# Learning and Evaluating Representations
for Deep One-class Classification

Kihyuk Sohn, Chun-Liang Li*, Jinsung Yoon, Minho Jin & Tomas Pfister
Google Cloud AI
<EMAIL_ADDRESS>Equal
contribution.

###### Abstract

We present a two-stage framework for deep one-class classification. We first
learn self-supervised representations from one-class data, and then build one-
class classifiers on learned representations. The framework not only allows to
learn better representations, but also permits building one-class classifiers
that are faithful to the target task. In particular, we present a novel
distribution-augmented contrastive learning that extends training
distributions via data augmentation to obstruct the uniformity of contrastive
representations. Moreover, we argue that classifiers inspired by the
statistical perspective in generative or discriminative models are more
effective than existing approaches, such as an average of normality scores
from a surrogate classifier. In experiments, we demonstrate state-of-the-art
performance on visual domain one-class classification benchmarks. Finally, we
present visual explanations, confirming that the decision-making process of
our deep one-class classifier is intuitive to humans. The code is available
at:

https://github.com/google-research/google-
research/tree/master/deep_representation_one_class.

##  1 Introduction

One-class classification aims to identify if an example belongs to the same
distribution as the training data. There are several applications of one-class
classification, such as anomaly detection or outlier detection, where we learn
a classifier that distinguishes the anomaly/outlier data without access to
them from the normal/inlier data accessible at training. The data could come
from various domains, such as industrial manufacturing (defect detection),
finance (fraudulent transaction detection), etc.

Generative models, such as kernel density estimation (KDE), have been popular
for one-class classification (Breunig et al., 2000; Latecki et al., 2007) as
they model the distribution by assigning high density to the training data,
and at test time, low density examples are determined as outliers.
Unfortunately, the curse of dimensionality hinders estimating the density of
the high-dimensional data accurately (Tsybakov, 2008). Deep generative models
(Kingma & Welling, 2013; Van Den Oord et al., 2016; Kingma & Dhariwal, 2018),
have demonstrated success in modeling high-dimensional data (e.g., images) and
have been applied to anomaly detection (Zhai et al., 2016; Zong et al., 2018;
Choi et al., 2018; Ren et al., 2019; Morningstar et al., 2020). However,
learning deep generative models on raw inputs remains challenging as they
appear to assign high density to background pixels (Ren et al., 2019) or learn
local pixel correlations (Kirichenko et al., 2020). A good representation
might still be beneficial to those models.

Alternately, discriminative models like one-class SVM (OC-SVM) (Schölkopf et
al., 2000) or support vector data description (SVDD) (Tax & Duin, 2004) learn
classifiers describing the support of one-class distributions to distinguish
them from outliers. These methods are more powerful with non-linear kernels
that implicitly transform the raw data into a high-dimensional feature space.
However, the performance of discriminative models is still limited by the
quality of data representations.

The fundamental limitation of one-class classification centers on learning
good high-level data representations. Following the success of deep learning
(LeCun et al., 2015), deep one-class classifications, which extend the
discriminative one-class classification using trainable deep neural networks,
have shown promising results compared to their kernel counterparts (Ruff et
al., 2018). However, a naive training of deep one-class classifiers leads to a
degenerate solution that maps all data into a single representation, also
known as “hypersphere collapse” (Ruff et al., 2018). Previous works circumvent
such issues by constraining network architectures (Ruff et al., 2018),
autoencoder pretraining (Ruff et al., 2018; Chalapathy et al., 2018), training
with a surrogate multi-class classification loss by introducing simulated
outliers (Hendrycks et al., 2018; Golan & El-Yaniv, 2018; Hendrycks et al.,
2019; Bergman & Hoshen, 2020) or injecting noise (Oza & Patel, 2018).

In this work, we present a two-stage framework for building deep one-class
classifiers. As in Figure 1, in the first stage, we train a deep network to
obtain a high-level data representation. Then, in the second stage, we build a
one-class classifier, such as OC-SVM or KDE, using representations from the
first stage. Comparing to those using surrogate losses (Golan & El-Yaniv,
2018; Hendrycks et al., 2019), our framework allows to build a classifier that
is more faithful to one-class classification. Decoupling representation
learning from classifier construction further opens up opportunities of using
state-of-the-art representation learning methods, such as self-supervised
contrastive learning (Chen et al., 2020). While vanilla contrastive
representations are less compatible with one-class classification as they are
uniformly distributed on the hypersphere (Wang & Isola, 2020), we show that,
with proper fixes, it provides representations achieving competitive one-class
classification performance to previous state-of-the-arts. Furthermore, we
propose a distribution-augmented contrastive learning, a novel variant of
contrastive learning with distribution augmentation (Jun et al., 2020). This
is particularly effective in learning representations for one-class
classification, as it reduces the class collision between examples from the
same class (Saunshi et al., 2019) and uniformity (Wang & Isola, 2020). Lastly,
though representations are not optimized for one-class classification as in
end-to-end trainable one-class classifiers (Ruff et al., 2018), we demonstrate
state-of-the-art performance on visual one-class classification benchmarks. To
summarize our contributions:

  * •

We present a two-stage framework for building deep one-class classifiers using
unsupervised and self-supervised representations followed by shallow one-class
classifiers.

  * •

We systematically study representation learning methods for one-class
classification, including augmentation prediction and contrastive learning,
and the proposed distribution-augmented contrastive learning method that
extends training data distributions via data augmentation.

  * •

We show that, with a good representation, both discriminative (OC-SVM) and
generative (KDE) classifiers, while being competitive with each other, are
better than surrogate classifiers based on the simulated outliers (Golan & El-
Yaniv, 2018; Hendrycks et al., 2019).

  * •

We achieve strong performance on visual one-class classification benchmarks,
such as CIFAR-10/100 (Krizhevsky, 2009), Fashion MNIST (Xiao et al., 2017),
Cat-vs-Dog (Elson et al., 2007) and CelebA (Liu et al., 2015). We make the
code available for reproducible research.

  * •

We extensively study the one-class contrastive learning and the realistic
evaluation of anomaly detection under unsupervised and semi-supervised
settings. Finally, we present visual explanations of our deep one-class
classifiers to better understand their decision making processes.

(a) Self-supervised representation learning

(b) One-class classifier

Figure 1: Overview of our two-stage framework for building deep one-class
classifier. (a) In the first stage, we learn representations from one-class
training distribution using self-supervised learning methods, and (b) in the
second stage, we train one-class classifiers using learned representations.

##  2 A Two-stage Framework for Deep One-Class Classification

In Section 2.1, we review self-supervised representation learning methods,
discuss how they connect to existing one-class classification methods, raise
issues of state-of-the-art contrastive representation learning (Chen et al.,
2020) for one-class classification, and propose ways to resolve these issues.
Then, in Section 2.2, we study how to leverage learned representations for
one-class classification.

###  2.1 Learning Representations for One-Class Classification

Let $\mathcal{A}$ be the stochastic data augmentation process, which is
composed of resize and crop, horizontal flip, color jittering, gray-scale and
Gaussian blur, following Chen et al. (2020). As in Figure 1, self-supervised
learning methods consist of a feature extractor $f$ parameterized by deep
neural networks and the proxy loss $\mathcal{L}$. Below, we discuss details of
different self-supervised learning methods.

| Proxy Task | Normality Score at Test Time ($x_{t}$)
---|---|---
Golan & El-Yaniv (2018) | $\mathcal{L}_{\mathrm{rot}}$ | $p_{q}\left(0|f(x_{t})\right)$ or $\sum_{y{\in}\\{0,1,2,3\\}}p_{q}\left(y|f(\mathrm{rot90}(x_{t},y))\right)$
Hendrycks et al. (2019)
Ours |  $\mathcal{L}_{\mathrm{rot}}$, $\mathcal{L}_{\mathrm{clr}}$, $\mathcal{L}_{\mathrm{clr}}^{\mathrm{distaug}}$, … |  $\mathrm{KDE}\left(f(x_{t}))\right)$ or $\mbox{OC-SVM}\left(f(x_{t}))\right)$

Table 1: A comparison between deep one-class classifiers based on self-
supervised learning. Previous works (Golan & El-Yaniv, 2018; Hendrycks et al.,
2019) train one-class classifiers using augmentation prediction (e.g.
$\mathcal{L}_{\mathrm{rot}}$) and determine outliers using augmentation
classifiers ($p_{q}$). We learn representations using proxy tasks of different
self-supervised learning methods, such as contrastive learning
($\mathcal{L}_{\mathrm{clr}}$), and build simple classifiers, such as KDE or
OC-SVM, on learned representations.

####  2.1.1 Augmentation Prediction

One popular method for learning representations learns by discriminating the
type of augmentation applied to the data. Rotation prediction (Gidaris et al.,
2018), which is one of the most representative of kinds, learns deep
representations by predicting the degree of rotation augmentations. The
training objective of the rotation prediction proxy task is written as
follows:

$\mathcal{L}_{\mathrm{rot}}=\mathbb{E}_{x\sim
P_{\mathcal{X}},\mathcal{A}}\Big{[}\mathrm{CrossEntropy}\big{(}y,p_{q\circ
f}\big{(}y|\mathrm{rot90}\left(\mathcal{A}(x),y)\right)\big{)}\Big{]}$ (1)

where $y\,{\in}\,\\{0,1,2,3\\}$ is a prediction target representing the
rotation degree, and $\mathrm{rot90}(x,y)$ rotates an input $x$ by
$90^{\circ}$ $y$ times. We represent the classifier $p_{q\circ f}(y|x)$ as
$p_{q}(y|f(x))\,{\propto}\,\exp(q\circ f(x))[y]$ which contains the
representation $f$ and a linear layer $q$ with 4 output units corresponding to
each rotation degree. It is shown that the learned network $f$ yields a good
representation for downstream classification tasks (Gidaris et al., 2018).

Application to One-Class Classification.  While proposed for learning
representations, augmentation prediction has been successfully adopted to
learn deep one-class classifiers (Golan & El-Yaniv, 2018; Hendrycks et al.,
2019; Bergman & Hoshen, 2020). Although it is not trained to do so, the
likelihood of learned rotation classifiers111For presentation clarity, we use
the rotation as an example for augmentations. Note that one may use more
geometric transformations, such as rotation, translation, or flip of an image,
as in Golan & El-Yaniv (2018); Hendrycks et al. (2019); Bergman & Hoshen
(2020). $p_{q}(y\,{=}\,0|f(x))$ is shown to well approximate normality score.
Plausible explanation is via outlier exposure (Hendrycks et al., 2018), where
the classifier learns a decision boundary distinguishing original images from
simulated outliers created by image rotation. However, it assumes inlier
images are not rotated. Moreover, the classifier may not generalize to one-
class classification task if it overfits to the proxy task (e.g., rotation
prediction).

####  2.1.2 Contrastive Learning

Unlike augmentation prediction that learns discriminative representations to
data augmentation, contrastive learning (Chen et al., 2020) learns
representation by distinguishing different views (e.g., augmentations) of
itself from other data instances. Let
$\phi(x)\,{=}\,\mathrm{normalize}(f(x))$, i.e., $\|\phi(x)\|\,{=}\,1$.
Following the formulation of Sohn (2016), the proxy task loss of contrastive
learning is written as:

$\mathcal{L}_{\mathrm{clr}}\,{=}\,-\mathbb{E}_{x,x_{i}\sim
P_{\mathcal{X}},\mathcal{A},\mathcal{A}^{\prime}}\Bigg{[}\log\frac{\exp\left(\frac{1}{\tau}\phi(\mathcal{A}(x))^{\top}\phi(\mathcal{A}^{\prime}(x))\right)}{\exp\left(\frac{1}{\tau}\phi(\mathcal{A}(x))^{\top}\phi(\mathcal{A}^{\prime}(x))\right)+\sum_{i=1}^{M{-}1}\exp\left(\frac{1}{\tau}\phi(\mathcal{A}(x))^{\top}\phi(\mathcal{A}(x_{i}))\right)}\Bigg{]}$

(2)

where $\mathcal{A}$ and $\mathcal{A}^{\prime}$ are identical but independent
stochastic augmentation processes for two different views of $x$.
$\mathcal{L}_{\mathrm{clr}}$ regularizes representations of the same instance
with different views ($\mathcal{A}(x),\mathcal{A}^{\prime}(x)$) to be similar,
while those of different instances
($\mathcal{A}(x),\mathcal{A}^{\prime}(x^{\prime})$) to be unlike.

Class Collision and Uniformity for One-Class Classification.  While
contrastive representations have achieved state-of-the-art performance on
visual recognition tasks (Oord et al., 2018; Hénaff et al., 2019; Chen et al.,
2020; Wang & Isola, 2020) and have been theoretically proved to be effective
for multi-class classification (Saunshi et al., 2019; Tosh et al., 2020), we
argue that this could be problematic for one-class classification.

First, a class collision (Saunshi et al., 2019). The contrastive loss in
Equation (2) is minimized by _maximizing_ the distance between representations
of negative pairs $(x,x_{i}),x\,{\neq}\,x_{i}$, even though they are from the
same class when applied to the one-class classification. This seems to
contradict to the idea of deep one-class classification (Ruff et al., 2018),
which learns representations by _minimizing_ the distance between
representations with respect to the center:
$\min_{f}\mathbb{E}_{x}\|f(x)-c\|^{2}$.

Second, a uniformity of representations (Wang & Isola, 2020). It is proved
that the optimal solution for the denominator of Equation (2) is _perfect
uniformity_ as $M\,{\rightarrow}\,\infty$ (Wang & Isola, 2020), meaning that
$\phi(x)$ follows a uniform distribution on the hypersphere. It is problematic
since one can always find an inlier $x\,{\in}\,\mathcal{X}$ in the proximity
to any outlier $x^{\prime}\,{\notin}\,\mathcal{X}$ on the hypersphere, as
shown in Figure 2(a). In contrast, with reduced uniformity as in Figure 2(b),
it is easier to isolate outliers from inliers.

(a) Perfect uniformity.

(b) Reduced uniformity.

(c) Distribution augmentation.

Figure 2: One-class classification on different levels of uniformity of inlier
distributions. (a) When representations are uniform, isolating outliers is
hard. (b) Reducing uniformity makes boundary between inlier and outlier clear.
(c) Distribution augmentation allows inlier distribution more compact.

One-Class Contrastive Learning.  First, to reduce the uniformity of
representations, we propose to use a moderate $M$ (batch size). This is in
contrast with previous suggestions to train with large $M$ for contrastive
representations to be most effective on multi-class classification tasks
(Hénaff et al., 2019; Bachman et al., 2019; Chen et al., 2020). The impact of
batch size $M$ for one-class classification will be discussed in Section 5.1.
In addition, we propose _distribution augmentation_ 222We note that the
distribution augmentation has been applied to generative modeling (Jun et al.,
2020) as a way of improving regularization and generalization via multi-task
learning. for one-class contrastive learning. The idea is that, instead of
modeling the training data distribution $P_{\mathcal{X}}$, we model the union
of augmented training distribution $P_{\bigcup_{a}a(\mathcal{X})}$, where
$a(\mathcal{X})\,{=}\,\\{a(x)|x\,{\in}\,\mathcal{X}\\}$. Note that
augmentation $a$ for augmenting distribution is disjoint from those for data
augmentation $\mathcal{A}$ that generates views. Inspired by Golan & El-Yaniv
(2018), we employ geometric transformations, such as rotation or horizontal
flip, for distribution augmentation. For example, as in Figure 3, $x$ and
$\mathrm{rot}90(x)$ (German shepherds in the top row) are considered as two
separate instances and therefore are encouraged to be distant in the
representation space. Not only increasing the number of data instances to
train on (e.g., distribution augmentation by rotating $90^{\circ}$,
$180^{\circ}$, $270^{\circ}$ increases the dataset by 4 times), but it also
eases the uniformity of representations on the resulted hypersphere. A
pictorial example is in Figure 2(c), where thanks to augmented distribution,
the inlier distribution may become more compact.

Figure 3: Distribution-augmented contrastive learning. Not only learning to
discriminate different instances from an original distribution (e.g., two
images of different dogs on the left), it also learns to discriminate
instances from different distributions created via augmentations, such as
rotations (e.g., two images of the same dog with different rotation degrees on
the top).

####  2.1.3 Extracting Richer Representation by Learning with Projection Head

Self-supervised representations are prone to overfitting to the proxy task.
Gidaris et al. (2018) show that $f$ is not the best representation for
downstream tasks as $f$ becomes overly discriminative to augmentation
prediction while losing useful information of the data. Instead, lower-layer
representations are suggested for downstream tasks. While the output of $f$
trained by contrastive loss has been theoretically proven to be useful for
multi-class classification (Saunshi et al., 2019; Tosh et al., 2020; Lee et
al., 2020), the efficacy of projection head parameterized by MLP has been
confirmed in practice (Chen et al., 2020; Grill et al., 2020).

The effect of projection heads may be explained from the information theory
perspective. To illustrate, we extend the network structure of augmentation
prediction as $q\circ g\circ f$, where $q$ is the linear classification head,
$g$ is the projection head, and $f$ outputs self-supervised representations
used for downstream tasks. We note that using an identity head $g(x)\,{=}\,x$
recovers the network structure of previous works (Gidaris et al., 2018; Golan
& El-Yaniv, 2018; Hendrycks et al., 2019). The data processing inequality
(Cover, 1999) $I(g\circ f(x);x)\,{\leq}\,I(f(x);x)$ tells that $f$ can retain
more information than $g$, thus more suitable for downstream tasks that are
not necessarily correlated with the proxy tasks. The effectiveness of deep MLP
projection head is empirically shown for both augmentation prediction and
contrastive learning in Section 4.

###  2.2 Building Deep One-Class Classifiers with Learned Representations

We present a two-stage framework for deep one-class classification that builds
one-class classifiers on learned representations as in Figure 1. Compared to
end-to-end training (Ruff et al., 2018; Golan & El-Yaniv, 2018; Hendrycks et
al., 2019; Bergman & Hoshen, 2020), our framework provides flexibility in
using various representations as the classifier is not bound to representation
learning. It also allows the classifier consistent with one-class
classification objective.

To construct a classifier, we revisit an old wisdom which considers the full
spectrum of the distribution of the learned data representation. For
generative approaches, we propose to use nonparametric kernel density
estimation (KDE) to estimate densities from learned representations. For
discriminative approaches, we train one-class SVMs (Schölkopf et al., 2000).
Both methods work as a black-box and in experiment we use the default training
setting except the kernel width where we reduce by 10 times than default. We
provide detailed classifier formulations in Appendix A.1.

####  2.2.1 Gradient-based Explanation of Deep One-Class Classifier

Explaining the decision making process helps users to trust deep learning
models. There have been efforts to visually explain the reason for model
decisions of multi-class classifiers (Zeiler & Fergus, 2014; Bach et al.,
2015; Zhou et al., 2016; Selvaraju et al., 2017; Sundararajan et al., 2017;
Adebayo et al., 2018; Kapishnikov et al., 2019) using the gradients computed
from the classifier. In this work, we introduce a gradient-based visual
explanation of one-class classification that works for any deep
representations. To construct an end-to-end differentiable decision function,
we employ a KDE detector built on top of any differentiable deep
representations: $\frac{\partial\mathrm{KDE}(f(x))}{\partial
x}=\frac{\partial\mathrm{KDE}(f(x))}{\partial f(x)}\frac{\partial
f(x)}{\partial x}$.

##  3 Related Work

One-class classification (Moya et al., 1993) has broad use cases, including
fraud detection (Phua et al., 2010), spam filtering (Santos et al., 2011),
medical diagnosis (Schlegl et al., 2017), manufacturing defect detection
(Bergmann et al., 2019), to name a few. Due to the lack of granular semantic
information for one-class data, methods for learning from unlabeled data have
been employed for one-class classification. Generative models, which model the
density of training data distribution, are able to determine outlier when the
sample shows low density (Schlegl et al., 2017; Zong et al., 2018; Huang et
al., 2019). These include simple methods such as kernel density estimation or
mixture models (Bishop, 2006), as well as advanced ones (Bengio et al., 2007;
Kingma & Welling, 2013; Goodfellow et al., 2014; Van Den Oord et al., 2016;
Van den Oord et al., 2016; Dinh et al., 2016; Kingma & Dhariwal, 2018).
However, the density from generative models for high-dimensional data could be
misleading (Nalisnick et al., 2018; Škvára et al., 2018; Choi et al., 2018;
Kirichenko et al., 2020). New detection mechanisms based on the typicality
(Nalisnick et al., 2019) or likelihood ratios (Ren et al., 2019) have been
proposed to improve out-of-distribution detection.

Self-supervised learning is commonly used for learning representations from
unlabeled data by solving proxy tasks, such as jigsaw puzzle (Noroozi &
Favaro, 2016), rotation prediction (Gidaris et al., 2018), clustering (Caron
et al., 2018), instance discrimination (Ye et al., 2019) and contrastive
learning (Oord et al., 2018; Chen et al., 2020; He et al., 2020). The learned
representations are then used for multi-class classification, or transfer
learning, all of which require labeled data for downstream tasks. When it
comes to its application to one-class classification, contrastive learning is
adopted to improve the out-of-distribution detection under multi-class setting
(Winkens et al., 2020), whereas our work focuses on learning from a single
class of examples, leading to propose a novel distribution-augmented
contrastive learning. Notably, learning to predict geometric transformations
(Golan & El-Yaniv, 2018; Hendrycks et al., 2019; Bergman & Hoshen, 2020)
extends rotation prediction to using more geometric transformations as
prediction targets. Unlike typical applications of self-supervised learning
where the classifier or projection head (Chen et al., 2020) are discarded
after training, the geometric transformation classifier is used as a surrogate
for one-class classification. As in Section 4.1, however, the surrogate
classifier optimized for the self-supervised proxy task is suboptimal for one-
class classification. We show that replacing it with simple one-class
classifiers consistently improve the performance. Furthermore, we propose
strategies for better representation learning for both augmentation prediction
and contrastive learning.

Distribution-augmented contrastive learning is concurrently developed in (Tack
et al., 2020) as a part of their multi-task ensemble model. While sharing a
similar technical formulation, we motivate from fixing the uniformity of
contrastive representations. We note that our study not only focuses on
representation learning, but also on the importance of detection algorithms.

##  4 Experiments

Following Golan & El-Yaniv (2018), we evaluate on one-class classification
benchmarks, including CIFAR-10, CIFAR-10033320 super class labels are used for
CIFAR-100 experiments (Golan & El-Yaniv, 2018). (Krizhevsky, 2009), Fashion-
MNIST (Xiao et al., 2017), and Cat-vs-Dog (Elson et al., 2007). Images from
one class are given as inlier and those from remaining classes are given as
outlier. We further propose a new protocol using CelebA eyeglasses dataset
(Liu et al., 2015), where face images with eyeglasses are denoted as an
outlier. It is more challenging since the difference between in and outlier
samples is finer-grained. We resize images into $64{\times}64$ for cat-vs-dog
and CelebA datasets and $32{\times}32$ for the rest.

We evaluate the performance of (1) representations trained with unsupervised
and self-supervised learning methods, including denoising autoencoder (Vincent
et al., 2008), rotation prediction (Gidaris et al., 2018), contrastive
learning (Ye et al., 2019; Chen et al., 2020), and (2) using different one-
class classifiers, such as OC-SVM or KDE. We use rotation augmentations for
distribution-augmented contrastive learning, denoted as Contrastive (DA). We
train a ResNet-18 (He et al., 2016) for feature extractor $f$ and a stack of
linear, batch normalization, and ReLU, for MLP projection head $g$. More
experiment details can be found in Appendix A.4.

###  4.1 Main Results

We report the mean and standard deviation of AUCs averaged over classes over 5
runs in Table 2. The mean of 5 datasets is weighted by the number of classes
for each dataset. Besides those using self-supervised representations, we
provide results using ImageNet pretrained ResNet-50 to highlight the
importance of learning representations from in-domain distribution.

ImageNet pretrained ResNet-50 achieves the best performance of $84.0$ mean AUC
over 5 datasets. Compared to representations learned with denoising objective,
it works particularly well on datasets such as CIFAR-100, cat-vs-dog, and
CelebA, which we attribute it to the subset of ImaegNet classes is closely
related to the classes of these datasets.

Similar to the findings from Golan & El-Yaniv (2018); Hendrycks et al. (2019),
we observe significant performance gains with self-supervised learning.
Moreover, while RotNet (Golan & El-Yaniv, 2018), an end-to-end trained
classifier using rotation prediction, achieves $83.1$ AUC, the RotNet
_representation_ evaluated with the _KDE detector_ achieves $86.6$,
emphasizing the importance of a proper detector in the second stage. Finally,
the quality of RotNet representation improves when trained with the MLP
projection head, resulting in $88.2$ AUC.

The representations learned with vanilla contrastive loss achieve $86.9$ AUC
with OC-SVM, which under-perform those trained with rotation prediction loss
($88.2$). The distribution-augmented contrastive loss achieves the highest
mean AUC of $\mathbf{89.9}$ among all methods, demonstrating performance gains
on all datasets by a large margin upon its vanilla counterparts.

Comparison to Previous Works.  We make comparisons to previous works in Table
3. While some comparisons may not be apples-to-apples as different works use
different implementations of networks, optimizers, etc., we note that our
implementation is based on the common choices of network (e.g., ResNet-18) and
optimizer (e.g., momentum SGD) for image classification. We advance the
previous state-of-the-art on one-class classification benchmarks by a large
margin without test-time augmentation nor ensemble of models. We further
improve the performance with model ensemble, which we report in Appendix
A.2.2.

Representation | Classifier | CIFAR-10 | CIFAR-100 | f-MNIST | Cat-vs-Dog | CelebA | Mean
---|---|---|---|---|---|---|---
ResNet-50 (ImageNet) | OC-SVM | 80.0 | 83.7 | 91.8 | 74.5 | 81.4 | 84.0
KDE | 80.0 | 83.7 | 90.5 | 74.6 | 82.4 | 83.7
RotNet (Golan & El-Yaniv, 2018) | Rotation Classifier | 86.8$\pm 0.4$ | 80.3$\pm 0.5$ | 87.4$\pm 1.7$ | 86.1$\pm 0.3$ | 51.4$\pm 3.9$ | 83.1
KDE | 89.3$\pm 0.3$ | 81.9$\pm 0.5$ | 94.6$\pm 0.3$ | 86.4$\pm 0.2$ | 77.4$\pm 1.0$ | 86.6
Denoising | OC-SVM | 83.4$\pm 1.0$ | 75.2$\pm 1.0$ | 93.9$\pm 0.4$ | 57.3$\pm 1.3$ | 66.8$\pm 0.9$ | 80.4
KDE | 83.5$\pm 1.0$ | 75.2$\pm 1.0$ | 93.7$\pm 0.4$ | 57.3$\pm 1.3$ | 67.0$\pm 0.7$ | 80.4
Rotation Prediction | OC-SVM | 90.8$\pm 0.3$ | 82.8$\pm 0.6$ | 94.6$\pm 0.3$ | 83.7$\pm 0.6$ | 65.8$\pm 0.9$ | 87.1
KDE | 91.3$\pm 0.3$ | 84.1$\pm 0.6$ | 95.8$\pm 0.3$ | 86.4$\pm 0.6$ | 69.5$\pm 1.7$ | 88.2
Contrastive | OC-SVM | 89.0$\pm 0.7$ | 82.4$\pm 0.8$ | 93.9$\pm 0.3$ | 87.7$\pm 0.5$ | 83.5$\pm 2.4$ | 86.9
KDE | 89.0$\pm 0.7$ | 82.4$\pm 0.8$ | 93.6$\pm 0.3$ | 87.7$\pm 0.4$ | 84.6$\pm 2.5$ | 86.8
Contrastive (DA) | OC-SVM | 92.5$\pm 0.6$ | 86.5$\pm 0.7$ | 94.8$\pm 0.3$ | 89.6$\pm 0.5$ | 84.5$\pm 1.1$ | 89.9
KDE | 92.4$\pm 0.7$ | 86.5$\pm 0.7$ | 94.5$\pm 0.4$ | 89.6$\pm 0.4$ | 85.6$\pm 0.5$ | 89.8

Table 2: We report the mean and standard deviation of one-class classification
AUCs averaged over classes over 5 runs. The best methods are bold-faced for
each setting. The per-class AUCs are reported in Appendix A.5. All methods are
implemented and evaluated under the same condition.

| CIFAR-10 | CIFAR-100 | f-MNIST | cat-vs-dog
---|---|---|---|---
Ruff et al. (2018) | 64.8 | – | – | –
Golan & El-Yaniv (2018)† | 86.0 | 78.7 | 93.5 | 88.8
Bergman & Hoshen (2020)† | 88.2 | – | 94.1 | –
Hendrycks et al. (2019)† | 90.1 | – | – | –
Huang et al. (2019)† | 86.6 | 78.8 | 93.9 | –
Ours: Rotation prediction | 91.3$\pm 0.3$ | 84.1$\pm 0.6$ | 95.8$\pm 0.3$ | 86.4$\pm 0.6$
Ours: Contrastive (DA) | 92.5$\pm 0.6$ | 86.5$\pm 0.7$ | 94.8$\pm 0.3$ | 89.6$\pm 0.5$

Table 3: Comparison to previous works on one-class classification. † denotes
evaluation methods using test time data augmentation. Our methods are both
more accurate and computationally efficient.

##  5 Analysis and Ablation Study

In Section 5.1, we analyze behaviors of one-class contrastive representations
and in Section 5.4, we report visual explanations of various deep one-class
classifiers. Due to a space constraint, more studies, including an in-depth
study on distribution-augmented contrastive representations and data
efficiency of self-supervised learning for one-class classification, are in
Appendix A.2.

(a) MMDs with various batch sizes.

(b) AUCs with various batch sizes.

(c) AUCs with various MLP depths.

Figure 4: Ablation study of contrastive representations trained with different
batch sizes. We use (a) the MMD distance between the representations and the
data sampled from uniform distributions to measure uniformity. Small MMD
distance means being similar to uniform distribution. We also report (b) one-
class classification performance in AUCs evaluated by kernel OC-SVMs and (c)
the performance with MLP heads of different depths.

(a) Unsupervised settings for learning representation and building detector
with various outlier ratio.

(b) Unsupervised setting for learning representation and 1% (50) one-class
data for building detector.

(c) Unsupervised setting for learning representation and 2% (100) one-class
data for building detector.

Figure 5: Realistic evaluation of anomaly detection under (5(a)) unsupervised
and (5(b), 5(c)) one-class or semi-supervised settings. For unsupervised
settings, we either learn representation or build detector from a training set
containing both inlier and outlier data without their labels. For one-class or
semi-supervised settings, we learn from a training set containing only a small
amount of one-class (inlier) data. We report AUCs on CIFAR-10 and OC-SVM with
RBF kernel is used for evaluation.

###  5.1 Uniformity, Batch Size and Distribution Augmentation

Oord et al. (2018); Hénaff et al. (2019); Bachman et al. (2019); Chen et al.
(2020) have shown substantial improvement on contrastive representations
evaluated on multi-class classification using very large batch sizes, which
results in uniformity. However, uniformity (Wang & Isola, 2020) and class
collision (Saunshi et al., 2019) can be an issue for one-class classification
as discussed in Section 2.1.2. Here we investigate the relations between
performance and uniformity, and how we can resolve the issue via batch size,
MLP head and distribution-augmented contrastive learning.

We measure the uniformity via MMD distance (Gretton et al., 2012) between the
learned representation and samples from uniform distributions on hypersphere.
Smaller MMD distance implies the distribution of representation is closer to
uniform distributions. We train models with various batch sizes
$\\{2^{3},\dots,2^{9}\\}$. We report in Figure 4 the performance over 5 runs
on CIFAR-10 validation set.

Distribution Augmentation.  In Figure 4(a), the representations from the
projection head $g\circ f$ trained by standard contrastive learning
(Contrastive $g\circ f$) are closer to be uniformly distributed when we
increase the batch size as proven by Wang & Isola (2020). Therefore, in one-
class classification, the nearly uniformly distributed representation from
contrastive learning is only slightly better than random guess (50 AUC) as in
Figure 4(b). In contrast, with distribution augmentations, representations
(Contrastive (DA) $g\circ f$) are less uniformly distributed and result in a
significant performance gain.

Batch Size.  Following the discussion, large batch sizes result in nearly
uniformly distributed representations ($g\circ f$), which is harmful to the
one-class classification. On the other hand, small batch sizes (${\leq}16$),
though less uniform, hinders us learning useful representations via
information maximization (Poole et al., 2019). As in Figure 4(b), there is a
trade-off of batch size for one-class classification, and we find batch size
of $32$ results in the best one-class classification performance.

MLP Projection Head.  As discussed in Section 2.1.3, $f$, an input to
projection head, contains no less information than the output $g\circ f$, but
overfits less to the proxy task. As in Figure 4(a), $f$’s are not at all
uniformly distributed and results in significantly better performance than
$g\circ f$, as in Figure 4(b).

Lastly, we emphasize that all these fixes contribute to an improvement of
contrastive representations for one-class classification. As in Figure 4 AUC
drops when any of these components are missing.

###  5.2 Analysis on Different Distribution Augmentations

The choice of distributions affects the performance. The ablation study using
horizontal flip ($\mathrm{hflip}$) and rotation augmentations is reported in
Figure 7. Note that $\mathrm{hflip}$ is used only to augment distribution in
this experiment. Interestingly, simply adding $\mathrm{hflip}$ improves the
AUC to $90.7$. This suggests a different insight from Tack et al. (2020) who
augments distribution as a means to outlier exposure (Hendrycks et al., 2018).
Although we report the numbers with rotation augmentations in Table 2, with
$\mathrm{hflip}$, $\mathrm{rot90}$, $\mathrm{rot90}{+}\mathrm{hflip}$ as
augmented distributions, we achieve the best mean AUC, $93.7$, on CIFAR-10,
without any test-time augmentation. More study follows in Appendix A.2.2.

###  5.3 Applications to Unsupervised and Semi-Supervised Anomaly Detection

We conduct experiments for unsupervised anomaly detection, where the training
set may contain a few outlier data.444We follow the definition of Chandola et
al. (2009) to distinguish unsupervised and semi-supervised settings of anomaly
detection. Please see Appendix A.3 for additional description on their
definitions. We study two settings: 1) Unsupervised settings for both learning
representation and building detector, and 2) unsupervised setting for learning
representation, but building detector with a small amount (as few as 50 data
instances) of one-class data only. For both settings, we vary the outlier
ratio in the training set from $0.5\%$ to $10\%$. We show results in Figure 5.
As in Figure 5(a), we observe the decrease in performance when increasing the
outlier ratio as expected. Rotation prediction is slightly more robust than
contrastive learning for high outlier ratio. On the other hand, when
classifier is built with clean one-class data, contrastive representations
performs better. Interestingly, contrastive learning benefits from outlier
data, as it naturally learn to distinguish inlier and outlier. Due to space
constraint, we provide more results and analysis in Appendix A.3.

###  5.4 Visual Explanation of Deep One-Class Classifiers

We investigate the decision making process of our deep one-class classifiers
using the tools described in Section 2.2.1. Specifically, we inspect by
highlighting the most influential regions using two popular visual explanation
algorithms, namely, integrated gradients (IG) (Sundararajan et al., 2017) and
GradCAM (Selvaraju et al., 2017), for distribution-augmented contrastive
representation, as well as RotNet and DAE on images from cat-vs-dog and CelebA
eyeglasses datasets. For RotNet, we test using both a rotation classifier and
KDE (RotNet∗ in Figure 6) to compute gradients.

As in Figure 6, the proposed visual explanation method based on the KDE one-
class classifier permits highlighting human-intuitive, meaningful regions in
the images, such as dog faces or eyeglasses instead of spurious background
regions (Figure 6(b)). On the other hand, even though the classification AUC
is not too worse ($86.1$ AUC on cat-vs-dog as opposed to $89.6$ for ours), the
visual explanation based on the rotation classifier (Figure 6(c)) suggests
that the decision may be made sometimes based on the spurious features (e.g.,
human face in the background). We present more examples for visual explanation
in Appendix A.6.

(a) Input

(b) Ours

(c) RotNet

(d) RotNet∗

(e) DAE

(f) Ours

(g) RotNet

(h) RotNet∗

(i) DAE

Figure 6: Visual explanations of deep one-class classifiers. (a) Input, (b–e)
images with heatmaps using integrated gradients (Sundararajan et al., 2017),
and (f–i) those using GradCAM (Selvaraju et al., 2017). RotNet∗: RotNet + KDE.
More examples are in Appendix A.6. Figure 7: AUCs of contrastive
representations trained from various augmented distributions on CIFAR-10. From
left to right, we train by accumulating distributions. For example, an entry
with “$\mathrm{+rot90,hflip}$” is trained with 4 distributions, i.e.,
$\mathrm{original}$, $\mathrm{hflip}$, $\mathrm{rot90}$ and
$\mathrm{rot90,hflip}$.

##  6 Conclusion

Inspired by an old wisdom of learning representations followed by building
classifiers, we present a two-stage framework for deep one-class
classification. We emphasize the importance of decoupling building classifiers
from learning representations, which allows classifier to be consistent with
the target task, one-class classification. Moreover, it permits applications
of various self-supervised representation learning methods, including
contrastive learning (Chen et al., 2020) with proper fixes, for one-class
classification, achieving strong performance on one-class classification
benchmarks.

#### Acknowledgments

We thank Fan Yang, Clayton Mellina, Ting Yu, Sercan Arik, and Klaus-Robert
Müller for helpful discussion. We thank Chen-Yu Lee for the proofread of our
manuscript.

## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: A system for large-scale machine learning.  In _12th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 16)_, pp. 265–283, 2016.
  * Adebayo et al. (2018) Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  In _Advances in Neural Information Processing Systems_ , pp. 9505–9515, 2018.
  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7):e0130140, 2015.
  * Bachman et al. (2019) Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  In _Advances in Neural Information Processing Systems_ , pp. 15535–15545, 2019.
  * Bengio et al. (2007) Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  In _Advances in neural information processing systems_ , pp. 153–160, 2007.
  * Bergman & Hoshen (2020) Liron Bergman and Yedid Hoshen.  Classification-based anomaly detection for general data.  In _Eighth International Conference on Learning Representations_ , 2020.
  * Bergmann et al. (2019) Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 9592–9600, 2019.
  * Bishop (2006) Christopher M Bishop.  _Pattern recognition and machine learning_.  springer, 2006.
  * Breunig et al. (2000) Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander.  Lof: identifying density-based local outliers.  In _Proceedings of the 2000 ACM SIGMOD international conference on Management of data_ , pp. 93–104, 2000.
  * Caron et al. (2018) Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 132–149, 2018.
  * Chalapathy et al. (2018) Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla.  Anomaly detection using one-class neural networks.  _arXiv preprint arXiv:1802.06360_ , 2018.
  * Chandola et al. (2009) Varun Chandola, Arindam Banerjee, and Vipin Kumar.  Anomaly detection: A survey.  _ACM computing surveys (CSUR)_ , 41(3):1–58, 2009\.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  _arXiv preprint arXiv:2002.05709_ , 2020.
  * Choi et al. (2018) Hyunsun Choi, Eric Jang, and Alexander A Alemi.  Waic, but why? generative ensembles for robust anomaly detection.  _arXiv preprint arXiv:1810.01392_ , 2018.
  * Cover (1999) Thomas M Cover.  _Elements of information theory_.  John Wiley & Sons, 1999.
  * Dinh et al. (2016) Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  _arXiv preprint arXiv:1605.08803_ , 2016.
  * Elson et al. (2007) Jeremy Elson, John R Douceur, Jon Howell, and Jared Saul.  Asirra: a captcha that exploits interest-aligned manual image categorization.  In _ACM Conference on Computer and Communications Security_ , volume 7, pp. 366–374, 2007.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In _Sixth International Conference on Learning Representations_ , 2018\.
  * Golan & El-Yaniv (2018) Izhak Golan and Ran El-Yaniv.  Deep anomaly detection using geometric transformations.  In _Advances in Neural Information Processing Systems_ , pp. 9758–9769, 2018.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pp. 2672–2680, 2014.
  * Gretton et al. (2012) Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773, 2012.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  _arXiv preprint arXiv:2006.07733_ , 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9729–9738, 2020.
  * Hénaff et al. (2019) Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  _arXiv preprint arXiv:1905.09272_ , 2019.
  * Hendrycks et al. (2018) Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich.  Deep anomaly detection with outlier exposure.  _arXiv preprint arXiv:1812.04606_ , 2018.
  * Hendrycks et al. (2019) Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song.  Using self-supervised learning can improve model robustness and uncertainty.  In _Advances in Neural Information Processing Systems_ , pp. 15637–15648, 2019.
  * Huang et al. (2019) Chaoqing Huang, Jinkun Cao, Fei Ye, Maosen Li, Ya Zhang, and Cewu Lu.  Inverse-transform autoencoder for anomaly detection.  _arXiv preprint arXiv:1911.10676_ , 2019.
  * Jun et al. (2020) Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever.  Distribution augmentation for generative modeling.  In _Proceedings of Machine Learning and Systems 2020_ , pp. 10563–10576, 2020.
  * Kapishnikov et al. (2019) Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry.  Xrai: Better attributions through regions.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 4948–4957, 2019.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma & Dhariwal (2018) Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  In _Advances in neural information processing systems_ , pp. 10215–10224, 2018.
  * Kirichenko et al. (2020) Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Why normalizing flows fail to detect out-of-distribution data.  _arXiv preprint arXiv:2006.08545_ , 2020.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * Latecki et al. (2007) Longin Jan Latecki, Aleksandar Lazarevic, and Dragoljub Pokrajac.  Outlier detection with kernel density functions.  In _International Workshop on Machine Learning and Data Mining in Pattern Recognition_ , pp. 61–75. Springer, 2007.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Lee et al. (2020) Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo.  Predicting what you already know helps: Provable self-supervised learning.  _arXiv preprint arXiv:2008.01064_ , 2020.
  * Li et al. (2017) Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos.  Mmd gan: Towards deeper understanding of moment matching network.  In _Advances in Neural Information Processing Systems_ , pp. 2203–2213, 2017.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 3730–3738, 2015.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  SGDR: Stochastic gradient descent with warm restarts.  In _Fifth International Conference on Learning Representations_ , 2017\.
  * Morningstar et al. (2020) Warren R Morningstar, Cusuh Ham, Andrew G Gallagher, Balaji Lakshminarayanan, Alexander A Alemi, and Joshua V Dillon.  Density of states estimation for out-of-distribution detection.  _arXiv preprint arXiv:2006.09273_ , 2020.
  * Moya et al. (1993) Mary M Moya, Mark W Koch, and Larry D Hostetler.  One-class classifier networks for target recognition applications.  _STIN_ , 93:24043, 1993.
  * Nalisnick et al. (2018) Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  _arXiv preprint arXiv:1810.09136_ , 2018.
  * Nalisnick et al. (2019) Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan.  Detecting out-of-distribution inputs to deep generative models using a test for typicality.  _arXiv preprint arXiv:1906.02994_ , 5, 2019.
  * Noroozi & Favaro (2016) Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In _European Conference on Computer Vision_ , pp. 69–84. Springer, 2016.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Oza & Patel (2018) Poojan Oza and Vishal M Patel.  One-class convolutional neural network.  _IEEE Signal Processing Letters_ , 26(2):277–281, 2018.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Phua et al. (2010) Clifton Phua, Vincent Lee, Kate Smith, and Ross Gayler.  A comprehensive survey of data mining-based fraud detection research.  _arXiv preprint arXiv:1009.6119_ , 2010.
  * Poole et al. (2019) Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker.  On variational bounds of mutual information.  In _International Conference on Machine Learning_ , pp. 5171–5180, 2019.
  * Ren et al. (2019) Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan.  Likelihood ratios for out-of-distribution detection.  In _Advances in Neural Information Processing Systems_ , pp. 14707–14718, 2019.
  * Ruff et al. (2018) Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft.  Deep one-class classification.  In _International conference on machine learning_ , pp. 4393–4402, 2018.
  * Santos et al. (2011) Igor Santos, Carlos Laorden, Xabier Ugarte-Pedrero, Borja Sanz, and Pablo G Bringas.  Spam filtering through anomaly detection.  In _International Conference on E-Business and Telecommunications_ , pp. 203–216. Springer, 2011.
  * Saunshi et al. (2019) Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar.  A theoretical analysis of contrastive unsupervised representation learning.  In _International Conference on Machine Learning_ , pp. 5628–5637, 2019.
  * Schlegl et al. (2017) Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery.  In _International conference on information processing in medical imaging_ , pp. 146–157. Springer, 2017.
  * Schölkopf et al. (2000) Bernhard Schölkopf, Robert C Williamson, Alex J Smola, John Shawe-Taylor, and John C Platt.  Support vector method for novelty detection.  In _Advances in neural information processing systems_ , pp. 582–588, 2000.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 618–626, 2017.
  * Škvára et al. (2018) Vít Škvára, Tomáš Pevnỳ, and Václav Šmídl.  Are generative deep models for novelty detection truly better?  _arXiv preprint arXiv:1807.05027_ , 2018.
  * Sohn (2016) Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In _Advances in neural information processing systems_ , pp. 1857–1865, 2016.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning_ , pp. 3319–3328, 2017.
  * Tack et al. (2020) Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  _arXiv preprint arXiv:2007.08176_ , 2020.
  * Tax & Duin (2004) David MJ Tax and Robert PW Duin.  Support vector data description.  _Machine learning_ , 54(1):45–66, 2004.
  * Tosh et al. (2020) Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu.  Contrastive learning, multi-view redundancy, and linear models.  _arXiv preprint arXiv:2008.10150_ , 2020.
  * Tsybakov (2008) Alexandre B Tsybakov.  _Introduction to nonparametric estimation_.  Springer Science & Business Media, 2008.
  * Van den Oord et al. (2016) Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with pixelcnn decoders.  In _Advances in neural information processing systems_ , pp. 4790–4798, 2016.
  * Van Den Oord et al. (2016) Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48_ , pp. 1747–1756, 2016\.
  * Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In _Proceedings of the 25th international conference on Machine learning_ , pp. 1096–1103, 2008.
  * Wang & Isola (2020) Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  _arXiv preprint arXiv:2005.10242_ , 2020.
  * Wilson et al. (2016) Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing.  Deep kernel learning.  In _Artificial intelligence and statistics_ , pp. 370–378, 2016\.
  * Winkens et al. (2020) Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al.  Contrastive training for improved out-of-distribution detection.  _arXiv preprint arXiv:2007.05566_ , 2020.
  * Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * Ye et al. (2019) Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang.  Unsupervised embedding learning via invariant and spreading instance feature.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 6210–6219, 2019.
  * Zeiler & Fergus (2014) Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , pp. 818–833. Springer, 2014.
  * Zhai et al. (2016) Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang.  Deep structured energy based models for anomaly detection.  _arXiv preprint arXiv:1605.07717_ , 2016.
  * Zhou et al. (2016) Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2921–2929, 2016.
  * Zong et al. (2018) Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen.  Deep autoencoding gaussian mixture model for unsupervised anomaly detection.  In _International Conference on Learning Representations_ , 2018.

##  Appendix A Appendix

###  A.1 Formulations of One-Class Classifiers

For completeness, we provide formulations of one-class classifiers, such as
one-class support vector machine (OC-SVM) (Schölkopf et al., 2000) and kernel
density estimator, used in our work.

One-Class Support Vector Machine solves the following optimization problem to
find support vectors describing the boundary of one-class distribution:

$\displaystyle\min_{w,\rho,\xi_{i}}\frac{\|w\|^{2}}{2}+\frac{1}{\nu
n}\sum_{i=1}^{n}\xi_{i}-\rho$ (3) $\displaystyle{\mbox{subject to
}}w^{\top}f_{i}\geq\rho-\xi_{i},\,\xi_{i}\geq 0,\forall i=1,\cdots,n$ (4)

where $f_{i}\,{=}\,f(x_{i})$ is a feature map. The decision score is given as
follows:

$s(x)=\sum_{i=1}^{n}\alpha_{i}k(x_{i},x)-\rho$ (5)

with coefficients $\alpha_{i}\,{>}\,0$ for support vectors. Linear or RBF
($k_{\gamma}(x,y)=\exp\left(-\gamma\|x-y\|^{2}\right)$) kernels are used for
experiments.

Kernel Density Estimation is a nonparametric density estimation algorithm. The
normality score of KDE with RBF kernel parameter $\gamma$ is written as
follows:

$\mathrm{KDE}_{\gamma}(x)\propto\frac{1}{\gamma}\sum_{y}\exp\left(-\gamma\|x-y\|^{2}\right)$
(6)

###  A.2 Additional Ablation Study

####  A.2.1 Well Behaved Contrastive Representation

##### Linear Separability.

A common belief is the good representations are linearly separable with
respect to some underlying labels, which is supported by several empirical
success (Chen et al., 2020). The linear classifiers are proven to be nearly
optimal (Tosh et al., 2020) when we learn representations on data with _all
possible labels_ along with some additional assumptions. In one-class
classification, although we violate the assumptions that we only train on one
class of data, we are interested in the linear separability of the data to
understand the characteristics of the learned embedding. To this end, in
addition to training RBF kernel OC-SVM, we train OC-SVM with linear kernels on
the learned embedding. We note that it is atypical to use linear OC-SVM, which
is usually considered to be suboptimal for describing decision boundaries of
one-class classification problems. However, with good representations, linear
classifiers show good performance on one-class classification as shown in
Table 4, and are better than existing works (Golan & El-Yaniv, 2018) in Table
2. Lastly, we note that OC-SVM with nonlinear RBF kernel is still better than
linear models in Table 4. Similar observations are found in other problems
such as regression (Wilson et al., 2016) and generative models (Li et al.,
2017).

##### Parametric Models.

In addition to studying from the perspective of discriminative model, we also
dive into the generative model side. Instead of using the nonparametric KDE
for density estimation, we use a parametric model, single multivariate
Gaussian, whose density is defined as

$p(x)\propto\det\big{(}{\Sigma}\big{)}^{-\frac{1}{2}}\exp\big{\\{}-\frac{1}{2}(f(x)-\mu)^{\top}\Sigma^{-1}(f(x)-\mu)\big{\\}}.$
(7)

A shown in Table 4, the Gaussian density estimation (GDE) model shows
competitive performance as KDE. It suggests the learned representations from
distribution-augmented contrastive learning is _compact_ as our expectation,
which can be well approximated by a simple parametric model with single
Gaussian. Compared with nonparametric methods, although parametric models have
strong assumptions on the underlying data distributions, using parametric
model is more data efficient if the assumption holds555The error convergence
rate is $O(n^{-1/2})$, while that of nonparametric KDE is $O(n^{-1/d})$
(Tsybakov, 2008).. In addition, parametric models have huge advantage in
computation during testing. Therefore, there is trade-off between assumptions
of the data as well as model flexibility and computational efficiency. We note
that the single Gaussian parametric model may not be a good alternative of KDE
universally. A candidate with good trade-off is Gaussian Mixture Models, which
can be treated as a middle ground of nonparametric KDE and parametric single
Gaussian model. We leave the study for future works.

Representation | Classifier | CIFAR-10 | CIFAR-100 | f-MNIST | Cat-vs-Dog | CelebA | Mean
---|---|---|---|---|---|---|---
Contrastive (DA) | OC-SVM (linear) | 90.7$\pm 0.8$ | 81.1$\pm 1.3$ | 93.7$\pm 0.8$ | 86.3$\pm 0.7$ | 88.4$\pm 1.4$ | 86.7
OC-SVM (kernel) | 92.5$\pm 0.6$ | 86.5$\pm 0.7$ | 94.8$\pm 0.3$ | 89.6$\pm 0.5$ | 84.5$\pm 1.1$ | 89.9
KDE | 92.4$\pm 0.7$ | 86.5$\pm 0.7$ | 94.5$\pm 0.4$ | 89.6$\pm 0.4$ | 85.6$\pm 0.5$ | 89.8
GDE | 92.0$\pm 0.5$ | 86.0$\pm 0.8$ | 95.5$\pm 0.3$ | 88.9$\pm 0.3$ | 90.6$\pm 0.4$ | 89.8

Table 4: One-class classification results using different one-class
classifiers on rotation-augmented contrastive representations. In addition to
OC-SVM and KDE, both of which with RBF kernels, we test with the linear OC-SVM
and the Gaussian density estimator (GDE).

####  A.2.2 Analysis on Distribution Augmented Contrastive Representations

Relation to Outlier Exposure (Hendrycks et al., 2018).  Distribution
augmentation shares a similarity to outlier exposure (Hendrycks et al., 2018)
in that both methods introduce new data distributions for training. However,
outlier exposure requires a stronger assumption on the data distribution that
introduced outlier should not overlap with inlier distribution, while such an
assumption is not required for distribution augmentation. For example, let’s
consider rotation prediction as an instance of outlier exposure. When training
the model on randomly-rotated CIFAR-10, where we randomly rotate images of
CIFAR-10, the performance of rotation prediction representation drops to
$60.7$ AUC, which is slightly better than a random guess. On the other hand,
rotation-augmented contrastive representation is not affected by random
rotation and achieves $92.4$ AUC.

Ensemble of Contrastive Representations.  We note that most previous methods
have demonstrated improved performance via extensive test-time data
augmentation (Golan & El-Yaniv, 2018; Hendrycks et al., 2019; Bergman &
Hoshen, 2020; Tack et al., 2020). While already achieving state-of-the-art
one-class classification performance without test-time data augmentation, we
observe marginal improvement using test time data augmentation. Instead, we
find that an ensemble of classifiers built on different representations
significantly improve the performance. In Table 5, we report the performance
of distribution-augmented contrastive representations trained with different
sets of augmented distributions on CIFAR-10. We observe that ensemble of 5
models trained with different seeds consistently improves the performance.
Moreover, not only we improve one-class classification AUCs when aggregating
scores across models trained with different distribution augmentations, we
also observe lower standard deviation across different seeds. Finally, when
ensemble over $5{\times}5=25$ models, we achieve $94.6$ AUC.

Representations | +$\mathrm{hflip}$,+$\mathrm{rot90}$ | +$\mathrm{rot90}$,$\mathrm{hflip}$ | +$\mathrm{rot180}$ | +$\mathrm{rot180}$,$\mathrm{hflip}$ | +$\mathrm{rot270}$ | Ensemble of 5
---|---|---|---|---|---|---
single model | 93.1$\pm 0.4$ | 93.7$\pm 0.6$ | 93.6$\pm 0.4$ | 93.5$\pm 0.5$ | 93.1$\pm 0.8$ | 94.4$\pm 0.2$
ensemble of 5 models | 93.9 | 94.4 | 94.3 | 94.2 | 93.9 | 94.6

Table 5: Performance of single and ensemble models of distribution augmented
contrastive representations on CIFAR-10. For each augmented distribution, we
report the mean and standard deviation of single model performance (“single
model”) and that of ensemble model whose ensemble score is aggregated from 5
models trained with different random seeds (“ensemble of 5 models”). “Ensemble
of 5” aggregates score from 5 models with different augmentation
distributions.

Experiments on All Datasets. We evaluate the performance of distribution-
augmented contrastive representations on all datasets.

Augmentation | CIFAR-10 | CIFAR-100 | f-MNIST | Cat-vs-Dog | Mean
---|---|---|---|---|---
– | 88.7$\pm 0.6$ | 80.5$\pm 0.9$ | 93.8$\pm 0.3$ | 85.4$\pm 0.6$ | 85.9
+$\mathrm{hflip}$ | 90.7$\pm 0.3$ | 82.4$\pm 0.8$ | 93.6$\pm 0.5$ | 88.3$\pm 0.2$ | 87.3
+$\mathrm{rot90}$ | 93.1$\pm 0.4$ | 85.9$\pm 0.7$ | 94.5$\pm 0.3$ | 89.9$\pm 0.4$ | 89.9
+$\mathrm{rot90,hflip}$ | 93.7$\pm 0.6$ | 87.4$\pm 0.6$ | 94.6$\pm 0.6$ | 91.0$\pm 0.5$ | 90.8
+$\mathrm{rot180}$ | 93.6$\pm 0.4$ | 86.8$\pm 0.7$ | 94.7$\pm 0.4$ | 91.1$\pm 0.3$ | 90.5
+$\mathrm{rot180,hflip}$ | 93.5$\pm 0.5$ | 86.7$\pm 0.8$ | 94.5$\pm 0.4$ | 90.7$\pm 0.2$ | 90.4
+$\mathrm{rot270}$ | 93.1$\pm 0.8$ | 86.4$\pm 0.8$ | 94.3$\pm 0.5$ | 90.6$\pm 0.6$ | 90.1
+$\mathrm{rot270,hflip}$ | 93.0$\pm 0.5$ | 86.4$\pm 0.9$ | 94.5$\pm 0.5$ | 90.6$\pm 0.5$ | 90.1
Table 6: Performance of distribution augmented contrastive representations on
CIFAR-10, CIFAR-100, f-MNIST and Cat-vs-Dog datasets. For each row, we include
training examples from distribution via the specified augmentation.
Representations are evaluated via OC-SVM using RBF kernel.

####  A.2.3 Data Efficiency of Self-Supervised Representation Learning

While previous works on self-supervised learning have demonstrated the
effectiveness on learning from large-scale unlabeled data, not much has shown
for the data efficiency of these methods. Unlike multi-class classification
tasks where the amount of data could scale multiplicative with the number of
classes, data efficiency of representation learning becomes of particular
interest for one-class classification as it is hard to collect large-scale
data even without an annotation.

We present one-class classification AUCs of representations with various
training data sizes, along with two baseline representations such as ResNet-18
using random weights or ImageNet-pretrained ResNet-50 in Figure 8. Note that
we vary the training set sizes at representation learning phase only, but use
a fixed amount ($5000$) to train classifiers for fair evaluation of the
representation quality. We find that even with 50 examples, classifiers
benefit from self-supervised learning when comparing against raw or deep
features with random weights. The proposed distribution-augmented contrastive
loss could match the performance of ImageNet-pretrained ResNet-50 with only
$100$ examples, while rotation prediction loss and vanilla contrastive loss
require $250$ and $1000$ examples, respectively.

Figure 8: Self-supervised representations are trained from different data
sizes, from 50 to 5000, on CIFAR-10. Standard deviations are obtained by
sampling subsets 5 times. Classifiers are trained with the full (5000) train
set for fair evaluation of representations. We provide baseline
representations of ResNet-18 using random weights and ImageNet-pretrained
ResNet-50. Standard deviations are computed by running 5 times with different
seeds.

Representation | Classifier | CIFAR-10 | CIFAR-100 | f-MNIST | cat-vs-dog | CelebA | Mean
---|---|---|---|---|---|---|---
ResNet-18 (Random) | OC-SVM (linear) | 50.1$\pm 0.9$ | 50.1$\pm 1.7$ | 50.3$\pm 0.9$ | 50.0$\pm 0.4$ | 49.9$\pm 0.3$ | 50.1
OC-SVM (kernel) | 61.2$\pm 2.3$ | 59.6$\pm 1.9$ | 89.8$\pm 0.9$ | 50.7$\pm 0.5$ | 57.1$\pm 0.6$ | 66.5
KDE | 60.5$\pm 2.4$ | 58.7$\pm 2.2$ | 89.4$\pm 1.0$ | 50.8$\pm 0.6$ | 57.3$\pm 0.8$ | 65.9
ResNet-50 (ImageNet) | OC-SVM (linear) | 67.9 | 71.0 | 77.0 | 61.0 | 58.0 | 70.9
OC-SVM (kernel) | 80.0 | 83.7 | 91.8 | 74.5 | 81.4 | 84.0
KDE | 80.0 | 83.7 | 90.5 | 74.6 | 82.4 | 83.7
RotNet | Rotation Classifier | 86.8$\pm 0.4$ | 80.3$\pm 0.5$ | 87.4$\pm 1.7$ | 86.1$\pm 0.3$ | 51.4$\pm 3.9$ | 83.1
(Golan & El-Yaniv, 2018) | KDE | 89.3$\pm 0.3$ | 81.9$\pm 0.5$ | 94.6$\pm 0.3$ | 86.4$\pm 0.2$ | 77.4$\pm 1.0$ | 86.6
Denoising | OC-SVM (linear) | 72.6$\pm 1.8$ | 62.4$\pm 2.1$ | 55.0$\pm 3.9$ | 61.0$\pm 1.5$ | 59.8$\pm 4.0$ | 62.9
OC-SVM (kernel) | 83.4$\pm 1.0$ | 75.2$\pm 1.0$ | 93.9$\pm 0.4$ | 57.3$\pm 1.3$ | 66.8$\pm 0.9$ | 80.4
KDE | 83.5$\pm 1.0$ | 75.2$\pm 1.0$ | 93.7$\pm 0.4$ | 57.3$\pm 1.3$ | 67.0$\pm 0.7$ | 80.4
Rotation Prediction | OC-SVM (linear) | 88.5$\pm 0.8$ | 72.3$\pm 1.7$ | 89.0$\pm 1.5$ | 81.4$\pm 1.4$ | 62.6$\pm 4.1$ | 80.1
OC-SVM (kernel) | 90.8$\pm 0.3$ | 82.8$\pm 0.6$ | 94.6$\pm 0.3$ | 83.7$\pm 0.6$ | 65.8$\pm 0.9$ | 87.1
KDE | 91.3$\pm 0.3$ | 84.1$\pm 0.6$ | 95.8$\pm 0.3$ | 86.4$\pm 0.6$ | 69.5$\pm 1.7$ | 88.2
Contrastive | OC-SVM (linear) | 85.6$\pm 1.2$ | 74.9$\pm 1.4$ | 92.3$\pm 1.1$ | 82.8$\pm 1.1$ | 91.7$\pm 0.7$ | 82.2
OC-SVM (kernel) | 89.0$\pm 0.7$ | 82.4$\pm 0.8$ | 93.9$\pm 0.3$ | 87.7$\pm 0.5$ | 83.5$\pm 2.4$ | 86.9
KDE | 89.0$\pm 0.7$ | 82.4$\pm 0.8$ | 93.6$\pm 0.3$ | 87.7$\pm 0.4$ | 84.6$\pm 2.5$ | 86.8
Contrastive (DA) | OC-SVM (linear) | 90.7$\pm 0.8$ | 81.1$\pm 1.3$ | 93.7$\pm 0.8$ | 86.3$\pm 0.7$ | 88.4$\pm 1.4$ | 86.7
OC-SVM (kernel) | 92.5$\pm 0.6$ | 86.5$\pm 0.7$ | 94.8$\pm 0.3$ | 89.6$\pm 0.5$ | 84.5$\pm 1.1$ | 89.9
KDE | 92.4$\pm 0.7$ | 86.5$\pm 0.7$ | 94.5$\pm 0.4$ | 89.6$\pm 0.4$ | 85.6$\pm 0.5$ | 89.8

Table 7: One-class classification results using different representations and
one-class classifiers. We report the mean and standard deviation over 5 runs
of AUCs averaged over classes. The best methods are bold-faced for each
setting.

###  A.3 Unsupervised and Semi-Supervised Anomaly Detection

In this section, we conduct experiments for unsupervised anomaly detection.
Note that one-class classification assumes access to training data drawn
entirely from one-class, inlier distribution and is often referred to semi-
supervised anomaly detection (Chandola et al., 2009) as it requires human
effort to filter out training data from outlier distribution. On the other
hand, unsupervised anomaly detection assumes to include training data both
from inlier and outlier distributions without knowing their respective labels.
In other words, unsupervised anomaly detection may be viewed as an one-class
classification with noisy data.

Here, we are interested in analyzing the impact of label noise (e.g., outlier
examples are given as inlier) on one-class classification methods. To this
end, we conduct experiments under unsupervised settings that includes
different ratios, such as $0.5\%$, $1\%$, $2\%$, $5\%$, or $10\%$, of outlier
examples in the train set without their labels. Note that the total amount of
training examples for different settings remain unchanged. We show results in
Figure 9 of 4 models: rotation prediction without (“RotNet”) and with
(“Rotation Prediction”) MLP projection head, contrastive (“Contrastive”) and
distribution-augmented contrastive (“Contrastive (DA)”) learning. We also
report thee performance evaluated with different classifiers, including OC-SVM
with RBF (Figure 9(a)) and linear (Figure 9(c)) kernels, and Gaussian Density
Estimator (GDE, Figure 9(b)). For all models, we observe performance
degradation of deep one-class classifiers trained with outlier examples as
expected. Rotation prediction has shown more robust when outlier ratio is high
($5$ or $10\%$). While showing stronger performance under one-class setting,
kernel-based classifiers (OC-SVM with RBF kernel, KDE) have shown less robust
than parameteric (GDE) or linear classifiers under high outlier ratios, as
kernel-based methods focus more on the local neighborhood structures to
determine a decision boundary.

In addition, we conduct a study under another semi-supervised setting, where
we are given a small amount of labeled inlier data and a large amount of
unlabeled training data composed of both inlier and outlier examples. Note
that this is more realistic scenario for anomaly detection problems since it
is easier to obtain some portions of labeled inlier examples than outlier
examples. To demonstrate its effectiveness, we apply our proposed framework
without any modification by first training representations on unlabeled
training data and then building classifiers on these representations using
small amount of inlier data. Results are shown in Figure 10. Interestingly, we
observe consistent improvement in classification performance for contrastive
representations trained on data with higher proportions of outlier examples
when classifier is trained on a pure one-class data. Plausible explanation is
that the model learns better contrastive representations when trained with
both inlier and outlier as it naturally learn to distinguish inlier and
outlier. On the other hand, representations from rotation prediction still
show performance degradation as it learns to classify inlier and outlier into
the same category.

(a) OC-SVM (RBF kernel)

(b) Gaussian density estimation

(c) OC-SVM (linear kernel)

Figure 9: Classification performance under unsupervised learning setting of
representation and classifier. For unsupervised setting, training set contains
both inlier and outlier examples without their labels, whereas for one-class
setting, training set contains only inlier examples. For representations
trained with different outlier ratios, classification performances are
evaluated with different classifiers, such as OC-SVMs with RBF and linear
kernels, and Gaussian density estimation.

(a) 1% (50) one-class data

(b) 2% (100) one-class data

(c) 10% (500) one-class data

Figure 10: Classification performance under unsupervised representation
learning and one-class classifier learning settings. For unsupervised
representation learning, training set contains both inlier and outlier
examples without their labels. For one-class classifier learning, a small
portion of inlier data is used for training an OC-SVM with RBF kernel.

###  A.4 Details of Experimental Setting

Unless otherwise stated, models are trained for 2048 epochs with momentum
($0.9$) SGD and a single cycle of cosine learning rate decay (Loshchilov &
Hutter, 2017). L2 weight regularization with coefficient of $0.0003$ is
applied. We use scikit-learn (Pedregosa et al., 2011) implementation of OC-
SVMs with default value of $\nu$. We use
$\gamma\,{=}\,\nicefrac{{10}}{{|f|\cdot\mathrm{Var}(f)}}$, which is 10 times
larger than the default value for kernel OC-SVM. Same value of $\gamma$ is
used for KDE. We use scikit-learn implementation of Gaussian mixture model
using a single component for Gaussian density estimator (GDE). No
hyperparameters are tuned for GDE. Finally, all experiments are conducted
using TensorFlow (Abadi et al., 2016).

Other hyperparameters, such as learning rate or the MLP projection head depth,
are cross-validated using small labeled data. While this may violate the
assumption of one-class classification, supervised model selection is
inevitable for deep learning models as their behaviors may be largely
dependent on hyperparameters. To this end, we use 10% of inlier ($500$) and
the same number of outlier examples of CIFAR-10 for hyperparameter selection,
and use the same set of hyperparameters to test methods on other datasets,
which could demonstrate the algorithm robustness. Learning rate
${\in}\\{0.1,0.03,0.01,0.003,0.001\\}$ and the depth ${\in}\\{0,\cdots,8\\}$
of an MLP projection head are tuned for all methods. In addition, the
temperature $\tau{\in}\\{1,0.5,0.2,0.1\\}$ and the batch size
${\in}\\{2^{n},n=3,...,9\\}$ of contrastive loss are tuned. To this end, we
train all models across all datasets using the same hyperparameter
configurations, such as learning rate of $0.01$, projection head of depth $8$
($[512{\times}8,128]$), temperature $\tau$ of $0.2$, or batch size of $32$.

###  A.5 Per-Class AUCs

Representation | RotNet (Golan & El-Yaniv, 2018) | Denoising | Rot. Prediction | Contrastive | Contrastive (DA)
---|---|---|---|---|---
Classifier | Rot. Cls | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE
airplane | 80.3$\pm 0.4$ | 81.4$\pm 0.3$ | 81.6$\pm 0.7$ | 81.6$\pm 0.7$ | 83.6$\pm 0.6$ | 88.5$\pm 0.3$ | 88.8$\pm 0.3$ | 88.8$\pm 0.2$ | 90.9$\pm 0.5$ | 91.0$\pm 0.5$
automobile | 91.2$\pm 0.3$ | 96.7$\pm 0.1$ | 92.4$\pm 0.4$ | 92.4$\pm 0.4$ | 96.9$\pm 0.2$ | 97.5$\pm 0.3$ | 97.5$\pm 0.2$ | 97.5$\pm 0.2$ | 98.9$\pm 0.1$ | 98.9$\pm 0.1$
bird | 85.3$\pm 0.3$ | 86.4$\pm 0.3$ | 75.9$\pm 1.3$ | 75.9$\pm 1.3$ | 87.9$\pm 0.4$ | 88.2$\pm 0.3$ | 87.7$\pm 0.4$ | 87.7$\pm 0.4$ | 88.1$\pm 0.1$ | 88.0$\pm 0.2$
cat | 78.1$\pm 0.3$ | 78.9$\pm 0.3$ | 72.3$\pm 1.9$ | 72.3$\pm 1.9$ | 79.0$\pm 0.6$ | 78.3$\pm 1.0$ | 82.0$\pm 0.4$ | 82.1$\pm 0.4$ | 83.1$\pm 0.8$ | 83.2$\pm 0.9$
deer | 85.9$\pm 0.5$ | 88.0$\pm 0.6$ | 82.3$\pm 0.6$ | 82.4$\pm 0.6$ | 90.5$\pm 0.2$ | 90.2$\pm 0.2$ | 82.4$\pm 1.8$ | 82.3$\pm 1.8$ | 89.9$\pm 1.3$ | 89.4$\pm 1.8$
dog | 86.7$\pm 0.4$ | 88.1$\pm 0.3$ | 83.1$\pm 0.6$ | 83.1$\pm 0.6$ | 89.5$\pm 0.4$ | 88.2$\pm 0.6$ | 89.2$\pm 0.4$ | 89.2$\pm 0.5$ | 90.3$\pm 1.0$ | 90.2$\pm 1.2$
frog | 89.6$\pm 0.5$ | 90.7$\pm 0.4$ | 86.7$\pm 0.6$ | 86.8$\pm 0.6$ | 94.1$\pm 0.3$ | 94.6$\pm 0.3$ | 89.7$\pm 1.4$ | 89.8$\pm 1.4$ | 93.5$\pm 0.6$ | 93.5$\pm 0.6$
horse | 93.3$\pm 0.6$ | 95.7$\pm 0.3$ | 91.2$\pm 0.4$ | 91.2$\pm 0.4$ | 96.7$\pm 0.1$ | 97.0$\pm 0.1$ | 95.6$\pm 0.2$ | 95.6$\pm 0.2$ | 98.2$\pm 0.1$ | 98.1$\pm 0.1$
ship | 91.8$\pm 0.5$ | 94.0$\pm 0.2$ | 78.0$\pm 2.6$ | 78.0$\pm 2.6$ | 95.0$\pm 0.2$ | 95.7$\pm 0.1$ | 86.0$\pm 0.7$ | 85.8$\pm 0.6$ | 96.5$\pm 0.3$ | 96.5$\pm 0.3$
truck | 86.0$\pm 0.6$ | 93.0$\pm 0.3$ | 91.0$\pm 0.6$ | 91.0$\pm 0.6$ | 94.9$\pm 0.2$ | 94.9$\pm 0.2$ | 90.6$\pm 0.9$ | 90.7$\pm 0.9$ | 95.2$\pm 1.3$ | 95.1$\pm 1.3$
mean | 86.8$\pm 0.4$ | 89.3$\pm 0.3$ | 83.4$\pm 1.0$ | 83.5$\pm 1.0$ | 90.8$\pm 0.3$ | 91.3$\pm 0.3$ | 89.0$\pm 0.7$ | 89.0$\pm 0.7$ | 92.5$\pm 0.6$ | 92.4$\pm 0.7$

Table 8: Per-class one-class classification AUCs on CIFAR-10.

Representation | RotNet (Golan & El-Yaniv, 2018) | Denoising | Rot. Prediction | Contrastive | Contrastive (DA)
---|---|---|---|---|---
Classifier | Rot. Cls | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE
0 | 77.7$\pm 0.6$ | 79.1$\pm 0.6$ | 74.6$\pm 0.7$ | 74.4$\pm 0.8$ | 77.1$\pm 0.8$ | 78.8$\pm 0.6$ | 79.9$\pm 1.0$ | 79.9$\pm 1.0$ | 83.0$\pm 1.4$ | 82.9$\pm 1.4$
1 | 73.0$\pm 0.9$ | 75.4$\pm 0.8$ | 71.8$\pm 1.6$ | 71.8$\pm 1.7$ | 75.2$\pm 0.9$ | 77.9$\pm 1.1$ | 81.1$\pm 0.7$ | 81.1$\pm 0.8$ | 84.6$\pm 0.3$ | 84.3$\pm 0.4$
2 | 72.9$\pm 0.6$ | 76.1$\pm 0.7$ | 81.4$\pm 0.5$ | 81.4$\pm 0.5$ | 84.4$\pm 0.3$ | 85.7$\pm 0.4$ | 87.4$\pm 0.4$ | 87.5$\pm 0.4$ | 88.5$\pm 0.6$ | 88.6$\pm 0.6$
3 | 79.6$\pm 0.6$ | 81.3$\pm 0.3$ | 77.6$\pm 1.6$ | 77.6$\pm 1.6$ | 85.1$\pm 0.6$ | 87.3$\pm 0.4$ | 84.7$\pm 0.8$ | 84.7$\pm 0.8$ | 86.3$\pm 0.8$ | 86.4$\pm 0.8$
4 | 78.4$\pm 0.4$ | 80.8$\pm 0.5$ | 76.1$\pm 1.5$ | 76.1$\pm 1.5$ | 81.1$\pm 0.8$ | 84.8$\pm 0.8$ | 89.8$\pm 0.6$ | 89.9$\pm 0.6$ | 92.6$\pm 0.3$ | 92.6$\pm 0.3$
5 | 73.1$\pm 0.9$ | 73.7$\pm 0.5$ | 70.2$\pm 0.5$ | 70.2$\pm 0.5$ | 72.0$\pm 1.3$ | 78.6$\pm 1.0$ | 82.6$\pm 1.2$ | 82.7$\pm 1.2$ | 84.4$\pm 1.3$ | 84.5$\pm 1.3$
6 | 89.8$\pm 0.5$ | 90.4$\pm 0.4$ | 68.5$\pm 2.4$ | 68.4$\pm 2.3$ | 88.6$\pm 0.3$ | 87.5$\pm 0.4$ | 77.0$\pm 1.6$ | 77.0$\pm 1.6$ | 73.4$\pm 0.9$ | 73.4$\pm 0.9$
7 | 64.1$\pm 0.7$ | 66.1$\pm 0.6$ | 77.6$\pm 0.4$ | 77.6$\pm 0.4$ | 73.0$\pm 1.0$ | 76.2$\pm 0.6$ | 80.5$\pm 0.3$ | 80.4$\pm 0.3$ | 84.3$\pm 0.6$ | 84.2$\pm 0.6$
8 | 83.5$\pm 0.6$ | 85.3$\pm 0.4$ | 81.7$\pm 0.6$ | 81.7$\pm 0.6$ | 87.8$\pm 0.1$ | 86.4$\pm 0.2$ | 85.2$\pm 0.6$ | 85.2$\pm 0.6$ | 87.5$\pm 0.9$ | 87.7$\pm 0.9$
9 | 91.4$\pm 0.2$ | 91.7$\pm 0.4$ | 73.7$\pm 1.1$ | 73.7$\pm 1.1$ | 88.0$\pm 0.7$ | 88.8$\pm 0.5$ | 89.1$\pm 0.9$ | 89.2$\pm 0.9$ | 94.1$\pm 0.3$ | 94.1$\pm 0.3$
10 | 86.1$\pm 0.6$ | 87.0$\pm 0.6$ | 58.6$\pm 2.2$ | 58.5$\pm 2.2$ | 79.5$\pm 1.1$ | 82.4$\pm 0.7$ | 60.0$\pm 2.0$ | 59.8$\pm 2.0$ | 85.1$\pm 1.1$ | 85.2$\pm 1.1$
11 | 83.7$\pm 0.5$ | 85.0$\pm 0.3$ | 76.5$\pm 0.7$ | 76.5$\pm 0.7$ | 87.2$\pm 0.2$ | 84.9$\pm 0.4$ | 84.3$\pm 0.2$ | 84.2$\pm 0.1$ | 87.8$\pm 0.6$ | 87.8$\pm 0.6$
12 | 82.8$\pm 0.4$ | 84.3$\pm 0.4$ | 79.0$\pm 1.1$ | 79.1$\pm 1.1$ | 85.8$\pm 0.3$ | 85.2$\pm 0.7$ | 84.4$\pm 0.5$ | 84.4$\pm 0.5$ | 82.0$\pm 1.0$ | 82.0$\pm 1.1$
13 | 61.8$\pm 1.2$ | 64.1$\pm 0.8$ | 70.2$\pm 0.7$ | 70.2$\pm 0.7$ | 70.8$\pm 0.7$ | 73.6$\pm 0.6$ | 79.3$\pm 1.4$ | 79.2$\pm 1.4$ | 82.7$\pm 0.4$ | 82.7$\pm 0.4$
14 | 89.7$\pm 0.3$ | 90.3$\pm 0.2$ | 73.1$\pm 1.0$ | 73.1$\pm 1.0$ | 91.3$\pm 0.3$ | 90.2$\pm 0.6$ | 89.1$\pm 0.7$ | 89.1$\pm 0.6$ | 93.4$\pm 0.2$ | 93.4$\pm 0.2$
15 | 69.4$\pm 0.4$ | 70.6$\pm 0.3$ | 71.2$\pm 0.7$ | 71.1$\pm 0.7$ | 73.4$\pm 0.5$ | 74.9$\pm 0.5$ | 71.5$\pm 1.2$ | 71.4$\pm 1.2$ | 76.1$\pm 1.3$ | 75.8$\pm 1.4$
16 | 78.0$\pm 0.3$ | 79.4$\pm 0.4$ | 74.7$\pm 1.0$ | 74.8$\pm 1.0$ | 79.8$\pm 0.9$ | 80.5$\pm 0.9$ | 79.3$\pm 0.2$ | 79.3$\pm 0.2$ | 80.4$\pm 0.7$ | 80.3$\pm 0.7$
17 | 92.5$\pm 0.4$ | 93.7$\pm 0.2$ | 88.8$\pm 0.5$ | 88.8$\pm 0.5$ | 93.9$\pm 0.3$ | 94.6$\pm 0.2$ | 91.2$\pm 0.3$ | 91.2$\pm 0.3$ | 97.5$\pm 0.1$ | 97.5$\pm 0.1$
18 | 90.8$\pm 0.3$ | 92.5$\pm 0.3$ | 80.3$\pm 0.7$ | 80.2$\pm 0.7$ | 93.4$\pm 0.3$ | 93.1$\pm 0.1$ | 87.4$\pm 0.3$ | 87.2$\pm 0.3$ | 94.4$\pm 0.2$ | 94.4$\pm 0.2$
19 | 88.4$\pm 0.4$ | 90.3$\pm 0.3$ | 78.0$\pm 0.8$ | 77.8$\pm 0.8$ | 89.3$\pm 0.1$ | 89.8$\pm 0.2$ | 84.0$\pm 1.6$ | 83.9$\pm 1.6$ | 92.5$\pm 0.7$ | 92.4$\pm 0.6$
mean | 80.3$\pm 0.5$ | 81.9$\pm 0.5$ | 75.2$\pm 1.0$ | 75.2$\pm 1.0$ | 82.8$\pm 0.6$ | 84.1$\pm 0.6$ | 82.4$\pm 0.8$ | 82.4$\pm 0.8$ | 86.5$\pm 0.7$ | 86.5$\pm 0.7$

Table 9: Per-class one-class classification AUCs on CIFAR-20.

Representation | RotNet (Golan & El-Yaniv, 2018) | Denoising | Rot. Prediction | Contrastive | Contrastive (DA)
---|---|---|---|---|---
Classifier | Rot. Cls | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE
0 | 83.7$\pm 2.2$ | 94.5$\pm 0.3$ | 91.8$\pm 0.7$ | 92.3$\pm 0.7$ | 94.5$\pm 0.3$ | 95.2$\pm 0.5$ | 93.4$\pm 0.2$ | 93.3$\pm 0.2$ | 93.0$\pm 0.9$ | 92.3$\pm 1.6$
1 | 96.2$\pm 1.9$ | 99.5$\pm 0.1$ | 97.4$\pm 0.4$ | 97.7$\pm 0.4$ | 99.4$\pm 0.1$ | 99.7$\pm 0.1$ | 98.1$\pm 0.2$ | 98.3$\pm 0.2$ | 99.2$\pm 0.4$ | 99.1$\pm 0.6$
2 | 78.1$\pm 2.0$ | 93.8$\pm 0.2$ | 91.6$\pm 0.4$ | 91.6$\pm 0.4$ | 93.1$\pm 0.3$ | 94.2$\pm 0.3$ | 93.3$\pm 0.2$ | 93.5$\pm 0.2$ | 93.1$\pm 0.2$ | 93.4$\pm 0.1$
3 | 83.7$\pm 2.1$ | 95.0$\pm 0.4$ | 91.1$\pm 1.1$ | 91.0$\pm 1.1$ | 93.3$\pm 0.5$ | 95.4$\pm 0.4$ | 92.2$\pm 0.5$ | 91.4$\pm 0.7$ | 93.2$\pm 0.3$ | 92.2$\pm 0.3$
4 | 83.5$\pm 2.1$ | 91.9$\pm 0.1$ | 93.3$\pm 0.2$ | 92.7$\pm 0.2$ | 92.1$\pm 0.2$ | 93.9$\pm 0.2$ | 93.4$\pm 0.1$ | 92.5$\pm 0.2$ | 94.2$\pm 0.2$ | 93.4$\pm 0.3$
5 | 87.6$\pm 1.5$ | 94.3$\pm 0.4$ | 94.9$\pm 0.4$ | 94.7$\pm 0.5$ | 95.1$\pm 0.3$ | 97.3$\pm 0.3$ | 93.8$\pm 0.1$ | 93.0$\pm 0.2$ | 94.3$\pm 0.2$ | 94.3$\pm 0.3$
6 | 77.3$\pm 1.2$ | 83.4$\pm 0.2$ | 86.1$\pm 0.5$ | 85.5$\pm 0.4$ | 82.5$\pm 0.4$ | 85.9$\pm 0.4$ | 88.3$\pm 0.1$ | 87.4$\pm 0.1$ | 87.6$\pm 0.2$ | 86.2$\pm 0.1$
7 | 96.9$\pm 0.8$ | 99.2$\pm 0.1$ | 96.9$\pm 0.3$ | 96.9$\pm 0.3$ | 98.7$\pm 0.1$ | 99.4$\pm 0.1$ | 97.3$\pm 0.2$ | 97.0$\pm 0.2$ | 98.5$\pm 0.1$ | 98.4$\pm 0.1$
8 | 92.2$\pm 1.5$ | 96.2$\pm 0.5$ | 90.9$\pm 0.8$ | 90.0$\pm 0.9$ | 98.5$\pm 0.3$ | 98.3$\pm 0.3$ | 92.3$\pm 1.2$ | 92.3$\pm 1.2$ | 97.6$\pm 0.3$ | 97.6$\pm 0.3$
9 | 95.0$\pm 1.3$ | 98.1$\pm 0.3$ | 93.3$\pm 0.5$ | 92.9$\pm 0.5$ | 99.0$\pm 0.1$ | 98.6$\pm 0.2$ | 96.9$\pm 0.3$ | 96.9$\pm 0.3$ | 97.9$\pm 0.3$ | 98.0$\pm 0.3$
mean | 87.4$\pm 1.7$ | 94.6$\pm 0.3$ | 92.7$\pm 0.5$ | 92.5$\pm 0.5$ | 94.6$\pm 0.3$ | 95.8$\pm 0.3$ | 93.9$\pm 0.3$ | 93.6$\pm 0.3$ | 94.8$\pm 0.3$ | 94.5$\pm 0.4$

Table 10: Per-class one-class classification AUCs on Fashion MNIST.

Representation | RotNet (Golan & El-Yaniv, 2018) | Denoising | Rot. Prediction | Contrastive | Contrastive (DA)
---|---|---|---|---|---
Classifier | Rot. Cls | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE | OC-SVM | KDE
cat | 86.4$\pm 0.3$ | 86.1$\pm 0.2$ | 41.3$\pm 1.7$ | 41.2$\pm 1.8$ | 81.9$\pm 0.7$ | 84.1$\pm 0.8$ | 89.8$\pm 0.4$ | 89.7$\pm 0.4$ | 91.6$\pm 0.3$ | 91.7$\pm 0.3$
dog | 85.8$\pm 0.3$ | 86.6$\pm 0.2$ | 60.6$\pm 1.1$ | 60.3$\pm 1.0$ | 85.5$\pm 0.6$ | 88.7$\pm 0.4$ | 85.7$\pm 0.5$ | 85.7$\pm 0.5$ | 87.5$\pm 0.6$ | 87.5$\pm 0.6$
mean | 86.1$\pm 0.3$ | 86.4$\pm 0.2$ | 51.0$\pm 1.4$ | 50.7$\pm 1.4$ | 83.7$\pm 0.6$ | 86.4$\pm 0.6$ | 87.7$\pm 0.5$ | 87.7$\pm 0.4$ | 89.6$\pm 0.5$ | 89.6$\pm 0.4$

Table 11: Per-class one-class classification AUCs on Cat-vs-Dog.

###  A.6 More Examples for Visual Explanation of Deep One-class classifiers

(a) Input

(b) Ours

(c) RotNet

(d) RotNet∗

(e) DAE

(f) Ours

(g) RotNet

(h) RotNet∗

(i) DAE

Figure 11: Visual explanations on CelebA eyeglasses dataset. (a) input images,
(b–e) images with heatmaps using integrated gradients (Sundararajan et al.,
2017), and (f–i) those using GradCAM (Selvaraju et al., 2017). RotNet∗: RotNet
+ KDE.

(a) Input

(b) Ours

(c) RotNet

(d) RotNet∗

(e) DAE

(f) Ours

(g) RotNet

(h) RotNet∗

(i) DAE

Figure 12: Visual explanations on CelebA eyeglasses dataset. (a) input images,
(b–e) images with heatmaps using integrated gradients (Sundararajan et al.,
2017), and (f–i) those using GradCAM (Selvaraju et al., 2017). RotNet∗: RotNet
+ KDE.

(a) Input

(b) Ours

(c) RotNet

(d) RotNet∗

(e) DAE

(f) Ours

(g) RotNet

(h) RotNet∗

(i) DAE

Figure 13: Visual explanations on cat-vs-dog dataset. (a) input images, (b–e)
images with heatmaps using integrated gradients (Sundararajan et al., 2017),
and (f–i) those using GradCAM (Selvaraju et al., 2017). RotNet∗: RotNet + KDE.

(a) Input

(b) Ours

(c) RotNet

(d) RotNet∗

(e) DAE

(f) Ours

(g) RotNet

(h) RotNet∗

(i) DAE

Figure 14: Visual explanations on cat-vs-dog dataset. (a) input images, (b–e)
images with heatmaps using integrated gradients (Sundararajan et al., 2017),
and (f–i) those using GradCAM (Selvaraju et al., 2017). RotNet∗: RotNet + KDE.