# Understanding Likelihood of Normalizing Flow and Image Complexity through
the Lens of Out-of-Distribution Detection

Genki Osada  LY Corporation, Japan  Tsubasa Takahashi  LY Corporation, Japan
Takashi Nishide  University of Tsukuba, Japan

###### Abstract

Out-of-distribution (OOD) detection is crucial to safety-critical machine
learning applications and has been extensively studied. While recent studies
have predominantly focused on classifier-based methods, research on deep
generative model (DGM)-based methods have lagged relatively. This disparity
may be attributed to a perplexing phenomenon: DGMs often assign higher
likelihoods to unknown OOD inputs than to their known training data. This
paper focuses on explaining the underlying mechanism of this phenomenon. We
propose a hypothesis that less complex images concentrate in high-density
regions in the latent space, resulting in a higher likelihood assignment in
the Normalizing Flow (NF). We experimentally demonstrate its validity for five
NF architectures, concluding that their likelihood is untrustworthy.
Additionally, we show that this problem can be alleviated by treating image
complexity as an independent variable. Finally, we provide evidence of the
potential applicability of our hypothesis in another DGM, PixelCNN++.

##  1 Introduction

Deep neural network (DNN) models deployed in real-world systems often
encounter out-of-distribution (OOD) inputs — samples from a different
distribution of the training set. DNN models often make incorrect predictions
with high confidence for OOD inputs [1], making it crucial to distinguish OOD
inputs from in-distribution (In-Dist) ones at test time, especially for
safety-critical applications such as autonomous driving [2] and medical
diagnosis [3]. Numerous methods have been proposed to improve empirical
performance [4, 5], and analytical approaches have also been studied [6, 7].

One major approach to OOD detection is using deep generative models (DGMs).
Particularly, Normalizing Flows (NFs) and Autoregressive (AR) models are the
two most commonly selected DGMs due to their ability to compute exact model
likelihoods. The DGM-based approach is attractive for its strengths: it does
not require labeled data to train detection models, and the performance of OOD
detection is independent of the number of classification classes [8]. However,
most of the recent progress in this field has been based on another prevailing
approach, the classifier-based method [4, 5]. The primary factor hindering the
progress of DGM-based methods may be the observation presented by [9, 10]:
DGMs were expected to assign a higher likelihood, i.e., probabilistic density,
to In-Dist inputs than to OOD inputs, but it turned out that this is not the
case in some particular cases. We refer to this phenomenon as the _failure of
likelihood_. When an OOD detection method is deployed to real-world
applications with no control over its inputs, this unreliability is
unacceptable. Above all, correctly estimating the likelihood is a fundamental
feature that DGMs are expected to fulfill. Hence, research on DGM-based
methods has focused on addressing this issue before enhancing detection
performance and tackling more challenging tasks.

Several hypotheses have been proposed to explain this phenomenon. One argues
that image complexity has some influence on the likelihood [9, 11], while
another suggests the need to account for the notion of the _typical set_ [10,
12]. However, these methods still fail in specific cases, as we will show
later, and the mechanism behind the failure of likelihood remains unclear.

In this paper, we address the cause of the failure of likelihood. We focus on
Normalizing Flows due to their analytical tractability. The contributions of
this work are as follows:

  * •

We present a hypothesis that explains how image complexity can arbitrarily
control the likelihood of NFs and how the density concentration in latent
distribution causes this phenomenon.

  * •

Our hypothesis provides a unified explanation for two separate questions: why
images with less complexity are assigned high likelihood and why OOD inputs
are regarded as the typical set samples in some cases.

  * •

We experimentally verify our argument with five different NF architectures and
conclude that OOD detections based on the likelihood of NFs are untrustworthy.

The relationship between the likelihood and image complexity that we explain
in this paper finds implicit support in Independent Component Analysis (ICA)
[13], as well as in Shannon’s source coding theorem [14], as discussed later
in this section.

Additionally, we propose a countermeasure and argue that the failure of
likelihood can be overcome by leveraging information about the root cause of
the problem, namely image complexity. We demonstrate that a Gaussian mixture
model (GMM) with two variables, image complexity and likelihood in latent
space, can detect all of the OOD datasets we used in our evaluation, thereby
highlighting the validity of our argument from another perspective.

Finally, given a recent study in which AR models can be seen as a specific
type of NF [15], we show the potential applicability of our hypothesis to AR
models as well through the experiments using PixelCNN++.

##### Related works.

Two main hypotheses have been proposed to explain the failure of likelihood-
based OOD detection: the involvement of _image complexity_ and the necessity
of considering _typical set_. Experimental studies by [11, 16] demonstrated
that the log-likelihood $\log p({\bf x})$ increases with less complex images,
but the underlying mechanism remained unclear. [17, 18] attributed this
phenomenon to the architecture of the NFs. However, the same phenomenon has
also been observed in PixelCNN [9, 16], implying that it is not exclusively
caused by the architecture of the NFs. [9] suggested that the phenomenon could
be theoretically explained. Their analysis implicitly assumed that the
likelihood in the latent space and the determinant of the Jacobian matrix
remain consistent across different datasets. However, as shown in Figs. 3
(right) and 4, in reality, these values differ considerably between datasets.
We explain that this variation arises from differences in image complexities,
ultimately serving as a factor that can arbitrarily influence $\log p({\bf
x})$. We experimentally show that our hypothesis likely applies to PixelCNN as
well.

[10, 12] introduced a testing method based on the typical set in latent space.
However, [19, 10, 20] concluded that its performance was insufficient. Our
hypothesis also explains _why_ typicality-based testing fails. Additionally,
in Appendix A, we present other DGM-based approaches and an overview of
classifier-based detection methods.

Beyond the context of OOD detection, the credibility of likelihood has been
discussed. [21] has shown that high likelihood does not necessarily indicate
high sample quality produced by the model. [22] proposed that image complexity
can be estimated by $\log p({\bf x})$ of ICA [13], which can be regarded as a
linear NF [23]. Their study implies that less complex image ${\bf x}$ is
assigned a higher $\log p({\bf x})$ in ICA, aligning with our assertion.
Moreover, Shannon’s source coding theorem [14] states that the expected code
length of a symbol ${\bf x}$ is bounded by its entropy:
$\operatorname{\mathbb{E}}_{{\bf x}\sim P_{\text{x}}}[-\log p({\bf
x})]\leq\operatorname{\mathbb{E}}_{{\bf x}\sim P_{\text{x}}}[L({\bf x})].$
Here, $L({\bf x})$ represents the length of the encoded message for ${\bf x}$
using a lossless compression algorithm, and $P_{\text{x}}$ denotes a data
distribution, which is generally unknown. Minimizing
$\operatorname{\mathbb{E}}_{{\bf x}\sim P_{\text{x}}}[L({\bf x})]$ is achieved
by assigning a smaller $L({\bf x})$ to ${\bf x}$ with a larger $\log p({\bf
x})$ and a larger $L({\bf x})$ to ${\bf x}$ with a smaller $\log p({\bf x})$,
ensuring efficient encoding [24]. The complexity of an image ${\bf x}$ is
defined in the next section as $L({\bf x})/d$, where ${\bf x}$ with less
complexity have a smaller $L({\bf x})$. Therefore, the theorem implies that
images with less complexity are assigned higher likelihoods, aligning with our
claim.

##  2 Background and Problem Statement

This paper shows that the complexity of input images can control the
likelihood of Normalizing Flows (NFs). We first define the image complexity
and describe the NFs. Then, we discuss existing approaches and their failures,
leading us to articulate the problem statement addressed in this paper.

###  2.1 Preliminary

##### Image complexity and entropy coding.

Estimating the complexity of images lacks a definitive method; however, two
common options are available: Kolmogorov complexity and Shannon’s entropy
[25]. While a connection is suggested between the two [26], the former is
uncomputable. Thus, the prevalent method is based on entropy, which is
directly used in lossless compression (coding) [27, 28]. In this work, we use
the following definition for image complexity used in previous studies [11,
29].

######  Definition 1 (Image complexity).

Let $L({\bf x})$ be the length of the bit string obtained after compressing
${\bf x}\in\\{0,1,\ldots,255\\}^{d}$ using a lossless compression algorithm,
comp. Image complexity for ${\bf x}$ is defined as $C({\bf
x})=\frac{1}{d}L({\bf x})$. The more complex ${\bf x}$ is, the larger $C({\bf
x})$ is, and vice versa.

We use the JPEG2000 compression, which is based on entropy coding, as the comp
in all our experiments.

##### Normalizing Flow.

We focus our investigation on NFs [30, 23, 31] among DGMs. NFs offer distinct
advantages over other types of DGMs, as they enable the exact computation of
likelihoods, unlike VAEs, and allow for the separation of the volume term,
facilitating analysis [9]. NFs learn an invertible mapping
$f:\mathcal{X}\rightarrow\mathcal{Z}$ that maps observable data ${\bf x}$ to
the latent vector ${\bf z}=f({\bf x})$, where $\mathcal{X}\in\mathbb{R}^{d}$
is the data space and $\mathcal{Z}\in\mathbb{R}^{d}$ is the latent space. We
denote a distribution on $\mathcal{Z}$ as $P_{\text{z}}$ with probability
density $p({\bf z})$. NFs learn an approximate model distribution
$\widehat{P}_{\text{x}}$ to match the unknown true distribution $P_{\text{x}}$
on $\mathcal{X}$. Under the change of variable rule, the log density of
$\widehat{P}_{\text{x}}$ is expressed as:

$\displaystyle\log p({\bf x})=\log p({\bf z})+\log|\text{det}\ J_{f}({\bf
x})|$ (1)

where $J_{f}({\bf x})=df({\bf x})/d{\bf x}$ is the Jacobian matrix of $f$ at
${\bf x}$, and $\log|\text{det}\ J_{f}({\bf x})|$ is referred to as the
_volume_. By maximizing $\log p({\bf z})$ and $\log|\text{det}\ J_{f}({\bf
x})|$ simultaneously with respect to samples ${\bf x}\sim P_{\text{x}}$, the
NF model $f$ is trained to match $\widehat{P}_{\text{x}}$ with $P_{\text{x}}$.
In our work, $P_{\text{x}}$ represents In-Distribution (In-Dist), and we train
an NF model using samples from In-Dist.

We investigate five architectures: Glow [32], CV-Glow [9], iResNet [33],
ResFlow [34], and IDF [35]. iResNet and ResFlow improve the stability of NFs
by controlling the Lipschitz constant of $f$ to be less than one while
maintaining high expressive power. IDF uses a categorical distribution for
$P_{\text{z}}$, while the other four architectures use the standard Gaussian
distribution. CV-Glow, as the abbreviation for _Glow with the constant volume_
, has a fixed volume that depends only on the weight matrix of the $1\times 1$
convolutions, resulting in a constant volume across all inputs ${\bf x}$.
Similarly, the volume of IDF is fixed at 0 to build a latent space with
integer values. It is worth noting that CV-Glow, IDF, and even PixelCNN++ [36]
exhibit a similar behavior due to the commonality of the constant volume, as
we will see later. The implementation of these models are described in
Appendix D.3.

Figure 1: DCAS (Remark 1) attracts less complex images to the high-density
region in latent space. $\mathcal{Z}$ represents a Gaussian latent space
trained on CIFAR-10. $O_{\mathcal{Z}}$ represents the origin of $\mathcal{Z}$.
The dark blue circle represents the typical set in $\mathcal{Z}$, identified
as In-Dist by the typicality test. Complex OOD images like Noise-1 and
CIFAR-10 (Noise-4-16) are mapped far from $O_{\mathcal{Z}}$. However, due to
DCAS, less complex images like Noise-16 and CIFAR-10 (Pool-4) are mapped
closer to $O_{\mathcal{Z}}$ than the circle. We hypothesize that SVHN (an OOD
image) should be mapped far beyond the circle. However, due to its less
complexity, it is attracted towards $O_{\mathcal{Z}}$ and coincidentally falls
on the circle, leading to the misidentification of SVHN as In-Dist by the
typicality test.

###  2.2 Failure of Existing Approaches

##### Likelihood test.

The likelihood test, introduced by [37], is an OOD detection method that
relies a density estimation model. Treating the probabilistic density of input
as a likelihood, it assumes that OOD examples would be assigned lower
likelihoods compared to In-Dist examples. However, counter-evidence presented
by [9, 10] showed that DGMs trained on CIFAR-10 assigned higher likelihoods to
samples from SVHN (OOD) than to samples from CIFAR-10 (In-Dist), for instance.
This has sparked controversy and led to the proposal of two approaches
described below as methods for improvement.

##### Complexity-aware likelihood test.

Several studies have linked the failure of the likelihood test to image
complexity [9, 11, 16, 17, 18]. These studies experimentally showed that low-
complexity regions in images contribute to an increase in the likelihood $\log
p({\bf x})$. One such study by [11] introduced a complexity-aware likelihood
test (CALT), which directly incorporates image complexity $C({\bf x})$.
Specifically, CALT computes the score $S_{\text{CALT}}({\bf x})=\log p({\bf
x})+C({\bf x})$. However, our experiments have revealed that CALT still
exhibits failures in certain cases, indicating that merely adding $C({\bf x})$
to $\log p({\bf x})$ is insufficient for effective OOD detection. (See
Appendix B.2 and Section 5.1.)

##### Typicality test.

[10, 12] attributed the failure of the likelihood test to an ignorance of the
notion of the _typical set_. In a $d$-dimensional isotropic Gaussian
$\mathcal{N}(0,\textbf{I}_{\text{d}})$, the typical set is expected to reside
in a hypersphere with a radius of $\sqrt{d}$ with high probability, rather
than around its mean (See Appendix B.1). Utilizing this property, [10, 12]
have proposed a method that identifies a test input ${\bf x}$ as OOD when it
fall outside of the distribution’s typical set, which we refer to as the
typicality test in latent space (TTL). TTL computes the score
$S_{\text{TTL}}({\bf x})={\tt abs}(\left\lVert{\bf z}\right\rVert-\sqrt{d})$,
where ${\bf z}=f({\bf x})$ and $f$ is a trained NF model. If ${\bf x}$ is OOD,
$S_{\text{TTL}}({\bf x})$ is expected to be large. However, [20, 19, 10]
concluded that the performance of TTL was insufficient, and our experiments
also revealed failure cases. We observed that $\left\lVert{\bf z}\right\rVert$
for In-Dist samples is concentrated around the theoretical value, $\sqrt{d}$
(i.e., $\sqrt{32\times 32\times 3}\simeq 55.4$ for CIFAR-10 and SVHN). This
indicates that NFs can correctly perceive test samples from In-Dist as the
typical set. However, in the case where the In-Dist is CIFAR-10,
$\left\lVert{\bf z}\right\rVert$ for SVHN (OOD) is also highly concentrated
around $55.4$, indicating the failure of TTL to detect SVHN samples as OOD.
This situation is depicted in Fig. 1. (For experimental results, see Fig. 5
(bottom two) in Appendix B.2.) It becomes evident that even OOD inputs can
reside within the typical set, challenging the effectiveness of the typicality
test.

###  2.3 Problem Statement

The aforementioned issues can be summarized in the following two questions:

  * •

Why does less image complexity result in the failure of the likelihood test?

  * •

Why are OOD inputs often misclassified as typical set samples?

We address these questions in this study.

##  3 Hypothesis and Experimental Validation

We present Hypothesis 1, which comprehensively explains the questions posed in
Section 2.3. Subsequently, we provide experimental results that support the
hypothesis.

###  3.1 Our Hypothesis

######  Definition 2 (Local Lipschitz continuity).

For a subset $\mathcal{A}\subset\mathcal{Z}$, we define an invertible function
$f:\mathcal{X}\rightarrow\mathcal{Z}$ as locally $L_{\mathcal{A}}$-Lipschitz
as follows:

$\displaystyle\left\lVert f({\bf x}_{1})-f({\bf x}_{2})\right\rVert\leq
L_{\mathcal{A}}\left\lVert{\bf x}_{1}-{\bf x}_{2}\right\rVert,\forall f({\bf
x}_{1}),f({\bf x}_{2})\in\mathcal{A}.$ (2)

######  Assumption 1.

We consider that $f$ is implemented by an NF and assume that the latent space
$\mathcal{Z}$ is semantically continuous [31].

######  Hypothesis 1.

Let $f:\mathcal{X}\rightarrow\mathcal{Z}$ be an invertible function locally
$L_{\mathcal{A}}$-Lipschitz for $\mathcal{A}\subset\mathcal{Z}$. For all ${\bf
z}^{\prime}\in\mathcal{A}$, let ${\bf x}^{\prime}=f^{-1}({\bf z}^{\prime})$.
Also, let $\mathcal{B}^{\epsilon}_{{\bf z}}=\\{{\bf
z}^{\prime}\in\mathcal{A}:\left\lVert{\bf z}^{\prime}-{\bf
z}\right\rVert<\epsilon\\}$ and $\mathcal{B}^{\epsilon}_{{\bf
z}}\subset\mathcal{A}$ for a constant $\epsilon\geq 0$. Then, letting $C({\bf
x})$ be the image complexity of ${\bf x}$ (Definition 1) and $C_{1}$ be a
constant, we have

$\displaystyle\frac{\epsilon^{2}}{{L_{\mathcal{A}}}^{2}}\left(1-\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf
z}})\right)\leq C_{1}\exp(C({\bf x})).$ (3)

Considering $\mathcal{A}$ to be a very small region in $\mathcal{Z}$ and based
on Assumption 1, we posit that samples ${\bf x}=f^{-1}({\bf z})$ and ${\bf
x}^{\prime}=f^{-1}({\bf z}^{\prime})$ for ${\bf z},{\bf
z}^{\prime}\in\mathcal{A}$ share common semantics. This allows us to treat
their complexities as approximately equal, i.e., $C({\bf x})\approx C({\bf
x}^{\prime})$, which we use in the derivation. We derive Eq. (3) from
experimental observations and present it in Appendix C.2. Assuming ${\bf x}$
follows a diagonal Gaussian distribution, we can analytically derive it as
stated in Appendix C.1, resulting in the right-hand side being $\frac{d}{2\pi
e}\delta_{x}^{\frac{2}{d}}\exp\left(2C({\bf x})\right)$ where $\delta_{x}$ is
the volume of bins used in discretization. Next, we present an observation
necessary to state the following Remarks.

######  Observation 1.

A positive correlation exists between $\log p({\bf z})$ and the volume
$\log|\text{det}\ J_{f}({\bf x})|$ in response to the variation of input ${\bf
x}$.

We defer the explanation to Section 3.2, Experiment 3. In the following
Remarks, we consider that ${\bf z}=f({\bf x})$ is determined for a given input
${\bf x}$ such that $\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf z}})$ satisfies
the inequality in Eq. (3).

Pooling Noise Images :

Manipulated CIFAR-10 :

Figure 2: Complexity controlled images. Image complexity increases from left
to right in both rows. Top: _Pooling noise images_ with pooling size $\kappa$
decreases as 32, 16, 8, 4, 2, and 1 from left to right. Bottom: _Manipulated
CIFAR-10_ with Pool-8, Pool-4, Pool-2, Noise-4-4, Noise-4-8, and Noise-4-16
from left to right.

Figure 3: Plots for manipulated CIFAR-10. Left plot shows complexity vs.
$\left\lVert{\bf z}\right\rVert$, supporting Remark 1. Right plot shows volume
vs. $\left\lVert{\bf z}\right\rVert$, supporting Remark 2 and Observation 1.
We note that $\left\lVert{\bf z}\right\rVert\propto-\sqrt{\log p({\bf z})}$.

######  Remark 1.

As ${\bf x}$ becomes less complex, i.e., as $C({\bf x})$ becomes small,
$\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf z}})$ becomes large. In order to
increase $\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf z}})$, ${\bf z}$ needs to be
in a high-density region in $\mathcal{Z}$, because the volume of
$\mathcal{B}^{\epsilon}_{{\bf z}}$, $\text{vol}(\mathcal{B}^{\epsilon}_{{\bf
z}})$, is fixed by $\epsilon$. In other words, ${\bf z}$ for a less complex
${\bf x}$ will concentrate on a high-density region in $\mathcal{Z}$, meaning
that $\log p({\bf z})$ for them will be large. Specifically, when the
distribution of $\mathcal{Z}$ is $\mathcal{N}(0,\textbf{I}_{\text{d}})$, ${\bf
z}$ is mapped to the region close to the origin, meaning that $\left\lVert{\bf
z}\right\rVert$ will be small for a less complex ${\bf x}$ (See Fig. 1). We
refer to this effect as Density Concentration Attraction for Simpleness, DCAS.

######  Remark 2.

As the input becomes less complex, $L_{\mathcal{A}}$ becomes large. Since
$|\text{det}\ J_{f}({\bf x})|<L_{\mathcal{A}}^{d}$ [38], the volume term,
$\log|\text{det}\ J_{f}({\bf x})|$, for a less complex input is allowed to
become large.

######  Remark 3.

From Observation 1 and Remarks 1, 2, the decrease in image complexity
increases both $\log p({\bf z})$ and $\log|\text{det}J_{f}({\bf x})|$.
Consequently, according to Eq. (1), $\log p({\bf x})$ increases for a less
complex input.

###  3.2 Experimental Results

To support the above Remarks, we conducted experiments using a dataset in
which image complexity was systematically controlled.

##### Datasets.

We constructed two datasets with controlled image complexity: _pooling noise
images_ and _manipulated CIFAR-10_. The details are provided in Appendix D.1,
and samples are shown in Fig. 2. The pooling noise images are generated by
applying average pooling with different filter sizes, denoted as $\kappa$, to
random noise images. Each set of images is labeled as Noise-$\kappa$. For the
manipulated CIFAR-10 dataset, the complexity increases from Noise-4-1 to
Noise-4-32 and decreases from Pool-2 to Pool-16, compared to the original
image. The NF models used in this section were trained on CIFAR-10 [39].

##### Experiment 1: Complexity vs. $\log p({\bf z})$.

We examined Remark 1 using the five NF models introduced in Section 2.1. We
used $\left\lVert{\bf z}\right\rVert(\propto-\sqrt{\log p({\bf z})})$ as a
proxy for $\log p({\bf z})$ for the convenience of considering the typical set
(for the four NFs with Gaussian latent distribution). The plot for ResFlow on
the manipulated CIFAR-10 is depicted in Fig. 3 (left). It shows that $\log
p({\bf z})$ increases (equivalently, $\left\lVert{\bf z}\right\rVert$
decreases) as the complexity decreases, providing support for Remark 1.
Similar plots were observed for all other nine cases, as shown in Figs. 9 and
10 in Appendix E.1.

##### Experiment 2: $\log p({\bf z})$ vs. volume.

We examined Remark 2 using Glow, iResNet, and ResFlow, excluding CV-Glow and
IDF, which have a constant (or zero) volume. Fig. 3 (right) presents the plot
for ResFlow on the manipulated CIFAR-10. It shows that a decrease in
complexity (from Noise-4-16 to Pool-32) causes an increase in the volume,
providing support for Remark 2. Similar plots were observed for all other five
cases, as shown in Fig. 11 in Appendix E.2. Notably, Remark 2 asserts the
opposite of the speculation presented in a previous study [29] (that higher
image complexity corresponds to a larger volume).

##### Experiment 3: Correlation between $\log p({\bf z})$ and volume.

Fig. 3 (right) also shows that the positive correlation between $\log p({\bf
z})$ ($\propto-\left\lVert{\bf z}\right\rVert^{2}$) and the volume
$\log|\text{det}\ J_{f}({\bf x})|$. It indicates that the two increase
simultaneously in a balanced manner, which we have already presented as
Observation 1. The balance between the two is a result of the learning process
of NFs. By maximizing Eq. (1), an NF is trained to reach an equilibrium
between two opposing forces: 1) attracting ${\bf z}$ to high-density region in
$\mathcal{Z}$ to maximize $\log p({\bf z})$, and 2) scattering ${\bf z}$ to
expand the volume $|\text{det}\ J_{f}({\bf x})|$ [23, 40]. In other words,
training an NF model involves optimizing the ratio between $\log p({\bf z})$
and the volume to partition the variation in $\log p({\bf x})$. The
correlation strength between the two, as discussed in Appendix D.4, is
dependent on the architecture of NFs. We utilize this knowledge in the
selection of architectures for OOD detection in Section 5.1.

##  4 Likelihood is Untrustworthy

With Remarks presented in the previous section, we address the questions posed
in Section 2.3 and assert the untrustworthiness of the likelihood of
Normalizing Flows.

#####  _Why does less image complexity result in the failure of the likelihood
test?_

Remark 3 directly answers this question. Inputs with less image complexity,
even OOD ones, cause an increase in $\log p({\bf x})$, leading to their
misidentification as In-Dist. While some studies have experimentally shown
that a decrease in the image complexity causes an increase in $\log p({\bf
x})$, we have formulated the underlying mechanism in Eq. (3) and identified
that the culprit is the effect caused by a density concentration in latent
distribution, which we refer to as DCAS in Remark 1.

#####  _Why are OOD inputs often misclassified as typical set samples?_

Based on Remark 1, we explain the mechanism underlying the failure of TTL. We
provide an illustration in Fig. 1 for better intuition. We posit that an NF
trained on In-Dist data inherently assigns a smaller $\log p({\bf z})$ to an
OOD input ${\bf x}_{\text{ood}}$ compared to an In-Dist input ${\bf
x}_{\text{in}}$. In the case of NFs with a Gaussian latent space, the NF, $f$,
attempts to map ${\bf z}_{\text{ood}}=f({\bf x}_{\text{ood}})$ to a region
farther than the distance $\sqrt{d}$ from the origin, where the typical set
samples, including ${\bf z}_{\text{in}}=f({\bf x}_{\text{in}})$, concentrate.
However, when the complexity of ${\bf x}_{\text{ood}}$ is lower, the effect of
DCAS comes into play: ${\bf z}_{\text{ood}}$ is attracted to high-density
region on $\mathcal{Z}$, i.e., the origin of the standard Gaussian, causing
$\left\lVert{\bf z}_{\text{ood}}\right\rVert$ to become smaller. Consequently,
$\left\lVert{\bf z}_{\text{ood}}\right\rVert$ can be as small as $\sqrt{d}$,
rendering the TTL unable to differentiate such ${\bf x}_{\text{ood}}$ from the
typical set, i.e., the In-Dist examples. Therefore, the failure of TTL is
caused by the balance between the _original_ $\left\lVert{\bf
z}_{\text{ood}}\right\rVert$ assuming the effect of DCAS could be removed, and
the extent to which the DCAS shrinks $\left\lVert{\bf
z}_{\text{ood}}\right\rVert$. Both factors depend on the combination of In-
Dist dataset and OOD inputs. The combination of CIRAR-10 for In-Dist and SVHN
for OOD is a case where $\left\lVert{\bf z}_{\text{ood}}\right\rVert$ is
coincidentally shrunk to approximately $\sqrt{d}$. On the other hand, inputs
with even less complexity than SVHN, such as Pool-16/32, are more intensely
affected by the DCAS, resulting in $\left\lVert{\bf
z}_{\text{ood}}\right\rVert<\sqrt{d}$. In this case, ${\bf z}_{\text{ood}}$
falls out of the annulus where the typical set samples reside, and thus the
TTL can successfully identify such $\left\lVert{\bf
x}_{\text{ood}}\right\rVert$ as OOD.

##### Untrustworthiness of $\log p({\bf z})$ and $\log p({\bf x})$.

We explained the reasons why existing OOD detection methods fail above. Now,
we present the main claim of this study. The impact of image complexity on the
volume term can be mitigated by the selection of the NF architecture, and
indeed, in fixed-volume architectures such as CV-Glow and IDF, the effect
described in Remark 2 is nullified (Appendix D.4). However, the effect of DCAS
described in Remark 1 affects $\log p({\bf z})$ whenever the latent
distribution $P_{\text{z}}$ has a density concentration, regardless of its
distributional form. Since there is no way to disable the DCAS, $\log p({\bf
z})$ is unreliable. This not only undermines the trustworthiness of the TTL,
but also renders $\log p({\bf x})$, which incorporates $\log p({\bf z})$,
untrustworthy. Therefore, we conclude that the likelihood of Normalizing Flows
is untrustworthy.

Figure 4: Complexity vs. $\left\lVert{\bf z}\right\rVert(\propto-\sqrt{\log
p({\bf z})})$ for OOD datasets. Blue is In-Dist (CIFAR-10), green is SVHN, and
Red is CelebA. From left to right, Glow, ResFlow, and IDF. Glow and ResFlow
exhibit more pronounced separation between datasets compared to IDF. The GMM
is trained to capture the in-distribution in these two-dimensional spaces.

| SVHN | CelebA | TIN | Bed | Living | Tower | N-1 | N-2 | N-4 | N-8 | N-16 | N-32
---|---|---|---|---|---|---|---|---|---|---|---|---
TTL | 47.86 | 89.46 | 84.62 | 90.77 | 91.70 | 89.76 | 100 | 57.32 | 36.88 | 91.70 | 96.41 | 99.97
CALT | 94.08 | 52.78 | 36.01 | 54.28 | 36.12 | 61.35 | 97.74 | 100 | 100 | 100 | 100 | 100
LRB | 54.16 | 65.69 | 38.33 | 34.56 | 32.81 | 28.59 | 0.79 | 0.26 | 6.47 | 15.31 | 14.87 | 11.81
LRG | 75.08 | 94.08 | 81.41 | 67.92 | 61.96 | 68.97 | 100 | 73.51 | 99.21 | 99.23 | 97.51 | 96.37
WAIC | 74.78 | 33.97 | 78.31 | 78.43 | 85.97 | 83.60 | 100 | 100 | 88.3 | 90.79 | 98.08 | 99.60
Ours (Glow) | 91.47 | 83.15 | 88.59 | 86.97 | 89.25 | 88.42 | 100 | 99.23 | 100 | 100 | 100 | 100
Ours (ResFlow) | 93.51 | 99.92 | 93.53 | 94.04 | 94.10 | 91.92 | 100 | 99.99 | 100 | 99.96 | 100 | 100
Ours (iResNet) | 87.51 | 62.27 | 61.98 | 81.41 | 86.04 | 80.52 | 100 | 99.94 | 10.80 | 99.47 | 100 | 100

| CIFAR-10 | CelebA | TIN | Bed | Living | Tower | N-1 | N-2 | N-4 | N-8 | N-16 | N-32
---|---|---|---|---|---|---|---|---|---|---|---|---
TTL | 97.59 | 99.98 | 99.83 | 99.99 | 100 | 99.85 | 100 | 100 | 99.98 | 78.34 | 81.69 | 99.89
CALT | 8.12 | 4.12 | 8.80 | 14.14 | 9.17 | 21.58 | 100 | 100 | 100 | 100 | 100 | 100
LRB | 1.45 | 5.05 | 7.01 | 14.10 | 9.89 | 14.67 | 53.86 | 0.00 | 0.11 | 6.29 | 16.20 | 11.68
LRG | 94.99 | 99.96 | 100 | 100 | 100 | 100 | 100 | 100 | 99.72 | 99.04 | 96.71 | 96.24
WAIC | 99.56 | 99.45 | 99.86 | 99.96 | 99.99 | 99.74 | 100 | 100 | 100 | 98.25 | 95.87 | 98.04
Ours (Glow) | 98.01 | 99.96 | 99.91 | 99.98 | 100 | 99.86 | 100 | 100 | 100 | 100 | 100 | 100
Ours (ResFlow) | 65.96 | 63.76 | 85.76 | 68.96 | 84.86 | 57.04 | 100 | 92.12 | 100 | 100 | 100 | 100
Ours (iResNet) | 89.60 | 98.86 | 98.99 | 99.69 | 99.89 | 99.03 | 100 | 100 | 99.97 | 65.61 | 99.16 | 99.22

Table 1: AUROC (%)$\uparrow$. In-Dist datasets are CIFAR-10 (top) and SVHN
(bottom). Due to space constraints, Noise-$\kappa$ is abbreviated as
N-$\kappa$. Failure cases (lower than $60\%$) are underlined.

| SVHN | CelebA | TIN | Bed | Living | Tower | N-2 | N-4 | N-8 | N-16 | N-32
---|---|---|---|---|---|---|---|---|---|---|---
PixelCNN++ | 83.38 | 14.37 | 23.20 | 15.53 | 10.33 | 24.62 | 0.03 | 36.47 | 98.92 | 100 | 100
Ours | 95.34 | 75.71 | 74.91 | 76.74 | 78.77 | 80.68 | 100 | 100 | 100 | 100 | 100

| CIFAR-10 | CelebA | TIN | Bed | Living | Tower | N-2 | N-4 | N-8 | N-16 | N-32
---|---|---|---|---|---|---|---|---|---|---|---
PixelCNN++ | 1.86 | 0.02 | 0.75 | 0.18 | 0.05 | 0.76 | 0.00 | 0.00 | 66.80 | 99.30 | 96.96
Ours | 91.48 | 96.65 | 96.94 | 98.42 | 99.31 | 97.61 | 100 | 100 | 100 | 100 | 100

Table 2: AUROC (%)$\uparrow$. ‘Ours’ indicates complexity-aware PixelCNN++.
‘PixelCNN++’ indicates the likelihood test with PixelCNN++ as a baseline. In-
Dist datasets are CIFAR-10 (top) and SVHN (bottom). Due to space constraints,
Noise-$\kappa$ is abbreviated as N-$\kappa$. Failure cases (lower than $60\%$)
are underlined. The significant improvements shown here suggest that our
hypothesis may be applicable not only to NFs but also to autoregressive
models.

##  5 OOD Detection with Complexity Awareness

In the previous section, we have shown that the OOD detection based on the
likelihood of Normalizing Flows is untrustworthy. In this section, we show
that this situation can be overcome by exploiting the information on the root
cause of the problem, i.e., image complexity. We aim to further validate
Hypothesis 1 through the presented demonstration.

###  5.1 Experiment 4: Normalizing Flows

We aim to isolate the influence of image complexity $C({\bf x})$ on the
likelihood. To achieve this, we treat $C({\bf x})$ as an independent variable
and train a multivariate detection model that takes two variables, $C({\bf
x})$ and $\log p({\bf z})$. While the volume term $\log|\text{det}\ J_{f}({\bf
x})|$ could also be used as an input variable, we choose not to include it due
to its dependence on the NF architecture and higher computational cost for
iResNet and ResFlow. Among the various methods available for multivariable
detection, we employ one of the simplest models, a Gaussian mixture model
(GMM), for demonstration.

##### NF models.

We use Glow, iResNet, and ResFlow in this demonstration because in these
architectures, $\log p({\bf z})$ is insensitive to $C({\bf x})$, which we
believe is advantageous for distinguishing between different datasets. For
details regarding this architecture selection, refer to Appendices D.5 and
D.4. Fig. 4 visually demonstrates the superior capability of Glow and ResFlow
to separate and differentiate each dataset, outperforming IDF.

##### Detection with GMM.

Using the samples in the training portion of the In-Dist datasets, ${\bf
x}_{\text{train}}$, we obtain a set of $C({\bf x}_{\text{train}})$ and $\log
p(f({\bf x}_{\text{train}}))$ where $f$ is a trained NF model. Then, we train
a GMM $\mathcal{G}$ to capture the distribution of two-dimensional vectors
composed of $C({\bf x}_{\text{train}})$ and $\log p({\bf z}_{\text{train}})$.
At the testing phase, given a test sample ${\bf x}_{\text{test}}$, we input
$C({\bf x}_{\text{test}})$ and $\log p({\bf z}_{\text{test}})$ into the
trained $\mathcal{G}$, obtaining the likelihood score $S_{\text{GMM}}({\bf
x}_{\text{test}})$ from $\mathcal{G}$. A larger $S_{\text{GMM}}({\bf
x}_{\text{test}})$ indicates a higher likelihood that ${\bf x}_{\text{test}}$
belongs to the In-Dist dataset. For more details, see Appendix D.5.

##### Datasets.

When the In-Dist dataset is CIFAR-10 and SVHN, we use CelebA [41],
TinyImageNet (TIN) [42], LSUN [43], and the pooling noise images (Section 3.2)
as OOD datasets. For LSUN, which comprises various scene categories, we select
the Bedroom (_Bed_), Living room (_Living_), and _Tower_ categories, treating
each of them as individual OOD datasets. When the In-Dist dataset is MNIST and
FMNIST, we use OMNIGLOT [44] and NotMNIST [45] as OOD datasets. In experiments
using ImageNet of size $224\times 224$ as the In-Dist dataset, we use the
pooling noise images as OOD datasets.

##### Evaluation metrics.

We evaluate our method using two standard metrics in OOD detection literature:
the area under the receiver operating characteristic curve (AUROC) and the
area under the precision-recall curve (AUPR). These metrics provide an overall
assessment of performance by varying the detection threshold. Higher values
for both metrics indicate better performance. Given that the chance level for
AUROC is 50%, we set a minimum threshold of 60% to evaluate detectability. Our
primary focus is on the method’s capability to detect any type of OODs. Thus,
we evaluate the methods based on whether the AUROC scores exceed 60% for each
OOD dataset used in the evaluation. In the tables, scores below 60% are
underlined to indicate that they do not meet the detectability threshold.

##### Competitors.

In addition to TTL and CALT, we compare our method to three other existing NF-
based methods on CIFAR-10 and SVHN: the Watanabe-Akaike Information Criterion
(WAIC) [10], the likelihood-ratio to background model (LRB) [16], and the
likelihood-ratio to general model (LRG) [17]. Appendix D.5 provides detailed
descriptions of each method.

##### Results on MNIST and FMNIST.

The AUROC and AUPR are shown in Appendix E.2 (Tables 3 and 4). Our methods
detected all cases, with Glow achieving the highest performance, followed by
ResFlow and iResNet.

##### Results on CIFAR-10 and SVHN.

The AUROC and AUPR are shown in Tables 1 and 5 in Appendix E.2, respectively.
Once again, our method with Glow demonstrated the most superior performance
overall. Our method with ResFlow performed best on CIFAR-10 but showed weaker
performance on SVHN, scoring below 60% on Tower. Among the other methods, only
LRG scored higher than 60% in all test cases, excluding our methods, our
speculation on which is included in Appendix E.3.

##### Results on ImageNet.

The AUROC and AUPR are shown in Table 7 in Appendix E.2. Our method achieved
detection accuracy of 100% or close to it across all evaluated pooling noise
images, validating our claims for large images.

##### Benefits of using two variables

are visually shown in Fig. 4 (left and center). Relying solely on $C({\bf x})$
results in the inability to differentiate between CIFAR-10 (In-Dist.) and
CelebA. Similarly, using only $\log p({\bf z})$ fails to differentiate between
CIFAR-10 (In-Dist.) and SVHN. Effective separation is achieved by utilizing
both variables.

##  6 Applicability to Autoregressive Model

We now shift our focus from Normalizing Flows (NFs) to Autoregressive (AR)
models, which are another type of DGM where instances of the failure of the
likelihood test have also been observed [9]. While our Hypothesis 1 is
developed based on a function invertible between latent space and data space,
AR models are designed without the concept of latent space. However, a recent
interpretation has proposed considering AR models as a single-layer NF, thus
implying the presence of an implicit latent space [15]. In this
interpretation, the latent distribution of PixelCNN++ is a discretized mixture
of logistics. With this perspective, we argue that our hypothesis is also
applicable to AR models and can provide an explanation for the failure of the
likelihood tests with AR models. To substantiate this claim, we present the
following experimental results.

###  6.1 Experiment 5: PixelCNN++

First, we present the results of the same experiments using GMM, previously
done, but this time applied to PixelCNN++. Similar to CV-Glow and IDF, the
volume term in PixelCNN++ is fixed (zero) as $\log p({\bf x})=\log p({\bf
z})$, so the GMM was constructed on $\log p({\bf x})$ and $C({\bf x})$. The
AUROC results are provided in Table 2, while the AUPR results are available in
Table 6 in Appendix E.3. The results clearly show that our complexity-aware
method significantly improves the OOD detection performance for PixelCNN++ as
well. These outcomes suggest the existence of the DCAS described in Remark 1
in PixelCNN++.

Second, as demonstrated in Appendix E.3, the response of $\log p({\bf x})$ (or
$\log p({\bf z})$) in PixelCNN++ to image complexity closely resembles that
observed in NFs with a fixed volume architecture, i.e., CV-Glow and IDF. This
finding suggests that the discussion pertaining to Remark 2 (Appendix D.4) is
also applicable to PixelCNN++. While further theoretical verification may be
necessary to conclusively establish the applicability of Hypothesis 1 to AR
models, our results provide plausible evidence supporting this assertion.

##  7 Limitation

The OOD detection task we addressed was aimed to discern differences between
datasets, such as CIFAR-10 and SVHN. More challenging tasks such as semantic
OOD detection — aimed at identifying objects not present in the training data
— have remained unexplored in this study. This is the limitation of this paper
and is an avenue left for future work.

##  8 Conclusion

In this study, we have proposed a hypothesis that explains the failure of OOD
detection methods based on the likelihood of Normalizing Flows and
Autoregressive models and delineates how the likelihood in those models is
affected by varying image complexity. We believe that the findings presented
in this paper will contribute to the future advancement of DGM applications,
particularly OOD detection.

## References

  * [1] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 427–436, 2015.
  * [2] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li.  Vos: Learning what you don’t know by virtual outlier synthesis.  In Proceedings of the International Conference on Learning Representations, 2022.
  * [3] Jasper Linmans, Jeroen van der Laak, and Geert Litjens.  Efficient out-of-distribution detection in digital pathology using multi-head convolutional neural networks.  In Tal Arbel, Ismail Ben Ayed, Marleen de Bruijne, Maxime Descoteaux, Herve Lombaert, and Christopher Pal, editors, Proceedings of the Third Conference on Medical Imaging with Deep Learning, volume 121 of Proceedings of Machine Learning Research, pages 465–478. PMLR, 06–08 Jul 2020\.
  * [4] Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, WENXUAN PENG, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu.  OpenOOD: Benchmarking generalized out-of-distribution detection.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [5] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.  Generalized out-of-distribution detection: A survey.  CoRR, abs/2110.11334, 2021.
  * [6] Peyman Morteza and Yixuan Li.  Provable guarantees for understanding out-of-distribution detection.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
  * [7] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan.  Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7498–7512. Curran Associates, Inc., 2020.
  * [8] Rui Huang and Yixuan Li.  Mos: Towards scaling out-of-distribution detection for large semantic space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8710–8719, June 2021.
  * [9] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  In International Conference on Learning Representations, 2019.
  * [10] Hyunsun Choi, Eric Jang, and Alexander A. Alemi.  Waic, but why? generative ensembles for robust anomaly detection, 2018\.
  * [11] Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque.  Input complexity and out-of-distribution detection with likelihood-based generative models.  In International Conference on Learning Representations, 2020.
  * [12] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan.  Detecting out-of-distribution inputs to deep generative models using typicality, 2020.
  * [13] A. Hyvärinen and E. Oja.  Independent component analysis: Algorithms and applications.  Neural Netw., 13(4–5):411–430, May 2000.
  * [14] Claude E Shannon.  A mathematical theory of communication.  The Bell system technical journal, 27(3):379–423, 1948.
  * [15] Didrik Nielsen and Ole Winther.  Closing the dequantization gap: Pixelcnn as a single-layer flow.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3724–3734. Curran Associates, Inc., 2020.
  * [16] Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan.  Likelihood ratios for out-of-distribution detection.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [17] Robin Schirrmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang.  Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21038–21049. Curran Associates, Inc., 2020.
  * [18] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson.  Why normalizing flows fail to detect out-of-distribution data.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20578–20589. Curran Associates, Inc., 2020.
  * [19] Ziyu Wang, Bin Dai, David Wipf, and Jun Zhu.  Further analysis of outlier detection with deep generative models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8982–8992. Curran Associates, Inc., 2020.
  * [20] Lily Zhang, Mark Goldstein, and Rajesh Ranganath.  Understanding failures in out-of-distribution detection with deep generative models.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12427–12436. PMLR, 18–24 Jul 2021.
  * [21] L. Theis, A. van den Oord, and M. Bethge.  A note on the evaluation of generative models.  In International Conference on Learning Representations, Apr 2016\.
  * [22] Jukka Perkiö and Aapo Hyvärinen.  Modelling image complexity by independent component analysis, with application to content-based image retrieval.  In Cesare Alippi, Marios Polycarpou, Christos Panayiotou, and Georgios Ellinas, editors, Artificial Neural Networks – ICANN 2009, pages 704–714, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.
  * [23] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  In International Conference on Learning Representations, 2015.
  * [24] Christopher M Bishop and Nasser M Nasrabadi.  Pattern recognition and machine learning, volume 4.  Springer, 2006.
  * [25] J. Rigau, M. Feixas, and M. Sbert.  An information-theoretic framework for image complexity.  In Proceedings of the First Eurographics Conference on Computational Aesthetics in Graphics, Visualization and Imaging, Computational Aesthetics’05, page 177–184, Goslar, DEU, 2005. Eurographics Association.
  * [26] Peter D. Grünwald and Paul M. B. Vitányi.  Kolmogorov complexity and information theory. with an interpretation in terms of questions and answers.  Journal of Logic, Language and Information, 12(4):497–529, 2003\.
  * [27] Khalid Sayood.  Introduction to data compression.  Morgan Kaufmann, 2017.
  * [28] Zhenghao Chen, Shuhang Gu, Guo Lu, and Dong Xu.  Exploiting intra-slice and inter-slice redundancy for learning-based lossless volumetric image compression.  IEEE Transactions on Image Processing, 31:1697–1707, 2022.
  * [29] Amirhossein Ahmadian and Fredrik Lindsten.  Likelihood-free out-of-distribution detection with invertible generative models.  In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2119–2125. International Joint Conferences on Artificial Intelligence Organization, 8 2021.  Main Track.
  * [30] Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530–1538, Lille, France, 07–09 Jul 2015. PMLR.
  * [31] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  In International Conference on Learning Representations, 2017.
  * [32] Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  In Advances in Neural Information Processing Systems 31, pages 10215–10224. Curran Associates, Inc., 2018.
  * [33] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen.  Invertible residual networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 573–582. PMLR, 09–15 Jun 2019.
  * [34] Ricky T. Q. Chen, Jens Behrmann, David K Duvenaud, and Joern-Henrik Jacobsen.  Residual flows for invertible generative modeling.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [35] Emiel Hoogeboom, Jorn Peters, Rianne van den Berg, and Max Welling.  Integer discrete flows and lossless compression.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [36] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma.  PixelCNN++: Improving the pixelCNN with discretized logistic mixture likelihood and other modifications.  In International Conference on Learning Representations, 2017.
  * [37] Chris M Bishop.  Training with noise is equivalent to tikhonov regularization.  Neural computation, 7(1):108–116, 1995.
  * [38] Herbert Federer.  Geometric measure theory.  Springer, 2014.
  * [39] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [40] Jens Behrmann, Paul Vicol, Kuan-Chieh Wang, Roger Grosse, and Joern-Henrik Jacobsen.  Understanding and mitigating exploding inverses in invertible neural networks.  In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1792–1800. PMLR, 13–15 Apr 2021.
  * [41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [43] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  CoRR, abs/1506.03365, 2015.
  * [44] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum.  Human-level concept learning through probabilistic program induction.  Science, 350(6266):1332–1338, 2015.
  * [45] Yaroslav Bulatov.  Notmnist dataset.  Google (Books/OCR), Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html, 2, 2011.
  * [46] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In International Conference on Learning Representations, 2014.
  * [47] Mu Cai and Yixuan Li.  Out-of-distribution detection via frequency-regularized generative models.  In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
  * [48] Thomas M Cover and Joy A Thomas.  Elements of information theory.  John Wiley & Sons, 2012.
  * [49] Carlos Fernandez-Granda.  Optimization-based data analysis: Lecture notes 3: Randomness, Fall 2017\.
  * [50] Jonathan Ho, Evan Lohn, and Pieter Abbeel.  Compression with flows via local bits-back coding.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [51] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations, 2015.
  * [52] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.

##  Appendix A Additional Related Works

##### DGM-based approaches.

[20] investigated the failures of the likelihood test and the typicality test
for OOD detection. They argued that without specifying the type of OOD to be
detected, it is impossible to rely on a single statistical test, even if it is
based on the likelihood. They also discussed the failure of the typicality
test when the out-distribution overlaps with the support of the in-
distribution. In [19], the failure of the typicality test on latent space of a
VAE [46] was attributed to the fact that its latent space did not form a
perfect Gaussian. While these studies demonstrated the failures of the
likelihood test and the typicality test, our work takes a step further by
providing an explanation about _why_ they fail. Importantly, we show that the
underlying causes of both testing methods’ failures are fundamentally the
same.

Several studies have highlighted that DGMs exhibit a sensitivity to the
_smoothness_ of images, which can lead to the failure of likelihood test. The
term _smooth_ refers to regions where adjacent pixels have constant or very
close values. For example, [16] demonstrated in experiments on MNIST and
FMNIST that backgrounds with black pixels were assigned high likelihoods.
Similarly, [9] showed that converting RGB images to black and white increased
their likelihoods. The influence of smoothness on likelihood was also observed
by [17], which attributed it to the nature of convolutional networks.
Additionally, [18] focused on NFs and argued that the coupling layer structure
in NFs makes them more predictable in smooth regions, and thus images
containing many smooth regions will have higher likelihoods. Moreover, [47]
proposed a method that emphasizes high-frequency features associated with
object contours. The concept of _less complexity_ mentioned in this paper
aligns with the notion of smoothness discussed in these studies. However, we
emphasize that the cause of irrationally increased likelihoods is not specific
to a particular network structure but rather a density concentration in the
latent space.

##  Appendix B Existing Methods

###  B.1 Typical Set and Typicality Test

The concept of the _typical set_ refers to a set or region that contains a
significant amount of probability mass in a given distribution. It is a
fundamental concept in probability theory (see [12, 48] for formal
definitions). In the context of OOD detection, [10, 12] argued that the
failure of the likelihood test can be attributed to ignorance of the notion of
the typical set. While samples drawn from a DGM are highly likely to come from
its typical set, in high-dimensional spaces, the typical set may not
necessarily coincide with high-density or high-likelihood regions. This
phenomenon becomes more pronounced as the dimensionality increases. While
specifying the region of the typical set for arbitrary distributions is
difficult, it is possible for an isotropic Gaussian distribution. It is well
known that the typical set of samples from a $d$-dimensional Gaussian
$\mathcal{N}(0,\textbf{I}_{\text{d}})$ resides in a hypersphere with a radius
of $\sqrt{d}$ with high probability. Specifically, for any $\epsilon\in(0,1)$,
we have

$\displaystyle\mathbb{P}\left(\sqrt{d(1-\epsilon)}<\left\lVert{\bf
z}\right\rVert<\sqrt{d(1+\epsilon)}\right)\geq
1-2\text{exp}\left(-\frac{d\epsilon^{2}}{8}\right).$ (4)

We refer the reader to [49], for example, for the derivation. As the dimension
$d$ increases, the radius of the hypersphere also increases, causing the
region where the typical set (i.e., In-Dist samples) resides to move away from
the region of highest likelihood, which corresponds to the mean of the
Gaussian distribution. Consequently, In-Dist examples are often assigned low
likelihoods in high-dimensional spaces. To address this issue, [10, 12]
proposed a method, the typicality test in latent space (TTL). The TTL utilizes
NFs with latent distribution of $\mathcal{N}(0,\textbf{I}_{\text{d}})$ to
obtain the latent vector ${\bf z}$ corresponding to a test input ${\bf x}$. It
then computes the score $S_{\text{TTL}}({\bf x})={\tt abs}(\left\lVert{\bf
z}\right\rVert-\sqrt{d})$, where $\lVert\cdot\rVert$ denotes the $L_{2}$ norm.
If ${\bf x}$ falls outside the typical set, the score $S_{\text{TTL}}({\bf
x})$ will be large, indicating that ${\bf x}$ is likely to be an OOD input. We
present our experimental results on the TTL in Appendix B.2.

###  B.2 Our Experiments

Figure 5: Existing methods fail in specific combinations. The cases where In-
Dist is CIFAR-10 are shown in the left columns, and the cases where In-Dist is
SVHN are shown in the right columns. Top row: histograms of CALT. The $x$-axis
is $S_{\text{CALT}}({\bf x})=\log p({\bf x})+C({\bf x})$. Bottom row:
histograms of TTL. The $x$-axis is $\left\lVert{\bf
z}\right\rVert(\propto-\sqrt{\log p({\bf z})})$. CALT fails to detect CIFAR-10
samples as OOD when SVHN is In-Dist (top-right). TTL cannot identify SVHN
samples as OOD when CIFAR-10 is In-Dist (bottom-left).

Figure 6: Decomposing $S_{\text{CALT}}({\bf x})$ into $\log p({\bf x})$ and
$C({\bf x})$ on CIFAR-10 (left) and SVHN (right).

We experimented two existing OOD detection methods, the complexity aware
likelihood test (CALT) and the typicality test in latent space (TTL). The
experiments were performed with Glow on SVHN and CIFAR-10, and we also used
CelebA as the OOD datasets with resizing to $32$. The test samples are the
same we used in our experiments. Fig. 5 (top two) shows the histogram of the
score that CALT bases on, and Fig. 6 shows its components. It shows that CALT
fails to detect CIFAR-10 samples as OOD when SVHN is In-Dist (top-right). Fig.
5 (bottom two) shows the histogram of $\left\lVert{\bf z}\right\rVert$ that
the TTL bases on. It shows that TTL cannot identify SVHN samples as OOD when
CIFAR-10 is In-Dist (bottom-left).

##  Appendix C Derivation of Hypothesis 1

We establish Hypothesis 1. In the derivation, we consider generating samples
${\bf x}$ in the data space $\mathcal{X}$ from a small region $\mathcal{A}$ in
the latent space $\mathcal{Z}$. In the derivation provided in Appendix C.1, we
assume that the set of ${\bf x}$ generated from
$\mathcal{A}\subset\mathcal{Z}$ follows a Gaussian distribution. A more
generalized scenario, without the Gaussian assumption, is derived from
experimental observations in Appendix C.2 .

###  C.1 Analysis

Introducing two lemmas, we derive Observation 3, which aligns with Hypothesis
1. We assume that the probability density function is Gaussian in a subdomain
$f^{-1}(\mathcal{A})\subset\mathcal{X}$. The invertible function (practically,
we assume Normalizing Flow) $f:\mathcal{X}\rightarrow\mathcal{Z}$ is locally
$L_{\mathcal{A}}$-Lipschitz with respect to a small region
$\mathcal{A}\subset\mathcal{Z}$. We note that in a different region
$\mathcal{A}^{\prime}$, the Lipschitz coefficient of $f$ differs and becomes
$L_{\mathcal{A}^{\prime}}$. In other words, the local Lipschitz coefficient of
$f$ depends on which region in $\mathcal{Z}$ the sample ${\bf x}$ is mapped
to. The relationship assumed in Lemmas 1 and 2 is illustrated in Fig. 7.

Figure 7: Illustration of the relationship in Lemmas 1 and 2. In Lemma 2, the
green region $f^{-1}(\mathcal{A})\subset\mathcal{X}$ is assumed to follow a
Gaussian distribution with a mean ${\bf
x}=\operatorname{\mathbb{E}}[f^{-1}({\bf z}^{\prime})]$. Assuming that the
latent space $\mathcal{Z}$ is semantically continuous, when ${\bf x}^{\prime}$
is generated from a sufficiently small $\mathcal{A}$, they share common
semantics, and thus the image complexities $C({\bf x}^{\prime})$ and $C({\bf
x})$ can be considered nearly equal.

######  Lemma 1.

Let $f:\mathcal{X}\rightarrow\mathcal{Z}$ be an invertible function being
locally $L_{\mathcal{A}}$-Lipschitz for $\mathcal{A}\subset\mathcal{Z}$. For
all ${\bf z}^{\prime}\in\mathcal{A}$, let ${\bf x}^{\prime}=f^{-1}({\bf
z}^{\prime})$, ${\bf z}=\operatorname{\mathbb{E}}{\bf
z}^{\prime}=[\operatorname{\mathbb{E}}z_{1}^{\prime},\dots,\operatorname{\mathbb{E}}z_{d}^{\prime}]^{T}$,
and ${\bf x}=f^{-1}({\bf z})$. Assume ${\bf z}\in\mathcal{A}$, then ${\bf
x}\in f^{-1}(\mathcal{A})$. Also, let $\mathcal{B}^{\epsilon}_{{\bf
z}}=\\{{\bf z}^{\prime}\in\mathcal{A}:\left\lVert{\bf z}^{\prime}-{\bf
z}\right\rVert<\epsilon\\}$ and $\mathcal{B}^{\epsilon}_{{\bf
z}}\subset\mathcal{A}$ for a constant $\epsilon\geq 0$. Then, we have

$\displaystyle\frac{\epsilon^{2}}{{L_{\mathcal{A}}}^{2}}\left(1-\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf
z}})\right)\leq\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim
f^{-1}(\mathcal{A})}\left\lVert{\bf x}^{\prime}-{\bf x}\right\rVert^{2}.$ (5)

###### Proof.

Since $f$ is (locally) $L_{\mathcal{A}}$-Lipschitz, we know $\left\lVert{\bf
z}^{\prime}-{\bf z}\right\rVert=\left\lVert f({\bf x}^{\prime})-f({\bf
x})\right\rVert\leq L_{\mathcal{A}}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert$. Then, we have

$\displaystyle\mathbb{P}\left(\left\lVert{\bf z}^{\prime}-{\bf
z}\right\rVert\geq\epsilon\right)$ $\displaystyle\leq$
$\displaystyle\mathbb{P}\left(L_{\mathcal{A}}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert\geq\epsilon\right)$ (6) $\displaystyle=$
$\displaystyle\mathbb{P}\left(\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert\geq\frac{\epsilon}{L_{\mathcal{A}}}\right)$ (7)
$\displaystyle=$ $\displaystyle\mathbb{P}\left(\left\lVert{\bf
x}^{\prime}-{\bf
x}\right\rVert^{2}\geq\frac{\epsilon^{2}}{{L_{\mathcal{A}}}{{}^{2}}}\right)$
(8) $\displaystyle\leq$
$\displaystyle\frac{{L_{\mathcal{A}}}^{2}}{\epsilon^{2}}\operatorname{\mathbb{E}}_{{\bf
x}^{\prime}\sim f^{-1}(\mathcal{A})}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2}.$ (9)

We applied Markov’s inequality to Eq. (8). Since
$\mathbb{P}\left(\left\lVert{\bf z}^{\prime}-{\bf
z}\right\rVert\geq\epsilon\right)=1-\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf
z}})$, the statement therefore follows. ∎

######  Lemma 2.

The mean square error (MSE) for $d$-dimensional random vectors sampled from
Gaussian distribution with mean vector ${\bf x}$ and isotropic covariance
matrix $\Sigma_{{\bf x}}=\sigma^{2}\mathbb{I}_{\text{d}}$, $\mathcal{N}({\bf
x},\Sigma_{{\bf x}})$, can be expressed using entropy $H_{\Sigma_{{\bf
x}}}=\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}[-\log p({\bf x}^{\prime})]$ as follows:

$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2}=\frac{d}{2\pi e}\exp\left(\frac{2}{d}H_{\Sigma_{{\bf
x}}}\right).$ (10)

###### Proof.

The MSE can be expressed as the trace of covariance matrix as follows:

$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}\left\lVert{\bf x}^{\prime}-{\bf x}\right\rVert^{2}$ (11)
$\displaystyle=$ $\displaystyle\operatorname{\mathbb{E}}_{{\bf
x}^{\prime}\sim\mathcal{N}({\bf x},\Sigma_{{\bf x}})}[({\bf x}^{\prime}-{\bf
x})^{T}({\bf x}^{\prime}-{\bf x})]$ (12) $\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}\left[\operatorname{Tr}{(({\bf x}^{\prime}-{\bf x})({\bf
x}^{\prime}-{\bf x})^{T})}\right]$ (13) $\displaystyle=$
$\displaystyle\operatorname{Tr}{\left(\operatorname{\mathbb{E}}_{{\bf
x}^{\prime}\sim\mathcal{N}({\bf x},\Sigma_{{\bf x}})}[({\bf x}^{\prime}-{\bf
x})({\bf x}^{\prime}-{\bf x})^{T}]\right)}$ (14) $\displaystyle=$
$\displaystyle\operatorname{Tr}{\left(\operatorname{\mathbb{E}}_{{\bf
x}^{\prime}\sim\mathcal{N}({\bf x},\Sigma_{{\bf x}})}[({\bf
x}^{\prime}-\operatorname{\mathbb{E}}{\bf x}^{\prime})({\bf
x}^{\prime}-\operatorname{\mathbb{E}}{\bf x}^{\prime})^{T}]\right)}$ (15)
$\displaystyle=$ $\displaystyle\operatorname{Tr}{(\Sigma_{{\bf x}})}.$ (16)

Since we assumed $\Sigma_{{\bf x}}=\sigma^{2}\mathbb{I}_{\text{d}}$, equality
holds in Hadamard’s inequality for $\Sigma_{{\bf x}}$ as

$\displaystyle\frac{1}{d}\operatorname{Tr}{(\Sigma_{{\bf
x}})}=\left(\det\Sigma_{{\bf x}}\right)^{\frac{1}{d}}.$ (17)

Also, it is well known that entropy of multivariate Gaussian is

$\displaystyle H_{\Sigma_{{\bf x}}}$ $\displaystyle:=$
$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}[-\log p({\bf x}^{\prime})]$ (18) $\displaystyle=$
$\displaystyle\frac{d}{2}(1+\log(2\pi))+\frac{1}{2}\log\det\Sigma_{{\bf x}}$
(19) $\displaystyle=$ $\displaystyle\frac{d}{2}\log(2\pi
e\left(\det\Sigma_{{\bf x}}\right)^{\frac{1}{d}}).$ (20)

By replacing $\left(\det\Sigma_{{\bf x}}\right)^{\frac{1}{d}}$ with
$\frac{1}{d}\operatorname{\mathbb{E}}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2}$, we have

$\displaystyle H_{\Sigma_{{\bf x}}}=\frac{d}{2}\log(\frac{2\pi
e}{d}\operatorname{\mathbb{E}}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2})$ (21)

and thus we obtain the statement. ∎

##### Discretizing continuous data.

We aim to establish a connection between Lemma 1 and 2 and the notion of image
complexity. In Definition 1, image complexity for an image ${\bf
x}\in\\{0,1,\ldots,255\\}^{d}$ is defined using the code length, denoted as
$L({\bf x})$, as $C({\bf x})=\frac{1}{d}L({\bf x})$, and thus we aim to
connect Lemma 1 and 2 to $L({\bf x})$. However, in Lemma 1 and 2, ${\bf x}$ is
a continuous variable, and thus establishing a direct connection with the code
length is unfeasible. Thus, we discretize ${\bf x}$ as $\bar{{\bf x}}$ and
code $\bar{{\bf x}}$ with a certain probability mass function (PMF),
$P(\bar{{\bf x}})$. This approach is detailed in [50, section 3.1] and [48,
section 8.3]. We divide $\mathbb{R}^{d}$ into bins of volume
$\delta_{x}:=2^{-kd}$ with precision $k$. When the probability density
function $p({\bf x})$ is sufficiently smooth, the PMF can be approximated as

$\displaystyle P(\bar{{\bf x}})\approx p(\bar{{\bf x}})\delta_{x}.$ (22)

For instance, with $k=8$, a continuous data ${\bf x}$ is discretized as
$\bar{{\bf x}}\in\\{0,1,\ldots,255\\}^{d}$. We utilize this approach in the
following.

######  Observation 2.

The MSE exhibits the following relationship with the image complexity of
discretized version of the mean of ${\bf x}^{\prime}$, represented as
$C(\bar{{\bf x}})$, with the volume of bins used in discretization,
$\delta_{x}$:

$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2}\leq\frac{d}{2\pi
e}\delta_{x}^{\frac{2}{d}}\exp\left(2C(\bar{{\bf x}})\right).$ (23)

Letting $\bar{{\bf x}}^{\prime}$ represent a discretized version of ${\bf
x}^{\prime}\in\mathbb{R}^{d}$, we approximate the entropy $H_{\Sigma_{{\bf
x}}}$ as follows:

$\displaystyle H_{\Sigma_{{\bf x}}}$ $\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}[-\log p({\bf x}^{\prime})]$ (24) $\displaystyle\approx$
$\displaystyle\operatorname{\mathbb{E}}_{\bar{{\bf x}}^{\prime}}[-\log
p(\bar{{\bf x}}^{\prime})]$ (25) $\displaystyle\approx$
$\displaystyle\operatorname{\mathbb{E}}_{\bar{{\bf
x}}^{\prime}}[-\log\frac{P(\bar{{\bf x}}^{\prime})}{\delta_{x}}]$ (26)
$\displaystyle=$ $\displaystyle\operatorname{\mathbb{E}}_{\bar{{\bf
x}}^{\prime}}[-\log P(\bar{{\bf x}}^{\prime})]+\log\delta_{x}.$ (27)

We applied Eq. (22) to Eq. (25). Applying Shannon’s source coding theorem [14]
and the definition of image complexity (Definition 1), we have

$\displaystyle\operatorname{\mathbb{E}}_{\bar{{\bf x}}^{\prime}}[-\log
P(\bar{{\bf x}}^{\prime})]$ $\displaystyle\leq$
$\displaystyle\operatorname{\mathbb{E}}_{\bar{{\bf x}}^{\prime}}[L(\bar{{\bf
x}}^{\prime})]$ (28) $\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}_{\bar{{\bf x}}^{\prime}}[dC(\bar{{\bf
x}}^{\prime})].$ (29)

Then, we employ Assumption 1, which posits that latent space $\mathcal{Z}$ is
semantically continuous. When considering $\mathcal{A}\subset\mathcal{Z}$ as a
very small region, generated samples ${\bf x},{\bf x}^{\prime}\in
f^{-1}(\mathcal{A})$ share common semantics. Consequently, we can approximate
the image complexities of the discretized version of ${\bf x}$ and ${\bf
x}^{\prime}$ as equivalent, i.e., $C(\bar{{\bf x}})\approx C(\bar{{\bf
x}}^{\prime})$, and thus we have $\operatorname{\mathbb{E}}_{\bar{{\bf
x}}^{\prime}}[C(\bar{{\bf x}}^{\prime})]\approx C(\bar{{\bf x}})$. Putting
them together, we have $H_{\Sigma_{{\bf x}}}\leq dC(\bar{{\bf
x}})+\log\delta_{x}$. By substituting this inequality to Eq. (10), we have
$\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim\mathcal{N}({\bf
x},\Sigma_{{\bf x}})}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2}\leq\frac{d}{2\pi e}\exp\left(\frac{2}{d}\left(dC(\bar{{\bf
x}})+\log\delta_{x}\right)\right)$ and obtain Eq. (23).

######  Observation 3.

If $f^{-1}(\mathcal{A})$ follows Gaussian $\mathcal{N}({\bf x},\Sigma_{{\bf
x}})$, we have

$\displaystyle\frac{\epsilon^{2}}{{L_{\mathcal{A}}}^{2}}\left(1-\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf
z}})\right)\leq\frac{d}{2\pi e}\delta_{x}^{\frac{2}{d}}\exp\left(2C(\bar{{\bf
x}})\right).$ (30)

Combining Eq. (23) to Eq. (5) yields the above.

In the following section, we will verify whether the same relationship holds
true, even when we remove the assumption of Gaussian distribution, through
experimentation.

###  C.2 Experimental Observation

In the previous section C.1, Eq. (23) holds under the constraint that samples
${\bf x}$ generated from $\mathcal{A}\subset\mathcal{Z}$ follow a Gaussian
distribution. In the following, we experimentally demonstrate that the
inequality between the MSE and the image complexity holds true even when the
constraint is relaxed.

The details of the experiments are presented in Appendix D.2. We generate
samples from the latent space $\mathcal{Z}$ of a trained Glow and use them as
${\bf x}$’ in Section C.1. Specifically, for a source image ${\bf x}$, we
define a hypersphere $\mathcal{B}^{\delta}_{f({\bf x})}$ with radius $\delta$
centered at ${\bf z}=f({\bf x})$ in $\mathcal{Z}$. We randomly sample ${\bf
z}^{\prime}$ from $\mathcal{B}^{\delta}_{f({\bf x})}$ and generate ${\bf
x}^{\prime}$ from these ${\bf z}^{\prime}$. Note that ${\bf x}^{\prime}$is not
guaranteed to follow a Gaussian distribution.

######  Observation 4.

For an image data ${\bf x}$, we experimentally observed the following
relationship between the MSE and the image complexity $C({\bf x})$:

$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim
f^{-1}(\mathcal{B}^{\delta}_{f({\bf x})})}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2}\propto\exp\left(C({\bf x})\right).$ (31)

To measure $C({\bf x})$, we used the JPEG2000, which is an entropy-based
lossless compression algorithm. This observation was established through
experiments conducted on CIFAR-10 and SVHN with different values of $\delta$
as $\delta=\\{0.01,0.1,1.0\\}$. The details of the experiment are explained in
Appendix D.2.

##### Derivation of Hypothesis 1.

By combining the observation of Eq. (31) with Eq. (5), we obtain Eq. (3):

$\displaystyle\frac{\epsilon^{2}}{L_{\mathcal{A}}^{2}}\left(1-\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf
z}})\right)$ $\displaystyle\leq$ $\displaystyle\operatorname{\mathbb{E}}_{{\bf
x}^{\prime}\sim f^{-1}(\mathcal{B}^{\delta}_{f({\bf x})})}\left\lVert{\bf
x}^{\prime}-{\bf x}\right\rVert^{2}$ (32) $\displaystyle=$ $\displaystyle
C_{1}\exp(C({\bf x})),$ (33)

where $C_{1}$ is a constant.

##  Appendix D Experiments

The experiments were conducted on a CentOS server equipped with a single
NVIDIA V100 GPU. Compression with JPEG2000 was performed using the Python PIL
package.

###  D.1 Datasets with controlled Image complexity

#### Pooling Noise Image

We generate images with controlled complexity by following the method proposed
in [11]. First, we sample RGB noise images with a size of $32\times 32$ from a
uniform random distribution. Next, we apply average pooling to these images
and resize them back to $32\times 32$. The complexity of the images is
controlled by the pooling size, denoted as $\kappa$. Images with $\kappa=1$
(i.e., no pooling) are the most complex, while those with $\kappa=32$ (i.e.,
constant-color image) are the least complex. For convenience, we refer to
these images as _pooling noise image_. Sample images are shown in Fig. 2.

#### Manipulated CIFAR-10

To create the _manipulated CIFAR-10_ dataset, we randomly select 1024 images
from the test set of CIFAR-10. We manipulate the image complexity by adding a
$4\times 4$ patch of weak noise ($[-0.2,0.2]$) to a randomly selected area of
each image. The complexity of the images increases with the number of patches,
denoted as $n$. We label the set of images with $n$ patches as Noise-4-$n$. In
addition, we decrease the image complexity by applying the pooling operation
in the same manner as the pooling noise images. We denote the set with a
pooling size of $\kappa$ as Pool-$\kappa$, and the image complexity decreases
from Pool-2 to Pool-16. Sample images are shown in Fig. 2.

###  D.2 Measuring MSE and Image Complexity

Figure 8: Relationship between the Mean Squared Error (MSE),
$\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim
X_{\kappa}^{\delta}}\left\lVert{\bf x}^{\prime}-{\bf
x}_{\kappa}\right\rVert^{2}$, and image complexity. The $x$-axis represents
$C({\bf x}_{\kappa})$ computed with JPEG2000. The $y$-axis is displayed on a
logarithmic scale, indicating that $C({\bf x}_{\kappa})$ is proportional to
$\log\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim
X_{\kappa}^{\delta}}\left\lVert{\bf x}^{\prime}-{\bf
x}_{\kappa}\right\rVert^{2}$.

Using the pooling noise images with $\kappa=\\{1,2,4,8,16,32\\}$, we have six
_source images_ ${\bf x}_{\kappa}$ with varying complexity. We then obtain the
corresponding latent vector ${\bf z}_{\kappa}=f({\bf x}_{\kappa})$ for each
source image, where $f$ represents a trained Glow model. Next, we define the
set $B^{\delta}_{{\bf z}_{\kappa}}=\\{{\bf z}^{\prime}:\left\lVert{\bf
z}^{\prime}-{\bf z}_{\kappa}\right\rVert\leq\delta\\}$, which represents
points within a hypersphere with a radius of $\delta$ centered at ${\bf
z}_{\kappa}$. We sample ${\bf z}^{\prime}$ from $B^{\delta}_{{\bf
z}_{\kappa}}$ and obtain a set of images $X_{\kappa}^{\delta}=\\{{\bf
x}^{\prime}=f^{-1}({\bf z}^{\prime}):{\bf z}^{\prime}\sim B^{\delta}_{{\bf
z}_{\kappa}}\\}$. In our experiment, we varied $\delta$ as
$\delta=\\{0.01,0.1,1.0\\}$. We randomly sampled $1024$ ${\bf z}^{\prime}$
from $B^{\delta}_{{\bf z}_{\kappa}}$ and obtained $1024$ corresponding images
in $X_{\kappa}^{\delta}$. For two Glow models trained on CIFAR-10 and SVHN
datasets, we measured $\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim
X_{\kappa}^{\delta}}\left\lVert{\bf x}^{\prime}-{\bf
x}_{\kappa}\right\rVert^{2}$ for each source image ${\bf x}_{\kappa}$. We then
represented these measurements on the $y$-axis in Fig. 8. The $x$-axis
represents the image complexity $C({\bf x}_{\kappa})$ computed using JPEG2000.
$C({\bf x}_{\kappa})$ varies with different values of ${\kappa}$. From the
plotted figures, we observed that, irrespective of the value of $\delta$, the
relationship between $C({\bf x})$ and $\log\operatorname{\mathbb{E}}_{{\bf
x}^{\prime}\sim X_{\kappa}^{\delta}}\left\lVert{\bf x}^{\prime}-{\bf
x}_{\kappa}\right\rVert^{2}$ approximately follows a linear trend. This
suggests the following connection:

$\displaystyle\operatorname{\mathbb{E}}_{{\bf x}^{\prime}\sim
f^{-1}(\mathcal{B}^{\delta}_{f({\bf x})})}\left\lVert{\bf x}^{\prime}-{\bf
x}\right\rVert^{2}\propto\exp\left(C({\bf x})\right).$ (34)

###  D.3 Models

#### Model Specification

We implemented the DGMs using publicly available code. The architecture of
Glow and CV-Glow111github.com/kmkolasinski/deep-learning-
notes/tree/master/seminars/2018-10-Normalizing-Flows-NICE-RealNVP-GLOW mainly
depends on two parameters: the depth of flow, denoted as $K$, and the number
of levels, denoted as $L$. The flow consists of affine coupling and $1\times
1$ convolutions, which are performed $K$ times, followed by the factoring-out
operation [31]. This sequence is repeated $L$ times. For CIFAR-10 and SVHN, we
set $K=32$ and $L=3$. For MNIST and FMNIST, we set $K=48$ and $L=2$. For
ImageNet, we set $K=64$ and $L=5$. We applied data augmentation techniques of
random $2\times 2$ translations and horizontal flipping on CIFAR-10. Random
cropping at a size of $224\times 224$ and horizontal flipping were applied to
ImageNet. We trained the models for $99100$ iterations for ImageNet and
$45100$ iterations for other datasets using the Adam optimizer [51] with
$\beta_{1}=0.9$ and $\beta_{2}=0.999$. The batch size is $8$ for ImageNet and
$256$ for other datasets. The learning rate was set to $1e-4$ for the first
$5100$ iterations and 5e-5 for the remaining iterations for all datasets. For
ResFlow,222github.com/rtqichen/residual-flows we used the pre-trained model on
CIFAR-10 and MNIST and trained the models on SVHN and FMNIST with the same
setting as CIFAR-10 and MNIST, respectively. For
iResNet,333github.com/jhjacobsen/invertible-resnet we used the pre-trained
model on CIFAR-10 and trained the models on SVHN, MNIST, and FMNIST with the
same setting as for CIFAR-10. For
IDF,444github.com/jornpeters/integer_discrete_flows we trained the models on
all datasets using the settings provided by the original author for the
CIFAR-10 model. For PixelCNN++,555github.com/pclucas14/pixel-cnn-pp we
utilized the pre-trained model on CIFAR-10 and trained the models on SVHN with
the same setting as CIFAR-10.

###  D.4 Sensitivity to Image Complexity

The strength of the positive correlation between $\log p({\bf z})$ and the
volume $\log|\text{det}\ J_{f}({\bf x})|$ discussed in Section 3.2, Experiment
3, depends on the architecture of the NF, specifically its flexibility
measured as the Lipschitz constant, $L$. This can be observed from the
relationship $|\text{det}\ J_{f}({\bf x})|<L^{d}$ (Remark 2). For CV-Glow and
IDF, where the volume term is fixed, it was observed that the increase in
$\log p({\bf z})$ was equivalent to the increase in $\log p({\bf x})$. On the
other hand, for iResFlow and ResFlow, which restrict the Lipschitz constant
($L\leq 1$), the impact of variation in $\log p({\bf x})$ on the volume was
reduced. As a result, more of the variation in $\log p({\bf x})$ was directed
towards the variation in $\log p({\bf z})$. In Glow, there is no such
restriction on the Lipschitz constant. Therefore, the sensitivity of $\log
p({\bf z})$ to variations in image complexity is the lowest for Glow, followed
by iResFlow and ResFlow, and the highest for CV-Glow and IDF. The same
conclusion can be drawn from Eq. (3), where the sensitivity of the density
around ${\bf z}$, $\mathbb{P}(\mathcal{B}^{\epsilon}_{{\bf z}})$, to
variations in $C({\bf x})$ increases when the flexibility is restricted (i.e.,
$L$ is small). Based on this property, we select the NF architecture for OOD
detection, as discussed in Section 5.1 and Appendix D.5.

###  D.5 Preparation for Experiments 4 and 5

#### Selection of NF architecture for OOD detection

For Experiment 4, we employed three Normalizing Flow (NF) models: Glow,
iResNet, and ResFlow. In this study, OOD detection is treated as the task of
distinguishing an in-distribution dataset from other datasets. The
architecture of the NF model plays a pivotal role in effectively capturing
large-scale differences between datasets, while remaining less sensitive to
small-scale variations among individual samples. To achieve robustness in this
regard, it is desirable for NF models to be somewhat insensitive to subtle
differences between individual samples. The flexibility of $J_{f}({\bf x})$ in
NFs, as discussed in Appendix D.4, determines the sensitivity of $\log p({\bf
z})$ to sample-specific differences in image complexity. From this standpoint,
the discriminative capacity using $\log p({\bf z})$ for dataset
differentiation can be ranked. Glow exhibits the lowest sensitivity, followed
by iResNet and ResFlow, while CV-Glow and IDF show the highest sensitivity. We
validate these characteristics for Glow, ResFlow, and IDF in Fig. 4, with
supplementary results for the other two NF models presented in Appendix E.3.
The plots for IDF exhibit a high sensitivity to $\log p({\bf z})$, leading to
significant overlap between the In-Dist and OOD datasets. Similar overlapping
plots were also observed for CV-Glow. In contrast, Glow, iResNet, and ResFlow
exhibit less overlap and provide enhanced discrimination. Consequently, we
opted for Glow, iResNet, and ResFlow.

#### OOD detection with GMM

We present a complexity-aware OOD detection method using GMM. In the problem
set-up of OOD detection, In-Dist data are assumed to be available, and we
train a GMM to fit the In-Dist data at hand. Here is the operational procedure
of our proposed method:

  1. 1.

Compute the image complexity $C_{{\bf x}_{\text{train}}}=C({\bf
x}_{\text{train}})$ and the log likelihood in latent space $L_{{\bf
z}_{\text{train}}}=\log p({\bf z}_{\text{train}})$ for the training images
${\bf x}_{\text{train}}$ from the In-Dist dataset (i.e., CIFAR-10, SVHN,
MNIST, FMNIST, and ImageNet). ${\bf z}_{\text{train}}=f({\bf
x}_{\text{train}})$ is obtained by applying the NF model $f$ to ${\bf
x}_{\text{train}}$.

  2. 2.

Train a GMM, denoted as $\mathcal{G}=\\{\pi_{k},\mu_{k},\Sigma_{k}\\}$,
utilizing the computed $C_{{\bf x}_{\text{train}}}$ and $L_{{\bf
z}_{\text{train}}}$. In this representation, $\pi_{k},\mu_{k}$, and
$\Sigma_{k}$ correspond to the mixing coefficient, mean, and covariance matrix
for the $k$-th component of the GMM. The parameter $K$ signifies the number of
mixture components.

  3. 3.

For a given test image ${\bf x}_{\text{test}}$, compute its image complexity
$C_{{\bf x}_{\text{test}}}=C({\bf x}_{\text{test}})$ and the log likelihood in
latent space $L_{{\bf z}_{\text{test}}}=\log p({\bf z}_{\text{test}})$, where
${\bf z}_{\text{test}}=f({\bf x}_{\text{test}})$.

  4. 4.

Compute the log likelihood for the trained GMM $\mathcal{G}$ as
$S_{\text{GMM}}({\bf
x}_{\text{test}})=\log\Sigma_{k=1}^{K}\pi_{k}\mathcal{N}\left((C_{{\bf
x}_{\text{test}}},L_{{\bf z}_{\text{test}}})|\mu_{k},\Sigma_{k}\right)$.
$S_{\text{GMM}}({\bf x}_{\text{test}})$ represents the In-Dist score of ${\bf
x}_{\text{test}}$, where a larger value indicates that ${\bf x}_{\text{test}}$
is more likely to belong to the In-Dist.

The dimensionality of GMM is as small as two, so there is no need to concern
about the discrepancy between the high-density region and the typical set
region, which occurs in high-dimensional spaces (Appendix B.1).

In the experiments, we had $1024$ training samples ${\bf x}_{\text{train}}$
and $2048$ test samples ${\bf x}_{\text{test}}$, consisting of $1024$ samples
from the OOD dataset and $1024$ samples from the test portion of the In-Dist
dataset. We used the scikit-learn [52] with its default hyper-parameter
values, including the number of Gaussian components $K=3$. We also tried
different values for $K$, such as $K=2$ and $K=4$, but the results were
similar to using $K=3$.

#### Competitors

In addition to TTL and CALT, we compare the performance of the proposed
methods to the existing NF-based methods on CIFAR-10 and SVHN as follows:

  1. 1.

The Watanabe-Akaike Information Criterion (WAIC) [10] measures
$\mathbb{E}[\log p({\bf x})]-\text{Var}[\log p({\bf x})]$ with an ensemble of
five Glow models. The five models were trained separately with different
initial parameters. CV-Glow and a background model used in the LRB method are
included as components of the ensemble.

  2. 2.

The likelihood-ratio to background model (LRB) [16] measures the likelihood
ratio $\log p({\bf x})-\log p_{0}({\bf x})$ where $p_{0}({\bf x})$ is a
background model trained using the training data with additional random noise
sampled from the Bernoulli distribution with a probability 0.15. LRB utilizes
two Glow models for the likelihood computation.

  3. 3.

The likelihood-ratio to general model (LRG) [17] measures the likelihood ratio
$\log p({\bf x})-\log p_{g}({\bf x})$ where $p_{g}({\bf x})$ is a general
model trained on broader and more general data than In-Dist data.
Specifically, TinyImageNet [42] is used for training the general model.

##  Appendix E More Experimental Results

###  E.1 Additional Results of Experiment 1

Here are the additional results of Experiment 1 with other NF models: Fig. 9
shows the results on the pooling noise images. Fig. 10 shows the results on
the manipulated CIFAR-10. It can be observed that CV-Glow assigns an extremely
small $\log p({\bf z})$ to images with high complexity, such as Noise-2,
Noise-1 in Fig. 9, as well as Noise-4-8, and Noise-4-16 in Fig. 10, This leads
to divergent values of $\left\lVert{\bf z}\right\rVert$. To ensure proper
visualization of the figures, the $y$-axis scale, (upper limit of
$\left\lVert{\bf z}\right\rVert$) was capped at $100$ in the CV-Glow plots.

Figure 9: Complexity vs. $\left\lVert{\bf z}\right\rVert(\propto-\sqrt{\log
p({\bf z})})$ on the pooling noise images. In-Dist is CIFAR-10. The NF models
are Glow, ResFlow, and iResNet for the top three and IDF and CV-Glow for the
bottom two. In CV-Glow, $\left\lVert{\bf z}\right\rVert$ for the samples of
Noise -1 and Noise-2 became extremely large, so we omitted plotting them.

Figure 10: Complexity vs. $\left\lVert{\bf z}\right\rVert(\propto-\sqrt{\log
p({\bf z})})$ on the manipulated CIFAR-10. In-Dist is CIFAR-10. The NF models
are Glow, iResNet, CV-Glow, and IDF, from left to right and top to bottom.

###  E.2 Additional Results of Experiment 2 and 3

We present the results of Experiment 2 and 3 with other NF models in Fig. 11.

Figure 11: Volume vs. $\left\lVert{\bf z}\right\rVert(\propto-\sqrt{\log
p({\bf z})})$. In-Dist is CIFAR-10. Upper three are for the pooling noise
images with ResFlow, Glow, and iResNet. Lower two are for the manipulated
CIFAR-10 with Glow and iResNet.

###  E.3 Additional Results of Experiment 4 and 5

We present the results of Experiment 3 with other NF models in Fig. 12.

Figure 12: Complexity vs. $\left\lVert{\bf z}\right\rVert$. In-Dist is
CIFAR-10. Top is for iResNet and middle is for CV-Glow. $-\log p({\bf x})$ is
used instead of $\left\lVert{\bf z}\right\rVert$ for PixelCNN++ in the bottom.

#### AUROC and AUPR on MNIST and FMNIST

We show the detection performance of our methods on MNIST and FMNIST are
shown. Tables 3 and 4 show the AUROC and AUPR, respectively. The proposed
method successfully detects all cases; Glow performed best, followed by
ResFlow and iResNet. The iResNet is less effective in the case where In-Dist
is MNIST and OOD is FMNIST. While CALT fails in the two cases, TTL is
comparable to our method with Glow.

Table 3: AUROC (%)$\uparrow$. In-Dist datasets are MNIST (left) and FMNIST
(right). CALT fails in two cases (lower than $60\%$) on FMNIST.

[h]  FMNIST OMNIGLOT NotMNIST TTL 98.88 98.84 99.96 CALT 84.83 93.54 64.11
Ours (Glow) 98.82 99.97 99.98 Ours (ResFlow) 95.53 99.87 99.98 Ours (iResNet)
61.03 94.52 98.54 MNIST OMNIGLOT NotMNIST TTL 99.06 97.01 99.06 CALT 52.51
97.37 52.51 Ours (Glow) 98.55 99.55 98.55 Ours (ResFlow) 93.32 88.32 93.32
Ours (iResNet) 97.59 86.63 97.59

Table 4: AUPR (%)$\uparrow$. In-Dist datasets are MNIST (left) and FMNIST
(right).

[t]  FMNIST OMNIGLOT NotMNIST TTL 99.17 99.20 99.97 CALT 86.16 95.10 52.69
Ours (Glow) 98.68 99.97 99.98 Ours (ResFlow) 93.88 99.88 99.98 Ours (iResNet)
57.93 96.64 97.79 MNIST OMNIGLOT NotMNIST TTL 84.17 97.26 99.15 CALT 85.41
97.60 47.03 Ours (Glow) 89.54 99.64 98.60 Ours (ResFlow) 87.63 87.72 88.80
Ours (iResNet) 51.67 90.08 96.21

#### AUPR on CIFAR-10 and SVHN

We show the detection performance of our methods as the area under the
precision-recall curve (AUPR). Table 5 shows the AUPR on CIFAR-10 and SVHN.
Table 6 shows the AUPR for PixelCNN++ on CIFAR-10 and SVHN.

Other than our methods, LRG is the only method that scores higher than 60% in
the AUROC results shown in Tables 1 in all test cases. We consider this result
as follows: LRG uses the general likelihood $\log p_{g}({\bf x})$ computed by
the NF model trained on more ‘general’ data than the In-Dist data and computes
the ratio as $S_{\text{LRG}}=\log p({\bf x})-\log p_{g}({\bf x})$.666We
trained the NF model for $\log p_{g}({\bf x})$ using TinyImageNet. Because the
variation caused by $C({\bf x})$ that occurs in $\log p({\bf x})$ would have
occurred in $\log p_{g}({\bf x})$ as well, subtracting $\log p_{g}({\bf x})$
could cancel out the influence of $C({\bf x})$ indirectly, leading to the
success of LRG.

Table 5: AUPR (%)$\uparrow$. In-Dist datasets are CIFAR-10 (top) and SVHN
(bottom)

SVHN CelebA TIN Bed Living Tower Noise-1 Noise-2 Noise-4 Noise-8 Noise-16
Noise-32 TTL 47.03 87.58 86.06 91.02 91.89 90.23 100 53.83 40.19 85.98 95.41
99.97 CALT 92.06 59.64 39.21 55.43 42.28 57.11 98.77 100 100 100 100 100 LRB
46.78 72.81 42.16 43.78 42.69 38.81 30.81 30.69 33.40 33.31 32.97 32.41 LRG
62.08 95.37 66.44 71.78 67.17 69.74 100 78.26 99.29 99.31 97.73 96.72 WAIC
78.09 42.76 74.10 81.10 87.21 84.32 100 100 89.9 94.70 98.87 99.78 Ours (Glow)
92.17 83.35 86.10 86.34 88.20 87.09 100 99.58 100 100 100 100 Ours (ResFlow)
93.10 99.93 91.84 93.69 94.25 92.03 100 99.99 100 99.97 100 100 Ours (iResNet)
88.38 65.42 59.11 81.47 86.56 78.76 100 99.95 32.58 99.64 100 100

[t]  CIFAR10 CelebA TIN Bed Living Tower Noise-1 Noise-2 Noise-4 Noise-8
Noise-16 Noise-32 TTL 98.26 99.98 99.90 99.99 100 99.90 100 100 99.98 78.25
83.79 99.93 CALT 31.58 31.09 31.70 33.10 31.76 35.01 100 100 100 100 100 100
LRB 30.76 31.19 31.77 33.13 32.34 33.19 46.35 30.69 30.70 31.44 33.47 32.56
LRG 92.82 99.96 100 100 100 100 100 100 99.67 98.84 95.89 95.42 WAIC 99.58
99.44 99.83 99.96 99.99 99.56 100 100 100 98.82 96.95 98.61 Ours (Glow) 97.03
99.96 99.88 99.97 100 99.80 100 100 100 100 100 100 Ours (ResFlow) 57.80 66.50
78.01 65.91 83.07 51.01 100 94.77 100 100 100 100 Ours (iResNet) 86.56 98.79
98.56 99.66 99.88 98.25 100 100 99.97 70.85 98.96 99.10

Table 6: AUPR (%) $\uparrow$ for our proposed complexity-aware method and the
likelihood test with PixelCNN++ as a baseline. In-Dist datasets are CIFAR-10
(top) and SVHN (bottom).

[t]  SVHN CelebA TIN Bed Living Tower Noise-2 Noise-4 Noise-8 Noise-16
Noise-32 PixelCNN++ 85.30 35.58 35.13 33.27 31.97 35.81 30.69 58.34 99.39 100
100 Ours 96.04 83.57 71.79 80.09 80.79 82.46 100 100 100 100 100

[t]  CIFAR-10 CelebA TIN Bed Living Tower Noise-2 Noise-4 Noise-8 Noise-16
Noise-32 PixelCNN++ 30.88 30.69 30.77 30.70 30.69 30.76 30.69 30.69 77.85
99.27 95.55 Ours 89.55 97.29 96.57 98.20 99.29 97.17 100 100 100 100 100

#### AUROC and AUPR on ImageNet

We verified the validity of our claims even for large-sized images of
$224\times 224$. Table 7 show the AUROC and AUPR on ImageNet. Our proposed
approach using a GMM, which utilizes the two variables $\log p({\bf z})$ from
a trained Glow and the image complexity, shows an almost flawless ability to
detect artificially generated noise images from the pooling noise image
dataset.

Table 7: The results on ImageNet. Top is AUROC (%)$\uparrow$ and bottom is
AUPR (%)$\uparrow$. Failure cases (lower than $60\%$) are underlined in the
AUROC table.

[t]  Noise-1 Noise-2 Noise-4 Noise-8 Noise-16 Noise-32 Noise-64 Noise-128
Noise-224 TTL 100 100 57.45 99.53 61.78 89.42 99.69 99.99 99.93 CALT 100 100
74.39 100 100 100 100 100 100 Ours (Glow) 100 100 98.09 100 99.51 99.80 100
100 100

[t]  Noise-1 Noise-2 Noise-4 Noise-8 Noise-16 Noise-32 Noise-64 Noise-128
Noise-224 TTL 100 100 48.44 98.65 50.95 78.05 99.59 99.99 99.93 CALT 100 100
86.18 100 100 100 100 100 100 Ours (Glow) 100 100 99.03 100 99.75 99.89 100
100 100