Date of publication March 29, 2023, date of current version April 4, 2023. DOI
10.1109/ACCESS.2023.3263210

Corresponding author: Jinsol Lee (e-mail: jinsol.lee@gatech.edu).

# Probing the Purview of Neural Networks via Gradient Analysis

JINSOL LEE, , CHARLIE LEHMAN, MOHIT PRABHUSHANKAR  AND GHASSAN ALREGIB  Omni
Lab for Intelligent Visual Engineering and Science (OLIVES)
School of Electrical and Computer Engineering, Georgia Institute of
Technology, Atlanta, GA 30332, USA

###### Abstract

We analyze the data-dependent capacity of neural networks and assess anomalies
in inputs from the perspective of networks during inference. The notion of
data-dependent capacity allows for analyzing the knowledge base of a model
populated by learned features from training data. We define purview as the
additional capacity necessary to characterize inference samples that differ
from the training data. To probe the purview of a network, we utilize
gradients to measure the amount of change required for the model to
characterize the given inputs more accurately. To eliminate the dependency on
ground-truth labels in generating gradients, we introduce confounding labels
that are formulated by combining multiple categorical labels. We demonstrate
that our gradient-based approach can effectively differentiate inputs that
cannot be accurately represented with learned features. We utilize our
approach in applications of detecting anomalous inputs, including out-of-
distribution, adversarial, and corrupted samples. Our approach requires no
hyperparameter tuning or additional data processing and outperforms state-of-
the-art methods by up to 2.7%, 19.8%, and 35.6% of AUROC scores, respectively.

###### Index Terms:

Gradients, model capacity, robustness, out-of-distribution detection,
adversarial detection

=-15pt

  * Citation

J. Lee, C. Lehman, M. Prabhushankar and G. AlRegib, “Probing the Purview of
Neural Networks via Gradient Analysis,” in IEEE Access, vol. 11, pp.
32716-32732, 2023, doi: 10.1109/ACCESS.2023.3263210.

  * BibTeX

@ARTICLE{lee2023purview,
author={Lee, Jinsol and Lehman, Charlie and Prabhushankar, Mohit and AlRegib,
Ghassan},
journal={IEEE Access},
title={Probing the Purview of Neural Networks via Gradient Analysis},
year={2023},
volume={11},
pages={32716-32732},
publisher={IEEE},
doi={10.1109/ACCESS.2023.3263210}}

  * Copyright

©2023 Lee, Lehman, Prabhushankar and AlRegib. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (CC
BY). The use, distribution or reproduction in other forums is permitted,
provided the original author(s) and the copyright owner(s) are credited and
that the original publication in this journal is cited, in accordance with
accepted academic practice. No use, distribution or reproduction is permitted
which does not comply with these terms.

  * Contact

{jinsol.lee<EMAIL_ADDRESS>
https://ghassanalregib.info/

##  I Introduction

Deep neural networks are prone to failure when deployed in real-world
environments as they often encounter data that diverge from training
conditions [1, 2, 3]. Neural networks rely on the implicit assumption that any
given input during inference is drawn from the same distribution as the
training data. Limited to classes seen during training, neural networks
classify any input image among such in-distribution classes, even if the image
is significantly different from training data. In addition, it is widely
accepted that neural networks tend to make overconfident predictions even for
inputs that differ from training data [4, 5, 6, 7], making it more challenging
to distinguish inputs of anomalous conditions. This behavior can have serious
consequences when utilized in safety-critical applications, such as autonomous
vehicles and medical diagnostics [8, 9]. To ensure reliable performance for
practical applications of neural networks, models must be able to distinguish
inputs that differ from training data and cannot be handled adequately based
on their capacity.

The capacity of neural networks is broadly discussed in terms of the size of
the networks (_i.e._ , the number of model parameters) [10, 11, 12]. It is
central to the generalization performance of neural networks in traditional
statistical learning theories as models with larger capacity are expected to
overfit training data, leading to poor generalization performance [13, 12].
However, recent studies show that over-parameterization of deep neural
networks only helps with their generalization performance [14, 15]. Many
researchers aim to understand this phenomenon by examining the
representational capacity, which is the types of functions a model can learn
[16, 17]. Others discuss it from the optimization point of view [12]. They
recognize that a learning algorithm, defined by model architecture and
training procedure, is unlikely to find the best function among all possible
functions, although it can still learn one that performs well for a given
task. This notion of capacity, defined as the types of functions that can be
reached via some learning algorithm, is denoted as the effective capacity [14,
15]. These perspectives of data-independent network capacity analyze the
generalization behavior of neural networks by changing network architectures
or forcing memorization. They assume that training data is available to be
manipulated for memorization and repetitive training of networks with altered
architectures. However, this assumption does not hold for deployed models in
which the training data and network architectures are fixed and novel
situations cannot be foreseen or simulated during training. Hence, they are
not suitable for analyzing the capacity of deployed models and their ability
to handle samples given during inference.

In this work, we examine the capacity of trained neural networks in terms of
their knowledge base and reformulate the definition of the model capacity. The
knowledge base of a network is established by training data, allowing the
model to characterize the inputs observed during inference with its learned
features. We argue that the capacity of models should be investigated not only
in terms of the learning algorithm (_i.e._ , model architectures and training
procedures) but also in terms of the training data. To analyze the data-
dependent capacity of the trained networks, we introduce the concept of
purview from top-down and bottom-up perspectives, as illustrated in Fig. 1.
The top-down purview, derived from the model perspective, is built on the
inclusive relationship between representational capacity ($RC$) and effective
capacity ($EC$), where the latter is a subset of the former. The purview lies
in the region described by $RC-EC$ and shown in blue, which denotes the
additional capacity in $RC$ that can be utilized to enhance $EC$. The bottom-
up purview, established from the data perspective, is based on the learned
features $\mathcal{T}$ that populate the knowledge base of a trained model,
_i.e._ , the data-dependent $EC$ and its features $\mathcal{T}_{EC}$. It is
the gap between the learned features $\mathcal{T}^{*}_{EC}$ established on the
training data and the ideal $EC$ with additional features in
$\mathcal{T}^{*}_{purview}$ that is necessary to handle inference samples in
addition to the training data.

Figure 1: Definition of purview, shown in the region highlighted in blue. A
top-down purview is established on the inclusive relationship between $RC$ and
$EC$. A bottom-up purview depends on learned features.

To probe the purview of the trained networks, we utilize gradients to
determine anomalous inputs from the perspective of the models. Our intuition
is that gradients correspond to the amount of change that a model requires to
properly represent a given sample. We hypothesize that the required change
captured in gradients would be more significant for inputs that cannot be
represented accurately with the learned features of a network. From the data-
dependent view of model capacity, a model trained with simpler data would have
a less extensive feature set, and thus, lower $EC$. In turn, there is more
room for improvement in its feature set (_i.e._ , larger $RC-EC$ region,
larger purview), leading to more significant gradient responses. During
training, gradients are generated with respect to the model outputs and
ground-truth labels for given inputs, the latter of which are unavailable
during inference. To remove dependency on information regarding given samples
during inference, we introduce confounding labels, which are labels formulated
by combining multiple categorical labels. Although the use of manipulated
labels has been explored in existing studies [18, 19, 20, 21, 22], it is only
analyzed during model training, and the label designs rely on the statistics
of the training data. In comparison, a confounding label does not require any
knowledge of the training data and can be used during inference with no access
to ground-truth labels. The contributions of this paper are five-fold:

  * •

We define purview by integrating the top-down view from the representational
capacity to the effective capacity and the bottom-up view based on the learned
features of the trained models.

  * •

We introduce the concept of confounding labels as an unsupervised tool to
elicit a model response that can be utilized to probe the purview of trained
neural networks.

  * •

We demonstrate the relationship between the degree of exposure to diverse data
in model training and the purview of trained networks via gradient analysis.

  * •

We utilize our proposed method in applications of OOD detection, adversarial
detection, and corrupted input detection and achieve state-of-the-art
performance.

  * •

We conduct extensive ablation studies to study the manipulation of supervised
and unsupervised confounding labels.

##  II Related Work

###  II-A Capacity of Neural Networks

Following the distinction between representational capacity and effective
capacity established in Section I, we introduce how they have been studied in
the literature. VC (Vapnik-Chervonenkis) dimension [13] is a traditional
approach for quantifying the representational capacity of a learning machine
with cardinality of data samples to provide the theoretical bounds. More
recent studies have examined the representational capacity of deep neural
networks from the viewpoint of generalization, which diverges from the
traditional understanding of the overfitting nature of larger models. To do
so, some proposed to utilize sample complexity to understand standard
generalization [14, 23] and adversarial generalization [24]. Others employed
the intrinsic dimensions of networks [25, 26]. Zhang  _et al._ [15] proposed
the concept of the effective capacity to analyze the memorization behavior of
deep neural networks, extended by Arpit  _et al._ [27]. They demonstrated the
memorization behavior by searching for the network of the smallest capacity
required to learn perturbed images and labels. These studies of
representational capacity and effective capacity analyze networks during
training by altering the model architectures. Their interpretation of the
model capacity is data-independent. They focus on the varying sizes of the
models and the functions they are capable of learning, rather than on the
effect of the training data, in addition to the network architectures and
training algorithms. We argue that training data is critical in defining the
effective capacity of networks. This data-dependent view of the effective
capacity of trained models allows for analyzing their ability to handle
different inputs during inference and determining the validity of model
predictions. We aim to demonstrate the connection between representational
capacity and effective capacity from a data-dependent perspective and the
proposed concept of purview.

###  II-B Anomalous Input Detection

Many studies have aimed to detect anomalous samples due to statistical shifts
in the data distribution or adversarial generation. This application exploits
the insufficient knowledge base of models trained on relatively small datasets
to distinguish inputs that differ from the training samples. There are mainly
two types of approaches: analyzing the output distributions of classifiers to
differentiate anomalous inputs or employing auxiliary networks for detection.
Hendrycks and Gimpel [28] introduced a baseline method of thresholding samples
based on predicted softmax distributions. Liang  _et al._ [29] employed
additional input and output processing to the previous method to further
improve detection performance. DeVries and Taylor [30] utilized prediction
confidence scores obtained from an augmented confidence estimation branch on a
pretrained classifier. Ma  _et al._ [31] proposed to characterize the
dimensional properties of adversarial regions with local intrinsic
dimensionality. Lee  _et al._ [32] proposed a confidence metric using
Mahalanobis distance. Liu  _et al._ [33] utilized energy scores to capture the
likelihood of occurrence during inference and training time. These approaches
utilize activation-based representations to characterize the anomaly in inputs
via learned features that establish the effective capacity of a model, _i.e._
, what the network knows about the given inputs. However, the overconfident
nature of neural networks makes it counterintuitive to characterize anomalous
inputs solely with activations that are known to be poorly calibrated [5].
Rather, we argue that the anomaly in inputs should be established based on
what the model is unfamiliar with and thus incapable of representing
accurately. Instead of focusing solely on the learned features based on the
effective capacity of the networks, we focus on the additional features
necessary to represent the anomaly in the regions of purview.

###  II-C Gradients as Features

At the core of the advancement of deep neural networks lies gradient-based
optimization techniques that allow for finding solutions to tasks at hand
[34]. In addition to its utility as an optimization tool, gradients have been
utilized for various purposes. Goodfellow  _et al._ [4] first demonstrated
that small, hardly perceptible perturbations, known as adversarial attacks,
can deliberately fool trained networks to make irrelevant predictions with
high confidence, extended by numerous studies including [35, 36]. These
gradient-based adversarial attacks force models to deviate from their data-
dependent $EC$ and yield perturbed outputs. Gradients are also commonly used
in visual explanation techniques to produce localization maps of pixels
relevant to model predictions [37, 38], which accentuate the existing features
that define the effective capacity. Others expanded their work with
contrastive explanations, in which they visualized features of other potential
predictions for actual predictions [39, 40]. Another application using
gradients is anomaly detection by constraining gradients in autoencoder
learning such that inliers form a specific knowledge base that can be used to
differentiate outliers [41]. Some have utilized gradients to quantify the
uncertainty of trained neural networks [42, 6, 43]. In general, gradients are
used to illuminate the learned features of networks and examine the divergence
from the established knowledge base. Our focus of purview exploits this nature
of gradients in terms of model capacity, bridging the gap between effective
capacity and representational capacity. We show that gradients enable the
analysis of data samples during inference from the perspective of models by
considering the adequacy of the effective capacity established on training
data and the additional capacity (_i.e._ , purview), stepping further into the
representational capacity needed for more accurate representation.

###  II-D Manipulating Label Encodings

One-hot encoding is the most widely used approach for formulating labels to
handle nominal data (_i.e._ , categorical data with no quantitative
relationship between categories). A combination of multiple one-hot encodings
is used to indicate information regarding multiple classes in each image.
Based on these two, recent studies have analyzed different formulations of
labels for various purposes. Some work [18, 19, 20] have proposed the mixing
of two input images and their labels with a pre-computed ratio as a data
augmentation technique. For the interpretability of neural networks,
Prabhushankar and AlRegib [44] proposed to extract causal visual features
using combinations of binary classification labels. To address the problem of
missing labels in a multi-label classification setting, Durand  _et al._ [21]
utilized partial labels formulated using the proportions of known labels.
Duarte  _et al._ [22] proposed a similar approach to handle imbalanced
datasets by masking parts of the labels based on the ratio of positive samples
to negative samples. These methods impose constraints on feature learning to
improve the generalizability and robustness. However, they explored different
label encodings only for model training and assumed the availability of
information regarding the training data. We propose manipulating label
encodings to probe the purview of networks built upon their knowledge base of
learned features. Contrary to existing studies, our approach does not depend
on the availability of training data statistics or label information for the
inference data.

##  III Probing the Purview of Neural Networks

In this section, we expand the definition of the purview of neural networks
based on data-dependent effective capacity. We discuss the purview of neural
networks in terms of the knowledge base established by the training data and
the additional knowledge necessary to handle samples during inference. We then
introduce our gradient-based approach to probe the purview of the trained
models.

###  III-A Feature-Based Capacity Analysis

We define purview as the additional capacity required for a model to handle
samples observed during inference that differ from the training data. Our
definition of purview depends on the effective capacity established by the
training data, as well as network architectures and training procedures.
Assuming a fixed architecture and training procedure, we first discuss the
effective capacity of a trained model in terms of the features learned from
the training data. Following the notations utilized by AlRegib and
Prabhushankar [39], we introduce our setup for feature-based effective
capacity analysis. Let $f(\cdot)$ be a neural network trained for an image
classification task,

$f:\mathbb{R}^{h\times w\times c}\rightarrow\mathbb{R}^{N},$ (1)

where it maps an input of dimension $h\times w\times c$ to an output vector of
dimension $N$, which is the number of classes defined in the training dataset.
Given an input image $x$, the model $f$ utilizes its learned features to
produce a model output (_i.e._ , logits)

$y=f(x).$ (2)

The predicted class $C$ is determined by taking the index of the largest logit
value,

$C=\operatorname*{arg\,max}~{}y,~{}~{}C\in\\{1,2,\cdots,N\\}.$ (3)

Let $\mathcal{T}$ be the set of all features that the network learned to
extract during training for the classification task,

$\mathcal{T}=\left\\{\mathcal{T}_{1},\mathcal{T}_{2},\cdots,\mathcal{T}_{P}\right\\},$
(4)

where $P$ denotes the total number of learned features. Each feature captures
a unique characteristic of the training data. Depending on the classes
represented in the training data and their similarities, some classes have
unique features whereas others have shared features for their similar
characteristics. The model prediction of $C$ for image $x$ means that the
network’s decision is based on the set of features relevant to class $C$,
which is a subset of all available features $\mathcal{T}$.

(a) Model $f$ approximating for $x$

(b) Model $g$ using an appropriate feature for $x$

Figure 2: Illustrations of the feature-based effective capacity of trained
neural networks established by their training data, given that the networks
share the same architecture and training procedures.

We describe our interpretation of data-dependent effective capacity using the
toy example in Fig. 2. Consider two networks, $f$ and $g$, which share the
same architecture and training procedures, but have different training data.
We assume that the network architecture has a sufficient number of model
parameters to learn their training data and that the training procedures allow
for optimization. The first network $f$ is trained on handwritten digit images
of $0,6$, shown in Fig. 2(a). Based on the training data, it learns the
features of curves shared among all digits ($\mathcal{T}_{1}$) and 3-way
intersections for the digit $6$ ($\mathcal{T}_{2}$), forming the feature set
$\mathcal{T}^{f}$,

$\mathcal{T}^{f}=\\{\mathcal{T}_{1},\mathcal{T}_{2}\\}.$ (5)

On the other hand, the second network $g$ is trained on the images of the same
handwritten digits as $f$ and an additional digit of $7$. Some of the learned
features of $g$ are similar to those of $f$, and are written as
$\mathcal{T}^{\prime}_{1}$ and $\mathcal{T}^{\prime}_{2}$. Moreover, the
additional training data of digit $7$ leads to extra features of straight
lines ($\mathcal{T}_{3}$) and corners ($\mathcal{T}_{4}$), constructing the
overall feature set $\mathcal{T}^{g}$,

$\mathcal{T}^{g}=\\{\mathcal{T}_{1}^{\prime},\mathcal{T}_{2}^{\prime},\mathcal{T}_{3},\mathcal{T}_{4}\\}.$
(6)

Model $g$ has a higher effective capacity with a more comprehensive feature
set due to its exposure to relatively diverse training data compared to $f$,
which is described with more ridges in Fig. 2(b). Given these two models, we
now consider an inference scenario in which an input image $x$ of the
handwritten digit $1$ is presented. For model $f$, the feature span
$\mathcal{T}^{f}$ is insufficient to accurately represent $x$ due to the lack
of the features for the straight lines. The best the model $f$ can do to
capture the characteristics of the sample $x$ is to utilize the existing
features for an approximation,

$\mathcal{T}^{f}_{x}=h(\mathcal{T}_{1},\mathcal{T}_{2}),$ (7)

where $h$ is a function that combines the learned features. For model $g$,
however, the same image of digit $1$ can be handled more properly because the
feature set $\mathcal{T}^{g}$ includes the straight-line feature, learned from
digit $7$ during training. The model $g$ can utilize this learned feature to
represent the inference sample,

$\mathcal{T}^{g}_{x}=\mathcal{T}_{3}.$ (8)

In contrast to $\mathcal{T}^{g}_{x}$, $\mathcal{T}^{f}_{x}$ is still
inadequate for representing the sample because each learned feature captures a
unique characteristic, and the combination of existing features cannot
precisely account for the lack of relevant feature. While the predictions from
both models are irrelevant because the class of $1$ does not exist in the
training data, this feature-based focus enables the analysis of the anomaly in
inputs from the perspective of models that is not limited to the predicted
class distributions.

The comparison between these two models highlights the core of our approach,
which focuses on the absence of relevant features in the model’s knowledge
base to distinguish inputs that cannot be represented accurately, _i.e._ ,
anomalous samples. Although the knowledge base of a model can be broadened via
training with comprehensive datasets and various data augmentation techniques,
it is still impractical for a model to learn everything in existence.
Consequently, in practice, we only have access to models that lack some
features (_i.e._ , $f$). Writing in general terms,

$\displaystyle\mathcal{T}^{f}$
$\displaystyle=\\{\mathcal{T}_{1},\cdots,\mathcal{T}_{P}\\},$ (9)
$\displaystyle\mathcal{T}^{g}$
$\displaystyle=\\{\mathcal{T}_{P+1},\cdots,\mathcal{T}_{P+Q}\\}$ (10)

Ideally, we want the approximation $\mathcal{T}^{f}_{x}$ to be equivalent to
the lacking feature,

$h(\mathcal{T}_{1},\cdots,\mathcal{T}_{P})=\mathcal{T}_{P+1},$ (11)

where $\mathcal{T}_{P+1}$ is an arbitrary feature that is relevant to input
$x$ and exists only in $\mathcal{T}^{g}$. This would allow for the
representation of the sample considered anomalous from the perspective of the
models with its learned features. With the purview, we examine the gap in the
knowledge base of $f$ that needs to be offset to fully grasp the unfamiliar
characteristics of $x$.

###  III-B Purview of Networks and Gradients

Building on the effective capacity analysis setup, we discuss the purview of
the models. We define purview as the additional feature-based capacity
necessary for a trained network to characterize inputs given during inference
that differ from the training data. It is centered on the absence of features
in the network’s feature span that are relevant for the given inputs. Consider
the models $f$ and $g$ and their features used in response to input $x$ during
inference $\mathcal{T}^{f}_{x}$ and $\mathcal{T}^{g}_{x}$. With the purview,
we examine the gap in the knowledge base of $f$ that needs to be offset such
that $f$ can represent $x$ more accurately. This gap is effectively the
necessary change to be made to $f$ to bring the best approximation
$h(\mathcal{T}_{1},\cdots,\mathcal{T}_{P})$ closer to the necessary feature
$\mathcal{T}_{P+1}$.

To assess the necessary change for the model such that
$h(\mathcal{T}_{1},\cdots,\mathcal{T}_{P})=\mathcal{T}_{P+1}$, we make
assumptions regarding the learned features. Each learned feature captures a
unique aspect of training data, so the features from each model are orthogonal
to each other,

$\displaystyle\mathcal{T}_{1}\perp\mathcal{T}_{2}\perp$
$\displaystyle\cdots\perp\mathcal{T}_{P},$ (12)
$\displaystyle\mathcal{T}_{P+1}\perp\mathcal{T}_{P+2}\perp$
$\displaystyle\cdots\perp\mathcal{T}_{P+Q}.$ (13)

We also assume that the function $h$ for approximation is linear. Based on the
orthogonality of features in Eq. 13 and the relationship between
$\mathcal{T}^{f}_{x}$ and $\mathcal{T}^{g}_{x}$, we write

$h(\mathcal{T}_{1},\cdots,\mathcal{T}_{P})\perp\mathcal{T}_{P+2}\perp\cdots\perp\mathcal{T}_{P+Q}$
(14)

This relationship also assures that the approximated feature is orthogonal to
the $\operatorname{span}$ of the rest of the features,

$\operatorname{span}(\mathcal{T}_{P+2},\cdots,\mathcal{T}_{P+Q})\perp
h(\mathcal{T}_{1},\cdots,\mathcal{T}_{P})$ (15)

To measure the necessary change, we employ gradients based on their utility in
model optimization, where they correspond to the amount of change that a model
requires to properly represent inputs. Gradients have a unique property of
orthogonality, which ensures that the gradients of the approximated feature
would lie in the $\operatorname{span}$ of the other features,

$\operatorname{span}(\mathcal{T}_{P+2},\cdots,\mathcal{T}_{P+Q})\supset\nabla
h(\mathcal{T}_{1},\cdots,\mathcal{T}_{P}).$ (16)

With the linear assumption on $h$, the gradients of a linear combination of
orthogonal features can be re-written as

$\operatorname{span}(\mathcal{T}_{P+2},\cdots,\mathcal{T}_{P+Q})\supset\nabla\mathcal{T}_{1},\cdots,\nabla\mathcal{T}_{P}.$
(17)

This shows that the gap in the knowledge base for the absent feature can be
examined by approximating $\mathcal{T}_{P+2},\cdots,\mathcal{T}_{P+Q}$ with
the gradients of the existing features $\mathcal{T}^{f}$. We hypothesize that
the amount of necessary change for a model to represent inputs seen during
inference would be more significant for inputs that differ greatly from the
training data. Equivalently, the amount of necessary change is inversely
related to the effective capacity of the model established by the training
data. Networks that are exposed to diverse training data would have enhanced
effective capacity with more generalizable features, leading to a smaller
amount of required change to handle inference samples. We propose to exploit
this relationship and probe the purview using gradients. Similar to the use of
gradients during model training, gradients can be generated during the
inference of a trained model. The obtained gradients are not applied for the
actual model updates. Because the model in question is a converged solution,
the required model update would not be drastic for inputs within its knowledge
base. These gradients can be used to probe the purview of a trained network.

###  III-C Confounding Labels for Gradients

Consider an image classifier during inference. The model will make a
prediction for any given input, but the validity of model predictions remains
in question due to the lack of access to labels or information on input data
distribution. This presents a challenge in utilizing gradients to probe the
purview of a trained network. We introduce confounding labels to remove the
dependency on ground-truth labels in gradient generation during inference.
Confounding label $y_{c}$ is a label formulated by combining multiple
categorical labels. In an image classification setting, an ordinary label
consists of a single class (_i.e._ , one-hot encodings) for a model trained to
minimize cross-entropy loss, whereas a confounding label may include multiple
classes or none.

$\displaystyle y_{c}~{}\in~{}\big{\\{}y:$
$\displaystyle~{}~{}y~{}\in~{}\\{0,1\\}^{N},$
$\displaystyle~{}~{}\textstyle\sum
y_{i}\in\\{0,\cdots,N\\}\setminus\\{1\\}\big{\\}}$ (18)

Our approach focuses on the absence of relevant features in the knowledge base
of a trained model, examined via gradients, to fully grasp the anomaly in the
given inputs during inference. A confounding label provides an unsupervised
methodology to elicit a gradient response that can be utilized to probe the
purview of trained neural networks.

Figure 3: Generating gradient-based representations with confounding labels.

We discuss the framework for collecting gradient-based representations with
confounding labels during inference in Fig. 3. We utilize the binary cross-
entropy loss between the logits and a confounding label, where $\hat{y_{i}}$
is the predicted probability for class $i$ and $y_{c,i}$ is the true
probability represented by the confounding label.

$\small
J(\theta)=-\frac{1}{N}\sum^{N-1}_{i=0}\left(y_{c,i}\cdot\log\left(\hat{y_{i}}\right)+\left(1-y_{c,i}\right)\cdot\log\left(1-\hat{y_{i}}\right)\right)$
(19)

With backpropagation of the loss, gradients are generated at each set of model
parameters (_i.e._ , the weight and bias parameters of the network layers).
While any form of gradient that preserves its magnitude would be valid, we
measure the squared $L_{2}$-norm for each parameter set and concatenate them
to represent the given input. The obtained gradient-based representation has
the following form:

$\begin{gathered}\big{[}~{}\|\nabla
J_{\theta_{0}}(\theta;x,y)\|^{2}_{2}~{},~{}\cdots~{},~{}\|\nabla
J_{\theta_{L-1}}(\theta;x,y)\|^{2}_{2}~{}\big{]},\end{gathered}$ (20)

where $L$ is the number of layers or parameter sets in the given network. We
highlight that the gradient generation process involves no hyperparameters
compared with other approaches that distinguish between in-distribution and
OOD.

(a) Gradients

(b) Activations

(c) Loss

Figure 4: Comparison between the magnitudes of layer-wise gradients
$\nabla_{\ell}J$, layer-wise activations $a_{\ell}$, and loss in
distinguishing in-distribution and OOD datasets. Each plot shows distributions
of per-class average values of considered attributes. The layer-wise values
are obtained from a convolutional layer in each residual block of the
ResNet-18 model.

####  III-C1 Gradients vs. Activations vs. Loss

We demonstrate the effectiveness of gradients obtained with confounding labels
compared to activations and loss values in differentiating anomalous inputs.
In an out-of-distribution detection setup, we show the disparity between the
features learned during training and the features necessary to represent test
images of both in-distribution and OOD. The key idea is that the model will
require more significant updates in its feature set to handle OOD samples than
to handle in-distribution samples. Based on our hypothesis, the gradient
magnitudes obtained from the OOD samples should be larger than those of the
in-distribution samples. As discussed in Section II, most OOD detection
approaches use activation-based measures. We will show that gradients can
capture distributional shifts more effectively than activations and loss
values.

For the demonstration, a ResNet classifier [45] is trained with MNIST [46] and
used to generate gradients with confounding labels on the test sets of MNIST,
SVHN [47], CIFAR-10/100 [48], LSUN [49], and ImageNet [50], where the last two
are resized subsets of their original versions provided by Liang  _et al._
[29]. We employ a network architecture that is sufficiently large for all
datasets. We collect the squared $L_{2}$-norm of the layer-wise gradients and
activations, as well as the loss values, and visualize their distributions in
Fig. 4(a). For gradient generation, we utilize a confounding label that
combines one-hot encodings of all classes (_i.e._ , all-hot encoding). For
visualization, we select a convolutional layer from each residual block of the
ResNet architecture. The distributions of the gradient and activation
magnitudes and loss values for the in-distribution dataset are highlighted by
red circles in each plot for clarity. The separation in the ranges of gradient
magnitudes between the in-distribution and OOD datasets is more evident in
some parts of the network because each layer captures information about
different aspects of the given inputs. Nevertheless, we observe a sharp
distinction based on the purview of the model, with smaller gradient
magnitudes for the in-distribution datasets and significantly larger
magnitudes for the OOD datasets throughout the network layers. Given that the
in-distribution dataset is the simplest of all considered datasets, the
learned features are insufficient to characterize the OOD samples, leading to
larger purview and thus larger gradient magnitudes. In contrast, the
activations in Fig. 4(b) show apparent overlaps in the magnitude ranges
between the in-distribution and OOD datasets throughout the network. This
supports our intuition that gradients can capture the anomaly (_i.e._ ,
distributional shift in the case of OOD) in inputs better than activations
based on the unfamiliar aspects of the given inputs from the model’s
perspective. Similar to the activations, we observe the apparent overlap of
the loss values across different datasets, as shown in Fig. 4(c). Loss and
gradients are intertwined in the process of backpropagation, but the loss is
determined based on the last layer of activation from the network and is
limited to a single value per sample. On the contrary, the gradients have the
same dimensions as their corresponding parameter sets, preserving more
information about the current state of the model and the necessary adjustment
for better representation of the given inputs. The overlap in the loss value
range between the in-distribution and OOD datasets shows that the loss alone
is inadequate for differentiating anomalous inputs from familiar inputs from
the model’s perspective. Overall, gradients are more effective in
characterizing anomalies in inputs than activations or loss values.

####  III-C2 Data-Dependent Capacity and Purview

(a) In-distribution dataset: MNIST

(b) In-distribution dataset: CIFAR-10

(c) In-distribution dataset: ImageNet

Figure 5: Comparison of gradient magnitudes obtained from models trained with
different datasets. In-distribution datasets are highlighted with red dotted
circles. The model trained on the simplest dataset (_i.e._ , MNIST) shows the
largest required model updates to represent OOD samples accurately, and the
distinctions in the gradient magnitudes between in-distribution datasets and
OOD datasets decrease as the complexity of in-distribution dataset increases.

We demonstrate the effectiveness of our gradient-based approach in probing the
purview of neural networks trained with data of varying levels of complexity.
In particular, we analyze the asymmetry in the data-dependent effective
capacity and the purview observed in the gradients. The core of this analysis
is to show that a model trained on simpler data would have lower effective
capacity and more room for improvement in its feature set (_i.e._ , purview),
leading to larger gradient responses observed throughout the network. The
experiment follows the setup of the OOD detection experiment introduced in the
previous section, with the exception of the in-distribution dataset for each
model. We utilize training datasets of various complexities: MNIST, CIFAR-10,
and ImageNet in ascending order of complexity in terms of the represented
types and numbers of class categories. The magnitudes of the gradients
obtained with the confounding labels are shown in Fig. 5 to compare the models
with different degrees of effective capacity. Each row represents the gradient
magnitudes obtained from a model trained with the specified dataset, and each
column corresponds to the gradient magnitudes from a convolutional layer of
the model. As shown in Fig. 5(a), the model trained on MNIST (_i.e._ , the
dataset with the lowest complexity) exhibits the largest gap in gradient
magnitudes between the in-distribution and OOD datasets. It captures larger
model purview in response to the OOD samples, which is attributed to the
limited data-dependent effective capacity. In contrast, the increase in the
complexity of the in-distribution datasets leads to smaller gaps in the
gradient magnitudes between the in-distribution and OOD, as seen in Fig. 5(b)
and 5(c). This indicates that by exposing models to datasets of higher
complexity in training and enhancing the data-dependent effective capacity,
the learned feature sets are sufficient to capture the characteristics of the
OOD samples. However, models with less extensive feature sets would require
more significant amounts of model updates to accurately represent the OOD
samples. This highlights the value of our data-dependent perspective on model
capacity, which allows for learned feature-based analysis.

##  IV Experiments

TABLE I: Out-of-distribution detection results. All values are in percentages and the best results are highlighted in bold. Dataset Distribution |  Baseline [28] / ODIN [29] / Mahalanobis [32] / Energy [33] / GradNorm [43] / Purview
---|---
In | Out | Detection Accuracy | AUROC | AUPR
SVHN | CIFAR-10 |  86.61 / 86.25 / 88.84 / - / - / 97.55 |  91.90 / 91.76 / 95.05 / 98.85 / 97.99 / 99.74 |  92.12 / 92.38 / 90.25 / 99.57 / 96.62 / 97.76
(ResNet) | ImageNet |  90.22 / 90.31 / 96.17 / - / - / 97.71 |  94.78 / 95.08 / 99.23 / 98.18 / 98.07 / 99.72 |  94.76 / 95.53 / 98.17 / 99.30 / 96.90 / 97.65
| LSUN |  89.14 / 89.13 / 97.50 / - / - / 98.82 |  94.12 / 94.52 / 99.54 / 97.72 / 98.10 / 99.91 |  94.32 / 95.20 / 98.84 / 99.09 / 96.80 / 99.16
| CIFAR-100 |  79.41 / 79.58 / 89.69 / - / - / 96.49 |  81.28 / 81.32 / 95.45 / 97.61 / 97.30 / 99.39 |  80.86 / 80.84 / 85.04 / 99.06 / 95.85 / 95.70
| ImageNet(F) |  80.62 / 80.72 / 89.55 / - / - / 96.93 |  82.44 / 82.46 / 95.79 / 98.12 / 98.01 / 99.56 |  81.59 / 81.56 / 86.64 / 99.28 / 96.88 / 96.52
| LSUN(F) |  80.04 / 80.19 / 90.46 / - / - / 97.41 |  81.52 / 81.61 / 96.14 / 97.44 / 97.80 / 99.71 |  80.93 / 80.94 / 85.39 / 98.96 / 96.48 / 97.55
CIFAR-10 | SVHN |  87.71 / 86.80 / 91.95 / - / - / 98.44 |  93.51 / 94.51 / 97.10 / 95.72 / 59.43 / 99.90 |  94.60 / 95.20 / 96.12 / 97.26 / 56.56 / 99.99
(ResNet) | ImageNet |  84.93 / 91.16 / 97.45 / - / - / 90.97 |  91.50 / 96.93 / 99.68 / 84.95 / 60.69 / 96.88 |  91.02 / 96.85 / 99.60 / 84.60 / 60.18 / 97.20
| LSUN |  85.58 / 91.91 / 98.60 / - / - / 98.59 |  92.09 / 97.51 / 99.86 / 92.39 / 70.38 / 99.90 |  91.71 / 97.51 / 99.82 / 95.40 / 69.54 / 99.91
| CIFAR-100 |  81.94 / 80.53 / 82.28 / - / - / 78.50 |  87.71 / 85.88 / 89.52 / 83.28 / 58.94 / 85.78 |  86.50 / 84.58 / 89.22 / 88.02 / 58.79 / 83.28
| ImageNet(F) |  84.11 / 82.61 / 85.63 / - / - / 81.58 |  89.60 / 88.30 / 92.59 / 86.41 / 62.22 / 89.02 |  88.27 / 86.86 / 92.44 / 90.47 / 61.57 / 86.76
| LSUN(F) |  84.47 / 83.37 / 84.75 / - / - / 84.28 |  89.92 / 89.00 / 91.70 / 87.94 / 64.12 / 92.23 |  88.79 / 87.78 / 91.18 / 91.11 / 63.88 / 91.00
SVHN | CIFAR-10 |  86.61 / 86.25 / 96.50 / - / - / 98.06 |  91.90 / 91.76 / 99.02 / 90.63 / 37.76 / 99.79 |  92.12 / 92.38 / 94.40 / 94.30 / 46.65 / 98.16
(DenseNet) | ImageNet |  90.22 / 90.31 / 98.89 / - / - / 98.01 |  94.78 / 95.08 / 99.84 / 87.51 / 40.28 / 99.77 |  94.76 / 95.53 / 99.37 / 92.15 / 48.76 / 98.18
| LSUN |  89.14 / 89.13 / 99.49 / - / - / 99.46 |  94.12 / 94.52 / 99.92 / 83.45 / 38.85 / 99.95 |  94.32 / 95.20 / 99.54 / 89.33 / 47.05 / 99.61
| CIFAR-100 |  86.59 / 85.85 / 96.49 / - / - / 97.12 |  91.42 / 90.95 / 99.14 / 86.09 / 37.66 / 99.63 |  91.60 / 91.73 / 95.52 / 90.94 / 47.09 / 97.09
| ImageNet(F) |  87.28 / 86.50 / 96.26 / - / - / 97.35 |  92.18 / 91.68 / 99.15 / 87.06 / 39.04 / 99.69 |  92.33 / 92.42 / 95.77 / 91.58 / 48.08 / 97.64
| LSUN(F) |  86.63 / 86.40 / 96.07 / - / - / 98.57 |  91.69 / 91.69 / 99.04 / 86.01 / 42.45 / 99.83 |  91.93 / 92.34 / 94.63 / 91.34 / 49.79 / 98.79
CIFAR-10 | SVHN |  87.71 / 86.80 / 95.75 / - / - / 98.89 |  93.51 / 94.51 / 98.96 / 89.33 / 88.75 / 99.95 |  94.60 / 95.20 / 97.21 / 94.01 / 83.94 / 99.99
(DenseNet) | ImageNet |  84.93 / 91.16 / 96.83 / - / - / 89.47 |  91.50 / 96.93 / 99.45 / 86.17 / 74.25 / 95.99 |  91.02 / 96.85 / 99.29 / 87.20 / 72.69 / 95.92
| LSUN |  85.58 / 91.91 / 98.08 / - / - / 99.24 |  92.09 / 97.51 / 99.74 / 92.77 / 84.15 / 99.96 |  91.71 / 97.51 / 99.72 / 95.57 / 83.71 / 99.96
| CIFAR-100 |  81.12 / 81.32 / 65.75 / - / - / 74.05 |  87.92 / 88.89 / 71.90 / 83.32 / 71.91 / 82.12 |  87.08 / 88.36 / 71.61 / 89.17 / 70.39 / 80.57
| ImageNet(F) |  82.22 / 83.35 / 69.68 / - / - / 81.84 |  88.89 / 90.51 / 77.35 / 87.06 / 72.72 / 89.95 |  88.16 / 90.02 / 77.00 / 92.00 / 71.01 / 88.65
| LSUN(F) |  81.75 / 81.89 / 75.03 / - / - / 90.28 |  88.42 / 89.57 / 82.05 / 86.77 / 68.12 / 96.39 |  87.64 / 89.19 / 81.21 / 92.01 / 66.49 / 96.14

In this section, we apply our gradient-based approach to probe the purview of
neural networks to detect anomalous inputs: OOD detection, adversarial
detection, and corrupted input detection. We distinguish the anomaly based on
the application and training datasets. For OOD detection, the in-distribution
datasets are referred to as normal, and the OOD datasets are anomalous. For
adversarial and corrupted input detection, clean images are considered normal,
and adversarial attack images or corrupted images are considered anomalous.
From a trained classifier, we obtain gradient-based representations from the
test sets of both normal and anomalous datasets using a confounding label
where all classes are positive (_i.e._ , all-hot encoding). For each
experiment, the collected gradient representations are used to train a simple
binary detector with two fully-connected layers as an anomalous input
detector. Each reported detection result is an average of 5-fold cross-
validation results repeated with two different random seeds, totaling ten
rounds with randomly initialized detectors for all detection experiments.
Further details regarding the implementation are provided in Section IV-D. In
addition, we provide an ablation study on detector network architectures and
training, as well as the designs of confounding labels in Section IV-E. We
note that our approach is not specifically devised to perform well for these
applications. Rather, it is a byproduct of the focus on understanding the
purview of models and the apparent gap in the data-dependent capacity observed
in the distribution of the gradient responses and has proven useful in such
applications.

###  IV-A Out-of-Distribution Detection

We apply our gradient-based approach on OOD detection using various image
classification datasets. We utilize CIFAR-10 and SVHN datasets as in-
distribution. For OOD, we consider the following additional datasets:
CIFAR-100 [48], resized LSUN and ImageNet [29], and fixed LSUN and ImageNet
[51], denoted as LSUN (FIX) and ImageNet (FIX), respectively. ResNet-18 and
DenseNet [52] architectures are utilized as classifiers from which gradient-
based representations are obtained. We measure the detection performance with
the following evaluation metrics:

  * •

Detection accuracy measures the maximum classification accuracy over all
possible confidence thresholds $\delta$,

$\small\operatorname*{max}_{\delta}\left\\{0.5P_{in}(q(x)\leq\delta)+0.5P_{out}(q(x)>\delta)\right\\},$

where $q(x)$ denotes the confidence score from the detector. For our method,
we fix the threshold $\delta$ to $0.5$.

  * •

AUROC measures the area under the receiver operating characteristic curve,
which plots the true positive rate against the false positive rate obtained
with varying threshold settings.

  * •

AUPR measures the area under the precision-recall curve, which plots the
precision (the ratio between true positives and all positives) against the
true positive rate with varying threshold values.

(a) CIFAR-10

(b) SVHN

(c) ImageNet

(d) LSUN

Figure 6: Sample dataset images. TABLE II: Adversarial detection results for CIFAR-10 in AUROC. For detecting unknown attacks, attacks denoted by “seen” are used for detector training and evaluated on unknown attacks. Model | Attacks | Baseline | LID | M(V) | M | Hu  _et al._ | Purview | (Seen: BIM) | (Seen: PGD)
---|---|---|---|---|---|---|---|---|---
M | Purview | M | Purview
ResNet | FGSM | 68.89 | 90.06 | 81.69 | 99.95 | 84.06 | 93.45 | 99.88 | 90.11 | 83.97 | 91.33
BIM | 54.09 | 99.21 | 87.09 | 100.0 | 83.65 | 96.19 |  (100.0) | (96.19) | 74.44 | 95.89
C&W | 87.58 | 76.47 | 74.51 | 92.79 | 84.02 | 97.07 | 86.65 | 94.98 | 77.54 | 95.37
PGD | 57.22 | 67.48 | 56.27 | 75.98 | 83.93 | 95.82 | 60.98 | 94.87 | (75.98) |  (95.82)
IterLL | 82.00 | 85.17 | 62.32 | 92.10 | 84.22 | 98.17 | 81.48 | 95.10 | 92.24 | 96.42
Semantic | 78.88 | 86.25 | 64.18 | 84.38 | 80.95 | 90.15 | 70.62 | 60.50 | 83.22 | 60.81
AutoAttack | 5.13 | 96.37 | 76.15 | 95.98 | 69.12 | 99.30 | 85.71 | 97.71 | 95.55 | 96.33
DenseNet | FGSM | 75.25 | 98.23 | 86.88 | 99.97 | 91.67 | 96.83 | 99.87 | 95.15 | 99.70 | 95.60
BIM | 47.01 | 100.0 | 89.19 | 100.0 | 90.32 | 96.85 |  (100.0) | (96.85) | 100.0 | 96.99
C&W | 88.59 | 80.58 | 75.77 | 90.76 | 93.88 | 97.05 | 78.06 | 95.66 | 84.36 | 96.11
PGD | 50.30 | 83.01 | 70.39 | 83.61 | 93.55 | 96.77 | 75.64 | 96.71 | (83.61) |  (96.77)
IterLL | 63.68 | 83.16 | 70.17 | 77.84 | 91.67 | 98.53 | 75.68 | 95.54 | 79.39 | 96.23
Semantic | 79.22 | 81.41 | 62.16 | 67.29 | 94.12 | 89.55 | 66.59 | 57.63 | 45.85 | 58.38
AutoAttack | 31.93 | 95.06 | 90.15 | 97.31 | 36.00 | 98.16 | 97.57 | 96.91 | 94.30 | 95.45

The results of OOD detection are reported in Table I along with other state-
of-the-art methods, including Baseline [28], ODIN [29], Mahalanobis [32],
Energy [33], and GradNorm [43]. For the Energy and GradNorm methods, the
detection accuracy is omitted because they do not specifically determine the
threshold values for their OOD scores for detection. We observe that our
method, shown as Purview, outperforms all state-of-the-art methods in AUROC
when SVHN is used as in-distribution in 11 out of 12 cases. With CIFAR-10 as
in-distribution, our method outperforms all others in AUROC when SVHN, LSUN,
and LSUN (FIX) are used as OOD. Our method is particularly effective when
there is a considerable difference in the complexity of the in-distribution
and OOD datasets. We provide sample images from each dataset for the visual
analysis in Fig. 6. ImageNet shows the most visual similarity with CIFAR-10 in
terms of the types and scales of objects in the images among the three OOD
datasets shown in Fig. 6 6(b)-6(d). SVHN is a street-view house number dataset
that is vastly different from CIFAR-10 dataset of natural images. LSUN is a
scene-understanding dataset with scene categories for classification. The
results support our intuition behind the purview of a trained network based on
its data-dependent effective capacity. The models trained on CIFAR-10 have
higher effective capacity than those trained on the simpler SVHN. Their
feature sets include more diverse features that can be utilized to approximate
the unfamiliar characteristics of the OOD datasets. On the other hand, models
trained on SVHN have relatively simple features. They require more extensive
updates to handle OOD datasets, exhibiting a more apparent gap in the gradient
responses and leading to a better OOD detection performance.

Similar reasoning can be applied to the inferior detection performance
observed with ImageNet and CIFAR-100 as OOD. With SVHN as in-distribution, our
approach achieves the best performance in all cases except one, and a very
close second in one case. However, with CIFAR-10 as in-distribution, our
approach is outperformed by up to 3.5% AUROC for ImageNet, 3.7% for
ImageNet(FIX), and 6.8% for CIFAR-100. Our experiment results show that the
more similar the in-distribution and OOD datasets are, the smaller the gap in
the spreads of gradient magnitudes, which makes it more challenging to
differentiate OOD samples from in-distribution samples. Overall, our gradient-
based method outperforms all activation-based methods in 17 out of 24 cases in
terms of AUROC.

###  IV-B Adversarial Detection

We utilize ResNet and DenseNet classifiers trained on CIFAR-10 training set
for adversarial detection. The test set of CIFAR-10 is utilized to generate
the following adversarial attacks: fast gradient sign method (FGSM) [4], basic
iterative method (BIM) [35], Carlini & Wagner attack (C&W) [53], projected
gradient descent (PGD) [36], iterative least-likely class method (IterLL)
[35], semantic attack [54], and AutoAttack [55]. We utilize the pristine
CIFAR-10 test set as normal samples and adversarial images as anomalous
samples. For comparison, we employ the Baseline method, local intrinsic
dimensionality (LID) scores [31], the Mahalanobis method, and the approach
introduced by Hu  _et al._ [56]. The performance is measured in AUROC and
reported in Table II. For the Mahalanobis method (M), we report vanilla
results (V), _i.e._ , without the input pre-processing or feature ensemble,
and results with both to demonstrate its dependency on the delicate
calibration of hyperparameters, neither of which our approach requires.

The Mahalanobis method performs best for the adversarial attack types of FGSM
and BIM. In contrast, our approach outperforms all state-of-the-art methods
for C&W, PGD, IterLL, Semantic and AutoAttack attacks. Note that the
Mahalanobis method requires fine-tuning of hyperparameters for each attack
type, and we report their results with the best-performing parameters selected
within the range of values explored in their study. Interestingly, the
Mahalanobis approach shows a significant drop in performance for the types of
attacks that were not included in their work. This indicates the need for
additional hyperparameter tuning outside of the suggested parameter values. It
highlights the benefit of our approach in its simplicity of obtaining
gradient-based representations without the need for pre- or post-processing.
Our method can effectively characterize adversarial attack samples, even for
attack types widely considered highly challenging to detect: C&W, PGD, and
AutoAttack.

TABLE III: Corrupted input detection results for CURE-TSR dataset in detection accuracy. All values are in percentages and the best results are highlighted in bold. The results shown in colors are statistically significant, where green shows purview outperforms Mahalanobis, and red vice versa. (Lev.: corruption severity levels, Avg.: average scores across all levels for each corruption type.) |  Mahalanobis [32] / Purview
---|---
Lev. | Decolor | LensBlur | CodecError | Darkening | DirtyLens | Exposure | GaussianBlur | Noise | Rain | Shadow | Snow | Haze
1 |  62.82 / 59.30 |  58.77 / 66.50 |  [HTML]B7E1CD56.22 / 72.14 |  [HTML]B7E1CD67.84 / 79.75 |  53.37 / 50.45 |  [HTML]B7E1CD70.54 / 80.03 |  [HTML]B7E1CD61.69 / 75.16 |  53.67 / 50.22 |  [HTML]B7E1CD74.81 / 93.83 |  [HTML]F4C7C362.37 / 56.48 |  51.35 / 56.15 |  [HTML]B7E1CD70.39 / 89.55
2 |  70.01 / 73.56 |  [HTML]B7E1CD67.54 / 79.21 |  [HTML]B7E1CD59.00 / 74.63 |  [HTML]B7E1CD74.21 / 96.51 |  51.27 / 57.07 |  [HTML]F4C7C399.40 / 91.24 |  [HTML]B7E1CD70.99 / 78.31 |  62.07 / 59.51 |  [HTML]B7E1CD89.81 / 96.24 |  57.72 / 62.41 |  [HTML]B7E1CD64.47 / 72.26 |  95.50 / 96.48
3 |  [HTML]B7E1CD68.37 / 81.92 |  81.78 / 84.69 |  [HTML]B7E1CD55.62 / 79.30 |  [HTML]B7E1CD98.20 / 99.45 |  [HTML]B7E1CD57.80 / 74.99 |  [HTML]F4C7C399.33 / 95.70 |  87.48 / 86.63 |  [HTML]B7E1CD59.37 / 73.94 |  [HTML]B7E1CD93.10 / 96.30 |  [HTML]B7E1CD54.27 / 65.64 |  82.01 / 82.71 |  [HTML]F4C7C3100.0 / 99.18
4 |  [HTML]B7E1CD78.71 / 87.78 |  86.43 / 86.72 |  [HTML]B7E1CD55.55 / 82.96 |  100.0 / 99.93 |  [HTML]B7E1CD67.54 / 83.49 |  99.70 / 97.60 |  [HTML]B7E1CD80.28 / 87.49 |  [HTML]B7E1CD63.49 / 83.31 |  [HTML]B7E1CD94.53 / 96.35 |  [HTML]B7E1CD51.35 / 69.00 |  [HTML]F4C7C395.50 / 88.89 |  100.0 / 98.41
5 |  [HTML]B7E1CD80.88 / 90.34 |  90.33 / 88.68 |  [HTML]B7E1CD56.07 / 84.27 |  100.0 / 99.79 |  95.28 / 94.48 |  100.0 / 99.51 |  [HTML]F4C7C392.28 / 89.63 |  [HTML]B7E1CD75.64 / 87.68 |  96.63 / 97.75 |  [HTML]B7E1CD63.34 / 72.11 |  [HTML]F4C7C399.33 / 91.77 |  100.0 / 98.79
Avg. |  [HTML]B7E1CD72.16 / 78.58 |  [HTML]B7E1CD76.97 / 81.16 |  [HTML]B7E1CD56.49 / 78.66 |  [HTML]B7E1CD88.05 / 95.08 |  [HTML]B7E1CD65.05 / 72.09 |  93.79 / 92.82 |  [HTML]B7E1CD78.55 / 83.44 |  [HTML]B7E1CD62.85 / 70.93 |  [HTML]B7E1CD89.78 / 96.10 |  [HTML]B7E1CD57.81 / 65.13 |  78.53 / 78.36 |  [HTML]B7E1CD93.18 / 96.48
TABLE IV: Corrupted input detection results for CIFAR-10-C dataset in
detection accuracy. All values are in percentages and the best results are
highlighted in bold. The results shown in colors are statistically
significant, where green shows purview outperforms Mahalanobis, and red vice
versa.

|  Mahalanobis [32] / Purview
---|---
Levels | GaussianNoise | ShotNoise | ImpulseNoise | SpeckleNoise | GaussianBlur | DefocusBlur | GlassBlur | MotionBlur | ZoomBlur
1 |  [HTML]B7E1CD90.88 / 99.17 |  [HTML]B7E1CD88.93 / 98.86 |  [HTML]B7E1CD90.65 / 99.07 |  [HTML]B7E1CD88.75 / 98.95 |  [HTML]B7E1CD87.78 / 99.01 |  [HTML]B7E1CD87.90 / 99.16 |  [HTML]B7E1CD96.95 / 99.61 |  [HTML]B7E1CD94.30 / 99.67 |  [HTML]B7E1CD96.18 / 99.82
2 |  [HTML]B7E1CD94.93 / 99.44 |  [HTML]B7E1CD91.20 / 99.11 |  [HTML]B7E1CD93.50 / 99.55 |  [HTML]B7E1CD91.83 / 99.18 |  [HTML]B7E1CD96.33 / 99.78 |  [HTML]B7E1CD92.33 / 99.57 |  [HTML]B7E1CD96.70 / 99.61 |  [HTML]B7E1CD97.15 / 99.85 |  [HTML]B7E1CD97.13 / 99.86
3 |  [HTML]B7E1CD97.10 / 99.66 |  [HTML]B7E1CD94.95 / 99.37 |  [HTML]B7E1CD95.73 / 99.64 |  [HTML]B7E1CD93.30 / 99.37 |  [HTML]B7E1CD98.30 / 99.93 |  [HTML]B7E1CD96.30 / 99.81 |  [HTML]B7E1CD96.05 / 99.80 |  [HTML]B7E1CD98.10 / 99.94 |  [HTML]B7E1CD98.05 / 99.96
4 |  [HTML]B7E1CD97.75 / 99.65 |  [HTML]B7E1CD96.45 / 99.55 |  [HTML]B7E1CD98.18 / 99.77 |  [HTML]B7E1CD94.93 / 99.51 |  [HTML]B7E1CD99.25 / 99.97 |  [HTML]B7E1CD98.38 / 99.90 |  [HTML]B7E1CD97.40 / 99.76 |  [HTML]B7E1CD98.05 / 99.92 |  [HTML]B7E1CD98.40 / 99.93
5 |  [HTML]B7E1CD98.15 / 99.67 |  [HTML]B7E1CD97.53 / 99.53 |  [HTML]B7E1CD99.08 / 99.85 |  [HTML]B7E1CD96.05 / 99.52 |  [HTML]B7E1CD99.18 / 99.95 |  [HTML]B7E1CD99.53 / 99.93 |  [HTML]B7E1CD96.63 / 99.73 |  [HTML]B7E1CD98.60 / 99.92 |  [HTML]B7E1CD98.85 / 99.92
Average |  [HTML]B7E1CD95.76 / 99.52 |  [HTML]B7E1CD93.81 / 99.28 |  [HTML]B7E1CD95.43 / 99.57 |  [HTML]B7E1CD92.97 / 99.30 |  [HTML]B7E1CD96.17 / 99.73 |  [HTML]B7E1CD94.89 / 99.67 |  [HTML]B7E1CD96.75 / 99.70 |  [HTML]B7E1CD97.24 / 99.86 |  [HTML]B7E1CD97.72 / 99.89

Levels | Snow | Frost | Fog | Brightness | Spatter | Contrast | Elastric | Pixelate | Jpeg | Saturate
---|---|---|---|---|---|---|---|---|---|---
1 |  [HTML]B7E1CD87.03 / 98.95 |  [HTML]B7E1CD86.85 / 98.94 |  [HTML]B7E1CD89.58 / 99.27 |  [HTML]B7E1CD84.10 / 98.48 |  [HTML]B7E1CD86.45 / 98.92 |  [HTML]B7E1CD91.08 / 99.52 |  [HTML]B7E1CD91.93 / 99.68 |  [HTML]B7E1CD86.28 / 98.98 |  [HTML]B7E1CD87.95 / 99.37 |  [HTML]B7E1CD87.15 / 99.22
2 |  [HTML]B7E1CD90.80 / 99.21 |  [HTML]B7E1CD88.08 / 99.03 |  [HTML]B7E1CD94.23 / 99.74 |  [HTML]B7E1CD83.78 / 98.43 |  [HTML]B7E1CD88.80 / 99.14 |  [HTML]B7E1CD97.58 / 99.88 |  [HTML]B7E1CD92.45 / 99.65 |  [HTML]B7E1CD88.48 / 99.09 |  [HTML]B7E1CD88.95 / 99.52 |  [HTML]B7E1CD87.28 / 99.22
3 |  [HTML]B7E1CD88.15 / 99.17 |  [HTML]B7E1CD92.23 / 99.35 |  [HTML]B7E1CD96.60 / 99.80 |  [HTML]B7E1CD83.45 / 98.58 |  [HTML]B7E1CD89.33 / 99.21 |  [HTML]B7E1CD99.00 / 99.94 |  [HTML]B7E1CD95.28 / 99.78 |  [HTML]B7E1CD89.38 / 99.22 |  [HTML]B7E1CD89.35 / 99.54 |  [HTML]B7E1CD83.18 / 98.19
4 |  [HTML]B7E1CD88.18 / 98.92 |  [HTML]B7E1CD92.68 / 99.51 |  [HTML]B7E1CD98.00 / 99.86 |  [HTML]B7E1CD83.55 / 98.51 |  [HTML]B7E1CD89.43 / 98.75 |  [HTML]B7E1CD99.05 / 99.95 |  [HTML]B7E1CD95.55 / 99.85 |  [HTML]B7E1CD92.25 / 99.48 |  [HTML]B7E1CD88.60 / 99.68 |  [HTML]B7E1CD82.93 / 98.08
5 |  [HTML]B7E1CD89.20 / 99.17 |  [HTML]B7E1CD94.93 / 99.75 |  [HTML]B7E1CD99.38 / 99.95 |  [HTML]B7E1CD83.83 / 98.57 |  [HTML]B7E1CD92.73 / 99.31 |  [HTML]B7E1CD99.38 / 99.99 |  [HTML]B7E1CD95.20 / 99.75 |  [HTML]B7E1CD95.05 / 99.52 |  [HTML]B7E1CD89.43 / 99.75 |  [HTML]B7E1CD83.70 / 98.27
Average |  [HTML]B7E1CD88.67 / 99.08 |  [HTML]B7E1CD90.95 / 99.32 |  [HTML]B7E1CD95.56 / 99.72 |  [HTML]B7E1CD83.74 / 98.51 |  [HTML]B7E1CD89.35 / 99.06 |  [HTML]B7E1CD97.22 / 99.86 |  [HTML]B7E1CD94.08 / 99.74 |  [HTML]B7E1CD90.29 / 99.26 |  [HTML]B7E1CD88.86 / 99.57 |  [HTML]B7E1CD84.85 / 98.59

Along with known attack detection experiments, we show our approach’s unknown
attack detection performance against the Mahalanobis method, which showed the
second-best performance following ours. The detectors are trained on attack
images of BIM and PGD (denoted as “seen”) and evaluated on unknown attacks.
The detection performance on the known attack is also included but shown in
parentheses for reference. We observe a similar trend to the known attack
detection, where the Mahalanobis method shows saturated performance on the
attacks for which the hyperparameters are tuned. On the contrary, our approach
shows more consistent performance across all unknown attacks except Semantic
attack with an average of over 95% AUROC and outperforms the compared method
in 18 out of 28 cases. Our approach can effectively characterize adversarial
perturbations even for unknown attacks.

###  IV-C Corrupted Input Detection

In addition to the widely accepted OOD and adversarial detection setup, we
consider corrupted inputs as another type of anomaly. Deployed in the real
world, neural networks are known to suffer from imperfect samples due to the
data acquisition process and environmental factors, such as motion blur or
weather conditions [1, 3, 2]. We use image classification datasets designed to
benchmark the robustness of neural networks under realistic challenging
conditions. CIFAR-10-C [2] consists of 19 diverse corruption types in four
categories, including noise, blur, weather, and digital, at five different
severity levels that are applied to test images of CIFAR-10 dataset. CURE-TSR
[8] is a traffic sign recognition dataset that includes real-world and
simulated challenging conditions of 12 types and five severity levels. For
each dataset, a ResNet model is trained on corruption-free images, and the
gradients are collected from pristine images in the test sets and their
corrupted versions. We utilize the Mahalanobis method [32] for comparison
because it showed the best performance among all the compared methods for OOD
detection and adversarial detection.

TABLE V: Classification accuracy for models used in OOD and adversarial detection (%). For OOD detection, we report the accuracy on test sets of the in-distribution datasets. For adversarial detection, the model trained on clean CIFAR-10 is tested on the adversarial attack images generated using the test set. We also include the accuracy of models used in Section III-C2. Network Architecture | OOD | Adversarial Attacks on CIFAR-10 | (OOD)
---|---|---|---
SVHN | CIFAR-10 |  FGSM | BIM | C&W | PGD | IterLL | Semantic | AutoAttack | MNIST | ImageNet
ResNet | 96.65 | 93.84 |  68.17 | 22.37 | 5.38 | 27.12 | 44.31 | 66.1 | 0.06 | 98.24 | 32.06
DenseNet | 96.52 | 94.34 |  53.21 | 7.29 | 4.5 | 8.67 | 17.88 | 68.9 | 0.02 | - | -

In the experiments, we observed that the AUROC scores are highly saturated for
both methods in many cases, particularly for CIFAR-10-C dataset, which calls
for a more comprehensive comparison. To better facilitate the performance
comparison, we employ corrected repeated $k$-fold cross-validated (CV) paired
t-test [57] as a measure of statistical significance. Paired t-test is a
statistical test for comparing two different learning schemes, $A$ and $B$,
based on a number of observations; in this case, predictive accuracy $a$ and
$b$. To improve the stability of the test, Nadeau and Bengio [58] proposed a
corrected variant for the CV setup, and Bouckaert and Frank [57] extended it
to a repeated CV setup. For the corrected repeated $k$-fold CV test, each
model is trained using $k$-fold cross-validation sets and the process is
repeated $r$ times ($r>1$), resulting in $a_{ij}$ and $b_{ij}$ for each fold
$i,1\leq i\leq k$ and each run $j,1\leq j\leq r$. Let $x_{ij}$ be the observed
difference $x_{ij}=a_{ij}-b_{ij}$, and $m$ and $\sigma^{2}$ be the estimates
for the mean $m=\frac{1}{kr}\sum_{i=1}^{k}\sum_{j=1}^{r}x_{ij}$ and variance
$\sigma^{2}=\frac{1}{kr-1}\sum_{i=1}^{k}\sum_{j=1}^{r}(x_{ij}-m)^{2}$. The
test statistic $t$ is computed as following:

$t=\frac{m}{\sqrt{(\frac{1}{kr}+\frac{n_{2}}{n_{1}})\sigma^{2}}}$ (21)

where $n_{1}$ and $n_{2}$ are the numbers of training and testing instances
for the variance estimate correction, respectively. $t$ is compared to a
threshold value based on the significance level $p$ to determine whether the
performance of the two models is significantly different (_i.e._ ,
statistically significant, SS). In our experiments, we use $k=5$, $r=2$, and
$p=0.05$.

We report the detection performance on CURE-TSR in Table III and CIFAR-10-C in
Table IV. As the statistical test is conducted using model predictions, we
report the performance in terms of detection accuracy rather than AUROC. We
highlight the SS detection accuracy in green and red, where green indicates
that our method outperforms the Mahalanobis method, and red shows the opposite
case. Both methods show higher variability in performance across different
severity levels of all corruption types for CURE-TSR compared to the
relatively saturated results for CIFAR-10-C. The proposed method outperforms
the compared method in all 95 cases for CIFAR-10-C and 30 out of 37 SS cases
(60 overall cases) for CURE-TSR. The other 23 cases of CURE-TSR show no
statistically significant differences. Our approach also shows statistically
superior performance in 29 out of 31 cases in the average detection accuracy
across all severity levels of each corruption type in both datasets. The
remaining two cases show no statistical difference. Note the superior SS
performance of our approach, even at lower severity levels for both datasets.
This shows that our gradient-based representations can characterize corruption
more effectively, even at a subtle degree. Overall, we highlight that our
approach outperforms the activation-based approach in terms of SS results:
94.7% among the corruption-level-specific performances and 100% among the
corruption-wise average performances.

###  IV-D Implementation Details

Classification Networks. For the image classification task, we employ two
state-of-the-art neural network architectures, which are then utilized to
generate gradient-based representations with confounding labels: ResNet [45]
and DenseNet [52]. We use ResNet with 18 layers and DenseNet with 100 layers,
growth rate $k=12$, and dropout rate of $0$. Both networks with no pre-
training are trained to minimize the cross-entropy loss with SGD optimization
(Nesterov momentum factor of $0.9$ and weight decay of
$5\text{\times}{10}^{-4}$. A learning rate scheduler is implemented to decay
the learning rate by a factor of $0.1$ at 50% and 75% of the total number of
training epochs. The models are trained for 300 epochs with a starting
learning rate of $0.1$ and batch size of 64. The classification accuracy on
test sets for the models used in the paper is listed in Table V.

Detection Networks. For anomalous input detection tasks, we utilize a simple
multilayer perceptron (MLP) architecture of two fully-connected layers with
ReLU non-linearity and dropout following the first fully-connected layer and
Sigmoid after the second layer. Given an input of gradient-based
representations with dimension $d$, the fully-connected layers have dimensions
of $d\times 40$ and $40\times 1$. The detectors are trained to minimize the
binary cross-entropy loss using the Adam optimizer and a learning rate of
$1\text{\times}{10}^{-3}$ for 30 epochs.

Adversarial Attack Hyperparameters. To generate adversarial attack images, we
borrow publicly available code
bases111https://github.com/Harry24k/adversarial-attacks-pytorch,
https://github.com/cleverhans-lab/cleverhans and utilize the hyperparameter
values listed in Table VI.

  * •

$\epsilon$ : maximum magnitude of perturbation

  * •

$\alpha$ : step size in each iteration for iterative attacks

  * •

$c$ : box constraint weight for optimization

  * •

$k$ : confidence threshold for misclassification

  * •

steps : number of iterations

TABLE VI: Hyperparameters for adversarial attacks. | $\epsilon$ |  $\alpha$ |  $c$ |  $k$ | steps
---|---|---|---|---|---
FGSM | 0.03 | - | - | - | -
BIM | 0.03 | 0.008 | - | - | 100
C&W | - | - | 1 | 0 | 1000
PGD | 0.03 | 0.008 | - | - | 10
IterLL | 0.05 | 0.005 | - | - | 15
AutoAttack | 0.03 | - | - | - | 100

###  IV-E Ablation Studies

Our approach to probing the purview of neural networks is free of
hyperparameters. It involves some parameters that can be adjusted, but they
remain fixed for the experiments shown throughout the paper: 1) the design of
confounding labels and 2) the anomalous input detector architecture and
training. This section provides detailed ablation studies on these two factors
to further validate that our method does not require hyperparameter tuning.

TABLE VII: Detection results using NR confounding labels for ablation studies in AUROC. Confounding Label Designs | Adversarial | Out-of-Distribution |  (OOD)
---|---|---|---
FGSM | BIM | C&W | PGD | IterLL | Semantic |  SVHN | ImageNet | LSUN |  STL10
All-hot label | 93.45 | 96.19 | 97.07 | 95.82 | 98.17 | 90.15 | 99.90 | 96.88 | 99.90 | 74.94
Top-k Prediction | 1 | 89.47 | 88.82 | 97.62 | 93.15 | 94.94 | 88.82 | 99.84 | 95.09 | 99.83 | 70.97
2 | 88.98 | 94.29 | 97.64 | 93.31 | 95.09 | 88.61 | 99.83 | 94.89 | 99.87 | 70.37
3 | 89.21 | 94.34 | 97.49 | 93.48 | 94.88 | 89.08 | 99.84 | 94.94 | 99.83 | 70.64
4 | 89.55 | 93.90 | 97.64 | 93.36 | 94.94 | 88.86 | 99.84 | 95.22 | 99.82 | 70.70
5 | 89.46 | 94.40 | 97.63 | 93.49 | 95.13 | 88.43 | 99.82 | 95.25 | 99.81 | 70.46
Taxanomy | Animal | 93.65 | 95.61 | 97.16 | 95.00 | 97.33 | 90.68 | 99.82 | 92.09 | 99.81 | 72.88
Vehicle | 93.19 | 95.33 | 96.43 | 94.59 | 96.82 | 90.89 | 99.87 | 92.07 | 99.88 | 72.44
Max Logit Value | 91.02 | 95.53 | 97.67 | 94.68 | 96.73 | 89.49 | 99.85 | 95.85 | 99.89 | 71.67
TABLE VIII: Detection results using FR confounding labels in comparison with NR all-hot encoding in AUROC. Confounding Designs | Adversarial | (OOD)
---|---|---
FGSM | BIM | C&W | PGD | STL10
NR | All-hot | 93.45 | 96.19 | 97.07 | 95.82 | 74.94
FR | Target | 92.75 | 96.78 | 98.17 | 96.31 | -
Common | - | 73.64
Unique | 75.11
TABLE IX: Ablation study on the hyperparameters for the detector network architectures and training. The hyperparameter values used for experiments in Section IV are underlined for reference. Hyperparameter | SVHN | ImageNet | LSUN
---|---|---|---
Accuracy | AUROC | AUPR | Accuracy | AUROC | AUPR | Accuracy | AUROC | AUPR
Layers | 2 | 98.44 | 99.90 | 99.99 | 90.97 | 96.88 | 97.20 | 98.59 | 99.90 | 99.91
3 | 98.90 | 99.95 | 99.99 | 91.75 | 97.16 | 97.28 | 99.22 | 99.97 | 99.97
4 | 99.01 | 99.95 | 99.99 | 91.40 | 97.12 | 97.24 | 99.14 | 99.95 | 99.95
Neurons | 10 | 98.08 | 99.84 | 99.98 | 90.73 | 96.54 | 96.78 | 98.04 | 99.80 | 99.81
15 | 98.05 | 99.83 | 99.98 | 90.84 | 96.74 | 96.92 | 98.40 | 99.86 | 99.86
20 | 98.21 | 99.86 | 99.98 | 90.92 | 96.73 | 96.92 | 98.57 | 99.90 | 99.91
25 | 98.33 | 99.87 | 99.99 | 91.09 | 96.87 | 97.08 | 98.66 | 99.90 | 99.91
30 | 98.25 | 99.87 | 99.99 | 90.92 | 96.78 | 97.15 | 98.61 | 99.90 | 99.91
35 | 98.31 | 99.88 | 99.99 | 90.96 | 96.82 | 96.90 | 98.60 | 99.89 | 99.90
40 | 98.44 | 99.90 | 99.99 | 90.97 | 96.88 | 97.20 | 98.59 | 99.90 | 99.91
45 | 98.37 | 99.90 | 99.99 | 91.23 | 97.00 | 97.42 | 98.62 | 99.90 | 99.90
Epochs | 10 | 97.45 | 99.76 | 99.97 | 91.07 | 96.74 | 97.09 | 97.64 | 99.78 | 99.78
15 | 98.08 | 99.84 | 99.98 | 90.78 | 96.69 | 96.96 | 97.99 | 99.81 | 99.82
20 | 98.06 | 99.87 | 99.99 | 90.82 | 96.70 | 96.80 | 98.57 | 99.88 | 99.88
25 | 98.46 | 99.89 | 99.99 | 91.09 | 96.79 | 97.00 | 98.69 | 99.92 | 99.92
30 | 98.44 | 99.90 | 99.99 | 90.97 | 96.88 | 97.20 | 98.59 | 99.90 | 99.91
35 | 98.57 | 99.90 | 99.99 | 90.88 | 96.81 | 97.15 | 98.82 | 99.91 | 99.92
Learning Rates | 0.1 | 98.73 | 99.90 | 99.99 | 91.15 | 96.76 | 96.84 | 98.89 | 99.89 | 99.87
0.05 | 98.80 | 99.92 | 99.99 | 91.33 | 96.95 | 97.31 | 98.98 | 99.93 | 99.94
0.01 | 98.97 | 99.94 | 99.99 | 91.54 | 97.04 | 97.35 | 99.08 | 99.96 | 99.96
0.005 | 98.87 | 99.93 | 99.99 | 91.47 | 97.11 | 97.35 | 99.06 | 99.96 | 99.96
0.001 | 98.44 | 99.90 | 99.99 | 90.97 | 96.88 | 97.20 | 98.59 | 99.90 | 99.91
0.0005 | 97.96 | 99.83 | 99.98 | 90.91 | 96.77 | 97.13 | 98.17 | 99.87 | 99.88

####  IV-E1 Confounding Label Designs

We introduced confounding labels as a methodology to elicit a gradient
response without relying on any information regarding the inputs given during
inference. The confounding labels are devised by combining the one-hot
encodings of multiple classes, and we utilize the default choice of the
confounding label where the one-hot encodings of all classes are combined
(_i.e._ , an all-hot label) throughout the experiment section. In this
section, we present ablation studies on different designs of confounding
labels, as opposed to the choice of all-hot labels in generating gradients. We
establish two types of confounding labels based on reference to information
regarding inputs during inference: no-reference (NR) and full-reference (FR)
designs. The terminologies are borrowed from the field of image quality
assessment [59], where the reference to clean images is a critical factor in
evaluating the quality of corrupted images. Our goal is to probe the purview
of neural networks during inference, where we generally do not have access to
any information regarding the given inputs. Nonetheless, we consider both
options to explore the effect of confounding label designs and show the most
viable options in each case in the OOD and adversarial detection setups.
Following the experimental setup presented in Sections IV-A and IV-B, we
utilize a ResNet classifier trained on CIFAR-10. For adversarial detection, we
employ FGSM, BIM, C&W, PGD, IterLL, and Semantic attacks. For OOD detection,
we utilize the originally utilized SVHN, resized LSUN, resized ImageNet, and
an additional dataset of STL-10 [60], given the shared classes with CIFAR-10,
which allow for unique insights.

No-Reference (NR) Labels. To construct NR confounding labels, we only utilize
the information that is safely assumed to be available at inference time:
trained classes of the classifier and model outputs in response to given
inputs during inference. We formulate three types of NR labels based on 1)
top-$k$ predictions, 2) taxonomy of trained classes, and 3) maximum logit
values. The top-$k$ prediction-based confounding labels are devised by
combining the one-hot encodings of the top $1$ through $k$ classes based on
the predicted class probabilities. The class taxonomy-based labels are
constructed based on the two superclasses of CIFAR-10: animals (six
subclasses: bird, cat, deer, dog, frog, and horse) and vehicles (four
subclasses: airplane, car, ship, and truck). For each superclass, the labels
are implemented by combining the one-hot encodings of the subclasses. The
logit-based label design is based on the literature introduced in Section
II-C, where the maximum logit value is directly backpropagated in generating
gradients instead of loss values between the model outputs and some labels.

We employ the three types of NR labels in gradient generation for adversarial
and OOD detection and report the detection performance in AUROC in Table VII.
Overall, the all-hot label performs best in 7 out of 10 cases. For adversarial
detection, all-hot label performance is followed by taxonomy-based labels and
maximum logit values. Top-$k$ prediction-based labels are the least effective
in generating gradient-based representations that capture perturbations. This
is because of the nature of adversarial attacks, specifically designed to
deceive the classifiers and result in perturbed probability distributions.
Consequently, the top-$k$ predictions are unreliable for eliciting meaningful
gradient responses. On the other hand, the trained class-based labels can
generate gradients for specific semantic categories, which capture adversarial
perturbations more effectively. For OOD detection, the results exhibit less
variation, except for the inferior performance of taxonomy-based labels for
ImageNet and superior performance for STL-10. Contrary to adversarial attacks,
OOD samples are not intentionally devised for incorrect classification but are
drawn far from the training data distribution. The trained classes of CIFAR-10
may be insufficient for the exponentially larger number of classes represented
in the ImageNet dataset, leading to an inferior performance. Conversely, the
shared classes between STL-10 and CIFAR-10 led to more relevant gradient
responses, and thus, superior detection performance. Overall, the all-hot
confounding label is the most viable design for generating gradient-based
representations that can distinguish adversarial and OOD samples from clean
and in-distribution data, with no reference to information regarding the data
samples.

Full-Reference (FR) Labels. When there is no available information concerning
the given inputs during inference, the all-hot confounding label is proven to
be the best option. However, if we have access to information regarding the
inputs, we can devise better confounding label designs. For adversarial
detection, we consider targeted adversarial attacks where the perturbation is
added to the images so that they can be classified as specific target classes.
The target classes can then be used to formulate the confounding labels using
their one-hot encodings. For OOD detection, we utilize STL-10 dataset for its
nine common classes with CIFAR-10 and one unique class. A common class-based
confounding label can be constructed by combining the one-hot encodings of the
nine common classes and the unique class-based label with the unique class’
one-hot encoding.

The detection results obtained using FR labels are shown in Table VIII. For
adversarial detection, we utilize four attack types (FGSM, BIM, C&W, and PGD)
and present the average results across all target classes. The target-class-
based confounding labels perform better than the all-hot labels for three out
of four adversarial attacks. For OOD detection with STL-10, the unique class-
based label performs better than the all-hot label, and the common class-based
labels perform worse than the all-hot label. The unique class-based
confounding label can elicit gradient responses with respect to the features
of the unique class of CIFAR-10, which are absent in STL-10. As a result, the
gradient distributions exhibit a more significant gap between the datasets and
lead to a better OOD detection performance. Overall, the knowledge about
inputs given during inference has proven useful in designing confounding
labels to generate gradient responses that can better characterize the anomaly
in inputs, leading to improved adversarial and OOD detection performances
compared to NR label designs.

####  IV-E2 Detector Designs and Training

We utilize a simple multilayer perceptron (MLP) as an anomalous input detector
using gradients. This section provides ablation studies on the detector
network design and its training. We consider the ablation settings for the
following hyperparameters:

  * •

Layers: number of layers for the MLP

  * •

Neurons: number of neurons in each layer

  * •

Epochs: number of passes through the dataset in training

  * •

Learning rates: step size at each iteration in optimization

Following Section IV-A, we utilize the OOD detection setup with a ResNet
classifier and CIFAR-10 dataset as in-distribution. The detection performances
using various hyperparameter values are reported in Table IX, measured in
detection accuracy, AUROC, and AUPR, using 5-fold cross-validation. The values
used for the experiments in Sections IV-A through IV-C are referred to as the
default values and are underlined for reference. For the ablation of each
hyperparameter, all other hyperparameters are held constant at the default
values, and only the specific hyperparameter in question is changed.

We observe that varying the hyperparameter values has little effect on the
detection performance for all hyperparameters. The maximum AUROC gap observed
for SVHN and LSUN is only 0.14% when varying the number of epochs and 0.46%
for ImageNet with a different number of neurons. The effect of different
hyperparameter values only leads to an average AUROC gap of 0.09% for SVHN,
0.1% for LSUN, and 0.32% for ImageNet. Note that the default values for the
hyperparameters do not always lead to the best performance. This ablation
study further highlights the simplicity of our approach with no
hyperparameters involved in detector design and training as well as generating
gradients to effectively characterize the anomaly in inputs. The gradient-
based representations generated with confounding labels have proven useful in
differentiating anomalous inputs from the perspective of models without
relying on a sophisticated detector.

##  V Conclusion

In this paper, we propose to examine the data-dependent effective capacity of
neural networks to probe their purview. We define purview of a model as the
capacity required to characterize given samples during inference, in addition
to the capacity defined by its training data. Inspired by the utility of
gradients used in model training, we utilize gradients to measure the amount
of change required for a model to characterize inputs more accurately during
inference. To facilitate gradient generation during inference, we introduce
confounding labels that can be formulated with no information regarding the
given inputs. We validate the effectiveness of gradients generated with
confounding labels in capturing anomalies in inputs for detecting OOD,
adversarial, and corrupted inputs. Finally, we provide extensive ablation
studies for the design of confounding labels and hyperparameter settings.

## References

  * [1] D. Temel, J. Lee, and G. AlRegib, “Cure-or: Challenging unreal and real environments for object recognition,” in _IEEE International Conference on Machine Learning and Applications_ , 2018, pp. 137–144.
  * [2] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” _Proceedings of the International Conference on Learning Representations_ , 2019.
  * [3] D. Temel, J. Lee, and G. AlRegib, “Object recognition under multifarious conditions: A reliability analysis and a feature similarity-based performance estimation,” in _IEEE International Conference on Image Processing_ , 2019, pp. 3033–3037.
  * [4] I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _International Conference on Learning Representations_ , 2015.
  * [5] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _Proceedings of the International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 1321–1330.
  * [6] J. Lee and G. AlRegib, “Gradients as a measure of uncertainty in neural networks,” in _IEEE International Conference on Image Processing_ , 2020\.
  * [7] J. Lee, M. Prabhushankar, and G. AlRegib, “Gradient-based adversarial and out-of-distribution detection,” _ICML Workshop on New Frontiers in Adversarial Machine Learning_ , 2022.
  * [8] D. Temel, G. Kwon, M. Prabhushankar, and G. AlRegib, “Cure-tsr: Challenging unreal and real environments for traffic sign recognition,” _NeurIPS Workshop on Machine Learning for Intelligent Transportation Systems_ , 2017.
  * [9] G. K. Y. Logan, K. Kokilepersaud, C. W. G. AlRegib, and H. Yu, “Multi-modal learning using physicians diagnostics for optical coherence tomography classification,” _IEEE International Symposium on Biomedical Imaging_ , 2022\.
  * [10] Y. Bengio _et al._ , “Learning deep architectures for ai,” _Foundations and trends® in Machine Learning_ , vol. 2, no. 1, pp. 1–127, 2009.
  * [11] Y. N. Dauphin and Y. Bengio, “Big neural networks waste capacity,” _arXiv preprint arXiv:1301.3583_ , 2013.
  * [12] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [13] V. N. Vapnik and A. Y. Chervonenkis, “On the uniform convergence of relative frequencies of events to their probabilities,” in _Theory of Probability and Its Applications_ , 1971.
  * [14] B. Neyshabur, R. Tomioka, and N. Srebro, “In search of the real inductive bias: On the role of implicit regularization in deep learning.” in _International Conference on Learning Representation (ICLR) Workshop_ , 2015\. [Online]. Available: http://arxiv.org/abs/1412.6614
  * [15] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” _Communications of the ACM_ , vol. 64, no. 3, pp. 107–115, 2021.
  * [16] G. Cybenko, “Approximation by superpositions of a sigmoidal function,” _Mathematics of control, signals and systems_ , vol. 2, no. 4, pp. 303–314, 1989.
  * [17] G. F. Montufar, R. Pascanu, K. Cho, and Y. Bengio, “On the number of linear regions of deep neural networks,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [18] Y. Tokozume, Y. Ushiku, and T. Harada, “Between-class learning for image classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5486–5494.
  * [19] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [20] S. Yun, D. Han, S. Chun, S. J. Oh, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019.
  * [21] T. Durand, N. Mehrasa, and G. Mori, “Learning a deep convnet for multi-label classification with partial labels,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 647–657.
  * [22] K. Duarte, Y. Rawat, and M. Shah, “Plm: Partial label masking for imbalanced multi-label classification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2739–2748.
  * [23] S. Mei, C. Zhao, B. Ni, and S. Yuan, “Towards bridging sample complexity and model capacity,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * [24] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry, “Adversarially robust generalization requires more data,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [25] C. Li, H. Farkhoor, R. Liu, and J. Yosinski, “Measuring the intrinsic dimension of objective landscapes,” in _International Conference on Learning Representations_ , 2018.
  * [26] A. Ansuini, A. Laio, J. H. Macke, and D. Zoccolan, “Intrinsic dimension of data representations in deep neural networks,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [27] D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio _et al._ , “A closer look at memorization in deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 233–242.
  * [28] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” _Proceedings of the International Conference on Learning Representations_ , 2017.
  * [29] S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” in _Proceedings of International Conference on Learning Representations_ , 2018\.
  * [30] T. DeVries and G. W. Taylor, “Learning confidence for out-of-distribution detection in neural networks,” _arXiv preprint arXiv:1802.04865_ , 2018.
  * [31] X. Ma, B. Li, Y. Wang, S. M. Erfani, S. Wijewickrema, G. Schoenebeck, D. Song, M. E. Houle, and J. Bailey, “Characterizing adversarial subspaces using local intrinsic dimensionality,” in _Proceedings of the International Conference on Learning Representations_ , 2018.
  * [32] K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 7167–7177.
  * [33] W. Liu, X. Wang, J. Owens, and Y. Li, “Energy-based out-of-distribution detection,” in _Advances in Neural Information Processing Systems_ , 2020\.
  * [34] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” _nature_ , vol. 323, no. 6088, pp. 533–536, 1986.
  * [35] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” _International Conference on Learning Representations Workshop_ , 2017.
  * [36] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _Proceedings of the International Conference on Learning Representations_ , 2017.
  * [37] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [38] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in _2018 IEEE winter conference on applications of computer vision (WACV)_. IEEE, 2018, pp. 839–847.
  * [39] G. AlRegib and M. Prabhushankar, “Explanatory paradigms in neural networks: Towards relevant and contextual explanations,” _IEEE Signal Processing Magazine_ , vol. 39, no. 4, pp. 59–72, 2022.
  * [40] M. Prabhushankar and G. AlRegib, “Introspective learning: A two-stage approach for inference in neural networks,” _Advances in neural information processing systems_ , 2022.
  * [41] G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, “Backpropagated gradient representations for anomaly detection,” in _Proceedings of the European Conference on Computer Vision_ , 2020.
  * [42] P. Oberdiek, M. Rottmann, and H. Gottschalk, “Classification uncertainty of deep neural networks based on gradient information,” in _IAPR Workshop on Artificial Neural Networks in Pattern Recognition_ , 2018, pp. 113–125.
  * [43] R. Huang, A. Geng, and Y. Li, “On the importance of gradients for detecting distributional shifts in the wild,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 677–689, 2021.
  * [44] M. Prabhushankar and G. AlRegib, “Extracting causal visual features for limited label classification,” in _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021, pp. 3697–3701.
  * [45] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [46] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [47] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” _NeurlIPS Workshop on Deep Learning and Unsupervised Feature Learning_ , 2011.
  * [48] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Tech. Rep., 2009.
  * [49] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” _arXiv preprint arXiv:1506.03365_ , 2015.
  * [50] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [51] J. Tack, S. Mo, J. Jeong, and J. Shin, “Csi: Novelty detection via contrastive learning on distributionally shifted instances,” _Advances in neural information processing systems_ , vol. 33, pp. 11 839–11 852, 2020.
  * [52] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4700–4708.
  * [53] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _IEEE Symposium on Security and Privacy_ , 2017, pp. 39–57.
  * [54] H. Hosseini, B. Xiao, M. Jaiswal, and R. Poovendran, “On the limitation of convolutional neural networks in recognizing negative images,” in _IEEE International Conference on Machine Learning and Applications_ , 2017, pp. 352–358.
  * [55] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _International conference on machine learning_. PMLR, 2020, pp. 2206–2216.
  * [56] S. Hu, T. Yu, C. Guo, W.-L. Chao, and K. Q. Weinberger, “A new defense against adversarial images: Turning a weakness into a strength,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [57] R. R. Bouckaert and E. Frank, “Evaluating the replicability of significance tests for comparing learning algorithms,” in _PAKDD_ , vol. 3056. Springer, 2004, pp. 3–12.
  * [58] C. Nadeau and Y. Bengio, “Inference for the generalization error,” _Advances in neural information processing systems_ , vol. 12, 1999.
  * [59] D. Temel, M. Prabhushankar, and G. AlRegib, “Unique: Unsupervised image quality estimation,” _IEEE signal processing letters_ , vol. 23, no. 10, pp. 1414–1418, 2016.
  * [60] A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in _Proceedings of the International Conference on Artificial Intelligence and Statistics_ , 2011, pp. 215–223.

|  Jinsol Lee (Member, IEEE) received the B.S. degree in electrical
engineering from Purdue University and the M.S. degree in electrical and
computer engineering from Georgia Institute of Technology, where she is
currently pursuing the Ph.D. degree with the Omni Lab for Visual Engineering
and Science (OLIVES). She is working in the fields of machine learning, deep
learning, computer vision, and image processing. Her research interests
include the robustness of machine learning models and learning with limited
data.
---|---
|  Charlie Lehman is a Machine Learning Ph.D. student at the Georgia Tech Omni
Lab for Visual Engineering and Science (OLIVES). His research interests
include robustness and explainability of deep vision models. He is also an
Engineering Duty Officer with the U.S. Navy Reserves, where he specializes in
the maintenance and repair of surface vessels.
---|---
|  Mohit Prabhushankar (Member, IEEE) received his Ph.D. degree in electrical
engineering from the Georgia Institute of Technology in 2021. He is currently
a Postdoctoral Research Fellow in the School of Electrical and Computer
Engineering at the Georgia Institute of Technology in the Omni Lab for
Intelligent Visual Engineering and Science (OLIVES). He is working in the
fields of image processing, machine learning, active learning, healthcare, and
robust and explainable AI. He is the recipient of the Best Paper award at ICIP
2019 and Top Viewed Special Session Paper Award at ICIP 2020. He is the
recipient of the ECE Outstanding Graduate Teaching Award, the CSIP Research
award, and of the Roger P Webb ECE Graduate Research Assistant Excellence
award, all in 2022.
---|---
|  Ghassan AlRegib (Fellow, IEEE) is currently the John and Marilu McCarty
Chair Professor in the School of Electrical and Computer Engineering at the
Georgia Institute of Technology. In the Omni Lab for Intelligent Visual
Engineering and Science (OLIVES), he and his group work on robust and
interpretable machine learning algorithms, uncertainty and trust, and human-
in-the-loop algorithms. The group has demonstrated their work on a wide range
of applications such as Autonomous Systems, Medical Imaging, and Subsurface
Imaging. The group is interested in advancing the fundamentals as well as the
deployment of such systems in real-world scenarios. He has been issued several
U.S. patents and invention disclosures. He is active in the IEEE as an IEEE
Fellow. He served on the editorial board of several transactions and served as
the TPC Chair for ICIP 2020, ICIP 2024, and GlobalSIP 2014. He was area editor
for the IEEE Signal Processing Magazine. In 2008, he received the ECE
Outstanding Junior Faculty Member Award. In 2017, he received the 2017 Denning
Faculty Award for Global Engagement. He and his students received the Best
Paper Award in ICIP 2019.
---|---