Citation

M. Prabhushankar and G. AlRegib, ”VOICE: Variance of Induced Contrastive
Explanations to Quantify Uncertainty in Neural Network Interpretability,”
Journal of Selected Topics in Signal Processing (J-STSP) Special Series on AI
in Signal & Data Science, May 23, 2024.

Review

Data of Initial Submission: 17 Aug 2023
Date of Minor Revision: 11 May 2024
Date of Acceptance: 23 May 2024

Codes

https://github.com/olivesgatech/VOICE-Uncertainty

Copyright

©2024 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

Contact

<EMAIL_ADDRESS>OR<EMAIL_ADDRESS>
https://alregib.ece.gatech.edu/

# VOICE: Variance of Induced Contrastive Explanations to quantify Uncertainty
in
Neural Network Interpretability

Mohit Prabhushankar, , Ghassan AlRegib M. Prabhushankar was with the
Department of Electrical and Computer Engineering, Georgia Institute of
Technology, Atlanta, GA, 30332 USA e-mail<EMAIL_ADDRESS>AlRegib was
with the Department of Electrical and Computer Engineering, Georgia Institute
of Technology, Atlanta, GA, 30332 USA e-mail<EMAIL_ADDRESS>received - August 27, 2023; revised - May 11, 2024; accepted - May 23, 2024

###### Abstract

In this paper, we visualize and quantify the predictive uncertainty of
gradient-based _post hoc_ visual explanations for neural networks. Predictive
uncertainty refers to the variability in the network predictions under
perturbations to the input. Visual _post hoc_ explainability techniques
highlight features within an image to justify a network’s prediction. We
theoretically show that existing evaluation strategies of visual explanatory
techniques partially reduce the predictive uncertainty of neural networks.
This analysis allows us to construct a plug in approach to visualize and
quantify the remaining predictive uncertainty of any gradient-based
explanatory technique. We show that every image, network, prediction, and
explanatory technique has a unique uncertainty. The proposed uncertainty
visualization and quantification yields two key observations. Firstly,
oftentimes under incorrect predictions, explanatory techniques are uncertain
about the same features that they are attributing the predictions to, thereby
reducing the trustworthiness of the explanation. Secondly, objective metrics
of an explanation’s uncertainty, empirically behave similarly to epistemic
uncertainty. We support these observations on two datasets, four explanatory
techniques, and six neural network architectures. The code is available at
https://github.com/olivesgatech/VOICE-Uncertainty

###### Index Terms:

Predictive Uncertainty, Gradients, Contrastive explanations, Counterfactual
explanations, Neural Networks, Deep Learning

##  I Introduction

Visual explanations provide rationales that justify a neural network’s
prediction at inference by highlighting features in an image [1]. These
explanations are a popular and intuitive methodology for researchers,
engineers, policymakers, and users to interpret systems that use deep neural
networks. The complexity of visual tasks tackled by neural networks range from
microscopic textures [2] to earth’s subsurface [3]. They are used in sensitive
tasks like detecting cardiovascular [4] and diabetic [5] risk factors in
humans through retinal images. Hence, the utility of neural networks in such
wide-ranging and sensitive applications call for explainability regarding
their decisions. Explainability is a core tenet of neural network deployment,
and is a critical field of study. A number of techniques have been proposed to
explain neural network decisions at inference [6, 7, 8, 9, 10, 11, 12, 13].
All these methods provide visual heatmaps that attribute regions or features
within an image that explain the neural network’s decision. The authors in [1]
provide an overview of these methods by categorizing them based on design
choices. One such design choice is the utility of gradients as features.

Figure 1: GradCAM [6] explanations and the proposed uncertainty visualization.
Top row are results obtained on VGG-16 [14] while bottom row are results
obtained on Swin Transformer [15]. Figs. (a), (b), (e), and (f) are obtained
on a clean image where both VGG-16 and Swin Transformer predict correctly
while Figs. (c), (d), (g), and (h) are results on noisy image where both
networks predict incorrectly.

Loss gradients are used in training to update network parameters towards a
global minima. Gradients provide an intelligent and intuitive search direction
to stochastically update the network parameters towards the minima. Recently,
loss gradients have been utilized at inference to characterize data [16, 17,
18, 19, 20]. Specific to explanations, a number of gradient-based methods have
been proposed including GradCAM [6], GradCAM++ [21], Guided Backpropagation
[9], and SmoothGrad [10] among others. Gradient-based explanatory techniques
are generally both non-explicit and non-interventionist [1]. Explicit
explanatory techniques either require a change to the network architecture
[12], or use an approximation of the architecture [13], or require additional
information to create explanations [11]. Interventionist explanatory
techniques require changes within data to create explanations [8]. Being non-
explicit and non-interventionist allows gradient-based techniques to induce
alternative paradigms of explanations. The authors in [22] postulate one such
alternative paradigm to propose ContrastCAM. While GradCAM backpropagates the
logits of the predicted class $P$ to highlight regions that answer the
question _‘Why P?’_ , ContrastCAM backpropagates the loss between the
predicted class $P$ and some contrast class $Q$ to highlight _‘Why P, rather
than Q?’_. This paradigm of questioning is contrastive in nature. ContrastCAM
induces contrastive explanations using the same methodology as the original
GradCAM, but by backpropagating a loss quantity rather than the prediction. In
this paper, we draw inspiration from these induced contrastive explanations to
construct the uncertainty map of the base explanation. We provide a plug-in
framework that works on any gradient-based neural network and explanatory
technique. We further examine explainability methods and show that each
explainability heatmap has an inherent uncertainty associated with it and this
uncertainty is unique to the network, the image, and the technique that
generated it.

In broad terms, uncertainty is the lack of knowledge within a neural network
that leads to mismatched confidence at inference. For instance, neural
networks are vulnerable to engineered adversarial noise [23] where the
networks predict the wrong class with high confidence. Even under pristine
conditions, there is a significant gap between a network’s prediction accuracy
and its confidence [24]. Hence, a neural network does not know when to trust
itself. Research in neural network uncertainty generally decomposes it into
source factors [25]. A number of techniques have been proposed to quantify
uncertainties in neural networks [26, 27, 28] and reduce them [29]. Note that
all these methods quantify the uncertainty of the neural network’s inference.
In this paper, we visualize and quantify uncertainties of explanatory
techniques that act on the model _after inference_. We show one such
explanatory technique, GradCAM [6], in Fig. 1. In Fig. 1(a), GradCAM explains
VGG-16’s [14] decision of bull-mastiff by highlighting the face of the dog.
The proposed uncertainty method in Fig. 1(b) highlights the region above and
below GradCAM’s explanation with an emphasis on the snout of the dog. Hence,
GradCAM is uncertain of VGG-16’s feature attributions based on the dog’s snout
and the dog’s upper body. Next, GradCAM’s explanation and uncertainty of Swin
Transformer [15] is visualized in Figs. 1(e) and (f) respectively. Notice
that, similar to Fig. 1(b), Fig. 1(f) also curls around its explanation.
However, unlike VGG-16, Swin Transformer is not uncertain about the dog’s
snout since the snout forms an important attribute within its original
explanation. We further complicate this analysis by adding Additive White
Gaussian Noise to the image so that both the networks predict incorrectly. The
results are shown in Figs. 1(c), (d), (g), and (h). In contrast to the results
on pristine images, we make the two following observations. Firstly, both the
explanation and its associated uncertainty are more dispersed across the
image. Secondly, there is a larger overlap between the explanation and its
uncertainty. The second observation indicates that GradCAM is uncertain about
the region it is attributing the decision to, thereby reducing the
trustworthiness of the method. While the goal of the paper is to visualize the
uncertainties in Figs. 1(b), (d), (f), and (h), we also provide two metrics
based on the above observations to quantify these visualizations.

In Section II, we take a deeper look into existing uncertainty quantification
techniques to establish the novelty of the proposed algorithm. We
theoretically motivate the need for uncertainty in explainability in Section
III. The methodology of obtaining uncertainty visualizations as well as
quantifying them is described in Section IV. The results from the proposed
framework are discussed in Section V. The key contributions of the paper
include,

  * •

We propose a rigorous framework for quantifying predictive uncertainty of
gradient-based visual explanations.

  * •

The proposed framework is a plug-in on top of any existing gradient-based
visualization technique.

  * •

We provide two quantitative metrics to characterize the proposed uncertainty.

  * •

The proposed framework allows us to conduct extensive evaluations of existing
visual explanatory techniques when differentiating between correct and
incorrect predictions as well as in the presence of perturbations.

##  II Related Work

###  II-A Explanations and Uncertainty in Neural Networks

Neural networks reason inductively, i.e. they make decisions with uncertainty
which then allows speculation regarding cause [30]. This is opposed to
deductive reasoning where the exact nature of the cause is ascertained before
making decisions. Hence, induction-based neural network reasoning allows a
statistical inference that is equipped with uncertainty [31]. Explanations are
a _post hoc_ function that attempt to justify the intractable inference of
neural networks to humans. Post hoc refers to explanation functions acting on
the network after the network’s decision is made. Computational explanation
models that attempt this justification are a function of the neural network’s
decision and hence the decision’s uncertainty.

###  II-B Uncertainty Quantification in neural networks

The science of uncertainty quantification (UQ) deals with assigning
probabilities regarding decisions made under some unknown states of the
system. When the system is a deep neural network, the primary research
direction in UQ examines the uncertainty associated with the neural network
itself. This research can be summarized by considering the sources of
uncertainties, namely the data and model [27] uncertainties. A secondary
research direction in UQ applies estimated uncertainty to select additional
data for training [32], interpreting existing results [33], estimating image
quality [34], and detecting out-of-distribution and adversarial samples [35].
In all cases, the statistical uncertainty of the decision is quantified. In
this work, we do not estimate the network or the data uncertainty. Rather, we
quantify the uncertainty of the _post hoc_ explanation method that acts on the
network.

Uncertainty in explainability is less researched in literature. This is partly
because the goal of explainability is _post hoc_ , i.e. explanatory techniques
act after a decision is made. On the other hand, UQ deals with estimating and
reducing uncertainty of the decision itself. The authors in [36] incorporate
uncertainty estimation during the training process to improve visual
explanations at inference for the task of visual question answering. In [37],
causal metrics of necessity and sufficiency are used to evaluate the
uncertainty of explanations in tabular data. The authors in [38] propose an
uncertainty framework using explanations. They propose BayesLIME and
BayesSHARP by measuring the changes in the disagreement between explanations
with slight perturbation. The idea of disagreement as a measure of uncertainty
has been explored in multiple domains [39, 34]. In this paper we show that
each individual perturbation itself creates uncertainty within an explanatory
framework and that the explanatory technique has its own uncertainty. To do
so, we consider a combination of the data and model uncertainties, which is
termed as the predictive uncertainty [40].

Predictive uncertainty is measured as the variance of the model outputs [41].
The authors in [42] discuss three methods to estimate predictive uncertainty
estimates: Deep Ensembles [26], MonteCarlo dropout (MC-dropout [43]), and a
combination of both [28]. The variance for a prediction is calculated across
multiple model outputs to obtain uncertainty. Predictive uncertainty
techniques are generally evaluated against log-likelihood and brier score. In
this paper, we utilize the definition of predictive uncertainty to estimate
the uncertainty of explanation techniques.

###  II-C Visual Explanations

A popular means of visual explanations are attribution masks as shown in Fig.
1. These masks highlight features that led a neural network to make its
decision. The authors in [1] provide a taxonomy of explanatory techniques
based on the design choices of the explanatory techniques and expand on the
advantages of gradient based explanatory methods. Popular gradient-based
methods including GradCAM [6], GradCAM++ [21], Guided Backpropagation [9], and
SmoothGrad [10] all function by backpropagating the prediction logit and
either directly visualizing some processed gradients [9, 10] or using
gradients as a weighing function on other features [6, 21]. Furthermore, the
authors in [22] posit a psychological definition to explanations whereby all
the above feature attribution methods answer _‘Why P?’_ questions where $P$ is
the prediction. This leads to other paradigms of explanations namely
contrastive and counterfactual explanations. Specifically, contrastive methods
answer the question _‘Why P, rather than Q?’_. In this explanatory paradigm,
the network highlights features that separate the prediction $P$ from a
contrast class $Q$. [22] propose ContrastCAM by backpropagating a loss
function between $P$ and $Q$, instead of the prediction logit through any
gradient-based _‘Why P?’_ framework. However, a number of recent works have
questioned the validity of these gradient-based visual explanations. The
authors in [44] suggest that all gradient-based visualization techniques
partially recover the input image. In this paper, we show that it is the
evaluation of any gradient-based visual explanation that lends itself to
reducing the predictive uncertainty in a neural network. In other words,
explanations highlight features that reduce the predictive uncertainty of the
neural network. We describe the evaluation of explanations in literature
before discussing its implications on predictive uncertainty in Section III.

###  II-D Evaluation of visual explanations

A number of techniques have been proposed for evaluating visual explanations.
Specifically, the authors in [1] provide a taxonomy of explanatory evaluation.
The first evaluation strategy, utilized in multiple explanation methods, is
direct human evaluation. Qualitative results from explanatory techniques
$\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ are shown to humans in controlled
scenarios and their preferences between the methods are noted. However, large
scale experiments involving humans is expensive and time consuming. Hence, an
indirect and targeted class of strategies is utilized to evaluate
explanations.

Indirect and targeted evaluation techniques involve masking a given image
using an explanatory map and checking for accuracy of the masked image on the
model. If the decision is unchanged, then the explanation has captured the
essential features, hence fulfilling the requirements of being a _post hoc_
explanation. A number of works propose different methodologies of masking. For
instance, the authors in [8] mask pixel by pixel, by using the explanatory
heatmap as a probability map. They mask the pixels in the original image based
on descending order of probability. The authors in [45] disagree with the
assumption of pixelwise masking and instead mask structure-wise, since the
human vision system is attuned to structures rather than pixels. They use
Huffman encoding as a proxy for structures and mask the unimportant structures
based on explanations. The authors in [21] directly threshold the mask and
only pass those regions in the input image which are above the threshold
value. In Section III, we show that this masking creates a subset of selected
features, which when passed through the network, creates its own unique
uncertainty.

###  II-E Gradients as uncertainty features

Gradients provide a measurable change to the network parameters. During
training, a loss function is backpropagated across the network and the network
parameters are updated until a local minima is reached [46]. Recently, a
number of works have used gradients as features to characterize data as a
function of network weights. This characterizations has shown promising
results in a number of disparate applications including novelty [18], anomaly
[17], and adversarial image detection [35], severity detection [20], image
quality assessment [16], and human visual saliency detection [47]. A number of
theories as to their efficacy has been put forward including neurobiological
[48], behavioral [29], and reasoning-based [30]. In this paper, we adopt the
interpretation of gradients as encoding the uncertainty of a loss function
[19, 49]. This uncertainty is reflected in all explanatory techniques that
backpropagate loss, i.e. gradient-based explanatory methods.

##  III Theory

###  III-A Visual Explanations

In mathematical notations, given a trained neural network
$f:\mathcal{X}\rightarrow\mathcal{Y}$ and an input $x\in\mathcal{X}$, a _post
hoc_ explanation $\mathcal{M}(\cdot)$ highlights features $\mathcal{T}_{P}$
that lead to a decision $P\in\mathcal{Y}$. This is a probabilistic definition
of _observed correlation_ explanation as provided in [1]. Observed correlation
explanations answer the question _‘Why P?’_. The explanatory technique
$\mathcal{M}(\cdot)$ is a function of the prediction $P$, the input $x$, the
network $f(\cdot)$, and the methodology used to derive the explanation,
denoted by $m$. Hence, an explanation is represented as
$\mathcal{M}_{m}(f,x,P)$. Fig. 1(a) represents
$\mathcal{M}_{GradCAM}(\text{VGG, cat-dog, bull mastiff})$, Fig. 1(e) is
$\mathcal{M}_{GradCAM}(\text{Swin Transformer, cat-dog, bull mastiff})$, Fig.
1(c) depicts $\mathcal{M}_{GradCAM}(\text{VGG, cat-dog, coral reef})$ and Fig.
1(g) is $\mathcal{M}_{GradCAM}(\text{Swin, cat-dog, Border terrier})$. Note
that the difference in $y$ in Fig. 1(c) and (g) is because VGG and Swin
predicted $x$ as coral reef and border terrier respectively. Similarly, Fig. 4
shows the results when varying $m$ between GradCAM, GradCAM++, Guided
Backpropagation, and SmoothGrad and $x$ between random images from ImageNet.

Alternative to _observed correlation_ explanations are _observed contrastive_
explanations [1]. Contrastive explanations are ‘ _Why P, rather than Q?_ ’
explanatory techniques that provide contextual explanations that highlight
features for why the network predicted class $P$ rather than some other class
$Q$. The authors in [22] provide a plug-in approach that induces gradient-
based correlation explanations to produce contrastive explanations. They
backpropagate a loss $J(P,Q)$ instead of the logit $y_{P}$ within
$\mathcal{M}_{m}(f,x,P)$. Hence, these explanations can be represented as
$\mathcal{M}_{m}(f,x,J_{P,Q})$. In this paper, we make use of these plug-in
methods to derive uncertainty of the base correlation explanatory maps
$\mathcal{M}_{m}(f,x,P)$.

###  III-B Predictive Uncertainty Decomposition

Under the assumption of a fixed model, predictive uncertainty is defined as
the variability in the model prediction associated with changing the input
values [41]. In a neural network, predictive uncertainty is the variance in
the probability distribution of the outputs of the network given by $f(x)$.
Hence, the predictive uncertainty is given by $V[f(x)]$ where $V[\cdot]$ is
the variance of the predicted logit probabilities. For ease of notation, we
refer to $f(x)$ as the output $y$. The author in [41] decomposes the variance
of logit probabilities as,

$V[y]=V[E(y|S_{x})]+E[V(y|S_{x})].$ (1)

where $S_{x}$ is some perturbation of the input $x$. In this paper, $S_{x}$ is
the explanation-masked input image given by $S_{x}=\mathcal{M}_{m}(f,x,P)\odot
x$. $\odot$ is the element-wise Hadamard product. The first term of the RHS in
Eq. 1 is the variance in the expectation of $y$ under explanation masked image
$S_{x}$. Note that current explanatory techniques use $V[E(y|S_{x})]$ as a
means of evaluating their methods. The explanations, $\mathcal{M}_{1}$ and
$\mathcal{M}_{2}$, mask (or perturb) the image and are passed back in the
network to show that the prediction $y$ does not change in one of the methods.
Mathematically, this equals $E(y|S_{x1})>E(y|S_{x2})$, where
$S_{x1}=\mathcal{M}_{1}(f,x,y)\odot x$ and $S_{x2}=\mathcal{M}_{2}(f,x,y)\odot
x$. Hence this leads to our first observation.

######  Observation III.1.

Visual explanations are evaluated to (partially) reduce the predictive
uncertainty in a neural network.

We say partially since from Eq. 1, only the first term $E(y|S_{x})$ is used
for evaluation and its variance is reduced. The second term is the residual
between the chosen $S_{x}$ and all other possible $S_{x}^{\prime}$ where
$x=S_{x}\cup S_{x}^{\prime}$. In other words, the features _not_ chosen by the
explanation $\mathcal{M}(\cdot)$ that are present in $S_{x}^{\prime}$ are
responsible for the predictive uncertainty $V[y]$. A demonstration of this is
in the results of uncertainty in Fig. 1(b). The network is uncertain regarding
its classification based on the snout of the dog since it shares this feature
with other dog breeds in the dataset. However, GradCAM $m(\cdot)$ does not
consider the snout as salient as the jowls of the dog for its decision. In our
proposed method, we consider the second residual term given by $E[V(y|S_{x})]$
to calculate this uncertainty. This leads to our second observation.

######  Observation III.2.

Uncertainty in explainability occurs due to all combinations of features that
the explanation did not attribute to the network’s decision.

The simple methodology for evaluating $V(y|S_{x})$ is by changing $S_{x}$
randomly to obtain $S_{x}^{\prime}$ in a Monte-Carlo fashion. However, all
combination of features on complex data like images are long and impractical
[50]. We propose a methodology that effectively computes this term.

###  III-C VOICE Theory

The second variance term in Eq. 1 $V(y|S_{x})]$ is the variability in $y$
under the presence of residual $S_{x}^{\prime}$. For a large majority of
changes in $S_{x}$, there is no change in $y$, leading to
$V(y|S_{x}^{\prime})=0$. Hence, Monte-Carlo sampling of $S_{x}^{\prime}$ can
be ineffective for a well trained network. Instead, we use contrastive
explanations that target changes within $x$ that lead to change in prediction
from $P$ to $Q$. In other words, contrastive explanations highlight those
feature $S_{P,Q}$ that differentiate prediction $P$ from $Q$. The prediction
$y$ changes when the features $S_{P,Q}$ are deleted from $S_{x}$.
Mathematically the changed probability $y^{\prime}$ is given by,

$\displaystyle y^{\prime}$ $\displaystyle=f(S_{x}^{\prime})$ (2)
$\displaystyle S_{x}^{\prime}$ $\displaystyle=S_{x}-S_{P,Q}$ (3)

Therefore, we induce changes in $y$ by obtaining contrastive explanations and
taking a variance across the features $S_{x}^{\prime}$ that lead to a change
in outputs $y$. We term this methodology as VOICE: Variance Of Induced
Contrastive Explanations. The exact procedure for generation of VOICE
uncertainty is described in Section IV.

Figure 2: Proposed VOICE uncertainty generation explained using three steps. A
prediction $P$ is made in Step 1. Contrastive explanations are induced using
$R$ contrast classes in Step 2. $R$ is the number of class probabilities that
exceed a threshold $p_{t}$. In Step 3, variance across vertically stacked
explanations is calculated pixelwise and normalized to produce VOICE
uncertainty. The red boxes are plug-in frameworks. Green $p_{t}$ is a
hyperparameter.

##  IV Methodology

In this section, we describe the methodology for obtaining VOICE along with
some objective metrics to characterize VOICE uncertainty.

###  IV-A VOICE: Variance Of Induced Contrastive Explanations

The block diagram for obtaining VOICE uncertainty is shown in Fig. 2. Step 1
in Fig. 2 involves a forward pass of an image $x$ to obtain a prediction $P$
from a trained network $f(\cdot)$. Existing explanatory techniques
$m_{m}(f,x,P)$ where $m$ is any gradient-based method acts on $P$ to answer
the question ‘ _Why P?_ ’ by highlighting visual features that led to the
decision $P$. Note that any explanatory method that backpropagates the logit
$y_{P}$ of the prediction $P$ is a necessary requirement for our method. This
is because in Step 2, a contrastive explanation is induced by backpropagating
a loss function $J(\cdot)$ between a prediction $P$ and some contrast class
$Q$. In this paper, we use cross entropy loss since it is commonly used in
training the network. Note that there are $N$ possible contrast classes to
create contrastive explanations where $N$ is the number of classes the network
is trained to discriminate against. In ImageNet, $N$ is $1000$. Qualitatively,
each contrastive explanation answers the question _’Why P, rather than Q?’_.
For a well trained network, the softmax probabilities of only a subset of the
$1000$ classes are above a threshold $p_{t}$. Let $R$ denote these classes. In
step 2, we backpropagate a loss $J(P,Q)$ through the explanatory methods
$m_{m}(f,x,J_{P,Q})$ where the loss replaces the prediction logit $y_{P}$
similar to Contrast-CAM from [22]. We do this for $R$ classes to obtain $R$
contrastive explanations. In Step 3, the $R$ induced contrastive explanations
are vertically stacked, and their variance is taken across each pixel to
obtain a single heatmap. This heatmap is normalized to obtain the Variance Of
Induced Contrastive Explanations (VOICE) uncertainty map $u_{m}$, that is a
function of the network $f(\cdot)$, the image $x$, the decision $P$ and the
explanatory technique $m(\cdot)$.

Choice of $R$: In Fig. 3, we show VOICE uncertainties on the cat-dog image
from ImageNet when it is passed through VGG-16 and explained using GradCAM at
different probability thresholds $p_{t}$. Fig. 3(a) with $p_{t}=0.001$
represents the case when only those contrastive maps whose class probabilities
have a greater than random chance $\big{(}\frac{1}{N}\big{)}$ are induced and
used, where $N=1000$ is the number of classes in ImageNet dataset.
Progressively increasing the threshold leads to a better uncertainty heatmap
until $p_{t}=0.00001$ in Fig. 3(c), after which there is no visual difference.
In this paper, we use $p=0.00001$ for all visualizations.

Figure 3: Proposed VOICE uncertainty on GradCAM [6] explanation with multiple
threshold probabilities $p_{t}$ with (a) $p_{t}=0.001$, (b) $p_{t}=0.0001$,
(c) $p_{t}=0.00001$, (d) $p_{t}=0.000001$, (e) $p_{t}=0.0000001$.

###  IV-B Quantitative Measures on VOICE

Variance across the contrastive explanations provides a heatmap that can be
visualized. This is shown in Fig. 1. In this subsection, we provide objective
measures to quantify the uncertainty map. We provide two measures, one of
which is a full reference metric, i.e. needing access to the base explanatory
map $m$ to quantify $u_{m}$, and another is a no-reference metric, i.e. does
not require access to $m$ to quantify $u_{m}$. The terminologies of full-
reference and no-reference are derived from the image quality assessment
literature [51].

#### Intersection over Union (IoU)

The explanatory map $m$ are the feature attributions that lead to a decision
$P$. If the explanation-specific uncertainty map $u_{m}$ overlaps with the
explanation $m$, then the overlap can be interpreted as the network being
uncertain about the regions in the image that it is using to make its
decision. Hence, with a high overlap, networks do not trust their own
decisions. We quantify this overlap using Intersection over Union (IoU)
metric, calculated as,

$IoU_{t}=\frac{|u_{m}\cap m|}{|u_{m}\cup m|},$ (4)

where the LHS is the intersection over union given a threshold $t$. The
numerator of the RHS denotes the number of pixels where $u_{m}$ and $m$
overlap and the denominator signifies the number of pixels which forms the
union of $u_{m}$ and $m$. Higher $IoU_{t}$, less trust can be placed on the
attributions made by the explanatory map. We show this in Section V where
wrong predictions made by networks generally have higher IoU compared to
correct predictions. Additionally, since both $m$ and $u_{m}$ are required in
Eq. 4, it is a full-reference metric.

#### Signal to Noise Ratio (SNR)

Signal-to-Noise Ratio is a metric in image processing that is used to compare
the intensity of a desired signal to the intensity of background noise. SNR is
generally calculated as,

$SNR=\frac{\mu(u_{m})}{\sigma(u_{m})},$ (5)

where $u_{m}$ is the uncertainty map, $\mu$ is the expected value or mean of
the uncertainty map and $\sigma$ is the standard deviation of the uncertainty
map. SNR expressed as Eq. 5 is the inverse of the coefficient of variation
(CV). CV is a standardised measure of the dispersion of a probability
distribution. Hence, in the context of uncertainty quantification, SNR
measures the ratio of the mean intensity of $u_{m}$ against the dispersion of
the intensity of $u_{m}$ across the pixels. Higher the SNR of $u_{m}$, more is
the uncertainty. Therefore, for a good explanatory map, SNR of its uncertainty
must be low. Note that unlike IoU, SNR is calculated using $u_{m}$ only
without needing access to $m$ and hence, is a no-reference metric.

In literature, log-likelihood and brier score are used as a proxy to quantify
uncertainty [27]. The authors in [34] note that both brier score and log-
likelihood metrics require ground truth annotations for their calculation.
However, under label uncertainty in practical scenarios of biomedical
diagnosis, these annotations are sometimes unavailable. Hence, the proposed
methodology provides a means to approximate uncertainty without requiring
ground truth. In Section V, we show that both SNR and IoU are correlated with
log-likelihood measure since they measure the residual term of predictive
uncertainty from Eq. 1.

Figure 4: Visualization of GradCAM, GradCAM++, Guided Backpropagation, and
SmoothGrad explanations and their corresponding uncertainties on three
correctly classified and two incorrectly classified randomly selected images
from ImageNet validation set. Guided Backpropagation uncertainty is not easily
visible and red boxes are used to show highlights.

##  V Results

TABLE I: Comparison among 4 explanatory techniques and 6 architectures on ImageNet dataset. Top-1 accuracy is shown. Ideally, the % difference between IoU and SNR must be higher. The highlights in red are lower and explained in Table II. Architecture | Accuracy | Explanation | IoU | SNR
---|---|---|---|---
(%) | Correct | Wrong | % Difference | Correct | Wrong | % Difference
AlexNet [52] | 51.22 |  GradCAM [6] | 0.5329 | 0.6242 | 17.13 | 1.0445 | 1.2952 | 24.00
GradCAM++ [21] | 0.5469 | 0.6732 | 23.09 | 1.0933 | 1.2620 | 15.44
Guided Backprop [9] | 0.0469 | 0.0761 | 62.41 | 0.4686 | 0.4964 | 5.93
SmoothGrad [10] | 0.4507 | 0.5904 | 30.99 | 1.0267 | 1.2010 | 16.97
SqueezeNet [53] | 54.17 |  GradCAM [6] | 0.7163 | 0.7878 | 9.98 | 0.7869 | 0.9268 | 17.77
GradCAM++ [21] | 0.7535 | 0.8210 | 8.96 | 0.8455 | 0.9634 | 13.94
Guided Backprop [9] | 0.1593 | 0.2166 | 35.98 | 0.3989 | 0.4280 | 7.29
SmoothGrad [10] | 0.8556 | 0.9169 | 7.16 | 1.2719 | 1.4645 | 15.14
ResNet-18 [54] | 65.23 |  GradCAM [6] | 0.7232 | 0.8300 | 14.77 | 1.4019 | 1.7366 | 26.51
GradCAM++ [21] | 0.7110 | 0.8189 | 15.18 | 1.2482 | 1.4113 | 13.07
Guided Backprop [9] | 0.0322 | 0.0446 | 38.28 | 0.4056 | 0.4309 | 6.26
SmoothGrad [10] | 0.3626 | 0.4620 | 27.43 | 0.9306 | 1.0755 | 15.57
VGG-16 [14] | 63.57 |  GradCAM [6] | 0.3939 | 0.5234 | 32.89 | 0.7877 | 0.9536 | 21.06
GradCAM++ [21] | 0.3905 | 0.5474 | 40.17 | 0.7923 | 0.9653 | 21.83
Guided Backprop [9] | 0.0328 | 0.0493 | 50.31 | 0.3683 | 0.4004 | 8.73
SmoothGrad [10] | 0.3083 | 0.4493 | 45.73 | 0.8736 | 1.0961 | 25.47
DenseNet-169 | 67.83 |  GradCAM [6] | 0.5892 | 0.7860 | 33.39 | 1.1521 | 1.4572 | 26.48
GradCAM++ [21] | 0.5950 | 0.7645 | 28.48 | 1.1551 | 1.3261 | 14.8
Guided Backprop [9] | 0.031 | 0.0522 | 68.54 | 0.3959 | 0.4305 | 8.74
SmoothGrad [10] | 0.2735 | 0.3962 | 44.87 | 0.7937 | 0.9870 | 24.35
Swin Base [15] | 80.61 |  GradCAM [6] | 0.7151 | 0.7524 | 5.22 | 0.8655 | 1.0123 | 16.96
GradCAM++ [21] | 0.2111 | 0.1806 | -14.44 | 0.9608 | 1.0809 | 12.50
Guided Backprop [9] | 0.0606 | 0.0524 | -13.49 | 0.2556 | 0.2708 | 5.93
SmoothGrad [10] | 0.3394 | 0.3723 | 9.69 | 1.1868 | 1.2606 | 6.22

In this section, we show both qualitative and quantitative results for VOICE
uncertainty for the application of image recognition. We use four popular
gradient based explanatory techniques including GradCAM [6], GradCAM++ [21],
Guided Backpropagation [9], and SmoothGrad [10]. Five commonly used neural
network architectures including AlexNet [52], VGG-16 [14], ResNet-18 [54],
DenseNet-169 [55], and SqueezeNet [53] and a state-of-the-art Swin Transformer
model [15] are used to showcase VOICE uncertainty. All models are pretrained
and are publicly available from the PyTorch Torchvision library. The
qualitative and quantitative results on clean data are presented in Sections
V-A and V-B respectively while the results on challenging noisy data are
presented in Sections V-C and V-D respectively.

###  V-A Qualitative results on clean data

We show qualitative results in Fig. 4 for a pretrained VGG-16 architecture
[14]. GradCAM [6] and GradCAM++ [21] explanations are extracted from the last
convolution layer to maximize the semantic content within the activations
while preserving spatial locality. Guided Backpropagation [9] and SmoothGrad
[10] are extracted by backpropagating the gradients to the input pixel space.
Hence, GradCAM and GradCAM++ highlight semantic structures directly while
Guided Backpropagation and SmoothGrad highlight pixels that collectively form
the structure. In this paper, we choose these two classes of semantic and
pixel explanations to illustrate our proposed uncertainty quantification.

Similar to Fig. 1, we differentiate between whether the network classified an
image correctly or incorrectly. Note that all four considered explanations are
_post hoc_ and hence, we have access to their classifications. We randomly
choose three correctly classified images in the ImageNet [56] validation set.
These are a mosque, a bald eagle, and a motor scooter in Figs. 4(a), (b), and
(c) respectively. In Fig. 4(a), the explanations provided by GradCAM and
GradCAM++ both highlight the minaret. Additionally, GradCAM++ highlights the
dome. The uncertainties of both GradCAM and GradCAM++ centre around their
explanations with minor overlaps. This is more apparent in Fig. 4(b), where
the base of the beak of the bald eagle is a strong indicator of its class
according to GradCAM. The uncertainty in GradCAM is noticeably centred around
the base of the beak. This shows that GradCAM is certain in its explanation.
However, this is not true for GradCAM++. A similar analysis holds for Fig.
4(c) where the overlap between the explanation and its uncertainty is higher
for GradCAM++ than GradCAM. In Section V-B, we show that this observation can
be generalized across multiple images. The results for Guided Backpropagation
and SmoothGrad are less clear than their semantic counterparts due to the
nature of pixelwise explanations. Guided backpropagation provides only a
limited number of pixel highlights that make variance across that index across
multiple contrastive explanations low. The uncertainties are highlighted in
red. While the uncertainties are visible in SmoothGrad, it is more diffused
compared to the uncertainties in GradCAM and GradCAM++. Also, more emphasis is
placed on edges, which is a natural artifact of obtaining gradients from the
input layer. Hence, by qualitatively observing visualizations of the
uncertainties for Guided Backpropagation and SmoothGrad, no conclusions
regarding the overlap between explanations and their uncertainties can be
drawn. Quantitative evaluation in Section V-B provides better results.

In Figs. 4(d) and (e), we show explanations and their uncertainties when the
network prediction is incorrect. The network predicts dogsled for Fig. 4(d)
and barbershop for Fig. 4(e). In Fig. 4(d), there is a noticeable large
overlap between the explanations and uncertainties of GradCAM and GradCAM++.
This indicates that the region that the network uses for classification is the
same region it is uncertain about. This observation holds for SmoothGrad as
well. Moreover, the uncertainties are dispersed in Figs. 4(d) and (e),
compared to the correctly classified images.

###  V-B Quantitative results on clean data

In this section, we present a quantitative analysis to validate the overlap
and dispersion hypotheses. We randomly select $500$ images from the ImageNet
[56] validation dataset and pass them through $6$ pretrained neural network
architectures. The resulting predictions are analyzed based on being correct
or incorrect in Table I. IoU and SNR metrics, differentiated between correct
and incorrect predictions are shown. The third column in both IoU and SNR
metrics titled $\%$ Difference in Table I is calculated as $\frac{(Wrong-
Correct)}{Correct}\times 100$. Hence, it shows the $\%$ increase of IoU on
wrong predictions from the correct predictions. This varies between different
networks. For all networks except Swin Transformer, the IoU between each
prediction’s explanation and the uncertainty associated with that explanation
is greater when the network is incorrect. Given that the network predicts
incorrectly, it is more uncertain about the regions it is using to predict
than if it were to have predicted correctly. This shows that the trust placed
by the explanatory technique in its explanation is lesser since it is using
the same features to explain its decision that it is uncertain about.
Similarly, the SNR or dispersion metric is higher when the network predictions
are incorrect across all networks and explanatory techniques.

Effect of threshold $t$ on Swin transformer In Table II, we analyze the effect
of $t$ on the IoU metric on Swin Transformer. In Table I, $t$ is kept
consistent at $0.1$ among all the networks. Increasing the value of $t$
increases the $\%$ difference in overlap between the correct and wrong
predictions. This behavior is seen among both GradCAM and GradCAM++. Note that
even at $0.1$, all CNN architectures in Table I show a positive $\%$
difference. On the other hand, Swin transformer has $-14.44\%$ difference
indicating that the overlap on the correct predictions is higher than the
incorrect predictions. This is because the explanations for GradCAM and
GradCAM++ are extracted from a $7\times 7$ kernel which is interpolated to
create a $224\times 224$ image. This adds spurious overlaps between the
explanations and uncertainties. Increasing threshold $t$ eliminates the
spurious overlaps, thereby increasing the $\%$ difference as depicted in Table
II.

Figure 5: Explanations and uncertainty of a Spoonbill perturbed with AWGN at noise power levels of (a) 0, (b) 450, and (c) 11000, respectively. At 11000, the network incorrectly predicts a coral reef. (d) provides normalized log-likelihood, IoU and SNR metrics across $15$ noise levels. Shading indicates variance across three runs. Background green indicates correct prediction and red indicates incorrect prediction. TABLE II: Results of IoU on Swin Transformer with a change in threshold $t$. Threshold | GradCAM | GradCAM++
---|---|---
$t$ | Correct | Wrong | % Difference | Correct | Wrong | % Difference
0.1 | 0.7151 | 0.7524 | 5.22 | 0.2111 | 0.1806 | -14.44
0.3 | 0.3604 | 0.3756 | 4.21 | 0.1472 | 0.1361 | -7.55
0.4 | 0.1970 | 0.2103 | 6.77 | 0.0994 | 0.1033 | 3.90
0.5 | 0.0930 | 0.1055 | 13.44 | 0.0592 | 0.0737 | 24.52
0.6 | 0.0384 | 0.0485 | 26.29 | 0.0347 | 0.0531 | 53.15
0.7 | 0.0830 | 0.1206 | 45.34 | 0.0210 | 0.0364 | 73.04

###  V-C Qualitative results on challenging data

In this subsection, we analyze the qualitative and quantitative results on a
single image of a spoonbill (ImageNet class $129$) across $15$ increasing
levels of zero-mean Additive White Gaussian Noise (AWGN). Level 0 indicates
the original image. Increasing levels indicate increase in noise power levels
with Level 15 being the maximum considered noise. Note that after level 7, we
increase the power level from $450$ to $7000$ so as to depict the trends
between correct and incorrect predictions. We present these results for VGG-16
[14] in Fig. 5.

Figs. 5(a), (b), and (c) show the explanations from GradCAM, GradCAM++, and
SmoothGrad along with their VOICE uncertainties respectively. Row (a) in Fig.
5 are results on the original spoonbill image, row (b) are results on level 7
AWGN spoonbill, and row (c) are results on level 15 AWGN spoonbill. Row (d) in
Fig. 5 shows the normalized quantitative metrics on IoU and SNR along with
normalized log-likelihood on the y-axis, across the noise levels on the
x-axis. Rows (a) and (b) spoonbills are correctly predicted as spoonbill by
the VGG-16 network while the level 15 AWGN spoonbill in Row (c) is incorrectly
predicted as coral reef. In row (d), the green background indicates that the
network predicts the class correctly across levels 0 - 7, while levels 8 - 15
are incorrectly predicted and are represented by a red background. To obtain
row (d) in Fig. 5, we run 3 separate runs of the experiment at every noise
level for each explanation of GradCAM, GradCAM++, and SmoothGrad. The
epistemic uncertainty, represented by log-likelihood, follows the expected
trajectory of gradual decrease with increase in noise levels. The steep fall
between levels 7 and 8 is due to a large increase in noise power from $450$ to
$7000$. Note that at lower levels of noise, the SNR and IoU of VOICE among all
three explanations is lower. In SmoothGrad, the increase in IoU can be
explained by looking at the VOICE explanations themselves. In row (b), the
SmoothGrad VOICE is more prominent than row (a). Hence there is more overlap
with the SmoothGrad explanation. In GradCAM++, the explanation and VOICE
overlap remains consistent between rows (a) and (b). However, in GradCAM, the
uncertainty _decreases_ with an increase in noise levels. This is reflected in
a gradual decline in IoU in GradCAM row (d) when the background is green.
Moreover, the variance across the $3$ runs is higher in IoU and SNR as
compared to log-likelihood. This is because during each of the 3 experiments
at every noise level, the added AWGN is random so that the predictions $P$
from the network are different across the $3$ runs. Hence, the VOICE
uncertainty is asking _’Why $P_{r}$, rather than Q?’_ where $P_{r}$ is the
prediction at run $r$ with $P_{1}\neq P_{2}\neq P_{3}$. Hence, the
explanations and their corresponding uncertainties are different across the
$3$ runs which is reflected in the higher variance. Note that in row (d) of
Fig 5, the values are normalized between $0$ and $1$.

Figure 6: Quantitative results on three challenges from CIFAR-10C data on five
progressively increasing challenge levels on the x-axis. First two columns are
derived using GradCAM explanations and last two are derived from GradCAM++.

###  V-D Quantitative results on challenging data

We use CIFAR-10C [57] as the challenging dataset to benchmark the VOICE
uncertainty metrics. CIFAR-10C provides $19$ environmental, acquisition, and
noise challenges on CIFAR-10 data. The $10000$ images from CIFAR-10 testset
are perturbed using $5$ progressively increasing levels of perturbations.
Hence, each challenge has $50000$ images to benchmark VOICE uncertainty
metrics. We obtain results on all $19$ challenges but show only three
challenges in Fig. 6 due to space constraints. These three challenges
including gaussian blur, contrast, and JPEG compression are chosen to showcase
the diverse behaviors between IoU, SNR and log-likelihood. Fig. 6 consists of
plots where x-axis depicts the challenge levels and y-axis plots normalized
log-likelihood and VOICE metrics. The metrics are plotted for GradCAM and
GradCAM++ explanations. Plots from each challenge and each explanation are
divided based on correct or incorrect predictions. On contrast and gaussian
blur challenges, the SNR and IoU VOICE metrics exhibit similar behavior as
log-likelihood among both GradCAM and GradCAM++ explanations. However, in JPEG
compression the behavior on challenge levels $3$ and $4$ between the three
metrics deviates under incorrect predictions. This is observed in both GradCAM
and GradCAM++ explanations. Among all $19$ challenges on CIFAR-10C, JPEG
compression shows the largest deviation. Note that log-likelihood requires
ground truth to quantify uncertainty while the proposed metrics do not.

TABLE III: Comparing explanatory techniques using AUC scores on CIFAR-10C dataset. The best performing explanation for every challenge and metric is bolded. Challenges | Explanation | IoU | SNR
---|---|---|---
Gaussian Blur | GradCAM | 0.24 | 0.31
GradCAM++ | 0.27 | 0.29
Guided Backprop | 0.46 | 0.19
SmoothGrad | 0.32 | 0.49
Contrast | GradCAM | 0.17 | 0.17
GradCAM++ | 0.19 | 0.16
Guided Backprop | 0.22 | 0.29
SmoothGrad | 0.42 | 0.23
JPEG Compression | GradCAM | 0.30 | 0.33
GradCAM++ | 0.31 | 0.39
Guided Backprop | 0.27 | 0.41
SmoothGrad | 0.27 | 0.47

###  V-E Comparing explanatory techniques

Table I provides IoU and SNR quantification of VOICE that validates the
overlap and dispersion hypotheses. However, these results are insufficient to
compare between explanations. For instance, Guided Backpropagation has the
highest difference between correct and incorrect prediction IoU across all
architectures, except Swin transformer. An incorrect conclusion from the
preceding statement is that guided backpropagation is the best method (as
compared against GradCAM, GradCAM++, and SmoothGrad) since it has less overlap
between the explanation and its uncertainty. This is an incorrect conclusion
since the values of IoU is not comparable between explanatory methodologies.
On AlexNet, the IoU of Guided Backpropagation is $0.047$ while on the other
explanations, IoU is around $0.5$. Visually, the low IoU value of Guided
Backpropagation uncertainty manifests itself as mostly black regions in Fig. 4
with the uncertain regions highlighted within the red box to make them
apparent.

Note that the variability in metric range is a common issue within uncertainty
quantification literature. For instance, log-likelihood measure is unbounded
and differs based on network, data, and predictions. Since there is no ground
truth, uncertainty evaluation strategies utilize a set of samples rather than
individual predictions for a pseudo-classification task. In [58], the authors
utilize their proposed uncertainty metric to classify between in-distribution
and out-of-distribution data. Higher the classification accuracy, better the
uncertainty metric. In [19], the authors use Area Under Accuracy Curve (AUAC)
on CIFAR-10C dataset to showcase that their uncertainty metric is robust
against increasing level of challenges. Ideally, AUAC must remain high, even
when a network is presented with challenging data. We follow the same
procedure as [19] and provide the Area Under IoU and Area Under SNR Curves for
each of the four explanations on three challenges used in Section V-D. From
Fig. 6, the IoU and SNR VOICE quantification must ideally be zero across
increasing level of challenges. An explanation with its AUC closer to zero is
a better explanation. We calculate the area under both IoU and SNR measures
and present them in Table III. The best performing explanation for every
challenge and metric is bolded. Note that on average, GradCAM has both IoU and
SNR values closer to zero compared to the other explanations. Our proposed
explanation overcomes the masking limitations in existing explanatory
evaluation techniques to objectively quantify subjective explanations.

##  VI Discussion

Figure 7: Visualization of explanations and their corresponding uncertainties
under (a) normal, (b) out-of-distribution, and (c) adversarial data.

###  VI-A Takeaways

#### Non-gradient based explanation uncertainty

The current work of obtaining uncertainty of explanations is applicable to
only gradient-based attribution methods. The proposed VOICE methodology is a
white-box method, i.e. it requires access to all the network parameters and
its gradients to induce uncertainty. While gradient-based methods are a
popular means of visualizing feature attributions, there are other methods
that provide explanations. These include TCAV [11], LIME [13], and Graph-CNN
[59] among others. Future work on obtaining method-based uncertainty must
focus on non-gradient explanatory techniques as well.

#### Objective measures for subjective explanations

The theoretical analysis of predictive uncertainty shows that the current
methodology of objectively evaluating explanations only, partly, reduces
uncertainty. Masking images and decreasing the $V[E(y|S_{x})]$ term ignores
the second term $E[V(y|S_{x})]$ in Eq. 1. A simple way of decreasing
$V[E(y|S_{x})]$ is by having bigger masks. But this invariably leads to a
larger area of uncertainty and hence larger $E[V(y|S_{x})]$. Any objective
evaluation methodology that relies on masking suffers from the same issue.
Hence, either uncertainty must also be reported along with accuracy
evaluations or the proposed area under SNR and IoU curves must be reported.

#### VOICE uncertainty on network-targeted challenges

In this paper, we used challenging data from CIFAR-10C dataset where
perturbations are obtained based on acquisition challenges like contrast, shot
noise, motion blur, environmental challenges like snow, rain, fog, as well as
noise characteristics. These are natural challenges that have shown to affect
the recognition performance of neural networks. However, neural network
specific challenges like adversarial images and out-of-distribution images
that are not present in the training domain also affect the network
performance. We show two such examples in Fig. 7. VGG-16 architecture is used
to obtain all explanations and uncertainties. The out-of-distribution (OOD)
image in Fig. 7(b) is correctly classified despite not being a part of
ImageNet. There is a larger overlap between the explanations and uncertainties
among both explanations in Fig. 7(b) compared to Fig. 7(a). Note that the
prediction is correct for the OOD image unlike in Section V-B where a larger
overlap is observed when the prediction is incorrect. The adversarial image in
Fig. 7(c) is derived using I-FGSM method [23]. In Fig. 7(a), the image is
recognized correctly as a bull-mastiff. In Fig. 7(c), the image is recognized
incorrectly as a boxer. However, both GradCAM and GradCAM++ highlight the
features of the dog in Fig. 7(a) and Fig. 7(c) to explain different
predictions. In contrast, the uncertainties of GradCAM and GradCAM++ in Fig.
7(c) highlight the face of the cat while the uncertainties of GradCAM and
GradCAM++ in Fig. 7(a) highlight the features around the explanation. Hence,
there is a spatial difference in occurrence of uncertainties while there is no
change in the explanations themselves. This indicates that while explanations
by themselves may not be sufficient to detect adversarial images, VOICE
uncertainty may be capable. However, the overlap metric is insufficient here
and future work must focus on new VOICE measures on adversarial and OOD image
detection.

###  VI-B Conclusion

In this paper, we show that all visual explanatory techniques have some
inherent uncertainty associated with them. We provide a theoretical analysis
based on predictive uncertainty and observe that existing evaluation
techniques for explanations only partially reduce predictive uncertainty. We
propose a methodology based on contrastive explanations to visualize the
remaining predictive uncertainty. In doing so, we observe that under incorrect
predictions, neural networks are often uncertain about the same features that
they use to make predictions. Moreover, the dispersion of uncertainty as well
as the overlap of an explanation and its uncertainty can be used to quantify
the explanation’s uncertainty. These quantities exhibit similar behavior as
log-likelihood, a measure of epistemic uncertainty.

## References

  * [1] G. AlRegib and M. Prabhushankar, “Explanatory paradigms in neural networks: Towards relevant and contextual explanations,” _IEEE Signal Processing Magazine_ , vol. 39, no. 4, pp. 59–72, 2022.
  * [2] Y. Hu, Z. Long, A. Sundaresan, M. Alfarraj, G. AlRegib, S. Park, and S. Jayaraman, “Fabric surface characterization: Assessment of deep learning-based texture representations using a challenging dataset,” _The Journal of The Textile Institute_ , vol. 112, no. 2, pp. 293–305, 2021\.
  * [3] M. A. Shafiq, M. Prabhushankar, H. Di, and G. AlRegib, “Towards understanding common features between natural and seismic images,” in _SEG International Exposition and Annual Meeting_. SEG, 2018, pp. SEG–2018.
  * [4] R. Poplin, A. V. Varadarajan, K. Blumer, Y. Liu, M. V. McConnell, G. S. Corrado, L. Peng, and D. R. Webster, “Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning,” _Nature Biomedical Engineering_ , vol. 2, no. 3, pp. 158–164, 2018.
  * [5] K. Kokilepersaud, S. T. Corona, M. Prabhushankar, G. AlRegib, and C. Wykoff, “Clinically labeled contrastive learning for oct biomarker classification,” _IEEE Journal of Biomedical and Health Informatics_ , 2023.
  * [6] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [7] M. Prabhushankar and G. AlRegib, “Extracting causal visual features for limited label classification,” in _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021, pp. 3697–3701.
  * [8] V. Petsiuk, A. Das, and K. Saenko, “Rise: Randomized input sampling for explanation of black-box models,” _arXiv preprint arXiv:1806.07421_ , 2018\.
  * [9] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” _arXiv preprint arXiv:1412.6806_ , 2014.
  * [10] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” _arXiv preprint arXiv:1706.03825_ , 2017\.
  * [11] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas _et al._ , “Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav),” in _International conference on machine learning_. PMLR, 2018, pp. 2668–2677.
  * [12] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2921–2929.
  * [13] M. T. Ribeiro, S. Singh, and C. Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [14] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [15] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 012–10 022.
  * [16] G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, “Distorted representation space characterization through backpropagated gradients,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 2651–2655.
  * [17] G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, “Backpropagated gradient representations for anomaly detection,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_. Springer, 2020, pp. 206–226.
  * [18] G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib, “Novelty detection through model-based characterization of neural networks,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 3179–3183.
  * [19] J. Lee, C. Lehman, M. Prabhushankar, and G. AlRegib, “Probing the purview of neural networks via gradient analysis,” _IEEE Access_ , vol. 11, pp. 32 716–32 732, 2023.
  * [20] K. Kokilepersaud, M. Prabhushankar, G. AlRegib, S. T. Corona, and C. Wykoff, “Gradient-based severity labeling for biomarker classification in oct,” in _2022 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2022, pp. 3416–3420.
  * [21] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2018, pp. 839–847.
  * [22] M. Prabhushankar, G. Kwon, D. Temel, and G. AlRegib, “Contrastive explanations in neural networks,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 3289–3293.
  * [23] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [24] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1321–1330.
  * [25] J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher _et al._ , “A survey of uncertainty in deep neural networks,” _Artificial Intelligence Review_ , vol. 56, no. Suppl 1, pp. 1513–1589, 2023.
  * [26] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [27] Y. Gal _et al._ , “Uncertainty in deep learning.”
  * [28] L. Smith and Y. Gal, “Understanding measures of uncertainty for adversarial example detection,” _arXiv preprint arXiv:1803.08533_ , 2018.
  * [29] M. Prabhushankar and G. AlRegib, “Introspective learning: A two-stage approach for inference in neural networks,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 12 126–12 140, 2022.
  * [30] M. Prabhushankar and G. AlRegib, “Contrastive reasoning in neural networks,” _arXiv preprint arXiv:2103.12329_ , 2021.
  * [31] S. R. Lele, “How should we quantify uncertainty in statistical inference?” _Frontiers in Ecology and Evolution_ , vol. 8, p. 35, 2020.
  * [32] R. Benkert, M. Prabhushankar, G. AlRegib, A. Pacharmi, and E. Corona, “Gaussian switch sampling: A second order approach to active learning,” _IEEE Transactions on Artificial Intelligence_ , pp. 1–14, 2023.
  * [33] R. Benkert, M. Prabhushankar, and G. AlRegib, “Reliable uncertainty estimation for seismic interpretation with prediction switches,” in _Second International Meeting for Applied Geoscience & Energy_. Society of Exploration Geophysicists and American Association of Petroleum …, 2022, pp. 1740–1744.
  * [34] C. Zhou, M. Prabhushankar, and G. AlRegib, “On the ramifications of human label uncertainty,” _arXiv preprint arXiv:2211.05871_ , 2022.
  * [35] J. Lee, M. Prabhushankar, and G. AlRegib, “Gradient-based adversarial and out-of-distribution detection,” _arXiv preprint arXiv:2206.08255_ , 2022\.
  * [36] B. N. Patro, M. Lunayach, S. Patel, and V. P. Namboodiri, “U-cam: Visual explanation using uncertainty based class activation maps,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7444–7453.
  * [37] P. Chowdhury, M. Prabhushankar, and G. AlRegib, “Explaining explainers: Necessity and sufficiency in tabular data,” in _NeurIPS 2023 Second Table Representation Learning Workshop_ , 2023.
  * [38] D. Slack, A. Hilgard, S. Singh, and H. Lakkaraju, “Reliable post hoc explanations: Modeling uncertainty in explainability,” pp. 9391–9404, 2021.
  * [39] W. A. Bomberger, “Disagreement as a measure of uncertainty,” _Journal of Money, Credit and Banking_ , vol. 28, no. 3, pp. 381–392, 1996.
  * [40] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [41] M. D. McKay, “Evaluating prediction uncertainty,” Nuclear Regulatory Commission, Tech. Rep., 1995.
  * [42] J. M. Köhler, M. Autenrieth, and W. H. Beluch, “Uncertainty based detection and relabeling of noisy image labels.” in _CVPR Workshops_ , 2019, pp. 33–37.
  * [43] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [44] W. Nie, Y. Zhang, and A. Patel, “A theoretical explanation for perplexing behaviors of backpropagation-based visualizations,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 3809–3818.
  * [45] M. Prabhushankar and G. AlRegib, “Extracting causal visual features for limited label classification,” in _IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021.
  * [46] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” _nature_ , vol. 323, no. 6088, pp. 533–536, 1986.
  * [47] Y. Sun, M. Prabhushankar, and G. AlRegib, “Implicit saliency in deep neural networks,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 2915–2919.
  * [48] M. Prabhushankar and G. AlRegib, “Stochastic surprisal: An inferential measurement of free energy in neural networks,” _Frontiers in Neuroscience_ , vol. 17, p. 926418, 2023.
  * [49] J. Lee and G. AlRegib, “Gradients as a measure of uncertainty in neural networks,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 2416–2420.
  * [50] D. Lopez-Paz, R. Nishihara, S. Chintala, B. Scholkopf, and L. Bottou, “Discovering causal signals in images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6979–6987.
  * [51] D. Temel, M. Prabhushankar, and G. AlRegib, “Unique: Unsupervised image quality estimation,” _IEEE signal processing letters_ , vol. 23, no. 10, pp. 1414–1418, 2016.
  * [52] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [53] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size,” _arXiv preprint arXiv:1602.07360_ , 2016.
  * [54] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [55] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [56] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [57] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” _arXiv preprint arXiv:1903.12261_ , 2019.
  * [58] R. Benkert, M. Prabhushankar, and G. AlRegib, “Transitional uncertainty with intermediate neural gaussian processes.”
  * [59] Q. Zhang, R. Cao, F. Shi, Y. N. Wu, and S.-C. Zhu, “Interpreting cnn knowledge via an explanatory graph,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.

|  Mohit Prabhushankar received his Ph.D. degree in electrical engineering
from the Georgia Institute of Technology (Georgia Tech), Atlanta, Georgia,
30332, USA, in 2021. He is currently a Postdoctoral Research Fellow in the
School of Electrical and Computer Engineering at the Georgia Institute of
Technology in the Omni Lab for Intelligent Visual Engineering and Science
(OLIVES). He is working in the fields of image processing, machine learning,
active learning, healthcare, and robust and explainable AI. He is the
recipient of the Best Paper award at ICIP 2019 and Top Viewed Special Session
Paper Award at ICIP 2020. He is the recipient of the ECE Outstanding Graduate
Teaching Award, the CSIP Research award, and of the Roger P Webb ECE Graduate
Research Assistant Excellence award, all in 2022. He has delivered short
courses and tutorials at IEEE IV’23, ICIP’23, BigData’23, WACV’24 and AAAI’24.
---|---
|  Ghassan AlRegib is currently the John and Marilu McCarty Chair Professor in
the School of Electrical and Computer Engineering at the Georgia Institute of
Technology. In the Omni Lab for Intelligent Visual Engineering and Science
(OLIVES), he and his group work on robust and interpretable machine learning
algorithms, uncertainty and trust, and human in the loop algorithms. The group
has demonstrated their work on a wide range of applications such as Autonomous
Systems, Medical Imaging, and Subsurface Imaging. The group is interested in
advancing the fundamentals as well as the deployment of such systems in real-
world scenarios. He has been issued several U.S. patents and invention
disclosures. He is a Fellow of the IEEE. Prof. AlRegib is active in the IEEE.
He served on the editorial board of several transactions and served as the TPC
Chair for ICIP 2020, ICIP 2024, and GlobalSIP 2014. He was area editor for the
IEEE Signal Processing Magazine. In 2008, he received the ECE Outstanding
Junior Faculty Member Award. In 2017, he received the 2017 Denning Faculty
Award for Global Engagement. He received the 2024 ECE Distinguished Faculty
Achievement Award at Georgia Tech. He and his students received the Best Paper
Award in ICIP 2019 and the 2023 EURASIP Best Paper Award for Image
communication Journal.
---|---


[affiliation=*]ShreyanGanguly [affiliation=*]RoshanNayak
[affiliation=*]RakshithRao [affiliation=*]UjanDeb [affiliation=]PrathoshAP

# AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss
Weighting

###### Abstract

Knowledge distillation, a widely used model compression technique, works on
the basis of transferring knowledge from a cumbersome teacher model to a
lightweight student model. The technique involves jointly optimizing the task
specific and knowledge distillation losses with a weight assigned to them.
Despite these weights playing a crucial role in the performance of the
distillation process, current methods provide equal weight to both losses,
leading to suboptimal performance. In this paper, we propose Adaptive
Knowledge Distillation, a novel technique inspired by curriculum learning to
adaptively weigh the losses at instance level. This technique goes by the
notion that sample difficulty increases with teacher loss. Our method follows
a plug-and-play paradigm that can be applied on top of any task-specific and
distillation objectives. Experiments show that our method performs better than
conventional knowledge distillation method and existing instance-level loss
functions.

###### keywords:

speech recognition, knowledge distillation, curriculum learning

111∗Equal contribution

##  1 Introduction

In recent years automatic speech recognition (ASR) has seen radical
improvements with the advent of deep neural network based models. Inspired by
successful unsupervised pre-training approaches for text [1] and images [2,
3], Schneider et. al. introduced wav2vec [4] which trains a deep convolutional
neural network with a noise contrastive objective on unlabeled data, improving
upon previous baselines by upto 36% [4]. Baevski et. al. [5] further improved
the state-of-the-art by pre-training a transformer encoder with quantized
latent representations and a similar contrastive task. The widespread adoption
of the transformer architecture across a wide range of modalities led to more
transformer based ASR models such as HuBERT [6] and data2vec [7], all of which
continue to push ASR performance to be more and more accurate. Even more
recently, Radford et. al. introduced Whisper [8], a transformer trained on
680,000 hours of labeled data. In addition to matching the state of the art
error rates on clean audio, Whisper far outperforms other models on zero shot,
out of distribution generalization, advancing ASR robustness closer to human
levels [8]. Thus the current landscape in ASR is populated by many robust and
powerful neural network models, which are getting closer and closer to human
speech recognition capabilities.

A common use case of ASR models is real time transcription of speech. However
despite their impressive performance, current neural network based ASR models
are typically deep networks with parameters ranging from hundreds of millions
to several billions [5, 8]. This constrains their usage to devices with high
compute resources. Large neural nets like large transformers also have a high
inference latency, especially on CPU. Inference time for the whisper-medium
model has been reported to be more than 60 secs for 30 sec audio inputs, even
for high end CPUs like the AMD Ryzen 5 5600X
222https://github.com/openai/whisper/discussions/454. This prohibits real time
usage for even the whisper-medium model, let alone larger more performant
models like the whisper-large. This problem is further exacerbated when the
models need to run on an edge device and cannot rely on remote compute over a
network to do the transcription.

Figure 1: A schematic diagram of the proposed knowledge distillation method
with dynamic loss weights based on sample difficulty. The teacher model
produces an output $y^{t}$ and incurs a loss $T_{l}$. The difficulty factor
$d_{f}$ is calculated based on the teacher loss $T_{l}$ and two
hyperparameters ($k$, $t$). The loss weight $\alpha$ for distillation is
calculated using the difficulty factor $d_{f}$. The student model produces an
output $y^{s}$ and incurs a loss $L_{ts}$. The final loss $L_{st}$ is a
combination of the student loss $L_{ts}$ and the distillation loss $L_{kd}$.

Therefore in order to run transformer based ASR models in real time we are
prevented from directly using the large and powerful models. But large
transformer models far outperform their smaller counterparts in terms of
accuracy and robustness [8]. A natural goal then is to attain the performance
of a large model with a smaller and faster model. Model distillation aims to
achieve this objective. Model distillation was formally introduced by Hinton
et. al. [9] where the authors showed that task knowledge possessed by an
ensemble of models can be distilled into a single smaller model, evidenced by
its improvement on the task. In it's simplest case, the student model in
addition to being trained on ground truth data is also asked to match the
output logits of a teacher model, softened by a temperature hyperparameter.
With respect to effective distillation approaches for transformers, Sanh et.
al. introduced DistilBERT [10], a distilled version of the BERT [1] model,
where the authors performed knowledge distillation during pre-training,
reducing model size and inference time by 40% and 60% respectively, while
preserving performance of upto 97% of the teacher BERT model [1]. Vision
transformers [11] also saw successful distillation techniques applied to them
[12].

In speech recognition, Peng et. al. distilled knowledge from a wav2vec2 model
[5] into smaller model with 4.8 times less parameters at the cost of about 7%
performance degradation. Fu et. al. and Kim et. al. further improved
distillation on wav2vec2 by reducing model parameters by upto 91% [13, 14].
Lee et. al. applied knowledge distillation on the HuBERT [6] model to obtain a
model with 23.8% less parameters with minimal performance degradation. The
distillation approaches are a mostly a combination of logit and feature
destillation and their variations. Peng et. al. apply logit distillation with
a feature penalty [15]. Fu et. al. adds a maximum mutual information (MMI)
criterion on the layer representations [13]. To aid in compression, Kim et.
al. share parameters and reuse attention alignments throughout the model [14].

While these different distillation approaches work well there is plenty of
room for improvement. In particular, we notice that the distillation
objectives employed above do not take into account the difficulty of the
training samples, as encountered by the teacher. We show that this information
can be captured from the teacher loss and utilized in order to improve
knowledge distillation to the student model. To this end we propose a novel
method where the distillation objective is dynamically adjusted throughout
training so as to progressively shift knowledge distillation from easier to
more difficult samples. We show consistent performance improvements across
wav2vec2 [5] and whisper [8] models and across 3 different languages.

##  2 Related Works

Knowledge Distillation: Leveraging the powerful generalization capacity of
large cumbersome models to train smaller compact models with the aim of model
compression was first proposed by Buciluǎ et. al. [16]. This technique was
extended in the work by Hinton et. al. [9] under the name of "Knowledge
Distillation" using a new objective function that could be applied to both
labeled and unlabeled sets. A brief description of knowledge distillation is
provided in section 3.

The focus of recent works in this domain has been on improving the
distillation performance by designing various forms of knowledge transfer
schemes. It can be broadly categorized into logit distillation and feature
distillation. In logit distillation, there have been works on choosing optimal
teacher-student network architecture [17], noise-based regularizer for KD
[18], using conditional adversarial network to learn a loss function [19].
Park et. al. [20] proposed relational KD with the aim of transferring
structural knowledge in logits from one model to another. Cho et. al. [21]
provided a interpretation of the original KD, claiming a better and more
accurate teacher does not ensure a more accurate student.
Knowledge Distillation for ASR: There has been works on compressing ASR models
like Wav2Vec [4] by Lee et. al. [22], Peng et. al. [23]. However, for recent
robust ASR models like Whisper [8], there has not been much advancement in the
field. Gandhi et. al. [24] used the vanilla knowledge distillation to distill
a compressed version of a model from its base version. To this end, we try to
present a novel KD-based approach for improving performance of Whisper model.

Curriculum Learning: Originally inspired by the human learning process,
Curriculum Learning was introduced by Bengio et al. [25] as a network training
strategy involving the presentation of samples in a meaningful order. The
recent trend in curriculum learning has predominantly focused on sequencing
samples in increasing order of complexity during training. Castells et al.
[26] introduced a novel confidence-aware loss function for dynamic instance-
wise curriculum learning. Wu et al. [27] expanded on this approach by adopting
a new scheme that separates difficult from incorrect samples within a
curriculum learning framework. Recent literature also explores the application
of curriculum learning in knowledge distillation. Li et al. [28] and Zhu et
al. [29] proposed innovative knowledge distillation methods based on
curriculum learning, progressing from easy to difficult samples using
temperature and sample metrics. However, these studies have not delved into
the exploration of incorporating increasing order of hidden knowledge
difficulty, obtained while distilling, within the framework of curriculum
learning.

Table 1: A comaprison of Ada-Kd against other distillation methods. Whisper-small and Wav2Vec 2.0 XLS-R-1B finetuned on the mentioned datasets is the teacher and Whisper-tiny, Wav2Vec 2.0 XLS-R-300M pretrained as student. The value of hyperparameter $t$ taken here is the mean of teacher losses. Experiments for CV-Chinese and Aishell2 have not been performed for Wav2Vec 2.0 owing to poor teacher performance. CER metric has been used to compare the techniques. Dataset/Model | Finetune | Normal KD [9] | Super Loss [30] | Focal Loss [31] | Annealing KD [32] | Adaptive KD
---|---|---|---|---|---|---
CV-Hindi/Whisper | 25.59 | 26 | 26.53 | 26.84 | 36.01 | 23.27
CV-Chinese/Whisper | 26.5 | 29.2 | 29.21 | 28.68 | 36.37 | 25.20
Aishell2/Whisper | 13.37 | 13.79 | 14.75 | 14.82 | 18.88 | 14.30
CV-Tamil/Whisper | 13.39 | 13.12 | 12.32 | 14.31 | 13.59 | 12.07
CV-Hindi/Wav2Vec2 | 17.52 | 15.04 | 16.63 | 15.78 | 25.59 | 14.26

##  3 Methodology

In certain cases, the standard loss functions fail to perform well on the task
at hand and slight modifications on them will result in enhanced performance.
In this section, we briefly touch upon the technique of knowledge distillation
and then introduce the adaptive knowledge distillation technique aimed at
instance-level adaptive loss weighting for knowledge distillation tasks. In
addition, we try to delve into and elucidate the rationale behind the
algorithm.

###  3.1 Knowledge Distillation

Model compression is a problem in which we aim to get the performance of a
large model on a smaller one. Knowledge distillation is one such widely used
model compression technique in which the large model, called the teacher is
used as a guide while training another model smaller than its size called the
student. In an ideal setting, the student will improve their performance by
mimicking the teacher's behaviour. Hinton et. al. [9], approached this problem
by utilizing Kullback–Leibler divergence as the objective function to match
the output logits of the teacher and student model as shown in Eq(1).

$\displaystyle L_{kd}(y^{s},y^{t})=\tau^{2}KL(y^{s},y^{t})$ (1)

In addition, training can be done using both distillation and task-specific
objectives to ensure that the student does not miss out on the particular
goals of the task. Thus, the distillation objective comes out to be,

$\displaystyle L_{student}=\sum_{i}(1-\alpha)L_{ts}(y_{i},x_{i}|\theta)+\alpha
L_{kd}(y^{s}_{i},y^{t}_{i})$ (2)

Here, the parameter $\alpha$ governs the significance of the distillation
objective $L_{kd}$ in the training process. Moreover, the task-specific
objective, denoted as $L_{ts}$, is assigned an importance level such that the
combined importance of both objectives sum to one.

###  3.2 Adaptive Knowledge Distillation

The vanilla knowledge distillation method assigns equal weightage to task-
specific and knowledge distillation losses. This does not capture the varying
difficulty involved in transferring knowledge from teacher to student for
different samples. In the past, instance-specific techniques [31] [30] have
demonstrated promising results across a wide range of tasks. Inspired by
curriculum learning [25], Adaptive Knowledge Distillation dynamically weighs
the objectives for each instance based on its difficulty and the current
training stage. We determine the difficulty of a sample based on the teacher
loss, where the difficulty rises proportionally with it. We distill the
knowledge from easier samples first, gradually increasing the distillation
weightage for hard samples over the course of training. Moreover, we noticed
that simultaneously decreasing the loss weights for easy samples assists in
achieving optimal results.

$\displaystyle\alpha=e^{\frac{-1}{\sqrt{d_{f}}}}$ (3)

To achieve this, we devised an equation (3) that calculates the loss weight
for distillation for each instance in a batch of 16. Where d is the difficulty
factor and is calculated by the equation,

$\displaystyle d_{f}=e^{-k(x-t)}$ (4)

Here $d_{f}$ controlled by two hyperparameters, namely $k$ and $t$. The
optimal values of these hyper-parameters depend on the distribution of the
teacher losses on the training set used for distillation. The details of the
values of these parameters have been discussed in section 4. Ideally, $t$ can
be the mean of teacher losses as shown below.

$\displaystyle\text{Where},t=\frac{1}{N}\sum_{i=1}^{N}L_{ts}(y_{i},x_{i}|\theta)$
(5)

Where N denotes the number of samples in the training dataset. While $k$
dictates the importance of distillation loss for each instance and is linearly
decreased during the training. As a deliberate design choice, we set the
initial value, $k_{+}$, such that the distillation loss weight for the sample
with the highest teacher loss is a minimal value, specifically 0.1.
Furthermore, $k_{-}$ is to be tuned. We conducted extensive experiments to
find the optimal value for $k_{-}$. Figure 1 shows the alphas for samples
based on teacher loss at two extreme values of $k$, i.e. $k_{+}$ and $k_{-}$
and also at $k=0$.

[-1.2em] Algorithm Adaptive Knowledge Distillation

Required: Training data $X$, Validation data $V$, Student model initialization
${\theta}_{s}$, Teacher model ${\theta}_{t}$, Teacher losses $L_{teacher}$,
task-specific loss $L_{ts}$, distillation loss $L_{kd}$, Range of
hyperparameter $K$, Iteration steps $S$, learning rate $\eta$.

  1. 1:

Initialize student model parameters ${\theta_{s}}$

  2. 2:

Compute $L_{ts}(x,y|\theta_{t})$ for all $(x,y)\in X$ and choose $t$, $k_{+}$,
and $k_{-}$.

  3. 3:

for $s\in\\{0,1,\ldots,S\\}$ do

  4. 4:

$x^{\text{samples}},y^{\text{samples}}\xleftarrow{}MiniBatchSampler(X)$

  5. 5:

Compute $\alpha$ for all $x^{\text{samples}}$ using Eq. (3).

  6. 6:

Form $L_{student}$ as given by Eq. (2)

  7. 7:

Update $\theta_{s}\xrightarrow{}\theta_{s}-\eta\nabla L_{student}$

  8. 8:

end for

###  3.3 Rationale behind this algorithm

The soft probabilities from the teacher model play a crucial role in imparting
insight into class relationships to the student. In instances where a robust
teacher model consistently produces accurate predictions, a very small number
of samples contribute to knowledge distillation. Furthermore, the vanilla
knowledge distillation method assigns equal importance to all samples, failing
to put emphasis on the harder ones to leverage the information about class
relations. In contrast, our proposed method adopts a nuanced strategy by
prioritizing the distillation of easier samples initially and progressively
shifting focus to more challenging ones. This strategic shift is motivated by
the understanding that the harder samples possess rich hidden knowledge, which
conveys intricate class relationships, making them crucial for effective
distillation once the knowledge from the easier samples has been distilled.

##  4 Experiments and Results

###  4.1 Experimental setup

Toolkits. Our experiments were implemented with the PyTorch [33] and
TorchAudio [34] deep learning frameworks. Pre-trained models were downloaded
from HuggingFace [35] repository.
Datasets. Common Voice 11 and AiShell2 datasets were used for the experiments.
Multiple languages were used, including Hindi, Tamil, and Mandarin, to test
the robustness of our approach. AiShell2, an open source 1000 hours dataset
that was split in a ratio of 80-20% for train and test sets, respectively.

Table 2: Datasets used and amount of train and test hours of validated data

Dataset | Number of samples | Train hours | Test hours
---|---|---|---
CV-Hindi | 7.3K | 5 | 4
CV-Tamil | 53k | 95 | 9
CV-Chinese | 40k | 43 | 7
Aishell2 | 250k | 800 | 200

Model. We have used the variants of Whisper [8] for our experiments, with
Whisper-small as teacher and Whisper-tiny as student having 244 and 39 million
parameters respectively. To verify the generalizability of our approach we
further experimented on Wav2vec 2.0 [5], with Wav2vec 2.0 XLS-R 1B as the
teacher and Wav2vec 2.0 XLS-R 300M as the student.
Training. The models were trained on Nvidia A100 GPU (80GB) with batch size of
16, learning rate of 1e-4 and Adam with the default initializations as
optimizer. Cross-entropy was used as task specific loss and Kullback–Leibler
divergence as distillation loss.

###  4.2 Baselines

We compare our methods with the existing instance-specific adaptive loss
functions. Focal loss [31] modifies the standard loss function in a way that
reduces the loss of easy samples and, conversely, amplifies the same for hard
samples. Thus, forcing the model to focus more on hard samples during the
training. In contrast, super-loss [30] calculates a confidence score of the
model on each sample and downweights the contribution of samples to the loss
with minimal confidence. Furthermore, annealing knowledge distillation [32] is
aimed at progressively aligning the student model with the annealed soft
probabilities of the teacher model during training. This is achieved by the
introduction of a dynamic temperature term applied to the teacher model's soft
probabilities.

###  4.3 Results

The results of the comparative study between our approach and the baseline
techniques with a consistent experimental setup are presented in the Table 1.
We have set the value of hyper-parameter $t$ to be the mean of teacher losses
across all the experiments. It can be observed that the proposed technique
outperforms both the language-specific fine-tuned model and vanilla knowledge
distillation approaches across languages - solidifying its status to be a
functional model compression technique. In case of dataset Aishell 2, the
performance of the fine-tuned model and vanilla distillation slightly
outperforms the proposed technique, implying that with a very large data set,
distillation methods cannot improve compared to fine-tuning. We compare our
method against other baseline techniques. Super-loss, focal-loss, and
annealing KD. For Annealing KD, we experimented with different $\tau$
hyperparameters in their formula to get the lowest CER. $\tau=4$ was the
optimal choice for CV-Hindi whereas for other languages, $\tau=7$ performed
the best for both the models. In terms of CER, it can be noted that our method
outperforms these baselines considerably with the CER percentage 23. 27%, 25.
20%, 14. 30%, 12. 07% in the CV-Hindi, CV-Chinese, AiShell2, and CV-Tamil
datasets, respectively. We have used $k$ ranges of 15 to -10 for Hindi, 8 to
-8 for other whisper experiments. While the $k$ range for Wav2Vec 2.0 is
observed to be very small of 0.024 to -0.015.

###  4.4 Ablation Study

In this study, we try to study how changing the hyperparameter $t$ will affect
the performance of the distillation. In Table 1 we had set $t$ to be the mean
of teacher losses. We now run ablations by setting $t$ to the 25th, 50th, and
75th percentiles of the teacher losses and compare them against the mean.

We see that for CV-Hindi optimal results are obtained when $t$ is set to be
the mean of the teacher losses. For CV-Chinese and CV-Tamil the mean does not
come out to be the optimal choice. Even lower CERs are obtained for different
$t$. These values perform close with an average deviation of 0.33 and 0.43 CER
respectively from the CER of $t_{mean}$. This shows that, while $t_{mean}$
could be a good choice, depending on the distribution of the teacher losses,
other values of $t$ could give optimal results.

##  5 Conclusion

In this work, we introduced a novel distillation technique to adaptively weigh
the importance of losses at the instance level depending on teacher model
performance. Inspired by curriculum learning, our method gradually transfers
the relation between classes from the teacher to the student. The easy-to-use
equation can be utilized on top of any task-specific and distillation loss
function. Experimental results demonstrate that our method performs
consistently better than existing instance-level loss functions on data sets
of various sizes. We observe that our approach, like other distillation
methods, falls short of standard fine-tuning if the dataset size is too large,
as in case of AiShell2. Another potential shortcoming of our current approach
is that $t$ and $k$ need to be tuned as hyperparameters. As future work we
will look to address this by making these values learnable. We will also look
to integrate this distillation technique with other efficiency approaches such
as pruning and quantization.

## References

  * [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' _arXiv, abs/1810.04805_ , 2018.
  * [2] C. Doersch, A. Gupta, and A. A. Efros, ``Unsupervised visual representation learning by context prediction,'' _In Proc. of ICCV_ , 2015.
  * [3] O. J. Henaff, A. Razavi, C. Doersch, S. M. A. Eslami, and A. van den Oord, ``Data-efficient image recognition with contrastive predictive coding,'' _arXiv, abs/1905.09272_ , 2019.
  * [4] S. Schneider, A. Baevski, R. Collobert, and M. Auli, ``wav2vec: Unsupervised pre-training for speech recognition,'' _arXiv, abs/1904.05862_ , 2019.
  * [5] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _arXiv, abs/2006.11477_ , 2020.
  * [6] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _arXiv, abs/2106.07447_ , 2021.
  * [7] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, ``data2vec: A general framework for self-supervised learning in speech, vision and language,'' _arXiv, abs/2202.03555_ , 2022.
  * [8] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' _arXiv, abs/2212.04356_ , 2022.
  * [9] G. Hinton, O. Vinyals, and J. Dean, ``Distilling the knowledge in a neural network,'' _arXiv preprint arXiv:1503.02531_ , 2015.
  * [10] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, ``Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,'' _arXiv, abs/1910.01108_ , 2019.
  * [11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, ``An image is worth 16x16 words: Transformers for image recognition at scale,'' _arXiv, abs/2010.11929_ , 2020.
  * [12] K. Wu, J. Zhang, M. L. Houwen Peng, B. Xiao, J. Fu, and L. Yuan, ``Tinyvit: Fast pretraining distillation for small vision transformers,'' _arXiv, abs/2207.10666_ , 2022.
  * [13] Y. Fu, Y. Kang, S. Cao, and L. Ma, ``Distillw2v2: A small and streaming wav2vec 2.0 based asr model,'' _arXiv, abs/2303.09278_ , 2023.
  * [14] D.-H. Kim, J.-H. Lee, J.-H. Mo, and J.-H. Chang, ``W2v2-light: A lightweight version of wav2vec 2.0 for automatic speech recognition,'' in _Proc. Interspeech 2022_ , 2022, pp. 3038–3042.
  * [15] Z. Peng, A. Budhkar, I. Tuil, J. Levy, P. Sobhani, R. Cohen, and J. Nassour, ``Shrinking bigfoot: Reducing wav2vec 2.0 footprint,'' _arXiv, abs/2103.15760_ , 2021.
  * [16] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, ``Model compression,'' in _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2006, pp. 535–541.
  * [17] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, ``Fitnets: Hints for thin deep nets,'' _arXiv preprint arXiv:1412.6550_ , 2014.
  * [18] B. B. Sau and V. N. Balasubramanian, ``Deep model compression: Distilling knowledge from noisy teachers,'' _arXiv preprint arXiv:1610.09650_ , 2016.
  * [19] E. J. Crowley, G. Gray, and A. J. Storkey, ``Moonshine: Distilling with cheap convolutions,'' _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [20] W. Park, D. Kim, Y. Lu, and M. Cho, ``Relational knowledge distillation,'' in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3967–3976.
  * [21] J. H. Cho and B. Hariharan, ``On the efficacy of knowledge distillation,'' in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 4794–4802.
  * [22] Y. Lee, K. Jang, J. Goo, Y. Jung, and H. Kim, ``Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning,'' _arXiv preprint arXiv:2207.00555_ , 2022.
  * [23] Z. Peng, A. Budhkar, I. Tuil, J. Levy, P. Sobhani, R. Cohen, and J. Nassour, ``Shrinking bigfoot: Reducing wav2vec 2.0 footprint,'' _arXiv preprint arXiv:2103.15760_ , 2021.
  * [24] H. Shao, W. Wang, B. Liu, X. Gong, H. Wang, and Y. Qian, ``Whisper-kdq: A lightweight whisper via guided knowledge distillation and quantization for efficient asr,'' _arXiv preprint arXiv:2305.10788_ , 2023.
  * [25] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, ``Curriculum learning,'' vol. 60, 06 2009, p. 6.
  * [26] T. Castells, P. Weinzaepfel, and J. Revaud, ``Superloss: A generic loss for robust curriculum learning,'' in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 4308–4319. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/2cfa8f9e50e0f510ede9d12338a5f564-Paper.pdf
  * [27] T. Wu, X. Ding, H. Zhang, J. Gao, M. Tang, L. Du, B. Qin, and T. Liu, ``Discrimloss: A universal loss for hard samples and incorrect samples discrimination,'' _IEEE Transactions on Multimedia_ , vol. 26, pp. 1957–1968, 2024.
  * [28] Z. Li, X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, and J. Yang, ``Curriculum temperature for knowledge distillation,'' in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 2, 2023, pp. 1504–1512.
  * [29] Q. Zhu, X. Chen, P. Wu, J. Liu, and D. Zhao, ``Combining curriculum learning and knowledge distillation for dialogue generation,'' in _Findings of the Association for Computational Linguistics: EMNLP 2021_ , 2021, pp. 1284–1295.
  * [30] T. Castells, P. Weinzaepfel, and J. Revaud, ``Superloss: A generic loss for robust curriculum learning,'' in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 4308–4319. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/2cfa8f9e50e0f510ede9d12338a5f564-Paper.pdf
  * [31] T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár, ``Focal loss for dense object detection,'' _CoRR_ , vol. abs/1708.02002, 2017. [Online]. Available: http://arxiv.org/abs/1708.02002
  * [32] A. Jafari, M. Rezagholizadeh, P. Sharma, and A. Ghodsi, ``Annealing knowledge distillation,'' _CoRR_ , vol. abs/2104.07163, 2021. [Online]. Available: https://arxiv.org/abs/2104.07163
  * [33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , ``Pytorch: An imperative style, high-performance deep learning library,'' _Advances in neural information processing systems_ , vol. 32, 2019.
  * [34] Y.-Y. Yang, M. Hira, Z. Ni, A. Chourdia, A. Astafurov, C. Chen, C.-F. Yeh, C. Puhrsch, D. Pollack, D. Genzel, D. Greenberg, E. Yang, J. Lian, J. Mahadeokar, J. Hwang, J. Chen, P. Goldsborough, P. Roy, S. Narenthiran, and Y. Shi, ``Torchaudio: Building blocks for audio and speech processing,'' 10 2021.
  * [35] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, ``Huggingface's transformers: State-of-the-art natural language processing,'' _CoRR_ , vol. abs/1910.03771, 2019. [Online]. Available: http://arxiv.org/abs/1910.03771