# Improving Deep Learning Interpretability by Saliency Guided Training

Aya Abdelsalam Ismail, Soheil Feizi , Héctor Corrada Bravo 11footnotemark: 1
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Department of Computer Science, University of Maryland
Data Science and Statistical Computing, Genentech, Inc.
Authors contributed equally

###### Abstract

Saliency methods have been widely used to highlight important input features
in model predictions. Most existing methods use backpropagation on a modified
gradient function to generate saliency maps. Thus, noisy gradients can result
in unfaithful feature attributions. In this paper, we tackle this issue and
introduce a saliency guided training***Code:
https://github.com/ayaabdelsalam91/saliency_guided_training procedure for
neural networks to reduce noisy gradients used in predictions while retaining
the predictive performance of the model. Our saliency guided training
procedure iteratively masks features with small and potentially noisy
gradients while maximizing the similarity of model outputs for both masked and
unmasked inputs. We apply the saliency guided training procedure to various
synthetic and real data sets from computer vision, natural language
processing, and time series across diverse neural architectures, including
Recurrent Neural Networks, Convolutional Networks, and Transformers. Through
qualitative and quantitative evaluations, we show that saliency guided
training procedure significantly improves model interpretability across
various domains while preserving its predictive performance.

##  1 Introduction

Deep Neural Networks (DNNs) have been widely used in a variety of different
tasks [32, 27, 44, 38]; yet interpreting complex networks remains a challenge.
Reliable explanations are necessary for critical domains like medicine,
neuroscience, finance, and autonomous driving [9, 35]. Explanations are also
useful for model debugging [64, 36]. As a result, various interpretability
methods were developed to understand DNNs [5, 50, 22, 54, 53, 33, 52]. A
common approach for understanding model decisions is to identify features in
the input that highly influenced the final classification decision [6, 64, 54,
53, 48, 37, 65]. Such approaches, known as saliency maps, often use gradient
calculations to assign an importance score to individual features, reflecting
their influences on the model prediction.

Saliency methods aim to highlight meaningful input features in model
predictions to humans; however, the maps produced are often noisy (i.e.,
contain visual noise). To improve the faithfulness of saliency maps,
explanations methods that depend on more than one or higher-order gradient
calculations were developed. For example, SmoothGrad [53] reduces saliency
noise by adding noise to the input multiple times and then taking the average
of the resulting saliency maps for each input. Integrated gradients [54],
DeepLIFT [48] and Layer-wise Relevance Propagation [5] backpropagate through a
modified gradient function [3] while Singla et al. [52] studies the use of
higher-order gradients in saliency maps.

In this paper, we take a different approach to improve the interpretability of
deep neural networks— instead of developing yet another saliency method, we
propose a new training procedure that naturally leads to improved model
explanations using current saliency methods. Our proposed training procedure,
called saliency guided training, trains models that produce sparse,
meaningful, and less noisy gradients without degrading model performance. This
is done by iteratively masking input features with low gradient values (i.e.,
less important features) and then minimizing a loss function that combines (a)
the KL divergence [28] between model outputs from the original and masked
inputs, and (b) the appropriate loss function for the model prediction. This
procedure reduces noise in model gradients without sacrificing its predictive
performance.

To demonstrate the effectiveness of our proposed saliency guided training
approach, we consider a variety of classification tasks for images, language,
and multivariate time series across diverse neural architectures, including
Convolutional Neural Networks (CNNs), Recurrent Neural Network (RNNs), and
Transformers. In particular, we observe that using saliency guided training in
image classification tasks leads to a reduction in visual saliency noise and
sparser saliency maps, as shown in Figure 2. Saliency guided training also
improves the comprehensiveness of the produced explanations for sentiment
analysis, and fact extraction tasks as shown in Table 1.

In multivariate time series classification tasks, we observe an increase in
the precision and recall of saliency maps when applying the proposed saliency
guided training. Interestingly, we also find that the saliency guided training
reduces the vanishing saliency issue of RNNs [20] as shown in Figure 6.
Finally, we note that although we use the vanilla gradient for masking in the
saliency guided training procedure, we observe significant improvements in the
explanations produced after training by several other gradient-based saliency
methods.

##  2 Background and Related Work

Interpretability is a rapidly growing area with several diverse lines of
research. One strand of interpretability considers post-hoc explanation
methods, aiming to explain why a trained model made a specific prediction for
a given input. Post-hoc explanation methods can be divided into gradient-based
methods [6, 54, 53, 48, 37, 47] that can be reformulated as computing
backpropagation for a modified gradient function and perturbation-based
approaches [64, 55, 43, 58] that perturb areas of the input and measure how
much this changes the model output. Perkins et al. [41] uses gradients for
feature selection through Grafting. Another line of works aims to measure the
reliability of interpretability methods. This can be done by creating
standardize benchmarks with interpretability metrics [18, 21, 11, 57, 46, 42]
or debugging explanations [1, 24, 15, 2] by identifying test cases where
explanations fail. Others [4, 12, 45, 62, 20] focus on modifying neural
architectures for better interpretability. Similar to our line of work,
Ghaeini et al. [14] and Ross et al. [45] incorporate explanations into the
learning process. However, Ghaeini et al. [14] relies on the existence of the
ground truth explanations while Ross et al. [45] relies on the availability of
annotations about incorrect explanations for a particular input. Our proposed
learning approach does not rely on such annotations; since most datasets only
have ground truth labels, it may not be practical to assume the availability
of positive or negative explanations.

Input level perturbation during training has been previously explored. [34,
19, 61, 51] use attention maps to improve segmentation for weakly supervised
localization. Wang et al. [60] incorporates attention maps into training to
improve classification accuracy. DeVries and Taylor [10] masks out square
regions of input during training as a regularization technique to improve the
robustness and overall performance of convolutional neural networks. Our work
focuses on a different task which is increasing model interpretability through
training in a self-supervised manner.

In this paper, we evaluate our learning procedure with the following saliency
methods: Gradient (GRAD) [6] is the gradient of the output w.r.t the input.
Integrated Gradients (IG) [54] calculates a path integral of the model
gradient to the input from a non-informative reference point. DeepLIFT (DL)
[48] compares the activation of each neuron to a reference activation; the
relevance is the difference between the two activations. SmoothGrad (SG) [53]
samples similar input by adding noise to the input and then takes the average
of the resulting sensitivity maps for each sample. Gradient SHAP (GS) [37]
adds noise to the input, then selects a point along the path between a
reference point and input, and computes the gradient of outputs w.r.t those
points.

We demonstrate the effectiveness of our training procedure using several
neural network architectures: Convolution neural networks (CNNs) including
VGG-16 [49], ResNet [16] and Temporal Convolutional Network (TCN) [39, 29, 7],
a CNN that handles sequences; Recurrent neural networks (RNNs) including LSTM
[17] and LSTM with Input-Cell Attention [20]; as well as Transformers [59].

##  3 Notation

First, consider a classification problem on the input data
$\\{(X_{i},y_{i})\\}_{i=1}^{n}$ such that each
$X=[x_{1},\dots,x_{N}]\in\mathbb{R}^{N}$ has $N$ features and $y$ is the
label. Let $f_{\theta}$ denote a neural network parameterized by $\theta$. The
standard training of the network involves minimizing the cross-entropy loss
$\mathcal{L}$ over the training set as follows:

$\displaystyle\operatorname*{minimize}_{\theta}\leavevmode\nobreak\
\leavevmode\nobreak\
\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}\left(f_{\theta}\left(X_{i}\right),y_{i}\right)$
(1)

The gradient of the network output $f_{\theta}\left(X\right)$ with respect to
the input $X$ is given by $\nabla_{X}f_{\theta}\left(X\right)$. Let $S(.)$ be
a sorting function such that $S_{e}(Z)$ is the $e^{th}$ smallest element in
$Z$. Hence, $S\left(\nabla_{X}f_{\theta}\left(X\right)\right)$ is the sorted
gradient. We define the input mask function $M_{k}(.)$ such that
$M_{k}(S(X),X)$ replaces all $x_{i}$ where
$S(x_{i})\in\\{S_{e}\left(x_{i}\right)\\}_{e=0}^{k}$ with a mask distribution,
i.e., $M_{k}(S(X),X)$ removes the $k$ lowest features from $X$ based on the
order provided by $S(X)$.

For a language input, we use $X=[x_{1},\dots,x_{N}]$ where
$x_{i}\in\mathbb{R}^{d}$ is the feature embedding representing the $i^{th}$
word of the input. In that case, $S(X)$ would sort elements of $X$ based on
the sum of the gradient of the embeddings for each word $x$ and
$M_{k}(S(X),X)$ would mask the bottom $k$ words according to that sorting. For
a multivariate time series input, we use
$X=[x_{1,1},\dots,x_{F,1},\dots,x_{F,T}]\in\mathbb{R}^{F\times T}$ where $T$
is the number of time steps and $F$ is the number of features per time step.
$x_{i,t}$ is the input feature $i$ at time $t$; sorting and masking would be
done at the $x_{i,t}$ level.

For two discrete probability distributions $P$ and $Q$ defined on the same
probability space ${\mathcal{X}}$, the Kullback–Leibler (KL) divergence [28]
(or, relative entropy) from $Q$ to $P$ is given as ${\displaystyle
D_{\text{KL}}}$:

$\displaystyle{\displaystyle D_{\text{KL}}(P\parallel
Q)=\sum_{x\in{\mathcal{X}}}P(x)\log\left({\frac{P(x)}{Q(x)}}\right).}$ (2)

##  4 Saliency Guided Training

Existing gradient-based methods can produce noisy saliency maps as shown in
Figure 1. The saliency map noise may be partially due to some uninformative
local variations in partial derivatives. Using a standard training procedure
based on ERM (expectation risk minimization), the gradient of the model w.r.t.
the input (i.e., $\nabla_{X}f_{\theta}\left(X\right)$) may fluctuate sharply
via small input perturbations [53].

Figure 1: Saliency maps produced by typical training versus saliency guided
training.

If gradient-based explanation methods faithfully interpret the model’s
predictions, irrelevant features should have gradient values close to zero.
Building on this intuition, we introduce saliency guided training, a procedure
to train neural networks such that input gradients computed from trained
models provide more faithful measures to downstream (gradient-based) saliency
methods. Saliency guided training aims to reduce gradient values of irrelevant
features without sacrificing the model performance. During saliency guided
training, for every input $X$, we create a new input $\widetilde{X}$ by
masking the features with low gradient values as follows:

$\displaystyle\widetilde{X}=M_{k}(S(\nabla_{X}f_{\theta}\left(X\right)),X)$
(3)

$\widetilde{X}$ is then passed through the network which results in an output
$f_{\theta}(\widetilde{X})$. In addition to the classification loss, the
saliency guided training minimizes the KL divergence between
$f_{\theta}\left(X\right)$ and $f_{\theta}(\widetilde{X})$ to ensure that the
trained model produces similar output probability distributions over labels
for both masked and unmasked inputs. The optimization problem for the saliency
guided training is:

$\displaystyle\operatorname*{minimize}_{\theta}\;\frac{1}{n}\;{\sum}_{i=1}^{n}\biggl{[}\mathcal{L}\Bigl{(}f_{\theta}\left(X_{i}\right),y_{i}\Bigr{)}+\lambda
D_{KL}\Bigl{(}f_{\theta}\left(X_{i}\right)\parallel
f_{\theta}(\widetilde{X_{i}})\Bigr{)}\biggr{]}$ (4)

where $\lambda$ is a hyperparameter to balance between the cross-entropy
classification loss and the KL divergence term. Since this loss function is
differentiable with respect to $\theta$, it can be optimized using existing
gradient-based optimization methods. The KL divergence term encourages the
model to produce similar outputs for the original input $X$ and masked input
$\widetilde{X}$. For this to happen, the model will need to learn to assign
low gradient values to irrelevant features in model predictions. This
potentially results in sparse and more faithful gradients as shown in Figure
1.

Masking functions: In images and time series data, features with low gradients
are replaced with random values within the feature range. In language tasks,
the masking function replaces the low salient word with the previous high
salient word. This allows us to emphasize on high salient words and remove
non-salient ones while maintaining the sentence length. The selection of $k$
is dataset-dependent. It depends on the amount of irrelevant information in a
training sample. For example, since most pixels in MNIST are uninformative, a
larger $k$ is desired. Detailed hyperparameters used is available in the
appendix. Note that, only input features are masked during the saliency guided
training.

Limitations: (a) Compared to traditional training, our proposed training
procedure is more computationally expensive. Specifically, the memory needed
is doubled since now in addition to storing the batch, we are storing the
masked batch as well. Similar to adversarial training, this training process
is slow and takes a larger number of epochs to converge. For example, the
standard training of a CIFAR-10 model usually takes on average 118 epochs to
converge where each epoch is roughly 24 seconds. Using the saliency guided
training, the convergence takes about 124 epochs where each epoch takes
roughly 75 seconds (all experiments on the same GPU). (b) Our training
procedure requires two hyperparameters $k$ and $\lambda$ which might require a
hyperparameter search (we find that $\lambda=1$ works well in all of our
experiments).

Given: Training samples $X$, # of features to be masked $k$, learning rate
$\tau$, hyperparameter $\lambda$

Initialize $f_{\theta}$

for  _$i\leftarrow 1$ to epochs _ do

for  _minibatch_ do

Compute the masked input:

Get sorted index $I$ for the gradient of output with respect to the input.

$I=S\Bigl{(}\nabla_{X}f_{\theta{{}_{i}}}\left(X\right)\Bigr{)}$

Mask bottom $k$ features of the original input.

$\widetilde{X}=M_{k}(I,X)$

Compute the loss function:

$L_{i}=\mathcal{L}\Bigl{(}f_{\theta_{i}}\left(X\right),y\Bigr{)}+\lambda
D_{KL}\Bigl{(}f_{\theta_{i}}\left(X\right)\parallel
f_{\theta_{i}}(\widetilde{X})\Bigr{)}$

Use the gradient to update network parameters:

$f_{\theta_{i+1}}=f_{\theta_{i}}-\tau\;\nabla_{\theta_{i}}L_{i}$

end for

end for

Algorithm 1 Saliency Guided Training

##  5 Experiments

All experiments have been repeated 5 times; the results reported below are the
average of the 5 runs. Hyperparameters used for each experiment, along with
the standard error bars and details on computational resources are available
in supplementary materials.

###  5.1 Saliency Guided Training for Images

In the following section, we compare gradient-based explanations produced by
regular training versus saliency guided training for MNIST [31] trained on a
simple CNN [30], for CIFAR10 [26] trained on ResNet18 [16] and for BIRD [13]
trained on VGG-16 [49]. Further details about the datasets and models are
available in the supplementary material.

Figure 2: (A) Comparison between different training methods on MNIST along
with distributions of gradient values in each sample. (B) Saliency maps for
CIFAR10 and BIRD datasets using regular and saliency guided training. (C)
Distribution of gradient means across examples. Maps produced by saliency
guided training are more precise: most features have gradient values around
zero with large gaps between mean and outliers. Here gradients around zero
indicate uninformative features, while very large and very small gradients
indicate informative features. Saliency guided training helps reduce noisy
fluctuating gradients in between as shown in the box plots.

#### Quality of Saliency Maps for Images

For an image classification problem, in many cases, most features are
redundant and not needed by the model to make the prediction. Consider the
background of an object in an image; although it covers most of the image,
backgrounds are often not essential in the classification task. If the model
is focusing on the object rather than the background, we would want the
background gradient (i.e., most of the features) to be close to zero.

The examples shown in Figure 2 were correctly classified by both models.
Gradients are scaled per sample to have values between -1 and 1. In Figure 2
(A) and Figure 2 (B), saliency maps produced by a model trained with saliency
guided training were more precise than that trained traditionally. Most
saliency maps produced by saliency guided training highlight the object itself
rather than the background across different datasets. The distributions of
gradient values per sample in Figure 2 (A) show that most features have small
gradient values (near zero) with a large separation of high salient features
away from zero for the saliency guided training. Similarly, in Figure 2 (C),
we find that over the entire dataset, gradient values produced by the saliency
guided training tend to be concentrated around zero with a large separation
between the mean and outliers (highly salient features), indicating the
model’s ability to differentiate between informative and non-informative
features.

Figure 3: Model accuracy drop when removing features with high saliency using
traditional and the saliency guided training for different gradient-based
methods against a random baseline. A steeper drop indicates a better
performance. We find that regardless of the saliency method used, the
performance improves by the saliency guided training.

#### Model Accuracy Drop

We compare the saliency guided training and traditional training for different
saliency methods with modification-based evaluation [46, 42, 23]: First,
features are ranked according to the saliency values. Then, higher-ranked
features are recursively eliminated (the original background in MNIST replaces
the eliminated features). Finally, the degradation to the trained model
accuracy is reported. This is done at different feature percentages. A steeper
drop indicates that the removed features affected the model accuracy more.
Figure 10 compares the model performance degradation on different gradient-
based methods; the saliency guided training shows a steeper accuracy drop
regardless of the saliency method used.

Figure 4:  Accuracy drop in different modification-based evaluation masking
approaches.

This experiment can only be performed on a dataset like MNIST since the
uninformative feature distribution is known (black background), while this is
not the case in other datasets that we have considered. Although such
modification-based evaluation methods have been applied to other datasets,
[46, 42, 23]; Hooker et al. [18] showed that removing features produces
samples from a different data distribution violating the underlying IID
assumption (i.e., the training and evaluation data come from identical
distributions). When the feature replacement comes from a different
distribution, it is unclear whether the degradation in the model performance
is from the distribution shift or the removal of informative features. For
that reason, we need to make sure that the model is trained on the mask used
during testing to avoid this undesired effect.

Hooker et al. [18] proposes ROAR where the model is retrained after the
feature elimination. However, due to the data redundancy, the retrained model
can rely on different features to achieve the same accuracy. Figure 4 shows
the model accuracy drop on traditionally trained MNIST when removing the
salient features. The IID line represents replacing features with the black
MNIST background (known uninformative distribution), which acts as the ground
truth in this particular dataset. The OOD line represents replacing the
features with the mean image pixel value as done by [46, 42, 23]; and ROAR
shows replacing features with the mean value and retraining the model as
proposed by Hooker et al. [18]. Since neither OOD nor ROAR produce results
similar to those produced by the IID feature replacement, we argue that
modification-based evaluation methods may provide unreliable results unless
the uninformative IID distribution is known. We leave further exploration of
modification-based evaluation methods to future work.

###  5.2 Saliency Guided Training for Language

We compare the interpretability of recurrent models trained on language tasks
using the ERASER [11] benchmark. ERASER was designed to capture how well an
explanation provided by models aligns with human rationales and how faithful
these explanations are (i.e., the degree to which explanation influences the
predictions). For our purpose, we only focus on the faithfulness of the
explanations.

ERASER provides two metrics to measure interpretability. Comprehensiveness
evaluates if all features needed to make a prediction are selected. To
calculate an explanation comprehensiveness, a new input $\overline{X}_{i}$ is
created such that $\overline{X}_{i}=X_{i}-R_{i}$ where $R_{i}$ is predicted
rationales (i.e. the words selected by saliency method as informative). Let
$f_{\theta}\left(X_{i}\right)_{j}$ be the prediction of model for class $j$.
The model comprehensiveness is calculated as:

$\displaystyle\text{Comprehensiveness}=f_{\theta}\left(X_{i}\right)_{j}-f_{\theta}\left(\overline{X}_{i}\right)_{j}$
(5)

A high score here implies that the explanation removed was influential in the
predictions. The second metric is Sufficiency that evaluates if the extracted
explanations contain enough signal to make a prediction. The following
equation gives the explanation sufficiency:

$\displaystyle\text{Sufficiency}=f_{\theta}\left(X_{i}\right)_{j}-f_{\theta}\left(R_{i}\right)_{j}$
(6)

A lower score implies that the explanations are adequate for a model
prediction. The comprehensiveness and sufficiency were calculated at different
percentages of features (similar to [11] percentages are $1\%,5\%,10\%,20\%$
and $50\%$), and Area Over the Perturbation Curve (AOPC) is reported.

We focus on datasets that can be formulated as a classification problem: Movie
Reviews: [63] positive/negative sentiment classification for movie reviews.
FEVER: [56] a fact extraction and verification dataset where the goal is
verifying claims from textual sources; each claim can either be supported or
refuted. e-SNLI: [8] a natural language inference task where sentence pairs
are labeled as entailment, contradiction, neutral and, supporting.

Word embeddings are generated from Glove [40]; then passed to a bidirectional
LSTM [17] for classification. Table 1 compares the scores produced by
different saliency methods for traditional and saliency guided training
against random assignment baseline. We found that saliency guided training
results in a significant improvement in both comprehensiveness and
sufficiently for sentiment analysis task Movie Reviews dataset. While for fact
extraction task FEVER dataset, and natural language inference task e-SNLI
dataset saliency guided training improves comprehensiveness and there is no
obvious improvement in sufficiency (this might be due to the adversarial
effect of shrinking the sentence to a much smaller size since the number of
words identified as “rationales” is smaller than the remaining words).

| Gradient | Integrated Gradient | SmoothGrad | Random
---|---|---|---|---
| Trad. | Sal. Guided | Trad. | Sal. Guided | Trad. | Sal. Guided |
Movies |  |  |  |  |  |  |
Comprehensiveness $\uparrow$ | 0.200 | 0.240 | 0.265 | 0.306 | 0.198 | 0.256 | 0.056
Sufficiency $\downarrow$ | 0.042 | 0.013 | 0.054 | 0.002 | 0.034 | 0.008 | 0.294
FEVER |  |  |  |  |  |  |
Comprehensiveness$\uparrow$ | 0.007 | 0.008 | 0.008 | 0.009 | 0.007 | 0.008 | 0.001
Sufficiency$\downarrow$ | 0.012 | 0.011 | 0.005 | 0.004 | 0.006 | 0.006 | 0.003
e-SNLI |  |  |  |  |  |  |
Comprehensiveness $\uparrow$ | 0.117 | 0.126 | 0.099 | 0.104 | 0.117 | 0.118 | 0.058
Sufficiency$\downarrow$ | 0.420 | 0.387 | 0.461 | 0.419 | 0.476 | 0.455 | 0.366

Table 1: Eraser benchmark scores: Comprehensiveness and sufficiency are in
terms of AOPC. ‘Random’ is a baseline when words are assigned random scores.

###  5.3 Saliency Guided Training for Time Series

We evaluated saliency guided training for multivariate time series, both
quality on multivariate time series MNIST and quantitatively through synthetic
data.

#### Saliency Maps Quality for Multivariate Time Series

We compare the saliency maps produced on MNIST treated as a multivariate time
series where one image axis is time. Figure 5 shows the saliency maps produced
by different (neural architecture, saliency method) pairs when different
training procedures were used. There is a visible improvement in saliency
quality across different networks when saliency guided training is used.

Figure 5: Saliency maps produced for (neural architecture, saliency method)
pairs. Traditional training was used for networks in the $1^{\text{st}}$ row,
while saliency guided training was used for the $2^{\text{nd}}$ row. Grad, DL,
GS and DLS stand for Gradient, DeepLift, Gradient SHAP and DeepSHAP,
respectively. There is an improvement in the quality of saliency maps when
saliency guided training is used.

#### Quantitative Analysis on Synthetic Data

We evaluated the saliency guided training on a multivariate time series
benchmark proposed by Ismail et al. [21]. The benchmark consists of 10
synthetic datasets, each examining different design aspects in typical time
series datasets. Informative features are highlighted by the addition of a
constant $\mu$ to the positive class and subtraction of $\mu$ from the
negative class. Following Ismail et al. [21], we compare 4 neural
architectures: LSTM [17], LSTM with Input-Cell Attention [20], Temporal
Convolutional Network (TCN) [29] and, Transformers [59]. Additional details
about the dataset and architectures are provided in the supplementary
material.

Quantitatively measuring the interpretability of a (neural architecture,
saliency method) pair involves applying the saliency method, ranking features
according to the saliency values, replacing high salient features with
uninformative features from the original distribution at different
percentages. Finally, the area under the precision curve (AUP) and the area
under the recall curve (AUR) is calculated by the precision/recall values at
different levels of degradation. Similar to Ismail et al. [21], we compare the
AUP and AUR with a random baseline; since the baseline might be different for
different models, we reported the difference between metrics values generated
using the saliency method and the baseline. For example, the difference
between gradient and random baseline Diff(AUP) when the model is trained
traditionally is calculated as:

$\displaystyle\text{{Diff}(AUP)}_{Grad,Trad.}=AUP_{Grad,Trad.}-AUP_{Random,Trad.}$
(7)

Similarly difference when the model is trained using saliency guided training
is:

$\displaystyle\text{{Diff}(AUP)}_{Grad,Sal.}=AUP_{Grad,Sal.}-AUP_{Random,Sal.}$
(8)

The mean metrics over all 10 datasets is shown in Table 2. Higher values
indicate better performance; negative values indicate performance similar to
random feature assignment. Overall, the best performance was achieved by (TCN,
Integrated gradients) when using saliency guided training. Detailed results
for each dataset are available in the supplementary material.

Metric | Architecture | Gradient | Integrated Gradient | DeepLIFT | Gradient SHAP | DeepSHAP | SmoothGrad
---|---|---|---|---|---|---|---
Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal.
Diff(AUP) | LSTM | -0.113 | -0.119 | -0.083 | -0.024 | -0.097 | -0.108 | -0.088 | -0.069 | -0.098 | -0.109 | -0.110 | -0.097
LSTM + Input. | 0.060 | 0.118 | 0.188 | 0.245 | 0.202 | 0.263 | 0.198 | 0.250 | 0.214 | 0.272 | 0.040 | 0.084
TCN | 0.106 | 0.168 | 0.233 | 0.291 | 0.248 | 0.270 | 0.235 | 0.288 | 0.263 | 0.280 | 0.088 | 0.155
Transformer | -0.054 | -0.062 | 0.061 | 0.044 | -0.040 | -0.032 | 0.069 | 0.023 | -0.014 | -0.055 | -0.018 | -0.046
Diff(AUR) | LSTM | -0.017 | 0.019 | 0.062 | 0.121 | 0.047 | 0.089 | 0.060 | 0.102 | 0.031 | 0.075 | 0.007 | 0.004
LSTM + Input. | 0.075 | 0.136 | 0.185 | 0.198 | 0.187 | 0.204 | 0.182 | 0.196 | 0.183 | 0.201 | 0.043 | 0.111
TCN | 0.125 | 0.171 | 0.191 | 0.210 | 0.202 | 0.204 | 0.185 | 0.209 | 0.196 | 0.192 | 0.046 | 0.138
Transformer | 0.102 | 0.104 | 0.182 | 0.176 | 0.145 | 0.146 | 0.171 | 0.162 | 0.101 | 0.065 | 0.040 | 0.018

Table 2: The mean difference in weighted AUP and AUR for different (neural
architecture, saliency method) pairs. Overall, the best preference was
achieved by TCN when using Integrated gradients as a saliency method and
saliency guided training procedure.

#### Saliency Guided Training reduces vanishing saliency of recurrent neural
networks

Ismail et al. [20] showed that saliency maps in RNNs vanish over time, biasing
detection of salient features only to later time steps. This section
investigates if using saliency guided training reduces the vanishing saliency
issue in RNNs. Repeating experiments done by Ismail et al. [20], three
synthetic datasets were generated as shown Figure 6 (A). The specific features
and the time intervals (boxes) on which they are considered important are
varied between datasets to test the model’s ability to capture importance at
different time intervals. We trained an LSTM with traditional and saliency
guided training procedures.

The area under precision curve (AUP) and the area under the recall curve (AUR)
are calculated by the precision/recall values at different levels of
degradation. Higher AUP and AUR suggest better performance. Results are shown
in Figure 6 (B).

A traditionally trained LSTM shows clear bias in detecting features in the
later time steps; AUP and AUR increase as informative features move to later
time steps. When saliency guided training is used, LSTM was able to identify
informative features regardless of their locations in time.

Figure 6: (A) Samples from 3 different simulated datasets, informative
features are located at the earlier, intermediate, and later time steps. (B)
AUP and AUR were produced by LSTM by traditional and saliency guided training
procedures. Traditionally trained LSTM shows clear bias in detecting features
in the later time steps. When saliency guided training is used, there is no
time bias.

##  6 Summary and Conclusion

We propose saliency guided training as a new training procedure that improves
the quality of explanations produced by existing gradient-based saliency
methods. saliency guided training is optimized to reduce gradient values for
irrelevant features. This is done by masking input features with low gradients
and then minimizing the KL divergence between outputs from the original and
masked inputs along with the main loss function. We demonstrated the
effectiveness of the saliency guided training on images, language, and
multivariate time series.

Our proposed training method encourages models to sharpen the gradient-based
explanations they provide. It does this however without requiring explanations
as input. It instead may be cast as a regularization procedure where
regularization is provided by feature sparsity driven by a gradient-based
feature attribution. This is an alternative approach to using ground truth
explanations to force the model to be right for the right reasons [45]. We
found that training model explanations in an unsupervised fashion also
improves model faithfulness. This opens an interesting avenue for other
unsupervised, perhaps regularization-based, methods to improve the
interpretability of prediction models.

##  7 Acknowledgments

This project was supported in part by NSF CAREER AWARD 1942230, a grant from
NIST 60NANB20D134, NSF award CDS&E:1854532, ONR grant 13370299 and AWS Machine
Learning Research Award.

## References

  * Adebayo et al. [2018] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Adebayo et al. [2020] Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim.  Debugging tests for model explanations.  _arXiv preprint arXiv:2011.05429_ , 2020.
  * Ancona et al. [2018] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross.  Towards better understanding of gradient-based attribution methods for deep neural networks.  _International Conference on Learning Representations_ , 2018.
  * Ba and Caruana [2014] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  In _Advances in Neural Information Processing Systems_ , 2014.
  * Bach et al. [2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  In _PLoS ONE_ , 2015.
  * Baehrens et al. [2010] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert MÃžller.  How to explain individual classification decisions.  In _Journal of Machine Learning Research_ , 2010.
  * Bai et al. [2018] Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  _arXiv preprint arXiv:1803.01271_ , 2018.
  * Camburu et al. [2018] Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom.  e-snli: Natural language inference with natural language explanations.  _arXiv preprint arXiv:1812.01193_ , 2018.
  * Caruana et al. [2015] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad.  Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.  In _International conference on knowledge discovery and data mining_ , 2015.
  * DeVries and Taylor [2017] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_ , 2017.
  * DeYoung et al. [2019] Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace.  Eraser: A benchmark to evaluate rationalized nlp models.  _arXiv preprint arXiv:1911.03429_ , 2019.
  * Frosst and Hinton [2017] Nicholas Frosst and Geoffrey Hinton.  Distilling a neural network into a soft decision tree.  _arXiv preprint arXiv:1711.09784_ , 2017.
  * Gerry [2021] Gerry.  265 bird species, 2021.  URL https://www.kaggle.com/gpiosenka/100-bird-species/.
  * Ghaeini et al. [2019] Reza Ghaeini, Xiaoli Z Fern, Hamed Shahbazi, and Prasad Tadepalli.  Saliency learning: Teaching the model where to pay attention.  _arXiv preprint arXiv:1902.08649_ , 2019.
  * Ghorbani et al. [2019] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In _AAAI Conference on Artificial Intelligence_ , 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  In _Neural computation_ , 1997.
  * Hooker et al. [2019] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim.  A benchmark for interpretability methods in deep neural networks.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Hou et al. [2018] Qibin Hou, Peng-Tao Jiang, Yunchao Wei, and Ming-Ming Cheng.  Self-erasing network for integral object attention.  _arXiv preprint arXiv:1810.09821_ , 2018.
  * Ismail et al. [2019] Aya Abdelsalam Ismail, Mohamed Gunady, Luiz Pessoa, Hector Corrada Bravo, and Soheil Feizi.  Input-cell attention reduces vanishing saliency of recurrent neural networks.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Ismail et al. [2020] Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi.  Benchmarking deep learning interpretability in time series predictions.  _arXiv preprint arXiv:2010.13924_ , 2020.
  * Kindermans et al. [2016] Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, and Sven Dähne.  Investigating the influence of noise and distractors on the interpretation of neural networks.  _arXiv preprint arXiv:1611.07270_ , 2016.
  * Kindermans et al. [2017] Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne.  Learning how to explain neural networks: Patternnet and patternattribution.  _arXiv preprint arXiv:1705.05598_ , 2017.
  * Kindermans et al. [2019] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim.  The (un) reliability of saliency methods.  In _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_ , 2019.
  * Kokhlikyan et al. [2020] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson.  Captum: A unified and generic model interpretability library for pytorch, 2020.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  _Citeseer_ , 2009.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 2012.
  * Kullback and Leibler [1951] Solomon Kullback and Richard A Leibler.  On information and sufficiency.  _The annals of mathematical statistics_ , 1951.
  * Lea et al. [2017] Colin Lea, Michael Flynn, Rene Vidal, Austin Reiter, and Gregory Hager.  Temporal convolutional networks for action segmentation and detection.  In _Conference on Computer Vision and Pattern Recognition_ , 2017\.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 1998.
  * LeCun et al. [2010] Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database, 2010.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 2015.
  * Levine et al. [2019] Alexander Levine, Sahil Singla, and Soheil Feizi.  Certifiably robust interpretation in deep learning.  _arXiv preprint arXiv:1905.12105_ , 2019.
  * Li et al. [2018] Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu.  Tell me where to look: Guided attention inference network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * Lipton [2018] Zachary C Lipton.  The mythos of model interpretability.  In _Queue_ , 2018.
  * Lou et al. [2012] Yin Lou, Rich Caruana, and Johannes Gehrke.  Intelligible models for classification and regression.  In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2012.
  * Lundberg and Lee [2017] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Obermeyer and Emanuel [2016] Ziad Obermeyer and Ezekiel J Emanuel.  Predicting the future—big data, machine learning, and clinical medicine.  In _The New England journal of medicine_ , 2016.
  * Oord et al. [2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014.
  * Perkins et al. [2003] Simon Perkins, Kevin Lacker, and James Theiler.  Grafting: Fast, incremental feature selection by gradient descent in function space.  _The Journal of Machine Learning Research_ , 2003.
  * Petsiuk et al. [2018] Vitali Petsiuk, Abir Das, and Kate Saenko.  Rise: Randomized input sampling for explanation of black-box models.  _arXiv preprint arXiv:1806.07421_ , 2018.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016.
  * Rich [2016] Michael L Rich.  Machine learning, automated suspicion algorithms, and the fourth amendment.  In _University of Pennsylvania Law Review_ , 2016.
  * Ross et al. [2017] Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez.  Right for the right reasons: Training differentiable models by constraining their explanations.  _arXiv preprint arXiv:1703.03717_ , 2017.
  * Samek et al. [2016] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller.  Evaluating the visualization of what a deep neural network has learned.  _IEEE transactions on neural networks and learning systems_ , 2016\.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * Shrikumar et al. [2017] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In _International Conference on Machine Learning_ , 2017.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Simonyan et al. [2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _CoRR_ , 2013.
  * Singh and Lee [2017] Krishna Kumar Singh and Yong Jae Lee.  Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization.  In _2017 IEEE international conference on computer vision (ICCV)_. IEEE, 2017.
  * Singla et al. [2019] Sahil Singla, Eric Wallace, Shi Feng, and Soheil Feizi.  Understanding impacts of high-order loss approximations and features in deep learning interpretation.  In _International Conference on Machine Learning_. PMLR, 2019.
  * Smilkov et al. [2017] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  _arXiv preprint arXiv:1706.03825_ , 2017.
  * Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning_ , 2017.
  * Suresh et al. [2017] Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi.  Clinical intervention prediction and understanding using deep networks.  _arXiv preprint arXiv:1705.08498_ , 2017.
  * Thorne et al. [2018] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.  Fever: a large-scale dataset for fact extraction and verification.  _arXiv preprint arXiv:1803.05355_ , 2018.
  * Tomsett et al. [2020] Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece.  Sanity checks for saliency metrics.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2020.
  * Tonekaboni et al. [2020] Sana Tonekaboni, Shalmali Joshi, Kieran Campbell, David Duvenaud, and Anna Goldenberg.  What went wrong and when? instance-wise feature importance for time-series models.  _arXiv preprint arXiv:2003.02821_ , 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Wang et al. [2019] Lezi Wang, Ziyan Wu, Srikrishna Karanam, Kuan-Chuan Peng, Rajat Vikram Singh, Bo Liu, and Dimitris N Metaxas.  Sharpen focus: Learning with attention separability and consistency.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019.
  * Wei et al. [2017] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan.  Object region mining with adversarial erasing: A simple classification to semantic segmentation approach.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017.
  * Wu et al. [2018] Mike Wu, Michael C Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez.  Beyond sparsity: Tree regularization of deep models for interpretability.  In _AAAI Conference on Artificial Intelligence_ , 2018.
  * Zaidan and Eisner [2008] Omar Zaidan and Jason Eisner.  Modeling annotators: A generative approach to learning from annotator rationales.  In _Proceedings of the 2008 conference on Empirical methods in natural language processing_ , 2008.
  * Zeiler and Fergus [2014] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , 2014.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.

## Checklist

Please do not modify the questions and only use the provided macros for your
answers. Note that the Checklist section does not count towards the page
limit. In your paper, please delete this instructions block and only keep the
Checklist section heading above, along with the questions/answers below.

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]  section 4

    3. (c)

Did you discuss any potential negative social impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]  In the supplemental material

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]  In the supplemental material

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]  In the
supplemental material

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [N/A]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL?[N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

## Supplementary

## Experimental Details

Computational resources: All experiments were ran on a single 12GB NVIDIA
RTX2080Ti GPU. Saliency Methods: Captum [25] implementation was used for
different saleincy methods.

### Saliency Guided Training for Images

##### Datasets and Classifiers

  * •

MNIST [31] a database of handwritten digits. The classifier consists of two
CNN layers with kernel size 3 and stride of 1 followed by two fully connected
layers, two dropout layers with $p=0.25$ and $p=0.5$, and the 10 output
neurons.

  * •

CIFAR10 [26] a low-resolution classification dataset with 10 different classes
representing airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships,
and trucks. ResNet18 [16] was used as a classifier, ResNet18 is a very deep
CNN with “identity shortcut connection,” i.e., skip connections, that skip one
or more layers to solve the vanishing gradient problem faced by deep networks.

  * •

BIRD [13] A kaggle datasets of 260 bird species. Images were gathered from
internet searches by species name. VGG16 [49] was used as a classifier, the
last few dense layers and the output layer were modified to accommodate the
number of classes in this dataset.

Dataset | # Training | # Testing | # Classes | Features | Test Accuracy | $\lambda$ | $k$
---|---|---|---|---|---|---|---
|  |  |  |  | Tradtional | Sal. Guided |  | (as a % of feature)
MNIST | 60000 | 10000 | 10 | $1\times 28\times 28$ | 99.4 | 99.3 | 1 | 50%
CIFAR10 | 50000 | 10000 | 10 | $3\times 32\times 32$ | 92.0 | 91.5 | 1 | 50%
BIRD | 38518 | 1350 | 260 | $3\times 224\times 224$ | 96.6 | 96.9 | 1 | 50%

Table 3: Datasets used for Image experiments. $k$ is the percentage of overall
features masked during saliency guided training. For example, in MNIST number
of features masked $\big{\lceil}0.5\times 28\times 28\big{\rceil}=392$.

##### Masking

For images, low salient features are replaced by a random variable within the
color channel input range. For example, in an RGB image, if pixel $2\times 3$
is to be masked $1\times 2\times 3$ would be replaced with a random variable
within R channel range, similarly $2\times 2\times 3$ and $3\times 2\times 3$
would be replaced with a random variable within G and B channel range
respectively.

#### Saliency Map Quality for Images

The examples shown in Figure 7, Figure 8, and Figure 9 were correctly
classified by both models. Gradients are scaled per sample to have values
between -1 and 1. Overall, saliency maps produced by saliency guided training
are less noisy than those produced by traditional training and tend to
highlight the object itself rather than the background. The distributions of
gradient values per sample show that most features have small values (near
zero) with a higher separation of high saliency features away from zero for
saliency guided training.

Figure 7: Saliency maps and saliency distribution for Traditional an Saliency
Guided Training on MNIST

Figure 8: Saliency maps and saliency distribution for Traditional an Saliency
Guided Training on CIFAR10

Figure 9: Saliency maps and saliency distribution for Traditional an Saliency
Guided Training on BIRD

#### Model Accuracy Drop

We compare interpretable and traditional training for different saliency
methods with modification-based evaluation. Each experiment is repeated five
times. Figure 10 shows the mean and standard error for model degradation on
different gradient-based methods.

Figure 10:  The mean and standard error for model accuracy drop when removing
features with high saliency using traditional and saliency guided training for
different gradient-based methods against a random baseline. A steeper drop
indicates better performance. We find that regardless of the saliency method
used, the performance improves by saliency guided training.

#### Fine-tuning with Saliency guided Training

We investigate the effect of training traditionally and fine-tuning with
saliency guided training. This would be particularly useful for large datasets
like imagenet. Table 4 shows the area under accuracy drop curve (AUC) on MNIST
Figure LABEL:fig:AccuracyDropAll for gradient when training traditionally,
training using saliency guided procedure and fine-tuning (smaller AUC
indicates better performance). We find that fine-tuning improves the
performance over traditionally trained networks.

Training Procedures | AUC
---|---
Traditional | 3360.4
Saliency Guided | 1817.6
Fine-tuned | 2258.8
Table 4: Area under accuracy drop curve on MNIST for different training
procedures

Note that, there is not much gain in training performance when training from
scratch versus fine-tuning for small datasets like MNIST. However, for larger
datasets like CIFAR10, we observed a clear decrease in the number of epochs
when fine-tuning the network. The number of epochs for traditional training
CIFAR10 is on average 118, saliency training is 124 while fine-tuning takes
only 70 epochs.

### Saliency Guided Training for Language

We compare the interpretability of different models trained on language tasks
using the ERASER [11] benchmark.

##### Datasets

For all datasets, words embbedding were generated from Glove [40] and a
bidirectional LSTM [17] was used for classifications. Details about each
dataset is available in Table 5

Dataset | # Training | # Testing | # Classes | Tokens | Sentences | Test Accuracy | $\lambda$ | $k$
---|---|---|---|---|---|---|---|---
|  |  |  |  |  | Tradtional | Sal. Guided |  | (as a % of tokens)
Movie Review | 1600 | 200 | 2 | 774 | 36.8 | 0.8890 | 0.8980 | 1 | 60%
FEVER | 97957 | 6111 | 2 | 327 | 12.1 | 0.7234 | 0.7255 | 1 | 80%
e-SNLI | 911928 | 16429 | 3 | 16 | 1.7 | 0.9026 | 0.9068 | 1 | 70%

Table 5:  Overview of datasets in the ERASER benchmark. Number of labels,
dataset size, and average numbers of sentences and tokens in each document.
$k$ is the percentage of overall tokens within a particular document.

##### Masking

For language tasks, masking is a bit more tricky. We tried multiple masking
function, including:

  * •

Removing the masking function creates new input such that $\widetilde{X}$
contains only high salient word from the original input $X$.

  * •

Replace with token “[UNK]” the masking function replaces the low salient word
with the token “[UNK]” i.e., unknown.

  * •

Replace with token “[SEP]” the masking function replaces the low salient word
with the token “[SEP]” i.e., white space.

  * •

Replace with random word the masking function replaces the low salient with a
random word from vocabulary.

  * •

Replace with last high salient word the masking function replaces the low
salient word with the previous high salient word.

Over the three datasets, we found that the last masking function (replace with
last high salient word) gave the best results. We believe that the masking
function can also be dataset-dependent. This particular experiment aims to
prove that saliency guided training improves interpretability on language
tasks. We will consider finding the optimal masking function for different
language tasks in our future work.

##### Metrics

ERASER provides two metrics to measure interpretability. Comprehensiveness
evaluates if all features needed to make a prediction are selected. To
calculate an explanation comprehensiveness, a new input $\mkern
1.5mu\overline{\mkern-1.5muX\mkern-1.5mu}\mkern 1.5mu_{i}$ is created such
that $\mkern 1.5mu\overline{\mkern-1.5muX\mkern-1.5mu}\mkern
1.5mu_{i}=X_{i}-R_{i}$ where $R_{i}$ is predicted rationales. Let
$f_{\theta}\left(X_{i}\right)_{j}$ be the prediction of model for class $j$.
The model comprehensiveness is calculated as:

$\text{Comprehensiveness}=f_{\theta}\left(X_{i}\right)_{j}-f_{\theta}\left(\mkern
1.5mu\overline{\mkern-1.5muX\mkern-1.5mu}\mkern 1.5mu_{i}\right)_{j}$

A high score here implies that the explanation removed was influential in the
predictions. The second metric is Sufficiency that evaluates if the extracted
explanations contain enough signal to make a prediction. The following
equation gives the explanation sufficiency:

$\text{Sufficiency}=f_{\theta}\left(X_{i}\right)_{j}-f_{\theta}\left(R_{i}\right)_{j}$

A lower score implies that the explanations are adequate for a model
prediction.

To evaluate the faithfulness of continuous importance scores assigned to
tokens by models, the soft score over features provided by the model is
converted into discrete rationales $R_{i}$ by taking the top-$k_{d}$ values,
where $k_{d}$ is a threshold for dataset $d$. Denoting the tokens up to and
including bin $k$, for instance, $i$ by $R_{ik}$, an aggregate
comprehensiveness measure is defined as:

$\frac{1}{|\mathcal{B}|+1}\left(\sum_{k=0}^{|\mathcal{B}|}f_{\theta}\left(X_{i}\right)_{j}-f_{\theta}\left(\mkern
1.5mu\overline{\mkern-1.5muX\mkern-1.5mu}\mkern 1.5mu_{ik}\right)_{j}\right)$

Sufficiency is defined similarly. Here tokens are grouped into k = 5 bins by
grouping them into the top $1\%,5\%,10\%,20\%$ and $50\%$ of tokens, with
respect to the corresponding importance score. This metrics is referred to as
Area Over the Perturbation Curve (AOPC). For reference, we report these when
random scores are assigned to tokens. Results are shown in the main paper
Table 1.

### Saliency Guided Training for Time Series

We evaluated saliency guided training on a multivariate time series, both
quality on multivariate time series MNIST and quantitatively through synthetic
data.

#### Saliency Maps Quality for Multivariate Time Series

We compare the saliency maps produced on MNIST treated as a multivariate time
series with 28 time steps each having 28 features. Figure 11, Figure 12, and
Figure 13 shows the saliency maps produced by different saliency methods for
Temporal Convolutional Network (TCN), LSTM with Input-Cell Attention and,
Transformers respectively. There is a visible improvement in saliency quality
across different networks when saliency guided training is used. The most
significant improvement was found in TCNs.

Figure 11: Saliency maps produced for (TCN, saliency method) pairs. Figure 12:
Saliency maps produced for (LSTM with Input-Cell Attention, saliency method)
pairs.  Figure 13: Saliency maps produced for (Transformers, saliency method)
pairs.

#### Quantitative Analysis on Synthetic Data

We evaluated saliency guided training on a multivariate time series benchmark
proposed by Ismail et al. [21]. The benchmark consists of 10 synthetic
datasets, each examining different design aspects in typical time series
datasets. Properties of each dataset is shown in Figure 14. Informative
features are highlighted by the addition of a constant $\mu$ to the positive
class and subtraction of $\mu$ from the negative class. For the following
experiments $\mu=1$. Details of each dataset is available in table 6.

Figure 14: Figure from Ismail et al. [21]: Different evaluation datasets used
for benchmarking saliency methods. Some datasets have multiple variations
shown as sub-levels. N/S: normal and small shapes, T/F: temporal and feature
positions, M: moving shape. All datasets are trained for binary
classification. Examples are shown above each dataset, where dark red/blue
shapes represent informative features.

Dataset | # Training | # Testing | # Time Steps | # Feature | # Informative | # Informative
---|---|---|---|---|---|---
|  |  |  |  | Time steps | Features
Middle | 1000 | 100 | 50 | 50 | 30 | 30
Small Middle | 1000 | 100 | 50 | 50 | 15 | 15
Moving Middle | 1000 | 100 | 50 | 50 | 30 | 30
Moving Small Middle | 1000 | 100 | 50 | 50 | 15 | 15
Rare Time | 1000 | 100 | 50 | 50 | 6 | 40
Moving Rare Time | 1000 | 100 | 50 | 50 | 6 | 40
Rare Features | 1000 | 100 | 50 | 50 | 40 | 6
Moving Rare Features | 1000 | 100 | 50 | 50 | 40 | 6
Postional Time | 1000 | 100 | 50 | 50 | 20 | 20
Postional Feature | 1000 | 100 | 50 | 50 | 20 | 20

Table 6: Synthetic dataset details: Number of training samples, number of
testing samples, number of time steps per sample, number of features per time
step, number of time steps with informative features, and number of
informative features in an informative time step.

Following Ismail et al. [21], we compare 4 neural architectures: LSTM [17],
LSTM with Input-Cell Attention [20], Temporal Convolutional Network (TCN) [29]
and, Transformers [59]. Each (neural architecture, dataset) pair was trained
both traditionally and using saliency guided training. Test accuracy is
reported in Table 7

Datasets | LSTM | LSTM+ Input-Cell | TCN | Transformer
---|---|---|---|---
Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal.
Middle | 99 | 100 | 100 | 100 | 100 | 100 | 100 | 100
Small Middle | 100 | 100 | 99 | 100 | 100 | 100 | 100 | 100
Moving Middle | 100 | 100 | 100 | 100 | 100 | 100 | 99 | 100
Moving Small Middle | 100 | 100 | 99 | 100 | 100 | 100 | 99 | 100
Rare Time | 100 | 100 | 99 | 100 | 100 | 100 | 100 | 100
Moving Rare Time | 100 | 100 | 100 | 100 | 100 | 100 | 99 | 100
Rare Features | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100
Moving Rare Features | 99 | 100 | 99 | 99 | 100 | 100 | 99 | 100
Positional Time | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100
Positional Feature | 100 | 100 | 99 | 100 | 99 | 100 | 100 | 100

Table 7: Test accuracy of different (neural architecture, dataset) pairs.

Quantitatively measuring the interpretability of a (neural architecture,
saliency method) pair involves applying the saliency method, ranking features
according to the saliency values, replacing high salient features with
uninformative features from the original distribution at different
percentages. Finally, we measure the model accuracy drop, weighted precision,
and recall.

The area under precision curve (AUP) and the area under the recall curve (AUR)
are calculated by the precision/recall values at different levels of
degradation. Similar to Ismail et al. [21], we compare the AUP and AUR with a
random baseline; since the baseline might be different for different models,
we reported the difference between metrics values generated using the saliency
method and the baseline. All experiments were ran 5 times the mean Diff(AUP),
and Diff(AUR) is shown in Tables [8-11].

The results in Tables [8-11] show the follows: LSTM: Saliency guided training
along with Integrated Gradient has the best precision and recall. LSTM with
Input Cell Attention: Saliency guided training improves the performance of
different saliency methods and datasets. DeepSHAP gives the best precision,
while DeepSHAP gives the best recall. TCN: overall, saliency guided training
improves the performance of different saliency methods and datasets.
Integrated Gradient, Gradient SHAP, and DeepSHAP are best performing saliency
methods. Transformers: have the worst interpretability. Using saliency guided
training improved recall but not precision.

Metric | Datasets | $\lambda$ | $k$ | Gradient | Integrated Gradient | DeepLIFT | Gradient SHAP | DeepSHAP | SmoothGrad
---|---|---|---|---|---|---|---|---|---
Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal.
Diff(AUP) | Middle | 1 | 30% | -0.280 | -0.280 | -0.261 | -0.036 | -0.267 | -0.270 | -0.263 | -0.124 | -0.267 | -0.271 | -0.283 | -0.269
Small Middle | 1 | 60% | -0.071 | -0.066 | -0.053 | 0.052 | -0.070 | -0.055 | -0.056 | -0.033 | -0.070 | -0.055 | -0.072 | -0.044
Moving Middle | 1 | 60% | -0.265 | -0.277 | -0.218 | -0.169 | -0.237 | -0.264 | -0.222 | -0.237 | -0.239 | -0.264 | -0.259 | -0.263
Moving Small Middle | 1 | 5% | -0.059 | -0.060 | -0.035 | 0.051 | -0.043 | -0.045 | -0.042 | -0.013 | -0.044 | -0.046 | -0.056 | -0.037
Rare Time | 1 | 30% | -0.076 | -0.076 | -0.075 | -0.065 | -0.076 | -0.076 | -0.075 | -0.071 | -0.076 | -0.076 | -0.076 | -0.068
Moving Rare Time | 1 | 50% | -0.067 | -0.058 | -0.042 | 0.016 | -0.053 | -0.042 | -0.045 | -0.010 | -0.054 | -0.043 | -0.061 | -0.032
Rare Feature | 1 | 30% | -0.063 | -0.075 | -0.039 | 0.006 | -0.047 | -0.073 | -0.038 | -0.027 | -0.048 | -0.073 | -0.069 | -0.059
Moving Rare Feature | 1 | 10% | -0.062 | -0.069 | -0.021 | 0.012 | -0.040 | -0.056 | -0.032 | -0.029 | -0.041 | -0.056 | -0.059 | -0.044
Postional Time | 1 | 30% | -0.116 | -0.119 | -0.040 | -0.006 | -0.107 | -0.112 | -0.058 | -0.046 | -0.108 | -0.113 | -0.111 | -0.102
Postional Feature | 1 | 2% | -0.064 | -0.104 | -0.042 | -0.104 | -0.028 | -0.089 | -0.043 | -0.105 | -0.031 | -0.091 | -0.055 | -0.053
Diff(AUR) | Middle | 1 | 30% | 0.072 | 0.076 | 0.128 | 0.153 | 0.125 | 0.135 | 0.122 | 0.132 | 0.114 | 0.126 | 0.070 | 0.031
Small Middle | 1 | 60% | -0.043 | 0.037 | 0.048 | 0.157 | 0.029 | 0.116 | 0.038 | 0.129 | 0.007 | 0.102 | -0.032 | 0.009
Moving Middle | 1 | 60% | 0.060 | 0.073 | 0.119 | 0.124 | 0.110 | 0.124 | 0.119 | 0.117 | 0.099 | 0.115 | 0.061 | 0.042
Moving Small Middle | 1 | 5% | -0.032 | -0.004 | 0.046 | 0.135 | 0.043 | 0.073 | 0.042 | 0.093 | 0.025 | 0.060 | -0.023 | -0.025
Rare Time | 1 | 30% | -0.244 | -0.137 | -0.132 | 0.043 | -0.169 | -0.021 | -0.116 | 0.005 | -0.189 | -0.043 | -0.145 | -0.108
Moving Rare Time | 1 | 50% | -0.222 | -0.070 | -0.092 | 0.075 | -0.103 | 0.018 | -0.065 | 0.060 | -0.131 | 0.002 | -0.144 | -0.035
RareFeature | 1 | 30% | 0.182 | 0.197 | 0.219 | 0.218 | 0.217 | 0.223 | 0.216 | 0.216 | 0.211 | 0.219 | 0.191 | 0.166
Moving Rare Feature | 1 | 10% | 0.143 | 0.162 | 0.191 | 0.196 | 0.191 | 0.202 | 0.194 | 0.196 | 0.183 | 0.197 | 0.162 | 0.107
Postional Time | 1 | 30% | -0.032 | -0.073 | 0.072 | 0.119 | 0.029 | 0.021 | 0.046 | 0.082 | 0.012 | 0.001 | -0.019 | -0.064
Postional Feature | 1 | 2% | -0.053 | -0.070 | 0.016 | -0.005 | -0.002 | -0.005 | 0.004 | -0.009 | -0.018 | -0.025 | -0.056 | -0.083

Table 8: Difference in weighted AUP and AUR for (LSTM, saliency method) pairs.
Overall, the best preference was achieved when using Integrated Gradients as a
saliency method and saliency guided training as a training procedure.

Metric | Datasets | $\lambda$ | $k$ | Gradient | Integrated Gradient | DeepLIFT | Gradient SHAP | DeepSHAP | SmoothGrad
---|---|---|---|---|---|---|---|---|---
Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal.
Diff(AUP) | Middle | 1 | 40% | 0.014 | 0.046 | 0.233 | 0.252 | 0.218 | 0.237 | 0.244 | 0.261 | 0.232 | 0.247 | -0.006 | 0.026
Small Middle | 1 | 60% | 0.049 | 0.150 | 0.161 | 0.273 | 0.169 | 0.305 | 0.170 | 0.273 | 0.180 | 0.312 | 0.038 | 0.091
Moving Middle | 1 | 80% | 0.044 | 0.082 | 0.262 | 0.251 | 0.260 | 0.256 | 0.276 | 0.256 | 0.277 | 0.261 | 0.010 | 0.044
Moving Small Middle | 1 | 5% | 0.044 | 0.055 | 0.181 | 0.201 | 0.179 | 0.196 | 0.187 | 0.200 | 0.190 | 0.204 | 0.022 | 0.029
Rare Time | 1 | 40% | 0.186 | 0.278 | 0.271 | 0.378 | 0.323 | 0.412 | 0.279 | 0.373 | 0.338 | 0.424 | 0.133 | 0.209
Moving Rare Time | 1 | 80% | 0.144 | 0.276 | 0.233 | 0.388 | 0.269 | 0.417 | 0.238 | 0.381 | 0.282 | 0.429 | 0.103 | 0.167
Rare Feature | 1 | 30% | 0.032 | 0.101 | 0.163 | 0.270 | 0.166 | 0.266 | 0.174 | 0.278 | 0.180 | 0.274 | 0.039 | 0.105
Moving Rare Feature | 1 | 5% | -0.002 | -0.004 | 0.120 | 0.124 | 0.116 | 0.116 | 0.124 | 0.127 | 0.126 | 0.126 | -0.003 | -0.004
Postional Time | 1 | 40% | 0.117 | 0.186 | 0.184 | 0.225 | 0.236 | 0.314 | 0.197 | 0.252 | 0.248 | 0.316 | 0.093 | 0.187
Postional Feature | 1 | 5% | -0.021 | 0.007 | 0.072 | 0.083 | 0.080 | 0.113 | 0.089 | 0.101 | 0.088 | 0.122 | -0.031 | -0.012
Diff(AUR) | Middle | 1 | 40% | 0.028 | 0.084 | 0.163 | 0.176 | 0.160 | 0.180 | 0.162 | 0.173 | 0.157 | 0.177 | -0.001 | 0.044
Small Middle | 1 | 60% | 0.064 | 0.176 | 0.186 | 0.217 | 0.189 | 0.217 | 0.182 | 0.212 | 0.183 | 0.213 | 0.031 | 0.159
Moving Middle | 1 | 80% | 0.060 | 0.117 | 0.174 | 0.180 | 0.175 | 0.187 | 0.173 | 0.177 | 0.175 | 0.183 | 0.021 | 0.072
Moving Small Middle | 1 | 5% | 0.079 | 0.101 | 0.202 | 0.201 | 0.199 | 0.194 | 0.198 | 0.196 | 0.194 | 0.186 | 0.029 | 0.052
Rare Time | 1 | 40% | 0.139 | 0.203 | 0.214 | 0.225 | 0.214 | 0.233 | 0.211 | 0.223 | 0.211 | 0.233 | 0.103 | 0.191
Moving Rare Time | 1 | 80% | 0.118 | 0.213 | 0.198 | 0.226 | 0.200 | 0.233 | 0.193 | 0.224 | 0.194 | 0.232 | 0.070 | 0.197
RareFeature | 1 | 30% | 0.077 | 0.181 | 0.196 | 0.223 | 0.197 | 0.224 | 0.193 | 0.222 | 0.193 | 0.223 | 0.074 | 0.172
Moving Rare Feature | 1 | 5% | 0.059 | 0.039 | 0.188 | 0.189 | 0.191 | 0.186 | 0.182 | 0.183 | 0.186 | 0.180 | 0.038 | 0.028
Postional Time | 1 | 40% | 0.140 | 0.201 | 0.188 | 0.200 | 0.203 | 0.225 | 0.185 | 0.202 | 0.201 | 0.224 | 0.109 | 0.188
Postional Feature | 1 | 5% | -0.017 | 0.043 | 0.141 | 0.146 | 0.145 | 0.166 | 0.141 | 0.150 | 0.132 | 0.157 | -0.041 | 0.005

Table 9: The difference in weighted AUP and AUR for different (LSTM with
Input-Cell Attention, saliency method) pairs. The use of saliency guided
training improved the performance of most saliency methods. Overall, DeepSHAP
and DeepLIFT produced the best precision and recall, respectively, when
combined with saliency guided training.

Metric | Datasets | $\lambda$ | $k$ | Gradient | Integrated Gradient | DeepLIFT | Gradient SHAP | DeepSHAP | SmoothGrad
---|---|---|---|---|---|---|---|---|---
Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal.
Diff(AUP) | Middle | 1 | 50% | 0.127 | 0.217 | 0.283 | 0.393 | 0.350 | 0.398 | 0.290 | 0.384 | 0.365 | 0.416 | 0.090 | 0.194
Small Middle | 1 | 40% | 0.164 | 0.260 | 0.299 | 0.433 | 0.312 | 0.419 | 0.302 | 0.418 | 0.328 | 0.442 | 0.156 | 0.253
Moving Middle | 1 | 70% | 0.122 | 0.197 | 0.287 | 0.342 | 0.332 | 0.367 | 0.286 | 0.329 | 0.345 | 0.387 | 0.047 | 0.182
Moving Small Middle | 1 | 80% | 0.065 | 0.043 | 0.194 | 0.151 | 0.169 | 0.191 | 0.190 | 0.152 | 0.183 | 0.200 | 0.037 | 0.023
Rare Time | 1 | 50% | 0.184 | 0.290 | 0.314 | 0.363 | 0.324 | 0.309 | 0.314 | 0.360 | 0.352 | 0.319 | 0.177 | 0.226
Moving Rare Time | 1 | 50% | 0.142 | 0.182 | 0.260 | 0.333 | 0.257 | 0.243 | 0.258 | 0.330 | 0.275 | 0.251 | 0.122 | 0.179
Rare Feature | 1 | 30% | 0.058 | 0.244 | 0.246 | 0.451 | 0.252 | 0.422 | 0.249 | 0.453 | 0.286 | 0.450 | 0.085 | 0.259
Moving Rare Feature | 1 | 5% | -0.003 | 0.004 | 0.116 | 0.134 | 0.112 | 0.114 | 0.122 | 0.129 | 0.122 | 0.123 | 0.007 | 0.005
Postional Time | 1 | 70% | 0.115 | 0.072 | 0.180 | 0.114 | 0.233 | 0.069 | 0.187 | 0.117 | 0.237 | 0.035 | 0.106 | 0.066
Postional Feature | 1 | 10% | 0.082 | 0.176 | 0.151 | 0.199 | 0.136 | 0.162 | 0.155 | 0.203 | 0.137 | 0.175 | 0.058 | 0.159
Diff(AUR) | Middle | 1 | 50% | 0.133 | 0.161 | 0.190 | 0.207 | 0.202 | 0.209 | 0.188 | 0.205 | 0.201 | 0.205 | 0.054 | 0.128
Small Middle | 1 | 40% | 0.086 | 0.230 | 0.194 | 0.240 | 0.202 | 0.240 | 0.189 | 0.239 | 0.196 | 0.241 | 0.039 | 0.230
Moving Middle | 1 | 70% | 0.134 | 0.144 | 0.191 | 0.195 | 0.201 | 0.208 | 0.186 | 0.194 | 0.201 | 0.203 | 0.036 | 0.121
Moving Small Middle | 1 | 80% | 0.118 | 0.117 | 0.204 | 0.199 | 0.193 | 0.195 | 0.196 | 0.196 | 0.186 | 0.190 | -0.001 | 0.065
Rare Time | 1 | 50% | 0.173 | 0.215 | 0.199 | 0.233 | 0.225 | 0.221 | 0.193 | 0.230 | 0.226 | 0.204 | 0.125 | 0.151
Moving Rare Time | 1 | 50% | 0.106 | 0.198 | 0.177 | 0.220 | 0.195 | 0.189 | 0.167 | 0.224 | 0.191 | 0.179 | -0.057 | 0.149
RareFeature | 1 | 30% | 0.152 | 0.222 | 0.222 | 0.239 | 0.223 | 0.239 | 0.219 | 0.239 | 0.224 | 0.239 | 0.130 | 0.217
Moving Rare Feature | 1 | 5% | 0.101 | 0.122 | 0.198 | 0.204 | 0.206 | 0.205 | 0.195 | 0.201 | 0.196 | 0.198 | 0.048 | 0.055
Postional Time | 1 | 70% | 0.126 | 0.128 | 0.156 | 0.172 | 0.194 | 0.160 | 0.147 | 0.165 | 0.181 | 0.110 | 0.039 | 0.102
Postional Feature | 1 | 10% | 0.126 | 0.174 | 0.177 | 0.196 | 0.180 | 0.175 | 0.172 | 0.194 | 0.164 | 0.154 | 0.049 | 0.160

Table 10: The difference in weighted AUP and AUR for different (TCN, saliency
method) pairs. The use of saliency guided training improved the performance of
most saliency methods. Overall, when combined with saliency guided training,
Integrated Gradients and DeepSHAP produced the best precision. For recall,
Integrated Gradients, DeepLift, Gradient SHAP, and DeepSHAP seem to perform
similarly, again, the best performance was achieved when saliency guided
training is used.

Metric | Datasets | $\lambda$ | $k$ | Gradient | Integrated Gradient | DeepLIFT | Gradient SHAP | DeepSHAP | SmoothGrad
---|---|---|---|---|---|---|---|---|---
Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal. | Trad. | Sal.
Diff(AUP) | Middle | 1 | 30% | -0.179 | -0.213 | 0.051 | -0.004 | -0.116 | -0.176 | 0.067 | -0.064 | -0.062 | -0.222 | -0.069 | -0.150
Small Middle | 1 | 60% | -0.034 | -0.057 | 0.054 | 0.042 | -0.018 | -0.034 | 0.066 | 0.024 | 0.009 | -0.060 | 0.006 | -0.022
Moving Middle | 1 | 90% | -0.188 | -0.146 | 0.062 | 0.018 | -0.142 | -0.067 | 0.065 | -0.011 | -0.130 | -0.143 | -0.091 | -0.157
Moving Small Middle | 1 | 70% | -0.002 | -0.008 | 0.031 | 0.039 | 0.017 | 0.029 | 0.026 | 0.037 | 0.036 | 0.016 | -0.021 | -0.035
Rare Time | 1 | 50% | 0.038 | -0.006 | 0.118 | 0.057 | -0.019 | 0.017 | 0.132 | 0.049 | 0.014 | -0.010 | -0.029 | -0.031
Moving Rare Time | 1 | 50% | 0.066 | 0.062 | 0.110 | 0.049 | -0.021 | 0.046 | 0.117 | 0.055 | -0.009 | 0.033 | -0.026 | -0.027
Rare Feature | 1 | 30% | -0.049 | -0.045 | 0.029 | 0.139 | -0.002 | -0.004 | 0.033 | 0.088 | 0.008 | -0.015 | 0.005 | 0.028
Moving Rare Feature | 1 | 10% | -0.034 | -0.031 | 0.041 | 0.055 | 0.008 | 0.014 | 0.038 | 0.049 | 0.008 | 0.022 | -0.003 | -0.013
Postional Time | 1 | 60% | -0.060 | -0.078 | 0.084 | 0.029 | -0.048 | -0.047 | 0.102 | -0.001 | 0.013 | -0.072 | 0.026 | -0.057
Postional Feature | 1 | 10% | -0.094 | -0.099 | 0.032 | 0.012 | -0.061 | -0.097 | 0.046 | 0.008 | -0.029 | -0.098 | 0.019 | 0.006
Diff(AUR) | Middle | 1 | 30% | 0.087 | 0.053 | 0.167 | 0.146 | 0.155 | 0.112 | 0.157 | 0.111 | 0.119 | -0.051 | 0.040 | -0.025
Small Middle | 1 | 60% | 0.085 | 0.030 | 0.186 | 0.189 | 0.128 | 0.113 | 0.173 | 0.171 | 0.077 | -0.025 | 0.060 | 0.036
Moving Middle | 1 | 90% | 0.071 | 0.134 | 0.164 | 0.185 | 0.136 | 0.181 | 0.150 | 0.183 | 0.057 | 0.130 | 0.019 | 0.040
Moving Small Middle | 1 | 70% | 0.118 | 0.137 | 0.171 | 0.177 | 0.157 | 0.171 | 0.160 | 0.171 | 0.098 | 0.117 | -0.004 | -0.019
Rare Time | 1 | 50% | 0.152 | 0.139 | 0.206 | 0.172 | 0.116 | 0.135 | 0.199 | 0.157 | 0.077 | 0.088 | -0.027 | -0.073
Moving Rare Time | 1 | 50% | 0.184 | 0.185 | 0.198 | 0.170 | 0.124 | 0.175 | 0.186 | 0.168 | 0.059 | 0.127 | -0.033 | -0.013
RareFeature | 1 | 30% | 0.115 | 0.152 | 0.184 | 0.217 | 0.187 | 0.188 | 0.173 | 0.203 | 0.144 | 0.129 | 0.087 | 0.135
Moving Rare Feature | 1 | 10% | 0.101 | 0.122 | 0.179 | 0.183 | 0.174 | 0.180 | 0.165 | 0.176 | 0.125 | 0.149 | 0.060 | 0.034
Postional Time | 1 | 60% | 0.091 | 0.071 | 0.193 | 0.172 | 0.154 | 0.150 | 0.184 | 0.151 | 0.145 | 0.059 | 0.103 | -0.004
Postional Feature | 1 | 10% | 0.017 | 0.013 | 0.170 | 0.149 | 0.123 | 0.057 | 0.160 | 0.131 | 0.105 | -0.072 | 0.094 | 0.073

Table 11: The difference in weighted AUP and AUR for different (Transformers,
saliency method) pairs. In this benchmark, Transformers seem to have the worst
interpretability. Using saliency guided training improved recall but not
precision. Overall best precision was achieved when combining traditional
training with Gradient SHAP. While best recall was achieved when using
saliency guided training and Integrated Gradients.