# Token Drop mechanism for Neural Machine Translation

Huaao Zhang
Soochow University / Suzhou China
<EMAIL_ADDRESS>
Xiangyu Duan
Soochow University / Suzhou China
<EMAIL_ADDRESS>
Shigui Qiu
Soochow University / Suzhou China
<EMAIL_ADDRESS>
Min Zhang
Soochow University / Suzhou China
<EMAIL_ADDRESS>
* Corresponding author.

###### Abstract

Neural machine translation with millions of parameters is vulnerable to
unfamiliar inputs. We propose Token Drop to improve generalization and avoid
overfitting for the NMT model. Similar to word dropout, whereas we replace
dropped token with a special token instead of setting zero to words. We
further introduce two self-supervised objectives: Replaced Token Detection and
Dropped Token Prediction. Our method aims to force model generating target
translation with less information, in this way the model can learn textual
representation better. Experiments on Chinese-English and English-Romanian
benchmark demonstrate the effectiveness of our approach and our model achieves
significant improvements over a strong Transformer baseline111Our code is
released at https://github.com/zhajiahe/Token_Drop.

##  1 Introduction

11footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

Neural machine translation (NMT) achieved enormous success in advancing the
quality of translation [Bahdanau et al. (2015, Vaswani et al. (2017, Gehring
et al. (2017]. In spite of the impressive performance, NMT models are still
vulnerable to perturbations in the input sentences [Belinkov and Bisk (2018,
Cheng et al. (2019] , i.e. a tiny perturbation will affect hidden
representation and lead to low quality of translation [Zhao et al. (2018].
Moreover, NMT commonly consists of millions of parameters, which making it
prone to overfitting especially in low resource scene.

A natural way to improve generalization is synthesizing natural noise
[Karpukhin et al. (2019] or adopting arbitrary noise [Cheng et al. (2018,
Ebrahimi et al. (2018]. Another way is exploring regularization techniques to
avoid overfitting [Miceli et al. (2017], making model robust to unseen or
unfamiliar inputs. However, as discrete data, the text is hard to retain the
semantic information after corruption.

In this paper, we propose Token Drop to prevent overfitting and improve
generalization. Different from standard dropout [Srivastava et al. (2014] that
drops neurons in network randomly, we drop tokens of the input sentences. In
order to retain semantic information, we replace tokens with a special symbol
$<\textit{unk}>$ . This allows model learn hidden representation from rest
token’s context, and predict target translation condition on latent variable.
On the one hand, our method allows model meeting exponentially different
sentences can be explained as data augmentation; On the other hand, our method
corrupts input sentences with natural noise can be seen as regularization term
for NMT.

We investigate two self-supervised objectives: Replaced Token Detection and
Dropped Token Prediction. Considering our Token Drop method regularize
parameters by weakening model inputs, making NMT suitable for applying self-
supervised objective. During training: (1) use a discriminator to detect
whether input tokens are dropped or not; (2) leverage hidden state to predict
original tokens of dropped tokens inspired by Cloze task [Devlin et al.
(2019]. Both of them guide model to generate semantically similar
representation, leading to a better generalization capacity.

##  2 Token Drop Training

Standard dropout prevents overfitting by setting input neurons or hidden
neurons to zero with a certain probability $p$ [Hinton et al. (2012,
Srivastava et al. (2014]. whereas we consider the input sequences of machine
translation models instead of the network’s neurons, which named Token Drop.
Given a input tokens sequence of sentence $X=\\{x_{1},x_{2},...,x_{n}\\}$ and
posit a $|X|$ independent drop rate $p$. The token $x_{i}$ in $X$ will be
dropped if $m_{i}$ is 1 . This process as Equation 1:

$\textbf{m}\sim
Bernoulli(p)\qquad\hat{X}=\textbf{REPLACE}(X,\textbf{m},<\textit{unk}>)$ (1)
$\mathcal{L}_{M}=-\log\sum P(Y|\hat{X},\theta_{M})$ (2)

The Token Drop can be interpreted as data augmentation and regularization
technique for NMT. Seeing that NMT model commonly adopts encoder and decoder
architecture, therefore our method drops tokens for both source and target
inputs. For the source side, model encoder learns intermediate representation
by exponentially different incomplete sentences. For the target side, model
decoder generates target translation condition on latent variable, weakening
the constraint caused by teacher forcing. Both of them receives incomplete
information from inputs, simulating the real situation (e.g unknown or
unfamiliar data) at test time.

###  2.1 Token Drop methods

We adopt three drop strategy for Token Drop:

Zero-Out is introduced by ?), different from the standard dropout, the method
drops full word by setting zero to word embedding during training. The
deficiency is zero vector can not learn representation from its context in the
self-attention layer.

Drop-Tag [Kågebäck and Salomonsson (2016] replaces token with a
$<\textit{dropped}>$ tag. The tag is subsequently treated just like any other
word in the vocabulary and has a corresponding word embedding that is trained.
We adopt this technique for NMT to learn better feature representation.

Unk-Tag replaces token with generic unknown word token $<\textit{unk}>$. ?)
and ?) apply it to RNN decoder to force model make prediction by latent
variable. We found this perfectly suits for NMT system especially on self-
attention layers. Better than Drop-Tag method, it need not to add an extra
token as well as parameters.

###  2.2 Replaced Token Detection

We propose the Replaced Token Detection task to promote generalization ability
of the model encoder. We regard dropped information as self-supervised label,
following ?), we train a discriminator $D(G(x))$ to detect whether tokens are
dropped or not. On account of our dropped tokens are obvious to distinguish,
so we add a simple linear classifier to detect replaced tokens. The objective
is :

$\mathcal{L}_{RTD}=\mathbb{E}_{x\sim\mathcal{S}}[-\log
D(G(\mathbf{x}))]+\mathbb{E}_{\hat{x}\sim d(x)}[1-\log D(G(\hat{x}))]$ (3)

Where $d(x)$ denotes the dropped tokens. In our model, the encoder serves as a
generator $G$, which generates hidden state of input tokens. The discriminator
$D$ tries to distinguish whether a token is dropped or not, while the
generator $G$ has to produce similar representation for $x$ and $\hat{x}$,
making the model robust to noisy and unknown inputs.

###  2.3 Dropped Token Prediction

In consideration of our Token Drop model randomly replaces tokens of input
sentence, similar to Masked Language Model [Devlin et al. (2019], which masks
then predicts masked tokens by the rest of the tokens, making use of
contextual information. Accordingly, we propose Dropped Token Prediction
(DTP), predicting dropped tokens by their hidden states. The DTP objective is
:

$\mathcal{L}_{DTP}=-\log\sum_{\hat{x}\in d(x)}P(\hat{x}|X_{/d(x)},\theta_{M})$
(4) $P(\hat{x}|X_{/d(x)})=E(G(X_{/d(x)}))$ (5)

Where $d(x)$ and $X_{/d(x)}$ denote the dropped tokens and the rest tokens
respectively. $G$ is model encoder, $E(.)$ is prediction layer. In our
implementation of DTP, we adopt weight tying [Press and Wolf (2017] , that is
to share the same weight matrix between embedding layer and token prediction
classifier.

At the end we train our model jointly with DTP and RTD objective:

$\mathcal{L}=\mathcal{L}_{M}+\alpha\mathcal{L}_{RTD}+\beta\mathcal{L}_{DTP}$
(6)

##  3 Experiment

We conduct our approach on two machine translation benchmarks: LDC (ZH-EN) and
WMT16 (EN-RO)222http://www.statmt.org/wmt16/

###  3.1 Dataset and Evaluation

For ZH-EN translation, we used 1.25M sentence pairs extract from LDC corpus.
Byte-pair encoding is employed separate vocabulary of about 42K and 31K tokens
with 32K merge operations [Sennrich et al. (2016b]. we chose NIST06 as the
valid set and NIST02, NIST03, NIST04, NIST05, NIST08 as the test set, which
contains 878, 919, 1788, 1082, 1357 sentence pairs respectively. We measure
BLEU score with multi-bleu.pl333https://github.com/moses-
smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl. For WMT16 EN-RO
data which consists of 610K pairs, we adopt ?)’s preprocessing. The vocabulary
is 35K joint source and target subwords [Sennrich et al. (2016b]. we use
newstest-2016 as test set and report tokenized BLEU score.

###  3.2 Models and Settings

We adopt the Transformer model [Vaswani et al. (2017] implemented in PyTorch
in the fairseq-py toolkit [Ott et al. (2019]. We closely followed settings by
?)$(d_{model}=512,d_{hidden}=512,d_{FFN}=2048,n_{layer}=6,n_{head}=8)$ and
used dropout of $p_{dropout}=0.3$. As for our approach, we set drop rate
$p_{s}=0.15$ and $p_{t}=0.3$ respectively for ZH-EN task. For EN-RO, we set
$p_{s}=0.15$ and $p_{t}=0.2$. To train with DTP and RTD objective, we simply
set $\alpha,\beta=1$ .

##  4 Result

Models | LDC ZH-EN | WMT EN-RO
---|---|---
NIST02 | NIST03 | NIST04 | NIST05 | NIST08 | AVG | EN$\rightarrow$RO | RO$\rightarrow$ EN
Transformer | 47.17 | 46.78 | 47.46 | 47.97 | 38.01 | 45.47 | 34.60 | 33.96
Zero-Out | 47.86 | 47.12 | 48.86 | 47.90 | 38.65 | 46.08 | 35.21 | 35.00
Drop-Tag | 48.99 | 48.50 | 49.81 | 49.60 | 39.31 | 47.24 | 35.34 | 35.19
Unk-Tag | 48.96 | 48.52 | 49.50 | 49.49 | 39.31 | 47.16 | 35.28 | 35.13
\+ RTD | 49.00 | 49.36 | 49.43 | 49.25 | 39.92 | 47.39 | 35.49 | 35.38
\+ DTP | 49.16 | 49.19 | 49.99 | 49.83 | 40.20 | 47.75 | 35.75 | 35.38
\+ RTD&DTP | 49.52 | 49.29 | 50.17 | 49.82 | 40.38 | 47.84 | 35.67 | 35.69
Table 1:  BLEU scores on test set for LDC Chinese-English and WMT16 English-
Romanian tasks

The results of our experiment on NIST Chinese-English and WMT16 English-
Romanian tasks are shown in Table 1. We first conduct Token Drop through three
drop methods (Zero-out, Drop-Tag, Unk-Tag), the results show that Token Drop
model significantly outperform baseline on two languages. Furthermore, we
combine Unk Tag method with DTP and RTD training objective, the results show
that both DTP and RTD provide a further improvement on Token Drop training.
Overall, we get a gain of 2.37, 1.15 and 1.73 BLEU score on three tasks
respectively.

Method | 0.00 | 0.05 | 0.10 | 0.15
---|---|---|---|---
Beseline | 47.44 | 39.26 | 30.83 | 23.20
[Cheng et al. (2019] | 46.95 | 44.20 | 41.71 | 39.89
Ours | 48.75 | 46.55 | 44.12 | 41.64
Table 2:  Result on incomplete inputs with different $p_{s}$ for ZH-EN valid
set (NIST06).

In order to demonstrate the generalization capacity of our model on real
situation. We constrain input information by replacing words with generic
unknown symbol $<\textit{unk}>$. For each sentence, we generate 100 noisy
sentences then report average BLEU score. We also compare our method with ?),
who introduced white-box adversarial noisy inputs to improve robustness. Table
2 reports our result on incomplete inputs, from where we can see our approach
outperforms previous method. This improvement confirms that our model obtains
larger generalization capacity over baseline.

Figure 1:  Effect of different source drop rate $p_{s}$
($p_{t}=0.3$) on LDC ZH-EN translation task

Figure 2:  Learning curves of baseline model
of our Token Drop model

To examine the impact of Token Drop, we train our model with different source
drop rate $p_{s}=[0.0,0.05,0.1,0.15,0.2,0.25,0.3]$. From Figure 2 we can see
that model training with a moderate drop rate $p$ would advanced in
performance significantly. Drop-Tag and Unk-Tag are quite similar, both of
them outperforms Zero-Out method. We plot learning curves of baseline model
and our Token Drop model. Figure 2 shows that with the increase of training
iterations, our model achieves lower and more stable perplexity than baseline,
demonstrating the effectiveness of our approach to prevent overfitting and
improve translation quality.

##  5 Related Work

Word Dropout ?) proposed word dropout as feature extractor for text
classification task. ?) and ?) applied word dropout to RNN decoder can be
regard as a smoothing technique. For machine translation task, ?) randomly set
zero to words of input sentence to prevent overfitting, advancing in
considerable performance on noisy dataset. In this paper, we explain word
dropout as data augmentation (i.e. allows model meeting exponentially
different sentences) and a regularization technique (i.e. weakens the encoder
and decoder, obtaining better intermediate representations).

##  6 Conclusion

In this paper, we have proposed Token Drop mechanism for neural machine
translation task. Inspired by self-supervised learning, we introduced Replaced
Token Detection and Dropped Token Prediction training objective. We found that
NMT model trained with Token Drop gains larger generalization capacity and
reduction in overfitting. Even without prior knowledge and additional
parameters, our proposed approach reports convincing results on neural machine
translation. In future work, we plan to investigate impact of dropping on
different words, e.g. word importance and word type.

## Acknowledgements

The authors would like to thank the anonymous reviewers for the helpful
comments. This work was supported by National Natural Science Foundation of
China (Grant No. 61673289, 61525205).

## References

  * [Bahdanau et al. (2015]  Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  2015\.  Neural machine translation by jointly learning to align and translate.  In Proc. of ICLR.
  * [Belinkov and Bisk (2018]  Yonatan Belinkov and Yonatan Bisk.  2018\.  Synthetic and natural noise both break neural machine translation.  In Proc. of ICLR.
  * [Bowman et al. (2016]  Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio.  2016\.  Generating sentences from a continuous space.  In Proc. of SIGNLL.
  * [Cheng et al. (2018]  Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu.  2018\.  Towards robust neural machine translation.  In Proc. of ACL.
  * [Cheng et al. (2019]  Yong Cheng, Lu Jiang, and Wolfgang Macherey.  2019\.  Robust neural machine translation with doubly adversarial inputs.  In Proc. of ACL.
  * [Clark et al. (2020]  Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  2020\.  Electra: Pre-training text encoders as discriminators rather than generators.  In Proc. of ICLR.
  * [Devlin et al. (2019]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proc. of NAACL.
  * [Ebrahimi et al. (2018]  Javid Ebrahimi, Daniel Lowd, and Dejing Dou.  2018\.  On adversarial examples for character-level neural machine translation.  In Proc. of COLING.
  * [Gehring et al. (2017]  Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin.  2017\.  Convolutional sequence to sequence learning.  In Proc. of ICML.
  * [Hinton et al. (2012]  Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov.  2012\.  Improving neural networks by preventing co-adaptation of feature detectors.  arXiv:1207.0580.
  * [Iyyer et al. (2015]  Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III.  2015\.  Deep unordered composition rivals syntactic methods for text classification.  In Proc. of ACL.
  * [Kågebäck and Salomonsson (2016]  Mikael Kågebäck and Hans Salomonsson.  2016\.  Word sense disambiguation using a bidirectional lstm.  arXiv:1606.03568.
  * [Karpukhin et al. (2019]  Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad.  2019\.  Training on synthetic noise improves robustness to natural noise in machine translation.  In Proc. of W-NUT.
  * [Lee et al. (2018]  Jason Lee, Elman Mansimov, and Kyunghyun Cho.  2018\.  Deterministic non-autoregressive neural sequence modeling by iterative refinement.  In Proc. of ACL.
  * [Miceli et al. (2017]  Barone Miceli, Valerio Antonio, Barry Haddow, Ulrich Germann, and Rico Sennrich.  2017\.  Regularization techniques for fine-tuning in neural machine translation.  In Proc. of EMNLP.
  * [Ott et al. (2019]  Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  2019\.  fairseq: A fast, extensible toolkit for sequence modeling.  In Proc. of NAACL.
  * [Press and Wolf (2017]  Ofir Press and Lior Wolf.  2017\.  Using the output embedding to improve language models.  In Proc. of EACL.
  * [Sennrich et al. (2016a]  Rico Sennrich, Barry Haddow, and Alexandra Birch.  2016a.  Edinburgh neural machine translation systems for WMT 16.  In Proc. of WMT.
  * [Sennrich et al. (2016b]  Rico Sennrich, Barry Haddow, and Alexandra Birch.  2016b.  Neural machine translation of rare words with subword units.  In Proc. of ACL.
  * [Srivastava et al. (2014]  Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  2014\.  Dropout: a simple way to prevent neural networks from overfitting.  JMLR.
  * [Vaswani et al. (2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Proc. of NIPS.
  * [Xie et al. (2017]  Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng.  2017\.  Data noising as smoothing in neural network language models.  arXiv:1703.02573.
  * [Yang et al. (2017]  Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick.  2017\.  Improved variational autoencoders for text modeling using dilated convolutions.  In Proc. of ICML.
  * [Zhao et al. (2018]  Zhengli Zhao, Dheeru Dua, and Sameer Singh.  2018\.  Generating natural adversarial examples.  In Proc. of ICLR.


# SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localization

Haofan Wang* Electrical and Computer Engineering Carnegie Mellon University
<EMAIL_ADDRESS>Rakshit Naidu* and Joy Michael Computer Science and
Engineering Manipal Institute of Technology
<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>Soumya Snigdha Kundu  Computer Science and
Engineering
SRM Institute
of Science and Technology
<EMAIL_ADDRESS>
###### Abstract

Interpretation of the underlying mechanisms of Deep Convolutional Neural
Networks has become an important aspect of research in the field of deep
learning due to their applications in high-risk environments. To explain these
black-box architectures there have been many methods applied so the internal
decisions can be analyzed and understood. In this paper, built on the top of
Score-CAM, we introduce an enhanced visual explanation in terms of visual
sharpness called SS-CAM, which produces centralized localization of object
features within an image through a smooth operation. We evaluate our method on
the ILSVRC 2012 Validation dataset, which outperforms Score-CAM on both
faithfulness and localization tasks.

††*Equal Contribution

##  I Introduction

The primary solution to vision problems in the past years have been
Convolutional Neural Networks (CNNs). There have been sufficient advancements
in its architectures[6][21] to cope with complex problems such as image
captioning[8], image classification[20], semantic segmentation[10] and many
other problems[13, 22, 7]. Despite their progressive nature of solving major
vision tasks, they still act like black box architectures and interpreting
these models has become a difficult task. As these architectures are applied
to highly sensitive environments such as medicine, finance and autonomous
transportation facilities, one has to know the construct for the basis of the
architecture’s findings to prevent any type of irreparable damage. The task
continues to become increasingly difficult as the complexity of the
architecture increases.

To avoid instances where sensitive issues are hampered, these unexplainable
architectures need to be justified with visual reasoning to explain the
decision that they make. Not only can the visual information help in
identifying the pit-falls of the architectures but also help in generating
insights to improve on the model as well. Identifying the pit-falls can help
in debugging the network. Models generally tend to mis-classify while
performing the above tasks but it takes a lot of time and effort to understand
why exactly the model is making such mistakes. The visual information can help
in pin-pointing the mistakes of the model. Additional insights can be those
related to improving the model in general to generate better localization,
faithfulness and gaining human trust. They can also help in solving the issues
explained in [1] which focuses on the racial and gender biases involved in
non-inclusive datasets which has become a raising concern for many developers.
Any architecture which fails to deal with racism and gender bias will not be
acceptable on any front.

[26] by the means of Deconvolution and [19] by the means of Guided
Backpropagation were the first ones to address the issue. But recent progress
came to notice with the culmination of attribution maps by giving importance
to the weights of each region in the image before the final Pooling layer.
Gradient-based visualisation, Perturbation-based visualisation and Class
Activation Mappings[27] are the 3 types of maps which help in developing the
visual aid. Gradient-based approaches generate saliency map using the
derivative of the target class score to the input image through
backpropagation [19]. Perturbing [14, 11] is also a commonly adopted technique
to find the region of interest in the image, which works by masking some
specific region in the input. The region that causes the largest drop on
target class is regarded as the important region. Other works[24] add
regularizers to make these attribution maps more robust and some packages[25]
have been developed

Our work builds on CAM-based approaches[27, 15, 2], which obtain attribution
maps by a linear combination of the weights and the activation maps. Recent
CAM-based approaches all generalize the original CAM and are not limited to
CNNs with a special global pooling layer. They can be divided into two
branches, one is gradient-based CAMs[15, 2], which represent the linear
weights corresponding to internal activation maps by gradient information. The
other is gradient-free CAMs[23, 4] which capture the importance of each
activation map by the target score in forward propagation. Although Score-
CAM[23] has achieved good performance on both visual comparison and fairness
evaluation, its localization result is coarse which leads to certain instances
of non-interpretability.

To achieve better visual feature localizing, in this paper, we propose a novel
method called SS-CAM, built on top of Score-CAM[23], for enhancing object
understanding and providing better post-hoc explanations about the
centralized, target object in the image using Smoothing. We evaluate our
approach on ILSVRC 2012 dataset on the basis of 4 major metrics. Our
contributions can be summarized as below:

  * •

We introduce an enhanced visual feature localization method SS-CAM, which
combine Score-CAM with an extra smooth operation and leads to a visually
sharper attribution map.

  * •

Two types of smoothing are introduced, we quantitatively evaluate the
differences in smoothing for different positions.

  * •

We achieve better visual performance than previous CAM-based methods. We
quantitatively evaluate on recognition, localization and human trust tasks and
show that SS-CAM better localizes decision-related features.

##  II Related Work

A Gradient-based approach generates a saliency map using the derivative of the
target class score to the input image through backpropagation[19].

SmoothGrad [18]: SmoothGrad aims to reduce the visual noise by adding
sufficient noise to an image and generating similar images. Then the average
of the resulting sensitivity maps of each of these sampled, similar images is
taken. Training with the noise and inferring the decision of the model with
the noise seems to have a better effect at yielding the best results.
Visually-sharp gradient-based sensitivity maps are created by taking random
samples in the neighbourhood of $x$ and averaging the resulting maps. It is
indicated by:

$\widehat{M_{c}}\left(x\right)=\dfrac{1}{n}\sum^{n}_{1}M_{c}\left(x+\mu\left(0,\sigma^{2}\right)\right)$

where ${n}$ refers to the number of samples and $\mu\left(0,\sigma^{2}\right)$
refers to the Gaussian noise with a standard deviation $\sigma$ that is added
to an image input ${x}$.

CAM [27]: CNNs utilise targeted and distinguishable image regions for
identifying particular categories by linearly combining weights with the
activation maps. These categories are known as Class Activation Maps (CAMs).
The regions that are highlighted as the predicted class score values are
mapped back to the previous convolutional layer. It is denoted by :

$L^{c}_{CAM}=ReLU\left(\sum_{k}\alpha^{c}_{k}A^{k}_{l-1}\right)$ (1)

$where$

$\alpha^{c}_{k}=w^{c}_{l,l+1}[k]$ (2)

$\alpha^{c}_{k}=w^{c}_{l,l+1}[k]$ refers to the weight of the k-th neuron
after the pooling operation.

The generalisation of CAMs take place with Grad-CAM[15]. To obtain fine-
grained visualizations, Grad-CAM activation maps are revamped by multiplying
with the visualization obtained via Guided Backpropagation. This is better
known as Guided Grad-CAM.

Grad-CAM [15]: This technique utilizes local gradient to represent linear
weight, and can be applied to any Average Pooling-based CNN architectures
without the re-training process. It is denoted by :

$L^{c}_{Grad-CAM}=ReLU\left(\sum_{k}\alpha^{c}_{k}A^{k}_{l}\right)$ (3)

$where$

$\alpha^{c}_{k}=GAP\left(\frac{\partial Y^{c}}{\partial A^{k}_{l}}\right)$ (4)

GAP(.) refers to the Global Average Pooling operation.

However, this approach has some limitations. It fails to provide worthy
explanations in highly confident scenarios due to gradient saturation and
fails to highlight multiple occurrences of the same object in a single image.
Also, the heatmaps generated with Grad-CAM do not capture the entire object
completely as the gradients tends to be noisy, which is required to gain
better recognition of the object in the image.

This technique was further improved with Grad-CAM++[2] which gives better
results by localizing the entire object in image, instead of bits and pieces
of it. However, as the original CAM is restricted to CNNs having the GAP
layer, they do not generate feasible explanations on a wide array of CNNs. As
the output layer is a non-linear function, gradient-based CAMs tend to
diminish the backpropagating gradients which cause gradient saturation thereby
making it difficult to provide concrete explanations. To solve these problems,
Score-CAM[23] bridges the gap between perturbation-based and CAM-based method
by being the first gradient-free generalization CAM.

Score-CAM [23]: Score-CAM obtains the weights of each activation map in its
forward passing score on the target class. The final result is a linear
combination of both the weights and the activation maps. In Score-CAM, $L^{c}$
is denoted as follows:

$L^{c}_{Score-CAM}=ReLU\left(\sum_{k}\alpha^{c}_{k}A^{k}_{l}\right)$ (5)

$where$

$\alpha^{c}_{k}=C(A^{K})$ (6)

where $c$ is the class of interest, $l$ is the convolution layer and $C(.)$ is
the Channel-wise Increase of Confidence(CIC) score for the activation map
$A^{k}_{l}$.

##  III Proposed Approach

Detecting model misbehavious and a good post-hoc visualization were the
primary applications of Score-CAM [23]. With SS-CAM, we generate better
visualisations with higher faithfulness evaluation which leads one to analyse
the model with a better intuition. We borrow the idea from SmoothGrad[18] and
integrate the smooth into the pipeline of Score-CAM to generate ”smoother”
feature localization. The higher ”smoothness” helps one perform a higher level
of analysis on the architecture. In this paper, we evaluate two ways to
smooth. The pipeline for the proposed method is shown in Figure 1. Before
diving into the details of techniques, we define our motivation to pursue this
topic of interest and why we address it.

Figure 1: Pipeline of the proposed SS-CAM appraoch. The saliency map is produced by the linear combination of the average score after the smoothing operation and the upsampled activation maps. The average score is obtained from these noisy images and utilized as the final score to provide enhanced feature localizations. Input Image | Score-CAM | SS-CAM(2)
---|---|---
|  |
Figure 2: Depicts the Imagenet Label: Bull Mastiff, and the outputs of Score-
CAM and SS-CAM(2). As you can see the Score-CAM output is quite noisy by
itself (probably due to ReLU non-linearity), this is what encouraged us to
tackle this issue.

We choose to investigate Smoothing in Score-CAM to generate more feasible
explanations. As one can see the Score-CAM map (Figure 2) clearly doesn’t
highlight regions that aren’t localized to the target object (here, Bull
Mastiff). This probably due to the noise generated because of the ReLU non-
linearity. The map also spreads all over which clearly makes it highly
unreliable to convery any type of result but SS-CAM generates a concentrated
heat-map which completely focuses on the object. Such Concentrated heat maps
can substantially increase the chance of finding faults with the predictions.
If a model is mis-classifying, the mis-classified heat map will appropriately
show the part of the image which is being represented and help in debugging
the model. We now proceed to explain our approaches starting by measuring the
importance of each activation map with CIC:

|  |  | Smooth |  |  |
---|---|---|---|---|---|---
Input Image |  Grad-CAM |  Grad-CAM++ |  Grad-CAM++ |  Score-CAM |  SS-CAM(1) |  SS-CAM(2)

Figure 3: Depicts the Imagenet Labels (Row-wise): Tennis ball, Doberman and
Black-and-tan Coonhound. This figure is used for a Visual Comparison of our
approach with the other existing approaches. TABLE I: AUC scores of the
Deletion curve(the lower the better) and Insertion curve(the higher the
better) for the Black-and-tan Coonhound saliency maps as generated in Figure
3.

Metrics | Grad-CAM | Grad-CAM++ | Smooth | Score-CAM | SS-CAM(1) | SS-CAM(2)
---|---|---|---|---|---|---
|  |  | Grad-CAM++ |  |  |
Deletion(%) | 48.08 | 52.53 | 44.41 | 28.29 | 43.96 | 29.94
Insertion(%) | 59.85 | 51.68 | 64.91 | 59.41 | 70.02 | 61.76

TABLE II: Average Drop(the lower the better) and Average Increase In
Confidence(the higher the better) across 2000 ILSVRC Validation images with
VGG-16.

Metrics | Grad-CAM | Grad-CAM++ | Smooth | Score-CAM | SS-CAM(1) | SS-CAM(2)
---|---|---|---|---|---|---
|  |  | Grad-CAM++ |  |  |
Average Drop(%) | 89.69 | 89.76 | 90.92 | 90.07 | 89.01 | 90.17
Average Increase(%) | 0.65 | 0.40 | 0.45 | 0.60 | 0.65 | 0.65

Channel-wise Increase of Confidence(CIC) : Given a model $f(X)$ that takes an
image input $X$, the $k$-th channel of an activation map $A$ of a
convolutional layer $l$ in $f$ is given as $A^{k}_{l}$. For a baseline input
$X_{b}$ the importance of $A^{k}_{l}$ is denoted as

$C(A^{k}_{l})=f(X\cdot H^{k}_{l})-f(X_{b})$ (7)

$where$

$H^{k}_{l}=s(U(A^{k}_{l}))$ (8)

Here, $U(.)$ is the Upsampling operation of the activation map $A^{k}_{l}$
into the size of the input and $s(.)$ refers to the normalization function (as
explained in Equation 15)) so that the elements are within the [0,1] range.

A few similar approaches to CIC are DeepLIFT[16] and Average Drop% and
Increase in Confidence from [2]. The significance of activation maps are
represented by the gradient information that flows into the final
convolutional layer.

Smooth on activation map We set the number of noised sample images $N$ to be
generated by adding Gaussian noise to each activation map $A_{i}$. For each
activation map $A_{i}$, $N$ scores are generated and averaged to a final
score, which is considered as the importance of the activation map $A_{i}$.

An activation map $A$ of a convolutional layer $l$ with channel $k$ is given
as $A^{k}_{l}$. SS-CAM with smooth on feature space (type1) can be depicted
as:

$L^{c}_{SS-CAM}=ReLU\left(\sum_{k}\alpha^{c}_{k}A^{k}_{l}\right)$ (9)

$where$

$\alpha^{c}_{k}=\dfrac{\sum^{N}_{1}\left(C(M)\right)}{N}$ (10)

Here, $c$ is the class of interest, $l$ is the convolutional layer and
$\alpha^{c}_{k}$ is the average Channel-wise score of the activation map
$A^{k}_{l}$ which accounts for the scores generated by the noisy maps
$A^{k}_{l}+N(0,\sigma)$. $X_{0}$ is the input image and $A^{k}_{l}$ refers to
the activation map at layer $l$.

$M=\sum^{N}_{1}(X_{0}*(A^{k}_{l}+N(0,\sigma)))$ (11)

Smooth on input : An activation map $A$ of a convolutional layer $l$ with
channel $k$ is given as $A^{k}_{l}$. SS-CAM with smooth on input space (type2)
can be depicted as:

$L^{c}_{SS-CAM}=ReLU\left(\sum_{k}\alpha^{c}_{k}A^{k}_{l}\right)$ (12)

$where$

$\alpha^{c}_{k}=\dfrac{\sum^{N}_{1}\left(C(M)\right)}{N}$ (13)

Here, $c$ is the class of interest, $l$ is the convolutional layer and
$\alpha^{c}_{k}$ is the Channel-wise average score of the noisy normalized
activation map ($A^{k}_{l}+N(0,\sigma)$). $X_{0}$ is the input image and here,
$X_{0}*A^{k}_{l}$ refers to the normalized input mask.

$M=\sum^{N}_{1}((X_{0}*A^{k}_{l})+N(0,\sigma))$ (14)

Normalization: The binary mask over the normalized map would not be great as
our aim is to focus on the spatial region where the object lies and the binary
mask would lose sight of the important features. Hence, we employ a similar
normalization function as used in [23] to elevate the features within the
specific region.

The normalization used in the algorithm is given as :

$s\left(A^{k}_{l}\right)=\dfrac{A^{k}_{l}-min(A^{k}_{l})}{max(A^{k}_{l})-min(A^{k}_{l})}$
(15)

##  IV Experiments

In this section, we explain our experimental setting to assess the proposed
method(s). Our setting is similar to that in [2, 23, 12]. First, a qualitative
comparison of the the outputs of the architectures via a visualization on the
ILSVRC 2012 Validation set in A. Second, we evaluate the fairness of the
explanations of the architectures on object recognition in B. Next, the
bounding box evaluations are performed for the class-conditional localization
of objects in a given image in C using the Energy-based pointing game
(introduced in [2]) over 2000 uniformly random selected images from the ILSVRC
Validation Set 2012. Lastly, we show a quantitative study of Human Trust
evaluation to understand how our method achieves sufficient interpretability.

Our comparison extends with namely 4 other existing CAM techniques, Grad-CAM,
Grad-CAM++, Smooth Grad-CAM++ and Score-CAM. The images are resized with a
definite size (224, 224, 3), transformed into the [0,1] range and then,
normalized using ImageNet[3] weights (mean vector : [0.485, 0.456, 0.406] and
standard deviation vector [0.229, 0.224, 0.225]). For simplicity, baseline
image $X_{b}$ is set to 0.

###  IV-A Visual Comparison

To conduct this experiment, 2000 images were randomly selected from the ILSVRC
2012 Validation set. Figure 3 displays a few images that compare our
approach(es) with existing CAM approaches namely Grad-CAM, Grad-CAM++, Smooth
Grad-CAM++ and Score-CAM. Here, we used $N$ = 35 and $\sigma$ = 2. A further
depth to this comparison was added to sub-section D.

###  IV-B Faithfulness Evaluations

The faithfulness evaluations are carried out as depicted in Grad-CAM++[2] for
the purpose of Object Recognition. Three metrics called Average Drop, Average
Increase In Confidence and Win % are introduced.

These metrics are evaluated over the pre-trained VGG-16 model for 2000 images
randomly selected from the ILSVRC 2012 Validation set. We used $N$ = 10 and
$\sigma$ = 2 to conduct this sub-experiment.

Insertion and Deletion Curves are used to calculate the Area Under Curve(AUC)
metric to understand how many pixels of the saliency map can either contribute
or decrease the scores of the resulting fractioned maps. We adapt a similar
setting as introduced in [12]. The Deletion operation shows the ability to
destroy the map information pixel-wise. A sharp drop and a lower AUC of the
scores generated is indicative of a good explanation. The Insertion operation
determines the ability to reconstruct the saliency map from a given baseline.
A sharp increase and a higher AUC of the scores generated is indicative of a
good explanation.

Figure 4: Insertion and Deletion curve of the Black-and-tan Coonhound maps
generated above for SS-CAM(2). You can notice a sharp increase midway in the
Insertion curve, which indicates that our approach is capable of generating
viable explanations.

####  IV-B1 Average Drop %

The Average Drop refers to the maximum positive difference in the predictions
made by the prediction using the input image and the prediction using the
saliency map. It is given as:
$\sum^{N}_{1}\dfrac{\max\left(0,Y^{c}_{i}-O^{c}_{i}\right)}{Y^{c}_{i}}$
$\times 100$. Here, $Y^{c}_{i}$ refers to the prediction score on class $c$
using the input image $i$ and $O^{c}_{i}$ refers to the prediction score on
class $c$ using the saliency map produced over the input image $i$.

####  IV-B2 Increase in Confidence %

The Average Increase in Confidence is denoted as:
$\sum^{N}_{1}\dfrac{Func\left(Y^{c}_{i}<O^{c}_{i}\right)}{N}$ $\times$ 100
where $Func$ refers to a boolean function which returns $1$ if the condition
inside the brackets is true, else the function returns $0$. The symbols are
referred to as shown in the above experiment for Average Drop.

####  IV-B3 Win %

The Win percentage refers to the decrease in the model’s confidence for an
explanation map generated by SS-CAM. This metric is compared to the confidence
generated by Score-CAM[23] maps with both SS-CAM(1) and SS-CAM(2) maps. SS-
CAM(1) when compared with Score-CAM shows 49.8% of the scores towards its
favour while SS-CAM(2) displays a Win % of 47.8% scaled on its behalf when
compared with the scores by Score-CAM. We use $N$=10 and $\sigma$=2 for this
experiment.

Table II depicts moderate to similar performance in regards to average drop
and increase in confidence. While Table I depicts SS-CAM(2) just slightly
falling short of Score-CAM in terms of the deletion curve, SS-CAM(1) performs
quite well with a 5 % increase in the insertion curve.

###  IV-C Localization Evaluations

Bounding box evaluations are accomplished in this section. We employ the
similar metric, as specified in Score-CAM, called the Energy-based pointing
game. Here, the amount of energy of the saliency map is calculated by finding
out how much of the saliency map falls inside the bounding box. Specifically,
the input is binarized with the interior of the bounding box marked as 1 and
the region outside the bounding box as 0. Then, this input is multiplied with
the generated saliency map and summed over to calculate the proportion ratio,
which given as - $Proportion=\dfrac{\sum L^{c}_{(i,j)\in bbox}}{\sum
L^{c}_{(i,j)\in bbox}+\sum L^{c}_{(i,j)\not\in bbox}}$. Three pretrained
models namely, VGG-16[17], ResNet-18(Residual Network with 18-layers)[6] and
SqueezeNet1.0[9] model are used to conduct the Energy-based pointing game on
the 2000 randomly chosen images from the ILSVRC 2012 Validation set. We used
$N$ = 10 and $\sigma$ = 2 to conduct this sub-experiment. Though the margins
are small Table III portrays a better localization evaluation for 2 of the 3
architectures.

TABLE III: Localization Evaluations using Energy-based Pointing Game (Higher
the better)

Metrics | Grad-CAM | Grad-CAM++ | Smooth | Score-CAM | SS-CAM(1) | SS-CAM(2)
---|---|---|---|---|---|---
|  |  | Grad-CAM++ |  |  |
VGG-16 |  |  |  |  |  |
Proportion(%) | 39.95 | 40.16 | 40.13 | 40.10 | 40.27 | 39.70
ResNet18 |  |  |  |  |  |
Proportion(%) | 40.90 | 40.85 | 40.86 | 40.76 | 41.55 | 40.51
SqueezeNet1.0 |  |  |  |  |  |
Proportion(%) | 39.47 | 39.30 | 39.25 | 39.56 | 37.07 | 39.56

TABLE IV: Human Trust Evaluations using the visualizations generated by VGG-16
(Higher the better)

Metrics | Grad-CAM | Grad-CAM++ | Smooth | Score-CAM | SS-CAM(1) | SS-CAM(2) | Not Sure
---|---|---|---|---|---|---|---
|  |  | Grad-CAM++ |  |  |  |
Responses(%) | 9.76 | 14.44 | 16.84 | 14.92 | 14.60 | 20.60 | 8.84

###  IV-D Evaluating Human Trust

In this experiment, we evaluate on human trust or interpretability of the
produced visual explanations on the basis of blind reviews. We generate
explanation maps for 50 images of the 5 randomly selected classes out of 1000
classes from the ILSVRC 2012 Validation dataset[3], totalling to 250 images.
The Grad-CAM maps are treated as the base-line for the visualization maps
generated. These maps, along with the input image, are shown to 10 subjects
(who have no knowledge in this field or Deep Learning) and they are testified
to answer which map gives more insight to the captured object, thereby
invoking more trust and higher interpretability. A pretrained VGG-16 model was
used to create the explanation maps. We choose 5 classes at random as this
would give an equal probability of all the classes getting to be picked from
the 1000 ImageNet classes, hence, removing any bias.

For each image, six explanation maps were generated namely, Grad-CAM, Grad-
CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM(1) and SS-CAM(2). The subjects
were asked to choose which maps demonstrated a better visual explanation of
the object in the image (without having any knowledge of which map
corresponded to which method). The responses were normalized so that the total
score for each image would be equal to 1. These normalized scores were then
added and so that the maximum score achievable would be 250 for each subject.
The total number of responses recorded are 10*250 = 2500 responses. A ”Not
sure” option was also provided for the subjects if they were unable to
distinguish the visual maps. Table IV shows the corresponding score achieved
using this evaluation metric. We used $N$ = 35 and $\sigma$ = 2 to conduct
this experiment. The values presented in Table IV clearly poses the higher
response to the SS-CAM(2) approach which leads to the conclusion of higher
human interpretability and trust.

##  V Conclusion

Our proposed method significantly enhances the localization of the features of
the target class in an image, thereby explaining the images profoundly. In
this aspect, our method fairs well above the existing CAM approaches as
evaluated using the Energy-based Pointing game (Table III). The higher AUC
scores presented in (Table I) further supports the capabilities of the
techniques in terms of faithfulness evaluations and (Table IV) shows a clear
superiority in terms of human trust evaluations. The Insertion curve(Figure 4)
also suggests that SS-CAM has great potentials due to a significant increase
in the score as the pixels get inserted.

The Gaussian noise with the paramater $\sigma$[18, 5] is used to smoothen the
input space of the attribution maps and construct visually sharp attribution
maps. If the parameter $\sigma$ is played around with, it has been found that
if $\sigma$ is low, the score generated is quite ”noisy” because the resulting
attribution maps that tend to be too noisy and if the parameter is high, the
attribution map generated is expected to be much more generalized across
different classes which would mean that the maps would be rendered useless and
irrelevant. Hence, an ideal $\sigma$ is carefully chosen for our evaluations
to get the best results.

The limitations of SS-CAM are that it takes a lot of time to process these
explanations when compared to Grad-CAM[15]. This occurs due to the iteration
over each feature map to calculate the scores of $N$ noisy samples during the
upsampling stage and averaging them for generating smoother attribution maps.
But one should note that the application of such techniques are focused on
environments where even a single error can cause great harm and damage. Hence
we gave priority to results over the time taken to the obtain the same.

##  VI Future Work

The future prospect is to extend other technologies to CAM and establish a
unified framework for each CAM-based method in mathematical expression.

## References

  * [1] J. Buolamwini and T. Gebru, “Gender shades: Intersectional accuracy disparities in commercial gender classification,” in _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ , ser. Proceedings of Machine Learning Research, S. A. Friedler and C. Wilson, Eds., vol. 81. New York, NY, USA: PMLR, 23–24 Feb 2018, pp. 77–91. [Online]. Available: http://proceedings.mlr.press/v81/buolamwini18a.html
  * [2] A. Chattopadhyay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” _CoRR_ , vol. abs/1710.11063, 2017. [Online]. Available: http://arxiv.org/abs/1710.11063
  * [3] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [4] S. Desai and H. G. Ramaswamy, “Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization,” in _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2020, pp. 972–980.
  * [5] G. S. W. Goh, S. Lapuschkin, L. Weber, W. Samek, and A. Binder, “Understanding integrated gradients with smoothtaylor for deep neural network attribution,” _ArXiv_ , vol. abs/2004.10484, 2020.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _CoRR_ , vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
  * [7] G. Hu, Y. Yang, D. Yi, J. Kittler, W. Christmas, S. Z. Li, and T. Hospedales, “When face recognition meets with deep learning: an evaluation of convolutional neural networks for face recognition,” in _Proceedings of the IEEE international conference on computer vision workshops_ , 2015, pp. 142–150.
  * [8] Y. Huang, C. Li, T. Li, W. Wan, and J. Chen, “Image captioning with attribute refinement,” in _2019 IEEE International Conference on Image Processing (ICIP)_ , 2019, pp. 1820–1824.
  * [9] F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size,” _CoRR_ , vol. abs/1602.07360, 2016. [Online]. Available: http://arxiv.org/abs/1602.07360
  * [10] X. Liu, Z. Deng, and Y. Yang, “Recent progress in semantic image segmentation,” _CoRR_ , vol. abs/1809.10198, 2018. [Online]. Available: http://arxiv.org/abs/1809.10198
  * [11] S. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” 2017.
  * [12] V. Petsiuk, A. Das, and K. Saenko, “RISE: randomized input sampling for explanation of black-box models,” _CoRR_ , vol. abs/1806.07421, 2018. [Online]. Available: http://arxiv.org/abs/1806.07421
  * [13] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [14] M. T. Ribeiro, S. Singh, and C. Guestrin, “”Why Should I Trust You?”: Explaining the predictions of any classifier,” _CoRR_ , vol. abs/1602.04938, 2016. [Online]. Available: http://arxiv.org/abs/1602.04938
  * [15] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 618–626.
  * [16] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” _CoRR_ , vol. abs/1704.02685, 2017. [Online]. Available: http://arxiv.org/abs/1704.02685
  * [17] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014.
  * [18] D. Smilkov, N. Thorat, B. Kim, F. B. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” _CoRR_ , vol. abs/1706.03825, 2017. [Online]. Available: http://arxiv.org/abs/1706.03825
  * [19] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. A. Riedmiller, “Striving for simplicity: The all convolutional net,” _CoRR_ , vol. abs/1412.6806, 2015\.
  * [20] F. Sultana, A. Sufian, and P. Dutta, “Advancements in image classification using convolutional neural network,” _CoRR_ , vol. abs/1905.03288, 2019. [Online]. Available: http://arxiv.org/abs/1905.03288
  * [21] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” 2019.
  * [22] H. Wang, Z. Chen, and Y. Zhou, “Hybrid coarse-fine classification for head pose estimation,” _arXiv preprint arXiv:1901.06778_ , 2019.
  * [23] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu, “Score-cam: Score-weighted visual explanations for convolutional neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 24–25.
  * [24] Z. Wang, H. Wang, S. Ramkumar, M. Fredrikson, P. Mardziel, and A. Datta, “Smoothed geometry for robust attribution,” _arXiv preprint arXiv:2006.06643_ , 2020.
  * [25] F. Yang, Z. Zhang, H. Wang, Y. Li, and X. Hu, “Xdeep: An interpretation tool for deep neural networks,” _arXiv preprint arXiv:1911.01005_ , 2019.
  * [26] M. Zeiler and R. Fergus, “Visualizing and understanding convolutional neural networks,” vol. 8689, 11 2013.
  * [27] B. Zhou, A. Khosla, À. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” _CoRR_ , vol. abs/1512.04150, 2015. [Online]. Available: http://arxiv.org/abs/1512.04150


decor/.style = label/.expanded = [inner sep = 0.2ex, font=]right:$#1$ ,
root/.style = minimum size = 0.1ex, decorated/.style = for tree = circle,
fill, inner sep = 0.3ex, minimum size = 1.ex, grow’ = south, l = 0, l sep =
1.2ex, s sep = 0.7em, fit = tight, parent anchor = center, child anchor =
center, delay = decor/.option = content, content = , default preamble =
decorated, root, begin draw/.code=