# MGI: Multimodal Contrastive pre-training of Genomic and Medical Imaging

Jiaying Zhou *
Department of Ophthalmology and Visual Sciences
The Chinese University of Hong Kong
<EMAIL_ADDRESS>
&Mingzhou Jiang *
Tsinghua Shenzhen International Graduate School
Tsinghua University
<EMAIL_ADDRESS>
&Junde Wu *, project lead
National University of Singapore
University of Oxford
<EMAIL_ADDRESS>
&Jiayuan Zhu
University of Oxford
&Ziyue Wang
National University of Singapore
&Yueming Jin
National University of Singapore
<EMAIL_ADDRESS>

###### Abstract

Medicine is inherently a multimodal discipline. Medical images can reflect the
pathological changes of cancer and tumors, while the expression of specific
genes can influence their morphological characteristics. However, most deep
learning models employed for these medical tasks are unimodal, making
predictions using either image data or genomic data exclusively. In this
paper, we propose a multimodal pre-training framework that jointly
incorporates genomics and medical images for downstream tasks. To address the
issues of high computational complexity and difficulty in capturing long-range
dependencies in genes sequence modeling with MLP or Transformer architectures,
we utilize Mamba to model these long genomic sequences. We aligns medical
images and genes using a self-supervised contrastive learning approach which
combines the Mamba as a genetic encoder and the Vision Transformer (ViT) as a
medical image encoder. We pre-trained on the TCGA dataset using paired gene
expression data and imaging data, and fine-tuned it for downstream tumor
segmentation tasks. The results show that our model outperformed a wide range
of related methods.

##  1 Introduction

The medical field encompasses a variety of data modalities. Different types of
medical data can reflect various disease characteristics and they are often
complementary. For example, in tumor diagnosis, imaging data reveal the
morphological characteristics and growth patterns of tumors, while genetic
data provide information on disease susceptibility and potential biomarkers.
Specific gene expressions can influence the morphological features of tumors,
such as shape, size, and location. Therefore, integrating these different but
complementary data can help clinicians develop a more comprehensive
understanding of disease characteristics.

Most deep learning methods applied in the medical field today are unimodal,
analyzing disease characteristics presented by single-modal data. Tumor
formation and development are often closely related to specific gene
expressions. Therefore, relying solely on imaging and ignoring molecular
biology data may lead to misdiagnosis. For instance, a doctor might diagnose a
tumor based only on imaging data, whereas gene expression data do not support
this conclusion, resulting in a false positive—believing a tumor exists when
it does not.

To address this issue, constructing methods that integrate multimodal data can
learn information from different types of data, helping the model make more
accurate judgments. This approach helps reduce the partial insights that come
from relying on single-modal data. Recently, an increasing number of studies
in the medical field have utilized deep neural networks to jointly learn from
both gene and image data. For example, (Gundersen et al., 2020), (Venugopalan
et al., 2021), (Fujinami-Yokokawa et al., 2021) have attempted to use the
combination of gene and image data for comprehensive diagnosis in the medical
field. These multimodal paradigms typically use different encoders to
separately encode images and genes. The gene encoders commonly employed are
MLP or Transformer like(Kirchler et al., 2022), while the image encoders are
usually CNN or ViT like (Yang et al., 2021).

However, gene sequences are typically long sequences. MLP lacks the capability
to model long sequences as it processes each input independently without
considering relationships between inputs, like (Taleb et al., 2022), thus
failing to effectively capture long-range dependencies in genes sequence. The
core of the Transformer is the attention mechanism, which, despite its
excellent performance in capturing sequence dependencies, has a quadratic
computational complexity. This results in extremely high computational and
memory consumption when processing very long gene sequences, making it
impractical for real-world applications, such as (Abdine et al., 2024).
Therefore, it is needed to develop more efficient methods to learn the long-
range dependencies in long sequence data, such as gene expression.

In this paper, we propose a multimodal framwork named MGI (Multimodal
Contrastive Learning of Genomic and Medical Imaging), which combines gene
sequence expression and medical imaging data. MGI uses Mamba to model gene
sequences, leveraging its strengths in long sequence modeling to enhance the
gene modeling capability within the multimodal framework. Simultaneously, it
employs ViT to model images and integrates information from gene sequence
expression data to achieve more accurate medical image segmentation. During
the pre-training phase, we use a self-supervised contrastive learning strategy
to align the visual encoder and gene encoder on paired genetic data and image
data, enabling the visual encoder to learn relevant features from the
perspective of genes. For the downstream segmentation task, we introduce a
lightweight multimodal attention fusion decoder to integrate image and gene
data, thereby improving segmentation accuracy.

Our contributions can be summarized as follows:

(1) We propose a new multimodal model called MGI, which is based on both
genetic and imaging data. We use Mamba for feature extraction from the gene
data, successfully addressing the issue faced by previous gene encoders in
capturing long-range dependencies in long gene sequences.

(2) We align image embeddings with gene embeddings through a gene-image
contrastive loss, enabling the image encoder to understand genetic
information.

(3) We propose a new lightweight multimodal fusion module for downstream
tasks.

##  2 Method

Before introducing our method, we first motivate the imaging and genetic
modalities chosen in this work in Sec 2.1. Then, we describe our multimodal
pre-training framework MGI in Sec 2.2. After that, in Sec 2.3, we introduce
the contrastive loss which aligns the images and genes. Finally, in Sec 2.4,
we describe the multimodal fusion method that we design for transferring
learning of downstream segmentation task.

###  2.1 Modalities of Imaging and Genetic Data

#### Medical imaging modality.

Magnetic resonance imaging (MRI), especially the fluid-attenuated inversion
recovery (FLAIR) sequence, is of significant importance in the diagnosis and
monitoring of lower-grade gliomas. The FLAIR sequence can suppress
cerebrospinal fluid signals, thereby more clearly displaying the lesion areas
and aiding physicians in identifying and evaluating the extent and location of
the tumor. Therefore, in this paper, we chose the FLAIR sequence images from
The Cancer Genome Atlas (TCGA) lower-grade glioma dataset as the image
modality for pre-training.

#### Genetic modality.

The human genome consists of over 3 billion base pairs. Genes are made up of
different numbers of base pairs arranged in specific sequences. Due to the
limitations of sequencing technology and cost, only a small portion of genes
have been sequenced, but these sequences are still quite long by current
standards. Human complex traits are influenced by the expression of numerous
genes; for instance, the development of Alzheimer’s disease and endometrial
cancer is associated with specific gene segments(Pietzner et al., 2021).

Therefore, when using deep learning methods to analyze medical images for
disease diagnosis, it is beneficial to consider different levels of gene
expression. This approach helps in understanding the impact of specific genes
on the morphology of pathological regions in the images, thereby constructing
more reliable and accurate models. For our pre-training, we choose the gene
expression data from RNA-sequence. This data is collected from lower-grade
glioma samples where mRNA is reverse-transcribed into complementary DNA,
reflecting the expression levels of different gene segments during sequencing.
The expression of certain genes is associated with the disease.

###  2.2 Multimodal pre-training framework

As shown in Fig. 1, our MGI framework consists of three parts: the inputs of
multimodal samples, encoders of medical images and genes, contrastive loss for
images and genetics. We assume a bacth input including B multimodal samples,
one for each individual person. There is a single medical image and the gene
expression sequences corresponding to the patient in each sample.

To process these different modalities of inputs, we use one Images Encoder and
one Genes Encoder to encoding images and genes separately. We seek to exploit
the Vision Transformer (ViT)’s capability to capture global image features for
encoding purposes, (Dosovitskiy et al., 2021) aiming to enhance the
scalability and applicability of image embeddings for various downstream
tasks. Simultaneously, we highly endorse Mamba’s approach of utilizing state
to memory the global information, which demonstrates superior long-sequence
modeling capabilities and higher computational efficiency (Gu & Dao, 2023).
Therefore, in our method, the Images Encoder consists of $L_{1}$ Vision
Transformer(ViT) blocks and Genes Encoder consists of $L_{2}$ Mamba blocks.

Images and genetic sequences are encoded into embeddings by their respective
encoders. Then, mixed pooling and linear projection are applied to the
embeddings, mapping the image and genetic embeddings into vectors within the
feature space. We denote imaging feature vector by $I_{i}$ and genes sequences
feature vector by $G_{i}$ ($i\in 1,2,3,...,B$). Feature vectors $I_{i}$ and
$G_{i}$ from different samples are used to calculate contrastive loss, and
backpropagation is employed to update the model parameters.

Figure 1: MGI framework.

###  2.3 Contrastive loss for Images and Genetics

We present the images-genes contrastive loss to align the images embeddings
and genes embeddings. Given the imaging feature vector $I_{i}$ and genes
sequences feature vector $G_{i}$, each $I_{i}$ and $G_{i}$ is computed for
cosine similarity with the all of the $G_{i}$ and $I_{i}$ within the same
batch, respectively. As shown in Fig. 1, we aim to maximize the cosine
similarity between $I_{i}$ and $G_{i}$ from the same sample. The loss function
is shown in formulation (1).

$\mathcal{L}\left(I,G\right)=\displaystyle\sum_{i=1}^{B}log\frac{exp\left(\cos\left({I}_{i},{G}_{i}\right)/\tau\right)}{\displaystyle\sum_{j=1}^{B}\displaystyle\sum_{k=1}^{B}exp\left(\cos\left({I}_{j},{G}_{k}\right)/\tau\right)}$
(1)

where $\tau$ is a temperature parameter, cos is the cosine similarity. This
formulation ensures that the learned visual representations and genetic
sequence representations acquire useful information from each other.

Figure 2: Multimodal fusion module.

###  2.4 Multimodal fusion for transferring learning

There exits a gap between medical images and genetic expression sequence
modalities. Our motivation for multimodal fusion is that the inter-modality
interactions between medical images and genetic features would be able to
enhance the effectiveness of visual representations. For example, after
integrating multimodal genetic and image data, the performance of Alzheimer’s
disease diagnostic classification shows significant improvement(Zhou et al.,
2019).

We present a lightweight Multimodal fusion module as the maskdecoder for
downstream segmentation task. The Multimodal fusion module uses the two-way
attention mechanism to integrate images embeddings and genes embeddings. It
consists of two Attention interaction modules and a Mask output module. As
shown in Fig. 2, each Attention interaction module performs 4 steps:(1) self
attention on the genes embeddings, (2) cross attention from genes embeddings
to image embeddings, (3) a MLP to update features, (4) cross attention from
image embeddings to genes embeddings. Finally, in Mask output module, we
upsample the updated image embeddings by transposed convolutional layer, and a
MlP block after one cross attention from genes embeddings to image embeddings
is applied to update genes embeddings matching the dimension of image
embeddings. We pre- dict a segmentation mask with a spatially point-wise
product between the image embeddings and genes embeddings.

##  3 Experiment

###  3.1 Dataset

We pretrained our models on the TCGA-LGG dataset, a genomic and clinical
dataset for Low-Grade Glioma collected by The Cancer Genome Atlas (TCGA)
project. This dataset contains multimodal data for nearly 530 individuals.
Since only a subset of the image data in LGG is available, we can only use
that subset in our experiment. It includes 3,929 CT images and their
corresponding masks from 106 patients. Each patient’s genetic data includes
the expression levels of 60,484 genes, which have been screened by doctors and
are considered potentially related to low-grade glioma. We hold out a train
split (80%) from the LGG dataset, using the remaining (20%) for testing. For
the downstream task, we fine-tuned MGI for tumor segmentation task on the
TCGA-LGG.

###  3.2 Results

We compare our MGI with several baseline methods in Tab. 1. Our method
outperforms other approaches, achieving a Dice score of 0.901.

Table 1: The results of segmentation on the TCGA-LGG by Dice score Method | TCGA-LGG
---|---
U-Net | 0.735
ResU-Net | 0.772
DeepLabv3+ | 0.840
Attention U-Net | 0.830
MGI(ours) | 0.901

##  4 Conclusion

In these paper, we present a new multimodal framework of genomic and medical
image, which utilize ViT as the image encoder and Mamba as the genes encoder.
This is the first attempt at multimodal learning using ViT and Mamba. We
design a image-gene contrastive loss to align image embeddings and genes
embeddings. For downstream segmentation task, we present a new lightweight
multimodal fusion module to integrate the features of genetic and medical
image. Our method performs well on the Low-Grade Glioma segmentation task.

## References

  * Abdine et al. (2024) Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, and Michalis Vazirgiannis.  Prot2text: Multimodal protein’s function generation with gnns and transformers.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pp. 10757–10765, 2024.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * Fujinami-Yokokawa et al. (2021) Yu Fujinami-Yokokawa, Hideki Ninomiya, Xiao Liu, Lizhu Yang, Nikolas Pontikos, Kazutoshi Yoshitake, Takeshi Iwata, Yasunori Sato, Takeshi Hashimoto, Kazushige Tsunoda, et al.  Prediction of causative genes in inherited retinal disorder from fundus photography and autofluorescence imaging using deep learning techniques.  _British Journal of Ophthalmology_ , 105(9):1272–1279, 2021.
  * Gu & Dao (2023) Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  _arXiv preprint arXiv:2312.00752_ , 2023.
  * Gundersen et al. (2020) Gregory Gundersen, Bianca Dumitrascu, Jordan T Ash, and Barbara E Engelhardt.  End-to-end training of deep probabilistic cca on paired biomedical observations.  In _Proceedings of The 35th Uncertainty in Artificial Intelligence Conference_ , 2020.
  * Kirchler et al. (2022) Matthias Kirchler, Stefan Konigorski, Matthias Norden, Christian Meltendorf, Marius Kloft, Claudia Schurmann, and Christoph Lippert.  transfergwas: Gwas of images using deep transfer learning.  _Bioinformatics_ , 38(14):3621–3628, 2022.
  * Pietzner et al. (2021) Maik Pietzner, Eleanor Wheeler, Julia Carrasco-Zanini, Adrian Cortes, Mine Koprulu, Maria A Wörheide, Erin Oerton, James Cook, Isobel D Stewart, Nicola D Kerrison, et al.  Mapping the proteo-genomic convergence of human diseases.  _Science_ , 374(6569):eabj1541, 2021.
  * Taleb et al. (2022) Aiham Taleb, Matthias Kirchler, Remo Monti, and Christoph Lippert.  Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20908–20921, 2022.
  * Venugopalan et al. (2021) Janani Venugopalan, Li Tong, Hamid Reza Hassanzadeh, and May D Wang.  Multimodal deep learning models for early detection of alzheimer’s disease stage.  _Scientific reports_ , 11(1):3254, 2021.
  * Yang et al. (2021) Karren Dai Yang, Anastasiya Belyaeva, Saradha Venkatachalapathy, Karthik Damodaran, Abigail Katcoff, Adityanarayanan Radhakrishnan, GV Shivashankar, and Caroline Uhler.  Multi-domain translation between single-cell imaging and sequencing data using autoencoders.  _Nature communications_ , 12(1):31, 2021.
  * Zhou et al. (2019) Tao Zhou, Kim-Han Thung, Xiaofeng Zhu, and Dinggang Shen.  Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis.  _Human brain mapping_ , 40(3):1001–1016, 2019.


6em

G. AlRegib and M. Prabhushankar, "Explanatory Paradigms in Neural Networks,"
_IEEE Signal Processing Magazine, Special Issue on Explainability in Data
Science_ , accepted on Feb. 18 2022.

Data of White Paper Submission : 22 March 2021
Date of Invitation : 29 April 2021
Date of Initial Submission : 25 August 2021
Date of First Revision : 16 November 2021
Date of Second Revision : 18 January 2022
Date of Accept: 18 Feb 2022

https://github.com/olivesgatech/Explanatory-Paradigms

©2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

<EMAIL_ADDRESS>OR<EMAIL_ADDRESS>
http://ghassanalregib.com/

# Explanatory Paradigms in Neural Networks

Ghassan AlRegib, and Mohit Prabhushankar

###### Abstract

In this article, we present a leap-forward expansion to the study of
explainability in neural networks by considering explanations as answers to
abstract reasoning-based questions. With _P_ as the prediction from a neural
network, these questions are _‘Why P?’_ , _‘What if not P?’_ , and _‘Why P,
rather than Q?’_ for a given contrast prediction _Q_. The answers to these
questions are observed correlations, observed counterfactuals, and observed
contrastive explanations respectively. Together, these explanations constitute
the abductive reasoning scheme. We term the three explanatory schemes as
observed explanatory paradigms. The term observed refers to the specific case
of _post-hoc_ explainability, when an explanatory technique explains the
decision $P$ after a trained neural network has made the decision $P$. The
primary advantage of viewing explanations through the lens of abductive
reasoning-based questions is that explanations can be used as reasons while
making decisions. The post-hoc field of explainability, that previously only
justified decisions, becomes active by being involved in the decision making
process and providing limited, but relevant and contextual interventions. The
contributions of this article are: ($i$) realizing explanations as reasoning
paradigms, ($ii$) providing a probabilistic definition of observed
explanations and their completeness, ($iii$) creating a taxonomy for
evaluation of explanations, and ($iv$) positioning gradient-based complete
explanainability’s replicability and reproducibility across multiple
applications and data modalities, ($v$) code repositories, publicly available
at https://github.com/olivesgatech/Explanatory-Paradigms

##  I Introduction

The generalizability of machine learning has fostered its ubiquitous usage
across diverse applications within Artificial Intelligence (AI). AI takes an
inferential and interpretive role in safety-critical fields of medicine,
remote sensing, and bioinformatics [1] among others. For example, AI
algorithms detect cardiovascular risk factors in humans through retinal fundus
images [2]. In [3], the authors propose an AI-based intervention system to
identify early red flags in law enforcement to prevent police-public adverse
events. In such sensitive applications, it is apparent that the role of
inference in an AI system is not a binary conclusion of whether risks exist.
Rather, AI inference must provide an explanation about the risks involved in
the decision, thereby allowing humans to assess and act on the risks.
Consequently, the authors in [2] provide a visual explanatory map to show why
and where an AI model detects cardiovasular risk symptoms. A number of works
exist that track the larger concept of interpretability in AI. The authors in
[4] provide a comprehensive overview of AI interpretability and its larger
denomination with related fields of privacy, transparency, safety,
explainability, and ethics. In this article, we specifically focus on
explainability on visual data for a specific modality of machine learning,
namely neural networks.

The generalization and task-independent modeling capabilities of neural
networks have resulted in their widespread applicability. Generalization
refers to a neural network’s ability to regularize to unseen data. Effective
generalization is required for large-scale learning. Task-independent modeling
refers to transference of low-level features between tasks and networks. For
instance, pretrained neural nets that are trained on natural images are used
on computed seismic images [5]. These two capabilities coupled with hardware
improvements have allowed neural nets to achieve state-of-the-art performance
on a number of tasks. Specifically in image classification, neural networks
surpassed top-5 human accuracy of $94.9\%$ on ImageNet dataset [6]. While
gaining such high performance accuracy is an achievement, it is equally
important to understand the reasons that lead to such high accuracy rates.
Thus, it is imperative to study explainability in neural networks to better
understand the underlying science behind the phenomenon. However,
explainability in such networks is challenging since neural nets do not
conform to the traditional notion of AI interpretability. An example of AI
interpretability is through decision trees [7]. The idea that neural networks
should obtain explainability similar to decision trees is not unreasonable.
Decision trees are non-linear models that create local binary partitions. This
is not dissimilar to neural net neurons with ReLU activation functions that
either propagate or suppress features. Providing such an explanation requires
a complete understanding of features at each node. While decision trees are
designed using disentangled and interpretable features, neural networks learn
such features along with feature interactions. This challenge has recently
been compounded by the scale of neural networks. The natural language
processing model GPT-3 [8] consists of 175 billion parameters. Parsing all
parameters and decisions within such models is impractical. Hence,
explainability in neural networks is abstract. This abstract definition of
explainability in neural networks is motivated by the broad definition of
explanations themselves.

Figure 1: Role of explainability demonstrated through targeted toy examples.
Note that $f(\cdot)$ is not trained to answer abstract questions. The
explanations provide additional insights beyond the trained solution.

Explanations are defined as a set of rationales used to understand the reasons
behind a decision [9]. If the decision is based on visual characteristics
within the data, the decision-making reasons are visual explanations. Consider
the image of a spoonbill in Fig. 1. A trained neural network correctly
recognizes it as a spoonbill. Consider the form of explanations that are
acceptable for such a decision. In Fig. 1, we present three such toy
explanations all of which allude to the general goals of explainability. All
three explanations are abstract answers to specific questions. When a kid is
interested to know _‘Why Spoonbill?’_ , the network’s explanation highlights
the body and head of the bird leading to that decision. This is an abstract
explanation that requires humans to take the extra step of knowing that the
highlighted regions depict the spoonbill’s flat-beak and its pink and round
body. For the neural network, these characteristics determine the class of the
given bird. Note that these are not causal factors. Flamingos have the same
body shape and color that the explanation highlights. Consider the scenario
when an ornithologist, armed with the knowledge that flamingos and spoonbills
share the same body shape and color, asks the question _‘Why Spoonbill, rather
than a Flamingo?’_. Such a targeted and contextualized question elicits the
explanation that the two birds differ in the shape of their necks -
specifically, the lack of an S-shaped neck in a spoonbill is highlighted as to
why it is not a flamingo. The relevance of the lack of an S-shaped neck is
accentuated by the question, _‘Why Spoonbill, rather than a Flamingo?’_. Such
an explanation can be used to assess the network. In the last toy example, the
network answers the question, _‘Why Spoonbill, rather than a Fox?’_ by
highlighting all the body parts of the bird. Note that the network is only
trained to classify birds and has _not_ explicitly been trained to answer
these questions. Hence, all these intelligible and abstract explanations
together serve to garner _trust_ in the network and its inference
capabilities.

The central thesis of this article is to motivate and categorize existing
abstract explanatory techniques based on specific questions. We consider three
types of questions, namely observed correlations _‘Why P?’_ , observed
counterfactual _‘What if?’_ , and observed contrastive _‘Why P, rather than
Q?’_. Here $P$ is any prediction from a neural network, like a spoonbill, and
$Q$ is any contrast class, like a flamingo or a fox. The answers to these
questions are _observed_ in the sense that the decision is already made.
Together, these three questions provide complementary explanations that we
term as explanatory paradigms. Their complementary nature allows us to define
complete explanations as justifications for any and all posed contextually
relevant questions. For the observed counterfactual and contrastive paradigms,
there is an additional intervention that allows for their respective
questions. These explanations are not _interventionist_ from a causal
perspective [10]. Interventions refer to manipulations in data that are
designed to probe a network’s class causality and construct counterfactuals.
In this article, the interventions considered are based on a network’s notion
of classes, through gradients. By considering such interventions, we elicit
model behaviors that provide responses to the ornithologist and fox from Fig.
1 without the need for engineered interventions. The exact definition of each
of the question-based paradigm and its relationship with existing
categorizations is presented in Section III. We start by describing the
existing definitions of explainability in neural networks and provide our own
definition that is both intuitive and mathematically rigorous in Section II.
We then summarize the explanatory paradigms and their advantages in Section
III before combining them into complete explanations. A taxonomy of
explanatory evaluation is provided in Section IV followed by a summary of
existing methods of explaianbility in Table I. We discuss existing challenges
in explainability that inform future research in Section VI before concluding
in Section VII.

##  II Background

We first discuss requisite neural network preliminaries. While the focus is on
classification networks, the preliminary discussion extends to other
discriminative and generative networks as well. This is followed by existing
and our own definitions of explainability.

###  II-A Neural Network Preliminaries

Classification neural networks learn non-linear transformations to obtain
discriminative representation spaces on any given data. The inherent
mechanisms behind the discriminative process can be formulated as follows. Let
$f(\cdot)$ be a neural network trained to distinguish among $N$ classes. If
$x$ is any input to the network, the output logits in a classification network
are given by

$\begin{gathered}\hat{y}=f(x),\forall\hat{y}\in\Re^{N\times 1},\end{gathered}$
(1)

where $\hat{y}$ is an $(N\times 1)$ vector. The predicted class of $x$, given
by $P$, is the index of the maxima of $\hat{y}$. i.e.,

$\begin{gathered}P=\operatorname*{arg\,max}\hat{y},P\in[1,N].\end{gathered}$
(2)

During training, the network is penalized for predicting the wrong class.
Penalty is calculated by using an empirical loss function, $J(P,y)$, where $y$
is the true label for input $x$. This loss is backpropagated through the
network. The network parameters are updated based on gradients $\frac{\partial
J(P,y)}{\partial\theta}$ where $\theta$ are the network parameters, namely its
weights and biases. Mathematically, this update is given by,

$\theta^{\prime}=\theta-\frac{\partial J(P,y)}{\partial\theta},$ (3)

where $\theta^{\prime}$ are the parameters after the update. A full
description of backpropagation and the associated math is presented in [11].
The gradients can be backpropagated all the way back to the input $x$ to
obtain $\frac{\partial J(P,y)}{\partial x}$ or to the activations at some
layer $L$, given by $A_{L}$, to obtain $\frac{\partial J(P,y)}{\partial
A_{L}}$. These quantities represent sensitivity which is defined as a measure
of the change in the prediction loss given a small perturbation either in $x$
or $A_{L}$. Parameter gradients and sensitivity are widely used as explanatory
features among a number of explanatory methods including [12, 13, 14, 15, 16,
17, 18].

###  II-B Interventions, causality, and _post-hoc_ explanations

Interventions in data are manipulations that are designed to test for causal
factors [10]. The authors in [19] use the presence of interventions to
motivate two types of data, namely observational and interventionist. The
interventionist data is specifically engineered to test for causal variables.
It employs the _do_ operator and is generally given as $\mathbb{P}(Y|do(x))$
for data $x$ and label $Y$. The _do_ operator represents interventions.
However, all possible interventions in data can be long, complex and
impractical [20]. Even with interventions, it is challenging to estimate if
the resulting effect is a consequence of the intervention or due to other
uncontrolled interventions. Instead of interventions, _post-hoc_ explanations
highlight features using a trained network that aid inference. In this
article, we use the term _observed_ to denote a specific case of _post-hoc_
explainability after a network has made its decision. We then construct
gradient-based observed interventions to create complete observed
explanations.

###  II-C Explanation Maps

The definition of explanatory maps in neural networks has evolved over time.
The early works for explainability [21, 22] consider explanations to be
processes that illustrate the workings of neural networks. The deconvolution-
based explainability technique proposed in [21] shows that at different layers
and different kernels, different structures in the data activate to create
local decisions. Deconvolution, unpooling, and rectification are used to
reconstruct and visualize activations at all layers. These visualizations are
analyzed based on layers and kernels to obtain insight regarding individual
parameters within the network. Such a hierarchial representation is followed
in [23] where the authors consider the neural network as an explanatory graph.
From this methodology of obtaining explanations, one could define explanations
as visualizations that require further deductions regarding the network and
its outputs. We term such explanations as _indirect explanations_. An
alternative definition of explainability is based on [24] that defines visual
explanations as localization maps. Such a definition is based on the argument
that neural networks, with no explicit training, implicitly localize objects
that they recognize. We term these types of explanations as _direct
explanations_. Direct explanations passively highlight all regions in an image
that are used to predict $P$. [24] produces a Class Activation Map (CAM) that
highlights parts of the image leading to the final decision. Direct
explanations are passive in the sense that they retroactively justify
decisions.

Recently however, direct and indirect definitions of explanations are
insufficient to categorize some of the newer explanatory techniques [12, 13].
The authors in [12] show explanations for any class in any given image even
when objects from that class are not present in the image. The authors in [13]
show the contrast between any two classes in the image. This is depicted in
the contrast between a spoonbill and flamingo in Fig. 1. Neither the
definitions of localization, nor reconstruction encompass these works.
However, such works lead to explanations taking an active role in neural nets
by participating in decision making or assessing their parent networks [14].
These explanations encompass all three explanatory roles sought in Fig. 1. In
this article we propose a new and simple probabilistic definition for
explanations that combine newer and older explainability techniques, and
construct a more comprehensive paradigm.

In order to better understand and distinguish the definitions of
explainability, consider an image $x$ fed into a trained network $f(\cdot)$.
The network is trained to distinguish $x$ between $N$ classes as $P\in Y$. Let
$\mathcal{T}$ be the set of all features on which the network has learned to
base its decision. In other words, a network infers that $x\in P$, if $x$ has
features $\mathcal{T}_{p}$. Hence
$\mathcal{T}=[\mathcal{T}_{1},\mathcal{T}_{2}\dots\mathcal{T}_{M}]$, where $M$
is the total number of features that is learned by a network. Note that we are
not considering $\mathcal{T}_{i},i\in[1,M]$ to be the causal features. In
other words, $\mathcal{T}_{i}\cap\mathcal{T}_{j}\neq{0},\forall i\neq j$.
Features from multiple sets constitute $\mathcal{T}_{p}$ and are used to make
inferences on $x$, i.e,
$\mathcal{T}_{p}=\bigcup\limits_{i=1}^{K_{p}}\mathcal{T}_{i}$,
$K_{p}\in[1,M]$. We define an explanatory map $\mathcal{M}$ as follows:

######  Definition II.1 (Observed Explanation).

An explanatory map is any function $\mathcal{M}(\cdot)$ that maximizes the
conditional probability
$\mathbb{P}(\bigcup\limits_{i=1}^{K}\mathcal{T}_{i}|Y)$, where $Y$ is any
possible class, $K\in[1,M]$ and $\bigcup\limits_{i=1}^{K}\mathcal{T}_{i}$ is
the union of all required features $\mathcal{T}_{i}$, that specifically lead
to $Y$.

In Definition II.1, we assume the decision $Y$ before maximizing the features
$\mathcal{T}_{i}$. However, by not specifying the value for $Y$, we allow for
interventions in labels to create contrastive paradigms. By specifying the
features as non causal $\mathcal{T}_{i}$, we open the possibility of changing
the features to create counterfactual paradigms. This allows specific
interventions in $Y$ and $\mathcal{T}_{i}$. Definition II.1 is not _causal
interventionist_ since we neither alter the feature set T \- interventionist
counterfactual - nor probe on $x$ for interventionist causality. Our
definition provides for restricted interventions for a decision $P$ and we
term it _observed_. Our observed definition of explanation comprises both
indirect and direct explanatory techniques. Direct and indirect explanations
maximimally highlight features that explain the decision $P$. In other words,
their goal is to maximize $\mathbb{P}(\mathcal{T}_{p}|P)$ for the specific
instance of $Y=P$. Our general definition explains any decision, $Y$, by
maximally highlighting features $\bigcup\limits_{i=1}^{K}\mathcal{T}_{i}$. The
newer explanatory techniques like [12, 13] that do not conform to direct and
indirect definitions of explanations can be explained using Definition II.1.
This is presented in Section III.

Figure 2: Three paradigms of explanations, each answering observed
correlation, counterfactual, and contrastive questions, respectively.

##  III Explanatory Paradigms

###  III-A Observed Explanatory Paradigms via Reasoning

We first motivate our observed paradigms through reasoning. The primary goal
of neural networks is to infer decisions. Any network $f(\cdot)$, is trained
to maximize $\mathbb{P}(P|\mathcal{T}_{p})$, i.e, $\mathcal{T}_{p}$ is the
reason that leads $f(\cdot)$ to predict $P$. Hence, explanations are a means
to extract these reasons. There are three ways in which reasoning can occur -
deductive, inductive, and abductive reasoning [14]. Deductive reasoning occurs
when networks are provided with a set of features and logic to use the given
features. Since both the features, and the logic to manipulate the features
are learned by the network, deductive reasoning does not play a large role in
neural nets. Inductive reasoning forms the core of the reasoning behind
current networks. A set of parameters are learned consisting of some
underlying rules and knowledge. During inference, neural nets associate given
data with the knowledge in their parameters and make their predictions with a
probability. Explanations are then generated to justify such a decision.
Direct and indirect explanations are a product of this reasoning scheme. The
third form of reasoning is abductive reasoning. The primary difference between
inductive and abductive reasoning is the usage of explanations. In abductive
reasoning, explanation is any hypothesis that supports a prediction. In Fig.
1, all three explanations support the prediction that the given image is that
of a spoonbill. The network can detect the presence of the pink and round
body, and straight beak to inductively infer the class of the bird as
spoonbill. Or, alternatively, it can hypothesize as to why the bird cannot be
a flamingo - lack of S-shaped neck - and then decide that the bird has to be a
spoonbill. Such an inference is abductive. A number of such hypotheses can be
constructed, fused and incorporated together, and then used to make a
decision.

In this article, we present three specific ways of generating such hypotheses
- as answers to causal questions _‘Why P?’_ , counterfactual questions _‘What
if not P?’_ , and contrastive questions _‘Why P, rather than Q?’_. However,
since we use observed techniques, the network provides correlation features to
the causal question and we name it observed correlation. In Section III-E, we
show that these three explanatory paradigms are sufficient to generate
probabilistically complete explanations. A visual depiction of the three
paradigms is presented in Fig. 2. Consider a bullmastiff as the given input
image. The observed correlation explanation answers _‘Why Bullmastiff?’_ by
highlighting the face of the dog.111Grad-CAM [12] is used to obtain this
explanation. The question _‘What if the bullmastiff was not in the image?’_ is
a counterfactual question. As highlighted using the technique in [12], the
network would have made a decision based on the cat for the given
counterfactual. The contrastive question is of the form _‘Why Bullmastiff,
rather than Boxer?’_. The heavy jowls of the bullmastiff and the presence of a
cat preclude the network from predicting a boxer as highlighted by Contrast-
CAM from [13]. Contrastive explanations take into account the context provided
by the questionnaire thus making explanations more relevant. Note that in all
three cases, the features set $\mathcal{T}$ highlighted by the explanations
are different. However, finding any feature set $\mathcal{T}$ in neural
networks is challenging because features $[\mathcal{T}_{i}],\forall i\in[1,M]$
are not disentangled. Rather, feature extraction and inference occur
simultaneously by projecting data onto network parameters. We consider
$\mathcal{T}$ to be the span of network parameters or as manifolds in a high
dimensional space. This manifold is, however, different for different network
architectures. Hence, each feature set $\mathcal{T}$ is tied to a specific
neural network $f(\cdot)$ as $\mathcal{T}^{f}$. To keep the notations simple,
we ignore the superscript and name the three paradigms as observed
correlation, observed counterfactual, and observed contrastive explanations.
Observed, in this article, is from the perspective of a neural network.

Figure 3: Three paradigms of explanations, and their manifolds. All requisite
changes in the manifolds can be tracked using gradients from Eq. 3.

###  III-B Observed Correlation Explanations

Observed correlation explanations are answers to _‘Why P?’_ questions. $P$ can
be any learned class but is generally the prediction made by the network.
Observed correlation explanations can be modeled using any of indirect or
direct explanations from Section II-C. Using Definition II.1, observed
correlation explanations $\mathcal{M}_{cu}(\cdot)$ find the feature set
$\mathcal{T}_{p}$ that satisfy $\mathbb{P}(\mathcal{T}=\mathcal{T}_{p}|Y=P)$.
In Fig. 3a, we visually represent the search for the observed correlation
features for the example of the spoonbill image. Consider the blue manifold as
the learned manifold where the image of the spoonbill is classified as a
spoonbill. The set of all possible features $[\mathcal{T}_{i}],\forall
i\in[1,M]$ exists on the blue manifold. The goal of any observed correlation
explanation is to find the features
$\mathcal{T}_{p}=\bigcup\limits_{i=1}^{K}\mathcal{T}_{i}$, that span the
purple manifold within the larger blue manifold. While the blue manifold uses
all parts of $x$ to recognize a spoonbill, only the features in its legs and
neck are sufficient for $x$’s recognition. Hence $\mathcal{T}_{p}$ are
features in a spoonbill’s neck and legs.

Note that the observed correlation features presented here are different from
interventionist causal factors discussed in [25][19][10] even though the
question it answers is causal. These are _post-hoc_ explanations observed from
a network. The observed correlation explanations take the prediction $P$ into
consideration to retroactively extract features $\mathcal{T}_{p}$. A list of
common existing observed correlation techniques are shown in Table I.

###  III-C Observed Counterfactual Explanations

Counterfactual Explanations are contextually relevant explanations that are of
the paradigm _‘What if?’_. The explanation to _‘What if the bullmastiff was
not in the given image?’_ is shown in Fig. 2. Other counterfactual questions
can take the form _‘What if the dog had feathers?’_ or _‘What if the dog had a
longer neck?’_. Generally, these questions are tackled by intervening within
the image in [26][10][19]. In the observed paradigm, we intervene in the
feature space $\mathcal{T}$. Conceptually, this is similar to maximizing
$\mathbb{P}(do(\mathcal{T})|Y=y)$ from Section II-B. These _do_ interventions
are limited since the original feature set $\mathcal{T}$ remains unchanged.
Depending on the question asked, observed counterfactual explanations
$\mathcal{M}_{cf}(\cdot)$, are given by the probability of
$\mathcal{T}_{\hat{y}}$ that satisfy
$\mathbb{P}(\mathcal{T}=\mathcal{T}_{\hat{y}}|Y=y)$. $\mathcal{T}_{\hat{y}}$
is defined similarly to observed correlation explanations except that there is
an intervention in the feature set of the label depicted by $\hat{y}$. This
intervention can either be the absence of $y$, like the counterfactual
question in Fig. 2, or alterations in the data that changes $y$, like in [26].

In Fig. 3b, we visualize three possible interventions on the spoonbill when
$y=P$. The first two interventions occur on the legs and neck of the spoonbill
respectively. These intervened images $x_{1}$ and $x_{2}$ are projected onto
separate counterfactual manifolds and are classified as flamingos. The last
intervention occurs in the body of the flamingo but is insufficient to change
its classification. Nevertheless, $x_{3}$ is projected onto a different set of
features that span a new manifold. Hence, in this toy example, there are three
separate $\hat{p}$ and three $\mathcal{T}_{\hat{p}}$. Note that the underlying
features, $\mathcal{T}=[\mathcal{T}_{1},\mathcal{T}_{2}\dots\mathcal{T}_{M}]$
remain the same. The union of the subsets given by
$\bigcup\limits_{i=1}^{K}\mathcal{T}_{i}$ is different. This suggests that no
new features are learned, rather a new union of existing features is formed.
The result shown in Fig. 2 illustrates how the network can highlight the
features in the cat when denied the features of the dog. The authors in [20]
show that interventions can be long, complex and impractical.

###  III-D Observed Contrastive

The third explanatory paradigm of contrastive explanations provide an
intuitive resolution to the intervention complexity challenge by taking
advantage of the implicit notion about individual classes learned by the
network. They answer questions of the form _‘Why P, rather than Q?’_ , where
$P$ is the predicted class and $Q$ is any learned contrast class. Contrastive
explanations take into account the context provided by the questionnaire thus
making explanations more relevant. For a given class $Q$ and predicted class
P, contrastive explanations $\mathcal{M}_{ct}(\cdot)$ maximize the probability
$\mathbb{P}(\mathcal{T}=\mathcal{T}_{p,q}|Y=Q)$, where
$\mathcal{T}_{p,q}=\bigcup\limits_{q=1}^{K_{1}}\mathcal{T}_{p}\cap\mathcal{T}_{q}$.
From Section II-B, we are intervening in the label space as
$\mathbb{P}(\mathcal{T}|do(Y))$ to create observed contrastive paradigm.

Consider the learned blue manifold in Fig. 3c which consists of all features
$\mathcal{T}$ that are used to recognize a spoonbill. Contrastive explanations
require a new hypothetical manifold that produces enough interventions in
$\mathcal{T}$ to predict a spoonbill as a flamingo. This occurs when the
parameters of the blue manifold are changed to obtain the contrastive purple
manifold. The gradients described in Eq. 3 is used to adjust parameters
between learned and contrastive manifolds in [13].

###  III-E Probabilistic Completeness of explanations

An advantage of defining the three observed explanatory maps and their
associated paradigms in a probabilistic fashion is that they can be combined
into complete explanations. The authors in [27] describe completeness in
neural networks as explaining all parameters. In this article, we explain all
probabilities. From Eq. 2, a neural network can have $N$ class probabilities.
An explanatory technique must answer for any one of these possible decisions
in the form of correlation-based, counterfactual, and contrastive questions.
Such an explanation is termed as a complete explanation. A rigorous definition
is provided below:

######  Definition III.1 (Complete Explanation).

A complete explanation, $\mathcal{M}_{c}(\cdot)$, is a union of all functions
$\mathcal{M}(\cdot)$ that simultaneously maximize the conditional
probabilities $\mathbb{P}(\mathcal{T}|Y)$, for all $Y\in[1,N]$ and all
features $\mathcal{T}$, i.e,
$\mathcal{M}_{c}(\cdot)=\bigcup\limits_{j=1}^{N}\mathcal{M}_{j}(\cdot)=\bigcup\limits_{j=1}^{N}\mathbb{P}(\bigcup\limits_{i=1}^{K}\mathcal{T}_{i}|Y_{j})$,

where $\mathcal{M}(\cdot),\mathcal{T},Y$ and $K$ are defined in Definition
II.1. Complete explanations can simultaneously answer questions regarding any
class $Y$. For instance, in Fig. 2, given a dog and a cat image, a question of
the form _‘Why Bluejay?’_ is an acceptable question based on Definition III.1
since it probes $Y=y_{\text{Bluejay}}$ and since $f(\cdot)$ has learned the
Bluejay class. However, it is irrelevant seeing that there is no Bluejay in
the image. An alternative and relevant question of the form _‘Why Bullmastiff,
rather than a Bluejay?’_ , can be used to probe $Y=y_{\text{Bluejay}}$. An
explanation for such a question highlights the face of the dog [14]. Note that
the three paradigms provide a way to probe all class probabilities in a
contextually relevant fashion.

Consider a binary classifier $f(\cdot)$ with $P$ and $Q$ as the two possible
classes. Let $\mathbb{P}(P)$ and $\mathbb{P}(Q)$ be the probabilities of
$f(\cdot)$ predicting a given image $x$ to belong to $P$ and $Q$,
respectively. For a binary classifier, the sum of both probabilities add to
$1$,

$1=\mathbb{P}(P)+\mathbb{P}(Q).$ (4)

Decomposing Eq. 4 using the law of total probability, we have

$1=\mathbb{P}(P|\mathcal{T}_{p})\mathbb{P}(\mathcal{T}_{p})+\mathbb{P}(P|\mathcal{T}_{p}^{c})\mathbb{P}(\mathcal{T}_{p}^{c})+\mathbb{P}(Q|\mathcal{T}_{q})\mathbb{P}(\mathcal{T}_{q})+\mathbb{P}(Q|\mathcal{T}_{q}^{c})\mathbb{P}(\mathcal{T}_{q}^{c}),$
(5)

where $\mathcal{T}_{p}^{c}$ and $\mathcal{T}_{q}^{c}$ are complements. In
other words, $\mathcal{T}_{p}^{c}$ refers to the set of features not used to
make the decision $P$ and $\mathcal{T}_{q}^{c}$ are the features not used to
decide $Q$. Using Bayes’ theorem on all terms in the right hand side, by
eliminating the probabilities of the features in the numerator and
denominator, and rearranging to obtain $\mathcal{T}_{p}^{c}$ with
$\mathcal{T}_{q}^{c}$, we get,

$1=\mathbb{P}(\mathcal{T}_{p}|P)\times\mathbb{P}(P)+\mathbb{P}(\mathcal{T}_{q}|Q)\times\mathbb{P}(Q)+\mathbb{P}(\mathcal{T}_{p}^{c}|P)\times\mathbb{P}(P)+\mathbb{P}(\mathcal{T}_{q}^{c}|Q)\times\mathbb{P}(Q).$
(6)

$\mathcal{T}_{p}^{c}$ is analogous to the form _‘What if the bullmastiff was
not in the image?’_ in Fig. 2, where the network makes a decision without
considering the dog features. Similarly, $\mathcal{T}_{q}^{c}$ is of the form
_‘What if the tigercat was not in the image?’_. Hence, a function that
maximizes these terms, $\mathcal{M}_{cf}(\cdot)$, are counterfactuals from
Section III-C. Therefore, any explanatory map that maximizes the probability
term with $\mathcal{T}^{c}$ is an observed counterfactual explanation.
Similarly, $\mathbb{P}(\mathcal{T}_{p}|P)$ and $\mathbb{P}(\mathcal{T}_{q}|Q)$
probability terms are maximized by functions that are observed correlation and
contrastive explanations from Sections III-B and III-D respectively. The sum
of these functions maximizes all probabilities for data $x$, and hence is a
complete explanation $\mathcal{M}_{c}(x)$. Substituting these functions in Eq.
6, and using Definition III.1, we have,

$\mathcal{M}_{c}(x)=\mathcal{M}_{cu}(x)+\mathcal{M}_{ct}(x)+\mathcal{M}_{cf}(x),$
(7)

Hence, a complete explanation that accounts for all probabilities in a network
simultaneously answers the three observed correlation, counterfactual, and
contrastive questions. In other words, an explanatory technique that answers
observed correlation, counterfactual, and contrastive questions is
sufficiently complete. In this article, we advocate for creating complete
explanations. We show that a combination of gradient-based methods from [12],
and [13] can be used to achieve such an explanation in Section III-F. In
Section III-G, we elaborate on the significance of a complete explanation.

###  III-F Gradient-based Complete Explanations

In Fig. 3, the requisite change in manifolds for observed correlation,
counterfactual, and contrastive explanations occurs because of the change in
the network parameters that span the manifolds. From Eq. 3, the change in
parameters can be characterized by gradients. In this section, we show that a
gradient-based technique can provide a complete explanation. The authors in
[12] propose Grad-CAM, an explanatory technique that provides a gradient based
localization map leading to a decision. Grad-CAM is also used to create
counterfactual explanations by altering the gradients. Grad-CAM is further
extended as Contrast-CAM in [13] that answers contrastive questions. All these
works use the same underlying explanatory mechanism. The difference is in the
gradients used within each technique to access the requisite manifolds. Each
of these techniques is elaborated below.

Grad-CAM [12] Grad-CAM is used to visually justify the decision $P$ made by a
classification network by answering _‘Why P?’_. The activations from the last
convolutional layer of a network are used to create these visualizations since
they possess high-level semantic information while retaining spatial
information. For any class $i,\forall i\in[1,N]$, the logit $y_{i}$ is
backpropagated to the feature map $A_{l}$ where $l$ is the last convolutional
layer. The gradients at every feature map are $\frac{\partial y_{i}}{\partial
A_{l}^{k}}$ for a channel $k$. These gradients are global average pooled to
obtain importance scores $\alpha_{k}^{cu}$ of every feature map in $l^{th}$
layer and $k^{th}$ channel. The individual maps $A_{l}^{k}$ are multiplied by
their importance scores $\alpha_{k}$ and averaged to obtain a heat map. The
Grad-CAM map at layer $l$ for class $i$ is given by
$\mathcal{M}^{i}=ReLU(\sum_{k=1}^{K}\alpha_{k}^{cu}A^{k}_{l})$.

Counterfactual-CAM [12] The authors in [12] extend Grad-CAM by negating the
gradients as $-\frac{\partial y_{i}}{\partial A_{l}^{k}}$ thereby decreasing
the effect of the predicted class. The resulting importance score
$\alpha_{k}^{cf}$ de-emphasizes the region used for classification.
Effectively, these regions will be eliminated and the highlighted regions are
the ones that the network would have used to make its decision if the object
used to make the decision were not present. Hence, it is of the _‘What if?’_
modality. The final counterfactual explanatory map at layer $l$ is given by
$\mathcal{M}=ReLU(\sum_{k=1}^{K}\alpha_{k}^{cf}A^{k}_{l})$.

Contrast-CAM [13] In Grad-CAM, the importance score $\alpha_{k}^{cu}$ weighs
the activations in a layer $l$ based on $A_{l}^{k}$’s contribution towards
classification. The activations are projections on network parameters and
hence have access to both the correlation and contrastive information. The
goal is to obtain contrast-importance score $\alpha_{k}^{ct}$ that highlights
contrastive information in $A_{l}^{k}$. Firstly, a loss between predicted
class $P$ and any contrast class $Q$ is constructed as $J(P,Q)$. This loss
function inherits the difference between the classes $P$ and $Q$. Next,
$J(P,Q)$ is backpropagated within the Grad-CAM framework to obtain
$\alpha_{k}^{ct}$ as global average pooled contrastive features i.e
$\alpha_{k}^{c}=\sum_{u}\sum_{v}\nabla_{W_{l}}J(P,Q)$, where $u,v$ are the
channel sizes at layer $l$. Contrast-CAM map for contrast class $Q$ is given
by, $\mathcal{M}^{i}=ReLU(\sum_{k=1}^{K}\alpha_{k}^{ct}A^{k}_{l})$.

Figure 4: Observed correlation, counterfactual, representative contrast class
and contrastive explanations, respectively on a fault image. The _‘Why
Fault?’_ is answered by highlighting the discontinuity features
$\mathcal{T}_{P}$ in the rocks that correlates to presence of a fault. The
counterfactual explanation analyzes the decision after removing
$\mathcal{T}_{P}^{c}$. The contrastive explanation highlights features that
are an intersection of $\mathcal{T}_{P}$ and $\mathcal{T}_{Q}$. Here, $P$ is
the fault class and $Q$ is the salt dome class.

###  III-G Significance of Observed Explanatory Paradigms and Complete
Explanations

In this section, we elaborate on the significance of all three paradigms of
observed explanations.

####  III-G1 Results on non-interventionist data

The results shown in Figs. 1 and 2 are on natural images when $f(\cdot)$ is a
VGG-16 architecture. The observed correlation and counterfactual explanations
are extracted using [12], and observed contrastive is extracted through
Contrast-CAM [13]. In Fig. 4, a fault in the seismic section of the
Netherlands F3 block [28] is analyzed. On such data, active interventions from
a causal perspective are not possible. Instead, we rely on our observed
paradigms. The dataset [28] has four classes - faults, salt domes, chaotic
regions, and horizons. A ResNet-18 network is trained and the corresponding
Grad-CAM [12], Counterfactual-CAM [12], and Contrast-CAM [13] explanations
extracted from the trained model are shown. The observed correlation
explanation _‘Why fault?’_ , highlights the region where faults are
recognizable. The observed contrastive explanation to the question _‘Why
fault, rather than salt dome?’_ , performs the harder task of tracking the
fault since the Grad-CAM explanation can also be confused to be highlighting
the salt dome region. The counterfactual explanation answers _‘What if the
fault did not exist?’_ , and highlights the horizons class indicating that
horizons would be the classification. All three paradigms provide new insights
to seismic interpreters who are then qualified to assess the final decision.
Note the complementary nature of the explanations. All features are
highlighted across the seismic section, thereby visually validating Eq. 7.

####  III-G2 Using complete explanations to differentiate context features
from observed correlation features

The authors in [15] use the completeness of observed gradient-based methods to
provide a methodology to obtain separate out context features
$\mathcal{T}_{c}$ from observed correlation features $\mathcal{T}_{p}$. They
analyze Grad-CAM and postulate that the Grad-CAM explanation _‘Why P?’_ is a
combination of causal and context features making it correlations. They use a
set-theoretic approach to provide relationships between the two sets of
features and define context features interms of contrastive and counterfactual
features. This work can be considered as an explanatory bootstrap where
multiple methods are tied together to produce a new explanation. Such a
bootstrap is possible because each individual method is answering a specific
question.

####  III-G3 Using gradient features for applications

_Post-hoc_ explanations are passive in the sense that they justify decisions
once decisions are made. However, the observed contrastive and counterfactual
explanations can actively be used for inference and training. The authors in
[29] construct contrastive explanations to mine pertinent negatives to train
networks. The authors in [14] use contrastive explanations and features to
create robust networks. They show that usage of contrastive features
$\mathcal{T}_{p,q}$ in addition to observed correlation features
$\mathcal{T}_{p}$ increases the robustness of a network to noise by $4\%$ on
CIFAR-10C [30] dataset. This robustness is also showcased on domain shifted
data where the accuracy increases by $5.2\%$ on Office [31] dataset. The
authors use these contrastive $\mathcal{T}_{p,q}$ features as a plug-in on top
of existing $\mathcal{T}_{p}$ features thereby heuristically validating the
significance of complementary observed paradigms.

####  III-G4 Analyzing explanatory techniques through observed paradigms

By defining explanations in terms of probabilities, we tie existing
explanatory techniques to reasoning and complete them based on abstract
questions that use learned features $\mathcal{T}$. This provides insights into
existing explanatory techniques along with a unique avenue for analyzing
trends in the field of explainability. Comparisons of existing methods is
performed in Table I. We first provide a taxonomy of evaluation for the
observed paradigms before analyzing the trends. We then showcase challenges in
existing explainability techniques as analyzed through the lens of our
question-based paradigms.

Figure 5: Three taxonomies of evaluation based on human involvement. The first
is a direct evaluation of explanations by humans. The second defines
explanations as proxy tasks that require human evaluation. The third uses
existing tasks that have shown to require interpretability.

##  IV Evaluating Explanations

The subjectivity of explainability creates a challenge for its evaluation. For
a number of applications including recognition, the only available ground-
truths are object labels. Hence, explainability is not a task with ground-
truth labels which can be easily evaluated. The authors in [4] provide a broad
evaluation overview for AI techniques. We adapt this framework for neural
network explainability and present it as an Explanatory Evaluation Taxonomy.
An overview is provided in Fig. 5. Inherently, explanations are for humans.
All evaluation criteria must be grounded in either direct or indirect
interactions with humans. Consequently, there are three possible taxonomies
that existing explanatory techniques use for evaluation. We first introduce
and elaborate on these taxonomies. Specific explanatory paradigm techniques
and their related evaluation criteria are provided in Section V.

###  IV-A Direct-Human Evaluation

In this taxonomy, explanations serve to directly convince humans of the
decisions made by neural networks. As such, subjectively assessing the quality
of explanations involves humans in the loop. With the widespread applicability
of neural networks, the most common evaluation of explainability is
qualitative Direct-Human evaluation. Authors show application results that
$f(\cdot)$ was trained for, like recognition accuracy, along with selected
explanations for images. This is true for all data domains including natural
images [21], medical images [15], seismic images [5] among others. Early
explainability techniques including [21, 17, 18] all follow this approach. A
quantitative approach to direct-human evaluation is shown in [12]. The authors
in [12] create explanation maps on random $90$ images from ImageNet and use
Amazon Mechanical Turk to find and ask people to rate their explanation maps
as compared against explanations created by [24, 18]. Note that the goal here
is not to evaluate the efficacy of the explanatory map by itself, but rather a
way to compare between explanatory methods.

While subjectively evaluating explanation maps is resource intensive, it
provides new insights into data and applications. In the application of Image
Quality Assessment (IQA), the authors in [13] show that contrastive
explanations provide additional context to the explanations. In Fig. 6, $P$ is
the estimated quality score and $Q$ is a given contrast score. When $Q>P$, the
contrastive explanations describe why a lower score is predicted. According to
the obtained visualizations when $Q=1$ and $Q=0.75$, the estimated quality is
primarily due to the distortions within the foreground portions of the image
as opposed to the background. Further analysis is provided in [13]. Since
Direct-Human evaluations provide new qualitative information, the authors
constructed contrastive features from existing IQA metric [32] and showed that
quantitative results increase across multiple IQA datasets [33].

Figure 6: Grad-CAM and Contrastive explanations of questions shown below each
image. Best viewed in color.

###  IV-B Indirect-Application Evaluation

In this taxonomy, tasks are designed that indirectly define explainability.
Human evaluation is conducted on such tasks and since explainability is
defined as a function of the task, explainability is evaluated simultaneously.
The advantage of such an approach is twofold : 1) The existing framework for
the application can be utilized, 2) Explanations are not directly measured
which validates the original supposition that explanations are not a task by
themselves but rather auxiliaries that support the reasoning behind tasks.

Evaluation tasks: In the the application of recognition, the authors in [24]
define explanations as localization maps of a given object within the image.
They evaluate explanations as localization, fine-grained recognition, weakly
supervised localization, pattern discovery, and Visual Question Answering. The
authors in [16] derive explanation $\mathcal{M}$ for an image $x$ and mask $x$
as $x^{\prime}=x\times\mathcal{M}$. They pass the masked image through the
network and check for accuracy. This is conducted across a dataset of images
for a number of explanatory techniques. Higher the accuracy, better is the
localization capability of the explanation, and hence better is the
explanation. Note that the definition of explanation here is localization of
objects. The authors in [34] use visual saliency application as a proxy for
explanation. The intuition is that human gaze lingers on the most salient
parts of a given image. These salient regions are explanations for decisions
made on that image since it attracts human gaze. The authors construct
explanation maps using existing techniques of Grad-CAM [12] and Guided
Backpropagation [18] and show that their own implicit saliency tracks the
human gaze with a higher correlation. Other applications include weak
segmentation [12], pointing game [35, 12] and image captioning [35].

Evaluation metrics: The novelty in Indirect-Application evaluation is in
aligning the definition of explanations with existing tasks. Once they are
aligned, the established metrics for each application is used for evaluation.
For instance, classification accuracy is used to evaluate masked images in
[16]. Normalized Scanpath Saliency (NSS) and Correlation Coefficient (CC) are
used in [34] for evaluating visual saliency.

###  IV-C Targeted-Network Evaluation

The authors in [35] question the nature of explainability as a tool for human
justification and instead suggest that explanations provide an insider’s view
into a neural network’s decision making process. Such a process need not
conform to that of a human’s. Explainability is connected to other
applications like robustness. The goal in this taxonomy is to create
experiments that increases performance on auxiliary applications thereby
increasing its explainability.

Evaluation Tasks: In [14], the application of robustness is tied to
explainability. The authors show observed correlation and contrastive
explanations for noisy images and how having a large corpus of explanations
can be robust to distortions. Consequently, the gradient features used in
explanations are used to train a simple multi-layer perceptron that recognizes
noisy images in a more robust fashion compared to a feed-forward network. The
same robustness is observed for domain adaptation experiments. These gradients
are used in other robustness applications including anomaly detection [36],
novelty detcetion [37], and out-of-distribution detection [38]. Gradients used
in this fashion across applications connected to explainability provide
indirect validation of their explanatory capacity. The authors in [26] propose
using counterfactual explanations for the application of machine teaching -
where humans learn about birds categorizations from neural networks. Note that
even though humans are involved, the task for humans is not to evaluate
explanations. Rather, they learn from explanations and the evaluation measures
how well humans learn.

Evaluation Metrics: The authors in [35] and [15] propose three new evaluation
criteria - all for the application of recognition. These include Probabilistic
deletion and insertion [35], accuracy deletion and insertion [15], and
transference of features [15]. All three metrics however, are for observed
correlation explanations. There are currently no specific evaluation metrics
designed for observed counterfactual or contrastive explanations. The
challenge is currently defining tasks for these two paradigms and the metrics
follow from the tasks - similar to Indirect-Human evaluation.

###  IV-D Correlation, Counterfactual, and Contrastive Evaluations

A complete summary of existing techniques and their evaluation strategies are
presented in Table I. These summaries provide insight into evaluation
strategies followed in each of the observed correlation, counterfactual, and
contrastive frameworks. Useful trends regarding evaluation, extracted from the
table are presented in Section V.

TABLE I: Explanation Map Categorizations. Paradigms | Methods | Definition | Technique Categorization | Evaluation
---|---|---|---|---
|  Indirect |  Direct |  Targeted |  Implicit |  Explicit |  Black-Box |  White-Box |  Intervention |  Non-Intervention |  Gradient-based |  Non-gradient based |  Direct-Human |  Indirect-Application |  Targeted-Network
Correlation |  Deconvolution [21] | ✓ |  |  |  | ✓ |  | ✓ |  | ✓ |  | ✓ | ✓ |  |
Inverted Representations [22] | ✓ |  |  |  | ✓ |  | ✓ |  | ✓ |  | ✓ | ✓ |  |
Guided-Backpropagation [18] |  | ✓ |  | ✓ |  |  | ✓ |  | ✓ | ✓ |  |  | ✓ |
SmoothGrad [17] |  | ✓ |  | ✓ |  |  | ✓ | ✓ |  | ✓ |  |  | ✓ |
LIME [39] |  | ✓ |  |  | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ | ✓ |
CAM [24] |  | ✓ |  |  | ✓ |  | ✓ |  | ✓ | ✓ |  |  | ✓ |
Graph-CNN [23] | ✓ |  |  |  | ✓ |  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |
GradCAM [12] |  |  | ✓ | ✓ |  |  | ✓ |  | ✓ | ✓ |  | ✓ | ✓ |
TCAV [40] |  | ✓ |  |  | ✓ |  | ✓ |  | ✓ | ✓ |  | ✓ | ✓ |
|  GradCAM++ [16] |  |  | ✓ | ✓ |  |  | ✓ |  | ✓ | ✓ |  | ✓ | ✓ |
|  RISE [35] |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  |  | ✓ |  | ✓ | ✓
|  Causal-CAM [15] |  |  | ✓ | ✓ |  |  | ✓ |  | ✓ | ✓ |  | ✓ |  | ✓
Counterfactual |  Counterfactual-CAM [12] |  |  | ✓ | ✓ |  |  | ✓ |  | ✓ | ✓ |  | ✓ |  |
Goyal et.al [26] |  |  | ✓ | ✓ |  |  | ✓ | ✓ |  |  | ✓ | ✓ | ✓ |
Contrastive |  CEM [29] |  |  | ✓ |  | ✓ |  | ✓ | ✓ |  |  | ✓ | ✓ | ✓ |
Contrast-CAM [13] |  |  | ✓ | ✓ |  |  | ✓ |  | ✓ | ✓ |  | ✓ |  |
Contrastive Reasoning [14] |  |  | ✓ | ✓ |  |  | ✓ |  | ✓ | ✓ |  | ✓ |  | ✓

##  V Categorizations, Trends, and Insights into Explainability

The explanatory paradigms are natural extensions of exiting works. While
categorization based on abstract explanations seem intuitive, existing
techniques derive their novelty by other means. Specifically, existing
explanatory categorizations are based on design choices within the explanatory
techniques themselves. These design choices include - need for $f(\cdot)$’s
parameters during explanations, architectural design change in $f(\cdot)$ for
deriving explanations among others. We first expand on these categorizations
before summarizing existing methods in Table I.

Implicit vs explicit explanations A number of techniques require an
architectural change to the network or a separate network to generate
explanations [21, 24, 29, 39, 40]. We term them as explicit explanations. In
some cases, the change in network architecture can potentially alter
underlying results [12]. Implicit explanations follow the original definition
of explanations which is that explanations justify inference.

Black-box vs White-box explanations Techniques including [21, 24, 12, 13] all
require access to the model, its parameters and/or network states to provide
explanations. These are termed white-box techniques. Other techniques exist
that treat neural networks as black-boxes to make decisions [35]. Both RISE
[35] and LIME [39] require model outputs and not parameters.

Interventionist vs Non-interventionist Explanations When an explanatory
technique requires changes in data, we term them as interventionist technique.
SmoothGrad [17] requires noise added to the images. Counterfactual visual
explanations [26] require parts of the input image substituted by another.
RISE [35] requires generation of a large quantity of masked input images. Non-
interventionist techniques do not require interventions in input data.

Gradient-based vs Non-gradient based Explanations A number of methods use
gradients described in Section II-A as features. Some methods like SmoothGrad
[17], Guided Backpropagation [18] directly use gradients as explanation maps.
Others including Grad-CAM [12], and Contrast-CAM [13] use them as features
within their frameworks.

The existing methods are all presented in Table I. They are chronologically
ordered within each paradigm. The rows ascertain memberships to paradigms
while there are three categorizations within columns. The first is the
definitions of explanations from Section II, the second is the technique-based
categorization presented earlier in the section, and finally the evaluation
categorization from Section IV. Note that in Table I, a targeted definition is
added to indirect and direct explanations. This includes all newer techniques
related to the reasoning paradigms. The Table provides key insights into the
evolution of explainability techniques and potential research directions.
These include:

  * •

Correlation methods are studied more than the other explanatory paradigms. The
indirect and direct definitions of explainability feeds observed correlations
and has been well studied over time.

  * •

Indirect and direct explanations have given way to targeted explanations.
Targeted explanations refer to explanations justifying contextually relevant
questions.

  * •

Of the 10 techniques that use direct human evaluation, only three [12, 16, 26]
do so by conducting random trials on humans. Others are qualitative works
whose images are potentially chosen by the researchers to showcase their
methods. More rigor is required in direct-human evaluation strategies.

  * •

The difficulty in evaluation has encouraged authors to evaluate their works
using multiple strategies in recent times. Since 2017, all but one work [13]
use more than one taxonomy to evaluate explanatory techniques.

  * •

There is no consensus among researchers regarding the targeted-network task
that evaluates explainability. The primary task using explanation masked
images [16] as well as robustness tasks [14] are used in a majority of these
evaluations.

  * •

While technique-based categorization provides some novelty to the methods
themselves, they are ineffective in predicting the trends in explainability
over time.

Figure 7: The contrastive explanation in (a) shows the disparity between human
and machine explanations. Both methods in (b) fail to highlight the red
circle.

##  VI Challenges in Explainability and Future Research Directions

###  VI-A Disparity between human and machine explanations

Consider the case in Fig. 7a when a network, trained on the CURE-TSR [41]
traffic sign dataset, is analyzed constructively. Given a No-Right sign and
asked a contrastive question of the form _‘Why No-Left, rather than Stop?’_ ,
the network highlights the bottom left corner of the image. The letters that
spell STOP not being in the sign is the intuitive human response to the above
question. This disparity between human intuition and machine explanation is
explained based on the dataset. Among the 14 traffic signs in CURE-TSR, Stop
sign is the only class that has a hexagonal shape. Hence, the network has
learned to check for a straight side in the bottom left. The absence of this
side in $x$ indicates to the network that $x$ is not a STOP sign. Hence, the
observed contrastive explanation clearly illustrates the disparity between the
notion of classes between humans and machines while observed correlations
could not. This calls for the need for considering complete explanations in
the absence of interventions. It is not very clear _if_ the two explanations
must align in all applications. However, for safety-critical applications in
medical fields, it is required that these anomalies are analyzed and
understood.

###  VI-B Failure of existing objective evaluation metrics

In Fig. 7b, we visualize Grad-CAM [12] and Causal-CAM [15] maps. The original
scan is from a COVID positive patient. Both the Grad-CAM and Causal-CAM
explanations fail to highlight the circled red region that depicts COVID.
Feeding the masked image back into $f(\cdot)$ for indirect-application
evaluation, the network classifies both correctly but with a higher confidence
for Causal-CAM. Hence, in both the Probabilistic and Accuracy-Deletion and
Insertion metrics, Causal-CAM performs better than Grad-CAM. This essentially
shows that the network has learned to classify correctly but with the wrong
features. Such a feature discrepancy is not reflected in the objective metrics
of evaluation. In real-world biomedical applications like in COVID-19
detection, it is imperative to identify and make decisions that are
interpretable. Further study is required in designing better objective metrics
and networks whose features are more human interpretable.

##  VII Conclusion

The key takeaways from this article include : ($i$) realizing explanations as
reasoning paradigms, ($ii$) definition of complete explanations in neural
networks based on reasoning ($iii$) utility of gradient-based explanatory
paradigm’s replicability across multifarious applications.

###  VII-A Explanations as Reasoning Paradigms

By having explanations answer targeted questions, the goal of explainability
shifts from justifying decisions to reviewing and participating in making
decisions. This is inline with abductive reasoning scheme that humans often
use in practice.

###  VII-B Complete Explanations

Existing techniques treat each correlation, counterfactual, and contrastive
paradigms as separate entities. However, each paradigm answers complementary
questions. A complete explanation is one that can answer any given question.
In practice, the combination of Grad-CAM [12], Counterfactual-CAM [12], and
Contrast-CAM [13] achieve this. Similar research into methodologies that unify
these paradigms and provide complete explanations is required and pertinent.

###  VII-C Applicability of gradients-as-features

From Table I, it is clear that gradients are used as features in explanations
across multiple categorizations and paradigms. The same gradients are also
used as features for recognition robustness [14], anomaly [36], novelty [37]
and out-of-distribution detection robustness [38], Image Quality Assessment
[33] and Visual Saliency Detection [34]. The adoption of a network’s gradients
as features across applications is a sign of its multifarious utility.

## References

  * [1] Alex A Freitas,  “Comprehensible classification models: a position paper,”  ACM SIGKDD explorations newsletter, vol. 15, no. 1, pp. 1–10, 2014\.
  * [2] Ryan Poplin, Avinash V Varadarajan, Katy Blumer, Yun Liu, Michael V McConnell, Greg S Corrado, Lily Peng, and Dale R Webster,  “Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning,”  Nature Biomedical Engineering, vol. 2, no. 3, pp. 158–164, 2018\.
  * [3] Samuel Carton, Jennifer Helsby, Kenneth Joseph, Ayesha Mahmud, Youngsoo Park, Joe Walsh, Crystal Cody, CPT Estella Patterson, Lauren Haynes, and Rayid Ghani,  “Identifying police officers at risk of adverse events,”  in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 67–76.
  * [4] Finale Doshi-Velez and Been Kim,  “Towards a rigorous science of interpretable machine learning,”  arXiv preprint arXiv:1702.08608, 2017.
  * [5] Muhammad A Shafiq, Mohit Prabhushankar, Haibin Di, and Ghassan AlRegib,  “Towards understanding common features between natural and seismic images,”  in SEG Technical Program Expanded Abstracts 2018, pp. 2076–2080. Society of Exploration Geophysicists, 2018.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [7] Leo Breiman, Jerome Friedman, Charles J Stone, and Richard A Olshen,  Classification and regression trees,  CRC press, 1984.
  * [8] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.,  “Language models are few-shot learners,”  arXiv preprint arXiv:2005.14165, 2020.
  * [9] Philip Kitcher and Wesley C Salmon,  Scientific explanation, vol. 13,  U of Minnesota Press, 1962.
  * [10] Judea Pearl et al.,  “Causal inference in statistics: An overview,”  Statistics surveys, vol. 3, pp. 96–146, 2009.
  * [11] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams,  “Learning representations by back-propagating errors,”  nature, vol. 323, no. 6088, pp. 533–536, 1986.
  * [12] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra,  “Grad-cam: Visual explanations from deep networks via gradient-based localization,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 618–626.
  * [13] Mohit Prabhushankar, Gukyeong Kwon, Dogancan Temel, and Ghassan AlRegib,  “Contrastive explanations in neural networks,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3289–3293.
  * [14] Mohit Prabhushankar and Ghassan AlRegib,  “Contrastive reasoning in neural networks,”  in under review at IEEE Transactions on pattern analysis and machine intelligence. IEEE, 2021, published on ArXiv.
  * [15] Mohit Prabhushankar and Ghassan AlRegib,  “Extracting causal visual features for limited label classification,”  in IEEE International Conference on Image Processing (ICIP). IEEE, 2021.
  * [16] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian,  “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,”  in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018, pp. 839–847.
  * [17] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg,  “Smoothgrad: removing noise by adding noise,”  arXiv preprint arXiv:1706.03825, 2017.
  * [18] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller,  “Striving for simplicity: The all convolutional net,”  arXiv preprint arXiv:1412.6806, 2014.
  * [19] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio,  “Toward causal representation learning,”  Proceedings of the IEEE, vol. 109, no. 5, pp. 612–634, 2021.
  * [20] David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou,  “Discovering causal signals in images,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6979–6987.
  * [21] Matthew D Zeiler and Rob Fergus,  “Visualizing and understanding convolutional networks,”  in European conference on computer vision. Springer, 2014, pp. 818–833.
  * [22] Aravindh Mahendran and Andrea Vedaldi,  “Understanding deep image representations by inverting them,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 5188–5196.
  * [23] Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Zhu,  “Interpreting cnn knowledge via an explanatory graph,”  in Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [24] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba,  “Learning deep features for discriminative localization,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2921–2929.
  * [25] Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt,  “Visual causal feature learning,”  arXiv preprint arXiv:1412.2309, 2014.
  * [26] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee,  “Counterfactual visual explanations,”  arXiv preprint arXiv:1904.07451, 2019.
  * [27] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal,  “Explaining explanations: An overview of interpretability of machine learning,”  in 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA). IEEE, 2018, pp. 80–89.
  * [28] Yazeed Alaudah, Motaz Alfarraj, and Ghassan AlRegib,  “Structure label prediction using similarity-based retrieval and weakly supervised label mappingstructure label prediction,”  Geophysics, vol. 84, no. 1, pp. V67–V79, 2019.
  * [29] Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das,  “Explanations based on the missing: Towards contrastive explanations with pertinent negatives,”  arXiv preprint arXiv:1802.07623, 2018.
  * [30] Dan Hendrycks and Thomas Dietterich,  “Benchmarking neural network robustness to common corruptions and perturbations,”  arXiv preprint arXiv:1903.12261, 2019.
  * [31] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell,  “Adapting visual category models to new domains,”  in European conference on computer vision. Springer, 2010, pp. 213–226.
  * [32] Dogancan Temel, Mohit Prabhushankar, and Ghassan AlRegib,  “Unique: Unsupervised image quality estimation,”  IEEE signal processing letters, vol. 23, no. 10, pp. 1414–1418, 2016.
  * [33] Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib,  “Distorted representation space characterization through backpropagated gradients,”  in 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019, pp. 2651–2655.
  * [34] Yutong Sun, Mohit Prabhushankar, and Ghassan AlRegib,  “Implicit saliency in deep neural networks,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 2915–2919.
  * [35] Vitali Petsiuk, Abir Das, and Kate Saenko,  “Rise: Randomized input sampling for explanation of black-box models,”  arXiv preprint arXiv:1806.07421, 2018.
  * [36] Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib,  “Backpropagated gradient representations for anomaly detection,”  in European Conference on Computer Vision. Springer, 2020, pp. 206–226.
  * [37] Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib,  “Novelty detection through model-based characterization of neural networks,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3179–3183.
  * [38] Jinsol Lee and Ghassan AlRegib,  “Gradients as a measure of uncertainty in neural networks,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 2416–2420.
  * [39] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin,  “" why should i trust you?" explaining the predictions of any classifier,”  in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144.
  * [40] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.,  “Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav),”  in International conference on machine learning. PMLR, 2018, pp. 2668–2677.
  * [41] Dogancan Temel, Gukyeong Kwon, Mohit Prabhushankar, and Ghassan AlRegib,  “Cure-tsr: Challenging unreal and real environments for traffic sign recognition,”  arXiv preprint arXiv:1712.02463, 2017.