Citation

P. Chowdhury, M. Prabhushankar, G. AlRegib and M. Deriche, ”Are Objective
Explanatory Evaluation metrics Trustworthy? An Adversarial Analysis,” in IEEE
International Conference on Image Processing (ICIP), Abu Dhabi, UAE, Oct. 27 -
30, 2024.

Review

Date of submission: Feb. 10, 2024
Date of acceptance: Jun. 06, 2024

Code

https://github.com/olivesgatech/SHAPE

Copyright

©2024 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other work.

Keywords

Explainable AI, Causal Metrics, Adversarial Attacks, Visual Causality,
Importance Maps

Contact

<EMAIL_ADDRESS>OR<EMAIL_ADDRESS>OR<EMAIL_ADDRESS>
https://ghassanalregib.info/

# Are Objective Explanatory Evaluation metrics Trustworthy? An Adversarial
Analysis

###### Abstract

Explainable AI (XAI) has revolutionized the field of deep learning by
empowering users to have more trust in neural network models. The field of XAI
allows users to probe the inner workings of these algorithms to elucidate
their decision-making processes. The rise in popularity of XAI has led to the
advent of different strategies to produce explanations, all of which only
occasionally agree. Thus several objective evaluation metrics have been
devised to decide which of these modules give the best explanation for
specific scenarios. The goal of the paper is twofold: (i) we employ the
notions of necessity and sufficiency from causal literature to come up with a
novel explanatory technique called SHifted Adversaries using Pixel
Elimination(SHAPE) which satisfies all the theoretical and mathematical
criteria of being a valid explanation, (ii) we show that SHAPE is, infact, an
adversarial explanation that fools causal metrics that are employed to measure
the robustness and reliability of popular importance based visual XAI methods.
Our analysis shows that SHAPE outperforms popular explanatory techniques like
GradCAM and GradCAM++ in these tests and is comparable to RISE, raising
questions about the sanity of these metrics and the need for human involvement
for an overall better evaluation.

Index Terms—  Explainable AI, Causal Metrics, Adversarial Attacks, Visual
Causality, Importance Maps

##  1 Introduction

Fig. 1: Even though the adversarial explanation generated by SHAPE (b) is much
less comprehensible as compared to the GradCAM explanation (a), it ourperforms
the later in both the causal metric test (c) & (d) by a significant margin
thus claiming to be the more robust and reliable among the two.

The recent success of neural networks (NN) has resulted in a notable expansion
in artificial intelligence (AI) research. Despite it’s popularity, there is
still much to learn about how a given neural network makes decisions, and when
and if it can be trusted. It’s crucial that decision-making models are
transparent in fields like medical diagnosis [1], autonomous driving [2],
geophysics [3] and others where decisions can have dire repercussions. The
field of interpretable machine learning (or Explainable Artificial
Intelligence, or XAI) focuses on post-hoc, model-independent explanation tools
to overcome this problem [4, 5, 6].

An intuitive approach to generate explanations for a NN is to visualize a
certain quantity of interest, e.g. importance of input features or learned
weights. In case of Convolution Neural Networks (CNNs) which deal with image
data, this can be represented in the form of explanation maps, highlighting
regions of pixels in a certain order of importance. Two common methods to
generate these explanatory importance maps are: gradient based (e.g. GradCAM
[7], GradCAM++[8], CausalCAM [9], ContrastCAM [10] etc.) and perturbation
based (e.g. RISE[11] and LIME [12]). Despite the potential of these XAI
strategies, the discrepancies and the disagreement among them raise concerns
about their reliability [13, 14, 15]. Hence, scholars have come up with
metrics and benchmarks to test the robustness of these XAI strategies
themselves. Recent work [16, 17] has argued that a convincing importance-
ranking explanation should always satisfy the desirable properties of
causality and answer the question: How necessary and sufficient are the
important regions? [11] has formulated this question using their two tests:
insertion game and deletion game. As the names suggest, they introduce and
remove regions of an image according to their importance proposed by the XAI
strategy to see how fast or slow the model changes the prediction probability
of the concerned class. However since most explanation strategies tend to
approximate the idea of correlation as causation in their algorithms, they
fail when subjected to the tests of these causal metrics. This raises a
follow-up question in this ongoing conversation: Are causal metrics the
desirable test to assess the robustness of XAI modules?

In our work we take the concept of adversarial attack [18] and extend them to
explanations. Adversarial attacks are originally designed to deceive or
mislead machine learning models by introducing carefully crafted perturbations
in the input data, thereby questioning the reliability of deep learning
models. In this paper, we generate adversarial explanations to test the sanity
of the causal XAI evaluation metrics. Fig.1. shows the explanatory maps as
well as the insertion and deletion performance of explanations generated by
GradCAM (Fig.1a.) and our adversarial method (SHAPE) (Fig.1b.) for the same
class prediction (Rooster). Subjectively, GradCAM is a better explanation for
why the network predicted a Rooster. Objectively, GradCAM’s explanation should
be considered _superior_ than SHAPE’s if it’s Area Under Curve (AUC) for the
insertion test is higher. Similarly, AUC must be lower in the deletion test to
be considered superior. In both cases (Fig.1c.& d.), however, our SHAPE
explanatory technique outperforms GradCAM, contrarily to our subjective
expectation. Hence, Fig. 1 reveals the flaws in objective evaluation methods
and raises concerns about how explanations are produced in general. In summary
the contributions of this paper include:

  1. 1.

Generating model-faithful adversarial explanation generations based on the
idea of necessity and sufficiency.

  2. 2.

Evaluating the validity of the current objective causal metrics used to test
the robustness and reliability of importance map based XAI modules.

##  2 Background

###  2.1 Causal Definitions of Importance: Necessity and Sufficiency

Necessity and sufficiency are concepts that have been extensively explored in
philosophy, encompassing various logical, probabilistic, and causal
interpretations. ([19]) defines necessity and sufficiency as follows:

Necessary condition: A condition A is said to be necessary for a condition B,
iff the falsity of A guarantees the falsity of B. Example:“Air is necessary
for human life.”

Sufficient condition: A condition A is said to be sufficient for a condition
B, iff the truth of A guarantees the truth of B. Example:“Being a mammal is
sufficient to have a spine.”

The author in [20] has hypothesised that these two concepts from causal
analysis help to assess the degree of importance or significance displayed by
any cause of an event. Recent work [16, 21, 22] has claimed that a models
prediction and its relation to its input space can be understood using the
same logic of cause and effect. Intervention is done by removing or
introducing features from the input space and studying how the model’s
behavior changes from the non-intervened case [23]. A necessary feature will
change a models decision when it is removed and a sufficient one will do so
when it is introduced. by formulating these two causal concepts using
probabilistic approaches we can attempt to calculate and quantify the
importance of input features for a specific models prediction. Attribute
specific XAI modules are designed to do just that.

###  2.2 Visual Explanations: Importance Maps

Importance maps are a type of visual XAI tool used in deep learning and
computer vision to determine which regions of an input image have the greatest
influence on a neural network model’s decision [24].

Methods to generate importance maps can be broadly classified into two
techniques:

  1. 1.

Gradient-based Backpropagation Method: Gradient-based backpropagation
techniques for importance maps in computer vision and image processing entail
iteratively adjusting a neural network’s weights to minimize a predetermined
loss function that measures the discrepancy between predicted importance maps
and ground truth annotations. Some examples of such methods are GradCAM [7]
GradCAM++ [8] CausalCAM [9], ContrastCAM [10] and Guided Backpropagation [25,
26].

  2. 2.

Perturbation-based Forward Propagation Method: These techniques entail
systematically modifying the input images and recording how these
modifications impact the neural network model’s output [27]. The method RISE
provides a way to create explanatory maps that emphasize the important of the
input data for the model’s decision-making process by masking different
regions of an image and examining how the remaining area affects the model’s
predictions.

###  2.3 Causal Metrics for Evaluation of Explanations

With the emergence of numerous Explainable Artificial Intelligence (XAI)
methods focused on importance maps, there’s an increased demand for objective
evaluation strategies that determine which properties yield accurate and
reliable explanations. Motivated by the work of [28], the authors in [11]
formulate the definitions of necessity and sufficiency to establish two causal
metrics for explanation evaluation: deletion and insertion.

  1. 1.

Deletion: The deletion metric quantifies the definition of necessity which is
based on the logical assumption that eliminating the ”cause” will compel the
base model to change its decision. In particular, when increasing numbers of
significant pixels are eliminated—the significance being derived from the
explanation map—this metric quantifies a decline in the likelihood of the
anticipated class. A good explanation is indicated by a steep drop and
consequently a low area under the probability curve or AUC (as a function of
the fraction of eliminated pixels). Based on this conclusion, SHAPE in Fig.1
is a better strategy for class explaining the class rooster because it has a
smaller AUC (a 76.66% decrease) than GradCAM.

  2. 2.

Insertion: In contrast, the insertion metric employs a complimentary
methodology in order to calculate the sufficiency of the explanation. As more
and more pixels are added, it calculates the likelihood increase, with a
broader curve and higher AUC suggesting a more compelling explanation. The
insertion AUC of SHAPE for Fig.1 is almost 50% greater than GradCAM
establishing it as a better explanation according to both the causal metrics.

###  2.4 Adversarial Explanations: A New Robustness Test

In deep learning, especially computer vision, an adversarial attack is defined
as the purposeful manipulation of the input image in order to trick the
machine learning model to produce false decisions [18, 29]. These adversarial
images have barely any imperceptible changes in them while making wrong
predictions thereby questioning the softmax probability metric. We have
adopted a converse approach to create our adversarial explanations.
Objectively, our explanations employ the idea of necessary features to produce
the importance maps and have better insertion and deletion game scores than
other explanatory maps present in current literature. However subjectively,
these explanations are not human interpretable and thus don’t contribute to
increase trust between the user and the neural network- a condition any
dependable XAI module needs to satisfy. Hence these adversarial explanations
take a reverse approach to raise questions about the robustness of these
causal metrics being suitable evaluation tests for importance map based XAI
approaches.

##  3 Methodology

In this work we propose to leverage the two notions of causality, necessity
and sufficiency, to come up with a framework to generate adversarial
explanations. Our method is simultaneously faithful to the model itself and
mathematically sound when generating visual explanations. However, our
explanations are adversarial - they are almost entirely incomprehensible by
humans (Fig. 1 & 3) thus voiding the whole idea of an explanation helping its
user understand the inner workings of the model while increasing trust in its
decision-making process.

Our algorithm is an inverse approach to the popular perturbation based method,
RISE [11]. RISE explanations quantify the sufficiency of a region in the image
by masking out all other regions and allowing the model to make predictions on
the area of interest only. Our method creates explanations by quantifying the
shift in model response when a specific portion of pixels are eliminated. In
other words, our explanations quantify the necessity of a region of pixels.

###  3.1 SHifted Adversaries using Pixel Elimination (SHAPE)

Fig. 2: Overview of SHAPE to generate adversarial explanations: Input image
$I$ is element-wise multiplied by the random masks $M_{i}$ and are fed into
the model $f$ along with the original image to calculate the change in
prediction scores.The importance map is a weighted sum of masks where the
weights for each mask is its corresponding change in probability scores.

Based on the random masking method introduced in [11], we use the definition
of necessary cause to define the importance of pixels $\lambda$ as the
expected change in prediction probability from the original image $(I)$ over
all masks $M$ on the condition that the pixels are not observed, i.e.,
$M(\lambda)=0$:

$N_{I,f}(\lambda)=\mathbb{E}_{M}[f(I)\sim f(I\odot M)\mid M(\lambda)=0]$ (1)

where $I\odot M$ refers to the image with the mask applied.

For a summation over all masks $m$, Eq. 1 is:

$\displaystyle N_{I,f}(\lambda)$ (2) $\displaystyle=\sum_{m}[f(I)\sim f(I\odot
m)]P[M=m\mid M(\lambda)=0]$

which can be written as:

$\displaystyle\frac{1}{P[M(\lambda)=0]}\sum_{m}[f(I)\sim f(I\odot
m)]P[M=m,M(\lambda)=0]$ (3)

Here,

$\displaystyle P[M=m,M(\lambda)=0]$
$\displaystyle=\begin{cases}P[M=m],&\text{if }m(\lambda)=0\\\ 0,&\text{if
}m(\lambda)=1\end{cases}$ $\displaystyle=(1-m(\lambda))P[M=m]$ (4)

Substituting $P[M=m,M(\lambda)=0]$ from (3.1) in (3) we get:

$N_{I,f}(\lambda)=\frac{1}{P[M(\lambda)=0]}\sum_{m}[f(I)\sim f(I\odot
m)]\cdot(1-m(\lambda))\cdot P[M=m]$ (5)

We then use Monte Carlo’s approximation to reduce the summation to a weighted
average of the mass samples $M(\lambda)$:

$\small
N_{I,f}(\lambda)\stackrel{{\scriptstyle\mathrm{MC}}}{{\approx}}\frac{1}{P[M(\lambda)=0]\cdot
N}\sum_{i=1}^{N}[f(I)\sim f(I\odot M_{i})]\cdot(1-M_{i}(\lambda)).$ (6)

The final $N_{I,f}(\lambda)$ is an approximation of region-wise necessity
score for a particular class prediction for an image $I$ and using model $f$
(See Fig 2).

##  4 Experiments and Observations

###  4.1 Experimental Setup for Adversarial Explanation Generation

Similar to [11] we generate the masks for pixel deletion. Smaller binary masks
are randomly upsampled using bilinear interpolation to generate masks of
larger resolution to ensure smooth importance map generation. The dimensions
of binary masks have been kept at a constant size of $7X7$ pixels with pixel
restoration ($\lambda=1$) probability being $0.1$. Using this binary mask we
generate $6000$ large masks to be used in all our experiments. We generate
adversarial explanations for images inferred using a ResNet50, RestNet101, and
VGG16 model, all pretrained on the ImageNet dataset.

Fig. 3: SHAPE Maps for image (a) for a RestNet101 model. (b) shows the
importance map for class Bull mastiff (prediction accuracy = $38.42\%$), (c)
shows the importance map for class Tiger cat (prediction accuracy = $9.41\%$)
and (d) shows the map for class Tabby (prediction accuracy = $04.97\%$). Fig.
4: CAM for prediction “great white shark” using (a) GradCAM, (b) GradCAM++,
(c) RISE and d) SHAPE(ours). SHAPE significantly outperforms all other methods
in both (e) deletion (lower AUC is better) and (f) deletion (higher AUC is
better) game

Fig.3 shows some of the adversarial explanations generated using SHAPE scores
for different class predictions. Subjectively, only the SHAPE map for boxer in
(c) seems to be the most human understandable explanation, while the one for
bull mastiff as well as doormat are almost indistinguishable and simply
represented by splotches of importance region spread throughout the image.

Subjectively, most of the visual explanations generated using SHAPE are not
human interpretable (See Fig 4 for another example). However they are sound on
the basis of the fact that this map shows what region a model considers
important or necessary during generation of it’s prediction for a certain
class. Thus this method satisfies the definition of an activation based visual
explanation strategy: it should provide insights into which regions of an
input image are important for a neural network’s decision-making process.
Hence for all experiments and mathematical analysis SHAPE can be used as a
valid explanation.

###  4.2 Faithfulness Test of Causal Evaluation Metrics

We notice SHAPE, our adversarial methods performs better on both the deletion
and insertion game for the particular example in Fig.4 than all the other
three explanation methods (GradCAM, GradCAM++ and RISE). It shows almost a 50%
better AUC improvement in both metrics when compared to the second best
explanation strategy, RISE. Thus according to the causal metric evaluation
SHAPE explantions are the best and most faithful to explain this model’s
decision making process. However human inspection of the explanation map
proves otherwise. This holds true for several other test images from the
ImageNet dataset as well, with severely misconstrued importance maps
displaying the best performances in both the causal tests (also Fig.1).

Method | ResNet50 | ResNet101 | VGG16
---|---|---|---
Insertion | Deletion | Insertion | Deletion | Insertion | Deletion
GradCAM | 0.684 | 0.223 | 0.6871 | 0.174 | 0.614 | 0.118
GradCAM++ | 0.712 | 0.209 | 0.701 | 0.166 | 0.642 | 0.134
RISE | 0.769 | 0.091 | 0.788 | 0.150 | 0.755 | 0.099
ShAPE(Ours) | 0.771 | 0.104 | 0.844 | 0.088 | 0.731 | 0.101

Table 1: Mean insertion (higher is better) and deletion (lower is better)
evaluation scores for GradCAM, GradCAM++, RISE and SHAP for models: ResNet50,
RestNet101 and VGG16 for same number of test images randomly sampled from
ImageNET DET test set.

Table 1 shows that for samples queried randomly from ImageNet DET test the
mean deletion and insertion scores for SHAPE makes it a much more trustworthy
explanation than GradCAM and GradCAM++ for all three (ResNet50, ResNet101 and
VGG16) models and even comparable to RISE.

###  4.3 Discussion

Although SHAPE doesn’t overload the explanation map with high importance
scores like GradCAM or GradCAM++ does in Fig.4 a.& b., it fools both the
insertion and deletion metrics despite highlighting important pixel regions
outside the area of interest (where the object is actually located) for the
particular class prediction. This can be attributed to the fact that the
_important_ regions shown by these XAI modules aren’t always the same on a
causal scale. This is because causal features cannot be extracted exclusively
from pixels [9]. Unlike in structured data where causal relationships might be
more discernible, images present a complex web of interconnected pixels,
making it difficult to isolate and manipulate individual causal factors. While
neural networks excel at learning complex patterns and making predictions
based on correlations within the data, they lack the inherent capacity to
infer causality without explicit guidance. While predictive features may
suffice for accurate predictions in normal circumstances, causal features are
indispensable for maintaining accuracy even in the face of corruptions.
Consequently even minor alterations can trigger substantial shifts in the
representation. Hence, causal metrics like insertion and deletion can fail to
hold as baselines for sanity checks of these explanations, which are but a
pixel level approximation of what causal features might represent.

##  5 Conclusion

In this paper we propose a novel method to create adversarial explanations by
formulating the definition of necessary features, as introduced by causality,
and quantifying the changes in model prediction. We use this adversarial
strategy to study the rationality behind using casual evaluation metrics like
insertion and deletion of pixels as a valid technique for comparing the
robustness of importance based visual explanation algorithms. Our adversarial
technique outperforms existing visual explanations in these evaluation
metrics. Hence, our work motivates the need for devising objective explanatory
evaluation techniques that matches subjective human opinions. Until such
metrics are devised, human intervention and assessment is necessary for proper
evaluation of visual explanations.

## References

  * [1] Mohit Prabhushankar, Kiran Kokilepersaud, Yash-yee Logan, Stephanie Trejo Corona, Ghassan AlRegib, and Charles Wykoff,  “Olives dataset: Ophthalmic labels for investigating visual eye semantics,”  Advances in Neural Information Processing Systems, vol. 35, pp. 9201–9216, 2022.
  * [2] Kiran Kokilepersaud, Yash-Yee Logan, Ryan Benkert, Chen Zhou, Mohit Prabhushankar, Ghassan AlRegib, Enrique Corona, Kunjan Singh, and Mostafa Parchami,  “Focal: A cost-aware video dataset for active learning,”  in 2023 IEEE International Conference on Big Data (BigData). IEEE, 2023, pp. 1269–1278.
  * [3] Prithwijit Chowdhury, Ahmad Mustafa, Mohit Prabhushankar, and Ghassan AlRegib,  “Counterfactual uncertainty for high dimensional tabular dataset,”  in SEG International Exposition and Annual Meeting. SEG, 2023, pp. SEG–2023.
  * [4] Ghassan AlRegib and Mohit Prabhushankar,  “Explanatory paradigms in neural networks: Towards relevant and contextual explanations,”  IEEE Signal Processing Magazine, vol. 39, no. 4, pp. 59–72, 2022.
  * [5] Mohit Prabhushankar and Ghassan AlRegib,  “Introspective learning: A two-stage approach for inference in neural networks,”  Advances in Neural Information Processing Systems, vol. 35, pp. 12126–12140, 2022.
  * [6] Mohit Prabhushankar, Gukyeong Kwon, Dogancan Temel, and Ghassan AlRegib,  “Contrastive explanations in neural networks,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3289–3293.
  * [7] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra,  “Grad-cam: Visual explanations from deep networks via gradient-based localization,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 618–626.
  * [8] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian,  “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,”  in 2018 IEEE winter conference on applications of computer vision (WACV). IEEE, 2018, pp. 839–847.
  * [9] Mohit Prabhushankar and Ghassan AlRegib,  “Extracting causal visual features for limited label classification,”  in 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, pp. 3697–3701.
  * [10] Mohit Prabhushankar, Gukyeong Kwon, Dogancan Temel, and Ghassan AlRegib,  “Contrastive explanations in neural networks,”  CoRR, vol. abs/2008.00178, 2020.
  * [11] Vitali Petsiuk, Abir Das, and Kate Saenko,  “Rise: Randomized input sampling for explanation of black-box models,”  arXiv preprint arXiv:1806.07421, 2018.
  * [12] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin,  “” why should i trust you?” explaining the predictions of any classifier,”  in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144.
  * [13] W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu,  “Definitions, methods, and applications in interpretable machine learning,”  Proceedings of the National Academy of Sciences, vol. 116, no. 44, pp. 22071–22080, 2019.
  * [14] Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju,  “The disagreement problem in explainable machine learning: A practitioner’s perspective,”  arXiv preprint arXiv:2202.01602, 2022.
  * [15] Mohit Prabhushankar and Ghassan AlRegib,  “Voice: Variance of induced contrastive explanations to quantify uncertainty in neural network interpretability,”  arXiv preprint arXiv:2406.00573, 2024.
  * [16] Ramaravind Kommiya Mothilal, Divyat Mahajan, Chenhao Tan, and Amit Sharma,  “Towards unifying feature attribution and counterfactual explanations: Different means to the same end,”  in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021, pp. 652–663.
  * [17] Yanou Ramon, David Martens, Foster Provost, and Theodoros Evgeniou,  “A comparison of instance-level counterfactual explanation algorithms for behavioral and textual data: Sedc, lime-c and shap-c,”  Advances in Data Analysis and Classification, vol. 14, pp. 801–819, 2020.
  * [18] Naveed Akhtar and Ajmal Mian,  “Threat of adversarial attacks on deep learning in computer vision: A survey,”  Ieee Access, vol. 6, pp. 14410–14430, 2018.
  * [19] Norman Swartz,  “The concepts of necessary conditions and sufficient conditions,”  Department of Philosophy Simon Fraser University, 1997.
  * [20] Judea Pearl,  Causality,  Cambridge university press, 2009.
  * [21] David S Watson, Limor Gultchin, Ankur Taly, and Luciano Floridi,  “Local explanations via necessity and sufficiency: Unifying theory and practice,”  in Uncertainty in Artificial Intelligence. PMLR, 2021, pp. 1382–1392.
  * [22] Sainyam Galhotra, Romila Pradhan, and Babak Salimi,  “Explaining black-box algorithms using probabilistic contrastive counterfactuals,”  in Proceedings of the 2021 International Conference on Management of Data, 2021, pp. 577–590.
  * [23] Prithwijit Chowdhury, Mohit Prabhushankar, and Ghassan AlRegib,  “Explaining explainers: Necessity and sufficiency in tabular data,”  in NeurIPS 2023 Second Table Representation Learning Workshop, 2023.
  * [24] Inam Ullah, Muwei Jian, Sumaira Hussain, Jie Guo, Hui Yu, Xing Wang, and Yilong Yin,  “A brief survey of visual saliency detection,”  Multimedia Tools and Applications, vol. 79, pp. 34605–34645, 2020.
  * [25] Weili Nie, Yang Zhang, and Ankit Patel,  “A theoretical explanation for perplexing behaviors of backpropagation-based visualizations,”  in International conference on machine learning. PMLR, 2018, pp. 3809–3818.
  * [26] Jinsol Lee, Charlie Lehman, Mohit Prabhushankar, and Ghassan AlRegib,  “Probing the purview of neural networks via gradient analysis,”  IEEE Access, vol. 11, pp. 32716–32732, 2023.
  * [27] Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols,  “Perturbation-based methods for explaining deep neural networks: A survey,”  Pattern Recognition Letters, vol. 150, pp. 228–234, 2021.
  * [28] Ruth C Fong and Andrea Vedaldi,  “Interpretable explanations of black boxes by meaningful perturbation,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 3429–3437.
  * [29] Pedro Tabacof and Eduardo Valle,  “Exploring the space of adversarial images,”  in 2016 international joint conference on neural networks (IJCNN). IEEE, 2016, pp. 426–433.


# Mutual information estimation for graph convolutional neural networks

Marius C. Landverk<EMAIL_ADDRESS>Signe Riemer-Sørensen
<EMAIL_ADDRESS>Analytics and AI, SINTEF Digital, P.O. Box
124 Blindern, N-0314 Oslo, Norway

###### Abstract

Measuring model performance is a key issue for deep learning practitioners.
However, we often lack the ability to explain why a specific architecture
attains superior predictive accuracy for a given data set. Often, validation
accuracy is used as a performance heuristic quantifying how well a network
generalizes to unseen data, but it does not capture anything about the
information flow in the model.

Mutual information can be used as a measure of the quality of internal
representations in deep learning models, and the information plane may provide
insights into whether the model exploits the available information in the
data.

The information plane has previously been explored for fully connected neural
networks and convolutional architectures. We present an architecture-agnostic
method for tracking a network’s internal representations during training,
which are then used to create the mutual information plane. The method is
exemplified for graph-based neural networks fitted on citation data. We
compare how the inductive bias introduced in graph-based architectures changes
the mutual information plane relative to a fully connected neural network.

##  1 Introduction

###  1.1 Motivation

For classification problems, the validation accuracy is a common heuristic to
gauge the generalization capabilities of a model. Whilst it is a useful
metric, it leaves something to be desired in terms of understanding why a
model is able to perform as well, or as ill, as it does. The ability of a
model to fit the data is directly related to the quality of the
representations generated by the model. The information plane is given as the
mutual information between a model’s representations, $Z_{i}$, with the input
$X$ and the true labels $Y$ [11, 16, 14]. Visual inspection of the information
plane at the point in training where the performance plateaus can provide
qualitative guidance. In this paper, we seek to use the information plane from
the information bottleneck method [14] in order to gauge model fit across
architectures with different inductive biases, and we exemplify by comparing a
fully-connected neural network to graph-based networks trained on the same
classification task.

###  1.2 Mutual information

The core of the information bottleneck method is a quantification of shared
information between random variables originally suggested by [11]. For two
random variables $X$ and $Y$, the mutual information $I(X,Y)$ is a symmetric
quantity measuring the amount of information that $X$ contains about $Y$, and
vice versa. If we want to predict the true labels $Y$ from $X$, we can define
a compressed version of $X$, called $Z$, with shared information $I(X,Z)$ and
$I(Z,Y)$. The minimal representation $Z$ (maximally compressed information on
$Y$ from $X$), that simultaneously maximises the mutual information with $Y$
is called the ’information bottleneck’ [14, 10]:

$\mathrm{argmax}_{Z\in\Delta}I(Y,Z)\;\mathrm{such\,that}\;I(X,Z)\leq R,$ (1)

where $R$ is a given scalar threshold. Mutual information can be expressed in
terms of the entropy $H(X),\;H(Y)$, as $I(X,Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$, where
$H(Y|X)$ is the conditional entropy of $Y$ given $X$ [3].

If $Y,X$ and $Z$ are random variables in a Markov chain where $Y\rightarrow
X\rightarrow Z$, the data-processing inequality applies to the mutual
information between the variables [3]:

$I(X,Y)\geq I(Z,Y).$ (2)

In other words, the information about an input $X$ contained in $Z$ cannot be
increased by applying a function to $Z$, as long as the function does not use
additional information about $X$.

For neural networks, the representations $Z_{i}$ can be associated with the
successive hidden layers. For fully connected neural networks, the data
processing inequality can then be applied to obtain a set of inequalities.
However, in more complicated neural network structures such as recurrent
neural networks or graph convolutional neural networks, the hidden layers $Z$
draws on information about $X$, and the data processing inequality is broken.
In the case of graph convolutional neural network, the data processing
inequality is broken explicitly because all hidden layers uses the edge
information in $X$ to compute the forward pass.

###  1.3 Related work

It has been debated whether the mutual information follows a specific pattern
during training with a fitting phase followed by a compression phase [14, 13].
So far, the compression phase is only observed for symmetric activation
functions such as sigmoid and tanh (see [5] for an overview of experiments).
Here we do not make any claims regarding the compression phase, but provide a
framework for comparing the mutual information across neural network
architectures.

Early works [16, 14] use a binning procedure to estimate the mutual
information. This is only meaningful for activation functions with restricted
output range e.g. sigmoid, and the choice of binning has been proven to affect
the resulting mutual information estimate significantly [15]. This can be
alleviated by assuming a distribution for the hidden variables $Z$ and
rewriting the mutual information to only include the hidden layer
representations. In Sec. 2 we focus on this method which allows to derive both
upper and lower bounds for $I(X,Z)$ and $I(Z,Y)$ [9, 10, 13], and only briefly
comment here on alternative methods. Instead of assuming the distributions,
the approach of [2] relies on a completely separate neural network to estimate
the mutual information of the problem-specific neural network. However, this
introduces additional hyper parameters and high risk of numerical
instabilities.

The method of [20] uses an estimate for the entropy based on an eigenvalue
decomposition of $X$ and a smoothing with an infinitely-divisible, real-valued
positive definite kernel. However, since the method relies on computing
eigenvalues, the computational requirements fast become prohibitive, as this
computation is done several times in order to obtain a single mutual
information plot.

The authors of [4] apply the mutual information plane as a way to explain
convolutional neural networks trained for image classification (e.g. ResNET),
but they only consider that specific architecture and image-like data.

###  1.4 Our contribution

We present a framework for tracking the activations of a generic neural
network. To produce an information plane, a classification task is required,
since otherwise the conditional entropy $H(X|Y)$ becomes intractable. The
framework has been developed in PyTorch [12] and can be found on
GitHub111https://github.com/mariusmcl/info-bottleneck-tracking.

The novelty of the developed framework is that it is agnostic to the kind of
model architecture, as long as all the submodules to be tracked are defined
explicitly in the initialization method of the PyTorch module. The original
methods for mutual information estimations were developed on fully connected
neural networks. To our knowledge, mutual information has not previously been
estimated for graph neural networks or recurrent networks (see e.g. table 1 in
[5]). The already existing codebases from [11, 13] are well suited for the
specifically investigated network architectures and setup, but they lack the
flexibility to be easily applied on different neural architectures, and hence
do not invite comparisons across architectures.

In Sec. 3, we exemplify on graph-like data and discuss how the inductive bias
introduced by the imposed structure of the graph neural network affects the
model quality and how that is expressed in the mutual information plane.

##  2 Method

We estimate the mutual information based on the method from Kolchinsky et. al.
[9, 10]. Given a batch $B$ with $N_{B}$ samples, the upper bound
estimates222The lower bound is obtained similarly by replacing the 2 in
$K_{ij}$ with 8. for the mutual information $I(X,Z_{B})$ and $I(Z_{B},Y)$ are

$I(X,Z_{B})\leq-\frac{1}{N_{B}}\sum_{i}\log\frac{1}{N_{B}}\sum_{j}K_{ij}$ (3)

with $K_{ij}=\exp\left(-\|Z_{i}-Z_{j}\|_{2}^{2}/(2\sigma^{2})\right)$, and

$\displaystyle I(Z_{B},Y)$
$\displaystyle\leq-\frac{1}{N_{B}}\sum_{i}\log\frac{1}{N_{B}}\sum_{j}K_{ij}$
(4)
$\displaystyle-\sum_{c=1}^{C}p_{c}\left[-\frac{1}{N_{c}}\sum_{\\{i|Y_{i}=c\\}}\log\frac{1}{N_{c}}\sum_{\\{j|Y_{j}=c\\}}K_{ij}\right],$

with $p_{c}=\frac{N_{c}}{N}$ where $N_{c}$ is the total number of examples in
category $c$ and $N$ is the total number of samples.

The estimate is based on pairwise distances between $Z_{i}$ and $Z_{j}$. If
the components in the representation of a class (some/all classes) are very
clustered, their pairwise distances are small and hence the second term in
equation (4) becomes small, leading to large mutual information. If the
representation is e.g. random and thereby not clustered at all, the mutual
information becomes large. Hence, if the inter-class distances are small, one
can relate a sub-region of the representation’s domain to a class label, which
in turn would yield a higher estimate of mutual information.

The only hyperparameter in the method is the homoscedastic noise variance
$\sigma^{2}$ of the assumed normal distribution. In equations (3) and (4),
$\sigma^{2}$ can be interpreted as a bandwidth term, with a higher
$\sigma^{2}$ allowing more interactions between batch elements $i$ and $j$,
leading to higher values of $K_{ij}$. Different choices of $\sigma^{2}$
produce quantitatively and qualitatively different results. If the variance is
too small, the estimates become noisy (large scatter in the plots), and if it
becomes too big the estimates have large uncertainty, which also tends to
break the data processing inequality even for fully connected neural networks.
This is consistent with previous observations based on independent estimators
[5, 18, 19].

The developed framework is designed to be compatible with the PyTorch training
workflow and relies on recording the activation values of the model during
training by attaching a ”hook” on the forward pass333Pytorch: Forward And
Backward Function Hooks,
https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html##forward-
and-backward-function-hooks. We show an example training loop with activation
tracking during training in code listing 1. The forward hooks are defined
through the model’s named_children attribute444Pytorch: Modules
https://pytorch.org/docs/stable/generated/torch.nn.Module.html.

⬇

for epoch in range(num_epochs):

hooks = list(tracker.forward_hooks.keys())

tracker.register_new_epoch(hooks)

for x, y in train_loader:

optimizer.zero_grad()

out = classifier(x)

loss = crossentropy_loss(out, y)

loss.backward()

optimizer.step()

tracker.save()

Figure 1: An example of activation tracking during a training loop in PyTorch
for training a network for a classification task. The forward hooks are
defined through the model’s $\text{named}\\_\text{children}$ attribute.

We have verified that the method reproduces the mutual information estimates
of [14] and [4] (as far as possible with the details provided in the paper).

##  3 Example on graph-like data

We exemplify the method on two different graph-like datasets (Sec. 3.1, Sec.
3.2) and and three different architectures (Sec. 3.3-Sec. 3.5).

###  3.1 The Cora data

The Cora data555https://relational.fit.cvut.cz/dataset/CORA has a graph like
structure with 2708 nodes representing scientific papers and 5429 edges given
by citations between papers. The edges are unidirectional so paper A can cite
paper B or opposite, or they can both cite each other. The papers are
described by a feature vector of length $D=1433$ where each element is the
number count of a predetermined word (bag-of-words feature vector). In
addition, each of the papers belong to one of seven categories indicating
which scientific field the article is published in. For the Cora data, we
consider the task of classifying the papers into one of seven publication
categories. We use the built in masks for splitting in training (140 nodes),
validation (500 nodes) and test (1000 nodes) samples.

###  3.2 The arxiv data

The arxiv dataset666https://ogb.stanford.edu/docs/nodeprop/#ogbn-arxiv
contains the citation network between preprint papers in computer science
submitted to arXiv. The nodes represents 169,343 preprints connected by
1,166,243 directed edges representing one preprint citing another. The papers
are described by a feature vector of length $D=128$ where each element is the
number count of a predetermined word (bag-of-words feature vector). The
targets are one of 40 sub-categories assigned to each paper (e.g. cs.AI, cs.LG
etc. which are manually assigned by the authors and moderators), and hence the
task is classification. We follow the recommendation and split into training
and test nodes based on submission dates such that we train on papers
published until 2017, validate on those published in 2018, and test on those
published since 2019. This is known to be slightly more challenging than the
random split used in the Cora data [7].

###  3.3 Multilayer perceptron model

The simplest model is a vanilla multilayer perceptron (MLP) model [6]. The
propagation rule for the MLP is given by

$Z_{l+1}=\sigma\bigl{(}Z_{l}W_{l+1}\bigr{)},$ (5)

where $W$ is the weight matrix, and $\sigma$ indicates the activation
function. The MLP was chosen as reference to explore if the mutual information
plane is affected by inductive bias in the model.

###  3.4 Graph convolution model

Graph convolutions are a generalisation of the convolutionary operator that
have become a standard workhorse in deep learning on images, to operate on
arbitrary graphs [8, 1]. The propagation rule for a graph convolutional
network is achieved through a weighted sum of neighboring nodes’ features,
followed by a matrix multiplication. The propagation rule for a node $n$ is
given by

$Z_{l+1}^{n}=\sigma\left(\left(\sum_{j\in\mathcal{N}(n)}\frac{1}{\sqrt{\hat{d}_{j}\cdot\hat{d}_{n}}}Z_{l}^{j}\right)W_{l+1}\right),$
(6)

with $\hat{d}_{i}$ being defined as
$\hat{d}_{i}=1+\sum_{j\in\mathcal{N}(i)}e_{ij}$ with $e_{ij}$ being the edge
weight between nodes $i$ and $j$, with the default being $e_{ij}=1$. The graph
convolutional architecture thus has a fixed weighing of a nodes’ attribute
based on that nodes’ edge degree.

###  3.5 Graph attention model

Graph attention networks follows the same propagation rule as in equation (6)
but apply an attention mechanism at the point when the information is
aggregated from the neighbouring nodes through normalised attention scores
assigned to each source node before the summation [17]. For a single node $n$
the propagation can be written as:

$Z_{l+1}^{n}=\sigma\left(\left(\sum_{j\in\mathcal{N}(n)}\alpha^{j,n}Z_{l}^{j}\right)W_{l+1}\right)\,,$
(7)

where the attention scores $\alpha^{j,n}$ are trainable and thus not fixed as
for the graph convolutional network.

###  3.6 Training the models

All architectures are fitted using three hidden layers, with 300, 200 and 100
neurons each. For the Cora data we use relu activations, while for the arxiv
data we fit two models for each architecture using tanh and relu activations
respectively. We train for 600 epochs for the arxiv dataset, and for 100
epochs for the Cora dataset. Due to the difficulties of batching a graph
dataset, all models are trained with full-batch gradient descent, and as such
each epoch corresponds to one update of the model parameters.

We estimate the upper bound on the mutual information using Listing 1 with a
noise parameter of $\sigma^{2}=0.1$ for the Cora dataset and
$\sigma^{2}=0.001$ for the arxiv dataset. We have tested multiple values of
$\sigma$. Generally, we find that small values ($\sigma=10^{-5}$) lead to very
scattered values of mutual information, while large values ($\sigma=0.1$)
reduce the scatter at the cost of increased uncertainty, which may lead to
apparent violation of the data processing inequality for the MLP model on some
datasets. We selected the values of $\sigma$ such that the data processing
inequality was not (strongly) violated for the MLP model (see also Sec. 2).

###  3.7 Results on Cora data

Fig. 2 shows the training and validation accuracy during training. We see that
both models obtain a training accuracy of almost 100%, but the validation
accuracy of the MLP model remains low, while the GCN structure enforces
specific relationships in the model that appears to prevent some overfitting
and consequently higher validation accuracy.

Figure 2: The training (solid lines) and validation accuracies (dashed lines)
during the fitting process. To the left is the Cora data with the MLP model in
blue and GCN model in black. The to right is the arxiv data with the GAT model
is blue, MLP model is orange and GCN model is green.

Fig. 3 shows the information planes for the MLP and GCN models fitted to the
Cora data. For the MLP model we see that the mutual information between input
and the individual layers ($I(X,Z_{i})$) and the individual layers and output
($I(Z_{i},Y)$) increase with training, also referred to as the fitting phase.
As expected, we do not observe any compression phase due to the choice of the
asymmetric relu activation function. For the MLP, the layers follow the data
processing inequality (equation (2)) with $I(X,Z_{1})\geq I(X,Z_{2})\geq
I(X,Z_{3})$ and $I(Z_{1},Y)\geq I(Z_{2},Y)\geq I(Z_{3},Y)$. We note that even
though the MLP model has less information about $X$ in its final layer
compared to the GCN model, it is not able to use this in order to create
generalized features which perform better on the validation set. Seen together
with the gap between training and validation accuracy in Fig. 2 this is a
clear sign that the MLP architecture is unable to generalize well on this
dataset.

Figure 3: The information plane of the training process for MLP model (left)
and GCN model (right) on the Cora citation dataset, with noise parameter
$\sigma^{2}=0.1$ and relu activations. The different symbols/numbers refer to
layers in the networks ($\bigcirc$:1, $\triangle$:2, $\square$:3) whereas the
colours refer to fitting epoch. The inserts provide a zoom-in of the
convergence region. For the MLP the third layer (squares) ends up with smaller
mutual information with X than the previous two layers (triangles and circles)
adhering to the data processing inequality. Through its pooling operation, the
GCN architecture is able to retain more information about $X$ than the MLP
model, which is shown by the final layer (square) having noticeably higher
mutual information about $X$ compared to its fully connected counterpart.

For the GCN model we also observe that the mutual information between input
and the individual layers ($I(X,Z_{i})$) and the individual layers and output
($I(Z_{i},Y)$) increase with training. By choosing $\sigma^{2}=0.1$ the data
processing inequality is not violated, however for lower values of
$\sigma^{2}$ the data processing inequality was violated for the GCN model. It
is not always expected that the GCN architecture fulfills the data processing
inequality. This is due to the successive representations
$Z_{i}^{1},Z_{i}^{2}$ drawing upon knowledge of the input data $X$ to generate
the representations. Specifically for the GCN, they are obtained through an
averaging of previous features from neighbouring nodes. This introduces an
explicit dependence upon the structure of $X$ in the update step in equation
(6), violating the data processing inequality.

It is somewhat surprising that the GCN model retains more information about
$X$ than the MLP model while at the same time performing better on the
validation set. Usually one would seek to generalize away as much unnecessary
variation in $X$ as possible in order to only retain variation related to
predicting $Y$. It seems the MLP architecture is bottlenecked by its ability
to only convey information through matrix multiplications, whilst the GCN
model alleviates this by also pooling each neighboring nodes’ features in its
propagation step.

###  3.8 Results on arxiv data

Fig. 2 (right) shows the training and validation accuracy during training
using relu activations on the arxiv data. On the arxiv data, the models obtain
training accuracies of 64-77%, and validation accuracies of 54% for the MLP
model and 58% for the two graph models777These are slightly worse than the
official best fit models obtained on the data, but we have not tuned the
individual models [7] in order to keep them comparable. The gap between
training and validation accuracies indicate some overfitting, which is worst
for the MLP. We assume this is due to the graph-based models enforcing
specific relationships that helps with the generalization. The accuracies have
very similar behaviour for the tanh activations (not shown).

Fig. 4 shows the information planes for all three models fitted to the arxiv
data with relu activations. For all models we observe a rapid fitting phase
with increasing mutual information along both axes. For the MLP model we see
that all the layers ends up within a very small region and while the inset
indicates that the ordering breaks the data processing inequality, this is
most likely due to uncertainty in the estimation. We associate the closeness
of the layers to a lack of compression during training and a poor
generalization performance. The GCN and GAT show similar patterns, but with
fewer iterations needed for the rapid increase of mutual information. The GAT
performs a small compression of the last layer. In both cases, the closeness
of the layers indicate low compression between the layers.

Figure 4: The information plane of the training process for MLP model (top)
and GCN model (middle) and GAT model (bottom) on the arxiv citation dataset,
with noise parameter $\sigma^{2}=0.001$ and relu activation functions. The
different symbols/numbers refer to layers in the networks ($\bigcirc$:1,
$\triangle$:2, $\square$:3) whereas the colours refer to fitting epoch. The
inserts provide a zoom-in of the convergence region.

At the current stage, we are unable to draw general conclusions from the
mutual information planes, but the presented framework will ease further study
of the mutual information plane of inductively biased models.

##  4 Conclusion

We provide a framework for performing the information plane analysis by
tracking the activations of a general PyTorch model. The work was partly
motivated by the need for evaluating generalisation performance based on
training data alone, and partly by the need for informed model architecture
selection. Being able to compare the mutual information plane across neural
architectures is reliant on whether the architectures violate the data
processing inequality. Ideally the data processing inequality would hold, but
for more complicated architectures, and as shown in this work, that is not
necessarily the case but the information plane can still provide insight in
some cases.

## References

  * Battaglia et al. [2018] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, and others.  Relational inductive biases, deep learning, and graph networks.  _arXiv e-prints_ , 1806.01261:arXiv:1806.01261, June 2018\.
  * Belghazi et al. [2018] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm.  Mutual information neural estimation.  In J. Dy and A. Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 531–540. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/belghazi18a.html.
  * Cover and Thomas [2006] T. M. Cover and J. A. Thomas.  _Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing)_.  Wiley-Interscience, USA, 2006.  ISBN 0471241954.
  * Fang et al. [2018] H. Fang, V. Wang, and M. Yamaguchi.  Dissecting deep learning networks—visualizing mutual information.  _Entropy_ , 20(11), 2018.  ISSN 1099-4300.  10.3390/e20110823.  URL https://www.mdpi.com/1099-4300/20/11/823.
  * Geiger [2020] B. C. Geiger.  On Information Plane Analyses of Neural Network Classifiers – A Review.  _arXiv e-prints_ , 2003.09671:arXiv:2003.09671, Mar. 2020\.
  * Goodfellow et al. [2016] I. Goodfellow, Y. Bengio, and A. Courville.  _Deep Learning_.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * Hu et al. [2020] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec.  Open Graph Benchmark: Datasets for Machine Learning on Graphs.  _arXiv e-prints_ , 2005.00687:arXiv:2005.00687, May 2020\.
  * Kipf and Welling [2016] T. N. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv e-prints_ , 1609.02907, 2016.  URL http://arxiv.org/abs/1609.02907.
  * Kolchinsky and Tracey [2017] A. Kolchinsky and B. D. Tracey.  Estimating mixture entropy with pairwise distances.  _arXiv e-prints_ , 1706.02419, 2017.  URL http://arxiv.org/abs/1706.02419.
  * Kolchinsky et al. [2019] A. Kolchinsky, B. D. Tracey, and D. H. Wolpert.  Nonlinear information bottleneck.  _Entropy_ , 21(12), 2019.  URL http://arxiv.org/abs/1705.02436.
  * Naftali Tishby [2000] W. B. Naftali Tishby, Fernando C. Pereira.  The information bottleneck method.  _arXiv e-prints_ , physics/0004057, 2000.  URL https://arxiv.org/abs/physics/0004057.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _arXiv e-prints_ , 1912.01703, 2019.  URL http://arxiv.org/abs/1912.01703.
  * Saxe et al. [2019] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox.  On the information bottleneck theory of deep learning.  _Journal of Statistical Mechanics: Theory and Experiment_ , 2019(12):124020, dec 2019.  10.1088/1742-5468/ab3985.  URL https://doi.org/10.1088/1742-5468/ab3985.
  * Shwartz-Ziv and Tishby [2017] R. Shwartz-Ziv and N. Tishby.  Opening the black box of deep neural networks via information.  _arXiv e-prints_ , 1703.00810, 2017.  URL http://arxiv.org/abs/1703.00810.
  * Sloth Lorenzen et al. [2021] S. Sloth Lorenzen, C. Igel, and M. Nielsen.  Information Bottleneck: Exact Analysis of (Quantized) Neural Networks.  _arXiv e-prints_ , 2106.12912:arXiv:2106.12912, June 2021\.
  * Tishby and Zaslavsky [2015] N. Tishby and N. Zaslavsky.  Deep learning and the information bottleneck principle.  In _2015 IEEE Information Theory Workshop (ITW)_ , pages 1–5. IEEE, 2015.  10.1109/ITW.2015.7133169.
  * Veličković et al. [2017] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio.  Graph Attention Networks.  _arXiv e-prints_ , 1710.10903:arXiv:1710.10903, Oct. 2017\.
  * Yu and Príncipe [2019] S. Yu and J. C. Príncipe.  Understanding autoencoders with information theoretic concepts.  _Neural Networks_ , 117:104–123, 2019.  ISSN 0893-6080.  10.1016/j.neunet.2019.05.003.  URL https://www.sciencedirect.com/science/article/pii/S0893608019301352.
  * Yu et al. [2021a] S. Yu, K. Wickstrøm, R. Jenssen, and J. C. Príncipe.  Understanding convolutional neural networks with information theory: An initial exploration.  _IEEE Transactions on Neural Networks and Learning Systems_ , 32(1):435–442, 2021a.  10.1109/TNNLS.2020.2968509.
  * Yu et al. [2021b] X. Yu, S. Yu, and J. C. Príncipe.  Deep deterministic information bottleneck with matrix-based entropy functional.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 3160–3164, 2021b.  10.1109/ICASSP39728.2021.9414151.