[1]

[1]The acronym SHAMSUL, derived from a Semitic word meaning "the Sun," serves
as a symbolic representation of our heatmap analysis approach aimed at
unveiling the medical significance inherent in the predictions of black box
deep learning models.

[a] [1] [1]Corresponding author

[a]

[b]

[a]

# SHAMSUL: Simultaneous Heatmap-Analysis to investigate Medical Significance
Utilizing Local interpretability methods

Mahbub Ul Alam<EMAIL_ADDRESS>Jaakko Hollmén<EMAIL_ADDRESS>Jón
Rúnar Baldvinsson<EMAIL_ADDRESS>Rahim Rahmani<EMAIL_ADDRESS>Department of Computer and Systems Sciences, Stockholm University, Stockholm,
Sweden  Skatturinn (Iceland Revenue and Customs), Reykjavík, Iceland

###### Abstract

[S U M M A R Y]

The interpretability of deep neural networks has become a subject of great
interest within the medical and healthcare domain. This attention stems from
concerns regarding transparency, legal and ethical considerations, and the
medical significance of predictions generated by these deep neural networks in
clinical decision support systems. To address this matter, our study delves
into the application of four well-established interpretability methods: Local
Interpretable Model-agnostic Explanations (LIME), Shapley Additive
exPlanations (SHAP), Gradient-weighted Class Activation Mapping (Grad-CAM),
and Layer-wise Relevance Propagation (LRP). Leveraging the approach of
transfer learning with a multi-label-multi-class chest radiography dataset, we
aim to interpret predictions pertaining to specific pathology classes. Our
analysis encompasses both single-label and multi-label predictions, providing
a comprehensive and unbiased assessment through quantitative and qualitative
investigations, which are compared against human expert annotation. Notably,
Grad-CAM demonstrates the most favorable performance in quantitative
evaluation, while the LIME heatmap segmentation visualization exhibits the
highest level of medical significance. Our research highlights the strengths
and limitations of these interpretability methods and suggests that a
multimodal-based approach, incorporating diverse sources of information beyond
chest radiography images, could offer additional insights for enhancing
interpretability in the medical domain.

###### keywords:

Deep Learning Interpretability Methods LIME SHAP Grad-CAM LRP Chest X-ray
Heatmap Visualization Clinical Decision Support System

##  1 Introduction

Machine learning (ML) [1] has emerged as a field utilizing probabilistic
computational methods to enable machines to learn from examples. ML models,
for instance, can be trained to diagnose patients by leveraging labeled
medical data [2]. To achieve accurate predictions, these models require fine-
tuning of certain parameter values, a process referred to as ’learning the
model’ [3].

Deep Learning (DL) [4], a subfield of ML, enables models to learn intricate
representations of data by progressively synthesizing lower-level information
into higher-level abstractions. This process minimizes the need for human-
engineered features. Deep neural networks (DNNs) are commonly employed in this
regard, as they can learn the necessary features from input data through
complex compositions of numerous tuning parameters.

The remarkable success of deep learning-based models presents promising
opportunities in the healthcare and medicine sectors. Specifically, the
medical diagnosis field can benefit from employing deep learning methods as
aids for human specialists [5]. However, the use of deep learning methods in
critical environments necessitates transparency in the decision-making
process. One major challenge with DNNs is their lack of interpretability in
this complex decision-making process. To address this, various
interpretability methods have been proposed [6, 7]. In the domain of ML/DL,
interpretability methods aim to explain the behavior of models, enabling
humans to understand specific decisions to a certain extent [8, 9].
Interpretable models facilitate comprehension of outcomes, thereby aiding
evaluations of ethics, trust, causality, and fairness [10].

Interpretability plays a crucial role in assessing these criteria, making it
imperative to include interpretability in the work involving deep learning
models from legal and ethical perspectives. Regulatory frameworks such as the
European Union (EU) General Data Protection Regulation (GDPR) [11], EU
Artificial Intelligence Act [12], and the US Algorithmic Accountability Act of
2022 [13] all emphasize the importance of interpretability.

Interpretability methods can be categorized in various ways. They can be
classified as local or global based on the scope of their interpretation [14].
Local methods interpret individual predictions, while global methods interpret
all predictions of a model simultaneously. Another classification [15]
categorizes interpretability methods as follows: (a) Visualization methods,
which highlight specific input features that significantly influence a
particular prediction; (b) Model distillation, employing a relatively simple
surrogate model to explain the predictions of the main model, such as a DNN;
and (c) Intrinsic methods, utilizing specialized DNNs that provide
interpretations for predictions. If an interpretability method focuses solely
on a particular model, it is considered model-specific; otherwise, it is
regarded as model-agnostic [3].

When using interpretability methods on models trained for medical diagnosis,
it is crucial to focus on interpreting the medical significance. Proper
consideration of this significance ensures the usability of such models in
healthcare sectors, including clinical decision support systems [16, 17]. The
medical significance of symptoms or disease processes pertains to their
clinical manifestations, which can disrupt the normal functioning of cells,
tissues, organs, or organ systems [18].

Radiography images serve as primary diagnostic tools and enable the analysis
of medical significance by evaluating the distinctive shapes and colors in the
images [18]. Given the widespread usage of radiography, abundant data is
available, making it a suitable choice for DL models [19] and their
interpretability aspects [20].

In this study, we introduce SHAMSUL, an approach that stands for Simultaneous
Heatmap-Analysis to investigate Medical Significance Utilizing Local
interpretability methods. Our primary focus is to analyze the medical
significance of specific pathology classes using a DNN model trained on
CheXpert [21], a multi-label multi-class dataset comprising chest radiography
images. This DNN model employs transfer learning on a deep convolutional
neural network (CNN) architecture [22].

Our work extends our previous research [23] and makes the following
contributions:

  1. 1.

In addition to the previously used local interpretability methods, Gradient-
weighted Class Activation Mapping (Grad-CAM) [24] and Layerwise-Relevance
Propagation (LRP) [25], we incorporate two widely-used interpretability
methods: Local Interpretable Model-agnostic Explanations (LIME) [26] and
Shapley Additive exPlanations (SHAP) [27]. We employ these methods to
investigate the medical significance of selected pathology classes using
heatmap visualization.

  2. 2.

We present a meticulous and comprehensive quantitative evaluation of these
interpretability methods through an assessment of their performance across
diverse pathology classes. In particular, we undertake a comparative analysis
by juxtaposing the heatmap performance of these methods with human expert
annotation.

  3. 3.

Moreover, in order to provide a comprehensive understanding of
interpretability, we extensively delve into the qualitative aspects of
performance within two selected pathology classes. By conducting a holistic
overview, we offer an in-depth analysis of the qualitative attributes of the
interpretability methods. In particular, we focus on the instance that
achieves the highest quantitative interpretability performance within each
selected class, thereby highlighting its significance. Additionally, to ensure
methodological consistency, systematic analysis, and impartiality, we also
select the best predicted instance based on probability score for each class,
as well as one multi-labelled instance.

Through this comprehensive analysis, we aim to present a nuanced and balanced
overview of the practicality and effectiveness of these interpretability
methods, while shedding light on the underlying reasoning behind both
successful and unsuccessful predictions, rooted in theoretical rationale.

The code implementation of SHAMSUL can be accessed via the following link
(accessed on 09 July 2023):
https://github.com/anondo1969/SHAMSUL

##  2 Related Works

This section provides a concise overview of the interpretability methods
mentioned earlier and their application in the context of chest radiography
images and related biomedical image-based prediction tasks.

LIME has been widely employed for COVID-19 prediction tasks using chest
radiography images [28, 29, 30]. Additionally, LIME and Grad-CAM have been
utilized to investigate the impact of lung segmentation on the prediction of
COVID-19 in chest radiography images [31]. DeepCOVIDExplainer utilized
gradient-guided class activation maps (Grad-CAM++) and layer-wise relevance
propagation (LRP) [32]. SHAP has been employed to explain the prediction of
choroidal neovascularization (CNV), diabetic macular edema (DME), and drusens
using optical coherence tomography (OCT) scans [33]. It has also been used for
COVID-19 prediction in chest radiography images [34]. CAM (class activation
mappings) heatmaps have been utilized to interpret predictions for pneumonia
detection [35] and multi-pathology detection [36]. The abnormality in chest
radiography images was investigated using Grad-CAM [37]. The visualization of
tuberculosis was explored using saliency maps and Grad-CAM [38]. Notable works
utilizing LRP include MRI-based Alzheimer’s disease classification [39] and
automatic liver tumor segmentation [40]. The quantitative evaluation of Grad-
CAM, LRP, and five additional saliency methods was conducted using chest X-ray
data [41]. It is worth noting that these works did not perform a comprehensive
comparison of the four interpretability methods, as previously mentioned.

##  3 Methods and the Dataset

In this section, we present the details of the interpretability methods and
datasets utilized in SHAMSUL.

###  3.1 LIME

One of the interpretability methods employed in SHAMSUL is LIME, which stands
for Local Interpretable Model-agnostic Explanations [26]. LIME operates under
the assumption that the local behavior of a model is simpler to compute than
its global behavior. Based on this assumption, LIME approximates the local
behavior of the model to provide explanations for individual predictions. This
method can be applied to various types of inputs, including images, tabular
data, and text.

LIME focuses on explaining the prediction for a single instance by considering
its local neighborhood. It expands the input data around the specific instance
to gather more information about the local area. Subsequently, LIME employs a
sparse linear model, such as the least absolute shrinkage and selection
operator (lasso) [42], to train on the augmented data. By using a linear
model, LIME ensures interpretability, as linear models are inherently more
interpretable. The regularization factor in the linear model is fine-tuned to
obtain a suitable number of non-zero coefficients, facilitating the
interpretation of the specific input.

To assign weights to the surrounding points in the local neighborhood, LIME
employs a kernel function ([43], with the default choice being a radial basis
function kernel. The weights are calculated based on the distance between the
points and the instance under consideration.

$\xi(y)=argmin_{g\in G}L(f,g,\pi_{x^{\prime}})+\Omega(g)$ (1)

The mathematical representation of the LIME explanation for a particular input
$x$ is provided by Equation 1. Let $f$ denote the trained model, and $g$
represent the linear surrogate model. The proximity area of $x$ is denoted as
$\pi_{x^{\prime}}$, where $x^{\prime}$ represents the surrounding points.
$\Omega(g)$ represents the penalty estimation for the complexity of $g$. The
aim of LIME is to minimize the locality-aware squared case, denoted by $L$.

###  3.2 SHAP

The SHAP (Shapley Additive exPlanations) method [27] is employed in this study
to provide interpretability for individual predictions. Consider a model
$f:X\rightarrow\mathbb{R}$, where we aim to interpret the prediction $f(x)$
for a given instance $x\in X$. Let $M$ denote the set of features present in
$x$. The binary vector $z\in{0,1}^{M}$ represents the absence or presence of
each feature. By removing the features set to $0$ in $z$ from $x$, we obtain
an input sample denoted as $h(z)\in X$.

$\varphi_{i}(x)=\sum_{z{\in\\{0,1\\}^{M}}\setminus\\{i\\}}\frac{\mid
z\mid!(M-\mid z\mid-1)!}{M!}B$ (2)

$B=(f(h(z\cup\\{i\\}))-f(h(z)))$ (3)

The attribution score, determined by the Shapley value, can be calculated for
each feature $1\leq i\leq M$. Here, the cardinality operator $\mid\cdot\mid$
represents the number of non-zero elements in a set. To approximate the
Shapley values in our work, we utilized KernelSHAP [44], which combines the
concepts of LIME and Shapley values. In KernelSHAP, a linear model
$g:\mathbb{R}^{M}\rightarrow\mathbb{R}$ is trained using a weighted least
squares (WLS) regression method, with its optimal coefficients serving as the
Shapley values. The data points are uniformly sampled from the simplified
input space ${0,1}^{M}$, considering the outputs as the value of $f\circ h$.

###  3.3 Grad-CAM

Grad-CAM (Gradient-weighted Class Activation Mapping) [24] is a technique that
assesses the significance of individual neurons in an input using a feature
map. This feature map is generated by leveraging the gradient scores
propagated to the final convolutional layer of a Convolutional Neural Network
(CNN). By utilizing Grad-CAM, it becomes possible to produce a heatmap that
highlights the salient features of the input.

Let $y_{c}$ denote the class score for a specific class $c$. Each neuron’s
significance can be interpreted as $y_{c}$ and is derived by calculating the
gradient of $y_{c}$ with respect to the activation $A$. This process yields an
importance weight $\alpha_{k}^{c}$ for each feature map $k$, given by:

$\alpha_{k}^{c}=\frac{1}{z}\sum_{i}\sum_{j}\frac{\partial y_{c}}{\partial
A_{ij}^{k}}$ (4)

The heatmap can now be generated as a weighted combination of the activation
maps. To emphasize positive cases exclusively, a Rectified Linear Unit (ReLU)
function [45] is employed to discard negative values. Hence, the Grad-CAM
heatmap $L_{Grad-CAM}^{c}$ is defined as:

$L_{Grad-CAM}^{c}=ReLU(\sum_{k}\alpha_{k}^{c}A^{k})$ (5)

This computational process yields the Grad-CAM heatmap, effectively
highlighting the important features within the input that are associated with
the class $c$.

###  3.4 LRP

LRP (Layerwise-Relevance Propagation) [25] operates under the assumption that
a model can be decomposed into multiple layers of computation. It leverages
the backpropagation algorithm [46] to calculate the importance of each neuron,
propagating from the final layer to the input layer while traversing through
the intermediate layers. The relevance of each neuron $R_{i}^{l}$ can be
computed using the following equation:

$R_{i}^{l}=\sum_{j}R_{i\leftarrow
j}^{(l,l+1)}=\sum_{j}\frac{z_{ij}}{z_{j}}R_{j}^{l+1}$ (6)

This equation represents the relevance $R_{i}^{l}$ of a neuron in a lower
layer, based on the relevance $R_{j}^{l+1}$ of the corresponding neuron in the
upper layer. The localized pre-activations are denoted as
$z_{ij}=x_{i}w_{ij}$, where $x_{i}$ represents the neuron’s input and $w_{ij}$
signifies the weight associated with the connection. Additionally,
$z_{j}=\sum_{i}z_{ij}+b_{j}$, where $b_{j}$ denotes the bias term. Through
this layerwise-relevance propagation mechanism, LRP enables the assessment of
the importance of neurons throughout the network architecture.

###  3.5 Dataset

Table 1: Classification details of the CheXpert dataset [21]. Pathology |  | Positive
---
(%)
| Uncertain
---
(%)
| Negative
---
(%)
| No
---
Finding
| 16627
---
(8.86)
| 0
---
(0.0)
| 171014
---
(91.14)
| Enlarged
---
Cardiomediastinum
| 9020
---
(4.81)
| 10148
---
(5.41)
| 168473
---
(89.78)
Cardiomegaly |  | 23002
---
(12.26)
| 6597
---
(3.52)
| 158042
---
(84.23)
| Lung
---
Lesion
| 6856
---
(3.65)
| 1071
---
(0.57)
| 179714
---
(95.78)
| Lung
---
Opacity
| 92669
---
(49.39)
| 4341
---
(2.31)
| 90631
---
(48.3)
Edema |  | 48905
---
(26.06)
| 11571
---
(6.17)
| 127165
---
(67.77)
Consolidation |  | 12730
---
(6.78)
| 23976
---
(12.78)
| 150935
---
(80.44)
Pneumonia |  | 4576
---
(2.44)
| 15658
---
(8.34)
| 167407
---
(89.26)
Atelectasis |  | 29333
---
(15.63)
| 29377
---
(15.66)
| 128931
---
(68.71)
Pneumothorax |  | 17313
---
(9.23)
| 2663
---
(1.42)
| 167665
---
(89.26)
| Pleural
---
Effusion
| 75696
---
(40.34)
| 9419
---
(5.02)
| 102526
---
(54.64)
| Pleural
---
Other
| 2441
---
(1.3)
| 1771
---
(0.94)
| 183429
---
(97.76)
Fracture |  | 7270
---
(3.87)
| 484
---
(0.26)
| 179887
---
(95.87)
| Support
---
Devices
| 105831
---
(56.4)
| 898
---
(0.48)
| 80912
---
(43.12)

####  3.5.1 CheXpert

The CheXpert (Chest eXpert) [21] dataset is a comprehensive collection of
264,316 chest radiography images derived from 65,240 patients. It is annotated
with three distinct labels: positive, negative, and uncertain, reflecting the
presence, absence, or ambiguity of pathologies. Spanning a period from October
2002 to July 2017, the dataset was curated from the Stanford University
Medical Center. To facilitate analysis and interpretation, the dataset is
further divided into 14 subgroups, each corresponding to a specific
pathological condition. Table 1 provides a summary of the dataset
characteristics. For the pathology prediction task, a downscaled version of
the original dataset, occupying approximately 11 gigabytes instead of the
original 439 gigabytes, was utilized. The downscaled version can be obtained
from the Stanford machine learning group [21].

####  3.5.2 CheXlocalize

The CheXlocalize [41] dataset serves as a valuable resource for the
localization of 10 distinct pathologies on chest X-rays, offering radiologist-
annotated segmentations. This dataset encompasses two types of radiologist
annotations: (1) pixel-level segmentations and (2) most-representative points.
These annotations were meticulously applied to images extracted from the two
different CheXpert data sets. The first set comprises 234 chest X-rays derived
from 200 patients, while the second set comprises 668 chest X-rays obtained
from 500 patients. Notably, the annotations were performed by board-certified
radiologists, ensuring a high level of expertise and accuracy in the
annotations.

##  4 Experimental Setup

###  4.1 Pathology Prediction

The experiment setup for our current study remained consistent with our
previous work on prediction and visualization [23], utilizing the same trained
model. The CheXpert dataset was employed, employing the provided train-test
split (223414 train data instances and 234 test data instances). The task at
hand involved multi-label classification for pathology prediction, wherein the
uncertain labels were treated as negative instances. To achieve this, a
DenseNet-121 (Dense Convolutional Network) [47] was adopted as the base
architecture for transfer learning, leveraging pre-trained weights from
ImageNet [48]. For compatibility with the DenseNet-121 model, the input chest
radiography images were resized to $224\times 224$ pixels. Subsequently, the
images were scaled by a factor of 255 and converted to RGB (red-green-blue)
values. The backpropagation algorithm employed was the ’Adam optimizer’, with
a minibatch size of 16 and training conducted for 100 epochs.

We used AUROC and AUPRC metrics for the pathology prediction evaluation score.
The AUROC (area under the receiver operating characteristic) metric considers
the trade-off between the true-positive rate (TPR) and the false-positive rate
(FPR) across various decision thresholds. Additionally, we present the
weighted average AUROC and AUPRC (area under the precision-recall curve)
scores, encompassing all classes except for the Fracture class, which lacks
positive instances in the test data. The precision-recall curve, which
illustrates the relationship between precision (y-axis) and recall (x-axis) at
various decision thresholds, is a valuable metric for evaluating the
performance of a classification model. It provides insights into the trade-off
between correctly identified positive instances (precision) and the model’s
ability to capture all positive instances (recall). [49].

###  4.2 Quantitative Interpretability Evaluation

In order to quantitatively evaluate and visualize the obtained relevance from
the interpretability methods, pixel-level segmentations were employed. These
segmentations were generated by converting the interpretability heatmap score
values, where high or positive scores were indicative of the relevance or
significance of specific pathology classes within the corresponding pixel
regions.

To assess the quality of the interpretability methods’ generated
segmentations, we utilized the mean Intersection over Union (IoU) metric [41].
The IoU measures the overlap between the predicted segmentations from the
interpretability methods and the ground-truth annotation segmentations. This
metric quantifies the ratio between the area of overlap and the area of union,
resulting in a value between 0 and 1. A value of 0 signifies no overlap, while
a value of 1 indicates perfectly aligned segmentations. By employing this
metric, we can evaluate the extent to which the interpretability methods’
generated segmentations align with the human expert annotation segmentations,
providing insight into the accuracy and effectiveness of the interpretability
methods.

###  4.3 Qualitative Interpretability Analysis

(a) Lung Opacity

(b) Support Devices

Figure 1: Instances of the Lung Opacity (prediction probability score: 0.63)
and Support Devices (prediction probability score: 0.93) classes with human
expert segmentation annotation for subsequent qualitative analysis.

(a) Lung Opacity

(b) Support Devices

Figure 2: Multi-label instance of the Lung Opacity (prediction probability
score: 0.58) and Support Devices (prediction probability score: 0.92) classes
with human expert segmentation annotation for subsequent qualitative analysis.

In this qualitative study, our research objective is to explore the
qualitative aspects of interpretability in two prominent pathology classes,
namely Lung Opacity and Support Devices, which are widely utilized in the
field. We visualize pixel-level segmentations of these selected pathology
classes. We focused on these classes due to their high prediction performance,
medical relevance, and prevalence within the domain. By employing various
local interpretable methods, we aim to gain insights into how these methods
shed light on the distinctive characteristics of the selected pathology
classes.

For our qualitative analysis, we have established two criteria:

  1. 1.

Examining the instance with the highest quantitative interpretability
performance within these two classes. Through this analysis, we seek to
evaluate how the quantitative performance can be interpreted qualitatively and
whether this superior quantitative score aligns with the medical significance
of the findings.

  2. 2.

Additionally, we select the best predicted instance based on the prediction
probability score for each of these classes, along with one instance
exhibiting multiple labels. This additional selection allows us to investigate
how different interpretability methods interpret the same instance in varying
ways. Figure 1(a) depicts the chosen instance from the Lung Opacity class,
while Figure 1(b) showcases the instance from the Support Devices class, both
of which have been annotated with human expert segmentation. The accompanying
figures also present the corresponding prediction probability scores. Figure 2
depicts the chosen multi-label instance for the Lung Opacity and Support
Devices classes.

####  4.3.1 Medical Significance of the Selected Classes

Lung Opacity, also referred to as "pulmonary opacification" [50], represents a
common observation in chest radiography images. It signifies a decreased ratio
of gas to soft tissue in the lungs. In a healthy lung, the appearance on a
chest radiograph is black, indicating unobstructed airspaces. The presence of
a gray or hazy region indicates the filling of airspaces with fluid, pus, or
other substances. It may also indicate anomalies in the alveolar walls or
thickening of the interstitial space between the lungs [18]. Lung Opacity can
be a symptom of various diseases, including COVID-19 and pneumonia [51].

Support Devices refers to the identification of medical device(s) within the
thoracic region. Prompt and accurate localization of these devices is crucial,
as any potential complications need to be identified and addressed promptly to
ensure patient safety. Various types of support devices exist, such as
cardiac, endovascular, bronchopulmonary, and esophageal devices [52].

##  5 Results

Figure 3: The evaluation of CheXpert pathology prediction. The outermost
circle displays the class names. The middle circle represents the AUROC value
for each class. The innermost circle displays the total number of positive
instances observed for each respective class.

###  5.1 Pathology Prediction

The evaluation of the model’s performance is depicted in Figure 3,
illustrating the AUROC scores for each class. For this evaluation, a total of
234 input evaluation data points were utilized. However, it should be noted
that the Fracture class is absent in this evaluation due to the absence of any
positive instances in the provided test data. The weighted average AUROC is
determined to be 0.76. Additionally, the weighted average AUPRC is calculated
to be 0.65.

An important observation is the considerable imbalance among positive
instances across the classes. However, despite this imbalance, the results can
be considered balanced. Notably, the Pleural Other class exhibits noteworthy
performance, even with only a single positive instance. These findings
highlight the robustness and potential effectiveness of the model in handling
varying class distributions and detecting positive instances accurately, even
in challenging scenarios.

It is pertinent to note that the task of pathology prediction encompasses four
additional classes in comparison to the quantitative evaluation tasks
conducted using the CheXlocalize dataset. The CheXlocalize dataset employed in
the quantitative evaluation comprised ten classes. The four supplementary
classes introduced in the pathology prediction task include: No Finding,
Pleural Other, Pneumonia, and Fracture. The inclusion of these classes expands
the scope of the pathology prediction task and enables the identification and
classification of a wider range of pathological conditions.

###  5.2 LIME Interpretability Investigation

Figure 4: LIME instance-wise quantitative IoU score evaluation.

Figure 4 depicts the evaluation of IoU scores, comparing the segmentation
generated by the LIME interpretability method against human expert
annotations. Each box-plot in the figure represents one of the ten pathology
classes considered. The mean IoU values are indicated on each box-plot, while
the dashed line represents the overall mean, and the solid line in close
proximity to it signifies the median value. Notably, the results reveal that
the LIME method yielded the most favorable outcomes for the Edema class,
exhibiting a mean score of 0.225. Conversely, it exhibited the poorest
performance on the Lung Lesion class, resulting in a score of zero. These
findings provide valuable insights into the efficacy of the LIME
interpretability method for different pathology classes.

(a) Ground Truth

(b) LIME Segmentation

Figure 5: LIME heatmap score segmentation of the Lung Opacity (IoU score:
0.38, prediction probability score: 0.42) class with human expert segmentation
annotation for comparison

Figure 5 presents the LIME segmentation visualization for the instance with
the highest IoU score in the Lung Opacity class. Upon examination, it is
evident that LIME accurately identifies the lower-left lung region. However,
there is a noticeable discrepancy between the LIME-generated segmentation and
the human expert annotation, as LIME tends to encompass a larger area in its
segmentation. Despite this slight disparity, it is noteworthy that LIME
exhibits commendable performance in correctly identifying the relevant region
in this instance. This observation underscores the effectiveness of the LIME
method in localizing and delineating the target area for the Lung Opacity
class.

(a) Ground Truth

(b) LIME Segmentation

Figure 6: LIME heatmap score segmentation of the best IoU scoring Support
Devices (IoU score: 0.48, prediction probability score: 0.86) class instance
with human expert segmentation annotation for comparison.

Figure 6 displays the LIME segmentation visualization for the instance with
the highest IoU score in the Support Devices class. Upon examination, it is
evident that LIME accurately identifies the upper-right lung region,
successfully localizing the target area of interest. However, it is important
to note that the LIME-generated segmentation does not entirely encompass the
entirety of the corresponding human expert annotation area. Despite this
partial coverage, the result demonstrates LIME’s ability to effectively
identify and localize the upper-right lung region for instances within the
Support Devices class. This finding underscores the utility of the LIME
interpretability method in identifying regions of interest, although further
refinement may be necessary to achieve full correspondence with human expert
annotations.

(a) Lung Opacity

(b) Support Devices

Figure 7: LIME heatmap score segmentation of the best prediction probability
scoring Lung Opacity (prediction probability score: 0.63, IoU score: 0.16) and
Support Devices (prediction probability score: 0.93, IoU score: 0.25) class
instances.

Figure 7 showcases the LIME segmentation visualization for the instance with
the highest prediction probability in both the Lung Opacity and Support
Devices classes. A comparative analysis with Figure 1, which presents the
human expert annotation segmentation for these classes, allows for insightful
observations. In the case of the Lung Opacity class, the LIME segmentation
only partially covers one of the three designated areas. Conversely, in the
Support Devices class, the LIME segmentation predominantly focuses on the
upper-right lung region, overlapping with and partially covering the
corresponding human expert annotation segmentation area, particularly its left
section.

(a) Lung Opacity

(b) Support Devices

Figure 8: LIME heatmap score segmentation of the best prediction probability
scoring multi-label class instance (Lung Opacity \- prediction probability
score: 0.58, IoU score: 0.31 and Support Devices \- prediction probability
score: 0.92, IoU score: 0.08).

Figure 8 depicts the LIME segmentation visualization for the instance with the
highest prediction probability among the multi-label instances in both the
Lung Opacity and Support Devices classes. A comparison with Figure 2, which
presents the corresponding human expert annotation segmentation for these
classes, allows for insightful analysis.

In the case of the Lung Opacity class, the LIME segmentation partially
identifies two of the three designated segmentation regions, aligning with
certain areas of interest. However, it is important to note that LIME
mistakenly identifies the left shoulder and an artifact situated on top of it
as relevant regions, despite their lack of association with the Lung Opacity
class. This highlights a potential limitation in the LIME method’s ability to
accurately discern between the true areas of interest and extraneous
artifacts.

Turning to the Support Devices class, the LIME segmentation covers a
substantial portion of the left side of the body, incorrectly marking both the
relevant region and an artifact positioned on top of it. This indicates a lack
of precision in the LIME method’s identification and localization of the
target region within the Support Devices class.

These observations emphasize the need for further refinement of the LIME
segmentation approach to ensure accurate and reliable identification of the
relevant regions while minimizing misclassifications of artifacts and
unrelated areas. Further refinement is warranted to improve its consistency
and alignment with human expert annotation segmentations for a more robust and
reliable analysis.

###  5.3 SHAP Interpretability Investigation

Figure 9: SHAP instance-wise quantitative IoU score evaluation.

Figure 9 presents the IoU evaluation scores, comparing the SHAP
interpretability method’s generated segmentation against the corresponding
human expert annotation. An examination of the results reveals that the SHAP
method did not demonstrate robust generalization across all pathology classes.
Particularly, for three classes, namely Consolidation, Enlarged
Cardiomediastinum, and Lung Lesion, SHAP failed to provide any meaningful
values for any of the instances assessed. This indicates a lack of
effectiveness in accurately identifying and segmenting these specific
pathology classes using the SHAP method.

Furthermore, the highest mean IoU score of 0.301 was achieved for the
Pneumothorax class. In contrast, the mean IoU scores for the remaining classes
were considerably lower when compared to this top score, indicating
comparatively poorer performance in accurately capturing the relevant regions
for those classes using the SHAP method.

(a) Ground Truth

(b) SHAP Segmentation

Figure 10: SHAP heatmap score segmentation of the Lung Opacity (IoU score:
0.34, prediction probability score: 0.52) class with human expert segmentation
annotation for comparison.

Figure 10 displays the SHAP segmentation visualization for the instance with
the highest IoU score within the Lung Opacity class. A careful examination of
the visualization reveals that the SHAP segmentation successfully covers the
region annotated by human experts, effectively localizing the target area of
interest. However, it is important to note that the SHAP segmentation also
erroneously encompasses both shoulders, the neck area, and an artifact located
in the top-right area.

(a) Ground Truth

(b) SHAP Segmentation

Figure 11: SHAP heatmap score segmentation of the best IoU scoring Support
Devices (IoU score: 0.26, prediction probability score: 0.63) class instance
with human expert segmentation annotation for comparison.

Figure 11 illustrates the SHAP segmentation visualization for the instance
with the highest IoU score within the Support Devices class. The SHAP
segmentation displays notable overlap with the ground truth areas,
successfully capturing certain regions of interest. However, it is important
to note that the SHAP segmentation is extensive and entirely overlaps the
bottom region of the image, thereby leading to incorrect labeling.

This observation highlights a limitation in the SHAP segmentation’s ability to
accurately delineate and localize the relevant region within the Support
Devices class. Although it achieves a reasonable degree of correspondence with
the ground truth areas, the excessive coverage beyond the intended region
compromises its accuracy and specificity.

(a) Lung Opacity

(b) Support Devices

Figure 12: SHAP heatmap score segmentation of the best prediction probability
scoring Lung Opacity (prediction probability score: 0.63, IoU score: 0.23) and
Support Devices (prediction probability score: 0.93, IoU score: 0.13) class
instances.

Figure 12 displays the SHAP segmentation visualization for the instance with
the highest prediction probability in both the Lung Opacity and Support
Devices classes. A comparison with Figure 1, which presents the corresponding
human expert annotation segmentation for these classes, allows for a
comprehensive analysis.

Upon examination, it becomes apparent that the SHAP segmentation almost
entirely covers the entire image for both classes. While this extensive
coverage includes the correct regions of interest, the overwhelming extent of
the segmentation diminishes its interpretive significance. Consequently, the
SHAP segmentation, despite accurately capturing the relevant regions, lacks
specificity and discriminative power due to its indiscriminate and excessive
nature.

(a) Lung Opacity

(b) Support Devices

Figure 13: SHAP heatmap score segmentation of the best prediction probability
scoring multi-label class instance (Lung Opacity \- prediction probability
score: 0.58, IoU score: 0.17 and Support Devices \- prediction probability
score: 0.92, IoU score: 0.03).

Figure 13 presents the SHAP segmentation visualization for the instance with
the highest prediction probability among the multi-label instances in both the
Lung Opacity and Support Devices classes. A comparative analysis with Figure
2, which displays the human expert annotation segmentation for these classes,
enables a comprehensive examination of the results.

In terms of the Lung Opacity class, the SHAP segmentation demonstrates partial
identification of two out of the three designated segmentation regions.
However, it incorrectly identifies the left shoulder, arm, neck, and an
artifact located in the left-top area of the body as relevant regions.
Moreover, the SHAP segmentation exhibits a somewhat extensive nature in
comparison to the ground truth segmentation regions. Consequently, while the
SHAP segmentation captures certain areas of interest, its inclusion of
unrelated structures and artifacts diminishes its overall accuracy and
interpretive value.

Regarding the Support Devices class, the SHAP segmentation covers the entire
regions of the image except for the middle part. However, when compared to
Figure 2(b), the relevance of the SHAP segmentation is insignificant. The
segmentation fails to accurately capture and delineate the specific areas of
interest within the Support Devices class, highlighting a limitation in the
SHAP method’s ability to provide precise and discriminative segmentations for
this class.

These observations underscore a limitation in the SHAP interpretability
method’s ability to provide concise and focused segmentations. Although it
successfully identifies the areas of interest, the lack of precision and the
encompassing nature of the SHAP segmentation limit its interpretive value.

###  5.4 Grad-CAM Interpretability Investigation

Figure 14: Grad-CAM instance-wise quantitative IoU score evaluation.

Figure 14 presents the IoU evaluation scores, comparing the Grad-CAM
interpretability method’s generated segmentation against the corresponding
human expert annotation. The results demonstrate a balanced performance of
Grad-CAM across all evaluated pathology classes.

Notably, the Cardiomegaly class achieves the highest mean IoU score of 0.234.
Conversely, the Lung Lesion class attains the worst mean IoU score of zero.
This score signifies a lack of correspondence and agreement between the Grad-
CAM segmentation and the human expert annotation for instances within this
class.

(a) Ground Truth

(b) Grad-CAM Segment.

Figure 15: Grad-CAM heatmap score segmentation of the Lung Opacity (IoU score:
0.35, prediction probability score: 0.54) class with human expert segmentation
annotation for comparison.

Figure 15 presents the Grad-CAM segmentation visualization for the instance
with the highest IoU score within the Lung Opacity class. Upon examination, it
becomes apparent that the Grad-CAM segmentation successfully encompasses the
region annotated by human experts, accurately localizing the intended area of
interest. However, it is important to note that the Grad-CAM segmentation
exhibits a broad and expansive nature, which can hinder the ability to
distinguish the precise boundaries of the human expert annotated region.

(a) Ground Truth

(b) Grad-CAM Segment.

Figure 16: Grad-CAM heatmap score segmentation of the best IoU scoring Support
Devices (IoU score: 0.32, prediction probability score: 0.63) class instance
with human expert segmentation annotation for comparison.

Figure 16 illustrates the Grad-CAM segmentation visualization for the instance
with the highest IoU score within the Support Devices class. Upon observation,
it is evident that the Grad-CAM segmentation successfully captures certain
regions in the top-right and middle-left parts of the body that correspond to
the correct areas of interest within the Support Devices class. However,
similar to previous cases, the Grad-CAM segmentation exhibits a relatively
large coverage area compared to the human expert annotated regions.

(a) Lung Opacity

(b) Support Devices

Figure 17: Grad-CAM heatmap score segmentation of the best prediction
probability scoring Lung Opacity (prediction probability score: 0.63, IoU
score: 0.23) and Support Devices (prediction probability score: 0.93, IoU
score: 0.18) class instances.

Figure 17 exhibits the Grad-CAM segmentation visualization for the instance
with the highest prediction probability in both the Lung Opacity and Support
Devices classes. A comparison with Figure 1, which displays the corresponding
human expert annotation segmentation for these classes, enables a
comprehensive analysis.

Upon examination, it becomes evident that the Grad-CAM segmentation, while
capturing the relevant regions of interest, covers a significantly broader
area compared to the human expert annotation segmentations for both classes.
This expansive coverage, while encompassing the correct regions, poses
challenges in precisely identifying the precise boundaries of the relevant
regions. Consequently, the Grad-CAM segmentations may not offer the level of
nuance required for a nuanced understanding of the medical significance of the
identified regions.

(a) Lung Opacity

(b) Support Devices

Figure 18: Grad-CAM heatmap score segmentation of the best prediction
probability scoring multi-label class instance (Lung Opacity \- prediction
probability score: 0.58, IoU score: NA and Support Devices \- prediction
probability score: 0.92, IoU score: 0.06).

In terms of the Lung Opacity class, the Grad-CAM segmentation fails to
identify any relevant area within the image. Turning to the Support Devices
class, the Grad-CAM segmentation exhibits extensive coverage of the middle and
upper-right area of the body. However, when compared to Figure 2(b), the
relevance of the Grad-CAM segmentation becomes insignificant. The broad
coverage of non-relevant areas diminishes the interpretability and
discriminative power of the Grad-CAM method for accurately identifying the
specific regions of interest within the Support Devices class.

Upon examination, it becomes evident that the Grad-CAM segmentation, while
capturing the relevant regions of interest, covers a significantly broader
area compared to the human expert annotation segmentations for both classes.
This expansive coverage, while encompassing the correct regions, poses
challenges in precisely identifying the precise boundaries of the relevant
regions. Consequently, the Grad-CAM segmentations may not offer the level of
nuance required for a nuanced understanding of the medical significance of the
identified regions.

###  5.5 LRP Interpretability Investigation

Figure 19: LRP instance-wise quantitative IoU score evaluation.

Figure 19 presents the IoU evaluation scores, which compare the LRP
interpretability method’s generated segmentation against the corresponding
human expert annotation. The results reveal valuable insights into the
performance of the LRP method in accurately delineating the regions of
interest.

Among the evaluated classes, the Enlarged Cardiomediastinum class achieved the
highest mean IoU score, with a value of 0.178. This indicates a relatively
better alignment between the LRP-generated segmentation and the human expert
annotation for this particular class. The higher mean IoU score suggests a
higher degree of correspondence and accuracy in identifying and delineating
the relevant regions. On the other hand, the Lung Lesion class obtained the
worst mean IoU score, with a value of zero.

(a) Ground Truth

(b) LRP Segmentation

Figure 20: LRP heatmap score segmentation of the Lung Opacity (IoU score:
0.24, prediction probability score: 0.63) class with human expert segmentation
annotation for comparison.

(a) Ground Truth

(b) LRP Segmentation

Figure 21: LRP heatmap score segmentation of the best IoU scoring Support
Devices (IoU score: 0.27, prediction probability score: 0.63) class instance
with human expert segmentation annotation for comparison.

(a) Lung Opacity

(b) Support Devices

Figure 22: LRP heatmap score segmentation of the best prediction probability
scoring Lung Opacity (prediction probability score: 0.63, IoU score: 0.24) and
Support Devices (prediction probability score: 0.93, IoU score: 0.14) class
instances.

(a) Lung Opacity

(b) Support Devices

Figure 23: LRP heatmap score segmentation of the best prediction probability
scoring multi-label class instance (Lung Opacity \- prediction probability
score: 0.58, IoU score: 0.12 and Support Devices \- prediction probability
score: 0.92, IoU score: 0.04).

The qualitative visualizations of all the cases, as presented in Figure 20,
21, 22, and 23, reveal a consistent pattern. Across these images, the LRP
segmentation covers the entirety of the image regions, exhibiting a uniform
and indiscriminate nature. Consequently, the LRP heatmaps lack class
discriminability, exhibit considerable noise, and fail to provide meaningful
insights specific to any particular classes. While the quantitative evaluation
in Figure 19 may yield acceptable values, the qualitative analysis underscores
the limited medical significance of the LRP method.

The visual examination of the LRP visualizations highlights the absence of
focused and interpretable segmentations. The complete coverage of image
regions by the LRP segmentation masks inhibits its ability to provide precise
and discriminative interpretations. This lack of specificity diminishes the
utility and usefulness of the LRP method from a medical significance
standpoint.

Thus, while the quantitative evaluation may provide some acceptable metrics,
the qualitative analysis reveals inherent limitations in the LRP
interpretability method. The non-discriminative, noisy, and indiscriminate
nature of the LRP heatmaps diminishes their practical value in medical image
analysis. Future research endeavors should focus on refining the LRP method to
address these limitations and develop more precise and medically significant
visualizations.

##  6 Discussion

This study undertakes an exploration of four prominent local interpretability
methods commonly utilized in the field, namely LIME, SHAP, Grad-CAM, and LRP.
Our primary objective is to comprehensively examine the applicability of these
methods through a quantitative evaluation of the heatmap score performance
across all pathology classes within a multi-label-multi-class pathology
prediction task. Furthermore, we aim to investigate the medical significance
of two selected pathology classes by employing visualizations in the form of
heatmap-generated segmentation. In both qualitative and quantitative analyses,
we compare the performance of these methods against human expert annotation.
To conduct our investigation, we employ the Chexpert dataset, which serves as
a suitable resource for the pathology prediction task at hand. Additionally,
we utilize the CheXlocalize dataset, a complementary dataset to Chexpert,
where human experts have provided annotations for the ground truth values, to
facilitate our interpretability investigation.

The field of deep learning offers a wide range of interpretability methods [6,
7] tailored to diverse prediction tasks and neural network models. Our
research aimed to incorporate both qualitative investigation and quantitative
evaluation, necessitating a careful selection of interpretability methods to
manage the scope of visualization. One of the chosen methods, LIME, was deemed
valuable for interpreting individual instance predictions in black box machine
learning models, as it obviates the need for direct manipulation of the
original model. Instead, a surrogate model with global interpretability is
employed, rendering LIME an attractive option provided that its performance
meets satisfactory standards. SHAP, on the other hand, can be viewed as an
integration of LIME and Shapley values. By utilizing Shapley values, it
quantifies the contribution of each feature, thereby discerning their
significance with respect to specific predictions. The foundation of SHAP in
game theory lends it theoretical credibility and opens avenues for its
potential applicability as supporting evidence in ethical or legal contexts.
Consequently, its explanation may be well-received if it aligns with both
medical significance and mathematical rigor perspectives. In the context of
convolutional neural network (CNN) models, saliency methods [53, 54] hold
promise, making Grad-CAM and LRP suitable representatives of this category. It
is important to acknowledge that our selection of methods is not exhaustive,
as there exist numerous other equally strong candidates. Our future work aims
to address those additional methods and expand the scope of investigation
accordingly.

Expanding upon the foundations laid in our prior research [23], we have
maintained methodological consistency by employing the pre-trained model
established in our earlier work. The choice of utilizing a DenseNet-121 model
was deliberate, as its superiority over alternative models like ResNet152 or
Inception-v4 has been substantiated by a recent study [41] with relevance to
our research domain. This decision was of paramount importance in facilitating
a comprehensive and equitable comparison among the various interpretability
methods under examination. Consequently, our primary emphasis resides in the
evaluation of pathology prediction outcomes, which we deemed satisfactory
within the context of our investigation.

The quantitative evaluation of interpretability methods offers a reasonably
impartial and equitable assessment of their performance to a certain extent.
However, the overall quantitative performance falls short of achieving
satisfaction when compared to human expert annotation. Notably, all four
methods examined failed to identify any medical significance for the Lung
Lesion class. This outcome can be partly attributed to the fact that the
predicted data contained only a solitary positive instance of the Lung Lesion
class, as visually represented in Figure 3.

Curiously, across all interpretability methods, the highest performing class
did not exhibit a correlation with either the AUROC pathology prediction
evaluation score or the total number of prediction instances within each
class. This observation implies that a simple comparison based on different
deep neural network models may not be effective, as our conventional
evaluation scores, which typically serve as indicators of model performance,
did not exhibit strong correlation in this context.

On the whole, Grad-CAM demonstrated the most balanced mean Intersection over
Union (IoU) scores across all pathology classes. In contrast, SHAP yielded the
poorest scores, as it failed to identify medical significance across four
classes and demonstrated highly inconsistent performance for the remaining
classes.

Image classification tasks, such as the well-known Imagenet classification
[55], predominantly rely on the identification of object segmentation
boundaries within an image to distinguish between different object classes.
Consequently, achieving improved classification results in such tasks is
relatively easier compared to ascertaining the medical significance of chest
radiography images. The medical significance of pathology classifications
encompasses more nuanced considerations, and the segmentation boundaries in
these images tend to be more compact than those encountered in standard image
classification tasks.

In light of the observed disparities, we have adopted a different approach in
contrast to our previous work [23]. Rather than focusing solely on the
analysis of best, worst, and ambiguous predictions along with their correct
and incorrect variants, we have designed a scenario that offers a fairer
evaluation. Specifically, we have selected two pathology classes, namely Lung
Opacity and Support Devices, for in-depth exploration due to their two
distinctive characteristics.

Firstly, these classes represent the top-predicted categories based on
prediction scores. The Lung Opacity prediction yielded AUROC and AUPRC scores
of 0.87 and 0.88, respectively. Similarly, the Support Devices prediction
achieved AUROC and AUPRC scores of 0.89 and 0.87, respectively. Regrettably,
as observed during the quantitative evaluation, the highest prediction scores
did not align well with improved interpretability in terms of medical
significance.

Secondly, these selected classes exhibit similarities to the image boundary
segmentation paradigm. For instance, the presence of Lung Opacity should
ideally be identified as a distinct segment within the lung area, while
Support Devices should be recognized as an anomalous shape within the chest
radiography image, as discussed in section 4.3.1. Building upon this
underlying intuition, we can view these classes as tasks involving the
detection of segmentation boundaries, akin to standard image classification.
Notably, the Support Devices class possesses a relatively well-defined shape,
making it intriguing to explore whether confusion arises between these two
distinct shapes during model identification. It is essential to emphasize that
while there is no predefined or general shape for lung opacity, the shape of
Support Devices is comparatively more distinguishable.

The LIME heatmap segmentation visualization emerges as a comprehensive tool
for assessing the medical significance of pathology classes. Its qualitative
investigation yields the best overall performance. In the case of single-label
instances, it demonstrates relatively satisfactory outcomes. However, when
examining the multi-label scenario, confusion arises between the selected
classes. This observation highlights a significant limitation of LIME,
particularly in situations where the definition of the ’right’ surrounding-
local area is flexible. In such cases, its performance tends to be suboptimal.
Previous research has illustrated that slight variations in the proximity of
two such ’right’ surrounding local areas, as observed in a simulated setting,
can result in significantly divergent heatmaps [56]. Furthermore, the
effectiveness of LIME heavily depends on the chosen sampling technique for the
local model, and variations in the sampling approach can yield vastly
dissimilar outcomes. Moreover, it is disconcerting to note that the
manipulation of LIME’s results can be accomplished relatively swiftly [57].
Considering these limitations, LIME may not be the most suitable option when
confronted with sensitive subjects, such as the detection of medical
significance.

The SHAP heatmap segmentation method exhibited limited success in identifying
instances of medical significance, as it failed to capture the majority of
relevant cases. In the few instances where it did identify significant
regions, it became apparent that the method also included numerous unrelated
regions. This outcome may be attributed to a notable drawback associated with
the specific variant of SHAP we employed, namely KernelSHAP, which tends to
overlook feature dependence—a critical factor in certain cases. Consequently,
we observed instances where SHAP erroneously indicated that the significant
region for Lung Opacity existed outside the actual lung area. Furthermore,
there is a legitimate concern that biases may persist undetected within the
SHAP framework [57].

Grad-CAM is specifically designed for convolutional neural network (CNN)
models. Its effectiveness is most apparent when there is a well-defined
boundary distinction in an image classification task, as it selectively
highlights pixels that have a positive impact on the classification outcome.
However, when examining the resulting heatmap segmentation visualizations, we
observe that the range of highlighted pixels is relatively extensive. As a
consequence, the important area becomes broader, making it challenging to
precisely identify the correct region, especially in cases requiring nuanced
classification. It is important to note that pixel attribution, which is the
basis of Grad-CAM, has been associated with potential risks [58] and a lack of
reliability [59]. Therefore, caution should be exercised when employing Grad-
CAM for quantitative explanations rather than relying solely on its
qualitative interpretability.

The LRP method yielded the least informative visualizations for investigating
medical significance, resulting in overall poor performance. LRP is
characterized by its production of highly intricate, noisy, and non-class
discriminative heatmaps [60]. As a consequence, interpreting the heatmap
visualizations becomes increasingly challenging, particularly when attempting
to discern intricate class explanations that are specific to particular
locations and possess nuanced medical significance.

The exploration of medical significance in deep neural network-based
predictions is a multifaceted undertaking that demands a thorough examination
from both quantitative and qualitative perspectives, as demonstrated in this
study. Relying solely on the quantitative aspects of LRP heatmap segmentation,
for instance, would have provided an incomplete understanding of the medical
significance, as the qualitative exploration presented a distinct viewpoint.
In conclusion, to achieve more robust and informative outcomes for medical
diagnosis prediction tasks employing deep neural networks, it is imperative to
consider the interplay of three key aspects: (1) the data, (2) the prediction
model, and (3) the interpretability method.

Firstly, it is crucial to recognize that an interpretability method is
inherently dependent on the underlying model. Hence, efforts should be
directed towards improving the model’s alignment with real-world aspects.
Enhancing the model’s ability to emulate reality increases the transferability
and reliability of the model’s interpretations. In this context, the
integration of multimodal data, encompassing not only chest radiography but
also patient history, clinical notes, and laboratory tests, can be
instrumental. By leveraging multimodal information, the explanations for
predictions can be mutually reinforcing, potentially enhancing the
interpretability of the results.

Secondly, infusing domain knowledge into the model architecture is of
significant value. Incorporating multimodal data is one approach to achieving
this goal. Exploring different model architectures is another avenue worth
exploring. While CNNs are well-suited for image data as they capture spatial
dependencies effectively, our findings indicate that their performance was
unsatisfactory in this particular context. Vision transformers [61, 62], on
the other hand, have demonstrated superior performance and should be
investigated [63] further. To leverage the benefits of multimodality,
employing techniques such as ClinicalBERT [64, 65], along with variations of
ImageBERT [66], fine-tuned on chest radiography data and tailored to the
specific diagnosis task(s), could yield promising outcomes.

Lastly, the implementation and refinement of interpretability methods should
be guided by causal considerations (if feasible), feature dependence, and a
deeper understanding of the landscape of chest radiography or medical data in
general. The relevant regions within these chest radiography images are often
in close proximity with frequent overlaps, and the nuances of their
interpretations should be adequately reflected in the methods employed.

The encountered challenges emphasize the necessity for continued research and
progress in this domain. To overcome these challenges, our future endeavors
will focus on adopting a multi-modality-based approach. By integrating
multiple sources of information, employing diverse modalities, and exploring
various model architectures, we aim to enhance the comprehension and
interpretation of medical significance within deep neural network predictions.
This avenue of investigation holds significant promise for enhancing the
reliability and practicality of these predictions in the realm of medical
diagnostics and decision-making.

##  7 Conclusion

This study presents a comprehensive investigation of four prominent
interpretability methods, namely LIME (Local Interpretable Model-agnostic
Explanations), SHAP (Shapley Additive exPlanations), Grad-CAM (Gradient-
weighted Class Activation Mapping), and LRP (Layer-wise Relevance
Propagation). The primary objective is to scrutinize the medical significance
within a deep learning-based pathology prediction model applied to multi-label
chest radiography data. Through a meticulous analysis encompassing
quantitative and qualitative assessments in comparison to human expert
annotation, we provide a comprehensive and realistic evaluation of the current
landscape pertaining to the detection of medical significance. The
quantitative evaluation highlights Grad-CAM as the most effective method,
while the LIME heatmap segmentation visualization demonstrates the highest
degree of medical significance. By elucidating the strengths and limitations
of these interpretability methods, this research contributes to the
advancement of their practical applicability in the field of medical
diagnosis. The findings presented in this study aim to serve as a valuable
resource for researchers, practitioners, and stakeholders seeking to enhance
the interpretability and reliability of deep learning models in the critical
domain of medical diagnostics. Furthermore, we recognize that there is still
much work to be done in this field, and we encourage further exploration and
refinement of interpretable methods to continue driving progress in the
medical diagnosis sector.

## Declaration of competing interest

The authors do not have any conflicts of interest to disclose.

## References

  * Bishop and Nasrabadi [2006] Christopher M Bishop and Nasser M Nasrabadi.  _Pattern recognition and machine learning_ , volume 4.  Springer, 2006.
  * Bakator and Radosav [2018] Mihalj Bakator and Dragica Radosav.  Deep learning and medical diagnosis: A review of literature.  _Multimodal Technologies and Interaction_ , 2(3):47, 2018.
  * Molnar [2022] Christoph Molnar.  _Interpretable Machine Learning_.  Leanpub, 2 edition, 2022.  URL https://christophm.github.io/interpretable-ml-book.
  * Goodfellow et al. [2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep Learning_.  MIT press, 2016.
  * Ching et al. [2018] Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, et al.  Opportunities and obstacles for deep learning in biology and medicine.  _Journal of The Royal Society Interface_ , 15(141):20170387, 2018.
  * Linardatos et al. [2021] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis.  Explainable AI: A Review of Machine learning Interpretability Methods.  _Entropy_ , 23(1), 2021.  ISSN 1099-4300.  10.3390/e23010018.  URL https://www.mdpi.com/1099-4300/23/1/18.
  * Samek et al. [2019] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller.  _Explainable AI: interpreting, explaining and visualizing deep learning_ , volume 11700.  Springer Nature, 2019.
  * Miller [2019] Tim Miller.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial intelligence_ , 267:1–38, 2019.
  * Kim et al. [2016] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo.  Examples are not enough, learn to criticize! criticism for interpretability.  _Advances in neural information processing systems_ , 29, 2016.
  * Lipton [2018] Zachary C Lipton.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ , 16(3):31–57, 2018.
  * Voigt and Von dem Bussche [2017] Paul Voigt and Axel Von dem Bussche.  The EU general data protection regulation (GDPR).  _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ , 10(3152676):10–5555, 2017.
  * Veale and Borgesius [2021] Michael Veale and Frederik Zuiderveen Borgesius.  Demystifying the Draft EU Artificial Intelligence Act—Analysing the good, the bad, and the unclear elements of the proposed approach.  _Computer Law Review International_ , 22(4):97–112, 2021.
  * Mökander et al. [2022] Jakob Mökander, Prathm Juneja, David S Watson, and Luciano Floridi.  The US Algorithmic Accountability Act of 2022 vs. The EU Artificial Intelligence Act: what can they learn from each other?  _Minds and Machines_ , pages 1–8, 2022.
  * Buhrmester et al. [2021] Vanessa Buhrmester, David Münch, and Michael Arens.  Analysis of explainers of black box deep neural networks for computer vision: A survey.  _Machine Learning and Knowledge Extraction_ , 3(4):966–989, 2021.
  * Ras et al. [2022] Gabrielle Ras, Ning Xie, Marcel van Gerven, and Derek Doran.  Explainable deep learning: A field guide for the uninitiated.  _Journal of Artificial Intelligence Research_ , 73:329–397, 2022.
  * Souza-Pereira et al. [2021] Leonice Souza-Pereira, Sofia Ouhbi, and Nuno Pombo.  Quality-in-use characteristics for clinical decision support system assessment.  _Computer Methods and Programs in Biomedicine_ , 207:106169, 2021.
  * Xiao et al. [2018] Cao Xiao, Edward Choi, and Jimeng Sun.  Opportunities and challenges in developing deep learning models using electronic health records data: A systematic review.  _J. Am. Med Inform. Assoc._ , 25(10):1419–1428, 2018.
  * Goodman [2014] Lawrence R Goodman.  _Felson’s principles of chest roentgenology, a programmed text_.  Elsevier Health Sciences, 2014.
  * Çallı et al. [2021] Erdi Çallı, Ecem Sogancioglu, Bram van Ginneken, Kicky G van Leeuwen, and Keelin Murphy.  Deep learning for chest X-ray analysis: A survey.  _Medical Image Analysis_ , 72:102125, 2021.
  * Salahuddin et al. [2022] Zohaib Salahuddin, Henry C Woodruff, Avishek Chatterjee, and Philippe Lambin.  Transparency of deep neural networks for medical image analysis: A review of interpretability methods.  _Computers in biology and medicine_ , 140:105111, 2022.
  * Irvin et al. [2019] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 590–597, 2019.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep Learning.  _nature_ , 521(7553):436–444, 2015.
  * Alam et al. [2022] Mahbub Ul Alam, Jón Rúnar Baldvinsson, and Yuxia Wang.  Exploring LRP and Grad-CAM visualization to interpret multi-label-multi-class pathology prediction using chest radiography.  In _2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS)_ , pages 258–263, 2022.  10.1109/CBMS55023.2022.00052.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * Bach et al. [2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7):e0130140, 2015.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  "why should I trust you?" Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144, 2016.
  * Lundberg and Lee [2017] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30, 2017.
  * Ahsan et al. [2020a] Md Manjurul Ahsan, Kishor Datta Gupta, Mohammad Maminur Islam, Sajib Sen, Md Lutfar Rahman, and Mohammad Shakhawat Hossain.  Covid-19 symptoms detection based on nasnetmobile with explainable AI using various imaging modalities.  _Machine Learning and Knowledge Extraction_ , 2(4):490–504, 2020a.
  * Ahsan et al. [2020b] Md Manjurul Ahsan, Kishor Datta Gupta, Mohammad Maminur Islam, Sajib Sen, Md Rahman, Mohammad Shakhawat Hossain, et al.  Study of different deep learning approach with explainable AI for screening patients with COVID-19 symptoms: Using ct scan and chest x-ray image dataset.  _arXiv preprint arXiv:2007.12525_ , 2020b.
  * Jaiswal et al. [2020] Amit Kumar Jaiswal, Prayag Tiwari, Vipin Kumar Rathi, Jia Qian, Hari Mohan Pandey, and Victor Hugo C Albuquerque.  Covidpen: A novel covid-19 detection model using chest x-rays and ct scans.  _Medrxiv_ , 2020.
  * Teixeira et al. [2021] Lucas O Teixeira, Rodolfo M Pereira, Diego Bertolini, Luiz S Oliveira, Loris Nanni, George DC Cavalcanti, and Yandre MG Costa.  Impact of lung segmentation on the diagnosis and explanation of COVID-19 in chest X-ray images.  _Sensors_ , 21(21):7116, 2021.
  * Karim et al. [2020] Md Rezaul Karim, Till Döhmen, Michael Cochez, Oya Beyan, Dietrich Rebholz-Schuhmann, and Stefan Decker.  Deepcovidexplainer: explainable COVID-19 diagnosis from chest X-ray images.  In _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pages 1034–1037. IEEE, 2020.
  * Singh et al. [2020] Amitojdeep Singh, Abdul Rasheed Mohammed, John Zelek, and Vasudevan Lakshminarayanan.  Interpretation of deep learning using attributions: application to ophthalmic diagnosis.  In _Applications of machine learning 2020_ , volume 11511, pages 39–49. SPIE, 2020.
  * Ong et al. [2021] Joe Huei Ong, Kam Meng Goh, and Li Li Lim.  Comparative Analysis of Explainable Artificial Intelligence for COVID-19 Diagnosis on CXR image.  In _2021 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)_ , pages 185–190, 2021.  10.1109/ICSIPA52582.2021.9576766.
  * Rajpurkar et al. [2017] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  _arXiv preprint arXiv:1711.05225_ , 2017.
  * Rajpurkar et al. [2018] Pranav Rajpurkar, Jeremy Irvin, Robyn L Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis P Langlotz, et al.  Deep learning for chest radiograph diagnosis: A retrospective comparison of the Chexnext algorithm to practicing radiologists.  _PLoS medicine_ , 15(11):e1002686, 2018.
  * Hsu et al. [2019] WH Hsu, FJ Tsai, G Zhang, CK Chang, PH Hsieh, SN Yang, SS Sun, KenYK Liao, and Eddie TC Huang.  Development of a Deep Learning Model for Chest X-Ray Screening.  _MEDICAL PHYSICS INTERNATIONAL_ , 7(3):314, 2019\.
  * Pasa et al. [2019] F Pasa, V Golkov, F Pfeiffer, D Cremers, and D Pfeiffer.  Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization.  _Scientific reports_ , 9(1):1–9, 2019.
  * Böhle et al. [2019] Moritz Böhle, Fabian Eitel, Martin Weygandt, and Kerstin Ritter.  Layer-wise relevance propagation for explaining deep neural network decisions in MRI-based Alzheimer’s disease classification.  _Frontiers in aging neuroscience_ , 11:194, 2019.
  * Chlebus et al. [2019] Grzegorz Chlebus, Nasreddin Abolmaali, Andrea Schenk, and Hans Meine.  Relevance analysis of MRI sequences for automatic liver tumor segmentation.  _arXiv preprint arXiv:1907.11773_ , 2019.
  * Saporta et al. [2022] Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G Blankenberg, Andrew Y Ng, et al.  Benchmarking saliency methods for chest x-ray interpretation.  _Nature Machine Intelligence_ , 4(10):867–878, 2022.
  * Tibshirani [1996] Robert Tibshirani.  Regression shrinkage and selection via the lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 58(1):267–288, 1996.
  * Bergman and Schiffer [1951] Stefan Bergman and M Schiffer.  Kernel functions and conformal mapping.  _Compositio Mathematica_ , 8:205–249, 1951.
  * Lundberg et al. [2020] Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee.  From local explanations to global understanding with explainable AI for trees.  _Nature machine intelligence_ , 2(1):56–67, 2020\.
  * Agarap [2018] Abien Fred Agarap.  Deep learning using rectified linear units (RELU).  _arXiv preprint arXiv:1803.08375_ , 2018.
  * Rumelhart et al. [1986] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning representations by back-propagating errors.  _nature_ , 323(6088):533–536, 1986.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Davis and Goadrich [2006] Jesse Davis and Mark Goadrich.  The relationship between Precision-Recall and ROC curves.  In _Proceedings of the 23rd international conference on Machine learning_ , pages 233–240, 2006.
  * Hansell et al. [2008] David M Hansell, Alexander A Bankier, Heber MacMahon, Theresa C McLoud, Nestor L Muller, Jacques Remy, et al.  Fleischner society: glossary of terms for thoracic imaging.  _Radiology_ , 246(3):697, 2008.
  * Parekh et al. [2020] Maansi Parekh, Achala Donuru, Rashmi Balasubramanya, and Sangita Kapur.  Review of the chest CT differential diagnosis of ground-glass opacities in the COVID era.  _Radiology_ , 2020.
  * Sigakis et al. [2018] Christopher JG Sigakis, Susan K Mathai, Thomas D Suby-Long, Nicole L Restauri, Daniel Ocazionez, Tami J Bang, Carlos S Restrepo, Peter B Sachs, and Daniel Vargas.  Radiographic review of current therapeutic and monitoring devices in the chest.  _Radiographics_ , 38(4):1027–1045, 2018.
  * Simonyan et al. [2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Tjoa and Cuntai [2022] Erico Tjoa and Guan Cuntai.  Quantifying explainability of saliency methods in deep neural networks with a synthetic dataset.  _IEEE Transactions on Artificial Intelligence_ , 2022.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pages 1097–1105, 2012.
  * Alvarez-Melis and Jaakkola [2018] David Alvarez-Melis and Tommi S Jaakkola.  On the robustness of interpretability methods.  _arXiv preprint arXiv:1806.08049_ , 2018.
  * Slack et al. [2020] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju.  Fooling lime and shap: Adversarial attacks on post hoc explanation methods.  In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , pages 180–186, 2020.
  * Ghorbani et al. [2019] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 3681–3688, 2019.
  * Kindermans et al. [2019] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim.  The (un) reliability of saliency methods.  In _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_ , pages 267–280. Springer, 2019.
  * Jung et al. [2021] Yeon-Jee Jung, Seung-Ho Han, and Ho-Jin Choi.  SLRP: Improved heatmap generation via selective layer-wise relevance propagation.  _Electronics Letters_ , 57(10):393–396, 2021\.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pages 10347–10357. PMLR, 2021.
  * Chefer et al. [2021] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 782–791, 2021.
  * Alsentzer et al. [2019] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical bert embeddings.  _arXiv preprint arXiv:1904.03323_ , 2019.
  * Huang et al. [2019] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _arXiv preprint arXiv:1904.05342_ , 2019.
  * Qi et al. [2020] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  _arXiv preprint arXiv:2001.07966_ , 2020.