11institutetext: University of Twente, Netherlands 22institutetext: Philipps-
University Marburg, Germany 33institutetext: Hospital Group Twente (ZGT), The
Netherlands 44institutetext: University of Mannheim, Germany
44email<EMAIL_ADDRESS>44email:
<EMAIL_ADDRESS>44email:
<EMAIL_ADDRESS>
# Prototype-based Interpretable Breast Cancer Prediction Models: Analysis and
Challenges

Shreyasi Pathak 11 0000-0002-6984-8208 Jörg Schlötterer 2244
0000-0002-3678-0390 Jeroen Veltman 33 0000-0002-6824-3987 Jeroen Geerdink 33
0000-0001-6718-6653 Maurice van Keulen 11 0000-0003-2436-1372 Christin Seifert
22 0000-0002-6776-3868

###### Abstract

Deep learning models have achieved high performance in medical applications,
however, their adoption in clinical practice is hindered due to their black-
box nature. Using explainable AI (XAI) in high-stake medical decisions could
increase their usability in clinical settings. Self-explainable models, like
prototype-based models, can be especially beneficial as they are interpretable
by design. However, if the learnt prototypes are of low quality then the
prototype-based models are as good as black-box. Having high quality
prototypes is a pre-requisite for a truly interpretable model. In this work,
we propose a prototype evaluation framework for coherence (PEF-C) for
quantitatively evaluating the quality of the prototypes based on domain
knowledge. We show the use of PEF-C in the context of breast cancer prediction
using mammography. Existing works on prototype-based models on breast cancer
prediction using mammography have focused on improving the classification
performance of prototype-based models compared to black-box models and have
evaluated prototype quality through anecdotal evidence. We are the first to go
beyond anecdotal evidence and evaluate the quality of the mammography
prototypes systematically using our PEF-C. Specifically, we apply three state-
of-the-art prototype-based models, ProtoPNet, BRAIxProtoPNet++ and PIP-Net on
mammography images for breast cancer prediction and evaluate these models
w.r.t. i) classification performance, and ii) quality of the prototypes, on
three public datasets. Our results show that prototype-based models are
competitive with black-box models in terms of classification performance, and
achieve a higher score in detecting ROIs. However, the quality of the
prototypes are not yet sufficient and can be improved in aspects of relevance,
purity and learning a variety of prototypes. We call the XAI community to
systematically evaluate the quality of the prototypes to check their true
usability in high stake decisions and improve such models further.

###### Keywords:

Explainable AI Prototype-based models Breast cancer prediction Mammography.

##  1 Introduction

Deep learning has achieved high performance on various medical applications,
e.g. breast cancer prediction [17, 16]. However, its adoption to clinical
practice is hindered due to its black-box nature. Explainable AI (XAI) aims to
address this gap by explaining the reasoning of black-box models in a post-hoc
manner (post-hoc explainable methods) or developing models which are
interpretable by design (self-explainable methods).

Post-hoc explainable methods explain a trained deep neural network with a
separate explainer model, e.g. Grad-CAM can highlight the important image
regions to explain the prediction from a neural network [11]. However, these
explanations are not always reliable or trustworthy [12], as they come from a
second explainer model. Self-explainable models were developed to address this
issue – one of the most prominent ones being prototype-based models, e.g.
ProtoPNet [2].

Breast cancer prediction using mammography is a challenging medical task as
the regions-of-interest (ROIs) are small compared to the whole mammogram and
presence of dense breast tissue may make it difficult to read the mammogram.
Depending on the finding it can be difficult for a radiologist to decide
whether to perform a biopsy, wait for follow up or diagnose the lesion as
certainly benign. Learning relevant malignant and benign prototypes from the
training data and showing the reasoning behind a model’s prediction might
assist radiologists in taking data based decision. Existing works [21, 20]
have extended prototype-based model, ProtoPNet [2] on breast cancer prediction
using whole mammography images. [1] extended ProtoPNet for breast cancer
prediction on ROIs from mammography images. However, most works show anecdotal
evidence for the explanations. The quality of the learnt prototypes from
mammography images have not been extensively evaluated yet.

A recent survey [8] found that most XAI methods are usually evaluated based on
anecdotal evidence by selecting a few good examples. However, anectodal
evidence is not sufficient to judge the quality of the explanations. Nauta et
al. [8] proposed a framework of twelve properties for evaluating the quality
of explanations. One of the properties, Coherence, evaluates whether the
explanations are correct with respect to domain knowledge. In the medical
domain, it is highly important to evaluate the explanation quality in terms of
Coherence, before the model can be adopted in a clinical setting.

In this work, we go beyond anecdotal evidence and propose a prototype
evaluation framework for coherence (PEF-C) to quantitatively evaluate the
quality of prototypes and to provide a broader evaluation of prototype-based
models based on the domain knowledge. Specifically, we apply state-of-the-art
prototype based models on breast cancer prediction using mammography images
and evaluate the quality of the prototypes both quantitatively and
qualitatively. Our contributions are as follows:

  1. 1.

We propose PEF-C, a Prototype Evaluation Framework for Coherence, for
evaluating the quality of prototypes in prototype-based models. Our framework
can be used as a comprehensible and generalizable evaluation framework for
prototype-based models of medical applications.

  2. 2.

We reproduced a state-of-the-art breast cancer prediction model,
BRAIxProtoPNet++, which had no public source code. We release the source code
of the evaluation framework, the implementations of all black-
box111https://github.com/ShreyasiPathak/multiinstance-learning-mammography and
interpretable models222https://github.com/ShreyasiPathak/prototype-based-
breast-cancer-prediction for further research.

  3. 3.

We systematically compare prototype-based models and blackbox models on three
standard benchmark datasets for breast cancer detection, and show that
interpretable models achieve comparative performance. Our analysis of
prototype quality identifies differences between prototype-based models and
points towards promising research directions to further improve model’s
coherence.

##  2 Related Work

Prototype-based models. Prototype-based models were first introduced with
ProtoPNet [2], an intrinsically interpretable model with similar performance
to black-box models. ProtoTree [9] reduces the number of prototypes in
ProtoPNet and uses a decision tree structure instead of a linear combination
of prototypes as decision layer. ProtoPShare [14] and ProtoPool [13] also
optimize towards reducing the number of prototypes used in classification.
TesNet [22] disentangles the latent space of ProtoPNet by learning prototypes
on a Grassman manifold. PIP-Net [6] addresses the semantic gap between latent
space and pixel space for the prototypes, adds sparsity to the classification
layer and can handle out-of-distribution data. XProtoNet [4] is a prototype-
based model developed for chest radiography and achieved state-of-the-art
classification performance on a chest X-ray dataset.

Interpretability in Breast Cancer Prediction. [23] developed a expert-in-a-
loop interpretation method for interpreting and labelling the internal
representation of the convolutional neural network for mammography
classification. BRAIxProtoPNet++ [21] extended ProtoPNet for breast cancer
prediction achieving a higher performance compared to ProtoPNet and
outperforming black-box models in classification performance. InterNRL [20]
extended on BRAIxProtoPNet++ using a student-teacher reciprocal learning
approach. IAIA-BL [1] extended ProtoPNet with some supervised training on
fine-grained annotations and applied on ROIs instead of whole mammogram
images.

Evaluating XAI. Most XAI methods are usually evaluated using anecdotal
evidence [8]. However, the field is moving towards standardizing evaluation of
explanations [8, 7] similar to the standardized approach for evaluating
classification performance. In this work, we develop a prototype evaluation
framework for coherence (PEF-C) and use it in the context of breast cancer
prediction. We use ProtoPNet, BRAIxProtoPNet++ and PIP-Net on breast cancer
prediction using mammography images and evaluate on 3 public datasets - CBIS-
DDSM [15], VinDr [10] and CMMD [3] for classification performance. We further
use PEF-C to evaluate the learnt prototypes of the 3 models on CBIS-DDSM
dataset.

##  3 Preliminaries

In this section, we provide a summary of the state-of-the-art models used in
this paper to make this paper self-contained.

### Problem Statement

We pose breast cancer prediction using mammography as a classification
problem. We define it as a binary classification problem for the datasets
CBIS-DDSM and CMMD containing 2 classes, benign and malignant as labels and a
multiclass classification problem for the dataset, VinDr, containing the 5 BI-
RADS [18] categories as labels (cf. Sec. 5.1). Suppose,
$\\{(x_{1},y_{1}),$…$,(x_{n},y_{n}),$…$,(x_{N},y_{N})\\}\in\mathcal{X}\times\mathcal{Y}$
is a set of N training images with each image having a class label
$y_{n}\in\\{0,1\\}$ ($0$: benign, $1$: malignant) for binary classification
and a class label $y_{n}\in\\{0,1,2,3,4\\}$ ($0$: BI-RADS 1, $1$: BI-RADS 2,
$2$: BI-RADS 3, $3$: BI-RADS 4, $4$: BI-RADS 5) for multiclass classification.
Our objective is to learn the prediction function $f$ in
$\hat{y}_{n}=f(x_{n})$ where $\hat{y}_{n}$ is the predicted label.

### Black-box Models

We used pretrained ConvNext [5] and EfficientNet [19] models and changed the
classification layer to 2 classes and 5 classes with a softmax activation
function. GMIC [17] is a state-of-the-art breast cancer prediction model
containing a global module with feature extractor for learning the features
from a whole image, a local module with feature extractor for learning the
features from the patches and a fusion module to concatenate the global and
local features. The global module generates a saliency map, which is used to
retrieve the top-k ROI candidates. Local module takes these ROI candidates
(patches) as input and aggregates the features using a multi-instance learning
method. The global, local and fusion features are separately used for
prediction and passed to the loss function for training GMIC. During
inference, only fusion features are used for prediction. We used their model
for binary classification with 1 neuron in the classification layer with a
sigmoid activation function. For multiclass classification, we used their
setup of multilabel classification and replaced sigmoid with a softmax
activation function.

### Prototye-based Models

ProtoPNet [2] consists of a convolutional feature extractor backbone, a layer
with prototype vectors and a classification layer. The feature extractor
backbone generates a feature map of $H\times W\times D$ where H, W, and D are
height, width and depth of the feature map. A prototype vector of size
$1\times 1\times D$ is passed over $H\times W$ spatial locations to find the
patch in the feature map that is most similar to the prototype vector. The
prototype layer contains equal number of prototype vectors for each class. In
the classification layer, the prototype vectors are connected to their own
class with a weight of 1 and to other classes with a weight of -0.5. After the
most similar patch for each prototype is saved, the patch is visualized in the
pixel space using upsampling.

BRAIxProtoPNet++ [21] extends on ProtoPNet to improve the classification
performance by including a global network and distilling the knowledge from
the global network to the ProtoPNet. It also increases the prototype diversity
by taking all prototypes from different training images.

PIP-Net [6] does not use a separate prototypical layer like ProtoPNet and
BRAIxProtoPNet++, but uses regularization to make the features in the last
layer of the convolutional backbone interpretable. For a feature map of
dimension $H\times W\times D$, the model can have a maximum of $D$ prototypes.
Each patch of dimension $1\times 1\times D$ in the feature map is softmaxed
over $D$ such that one patch tries to match to maximally one prototype. The
maximum similarity score is taken from each of the $D$ channels, which is then
used in the classification layer. Further, the classification layer is
optimized for sparsity to keep only the positive connections towards a
decision.

Table 1: Overview of prototype evaluation measures. Global prototypes (GP) is the number of all prototypes, relevant prototypes (RP) contain ROIs. Categories refer to fine-grained part annotations from a lexicon (abnormality type and BIRADS descriptors in our application case). Total number of categories (TC) are all possible categories from the lexicon, unique categories (UC) is the size of the set of categories found by the model. Level refers to whether the measure is calculated at the local (L), i.e., at the image-level or global (G) model level. Property |  Question | Level | Measure
---|---|---|---
Compactness |  What is the explanation size? | G | $GP$
|  | L | $LP$
Relevance |  How many of the learnt prototypes are relevant for diagnosis? | G | $|RP|/GP$
Specialization |  Are the prototypes meaningful enough to be assigned names of abnormality categories? | G |  Purity${}_{RP}/|RP|$
Uniqueness |  Does the model activate more than one prototype for one abnormality category? | G | $UC_{RP}/|RP|$
Coverage |  How many abnormality categories were the model able to learn? | G | $UC_{RP}/TC$
Localization |  To what extent is the model able to find correct ROIs? | L | IoU, DSC
Class-specific |  Is the model associating the prototypes with the correct class? | G | $\frac{Align(W_{RP},CD_{C_{RP}})}{|RP|}$

##  4 Prototype Evaluation Framework

We developed a Prototype Evaluation Framework for Coherence (PEF-C) to i)
quantitatively evaluate the quality of the learnt prototypes and ii) provide a
broader assessment of the prototype-based models, based on domain knowledge.
We use our PEF-C to assess the status quo of the prototype-based models on
breast cancer prediction using mammography. We evaluate the following 7
properties, summarized in Table 1:

  1. 1.

Compactness measures the size of the local and global explanation of the
model. The lower the explanation size, the easier it is to comprehend for
users. Note that while this property is associated with the presentation
aspect of the explanations and not with coherence [8], we decided to include
it here as the measurement of further properties is based on the explanation
size. We calculate this as follows:
Global prototypes (GP): Total number of prototypes with non-zero weights to
the classification layer.
Local prototypes (LP): Average number of prototypes that are active per
instance, i.e., whose contribution to the classification decision (prototype
presence score $p$ times weight $w$ to the classification layer) is non-zero.

  2. 2.

Relevance measures the number of prototypes associated with the ROIs among the
whole set of prototypes. The larger the number of relevant prototypes, the
better the capability of the model in finding the relevant regions. We measure
this property by calculating the ratio between the number of prototypes
activated on at least one ROI (relevant prototypes (RP)) and the number of
global prototypes:

$Rel.=\frac{|RP|}{GP}$

Specifically, to determine $RP$, we take the top-k instances with highest
prototype presence scores for each prototype (i.e., images, where the
prototype is most activated). If the corresponding patch for the prototype in
any of these top-k instances overlaps with the center of a ROI in this image,
we consider it relevant. We set $k=10$ to focus on the most representative
instances, but still allow a bit of variation (outliers).

  3. 3.

Specialization measures the purity of prototypes by matching the extent of its
alignment to lexical categories. A truly interpretable prototype represents a
single semantic concept that is comprehensible to humans, i.e., can be named
or given a meaning by users. For example in our case, a prototype is not fully
interpretable if it gets activated both on mass or calcification, as we don’t
know whether to call this prototype a mass prototype or a calcification
prototype. The higher the purity, the more interpretable the prototypes.

$Spec.=\frac{\sum_{p\in RP}Purity_{p}}{|RP|}$

where $RP$ is the set of relevant prototypes as defined above and $Purity_{p}$
is the share of the majority category in top-k patches (relevant prototypes
overlap with an ROI center and each ROI is labelled with a category). If all
top-k patches belong to the same category, the prototype is pure. Again,
$k=10$ from before. For a detailed evaluation by category, we assign the
majority category and report purity scores per category. If the category
lexicon constitutes a hierarchy like in our case, we can determine purity also
on more fine-grained category levels. In our application case, the top-level
categories are abnormality types (mass and calcification) and sub-categories
constitute fine-grained BIRADS descriptors, e.g., shape or margin of a mass
and morphology or distribution of a calcification.

  4. 4.

Uniqueness measures the number of unique categories of prototypes among all
relevant prototypes.

$Uniq.=\frac{UC_{RP}}{|RP|}$

where $UC_{RP}$ stands for unique categories (UC) of all relevant prototypes.
Each prototype has a single (majority) category assigned from the purity
calculation of the Specialization measure before. Ideally, we want each
prototype to represent a different, unique category and not multiple
prototypes that represent the same. The uniqueness measure should always be
evaluated in combination with relevance and coverage (next measure), as
trivially, a single relevant prototype would yield the maximum uniqueness
score of 1.

  5. 5.

Coverage measures the fraction of all categories in the dataset that are
represented by prototypes.

$Cov.=\frac{UC_{RP}}{TC}$

where $TC$ is the total number of categories.
Ideally, a model has at least one prototype associated with each category,
such that it is able to incorporate all variations in its reasoning process.

  6. 6.

Localization measures the capability of the model in finding the correct ROIs
through the activated prototypes.

$Loc.=\frac{\sum_{t=1}^{T}IoU(ROI_{t},Patch_{t})}{T}$

where IoU is the intersection over union (IoU) over groundtruth ROI and the
patch activated by the prototype, $t$ stands for the $t^{th}$ test instance of
the total $T$ test instances. IoU can also be replaced with dice similarity
coefficient (DSC). We calculate and report both in our experiments.
This property is a local measure, meaning that it’s calculated per test
instance and averaged over the whole test set. The higher the localization,
the better the model is at finding the correct ROIs.

  7. 7.

Class-specific measures the extent to which prototypes are associated with the
correct class. For example, a prototype that has an assigned category (cf.
Purity calculation in Specialization measure) that is more often found to be
benign in our dataset, should have a higher weight to the benign class in the
classification layer. A high class-specific score means the model is good at
associating categories with the correct class. We measure this property by the
following:

$Cl\text{-}spec.=\frac{\sum_{p\in RP}Align(W_{p},CD_{C_{p}})}{|RP|}$

where $Align$ stands for the alignment between the weights of a relevant
prototype towards the classes in the classification layer, $W_{RP}$, and the
class distribution (CD) of the category (C) assigned to this prototype,
$CD_{C_{RP}}$. $Align$ is a binary assessment: If a prototype has the highest
weight to the majority class of its category, the score is 1 and 0 otherwise.
The majority class is obtained from ROI category annotations in the dataset
(category class distribution $CD_{C_{RP}}$).

##  5 Experimental Setup

In this section, we describe the datasets, training details, setup of our
prototype evaluation framework and the visualization method of the prototypes.

###  5.1 Datasets

We performed our experiments on three commonly used mammography datasets:
CBIS-DDSM [15] is a public dataset containing 3,103 mammograms from 1,566
patients including mediolateral oblique (MLO) and craniocaudal (CC) views. The
dataset contains 1,375 malignant and 1,728 benign images. For our experiments,
we used the official train (2458 images) and test split (645 images) provided
by the dataset.
CMMD [3] is a Chinese public dataset collected between years 2012 and 2016,
containing 5,202 mammograms of CC and MLO views from 1,775 patients. It
contains 2,531 malignant and 2,570 benign images. We split the dataset in a
patient-wise manner into training set (3,199 images, D1-0001 to D2-0247) and
test set (2,002 images, D2-0248 to D2-0749) following [20] for fair comparison
to their results.
VinDr-Mammo [10] is a public dataset from two hospitals in Vietnam, containing
5,000 mammography cases (20,000 mammogram images) including CC and MLO views.
The images are annotated with BIRADS category 1 to 5 with 1 being normal
mammogram and 5 being most malignant. We used the official training (16,000
images) and test (4,000 images) split of the dataset for our experiments.

###  5.2 Training Details

We split the training set patient-wise into 90% train and 10% validation for
all datasets. We performed hyperparameter tuning on CMMD (and when needed also
on CBIS-DDSM) dataset and the hyperparameters with the highest validation AUC
were selected as the hyperparameter for our experiments on all datasets. The
models were trained till a certain epoch (details in the following paragraph)
and he model of the epoch with the highest validation AUC was selected for
test set inference.

EfficientNet-B0 [19] was trained with a fixed learning rate (lr) of 0.001 and
weight decay (wd) of 1e-5 for 60 epochs. ConvNext [5] was trained with a fixed
lr=1e-5 and wd=1e-5 for 50 epochs. GMIC [17] was trained with a fixed learning
rate $\approx$ 6e-5 and weight decay $\approx$ 3e-4 (exact values can be found
in our repository1) for 50 epochs. ProtoPNet [2] was trained in 3 stages - i)
warm optimizer stage where only the add on layer and the prototype vectors are
trained with lr=1e-4 for 5 epochs, ii) fine tuning stage where the whole
network except the last layer is trained for 55 epochs with lr=1e-5 for the
backbone, lr=1e-4 for the add on layers and the prototype vector, iii) the
last layer is trained every 11th epoch for 20 iterations with lr=1e-2.
BRAIxProtoPNet++ [21] was trained in 3 stages - i) GlobalNet training where
the GlobalNet and its classification layer was trained with lr=1e-5 for 3
epochs, ii) ProtoPNet training where the add on layer, the prototype vector
and the ProtoPNet classification layer was trained with lr=1e-5 for 3 epochs,
and iii) whole model fine-tuning stage, where all the layers were trained with
lr=1e-5 for the backbone, lr=1e-4 for the add on layer and the prototype
vector and lr=1e-2 for the classification layers for 54 epochs. This schedule
of BRAIxProtoPNet++ was followed for CBIS-DDSM dataset, however, for CMMD and
VinDr, the (iii) stage used a fixed lr=1e-5 for all layers. In PIP-Net [6],
the feature extractor backbone was trained with a fixed lr=0.00001 and the
classifier layer was trained with a lr=0.05 in a cosine annealing with warm
restart schedule. PIP-Net was pretrained in a self-supervised manner for 10
epochs, followed by full finetuning for 60 epochs following [6].

All images were resized to 1536x768 following [21, 20]. Data augmentation of
random horizontal flipping, shearing, translation and rotation was applied
(details can be found in our repository).

We report the average performance and standard deviation over three random
weight initializations with different seeds.

###  5.3 Prototype Evaluation Framework Setup

We calculated the seven properties of PEF-C for the three prototype-based
models, ProtoPNet, BRAIxProtoPNet++ and PIP-Net. All analysis were perfomed on
CBIS-DDSM, because it is the only dataset with fine-grained annotations of
abnormality types – mass and calcification, mass BIRADS descriptors - shape
and margin, and calcification BIRADS descriptors - morphology and
distribution. We took the top-$k$, where $k=10$ similar patches from the
training set for each prototype to calculate the relevance, specialized,
uniqueness and coverage.

For purity calculation in specialization, we define categories following the
hierarchical level of BIRADS lexicon – at the top level, abnormality type is
the category, i.e. mass or calcification. At a sub-level, mass abnormalities
(lesions) can be described with or categorized by shape and margin
descriptors. At the same level, calcification abnormalities can be described
with or categorized by a morphology and distribution. For uniqueness, we
define categories by combining the descriptors at all levels. A category for
mass is defined with a combination of abnormality type - shape - margin and
for calcification, with a combination of abnormality type - morphology -
distribution. For example, an abnormality category of a mass can be mass-oval-
circumscribed, i.e. a mass with oval shape and circumscribed margin, and an
abnormality category for a calcification can be calcification-pleomorphic-
segmental, i.e. a calcification with pleomorphic morphology and segmental
distribution. Following this rule, we found that the total abnormality
categories in the CBIS-DDSM is 132 (70 mass categories, and 62 calcification
categories) and we used this value for calculation of coverage property.

For calculating the class-specific measure, we selected the abnormality
categories that have instances for both malignant and benign classes in the
dataset.

Localization: We calculate the intersection over union (IoU) and Dice
Similarity Coefficient (DSC) for the top-1 (IoU1, DSC1), top-10 (IoU10, DSC10)
and all (IoUAll, DSCAll) activated prototypes with the annotated ROI. For
selecting the top-1 and top-10 prototypes, we score the prototypes using the
absolute magnitude of the similarity score $\times$ weight of the prototype
with the groundtruth class.

Table 2: Performance comparison of SOTA breast cancer prediction models. Showing mean and standard deviation; values normalized to percentages.  | CBIS | VinDr | CMMD
---|---|---|---
| F1 | AUC |  F1wM |  AUCM | F1 | AUC
Black box modles
EfficientNet [19] | $58\pm 6$ | $76\pm 1$ | $65\pm 4$ | $77\pm 1$ | $73\pm 3$ | $83\pm 1$
ConvNext [5] | $67\pm 3$ | $\textbf{80}\pm 2$ | $\textbf{72}\pm 1$ | $\textbf{83}\pm 0$ | $\textbf{74}\pm 2$ | $84\pm 1$
GMIC [17] | $\textbf{68}\pm 3$ | $\textbf{80}\pm 2$ | $64\pm 3$ | $80\pm 1$ | $72\pm 5$ | $83\pm 1$
Part-prototpe models with ConvNext backbone
ProtoPNet [2] | $65\pm 1$ | $77\pm 1$ | $58\pm 13$ | $79\pm 1$ | $70\pm 0$ | $83\pm 1$
BRAIxProtoPNet++ [21] | $64\pm 4$ | $79\pm 1$ | $63\pm 11$ | $81\pm 0$ | $72\pm 3$ | $\textbf{86}\pm 2$
PIP-Net [6] | $63\pm 3$ | $75\pm 2$ | $63\pm 3$ | $78\pm 1$ | $70\pm 1$ | $81\pm 1$

###  5.4 Visualization of prototypes

We visualize the local and global explanations of ProtoPNet, BRAIxProtoPNet++
and PIP-Net on CBIS-DDSM and CMMD dataset for qualitative evaluation. We
randomly select a test instance for the local explanation and show the top-3
activated prototypes. For global explanation, we show the top-10 image patches
for each of eight reasonable looking prototypes. ProtoPNet and PIP-Net have
different approaches for visualizing the prototypes. ProtoPNet does not have a
fixed patch size. It takes the $x$ percentile from the upsampled similarity
map to visualize the patch corresponding to the prototype. PIP-Net has a fixed
patch size for prototype visualization. It takes the highest similarity score
from the feature map, upsamples it to the patch size in the image space and
uses that patch to visualize the prototype. For local visualization, we used
the visualization method specific to each model, where we set percentile as
99.9 for ProtoPNet and BraixProtoPNet++ to select the relevant region and for
PIP-Net, we set the patch size as $130\times 130$. However, we used a fixed
patch size of $130\times 130$ for visualization of global explanation for all
models. We also used the same fixed patch size for calculation of purity
across all models for a fair comparison of the scores.

##  6 Results and Discussion

In this section, we report the classification performance of black-box vs
prototype-based models, the qualitative evaluation of the prototypes through
local and global explanation and the quantitative evaluation of the prototypes
using our prototype evaluation framework for coherence.

###  6.1 Performance Comparison of Black-box vs Prototype-based Models

We compared the classification performance of three intrinsically
interpretable prototype-based models (ProtoPNet [2], BRAIxProtoPNet++ [21] and
PIP-Net [6]), and three black-box model (EfficientNet [19], ConvNext [5] and
GMIC [17]) (cf. Table 2). ConvNext achieves the highest performance (F1 score)
for VinDr (F1: 0.72) and CMMD (F1: 0.74) dataset and GMIC achieves the highest
performance (F1: 0.68) for CBIS, with ConvNext having competitive performance
(F1: 0.67). However, prototype-based models have competitive performance with
black-box models, with BRAIxProtoPNet++ achieving highest AUC (0.86) on CMMD.
BRAIxProtoPNet++ outperforms ProtoPNet and PIP-Net in terms of F1 score on all
datasets. Note that though PIP-Net has a lower AUC score compared to ProtoPNet
and BRAIxProtoPNet++, the F1 score is competitive with the other models even
with a much lower number of activated prototypes. BRAIxProtoPNet++ did not
have a public code. We implemented their model and reach a nearby performance
to the ones reported in [20] on CMMD (BRAIxProtoPNet++
AUC${}^{\text{Reported}}$ 88% vs AUCOurs 86%; ProtoPNet
AUC${}^{\text{Reported}}$ 85% vs AUCOurs 84%).

###  6.2 Local and Global Visualization of Prototypes

For global explanation, we visualized the top-10 image patches (based on the
highest similarity score) activated by each prototype for the 3 prototype-
based models for CBIS-DDSM (cf. Fig. 1) and CMMD dataset (cf. Fig 2). We can
see prototypes representing the mass abnormality in the visualization from
CBIS-DDSM. In ProtoPNet global explanation (cf. Fig 1(a)), the prototype in
the second row contains some patches with mass ROI of irregular shape and
spiculated margin and the fourth row contains some mass ROIs of oval shape and
ill-defined margin, among other different looking patches. This shows that all
patches activated by a prototype in ProtoPNet may not be visually similar.
Further, separate prototypes may learn the same abnormality category as shown
in Fig 1(b), where prototypes in 5th, 6th and 7th row all contain mass ROIs of
irregular shape and spiculated margin. Further, the same prototype can get
activated for different abnormality categories. For example, in Fig. 1(c),
prototype in 4th row contains most ROIs of category mass with irregular shape
and spiculated margin. However, that prototype also contains ROIs of category
calcification with pleomorphic and amorphous morphology and segmental
distribution. Further, there are prototypes representing normal tissue and
black background in all prototype-based models. This is expected, however,
such prototypes should have lower weights to the classes as they usually do
not have much contribution to benign and malignant prediction. In general, it
can be observed that i) duplicate prototypes might be learnt by the models
(quite easily visually observed in ProtoPNet (cf. Fig 2(a))), ii) all top-10
patches activated by one prototype may not look visually the same or may not
belong to the same abnormality category, iii) not all learnt prototypes belong
to the ROIs, but can belong to edges and background with reasonable weights in
the classification layer.

(a) ProtoPNet

(b) BRAIxProtoPNet++

(c) PIP-Net

Figure 1: Global visualization of 3 prototype-based models trained on CBIS-
DDSM dataset. Each row represents one prototype visualized with the top-10
activated image patches.

(a) ProtoPNet

(b) BRAIxProtoPNet++

(c) PIP-Net

Figure 2: Global visualization of 3 prototype-based models trained on CMMD
dataset. Each row represents one prototype visualized with the top-10
activated image patches.

We visualized local explanation for one test instance from CMMD dataset for
ProtoPNet (cf. Fig. 3), BRAIxProtoPNet++ (cf. Fig. 4) and PIP-Net (cf. Fig. 5)
as an anecdotal evidence. We show the top-3 activated prototypes for each
model. For ProtoPNet (cf. Fig. 3), we observed that i) one of the activated
prototypes contained the correct ROI, ii) however, the top activated
prototypes also represented some irrelevant black region and edges, iii) the
test image patch does not always look similar to the prototype from the
training set, and iv) similar looking prototype may get activated for both
benign and malignant classes. For BRAIxProtoPNet++ (cf. Fig. 4), we observed
that i) the region surrounding the actual ROI gets activated rather than the
actual ROI, ii) most of the activated prototypes point to the same region (we
have only shown for 3, but we found other prototypes to also point to the same
region as the top 3), and iii) the activated prototypes of the benign class
(not predicted class) represent the background. For PIP-Net, we observed that
i) the top activated prototypes contain the ROIs, ii) the test image patch has
some similarity to the prototype from the training set, and iii) the benign
prototypes don’t belong to the ROI, but to the other regions of the breast.
Overall, the local explanation of PIP-Net for this test image looked more
reasonable compared to the other models.

(a) Prototypes activated for malignant class

(b) Prototypes activated for benign class

Figure 3: Local explanation from ProtoPNet showing the top-3 activated
prototypes for the malignant class and the benign class. Example image: CMMD,
malignant test case D2-0249, view RCC, predicted class malignant

(a) Prototypes activated for malignant class

(b) Prototypes activated for benign class

Figure 4: Local explanation from BRAIxProtoPNet++ showing the top-3 activated
prototypes for the malignant class and the benign class. Example image: CMMD,
malignant test case D2-0249, view RCC, predicted class malignant

(a) Prototypes activated for malignant class

(b) Prototypes activated for benign class

Figure 5: Local explanation from PIP-Net showing the top-3 activated
prototypes for the malignant class and the benign class. Example image: CMMD,
malignant test case D2-0249, view RCC, predicted class malignant

###  6.3 Automatic Quantitative Evaluation of Prototypes

We report the results of our prototype evaluation framework in Table 3 for
ProtoPNet, BRAIxProtoPNet++ and PIP-Net on CBIS-DDSM dataset. We observe that
PIP-Net has much lower number of global and local prototypes increasing the
sparsity of the model (compactness). ProtoPNet has higher relevance score
compared to the others suggesting more prototypes get activated on ROIs,
however, the standard deviation is quite high reducing the reliability of the
score. Specialization scores show that ProtoPNet has the highest purity at the
first granularity level of abnormality type, however, PIP-Net has the highest
purity in the second granularity level. This shows that though ProtoPNet is
better at learning prototypes representing only mass or calcification, PIP-Net
is better at learning prototypes representing a particular type of mass or
calcification, which is in line with the ‘semantic correspondence’
characteristic of the model. Further, PIP-Net has the highest uniqueness score
suggesting that most relevant prototypes belong to a unique category. However,
the high uniqueness score is also an effect of the low global prototypes. For
example, in one run of PIP-Net, 16 out of 48 global prototypes were relevant
and of these 16, 10 were unique, resulting in a high uniqueness score of
0.625. Whereas in one run of BRAIxProtoPNet++, 135 out of 400 global
prototypes were relevant, of which 33 were unique resulting in a lower
uniqueness score of 0.24. Therefore, it is better to look at the uniqueness
measure together with the coverage score, which shows BRAIxProtoPNet++ to have
a higher coverage than PIP-Net due to higher unique categories. We also
measured class-specific score, i.e. how much do the class weights of the
relevant prototypes (associated with a category) align with the class
distribution of that category from the dataset. PIP-Net has the highest class-
specific score suggesting that more relevant prototypes (representing the
ROIs) are associated the correct class compared to the other models. However,
PIP-Net also has a low number of relevant prototypes, compared to the others,
which can result in a higher class-specific score. Lastly, the localization
score shows that PIP-Net is better in finding the ROI with the top-1 activated
prototype. At top-10 activated prototypes, ProtoPNet has similar IoU to PIP-
Net and with all activated prototypes, BRAIxProtoPNet++ has the highest IoU
due to its high number of global prototypes (400) compared to PIP-Net (90).
The IoU6 score of the black-box model, GMIC, is similar to the IoU1 of the
prototype based models and with only 10 activated prototypes, the prototype
based models outperform GMIC is finding the correct ROI. This shows that
prototype-based models can be a good candidate for unsupervised ROI
extraction.

Table 3: Comparison of prototype-based models using our prototype evaluation
framework on CBIS-DDSM dataset. Mean and standard deviation reported across
model runs with different seeds.

Property | ProtoPNet | BRAIxProtoPNet++ | PIP-Net
---|---|---|---
Compactness |  |  |
Global | $400\pm 0.0$ | $400\pm 0.0$ | $90\pm 83$
Local |  |  |
Positive | $314\pm 20$ | $200\pm 0$ | $27\pm 31$
Negative | $84\pm 20$ | $200\pm 0$ | -
Sparsity | 0% | 0% | 70%
Relevance | $\textbf{0.36}\pm 0.23$ | $0.30\pm 0.07$ | $0.28\pm 0.05$
Specialization |  |  |
Abnorm. Type | $\textbf{0.37}\pm 0.02$ | $0.22\pm 0.09$ | $0.31\pm 0.13$
Mass Shape | $0.29\pm 0.04$ | $0.29\pm 0.07$ | $\textbf{0.35}\pm 0.08$
Mass Margin | $0.34\pm 0.03$ | $0.33\pm 0.10$ | $\textbf{0.38}\pm 0.04$
Calc. Morph. | $0.30\pm 0.22$ | $0.24\pm 0.06$ | $\textbf{0.32}\pm 0.09$
Calc. Distribution | $0.24\pm 0.11$ | $0.24\pm 0.03$ | $\textbf{0.27}\pm 0.09$
Uniqueness | $0.13\pm 0.07$ | $0.31\pm 0.08$ | $\textbf{0.53}\pm 0.12$
Coverage | $0.12\pm 0.05$ | $\textbf{0.25}\pm 0.02$ | $0.10\pm 0.08$
Class-specific | $0.52\pm 0.05$ | $0.61\pm 0.03$ | $\textbf{0.65}\pm 0.08$
Localization |  |  |
IoU1 | $0.04\pm 0.01$ | $0.04\pm 0.02$ | $\textbf{0.07}\pm 0.00$
IoU10 | $\textbf{0.13}\pm 0.02$ | $0.10\pm 0.01$ | $\textbf{0.13}\pm 0.01$
IoUAll | $0.23\pm 0.03$ | $\textbf{0.28}\pm 0.01$ | $0.18\pm 0.06$
DSC1 | $0.06\pm 0.02$ | $0.06\pm 0.03$ | $\textbf{0.10}\pm 0.0$
DSC10 | $\textbf{0.20}\pm 0.03$ | $0.15\pm 0.02$ | $0.19\pm 0.01$
DSCAll | $0.34\pm 0.04$ | $\textbf{0.39}\pm 0.02$ | $0.26\pm 0.09$

  * Note: The black-box model, GMIC has an IoU of $0.05\pm 0.0$ and DSC of $0.09\pm 0.0$ on CBIS-DDSM over 6 ROI candidates that are extracted from the GMIC model.

##  7 Conclusion and Future Work

We extensively compare state-of-the-art black-box models to state-of-the-art
prototype-based models in the context of breast cancer prediction using
mammography. We found that though black-box model, ConvNext has higher
classification performance compared to prototype-based models, prototype-based
models are competitive in performance. To go beyond anecdotal evidence in
evaluating the quality of the prototypes, we propose a prototype evaluation
framework for coherence (PEF-C) to quantitatively evaluate the prototype
quality based on domain knowledge. Our framework needs regions-of-interest
(ROIs) annotation and some fine-grained labels for automatic evaluation of
prototype quality. We used such annotations from a public dataset, CBIS-DDSM
and evaluate our framework on 3 prototype-based models. We found that around
30% prototypes were relevant, prototypes were around 37% pure at the first
granularity level of mass and calcification and 35% to 27% pure at the second
granularity level of a specific type of mass or calcification. We also found
that learnt prototypes covered around 25% of the total abnormality categories
from the dataset (categories from the BI-RADS lexicon) and around 60-65% of
the relevant prototypes are associated with the correct class. Further,
prototype-based model outperformed the black-box model, GMIC, in localizing
the ROIs, suggesting that prototype-based models can be a good candidate for
unsupervised region of interest extraction. However, the intersection over
union scores for top-10 activated prototypes are quite low (around 0.13). This
suggests that we still need to improve on learning relevant prototypes which
can get activated on all ROIs of the same category. We found slightly higher
scores for PIP-Net w.r.t some properties, but PIP-Net also had a much lower
number of learnt prototypes resulting in low coverage score. Having lower
number of prototypes (compactness) makes the explanation easier for users to
comprehend, but we need to increase the number of unique and relevant
prototypes learnt by the model to improve the model’s detection of various
ROIs.

In conclusion, our analysis shows that prototype-based models still need to
improve on i) purity of the prototypes, ii) reducing irrelevant prototypes,
iii) learning more unique prototypes such that the prototypes can cover more
abnormality categories and iv) finding the ROIs more accurately. In the
future, it would be also interesting to perform user evaluation of the learnt
prototypes to assess their quality with domain-experts. Our vision is to
integrate the interpretable model in the clinical workflow for the purpose of
assisting the clinicians with diagnosis. We envision a two-phase approach: a
bootstrap-phase where clinicians study the prototypes of the global
explanation and discuss their meaning giving them a name and a description;
and a use-and-tune phase where clinicians during their everyday diagnosis see
and amend the local explanation along with the prototype names and
descriptions. In this way, the documentation of the meaning of the prototypes
and thereby the quality of the model can gradually improve in a natural
manner.

## References

  * [1] Barnett, A.J., Schwartz, F.R., Tao, C., Chen, C., Ren, Y., Lo, J.Y., Rudin, C.: A case-based interpretable deep learning model for classification of mass lesions in digital mammography. Nature Machine Intelligence 3(12), 1061–1070 (2021)
  * [2] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., Su, J.K.: This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems 32 (2019)
  * [3] Cui, C., Li, L., Cai, H., Fan, Z., Zhang, L., Dan, T., Li, J., Wang, J.: The chinese mammography database (cmmd): An online mammography database with biopsy confirmed types for machine diagnosis of breast. (version 1) [data set] (2021), https://doi.org/10.7937/tcia.eqde-4b16, the Cancer Imaging Archive, Accessed: 08/09/2023
  * [4] Kim, E., Kim, S., Seo, M., Yoon, S.: Xprotonet: diagnosis in chest radiography with global and local explanations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15719–15728 (2021)
  * [5] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022)
  * [6] Nauta, M., Schlötterer, J., van Keulen, M., Seifert, C.: Pip-net: Patch-based intuitive prototypes for interpretable image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2744–2753 (2023)
  * [7] Nauta, M., Seifert, C.: The co-12 recipe for evaluating interpretable part-prototype image classifiers. In: World Conference on Explainable Artificial Intelligence. pp. 397–420. Springer (2023)
  * [8] Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., Seifert, C.: From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing Surveys 55(13s), 1–42 (2023)
  * [9] Nauta, M., Van Bree, R., Seifert, C.: Neural prototype trees for interpretable fine-grained image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14933–14943 (2021)
  * [10] Nguyen, H.T., Nguyen, H.Q., Pham, H.H., Lam, K., Le, L.T., Dao, M., Vu, V.: Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. medRxiv (2022). https://doi.org/10.1101/2022.03.07.22272009
  * [11] Oh, Y., Park, S., Ye, J.C.: Deep learning covid-19 features on cxr using limited training data sets. IEEE transactions on medical imaging 39(8), 2688–2700 (2020)
  * [12] Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence 1(5), 206–215 (2019)
  * [13] Rymarczyk, D., Struski, Ł., Górszczak, M., Lewandowska, K., Tabor, J., Zieliński, B.: Interpretable image classification with differentiable prototypes assignment. In: European Conference on Computer Vision. pp. 351–368. Springer (2022)
  * [14] Rymarczyk, D., Struski, Ł., Tabor, J., Zieliński, B.: Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. pp. 1420–1430 (2021)
  * [15] Sawyer-Lee, R., Gimenez, F., Hoogi, A., Rubin, D.: Curated breast imaging subset of digital database for screening mammography (cbis-ddsm) (version 1) [data set] (2016), https://doi.org/10.7937/K9/TCIA.2016.7O02S9CY, accessed: 28/04/2022
  * [16] Shen, L., Margolies, L.R., Rothstein, J.H., Fluder, E., McBride, R., Sieh, W.: Deep learning to improve breast cancer detection on screening mammography. Scientific reports 9(1), 1–12 (2019)
  * [17] Shen, Y., Wu, N., Phang, J., Park, J., Liu, K., Tyagi, S., Heacock, L., Kim, S.G., Moy, L., Cho, K., et al.: An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization. Medical image analysis 68, 101908 (2021)
  * [18] Sickles, E.A., D’Orsi, C.J., Bassett, L.W., Appleton, C.M., Berg, W.A., Burnside, E.S., et al.: Acr bi-rads® mammography. ACR BI-RADS® atlas, breast imaging reporting and data system 5, 2013 (2013)
  * [19] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)
  * [20] Wang, C., Chen, Y., Liu, F., Elliott, M., Kwok, C.F., Peña-Solorzano, C., Frazer, H., McCarthy, D.J., Carneiro, G.: An interpretable and accurate deep-learning diagnosis framework modelled with fully and semi-supervised reciprocal learning. IEEE Transactions on Medical Imaging (2023)
  * [21] Wang, C., Chen, Y., Liu, Y., Tian, Y., Liu, F., McCarthy, D.J., Elliott, M., Frazer, H., Carneiro, G.: Knowledge distillation to ensemble global and interpretable prototype-based mammogram classification models. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 14–24. Springer (2022)
  * [22] Wang, J., Liu, H., Wang, X., Jing, L.: Interpretable image recognition by constructing transparent embedding space. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 895–904 (2021)
  * [23] Wu, J., Peck, D., Hsieh, S., Dialani, V., Lehman, C.D., Zhou, B., Syrgkanis, V., Mackey, L., Patterson, G.: Expert identification of visual primitives used by cnns during mammogram classification. In: Medical Imaging 2018: Computer-Aided Diagnosis. vol. 10575, pp. 633–641. SPIE (2018)