# MetaCAM: Ensemble-Based Class Activation Map

Emily Kaczmareka,*, Olivier X. Miguelb, Alexa C. Bowieb, Robin Ducharmeb,
Alysha L.J. Dingwall-Harveya,b,
Steven Hawkena,b,c,g, Christine M. Armoura,i,j, Mark C. Walkera,b,c,d,e,f,g,
Kevin Dicka,*
a Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
b Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa,
Canada
c School of Epidemiology and Public Health, University of Ottawa, Ottawa,
Canada
d Department of Obstetrics and Gynecology, University of Ottawa, Ottawa,
Canada
e International and Global Health Office, University of Ottawa, Ottawa, Canada
f BORN Ontario, Children’s Hospital of Eastern Ontario, Ottawa, Canada
g ICES, Toronto, Canada
h Department of Obstetrics, Gynecology & Newborn Care, The Ottawa Hospital,
Ottawa, Canada
i Department of Pediatrics, University of Ottawa, Ottawa, Canada
Institution1 address
j Prenatal Screening Ontario, Better Outcomes Registry & Network, Ottawa,
Canada
*{ekaczmarek<EMAIL_ADDRESS>
###### Abstract

The need for clear, trustworthy explanations of deep learning model
predictions is essential for high-criticality fields, such as medicine and
biometric identification. Class Activation Maps (CAMs) are an increasingly
popular category of visual explanation methods for Convolutional Neural
Networks (CNNs). However, the performance of individual CAMs depends largely
on experimental parameters such as the selected image, target class, and
model. Here, we propose MetaCAM, an ensemble-based method for combining
multiple existing CAM methods based on the consensus of the top-$k$% most
highly activated pixels across component CAMs. We perform experiments to
quantifiably determine the optimal combination of 11 CAMs for a given MetaCAM
experiment. A new method denoted Cumulative Residual Effect (CRE) is proposed
to summarize large-scale ensemble-based experiments. We also present adaptive
thresholding and demonstrate how it can be applied to individual CAMs to
improve their performance, measured using pixel perturbation method Remove and
Debias (ROAD). Lastly, we show that MetaCAM outperforms existing CAMs and
refines the most salient regions of images used for model predictions. In a
specific example, MetaCAM improved ROAD performance to 0.393 compared to 11
individual CAMs with ranges from -0.101-0.172, demonstrating the importance of
combining CAMs through an ensembling method and adaptive thresholding.

##  1 Introduction

Convolutional neural networks (CNNs) are state-of-the-art deep learning
architectures developed for image analysis tasks, including classification,
segmentation, and object detection. These methods were originally considered
uninterpretable (i.e. ‘black boxes’) given that specific regions or features
of an image used to produce a model’s prediction were unknown. Having clear,
reliable model interpretation improves confidence and trust in deploying
artificial intelligence in real-world settings. This is of particular
importance in high-criticality fields, such as medicine, autonomous driving,
and automatic biometric identification [15]. Furthermore, interpretability can
identify decisions using incorrect information, such as biases or undesired
markings in images. There is a definitive need for dependable visualizations
of salient regions used in predictions.

Numerous studies have investigated improving the explainability of CNNs. An
increasingly popular class of explainable algorithms for interpreting CNN
model predictions are Class Activation Maps (CAMs) [19]. Originally developed
by Zhou et al., CAMs create heat-map visualizations indicating the most
salient regions of an image used by a CNN model for a given task. These
visualizations are typically generated through a linear weighting of the
feature maps produced by the final convolutional layer of a network. Many
variations have been propose to improve upon the original CAM formulation [1,
3, 4, 5, 6, 9, 12, 16, 18]. There is, however, little consensus regarding
which CAM method produces the most accurate reflection of important regions
within an image.

The comparison across different CAM variants has also been largely
inconsistent, both with what CAM methods are compared and what performance
metric is considered. Qualitatively, performance may be evaluated in a given
study by comparing visualizations between various CAMs. Quantitatively,
various performance metrics have been proposed including perturbation
analysis, object localization and segmentation, and human trust/class
discrimination, making relative CAM ranking infeasible. Furthermore, the
performance of CAM methods varies across the parameters of individual
experiments, such as the chosen images, their target classes, and the CNN
model. The combination of CAM visualizations for improved performance and
consistency has recently been investigated [13]; however, the selection of
contributing CAMs is arbitrary and understudied.

In this work, we address these issues by proposing a generalized method
MetaCAM, a consensus-based CAM method that outputs the top-$k$% pixels in
agreement across any number and combination of component CAM methods. This
consensus-based approach ensures that if a particular CAM performs poorly for
a specific task, its contribution will be mediated in the final MetaCAM
visualization. Consequently, MetaCAM may be used reliably in a diverse
application to generate valid CAM visualization depicting the most salient
regions of an image.

We additionally develop an adaptive thresholding method to determine optimal
top-$k$ values for maximizing MetaCAM performance. We further extend this
method to refine and improve existing CAM methods. Through large-scale
comparative experiments, we systematically determine what combinations of
component CAMs should be considered as part of MetaCAM among 11 unique
publicly implemented CAM methods[6].

Our key contributions are as follows:

  * •

We propose MetaCAM, a novel ensemble-based CAM method that combines existing
CAMs; we perform extensive experimentation to determine the best aggregation
of CAM methods.

  * •

We demonstrate that MetaCAM can be extended to include non-CAM visual
explanation methods such as FullGrad [17]. For simplicity, when referring to
CAM methods throughout this paper, we additionally subsume FullGrad.

  * •

We develop adaptive thresholding to improve the performance of MetaCAM. We
further demonstrate how this can be extended to individual CAM methods to
greatly improve performance and refine visualizations.

  * •

We perform a systematic evaluation of MetaCAM combinations of 11 individual
CAM methods. We summarize performance across MetaCAM and all individual CAMs
for numerous images, target classes, and CNN models using an unbiased
quantitative performance metric, Remove and Debias (ROAD).

##  2 Related Work

###  2.1 CAM Methods

For a given CAM method, the CAM visualization L is generated from a linearly
weighted summation of all k feature maps A at the chosen layer l of a CNN
architecture f. Each CAM method produces a map for a given image, x, and
class-discriminative methods further specify the desired class output, c. Most
CAM methods also perform a ReLU operation after the final summation to retain
positive activations (not included in Eq. 1). The CAM formulation is thus:

$L^{c}_{CAM(A)}=\sum\limits_{k}(\alpha_{k}^{c}A_{k}),\textit{where
}A=f^{l}(x)$ (1)

The original CAM determined $\alpha_{k}^{c}$, the importance of individual
feature maps, using the weights of the final dense layer leading to class
predictions in a CNN [19]. However, this required the final dense layer to be
preceded by a convolutional layer and global average pooling, which can reduce
performance and is not included in many modern CNNs. As such, new CAM methods
have been developed that are CNN model-agnostic and have improved upon
existing CAM performance. We have grouped 11 existing CAM variants into
distinct categories based on how they compute Eq. 1.

Basic GradCAMs: GradCAM was the first method to propose the use of gradients
to determine the importance of feature maps [16]. Specifically, the gradient
of the desired class with respect to the feature maps at a specified layer is
used as the $\alpha_{k}^{c}$ weights in Eq. 1. This eliminated the need for a
specific network architecture and demonstrated improved performance over the
original CAM computation. GradCAM++ is a variation of GradCAM which leverages
a weighted average of first-order gradients based on higher-order gradients
[1]. While these two methods are consistently used across studies and are
highly popularised, certain critiques (e.g. [4]) have led to the development
of further CAM variants.

GradCAM Variants: Different GradCAM variants have been proposed to further
improve the performance of GradCAM and GradCAM++ and address some of their
limitations. Fu et al. argue that there is little theoretical explanation to
explain averaging gradients in GradCAM [5]. Instead, they propose XGradCAM,
which calculates a weighted average of gradients determined through an
optimization problem based on axiom constraints of sensitivity and
conservation. EigenGradCAM is also a GradCAM variant that has also been
implemented by Gildenblat as a class-discriminative variation of EigenCAM [6,
12]. In this case, $\alpha_{k}^{c}$ remains as the gradient, but rather than
multiplying this by the feature map activations A, the principal components of
the activations are instead used.

Figure 1: Conceptual overview of the proposed MetaCAM visual explanation
method. For clarity, $w=1$ when thresholding is used ($k<100$%) and used
otherwise when $k=100$%.

Elementwise GradCAMs: Rather than weighting feature maps by the average of
first or second gradients, some methods suggest performing elementwise
multiplication of gradients by feature maps. Draelos and Carin show that
averaging gradients may cause certain image elements to falsely appear as
regions of importance [4]. For example, negative elementwise gradients may
become positive from averaging and therefore incorrectly highlight areas on
their respective feature maps. To avoid this, HiResCAM multiplies feature maps
by their elementwise gradients, which ensures each pixel is weighted for
importance by their respective gradient value [4]. A similar elementwise CAM
implementation, GradCAMElementwise, was proposed by Gildenblat [6]. In this
case, after performing elementwise multiplication of feature maps by
gradients, ReLU is employed prior to the summation of feature maps.

LayerCAM: Jiang et al. recognized that the final layers of a CNN produce
coarse feature maps, which may cause the CAM visualizations to lose important
fine-detailed information [9]. To counter this, LayerCAM feature map
activations are weighted using spatial-specific gradients (elementwise
gradients), using positive gradients only. This ensures detailed information
can be captured from any layer in the CNN, and these layers can be summed
together for more specific CAM visualizations.

Image Perturbation-Based Approaches: While high performance has been achieved
using gradient-based CAM methods, studies suggest that gradient saturation may
lead to noisy or diminished visualizations. Wang et al. also show that
gradient-based methods may result in incorrectly weighted feature maps [18].
ScoreCAM and AblationCAM are two alternative approaches that use perturbations
to identify feature map importance. ScoreCAM image perturbations are created
by masking the original input with each feature map [18]. The importance
weights are then based on the forward output score of the network for the
perturbed image. Conversely, AblationCAM zeroes out image regions that are
activated in feature maps [3]. The reduced performance based on the perturbed
image is then used as the $\alpha_{k}^{c}$ importance weights for AblationCAM.

Non-Discriminative Approaches: In addition to gradient saturation, the
calculation of gradients can be time-consuming and dependent on correct image
classification. EigenCAM eliminates these problems and the need for class
discrimination by computing the principal components of the feature maps (this
method does not follow the general CAM formulation in Eq. 1) [12].

Non-CAM Gradient-Based Approaches: While our study proposes MetaCAM, the
combination of any CAM-based approach, it can also be extended into non-CAM
feature maps. We demonstrate this with the inclusion of FullGrad, which is a
non-discriminative method that uses the summed gradients of all bias terms
throughout a CNN [17]. We include FullGrad and any other visual explanation
methods under the term ‘CAM’ throughout this paper.

###  2.2 Quantitative Evaluation

In order to evaluate performance and accuracy of CAM methods, there have been
numerous quantitative metrics proposed. Here, we summarize three of the most
commonly used metrics for CAM evaluation.

Perturbation Analysis: One method of evaluating the quality of CAM
visualizations is by perturbing the original image with the outputted
activations. Some papers choose to mask the least-activated regions from CAM
in the original image [1, 3, 17, 18], while others choose to perturb the most-
activated regions [5, 9, 16, 17]. The perturbated image is then processed by
the model, and the resulting classification score is used to determine the
drop or increase in confidence specifically attributable to the perturbation.

Object Localization and Segmentation: Another CAM performance metric is based
on object localization/segmentation, demonstrated in [1, 4, 9, 12, 16, 18].
Here, true bounding boxes or segmentation masks are provided to identify
objects within an image. The most highly activated pixels in CAM feature maps
are converted to a binary mask, and the mask is then compared to the desired
bounding box or segmentation mask. This is commonly measured through the
intersection over union (IoU).

Human Evaluation and Class Discrimination: One of the reasons CAM methods have
been developed is to instill higher trust in CNN model decisions. Certain
studies choose to evaluate this by asking human raters to identify which CAM
image (among different CAM methods) is more reliable [1, 3, 4]. Similarly, to
evaluate whether CAM methods can correctly identify classes, individuals are
asked to select which class is best highlighted in the image [3, 5, 16].

Certain metrics must be used with caution when evaluating CAM performance. A
CAM model may use areas of an image outside the desired class to make its
prediction and as such, there exists concern as to whether object localization
and human evaluation/class discrimination metrics are truly evaluating the
accuracy of CAM methods. Draelos and Carin highlight this using an example
that water in an image may help provide evidence for the classification of a
boat [4]. Thus, IoU and class discrimination should not be used to identify if
CAM methods accurately identify salient regions of an image used for model
prediction.

##  3 Proposed Method

In this section, we describe MetaCAM’s formulation; a conceptual overview of
our proposed method is depicted in Figure 1. First, we define the problem and
outline several methods of averaging individual CAM methods to develop
MetaCAM. Next, we describe how MetaCAM can formulated based on the consensus
of CAM methods. Briefly, the top k% most highly activated pixels in agreement
across all chosen CAMs are used to create MetaCAM. We also describe how this
thresholding technique can be used to refine individual CAMs. Lastly, we
define how CAMs are evaluated using the Remove and Debias (ROAD) method.

###  3.1 Problem Formulation

Consider a number of feature map visualizations $n$ generated by different CAM
methods. The objective is to determine a method of combining these
visualizations to improve upon individual CAM performance. MetaCAM should
create an accurate visualization of salient regions used in prediction
regardless of selected image, target class, or model.

###  3.2 Averaging-Based MetaCAM

An obvious initial formulation of MetaCAM is to simply take an average of all
individual CAM methods. Prior to combining CAM visualizations, all maps are
normalized between 0 and 1. CAMs that produce an invalid output for a given
image/model are removed prior to calculation.

$L^{c}_{MetaCAM(A)}=\frac{\sum\limits_{n}(L^{c}_{CAM(A)_{n}})}{n}$ (2)

where n is the number of individuals CAMs. If all CAMs included in the
formulation provide highly accurate visualizations, averaging across CAMs
should refine the activated regions and improve performance. However, if any
of the included CAMs activate ‘incorrect’ regions of the image, equally
weighting all CAMs will reduce the performance of MetaCAM and yield lower
performance than other top-performing CAMs. To diminish the inclusion of poor-
performing CAMs, a weighted average can be used:

$L^{c}_{MetaCAM(A)}=\frac{\sum\limits_{n}(w_{n}\cdot
L^{c}_{CAM(A)_{n}})}{\sum\limits_{n}w_{n}}$ (3)

The weights of each CAM can be determined by any quantitative measure of CAM
performance. In this study, we choose the ROAD value (section 3.4) to
numerically quantify and compare CAMs. In addition to using the ROAD values as
weights themselves, transformations can be applied to augment the difference
in performance between CAMs. We experiment with min-max, softmax (with initial
amplification of ROAD weights to a minimum of $10^{1}$), and exponential
normalization of weights.

While a weighted average of CAMs may improve MetaCAM performance over equal-
weighting, poor-performing CAMs are not entirely removed from the overall
formulation of MetaCAM and may still negatively affect performance. For this
reason, we opt for a consensus-based MetaCAM formulation.

###  3.3 Adaptive Thresholding-Based MetaCAM

To reduce the impact of poor-performing CAMs, we leverage a consensus-based
formulation of MetaCAM. Here, rather than averaging CAMs, MetaCAM is generated
using the top-$k$% of pixels in agreement across all CAM methods. Given the
consensus of pixel activations across all methods, regions from individual
CAMs that are incorrectly activated will be excluded from the final MetaCAM
formulation. The activation maps from all CAM methods are summed, and a
threshold is applied where the top-$k$% of summed activations are used as
MetaCAM’s activations, with all other activations set to zero:

$L^{c}_{MetaCAM(A_{i,j})}=\left\\{\begin{array}[]{ c l
}\sum\limits_{n}(L^{c}_{CAM(A_{i,j})_{n}})&\quad\textrm{if }\geq t\\\
0&\quad\textrm{otherwise}\end{array}\right.$

where t is the chosen threshold and i,j represent pixel locations. The best-
performing threshold for MetaCAM is dependent on a given image, target class,
and model. We therefore implement adaptive thresholding, which computes the
ROAD performance of MetaCAM at different thresholds and returns the highest-
performing resultant activation map. For a fair comparison with individual CAM
methods, we extend this adaptive thresholding approach for each component CAM
method. Thus, ROAD percentiles are calculated using the top-$k$% of highly
activated pixels in each activation map.

###  3.4 MetaCAM Evaluation

To quantify all CAM methods considered in this study we implement a pixel
perturbation method, Remove and Debias (ROAD) [14]. ROAD addresses issues of
data leakage found in other pixel perturbation methods and improves
computational efficiency [14]. To determine the performance of a CAM
visualization, ROAD uses noisy linear imputations to perturb either the most
or least activated image pixels. The imputations are applied to individual
pixels based on neighbouring pixel values, creating a blurred perturbation of
the original image as opposed to masking the pixels entirely. The perturbed
image is then evaluated by the network to determine the increase or decrease
in prediction confidence. Prediction confidence varies depending on the
percent of pixels perturbed; for this reason, we choose to evaluate ROAD at
20%, 40%, 60%, and 80% of perturbation, taking an average across all
percentiles for robust evaluation. We also consider Gildenblat’s proposition
to combine the confidence drop when the most activated pixels are perturbed,
and the confidence increase when the least activated pixels are perturbed [6].
Our final ROAD evaluation score is thus:

$ROAD(L^{c}_{CAM})=\sum\limits_{p}\frac{(C^{p}_{L_{LRP}}–C^{p}_{L_{MRP}})}{2}$
(4)

where p is the percentile, C is the confidence, LRP represents least relevant
pixels, and MRP represents most relevant pixels.

##  4 Experiments

In this section, we outline the experiments used to evaluate MetaCAM. Briefly,
we first describe the image dataset, element classes and models, and
experimental setup used to evaluate MetaCAM’s performance in sections 4.1 and
4.2. Next, we test different combinations of CAMs (CAM-sets) to determine both
the optimal formulation of MetaCAM and top-k pixel threshold, evaluated using
a novel method denoted Cumulative Residual Effect (CRE) and normalized ROAD
scores in sections 4.3, 4.4, 4.5, 4.6 4.7. Lastly, we compare MetaCAM to
individual CAMs both quantitatively and qualitatively in sections 4.8, 4.9.

###  4.1 Datasets & Pre-Trained Models

To compare MetaCAM performance to other individual CAM methods we selected
various images from the ImageNet ILSVRC 2012 validation dataset [2] in
addition to sample images commonly considered as benchmarks for CAM method
comparison. Each of the 2D RGB natural images with ground truth segmentation
maps has a corresponding numerical class ID. Images are preprocessed by
reshaping to an initial size of $256\times 256$ and center cropping to
$224\times 224$, followed by normalization. For simplicity, a given image
(e.g. $x$), class (e.g. $c$), and model (e.g.$f(\cdot)$) are hereafter denoted
as an $(x,c,f(\cdot))$ 3-tuple. We use ResNet152 [7] and DenseNet161 [8]
models (unless otherwise specified) pre-trained on the ImageNet-1K dataset
from PyTorch [11], comprised of 1,000 possible classes.

Figure 2: CAM inclusion study curves summarising the adaptive thresholding
performance across $m=64$ binary experiments. The mean performance is banded
by threshold-wise 95% confidence intervals. Figure 3: Frequency for which a
given top-$k$% pixel threshold produced the maximum ROAD score across $m=64$
binary experiments.

###  4.2 Experimental Setup

To experimentally determine what ideal combination of individual CAMs produces
the best ROAD score for a given $(x,c,f(\cdot))$ we might trivially consider
all available CAMs and/or vision explanation methods. While an initial
starting point, such an approach comes at a maximal computational expense and
does not provably guarantee optimal results. Rather, as much ensemble-based
research is conducted, the optimal combination of components must be
determined experimentally. To that end, we leveraged a systematic approach for
determining what CAMs to include/exclude in a particular MetaCAM application
to a given $(x,c,f(\cdot))$.

###  4.3 Systematically Determining the Ideal CAM-set for MetaCAM

The systematic and unbiased methodology to determine what CAM combination
produces the optimal MetaCAM result can be determined experimentally. To that
end, we leverage high-performance-computing (HPC) infrastructure to uniquely
explore the hyper-parameterized inclusion-/exclusion-space of all available
CAM methods.

For the $n=11$ individual CAM methods available under consideration, we opted
for six unique groupings of methodologies to reduce the computational expense
of our systematic experiments ($m=2^{11}$ unique experiments is
computationally taxing). Ultimately, we organized all CAM methods into six
groupings based on methodological similarity, as reflected in the
introduction; certain CAMs were excluded from these large-scale experiments
given their individual computational expense. Consequently, we grouped all CAM
methods into $n=6;m=2^{n}$ individual experiments that represent the
inclusion/exclusion of specific groups:

  1. A : [HiResCAM, GradCAMElementwise],

  2. B : [GradCAM, GradCAM++],

  3. C : [XGradCAM],

  4. D : [AblationCAM, ScoreCAM],

  5. E : [LayerCAM],

  6. F : [FullGrad]

Thus, all systematically determined experiments are summarized through a CAM-
specific binary number where $100000$ position-wise represents only the CAMs
of group A (1 indicates inclusion only for the binary position for group A).
Similarly, 111111 represents the inclusion of all CAMs in groups ABCDEFG. Our
work quantifies all ROAD performance of the $m=64$ individual experiments for
a given $(x,c,f(\cdot))$. The maximum achievable ROAD score is determined
across all $m=64$ experiments and a relative ranking of CAM
inclusion/exclusions is further explored in subsequent sections. Individual
experiments were allocated $4\times$ cores, 64GB of RAM, and either an NVIDIA
P100 Pascal or V100 Volta GPU through the Digital Research Alliance HPC
infrastructure.

Figure 4: Cumulative Residual Effect summarizing group-wise impact on MetaCAM
performance Figure 5: Comparison of normalized inter-experiment maximum ROAD
scores. Figure 6: Visualization of the MetaCAMs from the best performing
threshold across various images, classes, & model architectures. Figure 7:
MetaCAM outperforms individual methods even when augmented through adaptive
thresholding. Figure 8: Comparison of individual CAMs using a pre-trained
DenseNet161 and the cat class ID (281).

###  4.4 Experimentally Determining Ideal MetaCAM Top-$k$% Pixel Threshold

The proposed MetaCAM method leverages top-$k$ pixel adaptive thresholding that
can be experimentally determined across the $m=64$ individual experiments. For
a given range of values $k$ (typically $k\in[15,45]$, determined
experimentally) the mean ROAD score is depicted for various $(x,c,f(\cdot))$
in Figure 2. We note that there does not exist a universally applicable
threshold range and that in some cases (e.g. E) there does not exist a
definitive ideal threshold whereas for others (e.g. A,B,C,D,G) there are peak
threshold or ranges providing dramatic increases in ROAD score.

To further investigate the influence on MetaCAM’s top-$k$ thresholding, Figure
3 depicts a sample of four experiments and the frequency at which a given
value $k$ produced the top-1 maximum ROAD score. Interestingly, certain
$(x,c,f(\cdot))$ produce a definitive $k$ (e.g. A,C) while others suggest a
general range for $k$ (e.g. B,D). Intuitively, the complexity of the contents
of a given image and the classes represented therein are strong determinants
of the ideal $k$; images with a clear distinction of class-specific pixels and
the remainder of the image do not overly benefit from an adaptive threshold
(Figure 2E) while all others suggest tuning $k$. As a rule of thumb, a modest
amount of top-$k$ thresholding appears to consistently improve ROAD score,
often by a considerable margin.

###  4.5 Cumulative Residual Effect

The $m=64$ binary CAM-group inclusion/exclusion experiments for a given
$(x,c,f(\cdot))$ produce a unique relative ranking of experiment-specific
binary numbers according to the MetaCAM ROAD score. To quantify the influence
a given CAM group had on the resultant MetaCAM ROAD score we propose the
Cumulative Residual Effect method. CRE determines the relative
positive/negative effect of each CAM group by taking the residual of the
individual MetaCAM score with the median value of all scores for the $m=64$
experiments and sum this residual (either a positive or negative value) with
each contributing CAM groups within that experiment. This produces a group-
wise summary representing the relative impact of including/excluding that CAM
group; all CAM groups are included in exactly 32 experiments and excluded in
exactly 32 experiments.

Since this method leverages residuals, it has an additional useful property of
being aggregated with other experiments; that is, inter-experiment CREs are
obtained by summing intra-experiment group-wise values. In Figure 4 we
illustrate four example CRE plots that may be roughly interpreted in a way
analogous to SHAP force plots [10]. The inclusion of certain CAM groups will
positively improve MetaCAM ROAD score (e.g. A,B) relative to the median of all
experiments, and in other cases, their inclusion negatively influences overall
ROAD scores (e.g. C,D; it is critical to note that this quantification is
relative to the experiment-wide median ROAD score).

###  4.6 Comparison of MetaCAMs across Images, Classes, & Models

To fully depict the MetaCAM output across varying $(x,c,f(\cdot))$, we
visualize the best-performing MetaCAMs by ROAD score in Figure 5. We note that
our consensus-based approach reliably detects the target class in all cases
with dramatic improvements in ROAD score over individual CAMs (section 4.8).

###  4.7 “Bad” CAMs & RandomCAM can Improve Performance

Surprisingly, the inclusion of individually poor-performing CAMs (such as
EigenCAM which often resulted in low or negative ROAD scores) and random noise
(e.g. RandomCAM, an activation map generated from a random uniform
distribution between -1 and 1 [6]) resulted in improved MetaCAM performance.
Our initial top-performing MetaCAM combination using DenseNet161 to predict
the cat in the catdog image achieved ROAD performance of 0.295. However, the
inclusion of EigenCAM and RandomCAM improved ROAD performance to 0.393 and
0.314, respectively. In addition, the highest-performing threshold decreased
from 19 to 10 with EigenCAM, and 15 with RandomCAM. We posit that the
inclusion of poor-performing CAMs or random noise forces MetaCAM to further
refine its output to only those highest-consensus pixels at lower top-$k$
threshold values. Giving random or incorrect regions higher activations
results in a refinement of the specific area used in model predictions and
contributes to the exclusion other classes within the same image.
Interestingly, future work might consider incorporating various “bad”/random-
based CAMs to experimentally determine their impact on overall MetaCAM
performance.

###  4.8 MetaCAM Outperforms Individual Methods

To compare the performance of MetaCAM against other visual explanation
methods, we applied adaptive thresholds to each individual visualization,
displayed in Figure 7. MetaCAM outperforms all individual CAM methods, shown
by the largest peak at $k=10$. Most CAMs have a peak performance using a
threshold between the top 10%-30% of most highly activated pixels. This
indicates that adaptive thresholding is able to improve performance of all
visual explanation methods, by selecting the top relevant pixels for a given
$(x,c,f(\cdot))$. In addition, Figure 7 also shows the original performance of
CAMs without adaptive thresholding ($k=100$). Here, ROAD performance ranges
between -0.101-0.172, further demonstrating that MetaCAM (ROAD=0.393) has a
dramatic increase in performance compared to original CAMs.

###  4.9 Comparison of CAMs across different k% Pixel Thresholds

Figure 8 displays our adaptive thresholding technique applied to all 11 CAMs
explored in this study, in addition to MetaCAM. CAMs are shown using
thresholds of the top 15%, 30%, and 45% activated pixels, as well as the
original CAM visualization. These qualitative results present the usefulness
of thresholding; reducing the number of activated pixels focuses on the most
salient regions of each image. Many of the original CAM visualizations
activate large regions of the image, including both the cat and dog despite
only using the cat class ID (281) as the target. Adaptive thresholding is able
to refine the activations of all CAMs to focus on the desired target class. It
is important to note that thresholding is not performing object localization;
it is focusing on the regions of an image used by a model for prediction, as
measured by ROAD.

Interestingly, EigenCAM incorrectly highlights the dog in the image, instead
of the desired cat class. This explains the negative ROAD value for EigenCAM
in Figure 7. EigenCAM is a non-discriminative CAM method that uses principal
components to create activation maps. However, when there are multiple classes
within the same image, the order of prinicipal components must be specified
(e.g., first principal components vs. second principal components). EigenCAM
performs well on images with a “single-subject”, but otherwise requires a user
to determine the number and rank of various components within an image to
perform successfully. This requires a level of hand-engineering and data
leakage. Despite the potential of including ‘incorrect’ classes in an image,
EigenCAM may still be beneficial to the performance of MetaCAM as described
previously.

##  5 Conclusion

In this study, we propose MetaCAM, a consensus-based combination of any number
of existing CAM formulations using the top k% of pixels in agreement across
all methods. Our experiments demonstrate that MetaCAM is able to outperform
existing CAM methods, both with and without adaptive thresholding. We expect
MetaCAM to be of particular use in high-criticality fields.

## Acknowledgment

The authors would like to acknowledge Dr. Katherine Muldoon for her support of
this work. The authors also acknowledge that this study took place on unceded
Algonquin Anishinabe territory.

## References

  * [1] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839–847, 2018.
  * [2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [3] Saurabh Desai and Harish G. Ramaswamy.  Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization.  In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 972–980, 2020.
  * [4] Rachel Lea Draelos and Lawrence Carin.  Use hirescam instead of grad-cam for faithful explanations of convolutional neural networks, 2020.
  * [5] Ruigang Fu, Qingyong Hu, Xiaohu Dong, Yulan Guo, Yinghui Gao, and Biao Li.  Axiom-based grad-cam: Towards accurate visualization and explanation of cnns, 2020.
  * [6] Jacob Gildenblat and contributors.  Pytorch library for cam methods.  https://github.com/jacobgil/pytorch-grad-cam, 2021.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [8] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger.  Densely connected convolutional networks.  CoRR, abs/1608.06993, 2016.
  * [9] Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei.  Layercam: Exploring hierarchical class activation maps for localization.  IEEE Transactions on Image Processing, 2021.
  * [10] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  Advances in neural information processing systems, 30, 2017.
  * [11] TorchVision maintainers and contributors.  TorchVision: PyTorch’s Computer Vision library, 11 2016.
  * [12] Mohammed Muhammad and M. Yeasin.  Eigen-cam: Class activation map using principal components.  pages 1–7, 07 2020.
  * [13] Ahmet Haydar Ornek and Murat Ceylan.  Codcam: A new ensemble visual explanation for classification of medical thermal images.  Quantitative InfraRed Thermography Journal, 0(0):1–25, 2023.
  * [14] Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci.  A consistent and efficient evaluation strategy for attribution methods.  In Proceedings of the 39th International Conference on Machine Learning, pages 18770–18795. PMLR, 2022.
  * [15] Thomas Schmid, Wolfgang Hildesheim, Taras Holoyad, and Kinga Schumacher.  The ai methods, capabilities and criticality grid.  KI - Künstliche Intelligenz, 35:1–16, 08 2021.
  * [16] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017.
  * [17] Suraj Srinivas and François Fleuret.  Full-gradient representation for neural network visualization.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [18] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 24–25, 2020.
  * [19] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.


11institutetext: Univ. Grenoble Alpes, Inserm, U1216, Grenoble Institut
Neurosciences, 38000, FR 22institutetext: Pixyl, Research and Development
Laboratory, 38000 Grenoble, FR 33institutetext: Univ. Grenoble Alpes, Inria,
CNRS, Grenoble INP, LJK, 38000 Grenoble, FR

# Multi-layer Aggregation as a key to feature-based OOD detection

Benjamin Lambert 11 2 2 Florence Forbes 33 Senan Doyle 22 Michel Dojat 11

###### Abstract

Deep Learning models are easily disturbed by variations in the input images
that were not observed during the training stage, resulting in unpredictable
predictions. Detecting such Out-of-Distribution (OOD) images is particularly
crucial in the context of medical image analysis, where the range of possible
abnormalities is extremely wide. Recently, a new category of methods has
emerged, based on the analysis of the intermediate features of a trained
model. These methods can be divided into 2 groups: _single-layer_ methods that
consider the feature map obtained at a fixed, carefully chosen layer, and
_multi-layer_ methods that consider the ensemble of the feature maps generated
by the model. While promising, a proper comparison of these algorithms is
still lacking. In this work, we compared various feature-based OOD detection
methods on a large spectra of OOD (20 types), representing approximately 7800
3D MRIs. Our experiments shed the light on two phenomenons. First, _multi-
layer_ methods consistently outperform _single-layer_ approaches, which tend
to have inconsistent behaviour depending on the type of anomaly. Second, the
OOD detection performance highly depends on the architecture of the underlying
neural network.

###### Keywords:

Uncertainty Deep learning Anomaly Detection Medical images analysis

##  1 Introduction

Out-of-distribution (OOD) images correspond to samples that are significantly
different from the ones observed during training. Deep Learning (DL) models
tend to behave inconsistently for this type of inputs, making OOD image
detection crucial to avoid hidden model deficiencies. It is especially
required in real-world automated pipelines, where input images may not be
visually inspected before running the analysis. In the context of medical-
images analysis, a large variety of phenomenons in the input images can impact
a model and lead to unpredictable responses: noise, artifacts, variations in
the imaging acquisition protocol and device, or pathological cases that were
not included in the initial training dataset. Various methods were proposed
for their detection, which can roughly be divided into two different
categories [3]: methods that build a model specifically dedicated to OOD
detection; and methods that rely on the uncertainty or intermediate
activations of a task-specific model (e.g image segmentation) to detect
abnormal inputs.

Within the first category, the most straightforward approach is to build a
classifier to directly detect OOD images. For this, a Convolutional Neural
Network (CNN) can be trained in a supervised manner, thus requiring the
construction of an annotated dataset containing various types of real-world
OOD [4]. Unsupervised Anomaly Detection (UAD) proposes to model the appearance
of normal images by training an Auto-Encoder network (AE) to reconstruct in-
distribution (ID) samples. At test-time, reconstruction is expected to be
degraded for OOD samples, allowing for their detection [10].

Among the second category, uncertainty-based methods propose to detect OOD
inputs directly from the outputs of an existing neural network. They rely on
the hypothesis that the uncertainty of the deployed model should be high in
the presence of a train-test mismatch, allowing its detection. A standard
method consists of producing a set of diverse and plausible predictions for
the same input image, with MC dropout [9], Deep Ensemble [21] or Test Time
Augmentation [33] being popular approaches. Uncertainty can then be estimated
by computing the variance among the predictions. Alternatively, feature-based
methods propose to analyse the intermediate activations of an existing model
to detect OOD inputs [28]. It is based on the assumption that the hidden
activations of the model should be different for an ID image compared to an
OOD image. A taxonomy of these feature-based methods is possible based on the
number of layers used for OOD detection. _Single-layer_ methods only target
one specific convolutional layer. In the context of medical image
segmentation, popular choices are the end of the encoder [11] or the
penultimate convolutional layer [16, 7]. _Multi-layer_ methods are an
extension of the former that consider the entire set of convolutional layers
in the trained model for OOD detection [5]. Although gaining popularity, a
proper comparison of these algorithms is still lacking. The contributions of
our work are as follows:

  * •

We develop a large MRI segmentation benchmark comprising 20 different OOD
datasets of various types and strengths, representing 7796 3D MRI volumes. We
use this benchmark to compare 5 different feature-based OOD detectors as well
as one uncertainty baseline.

  * •

We adapt single-layer methods to a multi-layer fashion to demonstrate the
potential performance gain that can be obtained with this enhancement.

Figure 1: Illustration of ID and OOD data used in the experiments.

##  2 Compared methods

###  2.1 Feature-based methods

Feature-based methods rely on a trained segmentation model and follow the same
principle. First, a set of feature maps $F_{i}\in\mathbb{R}^{N\times
H_{i}\times W_{i}\times D_{i}}$ is collected from a ID dataset for one
convolution layer $i$ (single-layer methods) or all convolution layers (multi-
layer methods). Here, $N$ correponds to the number of convolutional filters in
the $i$-th layer, and $H_{i}\times W_{i}\times D_{i}$ to the spatial
dimensions of the feature map. Second, at inference time, a metric is computed
to estimate the distance between the test features and the ID features to
detect OOD samples.

#### Spectral signatures

[16] was proposed as a single-layer method focusing on the features obtained
at the penultimate convolutional layer. Features are flattened to a 2D matrix
$F_{2D}\in\mathbb{R}^{N\times HWD}$, and its singular values $S$ are
calculated. The spectral signature is then taken as $\phi=\frac{log(S)}{\lVert
log(S)\rVert_{2}}$. To detect OOD at test time, the distances $d_{j}$ between
the signature of the test image $\phi_{test}^{i}$ and the signatures of a set
of ID samples $\phi_{ID}^{j}$ is obtained by using the Euclidean distance. The
final proposed OOD score corresponds to the minimum of the $d_{j}$ distances.

#### Prototypes

[7] is a single-layer method that operates from the penultimate layer features
and the segmentation masks predicted by the segmentation network. To obtain a
prototype for a specific class and input image, features are multiplied with
the binarized class mask, and average pooling is applied on the masked
features. This yields to prototypes $P$ $\in\mathbb{R}^{N\times C}$, $C$ being
the number of segmented classes. An average ID prototype $P_{ID}$ is finally
obtained by averaging the prototypes collected on the ID dataset. At test
time, the OOD score is taken as the cosine dissimilarity between $P_{ID}$ and
the test image prototype $P_{j}$.

#### The Mahalanobis Distance

(MD) was recently investigated in 2 distinct studies for OOD detection in 3D
medical images. In [11], authors focus on the features from the end of the
encoder part of the network. They apply consecutive average pooling until the
number of elements $M$ in the feature maps falls below a defined threshold of
$1e4$, and then flatten it to obtain 1D vectors $z_{i}$ $\in\mathbb{R}^{M}$,
for each ID image $i$. From these vectors, they compute the parameters of a
multivariate Gaussian: the mean $\mu\in\mathbb{R}^{M}$ and covariance
$\Sigma\in\mathbb{R}^{M\times M}$. At test time, the MD is computed given the
fitted Gaussian and the test image feature representation. We refer to this
single-layer method as _MD Pool_. A similar approach is implemented in the
multi-layer Free Rejection of Out-of-Distribution (FRODO) approach [5]. This
work differs in two ways: first, they directly compute the average of the
feature map over the spatial dimensions ($H\times W\times D$) instead of
applying average poolings. Second, they fit a multivariate Gaussian
independently for each convolutional layer and compute the final OOD score as
the average of each layer score.

#### One-class SVM

(OCSVM) is an unsupervised algorithm for OOD detection that can be trained
using only ID samples. It aims at finding the optimal boundary around the
expected (ID) data. At test time, the distance to the boundary can be used as
an OOD score, with ID sample being attributed with negative distances, and OOD
samples with positive distances. Similar to [34], we fit a OCSVM per
convolution based on the averaged layer activations obtained from the ID
images. At test time, each OCSVM produces a score, and the final OOD score is
taken as the maximum of these scores.

###  2.2 Adapting single-layer methods to multi-layer methods

To assess the contribution of multi-layer aggregation to OOD detection, we
propose to adapt single-layer methods (Spectrum, Prototypes and MD Pool) to
multi-layer style. To achieve this, we replicate the OOD score computation
step for each convolutional layer independently, yielding to _layer-wise_
scores $l_{i}$. As in FRODO, the final multi-layer score $L_{multi}$ is taken
as the average of the scores of the individual layers:

$L_{multi}=\frac{1}{N}\sum^{N}_{i=1}l_{i}$ (1)

###  2.3 Uncertainty baseline

To compare feature-based OOD detectors with a more traditional uncertainty
methodology, we implement a MC dropout baseline [9]. In MC dropout, the
dropout layers of the segmentation model are kept activated at test-time and
$N=20$ predictions are repeatedly sampled for each input image. The average
voxel-wise variance among the MC samples is taken as uncertainty estimate at
test time.

##  3 Material and Method

###  3.1 In-distribution Datasets

Our work relies on the open-source BraTS 2021 dataset [2] containing 1251
patients. The dataset initially includes four MRI sequences for each patient
with four ground truth segmentation masks: the background, the necrotic tumor
core, the edematous and the GD-enhancing tumor. We choose to focus on T1w
sequences as this sequence is common and sufficient for experiment with
multiple OOD settings. We also simplify the prediction task by focusing on the
segmentation of the _whole tumor core_ , concatenation of all tumors sub-
classes. The dataset is randomly split into a training fold (651), a
calibration fold (200) used to fit the OOD detectors and a testing fold (400)
(referred to as Test ID in the following). Additionally, we propose to include
_control_ samples in our protocol, representing images that share the same
properties than the training samples (same modality, organ and pathology), but
that were acquired in a different imaging center. An effective model _should_
be able to generalize to these images and thus, the OOD detector should
identify them as ID samples to prevent false alarms. We thus propose to use
the LUMIERE glioblastoma dataset [30] as a _Control_ dataset, from which we
select 74 T1-w pre-operative brain MRI. Figure 1 illustrates the data used in
the different experiments.

###  3.2 Out-of-distribution Datasets

Following the categorization of [11], we propose to investigate
_Transformation_ , _Diagnosis_ and _Far_ OODs, as well as a new proposed
setting, _Modality_ shifts.

#### Transformation shifts.

Finding real images with a controlled amount of artifacts to allow evaluation
of OOD detection methods is difficult. We therefore generate realistic
synthetic artifacted images from the set of _Test ID_ images [8, 11]. We used
the TorchIO Data Augmentation library [19] to generate _Bias_ , _Motion_ ,
_Ghost_ , _Spikes_ , _Downsample_ , _Noise_ , and _Scaling_ artifacts. We add
a set of novel transformations: the _Registration_ that applies noise to the
registration matrix to simulate an erroneous registration, the _Gamma_ that
applies extreme gamma modification to the image to mimick errors in the
intensity normalization step, and the _Truncation_ that crops half of the
brain. Finally, we also implement _Adversarial_ attacks, using the popular
Fast Gradient Sign Method (FGSM) [12].

#### Diagnosis shifts.

DL segmentation models are usually trained with images showing a single
pathology (e.g brain tumor or strokes). However, once deployed, the model can
be confronted with images exhibiting unseen anomalies, which can lead to
incorrect predictions. To test OOD detection methods on this scenario, we use
T1w brain MRI with various diseases: 170 subjects from the White Matter
Hyperintensities (WMH) 2017 challenge [20], 655 subjects from the ATLAS-2
brain stroke dataset [22] and 162 subjects from the EPISURG dataset [27]
containing epileptic subjects who underwent resective brain surgery. We also
use 582 T1-w MRIs from healthy and young subjects from the IXI dataset [1].

#### Modality shifts.

Medical images are usually stored in DICOM formats, whose meta-data (headers)
may be incorrectly filled [13]. As a result, mismatches between the expected
input modality (e.g T1w) and the test image modality (e.g CT or T2w) may be
undetected. We construct 3 different _Modality_ shift OOD datasets. First, we
use the FLAIR and T1ce sequences corresponding to the 400 test subjects.
Second, we extract 437 brain CT-scans from the CQ500 dataset, exhibiting
intracranial hemorrhage or cranial fractures.

#### Far OOD

corresponds to images that show little to no similarity with the ID samples.
We use 2 non-brain T1w MRI datasets, respectively 80 abdominal MRI from the
CHAOS dataset [17] and 515 images from the Lumbar Spine MRI dataset [25], as
far OOD samples.

###  3.3 Influence of the segmentation model architecture

Feature-based OOD detection methods rely on the hypothesis that the
activations of the trained segmentation models are representative of the
conformity of the input sample. To verify if this holds true for any
segmentation model, we use the MONAI library [6] to train 6 different
segmentation models: an Attention UNet (AttUNet) [26], a Residual UNet
(ResUNet) [18], a Dynamic UNet (DynUNet) [15], a UNet++ [35], a VNet [24] and
a Transformer-based model, namely the UneTR [14]. All models are trained with
the Dice loss [24], instance normalization [31], a 3D dropout [29] rate of
$20\%$, and a batch size of 1, using the ADAM optimizer [19] with a learning
rate of $2e-4$.

Table 1: Number of model parameters (in millions) as well as the average segmentation performance (Dice) on the _Test ID_ dataset, for each segmentation model. | AttUNet | ResUNet | VNet | UNet++ | DynUNet | UNETR
---|---|---|---|---|---|---
Nb. parameters | 5.0 | 4.8 | 11.4 | 7.0 | 16.5 | 102
Test ID Dice $\uparrow$ | .83 | .81 | .81 | .81 | .82 | .76

###  3.4 Evaluation Setting

We cast OOD detection as a binary classification problem, where ID samples
correspond to the positive class and OOD samples to the negative class. Each
method produces a score for each OOD sample, which is compared with the scores
obtained on the ID data in order to compute AUROC classification scores. We
also report the segmentation performance using the Dice score, when the ground
truth for brain tumors is available.

Table 2: OOD detection performance (AUROC) for each dataset, obtained with the AttUNet segmentation model for the features and uncertainty approaches. We also report the number of samples in each dataset (N) and Dice score when the ground truth for brain tumors is available. S: single-layer. M: multi-layer. |  |  | Spectrum | Prototypes | MD Pool | Frodo | SVM | MC
---|---|---|---|---|---|---|---|---
Dataset | N |  | Dice
---
($\uparrow$)
S | M | S | M | S | M | M | M | -
Test ID | 400 | .83 | - | - | - | - | - | - | - | - |
Control | 74 | .85 | 0.56 | 0.84 | 0.36 | 0.48 | 0.42 | 0.40 | 0.45 | 0.38 | 0.50
Motion | 400 | .82 | 0.66 | 0.97 | 0.48 | 0.49 | 0.76 | 0.71 | 0.80 | 0.60 | 0.56
Ghost | 400 | .80 | 0.66 | 0.57 | 0.46 | 0.48 | 0.88 | 0.85 | 0.85 | 0.60 | 0.63
Bias | 400 | .78 | 1.00 | 1.00 | 0.86 | 0.82 | 1.00 | 0.98 | 1.00 | 1.00 | 0.77
Spikes | 400 | .79 | 0.90 | 1.00 | 0.69 | 0.82 | 0.86 | 1.00 | 1.00 | 1.00 | 0.63
Noise | 400 | .81 | 0.84 | 1.00 | 0.57 | 0.61 | 0.80 | 1.00 | 1.00 | 1.00 | 0.62
Downsample | 400 | .82 | 0.69 | 0.97 | 0.48 | 0.50 | 0.55 | 0.55 | 0.73 | 0.60 | 0.53
Gamma | 400 | .31 | 1.00 | 1.00 | 0.95 | 0.96 | 0.98 | 0.98 | 1.00 | 1.00 | 0.97
Truncation | 400 | .61 | 0.99 | 0.99 | 0.86 | 0.83 | 1.00 | 0.99 | 1.00 | 0.99 | 0.63
Registration | 400 | .01 | 1.00 | 1.00 | 0.93 | 0.88 | 1.00 | 1.00 | 1.00 | 1.00 | 0.84
Adversarial | 400 | .58 | 0.77 | 0.69 | 0.42 | 0.50 | 0.89 | 0.95 | 1.00 | 0.97 | 0.80
Scaling | 400 | .77 | 0.99 | 1.00 | 0.91 | 0.91 | 0.99 | 0.99 | 1.00 | 1.00 | 0.72
Transform | 4400 | - | 0.86 | 0.92 | 0.70 | 0.71 | 0.89 | 0.92 | 0.95 | 0.89 | 0.70
FLAIR | 400 | .10 | 0.99 | 0.99 | 0.40 | 0.91 | 0.99 | 0.97 | 1.00 | 0.99 | 1.00
T1Ce | 400 | .69 | 0.97 | 0.96 | 0.80 | 0.78 | 0.92 | 0.92 | 0.97 | 0.90 | 0.90
CT | 437 | - | 0.99 | 1.00 | 0.36 | 0.89 | 0.99 | 0.98 | 1.00 | 0.99 | 1.00
Modality | 1237 | - | 0.98 | 0.98 | 0.54 | 0.86 | 0.97 | 0.96 | 0.99 | 0.97 | 0.97
Healthy | 577 | - | 0.55 | 0.99 | 0.89 | 1.00 | 0.12 | 0.57 | 0.97 | 0.98 | 0.92
Strokes | 655 | - | 0.58 | 0.88 | 0.82 | 0.97 | 0.37 | 0.54 | 0.86 | 0.82 | 0.81
WMH | 170 | - | 0.73 | 0.96 | 0.89 | 0.98 | 0.30 | 0.54 | 0.96 | 0.93 | 0.76
Epilepsy | 162 | - | 0.68 | 0.94 | 0.89 | 0.99 | 0.41 | 0.60 | 0.88 | 0.83 | 0.82
Diagnosis | 1564 | - | 0.59 | 0.92 | 0.86 | 0.98 | 0.28 | 0.58 | 0.92 | 0.90 | 0.85
Lumbar | 515 | - | 1.00 | 1.00 | 0.86 | 0.96 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Abdominal | 80 | - | 1.00 | 1.00 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Far OOD | 595 | - | 1.00 | 1.00 | 0.88 | 0.96 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Overall | 7796 | - | 0.84 | 0.93 | 0.72 | 0.81 | 0.79 | 0.86 | 0.95 | 0.91 | 0.80
Figure 2: _Overall_ OOD detection performance (AUROC score) of feature-based
approaches with respect to the segmentation model architecture.

##  4 Results and Discussion

The average segmentation performance of each tested segmentation backbone is
presented in Table 1, with AttUnet being the best performer. OOD detection
performances of each method are presented in Table 2 using the AttUnet as
backbone for feature and uncertainty-based methods. Finally, Figure 2 presents
the _Overall_ OOD detection performance with respect to the neural
architecture.

All OOD detectors achieve high detection accuracy on _Far OOD_ with mixed
performances on other OOD types, showing that restricting to extreme OOD
examples is insufficient to robustly validate a method. The best performer is
FRODO, achieving a perfect detection of non-conform inputs (AUROC=$1.00$) in
12 out of 20 settings, followed by the multi-layer implementation of Spectrum,
and OCSVM. Overall, multi-layer methods outperform their single-layer version.
For the AttUnet, this enhancement allows an increase of the _Overall_ AUROC
score of $10.7\%$ for Spectrum, $12.5\%$ for Prototypes and $8.9\%$ for MD
Pool. Single-layer methods exhibit variable performances depending on the OOD
type, in accord with observations on 2D image classification [34]. Finally,
the MC dropout baseline has mixed performance in our benchmark, being
outperformed by all multi-layer feature-based detectors.

The rankings of OOD methods is roughly the same with different segmentation
architecture (Figure 2), with FRODO and Spectrum Multi-layer being the two
top-performers. Interestingly, converting single-layer methods to multi-layer
methods is beneficial for all architectures, with a gain on the AUROC for 6
out of 6 backbones for Spectrum and MD Pool, and 5 out of 6 backbones for
Prototypes. However, the global _Overall_ OOD detection performance is
variable depending on the segmentation architecture, e.g. FRODO achieves an
AUROC score of $0.95$ or of $0.86$ when implemented on an AttUnet or VNet
respectively. This indicates that certain popular medical image segmentation
architectures are more prone to _feature collapse_ [32], mapping OOD images to
ID feature representations. Several strategies have been proposed in the
context of 2D image classification to alleviate this issue, such as adding
Gradient Penalty [32], Lipschitz constraints [23], or a reconstruction term in
the loss [28]. These methods aims at enforcing a discriminative feature space
for OOD detection, requiring changes in the training paradigm of the model,
possibly resulting in sub-optimal predictive performances. To summarize, our
main findings are:

  * •

Feature-based methods monitoring the activation of _all_ convolution layers
are more performant and robust than methods only targeting a single layer,
whose performance is highly variable depending on the type of OOD.

  * •

The performance of these methods is dependent on the underlying segmentation
architecture, with some of them being more prone to feature collapse,
undermining the sensibility of OOD detection.

## References

  * [1] The ixi brain dataset. https://brain-development.org/ixi-dataset/
  * [2] Baid, U., Ghodasara, S., et al.: The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021)
  * [3] Berger, C., et al.: Confidence-based out-of-distribution detection: a comparative study and analysis. Unsure 2021 pp. 122–132 (2021)
  * [4] Bottani, S., et al.: Automatic quality control of brain t1-weighted magnetic resonance images for a clinical data warehouse. Medical Image Analysis 75, 102219 (2022)
  * [5] Çallı, E., Van Ginneken, B., et al.: Frodo: An in-depth analysis of a system to reject outlier samples from a trained neural network. IEEE Transactions on Medical Imaging 42(4), 971–981 (2022)
  * [6] Cardoso, M.J., Li, W., et al.: Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 (2022)
  * [7] Diao, Z., et al.: A unified uncertainty network for tumor segmentation using uncertainty cross entropy loss and prototype similarity. Knowledge-Based Systems 246, 108739 (2022)
  * [8] Fuchs, M., Gonzalez, C., Mukhopadhyay, A.: Practical uncertainty quantification for brain tumor segmentation. Medical Imaging with Deep Learning (MIDL) (2021)
  * [9] Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. ICML 48, 1050–1059 (2016)
  * [10] Gong, D., Liu, L., et al.: Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. ICCV pp. 1705–1714 (2019)
  * [11] González, C., Gotkowski, K., et al.: Distance-based detection of out-of-distribution silent failures for covid-19 lung lesion segmentation. Medical Image Analysis 82, 102596 (2022)
  * [12] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. 3rd International Conference on Learning Representations, ICLR 2015 (2015)
  * [13] Gueld, M.O., Kohnen, M., et al.: Quality of dicom header information for image categorization. Medical imaging 2002: PACS and integrated medical information systems: design and evaluation 4685, 280–287 (2002)
  * [14] Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H.R., Xu, D.: Unetr: Transformers for 3d medical image segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 574–584 (2022)
  * [15] Isensee, F., Jaeger, P.F., et al.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [16] Karimi, D., Gholipour, A.: Improving calibration and out-of-distribution detection in deep models for medical image segmentation. IEEE Transactions on Artificial Intelligence (2022)
  * [17] Kavur, A.E., Gezer, N.S., Barış, M., Aslan, S., Conze, P.H., Groza, V., Pham, D.D., Chatterjee, S., Ernst, P., Özkan, S., et al.: Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation. Medical Image Analysis 69, 101950 (2021)
  * [18] Kerfoot, E., Clough, J., et al.: Left-ventricle quantification using residual u-net. Statistical Atlases and Computational Models of the Heart. Atrial Segmentation and LV Quantification Challenges Workshop, Held in Conjunction with MICCAI 2018 pp. 371–380 (2019)
  * [19] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [20] Kuijf, H.J., et al.: Standardized assessment of automatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge. IEEE transactions on medical imaging 38(11), 2556–2568 (2019)
  * [21] Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems 30 pp. 6402–6413 (2017)
  * [22] Liew, S.L., Lo, B.P., et al.: A large, curated, open-source stroke neuroimaging dataset to improve lesion segmentation algorithms. Scientific data 9(1), 320 (2022)
  * [23] Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T., Lakshminarayanan, B.: Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems 33, 7498–7512 (2020)
  * [24] Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. 2016 fourth international conference on 3D vision (3DV) pp. 565–571 (2016)
  * [25] Natalia, F., Meidia, H., et al.: Development of ground truth data for automatic lumbar spine MRI image segmentation. HPCC/SmartCity/DSS 2018 pp. 1449–1454 (2018)
  * [26] Oktay, O., Schlemper, J., et al.: Attention u-net: Learning where to look for the pancreas. Medical Imaging with Deep Learning (MIDL) (2018)
  * [27] Pérez-García, F., Rodionov, R., et al.: Simulation of brain resection for cavity segmentation using self-supervised and semi-supervised learning. In: Medical Image Computing and Computer Assisted Intervention (MICCAI) 2020. pp. 115–125. Springer (2020)
  * [28] Postels, J., et al.: On the practicality of deterministic epistemic uncertainty. ICML 162, 17870–17909 (2022)
  * [29] Srivastava, N., Hinton, G., et al.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1), 1929–1958 (2014)
  * [30] Suter, Y., Knecht, U., et al.: The lumiere dataset: Longitudinal glioblastoma mri with expert rano evaluation. Scientific data 9(1), 768 (2022)
  * [31] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
  * [32] Van Amersfoort, J., et al.: Uncertainty estimation using a single deep deterministic neural network. International conference on machine learning pp. 9690–9700 (2020)
  * [33] Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T.: Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing 338, 34–45 (2019)
  * [34] Wang, H., Zhao, C., et al.: Layer adaptive deep neural networks for out-of-distribution detection. Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference pp. 526–538 (2022)
  * [35] Zhou, Z., Siddiquee, R., et al.: Unet++: A nested u-net architecture for medical image segmentation. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (DLMIA) 2018, Held in Conjunction with MICCAI 2018 pp. 3–11 (2018)