# Interpretability Benchmark for Evaluating Spatial Misalignment
of Prototypical Parts Explanations

Mikołaj Sacha1, 2, Bartosz Jura3, Dawid Rymarczyk1,2,
Łukasz Struski1, Jacek Tabor1, Bartosz Zieliński1,4

###### Abstract

Prototypical parts-based networks are becoming increasingly popular due to
their faithful self-explanations. However, their similarity maps are
calculated in the penultimate network layer. Therefore, the receptive field of
the prototype activation region often depends on parts of the image outside
this region, which can lead to misleading interpretations. We name this
undesired behavior a spatial explanation misalignment and introduce an
interpretability benchmark with a set of dedicated metrics for quantifying
this phenomenon. In addition, we propose a method for misalignment
compensation and apply it to existing state-of-the-art models. We show the
expressiveness of our benchmark and the effectiveness of the proposed
compensation methodology through extensive empirical studies.

##  1 Introduction

The lack of insights into the reasons behind model predictions is a major
limitation of current deep learning-based systems, particularly in high-stake
decision fields like medicine and autonomous driving (Rudin 2019). As a
result, eXplainable Artificial Intelligence (XAI) has gained significant
attention in recent years, with two main branches of research being
extensively developed: post hoc and self-explainable methods (Rudin 2019).

The post hoc approaches assume that an explainer model needs to be developed
to explain the predictions of a classic deep neural network. However, this
approach may be biased and unreliable (Adebayo et al. 2018a). That is why
self-explainable methods were introduced, such as prototypical parts-based
methods (Chen et al. 2019). They contain built-in interpretability components
and provide interpretation along with the prediction.

Figure 1: The receptive field of the prototypical part activation region often
depends on parts of the image outside this region. In this example, the
activation region (green bounding box) depends on the upper left background
outside the bounding box. Therefore, after changing this part of the
background (as in the modified image), the location of the activation region
also changes (red bounding box). Such behavior is unwanted because it misleads
explanations.

Prototypical parts-based networks, such as ProtoPNet (Chen et al. 2019),
utilize feature-matching learning theory (Rosch 1975) to identify important
image parts by comparing them with reference patterns from training data.
However, despite its ability to provide highly faithful explanations, this
approach has known shortcomings, such as ambiguity of prototypical parts
(Nauta et al. 2020) or non-resistance to image modifications, like JPEG
compression (Hoffmann et al. 2021). Consequently, tools such as PRP (Gautam et
al. 2023) have been introduced to improve interpretability.

In this paper, we identify another risk related to the fact that prototypical
parts similarity maps are calculated in the penultimate network layer.
Therefore, the receptive field of the prototypical part activation region
often depends on parts of the image outside this region. It can result in
misleading explanations because users usually identify the activation region
with the receptive field, while, as presented in Fig. 1, this assumption is
only sometimes fulfilled.

To assess the explanation misalignment of prototypical parts-based methods, we
introduce an interpretability benchmark with a set of dedicated metrics. It
adversarially modifies the input image to reduce the high activation of a
prototypical part by changing only the image area where a particular
prototypical part is almost inactive. We decided to use adversarial
modification due to its flexibility which allows us to choose the modified
pixels and strength of the modification without altering the input image too
much. We use the original and modified images to compute easy-to-interpret
explanation misalignment metrics.

Except for the interpretability benchmark, we propose a novel compensation
methodology preserving the spatial relationship between the prototypical part
activation region and its receptive field. It is based on a novel loss
function computed for the image passed twice through the network, with and
without a mask, and masking-based augmentation.

Finally, we provide an extensive experimental evaluation of our compensation
methodology using state-of-the-art prototypical parts methods, showing the
effectiveness of our benchmark and the constructed compensation method.

Our contributions can be summarized as follows:

  * •

We systematize the limitations of the basic visualization of prototypical
parts activation, which can lead to misleading explanations.

  * •

We propose an interpretability benchmark for measuring the spatial
misalignment of the prototypical part activation region and its receptive
field to assess the reliability of explanations.

  * •

We introduce a novel compensation methodology for this misalignment that can
be used with any prototypical parts-based model.

##  2 Related works

Methods used to explain deep learning models can be classified into post hoc
and self-explainable (Rudin 2019). Post hoc methods assume that the reasoning
process is hidden within a black box model, and a new explainer model must be
created to reveal it. Some of those methods generate saliency maps (Rebuffi et
al. 2020; Selvaraju et al. 2019; Simonyan, Vedaldi, and Zisserman 2014) or use
Concept Activation Vectors (CAV) to construct explanation with user-friendly
concepts (Chen, Bei, and Rudin 2020; Ghorbani et al. 2019; Kim et al. 2018;
Yeh et al. 2020). Others provide counterfactual examples (Abbasnejad et al.
2020; Niu et al. 2021) or analyze the network’s reaction to image
perturbations (Basaj et al. 2021; Fong, Patrick, and Vedaldi 2019; Ribeiro,
Singh, and Guestrin 2016). The post hoc methods are easy to implement as they
do not interfere with the architecture. However, they may produce biased and
unreliable explanations (Adebayo et al. 2018b). Therefore, considerable effort
has been devoted to designing self-explainable models (Alvarez Melis and
Jaakkola 2018; Brendel and Bethge 2019) that make the internal decision
process visible for the user. Many interpretable solutions use attention
mechanisms (Liu et al. 2021; Zheng et al. 2017, 2019) or exploit the
activation space (Guidotti et al. 2020; Puyol-Antón et al. 2020), such as
adversarial autoencoders. However, the most recent approaches are built on an
interpretable method introduced in (Chen et al. 2019) (ProtoPNet) using a
hidden layer of prototypical parts to discover visual concepts.

Multiple self-explainable methods enhance ProtoPNet (Chen et al. 2019). TesNet
(Wang et al. 2021) constructs the latent space on a Grassman manifold. PIP-Net
(Nauta et al. 2023) redefines the prototypical parts layer to allow out-of-
distribution data detection. ProtoVAE (Gautam et al. 2022) leverages a
variational autoencoder with prototypical parts. ProtoPShare (Rymarczyk et al.
2021), ProtoTree (Nauta et al. 2021), ProtoKNN (Ukai et al. 2023), and
ProtoPool (Rymarczyk et al. 2022) reduce the number of prototypical parts used
in the classification. ProtoPShare introduces data-dependent merge-pruning
that discovers prototypical parts of similar semantics and joins them.
ProtoTree uses a soft neural decision tree that may depend on the negative
reasoning process and is extended to a visual transformer by (Kim, Nam, and Ko
2022). ProtoPool proposes differentiable prototypical parts to class
assignments while ProtoKNN adapts distance-based classifiers to prototypical
parts. At the same time, more alternative approaches organize the prototypical
parts hierarchically (Hase et al. 2019) to classify input at every level of a
predefined taxonomy or transform prototypical parts from the latent space to
data space (Li et al. 2018). Moreover, prototype-based solutions are widely
adopted in various fields such as medical imaging (Afnan et al. 2021; Barnett
et al. 2021; Kim et al. 2021; Rymarczyk et al. 2023a; Singh and Yow 2021),
time-series analysis (Gee et al. 2019), graphs analysis (Rymarczyk,
Dobrowolski, and Danel 2023; Zhang et al. 2022), semantic segmentation (Sacha
et al. 2023), deepfake detection (Trinh et al. 2021), zero-shot learning (Xu
et al. 2020), and continual learning (Rymarczyk et al. 2023b).

As the number of published prototypical parts-based methods grows, the
community starts to contemplate the correct ways of comparing them, using not
only accuracy. For example, (Hoffmann et al. 2021) investigates ProtoPNets
interpretability and discovers a semantic gap between similarity in input and
latent space. At the same time, (Etmann et al. 2019; Zhang and Zhu 2019;
Tsipras et al. 2018) highlight connections between the explainability of
machine learning models and their adversarial robustness. However, according
to our knowledge, no systematic benchmark has been proposed for a
comprehensive comparison of prototypical parts-based models, such as the one
we propose.

##  3 Preliminaries

### Prototypical parts network

To make this work self-contained, in this section, we describe the ProtoPNet
model (Chen et al. 2019), which introduces the prototypical parts layer. The
following paragraphs include the architecture, inference, and basic
visualization. We provide the ProtoPNets’ training schema in Supplement.

#### Architecture.

Prototypical parts networks (Chen et al. 2019) consist of a backbone
convolutional network $f$, a prototypical part layer $g$, and a fully
connected layer $h$. The prototypical part layer $g$ consists of $K$
prototypical parts $p\in\mathbb{R}^{D}$ per class, whose assignment is coded
in the fully connected layer $h$. If the prototypical part $p$ is assigned to
class $c$, then the weight between them equals $1$. Otherwise, it is set to
$-0.5$. We will denote the set of all prototypical parts as $P$ and the set of
prototypical parts of class $c$ as $P_{c}$.

#### Inference.

Given an input image $x$, its representation $f(x)$ of shape $H\times W\times
D$ is generated with a backbone $f$. The $H$ and $W$ represent height and
width after the last convolutional layer, where $D$ is its depth. Then, each
prototypical part $p$ is compared to each of $H\times W$ representation
vectors $z_{j}\in f(x)$ to find the maximum similarity (i.e. the maximal
activation of this prototypical part on the input image)

$\vspace{-0.5ex}g_{p}(x)=\max_{z_{j}\in f(x)}sim(p,z_{j}),\vspace{-0.5ex}$ (1)

where

$\vspace{-0.25ex}sim(p,z_{j})=\log\frac{|z_{j}-p|_{2}+1}{|z_{j}-p|_{2}+\eta}$
(2)

and $\eta\ll 1$. Suppose $s$ of dimension $W\times H$ refers to the similarity
map generated for the whole $f(x)$ representation

$\vspace{-0.5ex}s=sim(p,f(x)).$ (3)

In that case, the final prediction is obtained by pushing similarity values
through the fully connected layer $h$.

#### Visualization.

Visualization of the regions corresponding to prototypical parts is obtained
with similarity maps calculated by the layer $g$ before max pooling, upscaled
from $H\times W$ to the resolution of the input image, and overlayed. To
further simplify the visualization, the authors of ProtoPNet (Chen et al.
2019) take the $90$th percentile of this upscaled similarity map and draw a
bounding box around the highest activation values to mark the prototypical
part.

### Spatial misalignment of explanations

######  Definition 3.1.

Let us consider a similarity map $s_{i}=sim(p_{i},f(x))$ of the prototypical
part $p_{i}$ for an input image $x$ (as defined in Eq. 3). Moreover, let
$m_{i}$ be a binarized mask obtained by interpolating $s_{i}$ to the input
resolution and assigning a positive mask value to pixels with an activation
value above the $90$th percentile. We define the spatial misalignment of
explanations as:

$\vspace{-0.5ex}\Delta=\max_{\bar{x}_{i}}\|sim(p_{i},f(x))-sim(p_{i},f(\bar{x}_{i}))\|,\vspace{-0.5ex}$
(4)

where $\bar{x}_{i}$ correspond to any $x$ modification outside mask $m_{i}$.

Observe that if $\Delta=0$, then changes outside the mask do not impact the
explanation. Therefore, they are spatially aligned. However, the higher
$\Delta$, the larger the misalignment.

Figure 2: Prototypical part Location Change ($PLC$), similarly to the
remaining metrics, is a two-step process. The first step is similar for all
metrics. It calculates the similarity map for the maximally activated
prototypical part and adversarially modifies the image outside the activated
region (outside the green bounding box). As a result, the activation region
can change (red bounding box). The second step differs between metrics. In the
case of $PLC$, it measures the location change of activation before and after
modification (difference between green and red bounding box, respectively).

##  4 Interpretability benchmark

This section introduces a benchmark for evaluating the spatial misalignment of
prototypical part explanations. It consists of three metrics that can be
considered complementary to the performance metrics by the community
researchers.

The central idea is to analyze differences in prototypical part activations
(similarity maps) obtained for the original and adversarially modified image.
Modifications are made only outside the activation region of the original
image. Therefore, in the case of perfect alignment, there should be no
difference between prototypical part activations obtained for both images.
Otherwise, there can be differences in explanation location, activation, and
ranking, considered in the following metrics.

#### Adversarial modification.

To formally describe our benchmark, let us assume that $X$ is a test set of
images used to quantify the spatial misalignment of the model. Each $x\in X$
is passed through the backbone convolutional network $f$ and compared to all
prototypical parts using Eq. 2.

Let $\bar{p}$ be a prototypical part with the largest activation on $x$, i.e.
$\bar{p}=\operatorname*{arg\,max}_{p\in P}g_{p}(x)$. We calculate the
similarity map $\bar{s}=sim(\bar{p},f(x))$ and interpolate it bilinearly to
the input resolution. Then, following the visualization method from (Chen et
al. 2019), we construct a bounding box $b(x)$ defined as the smallest
rectangular region containing all activation above the $90$th percentile (see
green bounding box in Fig. 2). This region is presented to the user as the one
activated by the prototypical part $\bar{p}$.

We propagate the gradient from $\bar{p}$ back to the input image using the
_projected gradient descent_ (PGD) method (Papernot et al. 2018) to generate
an adversarially modified version of the input image $\bar{x}$ (see modified
image in Fig. 2) with the goal to minimize the $g_{\bar{p}}(x)$. However, in
contrast to standard PGD, we only modify the input pixels outside $b(x)$ (see
adversarial modification in Fig. 2).

If PGD manages to decrease $g_{\bar{p}}(x)$, then we deal with misalignment,
as it is possible to modify activations inside $b(x)$ by modifying the region
outside of it. This phenomenon is quantified using specialized _spatial
misalignment metrics_ , which are introduced in the following paragraph.

#### Spatial misalignment metrics.

Figure 3: Prototypical part Activation Change ($PAC$) measures the relative
difference between the maximum activation of the prototypical part before and
after adversarial modification. In this example, the activation of the
prototypical part drops substantially (by $75\%$), which indicates a
misalignment of the prototypical part explanation. Figure 4: Prototypical part
Rank Change ($PRC$) corresponds to the difference in ranking of the
prototypical part activations before and after modification. The ranking
calculates the number of prototypical parts from the classes other than the
class of the considered prototypical part with greater maximum similarity.
$PRC$ close to $0$ indicates that the explanation is spatially aligned.

In this paragraph, we describe our metrics for evaluating the spatial
misalignment of prototypical parts explanations. They all operate on the
original image $x$, the prototypical part $p_{i}$, the adversarially modified
image $\bar{x_{i}}$, and two similarity maps ($s_{i}$ and $\bar{s_{i}}$).

The first metric, Prototypical part Location Change ($PLC$), corresponds to
the change in the explanation location (see Fig. 2)

$\vspace{-0.5ex}PLC=1-\mathbb{E}_{x\in X}\frac{|b(x)\cap
b(\bar{x_{i}})|}{|b(x)\cup b(\bar{x_{i}})|},\vspace{-0.5ex}$ (5)

where $b(x)$ corresponds to the minimal rectangular region covering the
binarized mask obtained by assigning a positive mask value to pixels with an
activation value above the $90$th percentile. It quantifies how much the
explanation region can be relocated due to changes made by adversarial
modification. $PLC=0$ indicates that the region location remains unchanged.
However, a high $PLC$ value suggests a significant shift in the explanation
location.

The second metric, Prototypical part Activation Change ($PAC$), corresponds to
the relative difference between the maximum activation of the prototypical
part before and after adversarial modification (see Fig. 3)

$PAC=\mathbb{E}_{x\in
X}\frac{g_{p_{i}}(x)-g_{p_{i}}(\bar{x_{i}})}{g_{p_{i}}(x)}.$ (6)

It quantifies the impact of adversarial modification on prototypical part
activation. $PAC=0$ indicates no activation change, while higher $PAC$ values
correspond to high activation changes.

The third metric, Prototypical part Rank Change, corresponds to the difference
in ranking of the prototypical parts activations (see Fig. 4)

$PRC=\mathbb{E}_{x\in X}[r(\bar{x_{i}})-r(x)],$ (7)

where $r(x)=|\\{p\in P\setminus P_{k}\colon
g_{p}(\bar{x_{i}})>g_{p_{i}}(\bar{x_{i}})\\}|$ calculates the number of
prototypical parts $p$ from the classes other than the ground truth class of
$x$ (here noted as class $k$) with maximum activation greater than this
obtained by $\bar{p}$. $PRC=0$ means that $\bar{p}$ remains the most activated
prototypical part. However, the higher the $PRC$, the more prototypical parts
from other classes become increasingly important, indirectly indicating
misalignment.

As the final metric, we define the Accuracy Change (AC), which is equal to the
difference between the accuracy obtained for the original and modified images,
expressed in percentage points.

Figure 5: The spatially-aligned training aims to maximize the alignment of
activation maps of a selected prototypical part between the original image and
the image with only the highest activated fragment visible to the model. As
shown in the picture, the training step on a single image consists of two
passes of the model. During the first pass, we compute the model output
together with the intermediate similarity maps to the prototypical parts. In
the second pass, we randomly select a prototypical part from the ground-truth
class and use its similarity map to mask the image ($\bar{x}$). This way, we
obtain a new similarity map ($\bar{s}$), which we compare with the original
one, obtaining loss $L_{align}$.

##  5 Misalignment compensation

In this section, we propose a compensation methodology to prevent spatial
misalignment of prototypical parts explanations. It is a general strategy as
its only assumption is that the model calculates the prototypical parts
similarity map over the full input image at some point in its pipeline. Hence,
this strategy can be used with all state-of-the-art models based on
prototypical parts. The main idea behind our training strategy is to enforce
the alignment by passing the image through the network twice: the original
image and the image with the area outside the activation region masked.

To formalize our approach, let us assume that the considered prototypical part
model consists of a backbone convolutional network $f$ used to calculate the
similarity map $s_{i}=sim(p_{i},f(x))$ to a prototypical part $p_{i}$, as
defined in Eq. 3. Similar to the standard prototypical parts-based approaches,
the similarity maps are aggregated and classified, as presented in Fig. 5.
Additionally, we interpolate bilinearly the detached $s_{i}$ to the input
resolution and binarize it so that positive values correspond to activation
value above the 90th percentile, obtaining $m_{i}$. This mask is used to
generate a masked input $\bar{x_{i}}=x\cdot m_{i}$ that is used in the second
pass of the model. By passing $\bar{x_{i}}$ trough $f$ and comparing it to the
prototypical part $p_{i}$, we obtain a new similarity map
$\bar{s_{i}}=sim(\bar{p_{i}},f(x))$.

Figure 6: Sample images without (top row) and with (bottom row) the masking
augmentation. Each of the left three columns presents only one type of
masking, while the column on the right combines all of them. The number and
types of masking augmentations are randomized. The masking augmentation steers
the prototypical part model towards learning more locally-focused prototypical
parts.

For aligned explanation, $\bar{s_{i}}$ should be very similar to $s_{i}$
because the area outside the activation region should not influence the final
results. Therefore, we introduce a spatial alignment loss function, which
penalizes the model for not fulfilling this condition

$\vspace{-0.5ex}L_{align}=\|s_{i}-\bar{s_{i}}\|_{2},$ (8)

We weight this loss component with $\lambda_{align}$.

Figure 7: Comparison of the spatial misalignment metrics between baseline
prototype-based models (non-hatched bars) and the best variants achieved using
the spatial misalignment compensation methods (hatched bars).

#### Masking Augmentation.

To further prevent explanation misalignment, we consider a special type of
augmentation during model training (Fig. 6). For every training image, we
apply masking with a given probability. For the modified samples, we randomly
select a number of rectangular regions together with their widths, heights,
and locations. We randomly modify each region, either by adding noise or by
replacing the region with gray color or random noise.

##  6 Experimental setup

In this section, we discuss the experimental setup of our spatial misalignment
compensation. We describe other details related to training the compared
models in Supplement.

#### Masking augmentation setup.

For the variants of the trained models that employ masking augmentation, we
apply it during all phases of the training, augmenting each sampled training
image with the probability of 50%. We sample the number of modified image
regions between $1$ and $6$. For each region, we sample the augmentation type
out of the three options. The width and the height of each region are randomly
selected between $0.1$ and $0.5$ of the image width and height, respectively.
The location of each region is randomly selected from each possible location
that is fully within the image. All random values are sampled from the uniform
probability on the respective intervals.

#### Spatial misalignment benchmark setup.

To evaluate the spatial misalignment of the tested models, we perform the
spatial misalignment test on each image from the test set of CUB-200-2011
dataset (Wah et al. 2011). For each image, we select the top activated
prototypical part in the image for a given model and modify it adversarially
according to the procedure described in Sec. 4. We use the following
parameters for the projected gradient descent function used within the
benchmark: maximum total perturbation: 0.4; maximum perturbation within one
iteration: 0.01; number of iterations: 40.

##  7 Results

#### What is the level of the spatial misalignment for vanilla models and how
it can be decreased with our compensation method?

Fig. 7 illustrates the values for spatial misalignment metrics and
classification accuracy (in percents) achieved by the ProtoPNet, TesNet,
ProtoPool, and ProtoTree models, when trained with and without spatial-
misalignment compensation on the CUB-200-2011 dataset. We show the results for
baseline models and the best variants achieved with our compensation methods.
The implementation of our training technique yields a notable enhancement in
the robustness of the explanations, as gauged by the proposed metrics, as well
as increased stability in prediction accuracy across all tested prototypical
parts-based models. More details are in the Supplement.

Relative improvements, relative to each model’s baseline, exhibit varying
degrees of prominence. Notably, the highest gain is observed for ProtoPNet,
whereas ProtoPool demonstrates the least one. This can be related to the
ProtoPool’s specific focal similarity function, which is designed to generate
salient explanations. Comparatively, the enhancements for ProtoTree align more
closely with those observed for ProtoPNet which correlates with the findings
of (Nauta et al. 2023) emphasizing ProtoTree’s limitations in capturing atomic
parts of objects as prototypical parts. In the case of TesNet, its basic
version presents robust interpretations, particularly evident when considering
the $PAC$ metric. A comprehensive analysis of TesNet’s robustness is provided
in the subsequent paragraphs.

Furthermore, ProtoPNet’s explanations exhibit the highest susceptibility to
spatial misalignment (with the exception of the $PLC$ metric), while TesNet’s
interpretations show the least vulnerability to misalignment.

In terms of computations, masking augmentation has minimal impact on model
training. On the other hand, computing the spatial alignment loss necessitates
an additional pass, leading to an average 40% increase in training time.
Potentially, calculating the loss using only a subset of the training dataset
or specific image subregions may address it.

Figure 8: Ablation on the value of $\lambda_{align}$ for ProtoPNet and TesNet
models, trained without the masking augmentation. Figure 9: Ablation on the
spatial-alignment compensation without and with the masking augmentation
technique. Figure 10: Comparison of the spatial misalignment benchmark results
on the baseline ProtoPNet model and the variants with the spatial-aligning
loss ($\lambda=10$) and with the spatial-aligning loss enhanced by the
additional masking augmentation technique. Our variants achieve better
consistency between the similarity map before and after modifying the image,
indicating better spatial alignment.

#### How do the explanations differ between the baselines and improved models?

In Fig. 10, we show the results of the spatial misalignment benchmark for the
baseline ProtoPNet model, as well as for the ProtoPNet model trained with
$\lambda_{align}=10$ with and without the masking augmentation technique. The
examples were selected at random from the test set of the CUB-200-2011
dataset. We observe that the activation maps of the baseline model are
diminished by the test, while the model trained with the spatial-aligning loss
is robust to the modification of the image area outside the high-activation
bounding box. Better spatial alignment of our improved models is also
indicated by the results of the metrics shown below the images in the figure.
We present more such examples in the Supplementary Materials.

#### What is the optimal weight for spatial alignment loss?

To investigate the optimal value of the weighting factor for spatial alignment
loss, we trained the models varying the value of
$\lambda_{align}\in\\{0.0,0.1,1,5,10\\}$. For this ablation, we turned off the
masking augmentation technique. The results presented in Fig. 8 show that a
larger value of the $\lambda_{align}$ weight allows obtaining more spatially-
aligned explanations, as evidenced by the decreasing metric’s values, with
observable improvements for large weights ($\lambda_{align}\geq 1$). We
provide more detailed results in the Supplement.

#### What is the gain from using masking augmentation?

Fig. 9 shows how training with and without masking augmentation (MA)
influences the spatial misalignment metrics and the model’s accuracy. We
observe that, while applying the compensating loss improves the values of
metrics, the additional usage of masking augmentation combined with the
compensating loss yields the best results. We provide more detailed results in
Supplementary Materials.

#### Does the spatial misalignment compensation generalize to other model
backbones and datasets?

To evaluate the generalization of our approach, we conduct experiments using
the ProtoPNet (Chen et al. 2019) and TesNet (Wang et al. 2021) models with
VGG16 (Simonyan, Vedaldi, and Zisserman 2014) backbone (instead of ResNet), as
well as we benchmark the approach using the Stanford Cars dataset (Krause et
al. 2013). With the VGG16 backbone, both models show enhanced spatial
misalignment metrics, likely attributed to the narrower receptive field of
VGG. Moreover, similar trends for spatial misalignment are observed on the
Stanford Cars dataset, mirroring the behavior on CUB. More comprehensive
results are in the Supplement.

#### Why is TesNet so robust?

In order to investigate what makes TesNet so robust to the adversarial
modifications of our benchmark, we trained it without its specific loss terms
and applied the benchmark. Specifically, we trained TesNet 1) without the
subspace orthogonality loss ($\lambda_{orth}$), and 2) without both the
$\lambda_{orth}$ and the subspace-separation loss ($\lambda_{ss}$). Results of
these experiments are provided in Supplementary Materials. Classification
accuracy as well as metrics for these two models are comparable to those
obtained for the baseline TesNet model. $PAC$ metric tends even to be slightly
better than for the baseline TesNet model. These results might suggest that it
is the prototype similarity function used by TesNet, i.e., projection of the
latent space patches onto the prototype vectors (instead of a function of
$L^{2}$-distance as used by ProtoPNet), that is primarily responsible for the
superior performance of TesNet in our benchmark, as compared to results
obtained by the ProtoPNet model.

##  8 Conclusions

In this article, we discuss the limitations of prototypical parts-based
methods, such as ProtoPNet, caused by the misalignment between input and
representation space. To address this issue, we propose an interpretability
benchmark that measures this misalignment and introduce a novel compensation
methodology. Experimental evaluations show the adequacy of the proposed
benchmark and the effectiveness of the compensation methodology. With the
proposed spatial misalignment benchmark, we can automatically assess the
accuracy of explanations before presenting them to the user, thus avoiding
potential misinformation.

We hope this benchmark will improve the faithfulness of visualizations
generated by the prototypical parts-based models and strengthen research
dedicated to the automatic assessment of models’ interpretability.

## Acknowledgements

This research was partially funded by the National Science Centre, Poland,
grants no. 2021/41/B/ST6/01370 (work by Mikołaj Sacha and Jacek Tabor),
2022/45/N/ST6/04147 (work by Dawid Rymarczyk), 2020/39/D/ST6/01332 (work by
Łukasz Struski), and 2022/47/B/ST6/03397 (work by Bartosz Zieliński). The
research of Bartosz Jura was carried out within the research project “Bio-
inspired artificial neural network” (grant no. POIR.04.04.00-00-14DE/18-00)
within the Team-Net program of the Foundation for Polish Science co-financed
by the European Union under the European Regional Development Fund. Moreover,
Dawid Rymarczyk received an incentive scholarship from the funds of the
program Excellence Initiative – Research University at the Jagiellonian
University in Kraków. Finally, some experiments were performed on servers
purchased with funds from a Priority Research Area (Artificial Intelligence
Computing Center Core Facility) grant under the Strategic Programme Excellence
Initiative at Jagiellonian University.

## References

  * Abbasnejad et al. (2020) Abbasnejad, E.; Teney, D.; Parvaneh, A.; Shi, J.; and Hengel, A. v. d. 2020.  Counterfactual vision and language learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10044–10054.
  * Adebayo et al. (2018a) Adebayo, J.; Gilmer, J.; Muelly, M.; Goodfellow, I.; Hardt, M.; and Kim, B. 2018a.  Sanity checks for saliency maps.  _Advances in neural information processing systems_ , 31.
  * Adebayo et al. (2018b) Adebayo, J.; Gilmer, J.; Muelly, M.; Goodfellow, I.; Hardt, M.; and Kim, B. 2018b.  Sanity Checks for Saliency Maps.  In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Afnan et al. (2021) Afnan, M. A. M.; Liu, Y.; Conitzer, V.; Rudin, C.; Mishra, A.; Savulescu, J.; and Afnan, M. 2021.  Interpretable, not black-box, artificial intelligence should be used for embryo selection.  _Human Reproduction Open_.
  * Alvarez Melis and Jaakkola (2018) Alvarez Melis, D.; and Jaakkola, T. 2018.  Towards Robust Interpretability with Self-Explaining Neural Networks.  In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Barnett et al. (2021) Barnett, A. J.; Schwartz, F. R.; Tao, C.; Chen, C.; Ren, Y.; Lo, J. Y.; and Rudin, C. 2021.  IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography.  _arXiv preprint arXiv:2103.12308_.
  * Basaj et al. (2021) Basaj, D.; Oleszkiewicz, W.; Sieradzki, I.; Górszczak, M.; Rychalska, B.; Trzcinski, T.; and Zielinski, B. 2021.  Explaining Self-Supervised Image Representations with Visual Probing.  In _International Joint Conference on Artificial Intelligence_.
  * Brendel and Bethge (2019) Brendel, W.; and Bethge, M. 2019.  Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet.  In _International Conference on Learning Representations_.
  * Chen et al. (2019) Chen, C.; Li, O.; Tao, D.; Barnett, A.; Rudin, C.; and Su, J. K. 2019.  This looks like that: deep learning for interpretable image recognition.  In _NeurIPS_ , 8930–8941.
  * Chen, Bei, and Rudin (2020) Chen, Z.; Bei, Y.; and Rudin, C. 2020.  Concept whitening for interpretable image recognition.  _Nature Machine Intelligence_ , 2(12): 772–782.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Etmann et al. (2019) Etmann, C.; Lunz, S.; Maass, P.; and Schönlieb, C.-B. 2019.  On the connection between adversarial robustness and saliency map interpretability.  _arXiv preprint arXiv:1905.04172_.
  * Fong, Patrick, and Vedaldi (2019) Fong, R.; Patrick, M.; and Vedaldi, A. 2019.  Understanding deep networks via extremal perturbations and smooth masks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2950–2958.
  * Gautam et al. (2022) Gautam, S.; Boubekki, A.; Hansen, S.; Salahuddin, S. A.; Jenssen, R.; Höhne, M. M.; and Kampffmeyer, M. 2022.  ProtoVAE: A Trustworthy Self-Explainable Prototypical Variational Model.  In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., _Advances in Neural Information Processing Systems_.
  * Gautam et al. (2023) Gautam, S.; Höhne, M. M.-C.; Hansen, S.; Jenssen, R.; and Kampffmeyer, M. 2023\.  This looks more like that: Enhancing self-explaining models by prototypical relevance propagation.  _Pattern Recognition_ , 136: 109172.
  * Gee et al. (2019) Gee, A. H.; Garcia-Olano, D.; Ghosh, J.; and Paydarfar, D. 2019.  Explaining deep classification of time-series data with learned prototypes.  In _CEUR workshop proceedings_ , volume 2429, 15. NIH Public Access.
  * Ghorbani et al. (2019) Ghorbani, A.; Wexler, J.; Zou, J. Y.; and Kim, B. 2019.  Towards Automatic Concept-based Explanations.  In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Guidotti et al. (2020) Guidotti, R.; Monreale, A.; Matwin, S.; and Pedreschi, D. 2020.  Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(09): 13665–13668.
  * Hase et al. (2019) Hase, P.; Chen, C.; Li, O.; and Rudin, C. 2019.  Interpretable image recognition with hierarchical prototypes.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , volume 7, 32–40.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hoffmann et al. (2021) Hoffmann, A.; Fanconi, C.; Rade, R.; and Kohler, J. 2021.  This looks like that… does it? shortcomings of latent space prototype interpretability in deep networks.  _arXiv preprint arXiv:2105.02968_.
  * Kim et al. (2018) Kim, B.; Wattenberg, M.; Gilmer, J.; Cai, C.; Wexler, J.; Viegas, F.; et al. 2018\.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , 2668–2677. PMLR.
  * Kim et al. (2021) Kim, E.; Kim, S.; Seo, M.; and Yoon, S. 2021.  XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 15719–15728.
  * Kim, Nam, and Ko (2022) Kim, S.; Nam, J.; and Ko, B. C. 2022.  ViT-NeT: Interpretable Vision Transformers with Neural Tree Decoder.  In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , 11162–11172. PMLR.
  * Krause et al. (2013) Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013.  3D Object Representations for Fine-Grained Categorization.  In _2013 IEEE International Conference on Computer Vision Workshops_ , 554–561.
  * Li et al. (2018) Li, O.; Liu, H.; Chen, C.; and Rudin, C. 2018.  Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Liu et al. (2021) Liu, N.; Zhang, N.; Wan, K.; Shao, L.; and Han, J. 2021.  Visual saliency transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 4722–4732.
  * Nauta et al. (2020) Nauta, M.; Jutte, A.; Provoost, J.; and Seifert, C. 2020.  This Looks Like That, Because… Explaining Prototypes for Interpretable Image Recognition.  _arXiv preprint arXiv:2011.02863_.
  * Nauta et al. (2023) Nauta, M.; Schlötterer, J.; van Keulen, M.; and Seifert, C. 2023.  PIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2744–2753.
  * Nauta et al. (2021) Nauta, M.; et al. 2021.  Neural Prototype Trees for Interpretable Fine-grained Image Recognition.  In _CVPR_ , 14933–14943.
  * Niu et al. (2021) Niu, Y.; Tang, K.; Zhang, H.; Lu, Z.; Hua, X.-S.; and Wen, J.-R. 2021.  Counterfactual vqa: A cause-effect look at language bias.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12700–12710.
  * Papernot et al. (2018) Papernot, N.; Faghri, F.; Carlini, N.; Goodfellow, I.; Feinman, R.; Kurakin, A.; Xie, C.; Sharma, Y.; Brown, T.; Roy, A.; Matyasko, A.; Behzadan, V.; Hambardzumyan, K.; Zhang, Z.; Juang, Y.-L.; Li, Z.; Sheatsley, R.; Garg, A.; Uesato, J.; Gierke, W.; Dong, Y.; Berthelot, D.; Hendricks, P.; Rauber, J.; and Long, R. 2018.  Technical Report on the CleverHans v2.1.0 Adversarial Examples Library.  _arXiv preprint arXiv:1610.00768_.
  * Puyol-Antón et al. (2020) Puyol-Antón, E.; Chen, C.; Clough, J. R.; Ruijsink, B.; Sidhu, B. S.; Gould, J.; Porter, B.; Elliott, M.; Mehta, V.; Rueckert, D.; et al. 2020.  Interpretable deep models for cardiac resynchronisation therapy response prediction.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 284–293. Springer.
  * Rebuffi et al. (2020) Rebuffi, S.-A.; Fong, R.; Ji, X.; and Vedaldi, A. 2020.  There and back again: Revisiting backpropagation saliency methods.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8839–8848.
  * Ribeiro, Singh, and Guestrin (2016) Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016.  ” Why should i trust you?” Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 1135–1144.
  * Rosch (1975) Rosch, E. 1975.  Cognitive representations of semantic categories.  _Journal of experimental psychology: General_ , 104(3): 192.
  * Rudin (2019) Rudin, C. 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5): 206–215.
  * Rymarczyk, Dobrowolski, and Danel (2023) Rymarczyk, D.; Dobrowolski, D.; and Danel, T. 2023.  ProGReST: Prototypical Graph Regression Soft Trees for Molecular Property Prediction.  In _Proceedings of the 2023 SIAM International Conference on Data Mining (SDM)_ , 379–387. SIAM.
  * Rymarczyk et al. (2023a) Rymarczyk, D.; Pardyl, A.; Kraus, J.; Kaczyńska, A.; Skomorowski, M.; and Zieliński, B. 2023a.  ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classification.  In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I_ , 421–436. Springer.
  * Rymarczyk et al. (2022) Rymarczyk, D.; Struski, Ł.; Górszczak, M.; Lewandowska, K.; Tabor, J.; and Zieliński, B. 2022.  Interpretable image classification with differentiable prototypes assignment.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XII_ , 351–368. Springer.
  * Rymarczyk et al. (2023b) Rymarczyk, D.; van de Weijer, J.; Zieliński, B.; and Twardowski, B. 2023b.  ICICLE: Interpretable Class Incremental Continual Learning.  _International Conference on Computer Vision_.
  * Rymarczyk et al. (2021) Rymarczyk, D.; et al. 2021.  ProtoPShare: Prototypical Parts Sharing for Similarity Discovery in Interpretable Image Classification.  In _SIGKDD_ , 1420–1430.
  * Sacha et al. (2023) Sacha, M.; Rymarczyk, D.; Struski, Ł.; Tabor, J.; and Zieliński, B. 2023\.  ProtoSeg: Interpretable Semantic Segmentation With Prototypical Parts.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 1481–1492.
  * Selvaraju et al. (2019) Selvaraju, R. R.; Lee, S.; Shen, Y.; Jin, H.; Ghosh, S.; Heck, L.; Batra, D.; and Parikh, D. 2019.  Taking a hint: Leveraging explanations to make vision and language models more grounded.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2591–2600.
  * Simonyan, Vedaldi, and Zisserman (2014) Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2014.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _In Workshop at International Conference on Learning Representations_. Citeseer.
  * Singh and Yow (2021) Singh, G.; and Yow, K.-C. 2021.  These do not Look Like Those: An Interpretable Deep Learning Model for Image Recognition.  _IEEE Access_ , 9: 41482–41493.
  * Trinh et al. (2021) Trinh, L.; Tsang, M.; Rambhatla, S.; and Liu, Y. 2021.  Interpretable and Trustworthy Deepfake Detection via Dynamic Prototypes.  In _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 1972–1982.
  * Tsipras et al. (2018) Tsipras, D.; Santurkar, S.; Engstrom, L.; Turner, A.; and Madry, A. 2018.  Robustness may be at odds with accuracy.  _arXiv preprint arXiv:1805.12152_.
  * Ukai et al. (2023) Ukai, Y.; Hirakawa, T.; Yamashita, T.; and Fujiyoshi, H. 2023.  This Looks Like It Rather Than That: ProtoKNN For Similarity-Based Classifiers.  In _The Eleventh International Conference on Learning Representations_.
  * Wah et al. (2011) Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011.  The caltech-ucsd birds-200-2011 dataset.
  * Wang et al. (2021) Wang, J.; et al. 2021.  Interpretable Image Recognition by Constructing Transparent Embedding Space.  In _ICCV_ , 895–904.
  * Xu et al. (2020) Xu, W.; Xian, Y.; Wang, J.; Schiele, B.; and Akata, Z. 2020.  Attribute Prototype Network for Zero-Shot Learning.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20. Red Hook, NY, USA: Curran Associates Inc.  ISBN 9781713829546.
  * Yeh et al. (2020) Yeh, C.-K.; Kim, B.; Arik, S.; Li, C.-L.; Pfister, T.; and Ravikumar, P. 2020.  On Completeness-aware Concept-Based Explanations in Deep Neural Networks.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 20554–20565. Curran Associates, Inc.
  * Zhang and Zhu (2019) Zhang, T.; and Zhu, Z. 2019.  Interpreting adversarially trained convolutional neural networks.  In _International conference on machine learning_ , 7502–7511. PMLR.
  * Zhang et al. (2022) Zhang, Z.; Liu, Q.; Wang, H.; Lu, C.; and Lee, C. 2022.  ProtGNN: Towards Self-Explaining Graph Neural Networks.
  * Zheng et al. (2017) Zheng, H.; Fu, J.; Mei, T.; and Luo, J. 2017.  Learning multi-attention convolutional neural network for fine-grained image recognition.  In _Proceedings of the IEEE international conference on computer vision_ , 5209–5217.
  * Zheng et al. (2019) Zheng, H.; Fu, J.; Zha, Z.-J.; and Luo, J. 2019.  Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5012–5021.

## Interpretability Benchmark for Evaluating Spatial Misalignment -
Supplementary Material

## Preliminaries: ProtoPNet training

#### Training.

ProtoPNet learning is conducted in three optimization phases: warm-up, joint
learning, and convex optimization of the last layer. In the first phase, the
prototypical part layer $g$ is trained. In the second phase, $g$ and the
backbone network $f$ are trained. In the last phase, the fully-connected $h$
layer is fine-tuned. The cross-entropy loss with two regularizers, cluster and
separation costs (Chen et al. 2019), is used as the learning criterion.
Cluster cost enforces that for each training image, there is a latent patch
close to at least one prototypical part of its ground truth class. On the
other hand, the separation cost enforces every latent patch of an input image
to be far away from the prototypical parts of other classes.

## Details of the Experimental Setup

We run our experiments on several standard protypical part-based models for
image classification: ProtoPNet (Chen et al. 2019), TesNet (Wang et al. 2021),
ProtoPool (Rymarczyk et al. 2022) and ProtoTree (Nauta et al. 2021)111We use
the code provided by the authors for training the models:
https://github.com/cfchen-duke/ProtoPNet
https://github.com/JackeyWang96/TesNet
https://github.com/gmum/ProtoPool
https://github.com/M-Nauta/ProtoTree. We train and evaluate the models on the
CUB-200-2011 (Wah et al. 2011) dataset. Apart for adding the spatial alignment
compensation techniques, we use the same training procedure as in the original
publications describing the models. We run multi-phase training with warmup,
joint training, prototypical part projection, prototypical part pruning and
last layer fine-tuning phases (Chen et al. 2019), and employ resnet34 (He et
al. 2016) network pretrained on ImageNet (Deng et al. 2009) as the model
backbone. We train each of the models for $\lambda_{align}\in\\{0,\allowbreak
0.1,\allowbreak 1,\allowbreak 5,\allowbreak 10\\}$ and with or without the
masking augmentation training (MA), yielding 10 variants for each of the
architectures. For each variant, we run three training runs with different
random number generator seeds and take the average of their metrics as the
final result. For the variants with non-zero $\lambda_{align}$, we add this
loss with the weight of $\lambda_{align}$ to the loss value used by the
original prototypical part-based model.

## Visual examples of spatial alignment compensation

In this section, we show more examples that compare the spatial alignment
benchmark results on the baseline ProtoPNet to the ProtoPNet with spatial
aligning loss ($\lambda=10$) and with the spatial-aligning loss enhanced by
the additional masking augmentation technique.

## Detailed results of spatial-aligning compensation

In the following sections, we show the detailed comparison of spatial
alignment metrics between baseline methods and methods improved by the
spatial-aligning loss and masking augmentation. For each of the evaluated
models, we show the values of Prototypical part Location Change (PLC),
Prototypical part Rank Change (PRC), Prototypical part Activation Change
(PAC), as well as the classification accuracy before (Acc. Before), and after
(Acc. After) the adversarial modification and the difference in percentage
points between these accuracy values (Accuracy Change - AC).

### Statistics over the values of metrics

In Fig. 11, we draw kernel density estimation plots over histograms of all
values of the spatial alignment metrics achieved by the trained ProtoPNet
models on the test set of CUB-200-2011. We observe that, on average, our
variants achieve better (lower) values of metrics, with the values of the
metrics getting better with both increasing $\lambda_{align}$ and employing
the masking augmentation.

Figure 11: Kernel density estimation over histograms of metrics values based
on the test images from CUB-200-2011 for the ProtoPNet variants trained with
different values of $\lambda_{align}$ as well as with and without the masking
augmentation (MA). We observe that, on average, our variants achieve better
(lower) metrics values, getting better with both increasing $\lambda_{align}$
and MA.

### Detailed results on different prototypical part-based models

In Tabs. 1, 2, 3 and 4 ((Chen et al. 2019; Wang et al. 2021; Rymarczyk et al.
2022; Nauta et al. 2021)), we compare the values of spatial alignment metrics
between baseline methods and methods improved by the spatial-aligning loss and
masking augmentation.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 24.0 $\pm$ 1.7 | 13.5 $\pm$ 3.1 | 23.7 $\pm$ 2.8 | 76.4 $\pm$ 0.2 | 68.2 $\pm$ 0.9 | 8.2 $\pm$ 1.1
ours | 0.1 | ✗ | 22.0 $\pm$ 0.6 | 10.3 $\pm$ 1.1 | 19.8 $\pm$ 0.3 | 76.9 $\pm$ 0.3 | 69.5 $\pm$ 0.6 | 7.4 $\pm$ 0.3
1 | 22.0 $\pm$ 1.0 | 9.0 $\pm$ 1.5 | 6.5 $\pm$ 0.5 | 77.2 $\pm$ 0.3 | 70.1 $\pm$ 0.8 | 7.1 $\pm$ 1.1
5 | 20.7 $\pm$ 1.3 | 7.8 $\pm$ 0.9 | 4.8 $\pm$ 0.8 | 76.8 $\pm$ 0.2 | 70.0 $\pm$ 0.5 | 6.8 $\pm$ 1.0
10 |  | 18.3 $\pm$ 1.7 | 4.8 $\pm$ 1.1 | 2.9 $\pm$ 1.1 | 73.3 $\pm$ 0.5 | 66.5 $\pm$ 0.5 | 6.8 $\pm$ 0.6
0 | ✓ | 16.7 $\pm$ 0.3 | 1.8 $\pm$ 0.1 | 5.0 $\pm$ 0.7 | 78.6 $\pm$ 0.6 | 77.0 $\pm$ 0.7 | 1.6 $\pm$ 0.3
0.1 | 15.7 $\pm$ 0.3 | 1.7 $\pm$ 0.1 | 6.3 $\pm$ 0.2 | 78.1 $\pm$ 0.2 | 75.9 $\pm$ 0.2 | 2.2 $\pm$ 0.1
1 | 15.3 $\pm$ 0.3 | 2.1 $\pm$ 0.1 | 2.7 $\pm$ 0.1 | 76.7 $\pm$ 0.2 | 74.7 $\pm$ 0.1 | 2.0 $\pm$ 0.2
| 5 | 13.3 $\pm$ 0.8 | 1.8 $\pm$ 0.4 | 2.5 $\pm$ 0.3 | 76.7 $\pm$ 0.2 | 74.6 $\pm$ 0.3 | 2.1 $\pm$ 0.4
| 10 |  | 8.5 $\pm$ 1.5 | 1.6 $\pm$ 0.8 | 1.7 $\pm$ 0.7 | 76.5 $\pm$ 0.1 | 74.0 $\pm$ 1.0 | 2.5 $\pm$ 0.9

Table 1: Metrics on the spatial-alignment test acquired by the baseline
ProtoPNet model and ProtoPNet trained with the spatial-aligning training and
masking augmentation. The variant with the spatially-aligning loss with weight
$\lambda_{align}=10$ and masking augmentation achieves overall the best
results, and it was used in the main paper results.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 16.0 $\pm$ 0.0 | 2.9 $\pm$ 0.3 | 3.4 $\pm$ 0.3 | 81.6 $\pm$ 0.2 | 75.8 $\pm$ 0.5 | 5.8 $\pm$ 0.6
ours | 0.1 | ✗ | 13.7 $\pm$ 0.3 | 1.8 $\pm$ 0.2 | 3.0 $\pm$ 0.3 | 81.5 $\pm$ 0.1 | 75.2 $\pm$ 0.0 | 6.3 $\pm$ 0.1
1 | 13.7 $\pm$ 0.7 | 2.0 $\pm$ 0.4 | 3.4 $\pm$ 0.8 | 81.4 $\pm$ 0.0 | 75.5 $\pm$ 0.3 | 5.9 $\pm$ 0.3
5 | 13.7 $\pm$ 0.3 | 1.1 $\pm$ 0.1 | 2.7 $\pm$ 0.1 | 79.9 $\pm$ 0.5 | 74.7 $\pm$ 0.7 | 5.2 $\pm$ 0.3
10 | 13.7 $\pm$ 0.3 | 1.4 $\pm$ 0.2 | 2.3 $\pm$ 0.4 | 79.1 $\pm$ 0.4 | 74.2 $\pm$ 0.9 | 4.9 $\pm$ 0.8
0 | ✓ | 9.0 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 0.1 $\pm$ 0.0 | 82.6 $\pm$ 0.2 | 81.5 $\pm$ 0.2 | 1.9 $\pm$ 0.1
0.1 | 9.0 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 0.3 $\pm$ 0.1 | 82.4 $\pm$ 0.2 | 80.4 $\pm$ 0.2 | 2.0 $\pm$ 0.1
1 | 9.0 $\pm$ 0.0 | 0.4 $\pm$ 0.1 | 0.8 $\pm$ 0.1 | 82.2 $\pm$ 0.2 | 80.3 $\pm$ 0.2 | 1.9 $\pm$ 0.1
5 | 9.0 $\pm$ 0.0 | 0.3 $\pm$ 0.0 | 0.8 $\pm$ 0.2 | 80.7 $\pm$ 0.5 | 79.4 $\pm$ 0.6 | 1.4 $\pm$ 0.2
10 | 10.0 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 0.8 $\pm$ 0.1 | 79.3 $\pm$ 0.5 | 77.1 $\pm$ 0.7 | 2.2 $\pm$ 0.3

Table 2: Metrics on the spatial-alignment test acquired by the baseline TesNet
model and TesNet trained with the spatial-aligning training and masking
augmentation. The variant with no spatially-aligning loss
($\lambda_{align}=0$) and masking augmentation achieves overall the best
results, and it was used in the main paper results.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 31.8 $\pm$ 0.8 | 4.5 $\pm$ 0.9 | 11.2 $\pm$ 1.3 | 80.8 $\pm$ 0.2 | 76.0 $\pm$ 0.3 | 4.8 $\pm$ 0.1
ours | 0.1 | ✗ | 30.5 $\pm$ 0.9 | 4.6 $\pm$ 0.3 | 11.5 $\pm$ 0.5 | 81.1 $\pm$ 0.4 | 75.9 $\pm$ 0.7 | 6.2 $\pm$ 0.3
1 | 30.8 $\pm$ 1.3 | 3.7 $\pm$ 0.4 | 8.6 $\pm$ 0.6 | 81.5 $\pm$ 0.1 | 76.0 $\pm$ 0.4 | 5.5 $\pm$ 0.3
5 | 31.4 $\pm$ 1.5 | 3.7 $\pm$ 0.4 | 8.7 $\pm$ 0.5 | 81.3 $\pm$ 0.1 | 76.4 $\pm$ 0.3 | 4.9 $\pm$ 0.3
10 |  | 32.4 $\pm$ 1.5 | 3.8 $\pm$ 0.5 | 8.9 $\pm$ 0.8 | 81.3 $\pm$ 0.1 | 76.9 $\pm$ 0.5 | 4.4 $\pm$ 0.4
0 | ✓ | 32.4 $\pm$ 1.0 | 3.4 $\pm$ 0.6 | 9.3 $\pm$ 1.1 | 80.9 $\pm$ 0.2 | 76.3 $\pm$ 0.2 | 4.6 $\pm$ 0.2
0.1 | 31.1 $\pm$ 1.6 | 3.7 $\pm$ 0.4 | 9.8 $\pm$ 0.5 | 80.4 $\pm$ 0.4 | 76.4 $\pm$ 0.2 | 4.0 $\pm$ 0.5
1 | 32.1 $\pm$ 1.9 | 3.5 $\pm$ 0.3 | 9.0 $\pm$ 0.9 | 81.5 $\pm$ 0.1 | 76.8 $\pm$ 0.3 | 4.7 $\pm$ 0.4
| 5 | 31.5 $\pm$ 1.2 | 3.6 $\pm$ 0.4 | 8.8 $\pm$ 0.8 | 81.4 $\pm$ 0.1 | 76.4 $\pm$ 0.5 | 5.1 $\pm$ 0.5
| 10 |  | 30.0 $\pm$ 1.6 | 4.1 $\pm$ 0.2 | 8.4 $\pm$ 0.7 | 81.2 $\pm$ 0.5 | 77.4 $\pm$ 0.6 | 3.8 $\pm$ 0.3

Table 3: Metrics on the spatial-alignment test acquired by the baseline
ProtoPool model and ProtoPool trained with the spatial-aligning training and
masking augmentation. The variant with the spatially-aligning loss with weight
$\lambda_{align}=10$ and masking augmentation achieves overall the best
results, and it was used in the main paper results.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 27.7 $\pm$ 0.3 | 12.7 $\pm$ 1.3 | 6.5 $\pm$ 0.1 | 82.5 $\pm$ 0.2 | 70.4 $\pm$ 1.2 | 12.1 $\pm$ 1.0
ours | 0.1 | ✗ | 28.0 $\pm$ 0.6 | 12.8 $\pm$ 0.6 | 6.2 $\pm$ 0.2 | 79.6 $\pm$ 2.3 | 66.5 $\pm$ 1.3 | 13.1 $\pm$ 1.4
1 | 28.0 $\pm$ 0.6 | 12.3 $\pm$ 0.9 | 5.3 $\pm$ 0.5 | 79.6 $\pm$ 2.3 | 66.5 $\pm$ 1.3 | 13.1 $\pm$ 1.4
5 | 24.3 $\pm$ 1.3 | 11.2 $\pm$ 0.3 | 4.7 $\pm$ 1.4 | 81.6 $\pm$ 0.2 | 66.4 $\pm$ 2.0 | 15.2 $\pm$ 2.2
10 | 21.7 $\pm$ 0.9 | 12.2 $\pm$ 0.8 | 5.6 $\pm$ 1.1 | 81.2 $\pm$ 1.6 | 67.4 $\pm$ 2.8 | 13.8 $\pm$ 1.3
0 | ✓ | 21.7 $\pm$ 0.3 | 8.4 $\pm$ 0.5 | 2.5 $\pm$ 0.4 | 82.9 $\pm$ 0.5 | 78.0 $\pm$ 0.6 | 4.9 $\pm$ 0.2
0.1 | 20.3 $\pm$ 0.3 | 5.4 $\pm$ 0.1 | 2.9 $\pm$ 0.1 | 81.9 $\pm$ 0.2 | 77.3 $\pm$ 0.3 | 4.6 $\pm$ 0.1
1 | 20.7 $\pm$ 0.9 | 6.2 $\pm$ 0.7 | 1.7 $\pm$ 0.5 | 82.9 $\pm$ 0.3 | 78.0 $\pm$ 0.5 | 4.9 $\pm$ 0.2
5 | 17.7 $\pm$ 1.5 | 5.9 $\pm$ 0.5 | 2.3 $\pm$ 0.8 | 81.7 $\pm$ 0.6 | 76.6 $\pm$ 0.6 | 5.1 $\pm$ 0.1
10 | 16.7 $\pm$ 0.3 | 5.7 $\pm$ 0.0 | 2.7 $\pm$ 0.3 | 72.4 $\pm$ 1.5 | 66.8 $\pm$ 1.5 | 5.7 $\pm$ 0.6

Table 4: Metrics on the spatial-alignment test acquired by the baseline
ProtoTree model and ProtoTree trained with the spatial-aligning training and
masking augmentation. The variant with the spatially-aligning loss with weight
$\lambda_{align}=1$ and masking augmentation achieves overall the best
results, and it was used in the main paper results.

### Detailed results on VGG backbone

In Tabs. 5 and 6, we show the detailed comparison of spatial alignment metrics
between baseline methods and methods improved by the spatial-aligning loss and
masking augmentation for ProtoPNet (Chen et al. 2019) and TesNet (Wang et al.
2021) trained using VGG16 backbone instead of Resnet34.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 17.9 $\pm$ 0.4 | 1.2 $\pm$ 0.1 | 7.0 $\pm$ 0.3 | 71.0 $\pm$ 0.2 | 64.9 $\pm$ 0.3 | 4.8 $\pm$ 0.1
ours | 0.1 | ✗ | 16.7 $\pm$ 0.3 | 0.9 $\pm$ 0.0 | 5.4 $\pm$ 0.2 | 71.3 $\pm$ 0.1 | 65.2 $\pm$ 0.1 | 6.1 $\pm$ 0.1
1 | 17.0 $\pm$ 0.5 | 1.1 $\pm$ 0.1 | 4.9 $\pm$ 0.3 | 72.7 $\pm$ 0.2 | 66.7 $\pm$ 0.5 | 5.9 $\pm$ 0.6
5 | 18.2 $\pm$ 0.3 | 1.8 $\pm$ 0.1 | 5.5 $\pm$ 0.7 | 73.6 $\pm$ 0.1 | 67.8 $\pm$ 0.1 | 5.8 $\pm$ 0.1
10 |  | 17.9 $\pm$ 0.3 | 1.8 $\pm$ 0.2 | 4.9 $\pm$ 0.7 | 73.6 $\pm$ 0.2 | 67.9 $\pm$ 0.3 | 5.7 $\pm$ 0.4
0 | ✓ | 12.9 $\pm$ 0.3 | 0.5 $\pm$ 0.0 | 3.6 $\pm$ 0.1 | 73.2 $\pm$ 0.1 | 70.4 $\pm$ 0.4 | 2.8 $\pm$ 0.5
0.1 | 11.8 $\pm$ 0.3 | 0.4 $\pm$ 0.0 | 2.9 $\pm$ 0.1 | 74.2 $\pm$ 0.2 | 71.4 $\pm$ 0.1 | 2.8 $\pm$ 0.2
1 | 12.0 $\pm$ 0.2 | 0.5 $\pm$ 0.0 | 2.2 $\pm$ 0.1 | 74.6 $\pm$ 0.1 | 72.0 $\pm$ 0.1 | 2.5 $\pm$ 0.1
| 5 | 12.6 $\pm$ 0.2 | 0.6 $\pm$ 0.0 | 2.6 $\pm$ 0.1 | 75.1 $\pm$ 0.3 | 72.6 $\pm$ 0.2 | 2.5 $\pm$ 0.1
| 10 |  | 13.2 $\pm$ 0.2 | 0.8 $\pm$ 0.0 | 3.0 $\pm$ 0.1 | 74.8 $\pm$ 0.3 | 72.3 $\pm$ 0.2 | 2.6 $\pm$ 0.2

Table 5: Metrics on the spatial-alignment test acquired by the baseline
ProtoPNet model and ProtoPNet trained with the spatial-aligning training and
masking augmentation, using VGG16 backbone instead of Resnet34.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 15.0 $\pm$ 0.6 | 0.9 $\pm$ 0.1 | 1.0 $\pm$ 0.1 | 79.3 $\pm$ 0.1 | 72.6 $\pm$ 0.2 | 6.7 $\pm$ 0.4
ours | 0.1 | ✗ | 15.0 $\pm$ 0.6 | 0.8 $\pm$ 0.1 | 1.2 $\pm$ 0.1 | 79.4 $\pm$ 0.2 | 72.1 $\pm$ 0.3 | 7.4 $\pm$ 0.5
1 | 12.7 $\pm$ 0.3 | 0.6 $\pm$ 0.0 | 1.3 $\pm$ 0.1 | 80.0 $\pm$ 0.2 | 73.4 $\pm$ 0.4 | 6.6 $\pm$ 0.4
5 | 13.3 $\pm$ 0.3 | 0.5 $\pm$ 0.1 | 1.0 $\pm$ 0.1 | 79.5 $\pm$ 0.2 | 71.9 $\pm$ 0.5 | 7.6 $\pm$ 0.5
10 | 12.3 $\pm$ 0.3 | 0.3 $\pm$ 0.0 | 0.6 $\pm$ 0.0 | 78.5 $\pm$ 0.1 | 71.7 $\pm$ 0.4 | 6.8 $\pm$ 0.4
0 | ✓ | 9.0 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 0.4 $\pm$ 0.1 | 80.4 $\pm$ 0.1 | 78.0 $\pm$ 0.3 | 2.3 $\pm$ 0.2
0.1 | 9.0 $\pm$ 0.0 | 0.3 $\pm$ 0.0 | 0.4 $\pm$ 0.1 | 80.5 $\pm$ 0.2 | 78.3 $\pm$ 0.2 | 2.3 $\pm$ 0.2
1 | 9.0 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 80.8 $\pm$ 0.3 | 78.7 $\pm$ 0.2 | 2.1 $\pm$ 0.2
5 | 9.0 $\pm$ 0.0 | 0.2 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 80.4 $\pm$ 0.2 | 78.1 $\pm$ 0.2 | 2.2 $\pm$ 0.1
10 | 9.0 $\pm$ 0.6 | 0.2 $\pm$ 0.0 | 0.3 $\pm$ 0.0 | 79.0 $\pm$ 0.2 | 76.6 $\pm$ 0.1 | 2.4 $\pm$ 0.1

Table 6: Metrics on the spatial-alignment test acquired by the baseline TesNet
model and TesNet trained with the spatial-aligning training and masking
augmentation, using VGG16 backbone instead of Resnet34.

### Detailed results on Stanford Cars dataset

In Tabs. 7 and 8, we show the detailed comparison of spatial alignment metrics
between baseline methods and methods improved by the spatial-aligning loss and
masking augmentation for ProtoPNet (Chen et al. 2019) and TesNet (Wang et al.
2021) trained and evaluated on Stanford Cars dataset (Krause et al. 2013).

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 35.3 $\pm$ 0.0 | 6.4 $\pm$ 0.1 | 28.4 $\pm$ 0.1 | 88.1 $\pm$ 0.0 | 78.1 $\pm$ 0.2 | 10.0 $\pm$ 0.2
ours | 0.1 | ✗ | 33.0 $\pm$ 2.4 | 5.4 $\pm$ 0.5 | 18.0 $\pm$ 0.8 | 87.9 $\pm$ 0.2 | 77.8 $\pm$ 1.4 | 10.1 $\pm$ 1.2
1 | 24.3 $\pm$ 2.9 | 12.4 $\pm$ 2.1 | 2.1 $\pm$ 0.7 | 88.2 $\pm$ 0.4 | 78.1 $\pm$ 1.0 | 10.2 $\pm$ 0.9
5 | 22.7 $\pm$ 2.8 | 7.5 $\pm$ 1.6 | 1.9 $\pm$ 0.6 | 88.1 $\pm$ 0.4 | 78.1 $\pm$ 1.0 | 10.0 $\pm$ 0.9
10 |  | 21.8 $\pm$ 2.5 | 8.4 $\pm$ 1.5 | 1.7 $\pm$ 0.5 | 87.8 $\pm$ 0.2 | 78.5 $\pm$ 1.2 | 10.2 $\pm$ 1.1
0 | ✓ | 35.2 $\pm$ 1.0 | 6.6 $\pm$ 0.5 | 28.3 $\pm$ 0.7 | 88.1 $\pm$ 0.2 | 77.8 $\pm$ 0.4 | 10.4 $\pm$ 0.4
0.1 | 25.1 $\pm$ 1.3 | 1.3 $\pm$ 0.3 | 8.1 $\pm$ 0.8 | 88.0 $\pm$ 0.1 | 84.9 $\pm$ 0.4 | 3.0 $\pm$ 0.4
1 | 15.6 $\pm$ 0.5 | 3.0 $\pm$ 0.2 | 1.6 $\pm$ 0,6 | 88.0 $\pm$ 0.1 | 85.2 $\pm$ 0.2 | 2.8 $\pm$ 0.3
| 5 | 13.5 $\pm$ 0.8 | 2.9 $\pm$ 0.2 | 1.4 $\pm$ 0.6 | 88.1 $\pm$ 0.1 | 86.1 $\pm$ 0.3 | 2.0 $\pm$ 0.3
| 10 |  | 12.5 $\pm$ 0.9 | 1.2 $\pm$ 0.3 | 1.3 $\pm$ 0.7 | 88.0 $\pm$ 0.1 | 86.4 $\pm$ 0.2 | 1.4 $\pm$ 0.2

Table 7: Metrics on the spatial-alignment test acquired by the baseline
ProtoPNet model and ProtoPNet trained with the spatial-aligning training and
masking augmentation, trained and evaluated on the Stanford Cars dataset.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 15.3 $\pm$ 0.3 | 1.7 $\pm$ 0.3 | 1.8 $\pm$ 0.2 | 91.8 $\pm$ 0.0 | 86.6 $\pm$ 0.2 | 5.2 $\pm$ 0.2
ours | 0.1 | ✗ | 15.3 $\pm$ 0.7 | 1.1 $\pm$ 0.1 | 1.1 $\pm$ 0.3 | 91.3 $\pm$ 0.1 | 86.1 $\pm$ 0.2 | 5.2 $\pm$ 0.3
1 | 13.7 $\pm$ 0.3 | 0.8 $\pm$ 0.1 | 2.7 $\pm$ 0.3 | 91.2 $\pm$ 0.1 | 86.7 $\pm$ 0.1 | 4.5 $\pm$ 0.1
5 | 16.7 $\pm$ 0.3 | 1.1 $\pm$ 0.1 | 3.0 $\pm$ 0.2 | 89.8 $\pm$ 0.2 | 84.6 $\pm$ 0.4 | 5.2 $\pm$ 0.3
10 | 15.0 $\pm$ 0.0 | 0.7 $\pm$ 0.1 | 1.3 $\pm$ 0.2 | 89.4 $\pm$ 0.1 | 85.1 $\pm$ 0.1 | 4.3 $\pm$ 0.0
0 | ✓ | 9.7 $\pm$ 0.3 | 0.5 $\pm$ 0.1 | 0.1 $\pm$ 0.1 | 92.2 $\pm$ 0.1 | 90.9 $\pm$ 0.1 | 1.3 $\pm$ 0.1
0.1 | 10.0 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 0.1 $\pm$ 0.1 | 91.9 $\pm$ 0.1 | 90.1 $\pm$ 0.2 | 1.8 $\pm$ 0.2
1 | 9.0 $\pm$ 0.0 | 0.3 $\pm$ 0.0 | 0.4 $\pm$ 0.1 | 91.7 $\pm$ 0.1 | 90.4 $\pm$ 0.2 | 1.3 $\pm$ 0.2
5 | 10.3 $\pm$ 0.3 | 0.3 $\pm$ 0.0 | 0.6 $\pm$ 0.0 | 90.7 $\pm$ 0.1 | 89.5 $\pm$ 0.1 | 1.2 $\pm$ 0.1
10 | 10.7 $\pm$ 0.3 | 0.2 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 90.1 $\pm$ 0.3 | 88.6 $\pm$ 0.3 | 1.5 $\pm$ 0.1

Table 8: Metrics on the spatial-alignment test acquired by the baseline TesNet
model and TesNet trained with the spatial-aligning training and masking
augmentation, trained and evaluated on the Stanford Cars dataset.

### Why is TesNet so robust?

In Tab. 9 shown are results of the benchmark applied to TesNet trained without
its specific loss terms (as described in the main text).

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 16.0 $\pm$ 0.0 | 2.9 $\pm$ 0.3 | 3.4 $\pm$ 0.3 | 81.6 $\pm$ 0.2 | 75.8 $\pm$ 0.5 | 5.8 $\pm$ 0.6
baseline w/o $\lambda_{orth}$ | ✗ | ✗ | 16.0 $\pm$ 0.0 | 3.7 $\pm$ 0.6 | 2.2 $\pm$ 0.1 | 80.0 $\pm$ 0.3 | 73.7 $\pm$ 0.1 | 6.3 $\pm$ 0.2
baseline w/o $\lambda_{orth}$, $\lambda_{ss}$ | ✗ | ✗ | 15.0 $\pm$ 0.6 | 2.2 $\pm$ 0.5 | 1.6 $\pm$ 0.1 | 80.6 $\pm$ 0.3 | 74.6 $\pm$ 0.1 | 6.0 $\pm$ 0.3

Table 9: Metrics on the spatial-alignment test acquired by the TesNet trained
without the TesNet-specific losses.

### Results of spatial-aligning training with a smooth mask

In Tabs. 10 and 11 we provide results of some additional experiments, obtained
for a variant of spatial-aligning training with a smooth mask. In the main
series of experiments with the $\lambda_{align}$ loss (described above and in
the main text), in the seconds pass we used a binarized mask modifying the
original image, that is, mask that was equal 1 inside the highly activated
region $b(x)$, and 0 outside of $b(x)$. In order to examine possible effects
of such binarization, we conducted additional experiments, this time with a
smooth mask, that is, mask taking values from the continuous range $[0,1]$,
proportional to the values of the prototype similarity map $sim(p,f(x))$. The
results for TesNet and ProtoTree models suggest that such a ’blurred’ mask
during training is generally not effective in spatially aligning the
prototypes (and might have a strong negative effect on the network
performance, as evidenced by results for ProtoTree), in contrast to the
binarized version of the mask, which is effective in aligning the prototypes.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 16.0 $\pm$ 0.0 | 2.9 $\pm$ 0.3 | 3.4 $\pm$ 0.3 | 81.6 $\pm$ 0.2 | 75.8 $\pm$ 0.5 | 5.8 $\pm$ 0.6
ours | 0.1 | ✗ | 17.7 $\pm$ 0.3 | 3.4 $\pm$ 0.1 | 4.1 $\pm$ 0.4 | 81.5 $\pm$ 0.2 | 74.0 $\pm$ 0.2 | 7.5 $\pm$ 0.2
1 | 16.0 $\pm$ 0.0 | 3.1 $\pm$ 0.2 | 3.9 $\pm$ 0.2 | 81.3 $\pm$ 0.2 | 74.9 $\pm$ 0.3 | 6.4 $\pm$ 0.1
10 | 15.0 $\pm$ 0.6 | 1.9 $\pm$ 0.3 | 5.1 $\pm$ 0.4 | 80.8 $\pm$ 0.1 | 75.1 $\pm$ 0.3 | 5.7 $\pm$ 0.2
0 | ✓ | 9.0 $\pm$ 0.0 | 0.4 $\pm$ 0.0 | 0.1 $\pm$ 0.0 | 82.6 $\pm$ 0.2 | 81.5 $\pm$ 0.2 | 1.9 $\pm$ 0.1
0.1 | 9.7 $\pm$ 0.3 | 0.5 $\pm$ 0.0 | 0.1 $\pm$ 0.0 | 82.2 $\pm$ 0.1 | 80.3 $\pm$ 0.2 | 1.9 $\pm$ 0.1
1 | 10.0 $\pm$ 0.0 | 0.5 $\pm$ 0.0 | 0.3 $\pm$ 0.1 | 82.9 $\pm$ 0.1 | 80.6 $\pm$ 0.3 | 2.4 $\pm$ 0.2
10 | 14.0 $\pm$ 4.0 | 0.7 $\pm$ 0.4 | 0.6 $\pm$ 0.2 | 81.2 $\pm$ 0.7 | 79.0 $\pm$ 0.7 | 2.3 $\pm$ 0.1

Table 10: Metrics on the spatial-alignment test acquired by the baseline
TesNet model (included here for ease of reference) and TesNet trained with the
spatial-aligning training with a smooth mask and masking augmentation.

method | $\lambda_{align}$ | MA |  $\downarrow$ PLC |  $\downarrow$ PRC |  $\downarrow$ PAC |  $\uparrow$ Acc. Before |  $\uparrow$ Acc. After |  $\downarrow$ AC
---|---|---|---|---|---|---|---|---
baseline | ✗ | ✗ | 27.7 $\pm$ 0.3 | 12.7 $\pm$ 1.3 | 6.5 $\pm$ 0.1 | 82.5 $\pm$ 0.2 | 70.4 $\pm$ 1.2 | 12.1 $\pm$ 1.0
ours | 0.1 | ✗ | 31.3 $\pm$ 0.3 | 19.6 $\pm$ 2.5 | 10.7 $\pm$ 0.3 | 79.4 $\pm$ 1.3 | 64.3 $\pm$ 1.9 | 15.1 $\pm$ 0.6
1 | 30.3 $\pm$ 1.9 | 18.8 $\pm$ 4.9 | 8.9 $\pm$ 1.6 | 68.6 $\pm$ 7.5 | 59.8 $\pm$ 3.1 | 8.9 $\pm$ 4.5
10 | 31.7 $\pm$ 1.7 | 19.5 $\pm$ 1.1 | 8.0 $\pm$ 2.3 | 78.9 $\pm$ 0.7 | 60.6 $\pm$ 3.4 | 18.3 $\pm$ 2.8
0 | ✓ | 21.7 $\pm$ 0.3 | 8.4 $\pm$ 0.5 | 2.5 $\pm$ 0.4 | 82.9 $\pm$ 0.5 | 78.0 $\pm$ 0.6 | 4.9 $\pm$ 0.2
0.1 | 23.0 $\pm$ 0.6 | 6.8 $\pm$ 0.6 | 3.3 $\pm$ 0.7 | 78.3 $\pm$ 2.5 | 73.7 $\pm$ 2.2 | 4.6 $\pm$ 0.4
1 | 22.7 $\pm$ 0.9 | 12.1 $\pm$ 2.2 | 4.7 $\pm$ 1.5 | 71.2 $\pm$ 4.8 | 68.1 $\pm$ 3.5 | 3.1 $\pm$ 1.3
10 | 24.7 $\pm$ 0.7 | 10.1 $\pm$ 0.3 | 3.6 $\pm$ 0.7 | 79.8 $\pm$ 0.3 | 74.6 $\pm$ 0.6 | 5.2 $\pm$ 0.9

Table 11: Metrics on the spatial-alignment test acquired by the baseline
ProtoTree model (included here for ease of reference) and ProtoTree trained
with the spatial-aligning training with a smooth mask and masking
augmentation.


519.21

# WAVELET-BASED SIMULATION
OF RANDOM PROCESSES FROM CERTAIN CLASSES
WITH GIVEN ACCURACY AND RELIABILITY

Ievgen Turchyn  Department of Mechanics and Mathematics, Oles Honchar Dnipro
National University, Gagarin av., 72, Dnipro, 49010, Ukraine
<EMAIL_ADDRESS>
(Date: ??.??.2019)

###### Abstract.

We consider stochastic processes $Y(t)$ which can be represented as
$Y(t)=(X(t))^{s},s\in\mathbb{N},$ where $X(t)$ is a stationary strictly sub-
Gaussian process and build a wavelet-based model that simulates $Y(t)$ with
given accuracy and reliability in $L_{p}([0,T])$. A model for simulation with
given accuracy and reliability in $L_{p}([0,T])$ is also built for processes
$Z(t)$ which can be represented as $Z(t)=X_{1}(t)X_{2}(t)$, where $X_{1}(t)$
and $X_{2}(t)$ are independent stationary strictly sub-Gaussian processes.

###### Key words and phrases:

Wavelets, Sub-Gaussian random processes, Simulation

###### 2010 Mathematics Subject Classification:

Primary 60G10; Secondary 42C40

##  1\. Introduction

Wavelet expansions and wavelet-based expansions form an interesting class of
representations of random processes. At present there exist many articles
devoted to such expansions and their properties, some of them are [1], [3],
[4], [8], [9] and [14]–[16]. Wavelet-based expansions with uncorrelated terms
(see, for instance, articles [4], [16]) are especially important since they
are very convenient for approximation and simulation of random processes.

We will consider simulation of stochastic processes with given accuracy and
reliability. This is simulation of a random process by a model which has
guaranteed rate of convergence in a certain sense — i.e., a model $\hat{X}(t)$
approximates a process $X(t)$ with given accuracy $\varepsilon$ and
reliability $1-\delta$ in a functional space $U(\mathbf{T})$ if

$P\\{\|X-\hat{X}\|_{U(\mathbf{T})}>\varepsilon\\}\leq\delta.$

Simulation of stochastic processes with given accuracy and reliability has
been studied, in particular, in [11] (see also e.g. [5], [10], [13] and [14]).
It is necessary to mention that results on simulation with given accuracy and
reliability are available mostly for light-tailed processes — Gaussian and
sub-Gaussian processes (although there are some exceptions, see, for instance,
[18]).

The article is devoted to simulation with given accuracy and reliability in
$L_{p}([0,T])$ of random processes which can be represented as

$Y(t)=(X(t))^{s},\ s\in\mathbb{N},$

where $X(t)$ is a stationary strictly sub-Gaussian process, and

$Z(t)=X_{1}(t)X_{2}(t),$

where $X_{1}(t)$ and $X_{2}(t)$ are stationary strictly sub-Gaussian
processes. So our approach allows simulation of processes with one-dimensional
distributions which have relatively heavy tails (i.e. heavier than the
Gaussian ones).

Our models are derived from a model which was studied in [14], where a
wavelet-based expansion was considered and used for construction of a model of
a process (this or similar wavelet-based expansions and their rate of
convergence were also studied in [4], [20] and [21]).

##  2\. Sub-Gaussian random variables and processes

######  Definition 2.1.

[2] Let $\\{\Omega,\mathfrak{F},\operatorname{\mathsf{P}}\\}$ be a standard
probability space. A random variable $\xi$ is called sub-Gaussian if

1) $\operatorname{\mathsf{E}}\xi=0$;

2) for all $\lambda\in\mathbb{R}$ there exists
$\operatorname{\mathsf{E}}\exp\\{\lambda\xi\\}$;

3) there exists such constant $a>0$ that inequality

$\operatorname{\mathsf{E}}\exp\\{\lambda\xi\\}\leq\exp\\{a^{2}\lambda^{2}/2\\}$

holds for all $\lambda\in\mathbb{R}$.

We will denote the set of all sub-Gaussian random variables by ${\rm
Sub}(\Omega)$. ${\rm Sub}(\Omega)$ is a Banach space with respect to the norm

$\tau(\xi)=\sup\limits_{\lambda\neq
0}\left(\frac{2\ln\operatorname{\mathsf{E}}\exp\\{\lambda\xi\\}}{\lambda^{2}}\right)^{1/2}$

(see [2]). Examples of sub-Gaussian random variables can be found in [2]. Let
us note that centered normal random variables belong to ${\rm Sub}(\Omega)$.

######  Definition 2.2.

[2] A sub-Gaussian random variable $\xi$ is called strictly sub-Gaussian if
$\tau(\xi)=(\mathsf{E}\xi^{2})^{1/2}$.

######  Definition 2.3.

[2] A family of sub-Gaussian random variables $\Delta$ is called strictly sub-
Gaussian if for any finite or countable set $\\{\xi_{i},i\in I\\}$ from
$\Delta$ and all $\lambda_{i}\in\mathbb{R}$ holds relation

$\tau^{2}\left(\sum\limits_{i\in
I}\lambda_{i}\xi_{i}\right)=\mathsf{E}\left(\sum\limits_{i\in
I}\lambda_{i}\xi_{i}\right)^{2}.$

######  Definition 2.4.

[2] A stochastic process $X=\\{X(t),t\in\mathbf{T}\\}$ is called strictly sub-
Gaussian if the family of random variables $\\{X(t),t\in\mathbf{T}\\}$ is
strictly sub-Gaussian.

######  Example 2.1.

[2] Let $X=\\{X(t),t\in\mathbf{T}\\}$ be a centered Gaussian process. Then
$X(t)$ is a strictly sub-Gaussian stochastic process.

######  Example 2.2.

Let $X=\\{X(t),t\in\mathbf{T}\\}$ be a random process such that

$X(t)=\sum\limits_{k=1}^{\infty}\xi_{k}f_{k}(t),$

where $\xi=\\{\xi_{k},k=1,2,\ldots\\}$ is a family of independent strictly
sub-Gaussian random variables and for all $t\in\mathbf{T}$

$\sum\limits_{k=1}^{\infty}\mathsf{E}\xi_{k}^{2}f_{k}^{2}(t)<\infty.$

Then $X(t)$ is a strictly sub-Gaussian stochastic process.

##  3\. Expansion of a random process into a wavelet-based series

######  Definition 3.1.

[6] Let $\phi\in L_{2}(\mathbb{R})$ be such a function that the following
assumptions hold:

i)

$\sum\limits_{k\in\mathbb{Z}}|\hat{\phi}(y+2\pi k)|^{2}=1$

almost everywhere, where $\hat{\phi}(y)$ is the Fourier transform of $\phi$;

ii) There exists a function $m_{0}\in L_{2}([0,2\pi])$ such that $m_{0}(x)$
has period $2\pi$ and almost everywhere

$\hat{\phi}(y)=m_{0}\left(\frac{y}{2}\right)\hat{\phi}\left(\frac{y}{2}\right);$

iii) $\hat{\phi}(0)\not=0$ and the function $\hat{\phi}(y)$ is continuous at
0.

The function $\phi(x)$ is called $f$-wavelet. Let $\psi(x)$ be the inverse
Fourier transform of the function

$\hat{\psi}(y)=\overline{m_{0}\left(\frac{y}{2}+\pi\right)}\exp\Bigl{\\{}\Bigr{.}-i\frac{y}{2}\Bigl{\\}}\Bigr{.}\hat{\phi}\left(\frac{y}{2}\right).$

The function $\psi(x)$ is called $m$-wavelet.

Let

$\phi_{jk}(x)=2^{j/2}\phi(2^{j}x-k),\,\,\,\psi_{jk}(x)=2^{j/2}\psi(2^{j}x-k),\,\,\,k\in\mathbb{Z},\,\,\,j=0,1,2,\ldots$

The family of functions
$\\{\phi_{0k},\psi_{jk},j=0,1,\ldots,k\in\mathbb{Z}\\}$ is an orthonormal
basis in $L_{2}(\mathbb{R})$ (see, for example, [6]).

######  Remark 3.1.

We will consider only real-valued wavelets below.

Let us now formulate a result which is very important for us.

######  Theorem 3.1.

[12] Suppose that $X(t),\,t\in\mathbb{R}$, is a centered second-order random
process such that its correlation function
$R(t,s)=\mathsf{E}X(t)\overline{X(s)}$ can be represented as

$R(t,s)=\int_{\mathbb{R}}u(t,y)\overline{u(s,y)}dy,$ (1)

where $u(t,y)$ is a Borel function which belongs to $L_{2}(\mathbb{R})$ for
all $t\in\mathbb{R},\\{\phi_{0k}(x),\linebreak\psi_{jk}(x),\,\,$
$k\in\mathbb{Z},\,j=0,1,\ldots\\}$ is an arbitrary wavelet basis,

$a_{0k}(t)=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}u(t,y)\overline{\hat{\phi}_{0k}(y)}dy,$
(2)
$b_{jk}(t)=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}u(t,y)\overline{\hat{\psi}_{jk}(y)}dy,$
(3)

$\hat{\phi}_{0k}(y)$ and $\hat{\psi}_{jk}(y)$ are the Fourier transforms of
$\phi_{0k}(y)$ and $\psi_{jk}(y)$ respectively.

Then

$X(t)=\sum_{k\in\mathbb{Z}}\xi_{0k}a_{0k}(t)+\sum_{j=0}^{\infty}\sum_{k\in\mathbb{Z}}\eta_{jk}b_{jk}(t),$
(4)

series $(\ref{p3eq-rozkl1})$ converges in $L_{2}(\Omega)$ for all
$\,t\in\mathbb{R}$, where $\xi_{0k},\,\eta_{jk}$ are centered random variables
such that

$\mathsf{E}\xi_{0k}\overline{\xi_{0l}}=\delta_{kl},\,\,\,\mathsf{E}\eta_{jk}\overline{\eta_{lm}}=\delta_{jl}\delta_{km},\,\,\,\mathsf{E}\xi_{0k}\overline{\eta_{nl}}=0.$

######  Corollary 3.1.

[12] Suppose that a centered second-order stationary process $X(t)$ has the
spectral density $f(y)$,
$\\{\phi_{0k}(x),\,\,\psi_{jk}(x),\,\,k\in\mathbb{Z},\,\,$ $j=0,1,\ldots\\}$
is a wavelet basis, $g(y)=\sqrt{f(y)}$. Then $X(t)$ can be represented as a
mean square convergent series $(\ref{p3eq-rozkl1})$ and

$a_{0k}(t)=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}g(y)\exp\\{-iy(t-k)\\}\overline{\hat{\phi}(y)}dy,$
(5)
$b_{jk}(t)=\frac{1}{\sqrt{2\pi}\,2^{j/2}}\int_{\mathbb{R}}g(y)\exp\left\\{-iy\left(t-\frac{k}{2^{j}}\right)\right\\}\overline{\hat{\psi}(y/2^{j})}dy,$
(6)

where the random variables $\xi_{0k},\,\eta_{jk}$ from $(\ref{p3eq-rozkl1})$
are such that

$\mathsf{E}\xi_{0k}=\mathsf{E}\eta_{jk}=0,$
$\mathsf{E}\xi_{0k}\overline{\xi_{0l}}=\delta_{kl},\,\,\,\mathsf{E}\eta_{jk}\overline{\eta_{lm}}=\delta_{jl}\delta_{km},\,\,\,\mathsf{E}\xi_{0k}\overline{\eta_{nl}}=0.$

##  4\. Simulation with given accuracy and reliability in $L_{p}([0,T])$

By a stationary process we will always mean a wide-sense stationary process
below.

######  Definition 4.1.

Suppose that a stationary random process $X=\\{X(t),\,t\in\mathbb{R}\\}$
satisfies the conditions of Corollary 3.1. We call the following process a
model of $X(t)$:

$\hat{X}(t)=\sum\limits_{k=-(N_{0}-1)}^{N_{0}-1}\xi_{0k}a_{0k}(t)+\sum\limits_{j=0}^{N-1}\sum\limits_{k=-(M_{j}-1)}^{M_{j}-1}\eta_{jk}b_{jk}(t),$
(7)

where $\xi_{0k}$, $\eta_{jk}$ are the random variables from the expansion (4),
$a_{0k}(t)$ and $b_{jk}(t)$ are calculated using formulae (5) and (6),
$N_{0}>1,N>1,M_{j}>1\ (j=0,1,\ldots,N-1)$.

Numerical characteristics which describe the rate of approximation of a
process by its model are accuracy and reliability.

######  Definition 4.2.

We say that a model $\hat{X}(t)$ approximates a process $X(t)$ with given
reliability $1-\delta$ ($0<\delta<1$) and accuracy $\varepsilon>0$ in
$L_{p}([0,T]),\,p>0,$ if

$P\left\\{\left(\int_{0}^{T}|X(t)-\hat{X}(t)|^{p}\right)^{1/p}>\varepsilon\right\\}\leq\delta.$

###  4.1. Simulation of $\mathbf{Y(t)=(X(t))^{s}}$

If $\hat{X}(t)$ is a model for a process $X(t)$ then a natural model for a
process $Y(t)=F(X(t))$ is a ‘‘plug-in’’ model $\hat{Y}(t)=F(\hat{X}(t))$. So
we will use $\hat{Y}(t)=(\hat{X}(t))^{s}$ as a model for $Y(t)=(X(t))^{s}$.

######  Theorem 4.1.

Let $Y(t)=(X(t))^{s},t\in\mathbb{R},s\in\mathbb{N},$ where
$X(t),t\in\mathbb{R},$ is a mean square continuous stationary strictly sub-
Gaussian stochastic process which has spectral density $f(y)$,
$g(y)=\sqrt{f(y)}$, $R(\tau)$ is the correlation function of $X(t)$, $\phi$ is
a $f$-wavelet, $\psi$ is the corresponding $m$-wavelet. Let the random
variables $\xi_{0k},\eta_{jk}$ in the expansion $(\ref{p3eq-rozkl1})$ of
$X(t)$ be independent and strictly sub-Gaussian. Suppose that the following
conditions hold: there exist the derivatives $g^{\prime}(y)$,
$\hat{\psi}^{\prime}(y)$, $\hat{\phi}^{\prime}(y)$, $|\hat{\psi}(y)|<C_{1}$,
$|\hat{\psi}^{\prime}(y)|<C_{2}$, $f(y)\to 0$ as $|y|\to\infty$, $g(y)$ and
$\hat{\phi}(y)$ are absolutely continuous,

$\sup_{y\in\mathbb{R}}|\hat{\phi}(y)|<\infty,\quad\sup_{y\in\mathbb{R}}g(y)<\infty,$
$\int_{\mathbb{R}}g(y)dy<\infty,\quad\int_{\mathbb{R}}|g^{\prime}(y)||y|dy<\infty,$
$\int_{\mathbb{R}}g(y)|y|dy<\infty\quad\int_{\mathbb{R}}|g^{\prime}(y)||\hat{\phi}(y)|dy<\infty,$
$\int_{\mathbb{R}}g(y)|\hat{\phi}^{\prime}(y)|dy<\infty.$

Denote

$A=\frac{C_{2}}{\sqrt{2\pi}}\int_{\mathbb{R}}(|g^{\prime}(y)||y|+g(y))dy,$
$B=\frac{C_{2}}{\sqrt{2\pi}}\int_{\mathbb{R}}g(y)|y|dy,$
$A_{1}=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}(|g^{\prime}(y)||\hat{\phi}(y)|+g(y)|\hat{\phi}^{\prime}(y)|)dy,$
$B_{1}=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}g(y)|\hat{\phi}(y)|dy.$

Let $p\geq 2,T>0,\varepsilon>0,0<\delta<1$ and let the model $\hat{X}(t)$ of
$X(t)$ be defined by $(\ref{pNR-model-hatX})$. Set
$\hat{Y}(t)=(\hat{X}(t))^{s}$.

If

$N_{0}>\frac{6}{\delta_{1}}(A_{1}+B_{1}T)^{2}+1,$ (8)
$N>\max\left\\{1+\log_{2}\left(\frac{72(A+BT)^{2}}{5\delta_{1}}\right),\,\,\,1+\log_{8}\left(\frac{18B^{2}}{7\delta_{1}}\right)\right\\},$
(9)
$M_{j}>1+\frac{12}{\delta_{1}}(A+BT)^{2}\left(1-\frac{1}{2^{N}}\right),\quad
j=0,1,\ldots,N-1,$ (10)

where

$\delta_{1}=\frac{\delta^{2/p}\varepsilon^{2}}{(D^{*})^{2/p}},$
$D^{*}=2^{(ps+3)/2}Tp\sqrt{\Gamma(p)}s^{p-1/2}(R(0))^{p(s-1)/2}$
$\cdot\left[(s-1)\Gamma(p(s-1))+\sum_{k=1}^{s-2}\sqrt{k(s-1-k)\Gamma(2pk)\Gamma(2p(s-1-k))}\right]^{1/2},$

then the model $\hat{Y}(t)$ approximates the process $Y(t)$ with given
accuracy $\varepsilon$ and reliability $1-\delta$ in $L_{p}([0,T])$.

###### Proof.

Denote $\Delta X(t)=X(t)-\hat{X}(t)$, $\Delta Y(t)=Y(t)-\hat{Y}(t)$,

$\|\Delta Y\|_{p}=\left(\int_{0}^{T}|\Delta Y(t)|^{p}dt\right)^{1/p}.$

According to Lemma 4.1 from [14] the following inequalities hold under the
conditions of the theorem:

$|a_{0k}(t)|\leq\frac{A_{1}+B_{1}|t|}{|k|},\ k\neq 0,$ (11)
$|b_{j0}(t)|\leq\frac{B}{2^{3j/2}},$ (12)
$|b_{jk}(t)|\leq\frac{A+B|t|}{|k|2^{j/2}},\ k\neq 0.$ (13)

It follows from (8)–(10) and (11)–(13) that

$\sup_{t\in[0,T]}\mathsf{E}|\Delta X(t)|^{2}\leq\delta_{1}.$

We will need the following inequality (see [17]): if $\xi$ is a sub-Gaussian
random variable, then

$\mathsf{E}|\xi|^{p}\leq p\,2^{p/2}(\tau(\xi))^{p}\,\Gamma(p/2),\ p>0.$ (14)

Let us estimate $\mathsf{E}\|\Delta{Y}\|_{p}^{2}$. Using the Lyapunov
inequality, we have:

$\mathsf{E}\|\Delta{Y}\|_{p}^{2}=\mathsf{E}\left(\int_{0}^{T}|Y(t)-\hat{Y}(t)|^{p}dt\right)^{2/p}\leq\left(\int_{0}^{T}\mathsf{E}|Y(t)-\hat{Y}(t)|^{p}dt\right)^{2/p}.$

Applying the CauchySchwarz inequality we obtain:

$\mathsf{E}|Y(t)-\hat{Y}(t)|^{p}=\mathsf{E}|(X(t))^{s}-(\hat{X}(t))^{s}|^{p}$
$=\mathsf{E}|X(t)-\hat{X}(t)|^{p}\left|\sum_{k=0}^{s-1}(X(t))^{k}(\hat{X}(t))^{s-1-k}\right|^{p}$
$\leq\left(\mathsf{E}|X(t)-\hat{X}(t)|^{2p}\right)^{1/2}\left(\mathsf{E}\left|\sum_{k=0}^{s-1}(X(t))^{k}(\hat{X}(t))^{s-1-k}\right|^{2p}\right)^{1/2}.$

It follows from (14) that

$\left(\mathsf{E}|X(t)-\hat{X}(t)|^{2p}\right)^{1/2}\leq(p\,2^{p+1}\,\Gamma(p))^{1/2}(\sigma(\Delta
X(t)))^{p}.$ (15)

Since

$\mathsf{E}|\zeta_{1}+\zeta_{2}+\ldots+\zeta_{s}|^{2p}\leq
s^{2p-1}(\mathsf{E}|\zeta_{1}|^{2p}+\mathsf{E}|\zeta_{2}|^{2p}+\ldots+\mathsf{E}|\zeta_{s}|^{2p})$
(16)

(an application of the power mean inequality) we have

$\mathsf{E}\left|\sum_{k=0}^{s-1}(X(t))^{k}(\hat{X}(t))^{s-1-k}\right|^{2p}\leq
s^{2p-1}\sum_{k=0}^{s-1}\mathsf{E}|X(t)|^{2pk}|\hat{X}(t)|^{2p(s-1-k)}.$ (17)

But using (14) and Cauchy-Schwarz inequality we obtain

$\mathsf{E}|X(t)|^{2pk}|\hat{X}(t)|^{2p(s-1-k)}\leq
C_{k}^{*}(R(0))^{p(s-1)},\quad 1\leq k\leq s-2,$ (18)

where

$C_{k}^{*}=4p\,2^{p(s-1)}\sqrt{k(s-1-k)\Gamma(2pk)\Gamma(2p(s-1-k))}$

(we used the inequality $\sigma(\hat{X}(t))<\sigma(X(t))$).

It follows from (14) that

$\mathsf{E}|X(t)|^{2p(s-1)}\leq
2p(s-1)2^{p(s-1)}\Gamma(p(s-1))(R(0))^{p(s-1)},$ (19)
$\mathsf{E}|\hat{X}(t)|^{2p(s-1)}\leq
2p(s-1)2^{p(s-1)}\Gamma(p(s-1))(R(0))^{p(s-1)}.$ (20)

Using (17)–(20) we have

$\int_{0}^{T}\mathsf{E}|Y(t)-\hat{Y}(t)|^{p}dt\leq\delta_{1}^{p/2}D^{*}=\delta\varepsilon^{p}.$

Applying the Markov inequality we get

$P\\{\|\Delta Y\|_{p}>\varepsilon\\}=P\\{\|\Delta
Y\|_{p}^{p}>\varepsilon^{p}\\}\leq\delta.$

So the theorem is proved. ∎

######  Example 4.1.

A stationary centered Gaussian process $X=\\{X(t),\,t\in\mathbb{R}\\}$ with
spectral density $f(y)=\frac{\textstyle 1}{\textstyle(1+y^{2n})^{2}},n\geq 2,$
and an arbitrary Daubechies wavelet satisfy the conditions of Theorem 4.1.

###  4.2. Simulation of $\mathbf{Z(t)=X_{1}(t)X_{2}(t)}$

Let us now consider a stochastic process $Z(t)$ which can be represented as

$Z(t)=X_{1}(t)X_{2}(t),$

where $X_{1}(t)$ and $X_{2}(t)$ are independent stationary strictly sub-
Gaussian stochastic processes which have spectral densities $f_{1}(y)$ and
$f_{2}(y)$ correspondingly. Let $\phi_{1},\psi_{1}$ and $\phi_{2},\psi_{2}$ be
two pairs of a $f$-wavelet and the corresponding $m$-wavelet.

According to Corollary 3.1, the processes $X_{1}(t)$ and $X_{2}(t)$ can be
expanded as

$X_{1}(t)=\sum_{k\in\mathbb{Z}}\xi_{0k}^{(1)}a_{0k}^{(1)}(t)+\sum_{j=0}^{\infty}\sum_{k\in\mathbb{Z}}\eta_{jk}^{(1)}b_{jk}^{(1)}(t),$
(21)
$X_{2}(t)=\sum_{k\in\mathbb{Z}}\xi_{0k}^{(2)}a_{0k}^{(2)}(t)+\sum_{j=0}^{\infty}\sum_{k\in\mathbb{Z}}\eta_{jk}^{(2)}b_{jk}^{(2)}(t),$
(22)

where

$a_{0k}^{(s)}(t)=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}g_{s}(y)\exp\\{-iy(t-k)\\}\overline{\hat{\phi}_{s}(y)}dy,$
(23)
$b_{jk}^{(s)}(t)=\frac{1}{\sqrt{2\pi}\,2^{j/2}}\int_{\mathbb{R}}g_{s}(y)\exp\left\\{-iy\left(t-\frac{k}{2^{j}}\right)\right\\}\overline{\hat{\psi}_{s}(y/2^{j})}dy,$
(24)

$g_{s}(y)=\sqrt{f_{s}(y)},$ random variables $\xi_{0k}^{(s)},\eta_{jl}^{(s)}$
are uncorrelated,

$\mathsf{E}|\xi_{0k}^{(s)}|^{2}=1,\mathsf{E}|\eta_{jl}^{(s)}|^{2}=1,$

$s=1,2.$

We will consider a ‘‘plug-in’’ model

$\hat{Z}(t)=\hat{X}_{1}(t)\hat{X}_{2}(t)$ (25)

for the process $Z(t)$, where $\hat{X}_{1}(t)$ and $\hat{X}_{2}(t)$ are models
of type (7) for $X_{1}(t)$ and $X_{2}(t)$ correspondingly, i.e.

$\hat{X}_{s}(t)=\sum\limits_{k=-(N_{0}^{(s)}-1)}^{N_{0}^{(s)}-1}\xi_{0k}^{(s)}a_{0k}^{(s)}(t)+\sum\limits_{j=0}^{N^{(s)}-1}\sum\limits_{k=-(M_{j}^{(s)}-1)}^{M_{j}^{(s)}-1}\eta_{jk}^{(s)}b_{jk}^{(s)}(t),$
(26)

$N_{0}^{(s)}>1,N^{(s)}>1,M_{j}^{(s)}>1\ (j=0,1,\ldots,N^{(s)}-1),\ s=1,2.$

######  Theorem 4.2.

Let $Z(t)=X_{1}(t)X_{2}(t),t\in\mathbb{R}$, where
$X_{1}(t),X_{2}(t),t\in\mathbb{R},$ are mean square continuous stationary
strictly sub-Gaussian stochastic processes which have spectral densities
$f_{s}(y)$, $g_{s}(y)=\sqrt{f_{s}(y)}$, $R_{s}(\tau)$ is the correlation
function of $X_{s}(t)(s=1,2)$, $\phi_{1},\psi_{1}$ and $\phi_{2},\psi_{2}$ are
two pairs of a $f$-wavelet and the corresponding $m$-wavelet. Let the random
variables $\xi_{0k}^{(s)},\eta_{jk}^{(s)}(s=1,2)$ in expansions
$(\ref{w1}),(\ref{w2})$ of $X_{1}(t)$ and $X_{2}(t)$ be independent and
strictly sub-Gaussian. Suppose that the following conditions hold: there exist
derivatives $g_{s}^{\prime}(y)$, $\hat{\psi}_{s}^{\prime}(y)$,
$\hat{\phi}_{s}^{\prime}(y)$, $|\hat{\psi}_{s}(y)|<C_{1}^{(s)}$,
$|\hat{\psi}_{s}^{\prime}(y)|<C_{2}^{(s)}$, $f_{s}(y)\to 0$ as $|y|\to\infty$,
$g_{s}(y)$ and $\hat{\phi}_{s}(y)$ are absolutely continuous,

$\sup_{y\in\mathbb{R}}|\hat{\phi}_{s}(y)|<\infty,\quad\sup_{y\in\mathbb{R}}g_{s}(y)<\infty,$
$\int_{\mathbb{R}}g_{s}(y)dy<\infty,\quad\int_{\mathbb{R}}|g_{s}^{\prime}(y)||y|dy<\infty,$
$\int_{\mathbb{R}}g_{s}(y)|y|dy<\infty\quad\int_{\mathbb{R}}|g_{s}^{\prime}(y)||\hat{\phi}_{s}(y)|dy<\infty,$
$\int_{\mathbb{R}}g_{s}(y)|\hat{\phi}_{s}^{\prime}(y)|dy<\infty,$

$s=1,2.$ Denote

$A^{(s)}=\frac{C_{2}^{(s)}}{\sqrt{2\pi}}\int_{\mathbb{R}}(|g_{s}^{\prime}(y)||y|+g_{s}(y))dy,$
$B^{(s)}=\frac{C_{2}^{(s)}}{\sqrt{2\pi}}\int_{\mathbb{R}}g_{s}(y)|y|dy,$
$A_{1}^{(s)}=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}(|g_{s}^{\prime}(y)||\hat{\phi}_{s}(y)|+g_{s}(y)|\hat{\phi}_{s}^{\prime}(y)|)dy,$
$B_{1}^{(s)}=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}g_{s}(y)|\hat{\phi}_{s}(y)|dy,$

$s=1,2.$

Let $p\geq 1,T>0,\varepsilon>0,0<\delta<1$ and let the models
$\hat{Z}(t),\hat{X}_{s}(t)$ of $Z(t),X_{s}(t)$ $(s=1,2)$ be defined by
$(\ref{mod-Zhat}),(\ref{mod-Xs})$ correspondingly.

If

$N_{0}^{(s)}>\frac{6}{\delta_{s}^{*}}(A_{1}^{(s)}+B_{1}^{(s)}T)^{2}+1,$ (27)
$N^{(s)}>\max\left\\{1+\log_{2}\left(\frac{72(A^{(s)}+B^{(s)}T)^{2}}{5\delta_{s}^{*}}\right),\,\,\,1+\log_{8}\left(\frac{18(B^{(s)})^{2}}{7\delta_{s}^{*}}\right)\right\\},$
(28)
$M_{j}^{(s)}>1+\frac{12}{\delta_{s}^{*}}(A^{(s)}+B^{(s)}T)^{2}\left(1-\frac{1}{2^{N^{(s)}}}\right),\quad
j=0,1,\ldots,N^{(s)}-1,$ (29)

$s=1,2,$ where

$\delta_{1}^{*}=\hat{\delta}/R_{2}(0),\ \delta_{2}^{*}=\hat{\delta}/R_{1}(0),$
$\hat{\delta}=\frac{\delta^{2/p}\varepsilon^{2}}{(2^{2p+1}p\Gamma(p)T)^{2/p}},$

then the model $\hat{Z}(t)$ defined by $(\ref{mod-Zhat})$ approximates the
process $Z(t)$ with given accuracy $\varepsilon$ and reliability $1-\delta$ in
$L_{p}([0,T])$.

###### Proof.

Denote $\Delta X_{s}(t)=X_{s}(t)-\hat{X}_{s}(t),$

$\|\Delta X_{s}\|_{p}=\left(\int_{0}^{T}|\Delta X_{s}(t)|^{p}dt\right)^{1/p},$

$s=1,2.$ Let us estimate

$\mathsf{E}\|X_{1}X_{2}-\hat{X_{1}}\hat{X_{2}}\|_{p}.$

Applying (14), the Cauchy-Schwarz inequality and the power mean inequality we
get

$\mathsf{E}|X_{1}(t)X_{2}(t)-\hat{X_{1}}(t)\hat{X_{2}}(t)|^{p}$ $\leq
2^{p-1}(\mathsf{E}|X_{2}(t)(X_{1}(t)-\hat{X_{1}}(t))|^{p}+\mathsf{E}|\hat{X_{1}}(t)(X_{2}(t)-\hat{X_{2}}(t))|^{p})$
$\leq
2^{p-1}\Big{(}(\mathsf{E}|X_{2}(t)|^{2p})^{1/2}(\mathsf{E}|X_{1}(t)-\hat{X_{1}}(t)|^{2p})^{1/2}$
$+(\mathsf{E}|\hat{X_{1}}(t)|^{2p})^{1/2}(\mathsf{E}|X_{2}(t)-\hat{X_{2}}(t)|^{2p})^{1/2}\Big{)}$
$\leq
2^{p-1}\Big{(}C_{R}(p)\sigma^{p}(X_{2}(t))\sigma^{p}(X_{1}(t)-\hat{X_{1}}(t))+C_{R}(p)\sigma^{p}(X_{1}(t))\sigma^{p}(X_{2}(t)-\hat{X_{2}}(t))\Big{)},$

where

$C_{R}(p)=2p2^{p}\Gamma(p).$

It follows from (27)–(29) that

$\sup_{t\in[0;T]}\sigma^{2}(\Delta X_{s}(t))\leq\delta_{s}^{*}$

$s=1,2.$ Now we see that

$\int_{0}^{T}\mathsf{E}|Z(t)-\hat{Z}(t)|^{p}dt$ $\leq
2^{p-1}C_{R}(p)T\Big{(}(R_{2}(0))^{p/2}\big{(}\sup_{t\in[0,T]}\sigma^{2}(\Delta
X_{1}(t))\big{)}^{p/2}+(R_{1}(0))^{p/2}\big{(}\sup_{t\in[0,T]}\sigma^{2}(\Delta
X_{2}(t))\big{)}^{p/2}\Big{)}$ $\leq\delta\varepsilon^{p}$

and, using the Markov inequality, we obtain

$P\\{\|Z-\hat{Z}\|_{p}>\varepsilon\\}\leq\delta.$

∎

######  Example 4.2.

Let us consider a process $Z(t)=X_{1}(t)X_{2}(t)$, where $X_{1}(t)$ and
$X_{2}(t)$ are independent centered stationary Gaussian stochastic processes
which have spectral densities

$f_{1}(y)=\frac{1}{(1+y^{2})^{2n}},\ n\geq 2,$

and

$f_{2}(y)=\left(\frac{1}{(1+(y-a)^{2})^{m}}+\frac{1}{(1+(y+a)^{2})^{m}}\right)^{2},\
m\geq 2,$

correspondingly. Let us take as $\phi_{1},\phi_{2}$ and $\psi_{1},\psi_{2}$
correspondingly two Daubechies $f$-wavelets and $m$-wavelets of any order.
These two pairs of a process and the corresponding wavelet satisfy the
conditions of the theorem.

##  5\. Conclusions

We built a wavelet-based model for simulation of a process which is an integer
power of a sub-Gaussian process. A wavelet-based model was also built for a
process $Z(t)$ which can be represented as $Z(t)=X_{1}(t)X_{2}(t)$, where
$X_{1}(t)$ and $X_{2}(t)$ are stationary strictly sub-Gaussian processes.

We proved theorems about simulation of stochastic processes by the above-
mentioned models with given accuracy and reliability in $L_{p}([0;T])$.

The author expresses gratitude to professor Yury Kozachenko for valuable
discussions.

## References

  * [1] A. Ayache, M. S. Taqqu, Rate optimality of wavelet series approximations of fractional Brownian motion, J. Fourier Anal. Appl. 9 (2003), no. 5, 451–471.
  * [2] V. V. Buldygin, Yu. V. Kozachenko, Metric Characterization of Random Variables and Random Processes, Amer. Math. Soc., Providence, RI, 2000\.
  * [3] S. Cambanis, E. Masry, Wavelet approximation of deterministic and random signals: convergence properties and rates, IEEE Trans. Inform. Theory, 40 (1994), no. 4, 1013–1029.
  * [4] G. Didier, V. Pipiras, Gaussian stationary processes: adaptive wavelet decompositions, discrete approximations, and their convergence, J. Fourier Anal. Appl. 14 (2008), no. 2, 203–234.
  * [5] B. V. Dovgay, Yu. V. Kozachenko, I. V. Rozora, Simulation of Random Processes in Physical Systems, Zadruga, Kyiv, 2010\. (Ukrainian)
  * [6] W. Härdle, G. Kerkyacharian, D. Picard, A. Tsybakov, Wavelets, Approximation and Statistical Applications, Springer, N.Y., 1998.
  * [7] E. Hernández, G. Weiss, A First Course on Wavelets, CRC Press Inc., Boca Rotan, FL, 1996\.
  * [8] Y. Kozachenko, A. Olenko, O. Polosmak, Uniform convergence of wavelet expansions of Gaussian random processes, Stoch. Anal. Appl. 29 (2011), no. 2, 169–184.
  * [9] Y. Kozachenko, A. Olenko, O. Polosmak, Uniform convergence of compactly supported wavelet expansions of Gaussian random processes, Comm. Statist. Theory Methods 43 (2014), no. 10-12, 2549–2562.
  * [10] Yu. Kozachenko, O. Pogoriliak, Simulation of Cox processes driven by random Gaussian field, Methodol. Comput. Appl. Probab. 13 (2011), 511–521.
  * [11] Yu. Kozachenko, O. Pogorilyak, I. Rozora, A. Tegza, Simulation of Stochastic Processes with Given Accuracy and Reliability, ISTE Press, London, 2016\.
  * [12] Yu. V. Kozachenko, I. V. Rozora, Ye. V. Turchyn, On an expansion of random processes in series, Random Oper. Stoch. Equ. 15 (2007), no. 1, 15–33.
  * [13] Yu. Kozachenko, T. Sottinen, O. Vasylyk, Simulation of weakly self-similar stationary increment $Sub_{\varphi}(\Omega)$-processes: a series expansion approach, Methodol. Comput. Appl. Probab. 7 (2005), 379–400.
  * [14] Yu. Kozachenko, Ye. Turchyn, On Karhunen-Loeve-like expansion for a class of random processes, Int. J. Stat. Manag. Syst. 3 (2008), 43–55.
  * [15] E. Masry, Convergence properties of wavelet series expansions of fractional Brownian motion, Appl. Comput. Harmon. Anal. 3 (1996), no. 3, 239–253.
  * [16] Y. Meyer, F. Sellan, M. S. Taqqu, Wavelets, generalized white noise and fractional integration: the synthesis of fractional Brownian motion, J. Fourier Anal. Appl. 5 (1999), no. 5, 465–494.
  * [17] O. Rivasplata, Subgaussian random variables: An expository note, DOI: 10.13140/RG.2.2.36288.23040
  * [18] I. Turchyn, A multiplicative wavelet-based model for simulation of a random process, Mod. Stoch. Theory Appl. 2 (2015), 309-325.
  * [19] Y. Turchyn, Simulation of sub-Gaussian processes using wavelets, Monte Carlo Methods Appl. 17 (2011), 215-231.
  * [20] G. Walter, J. Zhang, A wavelet-based KL-like expansion for wide-sense stationary random processes, IEEE Trans. Signal Process. 42 (1994), no. 7, 1737–1745.
  * [21] P. Zhao, G. Liu, C. Zhao, A matrix-valued wavelet KL-like expansion for wide-sense stationary random processes, IEEE Trans. Signal Process. 52 (2004), no. 4, 914–920.