# Faithfulness Measurable Masked Language Models

Andreas Madsen1,2 Siva Reddy1,3,4 Sarath Chandar1,2,5
1 Mila – Quebec AI Institute 2 Polytechnique Montréal 3 McGill University
4 Facebook CIFAR AI Chair 5 Canada CIFAR AI Chair
<EMAIL_ADDRESS>

###### Abstract

A common approach to explain NLP models, is to use importance measures that
express which tokens are important for a prediction. Unfortunately, such
explanations are often wrong despite being persuasive. Therefore, it is
essential to measure their faithfulness. One such metric is if tokens are
truly important, then masking them should result in worse model performance.
However, token masking introduces out-of-distribution issues and existing
solutions are computationally expensive and employ proxy-models. Furthermore,
other metrics are very limited in scope. In this work, we propose an
inherently faithfulness measurable model that addresses these challenges. This
is achieved by using a novel fine-tuning method that incorporates masking,
such that masking tokens become in-distribution by design. This differs from
existing approaches, which are completely model-agnostic but are inapplicable
in practice. We demonstrate the generality of our approach by applying it to
various tasks and validate it using statistical in-distribution tests.
Additionally, because masking is in-distribution, importance measures which
themselves use masking become more faithful, thus our model becomes more
explainable.

##  1 Introduction

As machine learning models are increasingly being deployed, the demand for
interpretability to ensure safe operation increases (Doshi-Velez and Kim,
2017). Importance measures, such as attention or integrated gradient, are a
popular way of explaining which input tokens are important for making a
prediction (Bhatt et al., 2019). Such explanations are not only used directly
to explain models, but are also used in other explanations such as contrastive
(Yin and Neubig, 2022), counterfactuals (Ross et al., 2021), and adversarial
explanations (Ebrahimi et al., 2018).

Unfortunately, importance measures are often found to provide false
explanations despite being persuasive (Jain and Wallace, 2019; Hooker et al.,
2019). The measure of explanations validity is called a faithfulness metric.
In this work, we propose a methodology that enables existing models to support
measuring faithfulness by design.

Measuring faithfulness is challenging, as there is generally no known ground-
truth for the correct explanation. Instead, faithfulness metrics have to use
proxies. One such proxy is the _erasure-metric_ by Samek et al. (2017): if
tokens are truly important, then masking them should result in worse model
performance compared to masking random tokens.

However, masking tokens can create out-of-distribution issues. This can be
solved by retraining the model, after allegedly important tokens have been
masked (Hooker et al., 2019; Madsen et al., 2022). Unfortunately, this is
computationally expensive, leaks the gold label, and the measured model is now
different from the model of interest.

In general, the cost of proxy-metrics is some combination of incorrect
assumptions, expensive computations, or using a proxy-model (Jain and Wallace,
2019; Bastings et al., 2021; Madsen et al., 2022). Based on previous
literature, we propose the following desirables that we satisfy, which to the
best of our knowledge no other method satisfies:

  1. a)

The method does not assume a true explanation.

  2. b)

The method measures faithfulness of an explanation w.r.t. a specific model
instance or single observation. For example, does not retrain or modify an
existing model.

  3. c)

The method uses only the original dataset, e.g. does not introduce spurious
correlations.

  4. d)

The method only uses inputs that are in-distribution w.r.t. the model.

  5. e)

The method is computationally cheap by not training/fine-tuning repeatedly and
only computes explanations of the test dataset.

  6. f)

The method can be applied to any classification task.

The key idea is to use the erasure-metric but without solving the out-of-
distribution (OOD) issue using retraining, which avoids all the limitations.
This is achieved by including masking in the fine-tuning procedure of masked
language models, such that masking is always in-distribution (Figure 1(a)).
This is possible because language models are heavily over-parameterized, and
can therefore support such additional complexity. Although, our approach
applies to Masked Language Models (MLMs), we suspect future work could apply
this idea to any language model with sufficient capacity.

Our approach is significantly different from previous literature, which is
completely model agnostic. Instead, we fine-tune a model such that measuring
the faithfulness is easy by design. We call such designed models: inherently
faithfulness measurable models (FFMs).

(a) Masked fine-tuning. Support for masking any permutation of tokens is
achieved by uniformly masking half of the mini-batch during fine-tuning, the
other half is left unmasked.

(b) In-distribution validation. CDFs of the model’s embeddings provides in-
distribution p-values, and validates that the masked test observations are in-
distribution.

(c) Faithfulness metric. Observations are masked according to an explanation.
Model performance lower than masking random tokens means faithful. The large
the area, the more faithful.

Figure 1: To measure faithfulness, a _faithfulness measurable masked language
model_ is created (a), then the model is checked for out-of-distribution
issues (b), and finally the faithfulness is measured by masking allegedly
important tokens (c). – _[ $\mathcal{M}$] is the masking token._

To validate that masking is in-distribution, we generalize previous OOD
detection work from computer vision (Heller and Soudry, 2022). This serves as
a statistically grounded meta-validation of the faithfulness measure itself
(Figure 1(b)). Finally, once the model is validated, the erasure-metric can be
applied (Figure 1(c)).

Note, the concept of an _inherently faithfulness measurable model_ (FMM) is
significantly different from inherently explainable models, which are
interpretable by design (Jacovi and Goldberg, 2020). An FMM does not guarantee
that an explanation exists, and an inherently explainable model doesn’t
provide a means of measuring faithfulness.

However, with an FMM, measuring faithfulness is computationally cheap.
Therefore, it is possible to optimize an explanation w.r.t. faithfulness, as
proposed by Zhou and Shah (2023). However, they did not solve the OOD issue
caused by masking, as it was “orthogonal” to their idea, but our _inherently
faithfulness measurable model_ fills that gap, making it indirectly inherently
explainable.

Finally, for completeness, we compare a large variety of existing explanation
methods and modify some existing explanations to be able to separate positive
from negative contributions. In general, we find that the explanations which
take advantage of masking (occlusion-based) are more faithful than gradient-
based methods. However, the robustness provided by a _faithfulness measurable
model_ also makes some gradient-based more faithful.

#### To summarize, our contributions are:

  * •

Introduce the concept of an _inherently faithfulness measurable model_ (FMM).

  * •

Propose _masked fine-tuning_ that enables masking to be in-distribution.

  * •

Establish a statistically grounded meta-validation for the faithfulness
measurable model, using out-of-distribution detection.

  * •

Existing occlusion-based explanations becomes more faithful, as they no longer
cause out-of-distribution issues.

  * •

Introduce signed variants of existing gradient-based methods, which can
separate between positive and negative contributing tokens.

##  2 Related Work

Much recent work in NLP has been devoted to investigating the faithfulness of
importance measures. In this section, we categorize these faithfulness metrics
according to their underlying principle and discuss their limitations. The
limitations are annotated as (a) to (f) and refers to the desirable mentioned
in the Introduction.

Note, we do not cover the faithfulness metrics that are specific to attention
(Moradi et al., 2021; Wiegreffe and Pinter, 2019; Vashishth et al., 2019), as
this paper presents a general approach.

###  2.1 Correlating importance measures

One of the early ideas was to compare two importance measures. The claim is
that a correlation would be a very unlikely coincidence unless they express a
fundamental truth. Both Jain and Wallace (2019) and Meister et al. (2021) find
that there is little correlation between attention, gradient, and leave-one-
out, they are therefore not faithful. Jain and Wallace (2019) do acknowledge
the limitations of their approach, as it assumes each importance measure is
faithful to begin with (a).

###  2.2 Known explanations in synthetic tasks

Arras et al. (2022) construct a purely synthetic task, where the true
explanation is known, therefore the correlation can be applied appropriately.
Unfortunately, this approach cannot be used on real datasets (f). Instead,
Bastings et al. (2021) introduce spurious correlations into real datasets,
creating partially synthetic tasks. They then evaluate if importance measures
can detect these correlations. It is assumed that if an explanation fails this
test, it is generally unfaithful. Bastings et al. (2021) conclude that
faithfulness is both model and task-dependent.

Both methods are valid when measuring faithfulness on models trained on
(partially) synthetic data. However, the model and task-dependent conclusion
also means that we can’t generalize this finding to the models (b) and
datasets of interest (c).

###  2.3 Similar inputs, similar explanation

Jacovi and Goldberg (2020) suggest that if similar inputs show similar
explanations, then the explanation method is faithful. Zaman and Belinkov
(2022) applies this idea, using a multilingual dataset (f), where each
language example is explained and aligned to the English example, using a
known alignment mapping. If the correlation between language-pairs is high,
this indicates faithfulness.

Besides being limited to multilingual datasets, the metric assumes the model
behaves similarly among languages. However, languages may have different
linguistic properties or spurious correlations. A faithful explanation would
then yield different explanations for each language.

###  2.4 Removing important information should affect the prediction

Samek et al. (2017) introduce the erasure-metric: if information (input
tokens) is truly important, then removing it should result in worse model
performance compared to removing random information. However, Hooker et al.
(2019) argue that removing tokens introduces an out-of-distribution issue (d).

Hooker et al. (2019) solve the out-of-distribution by retraining the model.
They point out a limitation: the method cannot separate an unfaithful
explanation from “there exist dataset redundancies”. However, this has later
been solved by Madsen et al. (2022), and conclude, like Bastings et al.
(2021), that faithfulness is both model and task-dependent.

Unfortunately, both methods are computationally expensive (e), due to the
model being retrained. Additionally, retraining means that it is no longer
possible to comment on the faithfulness of the deployed model (b). Finally,
the act of removing tokens and retraining can introduce redundancies, which
increases the performances and underestimates the faithfulness (Madsen et al.,
2022).

##  3 Inherently faithfulness measurable models (FMMs)

As an alternative to existing faithfulness methods, which all aim to work with
any models, we propose creating _inherently faithfulness measurable models_
(FMMs). These are models that both provide the typical output (e.g.
classification) for a given task and, by design, provide the means to measure
the faithfulness of an explanation. Importantly, this allows measuring the
faithfulness of a specific model, as there is no need for proxy models, an
important property in a real deployment setting.

An _inherently faithfulness measurable model_ does create the limitation, that
a specific model is required. However, our proposed method is very general, as
it only requires a modified fine-tuning procedure applied to a masked language
model.

###  3.1 Faithfulness of importance measures

In this paper, we look at importance measures (IMs), which are explanations
that either score or rank how important each input tokens are for making a
prediction. A faithfulness metric, measures how correct such explanation is.
Importantly, such a metric should work regardless of how the importance
measure is calculated.

For importance measures, there are multiple definitions of truth. In this
paper, we use the erasure-metric definition: _if information (tokens) is truly
important, then masking them should result in worse model performance compared
to masking random information (tokens)._ (Samek et al., 2017; Hooker et al.,
2019; DeYoung et al., 2020)

The challenge with an erasure-metric is that fine-tuned models do not support
masking tokens. Even masked language models are usually only trained with 12%
or 15% masking (Devlin et al., 2019; Liu et al., 2019; Wettig et al., 2023),
and an erasure-metric use between 0% and 100% masking. Furthermore,
catastrophic forgetting of the masking token is likely when fine-tuning.

Hooker et al. (2019) and Madsen et al. (2022) solve this by retraining the
model, with partially masked inputs, and call the approach ROAR (Remove and
Retrain). Unfortunately, as discussed in Section 2, retraining has its issues.
It is computationally expensive, leaks the gold label, and measures a proxy
model instead of the true model.

We find that the core issue is the need for retraining. If instead, the fine-
tuned model supports masking any permutation of tokens, then retraining would
not be required, eliminating all issues. To achieve this, we propose a new
fine-tuning procedure called _masked fine-tuning_.

To evaluate faithfulness of an importance measure, we propose a three-step
process, as visualized in Figure 1 (also see Appendix D for details):

  1. 1.

Create a faithfulness measurable masked language model, using _masked fine-
tuning_. – See Section 3.2 and Figure 1(a).

  2. 2.

Check for out-of-distribution (OOD) issues, by using a statistical in-
distribution test. – See Section 3.3 and Figure 1(b).

  3. 3.

Measure the faithfulness of an explanation. – See Section 3.4 and Figure 1(c).

###  3.2 Masked fine-tuning

To provide masking support in the fine-tuned model, we propose to randomly
mask the training dataset, by uniformly sampling a masking-rate between 0% and
100% for each observation and then randomly mask that ratio of tokens.
However, to maintain the regular unmasked performance, half of the mini-batch
remains unmasked. This is analog to multi-task learning, where one task is
masking support and one task is regular performance.

To include masking support in early-stopping, the validation dataset is
duplicated, where one copy is unmasked and one copy is randomly masked.

###  3.3 In-distribution validation

Erasure-based metrics are only valid when the input is in-distribution.
Unfortunately, _masked fine-tuning_ does not guarantee this. Additionally, in-
distribution is the statistical null-hypothesis and can never be confirmed.
However, we can validate using out-of-distribution (OOD) test.

We use the _MaSF_ method by Heller and Soudry (2022) as the OOD test, which
provides non-parametric p-values under the in-distribution null-hypothesis.
While _MaSF_ is developed and tested for computer vision, it is very general
and naturally applies to NLP. The method works by developing empirical CDFs of
the models embeddings. In the case of RoBERTa we use the embeddings after the
layer-normalization (Ba et al., 2016).

To develop the empirical CDFs, the validation dataset is used. Because the
validation dataset should be from the same distribution as the training
dataset, we use the same masking method as for the training dataset. Once the
CDFs are developed, any observation can be tested against the CDFs.

###  3.4 Faithfulness metric

To measure faithfulness on a model trained using _masked fine-tuning_ : the
importance measure (IM) is computed for a given input, then $x\%$ (e.g.
$10\%$) of the most important tokens are masked, then the IM is calculated on
this masked input, then an additional $x\%$ of the most important tokens are
masked. This is repeated until $100\%$ is masked. The importance measure is
re-calculated, because otherwise dataset redundancies will infer with the
metric as shown by Madsen et al. (2022).

At each iteration, the masked input is validated using MaSF and the
performance is measured. Faithfulness is shown, if and only if the performance
is less than when masking random tokens.

This procedure is identical to Recursive-ROAR by Madsen et al. (2022), but
without retraining and with in-distribution validation.

##  4 Importance measures (IMs)

Our proposed _inherently faithfulness measurable model_ and the erasure-metric
can be applied to a single observation, which is useful in practical settings
but not for statistical conclusions. Therefore, we focus on importance
measures, which are feasible to compute on the entire test dataset.

The importance measures used in this paper, are all from existing literature,
hence the details are in the Appendix C. Except, we introduce signed and
absolute variants of existing IMs. Additionally, we argue that occlusion-based
IMs are only valid in combination with _masked fine-tuning_.

###  4.1 Gradient-based vs occlusion-based

A common idea is that if a small change in the input causes a large change in
the output, then that indicates importance.

#### Gradient-based

The relationship of change can be modelled by the gradient w.r.t. the input.
Let $f(\mathbf{x})$ be the model with input111$V$ is vocabulary-size and $T$
is sequence-length. $\mathbf{x}\in\mathbb{R}^{T\times V}$, then the gradient
explanation (grad) is
$\nabla_{\mathbf{x}}f(\mathbf{x})_{y}\in\mathbb{R}^{T\times V}$ (Baehrens et
al., 2010; Li et al., 2016a). Later work (Kindermans et al., 2016) have
proposed using $\textbf{x}\odot\nabla_{\mathbf{x}}f(\mathbf{x})_{y}$ instead
(i.e. x $\odot$ grad), or to sample multiple gradients (IG) (Sundararajan et
al., 2017).

#### Occlusion-based

An alternative to using gradients, is to measure the change by removing or
masking a token, and measuring how it affects the output. One rarely addressed
concern, is that removing/masking tokens is likely to be out-of-distribution
(Zhou and Shah, 2023). This would cause an otherwise sound method to become
unfaithful. However, because our proposal _faithfulness measurable masked
language model_ supports masking, this should not be a concern. Importantly,
this exemplifies how a _inherently faithfulness measurable model_ may also
produce more faithful explanations.

A simple occlusion-based explanation is leave-on-out (LOO) which test the
effect of each token, one-by-one (Li et al., 2016b). However, this does not
account for when one token is masked, another becomes more important. Zhou and
Shah (2023) propose that we can optimize for the faithfulness metric itself,
rather than using heuristic approximations. The idea is to use beam-search
(Beam), where the generated sequence is the optimal masking order of tokens.

###  4.2 Signed and absolute variants

Most literature does not distinguish between positive and negative
contributing tokens222Unfortunately, the details of this are often omitted in
the literature and can often only be learned though reading the code. However,
Meister et al. (2021) do provide the details., which we categorize this as an
_absolute IM_. In the opposite case, when positive and negative are separated,
we categorize as a _signed IM_. In most cases, a signed IM can be transformed
to an absolute IM using $abs(\cdot)$.

##  5 Experiments

We use RoBERTa in size base and large, with the default GLUE hyperparameters
provided by Liu et al. (2019). We present results on 16 classification
datasets in the appendix, but do only include BoolQ, MRPC, and bAbI-2 in the
main paper. These were chosen as they represent the general trends we observe,
although we observe very consistent results across all datasets. The full list
of datasets are in Appendix A and model details are in Appendix B. The code is
available at http://github.com/AndreasMadsen/economical-roar.

For each experiment, we use 5 seeds and present their means with their 95%
confidence interval (error-bars or ribbons). The 95% confidence interval is
computed using the bias-corrected and accelerated bootstrap method (Buckland
et al., 1998; Michael R. Chernick and LaBudde, 2011). When relevant, each seed
is presented as a plus (+).

###  5.1 Masked fine-tuning

There are two criteria for learning our proposed _faithfulness measurable
masked language model_ :

  1. 1.

The usual performance metric, where no data is masked, should not decrease.

  2. 2.

Making any permutations of tokens should be in-distribution.

In Section 3.2 we propose _masked fine-tuning_ , where the one half of a mini-
batch is uniformly masked between 0% and 100% and the other half is unmasked.
Additionally, the validation dataset contains a masked copy and an unmasked
copy.

#### Unmasked performance.

To validate the first goal, Figure 2 presents an ablation study. It compares
_masked fine-tuning_ with using only unmasked data (_plain fine-tuning_), as
is traditionally done, and using only uniformly masked data (_only masking_).
The unmasked performance is then measured (the usual benchmark).

Figure 2: The unmasked performance for each fine-tuning strategy. _Plain fine-
tuning_ is the baseline (dashed line). We find that our _Masked fine-tuning_
does not decrease performance. _All_ is computed by taking the average of all
datasets. More datasets, and an ablation separation of the training and
validation strategy, can be found in Appendix F.

We verify that no performance is lost when using our _masked fine-tuning_ ,
while performance is lost when using _only masking_ and many seeds present
unstable convergence. For bAbI-2 and bAbI-3 we also observe unstable
convergence using _masked fine-tuning_ , however it is less frequent (see
Figure 8 for bAbI-3). Note, we don’t perform any hyperparameter optimization
in this paper, which would likely solve the stability issues with _masked
fine-tuning_. Finally, when using _masked fine-tuning_ , the models do need to
be trained for slightly more epochs (twice more or less), see Appendix G.

#### 100% Masked performance.

Measuring in-distribution support for masked data is challenging, as there is
generally no known performance baseline. We validate this in-depth in Section
5.2. However, for 100% masked data, only the sequence-length is left as
information. The performance of a model should therefore be at least that of
the class-majority baseline, where the most frequent class is “predicted” for
all observations. We present an ablation study using this baseline in Figure
3.

Figure 3: The 100% masked performance for each fine-tuning strategy. The
dashed line represents the class-majority baseline. Results show that masking
during training (either our _masked fine-tuning_ or _only masking_) is
necessary. More datasets, and an ablation separation of the training and
validation strategy, can be found in Appendix F.

From Figure 3 we observe that training with unmasked data (_Plain fine-
tuning_) performs worse than the class-majority baseline, clearly showing an
out-of-distribution issue. However, when using masked data, either _only
masking_ or _masked fine-tuning_ , both are effective at achieving in-
distribution results for 100% masked data.

#### The best approach used in the following experiments.

Appendix F contains a more detailed ablation study. However, the conclusion is
the same. _Masked fine-tuning_ is the only method that achieves good results
for both the unmasked and 100% masked cases. In Appendix F, we do find that
the choice of validation dataset is not very significant. However, we consider
it the most principled approach to using both unmasked and masked data, i.e.
_Masked fine-tuning_.

For all the following experiments in Section 5.2 and Section 5.3, the _masked
fine-tuning_ method is used. Additionally, we will only present results for
RoBERTa-base for brevity, but results for RoBERTa-large are included in the
appendix.

###  5.2 In-distribution validation

Because the expected performance for masked data is generally unknown, a
statistical in-distribution test called _MaSF_ (Heller and Soudry, 2022) is
used instead. The theory is briefly explained in Section 3.3, with details in
Section D.2.

MaSF provides an in-distribution p-value for each observation. To test if all
masked test observations are in-distribution, the p-values are aggregated
using Simes’s method (Simes, 1986). Because in-distribution is the null
hypothesis we can never confirm in-distribution, we can only validate it.
Rejecting the null hypothesis would mean that some observation is out-of-
distribution.

Figure 4: In-distribution p-values using MaSF, for RoBERTa-base with and
without masked fine-tuning. The masked tokens are chosen according to an
importance measure. P-values below the dashed line show out-of-distribution
(OOD) results, given a 5% risk of a false positive. Results show that only
when using masked fine-tuning, masked data is consistently not OOD. Because
the results are highly consistent, the overlapping lines do not hide any
important details. More datasets and models in Appendix H.

Because random uniform masking is not the same as strategically masking
tokens, we validate in-distribution for each importance measure, where the
masking is done according to the importance measure, identically to how the
faithfulness metric is computed (Section 3.4).

Additionally, because MaSF does not consider the model’s performance, it is
necessary to consider these results in combination with regular performance
metrics, see Section 5.1.

The results for when using _masked fine-tuning_ and _plain fine-tuning_ (no
masking) are presented in Figure 4. The results show that only when using
masked fine-tuning, are masked datasets consistently in-distribution. The
results are highly consistent, as such the overlapping lines do not hide any
important details.

In the case of BoolQ, we suspect that because the training dataset is fairly
small (7542 observations), the model does not completely forget the mask
token. Additionally, a few datasets, such as bAbI-2, become out-of-
distribution at 100% masking. This contradicts the performance results for
100% masked data (Figure 3) which clearly shows in-distribution performance.
We suspect this is just a limitation of MaSF, because the empirical
distribution that MaSF builds contains very little 100% masked data, as the
masking ratio is continuous uniform between 0% and 100%. Fortunately, because
Figure 3 shows in-distribution results for 100% masked data, this should not
be a concern.

Figure 5: The performance given the masked datasets, where masking is done for
the x% allegedly most important tokens according to the importance measure. If
the performance for a given explanation is below the _“Random”_ baseline, this
shows faithfulness. Although, faithfulness is not an absolute concept, so more
is better. This plot is for RoBERTa-base, and separates importance measures
based on their signed and absolute variants. More datasets and models in
Appendix I.

###  5.3 Faithfulness metric

Based on previous experiments, we can conclude that _masked fine-tuning_ is
able to achieve both objectives, unaffected regular performance and support
for masked inputs. Therefore, it is safe to apply the faithfulness metric to
these models.

Briefly, the faithfulness metric works by showing that masking using a given
importance measure, is more effective at removing important tokens than using
a known false explanation, such as a random explanation. That is, if a curve
is below the random baseline, the explanation is faithful. Although,
faithfulness is not an absolute, so further below would indicate more
faithful.

Figure 5 shows that most explanations are faithful, although some are much
more faithful than others. In particular, we find that occlusion-based
importance measures (LOO, Beam) are the most faithful. This is expected, as
they take advantage of the masking support that our _faithfulness measurable
masked language model_ offers.

Because signed importance measures are able to differentiate between positive
and negative contributing tokens, while absolute tokens are not, it is to be
expected that signed importance measures are more faithful. However, because
of this difference in capability, comparing them might not be fair. We let the
reader decide this for themselves.

#### Relative Area Between Curves (RACU)

Madsen et al. (2022) propose to compute the area between the random curve and
an explanation curve. This can then be normalized by the theoretical optimal
explanation, which would achieve the performance of 100% masking immediately.
They call this metric RACU. However, the normalization is only theoretically
optimal for an absolute importance measure. Signed importance measures are
able to trick the model into predicting the opposite label, and can thus
achieve even lower performance. For this reason, we also show the un-
normalized metric (ACU) in Table 1.

|  | Faithfulness [%]
---|---|---
Dataset | IM | Our | R-ROAR
|  | ACU | RACU | RACU
Anemia | Grad ($L_{2}$) | $23.8_{-0.5}^{+0.6}$ | $62.1_{-1.7}^{+1.4}$ | $18.2_{-13.8}^{+11.8}$
Grad ($L_{1}$) | $23.8_{-0.6}^{+0.6}$ | $62.2_{-2.1}^{+1.4}$ | –
x $\odot$ grad (sign) | $9.7_{-2.5}^{+2.6}$ | $25.1_{-6.2}^{+6.5}$ | –
x $\odot$ grad (abs) | $16.6_{-1.3}^{+1.3}$ | $43.2_{-3.7}^{+3.0}$ | $8.8_{-22.8}^{+22.7}$
IG (sign) | $62.0_{-1.5}^{+1.6}$ | $161.8_{-2.3}^{+2.7}$ | –
IG (abs) | $20.0_{-1.6}^{+0.9}$ | $52.1_{-4.5}^{+2.5}$ | $12.5_{-7.0}^{+11.3}$
LOO (sign) | $63.3_{-1.6}^{+1.4}$ | $165.2_{-3.2}^{+2.9}$ | –
LOO (abs) | $18.9_{-1.3}^{+1.0}$ | $49.2_{-3.8}^{+2.5}$ | –
Beam | – | – | –
Diabetes | Grad ($L_{2}$) | $19.7_{-0.7}^{+1.2}$ | $91.8_{-0.9}^{+0.6}$ | $57.9_{-19.8}^{+14.4}$
Grad ($L_{1}$) | $19.6_{-0.7}^{+1.0}$ | $91.6_{-0.9}^{+0.5}$ | –
x $\odot$ grad (sign) | $10.9_{-1.3}^{+1.9}$ | $51.1_{-7.8}^{+9.1}$ | –
x $\odot$ grad (abs) | $18.8_{-0.7}^{+1.3}$ | $87.9_{-2.0}^{+1.4}$ | $53.4_{-29.3}^{+23.2}$
IG (sign) | $24.8_{-2.1}^{+1.5}$ | $115.8_{-7.8}^{+2.8}$ | –
IG (abs) | $19.4_{-0.6}^{+1.0}$ | $90.5_{-1.2}^{+0.6}$ | $26.1_{-25.1}^{+12.0}$
LOO (sign) | $41.5_{-5.9}^{+2.3}$ | $193.4_{-17.4}^{+8.9}$ | –
LOO (abs) | $19.1_{-0.6}^{+1.1}$ | $89.0_{-0.6}^{+0.4}$ | –
Beam | – | – | –
SST2 | Grad ($L_{2}$) | $12.2_{-0.7}^{+0.6}$ | $40.4_{-1.7}^{+3.0}$ | $26.1_{-2.2}^{+1.6}$
Grad ($L_{1}$) | $12.1_{-0.7}^{+0.7}$ | $40.3_{-1.8}^{+3.3}$ | –
x $\odot$ grad (sign) | $-3.7_{-1.6}^{+1.5}$ | $-12.2_{-6.0}^{+4.5}$ | –
x $\odot$ grad (abs) | $7.1_{-0.2}^{+0.2}$ | $23.5_{-1.1}^{+1.9}$ | $18.6_{-4.6}^{+4.1}$
IG (sign) | $31.8_{-2.2}^{+2.8}$ | $105.6_{-7.7}^{+7.7}$ | –
IG (abs) | $13.7_{-0.8}^{+0.8}$ | $45.3_{-2.8}^{+4.1}$ | $32.9_{-1.5}^{+1.8}$
LOO (sign) | $51.6_{-0.9}^{+1.4}$ | $171.3_{-6.2}^{+5.8}$ | –
LOO (abs) | $16.6_{-1.0}^{+1.2}$ | $54.9_{-1.5}^{+2.1}$ | –
Beam | $56.4_{-0.7}^{+0.5}$ | $187.3_{-7.1}^{+8.1}$ | –
bAbI-2 | Grad ($L_{2}$) | $28.5_{-0.8}^{+0.8}$ | $96.3_{-2.8}^{+6.8}$ | $57.8_{-2.0}^{+2.0}$
Grad ($L_{1}$) | $28.5_{-0.8}^{+0.9}$ | $96.3_{-2.7}^{+6.8}$ | –
x $\odot$ grad (sign) | $19.7_{-8.1}^{+6.6}$ | $65.7_{-26.3}^{+24.1}$ | –
x $\odot$ grad (abs) | $27.3_{-1.5}^{+1.7}$ | $92.0_{-3.1}^{+2.5}$ | $48.1_{-3.5}^{+3.2}$
IG (sign) | $40.3_{-0.8}^{+0.9}$ | $136.3_{-6.4}^{+4.4}$ | –
IG (abs) | $29.1_{-1.3}^{+1.0}$ | $98.3_{-3.9}^{+5.5}$ | $42.0_{-4.8}^{+3.8}$
LOO (sign) | $40.2_{-0.8}^{+1.2}$ | $136.0_{-6.5}^{+4.1}$ | –
LOO (abs) | $28.5_{-1.4}^{+0.9}$ | $96.3_{-3.6}^{+9.2}$ | –
Beam | $41.1_{-0.7}^{+1.0}$ | $139.2_{-7.3}^{+5.0}$ | –

Table 1: Faithfulness scores using Relative Area Between Curves (RACU) and the
non-relative variant (ACU). Higher is better, and negative values indicates
not-faithful. The comparison with Recursive-ROAR (Madsen et al., 2022) is
imperfect because Recursive-ROAR has limitations. However, the results
generally show higher faithfulness using masked fine-tuning.
The upper-bound used in RACU is not valid for signed IMs, hence we also report
the non-relative variant (ACU) and gray out the RACU scores. For absolute
importance measures, AUC is valid, but RACU may be more informative. Hence,
ACU is grayed out for absolute importance measures. See Table 11 for all
datasets and Table 12 for RoBERTa-large.

Note, that comparing with Recursive ROAR (R-ROAR) (Madsen et al., 2022) is
troublesome, because R-ROAR have issues, such as it leaking the gold label.
Additionally, while they also use RoBERTa-base it’s not the same model because
we use masked fine-tuning.

##  6 Discussion

Our proposed _faithfulness measurable masked language model_ is able to
maintain a high unmasked performance (Figure 2), while providing in-
distribution results for any permutation of masking (Figure 3 and Figure 4).
We demonstrate this is consistent across 16 datasets. As such, there is not
much to discuss regarding this, we therefore focus our discussion on the
matter of faithfulness.

The beam-search (Beam) method (Zhou and Shah, 2023) is one of the two most
faithful explanations (the other is leave-on-out). This is to be expected, as
it optimizes the explanation for faithfulness, which our proposed faithfulness
measurable masked language model makes computationally efficient to evaluate.

It is worth considering the significance of this. While our proposed model is
not an _inherently explainable model_ (Jacovi and Goldberg, 2020), it is
_indirectly_ inherently explainable because it provides a built-in way to
measure faithfulness, which can then be optimized for. It does this without
sacrificing the generality of the model, as it is still a RoBERTa model. As
such, a _inherently faithfulness measurable model_ bridges the gap (Jacovi and
Goldberg, 2020) between _post-hoc_ (Madsen et al., 2023) and _inherent_
interpretability (Rudin, 2019).

That being said, beam-search (Beam) is still an approximation and will only
find the optimal explanation with an infinitely wide beam. In particular, we
observe that Leave-one-out (LOO) performs better than beam-search in some
cases and is consistently very faithful too. This is because it takes
advantage of the masking support that masked fine-tuning provides. Leave-one-
out has the additional advantage that it is much more computationally
efficient. We hope that future work can improve upon algorithms that optimize
for faithfulness.

#### Additional observations.

Table 1 show that the RACU scores have a lower variance (confidence interval)
using our methodology compared to Recursive ROAR. This is likely because
Recursive ROAR leaks the gold label (Madsen et al., 2022), which causes
oscillation in the faithfulness curve.

We also observe that gradient-based explanations are more faithful when using
our model. We suspect this is partially also because there is no leakage
issue. However, previous work has also shown that gradient-based methods
behave more favorably on robust models in computer vision (Bansal et al.,
2020). Using masked fine-tuning can be seen as a robustness objective, as the
model becomes robust to missing information.

Surprisingly, input times gradient (x $\odot$ grad) appears to be the worst
explanation. A curious result, because both _gradient_ (Grad) and _integrated
gradient_ (IG) perform much better. Recall that _integrated gradient_ is
essentially $x\odot\int\text{grad}$ (Sundararajan et al., 2017). Perhaps, a
version of _integrated gradient_ where the x-hadamard product (x $\odot$) is
not used, might improve the faithfulness.

Finally, Diabetes and bAbI-2 get 90% RACU faithfulness (using absolute
importance measures). This suggests that there are only a few words that are
critical for the prediction of these tasks. This may indicate a problem with
their efficacy as benchmarking datasets. Although, for Diabetes which is
purely a diagnostic dataset (Jain and Wallace, 2019; Johnson et al., 2016),
this may be fine if those tokens should be critical to the task.

##  7 Conclusion

Using only a modified fine-tuning method, called _masked fine-tuning_ , we are
able to turn a typical general-purpose masked language model (e.g. RoBERTa)
into an _inherently faithfulness measurable model_ (FMM). Meaning, the model,
by design, inherently provides a way to measure the faithfulness of any
importance measure explanation.

To the best of our knowledge, this is the first work that proposes creating a
model designed to be faithfulness measurable. Importantly, our approach is
very general, simple to apply, and stratifies critical desirables that
previous measures didn’t.

The _masked fine-tuning_ method, does not decrease performance on 16 tested
benchmarks, while adding in-distribution support for token masking. This is
verified using statistical OOD tests.

We find that occlusion-based importance measures (IMs) are consistently the
most faithful, likely due to the explicit support for token masking.

Finally, while an _inherently faithfulness measurable model_ does not directly
provide explanations itself, like an _inherently explainable model_ (Jacovi
and Goldberg, 2020) does, it is possible to optimize an explanation with
respect to faithfulness. This is possible because our approach makes
faithfulness evaluation computationally trivial. As such, we bridge the long-
standing gap between _post-hoc_ and _inherent_ interpretability, creating a
new direction for interpretability.

## Limitations

### Impossible to compare faithfulness with and without masked fine-tuning

We make the claim that _masked fine-tuning_ makes importance measures (IMs)
more faithful. However, we do not show an ablation study where we measure
faithfulness without _masked fine-tuning_. Unfortunately, _masked fine-tuning_
is required to measure faithfulness, hence performing such an ablation study
is impossible.

However, our argument for occlusion-based IMs has a theoretical foundation, as
occlusion (i.e. masking) is only in-distribution because of _masked fine-
tuning_. Additionally, we also observe that occlusion-based IMs are
consistently more faithful than gradient-based IMs. Finally, for gradient-
based IMs we compare with Recursive ROAR (Madsen et al., 2022) and our
approach provide more faithful explanations, although this comparison is
imperfect as discussed in Section 5.3.

### Not a post-hoc method

While this work solves existing limitations with previous methods, it
introduces the significant limitation that it by definition requires a
_faithfulness measurable model_. As such, the question of faithfulness needs
to be considered ahead of time, when developing a model. It can not be an
afterthought, which is often how interpretability is approached (Bhatt et al.,
2019; Madsen et al., 2023).

While this is a significant limitation, considering explanation ahead of
deployment is increasingly becoming a legal requirement (Doshi-Velez et al.,
2017). Currently, the European Union provides a “right to explanation” in
regard to automatic decisions, which includes NLP models (Goodman and Flaxman,
2017).

### Uses masked language models (MLMs)

The proposed masked fine-tuning method heavily relies on MLMs, as we leverage
the pre-trained model’s partial support for token masking. Therefore, a
natural question is: what about other models which do not use masked language
modeling? In our paper, we did not explore this question.

However, it is possible to introduce the mask tokens to, for example,
generative language models. This could be done by masking random tokens in the
input sequence, while keeping the generation objective the same (the mask
tokens are not inferred). This is similar to how unknown-word tokens have been
used. It may also be possible to add this support to an already pre-trained
model in an additional pre-training step, before fine-tuning. However, we
suspect that including masking support in generative models is a more complex
and computationally intensive task, which is why we limited the scope to MLMs.

### In-distribution is impossible to prove

Because in-distribution is always the null hypothesis, it is impossible to
statistically show that inputs are truly in-distribution. The typical approach
to similar statistical questions333A similar well explored statistical
question, is how to show that the error in a linear model is normally
distributed. is to keep validating in-distribution using a variety of methods.
Unfortunately, the literature on this topic in deep learning is extremely
limited (Yang et al., 2021; Sun and Lampert, 2020; Heller and Soudry, 2022;
Dziedzic et al., 2022; Kwon et al., 2020).

We would therefore advocate for more work on identifying out-of-distribution
inputs, using non-parametric methods that primarily consider the model’s
internal state. The use of parametric methods or works that use axillary
models is more well-explored, but not useful for our purpose.

### Requires repeated measures on the test dataset

Because datasets have redundancies, it is necessary to reevaluate the
importance measures (Hooker et al., 2019; Madsen et al., 2022). This leads to
an increased computational cost.

However, unlike previous work (Madsen et al., 2022), our method only requires
reevaluation of the test dataset, which is often quite small. Additionally,
some IMs, such as the beam-search method (Zhou and Shah, 2023), take dataset
redundancies into account and therefore do not require reevaluation.
Reevaluation could be done if desired, but would result in the exact same
results.

### Measures only faithfulness

There are other important aspects to an importance measure, such as if the
explanation is useful to humans (Doshi-Velez and Kim, 2017), but our method
does not measure this. We consider this to be a separate topic, as it is an
HCI question that requires experimental studies with humans. While the topic
of faithfulness can not be measured by humans (Jacovi and Goldberg, 2020), as
neural networks are too complicated for humans to manually evaluate if an
explanation is true.

## Ethics Statement

Interpretability is an essential component when considering the ethical
deployment of a model. In this paper, we particularly consider deployment a
key motivator in our work, as we specify that for a _faithfulness measurable
model_ the deployed model and measured model should be the same. Without this
criterion, there is a dangerous potential for discrepancies between the
analyzed model and the deployed model.

However, there is still a risk that the faithfulness measure itself can be
wrong. This is problematic, as a false faithfulness measure would create
unjustified confidence in the importance measure. A key assumption with our
proposed metric is that the model provides in-distribution support for any
permutation of masked tokens. We take extra care to validate this assumption
using sound methodologies. Note that our work is not the only one with this
assumption, in particular ROAR-based metrics (Hooker et al., 2019; Madsen et
al., 2022) assume in-distribution behavior, but do not test for this.
Additionally, we attempt to provide complete transparency regarding all the
limitations of our faithfulness metric.

### Use of anonymized medical data

The Diabetes and Anemia datasets used in this work are based on the anonymized
open-access medical database MIMIC-III (Johnson et al., 2016). These datasets
are used because they are common in the faithfulness literature (Jain and
Wallace, 2019; Madsen et al., 2022). In particular, they contain many
redundancies and are long-sequence datasets. The specific authors of this
paper, who have worked with this data, have undergone the necessary HIPAA
certification in order to access this data and comply with HIPAA regulations.

## Acknowledgements

SC and SR are supported by the Canada CIFAR AI Chairs program and the NSERC
Discovery Grant. Computing resources were provided by the Digital Research
Alliance of Canada.

## References

  * Arras et al. (2017) Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2017.  "What is relevant in a text document?": An interpretable machine learning approach.  _PLOS ONE_ , 12(8):e0181142.
  * Arras et al. (2022) Leila Arras, Ahmed Osman, and Wojciech Samek. 2022.  CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations.  _Information Fusion_ , 81:14–40.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016.  Layer Normalization.  _Arxiv_.
  * Baehrens et al. (2010) David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus Robert Müller. 2010.  How to explain individual classification decisions.  _Journal of Machine Learning Research_ , 11:1803–1831.
  * Bansal et al. (2020) Naman Bansal, Chirag Agarwal, and Anh Nguyen. 2020.  SAM: The Sensitivity of Attribution Methods to Hyperparameters.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 11–21. IEEE.
  * Bastings et al. (2021) Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. 2021.  "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification.  _arXiv_.
  * Bhatt et al. (2019) Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. 2019.  Explainable Machine Learning in Deployment.  _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 648–657.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Buckland et al. (1998) S. T. Buckland, A. C. Davison, and D. V. Hinkley. 1998.  Bootstrap Methods and Their Application.  _Biometrics_ , 54(2):795.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference_ , volume 1, pages 2924–2936.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The PASCAL Recognising Textual Entailment Challenge.  In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence D’Alché-Buc, editors, _Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment_ , pages 177–190. Springer Berlin Heidelberg, Berlin, Heidelberg.
  * Devlin et al. (2019) Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference_ , volume 1, pages 4171–4186. Association for Computational Linguistics (ACL).
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020.  ERASER: A Benchmark to Evaluate Rationalized NLP Models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically Constructing a Corpus of Sentential Paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , pages 9–16.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards A Rigorous Science of Interpretable Machine Learning.  _arXiv_.
  * Doshi-Velez et al. (2017) Finale Doshi-Velez, Mason Kortz, Ryan Budish, Christopher Bavitz, Samuel J. Gershman, David O’Brien, Stuart Shieber, Jim Waldo, David Weinberger, and Alexandra Wood. 2017.  Accountability of AI Under the Law: The Role of Explanation.  _SSRN Electronic Journal_ , Online.
  * Dziedzic et al. (2022) Adam Dziedzic, Stephan Rabanser, Mohammad Yaghini, Armin Ale, Murat A. Erdogdu, and Nicolas Papernot. 2022.  $p$-DkNN: Out-of-Distribution Detection Through Statistical Testing of Deep Representations.  _arXiv_.
  * Ebrahimi et al. (2018) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018.  HotFlip: White-Box Adversarial Examples for Text Classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , volume 2, pages 31–36, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Fisher (1992) R A Fisher. 1992.  Statistical Methods for Research Workers.  In Samuel Kotz and Norman L Johnson, editors, _Breakthroughs in Statistics: Methodology and Distribution_ , pages 66–70. Springer New York, New York, NY.
  * Goodman and Flaxman (2017) Bryce Goodman and Seth Flaxman. 2017.  European union regulations on algorithmic decision making and a "right to explanation".  _AI Magazine_ , 38(3):50–57.
  * Heller and Soudry (2022) Ruth Heller and Daniel Soudry. 2022.  A Statistical Framework for Efficient Out of Distribution Detection in Deep Neural Networks.  _International Conference on Learning Representations_.
  * Hooker et al. (2019) Sara Hooker, Dumitru Erhan, Pieter-Jan Jan Kindermans, and Been Kim. 2019.  A benchmark for interpretability methods in deep neural networks.  In _Advances in Neural Information Processing Systems_ , volume 32.
  * Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017.  First Quora Dataset Release: Question Pairs.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North_ , volume 1, pages 3543–3556, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Johnson et al. (2016) Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016.  MIMIC-III, a freely accessible critical care database.  _Scientific Data_ , 3(1):160035.
  * Kindermans et al. (2016) Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, and Sven Dähne. 2016.  Investigating the influence of noise and distractors on the interpretation of neural networks.  In _NIPS Workshop on Interpretable Machine Learning in Complex Systems_.
  * Kwon et al. (2020) Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. 2020.  Backpropagated Gradient Representations for Anomaly Detection.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 12366 LNCS:206–226.
  * Li et al. (2016a) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016a.  Visualizing and Understanding Neural Models in NLP.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 681–691, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Li et al. (2016b) Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b.  Understanding Neural Networks through Representation Erasure.  _arXiv_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , volume 1, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Madsen et al. (2022) Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. 2022.  Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1731–1751, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Madsen et al. (2023) Andreas Madsen, Siva Reddy, and Sarath Chandar. 2023.  Post-hoc Interpretability for Neural NLP: A Survey.  _ACM Computing Surveys_ , 55(8):1–42.
  * Marneffe et al. (2019) Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The CommitmentBank: Investigating projection in naturally occurring discourse.  _Proceedings of Sinn und Bedeutung_ , 23(2):107–124.
  * Meister et al. (2021) Clara Meister, Stefan Lazov, Isabelle Augenstein, and Ryan Cotterell. 2021.  Is Sparse Attention more Interpretable? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 122–129, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Michael R. Chernick and LaBudde (2011) Michael R. Chernick and Robert A. LaBudde. 2011.  _An introduction to bootstrap methods with applications to R_.  John Wiley & Sons.
  * Moradi et al. (2021) Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. 2021.  Measuring and improving faithfulness of attention in neural machine translation.  _EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference_ , pages 2791–2802.
  * Mudrakarta et al. (2018) Pramod K. Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018.  Did the model understand the question? In _ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)_ , volume 1, pages 1896–1906.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuad: 100,000+ questions for machine comprehension of text.  _EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings_ , pages 2383–2392.
  * Ross et al. (2021) Alexis Ross, Ana Marasović, and Matthew Peters. 2021.  Explaining NLP Models via Minimal Contrastive Editing (MiCE).  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3840–3852, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Rudin (2019) Cynthia Rudin. 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215.
  * Samek et al. (2017) Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus Robert Müller. 2017.  Evaluating the visualization of what a deep neural network has learned.  _IEEE Transactions on Neural Networks and Learning Systems_ , 28(11):2660–2673.
  * Simes (1986) R. J. Simes. 1986.  An Improved Bonferroni Procedure for Multiple Tests of Significance.  _Biometrika_ , 73(3):751.
  * Socher et al. (2013) Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013.  Parsing with compositional vector grammars.  In _ACL 2013 - 51st Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_ , volume 1, pages 455–465. Association for Computational Linguistics.
  * Sun and Lampert (2020) Rémy Sun and Christoph H. Lampert. 2020.  KS(conf): A Light-Weight Test if a Multiclass Classifier Operates Outside of Its Specifications.  _International Journal of Computer Vision_ , 128(4):970–995.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _34th International Conference on Machine Learning, ICML 2017_ , volume 7, pages 5109–5118.
  * Vashishth et al. (2019) Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019.  Attention Interpretability Across NLP Tasks.  _arXiv_.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  _Advances in Neural Information Processing Systems_ , 32(July):1–30.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019b.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  In _International Conference on Learning Representations_.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural Network Acceptability Judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Weston et al. (2016) Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016.  Towards AI-complete question answering: A set of prerequisite toy tasks.  _4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings_.
  * Wettig et al. (2023) Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023.  Should You Mask 15% in Masked Language Modeling? _EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference_ , pages 2977–2992.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not Explanation.  _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Yang et al. (2021) Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2021.  Generalized Out-of-Distribution Detection: A Survey.  _arXiv_.
  * Yin and Neubig (2022) Kayo Yin and Graham Neubig. 2022.  Interpreting Language Models with Contrastive Explanations.  _ArXiv_ , page 184.
  * Zaman and Belinkov (2022) Kerem Zaman and Yonatan Belinkov. 2022.  A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_ , pages 1556–1576.
  * Zhou and Shah (2023) Yilun Zhou and Julie Shah. 2023.  The Solvability of Interpretability Evaluation Metrics.  In _Findings of the Association for Computational Linguistics: EACL_.

##  Appendix A Datasets

The datasets used in this work are all public and listed below. The Diabetes
and Anemia datasets are from MIMIC-III which requires a HIPPA certification
for data analysis (Johnson et al., 2016). The first author complies with this
and have not shared the data with any, including other authors.

Note that when computing the importance measure on paired-sequence tasks, only
the first sequence is considered. This is to stay consistent with previous
work (Jain and Wallace, 2019; Madsen et al., 2022) and because for tasks like
document based Q&A (e.g. bAbI) it does not make sense to mask the question.

The essential statistics for each dataset, and which part is masked and
auxiliary, are specified in Table 2.

Type | Dataset | Size | Inputs | Performance | Citation
---|---|---|---|---|---
|  | Train | Validation | Test | masked | auxilary | metric | class-majority |
NLI | RTE | $1992$ | $498$ | $277$ | sentence1 | sentence2 | Accuracy | $47\%$ | Dagan et al.,2006
SNLI | $549367$ | $9842$ | $9824$ | premise | hypothesis | Macro F1 | $34\%$ | Bowman et al.,2015
MNLI | $314162$ | $78540$ | $9815$ | premise | hypothesis | Accuracy | $35\%$ | Williams et al.,2018
QNLI | $83794$ | $20949$ | $5463$ | sentence | question | Accuracy | $51\%$ | Rajpurkar et al.,2016
CB | $200$ | $50$ | $56$ | premise | hypothesis | Macro F1 | $22\%$ | Marneffe et al.,2019
Paraphrase | MRPC | $2934$ | $734$ | $408$ | sentence1 | sentence2 | Macro F1 | $41\%$ | Dolan et al., 2005
QQP | $291077$ | $72769$ | $40430$ | question1 | question2 | Macro F1 | $39\%$ | Iyer et al.,2017
Sentiment | SST2 | $53879$ | $13470$ | $872$ | sentence | – | Accuracy | $51\%$ | Socher et al.,2013
IMDB | $20000$ | $5000$ | $25000$ | text | – | Macro F1 | $33\%$ | Maas et al.,2011
Diagnosis | Anemia | $4262$ | $729$ | $1243$ | text | – | Marco F1 | $39\%$ | Johnson et al.,2016
Diabetese | $8066$ | $1573$ | $1729$ | text | – | Marco F1 | $45\%$ | Johnson et al.,2016
Acceptability | CoLA | $6841$ | $1710$ | $1043$ | sentence | – | Matthew | $0\%$ | Warstadt et al.,2019
QA | BoolQ | $7542$ | $1885$ | $3270$ | passage | question | Accuracy | $62\%$ | Clark et al.,2019
bAbI-1 | $8000$ | $2000$ | $1000$ | paragraph | question | Micro F1 | $15\%$ | Weston et al.,2016
bAbI-2 | $8000$ | $2000$ | $1000$ | paragraph | question | Micro F1 | $19\%$ | Weston et al.,2016
bAbI-3 | $8000$ | $2000$ | $1000$ | paragraph | question | Micro F1 | $18\%$ | Weston et al.,2016

Table 2: Datasets used, all datasets are either single-sequence or sequence-
pair datasets. All datasets are sourced from GLUE (Wang et al., 2019b),
SuperGLUE (Wang et al., 2019a), MIMIC-III (Johnson et al., 2016), or bAbI
(Weston et al., 2016). The decisions regarding which metrics are used, are
also from these sources. The class-majority baseline is when the most frequent
class is always selected.

##  Appendix B Models

In this paper, we use the RoBERTa model (Liu et al., 2019), although any
masked language model of similar size or larger is likely to work. We choose
RoBERTa model, because converges consistently and reasonable hyperparameters
are well established. This should make reproducing the results in this paper
easier. We use both the base (125M parameters) and large size (355M
parameters).

The hyperparameters are defined by Liu et al. (2019, Appendix C, GLUE).
Although these hyperparameters are for the GLUE tasks, we use them for all
tasks. The one exception, is that the maximum number of epoch is higher. This
is because when _masked fine-tuning_ require more epochs. In Table 3 we
specify the max epoch parameter. However, when using early stopping with the
validation dataset, the optimization is not sensitive to the specific number
of epochs, lower numbers are only used to reduce the compute time.

Dataset | max epoch | Performance
---|---|---
|  | RoBERTa- | RoBERTa-
|  | base | large
BoolQ | 15 | ${80\%}^{+0.2}_{-0.2}$ | ${85\%}^{+0.3}_{-0.3}$
CB | 50 | ${65\%}^{+17.6}_{-47.9}$ | ${87\%}^{+3.1}_{-8.2}$
CoLA | 15 | ${59\%}^{+1.3}_{-1.1}$ | ${66\%}^{+0.8}_{-0.8}$
IMDB | 10 | ${95\%}^{+0.2}_{-0.2}$ | ${96\%}^{+0.2}_{-0.4}$
Anemia | 20 | ${84\%}^{+0.8}_{-0.7}$ | ${84\%}^{+0.5}_{-0.8}$
Diabetes | 20 | ${76\%}^{+0.9}_{-0.9}$ | ${77\%}^{+0.6}_{-1.6}$
MNLI | 10 | ${87\%}^{+0.4}_{-0.2}$ | ${90\%}^{+0.3}_{-0.2}$
MRPC | 20 | ${86\%}^{+0.8}_{-0.7}$ | ${87\%}^{+0.5}_{-1.2}$
QNLI | 20 | ${92\%}^{+0.1}_{-0.1}$ | ${94\%}^{+0.1}_{-0.2}$
QQP | 10 | ${90\%}^{+0.1}_{-0.1}$ | ${91\%}^{+0.0}_{-0.1}$
RTE | 30 | ${75\%}^{+1.4}_{-2.9}$ | ${83\%}^{+1.3}_{-1.4}$
SNLI | 10 | ${91\%}^{+0.1}_{-0.2}$ | ${92\%}^{+0.1}_{-0.2}$
SST2 | 10 | ${94\%}^{+0.2}_{-0.2}$ | ${96\%}^{+0.2}_{-0.2}$
bAbI-1 | 20 | ${100\%}^{+0.0}_{-0.1}$ | ${100\%}^{+0.0}_{-0.0}$
bAbI-2 | 20 | ${99\%}^{+0.1}_{-0.1}$ | ${100\%}^{+0.1}_{-0.1}$
bAbI-3 | 20 | ${90\%}^{+0.2}_{-0.3}$ | ${90\%}^{+0.5}_{-0.5}$
Table 3: Dataset statistics. Performance metrics are the mean with a 95%
confidence interval.

##  Appendix C Importance measure details

This section provides additional details on the importance measures, which
were only briefly described in Section 4. In particular, not all importance
measures have both signed and absolute variants (Table 4), this section should
clarify why.

Category | Full name | Short name | Variants
---|---|---|---
Gradient | Gradient w.r.t. input | Grad | absolute
Input times gradient | x $\odot$ grad | both
Integrated gradient | IG | both
Occlusion | Leave-on-out | LOO | both
Beam-search | Beam | signed

Table 4: Overview of the importance measures used in this paper. Note that not
all importance measures exists in both signed and absolute variants due to
their mathematical construction.

###  C.1 Gradient-based

The common idea is that if a small change in the input causes a large change
in the output, then that indicates importance.

#### Gradient (Grad)

measures the mentioned relationship using the gradient, which is a linear-
approximation. Let $f(\mathbf{x})$ be the model with input
$\mathbf{x}\in\mathbb{R}^{T\times V}$ ($V$ is vocabulary-size and $T$ is
sequence-length), then the gradient is
$\nabla_{\mathbf{x}}f(\mathbf{x})_{y}\in\mathbb{R}^{T\times V}$ (Baehrens et
al., 2010; Li et al., 2016a). Because the desire is an importance measure for
each token (i.e. $\mathbb{R}^{T}$), the vocabulary dimension is
typically444The topic of vocabulary-dimention reduction is rarely discussed in
papers. reduced using either $L_{1}$ or $L_{2}$ norm. In this paper, we
consider both.

#### Input times Gradient (x $\odot$ grad)

multiplies the gradient with the input, which some argue to be better
(Kindermans et al., 2016) i.e.
$\mathbf{x}\odot\nabla_{\mathbf{x}}f(\mathbf{x})\in\mathbb{R}^{T\times V}$.
Because $\mathbf{x}$ is a one-hot-encoding, using any norm function is just
the absolute value of the non-zero element, which is what is typically used.
However, we observe that for NLP, one could also consider the signed version
of this, where the non-zero element is simply picked out. In this paper, we
consider both the signed and absolute variants.

#### Integrated Gradient (IG)

by Sundararajan et al. (2017) is a very popular explanation method. This can
be seen as an extension of the _input times gradient_ method, and therefore we
also consider a signed and absolute variant for this. The method works by
sampling gradients between a baseline $f(\mathbf{b})$ and the input
$f(\mathbf{x})$. We use a zero-vector baseline and 20 samples, as is commonly
done in NLP literature (Mudrakarta et al., 2018)

$\displaystyle\operatorname{IG}(\mathbf{x})$
$\displaystyle=(\mathbf{x}-\mathbf{b})\odot\frac{1}{k}\sum_{i=1}^{k}\nabla_{\tilde{\mathbf{x}}_{i}}f(\tilde{\mathbf{x}}_{i})_{c}$
(1) $\displaystyle\tilde{\mathbf{x}}_{i}$
$\displaystyle=\mathbf{b}+\frac{i}{k}(\mathbf{x}-\mathbf{b}).$

###  C.2 Occlusion-based

Rather than linear-approximating the relation between input and output using
gradients, the relationship can also be approximated by removing or masking
each token, one by one, and measuring how is affects the output.

One rarely addressed concern, is that removing/masking tokens is likely to be
out-of-distribution (Zhou and Shah, 2023). This would cause an otherwise sound
method to become unfaithful. However, because the proposed _faithfulness
measurable masked language model_ supports masking, this should not be a
concern. Importantly, this exemplifies how a _faithfulness measurable model_
may also produce more faithful explanations.

#### Leave-on-out (LOO)

directly computes the difference between the model output difference between
the unmasked input and the input with the $i$’th token masked, i.e. the
importance is
$\\{f(\mathbf{\hat{x}})_{y}-f(\mathbf{\tilde{x}}_{i})_{y}\\}_{i=1}^{T}\in\mathbb{R}^{T}$
(Li et al., 2016b). Even though this importance measure does not have a
vocabulary dimension, it is common to take its absolute (Meister et al.,
2021). However, we consider both absolute and signed variants.

#### Optimizing for faithfulness (Beam).

Zhou and Shah (2023) propose that we can optimize for the faithfulness metric
itself, rather than using heuristics. The central idea is to use beam-search,
where the generated sequence is the optimal masking order of tokens. Each
iteration of the beam-search masks one additional token, where the token is
selected by testing every possibility and maximizing the faithfulness metric.
This could be reframed as a recursive version of leave-on-out. They propose
several optimization targets, but since our faithfulness metric is analog to
comprehensiveness, we use this variation.

The number of forward passes is $\mathcal{O}(B\cdot T^{2})$, it is not exactly
$B\cdot T^{2}$ because many of them become redundant. This is quite
computationally costly, although one advantage is that this explanation is
inherently recursive, hence it is not necessary to reevaluate the importance
measure in each iteration of the faithfulness metric. However, for long
sequence datasets, such as IMDB, BaBi-3, Anemia, and Diabetes it is not
feasible to apply this explanation. In our experiments, we use a beam-size of
$B=10$.

##  Appendix D Workflow algorithms

This section provides the details and algorithms for the workflow, which was
described in Section 3.

###  D.1 Masked fine-tuning

Masked fine-tuning is a multi-task learning methods, where one task is the
typical unmasked performance and the other task is masking support. We achieve
this by masking half of a mini-batch uniformly, at between 0% and 100%
masking, and do not modify the other half. Note that this is slightly
different from some multi-task learning methods, which may sample randomly
between the two task, where we split deterministically. Other methods may also
switch between the two tasks in each step, we don’t do this as it can create
unstable oscillations. Instead, both tasks are included in the same mini-
batch.

There are many approaches to implementing masked fine-tuning with identical
results, however Algorithm 1 presents our implementation.

Algorithm 1 Creates the mini-batches used in masked fine-tuning.

1$B$ is a mini-batch with $N$ randomly sampled observations from the training
dataset. [$\mathcal{M}$] is the masking token.

2

3function MiniBatch($B$)

4 $M$ $\leftarrow$ $\varnothing$ $\triangleright$ Stores new mini-batch

5 for $i\leftarrow 1$ to $N$ do

6 if $i$ is even then

7 $r$ $\leftarrow$ SampleUniform(0, 1)

8 $M_{i}$ $\leftarrow$ MaskTokens($r$, $B_{i}$) $\triangleright$ Masks $r\%$
randomly selected tokens in $B_{i}$.

9 else

10 $M_{i}$ $\leftarrow$ $B_{i}$

11 return $M$

Algorithm 2 Continues Algorithm 1.

1

2function MaskTokens($x$, $r$)

3 $\tilde{x}$ $\leftarrow$ x

4 for $t\leftarrow 1$ to $T$ do

5 $s$ $\leftarrow$ SampleUniform(0, 1)

6 if $s<r$ then

7 $x_{t}$ $\leftarrow$ [$\mathcal{M}$] $\triangleright$ Masks token $t$.

8 return $\tilde{x}$

###  D.2 In-distribution validation (MaSF)

_MaSF_ is a statistical in-distribution test developed by Heller and Soudry
(2022). _MaSF_ is an acronym that stands for Max-Simes-Fisher, which is the
order of aggregation functions it uses. Heller and Soudry (2022) presented a
number of other combinations and orders of aggregation functions but found
this to be the best, although many combinations had similar performance in
their benchmark, so we do not consider this particular choice to be important
and assume it generalizes well to our case. In Section 5.2 we verified this
with an ablation study.

At is core, _MaSF_ is an aggregation of many in-distribution p-values, where
each p-value is from an in-distribution test of a latent embedding. That is,
given a history of embedding observations, which presents a distribution, what
is the probability of observing the new embedding or something more extreme.
If for example that probability is less than 5%, it could be classified as
out-of-distribution at a 5% risk of a false-positive.

However, a model has many internal embeddings and thus there will be many
p-values. If each were tested independently, there would be many false-
positives. This is known as p-hacking. To prevent this, the p-values are
aggregated using the Simes and Fisher methods, which are aggregation methods
for p-values that prevents this issue. Once the p-values are aggregated, it
becomes a “global null-test”. This means, the aggregated statistical test,
checks if any of the embeddings are out-of-distribution.

#### Emperical CDF.

Each p-value is computed using an empirical communicative density function
(CDF). A nice property of an empirical CDF is that it doesn’t assume any
distribution, a property called non-paramatic. It is however still a model,
only if an infinite amount of data was available would it represent the true
distribution.

A CDF measures the probability that of observing $z$ or less than $z$, i.e.
$\mathbb{P}(Z\leq z)$. The empirical version of this simply counts how many
embeddings were historically less than the tested embedding, as shown in (2).
However, as we are also interested in cases where the embedding is abnormally
large, hence we also use $\mathbb{P}(Z>z)=1-\mathbb{P}(Z\leq z)$. We are then
interested in the most unlikely case, which is known as the two-sided p-value,
i.e. $\min(\mathbb{P}(Z\leq z),1-\mathbb{P}(Z\leq z))$.

$\mathbb{P}(Z\leq
z)=\frac{1}{|Z_{\mathrm{emp}}|}\sum_{i=1}^{|Z_{\mathrm{emp}}|}1[Z_{\mathrm{emp},i}<z]$
(2)

The historical embeddings are collected by running the model on the validation
dataset. Note that for this to be accurate, the validation dataset should be
i.i.d. with the training dataset. This can easily be accomplished by randomly
splitting the datasets, which is common practice, and transform the validation
dataset the same way as the training dataset.

#### Algorithm.

In the case of _MaSF_ , the embeddings are first aggregated along the
sequence-dimension using the max-operation. Heller and Soudry (2022) only
applied _MaSF_ to computer vision, in which case it was the width and height
dimensions. However, we generalize this to NLP by swapping width and height
with the sequence-dimention.

The max-aggregated embeddings from the validation dataset, then provide the
historical data for the empirical CDFs. If a network has $L$ layers, each with
$H$ latent dimensions, there will be $L\cdot H$ CDFs. The same max-aggregated
embeddings are then transformed to p-values using those CDFs. Next, the
p-values are aggregated using Simes’s method (Simes, 1986) along the latent-
dimention, which provides another set of CDFs and p-values, one for each layer
($L$ CDFs and p-values). Finally, those p-values are aggregated using Fisher’s
method (Fisher, 1992), providing one CDF and one p-value for each observation.

The algorithm for _MaSF_ can be found in Algorithm 3. However, while this
algorithm does work, a practical implementation is in our experience non-
trivial, as for an entire test dataset ($\mathcal{D}_{T}$) there are
$\mathcal{O}(|\mathcal{D}_{T}|\cdot H\cdot L)$ CDFs evaluations, each
involving $\mathcal{O}(\mathcal{D}_{V})$ comparisons. While this is
computationally trivial on a GPU, it can require a lot of memory usage when
done in parallel. We therefore found that a practical implementation must
batch over both the test dataset and validation dataset.

Algorithm 3 MaSF algorithm, which provides p-values under the in-distribution
null-hypothesis.

1$x$ is the input. $f_{e}(x)\in\mathbb{R}^{T\times H\times L}$ provides the
model embeddings, where $T$ is sequence-length, $H$ is the hidden-size, and
$L$ is the number of layers. $\mathbb{P}$ are the emperical CDFs, these are
collected by running the MaSF algorithm on a validation dataset.

2

3function MaSF($x$, $\mathbb{P}$)

4 $e$ $\leftarrow$ $f_{e}(x)$ $\triangleright$ Get embeddings

5 for $l\leftarrow 1$ to $L$ do

6 for $h\leftarrow 1$ to $H$ do

7 $z_{l,h}^{(1)}$ $\leftarrow$ $\max_{t=1}^{T}e_{l,h,t}$ $\triangleright$ Ma-
step

8 $\tilde{p}_{l,h}^{(1)}$ $\leftarrow$
$\mathbb{P}^{(1)}_{l,h}(Z<z_{l,h}^{(1)})$

9 $p_{l,h}^{(1)}$ $\leftarrow$
$\min(\tilde{p}_{l,h}^{(1)},1-\tilde{p}_{l,h}^{(1)})$

10 $z_{l}^{(2)}$ $\leftarrow$ $\mathrm{Simes}(p_{l,:}^{(1)})$ $\triangleright$
S-step

11 $\tilde{p}_{l}^{(2)}$ $\leftarrow$ $\mathbb{P}^{(2)}_{l}(Z<z_{l}^{(2)})$

12 $p_{l}^{(2)}$ $\leftarrow$
$\min(\tilde{p}_{l}^{(2)},1-\tilde{p}_{l}^{(2)})$

13 $z^{(3)}$ $\leftarrow$ $\mathrm{Fisher}(p^{(2)})$ $\triangleright$ F-step

14 $\tilde{p}^{(3)}$ $\leftarrow$ $\mathbb{P}^{(3)}(Z<z^{(3)})$

15 $p^{(3)}$ $\leftarrow$ $1-\tilde{p}^{(3)}$

16 return $p^{(3)}$

17

18function Simes($p$)

19 $q$ $\leftarrow$ $\mathrm{SortAscending}(p)$

20 return $\min_{i=1}^{N}q_{i}\frac{N}{i}$

21

22function Fisher($p$)

23 return $-2\sum_{i=1}^{N}\log(p_{i})$

###  D.3 Faithfulness metric

The faithfulness is measured by masking $10\%$ the tokens according to an
importance measure, such that the most important tokens are masked. This new
10%-masked observation is then used as a new input, and the model prediction
is explained. This is then repeated, until 100% masking. This process is
identical to the Recursive ROAR approach by (Madsen et al., 2022), except re-
training is not needed. For an explanation, we define the evaluation procedure
in Algorithm 4. However, in terms of Figure 6, Algorithm 4 only provides the
“importance measure” curve. To get the random baseline, simply apply Algorithm
4 again with a random explanation.

Algorithm 4 Measures the masked model performance given an explanation.

1$\mathrm{IM}(f,x,y)\in\mathbb{R}^{T}$ explains the model $f$ for the input
$x$ and label $y$, with sequence-length $T$. $\delta$ is the iterations step-
size (e.g. 10%) and [$\mathcal{M}$] is the masking token.

2

3function RecursiveEval($\mathrm{IM}$, $f$, $x$, $y$, $\delta$)

4 $\tilde{x}_{0}$ $\leftarrow$ $x$

5 $p_{0}$ $\leftarrow$ $\mathrm{PerformanceMetric}(f(x),y)$

6 for $i\leftarrow 1$ to $1/\delta$ do

7 $e_{i}$ $\leftarrow$ $\mathrm{IM}(f,\tilde{x}_{i-1},y)$

8 $\tilde{x}_{i}$ $\leftarrow$
$\mathrm{AddMask}(e_{i},\tilde{x}_{i-1},\delta)$ $\triangleright$ Mask
$\delta\cdot T$ more tokens in $\tilde{x}_{i-1}$ using scores $e_{i}$.

9 $p_{i}$ $\leftarrow$ $\mathrm{PerformanceMetric}(f(\tilde{x}_{i}),y)$

10 return $p$

Figure 6: Visualization of the faithfulness calculation done in (3). The
_faithfulness_ area is the numerator in (3), while the _normalizer_ area is
the denominator. – Figure by Madsen et al. (2022), included with permission.

#### Relative Area between Curves (RACU)

ACU is the “faithfulness” area between the “importance measure” curve and the
“random” curve as shown in Figure 6. This was not explicitly defined by Madsen
et al. (2022) who defined RACU.

RACU, is a normalization of ACU, which normalizes by the theoretically optimal
explanation, which is one that archives the performance of 100% masking
immediately. However, as argued in Section 5.3 this normalization is only
valid for absolute importance measures. Therefore, we include the unnormalized
score (AUC) in this paper too. Both metrics are defined in Equation 3.

Unfortunately, because different models (e.g. RoBERTa-base versus RoBERTa-
large) have different performance caritaristics, it is not possible to compare
unnormalized score (AUC) between different models (Arras et al., 2017). For
this reason, RACU may still be useful for signed metrics, even though it can
not be interpreted as a 0%-100% scale.

$\displaystyle\operatorname{ACU}$
$\displaystyle=\sum_{i=0}^{I-1}\frac{1}{2}\Delta x_{i}(\Delta p_{i}+\Delta
p_{i+1})$ (3) $\displaystyle\operatorname{RACU}$
$\displaystyle=\frac{\operatorname{ACU}}{\sum_{i=0}^{I-1}\frac{1}{2}\Delta
x_{i}(\Delta b_{i}+\Delta b_{i+1})}$ $\displaystyle\text{where }\Delta x_{i}$
$\displaystyle=x_{i+1}-x_{i}\quad\textit{step size}$ $\displaystyle\Delta
p_{i}$ $\displaystyle=b_{i}-p_{i}\quad\textit{performance delta}$
$\displaystyle\Delta b_{i}$ $\displaystyle=b_{i}-b_{I}\quad\textit{baseline
delta}$

##  Appendix E Compute

This section reports the compute resources and requirements. The compute
hardware specifications are in Table 5 and were the same for all experiments.
All compute was performed using 99% hydroelectric power.

CPU |  12 cores, Intel Silver 4216 Cascade Lake @ 2.1GHz
---|---
GPU |  1x NVidia V100 (32G HBM2 memory)
Memory |  24 GB
Table 5: The compute hardware used. Note, that a shared user system were used,
only the allocated resources are reported.

Note that the importance measures computed are the same for both the
faithfulness results and the out-of-distribution results. Hence, these do not
need to be computed twice. Additionally, the beam-search method is in itself
recursive, so this was only computed for 0% masking.

###  E.1 Walltimes

We here include the walltimes for all experiments.

  * •

Table 6 shows wall-times for the masked fine-tuning.

  * •

Table 7 shows wall-times for the in-distribution validation, not including
importance measures.

  * •

Table 8 shows wall-times for the faithfulness evaluation, not including
importance measures.

  * •

Table 9 shows wall-times for the importance measures.

Dataset | Walltime [hh:mm]
---|---
| RoBERTa- | RoBERTa-
| base | large
BoolQ | 00:51 | 02:03
CB | 00:06 | 00:12
CoLA | 00:17 | 00:33
IMDB | 01:44 | 04:02
Anemia | 00:48 | 02:04
Diabetes | 01:34 | 04:04
MNLI | 06:39 | 14:47
MRPC | 00:12 | 00:27
QNLI | 04:03 | 09:12
QQP | 05:13 | 11:52
RTE | 00:18 | 00:43
SNLI | 04:57 | 10:38
SST2 | 01:19 | 02:44
bAbI-1 | 00:27 | 01:01
bAbI-2 | 00:50 | 02:05
bAbI-3 | 01:43 | 04:28
sum | 01:43 | 04:28
x5 seeds | 08:37 | 22:21
Table 6: Walltime for fine-tuning. Masked fine-tuning does not affect the total walltime in our setup. Dataset | Walltime [hh:mm]
---|---
| RoBERTa- | RoBERTa-
| base | large
BoolQ | 00:04 | 00:09
CB | 00:01 | 00:02
CoLA | 00:02 | 00:04
IMDB | 00:15 | 00:44
Anemia | 00:01 | 00:03
Diabetes | 00:01 | 00:04
MNLI | 00:20 | 00:57
MRPC | 00:02 | 00:03
QNLI | 00:05 | 00:12
QQP | 00:47 | 02:13
RTE | 00:02 | 00:04
SNLI | 00:04 | 00:09
SST2 | 00:09 | 00:25
bAbI-1 | 00:01 | 00:03
bAbI-2 | 00:02 | 00:05
bAbI-3 | 00:01 | 00:03
sum | 02:06 | 05:25
x5 seeds | 10:30 | 27:09
Table 7: Walltime for in-distribution validation. This does not include importance measure calculations. See Table 9. Dataset | Walltime [hh:mm]
---|---
| RoBERTa- | RoBERTa-
| base | large
BoolQ | 00:02 | 00:05
CB | 00:00 | 00:01
CoLA | 00:01 | 00:02
IMDB | 00:13 | 00:39
Anemia | 00:01 | 00:02
Diabetes | 00:01 | 00:03
MNLI | 00:02 | 00:05
MRPC | 00:01 | 00:02
QNLI | 00:01 | 00:04
QQP | 00:05 | 00:13
RTE | 00:01 | 00:02
SNLI | 00:01 | 00:03
SST2 | 00:01 | 00:01
bAbI-1 | 00:00 | 00:01
bAbI-2 | 00:01 | 00:02
bAbI-3 | 00:00 | 00:02
sum | 00:38 | 01:34
x5 seeds | 03:13 | 07:51
Table 8: Walltime for faithfulness evaluation. This does not include
importance measure calculations. See Table 9.

Dataset | IM | Walltime [hh:mm]
---|---|---
|  | RoBERTa- | RoBERTa-
|  | base | large
bAbI-1 | Beam | 00:54 | 02:24
Grad ($L_{1}$) | 00:01 | 00:04
Grad ($L_{2}$) | 00:02 | 00:04
x $\odot$ grad (abs) | 00:01 | 00:04
x $\odot$ grad (sign) | 00:01 | 00:04
IG (abs) | 00:04 | 00:12
IG (sign) | 00:04 | 00:11
LOO (abs) | 00:24 | 00:49
LOO (sign) | 00:24 | 00:49
Random | 00:00 | 00:00
bAbI-2 | Beam | 20:56 | 61:24
Grad ($L_{1}$) | 00:02 | 00:06
Grad ($L_{2}$) | 00:02 | 00:05
x $\odot$ grad (abs) | 00:02 | 00:05
x $\odot$ grad (sign) | 00:02 | 00:05
IG (abs) | 00:10 | 00:29
IG (sign) | 00:10 | 00:29
LOO (abs) | 00:39 | 01:26
LOO (sign) | 00:39 | 01:26
Random | 00:00 | 00:00
bAbI-3 | Beam | – | –
Grad ($L_{1}$) | 00:02 | 00:05
Grad ($L_{2}$) | 00:02 | 00:05
x $\odot$ grad (abs) | 00:01 | 00:04
x $\odot$ grad (sign) | 00:01 | 00:04
IG (abs) | 00:18 | 00:54
IG (sign) | 00:18 | 00:53
LOO (abs) | 01:09 | 03:18
LOO (sign) | 01:09 | 03:18
Random | 00:00 | 00:00
BoolQ | Beam | 00:33 | 01:22
Grad ($L_{1}$) | 00:05 | 00:11
Grad ($L_{2}$) | 00:05 | 00:11
x $\odot$ grad (abs) | 00:04 | 00:10
x $\odot$ grad (sign) | 00:04 | 00:10
IG (abs) | 00:39 | 01:48
IG (sign) | 00:39 | 01:49
LOO (abs) | 00:16 | 00:38
LOO (sign) | 00:16 | 00:38
Random | 00:00 | 00:00
CB | Beam | 00:45 | 02:09
Grad ($L_{1}$) | 00:01 | 00:03
Grad ($L_{2}$) | 00:01 | 00:03
x $\odot$ grad (abs) | 00:01 | 00:03
x $\odot$ grad (sign) | 00:01 | 00:03
IG (abs) | 00:01 | 00:04
IG (sign) | 00:01 | 00:04
LOO (abs) | 00:09 | 00:19
LOO (sign) | 00:09 | 00:19
Random | 00:00 | 00:00
CoLA | Beam | 00:11 | 00:19
Grad ($L_{1}$) | 00:02 | 00:05
Grad ($L_{2}$) | 00:02 | 00:04
x $\odot$ grad (abs) | 00:02 | 00:04
x $\odot$ grad (sign) | 00:02 | 00:04
IG (abs) | 00:04 | 00:09
IG (sign) | 00:04 | 00:09
LOO (abs) | 00:09 | 00:18
LOO (sign) | 00:09 | 00:18
Random | 00:00 | 00:00
Anemia | Beam | – | –
Grad ($L_{1}$) | 00:02 | 00:06
Grad ($L_{2}$) | 00:02 | 00:06
x $\odot$ grad (abs) | 00:01 | 00:05
x $\odot$ grad (sign) | 00:01 | 00:05
IG (abs) | 00:23 | 01:08
IG (sign) | 00:23 | 01:08
LOO (abs) | 02:23 | 06:58
LOO (sign) | 02:23 | 07:01
Random | 00:00 | 00:00
Diabetes | Beam | – | –
Grad ($L_{1}$) | 00:03 | 00:07
Grad ($L_{2}$) | 00:03 | 00:07
x $\odot$ grad (abs) | 00:02 | 00:06
x $\odot$ grad (sign) | 00:02 | 00:06
IG (abs) | 00:32 | 01:34
IG (sign) | 00:32 | 01:34
LOO (abs) | 03:19 | 09:44
LOO (sign) | 03:17 | 09:45
Random | 00:00 | 00:00

Dataset | IM | Walltime [hh:mm]
---|---|---
|  | RoBERTa- | RoBERTa-
|  | base | large
MRPC | Beam | 00:14 | 00:33
Grad ($L_{1}$) | 00:02 | 00:04
Grad ($L_{2}$) | 00:02 | 00:04
x $\odot$ grad (abs) | 00:02 | 00:04
x $\odot$ grad (sign) | 00:02 | 00:04
IG (abs) | 00:03 | 00:07
IG (sign) | 00:03 | 00:07
LOO (abs) | 00:08 | 00:17
LOO (sign) | 00:08 | 00:17
Random | 00:00 | 00:00
RTE | Beam | 01:32 | 04:26
Grad ($L_{1}$) | 00:02 | 00:04
Grad ($L_{2}$) | 00:02 | 00:04
x $\odot$ grad (abs) | 00:02 | 00:04
x $\odot$ grad (sign) | 00:02 | 00:04
IG (abs) | 00:04 | 00:09
IG (sign) | 00:04 | 00:09
LOO (abs) | 00:10 | 00:22
LOO (sign) | 00:10 | 00:22
Random | 00:00 | 00:00
SST2 | Beam | 00:18 | 00:43
Grad ($L_{1}$) | 00:02 | 00:04
Grad ($L_{2}$) | 00:02 | 00:04
x $\odot$ grad (abs) | 00:02 | 00:04
x $\odot$ grad (sign) | 00:02 | 00:04
IG (abs) | 00:04 | 00:09
IG (sign) | 00:04 | 00:09
LOO (abs) | 00:09 | 00:19
LOO (sign) | 00:10 | 00:19
Random | 00:00 | 00:00
SNLI | Beam | 01:10 | 02:38
Grad ($L_{1}$) | 00:05 | 00:07
Grad ($L_{2}$) | 00:06 | 00:07
x $\odot$ grad (abs) | 00:05 | 00:06
x $\odot$ grad (sign) | 00:04 | 00:06
IG (abs) | 00:21 | 00:57
IG (sign) | 00:21 | 00:56
LOO (abs) | 00:12 | 00:26
LOO (sign) | 00:12 | 00:26
Random | 00:01 | 00:00
IMDB | Beam | – | –
Grad ($L_{1}$) | 00:34 | 01:18
Grad ($L_{2}$) | 00:34 | 01:17
x $\odot$ grad (abs) | 00:22 | 01:03
x $\odot$ grad (sign) | 00:22 | 01:03
IG (abs) | 06:49 | 20:08
IG (sign) | 06:54 | 20:09
LOO (abs) | 25:02 | 73:17
LOO (sign) | 24:48 | 72:55
Random | 00:01 | 00:01
MNLI | Beam | 05:44 | 15:34
Grad ($L_{1}$) | 00:05 | 00:11
Grad ($L_{2}$) | 00:05 | 00:11
x $\odot$ grad (abs) | 00:04 | 00:09
x $\odot$ grad (sign) | 00:04 | 00:09
IG (abs) | 00:35 | 01:35
IG (sign) | 00:35 | 01:34
LOO (abs) | 00:19 | 00:46
LOO (sign) | 00:19 | 00:46
Random | 00:00 | 00:00
QNLI | Beam | 06:39 | 18:51
Grad ($L_{1}$) | 00:04 | 00:08
Grad ($L_{2}$) | 00:04 | 00:08
x $\odot$ grad (abs) | 00:03 | 00:07
x $\odot$ grad (sign) | 00:03 | 00:08
IG (abs) | 00:23 | 01:03
IG (sign) | 00:23 | 01:04
LOO (abs) | 00:17 | 00:43
LOO (sign) | 00:17 | 00:43
Random | 00:00 | 00:00
QQP | Beam | 04:44 | 11:12
Grad ($L_{1}$) | 00:12 | 00:26
Grad ($L_{2}$) | 00:12 | 00:26
x $\odot$ grad (abs) | 00:10 | 00:22
x $\odot$ grad (sign) | 00:10 | 00:22
IG (abs) | 01:48 | 04:57
IG (sign) | 01:48 | 04:59
LOO (abs) | 00:36 | 01:24
LOO (sign) | 00:36 | 01:23
Random | 00:01 | 00:01
sum | 145:00 | 406:58
x5 seeds | 725:02 | 2034:53

Table 9: Walltime for importance measures. Note that because the beam-search
method (Beam) scales quadratic with the sequence-length, it is not feasible to
compute for all datasets.

##  Appendix F Masked fine-tuning

In, Section 5.1 we show selected results for unmasked performance and 100%
masked performance. In this appendix, we extend those results to all 16
datasets. In addition to this, this appendix contains a more detailed ablation
study, where the training strategy and validation strategy are considered
separate. As such, the results in Section 5.1 are a strict subset of these
detailed results. In Table 10 we show how the terminologies relate.

Section 5.1 | Training strategy | Validation strategy
---|---|---
Maskedfine-tuning | Use 50/50 | Use both
Plainfine-tuning | No masking | No masking
Only masking | Masking | Masking
Table 10: This table relates terminologies between the fine-tuning strategies
mentioned in Section 5.1 and the training strategy and validation strategy
terms.

#### Training strategy

The training strategy applies to the training dataset during fine-tuning.

No masking

No masking applied to the training dataset. This is what is ordinarily done in
the literature.

Masking

Masking is applied to every observation. The masking is uniformly sampled, at
a masking rate between 0% and 100%.

Use 50/50

Half of the mini-batch using the _No masking_ strategy and the other half use
the _Masking_ strategy.

#### Validation strategy

The validation dataset is used to select the optional epoch. This is similar
to early stopping, but rather than stopping immediately. The training
continues, and the best epoch is chosen at the end of training.

The validation strategy applies to the validation dataset during fine-tuning.

No masking

No masking applied to the validation dataset. This is what is ordinarily done
in the literature.

Masking

Masking is applied to every observation. The masking is uniformly sampled, at
a masking rate between 0% and 100%.

Use both

A copy of the validation dataset has the _No masking_ strategy applied to it.
Another copy of the validation dataset has the _Masking_ strategy applied to
it. As such, the validation dataset is twice as long, but it does not add
additional observation.

###  F.1 Findings

We generally find that choice of validation strategy when using the _Use
50/50_ training strategy is not important. Interestingly, _Masking_ for the
validation dataset and _No masking_ for the training dataset often works too.

However, because _Use 50/50_ for training strategy and _Use both_ for
validation strategy, i.e. masked fine-tuning, work well in all cases and is
theoretically sound, this is the approach we recommend and use throughout the
paper.

###  F.2 All datasets aggregation

In Section 5.1 we also include an _All_ “dataset”. This is a simple arithmetic
mean over all the performance of all 16 datasets. This is similar to how the
GLUE benchmark (Wang et al., 2019b) works. To compute the confidence interval,
a dataset-aggregation is done for each seed, such that the all-observation are
i.i.d..

Because some seeds do not converge for some datasets, such as bAbI-2 and
bAbI-3 (as mentioned in Section 5.1), those outliers and not included in the
aggregation, also hyperparameter optimization will likely help. For complete
transparency, we do include them in the statistics for the individual datasets
and show all individual performances with a (+) symbol.

Figure 7: The all aggregation for the 100% masked performance and unmasked
performance. The baseline (dashed line) for 100% masked performance is the
class-majority baseline. For unmasked performance, it is when using no masking
for both validation and training.

###  F.3 Test dataset

Figure 8: The unmasked performance for each validation and training strategy,
using the test dataset. Not that _“No masking”_ as a _training strategy_ is
not a valid option only a baseline, as it creates OOD issues. We find that the
multi-task _training strategy_ _“Use 50/50”_ works best. This plot is page-1.
Corresponding main results in Figure 2. Figure 9: The unmasked performance for
each validation and training strategy, using the test dataset. Not that _“No
masking”_ as a _training strategy_ is not a valid option only a baseline, as
it creates OOD issues. We find that the multi-task _training strategy_ _“Use
50/50”_ works best. This plot is page-2. Corresponding main results in Figure
2. Figure 10: The 100% masked performance, using the test dataset. The dashed
line represents the class-majority classifier baseline. Results show that
masking during training (_“Masking”_ or _“Use 50/50”_) is necessary. This plot
is page-1. Corresponding main paper results in Figure 3. Figure 11: The 100%
masked performance, using the test dataset. The dashed line represents the
class-majority classifier baseline. Results show that masking during training
(_“Masking”_ or _“Use 50/50”_) is necessary. This plot is page-2.
Corresponding main paper results in Figure 3.

###  F.4 Validation dataset

Figure 12: The unmasked performance for each validation and training strategy,
using the validation dataset. Not that _“No masking”_ as a _training strategy_
is not a valid option only a baseline, as it creates OOD issues. We find that
the multi-task _training strategy_ _“Use 50/50”_ works best. This plot is
page-1. Figure 13: The unmasked performance for each validation and training
strategy, using the validation dataset. Not that _“No masking”_ as a _training
strategy_ is not a valid option only a baseline, as it creates OOD issues. We
find that the multi-task _training strategy_ _“Use 50/50”_ works best. This
plot is page-2. Figure 14: The 100% masked performance, using the validation
dataset. The dashed line represents the class-majority classifier baseline.
Results show that masking during training (_“Masking”_ or _“Use 50/50”_) is
necessary. This plot is page-1. Figure 15: The 100% masked performance, using
the validation dataset. The dashed line represents the class-majority
classifier baseline. Results show that masking during training (_“Masking”_ or
_“Use 50/50”_) is necessary. This plot is page-2.

##  Appendix G Convergence speed

Figure 16: The validation performance for each epoch. Note that the max number
of epochs vary depending on the dataset. This is only to limit the compute
requirements when fine-tuning. The best epoch is selected by the “early-
stopping” dataset, which has one copy with no masking and one copy with
uniformly sampled masking ratios. This plot is page-1. Figure 17: The
validation performance for each epoch. Note that the max number of epochs vary
depending on the dataset. This is only to limit the compute requirements when
fine-tuning. The best epoch is selected by the “early-stopping” dataset, which
has one copy with no masking and one copy with uniformly sampled masking
ratios. This plot is page-2.

##  Appendix H In-distribution validation

Figure 18: In-distribution p-values using MaSF, for RoBERTa-base with and
without masked fine-tuning, page-1. The masked tokens are chosen according to
an importance measure. P-values below the dashed line show out-of-distribution
(OOD) results, given a 5% risk of a false positive. Results show that only
when using masked fine-tuning, masked data is consistently not OOD.
Corresponding main paper results are in Figure 4. Figure 19: In-distribution
p-values using MaSF, for RoBERTa-base with and without masked fine-tuning,
page-2. The masked tokens are chosen according to an importance measure.
P-values below the dashed line show out-of-distribution (OOD) results, given a
5% risk of a false positive. Results show that only when using masked fine-
tuning, masked data is consistently not OOD. Corresponding main paper results
are in Figure 4. Figure 20: In-distribution p-values using MaSF, for RoBERTa-
large with and without masked fine-tuning, page-1. The masked tokens are
chosen according to an importance measure. P-values below the dashed line show
out-of-distribution (OOD) results, given a 5% risk of a false positive.
Results show that only when using masked fine-tuning, masked data is
consistently not OOD. Corresponding main paper results are in Figure 4. Figure
21: In-distribution p-values using MaSF, for RoBERTa-large with and without
masked fine-tuning, page-2. The masked tokens are chosen according to an
importance measure. P-values below the dashed line show out-of-distribution
(OOD) results, given a 5% risk of a false positive. Results show that only
when using masked fine-tuning, masked data is consistently not OOD.
Corresponding main paper results are in Figure 4.

##  Appendix I Faithfulness metrics

Figure 22: The performance given the masked datasets, where masking is done
for the x% allegedly most important tokens according to the importance
measure. If the performance for a given explanation is below the _“Random”_
baseline, this shows faithfulness. Although, faithfulness is not an absolute
concept, so more is better. This plot is page-1 for RoBERTa-base.
Corresponding main paper results in Section 5.3. Figure 23: The performance
given the masked datasets, where masking is done for the x% allegedly most
important tokens according to the importance measure. If the performance for a
given explanation is below the _“Random”_ baseline, this shows faithfulness.
Although, faithfulness is not an absolute concept, so more is better. This
plot is page-2 for RoBERTa-base. Corresponding main paper results in Section
5.3. Figure 24: The performance given the masked datasets, where masking is
done for the x% allegedly most important tokens according to the importance
measure. If the performance for a given explanation is below the _“Random”_
baseline, this shows faithfulness. Although, faithfulness is not an absolute
concept, so more is better. This plot is page-1 for RoBERTa-large.
Corresponding main paper results in Section 5.3. Figure 25: The performance
given the masked datasets, where masking is done for the x% allegedly most
important tokens according to the importance measure. If the performance for a
given explanation is below the _“Random”_ baseline, this shows faithfulness.
Although, faithfulness is not an absolute concept, so more is better. This
plot is page-2 for RoBERTa-large. Corresponding main paper results in Section
5.3.

###  I.1 Relative Area between Curves (RACU)

|  | Faithfulness [%]
---|---|---
Dataset | IM | Our | R-ROAR
|  | ACU | RACU | RACU
bAbI-1 | Grad ($L_{2}$) | $32.6_{-1.1}^{+1.2}$ | $91.9_{-3.2}^{+2.0}$ | $64.2_{-2.6}^{+2.6}$
Grad ($L_{1}$) | $32.7_{-0.9}^{+1.3}$ | $92.1_{-3.1}^{+2.0}$ | –
x $\odot$ grad (sign) | $21.4_{-7.3}^{+4.9}$ | $60.5_{-20.7}^{+13.7}$ | –
x $\odot$ grad (abs) | $27.0_{-1.4}^{+1.1}$ | $76.0_{-1.6}^{+3.0}$ | $52.1_{-3.7}^{+1.8}$
IG (sign) | $44.1_{-0.8}^{+0.8}$ | $124.2_{-4.5}^{+3.3}$ | –
IG (abs) | $33.3_{-2.4}^{+2.4}$ | $93.7_{-4.1}^{+5.5}$ | $48.2_{-5.7}^{+4.1}$
LOO (sign) | $46.1_{-0.7}^{+0.7}$ | $129.9_{-3.5}^{+3.4}$ | –
LOO (abs) | $32.4_{-1.2}^{+1.2}$ | $91.2_{-0.7}^{+1.2}$ | –
Beam | $45.9_{-0.7}^{+0.7}$ | $129.2_{-3.2}^{+3.1}$ | –
bAbI-2 | Grad ($L_{2}$) | $28.5_{-0.8}^{+0.8}$ | $96.3_{-2.8}^{+6.8}$ | $57.8_{-2.0}^{+2.0}$
Grad ($L_{1}$) | $28.5_{-0.8}^{+0.9}$ | $96.3_{-2.7}^{+6.8}$ | –
x $\odot$ grad (sign) | $19.7_{-8.1}^{+6.6}$ | $65.7_{-26.3}^{+24.1}$ | –
x $\odot$ grad (abs) | $27.3_{-1.5}^{+1.7}$ | $92.0_{-3.1}^{+2.5}$ | $48.1_{-3.5}^{+3.2}$
IG (sign) | $40.3_{-0.8}^{+0.9}$ | $136.3_{-6.4}^{+4.4}$ | –
IG (abs) | $29.1_{-1.3}^{+1.0}$ | $98.3_{-3.9}^{+5.5}$ | $42.0_{-4.8}^{+3.8}$
LOO (sign) | $40.2_{-0.8}^{+1.2}$ | $136.0_{-6.5}^{+4.1}$ | –
LOO (abs) | $28.5_{-1.4}^{+0.9}$ | $96.3_{-3.6}^{+9.2}$ | –
Beam | $41.1_{-0.7}^{+1.0}$ | $139.2_{-7.3}^{+5.0}$ | –
bAbI-3 | Grad ($L_{2}$) | $23.5_{-1.5}^{+0.9}$ | $97.3_{-4.2}^{+3.5}$ | $34.0_{-15.1}^{+14.6}$
Grad ($L_{1}$) | $23.5_{-1.3}^{+0.9}$ | $97.2_{-4.3}^{+3.6}$ | –
x $\odot$ grad (sign) | $23.1_{-2.5}^{+2.4}$ | $96.7_{-14.9}^{+21.8}$ | –
x $\odot$ grad (abs) | $24.4_{-1.2}^{+0.7}$ | $101.2_{-5.4}^{+4.3}$ | $22.4_{-12.4}^{+15.9}$
IG (sign) | $36.6_{-1.4}^{+1.3}$ | $152.5_{-17.4}^{+14.7}$ | –
IG (abs) | $24.4_{-1.3}^{+1.0}$ | $100.9_{-3.9}^{+4.0}$ | $-27.9_{-49.1}^{+18.0}$
LOO (sign) | $38.7_{-0.7}^{+0.9}$ | $160.5_{-13.2}^{+13.8}$ | –
LOO (abs) | $23.4_{-1.1}^{+0.7}$ | $97.0_{-4.6}^{+4.5}$ | –
Beam | – | – | –
BoolQ | Grad ($L_{2}$) | $4.1_{-0.3}^{+0.2}$ | $50.1_{-3.2}^{+3.6}$ | –
Grad ($L_{1}$) | $4.1_{-0.3}^{+0.2}$ | $50.1_{-3.1}^{+4.0}$ | –
x $\odot$ grad (sign) | $-1.3_{-0.4}^{+0.7}$ | $-16.0_{-4.5}^{+8.2}$ | –
x $\odot$ grad (abs) | $1.7_{-0.3}^{+0.2}$ | $21.3_{-3.3}^{+2.7}$ | –
IG (sign) | $9.6_{-0.6}^{+0.6}$ | $118.3_{-7.9}^{+10.3}$ | –
IG (abs) | $3.0_{-0.4}^{+0.2}$ | $37.8_{-5.3}^{+4.7}$ | –
LOO (sign) | $26.3_{-1.4}^{+1.9}$ | $323.1_{-15.2}^{+14.4}$ | –
LOO (abs) | $5.3_{-0.2}^{+0.3}$ | $65.2_{-2.2}^{+3.3}$ | –
Beam | $21.2_{-1.1}^{+1.4}$ | $261.4_{-14.5}^{+10.7}$ | –
CB | Grad ($L_{2}$) | $20.3_{-1.3}^{+1.3}$ | $64.7_{-9.9}^{+9.9}$ | –
Grad ($L_{1}$) | $20.2_{-1.3}^{+1.4}$ | $64.7_{-10.7}^{+10.7}$ | –
x $\odot$ grad (sign) | $6.0_{-3.6}^{+4.0}$ | $17.8_{-11.0}^{+7.0}$ | –
x $\odot$ grad (abs) | $15.9_{-3.2}^{+2.2}$ | $52.0_{-14.3}^{+15.7}$ | –
IG (sign) | $30.1_{-5.7}^{+5.7}$ | $97.6_{-28.3}^{+22.7}$ | –
IG (abs) | $21.1_{-2.3}^{+2.3}$ | $67.5_{-13.2}^{+10.8}$ | –
LOO (sign) | $40.9_{-5.5}^{+5.8}$ | $131.4_{-27.3}^{+26.7}$ | –
LOO (abs) | $21.0_{-2.4}^{+1.7}$ | $67.4_{-12.8}^{+12.8}$ | –
Beam | $51.0_{-6.8}^{+6.4}$ | $163.5_{-33.8}^{+32.7}$ | –
CoLA | Grad ($L_{2}$) | $12.7_{-1.1}^{+0.9}$ | $43.4_{-3.2}^{+4.0}$ | –
Grad ($L_{1}$) | $12.7_{-1.2}^{+0.8}$ | $43.2_{-3.4}^{+3.7}$ | –
x $\odot$ grad (sign) | $-2.4_{-1.1}^{+1.9}$ | $-8.3_{-3.7}^{+6.0}$ | –
x $\odot$ grad (abs) | $8.0_{-0.8}^{+1.0}$ | $27.1_{-2.9}^{+2.9}$ | –
IG (sign) | $8.1_{-0.8}^{+1.1}$ | $27.5_{-2.6}^{+2.8}$ | –
IG (abs) | $8.7_{-0.8}^{+1.7}$ | $29.8_{-3.0}^{+6.6}$ | –
LOO (sign) | $52.0_{-2.9}^{+1.7}$ | $177.2_{-7.8}^{+7.2}$ | –
LOO (abs) | $17.0_{-0.4}^{+0.4}$ | $57.9_{-0.9}^{+1.2}$ | –
Beam | $50.6_{-2.3}^{+1.3}$ | $172.7_{-7.2}^{+5.9}$ | –
Anemia | Grad ($L_{2}$) | $23.8_{-0.5}^{+0.6}$ | $62.1_{-1.7}^{+1.4}$ | $18.2_{-13.8}^{+11.8}$
Grad ($L_{1}$) | $23.8_{-0.6}^{+0.6}$ | $62.2_{-2.1}^{+1.4}$ | –
x $\odot$ grad (sign) | $9.7_{-2.5}^{+2.6}$ | $25.1_{-6.2}^{+6.5}$ | –
x $\odot$ grad (abs) | $16.6_{-1.3}^{+1.3}$ | $43.2_{-3.7}^{+3.0}$ | $8.8_{-22.8}^{+22.7}$
IG (sign) | $62.0_{-1.5}^{+1.6}$ | $161.8_{-2.3}^{+2.7}$ | –
IG (abs) | $20.0_{-1.6}^{+0.9}$ | $52.1_{-4.5}^{+2.5}$ | $12.5_{-7.0}^{+11.3}$
LOO (sign) | $63.3_{-1.6}^{+1.4}$ | $165.2_{-3.2}^{+2.9}$ | –
LOO (abs) | $18.9_{-1.3}^{+1.0}$ | $49.2_{-3.8}^{+2.5}$ | –
Beam | – | – | –
Diabetes | Grad ($L_{2}$) | $19.7_{-0.7}^{+1.2}$ | $91.8_{-0.9}^{+0.6}$ | $57.9_{-19.8}^{+14.4}$
Grad ($L_{1}$) | $19.6_{-0.7}^{+1.0}$ | $91.6_{-0.9}^{+0.5}$ | –
x $\odot$ grad (sign) | $10.9_{-1.3}^{+1.9}$ | $51.1_{-7.8}^{+9.1}$ | –
x $\odot$ grad (abs) | $18.8_{-0.7}^{+1.3}$ | $87.9_{-2.0}^{+1.4}$ | $53.4_{-29.3}^{+23.2}$
IG (sign) | $24.8_{-2.1}^{+1.5}$ | $115.8_{-7.8}^{+2.8}$ | –
IG (abs) | $19.4_{-0.6}^{+1.0}$ | $90.5_{-1.2}^{+0.6}$ | $26.1_{-25.1}^{+12.0}$
LOO (sign) | $41.5_{-5.9}^{+2.3}$ | $193.4_{-17.4}^{+8.9}$ | –
LOO (abs) | $19.1_{-0.6}^{+1.1}$ | $89.0_{-0.6}^{+0.4}$ | –
Beam | – | – | –

|  | Faithfulness [%]
---|---|---
Dataset | IM | Our | R-ROAR
|  | ACU | RACU | RACU
MRPC | Grad ($L_{2}$) | $7.8_{-0.6}^{+0.5}$ | $36.8_{-6.1}^{+4.3}$ | –
Grad ($L_{1}$) | $7.9_{-0.7}^{+0.5}$ | $37.4_{-6.3}^{+4.2}$ | –
x $\odot$ grad (sign) | $-2.9_{-0.6}^{+0.5}$ | $-13.7_{-2.9}^{+2.9}$ | –
x $\odot$ grad (abs) | $3.3_{-0.8}^{+0.6}$ | $15.6_{-4.9}^{+3.4}$ | –
IG (sign) | $12.7_{-1.0}^{+0.8}$ | $59.9_{-10.6}^{+6.2}$ | –
IG (abs) | $5.4_{-0.6}^{+1.0}$ | $25.4_{-3.9}^{+7.0}$ | –
LOO (sign) | $37.1_{-1.3}^{+2.4}$ | $174.4_{-18.9}^{+12.4}$ | –
LOO (abs) | $9.9_{-0.9}^{+0.8}$ | $47.0_{-8.9}^{+5.5}$ | –
Beam | $45.6_{-1.5}^{+2.6}$ | $214.8_{-25.1}^{+13.3}$ | –
RTE | Grad ($L_{2}$) | $9.4_{-1.0}^{+0.8}$ | $73.9_{-19.3}^{+23.6}$ | –
Grad ($L_{1}$) | $9.4_{-1.0}^{+0.8}$ | $73.5_{-19.1}^{+23.3}$ | –
x $\odot$ grad (sign) | $0.1_{-1.7}^{+2.3}$ | $-3.5_{-13.9}^{+17.3}$ | –
x $\odot$ grad (abs) | $5.7_{-0.5}^{+0.5}$ | $44.3_{-8.5}^{+11.8}$ | –
IG (sign) | $20.2_{-2.2}^{+1.3}$ | $156.6_{-28.4}^{+45.5}$ | –
IG (abs) | $7.3_{-0.9}^{+1.6}$ | $55.9_{-13.7}^{+14.8}$ | –
LOO (sign) | $44.4_{-1.9}^{+1.6}$ | $344.8_{-75.5}^{+92.4}$ | –
LOO (abs) | $9.4_{-0.6}^{+0.9}$ | $73.2_{-15.2}^{+21.1}$ | –
Beam | $51.3_{-1.7}^{+2.0}$ | $399.7_{-89.9}^{+109.9}$ | –
SST2 | Grad ($L_{2}$) | $12.2_{-0.7}^{+0.6}$ | $40.4_{-1.7}^{+3.0}$ | $26.1_{-2.2}^{+1.6}$
Grad ($L_{1}$) | $12.1_{-0.7}^{+0.7}$ | $40.3_{-1.8}^{+3.3}$ | –
x $\odot$ grad (sign) | $-3.7_{-1.6}^{+1.5}$ | $-12.2_{-6.0}^{+4.5}$ | –
x $\odot$ grad (abs) | $7.1_{-0.2}^{+0.2}$ | $23.5_{-1.1}^{+1.9}$ | $18.6_{-4.6}^{+4.1}$
IG (sign) | $31.8_{-2.2}^{+2.8}$ | $105.6_{-7.7}^{+7.7}$ | –
IG (abs) | $13.7_{-0.8}^{+0.8}$ | $45.3_{-2.8}^{+4.1}$ | $32.9_{-1.5}^{+1.8}$
LOO (sign) | $51.6_{-0.9}^{+1.4}$ | $171.3_{-6.2}^{+5.8}$ | –
LOO (abs) | $16.6_{-1.0}^{+1.2}$ | $54.9_{-1.5}^{+2.1}$ | –
Beam | $56.4_{-0.7}^{+0.5}$ | $187.3_{-7.1}^{+8.1}$ | –
SNLI | Grad ($L_{2}$) | $8.9_{-0.4}^{+1.0}$ | $62.2_{-1.7}^{+2.7}$ | $50.7_{-0.8}^{+1.1}$
Grad ($L_{1}$) | $8.9_{-0.4}^{+1.0}$ | $62.2_{-1.8}^{+2.6}$ | –
x $\odot$ grad (sign) | $1.3_{-0.8}^{+0.7}$ | $9.2_{-4.9}^{+5.2}$ | –
x $\odot$ grad (abs) | $6.4_{-0.3}^{+0.5}$ | $44.8_{-1.6}^{+1.6}$ | $41.0_{-0.5}^{+0.4}$
IG (sign) | $16.3_{-1.3}^{+1.3}$ | $113.5_{-6.4}^{+7.6}$ | –
IG (abs) | $8.9_{-0.5}^{+0.8}$ | $62.3_{-2.1}^{+2.5}$ | $56.7_{-1.1}^{+1.0}$
LOO (sign) | $26.6_{-0.4}^{+0.4}$ | $186.6_{-12.4}^{+9.1}$ | –
LOO (abs) | $10.5_{-0.5}^{+0.8}$ | $73.6_{-1.9}^{+2.0}$ | –
Beam | $29.3_{-0.3}^{+0.2}$ | $205.2_{-8.7}^{+8.1}$ | –
IMDB | Grad ($L_{2}$) | $24.9_{-4.3}^{+5.9}$ | $47.8_{-8.5}^{+10.9}$ | $25.4_{-2.0}^{+3.1}$
Grad ($L_{1}$) | $24.9_{-4.4}^{+5.7}$ | $47.8_{-8.3}^{+10.9}$ | –
x $\odot$ grad (sign) | $-3.2_{-1.0}^{+1.3}$ | $-6.2_{-1.9}^{+2.5}$ | –
x $\odot$ grad (abs) | $12.9_{-2.6}^{+4.5}$ | $24.7_{-5.1}^{+8.7}$ | $16.9_{-3.0}^{+1.1}$
IG (sign) | $40.3_{-1.9}^{+3.3}$ | $77.3_{-3.6}^{+6.3}$ | –
IG (abs) | $18.4_{-2.8}^{+4.2}$ | $35.4_{-5.9}^{+8.0}$ | $35.1_{-1.7}^{+2.4}$
LOO (sign) | $68.0_{-0.9}^{+0.9}$ | $130.7_{-1.5}^{+1.3}$ | –
LOO (abs) | $20.7_{-3.2}^{+4.2}$ | $39.8_{-6.0}^{+9.5}$ | –
Beam | – | – | –
MNLI | Grad ($L_{2}$) | $8.8_{-0.2}^{+0.3}$ | $49.6_{-1.2}^{+1.1}$ | –
Grad ($L_{1}$) | $8.8_{-0.2}^{+0.3}$ | $49.7_{-1.2}^{+1.1}$ | –
x $\odot$ grad (sign) | $0.6_{-0.7}^{+0.6}$ | $3.5_{-3.8}^{+3.7}$ | –
x $\odot$ grad (abs) | $5.7_{-0.2}^{+0.2}$ | $32.0_{-0.8}^{+0.7}$ | –
IG (sign) | $18.5_{-0.7}^{+0.6}$ | $104.3_{-5.6}^{+5.9}$ | –
IG (abs) | $9.1_{-0.2}^{+0.2}$ | $51.3_{-2.0}^{+1.0}$ | –
LOO (sign) | $34.7_{-0.5}^{+0.7}$ | $195.7_{-2.8}^{+3.2}$ | –
LOO (abs) | $11.8_{-0.3}^{+0.3}$ | $66.7_{-1.1}^{+0.9}$ | –
Beam | $40.5_{-0.4}^{+0.8}$ | $228.9_{-4.5}^{+4.5}$ | –
QNLI | Grad ($L_{2}$) | $12.7_{-0.7}^{+0.6}$ | $40.8_{-2.0}^{+2.4}$ | –
Grad ($L_{1}$) | $12.7_{-0.7}^{+0.6}$ | $40.8_{-1.9}^{+2.3}$ | –
x $\odot$ grad (sign) | $-3.7_{-1.1}^{+1.9}$ | $-11.9_{-3.4}^{+6.0}$ | –
x $\odot$ grad (abs) | $8.8_{-0.6}^{+0.6}$ | $28.3_{-1.7}^{+2.1}$ | –
IG (sign) | $24.4_{-2.0}^{+4.3}$ | $78.0_{-7.4}^{+10.4}$ | –
IG (abs) | $11.4_{-0.6}^{+0.7}$ | $36.5_{-2.6}^{+2.0}$ | –
LOO (sign) | $46.1_{-2.6}^{+1.8}$ | $147.7_{-8.7}^{+4.7}$ | –
LOO (abs) | $14.0_{-0.4}^{+0.6}$ | $44.9_{-1.3}^{+3.2}$ | –
Beam | $55.7_{-2.2}^{+1.5}$ | $178.4_{-8.6}^{+4.8}$ | –
QQP | Grad ($L_{2}$) | $4.5_{-0.1}^{+0.1}$ | $40.0_{-0.8}^{+0.4}$ | –
Grad ($L_{1}$) | $4.5_{-0.1}^{+0.1}$ | $40.0_{-0.7}^{+0.4}$ | –
x $\odot$ grad (sign) | $-0.8_{-0.6}^{+0.3}$ | $-7.3_{-5.5}^{+3.0}$ | –
x $\odot$ grad (abs) | $2.8_{-0.2}^{+0.2}$ | $24.5_{-1.5}^{+1.4}$ | –
IG (sign) | $9.1_{-0.3}^{+0.4}$ | $81.3_{-2.3}^{+2.6}$ | –
IG (abs) | $3.5_{-0.2}^{+0.2}$ | $31.1_{-1.3}^{+2.3}$ | –
LOO (sign) | $22.0_{-0.9}^{+0.6}$ | $195.4_{-7.7}^{+6.1}$ | –
LOO (abs) | $5.6_{-0.2}^{+0.1}$ | $49.6_{-1.8}^{+0.8}$ | –
Beam | $23.3_{-0.6}^{+0.5}$ | $207.3_{-5.7}^{+6.6}$ | –

Table 11: Faithfulness scores for RoBERTa-base. Shows Relative Area Between
Curves (RACU) and the non-relative variant (ACU), defined by Madsen et al.
(2022). Also compares with Recursive-ROAR by Madsen et al. (2022).

|  | Faithfulness [%]
---|---|---
Dataset | IM | Our | R-ROAR
|  | ACU | RACU | RACU
bAbI-1 | Grad ($L_{2}$) | $31.1_{-1.5}^{+1.0}$ | $87.6_{-2.3}^{+1.6}$ | –
Grad ($L_{1}$) | $31.1_{-1.6}^{+1.1}$ | $87.7_{-2.5}^{+1.6}$ | –
x $\odot$ grad (sign) | $13.9_{-6.5}^{+3.1}$ | $39.1_{-16.0}^{+8.8}$ | –
x $\odot$ grad (abs) | $24.0_{-3.2}^{+2.6}$ | $67.5_{-8.3}^{+5.7}$ | –
IG (sign) | $36.3_{-3.1}^{+2.9}$ | $102.4_{-6.9}^{+7.7}$ | –
IG (abs) | $31.6_{-1.5}^{+1.3}$ | $88.9_{-2.7}^{+2.5}$ | –
LOO (sign) | $44.4_{-1.0}^{+0.9}$ | $125.2_{-1.7}^{+1.7}$ | –
LOO (abs) | $32.4_{-0.6}^{+0.4}$ | $91.4_{-0.9}^{+0.4}$ | –
Beam | $44.7_{-1.0}^{+0.8}$ | $126.0_{-1.1}^{+1.1}$ | –
bAbI-2 | Grad ($L_{2}$) | $28.2_{-1.5}^{+0.9}$ | $94.0_{-3.4}^{+6.3}$ | –
Grad ($L_{1}$) | $28.0_{-1.4}^{+0.9}$ | $93.5_{-2.8}^{+5.3}$ | –
x $\odot$ grad (sign) | $8.2_{-4.9}^{+7.0}$ | $26.9_{-19.4}^{+20.4}$ | –
x $\odot$ grad (abs) | $22.6_{-1.5}^{+1.5}$ | $75.5_{-5.2}^{+7.7}$ | –
IG (sign) | $37.9_{-1.3}^{+1.3}$ | $126.6_{-9.7}^{+5.8}$ | –
IG (abs) | $27.4_{-1.7}^{+1.7}$ | $91.7_{-9.5}^{+6.8}$ | –
LOO (sign) | $40.6_{-0.7}^{+1.9}$ | $135.4_{-4.0}^{+4.0}$ | –
LOO (abs) | $28.1_{-0.9}^{+0.9}$ | $93.9_{-1.8}^{+2.8}$ | –
Beam | $41.7_{-0.7}^{+1.8}$ | $139.2_{-4.3}^{+4.3}$ | –
bAbI-3 | Grad ($L_{2}$) | $22.4_{-3.8}^{+3.8}$ | $94.7_{-0.2}^{+0.2}$ | –
Grad ($L_{1}$) | $22.2_{-3.6}^{+3.6}$ | $94.1_{-0.4}^{+0.4}$ | –
x $\odot$ grad (sign) | $8.5_{-3.5}^{+3.5}$ | $34.4_{-9.0}^{+9.0}$ | –
x $\odot$ grad (abs) | $19.9_{-4.6}^{+4.6}$ | $83.2_{-5.5}^{+5.5}$ | –
IG (sign) | $33.0_{-3.8}^{+3.8}$ | $141.3_{-7.7}^{+7.7}$ | –
IG (abs) | $24.3_{-3.7}^{+3.7}$ | $103.2_{-1.5}^{+1.5}$ | –
LOO (sign) | $35.0_{-3.0}^{+3.0}$ | $150.5_{-12.4}^{+12.4}$ | –
LOO (abs) | $23.3_{-4.2}^{+4.2}$ | $98.7_{-1.2}^{+1.2}$ | –
Beam | – | – | –
BoolQ | Grad ($L_{2}$) | $2.6_{-0.3}^{+0.1}$ | $24.8_{-1.9}^{+1.7}$ | –
Grad ($L_{1}$) | $2.7_{-0.3}^{+0.2}$ | $25.3_{-1.9}^{+2.3}$ | –
x $\odot$ grad (sign) | $-0.4_{-0.1}^{+0.2}$ | $-3.6_{-1.5}^{+1.6}$ | –
x $\odot$ grad (abs) | $1.2_{-0.3}^{+0.3}$ | $10.8_{-2.0}^{+2.0}$ | –
IG (sign) | $6.9_{-0.8}^{+1.2}$ | $65.6_{-6.9}^{+15.7}$ | –
IG (abs) | $3.2_{-0.8}^{+0.4}$ | $30.3_{-5.0}^{+4.6}$ | –
LOO (sign) | $25.6_{-2.7}^{+1.7}$ | $242.9_{-26.4}^{+39.2}$ | –
LOO (abs) | $6.2_{-0.5}^{+0.6}$ | $58.0_{-1.2}^{+3.3}$ | –
Beam | $21.5_{-2.4}^{+1.3}$ | $204.0_{-21.7}^{+30.8}$ | –
CB | Grad ($L_{2}$) | $10.1_{-4.2}^{+4.4}$ | $30.8_{-12.9}^{+15.2}$ | –
Grad ($L_{1}$) | $8.9_{-4.3}^{+3.9}$ | $27.3_{-13.2}^{+12.6}$ | –
x $\odot$ grad (sign) | $-0.1_{-3.3}^{+2.7}$ | $0.6_{-8.1}^{+9.5}$ | –
x $\odot$ grad (abs) | $5.6_{-2.9}^{+1.8}$ | $17.4_{-9.8}^{+6.1}$ | –
IG (sign) | $28.5_{-3.4}^{+3.9}$ | $85.3_{-14.0}^{+19.0}$ | –
IG (abs) | $17.1_{-2.2}^{+3.2}$ | $51.3_{-9.0}^{+10.5}$ | –
LOO (sign) | $39.0_{-2.4}^{+2.9}$ | $116.1_{-15.8}^{+9.5}$ | –
LOO (abs) | $19.0_{-2.4}^{+4.0}$ | $56.6_{-12.8}^{+8.8}$ | –
Beam | $51.8_{-3.0}^{+3.0}$ | $154.6_{-22.6}^{+14.0}$ | –
CoLA | Grad ($L_{2}$) | $10.9_{-0.7}^{+0.9}$ | $35.0_{-2.5}^{+2.9}$ | –
Grad ($L_{1}$) | $10.4_{-0.6}^{+0.9}$ | $33.3_{-2.2}^{+3.1}$ | –
x $\odot$ grad (sign) | $1.2_{-0.3}^{+0.2}$ | $3.8_{-0.9}^{+0.5}$ | –
x $\odot$ grad (abs) | $6.6_{-0.9}^{+0.9}$ | $21.3_{-2.9}^{+2.9}$ | –
IG (sign) | $17.3_{-1.1}^{+1.1}$ | $55.4_{-3.2}^{+3.6}$ | –
IG (abs) | $11.7_{-0.3}^{+0.4}$ | $37.5_{-0.9}^{+1.0}$ | –
LOO (sign) | $38.9_{-2.0}^{+4.1}$ | $124.9_{-6.0}^{+11.4}$ | –
LOO (abs) | $17.6_{-1.3}^{+0.9}$ | $56.7_{-3.4}^{+3.2}$ | –
Beam | $37.5_{-2.7}^{+4.7}$ | $120.3_{-8.2}^{+13.4}$ | –
Anemia | Grad ($L_{2}$) | $18.6_{-1.5}^{+1.7}$ | $47.9_{-3.6}^{+3.7}$ | –
Grad ($L_{1}$) | $18.4_{-1.8}^{+2.0}$ | $47.4_{-4.3}^{+4.3}$ | –
x $\odot$ grad (sign) | $4.6_{-1.6}^{+1.8}$ | $11.9_{-4.2}^{+4.6}$ | –
x $\odot$ grad (abs) | $11.6_{-2.1}^{+2.1}$ | $29.8_{-5.5}^{+5.5}$ | –
IG (sign) | $58.2_{-4.0}^{+2.9}$ | $150.1_{-5.0}^{+4.2}$ | –
IG (abs) | $16.4_{-1.9}^{+1.2}$ | $42.3_{-5.7}^{+2.4}$ | –
LOO (sign) | $60.5_{-2.8}^{+3.3}$ | $156.0_{-5.7}^{+7.4}$ | –
LOO (abs) | $15.6_{-1.7}^{+1.6}$ | $40.2_{-3.7}^{+3.5}$ | –
Beam | – | – | –
Diabetes | Grad ($L_{2}$) | $20.1_{-1.1}^{+1.8}$ | $90.7_{-0.4}^{+0.5}$ | –
Grad ($L_{1}$) | $20.1_{-1.1}^{+1.9}$ | $90.7_{-0.4}^{+0.6}$ | –
x $\odot$ grad (sign) | $7.4_{-4.0}^{+3.1}$ | $34.2_{-19.0}^{+16.6}$ | –
x $\odot$ grad (abs) | $18.6_{-0.9}^{+1.5}$ | $84.2_{-0.8}^{+0.6}$ | –
IG (sign) | $25.3_{-2.3}^{+4.0}$ | $113.5_{-5.1}^{+7.9}$ | –
IG (abs) | $19.6_{-1.0}^{+1.4}$ | $88.7_{-1.2}^{+0.5}$ | –
LOO (sign) | $34.8_{-4.9}^{+6.7}$ | $156.0_{-18.3}^{+12.2}$ | –
LOO (abs) | $19.2_{-1.0}^{+1.4}$ | $86.6_{-0.9}^{+0.6}$ | –
Beam | – | – | –

|  | Faithfulness [%]
---|---|---
Dataset | IM | Our | R-ROAR
|  | ACU | RACU | RACU
MRPC | Grad ($L_{2}$) | $6.6_{-1.1}^{+2.2}$ | $22.9_{-4.3}^{+3.1}$ | –
Grad ($L_{1}$) | $6.6_{-1.1}^{+1.9}$ | $22.7_{-3.7}^{+1.5}$ | –
x $\odot$ grad (sign) | $-0.9_{-1.1}^{+1.0}$ | $-3.4_{-2.2}^{+3.2}$ | –
x $\odot$ grad (abs) | $4.4_{-1.0}^{+1.2}$ | $15.7_{-4.8}^{+2.6}$ | –
IG (sign) | $15.7_{-2.0}^{+1.6}$ | $55.9_{-8.8}^{+9.5}$ | –
IG (abs) | $8.1_{-1.1}^{+1.3}$ | $28.8_{-3.7}^{+9.2}$ | –
LOO (sign) | $29.9_{-1.3}^{+0.8}$ | $110.0_{-24.2}^{+30.9}$ | –
LOO (abs) | $10.2_{-1.6}^{+1.6}$ | $36.1_{-5.2}^{+8.8}$ | –
Beam | $40.1_{-2.3}^{+1.4}$ | $146.9_{-30.6}^{+41.5}$ | –
RTE | Grad ($L_{2}$) | $7.1_{-1.4}^{+0.8}$ | $38.8_{-5.9}^{+5.9}$ | –
Grad ($L_{1}$) | $7.4_{-1.2}^{+0.9}$ | $40.4_{-7.6}^{+5.1}$ | –
x $\odot$ grad (sign) | $-0.1_{-1.3}^{+1.0}$ | $-1.7_{-7.2}^{+4.8}$ | –
x $\odot$ grad (abs) | $5.0_{-1.0}^{+1.3}$ | $27.1_{-3.0}^{+6.8}$ | –
IG (sign) | $22.6_{-2.0}^{+2.3}$ | $127.1_{-31.6}^{+28.9}$ | –
IG (abs) | $7.7_{-1.3}^{+1.2}$ | $43.0_{-10.7}^{+8.8}$ | –
LOO (sign) | $38.5_{-3.6}^{+1.9}$ | $213.6_{-40.5}^{+37.1}$ | –
LOO (abs) | $9.9_{-0.4}^{+0.4}$ | $55.4_{-9.9}^{+11.9}$ | –
Beam | $50.3_{-2.9}^{+1.2}$ | $280.1_{-45.7}^{+57.9}$ | –
SST2 | Grad ($L_{2}$) | $9.8_{-1.0}^{+1.1}$ | $32.4_{-3.0}^{+3.5}$ | –
Grad ($L_{1}$) | $9.7_{-0.9}^{+1.1}$ | $32.0_{-2.9}^{+3.2}$ | –
x $\odot$ grad (sign) | $-3.4_{-0.6}^{+0.7}$ | $-11.4_{-2.2}^{+2.4}$ | –
x $\odot$ grad (abs) | $5.4_{-1.1}^{+1.6}$ | $18.0_{-3.5}^{+5.2}$ | –
IG (sign) | $40.1_{-1.9}^{+3.2}$ | $133.1_{-6.5}^{+10.6}$ | –
IG (abs) | $15.6_{-0.7}^{+0.9}$ | $51.6_{-1.5}^{+3.4}$ | –
LOO (sign) | $49.4_{-1.4}^{+1.3}$ | $164.0_{-2.9}^{+1.7}$ | –
LOO (abs) | $17.2_{-0.7}^{+1.0}$ | $57.1_{-2.2}^{+3.6}$ | –
Beam | $55.6_{-0.5}^{+1.0}$ | $184.5_{-2.6}^{+2.1}$ | –
SNLI | Grad ($L_{2}$) | $8.2_{-0.5}^{+0.3}$ | $53.8_{-2.0}^{+1.3}$ | –
Grad ($L_{1}$) | $8.2_{-0.5}^{+0.4}$ | $53.3_{-2.1}^{+1.6}$ | –
x $\odot$ grad (sign) | $-0.3_{-0.3}^{+0.3}$ | $-2.2_{-2.2}^{+1.7}$ | –
x $\odot$ grad (abs) | $5.6_{-0.4}^{+0.3}$ | $36.5_{-2.0}^{+1.7}$ | –
IG (sign) | $14.0_{-0.3}^{+0.3}$ | $91.2_{-1.0}^{+1.0}$ | –
IG (abs) | $8.0_{-0.5}^{+0.4}$ | $52.6_{-1.7}^{+1.4}$ | –
LOO (sign) | $26.3_{-0.5}^{+0.4}$ | $172.3_{-7.3}^{+4.3}$ | –
LOO (abs) | $11.0_{-0.3}^{+0.5}$ | $71.8_{-0.9}^{+1.7}$ | –
Beam | $29.6_{-0.3}^{+0.3}$ | $193.7_{-4.5}^{+5.2}$ | –
IMDB | Grad ($L_{2}$) | $13.9_{-1.9}^{+3.5}$ | $29.4_{-2.4}^{+6.0}$ | –
Grad ($L_{1}$) | $13.7_{-1.9}^{+4.1}$ | $28.9_{-2.5}^{+6.0}$ | –
x $\odot$ grad (sign) | $-2.9_{-0.4}^{+0.4}$ | $-6.4_{-1.2}^{+1.3}$ | –
x $\odot$ grad (abs) | $7.7_{-1.2}^{+2.9}$ | $16.3_{-2.8}^{+3.9}$ | –
IG (sign) | $53.2_{-4.1}^{+3.4}$ | $114.2_{-11.5}^{+12.8}$ | –
IG (abs) | $18.9_{-4.2}^{+3.7}$ | $40.3_{-9.8}^{+6.2}$ | –
LOO (sign) | $60.5_{-1.0}^{+1.1}$ | $130.1_{-13.0}^{+13.0}$ | –
LOO (abs) | $16.7_{-1.9}^{+5.0}$ | $35.5_{-3.8}^{+5.6}$ | –
Beam | – | – | –
MNLI | Grad ($L_{2}$) | $7.9_{-0.2}^{+0.1}$ | $38.7_{-1.3}^{+0.9}$ | –
Grad ($L_{1}$) | $7.8_{-0.2}^{+0.2}$ | $38.3_{-1.5}^{+1.0}$ | –
x $\odot$ grad (sign) | $-0.5_{-0.2}^{+0.5}$ | $-2.3_{-1.1}^{+2.6}$ | –
x $\odot$ grad (abs) | $5.2_{-0.1}^{+0.1}$ | $25.4_{-1.1}^{+0.8}$ | –
IG (sign) | $18.6_{-1.1}^{+0.8}$ | $91.1_{-4.6}^{+4.3}$ | –
IG (abs) | $9.0_{-0.2}^{+0.3}$ | $44.2_{-0.7}^{+1.5}$ | –
LOO (sign) | $33.0_{-1.0}^{+0.9}$ | $161.9_{-8.4}^{+6.1}$ | –
LOO (abs) | $12.4_{-0.2}^{+0.1}$ | $60.6_{-0.9}^{+1.4}$ | –
Beam | $41.2_{-1.2}^{+0.9}$ | $201.7_{-9.1}^{+6.5}$ | –
QNLI | Grad ($L_{2}$) | $9.5_{-0.4}^{+0.5}$ | $28.3_{-0.9}^{+1.4}$ | –
Grad ($L_{1}$) | $9.4_{-0.3}^{+0.3}$ | $28.0_{-0.5}^{+1.3}$ | –
x $\odot$ grad (sign) | $-1.4_{-0.5}^{+0.4}$ | $-4.1_{-1.3}^{+1.4}$ | –
x $\odot$ grad (abs) | $6.5_{-0.2}^{+0.3}$ | $19.1_{-0.6}^{+0.7}$ | –
IG (sign) | $25.0_{-5.1}^{+3.3}$ | $74.0_{-13.7}^{+9.6}$ | –
IG (abs) | $9.7_{-1.8}^{+1.2}$ | $28.7_{-5.0}^{+3.2}$ | –
LOO (sign) | $40.4_{-2.1}^{+1.5}$ | $119.8_{-5.0}^{+4.0}$ | –
LOO (abs) | $12.5_{-0.3}^{+0.2}$ | $37.0_{-0.4}^{+0.3}$ | –
Beam | $53.8_{-2.1}^{+1.7}$ | $159.5_{-4.4}^{+4.6}$ | –
QQP | Grad ($L_{2}$) | $4.0_{-0.2}^{+0.3}$ | $33.5_{-1.9}^{+1.4}$ | –
Grad ($L_{1}$) | $4.0_{-0.2}^{+0.3}$ | $33.0_{-1.9}^{+1.3}$ | –
x $\odot$ grad (sign) | $-0.4_{-0.2}^{+0.3}$ | $-3.3_{-2.0}^{+2.3}$ | –
x $\odot$ grad (abs) | $2.5_{-0.2}^{+0.3}$ | $20.7_{-2.2}^{+1.3}$ | –
IG (sign) | $8.9_{-0.6}^{+1.0}$ | $73.7_{-3.6}^{+4.8}$ | –
IG (abs) | $3.8_{-0.3}^{+0.7}$ | $31.6_{-1.6}^{+2.5}$ | –
LOO (sign) | $20.4_{-0.4}^{+0.7}$ | $169.8_{-11.1}^{+8.6}$ | –
LOO (abs) | $5.7_{-0.3}^{+0.2}$ | $47.3_{-2.8}^{+2.3}$ | –
Beam | $22.5_{-0.8}^{+0.7}$ | $187.0_{-10.3}^{+10.4}$ | –

Table 12: Faithfulness scores for RoBERTa-large. Shows Relative Area Between
Curves (RACU) and the non-relative variant (ACU), defined by Madsen et al.
(2022). Note that Madsen et al. (2022) does not report results for Recursive-
ROAR with RoBERTa-large.