# Human Interpretation and Exploitation of Self-attention Patterns in
Transformers: A Case Study in Extractive Summarization

Raymond Li1, Wen Xiao11footnotemark: 11, Lanjun Wang2, Giuseppe Carenini1
Equal Contribution. Corresponding Author. This work has been done when Lanjun
was with Huawei Canada Technologies Co. Ltd.

###### Abstract

The transformer multi-head self-attention mechanism has been thoroughly
investigated recently. On one hand, researchers are interested in
understanding why and how transformers work. On the other hand, they propose
new attention augmentation methods to make transformers more accurate,
efficient and interpretable. In this paper, we synergize these two lines of
research in a human-in-the-loop pipeline to first find important task-specific
attention patterns. Then those patterns are applied, not only to the original
model, but also to smaller models, as a human-guided knowledge distillation
process. The benefits of our pipeline are demonstrated in a case study with
the extractive summarization task. After finding three meaningful attention
patterns in the popular BERTSum model, experiments indicate that when we
inject such patterns, both the original and the smaller model show
improvements in performance and arguably interpretability.

##  1 Introduction

With transformer-based models (Vaswani et al. 2017) dominating the leaderboard
for many key NLP tasks like summarization (Liu and Lapata 2019) and sentiment
analysis (Adhikari et al. 2019), their core multi-head self-attention
mechanism has been thoroughly investigated recently. In particular, to explain
why and how transformers work, researchers analyze the learnt self-attention
matrices of language models or task-specific models (e.g., Raganato and
Tiedemann (2018); Vig and Belinkov (2019)), with Voita et al. (2019) for
instance, exploring the patterns of attention heads in neural machine
translation during pruning. Instead of visualization on trained or fine-tuned
models, in a recent work, Li et al. (2021) introduce a visual analytic
framework, T3-Vis, to help researchers to better train and fine-tune
transformer models, by providing valuable insights about the model’s intrinsic
properties and behaviours.

Meanwhile, a parallel line of research has shown that injecting
linguistic/positional information into the attention matrices is useful for
reducing the size of the model while keeping competitive results, and even
improve the performance in some cases. This can be done by either enforcing
fixed attention patterns, like Raganato, Scherrer, and Tiedemann (2020) do
with positional attention in a machine translator, and Xiao, Huber, and
Carenini (2020) with fixed discourse tree attention in a summarizer; or
alternatively by guiding the attention weights through more flexible masking
strategies, like Yang et al. (2018); Fan et al. (2021) do by using masks to
enforce locality within a fixed-window, and others have done to inject
discourse information (Mihaylov and Frank 2019), syntactical dependencies (Bai
et al. 2021), and world knowledge (Liu et al. 2020). Interestingly, all the
above strategies can also be applied to the original model through injecting
the patterns into newly added attention heads using techniques such as
Projected Attention Layers (PAL) (Stickland and Murray 2019), in which case
while the size of the model is not reduced the injected information will be
more flexibly and effectively integrated into the original model.

In this paper, we propose and test a novel human-in-the-loop pipeline, that to
the best of our knowledge is the first attempt trying to synergize research on
analyzing self-attention, along with work on injecting information into
attention matrices. Initially, an NLP expert analyzes the attention heads of a
transformer model using an interactive visual
interface111https://github.com/raymondzmc/T3-Vis(Li et al. 2021) to identify
potentially meaningful and useful patterns/relations. Then, those patterns are
evaluated on the validation set to confirm their global relevance. Finally,
the patterns found to be useful are applied back to the original model, making
the resulting model more accurate and/or more interpretable, because those
patterns become inherent and transparent properties of such model. Besides,
those patterns can also be applied to models with smaller size, in what can be
regarded as a human-guided, interpretable knowledge distillation process.
Typically, in knowledge distillation a small or focused model is obtained from
a huge or general purpose model by automatically distilling the most useful
knowledge, with the resulting model being still however a black box, because
it is often unclear what information has been distilled and what has not (Gou
et al. 2021). In contrast, in our pipeline, a human can find attention
patterns that are useful for the task-specific model to make predictions, and
then apply those to the smaller model, improving its performance, with less
parameters as well as better interpretability.

In order to test the feasibility and potential benefits of our approach, we
run a case study on the extractive summarization task, in particular using the
popular BERTSum model (Liu and Lapata 2019), and we find that: (i) For some of
the important heads, the patterns they learn do have practical meaning, either
lexical, local or positional. For instance, matching token (i.e. the trend on
attending to other tokens with the same id) is an important clue for the
summarization model. (ii) By applying the patterns back into the original
model through PAL, the resulting model can achieve better task performance and
stronger interpretability. (iii) Remarkably, the human-guided knowledge
distilled model performs much better than the vanilla transformer (baseline),
and can be competitive even with fine-tuned BERT distilled models, which by
distilling all aspects of the model have an a priori substantial advantage
over our technique that only distils attention patterns.

##  2 Related Work

###  2.1 Attention Analysis in Transformers

Head attention matrices have been thoroughly investigated, often with the aid
of visualization tools (Vig 2019; Hoover, Strobelt, and Gehrmann 2020),
because they reveal how transformers work when applied to specific tasks, with
most studies focusing on language models. For instance, Vig and Belinkov
(2019) visually explore attention patterns in BERT and GPT-2, analyzing their
alignment with syntax. Clark et al. (2019) instead evaluate the attention
heads by finding the most attended to token with respect to a given one, and
recognize that while certain attention heads do specialize to specific
dependency relations, no individual attention head captures the entire
dependency structure. Furthermore, Kovaleva et al. (2019) categorize BERT’s
attention heads based on the patterns (e.g. diagonal, heterogeneous), and find
that some heads correspond to core frame-semantic relations. More recently,
Zhao and Bethard (2020) look at how BERT’s attention changes after fine-tuning
by focusing on tokens within the negation scope, while Guan et al. (2020)
categorize attention heads through unsupervised clustering based on aggregated
attention patterns.

Other tasks, besides language modelling, have been studied. For example,
Raganato and Tiedemann (2018) investigate the linguistic information learned
by the transformer encoder when trained for machine translation (MT), and
similarly, Voita et al. (2019) characterize the functions of the attention
heads in the MT model (positional, syntactic, rare words), and evaluate the
importance of those head functions. Finally, the work by Xiao, Huber, and
Carenini (2021) shows that transformer-based summarizers learn discourse
information implicitly in the attention heads. In addition, Li et al. (2021)
propose a visual interface that shows the attention maps as well as the
importance score of each head in an interactive way, allowing deeper and
specialized exploration on all the attention heads. In this paper, we aim to
find task-specific important attention patterns, but differently from previous
work which just identify and categorize attention patterns, we propose a
pipeline to leverage these patterns in improving models’ performance and
interpretability by using the new visual interface, T3-Vis(Li et al. 2021).

###  2.2 Attention Augmentation

There are two widely used ways to augment attention: fixed and masking. For
fixed attention patterns, Raganato, Scherrer, and Tiedemann (2020) use fixed
positional patterns in MT models and demonstrate benefits for low-resource
scenarios, while Tay et al. (2020) show that even a random attention matrix is
competitive with the scaled dot-product attention in standard transformers.
Later on, Xiao, Huber, and Carenini (2020) expand on their work by using
embedded RST-style discourse trees as fixed attention matrices and show their
effectiveness in extractive summarization. For masking attention patterns,
Yang et al. (2018) model localness through casting a learnable Gaussian mask;
Fan et al. (2021) apply either static or dynamic attention masks to better
model local semantic features; Ainslie et al. (2020) and Beltagy, Peters, and
Cohan (2020) use attention masks to create sparse attention matrices that
improve the scalability on long documents. By comparison, while in all these
previous works the applied attention patterns (either fixed or masking) are
pre-defined based on prior intuitions, in this paper we propose a strategy to
find and assess important attention patterns interactively. More recently, Yin
et al. (2021) explore the alignment between human attention and model’s learnt
attentions in the machine translation domain, and further propose a guided
attention strategy to make them more aligned. However, more task specific and
fine-grained annotations are required to have the human attentions. Instead,
in our pipeline, we explore the effective interpretable patterns directly from
the model, and guide the models with those patterns, which not only better
guarantees the effectiveness of the attention patterns, but also allowing
researchers to understand how the model works.

###  2.3 Knowledge Distillation in NLP

Knowledge distillation (KD) (Hinton, Vinyals, and Dean 2015) aims to compress
a large teacher model into a smaller student model. This is achieved by
training the student network to mimic the behaviors of the teacher model in
order to obtain a competitive performance. Specifically, the type of knowledge
distilled in KD can be categorized into three types (Gou et al. 2021):
response-based (e.g., DistilBert (Sanh et al. 2019)), feature-based
(e.g.,TinyBERT (Jiao et al. 2020), MobileBERT (Sun et al. 2020), MiniLM (Wang
et al. 2020)), and relation-based (e.g. Multi-head Graph Distillation
(Seunghyun Lee 2019)). Our approach can be considered as of type relation-
based because we exploit the relations between tokens in the attention
weights. Importantly, the key difference from existing KD works, is that in
our approach knowledge is explicitly extracted from the teacher model by a
human, and such knowledge becomes a transparent property of the student model,
arguably increasing its interpretability.

###  2.4 Extractive Summarization

Extractive summarizarion is the task to pick the most representative sentences
as the summary for the given document(s). Current state-of-the-art models,
which are mostly based on large-scale pretrained language models (Liu and
Lapata 2019; Zhong et al. 2020), can deliver good performances, but why and
how such models work so well still remains an open question. In this paper, we
run a case study for our proposed pipeline on the extractive summarization
task, exploring the discovery, assessment and application of useful attention
patterns in the context of the transformer-based BERTSum model (Liu and Lapata
2019).

##  3 Proposed Generic Pipeline

Figure 1: The generic pipeline, which contains two main parts - pattern
extraction (middle) and pattern application (right), given a trained task-
specific model (left)333The figures of (a) and (b) in the ‘Extract Patterns’
step are captured from the visual interface (Li et al. 2021)..

In this section, we will briefly describe the proposed pipeline (Figure. 3).
Specifically, given a trained model (the left part in Figure.3), NLP experts
are supposed to first extract important patterns on the visual interface(Li et
al. 2021) (middle part in Figure.3) by following these three steps.
Step 1: Estimate the importance scores for all the heads on the validation
set, and find important heads that stand out
Step 2: Recognize special patterns in the important attention heads
Step 3: Evaluate and validate the pattern to confirm global relevance

Once the important patterns are identified, there are two common approaches -
fixed and masking - to apply them as constraints to the attention matrices in
the transformer-based neural models (see §3.2). The pipeline also recommends
two scenarios to apply the patterns: the first one is to enhance the original
model, while the second one is to train a new model in which the patterns are
enforced. Additional potential ways to make use of the patterns are left as
future work.

###  3.1 Extract Patterns from Attention

#### Step 1: Estimate Attention Head Importance

Although the multi-head self attention mechanism in transformers allows the
model to learn multiple types of relationships between input representations
across a single hidden layer, the importance of the individual attention heads
can vary depending on the downstream tasks. Motivated by Molchanov et al.
(2019); Michel, Levy, and Neubig (2019), we assume the importance of each head
to be independent to avoid an NP-hard combinatorial search. In principle, the
importance of an attention head $I(h)$ can be defined as the loss increment on
the validation set when removing it during inference.

$I(h)=\sum_{(x,y)\in X}\bigg{(}L(y|x,H-{h})-L(y|x,H)\bigg{)}$ (1)

where $L(y|x,H)$ is the loss on the example $x$ with label $y$ with all the
heads $H$, while $L(y|x,H-{h})$ is the loss with the heads except $h$.
However, this leave-one-out strategy for computing importance scores,
requiring $|H|$ times inferences (on the validation set), is extremely time
consuming and not scalable to large datasets. Thus, following previous works
(Michel, Levy, and Neubig 2019; Molchanov et al. 2019), we explore approximate
importance score estimation methods, which only require a single forward and
backward pass for each example. As different methods are suitable for
different tasks, we have considered the following three popular estimation
methods.

Head Sensitivity: Proposed by Michel, Levy, and Neubig (2019), a mask variable
is applied to the output of each individual attention head, where its gradient
sensitivity is used as a proxy score for the importance.

Layer-wise Relevance Propagation (LPR): Designed to compute the contributions
of individual pixels for image classification (Bach et al. 2015). LRP is
adapted by Voita et al. (2019) for the transformer model to estimate the
attention head relevance to the prediction.

Taylor Estimation: Molchanov et al. (2019) propose using the Taylor expansion
to estimate the error induced from removing a parameter from the model. In our
work, we use the first-order expansion to avoid the overhead from computing
the Hessian, where the gradient with respect to validation loss is summed over
all parameters of an attention head to estimate its importance.

In the pipeline, NLP experts can select the most proper estimation method for
their own task either based on prior knowledge (e.g. Head Sensitivity should
be selected in MT following Michel, Levy, and Neubig (2019)), or empirically,
by verifying which of the three approximate estimation methods best aligns
(e.g. by cosine similarity) with ‘gold head importance scores’ computed by
leave-one-out (Eq. 1) on the validation set.

#### Step 2: Find Attention Patterns

In this step, the human expert should start with the most important heads and
visually inspect their attention distributions looking for patterns

Here, we define a pattern very broadly as a predicate $P$ that can be verified
on any pair of input tokens $(x_{i},x_{j})$. For instance, the positional
pattern ‘preceding token’ would be true if $x_{i}$ appears before $x_{j}$.
Candidate patterns should satisfy two criteria: 1) occur consistently among
relevant tokens; 2) be interpretable by human experts to be beneficial for the
downstream tasks. For example, in previous works on analyzing the attention
heads of pretrained language models (Vig and Belinkov 2019; Kovaleva et al.
2019), some attention heads show a high correlation with position or
linguistic properties (like syntactic dependency).

#### Step 3: Evaluate Attention Patterns

When a specific interesting pattern is uncovered from visualizing the
attention heads, the next step is to confirm its global relevance on each head
by empirically measuring the proportion of total attentions from the head
aligned with the pattern aggregated over data samples. By the evaluation on
each single head over the whole validation set, the NLP expert can verify if
the pattern generally exists across different data samples, instead of only
appearing by chance on certain data that the expert happened to look at.

Specifically, we define the global relevance (GR) of an attention pattern $P$
for the attention head $h$ as follows:

$\displaystyle\textrm{GR}(P,h)$ $\displaystyle=$ $\displaystyle\sum_{x\in
X}\frac{gr(x,P,h)}{|X|}$ (2) $\displaystyle\textrm{gr}(x,P,h)$
$\displaystyle=$
$\displaystyle\frac{\sum_{i}^{|x|}\sum_{j}^{|x|}\alpha_{i,j}^{h}\cdot\mathbbm{1}_{P(x_{i},x_{j})}}{|x|}$
(3)

where $\textrm{gr}(x,P,h)$ denotes the global relevance of a pattern $P$ for
head $h$ on a single data sample $x$, and to validate the generality,
$\textrm{GR}(P,h)$ is then computed as the average $\textrm{gr}(x,P,h)$ over
the validation set. The attention value from token $x_{i}$ to $x_{j}$ on the
head $h$, denoted $\alpha_{i,j}^{h}$, is aggregated if and only if
$P(x_{i},x_{j})$ holds. Note that
$\sum_{i}^{|x|}\sum_{j}^{|x|}\alpha_{i,j}^{h}=|x|$ due to the property of
attention matrices.

Given a pattern $P$, it will be kept if there exists at least one
significantly relevant head. There are several ways to decide whether the
relevant head exists based on GR, e.g., setting a threshold. Here, we suggest
to use one-tailed one-sample t-test on each head $h^{*}$ with the null
hypothesis as: $GR(P,h^{*})<\bar{GR}(P,h)$, where $\bar{GR}(P,h)$ is the
average of $GR(P,h)$ over $H$. The p-value is usually set as 0.01. If there is
at least one head rejecting this null hypothesis, i.e. showing its
significantly higher relevance than most other heads, we keep the pattern $P$
for further applications.

###  3.2 Apply Patterns

Once important and interpretable patterns have been identified, they can be
injected into a transformer model by either fixing or masking the attention
weights prior to the softmax function. Although the two strategies are very
similar with respect to what they can achieve, as we will see in the case
study they can be more or less appropriate depending on the nature of the
pattern that needs to be applied. For fixed attention weights, the attention
logits in the scaled-dot-product attention is replaced with a fixed (possibly
input dependent) matrix such that

$\textrm{FixAttn}(V,X)=\sigma(F(X))V$ (4)

where $\sigma$ is the softmax operation, $V$ is the value vectors, and
$F(X)\in[0,1]$ computes a binary matrix from the input sequence $X$ based on
the specific pattern. Similarly, a pattern can also be applied by casting a
mask over the attention weights computed from the key and query vectors, such
that

$\textrm{MaskAttn}(Q,K,V,X)=\sigma(M(X)+QK^{T})V$ (5)

where $M(X)\in[0,-\infty)$ computes the desired behaviour in the same fashion
as $F(X)$, and is added to the attention logits to approximate the
multiplication of the attention distribution by a weight.

In practice, patterns can be applied in at least two scenarios (i) to enhance
the original model on which they are discovered aiming to improve its accuracy
and interpretability (ii) to improve smaller models, in what can be regarded
as a human-guided distillation process. In the first scenario, although
patterns can be directly injected into the pretrained encoder, determining to
which heads the patterns should be applied to requires extensive
hyperparameter search and risk overfitting. Instead, we opt to inject the
patterns via additional attention heads through techniques such as the
Projected Attention Layers (Stickland and Murray 2019). As for the second
scenario, the patterns are simply applied on the heads (one per head) for each
layer, and the new models are trained from scratch.

##  4 Case Study: Extractive Summarization

###  4.1 Trained Summarization Model

In our case study, we adopt the architecture of the popular BERTSum model (Liu
and Lapata 2019)444We re-implemented the model using the BERT class from
hugging face. Results are comparable with original paper with details in
Appendix. A . The model first obtains the contextualized sentence
representation from the pretrained BERT encoder, and uses a single-layer
binary classifier to score the sentences for summary selection. More
specifically, a pair of BOS, EOS tokens are inserted before and after each
sentence to indicate the segment boundary, and during prediction, the last
hidden state of the BOS token of each sentence is used as the sentence
representation.We apply our pipeline on the BERTSum model trained on the
CNN/DM dataset (Hermann et al. 2015; See, Liu, and Manning 2017), a widely
used summarization dataset and as evaluation metrics we use standard ROUGE
scores (Lin 2004).

Figure 2: Example of Extracting Patterns in the extractive summarization case
study. Important heads are found ((a), up left). Then three pattern types are
identified ((b), up right)666The figures of (a) and (b) from Figure. 6 are
captured from the visual interface (Li et al. 2021).: Matching Token, Intra-
Sentence and Positional (-1, +1), shown in Green, Olive, and Blue
respectively. Finally, each pattern is evaluated with GR on all of the heads
((c), bottom) . For better visualization, we only label one head with
significantly larger GR in (c) for each pattern.

###  4.2 Extract Patterns from Attentions

The process of extracting patterns in the extractive summarization case study
is shown in Figure. 6.

#### Find Important Heads

To identify the most appropriate head importance estimation method for
extractive summarization, we evaluate the three proxy scores (Sensitivity, LRP
and Taylor) against the leave-one-out head importance score by using the
selection process outlined in §3.1. As shown in Table 1, the Taylor Estimation
(Molchanov et al. 2019) is the most aligned with leave-one-out in term of
cosine similarity and therefore it is used in the the rest of our analysis.

Method | Sensitivity | LRP | Taylor
---|---|---|---
Cosine Similarity | $0.6$ | $0.64$ | $\mathbf{0.83}$

Table 1: Cosine similarity between the leave-one-out loss increment and the
three estimation methods.

The resulting estimated importance score heatmap of all heads is shown in
Figure.6 (a), revealing that head importance is not uniformly distributed,
i.e. a small number of heads play a dominant role for the summarization task,
which is inline with the findings in Michel, Levy, and Neubig (2019).

#### Find and Evaluate Patterns

With help from a visual interface, we analyze the attention distributions of
the most important heads by looking for human interpretable relationships
encoded in the attention weights. From this analysis, three specific types of
patterns appear to be present in the most important heads. Those patterns are
then evaluated on the validation set to assess their global relevance on each
head.

Matching Token (Green in Figure.6) We observe that the attention weights of
some important heads appear to exhibit an ”attending to matching tokens”
pattern. Specifically, the attention value $\alpha_{i,j}^{h}$ between input
tokens $x_{i}$ and $x_{j}$ on head $h$ is high whenever $x_{i}=x_{j}$. For
example, as shown in Figure.6 (b), the token ”photo” mostly attends to other
appearances of the token ”photo” in the input sequence. To evaluate whether
this pattern has a large global relevance for any head, we only consider
tokens that appear at least twice within a single documents, and compute GR
(Eq. 2), in which $P(x_{i},x_{j})$ holds if and only if $x_{i}=x_{j}$, i.e.
$\mathbbm{1}_{P(x_{i},x_{j})}=(\mathbbm{1}_{\textrm{freq}(x_{i})>1})\times(\mathbbm{1}_{x_{i}=x_{j}})$.

The evaluation results show that there are several heads for which the
matching token pattern has high global relevance with (See the Green box in
Figure.6 (c)). Interestingly, these heads are prominent in the importance
heatmap, which suggests the matching token pattern is critical for the
summarization task.

Intra-Sentence (Olive in Figure.6) For some heads, the attentions among tokens
tend to be localized within the sentence boundaries, as shown on an example in
Figure.6 (b).

To evaluate this pattern, GR is computed with $P(x_{i},x_{j})$ holding if and
only if $x_{i}$ and $x_{j}$ occurs within the same sentence boundary. This
reveals that such pattern appears frequently, especially in the mid to upper
layers of the transformer encoder. (See Figure.6 (c))

Positional (Blue in Figure.6) Similar to findings in Kovaleva et al. (2019),
we observe ‘positional heads’, which focus specifically on either the
preceding or following tokens, i.e., both $\alpha_{i,i-1}^{h}$ and
$\alpha_{i,i+1}^{h}$ have high values. To evaluate this pattern, GR is
computed with $P(x_{i},x_{j})$ holding iff $j=i-1$ for preceding positional
heads and $j=i+1$ for succeeding positional heads. The pattern is verified to
exist in the lower layers of the encoder, as shown in the blue boxes of
Figure.6(c).

###  4.3 Apply Patterns to Summarizers

After uncovering potentially important patterns and confirming their
relevance, we apply them to transformer-based summarizers through masking and
fixing the attention weights. Aiming for more general ‘cross-dataset’
insights, in addition to the CNN/DM dataset, on which we trained the model
that we use to extract the patterns, we also evaluate the benefit of the
patterns when applied to the NYT-50 dataset (Sandhaus 2008).777Results shown
in Sec. 4 are without the Trigram Blocking trick, and more results with it are
in Appendix.C

#### Method

The patterns identified from our analysis can be applied on an attention head
through masking or fixing its corresponding attention weight matrix.
Specifically, for the matching token pattern, we apply an attention mask which
enforces that when a token appears more than once in the document, it should
attend only to other occurrences of itself:

$M^{(m)}_{i,j}=\begin{cases}1&(x_{i}=x_{j})\lor(\textrm{freq}(x_{i})=1)\\\
0&\textrm{otherwise}\end{cases}$ (6)

where the constraint is removed for tokens occurring only once in the
document.

Similarly, for intra-sentence attention, the attention mask specifies that
only tokens within the sentence boundary can attend to each others, where:

$M^{(s)}_{i,j}=\begin{cases}1&\textrm{SameSent}(x_{i},x_{j})\\\
0&\textrm{otherwise}\end{cases}$ (7)

Lastly, we use a fixed attention matrix to encode the two positional patterns
with:

$F^{(-1)}_{i,j}=\begin{cases}1&j=i-1\\\ 0&\textrm{otherwise}\end{cases}$ (8)

And $F^{(+1)}_{i,j}$ being the same, but equal to 1 for $j=i+1$. We opt for
fixed attention matrices for these patterns to save computational overhead
since it has the same effect as applying the mask (each row is a one-hot
vector). This is similar to the method proposed by Raganato, Scherrer, and
Tiedemann (2020), but we only fix for the preceding and following token
patterns.

Model | CNN/DM(in-dataset) | NYT-50(cross-dataset)
---|---|---
R-1 | R-2 | R-L | R-1 | R-2 | R-L
BERTSum | 42.33 | 19.88 | 38.86 | 48.37 | 29.25 | 40.72
\+ PAL | 42.34 | 19.88 | 38.86 | 48.56 | 29.41 | 40.91
\+ PAL (Ours) | 42.58 | 20.05 | 39.10 | 48.74 | 29.60 | 41.11

Table 2: ROUGE F-scores of PAL with pretrained models on corresponding test
sets.

#### Guided Knowledge Injection into Pre-trained Models

We first experiment with injecting the patterns back into the pre-trained
BERTSum summarizer. In particular, we apply them through additional attention
heads in the form of a Projected Attention Layer (PAL)(Stickland and Murray
2019), along with the existing parameters of the original model.

The hidden size of our PALs is $256$, which consists of $4$ additional
attention heads ($d_{k}=d_{v}=d_{q}=64$). PAL is added in each of the 12 BERT
layers, where our patterns are applied in the 4 PAL attention heads. To ensure
the changes in performance are due to the patterns rather than the additional
parameters, we also compare against adding PAL without applying the patterns.
Results in Table 2 indicate applying the patterns in the PAL (+PAL(Ours))
improves BERTSum’s performance on both datasets, where the performance gains
on the NYT-50 are similar (or even slightly better) than on the in-domain
CNN/DM dataset, supporting the generality of the discovered patters. This
suggests that following our pipeline can boost model performance, as well as
its interpretability, as the model follows meaningful patterns. Interestingly,
visualizing the head importance scores reveals that the PAL heads with
patterns applied are significantly more important (by 2 orders of magnitude)
than the PAL heads without patterns applied 888An illustrative example is
shown in Appendix.B.1 .

#### Knowledge Distillation for Vanilla Transformers

In a second round of experiments, we apply the three kinds of patterns on a
simpler and smaller summarizer, namely a non-pretrained 6-layer 8-head
transformer architecture (Vaswani et al. 2017) and compare against the vanilla
summarizer baseline. Since this can be seen as a form of human-guided
distillation, we also compare against two popular SOTA distillation models
from recent works: DistilBERT (Sanh et al. 2019) and TinyBERT (Jiao et al.
2020). Specifically, we compare against the 6-layer 12-head variant of these
two models. As these models use the embedding-layer from BERT (TinyBERT uses
projected embeddings but maintains the same dimensions), we also use the
pretrained embeddings for a fair comparison.

Under both settings, each of the four patterns (with 2 as positional patterns)
is applied in a separate attention head across all layers in the model. Note
that since the goal of these experiments is to assess the benefits of the
patterns, we do not perform extensive hyperparameters search when applying
these patterns (e.g. on which layer, on how many heads, etc.).

As shown in Table 3, for CNN/DM the pattern-infused models significantly
outperform the baseline vanilla summarizer under all three settings (6-8,
6-12, and 6-12 w/ BERT embeddings), and can also beat the two distilled models
with the same setting, which convincingly demonstrates the utility of applied
patterns. As for the cross-dataset experiments on the NYT-50 dataset, our
models significantly improves over the vanilla transformer and DistilBERT, but
lags behind TinyBERT in performace. Since TinyBERT also distilled the layer
output along with the attention weights and predicted distribution of BERT, it
has a substantial advantage over our technique that only distils attention
patterns. We suspect that the additional knowledge helped TinyBERT in
capturing long term dependencies required by NYT-50.

Overall, with the specific patterns applied, our models are arguably more
interpretable than both vanilla transformers and distilled models, as we
certainly know the information encoded in each masked/fixed attention heads.
Additionally, as it was the case for the enhanced pre-trained model, the
attention heads expressing the patterns tend to have higher importance scores
than the other heads 999An illustrative example is shown in Appendix. B.2 ,
suggesting that such patterns are effectively leveraged by the model.

Model | CNN/DM (in-dataset) | NYT-50 (cross-dataset)
---|---|---
R-1 | R-2 | R-L | R-1 | R-2 | R-L
6 Layer 8 Head
Transformer | 40.50 | 18.22 | 36.94 | 45.86 | 26.83 | 38.23
Ours | 41.42 | 18.94 | 37.92 | 47.11 | 27.89 | 39.34
6 Layer 12 Head
Transformer | 40.53 | 18.22 | 36.98 | 46.07 | 27.01 | 38.42
Ours | 41.58 | 19.10 | 38.10 | 46.84 | 27.68 | 39.08
6 Layer 12 Head (with BERT Embeddings)
Transformer | 40.74 | 18.40 | 37.20 | 46.07 | 26.98 | 38.37
DistillBERT | 40.36 | 18.13 | 36.83 | 45.49 | 26.52 | 37.90
TinyBERT | 41.43 | 19.09 | 37.93 | 47.76 | 28.57 | 40.04
Ours | 41.49 | 19.07 | 37.99 | 47.02 | 27.83 | 39.21

Table 3: Results on the knowledge distillation experiments with two settings -
as in the original transformer (Vaswani et al. 2017) (6 layers, 8 heads), and
as in the distilled models (Sanh et al. 2019; Jiao et al. 2020)(6 layers, 12
heads w/ or w/o BERT Embeddings)

To study the contribution of individual patterns, we perform an ablation study
by applying all combinations of patterns on CNN/DM using the transformer model
with 6 layers and 8 heads.

Model | R-1 | R-2 | R-L
---|---|---|---
Transformer | 40.50 | 18.22 | 36.94
\+ match (m) | +0.03 | +0.12 | +0.07
\+ intra (i) | +0.05 | +0.06 | +0.12
\+ pos (p) | -0.16 | -0.17 | -0.13
\+ m +i | +0.84 | +0.65 | +0.91
\+ m +p | +0.07 | +0.11 | +0.11
\+ i + p | -0.01 | +0.03 | +0.07
\+ all | +0.92 | +0.72 | +0.98

Table 4: Ablation study on the CNN/DM dataset with the basic transformer
setting.

According to Table 4, we observe that applying matching token and intra-
sentence together achieves the strongest improvement on the performance among
all combinations, only slightly lower than applying all patterns. Meanwhile,
the gains from applying them separately are only marginal. One intriguing
explanation is that these two patterns allows the model to learn sentence-
level features based on term frequency (plausibly similar to TF-IDF (Jones
1972)), where higher scores are assigned to sentences containing frequently
appearing tokens. Additionally, although applying only the positional patterns
causes the performance to degrade, it works better when combined with the two
other patterns. The reason why this happens is unclear and further study are
left as future work.

#### Discussion on Interpretability

Arguably, with human-interpretable patterns enhanced in the model for a
specific downstream task, we gain a better intuition about the model’s
behaviour and the type of features that are useful for the respective task,
i.e. extractive summarization in the case study.

##  5 Conclusion and Future Work

In this paper, we propose a generic human-in-the-loop pipeline, which
synergizes two popular research directions, where the findings from an
analysis of the multi-head self-attention mechanism in transformers can be
utilized to create more accurate and interpretable transformer models. To be
specific, a human expert analyzes the attention heads of a task-specific
model, identify and verify potentially meaningful patterns and finally inject
the patterns back into a the original model or in smaller models. By running a
case study on the extractive summarization task, we show the potential
benefits of our pipeline. We do find meaningful patterns in some important
heads, and the relationships encoded in the patterns help us understand the
features used by the model when performing summarization. Furthermore, by
applying the patterns into the original as well as smaller models, the
performance and interpretability improve in both cases.

One very promising direction for future work is to apply our generic pipeline
to other NLP tasks, like machine translation, whose models are known to
contain redundancy among the heads (Michel, Levy, and Neubig 2019; Voita et
al. 2019). It would also be worth exploring to verify whether the important
patterns from one task can be transferable to another task (e.g. extractive
summarization to abstractive summarization, or to discourse parsing, etc.), in
order to better understand the connections between different tasks. In
addition to the apparent patterns that human experts can identify through
visualization of the attention, our pipeline can be further expanded to
consider deeper linguistic features, i.e. human experts could propose
linguistic patterns that may be relevant for a task based on prior knowledge
(e.g. discourse may be important for summarization), and then evaluate those
patterns, to apply them back to the model once validated.

## References

  * Adhikari et al. (2019) Adhikari, A.; Ram, A.; Tang, R.; and Lin, J. 2019.  Docbert: Bert for document classification.  _arXiv preprint arXiv:1904.08398_.
  * Ainslie et al. (2020) Ainslie, J.; Ontanon, S.; Alberti, C.; Cvicek, V.; Fisher, Z.; Pham, P.; Ravula, A.; Sanghai, S.; Wang, Q.; and Yang, L. 2020.  ETC: Encoding Long and Structured Inputs in Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 268–284. Online: Association for Computational Linguistics.
  * Bach et al. (2015) Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.-R.; and Samek, W. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7): e0130140.
  * Bai et al. (2021) Bai, J.; Wang, Y.; Chen, Y.; Yang, Y.; Bai, J.; Yu, J.; and Tong, Y. 2021.  Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 3011–3020. Online: Association for Computational Linguistics.
  * Beltagy, Peters, and Cohan (2020) Beltagy, I.; Peters, M. E.; and Cohan, A. 2020.  Longformer: The Long-Document Transformer.  _arXiv:2004.05150_.
  * Clark et al. (2019) Clark, K.; Khandelwal, U.; Levy, O.; and Manning, C. D. 2019.  What Does BERT Look at? An Analysis of BERT’s Attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , 276–286. Florence, Italy: Association for Computational Linguistics.
  * Fan et al. (2021) Fan, Z.; Gong, Y.; Liu, D.; Wei, Z.; Wang, S.; Jiao, J.; Duan, N.; Zhang, R.; and Huang, X. 2021.  Mask Attention Networks: Rethinking and Strengthen Transformer.  _arXiv preprint arXiv:2103.13597_.
  * Gou et al. (2021) Gou, J.; Yu, B.; Maybank, S. J.; and Tao, D. 2021.  Knowledge distillation: A survey.  _International Journal of Computer Vision_ , 1–31.
  * Guan et al. (2020) Guan, Y.; Leng, J.; Li, C.; Chen, Q.; and Guo, M. 2020.  How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT’s Attention.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 3853–3860. Barcelona, Spain (Online): International Committee on Computational Linguistics.
  * Hermann et al. (2015) Hermann, K. M.; Kocisky, T.; Grefenstette, E.; Espeholt, L.; Kay, W.; Suleyman, M.; and Blunsom, P. 2015.  Teaching Machines to Read and Comprehend.  In Cortes, C.; Lawrence, N.; Lee, D.; Sugiyama, M.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Hinton, Vinyals, and Dean (2015) Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the Knowledge in a Neural Network.  In _NIPS Deep Learning and Representation Learning Workshop_.
  * Hoover, Strobelt, and Gehrmann (2020) Hoover, B.; Strobelt, H.; and Gehrmann, S. 2020.  exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 187–196. Online: Association for Computational Linguistics.
  * Jiao et al. (2020) Jiao, X.; Yin, Y.; Shang, L.; Jiang, X.; Chen, X.; Li, L.; Wang, F.; and Liu, Q. 2020.  TinyBERT: Distilling BERT for Natural Language Understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 4163–4174. Online: Association for Computational Linguistics.
  * Jones (1972) Jones, K. S. 1972.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of documentation_.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In _ICLR (Poster)_.
  * Kovaleva et al. (2019) Kovaleva, O.; Romanov, A.; Rogers, A.; and Rumshisky, A. 2019.  Revealing the Dark Secrets of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 4365–4374. Hong Kong, China: Association for Computational Linguistics.
  * Li et al. (2021) Li, R.; Xiao, W.; Wang, L.; Jang, H.; and Carenini, G. 2021.  T3-Vis: visual analytic for Training and fine-Tuning Transformers in NLP.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 220–230. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Lin (2004) Lin, C.-Y. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , 74–81. Barcelona, Spain: Association for Computational Linguistics.
  * Liu et al. (2020) Liu, W.; Zhou, P.; Zhao, Z.; Wang, Z.; Ju, Q.; Deng, H.; and Wang, P. 2020.  K-bert: Enabling language representation with knowledge graph.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 2901–2908.
  * Liu and Lapata (2019) Liu, Y.; and Lapata, M. 2019.  Text Summarization with Pretrained Encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 3730–3740. Hong Kong, China: Association for Computational Linguistics.
  * Michel, Levy, and Neubig (2019) Michel, P.; Levy, O.; and Neubig, G. 2019.  Are Sixteen Heads Really Better than One?  In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Mihaylov and Frank (2019) Mihaylov, T.; and Frank, A. 2019.  Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2541–2552. Hong Kong, China: Association for Computational Linguistics.
  * Molchanov et al. (2019) Molchanov, P.; Mallya, A.; Tyree, S.; Frosio, I.; and Kautz, J. 2019.  Importance estimation for neural network pruning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11264–11272.
  * Nallapati, Zhai, and Zhou (2017) Nallapati, R.; Zhai, F.; and Zhou, B. 2017.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Paulus, Xiong, and Socher (2018) Paulus, R.; Xiong, C.; and Socher, R. 2018.  A Deep Reinforced Model for Abstractive Summarization.  In _International Conference on Learning Representations_.
  * Raganato, Scherrer, and Tiedemann (2020) Raganato, A.; Scherrer, Y.; and Tiedemann, J. 2020.  Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 556–568. Online: Association for Computational Linguistics.
  * Raganato and Tiedemann (2018) Raganato, A.; and Tiedemann, J. 2018.  An Analysis of Encoder Representations in Transformer-Based Machine Translation.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , 287–297. Brussels, Belgium: Association for Computational Linguistics.
  * Sandhaus (2008) Sandhaus, E. 2008.  The new york times annotated corpus.  _Linguistic Data Consortium, Philadelphia_ , 6(12): e26752.
  * Sanh et al. (2019) Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * See, Liu, and Manning (2017) See, A.; Liu, P. J.; and Manning, C. D. 2017.  Get To The Point: Summarization with Pointer-Generator Networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1073–1083. Vancouver, Canada: Association for Computational Linguistics.
  * Seunghyun Lee (2019) Seunghyun Lee, B. C. S. 2019.  Graph-based Knowledge Distillation by Multi-head Attention Network.  In _British Machine Vision Conference (BMVC)_.
  * Stickland and Murray (2019) Stickland, A. C.; and Murray, I. 2019.  BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning.  In _International Conference on Machine Learning_ , 5986–5995. PMLR.
  * Sun et al. (2020) Sun, Z.; Yu, H.; Song, X.; Liu, R.; Yang, Y.; and Zhou, D. 2020.  MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2158–2170. Online: Association for Computational Linguistics.
  * Tay et al. (2020) Tay, Y.; Bahri, D.; Metzler, D.; Juan, D.-C.; Zhao, Z.; and Zheng, C. 2020.  Synthesizer: Rethinking self-attention in transformer models.  _arXiv preprint arXiv:2005.00743_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017.  Attention is All you Need.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vig (2019) Vig, J. 2019.  A Multiscale Visualization of Attention in the Transformer Model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 37–42. Florence, Italy: Association for Computational Linguistics.
  * Vig and Belinkov (2019) Vig, J.; and Belinkov, Y. 2019.  Analyzing the Structure of Attention in a Transformer Language Model.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , 63–76. Florence, Italy: Association for Computational Linguistics.
  * Voita et al. (2019) Voita, E.; Talbot, D.; Moiseev, F.; Sennrich, R.; and Titov, I. 2019.  Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 5797–5808. Florence, Italy: Association for Computational Linguistics.
  * Wang et al. (2020) Wang, W.; Wei, F.; Dong, L.; Bao, H.; Yang, N.; and Zhou, M. 2020.  MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 5776–5788. Curran Associates, Inc.
  * Xiao, Huber, and Carenini (2020) Xiao, W.; Huber, P.; and Carenini, G. 2020.  Do We Really Need That Many Parameters In Transformer For Extractive Summarization? Discourse Can Help !  In _Proceedings of the First Workshop on Computational Approaches to Discourse_ , 124–134. Online: Association for Computational Linguistics.
  * Xiao, Huber, and Carenini (2021) Xiao, W.; Huber, P.; and Carenini, G. 2021.  Predicting Discourse Trees from Transformer-based Neural Summarizers.  _CoRR_ , abs/2104.07058.
  * Xu and Durrett (2019) Xu, J.; and Durrett, G. 2019.  Neural Extractive Text Summarization with Syntactic Compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 3292–3303. Hong Kong, China: Association for Computational Linguistics.
  * Xu et al. (2020) Xu, J.; Gan, Z.; Cheng, Y.; and Liu, J. 2020.  Discourse-Aware Neural Extractive Text Summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 5021–5031. Online: Association for Computational Linguistics.
  * Yang et al. (2018) Yang, B.; Tu, Z.; Wong, D. F.; Meng, F.; Chao, L. S.; and Zhang, T. 2018.  Modeling Localness for Self-Attention Networks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 4449–4458. Brussels, Belgium: Association for Computational Linguistics.
  * Yin et al. (2021) Yin, K.; Fernandes, P.; Pruthi, D.; Chaudhary, A.; Martins, A. F. T.; and Neubig, G. 2021.  Do Context-Aware Translation Models Pay the Right Attention?  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 788–801. Online: Association for Computational Linguistics.
  * Zhang, Wei, and Zhou (2019) Zhang, X.; Wei, F.; and Zhou, M. 2019.  HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 5059–5069. Florence, Italy: Association for Computational Linguistics.
  * Zhao and Bethard (2020) Zhao, Y.; and Bethard, S. 2020.  How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4729–4747. Online: Association for Computational Linguistics.
  * Zhong et al. (2020) Zhong, M.; Liu, P.; Chen, Y.; Wang, D.; Qiu, X.; and Huang, X. 2020.  Extractive Summarization as Text Matching.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 6197–6208. Online: Association for Computational Linguistics.

##  Appendix A Experiment Settings

The dataset CNN/DM consists of news articles and multi-sentence highlights as
summaries. In our work, we used the non-anonymized version processed by See,
Liu, and Manning (2017) while following the standard dataset split that
contains 287,226 training examples, 13,368 validation examples and 11,490 test
examples 101010https://github.com/abisee/cnn-dailymail. Following previous
work (Xu and Durrett 2019; Zhang, Wei, and Zhou 2019; Xu et al. 2020), we
create the NYT-50 dataset from the New York Times Annotated Corpus by removing
the documents whose summaries are shorter than 50 words, and use the data
split that consists of 137,778 training examples, 17,222 validation examples
and 17,223 test examples. In both datasets, we use the same data pre-
processing steps from previous work (Liu and Lapata 2019; Xu et al. 2020), and
obtain sentence-level oracle labels for extractive summarization by greedily
select sentences that maximizes the ROUGE evaluation metric (Nallapati, Zhai,
and Zhou 2017). During training and inference, the documents are truncated to
$512$ and $800$ tokens, respectively, for the CNN/DM and NYT-50 datasets.

During training, we use the ADAM optimizer (Kingma and Ba 2015)
($\beta_{1}=0.9$, $\beta_{2}=0.999$) following the same learning rate
scheduler used in (Liu and Lapata 2019). We train all our models for a total
of $50,000$ steps where the validation loss is evaluated every $1,000$ steps
for selecting the top-3 checkpoints. We perform all our experiments on a
combination of NVIDIA GTX 1080 Ti and V100 under the single GPU setting, where
the true batch size is set to $36$ with gradient accumulation per step is set
to $9$ or $3$ for 1080 Ti and V100 respectively due to memory constraints.

##  Appendix B Head Importance of Pattern-Injected Models

###  B.1 Projected Attention Layers

We visualize the important scores of the PAL heads for BERTSum trained on
CNN/DM (Fig. 3), where there are four heads added to each BERT layer via
residual connection. Figure 3(a) shows the normalized importance score of the
PAL heads without any patterns applied, where the model is opting to use
almost entirely the representation from the BERT layers. In Figure 3(b), where
each of the four PAL heads are injected with our patterns, we can see that
importance score significantly increased from the score without the patterns
applied, indicating that the features encoded in our patterns are indeed being
utilized by the models in addition to the existing pretrained representations.

(a) Regular PAL

(b) PAL with patterns

Figure 3: PAL head importance without (a) and with our patterns (b), such
that: Matching Token (Green), Intra Sentence (Olive) and Positional (Blue)
(-1, +1)

###  B.2 Distilled Model

Similarly, We also visualize the head importance score (Fig. 4) using the
6-layer 12-head model on NYT-50, where the first four heads (index 1-4) of
each layer are injected with our patterns (matching token, intra-sentence and
positional, respectively). From this example, we can see that the heads with
patterns applied are considered to be more important across almost all layers,
with the most important head being the intra-sentence head in the last layer.
This fits our intuition since the output of the last layer is used as the
sentence-representation for the classifier.

Figure 4: Head importance heatmap for 6-layer 8-head transformer model, where
the first four heads are injected with the patterns: Matching Token (Green),
Intra Sentence (Olive) and Positional (Blue) (-1, +1).

##  Appendix C Summarization with Trigram Blocking

###  C.1 Trigram Blocking

In our experiments, we follow previous work (Paulus, Xiong, and Socher 2018;
Liu and Lapata 2019) in evaluating the models in two ways: with and without
the trigram blocking trick applied. Normally at inference time, the summary is
formed by selecting sentences with the highest prediction scores, but with the
trick applied, sentences containing the same trigrams as already-selected
sentences will not be selected. Trigram blocking has been shown to be an
effective method to deal with redundancy on some dataset (e.g. CNN/DM), but
may cause performance drop in others (e.g. Pubmed and arXiv).

###  C.2 Guided Knowledge Injection into pre-trained Models

In Table.5, we show all the results on both datasets with the Trigram Block
trick applied. While the same trend is present as with pattern applied PALs,
the model with Trigram Block seems to work better on both datasets.

Model | CNN/DM | NYT-50
---|---|---
R-1 | R-2 | R-L | R-1 | R-2 | R-L
BERTSum | 42.97 | 20.09 | 39.43 | 47.58 | 28.40 | 39.95
\+ PAL | 42.96 | 20.07 | 39.41 | 47.78 | 28.56 | 40.15
\+ PAL (Ours) | 43.07 | 20.12 | 39.50 | 48.25 | 29.10 | 40.70

Table 5: ROUGE F scores of PAL with pretrained models on corresponding test
sets with Trigram Blocking Trick.

###  C.3 Knowledge Distillation for Vanilla Transformers

In Table.6, we show the experimental results of the knowledge distillation
models with trigram blocking applied, and Table.7 shows the ablation results
on the CNN/DM dataset with Trigram Blocking. In line with the observations in
Sec.4.3, our human-guided knowledge distilled models work better than all the
other models on all of the settings on the CNNDM dataset, while better than
all models except TinyBERT on the NYT dataset. As for the ablation study, the
matching token pattern brings a higher performance increment compared with the
result without Trigram Block, and overall, with all of the patterns applied
the performance is the best.

Model | CNN/DM | NYT-50
---|---|---
R-1 | R-2 | R-L | R-1 | R-2 | R-L
6 Layer 8 Head
Transformer | 41.07 | 18.41 | 37.45 | 45.13 | 26.05 | 37.52
Ours | 41.93 | 19.04 | 38.37 | 46.36 | 27.03 | 38.58
6 Layer 12 Head
Transformer | 41.12 | 18.42 | 37.50 | 45.35 | 26.23 | 37.71
Ours | 42.01 | 19.12 | 38.44 | 46.09 | 26.84 | 38.35
6 Layer 12 Head (with BERT Embeddings)
Transformer | 41.38 | 18.65 | 37.79 | 45.35 | 26.20 | 37.66
DistillBERT | 40.84 | 18.22 | 37.23 | 44.77 | 25.77 | 37.21
TinyBERT | 42.16 | 19.35 | 38.61 | 46.97 | 27.70 | 39.24
Ours | 42.24 | 19.35 | 38.68 | 46.27 | 27.02 | 38.49

Table 6: The overall results on the knowledge distillation experiments with
two settings - the setting of the original transformer(Vaswani et al. 2017) (6
layers, 8 heads), and the setting of the distilled models(Sanh et al. 2019;
Jiao et al. 2020)(6 layers, 12 heads), with Trigram Blocking applied.

Model | w Tri-block
---|---
R-1 | R-2 | R-L
Transformer | 41.07 | 18.41 | 37.45
\+ match (m) | +0.67 | +0.54 | +0.71
\+ intra (i) | +0.20 | +0.12 | +0.27
\+ pos (p) | -0.13 | -0.13 | -0.10
\+ m +i | +0.84 | +0.57 | +0.89
\+ m +p | +0.46 | +0.38 | +0.52
\+ i + p | +0.27 | +0.20 | +0.34
\+ all | +0.86 | +0.63 | +0.92

Table 7: Ablation study on the CNN/DM dataset with the basic transformer
setting with Trigram Blocking trick.


:
LEAVE UNSET 2023 LEAVE UNSET LEAVE UNSET Machine Learning for Health (ML4H)
2023

# Validity problems in clinical machine learning by indirect data labeling
using consensus definitions††thanks: This article is an extended version of
Chapter 2.4.3 of Riezler and Hagmann (2021).

Michael Hagmann<EMAIL_ADDRESS>
Shigehiko Schamoni<EMAIL_ADDRESS>
Stefan Riezler<EMAIL_ADDRESS>
Department of Computational Linguistics
& Interdisciplinary Center for Scientific Computing (IWR)
Heidelberg University  Germany

###### Abstract

We demonstrate a validity problem of machine learning in the vital application
area of disease diagnosis in medicine. It arises when target labels in
training data are determined by an indirect measurement, and the fundamental
measurements needed to determine this indirect measurement are included in the
input data representation. Machine learning models trained on this data will
learn nothing else but to exactly reconstruct the known target definition.
Such models show perfect performance on similarly constructed test data but
will fail catastrophically on real-world examples where the defining
fundamental measurements are not or only incompletely available. We present a
general procedure allowing identification of problematic datasets and black-
box machine learning models trained on them, and exemplify our detection
procedure on the task of early prediction of sepsis.

###### keywords:

validity, consensus definitions, early disease diagnosis

##  1 Introduction

A machine learning research project is naturally divided into the tasks of
data curation — careful preparation of training, development and test data —
and of machine learning itself, where the practitioner only needs to focus on
improving the current state-of-the-art on the benchmark data, often being
agnostic about the compilation process of the data. This division of labor
incurs the risk of obtaining problematic models if the data curator determines
the target label by an indirect measurement, and the fundamental measurements
used for its determination are part of the input feature representation for
machine learning. Under this setup, the obtained model will be nothing else
than a nearly identical reconstruction of the applied target definition. This
constitutes a validity problem that leads to serious practical problems when
the model is applied to real-world data. Particularly, the model fails on data
where the fundamental measurements of the target definition are not or only
incompletely available, because it hasn’t learned alternative predictive
patterns relying on other features. Furthermore, usages of machine learning to
discover new explanatory predictive patterns will fail for exactly the same
reason.

Machine learning research in health care is exposed to this risk if labels are
created by indirect measurements given by consensus definitions of medical
diseases. For example, the Sepsis-3 definition (Singer et al., 2016; Seymour
et al., 2016) defines sepsis by the presence of a verified or suspected
infection in combination with an organ function deterioration. For patients
without or unknown pre-conditions, this deterioration is in practice
operationalized by a total SOFA score of at least 2 points. The SOFA scoring
system itself (Vincent et al., 1996) is defined for 6 organ systems (e.g.,
liver or kidney) whose scores are defined by thresholds on measurable
physiological quantities like creatinine, bilirubin, urine output, mean
arterial pressure, etc. As shown in a recent overview (Moor et al., 2021) that
examined 22 studies on machine learning approaches for sepsis prognosis, with
the exception of one, all studies define ground-truth sepsis labels using
rules such as the Sepsis-3 consensus definition, and use datasets such as
MIMIC (Johnson et al., 2016) for training and testing. Similarly, a recent
challenge on early prediction of sepsis from clinical data assigned labels in
accordance with the Sepsis-3 consensus definition (Reyna et al., 2019). The
approaches described in Moor et al. (2021) and Reyna et al. (2019) include the
fundamental physiological measurements involved in the consensus definition of
sepsis as input features for machine learning, which makes them _defining
features_ in the indirect measurement of the consensus definition.

Given the negative impact of defining features, we need a procedure to
identify them in training data. Starting from a set of candidate features,
e.g, identified by the absolute magnitude of their bivariate correlation with
the labels, we present the following two-step detection procedure, based on
interpretable _Generalized Additive Models (GAM)_ :

  1. 1.

In a first step, we search for the GAM with the _best data fit_ and _smallest
complexity_ that can be built using only candidate features.

  2. 2.

In a second step, we confirm that the features constituting the model
identified in the first step are indeed _defining_ by checking if the
contribution of all other candidate features to the prediction is _nullified_
in a model that includes the defining features as inputs.

We exemplify the problem on the consensus definitions involved in SOFA score
and more complex examples like the Sepsis-3 consensus definition. The
description of the experiments will rely on the informally defined steps of
the above detection procedure. Formal details are given in the Appendix. Code
and data to reproduce this work are available at
https://github.com/StatNLP/ml4h_validity_problems.

##  2 Experiments

### Data

The input data used in our experiments consist of several clinical
measurements for $620$ intensive care patients from the surgical intensive
care unit of the University Medical Centre Mannheim, Germany (see Lindner et
al. (2022); Schamoni et al. (2022) for a detailed description). We split the
data into train ($323{,}404$ measurement points) and test ($80{,}671$
measurement points) partitions. Furthermore, we label the data according to
the Sepsis-3 consensus definition (Singer et al., 2016; Seymour et al., 2016)
that combines the criteria of SOFA and infection.

Table 1: Definition of liver SOFA score. Serum bilirubin levels are compared
to specific thresholds listed in $mg/dl$ to determine the SOFAliver score.
Table 2: Step 1 of detection of defining features for liver SOFA: Top 5 GAMs
for liver SOFA, ranked by data fit(↓) and complexity(↑).

### Detecting defining features in datasets for liver SOFA scores

In the following example, we assume that
$D=\\{(\mathbf{x}^{n},{y}^{n})\\}_{n=1}^{N}$ is a dataset of input features
$\mathbf{x}=(x_{1},x_{2},\ldots,x_{p})$ and gold standard labels $y$ of liver
SOFA scores. We apply the two-step detection procedure described above to this
data. That means, we train a GAM on this dataset, and use the statistical
metrics of data fit, model complexity, and nullification to expose fundamental
measurements in $\mathbf{x}$ as defining features. This is achieved by showing
that they allow an exact reconstruction of the target functional definition
$y$, while nullifying the contribution of all other features.

Figure 1: Step 2 of detection of defining features for liver SOFA: Feature
shapes and data fit $D^{2}$ of GAMs for liver SOFA data without (left column)
and with bilirubin (right column) as input feature, showing perfect
reconstruction of the target definition and nullfication of non-defining
features.

For ease of presentation, we choose a subset of the original $43$ features
used in Schamoni et al. (2022). In the liver SOFA example we consider the
clinical measurements of bilirubin, aspartate aminotransferase (ASAT), a quick
international normalized ratio of the protein prothrombin (quick-INR), alanin
aminotransferase (ALAT), and thrombocytes.111The features were selected based
on the magnitude of Pearson correlation with the target. As shown in Table 1,
bilirubin is a fundamental measurement for the indirect measurement of liver
SOFA.

Step 1 of the detection procedure trains a GAM on the five candidate features,
yielding the result shown in Table 2. We see that any model including
bilirubin as feature explains the data with a data fit $D^{2}=100\%$. The
model that uses bilirubin as only feature has the smallest complexity. Step 2
of the detection procedure is depicted in Figure 1. It shows the feature
shapes of GAMs fitted to subsets of the training data with (right column) and
without bilirubin (left column). The plots show that the GAM trained without
access to bilirubin uses all other available features (rows 2–5 in left
panel), whereas their contribution to the prediction is nullified in the GAM
trained on data including bilirubin (rows 2–5 in right panel). The comparison
of the data fit metric $D^{2}$ for both GAMs illustrates that the inclusion of
bilirubin in the training data boosts the data fit from $26\%$ to $100\%$.
This means that the GAM trained on data including bilirubin is able to
perfectly replicate the liver SOFA scores for all test inputs. Additionally,
the first panel of the right column demonstrates that a GAM is able to nearly
perfectly reconstruct the liver SOFA definition from training data including
the defining feature bilirubin: The steps of the plateaus in the graph
correspond exactly to the thresholds listed in Table 1. This shows that our
detection procedure is able to successfully identify bilirubin as a defining
feature for the indirect measurement of liver SOFA, out of a candidate set of
fundamental features with a similarly high bivariate correlation to the target
label.

### Detecting defining features in datasets for kidney SOFA scores

Table 3: Definition of kidney SOFA score. Serum creatinine levels and urine
output are compared to thresholds listed in $mg/dl$ and $ml/day$,
respectively, to determine their component value. The final score is the
maximum of both: SOFAkidney
$=\mathrm{max}(\text{value}_{crea},\text{value}_{urine})$. Table 4: Step 1 of
two-step detection of defining features for kidney SOFA: Top 5 GAMs for kidney
SOFA, ranked by data fit($\downarrow$) and complexity ($\uparrow$). Figure 2:
Step 2 of two-step detection of defining features for kidney SOFA: Feature
shapes and data fit $D^{2}$ of GAMs trained on kidney SOFA data without (left
col.) and with creatinine and urine-24h (right col.) as input features,
showing reconstruction of target definition and nullification of non-defining
features.

As shown in Table 3, the kidney SOFA score is defined as a maximum of two step
functions. This bivariate function cannot be expressed as sum of smooth
functions using spline-based GAMs. Additionally, we face the problem that
creatinine and urine-24h output are significantly correlated ($r=-.28$,
$p<.0001$). However, as we will see, a GAM-based analysis of the univariate
feature shapes for urine-24h and creatinine still yields some useful insights.
Let us first apply step 1 of the detection procedure for defining features to
data with automatically assigned kidney SOFA labels. Table 4 displays the
ranking of models using the 6 candidate features, yielding a data fit
$D^{2}=95\%$ for any model including creatinine and urine-24h as features,
where the model that includes only these two features has the smallest
complexity.

Figure 2 depicts the feature shapes of a model using all features in the
dataset (right column) and a model that was trained without creatinine and
urine-24h (left column). The $D^{2}$ value of the model on the right reaches
$95\%$, compared to $25\%$ for the model without defining features. The bottom
four plots in the right column show that the contribution of any feature in
the model without creatinine and urine-24h (left column) is nullified by
inclusion of urine-24h and creatinine as features. Furthermore, the top two
plots in the right column show that the target definition of kidney SOFA given
in Table 3 can be satisfactorily reconstructed, despite the fact that the
kidney SOFA is a composition of two step functions. By combining the
conditions of data fit and nullification, our detection procedure was able to
successfully identify the fundamental measurements of creatinine and urine-24h
output as defining features for the indirect measurement of kidney SOFA,
compared to other fundamental features with a similarly high bivariate
correlation to the target label.

### Detecting defining features for Sepsis-3 consensus definition

Table 5: Operationalization of Sepsis-3 according to Singer et al. (2016). A
patient has sepsis if the maximum SOFA score in the last 24h is greater or
equal to 2 in combination with a suspected or verified infection; expressed
here as a product of the two conditions:
Sepsis${}_{Sepsis\text{-}3}=\text{Condition}_{Infection}\times\text{Condition}_{SOFA}$.
Table 6: Step 1 of two-step detection of defining features for Sepsis-3: Top 6
GAMs for Sepsis-3, ranked by data fit($\downarrow$) and complexity
($\uparrow$). Figure 3: Step 2 of two-step detection of defining features for
Sepsis-3: Feature shapes and data fit $D^{2}$ of GAMs trained on Sepsis-3 data
without (left col.) and with suspected infection (right col.) as input
features, showing satisfactory reconstruction of the target definition and
nullification of non-defining features.

SOFA-definitions as used in the previous section are the basis for further
experiments to predict sepsis according to the Sepsis-3 consensus definition
(Singer et al., 2016; Seymour et al., 2016). Sepsis according to Sepsis-3 is
present if the following two conditions are met: 1) The patient suffers from
deterioration of organ functions; 2) The patient has a suspected or verified
infection. According to Singer et al. (2016), a SOFA score of 2 points or more
indicates an organ dysfunction for patients without any pre-conditions. A
suspected or verified infection is operationalized by defining an infection
window by the first dose of antibiotics and the first draw of blood culture
samples (see eAppendix A of Seymour et al. (2016)).222The start of the window
in combination with the SOFA criterion is then the onset of sepsis. The length
of the sepsis episode is determined by the infection window. Subsequent
administered antibiotics and additional draws of blood samples can thus extend
a sepsis episode. The complete definition and the conditions we use are
compiled in Table 5.

The candidate features are leukocytes, cardiac output, sodium, and C-reactive
protein (CRP), suspected infection, and maximum SOFA score in 24 hours
(SOFA-24h). Step 1 of the detection procedure produces the results shown in
Table 6: We see that all models including suspected infection as feature
explain the data with a data fit of $D^{2}=98\%$ where the model that includes
only this feature has the smallest complexity. Suspected infection is thus
marked as potentially defining in step 1 of the detection procedure. Step 2 of
the detection procedure is shown in Figure 3. The left column displays the
feature shapes of the features of SOFA-24h, leukocytes, cardiac output,
sodium, and CRP in a GAM that does not use suspected infection as input
feature. All features show a clear contribution to the prediction of sepsis.
Including suspected infection as input feature to the GAM yields the feature
shapes shown in the right column: The contributions of all features except
suspected infection are nullified. Comparing the data fit metric $D^{2}$ for
both GAMs shows that the inclusion of suspected infection as input feature
boosts the data fit from $6\%$ to $98\%$. This means that a GAM trained on
data including suspected infection is able to perfectly replicate the Sepsis-3
status of each test input. We can thus identify suspected infection as
defining feature for the indirect measurement of Sepsis-3, compared to other
fundamental features with the highest bivariate correlation to the target
label.

Surprisingly, SOFA-24h is not identified as defining feature, but instead is
nullified. This can be explained by particularities of the dataset used in
this experiment (Lindner et al., 2022; Schamoni et al., 2022). In this
dataset, only $522$ out of $28,122$ infection observations had a SOFA-24h
measurement of less than $2$. This implies that the probability of having a
positive Sepsis-3 label is $0.9814$ if the patient has a suspected or verified
infection. This makes it possible to reduce the conditions in the Sepsis-3
definition effectively to testing for infection, and is the reason why our
method only unveiled suspected infection as the sole defining feature.

### Detecting defining features of black-box machine learning models

The proposed procedure can also be used to detect whether a state-of-the-art
machine learning model such as a deep neural network inherits a validity
problem present in the training data. This is possible without access to the
actual training data or the parameters of the model, which is thus treated as
a black-box model. Since a trained model is a function that maps input
features to labels, we can apply the detection procedure to a dataset
$T=\\{(\mathbf{x}^{m},\hat{y}^{m})\\}_{m=1}^{M}$ consisting of test inputs
$\mathbf{x}$ and labels $\hat{y}$ predicted by the black-box model. That
means, we apply the same procedure as above to a dataset of inputs and labels,
with the only difference that the labels are predictions of the black-box
model. After training a GAM on test inputs and predicted labels, we use the
statistical metrics of data fit, model complexity, and nullification to detect
if the trained black-box model relies on training data that was labeled by
indirect measurements.

Table 7: Step 1 of two-step detection of defining features for black-box
predictions: Top 5 GAMs for black-box predictions, trained with (upper) and
without (lower) bilirubin feature, ranked by data fit($\downarrow$) and
complexity ($\uparrow$). Figure 4: Step 2 of two-step detection of defining
features for black-box predictions: Feature shapes of GAMs trained on
predictions of black-box models without (cols. 1 and 2) and with (cols. 3 and
4) access to bilirubin during training. on of target function and
nullification shown in rightmost column.

We exemplify the detection procedure on the example of liver SOFA prediction.
Let us start with a neural network that has access to all $43$ original
features, including bilirubin. A first noteworthy observation is that the
accuracy of this neural network is $97\%$ on the test data, and $99.7\%$ on
the training data. This gap is exceptionally low, demanding a further
investigation by a detection procedure for defining features. For ease of
presentation of our detection procedure, we again limit the original $43$
features to the five features that are most highly correlated with the black-
box model predictions. For liver SOFA these are bilirubin, thrombocytes,
cardiac output, systematic vascular resistance index (SVRI), and urine output
in the previous $24$ hours (urine-24h). Step 1 of the detection procedure
consists of training a GAM on predictions of the neural network. As shown in
the upper half of Table 7, the GAM with lowest complexity and highest data fit
is the model using bilirubin as sole feature. This result is similar to the
outcome of step 1 displayed in Table 2. The result of step 2 of our detection
procedure is depicted in the fourth column of Figure 4. The topmost panel in
the fourth column shows that the predictions of the black-box model can be
reproduced nearly perfectly ($D^{2}=99\%$), with a feature shape for bilirubin
that is very similar to the liver SOFA definition given in Table 3. Without
access to bilirubin, the GAM in the third column of Figure 4 is shown to be
only capable to poorly reproduce the black-box model’s predictions
($D^{2}=40\%$). A comparison of the plots in the third and fourth column
illustrates that the non-negligible contribution of the features in the GAM
without access to bilirubin are nullified if bilirubin is included in the GAM.
We can therefore conclude that the black box model must have had access to the
defining feature of bilirubin in its training data, inheriting the validity
problem.

The first and second column of Figure 4 compare this result to an application
of the detection procedure to a black-box model that accesses all features
except bilirubin in its training data. The test accuracy of this model drops
to $76\%$, and a GAM trained on its predictions explains the data with a data
fit of $D^{2}=39\%$ if bilirubin is included in the GAM, and with $D^{2}=34\%$
if bilirubin is excluded. However, the feature shapes are similar and non-null
in both types of GAMs.

We furthermore conduct an ablation experiment where we deprive the neural
network trained on data including bilirubin of this very feature at test
time.333This is achieved by setting the feature value to $0$ (which is the
mean value of the standardized input features) for all test instances, and
evaluate accuracy of the new predictions. This neural network predicts SOFA
scores of $0$ in $99.7\%$ of all test predictions, and consequently its
accuracy drops to $76.3\%$ (which is approximately the frequency of gold
standard zeros in the test set).

Together, these results show that if the defining feature is included in the
training data, the black-box model is able to reconstruct the already known
definition almost perfectly. However, it lacks the ability to make correct
predictions when it is deprived of the defining features during training or
testing.

##  3 Related Work

The problem described in our work has not been discussed in the literature so
far. First, our detection procedure must not be confused with standard feature
selection (Hastie et al., 2008). These techniques are aimed at improving model
prediction while our method is a data analysis technique, aimed at
distinguishing defining features from _powerful features_ , i.e. other
features that are highly correlated with the target labels. Such features can
also take the form of _shortcuts_ where a spurious correlation between an
irrelevant input feature and the target label is exploited by the model
(Geirhos et al., 2020). This has been shown to happen in natural language
processing, image processing, and medical data science (Ribeiro et al., 2016;
Lapuschkin et al., 2019; DeGrave et al., 2021). The discrimination between
powerful features and defining features is achieved by step 2 of our method
since defining features will result in nullification even of powerful
features, while powerful features will not cause this effect.

Defining features also need to be distinguished from _leaking features_.
Rosset et al. (2009) and Kaufmann et al. (2011) define leakage in data mining
as the introduction of information about the data mining target, which should
not be legitimately available to mine from. Examples for leaking input
features are inclusion of the target itself, or of future information in a
prediction task. In contrast to illegitimate data mining from leaking
features, it is perfectly legitimate to diagnose an illness according to a
consensus definition, and the involved physiological measurements are a
legitimate part of input representations in machine learning. However, a
validity problem arises by a double usage — directly as input features and
indirectly in the definition of outputs — in difference to leaking features
that are illegitimate per se.

##  4 Discussion

We described a problem that potentially arises from the common practice of
generating machine learning data by automatically labeling clinical datasets
according to consensus definitions. Since consensus definitions are indirect
measurements based on fundamental clinical measurements, if the latter are
included as input features in machine learning datasets, they become defining
features causing machine learning models to learn nothing else but to
reconstruct the known consensus definition.

We demonstrated the applicability of our procedure to detect three cases of
indirect measurements based on three fundamentally different definitions: For
SOFAliver, the definition consists of a single additive component, which
naturally extends to definitions based on sums of multiple components. For
SOFAkidney, the definition is based on the more complex maximum-operation
between several components. In the final case of Sepsis-3, the definition is
based on a logical and-operation which can be expressed as a product of two
values. In all three cases, our procedure was able to detect a validity
problem caused by labeling via indirect measurements. Most of the consensus
definitions we are aware of can be modeled using these three basic operations.

The experiments conducted in this paper follow a cross-sectional experiment
design. However, we believe that our conclusions also extend to a longitudinal
design for prognosis where feature measurements at a current point in time are
used to predict the disease status at future points in time. In this setting,
the problem of indirect target definitions will be introduced by the auto-
correlation of the fundamental features in the time series, especially if data
imputation methods like last-value carried forward are used in dataset
creation.

One way to avoid the described validity problem is to keep the data labeling
process and the input features apart. Another option that restitutes the
application of consensus-based definitions while keeping all fundamental
features in the input representation could be a forecasting setup: In such a
scenario, the machine learning goal is to project the time series of clinical
measurements into the future, and then to apply the measurements of the
consensus definition to predict the sepsis label.

This research has been conducted in project SCIDATOS (Scientific Computing for
Improved Detection and Therapy of Sepsis), funded by the Klaus Tschira
Foundation, Germany (Grant number 00.0277.2015). We would like to thank our
colleagues at the University Medical Centre Mannheim (UMM), Germany — Manfred
Thiel, Holger Lindner, and Verana Schneider-Lindner — for permission to use
the data generated in SCIDATOS for the experiments presented in this paper.

## References

  * Agarwal et al. (2020) Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey E. Hinton.  Neural additive models: Interpretable machine learning with neural nets.  In _Proceedings of the ICML Workshop on Human Interpretability in Machine Learning_ , virtual, 2020.
  * DeGrave et al. (2021) A.J. DeGrave, J.D. Janizek, and SI Lee.  AI for radiographic COVID-19 detection selects shortcuts over signal.  _Nature Machine Intelligence_ , 3, 2021.
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2:665–673, 2020.
  * Hastie and Tibshirani (1986) Trevor Hastie and Robert Tibshirani.  Generalized additive models.  _Statistical Science_ , 1(3):297–318, 1986.
  * Hastie and Tibshirani (1990) Trevor Hastie and Robert Tibshirani.  _Generalized Additive Models_.  Chapman and Hall, 1990.
  * Hastie et al. (2008) Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  _The Elements of Statistical Learning: Data Mining, Inference, and Prediction_.  Springer, New York, NY, second edition, 2008.
  * Heckman (1986) Nancy E. Heckman.  Spline smoothing in a partly linear model.  _Journal of the Royal Statistical Society B_ , 48(2):244–248, 1986.
  * Johnson et al. (2016) Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark.  MIMIC-III, a freely accessible critical care database.  _Scientific Data_ , 3(1):160035, 2016.
  * Kaufmann et al. (2011) Shachar Kaufmann, Saharon Rosset, and Claudia Perlich.  Leakage in data mining: Formulation, detection, and avoidance.  In _Proceedings of the Conference on Knowledge Discovery and Data Mining (KDD)_ , San Diego, CA, USA, 2011.
  * Lapuschkin et al. (2019) Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Unmasking clever hans predictors and assessing what machines really learn.  _Nature Communications_ , 10(1):1–8, 2019.
  * Lindner et al. (2022) Holger A. Lindner, Shigehiko Schamoni, Thomas Kirschning, Corinna Worm, Bianka Hahn, Franz-Simon Centner, Jochen J. Schoettler, Michael Hagmann, Jörg Krebs, Dennis Mangold, Stephanie Nitsch, Stefan Riezler, Manfred Thiel, and Verena Schneider-Lindner.  Ground truth labels challenge the validity of sepsis consensus definitions in critical illness.  _Journal of Translational Medicine_ , 20(27), 2022.
  * Lou et al. (2012) Yin Lou, Rich Caruana, and Johannes Gehrke.  Intelligible models for classification and regression.  In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , London, UK, 2012.
  * McCullagh and Nelder (1989) P. McCullagh and J.A. Nelder.  _Generalized Linear Models_.  Chapman and Hall, second edition, 1989.
  * Moor et al. (2021) Michael Moor, Bastian Rieck, Max Horn, Catherine Jutzeler, and Karsten Borgwardt.  Early prediction of sepsis in the ICU using machine learning: A systematic review.  _Frontiers in Medicine_ , 8, 2021.
  * Reyna et al. (2019) Matthew A. Reyna, Christopher S. Josef, Russell Jeter, Supreeth P.B. Shashikumar, M. Brandon Westover, Shamim Nemati, Gari D. Clifford, and Ashish Sharma.  Early prediction of sepsis from clinical data: The physionet/computing in cardiology challenge 2019.  _Critical Care Medicine_ , 48(2):210–217, 2019\.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”Why Should I Trust You?” Explaining the Predictions of Any Classifier.  In _Proceedings of the Conference on Knowledge Discovery and Data Mining (KDD)_ , San Francisco, CA, USA, 2016.
  * Riezler and Hagmann (2021) Stefan Riezler and Michael Hagmann.  _Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science_.  Morgan & Claypool Publishers, 2021.
  * Rosset et al. (2009) Saharon Rosset, Claudia Perlich, Grzegorz Swirszcz, Prem Melville, and Yan Liu.  Medical data mining: insights from winning two competitions.  _Data Mining and Knowledge Discovery_ , 20:439–468, 2009\.
  * Schamoni et al. (2022) Shigehiko Schamoni, Michael Hagmann, and Stefan Riezler.  Ensembling neural networks for improved prediction and privacy in early diagnosis of sepsis.  _Proceedings of Machine Learning research: Machine Learning for Healthcare_ , 182:1–23, 2022.
  * Seymour et al. (2016) Christopher W. Seymour, Vincent X. Liu, Theodore J. Iwashyna, Frank M. Brunkhorst, Thomas D. Rea, André Scherag, Gordon Rubenfeld, Jeremy M. Kahn, Manu Shankar-Hari, Mervyn Singer, Clifford S. Deutschman, Gabriel J. Escobar, and Derek C. Angus.  Assessment of clinical criteria for sepsis for the third international consensus definitions for sepsis and septic shock (Sepsis-3).  _JAMA_ , 315(8):762–774, 2016.
  * Singer et al. (2016) Mervyn Singer, Clifford S. Deutschman, and Christopher Warren Seymour.  The third international consensus definitions for sepsis and septic shock (Sepsis-3).  _JAMA_ , 315(8):801–810, 2016.
  * Tan et al. (2018) Sarah Tan, Rich Caruana, Giles Hooker, and Yin Lou.  Distill-and-compare: Auditing black-box models using transparent model distillation.  In _Proceedings of AIES_ , New Orleans, LA, USA, 2018.
  * Vincent et al. (1996) J.L. Vincent, R. Moreno, J. Takala, S. Willatts, A. De Mendonça, H. Bruining, C. Reinhart, P. Suter, and L. Thijs.  The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure.  _Intensive Care Medicine_ , 22(7):707–710, 1996\.
  * Wood (2017) Simon N. Wood.  _Generalized Additive Models. An Introduction with R_.  Chapman & Hall/CRC, second edition, 2017.

## Appendix

### Statistical background

A prerequisite for our detection procedure for defining features is the
availability of an expressive and yet intelligible model that can be fitted to
data $D=\\{(\mathbf{x}^{n},{y}^{n})\\}_{n=1}^{N}$, where $\mathbf{x}^{n}$
denotes a $p$-dimensional vector of input features (covariates) and ${y}^{n}$
a scalar target label. As a model we adopt the class of _Generalized Additive
Models_ (GAMs) that originated in the area of biostatistics (Hastie and
Tibshirani, 1990) to lift the restriction of strictly linear features in
generalized linear regression models. GAMs have been used as interpretable
white-box models for machine learning (Lou et al., 2012) and as tools for
analyzing black-box models (Tan et al., 2018). The key idea of GAMs is
decomposing a multivariate function into a sum of usually univariate nonlinear
functions $f_{k}(x_{k})$ for each input feature $x_{k}$, called _feature
shapes_ , which are learned from the data. The general form of a GAM assumes
$Y$ to be a random variable from the exponential family, and $g(\cdot)$ to be
a nonlinear but invertible link function:

$g(\mathbb{E}[Y|\mathbf{x}])=\sum_{k=1}^{p}f_{k}(x_{k}).$ (1)

For example, a Gaussian regression model is obtained by using the identity
link function $g(x)=x$, and specifying the distribution of $Y^{n}$ to be of
the zero mean Gaussian subclass of the exponential family:

$\displaystyle Y^{n}$
$\displaystyle=\sum_{k=1}^{p}f_{k}(x_{k}^{n})+\epsilon^{n},$ (2)
$\displaystyle\textrm{ where
}\epsilon^{n}\sim\mathcal{N}(0,\sigma^{2})\textrm{ for }n=1,\ldots,N.$

For our application, we use GAMs that model feature shapes by _regression
spline_ functions. A regression spline represents a non-linear function as a
linear combination of so called fixed base functions which are used to
transform the input data. The estimated weights of this linear combination
determine the shape of the non-linear function and thus allow a model with
linear structure to learn non-linear functions from data.

In order to define a detection procedure for defining features we need two
measures, one that allows us to assess how good a model reproduces its
training data, and one that measures its complexity. We use the likelihood-
based criterion of _scaled deviance_ of a model to assess model fit. McCullagh
and Nelder (1989) define it as a metric proportional to the difference between
the log-likelihood $\ell(\mu)$ of a model $\mu$ to the log-likelihood
$\ell^{\ast}$ of the saturated model, which is the model in family that
achieves the highest possible likelihood value given the data:

$D^{\ast}_{\mu}=2(\ell^{\ast}-\ell(\mu)).$ (3)

The saturated model corresponds to an exact fit. For a Gaussian model with
known variance $\sigma^{2}$ and independent samples, the scaled deviance is:

$D^{\ast}_{\mu}=2(\ell^{\ast}-\ell(\mu))=\sum_{n=1}^{N}\frac{(y^{n}-\mu^{n})^{2}}{\sigma^{2}}.$
(4)

Following Hastie and Tibshirani (1986), we use the _percentage of deviance
explained_ to make the metric more interpretable, and denote it by $D^{2}$:

$D^{2}(\mu)=1-\frac{D^{\ast}_{\mu}}{D_{\mu_{0}}},$ (5)

where $D_{\mu_{0}}$ is the deviance for the model $\mu_{o}$ which is the
simplest possible model using only a intercept term and no other predictor.
This model just predicts a constant, namely the average $\bar{y}$ of all data
points. The likelihood of this model is the lowest possible, thus
$D^{2}(\mu)\in[0,1]$, where $0$ ($0\%$) means that the model $\mu$ fits the
data as good as $\mu_{0}$, and $1$ ($100\%$) means that $\mu$ fits the data as
good as the saturated model. For a Gaussian GAM with known variance and
independent samples, $D^{2}$ is

$D^{2}(\mu)=1-\frac{\sum_{n=1}^{N}(y^{n}-\mu^{n})^{2}}{\sum_{n=1}^{N}(y^{n}-\bar{y})^{2}}.$
(6)

In this case, $D^{2}$ equals the coefficient of determination $R^{2}$. It is
thus regarded as a generalization of the latter.

The typical model complexity measure employed for regression models are the
_degrees of freedom_ , corresponding to the number of model parameters
estimated from the data. This measure cannot be directly applied to GAMs due
to the involved smoothing process and the penalized regression estimation
procedure. Taking into account all the complexities involved with GAMs, Wood
(2017) has adapted the concept and defined a measure called _effective degrees
of freedom (edf)_ which we use to measure model complexity.

A further concept that is important in our detection procedure for defining
features is that of consistency of an estimator. It is one of the most
fundamental properties of an estimator and a necessary condition for model
interpretability.

######  Definition .1 (Consistency).

Let $M:=\left\\{p_{\theta}\colon\theta\in\Theta\right\\}$ be a parametric
statistical model where $\theta\mapsto p_{\theta}$ is injective. Further, let
$p_{\theta_{0}}\in M$ denote the true model of the data generating process for
a dataset $D=\\{(\mathbf{x}^{n},{y}^{n})\\}_{n=1}^{N}$. Then an estimator
${\theta}_{N}$ is called consistent iff for all $\epsilon>0$ holds

$\displaystyle
P\left(|{\theta}_{N}-\theta_{0}|>\epsilon\right)\xrightarrow{N\to\infty}0.$

Consistency of an estimator guarantees that the probability of learning the
true data-generating model converges to $1$ as the sample size increases,
given that the true model is among the candidates. Note that consistency holds
for spline based GAMs (Hastie and Tibshirani, 1990; Wood, 2017; Heckman,
1986), but has not been shown for variants where the features shapes are
modeled by non-spline machine learning methods like deep neural networks
(NAMs) (Agarwal et al., 2020). The consistency property of GAMs allows us to
identify defining features as those that approximate the data generating
process with a non-zero feature shape, and features that are not related to
the data generation process as those with constant zero feature shapes.

######  Proposition .2 (Nullification.).

Let $p^{\textrm{GAM}}_{{\theta}_{N}}$ be a GAM that optimizes the likelihood
of data $D=\\{(\mathbf{x}^{n},{y}^{n})\\}_{n=1}^{N}$,
$\mathbf{x}=(x_{1},\dots,x_{u},\dots,x_{p})$, that has been produced by a data
labeling function $l:(x_{1},\dots,x_{u})\mapsto y$. Furthermore, assume that
$l$ can be approximated by a model in
$M^{\textrm{GAM}}=\\{p^{\textrm{GAM}}_{\theta}:\theta\in\Theta\\}$. Then any
feature $x_{k\leq u}$ used by the data labeling function $l$ will have a non-
zero feature shape $\hat{f}_{k}(x_{k})\overset{p}{\rightarrow}f_{k}(x_{k})\neq
0$, and any other feature $x_{k>u}$ in the feature set will have a feature
shape of a constant zero function
$\hat{f}_{k}(x_{k})\overset{p}{\rightarrow}f_{k}(x_{k})=0$.

_Proof sketch._ The proposition follows directly from the consistency of
maximum likelihood estimators for GAMs. This has been shown, for example, by
Heckman (1986) for GAMs based on cubic regression splines. By consistency, the
maximum likelihood estimator $\theta_{N}$ will converge in probability to the
data generating parameters $\theta_{0}$. Since the model
$M^{\textrm{GAM}}=\\{p^{\textrm{GAM}}_{\theta}:\theta\in\Theta\\}$ is
identifiable, as by the injectivity of the mapping $\theta\mapsto p_{\theta}$,
the data generating parameters $\theta_{0}$ will identify the data generating
model $p^{\textrm{GAM}}_{{\theta}_{0}}$. By the additive structure of this
model, only features determining the feature-label relations in the data
$D=\\{(\mathbf{x}^{n},{y}^{n})\\}_{n=1}^{N}$ have non-zero feature shapes, and
the feature shapes of all other features in the feature set have constant zero
values. $\square$

### Detection procedure for defining features

Based on GAMs, the $D^{2}$ and $edf$ metrics, and the nullification criterion,
we define a detection procedure for defining features that is based on the
idea that in the presence of defining features, a simple model based solely on
such features will suffice to nearly perfectly reproduce the gold standard
target. The detection procedure for defining features starts from a set of
candidate features, for example, identified by the absolute magnitude of their
bivariate correlation with the labels, and then searches for the model with
highest deviance and lowest degrees of freedom over a set of candidate models
$\mathcal{M_{C}}$ (step 1), and confirms that the identified features are
indeed defining since all other features except the ones found in the first
step are nullified (step 2) in an extended model.

######  Definition .3 (Detection procedure).

Given a dataset of feature-label relations
$D=\\{(\mathbf{x}^{n},{y}^{n})\\}_{n=1}^{N}$ where
$\mathbf{x}=(x_{1},x_{2},\ldots,x_{p})$ is a $p$-dimensional feature vector,
let $C\subseteq\mathcal{P}(\\{1,\ldots,p\\})$ indicate the set of candidate
defining features in dataset $D$, and let
$\mathcal{M_{C}}\coloneqq\\{\mu_{c}\colon c\in C\\}$ be the set of models
obtained by fitting a GAM including a feature shape for every feature in $c$
(and nothing else) to the data $D$. A set of defining features $c^{*}$ is
detected by applying the following two-step detection procedure:

  1. 1.

$c^{*}=\operatorname*{argmax}_{c\in C}D^{2}(\mu_{c})$ where
$D^{2}(\mu_{c^{*}})$ is close to $1$, and in case the maximizer is not unique,
the maximizer is chosen such that the associated GAM $\mu_{c^{*}}$ has the
smallest effective degrees of freedom.

  2. 2.

The feature shapes of every feature $x_{j}:j\in\\{1,\ldots,p\\}\setminus
c^{*}$ added to the GAM $\mu_{c^{*}}$ is nullified in the resulting model.