# The NLP Task Effectiveness of Long-Range Transformers

Guanghui Qin Yukun Feng Benjamin Van Durme
Department of Computer Science, Johns Hopkins University
{gqin2, yfeng55<EMAIL_ADDRESS>
###### Abstract

Transformer models cannot easily scale to long sequences due to their
$O(N^{2})$ time and space complexity. This has led to Transformer variants
seeking to lessen computational complexity, such as Longformer and Performer.
While such models has theoretically greater efficiency, their effectiveness on
real NLP tasks has not been well studied. We benchmark 7 variants of
Transformer models on 5 difficult NLP tasks and 7 datasets. We design
experiments to isolate the effect of pretraining and hyperparameter settings,
to focus on their capacity for long-range attention. Moreover, we present
various methods to investigate attention behaviors, to illuminate model
details beyond metric scores. We find that attention of long-range
transformers has advantages on content selection and query-guided decoding,
but they come with previously unrecognized drawbacks such as insufficient
attention to distant tokens.

##  1 Introduction

Transformer-based models (Vaswani et al., 2017) have advanced the state of the
art in natural language processing. However, their quadratic time and space
complexity hinders their application on long texts. Various proposals have
been made to address these concerns (Tay et al., 2020c), with mathematical
guarantees on improved time or space. These models have been evaluated
primarily via perplexity (Dai et al., 2019) and non-NLP benchmarks (Tay et
al., 2020b). These metrics may not be ideal (Sun et al., 2021) and may not
reflect performance on complex NLP tasks (Arutiunian et al., 2020; Thorne et
al., 2021). We argue these metrics have not been sufficient for the
development of efficient Transformers and their practical application on long
texts, and that existing benchmarks are insufficient guides for architecture
selection.

It is not straightforward to have a fair and side-by-side comparison among
those models due to the differences between their pretraining and
hyperparameter settings (Tay et al., 2020c), and the metric scores cannot
convey detailed information inside the self-attention blocks (Sun et al.,
2021). We wish to fairly validate the effectiveness of long range attention
techniques, and to uncover the underlying factors behind model behaviors. We
critique the reliance on perplexity evaluations in previous work,
experimenting with five difficult, long-text NLP tasks. These tasks cover
typical NLP modeling scenarios: token or span-level prediction, sequence-level
classification, and sequence-to-sequence generation. We achieve competitive or
even state-of-the-art results. To our knowledge, this is the first work to
evaluate long-range transformers on such a wide spectrum of representative NLP
tasks.

To verify the key features of long range transformers, we sabotage their
distant attention to measure what they gain from long-range mechanisms. For
models without pretrained checkpoints, we migrate parameters from their
prototype models for fairness. We cover 3 main kinds of long range
transformers, including pattern-, recurrence-, and kernel-based methods. To
our knowledge we are the first to adopt all these methods to probe
transformers. To investigate the relationship between performance and document
lengths we break down the metric with a customized algorithm (Bagga and
Baldwin, 1998). Also, we use entropy and attribution analysis Li et al. (2017)
to test the effectiveness of cached memories in recurrent transformers and the
global tokens for query-based problems.

We find that long-range context brings performance gains to transformers in
some cases, which we attribute to more selective attention, especially when
the task comes with a query such as QA. Surprisingly we observe that some
long-range models do not effectively utilize distant information. We hope this
analysis helps practitioners better understand the current state of the art of
long-range attentions and suggest paths for future research.

##  2 Background

###  2.1 Long Range Transformers

Researchers have proposed a number of Transformer variants (Tay et al.,
2020c). Most of these models support decoding (causal masking) (Peng et al.,
2021a), while only a few of them have pretrained checkpoints (Beltagy et al.,
2020, inter alia). We cluster these approaches into 3 main categories.

#### Sparsified Patterns

Pattern-based methods try to make self-attention less dense. Some apply pre-
specified attention patterns, such as sliding or blocked windows, possibly
with global, random, or dilated patterns to facilitate information flow
(Zaheer et al., 2020; Ainslie et al., 2020; Brown et al., 2020, inter alia).
Although the attention of each layer is not full, the receptive field can be
increased as multiple layers are stacked. The selected or appended “global”
tokens can be task specific (Beltagy et al., 2020), allowing for direct
distant information exchange. Instead of pre-defined attention patterns, some
use content-based patterns so they become learnable, with techniques including
locality sensitive hashing (Kitaev et al., 2020), the differentiable Sinkhorn
algorithm (Tay et al., 2020a), or the learnable routing algorithm (Roy et al.,
2020).

#### Recurrence & Compressed Memory

These type of methods use segment-level recurrence to reuse the cached hidden
states of previous steps (Dai et al., 2019; Yang et al., 2019), while the
gradients are not propagated to previous segments. Instead of discarding
obsolete activations, Rae et al. (2020) compress them as memories for
efficient re-use with pooling or convolutions.

#### Low-Rank & Kernels

These methods approximate the self-attention calculation with random features
(Choromanski et al., 2021; Peng et al., 2021b) or low-rank methods (Wang et
al., 2020). Katharopoulos et al. (2020) completely discard the softmax
function with a more tractable linear dot-product of kernel features, reducing
the time complexity to linear and space complexity to constant.

###  2.2 Benchmarks and Analysis

There is not an agreed-upon standard benchmark for long sequence transformers.
Researchers have considered various tasks and domains, including language (Dai
et al., 2019), protein sequences (Choromanski et al., 2021), and images
(Katharopoulos et al., 2020). Few conduct experiments on long sequence NLP
tasks, including question answering (Beltagy et al., 2020) and summarization
(Zaheer et al., 2020). Tay et al. (2020b) proposed the Long Range Arena
comprised of six non-NLP tasks to exclude the factor of pretraining.
Concurrent to our efforts, Shaham et al. (2022) propose a suite of text-to-
text tasks as a long sequence NLP benchmark.

Researchers are also interested in the utility of context in transformers. Rae
and Razavi (2020) find that Transformer-XL doesn’t necessarily need long and
deep contexts. Sun et al. (2021) reveal that Longformer and Routing
transformers can only reduce the perplexity of LMs on a small set of tokens.
More related to our work, Lai et al. (2020) show that BERT can make use of a
larger scope of context than a BiLSTM.

##  3 Setup

###  3.1 Settings

It is non-trivial to contrast distinct transformer models, due to differences
between their pretraining and hyper-parameter settings. Our goal is to
minimize these confounding factors to allow a focus on the long range
attention ability of each model on different tasks. We therefore propose two
sets of experimental conditions.

#### Restricted Attention Range

To evaluate the performance gain from long-range attention, we evaluate models
in both their default context-aware settings, and a context-agnostic setting
that restricts the receptive range of the self-attention blocks. For pattern-
based transformers, we achieve the restriction by segmenting the input
sequence into chunks and run the transformers on segments independently. In
practice, we segment the texts into chunks that are so short ($n\leq 512$)
that all transformers can handle. For recurrent models, we could force the
model not to reuse past activations to eliminate the dependencies between
segments.

#### Parameter Migration

Kernel-based models usually don’t come with checkpoints for general tasks, but
they may have similar structure to other pretrained models like BERT and may
be designed to approximate the original results. Therefore, it is feasible to
migrate parameters from pretrained prototypes to their “efficient version” to
observe if the performance could be preserved. This type of method can be
suitable for models without additional parameters, such as Performer.

###  3.2 Models and Tasks

#### Models

We consider three approaches: 1) pattern-based: Longformer (Beltagy et al.,
2020), and Big Bird (Zaheer et al., 2020); 2) recurrent: XLNet (Yang et al.,
2019); and 3) kernel-based: Performer (Choromanski et al., 2021). We also
include the results of RoBERTa (Liu et al., 2019) and SpanBERT (Joshi et al.,
2020), where some of our approaches were initialized from those two non-long-
range models. Due to memory requirements we use the base version for all
models.111We used the codebase of Katharopoulos et al. (2020) for Performer
and Huggingface (Wolf et al., 2020) for the rest.

#### Tasks

We cover five tasks of three types, including 1) span-level predictions:
coreference resolution (Coref.) (Weischedel et al., 2011) and extractive
question answering (eQA) (Joshi et al., 2017); 2) sequence classification:
natural language inference (NLI) (Yin et al., 2021); and 3) seq2seq:
summarization (Summ.) (Chen et al., 2021; Huang et al., 2021), abstractive QA
(aQA) (Dasigi et al., 2021; Pang et al., 2021). We pick seven datasets that
involve long texts, whose statistics are shown in table 1. For more details
about the data preprocessing, please refer to appendix A.

Dataset | Task | #tokens | #docs
---|---|---|---
Ontonotes | Coref. | 467 | 3493
DocNLI | NLI | 399 | 1.44m
TriviaQA | eQA | 2895 | 95k
Qasper | aQA | 3.7k | 5.7k
QuALITY | aQA | 4.2k | 6.7k
SummFD | Summ. | 5.6k | 4.3k
GovRep | Summ. | 7.9k | 19k
Table 1: Task and dataset overview. The #tokens is the number of tokens per
doc on average.

##  4 Experiments

###  4.1 Coreference Resolution

Coreference resolution (coref.) is the task of identifying mention spans and
clustering them into entities. We consider multiple coreference strategies: 1)
the widely used Coarse2Fine (C2F) method222We used the re-implementation by
Gardner et al. (2018). (Lee et al., 2018) which relies on span representations
and 2) the current state of the art method called Start2End (S2E) (Kirstain et
al., 2021) that works on token representations. The dataset we use is
Ontonotes 5.0. Transformers considered include Longformer, XLNet, and
Performer. We migrate the parameters of SpanBERT and RoBERTa to Performer and
include results on these models as well. We segment the input tokens into
chunks with lengths of $L$, even if the model might be able to handle longer
texts. For Longformer with global tokens, we lack a natural choice for global
token so we simply consider all tokens global. For XLNet, we keep the memory
of the same length of the segments (e.g. we keep a memory length of 256 for a
model with segment length 256).333We adopt the same strategies with
segmentation and memories in the remainder of the paper. The results are shown
in table 2. Refer to appendix B for detailed metrics.

| Encoder |  $L$=128 |  $L$=256 |  $L$=512
---|---|---|---|---
Model: Coarse2Fine | Longformer | 75.74 | 76.72 | 77.36
LongformerG | 75.68 | 76.25 | 77.23
XLNet | 74.57 | 74.48 | 74.33
XLNetm | 74.73 | 75.76 | 76.29
RoBERTa | 74.64 | 76.45 | 76.83
RoBERTap | 51.58 | 51.71 | 50.39
SpanBERT | 75.04 | 75.84 | 76.59
SpanBERTp | 52.46 | 52.06 | 50.51
Longformer ($L$=1024) | 76.77 |
| Longformer ($L$=4096) | 76.32 |
Model: Start2End | Longformer | 74.77 | 76.27 | 77.73
LongformerG | 74.15 | 76.19 | 77.41
XLNet | 45.89 | 60.05 | 68.23
XLNetm | 52.61 | 56.37 | 66.91
RoBERTa | 71.96 | 76.27 | 77.78
RoBERTap | 40.06 | 42.35 | 41.69
SpanBERT | 68.70 | 74.27 | 75.32
SpanBERTp | 38.69 | 41.93 | 42.10
Longformer ($L$=1024) | 77.54 |
| Longformer ($L$=4096) | 77.57 |
Table 2:  Coref. results with the C2F and the S2E models. Numbers are averaged
F1 (MUC, B3, and CEAF${}_{\phi_{4}}$). Longformer with G uses global tokens.
XLNet with m uses recurrence memory. Encoders with p have their self-attention
replaced with a Performer kernel.

###  4.2 Natural Language Inference

NLI is a classification task concerning a premise and hypothesis with variable
lengths. The DocNLI dataset uses document-length inputs with binary labels. We
adopt the model proposed by Yin et al. (2021), and consider XLNet and
Longformer. In detail, we make the prediction on the CLS token, which is at
the beginning for Longformer and at the end for XLNet. Since it is unnecessary
to run encoders over multiple independent segments only to obtain the encoding
of CLS, we adopt the strategy of Yin et al. (2021) (truncate the sequence to
$L$ while preserving the hypothesis) for Longformer. The results are shown in
table 3.

Encoder |  $L$=128 |  $L$=256 |  $L$=512
---|---|---|---
XLNet | 29.95 | 40.39 | 24.31
XLNetm | 64.28 | 76.87 | 46.56
Encoder |  $L$=512 |  $L$=1024 |  $L$=4096
Longformer | 45.28 | 45.96 | 44.42
Table 3:  F1 scores on the test set for DocNLI. We keep $L$ large for
Longformer to encompass the hypotheses.

###  4.3 Question Answering

Encoder | F1 | EM
---|---|---
Longformer | 63.41 | 58.89
LongformerG | 72.96 | 67.88
LongformerS | 63.88 | 59.13
BigBird | 71.78 | 66.86
BigBirdS | 63.67 | 58.98
XLNet | 60.05 | 55.62
XLNetm | 62.85 | 58.13
Table 4:  Results for TraviaQA. S implies that the Longformer runs on
segmented inputs, where we set the segment length as 512. G indicates that the
Longformer sets question as the global tokens.

For question answering we experiment with eQA (TriviaQA) and aQA (Qasper and
QuALITY). We adopt the most straightforward method of concatenating the
question and passage, then extract or decode the answers. Longformer, BigBird,
XLNet are tested for encoder-only tasks, and Longformer-Encoder-Decoder (LED)
is used for seq2seq tasks. For models with global tokens (LongformerG and
BigBird), we set the query as global tokens.444The global tokens of BigBird
are located in the first 2 blocks and cannot be customized, so we place the
query at the beginning of the sequence.

TriviaQA is a question answering dataset that involves extracting answer spans
from reference documents. We use F1 and exact match (EM) to evaluate the
performance. Qasper addresses the QA task in the domain of academic papers. It
involves various answer types: extractive, abstractive, boolean, and
unanswerable. We unify such tasks as an abstractive QA task (cf. Shaham et al.
(2022)) and implement an LED-based decoder to generate answers. F1 score is
used for the evaluation of Qasper. QuALITY is a multiple choice QA task. Given
a question and a passage, the task is to select the correct answer from
several candidates. We regard it as a seq2seq problem, with the objective to
predict the correct answer conditioned on the concatenation of query and
passage. During inference, the candidate answer with the least perplexity is
selected. Results shown in tables 4 and 5.

Model |  Qasper (F1) |  QuALITY (Acc)
---|---|---
LED | 18.79 | 31.88
LEDG | 28.64 | 31.40
LEDS | 17.86 | 32.55
Table 5:  Accuracy on Qasper and QuALITY. LEDS indicates the segmented inputs
with length of 1536.

###  4.4 Summarization

Task | Model | R1 | R2 | RL
---|---|---|---|---
SummFD | LED | 31.99 | 6.73 | 18.05
LEDS | 31.73 | 6.43 | 17.98
GovRep | LED | 54.81 | 25.89 | 28.13
LEDS | 52.38 | 23.29 | 26.13
Table 6:  Results for Summarization on SummFD and GovRep. R1, R2, and RL
stands for ROUGE unigram, bigram, and longest common subsequence.

As a typical seq2seq problem, we adopt LED with context-aware and context-
agnostic settings to perform the summarization task. The input sequence is
chunked into segments in the context-agnostic setting, otherwise the cross
attention in LED can attend to the complete sequence in the context-aware
setting. Intuitively, the summary is likely to benefit from the contextual
representation with a broader view of the document. Two datasets are used: 1)
SummScreen, and 2) GovReport. SumScreen addresses the domain of TV shows.
Following Shaham et al. (2022), we use the subset of ForeverDreaming (SummFD)
consisting of 88 different shows. The goal is to summarize the transcript of
an episode, for which the recap is used as the ground truth summary. GovReport
is another long-document summarization dataset, in the domain of government
policies with human-written summaries. ROUGE (Lin, 2004) is used as the
evaluation metric. Results are shown in table 6.

##  5 Analysis

###  5.1 Overall Performance

Do long-range transformers improve the overall performance on a task that
involves long contexts? The answers vary for different tasks and encoders.

Considering table 2, we find the following. 1) RoBERTa ($L$=512) achieved the
best results, higher than all Longformers that were pretrained further upon
RoBERTa. 2) Longformer gains an advantage when the segments get longer, and
this advantage is saturated when the segment length reaches 512 – there is no
performance gain with an attention range larger than 512. 3) Performer-based
models under-perform their corresponding non-kernelized models by a huge gap.
4) XLNet performs better if cached memory is applied, and this is more
observable for the S2E model.

Reading table 3, we have a similar observation on XLNet that memory is very
helpful, where the incorporation of memory significantly boosts the prediction
performance. It is worth mentioning that our results with XLNet are state of
the art, higher than the RoBERTa-large results reported in Yin et al. (2021)
even with much fewer parameters. Longformer fails for this task, and the
performance even drops for longer segments, which is consistent with the
observation in Yin et al. (2021).

Table 4 and 5 show the results of three different QA tasks. Setting queries as
global tokens greatly benefits the overall performance on both eQA and aQA,
while other context-aware settings only slightly outperform the context-
agnostic models. We conclude that the performance gain of the context-aware
models mostly comes from enhanced attention to the query, which we analyze
further in section 5.5. The difference between context-agnostic and context-
aware models is not observable from the results of QuALITY, which is modeled
as a task to retrieve the most likely candidate by decoding perplexity. We
suspect that the decoder does not heavily rely on the long-range attention in
the encoder side, given that cross attention itself is already full.

Table 6 shows the results on summarization tasks and we do see some slight
superiority of the context-aware models. As we are summarizing long texts, it
is unnecessary for the model to pay attention to every token in the document,
so we speculate that the performance improvement may be related to the
selectivity of encoder-side attention, which is further analyzed in section
5.4.

###  5.2 Metric Breakdown

As the selling point of long-range transformers is handling long text, it is
interesting to compare performance on documents with different lengths.

It is straightforward for each task to cluster the documents into buckets
according to their lengths and run the evaluation on each bucket, except for
coref. For coref., mentions can be very close even for long documents. To
mitigate this problem, we propose a breakdown-version B3 (Bagga and Baldwin,
1998). The B3 algorithm is calculated by taking the average of the precision
and recall over all mentions. Suppose that we’d like to calculate the B3
metric for entity mentions with distance555Distance between the first
wordpiece of mentions is used. in the range $[L_{1},L_{2}]$, for the B3 of
mention $e$, we only take into account the coreferent mentions whose distances
from $e$ fall into $[L_{1},L_{2}]$ and ignore those out of the scope. We can
similarly get a breakdown B3 for all other ranges and depict the correlation
between B3 and the mention distances.

Figure 1:  Breakdown B3 metric for coref. Above: B3 scores of 4 models for the
buckets with the ranges from $[1,8)$ to $[512,\infty)$. Below: The B3 scores
relative to Longformer ($L$=4096) of the other models.

As an example of span/token-level prediction, we analyze coreference
resolution with both Longformer and XLNetm ($L$=512). The breakdown analysis
of coref. is shown in fig. 1, where we show both absolute B3 scores and
relative scores against the Longformer ($L$=4096) model. We adopt C2F as the
model and Ontonotes 5.0 as the dataset. We can see that the performance of all
models follow the same trend and peaks at the $[16,32)$ bucket. Also, the
relative B3 graph shows that Longformer with longer context encoding does NOT
benefit on distant mentions. On the contrary, they suffer more from distance
than shorter-context models, which we find counter-intuitive.

The breakdown of DocNLI is shown in fig. 2. We show the results on XLNetm with
different segment lengths (cf. fig3 in Yin et al. (2021)). All models have
similar performance for short documents, and XLNet ($L$=256) starts to
dominate when the document is longer. DocNLI is made up of examples pulled
from different datasets and then converted into NLI form, where different
datasets may have examples of different average lengths. Therefore the length
of an example in DocNLI may correlate with different domains making up the
dataset, which would interfere with our analysis. Future work will consider
other datasets without this confounding concern.

For query-based problems, the breakdown analysis of TriviaQA is shown in fig.
3. All models perform well on short texts, while the models with global tokens
obtain a greater advantage over the baseline models when the document gets
longer. We think that the global token mechanism could help the model be less
distracted on long texts via more attention on the queries (section 5.5),
which consequently improves the performance.

We do not conduct any breakdown analysis for seq2seq problems, given that the
sequence length directly contributes to the ROUGE metric.

Figure 2: The breakdown analysis of DocNLI. We adopt the same notation for
buckets as fig. 1. Figure 3: The performance of Longformer and BigBird on
different lengths of TraviaQA documents.

###  5.3 The Attribution of Recurrence Memories

Even if we know that distant context can help or hurt performance, it’s still
unclear how much they contribute to the predictions. One way to quantify this
is attribution analysis (Simonyan et al., 2014; Li et al., 2017). Suppose
$\ell$ is our loss function and $\mathbf{e}_{i}\in\mathbb{R}^{d}$ is the word
embedding of the $i$-th token. We use $\alpha_{i}$ to measure the attribution
of the $i$-th token to the final prediction where

$\alpha_{i}=\left\|\frac{\partial\ell}{\partial\mathbf{e}_{i}}\right\|_{l}.$
(1)

We set $l=1$ to take the $L_{1}$ norm in practice, and the ground truth labels
in the test set are used to calculate the gradient. Also, recurrent models
like XLNet usually stop the gradient from being propagated back to the cached
memory, and we temporarily turn off this feature for analysis.

Figure 4:  The gradient distribution over tokens for XLNetm on DocNLI
documents.

We apply this method to XLNet on DocNLI. We pick 128 documents of lengths
between 1,000 and 1,024 from the dataset for analysis. We normalize the
$\alpha_{i}$ over all tokens for each document, and take an average on each
token index666We use wordpieces as tokens. across all documents. The results
are shown in fig. 4. We see the attribution of tokens is stratified according
to their segment lengths, with a minor peak at the end of each segment where
the CLS token is located. As we have more and more segments, the attribution
of distant tokens to the final prediction becomes negligible, although this
problem can be alleviated by enlarging the segment length.

###  5.4 Content Selection in Cross Attention

Figure 5:  Attention distribution entropy (above) for documents with different
lengths and gradient distribution (below) over source tokens for LED on
SummFD.

The cross attention of LED is able to attend to the whole document no matter
if we segment the inputs or not. However, the problem for summarization is
whether the attention is selective, given that only a portion of the source
document is informative. We plot the gradient distribution and the entropy of
the attention distribution over the input tokens in fig. 5 with LED and
SummFD.

Reading the entropy curve, we find that the entropy of long context without
segmentation (LED) is consistently lower, which implies its attention is more
selective, and this may explain the reason behind its superiority in table 6.
Reading the gradient curve, we find that both models have a relatively uniform
attribution over tokens with a slight slope on the left side due to the
existence of short sequences. This is reasonable because summarization
requires the decoder to skim over the whole document, thus cross attention
shouldn’t have a locality preference.

###  5.5 Query-Guided Extraction and Decoding

Figure 6:  The accumulated gradient on query vs. the proportion of query, i.e.
(query length / doc length). We conduct experiment with Longformer and BigBird
on TriviaQA.

Different from other long-doc problems, the queries in QA can be treated as a
“guidance” on how to read the documents, and it is interesting to see if the
encoders can exploit this structure. Therefore, we conduct attribution
analysis on TriviaQA with Longformer and BigBird.777We adopt the same strategy
as in section 5.3 The results are shown on fig. 6, where we plot the
relationship between the proportion of accumulated gradients on queries and
the proportion of query tokens in the document. We find that models with
global tokens (LongformerG and BigBird) pay more attention on queries, which
in turn helps them to outperform the baseline models (cf. table 4).

For scenarios where seq2seq decoding meets queries, intuition suggests queries
could instruct the cross-attention to attend to specific tokens, making the
decoding more selective. Thus, we plot the entropy of attention distribution
on source sequence against the doc length in fig. 7. Longformer with global
token unquestionably has lower entropy, implying their decoding is more
targeted.

Figure 7: Entropy of source-side attention distribution vs. document length on
Qasper and QuALITY.

###  5.6 Error Propagation of Kernel Methods

We find that Performer couldn’t match the results of its prototypes (table 2).
We suspect that the error incurred by Kernel approximation may not be
acceptable for span-level tasks like coref. We conducted another set of
experiments: Instead of training the performer model from the checkpoints of
SpanBERT, we directly replace some layers from a fine-tuned SpanBERT model
with Performer layers with their parameters reserved. 888We find that further
fine-tuning can boost the results to certain extent, but no training is
included for analysis. Experiments are conducted with C2F on Ontonotes 5.0. In
fig. 8, we try to replace $P$ layers from SpanBERT in a top-down manner, where
$P=1,2,\dots,12$. We also try to use different feature dimensions to exclude
the possibility of insufficient approximations. We do find Performer with
large dimension of random features brings marginal advantages to the
performance. We find that the performance of Performer is not sensitive to the
dimension of random features, which indicates that we can economically use
very-low dimensional vectors. More importantly, the performance drops
dramatically as we replace more layers, implying that the accumulated errors
can be significant when the stacked transformers are deep.

Figure 8:  Results of SpanBERT ($L$=512) with layers replaced with Performer
layers. The baseline is the performance of the original model. “#Fea” is the
dimension of the random features.

##  6 Discussion

Researchers have proposed many innovative methods for efficient self-attention
over long sequences. The key ideas work as desired in certain cases, though we
demonstrate several drawbacks.

As the most popular approach, pattern-based methods do have advantages over
chunking methods in seq2seq problems, while they fall behind in coref and NLI.
Also, the global tokens work well when explicit query tokens are available
(section 5.5), making the queries more attended with higher attribution to the
prediction, and the cross attention more selective with lower entropy. Global
tokens hurt performance if they are applied to the whole document for common
tasks (table 2).

The memory of recurrence models generally improves performance, proving
historical hidden states are beneficial for transformers not only in LMs but
also in other tasks. However, XLNet does not fully exploit the history tokens,
with some information unattended (section 5.3). We speculate that it is
because XLNet was pretrained to predict local tokens, which does not
frequently need the participation of distant tokens. Also, the stop-gradient
trick may play a role in the gradient vanishing problem as well.

Kernel-based methods work for shallow networks, but face serious error
accumulation when transformers are deeply stacked (section 5.6). Although
further fine-tuning can somewhat boost the performance, they struggle to be on
par with corresponding prototypes (table 2).

##  7 Conclusion

We conduct experiments with various long-range transformers on NLP tasks that
involve long sequences, trying to fairly evaluate their long-range attention
ability. We analyze the attention behaviors of these transformers, which
reveals their characteristics for their success and some drawbacks for their
failures. We hope this work helps inform the development of new models and in
the selection of existing models for a given task.

## References

  * Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020.  ETC: Encoding Long and Structured Inputs in Transformers.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Arutiunian et al. (2020) Artashes Arutiunian, Morgan McGuire, Hallvar Gisnås, Sheik Mohamed Imran, Dean Pleban, Priyank Negi, and David Arnoldo Ortiz Lozano. 2020.  Reproducibility Challenge: Reformer.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , page 10.
  * Bagga and Baldwin (1998) Amit Bagga and Breck Baldwin. 1998.  Algorithms for Scoring Coreference Chains.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The Long-Document Transformer.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.
  * Chen et al. (2021) Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021.  SummScreen: A Dataset for Abstractive Screenplay Summarization.
  * Choromanski et al. (2021) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2021.  Rethinking Attention with Performers.  In _International Conference on Learning Representations (ICLR)_.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.  In _Association for Computational Linguistics (ACL)_.
  * Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021.  A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers.  In _North American Association for Computational Linguistics (NAACL)_.
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018.  AllenNLP: A Deep Semantic Natural Language Processing Platform.  In _Association for Computational Linguistics (ACL)_.
  * Huang et al. (2021) Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021.  Efficient Attentions for Long Document Summarization.  In _North American Association for Computational Linguistics (NAACL)_ , pages 1419–1436. Association for Computational Linguistics.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving Pre-training by Representing and Predicting Spans.  _Transactions of the Association for Computational Linguistics (TACL)_ , 8:64–77.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.  In _Association for Computational Linguistics (ACL)_.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020\.  Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.  In _International Conference on Machine Learning (ICML)_.
  * Kirstain et al. (2021) Yuval Kirstain, Ori Ram, and Omer Levy. 2021.  Coreference Resolution without Span Representations.  In _Association for Computational Linguistics (ACL)_.
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The Efficient Transformer.  In _International Conference on Learning Representations (ICLR)_.
  * Lai et al. (2020) Yi-An Lai, Garima Lalwani, and Yi Zhang. 2020.  Context Analysis for Pre-trained Masked Language Models.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3789–3804. Association for Computational Linguistics.
  * Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018.  Higher-order Coreference Resolution with Coarse-to-fine Inference.  In _North American Association for Computational Linguistics (NAACL)_.
  * Li et al. (2017) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2017.  Visualizing and Understanding Neural Models in NLP.  In _Association for Computational Linguistics (ACL)_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , page 8.
  * Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. S. Zettlemoyer, and V. Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.
  * Luo (2005) Xiaoqiang Luo. 2005.  On Coreference Resolution Performance Metrics.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , page 8.
  * Pang et al. (2021) Richard Yuanzhe Pang, A. Parrish, Nitish Joshi, N. Nangia, J. Phang, A. Chen, V. Padmakumar, J. Ma, J. Thompson, H. He, and S. R. Bowman. 2021.  QuALITY: Question Answering with Long Input Texts, Yes!
  * Peng et al. (2021a) Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A. Smith. 2021a.  ABC: Attention with Bounded-memory Control.
  * Peng et al. (2021b) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021b.  Random Feature Attention.  In _International Conference on Learning Representations (ICLR)_.
  * Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2020\.  Compressive Transformers for Long-Range Sequence Modelling.  In _International Conference on Learning Representations (ICLR)_.
  * Rae and Razavi (2020) Jack W. Rae and Ali Razavi. 2020.  Do Transformers Need Deep Long-Range Memory.  In _Association for Computational Linguistics (ACL)_.
  * Roy et al. (2020) Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020.  Efficient Content-Based Sparse Attention with Routing Transformers.  _Transactions of the Association for Computational Linguistics (TACL)_.
  * Shaham et al. (2022) Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022.  SCROLLS: Standardized CompaRison Over Long Language Sequences.
  * Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.
  * Sun et al. (2021) Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021.  Do Long-Range Language Models Actually Use Long-Range Context?  In _Empirical Methods in Natural Language Processing (EMNLP)_ , page 16.
  * Tay et al. (2020a) Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020a.  Sparse Sinkhorn Attention.  In _International Conference on Machine Learning (ICML)_.
  * Tay et al. (2020b) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020b.  Long Range Arena: A Benchmark for Efficient Transformers.
  * Tay et al. (2020c) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020c.  Efficient Transformers: A Survey.
  * Thorne et al. (2021) James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, and Alon Halevy. 2021.  Database Reasoning Over Text.  In _Association for Computational Linguistics (ACL)_.
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017.  Attention Is All You Need.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Vilain et al. (1995) Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995.  A model-theoretic coreference scoring scheme.  In _Conference on Message Understanding_ , page 45. Association for Computational Linguistics.
  * Wang et al. (2020) Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-Attention with Linear Complexity.
  * Weischedel et al. (2011) R. Weischedel, E. Hovy, M. Marcus, M. Palmer, R. Belvin, S. Pradhan, L. Ramshaw, and N. Xue. 2011.  OntoNotes: A large training corpus for enhanced processing.  In _Handbook of Natural Language Processing and Machine Translation. Springer_. Springer.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 38–45. Association for Computational Linguistics.
  * Yang et al. (2019) Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. 2019.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Yin et al. (2021) Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021.  DocNLI: A Large-scale Dataset for Document-level Natural Language Inference.  In _Association for Computational Linguistics (ACL)_.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big Bird: Transformers for Longer Sequences.  In _Advances in Neural Information Processing Systems (NeurIPS)_.

##  Appendix A Data Preprocessing

For TriviaQA, we use the scripts in Longformer Beltagy et al. (2020) to pre-
process the data. We adopt the dataset of QuALITY and GovReport from Shaham et
al. (2022). For other datasets, we process the data as instructed in their
original repositories.

##  Appendix B Detailed Results of Coreference Resolution

The full results of coreference resolution is shown in tables 7 and 8. We use
MUC (Vilain et al., 1995), B3 (Bagga and Baldwin, 1998), and
CEAF${}_{\phi_{4}}$ (Luo, 2005) as the evaluation metrics. We only reported
the “Avg.” in the main paper, which is the averaged F1 over 3 metrics.

Encoder | MUC | B3 | CEAF${}_{\phi_{4}}$ | Avg.
---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
Longformer ($L$=128) | 81.7 | 83.7 | 82.7 | 71.6 | 75.8 | 73.7 | 71.3 | 70.4 | 70.9 | 75.7
LongformerG ($L$=128) | 81.1 | 84.3 | 82.7 | 70.6 | 76.5 | 73.4 | 71.0 | 70.9 | 70.9 | 75.7
Longformer ($L$=256) | 81.6 | 85.2 | 83.4 | 71.8 | 78.1 | 74.8 | 71.6 | 72.3 | 72.0 | 76.7
LongformerG ($L$=256) | 81.4 | 84.8 | 83.1 | 71.4 | 77.6 | 74.4 | 71.0 | 71.7 | 71.3 | 76.3
Longformer ($L$=512) | 82.4 | 85.3 | 83.8 | 73.0 | 78.6 | 75.7 | 72.6 | 72.5 | 72.5 | 77.4
LongformerG ($L$=512) | 82.6 | 85.0 | 83.8 | 73.2 | 77.9 | 75.5 | 72.4 | 72.4 | 72.4 | 77.2
Longformer ($L$=1024) | 82.1 | 84.9 | 83.5 | 72.3 | 77.7 | 74.9 | 72.0 | 72.0 | 72.0 | 76.8
Longformer ($L$=4096) | 82.0 | 84.2 | 83.1 | 72.4 | 76.2 | 74.3 | 71.6 | 71.6 | 71.6 | 76.3
RoBERTa ($L$=128) | 81.4 | 82.5 | 81.9 | 70.7 | 73.9 | 72.2 | 71.4 | 68.2 | 69.7 | 74.6
RoBERTap ($L$=128) | 69.4 | 57.7 | 63.0 | 55.9 | 42.5 | 48.3 | 49.9 | 38.4 | 43.4 | 51.6
RoBERTa ($L$=256) | 82.0 | 84.5 | 83.2 | 72.2 | 77.2 | 74.6 | 72.3 | 70.7 | 71.5 | 76.5
RoBERTap ($L$=256) | 68.7 | 57.9 | 62.9 | 55.7 | 43.1 | 48.6 | 49.3 | 39.2 | 43.7 | 51.7
RoBERTa ($L$=512) | 81.6 | 85.0 | 83.3 | 72.1 | 78.2 | 75.1 | 72.1 | 72.2 | 72.2 | 76.8
RoBERTap ($L$=512) | 68.0 | 56.1 | 61.5 | 55.1 | 40.8 | 46.9 | 48.4 | 38.4 | 42.8 | 50.4
SpanBERT ($L$=128) | 82.0 | 82.2 | 82.1 | 72.0 | 73.7 | 72.8 | 71.5 | 69.0 | 70.2 | 75.0
SpanBERTp ($L$=128) | 70.6 | 56.8 | 63.0 | 58.1 | 42.9 | 49.4 | 50.8 | 40.5 | 45.1 | 52.5
SpanBERT ($L$=256) | 82.7 | 82.8 | 82.7 | 73.0 | 74.1 | 73.5 | 71.9 | 70.6 | 71.3 | 75.8
SpanBERTp ($L$=256) | 70.0 | 56.4 | 62.5 | 58.5 | 41.7 | 48.7 | 50.1 | 40.8 | 45.0 | 52.1
SpanBERT ($L$=512) | 83.3 | 82.9 | 83.1 | 74.4 | 74.8 | 74.6 | 72.6 | 71.7 | 72.1 | 76.6
SpanBERTp ($L$=512) | 67.6 | 55.7 | 61.1 | 56.2 | 40.8 | 47.3 | 47.2 | 39.8 | 43.2 | 50.5
XLNet ($L$=128, $m$=0) | 81.6 | 82.7 | 82.1 | 71.2 | 73.4 | 72.3 | 70.0 | 68.6 | 69.3 | 74.6
XLNet ($L$=128, $m$=128) | 81.7 | 82.6 | 82.1 | 71.9 | 73.3 | 72.6 | 69.9 | 69.0 | 69.5 | 74.7
XLNet ($L$=256, $m$=0) | 79.4 | 84.2 | 81.7 | 68.5 | 76.0 | 72.0 | 68.5 | 70.9 | 69.7 | 74.5
XLNet ($L$=256, $m$=256) | 84.3 | 81.8 | 83.0 | 75.4 | 72.2 | 73.8 | 72.2 | 68.9 | 70.5 | 75.8
XLNet ($L$=256, $m$=512) | 80.9 | 84.5 | 82.7 | 71.0 | 76.7 | 73.7 | 70.5 | 71.2 | 70.9 | 75.8
XLNet ($L$=512, $m$=0) | 79.0 | 85.2 | 82.0 | 67.4 | 77.4 | 72.1 | 69.1 | 68.8 | 69.0 | 74.3
XLNet ($L$=512, $m$=512) | 82.1 | 84.1 | 83.1 | 72.5 | 76.1 | 74.3 | 72.8 | 70.3 | 71.5 | 76.3
Table 7: Full results on Ontonotes with the coarse2fine model. Refer to table 2 for the definition of the notations. Encoder | MUC | B3 | CEAF${}_{\phi_{4}}$ | Avg.
---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
Longformer ($L$=128) | 84.4 | 80.0 | 82.1 | 75.3 | 70.5 | 72.8 | 72.9 | 66.0 | 69.3 | 74.8
LongformerG ($L$=128) | 84.4 | 79.1 | 81.7 | 75.1 | 69.2 | 72.0 | 72.8 | 65.2 | 68.8 | 74.2
Longformer ($L$=256) | 84.8 | 81.7 | 83.2 | 76.2 | 72.5 | 74.3 | 73.9 | 68.8 | 71.2 | 76.3
LongformerG ($L$=256) | 84.4 | 81.8 | 83.1 | 75.6 | 73.1 | 74.3 | 74.3 | 68.3 | 71.2 | 76.2
Longformer ($L$=512) | 85.5 | 82.6 | 84.0 | 78.0 | 74.5 | 76.2 | 75.2 | 70.7 | 72.9 | 77.7
LongformerG ($L$=512) | 84.5 | 83.4 | 83.9 | 76.2 | 75.2 | 75.7 | 74.3 | 70.9 | 72.6 | 77.4
Longformer ($L$=1024) | 86.0 | 82.1 | 84.0 | 78.7 | 73.4 | 76.0 | 75.2 | 70.2 | 72.6 | 77.5
LongformerG ($L$=1024) | 82.4 | 79.2 | 80.8 | 72.2 | 69.5 | 70.8 | 72.2 | 65.4 | 68.6 | 73.4
Longformer ($L$=4096) | 85.2 | 82.9 | 84.1 | 77.4 | 74.6 | 76.0 | 74.8 | 70.7 | 72.7 | 77.6
RoBERTa ($L$=128) | 81.1 | 78.0 | 79.6 | 70.5 | 68.0 | 69.3 | 71.2 | 63.3 | 67.0 | 72.0
RoBERTap ($L$=128) | 61.3 | 45.0 | 51.9 | 45.9 | 30.3 | 36.5 | 41.4 | 25.8 | 31.8 | 40.1
RoBERTa ($L$=256) | 84.7 | 81.8 | 83.2 | 76.0 | 72.6 | 74.3 | 74.2 | 68.5 | 71.3 | 76.3
RoBERTap ($L$=256) | 67.7 | 46.0 | 54.8 | 53.0 | 30.8 | 39.0 | 43.8 | 26.9 | 33.3 | 42.4
RoBERTa ($L$=512) | 85.7 | 82.6 | 84.1 | 78.1 | 74.3 | 76.2 | 75.2 | 70.9 | 73.0 | 77.8
RoBERTap ($L$=512) | 67.0 | 45.0 | 53.9 | 53.0 | 29.8 | 38.1 | 43.2 | 26.8 | 33.1 | 41.7
SpanBERT ($L$=128) | 78.2 | 75.5 | 76.8 | 66.6 | 64.3 | 65.5 | 68.2 | 60.5 | 64.1 | 68.7
SpanBERTp ($L$=128) | 56.5 | 45.7 | 50.5 | 39.8 | 31.5 | 35.1 | 38.4 | 25.1 | 30.4 | 38.7
SpanBERT ($L$=256) | 83.2 | 79.6 | 81.4 | 74.4 | 70.1 | 72.2 | 71.7 | 67.0 | 69.3 | 74.3
SpanBERTp ($L$=256) | 64.1 | 46.5 | 53.9 | 49.8 | 31.8 | 38.8 | 40.7 | 27.9 | 33.1 | 41.9
SpanBERT ($L$=512) | 83.5 | 81.3 | 82.4 | 74.6 | 71.9 | 73.2 | 72.3 | 68.5 | 70.3 | 75.3
SpanBERTp ($L$=512) | 63.7 | 47.0 | 54.1 | 48.7 | 32.0 | 38.6 | 42.2 | 27.9 | 33.6 | 42.1
XLNet ($L$=128, $m$=0) | 79.4 | 39.8 | 53.0 | 69.1 | 30.1 | 41.9 | 61.6 | 32.7 | 42.7 | 45.9
XLNet ($L$=128, $m$=128) | 78.1 | 49.1 | 60.3 | 66.0 | 39.4 | 49.3 | 63.8 | 38.8 | 48.2 | 52.6
XLNet ($L$=256, $m$=0) | 78.8 | 59.5 | 67.8 | 68.0 | 48.6 | 56.7 | 66.4 | 47.9 | 55.7 | 60.1
XLNet ($L$=256, $m$=256) | 64.6 | 67.5 | 66.0 | 48.4 | 55.0 | 51.5 | 59.8 | 45.3 | 51.6 | 56.4
XLNet ($L$=512, $m$=0) | 80.3 | 71.7 | 75.7 | 70.7 | 61.4 | 65.7 | 66.9 | 60.0 | 63.3 | 68.2
XLNet ($L$=512, $m$=512) | 76.2 | 73.0 | 74.6 | 64.2 | 63.8 | 64.0 | 66.4 | 58.3 | 62.1 | 66.9
Table 8: Same as table 7 but with the start2end model.