# Evaluation Discrepancy Discovery: A Sentence Compression Case-study

Yevgeniy Puzikov
Ubiquitous Knowledge Processing Lab (UKP Lab),
Department of Computer Science, Technical University of Darmstadt
https://www.ukp.tu-darmstadt.de Research done during an internship at
Bloomberg L.P., London, United Kingdom.

###### Abstract

Reliable evaluation protocols are of utmost importance for reproducible NLP
research. In this work, we show that sometimes neither metric nor conventional
human evaluation is sufficient to draw conclusions about system performance.
Using sentence compression as an example task, we demonstrate how a system can
game a well-established dataset to achieve state-of-the-art results. In
contrast with the results reported in previous work that _showed correlation_
between human judgements and metric scores, our manual analysis of state-of-
the-art system outputs demonstrates that high metric scores _may only indicate
a better fit to the data_ , but not better outputs, as perceived by humans.
The prediction and error analysis files are publicly released.
111https://github.com/UKPLab/arxiv2021-evaluation-discrepancy-nsc

##  1 Introduction

###  1.1 Task description

Sentence compression is a Natural Language Processing (NLP) task in which a
system produces a concise summary of a given sentence, while preserving the
grammaticality and the important content of the original input. Both
abstractive [Cohn and Lapata, 2008, Rush et al., 2015] and extractive
[Filippova and Altun, 2013, Filippova et al., 2015, Wang et al., 2017, Zhao et
al., 2018] approaches have been proposed to tackle this problem. Most
researchers have focused on the extractive methods, which treat this as a
deletion-based task where each compression is a subsequence of tokens from its
original sentence (Figure 1).

Dickinson, who competed in triple jump at the 1936 Berlin Games, was also a
bronze medalist in both the long jump and triple jump at the 1938 Empire
Games.

(a) Input sentence

Dickinson competed in triple jump at the 1936 Berlin Games.

(b) Reference compression

Dickinson was a bronze medalist in the long jump and triple jump at the 1938
Empire Games.

(c) Another possible compression

Figure 1: Sentence compression example from the Google Dataset: an input
sentence and a reference compression. The compression candidate at the bottom
is also a valid one, but would score low, because the n-gram overlap with the
reference is small.

In the past few years several novel methods have been proposed to tackle the
task of sentence compression. Most of these methods have been evaluated using
the Google Dataset [Filippova and Altun, 2013] or its derivatives. Most
authors present approaches that show better metric scores; a few of them also
describe human evaluation experiments and show that the proposed methods
outperform previous work. However, there has been a serious lack of analysis
done on actual model predictions. In this work we show that metric scores
obtained on the Google Dataset might be misleading; a closer look at model
predictions reveals a considerable amount of noise, which renders the trained
model predictions ungrammatical. Another problem is that valid system outputs
which do not match the references are severely penalized. For example, a
plausible compression for the introductory example we used above would be
Dickinson was a bronze medalist in the long jump and triple jump at the 1938
Empire Games. However, with the established evaluation protocol, this
compression would score very low because of the insignificant token overlap
with the reference. We showed that evaluating a system on the Google Dataset
is tricky, and even human evaluation done in the previous years could not
detect the issues described in this chapter.

To summarize, our contributions in this study are:

  * •

We introduce a simple method of sentence compression that established new
state-of-the-art results, as measured by common metrics.

  * •

We design an experiment with a contrived system which achieved even higher
scores, but produced less grammatical and less informative outputs.

  * •

We show that this discrepancy may be attributed to the noise in the dataset.

##  2 Data Analysis

In our experiments we use the Google Dataset introduced by ?)
222https://bit.ly/2ZvTK9z This dataset was constructed automatically by
collecting English news articles from the Internet, treating the first
sentence of each article as an uncompressed sentence and creating its
extractive compression using a set of heuristics and the headline of the
article. The dataset contains 200 000 training and 10 000 evaluation
instances; the first 1 000 data points from the latter are commonly used as a
test set and the remaining 9 000 as a development set.

Exploratory data analysis showed that the distribution of the training data is
highly skewed, which is not surprising though, given the nature of the data.

(a) Sentence length

(b) Reference length

(c) Token length

(d) Compression ratio (CR) values

Figure 2: Data analysis of the Google Dataset: length distributions of
sentences, ground-truth compressions and tokens, and the distribution of
compression ratio values. The numbers to the right of each box denote median
values.

In order to remove outliers and fit the computation budget, we removed
instances which contained sentences longer than 50 tokens and compressions
longer than 17 tokens. We also removed examples with tokens longer than 15
characters, since those in most cases denoted website links. Finally, we
excluded cases with a compression ratio of more than 0.85 — those rare cases
in most cases were too long to qualify as compressions. Evaluation on the
development and test sets was done without any data filtering.

##  3 BERT-based Sentence Compression

Most modern deletion-based compression systems adopt either a tree-pruning, or
a sequence labeling approach. The former uses syntactic information to
navigate over a syntactic tree of a sentence and decide which parts of it to
remove [Knight and Marcu, 2000, McDonald, 2006, Filippova and Altun, 2013].
With the advent of sequence-to-sequence models it became possible to skip the
syntactic parsing step and solve the task directly, by processing a sentence
one token at a time and making binary decisions as to whether to keep a token
or delete it [Filippova et al., 2015, Wang et al., 2017, Zhao et al., 2018,
Kamigaito and Okumura, 2020]. The advantages of such approaches include a
lesser chance of introducing error propagation from incorrect parsing
decisions, as well as higher training and inference speed.

For a long time the space of sequence-to-sequence models has been dominated by
different variants of Recurrent Neural Networks (RNN) [Rumelhart et al.,
1986]. However, a more recent Transformer architecture [Vaswani et al., 2017]
has shown very promising results in many NLP tasks. Given the success of
Bidirectional Encoder Representations from Transformers (BERT) [Devlin et al.,
2019], and the fact that there has been no empirical evaluation of its
performance in sentence compression, we decided to fill this gap and find out
how well BERT-based models would cope with the task.

We used pretrained BERT-base-uncased model weights
333https://huggingface.co/bert-base-uncased provided by the HuggingFace
library [Wolf et al., 2020], and implemented a simple BertUni model which
encodes the source sentence $S=\\{w_{1},w_{2},\dots w_{n}\\}$ and produces a
sequence of vectors $V=\\{v_{1},v_{2},\dots v_{n}\\},v_{i}\in\mathbb{R}^{h}$.
Each vector is fed into a dense layer with a logistic function as a non-linear
function to produce a score $s_{i}\in[0,1]$ (Figure 3). If $s_{i}\geq 0.5$,
the model output is 1 (and 0, otherwise).

Figure 3: Schematic of the BertUni architecture.

A simple decision rule was used to make a binary prediction:

$decision=\begin{cases}\text{1},&\text{if }\mathit{s}\geq 0.5,\\\
\text{0},&\text{otherwise}.\end{cases}$

##  4 Experiments

###  4.1 Automatic Metric Evaluation

For automatic evaluation of sentence compression systems, most researchers
follow ?; ?) and use the following two metrics:

  * •

F1-score: harmonic mean of the recall and precision in terms of tokens kept in
the target and the generated compressions.

  * •

Compression ratio (CR): the length of the compression divided over the
original sentence length.

The former metric shows how close the model outputs are to the references. The
latter one is supposed to measure the compression effectiveness.

To make our results comparable to previous work, in our experiments we
followed the same convention. However, we would like to note that _measuring
CR in sentence compression might be redundant_ for several reasons. The first
reason comes from the fact that data-driven sentence compressors are likely to
produce outputs with a compression ratio most commonly seen in the training
data references. In other words, CR is less a property of a system and more a
characteristic of the dataset. This is supported by the fact that most models
reported in the literature have the same compression ratio (in the range of
0.38–0.43, see Table 1).

Secondly, it is not even clear how to treat compression ratio values: is a CR
of 0.4 better than a CR of 0.5? Intuitively, yes, because it means a more
concise compression. However, the compression is really better only if it
retained more valuable information from the source. On the other side,
defining the notion of informativeness/importance in sentence compression (and
document summarization, in general) is an open problem and currently is not
measured automatically. This means that the CR metric is a very one-sided
proxy, too crude to be used for real automatic evaluation without balancing it
with some recall-oriented metric.

To put the evaluation of our approach into better context, we compare it with
the following systems. All systems predict a sequence of binary labels which
decide which tokens to keep or remove from the input sentence.

##### LSTM.

?) use a three-layer uni-directional Long Short-term Memory (LSTM) network
[Hochreiter and Schmidhuber, 1997] and pretrained word2vec [Mikolov et al.,
2013] embeddings as input representations. For comparison, we use the results
for the best configuration reported in the paper (LSTM-PAR-PRES). This system
parses the input sentence into a dependency tree, encodes the tree structure
and passes the aggregated feature representations to the decoder LSTM. Unlike
our approach, this system relies on beam search at inference time.

##### BiLSTM.

?) build upon LSTM approach, but introduces several modifications. It employs
a bi-directional LSTM encoder and enriches the feature representation with
syntactic context. In addition, it uses Integer Linear Programming (ILP)
methods to enforce explicit constraints on the syntactic structure and
sentence length of the output.

##### Evaluator-LM.

?) uses a bi-directional RNN to encode the input sentence and predict a binary
label for each input token. In addition to token embeddings, the network uses
vector representations of part-of-speech (POS) tags and dependency relations.
The system is trained using the REINFORCE algorithm [Williams, 1992], the
reward signal comes from a pretrained syntax-based language model (LM).

##### SLAHAN.

?) propose a modular sequence-to-sequence model that consists of several
components. The system encodes a sequence of tokens using a combination of
pretrained embeddings (Glove [Pennington et al., 2014], ELMO [Peters et al.,
2018], BERT) and parses the input into a dependency graph. Three attention
modules are employed to encode the relations in the graph, their weighted sum
is passed to a selective gate. The output of the latter forms an input to a
LSTM decoder.

Despite its simplicity, the proposed BERT-based approach achieved very
competitive scores (Table 1).

Model | F1$\uparrow$ | CR$\downarrow$
---|---|---
Evaluator-LM | 0.851 | 0.39
BiLSTM | 0.800 | 0.43
LSTM | 0.820 | 0.38
SLAHAN | 0.855 | 0.407
BertUni |  0.857 $\pm 0.002$ | 0.413 $\pm 0.004$
BertUni (dev) |  0.860 $\pm 0.001$ | 0.418 $\pm 0.004$
Table 1: The performance of the BertUni model on the test portion of the
Google Dataset, compared to recent approaches. The last row shows BertUni’s
performance on the development set.

Comparing single performance scores (and not score distributions) of neural
approaches is meaningless, because training neural models is non-deterministic
in many aspects and depends on random weight initialization, random shuffling
of the training data for each epoch, applying random dropout masks [Reimers
and Gurevych, 2017]. This makes it hard to compare the scores reported in
previous works and our approach. To facilitate a fair comparison with future
systems, we report the mean and standard deviation of the BertUni scores
averaged across ten runs with different random seeds.

In order to understand where BertUni fails and what we could potentially
improve upon, we conducted manual error analysis of its predictions.

###  4.2 Error analysis

The purpose of error analysis is to find weak spots of a system, from the
point of view of human evaluation. In sentence compression, previous work
typically analyzed system predictions of the first 200 sentences of the test
set, using a 5-point Likert scale to assess annotators’ opinions of the
compressions’ _readability_ and _informativeness_ [Filippova et al., 2015].

Since error analysis is used for further system improvement and test sets
should be used only for final evaluation, we perform error analysis on the
development set. In order to do that, we retrieved BertUni’s predictions on
the 200 dev set sentences which received the lowest F1 scores and manually
examined them. Note that those are not random samples; the reason why we chose
worst predictions is because we know that the system performed poorly on them.

As for the quality criteria, we had to make certain adjustments. ?) mention
that readability _covers the grammatical correctness, comprehensibility and
fluency of the output_ , while informativeness measures _the amount of
important content preserved in the compression_. In our opinion, merging
several criteria into one synthetic index is a bad idea, because annotators
can’t easily decide on the exact facet of evaluation. Given that there already
exists a problem of distinguishing fluency and grammaticality, adding both of
them to assess readability seems to be a bad design decision. The problem is
aggravated by the fact that readability as a text quality criterion is already
used by NLP researchers for estimating the _text complexity_ from a reader’s
point of view [Vajjala and Meurers, 2012, Štajner and Saggion, 2013, Venturi
et al., 2015, De Clercq and Hoste, 2016]. This made us conclude that
readability is another overloaded criterion. Instead, we chose
_grammaticality_ as the first quality criterion.

We manually analyzed BertUni predictions on the 200 aforementioned samples,
trying to identify common error patterns. The results are presented below.

##### Grammaticality.

Out of 200 compressions, 146 (73 %) were deemed to be grammatical. The errors
in the remaining instances have been classified into several groups (marked
with _G_ in Figure 4a).

(a) BertUni errors

(b) BertBi-TF errors

(c) Ground-truth errors

Figure 4: Number of errors made by the evaluated approaches on the 200
development set instances where BertUni achieved the lowest F1 scores, as well
as errors found in ground-truth compressions. Error types marked with _G_ are
_grammaticality_ flaws; the remaining ones are errors of _informativeness_.

Most of them were cases where grammatical clauses miss linking words, are
_stitched_ together, making the output ungrammatical, as in the following
compressions:

  * •

I ’m said It ’s not Kitty Pryde superhero is the leader of the X-Men .

  * •

He first Postal Vote result can be announced before 10PM .

Another large error category was _finish_ : the compression was grammatical
until the last retained token, where the sentence ended abruptly, rendering
the compression incomplete:

  * •

Activision Blizzard has confirmed some new statistics for its games including
.

  * •

The South Sydney star had no case to .

A few system outputs incorrectly started with a relative or demonstrative
pronoun. This happened when the system failed to retain parts of main clause
of the sentence (_rd-pron_):

  * •

That shows young people rapping while flashing cash and a handgun in a public
park .

Finally, one output missed a verb which was essential for ensuring
grammaticality (_verb-miss_):

  * •

People giant waves crash against the railway line and buildings at Dawlish .

##### Informativeness.

Out of 200 compressions, 105 (52.5 %) were deemed to be informative, the
errors in the remaining instances have been classified into several groups
(marked with _I_ in Figure 4a). Most of these erroneous cases were
compressions which missed certain information that was needed for
understanding the context (_info-miss_). For example:

  * •

Dolly Bindra filed a case .

  * •

Mount Hope became the third largest city .

A smaller, but still a large group of compressions started with unresolved
personal pronouns, which made it hard to understand the subject (_p-pron_):

  * •

She hopes her album Britney Jean will inspire people .

  * •

He should be allowed to work freely till proven guilty .

In some cases, omitting the context caused a change in the meaning of the
sentence (_mean-change_). For example:

  * •

Reference: […] Aleksandar Vucic […] voiced hope that Germany will give even
stronger support to Serbia […]

  * •

System: Aleksandar spoke Germany will give stronger support to Serbia .

A large number of both grammatical and informative compressions did not match
references (_I2_). Interestingly enough, in some cases the system outputs were
better then the references:

  * •

Reference: We saw their two and raised to three.

  * •

System: Newport beat Hartlepool 2 0 .

  * •

Reference: Who joins for the remainder of the season subject .

  * •

System: Watford have announced the signing of Lucas Neill .

More examples of compression errors are provided in Section A.1.

##  5 Evaluation Discrepancy

When assessing the sentence compressions, we needed to compare system outputs
with references. Manual examination revealed that many references themselves
were flawed. This, in turn, meant that noise is inherent to the Google
Dataset, and metric-based improvements on this data are misleading. To
corroborate this claim, we conducted two experiments: the first tested the
capacity of a more accurate system to ignore the noise and output compressions
of better quality. In the second, we verified whether the noise came from the
ground-truth data and attempted to quantify it.

At first, we decided to implement more complex models that could potentially
achieve better scores. We attempted to improve the grammatical quality of
BertUni compressions by using the history of model predictions for making more
informed decisions.

We impelented and tested models that use BERT-encoded lastly-retained tokens
at each prediction step as an additional input to the model (prediction
history), similar to n-gram language models. As a history, BertBi and BertTri
used one and two previously predicted tokens, respectively. BertBiSS and
BertTriSS were the same as BertBi and BertTri, but used scheduled sampling
training scheme to mitigate the exposure bias issue [Bengio et al., 2015].

According to the metric evaluation results, none of the more complex models
outperformed BertUni (Table 2).

Model | F1$\uparrow$ | CR$\downarrow$
---|---|---
BertUni |  0.860 $\pm 0.001$ | 0.418 $\pm 0.004$
BertBi | 0.849 $\pm 0.001$ | 0.423 $\pm 0.005$
BertBiSS | 0.840 $\pm 0.003$ |  0.370 $\pm 0.005$
BertTri | 0.847 $\pm 0.002$ | 0.423 $\pm 0.007$
BertTriSS | 0.843 $\pm 0.003$ | 0.382 $\pm 0.006$
BertBi-TF | 0.901 | 0.423
Table 2: BERT-based model variants’ performance on the development set (mean
and standard deviation across ten random seed values). BertBi-TF was run only
once, since it is a “cheating” model that is not meant to be used in
production.

We used an unrealistic scenario and artificially made it easier for the model
to make correct predictions. We trained a BertBi-TF model which builds upon
BertBi, but at prediction time for history instead of model predictions uses
ground-truth labels 444We call this model BertBi-TF, since it builds upon
BertBi, but uses teacher forcing (TF) both at training and prediction time..
The development set result of BertBi-TF was an F1 score of 0.901, a 4-point
improvement over BertUni. We retrieved this model’s predictions for the same
200 dev set sentences used for the error analysis of BertUni outputs, and
manually examined them. The usual evaluation practice is to draw samples
randomly, in order to not give an advantage to any system and not to bias the
evaluation. However, in this work we approached the problem from a system-
development perspective and attempted to assess the comparative performance of
the approaches in the _worst-case_ scenario. If such a comparison is biased,
then only in favor of BertBi-TF, because the drawn samples were the worst ones
for BertUni, not BertBi-TF. We view this as sanity step, a regression test to
ensure that the newer version of the system performs at least as well as the
baseline on the challenging cases.

We assessed BertBi-TF outputs from the same aspects of _grammaticality_ and
_informativeness_ , as described in Section 4.2.

##### Grammaticality.

Out of 200 compressions, only 44 (22 %) were found to be grammatical; we
classified the errors in the remaining instances into groups (marked with _G_
in Figure 4b). The first and most prevalent is the already mentioned _stitch_
group which comprises around 80 % of all grammatical errors:

  * •

The program has received FBS college game 2014 season .

  * •

Tskhinvali region with Russia .

The remaining errors are faulty compression endings (_finish_):

  * •

The fine has been described as a slap on the .

  * •

P Chidambaram sought .

##### Informativeness.

A similar situation was observed when assessing the compressions’
informativeness — only 41 (20.5 %) instances were considered as correct. The
distribution of errors (marked with _I_ in Figure 4b) indicates that more than
80 % of cases miss information by omitting important words:

  * •

Dickinson was a .

  * •

Wynalda is mixing .

A smaller fraction of errors was comprised by the cases with unresolved
personal pronouns:

  * •

He is an education .

  * •

It would win 45 to 55 seats in Odisha .

The remaining errors were the cases where the system compressions changed the
semantics of the input:

  * •

Sentence: 612 ABC Mornings intern Saskia Edwards hit the streets of Brisbane
to find out what frustrates you about other people.

  * •

System: Saskia frustrates people .

More examples of BertBi-TF errors are provided in Section A.2.

We counted the cases in which predictions of BertBi-TF had better or worse
quality, compared to BertUni. In terms of informativeness, BertBi-TF improved
15 and worsened 78 instances; in terms of grammaticality, 115 instances were
perceived as less grammatical, versus only 13 improved cases, which makes it
clear that BertBi-TF makes many more mistakes than BertUni, despite the higher
metric scores.

In order to verify our findings, we examined the ground-truth compressions in
more detail. Only 63 (31.5 %) of these compressions were both grammatical and
informative. Figure 4c shows a visualization of the error type distribution.
We provide examples of noisy ground-truth compressions in Section A.3.

The abundant errors related to the use of pronouns in the compressions were
predominantly caused by the fact that many instances contained ground-truth
compressions with unresolved pronouns; cleaning the data would likely result
in better outputs.

The _stitch_ , _finish_ and _info_miss_ errors can be attributed to the fact
that many references have missing information or artifacts remaining from the
automatic procedure that was used to create these compressions [Filippova and
Altun, 2013]. Resolving these issues may require more elaborate strategies,
beyond simple text substitution.

##  6 Discussion

In this study we advanced the state-of-the-art for the task of sentence
compression, and achieved that by designing a simple, but effective sequence
labeling system based on the Transformer neural network architecture. While
the proposed approach achieved the highest scores reported in the research
literature, the main message of the study is not a higher score — it is the
idea that NLP system evaluation might need to go beyond simple comparison of
metric scores with human judgements.

We found that a higher-scoring system can produce worse-quality outputs. We
further provided some empirical evidence that this issue is caused by the
noise in the training data. We call this finding a _discrepancy discovery_ ,
because existent sentence compression work does not explain our results, based
on the established evaluation practices. The research papers we analyzed
present automatic and human evaluation statistics that seem to overlook the
data quality issue. Of course, the approaches proposed so far could still
produce high-quality sentence compressions, but the absence of error analysis
plants a seed of doubt into the reader. In this work, we question not the
reported results, but the principles of the conventional evaluation workflow.

None of the examined research papers drew attention to the quality of the
data, even though it is known that the dataset was constructed automatically,
and therefore should contain noisy examples, which should affect the output
quality of any data-driven system. Previous work also overlooked the use of
the compression ratio which seems to be too simplistic to call it a metric
that measures the compression effectiveness. Finally, the employed sentence
compression evaluation protocols do not assume having multiple references. We
did not go into much detail about this issue, but provided an illustration at
the beginning of the paper (the Dickson example). The space of possible
compressions in deletion-based sentence compression is bound by sentence
length. But because the definition of importance is left out, the candidate
space is very large. The existence of only one reference brings additional
requirements for evaluation metrics to work, and commonly used n-gram overlap
metrics clearly do not satisfy these requirements.

##  7 Conclusion

The presented results show that system output analysis is indispensable when
assessing the quality of NLP systems. It is a well-established fact that
metric scores used for system development do not always reflect the actual
quality of a system; usually this is revealed via human evaluation
experiments. However, in our case study of automatic sentence compression we
have discovered that they might not be sufficient. Further investigation is
needed to make stronger claims; the study’s findings are yet to be confirmed
for other datasets and, perhaps, tasks.

## Acknowledgments

We thank Andy Almonte, Joshua Bambrick, Minh Duc Nguyen, Vittorio Selo, Umut
Topkara and Minjie Xu for their support and insightful comments.

## References

  * [Bengio et al., 2015]  Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer.  2015\.  Scheduled sampling for sequence prediction with recurrent neural networks.  In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1171–1179, Cambridge, MA, USA. MIT Press.
  * [Clarke and Lapata, 2006]  James Clarke and Mirella Lapata.  2006\.  Models for sentence compression: a comparison across domains, training requirements and evaluation measures.  In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 377–384, Sydney, Australia, July. Association for Computational Linguistics.
  * [Cohn and Lapata, 2008]  Trevor Cohn and Mirella Lapata.  2008\.  Sentence compression beyond word deletion.  In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 137–144, Manchester, UK, August. Coling 2008 Organizing Committee.
  * [De Clercq and Hoste, 2016]  Orphée De Clercq and Véronique Hoste.  2016\.  All mixed up? Finding the optimal feature set for general readability prediction and its application to English and Dutch.  Computational Linguistics, 42(3):457–490, September.
  * [Devlin et al., 2019]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * [Filippova and Altun, 2013]  Katja Filippova and Yasemin Altun.  2013\.  Overcoming the lack of parallel data in sentence compression.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1481–1491, Seattle, Washington, USA, October. Association for Computational Linguistics.
  * [Filippova et al., 2015]  Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals.  2015\.  Sentence compression by deletion with LSTMs.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 360–368, Lisbon, Portugal, September. Association for Computational Linguistics.
  * [Hochreiter and Schmidhuber, 1997]  Sepp Hochreiter and Jürgen Schmidhuber.  1997\.  Long Short-term Memory.  Neural Computation, 9(8):1735–1780.
  * [Kamigaito and Okumura, 2020]  Hidetaka Kamigaito and Manabu Okumura.  2020\.  Syntactically look-ahead attention network for sentence compression.  ArXiv, abs/2002.01145.
  * [Knight and Marcu, 2000]  Kevin Knight and Daniel Marcu.  2000\.  Statistics-based summarization - step one: Sentence compression.  In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, page 703–710. AAAI Press.
  * [McDonald, 2006]  Ryan McDonald.  2006\.  Discriminative sentence compression with soft syntactic evidence.  In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy, April. Association for Computational Linguistics.
  * [Mikolov et al., 2013]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc.
  * [Pennington et al., 2014]  Jeffrey Pennington, Richard Socher, and Christopher Manning.  2014\.  GloVe: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics.
  * [Peters et al., 2018]  Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  2018\.  Deep contextualized word representations.  CoRR, abs/1802.05365.
  * [Reimers and Gurevych, 2017]  Nils Reimers and Iryna Gurevych.  2017\.  Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 338–348, Copenhagen, Denmark. Association for Computational Linguistics.
  * [Rumelhart et al., 1986]  David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.  1986\.  Learning representations by back-propagating errors.  Nature, 323(6088):533–536, October.
  * [Rush et al., 2015]  Alexander M. Rush, Sumit Chopra, and Jason Weston.  2015\.  A neural attention model for abstractive sentence summarization.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal, September. Association for Computational Linguistics.
  * [Štajner and Saggion, 2013]  Sanja Štajner and Horacio Saggion.  2013\.  Readability indices for automatic evaluation of text simplification systems: a feasibility study for Spanish.  In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 374–382, Nagoya, Japan, October. Asian Federation of Natural Language Processing.
  * [Vajjala and Meurers, 2012]  Sowmya Vajjala and Detmar Meurers.  2012\.  On improving the accuracy of readability classification using insights from second language acquisition.  In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, NAACL HLT ’12, page 163–173, USA. Association for Computational Linguistics.
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  CoRR, abs/1706.03762.
  * [Venturi et al., 2015]  Giulia Venturi, Tommaso Bellandi, Felice Dell’Orletta, and Simonetta Montemagni.  2015\.  NLP–based readability assessment of health–related texts: a case study on Italian informed consent forms.  In Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis, pages 131–141, Lisbon, Portugal, September. Association for Computational Linguistics.
  * [Wang et al., 2017]  Liangguo Wang, Jing Jiang, Hai Leong Chieu, Chen Hui Ong, Dandan Song, and Lejian Liao.  2017\.  Can syntax help? Improving an LSTM-based sentence compression model for new domains.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1385–1393, Vancouver, Canada, July. Association for Computational Linguistics.
  * [Williams, 1992]  Ronald J. Williams.  1992\.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine Learning, 8(3–4):229–256, May.
  * [Wolf et al., 2020]  Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  2020\.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October. Association for Computational Linguistics.
  * [Zhao et al., 2018]  Yang Zhao, Zhiyuan Luo, and Akiko Aizawa.  2018\.  A language model based evaluator for sentence compression.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 170–175, Melbourne, Australia, July. Association for Computational Linguistics.

##  Appendix A Error Examples

This section contains error examples of BertUni and BertBi-TF models, as well
as errors in the gold standard. We publicly share the prediction and error
analysis files at https://github.com/UKPLab/arxiv2021-evaluation-discrepancy-
nsc.

##### Grammaticality.

Error types:

  * •

_finish_ : incomplete sentence with an abrupt ending, caused by omitting the
last token(s);

  * •

_stitch_ : grammatical clauses with missing linking words, as if they are
stitched together, which renders the sentence ungrammatical;

  * •

_rd-pron_ : incorrect sentence start with a relative or demonstrative pronoun;

  * •

_verb-miss_ : missing a verb which is essential for ensuring the
grammaticality of a sentence.

##### Informativeness.

Error types:

  * •

_info-miss_ : missing certain information that is needed for understanding the
context;

  * •

_p-pron_ : starting with an unresolved personal pronoun, which makes it hard
to understand the subject;

  * •

_mean-change_ : omitting the context causing a change of the meaning of the
sentence.

We also provide examples of alternative compressions which focus on parts of
the input sentence, which are different from those present in the reference
compressions. They are marked with an _I2_ label and are listed here, together
with erroneous cases, because due to mismatches with references they lower the
metric scores of the evaluated approaches.

###  A.1 BertUni

#### Grammatical Errors

Table 3: Manual error analysis results: examples of _ungrammatical_ outputs of BertUni. Error |  Sentence |  Compression
---|---|---
finish |  A young cow trying to grab a cooling drink from a river in Hampshire in the hot weather had to be rescued by firefighters when it got stuck in the mud. |  A cow trying to grab a drink had rescued .
|  Japan and the US reaffirmed Monday during a meeting in Tokyo with visiting Under Secretary for Terrorism and Financial Intelligence David S Cohen their ongoing cooperation on sanctions against Iran, a Japanese government source said Tuesday. |  Japan and the US reaffirmed .
|  612 ABC Mornings intern Saskia Edwards hit the streets of Brisbane to find out what frustrates you about other people. |  Saskia Edwards hit .
|  Google Helpouts, which was launched this week, is a service that allows you to pay for brief one-on-one webcam master classes with a range of experts in various fields. |  Google Helpouts is .
rd-pron |  Some parents and others in Bessemer City are complaining about a YouTube video that shows young people rapping while flashing cash and a handgun in a public park. |  That shows young people rapping while flashing cash and a handgun in a public park .
|  “In my district, already reeling from the shutdown of our largest private employer, the highest energy costs in the country, and reduced government revenues, this shutdown, if it continues any longer, can be the final nail in our economic coffin”, said Rep. Christensen. |  This shutdown can be the final nail in our economic coffin .
stitch |  Our obsession with Sachin Tendulkar and records has made us lose perspective to such an extent that what should have been widely condemned is being conveniently ignored. |  Should have been condemned is .
|  Direct Link Pisegna JR. Without will amoxicillin work for a uti this sweet little boy told his best friend that he loved him. |  boy told his he loved .
|  Visitors will find a mixture of old and new at Silver Springs State Park, which opened Tuesday near Ocala. |  will find and at Silver Springs State Park opened .
|  Twitter took the first step toward its IPO by filing with US regulators on Thursday, September 12, 2013. |  Twitter took by filing with US regulators .
|  Arsenio Hall lost control of his brand new Porsche Cayenne S and crashed his car on Monday night in El Lay! |  Arsenio Hall lost his and crashed his car in El Lay
verb-miss |  People watch giant waves crash against the already damaged railway line and buildings at Dawlish during storms in south west England February 8, 2014. |  People giant waves crash against the railway line and buildings at Dawlish .

#### Informativeness Errors

Table 4: Manual error analysis results: examples of _uninformative_ outputs of BertUni. We also show examples of alternative compressions (_I2_) which deviate from ground-truth compressions, but cannot be considered as errors. Error |  Sentence |  Compression
---|---|---
info-miss |  Buyers should beware, even though there was a safety recall on some GM cars those vehicles are still being sold on Craigslist. |  Buyers should beware .
|  New results show some improvement in test scores and graduation rates for local students, but experts say there’s still more work to be done. |  There ’s still more work to be done .
|  Counting of postal votes have already commenced while Elections Commissioner Mahinda Deshapriya stated that he first Postal Vote result can be announced before 10PM. |  He first Postal Vote result can be announced before 10PM .
|  When President Obama was elected in 2008 for his first term, he made a presidential decision that he would not give up his blackberry. |  He would not give up his blackberry .
mean-change |  On this week’s “Hostages” season 1, episode 13: “Fight or Flight,” Ellen reveals to Duncan that she will not kill the President but will help him get what he needs, as long as he gives her something in return. |  Ellen reveals she will not kill the President .
|  Serbia’s First Deputy Prime Minister Aleksandar Vucic spoke in Germany with former German chancellor Helmut Kohl about Serbia’s path towards the EU and its economic recovery/ During the talks, Vucic highlighted the important role of German investors in Serbia and voiced hope that Germany will give even stronger support to Serbia in the realisation of its European goals. |  Aleksandar spoke Germany will give stronger support to Serbia .
p-pron |  “Being a five-time champion, he knows how to handle pressure. Anand generally puts a lid on his emotions.” |  He knows how to handle pressure
|  Chelsea manager Jose Mourinho has made light of the managerial instability under Roman Abramovich by admitting he’s trying to break his own record. |  He ’s trying to break his own record .
|  Katy Perry wasn’t lying when she said she had some “beautiful news to share,” because she is now the new face of COVERGIRL. |  She had some beautiful news to share .
I2 |  Provisur Technologies has entered into an agreement with Scanico in which Scanico has become Provisur’s global partner in commercial freezing technology. |  Reference: Scanico has become Provisur ’s global partner in commercial freezing technology .
|  |  System: Provisur Technologies has entered into an agreement with Scanico .
|  Davina McCall has undergone medical tests after fears she may be suffering from hypothermia after battling severe weather in a Sport Relief challenge. |  Reference: she may be suffering from hypothermia after battling in a Sport Relief challenge .
|  |  System: Davina McCall has undergone medical tests .

###  A.2 BertBi-TF

#### Grammatical Errors

Table 5: Manual error analysis results: examples of _ungrammatical_ outputs of BertBi-TF Error |  Sentence |  Compression
---|---|---
finish |  A UKIP candidate who is standing for election in Enfield Town has defended a tweet in which he said a black comedian “should emigrate to a black country.” |  UKIP candidate has defended a tweet he said a black comedian should
|  Wynalda, who was introduced as the Silverbacks’ new manager on Tuesday, is mixing a bit of Europe with a bit of Mexico with a bit of Silicon Valley in an approach that will eliminate the head-coaching position. |  Wynalda is mixing .
|  Cell C has announced new pre-paid and contract packages that offer unlimited calls to any network. |  Cell C has announced .
|  Radnor police received a report Sept. 3 from a cadet at Valley Forge Military Academy that another cadet struck him in the face. |  Radnor police received a report another .
|  Diego Forlan scored directly from a corner this weekend to help Internacional to a 3-2 win over Fluminense. |  Diego Forlan scored to help Internacional to .
|  This 1930s-built four bedroom detached seafront home in Worthing is immaculately presented and has been expertly modernised. |  This seafront home is .
stitch |  Buyers should beware, even though there was a safety recall on some GM cars those vehicles are still being sold on Craigslist. |  Buyers should beware are being sold on Craigslist .
|  Davina McCall has undergone medical tests after fears she may be suffering from hypothermia after battling severe weather in a Sport Relief challenge. |  Davina McCall has undergone tests be suffering from hypothermia .
|  New results show some improvement in test scores and graduation rates for local students, but experts say there’s still more work to be done. |  Show some improvement in test scores graduation rates students but there .
|  POLICE are looking for witnesses after a car was hit by a van which failed to stop on Friday, January 7. |  POLICE a .
|  Our obsession with Sachin Tendulkar and records has made us lose perspective to such an extent that what should have been widely condemned is being conveniently ignored. |  With Sachin Tendulkar has made lose perspective what .
|  Watford have this evening announced the signing of experienced defender Lucas Neill, who joins for the remainder of the season subject to international clearance. |  Watford announced the signing Lucas season .
. |  |

#### Informativeness Errors

Table 6: Manual error analysis results: examples of _uninformative_ outputs of BertBi-TF. We also show examples of alternative compressions (_I2_) which deviate from ground-truth compressions, but cannot be considered as errors. Error |  Sentence |  Compression
---|---|---
info-miss |  “Former President Mandela is still in a critical condition in hospital but shows sustained improvement,” President Jacob Zuma said in a statement. |  Mandela is still in a critical condition shows improvement Jacob .
|  This 1930s-built four bedroom detached seafront home in Worthing is immaculately presented and has been expertly modernised. |  This seafront home is .
|  ASI’s additional director general BR Mani said he was hopeful of Nalanda making it to the list, claiming that Nalanda was an important centre of art and culture even before the university came into being. |  He Nalanda was .
|  Tata Martino explained that after two months in Barcelona, he is starting to understand how things work at the club, assuring that there seems to be a need to create a weekly crisis regarding the team’s style of play. |  He crisis .
mean-change |  Provisur Technologies has entered into an agreement with Scanico in which Scanico has become Provisur’s global partner in commercial freezing technology. |  Provisur has become Provisur ’s partner in commercial freezing technology .
|  TV presenter Katie Piper shows off her baby daughter for the first time and said motherhood has given her “true fulfillment”. |  Piper shows off her baby daughter motherhood .
|  AFL powerbrokers yesterday met Collingwood officials over convicted criminal Marley Williams, who faces a possible club ban or even a league suspension. |  Powerbrokers met ban suspension .
|  British rapper Professor Green was arrested on suspicion of drink driving on Sunday after he called police to report a robbery. |  Professor Green was arrested on suspicion of report a robbery .
p-pron |  “Being a five-time champion, he knows how to handle pressure. Anand generally puts a lid on his emotions.” |  He puts a lid on his emotions
|  PersonalWeb® today announced that it has reached a confidential settlement in its lawsuit with NEXSAN®, to which Nexsan has licensed the Truename patents from PersonalWeb. |  It has reached patents PersonalWeb .
|  Britney Spears has said she hopes her latest album Britney Jean will inspire people and she wants to ’project positive energy out into the world. |  She hopes her album Britney will inspire .
I2 |  On this week’s “Hostages” season 1, episode 13: “Fight or Flight,” Ellen reveals to Duncan that she will not kill the President but will help him get what he needs, as long as he gives her something in return. |  Reference: On this week ’s Hostages season 1 episode 13 Fight or Flight Ellen reveals .
|  |  System: Episode Fight or Flight Ellen reveals she will not kill the President .
|  Blustery winds arrived in Gwinnett on Wednesday and brought with them lower temperatures that caused the National Weather Service to issue a freeze warning for the area. |  Reference: that caused the National Weather Service to issue a freeze warning .
|  |  System: Blustery winds arrived issue a freeze warning .
|  BJP and JD today welcomed the five-year jail term handed down to RJD chief Lalu Prasad in the fodder scam case, saying it would send out a message that the law will catch up with the corrupt, however influential they might be. |  Reference: The law will catch up with the corrupt influential .
|  |  System: BJP law will catch up with the corrupt .

###  A.3 Ground Truth

#### Grammatical Errors

Table 7: Manual error analysis results: examples of _grammatical errors_ in ground-truth compressions, sampled from 200 development set instances with lowest BertUni F1 scores). Error |  Sentence |  Compression
---|---|---
finish |  Police investigating the unexplained death of a man in Taupo say his van appears to have broken down. |  Police investigating the unexplained death say .
|  Akkineni Nageswara Rao was one of the Indian cinema’s stalwarts, who will be remembered for his rich contribution. |  Akkineni Nageswara Rao was one .
|  Mortgage fees are going up so where does Pa. |  Where does Pa .
|  Coffee chain Starbucks has said guns are no longer “welcome” in its US cafes, although it has stopped short of an outright ban. |  Starbucks has said guns are .
rd-pron |  Way back in May 2011, Google filed a patent application for eye tracking technology, which would allow it to charge advertisers on a ’pay per gaze’ basis. |  Which would allow it to charge advertisers on a pay per gaze basis .
|  POLICE are looking for witnesses after a car was hit by a van which failed to stop on Friday, January 7. |  Which failed to stop .
|  Tomorrow South Africa will celebrate the centenary of the Union Buildings in Pretoria which have recently been declared a national heritage site by the South African Heritage Resources Agency. |  Which have been declared a national heritage site .
|  In a press release, Patrick said Goldstein will be replaced by Rachel Kaprielian, who is currently the state’s registrar of motor vehicles. |  Who is the state ’s registrar .
stitch |  Iran wants to end the stand-off with global powers over its nuclear programme swiftly, but will not sacrifice its rights or interests for the sake of a solution, President Hassan Rouhani said on Friday. |  Iran wants but will not sacrifice its rights Hassan Rouhani said .
|  Maggie Rose sheds her innocence in her brand new music video for “Looking Back Now.” |  Maggie Rose sheds for Looking Back Now
|  The Muskingum University chapter of Omicron Delta Kappa has made a donation of more than $600 to the New Concord Food Pantry in an effort to give back to the community. |  The Muskingum University chapter of Omicron Delta Kappa has made in an effort to give back to the community .
|  Dolly Bindra filed a case on an unknown person for having threatened her at gun point today in Oshiwara, Mumbai. |  Dolly Bindra filed for having threatened her at gun point in Oshiwara Mumbai .
|  North Korean leader Kim Jong-un has met with the top military leaders and warned them of a grave situation and threatened a new nuclear test. |  Kim Jong-un has met and warned of a grave situation and threatened a nuclear test .

#### Informativeness Errors

Table 8: Manual error analysis results: examples of _informativeness_ errors in ground-truth compressions, sampled from 200 development set instances with lowest BertUni F1 scores). Error |  Sentence |  Compression
---|---|---
info-miss |  Some parents and others in Bessemer City are complaining about a YouTube video that shows young people rapping while flashing cash and a handgun in a public park. |  Some parents in Bessemer City are complaining about a video .
|  Tata Martino explained that after two months in Barcelona, he is starting to understand how things work at the club, assuring that there seems to be a need to create a weekly crisis regarding the team’s style of play. |  There seems to be a need to create a weekly crisis .
|  Nothing is ever left behind in a BREACHED performance as the loud rocking, heavy amp cranky band announce Toronto show dates since performing last October at Indie Week. |  Band announce Toronto show dates
|  Prime Minister Kevin Rudd has missed the deadline for an August 24 election, with his deputy saying “people should just chill out” about the election date. |  People should chill out about the election date .
p-pron |  Davina McCall has undergone medical tests after fears she may be suffering from hypothermia after battling severe weather in a Sport Relief challenge. |  She may be suffering from hypothermia after battling in a Sport Relief challenge .
|  TV presenter Nick Knowles has been the recipient of some unexpected abuse as a result of an announcement that he will not be present at the birth of his child. |  He will not be present at the birth .
|  England fast bowler James Anderson does not feel sorry for Australia and has said his team wants to win the Ashes 5-0. |  His team wants to win the Ashes 5 0 .
|  If he decides to run for president, New Jersey Gov. Chris Christie will need to push back against the inevitable pressure that he will encounter to move to the right. |  He will encounter to move to the right .
|  Armaan will be taken for a medical examination and post that he will be presented in the court today. |  He will be presented in the court .