#  ‘Keep it Together’: Enforcing Cohesion in Extractive Summaries
by Simulating Human Memory

Ronald Cardenas
University of Edinburgh
<EMAIL_ADDRESS>
&Matthias Gallé
Cohere
<EMAIL_ADDRESS>
Shay B. Cohen
University of Edinburgh
<EMAIL_ADDRESS>

###### Abstract

Extractive summaries are usually presented as lists of sentences with no
expected cohesion between them. In this paper, we aim to enforce cohesion
whilst controlling for informativeness and redundancy in summaries, in cases
where the input exhibits high redundancy. The pipeline controls for redundancy
in long inputs as it is consumed, and balances informativeness and cohesion
during sentence selection. Our sentence selector simulates human memory to
keep track of topics –modeled as lexical chains–, enforcing cohesive ties
between noun phrases. Across a variety of domains, our experiments revealed
that it is possible to extract highly cohesive summaries that nevertheless
read as informative to humans as summaries extracted by only accounting for
informativeness or redundancy. The extracted summaries exhibit smooth topic
transitions between sentences as signaled by lexical chains, with chains
spanning adjacent or near-adjacent sentences.

##  1 Introduction

Text summarization is the task of processing a document(s) and producing a
shorter text, the summary, that retains the gist of the information Nenkova et
al. (2011). Extractive summarization selects content units (usually sentences)
and presents their concatenation as the summary. It remains challenging to
select the appropriate content units so that the summary ends up non-redundant
and informative, with much previous work modeling these qualities during
document understanding Peyrard et al. (2017); Xiao and Carenini (2020); Gu et
al. (2022). Moreover, the modeling of summary coherence previously relied on
capturing discourse patterns in nearby sentences Barzilay and Lapata (2008);
Steen and Markert (2022); Zhao et al. (2023). Cohesion, a special case of
local coherence, relies on the explicit textualization of contextual
connections called cohesive ties, making a text read as a unified whole Hassan
et al. (1976).

We introduce an extractive summarization methodology that implements two
control mechanisms at different stages of processing: the first one to control
redundancy during input understanding, and the second one to control the
trade-off between informativeness and cohesion during summary extraction. When
building extractive summaries by concatenating sentences, we argue that
controlling for cohesion is a better-defined task than aiming to control
coherence, especially if no sort of post-editing (e.g. replacing discourse
markers) is applied Zajic et al. (2007); West et al. (2019); Mallinson et al.
(2020). A potential benefit of producing a more cohesive text is that it is
easier to read and understand for humans, especially when the knowledge domain
is highly technical, as reported by previous work in psycholinguistics Kintsch
(1990) and automatic summarization Barzilay and Elhadad (2002).

In our pipeline, summary properties are controlled in the following way. On
the one hand, summary redundancy is addressed by controlling the redundancy
levels of the input text, following previous findings Carbonell and Goldstein
(1998); Xiao and Carenini (2020). The pipeline consumes input text in a
cascaded way: first splitting the input into contiguous passages, then
consuming passages one at a time so as to minimize their semantic similarity
with already selected passages.

On the other hand, informativeness and cohesion are directly modeled during
summary extraction. Extraction is done in a sentence-by-sentence fashion,
quantifying summary properties independently at each step. The objective is to
select a highly cohesive sentence that is informative enough. We introduce a
sentence selector that incrementally builds cohesive chains of noun phrases
and models chain interaction. The selector, KvD-Select, keeps track of chains
currently active by simulating human memory according to the Micro-Macro
theory, henceforth KvD Kintsch and van Dijk (1978), a psycholinguistic theory
of discourse comprehension and production. Working memory –a type of short-
term memory– is modeled as a limited-capacity buffer of lexical chains,
forcing the model to keep only the most salient chains.

We test our methodology on newswire multi-document summarization and single-
long document summarization of scientific articles, patents, and government
reports. Across domains, extensive experiments show that, first, our system is
effective at incrementally building an input sequence with lower content
redundancy, which translated to a significant reduction in summary redundancy.
Second, the proposed sentence selector managed to maintain summaries
informative while improving cohesion significantly: over 15% more noun phrases
and over 20% more sentences were connected through cohesive ties w.r.t a
greedy selector. Tailored human evaluation campaigns revealed that cohesion
has a positive impact on perceived informativeness, and that our extracted
summaries exhibit chains covering adjacent or near-adjacent sentences. Closer
inspection showed that topics flow smoothly across extracted summaries with no
abrupt change or jumps.

In summary, our contributions are as follows:

  * •

We propose a cascaded encoder capable of consuming arbitrary long textual
input that controls the level of content redundancy the rest of the pipeline
is exposed to.

  * •

We propose a summary extraction method that models informativeness and
cohesion independently and allows to control the balance between the two when
building the summary.

  * •

Automatic and human experiments show the effectiveness of our control
mechanisms and how summary properties can be balanced according to user needs
in a straightforward way.

##  2 Related Work

Previous work modeled cohesion by keeping track of named entities Barzilay and
Lapata (2008); Guinaudeau and Strube (2013), topic flow Barzilay and Elhadad
(2002), or by implementing discourse theories Jeon and Strube (2020). Most
similar to our approach, Fang (2019) introduced an KvD implementation that
models cohesion and informativeness jointly during reading, assigns a single
importance score to each sentence, and employs a greedy sentence selector. In
contrast, we quantify summary properties separately, and model cohesion by
implementing KvD during sentence selection. This approach allows more explicit
control of the contribution of each property during summary extraction.

Similar ways to control summary properties during selection have focused only
on minimizing redundancy Carbonell and Goldstein (1998); Fabbri et al. (2019);
Xiao and Carenini (2020), where the extractive summary is regarded as a list
of sentences with no particular order to them, a design choice possibly
influenced by the format of available benchmarks such as CNN/DM Hermann et al.
(2015). However, seminal work highlighted the role of redundancy Walker
(1993); Tauste (1995), and how its presence is a result of human memory
limitations Johnstone (1994).

In this work, we provide evidence that controlling for cohesion constitutes a
better strategy for providing the end-user with a more comprehensible summary,
formatted as a multi-sentence cohesive text. Our results show that this setup
is especially effective when the knowledge domain is highly technical, and
when a sentence ordering cannot be inferred from the input trivially, e.g. in
multi-document summarization.

Previous work on passage selection focused on selecting salient passages
(paragraphs in the a document or documents in a cluster) from long and noisy
input by ranking them according to TF-IDF similarity with the article title
Liu et al. (2018) or with a learned ranker Liu and Lapata (2019). In contrast,
our redundancy control method operates in an unsupervised fashion by selecting
passages according to their relevance, measured by LexRank Erkan and Radev
(2004), and to their redundancy w.r.t. already selected passages.

##  3 Problem Formulation

We tackle the task of extractive summarization as a sentence scoring step
followed by a selection step. Figure 1 shows the pipeline of the system, in
which sentences are scored in a cascaded fashion, as follows. First, the input
is segmented into blocks of contiguous sentences to be selected based on their
relevancy and their redundancy w.r.t. already selected blocks. Then, a local
encoder obtains block-level representations for each sentence in the block.
After all document blocks are processed, the encodings are concatenated into a
single embedding sequence and passed to the global context encoder, which will
obtain a document-aware representation of each sentence. Finally, a selection
module will extract a subset of sentences and present them as the summary in
the order they were extracted. We now proceed to elaborate on each module of
the proposed pipeline.

Figure 1: Our extraction pipeline: local extraction step $m$ adds local
sentences to $D^{\prime}$; at sentence selection step $t$, KvD-Select balances
informativeness of candidate $s_{t}$ with cohesion of summary $\hat{S}$.

###  3.1 Block Segmentation and Selection

Processing starts by segmenting the document(s) $D$ into fixed-length
overlapping blocks, each of which includes preceding and subsequent
wordpieces, providing surrounding context. Then, blocks are selected
iteratively until a predefined budget (total number of wordpieces) is met. At
step $m$, block $b_{m}$ is selected such that

$b_{m}=\operatorname*{argmax}_{b\in
B\setminus\hat{B}}[\lambda_{b}\text{LR}(b)-(1-\lambda_{b})\max_{b_{j}\in\hat{B}}\text{Sim}(b,b_{j})]$
(1)

where $\hat{B}$ is the set of blocks already selected, $\text{Sim}(x,y)$ is
the cosine similarity between TF-IDF vectors of blocks $x$ and $y$, and
$\lambda_{b}$ allows to control the mix of both terms. $\text{LR}(b)$ is the
continuous LexRank score of block $b$ Erkan and Radev (2004), calculated over
the complete graph of blocks in $D$,

$\text{LR}(b)=\frac{d}{|B|}+(1-d)\sum_{v\in\text{adj}[b]}\frac{\text{Sim}(b,v)}{\sum_{z\in\text{adj}[v]}\text{Sim}(z,v)}\text{LR}(v)$
(2)

where $d$ is the damping factor and $\text{adj}(b)$ is the set of block nodes
adjacent to $b$. This module balances block relevancy (as proxied by
centrality) and input redundancy in a straightforward way by linearly
combining their scores. After an optimal block is selected, it is sent to the
local encoder module.

###  3.2 Local Encoder (LE)

Given block $b$ as a sequence of wordpieces spanning contiguous sentences, the
local encoder will obtain representations for each sentence covered in $b$.
This module is trained as a local extractive summarizer itself, under sequence
labeling formulation where each sentence in the block is labeled as
$y^{\ell}_{i}\in\\{0,1\\}$ to indicate whether sentence $s_{i}$ is selected or
not. Then, sentence representation $h_{i}$ is defined as the average embedding
over $s_{i}$ wordpieces, obtained from a LongT5 encoder Guo et al. (2022).
Finally, the probability of $s_{i}$ being locally selected is defined as
$P(y^{\ell}_{i}\mid s_{i},b;\theta_{\ell})=\sigma(W^{\ell}\cdot h_{i})$, and
the module is trained using cross-entropy loss independently from the rest of
the pipeline. During inference, the local encoder consumes one block, selects
$N$ sentences and adds them to $D^{\prime}$ –containing all locally selected
sentences so far–, and their corresponding embeddings to $H^{\ell}$.

###  3.3 Global Context Encoder (GCE)

Given the sequence of local sentence embeddings $H^{\ell}$, this module
obtains the sequence of globally-aware representations $H^{g}$ as follows.
Sequence $H^{\ell}$ is passed through a self-attention layer Vaswani et al.
(2017), i.e. $g_{t}=\text{SelfAttn}(h_{t},H^{\ell}),\forall h_{t}\in
H^{\ell}$. Similarly to the LE module, the probability of selecting $s_{t}\in
D^{\prime}$ is $P(y^{g}_{t}\mid s_{t},D^{\prime};\theta_{g})=\sigma(W^{g}\cdot
g_{t})$, where $y^{g}_{t}\in\\{0,1\\}$ indicates whether $s_{t}$ is selected
or not for the final, global summary, and also trained using cross-entropy
loss.

###  3.4 Sentence Selection

Finally, candidate summary $\hat{S}$ is built by selecting one sentence at a
time from $D^{\prime}$, taking into account the informativeness and cohesion
of each candidate sentence w.r.t. the already selected sentences. At selection
step $t$, the optimal sentence is given by

$s_{t}=\operatorname*{argmax}_{s\in
D^{\prime}\setminus\hat{S}^{t-1}}[\lambda_{\text{sel}}f_{\text{I}}(s)+(1-\lambda_{\text{sel}})f_{\text{C}}(\hat{S}^{t})]$
(3)

where function $f_{\text{I}}$ estimates the informativeness of candidate
sentence $s$, $f_{\text{C}}$ estimates the cohesion of candidate summary
$\hat{S}^{t}=[\hat{S}^{t-1};s]$, and $\lambda_{\text{sel}}\in[0,1]$ is a
parameter that allows to control their trade-off. Following Xiao and Carenini
(2020), we take the probability of selecting $s$ given by the global context
encoder module as a proxy for informativeness, i.e.
$f_{\text{I}}(s)=P(y^{g}\mid s,D^{\prime};\theta_{g})$. In the next section,
we elaborate on how $f_{\text{C}}$ models and enforces cohesion during
sentence selection.

##  4 Cohesion during Summary Extraction

Cohesion is a language mechanism that enables a sequence of sentences to
function as a unified whole Hassan et al. (1976). It does so by linking
semantic units in a text through cohesive ties, regardless of the grammatical
or discourse structure these units are part of. In particular, lexical
cohesion links units with the same lexical form, synonyms, or units in the
same semantic field. Furthermore, units tied cohesively can be grouped in
chains by their semantic similarity. Whilst the mere presence of two or more
chains does not guarantee a cohesive effect, their interaction can be a
reliable proxy for cohesion Morris and Hirst (1991); Barzilay and Elhadad
(1997).

In this paper, we focus on modeling lexical cohesive ties between noun phrases
in nearby sentences of a summary by controlling the interaction between
lexical chains.

###  4.1 KvD Select

The proposed selector, KvD-Select, calculates cohesion score $f_{C}$ by
simulating the processes in working memory during text production. The
procedure is based on the Micro-Macro Structure theory Kintsch and van Dijk
(1978), which describes the cognitive processes involved in text comprehension
and production at the local (micro) and global (macro) level of discourse.
Following Fang (2019), we implement processes happening at micro-level, which
deal with the movement of content in and out of working memory.

Let $T$ be working memory and $G$ long-term memory (LTM), where both are
separate sets of cohesive chains, and each chain as a set of noun phrases
(NPs). At selection step $t$, the algorithm extracts NPs from $s_{t}$ and
connects them to the chains in $T$ and $G$, constraining the number of active
chains in $T$ afterward. Cohesive score $f_{\text{C}}$ then depends on the
average similarity between units added to $T$ and those added to $G$. We now
elaborate on each step of the algorithm.

##### Extracting Noun Phrases.

Given sentence $s_{t}\in D^{\prime}$, we obtain $P$, the set of extracted
nominal chunks, obtained by merging nominal nodes in dependency trees with
their children, following the procedure of Fang (2019). Specifically, given
that node u is nominal dependent of a clausal predicate, u will have its child
v merged if eitherv is a function word, a single-token modifier, or u and v
form part of a multi-word expression.

##### Adding Content to Memory.

Next, cohesive ties between $s_{t}$ and $\hat{S}^{t-1}$ are enforced by adding
each NP in $P$ to the chain with the highest element-wise semantic similarity.
Formally, the optimal chain to add $a\in P$ to is
$C^{*}=\operatorname*{argmax}_{C\in T}\\{\phi(p,C)\\}$, where $\phi$ is the
average BERTScore Zhang et al. (2019) between $a$ and each NP in $C$. In order
to make sure that chains maintain an acceptable level of semantic similarity
between elements, $a$ is added to chain $C$ only if $\phi(a,C)\geq\nu$, where
$\nu$ is the minimum admissible similarity. This way the algorithm can control
the similarity length between chain members, and avoid a single, long chain.

If similarity with chains in $T$ is not strong enough, we look at chains in
$G$, in which case the chosen chain is moved back to $T$. This step simulates
how humans recall content no longer present in WM, the recall mechanism
Kintsch and van Dijk (1978). If still no chain in $G$ meets the similarity
requirement, we proceed to create a brand new chain in $T$ with $a$ as its
sole element. By searching for a good enough candidate chain first in $T$ and
then in $G$, we encourage cohesive ties between NPs in nearby sentences.

##### Updating Memory.

After adding incoming NPs to chains in memory, $T$ is updated to retain only
the WM most recent chains, where recency of a chain is defined as the id of
the selection step in which this chain was last retained in $T$. For instance,
a chain currently in $T$ is more recent (higher step id) than a chain in $G$
discarded in an earlier step. This design choice mimics the recency effect
behaviour during free recall tasks in human subjects Glanzer (1972), a
behaviour attributed to short-term memory. Finally, discarded chains are moved
to $G$, concluding the selection step.

##### Candidate Scoring.

Next, we define cohesion score $f_{\text{coh}}$ which will be used to
discriminate amongst possible continuations to $\hat{S}^{t-1}$. The objective
is to encourage NPs in $P$ to be assigned to recent chains, in turn
encouraging chains to cover nearby sentences in the final summary. In
addition, we want to score down candidate sentences with NPs added to chains
in long-term memory.

Let $A_{T}=\\{a;a\in P,C_{a}\in T\\}$, where $C_{a}$ is the chain $a$ was
added to. Similarly, let $A_{G}=\\{b;b\in P,C_{b}\in G\\}$. Then, let
$\text{rec}(C)$ be the number of selection steps passed since the last time
chain $C$ was retained in $T$. Quantity $\text{rec}(C)$ functions as a proxy
for how spread chain $C$ is, i.e. how far away two sentences covered by $C$
are. Then,

$\displaystyle f_{\text{coh}}=\frac{1}{|A_{T}|}\sum_{a\in
A_{T}}\frac{\phi(a,C_{a})}{\text{rec}(C_{a})}+\frac{\gamma_{\text{rec}}}{|A_{G}|}\sum_{b\in
A_{G}}\frac{\phi(b,C_{b})}{\text{rec}(C_{b})}.$ (4)

Hence, the cohesive score depends on the contribution of each cohesive tie
formed. For each chunk in $A_{T}$ and $A_{G}$, its contribution depends
directly on the strength of similarity to its assigned chain and inversely on
the spread of said chain. The contribution of chunks in $A_{G}$ is scaled down
by hyper-parameter $\gamma_{\text{rec}}\in[0;1]$ as to simulate the higher
cognitive cost incurred when retrieving information from long-term memory.

##  5 Experimental Setup

We now describe the datasets used, training details, baselines, and evaluation
methodology.

###  5.1 Datasets

We employ datasets pairing single and multi-documents with human-written
summaries, specifically scientific articles from the biomedical domain
(PubMed; Cohan et al. (2018))111We use text from all sections as the source
document and the abstract as reference summary., patents in the Chemistry and
Metallurgy industry (BigPatent.C;Sharma et al. (2019)), legislature reports of
U.S. bills (GovReport; Huang et al. (2021)), and news articles
(MultiNews;Fabbri et al. (2019)).

###  5.2 Pipeline Parameters

Hyper-parameters were tuned over the validation sets of each dataset.

##### Document Segmentation and Block Selection.

We use a block size of $B=2048$, context size of $C=200$ pieces,
$\lambda_{b}=0.2$, and set a budget of $16\,384$ input wordpieces.

##### Local Encoder (LE), Global Context Encoder (GCE).

The block encoder in LE is initialized with a pretrained checkpoint of LongT5
with transient-global attention Guo et al. (2022),222HuggingFace,
google/long-t5-tglobal-base and an output layer of size $200$.

The LE module is trained independently from the GCE module, with LE being
trained first, then GCE trained whilst LE remains frozen. In both cases, we
used the Adam optimizer (Loshchilov and Hutter, 2018), a constant learning
rate of $1e^{-6}$, effective batch size of $64$, and $50$k training steps.
During inference, we extract a maximum of $N=10$ local sentences per block and
a maximum of $1000$ sentences in total.

##### Summary Extractor.

We set $\lambda_{sel}=0.8$, working memory WM$=6$, recall cost
$\gamma_{\text{rec}}=0.01$, and a minimum NP similarity of $\nu=0.6$. Word
budget is set to $200$, $100$, $650$, $250$ for PubMed, BigPatent.C,
GovReport, and MultiNews, respectively.

###  5.3 Comparison Systems

Baselines consist of encoders pre-trained to consume the whole input as a
sequence of up to $16\,384$ pieces long, topped with a sigmoid layer for
classification. As encoders, we employ the base pretrained checkpoints of
Longformer (LED-Ext;Beltagy et al. (2020)) and LongT5’s encoder (LT5-Ext;Guo
et al. (2022)).333Similar model such as BigBird Zaheer et al. (2020) and
Linformer Wang et al. (2020) do not have the same input capacity, hence they
were not considered. However, we do include other options of Local Encoders in
§ C.2.

The impact of cohesion modeling is assessed by employing a greedy selector
over GCE scores, equivalent to set $f_{\text{C}}=0$ in Eq. 3, dubbed
BlockSel[LT5]. Finally, we compare against the following (summary) property-
oriented selectors.

##### MMR-Select.

Xiao and Carenini (2020) Reduces redundancy by selecting $s_{i}$ (candidate
sentence at selection step $i$) such that cosine similarity w.r.t. the
partially extracted summary $\hat{S}$ is minimized. Informativeness and
redundancy are balanced in the same way as in Eq. 3.

##### N-gram passing (NPass).

Encourages repetition by allowing $p$ percent of n-grams in $s_{i}$ to overlap
with $\hat{S}$. When $p=0$, this method reduces to n-gram blocking, whereas
when $p=1.0$, to greedy selection. We report bi-gram passing with $p=0.8$.

##### Shuffle Classifier (CCL-Select).

Holistically quantifies local coherence using CCL Steen and Markert (2022), a
scorer trained to distinguish shuffled from unshuffled text. We use RoBERTa
Liu et al. (2019) as underlying model and use a window of $3$ consecutive
sentences.

###  5.4 Evaluation

Informativeness is assessed using ROUGE F1 Lin (2004), Redundancy is evaluated
using sentence-wise ROUGE-L (RdRL;Bommasani and Cardie (2020)), and inverse
uniqueness (IUniq;Peyrard et al. (2017)), defined as $1-\text{Uniqueness}$,
where ‘Uniqueness’ is the ratio of unique n-grams to the total number of
n-grams. We report the mean value between uni-, bi-, and trigrams. Higher
values denote higher redundancy.

Cohesion is measured using Entity Graph (EGr;Guinaudeau and Strube (2013)),
which models a text as a sentence graph with edges between sentences with
nouns in common, using the average edge weight as a proxy for cohesion.
Finally, local coherence is assessed using CCL Steen and Markert (2022).

####  5.4.1 Human Evaluation

We elicit human judgments to assess overall quality, informativeness, and
cohesion in two separate studies. We sampled 30 documents from the test set of
PubMed and compare systems BlockSel[LT5], MMR-Select, and KvD-Select.

##### Ranking Campaign.

Subjects were shown the abstract and the introduction of a scientific article
along with two system summaries, and then then asked to select the best
summary (or select both in case of tie) according to three criteria: (i)
overall quality, (ii) informativeness, and (iii) cohesion. In this setup,
cohesion is evaluated as a holistic property of the text, as perceived by a
reader.

##### Chaining Campaign.

Subjects were shown a single summary and were asked to annotate lexical chains
by grouping together pre-extracted NPs in the same semantic field. We report
the following chain properties: (i) chain spread, defined as the average
number of sentences between immediate-neighbor sentences covered by the same
chain;(ii) chain density, the number of chains covering the same sentence; and
(iii) sentence coverage, the percentage of sentences covered by at least one
chain.

Inter-annotator agreement is calculated as the average lexical overlap between
chains, expressed in F1 score, calculated pair-wise between subjects. For this
study, we include reference summaries as one more analysis system.

##  6 Results and Discussion

Next, we discuss the results of our analyses and the outcome of the human
evaluation campaigns.

###  6.1 Reducing Redundancy in Input Blocks

The following block selection strategies were compared: (i) Original,
consisting of selecting blocks in their original order in the source
document;444For MultiNews, we use the order provided in the dataset. (ii)
Oracle Selection, which selects the block that maximizes ROUGE F1 scores (mean
of ROUGE-1 and ROUGE-2) w.r.t. the reference summary; (iii) Max. Redundancy,
which selects the most similar block possible (by flipping the sign in Eq. 1);
and finally, (iv) BlockSelect, the proposed strategy.

The analysis, showcased in Figure 2, evaluates input redundancy at each block
selection step, as well as informativeness and redundancy of summaries
extracted from the blocks available at each step, using a greedy selector. We
observe that BlockSelect has a direct impact not only on input redundancy but
also on summary redundancy. As intended, it is effective at incrementally
building an input sequence with lower RdRL, and the extracted summaries
exhibit lower RdRL and comparable ROUGE scores than the compared block
selection strategies. Similar trends were observed in the other
datasets.555See Fig. 4 in § C.3 for results in other datasets.

Figure 2: Effect of block selection strategy over input redundancy (left),
summary informativeness (center), and summary redundancy (right), evaluated as
block selection proceeds on the MultiNews validation dataset.

###  6.2 Trading-off Informativeness and Cohesion

Next, we turn to the summary extraction module. Tables 1 and 2 present the
performance in terms of informativeness, whereas Table 3, in cohesion. In all
our experiments, statistical significance at the 95% confidence level is
estimated using Mann–Whitney U tests ($p<0.05$).

First, note the impact on cohesion when controlling for redundancy. MMR-Select
manages to obtain comparable informativeness levels to BlockSel[LT5], being
most effective for BigPatent.C. However, minimizing sentence similarity comes
at the expense of a significant decrease in cohesion (EGr) and coherence
(CCL). Second, we find that NPass is the only one capable of obtaining
comparable or better ROUGE scores but EGr scores indicate that lexical passing
is not enough to improve cohesion.

When guiding selection with a holistic shuffle scorer, as expected, CCL-Select
obtains remarkably high CCL scores. However, note that this selector does show
a significant reduction in EGr scores w.r.t. BlockSel[LT5], indicating that
CCL is measuring also discourse organization, possibly in the form of
rhetorical role ordering –first background, then method, and so on. Hence, it
can be said that summaries in CCL-Select are better organized in terms of
rhetorical roles but exhibit lower cohesion than greedily selected summaries.

Finally, KvD-Select manages to strike an even more aggressive trade-off
between informativeness and cohesion. Across datasets, the selector exhibits
lower ROUGE scores but the best EGr scores (except for PubMed), and second
highest CCL score after CCL-Select.

System | PubMed | BigPatent.C
---|---|---
R1 | R2 | RL | R1 | R2 | RL
LED-Ext | 40.20 | 13.87 | 36.85 | 36.65 | 11.07 | 31.94
LT5-Ext | 48.15 | 21.45 | 44.49 | 39.54 | 13.25 | 34.30
BlockSel[LT5] | 46.16† | 19.74† | 42.49† | 39.57† | 13.25† | 34.26†
+MMR-Select |  46.14† |  19.63† |  42.47† |  39.59† |  13.29† |  34.30†
+NPass |  46.38† |  19.92† |  42.74† |  39.59† |  13.26† |  34.29†
+CCL-Select |  45.91† |  19.60† |  42.45† |  39.16† | 12.95 |  33.92†
+KvD-Select | 44.90 | 18.47 | 41.27 | 38.37 | 12.41 | 33.13
Table 1: Informativeness in terms of ROUGE scores (R1, R2, RL) for PubMed and BigPatent.C datasets. †: no stat. difference between systems in the same column. Best systems are bolded; systems better than BlockSel[LT5] shown in blue and worse, in red. System | GovReport | MultiNews
---|---|---
R1 | R2 | RL | R1 | R2 | RL
LED-Ext | 57.59 | 23.40 | 54.56 | 45.28 | 15.78 | 41.35
LT5-Ext | 59.33 | 25.94 | 56.29 | 47.07 | 17.54 | 42.96
BlockSel[LT5] | 59.73† | 26.21† | 56.50† | 46.80† | 17.21† | 42.66†
+MMR-Select |  59.79† |  26.30† |  56.56† |  46.76† |  17.13† |  42.59†
+NPass |  59.79† |  26.25† |  56.56† |  46.91† |  17.27† |  42.78†
+CCL-Select |  59.72† |  26.24† | 56.50 |  46.85† |  17.29† |  42.71†
+KvD-Select | 57.88 | 23.66 | 54.57 | 45.85 | 16.13 | 41.62
Table 2: Informativeness in terms of ROUGE scores (R1, R2, RL) for GovReport and MultiNews datasets. See Table 1 for formatting details. Systems | PubMed | BigPatent.C | GovReport | MultiNews
---|---|---|---|---
RdRL | EGr | CCL | RdRL | EGr | CCL | RdRL | EGr | CCL | RdRL | EGr | CCL
LED-Ext | 14.70 | 1.00 | 0.36 | 14.94 | 0.69 | 0.35 | 15.06 | 1.91 | 0.30 | 11.25 | 0.81 | 0.30
LT5-Ext | 16.49 | 1.10 | 0.24 | 19.76 | 0.75 | 0.32 | 15.78 | 2.01 | 0.31 | 12.24 | 0.91 | 0.25
BlockSel[LT5] | 17.08† | 1.07† | 0.25† | 20.15† | 0.73 | 0.37† | 16.34 | 2.04† | 0.30† | 12.26 | 0.90† | 0.24
+MMR-Select |  16.99† | 1.07 |  0.23† |  19.17† | 0.73 |  0.33† | 16.16 |  2.03† | 0.30 |  12.05† |  0.88† | 0.22
+NPass | 16.39 | 1.07 | 0.25 |  19.79† | 0.73 | 0.37† | 16.24 | 2.04 |  0.31† |  12.03† |  0.89† | 0.24
+CCL-Select |  16.63† |  1.06† | 0.66 | 18.97 | 0.71 | 0.88 | 16.31 | 2.04† | 0.65 | 11.93 | 0.86 | 0.63
+KvD-Select |  16.24† |  1.05† | 0.27 | 21.09 | 0.78 |  0.40† | 16.69 | 2.15 | 0.32 | 12.97 | 0.99 | 0.28
Gold | 13.54 | 0.95 | 0.87 | 18.11 | 0.78 | 0.92 | 13.37 | 1.95 | 0.87 | 9.72 | 0.71 | 0.90
Table 3: Summary redundancy (RdRL), cohesion (EntityGraph; EGr), and local
coherence (CCL). For cohesion and coherence metrics, higher is better. See
Table 1 for formatting details.

##### Effect of Parameter $\lambda_{\text{sel}}$.

Figure 3 showcases how summary properties vary across increasing levels of
$\lambda_{\text{sel}}$, for MultiNews. Note that when $\lambda_{\text{sel}}=0$
selectors depend entirely on $f_{\text{C}}$, and $\lambda_{\text{sel}}=1.0$ is
the greedy selector. As expected, informativeness is higher as $f_{\text{I}}$
is weighted up (higher $\lambda_{\text{sel}}$) with all selectors except MMR-
Select. This indicates that it is possible to increase cohesion whilst
preventing redundancy from increasing, however at the expense of decreased
informativeness. Interestingly, KvD-Select seems robust to
$\lambda_{\text{sel}}$ in terms of EGr and RdRL. We hypothesize that KvD-
Select benefits from a signal indicating which cohesive ties are informative
and worth enforcing.

Figure 3: Informativeness (right), redundancy (center), and cohesion (right)
in summaries, across increasing values of trade-off parameter
$\lambda_{\text{sel}}$, on the validation set of MultiNews.

##### Impact of Block Selection.

Systems using our block selection mechanism exhibit lower ROUGE scores than
flat processing baselines (LED-Ext and LT5-Ext) in PubMed and MultiNews, and
comparable performance for BigPatent.C and GovReport. However, BlockSel[LT5]
shows slightly higher EGr scores in all datasets. This indicates that cascaded
processing puts a greedy selector in a better position to extract more
cohesive summaries at the expense of a slight decrease in informativeness.

###  6.3 Human Evaluation

In both studies, statistical significance between system scores was assessed
using a one-way ANOVA with posthoc Tukey tests with $95\%$ confidence interval
($p<0.01$). Results are presented in Table 4.

##### Ranking.

Krippendorff’s $\alpha$ Krippendorff (2011) showed an inter-annotator
agreement of $0.68$. For overall quality, subjects showed a significant
preference for KvD-Select over BlockSel[LT5]. For cohesion, KvD-Selec was
perceived as more cohesive compared to BlockSel[LT5], and BlockSel[LT5] was
more cohesive than MMR-Select.

##### Chaining.

Chain overlap was calculated at $0.90$. Differences between BlockSel[LT5] and
all other systems, as well as MMR-Select–Gold and KvD-Select–BlockSel[LT5]
were found to be significant, for all measurements of cohesion. Moreover, the
number of NPs annotated per chain was $2.30$, $2.33$, $2.80$, and $2.55$, for
systems BlockSel[LT5], MMR-Select, KvD-Select, and Gold, respectively.

We found that KvD-Select summaries exhibit more active and denser chains and
better-covered sentences than the baselines. Note that BlockSel[LT5] obtains
the lowest chain spread but also low coverage, indicating that its summaries
exhibit very few chains that happen to be close to each other. In contrast,
MMR-Select obtains the highest chain spread and low number of chains,
indicating content with low diversity and sparsely presented.

System | Ranking | Chaining
---|---|---
Ov$\downarrow$ | I$\downarrow$ | C$\downarrow$ | Spr$\downarrow$ | Den$\uparrow$ | Cov$\uparrow$
BlockSel[LT5] | 1.59 | 1.56 | 1.59 | 1.93 | 1.29 | 57.12
+MMR-Select | 1.50 | 1.48 | 1.47 | 2.36 | 1.28 | 53.21
+KvD-Select | 1.41 | 1.46 | 1.44 | 2.05 | 1.40 | 68.78
Gold | - | - | - | 1.91 | 1.36 | 69.65
Table 4: Ranking (left) w.r.t. (Ov)erall quality, (I)nformativeness, and
(C)ohesion; and properties of annotated chains (right): spread (Spr), density
(Den), and sentence coverage (Cov,%). Best systems are bolded.
($\uparrow$,$\downarrow$): higher, lower is better.

##  7 Conclusions

We presented an extractive summarization algorithm that controls each summary
quality independently, in scenarios where the input is highly redundant.
Redundancy is controlled as the input is consumed, and informativeness and
cohesion are balanced during sentence selection.

Results show that our input processing strategy is effective at retrieving
non-redundant yet relevant passages, reducing the redundancy levels the rest
of the pipeline is exposed to. In addition, our sentence selector emulates
human memory to keep track of cohesive chains while building the summary,
enforcing ties between noun phrases directly. Extensive automatic and human
experiments revealed that it is possible to extract highly cohesive summaries
that are as informative as summaries optimizing only for informativeness.

## Limitations

The proposed system presents the following limitations. First, the system
extracts complete sentences and concatenates them to form the final summary.
We do not perform any kind of post-editing of discourse markers that might
break coherence in the summary. However, our results show that the extracted
summaries are still perceived as cohesive by humans. Nevertheless, post-
editing is an interesting focus for future work.

Second, we argue about the usefulness of an extractive system in a generative
landscape where large language models are predominant. Recent large language
models have shown impressive capabilities at producing coherent, assertive
text, some even capable of consuming long sequences of tokens. However,
hallucinations are a pervasive problem in these systems, especially in highly
technical domains like the ones considered in this work. In this scenario, an
extractive summary has the advantage of presenting information from the source
verbatim and hence, without any hallucination. Moreover, extracted summaries
preserve the writing style of the input as well as technical, domain-specific
terms, avoiding altogether the problems of over-simplification and misstyling.

## References

  * Barzilay and Elhadad (1997) Regina Barzilay and Michael Elhadad. 1997.  Using lexical chains for text summarization.  In _Intelligent Scalable Text Summarization_.
  * Barzilay and Elhadad (2002) Regina Barzilay and Noemie Elhadad. 2002.  Inferring strategies for sentence ordering in multidocument news summarization.  _Journal of Artificial Intelligence Research_ , 17:35–55.
  * Barzilay and Lapata (2008) Regina Barzilay and Mirella Lapata. 2008.  Modeling local coherence: An entity-based approach.  _Computational Linguistics_ , 34(1):1–34.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Bommasani and Cardie (2020) Rishi Bommasani and Claire Cardie. 2020.  Intrinsic evaluation of summarization datasets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8075–8096.
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of mmr, diversity-based reranking for reordering documents and producing summaries.  In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 335–336.
  * Cho et al. (2022) Sangwoo Cho, Kaiqiang Song, Xiaoyang Wang, Fei Liu, and Dong Yu. 2022.  Toward unifying text segmentation and long document summarization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 106–118.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621.
  * Erkan and Radev (2004) Günes Erkan and Dragomir R Radev. 2004.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal of artificial intelligence research_ , 22:457–479.
  * Fabbri et al. (2021) Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  Summeval: Re-evaluating summarization evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Fabbri et al. (2019) Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019\.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084.
  * Fang (2019) Yimai Fang. 2019.  _Proposition-based summarization with a coherence-driven incremental model_.  Ph.D. thesis, University of Cambridge.
  * Glanzer (1972) Murray Glanzer. 1972.  Storage mechanisms in recall.  In _Psychology of learning and motivation_ , volume 5, pages 129–193. Elsevier.
  * Gu et al. (2022) Nianlong Gu, Elliott Ash, and Richard Hahnloser. 2022.  MemSum: Extractive summarization of long documents using multi-step episodic Markov decision processes.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6507–6522, Dublin, Ireland. Association for Computational Linguistics.
  * Guinaudeau and Strube (2013) Camille Guinaudeau and Michael Strube. 2013.  Graph-based local coherence modeling.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 93–103.
  * Guo et al. (2022) Mandy Guo, Joshua Ainslie, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022.  Longt5: Efficient text-to-text transformer for long sequences.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 724–736.
  * Hassan et al. (1976) Halliday Hassan, Rukaya, and Michael A. K. Halliday. 1976.  _Cohesion in English_.  Routledge.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28.
  * Huang et al. (2021) Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021.  Efficient attentions for long document summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1419–1436.
  * Jeon and Strube (2020) Sungho Jeon and Michael Strube. 2020.  Centering-based neural coherence modeling with hierarchical discourse segments.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7458–7472, Online. Association for Computational Linguistics.
  * Johnstone (1994) Barbara Johnstone. 1994.  _Repetition in discourse : interdisciplinary perspectives_.  Advances in discourse processes. Ablex Publishing Corporation.
  * Kintsch (1990) Eileen Kintsch. 1990.  Macroprocesses and microprocesses in the development of summarization skill.  _Cognition and instruction_ , 7(3):161–195.
  * Kintsch and van Dijk (1978) Walter Kintsch and Teun A van Dijk. 1978.  Toward a model of text comprehension and production.  _Psychological review_ , 85(5):363.
  * Krippendorff (2011) Klaus Krippendorff. 2011.  Computing krippendorff’s alpha-reliability.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2018) Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018.  Generating wikipedia by summarizing long sequences.  In _International Conference on Learning Representations_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , page 5070. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Mallinson et al. (2020) Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020.  Felix: Flexible text editing through tagging and insertion.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1244–1255.
  * Mesgar and Strube (2016) Mohsen Mesgar and Michael Strube. 2016.  Lexical coherence graph modeling using word embeddings.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1414–1423, San Diego, California. Association for Computational Linguistics.
  * Morris and Hirst (1991) Jane Morris and Graeme Hirst. 1991.  Lexical cohesion computed by thesaural relations as an indicator of the structure of text.  _Computational Linguistics_ , 17(1):21–48.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , page 280. Association for Computational Linguistics.
  * Nenkova et al. (2011) Ani Nenkova, Kathleen McKeown, et al. 2011.  Automatic summarization.  _Foundations and Trends® in Information Retrieval_ , 5(2–3):103–233.
  * Peyrard et al. (2017) Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017.  Learning to score system summaries for better content selection evaluation. In _Proceedings of the Workshop on New Frontiers in Summarization_ , pages 74–84, Copenhagen, Denmark. Association for Computational Linguistics.
  * Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019.  Bigpatent: A large-scale dataset for abstractive and coherent summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2204–2213.
  * Steen and Markert (2022) Julius Steen and Katja Markert. 2022.  How to find strong summary coherence measures? a toolbox and a comparative study for summary coherence measure evaluation.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 6035–6049, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Tauste (1995) Ana María Vigara Tauste. 1995.  Comodidad y recurrencia en la organización del discurso coloquial.  In _El español coloquial: actas del I Simposio sobre análisis del discurso oral: Almería, 23-25 de noviembre de 1994_ , pages 173–208. Servicio de Publicaciones.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Walker (1993) Marilyn A Walker. 1993.  _Informational redundancy and resource bounds in dialogue_.  Ph.D. thesis, Graduate School of Arts and Sciences, University of Pennsylvania.
  * Wang et al. (2020) Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_.
  * West et al. (2019) Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. 2019.  Bottlesum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3752–3761.
  * Xiao and Carenini (2020) Wen Xiao and Giuseppe Carenini. 2020.  Systematically exploring redundancy reduction in summarizing long documents.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 516–528.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in neural information processing systems_ , 33:17283–17297.
  * Zajic et al. (2007) David Zajic, Bonnie J Dorr, Jimmy Lin, and Richard Schwartz. 2007.  Multi-candidate reduction: Sentence compression as a tool for document summarization tasks.  _Information Processing & Management_, 43(6):1549–1570.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2023) Wei Zhao, Michael Strube, and Steffen Eger. 2023.  Discoscore: Evaluating text generation with bert and discourse coherence.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3847–3865.

##  Appendix A Dataset Preprocessing and Statistics

For all datasets, we homogenize the source-target length distributions by
discarding samples with references that were too short (less than 3 sentences,
not usefull for our cohesion analysis) or too long (more than 500 tokens in
all datasets except GovReport, for which this threshold is set to $1000$).
Similarly, samples with short input documents (less than 3 sentences or less
than 30 tokens in total) were also discarded. Sentences were re-split using
spaCy666https://spacy.io/ and trimmed to 100 tokens, whilst sentences with
less than $5$ tokens were discarded. Table 5 presents the statistics of all
dataset in terms of number of tokens.

It is worth noting that we found a discrepancy in PubMed.Text from the
‘article’ field (in theory the concatenated sections) would not always have
the same text as the ‘sections’ field. Hence, we chose data from the
‘sections’ field as input document.

##### Datasets with medium-length documents.

Previous work in redundancy reduction Xiao and Carenini (2020) showed that
medium-length documents (e.g. articles in CNN-DailyMail, Nallapati et al.
2016) do not exhibit content redundancy as acutely as long documents (e.g.
scientific articles). In this work, we focus on cases where input redundancy
is especially acute. However, we provide an analysis of the impact of input
redundancy on summary properties in Section 6.1. In Figure 2, we observe how
summary informativeness and redundancy change as we increase the permissible
input length, effectively simulating scenarios with documents of low-,
medium-, and high-length.

In addition, the CNN/DM dataset was not included in our experiments because
the reference summary is formatted as a list of sentences (highlights) where
each sentence exhibits a high degree of independence. In contrast, a format of
continuous sentences such as scientific abstracts constitutes a more
appropriate textual format for finding and analyzing cohesion phenomena.

Dataset | Input Length | Target Len.
---|---|---
Avg. | Max. | Q90 | Avg.
PubMed | 3150 | 119875 | 5844 | 206
BigPatent.C | 4534 | 72835 | 8655 | 119
GovReport | 8840 | 206622 | 15752 | 580
MultiNews | 2057 | 525348 | 3846 | 260
CNN/DM | 2057 | 525348 | 3846 | 260
NewsRoom | 2057 | 525348 | 3846 | 260
Table 5: Dataset statistics in terms of number of tokens showing average,
maximum, and 90% quantile (Q90).

##  Appendix B Optimization and Resource Details

Long-T5 models were trained using one NVIDA A100 (80Gb of GPU memory). Table 6
provides a comprehensive account of hyperparameter values used for training
and inference in our experiments, for all datasets. Hyper-parameters specific
to selector were optimized w.r.t. each dataset’s validation set.

The Local Encoder (LE) module is finetuned from pretrained Huggingface’s
checkpoint google/long-t5-tglobal-base, whereas the Global Context Encoder
(GCE) is trained from scratch. For training LE, we obtain extractive oracle
sentences from each block and train the module over blocks with
$\text{ROUGE-1}+\text{ROUGE-2}>0.5$. This threshold was obtained by observing
the mean of the distribution of ROUGE-1 + ROUGE-2 scores of these extractive
oracles w.r.t. references in preliminary analyses.

During inference, our system consumes input texts of up to $16\,384$ pieces in
length. This input is reduced to a maximum of $M=$$1000$ sentences, which are
pre-extracted by the LE module and then passed through the GCE module. From
these $M$ sentences, the summary extractor module extracts a summary with a
predefined word budget.

##### Artifact Lincense.

The models proposed in this paper extend the licenses of the artifacts they
are made of. LongT5 and LED were released under the Apache 2.0 license. LLaMa
is under META’s Community License
Agreement.777https://github.com/facebookresearch/llama/blob/main/LICENSE

Parameter | Value
---|---
Block Selection |
Block length in tokens | $2048$
Overlapping context size in tokens | 200
Damping factor ($d$) | 0.85
Trade-off param. ($\lambda_{b}$) | 0.2
Local Context Extractor |
Optimizer | Adam
Learning rate | 1E-06
Learning rate scheduler | Const.
Batch size | 64
Max. gradient norm | 2
Training steps | $100\,000$
Max. input length in tokens | $2048$
Max. # of sentences extracted | 10
Global Contetext Encoder |
# Attention heads | 8
# Layers | 1
Output layer size | 200
Dropout | 0.1
Optimizer | Adam
Learning rate | 1E-06
Learning rate scheduler | Const.
Max. input length in tokens | $16\,384$
Max. input length in sentences | $1000$
Batch size | 64
Max. gradient norm | 1
Training steps | $50\,000$
Sentence Selector |
All selectors. Trade-off param. ($\lambda_{\text{sel}}$) | 0.8
Summary budget in number of tokens |
PubMed | 200
BigPatent.C | 100
GovReport | 650
MultiNews | 250
KvD-Selector. |
Working memory (WM) | 6
Min. NP cos. similarity ($\nu$) | 0.6
Recall cost ($\gamma_{\text{rec}}$) | 0.01
Table 6: Hyper-parameter values for all modules in our summarization pipeline.

##  Appendix C Complementary Results

In this appendix, we present additional results in terms of metrics and
datasets for analysis in §6.

###  C.1 Additional Metrics

##### Semantic Relevance.

Table 7 shows BERTScore F1 scores Zhang et al. (2019) with importance
weighting (IDF) and with underlying models RoBERTa large Liu et al. (2019) and
DeBERTa v2 XLarge-MNLI He et al. (2021), with HuggingFace checkpoint names
roberta-large and deberta-xxlarge-mnli, respectively. Previous work found that
DeBERTa obtained higher correlation with human scores than RoBERTa
888https://github.com/microsoft/DeBERTa showed the same system-level ranking.

##### Cohesion.

The additional cohesion metrics considered are Lexical Graph Mesgar and Strube
(2016)) and the Noun variants of DiscoScore Zhao et al. (2023), DS-Focus[NN]
and DS-Sent[NN].

Lexical Graph (LexG) computes the adjacency matrix of the sentence graph of a
text, where two sentences are connected if they have at least two similar-
enough content words, i.e. if the cosine similarity between their embeddings
is greater than a threshold (zero). DS-Focus[NN] (DS-Foc) computes the
semantic difference between a set of nouns (foci) common to a candidate
summary and a reference. Similar to Lexical Graph, DS-Sent[NN] (DS-Sen)
computes the semantic similarity between nouns in adjacent sentences,
inversely weighted by the distance between sentences.

Table 8 presents results for the additional metrics in all datasets and
systems. In terms of LexG scores, we find that KvD-Select outperforms all
systems in all datasets except PubMed, indicating that consecutive sentences
in summaries extracted by our system contain more cohesive ties connecting
highly semantically similar nouns. Differences between systems in terms of DS-
Foc and DS-Sen scores are much less clearer, with DS-Foc seemingly being more
indicative of content coverage and DS-Sen more indicative of cohesion.
Nevertheless, KvD-Select is consistently competitive across datasets, even
obtaining the highest DS-Sen score for MultiNews.

Lastly, it is important to note that none of the score differences in Table 8
were found to be statistically significant, an indicative of the challenge of
measuring cohesion. For this reason, these results were not included in the
main argument of this paper.

System | PubMed | BigPatent.C | GovReport | MultiNews
---|---|---|---|---
| RoB | DeB | RoB | DeB | RoB | DeB | RoB | DeB
Ext-Oracle | 88.44 | 80.20 | 85.83 | 73.80 | 88.30 | 80.06 | 88.69 | 80.04
LT5-Abs | 85.71 | 73.94 | 84.09 | 70.16 | 86.49 | 76.52 | 85.13 | 74.22
LED-Ext | 83.67 | 69.83 | 83.04 | 68.19 | 85.95 | 74.81 | 85.51 | 73.34
LT5-Ext | 85.71 | 74.16 | 83.77 | 69.81 | 86.44 | 75.95 | 86.03 | 74.34
BlockSel[MemSum] | 83.52 | 70.05 | 82.60 | 67.82 | 85.06 | 73.08 | 85.07 | 72.76
BlockSel[LLaMa] | 82.86 | 68.52 | 83.17 | 68.65 | 84.80 | 72.82 | 85.33 | 73.03
BlockSel[LT5] | 85.05 | 73.08 | 83.65† | 69.75† | 86.46† | 76.06† | 85.97 | 74.10
+MMR-Select | 85.05 |  73.07† |  83.66† |  69.78† | 86.49 | 76.08 |  85.93† | 74.04
+NPass | 85.13 | 73.21 |  83.67† |  69.76† |  86.47† | 76.08 | 86.01 |  74.17†
+CCL-Select |  84.99† | 72.98 | 83.63 | 69.62 |  86.47† |  76.05† |  85.91† |  74.18†
+KvD-Select | 84.76 | 72.43 | 83.34 | 69.15 | 85.99 | 75.17 | 85.72 | 73.67
Table 7: Semantic relevance of system summaries in terms of BERTScore F1 using RoBERTa (RoB) and DeBERTa (DeB) as base models. †: no stat. difference between systems in the same column. Best systems are bolded; systems better than BlockSel[LT5] shown in blue and worse, in red. Systems | PubMed | BigPatent.C | GovReport | MultiNews
---|---|---|---|---
LGr | DS-Foc | DS-Sen | LGr | DS-Foc | DS-Sen | LGr | DS-Foc | DS-Sen | LGr | DS-Foc | DS-Sen
Gold | 1.1088 | 0.0000 | 1.0000 | 0.8368 | 0.0000 | 1.0000 | 1.9990 | 0.0000 | 1.0000 | 0.7756 | 0.0000 | 1.0000
Ext-Oracle | 1.1463 | 0.5185 | 0.9692 | 0.8022 | 0.5642 | 0.9499 | 1.9735 | 1.7069 | 0.9973 | 0.7488 | 0.3684 | 0.9789
LT5-Abs | 1.0953 | 1.2290 | 0.9262 | 0.8411 | 5.0947 | 0.9106 | 1.6333 | 7.9962 | 0.9896 | 0.8492 | 0.7973 | 0.9602
LT5-Flat | 1.2467 | 1.0925 | 0.9350 | 0.8084 | 1.0244 | 0.9256 | 2.0583 | 3.8380 | 0.9949 | 0.9687 | 0.7623 | 0.9642
LED-Flat | 1.1634 | 1.1075 | 0.9269 | 0.7800 | 1.0029 | 0.9250 | 1.9735 | 3.7556 | 0.9953 | 0.8762 | 0.6672 | 0.9646
MemSum-Casc | 0.9608 | 0.9232 | 0.9280 | 0.7042 | 0.9232 | 0.9174 | 1.8057 | 3.1476 | 0.9953 | 0.9081 | 0.6404 | 0.9630
LlaMa-Casc | 0.9222 | 0.9023 | 0.9247 | 0.7795 | 0.9315 | 0.9235 | 1.6541 | 3.1698 | 0.9947 | 0.8295 | 0.6774 | 0.9638
BlockSel[LT5] | 1.2204 | 1.0219 | 0.9303 | 0.7858 | 1.0291 | 0.9228 | 2.0838 | 3.4428 | 0.9947 | 0.9628 | 0.7353 | 0.9629
+MMR-Select | 1.2203 | 1.0117 | 0.9300 | 0.7803 | 1.0108 | 0.9239 | 2.0797 | 3.4391 | 0.9948 | 0.9464 | 0.7348 | 0.9632
+NPass | 1.2153 | 0.9833 | 0.9306 | 0.7837 | 1.0246 | 0.9229 | 2.0823 | 3.4129 | 0.9946 | 0.9595 | 0.7183 | 0.9630
+CCL-Select | 1.2075 | 0.9970 | 0.9305 | 0.7623 | 0.9869 | 0.9233 | 2.0867 | 3.4481 | 0.9948 | 0.9323 | 0.7150 | 0.9638
+KvD-Select(ours) | 1.1933 | 0.9948 | 0.9298 | 0.8207 | 1.0644 | 0.9210 | 2.1824 | 3.5580 | 0.9945 | 1.0536 | 0.7235 | 0.9651
Table 8: Summary cohesion in terms of Lexical Graph (LexG; Mesgar and Strube
(2016)) and DiscoScore’s Zhao et al. (2023) DS-Focus[NN] (DS-Foc) and DS-
Sent[NN] (DS-Sen). For all metrics, higher value is better except for DS-Foc.
Best systems are bolded; systems better than BlockSel[LT5] shown in blue and
worse, in red.

###  C.2 Additional Comparison Systems and Results

We compare against the following additional baselines.

##### Extractive Oracle.

The standard extractive oracle, labeled as Ext-Oracle, is obtained by greedily
selecting sentences maximizing $\text{ROUGE-1}+\text{ROUGE-2}$ against gold
summaries until the word budget is met.

##### Abstractive Baselines.

We compare against Longformer Encoder Decoder (LED-Abs;Beltagy et al. (2020))
and the complete LongT5 encoder-decoder architecture (LT5-Abs;Guo et al.
(2022)). Both systems were trained for 3k steps with batch size of $128$, and
AdamW optimizers. For LT5-Abs, we used a constant learning rate of 1E-3, using
the Huggingface’s checkpoints google/long-t5-tglobal-base. For LED-Abs, a
learning rate of 1E-5 was used, with triangular scheduling using 100 steps for
warm-up, and using HF’s checkpoint allenai/led-base-16384. For both systems,
inference was done using a beam size of 5 and length penalty of $0.5$, $0.5$,
$2.0$, and $1.0$ for PubMed, BigPatent.C, GovReport, and MultiNews,
respectively.

##### Choice of Local Encoder.

We assess the impact of architectural choice for the Local Encoder module in
our pipeline by using MemSum Gu et al. (2022) and LLaMA-7B Touvron et al.
(2023) as encoders. The systems, dubbed BlockSel[MemSum] and BlockSel[LLaMa],
employ a greedy selector.

##### Results.

The results on informativeness are presented in Table 9, whereas redundancy
and cohesion are in Table 10. The following insights can be drawn. Using LLaMA
as local encoder allows our system to select –greedily– sentences that have
little lexical overlap between them, prompting low summary redundancy scores
and in turn lowering cohesion scores. Moreover, the coverage is severely
impacted as seen by the low ROUGE scores. These results might indicate that
finetuning a large pretrained model like LLaMA does not necessarily translate
to better informativeness, performing much lower than a smaller model
pretrained on the summarization task. Perhaps unsurprisingly, task-specific,
smaller models can be competitive to massive foundation models trained on
1000x more data.

Using MemSum as the local encoder had a similar outcome, although not as
severe as when using LLaMa. Summaries in Gu et al. (2022) were obtained in a
scenario where only up to 500 sentences were consumed in the order they appear
in the document. In contrast, in our setup, the compared systems consume up to
$16\,384$ pieces of input text in the order the block selector module
retrieves. The performance gap between the results in Gu et al. (2022) and the
ones we report can then be explained by input length and ordering conditions.

System | PubMed | BigPatent.C | GovReport | MultiNews
---|---|---|---|---
R1 | R2 | RL | R1 | R2 | RL | R1 | R2 | RL | R1 | R2 | RL
Ext-Oracle | 65.10 | 37.99 | 60.76 | 53.85 | 23.20 | 46.90 | 72.66 | 40.90 | 69.36 | 62.66 | 33.73 | 57.93
LED-Abs | 45.31 | 20.73 | 41.82 | 35.89 | 14.66 | 31.45 | 54.34 | 24.78 | 51.48 | 46.73 | 18.93 | 43.11
LT5-Abs | 46.27 | 20.92 | 42.40 | 37.63 | 15.67 | 32.84 | 51.72 | 24.79 | 49.03 | 45.72 | 17.70 | 41.86
BlockSel[MemSum] | 40.29 | 14.85 | 37.09 | 36.07 | 10.79 | 30.97 | 54.91 | 19.66 | 51.75 | 44.47 | 15.28 | 40.32
BlockSel[LLaMa] | 37.60 | 11.86 | 34.51 | 36.82 | 11.24 | 32.00 | 54.20 | 19.02 | 50.90 | 45.02 | 15.48 | 41.00
BlockSel[LT5] | 46.16 | 19.74 | 42.49 | 39.57 | 13.25 | 34.26 | 59.73 | 26.21 | 56.50 | 46.80 | 17.21 | 42.66
Table 9: Informativeness in terms of ROUGE F1 scores (R1, R2, RL), for different choices of local encoder during block selection. All systems employ a greedy selector.  Systems | PubMed | BigPatent.C | GovReport | MultiNews
---|---|---|---|---
RdRL | EGr | CCL | RdRL | EGr | CCL | RdRL | EGr | CCL | RdRL | EGr | CCL
Ext-Oracle | 13.91 | 0.99 | 0.43 | 14.70 | 0.68 | 0.34 | 14.20 | 1.92 | 0.58 | 10.08 | 0.68 | 0.51
LED-Abs | 15.00 | 0.93 | 0.85 | 38.23 | 0.94 | 0.77 | 16.55 | 1.75 | 0.75 | 8.37 | 0.53 | 0.83
LT5-Abs | 16.15 | 0.96 | 0.86 | 38.04 | 0.82 | 0.81 | 15.89 | 1.60 | 0.81 | 12.60 | 0.78 | 0.78
BlockSel[MemSum] | 12.58 | 0.75 | 0.25 | 19.41 | 0.62 | 0.34 | 13.77 | 1.72 | 0.29† | 12.29† | 0.84 | 0.26†
BlockSel[LLaMA] | 11.61 | 0.70 | 0.27 | 17.51 | 0.70 | 0.39 | 12.43 | 1.58 | 0.28† | 10.87 | 0.77 | 0.25
BlockSel[LT5] | 17.08 | 1.07 | 0.26 | 20.15† | 0.73 | 0.39 | 16.34 | 2.04 | 0.27 | 12.26† | 0.90 | 0.26†
Gold | 13.54 | 0.95 | 0.78 | 18.11 | 0.78 | 0.83 | 13.37 | 1.95 | 0.75 | 9.72 | 0.71 | 0.80
Table 10: Summary redundancy (RdRL), cohesion (EGr), and local cohesion (CCL),
for complementary abstractive, flat, and cascaded systems. For cohesion and
coherence metrics, higher is better. Best systems are bolded. †: no stat.
difference between systems in the same column.

###  C.3 Reducing redundancy in block selection

Figure 4 presents the effect of block selection strategies for PubMed,
BigPatent.C, and GovReport.

(a) PubMed

(b) BigPatent.C

(c) GovReport

(d) MultiNews

Figure 4: Effect of block selection strategy over input redundancy (left),
summary informativeness (center), and summary redundancy (right), evaluated as
block selection proceeds on the validation splits of all datasets analysed.

###  C.4 Effect of Trade-off Parameter $\lambda_{\text{sel}}$

Figure 5 showcases how summary properties (informativeness, redundancy, and
cohesion) vary across increasing levels of $\lambda_{\text{sel}}$, for all
datasets analyzed.

(a) PubMed

(b) BigPatent.C

(c) GovReport

(d) MultiNews

Figure 5: Informativeness (left), redundancy (mid), and lexical cohesion
(right) across different values of the trade-off parameter $\lambda_{sel}$ on
the validation set of PubMed, BigPatent.C, GovReport, and MultiNews.

##  Appendix D Human Evaluation Campaigns

In this section, we provide further details about the two evaluation
campaigns. Both campaigns were run on Amazon Mechanical Turk, where Turkers
were required to have a Human Intelligence Task (HIT) approval rate higher
than $99\%$, a minimum of $10\,000$ approved HITs, be proficient in the
English language, and have worked in the healthcare or medical sector before.

Regarding the difficulty of the domain for non-expert annotators, previous
work Fabbri et al. (2021) showed that preference rankings of systems were the
same across expert and non-expert annotators, although with more marked
differences in preference for expert annotations. This indicates that non-
expert annotation still holds statistical value and conclusions can be drawn
from them, as has been done in previous work for arXiv Cho et al. (2022) and
PubMed Gu et al. (2022).

We requested three different annotators per HIT, and annotators were awarded
$\$1$ per HIT, translating to more than $15 per hour. These rates were
calculated by measuring the average annotation time per HIT in a pilot study.
Furthermore, we implemented the following catch controls: (i) we asked
participants to check checkboxes confirming they had read the instructions and
examples provided, and (ii) we discard HITs that were annotated in less than
$5$ minutes.999Time threshold obtained from pilot study measurements.
Annotations that failed the controls were discarded in order to maximize the
quality. Figure 6 depicts the instructions given to annotators for each
campaign.

###  D.1 Ranking Campaign

We collected three annotations (by different annotators) per system-pair
comparison and made sure that the same annotator was not exposed to the same
document twice. As an additional catch trial, we included in each annotation
batch an extra instance with summaries extracted by the extractive oracle and
the random baseline.

After discarding annotations that failed the controls, we are left with $708$
out of $810$ instances ($30$ documents, $3$ system pairs, $3$ dimensions, and
$3$ annotations per pair), and $90$ different annotators.

###  D.2 Chaining Campaign

Participants were shown a single system summary as a list of sentences where
tokens that belonged to the same noun phrase were colored the same.Then, the
task consists of selecting sets of colored text chunks that belong to the same
semantic field. Similarly to the previous study, we collected three
annotations per system summary (by different annotators) and included the gold
summary of an extra system in the campaign.

We collected $908$ human annotations of noun-phrase chains for 360 summaries
($30$ documents, $4$ systems including gold summaries, and $3$ annotations per
summary), from a total of $86$ different annotators. On average, annotators
identified $2.56$ groups per summary and $3.49$ NPs per group.

(a) Ranking Campaign

(b) Chaining Campaign

Figure 6: Instructions given to annotators in the ranking (top) and chaining
campaigns (bottom).

##  Appendix E Example Output

System Summary | Chain IDs
---|---
Gold (Avg. ROUGE=-; RdRL=8.3; EGr=0.72) |
Why did Microsoft buy Nokia’s phone business? |  1,2
We now know Microsoft’s answer: the computing giant released a 30-slide presentation today arguing that the move will improve Microsoft’s margins on Windows phones, which will allow it to invest more in the platform, which will accelerate sales and market share growth, The Washington Post reports. |  1,2,3,5
But John Herrman at BuzzFeed has another explanation: "fear of dying alone." | 3
Here’s what he and other pundits are saying: the presentation "manages to sound both insane and uninspiring, outlining modest goals that still sound unrealistic," Herrman argues - like capturing a whole 15% of the smartphone market. | 2,3,5
"It’s a fitting end for the close of Microsoft’s Ballmer era, during which the company…missed out on the most important change in consumer electronics in decades" while remaining profitable in unglamorous ways. |  1,2,4
Like everyone, Microsoft is trying to ape the Apple model, MobileOpportunity observes. |  1,3
But it’s not so sure that’s a good idea. | 3
"There already is an Apple," the blog points out, and other software/hardware hybrid companies, like Palm and BlackBerry, have been crushed under its heel. |  1,3
Maybe Microsoft should have tried to patch up its tried-and-true strategy of licensing its OS. |  1,2
The move risks complicating Microsoft’s crucial relationships with other PC and device manufacturers, one analyst tells ZDNet. |  1,2,3
But he adds that "Microsoft needed to make a bold move" or face "certain terminal decline," and that the price it paid for Nokia "seems extremely reasonable." |  1,3
Meanwhile, Matthew Yglesias at Slate digs up a fairly interesting memo from Nokia CEO (and, perhaps, Microsoft heir apparent) Stephen Elop, in which he uses the story of a Deepwater Horizon worker leaping from the burning oil platform - a seemingly desperate, yet necessary move \- to explain the company’s shift from its own failed OS to Windows Phone. |  1,2,3,4,11
Of course, Yglesias notes, that move "was basically a total failure." |  3,11
MMR-Select (Avg. ROUGE=28.25; RdRL=8.31; EGr=0.79) |
Summary: Microsoft’s acquisition of Nokia is aimed at building a devices and services strategy, but the joint company won’t take the same form as Apple. |  1,2,10
This crawl was run at level 1 (URLs, including their embeds, plus the URLs of all outbound links, including their embeds). | 6
Today’s sale price, which includes 1.65 billion euros in patents, is just 5.44 billion euros. | 2,7
It’s been a rough decade. | -
Microsoft is buying Nokia’s cell phone business and licensing its patent portfolio, according to both companies. |  1,2
In 2003, Nokia’s cell phone market share exceeded 35%. |  1,2
That same year, its phone business alone posted an operating profit of 5.48 billion euros. | 2,7
Nokia lashed itself to Microsoft’s mast after losing out to iOS and Android in the smartphone market share stakes and with the limited success of the Lumia range so far, enough to keep interest in Windows Phone alive, most analysts are seeing a certain amount of inevitability to the acquisition, even if they are split on what its biggest implications are. |  1,2,8,10
The seed for this crawl was a list of every host in the Wayback Machine. | 6
The WARC files associated with this crawl are not currently available to the general public. | 6
Five years ago was the year the App Store first opened. | 2
Windows Phone has barely dented the now much larger smartphone market. | 2
Many at the time wondered if Stephen Elop’s time at Nokia would be spent grooming the company for purchase —a foreigner in all possible ways, he began his time at the company with a memo rightly but offensively declaring Nokia’s proud platform a failure and quickly pledged the company’s commitment to the still-tiny Windows Phone. |  1,2,4
KvD-Select (Avg. ROUGE=26.33; RdRL=12.48; EGr=0.88) |
Summary: Microsoft’s acquisition of Nokia is aimed at building a devices and services strategy, but the joint company won’t take the same form as Apple. |  1,2,10
Microsoft has been working on its evolution into a devices and services company, moving away from the services business it has traditionally been, for several years now with limited success. |  1,2,8
Nokia lashed itself to Microsoft’s mast after losing out to iOS and Android in the smartphone market share stakes and with the limited success of the Lumia range so far, enough to keep interest in Windows Phone alive, most analysts are seeing a certain amount of inevitability to the acquisition, even if they are split on what its biggest implications are. |  1,2,8,10
Owning the desktop (via Windows) and building additional services on top, like Office or Search, has been vital for Microsoft’s strategy until now, so, as our interest shifts from the desktop to the tablet or smartphone, it’s essential to Microsoft’s broader business (even Azure) that it can retain that connection in some form. |  1,2,9
But he said Microsoft’s challenge remains how to unite the myriad services and brands - Windows, Nokia, Live, Surface, Xbox, Bing, and more - into a cohesive experience that will command and cement customer loyalty. |  1,2,9
It felt like a radical about-face, but no matter: Nokia and Microsoft were going to save each other. |  1,9
Table 11: Reference summary, along with summaries extracted by MMR-Select and
KvD-Select for a MultiNews sample with informativeness (average ROUGE score),
redundancy (RdRL), and cohesion (EGr) scores. Each sentence is annotated with
lexical chains, color-coded in the text and IDs shown to the right. Text was
detokenized and truecased for ease of reading.