linear system. We can prove that the operator
$L(v)=\mathtt{R}_{\pi}+\gamma\mathtt{P}_{\pi}v$ is a contraction and,
according to the Banach fixed point theorem, it can be solved by fixed point
iterations161616See, for instance, Puterman (2014).. In order to improve the
policy, we use the Q-function $Q^{\pi}(s,a)=E_{\pi}[R_{t}/s_{t},a_{t}=a]$,
that is the expected return for taking action $a$ in state $s$ and thereafter
following an optimal policy. Thus, if $Q^{\pi}(s,a)>v^{\pi}(s)$, then $\pi$
can be improved if $\pi(s)$ is replaced by $a$. There are two popular
algorithms to solve this problem, namely policy iteration and improvement
(Watkins, 1989) and value iteration (Puterman and Shin, 1978). While in the
former approach the policy improvement and policy evaluation tasks are run in
separate loops, in the latter, these are carried out in the same loop.

  7. 7.

When the transition probabilities and rewards are not known, we use Monte
Carlo methods to sample sequences of states, actions, and rewards (Michie and
Chambers, 1968, Barto and Duff, 1993, Singh and Sutton, 1996). The policy
evaluation and policy improvement steps are completed using average returns of
episodic tasks.

  8. 8.

Temporal difference learning methods combine Monte Carlo and dynamic
programming approaches (Sutton, 1988). They learn from experience like the
Monte Carlo methods and they update estimates like the dynamical programming
approach. Two algorithms can be used to implement the temporal difference
approach, namely SARSA (an Acronym for State-Action-
Reward-[next]State-[next]Action) and Q-learning. While the Q-learning approach
updates the value function using a greedy action approach (Rummery and
Niranjan, 1994, Sutton, 1995), SARSA updates with the actual action used for
generating experience by the agent (Watkins, 1989).

Ryang and Abekawa (2012), in order to formulate the problem of extractive
summarization as a reinforcement learning problem, reduce the document to be
summarized in a set of $n$ sentences, define a score function for any subset
of sentences of the document $S\subset d$, where $S$ is one of the possible
summaries and $d$ is the document, and also define the length function that
indicates the size of the summary. They use a temporal difference learning
approach to find the summary that maximizes the score function that considers
a trade-off between relevance and redundancy subject to the maximal summary
length. Thus, in this problem, a state is a summary of a given length. In this
work, the actions are deterministic and are basically the decision to include
or not a given sentence. The reward is defined in such a way that the agent
only receives the rewards when the summary reaches the final size. The score
function specifically depends on the coverage of important words (the count of
top-100 words in terms of the TF-IDF included), coverage ratio (the count of
top-100 elements included), redundancy ratio (the counting of the number of
elements that excessively cover the top 100 elements), length ratio (the ratio
between the length of the summary and length limitation) and the sum of the
inverse position of the sentences.

We may find an extension of the work of Ryang and Abekawa (2012) in Rioux et
al. (2014). In a multi-document setup and also working with a query-based
task, they use the SARSA algorithm, explore different types of rewards (not
only delayed rewards as in the case of Ryang and Abekawa (2012)) and also use
the ROUGE to evaluate the similarity between the reference summary and the
automatic summary.

There are other reinforcement-based approaches for ATS systems. We present a
significant compilation of these methods in Table 12.

Source |  Main contribution |  Dataset |  Evaluation
---|---|---|---
Ryang and Abekawa (2012) |  It introduces a reinforcement learning model for extractive summarization. |  DUC 2004 |  ROUGE-1, ROUGE-2 and ROUGE-3
1-4 Rioux et al. (2014) |  In a multi-document approach, it extends the work of Ryang and Abekawa (2012) using the SARSA algorithm, considering different types of rewards and using the ROUGE as a measure of similarity. |  DUC 2004 and DUC 2006 |  ROUGE-1, ROUGE-2 and ROUGE-L
1-4 Henß et al. (2015) |  In a multi-document summarization setup, it extends the work of Ryang and Abekawa (2012) using the Q-learning algorithm and considering the reference summaries in the training phase. |  DUC 2001, DUC 2002, DUC 2004, ACL and Wikipedia |  ROUGE-2, ROUGE-2 and ROUGE-L
1-4 Mollá (2017) |  In order to deal with a query-based task, it extends the work of Ryang and Abekawa (2012) including the use of the reference summary in the evaluation of the reward. |  BioASQ 5b Phase B |  ROUGE-L
1-4 Lee and Lee (2017) |  It explores the advantages of embedding features in a Q-learning model and also designs a deep neural network to approximate the value function. |  DUC 2001, DUC 2002, ACL-ARC and Wikipedia |  ROUGE-2
1-4 Narayan et al. (2018a) |  It formulates the problem of extractive summarization as a sentence ranking problem using reinforcement learning and it uses a neural network model to encode the features of the sentences. |  CNN and Daily-Mail |  ROUGE-1, ROUGE-2 and ROUGE-L
1-4 Yao et al. (2018) |  It develops a neural network for both encoding the features of the sentences and also choosing the sentences that must be included in the summary. |  CNN and Daily-Mail |  ROUGE-1, ROUGE-2 and ROUGE-L
Table 12: A significant compilation of ATS systems that use reinforcement
learning techniques.

##  7 Abstractive summarization

Different from extractive summarization, which builds a summary from the
combination of the important sentences previously extracted from the original
text, in abstractive summarization, we need a language model to rewrite the
summary from scratch. We split this section into two subsections. While in
subsection 7.1 we present the ATS systems based on classical linguistic
models, in Subsection 7.2 we present the modern methods based on sequence-to-
sequence neural network models.

###  7.1 Linguistic approaches

As we have mentioned in Section 6.3, we may split the linguistic approaches to
abstractive summarization into two large groups, namely structured-based and
semantic-based (Saranyamol and Sindhu, 2014).

An example of the structured-based approach that uses a rule-based scheme is
Genest and Lapalme (2012). In this work, in order to provide summaries in the
fields of “Accidents and Natural Disasters”, “Attacks, Health and Safety”,
“Endangered Resources”, and “Investigations/Trials”, the authors use
handcrafted information extraction rules, content selection heuristics and
generation patterns. In particular, to extract the information they need to
build the summary, they ask the following questions: (1) What: what happened;
(2) When: date, time, other temporal placement markers; (3) Where: physical
location; (4) Perpetrators: individuals or groups responsible for the attack;
(5) Why: reasons for the attack; (6) Who was affected: casualties (death,
injury), or individuals otherwise negatively affected; (7) Damages: damages
caused by the attack; (8) Countermeasures: countermeasures, rescue efforts,
prevention efforts, other reactions. With the answers to these questions in
hand, extracted using predefined extraction rules, they use previously
generated patterns to build the summaries. The reader may note that this is a
particular example of an informative summary defined in Section 2.

An example of the structured approach that uses a template for multi-document
summarization is Harabagiu and Lacatusu (2002). In this work, the authors use
templates that represent each topic of the piece of text that need to be
summarized and are populated using information extraction rules. In order to
generate the summaries, they use a parse tree to identify the Subject-Verb-
Object structures171717In linguistic, subject–verb–object (SVO) is a sentence
structure where the subject comes first, the verb second, and the object
third. and WordNet to classify the topic of each structure. The topics with a
high frequency are included in the summary.

We may find an example of the structured-approach that uses a domain ontology
in Lee et al. (2005). In this work, the authors extend a domain ontology using
concepts of fuzzy logic by embedding a set of membership degrees in each
concept of the domain ontology. They use this fuzzy domain ontology to extract
the sentences related to the text domain and to generate the abstract by
concatenating concepts with relations.

We may find one of the first examples of the semantic-based approach in Genest
and Lapalme (2011). This work proposes the concept of Information Items
(INITs) to help to define the abstract representation, which is the smallest
element of coherent information in a text or a sentence. The goal is to
identify all entities in the text, their properties, the predicates between
them, and the characteristics of the predicates. In that work, the
implementation of INITs is constrained to dated and located
subject–verb–object (SVO) triples and the summary generation as above-
mentioned is carried out using parse trees. This work is another example of an
ATS that generates informative summaries.

Another interesting example of the semantic based approach is the multimodal
model of Greenbacker (2011). We may summarize the implementation of this model
in three steps: (1) Building the semantic model: The semantic model, which
should consider both images and pieces of text, is built based on a knowledge
representation based on a domain ontology; (2) Rating the informational
content: In order to rate the content, the authors propose the so-called
information density metric (ID) which rates a concept’s importance based on
factors such as completeness of attributes, the number of connections with
other concepts and the number of expressions that put the concept in evidence;
(3) Generating a summary: The work uses the concepts of TAG (Tree Adjoining
Grammars) Derivation Trees181818A TAG is a formalism that builds grammatical
representations through the composition of smaller pieces of syntactic
structure (Joshi and Schabes, 1997). as in McDonald and Greenbacker (2010) to
express the concepts and relationships found in previous steps.

Moawad and Aref (2012) implement the semantic-based approach using a semantic
graph. The method consists of three steps: (1) The creation of the semantic
graph called Rich Semantic Graph (RSG) for the original document; (2) The
reduction of the generated semantic graph; (3) The generation of the final
abstractive summary from the reduced semantic graph. In RSG, the verbs and
nouns of the input document are represented as graph nodes along with edges
corresponding to semantic and topological relations between them. The graph
nodes are instances of the corresponding verb and noun classes in the domain
ontology. The Rich Semantic Graph Reduction Phase aims to reduce the generated
rich semantic graph of the source document to a reduced graph. A model of
heuristic rules is applied to reduce the graph by replacing, deleting, or
consolidating the graph nodes using the WordNet relations. Finally, the
Summarized Text Generation Phase aims to generate the abstractive summary from
the reduced rich semantic graph. To achieve its task, this phase accesses a
domain ontology, which contains the information needed in the same domain of
RSG to generate the final texts.

Table 13 presents a significant compilation of ATS systems based on linguistic
abstractive summarization.

Source | Type |  Main contribution |  Dataset |  Evaluation
---|---|---|---|---
Harabagiu and Lacatusu (2002) | Structured-template |  It presents a multi-document summarization approach that uses a template to identify the important pieces of information, a tree parser to build subject-verb-object structures, the WordNet to identify the topics and generates the abstract based on the high-frequency structures. |  DUC 2002 |  Human experts
1-5 Lee et al. (2005) | Structured-domain ontology |  It extends a domain ontology using concepts of fuzzy logic. It uses this domain ontology to extract the sentences and to generate the abstract |  Chinese news of three weather events, including “Typhoon event,” “Cold current event,” and “Rain event,” from the Chinatimes website in 2002, 2003, and 2004. |
1-5 Genest and Lapalme (2011) | Semantic-INITs |  It extracts sentences using a variety of methods and produces the summary with these sentences using a parse-tree. |  Text Analysis Conference (TAC) 2010 |  Precision and Recall evaluated by human experts
1-5 Greenbacker (2011) | Semantic-multimodal |  It implements the model in three steps: (1) Building the semantic model based on a domain ontology; (2) Rating the information content based on completeness of attributes, connections with other concepts and the number of expressions that put the concept in evidence; (3) Generating the summary using TAGs. |  An article from the May 29, 2006 edition of Businessweek magazine entitled, “Will Medtronic’s Pulse Quicken?”. |  Human experts.
1-5 Genest and Lapalme (2012) | Structured-rule |  It uses handcrafted information extraction rules, content selection heuristics and generation patterns. |  Attack category of Text Analysis Conference (TAC) 2011 (Owczarzak and Dang, 2011) |  Manual Pyramid
1-5 Moawad and Aref (2012) | Semantic-graph |  It implements the model in three steps: (1) It creates the semantic graph; (2) It reduces the generated semantic graph; (3) It generates the final abstractive summary from the reduced semantic graph. |  A simulated case study called “Graduate students”. |  Text coherence using the original words of the sentences and using the synonyms of the words of the sentences.
Table 13: A significant compilation of ATS systems based on linguistic
abstractive summarization.

###  7.2 Sequence-to-sequence deep learning methods

Sequence-to-sequence deep learning models used for tasks of NLP are NN models
in which the inputs and outputs are sequences of tokens of varying sizes. In
order to explore the most recent approaches of sequence-to-sequence models we
need to trace back to the first architectures of Recurrent Neural Networks
(RNN) (Jordan, 1986, Elman, 1990). RNNs are sequence models that deal directly
with two modeling constraints of the standard multilayer perceptrons that are
essential to model sequences. First, it is assumed that the input sequences
have the same length. Second, it’s an architecture that does not allow for the
same token in different positions of the text to have similar features. The
first ideas of RNNs arise in the seminal works of Jordan (1986) and Elman
(1990). We may summarize the vanilla RNN by the set of equations:

$s^{i}=g(W_{ss}s^{i-1}+W_{sx}x^{i}+b_{s})$ (9)

and

$y^{i}=g(W_{ys}s^{i}+b_{y}),$ (10)

where $x^{i}$ is a token in a piece of text, $y^{i}$ is the token we want to
predict, $s^{i}$ is the state of the RNN, $W_{ss}$, $W_{ax}$, $W_{ys}$,
$b_{s}$ and $b_{y}$ are the parameters of the network that we need to learn
and $s^{0}$ is a vector of zeros. Figure 3 represents this model. Like the
other models of language, we try to predict the probability of the next word.
Therefore, we may estimate the parameters of this model in a semi-supervised
fashion using the backpropagation through time algorithm. Suppose, for
instance, that your text includes the first sentence of the song Africa by the
band Toto “I hear the drums echoing tonight.”. Thus, $x^{1}=\mathrm{``I"}$,
$x^{2}=\mathrm{``hear"}$ $x^{3}=\mathrm{``the"}$, $x^{4}=\mathrm{``drums"}$,
$x^{5}=\mathrm{``echoing"}$, $y^{1}=\mathrm{``hear"}$ $y^{2}=\mathrm{``the"}$,
$y^{3}=\mathrm{``drums"}$, $y^{4}=\mathrm{``echoing"}$ and
$y^{5}=\mathrm{``tonight"}$. Thus, this algorithm tries to maximize the
probability that the token “hear” arises when the token “I” is an input, to
maximize the probability the token “the” happens when ”hear” is the input and
$s_{1}$ is the state of the system generated by “I” and so on. Unfortunately,
vanilla RNNs are not good at dealing with long sequences. Problems that arise
in this context are the so-called exploding and vanishing gradients (Bengio et
al., 1993, Pascanu et al., 2013, Ribeiro et al., 2020). This happens naturally
due to the algorithm of backpropagation, which is based on the chain rule of
calculus, and the consequent multiplication of the same shared matrix of the
parameters several times.

Figure 3: Vanilla
RNN.$s^{<0>}$$s^{<1>}$$s^{<2>}$…$s^{<T_{x}-1>}$$\hat{y}^{<1>}$$\hat{y}^{<2>}$$\hat{y}^{<3>}$$\hat{y}^{<T_{y}>}$$\hat{x}^{<1>}$$\hat{x}^{<2>}$$\hat{x}^{<3>}$$\hat{x}^{<T_{x}>}$

In order to overcome the difficulties of exploding and vanishing gradients,
two important models of RNNs were introduced, namely Long Short-Term Memory
(LSTM) (Hochreiter and Schmidhuber, 1997, Gers et al., 2000) and Gated
Recurrent Units (GRU) (Cho et al., 2014) empirically explored in Chung et al.
(2014). The basic idea behind these models is to replace the simple units of
the vanilla RNN model with complex units which include gates that control the
flow of information that passes from one unit to the other.

A natural extension of the vanilla RNNs is to consider the case where the
input sequence and output sequence have different sizes. These models are
called Encoder-Decoder models and they aim at mapping one sequence to another
sequence (Sutskever et al., 2014, Vinyals and Le, 2015), where the encoder
(decoder) is the part of the model that deals with the input (output)
sequence. Figure 4 presents an example of this topology, where each unit of
this model is the RNN unit (vanilla, LSTM or GRU). In this type of model, the
intention is, given a sequence, to predict another sequence not necessarily of
the same size. For example, consider that we want to use a sequence-to-
sequence model to translate the first sentence of the song “Smoke on the
Water” by the English band Deep Purple to Spanish. We may have $x^{1}=$“We”,
$x^{2}=$“all”, $x^{3}=$“came”, $x^{4}=$“out”, $x^{5}=$“to”,
$x^{6}=$“Montreux”, $y^{1}=$“Todos”, $y^{2}=$“salimos’”, $y^{3}=$“a” and
$y^{4}=$“Montreux” in Figure 4, where $T_{x}=6$ and $T_{y}=4$.

A fundamental contribution to improving the learning process of sequence-to-
sequence models is the attention mechanism (Bahdanau et al., 2014). The main
idea behind this paper is to include a set of weights to inform the model
about the context. For instance, consider again the sentence “We all came out
to Montreux”. We know that the phrasal verb “come out” has different meanings.
However, in this sentence, it is obvious that the sense of this verb is “to go
somewhere” and this happens because of the word “Montreux”, which is a town in
Switzerland. In the work by Bahdanau et al. (2014), the authors evaluate the
attention mechanisms by normalizing the output of a multilayer perceptron that
depends on the state of the decoder model and the activation signal that comes
from the encoder.

Figure 4: Sequence-to-sequence
models.……$\hat{y}^{<1>}$$\hat{y}^{<T_{y}>}$$\hat{x}^{<1>}$$\hat{x}^{<T_{x}>}$

The most recent models of NLP are based on transformers. Transformers are
networks models that comprise the idea of processing complex information in
parallel191919This idea comes from the architecture of the Convolution Neural
Network (CNN) that processes complex information in parallel (LeCun et al.,
2015). and an extension of the above-mentioned attention mechanism called
multi-head attention (Vaswani et al., 2017). Multi-head attention is the idea
of evaluating several attention models simultaneously. In a given sentence, we
may use several attention mechanisms to explore several different dimensions
at once. For instance, the chorus of the song “America” by Neil Diamond says
“They’re coming to America today”. We may think of a multi-head mechanism as a
way to help us ask and answer questions, i.e., it says where we should pay
attention in a sentence to answer a given question202020It has the same role
as the filters built automatically by NN models to identify dimensions of
images in CNN.. Thus, the first question could be: “What is happening?”. Then,
the answer is “They are coming somewhere.”. The second question could be
“Where?”. Then, the answer is “America”. The third question could be “When?”.
Then, the answer is “Today”. In the transformer network, this information is
encoded in vectors called Queries ($Q$), Keys ($K$), and Values ($V$). Like in
the recurrent sequence-to-sequence models, the architecture of Transformer
models has an encoder-decoder structure. The encoder model receives the
embeddings (discussed in Section 6.1.5) of the inputs with a position
encoding. The position encoding serves to inform the position of the token in
the sequence, which is necessary here since this is a parallel model where the
entire text is simultaneously inputted. The encoder block is formed by a stack
of multi-head attention blocks connected with a feedforward network to
generate the vectors $Q$, $K$ and $V$ that feed the following encoder blocks.
The decoder block is a stack of an additional multi-head attention block and a
block similar to the encoder block. The first block receives the output
embeddings and generates the vector $Q$ to be used together with the vectors
$K$ and $V$ it receives from the encoder block. This model is used to generate
the output sequence in a recursive fashion.

There is now a large list of transformers that have been used in successful
NLP tasks such as Google’s BERT (Devlin et al., 2018), PEGASUS (Zhang et al.,
2020), T5 (Raffel et al., 2019), and Switch (Fedus et al., 2021), Facebook’s
BART (Lewis et al., 2019), and Open AI’s Generative Pre-Training (GPT)
(Radford and Narasimhan, 2018), GPT-2 (Radford et al., 2019), and GPT-3 (Brown
et al., 2020).

An extension of these models is the so-called Longformer (Beltagy et al.,
2020), which is a modified Transformer architecture with a self-attention
operation that scales linearly with the sequence length, making it versatile
for processing long documents. Finally, it is worth mentioning that we may
find a survey of pre-trained language models for text generation in Li et al.
(2021a) and a survey of dynamic neural network models for natural language
processing in Xu and McAuley (2022).

Table 14 presents a summary of the most popular transformers used for
abstractive summarization with their main characteristics.

Name | Source |  Main characteristics |  Dataset |  Evaluation
---|---|---|---|---
BERT | Liu and Lapata (2019) |  It is a bidirectional encoder model. |  CNN-DailyMail, XSUM and NYT |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 BART | Lewis et al. (2019) |  It is a bidirectional encoder model with an autoregressive decoder architecture. |  CNN-DailyMail and XSUM |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 T5 | Raffel et al. (2019) |  It is essentially the original transformer model equivalent to Vaswani et al. (2017) with the normalization layer outside the residual path, and using a different position embedding scheme. |  CNN/DailyMail |  ROUGE-2
1-5 GPT | Radford et al. (2019) |  It is essentially the original transformer model equivalent to Vaswani et al. (2017). |  CNN/DailyMail |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 UniLM | Dong et al. (2019) |  It is a transformer model shared among three different self-attention masks. |  CNN/DailyMail |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 MASS | Song et al. (2019) |  It is a transformer model designed for encoder-decoder-based language generation tasks. |  Gigaword |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 UniLMv2 | Bao et al. (2020b) |  It is a transformer model shared among three different self-attention masks. |  CNN/DailyMail and XSUM |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 PEGASUS | Zhang et al. (2020) |  It is essentially the original transformer model equivalent to Vaswani et al. (2017). |  XSUM, CNN/DailyMail, NEWSROOM, Multi-News, Gigaword, arXiv, PubMed, BIGPATENT, WikiHow, Reddit TIFU, AESLC and BillSum |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 Ernie-Gen | Xiao et al. (2020) |  It is a transformer model with parameters shared among different tasks. |  Gigaword and CNN/DailyMail |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 ProphetNet | Qi et al. (2020) |  It is a transformer model in the so-called future n-gram prediction as described in |  Gigaword and CNN/DailyMail |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 Longformer | Beltagy et al. (2020) |  It is a modified Transformer model with a self-attention operation that scales linearly with the sequence length. |  arXiv |  ROUGE-1, ROUGE-2 and ROUGE-L

Table 14: A significant compilation of transformers used for abstractive
summarization.

Notes: BERT due to (Devlin et al., 2018) and trained by Liu and Lapata (2019)
is the acronym for Bidirectional Encoder Representations from Transformers.
GPT due to Radford and Narasimhan (2018) stands for Generative Pre-Trained
model. BART due to Lewis et al. (2019) is the acronym for Bidirectional and
Auto-Regressive. Pegasus due Vaswani et al. (2017) and trained by Zhang et al.
(2020) is the acronym for Pre-training with extracted gap-sentences for
abstractive summarization. T5 due to Raffel et al. (2019) stands for Text-to-
Text Transfer Transformer. UniLM due to Dong et al. (2019) stands for Unified
Language Model. MASS (Song et al., 2019) stands for MAsked Sequence-to-
Sequence.

##  8 Compressive extractive approaches

Compressive extractive approaches usually depend on two steps: (1) We extract
the sentences that should belong to the summary; (2) We compress the chosen
sentences that come from the original text in order to present only essential
information.

When selecting the sentences to be extracted, we usually rely on one of the
methods already discussed in Section 6. On the other hand, in order to
compress the sentences we need a model of language.

One advantage of the compressive extractive approaches over the pure
extractive approaches is that for the case of the summaries that need to have
a previously given constant length, the summaries created with the compressive
approach usually contain more information than the summaries created with
extractive approaches. This happens because by removing insignificant sentence
components, we make room for more relevant information in the summary.

We have discussed many methods to extract sentences from the whole text in
Section 6. On the other hand, sentence compression is by itself a research
problem. It starts with the input which is a sentence with $n$ words. The
algorithm for sentence compression may drop any subset of these words and the
words that remain with an unchanged order form a compression. Thus, the
problem is not trivial since there are $2^{n}$ possible pieces of text to be
chosen (Clarke and Lapata, 2006). In order to solve this problem, we have to
develop a method to determine what is the relevant piece of information in a
sentence and how to present this information grammatically.

Although our focus here is on text summarization, it is worth mentioning that
there are some interesting specific applications of solutions of the sentence
compression (Knight and Marcu, 2000), such as the generation of TV captions
that due to time and space constraints often requires only the most important
parts of sentences (Zdenek, 2011) and audio scanning services for the blind
(Grefenstette, 1998).

The precursors of compressive extractive summarization were the early attempts
to provide text summaries in the style of newspapers headlines as in Witbrock
and Mittal (1999) and Banko et al. (2000), where the summaries consist of a
single sentence or even less than a sentence extracted from the text.

In order to implement the compressive step of compressive extractive
summarization, we may use unsupervised, supervised methods or hybrid
approaches.

The unsupervised approaches delete words based on part-of-speech tags212121In
linguistics, Part-Of-Speech (POS) tags, are tags used in a text (corpus) to
associate a given word with its corresponding part of speech, based on both
its definition and its context. The list of universal POS tags is: ADJ
(adjective), ADP (adposition), ADV (adverb), AUX (auxiliary), CCONJ
(coordinating conjunction), DET (determiner), INTJ (interjection), NOUN
(noun), NUM (numeral), PART (particle), PRON (pronoun), PROPN (proper noun),
PUNCT (punctuation), SCONJ (subordinating conjunction), SYM (symbol), VERB
(verb) and X (other). or the lexical items222222A lexical item may be a single
word, a part of a word, or a sequence of words that forms the basic elements
of a language’s vocabulary. Examples of lexical items are: words (dog, table),
phrasal verbs (get up, get over, get in, get on, idioms (“better late than
never”, “pull yourself together”) and sayings (“An apple a day keeps the
doctor away.”, “Actions speak louder than words.”). alone. In particular, we
may find a very interesting approach for unsupervised compressive
summarization in Hori and Furui (2004). In order to generate a summary, their
approach focuses on extracting topic words, weighting correct-word
concatenations linguistically, and extracting reliable components of speech
recognition acoustically as well as linguistically. A set of words maximizing
a summarization score, indicating the appropriateness of a summarized
sentence, is selected from those using a Dynamic Programming (DP) technique.
The summarization score consists of word significance measured by the
frequency of each word in the sentence, word confidence measured by the
logarithm of the probability of $n$-grams, and the linguistic likelihood of
summarized sentences.

Another interesting approach is the graph-based approach due to Filippova
(2010) where a directed word graph is constructed. In this digraph, nodes
represent words and edges represent the adjacency between words in a sentence.
Thus, the authors can compress sentences by finding the k-shortest paths in
the digraph.

The supervised approaches may depend on a number of resources such as an
annotated corpus with the original sentences and their corresponding reduced
forms written by humans for training and testing purposes, a lexicon232323A
lexicon is the vocabulary of a language or a subject. For instance, the
lexicon of computer science must present keys such as “algorithm”, “big data”,
“class”, “design pattern” and so on. and a syntactic parser to generate a
parse tree242424A syntactic parsing converts the sentence into a tree whose
leaves hold POS tags, but the rest of the tree tells how exactly these words
join together to make the complete sentence. For example, a linking verb and a
verb may combine to be a Verb Phrase (VP) such as in “I have been studying
English for years.”, where the underlined piece of text is a verb phrase..

Jing (2000) focuses specifically on the problem of sentence reduction of
extracted sentences for summarization. The author assumes that the input of
his system is the collection of extracted sentences that we can build using
one of the methods of Section 6. On the other hand, his algorithm of sentence
reduction has five steps: (1) Syntactic parsing: He parses the input sentence
to produce the sentence parse tree; (2) Grammar checking: He determines which
components of the sentence must not be deleted to keep the sentence
grammatical. To do this, he traverses the parse tree generated in the first
step in top-down order and marks, for each node in the parse tree, which of
its children are grammatically obligatory; (3) Context information: The system
decides which components in the sentence are most related to the main topic
being discussed. To measure the importance of a phrase in the local context,
the system relies on lexical links between words; (4) Corpus evidence: The
program uses the annotated corpus consisting of sentences reduced by human
professionals and their corresponding original sentences to compute how likely
(measuring the probabilities) are humans to remove a certain phrase; (5) Final
decision: The final reduction decisions are based on the results from all the
earlier steps. To decide which phrases to remove, the system traverses the
annotated sentence parse tree and removes a phrase when it is not
grammatically obligatory, not the focus of the local context and has a
reasonable probability of being removed by humans.

Another interesting approach to supervised compressive summarization explores
the noisy channel framework (Knight and Marcu, 2000). In this framework, the
authors consider that every sentence was originally shorter and then someone
added some additional noisy text to it. Thus, the task of compressive
summarization is to find the original sentence. It is worth mentioning that it
is not relevant here whether or not the “original” string is real or
hypothetical. In this approach, the authors split the problem of sentence
compression into three sub-problems: (1) Source model: They assign to every
string (sentence) $s$ a probability $P(s)$, which gives the chance that $s$ is
generated as an “original short string”; (2) Channel model: They assign to
every pair of strings (sentences) $s$ and $t$ a probability $P(t|s)$, which
gives the chance that the expansion of the short string $s$ results in the
long string $t$; (3) Decoder: They search for the short string s that
maximizes $P(s|t)=P(s)P(t|s)$. In order to implement this, they assume that
the probabilities $P(s)$ and $P(t|s)$ are associated with the representation
of these sentences using parse trees.

In Cheng and Lapata (2016) and Zhang et al. (2018) different from the above-
mentioned works deal with both the extractive and compressive steps using
neural network models such as the ones presented in Section 7.2.

Although the most common compressive approaches focus on editing sentences
using compressive operations, other operations are also possible. Jing and
McKeown (2000), based on analysis of human written abstracts, call attention
to different types of operations such as sentence combination (merging
material from several sentences), syntactic transformation (for instance, to
change the position of the subject or to transform a piece of text from
passive voice to active voice), lexical paraphrasing (replacing phrases with
their paraphrases), generalization or specification (replacing phrases or
clauses with more general or specific descriptions) and reordering (changing
the order of specific sentences). However, besides compressing, they only
implement the fusion of sentences when two sentences are close to each other
and share the same subject or when a person or an entity is mentioned for the
first time in a summary and there is a description of this person or entity in
the text. Another approach is due to Ganesan et al. (2010) that, using a
graph-based approach where each word is a node in the graph, executes both
compressive and fusion tasks of highly redundant sentences.

Table 15 presents a significant compilation of ATS systems based on
compressive extractive summarization.

Source | Type |  Main contribution |  Dataset |  Evaluation
---|---|---|---|---
Jing and McKeown (2000) | Unsupervised |  It calls attention to other types of editions besides the compressive ones and it implements a limited set of sentence fusions. |  305 sentences from 50 summaries |  Human experts
1-5 Jing (2000) | Supervised |  It provides an algorithm based on 5 steps: (1) Synctatic parsing; (2) Grammar checking; (3) Context information; (4) Corpus evidence; (5) Final decision. |  Free daily news service “Communications-related headlines”, provided by the Benton Foundation252525http://www.benton.org. |  Percentage of system decisions that agree with human decisions
1-5 Knight and Marcu (2000) | Supervised |  It assumes that every sentence was originally shorter and then someone added some additional noisy text to it. It splits the problem of sentence compression into three sub-problems: (1) Source model: it assigns to every sentence probability that it was generated as an “original shorter string”; (2) Channel model: it assigns to every pair of sentences the probability that one is the expansion of the other; (3) Decoder: They search for the short strings that maximize the product of the two previous probabilities. |  The Ziff-Davis corpus, which is a collection of newspaper articles announcing computer products. |  Human experts
1-5 Hori and Furui (2004) | Unsupervised |  It uses a dynamic programming approach to maximize a summarization score that depends on the word significance, word confidence and the linguistic likelihood of summarized sentences. |  Japanese news broadcasts on TV in 1996. |  Summarization accuracy
1-5 Filippova (2010) | Unsupervised |  It builds a directed graph where nodes represent words and edges represent the adjacency between words in a sentence and compresses the sentences finding the k-shortest paths in this digraph. |  News articles presented in clusters on Google News. |  Human experts
1-5 Ganesan et al. (2010) | Unsupervised |  It provides a graph based algorithm based on two kinds of operations, namely compression and fusion. |  Reviews of hotels, cars and other products collected from Tripadvisor, Amazon and Edmunds. |  ROUGE-1, ROUGE-2, ROUGE-SU-4
1-5 Cheng and Lapata (2016) | Supervised |  It develops neural network models (LSTMs) to extract sentences and words in a supervised fashion as in Section 6.4. |  DUC 2002 and CNN/Dailymail |  ROUGE-1, ROUGE-2 and ROUGE-L
1-5 Zhang et al. (2018) | Supervised |  It develops neural network models (LSTMs) to extract sentences and words in a supervised fashion as in Section 6.4. One particular contribution of this paper is to use the human-generated summaries in the training process. |  CNN/Dailymail |  ROUGE-1, ROUGE-2 and ROUGE-L
Table 15: A significant compilation of ATS systems based on compressive
summarization.

##  9 Evaluation methods

The literature divides the methods for evaluating the quality of the generated
summaries by the so-called intrinsic and extrinsic methods (Jing et al., 1998,
Steinberger et al., 2009). While intrinsic methods measure the quality of the
summary, extrinsic methods measure a summary’s performance when involved in a
particular task such as document categorization and question answering. We may
divide the intrinsic methods into three groups: text quality evaluation,
content-based evaluation and hybrid. Text quality evaluation focuses on the
readability of the summary. On the other hand, content-based evaluation
considers the performance of the method according to the chosen words. The
hybrid methods consider both worlds. We may split the content group into three
subgroups: free-reference based, co-selection and content-based. Free-
reference based methods evaluate the content of the summaries without the need
of a human-made reference. Co-selection evaluators pay attention to the
sentences that were selected using metrics that come from the field of
information retrieval. Finally, content-based evaluators focus on the
selection of words. Considering the methods that need a human-made reference,
content-based evaluators are much more popular than co-selection evaluators,
since the former can be naturally applied to both extractive and abstractive
summarization. The application of the latter makes more sense in extractive
summarization.

Table 16 presents a digest of the methods used to evaluate ATS. DUC 2005
readability arises in this table as a quality-based approach. This entry
refers to the DUC 2005 that partially used this technique to evaluate the
summaries presented by the competitors (Dang, 2005). This method evaluates the
quality of the summary using the following dimensions: grammatical (the
summary should not have grammatical errors), non-redundancy (the summary
should not have unnecessary repetition), referential clarity (the summary
should be easy to identify who or what the pronouns and noun phrases in the
summary are referring to), focus (the summary should contain only sentences
with information that is related to the rest of the summary), structure and
coherence (the summary should be well-structured and well-organized). Based on
these dimensions, humans experts evaluate the summaries using a scale that
ranges from very Poor, poor, barely acceptable, good to very good. TAC 2008
also uses a quality-based approach and it considers exactly the same
dimensions and scale of DUC 2005 readability (Dang et al., 2008).

Another interesting attempt to provide a quality-based approach is due to
Grusky et al. (2018). The authors select two semantic dimensions, namely
informativeness and relevance, and two syntactic dimensions, namely fluency
and coherence, for evaluation. They ask Amazon Mechanical Turk crowd workers
to answer the following questions about these dimensions: (1) Informativeness:
“How well does the summary capture the key points of the article?” ; (2)
Relevance “Are the details provided by the summary consistent with details in
the article? ”; (3) Fluency: “Are the individual sentences of the summary
well-written and grammatical?”; (4) Coherence: “Do phrases and sentences of
the summary fit together and make sense collectively?”.

The idea considered in Pitler and Nenkova (2008) is to combine lexical,
syntactic, and discourse features to produce a predictive model of human
readers’ judgments of text readability. In this context, Xenouleas et al.
(2019) propose Sum-QE, a Quality Estimation model that adds a task-specific
layer to a pre-trained BERT model, which is fine-tuned on the task of
predicting scores for the five linguistic qualities assessed in DUC 2005. The
model achieves a high correlation with human scores by addressing linguistic
quality aspects that are only indirectly captured by recall-oriented
evaluation metrics.

There is a bunch of free-reference-based methods. The most simple methods are
based on the classical distribution divergences. For instance, we may evaluate
the divergence between $n$-grams of the candidate summary and the document
using the Kullback–Leibler divergence given by:

$D_{\text{KL}}(P\parallel
Q)=\sum_{x\in\mathcal{X}}P(x)\log_{2}\left(\frac{P(x)}{Q(x)}\right),$ (11)

where $P(x)$ is the probability of an event $x$ (the appearance of an
$n$-gram) in the document and $Q(x)$ is the probability of an event in the
candidate summary (Louis and Nenkova, 2013). Another option is to use the
Jensen–Shannon divergence, as in FRESA (Torres-Moreno et al., 2010) or Louis
and Nenkova (2013), that symmetrizes the evaluation of the KL divergence:

${\rm JSD}(P\parallel Q)=\frac{1}{2}D_{\text{KL}}(P\parallel
M)+\frac{1}{2}D_{\text{KL}}(Q\parallel M).$ (12)

SummTriver (Cabrera-Diego and Torres-Moreno, 2018) considers the possibility
of more than one candidate summary and it uses this piece of information to
evaluate the trivergence among the distribution of an event in the candidate
summary, the distribution of an event in a document formed by the set of
candidate summaries and the distribution of an event in the document. It may
evaluate the trivergence using different compositions of the divergences
between two of these given probabilities. We may also use the Hellinger
distance (Pollard, 2002, González-Castro et al., 2013) to evaluate the
distance between two distributions, given by:

$H(P,Q)=\frac{1}{\sqrt{2}}\sqrt{\sum_{x\in\mathcal{X}}(\sqrt{P(x)}-\sqrt{Q(x)})^{2}},$
(13)

where $0\leq H(P,Q)\leq 1$.

Furthermore, there are also other possibilities for evaluating the similarity
between a candidate summary and the original text. For instance, we may
evaluate the similarity by cosine of the representation of the text and
candidate summaries using vector space models (Louis and Nenkova, 2013) as
discussed in Section 6.1.1 or word embeddings (Sun and Nenkova, 2019) as
discussed in Section 6.1.5. Louis and Nenkova (2013) also suggests splitting
the text and candidate summary into topics and to evaluate the quality of the
summary by the fraction of the summary composed of text topic signatures or
the percentage of topic signatures from the text that also arise in the
summary. The summary likelihood approach considers the candidate summary as
being generated according to word distributions in the document. Louis and
Nenkova (2013) suggests two different ways to evaluate the likelihood of the
summaries, namely the unigram probability model and the multinomial
probability model. Another approach to free-reference-based evaluation is
SUPERT (SUmmarization evaluation with Pseudo references and bERT) presented in
Gao et al. (2020). In this work, the authors measure the relevance of a
candidate summary by comparing it with a pseudo-reference summary. Finally,
Bao et al. (2020a) uses the idea of transfer learning (Zhuang et al., 2020) to
train a neural network model using data from a dataset that has human-made
summaries to provide a score to a summary of a document that does not have
human-made summaries.

Two co-selection methods arise in Table 16. The information retrieval-based
approach, which considers the measures of precision, recall and F-score
(Baeza-Yates and Ribeiro-Neto, 2008) and the relative utility approach (Radev
et al., 2004). In order to apply the information retrieval approach to
evaluate the generated extractive summaries, we suppose that the objective of
the method is to recover the sentences that are part of the human-generated
summaries. Thus, we call the sentences that appear in the human-generated
summaries True and the ones that do not, False. With that in mind, we are able
to evaluate the precision, recall and F-metric. The precision is the ratio
between the number of True sentences recovered by the ATS systems and the
number total of sentences recovered by it as in Eq. (14):

$\text{precision}=\frac{|\\{\text{sentences in humans'
summary}\\}\cap\\{\text{ATS systems' retrieved sentences}\\}|}{|\\{\text{ATS
systems' retrieved sentences}\\}|}.$ (14)

The recall is the ratio between the number of True sentences recovered by the
automatic summarizer and the total number of sentences that should be
recovered according to the human-generated summary as in Eq. (15):

$\text{recall}=\frac{|\\{\text{sentences in humans'
summary}\\}\cap\\{\text{ATS systems' retrieved
sentences}\\}|}{|\\{\text{sentences in humans' summary}\\}|}.$ (15)

The $F$-measure is the harmonic mean of precision and recall (Baeza-Yates and
Ribeiro-Neto, 2008) as in:262626An extension of the $F$-measure is the
$F_{\beta}=(1+\beta^{2})\frac{\mathrm{precision}\times\mathrm{recall}}{\beta^{2}\mathrm{precision}+\mathrm{recall}}$,
where $\beta$ measures the relative importance between precision and recall.
If precision is as important as recall, we choose $\beta=1$ and the recover
the harmonic mean.

$F=2\frac{\mathrm{precision}\times\mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}.$
(16)

The relative utility approach is an extension of the idea of using the
information retrieval approach to evaluate summaries (Radev et al., 2003). The
starting point of this method is to assume that the human-generated summaries
present the level of confidence that a given sentence should belong to the
summary. With these weights in hand, we are able to evaluate the relative
utility, which is the ratio between the weighted importance of the sentences
recovered by the automatic summarizer and the weighted importance of the
sentences that should be recovered by the automatic summarizer. In order to
define this measure mathematically, we suppose that there are $N$ human
experts, the document to be summarized has $n$ sentences and $e$ is the number
of sentences in the extracted summary. Let $\delta_{sj}$ be the characteristic
function that equals 1 when sentence $j$ belongs to the extracted summary of
the ATS system and that equals 0 otherwise. Let also $\epsilon_{j}$ be the
characteristic function that equals 1 when sentence $j$ belongs to the summary
built based on the top $e$ sentences according to the average utilities of all
judges and equals 0 otherwise. Thus, we may define the relative utility as

$S=\frac{\sum_{j=1}^{n}\delta_{sj}\sum_{i=1}^{N}u_{ij}}{\sum_{j=1}^{n}\epsilon_{j}\sum_{i=1}^{N}u_{ij}}.$
(17)

We may note this is a kind of weighted recall.

As we can see in Table 16, there are several methods to evaluate ATS using the
content-based approach.

Factoid (Van Halteren and Teufel, 2003) and the Pyramid method (Nenkova and
Passonneau, 2004) are the content-based manual approaches in Table 16. In both
cases, the human evaluators assess whether the information available in the
human-made summaries is also in the automated summaries by comparing the
content units of both texts. A factoid is a pseudo-semantic representation
based on atomic information units that can be manually and robustly marked in
the text. On the other hand, a pyramid presents the distinct units of text
found in several reference summaries and the importance of these distinct
units based on how many human-made summaries they occur.

The cosine similarity is a natural approach to compare the tokens generated by
the ATS and the tokens of the human-made summaries when we represent the
summaries using the vector space model discussed in Section 6.1.1. In this
approach, we evaluate the cosine between the vectors that represent each
summary.

The unit overlap approach basically evaluates the Jaccard distance
(Levandowsky and Winter, 1971) between the words that occur in the human-made
and computer-made summaries (Saggion et al., 2002b) as in

$\text{unit overlap}=\frac{|A\cap B|}{|A|+|B|-|A\cap B|},$ (18)

where $A=\\{\text{sentences in humans' summary}\\}$ and $B=\\{\text{ATS
systems' retrieved sentences}\\}$.

BLEU (BiLingual Evaluation Understudy) is a metric that was originally created
for automatically evaluating machine-translated text. It measures the overlap
of $n$-grams between an automatically generated summary and a reference
summary (Papineni et al., 2002) in a precision fashion as in

$\text{BLEU}=\frac{|\\{n\text{-grams}\in A\\}\cap\\{n\text{-grams}\in
B\\}|}{|n\text{-grams}\in B|},$ (19)

where $A=\\{\text{humans' summary}\\}$ and $B=\\{\text{ATS systems'
summary}\\}$.

The next measure is the size of the Long Common Subsequence (LCS)272727Let
$X=\\{x_{1},x_{2},\cdots,x_{m}\\}$ and $Y=\\{y_{1},y_{2},\cdots,y_{n}\\}$. We
say the $Y$ is a subsequence of $X$ if there exists a strictly increasing
sequence of indexes $\\{i_{1},i_{2},\cdots,i_{K}\\}$ such that for all
$j\in\\{1,2,\cdots,K\\}$ we have $x_{i_{j}}=y_{j}$. Given two sequences $X$
and $Y$, the Longest Common Subsequence (LCS) of $X$ and $Y$ is a common
subsequence with maximum length. between the strings formed by the human-made
and computer-made summaries (Crochemore and Rytter, 1994). LCS is evaluated
using dynamic programming (Cormen et al., 2022).

ROUGE (Recall Oriented Understudy for Gisting Evaluation) is a general
approach to evaluating ATS systems. Since it comprises many different
approaches, it is one of the most popular. ROUGE-$n$282828Lin and Bilmes
(2011) calls attention that one interesting property of the ROUGE-$n$ is
submodularity suggesting that if this is an important way to evaluate
summaries, then optimal submodular approaches such as in their paper, Lin et
al. (2009) and Kågebäck et al. (2014) are well justified.  measures the
overlap of $n$-grams between an automatically generated summary and reference
summary as in

$\text{ROUGE-}n=\frac{|\\{n\text{-grams}\in A\\}\cap\\{n\text{-grams}\in
B\\}|}{|n\text{-grams}\in A|},$ (20)

where $A=\\{\text{humans' summary}\\}$ and $B=\\{\text{ATS system'
summary}\\}$. Although Lin and Hovy (2003) originally created it in a recall
fashion as in Eq. (20), the available softwares usually also present the
precision fashion of it (which is the BLEU in Eq. (19)) and the $F$-measure,
which is the harmonic mean between the precision and recall as shown in Eq.
(16).

It is worth mentioning that there are other commonly used variations of ROUGE
(Lin, 2004). ROUGE-$L$ intends to consider the length of the LCS such as in
Crochemore and Rytter (1994). In this case, the LCS is measured in precision
and recall fashions

$\text{precision}_{L}=\frac{LCS(\text{humans' summary},\text{ATS system'
summary})}{\text{length}(\text{ATS system' summary})},$ (21)

$\text{recall}_{L}=\frac{LCS(\text{humans' summary},\text{ATS system'
summary})}{\text{length}(\text{humans' summary})},$ (22)

and the $F$-measure is evaluated as in Eq. (16).

Note that ROUGE-L, as the LCS, does not differentiate whether the match
between the sequence and its subsequence is consecutive or not. For instance,
let $X=\text{[`a',`b',`c',`d',`e']}$, $Y_{1}=\text{[`a',`b',`c',`f',`g']}$ and
$Y_{2}=\text{[`a',`f',`b',`g',`c']}$. Both $Y_{1}$ and $Y_{2}$ have the same
ROUGE-L score. Thus, ROUGE-W measures the WLCS (Weighted Longest Common
Subsequence) that weights the LCS by the length of consecutive matches.

A skip-bigram is any pair of words in their sentence order, allowing for
arbitrary gaps. ROUGE-$S$ measures skip-bigram co-occurrence statistics
between the human expert’s summary and the ATS system’s summary. A potential
problem for ROUGE-$S$ is that it scores null if the sentence does not have any
word pair co-occurring with its reference. For instance, ROUGE-S gives a score
of 0 for the sequences $X=\text{[`a',`b',`c',`d']}$ and
$Y=\text{[`d',`c',`b',`a']}$. ROUGE-SU-$k$ counts both skip bigrams and
unigrams with maximal skip distance $k$.

One critique that we can make of the approaches that count the number of
common extracts between the computer-based summary and the human-based summary
is that the use of words with the same semantic is not considered. Thus, the
next line of Table 16 presents a collection of works that extend the ROUGE
methodology to consider semantically related words. Some of them use
dictionaries or corpora. METEOR (Banerjee and Lavie, 2005), Ganesan (2018) and
ShafieiBavani et al. (2018) use WordNet. Zhou et al. (2006) build an
independent paraphrase collection. On the other hand, other works apply neural
network representations, such as Ng and Abrecht (2015) (word embeddings),
Zhang et al. (2019) (BERT), Zhao et al. (2019) (BERT), Hailu et al. (2020)
(word embeddings) and Lee et al. (2020) (BERT), presented in Sections 6.1.5
and 7.2.

The next row considers methods based on relevance analysis that measure the
quality of the candidate summary by using a relative decrease in retrieval
performance when indexing summaries instead of full documents. They use a
search engine to find the most related documents (from a given set) to the
candidate summary and to its corresponding human-made summaries forming two
lists of retrieved documents. Radev et al. (2003) and Cohan and Goharian
(2016) evaluate respectively the candidate summary score using the Kendall or
the Spearman correlation measures and the intersection between the truncated
lists of retrieved documents.

The latent-based approach (Steinberger et al., 2009) uses singular value
decomposition discussed in Section 6.1.2 to capture the main topics of the
document. The computer-based summaries are ranked according to the similarity
of the main topics of their summaries and their reference documents.

The semi-automated pyramid methods (Harnly et al., 2005, Passonneau et al.,
2013, 2018) use respectively unigram overlap, cosine similarity of latent
vector representations and a weighted set cover algorithm to score the
summaries given the manual pyramids. On the other hand, the automated pyramid
methods (Yang et al., 2016, Peyrard and Eckle-Kohler, 2017, Gao et al., 2018,
2019) face the herculean task of building the pyramids automatically. In order
to do that, Yang et al. (2016) extract relation tuples using the Stanford Open
Information Extraction (Angeli et al., 2015) and Peyrard and Eckle-Kohler
(2017) use a phrase structure parse and dependency parse to convert each
sentence using the Stanford CoreNLP (Manning et al., 2014). In order to find
the semantic similarity, they respectively use WordNet and word embeddings.
SSAS (Semantic Similarity for Abstractive Summarization) (Vadapalli et al.,
2017) applies the model of Yang et al. (2016) to extract the SCUs and combine
various semantic and lexical similarity measures to score the quality of the
candidate summary.

The basic elements approach (Hovy et al., 2005, 2006), as in the case of BLEU,
ROUGE or pyramid methods, compares the content of the reference summaries and
generated summaries using small units as defined by the authors as (1) the
head of a major syntactic constituent (noun, verb, adjective or adverbial
phrases), expressed as a single item; or (2) a relation between a head and a
single dependent, expressed as a triple (head — modifier — relation). In order
to evaluate the score, the basic elements are evaluated according to lexical
identity (the words must match exactly), lemma identity (the root forms of the
words must match according to WordNet) and distributional similarity (words
are similar according to the cosine distance on mutual information-based
distributional similarity scores (Lin and Pantel, 2002)).

The SEE (Summary Evaluation Environment) (Lin, 2001, Lin and Hovy, 2003) is a
hybrid approach in Table 16, in the sense that comprises ingredients of both
the quality and content approaches. In this approach, the human evaluators
compare the human-made summary with the automatic summary by finding the
common pieces of text and specifying if the found pieces of text express all,
most, some, hardly any or none of the content of the current model unit. In
order to measure quality, the human evaluators rate grammar, cohesion, and
coherence at those five levels.

The TAC 2008 overall responsiveness (Dang et al., 2008) is also a manual
hybrid approach in Table 16. It evaluates the degree to which a summary is
responding to the information necessary to describe the topic state and the
linguistic quality. It uses the five-point scale: (1) very poor; (2) poor; (3)
barely acceptable; (4) good; (5) very good.

The extrinsic evaluation methods explore the quality of the summary in the
completion of a specific task. The difficulty with matching a computer-made
summary against an ideal summary is that the ideal summary is hard to
establish. In Table 16, some tasks are considered, namely document
categorization, information retrieval, question answering and masked token
task (Taylor, 1953).

Approach | Kind | Sub-kind | Method |  Source
---|---|---|---|---
Intrinsic | Quality |  | DUC 2005 readability |  Dang (2005)
4-5 |  | TAC 2008 readability |  Dang et al. (2008)
4-5 |  | Newsroom human evaluation |  Grusky et al. (2018)
4-5 |  | Sum-QE |  Xenouleas et al. (2019)
| Content | Free reference based | Distributions divergence |  Torres-Moreno et al. (2010) and Cabrera-Diego and Torres-Moreno (2018)
4-5 | Cosine similarity |  Louis and Nenkova (2013), Sun and Nenkova (2019)
4-5 | topic |  Louis and Nenkova (2013)
4-5 | likelihood |  Louis and Nenkova (2013)
4-5 | Pseudo reference |  Gao et al. (2020)
4-5 |  | Transfer learning |  Bao et al. (2020a)
| Co-selection | Information retrieval based |  Baeza-Yates and Ribeiro-Neto (2008)
4-5 | Relative utility |  Radev et al. (2004)
| Content-based | Manual |  Van Halteren and Teufel (2003), Nenkova and Passonneau (2004)
4-5 | Cosine similarity |  Salton (1989)
4-5 | Unity overlap |  Saggion et al. (2002b)
4-5 |  BLEU ($n$-grams matching) |  Papineni et al. (2002)
4-5 | Longest Common Subsequence |  Saggion et al. (2002a)
4-5 | Extracts matching (ROUGE) |  Lin and Hovy (2003), Lin (2004)
4-5 | Extracts matching with semantic |  Banerjee and Lavie (2005), Zhou et al. (2006), Ng and Abrecht (2015), Ganesan (2018), ShafieiBavani et al. (2018), Zhao et al. (2019) and Hailu et al. (2020).
4-5 | Search-based matching |  Radev et al. (2003) and Cohan and Goharian (2016)
4-5 | Latent-based |  Steinberger et al. (2009)
4-5 | Semi-automated pyramid |  Harnly et al. (2005), Passonneau et al. (2013, 2018)
4-5 |  | Automated pyramid |  Yang et al. (2016), Peyrard and Eckle-Kohler (2017), Vadapalli et al. (2017)
4-5 |  |  | Basic elements |  Hovy et al. (2005) and Hovy et al. (2006)
| Hybrid |  | SEE |  Lin (2001), Lin and Hovy (2003)
4-5 |  | Overall responsiveness |  (Dang et al., 2008)
Extrinsic | Task-based |  | Document categorization |  Brandow et al. (1995), Mani and Bloedorn (1997)
4-5 | Information retrieval |  Tombros and Sanderson (1998), Radev et al. (2003)
4-5 | Question answering |  Morris et al. (1992), Chen et al. (2018), Scialom et al. (2019) and Eyal et al. (2019)
4-5 |  |  | masked token task |  (Vasilyev et al., 2020)
Table 16: A digest of the methods used to evaluate ATS.

It is worth mentioning that, as we have shown in Table 16, due to the inherent
complexity of evaluating ATS systems, there are different methods for this
task. However, as reported in Blagec et al. (2022), BLEU and ROUGE dominate
the field of evaluation in spite of their low correlation with human
evaluation (Liu and Liu, 2008, Ng and Abrecht, 2015). We may find the same
conclusion if we consider carefully the last column of Tables 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14 and 15.

##  10 Open libraries

There are now several libraries with implementations of the most popular
methods of ATS. Table 17 presents a compendium of the extractive approaches
introduced in Section 6. Table 18 presents a compendium of summarization
methods based on transformer models discussed in Section 7.2. Finally, Table
19 presents a digest of open libraries with evaluation methods.

Name |  Source code | Method | Source | Section
---|---|---|---|---
Sumy |  https://github.com/miso-belica/sumy/ | Random |  |
3-5 | KL-Sum | Haghighi and Vanderwende (2009) | 6.1.1
3-5 | Luhn | Luhn (1958) | 6.1.1
3-5 | Edmundson | Edmundson (1969) | 6.1.1
3-5 | LSA | Steinberger et al. (2004) | 6.1.2
3-5 | Textrank | Mihalcea and Tarau (2004) | 6.1.3
3-5 | Lexrank | Erkan and Radev (2004) | 6.1.3
3-5 | Reduction |  | 6.1.3
Table 17: A compendium of the extractive approaches presented in Section 6. The methods come from the library Sumy (Belica, 2022). Name |  Source Code | Kind | Method | Source | Section
---|---|---|---|---|---
transformers (SummarizationPipeline) |  huggingface.co/ transformers/v3.0.2/_modules/ transformers/pipelines.html# SummarizationPipeline | Abstractive | bart-large-cnn | Lewis et al. (2019) | 7.2
4-6 | pegasus-xsum | Zhang et al. (2020) | 7.2
| pegasus-large
4-6 | t5-small | Raffel et al. (2019) | 7.2
| t5-base
| t5-large
| t5-3b
| t5-11b
transformerSum |  https://github.com/HHousen/ TransformerSum | Extractive | distilroberta-base-ext-sum | Sanh et al. (2019) | 6.4
distilbert-base-uncased-ext-sum
4-6 | roberta-large-ext-sum | Liu et al. (2019) | 6.4
| roberta-base-ext-sum
4-6 | bert-base-uncased-ext-sum | Devlin et al. (2018) | 6.4
| bert-large-uncased-ext-sum
4-6 | longformer-base-4096-ext-sum | Beltagy et al. (2020) | 6.4
| mobilebert-uncased-ext-sum
| Abstractive | led-base-16384 | Beltagy et al. (2020) | 7.2
| led-large-16384
| distil-led-large-cnn-16384
Table 18: A compendium of the transformer approaches presented in Sections 6.4 and 7.2. The methods come from the Transformers (Wolf et al., 2020) and TransformerSum libraries.(Housen, 2022). Library | Source Code | Method | Source
---|---|---|---
NLTK | https://www.nltk.org/ | Cosine simlarity | Salton (1989)
3-4 | Precision | Baeza-Yates and Ribeiro-Neto (2008)
3-4 | Recall | Baeza-Yates and Ribeiro-Neto (2008)
3-4 | F-Measure | Baeza-Yates and Ribeiro-Neto (2008)
3-4 | Likelihood | Louis and Nenkova (2013)
3-4 | Unit overlap (Jaccard) | Saggion et al. (2002b)
Gensim | https://radimrehurek.com/gensim/ | Kullback-Leibler | Louis and Nenkova (2013)
3-4 | Unit overlap (Jaccard) | Saggion et al. (2002b)
py-ROUGE | https://github.com/Diego999/py-rouge | ROUGE-N | Lin (2004)
3-4 | ROUGE-L | Lin (2004)
3-4 | ROUGE-W | Lin (2004)
ROUGE-metric | https://github.com/li-plus/rouge-metric | ROUGE-SU | Lin (2004)
Table 19: A compendium of the evaluation methods used in summarization and
discussed in Section 9. These methods come from the NLTK (Bird et al., 2009),
Gensim (Řehůřek and Sojka, 2010), Py-ROUGE, and ROUGE-metric libraries.

##  11 Empirical exercises

In this section, we present an empirical exploration of some of the free ATS
libraries presented in Section 10. For this exercise, we use CNN Corpus (Lins
et al., 2019) presented in Section 4. We have run all extractive approaches
presented in Section 10. In order to run the Edmundson summarizer we must
specify a list of bonus words, that is, words that are positively relevant, a
list of stigma words, negatively relevant, and a list of null words, deemed
irrelevant to the summary. While the null words were defined as the list of
stopwords of the NLTK library for the english language, the lists of bonus and
stigma words were obtained by calculating the document frequency of terms in a
subset of 200 golden summaries and source texts from each corpus, after
stopwords removal. The 10 most frequent words in the subset of golden
summaries were used as bonus words for each dataset, and the 10 words with the
highest ratio of term frequency in source texts to the frequency in golden
summaries were defined as stigma words.

We chose to run BART, mT5 and PEGASUS as the transformer models for
abstractive summarization in our experiments. Given our intention of selecting
ready-to-use summarization methods, these are some of the most popular
transformers on the summarization pipeline of the Huggingface package.
Therefore, these models are readily available in a pre-trained state by this
implementation. As for which fine-tuning for each model was chosen, we decided
upon pegasus-xsum, bart-large-cnn and mT5-multilingual-XLSum, trained with the
XSum, CNN/Daily Mail and XL-Sum datasets, respectively.

We also present the typical summaries generated by these methods for a
document from the CNN Corpus in Tables 21 and 22. This dataset has both
extractive and abstractive golden summaries for each text. The ones used in
our examples, as well as the source text, are presented below:  Source text: A
fourth infant has been discovered to have been infected with a rare, sometimes
fatal form of bacteria that can come from baby formula, but there is no
evidence the cases are related, federal health authorities said Friday. “Based
on test results to date, there is no need for a recall of infant formula and
parents may continue to use powdered infant formula, following the
manufacturer’s directions on the printed label”, the Centers for Disease
Control and Prevention and the Food and Drug Administration said in a joint
update. The latest case of Cronobacter infection occurred in Florida, the
update said. After cases occurred in Missouri and Illinois, authorities looked
for other cases, and found an Oklahoma case and the one in Florida. The
Florida and Missouri infants died this month of their infections. The Missouri
case prompted retail giant Wal-Mart to pull all cans of the same size and lot
number of baby formula from its shelves. But DNA fingerprinting of the
bacteria from the Missouri and Illinois cases found the bacteria differed,
suggesting they were not related, the agencies said. Bacteria from the other
two cases were not available for testing, they said. In the Missouri case,
Cronobacter bacteria were found in an opened bottle of nursery water and
prepared infant formula, but it was not clear how they became contaminated,
the update said. Tests on factory-sealed containers of powdered infant formula
and nursery water with the same lot numbers turned up no Cronobacter bacteria,
it said, adding, “There is currently no evidence to conclude that the infant
formula or nursery water was contaminated during manufacturing or shipping.”
Formula maker Mead Johnson Nutrition said the agencies’ test results
corroborated its own. “We’re pleased with the FDA and CDC testing, which
should reassure consumers, health care professionals and retailers everywhere
about the safety and quality of our products”, Tim Brown, senior vice
president and general manager for North America, said in a statement. “These
tests also reinforce the rigor of our quality processes throughout our
operations.” Cronobacter infection typically occurs during the first days or
weeks of life. In a typical year, the CDC said, it learns of four to six such
cases. This year, with increased awareness of the infection, it has tallied 12
cases. The bacteria can cause severe infection or inflammation of the
membranes that cover the brain. Symptoms can start with fever, poor feeding,
crying or listlessness. “Any young infant with these symptoms should be in the
care of a physician”, the update says. The bacteria can be found in the
environment and can multiply in powdered infant formula once it is mixed with
water, said the CDC, which recommends breastfeeding whenever possible.
Cronobacter is fatal in nearly 40% of cases, according to the CDC. Infection
survivors can be left with severe neurological problems.

Extractive golden summary: “Based on test results to date, there is no need
for a recall of infant formula and parents may continue to use powdered infant
formula, following the manufacturer’s directions on the printed label”, the
Centers for Disease Control and Prevention and the Food and Drug
Administration said in a joint update. But DNA fingerprinting of the bacteria
from the Missouri and Illinois cases found the bacteria differed, suggesting
they were not related, the agencies said. The bacteria can be found in the
environment and can multiply in powdered infant formula once it is mixed with
water, said the CDC, which recommends breastfeeding whenever possible.

Abstractive golden summary: “There is no need for a recall of infant formula”,
federal health officials say. DNA fingerprinting finds the Missouri and
Illinois bacteria are different, suggesting they’re not related. CDC
recommends breastfeeding whenever possible.

Based on the results of Table 20, there is no consensus on the best method.
However, we can see that some of the methods stand out. Among the classical
extractive methods, we note the one due to Luhn (1958) and Text Rank (Mihalcea
and Tarau, 2004). Among the transformers, we may cite the BART (Lewis et al.,
2019). This result makes sense since the training data of BART comprises a set
of news. Thus, in some sense, this exercise is an exercise of learning
transference (Zhuang et al., 2020).

CNN Corpus | ROUGE-1 | ROUGE-2 | ROUGE-3 | ROUGE-4
---|---|---|---|---
P | R | F | P | R | F | P | R | F | P | R | F
SumyKL | 34.20 | 32.59 | 33.37 | 16.95 | 16.28 | 16.61 | 14.12 | 14.09 | 13.89 | 13.08 | 13.09 | 12.88
SumyLexRank | 38.46 | 43.83 | 40.97 | 23.00 | 26.21 | 24.50 | 21.35 | 23.18 | 21.97 | 20.29 | 22.04 | 20.89
SumyLsa | 32.76 | 33.74 | 33.24 | 16.21 | 16.90 | 16.55 | 14.05 | 14.97 | 14.32 | 13.21 | 14.09 | 13.47
SumyLuhn | 35.86 | 52.83 | 42.72 | 24.14 | 34.37 | 28.36 | 25.50 | 28.95 | 26.89 | 24.61 | 27.94 | 25.95
SumyEdmundson | 35.99 | 41.39 | 38.50 | 21.36 | 24.64 | 22.89 | 20.33 | 22.44 | 21.07 | 19.43 | 21.48 | 20.15
SumyRandom | 33.45 | 30.77 | 32.05 | 15.33 | 14.52 | 14.91 | 12.82 | 12.54 | 12.42 | 11.94 | 11.72 | 11.59
SumyReduction | 33.47 | 49.65 | 39.98 | 21.61 | 30.81 | 25.40 | 22.36 | 25.20 | 23.50 | 21.51 | 24.23 | 22.60
SumySumBasic | 37.72 | 25.50 | 30.43 | 14.72 | 10.53 | 12.28 | 11.51 | 8.78 | 9.62 | 10.30 | 7.93 | 8.65
SumyTextRank | 33.31 | 50.26 | 40.07 | 21.63 | 31.34 | 25.60 | 22.66 | 25.54 | 23.82 | 21.83 | 24.58 | 22.93
bart-large-cnn | 33.29 | 33.54 | 33.41 | 15.87 | 15.88 | 15.87 | 12.82 | 12.97 | 12.74 | 9.38 | 9.42 | 9.28
mT5-multilingual-XLSum | 24.43 | 18.25 | 20.90 | 4.98 | 3.65 | 4.21 | 2.90 | 2.13 | 2.42 | 1.38 | 1.00 | 1.14
pegasus-xsum | 22.24 | 23.24 | 22.73 | 6.26 | 6.62 | 6.43 | 4.38 | 4.68 | 4.46 | 2.60 | 2.76 | 2.63

CNN Corpus | ROUGE-L |  ROUGE-SU4 | ROUGE-W
---|---|---|---
P | R | F | P | R | F | P | R | F
SumyKL | 25.15 | 24.19 | 24.66 | 19.14 | 18.33 | 18.73 | 21.73 | 8.99 | 12.47
SumyLexRank | 29.65 | 33.85 | 31.61 | 24.70 | 28.24 | 26.36 | 27.54 | 12.41 | 16.83
SumyLsa | 23.76 | 24.62 | 24.18 | 18.20 | 18.95 | 18.57 | 20.94 | 9.23 | 12.59
SumyLuhn | 29.28 | 42.62 | 34.71 | 25.37 | 36.66 | 29.99 | 30.48 | 14.53 | 19.40
SumyEdmundson | 27.99 | 32.30 | 29.99 | 23.04 | 26.66 | 24.72 | 26.30 | 12.05 | 16.26
SumyRandom | 23.89 | 22.11 | 22.96 | 17.75 | 16.64 | 17.18 | 20.88 | 8.25 | 11.55
SumyReduction | 26.93 | 39.42 | 32.00 | 22.97 | 33.31 | 27.19 | 27.85 | 13.21 | 17.66
SumySumBasic | 25.77 | 17.52 | 20.86 | 17.80 | 12.37 | 14.60 | 22.55 | 6.69 | 10.00
SumyTextRank | 26.86 | 39.99 | 32.13 | 22.97 | 33.84 | 27.36 | 28.11 | 13.34 | 17.83
bart-large-cnn | 25.29 | 25.53 | 25.41 | 15.89 | 15.93 | 15.91 | 24.33 | 11.77 | 15.61
mT5-multilingual-XLSum | 16.32 | 12.25 | 14.00 | 7.29 | 5.31 | 6.14 | 16.50 | 5.93 | 8.58
pegasus-xsum | 15.81 | 16.65 | 16.22 | 7.65 | 8.03 | 7.84 | 15.70 | 7.94 | 10.35

CNN Corpus | P | R | F | H | J | K-L | C
---|---|---|---|---|---|---|---
SumyKL | 31.92 | 28.99 | 30.38 | 0.24 | 0.84 | 0.21 | 0.43
SumyLexRank | 37.89 | 41.34 | 39.54 | 0.23 | 0.81 | 0.19 | 0.49
SumyLsa | 30.97 | 33.67 | 32.26 | 0.23 | 0.85 | 0.20 | 0.37
SumyLuhn | 37.03 | 49.10 | 42.22 | 0.24 | 0.80 | 0.18 | 0.54
SumyEdmundson | 35.04 | 38.99 | 36.91 | 0.23 | 0.82 | 0.19 | 0.46
SumyRandom | 30.89 | 29.19 | 30.02 | 0.23 | 0.85 | 0.21 | 0.39
SumyReduction | 34.14 | 45.55 | 39.03 | 0.24 | 0.81 | 0.18 | 0.54
SumySumBasic | 34.90 | 25.03 | 29.15 | 0.22 | 0.86 | 0.20 | 0.39
SumyTextRank | 33.95 | 46.11 | 39.11 | 0.24 | 0.81 | 0.18 | 0.54
bart-large-cnn | 33.80 | 33.91 | 33.85 | 0.22 | 0.83 | 0.19 | 0.35
mT5-multilingual-XLSum | 24.14 | 18.73 | 21.09 | 0.23 | 0.90 | 0.22 | 0.22
pegasus-xsum | 25.75 | 22.45 | 23.99 | 0.25 | 0.89 | 0.26 | 0.25

Table 20: CNN Corpus results. The evaluators used in this table are ROUGE-$n$
(1 to 4), ROUGE-L, ROUGE-SU4, ROUGE-W, Precision (P), Recall (R), the
F-measure, Hellinger (H), Jaccard distance (J), KL divergence (KL) and Cosine
similarity (C).

Method |  Summary
---|---
SumyRandom |  A fourth infant has been discovered to have been infected with a rare, sometimes fatal form of bacteria that can come from baby formula, but there is no evidence the cases are related, federal health authorities said Friday. In the Missouri case, Cronobacter bacteria were found in an opened bottle of nursery water and prepared infant formula, but it was not clear how they became contaminated, the update said. “We’re pleased with the FDA and CDC testing, which should reassure consumers, health care professionals and retailers everywhere about the safety and quality of our products”, Tim Brown, senior vice president and general manager for North America, said in a statement. Symptoms can start with fever, poor feeding, crying or listlessness.
SumyLuhn |  A fourth infant has been discovered to have been infected with a rare, sometimes fatal form of bacteria that can come from baby formula, but there is no evidence the cases are related, federal health authorities said Friday. “Based on test results to date, there is no need for a recall of infant formula and parents may continue to use powdered infant formula, following the manufacturer’s directions on the printed label”, the Centers for Disease Control and Prevention and the Food and Drug Administration said in a joint update. In the Missouri case, Cronobacter bacteria were found in an opened bottle of nursery water and prepared infant formula, but it was not clear how they became contaminated, the update said. Tests on factory-sealed containers of powdered infant formula and nursery water with the same lot numbers turned up no Cronobacter bacteria, it said, adding, “There is currently no evidence to conclude that the infant formula or nursery water was contaminated during manufacturing or shipping.”
SumyEdmundson |  A fourth infant has been discovered to have been infected with a rare, sometimes fatal form of bacteria that can come from baby formula, but there is no evidence the cases are related, federal health authorities said Friday. ”Based on test results to date, there is no need for a recall of infant formula and parents may continue to use powdered infant formula, following the manufacturer’s directions on the printed label”, the Centers for Disease Control and Prevention and the Food and Drug Administration said in a joint update. The latest case of Cronobacter infection occurred in Florida, the update said. In a typical year, the CDC said, it learns of four to six such cases.
SumyLsa |  A fourth infant has been discovered to have been infected with a rare, sometimes fatal form of bacteria that can come from baby formula, but there is no evidence the cases are related, federal health authorities said Friday. Bacteria from the other two cases were not available for testing, they said. Tests on factory-sealed containers of powdered infant formula and nursery water with the same lot numbers turned up no Cronobacter bacteria, it said, adding, “There is currently no evidence to conclude that the infant formula or nursery water was contaminated during manufacturing or shipping.” In a typical year, the CDC said, it learns of four to six such cases.
SumyLexRank |  “Based on test results to date, there is no need for a recall of infant formula and parents may continue to use powdered infant formula, following the manufacturer’s directions on the printed label”, the Centers for Disease Control and Prevention and the Food and Drug Administration said in a joint update. After cases occurred in Missouri and Illinois, authorities looked for other cases, and found an Oklahoma case and the one in Florida. Tests on factory-sealed containers of powdered infant formula and nursery water with the same lot numbers turned up no Cronobacter bacteria, it said, adding,” There is currently no evidence to conclude that the infant formula or nursery water was contaminated during manufacturing or shipping.” The bacteria can be found in the environment and can multiply in powdered infant formula once it is mixed with water, said the CDC, which recommends breastfeeding whenever possible.
SumyTextRank |  “Based on test results to date, there is no need for a recall of infant formula and parents may continue to use powdered infant formula, following the manufacturer’s directions on the printed label”, the Centers for Disease Control and Prevention and the Food and Drug Administration said in a joint update. But DNA fingerprinting of the bacteria from the Missouri and Illinois cases found the bacteria differed, suggesting they were not related, the agencies said. Tests on factory-sealed containers of powdered infant formula and nursery water with the same lot numbers turned up no Cronobacter bacteria, it said, adding, “There is currently no evidence to conclude that the infant formula or nursery water was contaminated during manufacturing or shipping.” The bacteria can be found in the environment and can multiply in powdered infant formula once it is mixed with water, said the CDC, which recommends breastfeeding whenever possible.
SumySumBasic |  The latest case of Cronobacter infection occurred in Florida, the update said. After cases occurred in Missouri and Illinois, authorities looked for other cases, and found an Oklahoma case and the one in Florida. Tests on factory-sealed containers of powdered infant formula and nursery water with the same lot numbers turned up no Cronobacter bacteria, it said, adding, “There is currently no evidence to conclude that the infant formula or nursery water was contaminated during manufacturing or shipping.” Infection survivors can be left with severe neurological problems.
SumyKL |  The latest case of Cronobacter infection occurred in Florida, the update said. After cases occurred in Missouri and Illinois, authorities looked for other cases, and found an Oklahoma case and the one in Florida. But DNA fingerprinting of the bacteria from the Missouri and Illinois cases found the bacteria differed, suggesting they were not related, the agencies said. The bacteria can cause severe infection or inflammation of the membranes that cover the brain.
SumyReduction |  “Based on test results to date, there is no need for a recall of infant formula and parents may continue to use powdered infant formula, following the manufacturer’s directions on the printed label”, the Centers for Disease Control and Prevention and the Food and Drug Administration said in a joint update. But DNA fingerprinting of the bacteria from the Missouri and Illinois cases found the bacteria differed, suggesting they were not related, the agencies said. Tests on factory-sealed containers of powdered infant formula and nursery water with the same lot numbers turned up no Cronobacter bacteria, it said, adding, “There is currently no evidence to conclude that the infant formula or nursery water was contaminated during manufacturing or shipping.” The bacteria can be found in the environment and can multiply in powdered infant formula once it is mixed with water, said the CDC, which recommends breastfeeding whenever possible.
Table 21: Examples of extractive summaries. Method |  Summary
---|---
Transformers- facebook/bart-large-cnn |  A fourth infant has been infected with a rare, sometimes fatal form of bacteria. There is no evidence the cases are related, federal health authorities say. The latest case of Cronobacter infection occurred in Florida. In a typical year, the CDC says, it learns of four to six such cases.
Transformers- google/pegasus-xsum |  There is no need for a recall of infant formula, US health officials say, after three babies died from an infection linked to the product in Missouri, Florida and Illinois. Tests on factory-sealed containers of powdered infant formula and nursery water with the same lot numbers turned up no Cronobacter.
Transformers- csebuetnlp/ mT5_multilingual _XLSum |  The US government has said there is no need for a recall of powdered infant formula and nursery water, following tests that show no evidence they were contaminated during manufacturing or shipment of the product.
Table 22: Examples of abstractive summaries.

##  12 Final remarks

In this work, we have provided a literature review of ATS systems. As we have
previously mentioned, this is not an easy task. Since work by Luhn (1958),
thousands of papers were introduced about this subject. In order to present a
comprehensive review, we have divided the contributions by the type of output
summary, as described in Section 2, namely extractive, abstractive and hybrid,
and the type of model used to generate the summary. Several models have been
used to generate summaries including the classical frequency-based models and
the state of art deep neural network sequence-to-sequence models. In general,
we may see in this work that any model of language can be used to generate a
summary.

Besides the presentation of the models, we also introduced the most popular
datasets used in the field, the methods used to evaluate the quality of the
summaries, the public libraries that can be used to generate summaries and
some empirical exercises exploring how the models discussed in the paper can
be applied using the public libraries reviewed in Section 10 to generate
useful summaries.

Although we may surely assert that the field of ATS systems is a mature field,
many difficulties still remain and these field difficulties will undoubtedly
be explored in the future:

  1. 1.

Text quality: The quality of a summary may strongly depend on the approach
used to implement the ATS system. In extractive approaches, they still fail to
provide solutions that avoid a lack of coherence between the sentences. In
particular, one of the biggest difficulties is the “dangling” anaphoras, i.e.,
sentences in the extracted summaries referring to other sentences that are not
in the summary or to the wrong previous sentence. Although there are some
solutions that try to minimize this problem such as extracting entire
paragraphs (Wu and Liu, 2003) or using heuristics to avoid extracting
sentences that make references to other sentences (Rush et al., 1971), this is
still an ongoing problem. In abstractive summarization systems, the quality of
the summary is strongly related to the capacity of text generation.

  2. 2.

Completeness: Most choices about what should be included in the summary are
based on assumptions made about the text. A large set of models have been made
available. Due to the complexity of human language, all models have their own
limitations. Frequency-based models strongly depend on the frequency of the
tokens. With these models, important pieces of information presented using
“rare” words may not be considered. One solution to deal with this is to use
semantically based approaches such as the ones presented in Section 6.1.5
based on word embeddings. Linguistic models usually depend on specific
structures, rules or ontologies that may not be correctly applied in many
situations. Sequence-to-sequence deep learning models are also kinds of
frequency-based models with the same merits and problems. Furthermore, they
need to be trained with large annotated datasets. If the dataset is not
available, we have to transfer the learning from one dataset to the other.
This transference may work or may not work depending on the differences and
similarities between the two pieces of text.

  3. 3.

Size of the input text: In extractive summarization, although the size of
input text usually is not directly a problem, the quality of the summary may
deteriorate when we concatenate sentences that come from very different parts
of the text. The size of the input text may not be a problem for the
linguistic-based abstractive approaches either. However, if the size of the
input text is positively correlated with the complexity of the text, this
might be a problem, since the structures used for linguistic-based abstractive
summarization may not be able to accommodate this richness of details. Due to
the computational complexity, deep learning sequence-to-sequence models suffer
directly from the size of input since most models are available to texts of
moderate size.

  4. 4.

Size of the abstract: Considering abstracts that convey the same pieces of
information, abstracts generated by extractive summarization approaches are
usually larger than the ones generated by abstractive methods. The point is
that in the case of the extractive approaches, the relevant pieces of
information are distributed in several sentences, and in many situations,
irrelevant pieces of information remain in the extracted sentences.

  5. 5.

Datasets: Although there are many datasets available today as we presented in
Section 4, many of these datasets are related to a specific field (news or
scientific papers) since in general, it is very costly to create datasets.
This problem increases when we need datasets to train extractive systems,
since human summaries are abstractive in nature, as we mentioned in Section
6.4.

  6. 6.

Evaluation: The biggest issue with the activity of evaluating the task of ATS
is that there is not a perfect summary that can be used as a model such as in
other machine learning tasks. Different human experts may generate great but
different summaries. Besides the fact that different human experts may choose
different pieces of text to include in the summary, they may include the same
pieces of text in semantically equivalent ways, making the task of comparing
machine summaries to human summaries very difficult.

##  13 Acknowledgment

The authors acknowledge and thank the partial financial support from the
Ministry of Science and Technology of Brazil (MCTI). DOC (grant number
302629/2019-0) and LW (309545/2021-8) also thank CNPQ for the partial
financial support.

## References

  * Abadi et al. (2015) M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Aliannejadi et al. (2015) M. Aliannejadi, S. A. Bahrainian, A. Giachanou, and F. Crestani.  University of lugano at trec 2015: Contextual suggestion and temporal summarization tracks.  In _TREC_ , 2015.
  * Allahyari et al. (2017) M. Allahyari, S. Pouriyeh, M. Assefi, S. Safaei, E. D. Trippe, J. B. Gutierrez, and K. Kochut.  Text summarization techniques: a brief survey.  _arXiv preprint arXiv:1707.02268_ , 2017.
  * Alomari et al. (2022) A. Alomari, N. Idris, A. Q. M. Sabri, and I. Alsmadi.  Deep reinforcement and transfer learning for abstractive text summarization: A review.  _Computer Speech & Language_, 71:101276, 2022.
  * Andhale and Bewoor (2016) N. Andhale and L. Bewoor.  An overview of text summarization techniques.  In _2016 international conference on computing communication control and automation (ICCUBEA)_ , pages 1–7. IEEE, 2016.
  * Angeli et al. (2015) G. Angeli, M. J. J. Premkumar, and C. D. Manning.  Leveraging linguistic structure for open domain information extraction.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 344–354, 2015.
  * Aslam et al. (2013) J. A. Aslam, M. Ekstrand-Abueg, V. Pavlu, F. Diaz, and T. Sakai.  Trec 2013 temporal summarization.  In _TREC_ , 2013.
  * Awasthi et al. (2021) I. Awasthi, K. Gupta, P. S. Bhogal, S. S. Anand, and P. K. Soni.  Natural language processing (nlp) based text summarization-a survey.  In _2021 6th International Conference on Inventive Computation Technologies (ICICT)_ , pages 1310–1317. IEEE, 2021.
  * Baeza-Yates and Ribeiro-Neto (2008) R. Baeza-Yates and B. Ribeiro-Neto.  _Modern Information Retrieval_.  Addison-Wesley Publishing Company, USA, 2nd edition, 2008.  ISBN 9780321416919.
  * Bahdanau et al. (2014) D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ , 2014.
  * Banerjee and Lavie (2005) S. Banerjee and A. Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72, 2005.
  * Banko et al. (2000) M. Banko, V. O. Mittal, and M. J. Witbrock.  Headline generation based on statistical translation.  In _Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics_ , pages 318–325, 2000.
  * Bao et al. (2020a) F. S. Bao, H. Li, G. Luo, C. Chen, Y. Yang, Y. He, and M. Qiu.  End-to-end semantics-based summary quality assessment for single-document summarization.  _arXiv preprint arXiv:2005.06377_ , 2020a.
  * Bao et al. (2020b) H. Bao, L. Dong, F. Wei, W. Wang, N. Yang, X. Liu, Y. Wang, S. Piao, J. Gao, M. Zhou, and H.-W. Hon.  Unilmv2: Pseudo-masked language models for unified language model pre-training.  In _Preprint_ , 2020b.
  * Barman et al. (2021) U. Barman, V. Barman, M. Rahman, and N. K. Choudhury.  Graph based extractive news articles summarization approach leveraging static word embeddings.  In _2021 International Conference on Computational Performance Evaluation (ComPE)_ , pages 008–011. IEEE, 2021.
  * Barrios et al. (2016) F. Barrios, F. López, L. Argerich, and R. Wachenchauzer.  Variations of the similarity function of textrank for automated summarization.  _arXiv preprint arXiv:1602.03606_ , 2016.
  * Barto and Duff (1993) A. Barto and M. Duff.  Monte carlo matrix inversion and reinforcement learning.  _Advances in Neural Information Processing Systems_ , 6, 1993.
  * Baxendale (1958) P. B. Baxendale.  Machine-made index for technical literature—an experiment.  _IBM Journal of research and development_ , 2(4):354–361, 1958.
  * Belica (2022) M. Belica.  Sumy.  https://github.com/miso-belica/sumy, 2022.
  * Beltagy et al. (2020) I. Beltagy, M. E. Peters, and A. Cohan.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ , 2020.
  * Belwal et al. (2021) R. C. Belwal, S. Rai, and A. Gupta.  Text summarization using topic-based vector space model and semantic measure.  _Information Processing & Management_, 58(3):102536, 2021.
  * Bengio et al. (1993) Y. Bengio, P. Frasconi, and P. Simard.  The problem of learning long-term dependencies in recurrent networks.  In _IEEE international conference on neural networks_ , pages 1183–1188. IEEE, 1993.
  * Bengio et al. (2000) Y. Bengio, R. Ducharme, and P. Vincent.  A neural probabilistic language model.  _Advances in neural information processing systems_ , 13, 2000.
  * Berkson (1944) J. Berkson.  Application of the logistic function to bio-assay.  _Journal of the American statistical association_ , 39(227):357–365, 1944.
  * Berkson (1951) J. Berkson.  Why i prefer logits to probits.  _Biometrics_ , 7(4):327–339, 1951.
  * Berry et al. (1995) M. W. Berry, S. T. Dumais, and G. W. O’Brien.  Using linear algebra for intelligent information retrieval.  _SIAM review_ , 37(4):573–595, 1995.
  * Bertsekas (2012) D. Bertsekas.  _Dynamic programming and optimal control: Volume I_ , volume 1.  Athena scientific, 2012.
  * Bertsekas and Tsitsiklis (1996) D. P. Bertsekas and J. N. Tsitsiklis.  _Neuro-dynamic programming_.  Athena Scientific, 1996.
  * Bertsekas et al. (2011) D. P. Bertsekas et al.  _Dynamic programming and optimal control 3rd edition, volume ii_.  Belmont, MA: Athena Scientific, 2011.
  * Bharti and Babu (2017) S. K. Bharti and K. S. Babu.  Automatic keyword extraction for text summarization: A survey.  _arXiv preprint arXiv:1704.03242_ , 2017.
  * Bilmes (2022) J. Bilmes.  Submodularity in machine learning and artificial intelligence.  _arXiv preprint arXiv:2202.00132_ , 2022.
  * Bird et al. (2009) S. Bird, E. Klein, and E. Loper.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  ” O’Reilly Media, Inc.”, 2009.
  * Bishop and Nasrabadi (2006) C. M. Bishop and N. M. Nasrabadi.  _Pattern recognition and machine learning_ , volume 4.  Springer, 2006.
  * Blagec et al. (2022) K. Blagec, G. Dorffner, M. Moradi, S. Ott, and M. Samwald.  A global analysis of metrics used for measuring performance in natural language processing.  _arXiv preprint arXiv:2204.11574_ , 2022.
  * Blei et al. (2003) D. M. Blei, A. Y. Ng, and M. I. Jordan.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022, 2003.
  * Bonacich (1972) P. Bonacich.  Factoring and weighting approaches to status scores and clique identification.  _Journal of mathematical sociology_ , 2(1):113–120, 1972.
  * Brandow et al. (1995) R. Brandow, K. Mitze, and L. F. Rau.  Automatic condensation of electronic publications by sentence selection.  _Information Processing & Management_, 31(5):675–685, 1995.
  * Breiman (2001) L. Breiman.  Random forests.  _Machine learning_ , 45(1):5–32, 2001.
  * Breiman et al. (1984) L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone.  _Classification and Regression Trees_.  Chapman and Hall/CRC, 1984.
  * Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Bullers et al. (2018) K. Bullers, A. M. Howard, A. Hanson, W. D. Kearns, J. J. Orriola, R. L. Polo, and K. A. Sakmar.  It takes longer than you think: librarian time spent on systematic review tasks.  _Journal of the Medical Library Association: JMLA_ , 106(2):198, 2018.
  * Burges et al. (2005) C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender.  Learning to rank using gradient descent.  In _Proceedings of the 22nd international conference on Machine learning_ , pages 89–96, 2005.
  * Cabral et al. (2014) L. d. S. Cabral, R. D. Lins, R. F. Mello, F. Freitas, B. Ávila, S. Simske, and M. Riss.  A platform for language independent summarization.  In _Proceedings of the 2014 ACM symposium on Document engineering_ , pages 203–206, 2014.
  * Cabrera-Diego and Torres-Moreno (2018) L. A. Cabrera-Diego and J.-M. Torres-Moreno.  Summtriver: A new trivergent model to evaluate summaries automatically without human references.  _Data & Knowledge Engineering_, 113:184–197, 2018.
  * Canhasi and Kononenko (2011) E. Canhasi and I. Kononenko.  Semantic role frames graph-based multidocument summarization.  In _Proc. SiKDD’11_ , 2011.
  * Cao et al. (2016) Z. Cao, W. Li, S. Li, F. Wei, and Y. Li.  Attsum: Joint learning of focusing and summarization with neural attention.  _arXiv preprint arXiv:1604.00125_ , 2016.
  * Carbonell and Goldstein (1998) J. Carbonell and J. Goldstein.  The use of mmr, diversity-based reranking for reordering documents and producing summaries.  In _Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’98, page 335–336, New York, NY, USA, 1998. Association for Computing Machinery.
  * Carenini et al. (2007) G. Carenini, R. T. Ng, and X. Zhou.  Summarizing email conversations with clue words.  In _Proceedings of the 16th international conference on World Wide Web_ , pages 91–100, 2007.
  * Chang and Chien (2009) Y.-L. Chang and J.-T. Chien.  Latent dirichlet learning for document summarization.  In _2009 IEEE international conference on acoustics, speech and signal processing_ , pages 1689–1692. IEEE, 2009.
  * Chen et al. (2021) J. Chen, D. Tam, C. Raffel, M. Bansal, and D. Yang.  An empirical survey of data augmentation for limited data learning in nlp.  _arXiv preprint arXiv:2106.07499_ , 2021.
  * Chen and Verma (2006) P. Chen and R. Verma.  A query-based medical information summarization system using ontology knowledge.  In _19th IEEE Symposium on Computer-Based Medical Systems (CBMS’06)_ , pages 37–42, 2006.
  * Chen et al. (2018) P. Chen, F. Wu, T. Wang, and W. Ding.  A semantic qa-based approach for text summarization evaluation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Cheng and Lapata (2016) J. Cheng and M. Lapata.  Neural summarization by extracting sentences and words.  _arXiv preprint arXiv:1603.07252_ , 2016.
  * Cho et al. (2014) K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Chung et al. (2014) J. Chung, C. Gulcehre, K. Cho, and Y. Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ , 2014.
  * Clarke and Lapata (2006) J. Clarke and M. Lapata.  Models for sentence compression: A comparison across domains, training requirements and evaluation measures.  In _Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics_ , pages 377–384, 2006.
  * Cohan and Goharian (2016) A. Cohan and N. Goharian.  Revisiting summarization evaluation for scientific articles.  _arXiv preprint arXiv:1604.00400_ , 2016.
  * Cohan et al. (2018) A. Cohan, F. Dernoncourt, D. S. Kim, T. Bui, S. Kim, W. Chang, and N. Goharian.  A discourse-aware attention model for abstractive summarization of long documents.  _arXiv preprint arXiv:1804.05685_ , 2018.
  * Collobert and Weston (2008) R. Collobert and J. Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _Proceedings of the 25th international conference on Machine learning_ , pages 160–167, 2008.
  * Conroy and O’leary (2001) J. M. Conroy and D. P. O’leary.  Text summarization via hidden markov models.  In _Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 406–407, 2001.
  * Cormen et al. (2022) T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein.  _Introduction to algorithms_.  MIT press, 2022.
  * Cortes and Vapnik (1995) C. Cortes and V. Vapnik.  Support-vector networks.  _Machine learning_ , 20(3):273–297, 1995.
  * Costa et al. (2007) L. d. F. Costa, F. A. Rodrigues, G. Travieso, and P. R. Villas Boas.  Characterization of complex networks: A survey of measurements.  _Advances in physics_ , 56(1):167–242, 2007.
  * Crochemore and Rytter (1994) M. Crochemore and W. Rytter.  _Text algorithms_.  Maxime Crochemore, 1994.
  * Dalal and Malik (2013) V. Dalal and L. Malik.  A survey of extractive and abstractive text summarization techniques.  In _2013 6th International Conference on Emerging Trends in Engineering and Technology_ , pages 109–110. IEEE, 2013.
  * Damova and Koychev (2010) M. Damova and I. Koychev.  Query-based summarization: A survey.  2010\.
  * Dang (2005) H. T. Dang.  Overview of duc 2005.  In _Proceedings of the document understanding conference_ , volume 2005, pages 1–12, 2005.
  * Dang et al. (2008) H. T. Dang, K. Owczarzak, et al.  Overview of the tac 2008 update summarization task.  In _TAC_ , 2008.
  * Dang et al. (2009) H. T. Dang, K. Owczarzak, et al.  Overview of TAC 2009 Summarization Track, 2009.  URL https://tac.nist.gov/publications/2009/presentations/TAC2009_Summ_overview.pdf.
  * Dang et al. (2010) H. T. Dang, K. Owczarzak, et al.  Overview of TAC 2010 Summarization Track, 2010.  URL https://tac.nist.gov/publications/2010/presentations/TAC2010_Summ_Overview.pdf.
  * Dang et al. (2011) H. T. Dang, K. Owczarzak, et al.  Overview of TAC 2011 Summarization Track, 2011.  URL https://www.nist.gov/tac/2011/Summarization/#:~:text=TAC%202011%20Summarization%20Tasks&text=The%202011%20Summarization%20Track%20has,frequencies%20to%20select%20important%20concepts.
  * Darling (2010) W. M. Darling.  Multi-document summarization from first principles.  In _TAC_ , 2010.
  * Das and Martins (2007) D. Das and A. Martins.  A survey on automatic text summarization (tech. rep.).  _Literature Survey for the Language and Statistics II course at Carnegie Mellon University_ , 2007.
  * Delort and Alfonseca (2012) J.-Y. Delort and E. Alfonseca.  Dualsum: a topic-model based approach for update summarization.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 214–223, 2012.
  * Denny and Spirling (2018) M. J. Denny and A. Spirling.  Text preprocessing for unsupervised learning: Why it matters, when it misleads, and what to do about it.  _Political Analysis_ , 26(2):168–189, 2018.
  * Dernoncourt et al. (2018) F. Dernoncourt, M. Ghassemi, and W. Chang.  A repository of corpora for summarization.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018.
  * Devlin et al. (2018) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Domingos and Pazzani (1997) P. Domingos and M. Pazzani.  On the optimality of the simple bayesian classifier under zero-one loss.  _Machine learning_ , 29(2):103–130, 1997.
  * Dong et al. (2019) L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/c20bb2d9a50d5ac1f713f8b34d9aac5a-Paper.pdf.
  * Duda et al. (1973) R. O. Duda, P. E. Hart, et al.  _Pattern classification and scene analysis_ , volume 3.  Wiley New York, 1973.
  * Dumais (1991) S. T. Dumais.  Improving the retrieval of information from external sources.  _Behavior Research Methods, Instruments, & Computers_, 23(2):229–236, Jun 1991.  ISSN 1532-5970.  doi: 10.3758/BF03203370.  URL https://doi.org/10.3758/BF03203370.
  * Edelman and Jeong (2021) A. Edelman and S. Jeong.  Fifty three matrix factorizations: A systematic approach.  _arXiv preprint arXiv:2104.08669_ , 2021.
  * Edmunds and Morris (2000) A. Edmunds and A. Morris.  The problem of information overload in business organisations: a review of the literature.  _International journal of information management_ , 20(1):17–28, 2000.
  * Edmundson (1969) H. P. Edmundson.  New methods in automatic extracting.  _Journal of the ACM_ , 16(2):264–285, apr 1969\.
  * Edmundson and Wyllys (1961) H. P. Edmundson and R. E. Wyllys.  Automatic abstracting and indexing—survey and recommendations.  _Communications of the ACM_ , 4(5):226–234, 1961\.
  * El-Kassas et al. (2021) W. S. El-Kassas, C. R. Salama, A. A. Rafea, and H. K. Mohamed.  Automatic text summarization: A comprehensive survey.  _Expert Systems with Applications_ , 165:113679, 2021.  ISSN 0957-4174.
  * Elman (1990) J. L. Elman.  Finding structure in time.  _Cognitive science_ , 14(2):179–211, 1990.
  * Erkan and Radev (2004) G. Erkan and D. R. Radev.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal Artificial Intelligence Research_ , 22(1):457–479, 2004.  ISSN 1076-9757.
  * Estrada (2012) E. Estrada.  _The structure of complex networks: theory and applications_.  Oxford University Press, 2012.
  * Eyal et al. (2019) M. Eyal, T. Baumel, and M. Elhadad.  Question answering as an automatic evaluation metric for news article summarization.  _arXiv preprint arXiv:1906.00318_ , 2019.
  * Fabbri et al. (2019) A. Fabbri, I. Li, T. She, S. Li, and D. Radev.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  pages 1074–1084, 2019.
  * Fattah and Ren (2009) M. A. Fattah and F. Ren.  Ga, mr, ffnn, pnn and gmm based models for automatic text summarization.  _Computer Speech & Language_, 23(1):126–144, 2009.
  * Fedus et al. (2021) W. Fedus, B. Zoph, and N. Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _arXiv preprint arXiv:2101.03961_ , 2021.
  * Fellbaum (2000) C. D. Fellbaum.  Wordnet : an electronic lexical database.  _Language_ , 76:706, 2000.
  * Feng et al. (2021) S. Y. Feng, V. Gangal, J. Wei, S. Chandar, S. Vosoughi, T. Mitamura, and E. Hovy.  A survey of data augmentation approaches for nlp.  _arXiv preprint arXiv:2105.03075_ , 2021.
  * Ferreira et al. (2013) R. Ferreira, L. de Souza Cabral, R. D. Lins, G. P. e Silva, F. Freitas, G. D. Cavalcanti, R. Lima, S. J. Simske, and L. Favaro.  Assessing sentence scoring techniques for extractive text summarization.  _Expert systems with applications_ , 40(14):5755–5764, 2013.
  * Filippova (2010) K. Filippova.  Multi-sentence compression: Finding shortest paths in word graphs.  In _Proceedings of the 23rd international conference on computational linguistics (Coling 2010)_ , pages 322–330, 2010.
  * Fleder and Hosanagar (2009) D. Fleder and K. Hosanagar.  Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity.  _Management science_ , 55(5):697–712, 2009.
  * Friedman et al. (2010) J. Friedman, T. Hastie, and R. Tibshirani.  Regularization paths for generalized linear models via coordinate descent.  _Journal of statistical software_ , 33(1):1, 2010\.
  * Friedman (2001) J. H. Friedman.  Greedy function approximation: a gradient boosting machine.  _Annals of statistics_ , pages 1189–1232, 2001.
  * Gambhir and Gupta (2017) M. Gambhir and V. Gupta.  Recent automatic text summarization techniques: a survey.  _Artificial Intelligence Review_ , 47(1):1–66, 2017.
  * Ganesan (2018) K. Ganesan.  Rouge 2.0: Updated and improved measures for evaluation of summarization tasks.  _arXiv preprint arXiv:1803.01937_ , 2018.
  * Ganesan et al. (2010) K. Ganesan, C. Zhai, and J. Han.  Opinosis: A graph based approach to abstractive summarization of highly redundant opinions.  2010\.
  * Gao et al. (2018) Y. Gao, A. Warner, and R. J. Passonneau.  Pyreval: An automated method for summary content analysis.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018.
  * Gao et al. (2019) Y. Gao, C. Sun, and R. J. Passonneau.  Automated pyramid summarization evaluation.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , 2019.
  * Gao et al. (2020) Y. Gao, W. Zhao, and S. Eger.  Supert: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization.  _arXiv preprint arXiv:2005.03724_ , 2020.
  * Genest and Lapalme (2011) P.-E. Genest and G. Lapalme.  Framework for abstractive summarization using text-to-text generation.  In _Proceedings of the workshop on monolingual text-to-text generation_ , pages 64–73, 2011.
  * Genest and Lapalme (2012) P.-E. Genest and G. Lapalme.  Fully abstractive approach to guided summarization.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 354–358, 2012\.
  * Gentzkow et al. (2019) M. Gentzkow, B. Kelly, and M. Taddy.  Text as data.  _Journal of Economic Literature_ , 57(3):535–74, September 2019.
  * Gers et al. (2000) F. A. Gers, J. Schmidhuber, and F. Cummins.  Learning to forget: Continual prediction with lstm.  _Neural computation_ , 12(10):2451–2471, 2000\.
  * Gholamrezazadeh et al. (2009) S. Gholamrezazadeh, M. A. Salehi, and B. Gholamzadeh.  A comprehensive survey on text summarization systems.  In _2009 2nd International Conference on Computer Science and its Applications_ , pages 1–6, 2009.
  * Gillick et al. (2008) D. Gillick, B. Favre, and D. Hakkani-Tür.  The icsi summarization system at tac 2008.  In _Tac_ , 2008.
  * Gillis (2020) N. Gillis.  _Nonnegative Matrix Factorization_.  SIAM, 2020.
  * Gliwa et al. (2019) B. Gliwa, I. Mochol, M. Biesek, and A. Wawer.  Samsum corpus: A human-annotated dialogue dataset for abstractive summarization.  _arXiv preprint arXiv:1911.12237_ , 2019.
  * Golub and Loan (2013) G. H. Golub and C. F. V. Loan.  _Matrix Computations_.  Johns Hopkins University Press, 2013.
  * Gong and Liu (2001) Y. Gong and X. Liu.  Generic text summarization using relevance measure and latent semantic analysis.  In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’01, page 19–25, New York, NY, USA, 2001. Association for Computing Machinery.  ISBN 1581133316.
  * González-Castro et al. (2013) V. González-Castro, R. Alaiz-Rodríguez, and E. Alegre.  Class distribution estimation based on the hellinger distance.  _Information Sciences_ , 218:146–164, 2013.
  * Goodfellow et al. (2016) I. Goodfellow, Y. Bengio, and A. Courville.  _Deep learning_.  MIT press, 2016.
  * Graff et al. (2003) D. Graff, J. Kong, K. Chen, and K. Maeda.  English gigaword.  _Linguistic Data Consortium, Philadelphia_ , 4(1):34, 2003.
  * Greenbacker (2011) C. Greenbacker.  Towards a framework for abstractive summarization of multimodal documents.  In _Proceedings of the ACL 2011 Student Session_ , pages 75–80, 2011\.
  * Grefenstette (1998) G. Grefenstette.  Producing intelligent telegraphic text reduction to provide an audio scanning service for the blind.  In _Intelligent Text Summarization, AAAI Spring Symposium Series_ , pages 111–117, 1998.
  * Grusky et al. (2018) M. Grusky, M. Naaman, and Y. Artzi.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  _arXiv preprint arXiv:1804.11283_ , 2018.
  * Gupta and Gupta (2019) S. Gupta and S. K. Gupta.  Abstractive summarization: An overview of the state of the art.  _Expert Systems with Applications_ , 121:49–65, 2019.
  * Gupta and Lehal (2010) V. Gupta and G. S. Lehal.  A survey of text summarization extractive techniques.  _Journal of emerging technologies in web intelligence_ , 2(3):258–268, 2010.
  * Gutiérrez and Keith (2018) L. Gutiérrez and B. Keith.  A systematic literature review on word embeddings.  In _International Conference on Software Process Improvement_ , pages 132–141. Springer, 2018.
  * Haghighi and Vanderwende (2009) A. Haghighi and L. Vanderwende.  Exploring content models for multi-document summarization.  In _Proceedings of human language technologies: The 2009 annual conference of the North American Chapter of the Association for Computational Linguistics_ , pages 362–370, 2009.
  * Hailu et al. (2020) T. T. Hailu, J. Yu, and T. G. Fantaye.  A framework for word embedding based automatic text summarization and evaluation.  _Information_ , 11(2):78, 2020.
  * Harabagiu and Lacatusu (2002) S. M. Harabagiu and F. Lacatusu.  Generating single and multi-document summaries with gistexter.  In _Document Understanding Conferences_ , pages 11–12, 2002.
  * Harnly et al. (2005) A. Harnly, A. Nenkova, R. Passonneau, and O. Rambow.  Automation of summary evaluation by the pyramid method.  In _Recent Advances in Natural Language Processing (RANLP)_ , pages 226–232, 2005.
  * Hasan and Ng (2014) K. S. Hasan and V. Ng.  Automatic keyphrase extraction: A survey of the state of the art.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1262–1273, 2014\.
  * Hasan et al. (2021) T. Hasan, A. Bhattacharjee, M. S. Islam, K. Samin, Y.-F. Li, Y.-B. Kang, M. S. Rahman, and R. Shahriyar.  Xl-sum: Large-scale multilingual abstractive summarization for 44 languages.  _arXiv preprint arXiv:2106.13822_ , 2021.
  * Hasler et al. (2003) L. Hasler, C. Orăsan, and R. Mitkov.  Building better corpora for summarisation.  In _Proceedings of Corpus Linguistics 2003_ , pages 309 – 319, Lancaster, UK, March 2003.  URL http://clg.wlv.ac.uk/papers/hasler-CL-03.pdf.
  * Haykin (1994) S. Haykin.  _Neural networks: a comprehensive foundation_.  Prentice-Hall, 1994.
  * Henß et al. (2015) S. Henß, M. Mieskes, and I. Gurevych.  A reinforcement learning approach for adaptive single-and multi-document summarization.  In _GSCL_ , pages 3–12, 2015.
  * Hermann et al. (2015) K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28, 2015.
  * Hervas-Drane (2008) A. Hervas-Drane.  Word of mouth and recommender systems: A theory of the long tail.  2008\.
  * Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Hori and Furui (2004) C. Hori and S. Furui.  Speech summarization: an approach through word extraction and a method for evaluation.  _IEICE TRANSACTIONS on Information and Systems_ , 87(1):15–25, 2004.
  * Housen (2022) H. Housen.  Transformersum.  https://github.com/HHousen/TransformerSum, 2022.
  * Hovy and Lim (1999) E. Hovy and C. Y. Lim.  Automated multilingual text summarization and its evaluation.  Technical report, University of Southern California, 1999.
  * Hovy et al. (2005) E. Hovy, C.-Y. Lin, and L. Zhou.  Evaluating duc 2005 using basic elements.  In _Proceedings of DUC_ , volume 2005. Citeseer, 2005.
  * Hovy et al. (2006) E. H. Hovy, C.-Y. Lin, L. Zhou, and J. Fukumoto.  Automated summarization evaluation with basic elements.  In _LREC_ , volume 6, pages 604–611. Citeseer, 2006.
  * Huang et al. (2021) L. Huang, S. Cao, N. Parulian, H. Ji, and L. Wang.  Efficient attentions for long document summarization.  _arXiv preprint arXiv:2104.02112_ , 2021.
  * Hulth (2003) A. Hulth.  Improved automatic keyword extraction given more linguistic knowledge.  In _Proceedings of the 2003 conference on Empirical methods in natural language processing_ , pages 216–223, 2003.
  * Izenman (2008) A. J. Izenman.  _Modern multivariate statistical techniques: Regression, classification and manifold learning_.  Springer, 2008.
  * Jain et al. (2017) A. Jain, D. Bhatia, and M. K. Thakur.  Extractive text summarization using word vector embedding.  In _2017 International Conference on Machine Learning and Data Science (MLDS)_ , pages 51–55, 2017.  doi: 10.1109/MLDS.2017.12.
  * Jalil et al. (2021) Z. Jalil, J. A. Nasir, and M. Nasir.  Extractive multi-document summarization: A review of progress in the last decade.  _IEEE Access_ , 2021.
  * Janin et al. (2003) A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, et al.  The icsi meeting corpus.  In _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)._ , volume 1, pages I–I. IEEE, 2003.
  * Jing (2000) H. Jing.  Sentence reduction for automatic text summarization.  In _Sixth applied natural language processing conference_ , pages 310–315, 2000.
  * Jing and McKeown (2000) H. Jing and K. McKeown.  Cut and paste based text summarization.  In _1st Meeting of the North American Chapter of the Association for Computational Linguistics_ , 2000.
  * Jing et al. (1998) H. Jing, R. Barzilay, K. McKeown, and M. Elhadad.  Summarization evaluation methods: Experiments and analysis.  In _AAAI symposium on intelligent summarization_ , pages 51–59. Palo Alto, CA, 1998.
  * Joachims (2006) T. Joachims.  Training linear svms in linear time.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 217–226, 2006.
  * Jones (1998) K. S. Jones.  Automatic summarising: Factors and directions.  In _Advances in Automatic Text Summarization_ , pages 1–12. MIT Press, 1998.
  * Jordan (1986) M. Jordan.  Serial order: a parallel distributed processing approach. technical report, june 1985-march 1986.  Technical report, California Univ., San Diego, La Jolla (USA). Inst. for Cognitive Science, 1986.
  * Joshi and Schabes (1997) A. K. Joshi and Y. Schabes.  Tree-adjoining grammars.  In _Handbook of formal languages_ , pages 69–123. Springer, 1997\.
  * Joulin et al. (2016) A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_ , 2016.  URL https://arxiv.org/abs/1607.01759.
  * Kågebäck et al. (2014) M. Kågebäck, O. Mogren, N. Tahmasebi, and D. Dubhashi.  Extractive summarization using continuous vector space models.  In _Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC)_ , pages 31–39, 2014.
  * Khurana and Bhatnagar (2022) A. Khurana and V. Bhatnagar.  Investigating entropy for extractive document summarization.  _Expert Systems with Applications_ , 187:115820, 2022.
  * Kim et al. (2018) B. Kim, H. Kim, and G. Kim.  Abstractive summarization of reddit posts with multi-level memory networks.  _arXiv preprint arXiv:1811.00783_ , 2018.
  * Klema and Laub (1980) V. Klema and A. Laub.  The singular value decomposition: Its computation and some applications.  _IEEE Transactions on automatic control_ , 25(2):164–176, 1980.
  * Klymenko et al. (2020) O. Klymenko, D. Braun, and F. Matthes.  Automatic text summarization: A state-of-the-art review.  _ICEIS (1)_ , pages 648–655, 2020.
  * Knight and Marcu (2000) K. Knight and D. Marcu.  Statistics-based summarization-step one: Sentence compression.  _AAAI/IAAI_ , 2000:703–710, 2000.
  * Kornilova and Eidelman (2019) A. Kornilova and V. Eidelman.  Billsum: A corpus for automatic summarization of us legislation.  _arXiv preprint arXiv:1910.00523_ , 2019.
  * Koupaee and Wang (2018) M. Koupaee and W. Y. Wang.  Wikihow: A large scale text summarization dataset.  _arXiv preprint arXiv:1810.09305_ , 2018.
  * Ku et al. (2006) L.-W. Ku, Y.-T. Liang, and H.-H. Chen.  Opinion extraction, summarization and tracking in news and blog corpora.  In _Proceedings of AAAI_ , pages 100–107, 2006.
  * Kumar and Salim (2012) Y. J. Kumar and N. Salim.  Automatic multi document summarization approaches.  In _KS Gayathri, Received BE degree in CSE from Madras University in 2001 and ME degree from Anna University, Chennai. She is doing Ph. D. in the area of Reasoning in Smart_. Citeseer, 2012.
  * Kupiec et al. (1995) J. Kupiec, J. Pedersen, and F. Chen.  A trainable document summarizer.  In _Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’95, page 68–73, 1995.
  * LeCun et al. (1989) Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural computation_ , 1(4):541–551, 1989.
  * LeCun et al. (2015) Y. LeCun et al.  Lenet-5, convolutional neural networks.  _URL: http://yann. lecun. com/exdb/lenet_ , 20(5):14, 2015.
  * Lee et al. (2005) C.-S. Lee, Z.-W. Jian, and L.-K. Huang.  A fuzzy ontology and its application to news summarization.  _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , 35(5):859–880, 2005.
  * Lee and Seung (2000) D. Lee and H. S. Seung.  Algorithms for non-negative matrix factorization.  _Advances in neural information processing systems_ , 13, 2000.
  * Lee et al. (2020) D. Lee, M. Shin, T. Whang, S. Cho, B. Ko, D. Lee, E. Kim, and J. Jo.  Reference and document aware semantic evaluation methods for korean language summarization.  _arXiv preprint arXiv:2005.03510_ , 2020.
  * Lee and Seung (1999) D. D. Lee and H. S. Seung.  Learning the parts of objects by non-negative matrix factorization.  _Nature_ , 401(6755):788–791, 1999.
  * Lee and Lee (2017) G. H. Lee and K. J. Lee.  Automatic text summarization using reinforcement learning with embedding features.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 193–197, 2017.
  * Lee et al. (2009) J.-H. Lee, S. Park, C.-M. Ahn, and D. Kim.  Automatic generic document summarization based on non-negative matrix factorization.  _Information Processing & Management_, 45(1):20–34, 2009.
  * Leskovec et al. (2004a) J. Leskovec, M. Grobelnik, and N. Milic-Frayling.  Learning semantic graph mapping for document summarization.  In _Proceedings of ECML/PKDD-2004 workshop on knowledge discovery and ontologies_ , 2004a.
  * Leskovec et al. (2004b) J. Leskovec, M. Grobelnik, and N. Milic-Frayling.  Learning sub-structures of document semantic graphs for document summarization.  In _LinkKDD Workshop_ , volume 133, page 138, 2004b.
  * Leskovec et al. (2005) J. Leskovec, N. Milic-Frayling, and M. Grobelnik.  Extracting summary sentences based on the document semantic graph.  2005\.
  * Levandowsky and Winter (1971) M. Levandowsky and D. Winter.  Distance between sets.  _Nature_ , 234:34–35, 1971.
  * Levy et al. (2015) O. Levy, Y. Goldberg, and I. Dagan.  Improving distributional similarity with lessons learned from word embeddings.  _TACL_ , 3:211–225, 2015.  URL https://levyomer.wordpress.com/2015/03/30/improving-distributional-similarity-with-lessons-learned-from-word-embeddings/.
  * Lewis et al. (2019) M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Li et al. (2021a) J. Li, T. Tang, W. X. Zhao, and J.-R. Wen.  Pretrained language models for text generation: A survey.  _arXiv preprint arXiv:2105.10311_ , 2021a.
  * Li et al. (2021b) Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou.  A survey of convolutional neural networks: analysis, applications, and prospects.  _IEEE transactions on neural networks and learning systems_ , 2021b.
  * Lin (2001) C.-Y. Lin.  Summary evaluation environment.  http://www1.cs.columbia.edu/nlp/tides/SEEManual.pdf, 2001.  Accessed: 2022-05-23.
  * Lin (2004) C.-Y. Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Lin and Hovy (1997) C.-Y. Lin and E. Hovy.  Identifying topics by position.  In _Fifth Conference on Applied Natural Language Processing_ , pages 283–290, 1997.
  * Lin and Hovy (2003) C.-Y. Lin and E. Hovy.  Automatic evaluation of summaries using n-gram co-occurrence statistics.  In _Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics_ , pages 150–157, 2003.
  * Lin and Pantel (2002) D. Lin and P. Pantel.  Concept discovery from text.  In _COLING 2002: The 19th International Conference on Computational Linguistics_ , 2002.
  * Lin and Bilmes (2011) H. Lin and J. Bilmes.  A class of submodular functions for document summarization.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , pages 510–520, 2011\.
  * Lin et al. (2009) H. Lin, J. Bilmes, and S. Xie.  Graph-based submodular selection for extractive summarization.  In _2009 IEEE Workshop on Automatic Speech Recognition & Understanding_, pages 381–386. IEEE, 2009.
  * Lins et al. (2019) R. D. Lins, H. Oliveira, L. Cabral, J. Batista, B. Tenorio, R. Ferreira, R. Lima, G. de França Pereira e Silva, and S. J. Simske.  The cnn-corpus: A large textual corpus for single-document extractive summarization.  In _Proceedings of the ACM Symposium on Document Engineering 2019_ , pages 1–10, 2019.
  * Liu and Liu (2008) F. Liu and Y. Liu.  Correlation between rouge and human evaluation of extractive meeting summaries.  In _Proceedings of ACL-08: HLT, short papers_ , pages 201–204, 2008\.
  * Liu et al. (2017) J. Liu, Z. Liu, and H. Chen.  Revisit word embeddings with semantic lexicons for modeling lexical contrast.  In _2017 IEEE International Conference on Big Knowledge (ICBK)_ , pages 72–79, Aug 2017.  doi: 10.1109/ICBK.2017.35.
  * Liu et al. (2013) Q. Liu, Y. Liu, D. Wu, and X. Cheng.  Ictnet at temporal summarization track trec 2013.  In _TREC_ , 2013.
  * Liu et al. (2020) Q. Liu, M. J. Kusner, and P. Blunsom.  A survey on contextual embeddings.  _arXiv preprint arXiv:2003.07278_ , 2020.
  * Liu and Lapata (2019) Y. Liu and M. Lapata.  Text summarization with pretrained encoders.  _arXiv preprint arXiv:1908.08345_ , 2019.
  * Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.  URL https://arxiv.org/abs/1907.11692.
  * Lloret and Palomar (2012) E. Lloret and M. Palomar.  Text summarisation in progress: a literature review.  _Artificial Intelligence Review_ , 2012.