# A General Contextualized Rewriting Framework for Text Summarization

Guangsheng Bao and Yue Zhang Manuscript submitted to TASLP.Guangsheng Bao is
with the School of Engineering, Westlake University, Hangzhou 310024, China
(email: baoguangsheng@westlake.edu.cn).Yue Zhang is with the School of
Engineering, Westlake University, Hangzhou 310024, China, and also with the
Institute of Advanced Technology, Westlake Institute of Advanced Study,
Hangzhou 310024, China (email: yue.zhang@wias.org.cn).

###### Abstract

The rewriting method for text summarization combines extractive and
abstractive approaches, improving the conciseness and readability of
extractive summaries using an abstractive model. Exiting rewriting systems
take each extractive sentence as the only input, which is relatively focused
but can lose necessary background knowledge and discourse context. In this
paper, we investigate contextualized rewriting, which consumes the entire
document and considers the summary context. We formalize contextualized
rewriting as a seq2seq with group-tag alignments, introducing group-tag as a
solution to model the alignments, identifying extractive sentences through
content-based addressing. Results show that our approach significantly
outperforms non-contextualized rewriting systems without requiring
reinforcement learning, achieving strong improvements on ROUGE scores upon
multiple extractors.

###### Index Terms:

Text Summarization, Abstractive, Rewriting, Contextualized Rewriting, Joint
Model.

##  I Introduction

Automatic text summarization [1, 2] is the task that compresses input
documents into shorter summaries while keeping their salient information. It
has been tackled mainly by two basic methods, namely, extractive and
abstractive. The _extractive_ method generates a summary by extracting
important text pieces (typically sentences) from a document and concatenating
them to form the summary [3, 4, 5]. It has the advantage in content selection
and faithfulness compared to the abstractive method [6, 7, 8]. However, the
extractive sentences may contain irrelevant or redundant information [9, 10,
11] and may have low coherence since the discourse relations and cross-
sentence anaphora are not maintained [12, 13]. The _abstractive_ method uses a
conditional language model to generate the summary from scratch token by token
[6, 7, 14], producing more fluent and readable content. A line of work
proposes a rewriting method to combine the advantage of extractive and
abstractive methods. It paraphrases the extractive sentences using an
abstractive model, removing irrelevant information and normalizing the
expressions. Various rewriting systems have been developed, including sentence
compression [9], syntax simplification [12], and paraphrasing [10, 15, 16,
17]. The human evaluation shows that rewriting methods can improve the
readability and conciseness of extractive summaries [15, 17].

Source Document:  “ Success Kid ” is likely the Internet ’s most famous baby .
You ’ve seen him in dozens of memes , fist clenched in a determined look of
persevering despite the odds . Success Kid – now an 8-year-old named Sammy
Griner – needs a little bit of that mojo to rub off on his family . His dad ,
Justin , needs a kidney transplant . About a week ago , Laney Griner , Justin
’s wife and Sammy ’s mother , created a GoFundMe campaign with a goal of $
75,000 to help cover the medical expenses that go along with a kidney
transplant … Extractive Summary:  His dad , Justin , needs a kidney transplant
. Rewritten Summary:  “ Success Kid ” Sammy Griner ’s dad , Justin , needs a
kidney transplant . Gold Summary:  Justin Griner , the dad of “ Success Kid ,
” needs a kidney transplant .

Figure 1: Key information from document context complements extractive
summary, improving its informativeness.

Source Document:  ‘ Queen of celebrity ’ is not lifetime role - before Kim
Kardashian there was Paris Hilton and now there is another glossy young female
fighting for the crown . A model and TV presenter known as the Mexican Kim
Kardashian is hoping to dethrone the real Kim after gathering a huge support
base on social media . 3 Jimena Sanchez , a 30 year-old Mexican model and TV
presenter , is four years younger than Kim and works as a sports presenter for
the Latin American division of Fox Sports , Fox Deportes . 1 Scroll down for
video . Jimena Sanchez ( left ) is a model and TV presenter from Mexico who
has been labelled the ‘ Mexican Kim Kardashian ’ for her similarity to the
reality star ( right ) . 2 But her real fame comes from the mass of social
media attention that she gets , with one million followers on Twitter and also
more than one million Facebook likes on her official page . Rewritten Summary:
Jimena Sanchez is a 30 year-old Mexican model and sports TV presenter . 1 The
star is called ‘ Mexican Kim Kardashian ’ because of their similar looks .  2
She is now hoping to become the most popular woman on social media . 3

Figure 2: Simple mentions to the entity in summary context simplify the
complex expression of subjects in extractive sentences and improves its
coherence.

There are still issues with these rewriting methods. First, these methods
generate summaries only according to the extractive sentences. Therefore, if
some critical information appears in the document but not in the extractive
sentences, it would be impossible for the rewriter to generate them. As an
example in Fig. 1 shows, an extractive summary can be created by extracting
the salient sentence “His dad , Justin , needs a kidney transplant .” A
rewriter can rearrange the expression but cannot resolve the entity “Sammy
Griner” and its mention “Success Kid” without considering the document context
during rewriting.

Second, these methods rewrite sentence by sentence independently. As a result,
the cross-sentence coherence, such as entity coreference, is not modeled and
controlled. Take the case in Fig. 2 as an example. The subjects of the three
extractive sentences are redundant and reference to the same entity “Jimena
Sanchez”, which requires simple mentions such as “the star” and “she” in the
summary sentences for conciseness and better coherence. A rewriter needs to
consider the summary context of each sentence in order to remove redundancy
and improve its readability.

We propose contextualized rewriting, considering both document and summary
context during rewriting, rather than conditioning only on the extractive
sentence. In particular, we represent extractive sentences as a part of the
document representation, introducing group-tags to represent the alignment
between summary and extractive sentences. Specifically, as Fig. 2 shows, we
allocate the group-tags \scriptsize1⃝ \scriptsize2⃝ and \scriptsize3⃝ to the
first, second, and third summary sentences. We mark the corresponding
extractive sentences using the same group-tags, through which the decoder can
locate the corresponding extractive sentence during rewriting.

Based on the scheme, we propose a general framework of seq2seq with group-tag
alignments for rewriting with an external or a joint internal extractor,
representing sentence selection as one of the token prediction steps during
decoding. This framework is independent of the implementation details of the
seq2seq model, thus common for different abstractive rewriters. We instantiate
three rewriters by applying the general framework to BERT [18] and BART [19]
models that include two rewriters with an external extractor naming BERT-
Rewriter and BART-Rewriter, and one rewriter with an internal extractor naming
BART-JointSR.

We evaluate the three rewriters using the popular benchmark dataset
CNN/DailyMail. The results suggest that contextualized rewriting can
significantly improve both the ROUGE [20] scores and human scores compared
with previous non-contextualized rewriting methods. The performance
improvement is further enlarged when contextualized rewriting works with the
stronger pre-trained model BART. We further prove that the joint modeling of
sentence selection and rewriting is much more beneficial than the separate
pipeline model from both the efficiency and the effectiveness. To our best
knowledge, we are the first rewriting method that improves ROUGE scores and
human scores against both extractive and abstractive baselines.

This paper significantly extends our conference work [21], which describes an
instance of the general framework using BERT. The addition of this article is
three-fold. First, we apply contextualized rewriting to the pre-trained model
BART and demonstrate the effectiveness of our method on a stronger baseline.
Second, we explore the possibility of joint modeling of sentence selection and
rewriting, further powering our rewriting method by removing the dependence on
an external extractor and demonstrating the advantage of joint modeling. Last,
we generalize our contextualized rewriting design to a framework that suits
both rewriters with an external extractor and an internal extractor, which can
be applied to different seq2seq models. Our code and models are released at
https://github.com/baoguangsheng/bart-rewriters.

Figure 3: Contextualized rewriting involves an extractor selecting sentences
from the input document and a rewriter generating a summary according to both
the selected sentences and the document. (Each line in the rectangles
represents a sentence, and each element w# denotes a word.)

##  II Related Work

Rewriting methods are widely used in various NLP applications, converting text
from one form to another while keeping the original semantics. In information
extraction, rewriting methods are used to generate different expressions of a
query to increase the recall of retrieval [22]. In conversational question
answering, rewriting methods are used to convert questions from a context-
dependent expression to a self-contained expression with referred entities
explicitly mentioned and omitted information recovered [23, 24, 25]. In neural
machine translation, rewriting methods are used to rewrite the retrieved
templates into translations by filling information from source sentences [26].
In text summarization, rewriting methods are used to compress sentences [27],
simplify expressions [28], or fill templates [29]. Studies [10, 15, 17] show
that rewriting methods can enhance extractive summaries on conciseness and
readability while reserving critical information. We further develop these
rewriting methods to consider document context and summary context during
rewriting.

Recent studies model sentence rewriting as conditioned text generation taking
extractive sentences as the inputs. Chen and Bansal [10] use an auto-
regressive sentence extractor and a seq2seq model with the copy mechanism [14]
to rewrite extractive sentences one by one. They tune the extractor using
reinforcement learning with reward signals from rewritten sentences and rerank
rewritten summaries to avoid repetition. Bae et al. [15] adopt a similar
strategy but use a BERT document encoder and reward signals from the whole
summary. Wei et al. [16] use a BERT document encoder to generate sentence
embeddings and upon which a binary classifier is trained to select sentences.
They use a Transformer decoder [30] equipped with the copy mechanism to
generate summary sentences. Xiao et al. [17] use a pointer network to select
sentences and make a decision of copying or rewriting accordingly. If the
model chooses to rewrite, a vanilla seq2seq model will be used to rewrite the
sentence. The model decisions on sentence selection and copying/rewriting are
tuned using reinforcement learning. Compared with these methods, our rewriting
method is _computationally simpler_ for both training and inference since we
do not use reinforcement learning and the copy mechanism as above.
Furthermore, as mentioned earlier, in contrast to these pure sentence
rewriting methods, we consider the _document context_ and the _summary
context_ during rewriting, thereby improving information recall and cross-
sentence coherence. In addition, we propose a joint model of internal
extractor and rewriter, reducing the total model size, the training cost, and
the inference cost of the pipeline approach.

Some hybrid extractive and abstractive summarization models align with our
work. Hsu et al. [31] propose inconsistency loss to encourage the consistency
between word-level and sentence-level attentions. Cheng and Lapata [13]
extract important words first and use them to constrain a language model to
generate a summary. Gehrmann et al. [11] propose a bottom-up method, selecting
important words using a neural classifier and restricting the copy source of a
pointer-generator network to the chosen words to generate a summary. Similar
to our method, they use extracted content to guide the abstractive summarizer.
However, different from their methods, which focus on word-level, we
investigate sentence-level constraints for guiding abstractive rewriting.

Our rewriting method can also be regarded as an abstractive summarizer [6, 7,
14] with attention guidance using group-tags, where the group-tags are
generated from the extractive summary. In comparison with the vanilla
abstractive model, the advantages are three-fold. First, extractive summaries
can guide the abstractive summarizer with more salient information. Second,
the training difficulty of the abstractive model can be reduced when important
contents are marked in inputs. Third, the summarization procedure is made more
interpretable by associating a crucial source sentence with each target
sentence.

##  III Problem Formulation

We formulate contextualized rewriting as conditioned text generation that
maximizes the probability of the summary given an input document. We explore
two alternatives of contextualized rewriters: a rewriter with extractive
summaries provided externally and a rewriter with an internal extractor. The
relative advantage of the former is generality, which can be used on top of
arbitrary extractive summarizers, while the advantage of the latter is
independent usability.

Formally, we use $X$ to denote an input document and $Y$ an output summary
that

$X=\\{X_{i}\\}|_{i=1}^{|X|}\text{ and }Y=\\{Y_{j}\\}|_{j=1}^{|Y|},$

where $X_{i}$ denotes each sentence in $X$ and $|X|$ the number of sentences.
$Y_{j}$ denotes each sentence in $Y$ and $|Y|$ the number of sentences in $Y$.
Here, we use a sentence-level notation instead of word-level or token-level
for the simplicity of discussions because we focus on sentence-level relations
between input document, extractive summary, and rewritten summary.

###  III-A Rewriter for External Extractive Summary

Given extractive summary $E$, our contextualized rewriter rewrites it into the
final summary $Y$ that

$\hat{Y}=\arg\max_{Y}P(Y|E,X),$ (1)

where $Y$ and $E$ have the same number of sentences.

As shown in Fig. 3, we use external extractor to select sentences from input
document $X$ and obtain extractive summary $E$ that

$E=\\{E_{j}=X_{k}|X_{k}\in X\\}|_{j=1}^{|E|},$ (2)

where $|E|$ denotes the number of sentences , which is equal to $|Y|$. Each
sentence $E_{j}$ is from a document sentence $X_{k}$. For instance, given a
document $X=\\{X_{1},X_{2},...,X_{6}\\}$, an extractive summary could be
$E=\\{E_{1}=X_{3},E_{2}=X_{5},E_{3}=X_{2}\\}$ that the first extractive
sentence $E_{1}$ is from the third document sentence $X_{3}$, and so on and so
forth.

The contextualized rewriter rewrites each extractive sentence $E_{j}$ into a
summary sentence $Y_{j}$ given the document context and summary context.
Therefore, the problem formulated in Eq 1 can be further detailed as

$\hat{Y}=\arg\max_{Y}\prod_{j=1}^{|Y|}P(Y_{j}|\mathop{\underline{E_{j}}}_{\begin{subarray}{c}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}rewriting}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}source}\end{subarray}},\mathop{\underline{E_{<j}}}_{\begin{subarray}{c}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}extractive}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}context}\end{subarray}},\mathop{\underline{Y_{<j}}}_{\begin{subarray}{c}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}summary}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}context}\end{subarray}},\mathop{\underline{X}}_{\begin{subarray}{c}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}document}\\\
\text{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}context}\end{subarray}}).$
(3)

It worth noting that contextualized rewriting as expressed in Eq 3 is
different from previous non-contextualized rewriters [15, 16, 17], which do
not use contextual information but directly calculate $P(Y_{j}|E_{j})$.

Figure 4: A general framework for contextualized rewriting which involves
group-tag alignments, sentence selection, and sentence rewriting.

###  III-B Joint Internal Extractor and Rewriter

The external extractor can be integrated into contextualized rewriter,
resulting in a self-contained rewriter. The self-contained rewriter models
both sentence extraction and rewriting in one seq2seq model. The joint problem
can be defined as

$\hat{Y}=\arg_{Y}\max_{Y,E}P(Y,E|X),$ (4)

where $E$ is the selected sentences from document $X$, which is defined as the
same as Eq 2. The equation can be further expanded as

$\hat{Y}=\arg_{Y}\max_{Y,E}\prod_{j=1}^{|Y|}P(Y_{j},E_{j}|Y_{<j},E_{<j},X),$
(5)

where the sentence selection $E_{j}$ and rewriting $Y_{j}$ both depend on the
history of selection and rewriting.

Given that the generation of $Y_{j}$ depends on the sentence selection of
$E_{j}$, we separate the two decision steps as

$\hat{Y}=\arg_{Y}\max_{Y,E}\prod_{j=1}^{|Y|}P(Y_{j}|E_{j},E_{<j},Y_{<j},X)P(E_{j}|E_{<j},Y_{<j},X),$
(6)

in which the current sentence selection $E_{j}$ depends on the previously
selected sentences and rewritten sentences, while current sentence rewriting
$Y_{j}$ depends on the current sentence selection and the history.

It worth noting that the internal extractor $P(E_{j}|E_{<j},Y_{<j},X)$ in Eq 6
is different from the external extractor $P(E_{j}|E_{<j},X)$ in Fig. 3 because
of the joint modeling of $Y$ and $E$. In addition, comparing detailed
expressions in Eq 3 and Eq 6, we could find that they share the same rewriting
part $P(Y_{j}|E_{j},E_{<j},Y_{<j},X)$, for which we will propose a general
implementation in next section.

##  IV A General Framework:
Seq2seq with Group-tag Alignments

As shown in Fig. 4, we model contextualized rewriting as a seq2seq mapping
problem with group-tag alignments. The alignments are expressed in two group-
tag sequences, identifying each sentence alignment by the same group-tag in
both sequences. We extend both the input document and output summary with
special sentence identifiers so that we can generate group-tag sequences
accordingly. We transfer the sentence selection problem into a problem of
predicting the sentence identifier tokens, which can be integrated into the
seq2seq framework easily.

###  IV-A Group-tag Alignments

We introduce sentence identifier tokens such as <S$1$> and <S$2$> to represent
the beginning of each sentence, extending document $X$ to $X^{\prime}$ and
summary $Y$ to $Y^{\prime}$.

####  IV-A1 Group-tag Generation

We generate group-tag sequences according to these identifiers. For a sentence
beginning with <S$k$>, we assign group-tag $k$ to each token in that sentence.
For example, we assign group-tag to each token of a document like “<S$3$>/3
w31/3 w32/3 </S>/3 <S$5$>/5 w52/5 w54/5 </S>/5 ”. Formally, we use
$tokens=\\{w_{i}\\}|_{i=1}^{N}$ to denote the mixed sequence of word tokens
and identifier tokens. Given $tokens$, we generate the group tag sequence
$tags=\\{t_{i}\\}|_{i=1}^{N}$ by Algorithm 1.

Input: $tokens=\\{w_{i}\\}|_{i=1}^{N}$
Output: $tags=\\{t_{i}\\}|_{i=1}^{N}$

1:function GroupTag(tokens)

2: $t\leftarrow 0$ $\triangleright$ default group-tag.

3: for $i\leftarrow 1$ to $N$ do

4: if $w_{i}$ == “<S$k$>” then

5: $t\leftarrow k$ $\triangleright$ new group-tag for new sentence.

6: end if

7: $t_{i}\leftarrow t$ $\triangleright$ group-tag for current sentence.

8: if $w_{i}$ == “</S>” then

9: $t\leftarrow 0$ $\triangleright$ reset group-tag if sentence ends.

10: end if

11: end for

12: return $tags$

13:end function

Algorithm 1 Generate group-tags from a mixed sequence of word tokens and
identifier tokens.

####  IV-A2 Alignment Representation

As Fig. 4 shows, we use group-tag sequences $G^{X^{\prime}}$ and
$G^{Y^{\prime}}$ to represent the extractive sentences $E$ and its one-one
mapping to summary sentences $Y^{\prime}$. Take the case in the figure as an
example, the $G^{X^{\prime}}=\\{1,...,2,2,2,2,2,2,3,3,3,3,3,3,...,6\\}$ and
$G^{Y^{\prime}}=\\{3,3,3,3,5,5,5,5,2,2,2,2\\}$, in which the tokens in
$Y^{\prime}$ corresponding to group-tag $3$ are generated by rewriting the
tokens in $X^{\prime}$ corresponding to the same group-tag $3$.

Given group-tag sequences $G^{X^{\prime}}$ and $G^{Y^{\prime}}$, we could
remove $E$ from Eq 3 and reformulate contextualized rewriting as

$P(Y_{j}|E_{j},E_{<j},Y_{<j},X)=\prod_{k}P(Y^{\prime}_{k}|G^{Y^{\prime}}_{<k},Y^{\prime}_{<k},G^{X^{\prime}},X^{\prime}),$
(7)

where $Y^{\prime}_{k}$ denotes the $k$-th token in $Y^{\prime}$,
$Y^{\prime}_{<k}$ the tokens before $k$, and $G^{Y^{\prime}}_{<k}$ the group-
tags before $k$ that $k$ ranges from the beginning to the end of sentence
$Y_{j}$. Note that the probability on the left of the equation is expressed on
sentences, while that on the right is expressed on tokens. We obtain group-tag
sequence by Algorithm 1, that

$G^{X^{\prime}}=\textsc{GroupTag}(X^{\prime}),$
$G^{Y^{\prime}}_{<k}=\textsc{GroupTag}(Y^{\prime}_{<k}).$

####  IV-A3 Group-tag Representation

In the encoder-decoder framework, we convert $G^{X^{\prime}}$ and
$G^{Y^{\prime}}$ into vector representations through a shared embedding table,
which is randomly initialized and jointly trained with the encoder and
decoder. The vector representations of $G^{X^{\prime}}$ and $G^{Y^{\prime}}$
are used to enrich vector representations of $X^{\prime}$ and $Y^{\prime}$,
respectively. As a result, all the tokens tagged with $k$ in both $X^{\prime}$
and $Y^{\prime}$ have the same vector component, through which content-based
addressing can be done by the attention mechanism [32]. Here, the group tag
serves as a mechanism to constrain the attention from $Y^{\prime}$ to the
corresponding part of $X^{\prime}$ during decoding. Unlike approaches that
modify a seq2seq model by using rules [33, 11], group tag enables the
modification to be flexible and trainable.

###  IV-B Document Encoding

We generate group-tags $G^{X^{\prime}}$ from the input $X^{\prime}$ and apply
the group-tag embeddings upon the document representation to produce the final
representation

$\begin{split}G^{X^{\prime}}&=\textsc{GroupTag}(X^{\prime}),\\\
x&=\textsc{Encoder}(X^{\prime}),\\\
x&=x+\textsc{Emb}_{tag}(G^{X^{\prime}}),\\\ \end{split}$ (8)

where Encoder denotes the encoder module of a seq2seq model, and
$\textsc{Emb}_{tag}$ denotes the embedding table of group-tags.

###  IV-C Summary Decoding

We extend a standard Transformer decoder with group-tag alignments. We
generate group-tag sequence $G^{Y^{\prime}}$ from summary $Y^{\prime}$ and
convert these group-tags into embeddings, adding them to token embeddings and
position embeddings for input representation

$\begin{split}G^{Y^{\prime}}&=\textsc{GroupTag}(Y^{\prime}),\\\
y&=\textsc{Emb}_{token}(Y^{\prime})+\textsc{Emb}_{pos}(Y^{\prime})+\textsc{Emb}_{tag}(G^{Y^{\prime}}),\\\
\end{split}$ (9)

where $\textsc{Emb}_{token}$ denotes the token embedding table,
$\textsc{Emb}_{pos}$ the position embedding table, and $\textsc{Emb}_{tag}$
the group-tag embedding table.

We predict the generation tokens according to encoder output and decoder
history. The rewriting formula in Eq 12 can be implemented by

$P(Y^{\prime}_{k}|G^{Y^{\prime}}_{<k},Y^{\prime}_{<k},G^{X^{\prime}},X^{\prime})=\textsc{Decoder}(y_{<k},x),\\\
$ (10)

where Decoder denotes Transformer decoder, $y$ represents the tagged token
embeddings, and $x$ the encoder outputs. Because of the same vector components
of group-tags in $x$ and $y$, the decoder addresses the $k$-th extracted
sentence in $x$ when generating the $k$-th rewritten sentence.

####  IV-C1 Sentence Selection

As shown in Fig. 4, on the decoder side of self-contained rewriter, we use
identifier tokens to denote the corresponding extractive sentences (e.g.<S$7$>
and <S$1$>). Therefore, the problem of sentence selection is transformed into
a problem of predicting sentence identifier tokens, which can be modeled as
one step in the decoding sequence. Formally, the sentence selection part in Eq
6 can be transformed to

$P(E_{j}|E_{<j},Y_{<j},X)=P(Y^{\prime}_{k}|G^{Y^{\prime}}_{<k},Y^{\prime}_{<k},G^{X^{\prime}},X^{\prime}),$
(11)

where the $k$-th token of the extended sequence $Y^{\prime}$ is the identifier
token of $j$-th selected sentence.

Figure 5: BERT-Rewriter and BART-Rewriter use extractive sentences provided by
external extractor, such as the sentences denoted by the sentence identifiers
<S$2$> and <S$1$> which are displayed as “S2” and “S1” in the encoder input.
The sentence selection does not play a role in the models as it only predicts
incremental identifiers. Figure 6: BART-JointSR jointly models sentence
selection and rewriting, using identifier tokens such as <S$7$> and <S$1$>
(displayed as “S7” and “S1”) in the decoder output to indicate the selected
sentences.

####  IV-C2 Sentence Rewriting

We can see that Eq 7 and Eq 11 have the same form and the $k$ in Eq 11 is
followed by the $k$ in Eq 7. So that we merge them into one decoding sequence,
which makes sentence selection first and then do contextualized rewriting.
Therefore, a general formula for both Eq 3 and Eq 6 is

$\hat{Y^{\prime}}=\arg\max_{Y^{\prime}}\prod_{k=1}^{K}P(Y^{\prime}_{k}|G^{Y^{\prime}}_{<k},Y^{\prime}_{<k},G^{X^{\prime}},X^{\prime}),$
(12)

where $K$ denotes the number of tokens in summary $Y^{\prime}$, and
$X^{\prime}$ is extended from $X$ while $Y^{\prime}$ is extended from $Y$.

The formula is the same for the rewriter with an external extractive summary
and the rewriter with an internal extractor, but the group-tag sequences are
different. For the former, the group-tag sequence $G^{X^{\prime}}$ tells where
the extractive sentences are located in $X^{\prime}$. In contrast, the group-
tag sequence $G^{Y^{\prime}}$ is just naturally ordered and increases per
sentence. For the latter, the group-tag sequence $G^{X^{\prime}}$ is naturally
ordered that the number increases per sentence that the group-tag sequence
$G^{Y^{\prime}}$ indicates the location of the extractive sentences in
$X^{\prime}$.

###  IV-D Training and Inference

We _train_ our contextualized rewriter on a pre-processed dataset labeled with
oracle extractions.

####  IV-D1 Oracle Extractions

To generate oracle extraction, we match each sentence in the human summary to
each document sentence, choosing the document sentence with the best matching
score as the oracle extraction. Specifically, we use the average recall of
ROUGE-1/2/L as the scoring function, which follows Wei et al.[16]. Differing
from existing work [5], which aims to find a set of sentences that maximizes
ROUGE matching with the whole summary, we find the best match for each summary
sentence. As a result, the number of extracted sentences is the same as the
number of sentences in the human summary. This strategy is also adopted by Wei
et al. [16] and Bae et al. [15].

####  IV-D2 Loss Function

We train contextualized rewriter using MLE loss, but with a different weight
assigned to identifier tokens because they serve for sentence selection

$\begin{split}Loss&=\sum_{j=k}^{|Y^{\prime}|}-w_{j}*\log
P(Y^{\prime}_{k}|G^{Y^{\prime}}_{<k},Y^{\prime}_{<k},G^{X^{\prime}},X^{\prime}),\\\
w_{j}&=\gamma\text{ if }Y^{\prime}_{k}\in\text{\it Indentifiers}\text{ else
}1,\end{split}$ (13)

where Identifiers is the set of identifier tokens and $\gamma$ is a hyper-
parameter determined by searching on the development set.

During _inference_ , we generate group tags incrementally with the tokens
predicted. The group tag of the next decoding step is uniquely determined by
the tokens that have been generated.

##  V Instances

###  V-A Instance I: BERT Rewriter

We describe our conference work BERT rewriter [21] as a first instance of the
framework, which follows the architecture design of Fig. 5, using a pretrained
BERT [18] as the document encoder and a randomly-initialized Transformer [30]
decoder as the summary decoder. The rewriter extends the abstractive
summarizer of [5] with group-tag alignments between the encoder and the
decoder.

In order to adapt to the _input_ definition of BERT model, we convert the
document input $X^{\prime}$ before feeding it to BERT model. We replace the
identifier tokens <S$k$> to a general sentence representation token [cls] and
the end-of-sentence token </S> to [sep], so that the BERT encoder can consume.
The randomly-initialized decoder shares the BERT token embedding table. The
identifier tokens <S$k$> in summary are replaced with [sep]. During decoding,
the first [sep] is translated as <S$1$>, the second <S$2$>, and so on.

Because we use a pre-trained model for the encoder and a randomly-initialized
model for the decoder, we _train_ them using different learning-rate and
warmup schedules that

$\begin{split}lr_{\textsc{Enc}}&=0.002\cdot min(step^{-0.5},step\cdot
warmup_{\textsc{Enc}}^{-1.5}),\\\ lr_{\textsc{Dec}}&=0.2\cdot
min(step^{-0.5},step\cdot warmup_{\textsc{Dec}}^{-1.5}).\\\ \end{split}$ (14)

For _inference_ , we constrain the decoding sequence to a minimum length of
50, a maximum length of 200, a length penalty [34] with $\alpha=0.95$, and a
beam size of 5. During beam search, we block the paths on which a repeated
trigram is generated, namely Trigram Blocking [35].

###  V-B Instance II: BART Rewriter

As a second instance, we apply the general framework described in section IV
to a pretrained BART [19], extending it with the rewriting mechanism as Fig. 5
describes.

BART-Rewriter relies on an external extractor to select the salient source
sentences. During training, the extractive sentences are selected by a
matching algorithm that each summary sentence is matched to a source sentence.
The group-tags for the summary follow a natural order for each sentence, while
the group-tags for the source document may not. During inference, we can use
BERT-Ext [21] to select sentences.

###  V-C Instance III: BART-JointSR (BART Joint Selector and Rewriter)

As a third instance, we build a joint model of internal extractor (selector)
and rewriter - BART-JointSR. As Fig. 6 shows, similar to BART-Rewriter, we
apply the general framework to the pre-trained BART, but instead of an
external extractor, we use an integrated extractor.

To train BART joint selector and rewriter, we generate different training
inputs that sentence identifiers are generated using a different strategy. We
use sentence identifiers with natural order for the input document but
sentence identifiers corresponding to the document sentence for the summary.
Therefore, the sentence identifiers in the decoder output indicate the
sentence selections. During inference, we use beam-search to find the best
sentence selection and rewriting combination.

##  VI Experimental Setup

We experiment on the standard benchmark dataset CNN/DailyMail [36], a single-
document summarization dataset comprising 312,085 online news articles with an
average of 766 words per article and human written highlights with an average
of 3.75 sentences per sample. We follow the standard splitting of Herman et
al. [36], containing 287,227 training samples, 13,368 validation samples, and
11,490 testing samples. We use the non-anonymous version and preprocess the
dataset following the BART baseline [19].

We evaluate the quality of summaries using the automatic metric ROUGE [20],
reporting ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L). The ROUGE-1/2
represent the n-gram overlap between generated summary and gold reference, and
ROUGE-L reflects the longest common sub-sequence between generated summary and
gold reference.

Method | R-1 | R-2 | R-L
---|---|---|---
Extractive
LEAD-3 [14] | 40.34 | 17.70 | 36.57
BERTSUMEXT [5] | 43.25 | 20.24 | 39.63
Abstractive
BERTSUMABS [5] | 41.72 | 19.39 | 38.76
BERTSUMEXTABS [5] | 42.13 | 19.60 | 39.18
BART (large) [19] | 44.16 | 21.28 | 40.90
RNN-Ext+Abs+RL [10] | 40.88 | 17.80 | 38.54
BERT-Hybrid [16] | 41.76 | 19.31 | 38.86
BERT-Ext+Abs+RL [15] | 41.90 | 19.08 | 39.64
BERT+Copy/Rewrite+HRL [17] | 42.92 | 19.43 | 39.35
Our Models
BERT-Ext | 41.04 | 19.56 | 37.66
\+ BERT-Rewriter (base) | 43.52 | 20.57 | 40.56
\+ BART-Rewriter (base) | 43.52 | 20.76 | 40.61
\+ BART-Rewriter (large) | 44.26 | 21.23 | 41.34
BART-JointSR (large) | 44.72* | 21.78* | 41.70*
TABLE I: Main results on CNN/DailyMail. The best scores are in bold, and
significantly better scores than BART baseline are marked with * ($p<0.001$,
t-test). Ext and Abs denote extractive and abstractive models, respectively,
and RL means reinforcement learning.

##  VII Automatic Evaluation

###  VII-A Main Results

We evaluate our rewriters on the dataset CNN/DailyMail, reporting ROUGE-1/2/L
(R-1/2/L). As shown in Table I, we compare our model with previous extractive
models, abstractive models, and rewriters.

Compared to previous BERT based extractive baseline BERTSUMEXT and abstractive
baseline BERTSUMEXTABS, our BERT-Rewriter (base) outperforms them for an
average of 0.5 and 1.2 ROUGE points, respectively. Compared with previous BERT
based rewriters, our BERT-Rewriter (base) achieves the best scores,
outperforming BERT+Copy/Rewrite+HRL for 1.0 points on average and especially
1.2 points on ROUGE-L, despite that our rewriter is purely abstractive without
leveraging complex techniques such as copying mechanism and reinforcement
learning. These results suggest the effectiveness of our rewriting method and
the advantage of contextualized rewriting compared to non-contextualized
rewriting.

Compared to the abstractive baseline BART, our BART-Rewriter (large) enhances
the scores by 0.2 ROUGE points on average and especially 0.44 on ROUGE-L. Our
joint model BART-JointSR gives much better ROUGE scores than both BART and
BART-Rewriter (large), outperforming them for an average of 0.6 and 0.4
points, respectively. The results demonstrate that our rewriting method works
with large pre-trained model and benefits from joint modeling.

The architecture and size of pre-trained models are important factors
influencing the performance of our rewriters. As we can see, our BART-Rewriter
(base) has similar ROUGE scores as BERT-Rewriter (base). Although the BART-
Rewriter is fine-tuned on the seq2seq pre-trained BART and the BERT-Rewriter
is fine-tuned on a pre-trained BERT encoder and a randomly initialized
decoder, the BART-Rewriter does not outperform the BERT-Rewriter because it
has about $14\%$ fewer parameters. Our BART-Rewriter (large) gives much better
scores than BART-Rewriter (base), improving the scores by 0.74, 0.47, and 0.73
on ROUGE-1/2/L, respectively. The results suggest that contextualized
rewriting method can work with various pre-trained models, taking advantage of
a larger model and further enhancing the model performance.

Method | R-1 | R-2 | R-L | #W
---|---|---|---|---
Oracle | 46.77 | 26.78 | 43.32 | 112
\+ BERT-Rewriter | 52.57 (+5.80) | 29.71 (+2.93) | 49.69 (+6.37) | 63
\+ BART-Rewriter | 55.01 (+8.24) | 32.07 (+5.29) | 52.08 (+8.76) | 64
LEAD3 | 40.34 | 17.70 | 36.57 | 85
\+ BERT-Rewriter | 41.09 (+0.75) | 18.19 (+0.49) | 38.06 (+1.49) | 55
\+ BART-Rewriter | 41.29 (+0.95) | 18.49 (+0.79) | 38.22 (+1.65) | 53
BERTSUMEXT | 42.50 | 19.88 | 38.91 | 80
\+ BERT-Rewriter | 43.31 (+0.81) | 20.44 (+0.56) | 40.33 (+1.42) | 54
\+ BART-Rewriter | 43.35 (+0.85) | 20.70 (+0.82) | 40.55 (+1.64) | 50
BERT-Ext | 41.04 | 19.56 | 37.66 | 105
\+ BERT-Rewriter | 43.52 (+2.48) | 20.57 (+1.01) | 40.56 (+2.90) | 66
\+ BART-Rewriter | 44.26 (+3.22) | 21.23 (+1.67) | 41.34 (+3.68) | 66
TABLE II: Contextualized rewriters work with different extractors, enhancing
their performance. The column #W represents the average number of words in
summary.

###  VII-B Universality of Rewriter with External Extractive Summary

Contextualized rewriters learn to compress, paraphrase, and abstract the
extractive sentences into more concise and readable expressions. Although the
rewriters are trained using oracle extractions, the learned rewriters are not
limited to these oracle extractive sentences.

We evaluate the rewriters with various external extractors, including LEAD-3,
BERTSUMEXT, BERT-Ext, and Oracle. As shown in Table II, the rewriters enhance
the quality of summaries generated by all four extractors. In particular,
working with the basic extractor LEAD-3, BERT-Rewriter improves the average
ROUGE score by 0.91, while BART-Rewriter improves the average score by 1.13.
Even working with the best extractor BERTSUMEXT, BERT-Rewriter and BART-
Rewriter enhance the average ROUGE score by 0.93 and 1.10, respectively. The
improvements on ROUGE-L are most significant, which are more than 1.4 points
for all extractors, suggesting a strong enhancement in fluency.

It is worth noting that for BERTSUMEXT, we conduct decoding without Trigram
Blocking [35], which blocks beam search paths containing repeated trigrams. As
a result, the extractive summaries contain more redundant information.
However, when we apply our rewriters to them, the redundancy can be reduced
and achieve higher scores.

###  VII-C Advantage of Joint Internal Extractor and Rewriter

Our joint rewriter BART-JointSR (large) gives the best ROUGE scores,
outperforming BART-Rewriter for 0.46, 0.55, and 0.36 on ROUGE-1/2/L,
respectively. We hypothesize that the improvement comes from two advantages of
the joint model compared to the pipeline model. First, BART-JointSR models
sentence selection in an auto-regressive manner, therefore, the order of
selected sentences is optimized, and the redundancy between sentences is
reduced. Second, BART-JointSR jointly models sentence selection and rewriting,
so that not only rewriting depends on selection history but vice versa. As a
result, the sentence selection and rewriting are better matched with each
other.

We conduct various experiments to verify our hypothesis. As shown in Table
III, we first study the impact of the order of selected sentences. We alter
BART-Rewriter model to enable the decoder to decide the sentence order before
rewriting, naming BART-Rewriter with reorder. Reordering extractive sentences
can improve the ROUGES scores by about 0.1 on average, which is although small
but confirms our first hypothesis about sentence order.

Method | R-1 | R-2 | R-L | #W
---|---|---|---|---
BERT-Ext | 41.04 | 19.56 | 37.66 | 105
\+ BART-Rewriter | 44.26 | 21.23 | 41.34 | 66
\+ BART-Rewriter with reorder | 44.38 | 21.31 | 41.43 | 61
BART-JointSR | 44.72 | 21.78 | 41.70 | 63
\- extractive summaries | 39.45 | 18.37 | 35.75 | 110
\- dedup extractive summaries | 42.43 | 20.06 | 38.64 | 93
BART-JointSR with two-stage | 43.57 | 20.78 | 40.57 | 57
\- extractive summaries | 37.12 | 16.81 | 33.24 | 131
\- dedup extractive summaries | 42.06 | 19.62 | 38.04 | 88
TABLE III: Joint modeling of sentence selection and rewriting further improves
the performance. The performance will be much lower if we separate selection
and rewriting, as the method BART-JointSR with two-stage shows.

We further compare the independent extractor BERT-Ext and the inner auto-
regressive extractor in BART-JointSR by concatenating the selected sentences.
As the item “extractive summaries” under BART-JointSR shows, the ROUGE scores
are much lower than that of BERT-Ext, which seems contradictive with our first
hypothesis about reducing redundancy. However, when we remove the repetitive
extractive sentences, we obtain much high scores, as the item “dedup
extractive summaries” under BART-JointSR illustrates. Given our rewriter’s
firm information compression and redundancy reduction abilities, these
duplicate extractive sentences are rewritten into different summary sentences
without redundant information. It suggests that combining an auto-regressive
extractor and a contextualized rewriter may be the key to its high
performance.

Figure 7: Distribution of extractive sentences show that the sentences
selected by joint rewriter is closer to oracle extractions than BERT-Ext.
Figure 8: Distribution of duplicate extractive sentences. Joint rewriter
generates duplicate extractions and the distribution is close to oracle
extractions. BERT-Ext do not generate duplicate extractions, so that we use 0
to denote it.

To clarify the impact of the auto-regressive extractor, we have a closer look
at the distribution of extractive sentences over the sentence position in
input document. As Fig. 7 shows, the sentences extracted by BART-JointSR have
better distribution than that by BERT-Ext, which is much closer to the oracle
extractions. We further observe that the oracle extractions also contain
duplicate sentences, and we compare the distribution of duplicate extractive
sentences in Fig. 8. We can see that the distribution of duplicate sentences
extracted by BART-JointSR is close to that of oracle extractions. In contrast,
BERT-Ext only returns identical extractions because it independently makes
classification decisions on each sentence.

To verify our second hypothesis about the advantage of joint modeling, we
evaluate an alternative of BART-JointSR, naming BART-JointSR with two-stage,
where sentence selections and sentence rewriting separated in the decoder.
Specifically, for a BART-JointSR decoding sequence “<S$3$> … <S$5$> … <S$2$>
…”, we put all sentence selection steps together at the beginning of the
decoding sequence as “<S$3$> <S$5$> <S$2$> </S> <S$3$> … <S$5$> … <S$2$> …”.
Therefore, sentence selection does not depend on rewriting, but rewriting
still depends on sentence selection. As shown in Table III, BART-JointSR is
much better than the two-stage version, outperforming it for more than 1 ROUGE
point on R-1/2/L. The results suggest that the extractor can do a better job
when it is conditioned on both previously selected and rewritten sentences,
which strongly supports our joint modeling design.

Figure 9: Models have different abilities to generate non-redundant summaries. The more redundant information is generated by the model, the bigger performance drop will happen without trigram blocking. Method | Info. | Conc. | Read. | Faith.
---|---|---|---|---
BERTSUMEXT | 4.01 | 3.44 | 3.41 | 5.00
BERTSUMEXTABS | 3.87 | 3.73 | 4.06 | 4.80
BERT-Rewriter | 4.12 | 3.69 | 4.01 | 4.91
BART | 4.19 | 3.42 | 4.27 | 4.89
BART-Rewriter | 4.31 | 3.62 | 4.21 | 4.93
BART-JointSR | 4.32 | 4.15 | 4.59 | 4.97
TABLE IV: Human evaluation on informativeness, conciseness, readability, and
faithfulness.

##  VIII Human Evaluation

The contextualized rewriter can improve extractive summaries in various ways.
First, it can recall critical information from the document contexts, thereby
increasing its informativeness. Second, it can compress unimportant/redundant
information from the summaries, enhancing its conciseness. Last, it models
summary discourse, therefore enhancing the readability. In addition, since the
rewritten summaries are paraphrases of the extractive summaries, they tend to
have fewer hallucinations than summaries generated from scratch using a pure
abstractive model. We confirm these hypotheses by conducting a human
evaluation. In particular, we randomly select 30 samples from the test set of
CNN/DailyMail, scoring informativeness, conciseness, readability, and
faithfulness. We assess the qualities with a number from 1(worst) to 5(best)
by three independent annotators and report the average score for each quality.

The results are shown in Table IV. Compared to the extractive baseline
BERTSUMEXT, our BERT-Rewriter improves the informativeness, conciseness, and
readability by a significant margin while maintaining a high faithfulness. The
improvement in informativeness mainly comes from document context, while the
improvement in conciseness and readability is mainly contributed by reduced
redundancy and improved coherence. Compared with the abstractive baseline
BERTSUMEXTABS and BART, our rewriter BERT-Rewriter and BART-Rewriter improve
faithfulness and informativeness while keeping the conciseness and readability
close. Our joint rewriter BART-JointSR gets much higher scores than BART-
Rewriter on conciseness and readability, suggesting the advantage of joint
modeling of sentence selection and rewriting, which enhances the coherence
between sentences.

##  IX Analysis

To better understand where the performance improvement comes from, we further
conduct quantitative studies on the ability of our rewriters to reduce
redundancy, avoid irrelevance, and enhance coherence.

###  IX-A Redundancy

Redundancy has been a significant problem for both extractive and abstractive
summarization [37, 38]. Previous work [35, 15, 39] adopts a simple strategy
during beam-search to filter out paths with duplicated n-grams. In particular,
3-gram is used by most existing methods, and the strategy is known as Trigram
Blocking [35].

As Fig. 9 shows, we compare two strategies for each model: one uses trigram-
blocking during beam-search, and another does not. All the models except BART-
JointSR give lower ROUGE scores when the trigram-blocking post-process is
removed. The extractive model BERTSUMEXT depends on trigram-blocking the most;
the ROUGE scores drop by 0.59 on average without it. The rewriting model BERT-
Rewriter decreases the gap to 0.22 on average, suggesting that our rewriter
can significantly reduce the redundant information in extractive summaries.
Despite that trigram-blocking influences BART and BART-Rewriter for about 0.1
ROUGE point, our joint rewriter BART-JointSR gives the same scores, indicating
that the joint modeling of sentence selection and rewriting can fully reduce
the redundant information.

Figure 10: Proportion of rewritten, compressed, and unchanged sentences after
rewriting.

###  IX-B Compression

Contextualized rewriters can remove unimportant phrases from extractive
summaries. For instance, an extractive summary “they returned to find
hargreaves and the girl, who has not been named, lying on top of each other.”
is compressed into “they returned to find hargreaves and the girl lying on top
of each other.” According to 20 samples of BERT/BART-Rewriter and BART-JointSR
generations, all the compressions are on the phrase-level instead of on single
words, and most removed phrases are likely unimportant since only $11\%$ of
them appear in gold summaries.

Our contextualized rewriters can largely compress the summaries. As the column
#W in Table II shows, the oracle summaries are compressed to almost a half by
our rewriters, while other extractive summaries are compressed more than one-
third. As Fig. 10 shows, most extractive sentences are rewritten or
compressed. We obtain these numbers on the test dataset by adopting the edit-
sequence-generation algorithm [40] to generate a sequence of word editing
actions that maps an extracted summary sentence to the rewritten one. We
categorize a sentence as “Rewritten” if the sequence contains an action of
adding or modifying, “Compressed” if it contains an action of deleting, and
“Unchanged” otherwise. Take BART-JointSR as an example. Only $15\%$ of
extractive sentences remain unchanged during rewriting.

Source Document:  British world dressage champion Charlotte Dujardin won the
Grand Prix at the World Cup in Las Vegas . 1 The 29-year-old , and her horse
Valegro , who won the world title in Lyon last year , recorded a score of
85.414 percent to finish clear of Dutchman Edward Gal with American Steffen
Peters in third . 2 Her score was short of the 87.129 she recorded in breaking
her own world record last year , but there was no wiping the smile off
Dujardin ’s face … The last few days , he was actually feeling not quite
himself and I was a bit worried . ‘ But he was feeling much better and I had a
really great ride . ’ Rewritten Summary:  Charlotte Dujardin won the Grand
Prix at the World Cup in Las Vegas . 1 The 29-year-old , and her horse Valegro
, won the world title in Lyon last year . 2 Swap Group Tags:  Charlotte
Dujardin won the world title in Lyon last year . 1 The 29-year-old won the
Grand Prix in Las Vegas . 2

Figure 11: Example of the ability to maintain coherence.

###  IX-C Coherence

Contextualized rewriters can improve extractive summaries on cohesion and
coherence. It is partially illustrated by the higher ROUGE-L scores in the
automatic evaluation and higher Readability scores in the human evaluation.
Here, we use a case to demonstrate how our rewriter maintains the quality.

The rewriting process of our rewriters is controlled by the extractive input.
By altering the extractive input and comparing the rewriting outputs, we can
qualitatively study the model’s ability. As Fig. 11 shows, we use different
sentence orders of extractive summary to test BART-Rewriter. We can see that
the athlete’s name is mentioned in the first summary sentence, while a
nominator is used in the second sentence. When we change the order of the two
extractive sentences, as the “Swap Group Tags” section shows, the content of
the two summary sentences interchange their positions. However, the athlete’s
name is still presented in the first sentence, and a nominator is used in the
second sentence. These observations demonstrate that our rewriter maintains
the cross-sentence anaphora correctly.

##  X Conclusion

We investigated contextualized rewriting for automatic summarization, building
both rewriters with an external extractor and an internal extractor, by
proposing a general framework naming seq2seq with group-tag alignments and
implement three rewriter instances. Results on standard benchmark show that
contextual information is highly beneficial for summary rewriting,
particularly when sentence selection and rewriting are jointly modeled. Our
contextualized rewriters outperform existing non-contextualized rewriters by a
significant margin, achieving strong ROUGE improvements upon multiple
extractors for the first time. Our framework of seq2seq with group-tag
alignments is general and can potentially be applied to other NLG tasks.

## References

  * [1] M. Maybury, _Advances in automatic text summarization_. MIT press, 1999.
  * [2] O. Tas and F. Kiyani, “A survey automatic text summarization,” _PressAcademia Procedia_ , vol. 5, no. 1, pp. 205–213, 2007.
  * [3] R. Nallapati, F. Zhai, and B. Zhou, “Summarunner: A recurrent neural network based sequence model for extractive summarization of documents,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , S. P. Singh and S. Markovitch, Eds. AAAI Press, 2017, pp. 3075–3081.
  * [4] S. Narayan, S. B. Cohen, and M. Lapata, “Ranking sentences for extractive summarization with reinforcement learning,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 1747–1759.
  * [5] Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3730–3740.
  * [6] A. M. Rush, S. Chopra, and J. Weston, “A neural attention model for abstractive sentence summarization,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 379–389.
  * [7] R. Nallapati, B. Zhou, C. dos Santos, Ç. Gulçehre, and B. Xiang, “Abstractive text summarization using sequence-to-sequence RNNs and beyond,” in _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 280–290.
  * [8] S. Chopra, M. Auli, and A. M. Rush, “Abstractive sentence summarization with attentive recurrent neural networks,” in _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. San Diego, California: Association for Computational Linguistics, Jun. 2016, pp. 93–98.
  * [9] G. Durrett, T. Berg-Kirkpatrick, and D. Klein, “Learning-based single-document summarization with compression and anaphoricity constraints,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1998–2008.
  * [10] Y.-C. Chen and M. Bansal, “Fast abstractive summarization with reinforce-selected sentence rewriting,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 675–686.
  * [11] S. Gehrmann, Y. Deng, and A. Rush, “Bottom-up abstractive summarization,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 4098–4109.
  * [12] B. Dorr, D. Zajic, and R. Schwartz, “Hedge trimmer: A parse-and-trim approach to headline generation,” in _Proceedings of the HLT-NAACL 03 Text Summarization Workshop_ , 2003, pp. 1–8.
  * [13] J. Cheng and M. Lapata, “Neural summarization by extracting sentences and words,” in _54th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, 2016, pp. 484–494.
  * [14] A. See, P. J. Liu, and C. D. Manning, “Get to the point: Summarization with pointer-generator networks,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1073–1083.
  * [15] S. Bae, T. Kim, J. Kim, and S.-g. Lee, “Summary level training of sentence rewriting for abstractive summarization,” in _Proceedings of the 2nd Workshop on New Frontiers in Summarization_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 10–20.
  * [16] R. Wei, H. Huang, and Y. Gao, “Sharing pre-trained BERT decoder for a hybrid summarization,” in _Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Kunming, China, October 18-20, 2019, Proceedings_ , ser. Lecture Notes in Computer Science, M. Sun, X. Huang, H. Ji, Z. Liu, and Y. Liu, Eds., vol. 11856. Springer, 2019, pp. 169–180.
  * [17] L. Xiao, L. Wang, H. He, and Y. Jin, “Copy or Rewrite: Hybrid Summarization with Hierarchical Reinforcement Learning,” _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2020.
  * [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [19] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 7871–7880.
  * [20] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [21] G. Bao and Y. Zhang, “Contextualized rewriting for text summarization,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 14, 2021, pp. 12 544–12 553.
  * [22] L. Dong, J. Mallinson, S. Reddy, and M. Lapata, “Learning to paraphrase for question answering,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 875–886. [Online]. Available: https://aclanthology.org/D17-1091
  * [23] A. Elgohary, D. Peskov, and J. Boyd-Graber, “Can you unpack that? learning to rewrite questions-in-context,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 5918–5924. [Online]. Available: https://aclanthology.org/D19-1605
  * [24] H. Su, X. Shen, R. Zhang, F. Sun, P. Hu, C. Niu, and J. Zhou, “Improving multi-turn dialogue modelling with utterance ReWriter,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 22–31. [Online]. Available: https://aclanthology.org/P19-1003
  * [25] R. Anantha, S. Vakulenko, Z. Tu, S. Longpre, S. Pulman, and S. Chappidi, “Open-domain question answering goes conversational via question rewriting,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 520–534.
  * [26] S. Ren, Y. Wu, S. Liu, M. Zhou, and S. Ma, “A retrieve-and-rewrite initialization method for unsupervised machine translation,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 3498–3504.
  * [27] G. Durrett, T. Berg-Kirkpatrick, and D. Klein, “Learning-based single-document summarization with compression and anaphoricity constraints,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1998–2008.
  * [28] B. Dorr, D. Zajic, and R. Schwartz, “Hedge trimmer: A parse-and-trim approach to headline generation,” in _Proceedings of the HLT-NAACL 03 Text Summarization Workshop_ , 2003, pp. 1–8.
  * [29] Z. Cao, W. Li, S. Li, and F. Wei, “Retrieve, rerank and rewrite: Soft template based neural summarization,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 152–161. [Online]. Available: https://aclanthology.org/P18-1015
  * [30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [31] W.-T. Hsu, C.-K. Lin, M.-Y. Lee, K. Min, J. Tang, and M. Sun, “A unified model for extractive and abstractive summarization using inconsistency loss,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 132–141.
  * [32] S. Garg, S. Peitz, U. Nallasamy, and M. Paulik, “Jointly learning to align and translate with transformer models,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 4453–4462.
  * [33] W.-T. Hsu, C.-K. Lin, M.-Y. Lee, K. Min, J. Tang, and M. Sun, “A unified model for extractive and abstractive summarization using inconsistency loss,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 132–141.
  * [34] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Ł. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,” in _https://arxiv.org/abs/1609.08144_ , 2016, pp. 1–23.
  * [35] R. Paulus, C. Xiong, and R. Socher, “A deep reinforced model for abstractive summarization,” _CoRR_ , vol. abs/1705.04304, 2017.
  * [36] K. M. Hermann, T. Kociský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines to read and comprehend,” in _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds., 2015, pp. 1693–1701.
  * [37] M. Zhong, P. Liu, Y. Chen, D. Wang, X. Qiu, and X.-J. Huang, “Extractive summarization as text matching,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 6197–6208.
  * [38] Q. Zhou, N. Yang, F. Wei, S. Huang, M. Zhou, and T. Zhao, “A joint sentence scoring and selection framework for neural extractive document summarization,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 671–681, 2020.
  * [39] Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3730–3740.
  * [40] F. Zhang and D. Litman, “Sentence-level rewriting detection,” in _Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications_. Baltimore, Maryland: Association for Computational Linguistics, Jun. 2014, pp. 149–154.