# Instructions for ACL-IJCNLP 2021 Proceedings

Alexander R. Fabbri$\dagger$ Xiaojian Wu$\ddagger$ Srini Iyer$\ddagger$ Mona
Diab$\ddagger$
$\dagger$ Yale University $\ddagger$ Facebook AI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
# Multi-Perspective Abstractive Answer Summarization

Alexander R. Fabbri$\dagger$ Xiaojian Wu$\ddagger$ Srini Iyer$\ddagger$ Mona
Diab$\ddagger$
$\dagger$ Yale University $\ddagger$ Facebook AI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Community Question Answering (CQA) forums such as Stack Overflow and Yahoo!
Answers contain a rich resource of answers to a wide range of questions. Each
question thread can receive a large number of answers with different
perspectives. The goal of multi-perspective answer summarization is to produce
a summary that includes all perspectives of the answer. A major obstacle for
multi-perspective, abstractive answer summarization is the absence of a
dataset to provide supervision for producing such summaries. This work
introduces a novel dataset creation method to automatically create multi-
perspective, bullet-point abstractive summaries from an existing CQA forum.
Supervision provided by this dataset trains models to inherently produce
multi-perspective summaries. Additionally, to train models to output more
diverse, faithful answer summaries while retaining multiple perspectives, we
propose a multi-reward optimization technique coupled with a sentence-
relevance prediction multi-task loss. Our methods demonstrate improved
coverage of perspectives and faithfulness as measured by automatic and human
evaluations compared to a strong baseline.

##  1 Introduction

Question: i found a puppy that is less then six weeks old an no mother around
what should i feed it?
---
Context: it a pit puppy i think
Answer 1: Go to a vet and get some and a small feeding bottle.
Answer 2: get a baby bottle warm milk best thing is to call a pet shop
Answer 3: it needs a certain type of milk,  dont feed it cows milk
Answer 4: call a vet and ask them. if you cannot do that then give them alot
of water and a little balony a day, than go into dog food…
Summary Bullet Points:
1. call the vet and tell them how old you think it is and what should you feed it…
2. the first thing you want to do if you plan to keep it is go to petsamrt or pet co and ask anyone that specializes on dogs and get the pup a baby bottle and feed it milk but  not cow milk try powder milk with water.
3. Try and find something soft to eat (as in a soft  dog food).
4. if it is not yet walking, then get a bottle
Table 1: An example bullet-point summary from our answer summarization
dataset, illustrating the multiple viewpoints present in the summaries created
through our pipeline, and a subset of the 14 user answers to which the target
summary can be aligned.

In a world of information overload and the ubiquity of discussion forums,
there is a need for text summarization as a means of distilling relevant
information into a concise form. The problem is even more pertinent for
question answering within the context of Comunity Question Answering (CQA)
forums, where a person poses a question and can get an abundance of answers to
sift through. Ideally, an answer summary should cover the multiple
perspectives found in the answers, where available. For example, in Table 1, a
person poses a question about finding a puppy and also provides context on the
type of dog. We present a sample of the 14 answers to that question on Yahoo!
Answers and an automatically-created summary consisting of bullet points
covering the answers’ main perspectives. We introduce a novel pipeline to
build such a multi-perspective, bullet-point summarization dataset and
introduce models to generate faithful multi-perspective summaries.

To date, most CQA forums have a notion of a ’best answer,’ which is either
manually chosen by the person who asked the question or by a moderator or
obtained via community ratings. Work in this field typically makes use of this
best answer as a proxy for summaries Tomasoni and Huang (2010); Chan et al.
(2012); Pande et al. (2013); Wang et al. (2014); Song et al. (2017). However,
the best answer only presents one person’s perspective and rarely captures the
variety of perspectives discussed in the thread. Datasets such as WikiHowQA
(Deng et al., 2020), which consists of a question, a long answer, and an
answer summary, focus on answer selection and the summarization of a single
answer. While CQASumm Chowdhury and Chakraborty (2019) uses the chosen best
answer as the answer summary, they also apply heuristics to ensure token
overlap with the remaining answers. However, we found that the heuristics
applied generally promotes only long answers instead of multiple perspectives.
To validate our hypothesis, we examine a set of 30 summaries from CQASumm and
found that only 37% of the examples contained multi-perspective answers.

Although multi-perspective answer summarization is an important research topic
with practical applications, there are no relevant datasets or techniques to
address it effectively. This paper tries to close this gap by developing a
dataset together with several modeling techniques for multi-perspective answer
summarization. To generate a multi-perspective summarization dataset, we
devise a pipeline to produce bullet point answer summaries. First, we select
and cluster salient answer sentences. Then, we use the cluster centroids as
our summary bullet points and remove them from the input to promote a more
challenging, more abstractive task. We further filter the data to improve our
dataset’s quality and promote desirable summary characteristics such as
compression. We find that a strong baseline model trained on our data
inherently outputs multi-perspective summaries. We focus our modeling efforts
on generating content implied by the input text and being faithful to the
underlying answers by covering multiple perspectives. To this end, we use a
reinforcement learning (RL) framework with new rewards and a sentence-
relevance multi-task loss, whereby the model learns to predict relevant
sentences for the current decoding step to more closely align the source and
generated output. Our models improve the coverage and faithfulness of
generated summaries when compared to a state-of-the-art abstractive baseline.

The main contribution of this paper is to develop, for the first time, a
method for multi-perspective abstractive answer summarization. To achieve
this, 1) We introduce a dataset generation pipeline for answer summarization
that goes beyond the best-answer summary, to create multi-perspective, bullet-
point summaries for training and evaluation 2) We introduce and evaluate RL
reward functions on answer summarization, including entailment as a measure of
faithfulness and volume of semantic space as a way to increase coverage of
multiple answer perspectives 3) We introduce a sentence-relevance prediction
loss to increase the faithfulness and interpretability of the generated answer
summaries. We will make our code available for reproducing our dataset
pipeline and model results.

##  2 Related Work

Extractive Answer Summarization: Much work has focused on the extractive
summarization setting as an answer-ranking problem (Chan et al., 2012; Pande
et al., 2013; Wang et al., 2014). Liu et al. (2008) find that only 48% of the
best answers on Yahoo! Answers are unique best answers; there are multiple
correct ways to answer a question. Other recent work has focused on sentence
extraction using metadata (Tomasoni and Huang, 2010) or sparse-coding
frameworks Song et al. (2017). Our focus is on representing multiple
perspectives in an abstractive summarization framework.

Abstractive Answer Summarization: Another line of work has attempted
abstractive answer summarization by treating the tagged best answer as the
gold summary of all the other answers (Chowdhury and Chakraborty, 2019;
Chowdhury et al., 2020). Chowdhury and Chakraborty (2019) present CQASumm, a
dataset of about 100k examples consisting of the best answer as the gold
summary, which, however, often only contains one perspective.

Multi-document Summarization: Answer summarization can be viewed as a query-
based multi-document summarization (MDS) problem. Several large-scale MDS
datasets have been introduced in the news domain (Fabbri et al., 2019; Gu et
al., 2020; Gholipour Ghalandari et al., 2020), for creating Wikipedia lead-
paragraphs (Liu et al., 2018) and for long-form question answering (Fan et
al., 2019). However, news-based MDS datasets are not query-based, Wikipedia
summarization is topic-based and less granular than our setting, and the ELI5
dataset (Fan et al., 2019) summarizes web documents rather than direct query
answers.

RL and Multi-task Learning for Summarization: Paulus et al. (2018) first apply
the REINFORCE algorithm Williams (1992) in the context of summarization. RL
has since been applied for both extractive Narayan et al. (2018b); Dong et al.
(2018), abstractive Pasunuru and Bansal (2018); Li et al. (2018); Huang et al.
(2020); Laban et al. (2020) and hybrid approaches Chen and Bansal (2018). Böhm
et al. (2019) stress the role of using rewards that correlate well with human
judgments on downstream performance. Our paper focuses on the selection of
rewards applicable for promoting faithful and diverse, abstractive answer
summaries. Previous work on entailment as an RL reward has focused on
document-level entailment in the news domain (Li et al., 2018; Pasunuru and
Bansal, 2018). In this work, we show the effect of the choice of entailment
model on downstream faithfulness prediction and the importance of using
sentence-level entailment. Recent work in multi-task learning with
summarization consists of sharing parameters between an abstractive generator
and auxiliary tasks such as entailment and question generation (Guo et al.,
2018) and text classification and syntax-labeling tasks (Lu et al., 2019).

##  3 Dataset Creation

Previous CQA work lacks multi-perspective supervision. To address this
research gap, we develop a system to create summaries covering multiple
perspectives of answers to a given question.

Figure 1: An illustration of our dataset pipeline. Given a question and
answers, we cluster relevant sentences and remove the cluster centroid of non-
singleton clusters from the input to use as bullet point summaries, filtering
the example if it does not meet quality-control criteria.

###  3.1 Overview of Data Generation Pipeline

The input to our pipeline is a question and its answers. We use question
threads from the Yahoo! Answers L6
corpus111https://webscope.sandbox.yahoo.com/catalog.php?datatype=l&did=11. Our
pipeline operates on the sentence level of these answers versus the answer
level, as we believe that this granularity allows us to capture additional
perspectives. Our dataset pipeline consists of the following components: 1) a
relevance model to remove irrelevant inputs, 2) a clustering model to cluster
similar content, and 3) input and summary creation from centroids.

Relevance model: We first aim to determine whether a given sentence is
relevant to answering a question and, therefore, to be considered as a
potential summary sentence. We use the ANTIQUE (Hashemi et al., 2020)
relevance data for training a query-sentence relevance model. The data
consists of Yahoo! answers and relevance labels on a scale from 1-4, with 1-2
not relevant and 3-4 relevant. We use a RoBERTa-based Liu et al. (2019) model
fine-tuned on answer selection on the TREC-QA dataset (Wang et al., 2007) as a
binary relevant/non-relevant classifier and further fine-tune it using the
Tanda (Garg et al., 2020) method. We experimented with training the relevance
classifier using Yahoo! Answers, treating the best answer as relevant and the
other answers as not relevant, and analogously on the sentence level, although
without improvements. The performance was measured using mean reciprocal rank
on a held-out relevance set.

As input to the clustering stage, we remove sentences that our relevance model
labels as irrelevant (our model tends to over-predict relevant sentences, as
many answers contain relevant sentences, thus removing only 16% of sentences).
Improving this relevance classifier to better filter irrelevant answer
sentences is a very interesting research direction, although we leave this for
future work.

Clustering: Most methods for short-text clustering (Hadifar et al., 2019; Xu
et al., 2017) require a known value of k, the number of clusters, which is
dynamic from question to question in our setting. In this work, we use the
sentence-transformers library (Reimers and Gurevych, 2019a) to perform
clustering. Specifically, we start with a RoBERTa-based model fine-tuned for
sentence embeddings on an entailment dataset, which is further fine-tuned for
semantic similarity. Clustering parameters were chosen based on a
StackOverflow clustering dataset containing labeled clusters commonly used in
short-text clustering. We used Agglomerative clustering with average linkage,
cosine distance, and a maximum distance of .65.

To create the final summaries, we locate the centroid of clusters with at
least two sentences and use these centroids as bullet-point summaries.
Further, we remove the centroid sentences from the sentence-tokenized input
answers to create a challenging abstractive summarization dataset analogous to
the XSum dataset (Narayan et al., 2018a). Since each cluster contains at least
two sentences, we assume that given a perfect clustering algorithm, a related
sentence can help generate the removed centroid sentence. While removing
sentences naturally decreases coherence, we believe that this introduces a
tolerable level of noise, considering the existing presence of noise through
ungrammatical and occasionally incoherent answers. To further account for
imperfections in the pipeline, we apply additional filtering techniques,
described below.

###  3.2 Postprocessing and Quantitative Analysis

We obtained question threads from Yahoo! Answers and applied heuristics
detailed in Tomasoni and Huang (2010) to find threads suitable for
summarization. Threads were removed if 1) there were less than five answers,
2) the longest answer was over 400 words, 3) the sum of the length of all
answers was outside of (100, 1000) words, and 4) the average length of answers
was outside of the (50, 300) words interval. This filtering left us with about
350k of the approximately 4.4 million threads and included both factoid and
non-factoid questions. Questions include the subject of the post as well as
the content of the post when available.

Example Filtering: We remove examples from the dataset based on desired
summarization characteristics. A desirable trait in summarization datasets is
compression, i.e., the ratio of the input size to the summary size (Grusky et
al., 2018). We remove examples with a compression ratio under 4, examples for
which the input length exceeded 1,100 tokens and for which the summary length
exceeded 250 tokens, leaving us with 284,295 examples. We further remove
target summaries labeled as contradictions from a RoBERTa-based entailment
model following Matsumaru et al. (2020). Furthermore, we remove examples with
more than 10 “+” or ”=” signs (math queries), those with very long ($>$50
characters) tokens, and those with a link in the target or more than one link
in the source. Finally, we filter to ensure that we have examples where the
named entities found in the target are also found in the source document.

Quality Analysis: The filtering process yielded 96,701 examples, which we
split into 88,512/4,032/4,157 training, validation, and testing examples. We
annotated a subset of 400 summaries created by our pipeline to conduct quality
checks. For each summary, the annotator reads the question, and if the answer
coverage of the summary was determined as reasonable, the summary was marked
as 1, otherwise 0. 370 of the 400 summaries were labeled as 1, showing that
the pipeline creates largely relevant content. Additionally, on examining 30
summaries, we found that 80% contained multiple perspectives versus the 37% we
found in CQASumm, showing the benefit of our dataset pipeline in encoding
multiple viewpoints. To further analyze the types of questions present in our
dataset, we trained a factoid/non-factoid question classifier using SQuAD
Rajpurkar et al. (2016) data as factoid examples and non-factoid Yahoo!
Questions dataset222https://ciir.cs.umass.edu/downloads/nfL6/ as non-factoid
examples. 8% of threads were labeled as factoid questions; the filtering steps
based on answer size likely filter out examples with short, factoid answers.

###  3.3 Relation to Existing Datasets

CQASumm is the closest dataset with our desired answer summarization
qualities, although it simply promotes answers as summaries rather than truly
summarizing answers. As discussed above, this dataset lacks our desired multi-
perspective summaries. A similar approach to dataset creation was taken by
Shapira and Levy (2020) for review summarization by clustering reviews using
pivot clustering, adding reviews to a cluster based on lexical overlap until a
max length and min number of review requirements are met. There are
significant differences to our approach in terms of granularity (reviews vs.
sentence clustering), type of clustering (lexical vs. embedding-based), as
well as the ultimate use of these clusters (they train a cluster summarizer
while we combine cluster centroids for creating an abstractive bullet point
combined with other cluster centroids). We present a comparison of dataset
statistics between our dataset, which we call AnswerSumm, and the standard
XSum and CNN-Daily Mail Nallapati et al. (2016) summarization datasets in
Table 2. In general, we find our dataset to be more abstractive than CNN-
DailyMail and less so than XSum. We also note that our generated dataset is
similar to CNN-DailyMail in that it consists of bullet points. While this may
create summaries with less coherence, or potentially contradictory answers, we
focus on producing multi-perspective summaries in this work and leave improved
summary coherence for future work.

Dataset | % Novel unigrams | Oracle Extractive | Length
---|---|---|---
AnswerSumm (ours) | 32.2 | 40.02/11.16/33.70 | 67
XSUM | 35.8 | 29.79/8.81/22.65 | 23
CNN | 16.8 | 50.38/28.55/46.58 | 46
DailyMail | 17.0 | 55.23/30.55/51.24 | 55
|  |  |

Table 2: Comparison between AnwerSumm and the XSum Narayan et al. (2018a) and
CNN-DailyMail Nallapati et al. (2016) datasets. Oracle Extractive and Length
refer to the maximum ROUGE Lin (2004) score achievable by an extractive model,
and the average length of the summaries, respectively.

##  4 RL Rewards and Auxiliary Losses

Cross-entropy loss suffers from exposure bias and also does not directly
optimize the evaluation metrics such as NLI and ROUGE-L Ranzato et al. (2016).
The REINFORCE algorithm Williams (1992), on the other hand, allows for
optimizing the evaluation metrics using non-differentiable rewards. Therefore,
we use an RL multi-reward objective to promote summaries with both high
coverage of the input answers and faithfulness. Additionally, we also
introduce an auxiliary loss function for more interpretable and faithful
summaries.

###  4.1 Multi-Reward Optimization

We follow the settings of Pasunuru and Bansal (2018) for optimizing multiple
rewards. In the equations which follow,
$x=\\{x_{1},\>x_{2},\>\dots,\>x_{n^{\prime}}\\}$ refers to the input source
tokens (e.g. a question and its answers), and
$y^{*}=\\{y^{*}_{1},\>y^{*}_{2},\>\dots,\>y^{*}_{N}\\}$ refers to the gold
target summary which consists of
$\\{y^{*}_{1_{s}},\>y^{*}_{s_{s}},\>\dots,\>y^{*}_{N_{s}}\\}$ sentences.
Standard training minimizes the negative log-likelihood (NLL) loss using
teacher forcing Williams and Zipser (1989):

$L_{ml}=-\sum_{t=1}^{N}\log p(y^{*}_{t}|y^{*}_{1},...,y^{*}_{t-1},x)$ (1)

For our RL optimization, we use self-critical policy gradient training as in
Paulus et al. (2018); Rennie et al. (2017). At each time-step, we produce an
output $y^{s}$ by sampling from the current decoding probability,
$p(y^{s}_{t}|y^{s}_{1},...,y^{s}_{t-1},x)$, as well as an output $\hat{y}$
obtained by greedily decoding from the current probability distribution. We
define a reward function $r(y,x,y^{*})\in[0,1]$, i.e., the reward function
compares $y$ with $x$ and $y^{*}$. The RL loss function $L_{rl}(x,y^{*})=$:

$(r(\hat{y},x,y^{*})-r(y^{s},x,y^{*}))\sum_{t=1}^{N}\log
p(y^{s}_{t}|y^{s}_{1},...,y^{s}_{t-1},x)$ (2)

As in Paulus et al. (2018) and Pasunuru and Bansal (2018), we use a mixture of
the above two losses:

$L_{mixed}=\gamma_{rl}L_{rl}+\gamma_{ml}L_{ml},$ (3)

where $\gamma_{rl}$ and $\gamma_{ml}$ are tunable hyperparameters used as
scaling factors. Rather than applying weights to each reward, we follow
Pasunuru and Bansal (2018) and optimize $L_{mixed}$ by alternating rewards in
each minibatch.

###  4.2 Rewards

We now describe the three RL reward functions used: (1) ROUGE Lin (2004) as a
proxy for content coverage, (2) entailment (NLI) for faithfulness, and (3)
semantic area to measure the coverage of a summary in a semantic space.

ROUGE Lin (2004): Similar to Paulus et al. (2018) and Pasunuru and Bansal
(2018), we use ROUGE-L as a reward to additionally promote important content
beyond the cross-entropy loss.

Method | % Correct
---|---
BERT NLI (Falke et al. (2019)) | 64.1%
FactCC Kryscinski et al. (2020) | 70.0%
QAGs Wang et al. (2020) | 72.1%
BART MNLI (sentence) | 71.9%
RoBERTa MNLI (sentence) | 89.8%
RoBERTa MNLI (article) | 78.6%
RoBERTa MNLI (max article sentence) | 85.0%

Table 3: Results from faithfulness ranking evaluation from Falke et al.
(2019), showing the importance, both of the strength of the NLI model on
downstream faithfulness performance,and the effect of input granularity on
performance. Sentence and article in parentheses indicate the granularity of
the source input to the NLI model; max sentence calculates the max score over
all article sentences as the score of a given target sentence.

Natural Language Inference (NLI) for Faithful Summarization: We use the degree
of entailment of summaries given input answers as a reward to promote
faithfulness of answer summarization. While entailment has been used as a
reward as well as a summarization metric, we find several gaps in the current
literature. Firstly, a discussion of the effect of the quality of the NLI
evaluation model on downstream faithfulness metrics is incomplete. Also,
summarization work typically uses NLI models with document-level input, while
NLI models are generally trained on sentence-level data.

Falke et al. (2019) analyze NLI models for ranking summaries; given an input
sentence and two summary sentences, one faithful and one unfaithful to the
input, a model should rank the faithful summary higher than an unfaithful
summary. They introduce a dataset of 377 examples and measure the rank
accuracy of NLI models. They define NLI as a measure of faithfulness for
ranking summaries in the following way: Let $\mathcal{N}$ be an NLI model
which, given a claim $c$ and a premise $p$, computes $\mathcal{N}(p,c)$, the
probability that the claim is entailed by the premise. We use this to
calculate the NLI score for a summary $y$ consisting of $N_{s}$ sentences:

$\text{NLI}(y,x)=\frac{1}{N_{s}}\sum_{i=1}^{N_{S}}\max_{s\in
x}\mathcal{N}(s,y_{i_{s}})$ (4)

For the original task introduced in Falke et al. (2019), $x$ consists of a
single source sentence from the CNN-DailyMail corpus. We present our findings
on this task in Table 3. We examine how the quality of the NLI model affects
performance by comparing BART Lewis et al. (2020) and RoBERTa fine-tuned on
the MNLI corpus Williams et al. (2018). Although the performance gap of these
two models is very small on MNLI (90.2% for RoBERTa and 89.9% for BART), the
performance gap is very large on ranking these sentences (89.8% for RoBERTa
and 71.9% for BART).

We also address the effect of the granularity of the NLI model input. As
discussed above, Falke et al. (2019) perform ranking based on sentence-level
input and output. Recent work in entailment as a summarization metric,
however, uses the entire input document as input to the NLI model for
faithfulness calculations (Maynez et al., 2020), rather than computing the max
over all the input sentences as in Equation (4). We locate the full source
articles for the 377 examples and perform two experiments, one using Equation
(4), and the other which uses the entire article to score the target sentence,
$\mathcal{N}(x,y_{i_{s}})$. Performance drops when using the entire article as
the input versus using Equation (4). To ensure that the performance drop was
not caused by content truncation due to the 512 input size limitation, we also
experimented with using the article starting from the relevant source
sentence, without improvements.

Furthermore, we find that the use of NLI is particularly suitable for
AnswerSumm. We sampled six threads from our dataset. Then for each thread, we
wrote sentences entailed by the source as well as sentences based on similar
themes but not stated in the source, totaling 50 faithful and 50 hallucinated
examples. We find that the RoBERTa MNLI model can correctly identify these
examples with 96% accuracy. We believe that NLI is intuitively more suitable
for our data, which is less entity-heavy when compared to the news domain.

Semantic Area for Multi-Perspective Summarization: We aim to reward summaries
that include more of the perspectives found in the input answers. To achieve
diverse extractive summarization, Yogatama et al. (2015) embed sentences in
semantic space and select those whose convex hull maximizes the volume in that
space. This idea of semantic volume is also used to measure the semantic
overlap between summaries and references in Jung et al. (2019). We use
semantic volume as a proxy for covering multiple perspectives; the summary
with the larger semantic volume covers a wider range of views discussed in the
input. We make use of sentence-transformers Reimers and Gurevych (2019b) to
obtain sentence embeddings for each sentence. We project each embedding onto
two dimensions using PCA, and thus, our volume calculation reduces to an area
calculation, which we call Semantic Area. We use min-max normalization to keep
the reward in the range of 0 to 1.

###  4.3 Relevant Sentence Prediction

We want to more closely align the decoded summary with the source text, as
hallucinations may be caused by the decoder acting more as a language model
rather than attending to the source text Maynez et al. (2020). Aligning the
source and generated output offers a potential interpretable output during
inference, which goes beyond using attention for interpretation Wiegreffe and
Pinter (2019). We introduce an auxiliary loss by which the model predicts,
based on the decoder representation, a span of source text relevant to the
current time-step, analogous to finding evidence to support a claim of
factuality Kryscinski et al. (2020).

Let $h_{e_{i}}\in R^{dim_{e}}$ be the representation of token $x_{i}$ from the
last layer of the encoder. Let $h_{d_{i}}\in R^{dim_{d}}$ be the
representation of token $y^{*}_{i}$ from the last layer of the decoder right
before the softmax layer. Here, $dim_{e}=dim_{d}=1024$. Let $h_{e}$ be the
concatenation of all $h_{e_{i}}$ and $h_{d}$ be the concatenation of all
$h_{d_{i}}$. We then pass these representations through separate layers
$L_{e}$ and $L_{d}$ which correspond to the typical layer used in BART
classification tasks except that it outputs a representation of size 2048:

$h^{*}_{e}=L_{e}(h_{e}),\;h^{*}_{d}=L_{d}(h_{d})$ (5)

We split the resulting representations in half along the hidden dimension,
resulting in encoder representations
$h^{*}_{e{\text{-}}start},h^{*}_{e{\text{-}}end}$ and decoder representations
$h^{*}_{d{\text{-}}start},h^{*}_{d{\text{-}}end}$ which will be used for start
and end relevant source span prediction. We then compute an inner product
between these representations, resulting in logits over the input
corresponding to potential start and end spans:

$\begin{split}\text{logit}_{start}=h^{*}_{e{\text{-}}start}\bullet
h^{*T}_{d{\text{-}}start}\\\ \text{logit}_{end}=h^{*}_{e{\text{-}}end}\bullet
h^{*T}_{d{\text{-}}end}\end{split}$ (6)

Cross entropy loss can then be calculated over the start and end logits with
reference to gold spans as in SQuAD question answering training. We call this
loss $L_{span}$. Our final loss function becomes:

$L_{mixed}=\gamma_{rl}L_{rl}+\gamma_{ml}L_{ml}+\gamma_{span}L_{span},$ (7)

where $L_{span}$ is the cross-entropy loss over start and end span
predictions. Specifically, we separate input sentences with special tokens and
predict sentence-level spans, which amounts to predicting a start and end
token corresponding to a relevant sentence, so we call this model variation
Sent Relevance. For each sentence in the gold target training data, we
calculate the BM25 scores of the sentences in the source to pick the gold
relevant source sentence for that target sentence. All the timesteps
corresponding to a target sentence use the same relevant input sentence. We
also experimented with just predicting relevant source sentences at the end of
each target, using a binary sentence classification loss and a regression loss
over the BM25 scores, without significant improvements.

Method | ROUGE-1/2/L
---|---
LexRank | 26.86/5.17/22.68
TextRank | 27.44/5.05/22.13
BertSum Liu and Lapata (2019) | 30.01/5.76/24.83

Table 4: ROUGE scores for baseline extractive models.

##  5 Experimental Settings

We use the fairseq codebase Ott et al. (2019) for our experiments. Our base
abstractive text summarization model is BART Lewis et al. (2020), a pretrained
denoising autoencoder that builds off of the sequence-to-sequence transformer
of Vaswani et al. (2017). Input to the model is the question concatenated with
input answers. We fine-tune BART using a polynomial decay learning rate
scheduler with learning rate $3\mathrm{e}{-5}$, using the Adam optimizer
(Kingma and Ba, 2015). We train with 500 warmup steps and 20,000 total steps
and pick the model with the best label-smoothed cross-entropy Szegedy et al.
(2016) validation loss. Cross-entropy loss is our main loss, while the RL
rewards and sentence-relevance prediction can be viewed as auxiliary losses.
In RL experiments, we train using BART from scratch, as opposed to using a
model already fine-tuned on answer summarization, as we found that this model
better learned to follow the given rewards. Following similar ratios as in Lu
et al. (2019), we set ($\gamma_{rl}$,$\gamma_{ml}$,$\gamma_{span}$) = (0.9,
0.1, 0.0) when experimenting without sentence-relevance loss, (0.00, 1.0, 1.0)
for experiments with just relevant sentence prediction and cross-entropy loss,
and (0.9, 0.5, 0.01) for experiments with all losses. Hyperparameters were
tuned on the validation set; we found a larger $\gamma_{ml}$ necessary when
combining rewards with sentence relevance prediction to ensure that the main
negative log-likelihood loss was not drowned out by the auxiliary losses.

##  6 Results

Extractive Baselines: We use standard extractive summarization baselines such
as Lexrank Erkan and Radev (2004) and TextRank Mihalcea and Tarau (2004), and
a BERT-based extractive model, BertSum Liu and Lapata (2019). Results are
presented in Table 4. We observe a large gap between these baselines and the
extractive oracle, which is the upper bound for extractive model performance,
showing potential for improvement. Since we focus on abstractive
summarization, we leave improving extractive models for future work.

Method | ROUGE-1/2/L | NLI
---|---|---
BART baseline |  33.37/8.39/29.41 | 48.13
BART + RL (ROUGE) | 33.26/8.30/29.46 | 49.29
BART + RL (NLI) | 33.05/8.36/29.23 | 56.68
BART + RL (Semantic Area) | 33.33/8.28/29.60 | 51.14
BART + RL (ALL) |  33.54/8.41/29.65 | 51.18
BART + Sent Relevance | 32.95/8.33/29.38 | 52.31
BART + Sent Relevance + RL (ALL) | 33.21/8.29/29.46 | 56.99

Table 5: ROUGE and NLI scores for proposed models, with the two highest scores
for each metric highlighted

Abstractive Models: We present the results of the abstractive models in Table
5. We note that while the model with ROUGE reward outperforms the baseline in
ROUGE-L (the ROUGE variant optimized), we do not see larger gains in ROUGE due
to the similarity between the ROUGE optimization and NLL on our datasets. For
bullet-point summaries, minimizing the NLL is analogous to rephrasing relevant
bullet-points from the source and increasing the ROUGE-L. The model that
combines all the RL rewards achieves the highest ROUGE performance, while the
model with all RL rewards and sentence-relevance loss achieves the highest NLI
score. The faithfulness of the model with only sentence relevance loss is
further improved by adding the RL rewards. In general, we see that the model
with a single RL reward achieves the highest score of the target summaries for
that metric, i.e., the highest NLI score is achieved using only the NLI-based
reward. Additionally, we calculate the average semantic area of the resulting
summaries. The baseline model, the model with just semantic reward, and the
final model with all rewards have semantic areas of 39.7, 46.5, and 42. To
further show the effect of our dataset on multi-perspective summarization, we
train a BART model on the most related answer summarization dataset CQASumm
and find that the semantic area of that model’s summaries is 31.54. This
result shows the importance of supervision from our dataset pipeline for
ensuring coverage of multiple perspectives in answer summarization.

As automatic metrics may not correlate perfectly with human judgments, we
perform a human evaluation to determine the differences in model output
qualities. We presented two annotators who are fluent in English with the
question, answers, and summaries from three models and asked them to annotate
the summaries along the following dimensions: 1) On a Likert scale from 1-5,
label the ability to capture multiple perspectives, with points deducted for
repetition 2) On a Likert scale from 1-5, label the extent of faithfulness of
the summary, with 5 being a completely faithful summary and 1 being an
entirely inaccurate summary.

We present results in Table 6. Annotations are averaged between each annotator
and then across examples for 50 questions threads from three models. We choose
to compare the BART baseline, the BART model with all RL rewards, and the BART
model with span prediction to determine the effect of our rewards and the
multi-task loss on output quality. Pearson correlations for faithfulness and
multi-perspective scores among the annotators were 0.41 and 0.31, displaying
moderate correlation. We find that most models can generate multiple
perspective summaries. The baseline already generates multi-perspective
output, likely because the dataset pipeline produces summaries that contain
multiple perspectives, so the baseline learns to produce such output. Using a
student’s t-test with a p-value of 0.05, we find that the improvement in
faithfulness between the RL models and the baseline is statistically
significant while the other differences are not. With the span-based model,
this improvement comes at the cost of some level of abstraction, as the
percentage of novel unigrams found in the summary is 10% vs. 13% found in the
baseline and RL-only models. This reduction in abstraction likely results
because the span loss more closely binds the decoder representation with the
encoder representation, encouraging the model to copy more from the source. We
demonstrate the added advantage of our span prediction model’s
interpretability by using it to provide explanations for the generated
summaries in the Appendix.

Method | Multi-Perspective | Faithful
---|---|---
BART | 4.45 | 3.72
BART + RL (ALL) | 4.57 | 4.13
BART + Sent Relevance + RL (ALL) | 4.55 | 4.24
|  |

Table 6: Human evaluations of model outputs measuring the ability to capture
multiple perspectives and faithfulness. Higher is better.

##  7 Conclusion and Future Work

We propose multi-perspective answer summarization by introducing a pipeline
for creating a suitable dataset for the task and by introducing models to
promote high-coverage, faithful answer summaries, as seen in automatic and
human evaluations. We aim to refine this pipeline for future work by improving
the relevance and clustering components and applying them to new data sources.
We plan to study the abstractiveness-faithfulness tradeoff further, explore
additional rewards for improved summary coherence, and move beyond bullet
point summaries by building on work in sentence fusion.

##  8 Ethical Considerations

As we propose a novel conversation summarization dataset creation pipeline and
modeling components, this section is divided into the following two parts.

###  8.1 New Dataset

#### Intellectual Properties and Privacy Rights

We will be providing scripts to run our dataset creation pipeline but will not
be releasing the data itself. Access to the Yahoo! Answers dataset requires
the submission of a user-agreement form through the webscope platform
333https://webscope.sandbox.yahoo.com/catalog.php?datatype=l&did=11. We do not
do any crowdsourcing. All human annotations are done in-house. We manually
reviewed our dataset output for quality and potential problems.

###  8.2 NLP Application

#### Bias

Biases may exist in the datasets, such as political bias and gender bias in
Yahoo! Answers. Thus, models trained on these datasets may propagate these
biases.

#### Misuse Potential and Failure Mode

When used as intended, applying the summarization models described in this
paper can save people much time. However, the current models are still prone
to producing hallucinated summaries, and in such a case, they may contribute
to misinformation on the internet. We move the needle in faithful
summarization in this paper, but further research is needed to ensure the
faithfulness of abstractive summaries to address this issue, as this issue is
present among all current abstractive summarization models.

#### Environmental Cost

The experiments described in the paper make use of V100 GPUs. We used up to 8
GPUs per experiment. The experiments may take several hours. Several dozen
experiments were run due to parameter search, and future work should
experiment with distilled models for more light-weight training. We note that
while our work required extensive experiments to draw sound conclusions,
future work will be able to draw on these insights and need not run as many
large-scale comparisons. Models in production may be trained once for use
using the most promising settings.

##  9 Appendix

We show the model-generated summaries for the model ”BART + Sent Relevance +
RL (ALL)” as well as the sources sentences the model predicts as relevant at
the end of each sentence generated during decoding. In the example below, the
model can correctly abstract meaning from the source sentences and formulate
summary bullet points. Occasionally the model will output a point which is not
coherent by itself (e.g. ’It’s a great book’) or may output related but not
supported text. We believe this is due to the BM25 relevance function used for
determining relevant sentences for training. Examining this mechanism sentence
relevance prediction as a model probing task as well as improving coherence in
summaries, to go beyond bullet point summaries via methods such as sentence
fusion, are valuable areas of future work.

We show model outputs from the three models examined in human evaluation in
Table 8. We see that the baseline hallucinates several times. We also notice
how the hallucinations, as opposed to typical hallucinations in the news
domain which may replace entities, are often plausible responses. For example,
although the baseline generates an output saying that it is not a good idea to
lose weight, which is not directly stated in the source, such an answer is
very plausible. We also found that there was occasionally a fine line between
what was a hallucination and what was a plausible generated text which is not
entirely implied in the source. For example, the text stating it is not a good
idea to lose weight echoes the sentiment that the user asking the question
should make the choice for themselves, although this is not stated in this
fashion. We believe that more precisely defining the degrees of hallucination
and plausibility to be an important direction for future work.

Question: What is the secret to work/life balance?
---
Summary Sentences: |  Associated Source
Sentences:
You have to find the right balance between work and life. |  I mean you keep looking outside of work for happiness, and you want a balance, so why not be happy everywhere
If you don’t try something new, you’ll never know what you’re doing. |  If what you’re doing now isn’t working, why not try something new
You have to make them both equal. |  Only then will they matter equally
It’s a great book, and you can get it at any book store. |  It’s absolutely possible, and in my sources is a book that you can get as cheap as $1.62
I think the trick is to go to work with the right attitude. |  It seems to me that people just go to work with the wrong attitude actually
Table 7: An example of the predicted sentences from our span-based model with
all rewards. On the left side are the generated summary sentences and on the
right side are the sentences predicted to be relevant at the end of sentence
timestep during generation.  Question: average 14 year old girls weight?
---
Context: im 14 years old, 145 pounds, 5’5” and 1/2, need to know if i need to
lose weight. my brother and his friend(who is also my friend) have told me i
do, but i dont know. is this a normal weight?
Answer 1: be comfortable in your own body, don’t worry what others think of
you! If you feel like you need to lose weight then exercise and make that
choice for yourself.
Answer 2: your bro and his friend are retards who cares what they say??? and
if you think you weigh alot it probaly doesn’t show
Answer 3: First off, that is not very nice for your brother and his friends to
be telling you to loose weight … It is actually more healthy to be about 10%
over what is normal for your age. Remember that muscle weights more than fat
but it takes up less room than fat does. If you eat well and exercise daily
you have nothing to worry about.
Answer 4: i’m 5’8” and wrestled at 126 lbs when i was a freshman and
sophomore.
Answer 5: Your BMI (Body Mass Index) is 24.1 Based on this number and your age
you are not overweight, but are considered ”at risk for overweight.”
BART Baseline Summary: You are not overweight if you are eating healthy and
exercising. $\langle S\rangle$ Your weight will change as you get older.
$\langle S\rangle$ If you are overweight, it is because you have too much
muscle and not enough fat. $\langle S\rangle$ I’m not sure how old you are,
but I’m assuming you are 14. $\langle S\rangle$ It is not a good idea to lose
weight.
BART + RL (ALL) Summary: If you feel like you need to lose weight then
exercise and make that choice for yourself. $\langle S\rangle$ You are not
overweight, and you are not at risk for overweight. $\langle S\rangle$ It is
normal for a fourteen year old to be a little over weight, but not overweight.
$\langle S\rangle$ If you are a wrestler, you will know how much muscle you
have.
BART + Sent Relevance + RL (ALL) Summary: If you feel like you need to lose
weight then do so, but don’t listen to your brother and his friend. $\langle
S\rangle$ You are not overweight, but you are at risk for being overweight.
$\langle S\rangle$ You should be comfortable with your weight. $\langle
S\rangle$ If you have muscle, you will be able to lose more weight than if you
had fat.
Table 8: Example question and answers along with bullet-point answer summaries
from three models. Possible hallucinations are shown in red.

## References

  * Böhm et al. (2019) Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019.  Better rewards yield better summaries: Learning to summarise without references.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3110–3120, Hong Kong, China. Association for Computational Linguistics.
  * Chan et al. (2012) Wen Chan, Xiangdong Zhou, Wei Wang, and Tat-Seng Chua. 2012.  Community answer summarization for multi-sentence question with group L1 regularization.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 582–591, Jeju Island, Korea. Association for Computational Linguistics.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–686, Melbourne, Australia. Association for Computational Linguistics.
  * Chowdhury and Chakraborty (2019) Tanya Chowdhury and Tanmoy Chakraborty. 2019.  Cqasumm: Building references for community question answering summarization corpora.  In _Proceedings of the ACM India Joint International Conference on Data Science and Management of Data_ , pages 18–26.
  * Chowdhury et al. (2020) Tanya Chowdhury, Sachin Kumar, and Tanmoy Chakraborty. 2020.  Neural abstractive summarization with structural attention.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 3716–3722. ijcai.org.
  * Deng et al. (2020) Yang Deng, Wai Lam, Yuexiang Xie, Daoyuan Chen, Yaliang Li, Min Yang, and Ying Shen. 2020.  Joint learning of answer selection and answer summary generation in community question answering.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7651–7658. AAAI Press.
  * Dong et al. (2018) Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018.  BanditSum: Extractive summarization as a contextual bandit.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3739–3748, Brussels, Belgium. Association for Computational Linguistics.
  * Erkan and Radev (2004) Günes Erkan and Dragomir R Radev. 2004.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal of artificial intelligence research_ , 22:457–479.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  ELI5: Long form question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
  * Garg et al. (2020) Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020.  TANDA: transfer and adapt pre-trained transformer models for answer sentence selection.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7780–7788. AAAI Press.
  * Gholipour Ghalandari et al. (2020) Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020.  A large-scale multi-document summarization dataset from the Wikipedia current events portal.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1302–1308, Online. Association for Computational Linguistics.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.
  * Gu et al. (2020) Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020.  Generating representative headlines for news stories.  In _WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , pages 1773–1784. ACM / IW3C2.
  * Guo et al. (2018) Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018.  Soft layer-specific multi-task summarization with entailment and question generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 687–697, Melbourne, Australia. Association for Computational Linguistics.
  * Hadifar et al. (2019) Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019.  A self-training approach for short text clustering.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 194–199, Florence, Italy. Association for Computational Linguistics.
  * Hashemi et al. (2020) Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W Bruce Croft. 2020.  Antique: A non-factoid question answering benchmark.  In _European Conference on Information Retrieval_ , pages 166–173. Springer.
  * Huang et al. (2020) Luyang Huang, Lingfei Wu, and Lu Wang. 2020.  Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5094–5107, Online. Association for Computational Linguistics.
  * Jung et al. (2019) Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. 2019.  Earlier isn’t always better: Sub-aspect analysis on corpus and system biases in summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3324–3335, Hong Kong, China. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Laban et al. (2020) Philippe Laban, Andrew Hsi, John Canny, and Marti A. Hearst. 2020.  The summary loop: Learning to write abstractive summaries without examples.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5135–5150, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2018) Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018.  Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1430–1441, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018.  Generating wikipedia by summarizing long sequences.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2008) Yuanjie Liu, Shasha Li, Yunbo Cao, Chin-Yew Lin, Dingyi Han, and Yong Yu. 2008.  Understanding and summarizing answers in community-based question answering services.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 497–504, Manchester, UK. Coling 2008 Organizing Committee.
  * Lu et al. (2019) Yao Lu, Linqing Liu, Zhile Jiang, Min Yang, and Randy Goebel. 2019.  A multi-task learning framework for abstractive text summarization.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 9987–9988. AAAI Press.
  * Matsumaru et al. (2020) Kazuki Matsumaru, Sho Takase, and Naoaki Okazaki. 2020.  Improving truthfulness of headline generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1335–1346, Online. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Narayan et al. (2018a) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Narayan et al. (2018b) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b.  Ranking sentences for extractive summarization with reinforcement learning.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1747–1759, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Pande et al. (2013) Vinay Pande, Tanmoy Mukherjee, and Vasudeva Varma. 2013.  Summarizing answers for community question answer services.  In _Language Processing and Knowledge in the Web_ , pages 151–161. Springer.
  * Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018.  Multi-reward reinforced summarization with saliency and entailment.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 646–653, New Orleans, Louisiana. Association for Computational Linguistics.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Reimers and Gurevych (2019a) Nils Reimers and Iryna Gurevych. 2019a.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers and Gurevych (2019b) Nils Reimers and Iryna Gurevych. 2019b.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017.  Self-critical sequence training for image captioning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 1179–1195. IEEE Computer Society.
  * Shapira and Levy (2020) Ori Shapira and Ran Levy. 2020.  Massive multi-document summarization of product reviews with weak supervision.
  * Song et al. (2017) Hongya Song, Zhaochun Ren, Shangsong Liang, Piji Li, Jun Ma, and Maarten de Rijke. 2017.  Summarizing answers in non-factoid community question-answering.  In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM 2017, Cambridge, United Kingdom, February 6-10, 2017_ , pages 405–414. ACM.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 2818–2826. IEEE Computer Society.
  * Tomasoni and Huang (2010) Mattia Tomasoni and Minlie Huang. 2010.  Metadata-aware measures for answer summarization in community question answering.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 760–769, Uppsala, Sweden. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Wang et al. (2014) Lu Wang, Hema Raghavan, Claire Cardie, and Vittorio Castelli. 2014.  Query-focused opinion summarization for user-generated content.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 1660–1669, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  * Wang et al. (2007) Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007.  What is the Jeopardy model? a quasi-synchronous grammar for QA.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 22–32, Prague, Czech Republic. Association for Computational Linguistics.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Hong Kong, China. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Williams (1992) R. J. Williams. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine Learning_ , 8:229–256.
  * Williams and Zipser (1989) Ronald J Williams and David Zipser. 1989.  A learning algorithm for continually running fully recurrent neural networks.  _Neural computation_ , 1(2):270–280.
  * Xu et al. (2017) Jiaming Xu, Bo Xu, Peng Wang, Suncong Zheng, Guanhua Tian, and Jun Zhao. 2017.  Self-taught convolutional neural networks for short text clustering.  _Neural Networks_ , 88:22–31.
  * Yogatama et al. (2015) Dani Yogatama, Fei Liu, and Noah A. Smith. 2015.  Extractive summarization by maximizing semantic volume.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1961–1966, Lisbon, Portugal. Association for Computational Linguistics.