# A Simple Proof of the Johnson–Lindenstrauss Extension Theorem

Manor Mendel  Mathematics & Computer Science Department, The Open University
of Israel<EMAIL_ADDRESS>
###### Abstract.

Johnson and Lindenstrauss proved that any Lipschitz mapping from an $n$-point
subset of a metric space into Hilbert space can be extended to the whole
space, while increasing the Lipschitz constant by a factor of $O(\sqrt{\log
n})$. We present a simplification of their argument that avoids dimension
reduction and the Kirszbraun theorem.

The Lipschitz constant of a mapping $f:T\to Y$ between metric spaces is
defined as $\mathrm{Lip}(f)=\sup_{x\neq y\in T}d_{Y}(f(x),f(y))/d_{T}(x,y)$.
The following is a Lipschitz extension theorem due to Johnson and
Lindenstrauss [2].

######  Theorem 1.

Let $T$ be an arbitrary $n$-point metric space, and $X\supset T$ an arbitrary
superspace. Let $\mathcal{H}$ be a Hilbert space and $f:T\to\mathcal{H}$ a
mapping . Then there exists an extension $F:X\to\mathcal{H}$ of $f$ such that
$F|_{T}=f$, and $\mathrm{Lip}(F)\lesssim\sqrt{\log n}\cdot\mathrm{Lip}(f)$.

As was pointed out in [3], Theorem 1 also follows from Ball’s extension
theorem [1] used in conjunction with the Markov type 2 estimates in [4]. The
proof of Theorem 1 in [2] introduced the celebrated Johnson–Lindenstrauss
lemma about dimension reduction of finite point sets in Hilbert space. In this
note we give a simple argument which is a reinterpretation of the original
from [2] that uses neither dimension reduction nor the Kirszbraun extension
theorem.

###### Proof of Theorem 1.

We may assume without loss of generality that $\mathcal{H}=\mathbb{R}^{n}$,
i.e., $f:T\to\mathbb{R}^{n}$. Furthermore, assume that by rescaling
$\mathrm{Lip}(f)=1$. Let $g_{1},\ldots,g_{n}$ be i.i.d. standard Gaussian
random variables defined on a probability space $\Omega$, and consider the
real-valued Hilbert space $H=L_{2}(\Omega)$.

Consider the linear mapping $k:\mathbb{R}^{n}\to H$ defined by
$k(x_{1},\ldots,x_{n})=\sum_{i=1}^{n}x_{i}g_{i}.$ It is a property of the
Gaussian distribution that if $g_{1},\ldots,g_{n}$ are independent standard
Gaussian variables, then $\sum_{i=1}^{n}x_{i}g_{i}$ is a centered Gaussian
random variable with variance $\sum_{i=1}^{n}x_{i}^{2}$, which means that
$\|\sum_{i}x_{i}g_{i}\|_{L_{2}(\Omega)}=\|(x_{1},\ldots,x_{n})\|_{2}$. Hence
$k$ is an isometric isomorphism of $\mathbb{R}^{n}$ in
$k(\mathbb{R}^{n})\subset H$. Thus, $u:=k\circ f:T\to k(\mathbb{R}^{n})$ is
also 1-Lipschitz. We will use the notation $u=(u_{\omega})_{\omega\in\Omega}$,
and $k=(k_{\omega})_{\omega\in\Omega}$.

Fix $\omega\in\Omega$. Since $u_{\omega}:T\to\mathbb{R}$, by the McShane
extension theorem (also known as the nonlinear Hahn–Banach theorem),
$u_{\omega}$ can be extended to $U_{\omega}:X\to\mathbb{\mathbb{R}}$ such that
$U_{\omega}|_{T}=u_{\omega}$, and
$\mathrm{Lip}(U_{\omega})=\mathrm{Lip}(u_{\omega})$. The McShane extension is
a simple result. For completeness, we provide a quick proof at the end of this
note, see also [5, Theorem 1.33]. We thus obtain
$U:=(U_{\omega})_{\omega\in\Omega}:X\to H$, an extension of $u$. Using the
orthogonal projection $P:H\to k(\mathbb{\mathbb{R}}^{n})$, we see that
$F:=k^{-1}\circ P\circ U:X\to\mathbb{R}^{n}$ is an extension of $f$. Since
$\mathrm{Lip}(k^{-1})=\mathrm{Lip}(P)=1$, we have
$\mathrm{Lip}(F)\leq\mathrm{Lip}(U)$. We next bound $\mathrm{Lip}(U)$. For
$x,y\in X$:

$\displaystyle\|U(x)-U(y)\|_{L_{2}(\Omega)}$
$\displaystyle=\Bigl{(}\mathbb{E}_{\Omega}\bigl{[}|U_{\omega}(x)-U_{\omega}(y)|^{2}\bigr{]}\Bigr{)}^{1/2}$
$\displaystyle\leq
d(x,y)\Bigl{(}\mathbb{E}_{\Omega}[\mathrm{Lip}(U_{\omega})^{2}]\Bigr{)}^{1/2}$
$\displaystyle=d(x,y)\Bigl{(}\mathbb{E}_{\Omega}[\mathrm{Lip}(u_{\omega})^{2}]\Bigr{)}^{1/2}$
$\displaystyle=d(x,y)\left(\mathbb{E}_{\Omega}\max_{s\neq t\in
T}\Bigl{(}\frac{\sum_{i}f(s)_{i}g_{i}(\omega)-\sum_{i}f(t)_{i}g_{i}(\omega)}{d(s,t)}\Bigr{)}^{2}\right)^{1/2}$
$\displaystyle\leq d(x,y)\biggl{(}\mathbb{E}_{\Omega}\max_{s\neq t\in
T}\Bigl{(}\sum_{i=1}^{n}\frac{f(s)_{i}-f(t)_{i}}{\|f(s)-f(t)\|_{2}}g_{i}(\omega)\Bigr{)}^{2}\biggr{)}^{1/2}$
(1) $\displaystyle=d(x,y)\Bigl{(}\mathbb{E}_{\Omega}\max_{s\neq t\in
T}G_{f(s)-f(t)}^{2}\Bigr{)}^{1/2},$

where $G_{z}:=\sum_{i}\frac{z_{i}}{\|z\|_{2}}g_{i}$.
$\\{G_{f(s)-f(t)}\\}_{s,t\in T}$ are standard Gaussians (but not necessarily
stochastically independent). It is a classical and elementary fact that for
$m\geq 2$ standard Gaussians $G_{1},\ldots,G_{m}$, with any dependence
structure,

(2) $\mathbb{E}[\max\\{G_{1}^{2},\ldots,G_{m}^{2}\\}]\lesssim\log m.$

For completeness, we provide a quick proof of (2) below. Since $|T|=n$, the
maximum in (1) is over at most $n^{2}$ squared Gaussians, and therefore (1) is
bounded above by a constant times $d(x,y)\cdot\sqrt{\log n}$. ∎

###### Proof of (2).

By a classical Gaussian tail bound, $\Pr[G_{i}\geq t]\leq e^{-t^{2}/2}$, and
therefore $\Pr[G_{i}^{2}\geq t]\leq 2e^{-t/2}$. By the union bound,
$\Pr[\max\\{G_{1}^{2},\ldots,G_{m}^{2}\\}\geq t]\leq 2me^{-t/2}$. For any
$s\geq 0$, we can estimate $\mathbb{E}[\max\\{G_{1}^{2},\ldots,G_{m}^{2}\\}]$
by partitioning the integral up to parameter $s\geq 0$ and from $s$ to
$\infty$ as follows:

$\mathbb{E}[\max\\{G_{1}^{2},\ldots,G_{m}^{2}\\}]\leq
s+\int_{s}^{\infty}\Pr\left[\max\\{G_{1}^{2},\ldots,G_{m}^{2}\\}\geq
t\right]dt\\\ \leq s+\int_{s}^{\infty}2me^{-t/2}dt=s+4me^{-s/2}.$

Substituting $s=2\log m$ (and noting that $m\geq 2$) proves (2). ∎

###### Proof of McShane extension for finite domains.

Let $T\subset X$, $T$ finite, and $q:T\to\mathbb{R}$ $L$-Lipschitz. Define
$Q:X\to\mathbb{R}$ by $Q(x)=\min_{t\in T}q(t)+Ld_{X}(x,t)$. It is left to the
reader to check that $Q|_{T}=q$ using the Lipschitz condition on $q$. Next we
prove the Lipshitz condition for $Q$. Fix $x,y\in X$, and let $t\in T$ satisfy
$Q(x)=q(t)+Ld_{X}(x,t)$. Applying the definition of $Q(y)$, and the triangle
inequality in $X$, we have

$Q(y)\leq q(t)+Ld_{X}(y,t)\leq q(t)+Ld_{X}(x,t)+Ld_{X}(x,y)=Q(x)+Ld_{X}(x,y).$

Hence $Q(y)-Q(x)\leq Ld_{X}(x,y)$. By symmetry, we also have $Q(x)-Q(y)\leq
Ld(x,y)$. ∎

## Acknowledgments.

The author thanks Assaf Naor and the anonymous referees for comments and
suggestions that considerably improved the presentation.

## References

  * [1] Ball, K. (1992).  Markov chains, Riesz transforms and Lipschitz maps.  Geom. Funct. Anal. 2(2): 137–172.  doi.org/10.1007/BF01896971
  * [2] Johnson, W. B., Lindenstrauss J., (1984).  Extensions of Lipschitz mappings into a Hilbert space.  In Beals, R., Beck, A., Bellow, A., Hajian, eds., Conference in Modern Analysis and Probability (New Haven, Conn.,1982). Contemporary Mathematics, 26. Providence, RI: American Mathematical Society, pp. 189–206.  doi.org/10.1090/conm/026/737400
  * [3] Mendel, M., Naor, A. (2006).  Some applications of Ball’s extension theorem.  Proc. Amer. Math. Soc. 134(9): 2577–2584. doi.org/10.1090/S0002-9939-06-08298-0
  * [4] Naor, A., Peres, Y., Schramm, O., Sheffield, S. (2006).  Markov chains in smooth Banach spaces and Gromov-hyperbolic metric spaces.  Duke Math. J., 134(1): 165–197, 2006.  doi.org/10.1215/S0012-7094-06-13415-4
  * [5] Weaver, N. (2018).  Lipschitz Algebras, 2nd ed. Hackensack, NJ:  World Scientific Publishing Co.


T

# Do Multi-Document Summarization Models Synthesize?

Jay DeYoung1, Stephanie C. Martinez1, Iain J. Marshall2, Byron C. Wallace1
1Khoury College of Computer Sciences, Northeastern University, Boston, MA, USA
2King’s College London, London
{deyoung.j, martinez.s<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Multi-document summarization entails producing concise synopses of collections
of inputs. For some applications, the synopsis should accurately _synthesize_
inputs with respect to a key property or aspect. For example, a synopsis of
film reviews all written about a particular movie should reflect the average
critic consensus. As a more consequential example, consider narrative
summaries that accompany biomedical _systematic reviews_ of clinical trial
results. These narratives should fairly summarize the potentially conflicting
results from individual trials.

In this paper we ask: To what extent do modern multi-document summarization
models implicitly perform this type of synthesis? To assess this we perform a
suite of experiments that probe the degree to which conditional generation
models trained for summarization using standard methods yield outputs that
appropriately synthesize inputs. We find that existing models do partially
perform synthesis, but do so imperfectly. In particular, they are over-
sensitive to changes in input ordering and under-sensitive to changes in input
compositions (e.g., the ratio of positive to negative movie reviews). We
propose a simple, general method for improving model synthesis capabilities by
generating an explicitly diverse set of candidate outputs, and then selecting
from these the string best aligned with the expected aggregate measure for the
inputs, or _abstaining_ when the model produces no good candidate. This
approach improves model synthesis performance. We hope highlighting the need
for synthesis (in some summarization settings), motivates further research
into multi-document summarization methods and learning objectives that
explicitly account for the need to synthesize.

##  1 Introduction

_Multi-document summarization_ (MDS) models aim to distill inputs into concise
synopses that preserve key content. Examples of MDS include summarizing news
articles (Dang, 2005; Fabbri et al., 2019; Gholipour Ghalandari et al., 2020;
Evans et al., 2004), answering questions from multiple sources (Dang, 2006),
and producing overviews of scientific literature (Liu et al., 2018; Lu et al.,
2020; Mollá and Santiago-Martínez, 2012; Wallace et al., 2020; DeYoung et al.,
2021). We expect summarization models to produce outputs consistent with
inputs (Kryscinski et al., 2020; Nan et al., 2021b), e.g., discussing the same
types of entities (Nan et al., 2021a) and allowing one to answer questions
similar in a way that is consistent with individual inputs (Wang et al., 2020;
Scialom et al., 2021).

In some applications models must _synthesize_ inputs—i.e., aggregate
potentially conflicting information—to yield an accurate synopsis (Figure 1).
As a simple example, consider the meta-reviews of movies featured on Rotten
Tomatoes,111A website that aggregates film reviews:
https://www.rottentomatoes.com/. which provide a consensus view of individual
critic opinions. These reviews should reflect the mean and range of sentiment
implicit in the input critiques: A summary of mostly negative reviews (e.g.,
_Gigli_) should communicate that the film was widely panned; a summary of
mixed reviews (_The Fifth Element_) ought to convey that critics disagreed and
discuss the main positive and negative attributes.

A more consequential example is the task of summarizing the evidence presented
in clinical trials. Individual trials will frequently present conflicting
evidence about whether or not a particular health intervention is effective.
An ideal summary of the evidence would appropriately weigh the findings
presented in the constituent inputs and reflect the evidence on balance.

Figure 1: Two multi-document summarization tasks where models must implicitly
synthesize inputs to produce accurate summaries. Left: Summarizing film
reviews with varying sentiment to yield a _critics consensus_. Right:
Summarizing trials that have evaluated a particular medical invention.

What are the desiderata of multi-document _synthesis_? First, summaries
produced by models should be consistent with the input data, with respect to
the latent property of interest. In the case of Rotten Tomatoes, the
_sentiment_ of the summary should be in line with the aggregate sentiment
expressed in the individual critic reviews. A corollary to this is that models
should be sensitive to changes in the composition of inputs, e.g., removing
most of the negative reviews from a set of inputs should yield a summary with
a corresponding increase in the expressed sentiment.

In this work we evaluate neural MDS models with respect to these criteria. To
this end we use a meta-reviews dataset from Rotten Tomatoes (Leone, 2020) and
a dataset of systematic reviews (meta-analyses) summarizing the evidence about
medical interventions (Wallace et al., 2020). For the former we probe the
degree to which generated meta-review sentiment agrees with the expected
aggregate sentiment score; for the latter we evaluate whether the generated
summary indicates that the input evidence suggests, on balance, that the
intervention under consideration was effective.

Our main contributions are summarized as follows. (1) To the best of our
knowledge, this is the first work to investigate implicit _synthesis_ in
summarization, and the degree to which modern models are capable of
this.222See Appendix A for discussion of related content aggregation work over
structured relations Shah et al. (2021a). (2) We show that “off-the-shelf”
neural MDS models are somewhat inconsistent and insensitive with respect to
performing synthesis in summarization. (3) We propose and evaluate a simple
and general technique of generating a diverse set of output candidates
(Vijayakumar et al., 2016) and then selecting from these on the basis of
agreement with an expected aggregate measure (based on inputs), with promising
results.

##  2 Synthesis and Summarization

In standard multi-document summarization, we assume inputs $(X_{i},y_{i})$,
where $X_{i}=\\{x_{i1},...,x_{i{|X_{i}|}}\\}$. We then typically train a
summarization model with parameters $\theta$, to consume $X_{i}$ and yield
summaries $\hat{y}_{i}$ as similar as possible to targets $y_{i}$. More
precisely, the standard objective entails finding estimates for $\theta$ which
maximize target token log-probabilities. Assuming the input documents $x_{ij}$
in $X_{i}$ have been linearized (i.e., concatenated, usually with adjoining
special tokens to demarcate individual inputs) into a string $x^{\oplus}_{i}$
of input tokens, this objective takes the form: $\sum_{t=1}^{|y_{i}|}\text{log
}p_{\theta}(y_{it}|y_{i1},...,y_{i(t-1)},x_{i}^{\oplus})$, where $p_{\theta}$
is a probability assigned to the token at position $t$ in the (linearized)
target $x^{\oplus}_{i}$ by a summarization model with parameters $\theta$. By
myopically focusing on encouraging the model to produce tokens that mimic the
targets, this objective aligns with standard (but flawed) measures of
automated summary quality like ROUGE (Lin, 2004), which quantify $n$-gram
overlap between targets $y_{i}$ and outputs $\hat{y}_{i}$.

We are interested in settings in which there is an additional, latent property
implicit in the constituent input texts $x_{ij}$, $z_{ij}$. For example,
$z_{ij}$ might reflect the sentiment in critique $j$ of the film indexed by
$i$. Summaries should _synthesize_ this aspect, i.e., the generated summary
$\hat{y}_{i}$ should implicitly convey an aggregated $z_{i}$ which reflects a
synthesis or aggregation $G$ over $Z_{i}=\\{z_{i1},...z_{i|X_{i}|}\\}$. That
is, we assume $z_{i}=G(Z_{i})$ . In both cases considered here—summaries of
film critiques and synopses of clinical trials evidence—$G$ can reasonably be
assumed to be a (weighted) mean,
$G(Z_{i})=\frac{1}{|X_{i}|}\sum_{j=1}^{|X_{i}|}\alpha_{ij}z_{ij}$. That is,
summaries should roughly reflect the average sentiment and reported treatment
effect in the cases of movie reviews and clinical trial reports, respectively.

We investigate the following questions. (1) Do model summaries $\hat{y}_{i}$
reflect the anticipated aggregate aspect of interest? That is, how well
calibrated is the aspect communicated in the generated summary
($z_{i\hat{y}}$) compared to the expected $z_{i}$? (2) Can we _improve_ the
ability of summarization models to synthesize by explicitly incorporating
synthesis targets $z_{i}$ into the decoding process?

We propose a simple inference-time procedure to explicitly preference output
candidates that align with the expected aggregate property of interest (e.g.,
average sentiment), and report promising results—under both automatic and
manual evaluation—for the approach. This strategy naturally lends itself to
_cautious_ summarization, i.e., approaches where the model can _abstain_ from
generating an output if it does not produce any candidates that reflect the
anticipated aggregate measure.

###  2.1 Movie Reviews

We first consider a dataset comprising movie reviews and associated meta-
reviews summarizing these from Rotten Tomatoes. An in-house staffer summarizes
audience reviews333written by designated “top-critics”, audience members
recognized for quality and quantity of reviews into meta-reviews. These meta-
reviews synthesize the constituent input reviews, reflecting the aggregate
critic reception of a film. Each meta-review is associated with a numerical
“Tomatometer” score, which is an overall measure of what percent reviews were
positive for the corresponding film ($G$ then is an average of the positive
indicator per review). The Rotten Tomatoes dataset we use comprises 9095
movies with meta-reviews constructed from 244,000 individual reviews (Table
1).

Measuring sentiment in movie reviews. As our measure $g$ we train a BERT model
(Devlin et al., 2019) using the continuous (fine-grained) sentiment targets
provided in the SST dataset (Socher et al., 2013).444SST is itself based on a
collection of Rotten Tomatoes critic reviews (Pang and Lee, 2005). We verified
that the SST text fragments do not overlap with our target reviews by manually
checking any (fragment, review) pair with substantial ($\geq 75\%$) overlap
for one quarter of all reviews. We trained this model for 3 epochs using a
learning rate of 5e-5 using the Huggingface
library555github.com/huggingface/transformers/blob/main/examples/pytorch/text-
classification with no hyperparameter tuning. While the raw text of the SST
dataset is in-domain, the targets themselves are not. We find a reasonably
strong correlation between our sentiment estimates and the “true” meta-review
sentiment (“Tomatometer” score): The R2 (centered) is 0.696, mean squared
error (MSE) of 0.022, and Pearson’s r of 0.836 (Figure 2, upper left).

| Train | Dev | Test | Train | Dev† | Test
---|---|---|---|---|---|---
Number of metareviews | 7251 | 932 | 912 | 1675 | 360 | 397
Avg. metareview length | 32.0 | 32.6 | 32.4 | 101 | 107 | 111
Total number of inputs | 195033 | 24336 | 24474 | 11054 | 1238 | 2669
Avg. number of inputs | 26.9 | 26.1 | 26.8 | 6.6 | 3.4 | 6.7
Avg length of individual input | 30.6 | 30.8 | 30.6 | 475 | 379 | 449
Avg length of concatenated inputs | 822 | 804 | 822 | 2641 | 1336 | 2544
Target Percent Positive | 59.5 | 62.1 | 61.2 | 31.9 | 31.4 | 35.0
Table 1: Dataset statistics for movie reviews (left) and systematic reviews
(right). Number of meta-reviews, average meta-review length (tokens), input
reviews per split, average number of inputs per instance, average total length
of instance-inputs. For movie reviews, the target percent positive reports the
fraction of metareviews with a positive sentiment; for systematic reviews this
refers to the fraction of metareviews reporting a significant effect.
$\dagger$ We subset the original dev set to instances of $\leq 4k$ tokens
(accommodating T5; other models can consume up to 16k).

###  2.2 Biomedical Systematic Reviews

Our second dataset is a collection of systematic reviews from the Cochrane
Collaboration.666An international non-profit dedicated to helping healthcare
providers make evidence-based decisions. This dataset comprises roughly 2600
systematic reviews summarizing a total of 16,500 clinical trials evaluating
interventions in healthcare (Table 1). Each review includes both a natural
language summary and accompanying statistical meta-analysis results. The
latter provides an aggregate statistical summary of the individual (study-
level) data extracted from the trials included in each review. The natural
language summary should accurately convey and contextualize the findings of
the meta-analysis. Therefore, the (lack of) treatment efficacy communicated in
a given summary should generally agree with the direction of the corresponding
meta-analytic point estimate.

Measuring effects in evidence syntheses For systematic reviews of clinical
trials, we resort to a less granular _classification_ model
$g(x_{ij}),g(y_{i})$ which attempts to infer whether a given piece of text
reports a significant result or not. In particular we use RobotReviewer
(Marshall et al., 2017; DeYoung et al., 2020). Given a narrative describing a
clinical trial result (or a systematic review summary of such results),
RobotReviewer predicts whether the reported result indicates a significant
effect of the treatment being investigated, or not. We can compare this
prediction to the “truth”, which here is derived from the meta-analytic result
(specifically by checking whether $p<0.05$). Applying this off-the-shelf model
to the manually composed summaries accompanying the meta-analyses in our
Cochrane set, we observe a macro-average F1 score of 0.577 (Table 10, Appendix
C), providing a reasonable (if weak) measure for this task.

##  3 Models

We evaluate a suite of transformer (Vaswani et al., 2017) summarization
models: Longformer (Beltagy et al., 2020), Pegasus (Zhang et al., 2020),
PRIMERA (Xiao et al., 2022), and T5 (Raffel et al., 2020). PRIMERA was
designed and pre-trained specifically for multi-document summarization. And
while not explicitly designed as multi-document summarization models, both
Pegasus Zhang et al. (2020) and
T5777https://huggingface.co/osama7/t5-summarization-multinews have been used
on multi-document tasks, while Longformer has been used for a related multi-
document summarization task (DeYoung et al., 2021). See Appendix B for
hyperparameter settings.

Figure 2: Movie Reviews: Actual vs. Predicted Sentiments on generated
summaries. We replaced LED with human outputs (upper left) for comparison; see
Figure 8 in Appendix C for all models.

##  4 Experiments

###  4.1 How well do summarization models synthesize?

We report sentiment performance for all models (Table 2). These metrics
quantify the strength of the relationship between (a) the continuous sentiment
inferred (via our text-regression $g$) over model generated or reference
(human written) summaries and (b) the reference sentiment (Tomatometer) score.
Across these metrics, correlations between the sentiment measured in model
generated outputs and the Tomatometer score are considerably lower than that
between the same measurement over human-composed summaries and said score.
This implies that human authors do a better job of synthesis than the models
when composing summaries.

For systematic reviews (Section 2.2), we are able to measure $g$ whether a
text appears to report significant treatment effect or not, and we can compare
this against the $p$-value from the corresponding statistical meta-analysis.
This permits only a coarse assessment of synthesis, as we are unable to
measure correlations. Instead we report classification metrics describing how
often the effect significance inferred from a summary (generated or manually
written) matches the ground truth derived from the meta-analysis (Table 2).
The results are qualitatively similar to the sentiment case, in that the
humans appear to do a better job of synthesis — as best we can measure, the
significance reported in their summaries better aligns with the statistical
results than in model generated summaries.

###  4.2 Sensitivity to Input Ordering

Synthesis of inputs should be invariant to ordering (e.g., the critics’
consensus on a film does not depend on the order in which one reads the
reviews). Here we evaluate if models are sensitive to input orderings with
respect to the synthesized aspect of interest ($z_{i\hat{y}}$) in the
resultant outputs. Specifically, $X_{i}=\\{x_{i1},...,x_{i{|X_{i}|}}\\}$ will
constitute an arbitrary ordering of inputs reflected in the linearized version
$x^{\oplus}_{i}$. This ordering should not affect the aggregate aspect
$z_{i\hat{y}}$ in the summary.

To evaluate if models realize this invariance, we permute the instance $i$
inputs $X_{i}$ (and, consequently, the linearized $x^{\oplus}_{i}$) one
hundred times, randomizing input orderings. For each such permutation
$\tilde{X}_{i}$ (and associated $\tilde{x}^{\oplus}_{i}$), we generate a
summary $\hat{y}_{i}$ and estimate of the resultant aspect
$\tilde{z}_{i\hat{y}}$, using the corresponding measurement model. By
repeating this process for each instance $i$, we can construct an empirical
distribution over $\tilde{z}_{i\hat{y}}$’s under different random orderings.

| R2 | Pearson’s r | MSE | ROUGE1
---|---|---|---|---
LED | 0.551 | 0.742 | 0.042 | 0.242
PRIMERA | 0.608 | 0.780 | 0.037 | 0.254
T5 | 0.516 | 0.720 | 0.046 | 0.253
Pegasus | 0.530 | 0.730 | 0.044 | 0.245
Reference | 0.697 | 0.836 | 0.023 |

| F1-score | ROUGE1
---|---|---
LED | 0.490 | 0.259
PRIMERA | 0.526 | 0.253
T5 | 0.521 | 0.206
Pegasus | 0.568 | 0.212
Reference | 0.577 |

Table 2: Base synthesis results. Movie reviews (left): correlations (R2,
Pearson’s r, mean-squared errors) between sentiment measured in model outputs
and targets. Systematic reviews (right): we report macro-averaged F1s.

Figure 3: The spread of sentiment/treatment effect measured in outputs
produced from permuted input orderings. Left: Movie review sentiment. Right:
Systematic review significance prediction entropy (0 indicates order
insensitivity) on the subset of reviews that report _significant_ effects.

Movie reviews. We zero-mean the $\tilde{z}_{i\hat{y}}$’s inferred over each
instance, and combine the distributions from all instances into a histogram
(Figure 3 left). This shows the spread of sentiments inferred over outputs
under random input orderings minus the corresponding instance mean sentiment.
Were a model completely invariant to ordering, the empirical distribution over
these differences would collapse to 0. Instead, we observe a relatively wide
spread in the sentiment measured over outputs generated from different
permutations, indicating a counter-intuitive sensitivity to orderings.888For a
ROUGE1 comparison, see Appendix D, Figure 10.

Systematic reviews. For each $X_{i}$ we have 100 order permutations and
associated summaries; we infer whether these report _significant results_ or
not, and record the fraction that do ($p_{i}$). If models were invariant to
ordering, this fraction would always be 0 or 1. Values in-between suggest the
model flips the report conclusion as a result of different input orderings.
Figure 3 (right) shows a histogram of entropies over $p_{i}$, computed over
the subset of examples where the associated meta-analysis indicates a
significant effect.999These are the more interesting cases; we provide results
over the entire dataset in Appendix Figure 9. Densities away from zero
indicate sensitivity to ordering.

###  4.3 Sensitivity to Input Composition

Figure 4: Model sentiment sensitivity to manipulated input sentiment
composition. The intensity patterns indicate that models tend to oscillate
between low and high sentiments in outputs, and are not responsive to subtler
shifts in input sentiment. For context we include a model regression (blue)
and the reference sensitivity regression (black).

Synthesis models should be responsive to changes in the distribution of the
attribute to be synthesized in the input composition: If we increase the ratio
of positive to negative reviews in an input set, we would anticipate a
concomitant change in the sentiment communicated in the meta-review
$z_{i\hat{y}}$. To assess if models meet this synthesis desiderata, we
manipulate model inputs $X_{i}$ in such a way to induce an expected change in
the target measure $z_{i\hat{y}}$; we then measure if the output yields a
summary that aligns with this expected change.

Movie reviews. We manipulate the ratio of positive to negative reviews and
observe the resultant change in the property of interest latent in the
corresponding output. We take movies with mixed reviews, and delete 10%, 20%,
30%, …, 100% of the positive inputs, retaining the negative inputs; we then
repeat the process but instead remove negative inputs. For each of these
permutations, we measure the input sentiment, the meta-review sentiment, and
how well they correlate (Table 3).

| R2 | P.’s r | MSE | F1 | Acc.
---|---|---|---|---|---
LED | 0.524 | 0.724 | 0.057 | 0.510 | 0.684
PRIMERA | 0.572 | 0.756 | 0.052 | 0.533 | 0.675
T5 | 0.481 | 0.694 | 0.063 | 0.469 | 0.658
Pegasus | 0.499 | 0.706 | 0.060 | 0.452 | 0.680
Table 3: Movie (left): Correlation (R2, Pearson’s r, MSE) and Systematic
(right) reviews: Classifications (F1, accuracy) for subsampled inputs and
generations.

Figure 4 plots the relationship between the fraction of positive reviews in
the (manipulated) input sets and the granular sentiment score inferred over
the resultant outputs. The models are generally undersensitive to changes in
their input: rather than having a change in meta-review sentiment equivalent
in size to changes in input sentiment (a slope of 1, as we observe when we fit
a model to the human written summaries). Models tend to have trouble changing
their sentiment, and require a large change in input distribution to
substantially change the sentiment communicated in the output.

Systematic Reviews. To measure sensitivity to changes in input composition, we
manipulate inputs $X_{i}$ such that the meta-analysis result (target
$z_{i\hat{y}}$) flips from a significant effect to no effect, or from no
effect to an effect. We first take a subset of the reviews that have
conflicting evidence (yielding 139 unique reviews). We then order inputs in
these by (weighted) effect sizes,101010In fixed effects meta-analysis the
weights are inverse variances associated with study-level effect estimates.
and remove subsets which ought to flip the significance result.

##  5 Improving Synthesis in Summarization

We propose a straightforward post-hoc approach to improving the synthesis
performed by multi-document summarization models: (1) Generate an explicitly
_diverse_ set of output candidates111111See Appendix Tables 11, 12 for an
ablation over diversity vs. standard beam search outputs; (2) Select from
these as the final output the candidate that best agrees with the expected
synthesis result (as predicted by an external model).121212For a related
generate-and-select approach Oved and Levy (2021) see Appendix A.131313We
experiment with an additional decoding method: constrain the beam search to
produce summaries with an approximately correct sentiment, Appendix E.

Figure 5: Proposed strategy to improve synthesis. We generate an intentionally
diverse set of output candidates (Vijayakumar et al., 2016) and then select
from these the text that best agrees with the _predicted_ aggregate property
of interest (here, sentiment). We can also _abstain_ when the model fails to
yield an appropriate output.

For (1), we rely on a previously proposed technique for generating diverse
outputs $\mathcal{C}_{i}$ from input $x^{\oplus}_{i}$, namely _Diverse Beam
Search_ (DBS) (Vijayakumar et al., 2016). This method modifies standard beam
search to maintain multiple _groups_ of beams. During decoding, a term is
added to the next-token log probabilities, penalizing production of strings
similar to candidates in _other_ groups.141414This penalty requires a
hyperparameter $\lambda$ that encodes the relative importance of diversity; we
use $\lambda$=0.5 and did not tune this. We also used 5 groups and 1 beam per
group.

In (2) we would like to select the output that best synthesizes the property
of interest; this requires a mechanism for specifying what we _expect_ the
synthesized property be, given the inputs. For example, if we know the
sentiment scores associated with input movie reviews, we might enforce that
the sentiment expressed in the output agrees with the average of these. To
realize this intuition, we can select as final output from $\mathcal{C}_{i}$
the string that best aligns with this anticipated aggregate property
(sentiment score or significance finding). Operationally, this requires an
external model to measure—or estimate—the aspect of interest as latent in a
given candidate output. This is a limitation of the approach, but in many
settings it may be feasible to identify or construct a model; we were able to
do so for both tasks considered in this paper.

There is no guarantee that _any_ member of $\mathcal{C}_{i}$ will align well
with the anticipated aggregated property. In such cases, we have no means of
yielding an output consistent with respect to synthesis, and it may be
desirable to _abstain_ from outputting anything at all in such cases; that is,
to be a _cautious_ summarizer (Ferri et al., 2004; Hechtlinger et al., 2018).
We consider this strategy in the case of generating narrative synopses of
evidence, as this constitutes a case in which (a) one would very much prefer
not to produce a misleading summary of clinical evidence (Kell et al., 2021),
and, (b) we observe many cases where the diverse decoding strategy yields an
output that seems to communicate (at a granular level) the aggregate findings
expected.

Movie Reviews For movie reviews we use BERT (Devlin et al., 2019), fine-tuned
on IMDB (Maas et al., 2011)151515https://huggingface.co/lvwerra/bert-imdb to
predict the sentiment of each input $x_{ij}$, using the proportion of
$x_{ij}\in X_{i}$ with a positive score as an approximation for the target
sentiment $z_{i\hat{y}}$. For each diverse prediction $\mathcal{C}_{i}$, we
predict a sentiment $\tilde{z}_{i\hat{y}}$ using our sentiment regression
model (Section 2.1), and select the prediction closest to the estimated target
sentiment $|\tilde{z}_{i\hat{y}}-z_{i\hat{y}}|$. We find this improves model
performance to human-like levels in terms of synthesis (Table 4, Figure 6).
Two authors annotated 100 paired instances over PRIMERA generations for
sentiment preference (matching the reference) between standard and diverse
outputs. We find a moderate agreement Cohen’s $\kappa$=0.59, with a
statistically significant preference for the diverse summaries (p=0.003;
Appendix F).

Systematic Reviews. In the case of systematic reviews, we can have only a
binary measure of _significant effect_ (or not). As a proxy for
$z_{i\hat{y}}$, we again use RobotReviewer to extract an effect for each of
the model inputs $x_{ij}$, using the majority vote (i.e., do the plurality of
$x_{ij}\in X_{i}$ indicate that there was an effect). We classify each output
candidate in $\mathcal{C}_{i}$ again using RobotReviewer to estimate
$\tilde{z}_{i\hat{y}}$. We then select for output the highest probability
candidate in $\mathcal{C}_{i}$ which agrees with the majority vote of the
inputs, and abstain where there are no viable candidates. For the models we do
choose a summary for, we find performance similar to our measure (Table 5).
Movie reviews show a wide range of sentiments; systematic reviews show some
improvement but are biased towards no effect (qualitatively observed in
Appendix H).

Figure 6: Differences relative to human summaries under vanilla decoding and
the proposed generate-diverse then select strategy on movie meta-reviews. We
report Pearson’s r and $R^{2}$ as measures of synthesis “calibration”. Vanilla
decoding yields synthesis performance worse than humans, but explicitly
considering synthesis at inference time results in performance comparable to
and sometimes better than human summaries (as best we can measure).

Figure 7: Distributions of outputs for the candiate summaries. Movie reviews (left) show a histogram for the range of differences between lowest and highest output sentiments. Systematic reviews (right) show histograms of the fractions of outputs reporting _significant_ results. | R2 | MSE | Pearson’s r | R1
---|---|---|---|---
LED | 0.656 | 0.032 | 0.821 | 0.229
Pegasus | 0.694 | 0.029 | 0.835 | 0.229
PRIMERA | 0.749 | 0.024 | 0.880 | 0.240
T5 | 0.721 | 0.026 | 0.856 | 0.231
Reference | 0.697 | 0.023 | 0.836 |
LED | 0.763 | 0.022 | 0.878 | 0.227
Pegasus | 0.799 | 0.019 | 0.894 | 0.232
PRIMERA | 0.890 | 0.011 | 0.948 | 0.240
T5 | 0.876 | 0.012 | 0.938 | 0.230
Table 4: Movie Reviews: Generate diverse movie meta-reviews and select among them using an approximate target sentiment (top) or the oracle sentiment (bottom).  | Multiple-then-select | Oracle
---|---|---
| F1 | %Abs. | R1 | Abs. | R1
LED | 0.557 | 0.386 | 0.252 | 0.233 | 0.259
PRIMERA | 0.581 | 0.336 | 0.251 | 0.213 | 0.248
T5 | 0.568 | 0.350 | 0.202 | 0.228 | 0.210
Pegasus | 0.588 | 0.383 | 0.211 | 0.242 | 0.225
Table 5: Systematic Review results with multiple-then-selected predictions. F1
is a macro-averaged F1 on the set of returned results. We abstain when no
output matches the expected synthesis result. Abs. refers to Abstention, R1 to
ROUGE1. Reference F1 is 0.577.

##  6 Related Work

Automatic (multi-document) summarization (Nenkova and McKeown, 2011; Maybury,
1999) has been an active subfield within NLP for decades. We have focused our
analysis on modern, neural abstractive models for conditional text generation
(Bahdanau et al., 2015). In light of their empirical success, we have
specifically evaluated a set of Transformer-based (Vaswani et al., 2017)
models which have recently been used for multi-document summarization (Beltagy
et al., 2020; Zhang et al., 2020; Xiao et al., 2022; Raffel et al., 2020).
There has been some work on highlighting conflicting evidence in health
literature specifically (Shah et al., 2021b, a), though this was focused
primarily on highlighting conflicting evidence, and explicitly aggregating
extracted content.

Sentence fusion One view on synthesis might be that is a particular kind of
_sentence fusion_ (Barzilay and McKeown, 2005). However, past work on “fusing”
sentences has assumed that the aim is to generate an output that contains the
information common to similar sentences (Thadani and McKeown, 2013). This is
intuitive in the context of, e.g., summarizing multiple news articles covering
the same event. But here we are interested in the more challenging setting in
which the output should reflect an aggregate measure of potentially
conflicting evidence or opinions.

Interpretation and analysis of neural models for NLP This work is also related
to the emerging body of work on analyzing neural NLP models, their behaviors,
“knowledge”, and “abilities” in general e.g., Linzen et al. (2016); Tenney et
al. (2019); Petroni et al. (2019); Niven and Kao (2019); Meng et al. (2022).
There has been some work specifically on analyzing neural summarization
models. Xu et al. (2020) investigated when a model is likely to extract (copy)
rather than abstract (generate). Xu and Durrett (2021) furthered this analysis
by assessing when models were relying on the local input to produce particular
output tokens, and when they instead rely mostly on a background language
distribution acquired in pre-training.

Factuality of neural summarizers Neural conditional generation models have
proven adept at producing fluent outputs, but when summarizing they are prone
to _hallucinating_ content unsupported by input documents (Maynez et al.,
2020; Kryscinski et al., 2019). Automated metrics such as ROUGE do not
reliably capture such phenomena (Falke et al., 2019; Maynez et al., 2020).
This has motivated several efforts to design automated factuality metrics; see
Pagnoni et al. (2021) for an overview.

##  7 Conclusions

We have outlined and investigated the problem of _synthesis_ as related to
some summarization tasks. We showed that existing models are partially able to
synthesize implicitly, but do so imperfectly: For instance, the aggregation
they perform is sensitive to input ordering, and they are not as sensitive to
perturbations in the composition of inputs as one would hope. We proposed and
validated a straightforward inference time method to improve model synthesis
capabilities by preferentially outputting summary candidates that align with a
predicted aggregate measure, and demonstrated empirically that this offers
gains in performance. We hope this work encourages additional research into
summarization models that explicitly optimize to accurately synthesize
potentially conflicting evidence.

## Limitations

This work investigates a narrow property in the realm of multi-document
summarization. It focuses solely on sentiment as a measure of synthesis for
movie meta-reviews, and automatically extracted effect findings for biomedical
systematic reviews. While both of these measures are _important_ to
understanding synthesis in these domains, they are not complete: neither
measure covers topicality, fluency, or any other measure of quality. Our
ability to measure the phenomenon of interest is limited by the quality of our
classifiers and our annotation efforts; this may fail in more subtle cases.

Though we have made an extensive effort to fine-tune several popular
summarization models, we are limited to transformer-based models of relatively
modest size (due to the GPU memory required to train long sequence
summarization models). Conceivably these behaviors may change as models scale
in size, or with a different flavor of model architecture. While we believe
these results are more due to model behaviors than the properties of any
particular language (English), this has not been experimentally confirmed.

## Ethics

Beyond limitations of our measurements, we caution against a naive deployment
of the methods introduced in this work. The exact aspects of a synthesis
depend deeply on the relevant domain, and without domain specific measures for
all quality aspects, these solutions may fail in unexpected ways. In general,
we caution against any current deployment of automated synthesis technology
without a human in the loop; evaluation of synthesis methodology is generally
understudied and requires domain expertise to assess quality.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _International Conference on Learning Representations (ICLR)_.
  * Barzilay and McKeown (2005) Regina Barzilay and Kathleen R. McKeown. 2005.  Sentence fusion for multidocument news summarization.  _Computational Linguistics_ , 31(3):297–328.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _ArXiv_ , abs/2004.05150.
  * Dang (2005) Hoa Trang Dang. 2005.  Overview of duc 2005.  In _Document Understand Conference_.
  * Dang (2006) Hoa Trang Dang. 2006.  Overview of DUC.  In _In Proceedings of HLT-NAACL_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * DeYoung et al. (2021) Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. 2021\.  MS^2: Multi-document summarization of medical studies.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7494–7513, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * DeYoung et al. (2020) Jay DeYoung, Eric Lehman, Benjamin Nye, Iain Marshall, and Byron C. Wallace. 2020\.  Evidence inference 2.0: More data, better models.  In _Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing_ , pages 123–132, Online. Association for Computational Linguistics.
  * Evans et al. (2004) David Kirk Evans, Judith L. Klavans, and Kathleen R. McKeown. 2004.  Columbia newsblaster: Multilingual news summarization on the web.  In _Demonstration Papers at HLT-NAACL 2004_ , pages 1–4, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Ferri et al. (2004) César Ferri, Peter Flach, and José Hernández-Orallo. 2004.  Delegating classifiers.  In _Proceedings of the twenty-first international conference on Machine learning_ , page 37.
  * Gholipour Ghalandari et al. (2020) Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020.  A large-scale multi-document summarization dataset from the Wikipedia current events portal.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1302–1308, Online. Association for Computational Linguistics.
  * Hechtlinger et al. (2018) Yotam Hechtlinger, Barnabás Póczos, and Larry Wasserman. 2018.  Cautious deep learning.  _arXiv preprint arXiv:1805.09460_.
  * Kell et al. (2021) Gregory Kell, Iain Marshall, Byron Wallace, and Andre Jaun. 2021.  What would it take to get biomedical QA systems into practice? In _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_ , pages 28–41, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Leone (2020) Stefano Leone. 2020.  Rotten tomatoes movies and critic reviews dataset.  https://www.kaggle.com/datasets/stefanoleone992/rotten-tomatoes-movies-and-critic-reviews-dataset.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam M. Shazeer. 2018.  Generating wikipedia by summarizing long sequences.  _ArXiv_ , abs/1801.10198.
  * Lu et al. (2020) Yao Lu, Yue Dong, and Laurent Charlin. 2020.  Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8068–8074, Online. Association for Computational Linguistics.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Marshall et al. (2017) Iain Marshall, Joël Kuiper, Edward Banner, and Byron C. Wallace. 2017.  Automating biomedical evidence synthesis: RobotReviewer.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 7–12, Vancouver, Canada. Association for Computational Linguistics.
  * Maybury (1999) Mani Maybury. 1999.  _Advances in automatic text summarization_.  MIT press.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual knowledge in gpt.  _arXiv preprint arXiv:2202.05262_.
  * Mollá and Santiago-Martínez (2012) Diego Mollá and María Elena Santiago-Martínez. 2012.  Creation of a corpus for evidence based medicine summarisation.  _The Australasian medical journal_ , 5(9):503.
  * Nan et al. (2021a) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021a.  Entity-level factual consistency of abstractive text summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2727–2733, Online. Association for Computational Linguistics.
  * Nan et al. (2021b) Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021b.  Improving factual consistency of abstractive summarization via question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6881–6894, Online. Association for Computational Linguistics.
  * Nenkova and McKeown (2011) Ani Nenkova and Kathleen McKeown. 2011.  _Automatic summarization_.  Now Publishers Inc.
  * Niven and Kao (2019) Timothy Niven and Hung-Yu Kao. 2019.  Probing neural network comprehension of natural language arguments.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4658–4664, Florence, Italy. Association for Computational Linguistics.
  * Oved and Levy (2021) Nadav Oved and Ran Levy. 2021.  PASS: Perturb-and-select summarizer for product reviews.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 351–365, Online. Association for Computational Linguistics.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 115–124, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _ArXiv_ , abs/1910.10683.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  QuestEval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Shah et al. (2021a) Darsh Shah, Lili Yu, Tao Lei, and Regina Barzilay. 2021a.  Nutri-bullets hybrid: Consensual multi-document summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5213–5222, Online. Association for Computational Linguistics.
  * Shah et al. (2021b) Darsh J Shah, Lili Yu, Tao Lei, and Regina Barzilay. 2021b.  Nutri-bullets: Summarizing health studies by composing segments.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13780–13788.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Tenney et al. (2019) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019\.  What do you learn from context? probing for sentence structure in contextualized word representations.  In _International Conference on Learning Representations_.
  * Thadani and McKeown (2013) Kapil Thadani and Kathleen McKeown. 2013.  Supervised sentence fusion with single-stage inference.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 1410–1418.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vijayakumar et al. (2016) Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016.  Diverse beam search: Decoding diverse solutions from neural sequence models.  _arXiv preprint arXiv:1610.02424_.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272.
  * Wallace et al. (2020) Byron C. Wallace, Sayantani Saha, Frank Soboczenski, and Iain James Marshall. 2020\.  Generating (factual?) narrative summaries of rcts: Experiments with neural multi-document summarization.  _ArXiv_ , abs/2008.11293.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wolhandler et al. (2022) Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Dagan. 2022.  How "multi" is multi-document summarization?
  * Xiao et al. (2022) Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022.  PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5245–5263, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2020) Jiacheng Xu, Shrey Desai, and Greg Durrett. 2020.  Understanding neural abstractive summarization models via uncertainty.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6275–6281, Online. Association for Computational Linguistics.
  * Xu and Durrett (2021) Jiacheng Xu and Greg Durrett. 2021.  Dissecting generation modes for abstractive summarization models via ablation and attribution.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6925–6940, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  _ArXiv_ , abs/1912.08777.

Variable | Definition
---|---
$X_{i}$ | The $i^{th}$ set of input documents, corresponding to instance $i$
$y_{i}$ | The target summary $y$ of the $i^{th}$ instance
$y_{i,t}$ | The $t^{th}$ token of target $y_{i}$
$\hat{y}_{i}$ | A generated summary of the $i^{th}$ instance
$x_{ij}$ | The $j^{th}$ input document for instance $i$
$x^{\oplus}_{i}$ | A particular linearization of the input documents $X_{i}$
$\theta$ | Model & parameters
$p_{\theta}$ | Probabily under parameters $\theta$
$p_{\theta}(y_{i,t}|y_{i,1..t-1},x_{i}^{\oplus})$ | Standard auto-regressive prediction of the next token given an
| input and partial summary
$z_{ij}$ | Latent property (sentiment, significance finding) of $x_{ij}$
$z_{i}$ | Aggregated latent property (sentiment, significance finding) of $X_{i}$
$z_{i\hat{y}}$ | Latent property measured over summary $\hat{y}$
$G$ | Aggregation function over latent properties $z_{ij}$, yields $z_{i}$
$g$ | Auxillary function to measure latent property $z_{ij}$ of $x_{ij}$, or $z_{i\hat{y}}$ of $\hat{y}$
$\alpha_{ij}$ | A weight for $z_{ij}$
Table 6: Notation.

##  Appendix A Additional Related Work

Shah et al. (2021a) created a “nutri-bullets” system for generating consensus-
based summaries of health and nutrition related content. They assume a low-
supervision setting in which one has a set of tuples extracted from texts with
which to train a content extractor, and where one can design heuristic rule-
based aggregation strategies on top of extracted tuples mapping onto discrete
categories like “consensus”. By contrast, we assume a more typical supervised
summarization setting and are interested in continuous aggregation of a latent
attribute of interest, and we do not assume (or have access to) relational
tuples over inputs. Indeed, recent work Wolhandler et al. (2022) has shown
that systematic reviews are categorically different than news summarization,
and that relational tuple extractors do not perform well in the medical
domain.

First, Shah et al. (2021a)’s focus primarily on settings in which training
data is (severely) limited, and motivate their pipeline approach on the basis
of this limited supervision assumption. For this reason they define separate
modules: The first performs content selection (tuple extraction; this does
require manual annotations of tuples on a subset of texts to train such an
extractor); The second applies (manually composed) deterministic aggregation
rules over these extracted tuples to combine them; a final module then
generates a “surface realization” conditioned on the aggregated result.

We have investigated more typical supervised settings (with thousands of input
and summary pairs), and we are training modern end-to-end transformer-based
summarization models. We have empirically assessed the extent to which model
outputs in this typical training regime are consistent with the continuous
synthesis result anticipated. We do not have annotated tuples on our inputs
(as would be required to use the Shah et al. (2021a) approach, as it assumes a
trained content extractor module). And while applying discrete (manually
composed) aggregation operators over inputs makes sense in some settings, we
are explicitly interested in the ability of models to aggregate variables of
interest continuously, for example producing “very positive” summaries when
movie reviews are overwhelmingly positive, and merely “positive” summaries
when they are only mostly positive.

In sum, the approach proposed by Shah et al. (2021a) is appropriate in, and
designed for, low-supervision settings (which we do not consider here) where
there are natural “tuples” to be extracted from inputs and supervision for
this sub-task (which we do not have) and where discrete aggregation categories
of inputs is natural (whereas we are interested in continuous aggregation,
e.g., mean sentiment).

Wolhandler et al. (2022) attempts to measure how challenging multi-document
summarization is, as a function of the unique knowledge (represented as
relational tuples) required to produce a summary. This work measures how many
new tuples each input document might add in contrast to subsets of other
inputs. By greedily building subsets of inputs as a function of new
information added, they find that standard multiple document summarization
datasets merely need to select two to four documents from inputs of up to ten,
whereas their approach breaks down in the case of systematic reviews. They
find that due to both technical constraints for relation extraction, as well
as the inability to model contradiction, relational extraction and aggregation
methods are insufficient for producing evidence syntheses.

Oved and Levy (2021) introduce the Perturb and Selection Summarizer (PASS)
system for summarizing Amazon product reviews. It works by perturbing model
inputs (i.e. keep random subsets of the input), generating a summary for each
perturbation (via standard beam search), and then selecting amongst outputs
(via a ranker) to produce a coherent, self-consistent, and fluent summary.

PASS is similar to our work in that it generates multiple outputs and selects
amongst them. However it differs in several key respects. The key conceptual
difference between PASS and our work is that PASS’s target is a summary’s
self-consistency (a product review might contradict itself on some aspect,
e.g. simultaneously discussing a product fitting well in addition to the
product running a size small), whereas our target is a continuous fact derived
from the inputs as a whole (e.g. aggregate sentiment or effect sizes). PASS is
designed to produce summaries that are plausible, as opposed (and
complementary) to summaries that reflect inherent contradiction in the input
data. As PASS produces summaries from subsets of each instance’s input, it
cannot perform an explicit synthesis on its own, as opposed to our work,
wherein each summary was produced with access to the whole of each instance’s
input.

##  Appendix B Models

We train all models using a modified Huggingface Transformers libraryWolf et
al. (2020). For the Pegasus model, we use a distilled version provided by
Huggingface (Table 7). All models were trained using their default
hyperparameters, except for batch size, optimization steps, learning rates,
and any parameters specified in Table 7. We fix our batch size to 16, using
gradient accumulation over single instances at a time, with floating point 16
(fp16) precision (due to data size), and perform an approximate (subject to
resource constraints) grid search over learning rates and training steps
(Tables 8, 9), keeping the model highlighted in bold. Earlier experimentation
was performed ad-hoc with Longformer and T5 models only; we found that while
lower numbers of steps could perform well, they had high variance and were
more sensitive to hyperparameter changes than longer runs. All training was
performed on 48G NVIDIA RTX8000 GPUs, most models are unable to fit single
instance gradient information into fewer than 40G, even at reduced precision.

Model | Huggingface Checkpoint | Optimizer | Schedule | Warmup | Smoothing
---|---|---|---|---|---
LED | allenai/led-base-16384 | Adam | P.nomial/decay 0.01 | 50 steps | 0.1
T5 | t5-base | Adam | Linear | 50 steps | 0
PRIMERA | allenai/PRIMERA | Adam | Linear | 50 steps | 0
Pegasus | sshleifer/distill-pegasus-cnn-16-4 | Adafactor | Linear | 500 steps | 0.1
Table 7: Model hyperparameters. We used optimizers, schedulers, weight decay, and label smoothing as best according to examples from source implementations (where available). Optimizer warmup was arbitrarily chosen. Non-specified parameters were the Huggingface defaults.  model | steps | lr | ROUGE1 | ROUGE2 | ROUGEL
---|---|---|---|---|---
led | 1000 | 1e-5 | 25.09 | 8.95 | 20.14
led | 1000 | 1e-6 | 25.25 | 8.33 | 19.48
led | 1000 | 3e-5 | 25.33 | 8.65 | 19.87
led | 1000 | 5e-5 | 25.14 | 8.35 | 19.89
led | 5000 | 1e-5 | 25.46 | 8.63 | 19.76
led | 5000 | 1e-6 | 25.31 | 8.76 | 20.12
led | 5000 | 3e-5 | 24.50 | 7.49 | 19.02
led | 5000 | 5e-5 | 23.99 | 7.07 | 17.61
led | 10000 | 1e-5 | 24.28 | 7.60 | 19.25
led | 10000 | 1e-6 | 25.58 | 8.64 | 20.32
led | 10000 | 3e-5 | 25.60 | 7.97 | 19.59
pegasus | 1000 | 1e-3 | 23.49 | 7.25 | 17.93
pegasus | 1000 | 1e-4 | 22.25 | 7.23 | 17.67
pegasus | 1000 | 1e-5 | 18.95 | 4.23 | 14.34
pegasus | 1000 | 1e-6 | 18.28 | 3.10 | 13.08
pegasus | 2500 | 1e-3 | 26.44 | 9.30 | 20.45
pegasus | 2500 | 1e-4 | 26.50 | 10.81 | 20.91
pegasus | 2500 | 1e-5 | 24.98 | 10.17 | 19.75
pegasus | 2500 | 1e-6 | 23.02 | 7.92 | 18.23
pegasus | 5000 | 1e-3 | 24.05 | 7.75 | 18.98
pegasus | 5000 | 1e-4 | 27.41 | 10.26 | 21.72
pegasus | 5000 | 1e-5 | 25.67 | 9.86 | 20.28
pegasus | 5000 | 1e-6 | 23.57 | 8.74 | 18.77
pegasus | 10000 | 1e-3 | 23.18 | 7.17 | 17.19
pegasus | 10000 | 1e-4 | 27.42 | 9.53 | 21.05
pegasus | 10000 | 1e-5 | 25.85 | 10.25 | 20.41
pegasus | 10000 | 1e-6 | 24.41 | 9.88 | 19.72
primera | 2500 | 1e-4 | 23.32 | 7.02 | 18.10
primera | 2500 | 1e-5 | 25.12 | 8.39 | 19.52
primera | 2500 | 1e-6 | 24.92 | 8.48 | 19.93
primera | 5000 | 1e-4 | 24.35 | 7.40 | 18.49
primera | 5000 | 1e-5 | 25.42 | 8.44 | 19.81
primera | 5000 | 1e-6 | 25.32 | 8.75 | 20.06
primera | 10000 | 1e-4 | 23.57 | 7.24 | 17.89
primera | 10000 | 1e-5 | 24.27 | 7.59 | 18.55
primera | 10000 | 1e-6 | 25.39 | 8.66 | 20.12
t5 | 1000 | 1e-4 | 25.24 | 9.13 | 19.67
t5 | 1000 | 1e-5 | 24.31 | 7.87 | 19.30
t5 | 1000 | 1e-6 | 22.39 | 6.62 | 17.90
t5 | 1000 | 5e-5 | 25.06 | 8.65 | 19.96
t5 | 2500 | 1e-4 | 25.82 | 8.46 | 19.59
t5 | 2500 | 1e-5 | 24.94 | 8.36 | 19.61
t5 | 2500 | 1e-6 | 23.82 | 7.59 | 19.09
t5 | 2500 | 5e-5 | 25.57 | 8.47 | 19.71
t5 | 5000 | 1e-4 | 25.11 | 8.17 | 19.84
t5 | 5000 | 1e-5 | 25.07 | 8.57 | 19.55
t5 | 5000 | 1e-6 | 23.87 | 7.99 | 19.36
t5 | 5000 | 5e-5 | 24.47 | 8.40 | 19.53
t5 | 7500 | 1e-4 | 24.33 | 7.58 | 18.93
t5 | 7500 | 1e-5 | 25.67 | 8.75 | 19.92
t5 | 7500 | 1e-6 | 24.17 | 7.73 | 19.43
t5 | 7500 | 5e-5 | 25.66 | 8.64 | 19.59
t5 | 10000 | 1e-4 | 24.41 | 7.78 | 18.89
t5 | 10000 | 1e-5 | 25.73 | 8.91 | 20.10
t5 | 10000 | 1e-6 | 24.29 | 7.98 | 19.31
t5 | 10000 | 5e-5 | 25.09 | 7.89 | 19.19
Table 8: Movie Reviews dev training results, best models bolded. Model | steps | lr | ROUGE1 | ROUGE2 | ROUGEL
---|---|---|---|---|---
led | 250 | 5e-5 | 24.30 | 6.70 | 18.68
led | 500 | 5e-5 | 23.99 | 6.91 | 17.09
led | 1000 | 5e-5 | 25.21 | 7.31 | 18.54
led | 2500 | 5e-5 | 26.05 | 6.52 | 17.65
led | 5000 | 5e-5 | 30.96 | 8.65 | 20.33
pegasus | 250 | 1e-4 | 20.22 | 5.64 | 15.86
pegasus | 500 | 1e-4 | 21.66 | 6.71 | 16.92
pegasus | 1000 | 1e-4 | 21.87 | 6.67 | 16.79
pegasus | 2500 | 1e-4 | 22.44 | 6.71 | 17.24
pegasus | 5000 | 1e-4 | 22.66 | 5.69 | 16.87
primera | 250 | 1e-4 | 23.21 | 7.68 | 17.96
primera | 500 | 1e-4 | 22.80 | 5.72 | 16.39
primera | 1000 | 1e-4 | 26.08 | 6.70 | 16.77
primera | 2500 | 1e-4 | 27.60 | 6.99 | 18.16
primera | 5000 | 1e-4 | 27.70 | 6.96 | 18.02
primera | 250 | 5e-5 | 22.52 | 6.90 | 17.98
primera | 500 | 5e-5 | 24.22 | 6.11 | 17.53
primera | 1000 | 5e-5 | 24.53 | 6.71 | 17.01
primera | 2500 | 5e-5 | 27.64 | 6.89 | 18.79
primera | 5000 | 5e-5 | 28.46 | 6.59 | 18.08
t5 | 250 | 5e-5 | 23.80 | 7.09 | 18.58
t5 | 500 | 5e-5 | 22.77 | 6.99 | 18.30
t5 | 1000 | 5e-5 | 0.00 | 0.00 | 0.00
t5 | 2500 | 5e-5 | 0.00 | 0.00 | 0.00
Table 9: Systematic Reviews dev training results, best models bolded. We
experimented with other parameters (in particular learning rates), and found
that total number of steps was more important.

##  Appendix C Detailed Results

Measure Validation As our results rely on using proxy metrics, we measure the
quality of these proxies. See Figure 8 for movie meta-review sentiment
correlation with human results, and Table 10 for how well the automatic
significance measures correlate with the underlying truth.

Figure 8: Actual sentiment vs. predicted sentiments on model outputs. | Precision | Recall | F1-score | Support
---|---|---|---|---
NSD | 0.726 | 0.870 | 0.792 | 247
Sig. Diff. | 0.500 | 0.283 | 0.362 | 113
Accuracy |  |  | 0.686 | 360
Macro avg | 0.613 | 0.577 | 0.577 | 360
Table 10: Systematic review significance validation results. NSD: No
Significant Difference. Sig. Diff.: Significant Difference

Figure 9: Entropy of instance predictions. Broken out by whether or not the
underlying truth is _not significant_ (left); or the whole dataset (right)

Diversity Sampling. We include detailed results for the importance of
diversity sampling; the diversity sampling procedure produces better metrics
in every dimension (Table 11 top left vs. bottom left.). In the systematic
reviews, most metrics drop slightly and abstention increases substantially
(Table 12).

| R2 | MSE | Pearson’s r | R1
---|---|---|---|---
LED | 0.656 | 0.032 | 0.821 | 0.229
Pegasus | 0.694 | 0.029 | 0.835 | 0.229
PRIMERA | 0.749 | 0.024 | 0.880 | 0.240
T5 | 0.721 | 0.026 | 0.856 | 0.231
Reference | 0.697 | 0.023 | 0.836 |

| R2 | MSE | Pearson’s r | R1
---|---|---|---|---
LED | 0.711 | 0.027 | 0.847 | 0.240
Pegasus | 0.705 | 0.028 | 0.840 | 0.247
PRIMERA | 0.731 | 0.025 | 0.857 | 0.255
T5 | 0.669 | 0.031 | 0.819 | 0.253
Reference | 0.697 | 0.023 | 0.836 |

| R2 | MSE | Pearson’s r | R1
---|---|---|---|---
LED | 0.653 | 0.033 | 0.815 | 0.241
PEGASUS | 0.649 | 0.033 | 0.809 | 0.248
PRIMERA | 0.685 | 0.029 | 0.833 | 0.254
T5 | 0.615 | 0.036 | 0.786 | 0.252
Reference | 0.697 | 0.023 | 0.836 |

| R2 | MSE | Pearson’s r | R1
---|---|---|---|---
LED | 0.763 | 0.022 | 0.878 | 0.227
Pegasus | 0.799 | 0.019 | 0.894 | 0.232
PRIMERA | 0.890 | 0.011 | 0.948 | 0.240
T5 | 0.876 | 0.012 | 0.938 | 0.230
Reference | 0.697 | 0.023 | 0.836 |

Table 11: Movie Reviews. First: Generate 5 diverse movie meta-reviews and then choose among them using an approximate target sentiment. Second: Generate 25 diverse movie meta-reviews and then choose among them using an approximate target sentiment; this was accidentally referenced in an earlier version of this work. Third: Generate 5 movie meta-reviews using standard beam search and choose among them using an approximate target sentiment. Fourth: Generate 5 diverse movie meta-reviews and select amongst them using the oracle sentiment. In all cases R1 refers to ROUGE1. | F1 | Abstention | ROUGE1 | Abstention-Oracle | ROUGE1-Oracle
---|---|---|---|---|---
LED | 0.521 | 0.503 | 0.258 | 0.358 | 0.263
PRIMERA | 0.551 | 0.464 | 0.256 | 0.342 | 0.248
T5 | 0.546 | 0.422 | 0.204 | 0.328 | 0.211
Pegasus | 0.589 | 0.469 | 0.211 | 0.281 | 0.222
Reference | 0.577 |  |  |  |
Table 12: Systematic reviews results with multiple generate-then-select
predictions, this time using the top-5 results from standard beam-search. F1
is a macro-averaged F1 on the set of returned results. We abstain when no
output matches the expected synthesis result.

##  Appendix D Rouge Results

We report mean differences in ROUGE outputs for both datasets in Figure 10.
Ideally, these would have all mass at zero.

Figure 10: ROUGE1 deviations from instance means for movie reviews (left) and
systematic reviews (right).

##  Appendix E Constrained Beam Search

We introduce an alternative post-hoc decoding scheme which constrains trained
models to only produce summaries with a desired overall attribute $z_{i}$ for
input $X_{i}$. _Constrained Decoding_ is a modification of the beam search
procedure: rather than considering every possible token when performing next-
token-prediction, we restrict models to only those tokens that are consistent
with the desired aggregate target attribute.

Specifically, when generating $\hat{y}_{i,t}$, we filter the top-$n$ most
likely tokens (according to $p_{\theta}(y_{i,t}|y_{i,1..t-1},x_{i}^{\oplus})$)
using classifier $g$ over this attribute. Once we have
$g(\hat{y}_{i,1,..,t})$, we exclude candidate tokens $y_{i,t}$ that are
inconsistent with the target attribute $z_{i}$:
$\left|g(\hat{y}_{i,1,..,t})-z_{i}\right|>\epsilon$. We apply this method for
$t>3$, allowing the first three tokens to establish candidate beams. When all
candidates would be eliminated, we do not filter any tokens.

We evaluate this method on PRIMERA, using a sentiment estimate derived from
the inputs for our target attribute $z_{i}$ as our factor. We perform a
hyperparameter sweep over $n$ and $\epsilon$. Ordinarily we would simply
choose the setting with the highest R2, however as this decoding process can
introduce disfluencies, we also consider ROUGE1 scores when choosing our
hyperparameter setting (Tables 15, 16).

While the Constrained Decoding method can produce a higher absolute
performance (Table 13), the summaries are not generally preferred by humans
(Table 17), as they often contain obvious disfluencies. We therefore did not
explore this approach further.

| R2 | MSE | Pearson’s r | ROUGE1
---|---|---|---|---
Base | 0.614 | 0.035 | 0.784 | 0.258
Constrained | 0.807 | 0.017 | 0.903 | 0.239
Diverse (target) | 0.749 | 0.024 | 0.880 | 0.240
Diverse (oracle) | 0.890 | 0.011 | 0.948 | 0.240
Table 13: Movie Reviews. Performance of the base, constrained, and diverse PRIMERA models on the Rotten Tomatoes set. The highest score for each metric is bolded. | $\epsilon$ | $n$ | R2 | MSE | Pearson’s r | ROUGE1
---|---|---|---|---|---|---
LED | 0.2 | 100 | 0.797 | 0.018 | 0.898 | 0.229
PRIMERA | 0.2 | 100 | 0.807 | 0.017 | 0.903 | 0.239
T5 | 0.3 | 100 | 0.674 | 0.030 | 0.822 | 0.246
PEGASUS | 0.2 | 100 | 0.793 | 0.019 | 0.894 | 0.254
Table 14: Movie Reviews. Performance of every model using constrained decoding on the Rotten Tomatoes set. The highest score for each metric is bolded. The hyperparameters for each model for the number of candidates, $n$, and the threshold, $\epsilon$, is included. $n$ | R2 | MSE | Pearson’s r | ROUGE1
---|---|---|---|---
5 | 0.815 | 0.018 | 0.914 | 0.250
10 | 0.805 | 0.018 | 0.906 | 0.250
25 | 0.813 | 0.017 | 0.911 | 0.250
50 | 0.807 | 0.018 | 0.909 | 0.253
100 | 0.818 | 0.017 | 0.915 | 0.252
Table 15: Movie Reviews. Performance of constrained PRIMERA when varying the number of candidates, $n$, and setting the threshold to $\epsilon=0.2$. $\epsilon$ | R2 | MSE | Pearson’s r | ROUGE1
---|---|---|---|---
0.1 | 0.855 | 0.014 | 0.947 | 0.241
0.2 | 0.818 | 0.017 | 0.915 | 0.252
0.3 | 0.723 | 0.026 | 0.860 | 0.249
0.4 | 0.755 | 0.023 | 0.874 | 0.242
0.5 | 0.776 | 0.021 | 0.886 | 0.257
0.6 | 0.704 | 0.028 | 0.844 | 0.256
0.7 | 0.675 | 0.031 | 0.824 | 0.259
0.8 | 0.689 | 0.029 | 0.834 | 0.259
0.9 | 0.688 | 0.029 | 0.834 | 0.259
1.0 | 0.688 | 0.029 | 0.834 | 0.259
Table 16: Movie Reviews. Performance of constrained PRIMERA when varying the
threshold $\epsilon$ and setting the number of candidates to $n=100$.

##  Appendix F Annotation

We conduct an annotation exercise over a subset of PRIMERA test-set outputs.
We order the moves by absolute difference in automatically extracted
sentiment, and taking the top-20%, subsample 100 movies for evaluation. We
present the reference summary to two authors, along with the base summary and
diverse beam search summary. We randomize the order of appearance of summaries
and blind the annotators to the selection (Figure 11). We report results of
this annotation in Table 17. We find a high-moderate agreement for
Cohen-$\kappa$ (computed via Pedregosa et al. (2011)) for the Diverse Beam
Search Strategy, and a low-moderate agreement for the Constrained decoding
method. We perform a binomial t-test Virtanen et al. (2020) and find the
annotators’ preference is significant at a p=0.05 level.

To assess the impact of disfluencies on author sentiment preferences, one
author conducted a (blinded) coarse fluency evaluation, assessing whether each
summary was almost error free, containing a small number of number of errors,
or entirely disfluent. Both the base and diverse decoding schemes yielded
mostly fluent summaries, while the constrained decoding scheme produced a
substantially larger number of disfluent summaries (Table 18).

Strategy | $\kappa$ | Pref. Base | Pref. Alt. | Excluded | p-val
---|---|---|---|---|---
Diverse | .594 | 73 | 115 | 12 | .0026
Constr. | .431 | 96 | 93 | 11 | .8843
Table 17: Annotation Results. We report a Cohen-$\kappa$, the total number of times the base-decoded summary was preferred over the alternative summary (either a result of Diverse Beam Search or Constrained Decoding), the number of instances excluded for having either a subject error or for not having a clear preference, and finally a significance value for a binomial t-test. Figure 11: Annotation example as presented to the annotators. Figure 12: Fluency annotation example as presented to the annotators. Figure 13: Fluency annotation example as presented to the annotators. | Base | Diverse | Constrained
---|---|---|---
Fluent, almost no errors | 186 | 178 | 151
Some error, still somewhat understandable | 9 | 17 | 35
Not fluent at all | 3 | 3 | 12
Table 18: Annotation Results. We report the fluency for the Base, Diverse Beam
Search, and Constrained Decoding summaries over the subset annotated for
sentiment.

##  Appendix G Additional Experiments

% Pos. | Phrase
---|---
0 - 20% | Reviews were overwhelmingly negative.
20 - 40% | Reviews were mostly negative.
40 - 60% | Reviews were mixed.
60 - 80% | Reviews were mostly positive.
80 - 100% | Reviews were overwhelmingly positive.

Significance | Phrase
---|---
None | The studies found no significant effect.
Sig. & $<$ 80% | The studies found a significant effect.
Sig. & $>$ 80% | The studies found a strong significant effect.

Table 19: Left: Sentiment phrases based on the percent of input reviews
expressing a positive sentiment. Right: Systematic review phrases based on the
significance of the effect, and if the effect is significant, is the effect in
the 80th percentile or higher?

A natural question to ask is whether a model can directly produce a coarse
synthesis textual summary. We ask: can a model identify approximate sentiment
(for movie reviews) or approximate effect strength (for systematic reviews in
addition to a more nuanced textual summary? We divide movie reviews into five
categories based on the percent of positive reviews, and systematic reviews
into three categories based on meta-analysis result. We retrain our best model
configurations, prefixing the targets with a sentence indicating the overall
synthesis result (Table 19).

Model | R2 | MSE | Pearson’s r | ROUGE1 | % Agree
---|---|---|---|---|---
LED | 0.531 | 0.044 | 0.729 | 0.235 | 0.594
PRIMERA | 0.552 | 0.042 | 0.743 | 0.250 | 0.629
T5 | 0.510 | 0.046 | 0.716 | 0.253 | 0.613
Pegasus | 0.529 | 0.044 | 0.729 | 0.244 | 0.622
Reference | 0.697 | 0.023 | 0.836 |  |
LED | 0.547 | 0.042 | 0.740 | 0.227 |
PRIMERA | 0.486 | 0.048 | 0.705 | 0.198 |
Pegasus | 0.290 | 0.066 | 0.541 | 0.181 |
Table 20: Sentiment Results with modified targets. ROUGE1 scores are for
generated reviews, with any generated coarse synthesis sentence removed. %
Agree is the fraction of coarse syntheses matching the target sentence. Top:
Generate as normally. Bottom: Generate (teacher-forcing) given the coarse
synthesis as a prefix.

##  Appendix H Examples of Diverse Generations

Summary | Sentiment
---|---
The Private Lives of Pippa Lee relies on a strong ensemble cast to deliver witty and poignant observations about life and relationships. | 0.800731
The Private Lives of Pippa Lee relies on a strong ensemble cast to deliver witty and poignant observations about life and relationships. | 0.800731
With a strong cast and Robin Wright Penn’s sharp performance, The Private Lives of Pippa Lee succeeds as both a witty tribute to lost characters and a showcase for Robin Wright Penn. | 0.809596
With a strong cast and Robin Wright Penn’s empathetic direction, The Private Lives of Pippa Lee succeeds as both a humorous look at domestic issues and a poignant look at relationships. | 0.809081
The Private Lives of Pippa Lee benefits from Robin Wright Penn’s superb performance, as well as a strong ensemble cast that includes Keanu Reeves, and Faye Dunaway. | 0.845693
The Private Lives of Pippa Lee has an affecting ensemble cast and Robin Wright Penn delivers a noteworthy performance, although the film is a bit too episodic. | 0.654905
Table 21: Different meta-reviews of "The Private Lives of Pippa Lee" and corresponding sentiments. The target sentiment for this meta-review is 70%, generating diverse candidates helps find a meta-review closer to the target. Summary | Sentiment
---|---
You Don’t Mess With the Zohan’s handful of laughs are almost enough to compensate for its inconsistent tone and stale, obvious jokes. | 0.242698
You Don’t Mess with the Zohan has a handful of crotch thrusts, but not enough of them land. | 0.429654
You Don’t Mess With the Zohan’s handful of laughs are almost enough to compensate for its aimless, crass script. | 0.287896
You Don’t Mess with the Zohan has its moments, but not all of them – and the jokes are embarrassingly crass and often crude. | 0.434442
You Don’t Mess with the Zohan has its moments, but not all of them – and the jokes are embarrassingly crass and often crude. The script | 0.406172
Table 22: Different meta-reviews for "You Don’t Mess With The Zohan"; a relatively panned movie with a target meta-review sentiment of 37%. Generated | Effect
---|---
Ketanserin versus placebo in the Raynaud’s phenomenon is neither effective nor safe. The Raynaud’s phenomenon is associated with significant adverse effects including dizziness and pain. The effectiveness of ketanserin for the Raynaud’s phenomenon is unknown. | no significant difference
Ketanserin versus placebo in the Raynaud’s phenomenon is neither effective nor safe. The Raynaud’s phenomenon is associated with significant adverse effects including dizziness and pain. | no significant difference
Ketanserin and serotonin receptor antagonists in the Raynaud’s phenomenon treatment of systemic scleroderma reduce the incidence of ischaemic ulcers and may reduce the frequency of adverse events. | significant difference
The Raynaud’s phenomenon is associated with a small number of adverse effects when administered orally to patients with Raynaud’s phenomenon. The frequency of Raynaud’s phenomenon is similar to that of other drugs. However, there is little evidence to aid the treatment of Raynaud’s phenomenon. | no significant difference
The Raynaud’s phenomenon is associated with a small number of adverse effects when administered orally to patients with Raynaud’s phenomenon. The frequency of Raynaud’s phenomenon is similar to that of other drugs. | no significant difference
Table 23: An instance where generating multiple reviews allows our models to find a candidate summary reporting a significant difference (the target).  Generated | Effect
---|---
The overall evidence supports the use of topical antibiotics in surgical patients who have undergone minor surgery, compared to no treatment. The effect on other outcomes, other than infection rate, is consistent. The safety profile of topical antibiotics is also of concern. Further well-designed RCTs are needed to assess effectiveness of topical antibiotics in surgical patients. | no significant difference
A single application of topical antibiotics in surgical site wounds reduces the risk of infection, and the risk of other complications, including wound dehiscence. The risk of infection recurrence is low. The use of topical antibiotics outside of surgery should be restricted to surgical site wounds. | no significant difference
A single application of topical antibiotics in surgical site wounds reduces the risk of infection, and the risk of other complications, including wound dehiscence. The risk of infection recurrence is low. | no significant difference
The overall evidence supports the use of topical antibiotics in surgical patients to reduce the risk of infection, and the risk of other complications, especially in high-risk patients. There is a lack of evidence in low-risk patients to support the use of topical antibiotics in this setting. | significant difference
A single application of topical antibiotics in surgical site infection prevention has been demonstrated to reduce the risk of infection in patients who have undergone surgery. The number of patients who have been treated with topical antibiotics has been small but this is due to risk of bias in the trials. Ointment use should be limited to patients whose primary wound is irradiated. | significantly difference
Table 24: An instance where generating multiple reviews allows our models to
find a candidate summary reporting a significant difference (the target).