Thus, such a court structure represents a set of Kings as well as their
witnesses. Denote by $C$ the set of all court structures for all sets
$\Gamma_{K}$ of unary types.

For each ${\cal A}$ with ${\cal A}\models\varphi$, the _court structure
induced by ${\cal A}$_ is the substructure of ${\cal A}$ induced by the Kings
of ${\cal A}$ as well as their witnesses.

Our goal is to construct formulas $\varphi_{\cal C}$ in spread normal form for
each ${\cal C}\in C$ such that the following conditions are equivalent:

  1. (1)

$\varphi$ has a (finite) model with court structure ${\cal C}$.

  2. (2)

$\varphi_{\cal C}$ has a (finite) model.

Then the statement of the proposition follows by observing that $\varphi$ has
a (finite) model if and only if $\bigvee_{{\cal C}\in C}\varphi_{\cal C}$ has
a (finite) model.

Thus we aim at constructing, for each ${\cal C}\in C$, a formula
$\varphi_{\cal C}$ in spread normal form with the above property. Recall that
a formula is in spread normal form, if it is of the form

$\displaystyle\quad\bigwedge_{\theta\in Z}\exists x\theta(x)\wedge\forall
x\forall y(x=y\vee\nu(x,y))$ $\displaystyle\wedge$
$\displaystyle\quad\bigwedge_{k=0}^{2}\bigwedge_{h=1}^{n}\forall x\exists
y(\lambda_{k}(x)\rightarrow(\lambda_{\lfloor
k+1\rfloor}(y)\wedge\mu_{h}(y)\wedge\theta_{h}(x,y)))$

where (i) $Z$ is a set of unary pure Boolean formulas; (ii) $\nu$,
$\theta_{1},\ldots,\theta_{n}$ are quantifier- and equality-free formulas;
(iii) $\lambda_{0}$, $\lambda_{1}$, and $\lambda_{2}$ are mutually exclusive
unary pure Boolean formulas; and (iv) $\mu_{1},\ldots,\mu_{n}$ are mutually
exclusive unary pure Boolean formulas.

The formula $\varphi_{\cal C}$ ensures a set of conditions on models. Suppose
that ${\cal C}$ has the elements $a_{1},\ldots,a_{t}$ ordered in such a way
that $a_{1},\ldots,a_{s}$ are its Kings and $a_{s+1},\ldots,a_{t}$ are its
non-King elements.

A first set of conditions, encoded in formulas $\delta_{1}$ and $\delta_{2}$,
ensures that the court structure of models of $\varphi_{\cal C}$ are
isomorphic to ${\cal C}$. To this end, let $Q_{1},\ldots,Q_{t}$ be fresh unary
relation symbols.

The formula $\delta_{1}$ ensures that each such relation $Q_{i}$ contains
exactly one element (intended to be the $i$th element $a_{i}$ of ${\cal C}$):

$\bigwedge_{i=1}^{t}\exists xQ_{i}(x)\;\wedge\;\forall x\forall
y\Big{(}x=y\vee\bigwedge_{i=1}^{t}\neg(Q_{i}(x)\wedge Q_{i}(y))\Big{)}$

The formula $\delta_{2}$ ensures that the substructure induced by elements in
the $Q_{i}$s is isomorphic to ${\cal C}$:

$\bigwedge_{i=1}^{t-1}\bigwedge_{j=i+1}^{t}\forall x\forall
y\Big{(}x=y\vee((Q_{i}(x)\wedge Q_{j}(y))\rightarrow\textsc{tp}^{\cal
C}[a_{i},a_{j}])\Big{)}$

A second set of conditions, encoded in formulas $\gamma_{1}$, $\gamma_{2}$,
and $\gamma_{3}$ ensures that witnesses are spread. To this end, let
$Q_{h,1},\ldots,Q_{h,s}$ be fresh unary relation symbols. The intuition is
that $Q_{h,i}$ shall contain all elements $a$ such that the $\psi_{h}$-witness
of $a$ is the $i$th King. For spreading non-King witnesses, further fresh
unary relation symbols $W_{1},\ldots,W_{m}$ and $N_{0},N_{1},N_{2}$ are used.

Witnesses for an element $a$ will then be ensured by distinguishing between
King witnesses, handled by $\gamma_{2}$, and non-King witnesses, handeled by
$\gamma_{3}$.

The formula $\gamma_{1}$ enforces that each element is either a King, or
occurs in exactly one $W_{i}$ and in exactly one $N_{i}$:

$\forall x\forall y\Big{(}x=y\vee\bigvee_{i=1}^{s}Q_{i}(x)\vee\mu(x)\Big{)}$

Here, $\mu(x)$ is a formula stating that $x$ occurs in exactly one $W_{i}$ and
in exactly one $N_{i}$. (Note that the quantification of $y$ is not really
necessary, but makes it easy to see that $\gamma_{2}$ adheres to the
conditions for spread normal form.)

The formula $\gamma_{2}$ ensures that King witnesses are present:

$\bigwedge_{i=1}^{s}\bigwedge_{h=1}^{m}\forall x\forall
y\Big{(}x=y\vee((Q_{h,i}(x)\wedge Q_{i}(y))\rightarrow\psi_{h}(x,y))\Big{)}$

Finally, the formula $\gamma_{3}$ ensures that each element has its non-King
witnesses:

$\bigwedge^{2}_{k=0}\bigwedge^{m}_{h=1}\forall x\exists
y\Big{(}N_{k}(x)\to(N_{\lfloor k+1\rfloor}(y)\wedge
W_{h}(y)\wedge\psi^{*}_{h}(x,y))\Big{)}$

where $\psi^{*}_{h}(x,y)$ is of the form

$\psi^{*}_{h}(x,y)\stackrel{{\scriptstyle\mathit{df}}}{{=}}\Big{(}\bigwedge_{i=1}^{s}\neg
Q_{h,i}(x)\Big{)}\to\psi_{h}(x,y)$

Now, formula $\varphi_{\cal C}$ can be obtained by transforming the
conjunction of $\delta_{1}$, $\delta_{2}$, $\gamma_{1}$, $\gamma_{2}$,
$\gamma_{3}$, and $\nu\stackrel{{\scriptstyle\mathit{df}}}{{=}}\forall
x\forall y(x=y\vee\psi(x,y))$ into spread normal form. Due to the form of
these formulas, this can be easily achieved. This concludes the construction
of $\varphi_{\cal C}$

It remains to show that $\varphi$ has a (finite) model with court structure
${\cal C}$ if and only if $\varphi_{\cal C}$ has a (finite) model.

From a model ${\cal A}_{\cal C}$ of $\varphi_{\cal C}$ one can construct a
model ${\cal A}$ of $\varphi$ by ignoring the interpretations of all new
relation symbols in $\varphi_{\cal C}$, that is, by ignoring the relations
$Q^{{\cal A}_{\cal C}}_{i}$, $Q^{{\cal A}_{\cal C}}_{h,i}$, $W^{{\cal A}_{\cal
C}}_{i}$ and $N^{{\cal A}_{\cal C}}_{i}$. The thus obtained structure ${\cal
A}$ satisfies the $\forall\forall$-constraints of $\varphi$ due to $\nu$, and
the $\forall\exists$-constraints due to $\delta_{2}$, $\gamma_{2}$, and
$\gamma_{3}$. The court structure induced by ${\cal A}$ is ${\cal C}$ due to
$\delta_{2}$.

Conversely suppose that ${\cal A}$ is a model of $\varphi$ with induced court
structure ${\cal C}$. A model ${\cal A}_{\cal C}$ of $\varphi_{\cal C}$ can be
constructed from ${\cal A}$ as follows. Let $A$ be the universe of ${\cal A}$,
let $A_{0}\stackrel{{\scriptstyle\mathit{df}}}{{=}}\\{a_{1},\ldots,a_{s}\\}$
be its set of Kings, and let
$B\stackrel{{\scriptstyle\mathit{df}}}{{=}}A\setminus A_{0}$.

The structure ${\cal A}_{\cal C}$ is constructed by cloning $B$. It has
universe $A_{0}\cup\bigcup^{2}_{k=0}\bigcup^{m}_{h=1}B_{k,h}$ where each
$B_{k,h}$ is a copy of $B$.

The relations $Q^{{\cal A}_{\cal C}}_{i}$ and $Q^{{\cal A}_{\cal C}}_{h,i}$
are defined as:

$Q^{{\cal A}_{\cal
C}}_{i}\stackrel{{\scriptstyle\mathit{df}}}{{=}}\\{a_{i}\\}\quad\text{and}\quad
Q^{{\cal A}_{\cal
C}}_{h,i}\stackrel{{\scriptstyle\mathit{df}}}{{=}}\\{b^{\prime}\mid\text{$b^{\prime}$
is a copy of an element $b$ with $\psi_{h}(b,a_{i})$}\\}$

The relations $W^{{\cal A}_{\cal C}}_{h}$ and $N^{{\cal A}_{\cal C}}_{k}$ are
defined as:

$W^{{\cal A}_{\cal
C}}_{h}\stackrel{{\scriptstyle\mathit{df}}}{{=}}\bigcup^{2}_{k=0}B_{k,h}\quad\text{
and }\quad N^{{\cal A}_{\cal
C}}_{k}\stackrel{{\scriptstyle\mathit{df}}}{{=}}\bigcup^{m}_{h=1}B_{k,h}$

It remains to argue that the thus constructed structure ${\cal A}_{\cal C}$ is
a model of $\varphi_{\cal C}$. The formulas $\delta_{1}$ and $\delta_{2}$ are
satisfied, as ${\cal C}$ is the court structure induced by ${\cal A}$. The
formula $\gamma_{1}$ is satisfied due to the construction of $W^{{\cal
A}_{\cal C}}_{h}$ and $N^{{\cal A}_{\cal C}}_{k}$. The formula $\gamma_{2}$ is
satisfied due the construction of $Q^{{\cal A}_{\cal C}}_{h,i}$. The formula
$\gamma_{3}$ is satisfied due to the construction of $W^{{\cal A}_{\cal
C}}_{h}$ and $N^{{\cal A}_{\cal C}}_{k}$, and because the
$\forall\exists$-constraints of $\varphi$ are satisfied in ${\cal A}$. For
seeing that $\nu$ is satisfed we observe that it is satisfied in ${\cal A}$,
and cloning does not lead to new binary types (see (i) in the definition of
cloning).

∎

The proofs of Lemma 4.3, Lemma 4.4, and Theorem 4.5 in [Pra18] implicitly
imply the following lemma.

######  Lemma .

Suppose ${\cal K}$ is a class of structures with signature $\Delta$ and
$\varphi$ is a $\mathrm{FO}^{2}({\cal K})$ formula over $\Delta\uplus\Theta$.
If $\varphi$ is in spread normal form then one can effectively construct an
equisatisfiable $\mathrm{FO}^{2}({\cal K})$ formula over signature
$\Delta\uplus\Theta^{\prime}$ where $\Theta^{\prime}$ only contains unary
relation symbols.

###### Proof.

We provide two ways of proving the statement.

##### _Adaption of the proof of Pratt-Hartmann[Pra18]._

The proofs of Lemma 4.3 and Lemma 4.4 in [Pra18] are independent of the class
${\cal K}$ of structures. Thus, in particular, Lemma 4.4 can be restated for
arbitrary classes ${\cal K}$ with a spread normal form.

Suppose that $\varphi$ is an $\mathrm{FO}^{2}({\cal K})$ formula in spread
normal form with signature $\Delta\uplus\Theta$. By Lemma 4.4 in [Pra18],
$\varphi$ can be translated into $\varphi^{\circ}$ such that (1)
$\varphi^{\circ}$ does not use atoms of the form $r(x,y)$ or $r(y,x)$ for
$r\in\Theta$, (2) the formula $\varphi$ implies $\varphi^{\circ}$, and (3) if
$\varphi^{\circ}$ has a model over domain $A$, then so has $\varphi$. As the
only binary atoms from $\Theta$ occurring in $\varphi^{\circ}$ are of the form
$r(x,x)$ or $r(y,y)$, they can be replaced by new unary atoms, see Theorem 4.5
in [Pra18]. This yields the intended equisatisfiable formula.

##### _Direct proof._

Let $\varphi$ be a $\mathrm{FO}^{2}({\cal K})$ formula over signature
$\Delta\uplus\Theta$ in spread normal form

$\displaystyle\quad\bigwedge_{\theta\in Z}\exists x\theta(x)\wedge\forall
x\forall y(x=y\vee\nu(x,y))$ $\displaystyle\wedge$
$\displaystyle\quad\bigwedge_{k=0}^{2}\bigwedge_{h=0}^{m-1}\forall x\exists
y(\lambda_{k}(x)\rightarrow(\lambda_{\lfloor
k+1\rfloor}(y)\wedge\mu_{h}(y)\wedge\theta_{h}(x,y)))$

where (i) $Z$ is a set of unary pure Boolean formulas; (ii) $\nu$,
$\theta_{0},\ldots,\theta_{m-1}$ are quantifier- and equality-free formulas;
and (iii) $\lambda_{0}$, $\lambda_{1}$, and $\lambda_{2}$ are mutually
exclusive unary pure Boolean formulas; and (iv) $\mu_{0},\ldots,\mu_{m-1}$ are
mutually exclusive unary pure Boolean formulas.

Suppose, without loss of generality, that the formulas $\nu$ and $\theta_{h}$
are of the form

$\bigvee_{i}\alpha_{i}(x,y)$

where all $\alpha_{i}$ are binary types.

For a binary type $\alpha$, denote by $\alpha^{*}$ the set of formulas
obtained from $\alpha$ by replacing all literals of the form $R(x,y)$,
$R(y,x)$, $\neg R(x,y)$, and $\neg R(y,x)$ for $R\in\Theta$ by _true_. Further
denote by $\nu^{*}$ and $\theta^{*}_{h}$ the formulas obtained by replacing
all binary types $\alpha$ by $\alpha^{*}$ in $\nu$ and $\theta_{h}$.

Consider the formula $\varphi^{*}$ defined as

$\displaystyle\quad\bigwedge_{\theta\in Z}\exists x\theta(x)\wedge\forall
x\forall y(x=y\vee\nu^{*}(x,y))$ $\displaystyle\wedge$
$\displaystyle\quad\bigwedge_{k=0}^{2}\bigwedge_{h=0}^{m-1}\forall x\exists
y(\lambda_{k}(x)\rightarrow(\lambda_{\lfloor
k+1\rfloor}(y)\wedge\mu_{h}(y)\wedge\theta^{*}_{h}(x,y)))$

This formula only uses literals from $\Delta$, unary literals over $\Theta$
and binary literals of the form $R(x,x),R(y,y),\neg R(x,x),\neg R(y,y)$ over
$\Theta$.

We claim that $\varphi^{*}$ and $\varphi$ are equisatisfiable. Certainly a
model ${\cal A}$ of $\varphi$ is also a model of $\varphi^{*}$ (as can be seen
by inspecting the conjuncts of $\varphi^{*}$).

For the converse suppose that ${\cal A}^{*}$ is a model of $\varphi^{*}$. Then
a model ${\cal A}$ of $\varphi$ can be obtained as follows (cf. proof of Lemma
4.4 in [Pra18]). The structure ${\cal A}$ has the same elements as ${\cal
A}^{*}$. Further ${\cal A}$ inherits the unary types from ${\cal A}^{*}$. It
remains to fix the binary types for ${\cal A}$.

Fix an element $a$. If $a$ satisfies one of the $\lambda_{i}$ (and since these
are disjoint, no other $\lambda_{j}$), then for all $h$ there exists $b$ such
that ${\cal A}\models\lambda_{i+1}(b)\wedge\mu_{h}(b)$. Let $\alpha^{*}$ be a
disjunct of $\theta^{*}_{h}$ that is satisfied by $(a,b)$. Suppose
$\alpha^{*}$ was obtained from a type $\beta$ of $\theta_{h}$ in the
construction of $\varphi^{*}$, then chose $\beta$ as the binary type of
$(a,b)$. Note that $\beta$ is consistent with the (already assigned) unary
types of $a$ and $b$, as $\alpha^{*}$ was obtained from $\beta$ by replacing
strictly binary literals by true. In this way, all witnesses for $a$ are
fixed. Since all $\mu_{h}$ are disjoint (since $\varphi$ is in spread normal
form), all such witnesses are distinct.

The above procedure is repeated for all $a$. Note that for a tuple $(a,b)$ the
binary types assigned for determining the witnesses of $a$ and $b$ do not
conflict since witnesses satisfy $\lambda_{\lfloor i+1\rfloor}$ which is
disjoint from $\lambda_{i}$. Furthermore, the types assigned in this fashion
do not conflict with $\nu$, because all types $\alpha$ used in the formulas
$\theta_{h}$ must be consistent with $\nu$.

It remains to assign the binary types in ${\cal A}$ for all remaining tuples
$(a,b)$. This has to be done in accordance with the condition $\nu$. As ${\cal
A}^{*}$ satisfies $\varphi^{*}$, the tuple $(a,b)$ satisfies $\nu^{*}$ and
hence some formula $\alpha^{*}$ that was obtained from some binary type
$\beta$ in $\nu$. The binary type of $(a,b)$ is chosen as $\beta$. This,
again, does not conflict with previously chosen unary types of $a$ and $b$.

Hence $\varphi$ and $\varphi^{*}$ are equisatisfiable. Now, an equisatisfiable
formula $\psi$ for $\varphi$ over signature $\Delta\uplus\Theta^{\prime}$
where $\Theta^{\prime}$ only contains unary relation symbols can be obtained
from $\varphi^{*}$ as follows (cf. proof of Theorem 4.5 in [Pra18]). All
binary atoms in $\varphi^{*}$ with relation symbols from $\Theta$ are of the
form $R(x,x)$ or $R(y,y)$. Replacing these atoms by atoms $\widehat{R}(x)$ and
$\widehat{R}(y)$ where $\widehat{R}$ is a fresh unary relation symbol yields
the equisatisfiable formula $\psi$ over signature
$\Delta\uplus\Theta^{\prime}$ where

$\Theta^{\prime}\stackrel{{\scriptstyle\mathit{df}}}{{=}}\\{S\in\Theta\mid
S\text{ is unary}\\}\uplus\\{\widehat{R}\in\Theta\mid R\in\Theta\text{ is
binary}\\}.$

∎

###### Proof of Proposition 5.2.

By Lemma 5.2 structures with two linear order allow cloning, and therefore
each $\mathrm{FO}^{2}(<_{1},<_{2})$ formula can be transformed into an
equisatisfiable disjunction $\bigvee_{i}\varphi_{i}$ of formulas $\varphi_{i}$
over signature $\\{<_{1},<_{2}\\}\uplus\Theta_{i}$ in spread normal form by
Lemma 5.2. Now, each $\varphi_{i}$ can be transformed into an equisatisfiable
formula $\varphi^{\prime}_{i}$ over
$\\{<_{1},<_{2}\\}\uplus\Theta^{\prime}_{i}$ where each $\Theta^{\prime}_{i}$
is unary, by Lemma 5.2. Now, Proposition 5.2 follows from the fact that
$\bigvee_{i}\varphi^{\prime}_{i}$ is equisatisfiable to
$\bigvee_{i}\varphi_{i}$. ∎

Recall that Theorem 5 implies the (general) decidability of
$\mathrm{EMSO}^{2}(<_{1},<_{2})$, see Corollary 5. Therefore, Proposition 5.2
yields Theorem 5.

## Acknowledgment

We are grateful to Nathan Lhote for insightful discussions, in particular on
the inclusion of MSO-definable atoms in Theorem 5. We also thank the anonymous
reviewers for helpful suggestions on improving the readability of the article.

The second author acknowledges the financial support by DFG grant 448468041.

## References

  * [BCL08] Achim Blumensath, Thomas Colcombet, and Christof Löding.  Logical theories and compatible operations.  In Logic and Automata: History and Perspectives [in Honor of Wolfgang Thomas], pages 73–106, 2008.
  * [BKL08] Christel Baier, Joost-Pieter Katoen, and Kim Guldstrand Larsen.  Principles of model checking.  MIT press, 2008.
  * [BMSS09] Mikołaj Bojańczyk, Anca Muscholl, Thomas Schwentick, and Luc Segoufin.  Two-variable logic on data trees and XML reasoning.  J. ACM, 56(3):13:1–13:48, 2009.
  * [CW13] Witold Charatonik and Piotr Witkowski.  Two-variable logic with counting and trees.  In LICS’13, pages 73–82, 2013.
  * [DFL18] Luc Dartois, Emmanuel Filiot, and Nathan Lhote.  Logics for word transductions with synthesis.  In Anuj Dawar and Erich Grädel, editors, Proceedings of the 33rd Annual ACM/IEEE Symposium on Logic in Computer Science, LICS 2018, Oxford, UK, July 09-12, 2018, pages 295–304. ACM, 2018.
  * [GKV97] Erich Grädel, Phokion G. Kolaitis, and Moshe Y. Vardi.  On the decision problem for two-variable first-order logic.  Bulletin of Symbolic Logic, 3(1):53–69, 1997.
  * [GO99] Erich Grädel and Martin Otto.  On logics with two variables.  Theor. Comput. Sci., 224(1-2):73–113, 1999.
  * [JLZ20] Jean Christoph Jung, Carsten Lutz, and Thomas Zeume.  On the decidability of expressive description logics with transitive closure and regular role expressions.  In Diego Calvanese, Esra Erdem, and Michael Thielscher, editors, Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, KR 2020, Rhodes, Greece, September 12-18, 2020, pages 529–538, 2020.
  * [KF94] Michael Kaminski and Nissim Francez.  Finite-memory automata.  Theor. Comput. Sci., 134(2):329–363, 1994.
  * [Kie11] Emanuel Kieronski.  Decidability issues for two-variable logics with several linear orders.  In CSL 2011, volume 12 of LIPIcs, pages 337–351, 2011.
  * [KO12] Emanuel Kieronski and Martin Otto.  Small substructures and decidability issues for first-order logic with two variables.  J. Symb. Log., 77(3):729–765, 2012.
  * [KT09] Emanuel Kieronski and Lidia Tendera.  On finite satisfiability of two-variable first-order logic with equivalence relations.  In LICS 2009, pages 123–132. IEEE Computer Society, 2009.
  * [Lib04] Leonid Libkin.  Elements of Finite Model Theory.  Springer, 2004.
  * [Mor75] Michael Mortimer.  On languages with two variables.  Zeitschr. f. math. Logik u. Grundlagen d. Math., 21:135–140, 1975\.
  * [Nev02] Frank Neven.  Automata theory for XML researchers.  ACM Sigmod Record, 31(3):39–46, 2002.
  * [NSV04] Frank Neven, Thomas Schwentick, and Victor Vianu.  Finite state machines for strings over infinite alphabets.  ACM Trans. Comput. Log., 5(3):403–435, 2004.
  * [Ott01] Martin Otto.  Two variable first-order logic over ordered domains.  J. Symb. Log., 66(2):685–702, 2001.
  * [Pra18] Ian Pratt-Hartmann.  Finite satisfiability for two-variable, first-order logic with one transitive relation is decidable.  Math. Log. Q., 64(3):218–248, 2018.
  * [Rab72] Michael Oser Rabin.  Automata on Infinite Objects and Church’s Problem.  Number 13. American Mathematical Soc., 1972.
  * [Sco62] Dana Scott.  A decision method for validity of sentences in two variables.  Journal of Symbolic Logic, 27(377):74, 1962.
  * [She75] Saharon Shelah.  The monadic theory of order.  Annals of Mathematics, pages 379–419, 1975.
  * [SZ12] Thomas Schwentick and Thomas Zeume.  Two-variable logic with two order relations.  Logical Methods in Computer Science, 8(1), 2012.
  * [Tho97] Wolfgang Thomas.  Languages, automata, and logic.  In Handbook of formal languages, pages 389–455. Springer, 1997\.
  * [TZ20] Szymon Torunczyk and Thomas Zeume.  Register automata with extrema constraints, and an application to two-variable logic.  In Holger Hermanns, Lijun Zhang, Naoki Kobayashi, and Dale Miller, editors, LICS ’20: 35th Annual ACM/IEEE Symposium on Logic in Computer Science, Saarbrücken, Germany, July 8-11, 2020, pages 873–885. ACM, 2020.
  * [ZH16] Thomas Zeume and Frederik Harwath.  Order-invariance of two-variable logic is decidable.  In Martin Grohe, Eric Koskinen, and Natarajan Shankar, editors, Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science, LICS ’16, New York, NY, USA, July 5-8, 2016, pages 807–816. ACM, 2016.


# Planning with Learned Entity Prompts for Abstractive Summarization

Shashi Narayan
Google Research
<EMAIL_ADDRESS>&Yao Zhao
Google Brain
<EMAIL_ADDRESS>&Joshua Maynez
Google Research
<EMAIL_ADDRESS>Gonçalo Simoes
Google Research
<EMAIL_ADDRESS>&Vitaly Nikolaev
Google Research
<EMAIL_ADDRESS>&Ryan McDonald
ASAPP
<EMAIL_ADDRESS>Work done while Ryan was at Google.

###### Abstract

We introduce a simple but flexible mechanism to learn an intermediate plan to
ground the generation of abstractive summaries. Specifically, we prepend (or
_prompt_) target summaries with entity chains – ordered sequences of entities
mentioned in the summary. Transformer-based sequence-to-sequence models are
then trained to generate the entity chain and then continue generating the
summary conditioned on the entity chain and the input. We experimented with
both pretraining and finetuning with this content planning objective. When
evaluated on CNN/DailyMail, XSum, SAMSum and BillSum, we demonstrate
empirically that the grounded generation with the planning objective improves
entity specificity and planning in summaries for all datasets, and achieves
state-of-the-art performance on XSum and SAMSum in terms of rouge. Moreover,
we demonstrate empirically that planning with entity chains provides a
mechanism to control hallucinations in abstractive summaries. By prompting the
decoder with a modified content plan that drops hallucinated entities, we
outperform state-of-the-art approaches for faithfulness when evaluated
automatically and by humans.

##  1 Introduction

Jones (1993) described text summarization – the task of generating accurate
and concise summaries from source document(s) – as a three-step process: (i)
Building the source representation from the source document(s), (ii) Learning
a summary representation from the source representation and (iii) Synthesizing
the output summary text. Common to most traditional methods, an input
representation was learned by semantically analyzing the source text, the
summary representation was then learned by modifying and refining the input
representation, and finally the summary was generated grounded to the
intermediate summary representation Luhn (1958); McKeown and Radev (1995);
Barzilay and Elhadad (1997); Mihalcea and Tarau (2004).

State-of-the-art neural summarizers are powerful representation learners and
conditional language models, thanks to sequence-to-sequence architectures
(seq2seq) with attention and copy mechanism Hochreiter and Schmidhuber (1997);
Bahdanau et al. (2015); See et al. (2017), Transformer architectures with
multi-headed self-attention Vaswani et al. (2017), and large pretrained
conditional language models Dong et al. (2019); Song et al. (2019); Lewis et
al. (2019); Rothe et al. (2020a); Raffel et al. (2019); Zhang et al. (2019).
However, the grounding of summary generation that was inherent to most
traditional methods is yet to be achieved in neural summarization. The
attention mechanism Bahdanau et al. (2015), especially in pretrained encoder-
decoder models Lewis et al. (2019); Raffel et al. (2019); Zhang et al. (2019),
plays a key role in aligning summary content to the input, yet undesired
hallucinations are common in generated summaries Maynez et al. (2020);
Kryscinski et al. (2020); Gabriel et al. (2020).

|
---|---
Figure 1: Pretraining and finetuning for abstractive summarization with entity
chains.

In this paper, we investigate Entity Chains – ordered sequences of
entities111We use the term ‘entity’ broadly and consider named entities, dates
and numbers to form an entity chain. in the summary – as an intermediate
summary representation to better plan and ground the generation of abstractive
summaries. During training, we construct an augmented target summary by
extracting and prepending its corresponding entity chain (Figure 1, right). At
test time, the model must generate both the entity chain followed by the
summary. Concretely, we use Transformer-based encoder-decoder Vaswani et al.
(2017) models; a transformer encoder first encodes the input and a transformer
decoder generates (i) an intermediate summary representation in the form of an
entity chain; and (ii) the summary conditioned on the entity chain and the
input. We evaluate our approach on four popular summarization datasets:
CNN/DailyMail highlight generation Hermann et al. (2015), XSum extreme
summarization Narayan et al. (2018), SAMSum dialogue summarization Gliwa et
al. (2019) and BillSum Kornilova and Eidelman (2019), and show that the state-
of-the-art Pegasus Zhang et al. (2019) pretrained models finetuned with the
planning objective clearly outperform regular finetuning in terms of entity
specificity and planning in generated summaries on all datasets. We further
demonstrate that this simple planning mechanism can be easily used for
pretraining summarization models to do entity-level content planning and
summary generation. Similar to Pegasus pretraining, we mask important
sentences from an input document, extract an entity chain from the masked
sentences, and generate these gap-sentences prepended with their entity chain
from the rest of the document (Figure 1, left). We see further gains with
pretraining achieving state of the art performance on XSum in terms of rouge.
We further demonstrate how the entity-level content planning in summarization
can be easily leveraged to mitigate hallucinations in abstractive summaries.
In particular, we modify the predicted entity chain to only keep entities that
are seen in the document and then generate the summary prompted with the
modified entity chain, outperforming state-of-the-art approaches when
evaluated automatically and by humans for faithfulness. Our main contributions
are as follows:

#### Planned and Grounded Abstractive Summarization

We introduce a novel training objective to neural summarization models for
content planning with entity chains, and study integrating it with supervised
finetuning and self-supervised pretraining objectives without altering the
models themselves. As the entity chains are extracted from the reference
summaries during training, our models learn to ground the generation of
summaries to the entity chains found in them. Hence, we refer to this
objective by Frost for its ability to try to “FReeze entity-level infOrmation
in abstractive SummarizaTion with planning.”

#### Controlled Abstractive Summarization with Entity Chains

Frost provides a very effective knob for entity-level content modification in
abstractive summaries. In this paper we empirically demonstrate how Frost is
critical for faithfulness by enabling the drop-prompt mechanism where we drop
out hallucinated entities from the predicted content plan and prompt the
decoder with this modified plan to generate faithful summaries. We further
qualitatively demonstrate that Frost enables generation of summaries (i) with
topical diversity by choosing different sets of entities from the source to
plan what we want to discuss in the summary, and (ii) with style diversity by
reordering entities in the predicted plan to get an equivalent summary but
with a different entity emphasis.

##  2 Related Work

#### Content Planning for Summarization.

Traditional methods argue on the granularity of linguistic, domain and
communicative information included in the source representation needed in
order to plan and build better summary representations. Some argued to use a
deep semantic analysis of the source text, such as Rhetorical Structure Theory
(RST; Mann and Thompson, 1988) or MUC-style representations McKeown and Radev
(1995) to interpret the source texts, while others used a shallow semantic
analysis using only word frequency Luhn (1958) or lexical chains Barzilay and
Elhadad (1997).

Recent encoder-decoder models for text generation Bahdanau et al. (2015);
Sutskever et al. (2014); Vaswani et al. (2017); Rothe et al. (2020b); Lewis et
al. (2019); Raffel et al. (2019); Zhang et al. (2019) tend to perform text
generation in an end-to-end setting, which means that most approaches do not
explicitly model content planning. Wiseman et al. (2018) and Hua and Wang
(2020) start the generation process by building templates which are then used
for realization. In data-to-text generation, Puduppully et al. (2019) generate
a content plan highlighting which information should be mentioned in the table
and in which order. In story generation, there has been some work on exploring
events Martin et al. (2018) or sequences of words Yao et al. (2019) to plan
ahead when creating a consistent story. We are not aware of any similar work
on content planning for summarization using encoder-decoder models.

#### Pretraining for Summarization and Planning.

Pretrained transformer-based models has dramatically changed the text
generation space and summarization is no exception to this. Most models focus
on task-agnostic pretraining using the left-to-right language modelling
objective Radford et al. (2018); Khandelwal et al. (2019); Dong et al. (2019)
or reconstructing the corrupted input text using a sequence-to-sequence
framework Song et al. (2019); Lewis et al. (2019); Raffel et al. (2019). There
has been few attempts towards task-specific pretraining for summarization to
teach models to do better content selection. Zhang et al. (2019) proposed to
select important sentences from an input document as a proxy for human-
authored summary and then to generate them from the rest of the document.
Narayan et al. (2020) proposed question generation pretraining to better align
with summarization. To the best of our knowledge we are the first one to
propose a solution that incorporates content planning directly into
pretraining.

#### Controlled Abstractive Summarization.

There is a growing interest in enabling users to specify high level
characteristics such as length, keywords, topic, in order to generate
summaries that better suit their needs. In most cases, these features are
first manually provided or estimated using third-party content selectors, and
then either (i) encoded along with the input Fan et al. (2018); He et al.
(2020); Dou et al. (2020) or (ii) used to filter beams for lexically
constrained decoding Mao et al. (2020), to control the summary generation. On
the contrary, our approach is more generic as we do not rely on external
systems or data to augment the input; users can prompt the decoder with a
desired content plan in the form of an entity chain to control the summary.

##  3 Content Planning with Entity Chains

We introduce a new training objective for encoder-decoder generative models to
do content planning while summarizing.

#### Model Formulation.

Let $d$ be an input document, we aim to teach our model to first generate a
content plan $c$ for the summary $s$ as $p(c|d)$, and then generate the
summary $s$ as $p(s|c,d)$. We define the ordered chain of entities observed in
the summary $s$ as its content plan. Instead of modeling $p(c|d)$ and
$p(s|c,d)$ separately, we take a simpler approach, we train an encoder-decoder
model to encode the document $d$ and generate the concatenated content plan
and summary sequences $c;s$, essentially the decoder first predicts the entity
chain $c$ and then continues predicting the summary $s$ using both $c$ and
$d$. We prefix $c$ and $s$ with special markers “[ENTITYCHAIN]” and
“[SUMMARY]”, respectively, as shown in Figure 1. If $s$ consists of multiple
sentences, we use sentence markers “$|||$” to mark them in $c$. The model is
trained with the standard maximum-likelihood objective generating the
augmented target $c;s$.

#### Pretraining Content Plans.

We modified Pegasus Zhang et al. (2019) to pretrain our models for entity-
level content planning and summary generation.222We experimented with Pegasus,
but our technique can be used with any pretraining objectives that require
sentence-level input corruptions. In particular, we select a maximum of $n$
most important sentences using self-rouge from an input document, the selected
sentences work as a proxy for a human-authored abstractive summaries for the
rest of the document. We construct a target by prepending the selected
sentences dynamically with their entity chain. Our model is then trained to
generate this target from the rest of the document.333The Pegasus objective
uses a summary-document length ratio to select $n$. This could lead to an
undesirably long summary when the input document is very long. Modeling such
summaries prepended with long entity chains effectively is beyond the limit of
our decoders (256 sentencepieces). Hence, we set $n=5$.

#### Modeling Entity-Level Lexical Cohesion and Coherence.

As entities in the summary contribute to the continuity of lexical meaning of
the summary, we hypothesize that by learning to predict the entity chain $c$
in advance, we enforce our model to learn entity-level lexical cohesion
Barzilay and Elhadad (1997) and coherence Halliday and Hasan (1976); Azzam et
al. (1999) in the summary. We hope that by doing so our model will be better
at predicting pertinent entities (entity specificity) in their right order
(entity planning) in generated summaries; the prediction of entities in the
entity chain $c$ (as $p(c|d)$ in Frost) will be less susceptible to local
correlations compared to when predicting them directly in the summary $s$ (as
$p(s|d)$). Furthermore, as $c$ is predicted in advance and the generation of
$s$ is grounded to $c$,444Here, $s$ is not strictly constrained to the entity
chain $c$. We hope that this will happen given $c$ is extracted from $s$
during the training time. Future work will focus on constraining $s$ to $c$,
e.g., using a checklist model Kiddon et al. (2016) or entity-chain constrained
decoding Mao et al. (2020). our model will be better equipped to predict
correct events relating to different entities in $c$ with full access to $c$
and not just entities to the left, already decoded.

Dataset | Size | Case | Target Summaries (Validation)
---|---|---|---
avg. | avg. | avg. | % target | total
| train/dev/test | sent. | ent. | uniq. ent. | (no ent.) | named | date | number
BillSum | 18.9k/–/3.3k | cased | 4.38 | 14.78 | 11.10 | 0.18 | 28931 | 2578 | 16646
CNN/DailyMail | 287k/13.4k/11.5k | cased | 4.11 | 7.55 | 6.92 | 0.10 | 74292 | 3569 | 23094
SAMSum | 14.7k/818/819 | cased | 2.03 | 3.59 | 3.01 | 0.37 | 2594 | 33 | 309
XSum | 204k/11.3k/11.3k | cased | 1.00 | 2.81 | 2.80 | 5.97 | 24682 | 777 | 6287
Table 1: Abstractive summarization datasets studied in this work. We report on
their train/validation/test sizes and how they were processed (cased/uncased).
To better understand the effect of summary planning with entity chains, we
report on average number of sentences (avg. sent.), average number of entities
(avg. ent.) and average number of unique entities (avg. uniq. ent.), per
target in validation sets. We also report on total number of named entities,
date and number in target summaries.

#### Controlled Generation with Entity Prompts.

An advantage of training to generate the summary $s$ following the generation
of the plan $c$ using the same decoder is that now during the inference time
the decoder can be easily prompted with any desired content plan $c^{\prime}$
to control the content in the output summary $s^{\prime}$ with a probability
of $p(s^{\prime}|c^{\prime},d)$. In Section 5, we prompt our decoder with
modified content plans to mitigate hallucinations and to generate diverse
summaries.

##  4 Experimental Setup

###  4.1 Base and Large Models

We experiment with both base and large transformer architectures Vaswani et
al. (2017). The base architecture has $L=12$, $H=768$, $F=3072$, $A=12$ (223M
parameters) and the large architecture has $L=16$, $H=1024$, $F=4096$, $A=16$
(568M parameters), where $L$ denotes the number of layers for encoder and
decoder Transformer blocks, $H$ for the hidden size, $F$ for the feed-forward
layer size, and $A$ for the number of self-attention heads. All pretrainings
are done with a batch size of 1024, whereas all finetuning experiments are
done with a smaller batch size of 256. For optimization, we use Adafactor
Shazeer and Stern (2018) with square root learning rate decay and dropout rate
of $0.01$ during pretraining and $0.0001$ during finetuning. All finetuned
models were decoded with a beam size of $8$ and a length-penalty of $0.8$.555
We will release all models and other resources for easy replication of our
results.

###  4.2 Datasets and Entity Annotations

#### Pretraining Datasets.

Following Zhang et al. (2019), our model pretraining also relied on two large
web corpus which were processed to look like plain text: (i) C4 Raffel et al.
(2019) is composed of 350M Web-pages that were obtained from Common Crawl, and
(ii) HugeNews Zhang et al. (2019) is composed of 1.5B news and news-like
articles from 2013-2019. This dataset includes articles from multiple
allowlisted sources including news publishers, high-school newspapers and
blogs.

#### Abstractive Summarization Datasets.

We evaluate our models on four summarization datasets: CNN/DailyMail highlight
generation Hermann et al. (2015), XSum extreme summarization Narayan et al.
(2018), SAMSum dialogue summarization Gliwa et al. (2019) and BillSum
summarizing US Congressional bills Kornilova and Eidelman (2019). We use the
publicly available versions through the TFDS Summarization
Datasets666https://www.tensorflow.org/datasets/catalog. We use the original
train/validation/test splits for them. For BillSum where the validation split
was not provided, we split 10% of the training set to serve as validation.
Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and,
1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights
into these datasets to better understand the effect of summary planning with
entity chains.

#### Entity Chain Annotation.

We experimented with entity chains consisting of named entities, dates and
numbers. We annotate the whole document in the pretraining datasets to allow
dynamic construction of summaries and their entity chains during pretraining.
We only annotate the reference summaries for the finetuning summarization
datasets. We use a BERT-based tagger trained on CoNLL-2003 data Tjong Kim Sang
and De Meulder (2003) to identify named entities, and regular expressions to
identify dates and numbers Guu et al. (2020). See Table 1 for the number of
named entities, dates and numbers found in different datasets.

Table 6 and 7 in Appendix A present hyperparameters used for pretraining and
finetuning Pegasus and Frost base and large sized models. We used Cloud TPU v3
accelerators for training.

###  4.3 Evaluation Measures

We evaluate our Frost models on rouge, entity specificity, entity planning,
faithfulness using automatic and human evaluations, and overall quality by
humans. Our models predict a summary plan in the form of an entity chain,
followed by a summary. All evaluations are done on the summary, the predicted
entity chains are stripped out.

#### Summary-level rouge.

We report rouge F1 scores Lin and Hovy (2003) to assess generated
summaries.777We lowercased candidate and reference summaries and used pyrouge
with parameters “-a -c 95 -m -n 4 -w 1.2.”

#### Entity Planning.

We evaluate the quality of content plans learned by our models by assessing
the entities and the order in which they appear in the summary. We annotate
both predicted and reference summaries with named entities, dates and numbers,
and report on rouge F1 scores for entity chains found in the predicted
summaries against corresponding reference entity chains.

#### Entity Specificity.

We compare entities in predicted and reference summaries, and report average
entity F1-scores (EntF1) in predicted summaries. We lower-case and remove
duplicate entities; we report on the exact match with reference entities.

#### Faithfulness.

For entity-level faithfulness, we report on EntPrec, measuring the precision
of entities in predicted summaries against their input documents. EntF1, in
comparison, evaluates predicted summaries for entity specificity against
reference summaries and is not a measure for faithfulness. For summary-level
faithfulness, we follow Maynez et al. (2020) and report on textual entailment
Pasunuru and Bansal (2018); Falke et al. (2019); Kryscinski et al. (2019). In
particular, we report the probability of a summary entailing (Entail.) its
input document using an entailment classifier trained by fine-tuning an
uncased BERT-Large pretrained model Devlin et al. (2019) on the Multi-NLI
dataset Williams et al. (2018).

We further assess summary faithfulness by humans. Our annotators, proficient
in English, were tasked to read the document carefully and then grade its
summary on a scale of 1-5 (fully unfaithful, somewhat unfaithful, 50-50,
somewhat faithful and fully faithful); a summary is “fully faithful” if all of
its content is fully supported or can be inferred from the document.

#### Overall Summary Quality.

Finally we assess the overall quality of summaries by humans. We ask our
annotators to read the document carefully and then grade its summary on a
scale of 1-5 (poor summary, better than poor, okay summary, better than okay
and great summary). To improve the annotator agreement, we clearly define 4
features that a “great summary” should have: Relevant (summary should have
most important information from the document), Accurate (summary should be
accurate with respect to the document or the expert knowledge),888With
accurate we mean factual to the background knowledge and not just faithful to
the document; as it is natural to construct summaries that integrate with the
author’s background knowledge Maynez et al. (2020). Concise (summary should
not have redundant or less-important content), and Fluent (summary should be
grammatically correct and coherent). A great summary should pass on all 4
features, a better than okay should have 3 out of 4 features, and so on.

For both human assessments, we collected 3 ratings for each (document,
summary) pair and report the average of all assigned labels (1-5) to a system.
We conducted 3 pilot studies for each setup to train our annotators with
examples to improve their understanding of the task. Additionally, extra
measures were taken to improve agreements among our annotators. For example,
for the faithfulness assessment, when one of somewhat unfaithful, 50-50 and
somewhat faithful were selected, annotators were asked to also specify what
was faithful or unfaithful in the summary. Similarly for the overall quality
assessment, when one of better than poor, okay summary and better than okay
were selected, they were asked to list all features on which the candidate
summary fails. Figure 8 and 9 in Appendix B show detailed instructions for
human evaluations for faithfulness and overall quality of summaries,
respectively.

##  5 Results

###  5.1 Frost Ablations

Sentence-level Entity Chain with Target: [EntityChain] Frozen $|$ Disney
[Summary] “Frozen,” the latest Disney musical, preaches the importance of
embracing your true nature. [EntityChain] Princess Anna $|$ Kristoff $|$ Snow
Queen Elsa [Summary] It depicts fearless Princess Anna who joins forces with
mountaineer Kristoff and his reindeer sidekick to find estranged sister, Snow
Queen Elsa, and break her icy spell.
---
Summary-level Entity Chain with Target: [EntityChain] Frozen $|$ Disney $|||$
Princess Anna $|$ Kristoff $|$ Snow Queen Elsa [Summary] “Frozen,” the latest
Disney musical, preaches the importance of embracing your true nature. It
depicts fearless Princess Anna who joins forces with mountaineer Kristoff and
his reindeer sidekick to find estranged sister, Snow Queen Elsa, and break her
icy spell.
Original Target: “Frozen,” the latest Disney musical, preaches the importance
of embracing your true nature. It depicts fearless Princess Anna who joins
forces with mountaineer Kristoff and his reindeer sidekick to find estranged
sister, Snow Queen Elsa, and break her icy spell.
Figure 2: An example of sentence-level and summary-level entity chains along
with its reference summary. RL-
SumR2-EPlanEntF1$40$$45$$50$$55$FscoresNoneSent-LevelSum-Level Figure 3:
Sentence-level vs summary-level entity chains. We report summary-level rouge-l
(RL-Sum), entity chain-level rouge-2 (R2-EPlan) and EntF1 on the CNN/DailyMail
validation set. Similar observations were made for other measures.

#### Sentence-Level vs Summary-Level Planning.

Let the summary $s$ consists of $m$ sentences $s_{1}\ldots s_{m}$ and $c_{i}$
be the entity chain for the sentence $s_{i}$, we consider generating the
summary $s$ in two ways. Sentence-level approach trains a model to generate
$s$ by consecutively generating the sentence-level content plan $c_{i}$
followed by its summary sentence $s_{i}$ with a probability
$p(c_{i}s_{i}|c_{1}s_{1}\ldots c_{i-1}s_{i-1};d)$; $d$ is the input document.
Summary-level approach first generates a summary-level content plan
$c=c_{1}|||\ldots|||c_{m}$ and then continue generating the summary $s$ with a
probability $p(s|c;d)$; $|||$ are sentence markers. The summary-level planning
is arguably better suited for summarization than the sentence-level planning.
By planning the whole summary beforehand, the summary-level planner would be
(i) less susceptible to local correlations than the sentence-level planning
while generating the entities, and (ii) a better microplanner in deciding
sentence boundaries and avoiding verbose summaries Reiter and Dale (1997). See
examples for sentence-level and summary-level entity chains in Figure 2.

We finetuned our large models initialized with the Pegasus checkpoint on the
CNN/DailyMail dataset for sentence-level vs summary-level entity chain
ablations. We did not do this study on the XSum dataset as the XSum summaries
are single sentences, which means that sentence-level and summary-level entity
chains are the same. In contrast, the CNN/DailyMail summaries consists of
multi-sentence highlights. Results are presented in Figure 3.

We found that the summary-level planning is not only superior to the sentence-
level planning, it helps with generating better quality summaries in terms of
summary-level rouge (RL-Sum), entity planning (R2-EPlan) and entity
specificity (EntF1). For the rest of the pretraining or finetuning
experiments, we focused on summary-level planning unless specified otherwise.

PegasusFrost(F)Frost(P+F)$22$$22.1$$22.2$$22.3$$22.4$R2-SummaryPegasusFrost(F)Frost(P+F)$37.3$$37.4$$37.5$$37.6$$37.7$$37.8$R2-EntityPlanPegasusFrost(F)Frost(P+F)$53$$53.1$$53.2$$53.3$$53.4$EntF1
Figure 4:  Finetuning results on the XSum validation set using one of the
base-sized pretrained models: Pegasus, Frost(F) and Frost(P+F). All pretrained
models were trained for 1.5m steps. See text for more details. We only report
on a subset of measures, similar observations were made for other measures.

#### Pretraining for Planning from Scratch.

We pretrained three base-sized models from scratch: Pegasus-base pretrained
with the original gap-sentence objective Zhang et al. (2019) for 1.5m steps,
Frost(F)-base pretrained with the gap-sentences prepended with their entity
chain for 1.5m steps, and Frost(P+F)-base first pretrained with the Pegasus
objective for 1m steps and then with the Frost objective for another 500k
steps. Maximum input and output lengths were set to 512 and 256 sentencepieces
during pretraining, respectively. We finetuned these three models on the XSum
dataset. Results are shown in Figure 4.

First of all, our results confirm that the pretraining for planning is
beneficial for summarization; both Frost(F) and Frost(P+F) learned better
content plans (in terms of entity chain-level rouge and EntF1) than Pegasus
without sacrificing the summary quality (in terms of summary-level rouge)
significantly. Interestingly, the Frost(P+F) finetuned model outperformed
Frost(F) on all measures confirming that the pretraining for planning and
summarization is more effective when started with summarization pretrained
models such as Pegasus than when trained jointly from scratch. Hence, for our
future pretraining experiments we initialize our model with existing Pegasus
checkpoint and continue pretraining for content planning and refer to them as
Frost for simplicity.

Pegasus0.1m1m1.5m$23$$24$$25$R2-SumPegasus0.1m1m1.5m$54$$55$$56$EntF1 Figure
5: Finetuning results on the XSum (in blue) and CNN/DailyMail (in red)
validation sets at various steps during pretraining Frost-Large. Instead of
pretraining from scratch, we start with a Pegasus-Large checkpoint, and
continue pretraining for additional 1.5m steps with the planning objective. We
report finetuning results for the Pegasus finetuned baseline and our models at
0.1m, 1m and 1.5m steps.

#### Effect of Pretraining Longer for Planning.

Based on our findings, we pretrain a large Frost model for planning for
summarization starting with an existing Pegasus-Large checkpoint. Figure 5
presents results for finetuning these models on the XSum and CNN/DailyMail
datasets at various steps during pretraining.

Similar to our findings in Figure 4, our results with large models further
confirm the advantages of the pretraining for planning; improvement over the
Pegasus baseline were larger for the XSum dataset than for the CNN/DailyMail
dataset. We achieved the best performance on both datasets at the 1m
pretraining step. We use the Frost model at 1m pretraining step for our
finetuning experiments.

Model | Summary | Entity Planning | Specificity
---|---|---|---
R1/R2/RL | R1/R2/RL | EntF1
CNN/DailyMail
Lead-3 | 40.31/17.83/36.45 | 57.41/42.32/47.16 | 46.22
Ext-Oracle∗ | 57.03/34.38/53.12 | 68.79/55.77/59.02 | 59.91
RoBERTaShare | 39.25/18.09/36.45 | – | –
MASS | 42.12/19.50/39.01 | – | –
UniLM | 43.33/20.21/40.51 | – | –
T5 | 43.52/21.55/40.69 | – | –
BART | 44.16/21.28/40.90 | – | –
ProphetNet | 44.20/21.17/41.30 | – | –
Pegasus | 44.16/21.56/41.30 | – | –
GSum |  45.94/22.32/42.48 | – | –
CTRLsum |  45.65/22.35/42.50 | 62.19/48.35/52.53 | 51.69
Pegasus (ours) | 44.05/21.69/40.98 | 61.12/46.46/52.40 | 49.93
Frost (ecp) | 44.85/21.83/41.80 |  64.34/49.85/54.98 | 53.17
Frost (ecpp) | 45.11/22.11/42.01 |  64.28/49.86/54.96 | 53.22
XSum
Lead-1 | 16.66/1.85/12.26 | 21.45/4.36/19.48 | 5.36
Ext-Oracle∗ | 28.81/8.61/21.97 | 32.64/13.34/28.85 | 18.78
RoBERTaShare | 38.52/16.12/31.13 | – | –
MASS | 39.75/17.24/31.95 | – | –
BART | 45.14/22.27/37.25 | – | –
GSum | 45.40/21.89/36.67 | – | –
Pegasus | 47.60/24.83/39.64 | – | –
Pegasus (ours) | 47.56/24.87/39.40 | 62.15/39.76/56.36 | 53.48
Frost (ecp) | 47.44/24.54/39.24 | 63.57/40.45/57.65 | 54.62
Frost (ecpp) |  47.80/25.06/39.76 |  64.09/41.07/58.18 | 55.49
SAMSum
Gliwa et al. (2019) | 40.99/17.72/38.30 | – | –
Pegasus (ours) |  52.27/28.34/47.83 | 72.42/51.07/64.85 | 81.17
Frost (ecp) |  52.39/27.70/47.82 |  74.42/55.32/66.35 | 83.60
Frost (ecpp) | 51.86/27.67/47.52 |  75.02/55.19/66.80 | 83.60
BillSum
Pegasus | 59.67/41.58/47.59 | – | –
Pegasus (ours) |  59.33/41.60/54.80 | 69.99/62.79/66.17 | 61.91
Frost (ecp) | 58.76/40.34/54.03 | 70.84/63.59/66.86 | 62.38
Frost (ecpp) |  59.50/41.17/54.85 |  71.67/64.56/67.79 | 63.38
Table 2: Final results on abstractive summarization datasets compared with the
previous SOTA. We report results from RoBERTaShare Rothe et al. (2020a), MASS
Song et al. (2019), UniLM Dong et al. (2019), T5 Raffel et al. (2019), BART
Lewis et al. (2019), ProphetNet Qi et al. (2020), Gliwa et al. (2019), CTRLsum
He et al. (2020), GSum Dou et al. (2020) and Pegasus Zhang et al. (2019). We
also include commonly reported Lead-$n$ baselines (selecting top $n$ sentences
from the document) and extractive oracle (Ext-Oracle; selecting best set of
sentences from the document with the most overlapping content with its
reference summary), for the CNN/DailyMail and XSum datasets. Ext-Oracles are
masked with $\ast$ and are not directly comparable. All results are reported
on the publicly available test sets.

###  5.2 Abstractive Summarization Results

Table 2 presents our final results from finetuning our large models on
summarization datasets. For each dataset, we first report results from earlier
work directly taken from corresponding papers; our results are in the bottom
blocks for each dataset. We finetune our own Pegasus using the standard
approach (document to summary). We report results from Frost (ecp; Entity
Chain Planning), i.e., Pegasus finetuned with the Frost objective (document to
content plan and summary). Finally, we report on Frost (ecpp; Entity Chain
Planning with Pretraining), our models both pretrained and finetuned with the
Frost objective.

We find that even simply finetuning an existing Pegasus pretrained model to do
content planning and summarization (as in Frost (ecp)) leads to improvements
in entity specificity (EntF1) and the quality of entity plans in summaries
(Entity Planning rouge), across all datasets. In fact in some cases, better
content plans lead to large improvements in summary-level rouge as well, for
example, Frost (ecp) improve on Pegasus from 44.05/21.69/40.98 to
44.85/21.83/41.80 on rouge scores for CNN/DailyMail summaries.

The pretraining for content planning and generation in Frost (ecpp) further
improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the
summary-level rouge for CNN/DailyMail and XSum. Our Frost models establish new
state-of-the-art rouge results on XSum. For CNN/DailyMail, we perform inferior
to CTRLsum He et al. (2020) and GSum Dou et al. (2020) on rouge scores.
However, we outperform CTRLsum on entity planning and specificity. Further
discussion on comparing our methods to controlled summarization systems such
as CTRLsum can be found in Section 5.5.

Model | Test set with Extractive Entity Chains Only (3.5k) | Test set with Non-Extractive Entity Chains Only (7.8k)
---|---|---
Summary | Entity Planning | Specificity | Summary | Entity Planning | Specificity
R1/R2/RL | R1/R2/RL | EntF1 | R1/R2/RL | R1/R2/RL | EntF1
Models trained on the original XSum dataset (204k/11.3k/11.3k)
Pegasus | 44.67/21.75/36.37 | 61.54/28.84/58.02 | 60.88 | 48.85/26.26/40.75 | 62.42/44.64/55.62 | 50.17
Frost (ecp) | 44.53/21.38/36.28 | 63.60/29.20/60.01 | 62.04 | 48.74/25.95/40.56 | 63.56/45.48/56.60 | 51.31
Frost (ecpp) | 45.08/22.01/36.80 | 63.73/30.08/60.10 | 63.03 | 49.02/26.42/41.08 | 64.25/45.98/57.32 | 52.12
($d\rightarrow c_{\text{drop}};s$) | 44.97/21.97/36.77 | 67.10/30.19/63.58 | 64.41 | 44.93/21.41/37.39 | 48.89/30.58/43.76 | 33.68
($d\rightarrow c_{\text{oracle}};s$)∗ | 50.26/27.33/43.12 | 98.55/53.94/98.53 | 98.36 | 61.80/40.55/56.20 | 99.10/92.10/99.08 | 98.69
Models trained on the filtered set with extractive entity chains only
(62.7k/3.5k/3.5k)
Pegasus | 44.10/21.24/35.87 | 65.66/28.69/61.86 | 63.47 | 44.72/21.80/36.94 | 52.87/35.39/46.99 | 40.77
Frost (ecp) | 44.29/21.26/36.04 | 67.05/29.11/63.02 | 64.69 | 44.02/20.91/36.21 | 49.59/31.16/43.55 | 38.97
Frost (ecpp) | 44.87/22.05/36.79 | 66.43/30.00/62.48 | 65.97 | 44.28/21.15/36.59 | 52.17/34.40/46.25 | 39.10
($d\rightarrow c_{\text{drop}};s$) | 44.56/21.80/36.53 | 65.94/29.51/62.04 | 65.08 | 42.93/19.41/35.40 | 47.29/28.27/41.95 | 31.17
($d\rightarrow c_{\text{oracle}};s$)∗ | 49.60/26.64/42.41 | 97.53/53.35/97.51 | 97.41 | 59.80/38.00/54.10 | 97.95/90.57/97.84 | 97.11
Table 3: Performance on XSum summaries when models are trained on the dataset
with extractive entity chains only (data filtering, the bottom block) and when
novel entities are dropped from the predicted entity chains using the drop-
prompt mechanism ($c_{\text{drop}}$). Results with ∗ are with oracle entity
chain prompts. We report results on the filtered test set with extractive
entity chains only (3.5k) and the rest of the test set with novel entities in
the targets (7.8k). Best results are bold faced.

###  5.3 Controlling Hallucinations

We demonstrate in two ways the planning with entity chains can be useful in
mitigating hallucinations in summarization: data filtering and drop-prompt
mechanism with entity chains. We focused on the XSum dataset for these
experiments.

#### Data Filtering using Entity Chains.

During training Frost prepends reference summaries with their entity chains to
better plan them. We leverage this to filter the dataset to only keep examples
where summaries have fully extractive entity chains; an entity chain is fully
extractive if all entities in it can be found in the input document. It’s
notable that this filtered dataset will not contain novel entities in the
summary targets. Frost models trained on this data will ground the summary
generation to extracted entity chains while allowing abstraction for non-
entity related generations. The resulting XSum dataset has 62.7k/3.5k/3.5k
train/validation/test instances. We finetune our models from Table 2 on this
filtered dataset and report their performance on the filtered test set (3.5k).
We also evaluate them on the rest of the test set (7.8k) where reference
entity chains are not fully extractive to their input documents.999Such data
divergence is not unique to XSum, around 30% of CNN/DailyMail summaries also
have reference entity chains that are not fully extractive to their input
documents. Writing these summaries requires either document-level inference or
the background knowledge of the input documents to generate novel entities or
numbers. See Figure 6 for examples.

#### Drop-Prompt Mechanism.

Frost decoders are trained to generate the summary $s$ following the
generation of its plan $c$. To improve faithfulness, we take the predictions
from Frost (ecpp) 101010The drop-prompt mechanism can be used with Frost (ecp)
also, we have simply used the best among Frost (ecp) and Frost (ecpp) on the
XSum set in terms of summary-level rouge, entity planning and EntF1. and
modify the generated plan $c$ to $c_{\text{drop}}$ by dropping entities (or
parts of them) that are not found in the input document. We then prompt our
decoder with $c_{\text{drop}}$ to generate a new summary. We conduct this with
both models, one trained on the full dataset and another on the filtered
subset. We also report results for the oracle entity chain prompts
$c_{\text{oracle}}$ for a comparison.

Gold: Walsall have signed defender Luke Leahy on a two-year contract from
Scottish Championship side Falkirk.
---
Trained on the original XSum dataset
Pegasus: Walsall have signed defender Paddy Leahy from Scottish Championship
side Falkirk on a three-year deal.
Frost (ecp): [ENTITYCHAIN] Walsall | Falkirk | Paddy Leahy | two [SUMMARY] Walsall have signed Falkirk defender Paddy Leahy on a two-year deal.
Frost (ecpp): [ENTITYCHAIN] Walsall | Falkirk | Liam Leahy | two [SUMMARY] Walsall have signed Falkirk defender Liam Leahy on a two-year deal.
Frost (ecpp), $c_{\text{drop}}$: [ENTITYCHAIN] Walsall | Falkirk | Leahy [SUMMARY] Walsall have signed Falkirk defender Leahy on a free transfer.
Trained on the filtered XSum dataset
Pegasus: Walsall have signed Falkirk defender Declan Leahy for an undisclosed
fee.
Frost (ecp): [ENTITYCHAIN] Walsall | Falkirk | Paddy Leahy [SUMMARY] Walsall have signed Falkirk defender Paddy Leahy for an undisclosed fee.
Frost (ecpp): [ENTITYCHAIN] Walsall | Falkirk | Conor Leahy [SUMMARY] Walsall have signed Falkirk defender Conor Leahy for an undisclosed fee.
Frost (ecpp), $c_{\text{drop}}$: [ENTITYCHAIN] Walsall | Falkirk | Leahy [SUMMARY] Walsall have signed Falkirk defender Leahy for an undisclosed fee.
Figure 6: Example XSum predictions for models presented in Table 3 and 4. We highlight entities in orange that are not faithful to the input document. Entities in green are faithful to the input document.  Model | Test set with Extractive Entity Chains Only (3.5k) | Test set with Non-Extractive Entity Chains Only (7.8k)
---|---|---
Faithfulness | Overall | Faithfulness | Overall
Entail. | EntPrec | Human | Agree. | Human | Agree. | Entail. | EntPrec | Human | Agree. | Human | Agree.
Models trained on the original XSum dataset (204k/11.3k/11.3k)
Pegasus | 0.613 | 0.800 | 4.20 | 0.74 | 4.09 | 0.69 | 0.402 | 0.361 | 3.15 | 0.71 | 2.93 | 0.80
Frost (ecp) | 0.606 | 0.770 | 4.22 | 0.75 | 4.23 | 0.70 | 0.379 | 0.317 | 3.11 | 0.73 | 2.85 | 0.77
Frost (ecpp) | 0.589 | 0.751 | 4.13 | 0.72 | 4.11 | 0.66 | 0.371 | 0.357 | 3.31 | 0.79 | 2.81 | 0.77
($d\rightarrow c_{\text{drop}};s$) | 0.650 | 0.943 | 4.09 | 0.73 | 4.09 | 0.64 | 0.441 | 0.746 | 3.53 | 0.75 | 3.13 | 0.79
Models trained on the filtered set with extractive entity chains only
(62.7k/3.5k/3.5k)
Pegasus | 0.667 | 0.887 | 4.39 | 0.80 | 4.14 | 0.69 | 0.389 | 0.501 | 3.37 | 0.79 | 3.09 | 0.76
Frost (ecp) | 0.581 | 0.858 | 4.27 | 0.76 | 4.16 | 0.66 | 0.442 | 0.548 | 3.43 | 0.72 | 3.01 | 0.73
Frost (ecpp) | 0.502 | 0.806 | 4.19 | 0.77 | 4.19 | 0.66 | 0.465 | 0.491 | 3.55 | 0.76 | 3.16 | 0.76
($d\rightarrow c_{\text{drop}};s$) | 0.533 | 0.943 | 4.41 | 0.75 | 4.18 | 0.72 | 0.453 | 0.826 | 3.85 | 0.74 | 3.46 | 0.77
Table 4: Faithfulness assessment using automatic (Entailment and EntPrec) and
human evaluations, and overall quality assessment by humans for models
presented in Table 3. Following Durmus et al. (2020), agreement (Agree.) is
computed by taking the percentage of the annotators that annotate the majority
class for the given (document, summary) pair. Best results are bold faced.

#### Effect on Summary-level rouge, Entity Planning and Specificity.

Results are presented in Table 3. First of all, similar to our results in
Table 2 on the full XSum test set, our results with the models trained on the
original dataset further validate that the pretraining and finetuning with the
content planning objective is equally useful for both test sets: one where
entities are simply copied from the input documents and another where novel
entities were inferred, to generate corresponding targets.

The data filtering using entity chains are particularly useful for test cases
with extractive reference entity chains; models trained on the filtered data
lead to much higher EntF1 (e.g., 65.97 vs 63.03, for Frost (ecpp)) and higher
quality entity plans (entitly-level rouge of 66.43/30.00/62.48 vs
63.73/30.08/60.10 for Frost (ecpp)), without sacrificing the summary quality
substantially (summary-level rouge of 44.87/22.05/36.79 vs 45.08/22.01/36.80
for Frost (ecpp)). Unsurprisingly, these models don’t do well on the test set
with non-extractive reference entity chains, such examples were not seen
during training.

The drop-prompt mechanism works particularly well for models trained on the
full dataset and evaluated on the test set with extractive reference entity
chains. For example, the entity planning rouge scores and EntF1 for Frost
models improve from 63.73/30.08/60.10 to 67.10/30.19/63.58, and, 63.03 to
64.41, respectively. Interestingly, we do not observe a significant drop in
the summary-level rouge scores (45.08/22.01/36.80 vs 44.97/21.97/36.77).
Basically, our results demonstrate that when models are trained on the noisy
data (with data divergence issues, common in summarization datasets Dhingra et
al. (2019); Maynez et al. (2020)), the drop-prompt mechanism is very effective
in generating high quality summaries when entities need to be simply copied
from the input documents.

The drop-prompt mechanism doesn’t help models when they are already trained on
the filtered training set. Also this mechanism is counter intuitive to use for
the test set where novel entities need to be inferred to generate targets, we
observe drops for models irrespective of how they were trained, either using
the full training set or the filtered set.

#### Effect on Faithfulness.

The planning objective in Frost itself does not ensure faithfulness; planning
is done with the entities in the target, if the target is noisy, its plan may
also be noisy (see Figure 6 for predicted entity chains with hallucinated
entities). But the planning with the entity chains facilitates the data
filtering and the drop-prompt mechanism, using entity chains. Table 4 presents
our results assessing them for faithfulness. We randomly selected 50 documents
for each of test sets (with extractive or non-extractive reference entity
chains) and assessed their summaries from all 8 systems (except with
$c_{\text{oracle}}$) from Table 3.

The data filtering using entity chains is extremely useful for improving
faithfulness in summaries. We see improvements for all models trained on the
filtered set (Table 4, bottom) compared to their counterparts trained on the
full training set (Table 4, top), when evaluated using EntPrec and by humans
for faithfulness. We don’t observe similar improvements in entailment.
Contrary to findings in Maynez et al. (2020), our results suggest that the
entailment scores are not a reliable indicator of faithfulness for documents
with extractive reference entity chains.

The drop-prompt mechanism is also very powerful in improving faithfulness.
Irrespective of which training data were used to train the models and which
test sets they were evaluated on, we see improvements in entailment scores,
EntPrec and the human assessment of faithfulness across the board, except for
a single case where the entailment score slightly drops from 0.465 to 0.453.
Figure 6 demonstrates how we drop hallucinated entities ‘Liam’, ‘two’ and
‘Conor’ from entity chains to enforce models to generate faithful summaries
grounded to the modified entity chains.

We achieve the best performance in terms of EntPrec and the human assessment
of faithfulness when both the data filtering and the drop-prompt mechanism
were used. We achieve 0.943 for EntPrec and 4.41 for faithfulness for the test
set with extractive entity chains only, and 0.826 for EntPrec and 3.85 for
faithfulness for the test set with non-extractive entity chains. Humans also
found that the predictions from these models were the best in terms of overall
summary quality.

Finally, we carried out pairwise comparisons for human assessments for
faithfulness and overall summary quality for all models (using a one-way ANOVA
with post-hoc Tukey HSD tests; $p<0.01$). Interestingly, differences among all
model pairs for both faithfulness and overall summary quality are
insignificant when evaluated on the test set with extractive reference entity
chains only. On the more challenging test set with non-extractive entity
chains, Frost (ecpp) trained on the filtered training set and with the drop-
prompt mechanism is significantly better than all other models except for (i)
Frost (ecpp) trained on the filtered training set but without the drop-prompt
mechanism and (ii) Frost (ecpp) trained on the original training set and with
the drop-prompt mechanism, for both faithfulness and overall summary quality.
All other pairwise differences are insignificant.

Gold:  [ENTITYCHAIN] BP | Lord Browne | North Sea [Summary] Former BP chief executive Lord Browne has warned North Sea oil operators’ costs must fall in order for them to compete globally.
---
Frost (ecpp): [ENTITYCHAIN] Browne | BP [Summary] Lord Browne, the former boss of BP, has warned that some oil companies could go out of business if costs continue to rise.
Frost (ecpp), $c_{\text{mod}}$, Style diversity
[ENTITYCHAIN] BP | Browne [Summary] The former boss of BP, Lord Browne, has warned that some oil companies could go out of business if costs continue to rise.
[ENTITYCHAIN] BBC | BP | Browne [Summary] In an interview with the BBC, former BP chief executive Lord Browne has warned the oil and gas industry is in danger of "going to the wall".
Frost (ecpp), $c_{\text{mod}}$, Topical diversity
[ENTITYCHAIN] Brent Crude [Summary] The former boss of the world’s biggest oil
company has warned that the price of Brent Crude will not recover until next
year.
[ENTITYCHAIN] Brent Crude | Unions [Summary] Oil companies will have to cut costs dramatically if they are to survive the fall in the price of Brent Crude, according to the former chief executive of the Unions.
Figure 7: An example of generating summaries with topical and style diversity using modified entity prompts $c_{\text{mod}}$. Model | XSum | CNN/DailyMail
---|---|---
Summary | Entity Planning | Specificity | Avg. | Summary | Entity Planning | Specificity | Avg.
R1/R2/RL | R1/R2/RL | EntF1 | Length | R1/R2/RL | R1/R2/RL | EntF1 | Length
Pegasus ($d\rightarrow s$) | 47.56/24.87/39.40 | 62.15/39.76/56.36 | 53.48 | 22.27 | 44.05/21.69/40.98 | 61.12/46.46/52.40 | 49.93 | 68.65
CTRLsum ($k;d\rightarrow s$) | – | – | – | – | 45.65/22.35/42.50 | 62.19/48.35/52.53 | 51.69 | 74.46
Frost ($d\rightarrow c;s$) | 47.80/25.06/39.76 | 64.09/41.07/58.18 | 55.49 | 21.09 | 45.11/22.11/42.01 | 64.28/49.86/54.96 | 53.22 | 65.75
CTRLsum ($k_{\text{oracle}};d\rightarrow s$) | – | – | – | – | 64.72/40.56/61.02 | 78.18/67.39/66.17 | 71.35 | 71.76
Pegasus ($d;c_{\text{oracle}}\rightarrow s$) | 56.58/34.64/49.87 | 94.48/75.17/93.32 | 93.59 | 22.12 | 56.46/33.62/53.51 | 86.35/80.54/83.76 | 82.41 | 64.84
Pegasus ($c_{\text{oracle}};d\rightarrow s$) | 57.60/35.62/51.11 | 98.13/79.27/97.58 | 97.90 | 22.13 | 61.66/38.43/58.75 | 97.97/96.11/97.72 | 97.43 | 64.24
Frost ($d\rightarrow c_{\text{oracle}};s$) | 58.24/36.47/52.16 | 98.93/80.32/98.91 | 98.59 | 21.80 | 61.85/38.95/59.00 | 98.02/96.19/97.96 | 97.31 | 63.86
Table 5: Comparison of Frost with encoder-guided control summarization. The
results in the top block are repeated from Table 2 for comparison. The bottom
block presents oracle results with access to oracle keywords
($k_{\text{oracle}}$) in CTRLSum or oracle entity promts ($c_{\text{oracle}}$)
in Frost and Pegasus. $d$ and $s$ stand for the input document and the output
summary, respectively. All the results are reported on the full test sets.

###  5.4 Generating Diverse Summaries

Frost provides a handle to easily control or manipulate content plan for
predicted summaries. In Figure 6, we saw how we can use this to control
entity-level hallucinations, but the strength of Frost models go beyond this.
Figure 7 shows how Frost models can be effectively used in generating
summaries with different entity focus by simply modifying the entity prompt.
Future work will focus on how to leverage Frost for synthetic data generation
for training improvements.

###  5.5 Comparison with Encoder-Guided Control Summarization

We compare the decoding strategy in Frost to encoder-guided control
summarization systems He et al. (2020); Dou et al. (2020). Table 5 presents
our results. In particular, we report on CTRLsum He et al. (2020); first, a
keyword extraction system (BERT-based sequence tagger) is used to extract
keywords ($k$) from the input document $d$, and then, the extracted keywords
are encoded along with the input document (as $k;d$) to generate the summary.
We also finetune Pegasus where the entity chain ($c$) is encoded along with
the input document $d$ (as $d;c$ or $c;d$). We only report the oracle results
(with $c_{\text{oracle}}$) for these models; like CTRLsum, generating an
entity chain $c$ during inference will require training an additional entity
chain generator, which is out of scope of this paper.

There are several advantages of Frost-style decoding. Unlike encoder-guided
control summarization systems, Frost models can be used in a usual way to
generate summaries for input documents without relying on external systems to
augment them during inference. Additionally, users can modify the generated
entity prompts or provide their desired entity prompts to control the
generated summaries (see Section 5.3 and 5.4). To the best of our knowledge,
Frost is the first decoder-prompted control summarization model.

Our results in Table 5 show that the prompting the decoder with entity prompts
are more effective in generating high-quality and grounded summaries compared
to when entity prompts are encoded with the input, especially for abstractive
summaries; both Pegasus versions ($d;c_{\text{oracle}}$ and
$c_{\text{oracle}};d$) perform inferior to Frost ($d\rightarrow
c_{\text{oracle}};s$) in terms of summary-level rouge, entity-level rouge and
EntF1 on the XSum dataset. CTRLsum ($k_{\text{oracle}};d\rightarrow s$)
achieves better rouge scores than Frost ($d\rightarrow c_{\text{oracle}};s$)
on CNN/DailyMail (64.72/40.56/61.02 vs 61.85/38.95/59.00). This is not
surprising as the oracle keywords $k_{\text{oracle}}$ in CTRLSum retain most
words from the summary found in the source document; as such due to the
extractive nature of CNN/DailyMail summaries, $k_{\text{oracle}}$ tend to be
closer to the surface forms of the reference summaries (rouge scores between
$k_{\text{oracle}}$ and reference summaries are 52.70/17.37/44.37). The
difference in rouge scores narrows down between CTRLsum and Frost to
45.65/22.35/42.50 vs 45.11/22.11/42.01 with automatically extracted keywords.
Frost outperforms CTRLsum on entity planning and specificity. It is not clear
how well CTRLsum’s keyword extraction system will generalize to more
abstractive datasets where words in reference summaries are often not found in
the source document.

Finally, our Frost models work as a better microplanner and produce concise
summaries than those generated by systems without doing content planning or
when its done on the input side. For example, the average lengths of
CNN/DailyMail test summaries are 68.65, 74.46, 65.75 and 60.70 tokens for
Pegasus, CTRLSum, Frost and humans, respectively. See example predictions
comparing CTRLSum and Frost in Figure 10, Appendix C.

##  6 Conclusion

In this work, we proposed the use of entity chains, both during pretraining
and finetuning, to better plan and ground the generation of abstractive
summaries. Our approach achieves state-of-the-art rouge results in XSum and
SAMSum, and competitive results in CNN/Dailymail. Compared to other guided
summarization models, CTRLSum and GSum, which perform slightly better in
CNN/Dailymail, our approach is drastically simpler to implement, trains by
augmenting the targets only, utilizes no additional parameters than the
baseline Pegasus model, and does not rely on any external systems to augment
the inputs during inference. We further demonstrate that by modifying the
entity chain prompts, we can easily control hallucinations in summaries.

## References

  * Azzam et al. (1999) Saliha Azzam, Kevin Humphreys, and Robert Gaizauskas. 1999.  Using coreference chains for text summarization.  In _Coreference and Its Applications_.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  _CoRR_ , abs/1409.0473.
  * Barzilay and Elhadad (1997) Regina Barzilay and Michael Elhadad. 1997.  Using lexical chains for text summarization.  In _Intelligent Scalable Text Summarization_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhingra et al. (2019) Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019.  Handling divergent reference texts when evaluating table-to-text generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4884–4895, Florence, Italy. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 13042–13054. Curran Associates, Inc.
  * Dou et al. (2020) Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020\.  Gsum: A general framework for guided neural abstractive summarization.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018.  Controllable abstractive summarization.  _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_.
  * Gabriel et al. (2020) Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2020\.  Go figure! a meta evaluation of factuality in summarization.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.
  * Halliday and Hasan (1976) M. A. K. Halliday and Ruqaiya Hasan. 1976.  _Cohesion in English_.  Longman, London.
  * He et al. (2020) Junxian He, Wojciech Kryściński, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2020.  Ctrlsum: Towards generic controllable text summarization.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 1693–1701. Curran Associates, Inc.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hua and Wang (2020) Xinyu Hua and Lu Wang. 2020.  PAIR: Planning and iterative refinement in pre-trained transformers for long text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 781–793, Online. Association for Computational Linguistics.
  * Jones (1993) Karen Sparck Jones. 1993.  What might be in a summary?  _Information Retrieval_ , pages 9–26.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. 2019.  Sample efficient text summarization using a single pre-trained transformer.  _CoRR_ , abs/1905.08836.
  * Kiddon et al. (2016) Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016.  Globally coherent text generation with neural checklist models.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 329–339, Austin, Texas. Association for Computational Linguistics.
  * Kornilova and Eidelman (2019) Anastassia Kornilova and Vladimir Eidelman. 2019.  BillSum: A corpus for automatic summarization of US legislation.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 48–56, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Evaluating the factual consistency of abstractive text summarization.  _CoRR_ , abs/1910.12840.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _CoRR_ , abs/1910.13461.
  * Lin and Hovy (2003) Chin Yew Lin and Eduard Hovy. 2003.  Automatic evaluation of summaries using n-gram co-occurrence statistics.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 150–157.
  * Luhn (1958) H. P. Luhn. 1958.  The automatic creation of literature abstracts.  _IBM J. Res. Dev._ , 2(2):159–165.
  * Mann and Thompson (1988) William C Mann and Sandra A Thompson. 1988.  Rhetorical structure theory: Toward a functional theory of text organization.  _Text_ , 8(3):243–281.
  * Mao et al. (2020) Yuning Mao, Xiang Ren, Heng Ji, and Jiawei Han. 2020.  Constrained abstractive summarization: Preserving factual consistency with constrained generation.
  * Martin et al. (2018) Lara J. Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O. Riedl. 2018.  Event representations for automated story generation with deep neural nets.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 868–875. AAAI Press.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * McKeown and Radev (1995) Kathleen McKeown and Dragomir R. Radev. 1995.  Generating summaries of multiple news articles.  In _Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’95, page 74–82, New York, NY, USA. Association for Computing Machinery.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Narayan et al. (2020) Shashi Narayan, Gonçalo Simões, Ji Ma, Hannah Craighead, and Ryan T. McDonald. 2020.  QURIOUS: question generation pretraining for text generation.  _CoRR_ , abs/2004.11026.
  * Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018.  Multi-reward reinforced summarization with saliency and entailment.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 646–653. Association for Computational Linguistics.
  * Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.  Data-to-text generation with content selection and planning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):6908–6915.
  * Qi et al. (2020) Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020.  ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2401–2410, Online. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _Technical report, OpenAI_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1901.07291.
  * Reiter and Dale (1997) Ehud Reiter and Robert Dale. 1997.  Building applied natural language generation systems.  _Natural Language Engineering_ , 3(1):57–87.
  * Rothe et al. (2020a) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020a.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ , 8:264–280.
  * Rothe et al. (2020b) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020b.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ , 8:264–280.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pages 4596–4604. PMLR.
  * Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.  MASS: Masked sequence to sequence pre-training for language generation.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97, pages 5926–5936. PMLR.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 3104–3112. Curran Associates, Inc.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30_ , pages 5998–6008.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1112–1122. Association for Computational Linguistics.
  * Wiseman et al. (2018) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2018.  Learning neural templates for text generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3174–3187, Brussels, Belgium. Association for Computational Linguistics.
  * Yao et al. (2019) Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019.  Plan-and-write: Towards better automatic storytelling.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 7378–7385. AAAI Press.
  * Zhang et al. (2019) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.

##  Appendix A Hyperparameters for Pretraining and Finetuning

Table 6 and 7 present hyperparameters used for pretraining and finetuning
Pegasus and Frost base and large sized models.

##  Appendix B Human Evaluation Instructions

Figure 8 and 9 show the exact instructions and the template used by our
annotators for faithfulness and overall summary quality, respectively.

##  Appendix C Comparison of Frost and CTRLSum

Figure 10 presents example predictions from Frost and CTRLSum along with their
entity prompts and keywords used for guidance, respectively.

Model, Size | Learning | Label | Init. | Steps | Batch | Corpus | Lengths (token)
---|---|---|---|---|---|---|---
rate | smoothing | size | input | target
Pegasus, base | 0.01 | 0.1 | random | 1.5m | 1024 | C4/Hugenews | 512 | 256
Frost(F), base | 0.01 | 0.1 | random | 1.5m | 1024 | C4/Hugenews | 512 | 256
Frost(P+F), base | 0.01 | 0.1 |  Pegasus (1m) | 0.5m | 1024 | C4/Hugenews | 512 | 256
Frost(P+F), large | 0.01 | 0.1 |  Pegasus (1.5m) | 1.5m | 1024 | C4/Hugenews | 512 | 256
Table 6: Hyperparameters used for pretraining Pegasus and Frost base (223M parameters) and large (568M parameters) models. See text in Section 5.1 for more details and how these models were ablated.  Dataset | Learning | Label | Steps | Batch | Beam | Beam | Lengths (token)
---|---|---|---|---|---|---|---
rate | smoothing | size | size | alpha | input | target
BillSum | 1e-4 | 0.1 | 200k | 256 | 8 | 0.8 | 1024 | 256
CNN/DailyMail | 1e-4 | 0.1 | 175k | 256 | 8 | 0.8 | 1024 | 256
Samsum | 1e-4 | 0.1 | 20k | 256 | 8 | 0.8 | 512 | 128
XSum | 1e-4 | 0.1 | 100k | 256 | 8 | 0.8 | 512 | 128
Table 7: Hyperparameters used for finetuning Pegasus and Frost models on
summarization datasets.
---
Figure 8: Instructions for human evaluations for faithfulness.
---
Figure 9: Instructions for human evaluations for overall quality of summaries.
Gold: Theia, a bully breed mix, was apparently hit by a car, whacked with a
hammer and buried in a field. “She’s a true miracle dog and she deserves a
good life,” says Sara Mellado, who is looking for a home for Theia.
---
CTRLSum: (Oracle keywords) hit car apparently whacked hammer buried field | dog bully breed mix Theia | true miracle deserves good life (Summary) A bully breed mix named Theia was hit by a car, apparently whacked with a hammer and buried in a field. Four days after her apparent death, the dog managed to stagger to a nearby farm. “She’s a true miracle dog and she deserves a good life,” the dog’s foster mother says.
Frost: (Oracle Entity Prompt) Theia ||| Sara Mellado | Theia (Summary) The dog, now named Theia, was apparently hit by a car and buried. Sara Mellado is raising money to help pay for Theia’s medical care.
CTRLSum: (Predicted keywords) stray pooch Washington State hit car head hammer mercy killing buried field survive | dog Theia | found (Summary) A stray pooch in Washington State was hit by a car and buried in a field. The dog was apparently hit on the head with a hammer in a mercy killing. She managed to survive and was found by a worker who took her to a vet. Theia is now being cared for at a veterinary hospital.
Frost: (Predicted Entity Prompt) Washington State ||| Theia ||| (Summary) A
stray dog in Washington State was apparently hit by a car and buried. The dog,
now named Theia, survived but still needs surgery. A fundraising page has been
set up to help pay for her care.
Figure 10: Example predictions from Frost and CTRLSum.

##  Appendix D Ablation on Entity Chains with Named Entities, Dates and
Numbers

We experimented with entity chains consisting of named entities, dates and
numbers. We finetuned Pegasus-large models on the XSum dataset to predict the
entity chain and then continue predicting the summary. Results are presented
in Figure 11.

We found that the entity chains with all named entities, dates and numbers
performed best for entity level measures (entity chain-level rouge and EntF1)
without sacrificing the final summary quality (summary-level rouge)
significantly. Following this, we always use entity chains with named
entities, dates and numbers to have better control over them.

RL-SumR2-EPlanEntF1$40$$45$$50$$55$FscoresNoneNENE/DNE/D/Num Figure 11: Ablations on entity chains with named entities, dates and numbers. We report summary-level rouge-l (RL-Sum), entity chain-level rouge-2 (R2-EPlan) and EntF1 on the XSum validation set. Similar observations were made for other measures, see detailed results in Table 8.  EntityChains | ROUGE | Entity Accuracy
---|---|---
Average | Corpus | Total Count (ref=31572)
R1/R2/RL | R/P/F | R/P/F | found/predicted
src $\rightarrow$ tgt
None | 47.57/25.05/39.59 | 54.83/60.37/53.99 | 54.17/58.02/56.03 | 17103/29480
src $\rightarrow$ ec$|$tgt
NE | 47.58/24.84/39.57 | 55.77/61.06/55.01 | 54.98/59.06/56.94 | 17357/29391
NE,D | 47.57/24.88/39.59 | 55.56/61.41/55.09 | 54.74/59.59/57.06 | 17284/29007
NE,D,Num | 47.36/24.69/39.44 | 55.52/61.43/55.18 | 54.73/60.04/57.26 | 17280/28779
Table 8: Detailed results for ablations on entity chains with named entities,
dates and numbers in Figure 11. We report results on XSum validation set.

##  Appendix E Ablation on Sentence-Level Vs Summary-Level Entity Chains

Table 9 presents detailed results for ablations on sentence-level vs summary-
level entity chains in Figure 3. We report results on CNN/DailyMail validation
set.

EntityChains | ROUGE | Entity Accuracy
---|---|---
Average | Corpus | Total Count (ref=92225)
R1/R2/RL | R/P/F | R/P/F | found/predicted
src $\rightarrow$ tgt
None | 45.01/22.57/41.99 | 55.29/51.86/50.66 | 54.17/49.92/51.96 | 49959/100083
src $\rightarrow$ ec$|$tgt
Sentence-Level | 44.71/22.51/41.83 | 54.45/54.83/52.06 | 52.87/53.93/53.40 | 48762/90416
Summary-Level | 45.70/22.61/42.69 | 57.49/54.87/53.82 | 56.08/54.02/55.03 | 51716/95740
Table 9: Detailed results for ablations on sentence-level vs summary-level entity chains in Figure 3. We report results on CNN/DailyMail validation set. Model | Test set with Extractive Entity Chains Only (7.8k) | Test set with Non-Extractive Entity Chains Only (3.7k)
---|---|---
Summary | Entity Planning | Specificity | Summary | Entity Planning | Specificity
R1/R2/RL | R1/R2/RL | EntF1 | R1/R2/RL | R1/R2/RL | EntF1
Models trained on the original CNN/DailyMail dataset (287k/13.4k/11.5k)
Pegasus | 45.09/22.99/42.04 | 61.61/47.88/53.41 | 52.22 | 41.88/18.98/38.77 | 60.10/43.49/50.29 | 45.15
Frost (ecp) | 45.80/22.96/42.73 | 64.78/51.19/56.02 | 55.49 | 42.87/19.47/39.86 | 63.42/47.05/52.81 | 48.33
Frost (ecpp) | 46.14/23.36/43.04 | 64.74/51.21/56.00 | 55.63 | 42.96/19.50/39.86 | 63.32/47.04/52.79 | 48.19
($d\rightarrow c_{\text{drop}};s$) | 46.14/23.43/43.06 | 64.75/51.18/56.03 | 55.62 | 42.87/19.45/39.76 | 62.96/46.64/52.48 | 47.96
($d\rightarrow c_{\text{oracle}};s$)∗ | 61.56/39.03/58.68 | 98.08/95.71/98.01 | 97.49 | 62.46/38.78/59.67 | 97.89/97.19/97.86 | 96.93
Models trained on the filtered set with extractive entity chains only
(203.5k/9k/7.8k)
Pegasus | 45.24/22.93/42.13 | 62.12/48.15/53.75 | 52.65 | 41.87/18.79/38.79 | 60.17/43.43/50.23 | 44.96
Frost (ecp) | 45.76/22.96/42.68 | 64.83/51.02/56.15 | 55.35 | 42.64/19.16/39.50 | 62.67/46.42/52.26 | 47.63
Frost (ecpp) | 46.11/23.41/43.00 | 65.03/51.35/56.31 | 55.72 | 42.53/19.12/39.42 | 62.81/46.53/52.42 | 47.69
($d\rightarrow c_{\text{drop}};s$) | 46.07/23.42/42.98 | 65.01/51.32/56.31 | 55.65 | 42.49/19.13/39.40 | 62.69/46.41/52.33 | 47.65
($d\rightarrow c_{\text{oracle}};s$)∗ | 61.25/38.72/58.32 | 97.88/95.46/97.78 | 97.23 | 61.84/37.92/59.03 | 97.66/96.66/97.53 | 96.15
Table 10: Performance on CNN/DailyMail summaries when models are trained on the dataset with extractive entity chains only (data filtering, the bottom block) and when novel entities are dropped from the predicted entity chains using the drop-prompt mechanism ($c_{\text{drop}}$). Results with ∗ are with oracle entity chain prompts. We report results on the filtered test set with extractive entity chains only (7.8k) and the rest of the test set with novel entities in the targets (3.7k). Model | Test set with Non-Extractive Entity Chains Only (3.7k)
---|---
Faithfulness | Overall
Entail. | EntPrec | Human | Human
Trained on the original CNN/DailyMail dataset (204k/11.3k/11.3k)
Pegasus | 0.718 | 0.942 | 4.67 | 4.21
Frost (ecp) | 0.785 | 0.946 | 4.63 | 4.09
Frost (ecpp) | 0.721 | 0.907 | 4.63 | 4.12
($d\rightarrow c_{\text{drop}};s$) | 0.798 | 0.966 | 4.63 | 4.21
Trained on the filtered set with extractive entity chains only
(62.7k/3.5k/3.5k)
Pegasus | 0.751 | 0.941 | 4.63 | 4.15
Frost (ecp) | 0.759 | 0.924 | 4.67 | 4.15
Frost (ecpp) | 0.732 | 0.903 | 4.61 | 4.10
($d\rightarrow c_{\text{drop}};s$) | 0.798 | 0.972 | 4.71 | 4.18
Table 11: Faithfulness assessment using automatic and human evaluations, and
overall quality assessment by humans for models presented in Table 10 for the
CNN/DailyMail dataset (Test set with Non-Extractive Entity Chains Only).

##  Appendix F CNN/DailyMail: Controlling Hallucinations Using Entity Chains

Here, we present results from controlling hallucinations on CNN/DailyMail
summaries (i) by training models on the dataset with extractive entity chains
only and (ii) by modifying the predicted entity chains to keep only the
supported entities.

Table 10 and 11 present results from these models on the filtered test set
with extractive chains only, i.e., 7.8k examples, and on the test set with
non-extractive chains only, i.e., 3.7k examples, for CNN/DailyMail.

Table 12 presents results from these models on the full test sets, i.e., 11.3k
examples for XSum and 11.5k examples for CNN/DailyMail.

Model | XSum | CNN/DailyMail
---|---|---
Summary | Entity Chain | Summary | Entity Chain
R1/R2/RL | R1/R2/RL | EntF1 | R1/R2/RL | R1/R2/RL | EntF1
| Trained on Full dataset (204k/11.3k/11.3k) | Trained on Full dataset (287k/13.4k/11.5k)
Pegasus (ours) | 47.56/24.87/39.40 | 62.15/39.76/56.36 | 53.48 | 44.05/21.69/40.98 | 61.12/46.46/52.40 | 49.93
Frost (ecp) | 47.44/24.54/39.24 | 63.57/40.45/57.65 | 54.62 | 44.85/21.83/41.80 | 64.34/49.85/54.98 | 53.17
Frost (ecpp) | 47.80/25.06/39.76 | 64.09/41.07/58.18 | 55.49 | 45.11/22.11/42.01 | 64.28/49.86/54.96 | 53.22
($d\rightarrow c_{\text{drop}};s$) | 44.94/21.58/37.20 | 54.51/30.46/49.88 | 43.17 | 45.08/22.14/41.99 | 64.17/49.71/54.88 | 53.14
($d\rightarrow c_{\text{oracle}};s$)∗ | 58.24/36.47/52.16 | 98.93/80.32/98.91 | 98.59 | 61.85/38.95/59.00 | 98.02/96.19/97.96 | 97.31
| Trained on the filtered set (62.7k/3.5k/3.5k) | Trained on the filtered set (203.5k/9k/7.8k)
Pegasus (ours) | 44.53/21.63/36.61 | 56.82/33.32/51.58 | 47.78 | 44.15/21.59/41.05 | 61.49/46.62/52.61 | 50.16
Frost (ecp) | 44.10/21.02/36.16 | 54.98/30.53/49.56 | 46.91 | 44.75/21.73/41.65 | 64.13/49.53/54.89 | 52.85
Frost (ecpp) | 44.46/21.43/36.65 | 56.57/33.04/51.26 | 47.40 | 44.95/22.02/41.84 | 64.31/49.79/55.05 | 53.12
($d\rightarrow c_{\text{drop}};s$) | 43.43/20.15/35.75 | 53.05/28.65/48.15 | 41.64 | 44.91/22.03/41.82 | 64.26/49.73/55.02 | 53.06
($d\rightarrow c_{\text{oracle}};s$)∗ | 56.65/34.49/50.49 | 97.82/79.08/97.74 | 97.20 | 61.44/38.46/58.55 | 97.81/95.85/97.70 | 96.88
Table 12: Results on the full test sets. Controlling hallucinations by
training models on the dataset with extractive entity chains only (the bottom
block) and by modifying the predicted entity chains to keep only the supported
entities ($c_{\text{drop}}$). Results with ∗ are with oracle entity chain
prompts. All the results are reported on the full test sets.