# An analysis of document graph construction methods for AMR summarization

Fei-Tzin Lee, Chris Kedzie, Nakul Verma, Kathleen McKeown
Computer Science Department
Columbia University
<EMAIL_ADDRESS>
###### Abstract

Abstract Meaning Representation (AMR) is a graph-based semantic representation
for sentences, composed of collections of concepts linked by semantic
relations. AMR-based approaches have found success in a variety of
applications, but a challenge to using it in tasks that require document-level
context is that it only represents individual sentences. Prior work in AMR-
based summarization has automatically merged the individual sentence graphs
into a document graph, but the method of merging and its effects on summary
content selection have not been independently evaluated. In this paper, we
present a novel dataset consisting of human-annotated alignments between the
nodes of paired documents and summaries which may be used to evaluate (1)
merge strategies; and (2) the performance of content selection methods over
nodes of a merged or unmerged AMR graph. We apply these two forms of
evaluation to prior work as well as a new method for node merging and show
that our new method has significantly better performance than prior work.

##  1 Introduction

Abstract Meaning Representation (AMR) is a graph-based semantic representation
that aims to capture semantics at a higher level of abstraction than plain
text (Banarescu et al., 2013). It has been used in a broad variety of
applications ranging from information extraction to natural language
generation. As originally designed, AMR is intended to represent the meaning
of a single sentence, and in tasks such as entailment or paraphrase detection,
the context of a single sentence suffices to perform the task. In tasks
requiring longer contexts, however, the sentence-level limitation forces any
application of AMR either to consider sentences independently or to link them
together in some way. We specifically focus on the use of document-level AMR
as an intermediate representation for automatic summarization. In contrast to
end-to-end neural approaches to abstractive summarization (e.g., Nallapati et
al. (2016); See et al. (2017); Chen and Bansal (2018); Lewis et al. (2020)),
the use of an intermediate semantic representation holds promise for more
precisely controlling the selected summary content.

SentenceAMR GraphsDocumentAMR GraphSummary AMR GraphSummary Text Figure 1: The
single document AMR summarization pipeline proposed by Liu et al. (2015). The
colored nodes indicate concepts that are coreferent across sentences and
merged into a single node in the document graph.

Previous work on summarization with AMR has used different merging strategies
to link nodes across sentence graphs in order to form a coherent document
graph (Liu et al., 2015; Hardy and Vlachos, 2018; Dohare et al., 2018).
However, the effects of the specific choices made about how to merge nodes
have not been substantially explored. Thus, in this paper, we collect a
dataset of annotations consisting of co-reference alignments between document
and summary nodes which can be used to evaluate the specific performance of a
merge strategy at each step of the content selection process. We also present
a novel, person-focused merge strategy through the use of co-reference across
sub-graphs, and demonstrate improvement upon the seminal approach to content
selection using AMR introduced by Liu et al. (2015).

In summary, our contributions are as follows111We provide our annotation
guidelines and a sample of our dataset in the supplementary data. We will
publicly release the full version of the data and code for the merge
strategies and evaluations upon publication.:

  * •

Adjudicated annotations for node alignment between documents and summaries
that can be used to evaluate the efficacy of document sentence merging and
node-level content selection (§ 3.2);

  * •

A merge strategy combining aspects of Liu et al. (2015)’s work with person-
focused co-reference across sub-graphs (§ 4.1);

  * •

An evaluation of our proposed merge strategy that demonstrates significantly
improved performance on both node merging and content selection (§ 5).

##  2 Related Work

To the best of our knowledge, O’Gorman et al. (2018) is the only prior work
providing a dataset for cross-sentence AMR node co-reference. The primary
distinction between their annotations and ours is that while they annotate co-
reference chains across sentences in individual documents, we focus
specifically on AMR as applied to summarization, and explicitly annotate
alignments between documents and summaries rather than within documents. This
allows us to evaluate not only performance on within-document merging but on
the additional step of node-level content selection as well. We apply these
annotations to evaluate merging and content selection in existing methods of
single-document AMR summarization.

Seminal work on summarization using AMR Liu et al. (2015) first merges the AMR
annotations for all sentences in a particular document into a single graph by
identifying nodes with identical text labels, then treats summarization of the
document as a subgraph selection task and solves it using integer linear
programming. They evaluate their selected subgraph by measuring overlap of
nodes selected from the merged graph with labels inferred from alignments to
the summary nodes. Subsequent research in this space builds on or compares to
this approach. For example, Dohare et al. (2018) suggest an alternative way of
merging the document AMR into a single graph using text coreference, but their
method merges nodes associated with the key word in co-referent noun phrases,
while we merge nodes by focusing on components of the graph. Additional work
on AMR summarization follows that of Liu et al. (2015) for merging but instead
focuses on multi-document summarization Liao et al. (2018) or methods for
natural language realization of the summary AMR graph Hardy and Vlachos
(2018). In contrast, our focus is on content selection for single-document
summarization.

##  3 Data

###  3.1 AMR Corpus

We use release 3.0 of the AMR annotation corpus Knight et al. (2020), which
consists of sections from a variety of domains including blogs, Aesop’s
Fables, Wikipedia, and newswire text. We specifically focus on the proxy
report section, which is the only section in the corpus with AMR for document-
summary pairs rather than standalone documents. Each document in this section
is a news article paired with a human-written abstractive summary, or “proxy
report". As in the other sections of the corpus, both documents and summaries
are annotated with AMR graphs for each sentence.

The proxy section contains 298 document-summary pairs in the training set, one
of which we discard because the associated summary is empty; 35 in the
development set; and 33 in the test set. The average document length is 16.9
sentences, with a standard deviation of 9.1; the average summary length is 1.6
sentences.

The AMR graph for each sentence is a directed graph with labels on nodes and
edges. Every node in the graph is associated with a concept that may be either
a noun or a verb sense, and is assigned a text label representing that
concept. Edges are directed according to the relation they represent, and are
labeled with one of a fixed set of relation types between nodes.

###  3.2 Annotations

Our annotations are pairings between document and summary nodes labeling co-
reference between those nodes.

We annotated 50 document-summary pairs in total, including the full 33
documents from the test set as well as an additional 17 from the development
set. We recruited ten students in total as annotators, all undergraduate or
graduate students with a background in NLP, nine of whom are native English
speakers. Each document was assigned to two annotators who first performed the
annotation task independently, then worked together to adjudicate
disagreements. A sample of the final adjudicated labels and the IDs of the
associated documents in the original corpus are provided as supplementary
data. We include further annotation details in Appendix A, and full guidelines
for the annotation task in the supplementary material.

#### Agreement and adjudication

As traditional agreement metrics such as Cohen’s $\kappa$ are not applicable
to our setting, which may be viewed as a binary multi-label classification
problem, we instead compute two set-based metrics for annotator agreement over
the set of nodes which received at least one alignment from either annotator.
The first is based on exact match: the label sets given by two annotators are
considered a match if they are precisely identical. The second is the softer
notion of Jaccard similarity between the two label sets.

The preliminary round of annotations achieved .487 exact match agreement and
.511 average Jaccard similarity, both of which are far higher than the
expected values from random annotation, given that each document node may be
aligned to on the order of 20-30 summary entities. However, as our annotations
are designed to be a relatively small but high-quality evaluation set, we
performed an adjudication round in which the pair of annotators assigned to
each document worked together to resolve any disagreements, yielding a
finalized set of adjudicated gold labels for each node.

##  4 Methods

Following Liu et al. (2015), we model AMR summarization as a three stage
pipeline (depicted in Figure 1) of (i) sentence graph combination, (ii)
content selection, and (iii) summary text generation.

#### Sentence Graph Combination

The input to the pipeline is an ordered sequence of $n$ disjoint AMR graphs,
$\left(V_{1},E_{1}\right),\ldots,\left(V_{n},E_{n}\right)$, where $V_{i}$ and
$E_{i}$ are the set of nodes and edges respectively of the AMR graph
representation of sentence $i$.222Technically, the $V_{i}$ are multi-sets
since they can contain two or more distinct instances of the same concept
(this happens in Figure 2, where there are multiple instances of person,
country, and other concepts). A node $v\in V_{i}$ corresponds to an instance
of an AMR concept and an edge $(v,v^{\prime})$ exists in $E_{i}$ if
$v^{\prime}$ occupies a role in the sentence with respect to $v$ (e.g., if
$v^{\prime}$ is the ARG0 of $v$). In this stage of the pipeline, the disjoint
sentence graphs are connected into a connected graph
$\left(\Lambda,\xi\right)$ representing the semantics of the entire document.
We describe several ways of automatically performing sentence graph merging in
§ 4.1.

#### Node Selection

We develop a model to identify a summary subgraph
$\left(S,R\right)\subseteq\left(\Lambda,\xi\right)$ corresponding to the AMR
graph of the summary text. We treat this as a node-level binary classification
task, predicting for each node $v\in\Lambda$ whether to include it in $S$ or
not. Noisy training labels for this task are inferred from the summary graphs
by assigning a label of 1 to a document node if there is at least one summary
node with the same concept label, and 0 otherwise. We train a graph attention
network (GAT) Veličković et al. (2018) to perform the classification. The
edges of the summary subgraph are determined implicitly by the summary nodes
(i.e., $R=\left\\{(v,v^{\prime})\in\xi|v,v^{\prime}\in S\right\\}).$
Additional details about the GAT model can be found in Appendix C.

#### Summary Text Generation

Finally, a natural language generation (NLG) model is tasked with mapping
$\left(S,R\right)$ to a natural language summary. We fine-tune BART (Lewis et
al., 2020) on the training split of the proxy corpus to generate summary text
from a linearization of the selected nodes.

###  4.1 Sentence Graph Combination

meet-01personhave-org-role-91ministerforeigncountryname"India"personhave-org-
role-91ministerforeigncountryname"China"say-01personhave-org-
role-91ministerforeigncountryname"India"say-01meet-01personhave-org-
role-91ministerforeigncountryname"India"countryname"China"say-01meet-01personhave-
org-role-91ministerforeigncountryname"India"personhave-org-
role-91ministerforeigncountryname"China"Concept MergingPerson Merging Fragment
A
…the Indian foreign minister met with the Chinese foreign minister… Fragment B
…according to the Indian foreign minister… Figure 2: Example of Concept
Merging versus Person Merging. Note that in this instance, Concept Merging
collapses two distinct individuals into a single person node while Person
Merging is able to preserve this information.

We assume each sentence of each input document is annotated with its own AMR
graph.333In this work we use the gold AMRs graphs provided by the AMR corpus
but they could in principal be provided by an AMR parser. This means that
while multiple references to the same instance of a concept are annotated as
such within sentences (e.g., if the same person is mentioned multiple times in
a sentence, they will only have a single person node in the corresponding AMR
graph), they are not annotated for co-reference across sentences.

Performing content selection directly on this disjoint collection of AMR
graphs is not ideal. First, under a strict interpretation of AMR, distinct
nodes correspond to distinct instances, which forces the content selection
model to consider each mention of a concept in isolation, making it difficult
to capture how the same instance of a concept might participate in many other
events in a document.

We hypothesize that an ideal document AMR graph would contain a single node
for each distinct instance of a concept mentioned in the sentence AMR graphs.
In other words, nodes that were co-referent across sentence AMRs would be
“merged” into a single node in the document AMR. This merged node would
inherit the incoming and outgoing edges to other concepts in the original
AMRs, and thus frequently occuring instances in the document would take on
greater graph centrality in the document AMR. Centrality is a useful feature
for summarization using lexically constructed graphs Erkan and Radev (2004),
and we suspect this to hold for semantically constructed graphs as well.

We explore three methods of merging nodes to produce a document AMR graph,
which we refer to as Concept Merging, Person Merging, and Combined
Merging.444We also considered evaluating the merge strategy outlined in Dohare
et al. (2018), but were unable to retrieve the node merges from their
published code, and so we do not compare against their method in this paper.

#### Concept Merging

Liu et al. (2015) merge any sentence-level nodes into a single node in the
document graph if they share the same AMR concept regardless of whether they
are different instances. Formally, Liu et al. partition the sentence nodes
into disjoint sets $C_{j}$ such that
$\bigcup_{i=1}^{n}V_{i}=\bigcup_{j=1}^{m}C_{j}$ and
$\operatorname{concept}(c)=\operatorname{concept}(c^{\prime})$ for any
$c,c^{\prime}\in C_{j}$. For each $C_{j}$, a single node $\nu$ of the same
concept is created in $\Lambda$. For any $\nu_{i},\nu_{j}\in\Lambda$ with
corresponding concept clusters $C_{i}$ and $C_{j}$ respectively, there exists
an edge $(\nu_{i},\nu_{j})\in\xi$ in the document graph if there exists a pair
of sentence nodes $c_{i}\in C_{i}$ and $c_{j}\in C_{j}$ with an edge
$(c_{i},c_{j})\in\bigcup_{k=1}^{n}E_{k}$.

The primary flaw of this strategy is the fact that, with some exceptions (due
to an initial step collapsing name and date nodes, which we elaborate upon in
Appendix B), the resulting document AMR has no way of distinguishing between
multiple instances of the same type of concept. For example, in many cases,
nodes representing different instances of the person concept will be merged
across sentences without regard to whether those nodes actually refer to the
same person or not. As an example, see the Concept Merging example in Figure 2
where two different people (the Indian and Chinese foreign ministers) are
collapsed to a single person node.

#### Person Merging

Our first strategy for merging is inspired by the observation that many of the
merge errors in concept merging involve incorrect merging of nodes that occur
in the descendant subgraph of a specific instance of some entity – for
example, identifying nodes that are attached to a particular person. To
address this issue, we take a person-focused approach to merging as follows:

  1. 1.

Identify the subgraph consisting of all descendant nodes of every node
representing the person concept.

  2. 2.

Identify the text spans that correspond to each of these nodes using automatic
AMR-to-text alignment.

  3. 3.

Use a textual co-reference system to determine whether any of the text spans
corresponding to each of any two sub-graphs are co-referent; and if so,
determine those two sub-graphs to be co-referent.

  4. 4.

For any two co-referent person sub-graphs, merge any nodes across the two sub-
graphs with the same node label.

We will refer to these graphs as “person-merged.” An example of person merging
can be found in Figure 2, where both distinct instances of the person concept
along with their sub-trees are preserved.

#### Combined Merging

Our final merge strategy combines both Concept and Person merging. In this
joint strategy, we first perform Person Merging. We then perform concept-
merging on all remaining nodes of the sentence graphs (but do not perform an
initial name or date collapse), excluding any person nodes or their subtrees
that have already been merged in the Person Merging phase. Detailed pseudocode
for the combined merging method can be found in Appendix D.

##  5 Experiments

In our experiments we evaluate the merge methods laid out in the previous
section. We additionally consider an unmerged baseline where sentences are
simply joined at the root by an artificial root node.

We evaluate these merge styles in two ways: we compare the proposed merge
clusters directly against those induced by our human annotations using
coreference metrics (node merging); and we evaluate the summary nodes
predicted by a classifier trained on each style of merged graph against our
annotated gold alignments with precision, recall and F-measure over the
unmerged sentence nodes aligned to reference summary nodes (node selection).

###  5.1 Data handling and merging

We use the TAMR aligner Liu et al. (2018) to generate node-to-text alignments.
We use the Penman Goodman (2020) and NetworkX packages in Python to read and
store the resulting AMR graph structures, respectively. For text coreference
we use SpanBERT-large Joshi et al. (2020). Regardless of merge method, we also
join sentences at the root with an artificial root node when forming the
document graph.

###  5.2 Evaluating merge clusters

Each merge strategy produces a partition of the sentence-level AMR graph.
Given this partition, we may consider the associated co-reference cluster for
each node of the graph to be the set of other nodes which occur within the
same element of the partition, and can evaluate these clusters against those
we derive from the annotations.

Although our collected annotations are for document-summary alignment rather
than document merging, it is simple to infer merge clusters from these
alignments: every document node can be said to be co-referent with all other
document nodes that are aligned to one of the same summary nodes. In this
manner we can collect a set of gold merge clusters that we may evaluate our
automatic clusters against. We note that this approach does mean that any
nodes not present in the summary will be treated as unmerged, but for
summarization purposes we are only interested in summary-aligned nodes.

| $B^{3}$ | LEA
---|---|---
Method | Prec. | Recall | F1 | Prec. | Recall | F1
Person | 0.011 | 0.023 | 0.015 | 0.052 | 0.144 | 0.074
Concept*$\dagger$ | 0.469 | 0.514 | 0.490 | 0.088 | 0.191 | 0.115
Combined*$\dagger$ | 0.743 | 0.704 | 0.723 | 0.086 | 0.241 | 0.122
Table 1: Precision, recall, and F1 for the $B^{3}$ and LEA metrics for person
merging, concept merging (Liu et al., 2015), and our full approach (combined
merge). Methods are marked with (*) if the $B^{3}$ F-score is statistically
significant over that of the row directly above, and with ($\dagger$) if the
LEA F-score is significant over the row above.

Although the co-reference task we are interested in is over nodes rather than
text, it shares with text co-reference the objective of determining which
semantic objects appear in the same cluster. We thus evaluate these clusters
using two textual co-reference metrics, $B^{3}$ Bagga and Baldwin (1998) and
LEA Moosavi and Strube (2016), implemented over our AMR data structure. We do
not use importance weighting for the latter.

###  5.3 Node selection

In the node selection phase, the merged or unmerged document graph produced in
the previous step is given as input to a node classifier which learns to
predict a binary label for each node indicating whether it should be present
in the summary. Here we follow prior work by using label-based alignment
between documents and summaries to generate the training labels - that is, a
document node is assigned a “gold" label of 1 for training purposes if there
is at least one node in the summary with the same concept label, and 0
otherwise. This is an imperfect method for the same reasons that apply to the
merging stage; while these noisy labels may be sufficient for training, we
therefore need an external benchmark to objectively evaluate content selection
with respect to the true summary.

Thus, we use our annotations as gold standard here as well: we consider the
true label of a node to be 1 if it is aligned to at least one node in the
summary in our annotations, and 0 otherwise. To avoid introducing confounding
effects by counting merged nodes once in a merged graph but multiple times in
the unmerged graph, we propagate the labels predicted on each node in a merged
document graph back to the original sentence nodes that formed it, and compute
precision, recall and F-measure against the gold labels induced by the
alignments over the unmerged nodes.

We use the same GAT architecture and training procedure to perform node
classification on all merge strategies. The details of both are described in
Appendix C.

###  5.4 Generation

Finally, to generate summary text from the nodes selected with each merge
strategy, we linearize the extracted nodes of the graph into a sequential
format. This linearized AMR is then passed to a pretrained BART-large model
which has been finetuned to generate the summary text from such linearized
input. We include further details on the linearization and finetuning
procedure in Appendix E.

We compare the generated output from each merge strategy using three kinds of
automatic metrics (ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and
MoverScore (Zhao et al., 2019)), as well as with a human evaluation designed
to investigate the influence of our proposed merge strategies on person-
related output, ranking each set of summaries on three criteria (fluency,
salience and faithfulness with regards to information about humans). We
provide sample summary outputs in Appendix E, and report details of the human
evaluation setup in Appendix F.

##  6 Results

In this section we report results for node merging and node selection. For
node selection, we report the average of precision, recall and F-score over
five classifier training runs in order to control for random variation in the
model initialization. For statistical significance we use the Approximate
Randomization Test Riezler and Maxwell (2005).

###  6.1 Node merging

We report precision, recall and F-score for the $B^{3}$ and LEA metrics over
the node clusters induced by person merging, concept merging and combined
merging, as compared to the gold annotated clusters (see Table 1).

Interestingly, combined merging strongly outperforms both person and concept
merging on nearly all metrics. While it is in line with expectations that
cluster precision would be higher for our method than for concept merging, as
it is more selective about which nodes may be merged, it is surprising that
our method also yields substantially higher recall. We discuss this phenomenon
in § 7.2.

###  6.2 Node selection

Merge Strategy | Prec. | Recall | F1
---|---|---|---
Person | 0.348 | 0.244 | 0.280
Concept (Liu et al.)* | 0.363 | 0.277 | 0.293
Unmerged* | 0.327 | 0.359 | 0.332
Combined* | 0.412 | 0.400 | 0.398
Table 2: Precision, recall, and F1 of document node selection under different
merge strategies. A method is marked with (*) if the difference between it and
the method one row above is statistically significant.

We evaluate the unmerged and concept merging baselines, as well as both our
combined merging strategy and person merging alone. Results are presented in
Table 2. Combined merging’s performance in node clustering seems to
effectively carry over to node selection, where it continues to outperform
concept merging and person merging in both precision and recall.

###  6.3 Generation

Merge | ROUGE-1 | ROUGE-2 | ROUGE-L | METEOR | MoverScore
---|---|---|---|---|---
Unmerged | 0.339 | 0.098 | 0.230 | 0.250 | 0.240
Concept | 0.284 | 0.066 | 0.210 | 0.175 | 0.211
Combined | 0.305 | 0.071 | 0.210 | 0.180 | 0.224
Seq-to-seq BART (CNN) | 0.417 | 0.237 | 0.330 | 0.288 | 0.390
Table 3: Scores on automatic metrics for each merge strategy, and the pretrained sequence-to-sequence BART-large model finetuned on CNN/DailyMail. Reported ROUGE numbers are f-scores. Merge | Fluency | Salience | Faithfulness
---|---|---|---
Unmerged | 20 (8) | 21 (8) | 16 (6)
Concept | 21 (10) | 15 (6) | 15 (5)
Combined | 22 (8) | 18 (3) | 17 (7)
Table 4: Number of test set documents for which each merge strategy was ranked
first by at least one annotator (both annotators) on each criterion. The test
set contains 33 documents, but the numbers do not sum to 33 because ties are
permitted.

We report the performance of our basic finetuned BART models when given as
input the linearized selected AMR from the unmerged, concept, and combined
merge strategies on a range of automatic metrics in Table 3.555While we also
provide the scores for the pretrained sequence-to-sequence BART model
finetuned on CNN/DailyMail as a performance ceiling, we note that our AMR-to-
text BART models are not intended to provide summary output competitive with
end-to-end BART, but are rather a diagnostic tool to investigate the effects
that the different merge strategies have upon the final generated output. Our
AMR-to-text models were finetuned on an extremely small amount of data and
should not be considered fully pretrained for this task; a significant amount
of further work on tuning would need to be done in order to make a comparison
between AMR-to-text and sequence-to-sequence BART models. We report the
performance of our finetuned BART models with the same three merge strategies
under human evaluation for fluency, salience and faithfulness in Table 4.

While BART outputs using the unmerged strategy appear superior under all
automatic metrics, the human evaluation reveals a different array of
strengths, in which unmerged outputs are rated the best in terms of salience
the most often by far, but the combined merge strategy actually yields the
most faithful outputs in general.

##  7 Analysis

Error type |  Example sentence fragment pair that produces error
---|---
Concept conflation (surface) |  …required to inform any person whose assets are being frozen…
…an European Union decision to freeze the assets of the People’s Mujahadeen of
Iran.
Concept conflation (hidden) |  …the foreign minister of India…
…the foreign minister of China…
Table 5: Examples of concept conflation errors in concept merging.

###  7.1 Concept error analysis

We identify nodes that concept merging clustered incorrectly using our
annotated alignments and perform a manual inspection of concept merging errors
on the development set, identifying five types of common errors: stopword
conflation, missing synonyms, name skipping, conflation of surface concepts
and conflation of hidden concepts. We present examples of the latter two kinds
of error in Table 5, and examples of all types of error in Appendix B.

While the first two types of errors can be resolved with straightforward fixes
(e.g., holding out stopwords when merging), and the third and fourth can be
resolved using text-based coreference, the final and most complex type of
error involves incorrectly merging AMR nodes that do not explicitly appear in
the text, and thus cannot be resolved simply by performing coreference on the
text itself. This error often occurs when there are multiple entities in the
text of a concept type that induces a large implicit structure, such as the
foreign ministers in Figure 2. Hidden concept conflation requires linking text
coreference to the graph structure of the AMR, as the person-focused and
combined merge strategies are designed to do.

###  7.2 The role of coreference

Perhaps the most curious observation from these results is that while person
merging has worse performance in both the cluster evaluation and on node
selection than either unmerged or concept merged graphs - in fact, it has
strikingly poor scores under merge cluster metrics - using it together with
concept merging in the combined strategy yields by far the best results. This
suggests that these two methods have strengths that are complementary.

Method | Proportion of merged nodes
---|---
Unmerged | 0
Concept | 0.509
Person | 0.012
Combined | 0.572
Table 6: Average proportion of document nodes that are merged for each
strategy, over the development set.

When we examine the clusters generated by person merging, we find that much of
the poor performance can simply be attributed to the fact that it hardly
merges any nodes at all (see Table 6). This is unsurprising: there are
relatively few nodes that occur both under a person node and within a
coreference cluster, and even fewer that share a label with another person-
descendant node in the same coreference cluster.

In this case, the benefit of using person merging as a first step in
conjunction with concept merging is not so much that it itself brings higher-
quality merges, but rather that it takes a number of low-quality merges out of
consideration by removing them from the pool of as yet unmerged nodes.

We further note that since the decision to merge nodes can only be made
between nodes of the same label in the combined strategy, with additional
restrictions between nodes in person subtrees that do not co-refer, we would
expect its recall to be no higher than that of concept merging, which merges
nodes with the same label without co-reference-based restriction.

The fact that there is also a large improvement in recall indicates that
concept merging misses a sizable proportion of merges because of some
condition that it does not share with combined merging. The only such
difference is the initial name and date collapse step. This means that adding
an identifying name to a node’s label indeed has the adverse effect of
preventing it from being merged with other instances of the same entity that
were not similarly referred to by name in many cases, as described in Appendix
B.

Thus, combined merging also averts this type of error by replacing the name
collapse step with co-reference-based subtree matching.

###  7.3 Generated summaries

The automatic metric scores indicate that while the combined merge strategy is
slightly better than concept merging, the unmerged strategy outperforms both
of the others. This is fairly surprising, as there does not seem to be any
intuitive reason that leaving the AMR unmerged should be better for generation
than merging it correctly. However, a manual inspection of the validation set
reveals that the unmerged strategy generally yields much longer linearized
inputs to the generator, and in fact on the validation set simply passing in
the linearized AMR for the entire document graph yields better scores on
automatic metrics than using the selected nodes from any merge strategy; we
hypothesize that BART is powerful enough to identify redundant information
even in AMR, which seemingly obviates the merging step.

However, the human evaluation tells a more nuanced story. While unmerged
inputs yield the most salient outputs - supporting the hypothesis that BART
performs its own internal pruning of redundant information - using the
combined merge strategy improves faithfulness on person-related information,
which suggests that narrowing down the field of consideration in the input
helps it focus better on the actual information it is supposed to summarize
rather than adding in details to fill in the gaps. The merge strategies are
roughly evenly matched on fluency, but combined merge also has a slight edge
here.

##  8 Conclusion

In this paper, we have presented a new annotated dataset of node alignments
between document-summary AMR pairs that can be used to evaluate two
fundamental components of the AMR-based summarization pipeline: document graph
construction via node merging, and node-level content selection over the
constructed document graph. Drawing on insights from error analysis of prior
work when evaluated against our annotations, we developed a new method for
node merging that combines co-reference with concept merging. We show that our
novel method significantly outperforms prior work using concept merging alone
when evaluated against our gold labels in both the node merging and node
selection steps, and analyze the causes of error that our method addresses.

The most obvious direction for future work would be to focus on the generation
component. Another extension would be to use the interpretability our dataset
affords to investigate the use of intermediate AMR representations for
controllability in neural summarization. All in all, our work lays the
foundation for a fine-grained understanding of content selection dynamics in
AMR summarization.

## References

  * Bagga and Baldwin (1998) Amit Bagga and Breck Baldwin. 1998.  Algorithms for Scoring Coreference Chains.  In _The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference_ , pages 563–566.
  * Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013\.  Abstract Meaning Representation for sembanking.  In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_ , pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–686, Melbourne, Australia. Association for Computational Linguistics.
  * Dohare et al. (2018) Shibhansh Dohare, Vivek Gupta, and Harish Karnick. 2018.  Unsupervised semantic abstractive summarization.  In _Proceedings of ACL 2018, Student Research Workshop_ , pages 74–83, Melbourne, Australia. Association for Computational Linguistics.
  * Erkan and Radev (2004) Günes Erkan and Dragomir R. Radev. 2004.  LexRank: Graph-based Lexical Centrality as Salience in Text Summarization.  _J. Artif. Intell. Res._ , 22:457–479.
  * Goodman (2020) Michael Wayne Goodman. 2020.  Penman: An open-source library and tool for AMR graphs.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 312–319, Online. Association for Computational Linguistics.
  * Hardy and Vlachos (2018) Hardy and Andreas Vlachos. 2018.  Guided neural language generation for abstractive summarization using Abstract Meaning Representation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 768–773, Brussels, Belgium. Association for Computational Linguistics.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Knight et al. (2020) Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, and Nathan Schneider. 2020.  Abstract meaning representation (AMR) annotation release 3.0.  Web Download.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Liao et al. (2018) Kexin Liao, Logan Lebanoff, and Fei Liu. 2018.  Abstract Meaning Representation for multi-document summarization.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1178–1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2015) Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A. Smith. 2015.  Toward abstractive summarization using semantic representations.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1077–1086, Denver, Colorado. Association for Computational Linguistics.
  * Liu et al. (2018) Yijia Liu, Wanxiang Che, Bo Zheng, Bing Qin, and Ting Liu. 2018.  An AMR aligner tuned by transition-based parser.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2422–2430, Brussels, Belgium. Association for Computational Linguistics.
  * Moosavi and Strube (2016) Nafise Sadat Moosavi and Michael Strube. 2016.  Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 632–642, Berlin, Germany. Association for Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar GuÌ‡lçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * O’Gorman et al. (2018) Tim O’Gorman, Michael Regan, Kira Griffitt, Ulf Hermjakob, Kevin Knight, and Martha Palmer. 2018.  AMR beyond the sentence: the multi-sentence AMR corpus.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3693–3702, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Riezler and Maxwell (2005) Stefan Riezler and John T. Maxwell. 2005.  On some pitfalls in automatic evaluation and significance testing for MT.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 57–64, Ann Arbor, Michigan. Association for Computational Linguistics.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph attention networks.  In _International Conference on Learning Representations_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Annotation interface

Annotators are given the full list of summary sentences, followed by up to 10
document sentences for each document-summary pair.666Since this is single-
document news summarization, the information in the summary typically occurs
very early in the document. For each sentence, they are presented with the
sentence text first, followed by the list of entities that occur in that
sentence, each represented by a handle and a label. The handle for each entity
in a document sentence consists of the index of the node within the document
(i.e., the handle of the third node of a sentence is ‘3’ for that sentence).
The handle for each entity in a summary sentence consists of the index of the
sentence within the summary, and a unique identifier within the sentence (for
example, the handle of the fourth node in the second summary sentence would be
‘2d’). The label provided for the entity is its concept label.

The main annotation interface is a spreadsheet containing the minimal
information for summary and document sentences. An example is provided in
Figure 3.

Figure 3: An example of the annotation interface for document-summary
alignment.

For many entities, the sentence text and concept label alone is sufficient to
determine the role the node plays in the sentence; however, in case further
disambiguation is necessary, we also provide both the AMR-to-text alignments
in an HTML file for each document as well as a labeled visualization of each
sentence graph.

Supplemental visualizations of the AMR graph structure and of the node-to-text
alignments for each sentence are provided as separate files. Examples are
provided in Figure 4 and Figure 5.

Figure 4: An example of the graph visualization. Figure 5: An example of the
HTML node-to-text alignment visualization.

For each document node in the main spreadsheet, the annotators are asked to
provide the following: (1) the space-separated list of handles for the set of
summary nodes that refer to the same specific entity as that document node, if
any (for example, “1b 2f"), and (2) an additional tag marking the alignment
list as “abstractive" if the document and summary nodes refer to the same
instance of an idea but are different concepts - for example, if a university
“organizes" a workshop in the document, but “funds" it in the summary, those
two verbs would be an abstractive match - or if one is an abstraction of the
other in another way, e.g. one is an aggregate that includes the other. We do
not consider the abstraction tags in this paper, but they would lend
themselves to future work on more operations such as aggregation and sentence
fusion.

##  Appendix B Types of errors in concept merging

Error type |  Example sentence fragment pair that produces error
---|---
Stopword conflation |  …a failure for Swedish law and order…
Both Norway and Sweden…
Missing synonyms |  …nuclear bombs…
…nuclear arms…
Name skipping |  Estonian informatics center spokeswoman Katrin pargmae stated…
Katrin pargmae also stated…
Concept conflation (surface) |  …required to inform any person whose assets are being frozen…
…an European Union decision to freeze the assets of the People’s Mujahadeen of
Iran.
Concept conflation (hidden) |  …the foreign minister of India…
…the foreign minister of China…
Table 7: Examples of common error types in concept merging.

The first two types of errors are relatively easy to fix. Stopword conflation
occurs whenever there are multiple instances of a stopword present in the
document, and can be easily avoided by simply holding out stopword nodes from
the concept merge process. Nodes that have synonymous but not identical labels
will not be merged even when they refer to the same entity; this can be
approximately solved by using a high word embedding similarity threshold
rather than exact concept matching.

The third error, name skipping, is a product of concept merging’s initial name
and date collapse step, in which name and date nodes are merged with their
parents if they are only children, renaming the parent nodes with the combined
concept. Name skipping occurs when a name node is merged into its parent in
some instances but not in others, which splits the resulting merge clusters
into those that were identified by name and those that were not. In the second
example in Table 7, the first instance of “Katrin pargmae" will not be merged
with its name node as it has additional children (the organization role of
spokeswoman), but the second instance will. This is an inherent issue of the
name collapse step, and can be addressed by replacing it with a more robust
method of detecting co-reference.

The final two types of errors can be viewed as special cases of the same
error, namely, merging entities that are different instances of the same
entity type, but the method of addressing the error differs between cases. In
the first case, a node representing an entity that is directly mentioned in
the text is erroneously merged with another of the same type. To address this,
we would need to perform text co-reference to determine whether nodes of the
same type directly co-refer.

In the second case, nodes that are not explicitly mentioned in the text, but
instead appear in the AMR as attachments to a node that does explicitly
appear, are erroneously merged with other implicit nodes of the same concept
type. Concept merging cannot distinguish between the hidden person nodes in
this case, even though they represent entities that do not co-refer.

It is this final type of error that the person-focused and combined merge
strategies we present are designed to directly address: the descendant subtree
of any person-type entity mentioned in the text must be merged only with the
subtrees of other person nodes coreferent with that entity. We focus on person
concepts specifically as this is a very common case that often induces a large
subtree.

##  Appendix C Content Selection Model

As our classification model we use a feedforward layer attached to a Graph
Attention Network, or GAT Veličković et al. (2018), that uses a modified the
scaled dot-product attention function of the Transformer architecture Vaswani
et al. (2017) that is masked according to the undirected AMR graph structure.
We use two layers, a single attention head, 256-dimensional intermediate
representations, and the ReLU activation function. We implement our model in
PyTorch. Node parameters in the GAT layers are initialized from a unit normal
distribution; the classification layer is initialized with the PyTorch
default.

The input representations for each node consist of the summed word embeddings
for all words in its concept label, as well as four discrete features: in-
degree, out-degree, sentence index and number of occurrences in the text. The
representation of every node at each graph layer is recalculated as an
attention-weighted sum over the representations of itself and its neighbors in
the AMR graph. The final node representation is passed to the single-layer
feedforward classification component.

We train our classifier with the Adam optimizer with a learning rate of .001
and batch size of 1 for a maximum of 128 epochs, using an early stopping
routine that halts training after training loss decreases by any amount
between epochs or development loss has not increased within the last three
epochs.

Using Transformer-style self-attention (masked to respect the AMR graph), the
learnable parameters for the GAT are a projection matrix for keys, queries and
values for each layer. As we use two layers, an intermediate representation
size of 256, and 304-dimensional inputs, the total number of parameters for
the GAT is therefore $2\times(256\times 304+304\times 256+304\times 304)$. The
binary classification layer has $304\times 2$ parameters.

A single training run of this model over the full proxy report dataset
typically runs for around 30 epochs (with early stopping) and takes
approximately three minutes on a single Tesla V100 GPU.

##  Appendix D Combined Merge Pseudo-code

We present pseudocode for the full combined merging strategy in Algorithm 1.

Input: A collection of sentence graphs $D=\\{S_{1},S_{2},...,S_{n}\\}$, where
$S_{i}=(V_{i},E_{i})$.

// Initialize document graph with artificial root

$V\leftarrow\bigcup_{i=1}^{n}V_{i}\cup\\{-1\\}$

$E\leftarrow\bigcup_{i=1}^{n}E_{i}\cup\\{(-1,root(S_{i}))\mid i\in[n]\\}$

// Collect subgraphs and aligned text spans

p $\leftarrow\\{n\in V_{i}\mid label(n)=person\\}$

c $\leftarrow\\{descendants(n)\mid n\in$ p$\\}$

// Assign cluster ids to subgraphs via spans

spans $\leftarrow\\{\\{\bigcup span(n)\mid n\in C_{i}\\}\mid C_{i}\in$ c$\\}$

ids $\leftarrow\\{cluster\\_id(span(p_{i}))$ if $p_{i}$ is clustered else
$cluster\\_id(C_{i})\mid C_{i}\in$ c$\\}$

// Partition coreferent type matches

$P\leftarrow\\{\\}$

for  _$k\in cluster\\_ids$_ do

cluster $\leftarrow\\{C_{i}\in$ c $\mid ids[i]=k\\}$

$C\leftarrow\\{n\in C_{i}\mid C_{i}\in$ cluster$\\}$

for  _unique $l\in\\{label(n)\mid n\in C\\}$_ do

$P\leftarrow P\cup\\{\\{n\in C\mid label(n)=l\\}\\}$

end for

end for

// Concept label partition on unmerged nodes

$U\leftarrow\\{n\in V\mid n\notin\bigcup_{i}\\{P_{i}\in P\\}\\}$

for  _unique $l\in\\{label(n)\mid n\in U\\}$_ do

$P\leftarrow P\cup\\{\\{n\in U\mid label(n)=l\\}\\}$

end for

// Merge from partition

$V\leftarrow\\{i|P_{i}\in P\\}$

$E\leftarrow\\{(i,j)|P_{i},P_{j}\in P,\exists k\in P_{i},l\in P_{j}\ni(k,l)\in
E\\}$

return _$(V,E)$_

Algorithm 1 Combined merging

##  Appendix E Finetuning setup for BART generator

For each merge strategy, we start with a vanilla pretrained BART-large model
and finetune it for four epochs with a learning rate of .00003 on the
generation task where the input is linearized selected AMR and the output is
the summary text. (We originally considered a range of up to 64 epochs, but
found that validation performance according to automatic metrics peaked before
epoch 10 in all cases, usually between epochs 3-5.)

The linearized AMR input is generated as follows: given a merged document
graph and the output of a node selection model on that graph, i.e., a binary
label for each node of the graph indicating whether that node is to be
included in the summary or not, we linearize each sentence in sequence,
keeping track of all nodes that have been seen before to avoid repetition. To
linearize each sentence, we start at the original root node of that sentence
(i.e., one of the children of the artificial root node of the document graph)
and perform a depth-first traversal from that node. Nodes that we touch that
are labeled 1 are added to the sequence; nodes that we touch that are labeled
0 are not added to the sequence unless they have a descendant labeled 1. Nodes
that we have already touched are added to the sequence again, but their
subtrees are not traversed again. The final sequence is converted to a string
in PENMAN format.

We present the pseudocode for the linearization procedure in Algorithm 2.

Input: A merged document graph $G=(V,E)$; labels $L$ for every node $n\in V$,
with $L(n)\in\\{0,1\\}$; original sentence roots $r_{1},r_{2},...,r_{s}\in V$.

// Initialize sequence

seq $\leftarrow[]$

touched $\leftarrow[]$

for  _$i\in[s]$_ do

// Traverse subtree of sentence root in depth-first order

for  _$n\in$ descendant subtree of $r_{i}$_ do

if  _n $\in$ touched_ then

append $n$ to seq

skip recursion and return to parent

else if  _L(n) = 1 or a descendant of n is labeled 1_ then

append $n$ to seq

continue recursion in depth-first order

else

skip recursion and return to parent

end for

end for

return _seq_

Algorithm 2 AMR linearization

For each merge strategy, we trained BART on linearized AMR for its selected
subgraphs and the associated text summaries, and used this model to generate a
text summary for each document. We provide examples of a few such summaries
using the combined merge strategy in Table 8.

Gold summary |  Generated summary output
---|---
Afghanistan’s counter-narcotic police have confiscated 1.2 metric tons (1.3 tons) of opium during a raid. In 2007 Afghanistan accounted for 93 percent of the world’s opium supply. |  Afghanistan’s Interior Ministry announced the confiscation of more than a ton of opium from a drug trafficker and the detention of a drug smuggler in the southernmost province of Helmand. The drug was smuggled into Afghanistan from Pakistan.
Russia proposes cooperation with India and China to increase security around Afghanistan to block drug supplies. Afghanistan is 1 of the world’s major opium producers and supplies Western markets through transit countries such as Russia. |  Russia’s Foreign Minister Lavrov stated that the Russian government will increase security at the borders with Afghanistan and Tajikistan in order to decrease the flow of illegal drugs from Afghanistan to the world market. The Russian government stated that Afghanistan is the world’s largest producer of opium and the largest supplier of heroin to Russia.
Russian Federation President Dmitry Medvedev promised on May 15, 2008 to provide funding for Russia’s nuclear missile program. |  Russian President Dmitry Medvedev made his first public appearance since being sworn in as the new President of the Russian Federation on November 9, 2008. The Russian President stated that the Russian military is ready to respond to any threat from the West.
Table 8: Sample generated summary output from BART given our selected content
as input.

##  Appendix F Human evaluation of generated summaries

We recruited five annotators with a background in NLP to perform an evaluation
of the summaries generated with the three different merge strategies. Of the
33 test set documents, each was assigned two annotators. For each document,
annotators were given the first ten sentences of the document text as well as
the three summaries produced on that text by the different merge strategies,
and asked to choose the best summary among those on each of three criteria:
fluency, salience and faithfulness, specifically with respect to people
mentioned in the document. The specific annotation instructions, reference
example, and task layout are provided in Figures 6, 7, and 8, respectively.

Figure 6: Instructions for human evaluation.

Figure 7: Reference example given for evaluation.

Figure 8: Document and summary text placeholders and ranking prompts for
evaluation.