# Automated Metrics for Medical Multi-Document Summarization Disagree with
Human Evaluations

Lucy Lu Wang1,2 Yulia Otmakhova3 Jay DeYoung4 Thinh Hung Truong3
Bailey E. Kuehl2 Erin Bransom2 Byron C. Wallace4
1University of Washington 2Allen Institute for AI 3University of Melbourne
4Northeastern University
<EMAIL_ADDRESS>{yotmakhova<EMAIL_ADDRESS>
{deyoung.j<EMAIL_ADDRESS>
###### Abstract

Evaluating multi-document summarization (MDS) quality is difficult. This is
especially true in the case of MDS for biomedical literature reviews, where
models must synthesize contradicting evidence reported across different
documents. Prior work has shown that rather than performing the task, models
may exploit shortcuts that are difficult to detect using standard $n$-gram
similarity metrics such as ROUGE. Better automated evaluation metrics are
needed, but few resources exist to assess metrics when they are proposed.
Therefore, we introduce a dataset of human-assessed summary quality facets and
pairwise preferences to encourage and support the development of better
automated evaluation methods for literature review MDS. We take advantage of
community submissions to the Multi-document Summarization for Literature
Review (MSLR) shared task to compile a diverse and representative sample of
generated summaries. We analyze how automated summarization evaluation metrics
correlate with lexical features of generated summaries, to other automated
metrics including several we propose in this work, and to aspects of human-
assessed summary quality. We find that not only do automated metrics fail to
capture aspects of quality as assessed by humans, in many cases the system
rankings produced by these metrics are anti-correlated with rankings according
to human annotators.111Dataset and analysis are available at
https://github.com/ allenai/mslr-annotated-dataset.

##  1 Introduction

Figure 1: Spearman correlations between rankings produced by human-assessed
quality facets (F1-F4), automated metrics (M1-M7), and combined pairwise
system rankings (PW-combined) on the Cochrane MSLR dataset. Rankings from
automated metrics are highly correlated as a group except for PIO-Overlap (A).
PIO-Overlap rankings are strongly correlated with rankings from human-assessed
facets, especially PIO agreement (B). Metrics most strongly associated with
PW-Combined rankings are Delta-EI and PIO-Overlap (C). Rankings from commonly
reported automated metrics like ROUGE and BERTScore are not correlated or
_anti_ -correlated with human-assessed system rankings (D).

Multi-document summarization (MDS) requires models to summarize key points
across a set of related documents. Variants of this task have drawn
significant attention in recent years, with the introduction of datasets in
domains like newswire Fabbri et al. (2019), Wikipedia Gholipour Ghalandari et
al. (2020), science Lu et al. (2020), medical literature reviews DeYoung et
al. (2021); Wallace et al. (2020), and law Shen et al. (2022); and substantial
methodological work to design model architectures tailored to this task Xiao
et al. (2022); Pasunuru et al. (2021); Liu and Lapata (2019).

In this work, we focus on MDS for literature reviews (MSLR), a challenging
variant of the task in which one attempts to synthesize all evidence on a
given topic. When manually performed, such reviews usually take teams of
experts many months to complete. Good review summaries aggregate the results
of different studies into a coherent passage, while the evidence presented in
the input studies will often be in conflict Wallace et al. (2020); DeYoung et
al. (2021); Wadden et al. (2022), complicating the synthesis task.222Indeed,
reviews conducted by different teams may themselves conflict Ioannidis (2016),
reflecting the inherent difficulty of the task; however this may owe to
differing methods of selecting input studies, a complication we ignore here,
though which has been explored in recent work Giorgi et al. (2022a).

Evaluating conditional text generation models is notoriously difficult,
impeding progress in the field. Prior work on summarization evaluation has
proposed various lexical and modeling-based approaches to assess generation
quality, but these metrics predominately use correlation with human-assessed
quality facets over relatively small numbers of examples to demonstrate
utility Fabbri et al. (2021); Wang et al. (2020); Deutsch and Roth (2020);
Yuan et al. (2021). This limitation of current metric evaluation implies that
existing automated measures may not generalize well. Further, evaluation in
the multi-document setting adds additional complexity, e.g., prior work has
shown that MDS models may sometimes exploit shortcuts that do not reflect as
detectable changes in automated metrics Wolhandler et al. (2022); Giorgi et
al. (2022a).

To address these challenges, we collect human annotations to evaluate current
models and to support automated metrics development for the medical MDS task.
We construct a dataset of such evaluations using public submissions from the
2022 MSLR shared task on literature review
MDS.333https://github.com/allenai/mslr-shared-task Selecting top-performing
models, we label the summary quality of a sample of these models’ outputs on
the Cochrane subtask Wallace et al. (2020). As part of our analysis, we
compare system rankings produced by automated metrics and human evaluations.
Strikingly, our results highlight consistent and significant disagreements
between automated metrics and humans, motivating the need for better automated
evaluation metrics in this domain.

We contribute the following:

  * •

A dataset of summaries and quality annotations on participant submissions to
the MSLR shared task. We include human annotations for 6 models on 8
individual quality facets (§3.2) and pairwise preferences provided by five
raters (§3.3).

  * •

An analysis of lexical features among inputs, generated, and target summaries
(§4), showing a large amount of undesirable copying behavior.

  * •

An analysis of correlations between automated evaluation metrics and human-
assessed quality (§5), and the differences in system rankings produced by
automated metrics versus human evaluation (§6). We propose several novel
evaluation metrics based on desired features of MSLR summaries (§5). We find
that system rankings derived from commonly reported automated metrics are
_not_ correlated or even _anti_ -correlated with rankings produced by human
assessments of quality, though some of the metrics we propose demonstrate
promise in capturing certain quality facets.

##  2 Background

The MSLR shared task was introduced to bring attention to the challenging task
of MDS for literature reviews. The shared task comprised two subtasks, based
on the Cochrane Wallace et al. (2020) and MS^2 DeYoung et al. (2021) datasets.
The Cochrane dataset consists of 4.6K reviews from the Cochrane database of
systematic reviews. Inputs are abstracts of papers cited by the review and
target summaries are the _Authors’ Conclusions_ subsections of review
abstracts. The MS^2 dataset includes 20K reviews and is semi-automatically
constructed from biomedical literature reviews indexed by PubMed. We refer the
reader to the original publications for details concerning dataset
construction Wallace et al. (2020); DeYoung et al. (2021).

Shared task organizers provided training and validation splits for both
datasets, and solicited model submissions to two public leaderboards, where
models were evaluated on a hidden test split. Models were ranked on the
leaderboard using ROUGE (-1, -2, -L; Lin 2004), BERTScore Zhang et al.
(2020a), and Delta-EI DeYoung et al. (2021); Wallace et al. (2020), a metric
based on evidence inference Lehman et al. (2019) classifications.

##  3 Dataset

We construct our dataset from system submissions to the Cochrane subtask
leaderboard for the 2022 MSLR shared task (provided to us by task organizers).
We only sample from the Cochrane subtask due to the greater number and variety
of successful submissions. We include all summaries from the leaderboard,
though we only perform human evaluation on summaries generated by 6 models
(discussion in §3.1). We define and apply two human evaluation protocols to a
sample of summaries from these 6 systems. The first (§3.2) is a facet-based
evaluation derived from the analysis conducted in Otmakhova et al. (2022b) and
the second (§3.3) is a pairwise preference assessment.

###  3.1 MDS systems

We perform human evaluation on the outputs of 6 MDS systems. Five of these are
community submissions to the MSLR-Cochrane
leaderboard,444https://leaderboard.allenai.org/mslr-cochrane/ while a sixth is
a baseline system (BART-Cochrane) included for reference. These systems
represent different Transformer model architectures (BART, BART-large,
Longformer, BigBird), input selection strategies (Shinde et al., 2022), and
differential representation/attention on input tokens (Otmakhova et al.,
2022a; DeYoung et al., 2021). We exclude some systems from human evaluation
due to poor summary quality (disfluent) or being baselines. We briefly
describe our 6 systems below.

#### ITTC-1 / ITTC-2

Otmakhova et al. (2022a) fine-tuned PRIMERA Xiao et al. (2022) for the
Cochrane subtask and exploited the use of global attention to highlight
special entities and aggregate them across documents. We include two settings
from the leaderboard, one that adds global attention to special entity marker
tokens (ITTC-1) and one that adds global attention to entity spans (ITTC-2).

#### BART-large

Tangsali et al. (2022) fine-tuned BART-large (Lewis et al., 2020) for the
subtask.

#### SciSpace

Shinde et al. (2022) defined an _extract-then-summarize_ approach, combining
BERT-based extraction of salient sentences from input documents with a BigBird
PEGASUS-based summarization model Zaheer et al. (2020).

#### LED-base-16k

Giorgi et al. (2022b) fine-tuned Longformer Encoder-Decoder (Beltagy et al.,
2020) for the Cochrane subtask following a similar protocol described in Xiao
et al. (2022).

#### BART (baseline)

The baseline follows the protocol in DeYoung et al. (2021) to fine-tune BART
(Lewis et al., 2020) for the Cochrane subtask.

Model rankings originally reported on the MSLR-Cochrane leaderboard are
provided in Table 1.

System | ROUGE* | BERTS. |  $\Delta$EI | ClaimV. | NLI | STS | PIO-Over. | Flu. | PIO | Dir. | Str. | PW-Comb.
---|---|---|---|---|---|---|---|---|---|---|---|---
ITTC-1 | 5 (4) | 5 (2) | 4 (6) | 4 | 4 | 4 | 1 | 3 | 1 | 3 | 3 | 1
ITTC-2 | 1 (2) | 2 (1) | 1 (2) | 2 | 2 | 2 | 5 | 1 | 4 | 6 | 6 | 2
BART-large | 3 (6) | 3 (5) | 2 (4) | 3 | 3 | 3 | 4 | 4 | 5 | 2 | 2 | 3
LED-base-16k | 4 (3) | 4 (3) | 5 (5) | 5 | 5 | 5 | 2 | 2 | 2 | 1 | 1 | 4
SciSpace | 2 (1) | 1 (6) | 3 (3) | 1 | 1 | 1 | 6 | 6 | 6 | 4 | 4 | 6
BART (baseline) | 6 (5) | 6 (4) | 6 (1) | 6 | 6 | 6 | 3 | 5 | 3 | 5 | 5 | 5
Table 1:  System rankings based on automated metrics and human evaluation
(best in green). Original system ranks from the MSLR leaderboard as assessed
based on ROUGE-L, BERTScore, and Delta-EI are provided in parentheses. The
ranks in this table are produced over subsamples of reviews from the Cochrane
test split (and macro-averaged for ROUGE and BERTScore), causing ranks to
differ from leaderboard rankings.
*Ranking for ROUGE is based on Avg-ROUGE-F, while leaderboard rank is based on ROUGE-L.

###  3.2 Facet-based Human Evaluation

We adapt a facet-based human evaluation procedure from the analysis in
Otmakhova et al. (2022b). In their work, the authors analyzed baseline model
outputs from MS^2 (DeYoung et al., 2021) with respect to fluency, PIO
alignment, evidence direction, and modality (or strength of claim). PIO stands
for Population (who was studied? e.g. women with gestational diabetes),
Intervention (what was studied? e.g. metformin), and Outcome (what was
measured? e.g. blood pressure), and is a standard framework for structuring
clinical research questions Huang et al. (2006). These are important elements
that _must_ align between generated and target summaries for the former to be
considered accurate. Evidence direction describes the effect (or lack thereof)
that is supported by evidence (e.g., the treatment shows a positive effect, no
effect, or a negative effect, comparatively). The strength of the claim
indicates how much evidence or how strong the evidence associated with the
effect might be.

We derive 8 questions based on this analysis:

  1. 1.

_Fluency_ : if the generated summary is fluent

  2. 2.

_Population_ : whether the population in the generated and target summaries
agree

  3. 3.

_Intervention_ : as above for intervention

  4. 4.

_Outcome_ : as above for outcome

  5. 5.

_Effect-target_ : effect direction in the target

  6. 6.

_Effect-generated_ : effect direction in the generated summary

  7. 7.

_Strength-target_ : strength of claim in the target

  8. 8.

_Strength-generated_ : strength of claim in the generated summary

Of the 470 reviews in the Cochrane test set, we sample 100 reviews per system
for facet annotations (600 summaries in total). For 50 reviews, we fully
annotate all summaries from the 6 systems (the overlapping set); for the other
50 reviews per system, we sample randomly from among the remaining reviews for
each system (the random set). All together, at least one system’s outputs are
annotated for 274 reviews in the test set. We elect for this sampling strategy
to balance thoroughness (having sufficient data points to make direct
comparisons between systems) and coverage (having annotations across more
review topics).

For each sampled instance, we show annotators a pair of (target, generated)
summaries from a review and ask them to answer 8 questions regarding these
(details in App. A). A sample of 10 reviews from the overlapping set (60
summary pairs) and 10 from the random set (10 summary pairs) are annotated by
two annotators. We compute inter-annotator agreement from these and report
Cohen’s Kappa and agreement proportions for all eight facets in Table 2.
Several facets have lower agreement (Population, Outcome, and Strength-
target), though most disagreements are between similar classes (e.g. partial
agree vs. agree); more on this in App. A.

Two annotators with undergraduate biomedical training annotated these samples.
We arrived at the final annotation protocol following two rounds of pilot
annotations on samples from the MS^2 dataset and discussing among authors to
resolve disagreements and achieve consensus.

Question | Classes | $\kappa$ | Agreement
---|---|---|---
Fluency | 3 | 0.52 | 0.87
Population | 4 | 0.33 | 0.56
Intervention | 4 | 0.60 | 0.77
Outcome | 4 | 0.24 | 0.36
Effect-target | 4 | 0.85 | 0.90
Effect-generated | 4 | 0.78 | 0.90
Strength-target | 4 | 0.30 | 0.54
Strength-generated | 4 | 0.77 | 0.90
Table 2: Inter-annotator agreement between experts on facets (Cohen’s $\kappa$
and proportion of agreement).

###  3.3 Pairwise Human Evaluation

We perform pairwise comparisons to elicit human preferences between system-
generated summaries and to study how facet-based quality maps to holistic
summary quality.

We sample pairs of system generations from our dataset, half from the
overlapping set of reviews annotated for facet evaluations, and half from
other reviews. A different subsample of these pairwise comparisons is provided
to each of 5 raters, who are asked to complete up to 100 judgments each. For
each comparison, the annotator is given the target summary, the system A
summary, the system B summary, and asked “Which of A or B more accurately
reflects the content of the target summary?” where the options are A, B, or
Neither. All annotators are knowledgable in BioNLP and one annotator has
biomedical training. Four annotators completed 100 pairwise comparisons; a
fifth completed 50 comparisons.

We first determine system rankings per individual annotator. To tally
annotations: if A is preferred over B, system A gets 1 point; if B over A,
system B gets 1 point; if Neither is preferred, neither system gets a point.
Systems are ranked by total points; tied systems receive the same ranking. To
determine a combined ranking based on the preferences of all 5 annotators, we
adopt the Borda count Emerson (2013), a ranked choice vote counting method
that maximizes the probability of selecting the Condorcet winner.555The
Condorcet winner is the candidate that would win a head-to-head election
against each of the other candidates assuming a plurality vote. In this
method, for each annotator (voter), we award each system the number of points
corresponding to the number of systems ranked below it, e.g., for a set of
systems ranked 1-6, the rank 1 system receives 5 points, the rank 2 system 4
points, and so on. System rankings resulting from the Borda count are shown in
Table 1 under Pairwise-Combined.

We perform bootstrapping over each annotator’s pairwise annotations to
estimate the error of the overall system rankings. We resample each
individual’s pairwise preferences with replacement and compute a new combined
ranking. Over 10000 bootstrap samples, the average Spearman $\rho$ of the
resampled rankings against the initial rankings is 0.716 (s/d = 0.197).

###  3.4 Dataset Statistics

Our final dataset consists of 4658 summaries generated by 10 systems over 470
review instances from MSLR-Cochrane. Of these summaries, 597 from 6 systems
are annotated on 8 quality facets. We also include 452 pairwise comparisons
from five annotators. In addition to annotations, we compute and include
automated metrics for each generated summary to facilitate analysis (more in
§5).

##  4 Analysis of generated summaries

We perform lexical analysis of input abstracts, system generated summaries,
and target summaries in our dataset, summarizing our findings below.

#### Input copying and synthesis

To assess similarity between inputs and summaries, we first apply the evidence
inference pipeline Lehman et al. (2019); DeYoung et al.
(2020)666https://github.com/bwallace/RRnlp to identify an evidence statement
in each input document and classify it with an effect direction. Between each
input evidence statement and the target and generated summaries, we compute
ROUGE-1 scores. We compute the _Synthesis_ rate as how often the effect
direction agrees between the most similar evidence statement (by ROUGE-1
score) and the generated summary. In Table 3, we find that system generations
match the effect of the closest input at a high rate (0.41-0.46), though no
more frequently than we would expect based on the synthesis rate for the
target summaries (0.48). Using ROUGE-1 scores, we also determine how often a
generated summary is closer to an input document than the target (_Input
Match_), which might indicate whether a system is performing an implicit
synthesis by selecting an input and copying it. We find that systems sometimes
copy inputs, but not in any consistent way.

System | Synthesis | Input Match
---|---|---
Targets | 0.48 | -
ITTC1 | 0.46 | 0.26
ITTC2 | 0.45 | 0.15
BART-Large | 0.41 | 0.31
LED-Base-16K | 0.45 | 0.36
SciSpace | 0.44 | 0.48
BART (baseline) | 0.44 | 0.38
Table 3: Results of summary vs. input lexical analysis.

####  n-gram self-repetition

Previously, Salkar et al. (2022) noted that models fine-tuned on the Cochrane
corpus tend to generate summaries containing repeating patterns; however, they
claim that the amount of such self-repetition777Salkar et al. (2022) define
self-repetition as the proportion of generated summaries containing at least
one $n$-gram of length $\geq$4 which also occurs in at least one other
summary. is fairly consistent between model-generated and human-written text.
We analyze self-repetition rates for long $n$-grams (5- to 10-grams) and show
that their occurrence rates are much higher in generated summaries than in
human-written summaries. These long $n$-grams do not just represent stylistic
patterns, but can contain important information such as the effect direction,
e.g., “there is insufficient evidence to support the use” (see App. B for
details), so the high rate of self-repetition is very concerning.

We find a clear distinction between generated and target summaries in the
self-repetition of longer sequences, such as 7- to 10-grams (Figure 5 in App.
B). Though the amount of self-repeating 10-grams in human-written summaries is
negligible, it reaches over 80% in some of the examined models’ outputs. The
self-repetition rate for specific $n$-grams (the number of documents in which
an $n$-gram appears) in generated summaries is also much higher than in the
targets: some 7-grams occur in up to 70% of generated summaries (Figure 2;
trends for other long $n$-grams are in App. B).

Figure 2: Percent of summaries in which each self-repeating 7-gram appears
(Test = target summaries). Figure 3: Distribution of 7-grams in Train vs.
Target set summaries (left) and vs. BART summaries (right).

To determine the origin of these long $n$-grams, we calculate their overlap
with summaries in the _Train_ set and their corresponding input documents.
While overlap with inputs is nearly zero, up to 90% of long $n$-grams are also
found in _Train_ set summaries (Figure 6 in App. C). Interestingly, models
with global attention (LED or PRIMERA-based) seem to replicate more long
sequences from the _Train_ set summaries than BART-based ones, while in the
Pegasus-based system (SciSpace) a smaller amount of self-repetition can be
explained by fine-tuning. Finally, we observe that though the distributions of
self-repeating $n$-grams in the target summaries of the _Test_ set and _Train_
set are very similar (Figure 3; left), in generated summaries the rate of
self-repetition increases up to 500x compared to occurrence in the _Train_ set
summaries (Figure 3; right). Models amplify repeating patterns from the
_Train_ set to unnatural proportions!

##  5 Automated evaluation metrics

We compute automated metrics for each generated summary and include instance-
level scores in our dataset. We investigate how these metrics correlate with
other metrics (§5.1) and with human evaluation facets (§5.2).

#### Metrics from the MSLR leaderboard:

  * ROUGE: The leaderboard reported system-level ROUGE-1, ROUGE-2, and ROUGE-L F-scores Lin (2004). We report these same three metrics; in some plots, due to space constraints, we show the average of these three ROUGE metrics, which we call Avg-ROUGE-F.

  * BERTScore: We compute and report BERTScore-F Zhang et al. (2020a) for each generated summary as computed using the RoBERTa-large model.

  * Delta-EI: We compute Delta-EI as introduced by Wallace et al. (2020) and modified by DeYoung et al. (2021) for the MSLR shared task. The metric computes the probability distributions of evidence direction for all intervention-outcome (I/O) pairs between inputs and the target and generated summaries. The final score is a sum over the Jensen-Shannon Divergence of probability distributions over all I/O pairs. Lower values indicate higher similarity to the target summary.

#### Other metrics we propose and examine:

  * NLI/STS/ClaimVer: These metrics leverage Sentence-BERT Reimers and Gurevych (2019) and are computed as the cosine similarity between the embedding of the target summary and the embedding of the generated summary when encoded with trained SBERT models. We use three pretrained variants of SBERT: RoBERTa fine-tuned on SNLI and MultiNLI (NLI); RoBERTa fine-tuned on SNLI, MultiNLI, and the STS Benchmark (STS); and PubMedBERT fine-tuned on MS-MARCO and the SciFact claim verification dataset (ClaimVer).

  * PIO-Overlap: Following Otmakhova et al. (2022a), we employ a strong PIO extractor (BioLinkBERT (Yasunaga et al., 2022) trained on EBM-NLP (Nye et al., 2018)) to extract PIO spans. For each target-generated pair, we define PIO-Overlap as the intersection of the two extracted sets of PIO spans normalized by the number of PIO spans in the target summary. Spans are only considered to overlap if they have the same label and one span is a subspan of the other.

###  5.1 Correlation between automated metrics

We compute Pearson’s correlation coefficients between pairs of metrics (Figure
8 in App. E). Most automated metrics are significantly correlated (p $<$
0.01), except Delta-EI and PIO-Overlap. ROUGE and BERTScore show a strong
positive correlation (r = 0.75), and NLI and STS have a strong positive
correlation (r = 0.92), unsurprising since the underlying models are trained
on similar data. Delta-EI presents as bimodal, with two peaks around 0 and 1.
Distributions of instance-level automated metrics per system are shown in App.
D.

System ranks (§6) produced by automated metrics are highly correlated except
for PIO-Overlap, which is anti-correlated (Figure 1). Ordering systems based
on these metrics generally result in the same or similar rankings ($\rho$
$\geq$ 0.77 for all pairs of metrics besides PIO-Overlap), e.g., rankings from
ClaimVer, NLI, and STS are identical ($\rho$ = 1).

###  5.2 Correlation between automated metrics and human judgements

We investigate the relationship between automated metrics and human facet-
based annotations. For this analysis, we normalize human facets to 4 agreement
scores: Fluency, PIO, Direction, and Strength, each in the range [0, 1]
(details in App. F).

Metric | Flu. | PIO | Dir. | Str.
---|---|---|---|---
ROUGE | -0.014 | -0.010 | 0.007 | -0.035
BERTScore | -0.000 | 0.022 | 0.036 | -0.033
Delta-EI | 0.066 | -0.080 | -0.060 | -0.054
ClaimVer | -0.051 | 0.142** | -0.017 | -0.093*
NLI | -0.026 | 0.053 | -0.011 | -0.063
STS | -0.042 | 0.066 | 0.001 | -0.056
PIO-Overlap | 0.043 | 0.358** | 0.033 | 0.050
Table 4: Correlation coefficients between automated metrics and human
evaluation facets. There is weak to no correlation between metrics and human-
assessed facets (aside from between PIO-overlap and PIO). Statistical
significance at $\alpha$ = 0.05 is marked with *, 0.01 with **, though these
thresholds for significance do not account for multiple hypothesis testing.

Correlation coefficients between automated metrics and these four agreement
scores are given in Table 4; PIO correlations are plotted in Figure 10 in App
E. In general, there is weak to no correlation between metrics and human-
assessed Fluency, PIO, Direction, and Strength, suggesting that automated
metrics may not be adequately capturing aspects of summaries that humans
determine to be important. The exception is PIO-Overlap, which has a
statistically significant correlation with human-assessed PIO agreement, and
presents as a promising future metric for the MSLR task; ClaimVer is also
weakly correlated with PIO agreement.

Disappointingly, Delta-EI does not correlate with human-assessed Direction
agreement. We investigate this further by computing empirical cumulative
distribution functions (ECDFs) for each of the metrics w.r.t. Direction
agreement (App. E). Delta-EI exhibits a small but desirable difference between
instances where Direction agrees and instances where Direction disagrees
(Agrees is more likely to have lower Delta-EI scores than Disagrees). In sum,
Delta-EI shows some promise in detecting differences in Direction agreement,
though further refinement of the metric is needed.

##  6 Comparing system rankings

Evaluation metrics for summarization can be used in two settings, to judge
performance at the _instance_ level (comparing individual summaries) or at the
_system_ level (comparing model performance over many instances). Here, we
compare system-level rankings produced by automated metrics, human facet
evaluation, and pairwise preference annotations to determine whether automated
metrics effectively rank systems as humans would.

System rankings are computed by averaging the instance-level metric values or
scores across all review instances for each system, and ranking from best to
worst average score (direction depends on metric; higher is better for all
scores except Delta-EI). We only average metrics over the subset of reviews
for which we have human annotations. This ensures a fair comparison in the
circumstance where we have selected an annotation sample that a system
performs particularly well or poorly on. By doing this, the system rankings we
present here are different than those computed using the same metrics from the
MSLR leaderboards. We do not intend our computed rankings to be interpreted as
the true system ranking; our analysis focuses on whether automated metrics and
human evaluation are able to produce _similar_ rankings of systems. Table 1
shows rankings as assessed by all automated metrics and human scores; Figure 1
shows Spearman correlation coefficients.

Figure 4: Spearman rank correlations between system ranks for each pairwise
annotator and ranks derived from facet-based annotation. Annotators weigh
quality facets differently when performing pairwise judgments.

#### Rankings by automated metrics are not correlated with rankings by human
evaluation

In general, system rankings from commonly reported automated metrics are not
correlated or anti-correlated (lighter blue) with system rankings produced by
human judgments. System rankings from automated metrics are highly correlated
among themselves ($\rho$ close to 1), aside from PIO-Overlap. PIO-Overlap
rankings are strongly correlated with rankings from human PIO agreement. PIO-
Overlap and Delta-EI ranks also correlate with the combined pairwise rankings,
again suggesting that these two metrics may be the most promising for
capturing human notions of summary quality.

#### Pairwise assessments do not weigh facets equally

Pairwise-combined rankings are correlated with facet-based rankings for
Fluency and PIO, but not Direction and Strength of claim. This may indicate
that Fluency and PIO are more detectable problems, or that issues in Fluency
and PIO are more prevalent in our data. The rank correlations also show that
Direction and Strength are highly correlated and may capture similar aspects
of system-level summary quality, making the case for dropping one of the two
(likely Strength) in future annotations.

#### Pairwise preferences suggest that annotators weigh facets differently

In Figure 4, we show Spearman correlation coefficients of facet-based rankings
against the rankings of five pairwise annotators and the combined pairwise
ranking. These coefficients suggest that annotators weigh facets differently
when comparing system output. Annotator 1 ranks similarly to Fluency and PIO
facets, Annotators 2 and 5 rank similarly to PIO and Direction facets, while
Annotators 3 and 4’s rankings are uncorrelated with most facets.

##  7 Related work

Beyond ROUGE Lin (2004) and BERTScore Zhang et al. (2020a), an extensive list
of $n$-gram Papineni et al. (2002); Banerjee and Lavie (2005) and model-based
Zhao et al. (2019); Gao et al. (2020); Martins et al. (2020); Sellam et al.
(2020); Yuan et al. (2021) summarization evaluation metrics have been proposed
in the literature. In particular, model-based approaches that use question
generation and question answering Wang et al. (2020); Durmus et al. (2020);
Deutsch et al. (2021) or NLI-based models Kryscinski et al. (2020) have been
proposed to assess summary factual consistency. Fabbri et al. (2021) and
Deutsch et al. (2022) provide more thorough evaluations of many of these
metrics on select summarization tasks. We perform evaluations using metrics
previously reported on the MSLR task, and leave a systematic evaluation of
metrics on this task and others to future work.

In Zhang et al. (2020b), the authors performed fact verification on generated
radiology reports using an information extraction module, by aligning the
extracted entities with entities found in the reference summary. Our PIO-
Overlap metric similarly uses a PIO entity extraction module to assess concept
overlap between generated and reference summaries. Falke et al. (2019)
proposed to use NLI models to rank summaries by average entailment score per
sentence against the input documents; this shares similarities with the Delta-
EI score we evaluated, which attempts to quantify agreement relative to the
reference summary with respect to the direction of evidence reported.

Deutsch et al. (2022) investigated system-level rankings produced by automated
metrics and human evaluation and found minimal correlation between them, a
finding corroborated by our work. Liu et al. (2022) introduced the robust
summarization evaluation (RoSE) benchmark, containing human judgments for
system outputs on the CNN/DM, XSum, and SamSum datasets. We extend such work
into a novel domain (medical MDS for literature review) and demonstrate
differences in automated metric performance and human evaluation in our domain
and task. For example, though ROUGE correlates with human preferences in
single-document (CNN/DM) and multi-document (MultiNews) news summarization, we
find that it is poorly correlated with human judgments and preferences in the
MSLR task.

Recent developments in large language modeling have also shifted the goalposts
for evaluation. Goyal et al. (2022) found that although humans overwhelmingly
prefer zero-shot GPT-3 summaries for news summarization, automated metrics
were unable to capture this preference; they introduced a benchmark of human
judgments and rationales comparing system outputs on the single-document news
summarization task. More recently, Shaib et al. (2023) demonstrated that GPT-3
can be adapted for the MSLR task, and though the model outputs are generally
found by human annotators to be faithful to the inputs, in the MDS setting the
evidence direction often disagrees with the reference. Detecting these
disagreements and developing automated metrics that can capture such
disagreements are valuable pursuits and one of the motivations for our work.
Further investigation into whether automated metrics developed using limited
human evaluation benchmarks such as the dataset we introduce here will be a
goal for future work.

##  8 Discussion

MDS for literature review may involve notions of summary quality not readily
captured by standard summarization evaluation metrics. For example, our
lexical analysis of generated summaries reveals a concerning level of self-
repetition behavior, which is not penalized by standard metrics. Through two
independent human evaluations (facet-based and pairwise preferences), we also
show that automated metrics such as ROUGE and BERTScore are poorly correlated
or even anti-correlated with human-assessed quality. This is not to say that
these metrics do not provide any utility. Rather, further work is needed to
understand what aspects of summary quality these metrics capture, and how to
use them in combination with other metrics, novel metrics yet unintroduced, as
well as human evaluation to better assess progress. We note that ours is not a
systematic analysis of all automated summarization evaluation metrics, but is
a focused study on evaluation metrics reported for the MSLR shared task and
which we introduce under the hypothesis that they may be useful for capturing
some quality facets associated with this task. For those interested in the
former, please refer to studies such as Fabbri et al. (2021) or Deutsch et al.
(2022).

A positive finding from our work is the promise of the PIO-Overlap and Delta-
EI metrics. Delta-EI shows some potential to capture evidence directional
agreement between summaries, though the metric as currently implemented is
noisy and does not cleanly separate summaries that agree and disagree on
direction. PIO-Overlap, a metric we introduce, correlates with human-assessed
PIO agreement, suggesting that it could be a performant, scalable alternative
to human evaluation of this quality facet. Still, more work is needed to probe
how variants of these metrics could be adapted to evaluate performance on MSLR
and other MDS tasks.

Finally, we note that human evaluation is difficult because people value
different qualities in summaries. The rank-based analysis we perform does not
account for interactions between related quality facets and is unable to
elicit relationships between overall quality and individual quality facets.
The majority of pairwise preference annotations in our dataset also include
short free text justifications for preference decisions, which could be used
to further study this problem. Other promising directions for future work
involve studying how to optimally elicit human preferences, such as how to
sample instances for labeling to maximize our confidence in the resulting
system-level rankings.

##  9 Conclusions

There have been major recent advances in the generative capabilities of large
language models. Models like ChatGPT,888https://openai.com/blog/chatgpt GPT-3
Brown et al. (2020), and PubmedGPT999https://hai.stanford.edu/news/stanford-
crfm-introduces-pubmedgpt-27b demonstrate aptitude on many tasks but have also
been shown to confidently produce factually incorrect outputs in specialized
and technical domains.101010Stack Overflow banned ChatGPT responses due to the
high rate of inaccurate and misleading information. Medicine is a specialized
domain where incorrect information in generated outputs is difficult to
identify and has the potential to do harm. There is therefore a pressing need
for the community to develop better methods to assess the quality and
suitability of generated medical texts. Our investigation confirms that there
is significant room for improvement on medical MDS evaluation. We hope that
the resources and findings we contribute in this work can assist the community
towards this goal.

## Limitations

Though we include 6 systems in our annotation which reflect the current state-
of-the-art, all of the models are Transformer-based and fine-tuned on just the
Cochrane dataset, which may limit the diversity of our generated summaries.
Additionally, none of the systems are generating summaries that approach the
accuracy of human-written summaries. As a consequence, though the summaries in
our dataset span the spectrum of quality, they may have less coverage on the
higher end of quality (summaries approaching the accuracy and utility of
human-written review summaries).

Our analysis of evaluation metrics also assumes the existence of reference
summaries. In many real-world summarization scenarios, reference summaries do
not exist, and reference-free evaluation metrics are needed for assessment. We
refer the reader to related work in reference-free summarization evaluation
Vasilyev et al. (2020); Gao et al. (2020); Luo et al. (2022), which have been
found in some settings by Fabbri et al. (2021) to exhibit even lower
correlation with human notions of summary quality; the performance of these
metrics on MSLR evaluation is unknown and is left to future work.

Our notions of summary quality also do not necessarily correspond to clinical
utility. As with anything in the medical setting, it is of utmost importance
to verify correctness and the quality of evidence before using any generated
text to make or guide clinical decisions.

## Ethical Considerations

As with other applications of NLP in the medical domain, results of MSLR
systems must be verified by domain experts before they should be considered
for use in clinical guidance. We do not intend the system outputs included in
our dataset and analysis to be used for such end applications, as this would
be clearly premature given the low quality of generated summaries and our lack
of ability to assess the prevalence of factuality errors in these summary
texts. Nonetheless, we believe that medical MDS holds eventual promise, and it
is of vital importance that we study its challenges and how to measure and
detect quality issues in generated text.

## Acknowledgements

This research was partially supported by National Science Foundation (NSF)
grant RI-2211954, and by the National Institutes of Health (NIH) under the
National Library of Medicine (NLM) grant 2R01LM012086. YO and THT are
supported by the Australian Government through the Australian Research Council
Training Centre in Cognitive Computing for Medical Technologies (project
number ICI70200030).

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _ArXiv_ , abs/2004.05150.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Deutsch et al. (2021) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021.  Towards question-answering as an automatic metric for evaluating the content quality of a summary.  _Transactions of the Association for Computational Linguistics_ , 9:774–789.
  * Deutsch et al. (2022) Daniel Deutsch, Rotem Dror, and Dan Roth. 2022.  Re-examining system-level correlations of automatic summarization evaluation metrics.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 6038–6052, Seattle, United States. Association for Computational Linguistics.
  * Deutsch and Roth (2020) Daniel Deutsch and Dan Roth. 2020.  SacreROUGE: An open-source library for using and developing summarization evaluation metrics.  In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_ , pages 120–125, Online. Association for Computational Linguistics.
  * DeYoung et al. (2021) Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. 2021\.  MS2: Multi-document summarization of medical studies.  In _EMNLP_.
  * DeYoung et al. (2020) Jay DeYoung, Eric Lehman, Benjamin Nye, Iain Marshall, and Byron C. Wallace. 2020\.  Evidence inference 2.0: More data, better models.  In _Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing_ , pages 123–132, Online. Association for Computational Linguistics.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Emerson (2013) Peter Emerson. 2013.  The original borda count and partial voting.  _Social Choice and Welfare_ , 40:353–358.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  SummEval: Re-evaluating summarization evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Gao et al. (2020) Yang Gao, Wei Zhao, and Steffen Eger. 2020.  SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1347–1354, Online. Association for Computational Linguistics.
  * Gholipour Ghalandari et al. (2020) Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020.  A large-scale multi-document summarization dataset from the Wikipedia current events portal.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1302–1308, Online. Association for Computational Linguistics.
  * Giorgi et al. (2022a) John Giorgi, Luca Soldaini, Bo Wang, Gary Bader, Kyle Lo, Lucy Lu Wang, and Arman Cohan. 2022a.  Exploring the challenges of open domain multi-document summarization.  _ArXiv_ , abs/2212.10526.
  * Giorgi et al. (2022b) John Giorgi et al. 2022b.  MSLR leaderboard: led-base-16384-ms2.  https://leaderboard.allenai.org/mslr-ms2/submission/ccfknkbml1mljnftf7d0.  Accessed: 2022-09-15.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News summarization and evaluation in the era of gpt-3.  _ArXiv_ , abs/2209.12356.
  * Huang et al. (2006) Xiaoli Huang, Jimmy Lin, and Dina Demner-Fushman. 2006.  Evaluation of pico as a knowledge representation for clinical questions.  In _AMIA annual symposium proceedings_ , volume 2006, page 359. American Medical Informatics Association.
  * Ioannidis (2016) John PA Ioannidis. 2016.  The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.  _The Milbank Quarterly_ , 94(3):485–514.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Lehman et al. (2019) Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C Wallace. 2019.  Inferring which medical treatments work from reports of clinical trials.  _arXiv preprint arXiv:1904.01606_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5070–5081, Florence, Italy. Association for Computational Linguistics.
  * Liu et al. (2022) Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir R. Radev. 2022.  Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation.  _ArXiv_ , abs/2212.07981.
  * Lu et al. (2020) Yao Lu, Yue Dong, and Laurent Charlin. 2020.  Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8068–8074, Online. Association for Computational Linguistics.
  * Luo et al. (2022) Ge Luo, Hebi Li, Youbiao He, and Forrest Sheng Bao. 2022.  PrefScore: Pairwise preference learning for reference-free summarization quality assessment.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 5896–5903, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Martins et al. (2020) Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2020.  Sparse text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4252–4273, Online. Association for Computational Linguistics.
  * Nye et al. (2018) Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. 2018.  A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 197–207, Melbourne, Australia. Association for Computational Linguistics.
  * Otmakhova et al. (2022a) Yulia Otmakhova, Thinh Hung Truong, Timothy Baldwin, Trevor Cohn, Karin Verspoor, and Jey Han Lau. 2022a.  LED down the rabbit hole: exploring the potential of global attention for biomedical multi-document summarisation.  In _Proceedings of the Third Workshop on Scholarly Document Processing_ , pages 181–187, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Otmakhova et al. (2022b) Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, and Jey Han Lau. 2022b.  The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5098–5111, Dublin, Ireland. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pasunuru et al. (2021) Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, and Markus Dreyer. 2021\.  Efficiently summarizing text and graph encodings of multi-document clusters.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4768–4779, Online. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Salkar et al. (2022) Nikita Salkar, Thomas Trikalinos, Byron Wallace, and Ani Nenkova. 2022.  Self-repetition in abstractive neural summarizers.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 341–350, Online only. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shaib et al. (2023) Chantal Shaib, Millicent Li, Sebastian Joseph, Iain James Marshall, Junyi Jessy Li, and Byron Wallace. 2023.  Summarizing, simplifying, and synthesizing medical evidence using gpt-3 (with varying success).
  * Shen et al. (2022) Zejiang Shen, Kyle Lo, Lauren Jane Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022.  Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities.  _ArXiv_ , abs/2206.10883.
  * Shinde et al. (2022) Kartik Shinde, Trinita Roy, and Tirthankar Ghosal. 2022.  An extractive-abstractive approach for multi-document summarization of scientific articles for literature review.  In _Proceedings of the Third Workshop on Scholarly Document Processing_ , Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Tangsali et al. (2022) Rahul Tangsali, Aditya Vyawahare, Aditya Mandke, Onkar Litake, and Dipali Kadam. 2022.  Abstractive approaches to multidocument summarization of medical literature reviews.  In _Proceedings of the Third Workshop on Scholarly Document Processing_ , Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Vasilyev et al. (2020) Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020.  Fill in the BLANC: Human-free quality estimation of document summaries.  In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pages 11–20, Online. Association for Computational Linguistics.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Online. Association for Computational Linguistics.
  * Wadden et al. (2022) David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022.  Scifact-open: Towards open-domain scientific claim verification.  _ArXiv_ , abs/2210.13777.
  * Wallace et al. (2020) Byron C. Wallace, Sayantani Saha, Frank Soboczenski, and Iain James Marshall. 2020\.  Generating (factual?) narrative summaries of RCTs: Experiments with neural multi-document summarization.  In _AMIA Annual Symposium_.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Wolhandler et al. (2022) Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Dagan. 2022.  How "multi" is multi-document summarization?  _ArXiv_ , abs/2210.12688.
  * Xiao et al. (2022) Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022.  PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5245–5263, Dublin, Ireland. Association for Computational Linguistics.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022.  LinkBERT: Pretraining language models with document links.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8003–8016, Dublin, Ireland. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _ArXiv_ , abs/2106.11520.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big bird: Transformers for longer sequences.  _ArXiv_ , abs/2007.14062.
  * Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a.  Bertscore: Evaluating text generation with BERT.  _ArXiv_ , abs/1904.09675.
  * Zhang et al. (2020b) Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D. Manning, and Curtis Langlotz. 2020b.  Optimizing the factual correctness of a summary: A study of summarizing radiology reports.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5108–5120, Online. Association for Computational Linguistics.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Facet-based Annotation

The questions and answer options shown to annotators for facet annotation are
shown in Table 5. If merging all Yes and Partial Yes classes, agreement
proportion between annotators increases for Fluency (0.87 $\rightarrow$ 0.97),
Population (0.56 $\rightarrow$ 0.64), Intervention (0.77 $\rightarrow$ 0.90),
and Outcome agreement (0.36 $\rightarrow$ 0.44).

Question |  Answer options
---|---
1\. Is the generated summary fluent? |  2: Yes–there are no errors that impact comprehension of the summary 1: Somewhat, there are some minor grammatical or lexical errors, but I can mostly understand 0: No, there are major grammatical or lexical errors that impact comprehension
2\. Is the *population* discussed in the generated summary the same as the population discussed in the target summary? |  2: Yes 1: Partially 0: No N/A: No population in generated summary Other: Comment
3\. Is the *intervention* discussed in the generated summary the same as the intervention discussed in the target summary? |  2: Yes 1: Partially 0: No N/A: No intervention in generated summary Other: Comment
4\. Is the *outcome* discussed in the generated summary the same as the outcome discussed in the target summary? |  2: Yes 1: Partially 0: No N/A: No outcome in generated summary Other: Comment
5\. What is the effect direction in the *target* summary for the main intervention and outcome considered? |  (+1): Positive effect 0: No effect (-1): Negative effect N/A: no effect direction is specified in the target summary Other: Comment
6\. What is the effect direction in the *generated* summary for the main intervention and outcome considered? |  (+1): Positive effect 0: No effect (-1): Negative effect N/A: no effect direction is specified in the generated summary Other: Comment
7\. What is the strength of the claim made in the *target* summary? |  3: Strong claim 2: Moderate claim 1: Weak claim 0: Not enough evidence (there is insufficient evidence to draw a conclusion) N/A: No claim (there is no claim in the summary) Other: Comment
8\. What is the strength of the claim made in the *generated* summary? |  3: Strong claim 2: Moderate claim 1: Weak claim 0: Not enough evidence (there is insufficient evidence to draw a conclusion) N/A: No claim (there is no claim in the summary) Other: Comment
Table 5: Questions and answer options used during facet annotation.

##  Appendix B Self-repetition rates in generated summaries

Most of the long n-grams repeating across documents contain meaningful
statements regarding the direction or strength of effect findings rather than
purely stylistic patterns, which means that the systems are prone to
introducing factuality mistakes by replicating common statements. In Table 6
we show the examples of the most repetitive 8-grams for the 6 models, together
with the percentage of generated summaries they occur in.

We also show that the self-repetition rate for $n$-grams with n > 4 have very
dissimilar trends for generated summaries in comparison to human-written
summaries (Figure 5) The amount of 5-grams and higher self-repetition also
differs between models .

Model | Most frequent 8-gram | Self-repetition rate (%)
---|---|---
Targets | the conclusions of the review once assessed . | 1.5
LED-base-16k | there is insufficient evidence to support or refute | 9.4
ITTC-1 | there is insufficient evidence to support the use | 18.7
BART-large | there is insufficient evidence to support the use | 22.8
SciSpace | there is insufficient evidence to support the use | 55.5
BART (baseline) | there is insufficient evidence from randomised controlled trials | 59.1
ITTC-2 | there is insufficient evidence to support the use | 65.1
Table 6: Examples of 8-grams which are most frequently repeated across
generated summaries, together with their self-repetition rate. Figure 5: Rate
of self-repetition for models generations and the human written summaries
(Test) Figure 6: Percentage of self-repeating $n$-grams in generated summaries
which also occur in the target summaries of the Train set. Figure 7:
Distribution of instance-level automated metric values by system (n=470 for
each system). Each subplot is sorted on system from best to worst by median
score. The median score ranking is typically not identical to the ranking of
the corresponding metric from the MSLR leaderboard, which are computed based
on micro-averaged metric values.

##  Appendix C Copying self-repeating $n$-grams from training set

In Figure 6, we show the percentages of self-repeating $n$-grams from
generated summaries which can also be found in the target summaries in the
Train set.

##  Appendix D Automated metric distributions per system

Distributions of automated metrics for all instances per system are shown in
Figure 7.

##  Appendix E Correlations between metrics in the Cochrane dataset

Figure 8: Correlations between automated metrics in the Cochrane dataset.
Pearson’s correlation coefficients (r) are shown, along with an * if p < 0.01.

We present correlations between all automated metrics along with correlation
coefficients (Figure 8). ROUGE and BERTScore are strongly correlated. NLI and
STS are strongly correlated. Delta-EI has a bimodal distribution. PIO-Overlap
is uncorrelated with other metrics.

Correlations between automated metrics and the normalized PIO facet score are
shown in Figure 9. In general, automated metrics are poor predictors of PIO
agreement, except PIO-Overlap, which is positively correlated with PIO
agreement (p < 0.05). This confirms that model extraction and alignment of PIO
spans is a promising direction for assessing PIO agreement. ClaimVer also
shows a weak but statistically significant correlation with PIO agreement. The
ClaimVer metric is computed based on embedding similarity between two texts
using a model trained on the SciFact scientific claim verification dataset
Wadden et al. (2020); the SciFact task measures whether evidence entails or
refutes a scientific claim, which is somewhat analogous to our evaluation task
for medical multi-document summarization.

We also assess whether metrics can distinguish between summaries where the
Direction agrees with the target and summaries where the Direction disagrees.
We present the empirical cumulative distribution functions (ECDF) for each
automated metric, showing the separation of metrics between when Direction
agrees and disagrees (Figure 10. The Delta-EI metric is somewhat sensitive to
human-assessed directional agreement (a higher proportion of generated
summaries where the Direction agrees with the target have lower Delta-EI
scores), though we note that the difference is small. PIO-Overlap also shows
some separation between the two Direction classes (a higher proportion of
disagrees have lower PIO-Overlap score than agrees), though again the
difference is subtle.

Figure 9: Correlations between automated metrics and the normalized PIO facet
score. Figure 10: Empirical cumulative distribution function (ECDF) of each of
the automated metrics and their values for summaries where humans assessed the
evidence direction to Agree versus those assessed to Disagree.

##  Appendix F Normalizing human facet scores

Responses to the Fluency question result in a 3-class ordinal variable that we
map to the range [0, 1], where 0.0 is disfluent, 0.5 is somewhat fluent, and
1.0 is fluent. PIO aggregates agreement over Population, Intervention, and
Outcome, where each of P, I, and O are 3-class ordinal variables that we map
to the range [0, 1] as we do Fluency; we average the three facets to get PIO
agreement. For evidence direction, though each of the two annotated questions
has 4 answers (positive, no effect, negative, or no direction given), we elect
to define Direction as a binary class. We normalize Direction to 1 if the
target direction and generated direction agree and 0 if they disagree. For
Strength, each of the two annotated questions has 4 answers (strong, moderate,
weak, and not enough evidence). We take the difference between the answers for
the target and generated summaries and normalize to the range [0, 1] to yield
our Strength agreement score.


# Superluminal matter waves

J.P. Palastro<EMAIL_ADDRESS>University of Rochester, Laboratory for
Laser Energetics, Rochester, New York 14623-1299 USA  D. Ramsey  University of
Rochester, Laboratory for Laser Energetics, Rochester, New York 14623-1299 USA
M. Formanek  ELI Beamlines Facility, The Extreme Light Infrastructure ERIC,
252 41 Dolní Břežany, Czech Republic  J. Vieira  GoLP/Instituto de Plasmas e
Fusão Nuclear, Instituto Superior Técnico, Universidade de Lisboa, Lisbon
1049-001, Portugal  A. Di Piazza  University of Rochester, Laboratory for
Laser Energetics, Rochester, New York 14623-1299 USA  Department of Physics
and Astronomy, University of Rochester, Rochester, New York 14627, USA

###### Abstract

The Dirac equation has resided among the greatest successes of modern physics
since its emergence as the first quantum mechanical theory fully compatible
with special relativity. This compatibility ensures that the expectation value
of the velocity is less than the vacuum speed of light. Here, we show that the
Dirac equation admits free-particle solutions where the peak amplitude of the
wavefunction can travel at any velocity, including those exceeding the vacuum
speed of light, despite having a subluminal velocity expectation value. The
solutions are constructed by superposing basis functions with correlations in
momentum space. These arbitrary velocity wavefunctions feature a near-constant
profile and may impact quantum mechanical processes that are sensitive to the
local value of the probability density as opposed to expectation values.

The discovery of the Dirac equation stands as a foundational achievement of
modern theoretical physics Dirac (1928). As the first quantum mechanical
theory fully compatible with special relativity, the Dirac equation resolved
several inconsistencies that beset the, otherwise widely successful,
Schrödinger equation Schrödinger (1926). The equation and its solutions
preserve many of the quantum mechanical concepts developed within the context
of the Schrödinger equation, such as probability currents, expectation values,
and operators. Unlike the Schrödinger equation, however, the Dirac equation
was formulated from the outset to exhibit a Lorentz-invariant and Hamiltonian
structure consistent with special relativity. In doing so, the Dirac equation
precluded unphysical phenomena fully allowed by the Schrödinger equation:
namely, expectation values for velocity that exceed the vacuum speed of light.

At a fundamental level, the Dirac equation is a wave equation. Within first
quantization, the solutions, or wavefunctions $\psi$, describe the quantum
mechanical state of a charged lepton, while their adjoint product
$\psi^{\dagger}\psi$ provides the probability density of finding the lepton
within a particular region of space–time or momentum–energy. In the absence of
fields (and vacuum nonlinearities), the simplest solution is a plane wave
modulating a constant spinor Peskin and Schroeder (1995). Physically
occurring, localized wavefunctions are typically formed by superposing these
solutions with amplitudes and phases that are uncorrelated in space–time or
momentum–energy. Superpositions featuring correlations in space–time or
momentum–energy allow for wavefunctions with more-complex and potentially
advantageous structures.

Insight into the fundamental properties of matter and light and the potential
for applications has driven a growing interest in structured solutions for
quantum mechanical and electromagnetic waves, with ideas from one often being
adapted to the other Berry and Balazs (1979); Allen  _et al._ (1992); Berry
(1998); Bliokh  _et al._ (2007); Verbeeck  _et al._ (2010); Kaminer  _et al._
(2012); Voloch-Bloch  _et al._ (2013); Grillo  _et al._ (2014); Kaminer  _et
al._ (2015); Lloyd  _et al._ (2017); Hall and Abouraddy (2023); Campos  _et
al._ (2024); Chirita Mihaila  _et al._ (2022); Wong  _et al._ (2024). One of
the most-iconic examples is the concept of orbital angular momentum from
quantum mechanics being adapted to electromagnetism Allen  _et al._ (1992);
Berry (1998). Within quantum mechanics, this “wavefunction engineering” has
been applied to: the formation of self-accelerating solutions, which appear to
move under the influence of a potential in the absence of any potential
Kaminer  _et al._ (2015); Campos  _et al._ (2024); transverse shaping of
electron wavefunctions for added flexibility in scanning electron microscopes
Chirita Mihaila  _et al._ (2022); and the formation of free-electron crystals
for enhancing x-ray radiation Wong  _et al._ (2024). In each of these, the
physical processes of interest are sensitive to the local properties of the
wavefunction as opposed to expectation values.

Here, we introduce a structured free-particle solution to the Dirac equation
where the peak amplitude of the wavefunction can travel at any velocity,
including those exceeding the vacuum speed of light, despite having a
subluminal velocity expectation value (Fig. 1). Motivated by similar solutions
found in electromagnetism (e.g., space–time wavepackets Longhi (2003);
Kondakci and Abouraddy (2017); Yessenov  _et al._ (2022) and flying focus
pulses Sainte-Marie  _et al._ (2017); Froula  _et al._ (2018); Palastro  _et
al._ (2018); Di Piazza (2021); Formanek  _et al._ (2023); Ramsey  _et al._
(2023); Palastro  _et al._ (2024)), the solutions feature a near-constant
profile and are constructed by superposing basis functions with correlations
in momentum–energy. We expect that the solutions, or approximations thereof,
could be produced by light-matter interactions, such as the Kapitza–Dirac
effect Kapitza and Dirac (1933); Freimund and Batelaan (2002); Shiloh  _et
al._ (2014); Grillo  _et al._ (2014); Vanacore  _et al._ (2018); Feist  _et
al._ (2020); Reinhardt and Kaminer (2020); García de Abajo and Konečná (2021);
Lin  _et al._ (2024), and generalized to exhibit more exotic properties, such
as a modified effective mass in the absence of fields. These arbitrary
velocity wavefunctions may impact quantum mechanical phenomena that are
sensitive to the local value and velocity of the probability density, like
Smith–Purcell or Cherenkov radiation Pupasov-Maksimov and Karlovets (2021), as
opposed to expectation values.

The wavefunction of a spin one-half charged particle evolves according to the
Dirac equation. In natural units ($\hbar=c=1$), the Schrödinger form of the
Dirac equation is given by

$i\partial_{t}\psi=H\psi,$ (1)

where $H=-i\bm{\alpha}\cdot\bm{\nabla}+\beta m$ is the Hamiltonian,

$\displaystyle\alpha^{j}=\begin{pmatrix}0&\sigma^{j}\\\
\sigma^{j}&0\end{pmatrix},$ $\displaystyle\beta=\begin{pmatrix}I&0\\\
0&-I\end{pmatrix},$

the $\sigma^{j}$ with $j=1,2,3$ are the Pauli matrices, $I$ is the identity
matrix, and $m$ is the mass of the particle. Here and throughout, bold font
denotes three-vectors, $\bm{a}^{2}\equiv\bm{a}\cdot\bm{a}$, the shorthand
notations $p\equiv p^{\mu}$ and $x\equiv x^{\mu}$ are used for the momentum
and position four-vectors, $a\cdot b\equiv
a^{\mu}b_{\mu}=a_{0}b_{0}-\bm{a}\cdot\bm{b}$, and relativistic notation for
sub and superscripts is not used.

The Dirac equation admits four plane wave solutions, corresponding to positive
and negative energy and spin states. A general wavefunction can be expressed
as a superposition of these solutions. For simplicity and definiteness, a
superposition of solutions with positive energy and spin in the rest frame
will be considered, such that

$\psi(x)=\frac{1}{(2\pi)^{3/2}}\int\frac{1}{\sqrt{2p_{0}}}u(p)f(p)e^{-ip\cdot
x}\delta(p_{0}-E_{\bm{p}})d^{4}p,$ (2)

where

$u(p)=\sqrt{\scriptstyle E_{\bm{p}}+m}\begin{pmatrix}\chi_{+}\\\
\frac{\bm{\sigma}\cdot\bm{p}}{E_{\bm{p}}+m}\chi_{+}\\\ \end{pmatrix},$ (3)

is the bispinor normalized so that $u^{\dagger}\beta u=2m$, $\dagger$
indicates a Hermitian conjugate, $\chi_{+}=(\begin{smallmatrix}1\\\
0\end{smallmatrix})$, $E_{\bm{p}}=(m^{2}+\bm{p}^{2})^{1/2}>0$, and the Dirac
delta function $\delta$ ensures the “on-shell” condition. The complex scalar
function $f(p)$ determines the relative phase and amplitude of each plane wave
that composes the wavefunction. Here, $f(p)$ is expressed as an integral over
an auxiliary parameter $q$:

$f(p)=\int N(p,q)dq.$ (4)

The utility of this auxiliary parameter will become clear below. The functions
$f(p)$ and $N(p,q)$ are constrained by the normalization condition

$\int\psi^{\dagger}\psi d^{3}x=\int|f(p)|^{2}d^{3}p=1,$ (5)

where $\psi^{\dagger}\psi$ is the probability density and, for the remainder,
$p_{0}=E_{\bm{p}}$ is implied.

The evolution of the wavefunction can be characterized by three types of
velocities: the phase velocities of the plane wave solutions, the eigenvalues
(or expectation values) of the velocity operator, and the group velocity. The
phase velocities, $\bm{v}_{p}=E_{\bm{p}}\bm{p}/\bm{p}^{2}$, are always
superluminal ($|\bm{v}_{p}|>1$). The velocity operator $\bm{v}$ is found by
evaluating the commutator of the Hamiltonian and position vector:
$\bm{v}=i[H,\bm{x}]=\bm{\alpha}$. Using Eq. (2), one finds
$\langle\bm{v}\rangle=\int\psi^{\dagger}\bm{v}\psi
d^{3}x=\int(\bm{p}/E_{\bm{p}})|f(p)|^{2}d^{3}p=\langle\bm{p}/E_{\bm{p}}\rangle$,
which, as expected, is always subluminal ($|\bm{p}/E_{\bm{p}}|<1$).

The group velocity $\bm{v}_{g}$ can take any value in any direction. For a
typical wavefunction, the components of the four momenta must satisfy the on-
shell condition but are otherwise independent. As a result, $\partial
p_{j}/\partial p_{k}=\delta_{jk}$ and

$v_{g,j}=\frac{\partial E_{\bm{p}}}{\partial p_{j}}=\frac{p_{j}}{E_{\bm{p}}},$
(6)

which is consistent with the eigenvalues of the velocity operator
$\langle\bm{v}\rangle=\langle\bm{p}/E_{\bm{p}}\rangle$. However, if the
momenta have some correlation, i.e., they are interdependent, then $\partial
p_{j}/\partial p_{k}\neq\delta_{jk}$ and

$v_{g,j}=\frac{\partial E_{\bm{p}}}{\partial
p_{j}}=\frac{p_{j}}{E_{\bm{p}}}+\frac{1}{2E_{\bm{p}}}\frac{\partial}{\partial
p_{j}}(\bm{p}^{2}-p_{j}^{2}).$ (7)

In this case, the correlated momenta introduce an additional contribution (the
second term on the right-hand side) that decouples the group velocity from
$\bm{p}/E_{\bm{p}}$.

Figure 1: Evolution of the probability density $\psi^{\dagger}\psi$ for a
lepton with a velocity expectation value $\langle v_{3}\rangle=0.9$ and a
group velocity $v_{a}=2$. The peak of the probability density travels
superluminally while maintaining a near-constant profile. The yellow (circle,
dotted), white (square, solid), and blue (diamond, dashed) lines indicate
signals travelling at the velocities 2, 1, and 0.9, respectively. In this
example, $\bar{\mathcal{P}}=2m$, $w=0.1m$, $\xi_{0}=250m^{-1}$, and
$\Delta\zeta=1400m^{-1}$ [see Eqs. (25) – (28)].

As an example, consider the case of an arbitrary, constant group velocity
$\bm{v}_{a}$. Direct integration of $\partial E_{\bm{p}}/\partial p_{j}$
yields

$E_{\bm{p}}=\bm{v}_{a}\cdot\bm{p}+\kappa,$ (8)

where $\kappa$ is a constant. Setting
$(m^{2}+\bm{p}^{2})^{1/2}=\bm{v}_{a}\cdot\bm{p}+\kappa$ ensures that the on-
shell condition is satisfied and determines the interdependence of the momenta
(Fig. 2). Specifically,

$\bm{p}^{2}-(\bm{v}_{a}\cdot\bm{p})^{2}-2\kappa(\bm{v}_{a}\cdot\bm{p})+m^{2}-\kappa^{2}=0.$
(9)

Without loss of generality, the specified group velocity $\bm{v}_{a}$ will be
aligned along the $j=3$ axis, i.e., $\bm{v}_{a}=v_{a}\bm{e}_{3}$, where
$\bm{e}_{j}$ denotes a unit vector. Substitution of Eqs. (8) and (9) into the
right-hand-side of Eq. (7) verifies that indeed $v_{g,3}=v_{a}$. The condition
on the momenta [Eq. (9)] is a quadratic equation whose solution provides
$p_{3}$ in terms of $\bm{p}_{\perp}\equiv p_{1}\bm{e}_{1}+p_{2}\bm{e}_{2}$.
Upon solving this equation, one finds

$p_{3}=p_{c}\equiv\kappa\gamma_{a}^{2}v_{a}\pm\left[(\kappa\gamma_{a}^{2})^{2}-\gamma_{a}^{2}(m^{2}+\bm{p}_{\perp}^{2})\right]^{1/2},$
(10)

where $\gamma_{a}\equiv 1/\sqrt{1-v_{a}^{2}}$. Note that $\gamma_{a}$ only
appears in even powers. Thus, a superluminal group velocity ($|v_{a}|>1$) does
not produce a complex valued $p_{c}$.

The interdependence of the momenta, as described by Eq. (10), can be built
into the wavefunction by making use of the auxiliary parameter $q$. Expressing
$q=q(\kappa)$ as a function of $\kappa$ (to be determined below) and setting

$N(p,q)=\mathcal{N}(p,q)\delta(p_{3}-p_{c})$ (11)

yields

$f(p)=\int\mathcal{N}(p,q)\delta(p_{3}-p_{c})dq,$ (12)

where the dependence of $p_{c}$ and $q$ on $\kappa$ is understood. Together,
Eqs. (2) and (12) describe a superposition of solutions with correlated
momenta parameterized by $\kappa$. Each solution has the same arbitrary group
velocity $v_{g,3}=v_{a}$.

To elucidate the physical meaning of the parameter $\kappa$ and facilitate
further calculation, it is helpful to substitute Eq. (12) into Eq. (2) and
define the function

$\displaystyle\Phi(x,q)=\int$
$\displaystyle\frac{1}{\sqrt{2E_{\bm{p}}}}u(p)\mathcal{N}(p,q)e^{-ip\cdot
x}\delta(p_{3}-p_{c})d^{3}p,$ (13)

such that $\psi(x)=(2\pi)^{-3/2}\int\Phi(x,q)dq$. Several properties of the
arbitrary group velocity solutions can be analyzed by considering the phase in
Eq. (13): $p\cdot x=p_{0}x_{0}-p_{3}x_{3}-\bm{p}_{\perp}\cdot\bm{x}_{\perp}$.
Making use of the delta function, the phase contribution $\phi\equiv
p_{0}x_{0}-p_{3}x_{3}$ becomes

$\displaystyle\phi$ $\displaystyle=(p_{3}v_{a}+\kappa)x_{0}-p_{3}x_{3}=\kappa
x_{0}+p_{3}(v_{a}x_{0}-x_{3}).$ (14)

The first term is proportional to $\kappa$ but does not depend on $p$. As a
result, the factor $e^{-i\kappa x_{0}}$ can be extracted from the integrand.
This factor shows that $\kappa$ quantifies the temporal phase advance of the
wavefunction. The second term in Eq. (14) reveals that the integrand depends
on $x_{0}$ and $x_{3}$ only in the combination $v_{a}x_{0}-x_{3}$. Thus, the
integrand advects at the group velocity $v_{a}$.

An intuitive, and often helpful, picture of a wavefunction is that of an
“envelope” modulated by a “carrier” wave. The envelope describes the bulk
motion of the probability density and propagates at the group velocity, while
the carrier describes the motion of the phase fronts, which propagate at the
phase velocity. The function $\Phi(x,q)$ can be expressed in this same
framework. Adding and subtracting a longitudinal (along $x_{3}$) momentum
offset $\mathcal{P}$ to Eq. (14) provides

$\phi=\mathcal{P}\left[(v_{a}+\tfrac{\kappa}{\mathcal{P}})x_{0}-x_{3}\right]+(p_{3}-\mathcal{P})(v_{a}x_{0}-x_{3}).$
(15)

With this offset, the first and second terms in $\phi$ describe the phase of
the carrier wave, which can be extracted from the integrand as before, and the
advection of the integrand (i.e., the envelope) at the group velocity.
Choosing

$\kappa=\mathcal{E}-v_{a}\mathcal{P},$ (16)

where $\mathcal{E}=(m^{2}+\mathcal{P}^{2})^{1/2}$, yields

$\phi=\mathcal{P}(\beta_{p}x_{0}-x_{3})+(p_{3}-\mathcal{P})(v_{a}x_{0}-x_{3})$
(17)

and ensures that the phase fronts of the carrier wave travel at a phase
velocity $\beta_{p}\equiv\mathcal{E/P}$. Note that this is equivalent to the
procedure of enveloping the wavefunction about the longitudinal momentum
$\mathcal{P}$ and energy $\mathcal{E}$.

With the auxiliary parameter $\kappa=\mathcal{E}-v_{a}\mathcal{P}$, the
condition on the interdependence of the momenta [Eq. (10)] simplifies to

$p_{c}-\mathcal{P}=\gamma_{a}^{2}(v_{a}\mathcal{E}-\mathcal{P})-\varpi\left[\gamma_{a}^{4}(v_{a}\mathcal{E}-\mathcal{P})^{2}-\gamma_{a}^{2}\bm{p}_{\perp}^{2}\right]^{1/2},$
(18)

where the choice of sign
$\varpi\equiv\mathrm{sign}[\gamma_{a}^{2}(v_{a}\mathcal{E}-\mathcal{P})]$
ensures that the envelope varies slowly compared to the phase fronts. Equation
(18) demonstrates two important points: First and foremost that $\mathcal{P}$
corresponds to the value of $p_{3}$ when $\bm{p}_{\perp}=0$. And second that
arbitrary group velocity solutions are precluded in one dimension. In one
dimension, the wavefunction is composed solely of plane waves with
$\bm{p}_{\perp}=0$, such that $p_{c}$ is always equal to $\mathcal{P}$. This
eliminates the second term in Eq. (17), which is responsible for the movement
of the envelope at $v_{a}$. Figure 2(b) displays a family of longitudinal
momenta $p_{3}$ [Eq. (18)] parameterized by $\mathcal{P}$ as a function of
perpendicular momentum $|\bm{p}_{\perp}|$ for a superluminal group velocity
$v_{a}=2$.

Figure 2: (a) Satisfying the on-shell condition for an arbitrary group
velocity $v_{a}$ is equivalent to finding the intersection between a four-
dimensional hyperboloid, $m^{2}=E_{p}^{2}-\bm{p}^{2}$, and the hyperplane
$E_{\bm{p}}=\bm{v}_{a}\cdot\bm{p}+\kappa.$ Here, the dimensionality has been
reduced for visualization purposes by setting $p_{2}=0$. (b) Projections of
the intersection in the $p_{3}$–$p_{1}$ plane for different values of
$\mathcal{P}$. The values of $\mathcal{P}$ define the vertices of the
projection, i.e., the values of $p_{3}$ where $p_{1}=p_{2}=0$ [Eq. (18)]. The
parameters are the same as in Fig. 1.

A distinguishing property of the Dirac equation is its Lorentz covariance.
Lorentz transformations of the arbitrary group velocity solutions [Eqs. (13)
and (18)] provides additional insight into their interpretation. Upon
performing a longitudinal Lorentz transformation to a frame moving at a
velocity $v_{L}$ with respect to the nominal frame, Eq. (17) becomes

$\displaystyle\phi$
$\displaystyle=\frac{\mathcal{P}}{\gamma_{L}(1+\beta_{p}^{\prime}v_{L})}(\beta_{p}^{\prime}x^{\prime}_{0}-x^{\prime}_{3})$
(19) $\displaystyle\hskip
16.0pt+\gamma_{L}(p_{3}-\mathcal{P})[(v_{a}-v_{L})x^{\prime}_{0}-(1-v_{a}v_{L})x^{\prime}_{3}],$

where ′ denotes coordinates in the moving frame and
$\beta^{\prime}_{p}=(\beta_{p}-v_{L})/(1-\beta_{p}v_{L})$ is the nominal phase
velocity in the moving frame. One may expect that $v_{L}=v_{a}$ would be a
natural choice for the frame velocity. However, special care must be taken to
ensure that the nominal phase velocity is always superluminal (i.e., the on-
shell condition is satisfied). For $|v_{a}|<1$, one can indeed set
$v_{L}=v_{a}$ to find

$\displaystyle\phi_{<}=\frac{\mathcal{P}}{\gamma_{a}(1+\beta_{p}^{\prime}v_{a})}(\beta_{p}^{\prime}x^{\prime}_{0}-x^{\prime}_{3})-\frac{(p_{3}-\mathcal{P})}{\gamma_{a}}x^{\prime}_{3},$
(20)

but for $|v_{a}|>1$ the on-shell condition requires $v_{L}=v^{-1}_{a}$, which
yields

$\displaystyle\phi_{>}=\frac{v_{a}\mathcal{P}}{\gamma_{a}(v_{a}+\beta_{p}^{\prime})}(\beta_{p}^{\prime}x^{\prime}_{0}-x^{\prime}_{3})-\frac{(p_{3}-\mathcal{P})}{v_{a}\gamma_{a}}x^{\prime}_{0}.$
(21)

Recall that the second terms in Eqs. (20) and (21) determine the space–time
dependence of the envelope. Thus, the subluminal and superluminal solutions
represent Lorentz transformations of solutions with time-independent and
longitudinal-coordinate-independent envelopes, respectively.

The phase [Eq. (17)] and momentum condition [Eq. (18)] appearing in Eq. (13)
depend directly on $\mathcal{P}$. Thus, a convenient choice for the auxiliary
parameter is $q(\kappa)=\mathcal{P}(\kappa)$, or more explicitly
$\mathcal{P}(\kappa)=\gamma_{a}^{2}v_{a}\kappa\pm(\gamma_{a}^{4}\kappa^{2}-\gamma_{a}^{2}m^{2})^{1/2}$
[see Eq. (16)]. Using this choice and applying the delta functions in Eq. (13)
provides

$\displaystyle\Phi(x,\mathcal{P})$ $\displaystyle=e^{-i\mathcal{P}\eta}\int
d\bm{p}_{\perp}\tilde{\mathcal{N}}(\bm{p}_{\perp},\mathcal{P})$ (22)
$\displaystyle\begin{pmatrix}\chi_{+}\\\
\dfrac{\sigma^{3}p_{c}+\bm{\sigma}_{\perp}\cdot\bm{p}_{\perp}}{v_{a}(p_{c}-\mathcal{P})+\mathcal{E}+m}\chi_{+}\\\
\end{pmatrix}e^{-i(p_{c}-\mathcal{P})\xi+i\bm{p}_{\perp}\cdot\bm{x}_{\perp}},$

where $\eta=\beta_{p}x_{0}-x_{3}$, $\xi=v_{a}x_{0}-x_{3}$,

$\tilde{\mathcal{N}}(\bm{p}_{\perp},\mathcal{P})=\sqrt{\tfrac{\scriptstyle
v_{a}(p_{c}-\mathcal{P})+\mathcal{E}+m}{2[\scriptstyle
v_{a}(p_{c}-\mathcal{P})+\mathcal{E}]}}\int\mathcal{N}(p,\mathcal{P})\delta(p_{3}-p_{c})dp_{3},$

and the $\mathcal{P}$ dependence of $\beta_{p}$, $\mathcal{E}$, and $p_{c}$
are understood. As before, the integral (envelope) depends solely on the
space–time coordinates in the combination $\xi=v_{a}x_{0}-x_{3}$ and advects
at $v_{a}$.

Approximate analytical solutions for $\Phi(x,\mathcal{P})$, and ultimately
$\psi(x)$, can be found for a particle moving predominantly in the
$\bm{e}_{3}$ direction with a velocity $\langle v_{3}\rangle$ that is
sufficiently distinct from the group velocity $v_{a}$. More specifically, one
assumes $|v_{a}\beta_{p}-1|\mathcal{P}^{2}\gg w^{2}$, where $w$ is the
transverse momentum spread. This condition describes a “paraxial” limit in
which Eq. (18) simplifies to

$p_{c}\approx\mathcal{P}+\frac{\bm{p}_{\perp}^{2}}{2\mathcal{P}(v_{a}\beta_{p}-1)}.$
(23)

With the quadratic dependence of $p_{c}$ on $\bm{p}_{\perp}$, a natural choice
of basis functions for $\tilde{\mathcal{N}}(\bm{p}_{\perp},\mathcal{P})$ are
the Laguerre–Gaussian modes:

$\tilde{\mathcal{N}}(\bm{p}_{\perp},\mathcal{P})=\sum_{n,\ell}\mathcal{A}_{n\ell}(\mathcal{P})\rho^{|\ell|}L_{n}^{|\ell|}(\rho^{2})\mathrm{exp}(-\tfrac{1}{2}\rho^{2})e^{i\ell\theta_{p}},$
(24)

where $\rho=w^{-1}(\bm{p}_{\perp}^{2})^{1/2}$, $L_{n}^{|\ell|}$ is a
generalized Laguerre polynomial, and $\theta_{p}=\mathrm{atan}(p_{2}/p_{1})$
is the azimuth in momentum space. Upon substituting Eqs. (23) and (24) into
Eq. (22), dropping terms higher order than $\bm{p}_{\perp}^{2}$ in the spinor,
and integrating, one finds

$\displaystyle\Phi(x,\mathcal{P})$ $\displaystyle\approx
e^{-i\mathcal{P}\eta}\sum_{n,\ell}\tilde{\mathcal{A}}_{n\ell}(\mathcal{P})\left(\begin{smallmatrix}U_{1}(x)\\\
U_{2}(x)\\\ U_{3}(x)\\\ U_{4}(x)\end{smallmatrix}\right),$ (25)

where $\tilde{\mathcal{A}}_{n\ell}(\mathcal{P})=2\pi
i^{\ell}(-1)^{n}\mathcal{A}_{n\ell}(\mathcal{P})$. Each component of the
spinor can be expressed in terms of $U_{1}$ as follows:

$\displaystyle U_{1}(x)$
$\displaystyle=\tfrac{w}{R}\left(\tfrac{r}{R}\right)^{|\ell|}L_{n}^{|\ell|}(\tfrac{r^{2}}{R^{2}})\mathrm{exp}\big{[}-(1-i\tfrac{\xi}{\xi_{0}})\tfrac{r^{2}}{2R^{2}}+i\Lambda(x)\big{]}$
(26) $\displaystyle U_{2}(x)$ $\displaystyle=0$ $\displaystyle U_{3}(x)$
$\displaystyle=\tfrac{\mathcal{P}}{\mathcal{E}+m}\left[1-\tfrac{(\mathcal{E}-v_{a}\mathcal{P}+m)}{2w^{2}\xi_{0}(\mathcal{E}+m)\mathcal{P}}\nabla_{\perp}^{2}\right]U_{1}(x)$
$\displaystyle U_{4}(x)$
$\displaystyle=-\tfrac{i}{(\mathcal{E}+m)}r^{|\ell|}e^{i\theta}\partial_{r}r^{-|\ell|}U_{1}(x),$

where $r=(\bm{x}_{\perp}^{2})^{1/2}$, $\theta=\mathrm{atan}(x_{2}/x_{1})$ is
the azimuth in configuration space, $R=w^{-1}[1+(\xi/\xi_{0})^{2}]^{1/2}$,
${\Lambda(x)=\ell\theta-(2n+|\ell|+1)\mathrm{atan}(\tfrac{\xi}{\xi_{0}})}$,
and

$\xi_{0}=\frac{\mathcal{P}(v_{a}\beta_{p}-1)}{w^{2}}.$ (27)

Equations (25) – (27) show that the envelope advects at the group velocity
$v_{a}$ while maintaining a constant profile characterized by the duration
$\xi_{0}/v_{a}$.

The $\mathcal{A}_{n\ell}(\mathcal{P})$ quantify the distribution of
$\mathcal{P}$ values for every $n$ and $\ell$ mode that compose the
wavefunction. Said differently, the wavefunction is comprised of solutions
that fall along a curve defined by Eq. (23) and parameterized by $\mathcal{P}$
[Fig. 2(b)]: $\psi(x)=(2\pi)^{-3/2}\int\Phi(x,\mathcal{P})d\mathcal{P}$. If
the $\mathcal{A}_{n\ell}(\mathcal{P})$ are narrowly peaked about some
$\mathcal{P}=\bar{\mathcal{P}}$, an approximate expression for this integral
and the wavefunction can be obtained. Expressing
$\Phi(x,\mathcal{P})=\sum_{n,\ell}\Phi_{n\ell}(x,\mathcal{P})$ yields

$\displaystyle\psi(x)\approx\sum_{n,\ell}\frac{\Phi_{n\ell}(x,\bar{\mathcal{P}})}{\tilde{\mathcal{A}}_{n\ell}(\bar{\mathcal{P}})}\int\tilde{\mathcal{A}}_{n\ell}(\bar{\mathcal{P}}+\Delta\mathcal{P})e^{-i\zeta\Delta\mathcal{P}}d\Delta\mathcal{P},$
(28)

where $\Delta\mathcal{P}=\mathcal{P}-\bar{\mathcal{P}}$,
$\zeta=v_{n}x_{0}-x_{3}$, and
$v_{n}=\bar{\mathcal{P}}/(m^{2}+\bar{\mathcal{P}}^{2})^{1/2}$. Figure 1
displays the components of a wavefunction with a superluminal velocity
$v_{a}=2$ for $n=\ell=0$ and
$\int\tilde{\mathcal{A}}_{00}(\bar{\mathcal{P}}+\Delta\mathcal{P})e^{-i\zeta\Delta\mathcal{P}}d\Delta\mathcal{P}\propto\mathrm{exp}[-(\zeta/\Delta\zeta)^{8}]$.

Solutions to the Dirac equation can feature a peak probability density that
moves at any velocity, including those exceeding the speed of light, while
maintaining a near-constant profile. This motion is independent of the
velocity expectation value. The solutions are on-shell and constructed by
superposing basis functions with correlations in the longitudinal and
transverse momenta. Future work will investigate electromagnetic structures
that can produce these wavefunctions through the Kapitza–Dirac effect; explore
the benefit of these wavefunctions in phenomena, such as Compton scattering
and Smith–Purcell and Cherenkov radiation; and pursue additional structures by
replacing Eq. (8) with other dependencies, like
$E_{\bm{p}}=p_{1}^{2}/\mathcal{M}+\kappa$, which could exhibit classical
dispersion with a modified effective mass $\mathcal{M}$ in vacuum.

###### Acknowledgements.

The authors would like to thank R. R. Almeida and D. H. Froula for insightful
discussions. The work of M.F. is supported by the European Union’s Horizon
Europe research and innovation program under the Marie Skłodowska-Curie Grant
Agreement No. 101105246-STEFF. The work of J.P.P., D.R., and A.D. is supported
by the Office of Fusion Energy Sciences under Award Numbers DE-SC0021057, the
Department of Energy National Nuclear Security Administration under Award
Number DE-NA0004144, the University of Rochester, and the New York State
Energy Research and Development Authority. This report was prepared as an
account of work sponsored by an agency of the US Government. Neither the US
Government nor any agency thereof, nor any of their employees, makes any
warranty, express or implied, or assumes any legal liability or responsibility
for the accuracy, completeness, or usefulness of any information, apparatus,
product, or process disclosed, or represents that its use would not infringe
privately owned rights. Reference herein to any specific commercial product,
process, or service by trade name, trademark, manufacturer, or otherwise does
not necessarily constitute or imply its endorsement, recommendation, or
favoring by the US Government or any agency thereof. The views and opinions of
authors expressed herein do not necessarily state or reflect those of the US
Government or any agency thereof.

## References

  * Dirac (1928) P. Dirac, Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character 117, 610–624 (1928).
  * Schrödinger (1926) E. Schrödinger, Phys. Rev. 28, 1049 (1926).
  * Peskin and Schroeder (1995) M. Peskin and D. Schroeder,  _An Introduction To Quantum Field Theory_, Frontiers in Physics (Avalon Publishing, 1995).
  * Berry and Balazs (1979) M. V. Berry and N. L. Balazs, American Journal of Physics 47, 264 (1979).
  * Allen  _et al._ (1992) L. Allen, M. W. Beijersbergen, R. J. C. Spreeuw, and J. P. Woerdman, Phys. Rev. A 45, 8185 (1992).
  * Berry (1998) M. V. Berry, in  _International Conference on Singular Optics_, Vol. 3487, edited by M. S. Soskin, International Society for Optics and Photonics (SPIE, 1998) pp. 6 – 11.
  * Bliokh  _et al._ (2007) K. Y. Bliokh, Y. P. Bliokh, S. Savel’ev, and F. Nori, Phys. Rev. Lett. 99, 190404 (2007).
  * Verbeeck  _et al._ (2010) J. Verbeeck, H. Tian, and P. Schattschneider, Nature 467, 301 (2010).
  * Kaminer  _et al._ (2012) I. Kaminer, R. Bekenstein, J. Nemirovsky, and M. Segev, Phys. Rev. Lett. 108, 163901 (2012).
  * Voloch-Bloch  _et al._ (2013) N. Voloch-Bloch, Y. Lereah, Y. Lilach, A. Gover, and A. Arie, Nature 494, 331 (2013).
  * Grillo  _et al._ (2014) V. Grillo, E. Karimi, G. C. Gazzadi, S. Frabboni, M. R. Dennis, and R. W. Boyd, Phys. Rev. X 4, 011013 (2014).
  * Kaminer  _et al._ (2015) I. Kaminer, J. Nemirovsky, M. Rechtsman, R. Bekenstein, and M. Segev, Nature Physics 11, 261 (2015).
  * Lloyd  _et al._ (2017) S. M. Lloyd, M. Babiker, G. Thirunavukkarasu, and J. Yuan, Rev. Mod. Phys. 89, 035004 (2017).
  * Hall and Abouraddy (2023) L. A. Hall and A. F. Abouraddy, Nature Physics 19, 435 (2023).
  * Campos  _et al._ (2024) A. G. Campos, K. Z. Hatsagortsyan, and C. H. Keitel, Phys. Rev. Res. 6, 023040 (2024).
  * Chirita Mihaila  _et al._ (2022) M. C. Chirita Mihaila, P. Weber, M. Schneller, L. Grandits, S. Nimmrichter, and T. Juffmann, Phys. Rev. X 12, 031043 (2022).
  * Wong  _et al._ (2024) L. W. W. Wong, X. Shi, A. Karnieli, J. Lim, S. Kumar, S. Carbajo, I. Kaminer, and L. J. Wong, Light: Science & Applications 13, 29 (2024).
  * Longhi (2003) S. Longhi, Phys. Rev. E 68, 066612 (2003).
  * Kondakci and Abouraddy (2017) H. E. Kondakci and A. F. Abouraddy, Nature Photonics 11, 733 (2017).
  * Yessenov  _et al._ (2022) M. Yessenov, J. Free, Z. Chen, E. G. Johnson, M. P. J. Lavery, M. A. Alonso, and A. F. Abouraddy, Nature Communications 13, 4573 (2022).
  * Sainte-Marie  _et al._ (2017) A. Sainte-Marie, O. Gobert, and F. Quéré, Optica 4, 1298 (2017).
  * Froula  _et al._ (2018) D. H. Froula, D. Turnbull, A. S. Davies, T. J. Kessler, D. Haberberger, J. P. Palastro, S.-W. Bahk, I. A. Begishev, R. Boni, S. Bucht, J. Katz, and J. L. Shaw, Nature Photonics 12, 262 (2018).
  * Palastro  _et al._ (2018) J. P. Palastro, D. Turnbull, S.-W. Bahk, R. K. Follett, J. L. Shaw, D. Haberberger, J. Bromage, and D. H. Froula, Phys. Rev. A 97, 033835 (2018).
  * Di Piazza (2021) A. Di Piazza, Phys. Rev. A 103, 012215 (2021).
  * Formanek  _et al._ (2023) M. Formanek, J. P. Palastro, M. Vranic, D. Ramsey, and A. Di Piazza, Phys. Rev. E 107, 055213 (2023).
  * Ramsey  _et al._ (2023) D. Ramsey, A. D. Piazza, M. Formanek, P. Franke, D. H. Froula, B. Malaca, W. B. Mori, J. R. Pierce, T. T. Simpson, J. Vieira, M. Vranic, K. Weichman, and J. P. Palastro, Physical Review A 107, 013513 (2023).
  * Palastro  _et al._ (2024) J. P. Palastro, K. G. Miller, R. K. Follett, D. Ramsey, K. Weichman, A. V. Arefiev, and D. H. Froula, Phys. Rev. Lett. 132, 095101 (2024).
  * Kapitza and Dirac (1933) P. L. Kapitza and P. A. M. Dirac, Mathematical Proceedings of the Cambridge Philosophical Society 29, 297–300 (1933).
  * Freimund and Batelaan (2002) D. L. Freimund and H. Batelaan, Phys. Rev. Lett. 89, 283602 (2002).
  * Shiloh  _et al._ (2014) R. Shiloh, Y. Lereah, Y. Lilach, and A. Arie, Ultramicroscopy 144, 26 (2014).
  * Vanacore  _et al._ (2018) G. M. Vanacore, I. Madan, G. Berruto, K. Wang, E. Pomarico, R. J. Lamb, D. McGrouther, I. Kaminer, B. Barwick, F. J. García de Abajo, and F. Carbone, Nature Communications 9, 2694 (2018).
  * Feist  _et al._ (2020) A. Feist, S. V. Yalunin, S. Schäfer, and C. Ropers, Phys. Rev. Res. 2, 043227 (2020).
  * Reinhardt and Kaminer (2020) O. Reinhardt and I. Kaminer, ACS Photonics 7, 2859 (2020).
  * García de Abajo and Konečná (2021) F. J. García de Abajo and A. Konečná, Phys. Rev. Lett. 126, 123901 (2021).
  * Lin  _et al._ (2024) K. Lin, S. Eckart, H. Liang, A. Hartung, S. Jacob, Q. Ji, L. P. H. Schmidt, M. S. Schöffler, T. Jahnke, M. Kunitski, and R. Dörner, Science 383, 1467 (2024).
  * Pupasov-Maksimov and Karlovets (2021) A. Pupasov-Maksimov and D. Karlovets, New Journal of Physics 23, 043011 (2021).