[1]Angela Giraldi 11footnotetext: For the CMS Collaboration.

# Precision luminosity measurement with proton-proton collisions at the CMS
experiment in Run 2

###### Abstract

A precision luminosity measurement is essential for LHC cross-section
measurements to determine fundamental parameters of the standard model and
constrain or discover beyond-the-standard-model phenomena. The luminosity of
the CMS detector has been measured at the LHC Interaction Point 5 using
proton-proton collisions at $\sqrt{s}=13$ TeV during the Run 2 data-taking
period (2015-2018). The absolute luminosity scale is obtained using beam-
separation scans and the Van der Meer (VdM) method, and several systematic
uncertainty sources are investigated, from the knowledge of the scale of beam
separation provided by LHC magnets to the nonfactorizability of the spatial
components of proton bunch density distributions in the transverse direction.
When the VdM calibration is applied to the entire data-taking period, the
detector linearity and stability measurements contribute significantly to the
total uncertainty in the integrated luminosity. In 2016-2018, the reported
integrated luminosity was among the most precise measurements at bunched-beam
hadron colliders.

##  1 Introduction

The luminosity $\mathcal{L}$ is a fundamental property of any collider
experiment as it is a measure of the collision rate. It is the proportionality
factor that relates the event rate $dN/dt$ of a process to its visible cross-
section $\sigma$:

$\frac{dN}{dt}=\mathcal{L}\cdot\sigma.$ (1)

Precision cross-section measurements require an accurate understanding of the
luminosity integrated over time, mainly because the uncertainty in the
integrated luminosity is an important uncertainty in most precise
measurements.

The CMS experiment [1] employs a two-step strategy to measure the integrated
luminosity of its collision data sets in Run 2 (2015-2018).

First various luminosity detectors are calibrated in a dedicated special run.
The calibration constant called visible cross section $\sigma_{\mathrm{vis}}$
relates the measured rate of the luminometer to the luminosity at the time of
the measurement. Second, the rate measurement is integrated over the data
collection period and normalized with $\sigma_{\mathrm{vis}}$ to yield the
integrated luminosity.

The methodology and dominant sources of systematic uncertainty are discussed
in these proceedings, based on the latest luminosity measurements from proton-
proton (pp) collision datasets collected at $\sqrt{s}=13$ TeV [2, 3, 4].

##  2 Luminosity detectors

During the LHC Run 2 period, the CMS experiment employed several sub-detectors
to monitor and measure the luminosity (luminometers). There are two dedicated
luminosity measurement systems: the Pixel Luminosity Telescope (PLT) [5],
consisting of eight three-layer telescopes located 1.8 m from the interaction
point in both directions, and the Fast Beam Conditions Monitor (BCM1F), made
of silicon and diamond sensors with a time resolution of 6.25 ns mounted on
the same carriages as PLT. Like the luminosity system installed as part of the
Hadron Forward Calorimeter (HF), they use a separate data acquisition (DAQ)
system that runs independently of the primary CMS readout. HF uses two
algorithms, one based on the transverse energy sum (HFET) and the other on the
fraction of occupied towers (HFOC). The results presented here are based
primarily on offline measurements made with the forward hadron calorimeter and
the pixel cluster counting (PCC) method, using silicon pixels to determine the
offline luminosity. In addition, other methods are used to perform luminosity
measurements using the main CMS DAQ system. First, the inner tracker
information is employed in the vertex counting (VTX) method, with strip
tracker information used for cross checks in low-pileup conditions. Then, the
rate of the muon track stubs in the muon barrel track finder is measured from
the barrel drift tubes (DT), and similarly, the rate of photons in ionization
chambers from the CERN radiation and environmental monitoring system (RAMSES)
is measured. DT and RAMSES are not sensitive to bunch-by-bunch measurements,
and their rates are cross-calibrated in special scans, resulting in stable
luminosity measurements during data collection.

Each luminometer reads out a rate $\mathcal{R}(t)$ of specific observables
(hits, tracks, clusters, etc.), proportional to the instantaneous luminosity
$\mathcal{L}(t)$ via Eq. 1 through $\sigma_{\mathrm{vis}}$. The luminosity
detectors are calibrated once per data-taking period in a special LHC fill by
calculating $\sigma_{\mathrm{vis}}$ as:

$\sigma_{\mathrm{vis}}=\frac{2\pi\Sigma_{x}\Sigma_{y}}{N_{1}N_{2}\nu_{\mathrm{LHC}}}\cdot\mathcal{R}_{0}\
,$ (2)

assuming that the bunch proton density function is factorizable into
independent $x$ and $y$ terms; with $\Sigma_{x}$ and $\Sigma_{y}$ being the
beam overlap width and height of the transverse luminous region where the
collisions occur, $N_{1}$ and $N_{2}$ the numbers of protons in the two
colliding bunches, $\nu_{\mathrm{LHC}}$ the revolution frequency, and
$\mathcal{R}_{0}$ the measured rate for head-on beam collisions. The
calibration constant $\sigma_{\mathrm{vis}}$ is determined from Van der Meer
(VdM) beam separation scans [6], where the two proton beams are separated in
the transverse plane and moved across each other in steps. $\Sigma_{x}$
($\Sigma_{y}$) is obtained as the width of the fitted double-Gaussian function
to the measured rate of the VdM scan data as a function of beam separation in
the horizontal (vertical) plane. An exemplary fit result is shown in Fig. 1
(left).

Figure 1: Left: Example of a VdM scan illustrating the normalized rate as a
function of beam separation in the x direction and the fitted curves. The
difference between the measured and fitted values is shown in the lower panel,
divided by the statistical uncertainty [2]. Right: The beam-separation
residuals in y [2].

##  3 Systematic uncertainties

The value of $\sigma_{\mathrm{vis}}$ derived from the VdM scan analysis is
affected by several systematic effects known as calibration uncertainties. In
contrast, integration uncertainties result from detector operations throughout
the year and from transferring the calibration of the luminometers measured in
the special VdM fill to the conditions in normal physics data taking fills.

###  3.1 Calibration uncertainties

The beam separation in the fit is derived from the current settings of the LHC
steering magnets (nominal separation) and is susceptible to several effects.
Via length scale scans (LSC), the absolute beam separation is calibrated by
comparing nominal beam positions to the measured one of reconstructed
interaction vertices with the CMS tracker. Using LHC beam position monitoring
(BPM) systems, a time-dependent movement of the proton beams away from their
standard orbit (orbit drift) is observed and accounted for both in the VdM
overlap fit and in the LSC procedure.

The electromagnetic interactions between the beams manifest themselves through
two beam-beam effects. First, due to electric repulsion, the transverse
distance between the beams increases and the magnitude of this deflection
relies on the nominal beam separation [7, 8]. Then, due to focusing and
defocusing (dynamic $\beta^{*}$ effect), the transverse shape of the proton
bunches varies, resulting in a separation-dependent rise in the collision
rate.

The proton numbers of the beams are accounted for in the VdM fit by
normalizing the measured rate with the beam currents as measured with LHC
devices and corrected for contributions from spurious charges in nominally
empty bunch slots (ghost charges) and non-colliding buckets of the colliding
bunches (satellite charges).

Several VdM scan pairs are conducted during a calibration. The bunch-to-bunch
and scan-to-scan consistency, also using different luminometers, is examined
to evaluate the systematic uncertainty of cross-detector calibration. After
applying all the corrections mentioned above, a residual difference is
consistently observed, see Fig. 1 (right); corrections are taken into account.

The assumption that the bunch proton density function is factorizable into
independent horizontal and vertical components, and hence Eq. (2) holds, is
tested to estimate the bias in the measurement of the beam overlap integral.
Different techniques are used to measure this effect, based on beam imaging
and offset scans [2, 9, 10].

###  3.2 Integration uncertainties

After determining the calibrated $\sigma_{\mathrm{vis}}$, a measurement of the
integrated luminosity can be assessed in the standard physics fills for all
data-taking periods.

Most detector measurements involve false signals from out-of-time pileup
contributions caused by electronic spillover or activation of the surrounding
detector material after collisions occurred. It does not affect the
calibration but results in a nonlinear detector response under normal data-
taking conditions. The observed rate from nominally empty bunches is used to
calculate these afterglow corrections.

Uncertainties in the integration procedure mainly arise from extrapolating the
calibration from the special to regular data-taking conditions (linearity), as
well as from changes in the detector over time (stability). Different
operational difficulties and radiation damage can impact individual
luminometers; therefore, their stability and linearity must be monitored and,
if feasible, independently corrected. This is accomplished by performing short
VdM-like beam separation scans in typical physics conditions at the beginning
and end of fills, so-called emittance scans [11]. An example of changing
detector conditions over time is the ageing of PMTs and fibers in the forward
hadron calorimeter, which causes instabilities in the HFET and HFOC response.
The efficiency of the HFET response as evaluated by emittance scans is shown
in Fig. 2 for 2017 and 2018, with a slope indicating a significant decrease in
efficiency with time.

After applying all corrections, the overall consistency of the individual
luminometers values to each other is evaluated, and the residual cross-
detector stability and linearity uncertainty is assigned. In addition, the
uncertainty associated with the deadtime of the CMS data acquisition system is
also included.

Figure 2: Relative efficiency of the HFET detector response as determined from emittance scans as a function of integrated luminosity for all LHC fills in 2017 and 2018 pp data. The slope indicates the instability [11]. Source | 2015 | 2016 | 2017 | 2018
---|---|---|---|---
Calibration uncertainty
Ghost and satellite charge | [%] | 0.1 | 0.1 | 0.1 | 0.1
Beam current normalization | [%] | 0.2 | 0.2 | 0.3 | 0.2
Orbit drift | [%] | 0.2 | 0.2 | 0.2 | 0.2
Residual differences | [%] | 0.8 | 0.5 | 0.2 | 0.1
Beam-beam effects | [%] | 0.5 | 0.5 | 0.6 | 0.2
Length scale calibration | [%] | 0.2 | 0.3 | 0.3 | 0.2
Transverse factorizability | [%] | 0.5 | 0.5 | 0.8 | 2.0
Integration uncertainty
Out-of-time pileup Type 1 corrections | [%] | 0.3 | 0.3 | 0.2 | 0.1
Out-of-time pileup Type 2 corrections | [%] | 0.1 | 0.3 | 0.3 | 0.4
Cross-detector stability | [%] | 0.6 | 0.5 | 0.5 | 0.6
Linearity | [%] | 0.5 | 0.3 | 1.5 | 1.5
CMS deadtime | [%] | 0.5 |  $<$0.1 | 0.5 |  $<$0.1
Total uncertainty | [%] | 1.6 | 1.2 | 2.3 | 2.5
Integrated luminosity | [fb-1] | 2.27 | 36.3 | 41.5 | 59.8
Table 1:  Summary of main contributions (in %) to the systematic uncertainty
in $\sigma_{\mathrm{vis}}$ at $\sqrt{s}$=13 TeV [2, 3, 4].

##  4 Summary

The integrated luminosity measurement of the CMS experiment is calibrated with
the Van der Meer (VdM) approach, considering several systematic uncertainties
sources, and corrected for nonlinearity and instability of the detector
response. Table 1 summarizes the sources of systematic uncertainty in the
luminosity calibration for the pp collision data sets at $\sqrt{s}$ = 13 TeV
due to the VdM scan procedure (calibration) and the detector operations
(integration).

## References

  * [1] CMS Collaboration, _The CMS Experiment at the CERN LHC_ , _JINST_ 3 (2008) S08004.
  * [2] CMS Collaboration, _Precision luminosity measurement in proton-proton collisions at $\sqrt{s}=$ 13 TeV in 2015 and 2016 at CMS_, _Eur. Phys. J. C_ 81 (2021) 800 [2104.01927].
  * [3] CMS Collaboration, _CMS luminosity measurement for the 2017 data-taking period at $\sqrt{s}=13~{}\mathrm{TeV}$_, CMS Physics Analysis Summary CMS-PAS-LUM-17-004, CERN, Geneva (2018).
  * [4] CMS Collaboration, _CMS luminosity measurement for the 2018 data-taking period at $\sqrt{s}=13~{}\mathrm{TeV}$_, CMS Physics Analysis Summary CMS-PAS-LUM-18-002, CERN, Geneva (2019).
  * [5] CMS BRIL collaboration, _The Pixel Luminosity Telescope: A detector for luminosity measurement at CMS using silicon pixel sensors_ , 2206.08870.
  * [6] S. van der Meer, _Calibration of the effective beam height in the ISR_ , Tech. Rep. CERN-ISR-PO-68-31, CERN, Geneva (1968).
  * [7] V. Balagura, _Van der Meer scan luminosity measurement and beam–beam correction_ , _Eur. Phys. J. C_ 81 (2021) 26 [2012.07752].
  * [8] W. Kozanecki, T. Pieloni and J. Wenninger, _Observation of Beam-beam Deflections with LHC Orbit Data_ , CERN Internal Note [CDS:1581723], CERN, Geneva (Aug, 2013).
  * [9] M. Klute, C. Medlock and J. Salfeld-Nebgen, _Beam Imaging and Luminosity Calibration_ , _JINST_ 12 (2017) P03018 [1603.03566].
  * [10] J. Knolle (CMS Collaboration), _The factorization bias in the Van der Meer method: Run 2 experiences at the CMS experiment_ , in proceeding of LHC Lumi Days 2019, 1910.14066 (2019).
  * [11] CMS Collaboration, _Long-term monitoring of delivered luminosity and calibration stability in CMS_ , CMS Detector Performance Note CMS-DP-2019-016, CERN, Geneva (Jun, 2019).


# Interactive Editing for Text Summarization

Yujia Xie Xun Wang Si-Qing Chen Wayne Xiong Pengcheng He
Microsoft
{yujiaxie, xunwang, sqchen, Wayne.Xiong<EMAIL_ADDRESS>
###### Abstract

Summarizing lengthy documents is a common and essential task in our daily
lives. Although recent advancements in neural summarization models can assist
in crafting general-purpose summaries, human writers often have specific
requirements that call for a more customized approach. To address this need,
we introduce REVISE, an innovative framework designed to facilitate iterative
editing and refinement of draft summaries by human writers. Within our
framework, writers can effortlessly modify unsatisfactory segments at any
location or length and provide optional starting phrases – our system will
generate coherent alternatives that seamlessly integrate with the existing
summary. At its core, REVISE incorporates a modified fill-in-the-middle model
with the encoder-decoder architecture while developing novel evaluation
metrics tailored for the summarization task. In essence, our framework
empowers users to create high-quality, personalized summaries by effectively
harnessing both human expertise and AI capabilities, ultimately transforming
the summarization process into a truly collaborative and adaptive experience.

##  1 Introduction

Human intelligence has been significantly augmented by the rapid development
of Artificial Intelligence (AI, Engelbart (1962); Lee et al. (2022)),
particularly with the emergence of large language models (Devlin et al., 2018;
Raffel et al., 2020; Brown et al., 2020; OpenAI, 2023). AIs have shown great
potential in various practical settings, helping users brainstorm ideas (e.g.,
Jasper, Copy.ai), paraphrase sentences (e.g.,Wordtune, QuillBot), reformulate
queries (Nogueira and Cho, 2017), autocomplete sentences (Chen et al., 2019),
and write code (e.g., Copilot, TabNine). One specific area where AI can
revolutionize our daily life is in the realm of document summarization, which
is the focus of this paper.

In this work, we present REVISE– Refinement and Editing Via Iterative
Summarization Enhancement, a novel framework that transforms the summarization
process into an interactive experience for human writers. Instead of
generating static summaries, our approach enables users to efficiently edit
and improve draft summaries iteratively, tailoring them to their specific
needs and preferences. This results in the creation of high-quality,
customized summaries that cater to individual requirements and contexts,
moving beyond the limitations of traditional, one-size-fits-all summarization
models. Figure 1 shows an illustration.

Figure 1:  Illustration on how REVISE interacts with human writer.

Our framework primarily consists of two models: one providing the initial
draft summary and the other supporting the writer in refining the summary
through interactive suggestions. We build upon a pretrained encoder-decoder
summarization model and enhance its ability to generate contextually relevant
and coherent suggestions for human edits. Through extensive experimentation
and evaluation, we demonstrate the superior performance of our proposed
framework in terms of salience, coherence, and adaptability to different
situations.

The key innovation in REVISE lies in fostering a seamless collaboration
between human writers and AI models, creating a truly interactive
summarization experience. This interactive approach not only enables users to
harness the power of AI to extract key information but also preserves the
creativity and adaptability offered by human input. By empowering users to
edit the summary iteratively until they are satisfied, our framework ensures
the delivery of personalized, high-quality summaries tailored to diverse
requirements.

We perform extensive human evaluations, suggesting the proposed framework can
not only improve the efficiency of human editing, but also significantly
enhance the summary quality.

##  2 Related Works

Interactive summarization. There are several works exploring the how to
facilitate human summary writing in an iterative way. For example, Yan et al.
(2011) generate new summaries after their users to click on sentences they
want to know more about. In Avinesh and Meyer (2017) and Avinesh et al.
(2018), users can indicate which bigrams of a candidate summary are relevant
to their interests. The APRIL system (Gao et al., 2020) first let users to
indicate preference between candidate summaries, and then train a summary-
ranking model to select the next pair of candidate summaries. Bohn and Ling
(2021) optimize document summaries for personal interest by collecting user
feedback in the normal flow of reading, such as dwell time or gaze location.
More recently, Shapira et al. (2021, 2022) allow users to interactively submit
queries in order to expand on the information on a topic in the summary. In
contrast, our framework is more versatile – users can either specify their
intent by the prompts, or let the model provide a few alternatives.

Text generation – interactive editing. The task of human editing text
interactively with AI is widely explored in other text generation tasks (Cheng
et al., 2022; Lee et al., 2022), e.g., machine translation (Barrachina et al.,
2009). Many works are prefix-based, i.e., new completions can only be
generated left-to-right (González-Rubio et al., 2013; Peris and Casacuberta,
2018, 2019a, 2019b). Few works (González-Rubio et al., 2016; Weng et al.,
2019) allow edits at arbitrary positions. However, unlike REVISE, the edits
can only be words or sentences.

Text infilling. There are two approaches for imbuing models with infilling
capabilities: first, through new architectures like SpanBERT (Joshi et al.,
2020) and XLNet (Yang et al., 2019). To list a few examples, XLNet modifies
the attention mask in a standard transformer to enable token generation in any
user-specified order, while Insertion Transformer (Stern et al., 2019), KERMIT
(Chan et al., 2019), and InDIGO (Gu et al., 2019) allow the model to predict a
location for the next token before predicting the token. Similarly, Blank
Language models (Shen et al., 2020) generate text by iteratively selecting a
blank and replacing it with a token (and optionally more blanks).

Alternatively, Zhu et al. (2019), Donahue et al. (2020), GLM (Du et al.,
2022), CM3 (Aghajanyan et al., 2022), InCoder (Fried et al., 2022), and
Bavarian et al. (2022) utilize left-to-right autoregressive modeling by moving
the infill regions to the end of context, with regions separated by sentinels.
Notably, Bavarian et al. (2022) show the computational efficiency and superior
performance of training in this way at scale. Our work extends Bavarian et al.
(2022) to encoder-decoder models, and demonstrates the feasibility of its
usage in summarization.

Text infilling can also be performed using a GAN-based method (Fedus et al.,
2018), where REINFORCE is needed, or through gradient search (Liu et al.,
2019).

##  3 Method

Figure 2:  Illustration on how the FIM model is trained.

The key component of our framework is a Fill-In-the-Middle (FIM) model, which
can provide alternatives to the human specified unsatisfactory part.
Specifically, the input of the model is a source document sequence, a prefix
sequence containing the summary before the deleted part and an optionally
human start, a suffix sequence containing the summary after the deleted part.
The goal of the model is to fill in a sequence that not only contains the key
information of document, but also connect coherently with the prefix and the
suffix. Following the advanced neural models for abstractive summarization
(Zhang et al., 2020; He et al., 2022), we adopt an encoder-decoder model
architecture.

###  3.1 Training for FIM

We start from a standard summarization training data
$\mathcal{D}=\\{d_{i},t_{i}\\}_{i=1}^{N}$, where $d_{i}$ is the $i$-th
document, $t_{i}$ is the corresponding summary, and they are both sequences of
tokens. During training, we randomly divide the summary $t_{i}$ into three
parts: the prefix $p_{i}$, the middle $m_{i}$, and the suffix $s_{i}$. Then,
we concatenate the prefix, the suffix, and the document, together with their
sentinel tokens, as the input for the encoder,

$\texttt{[PRE]}\circ p_{i}\circ\texttt{[SUF]}\circ
s_{i}\circ\texttt{[CLS]}\circ d_{i},$

where $\circ$ is the concatenation of tokens. We input the middle together
with its sentinel tokens into the decoder,

$\texttt{[BOS]}\circ m_{i}\circ\texttt{[EOS]}.$

Here, we follow Bavarian et al. (2022) and adopt an [EOS] token to signal a
successful concatenation of the middle and the suffix. The model is then
trained with a standard cross-entropy loss for sequence-to-sequence models.

###  3.2 Training for the Corner Cases

Human edits can appear anywhere – not only the middle, but also in the
beginning and the end. In our preliminary experiments, we find that if the
model is only trained for the data in the middle, it cannot handle the edits
in the beginning and the end of the summaries. Especially, if the edits are in
the end, the generation usually cannot reach [EOS] token, i.e., the generation
cannot end.

Therefore, we sample a proportion $\gamma$ of data for the edits in the
beginning and the end. Specifically, we randomly split the summary $t_{i}$
into the prefix $p_{i}$ and the suffix $s_{i}$. On the one hand, to train the
generation in the end, we input the concatenation of the prefix and the
document into the encoder,

$\texttt{[PRE]}\circ p_{i}\circ\texttt{[CLS]}\circ d_{i},$

and the suffix for the decoder,

$\texttt{[BOS]}\circ s_{i}\circ\texttt{[EOS]}.$

On the other hand, we exchange the position of $p_{i}$ and $s_{i}$ to train
the generation in the beginning.

##  4 Experiments

Model R-FIM R-begin R-end $\ell_{1}$ $\ell_{2}$ R-all Proposed 51.79/30.30
49.51/28.87 37.36/20.31 -3.14 -2.97 43.32/20.96 Context in decoder 45.72/25.62
44.25/24.72 36.32/19.41 -3.14 -3.84 43.37/20.94 No corner case training
51.86/30.24 46.67/27.11 26.97/12.67 -4.01 -3.04 10.67/2.42 Base model
47.79/27.20 46.01/25.98 34.46/18.13 -4.32 -3.19 41.92/19.36

Table 1:  Different variants of the FIM model. Here, for R-FIM, R-begin,
R-end, and R-all, the first number is ROUGE-1, the second number is ROUGE-2.

###  4.1 Settings

We use the CNN / Daily Mail (CNNDM) dataset (See et al., 2017) for training
and evaluation. We adopt the pretrained Z-Code++ (He et al., 2022) as the
backbone encoder-decoder model. The proportion $\gamma$ of the training for
the corner cases is $0.5$. We train the model for $10$ epochs with learning
rate $7\times 10^{-6}$.

###  4.2 Evaluation Metrics

We evaluate the FIM model for three aspects:

  1. 1.

Is the generation salient in the document?

  2. 2.

Does the generation connect coherently with the rest of the summary?

  3. 3.

Can the model handle any possible positions?

We propose to use three evaluation metrics for these three aspects,
respectively.

ROUGE score. We split the test set summaries of CNNDM into prefixes, middles,
and suffixes. We feed the golden prefixes and suffixes to the model, and
compute the ROUGE scores (Lin, 2004) between the generated texts and the
golden middles. In this way, we measure whether the generated texts captures
the important information as the golden summaries do.

GPT Likelihood. Large pretrained language models can be used to evaluate the
coherence of text (Yuan et al., 2021). Here, we adopt GPT-3.5 (Brown et al.,
2020). Specifically, given a sequence of tokens
$x=(x^{(1)},\cdots,x^{(j)},x^{(j+1)},\cdots,x^{(N)})$, we evaluate whether
$(x^{(1)},\cdots,x^{(j)})$ are locally coherently connect with
$(x^{(j+1)},\cdots,x^{(N)})$ by computing the log-likelihood of generating
$x^{(j+1)},x^{(j+2)},\cdots$, given the previous tokens,

$\displaystyle\ell_{H}((x^{(1)},\cdots,x^{(j)}),(x^{(j+1)},\cdots,x^{(N)}))$
$\displaystyle=$ $\displaystyle\log
p(x^{(j+1)},\cdots,x^{(j+H)}|x^{(1)},\cdots,x^{(j)})$ $\displaystyle=$
$\displaystyle\sum_{h=1}^{H}\log
p_{\texttt{GPT}}(x^{(j+h)}|x^{(1)},\cdots,x^{(j+h-1)})$

where $p_{\texttt{GPT}}(a^{(k)}|a^{(1)},\cdots,a^{(k-1)})$ is the probability
that next token is $a^{(k)}$ given previous token sequence
$a^{(1)},\cdots,a^{(k-1)}$, and the probability is provided by GPT model.
Here, we adopt the likelihood on a fixed-length sequence
$x^{(j+1)},\cdots,x^{(j+H)}$ instead of the complete sequence
$x^{(j+1)},\cdots,x^{(N)}$ to alleviate the length effect.

For summarization, we consider the connectivity of the prefix-middle and the
middle-suffix,

$\displaystyle\ell_{1}$ $\displaystyle=\ell_{H}(p_{i},m_{i}),$
$\displaystyle\ell_{2}$ $\displaystyle=\ell_{H}(p_{i}\circ m_{i},s_{i}),$

where $p_{i},m_{i},s_{i}$ is from the split test set summary in the above
section.

ROUGE score for corner cases. Following the training recipe in Section 3.2, we
split the test set summaries into the prefix and the suffix, ask the model to
generate the prefix or the suffix accordingly, and compute the corresponding
ROUGE scores.

Model R-FIM R-begin R-end R-all Base model 47.79/27.20 46.01/25.98 34.46/18.13
41.92/19.36 \- DA 46.52/25.97 44.99/25.19 34.04/17.52 35.44/15.53 \- DA - RTD
46.22/25.80 45.15/25.22 33.40/17.02 39.90/18.21

Table 2:  Different pretraining setting.

###  4.3 Empirical Results

Different design choices for text infilling. We include three different
variants of the proposed model in Table 1. Our proposed training method can
achieve the best performance in terms of salience and coherence. In Table 2 we
also show how the design choices in the pretraining stage affect the
performance.

Comparing to standard summarization model. The last column of Table 1 shows
the results when we do not provide summary context for generation, i.e., we
ask the model to generate a complete summary. In comparison, in He et al.
(2022), a vanilla finetuned summarization model can achieve a ROUGE-2 of 22.2.
Our corner case training can significantly improve the generation quality.

###  4.4 Human Evaluation

Figure 3:  Illustration on the human editing process with interaction. The
annotators will be prompted with multiple suggestions of summary completions,
and they choose one from them.

To validate our proposed pipeline can effectively help human writer to make
edits, we conduct a contrast experiment on whether using REVIVE can improve
editing efficiency. Specifically, we collect 120 document in three
domains111We release the data together with the evaluation results at
https://github.com/microsoft/Interactive-Summarization. – news, conversations,
and blogs, and ask human annotators to edit the draft summaries until they are
satisfied with the summary. The draft summaries are generated by the standard
summarization model. We compare the editing processes with and without
interaction.

Figure 3 shows an illustration of the editing process with interaction. In
practice, we prompt the annotators with 3 suggestions. We adopt the top-3
beams of beam search decoding method as suggestions, to enforce the
suggestions are at least one-token different.

Avg. Time Avg. Rating Accept / Accept w Edits / Reject Hallucination Rate
Draft Summary – 3.99 $\pm$ 1.62 0.12 Human w/o interaction 903.0 4.61 $\pm$
1.73 0.04 Human w interaction 645.5 5.52 $\pm$ 0.84 0.03

Table 3:  Average editing time and annotation quality. The ratings are the
general quality of the summary, with 1 meaning worst and 7 meaning best.

The experiment consists of two stages. In the first stage, we perform a
contrast experiment, to collect annotations with or without interaction. In
the second stage, we evaluate the annotations collected from the first stage,
to see whether the annotation quality and efficiency are improved. The
experiment takes 3 annotators, where each annotator will

  1. 1.

annotate 40 documents with interaction,

  2. 2.

annotate 40 documents without interaction,

  3. 3.

evaluate the summaries of 40 documents, where each document has 3 summaries,
i.e., the draft summary and the human annotated summary with and without
interaction.

The 40 documents in each of the above tasks are different for each annotator.
In this way, we enforce the annotation time and evaluation results are
unbiased, as the annotators will never see the same document across different
tasks.

Table 3 shows REVISE can significantly save editing time and improve
annotation quality. In practice, annotators trigger 2.2 suggestions for each
document on average. Due to model inference latency and network latency, the
suggestions will take 4.1s on average, which is also included in the
annotation time reported in Table 3.

##  5 Conclusion

In this work, we propose REVISE, offering an interactive and iterative
approach for human writers to efficiently edit and refine summaries tailored
to their specific needs. We adopt a fill-in-the-middle model for the encoder-
decoder architecture underlies the framework, empowering it to generate
coherent and contextually relevant suggestions for human edits, validated by a
set of novel evaluation metrics. We perform extensive human evaluation,
suggesting the proposed framework can improve both the editing efficiency and
summary quality. We hope REVISE can herald a new era in the summarization
experience, with potentially transformative implications for practical
applications and inspiring further research in interactive AI systems.

## References

  * Aghajanyan et al. (2022) Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. 2022.  Cm3: A causal masked multimodal model of the internet.  _arXiv preprint arXiv:2201.07520_.
  * Avinesh et al. (2018) PVS Avinesh, Carsten Binnig, Benjamin Hättasch, Christian M Meyer, and Orkan Özyurt. 2018.  Sherlock: A system for interactive summarization of large text collections.  _Proc. VLDB Endow._ , 11(12):1902–1905.
  * Avinesh and Meyer (2017) PVS Avinesh and Christian M Meyer. 2017.  Joint optimization of user-desired content in multi-document summaries by learning from user feedback.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1353–1363.
  * Barrachina et al. (2009) Sergio Barrachina, Oliver Bender, Francisco Casacuberta, Jorge Civera, Elsa Cubel, Shahram Khadivi, Antonio Lagarda, Hermann Ney, Jesús Tomás, Enrique Vidal, et al. 2009.  Statistical approaches to computer-assisted translation.  _Computational Linguistics_ , 35(1):3–28.
  * Bavarian et al. (2022) Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022.  Efficient training of language models to fill in the middle.  _arXiv preprint arXiv:2207.14255_.
  * Bohn and Ling (2021) Tanner Bohn and Charles X Ling. 2021.  Hone as you read: A practical type of interactive summarization.  _arXiv preprint arXiv:2105.02923_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chan et al. (2019) William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019\.  Kermit: Generative insertion-based modeling for sequences.  _arXiv preprint arXiv:1906.01604_.
  * Chen et al. (2019) Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, et al. 2019.  Gmail smart compose: Real-time assisted writing.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2287–2295.
  * Cheng et al. (2022) Ruijia Cheng, Alison Smith-Renner, Ke Zhang, Joel R Tetreault, and Alejandro Jaimes. 2022.  Mapping the design space of human-ai interaction in text summarization.  _arXiv preprint arXiv:2206.14863_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Donahue et al. (2020) Chris Donahue, Mina Lee, and Percy Liang. 2020.  Enabling language models to fill in the blanks.  _arXiv preprint arXiv:2005.05339_.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 320–335.
  * Engelbart (1962) Douglas C Engelbart. 1962.  Augmenting human intellect: A conceptual framework.  _Menlo Park, CA_ , page 21.
  * Fedus et al. (2018) William Fedus, Ian Goodfellow, and Andrew M Dai. 2018.  Maskgan: better text generation via filling in the_.  _arXiv preprint arXiv:1801.07736_.
  * Fried et al. (2022) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022.  Incoder: A generative model for code infilling and synthesis.  _arXiv preprint arXiv:2204.05999_.
  * Gao et al. (2020) Yang Gao, Christian M Meyer, and Iryna Gurevych. 2020.  Preference-based interactive multi-document summarisation.  _Information Retrieval Journal_ , 23(6):555–585.
  * González-Rubio et al. (2013) Jesús González-Rubio, Daniel Ortiz-Martínez, José-Miguel Benedí, and Francisco Casacuberta. 2013.  Interactive machine translation using hierarchical translation models.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 244–254.
  * González-Rubio et al. (2016) Jesús González-Rubio, Daniel Ortiz-Martínez, Francisco Casacuberta, and José-Miguel Benedí. 2016.  Beyond prefix-based interactive translation prediction.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 198–207.
  * Gu et al. (2019) Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019.  Insertion-based decoding with automatically inferred generation order.  _Transactions of the Association for Computational Linguistics_ , 7:661–676.
  * He et al. (2022) Pengcheng He, Baolin Peng, Liyang Lu, Song Wang, Jie Mei, Yang Liu, Ruochen Xu, Hany Hassan Awadalla, Yu Shi, Chenguang Zhu, et al. 2022.  Z-code++: A pre-trained language model optimized for abstractive summarization.  _arXiv preprint arXiv:2208.09770_.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Lee et al. (2022) Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. 2022.  Evaluating human-language model interaction.  _arXiv preprint arXiv:2212.09746_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2019) Dayiheng Liu, Jie Fu, Pengfei Liu, and Jiancheng Lv. 2019.  Tigs: An inference algorithm for text infilling with gradient search.  _arXiv preprint arXiv:1905.10752_.
  * Nogueira and Cho (2017) Rodrigo Nogueira and Kyunghyun Cho. 2017.  Task-oriented query reformulation with reinforcement learning.  _arXiv preprint arXiv:1704.04572_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Peris and Casacuberta (2018) Álvaro Peris and Francisco Casacuberta. 2018.  Active learning for interactive neural machine translation of data streams.  _arXiv preprint arXiv:1807.11243_.
  * Peris and Casacuberta (2019a) Álvaro Peris and Francisco Casacuberta. 2019a.  Interactive-predictive neural multimodal systems.  In _Iberian Conference on Pattern Recognition and Image Analysis_ , pages 16–28. Springer.
  * Peris and Casacuberta (2019b) Alvaro Peris and Francisco Casacuberta. 2019b.  Online learning for effort reduction in interactive neural machine translation.  _Computer Speech & Language_, 58:98–126.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Shapira et al. (2022) Ori Shapira, Ramakanth Pasunuru, Mohit Bansal, Ido Dagan, and Yael Amsterdamer. 2022\.  Interactive query-assisted summarization via deep reinforcement learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2551–2568.
  * Shapira et al. (2021) Ori Shapira, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2021.  Extending multi-document summarization evaluation to the interactive setting.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 657–677.
  * Shen et al. (2020) Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020.  Blank language models.  _arXiv preprint arXiv:2002.03079_.
  * Stern et al. (2019) Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019.  Insertion transformer: Flexible sequence generation via insertion operations.  In _International Conference on Machine Learning_ , pages 5976–5985. PMLR.
  * Weng et al. (2019) Rongxiang Weng, Hao Zhou, Shujian Huang, Lei Li, Yifan Xia, and Jiajun Chen. 2019\.  Correct-and-memorize: Learning to translate from interactive revisions.  _arXiv preprint arXiv:1907.03468_.
  * Yan et al. (2011) Rui Yan, Jian-Yun Nie, and Xiaoming Li. 2011.  Summarize what you are interested in: An optimization framework for interactive personalized summarization.  In _Proceedings of the 2011 conference on empirical methods in natural language processing_ , pages 1342–1351.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34:27263–27277.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * Zhu et al. (2019) Wanrong Zhu, Zhiting Hu, and Eric Xing. 2019.  Text infilling.  _arXiv preprint arXiv:1901.00158_.

##  Appendix A Human Annotation Guideline

Here we include the complete instructions for humen annotators, which include
experiment details such as evaluation portal snapshots.

###  A.1 Goal

This experiment is to validate the effectiveness of the interactive
summarization pipeline. Specifically, this pipeline is to facilitate the
summary annotation process. It not only provides the annotators with a draft
summary, but also interactively provides suggested completions to the
annotators while they are editing the summary. This collected data will be
used to further improve the annotation pipelines.

###  A.2 Experiment Design and Instruction

The experiment consists of two stages. In the first stage, we perform a
contrast experiment, to collect annotations with or without interaction. In
the second stage, we evaluate the annotations collected from the first stage,
to see whether the annotation quality and efficiency are improved. The
experiment needs 3 annotators, where each annotator will

  1. 1.

annotate 40 documents with interaction,

  2. 2.

annotate 40 documents without interaction,

  3. 3.

evaluate the summaries of 40 documents, where each document has 3 summaries.

####  A.2.1 Stage 1: Contrast experiment with or without interaction

The experiment consists of two groups, i.e., the experimental group (with
interaction) and the control group (without interaction), and there are 120
examples in each group. Since there are 3 annotators working on the
experiment, each annotator will label 80 examples in total, where 40 belong to
the experimental group and 40 belong to the control group. For each example,
the annotator is given a source document, a draft summary, and possibly some
interactive suggestions, and we will record

  1. 1.

the annotated summary,

  2. 2.

the annotation time.

Since the annotation time is also critical to this experiment, we would like
the annotators not to be distracted when doing annotation.

A.2.1.1 Experimental Group

Portal description. Figure 4 shows a snapshot of the annotation portal.

Figure 4:  A snapshot of annotation portal for the experiment group.

The portal contains a few components:

  1. 1.

Source document: This is the document to be annotated.

  2. 2.

Draft Summary: To ease the annotation, we provide a draft summary as the start
point of annotation. The draft summary may or may not be a good candidate
summary. The annotator is asked to edit the draft summary until it satisfy the
conditions described in the next section “Guidelines on What is a Good Summary
Annotation”.

  3. 3.

Suggested summary – Choice 1/2/3: While the annotator is editing the summary,
we provide three alternative suggestions interactively. Note that sometimes
when the possible alternatives are limited, the suggestions can be the same.

  4. 4.

Choose one you like for further editing: If the annotator would like to adopt
one of the suggestions, they can indicate so using this radio. After making a
selection, the adopted suggestion will appear in the draft summary box for
further editing.

  5. 5.

Save and Next button: Save current annotation and navigate to the next
example.

  6. 6.

Previous: Navigate to the previous example. The edits in the current example
will be lost. If the annotator navigates back to a previous example, make
edits and save the annotation again, the later annotation will be adopted.

Instruction on the usage of the suggestions. The generation of the suggested
summary is triggered by the ENTER key on the keyboard. The system will compare
the old summary and the current summary and determine which part should be
replaced.

As an illustrative example, if you are not satisfied with part of the summary,
e.g., the highlighted part below,

you can delete the highlighted part, hit the enter key, and it will trigger
the system to generate suggestions, as shown below.

In this example, the generated suggestion is not good – it has the same
content as the original one. To let the system know what you would like to
insert instead, you can type the start you like, e.g.,

By typing “Business practice” into the summary box following up by two spaces,
all suggestions will continue from “Business practice”. If you prefer the
second choice, select “Choice 2”, and then the second suggestion will appear
in the summary box for you to make further edits.

A.2.1.2 Control Group

Figure 5 shows a snapshot of the annotation portal.

Figure 5:  A snapshot of annotation portal for the control group.

The purpose of source document box, draft summary box, previous button, and
save & next button is similar to the experimental group. The only difference
is that in control group we do not provide interactive suggestions.

A.2.1.3 More Guidelines on How to Make a Good Summary Annotation

A good summary is a concise, fluent, and coherent text that can capture the
main points of the original document/transcript accurately without any grammar
errors, unrelated contents or hallucinations. Here are more detailed
guidelines:

  1. 1.

Although a draft summary is provided, we would suggest reading through the
document first to make a summary without being biased by the draft summary.
The draft summary may or may not be trustworthy – it may miss the point
completely or include nonexistent information.

  2. 2.

Keep perspective. If the document is of first-person perspective, use first-
person perspective in the summary, too. Same applies to the third-person
perspective.

  3. 3.

Summaries should cover important information in a concise way. It should avoid
redundancy or unimportant information and should not be too verbose.

  4. 4.

Summaries should be self-contained and understandable by another readers
without looking at the document. It should be fluent and coherent itself, like
a very short passage.

  5. 5.

Summaries should not be too generic. For example, “This document talks about
the economics of the developing countries.” is not as informative as “In a
developing country, there are trials and tribulations.”

  6. 6.

Summaries should not add or make up new information that is not supported by
the document.

####  A.2.2 Stage 2: Evaluation on Collected Summaries

Each annotator will be assigned to 40 documents, each with 3 summaries. They
will assign each summary a score on a 7-point Likert scale (1 = worst, 7 =
best) and answer seven binary questions about potential issues in the summary.