11institutetext: 1 University of Florida, Gainesville, FL32611, USA,
2 Roswell Park Comprehensive Cancer Center, Buffalo, NY14263, USA
3 Ph.D., Professor
4 corresponding author<EMAIL_ADDRESS>
432 Newell Dr, U of Florida, Gainesville, FL, 32611-6120,USA,
(352)382 1200 (fax 1220)

# Adding Safety Rules to Surgeon-Authored VR Training

Ruiliang Gao 11 Sergei Kurenov 22 Erik W. Black 1133 Jörg Peters 113344

###### Abstract

.
Introduction Safety criteria in surgical VR training are typically hard-coded
and informally summarized. The Virtual Reality (VR) content creation
interface, TIPS-author, for the Toolkit for Illustration of Procedures in
Surgery (TIPS) allows surgeon-educators (SE s) to create laparoscopic VR-
training modules with force feedback. TIPS-author initializes anatomy shape
and physical properties selected by the SE accessing a cloud data base of
physics-enabled pieces of anatomy.
Methods A new addition to TIPS-author are safety rules that are set by the SE
and are automatically monitored during simulation. Errors are recorded as
visual snapshots for feedback to the trainee. This paper reports on the
implementation and opportunistic evaluation of the snapshot mechanism as a
trainee feedback mechanism. TIPS was field tested at two surgical conferences,
one before and one after adding the snapshot feature.
Results While other ratings of TIPS remained unchanged for an overall Likert
scale score of 5.24 out of 7 (7 = very useful), the the rating of ‘The TIPS
interface helps learners understand the force necessary to explore the
anatomy’ improved from 5.04 to 5.35 out of 7 after the snapshot mechanism was
added.
Conclusions The ratings indicate the viability of the TIPS open-source SE-
authored surgical training units. Presenting SE-determined procedural missteps
via the snapshot mechanism at the end of the training increases acceptance.

###### Keywords:

laparoscopy; virtual reality; computer simulation; patient-specific modeling;
patient safety; education, medical; internship and residency

##  1 Introduction

Teaching laparoscopic surgery under one-on-one supervision in the operating
room (OR) is costly ranging, already a decade ago, from $50–$135 per minute
[1]. Less supervision is risky: cauterizing too close to a sensitive organ or
nicking a central vein are difficult to repair and may cause the patient
unnecessary suffering. Therefore alternative training methods are ethically
and fiscally prudent. Mentored self-study curricula, such as Fundamentals of
Laparoscopic Surgery (FLS), offer dexterity training and certification on peg-
board transfer, cutting and suturing of physical props as a foundation before
working on real patients [2]. However, FLS box training can not prepare for
the high variability of anatomy and soft tissue response that actual cases
present and provides no automatic checking of safety criteria.

The additional technical challenge that this work addresses is that entry must
interpret and trigger deployment of monitors for a palette of surgical safety
criteria set by surgeon educators. To explain the challenge, we first review
soft tissue simulation and VR trainers in general in Section 1.1, then a
customizable training framework (TIPS, Section 1.2) and then, in Section 1.3,
formulate the specific challenge and new contributions.

###  1.1 Soft tissue simulation and VR trainers

The last decade has witnessed progress in soft tissue simulation for a range
of surgical scenarios such as laparoscopic surgery, heart surgery,
neurosurgery, orthopedic and arthroscopic surgery. Early multilayered tissue
models for orthopedic trauma surgery were based on 3D mass-spring systems
accelerated with graphics hardware [3]. More recently, simulation of cardiac
electrophysiology simulation, pre-operative planning of cryosurgery and per-
operative guidance for laparoscopy use finite elements in real time in the
open source SOFA soft tissue simulation platform [4]. A real-time neurosurgery
simulator of skull drilling and surgical interaction with the brain was
proposed in [5] and [6] report on simulation of drilling and cutting of the
bone using the burr and the motorized oscillating saw based on the open source
iMSTK framework. [7] presented a framework for interactive outlining of
regions for simulation of reconstructive plastic surgery and [8] describes a
virtual surgical environment for training residents in less invasive
stabilization system surgery used to address fractures of the femur.

Several commercial VR training environments aim to reduce time spent teaching
in the OR by offering training modules with virtual anatomy that can be probed
using force feedback devices. Manual laparoscopic techniques lend themselves
particularly well to simulation that leverage force-feedback devices. Virtual
reality simulators allow trainees to practice decision-making and execution
prior to entering the OR [9, 10]. A number of commercial solutions have sunset
during the past 20 years (e.g. Simsurgery), or were merged or bought up by
larger companies (see e.g. SurgicalScience, Simbionics, Mimic). However,
commercial training environments neither capture the broad spectrum of
physical variations encountered in laparoscopic practice, nor prepare learners
for less common interventions.

(a) TIPS environment, no hand piece

(b) setup with wired handles

Figure 1: Physical setup of the TIPS environment: (a) left screen TIPS-
simulator; right screen TIPS-trainee providing instructions and quizzes. Two
six degrees-of-freedom haptic devices provide physical feedback. The haptic
setup can be augmented by (b) laparoscopic tool handles for a training setup
with fulcrum, similar to an FLS box trainer, but wiring is cumbersome and
brittle.

Building a robust virtual environment is a formidable challenge, leveraging
scientific advances in collision detection, real-time differential equation
solving, interactive visual and haptic feedback in a well-engineered
interface. Creating or updating VR scenarios is a second hurdle: due to the
back-and-forth between engineers, computer scientists and medical experts,
content creation is neither cheap nor fast and can take months or years; and
the result is not easily adjustable to create variants, uncommon or
specialized scenarios.

###  1.2 The Toolkit for Illustration of Procedures in Surgery (TIPS)

Figure 2: Screenshot of TIPS-trainee instructions directly generated from the
TIPS-author entries with embedded (real and) simulation footage video clips.

The open source Toolkit for Illustration of procedures in Surgery (TIPS)
addresses the fast prototyping challenge of missing variants of anatomy and of
less common laparoscopic procedures. Fig. 1 shows two physical setup to
transmit force via small robotic arms. The TIPS open source environment
consists of TIPS-simulator, an interactive soft-tissue laparoscopic simulation
with force feedback, (TIPS-trainee, a web-based component providing
instruction and examples to a novice surgeon and TIPS-author (see Section 2.1)
that allows the surgeon-educator to specify steps of a minimally invasive
procedure in the fixed format:

The triple ‘action, anatomy, tool’ is used to initialize the geometry and
physical properties of the virtual anatomy. This high-level initialization is
possible thanks to a rich database of simlets. A simlet is a piece of anatomy
with its physical properties, created by content artists. Simlets combine in a
Lego-like fashion [11, 12, 13]. For example, the cystic duct, cystic artery
and the fatty tissue covering them (each with their unique Young’s modulus
etc.), form an anatomy simlet. In our implementation, the web-based TIPS-
author interface auto-completes typed items once they are recognized to be in
the database and thereby steers the author towards existing simlets (see
Video, Supplemental Digital Content 1). Both the listing of steps and the
resulting simulation is peer-reviewed for completeness and relevance before
roll-out to trainees.

An initial study of 34 medical students [14] assessed whether the interactive
learning within the prototype TIPS environment has advantages over passive
learning from professional instructional videos [15]. The study showed that
inter-active TIPS platform instilled greater confidence in the ability to
reproduce the steps of the procedure (p=0.001) and was preferred by the
participants as a learning tool (p=0.011). Of course confidence is not always
positively correlated with proficiency [16].

###  1.3 The missing component and contribution: automatic initialization and
monitoring of safety criteria

The _missing component in earlier work_ is a lack of automatic interpretation
of a ‘safety’ entry in the specification of a surgical task. The technical
challenge is that this entry must interpret and trigger deployment of monitors
for a palette of surgical safety criteria set by surgeon educators. Examples
are: do not cauterize near sensitive organs, limit the force when separating
vessels from fatty tissue, etc. . The challenge addressed in this paper is to
provide a _generic mechanism_ for meaningful author-controlled yet automatic
(unsupervised) surgical safety feedback to the trainee – and so accommodate
current and future not yet specified laparoscopic training scenarios.

While on one hand, haptic interactive simulation is no different from (a
specialized) computer game, on the other hand the safety criteria can not be
hard-coded a priori as in a game environment – because the surgeon-author sets
the criteria. The challenge then is to insure that (a large class) of surgical
safety criteria can be (i) formulated, (ii) automatically translated into
measurable events during VR simulation and (iii) generate both immediate
feedback to the trainee and a meaningful final report to be shared with the
instructor. Section 2 introduces TIPS. Section 3 reports on the _new
contributions_ that

  * 1.

allow the surgeon-educator to specify the surgical safety criteria;

  * 2.

are automatically monitored within TIPS-simulator;

  * 3.

provide immediate feedback to the trainee; and

  * 4.

return, in a secure fashion, visual feedback to both the trainee and a summary
message to the instructor as a series of snapshots.

The list of errors and task-completion in the message (4.) record progress: a
repeatedly empty error-report and completed task report indicate proficiency
with respect to the training unit. This can be used to trigger the final
assessment by the instructor and complete a feedback loop to improve the
teaching unit by setting additional safety criteria or better specifying steps
of the procedure.

Figure 3: TIPS-author defines the interactive VR training simulation: (1) The
author specifies procedural steps and safety concerns in a fixed format. (2)
Instruction pages are generated from the author’s description. (3) Simlets
(pieces of anatomy and their physical properties) are combined to initialize
the scenario in TIPS-simulator. (4) Trainee achievements and safety violations
are screen-captured in TIPS-simulator for post-review. This is the focus and
contribution of the paper. (5) Completion and errors are reported to the
trainee as snapshots of missteps.

##  2 Methods

###  2.1 TIPS-author: a surgical simulation creation environment

TIPS-author enables a surgeon-educator (SE) to improve the specialization,
variety and relevance of laparoscopic VR-training. TIPS-author provides a
surgeon-educator (SE) with an open source environment to create and customize
hands-on, interactive force-feedback laparoscopy training units. Based on the
SE’s listing, TIPS-author extracts a set of compatible, computationally
efficient simlets from a database, and generates step-by-step instructions for
the web-based TIPS-trainee interface, quizzes and, as specified in Section
Section 2.2, the monitoring of surgical errors. The database is generated by a
scenario design cycle [12] that separates the roles of author, developer and
artist:

  * -

at the developer level, numerical simulation routines are selected or adjusted
and admissible parameter ranges are determined;

  * -

at the artist level, geometric models with collision and physics parameters
(simlets) are created; and

  * -

at the surgeon-author level, simlets are combined and the workspace and view
determined via a interactive webGL interface.

Key to this approach is that it can leverage the vibrant professional open-
source geometric modeling software _Blender_ (http://blender.org, [17]) and a
library of numerical, geometric and visual algorithms for soft-tissue
simulation, _SOFA_ (http://www.sofa-framework.org, [18, 19]). _Blender_ and
_SOFA_ are combined to generate hex-meshes on the fly for carvable fatty
tissue and thick-shell models of the stomach. Anatomy and physics are
customized via a menu added to _Blender_ that annotates the .xml files sent to
_SOFA_ for simulation. Combination and customization are possible due to a
small disciplined choice of representation primitives.

###  2.2 Adding safety monitoring and feedback to TIPS

Assessment, evaluation and feedback are critical components in the training of
novice surgeons and obeying safety rules is paramount when executing complex
sequences of maneuvers. Physician-training is an experiential process. That
is, learners acquire skill by engaging in supervised patient care. All US
physicians-in-training, including surgical trainees, must demonstrate
competency across a range of knowledge, skills and attitudes prior to
graduation [20, 21]. Assessing, evaluating and providing critical feedback and
instruction in the workplace is time intensive and stressful. And it requires
an experienced surgeon’s active participation and expert judgment to provide
safe and effective patient care and a quality learning experience. To ensure
that the assessment and evaluation of surgical trainees is reliable and valid
many training programs employ peer-reviewed evaluative tools such as the
objective structured assessment of technical skills (OSATS) for workplace-
based assessment [22].

Assessment is also central, but arguably less stressful, in popular computer
games where simple counters monitor progress. Psychometric games claim to
measure mental agility, attention, cognitive speed, spatial aptitude and
numerical processing ability. Increasingly, educational video games
incorporate stealth assessment, ubiquitous, unobtrusive, and real-time
assessments that intersect play, learning, and assessment. Stealth assessments
measure knowledge and skill, then provides learning supports, feedback,
instructions, or adapts challenges in the learning environment (e.g.,
difficulty) to students’ proficiency level, maximizing their learning [23].

Existing VR simulators typically report time to completion, task specific data
such as the number of staples used, and other general counters. TIPS’s
incorporation of SE-established safety criteria makes cases more relevant for
the specific procedure – but this approach also implies that the _criteria
cannot be hard-coded in the simulator ahead of time_. Consultation with the
clinical experts identified general classes of training errors (i–vi):

  * i

Incising or cauterizing at the wrong location.

  * ii

Injuring a nerve by applying too much force (pressure or over-stretching).

  * iii

Leaving foreign objects in the patient’s body (clips, tools).

  * iv

Applying surgical clips incorrectly.

  * v

Removing the wrong (part of) an organ.

  * vi

Suturing at the wrong location.

\begin{overpic}[trim=150.5625pt 100.37498pt 50.18748pt
110.41249pt,clip,scale={.25},tics=10,width=433.62pt]{FIG/ErrorCBDCut.png}
\put(73.0,40.0){{\color[rgb]{1,1,1}$\Downarrow$}} \end{overpic} (a) Wrong
incision on common bile duct.

\begin{overpic}[trim=150.5625pt 100.37498pt 50.18748pt
100.37498pt,clip,scale={.25},tics=10,width=433.62pt]{FIG/ErrorCDStrech.png}
\put(65.0,50.0){{\color[rgb]{1,1,1}$\Downarrow$}} \end{overpic} (b)
Overstretching the cystic duct.

\begin{overpic}[trim=100.37498pt 100.37498pt 100.37498pt
100.37498pt,clip,scale={.25},tics=10,width=433.62pt]{FIG/ErrorLeaveClipInside.png}
\put(45.0,45.0){{\color[rgb]{1,1,1}$\Downarrow$}} \end{overpic} (c) Clip drops
to abdominal wall due to bile duct cut at the wrong location.

\begin{overpic}[trim=100.37498pt 50.18748pt 100.37498pt
150.5625pt,clip,scale={.25},tics=10,width=433.62pt]{FIG/ErrorNotEnoughClips.png}
\put(50.0,45.0){{\color[rgb]{1,1,1}$\Downarrow$}} \end{overpic} (d) Bile leak
due to the lack of vascular clips on the left side.

Figure 4: Four types of common surgical errors in laparoscopic cholecystectomy
reported by TIPS-simulator. For immediate feedback the tool tip becomes red
(see $\Downarrow$) and the scene briefly freezes (see Video, Supplemental
Digital Content 1).

These surgical errors can be abstracted as: distance to anatomy, force
exerted, location and number of surgical safety clips, and incomplete
execution. Initialized by the ‘safety’ entry in TIPS-author, our solution is
to have TIPS-author parse these safety criteria and append the corresponding
safety tags to these simlets upon export to TIPS-Simulator. TIPS-simulator
monitors these data streams and reports violations both directly and as a
sequence of screen-shot images labeled by error types. Fig. 4 shows screen-
shots of four common surgical errors (corresponding to type i - iv) during
laparoscopic cholecystectomy.

In more detail, error class (i) is monitored by TIPS-simulator as a collision
event with an offset distance between the tool listed in the TIPS-authors
tuple of the task and an organ listed in the safety entry.

For example, for cholecystectomy, for the step ‘Explore the triangle of Calot’
(see Fig. 4a) the task tuple reads:

  * dissect Fatty tissue over the cystic ductus and cystic artery
using Curved Maryland Dissector not too close to Common bile duct.

Figure 5: Excessive force indicated by change of vessel to red (flashing)
color.

Here ‘dissect’ is the action, ‘Fatty tissue over the cystic ductus and cystic
artery’ is the anatomy (specifying a simlet), ‘Curved Maryland Dissector’
specifies the laparoscopic tool, ‘not too close’ indicates an error of type
(i) and ‘Common bile duct’ is an entry in the anatomy database that requires
monitoring. TIPS-simulator then monitors distance between the cauterizing tool
and the common bile duct. Distance below the offset triggers and registers the
error.

Type (ii) errors are monitored in terms of force feedback returned to the
haptic devices. This safety threshold is customized for veins or arteries with
different physical properties. Type (iii) and (iv) errors are detected by a
map tracking the vector of deployed clips on each clip-able object to monitor
not only the number but also placement of clips. For example, to prevent
bleeding or leaking, two clips should be applied on the part of the duct or
vein that remains inside the body. Type (v) errors are indirectly caught since
they terminate the simulation without generating an ‘achievement’ entry in the
final visual report and type (vi) errors are caught by initializing suturable
regions on the object, say the fundus of the gaster during Nissen
fundoplication.

Errors (i - iv) alert the trainee by a red flashing (Fig. 4a) instrument tip.
A corresponding screen-shot is saved for later named by time, error type, and
error values.

###  2.3 Summary feedback as a series of snapshots

Once the procedure completes, typically when the cancerous organ is retrieved
via the surgical pouch, all screen-shots of errors (and small ones for task
completions) are displayed to the trainee as a feedback report. This serves as
starting point for a discussion with the instructor. Proficiency with respect
to the training module is equivalent to repeated performance without errors
and a complete list of achievements. The final achievement is generically
checked by asserting that the cancerous body is free from the remaining organs
and tissues. Similarly, clip placement requires freeing the vessel and testing
that two clips remain within the body while a third clip ensures integrity of
the tissue to be removed. Such authored criteria provide more valuable
feedback than time taken or number of clips deployed.

Figure 6: Email reports to trainee (and the instructor) upon completion.

Additionally the unique directory of screen-shots and the filenames are
reported to the trainee by e-mail and, optionally, to an account set up for
the instructor (see Fig. 6) to document training progress and decide whether
the pattern and number of errors requires intervention and what errors should
be discussed.

In summary, faced with the complexity of supporting procedure-specific
proficiency assessment, we categorized laparoscopic safety violations into
several generic classes. This enables a simple but effective, implemented and
tested strategy to use TIPS-author safety entries to initialize, monitor and
report error events; and to create a record of progress towards proficiency.

##  3 Results

###  3.1 Evaluation of TIPS training and feedback

TIPS was demonstrated and experienced by a broad range of medical
professionals at the the American College of Surgeons Clinical Congress 2019
(ACS) and the Academic Surgical Congress 2020 (ASC). Besides testing the
technology ‘in the wild’, the venues allowed the team to conduct a survey of
TIPS.

Prior to field testing, face, construct and content validity of SE-authored
cholecystectomy and appendectomy TIPS modules had been established by
laparoscopic surgeons and residents at the Universities of Florida and
Buffalo. At the congress field tests, after training with the modules, 64
respondents (13 board certified surgeons, 17 medical residents, 27 medical
students and 7 other medical professionals) rated TIPS across several
usability items on a Likert scale from 1 to 7 (7 = strongly agree, see [24]).
The scale resolution was selected as a trade-off between scale complexity and
expressiveness. Table 1 lists the outcome of the four central questions of
usability and Fig. 7 breaks down the score on these four central questions.
(Four other questions established medical seniority, familiarity with virtual
trainers, and prior experience with laparoscopy). All questions were selected
in consultation with SEs at the authors’ institutions.

|  mean |  standard
---|---|---
TIPS … |  rating |  deviation
helps understand the force necessary to explore the anatomy |  5.34 |  1.46
interface does not distract from the surgical task |  5.02 |  1.52
enhances lap-competency attainment over current methods |  5.19 |  1.5
is compatible with the current lap training curricula |  5.39 |  1.43
overall score |  5.24 |  1.33
Table 1: TIPS with safety rules rated on the four key questions.

(a) cumulative

(b) by group

Figure 7: Breakdown of the average score on the 4 central TIPS evaluation
questions.

###  3.2 The effect of summative visual feedback via snapshots

When analyzing the data sets from the two conferences, we noticed agreement of
averages between the 13 ACS respondents and the 51 ASC respondents. The
agreement was within .2 on all rating categories except one. The only outlier
was the statement ‘The TIPS interface helps learners understand the force
necessary to explore the anatomy’. Here the rating improved from 5.04 at ACS
to 5.35 at ASC. The only change applied to the TIPS software after the ACS
survey and before the ASC survey, was the addition of the visual summary of
the achievements and procedural errors as a series of snapshots. The immediate
feedback by changing the color of a vessel was present in both tests.

##  4 Discussion

TIPS is a novel authoring environment that allows surgeon-educators to build
customizable VR lap scenarios. The bulk of the survey questions was aimed to
evaluate TIPS as a whole. Ratings collected from medical professionals at two
conference exhibitions indicate the viability of such SE-authored surgical
training. In particular, the high score for ‘enhances lap-competency
attainment over current methods’ speaks to the added value of customized TIPS
simulations over available current methods.

We did not set out to measure the impact of automatic visual summative
feedback on errors presented as screen snapshots. In fact, the immediate
feedback on SE-authored error measurements, a change of color, was present at
both field tests. It is therefore noteworthy that presenting trainee errors in
visual form at the end of the training increased acceptance noticeably.
Indeed, additional informal feedback from surgeons and trainees to the
question ’what feature of TIPS do you recall’ endorsed visual feedback via
screen shots as both meaningful and memorable.

## References

  * [1] A. Macario, What does one minute of operating room time cost?, J Clin Anesth. 22 (4) (2010) 233–6.  doi:10.1016/j.jclinane.2010.02.003.PMID:20522350.
  * [2] Z. B, Brydges, H. S. C. DA., State of the evidence on simulation-based training for laparoscopic surgery: a systematic review, Ann Surg. 257 (4) (2013) 586–593, pMID: 23407298.  doi:10.1097/SLA.0b013e318288c40b.
  * [3] J. Qin, W.-M. Pang, Y.-P. Chui, T.-T. Wong, P.-A. Heng, A novel modeling framework for multilayered soft tissue deformation in virtual orthopedic surgery, Journal of medical systems 34 (3) (2010) 261–271.
  * [4] H. Talbot, N. Haouchine, I. Peterlik, J. Dequidt, C. Duriez, H. Delingette, S. Cotin, Surgery training, planning and guidance using the sofa framework, in: Eurographics, 2015.
  * [5] G. Echegaray, I. Herrera, I. Aguinaga, C. Buchart, D. Borro, A brain surgery simulator, IEEE computer Graphics and Applications 34 (3) (2014) 12–18.
  * [6] V. S. Arikatla, M. Tyagi, A. Enquobahrie, T. Nguyen, G. H. Blakey, R. White, B. Paniagua, High fidelity virtual reality orthognathic surgery simulator, in: Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling, Vol. 10576, International Society for Optics and Photonics, 2018, p. 1057612.
  * [7] N. Mitchell, C. Cutting, E. Sifakis, Gridiron: an interactive authoring and cognitive training foundation for reconstructive plastic surgery procedures, ACM Transactions on Graphics (TOG) 34 (4) (2015) 1–12.
  * [8] J. Cecil, A. Gupta, M. Pirela-Cruz, An advanced simulator for orthopedic surgical training, International journal of computer assisted radiology and surgery 13 (2) (2018) 305–319.
  * [9] A. Gallagher, E. Ritter, H. Champion, G. Higgins, M. Fried, G. Moses, C. Smith, R. Satava, Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training, An. Surgery 241 (2) (2005) 364–372.
  * [10] L. P. K.S. Gurusamy, R. Aggarwal, B. Davidson, Virtual reality training for surgical trainees in laparoscopic surgery, The Cochrane Database of Systematic Reviews 1 (2009).
  * [11] Y.-I. Yeo, S. Dindar, G. Sarosi, J. Peters, Enabling surgeons to create simulation-based teaching modules, in: Medicine Meets Virtual Reality (MMVR) Feb 8-12 2011, Long Beach,CA, Studies in Health Technology and Informatics (SHTI), IOS Press, Amsterdam, 2011, pp. 1–6.
  * [12] S. Dindar, T. Nguyen, J. Peters, Towards surgeon-authored VR training: the scene-development cycle, in: Proceedings of Medicine Meets Virtual Reality (MMVR) April 9-12 2016, L.A. ,CA, Studies in Health Technology and Informatics (SHTI), IOS Press, Amsterdam, 2016, pp. 1–6.
  * [13] M. Sarov, R. Gao, J. Youngquist, G. Sarosi, S. Kurenov, J. Peters, An authoring interface for surgeon-authored VR training, International Journal of Computer Assisted Radiology and Surgery 13 (2018) 1–(1–4)–273.
  * [14] H. Lesch, E. Johnson, J. Cendan, J. Peters, VR simulation leads to enhanced procedural confidence for surgical trainees, Journal of Surgical Education 77 (1) (2019) 213–218, nIHMS1680076.
  * [15] N. G. S. of Medicine, A. C. of Surgeons (ACS), A. of Surgical Education (ASE), Wise-md.
URL http:/www.aquifer.org

  * [16] M. B, Belton, Powell, I. J., The relationship between fundamental movement skill proficiency and physical self-confidence among adolescents, J Sports Sci 35 (17) (2017) 1709–1714, pMID: 28282760.  doi:10.1080/02640414.2016.1235280.
  * [17] B. Foundation, Blender, http://blender.org/ (2015).
  * [18] INRIA, Simulation open framework architecture, http://www.sofa-framework.org (2015).
  * [19] J. Allard, S. Cotin, F. Faure, P.-J. Bensoussan, F. Poyer, C. Duriez, H. Delingette, L. Grisoni, Sofa an open source framework for medical simulation, in: Medicine Meets Virtual Reality (MMVR’15), Long Beach, USA, 2007\.
  * [20] S. Yardley, P. W. Teunissen, T. Dornan, Experiential learning: transforming theory into practice, Medical teacher 34 (2) (2012) 161–164.
  * [21] E. S. Holmboe, Realizing the promise of competency-based medical education, Academic Medicine 90 (4) (2015) 411–413.
  * [22] J. Martin, G. Regehr, R. Reznick, H. MacRae, J. Murnaghan, C. Hutchison, M. Brown, Objective structured assessment of technical skill (osats) for surgical residents, Br J Surg. (1997) 273–8.
  * [23] V. J. Shute, Stealth assessment in computer-based games to support learning, Computer games and instruction 55 (2) (2011) 503–524.
  * [24] Survey for 2019 American College of Surgeons Clinical Congress and the 2020 Academic Surgical Congress.
URL https://ufl.qualtrics.com/jfe/form/SV_b0WFv9hk3vl44fj


# Unlocking Structure Measuring: Introducing PDD,
an Automatic Metric for Positional Discourse Coherence

Yinhong Liu♠ Yixuan Su♠ Ehsan Shareghi♡♠ Nigel Collier♠
♠Language Technology Lab, University of Cambridge
♡Department of Data Science and AI, Monash University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Recent large language models (LLMs) have shown remarkable performance in
aligning generated text with user intentions across various tasks. When it
comes to long-form text generation, there has been a growing interest in
generation from a discourse coherence perspective. However, existing lexical
or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture
the discourse coherence. The development of discourse-specific automatic
evaluation methods for assessing the output of LLMs warrants greater focus and
exploration. In this paper, we present a novel automatic metric designed to
quantify the discourse divergence between two long-form articles. Extensive
experiments on three datasets from representative domains demonstrate that our
metric aligns more closely with human preferences and GPT-4 coherence
evaluation, outperforming existing evaluation methods. 111Our code is
available at https://github.com/williamLyh/pos_div_metric

Unlocking Structure Measuring: Introducing PDD,
an Automatic Metric for Positional Discourse Coherence

Yinhong Liu♠ Yixuan Su♠ Ehsan Shareghi♡♠ Nigel Collier♠ ♠Language Technology
Lab, University of Cambridge ♡Department of Data Science and AI, Monash
University<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Real-life texts often exhibit underlying structures. News articles, for
instance, adhere to a specific narrative order, as illustrated in Fig. 1,
employed by journalists to efficiently convey messages and improve reader
experience. Despite recent advances in generation of fluent text, the
generation of structurally coherent text remains an under-explored area.
Follow the theory of functional discourse structure, elaborated in Appendix
A.1, we leverage the discourse structure to model the coherence of long-form
texts. Several recent works (Spangher et al., 2022; Liu et al., 2022) have
addressed the problems of generating long-form text while following specific
in-domain discourse schema.

While established automatic metrics such as BLEU (Papineni et al., 2002),
ROUGE-L (Lin and Hovy, 2002) and BertScore (Zhang et al., 2019) exist for
Natural Language Generation evaluation, they predominantly measure lexical
n-gram overlaps or semantic similarities. The evaluation of structural
coherence has been a long-existing challenge (Guan et al., 2021; Cho et al.,
2019; Zhu and Bhat, 2020). A common baseline metric for measuring functional
discourse structure is the exact match, which compares structure elements one-
to-one at each exact position. However, this metric is notably sensitive to
local variations and differences in the lengths of articles.

Figure 1: A news article example with discourse role annotations. The
discourse schema follows the News discourse theory by Van Dijk (2013).

To address this gap, we propose a novel automatic model-free metric,
Positional Discourse Divergence (PDD), specifically designed to evaluate the
underlying discourse structure of articles in comparison to references. PDD
partitions the sentences of an article into multiple position bins and
calculates the divergence in discourse structures within each bin. This
approach renders PDD resilient to various challenges encountered in long-form
text generation, such as accommodating local variations and handling
misaligned numbers of sentences.

To validate the effectiveness and generalizability of the PDD, we evaluate the
inter-agreement with human evaluations and GPT-4 coherence evaluations on
three representative datasets with different discourse schema: News Discourse
(Choubey et al., 2020), Long-Form Question Answering (Xu et al., 2022a) and
Recipe1M+ (Liu et al., 2022). Across all three domains, PDD demonstrates the
highest agreement with human judgements on coherence.

##  2 Positional Discourse Divergence

Texts within a specific genre often exhibit similar patterns in their
discourse sequences, albeit with some variations at a local level. In other
words, the distribution of discourse roles is inherently tied to their
approximate positions within the articles. For instance, News reports commonly
present main events and their consequences at the beginning to capture the
reader’s interest, even though the precise order can differ. Likewise, recipes
tend to follow a predictable structure, where the preparation of ingredients
is generally mentioned first, followed by cooking actions towards the middle
or end of the text.

Despite the fluency achieved by (large) Language Models, they struggle to
organize discourse structures like humans. In Fig. 2, we observe disparities
between the discourse distributions of model predictions and human-written
references when the News articles are divided into 5 positional bins. To
quantitatively capture these gaps, we introduce the Positional Discourse
Divergence (PDD), denoted as $D_{pos}$, as an automatic metric. Equation 1
outlines the calculation for applying PDD to compare a _predicted_ article
against its corresponding _reference_ :

$D_{pos}=\frac{1}{N}\sum_{n=1}^{N}D_{KL}(p^{n}(r)+\bm{\epsilon}||q^{n}(r)+\bm{\epsilon})$
(1)

Firstly, both articles are segmented into $N$ positional bins. Note the number
of bin, $N$, should be smaller than the number of sentences in both the
reference and the prediction. We denote $p^{n}(r)$ to represent the
distribution of discourse role $r$ for the _reference_ in the $n$-th position
bin, and $q^{n}(r)$ to represent the distribution for the _generated_ article.
These discourse distributions are calculated by the frequency density of the
discourse roles within each bin. These discourse distributions are calculated
by the frequency density of the discourse roles within each bin. Then, for
each bin $n$, the KL divergence between the discourse distributions is
calculated. 222 Due to the asymmetry nature of KL divergence, $D_{KL}(P||Q)$
is interpreted as the information divergence of Q against P. Accordingly, we
employ $q^{n}(r)$ to denote the discourse distributions of the predictions and
$p^{n}(r)$ for the reference.  To address the sparsity in the discourse
distribution of a single article, small-value terms, denoted as
$\bm{\epsilon}$, are introduced. This addition helps in avoiding instances of
zero probabilities in the distribution.

To compute PDD or other discourse measurements like exact match, it is
inevitable to employ a discourse role classifier for labeling both prediction
and reference articles. An off-the-shelf discourse classifier, trained on
human-annotated data with a defined schema (e.g., the News Discourse dataset
for news domain), can serve this purpose. Further information regarding the
discourse classifiers is provided in Appendix B.

Figure 2: Positional discourse distribution comparisons (N=5). Top row: The
discourse distribution of model predictions on News Discourse test set
(Llama2-7b, finetuned on Kaggle All the News). Bottom row: Test set reference
distributions.

###  2.1 Interpreting the Metric

####  2.1.1 Set vs. Individual Predictions

Much like the BLEU score, the Positional Discourse Divergence can be applied
to a set of text, including both the set of model predictions and the set of
reference articles. The underlying assumption is that all articles within a
given set adhere to similar discourse structures, for example, being News
articles of the same sports genre. Consequently, the discourse distributions
of this set of articles offer a more accurate estimate of the discourse
distribution specific to that genre.

In the assessment of a single predicted article against a reference set, the
focus is on how well the article aligns with the target genre. In contrast,
when comparing a prediction set against a reference set, the evaluation exams
the model’s overall ability to generate content of that specific genre.

####  2.1.2 Bin Number

Figure 3: Positional Discourse Divergence vs. Bin number ($N$) for predictions
by two language models on the News Discourse test set. Training details in
Appendix C. Curves represent best-fit quadratic curves.

The bin number, $N$, plays a crucial role in determining the sensitivity of
PDD to local variations, as illustrated in Fig. 3. Therefore, the behavior of
PDD varies with the choice of $N$. Intuitively, a larger $N$ implies lower
tolerance for local perturbations. When $N$ equals the number of sentences,
the PDD is essentially equivalent to the exact match metric. Whereas, when $N$
equals 1, it describes the overall discourse role distribution gaps between
the prediction to the reference.

To illustrate, we fine-tuned Llama2-7b and Llama2-13b (Touvron et al., 2023)
and compared their predictions with the reference News articles. The details
of the supervised fine-tuning process are explained in Appendix C. The PDD
curves, illustrating the performance with different choices of bin numbers,
are presented in Fig. 3. The gaps in performance are effectively captured by
the disparities between the two PDD curves.

Metrics | Human |  | GPT-4
---|---|---|---
News Discourse | LFQA | Recipe1M+ |  | News Discourse | LFQA | Recipe1M+
Exact Match | 0.26 | 0.29 | 0.43 |  | 0.42 | 0.24 | 0.25
ROUGE-L | 0.30 | 0.24 | 0.39 |  | 0.44 | 0.19 | 0.26
BLEU | 0.24 | 0.31 | 0.46 |  | 0.48 | 0.28 | 0.32
BertScore | 0.45 | 0.42 | 0.63 |  | 0.63 | 0.42 | 0.68
PDD | 0.42 | 0.49 | 0.66 |  | 0.65 | 0.38 | 0.71
|  |  |  |  |  |  |

Table 1: Cohen’s Kappa with human and GPT-4 coherence evaluations. For News
Discourse, bin number $N=8$ was used, while for LFQA and Recipe1M+, $N=3$.
Human evaluation involves 50 randomly selected example pairs for each dataset,
while GPT-4 evaluation uses 300 pairs.

##  3 Metric Validation

To validate the efficacy of the Positional Discourse Divergence metric, we
evaluate its agreement with human assessments, and GPT-4 on article coherence.
Additionally, we compare PDD against baseline automatic metrics, such as exact
match, BLEU, and BertScore. To assess generalizability, we conduct this
validation across three different domains, each characterised by distinct
human annotated, sentence-level discourse schemas:

(I) News. We utilize the News Discourse dataset (Choubey et al., 2020),
comprising 802 documents across four genres and three media sources. The
average number of sentences per article is 14.6. Manual annotations for the
News Discourse dataset follow the theory of functional discourse schema
proposed by Van Dijk (1988, 2013). This schema defines discourse based on
eight types of relations between each sentence and the main event.

(II) Long-form QA. Long-Form Question Answering (LFQA) involves providing
comprehensive answers composed of multiple sentences. Xu et al. (2022a)
proposed an discourse ontology of six sentence-level functional roles also
following the theory of functional discourse structure. The discourse
annotations are collected on three recent LFQA datasets (ELI5 (Fan et al.,
2019), WebGPT (Nakano et al., 2022), and Natural Questions (Kwiatkowski et
al., 2019)). A total of 640 answer paragraphs were released, with an average
of 6.1 sentences per paragraph.

(III) Recipes. We adopt the discourse schema proposed by Liu et al. (2022)
which includes seven discourse roles based on cooking actions specifically
designed for recipes. They annotated the Recipe1M+ dataset (Moryossef et al.,
2019; Marín et al., 2021) with a rule-based annotation system following the
proposed schema. The Recipe1M+ contains over 1M textual recipes and
ingredients.

For further information regarding dataset details and schema definitions,
please refer to Appendix E.

###  3.1 Comparison with Other Metrics

We validate the effectiveness of our metric, PDD, by assessing its inter-
agreement with human evaluations and GPT-4 coherence evaluations. The human
evaluation setup details can be found in Appendix D. In our comparison, PDD is
evaluated alongside several automatic metrics, including exact match, BLEU
(Papineni et al., 2002), ROUGE-L (Lin and Hovy, 2002) and BertScore (Zhang et
al., 2019). Notably, only PDD and exact match focus on directly measuring
discourse structure, while the others are designed for assessing n-gram or
semantic similarity.

As rating long-form articles with absolute scores is a relatively complicated
and subjective task, we instead ask evaluators compare two perturbed
variations of the original reference article. Cohen’s Kappa is computed
between the metrics and evaluators based on these preference annotations. We
create two variations in the way that prevents resulting PDD values from
exhibiting a heavy left-tail issue and ensuring a more accurate kappa
estimation. In Variation 1, we randomly shuffle all the sentences, whereas in
Variation 2, we initially segment the article into a randomly selected number
of bins and then shuffle sentences only within their respective positional
bins.

In Tab. 1, we report the Kappa with both human and GPT-4 coherence
evaluations. The details of the prompt and survey templates are shown in
Appendix G. Our PDD metric demonstrates consistent good agreement (0.4-0.6) in
News Discourse and LFQA, achieving substantial agreement (>0.7) on Recipe1M+
dataset. The notable performance on the Recipe1M+ dataset can be attributed to
the strong order-dependent nature of recipes: A shuffled question-answer
format may be challenging to understand, but a disordered recipe is nearly
incomprehensible.

Another observation on News Discourse dataset, indicates Kappas with human
evaluations are generally lower than those with GPT-4. This discrepancy is
likely due to the much longer length of news articles compared to question
answering and recipe datasets, posing a more challenging task for human
readers.

Our PDD metric significantly outperforms baseline metrics of Exact Match,
Rouge-L and BLEU, while achieving comparable Kappa with the BertScore. We
attribute the good performance of BertScore to its ability in carrying textual
knowledge from the pre-trained BERT. In our experiment setup, both Variation 1
and 2 are shuffled from the same articles. Consequently, the metrics based on
n-gram and semantic similarities can effectively distinguish examples closer
to the original version and therefore achieve high kappa values. However, when
comparing the discourse structure between two different articles of the same
genre, they are likely to have very different n-gram or semantic content while
maintaining a similar discourse structure. In this case, only Exact Match and
PDD can capture the divergence between discourse structure.

###  3.2 Discussion

Our experimental findings yield the following noteworthy observations:

  * •

The behavior of PDD, as indicated by the formula in Eq. 1, converges towards
Exact Match as the chosen bin number increases. Conversely, with a smaller
value of $N$, PDD consistently outperforms Exact Match in terms of kappa. This
observation validates our initial hypothesis that permitting a certain level
of local variation does not detrimentally impact the overall reader
experience.

  * •

PDD consistently exhibits high kappa scores across diverse domains,
emphasizing the significance of preserving discourse structure in text across
various subject areas.

  * •

PDD is specifically designed to evaluate the underlying discourse structure.
It is not only simple and model-free, eschewing reliance on pre-trained
language models, but also interpretable because of its intrinsic use of KL
divergence.

##  4 Conclusion

In conclusion, the exploration of text generation with natural underlying
structure remains a significantly under-explored domain. Addressing this gap,
we introduced PDD, a simple and model-free metric designed to assess discourse
structure. By quantifying the divergence between discourse distributions
within position bins, PDD exhibits robust agreement with human and GPT-4
coherence evaluations across three representative domains, outperforming a
range of baseline metrics. Our hope is that PDD will stimulate future research
endeavors focused on unraveling the intricacies of underlying structure in
text generation.

## Limitations

##### Discourse classifier requirement

We note that our PDD requires a discourse classifier when applied to model
predictions. Although this necessity is inevitable in evaluating the discourse
structure alignment for any other metric such as Exact Match, it underscores
the dependence on annotated data with the target discourse schema for
training.

##### Choice of bin number $N$

The choice of bin number will affect the performance of the PDD. However, the
ideal choice of $N$ may vary for different articles: The optimal number of
sections the article should be segmented into. While trends may exist within
specific genres or datasets, in general, it requires certain level of domain
expertise to determine the optimal bin number.

## References

  * Beyer et al. (2021) Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021.  Is incoherence surprising? targeted evaluation of coherence prediction from language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4164–4173, Online. Association for Computational Linguistics.
  * Cho et al. (2019) Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, and Jianfeng Gao. 2019.  Towards coherent and cohesive long-form text generation.  In _Proceedings of the First Workshop on Narrative Understanding_ , pages 1–11, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Choubey et al. (2020) Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang, and Lu Wang. 2020.  Discourse as a function of event: Profiling discourse structure in news articles around the main event.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5374–5386, Online. Association for Computational Linguistics.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  ELI5: Long form question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
  * Grosz et al. (1995) Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995.  Centering: A framework for modeling the local coherence of discourse.  _Computational Linguistics_ , 21(2):203–225.
  * Guan et al. (2021) Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. 2021.  Long text generation by modeling sentence-level and discourse-level coherence.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6379–6393, Online. Association for Computational Linguistics.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Koto et al. (2021) Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021.  Discourse probing of pretrained language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3849–3864, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Laban et al. (2021) Philippe Laban, Luke Dai, Lucas Bandarkar, and Marti A. Hearst. 2021.  Can transformer models measure coherence in text: Re-thinking the shuffle test.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 1058–1064, Online. Association for Computational Linguistics.
  * Lavie and Agarwal (2007) Alon Lavie and Abhaya Agarwal. 2007.  METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 228–231, Prague, Czech Republic. Association for Computational Linguistics.
  * Liddy (1991) Elizabeth DuRoss Liddy. 1991.  The discourse-level structure of empirical abstracts: An exploratory study.  _Information Processing & Management_, 27(1):55–81.
  * Lin and Hovy (2002) Chin-Yew Lin and Eduard Hovy. 2002.  Manual and automatic evaluation of summaries.  In _Proceedings of the ACL-02 Workshop on Automatic Summarization_ , pages 45–51.
  * Liu et al. (2022) Yinhong Liu, Yixuan Su, Ehsan Shareghi, and Nigel Collier. 2022.  Plug-and-play recipe generation with content planning.  In _Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)_ , pages 223–234, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Mann and Thompson (1988) William C Mann and Sandra A Thompson. 1988.  Rhetorical structure theory: Toward a functional theory of text organization.  _Text-interdisciplinary Journal for the Study of Discourse_ , 8(3):243–281.
  * Marín et al. (2021) Javier Marín, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba. 2021.  Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(1):187–203.
  * Mizuta et al. (2006) Yoko Mizuta, Anna Korhonen, Tony Mullen, and Nigel Collier. 2006.  Zone analysis in biology articles as a basis for information extraction.  _International journal of medical informatics_ , 75(6):468–487.
  * Moryossef et al. (2019) Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019.  Step-by-step: Separating planning from realization in neural data-to-text generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2267–2277, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Nakano et al. (2022) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022.  Webgpt: Browser-assisted question-answering with human feedback.
  * Ng and Abrecht (2015) Jun-Ping Ng and Viktoria Abrecht. 2015.  Better summarization evaluation with word embeddings for ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Prasad et al. (2008) Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008.  The penn discourse treebank 2.0.  In _LREC_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Spangher et al. (2022) Alexander Spangher, Yao Ming, Xinyu Hua, and Nanyun Peng. 2022.  Sequentially controlled text generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6848–6866, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Van Dijk (1988) Teun A Van Dijk. 1988.  News analysis.  _Case Studies of International and National News in the Press. New Jersey: Lawrence_.
  * Van Dijk (2013) Teun A Van Dijk. 2013.  _News as discourse_.  Routledge.
  * Xu et al. (2022a) Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. 2022a.  How do we answer complex questions: Discourse structure of long-form answers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3556–3572, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2022b) Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. 2022b.  How do we answer complex questions: Discourse structure of long-form answers.  _arXiv preprint arXiv:2203.11048_.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34:27263–27277.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.
  * Zhao et al. (2023) Wei Zhao, Michael Strube, and Steffen Eger. 2023.  DiscoScore: Evaluating text generation with BERT and discourse coherence.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3865–3883, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Zhu and Bhat (2020) Wanzheng Zhu and Suma Bhat. 2020.  GRUEN for evaluating linguistic quality of generated text.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 94–108, Online. Association for Computational Linguistics.

##  Appendix A Background and Related Works

###  A.1 Discourse Structure

Discourse structure investigates the organization of language into larger
units like paragraphs, sections, and complete articles. In this work, we focus
on the communicative functions within entire articles served by those
linguistic units. Therefore, texts from different domains are characterized by
different discourse schemas, as their linguistic units also play different
functional roles. The discourse roles of scientific papers or experimental
abstracts (Liddy, 1991; Mizuta et al., 2006) include background, methodology,
experiments and findings. In the domain of long-form question answering Xu et
al. (2022b), the discourse function of each sentence can be answer, summary,
example and so on. Liu et al. (2022) developed a discourse schema for recipes
based on actions and controlled the generation process according to the
predicted discourse sequences. The explicit functional discourse structure of
news reports was addressed (Van Dijk, 2013; Choubey et al., 2020) by defining
roles based on their relations with the main event, such as consequence and
journalist evaluation.

Multiple established frameworks also proposed different definition of
discourse structure, which focus on how each linguistic unit relates to each
other through discourse connectives, such as causal, temporal, etc. For
instance, Rhetorical Structure Theory, RST (Mann and Thompson, 1988), seeks to
identify rhetorical relations between text segments and form a hierarchical
organization of discourse. The Penn Discourse Treebank, PDTB (Prasad et al.,
2008), defines its schema based on low-level discourse connectives presented
in the text.

###  A.2 NLG Metrics

Traditional NLG metrics, such as BLEU (Papineni et al., 2002) and ROUGE-L (Lin
and Hovy, 2002), measure lexical n-gram overlaps to assess fluency, but they
have limitations in capturing semantic similarity. Later works tried to
improve the hard lexical matching with soft word embedding matching (Ng and
Abrecht, 2015) or stemming and synonym matching (Lavie and Agarwal, 2007).

Recently, by leveraging contextual embeddings from BERT (Kenton and Toutanova,
2019), a series of metrics can successfully capture semantic similarity with
references or even textual quality without references(Zhao et al., 2019; Zhang
et al., 2019; Yuan et al., 2021). However, as BERT is argued that can only
capture limited discourse information (Koto et al., 2021; Laban et al., 2021;
Beyer et al., 2021), they are not suitable for evaluating the discourse
structure in long texts.

DiscoScore (Zhao et al., 2023) is a BERT-based metric, specifically designed
to model local discourse coherence for summarization and document-level
machine translation tasks. By leveraging Center theory (Grosz et al., 1995),
they modelled discourse similarity by focus frequency and transitions.

##  Appendix B Discourse Classifier

A discourse classifier is usually a lightweight language model trained on
sentence-discourse role pairs. Here we report the classifier performance
achieved:

For the News domain, we train a DistilBERT (Sanh et al., 2019) as the
discourse role classifier on the News Discourse training set and evaluated on
the validation set using human-annotated gold labels. The classifier achieves
an accuracy of $67\%$.

In the Recipe domain, the reported performance of the discourse role
classifier, by Liu et al. (2022), achieves an accuracy of $92\%$. It was a a
DistilBERT model trained on the Recipe1M+ training set and evaluated on the
validation set using silver annotations generated by a rule-based system.

For LFQA, Xu et al. (2022a) achieved an accuracy of $54\%$ by a T5-large
model, which shows comparable performances to human. The classifier was
trained and tested on the ELI5 dataset.

##  Appendix C LLM SFT Details

We fine-tuned two language models, the 8-bit LoRa versions of Llama2-7b and
Llama2-13b, using Kaggle All the News train set comprising 42.4K samples after
filtering. The models receive news headlines as input and aim to generate the
corresponding news articles. Training employed consistent hyper-parameters: a
learning rate of $3\times 10^{-4}$, 2 epochs, LoRa parameters $r=8$,
$\alpha=16$, and dropout set at $0.05$. The models were trained on a single
RTX a6000 48GB, requiring 12 and 23 hours for Llama2-7b and Llama2-13b,
respectively.

##  Appendix D Human Evaluation Details

The human evaluation was conducted using Amazon Mechanical Turk (MTurk). We
obtained three preference annotations for each example pair from native
English-speaking crowd workers. The final results were determined based on the
majority preference among the three evaluations. Crowd workers were
compensated at a rate of 15 pounds per hour for their participation in the
evaluation process.

##  Appendix E Discourse Schema

The definition of the discourse schema we used for news articles:

  * •

Main Event: The major subject of the news article.

  * •

Consequence: An event or phenomenon that is caused by the main event.

  * •

Previous Event: A specific event that occurred shortly before the main event.

  * •

Current Context: The general context or world state immediately preceding the
main event.

  * •

Historical Event: An event occurring much earlier than the main event.

  * •

Future Consequences: An analytical insight into future consequences or
projections.

  * •

Journalist Evaluation: A summary, opinion or comment made by the journalist.

  * •

Anecdotal Event: An event that is uncertain and cannot be verified. The
primary purpose is to provide more emotional resonance to the main event.

The definition of the discourse schema for LFQA:

  * •

Organizational sentence: An organizational sentence is to inform the reader
how the answer will be structured.

  * •

Answer summary: An answer sentence that plays a summary role, which can often
suffice by themselves as the answer to the question.

  * •

Answer: Answer sentences which explain or elaborate on the summary.

  * •

Example: The example provided in answers, which discussed a particular entity
or concept that is different from the rest of the answer sentences.

  * •

Auxiliary Information: Provide information that are related, but not directly
asked in the question.

  * •

Miscellaneous: Various other roles that shows up in human answers, such as the
limitation of the answer or the source of the answer.

The definition of the discourse schema we used for recipes:

  * •

Pre-processing means the preparations of ingredients or cooker.

  * •

Mixing includes actions of combining one or more ingredients together.

  * •

Transferring is for the actions of moving or transferring food or intermediate
food to a specific place.

  * •

Cooking represents the actual cooking actions, which could vary drastically
across different recipes.

  * •

Post-processing usually refers to the following up actions after the ‘cooking’
stage, such as ‘cooling down’, ‘garnish’.

  * •

Final refers to the last few actions before serving the food or the serving
action itself.

  * •

General includes the rest of actions which cannot be classified into the above
categories.

##  Appendix F Data Preprocessing

For News Discourse, we filtered the dataset based on the following conditions:

  * •

Containing special characters: @, [, +.

  * •

Having total number of words over 800 or below 100.

  * •

Containing random comments.

  * •

Containing more than two reports.

Then we pre-process the data by

  * •

Removing extra space.

  * •

Removing reporting source.

  * •

Removing journalist names.

  * •

Removing emoji.

For Recipe1M+, we filter it based on the following codintions:

  * •

Containing irrelevant information, such as advertisements, reviews and
comments.

  * •

Having total number of words over 300 or below 50.

  * •

Duplicate recipes.

For LFQA, we filtered out all model generated answer paragraphs, because they
contain sentences that do not have assigned discourse roles.

##  Appendix G Evaluation Templates

##### Human evaluation instruction

Below, we provide the instruction example for human evaluation on the News
Discourse dataset, where evaluators were directed to express their preference.
The instructions for LFQA and Recipe1M+ are similar, with certain domain-
specific keywords substituted, such as News headline becoming Recipe title.

“ Read the two versions of the news for the given headline and rank their
coherence following the guideline below. Coherence guidelines: 1\. Flow of
Sentences: Evaluate how well the sentences transition from one to another. A
fluent text should have seamless connections between sentences. 2\. Logical
Organization: Evaluate how well the sentences are organized and the ideas are
conveyed. A coherent text should have a clear and precise structure. General
guidelines: 1\. Be Objective: Please focus on the coherence of writing, not
the content or opinions expressed. 2\. Please rate which one is preferred
between the two versions. News headline: {headline} Version 1: {version1}
Version 2: {version2} ”

##### GPT-4 evaluation prompt

Below, we provide the prompt template for GPT-4 coherence evaluation on the
News Discourse dataset. Although the GPT-4 was instructed to rate with scores,
but the scores are converted to preference later. The instructions for LFQA
and Recipe1M+ are similar, with certain domain-specific keywords substituted,
such as News headline becoming Recipe title.

“ Pretend you are a human reader. Please evaluate the coherence of the two
given news articles. Guideline: 1\. Rate on a scale of 1 to 10, where 1
represents very low coherence, and 10 indicates very high coherence. 2\.
Consider the flow of ideas and the ordering of sentences. A highly coherent
article should have a better sentence ordering. 3\. Must return ratings in
JSON format only: {"score1": [your rating for version 1], "score2": [your
rating for version 2]} News headline: [ headline ] News version 1: [ version1
] News version 2: [ version2 ] Rating: ”