SF2A 2021

# Science with SKA

F. Combes  Observatoire de Paris, LERMA, Collège de France, CNRS, PSL
University, Sorbonne University, 75014, Paris, France

###### Abstract

Highlights are presented about the science to be done with SKA. as well as
state of the art science already done today with its precursors (MeerKAT,
ASKAP) and pathfinders (LOFAR, NenuFAR), with accent on the expected
breakthroughs.

###### keywords:

SKA, galaxies, cosmology, pulsars, pre-biotic, molecules

##  1 The key scientific projects of SKA

Some of the main puzzles of cosmology are the nature of dark matter and dark
energy, representing in total 95% of the content of the Universe. The dark
energy is presently compatible within the uncertainties with a cosmological
constant, but it is paramount to determine with greater precision whether its
evolution with time is dynamic, and could be due to a fifth element, a
quintessence, and new physics. The tools for making this diagnostic are the
same as for many other dark energy probes, either from ithe ground or in
space, with Euclid: the BAO (Baryon Acoustic Oscillations), playing the role
of a standard ruler, measuring the expansion at differnt redshifts, the Weak
Lensing (WL), or Redshift Space Distorsions (RSD), measuring the density and
amplitude of large-scale structures to constrain the evolution of $\Omega$ and
$\Lambda$. These tools will be exploited with optical tracers, and the novelty
of SKA is to use radio tracers, and the HI-21cm line to identify galaxies.
These tracers have different biases than the optical ones, and both studies
are very complementary. Optically, the massive galaxies are early-type
gathered in galaxy clusters, while the HI-rich galaxies are late-type in the
field.

Another key project is to explore the Epoch of Reionization (EoR), likely to
extend from z=20 to 6. If it is already possible to have some clues with
present searches of galaxies and quasars at z$>$6, the inter-galactic medium
will be uniquely explored by SKA, with the redshifted 21cm-HI line, as it is
now with pathfinders and precursors. The large galaxy surveys made over the
whole available sky, due to the wide field of view, will serve to determine
uniquely the large-scale structures, and the galaxy formation and evolution.

Pulsars will be discovered in a huge number with SKA, exploring the whole
Milky Way, while presently they are confined in the solar neighborhood. Milli-
second pulsars are extremely precise clocks, which can be used to detect very
long wavelength gravitational waves. Strong-gravity will be explored with
pulsars and black holes

Cosmic magnetism is another key project, and in particular the formation of
primordial magnetic fields will be tackled. Finally, the search for the origin
of life, the mapping of the protoplanetary disks, and the search for pre-
biotic molecules, will be carried out in synergy and complementarity with ALMA
at higher frequencies.

All key projects with SKA have been developed in many whitepapers and
conferences, (e.g., Carilli & Rawlings 2004; Carilli 2015).

##  2 Cosmology and galaxies

###  2.1 Dark sector and new physics

The state of the art constraints on the dark energy and the dark matter are
obtained by combining all available data, from the SNIa standard candels, i.e.
the Pantheon sample of 1048 SNIa between redshifts 0.01 $<$ z $<$ 2.3 (Scolnic
et al. 2018), and the 207 SN sample from DES-3yr (Abbott et al. 2019), with
the BAO results from SDSS (Alam et al. 2021), and the CMB data (Planck
Collaboration et al. 2020). The equation of state of the dark energy can be
written as the pressure proportional to the density P = w $\rho$, with w
negative, and the variation of w(a)= w0 \+ wa (1-a), where a is the
characteristic radius of the Universe, a=1/(1+z). Since SNIa are difficult to
observe at z$>$1, Inserra et al. (2021) propose to use and calibrate
superluminous supernovae (SLSNe), which will allow to go farther and faster.
The first results are promising, putting constraints in the w0-wa diagram.
With the enhanced precision acquired in the recent years, some tension has
grown between observations and the standard $\Lambda$CDM model, suggesting the
necessity of new physics (Smith et al. 2020). In particular the main tension
occurs between the Hubble constant H0= 73.48$\pm$1.66 km/s/Mpc measured
locally with Cepheids or other indicators (Riess et al. 2018), and the Planck
determination of 67.4$\pm$ 0.5 km/s/Mpc. The discrepancy reaches 3.7$\sigma$.
In radioastronomy, powerful masers (H2O, OH..) allow to observe in VLBI the
center of external galaxies, and their rotating circum-nuclear disk; measuring
the velocities through the Doppler effect and monitoring through VLBI the
gradient of maser poition with velocity, results in a precise distance
indicator, as shown beautifully by the prototypical example of NGC 4258
(Greenhill et al. 1995). SKA will measure many more masers around AGN at
various redshifts, and can give a complementary approach to the problem.
Already Pesce et al. (2020) with megamasers confirm H0 = 73.9$\pm$3 km/s/Mpc.

The most recent BAO and RSD results, including 147 000 quasars (Ata et al.
2018), and Ly$\alpha$ absorption surveys (Bautista et al. 2017), are
compatible with a flat $\Lambda$CDM cosmology, and perfectly compatible with
the Planck cosmological parameters (Bautista et al. 2021).

SKA-1 surveys of galaxies in the HI-21cm lines will be complementary and
competitive with the optical ones from the ground and in space (Euclid).
Surveys where galaxies are detected individually will be most useful for
galaxy formation and evolution, they will detect 4 million galaxies up to
z=0.2 in the all-sky survey, 2 million galaxies up to z=0.6 in the wide field,
and 0.4 million in the deep field survey up to z=0.8 (of 50 square degrees
area). For cosmology purposes, HI intensity mapping over 30 000 square
degrees, and covering redshifts up to 3 will be more competitive (Maartens et
al. 2015). Weak lensing in radio surveys up to redshift z=6 will consider a
billion objects. One of the strong advantages of SKA-1 is the much larger
volumes sampled, with respect to all other probes (Euclid, DESI, BOSS, Nancy-
Grace-Roman…). The second phase SKA-2 will surpass all.

Figure 1: Left: Radio continuum emission from ESO 137-006 detected by MeerKAT
at 1030 MHz. Three collimated synchrotron threads (CSTs) between the radio
lobes are indicated. The Sobel filter has been applied to the image, to better
show these features (Ramatsoku et al. 2020). The upper panel shows that the
galaxy is entering the Norma cluster, and its X-ray gas atmosphere (in red)..
Right: Some radio images of nearby galaxies from the LOFAR Two-metre Sky
Survey (LoTSS) (Shimwell et al. 2019).

###  2.2 Continuum and HI surveys

Simulations have been performed of wide-field images of the radio-continuum
sky with SKA, detecting both the very numerous star-forming galaxies, with
synchrtron emission coming from supernovae, and the stronger but less numerous
radio AGN, of FRI and FRII types, the latter even less numerous but stronger
(Jackson 2004). The AGN radio jets can be used easily as standard rods,
constraining the cosmological parameters, by themsleves, and also through weak
lensing.

For the all-sky survey at 1.4 GHz, in 2yrs of integration, SKA1 will achieve 3
$\mu$Jy rms, and detect $\sim$4 galaxies per arcmin2 (at more than
10$\sigma$), (Jarvis et al. 2015). The survey will be made with an excellent
quality circular Gaussian beam from about 0.6 to 100”, With almost uniform sky
coverage of 3$\pi$ str. This will provide a total of 0.5 billion radio
sources, yielding weak lensing and Integrated Sachs Wolfe (WL, ISW)
diagnostics. For the wide-field (5000 deg2), with 2 $\mu$Jy rms $\sim$6
galaxies per arcmin2 are expected (at more than 10$\sigma$). For the deep-
field (50deg2) with 0.1 $\mu$Jy rms, $\sim$20 galaxies per arcmin2 will be
detected, at more than 10$\sigma$.

Figure 1 shows some examples of radio images from the LOFAR Two-metre Sky
Survey (LoTSS), and also how the precursor MeerKAT has discovered new features
in typical radio-jets: collimated synchrotron threads, linking the radio lobes
from the sides, in parallel to the radio jets, in ESO 137-006. This galaxy is
moving inside the wind of the intra-cluster gas, entering the Norma cluster.
The radio lobes are distorted and bent, and the threads look like relics of
the previous radio jets, in previous episodes of ejection.

In HI-21cm surveys, SKA-1 will allow the imaging of substantial number of
high-redshift galaxies for the first time (Staveley-Smith & Oosterloo 2015).
While the present instruments are restricted to detect HI in individual
galaxies only to the local Universe up to z=0.1, the very deep survey will
permit the detection of galaxies at z=2, and even higher for SKA-2. A glance
of what intensities could be detected is given by recent stacking to detect
only ”globally” some remote galaxies. With GMRT deep (117h) field, Bera et al.
(2019) have stacked 445 blue galaxies between 0.2 $<$ z $<$ 0.4, and obtained
a detection at 7$\sigma$ of M(HI) = 5 10 9 M⊙. Stacking the continuum to
derive the star formation rate, they derive a depletion time of $\sim$ 9 Gyrs.
From GAMA survey, imaged on DINGO-VLA, Chen et al. (2021) have stacked HI
cubelets on a sample of 3622 galaxies, and obtained a clear detection, with
FWHM of 60km/s.

##  3 Reionization

Intensity mapping is the only technique able to determine the global
quantities searched for in the EoR. Continuum foregrounds are typically 1000
times brighter than the expected cosmological signal. The instrumental
responses to bright foregrounds with extended and multiple sidelobes, forming
a sea of confused signals, depending on their location on the field of view,
are a challenge to understand and subtract away (Santos et al. 2015). The
foregrounds to be eliminated produce a perturbing signal, which is not
necessarily spectrally smooth (Switzer et al. 2015).

The LOFAR key project on EoR has observed more than 1000hours on selected
clean fields, with the least possible foreground emission. However, even if
the nominal sensitivity is reached to detect easily the expected signal, the
confusion by foregrounds has prevented to draw any conclusion. Controlling the
calibration, and cleaning for the sidelobes down to the low intensity level
required is a long process. While in 2017, only 0.5% of data were understood
and used (Patil et al. 2017), recently up to 5% of data has been understood
and cleaned, resulting in an upper limit of the EoR signal two orders of
magnitude above the expected signal. There still remains noise that could be
due to residual emission from foreground sources or diffuse emission far away
from the phase centre, polarization leakage, chromatic calibration errors,
ionosphere, or low-level radiofrequency interference (Mertens et al. 2020)

##  4 Pulsars, Cosmic magnetism

###  4.1 Pulsars and gravitational waves

The large number of pulsars to be discovered by SKA, in combination with its
exceptional timing precision, will revolutionize the field of pulsar
astrophysics. SKA will provide a complete census of pulsars in both the Galaxy
and in Galactic globular clusters (Cordes et al. 2004). In the Milky Way,
about 30 000 pulsars should be present, and 10 000 milli-second ones. May be
20 000 pulsars will be detectable in the whole Galaxy (while today we know
only pulsars in the solar neighborhood). Pulsars and compact objects will
allow unique tests of the strong field limit of relativistic gravity and the
equation of state at extreme densities.

Through monitoring these pulsars, which are extremely precise clocks, (up to
10-15 in relative), gravitational waves of long wavelengths, of the order of
light-yrs, could be detected, and pulsars are the only way. The hope is to
detect the merger of super-massive black holes, and also the primordial waves,
signature of inflation (Janssen et al. 2015). A first preliminary detection
has been claimed with present telescopes (Arzoumanian et al. 2020).

###  4.2 Fast Radio Bursts

Since their discovery by Lorimer et al. (2007), our knowledge of Fast Radio
Bursts (FRB) have grown considerably: they have been detected in large
numbers, especially with the wide-field instrument CHIME: 540 are known, and
the occurence has been estimated at 800 per day on the whole sky (CHIME/FRB
Collaboration et al. 2019). With SKA-MID, it will be possible to detect 100
FRB/yr, with precise localisation (Keane 2018). The nature of the FRB
phenomenon is not yet clarified, although many repeaters have been detected,
and one FRB has been associated with a Galactic magnetar, SGR 1935+2154
(Bochenek et al. 2020). Due to their surface magnetic fields larger than 1014
gauss, magnetars are the source of high-energy phenomena, where their magnetic
energy decays. A florilege of theories have been proposed to explain the
phenomena (Platts et al. 2019).

Figure 2: Stacking radio and X-ray maps, around physically nearby pairs of
luminous red galaxies (LRG). From left to right the columns are GLEAM 154 MHz,
118 MHz, 88 MHz, OVRO-LWA 73 MHz, and ROSAT combined band 0.1 -2.4 keV. The
colour bars all have units of temperature in K, except the ROSAT maps which
are in counts per second per arcmin2 x104. From Vernstrom et al. (2021).

###  4.3 Magnetic Fields

Polarisation of radio emission, and Faraday rotation have been used
intensively to determine the intensity and orientation of the magnetic field
in spiral galaxies, at various depths according to the various wavelengths:
either from the halo, or the disk, and different distributions have been
obtained with respect to the spiral arms (Kierdorf et al. 2020). Turbulence
due to star formation in spiral arms paradoxically reduces alignment, and
frequent field reversals in the vertical direction contribute to distortions
that are not yet well understood. LOFAR has been used in combination with VLA
to determine the spectrum of the emission, separate thermal and non-thermal
components, the magnetic field strength and the cosmic ray electron losses
(Mulcahy et al. 2018).

All-sky survey of Faraday rotation will measure inter-galactic magnetic field,
as well as inside galaxies. The mechanisms to generate the field are not yet
settled, from inflation, phase transitions in the early Universe, and
batteries to amplify the seeds. Normally the field is frozen in the ionized
gas, but should dilute away in the expansion. When structures collapse, the
field is amplified again (Johnston-Hollitt et al. 2015).

Searches have been done in diffuse filaments connecting clusters, at the
cosmic web (15Mpc) scale, combining X-ray hot gas with eRosita with radio data
from ASKAP/EMU Early Science (Reiprich et al. 2021). Missing baryons are
searched for by studying the warm-hot gas in cluster outskirts and filaments.
The bridge between two clusters is detected; it may contain known galaxy
groupis, but not accounting for all the emission There are several clumps of
warm gas falling into the clusters, compatible to what is observed in
simulations.

LOFAR has also detected synchrotron emission in filaments between merging
galaxies, with possible shocks re-accelerating the electrons (Govoni et al.
2019; Botteon et al. 2020), but these were only short scales. Now with GLEAM
(the MWA survey), it is possible to search for longer filaments (see Figure
2). These are traced by Luminous red galaxies (LRGs), which are massive early-
types residing in the center of galaxies clusters or groups. The first large-
scale filament detection, has revealed a magnetic field of 30-60 nG, of
intensity higher than previously believed, with electrons subject to more
efficient shock acceleration (Vernstrom et al. 2021).

Figure 3: Cartoon of a protoplanetary disk (HD 100546) showing the regions
where CH3OH is detected and the different physical and chemical mechanisms
proposed (Booth et al. 2021).

##  5 Craddle for life

ALMA has made a breakthrough in the domain of the formation of planets and
protoplanetary disks, in imaging with superb resolution resonant rings and
gaps (Andrews 2020). Disks are formed first with gas and small-size dust
grains, the latter agglomerating progressively in mm- and cm-sized grains
before becoming planetesimals. These grains emit at longer and longer
wavelengths, and SKA-1 will be the prefered instrument to detect cm to m-sized
dust. At high resolution with 40mas beam, the nearest systems will be mapped
with 4AU resolution, sufficient to determine the snow line (Hoare et al.
2015). Large exoplanets, of Jupiter-size, could be detected with their
magnetic fields (Zarka et al. 2018). In synergy with ALMA, the detection of
complex organic molecules (COM) could be carried out, such as the methanol
CH3OH, with deuterated species CH2DOH, methanethiol CH3SH, formamide NH2CHO,
and heavier pre-biotic molecules in Band 5, such as amino acids and sugars. In
1000h SKA1-mid could detect clearly $\alpha$-alanine, with a hundred of lines,
for a column density of 1013 cm-2 (Hoare et al. 2015).

Methanol has been detected in protoplanetary disks, and is thought to come
from hydrogenation of CO on icy grains (cf Figure 3). Complex organic
molecules have now been detected, which are key to form amino-acids, and pre-
biotic molecules (Booth et al. 2021). These COM cannot form in situ, but must
come from ices formed previously in dark interstellar clouds.

##  6 Conclusions

SKA-1 will help to tackle the main puzzles of cosmology: the nature of dark
matter and dark energy, by using the common tools (BAO, WL, RSD) with high
precision, and with tracers with different biases than optical surveys (radio
continuum, HI in galaxies). With extragalactic masers, it will be possible to
bring a complementary constraint to the H0 problem, suggesting new physics.

With redshifted HI-21cm line, SKA will have a unique contribution to the Epoch
of Reionization, and the birth of the first galaxies. With the timing of
millisecond pulsars, SKA will make a breakthrough in probing strong ggavity,
and detecting gravitational waves in the very low frequency regime (nanoHz),
to search for primordial waves.

Our knowledge of magnetic genesis will considerably improve. SKA will work in
synergy with ALMA to determine the physics of protoplanetary disks, and detect
pre-biotic molecules.

All these key projects have begun to be tackled at very low frequency with the
NenuFAR pathfinder in Nançay, where the first large programs are ES1: Cosmic
Dawn; ES2: Exoplanets & Stars; ES3: Pulsars; ES4: Transients; ES5: Fast Radio
Bursts; ES6: Planetary Lightning; ES7: Joint Jupiter studies; ES8: Cluster of
galaxies & AGNs; and ES9: Cluster Filament & Cosmic Magnetism.

## References

  * Abbott et al. (2019) Abbott, T. M. C., Allam, S., Andersen, P., et al. 2019, ApJ, 872, L30
  * Alam et al. (2021) Alam, S., Aubert, M., Avila, S., et al. 2021, Phys. Rev. D, 103, 083533
  * Andrews (2020) Andrews, S. M. 2020, ARA&A, 58, 483
  * Arzoumanian et al. (2020) Arzoumanian, Z., Baker, P. T., Blumer, H., et al. 2020, ApJ, 905, L34
  * Ata et al. (2018) Ata, M., Baumgarten, F., Bautista, J., et al. 2018, MNRAS, 473, 4773
  * Bautista et al. (2017) Bautista, J. E., Busca, N. G., Guy, J., et al. 2017, A&A, 603, A12
  * Bautista et al. (2021) Bautista, J. E., Paviot, R., Vargas Magaña, M., et al. 2021, MNRAS, 500, 736
  * Bera et al. (2019) Bera, A., Kanekar, N., Chengalur, J. N., & Bagla, J. S. 2019, ApJ, 882, L7
  * Bochenek et al. (2020) Bochenek, C. D., Ravi, V., Belov, K. V., et al. 2020, Nature, 587, 59
  * Booth et al. (2021) Booth, A. S., Walsh, C., Terwisscha van Scheltinga, J., et al. 2021, Nature Astronomy
  * Botteon et al. (2020) Botteon, A., van Weeren, R. J., Brunetti, G., et al. 2020, MNRAS, 499, L11
  * Carilli (2015) Carilli, C. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 171
  * Carilli & Rawlings (2004) Carilli, C. L. & Rawlings, S. 2004, New A Rev., 48, 979
  * Chen et al. (2021) Chen, Q., Meyer, M., Popping, A., & Staveley-Smith, L. 2021, MNRAS, 502, 2308
  * CHIME/FRB Collaboration et al. (2019) CHIME/FRB Collaboration, Amiri, M., Bandura, K., et al. 2019, Nature, 566, 230
  * Cordes et al. (2004) Cordes, J. M., Kramer, M., Lazio, T. J. W., et al. 2004, New A Rev., 48, 1413
  * Govoni et al. (2019) Govoni, F., Orrù, E., Bonafede, A., et al. 2019, Science, 364, 981
  * Greenhill et al. (1995) Greenhill, L. J., Jiang, D. R., Moran, J. M., et al. 1995, ApJ, 440, 619
  * Hoare et al. (2015) Hoare, M., Perez, L., Bourke, T. L., et al. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 115
  * Inserra et al. (2021) Inserra, C., Sullivan, M., Angus, C. R., et al. 2021, MNRAS, 504, 2535
  * Jackson (2004) Jackson, J. C. 2004, J. Cosmology Astropart. Phys, 2004, 007
  * Janssen et al. (2015) Janssen, G., Hobbs, G., McLaughlin, M., et al. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 37
  * Jarvis et al. (2015) Jarvis, M., Bacon, D., Blake, C., et al. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 18
  * Johnston-Hollitt et al. (2015) Johnston-Hollitt, M., Govoni, F., Beck, R., et al. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 92
  * Keane (2018) Keane, E. F. 2018, Nature Astronomy, 2, 865
  * Kierdorf et al. (2020) Kierdorf, M., Mao, S. A., Beck, R., et al. 2020, A&A, 642, A118
  * Lorimer et al. (2007) Lorimer, D. R., Bailes, M., McLaughlin, M. A., Narkevic, D. J., & Crawford, F. 2007, Science, 318, 777
  * Maartens et al. (2015) Maartens, R., Abdalla, F. B., Jarvis, M., & Santos, M. G. 2015, arXiv e-prints, arXiv:1501.04076
  * Mertens et al. (2020) Mertens, F. G., Mevius, M., Koopmans, L. V. E., et al. 2020, MNRAS, 493, 1662
  * Mulcahy et al. (2018) Mulcahy, D. D., Horneffer, A., Beck, R., et al. 2018, A&A, 615, A98
  * Patil et al. (2017) Patil, A. H., Yatawatta, S., Koopmans, L. V. E., et al. 2017, ApJ, 838, 65
  * Pesce et al. (2020) Pesce, D. W., Braatz, J. A., Reid, M. J., et al. 2020, ApJ, 891, L1
  * Planck Collaboration et al. (2020) Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020, A&A, 641, A6
  * Platts et al. (2019) Platts, E., Weltman, A., Walters, A., et al. 2019, Phys. Rep., 821, 1
  * Ramatsoku et al. (2020) Ramatsoku, M., Murgia, M., Vacca, V., et al. 2020, A&A, 636, L1
  * Reiprich et al. (2021) Reiprich, T. H., Veronica, A., Pacaud, F., et al. 2021, A&A, 647, A2
  * Riess et al. (2018) Riess, A. G., Casertano, S., Yuan, W., et al. 2018, ApJ, 861, 126
  * Santos et al. (2015) Santos, M., Bull, P., Alonso, D., et al. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 19
  * Scolnic et al. (2018) Scolnic, D. M., Jones, D. O., Rest, A., et al. 2018, ApJ, 859, 101
  * Shimwell et al. (2019) Shimwell, T. W., Tasse, C., Hardcastle, M. J., et al. 2019, A&A, 622, A1
  * Smith et al. (2020) Smith, T. L., Poulin, V., & Amin, M. A. 2020, Phys. Rev. D, 101, 063523
  * Staveley-Smith & Oosterloo (2015) Staveley-Smith, L. & Oosterloo, T. 2015, in Advancing Astrophysics with the Square Kilometre Array (AASKA14), 167
  * Switzer et al. (2015) Switzer, E. R., Chang, T. C., Masui, K. W., Pen, U. L., & Voytek, T. C. 2015, ApJ, 815, 51
  * Vernstrom et al. (2021) Vernstrom, T., Heald, G., Vazza, F., et al. 2021, MNRAS, 505, 4178
  * Zarka et al. (2018) Zarka, P., Marques, M. S., Louis, C., et al. 2018, A&A, 618, A84


# Computing and Exploiting Document Structure to Improve Unsupervised
Extractive Summarization of Legal Case Decisions

Yang Zhong
University of Pittsburgh
Pittsburgh, PA, USA
<EMAIL_ADDRESS>
Diane Litman
University of Pittsburgh
Pittsburgh, PA, USA
<EMAIL_ADDRESS>

###### Abstract

Though many algorithms can be used to automatically summarize legal case
decisions, most fail to incorporate domain knowledge about how important
sentences in a legal decision relate to a representation of its document
structure. For example, analysis of a legal case summarization dataset
demonstrates that sentences serving different types of argumentative roles in
the decision appear in different sections of the document. In this work, we
propose an unsupervised graph-based ranking model that uses a reweighting
algorithm to exploit properties of the document structure of legal case
decisions. We also explore the impact of using different methods to compute
the document structure. Results on the Canadian Legal Case Law dataset show
that our proposed method outperforms several strong baselines.

##  1 Introduction

Single document summarization aims at rephrasing a long text into a shorter
version while preserving the important information Nenkova and McKeown (2011).
While recent years have witnessed a blooming of abstractive summarization
models that can generate fluent and coherent new wordings Rush et al. (2015);
Zhang et al. (2020b); Lewis et al. (2020), abstractive summaries often contain
hallucinated facts Kryscinski et al. (2019). In contrast, extractive
summarization models directly select sentences/phrases from the source
document to form a summary. In certain domains such as the law or science
Bhattacharya et al. (2019); Dong et al. (2021), using exact wordings may be
needed.

In this work, we focus on extractive summarization of legal case decisions.
Different from texts in the news domain, case texts tend to be longer (e.g.,
in Canadian legal case decisions Xu et al. (2021) there are on average 3.9k
words, while standard news articles Nallapati et al. (2016) range from 400 -
800 words) and also have more complicated document structures (e.g., legal
cases are likely to be split into sections while news articles are not). In
contrast to scientific domains, which also have long and structured texts,
large training sets of case decisions and reference summaries are generally
not freely available given the restrictions of the legal field. A currently
used case dataset has less than 30k training examples Xu et al. (2021), which
is ten times less than scientific datasets such as arXiv and PubMed Cohan et
al. (2018). Thus, for the legal domain, it is not surprising that unsupervised
extractive summarization methods are of interest. Unfortunately, when
researchers Saravanan et al. (2006); Bhattacharya et al. (2019) have attempted
to directly apply standard unsupervised models to legal data, they have
obtained mediocre results.

However, most such attempts have failed to utilize the document structure of
legal texts. In case law, important sentences about the issues versus the
decisions of the court occur in different places in the document structure. In
contrast, summarization algorithms typically flatten any structure during
initial processing (i.e., they concatenate sentences from different
sections/paragraphs of a document to form a sentence list), or select
sentences using structural biases from other domains (e.g., the importance of
leading sentences in news Zheng and Lapata (2019)). As shown in Figure 1,
while LexRank

Figure 1: An example case law document-summary pair (ID: 3_2000canlii19612)
and different summarization system outputs, where sentences are annotated with
argumentative Issue, Reason, and Conclusion labels. Our method better extracts
argumentative sentences from the source document by exploiting its structure.

correctly extracts the legal issue from the beginning of the source text, it
incorrectly extracts several redundant sentences (i.e., [20], [21] and [22]
which talk about similar content) as well as ignores a large part of the
article (e.g., no sentences are extracted from the last section of the
original case: RECAPITULATION OF CALCULATION OF DAMAGES). In contrast, the
human summary focuses on sentences related to the argument of the legal
decision (e.g., what are the issues, reasoning and conclusions of this court
case?), which tend to be spread across the document structure.

Recently, the HipoRank model was proposed to exploit discourse structure
patterns during unsupervised extractive summarization Dong et al. (2021).
However, the model was designed for long scientific articles, and the
experiments were based on data where the articles were already split into
document sections. For case decisions, document structures are generally
either missing or only implicitly conveyed by text formatting. For example, in
Figure 1, document sections are conveyed by bolding in the source HTML file.
Moreover, algorithms such as PACSUM and HipoRank compute sentence centrality
just once and greedily select the top-k candidates as the extractive summary.
Such a greedy selection algorithm fails to match the distribution of the
argumentative sentences that ultimately appear in human case law summaries.

To address these limitations, we investigate the utility of different methods
for automatically segmenting the sentences of legal case decisions into the
sections of a document structure. We posit that incorporating better views of
document structure could bring improvements in summarization quality when
discourse-aware methods such as HipoRank are applied to legal case decisions.
We also propose a novel reweighting algorithm to improve how HipoRank selects
sentences when creating extractive summaries of legal decisions. The algorithm
takes the history of already selected summary sentences into account, and
gradually updates the importance score of a sentence. We posit that
reweighting will decrease the selection of redundant sentences as well as
increase the selection of argumentative sentences from less-represented
document segments (e.g., in the middle).

We evaluate our proposed method111Our code is available at
https://github.com/cs329yangzhong/DocumentStructureLegalSum for summarizing
legal decisions using an annotated Canadian case summarization dataset
(CanLII) Xu et al. (2021). Based on the belief that argumentative sentences
will capture the important sentences to summarize in a legal decision Xu et
al. (2021); Elaraby and Litman (2022), a portion of the CanLLI dataset comes
with gold-standard sentence-level labels identifying which sentences are
related to the issue/reasoning/conclusion of the court’s decision in both
source and summary documents. We use these labels to additionally propose a
metric that can better evaluate the quality of the generated summary from a
legal expert’s perspective. Empirical results show that our method improves
performance over previous unsupervised models Zheng and Lapata (2019); Dong et
al. (2021); Erkan and Radev (2004) in automatic evaluation.

##  2 Related Work

Supervised Extractive Summarization Using Discourse Information Graph-based
methods have been exploited for extractive summarization tasks to better model
the inter-sentence relations based on document structure. Xu et al. (2020)
applied a GCN layer to aggregate information from the document’s discourse
graph based on RST trees and dependencies. More recently, HiStruct+ Ruan et
al. (2022) and HEGEL Zhang et al. (2020a) started to incorporate the
hierarchical structure and topic structure of scientific articles into
supervised model training, respectively. However, HiSruct+ relied on the
relatively fixed and explicit document structure of scientific
articles222Section titles following a shared pattern (introduction, method …
and conclusion) are encoded to provide structural information. However, in our
dataset, sectioning is often missing or not meaningful (e.g., titles such as
”section 1”)., while HEGEL relied on a large training set to identify the
topic distributions. Our work uses an unsupervised extractive summarization
approach in a lower-resource setting, as well as studies the effects of
computing different types of document structures. We leave the exploration of
the aforementioned supervised approaches on legal domain texts for future
work.

Unsupervised Extractive Summarization Traditional extractive summarization
methods are mostly unsupervised Radev et al. (2000); Yin and Pei (2015); Hirao
et al. (2013), where a large portion apply the graph-based algorithms Salton
et al. (1997); Steinberger and Jezek (2004); Erkan and Radev (2004) or are
based on term frequencies such as n-gram overlaps Nenkova et al. (2005) to
rank the sentences’ importance. More recently, pretrained transformer-based
models Devlin et al. (2019); Lewis et al. (2020); Zhang et al. (2020b) have
provided better sentence representations. For instance, Zheng and Lapata
(2019) built directed unsupervised graph-based models on news articles using
BERT-based sentence representations and achieved comparable performance to
supervised models on multiple benchmarks. Dong et al. (2021) augmented the
document graph of Zheng and Lapata (2019) with sentence position and section
hierarchy to reflect the document structure of scientific articles. Different
from these two works which are based on assumptions of news and scientific
article structures, our method uses reweighting to better utilize the document
structure of legal cases.

Extractive Summarization of Legal Texts Despite the success of supervised
neural network models in news and scientific article summarization Zhang et
al. (2020b); Lewis et al. (2020); Zaheer et al. (2020), they face challenges
in legal document summarization given the longer texts, distinct document
structure, and limited training data Bhattacharya et al. (2019). Instead,
prior work has tackled legal extractive summarization by applying domain
independent unsupervised algorithms Luhn (1958); Erkan and Radev (2004);
Saravanan et al. (2006), or designing domain specific supervised approaches
Saravanan et al. (2006); Polsley et al. (2016); Zhong et al. (2019). One
recent work Bhattacharya et al. (2021) frames the task as Integer Linear
Programming and demonstrates the importance of in-domain structure and legal
knowledge. In another line of research, Xu et al. (2021) propose a sentence
classification task with the hope of exploiting the court decision’s argument
structure by making explicit its issues, conclusions, and reasons (i.e.,
argument triples). Our work is unsupervised and implicitly reveals the
relations between argument triples to generate better summaries.

##  3 Case Decision Summarization Dataset

Recent work has introduced a number of legal document summarization or salient
information identification tasks with associated datasets, e.g., for bill
summarization Kornilova and Eidelman (2019) and for case sentence argumentive
classification Xu et al. (2021) and rhetorical role prediction Malik et al.
(2022).

Case length (avg. # words) | 3,971
---|---
Summary length (avg. # words) | 266
Training set (# case/summary pairs) | 27,241
Testing set (# case/summary pairs) | 1,049
Table 1: Dataset statistics of CanLII. Figure 2: Fraction of sentences
annotated as argumentative (using the IRC scheme) in the case documents versus
in the summaries of the CanLII test set. Though only a small fraction of
sentences in the original document are annotated as IRCs, IRCs are a large
fraction of the human-written summaries.

Similarly to Xu et al. (2021), we use the CanLII (Canadian Legal Information
Institute) dataset of legal case decisions and summaries333The data was
obtained through an agreement with the Canadian Legal Information Institute
(CanLII) (https://www.canlii.org/en/).. Full corpus statistics are provided in
Table 1, while an example case/summary pair from the test set is provided in
Figure 6 in Appendix D.

Xu et al. (2021) only used a small portion of this dataset for their work in
argumentative classification. Conjecturing that explicitly identifying the
decision’s argumentative components would be crucial in case summarization,
they annotated 1,049 case and human-written summary pairs curated from the
full dataset. In particular, they recruited legal experts to annotate the
document on the sentence level, adopting an IRC scheme (see Figure 6 in
Appendix D) which classifies individual sentences into one of four categories:
Issue (legal question addressed in the case), Conclusion (court’s decisions
for the corresponding issue), Reason (text snippets explaining why the court
made such conclusion) and Non_IRC (none of the above). The distributions of
the IRC labels in the cases and summaries are shown in Figure 2 and illustrate
that argumentative sentences do indeed play an important role in human
summaries. We utilized the unannotated 27,241 pairs to train a supervised
model baseline and the 1049 annotated pairs as our test set. While none of our
summarization methods use the IRC annotations, they are used during testing as
the basis of a domain-specific evaluation metric.

##  4 Method and Models

We propose a reweighting model that employs a graph-based ranking algorithm to
exploit the structures encoded in long legal case decisions.

###  4.1 Discourse-Aware Backbone Model

The HipoRank (Hierarchical and Positional Ranking) model recently developed by
Dong et al. (2021) constructs a directed graph for document representation
using document section and sentence hierarchies. HipoRank computes the
centrality score of each sentence as

$c(s_{i}^{I})=\mu_{1}c_{inter}(s_{i}^{I})+c_{intra}(s_{i}^{I})$ (1)

where $s_{i}^{I}$ refers to the $i$-th sentence in $I$-th section. $\mu_{1}$
is a tunable hyper-paramter, $c_{inter}(s_{i}^{I})$ computes the sentence’s
similarity to other section representations and $c_{intra}(s_{i}^{I})$
computes the average similarity of the current sentence with all others in the
same section. HipoRank then selects the top-K ranked sentences as the summary.
More details of the algorithm are provided in Appendix A. Directly applying
HipoRank to our data yielded multiple challenges (e.g., redundant neighboring
sentences (recall Figure 1) as well as too many sentences from the ends of the
article were selected).

###  4.2 Multiple Views of Document Structure

Before creating a HipoRank document graph, the document must be split into
sections and sentences. The scientific datasets previously used with HipoRank
were already split Dong et al. (2021). We investigate the summarization impact
of using different approaches to automatically compute linear sections of the
document structure. Figure 3 shows different structures for the same case.

Original Document Structure This approach extracts the structure by processing
the HTML files. We use a heuristic to mark the section names with an italic
and bold format as the boundaries and segment the documents into multiple
continuous sections. It is worth noting that 297 of the 1049 test case
documents do not come with explicit section splits, thus we treat them as
whole text spans444The Original Structure method processed HTML source files
and split sentences using a legal sentence splitter
(https://github.com/jsavelka/luima_sbd). The Topic and Thematic views used
non-HTML data preprocessed by Xu et al. (2021), but used the same sentence
splitter..

Topic Segment View Meanwhile, we also explore using a traditional, domain-
independent linear text segmentation algorithm. We use C99 Choi (2000) but
with advanced sentence representation from SBERT Reimers and Gurevych (2019)
to group neighboring sentences into topic blocks.

Thematic Stage View Early studies found that legal documents tend to have
well-defined, domain-dependent thematic structures Farzindar and Lapalme
(2004) or rhetorical roles Saravanan et al. (2008). Following work that
extracts stage views in conversations (introductions → problem exploration →
problem solving → wrap up) Chen and Yang (2020), we extract thematic stages
through a Hidden Markov Model (HMM). A fixed order of stages is imposed and
only consecutive transitions are allowed between neighboring stages. We again
represent the sentences using SBERT Reimers and Gurevych (2019) and set the
number of stages as 5, including the starting Decision Data, Introduction,
Context, Judicial Analysis, and Conclusion, as introduced by Farzindar and
Lapalme (2004).

Figure 3: Different document structure views of a legal case decision (ID:
c_2003skpc17) from our CANLLI test set. Original case sentences are annotated
with Issue, Reason, and Conclusion labels. On the left side, the green, yellow
and blue boxes refer to thematic stage, topic segmentation and the original
document structure, respectively. The boxes mean the corresponding sentences
on the right hand side are grouped into the same segments. For instance, for
the first blue box, the original article is split by the italicized and bolded
section name of “The Fact”.

CanLII Header Removing Preliminary analysis demonstrated that the raw CanLII
documents fail to distinguish the less important headers at the beginning
(i.e., the descriptive text before the main content, for example, the content
above the grey splitting line and BASIS OF CLAIM in Figure 1). Generated
summaries also tend to cover a large portion of such information. We thus
propose a legal case decision preprocessing procedure following certain
heuristics555See Appendix D.3 for details. to remove those headers, and
demonstrate the improved summarization quality (for all views of document
structure) in Section 6.

###  4.3 Reweighting the Centrality Score

The HipoRank document graph will not change once built, and the important
sentences are greedily selected based on the aggregated centrality scores.
This may introduce redundancies in selecting similar sentences and ignore the
contents in the middle of the source case decisions that are more important
once the argumentative sentences at the beginning and end are taken into
account. We propose a novel reweighting approach that can tackle this problem.
A prior attempt Tao et al. (2021) on multi-round selection looked at the local
similarity between selected sentences. They iteratively recompute the sentence
to sentence similarities between the selected summary sentences and recompute
the final sentence centrality scores after each sentence selection. Instead,
we are focusing on modeling the relationship between the selected sentence and
the other candidate sentences. Their method is also not directly applicable to
longer text due to the $n^{2}$ time complexity of computation given large
numbers of sentences (on average 205 sentences for CanLII dataset).

Algorithm 1 Reweighting Algorithm

computed centrality score $c(s_{i}^{I})$ for all sentence s, $c_{intra}(d)$
for different section d ’s embedding, and a threshold g for phase transition
and maximum summary length $max_{len}$.

$Summ\leftarrow[]$

PHASE 1

while $len(Summ)\leq g*max_{len}$ do

$Summ.append(topK(\\{c(s_{i}^{I})\\})$

end while

PHASE 2

while $len(Summ)\leq max_{len}$ do

$c(s_{i}^{I})\leftarrow c(s_{i}^{I})-sim(c_{intra}(I),c_{intra}(J))*\mu_{2}$
$\triangleright$ J is the section index of last selected sentence, $\mu_{2}$
is a hyperparameter

$Summ.append(top-1({c(s_{i}^{I})}))$

end whileReturn Summ

Our approach can be divided into two phases, as shown in Algorithm 1. In the
first phase, we assume that important argumentative sentences at the two ends
of the document can be easily detected (as shown in Figures 1 and 3, legal
case documents generally start by introducing the issues and end with
conclusions). A quantitative analysis of the top-5 selected sentences in
CanLII in fact provides an 80% coverage of issue or conclusion sentences. We
thus set up a threshold to pick the first k sentences based on the original
document graphs. Afterward, we gradually downweight the sentence’s centrality
score using the location of the latest selected sentence, that is, we set a
penalty score for sentences that are placed as a neighbor of the current
sentence selected for the summary. Our rationale is that reasoning sentences
are more likely to be located in different sections in the middle that are not
shared with issues and conclusions.

##  5 Experimental Setup

For supervised models, we split the training data in an 80:20 ratio for
training and validation. For unsupervised models, we tune the hyperparameters
on the validation set. Model training details can be found in Appendix B.

ID | Model | R-1 | R-2 | R-L | BS
---|---|---|---|---|---
Oracles
1 | IRC | 58.04 | 36.02 | 55.28 | 87.94
2 | EXT (ROUGE-L, F1) | 59.38 | 38.77 | 56.94 | 87.85
Extractive baselines (no document structure)
supervised
3 | BERT_EXT | 43.44 | 17.84 | 40.36 | 84.47
unsupervised
4 | LSA | 37.22 | 17.82 | 34.87 | 84.48
5 | LexRank | 37.90 | 18.17 | 35.62 | 84.32
6 | TextRank | 36.70 | 16.19 | 34.00 | 83.51
7 | PACSUM | 40.01 | 15.68 | 37.37 | 83.52
HipoRank backbone (with different computed document structures)
8 | Original Structure | 41.61 | 17.13 | 38.73 | 83.55
9 | C99-topic | 41.33 | 16.48 | 38.45 | 83.53
10 | HMM-stage | 40.71 | 15.64 | 37.93 | 83.57
11 | Original Structure w/o header | 42.58 | 18.01 | 39.63 | 83.62
12 | C99-topic w/o header | 43.25 | 18.02 | 40.25 | 84.48
13 | HMM-stage w/o header | 42.64 | 17.38 | 39.76 | 83.57
Ours (HipoRank backbone + Reweighting Algorithm)
14 | Original Structure w/o header | 43.14 | 18.46 | 40.23 | 84.20
15 | C99-topic w/o header | 43.90 | 18.67 | 41.00 | 84.34
16 | HMM-stage w/o header | 43.28 | 17.80 | 40.40 | 84.22
Table 2: The automatic evaluation results on the CanLII test set. Bold
represents the best non-oracle score, italic the best baseline/backbone score,
and underline the best unsupervised baseline/backbone score.

Upper Bound Oracles Based on Figure 2, we create a domain-dependent IRC_Oracle
model where test sentences manually annotated with the IRC labels are
concatenated to form the summary. Following Nallapati et al. (2017), we also
report results for EXT_Oracle, a domain-independent summarizer which greedily
selects sentences from the original document based on the ROUGE-L scores
compared to the abstractive human summary.

Extractive Baselines For unsupervised models, we compare with LSA Steinberger
and Jezek (2004), LexRank Erkan and Radev (2004), TextRank Barrios et al.
(2016), and PACSUM Zheng and Lapata (2019). We also include HipoRank Dong et
al. (2021) with document views. For supervised methods, we compare with
BERT_EXT Liu and Lapata (2019). Although not our focus, abstractive baselines
are in Appendix D.4.

Automic Evaluation Metrics We report ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L
(R-L) F1 scores, as well as BERTScore (BS) Zhang et al. (2020c). We also
propose metrics to measure the recall value of the annotated IRC types in the
test set, which exploits the structure of case documents. More details are in
Section 6.2.

##  6 Results

In this section, we aim to deal with three research questions: RQ1. How well
do the extractive baselines including the HipoRank backbone deal with legal
documents? RQ2. How well do the different views of document structures perform
with the HipoRank backbone? RQ3. Can the reweighting algorithm help select
important argumentative sentences and improve summary quality?

###  6.1 Automatic Summarization Evaluation

Table 2 compares our methods with prior extractive models. See Appendix D.1
for example summaries.

RQ1. Table 2 shows that there is still a gap between oracle models (rows 1 and
2) and current extractive baselines. There are around 20 points differences on
R-1, R-2, and R-L. Among the baselines, the supervised model works best only
for R-1 and R-L. Unsupervised methods obtain the highest BS (row 4) and R-2
(row 5), possibly due to the higher coverage of n-grams benefitting from
longer extracted summaries (row 3 model generated summaries have an average
length of 250; row 4-6 models generate on average 400-word summaries; row 7
and 8 models have a limit of 220 words). Without reweighting, the HipoRank
backbone never outperforms the best extractive baseline. However, if only
unsupervised baselines are considered, HipoRank in row 12 does show the best
performance for 3 of the 4 evaluation metrics.

RQ2. To examine the effects of the document views in Section 4.2, we split the
document into different types of linear segments and then used HipoRank to
generate summaries. Recall that HipoRank is the only model to exploit document
structure, and as noted for RQ1, with the right structure could obtain the
best unsupervised R-1, R-L, and BS baseline scores. When naively constructing
different document structures from the CanLII dataset without header removal,
using NLP algorithms (rows 9 - 10) versus just using the HTML formatting (row
8) generally degraded results. However, when we experimented with a modified
version of the input documents (rows 11-13) where the headers were filtered
through heuristics before computing the document structure, the scores in rows
11-13 were higher (or in one case the same) than the comparable scores in rows
8-10. Also, without headers, the C99 topic segmentation algorithm (row 12) now
outperforms the use of HTML (row 11) (obtaining an average improvement of 0.5
points across ROUGE and 0.8 for BS), suggesting that better structures can
improve summarization. As shown in Table 3 (and earlier in Figure 3), C99
creates many small sections (average number of sentences per section is 3.39
with standard deviation of 0.67). We hypothesize that this encourages the
selection of sentences from more fine-grained segments. In contrast, the other
two methods create lengthy sections (average of more than 50 sentences) with a
large standard deviation (135.40 for original structure without headers). In
sum, with improvements in automatic metrics, we find that document structures
play an important role in summarizing cases.

Model | avg. # secs | avg. # sents per sec
---|---|---
with header
Original Structure | 4.83 (6.44) | 83.82 (118.78)
C99-topic | 63.47 (70.34) | 3.38 (0.71)
HMM-stage | 4.00(0.83) | 54.32 (64.80)
without header
Original Structure | 3.67 (5.51) | 102.99 (135.40)
C99-topic | 59.74 (69.91) | 3.39 (0.67)
HMM-stage | 3.16 (1.08) | 70.19 (119.39)
Table 3: Statistics about the average number of sections (avg. # secs) and
average number of sentences per section (avg. # sents per sec) across the
documents with different computed document structures (standard deviation in
parenthesis).

RQ3. The final block of Table 2 presents the reweighting results (using the
"w/o header" version of the CanLLI documents as they performed best in the
prior block). By downweighting sentences that appear under the same section as
previously selected ones, we observe an F1 improvement of 0.65, 0.65, and 0.75
on R-1, R-2, and R-L, respectively, on the previously best-performing topic
segmented document (row 12 versus 15). Row 15 in fact has the best non-oracle
results for all ROUGE scores. This observation regarding the value of
reweighting also holds for the original structure (row 11 vs. 14) and the HMM-
stage segments (row 13 vs. 16).

(a) IRC Oracles

(b) HipoRank with headers

(c) HipoRank without headers

(d) HipoRank without headers and with reweighting

Figure 4: Sentence positions in source cases for extractive summaries
generated by different models using the original document structure on the
test set. For (b) (c) (d), documents on the x-axis are sorted in the same
order. For IRC Oracles, Issue, Reasoning and Conclusion sentences are colored
accordingly.

Finally, to better understand the behavior of different enhancements to the
HipoRank backbone model, Figure 4 visualizes the positions of IRC sentences in
the original article that are selected by a particular summarization method.
Plot (a) shows that the human-annotated IRC sentences in the summary tend to
span across the source documents, with issues appearing in the beginning and
conclusions in the end. Plot (b) shows that although HipoRank using the
original document structure can successfully pick middle section sentences,
the darkest band at the starting positions shows that the model still heavily
relies on the inductive bias to pick the beginning sentences. Plot (c) shows
that removing the headers reduces the starting sentence bias. Finally, plot
(d) shows that reweighting reduces the number of sentences appearing on both
ends. Further analyses on the complete automatic evaluation results666See
Appendix C for ROUGE precision and recall. suggest that the improvements come
from higher recall values.

###  6.2 Argumentative Sentence Coverage

Taking advantage of the sentence-level IRC annotations, we propose recall
metrics to better measure the summary quality from a legal argumentation
perspective (RQ3). We compute the recall of “IRC” sentences extracted from the
original case as source IRC coverage (src. IRC). We similarly compute the
coverage of IRC sentences in the human-written summaries as target IRC
coverage (tgt. IRC) and all sentences as target sentence coverage (trg. cov.).
To do so we apply the oracle summarizer (Section 5) to map the generated
extractive summaries to the human-written abstractive summaries.

We report these values for the IRC oracle, an unsupervised (LexRank) and
supervised (BERT_EXT) baseline, the discourse-aware HipoRank with the original
structure, and our best reweighting model using C99-topic segmentation. Table
4 shows that our model obtains the highest target IRC recall and coverage,
suggesting that the summaries are more similar to the references with respect
to the decision’s argumentation. Another unsupervised model, LexRank, obtains
the highest source IRC, but its off-the-shelf package requires a fixed
sentence ratio selected from the source. This produced longer summaries than
other approaches and thus captured more IRCs in the source.

Model | src. IRC | tgt. IRC | trg. cov.
---|---|---|---
Oracle |
IRC | 1 (0.00) | 0.918 (0.18) | 0.820 (0.25)
Baselines |
BERT_EXT | 0.804 (0.27) | 0.846 (0.23) | 0.833 (0.23)
LexRank |  0.912 (0.19) | 0.811 (0.26) | 0.800 (0.27)
HipoRank | 0.800 (0.25) | 0.851 (0.24) | 0.844 (0.22)
Ours | 0.823 (0.26) |  0.866 (0.20) |  0.850 (0.21)
Table 4: Average recall of IRC sentences matched in the original case (src.
IRC), gold summary (tgt. IRC), as well as target sentences coverage (trg.
cov.) for each document (standard deviation in parenthesis).

###  6.3 Human Evaluation Discussion

As a first step towards human evaluation, we tried to extend the HipoRank
setup in Dong et al. (2021) and designed a human evaluation protocol as
follows. We asked human judges777All judges should be native English speakers
who are at least pursuing a JD degree in law school and have experience in
understanding case law. to read the human-written reference summary and
presented extracted sentences from different summarization systems. The judges
were asked to evaluate a system-extracted sentence according to two criteria:
(1) Content Coverage \- whether the presented sentence contained content from
the human summary, and (2) Importance \- whether the presented sentence was
important for a goal-oriented reader even if it was not in the human
summary888Here we assumed the goal-oriented reader as the lawyers or law
students seeking information from the case.. The sentence selection was
anonymized and randomly shuffled. We used the same sampling strategy in Dong
et al. (2021) to pick ten reference summaries where the system outputs were
neutral (i.e., had similar R-2 scores compared to the human reference).
However, initial annotation on a small set by a legal expert demonstrated that
the selected sentences may not reflect the model’s capability. Most sampled
system outputs had low ROUGE-2 F1 scores compared to the reference (normally
below 10% while the average model performance is 17%), and the human evaluator
reported that some of the selected sentences were not meaningful. We thus
propose that a more careful sampling technique will be required for legal
annotation tasks such as ours.

To further guide our future work, we also reviewed how prior legal domain
research has performed human evaluations when automatically summarizing legal
documents Polsley et al. (2016); Zhong et al. (2019); Salaün et al. (2022).
Due to the burden of reading lengthy original documents, as in our human
evaluation, most prior work evaluated summary quality using reference
summaries rather than source documents. In addition, legal evaluations have
typically been small-scale (5-20 summaries) due to the need to have evaluators
with particular types of expertise (e.g., law graduate students or law
professors), which was a similar constraint in our exploratory human
evaluation. Researchers have also designed new types of legally-relevant
evaluation questions that evaluate the summary for task-specific properties
that go beyond more typical properties such as grammar, readability, and
style. In our case, we would like legal experts to assess IRC coverage in the
future.

##  7 Conclusion

We presented an unsupervised graph-based model for the summarization of long
legal case decisions. Our proposed approach incorporated diverse views of the
document structure of legal cases and utilized a reweighting scheme to better
select argumentative sentences. Our exploration of document structure
demonstrates how using different types of document structure impacts
summarization performance. Moreover, a document structure inspired reweighting
scheme yields performance gain on the CanLII case dataset.

## Ethical Considerations

The utilization of the generated summary results of legal documents remains
important. Current extractive methods avoid the problem of generating
hallucinated information Kryscinski et al. (2020); Maynez et al. (2020), which
has been observed in abstractive methods that use large-scale pre-trained
language models. The extracted sentences, however, may not capture the
important contents of the legal documents. Meanwhile, CanLII has taken
measures to limit the disclosure of defendants’ identities (such as blocking
search indexing). Thus, using the dataset may need to be taken good care of
and avoid impacting those efforts.

## Acknowledgement

We thank the Pitt AI Fairness and Law group, the Pitt PETAL group, and the
anonymous reviewers for their valuable feedback. This research is supported by
the National Science Foundation under Grant IIS-2040490 and a gift from
Amazon.

## Limitations

The dataset we used has a relatively small scale (1K) test set. Meanwhile, the
automatic evaluation metrics may fall short compared to human evaluations,
thus unfaithfully representing the final quality of generated summaries.
Although lightweight, there is still a large performance gap between our
unsupervised method and both the extractive oracles as well as abstractive
models (Appendix D.4), especially given the small-scale training data. There
are more graph-based methods to aggregate information from the built graphs
and we would like to explore and include more graph-based methods but selected
the most relevant one in this work. Moreover, our proposed reweighting
paradigm heavily relied on observations about the structure of legal cases.
Many other legal document types, such as bills and statutes, have inherently
distinct structures. Our results also show the importance of finding the
correct structure and weights, which can vary depending on the corpus. This
will require more advanced methods to find the correct structure and weights
for a dataset.

## References

  * Barrios et al. (2016) Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. 2016\.  Variations of the similarity function of textrank for automated summarization.  _CoRR_ , abs/1602.03606.
  * Baxendale (1958) P. B. Baxendale. 1958.  Machine-made index for technical literature—an experiment.  _IBM Journal of Research and Development_ , 2(4):354–361.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Bhattacharya et al. (2019) Paheli Bhattacharya, Kaustubh Hiware, Subham Rajgaria, Nilay Pochhi, Kripabandhu Ghosh, and Saptarshi Ghosh. 2019.  A comparative study of summarization algorithms applied to legal case judgments.  In _Advances in Information Retrieval_ , pages 413–428, Cham. Springer International Publishing.
  * Bhattacharya et al. (2021) Paheli Bhattacharya, Soham Poddar, Koustav Rudra, Kripabandhu Ghosh, and Saptarshi Ghosh. 2021.  Incorporating domain knowledge for extractive summarization of legal case documents.  In _Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law_ , pages 22–31.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  LEGAL-BERT: The muppets straight out of law school.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904, Online. Association for Computational Linguistics.
  * Chen and Yang (2020) Jiaao Chen and Diyi Yang. 2020.  Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4106–4118, Online. Association for Computational Linguistics.
  * Choi (2000) Freddy Y. Y. Choi. 2000.  Advances in domain independent linear text segmentation.  In _1st Meeting of the North American Chapter of the Association for Computational Linguistics_.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186.
  * Dong et al. (2021) Yue Dong, Andrei Mircea, and Jackie Chi Kit Cheung. 2021.  Discourse-aware unsupervised summarization for long scientific documents.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1089–1102, Online. Association for Computational Linguistics.
  * Elaraby and Litman (2022) Mohamed Elaraby and Diane Litman. 2022.  ArgLegalSumm: Improving abstractive summarization of legal documents with argument mining.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 6187–6194, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Erkan and Radev (2004) Günes Erkan and Dragomir R Radev. 2004.  Lexrank: Graph-based lexical centrality as salience in text summarization.  _Journal of artificial intelligence research_ , 22:457–479.
  * Farzindar and Lapalme (2004) Atefeh Farzindar and Guy Lapalme. 2004.  Legal text summarization by exploration of the thematic structure and argumentative roles.  In _Text Summarization Branches Out_ , pages 27–34, Barcelona, Spain. Association for Computational Linguistics.
  * Hirao et al. (2013) Tsutomu Hirao, Yasuhisa Yoshida, Masaaki Nishino, Norihito Yasuda, and Masaaki Nagata. 2013.  Single-document summarization as a tree knapsack problem.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1515–1520, Seattle, Washington, USA. Association for Computational Linguistics.
  * Kornilova and Eidelman (2019) Anastassia Kornilova and Vladimir Eidelman. 2019.  BillSum: A corpus for automatic summarization of US legislation.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 48–56, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Luhn (1958) H. P. Luhn. 1958.  The automatic creation of literature abstracts.  _IBM Journal of Research and Development_ , 2(2):159–165.
  * Malik et al. (2022) Vijit Malik, Rishabh Sanjay, Shouvik Kumar Guha, Shubham Kumar Nigam, Angshuman Hazarika, Arnab Bhattacharya, and Ashutosh Modi. 2022.  Semantic segmentation of legal documents via rhetorical roles.  _AAAI_.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents.  In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Nenkova and McKeown (2011) Ani Nenkova and Kathleen McKeown. 2011.  Automatic summarization.  _Foundations and Trends® in Information Retrieval_ , 5(2–3):103–233.
  * Nenkova et al. (2005) Ani Nenkova, Lucy Vanderwende, and Lucy Vanderwende. 2005.  The impact of frequency on summarization.
  * Polsley et al. (2016) Seth Polsley, Pooja Jhunjhunwala, and Ruihong Huang. 2016.  CaseSummarizer: A system for automated summarization of legal texts.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations_ , pages 258–262, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Radev et al. (2000) Dragomir R. Radev, Hongyan Jing, and Malgorzata Budzikowska. 2000.  Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies.  In _NAACL-ANLP 2000 Workshop: Automatic Summarization_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Ruan et al. (2022) Qian Ruan, Malte Ostendorff, and Georg Rehm. 2022.  HiStruct+: Improving extractive text summarization with hierarchical structure information.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1292–1308, Dublin, Ireland. Association for Computational Linguistics.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
  * Salaün et al. (2022) Olivier Salaün, Aurore Troussel, Sylvain Longhais, Hannes Westermann, Philippe Langlais, and Karim Benyekhlef. 2022.  Conditional abstractive summarization of court decisions for laymen and insights from human evaluation.  In _35th International Conference on Legal Knowledge and Information System_.
  * Salton et al. (1997) Gerard Salton, Amit Singhal, Mandar Mitra, and Chris Buckley. 1997.  Automatic text structuring and summarization.  _Information Processing and Management_ , 33(2):193–207.  Methods and Tools for the Automatic Construction of Hypertext.
  * Saravanan et al. (2006) M. Saravanan, B. Ravindran, and S. Raman. 2006.  Improving legal document summarization using graphical models.  In _Proceedings of the 2006 Conference on Legal Knowledge and Information Systems: JURIX 2006: The Nineteenth Annual Conference_ , page 51–60. IOS Press.
  * Saravanan et al. (2008) M. Saravanan, B. Ravindran, and S. Raman. 2008.  Automatic identification of rhetorical roles using conditional random fields for legal document summarization.  In _Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I_.
  * Steinberger and Jezek (2004) Josef Steinberger and Karel Jezek. 2004.  Using latent semantic analysis in text summarization and summary evaluation.  volume 4, pages 93–100. ISIM.
  * Tao et al. (2021) Dehao Tao, Yingzhu Xiong, Jin He, Yongfeng Huang, et al. 2021.  An unsupervised extractive summarization method based on multi-round computation.  _arXiv preprint arXiv:2112.03203_.
  * Teufel (1997) Simone Teufel. 1997.  Sentence extraction as a classification task.  In _Intelligent Scalable Text Summarization_.
  * Xu et al. (2021) Huihui Xu, Jaromir Savelka, and Kevin Ashley. 2021.  Accounting for sentence position and legal domain sentence embedding in learning to classify case sentences.  In _Legal Knowledge and Information System_.
  * Xu et al. (2020) Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Discourse-aware neural extractive text summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5021–5031, Online. Association for Computational Linguistics.
  * Yin and Pei (2015) Wenpeng Yin and Yulong Pei. 2015.  Optimizing sentence modeling and selection for document summarization.  In _Proceedings of the 24th International Conference on Artificial Intelligence_ , IJCAI’15, page 1383–1389. AAAI Press.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ , 33.
  * Zhang et al. (2020a) Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2020a.  Hegel: Hypergraph transformer for long document summarization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Zhang et al. (2020b) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020b.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * Zhang et al. (2020c) Tianyi Zhang, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020c.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zheng and Lapata (2019) Hao Zheng and Mirella Lapata. 2019.  Sentence centrality revisited for unsupervised summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6236–6247, Florence, Italy. Association for Computational Linguistics.
  * Zhong et al. (2019) Linwu Zhong, Ziyi Zhong, Zinian Zhao, Siyuan Wang, Kevin D. Ashley, and Matthias Grabmair. 2019.  Automatic summarization of legal decisions using iterative masking of predictive sentences.  In _Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law_ , ICAIL ’19, page 163–172, New York, NY, USA. Association for Computing Machinery.

##  Appendix A The HipoRank Algorithm

In this section, we provide a detailed recap of the HipoRank algorithm Dong et
al. (2021). Our approach mainly modifies the obtained document graphs by
building a section-section graph and changes the final summary selection
algorithms.

###  A.1 Hierarchical Document Graph Creation

The document is first split into its sections, then into sentences. Two levels
of connections are allowed in the built hierarchical graph: intra-sectional
connections and inter-sectional connections. Following the original paper, we
display a toy example of these two types of connections in Figure 5.

Figure 5:  (Reproduced from Dong et al. (2021)) An example of a hierarchical
document graph constructed by HipoRank approach on a toy document, which
contains two sections {T1, T2}, each containing three sentences for a total of
six sentences {s1, . . . , s6}. In the graph, each double-headed arrow
represents two edges with opposite directions. The solid and dashed arrows
indicate intra-section and inter-section connections respectively.

#### Intra-sectional connections

are designed to measure a sentence’s importance score inside its section. The
authors built a fully-connected subgraph over all sentences in the same
section, allowing for sentence-sentence edges, which are measured by a
weighted version of the similarities of sentence embeddings.

#### Inter-sectional connections

“aim to model the global importance of a sentence with respect to other
topics/sections in the document”, according to Dong et al. (2021). To reduce
the expensive computation of all sentence-sentence connections spanning across
a document, HipoRank’s authors introduce section nodes on top of sentence
nodes, and only allow for sentence-section edges to model the global
information.

###  A.2 Asymmetric Edge Weighting by Boundary Functions

In order to compute the weight of an edge, HipoRank measures the similarity of
sentence-sentence pairs by computing the cosine similarity of encoded sentence
embeddings. Similarly, for sentence-section pairs, it averages the sentences’
representations in the same section, uses it as the section vector, and
further computes the cosine similarity. Taking two discourse hypotheses of
long scientific documents into account ((1) important sentences are near the
boundaries (start or end) of a text Baxendale (1958) and (2) sections near the
text boundaries (start or end) are more important Teufel (1997)), the authors
of HipoRank capture this asymmetry by making their hierarchical graph directed
and inject asymmetric edge weighting over intra-section and inter-section
connections. We refer to the original paper for more detailed setups and
algorithm details.

###  A.3 Importance Computation and Summary Generation

We talk about the importance computation approach and summary generation
details in §4.1.

##  Appendix B Training Details and Hyperparameters

All of our experiments are conducted on Quadro RTX 5000 GPUs, each of which
has 16 GB RAM. For the extractive oracle baseline, we use the python package
of rouge999https://pypi.org/project/rouge/ to compute the ROUGE-L scores for
sentence scoring.

#### Document Segmentation

We provide details of segmentation methods mentioned in §4.2 below. For
sentence encoding, we use the sentence_transformer
library101010https://www.sbert.net/, and the checkpoint of “bert-base-nli-
stsb-mean-tokens” for sentence representations. For the HMM stage
segmentation, we train a GaussianHMM model with
hmmlearn111111https://hmmlearn.readthedocs.io/en/latest/, setting the number
of the components at 5 and train the model for 50 iterations on the validation
set. For C99 algorithm, we use an implementation121212https://github.com/GT-
SALT/Multi-View-Seq2Seq/blob/master/data/C99.py shared from Chen and Yang
(2020) in their original paper. We set the window size of 4 and
std_coefficient as 1. All data processing scripts are publicly available in a
combined package in
https://github.com/cs329yangzhong/DocumentStructureLegalSum.

#### Supervised Model

We build our BERT_EXT, the extractive model, on top of the official code base
of the work of Liu and Lapata (2019)131313https://github.com/nlpyang/PreSumm.
Since many original documents’ lengths go beyond the 512 token limits, we
break the full document into different chunks and train the model to extract
the top-3 sentences. For hyperparameters, we use 4 GPUs, set the learning rate
of 2e-3, and save the best checkpoints at every 5,000 steps. We set the batch
size as 3,000, the maximum training step at 100,000, and warm-up steps at
10,000.

#### Unsupervised Models

We use off-the-shelf packages for most traditional models. We use
LSA141414https://github.com/luisfredgs/LSA-Text-Summarization,
TextRank151515https://github.com/summanlp/textrank, and
LexRank161616https://github.com/crabcamp/lexrank accordingly.

For PACSUM model, we follow the re-
implementation171717https://github.com/mirandrom/HipoRank of Dong et al.
(2021) and keep the hyperparameters fixed with the original setup. BERT-based
sentence embeddings are extracted using the fine-tuned BERT model released
from the original paper Zheng and Lapata (2019). We also experimented with
LEGAL-BERT Chalkidis et al. (2020) in the early stages of our research but
found it degraded performance on the baselines.

For HipoRank, we use the publicly available code
base181818https://github.com/mirandrom/HipoRank. We experimented with various
hyperparameter settings on the validation set but we find that the original
hyperparamters used in the original paper for PubMed dataset seem to be the
most stable and produce the best results. ($\lambda_{1}=0.0$,
$\lambda_{2}=1.0$, $\alpha=1.0$, with $\mu_{1}=0.5$.)

We build our reweighting model on top of the HipoRank dataset. We search the
threshold g (for phase transition between phases 1 and 2) between [0.3, 0.5,
and 0.7], finding that 0.5 is the best for the CanLII dataset.

##  Appendix C The Effects of Reweighting Algorithm

We study the effects of our reweighting algorithm by comparing different
models’ performances on the input documents with original structures. As shown
in Table 5, with a minor sacrifice of precision, the recall values are greatly
improved with the reweighting algorithm, thus resulting in the final
improvements of F1 scores.

Document Structures | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
| P | R | F1 | P | R | F1 | P | R | F1
w/o header | 45.24 | 47.39 | 42.58 | 19.23 | 20.12 | 18.01 | 42.25 | 43.95 | 39.63
w/o header + Reweighting | 44.13 | 49.80 | 43.14 | 18.97 | 21.35 | 18.46 | 41.29 | 46.26 | 40.23
Table 5: The Precision (P), Recall (R) and F1 of ROUGE-1/2/L scores for the
inputs with original document structures, with and without reweighting
algorithm. We find that the reweighting algorithm improves the recall,
suggesting that more argumentative sentences in the references are covered.

##  Appendix D Examples

###  D.1 Summary Generation Results

We show the reference, best baseline, and our model’s output on the C99-topic
view of the without header version of documents in Table 6.

Model | Summary
---|---
Reference | FIAT: The defendants, Sims, Garbriel and Dumurs, bring separate motions, pursuant to
Queen’s Bench Rule 41(a), for severance of the claims against them or for an
order
staying the claims against them until the plaintiffs’ claim against the
primary defendant,
Walbaum have been heard and decided. || HELD: 1) || The Court will look at all
of the
circumstances in deciding whether to grant an application for severance. || In
this case
the plaintiffs should not be precluded from adducing evidence related to
Walbaum’s
dealings with each of the applicants or required to segregate the evidence
into two,
three or four separate trials. || Given the likelihood that the applicants
will be required
to attend portions of the trial in respect of the Walbaum Group in any event,
severance
would not necessarily result in a significant saving of time and expense. ||
2) The plain-
| -tiffs acknowledge that only relatively small portion of trial time (perhaps
less than 1
| day) will pertain to the claims against any one of the Sims, Gabriel or
Dumurs. || It wou-
| -ld be unfair to require all of the applicants to participate in all of the
trial when very little
| of it will be relevant to them. || Specific dates and times should be set
aside for the
| plaintiff to call evidence with respect to its claims against each applicant
group. || The
| applicants should be relieved from attending the trial at any other time.
HipoRank | QUEEN’S BENCH FOR SASKATCHEWAN || 2007 SKQB 296 || Judicial Centre: Regina
|| DUN-RITE PLUMBING & HEATING LTD. || (d) Robert Dumur || 593340 Saskatchewan
Ltd., carrying on || business as Dumur Industries (herein “the Dumurs”) || [2]
The Sims, Gab-
-riel and the Dumurs bring separate motions, pursuant to Queen’s Bench Rule 41(a) for
severance of the claims against them or for orders staying the claims until
the plaintiffs’ claims
against the Walbaum Group have been heard and decided. || ANALYSIS || [12] T
applications
are brought pursuant to Queen’s Bench Rule 41 which states: || 41 (1) Where
the joinder of
multiple claims or parties in the same action may unduly complicate or delay
the trial, or cause
undue prejudice to a party, the court may: || (a) order separate trials; ||
[21] It will also be left to
the trial judge (or the pre-trial management judge) to designate specific days
on which defence
evidence may be adduced during the trial and argument presented with respect
to each claim. ||
All defendants other than those comprising the Walbaum Group shall be relieved
from attending
the trial on any date not designated by counsel for the plaintiffs or
designated by the trial judge
| for adducing defence evidence and presenting argument. || [22] Costs will be
in the cause. || D.P.
| Ball
Ours | FIAT BALL J. || August 14, 2007 || [1] The plaintiff brings this action against nine defendants
(the claim against the defendant Albert Fazakas has been discontinued) who can
be separated
into four groups: || All-Rite Plumbing Heating Ltd. || [18] Although choeunsel
for the plaintiffs
asserts that the evidence against all of the defendants can be adduced in no
more than two and
one-half days, given the number and complexity of the claims against the
Walbaum Group this
estimate seems very unrealistic. || [19] The plaintiffs acknowledge that only
relatively small
portion of trial time (perhaps less than one day) will pertain to the claims
against any one of the
Sims, Gabriel or the Dumurs. || It would be unfair to require all of the
applicants to participate
in all of the trial when very little of it will be relevant to them. || The
applicants should be
relieved from attending the trial at any other time. || The plaintiffs shall
not call evidence in
| respect of those claims on any other date without leave of the court. || All
defendants other than
| those comprising the Walbaum Group shall be relieved from attending the
trial on any date not
| designated by counsel for the plaintiffs or designated by the trial judge
for adducing defence
| evidence and presenting argument.
Table 6: Generated summaries for a CanLII case decision (ID: 2_2007skqb296),
we use special symbol “||” to mark the sentence boundaries.

###  D.2 IRC Annotation

We show the IRC annotation of both a case and its human summary in Figure 6.

###  D.3 Document Cleaning Heuristics

The heuristics for filtering the headers from cases are provided below for
replication purposes; we also provide the
code191919https://github.com/cs329yangzhong/DocumentStructureLegalSum to
process the CanLII data (although it requires that the data must first be
obtained through an agreement with the Canadian Legal Information Institute).

  1. 1.

Cut the document until the sentence begins with “Introduction”.

  2. 2.

Cut the document until the sentence starts with an ordered number such as (1),
[1].

  3. 3.

Remove rows until the judge’s name or case date appeared.

###  D.4 Comparing to Abstractive Summarization

For supervised abstractive baselines, we experiment with BART Lewis et al.
(2020) and Longformer-Encoder-Decoder (LED) Beltagy et al. (2020). The latter
model can process longer input documents up to 16k tokens. The results in
Table 7 show that there still exists a gap between the extractive and
abstractive models.

Figure 6: An example of the annotated Issue, Reason, and Conclusion sentences in CanLII dataset’s case and summary pair (ID: 1991canlii4370). A portion of the beginning sentences in the case are not as important as the main document, including the meta-data of the case such as the participants’ names, time, locations, etc. Thus, we treated them as headers and filtered them out using a heuristic introduced in Appendix D.3.  |  | CanLII
---|---|---
ID | Model | R1/R2/RL | BS
Oracles |
1 | IRC | 58.04/36.02/55.28 | 87.94
2 | EXT | 59.38/38.77/56.94 | 87.85
Supervised Extractive |
3 | BERT_extractor | 43.44/17.84/40.36 | 84.47
Supervised Abtractive |
4 | BART | 50.50/25.58/46.82 | 87.25
5 | LED | 53.72/28.75/ 50.17 | 87.55
Table 7: The automatic evaluation results on the CanLII test set with
supervised abstractive models.