# Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple
Granularities

Zejiang Shen† Kyle Lo† Lauren Yu♢ Nathan Dahlberg
Margo Schlanger♢ Doug Downey†,♣
†Allen Institute for AI ♢University of Michigan ♣Northwestern University
{shannons, kylel<EMAIL_ADDRESS>
{laurenyu<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With the advent of large language models, methods for abstractive
summarization have made great strides, creating potential for use in
applications to aid knowledge workers processing unwieldy document
collections. One such setting is the Civil Rights Litigation Clearinghouse
(CRLC),111https://clearinghouse.net. which posts information about large-scale
civil rights lawsuits, serving lawyers, scholars, and the general public.
Today, summarization in the CRLC requires extensive training of lawyers and
law students who spend hours per case understanding multiple relevant
documents in order to produce high-quality summaries of key events and
outcomes. Motivated by this ongoing real-world summarization effort, we
introduce Multi-LexSum, a collection of 9,280 expert-authored summaries drawn
from ongoing CRLC writing. Multi-LexSum presents a challenging multi-document
summarization task given the length of the source documents, often exceeding
two hundred pages per case. Furthermore, Multi-LexSum is distinct from other
datasets in its multiple target summaries, each at a different granularity
(ranging from one-sentence “extreme” summaries to multi-paragraph narrations
of over five hundred words). We present extensive analysis demonstrating that
despite the high-quality summaries in the training data (adhering to strict
content and style guidelines), state-of-the-art summarization models perform
poorly on this task. We release Multi-LexSum for further research in
summarization methods as well as to facilitate development of applications to
assist in the CRLC’s mission.222The dataset can be accessed at
https://multilexsum.github.io.

##  1 Introduction

Automatic summarization is a longstanding goal of natural language processing.
Recently, abstractive summarization methods powered by large pretrained
language models have shown impressive results [36, 60]—raising the question of
whether these methods can help real-world summarization workloads currently
performed by human experts. In this paper, we present a new dataset, Multi-
LexSum, for studying automatic summarization in an important real-world
application setting found in the Civil Rights Litigation Clearinghouse (CRLC).
The CRLC currently collects and presents documents and information from modern
large-scale civil rights lawsuits in a manner easily understood by legal
practitioners and scholars and the general public alike [10]. Today, the
Clearinghouse relies on human legal experts to write summaries of civil rights
cases, explaining their events and outcomes. This cognitively demanding task
requires summary writers to comprehend multiple documents of different types
(often totaling over two hundred pages of text per case); extract entities,
events, and their interrelationships; and synthesize this information into a
summary that captures the key details in each case’s timeline. For a typical
summary, this process takes an expert 1-4 hours. And it needs to be repeated
as the case proceeds through the legal system, to keep the summary up-to-date.

Success in summarization automation would allow the Clearinghouse and other
efforts like it to greatly increase their coverage and update their summaries
in close to real time. Quicker and less costly narrative description of
important and routine lawsuits would benefit both the legal field and the
general public by increasing access to and understanding of disputes and their
resolutions.

We release Multi-LexSum, an abstractive summarization dataset for federal U.S.
large-scale civil rights lawsuits drawn from the CRLC. It consists of about
40,000 source documents and 9,000 expert-written summaries (covering about
half as many cases). Besides its potential to enable new summarization
capabilities to benefit the CRLC effort and others like it, Multi-LexSum has
unique characteristics that make it an interesting object of study for
summarization research more broadly:

  * •

Unlike some summarization workloads, the CRLC task requires production of
summaries at multiple target levels of granularity: tiny (25 words, on
average), short (130 words), and long (650 words). Variable granularity can be
valuable in many applications—e.g., short summaries are ideal to scan for
items of interest, and longer summaries to explore more deeply. But to our
knowledge, Multi-LexSum is the first dataset to provide summaries at multiple
levels of granularity. It enables study of multi-task methods that learn from
supervision at multiple granularities, and that provide controllable
generation at a specified granularity, as our experiments explore.

  * •

Other multi-document summarization datasets offer only 800-8,000 words in the
source documents, on average [14, 16, 39], even though many applications
require summarizing large collections of multiple documents. In Multi-LexSum,
the average source length is over 75,000 words.

  * •

Unlike other summarization datasets that are (semi-)automatically curated [26,
44, 11, 35, 22, 52, 28], Multi-LexSum consists of expert-authored summaries.
The experts—lawyers and law students—are trained to follow carefully created
guidelines, and their work is reviewed by an additional expert to ensure
quality (see Appendix B). This provides high-quality supervision and
evaluation and reduces the risk of training on summaries containing facts
unsupported by the source text, which can contribute to model hallucination
[33, 40].

We conduct a series of experiments on Multi-LexSum, and find that existing
summarization models perform poorly. Human assessments of model output result
in an average rating of 0.43 on a 0-3 scale, showing that significant
improvements are needed before the summaries can provide utility for the CRLC
project. Finally, multi-task approaches that train on the multiple
granularities of summaries in Multi-LexSum demonstrate promise for improving
long summary quality.

Long Summary l: This case is about an apprenticeship test that had a disparate
impact on Black apprenticeship applicants. The [a]Equal Employment Opportunity
Commission (EEOC) filed this lawsuit on [d]December 27, 2004, in [e]U.S.
District Court for the Southern District of Ohio. Filing [a]on behalf of
thirteen Black individuals and [f]similarly situated Black apprenticeship test
takers, the EEOC alleged that [g]the individuals’ employer, the Ford Motor
Company, as well as their union, the United Automobile, Aerospace, and
Agricultural implement workers of America (the "UAW"), and the Ford-UAW Joint
Apprenticeship Committee, violated [h]Title VII of the Civil Rights Act, 42
U.S.C. §1981, and Michigan state anti-discrimination law. At issue were the
selection tests for apprenticeship training programs, whose disparate impact
denied Black applicants eligibility and admission. [i]The EEOC sought
injunctive relief, as well as damages (including backpay) for the Black
apprenticeship applicants. The case was assigned to [j]Judge Susan J. Dlott.
[k]The individuals also brought a separate class action against Ford and the
UAW, Robinson v. Ford Motor Company, (No. 1:04-cv-00844), and the cases were
consolidated on January 6, 2005. As a result, the case was transferred to
[j]Judge S. Arthur Spiegel. Six months later, in June 2005, both cases were
resolved via a [l]class settlement agreement. Ford agreed to pay $8.55 million
and to implement a new selection process for its apprenticeship programs. This
agreement further required Ford to hire an industrial psychologist to design
this new selection process and to place 279 members of the settlement class on
the eligibility list for the Ford apprenticeship program.
On [m]June 15, 2005, the court found that the proposed settlement agreement
was fair. [n]2005 WL 5253339. The next day, the court ordered that Ford pay
$1.1 million to cover attorneys’ fees and expenses incurred during settlement
negotiations, and $567,000 to cover fees and expenses associated with the
implementation and monitoring of the settlement agreement. [n]2005 U.S. Dist.
LEXIS 12071. As [o]the settlement was initially scheduled to last for three
years, and [p]there is no further activity on the docket sheet, this case
presumably closed in 2008.
---
Short Summary s: This case is about an apprenticeship test that had a
disparate impact on Black apprenticeship applicants. The Equal Employment
Opportunity Commission (EEOC) filed this lawsuit on December 27, 2004, in U.S.
District Court for the Southern District of Ohio. Filing on behalf of thirteen
Black individuals and similarly situated Black apprenticeship test takers, the
EEOC alleged that the individuals’ employer, the Ford Motor Company, as well
as their union, the United Automobile, Aerospace, and Agricultural implement
workers of America (the “UAW”), and the Ford-UAW Joint Apprenticeship
Committee, violated Title VII of the Civil Rights Act, 42 U.S.C. § 1981, and
Michigan state anti-discrimination law. The EEOC sought injunctive relief and
damages for the Black apprenticeship applicants. The individuals also brought
a separate class action against Ford and the UAW, and the cases were
consolidated. In June 2005, both cases were resolved via a class settlement
agreement. Ford agreed to pay $8.55 million and to implement a new selection
process for its apprenticeship programs, and the court ordered Ford to cover
attorneys’ fees and expenses. This case is closed.
Tiny Summary t: 2005 class action settlement resulted in Ford paying $8.55m to
redesign its selection process for apprenticeship programs to address the
previous process’s disparate impact on Black applicants.
Checklist of Facts during Writing
a. Plaintiff description | e. Court’s full name | i. Remedy sought | m. Date of settlement/decree
b. Type of counsel | f. Class description | j. Judge’s name | n. Citation to an opinion
c. Type of action | g. Defendant description | k. Consolidated case | o. How long decrees lasted
d. Filling Date | h. Statutory basis for case | l. If class action | p. Last action in case

Table 1: Three different summaries for one case in Multi-LexSum. We highlight
and label spans of text according to which fact it covers.

##  2 Related work

###  2.1 Natural language processing for legal documents

Much recent work in Natural Language Processing (NLP) has focused on the legal
domain [2, 8, 3, 57, 6]. Lawsuits generate rich document sets with domain-
specific language and complex structures, which are challenging for state-of-
the-art language processing models [3]. Given the important societal role of
litigation, along with the extremely high cost of legal expertise, NLP methods
to help search, synthesize, and answer questions about legal corpora are of
strong interest.

NLP has been applied to a variety of legal document types, including patents
[52], legal provisions and contracts [54, 38, 49], legislative bills [34], and
court documents [42, 62, 21]. The NLP tasks studied in this work range from
document/sentence classification [5, 54, 7] to information extraction [4, 25],
question answering [32, 49, 30, 63], and—most relevant to our work—automatic
summarization [34, 52, 21, 29]. As found in other specialized domains of
language, legal NLP systems often benefit from starting from a large language
model pre-trained on legal text [57, 6, 62].

Our Multi-LexSum dataset is focused on automatic summarization of court
proceedings and outcomes. Previous work on this task mainly focuses on
extractive approaches, where the output summaries consist of sentences drawn
directly from the source [29]. Hachey and Grover [23] summarize UK court
judgments from the HOLJ corpus [21] by selecting the most summary-worthy
sentences from a document, while Kim et al. [31] develop a graph-based summary
sentence selection method on the same corpus. Yousfi-Monod et al. [59] propose
ProdSum, a Naive Bayes sentence classifier, for summarizing case decisions
from the Canadian Legal Information Institute (CanLII). Galgani et al. [18]
experiment with citation based summarization approaches on case reports from
the Australasian Legal Information Institute AustLII [20]. Systems like
CaseSummarizer [47], LetSum [17], and the pipeline by Zhong et al. [64] are
developed to extractively summarize documents from AustLII, CanLII, and Board
of Veterans’ Appeals (BVA), respectively. By contrast, our work focuses on
_abstractive_ summarization, where the target output is a paraphrase of the
source, creating the potential for providing more succinct summaries in more
accessible language. We release a large dataset of over 9,000 expert-written
summaries for court documents from about 4,500 U.S. federal civil rights
lawsuits, as a testbed for state-of-the-art abstractive summarizers.

###  2.2 Summarization datasets in other domains

Multi-LexSum contains expert-written summaries of up to three different
granularities for the same source; this is the first such published dataset to
the best of our knowledge. Perhaps the most similar work is BookSum [35];
however in contrast to our work, BookSum’s multiple summaries consider
different lengths of the source to be summarized—paragraphs, chapters, and the
whole content in a book. Multi-LexSum presents a new opportunity to study how
to learn from and produce summaries at varying granularity for the same
source, as we explore in our experiments.

Another key differentiating factor in Multi-LexSum is that its summaries are
expert-provided. In order to scale to impressive sizes, many existing
summarization datasets are created in a (semi-)automatic fashion—e.g., using
the first sentence [44] or summary bullets [43] as the target summary for a
piece of news, or automatically extracting and linking scientific paper
abstracts [11] and citing sentences [14]. These datasets lack a clear
specification of how the summary corresponds to the source, can have varying
quality, and often contain information that is not directly supported or
implied by the source, which can degrade the factual consistency of models
trained on the data [33, 40]. By contrast, Multi-LexSum contains “gold”
summaries. Experts are specifically trained to write the case summaries
following carefully crafted instructions (detailed in Appendix B), and the
written summaries are subsequently reviewed to ensure correctness and
stylistic consistency.

Compared to many existing single- or multi-doc summarization datasets for news
[44, 43, 15], scientific papers [11, 14, 39], patents [52], legislative bills
[34], and government reports [28], the summary context in Multi-LexSum comes
from multiple sources that are extraordinarily long—over 75k words, an order
of magnitude larger than most other datasets (see Table 3.1). One exception is
BookSum [35], which uses entire books as summary inputs; the books are on
average 127,000 words long. However, it has far fewer samples (403) than
Multi-LexSum does (4,500).

##  3 Multi-LexSum

###  3.1 Task definition

In the American legal system, civil lawsuits (“cases”) involve a set of
actions among two or more parties and the judge(s) [12]. Most steps in the
case are taken by way of formal document filings. The first step typically
occurs when the “plaintiffs”—people, groups of people, or entities—file a
“complaint” against one or more “defendants” in a state or federal trial
court. The case then proceeds as the parties file additional documents. It is
through these documents that the parties lay out the case background, explain
their arguments, rebut opposing parties’ arguments, and ask for specific
actions from the judge(s) (see Table 7 for a breakdown of document types). The
judge(s) also file documents which set schedules, ask questions, and
memorialize rulings—intermediate orders that frame the conflict or instruct
parties to take various steps or "final" orders that at least temporarily
resolve the case [13]. All a result, a case’s documents can extend to
hundreds, even thousands, of pages of text. Collectively, these documents
paint a full picture of the case, but they can be extremely time-consuming to
read and digest in order to understand the gist.

The goal of legal case summarization is to write a short article that captures
principal details and describes each case’s litigation history in plain
language—information that is otherwise often difficult to come by. The CRLC
summaries come in three different lengths:

  * •

Long (l) summaries typically contain multiple paragraphs, covering the case
background, parties involved, and proceedings. Major case events and outcomes
typically receive a paragraph each.

  * •

Short (s) summaries have only one paragraph with a shorter description of the
background, parties involved, and the outcome (so far) of the case.

  * •

Tiny (t) summaries are one-sentence summaries intended to appear on Twitter to
describe the case at a specific point in its history.

Figure 1: The pipeline of creating summaries and structured data for lawsuits.
Table 2: Comparison of Multi-LexSum to other single-document (SDS) and multi-
document (MDS) summarization datasets. Measurements include dataset size,
number of source documents per sample, number of words and sentences in source
and target texts, and source-target coverage, density, and compression ratio.
Except for number of samples, all reported values are averages across all
samples, including test sets when available.

|  | Source |  | Target |  | Source →Target
---|---|---|---|---|---|---
Dataset | Samples | Docs | Words | Sents |  | Words | Sents |  | Coverage | Density | Compress
Tiny
XSUM | 226,677 | 1 | 454.5 | 31.7 |  | 24.0 | 1.0 |  | 0.67 | 1.10 | 19.9
SciTLDR | 3,229 | 1 | 5847.7 | 232.3 |  | 22.2 | 1.1 |  | 0.95 | 4.85 | 310.8
Newsroom | 1,212,739 | 1 | 800.1 | 37.9 |  | 31.2 | 1.5 |  | 0.83 | 9.53 | 43.6
BookSum/Paragraph1 | 147,665 | 1 | 163.131 | 8.411 |  | 35.8 | 1.9 |  | 0.51 | 0.90 | 6.7
1-12[.4pt/1pt] Multi-LexSum $\textsc{d}\rightarrow\textsc{t}$ | 1,603 | 10.7 | 119072.6 | 5962.5 |  | 24.7 | 1.4 |  | 0.92 | 2.27 | 5449.6
Short
BigPatent | 1,341,362 | 1 | 3629.0 | 131.4 |  | 116.7 | 3.5 |  | 0.86 | 2.38 | 36.8
MS^2 | 16,212 | 24.0 | 7775.6 | 306.3 |  | 65.1 | 3.9 |  | 0.86 | 1.91 | 174.8
Multi-XScience | 40,528 | 5.1 | 817.0 | 32.0 |  | 119.7 | 4.9 |  | 0.67 | 1.30 | 7.7
CNN / Daily Mail | 311,971 | 1 | 805.2 | 39.3 |  | 59.9 | 6.1 |  | 0.85 | 3.49 | 14.9
BillSum | 23,455 | 1 | 1804.1 | 54.4 |  | 218.4 | 6.4 |  | 0.90 | 4.05 | 12.9
1-12[.4pt/1pt] Multi-LexSum $\textsc{d}\rightarrow\textsc{s}$ | 3,138 | 10.3 | 99378.2 | 5017.0 |  | 130.2 | 5.1 |  | 0.96 | 3.33 | 840.7
Long
Multi-News | 56,216 | 2.8 | 2168.1 | 92.2 |  | 264.0 | 10.4 |  | 0.83 | 5.01 | 8.2
BookSum/Chapter | 12,570 | 1 | 5339.6 | 302.1 |  | 421.0 | 21.7 |  | 0.78 | 1.47 | 16.6
BookSum/Book | 403 | 1 | 126537.2 | 6964.2 |  | 1163.1 | 56.0 |  | 0.90 | 1.79 | 146.3
1-12[.4pt/1pt] Multi-LexSum $\textsc{d}\rightarrow\textsc{l}$ | 4,534 | 8.8 | 75543.2 | 3814.2 |  | 646.5 | 28.8 |  | 0.94 | 4.07 | 97.4

  * 1

The BookSum number might be slightly different from those reported in original
paper because some samples weren’t successfully downloaded using the script
provided by the authors.

Using the different granularities of summary, we define a variety of distinct
summarization tasks. First, we consider three different multi-document
summarization (MDS) tasks that map from the source documents d to each of the
summary lengths above (e.g., $\textsc{d}\rightarrow\textsc{l}$ denotes the
task of mapping the source documents to the long summary). We also consider
three different single-document summarization (SDS) tasks that take a ground
truth summary as input and attempt to map to a shorter summary as output
(e.g., $\textsc{s}\rightarrow\textsc{t}$ denotes mapping from a short summary
to the corresponding tiny one). Finally, the multiple granularities in Multi-
LexSum create the opportunity to use sets of the data as input or output,
which we also explore (e.g. $\\{\textsc{l},\textsc{d}\\}\rightarrow\textsc{t}$
denotes taking a long summary and the source documents as input, and
outputting a tiny summary).

###  3.2 Creating Multi-LexSum summaries

All the data in Multi-LexSum, including the selected documents, summaries, and
the structured case metadata, are manually curated by legal experts: legal
scholars, attorneys, and law students who receive specialized training
relevant to their CRLC assignments. For a case where not much has happened
since the lawsuit was filed, it typically takes one to two hours for an
inexperienced law student to read source documents and write the summary.
Summarization of more developed cases requires more time—around two to four
hours. Even an experienced attorney might spend ten or more hours to
understand and summarize an unusually complex case.

Figure 1 illustrates the summary writing pipeline. After receiving a specific
lawsuit assignment, the summary writer reads through court documents,
especially the docket, which contains a chronological list of every document
filed (Appendix B.1). From the massive document collection, the summary writer
selects a small subset of documents (on average, eight) that provide
information about major events, and attaches them to the case in CRLC.

The summary writing then takes place, guided by the instructions defined in
Appendix B.2. To ensure the coverage of the principal information in a case,
writers can resort to a checklist of facts they need to include for the case.
They typically write the long summary first, and create the short and tiny
versions after, with the option to refer to the source documents as well as
the longer summaries. Cases can last a long time—sometimes several decades—so
the summaries may be updated with new material several or many times as the
case progresses. Table 1 shows summaries for the case EEOC v. Ford Motor
Company.

After a summary writer has completed a draft of a summary, another lawyer or
law student with additional experience and specialized training reviews the
summary for accuracy and readability. When needed, the reviewer edits the
summary to ensure it is factually correct and conforms with the writing style
guideline (Appendix B.3).

###  3.3 Dataset characterization

Table 3.1 compares key measurements between Multi-LexSum and other SDS and MDS
datasets. We report dataset sizes and the average number of source documents
per sample (which is 1 for SDS datasets). To calculate average number of words
and sentences in the source document(s) and target summaries, we use the SpaCy
library [27] en_core_web_sm model. Finally, we provide average extractive
fragment coverage, density, and compression ratio, as defined by Grusky et al.
[22]. Multi-LexSum is distinct in that its source text and long target
summaries are much lengthier than existing datasets, with the exception of
BookSum333We compare with the full book summarization task in BookSum given it
has the similar source and target lengths. which has far fewer samples and
focuses on the literature domain. Long source text poses a challenge for
identifying the salient information to include in the summary.

We find that Multi-LexSum’s summaries have a high fraction of terms that also
appear in the source, but are still abstractive. We follow Grusky et al.
[22]’s approach that analyzes the coverage and density based on extractive
fragments, which are shared spans of tokens that can be jointly identified in
the document and summary. Multi-LexSum has the top coverage for long
summaries, of 0.94, meaning that 94% of the words in the summary can be found
in the extractive fragments from the corresponding source documents. The
generally high coverage for all Multi-LexSum granularities suggests that its
summaries contain fewer unsupported entities and facts compared to the
datasets with lower coverage. At the same time, the density (or average length
of the extractive fragments) ranges from 2-4 for Multi-LexSum, suggesting that
most of the summary sentences are not verbatim extractions from the sources
and are instead abstractive.

##  4 Experiments

Our experiments on Multi-LexSum focus on two questions: (1) can models
generate and synthesize information from the massive source documents in MDS
tasks ($\textsc{d}\rightarrow\textsc{l}$, $\textsc{d}\rightarrow\textsc{s}$,
and $\textsc{d}\rightarrow\textsc{t}$); and (2) can models be configured to
produce summaries of the desired lengths and details for SDS tasks
($\textsc{l}\rightarrow\textsc{s}$, $\textsc{l}\rightarrow\textsc{t}$, and
$\textsc{s}\rightarrow\textsc{t}$)?

###  4.1 Experimental Setup

We split all cases into train (70%, 3177 samples), Dev (10%, 454), and Test
(30%, 908). All cases have long summaries, and 70% and 36% of the cases have
short or tiny summaries, respectively. The corresponding source and target
document lengths are reported in Table 3.1. Appendix E provides extra details
about split sizes and how the splits are determined.

Table 3: Performance of baseline models on different MDS tasks in Multi-
LexSum.

| $\textsc{d}\rightarrow\textsc{l}$ |  | $\textsc{d}\rightarrow\textsc{s}$ |  | $\textsc{d}\rightarrow\textsc{t}$
---|---|---|---|---|---
Models | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$ | Words |  | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$ | Words |  | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$ | Words
First $k$ Sent | 30.41 | 9.67 | 14.50 | 9.73 | 813.7 |  | 21.97 | 7.17 | 13.61 | -1.60 | 120.5 |  | 8.69 | 1.07 | 6.81 | -8.72 | 33.9
Random $k$ Sent | 35.62 | 9.15 | 13.78 | 10.91 | 838.7 |  | 24.37 | 3.79 | 12.92 | 5.28 | 141.4 |  | 10.39 | 0.84 | 7.78 | 0.91 | 45.9
BERT-EXT | 36.94 | 10.65 | 14.43 | 12.85 | 860.3 |  | 24.79 | 5.48 | 13.89 | 4.49 | 137.3 |  | 9.74 | 1.12 | 7.50 | -5.43 | 39.3
PEGASUS | 40.79 | 20.01 | 25.36 | 34.83 | 203.8 |  | 43.35 | 19.91 | 29.99 | 37.88 | 94.6 |  | 22.61 | 7.09 | 18.44 | 26.78 | 22.3
BART | 48.79 | 23.78 | 28.73 | 39.55 | 351.3 |  | 43.55 | 19.98 | 29.84 | 37.41 | 114.0 |  | 23.28 | 8.19 | 19.36 | 27.84 | 21.7
LED-4096 | 47.75 | 24.13 | 28.89 | 39.10 | 295.0 |  | 45.44 | 21.00 | 30.99 | 39.33 | 103.1 |  | 26.52 | 8.92 | 21.47 | 30.01 | 22.4
LED-16384 | 49.07 | 25.17 | 29.40 | 40.05 | 310.1 |  | 46.54 | 22.08 | 31.91 | 40.00 | 108.0 |  | 28.11 | 9.84 | 22.88 | 31.41 | 21.5
PRIMERA | 53.73 | 27.32 | 30.89 | 42.01 | 416.3 |  | 45.51 | 21.04 | 30.81 | 39.32 | 110.2 |  | 27.56 | 9.26 | 21.84 | 30.40 | 27.9

#### Models

We experiment with summarization models that are representative of the state-
of-the-art. BART [36] and PEGASUS [60] are two recent abstractive summarizers
based on the Transformer architecture [55] and have achieved state-of-the-art
performance on multiple summarization datasets. Owing to the large multi-
document source content in Multi-LexSum, we also experiment with two recent
summarizers tailored to this setting. Longformer Encoder and Decoder (LED) [1]
and PRIMERA [58] are two models that can handle longer inputs (16384 and 4096
tokens, respectively) by introducing sparsity into attention layers, and
PRIMERA adds an MDS-specific pre-training objective to improve performance on
MDS tasks.

#### Implementation and Computational Resources

For abstractive summarizers, we finetune the models based on the PyTorch [46]
implementations from the HuggingFace library [56]. For each task, the models
are trained for 6 epochs on two RTX A6000 GPUs from an internal cluster, with
a learning rate of 5e-5.444For LED training, we use 3e-5 following authors’
recommendations [1] and gradient checkpointing [9] to reduce GPU memory usage.
Following previous work [35], we use beam search with 5 beams and n-gram
repetition blocks for n>3 when decoding the generation outputs. The total GPU
hours used for training all the benchmark models sum to roughly 300 hours.

#### Automatic Evaluation

Rouge-{1,2,L} [37] and BERT Score (BS) [61] are used to compute the lexical
and estimated semantic overlap between the generated and gold summaries. We
use the DeBERTA [24] model for sentence embedding following the authors’
suggestion. We report the average of F1 measures for Rouge and BS on the test
set, and include the number of generated words for reference.

###  4.2 Multi-doc legal case summarization

Table 3 lists model performances on the three MDS tasks, in which the
summarizers are challenged to fetch key information from the extraordinarily
long input strings. We test a set of extractive baselines: following [51], we
develop two extractive heuristics that select the first $k$ or random $k$
sentences from the source documents ($k=35,6,2$ for
$\textsc{l},\textsc{s},\textsc{t}$, respectively). We compare them with the
BERT-Extractive-Summarizer (BERT-EXT) [41], which embeds the source documents
using sentence Transformers [50] and selects $k$ key sentences from the
embedding clusters. The best performing extractive models are worse than the
abstractive counterparts (PEGASUS) by 47%, 64%, 84% in terms of Rouge-2, and
the magnitude increases as the target is more abstractive. Because the
sentence embedder is not trained for legal text, BERT-EXT attains similar (or
worse in the case of $\textsc{d}\rightarrow\textsc{s}$) performance to the two
extractive heuristics.

For abstractive summarizers, models that allow long inputs (LED and PRIMERA)
perform better than BART and PEGASUS (with only 1024 input tokens at most) on
all three tasks, indicating the helpfulness of the longer input context.
Because LED and PRIMERA models provide pre-trained weights with different max
input lengths (16384 and 4096, respectively), we test two variants of LED
(LED-16384 and LED-4096) with the corresponding input lengths. The longer
input length brings consistent performance improvements for LED across the
three tasks, ranging from +4% to +10% of Rouge-2 in
$\textsc{d}\rightarrow\textsc{t}$. PRIMERA outperforms even the LED-16384
model on the $\textsc{d}\rightarrow\textsc{l}$ task, but achieves similar
results as LED-4096 on the other two tasks of shorter targets, aligned with
the authors’ observation [58].

All the summarizers fail to generate long summaries of lengths that match the
human summaries—PRIMERA produces the longest summaries of 416 words on
average, less than 65% of the ground-truths’ average length of 647—while their
generations for short and tiny summaries can match the gold label lengths (130
and 25 words on average). This highlights the limitations of existing
summarizers in producing long abstractive summaries, as required for Multi-
LexSum.

###  4.3 Generating shorter summaries from the longer version

To further explore the multiple granularities of summary in Multi-LexSum, we
train summarizers to generate shorter summaries from the longer versions.
Shown in Table 4(a), models trained on the $\textsc{l}\rightarrow\textsc{s}$,
$\textsc{l}\rightarrow\textsc{t}$, and $\textsc{s}\rightarrow\textsc{t}$ task
show significant improvements compared to their $\textsc{d}\rightarrow*$
counterparts: for example, the Rouge-2 of PEGASUS is improved by 79%, 104%,
and 128%, respectively, and exceeds scores from “long-input” models like LED
and PRIMERA. The model performance in $\textsc{s}\rightarrow\textsc{t}$ is
better than $\textsc{l}\rightarrow\textsc{t}$, providing further evidence that
inputs with more condensed information simplify the summarization task.

The high summary quality when condensing long summaries to shorter ones
suggests a strategy for leveraging training summaries at multiple
granularities—a pipeline approach where one model generates a long summary,
which is used as input in another model to generate a shorter summary. As an
initial test, we train PRIMERA models for each of
$\\{\textsc{l},\textsc{d}\\}\rightarrow\textsc{s}$ and
$\\{\textsc{s},\textsc{d}\\}\rightarrow\textsc{t}$, which generate a
short/tiny summary based on the corresponding long/short summary and the
source documents. We use ground-truth summaries l and s for training.
Illustrated in Table 4(b), when also provided with the gold long/short
summaries in the input at test time, PRIMER matches the performance of the
counterparts reported in 4(a). However, when we use the model in a pipeline
that does not assume a ground truth summary as input, substituting it with a
BART-generated one, the performance degrades and can be worse than the
corresponding $\textsc{d}\rightarrow*$ models by more than 20% (when
generating tiny summaries t).

Table 4: Model performance for generating shorter summaries from the longer
version. Comparing with Table 3, performance is much higher when the model is
given a ground truth summary of a different size as input.

Models | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$ | Words
---|---|---|---|---|---
$\textsc{l}\rightarrow\textsc{s}$ Fine-tuning
PEGASUS | 54.32 | 35.62 | 42.58 | 47.49 | 156.8
BART | 56.04 | 37.02 | 44.16 | 49.19 | 133.8
$\textsc{l}\rightarrow\textsc{t}$ Fine-tuning
PEGASUS | 32.86 | 14.44 | 27.20 | 34.62 | 24.6
BART | 31.65 | 13.05 | 25.52 | 33.59 | 24.0
$\textsc{s}\rightarrow\textsc{t}$ Fine-tuning
PEGASUS | 34.15 | 16.15 | 28.27 | 34.73 | 25.6
BART | 34.02 | 15.20 | 27.96 | 35.48 | 24.4

(a) Fine-tuning PEGASUS and BART on SDS tasks.

Target | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$ | Words
---|---|---|---|---|---
$\\{\textsc{l},\textsc{d}\\}\rightarrow\textsc{s}$
Gold l | 54.99 | 36.42 | 43.44 | 48.69 | 133.4
Predicted $\textsc{l}^{\prime}$ | 41.41 | 18.24 | 27.53 | 34.04 | 164.0
$\\{\textsc{l},\textsc{d}\\}\rightarrow\textsc{t}$
Gold l | 34.07 | 14.84 | 27.74 | 36.13 | 24.13
Predicted $\textsc{l}^{\prime}$ | 23.63 | 7.98 | 19.50 | 27.09 | 24.05
$\\{\textsc{s},\textsc{d}\\}\rightarrow\textsc{t}$
Gold l | 34.60 | 16.50 | 28.71 | 35.62 | 28.65
Predicted $\textsc{l}^{\prime}$ | 22.50 | 6.79 | 18.01 | 25.88 | 27.86

(b) PRIMERA models results on progressive summarization.

###  4.4 Multitask training for summaries of different lengths

Another strategy for leveraging summaries at multiple granularities is to
train _one_ model that can create summaries of different lengths. We indicate
the desired summary using prefixes [48], prepending one of “summary: long”,
“summary: short”, or “summary: tiny” to the input source when for generating
l, s, or t summaries, respectively. Table 5 compares this multitask model with
its single task counterparts. We fix the same number of training epochs, thus
multitask models are trained for more steps; however, increasing the number of
steps was not found to improve the single-task models.

We find training for three rather than two different tasks generally leads to
better performance. The added training samples bring greater performance
boosts for t summarization, which has only a third of the training samples
compared to l. s summary results are not improved much over single-task when
using three tasks, and are slightly worse using two tasks. Most interestingly,
l summarization is greatly improved (by 11-17% in the automated metrics) in
the three-task case. Since all training cases in the dataset have a long
summary, the only difference in the multi-task training is that the model is
exposed to the short and tiny views of the summary for the same cases.

##  5 Human evaluation

To assess the usability of these models, we conduct an evaluative study with
law students trained to contribute summaries to the CRLC. In all, despite
iterative efforts to improve system performance, we found today’s models
struggle to perform the task well.

#### Study Design

In coordination with the CRLC, we developed the following study setting. (1)
We scoped to only the $\textsc{d}\rightarrow\textsc{l}$ setting, which is the
most effort-intensive and could benefit the most from model-backed assistance.
(2) We used a BART model to generate summaries. (3) Participants included two
CLRC writers who edited the generations to produce summary text for 40 new
cases that aren’t present in Multi-LexSum; this process took them 180 hours in
total. (4) We recorded edits made to the summaries as well as asked writers to
rate the generation on a 4-point scale.555The rating levels were 0 (bad;
completely unusable); 1 (somewhat helpful but requiring $>$50% edits); 2
(requiring $<$50% edits); 3 (perfect; no edits needed).

Table 5: Comparing BART performance under multitask and single-task scenarios.
The three-task model improves performance over single-task models.

Lengths | Samples | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$ | Words |  | Samples | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$ | Words
---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  BART, Multitask: $\textsc{d}\rightarrow\\{\textsc{l},\textsc{s},\textsc{t}\\}$ |  |  BART, Single-task: $\textsc{d}\rightarrow\textsc{l},\textsc{d}\rightarrow\textsc{s},\textsc{d}\rightarrow\textsc{t}$
L | 6517 | 47.89 | 23.24 | 28.31 | 39.16 | 336.6 |  | 3177 | 40.79 | 20.01 | 25.36 | 34.83 | 203.8
S | 6517 | 43.80 | 20.14 | 29.89 | 38.00 | 122.6 |  | 2210 | 43.35 | 19.91 | 29.99 | 37.88 | 94.6
T | 6517 | 25.38 | 8.92 | 20.91 | 29.11 | 23.1 |  | 1130 | 22.61 | 7.09 | 18.44 | 26.78 | 22.3
|  BART, Multitask: $\textsc{l}\rightarrow\\{\textsc{s},\textsc{t}\\}$ |  |  BART, Single-task: $\textsc{l}\rightarrow\textsc{s},\textsc{l}\rightarrow\textsc{t}$
S | 3340 | 55.20 | 36.11 | 43.42 | 48.53 | 133.5 |  | 2210 | 56.04 | 37.02 | 44.16 | 49.19 | 133.8
T | 3340 | 32.51 | 13.68 | 26.46 | 35.22 | 23.1 |  | 1130 | 31.65 | 13.05 | 25.52 | 33.59 | 24.0

#### System Design

Initial feedback from CRLC experts indicated that the end-to-end generated
output of long summaries were too far from usable. Notably, they tended to
hallucinate key information (e.g., filing date or court’s name), and the
experts stated it would take longer to correct errors than to write the
summary from scratch. So we designed an alternative system based on iterative
CRLC expert feedback. System features included (1) a tool for writers to
select relevant text snippets while reading source documents, to aid the model
in salient information selection and (2) model generation of each summary
paragraph separately based on selected snippets. Given that this system was
developed in conjunction with CRLC stakeholders and greatly simplified the
computer task to improve performance, we view it as a more accurate reflection
of how modern summarization methods might be used in real-world applications.
It thus serves as a reasonable tool to assess the usability of these models.
Further details about this system can be found in the Appendix C.

#### Results

Comparing the generations to post-edited summary texts, Rouge-1, Rouge-2,
Rouge-L and BERT-Scores were 45.6, 30.0, 35.4 and 38.0, respectively; these
scores are similar to those of BART from our $\textsc{d}\rightarrow\textsc{l}$
experiments presented in 3. Yet, the system generations received a 0.43 user
rating, demonstrating the significant limitations of automated performance
metrics. Writers averaged 87 token edits per paragraph, 76% the average length
of paragraphs, and they on average extend generation lengths by 65%. Follow-up
interviews indicated the problem of erroneous or missing key fields continued
to prevent the generations from being useful.

##  6 Conclusion

In this paper, we introduce Multi-LexSum, an abstractive summarization dataset
for large-scale civil rights lawsuits from U.S federal courts. Multi-LexSum is
packed with unique features, including summaries of multiple levels of
granularity for the same source, large collections of long source documents,
and expert-authored summaries. Through a series of experiments, we find
existing summarization models struggle to produce the summaries directly from
the long source documents. The average rating of 0.43 on a 0-3 scale from
human assessments of current models also suggests substantial room for
improvement.

Multi-LexSum is not without its limitations. CRLC is more likely to include
cases where the plaintiff wins because such cases typically last longer and
receive more attention. This project is further limited to federal cases for
which dockets are available online. Performance might not generalize to under-
represented cases (e.g., where the defendant wins); we additionally provide
case metadata to facilitate future diagnosis of this bias.

We hope Multi-LexSum will aid development of real-world summarization systems
intended to assist the activities of both specialized projects like the CRLC
as well as more general sites geared toward dissemination of court documents
for the general public, e.g., https://www.courtlistener.com/recap/. More
broadly available and up-to-date case descriptions would be of enormous
assistance to reporters, advocates, and members of the general public. The
benefit would be even greater for larger “free law” projects that post
information about hundreds of thousands, rather than thousands, of lawsuits.

## Acknowledgements

We thank the following institutions and entities who generously provide the
support for the curation of the underlying Civil Rights Litigation
Clearinghouse data over its 15-year history, including: University of Michigan
Law School; Washington University in St. Louis School of Law; Center for
Empirical Research in Law; Arnold Ventures, “Improving Criminal Justice
Reformers’ Use of Litigation Information, Documents, and Insights”
(2021-2023); Vital Projects Fund, “Revamping the Civil Rights Litigation
Clearinghouse” (2021); Proteus Fund, “Revamping the Civil Rights Litigation
Clearinghouse” (2021); National Science Foundation SES-0718831, “The
Litigation Process in Government-Initiated Employment Discrimination Suits”
(2007). The construction of the Multi-LexSum dataset was also funded in part
by NSF Convergence Accelerator Award ITE-2132318.

We thank the hundreds of law student authors of the case summaries in Multi-
LexSum, listed in https://clearinghouse.net/people. We also appreciate the
advice from Adam Pah, Arman Cohan, Iz Beltagy, John Giorgi, Lucy Lu Wang,
Jonathan Bragg, Dan Weld, Sida Li, and Ruochen Zhang.

## References

  * Beltagy et al. [2020] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ , 2020.
  * Bommarito II et al. [2018] Michael J. Bommarito II, Daniel Martin Katz, and Eric M. Detterman.  LexNLP: Natural language processing and information extraction for legal and regulatory texts.  _arXiv:1806.03688 [cs, stat]_ , June 2018.  URL http://arxiv.org/abs/1806.03688.  arXiv: 1806.03688.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang.  On the Opportunities and Risks of Foundation Models.  _arXiv:2108.07258 [cs]_ , August 2021.  URL http://arxiv.org/abs/2108.07258.  arXiv: 2108.07258.
  * Chalkidis et al. [2018] Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos.  Obligation and prohibition extraction using hierarchical RNNs.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 254–259, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-2041.  URL https://aclanthology.org/P18-2041.
  * Chalkidis et al. [2019] Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras.  Neural legal judgment prediction in English.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4317–4323, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1424.  URL https://aclanthology.org/P19-1424.
  * Chalkidis et al. [2020] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos.  LEGAL-BERT: The Muppets straight out of Law School.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904, Online, 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.261.  URL https://www.aclweb.org/anthology/2020.findings-emnlp.261.
  * Chalkidis et al. [2021] Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos.  Multieurlex–a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer.  _arXiv preprint arXiv:2109.00904_ , 2021.
  * Chalkidis et al. [2022] Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras.  LexGLUE: A Benchmark Dataset for Legal Language Understanding in English.  _arXiv:2110.00976 [cs]_ , March 2022.  URL http://arxiv.org/abs/2110.00976.  arXiv: 2110.00976.
  * Chen et al. [2016] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_ , 2016.
  * Clearinghouse [2022] Civil Rights Litigation Clearinghouse.  Mission.  https://clearinghouse.net/about, 2022.  Last accessed June 5, 2022.
  * Cohan et al. [2018] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian.  A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621, New Orleans, Louisiana, 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-2097.  URL http://aclweb.org/anthology/N18-2097.
  * Courts [2022a] U.S. Courts.  Civil cases.  https://www.uscourts.gov/about-federal-courts/types-cases/civil-cases, 2022a.  Last accessed April 17, 2022.
  * Courts [2022b] U.S. Courts.  Covering civil cases – journalist’s guide.  https://www.uscourts.gov/statistics-reports/covering-civil-cases-journalists-guide, 2022b.  Last accessed April 17, 2022.
  * DeYoung et al. [2021] Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang.  MS2: Multi-Document Summarization of Medical Studies.  _arXiv:2104.06486 [cs]_ , November 2021.  URL http://arxiv.org/abs/2104.06486.  arXiv: 2104.06486.
  * Fabbri et al. [2019a] Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  _arXiv preprint arXiv:1906.01749_ , 2019a.
  * Fabbri et al. [2019b] Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev.  Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model.  _arXiv:1906.01749 [cs]_ , June 2019b.  URL http://arxiv.org/abs/1906.01749.  arXiv: 1906.01749.
  * Farzindar and Lapalme [2004] Atefeh Farzindar and Guy Lapalme.  Letsum, an automatic legal text summarizing.  In _Legal Knowledge and Information Systems: JURIX 2004, the Seventeenth Annual Conference_ , volume 120, page 11. IOS Press, 2004.
  * Galgani et al. [2014] Filippo Galgani, Paul Compton, and Achim Hoffmann.  Hauss: Incrementally building a summarizer combining multiple techniques.  _International journal of human-computer studies_ , 72(7):584–605, 2014.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92, 2021\.
  * Greenleaf et al. [1995] Graham Greenleaf et al.  Public access to law via internet: the australasian legal information institute.-paper presented at the asian pacific specials, health and law librarians conference (6th: 1995: Sydney)-.  _Journal of Law and Information Science_ , 6(1):49–69, 1995.
  * Grover et al. [2004] Claire Grover, Ben Hachey, and Ian Hughson.  The HOLJ corpus. supporting summarisation of legal texts.  In _Proceedings of the 5th International Workshop on Linguistically Interpreted Corpora_ , pages 47–54, Geneva, Switzerland, aug 29 2004. COLING.  URL https://aclanthology.org/W04-1907.
  * Grusky et al. [2018] Max Grusky, Mor Naaman, and Yoav Artzi.  Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 708–719, New Orleans, Louisiana, 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1065.  URL http://aclweb.org/anthology/N18-1065.
  * Hachey and Grover [2006] Ben Hachey and Claire Grover.  Extractive summarisation of legal texts.  _Artificial Intelligence and Law_ , 14(4):305–345, 2006.
  * He et al. [2020] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_ , 2020.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball.  Cuad: An expert-annotated nlp dataset for legal contract review.  _arXiv preprint arXiv:2103.06268_ , 2021.
  * Hermann et al. [2015] Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching Machines to Read and Comprehend.  _arXiv:1506.03340 [cs]_ , November 2015.  URL http://arxiv.org/abs/1506.03340.  arXiv: 1506.03340.
  * Honnibal et al. [2020] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd.  spaCy: Industrial-strength Natural Language Processing in Python.  2020\.  doi: 10.5281/zenodo.1212303.
  * Huang et al. [2021] Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang.  Efficient Attentions for Long Document Summarization.  _arXiv:2104.02112 [cs]_ , April 2021.  URL http://arxiv.org/abs/2104.02112.  arXiv: 2104.02112.
  * Jain et al. [2021] Deepali Jain, Malaya Dutta Borah, and Anupam Biswas.  Summarization of legal documents: Where are we now and the way forward.  _Computer Science Review_ , 40:100388, 2021.
  * Kien et al. [2020] Phi Manh Kien, Ha-Thanh Nguyen, Ngo Xuan Bach, Vu Tran, Minh Le Nguyen, and Tu Minh Phuong.  Answering legal questions by learning neural attentive text representation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 988–998, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  doi: 10.18653/v1/2020.coling-main.86.  URL https://aclanthology.org/2020.coling-main.86.
  * Kim et al. [2012] Mi-Young Kim, Ying Xu, and Randy Goebel.  Summarization of legal texts with high cohesion and automatic compression rate.  In _JSAI International Symposium on Artificial Intelligence_ , pages 190–204. Springer, 2012.
  * Kim et al. [2015] Mi-Young Kim, Ying Xu, and Randy Goebel.  A convolutional neural network in legal question answering.  In _JURISIN Workshop_ , 2015.
  * King et al. [2022] Daniel King, Zejiang Shen, Nishant Subramani, Daniel S Weld, Iz Beltagy, and Doug Downey.  Don’t say what you don’t know: Improving the consistency of abstractive summarization by constraining beam search.  _arXiv preprint arXiv:2203.08436_ , 2022.
  * Kornilova and Eidelman [2019] Anastassia Kornilova and Vlad Eidelman.  BillSum: A Corpus for Automatic Summarization of US Legislation.  _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 48–56, 2019.  doi: 10.18653/v1/D19-5406.  URL http://arxiv.org/abs/1910.00523.  arXiv: 1910.00523.
  * Kryściński et al. [2021] Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev.  BookSum: A Collection of Datasets for Long-form Narrative Summarization.  _arXiv:2105.08209 [cs]_ , May 2021.  URL http://arxiv.org/abs/2105.08209.  arXiv: 2105.08209.
  * Lewis et al. [2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Lin [2004] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * Lippi et al. [2019] Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni.  Claudette: an automated detector of potentially unfair clauses in online terms of service.  _Artificial Intelligence and Law_ , 27(2):117–139, 2019.
  * Lu et al. [2020] Yao Lu, Yue Dong, and Laurent Charlin.  Multi-xscience: A large-scale dataset for extreme multi-document summarization of scientific articles.  _arXiv preprint arXiv:2010.14235_ , 2020.
  * Maynez et al. [2020] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald.  On faithfulness and factuality in abstractive summarization.  _arXiv preprint arXiv:2005.00661_ , 2020.
  * Miller [2019] Derek Miller.  Leveraging bert for extractive text summarization on lectures.  _arXiv preprint arXiv:1906.04165_ , 2019.
  * [42] Ramesh Nallapati and Christopher D Manning.  Legal Docket-Entry Classification: Where Machine Learning stumbles.  page 9.
  * Nallapati et al. [2016] Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang.  Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond.  _arXiv:1602.06023 [cs]_ , August 2016.  URL http://arxiv.org/abs/1602.06023.  arXiv: 1602.06023.
  * Narayan et al. [2018] Shashi Narayan, Shay B. Cohen, and Mirella Lapata.  Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization.  _arXiv:1808.08745 [cs]_ , August 2018.  URL http://arxiv.org/abs/1808.08745.  arXiv: 1808.08745.
  * Pah et al. [2020] Adam R Pah, David L Schwartz, Sarath Sanga, Zachary D Clopton, Peter DiCola, Rachel Davis Mersey, Charlotte S Alexander, Kristian J Hammond, and Luís A Nunes Amaral.  How to build a more open justice system.  _Science_ , 369(6500):134–136, 2020.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * [47] Seth Polsley, Pooja Jhunjhunwala, and Ruihong Huang.  CaseSummarizer: A System for Automated Summarization of Legal Texts.  page 5.
  * Raffel et al. [2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ , 2019.
  * Ravichander et al. [2019] Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh.  Question answering for privacy policies: Combining computational and legal perspectives.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4947–4958, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1500.  URL https://aclanthology.org/D19-1500.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019\.  URL https://arxiv.org/abs/1908.10084.
  * See et al. [2017] Abigail See, Peter J. Liu, and Christopher D. Manning.  Get To The Point: Summarization with Pointer-Generator Networks.  _arXiv:1704.04368 [cs]_ , April 2017.  URL http://arxiv.org/abs/1704.04368.  arXiv: 1704.04368.
  * Sharma et al. [2019] Eva Sharma, Chen Li, and Lu Wang.  BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization.  _arXiv:1906.03741 [cs]_ , June 2019.  URL http://arxiv.org/abs/1906.03741.  arXiv: 1906.03741.
  * Shleifer and Rush [2020] Sam Shleifer and Alexander M Rush.  Pre-trained summarization distillation.  _arXiv preprint arXiv:2010.13002_ , 2020.
  * Tuggener et al. [2020] Don Tuggener, Pius von Däniken, Thomas Peetz, and Mark Cieliebak.  LEDGAR: A large-scale multi-label corpus for text classification of legal provisions in contracts.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 1235–1241, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.155.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wolf et al. [2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Xiao et al. [2021a] Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun.  Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents.  _arXiv:2105.03887 [cs]_ , May 2021a.  URL http://arxiv.org/abs/2105.03887.  arXiv: 2105.03887.
  * Xiao et al. [2021b] Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan.  Primer: Pyramid-based masked sentence pre-training for multi-document summarization.  _arXiv preprint arXiv:2110.08499_ , 2021b.
  * Yousfi-Monod et al. [2010] Mehdi Yousfi-Monod, Atefeh Farzindar, and Guy Lapalme.  Supervised machine learning for summarizing legal documents.  In _Canadian Conference on Artificial Intelligence_ , pages 51–62. Springer, 2010.
  * Zhang et al. [2020] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR, 2020.
  * Zhang et al. [2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.
  * Zheng et al. [2021] Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho.  When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset.  _arXiv:2104.08671 [cs]_ , July 2021.  URL http://arxiv.org/abs/2104.08671.  arXiv: 2104.08671.
  * Zhong et al. [2020] Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun.  Jec-qa: A legal-domain question answering dataset.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9701–9708, 2020.
  * Zhong et al. [2019] Linwu Zhong, Ziyi Zhong, Zinian Zhao, Siyuan Wang, Kevin D Ashley, and Matthias Grabmair.  Automatic summarization of legal decisions using iterative masking of predictive sentences.  In _Proceedings of the seventeenth international conference on artificial intelligence and law_ , pages 163–172, 2019.

##  Appendix A Multi-LexSum release

###  A.1 Accessing Multi-LexSum

The dataset source files are stored in JSON format, and they are uploaded to
Amazon S3 and can be downloaded publicly. Following the HuggingFace datasets
library,666https://github.com/huggingface/datasets we develop one Python
script (the `multi_lexsum.py` file) that handles both the downloading of the
source files and loading them into easily usable format:

⬇

1from datasets import load_dataset

2multi_lexsum = load_dataset("multi_lexsum.py", name="v20220616")

3# Download multi_lexsum locally and load it as a Dataset object

4example = multi_lexsum["test"][0] # The first instance of the test set

5example["sources"] # A list of source document text for the case

6for sum_len in ["long", "short", "tiny"]:

7 print(example["summary/" + sum_len]) # Summaries of three lengths

Currently, the `multi_lexsum.py` file can be retrieved from Multi-LexSum’s
GitHub repository: https://github.com/multilexsum/dataset. The authors are
working on incorporating the script as part of the HuggingFace datasets
library to further streamline the downloading and usage of Multi-LexSum. We
include a similar instruction on the project website,
https://multilexsum.github.io, which can be regularly updated to reflect the
latest changes and future updates or erratum to Multi-LexSum.

###  A.2 Multi-LexSum distribution and maintenance

#### License

Multi-LexSum is distributed under the Open Data Commons Attribution License
(ODC-By). The case summaries and metadata are licensed under the Creative
Commons Attribution License (CC BY-NC), and the source documents are already
in the public domain. Commercial users who desire a license for summaries and
metadata can contact<EMAIL_ADDRESS>which will allow free use but
limit summary reposting. The authors bear all responsibility in case of
violation of rights, and confirm the dataset licenses. The corresponding code
for downloading and loading the dataset is licensed under the Apache License
2.0.

#### Hosting and maintenance

The authors are committed to providing long-term support for the Multi-LexSum
dataset. At present, Multi-LexSum files are hosted on Amazon S3 by the authors
themselves. In the event when the authors are not able to host the data, we
will migrate the data to common dataset repositories (e.g., HuggingFace
Datasets) and update the documentation and code. The authors will closely
monitor the usage of the dataset, and develop necessary updates of bug fixes
when needed.

For additional details, we refer readers to the dataset documentation (or the
“datasheet” for datasets [19]) attached in Appendix F.

##  Appendix B Multi-LexSum summary writing and reviewing guidelines

###  B.1 Reading source documents

Different from multi-document summarization tasks in other domains [16, 14],
Multi-LexSum summary writers are required to read several long documents and
distill key facts therein. Strategically reading these documents saves time
and effort and also improves the chances of successfully extracting the
important information.

When summary writers first begin summarizing work on a case, they can orient
themselves in several ways:

  * •

Web searches. Summary writers are instructed to do a quick web search when
starting a case. They may find news articles or blog posts that help explain
what the case is about and developments in the case. They may also find
websites that are updated with documents filed in the case.

  * •

Recent judicial opinions. At the beginning of opinions, judges often summarize
developments in the case up to that point.

  * •

Notes from other summary writers. When someone takes on a case picked by
another person, that person usually includes notes, documents, or links to
websites about the case. These can provide context or background, provide part
of the narrative, or indicate some important events in the case.

Summary writers next skim their case’s trial court docket to get an overview.
A docket contains a chronological list of every document that is filed with a
court in a given case, whether the filer is a party, the judge(s), or someone
else. Each row or entry of a docket typically contains a number (its position
in the list), a date, a title and short description, and—in many systems,
including most federal court cases since 2003—a link to the filed document.
The description conveys what kind of document has been filed and who filed it.

Based on the description, summary writers determine whether a docket entry
relates to something important and warrants a deeper dive into the linked
document. While there can be up to hundreds of entries in a docket, the
writers are required to whittle the long list down to typically a dozen or
fewer of the documents most essential for understanding the case, which
constitutes the source documents in Multi-LexSum.777Some documents might be
helpful for understanding the case and writing the summary but do not need to
be added to the CRLC. Table 6 describes the rubric for whether to include a
document based on its type.

Table 6: Rubrics for whether to include a source document in Multi-LexSum.

Always Include | The first complaint
---|---
The last amended complaint
Settlement agreements, consent decrees, and litigated decrees
Opinions
Orders granting temporary restraining orders or preliminary injunctions
Orders granting or denying class certification
Orders awarding or denying attorneys’ fees
Monitor reports
Sometimes Include _Based on Writers’ Judgment_ | Amicus briefs
Orders that settle a contentious issue
Motions/briefs if there’s no order/opinion explaining their resolution
Rarely to Never Include | Answers
Amended complaints that are not the latest one
Orders that say nothing more than what’s on the docket
Motions/Briefs resolved by an opinion or order
Attorney appearances
Other orders

Document Type | Avg. Docs Per Case |  Description
---|---|---
_Common Document Types_
Complaint | 1.517 (0.88) |  The document that starts a case and will usually be the first thing filed. Plaintiffs can also file amended complaints to add or subtract parties or claims.
Opinion/Order | 3.300 (4.78) |  Created by judges, opinions or orders memorialize rulings in the case.
| Pleading/
---
Motion/
Brief
1.72 (7.05) |  This broadly covers documents filed by the parties in order to make requests or explain their arguments.
| Monitor/
---
Expert/
Receiver Report
0.221 (1.85) |  Reports created by non-parties to help with the litigation in various ways. A “monitor” is a court-appointed expert, usually superintending compliance with a court order; an “expert” works for one or the other side, during the litigation; a “receiver” is an entity appointed by the court to run defendant operations because the defendant has somehow demonstrated incapacity.
Settlement | 0.501 (0.83) |  An agreement among parties that resolves some or all of the issues in the lawsuit.
Press Release | 0.113 (0.46) |  What it sounds like––a press release.
Dockets | 1.089 (0.41) |  The docket is the court’s index to everything that has happened in a case, in that court.
_Less Common Document Types_
Correspondence | 0.03 (0.25) |  Letters NOT directed to the court.
(In some jurisdictions (particularly in New York City), parties will conduct
lots of litigation through letters to the court or “letter motions”—these are
classified as motions or briefs, not as correspondence,.)
| Declaration/
---
Affidavit
0.065 (0.8) |  Documents in which someone provides information under penalty of perjury.
| Discovery/
---
FOIA Material
0.018 (0.3) |  Discovery material is evidence turned over by one party to another. The Clearinghouse usually doesn’t collect it, so this document type is rare.
FOIA materials are documents produced in response to a Freedom of Information
Act (FOIA) request. These are also uncommon in the Clearinghouse.
FOIA Request | 0.003 (0.09) |  A request for information under the Freedom of Information Act or a state equivalent. This doesn’t come up much in the Clearinghouse.
| Internal
---
Memorandum
0.01 (0.12) |  An organization’s internal memo (different from litigation documents with “memorandum” in the title). This is a rare category.
| Magistrate Report/
---
Recommendation
0.044 (0.34) |  Decisions from magistrate judges.
| Statute/
---
Ordinance/
Regulation
30.018 (0.56) |  A law or rule of government entity—federal, state, city or county, or agency. This document type includes policies created by prisons, school districts, police departments, immigration authorities, etc.
Transcripts | 0.027 (0.32) |  Verbatim transcripts of court proceedings or depositions.
Other | 0.050 (0.38) |

Table 7: Different Types of Source Documents in Multi-LexSum

###  B.2 Writing summaries

The written summaries generally follow the order of events, as presented by
the docket. The best summaries tell the story of the court proceedings. The
student writes about the case’s developments, progressing through the most
important docket entries. If an entry’s document is also important, the writer
may also summarize the contents of the document as part of the narrative.

When writing the summary, writers also have a checklist of facts that they
need to include, as illustrated in Table 1.

###  B.3 Reviewing summaries

After a summary is written, a reviewer with additional training then checks
the summaries for factual accuracy. Reviewers may elect to go through a docket
and verify that the summary includes all the important entries, but more often
they just read the summary and keep an eye out for potential gaps in the
narrative, and for events that are confusing or seem implausible.

Reviewers also ensure that the writing style conforms to the general practices
as described below. In addition to checking spelling and grammar, they look
for ways to keep the writing concise and somewhat free of legal jargon. Some
examples of specific improvements:

  * •

Overall flow. A summary tells a story, and so writers are encouraged to avoid
too much repetition in terms of sentence structures. Following the chronology
presented by the docket, beginning writers often start every paragraph with
“on x date, the party did y,” but that can make a summary less interesting and
more difficult to read. In addition, while by default summaries present events
in chronological order, there are circumstances in which it makes sense for
the narrative to tell pieces of the story in a different order.

  * •

Level of detail. Documents that lay out the parties’ initial arguments, the
court’s reasoning, and the outcomes of the case can be lengthy, so students
need to make sure they are including enough detail for readers to understand
what happened while still summarizing the documents in a concise manner.

  * •

Party descriptions. Summary writers are instructed to describe the parties
beyond just their role in a case. This tends to result in including the names
of organizations or a description of individuals regarding why they would be
involved in the case (e.g., “individuals incarcerated in prison” for a case
about prison conditions).

  * •

Accurate terminology. As with any discipline, words can have a particular
meaning in the legal field, and so it is important with these summaries to
accurately convey the events of a case. These types of fixes include using the
right verbs around motions being “filed” and “granted,” as well as including
the full names of courts.

  * •

Avoiding legal jargon. These summaries are read by people other than attorneys
and law students, such as policymakers and reporters. This includes avoiding
the idiosyncratic capitalization sometimes found in legal writing and court
documents. Part of the task of summarizing a case is to translate legal
technicalities into a story more suitable for a general audience.

  * •

Adding references. If a summary discusses a judicial opinion that has a formal
citation (e.g., 123 F.2d 456—meaning volume 123 of the case reporter Federal
Reporter, second series, page 456), writers should include the citation, so
that lawyers and researchers are able to find and cite the opinion for their
own purposes. Writers may also choose to add links to news articles or blog
posts that help add detail to a summary should a reader want to learn more.

  * •

Grammar and spelling. In addition to correcting syntactical mistakes,
reviewers also ensure that acronyms are either avoided or spelled out the
first time. In addition, our style guideline requires reviewers to ensure that
the summaries are written in past tense, which beginning writers may overlook
because court documents and news articles may describe some events in present
tense. (Writing summaries in the past tense avoids revisions in later years to
change the tense.)

##  Appendix C Usability study system design

Figure 2: Illustration of the key components and functions in the user study
system.

For our human evaluation of automatic summarizers applied to Multi-LexSum, we
performed a usability study as described in Section 5. Our usability study
system is designed based on iterative feedback from CRLC experts to maximize
practical helpfulness. Illustrated in Figure 2, it breaks down case
summarization into four steps: (a) docket reading and important entry
selection, (b) summary outlining and content grouping, (c) source document
reading and extraction, and (d) summary selection/rating/editing.888We
implement the system except for the interface in (a) and (b), which is
developed based on Notion: https://www.notion.so/ and
https://github.com/lolipopshock/notion-df. The system is designed to enable
users to select relevant text snippets from the massive source documents to
aid the model in salient information selection, and to decrease the model
generation length to one paragraph at a time to reduce difficulty in both
model generation as well as human editing burden. We detail each step:

#### (a) Docket reading and important entry selection

The trial court docket is the governing document in a case: it contains a
chronological list of all documents filed with the court in the case,
including a brief text description of each document. It’s important for human
summary authors to read through the docket and arrive at a small subset of
important docket entries and documents that should be included in the summary
and may warrant further reading. This subset of candidate documents for in-
depth reading are a superset of the “source documents” included in Multi-
LexSum. The interface in Figure 2(a) supports this step by providing a tabular
interface: each row ➁ corresponds to an entry in the docket, and users can
click the checkbox in the first column ➀ when the referenced document is an
important event that may also need in-depth reading.

#### (b) Summary outlining and content grouping

After reading and selecting key source documents, human authors develop an
overall understanding of the case’s major events, each of which will generally
correspond to one paragraph in the long summary, as per the CRLC’s writing
guidelines. The interface instructs writers to create “event groups” via the
functions shown in Figure 2(b); only the candidate documents selected in the
previous step are displayed in individual “cards”. Writers specify a group
name ➀ and drag the relevant cards into that group ➁. The cards can also be
assigned to multiple groups when appropriate.

#### (c) Source document reading and extraction

The writers next begin in-depth reading for documents in each event group. In
a typical unassisted workflow, authors would manually take notes consisting
mainly of copied text snippets of key information from these source documents.
Our interface, shown in Figure 2(c), includes the selected document and
description from the docket in ➀, and ➁ allows the human writers to perform
this snippet-extraction workflow.

#### (d) Summary selection, rating, and editing

Assisted by human selection of important documents and manually extracted text
snippets, we run two summarizers, BART [36] and DistlBART [53], to produce the
draft summaries. In Figure 2(d), the writer can pick the preferred model
generation via ➀, provide the 4-point rating999Rating scale described in
Section 5. for the selected summary in ➁, and edit the summary text in ➂.

Because the models are provided human-selected salient text snippets, and are
only required to generate a single paragraph at a time, the summaries
generated in our setting are higher quality than those produced by end-to-end
systems; this was verified via feedback from the study participants. Despite
these efforts to improve generation quality, a mean rating of 0.43 for model
output can be viewed as an upper-bound on the ability of modern end-to-end
summarization models to produce usable summaries for this important task.

##  Appendix D Negative social impact

We believe that release of the Multi-LexSum dataset will have positive
scholarly and societal impact. However, there are some possible negatives:

One intended use case of Multi-LexSum is to support training automatic
summarizers for court documents. However, current summarization models are
known to often make up facts in the generated text [33, 40], and it is
difficult to differentiate between the “hallucinated” and faithful information
in the outputs. If such summarizers are deployed at scale without having
solved the hallucination problem, the factually incorrect summaries could lead
to misinterpretation of the case by anyone using the application. In addition,
the possibility of factual errors could undermine trust in the resource even
if automatic summarization is only used sparingly. Though this concern could
be resolved by future improvements of summarization models, we highlight this
risk to encourage particular care when deploying such summarizers.

Moreover, as we discussed in Section 6, the cases in Multi-LexSum are drawn
from a non-representative subset of all (civil rights) cases in U.S. courts.
Models trained on this dataset will tend to adapt the language to the cases
appeared in the dataset, which could be problematic when applied to other
types of cases, or to cases in different legal systems from other countries.
In the short term, we acknowledge this could lead to an “unfair” development
of NLP methods that work only for certain types of cases (though we note that
the cases in question are of both particularly high public interest and,
because they are non-commercial, are unlikely to spur private profit-driven
development). We strongly endorse efforts to increase the transparency of the
court system, including free release of court documents and case summaries for
other types of lawsuits in the U.S. and for different legal systems.101010For
example, the SCALES-OKN project [45].

##  Appendix E Multi-LexSum train-test split

Table 8: Multi-LexSum train-dev-test splits. |  Source d |  Long l |  Short s |  Tiny t | Total
---|---|---|---|---|---
Train (70%) | 28,557 | 3,177 | 2,210 | 1,130 | 6,517
Test (20%) | 7,428 | 908 | 616 | 312 | 1,836
Dev (10%) | 4,134 | 454 | 312 | 161 | 927
Table 9: The average and standard deviation of BART models’ performance on different test splits. Strategy | Test Split | $\text{R-1}_{f1}$ | $\text{R-2}_{f1}$ | $\text{R-L}_{f1}$ | $\text{BS}_{f1}$
---|---|---|---|---|---
Hold-out | 10% | 47.21(0.34) | 23.05(0.17) | 28.21(0.34) | 38.72(0.30)
15% | 47.00(0.13) | 22.89(0.13) | 27.94(0.14) | 38.30(0.26)
20% | 47.25(0.02) | 23.00(0.08) | 27.99(0.10) | 38.29(0.12)
K-fold | 20% | 47.24(0.49) | 23.12(0.25) | 28.23(0.29) | 38.79(0.33)

We randomly split all cases in Multi-LexSum into Train (70%), dev (10%) and
test (20%) sets, with detailed statistics reported in Table 8. The split
strategy is verified in terms of (1) whether the test split is large enough
for robust evaluation and (2) whether models are sensitive to a specific
splitting of the data. We train and evaluate BART models under different split
settings with the same hyperparameters, and we report the average and standard
deviation of the ROUGE scores across different splits.

#### Determining the optimal test split size

We test the same model (trained on 70% of the data) on different test split
sizes (10%, 15%, and 20%), which we refer to as the “hold-out” strategy. Table
9 shows that the increased test split size leads to more stable test results
(lower standard deviation). We decide to use 20% as the optimal test split
size, while keeping 70% of the dataset for training.

#### Verifying model stability on different splits

We conduct a 5-fold cross-validation experiment, where each time the model is
tested on one fold and trained on the other folds. Given our decision from the
earlier step to use 70% of the total data for training, for the purposes of
this experiment, we also cap the amount of training data used in the cross
validation experiments to this amount. Shown in Table 9, the low standard
deviation indicates the models are not sensitive to specific random splits.
Note that we used slightly different hyperparameters for this experiment than
those for the main results reported in Table 3.

##  Appendix F Multi-LexSum datasheet

Please see next page.

See pages - of datasheet.pdf