state and (2) the claim arises out of the nexus of the defendant’s contacts
with the state.

Under this rule, there are two paths for a court have jurisdiction over a
defendant: through domicile or through contacts.

  * •

Domicile: A defendant is domiciled in a state if they are a citizen of the
state (i.e. they live in the state). Changing residency affects a change in
citizenship.

  * •

Contacts: Alternatively, a court may exercise jurisdiction over a defendant
when that defendant has sufficient contacts with the court’s forum, and the
legal claims asserted arise from the nexus of the defendant’s contacts with
the state. In evaluating whether a set of contacts are sufficient, lawyers
look at the extent to which the defendant interacted with the forum, and
availed themselves of the benefits and privileges of the state’s laws.
Behavior which usually indicates sufficient contacts include: marketing in the
forum or selling/shipping products into the forum. In assessing nexus, lawyers
ask if the claims brought against the defendant arise from their contacts with
the forum. In short: is the conduct being litigated involve the forum or its
citizens in some capacity?

##### Task

The personal jurisdiction task requires an LLM to determine—given a fact
pattern describing the events leading up to a legal claim—whether a particular
court has personal jurisdiction over the defendant.

##### Construction process

We manually construct a dataset to test application of the personal
jurisdiction rule, drawing inspiration from exercises found online and in
legal casebooks. Each sample in our dataset describes a “fact pattern," and
asks if a court located in particular state (A) can exercise personal
jurisdiction over an individual (B) named in the fact pattern. In designing
the dataset, we use 5 base fact patterns, and create 4 slices, where each
slice evaluates a different aspect of the personal jurisdiction rule:

  * •

Domicile: Fact patterns where B is domiciled in A. Hence, personal
jurisdiction exists.

  * •

No contacts: Fact patterns where B has insufficient contacts with A. Hence
there is no personal jurisdiction.

  * •

Yes contacts, no nexus: Fact patterns where B has sufficient contacts with A,
but the claims against B do not arise from those contacts. Hence, personal
jurisdiction does not exist.

  * •

Yes contacts, yes nexus: Fact patterns where B has sufficient contacts with A,
and the claims against B arise from those contacts. Hence, there is personal
jurisdiction.

Caveat. Personal jurisdiction is a rich and complex doctrine. Our dataset
focuses on a narrow class of fact patterns, related to jurisdiction over
individuals. We don’t consider, for instance, more complex questions related
to adjudicating citizenship (e.g. the Hertz test) or the classic stream-of-
commerce problems. We leave this to future work.

Facts |  Personal jurisdiction?
---|---
Dustin is a repairman who lives in Arizona and repairs computers in California, Oregon, and Washington. Dustin is an avid skier, so his favorite place to go on vacation is Colorado. While travelling to repair a computer in Washington, Dustin is involved in a car crash in Oregon with Laura, a citizen of Oregon. After the accident, Dustin returns to Arizona. Laura sues him in Colorado. |  No
David is a citizen of California. He flies to New York for a vacation, where he meets Maggie, who is also visiting from Rhode Island. While they chat, Dave fraudulently tricks Maggie into giving him her savings. David then continues his vacation and visits Texas, Oregon, Florida, and New Mexico. After he returns home, Maggie sues David for fraud in Oregon. |  No
Ana is a lawyer who resides in Texas. While visiting Louisiana, she meets David, who runs a bike shop. David’s bike shop is famous, and he frequently advertises his bikes in Texas newspapers. Ana buys a bike from David and rides it back home. Right after she crosses the border, the bike seat explodes, injuring Ana. Ana sues David in Texas. |  Yes
Tony (from Texas) is a regional manager for a cookbook company, Tasty Eats Books (incorporated and principal place of business in Delaware). Tony’s job requires him to travel from city to city to show new cookbooks to chefs. In January 2022, he was scheduled to visit restaurants in Illinois, Indiana, and Michigan. While in Michigan, Tony goes to Lake Erie to blow off some steam. He ends up getting into a fight with Arthur, a lawyer from Detroit, Michigan. Tony and Arthur each blame the other for starting the fight. Arthur sues Tony in Texas. |  Yes
Table 41: Examples for personal_jurisdiction.

##### Significance and value

Identifying when personal jurisdiction exists is a skill that law students
learn in their first-year civil procedure course. The personal jurisdiction
task is interesting because applying even the simplified version of the rule
requires reasoning over the degree of connection between a defendant and the
forum state.

###  F.22 Privacy Policy Entailment

In LegalBench, the Privacy Policy Entailment task is denoted as
privacy_policy_entailment.

##### Background

The Privacy Policy Entailment task is created from the APP-350 corpus [161],
which consists of 350 Android app privacy policies annotated with different
privacy practices. In this corpus, individual clauses are annotated based on
whether they do or do not perform a certain practice (e.g., “We access your
location information”).

##### Task

Given a clause from a privacy policy and a description of the practice, the
LLM must determine if the clause describes the performance of that practice.
This is analagous to an entailment task, where the premise is the clause, and
the hypothesis is the practice description.

##### Construction process

For each practice coded in the APP-350 corpus, we derive a natural language
description of that practice, which serves as our “hypothesis.” Each instance
of this task corresponds to a triple containing a clause, a practice
description, and a binary classification (Yes/No) based on whether the clause
performs the practice. Across the dataset there are 57 unique policy
descriptions.

Clause |  Description |  Performed?
---|---|---
We may collect and record information through the SN Service in accordance with the policies and terms of that SN Service. The information we collect when you connect your user account to an SN Service may include: (1) your name, (2) your SN Service user identification number and/or user name, (3) locale, city, state and country, (4) sex, (5) birth date, (6) email address, (7) profile picture or its URL, and (8) the SN Service user identification numbers for your friends that are also connected to Supercell’s game(s). |  The policy describes receiving data from an unspecified single sign on service |  Yes
Your e-mail address will not be stored. |  The policy describes collection of the user’s e-mail by a party to the contract. |  No
Table 42: Example clause-description-label pairs for Privacy Policy Entailment
task.

##### Significance and value

The privacy policy entailment task is similar to ContractNLI, in that it
evaluate a LLM’s capacity to perform entailment-style reasoning over formal
legal language. From a lawyerly perspective, understanding whether a policy
performs certain functions or empowers one of the parties to pursue practices
is an essential element of legal comprehension. From a practical perspective,
the ability for LLMs to perform this task could empower researchers to conduct
broader studies of privacy agreements. As [161] observes, annotation cost
limitations often restrict the scope of empirical studies of privacy
agreements.

###  F.23 Privacy Policy QA

In name, the Privacy Policy QA task is denoted as privacy_policy_qa.

##### Background

The Privacy Policy QA task is derived from [112], which annotated clauses in
mobile application privacy policies based on whether they contain the answer
to a question.

##### Task

Given an excerpt from a privacy policy and a question, the LLM must determine
whether the excerpt is relevant to answering the question or not.

##### Construction process

We used the snippet annotations available in [112] to construct this task,
removing all snippets with fewer than 10 words. Examples of
excerpt/question/relevant tuples are shown in the table below. 5449 instances
correspond to a relevant question-clause pair, and 5474 instances correspond
to an irrelevant question-clause pair.

Excerpt |  Question |  Class
---|---|---
We also use cookies, tags, web beacons, local shared objects, files, tools and programs to keep records, store your preferences, improve our advertising, and collectNon-Identifying Information, includingDevice Dataand information about your interaction with the Site and ourBusiness Partners’web sites. |  is my search and purchase history shared with advertisers? |  Relevant
We collect information about the value added services you are using over Viber and/or apps (such as games) you have downloaded through Viber. |  does viber sell my information to advertisers and marketers? |  Irrelevant
Table 43: Examples for Privacy Policy QA

##### Significance and value

Determining when a particular legal excerpt is relevant to answering a
question is essential to interpreting legal documents. This task allows us to
evaluate LLMs for this capability. From a more practical standpoint, the
Privacy Policy QA task is a helpful evaluation task when developing LLM
systems which involve decompositions over long documents. A common approach—in
order to account for the fact that many long documents exceed ordinary context
windows—is to chunk documents into smaller segments, and apply a LLM
independently to filter out irrelevant segments. For QA tasks involving long
policies, this task allows practitioners to measure performance for the
filtering step.

###  F.24 Private Right of Action (PROA)

In LegalBench, the Private Right of Action task is denoted as proa.

##### Background

A private right of action (PROA) exists when a statute empowers an ordinary
individual (i.e. a private person) to legally enforce their rights by bringing
an action in court. In short, a PROA creates the ability for an individual to
sue someone in order to recover damages or halt some offending conduct. PROAs
are ubiquitous in antitrust law (in which individuals harmed by anti-
competitive behavior can sue offending firms for compensation) and
environmental law (in which individuals can sue entities which release
hazardous substances for damages) [51].

##### Task

In the PROA task, an LLM must determine if a statutory clause contains a
private right of action.

##### Construction process

We construct a dataset of PROAs by hand, drawing inspiration from clauses
found in different state codes. We construct 50 clauses which do contain a
PROA, and 50 clauses which do not. Clauses which do not contain a private
right may either create no cause of action, or empower a non-private
individual (e.g., an attorney general) to bring a claim. 5 randomly sampled
clauses constitute the training set, and the remaining 95 form the test set.

Input |  Answer
---|---
The attorney general or an attorney representing the state may bring an action for an injunction to prohibit a person from violating this chapter or a rule adopted under this chapter. |  No
The administrator may bring an action in a court of competent jurisdiction to enforce this chapter. |  No
The sheriff or the sheriff’s designee shall maintain a permanent personnel file oneach department employee. |  No
If any laborer, without good cause, shall abandon his or her employer before the expiration of his or her contract, he or she shall be liable to his or her employer for the full amount of any account he or she may owe his or her employer. |  Yes
No employer may discharge any employee by reason of the fact that earnings have been subjected to garnishment or execution. If an employer discharges an employee in violation of this section, the employee may within ninety days of discharge bring a civil action for recovery of twice the wages lost as a result of the violation and for an order requiring reinstatement. |  Yes
In addition to all other penalties, rights, or remedies provided by law, an individual or entity that uses or attempts to use its official authority or influence for the purpose of interfering with the right of a legislative employee to make a protected disclosure is liable in a civil action for damages brought by a legislative employee. |  Yes
Table 44: Examples for proa

##### Significance and value

The PROA task evaluates an LLM’s ability to perform a two-step reasoning test:
(1) does the statute allow a party to bring a claim in court, and (2) is that
party private? Law students and legal professionals should be capable of
performing this task at near-perfect accuracy.

The PROA task derives additional significance from a recent movement towards
studying state statutory language [155]. Legal scholars have long been unable
to conduct large scale empirical studies of state statutory language, given
the sheer volume of state statutes. The ability for LLMs to accurately
classify or annotate statutes could thus empower new empirical studies of
state statues.

###  F.25 Rule QA

In LegalBench, the Rule QA task is denoted as rule_qa.

##### Background

Lawyers are regularly required to recall specifical legal rules that are drawn
from cases, statutes, or other sources. Rules can take many shapes and forms.
For instance, the rule pertaining to the federal requirements for a class (in
a class action lawsuit) are codified in Rule 23(a) of the Federal Rules of
Civil Procedure and are simply known as the need for “numerosity, commonality,
typicality, and adequacy.”

##### Task

The Rule QA task evaluates a LLM’s ability to answer questions on different
legal rules. The rules are drawn from subjects typically studied in the first
year of law school (e.g., civil procedure, constitutional law, etc.). This is
an open-generation task.

##### Construction process

We manually wrote 50 question-answer pairs, focusing on the types of rules
which are regularly tested in law school courses on civil procedure, evidence,
and intellectual property. The questions ask the LLM to either (1) restate a
rule, (2) identify where a rule is codified, or (3) list the factors employed
in a particular rule. Several questions explicitly narrow their scope to a
jurisdiction (e.g., California state evidence law), in order to avoid bias
towards merely federal law.

Question |  Answer |  Area of law
---|---|---
What are the four categories of patentable subject matter? |  “process, machine, manufacture, or composition of matter.” |  IP
What are the requirements for diversity jurisdiction? |  Diversity jurisdiction exists when the amount in controversy exceeds $75,000 and the plaintiffs and defendants are completely diverse (i.e. no plaintiff shares a state of citizenship with any defendant) |  Civil Procedure
Under which statute are patentable subject matter requirements codified? |  35 USC 101 |  IP
What are the factors of the Mathews balancing test? |  A three-part test that determines whether an individual has received due process under the Constitution. The test balances (1) the importance of the interest at stake; (2) the risk of an erroneous deprivation of the interest because of the procedures used, and the probable value of additional procedural safeguards; and (3) the government’s interest. |  Constitutional law
Table 45: Examples for Rule QA

##### Significance and value

The Rule QA task is an initial effort to evaluate the propensity for legal
hallucination in LLMs. The questions asked are exceedingly basic, and law
students taking the relevant course would be expected to answer them nearly
perfectly.

###  F.26 SARA Tasks

In LegalBench, the SARA tasks are denoted as sara_*.

##### Background

An important skill for lawyers is to determine, given the facts of a case,
whether a given law applies and what it prescribes. For example, does the
payment received by the defendant on August 21st , 2017 qualify as wages under
§3306(b) of the US Tax Code? This task has been introduced by [78] as
statutory reasoning. [65] further introduce the Statutory Reasoning Assessment
dataset (SARA). SARA contains (1) a set of 9 sections, taken from US federal
tax law statutes, pruned and simplified; and (2) hand-crafted cases that test
the understanding of those 9 sections. In this context, a case is a paragraph
of text stating facts in plain language. Each case comes either with an
entailment prompt — a statement about the statutes and the case that may be
true or false — or a question — asking how much tax one of the case’s
protagonists owes. The SARA dataset is a simplified version of real-world
cases, that retains many of the features of statutory reasoning for tax law.
It poses, however, a significant challenge to NLP models [12].

##### Tasks

There are two SARA tasks. The first, sara_entailment, corresponds to the
entailment cases. The entailment cases state that a given law applies to a
given case, and require the LLM to produce a binary answer — akin to
Recognising Textual Entailment [40]. This is an approximation of real-world
statutory reasoning, where the answer is usually not strictly binary.

The second task, sara_numeric, consists of the numeric cases. Here, the goal
is to compute the amount of tax owed. We frame this as a floating point
number. To measure numerical accuracy, we use the metric introduced by [65],
which includes a tolerance for inaccurate predictions.

##### Construction process

We framed the SARA dataset for the paradigm of language modeling. Due to
dependencies across sections in the statutes, the entirety of the statutes are
generally relevant to determine the answer to any of the cases. However, all 9
sections do not fit into the LLM’s context window, and must be pruned. In
entailment cases, the entailment prompt specifies which law from the statutes
to apply. We automatically extract the text of that law, and use it as the
language model prompt. For numerical cases, the entirety of the statutes are
relevant, and we use that as the language model prompt. Pruning is left to the
LLM’s pre-processing.

##### Significance and value

Statutory reasoning is an important skill for lawyers, that is used within
many other legal tasks. It is a fundamental task for legal AI, probing whether
a computational model can understand and reason with legal rules expressed in
natural language. The types of reasoning involved in SARA are diverse —
defeasible, temporal, numerical reasoning, inter alia — and relevant beyond
the legal domain. Statutory reasoning combines natural language understanding
and logical reasoning, a major goal for AI.

If statutory reasoning were solved, it could serve as a basis for more complex
legal tasks. For example, it could be used to automate the computation of
taxes and benefits, without the need for coding the expert systems in use in
many parts of the world [101]. A system that can do statutory reasoning would
also be a step towards machine reading models that can analyze legislation and
anticipate its effects, coming up with possible application scenarios [11]. As
a final example, a statutory reasoning agent could be used for basic legal
advice, increasing access to justice.

Task name: sara_entailment
---
Statute: (2) an individual legally separated from his spouse under a decree of
divorce or of separate maintenance shall not be considered as married.
Description: Alice and Bob got married on April 5th, 2012. Alice and Bob were
legally separated under a decree of divorce on September 16th, 2017.
Statement: Section 7703(a)(2) applies to Alice for the year 2018.
Answer: Entailment
Task name: sara_numeric
Statute: §3301. Rate of tax

 There is hereby imposed on every
employer (as defined in section 3306(a)) for each calendar year an excise tax,
with respect to having individuals….[Ommitted from clarity]…This section shall
not apply to any taxable year beginning after December 31, 2017, and before
January 1, 2026.

Description: Bob is Charlie and Dorothy’s son, born on April 15th, 2015. Alice
married Charlie on August 8th, 2018. Alice’s and Charlie’s gross incomes in
2018 were $324311 and $414231 respectively. Alice, Bob and Charlie have the
same principal place of abode in 2018. Alice and Charlie file jointly in 2018,
and take the standard deduction.
Question: How much tax does Alice have to pay in 2018?
Answer: $259487
Table 46: Example of each SARA task. For sara_numeric, we ommit part of the
statute for brevity.

###  F.27 SCALR

In LegalBench, the SCALR task is denoted as scalr.

##### Background

Each case decided by the Supreme Court addresses a specific _question
presented for review_. Both the questions and the Court’s opinions are
published on the Supreme Court’s website.

Many of the Court’s opinions are briefly described by other judges who recount
the holdings of the Court in their own writing. For example, consider the
following passage from State of South Carolina v. Key, 27971 (S.C. 2020;
emphasis added):

> The United States Supreme Court has addressed the constitutionality of
> warrantless blood draws in several DUI cases. See Schmerber, 384 U.S. at
> 770-71 (holding the warrantless blood draw of a DUI suspect was valid
> because the law enforcement officer, dealing with a car accident, could
> “reasonably have believed that he was confronted with an emergency, in which
> the delay necessary to obtain a warrant, under the circumstances, threatened
> ‘the destruction of evidence’")…

We refer to these brief descriptions as ‘holding statements’ or ‘holding
parentheticals,’ since they are often enclosed by parentheses. Identifying the
holding parenthetical that corresponds to a question presented for review
requires a notion of ‘responsiveness’ or relevance between questions and
answers as well as an understanding of the kinds of legal issues that could be
implicated by a specific question presented for review.

##### Task

The SCALR benchmark is a collection of 571 multiple choice questions designed
to assess the legal reasoning and reading comprehension ability of large
language models. Each multiple-choice task gives the question presented for
review in a particular Supreme Court case. The solver must determine which
holding parenthetical describes the Court’s ruling in response to the question
presented. Here is an example from _AT &T Mobility LLC v. Concepcion_, 563
U.S. 333 (2011) with the correct response emphasized:

> Question: Whether the Federal Arbitration Act preempts States from
> conditioning the enforcement of an arbitration agreement on the availability
> of particular procedures–here, class -wide arbitration–when those procedures
> are not necessary to ensure that the parties to the arbitration agreement
> are able to vindicate their claims.
>

>> A: holding that when the parties in court proceedings include claims that
are subject to an arbitration agreement, the FAA requires that agreement to be
enforced even if a state statute or common-law rule would otherwise exclude
that claim from arbitration

>>

>> B: holding that the Arbitration Act “leaves no place for the exercise of
discretion by a district court, but instead mandates that district courts
shall direct the parties to proceed to arbitration on issues as to which an
arbitration agreement has been signed"

>>

>> C: holding that class arbitration “changes the nature of arbitration to
such a degree that it cannot be presumed the parties consented to it by simply
agreeing to submit their disputes to an arbitrator"

>>

>> D: holding that a California law requiring classwide arbitration was
preempted by the FAA because it “stands as an obstacle to the accomplishment
and execution of the full purposes and objectives of Congress," which is to
promote arbitration and enforce arbitration agreements according to their
terms

>>

>> E: holding that under the Federal Arbitration Act, a challenge to an
arbitration provision is for the courts to decide, while a challenge to an
entire contract which includes an arbitration provision is an issue for the
arbitrator

##### Construction

The data used to create this task comes from two sources:

  1. 1.

Questions presented were gathered from the Supreme Court of the United States’
website, which hosts PDFs of questions granted for review in each case dating
back to the 2001 Term.

  2. 2.

Holding statements that comprise the “choices" for each question were compiled
from (a) CourtListener’s collection of parenthetical descriptions and (b)
extraction of parenthetical descriptions from Courtlistener’s and the Caselaw
Access Project’s collections of court decisions using Eyecite.

Because questions presented for review in Supreme Court cases are not easily
available prior to 2001, the benchmark is limited to questions from cases
decided in the 2001 Term and later. To ensure that “holding" statements would
address the particular question presented, we limited the set of cases to
those in which exactly one question was granted for review. We also perform
some manual curation to exclude questions which are not answerable without
specific knowledge of a case. For example, we eliminated a case that presented
this question: “Whether this Court’s decision in Harris v. United States, 536
U.S. 545 (2002), should be overruled."

To create choices for each question presented, we first filter our set of
parenthetical descriptions as follows:

  1. 1.

We limited our parenthetical descriptions to only those that begin with
“holding that…", as these are most likely to describe the core holding of the
case, rather than some peripheral issue.

  2. 2.

We use only parentheticals that describe Supreme Court cases. This avoids the
creation of impossible questions that ask the solver to distinguish between
“holding" statements dealing with exactly the same issue at different stages
of appellate review.

  3. 3.

We then select for each case the longest parenthetical meeting the above
criteria. We use the longest parenthetical because it is most likely to be
descriptive enough to make the question answerable.

We then create a task for each case which has both a question presented and a
“holding" statement meeting the above requirements. (While question-correct
holding pairs are only for cases decided after 2001, we allow the use of
parentheticals describing any Supreme Court case as alternative answer
choices.) We then need to select the four alternative answer choices for each
question in a manner that makes the task challenging. To select choices that
are at least facially plausible, we find the four “holding" statements from
the remaining pool that are most TF-IDF similar to the question presented. The
inclusion of difficult alternative choices requires the solver to draw nuanced
distinctions between legal issues that share overlap in terminology.

##### Significance and value

This task is significant because it tracks the useful and challenging skill of
identifying a passage as relevant or responsive to a given query. LLMs that
are able to perform well at this task have the potential to be more useful for
complex legal question-answering and retrieval. The poor performance of
simpler models on this task demonstrates that it is a challenging one that
requires a level of understanding beyond the word/synonym level.

###  F.28 Securities Complaint Extraction

In LegalBench, the Securities Complaint Extraction tasks are denoted as
ssla_*.

##### Background

Securities Class Actions (SCAs) are lawsuits filed by, and on behalf of,
investors alleging economic injury as a result of material misstatements or
omissions in public disclosures made by corporate directors and officers.
These actions allege violations of the Securities Act of 1933 and Securities
Exchange Act of 1934 and are predominately filed in federal court, though in
2018 the United States Supreme Court determined actions brought under the ‘33
Act were permitted in state court.

“Plaintiff(s)” is the legal term to describe the individual, company, or
organization bringing forth a lawsuit. Under the class action system, one or
more plaintiffs are appointed “Lead Plaintiff” by the court to represent the
interests of a larger group of “similarly situated” parties. In securities
class actions, investors that suffered the greatest financial loss, often
public pensions or unions, are appointed lead plaintiff.

“Defendant(s)” is the legal term to describe the individual, company, or
organization that must defend themself against the alleged violations or
misconduct outlined in the lawsuit. There is always at least one defendant.
The majority of securities class actions name the company, its CEO and its
CFO. Many name additional C-suite level officers, members of the Board of
Directors and additional third-parties such as the company’s independent
auditor and the underwriters of public offerings.

Each designated lead plaintiff, and all named defendants, are explicitly
identified under the “Parties” section of the class action complaint.

##### Tasks

There are three extraction tasks.

  * •

The plaintiff task requires an LLM to extract the named plaintiffs within a
text.

  * •

The individual defendants tasks require an LLM to extract named defendants who
are individuals from within a text.

  * •

The company defendants tasks require an LLM to extract named defendants who
are corporations/companies from within a text.

For certain samples, the complaint excerpt may not exactly contain the answer.
For example, the correct answer may be “Strongbridge Biopharma PLC,” while the
complaint only mentions “Strongbridge.” In order to maintain fidelity to the
workflow used by SSLA, we evaluate an LLM’s ability to generate the official
name of the entity, as represented in the answer. This requires the LLM to
possess some background knowledge regarding official corporation names. We
find that larger models are generally able to account for this.

Sometimes, the provided text will not explicitly name the plaintiff, an
individual defendant, or a corporate defendant. In these cases, the LLM is
expected to return “Not named”.

Figure 5: Example of the typical SCA structure (Case 4:21-cv-05868-HSG)

##### Construction process

Stanford Securities Litigation Analytics (SSLA) identifies, tracks, and
aggregates data on the several hundred private shareholder lawsuits and public
SEC/DOJ enforcements filed each year. SSLA fellows manually extract and
analyze information including plaintiffs, defendants, judges, mediators,
plaintiff and defense firms, key litigation events, real-time case statuses,
settlement timing, settlement dollar amounts, attorneys’ fees and expenses,
and imposed SEC / DOJ penalties. There is no ambiguity regarding the answers
for this task given its nature.

This dataset is an extract from the corpus of texts of securities class action
complaints in the SSLA database. Given the typical structure and headings for
these types of cases, this dataset represents text extracted from the
complaint, between the sections titled “Parties” and “Substantive
Allegations”. For cases where the second heading was not found, texts
fragments were limited to 2,000 characters. Cases with both headings were then
filtered to include only those with texts up to 2,000 characters, which
excluded cases with longer “Parties” sections. Thus, all observations in this
dataset are 2,000 characters or less.

Text was scraped from complaints using python’s PyPDF2 library and left
unformatted and uncleaned. This training set includes several observations
where the text does not include all or some of the named entities due to the
method of text collection and variation in case structure. In several of these
cases, plaintiff names are not present in the selected text because the
plaintiff had been named earlier in the complaint.

Examples from Securities Complaint Tasks
---
Task
Table 46 – continued from previous page
Task
Task name: ssla_company_defendants
Excerpt: 6\. Plaintiff Don L. Gross, as set forth in the accompanying
certification and incorporated by reference herein, purchased the common stock
of KCS during the Cla ss Period and has been damaged thereby. 7\. Defendant
KCS, headquartered in Kansas C ity, Missouri, operates railroads in the
Midwest and Mexico that run north to south, unlike most other U.S. ra ilroads
that run east to west. The Company’s stock traded on the NYSE, an ef ficient
market, during th e Class Period under the ticker symbol “KSU.” As of October
11, 2013, there were more than 110 million shares issued and outstanding. 8\.
Defendant David L. Starling (“Starling”), at all relevant times, served as
KCS’s President and Chief Executive Officer (“CEO”). Case 4:14-cv-00345-BCW
Document 1 Filed 04/15/14 Page 2 of 293 9. Defendant David R. Ebbrecht
(“Ebbrecht”), at all relevant times, served as KCS’s Executive Vice President
and Chief Operating Officer (“COO”). 10\. Defendant Patrick J. Ottensmeyer
(“Ottensmeye r”), at all relevant times, served as KCS’s Executive Vice
President Sales & Marketing. 11\. Defendant Michael W. Upchurch (“Upchurch ”),
at all relevant times, served as KCS’s Executive Vice President and Chief
Financial Officer (“CFO”). 12\. Defendants Starling, Ebbrecht, Ottensmeyer an
d Upchurch are collec tively referred to herein as the “Individual
Defendants.” 13\. During the Class Period, the Individual Defe ndants, as
senior executive officers and/or directors of KCS, were privy to confidential
and proprie tary information concerning KCS, its operations, finances,
financial cond ition and present and future business prospects. The Individual
Defendants also had access to material adverse non-public information
concerning KCS, as discussed in detail below. Because of their positions with
KCS, the Individual Defendants had access to non-public information about its
busine ss, finances, products, markets and present and future business
prospects via interna
Answer: Kansas City Southern
Task name: ssla_individual_defendants
Excerpt: 6\. Plaintiff Don L. Gross, as set forth in the accompanying
certification and incorporated by reference herein, purchased the common stock
of KCS during the Cla ss Period and has been damaged thereby. 7\. Defendant
KCS, headquartered in Kansas C ity, Missouri, operates railroads in the
Midwest and Mexico that run north to south, unlike most other U.S. ra ilroads
that run east to west. The Company’s stock traded on the NYSE, an ef ficient
market, during th e Class Period under the ticker symbol “KSU.” As of October
11, 2013, there were more than 110 million shares issued and outstanding. 8\.
Defendant David L. Starling (“Starling”), at all relevant times, served as
KCS’s President and Chief Executive Officer (“CEO”). Case 4:14-cv-00345-BCW
Document 1 Filed 04/15/14 Page 2 of 293 9. Defendant David R. Ebbrecht
(“Ebbrecht”), at all relevant times, served as KCS’s Executive Vice President
and Chief Operating Officer (“COO”). 10\. Defendant Patrick J. Ottensmeyer
(“Ottensmeye r”), at all relevant times, served as KCS’s Executive Vice
President Sales & Marketing. 11\. Defendant Michael W. Upchurch (“Upchurch ”),
at all relevant times, served as KCS’s Executive Vice President and Chief
Financial Officer (“CFO”). 12\. Defendants Starling, Ebbrecht, Ottensmeyer an
d Upchurch are collec tively referred to herein as the “Individual
Defendants.” 13\. During the Class Period, the Individual Defe ndants, as
senior executive officers and/or directors of KCS, were privy to confidential
and proprie tary information concerning KCS, its operations, finances,
financial cond ition and present and future business prospects. The Individual
Defendants also had access to material adverse non-public information
concerning KCS, as discussed in detail below. Because of their positions with
KCS, the Individual Defendants had access to non-public information about its
busine ss, finances, products, markets and present and future business
prospects via interna
Answer: David L Starling, David R Ebbrecht, Patrick J. Ottensmeyer, Michael W.
Upchurch
Task name: ssla_plaintiff
Excerpt: 11\. Plaintiff, as set forth in th e attached certification, purchas
ed Catalyst securities at artificially inflated prices dur ing the Class
Period and has b een damaged upon the revelation of the alleged corrective
disclosures. 12\. Defendant Catalyst is a Coral Gates, Florida headquartered
company located at 355 Alhambra Circle Suite 1500 Coral Gates, FL 33134. The
common stock is traded on the NASDAQ Stock Market ("NASDAQ") unde r the ticker
symbol "CPRX." 13\. Defendant Patrick J. McEnany ("McEna ny") is the Company’s
co-founder, CEO and President. 14\. Defendant Dr. Hubert E. Huckel M.D.
("Huckel") is the Company’s co-founder and one of its directors. 15\.
Defendant Steven R. Miller Ph. D. ("M iller") is the company’s COO and CSO.
16\. The defendants referenced above in ¶¶ 13- 15 are sometimes referred to
herein as the "Individual Defendants." DEFENDANTS’ WRONGDOING Background Case
1:13-cv-23878-UU Document 1 Entered on FLSD Docket 10/25/2013 Page 4 of 20 5
17. Catalyst is a specialty pharmaceutical company which develops and
commercializes drugs treating orphan (rare) neuromuscular an d neurological
diseases. 18\. Lambert-Eaton Myasthenic Syndrome (“LEM S”) is an extremely
serious disase which is also extremely rare, afflicting about 3.4 persons per
million, and about one to two thousand patients in the United States. 19\. FDA
rules permit so-called “compassionate use” – use of a drug that has not been
approved by the FDA outside of clinical trials. A patient may be given drugs
under a compassionate use program if the patient may benefit from the
treatment, the therapy can be given safely outside the clinical trial setting,
no other alternative therapy is available, and the drug developer agrees to
provide access to the drug. 20\. Jacobus is a tiny privat e pharmaceutical
company in New Jersey, with only dozens of employees, and only 35 as of 2009.
Jacobus has b een manufacturing 3,4 DAP and providing it to patients through a
Answer: Not named

##### Significance and value

Extracting data from legal documents is an extraordinarily resource- and time-
intensive effort prone to human error. As a result, there are no known
databases of non-securities class action litigation, despite the obvious
public policy implications of the class action system. Automation of
identification tasks coupled with human approval would improve efficiency and
reduce collection costs and data errors. This task may be useful to other
legal researchers and industry practitioners extracting structured data from
complex texts. Identification is a very simple task that can be done by those
with an understanding and familiarity with the underlying legal documents and
legal system, but an LLM’s ability to accurately and precisely identify
entities is a useful metric to assess.

###  F.29 Successor Liability

In LegalBench, the Successor Liability task is denoted as successor_liability.

##### Background

When one company sells its assets to another company, the purchaser is
generally not liable for the seller’s debts and liabilities. Successor
liability is a common law exception to this general rule. In order to spot a
successor liability issue, lawyers must understand how courts apply the
doctrine.

The doctrine holds purchasers of all, or substantially all, of a seller’s
assets liable for the debts and liabilities of the seller if:

  1. 1.

the purchaser expressly agrees to be held liable;

  2. 2.

the assets are fraudulently conveyed to the purchaser in order to avoid
liability;

  3. 3.

there is a de facto merger between the purchaser and seller; or

  4. 4.

the purchaser is a mere continuation of the seller.

Express agreement is governed by standard contract law rules. In practice, if
a purchase agreement contains a provision to assume liabilities, litigation
will rarely arise. Courts, however, sometimes interpret an implied agreement
in the absence of a written provision.

Assets are fraudulently conveyed when the seller intends to escape liability
through a sale or knows that liability will be avoided through a sale.

De facto merger is a multifactor test that consists of (1) continuity of
ownership; (2) cessation of ordinary business and dissolution of the acquired
corporation as soon as possible; (3) assumption by the purchaser of the
liabilities ordinarily necessary for the uninterrupted continuation of the
business of the acquired corporation; and (4) continuity of management,
personnel, physical location, assets, and general business operation. Some
jurisdictions require a showing of all four elements. Others do not, and
simply emphasize that the substance of the asset sale is one of a merger,
regardless of its form.

Mere continuation typically requires a showing that after the asset sale, only
one corporation remains and there is an overlap of stock, stockholders, and
directors between the two corporations. There are two variations of the mere
continuation exception. The first variation is the “continuity of enterprise”
exception. In order to find continuity of enterprise, and thus liability for
the purchaser of assets, courts engage in a multifactor analysis. Factors
include: (1) retention of the same employees; (2) retention of the same
supervisory personnel; (3) retention of the same production facilities in the
same physical location; (4) production of the same product; (5) retention of
the same name; (6) continuity of assets; (7) continuity of general business
operations; and (8) whether the successor holds itself out as the continuation
of the previous enterprise. The second variation is the product line
exception. This exception imposes liability on asset purchasers who continue
manufacturing products of a seller’s product line. This exception generally
requires that defendants show that the purchaser of assets is able to assume
the risk spreading role of the original manufacturer, and that imposing
liability is fair because the purchaser enjoys the continued goodwill of the
original manufacturer.

Scholars have noted that fraud, de facto merger, and mere continuation (and
its variants) overlap. They share the common thread of inadequate
consideration, that is, the consideration given in exchange for the assets is
unable to fund the liabilities that underwrite those assets. Because of the
overlap, different courts might apply different doctrines to identical sets of
facts, but arrive at the same policy [50].

Successor liability doctrine is commonly taught in a course on corporate law
or business associations in law school. Sometimes it is reserved for upper
level courses in corporate finance or mergers and acquisitions. Students are
expected to spot successor liability issues and understand how to determine if
a successor will be held liable.

##### Task

The Successor Liability task requires an LLM to spot a successor liability
issue and identify its relevant exception to no liability. If more than one
exception is relevant, the LLM is required to state the additional
exception(s). The task does not include identification of the two variations
to the mere continuation exception (continuity of enterprise and product
line).

Facts |  Issue |  Relevant exception
---|---|---
Large Incarceration Services purchased a substantial amount of Small Prison’s assets last year. The asset purchase agreement expressly disclaimed Small Prison’s potential liability for an employment discrimination claim arising out of its prison service activities. Small conveyed its assets to Large because Small’s owners were concerned that the liability arising from the lawsuit would lead to bankruptcy. Several months following the asset purchase, Small Prison lost the discrimination lawsuit. The plaintiffs now seek relief from Large Incarceration Services. |  successor liability |  fraudulent conveyance, mere continuation
Large Incarceration Services purchased a substantial amount of Small Prison’s assets last year. The asset purchase agreement expressly assumed any liability arising out of its prison service activities. Several months following the asset purchase, Small Prison lost a number of discrimination lawsuits. The plaintiffs now seek relief from Large Incarceration Services. |  successor liability |  express agreement
Big Pharma purchases substantially all of DW I’s assets. The purchase agreement expressly provides for assumption of only those liabilities necessary for continuing operations of DW I. DW I had developed a successful drug that regulated oxygen levels in the blood. After the purchase of DW I’s assets, DW I dissolves. DW I’s shareholders maintain ownership in Big Pharma equivalent to their ownership in DW I. In addition, there is some overlap between the two companies’ management teams. Moreover, Big Pharma continues to employ seventy percent of DW I’s workforce. Past users of DW I’s drug bring a mass tort claim against Big Pharma alleging that DW I’s drug incorrectly measured oxygen levels in the blood leading to harm. |  successor liability |  de facto merger, mere continuation
Big Pharma purchases substantially all of DW I’s assets. The purchase agreement expressly provides for assumption of only those liabilities necessary for continuing operations of DW I. DW I had developed a successful drug that regulated oxygen levels in the blood. After the purchase of DW I’s assets, DW I dissolves. DW I’s shareholders do not own any stock in Big Pharma. However, there is some overlap between the two companies’ management teams. In addition, Big Pharma continues to employ seventy percent of DW I’s workforce. Past users of DW I’s drug bring a mass tort claim against Big Pharma alleging that DW I’s drug incorrectly measured oxygen levels in the blood leading to harm. |  successor liability |  mere continuation
Table 47: Examples for successor liability.

###  F.30 Supply Chain Disclosure Tasks

In LegalBench, the Supply Chain Disclosure Tasks are denoted as
supply_chain_disclosure_*.

##### Background

Corporations are frequently legally required to disclose information that may
be relevant to investors, regulators, or members of the public. One example of
this kind of disclosure requirement is laws that require corporations doing
business in particular jurisdictions to provide detailed information on their
supply chains, which is intended to ensure that the company’s business
practices are not supporting things like human trafficking or human rights
violations. One example of these kind of disclosure requirements is the
California Transparency in Supply Chains Act (CTSCA). The CTSCA applies to
corporations that are a: “[1] retail seller and manufacturer [2] doing
business in this state [of California] and [3] having annual worldwide gross
receipts that exceed one hundred million dollars ($100,000,000).”313131See
California Transparency in Supply Chains Act, Cal. Civ. Code § 1714.43(a)(1)
(West 2012). If a corporation meets these criteria, they are required to post
information on five topics:

  * •

Verification: “[A]t a minimum, disclose to what extent, if any, that the
retail seller or manufacturer . . . [e]ngages in verification of product
supply chains to evaluate and address risks of human trafficking and slavery.
The disclosure shall specify if the verification was not conducted by a third
party.”323232Cal. Civ. Code § 1714.43(c)(1).

  * •

Audits: “[A]t a minimum, disclose to what extent, if any, that the retail
seller or manufacturer . . . [c]onducts audits of suppliers to evaluate
supplier compliance with company standards for trafficking and slavery in
supply chains. The disclosure shall specify if the verification was not an
independent, unannounced audit.”333333Cal. Civ. Code § 1714.43(c)(2).

  * •

Certification: “[A]t a minimum, disclose to what extent, if any, that the
retail seller or manufacturer . . . [r]equires direct suppliers to certify
that materials incorporated into the product comply with the laws regarding
slavery and human trafficking of the country or countries in which they are
doing business.”343434Cal. Civ. Code § 1714.43(c)(3).

  * •

Accountability: “[A]t a minimum, disclose to what extent, if any, that the
retail seller or manufacturer . . . [m]aintains internal accountability
standards and procedures for employees or contractors failing to meet company
standards regarding slavery and trafficking.”353535Cal. Civ. Code §
1714.43(c)(4).

  * •

Training: “[A]t a minimum, disclose to what extent, if any, that the retail
seller or manufacturer . . . [p]rovides company employees and management, who
have direct responsibility for supply chain management, training on human
trafficking and slavery, particularly with respect to mitigating risks within
the supply chains of products.”363636Cal. Civ. Code § 1714.43(c)(5).

In addition to requiring corporations that meet the specified criteria to post
disclosures that provide this information, the California Attorney General’s
office has also posted a guide informing firms of “Best Practices” for what
specific information to provide on each of these five topics.373737Cal. Dep’t
of Justice, The California Transparency in Supply Chains Act: A Resource Guide
(2015), https://oag.ca.gov/sites/all/files/agweb/pdfs/sb657/resource-
guide.pdf.

However, prior research has suggested that companies do not always post
disclosures that cover each of these topics; and, even when they do, the
disclosures are not always consistent with the recommended best practices
[27].

##### Construction process

We constructed this task based on an existing dataset of supply chain
disclosures. In the summer of 2015, we, with the help of research assistants,
we searched the websites of corporations that had previously been identified
by an organization called “KnowTheChain” as being required to post supply
chain disclosures to be compliant with the California Supply Chain
Transparency Act. Through this process, we found disclosures for roughly 400
firms out of roughly 500 firms for which KnowTheChain suggested were required
to post disclosures.

For each of these roughly 400 firms, we saved copies of their supply chain
disclosures. We then had research assistants read the disclosures and code
whether they included each of the five required topics for disclosure and, if
so, whether the disclosures on those five topics were consistent with the best
practices outlined by the California Attorney General’s office.

We convert each of these 10 coded variables into a distinct binary
classificationt task, producing 10 tasks. Table 48 lists each task, along with
the precise question used to code the disclosure.

##### Significance and value

Corporate disclosure requirements are a commonly used regulatory tool, but
evidence suggests that firms do not always fully comply with these disclosure
requirements. The Supply Chain Disclosure task evaluates whether LLMs may be
able to determine whether corporations are complying with those disclosure
requirements. Because these disclosures are often formatted very differently,
written in complex language, and may be designed to obfuscate relevant
information, this task provides a useful measure of whether LLMs can parse the
content covered in legal documents.

Task |  Question
---|---
disclosed_verification |  Does the statement disclose to what extent, if any, that the retail seller or manufacturer engages in verification of product supply chains to evaluate and address risks of human trafficking and slavery? If the company conducts verification], the disclosure shall specify if the verification was not conducted by a third party.
disclosed_audits |  Does the statement disclose to what extent, if any, that the retail seller or manufacturer conducts audits of suppliers to evaluate supplier compliance with company standards for trafficking and slavery in supply chains? The disclosure shall specify if the verification was not an independent, unannounced audit.
disclosed_certification |  Does the statement disclose to what extent, if any, that the retail seller or manufacturer requires direct suppliers to certify that materials incorporated into the product comply with the laws regarding slavery and human trafficking of the country or countries in which they are doing business?
disclosed_accountability |  Does the statement disclose to what extent, if any, that the retail seller or manufacturer maintains internal accountability standards and procedures for employees or contractors failing to meet company standards regarding slavery and trafficking?
disclosed_training |  Does the statement disclose to what extent, if any, that the retail seller or manufacturer provides company employees and management, who have direct responsibility for supply chain management, training on human trafficking and slavery, particularly with respect to mitigating risks within the supply chains of products?
best_practice_verification |  Does the statement disclose whether the retail seller or manufacturer engages in verification and auditing as one practice, expresses that it may conduct an audit, or expressess that it is assessing supplier risks through a review of the US Dept. of Labor’s List?
best_practice_audits |  Does the statement disclose whether the retail seller or manufacturer performs any type of audit, or reserves the right to audit?
best_practice_certification |  Does the statement disclose whether the retail seller or manufacturer requires direct suppliers to certify that they comply with labor and anti-trafficking laws?
best_practice_accountability |  Does the statement disclose whether the retail seller or manufacturer maintains internal compliance procedures on company standards regarding human trafficking and slavery? This includes any type of internal accountability mechanism. Requiring independently of the supply to comply with laws does not qualify or asking for documentary evidence of compliance does not count either.
best_practice_training |  Does the statement disclose whether the retail seller or manufacturer provides training to employees on human trafficking and slavery? Broad policies such as ongoing dialogue on mitigating risks of human trafficking and slavery or increasing managers and purchasers knowledge about health, safety and labor practices qualify as training. Providing training to contractors who failed to comply with human trafficking laws counts as training.
Table 48: Supply Chain Disclosure Tasks

###  F.31 Telemarketing Sales Rule

In LegalBench, the Telemarketing Sales Rule task is denoted as
telemarketing_sales_rule.

##### Background

The Telemarketing Sales Rule (16 C.F.R. Part 310) is a set of regulations
promulgated by the Federal Trade Commission to implement the Telemarketing and
Consumer Fraud and Abuse Prevention Act. Its purpose is to protect consumers
from specified deceptive and abusive telemarketing practices. This task
focuses on 16 C.F.R. § 310.3(a)(1) and 16 C.F.R. § 310.3(a)(2), which outline
a series of specific telemarketing practices prohibited as "deceptive." 16
C.F.R. § 310.3(a)(1) lists information that must be disclosed to a consumer
before a sale is made, and 16 C.F.R. § 310.3(a)(2) lists categories of
information that a telemarketer is prohibited from misrepresenting. 16 C.F.R.
§ 310.2 provides definitions relevant to both of these subsections.

The Telemarketing Sales Rule (TSR) is not commonly taught in law school as its
own topic, but may be used as examples in courses on consumer protection law,
administrative law, telecommunications law, and the like. Because of its
simplicity, it has also been used in beginner-level legal research exercises
tasking students with finding the TSR in the Code of Federal Regulations and
applying it to a set of facts.

Applying the TSR would require an LLM to classify a set of facts as either
falling within or outside of the specific prohibitions outlined in the rule.
For example, the TSR requires that telemarketers disclose certain material
information before a sale is made, such as the total cost of the goods or
services, the quantities of goods or services being purchased, and exchange
and return restrictions. It also forbids telemarketers from making material
misrepresentations as to cost, quantity, quality, endorsement or sponsorship,
and the like. In many real-life situations, it would be ambiguous whether
certain telemarketer behavior would violate the TSR; for example, it could be
contentious whether a given misrepresentation fits the definition of
“material.” However, this task is limited to clear, unambiguous violations or
non-violations, such as if a telemarketer told a consumer that they were
selling four apples for four dollars, when in fact they were selling four
apples for six dollars.

The following subsections 16 C.F.R. § 310.3(a)(1) and 16 C.F.R. § 310.3(a)(2)
were ignored in the task, given their complexity or their reference to other
statutes and regulations:

  * •

16 C.F.R. § 310.3(a)(2)(vi)

  * •

16 C.F.R. § 310.3(a)(2)(viii)

##### Task

The TSR task is meant to test whether an LLM can classify simple sets of facts
as describing a violation of the TSR, or not describing a violation of the
TSR.

##### Construction process

We manually created 50 samples, such that examples of at least one violation
and at least one non-violation of each relevant subsection of 16 C.F.R. §
310.3(a)(1) and 16 C.F.R. § 310.3(a)(2) were present.

##### Significance and value

Determining whether a simple and unambiguous set of facts falls within the
ambit of 16 C.F.R. § 310.3(a)(1) or 16 C.F.R. § 310.3(a)(2) would be an easy
task for law students and lawyers, as well as many non-lawyers. However, an
LLM that was trained to recognize clear violations of consumer protection laws
could help administrative agencies like the Federal Trade Commission inform
normal citizens of their rights.

Input |  Answer
---|---
Acme Toys is a telemarketer subject to the Telemarketing Sales Rule. Acme Toys sold a customer a frisbee. It disclosed the brand of the frisbee, but did not tell the customer the frisbee was manufactured in Portugal. Is this a violation of the Telemarketing Sales Rule? |  No
Acme Toys is a telemarketer subject to the Telemarketing Sales Rule. Acme Toys told a customer that it would sell them a handful of frisbees at a very reasonable price, and that shipping would be $5. Then, the customer agreed to the sale. Is this a violation of the Telemarketing Sales Rule? |  Yes
Table 49: Examples for telemarketing_sales_rule

###  F.32 Textualism Tasks

In LegalBench, the Textualism tasks are denoted as textualism_tool_*.

##### Background

Courts regularly interpret statutes to determine the precise meaning of words
contained in the statute. For instance, suppose a statute specifies that “It
shall be illegal to park a vehicle inside public parks for longer than thirty
minutes.” A court may be asked to determine whether the statute prohibits
persons from parking bicycles inside public parks. This requires defining the
term “vehicle” and determining if a bicycle is a type of vehicle.

To guide the interpretation of ambigous statutory terms, American
jurisprudence has developed numerous principles of statutory construction or
interpretation. These principles—also referred to as tools or canons—are rules
which dictate how terms in statutes should be interpreted. For instance, the
principle of ejusdem generis states that where general words or phrases follow
a number of specific words or phrases, the general words are specifically
construed as limited and apply only to persons or things of the same kind or
class as those expressly mentioned [132].

One approach to statutory interpretation—known as textualism—states that only
the text of the statute should be considered [133]. In contrast, other
approaches to interpreting an ambigous term might call for a court to analyze
the purpose of the statute, the history of the statute, or the intent of the
legislature.

##### Task

The Textualism tasks ask a LLM to determine if an excerpt of judicial text is
applying a specific textual tool when performing statutory interpretation.
There are two tasks: dictionaries (textualism_tool_dictionaries) and .

  * •

The first task is plain-meaning (textualism_tool_plain), and it requires an
LLM to determine if a court is applying the “plain meaning” rule. The plain
meaning rule says that statutory text should be interpreted according to its
plain or ordinary meaning.

  * •

The second task is dictionaries (textualism_tool_dictionaries), and it
requires an LLM to determine if a court is using dictionaries to define the
statutory text.

##### Construction process

For each task we extracted paragraphs from Court of Appeals opinions and
manually annotated whether the paragraphs showed the court as “using” the
respective tool.

  * •

In order to count as using plain meaning, the paragraph must reference the
plain or ordinary meaning of the text. This includes directly saying “plain
meaning” or referencing the general logic of the plain meaning rule. There
must also be evidence that the court used the tool in its decision. This
latter condition is notable because legal scholars often care about whether
the court actually used the tool when defending its decision. Common examples
of using include stating it as a general rule of decision (“[0]ur obligation
is to look to the plain language of the statute to effectuate the intent of
congress”) or applying it to the facts (“The statute’s plain language
indicates the 150% fee cap applies if (1) the plaintiff was “a prisoner” at
the time he brought the action and (2) he was awarded attorney’s fees pursuant
to § 1988.”). “Using” does not, for example, include paragraphs that criticize
the use of the plain meaning rule.

  * •

In order to count as using dictionaries, the paragraph must reference a
dictionary. There must also be evidence that the court used a dictionary as
part of its rationale. This latter condition is notable because legal scholars
often care about whether the court actually used the tool when defending its
decision. Common examples of using include stating it as a general rule of
decision (“[We use a dictionary to help determine the plain meaning of the
statutory text”) or applying it to the facts (“According to the Websters
dictionary, a vehicle is any means in or by which someone travels, or
something is carried or conveyed”). “Using” does not, for example, include
paragraphs that criticize the use of dictionaries.

Input |  Answer
---|---
overcome our prior interpretation of a statute depends, in turn, on whether we regarded the statute as unambiguously compelling our interpretation. |  No
the statutory waiver is express, and its range is defined in unmistakable language. to say that a private person, but not the united states, is liable under title vii for interest as an element of an attorneys fee would rob the unambiguous statutory language of its plain meaning. it would defeat the statutory imposition upon the united states of a liability for costs, and the statutory inclusion of a reasonable attorneys fee as part of the costs, identical to that of a private party in similar circumstances. the scope-setting statutory words the same as a private person mark out the united states liability for attorneys fees as well as costs in the traditional sense. our responsibility as judges is to enforce this provision according to its terms. |  Yes
Table 50: Examples for textualism_tool_plain. Input |  Answer
---|---
we pause to note that even if congress sought, through the csra, to regulate the nonuse of interstate channels, it would still be within its constitutional command to do so. the supreme court has often held, in several contexts, that the defendants nonuse of interstate channels alone does not shield him from federal purview under the commerce clause. in heart of atlanta motel, inc. v. united states, 379 u.s. 241, 250, 85 s.ct. 348, 353, 13 l.ed.2d 258 (1964), the court upheld commerce clause jurisdiction over a local motel that failed to engage in interstate commerce when it refused to rent rooms to black guests. the court held that by failing to rent the rooms, the hotel inhibited black travelers from crossing state lines and thus obstructed interstate commerce that otherwise would have occurred. id. at 253, 85 s.ct. at 356. in standard oil co. v. united states, 221 u.s. 1, 68, 31 s.ct. 502, 518, 55 l.ed. 619 (1911), the court upheld the sherman act, 15 u.s.c. 1, 2, as permissible congressional action under the commerce clause. the sherman act prohibits restraints of trade and obstructions of interstate commerce in order to facilitate commerce that otherwise would occur absent the defendants monopolistic behavior. finally, in united states v. green, 350 u.s. 415, 420, 76 s.ct. 522, 525, 100 l.ed. 494 (1956), the court found constitutional the hobbs act, 18 u.s.c. 1951, which punishes interference with interstate commerce by extortion, robbery or physical violence [by] … outlaw[ing] such interference in any way or degree. to accept baileys nonuse argument would mean, as emphasized by the second circuit, that congress would have no power to prohibit a monopoly so complete as to thwart all other interstate commerce in a line of trade[;] or to punish under the hobbs act someone who successfully prevented interstate trade by extortion and murder. sage, 92 f.3d at 105. |  No
our primary area of concern with the district courts determination is its confident assertion that the language of 326(a) is unambiguous. see lan assocs., 237 b.r. at 56-57. in this day and age when we exchange by a keystroke or series of keystrokes what we used to handle only in cash, we do not think that the term moneys is so clear as the district court indicated. in fact, one of the definitions cited by the district court refers to money as a measure of value, see id. at 55-56 (citing websters third new intl dictionary 1458 (1986)), which surely is a concept that evolves along with and is dependent upon changing cultural, social, and economic practices and institutions. for example, in todays society the term money could easily encompass the concept of credit, which increasing numbers of people use as a method of payment. the term money might also encompass property, especially when property is used as a method of payment or a measure of wealth. see websters ii new college dictionary 707 (defining money as [a] medium that can be exchanged for goods and services and is used as a measure of their values on the market and as [p]roperty and assets considered in terms of monetary value); supra note 5 (describing the nabts argument that an exchange of property involves an exchange of value). but see in re brigantine beach hotel corp., 197 f.2d 296, 299 (3d cir.1952) (referring to precode statute governing receiver compensation and stating that [i]t is clear that the word moneys in the clause … upon all moneys disbursed or turned over … is not the equivalent of property.). these reasonable interpretations of the term moneys render it ambiguous for purposes of our interpretation of 326(a). see taylor v. continental group change in control severance pay plan, 933 f.2d 1227, 1232 (3d cir.1991) (a term is ambiguous if it is subject to reasonable alternative interpretations.); accord united states v. gibbens, 25 f.3d 28, 34 (1st cir.1994) (a statute is ambiguous if it reasonably can be read in more than one way.). |  Yes
Table 51: Examples for textualism_tool_dictionaries

##### Significance and value

Recognizing when a court is applying a particular canon of interpretation is a
classical skill law students are expected to learn. LLM performance on this
task thus offers a heuristic for comparing LLM comprehension of judicial text
to that of a law student’s. More practically, the capacity for LLMs to detect
when certain canons are being applied could make them a valuable tool for
empirical legal scholars.

###  F.33 UCC vs Common Law

In LegalBench, the UCC vs Common Law task is denoted as ucc_v_common_law.

##### Background

In the United States, contracts are typically governed by one of two different
bodies of law depending on the subject matter of the contract. Contracts for
the sale of goods (physical, moveable things) are governed by the Uniform
Commercial Code (UCC), a uniform set of laws created by the Uniform Law
Commission and adopted in all US jurisdictions. Contracts for services and
real estate, on the other hand, are governed by state common law. For example,
a contract for Alice to sell Bob her bike would be governed by the UCC (sale
of a good), but a contract for Bob to repair Alice’s bike would be governed by
the common law (service).

This distinction is significant because the UCC and the common law diverge on
numerous important legal issues such as:

  * •

Offer and acceptance: The common law requires an offeree’s acceptance to
exactly match the terms of the offeror’s offer in order for a contract to be
formed (the “mirror image” rule). The UCC, on the other hand, allows for some
variation in the terms under UCC Section 2-207.

  * •

Definiteness: For a common law contract to be enforceable, it must be
reasonably definite with respect to all material terms. For example, a service
contract would not be enforceable without a price term or an adequate
description of the service to be provided. The UCC only requires that a goods
contract include the good being sold and the quantity. If any other term is
missing from the contract (such as price or delivery), it will be filled in by
UCC default rules.

  * •

Options: To create an option contract (by which the offeror provides the
offeree with a defined period of irrevocability), the common law requires that
the offeree give the offeror separate consideration for the option. The UCC
allows merchants to make “firm offers” (effectively option contracts) without
the offeree providing separate consideration.

  * •

Modification: To modify an existing contract, the common law requires both
parties to provide new consideration (the “preexisting duty rule”) whereas the
UCC only requires that modifications be made in good faith.

##### Task

The UCC vs. Common Law task requires an LLM to determine whether a contract is
governed by the UCC or by the common law.

##### Construction process

The dataset was manually created to test an LLM’s ability to determine whether
a contract is governed by the UCC or by the common law. The dataset is
composed of 100 descriptions of simple contracts such as “Alice and Bob enter
into a contract for Alice to sell her bike to Bob for $50” (UCC) and “Aria
pays Owen $100 to mount a television on the wall of her living room” (common
law). Each description is followed the question, “Is this contract governed by
the UCC or the common law?”

The dataset does not include “mixed purpose” contracts which incorporate both
the sale of a good and a service. For example, a contract in which Alice sells
Bob her bike for $100 and agrees to inflate the tires each week for the first
month would be a mixed purpose contract. To determine whether a mixed purpose
contract is governed by the UCC or the common law, most jurisdictions apply
the “predominant purpose” test under which the predominant purpose of the
contract (good or service) determines which body of law applies.

##### Significance and value

The UCC vs. Common Law task is significant for a number of reasons. First, it
provides a measure of an LLM’s legal reasoning ability relative to a human
lawyer (who would almost certainly score a 100% on the task). Second, it
demonstrates an LLM’s ability to determine the subject matter of a legal text,
which has implications for the use of LLMs for legal tasks far beyond contract
classification. Third, this task could prove useful in the context of contract
lifecycle management (CLM) in which a CLM software product could automatically
sort contracts by subject matter for review purposes. Fourth, while the sample
contracts in the dataset were simple and easily identifiable as either UCC or
common law contracts, real-world mixed purpose contracts can be difficult to
classify and sometimes generate costly litigation. This task could be used as
a starting point for developing a more fine-tuned task that can classify mixed
purpose contracts.

##  Appendix G Full results

###  G.1 Models

HuggingFace links for the studied open-source models in Section 5.2 can be
found below.

LLM |  HuggingFace URL
---|---
Incite-Instruct-3B |  togethercomputer/RedPajama-INCITE-Instruct-3B-v1
Incite-Base-7B |  togethercomputer/RedPajama-INCITE-Base-7B-v0.1
Incite-Instruct-7B |  togethercomputer/RedPajama-INCITE-Instruct-7B-v0.1
BLOOM-3B |  bigscience/bloom-3b
BLOOM-7B |  bigscience/bloom-7b1
OPT-2.7B |  facebook/opt-2.7b
OPT-6.7B |  facebook/opt-6.7b
OPT-13B |  facebook/opt-13b
Falcon-7B-Instruct |  tiiuae/falcon-7b-instruct
MPT-7B-8k-Instruct |  mosaicml/mpt-7b-instruct
Vicuna-7B-16k |  lmsys/vicuna-7b-v1.5-16k
Vicuna-13B-16k |  lmsys/vicuna-13b-v1.5-16k
Flan-T5-XL |  google/flan-t5-xl
Flan-T5-XXL |  google/flan-t5-xxl
LLaMA-2-7B |  meta-llama/LLaMA-2-7b-hf
LLaMA-2-13B |  meta-llama/LLaMA-2-13b-hf
WizardLM-13B |  WizardLM/WizardLM-13B-V1.2
Table 52: HuggingFace links for open-source models.

###  G.2 Prompts

Prompts for all LegalBench experiments are available on the Github repository.
For experiments reported in Section 5.2, Table 53 provides the number of in-
context demonstrations used.

Number of in-context demonstrations |  Tasks
---|---
0 |  Canada Tax Court Outcomes, Consumer Contracts QA, Corporate Lobbying, International Citizenship Questions, Rule QA, Supply Chain Disclosure Tasks, SARA (Numeric)
1 |  MAUD Tasks, SCALR
2 |  Citation Prediction Tasks
3 |  Securities Complaint Extraction Tasks
4 |  Legal Reasoning Causality, Personal Jurisdiction, Successor Liability, SARA (Entailment) Telemarketing Sales Rule, Textualism Tools
5 |  Abercrombie, Hearsay, Insurance Policy Interpretation, Private Right of Action
6 |  CUAD Tasks, Diversity Tasks, J.Crew Blocker, Learned Hands Tasks, Overruling, UCC v. Common Law
7 |  Function of Decision Section, Oral Argument Question Purpose
8 |  Contract NLI Tasks, Contract QA, Definition Tasks, New York State Judicial Ethics, OPP-115 Tasks, Privacy Policy Entailment, Privacy Policy QA
9 |  Unfair Terms of Service
Table 53: Number of in-context demonstrations used for each type.

###  G.3 Results

We provide results for each LLM on each of the tasks. Models are divided into
four groups based on type: commercial models, 13B models, 7B models, and 3B
models. Model names are abbreviated to the family name to ensure well-formed
tables.

Task | GPT-4 | GPT-3.5 | Claude-1
---|---|---|---
abercrombie | 84.2 / 84.2 | 48.4 / 48.4 | 65.2 / 63.1
abercrombie | 84.2 / 84.2 | 48.4 / 48.4 | 65.2 / 63.1
diversity_1 | 96.7 / 96.7 | 86.7 / 86.7 | 86.7 / 86.7
diversity_2 | 100.0 / 100.0 | 53.3 / 50.0 | 73.3 / 73.3
diversity_3 | 96.7 / 96.7 | 83.3 / 66.7 | 53.3 / 53.3
diversity_4 | 93.3 / 93.3 | 70.0 / 66.7 | 43.3 / 43.3
diversity_5 | 76.6 / 76.6 | 66.7 / 66.7 | 36.7 / 36.7
diversity_6 | 80.0 / 80.0 | 6.7 / 6.7 | 53.3 / 53.3
hearsay | 75.5 / 47.9 | 55.3 / 23.4 | 68.1 / 41.5
personal_jurisdiction | 94.0 / 94.0 | 68.0 / 10.0 | 70.0 / 70.0
successor_liability | 19.1 / 19.1 | 15.2 / 15.2 | 38.3 / 38.3
telemarketing_sales_rule | 72.3 / 70.2 | 48.9 / 42.5 | 55.3 / 55.3
ucc_v_common_law | 97.8 / 97.8 | 100.0 / 47.8 | 93.6 / 93.6
Table 54: Performance on rule-application tasks for commercial models. We report correctness/analysis. Task | GPT-4 | GPT-3.5 | Claude-1
---|---|---|---
rule_qa | 72.0 | 46.0 | 77.0
international_citizenship_questions | 59.3 | 52.7 | 60.0
nys_judicial_ethics | 78.3 | 74.3 | 79.1
citation_prediction_classification | 71.3 | 54.6 | 61.1
citation_prediction_open | 15.1 | 3.8 | 11.3
Table 55: Commercial models on rule-recall tasks. Task | Flan | Llama-2 | OPT | Vicuna | WizardLM
---|---|---|---|---|---
rule_qa | 0.0 | 22.0 | 8.0 | 14.0 | 20.0
international_citizenship_questions | 52.4 | 50.0 | 18.0 | 21.5 | 49.8
nys_judicial_ethics | 69.1 | 67.6 | 63.3 | 61.3 | 63.8
citation_prediction_classification | 56.5 | 47.2 | 52.8 | 50.0 | 52.8
citation_prediction_open | 1.9 | 1.9 | 0.0 | 0.0 | 3.8
Table 56: 13B models on rule-recall tasks. Task | Bloom | Falcon | Incite-Base | Incite-Inst. | Llama-2 | MPT | OPT | Vicuna
---|---|---|---|---|---|---|---|---
rule_qa | 6.0 | 8.0 | 18.0 | 28.5 | 22.0 | 20.0 | 6.0 | 0.0
international_citizenship_questions | 0.0 | 10.2 | 49.5 | 47.0 | 27.2 | 2.4 | 2.4 | 3.0
nys_judicial_ethics | 62.5 | 55.8 | 62.7 | 54.9 | 67.5 | 57.2 | 59.7 | 52.3
citation_prediction_classification | 50.0 | 50.9 | 50.9 | 47.2 | 50.0 | 50.0 | 47.2 | 14.8
citation_prediction_open | 1.9 | 0.0 | 0.0 | 0.0 | 1.9 | 0.0 | 0.0 | 0.0
Table 57: 7B models on rule-recall tasks. Task | Bloom | Flan | Incite | Opt
---|---|---|---|---
rule_qa | 0.0 | 0.0 | 14.0 | 2.0
international_citizenship_questions | 0.0 | 50.0 | 18.9 | 3.0
nys_judicial_ethics | 54.0 | 58.5 | 49.9 | 56.7
citation_prediction_classification | 49.1 | 50.0 | 51.9 | 49.1
citation_prediction_open | 0.0 | 0.0 | 0.0 | 0.0
Table 58: 3B models on rule-recall tasks. Task | GPT-4 | GPT-3.5 | Claude-1
---|---|---|---
corporate_lobbying | 81.7 | 59.1 | 75.8
learned_hands_benefits | 87.9 | 62.1 | 66.7
learned_hands_business | 81.6 | 58.6 | 47.7
learned_hands_consumer | 76.2 | 59.3 | 58.1
learned_hands_courts | 52.6 | 54.2 | 46.9
learned_hands_crime | 81.0 | 62.4 | 59.0
learned_hands_divorce | 84.0 | 59.3 | 53.3
learned_hands_domestic_violence | 83.9 | 60.9 | 62.6
learned_hands_education | 91.1 | 57.1 | 55.4
learned_hands_employment | 69.9 | 67.7 | 49.3
learned_hands_estates | 96.6 | 59.0 | 74.7
learned_hands_family | 86.2 | 57.1 | 50.9
learned_hands_health | 87.2 | 65.0 | 49.6
learned_hands_housing | 85.0 | 63.9 | 58.7
learned_hands_immigration | 98.5 | 79.9 | 73.1
learned_hands_torts | 70.6 | 60.0 | 53.2
learned_hands_traffic | 95.3 | 49.8 | 52.3
Table 59: Commercial models on issue-spotting tasks. Task | Flan | Llama-2 | OPT | Vicuna | WizardLM
---|---|---|---|---|---
corporate_lobbying | 55.9 | 55.9 | 50.9 | 50.3 | 50.2
learned_hands_benefits | 68.2 | 50.0 | 50.0 | 4.5 | 0.0
learned_hands_business | 61.5 | 50.6 | 48.9 | 1.1 | 46.6
learned_hands_consumer | 72.8 | 50.0 | 45.0 | 0.0 | 44.6
learned_hands_courts | 58.9 | 49.5 | 50.0 | 50.0 | 0.0
learned_hands_crime | 83.0 | 50.1 | 51.5 | 48.4 | 26.7
learned_hands_divorce | 57.3 | 49.3 | 50.0 | 49.3 | 0.0
learned_hands_domestic_violence | 68.4 | 48.9 | 50.0 | 0.0 | 0.0
learned_hands_education | 89.3 | 50.0 | 46.4 | 50.0 | 26.8
learned_hands_employment | 74.2 | 49.2 | 49.9 | 23.1 | 0.0
learned_hands_estates | 67.4 | 50.0 | 68.5 | 49.4 | 38.8
learned_hands_family | 5.4 | 50.1 | 63.6 | 49.3 | 0.0
learned_hands_health | 66.8 | 49.6 | 63.7 | 50.0 | 38.1
learned_hands_housing | 68.7 | 49.8 | 49.5 | 47.8 | 0.0
learned_hands_immigration | 79.9 | 51.5 | 53.0 | 50.0 | 53.0
learned_hands_torts | 60.6 | 50.0 | 50.2 | 49.8 | 46.1
learned_hands_traffic | 84.2 | 49.8 | 58.6 | 10.8 | 38.7
Table 60: 13B models on issue-spotting tasks. Task | Bloom | Falcon | Incite-Base | Incite-Inst. | Llama-2 | MPT | OPT | Vicuna
---|---|---|---|---|---|---|---|---
corporate_lobbying | 40.3 | 44.8 | 50.1 | 49.7 | 50.0 | 49.6 | 50.9 | 45.5
learned_hands_benefits | 48.5 | 51.5 | 50.0 | 51.5 | 51.5 | 56.1 | 50.0 | 0.0
learned_hands_business | 55.7 | 60.3 | 50.0 | 55.7 | 50.0 | 50.0 | 59.8 | 0.0
learned_hands_consumer | 45.9 | 45.0 | 49.7 | 48.0 | 50.0 | 46.6 | 48.9 | 7.5
learned_hands_courts | 53.6 | 50.5 | 50.5 | 48.4 | 49.5 | 57.8 | 49.5 | 0.0
learned_hands_crime | 50.3 | 51.6 | 50.0 | 54.1 | 51.0 | 53.9 | 51.7 | 0.0
learned_hands_divorce | 48.0 | 50.0 | 50.0 | 62.0 | 46.7 | 58.0 | 49.3 | 0.0
learned_hands_domestic_violence | 48.9 | 48.9 | 51.7 | 59.8 | 51.1 | 49.4 | 50.6 | 0.0
learned_hands_education | 53.6 | 62.5 | 50.0 | 60.7 | 50.0 | 53.6 | 48.2 | 0.0
learned_hands_employment | 50.6 | 49.7 | 49.9 | 54.8 | 49.4 | 51.1 | 49.7 | 0.0
learned_hands_estates | 51.7 | 51.1 | 50.0 | 46.1 | 50.6 | 62.4 | 55.6 | 0.0
learned_hands_family | 55.0 | 57.3 | 49.6 | 60.8 | 50.2 | 56.0 | 53.3 | 0.0
learned_hands_health | 49.1 | 52.7 | 50.0 | 59.3 | 52.2 | 57.1 | 53.5 | 0.0
learned_hands_housing | 49.2 | 49.6 | 49.8 | 47.1 | 50.5 | 49.3 | 50.1 | 0.0
learned_hands_immigration | 53.7 | 47.0 | 50.0 | 64.9 | 50.0 | 67.9 | 62.7 | 0.0
learned_hands_torts | 51.2 | 48.8 | 50.0 | 56.0 | 50.0 | 48.4 | 50.0 | 0.0
learned_hands_traffic | 54.7 | 50.0 | 49.8 | 54.3 | 50.2 | 56.5 | 57.2 | 13.3
Table 61: 7B models on issue-spotting tasks. Task | Bloom | Flan | Incite | Opt
---|---|---|---|---
corporate_lobbying | 26.2 | 51.9 | 47.6 | 51.0
learned_hands_benefits | 43.9 | 43.9 | 47.0 | 53.0
learned_hands_business | 51.1 | 49.4 | 47.7 | 44.8
learned_hands_consumer | 39.7 | 70.8 | 50.2 | 38.1
learned_hands_courts | 56.2 | 46.9 | 47.4 | 60.4
learned_hands_crime | 49.7 | 60.6 | 51.9 | 51.7
learned_hands_divorce | 58.0 | 50.0 | 61.3 | 56.0
learned_hands_domestic_violence | 36.2 | 49.4 | 52.9 | 51.7
learned_hands_education | 50.0 | 67.9 | 51.8 | 46.4
learned_hands_employment | 50.7 | 46.6 | 48.0 | 50.1
learned_hands_estates | 42.7 | 66.3 | 52.8 | 62.4
learned_hands_family | 52.1 | 47.6 | 59.0 | 64.6
learned_hands_health | 46.9 | 58.4 | 53.1 | 52.2
learned_hands_housing | 51.2 | 51.7 | 47.1 | 51.8
learned_hands_immigration | 59.0 | 73.1 | 53.7 | 56.7
learned_hands_torts | 44.4 | 66.2 | 51.2 | 56.7
learned_hands_traffic | 46.8 | 65.3 | 46.4 | 65.3
Table 62: 3B models on issue-spotting tasks. Task | GPT-4 | GPT-3.5 | Claude-1
---|---|---|---
abercrombie | 85.3 | 63.2 | 66.3
diversity_1 | 100.0 | 88.4 | 83.4
diversity_2 | 99.8 | 87.3 | 92.9
diversity_3 | 97.0 | 89.4 | 84.3
diversity_4 | 100.0 | 90.1 | 97.9
diversity_5 | 93.2 | 92.6 | 81.0
diversity_6 | 90.5 | 77.3 | 56.4
hearsay | 83.8 | 69.2 | 76.4
personal_jurisdiction | 91.4 | 63.3 | 81.9
successor_liability | 57.1 | 52.3 | 72.7
telemarketing_sales_rule | 82.4 | 63.1 | 71.9
ucc_v_common_law | 98.8 | 100.0 | 88.8
Table 63: Commercial models on rule-conclusion tasks. Task | Flan | Llama-2 | OPT | Vicuna | WizardLM
---|---|---|---|---|---
abercrombie | 42.1 | 40.0 | 0.0 | 22.1 | 44.2
diversity_1 | 76.0 | 50.0 | 55.0 | 25.5 | 60.5
diversity_2 | 59.4 | 62.1 | 53.9 | 48.4 | 57.1
diversity_3 | 78.6 | 65.8 | 50.0 | 52.6 | 77.8
diversity_4 | 53.2 | 68.3 | 49.6 | 55.4 | 82.5
diversity_5 | 53.5 | 57.7 | 52.4 | 52.7 | 53.2
diversity_6 | 50.0 | 50.0 | 48.1 | 50.0 | 54.3
hearsay | 64.0 | 56.5 | 52.3 | 48.7 | 66.3
personal_jurisdiction | 62.6 | 57.9 | 51.3 | 0.0 | 64.0
successor_liability | 52.7 | 51.2 | 26.4 | 0.0 | 39.1
telemarketing_sales_rule | 69.8 | 74.3 | 50.0 | 63.1 | 72.7
ucc_v_common_law | 98.1 | 77.9 | 52.5 | 0.0 | 79.8
Table 64: 13B models on rule-conclusion tasks. Task | Bloom | Falcon | Incite-Base | Incite-Inst. | Llama-2 | MPT | OPT | Vicuna
---|---|---|---|---|---|---|---|---
abercrombie | 17.9 | 24.2 | 27.4 | 34.7 | 32.6 | 34.7 | 17.9 | 2.1
diversity_1 | 47.6 | 51.7 | 56.8 | 61.5 | 73.4 | 54.9 | 58.0 | 55.3
diversity_2 | 50.0 | 56.3 | 50.5 | 65.7 | 50.0 | 50.0 | 49.8 | 49.1
diversity_3 | 51.9 | 53.6 | 50.0 | 50.8 | 62.8 | 50.0 | 49.9 | 49.3
diversity_4 | 58.3 | 68.7 | 50.0 | 67.9 | 72.9 | 50.0 | 49.7 | 49.7
diversity_5 | 57.9 | 50.0 | 50.4 | 49.4 | 51.3 | 50.0 | 50.0 | 46.8
diversity_6 | 50.0 | 50.0 | 50.0 | 44.7 | 50.0 | 50.0 | 49.7 | 50.0
hearsay | 51.7 | 53.7 | 50.0 | 73.2 | 64.1 | 61.7 | 36.4 | 30.7
personal_jurisdiction | 51.8 | 54.6 | 40.0 | 46.1 | 52.6 | 43.8 | 50.6 | 50.0
successor_liability | 21.7 | 39.1 | 37.2 | 32.6 | 43.4 | 35.7 | 38.8 | 0.0
telemarketing_sales_rule | 57.4 | 55.3 | 51.8 | 58.6 | 61.3 | 55.8 | 54.5 | 43.6
ucc_v_common_law | 50.0 | 50.0 | 50.0 | 50.0 | 56.6 | 50.0 | 50.0 | 0.0
Table 65: 7B models on rule-conclusion tasks. Task | Bloom | Flan | Incite | Opt
---|---|---|---|---
abercrombie | 20.0 | 31.6 | 25.3 | 26.3
diversity_1 | 50.0 | 50.0 | 50.3 | 51.5
diversity_2 | 50.0 | 50.0 | 49.3 | 52.6
diversity_3 | 50.0 | 50.0 | 46.2 | 50.0
diversity_4 | 50.0 | 50.0 | 62.5 | 51.4
diversity_5 | 51.8 | 50.0 | 52.8 | 50.0
diversity_6 | 50.5 | 50.0 | 51.5 | 50.0
hearsay | 51.2 | 57.5 | 57.0 | 49.7
personal_jurisdiction | 50.0 | 51.1 | 46.1 | 50.0
successor_liability | 21.7 | 44.6 | 24.8 | 14.4
telemarketing_sales_rule | 44.5 | 51.4 | 53.5 | 55.9
ucc_v_common_law | 50.0 | 88.8 | 50.0 | 50.0
Table 66: 3B models on rule-conclusion tasks. Task | GPT-4 | GPT-3.5 | Claude-1
---|---|---|---
canada_tax_court_outcomes | 98.9 | 80.0 | 76.9
definition_classification | 96.6 | 80.2 | 87.9
definition_extraction | 81.8 | 85.0 | 82.7
function_of_decision_section | 43.3 | 35.2 | 37.6
legal_reasoning_causality | 84.5 | 72.1 | 66.9
oral_argument_question_purpose | 37.4 | 28.4 | 35.1
overruling | 95.2 | 88.9 | 95.4
scalr | 77.9 | 58.8 | 64.7
textualism_tool_dictionaries | 93.9 | 65.1 | 71.2
textualism_tool_plain | 84.7 | 73.2 | 70.2
Table 67: Commercial models on rhetorical-understanding tasks. Task | Flan | Llama-2 | OPT | Vicuna | WizardLM
---|---|---|---|---|---
canada_tax_court_outcomes | 71.7 | 34.4 | 1.8 | 2.9 | 75.1
definition_classification | 80.8 | 51.2 | 57.4 | 50.0 | 81.5
definition_extraction | 80.5 | 85.0 | 80.1 | 62.4 | 82.0
function_of_decision_section | 34.6 | 18.0 | 20.8 | 14.2 | 13.2
legal_reasoning_causality | 78.6 | 52.8 | 52.3 | 48.4 | 46.2
oral_argument_question_purpose | 24.5 | 20.0 | 15.0 | 16.1 | 29.3
overruling | 94.2 | 92.3 | 78.3 | 52.7 | 89.5
scalr | 66.5 | 56.7 | 20.2 | 5.0 | 45.6
textualism_tool_dictionaries | 93.9 | 66.5 | 54.9 | 5.6 | 61.1
textualism_tool_plain | 81.5 | 72.6 | 51.7 | 43.6 | 74.6
Table 68: 13B models on rhetorical-understanding tasks. Task | Bloom | Falcon | Incite-Base | Incite-Inst. | Llama-2 | MPT | OPT | Vicuna
---|---|---|---|---|---|---|---|---
canada_tax_court_outcomes | 0.0 | 0.0 | 0.0 | 5.2 | 35.4 | 0.0 | 28.5 | 0.0
definition_classification | 54.4 | 80.2 | 50.1 | 65.4 | 50.0 | 57.5 | 58.3 | 42.5
definition_extraction | 77.4 | 77.7 | 80.6 | 73.4 | 84.1 | 80.9 | 73.5 | 4.2
function_of_decision_section | 14.2 | 2.3 | 10.2 | 10.6 | 16.7 | 13.9 | 22.3 | 0.0
legal_reasoning_causality | 47.4 | 59.4 | 57.6 | 56.0 | 53.2 | 55.5 | 50.0 | 38.7
oral_argument_question_purpose | 13.7 | 15.5 | 20.2 | 27.0 | 14.5 | 17.2 | 14.4 | 1.2
overruling | 82.3 | 67.9 | 50.3 | 75.2 | 92.2 | 72.4 | 53.3 | 28.7
scalr | 18.6 | 20.7 | 22.0 | 23.0 | 33.7 | 25.7 | 19.3 | 0.0
textualism_tool_dictionaries | 48.5 | 60.9 | 55.7 | 59.6 | 47.4 | 59.5 | 51.0 | 19.8
textualism_tool_plain | 50.0 | 57.2 | 61.6 | 55.9 | 50.0 | 60.5 | 51.0 | 4.6
Table 69: 7B models on rhetorical-understanding tasks. Task | Bloom | Flan | Incite | Opt
---|---|---|---|---
canada_tax_court_outcomes | 16.0 | 62.3 | 11.7 | 0.3
definition_classification | 51.0 | 78.5 | 51.3 | 57.2
definition_extraction | 59.1 | 77.6 | 59.4 | 70.6
function_of_decision_section | 9.9 | 34.8 | 26.7 | 14.1
legal_reasoning_causality | 50.1 | 67.5 | 49.5 | 55.7
oral_argument_question_purpose | 5.3 | 19.9 | 21.5 | 14.3
overruling | 63.4 | 93.6 | 54.4 | 54.8
scalr | 17.7 | 64.5 | 21.5 | 20.0
textualism_tool_dictionaries | 39.7 | 92.9 | 55.6 | 54.8
textualism_tool_plain | 51.5 | 82.2 | 50.7 | 55.9
Table 70: 3B models on rhetorical-understanding tasks.

Commercial models on interpretation tasks. |  |  |
---|---|---|---
Task | GPT-4 | GPT-3.5 | Claude-1
Table 70 – continued from previous page |  |  |
Task | GPT-4 | GPT-3.5 | Claude-1
consumer_contracts_qa  | 93.6 | 85.9 | 90.3
contract_nli_confidentiality_of_agreement | 96.3 | 96.3 | 92.7
contract_nli_explicit_identification | 82.4 | 81.1 | 65.0
contract_nli_inclusion_of_verbally_conveyed_information | 90.7 | 83.0 | 83.4
contract_nli_limited_use | 86.6 | 85.4 | 80.8
contract_nli_no_licensing | 92.5 | 76.7 | 79.2
contract_nli_notice_on_compelled_disclosure | 97.2 | 97.2 | 96.5
contract_nli_permissible_acquirement_of_similar_information | 96.1 | 96.6 | 93.8
contract_nli_permissible_copy | 80.4 | 77.7 | 72.8
contract_nli_permissible_development_of_similar_information | 98.5 | 99.3 | 96.3
contract_nli_permissible_post-agreement_possession | 94.6 | 89.3 | 92.2
contract_nli_return_of_confidential_information | 95.6 | 92.5 | 89.4
contract_nli_sharing_with_employees | 94.6 | 94.8 | 95.9
contract_nli_sharing_with_third-parties | 93.3 | 75.0 | 86.6
contract_nli_survival_of_obligations | 94.0 | 74.5 | 78.3
contract_qa | 96.2 | 93.6 | 98.7
cuad_affiliate_license-licensee | 90.9 | 90.9 | 85.9
cuad_affiliate_license-licensor | 92.0 | 95.5 | 89.8
cuad_anti-assignment | 91.4 | 89.1 | 92.4
cuad_audit_rights | 97.9 | 89.5 | 92.7
cuad_cap_on_liability | 95.6 | 94.1 | 92.9
cuad_change_of_control | 88.9 | 89.7 | 89.2
cuad_competitive_restriction_exception | 84.1 | 80.0 | 71.4
cuad_covenant_not_to_sue | 95.8 | 88.0 | 89.3
cuad_effective_date | 92.8 | 75.0 | 74.2
cuad_exclusivity | 92.9 | 89.0 | 87.3
cuad_expiration_date | 82.0 | 87.0 | 78.8
cuad_governing_law | 99.3 | 98.3 | 98.7
cuad_insurance | 99.2 | 95.3 | 94.9
cuad_ip_ownership_assignment | 91.7 | 91.0 | 89.2
cuad_irrevocable_or_perpetual_license | 97.5 | 95.4 | 88.6
cuad_joint_ip_ownership | 94.3 | 91.1 | 85.4
cuad_license_grant | 94.0 | 90.3 | 91.0
cuad_liquidated_damages | 96.4 | 86.4 | 90.9
cuad_minimum_commitment | 89.1 | 86.1 | 88.6
cuad_most_favored_nation | 96.9 | 95.3 | 96.9
cuad_no-solicit_of_customers | 100.0 | 98.8 | 92.9
cuad_no-solicit_of_employees | 100.0 | 97.9 | 96.5
cuad_non-compete | 93.0 | 91.0 | 90.5
cuad_non-disparagement | 97.0 | 95.0 | 87.0
cuad_non-transferable_license | 90.2 | 82.1 | 87.6
cuad_notice_period_to_terminate_renewal | 95.9 | 97.7 | 96.4
cuad_post-termination_services | 94.6 | 89.0 | 77.8
cuad_price_restrictions | 95.7 | 87.0 | 89.1
cuad_renewal_term | 96.1 | 95.9 | 95.6
cuad_revenue-profit_sharing | 95.3 | 91.2 | 88.9
cuad_rofr-rofo-rofn | 88.6 | 81.9 | 86.2
cuad_source_code_escrow | 96.6 | 91.5 | 94.1
cuad_termination_for_convenience | 96.7 | 94.2 | 95.8
cuad_third_party_beneficiary | 89.7 | 83.8 | 82.4
cuad_uncapped_liability | 85.4 | 70.4 | 74.1
cuad_unlimited-all-you-can-eat-license | 93.8 | 93.8 | 87.5
cuad_volume_restriction | 80.7 | 68.6 | 80.1
cuad_warranty_duration | 77.8 | 81.2 | 78.1
insurance_policy_interpretation | 69.6 | 55.0 | 64.0
jcrew_blocker | 100.0 | 88.9 | 55.6
maud_ability_to_consummate_concept_is_subject_to_mae_carveouts | 50.0 | 50.0 | 31.5
maud_financial_point_of_view_is_the_sole_consideration | 50.0 | 38.8 | 50.0
maud_accuracy_of_fundamental_target_rws_bringdown_standard | 29.3 | 33.3 | 10.4
maud_accuracy_of_target_general_rw_bringdown_timing_answer | 63.6 | 51.0 | 46.9
maud_accuracy_of_target_capitalization_rw_(outstanding_shares)_bringdown_standard_answer | 20.7 | 16.2 | 13.4
maud_additional_matching_rights_period_for_modifications_(cor) | 57.4 | 43.3 | 24.5
maud_application_of_buyer_consent_requirement_(negative_interim_covenant) | 63.7 | 68.8 | 35.3
maud_buyer_consent_requirement_(ordinary_course) | 50.0 | 60.8 | 28.9
maud_change_in_law__subject_to_disproportionate_impact_modifier | 53.0 | 48.3 | 65.2
maud_changes_in_gaap_or_other_accounting_principles__subject_to_disproportionate_impact_modifier | 51.7 | 47.4 | 53.8
maud_cor_permitted_in_response_to_intervening_event | 50.0 | 52.5 | 50.1
maud_cor_permitted_with_board_fiduciary_determination_only | 21.4 | 50.0 | 50.6
maud_cor_standard_(intervening_event) | 0.5 | 36.5 | 0.0
maud_cor_standard_(superior_offer) | 40.5 | 45.5 | 0.0
maud_definition_contains_knowledge_requirement_-_answer | 25.0 | 33.7 | 20.7
maud_definition_includes_asset_deals | 33.3 | 30.5 | 0.5
maud_definition_includes_stock_deals | 33.3 | 37.5 | 27.4
maud_fiduciary_exception__board_determination_standard | 40.1 | 27.5 | 0.0
maud_fiduciary_exception_board_determination_trigger_(no_shop) | 50.0 | 48.8 | 50.0
maud_fls_(mae)_standard | 25.0 | 44.6 | 22.5
maud_general_economic_and_financial_conditions_subject_to_disproportionate_impact_modifier | 54.2 | 56.0 | 53.6
maud_includes_consistent_with_past_practice | 54.2 | 55.3 | 84.2
maud_initial_matching_rights_period_(cor) | 15.4 | 31.9 | 20.7
maud_initial_matching_rights_period_(ftr) | 49.4 | 32.8 | 14.0
maud_intervening_event_-_required_to_occur_after_signing_-_answer | 51.9 | 51.4 | 12.5
maud_knowledge_definition | 51.1 | 49.1 | 38.5
maud_liability_standard_for_no-shop_breach_by_target_non-do_representatives | 44.2 | 51.9 | 49.4
maud_ordinary_course_efforts_standard | 91.1 | 70.3 | 53.6
maud_pandemic_or_other_public_health_event__subject_to_disproportionate_impact_modifier | 48.7 | 50.0 | 51.3
maud_pandemic_or_other_public_health_event_specific_reference_to_pandemic-related_governmental_responses_or_measures | 79.5 | 70.9 | 60.8
maud_relational_language_(mae)_applies_to | 57.9 | 47.2 | 26.0
maud_specific_performance | 51.5 | 90.6 | 73.7
maud_tail_period_length | 68.1 | 39.5 | 60.4
maud_type_of_consideration | 99.5 | 82.7 | 75.1
opp115_data_retention | 67.0 | 70.5 | 55.7
opp115_data_security | 87.5 | 84.2 | 55.6
opp115_do_not_track | 99.1 | 93.6 | 90.0
opp115_first_party_collection_use | 76.7 | 80.6 | 63.0
opp115_international_and_specific_audiences | 92.3 | 82.6 | 79.4
opp115_policy_change | 91.9 | 89.3 | 83.8
opp115_third_party_sharing_collection | 80.1 | 77.0 | 71.0
opp115_user_access,_edit_and_deletion | 90.2 | 87.7 | 79.3
opp115_user_choice_control | 82.9 | 79.3 | 71.3
privacy_policy_entailment | 85.5 | 78.8 | 89.6
privacy_policy_qa | 71.3 | 65.5 | 63.0
proa | 99.0 | 90.6 | 88.5
ssla_company_defendants | 65.0 | 65.3 | 16.5
ssla_individual_defendants | 29.6 | 25.8 | 11.1
ssla_plaintiff | 92.0 | 86.5 | 86.7
sara_entailment | 86.8 | 68.4 | 67.6
sara_numeric | 8.3 | 4.2 | 6.2
supply_chain_disclosure_best_practice_accountability | 71.5 | 69.5 | 74.6
supply_chain_disclosure_best_practice_audits | 74.4 | 76.6 | 75.5
supply_chain_disclosure_best_practice_certification | 76.6 | 77.7 | 77.4
supply_chain_disclosure_best_practice_training | 83.3 | 87.1 | 85.3
supply_chain_disclosure_best_practice_verification | 68.3 | 59.4 | 64.3
supply_chain_disclosure_disclosed_accountability | 77.0 | 80.4 | 75.5
supply_chain_disclosure_disclosed_audits | 81.6 | 83.7 | 80.0
supply_chain_disclosure_disclosed_certification | 71.2 | 67.3 | 75.8
supply_chain_disclosure_disclosed_training | 89.1 | 83.0 | 75.6
supply_chain_disclosure_disclosed_verification | 56.6 | 62.0 | 67.6
unfair_tos | 9.1 | 13.7 | 5.5

13B models on interpretation tasks. |  |  |  |  |
---|---|---|---|---|---
Task | Flan | LLaMA-2 | OPT | Vicuna | WizardLM
Table 70 – continued from previous page |  |  |  |  |
Task | Flan | LLaMA-2 | OPT | Vicuna | WizardLM
consumer_contracts_qa  | 92.6 | 68.1 | 24.8 | 37.9 | 67.8
contract_nli_confidentiality_of_agreement | 85.4 | 50.0 | 54.9 | 48.8 | 76.8
contract_nli_explicit_identification | 81.4 | 49.4 | 51.6 | 50.0 | 67.0
contract_nli_inclusion_of_verbally_conveyed_information | 56.2 | 50.0 | 53.0 | 50.0 | 58.2
contract_nli_limited_use | 71.1 | 44.5 | 60.6 | 49.5 | 57.1
contract_nli_no_licensing | 54.2 | 53.3 | 47.0 | 51.2 | 58.4
contract_nli_notice_on_compelled_disclosure | 90.8 | 52.1 | 61.3 | 55.6 | 73.2
contract_nli_permissible_acquirement_of_similar_information | 89.3 | 50.0 | 45.5 | 52.2 | 61.2
contract_nli_permissible_copy | 84.3 | 47.8 | 47.8 | 50.0 | 56.2
contract_nli_permissible_development_of_similar_information | 98.5 | 50.0 | 55.1 | 54.4 | 86.0
contract_nli_permissible_post-agreement_possession | 93.4 | 48.8 | 44.0 | 50.0 | 52.5
contract_nli_return_of_confidential_information | 89.4 | 50.0 | 44.3 | 51.3 | 66.2
contract_nli_sharing_with_employees | 92.2 | 48.2 | 70.5 | 50.0 | 65.7
contract_nli_sharing_with_third-parties | 86.7 | 49.5 | 50.9 | 50.5 | 64.4
contract_nli_survival_of_obligations | 74.6 | 50.0 | 44.8 | 50.6 | 59.6
contract_qa | 96.3 | 82.7 | 56.8 | 73.1 | 35.9
cuad_affiliate_license-licensee | 83.8 | 58.1 | 49.0 | 50.0 | 65.7
cuad_affiliate_license-licensor | 90.9 | 72.7 | 53.4 | 50.0 | 54.5
cuad_anti-assignment | 85.0 | 52.2 | 48.1 | 50.2 | 76.8
cuad_audit_rights | 87.1 | 51.6 | 71.7 | 50.7 | 58.6
cuad_cap_on_liability | 81.5 | 74.0 | 0.6 | 50.0 | 57.5
cuad_change_of_control | 75.5 | 56.2 | 52.9 | 49.0 | 74.8
cuad_competitive_restriction_exception | 81.8 | 51.8 | 38.2 | 50.0 | 50.0
cuad_covenant_not_to_sue | 86.0 | 66.6 | 49.4 | 50.0 | 56.5
cuad_effective_date | 92.8 | 50.0 | 50.4 | 48.7 | 58.1
cuad_exclusivity | 84.0 | 86.1 | 61.7 | 49.6 | 58.7
cuad_expiration_date | 60.5 | 51.6 | 53.1 | 51.0 | 73.2
cuad_governing_law | 99.5 | 90.6 | 71.7 | 59.5 | 54.3
cuad_insurance | 90.2 | 55.5 | 66.3 | 55.0 | 55.7
cuad_ip_ownership_assignment | 89.8 | 74.0 | 49.8 | 49.7 | 59.9
cuad_irrevocable_or_perpetual_license | 95.7 | 85.4 | 67.9 | 50.0 | 73.9
cuad_joint_ip_ownership | 76.0 | 53.6 | 50.5 | 50.0 | 60.4
cuad_license_grant | 92.8 | 62.2 | 50.8 | 49.9 | 68.7
cuad_liquidated_damages | 73.6 | 50.9 | 48.6 | 49.5 | 71.8
cuad_minimum_commitment | 62.3 | 54.5 | 51.6 | 49.9 | 47.0
cuad_most_favored_nation | 75.0 | 56.2 | 43.8 | 50.0 | 57.8
cuad_no-solicit_of_customers | 97.6 | 57.1 | 56.0 | 50.0 | 85.7
cuad_no-solicit_of_employees | 95.8 | 97.9 | 69.0 | 50.0 | 75.4
cuad_non-compete | 91.6 | 52.7 | 46.8 | 49.3 | 55.2
cuad_non-disparagement | 83.0 | 73.0 | 67.0 | 49.0 | 74.0
cuad_non-transferable_license | 69.6 | 55.0 | 54.2 | 49.4 | 75.6
cuad_notice_period_to_terminate_renewal | 92.8 | 50.0 | 54.5 | 50.0 | 83.3
cuad_post-termination_services | 88.7 | 50.7 | 51.4 | 49.6 | 62.5
cuad_price_restrictions | 76.1 | 71.7 | 50.0 | 47.8 | 50.0
cuad_renewal_term | 89.1 | 50.5 | 50.3 | 57.5 | 83.9
cuad_revenue-profit_sharing | 72.1 | 58.4 | 54.5 | 49.7 | 52.7
cuad_rofr-rofo-rofn | 65.1 | 50.4 | 54.1 | 50.0 | 59.6
cuad_source_code_escrow | 66.1 | 58.5 | 67.8 | 50.0 | 52.5
cuad_termination_for_convenience | 91.4 | 50.9 | 49.8 | 55.8 | 84.2
cuad_third_party_beneficiary | 85.3 | 54.4 | 63.2 | 47.1 | 69.1
cuad_uncapped_liability | 50.3 | 55.4 | 46.6 | 51.0 | 74.5
cuad_unlimited-all-you-can-eat-license | 83.3 | 52.1 | 64.6 | 47.9 | 54.2
cuad_volume_restriction | 55.6 | 50.0 | 55.3 | 50.0 | 54.0
cuad_warranty_duration | 74.7 | 57.5 | 55.9 | 50.6 | 63.1
insurance_policy_interpretation | 44.8 | 46.6 | 38.0 | 13.2 | 51.9
jcrew_blocker | 86.7 | 66.7 | 51.1 | 50.0 | 8.9
maud_ability_to_consummate_concept_is_subject_to_mae_carveouts | 50.0 | 47.3 | 51.8 | 50.0 | 8.2
maud_financial_point_of_view_is_the_sole_consideration | 53.6 | 51.0 | 50.0 | 53.1 | 49.5
maud_accuracy_of_fundamental_target_rws_bringdown_standard | 12.5 | 33.3 | 33.3 | 31.7 | 33.3
maud_accuracy_of_target_general_rw_bringdown_timing_answer | 46.6 | 50.0 | 49.1 | 49.8 | 50.0
maud_accuracy_of_target_capitalization_rw_(outstanding_shares)_bringdown_standard_answer | 6.1 | 25.8 | 26.9 | 20.9 | 24.8
maud_additional_matching_rights_period_for_modifications_(cor) | 0.0 | 25.0 | 19.8 | 18.4 | 12.9
maud_application_of_buyer_consent_requirement_(negative_interim_covenant) | 63.4 | 47.8 | 45.0 | 3.1 | 49.4
maud_buyer_consent_requirement_(ordinary_course) | 34.4 | 56.5 | 44.4 | 45.1 | 50.0
maud_change_in_law__subject_to_disproportionate_impact_modifier | 50.0 | 55.9 | 46.6 | 21.3 | 0.0
maud_changes_in_gaap_or_other_accounting_principles__subject_to_disproportionate_impact_modifier | 50.0 | 57.5 | 47.1 | 20.6 | 0.0
maud_cor_permitted_in_response_to_intervening_event | 50.0 | 57.6 | 58.8 | 26.8 | 47.0
maud_cor_permitted_with_board_fiduciary_determination_only | 50.0 | 50.0 | 51.2 | 46.7 | 42.0
maud_cor_standard_(intervening_event) | 0.0 | 24.0 | 16.7 | 10.0 | 23.3
maud_cor_standard_(superior_offer) | 11.9 | 16.8 | 3.0 | 0.0 | 26.9
maud_definition_contains_knowledge_requirement_-_answer | 0.0 | 28.9 | 25.0 | 24.5 | 24.1
maud_definition_includes_asset_deals | 2.8 | 35.4 | 32.7 | 0.9 | 34.2
maud_definition_includes_stock_deals | 4.6 | 31.8 | 24.7 | 30.1 | 17.3
maud_fiduciary_exception__board_determination_standard | 1.2 | 6.5 | 12.5 | 0.5 | 14.1
maud_fiduciary_exception_board_determination_trigger_(no_shop) | 50.0 | 59.2 | 44.7 | 8.8 | 48.8
maud_fls_(mae)_standard | 17.1 | 5.0 | 24.5 | 25.0 | 4.6
maud_general_economic_and_financial_conditions_subject_to_disproportionate_impact_modifier | 50.0 | 53.6 | 52.4 | 6.0 | 0.0
maud_includes_consistent_with_past_practice | 62.3 | 50.0 | 52.6 | 53.2 | 61.0
maud_initial_matching_rights_period_(cor) | 0.0 | 8.3 | 20.8 | 0.9 | 14.2
maud_initial_matching_rights_period_(ftr) | 0.0 | 13.3 | 23.2 | 19.1 | 18.8
maud_intervening_event_-_required_to_occur_after_signing_-_answer | 39.2 | 46.2 | 48.9 | 47.1 | 47.1
maud_knowledge_definition | 50.6 | 46.0 | 48.4 | 0.0 | 51.2
maud_liability_standard_for_no-shop_breach_by_target_non-do_representatives | 48.7 | 48.7 | 49.4 | 0.6 | 50.0
maud_ordinary_course_efforts_standard | 81.1 | 57.8 | 33.7 | 0.0 | 67.3
maud_pandemic_or_other_public_health_event__subject_to_disproportionate_impact_modifier | 48.1 | 52.2 | 46.2 | 7.6 | 31.0
maud_pandemic_or_other_public_health_event_specific_reference_to_pandemic-related_governmental_responses_or_measures | 50.0 | 50.0 | 51.9 | 50.7 | 37.4
maud_relational_language_(mae)_applies_to | 51.0 | 44.2 | 1.4 | 51.7 | 5.1
maud_specific_performance | 94.9 | 52.1 | 50.0 | 0.0 | 56.7
maud_tail_period_length | 25.8 | 51.5 | 52.5 | 13.5 | 34.6
maud_type_of_consideration | 73.6 | 39.1 | 30.2 | 27.8 | 27.2
opp115_data_retention | 55.7 | 51.1 | 45.5 | 50.0 | 63.6
opp115_data_security | 75.1 | 49.8 | 51.2 | 55.5 | 59.4
opp115_do_not_track | 79.1 | 50.0 | 42.7 | 51.8 | 88.2
opp115_first_party_collection_use | 75.0 | 67.0 | 68.5 | 52.3 | 55.2
opp115_international_and_specific_audiences | 80.1 | 59.5 | 18.2 | 50.4 | 66.2
opp115_policy_change | 70.5 | 64.9 | 60.9 | 52.8 | 56.5
opp115_third_party_sharing_collection | 71.4 | 54.9 | 58.8 | 52.4 | 60.9
opp115_user_access,_edit_and_deletion | 75.4 | 54.3 | 60.9 | 49.1 | 59.4
opp115_user_choice_control | 80.9 | 53.7 | 50.0 | 47.5 | 58.0
privacy_policy_entailment | 58.9 | 56.2 | 50.1 | 0.6 | 65.9
privacy_policy_qa | 52.5 | 50.5 | 50.9 | 0.0 | 55.6
proa | 94.8 | 76.0 | 52.1 | 50.0 | 80.1
ssla_company_defendants | 34.4 | 63.4 | 56.6 | 3.1 | 2.8
ssla_individual_defendants | 21.9 | 21.6 | 16.2 | 0.0 | 0.0
ssla_plaintiff | 88.8 | 26.4 | 31.6 | 0.0 | 0.0
sara_entailment | 35.3 | 58.1 | 48.9 | 15.4 | 50.0
sara_numeric | 1.0 | 0.0 | 0.0 | 0.0 | 0.0
supply_chain_disclosure_best_practice_accountability | 73.6 | 49.4 | 48.8 | 67.2 | 52.1
supply_chain_disclosure_best_practice_audits | 69.2 | 49.2 | 66.6 | 46.5 | 65.7
supply_chain_disclosure_best_practice_certification | 69.9 | 51.6 | 56.1 | 67.2 | 64.8
supply_chain_disclosure_best_practice_training | 81.0 | 49.7 | 49.4 | 71.5 | 64.9
supply_chain_disclosure_best_practice_verification | 70.5 | 49.1 | 50.8 | 51.6 | 52.5
supply_chain_disclosure_disclosed_accountability | 80.7 | 49.4 | 45.8 | 59.1 | 58.8
supply_chain_disclosure_disclosed_audits | 83.3 | 49.3 | 48.9 | 64.1 | 64.7
supply_chain_disclosure_disclosed_certification | 68.7 | 49.4 | 53.8 | 61.6 | 59.3
supply_chain_disclosure_disclosed_training | 87.0 | 49.2 | 48.0 | 58.1 | 62.8
supply_chain_disclosure_disclosed_verification | 64.3 | 49.3 | 48.6 | 58.0 | 50.4
unfair_tos | 10.0 | 12.8 | 10.0 | 8.3 | 13.8

7B models on interpretation tasks. |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Task | BLOOM | Falcon | Incite-Base | Incite-Inst. | LLaMA-2 | MPT | OPT | Vicuna
Table 70 – continued from previous page |  |  |  |  |  |  |  |
Task | BLOOM | Falcon | Incite-Base | Incite-Inst. | LLaMA-2 | MPT | OPT | Vicuna
consumer_contracts_qa  | 0.6 | 57.9 | 42.4 | 49.0 | 63.5 | 40.5 | 14.7 | 34.0
contract_nli_confidentiality_of_agreement | 53.7 | 64.6 | 65.9 | 59.8 | 50.0 | 52.4 | 57.3 | 45.1
contract_nli_explicit_identification | 49.4 | 66.1 | 61.6 | 68.9 | 50.0 | 50.0 | 59.9 | 34.8
contract_nli_inclusion_of_verbally_conveyed_information | 50.0 | 67.5 | 66.7 | 60.9 | 50.0 | 49.3 | 55.5 | 33.1
contract_nli_limited_use | 56.2 | 46.0 | 59.0 | 66.3 | 51.1 | 61.8 | 61.7 | 24.8
contract_nli_no_licensing | 49.4 | 44.6 | 51.9 | 56.3 | 48.2 | 45.9 | 49.5 | 15.9
contract_nli_notice_on_compelled_disclosure | 51.4 | 51.4 | 64.1 | 62.7 | 50.7 | 65.5 | 68.3 | 27.5
contract_nli_permissible_acquirement_of_similar_information | 49.4 | 31.5 | 26.4 | 47.8 | 50.0 | 53.4 | 50.0 | 39.9
contract_nli_permissible_copy | 49.9 | 36.8 | 46.4 | 57.6 | 58.9 | 55.0 | 49.6 | 33.2
contract_nli_permissible_development_of_similar_information | 49.3 | 43.4 | 59.6 | 44.1 | 50.0 | 54.4 | 53.7 | 41.9
contract_nli_permissible_post-agreement_possession | 48.2 | 43.1 | 53.2 | 55.6 | 50.0 | 55.3 | 44.5 | 2.4
contract_nli_return_of_confidential_information | 50.0 | 57.7 | 50.2 | 70.0 | 50.0 | 56.5 | 75.6 | 5.9
contract_nli_sharing_with_employees | 55.6 | 61.6 | 48.2 | 70.3 | 50.0 | 48.1 | 58.9 | 6.1
contract_nli_sharing_with_third-parties | 49.5 | 48.1 | 39.3 | 53.7 | 50.0 | 48.4 | 48.7 | 11.9
contract_nli_survival_of_obligations | 49.5 | 55.1 | 41.2 | 43.2 | 50.0 | 41.7 | 49.3 | 1.8
contract_qa | 14.8 | 7.7 | 11.4 | 87.7 | 31.9 | 9.0 | 39.0 | 50.3
cuad_affiliate_license-licensee | 53.0 | 50.5 | 68.7 | 71.7 | 50.0 | 69.2 | 66.2 | 34.3
cuad_affiliate_license-licensor | 45.5 | 70.5 | 64.8 | 85.2 | 50.0 | 76.1 | 52.3 | 38.6
cuad_anti-assignment | 47.7 | 33.2 | 60.4 | 76.2 | 58.8 | 56.5 | 55.1 | 37.8
cuad_audit_rights | 67.4 | 59.7 | 80.0 | 71.5 | 50.7 | 52.5 | 72.8 | 28.2
cuad_cap_on_liability | 41.8 | 42.1 | 40.4 | 57.4 | 50.2 | 70.3 | 37.0 | 16.8
cuad_change_of_control | 55.0 | 49.8 | 57.0 | 67.8 | 50.0 | 56.0 | 55.5 | 39.2
cuad_competitive_restriction_exception | 37.3 | 30.9 | 50.0 | 47.7 | 48.6 | 46.8 | 36.4 | 16.4
cuad_covenant_not_to_sue | 47.7 | 48.1 | 67.9 | 78.2 | 57.5 | 71.8 | 50.0 | 0.0
cuad_effective_date | 56.8 | 60.2 | 53.0 | 46.6 | 50.0 | 44.5 | 65.7 | 6.4
cuad_exclusivity | 62.9 | 57.1 | 66.4 | 65.4 | 63.4 | 56.7 | 60.0 | 14.4
cuad_expiration_date | 75.9 | 62.7 | 55.5 | 67.6 | 50.1 | 52.9 | 85.0 | 10.5
cuad_governing_law | 79.3 | 77.3 | 28.5 | 68.2 | 55.8 | 66.3 | 73.1 | 17.6
cuad_insurance | 83.2 | 66.8 | 65.4 | 72.0 | 50.1 | 72.7 | 77.2 | 39.1
cuad_ip_ownership_assignment | 53.0 | 65.8 | 61.1 | 73.8 | 51.6 | 60.4 | 70.8 | 14.4
cuad_irrevocable_or_perpetual_license | 57.5 | 58.6 | 72.1 | 83.2 | 52.9 | 59.6 | 80.0 | 46.1
cuad_joint_ip_ownership | 55.2 | 58.3 | 60.9 | 74.5 | 50.0 | 55.2 | 72.4 | 30.2
cuad_license_grant | 62.7 | 60.3 | 65.5 | 77.0 | 63.0 | 39.0 | 67.9 | 20.8
cuad_liquidated_damages | 54.1 | 65.0 | 65.0 | 70.5 | 50.0 | 51.4 | 57.7 | 36.4
cuad_minimum_commitment | 51.4 | 58.4 | 55.3 | 53.8 | 49.9 | 55.3 | 59.2 | 31.3
cuad_most_favored_nation | 51.6 | 48.4 | 60.9 | 59.4 | 51.6 | 57.8 | 48.4 | 23.4
cuad_no-solicit_of_customers | 47.6 | 38.1 | 61.9 | 77.4 | 50.0 | 51.2 | 27.4 | 22.6
cuad_no-solicit_of_employees | 39.4 | 35.9 | 54.9 | 80.3 | 69.0 | 70.4 | 42.3 | 19.7
cuad_non-compete | 32.8 | 42.5 | 55.2 | 67.4 | 63.3 | 52.3 | 29.0 | 42.3
cuad_non-disparagement | 41.0 | 50.0 | 64.0 | 70.0 | 64.0 | 51.0 | 47.0 | 34.0
cuad_non-transferable_license | 67.2 | 65.9 | 56.5 | 78.0 | 50.0 | 48.5 | 68.5 | 27.9
cuad_notice_period_to_terminate_renewal | 50.0 | 58.6 | 50.5 | 76.6 | 50.0 | 50.9 | 81.5 | 35.6
cuad_post-termination_services | 48.1 | 45.7 | 60.0 | 57.2 | 50.0 | 64.7 | 57.7 | 20.0
cuad_price_restrictions | 58.7 | 43.5 | 45.7 | 58.7 | 50.0 | 54.3 | 52.2 | 41.3
cuad_renewal_term | 50.3 | 57.3 | 45.1 | 75.6 | 50.0 | 53.1 | 82.1 | 35.2
cuad_revenue-profit_sharing | 54.3 | 57.9 | 50.6 | 65.2 | 52.3 | 50.0 | 62.4 | 11.1
cuad_rofr-rofo-rofn | 54.6 | 43.3 | 55.9 | 57.0 | 50.0 | 59.9 | 53.0 | 20.6
cuad_source_code_escrow | 75.4 | 59.3 | 62.7 | 65.3 | 52.5 | 51.7 | 79.7 | 24.6
cuad_termination_for_convenience | 70.5 | 67.7 | 47.9 | 83.7 | 50.0 | 49.8 | 64.4 | 42.8
cuad_third_party_beneficiary | 70.6 | 50.0 | 69.1 | 79.4 | 50.0 | 67.6 | 70.6 | 29.4
cuad_uncapped_liability | 46.9 | 53.4 | 53.4 | 61.2 | 51.0 | 77.9 | 36.4 | 33.0
cuad_unlimited-all-you-can-eat-license | 72.9 | 62.5 | 75.0 | 79.2 | 62.5 | 60.4 | 72.9 | 41.7
cuad_volume_restriction | 52.8 | 47.5 | 55.0 | 54.0 | 50.9 | 52.5 | 63.0 | 42.2
cuad_warranty_duration | 67.5 | 59.4 | 57.8 | 64.1 | 51.9 | 54.4 | 65.3 | 5.3
insurance_policy_interpretation | 34.5 | 42.8 | 35.4 | 36.9 | 43.2 | 34.9 | 33.7 | 29.1
jcrew_blocker | 45.6 | 46.7 | 47.8 | 51.1 | 58.9 | 57.8 | 56.7 | 11.1
maud_ability_to_consummate_concept_is_subject_to_mae_carveouts | 42.3 | 48.2 | 30.9 | 50.0 | 50.0 | 0.9 | 50.5 | 45.5
maud_financial_point_of_view_is_the_sole_consideration | 48.5 | 56.6 | 45.9 | 43.9 | 48.5 | 4.1 | 63.3 | 50.0
maud_accuracy_of_fundamental_target_rws_bringdown_standard | 33.3 | 34.5 | 31.7 | 35.9 | 34.4 | 31.6 | 37.5 | 33.3
maud_accuracy_of_target_general_rw_bringdown_timing_answer | 50.0 | 45.8 | 53.7 | 48.3 | 51.7 | 50.8 | 53.9 | 50.0
maud_accuracy_of_target_capitalization_rw_(outstanding_shares)_bringdown_standard_answer | 30.7 | 30.2 | 25.1 | 20.2 | 22.1 | 22.3 | 24.7 | 18.1
maud_additional_matching_rights_period_for_modifications_(cor) | 19.5 | 18.3 | 21.1 | 16.3 | 22.0 | 8.4 | 20.3 | 20.0
maud_application_of_buyer_consent_requirement_(negative_interim_covenant) | 47.8 | 40.0 | 50.0 | 55.6 | 50.9 | 50.0 | 47.5 | 43.1
maud_buyer_consent_requirement_(ordinary_course) | 50.6 | 54.1 | 49.7 | 50.0 | 50.0 | 7.4 | 57.5 | 50.0
maud_change_in_law__subject_to_disproportionate_impact_modifier | 36.4 | 45.8 | 11.2 | 49.4 | 50.6 | 12.2 | 50.0 | 50.0
maud_changes_in_gaap_or_other_accounting_principles__subject_to_disproportionate_impact_modifier | 35.9 | 48.4 | 10.0 | 53.3 | 50.6 | 7.1 | 50.0 | 50.0
maud_cor_permitted_in_response_to_intervening_event | 47.6 | 62.5 | 47.5 | 49.4 | 58.6 | 7.3 | 65.6 | 50.7
maud_cor_permitted_with_board_fiduciary_determination_only | 45.8 | 42.1 | 37.2 | 48.8 | 50.0 | 13.5 | 49.4 | 49.4
maud_cor_standard_(intervening_event) | 26.7 | 9.3 | 10.9 | 16.7 | 21.6 | 13.8 | 16.7 | 16.7
maud_cor_standard_(superior_offer) | 16.8 | 7.8 | 7.0 | 15.8 | 17.8 | 9.7 | 11.5 | 6.4
maud_definition_contains_knowledge_requirement_-_answer | 25.3 | 26.9 | 20.8 | 23.1 | 22.8 | 28.0 | 25.3 | 25.0
maud_definition_includes_asset_deals | 23.9 | 29.4 | 18.2 | 28.9 | 32.8 | 6.8 | 35.1 | 28.6
maud_definition_includes_stock_deals | 27.7 | 16.4 | 5.0 | 24.5 | 18.4 | 7.6 | 17.2 | 22.6
maud_fiduciary_exception__board_determination_standard | 4.3 | 9.6 | 6.9 | 12.7 | 14.6 | 9.9 | 13.7 | 1.0
maud_fiduciary_exception_board_determination_trigger_(no_shop) | 48.8 | 44.7 | 43.3 | 49.3 | 55.8 | 20.1 | 57.0 | 42.1
maud_fls_(mae)_standard | 38.3 | 16.0 | 13.0 | 28.6 | 1.8 | 1.8 | 24.6 | 0.0
maud_general_economic_and_financial_conditions_subject_to_disproportionate_impact_modifier | 54.8 | 62.5 | 38.1 | 48.8 | 50.6 | 14.3 | 50.0 | 50.0
maud_includes_consistent_with_past_practice | 50.0 | 48.2 | 61.6 | 49.6 | 52.5 | 5.4 | 52.8 | 50.0
maud_initial_matching_rights_period_(cor) | 20.7 | 11.2 | 21.5 | 23.4 | 27.2 | 10.8 | 21.0 | 18.9
maud_initial_matching_rights_period_(ftr) | 20.6 | 14.5 | 16.4 | 11.1 | 22.8 | 7.7 | 20.0 | 16.9
maud_intervening_event_-_required_to_occur_after_signing_-_answer | 50.2 | 51.5 | 43.4 | 50.0 | 42.6 | 43.5 | 47.5 | 50.0
maud_knowledge_definition | 47.1 | 46.8 | 40.0 | 46.7 | 51.8 | 49.3 | 50.7 | 45.5
maud_liability_standard_for_no-shop_breach_by_target_non-do_representatives | 59.6 | 50.0 | 50.0 | 53.8 | 50.0 | 46.8 | 58.3 | 50.0
maud_ordinary_course_efforts_standard | 34.2 | 32.8 | 41.9 | 32.9 | 77.4 | 32.7 | 34.2 | 33.8
maud_pandemic_or_other_public_health_event__subject_to_disproportionate_impact_modifier | 48.2 | 10.8 | 49.4 | 48.9 | 51.3 | 8.5 | 47.5 | 49.3
maud_pandemic_or_other_public_health_event_specific_reference_to_pandemic-related_governmental_responses_or_measures | 41.7 | 51.9 | 46.0 | 48.1 | 50.0 | 4.2 | 50.8 | 50.0
maud_relational_language_(mae)_applies_to | 37.0 | 47.1 | 18.8 | 50.0 | 7.1 | 16.5 | 28.3 | 46.4
maud_specific_performance | 60.4 | 50.0 | 49.2 | 43.5 | 58.5 | 0.6 | 39.4 | 50.0
maud_tail_period_length | 6.9 | 32.0 | 25.3 | 31.4 | 63.9 | 14.2 | 8.6 | 3.1
maud_type_of_consideration | 25.0 | 28.5 | 27.1 | 26.4 | 25.3 | 39.3 | 24.3 | 25.0
opp115_data_retention | 48.9 | 37.5 | 46.6 | 50.0 | 50.0 | 50.0 | 51.1 | 42.0
opp115_data_security | 53.4 | 60.3 | 53.8 | 63.6 | 50.2 | 49.8 | 63.7 | 45.9
opp115_do_not_track | 43.6 | 34.5 | 47.3 | 69.1 | 50.0 | 49.1 | 31.8 | 45.5
opp115_first_party_collection_use | 63.9 | 59.3 | 69.5 | 69.9 | 61.3 | 59.2 | 70.4 | 46.4
opp115_international_and_specific_audiences | 58.1 | 60.3 | 50.1 | 64.9 | 51.1 | 52.7 | 57.5 | 37.4
opp115_policy_change | 68.2 | 66.2 | 55.6 | 55.9 | 50.0 | 50.0 | 74.5 | 44.0
opp115_third_party_sharing_collection | 53.8 | 57.7 | 50.9 | 68.4 | 50.4 | 50.3 | 54.1 | 36.5
opp115_user_access,_edit_and_deletion | 60.2 | 51.5 | 56.1 | 64.5 | 54.7 | 50.8 | 59.0 | 41.7
opp115_user_choice_control | 63.6 | 40.0 | 46.5 | 64.5 | 50.2 | 48.6 | 47.4 | 44.2
privacy_policy_entailment | 51.4 | 49.3 | 56.8 | 58.1 | 57.7 | 64.1 | 52.8 | 26.6
privacy_policy_qa | 50.0 | 49.7 | 52.0 | 56.3 | 50.2 | 50.0 | 50.4 | 0.1
proa | 52.1 | 56.2 | 50.0 | 71.6 | 52.1 | 58.3 | 51.0 | 47.9
ssla_company_defendants | 35.5 | 44.9 | 54.8 | 54.1 | 60.7 | 51.0 | 50.9 | 0.0
ssla_individual_defendants | 14.3 | 17.4 | 20.8 | 19.6 | 23.1 | 20.8 | 13.9 | 0.0
ssla_plaintiff | 26.1 | 9.3 | 52.3 | 64.4 | 60.5 | 76.0 | 59.8 | 0.0
sara_entailment | 50.4 | 50.0 | 50.0 | 51.1 | 50.0 | 50.0 | 50.0 | 0.0
sara_numeric | 2.1 | 1.0 | 2.1 | 0.0 | 0.0 | 2.1 | 1.0 | 0.0
supply_chain_disclosure_best_practice_accountability | 6.2 | 55.8 | 50.0 | 58.4 | 50.4 | 56.8 | 32.7 | 31.5
supply_chain_disclosure_best_practice_audits | 1.5 | 71.6 | 55.9 | 63.0 | 55.8 | 42.2 | 31.1 | 8.7
supply_chain_disclosure_best_practice_certification | 2.8 | 67.1 | 52.1 | 57.6 | 52.3 | 40.8 | 51.5 | 11.3
supply_chain_disclosure_best_practice_training | 6.6 | 52.3 | 50.9 | 55.4 | 49.3 | 56.2 | 50.5 | 30.7
supply_chain_disclosure_best_practice_verification | 4.5 | 55.8 | 49.8 | 54.3 | 49.1 | 36.0 | 38.1 | 13.9
supply_chain_disclosure_disclosed_accountability | 2.1 | 48.2 | 49.7 | 48.7 | 49.0 | 24.6 | 20.4 | 15.7
supply_chain_disclosure_disclosed_audits | 3.6 | 50.2 | 48.7 | 49.6 | 51.5 | 52.0 | 49.7 | 16.3
supply_chain_disclosure_disclosed_certification | 3.2 | 50.5 | 52.0 | 52.2 | 54.9 | 45.8 | 32.3 | 20.1
supply_chain_disclosure_disclosed_training | 5.6 | 48.6 | 49.1 | 49.2 | 49.3 | 44.0 | 39.5 | 10.1
supply_chain_disclosure_disclosed_verification | 5.3 | 48.9 | 49.6 | 47.6 | 48.6 | 48.5 | 49.3 | 23.1
unfair_tos | 8.3 | 12.3 | 12.7 | 8.7 | 10.6 | 11.0 | 10.5 | 0.0

3B models on interpretation tasks. |  |  |  |
---|---|---|---|---
Task | BLOOM | Flan | Incite | Opt
Table 70 – continued from previous page |  |  |  |
Task | BLOOM | Flan | Incite | Opt
consumer_contracts_qa  | 0.8 | 93.2 | 46.1 | 31.7
contract_nli_confidentiality_of_agreement | 57.3 | 86.6 | 56.1 | 46.3
contract_nli_explicit_identification | 57.2 | 81.2 | 63.6 | 53.3
contract_nli_inclusion_of_verbally_conveyed_information | 50.1 | 82.2 | 67.8 | 60.9
contract_nli_limited_use | 63.1 | 72.8 | 60.1 | 63.0
contract_nli_no_licensing | 50.0 | 65.1 | 43.5 | 38.5
contract_nli_notice_on_compelled_disclosure | 57.0 | 66.2 | 65.5 | 71.8
contract_nli_permissible_acquirement_of_similar_information | 59.6 | 69.1 | 46.6 | 54.5
contract_nli_permissible_copy | 51.3 | 83.7 | 55.3 | 56.2
contract_nli_permissible_development_of_similar_information | 60.3 | 75.0 | 49.3 | 59.6
contract_nli_permissible_post-agreement_possession | 43.8 | 66.3 | 43.4 | 42.4
contract_nli_return_of_confidential_information | 51.6 | 82.3 | 61.3 | 51.9
contract_nli_sharing_with_employees | 53.3 | 69.7 | 47.1 | 52.8
contract_nli_sharing_with_third-parties | 50.0 | 80.5 | 51.4 | 49.6
contract_nli_survival_of_obligations | 50.0 | 72.9 | 49.4 | 44.1
contract_qa | 35.9 | 0.0 | 82.9 | 44.6
cuad_affiliate_license-licensee | 59.6 | 66.7 | 60.1 | 68.7
cuad_affiliate_license-licensor | 48.9 | 77.3 | 71.6 | 50.0
cuad_anti-assignment | 50.0 | 74.7 | 51.5 | 33.7
cuad_audit_rights | 50.0 | 56.3 | 56.2 | 64.6
cuad_cap_on_liability | 49.2 | 50.9 | 49.4 | 26.8
cuad_change_of_control | 51.4 | 59.6 | 60.8 | 47.4
cuad_competitive_restriction_exception | 46.4 | 73.2 | 45.5 | 40.5
cuad_covenant_not_to_sue | 50.0 | 59.1 | 49.4 | 44.8
cuad_effective_date | 51.7 | 92.4 | 47.0 | 65.7
cuad_exclusivity | 64.6 | 55.2 | 53.9 | 60.9
cuad_expiration_date | 50.2 | 67.7 | 52.1 | 69.3
cuad_governing_law | 56.8 | 84.4 | 50.5 | 74.8
cuad_insurance | 53.5 | 53.9 | 57.5 | 70.3
cuad_ip_ownership_assignment | 49.0 | 57.6 | 57.3 | 65.1
cuad_irrevocable_or_perpetual_license | 51.8 | 76.1 | 53.6 | 71.1
cuad_joint_ip_ownership | 48.4 | 65.1 | 64.6 | 68.8
cuad_license_grant | 50.1 | 69.1 | 51.4 | 69.2
cuad_liquidated_damages | 50.0 | 52.3 | 57.3 | 34.5
cuad_minimum_commitment | 52.7 | 54.1 | 50.6 | 57.0
cuad_most_favored_nation | 50.0 | 57.8 | 56.2 | 43.8
cuad_no-solicit_of_customers | 52.4 | 47.6 | 59.5 | 32.1
cuad_no-solicit_of_employees | 48.6 | 63.4 | 66.9 | 42.3
cuad_non-compete | 48.0 | 63.1 | 56.1 | 30.3
cuad_non-disparagement | 49.0 | 57.0 | 60.0 | 44.0
cuad_non-transferable_license | 56.8 | 58.7 | 50.9 | 62.0
cuad_notice_period_to_terminate_renewal | 59.5 | 61.3 | 54.5 | 68.5
cuad_post-termination_services | 50.0 | 64.0 | 50.7 | 55.9
cuad_price_restrictions | 56.5 | 50.0 | 58.7 | 47.8
cuad_renewal_term | 50.3 | 70.7 | 59.8 | 73.3
cuad_revenue-profit_sharing | 54.1 | 60.9 | 52.5 | 64.1
cuad_rofr-rofo-rofn | 50.3 | 49.4 | 51.6 | 55.4
cuad_source_code_escrow | 55.9 | 42.4 | 70.3 | 56.8
cuad_termination_for_convenience | 50.0 | 64.4 | 58.1 | 38.4
cuad_third_party_beneficiary | 58.8 | 79.4 | 50.0 | 64.7
cuad_uncapped_liability | 50.0 | 53.4 | 49.7 | 28.9
cuad_unlimited-all-you-can-eat-license | 66.7 | 81.2 | 54.2 | 70.8
cuad_volume_restriction | 50.3 | 49.4 | 57.5 | 59.6
cuad_warranty_duration | 53.1 | 61.9 | 51.6 | 54.4
insurance_policy_interpretation | 32.2 | 38.8 | 37.1 | 32.9
jcrew_blocker | 55.6 | 56.7 | 48.9 | 53.3
maud_ability_to_consummate_concept_is_subject_to_mae_carveouts | 47.3 | 13.4 | 50.0 | 49.1
maud_financial_point_of_view_is_the_sole_consideration | 53.1 | 49.0 | 50.0 | 43.4
maud_accuracy_of_fundamental_target_rws_bringdown_standard | 32.6 | 0.0 | 32.6 | 34.8
maud_accuracy_of_target_general_rw_bringdown_timing_answer | 50.0 | 0.0 | 48.2 | 53.2
maud_accuracy_of_target_capitalization_rw_(outstanding_shares)_bringdown_standard_answer | 26.1 | 0.0 | 23.4 | 28.2
maud_additional_matching_rights_period_for_modifications_(cor) | 20.0 | 0.0 | 18.8 | 20.3
maud_application_of_buyer_consent_requirement_(negative_interim_covenant) | 49.1 | 62.5 | 50.3 | 50.0
maud_buyer_consent_requirement_(ordinary_course) | 49.4 | 50.0 | 50.0 | 49.7
maud_change_in_law__subject_to_disproportionate_impact_modifier | 44.3 | 6.3 | 44.0 | 51.1
maud_changes_in_gaap_or_other_accounting_principles__subject_to_disproportionate_impact_modifier | 41.9 | 5.9 | 51.0 | 47.9
maud_cor_permitted_in_response_to_intervening_event | 51.2 | 48.8 | 71.7 | 50.6
maud_cor_permitted_with_board_fiduciary_determination_only | 49.4 | 36.3 | 50.0 | 45.9
maud_cor_standard_(intervening_event) | 16.7 | 1.0 | 16.7 | 16.7
maud_cor_standard_(superior_offer) | 10.8 | 4.1 | 8.8 | 5.4
maud_definition_contains_knowledge_requirement_-_answer | 24.7 | 0.0 | 26.2 | 25.0
maud_definition_includes_asset_deals | 34.4 | 0.0 | 22.5 | 33.3
maud_definition_includes_stock_deals | 40.4 | 0.0 | 9.9 | 0.0
maud_fiduciary_exception__board_determination_standard | 6.3 | 0.5 | 12.9 | 13.3
maud_fiduciary_exception_board_determination_trigger_(no_shop) | 56.7 | 50.0 | 50.5 | 50.0
maud_fls_(mae)_standard | 23.9 | 10.5 | 24.2 | 1.2
maud_general_economic_and_financial_conditions_subject_to_disproportionate_impact_modifier | 50.0 | 3.0 | 66.1 | 53.0
maud_includes_consistent_with_past_practice | 50.4 | 89.7 | 50.0 | 43.3
maud_initial_matching_rights_period_(cor) | 13.5 | 0.0 | 18.0 | 17.0
maud_initial_matching_rights_period_(ftr) | 18.9 | 0.0 | 19.6 | 19.4
maud_intervening_event_-_required_to_occur_after_signing_-_answer | 48.5 | 3.3 | 50.0 | 50.0
maud_knowledge_definition | 46.0 | 50.0 | 47.5 | 49.4
maud_liability_standard_for_no-shop_breach_by_target_non-do_representatives | 50.0 | 54.5 | 50.0 | 50.0
maud_ordinary_course_efforts_standard | 33.3 | 58.3 | 32.1 | 33.3
maud_pandemic_or_other_public_health_event__subject_to_disproportionate_impact_modifier | 2.6 | 62.3 | 60.4 | 47.4
maud_pandemic_or_other_public_health_event_specific_reference_to_pandemic-related_governmental_responses_or_measures | 50.0 | 59.9 | 50.0 | 50.0
maud_relational_language_(mae)_applies_to | 32.6 | 0.0 | 50.0 | 49.3
maud_specific_performance | 50.0 | 89.3 | 48.0 | 50.0
maud_tail_period_length | 1.5 | 0.0 | 35.6 | 25.0
maud_type_of_consideration | 24.4 | 2.1 | 27.2 | 25.2
opp115_data_retention | 45.5 | 52.3 | 64.8 | 39.8
opp115_data_security | 60.7 | 69.7 | 55.9 | 53.6
opp115_do_not_track | 45.5 | 55.5 | 42.7 | 37.3
opp115_first_party_collection_use | 60.9 | 52.3 | 69.5 | 56.1
opp115_international_and_specific_audiences | 59.9 | 70.5 | 51.8 | 57.0
opp115_policy_change | 52.1 | 60.4 | 61.2 | 57.4
opp115_third_party_sharing_collection | 55.6 | 63.3 | 64.8 | 47.0
opp115_user_access,_edit_and_deletion | 49.2 | 82.3 | 66.4 | 53.5
opp115_user_choice_control | 48.5 | 58.5 | 59.0 | 47.5
privacy_policy_entailment | 50.0 | 53.6 | 66.1 | 54.1
privacy_policy_qa | 50.3 | 61.0 | 55.1 | 50.5
proa | 54.1 | 80.1 | 64.5 | 53.1
ssla_company_defendants | 36.2 | 13.1 | 51.1 | 47.0
ssla_individual_defendants | 11.9 | 20.0 | 20.5 | 17.3
ssla_plaintiff | 40.1 | 86.4 | 36.8 | 42.8
sara_entailment | 48.5 | 0.0 | 50.7 | 48.9
sara_numeric | 3.1 | 1.0 | 0.0 | 1.0
supply_chain_disclosure_best_practice_accountability | 43.0 | 74.5 | 55.1 | 37.4
supply_chain_disclosure_best_practice_audits | 42.7 | 74.3 | 61.0 | 54.6
supply_chain_disclosure_best_practice_certification | 46.8 | 77.4 | 52.0 | 46.6
supply_chain_disclosure_best_practice_training | 44.9 | 83.4 | 64.2 | 41.0
supply_chain_disclosure_best_practice_verification | 47.7 | 54.9 | 53.1 | 21.8
supply_chain_disclosure_disclosed_accountability | 46.9 | 77.7 | 60.6 | 14.6
supply_chain_disclosure_disclosed_audits | 46.5 | 73.0 | 57.3 | 2.6
supply_chain_disclosure_disclosed_certification | 45.3 | 77.3 | 52.0 | 12.6
supply_chain_disclosure_disclosed_training | 47.9 | 87.2 | 58.8 | 16.9
supply_chain_disclosure_disclosed_verification | 44.1 | 66.1 | 55.3 | 0.2
unfair_tos | 9.0 | 8.2 | 4.0 | 15.3