Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

SwissText’23: The 8th edition of the Swiss Text Analytics Conference –
Generative AI & LLM, June 12–14, 2023, Neuchâtel, Switzerland

[orcid=0000-0003-0858-6977<EMAIL_ADDRESS>]

# Large Language Model Prompt Chaining for
Long Legal Document Classification

Dietrich Trautmann  Thomson Reuters Labs, Zug, Canton of Zug, Switzerland

(2023)

###### Abstract

Prompting is used to guide or steer a language model in generating an
appropriate response that is consistent with the desired outcome. Chaining is
a strategy used to decompose complex tasks into smaller, manageable
components. In this study, we utilize prompt chaining for extensive legal
document classification tasks, which present difficulties due to their
intricate domain-specific language and considerable length. Our approach
begins with the creation of a concise summary of the original document,
followed by a semantic search for related exemplar texts and their
corresponding annotations from a training corpus. Finally, we prompt for a
label - based on the task - to assign, by leveraging the in-context learning
from the few-shot prompt. We demonstrate that through prompt chaining, we can
not only enhance the performance over zero-shot, but also surpass the micro-F1
score achieved by larger models, such as ChatGPT zero-shot, using smaller
models.

###### keywords:

Prompt Chaining Prompt Engineering Long Legal Documents Legal NLP Legal AI

##  1 Introduction

The legal domain, with its often challenging tasks and complex long documents,
is an important field of study for natural language processing (NLP) and
machine learning [1, 2]. Long legal document text classification tasks can be
challenging due to several factors, including large size, complex language and
specific vocabulary, highly specialized content structure, imbalanced data
(many common cases vs. a long-tail of peculiar ones), subjectivity (open to
interpretation and debate), and the need for expensive, manual annotations
from subject matter experts.

The recent surge in the utilization of legal benchmarks has stimulated a
proliferation of innovative solutions harnessing pre-trained language models
[3]. Conventionally, these methodologies necessitate an intensive annotation
process (though some utilize metadata annotations), followed by a costly fine-
tuning process for the models [3, 4].

The advent of large-scale pre-training of large language models (LLMs) has
presented an opportunity to leverage them directly through natural language
prompting [5], circumventing the need for additional task-dependent fine-
tuning. Prompting involves providing a specific instruction, query, or
question to an LLM to generate a specific output or response. The input, or
prompt, steers the system towards producing a response meaningfully related to
it.

The technique of prompt chaining [6, 7] has shown promise in NLP, sequentially
linking multiple prompts to guide the generation process (Fig. 1). Through the
utilization of consecutive prompts, the system can produce more contextually
relevant responses for each step and more complex responses for the overall
task.

Figure 1: _Prompt Chaining_ for Legal Document Classification

Prompt chaining proves particularly advantageous in long legal document
classification, improving task performance, efficiency, flexibility, and
consistency via the inspection of individual steps in the chain [8]. The
technique enhances the interpretability of the overall classification and
permits debugging of complex reasoning tasks upon failure [9].

Overall, prompt chaining is a valuable tool in the classification of long
legal documents, helping to improve the performance and efficiency of the
classification process. Prompt chaining allows language models to build on
their previous outputs and provide more nuanced classification results, and it
can be customized to meet the specific needs of the legal document
classification task.

Our contributions in this work are:

  * •

We show that we can successfully chain prompts for legal document
classification tasks (visualized in Fig. 1).

  * •

We apply prompt chaining on one binary classification task and on one multi-
class text classification task.

  * •

We improve the results over zero-shot prompting with our chaining approach.

  * •

Our prompt chaining approach even outperforms zero-shot ChatGPT prompting on
the micro-f1 score.

##  2 Related Work

In terms of related literature we focus on the legal document classification
work and on current prompting approaches, as well as the combination of the
two fields.

###  2.1 Legal Document Classification

Documents, with their characteristic long textual data, often pose significant
challenges for automated machine learning methods in processing and
classification tasks [10], [11]. These challenges become more pronounced in
the legal domain due to additional complexities such as intricate grammar,
nested sentences, domain-specific vocabulary, and extensive use of
abbreviations [12].

The _LexGLEU_ benchmark [3] represents a comprehensive consolidation of recent
datasets involving long legal documents, exclusively in the English language.
It includes legal documents related to EU & US Law, as well as contracts with
tasks pertaining to multi-label and multi-class classification and multiple
choice question-answering. In [13], the authors evaluated a hierarchical
approach for modeling long documents, and [14] investigated strategies to
augment the context-window of transformers for domain-specific tasks,
including the aforementioned _LexGLEU_ benchmark. While benchmarks in other
languages do exist, such as _MultiEURLEX_ [4], our work will also focus solely
on the English language.

###  2.2 Prompting

Several noteworthy projects aim to consolidate, evaluate, and standardize
prompting approaches across diverse tasks and domains. The two most
substantial projects include OpenPrompt [15] and PromptSource [16].

##### OpenPrompt

provides a user-friendly and research-friendly toolkit for prompt-learning in
pre-trained language models (PLMs). The advent of prompt-learning in natural
language processing sparked a need for a standardized implementation
framework. OpenPrompt caters to this need by delivering a modular and
extendable toolkit that accommodates various PLMs, task formats, and prompting
modules in a unified paradigm.

##### PromptSource

is a toolkit designed to facilitate the development and sharing of natural
language prompts for training and querying language models in NLP. It offers a
templating language for creating data-linked prompts, a swift iterative
development interface, and a community-driven set of guidelines for
contributing new prompts. Currently, the platform offers over 2,000 prompts
for approximately 170 datasets, promoting collaboration and efficient
utilization of prompts for language model training and querying.

##### Prompt Chaining

as a concept was explored in [8], where prompts were chained using a visual
program editor. Now, there are frameworks with prompt chaining at their core,
such as LangChain [17], LLamaIndex [18], and MiniChain [19].

###  2.3 Legal Prompting

Recent research has combined prompting and natural language tasks in the legal
domain. In [20], authors evaluated zero-shot prompting on the legal judgment
prediction task using multilingual data from the European Court for Human
Rights (ECHR) and the Federal Supreme Court of Switzerland (FSCS). Meanwhile,
[21] appraised _GPT-3_ ’s zero- and few-shot capabilities for legal reasoning
tasks on the COLLIE entailment task (using English translations of the
Japanese bar exam).

The GPT-3.5 model was evaluated on the US Bar Exam [22], and GPT-4 [23] has
demonstrated proficiency in passing multiple tests through zero-shot
prompting.

##  3 Data

The datasets utilized in our study are sourced from two widely recognized
benchmarks comprising lengthy documents in the legal domain: the European
Court of Human Rights (ECHR) and the Supreme Court of the United States
(SCOTUS). These datasets form part of the _LexGLUE_ benchmark [3].

###  3.1 ECHR

The ECHR dataset comprises approximately $11,000$ cases sourced from the
European Court of Human Rights public database. This dataset is split into
training, development, and test sets. Each case includes factual paragraphs,
along with the corresponding ECHR articles that were violated or alleged to be
violated. The original task involves to predict the violated human rights
articles for a case’s facts. However, for the purpose of our study, we have
simplified this task to a binary classification problem. We distinguish cases
based on whether there was a violation of any human rights articles,
irrespective of which specific articles were violated.

###  3.2 SCOTUS

The SCOTUS dataset provides insight into the highest federal court in the USA,
which handles complex or controversial cases unresolved by lower courts. This
dataset combines information from SCOTUS opinions and the Supreme Court
DataBase (SCDB). The SCDB offers metadata for all cases spanning from 1946 to
2020. Utilizing the SCDB, the dataset classifies court opinions into 14 issue
areas (refer to App. B). The dataset is divided chronologically into training
(#5000 samples, 1946–1982), development (#1400 samples, 1982–1991), and test
(#1400 samples, 1991–2016) sets, each covering a distinct time period.

##  4 Models

Our experimental design incorporates both general-purpose text generation
models and task-specific summarization models.

###  4.1 Generation Models

We used two different 20 billion parameter LLMs in our text generation steps.
Both of the models have a context window of up to 2048 tokens.

####  4.1.1 GPT-NeoX

The _GPT-NeoX_ model [24] is an autoregressive language model, specifically a
decoder-only model111https://hf.co/EleutherAI/gpt-neox-20b, trained on the
Pile dataset [25]. This models’ weights are openly available under a
permissive license (Apache 2.0
222https://www.apache.org/licenses/LICENSE-2.0.html).

####  4.1.2 Flan-UL2

_Flan-UL2_ [26] is an encoder-decoder model333https://hf.co/google/flan-ul2
based on the T5 architecture, trained with the mixture-of-denoisers objectives
(diverse span corruption and prefix language modeling tasks). This model was
further instruction fine-tuned using the Flan prompting collection [27]. The
collection contains instructions for a diverse set of tasks (e.g., to
summarize a text; to classify based on a list of options). The model is
publicly available with an open source license (Apache 2.0).

###  4.2 Summarization Models

We also used task-specific summarization models for the creation of the legal
summaries. In our experiments we found that – due to the lack of ground-truth
summaries for our long legal documents – the task-specific summarization
models created more coherent summaries compared to results from prompting the
general generation models.

####  4.2.1 BRIO

The BRIO444https://hf.co/Yale-LILY/brio-cnndm-uncased model [28] is an
abstractive summarization model that achieves state-of-the-art result on the
_CNN/DailyMail_ and _XSum_ datasets. It uses BART [29] as its base model and
has a context window of 1024 tokens.

####  4.2.2 PRIMERA

The PRIMERA555https://hf.co/allenai/primera-multi_lexsum-source-short model
[30] is an abstractive summarization model that was trained on the _Multi-
LexSum_ dataset at different granularities. We used the model that was trained
on the granularity from the full source document to create a short summary and
it has a context window of 1024 tokens. The other options are – besides
different model architectures – long and tiny summaries.

###  4.3 Semantic Similarity Search

We use semantic similarity search for the few-shot prompt building were we
retrieve semantic similar summaries from the training set to a target summary
(either from the development or test set). For this purpose, our summaries
were encoded using the _sentence-transformers_ library [31] and the _custom-
legalbert_ 666https://hf.co/zlucia/custom-legalbert model [32]. Furthermore,
we used the _annoy_ 777https://github.com/spotify/annoy library for the
approximate nearest neighbor search (semantic similarity search).

##  5 Prompt Chaining

Prompt Chaining is a methodology employed to decompose complex tasks into
smaller, manageable sub-tasks. A prompt chain typically comprises several
prompts - either task-specific or general-purpose, each serving a single
purpose. The output of one prompt feeds into the next as an input. In our
approach, we utilize pre-defined steps; however, it’s worth noting that this
methodology could be further optimized by incorporating a selection process
for the next step or by introducing stopping criteria in the output
generation, as exemplified in [9] and [33]. The steps for our prompt chaining
process, specifically utilized for the classification of long legal documents,
are depicted in Fig. 1. In the following sections, we will delve into each of
the primary steps, namely summarization, few-shot prompt building, and final
label generation.

###  5.1 Summary Generation

The inaugural step in our prompt chaining approach is the generation of a
succinct summary of the legal case text. As a majority of legal documents are
lengthy, often exceeding the large context window of $2048$ tokens provided by
contemporary language models, we advocate the creation of summaries for chunks
of the whole document. These chunks are crafted based on the model’s context
window. Sentences are sequentially stacked until the context limit of the
respective models is reached ($1024$ or $2048$ tokens). Post the initial
parsing of the full document, the summary generation process for chunks is
iteratively continued until the desired summary length, in our case up to
$128$ tokens, is achieved. These summaries typically consist of a few
sentences (up to 5) derived from our data.

Initial experimentation with direct prompting approaches on large language
models resulted in variable outcomes. The templates used for prompting
included:

  * •

INPUT TEXT _In summary,_

  * •

INPUT TEXT _TLDR:_

Since we do not possess ground-truth summaries for the documents, our
assessments relied on manual inspection of a subset of the summaries generated
(from the training set). The inspection indicated that the summaries were
relatively generic and often omitted the core legal issues of interest.

Consequently, our investigation steered towards task-specific summarization
models. Notably, the _BRIO_ model, being pre-trained and fine-tuned on news
articles, generated more generic summaries. In contrast, the _PRIMERA_ model,
fine-tuned specifically on legal documents, generated summaries where the core
legal context was mostly preserved. This iterative summarization was uniformly
applied across all documents using the same parameters.

###  5.2 Semantic Similarity Search

The objective at this stage was to construct few-shot prompts for the
subsequent label generation step. Thus, we embedded all our summaries with the
models discussed in section 4.3 and calculated the semantically closest
neighbors (up to eight) of each summary in the development and test sets, as
compared to the training set. These summaries from the training set, along
with their true labels and the current target sample (excluding its true label
from the development and test sets), served as the few-shot prompts. This
approach leverages the in-context learning abilities of large language models.

As illustrated by [34], an alternative approach could involve prompting a
large language model to generate contextual texts based on an input, instead
of retrieving from a corpus database as we have done. These generated samples
could then be incorporated as in-context samples in the following step. This
option can also be potentially implemented via LLMs prompting in our prompting
pipeline. The evaluation of this approach’s feasibility is left for future
work.

###  5.3 Label Generation

The final step in our prompt chain involves label generation. Here, we queried
the LLMs with the few-shot prompts previously constructed, in conjunction with
an instruction for the corresponding task and a list of potential labels to
choose from. For the ECHR experiments, the binary options provided were _YES_
or _NO_ , while for the SCOTUS experiments, up to 13 issue area labels were
presented for model prediction. This prompt construction enabled the models to
yield the desired label in all our experiments. Greedy decoding was used in
this generation step for all results.

Another strategy employed involved sampling from the model output multiple
times, a technique known as self-consistency via output sampling [35]. It has
been demonstrated that querying multiple times enhances the probability of
generating the true label. At the conclusion of each such sampling (up to 10
times), the majority count of the generated labels was taken as the final
prediction.

##  6 Experiments

Our experiments pursued two objectives. Firstly, we aimed to enhance the zero-
shot outcomes from prior work on the binary classification task on the ECHR
dataset, leveraging prompting techniques without any parameter adjustments to
the models. Following the successful demonstration of the efficacy of the few-
shot approach, we expanded our focus. The second experiment involved extending
the process to the 13 labels in the SCOTUS corpus, a task significantly more
challenging than the prior binary classification.

In addition to this, we compared our results to the zero-shot ChatGPT results
reported in [36], which covered a subset of the overall samples in the SCOTUS
data. We selected the corresponding samples and provided our results for
comparison.

Over multiple iterations, we developed the few-shot prompts on a randomly
selected portion (n = 40) of the development sets for both the _ECHR_ and the
_SCOTUS_ datasets. The final eight-shot prompt incorporated the eight
semantically closest summaries from the training set to the corresponding
target set, as well as the respective gold label from the training set.
Notably for the _SCOTUS_ corpus, we limited the available issue area labels
for the model, based on the labels of the included eight samples. Once we
identified the most effective composition of the few-shot prompt on the random
sample, we applied it to the full development and test sets, and reported the
results.

Our computational requirements were limited to CPU resources, as we only
performed inference calls on the generative models. Our work did not involve
the use of any GPUs. Detailed computational information is available in the
appendix (see App. A).

##  7 Results

In this section, we discuss our results on both benchmarks. We have included
the confusion matrices for the development sets (see Fig. 2 for ECHR and Fig.
3 for SCOTUS), the labelwise F1-scores (see Tab. 1 and Tab. 4), and the
overall results (see Tab. 2 and Tab. 3).

###  7.1 ECHR Results

With respect to the ECHR results, we managed to improve upon the zero-shot
results from previous work. However, the few-shot context still did not
suffice to reach the performance of supervised trained models. It’s important
to remember that full supervised fine-tuning involves hours of update runs
with thousands of annotated samples, while our experiments only included
inference calls. The confusion matrix (Fig. 2) also demonstrates some
misclassification along the off-diagonal axis, although the few-shot prompting
did capture more of the minority (_NO_) class.

Figure 2: Confusion matrix for the dev. set of _ECHR_. # | Label | # Samples | F1
---|---|---|---
1 | Yes | 825 | 0.864
2 | No | 175 | 0.248
Table 1: Labelwise F1-Scores for the development set of _ECHR_.

###  7.2 SCOTUS Results

The results for the multi-class text classification task for _SCOTUS_ are
presented in Tab. 2. Alongside our results on the development and test sets,
we also included external (ext.) results from [3] and [36].

While we achieved satisfactory performance on the development set, we observed
a substantial drop in performance on the test set. The test set performance in
terms of macro-f1 score was below the zero-shot ChatGPT results. However, our
prompt chaining approach was more effective in retrieving the higher frequency
classes, as reflected in the better micro-f1 score.

This trend is also evident in the label-wise scores (Tab. 4), where the higher
frequency classes received better scores than the minority classes. The
confusion matrix (Fig. 3) for this experiment showed that particularly many
issue areas were predicted as _civil rights_ , while also the _criminal
procedure_ , _judicial power_ and _federalism_ were misclassified as others.

| Model | Precision | Recall | macro-F1 | micro-F1 | weighted-F1 | Accuracy
---|---|---|---|---|---|---|---
dev. set | minority class | .088 | .500 | .149 | .175 | .052 | .175
random class | .506 | .510 | .451 | .514 | .572 | .514
majority class | .412 | .500 | .452 | .825 | .746 | .825
GPT-NeoX (0-shot, (F)) | .527 | .536 | .526 | .709 | .731 | .709
GPT-NeoX (8-shot, (S)) | .566 | .552 | .556 | .770 | .756 | .770
test set | minority class | .077 | .500 | .133 | .153 | .041 | .153
random class | .479 | .460 | .410 | .484 | .555 | .484
majority class | .423 | .500 | .459 | .847 | .777 | .847
GPT-NeoX (0-shot, (F)) | .522 | .530 | .521 | .707 | .728 | .707
GPT-NeoX (8-shot, (S)) | .525 | .537 | .527 | .779 | .768 | .779
Table 2:  The results for the ECtHR development and test sets. Besides the macro-averaged F1-score, precision and recall, we report also the micro-averaged and weighted-F1 and the accuracy scores. The (F) stands for the full document that was used in the input to the model, while the (S) stands for the summaries (concatenated in the few-shot prompts) as the input to the model.  | Model | Precision | Recall | macro-F1 | micro-F1 | weighted-F1 | Accuracy
---|---|---|---|---|---|---|---
dev. set | majority class | .020 | .077 | .031 | .257 | .105 | .257
random class | .070 | .064 | .057 | .071 | .084 | .071
FLAN-UL2 (8-shot, (S)) | .529 | .455 | .461 | .545 | .543 | .545
test set | majority class | .020 | .077 | .032 | .266 | .112 | .266
random class | .079 | .074 | .060 | .077 | .095 | .077
FLAN-UL2 (8-shot, (S)) | .427 | .373 | .359 | .486 | .483 | .486
FLAN-UL2 (8-shot, (S))† | .435 | .388 | .371 | .484 | .480 | .484
ext. | ChatGPT (0-shot, (F)) | - | - | .420 | .438 | - | -
supervised (full) | - | - | .695 | .782 | - | -
Table 3:  The results for the SCOTUS development and test sets. Besides the macro-averaged F1-score, precision and recall, we report also the micro-averaged and weighted-F1 and the accuracy scores. The (F) stands for the full document that was used in the input to the model, while the (S) stands for the summaries (concatenated in the few-shot prompts) as the input to the model. †We calculated the scores based on the same reduced set of documents (1k) as the ChatGPT work. The ext. rows are external results copied from the corresponding papers [3, 36].  # | Label | # Samples | F1
---|---|---|---
1 | Criminal Procedure | 360 | 0.742
2 | Federal Taxation | 226 | 0.695
3 | First Amendment | 218 | 0.644
4 | Unions | 165 | 0.590
5 | Economic Activity | 108 | 0.577
6 | Civil Rights | 83 | 0.551
7 | Privacy | 70 | 0.529
8 | Interstate Relations | 51 | 0.500
9 | Federalism | 38 | 0.308
10 | Judicial Power | 35 | 0.299
11 | Attorneys | 22 | 0.255
12 | Due Process | 14 | 0.173
13 | Miscellaneous | 10 | 0.133
Table 4: Labelwise F1-Scores for the development set of _SCOTUS_. Figure 3:
Confusion matrix for the dev. set of _SCOTUS_.

##  8 Conclusion

Our experiments successfully demonstrated that the implementation of few-shot
prompts can lead to improvements upon the zero-shot results. We also showed
that it is feasible to predict frequent labels with appreciable F1 scores
using this approach.

The strategy of prompting, and as we have demonstrated, the concept of prompt
chaining, represent promising avenues for future exploration. These techniques
are particularly advantageous as they circumvent the need for costly data
annotation and the development of custom models.

Last but not least, established prompting pipelines can be adapted for use
with different (updated) models and, as shown in [23], they offer across-the-
board enhancements for a diverse range of tasks, free of cost. Looking ahead,
our future work aims to experiment with even larger models on additional legal
benchmarks.

## References

  * Dale [2019] R. Dale,  Law and word order: Nlp in legal tech,  Natural Language Engineering 25 (2019) 211–217.
  * Zhong et al. [2020] H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, M. Sun,  How does nlp benefit legal system: A summary of legal artificial intelligence,  in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 5218–5230.
  * Chalkidis et al. [2022] I. Chalkidis, A. Jana, D. Hartung, M. Bommarito, I. Androutsopoulos, D. Katz, N. Aletras,  Lexglue: A benchmark dataset for legal language understanding in english,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 4310–4330.
  * Chalkidis et al. [2021] I. Chalkidis, M. Fergadiotis, I. Androutsopoulos,  Multieurlex-a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer,  in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 6974–6996.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.,  Training language models to follow instructions with human feedback,  Advances in Neural Information Processing Systems 35 (2022) 27730–27744.
  * Wu et al. [2022] T. Wu, M. Terry, C. J. Cai,  Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts,  in: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022, pp. 1–22.
  * Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, D. Zhou,  Chain of thought prompting elicits reasoning in large language models,  arXiv preprint arXiv:2201.11903 (2022).
  * Wu et al. [2022] T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, C. J. Cai,  Promptchainer: Chaining large language model prompts through visual programming,  in: CHI Conference on Human Factors in Computing Systems Extended Abstracts, 2022, pp. 1–10.
  * Khattab et al. [2022] O. Khattab, K. Santhanam, X. L. Li, D. Hall, P. Liang, C. Potts, M. Zaharia,  Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp,  arXiv preprint arXiv:2212.14024 (2022).
  * Wagh et al. [2021] V. Wagh, S. Khandve, I. Joshi, A. Wani, G. Kale, R. Joshi,  Comparative study of long document classification,  in: TENCON 2021-2021 IEEE Region 10 Conference (TENCON), IEEE, 2021, pp. 732–737.
  * Park et al. [2022] H. H. Park, Y. Vyas, K. Shah,  Efficient classification of long documents using transformers,  arXiv preprint arXiv:2203.11258 (2022).
  * Garimella et al. [2022] A. Garimella, A. Sancheti, V. Aggarwal, A. Ganesh, N. Chhaya, N. Kambhatla,  Text simplification for legal domain: Insights and challenges,  NLLP 2022 2022 (2022) 296–304.
  * Chalkidis et al. [2022] I. Chalkidis, X. Dai, M. Fergadiotis, P. Malakasiotis, D. Elliott,  An exploration of hierarchical attention transformers for efficient long document classification,  arXiv preprint arXiv:2210.05529 (2022).
  * Mamakas et al. [2022] D. Mamakas, P. Tsotsi, I. Androutsopoulos, I. Chalkidis,  Processing long legal documents with pre-trained transformers: Modding legalbert and longformer,  arXiv preprint arXiv:2211.00974 (2022).
  * Ding et al. [2022] N. Ding, S. Hu, W. Zhao, Y. Chen, Z. Liu, H. Zheng, M. Sun,  Openprompt: An open-source framework for prompt-learning,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2022, pp. 105–113.
  * Bach et al. [2022] S. Bach, V. Sanh, Z. X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Févry, et al.,  Promptsource: An integrated development environment and repository for natural language prompts,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2022, pp. 93–104.
  * Chase [2022] H. Chase, LangChain, 2022. URL: https://github.com/hwchase17/langchain.
  * Liu [2022] J. Liu, LlamaIndex, 2022. URL: https://github.com/jerryjliu/gpt_index.
  * Rush [2023] S. Rush, Mini-Chain, 2023. URL: https://github.com/srush/MiniChain/.
  * Trautmann et al. [2022] D. Trautmann, A. Petrova, F. Schilder,  Legal prompt engineering for multilingual legal judgement prediction,  arXiv preprint arXiv:2212.02199 (2022). URL: https://doi.org/10.48550/arXiv.2212.02199.
  * Yu et al. [2022] F. Yu, L. Quartey, F. Schilder,  Legal prompting: Teaching a language model to think like a lawyer,  arXiv preprint arXiv:2212.01326 (2022).
  * Bommarito II and Katz [2022] M. Bommarito II, D. M. Katz,  Gpt takes the bar exam,  arXiv preprint arXiv:2212.14402 (2022).
  * Katz et al. [2023] D. M. Katz, M. J. Bommarito, S. Gao, P. Arredondo,  Gpt-4 passes the bar exam,  Available at SSRN 4389233 (2023).
  * Black et al. [2022] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al.,  Gpt-neox-20b: An open-source autoregressive language model,  Challenges & Perspectives in Creating Large Language Models (2022) 95.
  * Gao et al. [2020] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al.,  The pile: An 800gb dataset of diverse text for language modeling,  arXiv preprint arXiv:2101.00027 (2020).
  * Tay et al. [2022] Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, D. Bahri, T. Schuster, H. S. Zheng, N. Houlsby, D. Metzler,  Unifying language learning paradigms,  arXiv preprint arXiv:2205.05131 (2022).
  * Longpre et al. [2023] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al.,  The flan collection: Designing data and methods for effective instruction tuning,  arXiv preprint arXiv:2301.13688 (2023).
  * Liu et al. [2022] Y. Liu, P. Liu, D. Radev, G. Neubig,  Brio: Bringing order to abstractive summarization,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 2890–2903.
  * Lewis et al. [2020] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer,  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,  in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7871–7880.
  * Shen et al. [2022] Z. Shen, K. Lo, L. Yu, N. Dahlberg, M. Schlanger, D. Downey,  Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities,  in: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * Reimers and Gurevych [2019] N. Reimers, I. Gurevych,  Sentence-bert: Sentence embeddings using siamese bert-networks,  in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2019\. URL: http://arxiv.org/abs/1908.10084.
  * Zheng et al. [2021] L. Zheng, N. Guha, B. R. Anderson, P. Henderson, D. E. Ho,  When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings,  in: Proceedings of the eighteenth international conference on artificial intelligence and law, 2021, pp. 159–168.
  * Schick et al. [2023] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, T. Scialom,  Toolformer: Language models can teach themselves to use tools,  arXiv preprint arXiv:2302.04761 (2023).
  * Yu et al. [2022] W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, M. Jiang,  Generate rather than retrieve: Large language models are strong context generators,  arXiv preprint arXiv:2209.10063 (2022).
  * Wang et al. [2022] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, D. Zhou,  Self-consistency improves chain of thought reasoning in language models,  arXiv preprint arXiv:2203.11171 (2022).
  * Chalkidis [2023] I. Chalkidis,  Chatgpt may pass the bar exam soon, but has a long way to go for the lexglue benchmark (2023). URL: https://dx.doi.org/10.2139/ssrn.4385460.

##  Appendix A Compute Requirements

We used the following Amazon EC2 M5 instance:

Standard Instance | vCPU | Memory
---|---|---
ml.m5d.24xlarge | 96 | 384 GiB

We haven’t used any GPUs in our experiments.

##  Appendix B SCOTUS issue areas

  * •

Criminal Procedure

  * •

Civil Rights

  * •

First Amendment

  * •

Due Process

  * •

Privacy

  * •

Attorneys

  * •

Unions

  * •

Economic Activity

  * •

Judicial Power

  * •

Federalism

  * •

Interstate Relations

  * •

Federal Taxation

  * •

Miscellaneous

  * •

Private Action was not available in the data