[2]Aman Tiwari

These authors contributed equally to this work.

These authors contributed equally to this work.

[1] Agami [2] Thoughtworks Technologies India Pvt Ltd

# Aalap: AI Assistant for Legal & Paralegal Functions in India

<EMAIL_ADDRESS>Prathamesh Kalamkar<EMAIL_ADDRESS>Atreyo Banerjee<EMAIL_ADDRESS>Saurabh Karn<EMAIL_ADDRESS>Varun Hemachandran<EMAIL_ADDRESS>Smita Gupta<EMAIL_ADDRESS>*  *

###### Abstract

Using proprietary Large Language Models on legal tasks poses challenges due to
data privacy issues, domain data heterogeneity, domain knowledge
sophistication, and domain objectives’ uniqueness. We created Aalalp, a fine-
tuned Mistral 7B model on instructions data related to specific Indian legal
tasks. The performance of Aalap is better than gpt-3.5-turbo in 31% of our
test data and obtains an equivalent score in 34% of the test data as evaluated
by GPT4. Training Aalap mainly focuses on teaching legal reasoning rather than
legal recall. Aalap is definitely helpful for the day-to-day activities of
lawyers, judges, or anyone working in legal systems.

##  1 Introduction

Large Language Models (LLMs) have triggered the creation of innovative
applications in the legal field. While many LLM-powered applications in the
legal domain use cloud-hosted LLMs like OpenAI, Bard, etc., there are some
challenges. Data privacy issues and regulatory requirements often require that
data can’t leave an organization’s boundary. This may require an LLM to be
hosted on-premise within the organization. Hosting larger models with tens of
billions of parameters can be cost-prohibitive. Hence, there is a need to
distill the knowledge of larger models into smaller models that work well on
specific legal tasks. While state-of-the-art models are very good at bar exams
and other objective legal assessments [1], there is still room for
improvement. [2], [3] and [4] show areas where the gaps of state-of-the-art
LLMs for legal tasks.

Fine-tuning a general-purpose LLM on specific legal tasks can alleviate some
of these concerns. Hence, we created Aalap, which stands for AI Assistant for
Legal and Paralegal functions in India. The following are the main
contributions of this paper.

  * •

Aalap instructions
dataset111https://huggingface.co/datasets/opennyaiorg/aalap_instruction_dataset:
22272 legal instructions and synthetically generated responses with a maximum
input token length of 16k

  * •

Aalap model 222https://huggingface.co/opennyaiorg/Aalap-Mistral-7B-v0.1-bf16:
7B Mistral fine-tuned on the Aalap dataset with a maximum context length of
32k which can perform well for specific legal tasks in Indian context.

  * •

AIBE data 333https://huggingface.co/datasets/opennyaiorg/aibe_dataset: AIBE
dataset containing 1158 multiple-choice questions and answers from past 12
AIBE exams.

Aalap model weights and the dataset are available on huggingface.

##  2 Related Work

Domain adaption of LLMs can be done in three broad ways [5]. The first one is
pretraining from scratch on domain-specific raw text datasets. Some examples
of this approach are BloombergGPT [6] and Med-PaLM2 [7]. While this approach
seems promising, it needs a lot of computing power and data as guided by
Chinchilla scaling laws [8]. The second approach is fine-tuning an LLM using
domain-specific supervised datasets. This approach needs less computing power
and data than pre-training from scratch. Such smaller fine-tuned models can
sometimes match the performance of much larger models on specific tasks [9].
approach. The third approach is using Retriever Augmented Generation [10]
[11], which augments the prompt of an LLM with retrieved domain, and no
pretraining or fine-tuning is involved. This method also involves prompt
engineering techniques that guide LLMs to accomplish specific tasks.

Domain adaptation of general-purpose LLMs to law via supervised fine-tuning
has been widely studied. Notable ones are LlawyerLlama [12], a Chinese legal
LLM trained on a large-scale legal dataset, and ChatLaw [11], a Legal Large
Language Model with Integrated External Knowledge Bases. Both models rely on
retrievers to extract relevant external knowledge and generate the required
answers using LLM abilities.

Prompting general-purpose LLMs using legal reasoning frameworks like IRAC
(Issue, Rule, Application, Conclusion) coupled with few-shot examples is an
effective way to extract more from LLMs without the need of fine-tuning [13].
[14] have used legal prompt engineering for Multilingual Legal Judgement
Prediction.

Evaluation of law-specific LLMs needs benchmarks that focus on legal tasks.
Since variations in legal processes by country are significant, developing a
single global legal LLM evaluation benchmark is hard. Legalbench is a
collaboratively constructed legal reasoning benchmark comprising 162 tasks in
the USA law context [2]. LawBench [15] is a similar benchmark in the Chinese
context. However, such legal benchmarks in the Indian context do not exist to
the best of our knowledge.

Legal texts are usually longer; tasks like argument generation need input
information from multiple documents. Hence, a legal LLM needs to handle longer
input texts. Recent advances in training and inference of LLMs allow for
handling longer texts at lesser compute. The most significant advances are
Grouped Query Attention [16], and Sliding Window Attention [17]. Mistral 7B
[18] is an open-source LLM that has leveraged such advances and performs well
on a wide range of benchmarks. Instruction fine-tuning methods like LoRA [19]
and QLoRA [20] allow for parameter-efficient fine-tuning of LLMs.

##  3 Aalap Dataset

The main objective behind creating Aalap was to train a relatively smaller LLM
specializing in specific legal tasks focusing on legal reasoning. Hence,
prioritizing which legal tasks to focus on was an important step. We discussed
with Legal practitioners which legal tasks they want to be automated. Then, we
selected the most common legal tasks for publicly available input datasets.
The selected legal tasks are mentioned in table 1

We followed the following principles during data creation

  1. 1.

Focus on legal reasoning rather than legal recall: Relying on LLMs to generate
legal precedents and statute definitions is a bad idea. This is because LLMs
are more likely to hallucinate and generate false content. Instead, precedents
information and statute definitions should be retrieved from authentic
sources, given as input to LLMs, and let LLMs do the legal reasoning.

  2. 2.

Use real-life situations: The datasets should contain real-life situations as
much as possible. Hence, we used descriptions from court judgments and police
First Investigation Reports (FIR) so that the model learns the language used
in such texts.

  3. 3.

Explain answers: ORCA [21] style explanations of answers help teach models the
reasons behind coming up with answers to multiple-choice questions. We created
GPT4 to create explanations or reuse relevant legal datasets that provided
such explanations.

  4. 4.

Use synthetically generated responses if needed: Many times, the actual
documents, like lawyers’ pleadings, witness statements, medical reports, etc.,
are not available publicly. So, synthetically generated responses from OpenAI
models were used wherever needed.

###  3.1 Common Data Sources

The following are the commonly used data sources for preparing the input
instructions for Aalap dataset.

  * •

Judgments Facts: Case facts are typically written in a court judgment. Such
facts from judgments texts were extracted using Opennyai’s rhetorical roles
model [22] on randomly chosen Indian Supreme Court & high court judgments.
These judgment facts were inputs for tasks like argument generation, issues
generation, and event timeline creation.

  * •

Judgments Statutes: Relevant Statutes of a judgment are the ones that were
extracted from corresponding judgment text using Opennyai’s Named Entity
Recognition model [23].

  * •

FIRs: Publicly available police First Investigation Reports (FIRs) from the
states of Maharashtra and Delhi were used to collect descriptions of real-life
events. These FIRs were used to create event timeline creation tasks.

###  3.2 Data Creation

The datasets for prioritized legal tasks were created as described in table 1.

Task |  Task Description |  Data Creation Methodology
---|---|---
Issues Generation |  Create legal issues for a court based on the case facts. Legal issues are the key points on which the verdict needs to be delivered. |  Judgment facts were then sent to gpt-3.5-turbo to generate legal issues.
Argument Generation |  Based on the facts of a case, legal issues, and applicable statute names, generate arguments for a given party. |  Arguments for the petitioners were created using gpt-3.5-turbo, using judgment facts, generated issues, and relevant statutes as inputs. Counterarguments for defendants were created using petitioners’ generated arguments.
Event Timeline |  Extract important events and their dates from the input text descriptions and output a chronologically sorted event list with dates and a brief description. |  FIRs and judgment facts were used as input text descriptions. These were sent to gpt-3.5-turbo to create event timelines.
Combine Event Timelines |  For extraction of event timelines from very long texts, it is often split into chunks, and the event timeline for each chunk is created independently, which is merged later. |  Individually created timelines from the same judgment or FIR were merged using gpt-3.5-turbo.
Legalbench |  Training data for the legalbench [2] was filtered to keep only the legal reasoning tasks. |  ORCA-style explanations of these MCQs were created using GPT4 in a zero-shot setting.
Statute Ingredients |  Break the input statute definition into the ingredients needed to apply the statute. |  Definitions of the most popular sections of Indian central acts were used to generate the statute ingredients using gpt-3.5-turbo.
Summary Generation |  Create a summary in judgment headnotes format using the input court judgment text |  Indian Supreme Court judgments from 1950 to 1994 are published with headnotes, which are summaries of those judgments.
Legal Open ORCA |  OpenORCA dataset [24] is an ORCA-style explanation of the Natural Instructions dataset. |  The Natural Instruction dataset (NI) is filtered for law and matched against the 1M GPT4 openORCA dataset using tf-idf matching to get legal OpenORCA data.
Contract Clause Generation |  Generation of new contract clauses and modification of existing contract clauses. |  Existing Data111https://huggingface.co/datasets/NebulaSense/Legal_Clause_Instructions
Legal NIv2 MCQ |  Natural Instructions v2 data [25] was filtered for law-related questions. |  A random sample of NI dataset MCQs filtered for law.
Constitution General Knowledge |  Q&A about the Indian Constitution. |  Existing data222https://huggingface.co/datasets/nisaar/Constitution_of_India
Incomplete Instructions |  The given information is incomplete for completing the task. In such cases, the response is to ask for the required information. |  Randomly selected instructions belonging to each of these tasks above where the information is incomplete.
Table 1:  Prioritized Legal Tasks

Summary statistics of various task categories and licenses associated with
each dataset are shown in Table 5.

###  3.3 Data Limitations

Precedent Information is missing in the argument generation. Precedents are
essential elements to support generated arguments. Fetching the proper
paragraphs from authentic precedent sources based on the situation is
necessary to build an argument. Since we did not have access to a semantic
precedents search engine that can do this, we have excluded precedents
information from generated arguments.

Facts in the judgments can be worded much differently than what is available
from statements, filings, and other reports that lawyers get as inputs. Since
this information is not public, we had to rely on publicly available datasets.

Since the Indian judiciary operates mainly in English, all the datasets are
currently in the English language.

##  4 Model Training

To create a general-purpose Large Language Model (LLM) tailored for specific
legal tasks, a model architecture with a substantial context length of 32K,
namely Mistral 7B [18], was chosen. This intentional selection is derived from
our dataset’s distinctive nature and extensive length as well as Mistral 7B’s
superior capabilities when compared to other models, including the LLama2
family. However, the fine-tuning of such a model necessitates considerable
computational resources and comprehensive training data, as per conventional
approaches. In light of the constraints imposed by limited computational
resources and the specificity of our legal tasks-related dataset, it becomes
imperative to adopt methods prioritizing efficiency in both computational
costs and data requirements. This goal is achieved by utilizing parameter-
efficient tuning methods [26] [19], which better leverage available data and
minimize the necessity for extensive resource allocation. Specifically, the
Low-Rank Adaption method [19] was employed to fine-tune the Mistral 7B [18]
model for developing Aalap.

###  4.1 Training Procedure

The fine-tuning process was conducted by loading the Mistral 7B model in
Bfloat-16 precision. The training infrastructure utilized 4 X A100 80GB GPUs,
affording a compute size of 320GB GPU memory resources. Considering limited
resources, parameters that fully leveraged the available compute were
selected, resulting in generating a commendable model within a reasonable time
frame of 86 hours, which cost us $950.

Throughout the training phase, the maximum token length was designated as
16384, aligning with the maximum length of data present in the training
corpus. The LoRA rank was initialized to 8, and alpha was set to 16, with a
dropout rate of 0.1. LoRA was applied to all model layers to achieve
performance similar to that of a fully fine-tuned model. Integration of flash-
attention 2 [27] during the training process significantly enhanced training
speed. A Paged Adam 32-bit optimizer with a learning rate of 5e-5 was
employed, along with a ”constant” learning rate scheduler. Weight decay was
set to 0.01, and the warm-up ratio was established at 0.03. Deepseed stage 3
was employed in the training process, with per_device_batch_size to 4 and
gradient accumulation set to 2, resulting in a global batch size of 32 across
4 GPUs. The model was trained for a total of 6500 steps which come down to a
total of 10 epochs.

Our runs were tracked through Weights & Biases [28]. The progression of model
training and evaluation losses during the fine-tuning process is depicted in
Figure 2. It is observed that there is a steep decline in the initial phase of
training, followed by a slow but constant decrease in losses as the training
proceeds. This steep decline suggests that, despite differences between the
task and the pretraining of the model, the model demonstrates the ability to
adapt quickly. As the training progresses, the model appears to focus on
learning the subtle nuances of the data.

Throughout the training process, checkpoints of the model are consistently
extracted for human evaluation. As the training continues, model showcases a
noticeable improvement in its capability of generating responses that are not
only coherent but also contextually correct for the specific legal tasks under
consideration. A detailed discussion of the evaluation of Aalap is provided in
the following section.

##  5 Model Evaluations

The aalap model was evaluated using three methodologies.

  * •

Using GPT4 as an evaluator on Aalap test data

  * •

Using Legalbench data

  * •

Using All India Bar Exam (AIBE) data

###  5.1 Evaluation using GPT4 as an evaluator on Aalap test data

The performance of Aalap was compared with Mistral 7B & gpt-3.5-turbo on Aalap
test data. The evaluation approach proposed by [29] was followed. The
‘gpt-4-1106-preview’ model was used for the evaluations to give a score to
each response ranging from 0 to 10 by comparing it with the reference answer
provided. The evaluation was done mainly for correctness and helpfulness.

Figure 1: Boxplot of Aalap evaluation scores by GPT4 grouped by tasks

The table 2 gives the pairwise comparison of evaluation scores.

Model | Aalap is worse | Tie | Aalap is Better
---|---|---|---
Mistal 7B | 37 | 132 | 925
gpt-3.5-turbo | 385 | 373 | 336
|  |  |
Table 2:  Comparison of Aalap vs. Mistral7B vs. gpt3.5turbo

We can see that Aalap is doing better than Mistral 7B in 85% of test data. The
performance of Aalap is better than gpt-3.5-turbo in 31% of our test data and
obtains an equivalent score in 34% of the test data.

Figure 1 shows the evaluation scores of Aalap calculated by GPT4 for various
task categories on the test data. For verbose tasks like issues generation,
argument generation, and contract clause generation, Aalap is doing a good
job. We can see that Aalap is not doing well on tasks like summary generation,
event timeline generation, and constitution general knowledge.

###  5.2 Evaluation using Legalbench data

Legalbench test data [2] evaluates the model on 162 legal tasks, which can be
categorized into Issue, Rule, Conclusion, Interpretation, and Rhetoric. Since
this test data is huge, we took a random sample of 100 examples from each of
the 162 tasks. The questions are mostly multiple-choice questions where the
model’s job is to look at a few shot examples and select the right option for
the question asked. Since Aalap is trained to produce verbose output and
explanations, we created longer answers for these questions and asked
gpt-3.5-turbo to evaluate against ground truth. This was necessary because,
many times, simple regular expressions could not evaluate the outputs
correctly. For evaluating Mistral and gpt-3.5-turbo, we created the outputs
with limited tokens and used regular expressions to evaluate.

Task Category | gpt-3.5-turbo | Mistral 7B | Aalap
---|---|---|---
Issue | 51.2 | 51.8 | 45.3
Rule | 34.0 | 24.0 | 32.0
Interpretation | 61.9 | 57.0 | 53.6
Rhetoric | 62.0 | 42.0 | 33.0
Conclusion | 78.0 | 42.6 | 61.7
Overall | 61.0 | 53.2 | 51.2
|  |  |
Table 3:  The average performance of various LLMs on the sample Legalbench
data

As we can see from Table 3, Aalap is doing worse than the Mistral 7B model.
This is primarily due to the loss of few-shot abilities of LLMs after
finetuning. This behavior is demonstrated by other studies as well [30]. Even
though the Legalbench training data is part of the training data for Aalap,
the format is in the form of an explanation of an individual example
explanations.

###  5.3 Evaluation using All India Bar Exam

All India Bar Exam (AIBE) questions and answers for the past 12 years were
compiled 444https://huggingface.co/datasets/opennyaiorg/aibe_dataset ranging
from AIBE 4 to AIBE 16. The scanned question papers with answer keys were
converted to text using optical character recognition. This data was manually
verified for correctness. There are 1158 multiple-choice questions from
various areas of law. We compared the performance of Aalap with Mistral 7B &
gpt-3.5-turbo on this data. Most AIBE questions are recall-based, and a few
test legal reasoning. The minimum passing percentage for this exam is 40%.

gpt-3.5-turbo | Mistral 7B | Aalap
---|---|---
58.72% | 23.48% | 25.56%
|  |
Table 4:  Average AIBE scores for various LLMs

Aalap is doing slightly better than Mistral but still falls short of the
passing percentage.

###  5.4 Discussion

For the tasks that are present in the training data, we can say that Aalap is
performing comparably to ‘gpt-3.5-turbo’. But for the AIBE exam and Legalbench
data, Aalap is not doing any better than the Mistral 7B base model. The main
reasons behind poor performance on AIBE data is recall-oriented nature of the
questions. The main reasons for poor performance on Legalbench data are loss
of few-shot abilities after fine-tuning and the dataset is in the context of
US legal systems. The performance of these benchmarks could be improved by
adding such related datasets in the training process.

##  6 Conclusion & Next Steps

We created Aalap data and model, which is a fine-tuned LLM for specific Indian
legal tasks. Aalap model can do specific legal tasks well and is not a general
purpose legal LLM. Given the size of the model, this can be hosted on-premise
which can help with handling of sensitive data. Hence we believe that Aalap
can help to automate some mundane tasks for legal practitioners and help to
improve their efficiency.

The training data quality can be improved significantly by adding expert
reviews. The entire data generation process can be more aligned with the
practical workflows of lawyers by seeking their feedback. Seeking feedback
about Aalap’s performance in real-world scenarios will help to improve this
model further. It will be useful to make this dataset & model multi-lingual.

## Acknowledgements

This work is part of the OpenNyAI mission, which is funded by EkStep and
Agami. We thank all the lawyers, including Professor Rangin Tripathi from
National Law University Odisha, for contributing to the task selection. The
authors thank Sankalp Bhoyar for helping build the front end for Aalap and
Kushagra Bhatia for verifying and correcting AIBE dataset. We thank E2E
Networks for providing GPUs for this work.

##  Appendix A Summary statistics of various task categories

Summary statistics of train and test Aalap data is show in Table 5. Average
input and output tokens are calculated using tiktoken library.

Task Category |  train count |  test count |  Average input tokens |  Average output tokens |  License
---|---|---|---|---|---
Issue Generation |  577 |  24 |  1376 |  161 |  CC0-1.0
Argument Generation |  1142 |  58 |  2381 |  943 |  CC0-1.0
Event Timeline |  676 |  49 |  3476 |  342 |  CC0-1.0
Combine Event Timeline |  195 |  9 |  883 |  772 |  CC0-1.0
legalbench |  580 |  27 |  229 |  218 |  Other
Statute Ingredients |  499 |  27 |  402 |  111 |  CC0-1.0
Summary Generation |  686 |  14 |  7635 |  1413 |  CC0-1.0
Legal Open ORCA |  8142 |  413 |  449 |  91 |  MIT
Contract Clause Generation |  4325 |  232 |  76 |  179 |  cc-by-nc-4.0
Legal NIv2 MCQ |  1891 |  109 |  408 |  9 |  Apache 2.0
Constitution General Knowledge |  889 |  44 |  36 |  85 |  Apache 2.0
Incomplete Instructions |  1464 |  82 |  97 |  81 |  CC0-1.0
General Alap |  112 |  6 |  31 |  25 |  CC0-1.0
Table 5:  Summary statistics of various task categories

##  Appendix B Aalap Training & Eval Loss

Training and evaluation loss are shown in Figure 2.

Figure 2: Comparison of Training and Evaluation Loss During Fine-tuning: The
graph illustrates the progression of training and evaluation loss over the
course of fine-tuning, providing insights into the model’s learning dynamics
and generalization performance.

## References

  *   * Katz et al. [2023] Katz, D.M., Bommarito, M.J., Gao, S., Arredondo, P.: GPT-4 passes the bar exam (2023). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233
  * Guha et al. [2023] Guha, N., Nyarko, J., Ho, D.E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D.N., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G.M., Porat, H., Hegland, J., Wu, J., Nudell, J., Niklaus, J., Nay, J., Choi, J.H., Tobia, K., Hagan, M., Ma, M., Livermore, M., Rasumov-Rahe, N., Holzenberger, N., Kolt, N., Henderson, P., Rehaag, S., Goel, S., Gao, S., Williams, S., Gandhi, S., Zur, T., Iyer, V., Li, Z.: LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models (2023)
  * Choi et al. [2023] Choi, J.H., Hickman, K.E., Monahan, A., Schwarcz, D.: Chatgpt goes to law school (2023). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4335905
  * Chalkidis [2023] Chalkidis, I.: ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark (2023)
  * Ling et al. [2023] Ling, C., Zhao, X., Lu, J., Deng, C., Zheng, C., Wang, J., Chowdhury, T., Li, Y., Cui, H., Zhang, X., Zhao, T., Panalkar, A., Cheng, W., Wang, H., Liu, Y., Chen, Z., Chen, H., White, C., Gu, Q., Pei, J., Zhao, L.: Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey (2023)
  * Wu et al. [2023] Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., Mann, G.: BloombergGPT: A Large Language Model for Finance (2023)
  * Singhal et al. [2023] Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., Schaekermann, M., Wang, A., Amin, M., Lachgar, S., Mansfield, P., Prakash, S., Green, B., Dominowska, E., Arcas, B.A., Tomasev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S.S., Barral, J., Webster, D., Corrado, G.S., Matias, Y., Azizi, S., Karthikesalingam, A., Natarajan, V.: Towards Expert-Level Medical Question Answering with Large Language Models (2023)
  * Hoffmann et al. [2022] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Las Casas, D., Hendricks, L.A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J.W., Vinyals, O., Sifre, L.: Training Compute-Optimal Large Language Models (2022)
  * Cheng et al. [2023] Cheng, D., Huang, S., Wei, F.: Adapting Large Language Models via Reading Comprehension (2023)
  * Lewis et al. [2021] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2021)
  * Cui et al. [2023] Cui, J., Li, Z., Yan, Y., Chen, B., Yuan, L.: ChatLaw: Open-Source Legal Large Language Model with Integrated External Knowledge Bases (2023)
  * Huang et al. [2023] Huang, Q., Tao, M., Zhang, C., An, Z., Jiang, C., Chen, Z., Wu, Z., Feng, Y.: Lawyer LLaMA Technical Report (2023)
  * Yu et al. [2023] Yu, F., Quartey, L., Schilder, F.: Exploring the effectiveness of prompt engineering for legal reasoning tasks. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 13582–13596 (2023)
  * Trautmann et al. [2022] Trautmann, D., Petrova, A., Schilder, F.: Legal prompt engineering for multilingual legal judgement prediction. arXiv preprint arXiv:2212.02199 (2022)
  * Fei et al. [2023] Fei, Z., Shen, X., Zhu, D., Zhou, F., Han, Z., Zhang, S., Chen, K., Shen, Z., Ge, J.: Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289 (2023)
  * Ainslie et al. [2023] Ainslie, J., Lee-Thorp, J., Jong, M., Zemlyanskiy, Y., Lebrón, F., Sanghai, S.: Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)
  * Child et al. [2019] Child, R., Gray, S., Radford, A., Sutskever, I.: Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019)
  * Jiang et al. [2023] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.-A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7B (2023)
  * Hu et al. [2021] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * Dettmers et al. [2023] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QLoRA: Efficient Finetuning of Quantized LLMs (2023)
  * Mukherjee et al. [2023] Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., Awadallah, A.: Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (2023)
  * [22] Kalamkar, P., Tiwari, A., Agarwal, A., Karn, S., Gupta, S., Raghavan, V., Modi, A.: Corpus for automatic structuring of legal documents
  * Kalamkar et al. [2022] Kalamkar, P., Agarwal, A., Tiwari, A., Gupta, S., Karn, S., Raghavan, V.: Named entity recognition in indian court judgments. In: Proceedings of the Natural Legal Language Processing Workshop 2022, pp. 184–193 (2022)
  * Lian et al. [2023] Lian, W., Goodson, B., Pentland, E., Cook, A., Vong, C., ”Teknium”: OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces. HuggingFace (2023)
  * Mishra et al. [2022] Mishra, S., Khashabi, D., Baral, C., Hajishirzi, H.: Cross-task generalization via natural language crowdsourcing instructions. In: ACL (2022)
  * Li and Liang [2021] Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation (2021)
  * Dao [2023] Dao, T.: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023)
  * Biewald [2020] Biewald, L.: Experiment Tracking with Weights and Biases. Software available from wandb.com (2020). https://www.wandb.com/
  * Zheng et al. [2023] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E., Stoica, I.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
  * Wang et al. [2023] Wang, Y., Si, S., Li, D., Lukasik, M., Yu, F., Hsieh, C.-J., Dhillon, I.S., Kumar, S.: Two-stage LLM Fine-tuning with Less Specialization and More Generalization (2023)