# SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain

###### Abstract

In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language
models (LLMs) tailored for the legal sector. These models, which feature
architectures of 54 billion and 141 billion parameters, respectively, are
based on the Mixtral architecture. The development of SaulLM-54B and
SaulLM-141B is guided by large-scale domain adaptation, divided into three
strategies: (1) the exploitation of continued pretraining involving a base
corpus that includes over $540$ billion of legal tokens, (2) the
implementation of a specialized legal instruction-following protocol, and (3)
the alignment of model outputs with human preferences in legal
interpretations. The integration of synthetically generated data in the second
and third steps enhances the models’ capabilities in interpreting and
processing legal texts, effectively reaching state-of-the-art performance and
outperforming previous open-source models on LegalBench-Instruct. This work
explores the trade-offs involved in domain-specific adaptation at this scale,
offering insights that may inform future studies on domain adaptation using
strong decoder models. Building upon SaulLM-7B, this study refines the
approach to produce an LLM better equipped for legal tasks. We are releasing
base, instruct, and aligned versions on top of SaulLM-54B and SaulLM-141B
under the MIT License to facilitate reuse and collaborative research.

##  1 Introduction

LLMs have demonstrated exceptional capabilities across various domains
achiam2023gpt; scao2022bloom; penedo2023refinedweb; touvron2023llama;
jiang2023mistral; jiang2024mixtral; touvron2023llama2; bai2023qwen;
faysse2024croissantllm, excelling in tasks such as language translation
alves2024tower, medical diagnostics chen2023meditron; bolton2024biomedlm;
bolton2024assessing, and automated code generation allal2023santacoder;
li2023starcoder; guo2024deepseek, among others. These achievements highlight
the potential for human-like communication through large language models
(LLMs). Despite the significant potential benefits, the adaptation of most
recent LLMs for legal tasks has not been extensively examined, with only two
recent studies cited from (colombo2024saullm7b; niklaus2024flawn;
yao2021adapt), and its impact on society could be substantial. Indeed, at a
time when legal systems in many countries are overburdened sourdin2020court,
the development of robust and high-performing legal LLMs could provide
critical support to lawyers and judicial systems re2019developing;
bhambhoria2024evaluating. However, adapting LLMs to the legal domain presents
unique challenges, particularly because of the vast scale involved, with
hundreds of billions of existing legal data available.

Previous efforts to tailor LLMs to the legal sector have encountered
significant challenges (colombo2024saullm7b; niklaus2024flawn; yao2021adapt):
first, a limited model scale, capped at $7/12$B parameters, which is
considerably smaller than the largest open-source models bai2023qwen;
jiang2024mixtral; second, training datasets restricted to no more than $30$
billion tokens, significantly fewer than potentially available tokens
gao2020pile; niklaus2023multilegalpile. Given the importance of scale and
breadth in effectively adapting LLMs to new domains, this paper aims to answer
the following research question:

How much can we improve the specialization of generic LLMs for legal tasks by
scaling up both model and corpus size?

In this paper, we present an empirical study on the scalability and domain
adaptation of LLMs in the legal sector. Relying on a corpus exceeding $500$B
tokens and models up to $141$B parameters, our research seeks to address the
gaps in the examination of legal applications. A novel aspect of our approach
is the adaptation of large-scale Mixture of Experts (MoE) models with $54$B
and $141$B parameters, which have gained significant traction in recent months
zhou2022mixture; dai2024deepseekmoe; lin2024moe; zhang2024efficient;
yu2024boosting; pioro2024moe. Formally, this study makes two principal
contributions:
1\. Comprehensive Analysis of Domain Adaptation Strategies for Legal LLMs
Domain adaptation for legal LLMs remains a challenging and somewhat
underexplored area. This work advances the field by specializing each step in
the process of developing modern LLMs, from continued pretraining to
instruction fine-tuning and alignment, relying on both synthetic and real
data. This paper offers a fresh perspective on the efficacy of each step and
its value for adapting to the legal domain, potentially guiding further
research in the legal domain as well as in other expert domains.
2\. SaulLM-54B & SaulLM-141B: Joining SaulLM-7b to form a Family of Legal LLMs
under Permissive License111Model will be made available at
https://huggingface.co/. We specialize general-purpose, large-scale LLMs for
the law. This work represents an ambitious advancement in terms of scale and
leveraging the increasingly popular MoE architecture. While this architecture
is widely used, its specific applications within focused domains, particularly
the legal sector, are still largely unexplored. By releasing these models, we
aim to foster further research in legal NLP and contribute to unlocking the
full potential of LLMs.

##  2 Related Work

###  2.1 Domain Specialization For Large Language Models

The process of domain specialization for LLMs has demonstrated promising
results in areas such as medicine chen2023meditron, science
taylor2022galactica, translation alves2024tower; alves2023steering, or code
roziere2023code; li2023starcoder; allal2023santacoder. Models like SciBERT
beltagy2019scibert, PubMedBERT tinn2023fine, Galactica taylor2022galactica and
Meditron chen2023meditron have been specifically trained on domain-related
corpora to enhance their performance. Studies have identified that both the
scale of the model and the size of the in-domain data are crucial for
achieving strong domain adaptation chen2023meditron; roziere2023code.

In the legal domain, earlier models such as LegalBERT chalkidis2020legal,
InCaseLawBERT paul-2022-pretraining, and SaulLM-7B colombo2024saullm7b, among
others, while pioneering, have been constrained by their relatively small
scale and the specificity of their training data, which covers a limited
number of documents and jurisdictions. Our work aims to build on these efforts
by deploying LLMs at an unprecedented scale, utilizing models of up to 141B
parameters and a base corpus exceeding 500 billion tokens to significantly
enhance the depth and breadth of legal language comprehension and generation.

###  2.2 Legal Domain Adaptation for Modern LLM

The field of legal domain adaptation has traditionally concentrated on
refining models through pretraining on specialized corpora chalkidis2020legal;
colombo2024saullm7b; cui2023chatlaw. Yet, in the current paradigm, pretraining
represents just one aspect of the solution, as LLMs often utilize techniques
like instruction fine-tuning and alignment, employing algorithms such as DPO
rafailov2024direct, PPO schulman2017proximal or RLHF ouyang2022training;
li2023reinforcement.

Recent domain-adapted models, such as SaulLM or Legal-FLAN-T5 (a closed
model), have tried to improve alignment with legal instructions. However,
SaulLM is a smaller model, and Legal-FLAN-T5, is based on an outdated
architecture and does not leverage the extreme scale pretraining that
contemporary models do. Moreover, it not being publicly available stymies
progress vital for advancing research and applications in the legal sector.

We believe this work pioneers a holistic approach to domain adaptation by
training modern LLMs specifically for the legal domain, from pretraining to
instruction fine-tuning and legal preference alignment. We demonstrate that
synthetic data can be effectively utilized for alignment, advancing beyond
SaulLM-7B’s use solely of instruction fine-tuning. The resulting models,
SaulLM-54B and SaulLM-141B, lay the groundwork for further research and
development, and expand access to high-performance legal LLMs.

##  3 Data Collection and Corpus Construction

This section outlines our approach to assembling and refining a comprehensive
legal text corpus tailored for training large language models in the legal
domain.

###  3.1 Pretraining Corpora

The diversity of legal systems worldwide, from common law to civil law
traditions, presents unique challenges and opportunities
niklaus2023multilegalpile; henderson2022pile. To address this, we compiled an
extensive English-language corpus from various jurisdictions including the
U.S., Europe, Australia, and others aletras2016predicting;
gutierrez2021spanish, which comprises $500$ billion tokens before cleaning and
deduplication.

####  3.1.1 Legal Sources

Our base corpus combines various legal datasets (niklaus2022budgetlongformer)
with newly sourced public domain documents. It includes significant
collections such as the FreeLaw subset and the MultiLegal Pile, augmented with
extensive web-scraped content. subsubsection 3.1.2 summarizes the composition
and scale of our dataset.

####  3.1.2 Other Sources

Replay Sources. To mitigate the risk of catastrophic forgetting during model
training, we reintroduced data from earlier training distributions
(MCCLOSKEY1989109; chen2023meditron; sun2020distill). This replay strategy
incorporates general data sources such as Wikipedia, StackExchange, and
GitHub, and makes up approximately $2\%$ of the total training mix. These
datasets are sampled from SlimPajama shen2023slimpajama;
together2023redpajama; soboleva2023slimpajama.

Additionally, we included $5\%$ of math datasets in the pretraining mix using
commercially available math sources. We found this approach usefull for
retaining the reasoning performance of the final model and avoiding the weaker
performance observed in previous research attempts like SaulLM-7B222These
findings also align with the high percentage of math and STEM in the
pretraining mix from ormazabal2024reka..

Table 1: Sources of Legal Pretraining Data Source Name | Tokens (B)
---|---
FreeLaw Subset from The Pile | 15
EDGAR Database | 5
English MultiLegal Pile | 50
English EuroParl | 6
GovInfo Statutes, Opinions & Codes | 11
Law Stack Exchange | 0.019
Comm Open Australian Legal Corpus | 0.5
EU Legislation | 0.315
UK Legislation | 0.190
Court Transcripts | 0.350
UPSTO Database | 4.7
Web Data (legal) | 400
Other | 30
Total | 520

Instruction Sources. We found that incorporating conversational data during
pretraining is advantageous, drawing inspiration from recent breakthroughs in
neural machine translation alves2024tower. Studies suggest that the enhanced
translation capabilities of LLMs can be attributed to the presence of
accidental parallel data within their training corpora. Accordingly, we have
integrated the Super Natural Instruction wang2022super and FLAN collection
longpre2023flan into our pretraining mix, enriching the dataset with diverse
instructional content.

Data for Model Annealing. Model annealing is primarily achieved through a
methodical reduction of the learning rate parmar2024nemotron; hu2024minicpm,
known as learning rate annealing.

In our experiments, model annealing with high-quality, domain-relevant data
significantly enhanced performance. Conversely, repetitive synthetic data from
initial instruction fine-tuning harmed performance. Therefore, we used the
commercial portion of the LawInstruct dataset for model annealing, which
proved more effective than for instruction finetuning. We also included
UltraChat ding2023enhancing as generic instructions during the annealing
phase.

####  3.1.3 Data Preprocessing

Our data processing pipeline closely follows colombo2024saullm7b. In
particular, we do:
1\. Text extraction: a significant fraction of the collected data is in PDF
format. We used Poppler to extract the text.
2\. Data cleaning: extraction from PDF files creates some artifacts like page
and line numbers in the middle of sentences, as well as broken lines of text,
non-normalized Unicode characters, etc.

  * •

Text normalization. We normalize all text using the NFKC method, available
through the unicodedata Python package.

  * •

Rule-based filters. We created regex rules for filtering commonly undesirable
but commonly recurring patterns, like page and line numbers in the middle of
the text, HTML tags, etc. Following colombo2024saullm7b, we found that some of
the most common 10-grams in our dataset were repeated characters and
whitespace and removed them.

  * •

Perplexity filtering. Similarly to colombo2024saullm7b we used a KenLM
(heafield-2011-kenlm) model trained on a small subset of carefully cleaned
legal data to filter documents with high perplexity. Concretely, we filtered
any document whose normalized perplexity was larger than $1500$.

3\. Text deduplication: we used chenghao_mou_2023_8364980 to remove duplicates
and near-duplicate examples from the training set. We used default parameters
except for the similarity threshold, which we set to $0.5$.

Finally, we packed the individual documents together to build $8192$ tokens-
long training examples. Documents longer than this value were chunked into
several examples.

###  3.2 Instruction Data

Instruction fine-tuning is essential for making an LLM follow instructions and
optimize the performance of pretrained models across a variety of tasks
wang2023far; wei2021finetuned; chung2022scaling; Faysse_2023;
ding2023enhancing; wang2023selfinstruct. To this end, we employ a strategic
mix of general and domain-specific (legal) instructions, aimed at enhancing
the model’s ability to precisely interpret and execute commands, with a
particular focus on legal scenarios.
General Instructions Our methodology for sourcing general instructions
involves the integration of a diverse array of datasets, each selected to
augment different aspects of the model’s capabilities across various domains
cao2023instruction; zhou2023lima:
1\. General Instruction from UltraInteract yuan2024advancing: UltraInteract is
an extensive, high-quality dataset designed to foster complex reasoning,
featuring structured instructions that include preference trees, reasoning
chains, and multi-turn interaction trajectories.
2\. General Instruction from Dolphin
333https://huggingface.co/datasets/cognitivecomputations/dolphin: This dataset
provides additional conversational data, further broadening the model’s
exposure to diverse communication styles and contexts.

Each dataset is subjected to rigorous filtering, deduplication, and curation
processes, culminating in a refined compilation of approximately $1,000,000$
instructions meticulously prepared for the instruction fine-tuning phase.
Legal Instruction Construction For legal instructions, we synthesize dialogues
and question/answer pairs that capture key legal concepts and document types
to emulate legal analysis. In accordance with the model scale, we used
Mistral-54B-Instruct for SaulLM-54B and Mistral-141B-Instruct for SaulLM-141B.
The generation follows the recipe from colombo2024saullm7b and begins with a
three-turn sequence: (1) a user inquires about a legal document, (2) the
assistant reformulates this inquiry by integrating metadata such as document
type or issue date, and (3) the user asks for further explanation of the
assistant’s reasoning. The dialogue progressively deepens, with the assistant
methodically unpacking the legal reasoning in response to increasingly nuanced
questions from the user.

Figure 1: Domain adaptation method model for turning a Mixtral to a
SaulLM-141B. Training involves different stages: legal domain pretraining,
instruction filtering, and preference filtering.

###  3.3 Preference Data

We enhance our models’ adaptability and precision by incorporating preference
data from both general and legal-specific sources tunstall2023zephyr;
rafailov2023direct; munos2023nash; vonwerra2022trl. _General datasets_ are
UltraFeedback cui2023ultrafeedback and Orca. For the _legal domain_ , we
employ synthetic scenarios crafted to simulate complex legal reasoning and
generate accepted/rejected responses. The Mixtral-142B-Instruct model
evaluates these responses based on factual accuracy, relevance, and logical
coherence, selecting the most appropriate responses as preferred outcomes
(similar to zheng2024judging).

##  4 Implementation Details & Evaluation Protocol

###  4.1 Model Selection

We used Mixtral models jiang2024mixtral, which are built on a Transformer
architecture vaswani2017attention enhanced with a Mixture of Experts to
improve computational efficiency and adaptability for handling extensive
contexts. The Mixtral-54B and Mixtral-141B architecture respectively consists
of $32$ (resp. $56$) layers, a model dimension of $4096$ (resp. $6144$), and a
hidden dimension of $14,336$ (resp. $16384$). Although it supports a context
length of up to $32,768$ (resp. $65536$) tokens, we continue pretraining on
$8,192$ tokens. Extending the context length is beyond the scope of this
paper. The MoE layers in Mixtral rely on 8 experts with 2 active experts
selectively based on the input, efficiently utilizing computational resources
and providing significant model capacity. Interestingly, Mixtral is the only
model available in dual configurations (Mixtral-54B and Mixtral-141B),
allowing us to evaluate the scalability of our domain adaptation approaches.

At the time of the training, Mixtral was the most powerful decoder in its
class, surpassing all competitors including Llama touvron2023llama;
touvron2023llama2; zhang2024tinyllama, Yi, Qwen bai2023qwen, and CroissantLLM
faysse2024croissantllm in terms of both cost-effectiveness and performance.

###  4.2 Engineering Details

Codebase Configuration Our training framework uses PyTorch. The integration of
DeepSpeed rasley2020deepspeed (level 3) and Flash attention
dao2022flashattention mechanisms enhances our training efficiency and
scalability. We make our models available through the Huggingface hub
wolf2019huggingface.
Compute Infrastructure The computational backbone for the continuous
pretraining phase of our project consists of $384$ AMD MI250 GPUs. We can
reach $40\%$ GPU utilization with our implementation. For instruction fine-
tuning and preference optimization, we rely on 64 AMD MI250 GPUs. Evaluation
protocols are executed on a single node of AMD MI250 GPU.
Synthetic Data Generation For synthetic data generation, we used vLLM on a
node of NVIDIA-A100, primarily due to limited support of libraries on
MI250444VLLM is not supported on AMD-MI250 and HuggingFace’s text-generation-
inference had a few bugs that prevented its use..

###  4.3 Training Details

The model training process is divided into three distinct phases: continued
pretraining, instruction finetuning (IFT), and preference alignment using
domain-specific optimization (DPO). A full schema of the pipeline can be found
in Figure 1.
Continued Pretraining For continued pretraining, we use the AdamW
kingma2014adam; loshchilov2017decoupled; barakat2021convergence optimizer with
hyperparameters $\beta_{1}=0.99$, $\beta_{2}=0.90$, and a learning rate of
$2\times 10^{-5}$. We utilize a cross-entropy loss function to optimize model
predictions. The training protocol sets gradient accumulation to $4$, with
tailored batch sizes of $8$ for SaulLM-54B and $4$ for SaulLM-141B, optimizing
both GPU utilization and training efficiency.
Instruction Fine-Tuning (IFT) IFT uses the AdamW optimizer (learning rate of
$1\times 10^{-5}$), reinitialized to reset training states and maintain
training stability. We limit this phase to a single training epoch, as our
experiments suggest this maximizes performance gains.

Preference Training Using DPO We adjust the learning rate of the AdamW
optimizer to $1\times 10^{-6}$ during DPO. Our choice of DPO over IPO
calandriello2024human, KTO ethayarajh2024kto or ORPO hong2024reference was
based on preliminary experiments.

###  4.4 Evaluation Protocol

LegalBench-Instruct We rely on LegalBench-Instruct colombo2024saullm7b, which
refined the prompts from LegalBench guha2022legalbench by eliminating
distracting elements and specifying a response format to enhance precision.
Like LegalBench, it evaluates LLMs across six types of legal reasoning: issue-
spotting, rule-recall, rule-application, rule-conclusion, interpretation, and
rhetorical understanding. Grounded in American legal frameworks but applicable
globally, these categories provide a comprehensive evaluation of the models’
legal reasoning capabilities. This structured approach helps in accurately
assessing and guiding the enhancement of LLMs within and beyond American legal
contexts. We follow previous work and rely on balanced accuracy as the primary
metric across all tasks.

Massive Multitask Language Understanding (MMLU) Previous works utilize MMLU
hendrycks2020measuring, a widely-recognized benchmark, focusing on its legal-
specific tasks in international law, professional law, and jurisprudence, with
$120$, $1500$, and $110$ examples respectively. These tasks are crucial for
assessing our models’ understanding and application of complex legal concepts,
highlighting their proficiency in nuanced legal environments.

Choice of Baseline & Model Naming For our evaluation, we aim for a direct,
apples-to-apples comparison of models. It is important to note that not all
competing models are open source, and detailed information on their alignment
procedures and instruction fine-tuning processes is not available. This lack
of transparency complicates the establishment of fully equitable baseline
comparisons. In what follows, we use OpenAI’s GPT-4 as of 10 May 2024, Meta’s
Llama3 (the Instruct variant) and the Instruct variants of Mixtral-54B, and
Mixtral-141B. Additionally, SaulLM-54B-IFT is the IFT version built on
SaulLM-54B-base and SaulLM-medium for the DPO version based on SaulLM-54B-IFT.
SaulLM-large is the final version DPO and IFT based on SaulLM-141B.

##  5 Experimental Results

###  5.1 Global Results: LegalBench-Instruct & Legal-MMLU

Figure 2 presents the results of SaulLM-large and SaulLM-medium on LegalBench-
Instruct, from which we make several observations:

Figure 2: Overall Results. Comparison of SaulLM-large and SaulLM-medium with
existing models.

Our domain adaptation strategy is achieving strong results. SaulLM-medium
outperforms Mixtral-54B, and similar findings are observed with SaulLM-large
compared to Mixtral-141B. Interestingly, domain adaptation at both the
instruction tuning stage and preference alignment enables our smaller models
to outperform larger ones, such as GPT-4 and LLama3-70B. These results
validate our approach and demonstrate that specializing the entire pipeline
(i.e., from continued pretraining to preference alignment) is a promising
direction for improving performance in legal-related tasks.

Adaptation results on MMLU Results on the legal subset of MMLU align with our
observations on LegalBench-Instruct. Figure 5, Figure 15 and Figure 5 show
that continued

Figure 3: Jurisprudence

Figure 4: Professional Law

Figure 5: International Law

pretraining significantly enhances model performance across all legal tasks in
MMLU. SaulLM-large consistently performs well, achieving the highest scores in
both jurisprudence and professional law, while SaulLM-medium excels in
international law. The models show substantial improvement over the Mixtral
across all tasks. These findings highlight the importance of continued
pretraining in improving the understanding and application of complex legal
concepts, with larger models demonstrating better gains.

As for LegalBench-Instruct, inverse scaling effect can also be observed (e.g,
on international law - Figure 5), where the largest models (SaulLM-large and
Mixtral-141B) do not outperform the smaller SaulLM-medium and Mixtral-54B
models. This result suggests that while continued pretraining improves
performance, optimal model size may vary depending on the task.

Domain adaptation works across scales and MoE models. The results from SaulLM-
medium and SaulLM-large confirm previous findings from colombo2024saullm7b and
confirms that domain adaptation is effective across different scales,
including on MoE models. Interestingly, most of the data collected for this
work comes from public sources, which were likely seen during the pretraining
of the base models.

A Path Towards Stronger Models. The results of LLama3-70B and the scalability
of our methods suggest that applying the same approach to the LLama3-70B base
model could lead to even better performance than our best model, SaulLM-141B.
It is worth noting that SaulLM-141B has only 44B active parameters making it
appealing for efficient serving.

###  5.2 How much does continued pretraining help for the legal domain?

Figure 6: Global Analysis. Role of continued pretraining.

Figure 7: Category Analysis: Role of continue pretraining.

Previous works on domain adaptation via continued pretraining primarily
focused on instruction finetuning chen2023meditron; colombo2024saullm7b;
niklaus2024flawn. In Figure 7 and Figure 7, we report the performance of
Mixtral-54B trained with the IFT mix described in subsection 3.2
(Mixtral-54-IFT) and subsequently aligned using the DPO dataset
(Mixtral-54-IFT+DPO), as described in subsection 3.3. We also compare these
results to the instruct version of Mixtral (Mixtral-54B), as outlined in
jiang2024mixtral.

Table 2: Quantifying the role of DPO. We report the percentage of tasks where the difference in performance ($\Delta$) between SaulLM-54B and SaulLM-54B-IFT is positive (resp. negative). Category | $\Delta\geq 0$ | $\Delta\leq 0$
---|---|---
Conclusion | 37.5% | 62.5%
Interpretation | 65.1% | 34.9%
Rhetoric | 44.4% | 55.6%
Rules | 60.0% | 40.0%
Issue | 100.0% | -

Table 3: Quantifying the role of scaling. We report the percentage of tasks where the difference in performance ($\Delta$) between SaulLM-medium and SaulLM-large is positive (resp. negative). Category | $\Delta\geq 0$ | $\Delta\leq 0$
---|---|---
Conclusion | 18.2% | 81.8%
Interpretation | 23.7% | 76.3%
Rules | 25.0% | 75.0%
Issue | - | 100.0%
Rhetoric | - | 100.0%

Continuing pretraining significantly enhances model performance in the legal
domain, benefiting both the IFT and DPO stages. From Figure 7, we observe that
both IFT and DPO benefit from a notable improvement (approximately $+7\%$).
Interestingly, this improvement is consistent across all five categories, as
shown in Figure 7.

Adding legal data to the IFT and DPO datasets improves the model’s legal
capabilities. By comparing the performance of Mixtral-54-IFT+DPO and
Mixtral-54, we observe that the mix used for IFT and DPO enhanced with legal
data leads to stronger legal performance than that of Mixtral-54, which does
not publicly describe the alignment methods used. This result aligns with
findings reported in niklaus2024flawn; colombo2024saullm7b.

###  5.3 How Much Does Legal Preference Alignment Help?

Our findings from Figure 7 indicate that alignment significantly improves the
results. In particular, DPO improvement is mostly consistent across tasks and
categories. As shown in Table 3, the alignment version SaulLM-medium
demonstrates significant improvements over the IFT version across most tasks,
including conclusion, rhetoric, rules, and issue tasks. We observe, however, a
drop in performance in some interpretation tasks. Upon closer examination, we
found that this decline is often due to the model becoming more verbose, which
causes the evaluation process to fail in correctly parsing the answers555To
further explore this phenomenon, we present the results in Figure 16 at the
task level between the alignment version SaulLM-medium and the instruction
fine-tuning version SaulLM-54B-IFT., i.e., this issue primarily arises from a
benchmark limitation. Addressing model verbosity and the challenge of more
reliable benchmarks is beyond the scope of this work, but it is a well-known
problem identified in many concurrent studies Faysse_2023. Enhancing the
evaluation process is one of the key improvements we plan to contribute to in
the future.

Figure 8: Continue Pretraining Analysis. Training loss for SaulLM-141B-base
and SaulLM-54B-base over normalized epochs.

Figure 9: Energy Consumption Analysis. Mean Power per Node for training jobs
on the XXXXX Supercomputer.

###  5.4 Can We Achieve Further Improvements by Continuing Pretraining?

Training longer can potentially improve the results. Figure 9 illustrates the
normalized log loss over normalized epochs for both model sizes,
SaulLM-54B-base and SaulLM-141B-base. The figure presents both the raw and
smoothed loss curves, which exhibit a clear downward trend throughout the
training period, with no indication of saturation.

This observation suggests that continuing the pretraining process beyond the
current SaulLM-base can lead to further improvements. The consistent decrease
in loss implies that the models have not yet reached their full potential and
that additional pretraining could enhance their performance further, which is
consistent with findings from other works penedo2023refinedweb;
albalak2024survey; muennighoff2024scaling.

###  5.5 How Much Does Scaling Help?

Table 3 quantifies the impact of scaling the model and compares the
performance between SaulLM-medium and SaulLM-large. Due to space constraints
in-depth analysis is reported in Figure 17.

The main takeaway is that scaling generally improves overall results, but we
also observe inverse scaling on some legal tasks mckenzie2023inverse;
li2024inverse. Unsurprisingly, for the majority of tasks across all
categories, increasing the model size leads to improvements, but for tasks
involving conclusion, interpretation, and rules, we observe a proportion of
tasks ($20\%$) that follow inverse scaling laws.

###  5.6 Energy Consumption

The training was conducted on XXXXX666For anonymity reasons we hide the name
of the super computer., ranked XXX in the Green500 since November
2022777https://top500.org/lists/green500/2023/11/, as one of the world’s most
efficient machines in terms of performance per watt.

Experiments for training SaulLM were performed between February 20th and May
15th. Energy consumption for each job was meticulously tracked, and we
calculated and displayed the average power used per node for each job involved
in this training in Figure 9888The high number of jobs (over 1800) for this
project arise from (i) the technical difficulties in efficiently using AMD
GPUs, and (ii) numerous hardware failures when scaling up the number of
nodes.. The mean power usage ranged from 600W to 2500W, reflecting the varying
utilization of the GPUs. Each node contains four MI250X GPUs, which have a
theoretical Thermal Design Power (TDP) of 560W. This configuration explains
the maximum consumption of 2500W during high-intensity GPU usage.

Overall, the project involves over $160,000$ hours of MI250 for debugging,
continued pretraining, instruction finetuning and preference alignment. The
total energy consumed was $65,480.4$kWh. Consumption is significantly lower
than the typical energy requirements for full LLM training, showing that
continued pretraining is an effective strategy for specializing in new LLMs
while optimizing energy efficiency. Further information on energy
considerations is reported in Appendix A.

##  6 Conclusion & Limitations

Conclusions: This study released two new legal LLMs under the MIT license:
SaulLM-54B and SaulLM-141B. They leveraged the Mixtral architecture and
continued pretraining on a large legal corpus. Our findings show significant
advancements in processing and understanding complex legal documents. Through
continued pretraining, instruction fine-tuning, and preference alignment using
domain-specific optimization, we have demonstrated substantial improvements
compared to GPT-4, Llama3 and original Mixtral models as measured on
LegalBench-Instruct.
Limitations: Our experiments suggest that the instruction finetuning and
alignment processes utilized by Mixtral-Instruct and Llama3 are advanced and
challenging to replicate. These processes often rely on proprietary datasets
and significant computational resources that are not readily available in
open-source frameworks. Although both SaulLM-54B and SaulLM-141B achieve
stronger performances than Llama3 and Mixtral Instruct on legal benchmarks, we
found that they are slightly weaker at following generic instructions.Thus
future work will strive to address the current limitations by developing more
robust methods for instruction finetuning and alignment that are accessible to
the broader research community.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: We propose a new domain adaptation recipe.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: See limitations.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [N/A]

  14. Justification: No theoretical paper.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: Dataset and model are available with a list of hyperparameter in training details.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: Models are released under a commercial license and data are described.

  25. Guidelines:

     * •

The answer NA means that the paper does not include experiments requiring
code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: We relied on previous works that are cited.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [Yes]

  34. Justification: We follow previous works.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: See model details and energy analysis.

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: We fully comply.

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [Yes]

  49. Justification: See limitation.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification:

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: MIT-liscence

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes]

  64. Justification: We introduce new models that will be publically released under MIT liscence.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: [TODO]

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: [TODO]

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.

##  Appendix A Energy Analysis

Figure 13, Figure 13, Figure 13, and Figure 13 show different analysis of
energy consumption while developing and training SaulLM-54B and SaulLM-141B.

Figure 10: Energy Analysis. GPU Load vs Elapsed Time for Different Numbers of
Nodes.

Figure 11: Energy Analysis. Number of Jobs vs Number of Nodes.

Figure 12: Energy Analysis. Consumed Energy vs Elapsed Time for Different
Numbers of Nodes.

Figure 13: Energy Analysis. Log-Scaled Total Power Consumption vs GPU Load for
Different Numbers of Nodes.

##  Appendix B Further Results on LegalBench

###  B.1 Per Category Analysis

Figure 14: Category Analysis on LegalBench-Instruct.

Figure 15: Category Analysis on LegalBench-Instruct.

###  B.2 Per Task Analysis

Figure 16: Per Task Analysis: Quantifying the role of DPO w.r.t. instruction
fine-tuning only. Figure 17: Per Task Analysis: Quantifying the role of DPO
w.r.t. instruction fine-tuning only.