Low-rank adaptation of large models, particularly LoRA, has gained traction due to its computational efficiency.
This efficiency, contrasted with the prohibitive costs of full-model fine-tuning, means that practitioners often turn to LoRA and sometimes without a complete understanding of its ramifications.
In this study, we focus on fairness and ask whether LoRA has an unexamined impact on utility, calibration, and resistance to membership inference across different subgroups (e.g., genders, races, religions) compared to a full-model fine-tuning baseline.
We present extensive experiments across vision and language domains and across classification and generation tasks using ViT-Base, Swin-v2-Large, Llama-2 7B, and Mistral 7B.
Intriguingly, experiments suggest that while one can isolate cases where LoRA exacerbates model bias across subgroups, the pattern is inconsistent—in many cases, LoRA has equivalent or even improved fairness compared to the base model or its full fine-tuning baseline.

We also examine the complications of evaluating fine-tuning fairness relating to task design and model token bias, calling for more careful fairness evaluations in future work.

§ INTRODUCTION & MOTIVATION

The challenge of efficiently scaling large models has led to the growing interest and reliance on parameter-efficient fine-tuning, which focuses on adjusting only a small, deliberately chosen set of parameters in the base model [Hu et al., 2021, Dettmers et al., 2023, Li & Liang, 2021, Lester et al., 2021].
Of particular interest is the low-rank adaptation (LoRA) technique [Hu et al., 2021], in which the pre-trained weight matrices are frozen while their changes from fine-tuning are approximated by low-rank decompositions. LoRA has received significant attention due to its simplicity and effectiveness in a variety of tasks across both language [Liu et al., 2022] and vision [Gandikota et al., 2023] domains.

Despite the popularity of LoRA, however, little is known about its effects on trustworthiness, such as fairness and robustness. The lack of understanding together with LoRA's wide adoption implies that practitioners may be deploying models with unintended and potentially harmful consequences in high-stakes applications. To this end, this work initiates a study on fairness and asks the following:

What are the effects of LoRA, if any, on subgroup fairness?

Central to the existing knowledge gap is the prohibitive cost of full fine-tuning that deters a direct comparison against LoRA. This is troubling since the increased adoption of large models often involves taking off-the-shelf pre-trained models (e.g., Mistral [Jiang et al., 2023]), fine-tuning them on custom data (if said models cannot reason in-context with few-shot prompts), and running them as part of decision-making processes.
In many scenarios such as enterprise [Tran, 2023], healthcare [Yu et al., 2023], and banking [Loukas et al., 2023], practitioners may gravitate towards LoRA solely for its cost-effectiveness without consideration for unfair outcomes; this can lead to tangible harm at tasks such as risk assessment, credit score estimation, loan approvals, and hiring/promotion evaluations.

Apart from the real-world motivations above, tangential prior work also inspired this study from an algorithmic standpoint.
Specifically, LoRA is characterized by its reduced fitting capacity through low-rank approximations; a similar property is also inherent to model pruning and differentially private training. Respectively, [Tran et al., 2022] and [Bagdasaryan et al., 2019] found that both pruning and private training can worsen the fairness of accuracy across subgroups (despite achieving good overall accuracy), as the sparsity and noisy gradients (due to private training) can both impact a model's ability to fit minority and underrepresented inputs.
On the other hand, [Langenberg et al., 2019] and [Awasthi et al., 2020] showed that low-rank weights and representations can lead to better adversarial robustness.

Prompted by these studies, we ask whether LoRA exhibits similar side effects and, if so, whether they are consistent across tasks and datasets.
All in all—is LoRA's efficiency a “free lunch”?

In this study, we seek to better understand the fairness implications of LoRA on large models via extensive experimentation.

Our study and findings can be summarized as follows:

* We fine-tune both vision and language pre-trained models across sizes 86M-7B (ViT-Base [Dosovitskiy et al., 2020], Swin-v2-Large [Liu et al., 2022], Llama-2 7B [Touvron et al., 2023], and Mistral 7B [Jiang et al., 2023]), and across tasks including hatespeech detection, gender classification, machine translation, multiple-choice QA, and cloze completions, juxtaposing full-model fine-tuning and LoRA and measuring the subgroups disparities on accuracy, calibration, privacy as resistance to membership inference, and gender bias. To our knowledge, our work is the first to provide a comprehensive empirical investigation into the fairness properties of low-rank adaptation.

* Intriguingly, our experiments reveal no consistent pattern of LoRA worsening subgroup fairness, compared to full fine-tuning across different architectures and domains (<ref>).
Note that isolated examples do exist where LoRA worsens fairness across subgroups, though such cases should be viewed with target applications and metric sensitivity in mind [Kleinberg et al., 2016];
e.g., LoRA may appear less fair via worst subgroup accuracy but equally fair under demographic parity difference (DPD), which only considers positive predictions. Nonetheless, for any fixed task and its appropriate fairness metrics that we experimented on, we found no strong evidence that LoRA is less fair.

* The fairness implications may depend on the quality of the underlying pre-trained model (<ref>). We also observe cases where LoRA does exacerbate unfairness can disappear when the base pre-trained model is stronger (<ref>) when all else is kept constant.
This suggests that the fairness properties of LoRA are not merely a function of its parameter efficiency (cf. model pruning [Tran et al., 2022]).

* The LoRA rank has little impact on subgroup fairness (<ref>).
While rank can be a confounding factor for its impact on model capacity and thus fairness (cf. pruning and private training), we did not observe a significant influence of rank on either utility or fairness.
Our finding is in line with existing utility analysis [Hu et al., 2021] across tasks of varying difficulty (binary image classification, machine translation, language modeling).

* LLMs can exhibit token biases, complicating fairness evaluations for generative tasks.
A common strategy for eliciting model preferences is to compare token likelihoods for completing prompt templates [Wang et al., 2023].
However, we found that (1) small-scale LLMs (7B) may have strong and often unpredictable biases towards specific tokens for both full fine-tuning and LoRA (also reported recently by [Zheng et al., 2024]), and that (2) such biases are not alleviated by re-ordering answer options, switching base pre-trained models, or using rarer tokens (e.g., emojis and special UTF-8 characters).
This implies that fairness conclusions can be confounded by such bias.

§ PRELIMINARIES & RELATED WORK

An important paradigm in modern machine learning (ML) is to adapt large pre-trained models to downstream tasks through fine-tuning. The benefits of fine-tuning are two-fold: (1) it leverages the extensive knowledge stored in these pre-trained models, and (2) it promises greater efficiency compared to training from scratch.
However, as models grow in size, this efficiency advantage becomes elusive due to increased demand on compute; for example, simply keeping the gradients of Llama-2 70B [Touvron et al., 2023] in 16-bit precision requires 130GB of memory, which is already infeasible for most commodity hardware. This gap motivates parameter-efficient fine-tuning methods and subsequent novel trustworthiness concerns. Here, we briefly outline work most closely related to the focus of this paper and some preliminaries that ground our analyses.

Low-Rank Adaptation.

LoRA [Hu et al., 2021] is a widely used parameter-efficient fine-tuning algorithm for large models. It proposes to separate out the weight deltas from fine-tuning and approximate them using low-rank matrices; inference then involves forward passing both the (frozen) pre-trained model and the low-rank model deltas, also known as adapters, and summing the activations. Specifically, for a pre-trained weight matrix $\mathbf{W} \in \mathbb R^{d\times k}$ with dimensions $d, k$, LoRA approximates its changes from fine-tuning as $\Delta\mathbf{W} \approx \mathbf{B}\mathbf{A}$ where $\mathbf{B} \in \mathbb R^{d\times r}$ and $\mathbf{A} \in \mathbb R^{r\times k}$ with rank $r \ll \min(d, k)$, and thus inference on input $\mathbf{x} \in \mathbb R^d$ is $\mathbf{W}\mathbf{x} +\mathbf{BA}\mathbf{x} \approx (\mathbf{W} + \Delta\mathbf{W})\mathbf{x}$ if $\Delta\mathbf{W}$ is obtained through full fine-tuning.
$\mathbf{A}$ and $\mathbf{B}$ can be updated directly via backpropagation.
Typically, implementations of LoRA apply to all query and value matrices of self-attention layers in the pre-trained transformer.
To fine-tune for supervised tasks, an additional head is also attached to the last layer of the model.

Fairness evaluations in machine learning.

Fairness is a pivotal concern as biased models from training data/algorithms can lead to misleading and even catastrophic consequences, and understanding and mitigating such bias has been an active area of research [Kearns et al., 2018, Barocas et al., 2023, Chouldechova & Roth, 2018, Mehrabi et al., 2021].
The precise definitions and measurements of fairness, however, are often application-dependent.

Fairness of classification.
Classification tasks have well-accepted fairness evaluation methods and metrics.
Subgroup accuracy parity and worst subgroup accuracy (relatedly, best-worst spread) are two metrics commonly studied in prior work [Kearns et al., 2018, Bagdasaryan et al., 2019, Yang et al., 2020, Tran et al., 2022], which measure differences in accuracy. E.g., are people with different skin colors equally well-classified? Does the subgroup with the worst utility get “poorer” under the ML algorithm?

We also consider the two common fairness metrics seen in recent work [Wang et al., 2023, Hong et al., 2024]. Demographic parity difference (DPD) [Agarwal et al., 2018, Agarwal et al., 2019] measures how varied are the model's positive predictions are across attributes.
Equalized odds difference (EOD) measures if the model has similar predictive performance across both true and false positive rates, regardless of the protected attribute.

Typically, in situations where ensuring equal representation or opportunity is the goal, such as fair hiring decisions or loan approvals across demographic groups, the “one-sided” DPD might be preferred. In scenarios where equitable outcomes are critical, such as the success of medical diagnosis across different demographic groups, the “balanced” EOD may be more appropriate.
See <ref> for formal definitions.

Fairness of generation.
For generative tasks, fairness evaluations can be nuanced due to the open-ended nature of outputs—what does it mean for generated pixels/tokens to be “fair”? Prior work explored biases in word embeddings [Bolukbasi et al., 2016], human-rated fairness scores on generated outputs [Lee et al., 2023], or reducing generative models to classifiers through prompting [Wang et al., 2023].
Recent work in the surge of large generative models such as [Esiobu et al., 2023] and [Bianchi et al., 2023] focuses on the behavioral fairness of the generative models (e.g., whether the output text is stereotypical) as opposed to establishing formal algorithmic fairness notions.

Fairness of fine-tuning.
When evaluating the fairness properties of fine-tuning algorithms, we argue for the following key desiderata: (1) the fine-tuning task should not teach the model to be fair (or else we cannot extrapolate the evaluation to new tasks); (2) there is a “side-channel” through which we can measure fairness (e.g., measuring gender bias for machine translation); and (3) the fairness implications are directly relevant to the task being fine-tuned on (so that any observed fairness issues are indicative of realistic harm).
We strive to achieve all these desiderata when designing fairness evaluations, though experiments forgoing desideratum (3) may still serve as “probes” and provide useful insights.

§ FAIRNESS EVALUATIONS OF LORA

We now turn to the experiments. We first describe the task setup and datasets, and then present results across dimensions of accuracy (<ref>), calibration (<ref>), privacy as (group-)differential resistance to membership inference attacks (MIAs) (<ref>), and gender bias in generative tasks (<ref>), along with their appropriate fairness considerations.
We report a subset of the relevant metrics for each dataset and task and defer full results and additional implementation details to the appendix.

§.§ Tasks, Datasets, and Their Fairness Considerations

Hatespeech detection.We construct 4 data subsets from the Berkeley D-Lab Hatespeech dataset [Kennedy et al., 2020]: Gender, Race, Religion, and Sexuality. The subsets contain 13976, 11670, 6081, and 7297 examples, respectively, where each example is a tweet-sized text snippet targeting a specific subgroup within the subset (e.g., hatespeech in the Religion subset may target Buddhists or Christians), and fairness is measured across these subgroups. Each example has a scalar hatespeech score, which we binarize into labels to turn regression into classification for the ease of applying and contrasting standard fairness metrics (<ref>).

Face image classification.We use the UTK-Face dataset [Zhang et al., 2017], where each face image is labeled with gender, age, and specified race of the person.
We consider gender classification (binary) and age classification (9-bins) as the fine-tuning tasks, and race attributes are used as subgroups to evaluate fairness, following [Bagdasaryan et al., 2019] and [Tran et al., 2022].

For hatespeech detection and face image classification, a fair fine-tuning method should produce models that:

(1) perform well across all subgroups (i.e., accuracy parity); (2) do not worsen the worst subgroup accuracy; and (3) make errors equally often across subgroups (captured by EOD).
For hatespeech detection and related applications such as credit scoring, content moderation, and hiring processes, the model should also make positive predictions (e.g., flagging hatespeech) equally often across subgroups (captured by DPD).

Machine translation.We use the WinoMT dataset [Stanovsky et al., 2019], which consists of tuples of English sentences that include both the pro- and anti-stereotypical gender constructions by varying the subject pronouns and compositions. For example, in the sentence pair “The developer argued with the designer because [pronoun] did not like the design” and “The developer argued with the designer because [pronoun] idea cannot be implemented.”, the gender pronouns can be varied between she/he (or her/his) to produce a 4-tuple.

We then construct the fine-tuning task as translating these sentences from a gender-neutral language (we used Turkish) back to English and observe whether the fine-tuned model surfaces gender bias (e.g., prefers the stereotypical English translation). We experimented both a balanced and a pro-stereotype set of Turkish-to-English translation examples for the fine-tuning (<ref>).

Language modeling: multiple-choice QA and cloze completions.To probe the fairness implications of fine-tuning from a different angle, we also explore how the base model's gender bias may surface differently as it fits on gender-neutral text under LoRA vs. full fine-tuning.
We sample 50,000 Yelp reviews from the multi-dimensional gender bias dataset [Subramanian et al., 2018], which consists of reviews where: (1) the rating is 3/5 such that the sentiment tends to be neutral, and (2) the gender is not easily identifiable.

We then elicit the model's bias (before and after fitting next-token prediction on these reviews) by prompting it to deduce the gender of the review author, either through multiple-choice QA or cloze completions. Because the reviews are selected to be gender-neutral, bias is measured by how much LoRA and full fine-tuning deviate from the golden behavior of guessing male/female equally often, compared to the base models.

For example, a cloze task with the template [Describing their most recent experience: “{review}”, says a {gender}] elicits model preference by comparing token likelihood for “male” and “female” at the slot {gender}. We also consider multiple-choice setups with options for the model to guess gender-neutral/non-binary.

Because generative models give open-ended outputs, part of the difficulty in designing fairness evaluations is finding a “side-channel” through which we can easily and quantitatively probe the model's bias (recall <ref>).
For both the above tasks, we focus on gender bias because it is interpretable and orthogonal to the fine-tuning tasks, has a clear societal impact, and is frequently considered in prior work [Wang et al., 2023, Hong et al., 2024].

Training settings.On hatespeech detection, machine translation, and language modeling, we fine-tune (LoRA and full fine-tune) on Llama-2 7B [Touvron et al., 2023] and Mistral 7B [Jiang et al., 2023]; on image classification, we fine-tune on ViT-Base [Dosovitskiy et al., 2020] and Swin-v2-Large [Liu et al., 2022].
On all datasets and tasks, LoRA can match full-model fine-tuning in terms of both train/test performance (though some tasks need higher LoRA rank), allowing fair comparison as absolute performance advantage can be a confounding factor in fairness evaluations.

For language modeling, we also experiment on the instruction-following versions of Llama-2 7B and Mistral 7B.
The evaluation data for each experiment is a random 20% split (varies across random seeds) except for language modeling where evaluation (via prompting) is done on the same training set.

LoRA vs. full fine-tuning on group-wise accuracy and equalized odds difference (EOD, lower is fairer) on UTK-Face gender and age classification for ViT-Base (figs 1, 3) and Swin-v2-Large (figs 2, 4).
Error bars: 95% CI across 5 seeds.
By all metrics LoRA may be considered less fair than full fine-tuning on ViT-Base but equally as fair when switched to a better base model Swin-v2-Large.

LoRA vs. full fine-tuning on group-wise accuracy, demographic parity difference (DPD, lower is fairer), and equalized odds difference (EOD, lower is fairer).
Error bars: 95% CI across five seeds.
Numbers in brackets: subgroup sizes.
Rows: Llama-2 7B and Mistral 7B on D-Lab religion hatespeech detection.
Columns: group-wise accuracy, DPD, EOD.
No consistent pattern that LoRA worsens subgroup fairness compared to full fine-tune, and tendency can flip across the base models.

§.§ Accuracy

<ref> present results on UTK-Face age classification and D-Lab religion hatespeech detection across four base model architectures; more results are deferred to <ref>.
There are several interesting observations:

No consistent pattern of LoRA worsening subgroup fairness compared to full fine-tuning.
Overarchingly, LoRA and full fine-tuning exhibit similar performance across all subgroups, with the worst subgroup performance and best-worse spread for LoRA being consistently on par with full fine-tuning. Observe also that for most subgroups, LoRA does not worsen either DPD or EOD and may even improve them in some cases.

Fairness implications can depend on the quality of pre-trained model.A closer look at <ref> suggests that while LoRA may be considered less fair than full fine-tuning on ViT-Base—by decreased worst subgroup utility on Black group for age classification (1st subplot) and by increased EOD on Asian group for gender classification (3rd subplot)—the tendency disappears when the base model is switched to the more powerful Swin-v2-Large (all else kept the same).

This is interesting as it suggests that the fairness properties of LoRA are not only a function of its parameter efficiency and they provide a separation from model pruning where [Tran et al., 2022] found that the fairness ramifications persist across model sizes.

It is nonetheless possible to isolate cases where LoRA is less fair, but such cases should be viewed with target applications and metric sensitivity in mind.

Another interpretation of the above observation is that one can single out cases where LoRA is less fair than full fine-tuning.
We note that different fairness metrics may be more or less relevant depending on the goals and priorities of the task at hand.
Take, for example, UTK-Face gender classification where the female category is labeled as 1; for applications where correctly classifying females are important (e.g., when there is drastically less data for females than males), the unfairness of LoRA according to EOD (<ref>) may be less relevant than DPD which only looks at “positive” (i.e., female) predictions. There, DPD may very well lead to a different conclusion that LoRA is equally as fair  (<ref> in <ref>).
In the context of fairness metric sensitivity [Kleinberg et al., 2016], it is therefore crucial for practitioners to adopt a task-centric perspective to ensure a meaningful and relevant fairness evaluation.

§.§ Calibration

Confidence histograms (top) and reliability diagrams (bottom) for Llama-2 7B on D-Lab religion (left), Swin-v2-Large on UTK-Face gender (middle), and Llama-2 7B on subgroups with highest ECE on D-Lab religion (right). Dotted purple line indicates perfect calibration. Gap is calculated by confidence minus accuracy. Model with a lower ECE is better calibrated.

While metrics in <ref> concentrate on equality in error rates across groups, the measure of calibration within groups is another important fairness metric to ensure the probability estimates align with real-world outcomes, both globally and across different subgroups [Kleinberg et al., 2016].

To measure calibration, we extract the model's confidence on the 20% evaluation set by examining the probability outputs from the classification head. We then follow [Guo et al., 2017] and generate the confidence histograms and the reliability diagrams. We defer background on calibration to <ref> and more results to <ref>.

LoRA and full fine-tuning show comparable calibration levels, though LoRA shows signs of overconfidence.
<ref> shows that both LoRA and full fine-tuning exhibit a reasonable level of calibration, with their expected calibration error (ECE) being relatively low and comparable across different datasets and subgroups. The reliability diagrams illustrate that the probabilities predicted by both methods are well-aligned with the observed accuracies. Neither method consistently yields less calibrated models than the other, and the conclusion holds even when we specifically look at the respective subgroups with highest ECE (<ref> right).
One subtle observation is that

LoRA shows a tendency for its predicted probabilities to cluster at the lower and upper ends of the scale, particularly in the 0-0.1 and 0.9-1 confidence bins (top row of <ref>). This skewness indicates a degree of overconfidence in LoRA's predictions, leading to less reliable decision-making [Niculescu-Mizil & Caruana, 2005] that could potentially affect subgroups disparately.

§.§ Resistance to Membership Inference Attacks (MIA)

Membership inference attacks (MIA) involve predicting whether an example was in the training set of a target model. MIA has downstream implications for data privacy, copyright protection [Henderson et al., 2023], as well as (detecting) data contamination [Jiang et al., 2024, Oren et al., 2024, Zhang et al., 2024], and it is useful to understand the impact of fine-tuning on the model's resistance to MIA.
One reason to hypothesize that LoRA may exhibit different behaviors than full fine-tuning is its parameter efficiency and thus its decreased capacity to memorize and overfit. Past work showed that overfitting tends to result in higher vulnerability of MIA [Carlini et al., 2022, Yeom et al., 2018], and minority groups tend to be outliers and thus possibly memorized more often [Feldman, 2020].

Motivated by the above hypothesis and relevant observations, we evaluate the resistance of fine-tuned models against MIA to see whether LoRA makes the fine-tuned model more (or less) vulnerable compared to full fine-tuning. In particular, we focus on the Likelihood Ratio Attack (LiRA, [Carlini et al., 2022]) due to its efficacy. (See <ref> for background and implementation of MIA and <ref> for additional results.) We attack ViT-Base and Swin-v2-Large fine-tuned with UTK-Face dataset for binary gender classification; and Llama-2 7B and Mistral 7B fine-tuned with D-Lab Hatespeech dataset for binary hatespeech classification. We repeat this for both LoRA and full fine-tuning and compare their resistance to MIA when the training loss is about the same for both methods. We refer the reader to <ref> for the details on how we partitioned each dataset to train the shadow models.

LoRA is generally as resistant to MIA as full fine-tuning.
For each model and dataset pair described above, we obtain receiver operating characteristic (ROC) curves by varying the confidence thresholds. <ref> show the ROC curves in log-scale to emphasize true positive rates at low false positives. We defer results with ViT-Base and Mistral 7B models and a simpler MIA attack (LOSS) to <ref>. From <ref>, we see that there is no clear evidence that LoRA makes the model less resistant to MIA compared to full fine-tuning. On Swin-v2-Large with UTK-Face, LoRA seems more resistant than full fine-tuning overall and also at the subgroup level across different races. On Llama-2 7B with D-Lab, while LoRA seems marginally less resistant on average, there are subgroups for which LoRA provides higher resistance than full fine-tuning. In general, we also do not observe a significant impact of the subgroup size on their resistance to MIA.

Likelihood Ratio Attack (LiRA) on Swin-v2-Large for membership inference on UTK-Face gender. LoRA models are slightly more resistant to MIA than full fine-tuning.

Likelihood Ratio Attack (LiRA) on Llama-2 7B for membership inference on D-Lab religion. LoRA models are roughly equally resistant to MIA compared to full fine-tuning.

§.§ Gender Bias in Generative Tasks

Cloze completion gender bias of base model, LoRA, and full FT. Red dotted line is the ideal behavior of guessing two genders equally often. Error bars are over five cloze templates.

Ratios of pro-stereotypical and anti-stereotypical gendered English translations generated by models. Left: Llama-2 7B and Mistral 7B trained on gender-unbiased dataset. Right: same models trained on pro-stereotypical gendered dataset. The golden behavior should be 0.5/0.5 ratios.

We also explore how gender bias may surface from fine-tuning for machine translation and language modeling (recall <ref>).

Respectively, <ref> present gender bias results on Yelp review language modeling and Turkish-to-English translation on Llama-2 7B and Mistral 7B; more results are deferred to <ref>. The average eval BLEU scores [Papineni et al., 2002] range between 58 and 63, reflecting good and fluent translations [Lavie, 2010].

No definitive evidence of LoRA exacerbating gender bias.
On language modeling evaluations, we observe that: (1) compared to the pre-trained base models (both raw and instruction-tuned), the fine-tuned models tend to reduce bias, and (2) LoRA does not exhibit more bias than full-model fine-tuning.
On machine translation evaluations, we see that: (1) when trained on a gender-unbiased dataset, both Llama-2 7B and Mistral 7B models demonstrate balanced frequencies of gender representation, indicating unbiased behavior regardless of the fine-tuning approach; and
(2) when trained on a pro-stereotypical gendered dataset, this bias is transferred heavily onto the fine-tuned model's behavior. Specifically, while Mistral 7B exhibits comparable levels of gender bias with either LoRA or full-model fine-tuning, Llama-2 7B presents less gender bias with LoRA fine-tuning, suggesting that LoRA may sometimes lead to a less severe gender bias than full-model fine-tuning.

§.§ Effect of LoRA Rank

Subgroup accuracy and EOD across of LoRA ranks from 0 to 768 on ViT-Base on UTK-Face gender classification.

We also explore the choice of rank for LoRA, as it may also be a confounding factor in the model's fitting capacity and fairness impact.
Results from UTK-Face gender classification (<ref>) reveal that accuracy and fairness metrics (EOD) are not influenced by rank, aligning with findings from [Hu et al., 2021]. In review generation (<ref>), where a low rank might result in underfitting due to limited capacity, no definitive connection between rank and fairness was found.
Similarly, for machine translation (<ref>), fairness remains largely unchanged beyond a rank threshold that ensures quality translation (indicated by BLEU scores), as shown by stable fairness measures (flat lines in the plot) despite an increase in rank. More results are deferred to <ref>.

Fairness and BLEU score (twin axes) for Mistral 7B on gender-unbiased (left) and pro-stereotype biased (right) training sets. LoRA rank ranges from 1 to 4096. Dotted lines: full fine-tuning.

§ DISCUSSIONS, LIMITATIONS, AND FUTURE WORK

We have presented extensive empirical analyses and found no conclusive evidence that LoRA may exacerbate subgroup fairness compared to full fine-tuning. Does this imply that the parameter efficiency of LoRA is a free lunch? Possibly, but not necessarily—no evidence of unfairness does not imply fairness.
While our study aims to be comprehensive, we outline some limitations and directions for future work.

Token bias in LLMs complicates fairness evaluations on generative tasks.
Unlike classification tasks where model predictions can be used for downstream decisions directly (and thus fairness can be evaluated directly), generative tasks involve diverse outputs that do not always reveal the model's preferences.
For language models, practitioners often instead elicit such preferences through prompting [Lee et al., 2023], e.g., via QA [Cobbe et al., 2021] and multiple choice [Hendrycks et al., 2020], and this underpins our gender bias experiments (<ref>).

This elicitation process, however, introduces an ambiguity between the model's preference for specific tokens vs. its actual preference on the subject matter: if a model responds “yes” to a question, is it because “yes” is the correct answer, or that the model simply generates “yes” more often?

Indeed, we found that models have strong and often unpredictable preferences towards specific tokens.
For example, full fine-tuned Llama-2 7B chose “Yes” over 99% of the 50k Yelp reviews, while surprisingly, LoRA preferred “No” 99% of the time.
This phenomenon persists across various setups—“Yes/No” answers and multiple-choice QA with numeric and letter options (see <ref>).
Moreover, these biases are not easily mitigated:
(1) negating the semantic meanings of the prompts to flip “Yes/No” options (e.g., male + yes $\to$ female + no) did not change model preferences (<ref>);
(2) models may favor token “A” even when it denoted opposite answers (<ref>);
(3) the preference remains even when the ordering of choices was modified (e.g., ABC to BAC; <ref>); and
(4) the above issues can persist when switching to a different base model and even when answer options are presented with rare symbols (e.g.,  (U+1F7E0) and  (U+25D1); <ref>). Among different tokens to compare model preferences, we found using “male/female” tokens mitigates the strong token bias (<ref>) and focused on reporting these results in <ref> and deferring the rest to <ref>.

Future work should prioritize evaluation methods beyond the token level and consider more nuanced effects of bias through semantics, discourse structure, and the holistic content of the generated text.

Considerations for fairness gerrymandering.
Fairness gerrymandering happens when change in subgroup definitions (e.g., adding/removing subgroups with intersections/unions of protected attributes) alters the fairness conclusions [Kearns et al., 2018, Yang et al., 2020]. For example, while we found no evidence for LoRA worsening fairness on D-Lab religion (<ref>) and race (<ref>) data, this conclusion may not transfer to the intersection of these subgroups (e.g., Asian atheists). Addressing fairness gerrymandering can be computationally demanding both theoretically [Kearns et al., 2018] and empirically with large models; we leave exhaustive experimentation on varying subgroup definitions to future work.

Concluding remarks.
Our study sheds light on the fairness properties of low-rank adaptation (LoRA) across architectures, model sizes, datasets, and fairness considerations.

In future work, we hope to extend fairness evaluations in generative settings by exploring better experiment design that minimizes the impact of model token bias and/or reliance on reasoning capacity for evaluating generative models.
Probing techniques (e.g., [Alain & Bengio, 2016, Hewitt & Liang, 2019, Stoehr et al., 2023, Zou et al., 2023]) emerge as a promising tool to assess models while circumventing their token biases, though the use of additional classifier heads resemble our supervised evaluations. It is also worth exploring and comparing other parameter-efficient methods (e.g., [Li & Liang, 2021, Liu et al., 2022]) and their intersection with related techniques such as quantization [Dettmers et al., 2023, Hong et al., 2024] and pruning [Dery et al., 2024, Gromov et al., 2024]; this may offer insight whether our findings with LoRA is unique to its algorithmic constructions.

§ ETHICS STATEMENT

This work comprehensively evaluates the fairness implications of low-rank adaptation (LoRA) of language models in comparison to full-model fine-tuning methods across multiple critical dimensions: disparate accuracy across subgroups, calibration, resistance to membership inference attacks, and gender bias. The study spans both vision and language domains, acknowledging the profound impact these technologies have on society.

We recognize the ethical implications of our findings, particularly concerning the equitable treatment of diverse populations and the protection of individuals' privacy. Our research discovers areas where bias and inequities are amplified when switching from full-model fine-tuning to LoRA (and sometimes, the other way around).  We urge the community to consider the ethical ramifications of the choice of fine-tuning method and caution against adopting the method that gives the overall best utility without careful consideration of fairness implications. It is essential to continue efforts to mitigate bias, enhance fairness, and protect privacy in machine learning systems. This includes ongoing evaluation, adopting ethical AI frameworks, and engaging with diverse stakeholders to understand and address potential impacts comprehensively.

Future work should not only extend the technical dimensions evaluated but also deepen the engagement with interdisciplinary approaches to understand and address the societal implications of different fine-tuning methods. By doing so, we can strive towards the development of scalable machine learning technologies that are not only advanced but also aligned with the principles of equity, fairness, and respect for all individuals.

§ REPRODUCIBILITY STATEMENT

The code for this project will be open-sourced at <https://github.com/kenziyuliu/lora-fairness>. All experiments are done using standard machine learning packages, including PyTorch [Paszke et al., 2019], Hugging Face (transformers [Wolf et al., 2019] and PEFT [Mangrulkar et al., 2022]). Distributed training made use of Hugging Face DeepSpeed [Rasley et al., 2020] integration. Other packages used in this work are summarized in the attached  file.

All experiments are conducted with up to 8 NVIDIA A100-SXM4-80GB GPUs. Most, if not all, individual fine-tuning runs can be finished within one GPU day. Additional experimental details can be found in <ref>. This includes
dataset availability information and preprocessing (<ref>), additional implementation details (<ref>), prompt templates used for evaluations on generative tasks (<ref>).

§ ACKNOWLEDGEMENTS

The authors would like to thank Nicolas Papernot for valuable suggestions on an earlier draft of this work. KZL acknowledges support from the Ravi Family Graduate Fellowship from School of Engineering at Stanford University. BI was supported by a Google PhD Fellowship. SK acknowledges support by NSF III 2046795, IIS 1909577, CCF 1934986, NIH 1R01MH116226-01A, NIFA award 2020-67021-32799, the Alfred P. Sloan Foundation, and Google Inc.

[Achiam et al., 2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.

[Agarwal et al., 2018]
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach.
A reductions approach to fair classification.
In International conference on machine learning, pp.  60–69. PMLR, 2018.

[Agarwal et al., 2019]
Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu.
Fair regression: Quantitative definitions and reduction-based algorithms.
In International Conference on Machine Learning, pp.  120–129. PMLR, 2019.

[Alain & Bengio, 2016]
Guillaume Alain and Yoshua Bengio.
Understanding intermediate layers using linear classifier probes.
arXiv preprint arXiv:1610.01644, 2016.

[Awasthi et al., 2020]
Pranjal Awasthi, Himanshu Jain, Ankit Singh Rawat, and Aravindan Vijayaraghavan.
Adversarial robustness via robust low rank representations.
Advances in Neural Information Processing Systems, 33:0 11391–11403, 2020.

[Bagdasaryan et al., 2019]
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov.
Differential privacy has disparate impact on model accuracy.
Advances in neural information processing systems, 32, 2019.

[Barocas et al., 2023]
Solon Barocas, Moritz Hardt, and Arvind Narayanan.
Fairness and machine learning: Limitations and opportunities.
MIT Press, 2023.

[Bianchi et al., 2023]
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan.
Easily accessible text-to-image generation amplifies demographic stereotypes at large scale.
In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp.  1493–1504, 2023.

[Bolukbasi et al., 2016]
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.
Man is to computer programmer as woman is to homemaker? debiasing word embeddings.
Advances in neural information processing systems, 29, 2016.

[Carlini et al., 2022]
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer.
Membership inference attacks from first principles.
In 2022 IEEE Symposium on Security and Privacy (SP), pp.  1897–1914. IEEE, 2022.

[Chouldechova & Roth, 2018]
Alexandra Chouldechova and Aaron Roth.
The frontiers of fairness in machine learning.
arXiv preprint arXiv:1810.08810, 2018.

[Cobbe et al., 2021]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.

[Dery et al., 2024]
Lucio Dery, Steven Kolawole, Jean-Francois Kagey, Virginia Smith, Graham Neubig, and Ameet Talwalkar.
Everybody prune now: Structured pruning of llms with only forward passes.
arXiv preprint arXiv:2402.05406, 2024.

[Dettmers et al., 2023]
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.
Qlora: Efficient finetuning of quantized llms.
arXiv preprint arXiv:2305.14314, 2023.

[Dosovitskiy et al., 2020]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
In International Conference on Learning Representations, 2020.

[Esiobu et al., 2023]
David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, and Eric Michael Smith.
Robbie: Robust bias evaluation of large generative language models.
In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.

[Feldman, 2020]
Vitaly Feldman.
Does learning require memorization? a short tale about a long tail.
In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp.  954–959, 2020.

[Gandikota et al., 2023]
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau.
Concept sliders: Lora adaptors for precise control in diffusion models.
arXiv preprint arXiv:2311.12092, 2023.

[Gromov et al., 2024]
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts.
The unreasonable ineffectiveness of the deeper layers, 2024.

[Guo et al., 2017]
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.
On calibration of modern neural networks, 2017.

[Henderson et al., 2023]
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang.
Foundation models and fair use.
arXiv preprint arXiv:2303.15715, 2023.

[Hendrycks et al., 2020]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
arXiv preprint arXiv:2009.03300, 2020.

[Hewitt & Liang, 2019]
John Hewitt and Percy Liang.
Designing and interpreting probes with control tasks.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 2019.

[Hong et al., 2024]
Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, et al.
Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression.
arXiv preprint arXiv:2403.15447, 2024.

[Hu et al., 2021]
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
In International Conference on Learning Representations, 2021.

[Jiang et al., 2023]
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.
Mistral 7b.
arXiv preprint arXiv:2310.06825, 2023.

[Jiang et al., 2024]
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo.
Investigating data contamination for pre-training language models.
arXiv preprint arXiv:2401.06059, 2024.

[Kearns et al., 2018]
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu.
Preventing fairness gerrymandering: Auditing and learning for subgroup fairness.
In International conference on machine learning, pp.  2564–2572. PMLR, 2018.

[Kennedy et al., 2020]
Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano.
Constructing interval variables via faceted rasch measurement and multitask deep learning: a hate speech application.
arXiv preprint arXiv:2009.10277, 2020.

[Kleinberg et al., 2016]
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan.
Inherent trade-offs in the fair determination of risk scores.
arXiv preprint arXiv:1609.05807, 2016.

[Langenberg et al., 2019]
Peter Langenberg, Emilio Rafael Balda, Arash Behboodi, and Rudolf Mathar.
On the effect of low-rank weights on adversarial robustness of neural networks.
arXiv preprint arXiv:1901.10371, 2019.

[Lavie, 2010]
Alon Lavie.
Evaluating the output of machine translation systems.
In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Tutorials, Denver, Colorado, USA, October 31-November 4 2010. Association for Machine Translation in the Americas.
URL <https://aclanthology.org/2010.amta-tutorials.4>.

[Lee et al., 2023]
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al.
Holistic evaluation of text-to-image models.
Advances in Neural Information Processing Systems, 36, 2023.

[Lester et al., 2021]
Brian Lester, Rami Al-Rfou, and Noah Constant.
The power of scale for parameter-efficient prompt tuning.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.  3045–3059, 2021.

[Li & Liang, 2021]
Xiang Lisa Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation.
In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.  4582–4597, 2021.

[Liu et al., 2022]
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel.
Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
Advances in Neural Information Processing Systems, 35:0 1950–1965, 2022a.

[Liu et al., 2022]
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.
Swin transformer v2: Scaling up capacity and resolution.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  12009–12019, 2022b.

[Loukas et al., 2023]
Lefteris Loukas, Ilias Stogiannidis, Odysseas Diamantopoulos, Prodromos Malakasiotis, and Stavros Vassos.
Making llms worth every penny: Resource-limited text classification in banking.
In Proceedings of the Fourth ACM International Conference on AI in Finance, pp.  392–400, 2023.

[Mangrulkar et al., 2022]
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan.
Peft: State-of-the-art parameter-efficient fine-tuning methods.
<https://github.com/huggingface/peft>, 2022.

[Mehrabi et al., 2021]
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.
A survey on bias and fairness in machine learning.
ACM computing surveys (CSUR), 540 (6):0 1–35, 2021.

[Naeini et al., 2015]
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht.
Obtaining well calibrated probabilities using bayesian binning.
In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pp.  2901–2907. AAAI Press, 2015.
ISBN 0262511290.

[Niculescu-Mizil & Caruana, 2005]
Alexandru Niculescu-Mizil and Rich Caruana.
Predicting good probabilities with supervised learning.
In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, pp.  625–632, New York, NY, USA, 2005. Association for Computing Machinery.
ISBN 1595931805.
URL <https://doi.org/10.1145/1102351.1102430>.

[Oren et al., 2024]
Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto.
Proving test set contamination for black-box language models.
In The Twelfth International Conference on Learning Representations, 2024.

[Papineni et al., 2002]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp.  311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
URL <https://aclanthology.org/P02-1040>.

[Paszke et al., 2019]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
Pytorch: An imperative style, high-performance deep learning library.
Advances in neural information processing systems, 32, 2019.

[Rasley et al., 2020]
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp.  3505–3506, New York, NY, USA, 2020. Association for Computing Machinery.
ISBN 9781450379984.
URL <https://doi.org/10.1145/3394486.3406703>.

[Stanovsky et al., 2019]
Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer.
Evaluating gender bias in machine translation.
In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.  1679–1684, 2019.

[Stoehr et al., 2023]
Niklas Stoehr, Pengxiang Cheng, Jing Wang, Daniel Preotiuc-Pietro, and Rajarshi Bhowmik.
Unsupervised contrast-consistent ranking with language models.
arXiv preprint arXiv:2309.06991, 2023.

[Subramanian et al., 2018]
Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau.
Multiple-attribute text style transfer.
arXiv preprint arXiv:1811.00552, 2018.

[Touvron et al., 2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023a.

[Touvron et al., 2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023b.

[Tran et al., 2022]
Cuong Tran, Ferdinando Fioretto, Jung-Eun Kim, and Rakshit Naidu.
Pruning has a disparate impact on model accuracy.
Advances in Neural Information Processing Systems, 35:0 17652–17664, 2022.

[Tran, 2023]
Hoang Tran.
How to fine-tune large language models for enterprise use cases.
<https://snorkel.ai/how-to-fine-tune-large-language-models-for-enterprise-use-cases/>, November 2023.
Accessed: 2024-03-21.

[Wang et al., 2023]
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al.
Decodingtrust: A comprehensive assessment of trustworthiness in gpt models.
In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.

[Wolf et al., 2019]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.
Huggingface's transformers: State-of-the-art natural language processing.
arXiv preprint arXiv:1910.03771, 2019.

[Yang et al., 2020]
Forest Yang, Mouhamadou Cisse, and Sanmi Koyejo.
Fairness with overlapping groups; a probabilistic perspective.
Advances in neural information processing systems, 33:0 4067–4078, 2020.

[Yeom et al., 2018]
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha.
Privacy risk in machine learning: Analyzing the connection to overfitting.
In 2018 IEEE 31st computer security foundations symposium (CSF), pp.  268–282. IEEE, 2018.

[Yu et al., 2023]
Ping Yu, Hua Xu, Xia Hu, and Chao Deng.
Leveraging generative ai and large language models: a comprehensive roadmap for healthcare integration.
In Healthcare, volume 11, pp.  2776. MDPI, 2023.

[Zhang et al., 2024]
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al.
A careful examination of large language model performance on grade school arithmetic.
arXiv preprint arXiv:2405.00332, 2024.

[Zhang et al., 2017]
Zhifei Zhang, Yang Song, and Hairong Qi.
Age progression regression by conditional adversarial autoencoder.
In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017.

[Zheng et al., 2024]
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang.
Large language models are not robust multiple choice selectors.
In The Twelfth International Conference on Learning Representations, 2024.
URL <https://openreview.net/forum?id=shr9PXz7T0>.

[Zou et al., 2023]
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al.
Representation engineering: A top-down approach to ai transparency.
arXiv preprint arXiv:2310.01405, 2023.

§ ADDITIONAL BACKGROUND

§.§ Fairness Definitions

Both demographic parity difference (DPD) and equalized odds difference (EOD) quantify the fairness of the model predictions for the sensitive attribute \( A \).

Demographic Parity Difference (DPD)
is the absolute difference in the probability of positive outcomes between two groups distinguished by a sensitive attribute. Specifically,

\begin{equation*}
M_{\text{dpd}} = \left| P(f(X) = 1 \mid A = 1) - P(f(X) = 1 \mid A = 0) \right|,
\end{equation*}

where \( A \) is the sensitive attribute, \( f(X) \) is the prediction of the machine learning model, and \( X \) is the feature vector. A non-zero \( M_{\text{dpd}} \) indicates a disparity in prediction outcomes that are independent of the ground truth and solely based on the sensitive attribute. Furthermore, a large DPD means that there is a large prediction gap between the groups with $A = 1$ and $A = 0$, indicating the unfairness of the model prediction.

Equalized Odds Difference (EOD) is calculated as the larger of two quantities: the disparity in true positive rates and the disparity in false positive rates between two groups distinguished by a sensitive characteristic. Specifically,

\begin{equation*}
M_{\text{eod}} = \max\left\{ M_{\text{TP}}, M_{\text{FP}} \right\},
\end{equation*}

with the components defined as follows (\( Y \) here is the true label):

* True positive equalized odds difference:

\begin{align*}
M_{\text{TP}} = \left| P(f(X) = 1 \mid Y = 1, A = 1) - P(f(X) = 1 \mid Y = 1, A = 0) \ \right|
\end{align*}

* False positive equalized odds difference:

\begin{align*}
M_{\text{FP}} = \left| P(f(X) = 1 \mid Y = 0, A = 1) - P(f(X) = 1 \mid Y = 0, A = 0) \ \right|
\end{align*}

A significant \( M_{\text{eod}} \) reflects a discrepancy in error rates between the groups, indicating the unfairness of the model prediction.

§.§ Definitions in the context of model calibration

Reliability diagrams (or calibration curves) [Niculescu-Mizil & Caruana, 2005] plot the model's predicted confidence against its observed accuracy. To construct these diagrams, predictions are grouped into $M$ interval bins (each of size $1/M$). Within each bin (say $B_m)$, the model's accuracy, denoted as \( \text{acc}(B_m) \), is computed as:

\[
\text{acc}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} 1(\hat{y}_i = y_i),
\]

where \( \hat{y}_i \) represents the predicted class label for sample \(i\), and \(y_i\) is the corresponding true label. The average confidence for bin \(B_m\), expressed as \( \text{conf}(B_m) \), is the mean predicted probability for the samples within that bin:

\[
\text{conf}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} \hat{p}_i,
\]

with \( \hat{p}_i \) denoting the confidence for sample \(i\). A perfectly calibrated model should exhibit \( \text{acc}(B_m) = \text{conf}(B_m) \) for each bin, i.e., the diagram should plot the identity function. The distance between the observed accuracy and the predicted probability in each bin represents the calibration gap.

Expected calibration error (ECE) [Naeini et al., 2015] is a scalar summary statistic of how well the model is calibrated by calculating the weighted absolute difference between predicted probability (i.e., confidence) and accuracy across all the confidence bins. Specifically,

\[
\text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \left| \text{acc}(B_m) - \text{conf}(B_m) \right|,
\]

where \( n \) is the total number of samples. The ECE reflects the calibration gap, with lower values indicating a model whose predicted probabilities are closer to the true outcomes.

Importance of calibration for fairness.Well-calibrated models produce probability estimates that can be trusted equally across different demographic groups. If a model is not well-calibrated, some groups may systematically receive overconfident or underconfident predictions. For example, in the scenario of hatespeech detection, it's imperative that the model operates impartially among all demographics to avoid unfairly censoring content from specific groups.

§.§ Membership Inference Attacks (MIAs)

The goal of membership inference attacks is to estimate whether a query example is part of a model's training set as accurately as possible. To this end, consider a dataset space $\mathcal{X}$, label space $\mathcal{Y}$, a real-world distribution $\mathbb{D}$ over $\mathcal{X} \times \mathcal{Y}$, a training dataset $D$ sampled from $\mathbb{D}$, and a training procedure $f_D \leftarrow \mathcal{T}(D)$ where $f_D$ is a machine learning model that outputs a probability distribution over $\mathcal{Y}$ for any given instance $x \in \mathcal{X}$. The attacker has black-box access to the model $f_D$ (known as the target model) and wishes to determine whether the query example $(x, y) \in D$. Following [Carlini et al., 2022], evaluating the effectiveness of an MIA can be done by measuring the true positive rate at a low false positive rate. The justification is that being able to confidently infer that just one data point is a member is a much bigger breach in privacy than being $51\%$ confident in the membership of a larger number of datapoints, even though both instances may score the same in other classification metrics such as accuracy or AUROC. In other words, an attack is only an effective privacy breach when it has a low false positive rate.

LiRA.The Likelihood Ratio Attack (LiRA) is an important MIA algorithm due to its effectiveness [Carlini et al., 2022]. Here, the attacker trains $N$ shadow models $f_{D_i} \leftarrow\mathcal{T}(D_i)$ where each $D_i$ (for $i \in \{1, \dots, N\}$) is randomly sampled from $\mathbb{D}$ (such that for all $i$, the query example $(x, y) \not\in D_i$) in order to mimic the behavior of the target model $f_D$. For any given model $f: \mathcal{X} \to \mathcal{Y}$, let $f(x)_y$ denote the confidence of $f$ on $(x, y)$, in other words, the probability value output by $f(x)$ for label $y$. Although the attacker has no information about $D$, it has complete information about each $D_i$ as they were the one who sampled the dataset themselves. Thus, the attacker models a distribution of the confidences $f_{D_i}(x)_y$ for each shadow model $i$, where $D_i$ is sampled from $\mathbb{D}$ and $(x, y) \not\in D_i$. From this, the attacker can then compare the target model's confidence $f_D(x)_y$ and perform a hypothesis test against the null hypothesis of $(x, y) \not\in D$ (rejecting the null hypothesis and inferring membership whenever the cumulative distribution function of $f_D(x)_y$ is above some threshold $\tau$). In our implementation, we are performing the “offline” version of the attack, as the traditional “online” approach is infeasibly computationally expensive. For more details about LiRA, we refer the reader to [Carlini et al., 2022].

In our experiments, for each dataset, we partitioned a small subset (20%) from both the training and evaluation sets (for the target models) for membership inference evaluation and used the remaining data to train the shadow models. This way, we ensure that the membership inference target dataset is disjoint from the shadow training dataset, which is a necessary assumption for the offline LiRA attack. To ensure variability between the different shadow datasets, we randomly sample 50% of the shadow training dataset to train each shadow model. The shadow models are fine-tuned via LoRA using the target model's pre-trained model.

LOSS.The LOSS attack [Yeom et al., 2018] is based on the observation that machine learning models are trained to minimize the loss of their training examples. Thus, examples with lower loss are, on average, more likely to be members of the training data. Specifically,

\begin{align}
A_{\text{loss}}(x, y) = \mathbbm{1}[-\ell(f(x)_y) > \tau],
\end{align}

where $\ell(\cdot)$ is the loss function, $\tau$ is a tunable decision threshold parameter, and $A_{\text{loss}}(x, y)$ is the attack model which outputs 1 if the loss is below the threshold $\tau$ (indicating membership) and 0 otherwise. It is a very simple attack that tends to be less effective than more sophisticated attacks like LiRA, but it is also much faster to run.

§ ADDITIONAL DISCUSSIONS

Gender bias evaluations via cloze completion with Yelp reviews. Recall that in <ref>, we performed experiments to evaluate model fairness by comparing the model's preference for gendered completions on Yelp reviews.
Because the focus of the fairness evaluation is directly related to the performance of the model on the specific task it's being fine-tuned for, one limitation of our task setup for evaluating gender bias on Yelp reviews arises: the use of multiple choice QA or cloze completions to elicit model preference primarily compares how LoRA and full fine-tuning surface the underlying gender bias from a fairness-agnostic task, rather than their inherent impact on fairness.

This is a nuanced distinction: although the task setups on supervised classification and cloze completion mirror each other in that any fairness implications would emerge because of the fine-tuning, in the latter case such fairness implications do not directly hinder the model's ability to do the downstream task well (writing gender-neutral Yelp reviews vs. classifying people with darker skin).

§ ADDITIONAL EXPERIMENTAL DETAILS

§.§ Dataset Access and Preprocessing

Hatespeech Detection on Berkeley D-Lab.
The Berkeley D-Lab hatespeech detection dataset [Kennedy et al., 2020] can be accessed via Hugging Face: <https://huggingface.co/datasets/ucberkeley-dlab/measuring-hate-speech>.

We first de-duplicate the original dataset, and take one human annotation of the text example when there exists multiple annotations from multiple raters; then we binarize the annotation for each example as either hatespeech or not by thresholding the assigned hatespeech score at 0.5. To obtain the different subsets of the D-Lab hatespeech dataset (hatespeech examples on Gender, Race, Religion, and Sexuality), we use the provided binary attribute labels to filter the dataset. For example, we use the column  to take only the examples that may target a specific race group; within these examples, there are more granular attribute labels such as  and  through which we can split the dataset into groups and assess model fairness. The Gender, Religion, and Sexuality subsets are similarly created using the columns , , and  and their corresponding granular attribute labels, respectively.

Face Image Classification on UTK-Face.
The UTK-Face dataset [Zhang et al., 2017] can be accessed via <https://susanqq.github.io/UTKFace/>. Each face image is labeled with the age, gender, and racial group of the person in the image. The image is resized to the input dimensions of the base model and normalized before being fed into the model. During training, the images are augmented via random horizontal flips.

Machine Translation on WinoMT.
The WinoMT dataset can be accessed via GitHub.[<https://github.com/gabrielStanovsky/mt_gender/tree/5862928/data/aggregates>]. The linked directory contains three text files that specify all data (), only pro-stereotypical sentence tuples (), and only anti-stereotypical sentence tuples ().

Language Modeling on Yelp Reviews.
The Yelp reviews subset of the multi-dimensional gender bias dataset [Subramanian et al., 2018] can be accessed via <https://huggingface.co/datasets/md_gender_bias/viewer/yelp_inferred>. Note that we only take the text examples from the dataset for fine-tuning the models on next-token prediction, and do not used the inferred gender labels for each review. For fine-tuning training, the text examples are tokenized and concatenated into sequences of length 256 (most examples are much shorter), and then fed into the model as input. Due to computational constraints, we subsample 50,000 examples from the training set for fine-tuning, though our initial experiments on the full dataset ($>$1M examples) suggest that the results are consistent.

For supervised tasks, The training and evaluation split is 80% and 20%, respectively. For language modeling, we focus on fitting next-token prediction on the given set of reviews and fairness is evaluated on the same training set.

§.§ Additional Implementation Details

The instruction-following variants of Llama-2 7B and Mistral 7B are accessible as  and  on Hugging Face (<https://huggingface.co/models>).

§.§ Prompt Templates for Language Modeling Evaluations

Recall from <ref> that to perform fairness evaluations on the language modeling task, we use various prompt templates to elicit the fine-tuned model's preferences and gauge how much the model favors different identity groups (genders in the case of Yelp reviews).

<ref> below lists the prompt templates we use for the language modeling evaluations.
These templates cover a range of scenarios across yes-no questions, multiple-choice questions (with numbers, letters, or special symbols as answer options), as well as different styles of questions (e.g., direct questions, indirect questions, and questions with negation). The prompt templates are generated with the assistance of GPT-4 [Achiam et al., 2023].

The prompts are roughly groupped into the following types:

* YN*: These are yes-no questions that prompt the model to generate text that contains specific identity groups. Since “Yes” and “No” are both treated as a single token, we can directly measure the model's preference by comparing the likelihood of the two tokens being generated at the end of the prompt templates. In these templates, we compare “male” and “female” as the gender groups, and thus for a specific template, we can take four measurements (“male” + “yes”, “male” + “no”, “female” + “yes”, “female” + “no).

* MC*: These are multiple-choice questions that prompt the model to select an answer that corresponding to a specific identity group. The text of the prompt templates are different from YN* templates. Similarly to YN* templates, we can measure the model's preference by comparing the likelihood of the tokens being generated at the end of the prompt templates. The tokens denoting the answer options all have the same length (they are either single tokens, or token sequences with common prefixes in the case of special symbols), and thus we can directly compare the token likelihoods. With these templates, we can also allow the model to select “gender-neutral” or “non-binary” as an answer option beyond “male” and “female”.

* Cloze*: These are cloze templates that prompt the model to complete the sentence with a specific identity group. Unlike the YN* and MC* templates, the cloze templates are more of a fill-in-the-blank style statements than questions. Here, we rely on the fact that “male” and “female” are both treated as a single token and evaluate across these two gender groups.

* *-numeric: These are the same templates as specified by the prompt ID prefixes, but with the answer options replaced by numbers. The goal is to examine whether standard choice tokens like “yes”, “no”, “A”, “B”, etc. are treated differently from numbers. Compare, for example, the templates YN1 and Yn1-numeric in <ref>.

* *-inverted: These are the same templates as specified by the prompt ID prefixes, but with the content of the answer options inverted. In the YN* templates, this means answer options are presented as “no” and “yes”; in the MC* templates, this means what A, B, C denote are changed. Compare, for example, the templates MC3 and MC3-inverted in <ref>.

* *-inverted-symbol: Same as above, except the answer option ordering is changed. Compare, for example, the templates [MC3, MC3-inverted, and MC3-inverted-symbol] or [MC1-numeric, MC1-numeric-inverted, and MC1-numeric-inverted-symbol] in <ref>.

* *-special: These are the same templates as specified by the prompt ID prefixes, but with the answer options replaced by special symbols. The goal is to examine whether standard choice tokens like “yes”, “no”, “A”, “B”, etc. are treated differently from special symbols. We consider two sets of special symbols: one set with the symbols “” and “” (for binary choices) and another set with emojis , ,  for multi-choice questions.

§ ADDITIONAL RESULTS

§.§ Accuracy

<ref> show the classification fine-tuning results for Llama-2 7B and Mistral 7B on all Berkeley D-Lab hatespeech subsets (gender, race, religion, sexuality).
<ref> shows the results for UTK-Face gender and age classification for ViT-Base and Swin-v2-Large.

In all of these figures, we present the subgroup F1 score, accuracy, demographic parity difference (DPD), and equal opportunity difference (EOD) for each subset of the dataset. In the case of UTK-Face age classification, we only present the subgroup accuracy, as F1, DPD, and EOD are not directly applicable.

The results are consistent with the main results described in <ref>:

* By worst group performance, best-worst group performance spread, demographic parity difference (DPD), and equal opportunity difference (EOD), Llama-2 7B and Mistral 7B exhibit similar fairness performance across the different subsets.

* In most cases, LoRA does not worsen either the DPD or the EOD.

* The fairness assessment of the fine-tuning methods can be sensitive to the choice of the metrics.

Classification fine-tuning results for Llama-2 7B and Mistral 7B on the Berkeley D-Lab hatespeech gender subset. Rows from top to bottom: model Llama-2 7B to Mistral 7B. Columns from left to right: subgroup F1 score, accuracy, DPD, and EOD. See <ref> and <ref> for more details.

Classification fine-tuning results for Llama-2 7B and Mistral 7B on the Berkeley D-Lab hatespeech race subset. Rows from top to bottom: model Llama-2 7B to Mistral 7B. Columns from left to right: subgroup F1 score, accuracy, DPD, and EOD. See <ref> and <ref> for more details.

Classification fine-tuning results for Llama-2 7B and Mistral 7B on the Berkeley D-Lab hatespeech religion subset. Rows from top to bottom: model Llama-2 7B to Mistral 7B. Columns from left to right: subgroup F1 score, accuracy, DPD, and EOD. See <ref> and <ref> for more details.

Classification fine-tuning results for Llama-2 7B and Mistral 7B on the Berkeley D-Lab hatespeech sexuality subset. Rows from top to bottom: model Llama-2 7B to Mistral 7B. Columns from left to right: subgroup F1 score, accuracy, DPD, and EOD. See <ref> and <ref> for more details.

Classification fine-tuning results for ViT-Base and Swin-v2-Large on UTK-Face (gender and age classification).
Top row: ViT-Base on gender classification; metrics are subgroup F1 score, accuracy, DPD, and EOD.
Middle row: Swin-v2-Large on gender classification with the same metrics.
Bottom row: Subgroup accuracy of ViT-Base and Swin-v2-Large on age classification.
See <ref> and <ref> for more details.

§.§ Calibration

<ref> show the calibration results (i.e., confidence diagrams and reliability diagrams) for Llama-2 7B and Mistral 7B across all Berkeley D-Lab hatespeech subsets, and for Mistral 7B's highest ECE subgroups within each subset.
<ref> shows the calibration results for UTK-Face gender classification for ViT-Base and SwinV2-Large.

The results are consistent with the main results described in <ref>:

* LoRA and full fine-tuning show comparable and reasonable calibration levels. No significant differences are observed between the two fine-tuning methods.

* LoRA tends towards overconfidence, with predictions clustering at extreme scales (bins 0.0-0.1 and 0.9-1.0), potentially affecting reliability across subgroups.

Confidence histograms (top) and reliability diagrams (bottom) for Llama-2 7B on D-Lab gender (left), Mistral 7B on D-Lab gender (middle), and Mistral 7B on subgroups with highest ECE within D-Lab gender (right).
See <ref> and <ref> for more details.

Confidence histograms (top) and reliability diagrams (bottom) for Llama-2 7B on D-Lab race (left), Mistral 7B on D-Lab race (middle), and Mistral 7B on subgroups with highest ECE within D-Lab race (right).
See <ref> and <ref> for more details.

Confidence histograms (top) and reliability diagrams (bottom) for Llama-2 7B on D-Lab religion (left), Mistral 7B on D-Lab religion (middle), and Mistral 7B on subgroups with highest ECE within D-Lab religion (right).
See <ref> and <ref> for more details.

Confidence histograms (top) and reliability diagrams (bottom) for Llama-2 7B on D-Lab sexuality (left), Mistral 7B on D-Lab sexuality (middle), and Mistral 7B on subgroups with highest ECE within D-Lab sexuality (right).
See <ref> and <ref> for more details.

Confidence histograms (top) and reliability diagrams (bottom) for vit-base on UTK-Face gender classification (left), swinv2-large on UTK-Face gender classification (middle), and swinv2-large on subgroups with highest ECE within different races (right).
See <ref> and <ref> for more details.

§.§ Resistance to Membership Inference Attacks (MIA)

<ref> show Likelihood Ratio Attack (LiRA) on Llama-2 7B and Mistral 7B for membership inference on the D-Lab religion subset.
<ref> show LiRA on ViT-Base and Swin-v2-Large for membership inference on UTK-Face gender classification.

We also present the LOSS attack results in <ref>. See <ref> for background on the definitions and implementations of membership inference attacks.

Likelihood Ratio Attack (LiRA) on Mistral 7B for membership inference on the D-Lab religion subset. Results indicate that the full fine-tuned model is slightly more resistant to membership inference than LoRA fine-tuning.

Likelihood Ratio Attack (LiRA) on ViT-Base for membership inference on UTK-Face gender classification. Results indicate that the LoRA fine-tuned model is roughly equally resistant to membership inference compared to full fine-tuning.

LOSS attack on Llama-2 7B for membership inference on the D-Lab religion subset. Results indicate that the LoRA fine-tuned model is slightly more resistant to membership inference compared to full fine-tuning.

LOSS attack on Mistral 7B for membership inference on the D-Lab religion subset. Results indicate that the LoRA fine-tuned model is roughly equally resistant to membership inference compared to full fine-tuning.

LOSS attack on ViT-Base for membership inference on UTK-Face gender classification. Results indicate that the LoRA fine-tuned model is roughly equally resistant to membership inference compared to full fine-tuning.

LOSS attack on Swin-v2-Large for membership inference on UTK-Face gender classification. Results indicate that the LoRA fine-tuned model is roughly equally resistant to membership inference compared to full fine-tuning.

§.§ Effect of LoRA Rank

Recall from <ref> that we evaluate the effect of LoRA ranks on the fairness of the fine-tuned models. <ref> presents the results for Llama-2 7B on all Berkeley D-Lab hatespeech subsets, <ref> presents the results for ViT-Base on UTK-Face gender classification, and <ref> presents gender stereotype results for Llama-2 7B on Turkish to English machine translation. Following the main discussions in <ref>, we found that the choice of rank tends to have little effect on the fairness of the fine-tuned models.

Effect of LoRA ranks on Llama-2 7B on all Berkeley D-Lab hatespeech subsets (Gender, Race, Religion, Sexuality). Rows from top to bottom: D-Lab subsets. Columns from left to right: subgroup accuracy, DPD, and EOD across rank values from 0 to 4096.

Effect of LoRA ranks on ViT-Base on UTK-Face gender classification.
Left to right: subgroup accuracy, DPD, and EOD across rank values from 0 to 768.

Effect of LoRA ranks on Llama-2 7B on Turkish to English machine translation.
Left to right: gender stereotype results on the gender-unbiased dataset and pro-stereotypical gendered dataset across rank values from 1 to 4096.

§.§ Effect of Subgroup Size

<ref> illustrate the effect of increasing group size on utility (e.g., accuracy in the plots) and fairness. We observe that these metrics are not solely dependent on the size of the group:

* On the D-Lab dataset with language models, accuracy does not consistently increase or decrease with the size of the groups. On the UTK-Face gender dataset with vision models, while there is a general trend of increasing accuracy with larger group sizes, the practical impact of group size on accuracy is limited, since the absolute difference in accuracy across these sizes is marginal.

* The demographic parity difference and equalized odds difference both exhibit fluctuations across different group sizes without showing a clear trend that correlates with group size. The equalized odds difference tends to decrease as the group size increases in the Llama-2 7B model, but this trend is less clear in the Mistral 7B model (varies in model architectures).

Accuracy and fairness (DPD, EOD) on D-Lab religion subset with subgroups sorted by size.

Accuracy and fairness (DPD, EOD) on UTK-Face gender classification with subgroups sorted by size.

§.§ Gender Bias

Yelp Review Language Modeling

Recall <ref> for the task setup, <ref> for the prompt templates used for the evaluations, <ref> for results on cloze completions, and <ref> that we also explore the effects of model token bias on the generative evaluations.

We present the results for Llama-2 7B and Mistral 7B on the subsampled Yelp review dataset. For the two models respectively (tables are in <ref> and <ref>):

* <ref> and <ref> show the results for YN* prompts.

* <ref> and <ref> show the results for MC* prompts.

* <ref> and <ref> show the results for *-special prompts.

* <ref> and <ref> show the results for cloze prompts.

In these tables, the text “ratio_{}" in the metric field measures the percentage of the 50k Yelp reviews, given the specific prompt template, the model selected that choice. There is a slight difference between the metrics for YN* prompts and MC* prompts. For YN* prompts, the metric “ratio_{gender}_{choice}" means the ratio model answers “{choice}" when asking specifically whether the reviewer is “{gender}". For MC* prompts, the metric “ratio_{token}" means the ratio of the reviews the model selects “{token}". The value is bold if it is either greater than 99% or less than 1%, showing a strong preference towards one answer.

The results are consistent and the model token bias is clear:

* LoRA does not exhibit more bias than full-model fine-tuning.

* On yes/no and multiple choice QA, both Llama-2 7B and Mistral 7B models exhibit strong biases, often preferring specific responses like the token "A" regardless of context (where selection rates for that token exceed 99% as indicated in the tables), which complicates fairness in evaluations.

* Attempts to reduce these biases by changing prompts or varying tokens used in the multiple-choice options have been ineffective, suggesting such biases are ingrained and not easily correctable.

Turkish to English Machine Translation
Recall <ref> for the task setup and <ref> for results of gender stereotypes. In addition, <ref> shows results on Llama-2 7B and Mistral 7B for gender predispositions. Pro-stereotypical or anti-stereotypical sentences can be broken down into male-predisposed and female-predisposed sentences depending on the context. For example, the sentence “the developer argued with the designer because [he] did not like the design" is pro-stereotypical and male-predisposed since people usually fill in the male-gendered pronoun given the context, while the sentence “the developer argued with the designer because her idea cannot be implemented" is pro-stereotypical and female-predisposed.

The results are consistent with the main results described in <ref>:

* On the gender-unbiased training set, both Llama-2 7B and Mistral 7B show fair gender-unbiased behavior, regardless of the fine-tuning methods.

* On the pro-stereotypical gendered training set, the bias depends heavily on the model architecture: Mistral 7B shows significant gender bias regardless of the fine-tuning method, while Llama-2 7B shows less gender bias when fine-tuned using LoRA.

Model gender-predispositions for Llama-2 7B and Mistral 7B on both balanced and pro-stereotypical gendered training sets. The training and evaluation sets are the same as <ref> and only the metric differs. The x-axis label (gender1)_predispose_(gender2): (gender1) refers to the gender that people usually predispose the pronoun to be given the context in the English sentence. (gender2) refers to the gender pronoun that a model-translated English sentence contains. E.g., “the developer argued with the designer because she did not like the design” (where “she” refers to “developer”) is a (male)_predispose_(female) sentence. Ideally, the model should translate similar ratios (50%/50%) of male_predispose_female and male_predispose_male sentences, the same for female_predispose_female and female_predispose_male sentences.

§.§.§ Additional Results for Yelp Review Language Modeling: Yes/No and Multiple Choice

Evaluting Llama-2 7B fine-tuned on Yelp reviews on YN* prompts with “yes” and “no” as answer options. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.

Evaluting Mistral 7B fine-tuned on Yelp reviews on YN* prompts with “yes” and “no” as answer options. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.

Evaluting Llama-2 7B fine-tuned on Yelp reviews on MC* prompts with multiple choices as answer options where symbols are sets of “ABCD” or “1234”. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.

Evaluting Mistral 7B fine-tuned on Yelp reviews on MC* prompts with multiple choices as answer options where symbols are sets of “ABCD” or “1234”. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.

Evaluting Llama-2 7B fine-tuned on Yelp reviews on *-special prompts with special symbols as answer options. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.

Evaluting Mistral 7B fine-tuned on Yelp reviews on *-special prompts with special symbols as answer options. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.

§.§.§ Additional Results for Yelp Review Language Modeling: Cloze Completions

Evaluting Llama-2 7B fine-tuned on Yelp reviews on cloze prompts with “male" or “female" as answer options. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.

Evaluting Mistral 7B fine-tuned on Yelp reviews on cloze prompts with “male" or “female" as answer options. Bold values denote strong preference ($>99$% or $<1$%) towards an answer. See <ref> for task set up, <ref> for prompt templates, and <ref> and <ref> for result analysis.