# Inference-Time Selective Debiasing

Gleb Kuzmin3,5 Nemeesh Yadav4 Ivan Smirnov5
Timothy Baldwin1,2 Artem Shelmanov1
1MBZUAI 2The University of Melbourne 3AIRI
4IIIT Delhi 5FRC CSC RAS
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
{timothy.baldwin<EMAIL_ADDRESS>
###### Abstract

We propose selective debiasing – an inference-time safety mechanism that aims
to increase the overall quality of models in terms of prediction performance
and fairness in the situation when re-training a model is prohibitive. The
method is inspired by selective prediction, where some predictions that are
considered low quality are discarded at inference time. In our approach, we
identify the potentially biased model predictions and, instead of discarding
them, we debias them using LEACE – a post-processing debiasing method. To
select problematic predictions, we propose a bias quantification approach
based on KL divergence, which achieves better results than standard UQ
methods. Experiments with text classification datasets demonstrate that
selective debiasing helps to close the performance gap between post-processing
methods and at-training and pre-processing debiasing techniques.

Inference-Time Selective Debiasing

Gleb Kuzmin3,5 Nemeesh Yadav4 Ivan Smirnov5 Timothy Baldwin1,2 Artem
Shelmanov1 1MBZUAI 2The University of Melbourne 3AIRI 4IIIT Delhi 5FRC CSC RAS
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>{timothy.baldwin,
<EMAIL_ADDRESS>

##  1 Introduction

Fairness is an important safety characteristic of a machine learning (ML)
model, representing the model’s ability to classify instances without
discrimination based on various sensitive attributes, such as race, gender,
and age Blodgett et al. (2020). In recent years, numerous works have
investigated and promoted fairness, and a variety of fairness definitions have
been proposed Blodgett et al. (2020); Han et al. (2022b). One prominent type
of fairness is group fairness, also known as the equal opportunity criterion,
which reflects the inequality of opportunities across different groups Han et
al. (2022a). The inequality in the model predictions usually comes from
inadequate or biased training data, and to address this problem and achieve
better fairness, researchers have proposed various debiasing techniques Li et
al. (2018); Han et al. (2021, 2022a); Belrose et al. (2023). The majority of
these techniques assume that one has access to the complete training data and
the ability to retrain the model from scratch using some special loss function
or reweighting the training instances. However, there are many situations when
this assumption does not hold. There is a need for inference-time safety
mechanisms that protect users from inadequate model behavior.

Inference time safety mechanisms are primarily associated with uncertainty
quantification (UQ) techniques Gal and Ghahramani (2016) and selective
classification Geifman and El-Yaniv (2017). Selective classification aims to
enhance the reliability of ML-based applications by abstaining from unreliable
predictions with high uncertainty. We suggest that the same approach could
equally be applied to increase fairness.

In this work, we propose an inference-time safety mechanism that aims to
increase the overall quality of models in terms of prediction performance and
fairness in situations when model re-training is prohibitive. We call this
approach selective debiasing. Instead of rejecting predictions of selected
instances as in selective classification, we apply to them inference-time
debiasing using post-processing debiasing techniques. We note that applying
debiasing to all model decisions might reduce its overall performance.
However, the selected predictions often have a high potential to be of low
quality or carry some bias. Therefore, modifying these predictions is less
risky in terms of worsening the situation, while still holding the potential
to correct errors. To the best of our knowledge, this style of approach is
novel to the NLP community.

Our main contributions are as follows:

  * •

We propose selective debiasing, an inference-time safety mechanism that aims
to improve both the performance and fairness of model predictions by applying
a post-processing debiasing method to only a selected subset of predictions.

  * •

We suggest a scoring criterion that aims to select the most unreliable and
biased predictions. Experiments demonstrate that this scoring criterion is
generally better than UQ techniques in selective debiasing.

##  2 Background

Debiasing techniques can be categorized into three groups: at-training, pre-
processing, and post-processing Han et al. (2022b). The first two groups
require retraining the model from scratch and access to the whole training
set. They also cannot be selectively applied to a subset of predictions. Post-
processing techniques do not involve changes to the model itself, can be
trained on a subset of data, and can be applied to predictions selectively.
However, their performance is usually worse.

#### At-training and pre-processing methods.

Adversarial training (Adv) Li et al. (2018) aims to solve a minimax game
between minimizing the loss for the primary task and maximizing the loss for
predicting the protected attribute. Diverse adversaries (DAdv) Han et al.
(2021) extends Adv by using an ensemble of diverse discriminators instead of
just one. Balanced Training with Equal Opportunity (BTEO) Han et al. (2022a)
is a dataset balancing technique. It aims to minimize the TPR gap between two
protected groups. Balanced Training with Joint balance (BTJ) Lahoti et al.
(2020) aims to improve the worst-case performance over all unobserved
protected groups by focusing on the computationally identifiable regions of
error.

#### Post-processing methods.

There are two well-known approaches to post-processing debiasing: Iterative
Null-space Projection (INLP) Ravfogel et al. (2020) and LEAst-squares Concept
Erasure (LEACE) Belrose et al. (2023).

INLP proposes an iterative method that involves finding an orthogonal
projection of a linear classifier matrix, which is initially learned to
predict protected attributes from representations (e.g. hidden states of the
standard model). This orthogonal projection is then iteratively used to remove
all relevant information from these representations, which was used by the
classifier to predict protected attributes.

LEACE is a concept erasure technique that renders representations impervious
to the prediction of a specific concept while minimizing change to the
original representations. LEACE first whitens the data by equalizing the
variance in all directions. Then the data is orthogonally projected onto the
relevant subspace, and finally, the data is unwhitened with the same
covariance matrix before subtracting from the original representation.

##  3 Proposed Methods

#### Selective debiasing.

Selective classification is a well-known human-in-the-loop approach that makes
predictions for some instances and dispatches others to human experts for
labeling. Instead of rejecting instances completely as in selective
classification, we suggest applying debiasing to selected model predictions
during the inference time. This is different from the standard post-processing
debiasing methods since we suggest changing the predictions for only some
instances. In particular, we identify the potentially most biased instances by
a scoring criterion and replace the original model’s prediction with
predictions debiased using a post-processing method. This does not introduce
much computational overhead, as applying a post-processing method to a model
is a matter of one or two matrix multiplications. Finally, we obtain
predictions for the model that represent a mixture of original $p^{i}$ and
debiased predictions $q^{i}$:

$r^{i}=\begin{cases}p^{i},\text{ if $i\in S_{h}$}\\\ q^{i},\text{ if $i\notin
S_{h}$},\end{cases}$ (1)

where $S_{h}$ is the set of samples with high bias scores. Membership in
$S_{h}$ is determined by a threshold criterion (e.g. if the bias score is in
the top-$N$ percent of highest scores). Such an approach allows flexible
adjustment of the accuracy–fairness trade-off.

#### Bias quantification method.

For selective debiasing, we need a good instance-level bias quantification
method. Selective classification is usually based on UQ methods. However,
uncertainty on its own is not reflective of group fairness; it simply tries to
eliminate potentially erroneous predictions. Figure 1 presents a motivational
example. It shows the rejection plots for oracle rejection strategies in
selective classification for both accuracy and fairness (see the exact
definition of fairness in Appendix E). We can see that the fairness oracle
outperforms the UQ oracle in terms of fairness while keeping the same
performance in terms of accuracy. These results illustrate that to improve
fairness without penalty to accuracy, it is necessary to change the order of
instances being eliminated, i.e. we need a different selection criterion.

Consider a model that is trained to label instances with classes $c\in C$. To
quantify how biased a prediction is for a given instance, we suggest using the
Kullback-Leibler divergence (KL) Kullback and Leibler (1951) between the
standard model prediction $p_{c}^{i}$ and the debiased prediction $q_{c}^{i}$
via LEACE for the $i$-th instance:

$U_{KL}^{i}=\sum_{c\in
C}p_{c}^{i}\log\left(\frac{p_{c}^{i}}{q_{c}^{i}}\right).$ (2)

The key idea behind the usage of KL divergence is that the distance quantifies
the difference in predictions between the standard and debiased model. The
bigger the difference, the bigger the amount of information about the
protected attribute removed from the original representation of the instance.
The proposed approach allows for improvement over the baseline debiasing
method by updating only the most biased instances. It is especially beneficial
in the case of LEACE because it prevents excessive change in less-biased
instances, which could lead to a drop in both accuracy and fairness. Moreover,
selective debiasing adds a mechanism for tuning the accuracy–fairness trade-
off for LEACE. This proposed method also could be applied in combination with
INLP.

##  4 Experiments

Figure 1: Rejection results for fairness and accuracy with oracle scores on a
synthetic dataset with LogReg model; the FR-AUC and Acc-AUC are the areas
under fairness– and accuracy–rejection curves correspondingly. The details are
presented in Appendix F.

###  4.1 Experimental Setup

#### Datasets.

For our experiments, we use two English text classification datasets that in
addition to target variables, provide explicit protected attributes. The first
is MOJI Blodgett et al. (2016), a dataset for sentiment analysis with a binary
class (happy and sad) and a binary protected attribute, which corresponds to
the author’s ethnicity (African American English (AAE) vs. Standard American
English (SAE)). The second is a version of the widely used BIOS dataset De-
Arteaga et al. (2019) for occupation classification with binary gender as the
protected attribute. BIOS-2 Subramanian et al. (2021) is a two-class subsample
of the original BIOS dataset with a highly-skewed joint distribution of
classes and protected attribute values. As it has been shown to be beneficial
to report results for both “balanced” and “imbalanced” versions of datasets
Kuzmin et al. (2023), we conduct experiments on both versions. Detailed
information and statistics of the datasets are presented in Appendix D.

#### Metrics.

We employ several metrics to quantify the predictive performance of the model
and also its fairness. To estimate the model’s performance, we use accuracy.
For fairness, we consider the widely-used equal opportunity criterion Hardt et
al. (2016); Han et al. (2022a, b). We also use two aggregated metrics to
obtain the overall model performance in terms of accuracy and fairness. The
first one is the distance to the optimal point (DTO) Han et al. (2021):

$\text{DTO}=\sqrt{(1-\text{Accuracy})^{2}+(1-\text{Fairness})^{2}}.$ (3)

The second one is the Fairness F-score (FF) – a smoothed minimum of accuracy
and fairness:

$\text{FF-
score}=\frac{2\cdot\text{Accuracy}\cdot\text{Fairness}}{\text{Accuracy}+\text{Fairness}}.$
(4)

Details of the equal opportunity fairness calculation are presented in
Appendix E.

Dataset | Metric | Standard | Adv | DAdv | BTEO | BTJ |  | LEACE-
---
last
| LEACE-
---
last+SR,
opt. perc.
| LEACE-
---
last+KL,
opt. perc.
| LEACE-
---
cls
| LEACE-
---
cls+SR,
opt. perc.
| LEACE-
---
cls+KL,
opt. perc.
INLP |  | INLP+SR,
---
opt. perc.
| INLP+KL,
---
opt. perc.
MOJI imbalanced | Fairness $\uparrow$ | 61.8$\pm$0.7 | 73.7$\pm$0.6 | 73.4$\pm$0.4 | 75.2$\pm$0.6 | 74.8$\pm$0.6 | 75.8$\pm$2.6 | 68.6$\pm$1.4 | 75.7$\pm$0.8 | 75.2$\pm$3.0 | 68.4$\pm$1.1 | 77.2$\pm$0.7 | 77.3$\pm$7.3 | 63.5$\pm$1.1 | 70.5$\pm$2.4
Accuracy $\uparrow$ | 79.1$\pm$0.7 | 72.0$\pm$0.7 | 72.4$\pm$0.5 | 73.6$\pm$0.6 | 73.2$\pm$0.4 | 68.3$\pm$2.6 | 77.6$\pm$0.9 | 72.7$\pm$1.2 | 66.8$\pm$3.0 | 77.6$\pm$1.0 | 71.8$\pm$1.2 | 68.4$\pm$6.8 | 78.0$\pm$0.6 | 73.5$\pm$1.4
DTO $\downarrow$ | 43.6$\pm$0.6 | 38.4$\pm$0.5 | 38.3$\pm$0.4 | 36.2$\pm$0.1 | 36.7$\pm$0.4 | 39.9$\pm$3.6 | 38.6$\pm$0.7 | 36.6$\pm$0.5 | 41.4$\pm$4.1 | 38.8$\pm$0.6 | 36.2$\pm$1.2 | 40.0$\pm$3.5 | 42.6$\pm$0.8 | 39.7$\pm$1.8
FF-score $\uparrow$ | 69.4$\pm$0.4 | 72.8$\pm$0.4 | 72.9$\pm$0.3 | 74.4$\pm$0.1 | 74.0$\pm$0.3 | 71.8$\pm$2.6 | 72.8$\pm$0.5 | 74.1$\pm$0.3 | 70.8$\pm$3.0 | 72.7$\pm$0.4 | 74.4$\pm$0.8 | 71.9$\pm$2.2 | 70.0$\pm$0.6 | 71.9$\pm$1.3
MOJI balanced | Fairness $\uparrow$ | 69.5$\pm$0.2 | 83.8$\pm$0.8 | 84.7$\pm$1.5 | 85.5$\pm$0.5 | 85.6$\pm$0.6 | 79.7$\pm$3.9 | 77.1$\pm$0.9 | 86.6$\pm$0.5 | 77.6$\pm$4.2 | 77.0$\pm$0.8 | 87.5$\pm$0.5 | 85.8$\pm$8.3 | 71.7$\pm$0.6 | 77.9$\pm$4.4
Accuracy $\uparrow$ | 71.9$\pm$0.4 | 74.0$\pm$0.4 | 74.1$\pm$0.6 | 74.8$\pm$0.3 | 74.5$\pm$0.4 | 73.6$\pm$0.8 | 74.0$\pm$0.3 | 74.0$\pm$0.2 | 73.0$\pm$1.2 | 74.0$\pm$0.4 | 73.7$\pm$0.5 | 63.0$\pm$6.9 | 71.9$\pm$0.4 | 68.5$\pm$2.0
DTO $\downarrow$ | 41.5$\pm$0.4 | 30.7$\pm$0.7 | 30.1$\pm$0.7 | 29.0$\pm$0.1 | 29.3$\pm$0.4 | 33.4$\pm$3.0 | 34.7$\pm$0.7 | 29.3$\pm$0.3 | 35.2$\pm$3.7 | 34.7$\pm$0.6 | 29.1$\pm$0.6 | 40.9$\pm$4.4 | 39.9$\pm$0.6 | 38.8$\pm$1.2
FF-score $\uparrow$ | 70.7$\pm$0.3 | 78.6$\pm$0.5 | 79.1$\pm$0.6 | 79.8$\pm$0.1 | 79.6$\pm$0.3 | 76.5$\pm$2.2 | 75.5$\pm$0.5 | 79.8$\pm$0.2 | 75.2$\pm$2.6 | 75.5$\pm$0.4 | 80.0$\pm$0.4 | 71.9$\pm$3.2 | 71.8$\pm$0.4 | 72.8$\pm$1.0
BIOS-2 imbalanced | Fairness $\uparrow$ | 90.4$\pm$0.8 | 97.2$\pm$0.8 | 96.4$\pm$0.4 | 95.8$\pm$1.0 | 96.6$\pm$0.8 | 92.8$\pm$9.3 | 93.0$\pm$2.3 | 94.5$\pm$4.4 | 77.3$\pm$6.5 | 94.8$\pm$2.3 | 96.7$\pm$0.9 | 92.0$\pm$1.6 | 91.7$\pm$1.4 | 91.9$\pm$1.7
Accuracy $\uparrow$ | 96.7$\pm$0.1 | 94.8$\pm$0.4 | 95.0$\pm$0.3 | 95.2$\pm$0.3 | 95.0$\pm$0.5 | 60.5$\pm$3.6 | 94.6$\pm$0.2 | 92.0$\pm$0.4 | 64.0$\pm$5.5 | 94.6$\pm$0.1 | 93.2$\pm$0.3 | 95.8$\pm$0.6 | 96.2$\pm$0.3 | 95.8$\pm$0.6
DTO $\downarrow$ | 10.1$\pm$0.7 | 5.9$\pm$0.2 | 6.2$\pm$0.2 | 6.5$\pm$0.6 | 6.1$\pm$0.3 | 41.3$\pm$2.1 | 9.0$\pm$1.7 | 10.3$\pm$2.8 | 43.4$\pm$2.4 | 7.7$\pm$1.7 | 7.6$\pm$0.5 | 9.1$\pm$1.3 | 9.1$\pm$1.3 | 9.2$\pm$1.4
FF-score $\uparrow$ | 93.5$\pm$0.4 | 96.0$\pm$0.2 | 95.7$\pm$0.1 | 95.5$\pm$0.4 | 95.8$\pm$0.2 | 72.8$\pm$2.3 | 93.8$\pm$1.2 | 93.2$\pm$2.3 | 69.6$\pm$1.7 | 94.7$\pm$1.2 | 94.9$\pm$0.4 | 93.8$\pm$0.7 | 93.9$\pm$0.7 | 93.8$\pm$0.8
BIOS-2 balanced | Fairness $\uparrow$ | 89.7$\pm$0.6 | 97.8$\pm$0.8 | 98.0$\pm$0.8 | 95.9$\pm$0.8 | 96.4$\pm$0.3 | 90.6$\pm$9.8 | 93.7$\pm$2.6 | 94.6$\pm$4.2 | 74.8$\pm$9.2 | 96.6$\pm$1.8 | 97.5$\pm$0.9 | 91.8$\pm$2.0 | 91.6$\pm$1.9 | 91.3$\pm$1.9
Accuracy $\uparrow$ | 92.4$\pm$0.3 | 91.9$\pm$0.6 | 91.9$\pm$1.5 | 92.6$\pm$0.5 | 92.9$\pm$0.6 | 49.9$\pm$9.4 | 90.9$\pm$1.3 | 89.3$\pm$1.8 | 63.8$\pm$10.1 | 91.9$\pm$0.7 | 90.6$\pm$1.3 | 91.9$\pm$1.2 | 92.2$\pm$1.0 | 91.9$\pm$1.2
DTO $\downarrow$ | 12.8$\pm$0.6 | 8.5$\pm$0.4 | 8.4$\pm$1.4 | 8.5$\pm$0.2 | 8.0$\pm$0.6 | 52.4$\pm$6.0 | 11.1$\pm$2.4 | 12.4$\pm$3.4 | 46.0$\pm$4.3 | 8.9$\pm$1.4 | 9.7$\pm$1.5 | 11.7$\pm$1.7 | 11.5$\pm$1.7 | 12.0$\pm$1.6
FF-score $\uparrow$ | 91.1$\pm$0.4 | 94.7$\pm$0.1 | 94.9$\pm$0.7 | 94.2$\pm$0.2 | 94.6$\pm$0.3 | 63.0$\pm$4.6 | 92.3$\pm$1.9 | 91.9$\pm$2.9 | 67.5$\pm$3.0 | 94.2$\pm$1.2 | 93.9$\pm$1.1 | 91.8$\pm$1.1 | 91.9$\pm$1.2 | 91.6$\pm$1.1

Table 1:  Comparison of debiasing methods and selective debiasing. The best
results in the group are in bold, and the best results overall are underlined.
The results averaged over 5 random seeds.

#### Models.

For the BIOS-2 dataset, we use BERT (“bert-base-cased”) Devlin et al. (2019).
For the MOJI dataset, we use the domain-specific BERTweet model Nguyen et al.
(2020) given that the data is from social media sources. For both models, we
added a three-layer MLP as a classification head, following Han et al.
(2022b). Model hyperparameters are described in Appendix A.

#### Baselines.

We compare the proposed selective debiasing approach to inference-time
debiasing of all predictions using LEACE and INLP, as well as to training-time
debiasing techniques. We also compare the proposed KL-based bias
quantification score with a UQ baseline: Softmax Response (SR: Geifman and El-
Yaniv (2017)), calculated as $U_{SR}=1-\max_{c\in C}p_{c}$.

###  4.2 Results

Table 1 compares fairness, accuracy, DTO, and FF-score for various at-training
and pre-processing debiasing methods, post-processing debiasing methods, and
for selective debiasing based on LEACE with SR and the proposed KL-based bias
quantification score. Here, we show results only for the optimal selection
percentage. The full results with various selection percentages are presented
in Appendix B. The optimal percentage was chosen over the first 15% of the
validation set. “LEACE-last” represents LEACE applied to the outputs of the
last hidden layer of the classifier, while “LEACE-cls” is LEACE applied to
each linear layer of the classification head of the model. For comparison, we
also provide the results with selective debiasing using INLP. The post-
processing methods were trained on only 20% of the training set.

In the majority of cases, the best results are unsurprisingly achieved by at-
training and pre-processing debiasing techniques, as these methods retrain the
models from scratch on the full training data. Nevertheless, the proposed
selective debiasing approach based on LEACE substantially enhances the results
of inference-time debiasing using post-processing techniques in terms of
metrics that take into account both fairness and performance: FF-score and
DTO. It is also competitive with at-training and pre-processing techniques.
For LEACE-cls with KL selection, selective debiasing even outperforms these
methods on MOJI-balanced. The results in Tables 10, 11 and 12 also show that
selective debiasing consistently outperforms standard inference-time debiasing
in terms of FF-score across various percentages.

LEACE-cls generally achieves better fairness than LEACE-last and slightly
better joint fairness–performance, in terms of DTO and FF-score. INLP-based
selective debiasing improves the FF-score only on MOJI-balanced, while on
other datasets, it is consistent with the base inference-time debiasing
method. INLP-based approaches overall fall behind the corresponding LEACE-
based techniques.

When comparing the results of the proposed bias quantification method based on
the KL distance with SR, we can see that our method notably outperforms SR on
MOJI datasets and is on par with SR on BIOS-2. We further explore other
distance-based bias quantification methods (Euclidean and cosine distances),
which are presented in Appendix C. Results in Tables 10, 11 and 12 show that
in most cases, selection by KL works comparably or better than other distance-
based measures. Moreover, KL scores are easier to compute than distance-based
scores.

##  5 Conclusion and Future Work

We proposed selective debiasing – a new simple inference-time safety mechanism
for increasing model performance and fairness. We showed that it is helpful in
the case that re-training a model from scratch for better fairness is
prohibitive or there is no access to full training data. Additionally, for the
selection of problematic predictions, we suggest a bias quantification
approach based on KL distance that achieves better results than the standard
UQ method. The proposed mechanism fills the gap for efficient techniques that
can be applied at inference time and opens the door for safer ML-based
systems. In future work, we aim to investigate a deeper integration between UQ
and debiasing methods.

## Limitations

In this work, we considered only group fairness (equal opportunity criterion),
where there exist many other fairness definitions. However, this research is
focused particularly on group fairness, and the equal opportunity criterion is
the metric of choice in previous work on the same datasets. During all
experiments, we assume that we have access to the protected attributes, which
is not always the case. But this is a common assumption for any work on
debiasing; moreover, it is necessary for the calculation of the fairness
metric. Finally, all of the experiments were conducted on the English
language, but the used methods are language-independent, so we do not expect
significant differences in results for other languages.

## Ethical Considerations

In this work, we consider group fairness and instance-level bias
quantification. We used only publicly available datasets and models, and only
for the intended use. In our research, we used protected attributes to apply
debiasing methods and to compute metrics; however, this is necessary for all
debiasing methods. To avoid possible harm, we used only attributes that users
self-disclosed for the experiments.

## References

  * Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023.  Leace: Perfect linear concept erasure in closed form.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 66044–66063. Curran Associates, Inc.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476, Online. Association for Computational Linguistics.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016.  Demographic dialectal variation in social media: A case study of African-American English.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1119–1130, Austin, Texas. Association for Computational Linguistics.
  * De-Arteaga et al. (2019) Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  In _proceedings of the Conference on Fairness, Accountability, and Transparency_ , pages 120–128.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR.
  * Geifman and El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv. 2017.  Selective classification for deep neural networks.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 4878–4887.
  * Han et al. (2021) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021.  Diverse adversaries for mitigating bias in training.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2760–2765, Online. Association for Computational Linguistics.
  * Han et al. (2022a) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2022a.  Balancing out bias: Achieving fairness through balanced training.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11335–11350, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Han et al. (2022b) Xudong Han, Aili Shen, Yitong Li, Lea Frermann, Timothy Baldwin, and Trevor Cohn. 2022b.  FairLib: A unified framework for assessing and improving fairness.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 60–71, Abu Dhabi, UAE. Association for Computational Linguistics.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc.
  * Kullback and Leibler (1951) S. Kullback and R. A. Leibler. 1951.  On Information and Sufficiency.  _The Annals of Mathematical Statistics_ , 22(1):79 – 86.
  * Kuzmin et al. (2023) Gleb Kuzmin, Artem Vazhentsev, Artem Shelmanov, Xudong Han, Simon Suster, Maxim Panov, Alexander Panchenko, and Timothy Baldwin. 2023.  Uncertainty estimation for debiased models: Does fairness hurt reliability? In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 744–770, Nusa Dua, Bali. Association for Computational Linguistics.
  * Lahoti et al. (2020) Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. 2020.  Fairness without demographics through adversarially reweighted learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 728–740. Curran Associates, Inc.
  * Li et al. (2018) Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018.  Towards robust and privacy-preserving text representations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 25–30, Melbourne, Australia. Association for Computational Linguistics.
  * Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020.  BERTweet: A pre-trained language model for English tweets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 9–14, Online. Association for Computational Linguistics.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020.  Null it out: Guarding protected attributes by iterative nullspace projection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7237–7256, Online. Association for Computational Linguistics.
  * Subramanian et al. (2021) Shivashankar Subramanian, Xudong Han, Timothy Baldwin, Trevor Cohn, and Lea Frermann. 2021.  Evaluating debiasing techniques for intersectional biases.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2492–2498, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

##  Appendix A Training Setup and Hyperparameters

To find an optimal set of hyperparameters, we conducted a grid search on the
validation set. We used accuracy as an optimization target for standard
models, and DTO for models with debiasing. The grid and optimal parameters for
the standard models are described in Table 2. For each debiasing method, we
tuned the method’s parameters and kept training parameters of the base model –
the grid and optimal values for debiasing methods presented in Table 3. The
training was conducted on a cluster with Nvidia V100 GPUs. An approximate
number of GPU-hours spent during the experiments is presented in Table Table
4.

Dataset |  | Num.
---
Epochs
| Batch
---
Size
| Learning
---
Rate
| Weight
---
Decay
| Dropout
---
Rate
MOJI imbalanced | 20 | 32 | 1e-6 | 0 | 0.1
MOJI balanced | 20 | 32 | 1e-6 | 0 | 0.1
BIOS-2 imbalanced | 20 | 16 | 1e-6 | 0 | 0.1
BIOS-2 balanced | 20 | 32 | 1e-6 | 1e-4 | 0.1

Table 2:  Optimal training hyperparameters for BERTweet on MOJI and BERT on
BIOS-2 for standard model. We use a grid search with the following grid
values: batch size: [16, 32], learning rate: [1e-6, 5e-6, 1e-5, 3e-5, 5e-5],
weight decay: [0, 1e-4]. The number of epochs is determined by early-stopping.

Dataset |  | Debiasing
---
Method
| Adv.
---
Lambda
| Adv. Diverse
---
Lambda
| INLP
---
by Class
| INLP Discriminator
---
Reweighting
Moji (imbalanced) | Adv | 1.0 | - | - | -
DAdv | 1.0 | 1.0 | - | -
INLP | - | - | False | True
Moji (balanced) | Adv | 1.0 | - | - | -
DAdv | 1.0 | 1.0 | - | -
INLP | - | - | False | False
BIOS-2 (imbalanced) | Adv | 1.0 | - | - | -
DAdv | 1.0 | 1.0 | - | -
INLP | - | - | False | True
BIOS-2 (balanced) | Adv | 1.0 | - | - | -
DAdv | 1.0 | 1.0 | - | -
INLP | - | - | False | True

Table 3:  Optimal debiasing hyperparameters for BERTweet on MOJI and BERT on BIOS-2 for various debiasing methods. The base training parameters are the same as for the vanilla model. We use a grid search with the following grid values: Adv. Lambda/Adv. Diverse Lambda: [1e-4, 1e-3, 1e-2, 1e-1, 1, 1e2, 1e3], INLP by Class/INLP Discriminator Reweighting: [False, True]. The remaining parameters for each method used default values from Han et al. (2022b). For DAdv Adv. Lambda/Adv. Diverse Lambda parameters were tuned jointly, as in Han et al. (2022b). Dataset | Model | GPU hours |  | Num. of
---
Params
Moji | BERTweet | 339 | 135m
Bios-2 | BERT | 119 | 110m
Table 4:  Overall computation statistics. GPU hours specify the approximate
number of GPU hours spent for training and evaluating the corresponding model
for all experiments on both imbalanced and balanced sets. The column Num. of
Params contains the number of parameters of a single model.

##  Appendix B Selective Debiasing on Various Percentages

To check how stable the proposed methods are, we compare selective debiasing
results over 5%, 10%, and 15% of selection for random, SR, and KL scores. The
results are presented in Tables 6, 7 and 8. The optimal percentage selected on
the validation set from values from 1% to 15%; results for each dataset-method
pair in Table 9. In general, optimal scores are better or comparable with
results on various percentages, which allows us to use this approach to detect
the optimal percentage of selection. Table 5 shows the performance of
selective debiasing and post-processing debiasing methods trained on a full
training set. As one can see, the performance on the full set is comparable
with the results on only 20% from Table 1.

Dataset | Metric | Standard | Adv | DAdv | BTEO | BTJ |  | LEACE-
---
last
| LEACE-
---
last+SR,
opt. perc.
| LEACE-
---
last+KL,
opt. perc.
| LEACE-
---
cls
| LEACE-
---
cls+SR,
opt. perc.
| LEACE-
---
cls+KL,
opt. perc.
INLP |  | INLP+SR,
---
opt. perc.
| INLP+KL,
---
opt. perc.
MOJI imbalanced | Fairness $\uparrow$ | 61.8$\pm$0.7 | 73.7$\pm$0.6 | 73.4$\pm$0.4 | 75.2$\pm$0.6 | 74.8$\pm$0.6 | 75.7$\pm$2.6 | 68.5$\pm$1.3 | 75.9$\pm$1.3 | 74.5$\pm$2.4 | 68.4$\pm$1.2 | 77.0$\pm$0.9 | 88.2$\pm$6.3 | 64.1$\pm$1.7 | 73.2$\pm$1.3
Accuracy $\uparrow$ | 79.1$\pm$0.7 | 72.0$\pm$0.7 | 72.4$\pm$0.5 | 73.6$\pm$0.6 | 73.2$\pm$0.4 | 68.3$\pm$2.3 | 77.7$\pm$1.0 | 72.2$\pm$1.1 | 66.8$\pm$2.5 | 77.6$\pm$1.0 | 71.6$\pm$1.1 | 59.9$\pm$7.3 | 77.6$\pm$1.3 | 71.6$\pm$1.9
DTO $\downarrow$ | 43.6$\pm$0.6 | 38.4$\pm$0.5 | 38.3$\pm$0.4 | 36.2$\pm$0.1 | 36.7$\pm$0.4 | 40.0$\pm$3.4 | 38.6$\pm$0.7 | 36.8$\pm$0.7 | 41.9$\pm$3.4 | 38.8$\pm$0.7 | 36.6$\pm$1.0 | 42.6$\pm$5.1 | 42.4$\pm$1.3 | 39.0$\pm$1.8
FF-score $\uparrow$ | 69.4$\pm$0.4 | 72.8$\pm$0.4 | 72.9$\pm$0.3 | 74.4$\pm$0.1 | 74.0$\pm$0.3 | 71.8$\pm$2.4 | 72.8$\pm$0.5 | 74.0$\pm$0.5 | 70.4$\pm$2.4 | 72.7$\pm$0.5 | 74.2$\pm$0.7 | 70.8$\pm$3.4 | 70.2$\pm$0.9 | 72.4$\pm$1.3
MOJI balanced | Fairness $\uparrow$ | 69.5$\pm$0.2 | 83.8$\pm$0.8 | 84.7$\pm$1.5 | 85.5$\pm$0.5 | 85.6$\pm$0.6 | 79.7$\pm$3.5 | 77.0$\pm$0.9 | 86.7$\pm$0.6 | 77.0$\pm$3.4 | 77.0$\pm$0.8 | 87.3$\pm$0.7 | 77.3$\pm$5.6 | 70.4$\pm$1.3 | 74.0$\pm$2.8
Accuracy $\uparrow$ | 71.9$\pm$0.4 | 74.0$\pm$0.4 | 74.1$\pm$0.6 | 74.8$\pm$0.3 | 74.5$\pm$0.4 | 73.6$\pm$0.7 | 74.0$\pm$0.4 | 73.9$\pm$0.2 | 73.0$\pm$0.9 | 74.0$\pm$0.4 | 73.6$\pm$0.5 | 65.9$\pm$4.6 | 71.8$\pm$0.4 | 69.0$\pm$1.8
DTO $\downarrow$ | 41.5$\pm$0.4 | 30.7$\pm$0.7 | 30.1$\pm$0.7 | 29.0$\pm$0.1 | 29.3$\pm$0.4 | 33.4$\pm$2.7 | 34.7$\pm$0.7 | 29.3$\pm$0.4 | 35.5$\pm$3.0 | 34.7$\pm$0.6 | 29.3$\pm$0.5 | 41.3$\pm$4.5 | 40.9$\pm$0.9 | 40.6$\pm$2.3
FF-score $\uparrow$ | 70.7$\pm$0.3 | 78.6$\pm$0.5 | 79.1$\pm$0.6 | 79.8$\pm$0.1 | 79.6$\pm$0.3 | 76.5$\pm$2.0 | 75.5$\pm$0.5 | 79.8$\pm$0.3 | 74.9$\pm$2.1 | 75.5$\pm$0.4 | 79.9$\pm$0.4 | 71.0$\pm$3.4 | 71.1$\pm$0.7 | 71.4$\pm$1.6
BIOS-2 imbalanced | Fairness $\uparrow$ | 90.4$\pm$0.8 | 97.2$\pm$0.8 | 96.4$\pm$0.4 | 95.8$\pm$1.0 | 96.6$\pm$0.8 | 93.3$\pm$8.1 | 93.1$\pm$2.3 | 92.9$\pm$2.1 | 78.0$\pm$5.5 | 95.2$\pm$2.5 | 96.4$\pm$1.0 | 91.6$\pm$1.6 | 91.9$\pm$0.8 | 91.5$\pm$1.5
Accuracy $\uparrow$ | 96.7$\pm$0.1 | 94.8$\pm$0.4 | 95.0$\pm$0.3 | 95.2$\pm$0.3 | 95.0$\pm$0.5 | 61.1$\pm$4.0 | 94.6$\pm$0.3 | 94.7$\pm$0.2 | 65.4$\pm$5.6 | 94.6$\pm$0.1 | 93.2$\pm$0.3 | 95.9$\pm$0.8 | 95.9$\pm$0.6 | 95.9$\pm$0.8
DTO $\downarrow$ | 10.1$\pm$0.7 | 5.9$\pm$0.2 | 6.2$\pm$0.2 | 6.5$\pm$0.6 | 6.1$\pm$0.3 | 40.4$\pm$2.2 | 8.9$\pm$1.7 | 9.0$\pm$1.6 | 41.7$\pm$2.0 | 7.4$\pm$1.8 | 7.7$\pm$0.6 | 9.5$\pm$1.1 | 9.1$\pm$0.4 | 9.5$\pm$1.1
FF-score $\uparrow$ | 93.5$\pm$0.4 | 96.0$\pm$0.2 | 95.7$\pm$0.1 | 95.5$\pm$0.4 | 95.8$\pm$0.2 | 73.5$\pm$2.0 | 93.8$\pm$1.2 | 93.7$\pm$1.1 | 70.7$\pm$1.2 | 94.9$\pm$1.3 | 94.8$\pm$0.6 | 93.7$\pm$0.6 | 93.8$\pm$0.2 | 93.6$\pm$0.6
BIOS-2 balanced | Fairness $\uparrow$ | 89.7$\pm$0.6 | 97.8$\pm$0.8 | 98.0$\pm$0.8 | 95.9$\pm$0.8 | 96.4$\pm$0.3 | 91.2$\pm$10.2 | 93.2$\pm$2.6 | 94.3$\pm$3.6 | 74.7$\pm$9.2 | 96.7$\pm$1.6 | 97.6$\pm$1.1 | 91.8$\pm$1.1 | 91.4$\pm$0.9 | 91.8$\pm$1.1
Accuracy $\uparrow$ | 92.4$\pm$0.3 | 91.9$\pm$0.6 | 91.9$\pm$1.5 | 92.6$\pm$0.5 | 92.9$\pm$0.6 | 50.4$\pm$8.8 | 90.7$\pm$1.2 | 90.0$\pm$1.8 | 64.0$\pm$9.7 | 91.9$\pm$0.9 | 90.6$\pm$1.4 | 90.7$\pm$1.2 | 91.1$\pm$1.1 | 90.7$\pm$1.1
DTO $\downarrow$ | 12.8$\pm$0.6 | 8.5$\pm$0.4 | 8.4$\pm$1.4 | 8.5$\pm$0.2 | 8.0$\pm$0.6 | 51.9$\pm$5.1 | 11.6$\pm$2.4 | 11.7$\pm$3.3 | 45.8$\pm$3.5 | 8.8$\pm$1.4 | 9.7$\pm$1.6 | 12.5$\pm$1.2 | 12.4$\pm$1.1 | 12.4$\pm$1.1
FF-score $\uparrow$ | 91.1$\pm$0.4 | 94.7$\pm$0.1 | 94.9$\pm$0.7 | 94.2$\pm$0.2 | 94.6$\pm$0.3 | 63.7$\pm$3.6 | 91.9$\pm$1.9 | 92.1$\pm$2.6 | 67.7$\pm$2.4 | 94.2$\pm$1.2 | 94.0$\pm$1.1 | 91.2$\pm$0.9 | 91.3$\pm$0.8 | 91.3$\pm$0.8

Table 5:  Comparison of debiasing methods and selective debiasing; the post-
processing methods trained on full training set. The best results in the group
are in bold, and the best results overall are underlined.

Dataset | Standard | LEACE |  | Random,
---
5%
| SR,
---
5%
| KL,
---
5%
| Random,
---
10%
| SR,
---
10%
| KL,
---
10%
| Random,
---
15%
| SR,
---
15%
| KL,
---
15%
| Random,
---
optimal
percentage
| SR,
---
optimal
percentage
| KL,
---
optimal
percentage
MOJI imbalanced | 69.4$\pm$0.4 | 71.8$\pm$2.6 | 70.6$\pm$0.3 | 70.8$\pm$0.5 | 72.5$\pm$0.6 | 71.4$\pm$0.3 | 71.8$\pm$0.6 | 73.7$\pm$0.7 | 72.0$\pm$0.4 | 72.8$\pm$0.5 | 74.1$\pm$0.3 | 72.0$\pm$0.4 | 72.8$\pm$0.5 | 74.1$\pm$0.3
MOJI balanced | 70.7$\pm$0.3 | 76.5$\pm$2.2 | 71.8$\pm$0.2 | 72.4$\pm$0.1 | 75.4$\pm$0.2 | 72.5$\pm$0.0 | 74.0$\pm$0.2 | 78.2$\pm$0.3 | 73.3$\pm$0.1 | 75.5$\pm$0.5 | 79.8$\pm$0.2 | 73.3$\pm$0.1 | 75.5$\pm$0.5 | 79.8$\pm$0.2
BIOS-2 imbalanced | 93.5$\pm$0.4 | 72.8$\pm$2.3 | 92.9$\pm$0.5 | 93.7$\pm$1.0 | 93.3$\pm$2.0 | 92.0$\pm$0.8 | 93.7$\pm$1.3 | 90.3$\pm$2.1 | 91.1$\pm$1.2 | 93.1$\pm$1.1 | 86.3$\pm$2.2 | 93.4$\pm$0.4 | 93.8$\pm$1.2 | 93.2$\pm$2.3
BIOS-2 balanced | 91.1$\pm$0.4 | 63.0$\pm$4.6 | 90.6$\pm$0.3 | 91.6$\pm$1.0 | 92.0$\pm$2.1 | 89.3$\pm$0.9 | 92.2$\pm$1.9 | 89.6$\pm$2.4 | 88.7$\pm$1.2 | 92.6$\pm$2.2 | 85.4$\pm$2.4 | 90.9$\pm$0.3 | 92.3$\pm$1.9 | 91.9$\pm$2.9

Table 6:  FF-score of selective debiasing for LEACE on the last layer for
various percentages.

Dataset | Standard | LEACE |  | Random,
---
5%
| SR,
---
5%
| KL,
---
5%
| Random,
---
10%
| SR,
---
10%
| KL,
---
10%
| Random,
---
15%
| SR,
---
15%
| KL,
---
15%
| Random,
---
optimal
percentage
| SR,
---
optimal
percentage
| KL,
---
optimal
percentage
MOJI imbalanced | 69.4$\pm$0.4 | 70.8$\pm$3.0 | 70.7$\pm$0.4 | 70.8$\pm$0.5 | 72.5$\pm$0.8 | 71.5$\pm$0.4 | 71.7$\pm$0.5 | 74.2$\pm$0.6 | 72.1$\pm$0.5 | 72.7$\pm$0.4 | 74.4$\pm$0.8 | 72.1$\pm$0.5 | 72.7$\pm$0.4 | 74.4$\pm$0.8
MOJI balanced | 70.7$\pm$0.3 | 75.2$\pm$2.6 | 71.8$\pm$0.3 | 72.4$\pm$0.1 | 75.5$\pm$0.2 | 72.7$\pm$0.2 | 74.0$\pm$0.2 | 78.7$\pm$0.3 | 73.5$\pm$0.1 | 75.5$\pm$0.4 | 80.0$\pm$0.4 | 73.5$\pm$0.1 | 75.5$\pm$0.4 | 80.0$\pm$0.4
BIOS-2 imbalanced | 93.5$\pm$0.4 | 69.6$\pm$1.7 | 93.4$\pm$0.7 | 94.6$\pm$1.0 | 95.1$\pm$0.4 | 93.1$\pm$1.0 | 94.7$\pm$0.9 | 91.0$\pm$1.0 | 92.9$\pm$1.3 | 93.2$\pm$0.6 | 85.1$\pm$1.1 | 93.5$\pm$0.4 | 94.7$\pm$1.2 | 94.9$\pm$0.4
BIOS-2 balanced | 91.1$\pm$0.4 | 67.5$\pm$3.0 | 91.2$\pm$0.5 | 92.4$\pm$0.6 | 93.9$\pm$1.1 | 90.8$\pm$1.1 | 93.7$\pm$1.2 | 90.1$\pm$2.3 | 90.8$\pm$1.3 | 94.4$\pm$0.9 | 83.4$\pm$2.2 | 91.2$\pm$0.7 | 94.2$\pm$1.2 | 93.9$\pm$1.1

Table 7:  FF-score of selective debiasing for LEACE on the classifier level
for various percentages.

Dataset | Standard | INLP |  | Random,
---
5%
| SR,
---
5%
| KL,
---
5%
| Random,
---
10%
| SR,
---
10%
| KL,
---
10%
| Random,
---
15%
| SR,
---
15%
| KL,
---
15%
| Random,
---
optimal
percentage
| SR,
---
optimal
percentage
| KL,
---
optimal
percentage
MOJI imbalanced | 69.4$\pm$0.4 | 71.9$\pm$2.2 | 70.0$\pm$0.4 | 69.5$\pm$0.3 | 71.0$\pm$0.8 | 70.1$\pm$0.4 | 69.6$\pm$0.5 | 71.8$\pm$1.0 | 70.2$\pm$0.5 | 70.1$\pm$0.5 | 71.9$\pm$1.3 | 70.2$\pm$0.5 | 70.0$\pm$0.6 | 71.9$\pm$1.3
MOJI balanced | 70.7$\pm$0.3 | 71.9$\pm$3.2 | 71.0$\pm$0.4 | 71.2$\pm$0.1 | 72.0$\pm$0.6 | 71.2$\pm$0.4 | 71.6$\pm$0.4 | 72.5$\pm$0.9 | 71.3$\pm$0.5 | 71.8$\pm$0.5 | 72.8$\pm$1.0 | 71.3$\pm$0.5 | 71.8$\pm$0.4 | 72.8$\pm$1.0
BIOS-2 imbalanced | 93.5$\pm$0.4 | 93.8$\pm$0.7 | 93.5$\pm$0.4 | 93.9$\pm$0.7 | 93.8$\pm$0.8 | 93.5$\pm$0.4 | 93.8$\pm$0.7 | 93.8$\pm$0.8 | 93.6$\pm$0.4 | 93.8$\pm$0.7 | 93.8$\pm$0.8 | 93.6$\pm$0.4 | 93.9$\pm$0.7 | 93.8$\pm$0.8
BIOS-2 balanced | 91.1$\pm$0.4 | 91.8$\pm$1.1 | 91.1$\pm$0.4 | 91.5$\pm$0.9 | 91.6$\pm$1.1 | 91.1$\pm$0.5 | 91.8$\pm$1.1 | 91.7$\pm$1.1 | 91.2$\pm$0.5 | 91.9$\pm$1.2 | 91.7$\pm$1.2 | 91.2$\pm$0.5 | 91.9$\pm$1.2 | 91.6$\pm$1.1

Table 8:  FF-score of selective debiasing for INLP for various percentages.

Dataset | LEACE-last | LEACE-cls | INLP
---|---|---|---
Random | SR | KL | Random | SR | KL | Random | SR | KL
MOJI imbalanced | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 14
MOJI balanced | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 13 | 11
BIOS-2 imbalanced | 1 | 6 | 3 | 1 | 6 | 4 | 9 | 12 | 8
BIOS-2 balanced | 1 | 11 | 6 | 7 | 12 | 5 | 8 | 15 | 13

Table 9:  Optimal selection percentages for various debiasing methods.

##  Appendix C Comparison with Other Distances

We also conducted additional experiments to compare how proposed selection
strategies differ from other similarity measures. Here we consider several
measures, calculated over output from the last hidden layer of the model, and
compare it with SR and KL strategies. The results are presented in Tables 10,
11 and 12. In most cases, selection by KL works comparably or better than the
best-performing distance-based measure. Moreover, KL scores are easier to
compute than distance-based scores. However, in some cases, cosine distance
could serve as a replacement for the KL score due to its similar performance.

Dataset | Standard | LEACE | SR, 5% | KL, 5% | Euclidean, 5% | Cosine, 5% | SR, 10% | KL, 10% | Euclidean, 10% | Cosine, 10% | SR, 15% | KL, 15% | Euclidean, 15% | Cosine, 15%
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
MOJI imbalanced | 69.4$\pm$0.4 | 71.8$\pm$2.6 | 70.8$\pm$0.5 | 72.5$\pm$0.6 | 71.4$\pm$0.7 | 72.2$\pm$0.6 | 71.8$\pm$0.6 | 73.7$\pm$0.7 | 72.5$\pm$0.8 | 73.8$\pm$0.6 | 72.8$\pm$0.5 | 74.1$\pm$0.3 | 73.0$\pm$0.8 | 74.3$\pm$0.8
MOJI balanced | 70.7$\pm$0.3 | 76.5$\pm$2.2 | 72.4$\pm$0.1 | 75.4$\pm$0.2 | 74.0$\pm$0.4 | 75.0$\pm$0.2 | 74.0$\pm$0.2 | 78.2$\pm$0.3 | 76.2$\pm$0.4 | 78.1$\pm$0.3 | 75.5$\pm$0.5 | 79.8$\pm$0.2 | 77.5$\pm$0.4 | 79.2$\pm$0.5
BIOS-2 imbalanced | 93.5$\pm$0.4 | 72.8$\pm$2.3 | 93.7$\pm$1.0 | 93.3$\pm$2.0 | 93.1$\pm$1.5 | 92.0$\pm$2.0 | 93.7$\pm$1.3 | 90.3$\pm$2.1 | 90.1$\pm$1.0 | 88.7$\pm$1.5 | 93.1$\pm$1.1 | 86.3$\pm$2.2 | 86.6$\pm$1.5 | 85.6$\pm$1.9
BIOS-2 balanced | 91.1$\pm$0.4 | 63.0$\pm$4.6 | 91.6$\pm$1.0 | 92.0$\pm$2.1 | 90.3$\pm$1.9 | 89.5$\pm$1.6 | 92.2$\pm$1.9 | 89.6$\pm$2.4 | 87.9$\pm$3.0 | 86.0$\pm$1.7 | 92.6$\pm$2.2 | 85.4$\pm$2.4 | 84.4$\pm$1.9 | 82.8$\pm$2.1

Table 10:  Comparison of FF-score of distance-based scores for LEACE-last for
various percentages.

Dataset | Standard | LEACE | SR, 5% | KL, 5% | Euclidean, 5% | Cosine, 5% | SR, 10% | KL, 10% | Euclidean, 10% | Cosine, 10% | SR, 15% | KL, 15% | Euclidean, 15% | Cosine, 15%
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
MOJI imbalanced | 69.4$\pm$0.4 | 70.8$\pm$3.0 | 70.8$\pm$0.5 | 72.5$\pm$0.8 | 71.9$\pm$0.5 | 72.2$\pm$0.5 | 71.7$\pm$0.5 | 74.2$\pm$0.6 | 73.3$\pm$0.8 | 73.7$\pm$1.1 | 72.7$\pm$0.4 | 74.4$\pm$0.8 | 73.9$\pm$0.9 | 74.3$\pm$1.3
MOJI balanced | 70.7$\pm$0.3 | 75.2$\pm$2.6 | 72.4$\pm$0.1 | 75.5$\pm$0.2 | 74.7$\pm$0.3 | 74.4$\pm$0.5 | 74.0$\pm$0.2 | 78.7$\pm$0.3 | 77.2$\pm$0.4 | 77.3$\pm$1.0 | 75.5$\pm$0.4 | 80.0$\pm$0.4 | 78.7$\pm$0.4 | 78.9$\pm$1.1
BIOS-2 imbalanced | 93.5$\pm$0.4 | 69.6$\pm$1.7 | 94.6$\pm$1.0 | 95.1$\pm$0.4 | 94.3$\pm$0.6 | 94.1$\pm$0.6 | 94.7$\pm$0.9 | 91.0$\pm$1.0 | 89.9$\pm$1.2 | 90.3$\pm$1.2 | 93.2$\pm$0.6 | 85.1$\pm$1.1 | 83.9$\pm$1.0 | 84.4$\pm$1.0
BIOS-2 balanced | 91.1$\pm$0.4 | 67.5$\pm$3.0 | 92.4$\pm$0.6 | 93.9$\pm$1.1 | 93.3$\pm$1.4 | 92.6$\pm$1.1 | 93.7$\pm$1.2 | 90.1$\pm$2.3 | 87.2$\pm$1.8 | 87.0$\pm$1.3 | 94.4$\pm$0.9 | 83.4$\pm$2.2 | 80.6$\pm$2.0 | 80.9$\pm$1.5

Table 11:  Comparison of FF-score of distance-based scores for LEACE-cls for
various percentages.

Dataset | Standard | LEACE | SR, 5% | KL, 5% | Euclidean, 5% | Cosine, 5% | SR, 10% | KL, 10% | Euclidean, 10% | Cosine, 10% | SR, 15% | KL, 15% | Euclidean, 15% | Cosine, 15%
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
MOJI imbalanced | 69.4$\pm$0.4 | 71.9$\pm$2.2 | 69.5$\pm$0.3 | 71.0$\pm$0.8 | 69.5$\pm$0.5 | 69.5$\pm$0.5 | 69.6$\pm$0.5 | 71.8$\pm$1.0 | 69.7$\pm$0.7 | 69.6$\pm$0.6 | 70.1$\pm$0.5 | 71.9$\pm$1.3 | 69.8$\pm$0.9 | 69.7$\pm$0.8
MOJI balanced | 70.7$\pm$0.3 | 71.9$\pm$3.2 | 71.2$\pm$0.1 | 72.0$\pm$0.6 | 70.9$\pm$0.4 | 71.1$\pm$0.5 | 71.6$\pm$0.4 | 72.5$\pm$0.9 | 71.2$\pm$0.5 | 71.3$\pm$0.6 | 71.8$\pm$0.5 | 72.8$\pm$1.0 | 71.4$\pm$0.6 | 71.5$\pm$0.6
BIOS-2 imbalanced | 93.5$\pm$0.4 | 93.8$\pm$0.7 | 93.9$\pm$0.7 | 93.8$\pm$0.8 | 93.5$\pm$0.4 | 93.5$\pm$0.4 | 93.8$\pm$0.7 | 93.8$\pm$0.8 | 93.5$\pm$0.4 | 93.5$\pm$0.4 | 93.8$\pm$0.7 | 93.8$\pm$0.8 | 93.5$\pm$0.4 | 93.5$\pm$0.4
BIOS-2 balanced | 91.1$\pm$0.4 | 91.8$\pm$1.1 | 91.5$\pm$0.9 | 91.6$\pm$1.1 | 91.1$\pm$0.4 | 91.1$\pm$0.4 | 91.8$\pm$1.1 | 91.7$\pm$1.1 | 91.1$\pm$0.4 | 91.1$\pm$0.4 | 91.9$\pm$1.2 | 91.7$\pm$1.2 | 91.1$\pm$0.4 | 91.1$\pm$0.4

Table 12:  Comparison of FF-score of distance-based scores for INLP for
various percentages.

##  Appendix D Datasets Statistics

The synthetic dataset was generated as a random 2 classes classification task
using `make_classification` function from Scikit-learn library Pedregosa et
al. (2011) with following parameters: `n_features=10`, `n_informative=5`,
`n_clusters_per_class=2`, `random_state=42`, `n_redundant=2`. The protected
attribute for the synthetic dataset is designed as a condition over the first
informative feature and equals 1 if this feature is greater than 0, and 0
otherwise. The overall statistics for each dataset are presented in Table 13.
Tables 14 and 15 shows the joint distribution of target variable and protected
attributes.

Dataset |  | Num. of
---
classes/attributes
| Protected
---
attribute
Train/Val/Test
Synthetic | 2/2 | Geometric | 6k/2k/2k
Moji (balanced) | 2/2 | Race | 100k/8k/8k
Moji (imbalanced) | 2/2 | Race | 100k/5k/5k
Bios-2 (imbalanced) | 2/2 | Gender | 21k/3k/8k
Bios-2 (balanced) | 2/2 | Gender | 21k/1k/2k

Table 13:  Dataset statistics.

Split | Gender | Profession
---|---|---
Nurse | Surgeon | Total
Train | Female | 53.34 | 5.74 | 59.08
| Male | 5.50 | 35.42 | 40.92
| All | 58.84 | 41.16 | 100.00
Val | Female | 53.32 | 5.08 | 58.40
| Male | 5.52 | 36.08 | 41.60
| All | 58.83 | 41.17 | 100.00
Test | Female | 53.82 | 7.51 | 61.33
| Male | 5.01 | 33.66 | 38.67
| All | 58.83 | 41.17 | 100.00
Val (balanced) | Female | 26.02 | 23.98 | 50.00
| Male | 26.02 | 23.98 | 50.00
| All | 52.05 | 47.95 | 100.00
Test (balanced) | Female | 20.02 | 29.98 | 50.00
| Male | 20.02 | 29.98 | 50.00
| All | 40.04 | 59.96 | 100.00

Table 14:  Joint distribution for BIOS-2 dataset.

Split | Ethnicity | Target
---|---|---
Sad | Happy | Total
Train | SA | 40.00 | 10.00 | 50.00
| AA | 10.00 | 40.00 | 50.00
| All | 50.00 | 50.00 | 100.00
Val | SA | 40.02 | 9.98 | 50.00
| AA | 9.98 | 40.02 | 50.00
| All | 50.00 | 50.00 | 100.00
Test | SA | 40.02 | 9.99 | 50.01
| AA | 9.99 | 40.00 | 49.99
| All | 50.01 | 49.99 | 100.00
Val (balanced) | SA | 25.00 | 25.00 | 50.00
| AA | 25.00 | 25.00 | 50.00
| All | 50.00 | 50.00 | 100.00
Test (balanced) | SA | 25.01 | 25.01 | 50.01
| AA | 24.99 | 24.99 | 49.99
| All | 50.00 | 50.00 | 100.00

Table 15:  Joint distribution for MOJI dataset.

##  Appendix E Equal Opportunity

There are numerous amount of group fairness definitions; to avoid any
mismatches, we are presenting the step-by-step process of equal opportunity
criterion calculation. This criterion is based on recall values, or true
positive rate (TPR) for each class and protected group.

  * •

TPR (recall) for each protected group defined as follows:

$TPR=\frac{TP}{TP+FN},$ (5)

where $TP$, $FN$ \- is true positives and false negatives for specific group.

  * •

After we calculate TPR-gap:

$\delta=\sqrt{\frac{1}{C}\sum_{c}(\sum_{g}|TPR_{c,g}-\overline{TPR_{c}}|)^{2}},$
(6)

here $g$ is group index, $c$ \- class index, $\overline{TPR_{c}}$ \- $TPR$
averaged across all groups for class $c$.

  * •

Finally, we calculate fairness with the following equation:

$Fairness=100\cdot(1-\delta).$ (7)

##  Appendix F Fairness and UQ Oracles

In this section, we describe in detail oracle strategies for fairness and
accuracy. For both strategies, we assume access to the ground-truth labels,
while for fairness oracle we also use protected attributes. Accuracy oracle
built as follows – we find all erroneously classified instances and replace
predictions on these instances with ground-truth labels while keeping all
other predictions unchanged. This oracle shows the best possible UQ strategy
that allows the detection of all erroneous predictions and gives the maximal
increase in accuracy. The same idea is behind fairness oracle, but instead of
accuracy, we used fairness as a target metric. For fairness, we first replace
predictions for instances, which gives the maximal increase in fairness. These
predictions were chosen greedily from the erroneous ones. To measure the
quality of these oracle strategies and to compare it with other scores, we
calculated several metrics: FR-AUC, Acc-AUC, and FF-score-AUC. Each
corresponds to the area under the target metric-rejection curve, where the
target metric is fairness, accuracy, or FF-score; the area under the curve is
calculated on binarized over 100 points target metric values.