11affiliationtext: Basque Center for Applied Mathematics (BCAM),
Spain22affiliationtext: Predictive Analytics Lab (PAL), University of Sussex,
UK33affiliationtext: Universidad Pública de Navarra (UPNA),
Spain44affiliationtext: Mark3Institute for Advanced Materials and Mathematics
(INAMAT2), Spain55affiliationtext: University of the Basque Country (UPV/EHU),
Spain66affiliationtext: Monash University, Indonesia

# Dancing in the Shadows: Harnessing Ambiguity for Fairer Classifiers

Ainhize Barrainkua  Paula Gordaliza  Jose A. Lozano  Novi Quadrianto

###### Abstract

This paper introduces a novel approach to bolster algorithmic fairness in
scenarios where sensitive information is only partially known. In particular,
we propose to leverage instances with uncertain identity with regards to the
sensitive attribute to train a conventional machine learning classifier. The
enhanced fairness observed in the final predictions of this classifier
highlights the promising potential of prioritizing ambiguity (i.e., non-
normativity) as a means to improve fairness guarantees in real-world
classification tasks.

##  1 Introduction

Algorithmic systems, designed to streamline decision processes and enhance
efficiency, have permeated virtually every aspect of our lives. From credit
approvals to hiring decisions, from predictive policing to healthcare
recommendations, algorithms wield significant influence. Yet, this influence
is not neutral, and the consequences could be disproportionate for diverse
communities. Subtle biases embedded in training data, the choices made during
model development, and the very nature of algorithmic decision-making are some
potential reasons for inequitable treatment of certain demographic groups,
perpetuating and, in some instances, exacerbating societal disparities.
Consider, for instance, the use of predictive policing algorithms, where
certain communities are subjected to heightened surveillance based on
historical crime data, perpetuating a cycle of over-policing [9]. Similarly,
in hiring practices, algorithms may inadvertently favor certain demographics,
leading to underrepresentation and reinforcing existing inequalities in the
workplace [6, 5]. Therefore, it is crucial to acknowledge the inherent biases
and disparities that have emerged within these systems and propose innovative
solutions to enhance their fairness guarantees.

It is already well-known that the naïve strategy of simply removing the
sensitive information (e.g., gender, race, age, sexual orientation) of the
instances does not improve the fairness guarantees of the classifier. This
happens because there are several ’objective’ attributes that are highly
correlated with the sensitive attribute, and therefore, they encode redundant
information [10, 15, 4]. As an alternative, numerous fairness-enhancing
interventions have been proposed, categorized as pre-, in-, and post-
processing methods based on the stage at which fairness requirements are
integrated. Pre-processing methods aim to mitigate bias in the dataset, in-
processing techniques adjust classifier learning to incorporate fairness
guarantees, and post-processing methods alter algorithmic predictions to
ensure fair final outcomes. Refer to [19] for a comprehensive review on
fairness-enhancing interventions.

Conventional fairness-enhancing interventions can only operate under complete
knowledge of the sensitive information. However, such assumption is
unrealistic: in many real-world applications, obtaining the sensitive
attribute value of all the instances is either illegal or impractical, as
individuals may be reluctant to provide such information due to privacy
concerns or fear of discrimination. This constraint underscores the importance
of developing fair and effective algorithms that can operate with only partial
access to sensitive attributes, without compromising privacy or perpetuating
biases.

Numerous methods operating within these conditions utilize an attribute
classifier to forecast the sensitive information linked with each instance [3,
12, 7, 20, 18, 17, 16]. Afterwards, they select instances for which confident
predictions were made and utilize them to train a traditional fairness-
enhancing intervention. In our study, we also advocate for the use of
attribute classifiers, albeit with a distinct (opposite) purpose. Unlike
conventional methods that depend on clear-cut predictions—a practice that has
sparked ethical apprehensions— we embrace those characterized by high
uncertainty. This decision stems from our hypothesis that leveraging instances
with high ambiguity in the sensitive information, or what we term non-
normative cases, holds the potential to improve the fairness guarantees of the
final classifier that will be blind to the sensitive information. By embracing
the richness of diversity and the inclusion of non-normative instances in our
training data, we aim to develop classification rules that transcend
traditional norms and promotes equitable outcomes for all individuals,
regardless of their sensitive attributes.

Experiments conducted on benchmark real-world high-stake classification tasks
demonstrate that incorporating non-normativity indeed enhances the fairness
guarantees of conventional machine learning (ML) classifiers. Moreover, these
findings indicate that these fairness assurances are on par with those
provided by popular fairness-enhancing interventions that presume complete
knowledge of sensitive information, whereas our method only requires partial
knowledge of such information.

##  2 Notation and Theoretical Background

###  2.1 Notation and Problem Setting

We consider a dataset consisting of both non-sensitive and sensitive
information, denoted as $\mathcal{D}_{1}=\\{(x_{i},s_{i})\\}_{i=1}^{N}$, where
$x\in\mathcal{X}\subset\mathbb{R}^{d}$ represents the non-sensitive features,
and $s\in\mathcal{S}$ denotes the sensitive feature(s). This dataset will
serve as the training data for the algorithm to detect instances with
uncertain sensitive information. Additionally, we consider a second dataset
containing non-sensitive features and class information, but without sensitive
information, denoted as $\mathcal{D}_{2}=\\{(x_{i},y_{i})\\}_{i=1}^{M}$, where
$y\in\mathcal{Y}$ represents the class label. Non-normative instances will be
identified from this dataset and utilized to train the final classification
rule, which will be blind to the sensitive information. For simplicity, we
focus on binary classification tasks and binary sensitive attributes, where
$\mathcal{Y}=\\{0,1\\}$ and $\mathcal{S}=\\{0,1\\}$, respectively. However,
extending this work to multi-class and multi-group settings is
straightforward.

###  2.2 Measuring Algorithmic Fairness

There are many definitions to measure the fairness guarantees of classifiers,
and are conventionally divided into two groups: individual and statistical (or
group) notions. Individual fairness notions operate under the principle that
comparable individuals should receive similar treatment. On the other hand,
statistical fairness notions examine whether a particular statistical quantity
is consistent across different sensitive groups. The primary limitations of
individual notions include the necessity of defining a non-trivial similarity
metric and the requirement to compare all potential pairs of instances, which
can be impractical at scale. In contrast, statistical notions offer a more
computationally efficient alternative, contributing to their widespread
adoption in the literature. Within statistical notions, the name of the
fairness metrics varies depending on the statistical quantity being
considered. For instance, demographic parity (DP) [8] assesses whether the
acceptance rate is consistent across different sensitive groups, that is:

$\textrm{DP}=|p(\hat{y}=1|s=1)-p(\hat{y}=1|s=0)|.$ (1)

On the other hand, equality of opportunity (EOp) [11] examines whether the
true positive rate (TPR) is uniform across these groups:

$\textrm{EOp}=|p(\hat{y}=1|s=1,y=1)-p(\hat{y}=1|s=0,y=1)|$ (2)

See [19] for a comprehensive review on fairness metrics and fairness-enhancing
interventions.

##  3 Methodology

Our approach comprises three primary components: the norm breaker ensemble,
the non-normative dataset, and the unbiased classifier. In the following
lines, we present detailed explanations of each.

#### Norm Breaker Ensemble (NBE)

The primary objective of the NBE is to identify instances that deviate from
the norm. NBE operates as a conventional ensemble classifier, flagging
instances with elevated predictive uncertainty as less normative. Each base
classifier within the ensemble serves as an attribute classifier
($h_{S}:\mathcal{X}\rightarrow\mathcal{S}$). These attribute classifiers
provide varying perspectives on the relationship between the ’objective’
features and the sensitive attribute. The ensemble is trained with the
instances in $\mathcal{D}_{1}$, for which the sensitive information is known.
The objective behind building such an ensemble is to detect non-normative
instances. These instances are characterized by the highest ambiguity in the
predicted sensitive attribute. In other words, they are the instances with
significant disagreement in predictions among the base estimators of the
ensemble. Specifically, each instance $(x,y)$ will be assigned an uncertainty
value $u_{x}$, calculated as follows: $u_{x}=1-\max\\{p(s=1|x),1-p(s=1|x)\\}$,
where $p(s=1|x)$ refers to the probability given by the ensemble to assigning
sensitive attribute value $s=1$ to instance $(x,y)$. Uncertainty reaches its
highest value when the ensemble exhibit the greatest disagreement, meaning
half of the learners support one value of the sensitive attribute while the
remainder advocate for the opposite value.

#### Non-Normative Dataset ($\mathcal{D}^{\prime}_{2}$)

The non-normative dataset comprises instances with uncertain or ambiguous
sensitive information. Let $u_{x}$ be the predictive uncertainty of the NBE
regarding the sensitive information of instance $(x,y)$, then the non-
normative dataset is created as follows:
$\mathcal{D}^{\prime}_{2}=\\{(x,y)\;|\;u_{x}\geq U\\}$, where $U$ is the
uncertainty threshold. Our approach targets the elimination of bias from the
dataset utilized to train the final conventional ML classifier. Thus, it
serves as a pre-processing fairness-enhancing intervention.

#### Unbiased Classifier ($h_{Y}$)

The final unbiased classifier is also a conventional unconstrained ML
classifier (blind to the sensitive information), which does not inherently
include any explicit fairness-enhancing interventions (i.e.,
$h_{Y}:\mathcal{X}\rightarrow\mathcal{Y}$). This classifier will be trained
using instances from $\mathcal{D}^{\prime}_{2}$. Given that this dataset
exhibits less bias than the original, it is expected that the ML classifier
will generate less biased decisions. It is important to note that the family
of functions selected to build this classifier may differ from the one chosen
for the base learners of the NBE.

##  4 Experiments

###  4.1 Experimental Setting

#### Models

We explore different conventional ML classifiers: logistic regression (LR),
support vector machine (SVM) and light gradient boosting method (LGBM) [14].
These models serve as the base learners for the NBE, the final unbiased
classifier and the classifer on top of which the state-of-the-art (SOTA)
fairness-enhancing interventions are added. We construct the NBE using a
particular family of functions by bagging (a.k.a. boostrapping). Furthermore,
the SOTA interventions considered are the pre-processing method by [13] and
the post-processing method by [11], respectively referenced as RW and PP
throughout the remainder of the paper. Note that, unlike our approach, these
fairness-enhancing interventions operate under the assumption of complete
knowledge regarding the sensitive information.

#### Datasets

(a) German Credit, data from a German bank to predict good/bad credit score,
publicly available in the UCI repository [2]; (b) Adult Income, US census
based data from 1994 for income prediction, publicly available in the UCI
repository [2]; and (c) COMPAS [1], arrest data from Broward County, Florida,
originally compiled by ProPublica. Their corresponding details are outlined in
Table 1.

Table 1: Datasets: name, number of instances (size), number of features ($\\#$feat), application domain and sensitive attribute (S). Dataset | Size |  $\\#$feat | Domain | S
---|---|---|---|---
German Credit | 1,000 | 20 | Finance | Age
Adult Income | 48,842 | 15 | Finance | Race
COMPAS | 6,000 | 11 | Criminal justice | Race

#### Evaluation

We evaluate our approach for 10 different train/test (70/30) paritions and
report averaged result in fairness guarantees (EOp and DP) and accuracy. We
partition the training dataset into two disjoint subsets of equal size to form
$\mathcal{D}_{1}$ and $\mathcal{D}_{2}$. In our method, $\mathcal{D}_{1}$ is
used to train the attribute classifier and identify the non-normative
instances from $\mathcal{D}_{2}$, resulting in the creation of
$\mathcal{D}^{\prime}_{2}$. On the other hand, the pre- and post-processing
method considered are trained using the original dataset $\mathcal{D}_{2}$,
but further assuming that the sensitive information of those instances is
known. Additionally, we investigate the average uncertainty of the sensitive
information in dataset $\mathcal{D}_{2}$ using NBE, as well as the accuracy of
NBE in predicting the sensitive information of $\mathcal{D}_{2}$.

###  4.2 Results

The empirical evaluation carried out in the German Credit (see Figure 1),
Adult Income (see Figure 2) and COMPAS (see Figure 3) datasets clearly
demonstrate that harnessing non-normativity significantly improves the
fairness guarantees of ML classifiers. Indeed, our approach consistently
outperforms SOTA approaches in terms of fairness guarantees across nearly all
cases. It is worth highlighting that this improvement in fairness typically
comes at the expense of accuracy. Nonetheless, the deterioration in accuracy
is lower than the gain in fairness. Further, in specific scenarios, the
decrease in accuracy resulting from fairness enhancement is negligible. This
decrease in accuracy may result from the substantial reduction in training
instances when only non-normative examples are used, which can potentially
harm predictive performance. Moreover, a crucial question arises: if the
labels are potentially biased, is accuracy a reliable measure of predictive
performance?

Besides, Table 2 presents the uncertainty regarding the sensitive information
in dataset $\mathcal{D}_{2}$, along with the accuracy of NBE in predicting the
sensitive information from the instances of that dataset. The German Credit
and COMPAS datasets exhibit the highest uncertainty regarding sensitive
information, and correspondingly, they also demonstrate the lowest accuracy in
the predictions of the NBE. Interestingly, it is within these two datasets
that our approach achieves the most favorable fairness-accuracy trade-offs.

(a) LR (EOp)

(b) SVM (EOp)

(c) LGBM (EOp)

(d) LR (DP)

(e) SVM (DP)

(f) LGBM (DP)

(g) LR (EOp vs. acc)

(h) SVM (EOp vs. acc)

(i) LGBM (EOp vs. acc)

Figure 1:  Average results in the German Credit dataset for varying
uncertainty threshold values, using different base learners. The first two
rows depict fairness guarantees in terms of EOp and DP, respectively, with the
shaded region indicating variance. The last row illustrates the results in the
joint space of fairness(EOp) and accuracy. Our method outperforms the SOTA
interventions in terms of fairness across all ML classifiers. Specifically,
SVM and LGBM offer the most optimal fairness guarantees, almost reaching
perfection, with minimal compromise in accuracy. Moreover, SVM shows the best
fairness-accuracy trade-off curve.

(a) LR (EOp)

(b) SVM (EOp)

(c) LGBM (EOp)

(d) LR (DP)

(e) SVM (DP)

(f) LGBM (DP)

(g) LR (EOp vs. acc)

(h) SVM (EOp vs. acc)

(i) LGBM (EOp vs. acc)

Figure 2: Average results in the Adult Income dataset for varying uncertainty
threshold values, using different base learners. The first two rows depict
fairness guarantees in terms of EOp and DP, respectively, with the shaded
region indicating variance. The last row illustrates the results in the joint
space of fairness(EOp) and accuracy. Our method outperforms the SOTA pre- and
post-processing interventions in terms of fairness guarantees across all
considered ML classifiers. Specifically, employing SVM and LGBM yields the
most favorable fairness guarantees, with a similar decrease in accuracy.

(a) LR (EOp)

(b) SVM (EOp)

(c) LGBM (EOp)

(d) LR (DP)

(e) SVM (DP)

(f) LGBM (DP)

(g) LR (EOp vs. acc)

(h) SVM (EOp vs. acc)

(i) LGBM (EOp vs. acc)

Figure 3:  Average results in the COMPAS dataset for varying uncertainty threshold values, using different base learners. The first two rows depict fairness guarantees in terms of EOp and DP, respectively, with the shaded region indicating variance. The last row illustrates the results in the joint space of fairness(EOp) and accuracy. Our approach, employing LR and LGBM, outperforms the pre-processing approach in terms of fairness. Further, using SVM enables to outperform both SOTA methods in terms of fairness, providing the best fairness guarantees for this classification task. However, this enhancement in fairness typically accompanies a decrease in accuracy. Notably, the reduction in accuracy is least pronounced when employing LR (a more vertical trade-off curve).  Table 2: Mean uncertainty of data regarding the sensitive information and accuracy of NBE in $\mathcal{D}_{2}$. Dataset | Model | Uncertainty | Accuracy of NBE
---|---|---|---
| LR | $0.24\pm 0.01$ | $0.69\pm 0.03$
German Credit | SVM | $0.34\pm 0.01$ | $0.70\pm 0.02$
| LGBM | $0.22\pm 0.01$ | $0.70\pm 0.02$
| LR | $0.08\pm 0.00$ | $0.92\pm 0.00$
Adult Income | SVM | $0.05\pm 0.00$ | $0.95\pm 0.00$
| LGBM | $0.03\pm 0.00$ | $0.96\pm 0.00$
| LR | $0.29\pm 0.01$ | $0.67\pm 0.01$
COMPAS | SVM | $0.34\pm 0.01$ | $0.67\pm 0.01$
| LGBM | $0.29\pm 0.01$ | $0.67\pm 0.01$

##  5 Conclusions

We propose a new approach to enhance fairness when the sensitive information
is only partially observed. In particular, we propose to leverage non-
normative instances to improve the fairness guarantees of conventional ML
classifiers that are blind to the sensitive information. Through comprehensive
experimentation across various real-world classification tasks, we demonstrate
the effectiveness of our approach in reducing the discrimination in classifier
performance. Remarkably, our method demonstrates comparable or better
performance than certain well-known fairness-enhancing interventions that
operate with full knowledge of the sensitive information.

As part of future venues, we seek to develop an intervention that minimizes
the loss in accuracy. Additionally, we aim to improve our uncertainty
estimation strategy to provide more accurate uncertainty estimations.
Furthermore, we aspire to differentiate between aleatoric and epistemic
uncertainties, which could provide a more nuanced understanding of non-
normative instances.

## Acknowledgments

This work is supported by the European Research Council under the European
Union’s Horizon 2020 research and innovation programme Grant Agreement no.
851538 - BayesianGDPR, Horizon Europe research and innovation programme Grant
Agreement no. 101120763 - TANGO. This work is also supported by the Basque
Government under grant IT1504-22 and through the BERC 2022-2025 program; by
the Spanish Ministry of Science and Innovation under the grants
PID2022-137442NB-I00 and PID2021-128314NB-I00, and through BCAM Severo Ochoa
accreditation CEX2021-001142-S / MICIN / AEI / 10.13039/501100011033.

## References

  * [1] Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner  “Machine bias”  In _Ethics of data and analytics_ Auerbach Publications, 2022, pp. 254–264
  * [2] Arthur Asuncion and David Newman  “UCI machine learning repository”  Irvine, CA, USA, 2007
  * [3] Pranjal Awasthi, Alex Beutel, Matthäus Kleindessner, Jamie Morgenstern and Xuezhi Wang  “Evaluating fairness of machine learning models under uncertain and incomplete information”  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , 2021, pp. 206–214
  * [4] Philippe Besse, Eustasio Barrio, Paula Gordaliza, Jean-Michel Loubes and Laurent Risser  “A survey of bias in machine learning through the prism of statistical parity”  In _The American Statistician_ 76.2 Taylor & Francis, 2022, pp. 188–198
  * [5] Jeffrey Dastin  “Amazon scraps secret AI recruiting tool that showed bias against women”  In _Ethics of data and analytics_ Auerbach Publications, 2022, pp. 296–299
  * [6] Amit Datta, Michael Carl Tschantz and Anupam Datta  “Automated Experiments on Ad Privacy Settings”  In _Proceedings on Privacy Enhancing Technologies_ 2015.1, 2015, pp. 92–112
  * [7] Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, Aaron Roth and Saeed Sharifi-Malvajerdi  “Multiaccurate proxies for downstream fairness”  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , 2022, pp. 1207–1239
  * [8] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold and Richard Zemel  “Fairness through awareness”  In _Proceedings of the 3rd innovations in theoretical computer science conference_ , 2012, pp. 214–226
  * [9] Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger and Suresh Venkatasubramanian  “Runaway feedback loops in predictive policing”  In _Conference on fairness, accountability and transparency_ , 2018, pp. 160–171  PMLR
  * [10] Sara Hajian and Josep Domingo-Ferrer  “A methodology for direct and indirect discrimination prevention in data mining”  In _IEEE transactions on knowledge and data engineering_ 25.7 IEEE, 2012, pp. 1445–1459
  * [11] Moritz Hardt, Eric Price and Nati Srebro  “Equality of opportunity in supervised learning”  In _Advances in neural information processing systems_ 29, 2016, pp. 3315–3323
  * [12] Nathan Kallus, Xiaojie Mao and Angela Zhou  “Assessing algorithmic fairness with unobserved protected class using data combination”  In _Management Science_ 68.3 INFORMS, 2022, pp. 1959–1981
  * [13] Faisal Kamiran and Toon Calders  “Data preprocessing techniques for classification without discrimination”  In _Knowledge and information systems_ 33.1 Springer, 2012, pp. 1–33
  * [14] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye and Tie-Yan Liu  “Lightgbm: A highly efficient gradient boosting decision tree”  In _Advances in neural information processing systems_ 30, 2017
  * [15] Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing and Bernhard Schölkopf  “Avoiding discrimination through causal reasoning”  In _Advances in neural information processing systems_ 30, 2017
  * [16] Yueqing Liang, Canyu Chen, Tian Tian and Kai Shu  “Fair classification via domain adaptation: A dual adversarial learning approach”  In _Frontiers in Big Data_ 5 Frontiers, 2023, pp. 129
  * [17] Vishnu Suresh Lokhande, Kihyuk Sohn, Jinsung Yoon, Madeleine Udell, Chen-Yu Lee and Tomas Pfister  “Towards group robustness in the presence of partial group labels”  In _arXiv preprint arXiv:2201.03668_ , 2022
  * [18] Junhyun Nam, Jaehyung Kim, Jaeho Lee and Jinwoo Shin  “Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation”  In _International Conference on Learning Representations_ , 2022  PMLR
  * [19] Dana Pessach and Erez Shmueli  “Algorithmic fairness”  In _Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook_ Springer, 2023, pp. 867–886
  * [20] Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn and Christopher Ré  “Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations”  In _arXiv preprint arXiv:2203.01517_ , 2022