.pngpdf.pdfconvert #1

#  DispaRisk: Assessing and Interpreting Disparity Risks in Datasets111Work in
progress

Jonathan Vasquez1,2 Carlotta Domeniconi1&Huzefa Rangwala1 1George Mason
University
2Universidad de Valparaíso
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Machine Learning algorithms (ML) impact virtually every aspect of human lives
and have found use across diverse sectors, including healthcare, finance, and
education. Often, ML algorithms have been found to exacerbate societal biases
presented in datasets, leading to adversarial impacts on subsets/groups of
individuals, in many cases minority groups. To effectively mitigate these
untoward effects, it is crucial that disparities/biases are identified and
assessed early in a ML pipeline. This proactive approach facilitates timely
interventions to prevent bias amplification and reduce complexity at later
stages of model development. In this paper, we introduce DispaRisk, a novel
framework designed to proactively assess the potential risks of disparities in
datasets during the initial stages of the ML pipeline. We evaluate DispaRisk’s
effectiveness by benchmarking it with commonly used datasets in fairness
research. Our findings demonstrate the capabilities of DispaRisk to identify
datasets with a high-risk of discrimination, model families prone to biases,
and characteristics that heighten discrimination susceptibility in a ML
pipeline. The code for our experiments is available in the following
repository: https://github.com/jovasque156/disparisk

##  1 Introduction

Extensive research on fairness in machine learning (ML) has shown that
algorithms can exacerbate historical and societal biases when trained on
unchecked/biased datasets. This leads to harmful effects on minorities and
disadvantaged groups Pessach and Shmueli (2023); Kizilcec and Lee (2022); Wen
et al. (2021); Fu et al. (2021), e.g., in criminal justice ProPublica (2016),
healthcare Chen et al. (2023), and education Vasquez et al. (2022). Previous
research emphasizes the importance of identifying and mitigating biases in the
early stages of a ML pipeline Feldman et al. (2015); Hardt et al. (2016). One
solution is to identify potential risks of discrimination via pre-training
metrics, which can be computed directly from the data before they are used for
training the model. Several metrics have been proposed Hardt et al. (2021)
including Class Imbalance (CL), Conditional Demographic Disparity in Labels
(CDDL), and Difference in positive proportions in observed labels (DPL), and
Mutual Information (MI) between the sensitive attribute and the rest of the
features, since Gupta et al. (2021) showed that group disparities are bounded
by this estimate. However, these approaches do not provide evaluations that
consider the characteristics of downstream ML models in a given pipeline. This
limitation makes it challenging to answer questions such as which type of
model is more likely to produce disparate outcomes when applied on the
assessed dataset.

Prior research involves training a diverse set of models, followed by the
computation of post-training metrics. This approach, discussed by Hardt et al.
(2021); Vasquez et al. (2022), serve two main purposes: firstly, to identify
any existing discrimination (e.g., through Demographic Parity (DP), Accuracy
Difference (AD), and Equalized Odd (ODD)); and secondly, to generate
explanations of disparate predictions (e.g., through KernelSHAP). Despite the
utility of this strategy for the late steps of a ML pipeline, they are
applicable only to specific trained models of the pipeline. Therefore, if the
models are retrained, or their hyperparameters are altered, these metrics need
to be recalculated.

Irrespective of the advancements made, we still lack assessment approaches
that are insightful, specific for the early stages of a ML pipeline, and, as a
critical need, to be less dependent on specific instances of trained models of
the ML pipeline. This work addresses this gap by leveraging recent research on
usable information–actionable information under statistical and computational
limitations Xu et al. (2020); Ethayarajh et al. (2022); Hewitt et al. (2021)–,
and introduces DispaRisk, a framework for detecting and interpreting disparity
risks in datasets prior to their use in downstream ML workflows. Given a ML
pipeline context, DispaRisk: (1) identifies datasets that are more likely to
lead to disparate outcomes; (2) identifies model families prone to produce
disparities; (3) generates insights about the features that leak information
about sensitive attributes; and (4) identifies individuals at a high-risk of
being discriminated.

##  2 Fairness in Algorithms

All subsequent discussions are based on the setting described in this section.
For a given ML pipeline, let $X,S,Y$ be random variables in
$\mathcal{X}\times\mathcal{S}\times\mathcal{Y}$ representing features of
individuals, the sensitive attributes, and a target variable to be predicted
by a mapping function $f:X\mapsto Y$, respectively. Also, let $\mathcal{A}$ be
an agent with access to a space of possible mapping functions, which is
constrained by computational and statistical constraints. Additionally, we
assume that $\mathcal{A}$ has access to samples
$(x_{i},s_{i},y_{i})\in\mathcal{X}\times\mathcal{S}\times\mathcal{Y}$, and
that learns a mapping function $f$ from a given dataset
$\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{n}$.

Given this setting, we now outline three common fairness notions that will be
used in this study, and then explore the relationship of fairness with recent
information theories.

###  2.1 Fairness Notions.

Fairness in algorithms is commonly certified through independence ($f(X)\perp
S$), separation ($f(X)\perp S|Y$), or individual notions ($f(x_{i})\approx
f(x_{j})\quad\forall i,j$ s.t. $d(x_{i},x_{j})<\epsilon$, with $\epsilon>0$
and $d$ a distance function) Feldman et al. (2015); Hardt et al. (2016);
Pessach and Shmueli (2023); Vasquez et al. (2022). Pessach and Shmueli (2023)
provide a comprehensive survey of which we replicate the three most commonly
used metrics:

######  Definition 1 (Demographic Parity (DP)).

Demographic Parity evaluates the difference in the predictive positive rate
across demographic groups:

$\displaystyle\begin{aligned} \Delta_{DP}(f,S)=\max_{s\in S,s`\in
S}&\left|P(f(X)=1|S=s)\right.\\\
&\left.-P(f(X)=1|S=s^{\prime})\right|\end{aligned}$

######  Definition 2 (Equalized Odds (ODD)).

Equalized Odds measures the difference in the true and false positive rates
across demographic groups:

$\displaystyle\begin{aligned} \Delta_{ODD}(f,&S,Y)=&\\\ &\max_{s\in S,s`\in
S}\frac{1}{2}&\sum_{c\in\\{0,1\\}}\left|P(f(X)=1|S=s,Y=c)\right.\\\
&&\left.-P(f(X)=1|S=s^{\prime},Y=c)\right|\end{aligned}$

######  Definition 3 (Equalized Opportunity (OPP)).

Equalized Opportunity is a weaker requirement of Equalized Odds, focusing only
on the true positive rate:

$\displaystyle\Delta_{OPP}(f,S)=\max_{s\in S,s`\in S}$
$\displaystyle\left|P(f(X)=1|S=s,Y=1)\right.$
$\displaystyle\left.-P(f(X)=1|S=s^{\prime},Y=1)\right|$

###  2.2 Fairness Through Usable Information.

Mutual Information (MI) measures the reduction in uncertainty of one random
variable due to the knowledge of another Cover and Thomas (2006). It is
formally defined as $I(A;B)=H(A)-H(A|B)$, where $A$ and $B$ are random
variables, and $H(\cdot)$ is the entropy. Previous studies leveraged MI to
mitigate bias in datasets by learning a representation $Z$ of the input space
that minimizes the amount of information about the sensitive attribute $S$
Gitiaux and Rangwala (2021, 2022); Gupta et al. (2021). These approaches are
substantiated by the theorem shown by Gupta et al. (2021), which demonstrates
that for any mapping function $f$ acting on $Z$, the respective DP is bounded
by $I(Z;S)$. Consequently, any reduction in $I(Z;S)$ will also decrease the
disparity of $f$ acting on $Z$. The underlying intuition is that the more
information $Z$ contains about $S$, the higher is the risk of disparate
outcomes for any function $f$ acting on $Z$. Based on this notion, one could
assess a given dataset by estimating $I(X;S)$, and generate insights on risks
of disparities by assessing whether a higher estimation of MI is obtained.
However, while this evaluation approach is useful for identifying the risk
that any function $f$ acting on $X$ might pose, it does not allow to
distinguish which specific type of function $f$ is more likely to result in
higher disparate outcomes, since the estimation of MI serves as an upper bound
for all functions. To address this limitation, we propose to employ usable
information, a concept that generalizes Shannon information formulated by Xu
et al. (2020). whose underlying intuition usable information and formulation
are described next.

Usable information intuitively suggests that different agents using the same
dataset for a task can leverage a different amount of information, depending
on their computational capabilities. Thus, from each agent’s perspective, the
dataset is a resource of a varying amount of usable information. For example,
consider the case of a decryption task. Under Shannon information theory, the
original and encrypted messages have high MI; thus, two different decrypters
with unlimited computational capabilities should be able to obtain the
original message. However, in practice, the assumption of unlimited
computational resources is not met, and their success depends on how much
usable information is in the encrypted message for each decrypter. Consider a
second example where, for binary classification, one of the agents has access
only to logistic regression mapping functions, while the second has access to
mapping functions comprised by affine functions followed by softmax. Similar
to the decryption task, by computing the usable information on the target
variable for each agent, we can now assess which one is more prone to produce
better predictions. Indeed, by adhering to the fundamental principle of the
theorem of Gupta et al. (2021), we can compute the usable information on the
sensitive attribute for each agent, and estimate which agent is at a higher
risk of discriminating by using the corresponding family of mapping functions.

From Xu et al. (2020), the constraints of an agent are modeled as a predictive
family $\mathcal{V}$, which represents a collection of predictive models an
agent can use for their tasks. As such, a predictive family may be restricted
for statistical constraints or computational limitations, and it must satisfy
optional ignorance. For the formal definitions of these two, let
$\mathcal{P}(\mathcal{Y})$ represent the set of all probability measures on
$\mathcal{Y}$, and $\varnothing$ the absence of any side information. Side
information refers to samples from a feature space. For instance, in toxicity
analysis, a side information can be a text sampled from a corpus of tweets
$X$, and $\varnothing$ a post of zero characters. Given this, a predictive
family is formally defined as Xu et al. (2020):

######  Definition 4 (Predictive Family).

A set
$\mathcal{V}\subseteq\Omega=\\{f:\mathcal{X}\cup\varnothing\mapsto\mathcal{P}(\mathcal{Y})\\}$
is a predictive family if it satisfies the optional ignorance condition:

$\displaystyle\begin{aligned} \forall f\in\mathcal{V},\forall
P\in\textrm{range}(f),\quad\exists f^{\prime}\in\mathcal{V},\\\
\textrm{s.t}\quad\forall
x\in\mathcal{X},f^{\prime}[x]=P,f^{\prime}[\varnothing]=P\end{aligned}$

where $range(f)$ is the range of function $f$.

The _optional ignorance_ condition captures the ability to disregard the side
information when making predictions. For example, consider a predictive family
built with linear regression functions. By setting certain parameters
associated with a subset of features to $0$, agent can ignore those features
and generate predictions of the target variable $Y$ within the set
$\mathcal{P(\mathcal{Y})}$. The fulfillment of this condition is crucial to
achieve desirable properties described at the end of this section.

We now reproduce the definition of predictive $\mathcal{V}$-_information_ ,
also from Xu et al. (2020), which refers to the measure of the change in the
predictability of variable $Y$ when additional side information $X$ is
provided to a model in family $\mathcal{V}$. Formally:

######  Definition 5 (Predictive $\mathcal{V}$-_information_).

Given the random variables $X,Y$ taking values in
$\mathcal{X}\times\mathcal{Y}$ and the predictive family
$\mathcal{V}\subseteq\Omega=\\{f:\mathcal{X}\cup{\varnothing}\mapsto\mathcal{P}(\mathcal{Y})\\}$,
the _predictive_ $\mathcal{V}$-_information_ from $X$ to $Y$ is defined as:

$I_{\mathcal{V}}(X\mapsto
Y)=H_{\mathcal{V}}(Y|\varnothing)-H_{\mathcal{V}}(Y|X)$

where $H_{\mathcal{V}}(\cdot)$ is the predictive $\mathcal{V}$-entropy:

$\displaystyle H_{\mathcal{V}}(Y|X)$
$\displaystyle=\inf_{f\in\mathcal{V}}\mathbb{E}_{x,y\sim X,Y}[-\log f[x](y)]$
$\displaystyle H_{\mathcal{V}}(Y|\varnothing)$
$\displaystyle=\inf_{f\in\mathcal{V}}\mathbb{E}_{x,y\sim X,Y}[-\log
f[\varnothing](y)]$

with $f[x]\in\mathcal{P}(\mathcal{Y})$ chosen for the given $x$;
$f[\varnothing]$ implies that no side information is provided; and $f[x](y)$
represents the density value evaluated at $y\in\mathcal{Y}$. The base of the
$\log$ determines the units of information: $2$ for bits and $e$ for nats.

Finally, $\mathcal{V}$-_information_ metrics satisfy the desirable properties
listed below Xu et al. (2020). From these properties, we can infer that if
$\mathcal{M}\subseteq\mathcal{V}$, then $I_{\mathcal{M}}(X\mapsto Y)\leq
I_{\mathcal{V}}(X\mapsto Y)$. Therefore, if $\mathcal{V}$ is large enough to
contain all possible models, i.e. $\mathcal{V}=\Omega$, the
$\mathcal{V}$-_information_ reduces to Shannon information.

  1. 1.

Non-Negativity: $I_{\mathcal{V}}(X\mapsto Y)\geq 0$.

  2. 2.

Independence: If $X$ is independent of $Y$, $I_{\mathcal{V}}(X\mapsto
Y)=I_{\mathcal{V}}(Y\mapsto X)=0$.

  3. 3.

Monotonicity: If $\mathcal{V}\subseteq\mathcal{U}$, then
$H_{\mathcal{V}}(Y)\geq H_{\mathcal{U}}(Y)$, $H_{\mathcal{V}}(Y|X)\geq
H_{\mathcal{V}}(Y|X)$.

Building on the theory discussed thus far, in the following section we present
DispaRisk.

##  3 The DispaRisk Framework.

DispaRisk operates on two inputs: the sensitive attribute $S$, and the
computational constraints $\mathcal{V}$ of the downstream workflow.
$\mathcal{V}$ must satisfy the optional ignorance condition. Given these
inputs, we can compute different estimates for $\mathcal{V}$-_information_ ,
which is crucial to identify and interpret disparities in datasets. We
categorize the corresponding estimations in three types of analyses: (1)
dataset-level, (2) slicing, and (3) instance-level.

Dataset-level: first we assess whether $X$, which is used to predict $Y$,
contains a significant amount of $\mathcal{V}$-_information_ for predicting
the variable $S$. If it does, models in $\mathcal{V}$ are prone to produce
biased outcomes. Consequently, the first metric to estimate is
$I_{\mathcal{V}}(X\mapsto
S)=H_{\mathcal{V}}(S|\varnothing)-H_{\mathcal{V}}(S|X)$. This proposed metric
can generate insights through dataset-level comparisons. For example, given a
dataset and several predictive families, this metric can issue warnings about
the families that are likely to be more discriminatory. Similarly, given a
predictive family and multiple datasets for the same task, the metric can
assist in identifying which dataset poses higher risks when used to learn a
function from the family.

Slicing: the second assessment consists in finding the subsets of features
that might potentially contribute to higher disparities. We call this goal
slicing analysis. To this end, we build on the work of Hewitt et al. (2021),
whose purpose is to capture the reduction in predictability when a certain
subset of input features is known already. From Hewitt et al. (2021), let’s
assume the input space is partitioned into two subsets: a known subset $C$ and
an uknown subset $\bar{C}$, where features in $\bar{C}$ are replaced by a
constant value $\bar{a}$. Hewitt et al. (2021) formulate conditional
$\mathcal{V}$-_information_ as follows:

######  Definition 6 (Conditional $\mathcal{V}$-_information_).

Given a predictive family $\mathcal{V}\subseteq\Omega$, the predictive
$\mathcal{V}$-_information_ from $X_{l}$ to $Y$ conditioned on the prior
knowledge of $C\subset\\{X_{1},\dots,X_{n}\\}=X$ is defined as:

$I_{\mathcal{V}}(X_{l}\mapsto
Y|C)=H_{\mathcal{V}}(Y|C)-H_{\mathcal{V}}(Y|C\cup{X_{l}})$

where $\mathcal{V}$-_entropy_ $H_{\mathcal{V}}(\cdot|\cdot)$ is now defined
as:

$H_{\mathcal{V}}(Y|C)=\inf_{f\in V}\mathbb{E}_{c,y}\left[-\log
f(c,\bar{a})[y]\right]$

with $c\in C$ and $\bar{a}\in\bar{\mathcal{C}}$.

We propose to estimate the conditional $\mathcal{V}$-_information_ for
predicting the sensitive attribute and the target variable, given a specific
feature $X_{l}$. This is represented as $I_{\mathcal{V}}(X_{l}\mapsto
T)=H_{\mathcal{V}}(T|\varnothing)-H_{\mathcal{V}}(T|X_{l})$, where $T$ is set
to either $S$ or $Y$. We also propose to measure the amount of
$\mathcal{V}$-_information_ provided by variable $l$, information otherwise
not present in the input space when $X_{l}$ is excluded. To this end, we
define $C_{-l}=\\{X_{1},\dots,X_{n}\\}\setminus X_{l}$, where
$l\in\\{1,\dots,n\\}$. The wanted conditional $\mathcal{V}$-_information_ is
then given by: $I_{\mathcal{V}}(X\mapsto
T|C_{-l})=H_{\mathcal{V}}(T|C_{-l})-H_{\mathcal{V}}(T|X)$.

The fundamental intuition behind the proposed estimations is to determine
whether, for a given ML pipeline, we are in an ideal low-risk scenario
concerning the variable $X_{l}$. This scenario is characterized by high
$\mathcal{V}$-_information_ values for predicting the sensitive attribute $S$,
and low values for the prediction of the target variable $Y$. This scenario is
desirable because slicing out $X_{l}$ from the input space would not
compromise the useful information needed to predict $Y$. Any other scenarios
would indicate a risk of disparities. For instance, a scenario with high
values for both $I_{\mathcal{V}}(X_{l}\mapsto S)$ and
$I_{\mathcal{V}}(X_{l}\mapsto Y)$, but low values for
$\mathcal{V}$-_information_ when conditioned on $C_{-l}$, implies a
significant risk of disparity. In fact, in this case $X_{l}$ likely leaks
usable information about $S$; as such, outcomes can result in disparate
predictions, and the removal of $X_{l}$ cannot solve the problem since its
$\mathcal{V}$-_information_ is already encompassed within the other features.
In this scenario, the dataset presents a high level of risk and poses
substantial challenges for the implementation of mitigation approaches.

Instance-level: here, we aim to identify subsets of instances that are
potentially at risk of being discriminated. We build on the work of Ethayarajh
et al. (2022), which introduces the pointwise $\mathcal{V}$-_information_
metric to estimate the degree of usable information in individual instances.
This metric is defined as the difference in log-likelihood predictions of the
variable to be predicted (in this case, we assume $s$) when no side
information $\varnothing$ is given, and when side information $x$ is provided:

######  Definition 7 (Pointwise $\mathcal{V}$-_information_ (PVI)).

Given random variables $X$ and $S$, and a predictive family $\mathcal{V}$, the
pointwise $\mathcal{V}$-_information_ (PVI) of $(x,s)$ is

$\texttt{PVI}(x\mapsto s)=-\log f^{\prime}[\varnothing](s)+\log f[x](s)$

where $f^{\prime}\in\mathcal{V}\textrm{ s.t. }\mathbb{E}[-\log
f^{\prime}[\varnothing](Y)]=H_{\mathcal{V}}(Y)$ and $f\in\mathcal{V}\textrm{
s.t. }\mathbb{E}[-\log f[X](Y)]=H_{\mathcal{V}}(Y|X)$.

Unlike $\mathcal{V}$-_information_ , PVI can be negative. Intuitively,
$\texttt{PVI}<0$ implies that $f\in\mathcal{V}$ is better at predicting $s$
when $\varnothing$ is given, rather than $x$ (i.e., by outputting the majority
class). This can occur due to various factors, such as mislabeling Ethayarajh
et al. (2022), or an intrinsic error due to the underlying uncertainty in the
ground truth.

For instance-level analysis, we propose estimating
$\texttt{PVI}_{\mathcal{V}}(x\mapsto s)$ where negative values suggest that
models in $\mathcal{V}$ are more accurate at predicting sensitive attributes
by simply guessing the majority class rather than using the side information
$x$. Conversely, positive values indicate that the models are better off using
the side information to predict $s$. From a fairness perspective, both cases
are problematic: when the model guesses the sensitive attribute to predict
$y$, it may exacerbate disparities for underrepresented groups, and if a model
can accurately predict $s$ using information in $x$, it may be inadvertently
identifying a feature highly correlated to the sensitive attribute, or it may
reconstruct it. In contrast, a $\texttt{PVI}_{\mathcal{V}}$ around $0$
indicates less disparity risks, making it the preferable case.

##  4 Experimental Results

###  4.1 Datasets

We evaluated DispaRisk on four tabular and an image fairness benchmarks.

Census Income (CI): CI has $48,844$ individuals with $14$ features Dheeru and
Taniskidou (2017). The sensitive attribute is gender. The target variable
encodes whether the income is greater than $50K$ Asuncion and Newman (2007).

Dutch Census (DC): this dataset has $60,420$ observations of Dutch individuals
from 1973-1975 Le Quy et al. (2022) with $12$ features, such as age and
education. The sensitive attribute is sex and the binary target is the
occupation feature with high income level and low income level classes.

Compas Recidivism (CR): this dataset has $7,214$ observations of individuals
in the criminal justice system with features on felonies, misdemeanors,
charges, age, sex, race and jailed days. Race is the sensitive attribute,
describing whether an individual is African-American or not. The target
variable encodes whether the individual re-offended in two years ProPublica
(2016).

Credit Card (CC): this dataset has $30,000$ observations of Taiwanese
customres’ default payments in October 2005 with 24 features Le Quy et al.
(2022). We use sex as sensitive attribute and as target variable whether
customer faces a default situation in the following month.

CelebFaces Attribute (CelebA): This dataset includes over $200,000$ face
images of celebrities. The source provides also 39 binary attributes and 5
landmarks that specify the pixel location of five facial features Liu et al.
(2015). Following Zhang et al Zhang et al. (2022), gender is the sensitive
attribute and the binary variable BlondHair is the target.

For the each dataset, we use the train/test partitions given by the source of
the data. If no partitions are given, we randomly split the dataset using a
$70/30$ ratio for training and test sets. The training sets are used to fine-
tune the predictive families and the test sets are used to estimate the
$\mathcal{V}$-_information_ measures. During training, we estimate
performances under cross-validation and one held-out validation set strategies
in tabular and CelebA datasets, respectively. In addition, we apply label
encoding to categorical variables and scale all features by subtracting the
mean and normalizing into unit variance. For the CelebA dataset, following
Zhang et al Zhang et al. (2022), the images are resized, cropped and flipped.

Table 1: $I_{\mathcal{V}}(X\mapsto S)$ (abbreviated as $I_{\mathcal{V}}(S)$), Demographic Parity, and Accuracy on predicting $Y$ for each $\mathcal{V}$ across tabular datasets. Reported values are computed as the mean of the top $25\%$ performing models. The $I_{\mathcal{V}}(X\mapsto S)$, metric helps identify predictive families with higher unfairness concerns on a given dataset. Bold font indicates highest values. $\mathcal{V}$-family | Census Income | Dutch Census | Compas Recidivism | Credit Card
---|---|---|---|---
$I_{\mathcal{V}}(S)$ | $\Delta_{DP}$ | acc | $I_{\mathcal{V}}(S)$ | $\Delta_{DP}$ | acc | $I_{\mathcal{V}}(S)$ | $\Delta_{DP}$ | acc | $I_{\mathcal{V}}(S)$ | $\Delta_{DP}$ | acc
Linear | .185 | .132 | .820 | .060 | .186 | .805 | .042 | .199 | .688 | .010 | .017 | .811
1MLP | .317 | .176 | .851 | .099 | .196 | .821 | .046 | .211 | .692 | .024 | .020 | .819
2MLP | .313 | .169 | .851 | .099 | .192 | .822 | .046 | .215 | .690 | .022 | .018 | .819
3MLP | .310 | .168 | .851 | .103 | .193 | .822 | .046 | .220 | .692 | .025 | .018 | .819

###  4.2 Predictive Families and Estimates

We explore five predictive families. The first family consists of linear
classifier models followed by the sigmoid function $\sigma$; we call this
family Linear and set $w$ and $b$ its parameters. The next three families
consist of nested affine function classifiers with ReLU as activation function
and a sigmoid at the top. We construct the three families with Multi-Layer
Perceptrons (MLP) function, denoting with 1MLP, 2MLP, and 3MLP the one-layer,
two-layers, and 3-layers MLP, respectively. We also use $m_{i}$ to denote the
perceptron function, where $i$ indicates the number of layers. The last two
families are used only for the CelebA dataset, and consist of convolutional
neural networks (CNNs), followed by fully connected layers and a softmax
output layer. We follow ResNet18 (RN) He et al. (2016) and AlexNet (AN)
Krizhevsky et al. (2017) architectures to construct two families of CNNs:

$\displaystyle\mathcal{V}_{Linear}$ $\displaystyle=\left\\{h:X\mapsto
f[x](y)=\sigma(-w[x]+b)\right\\}$ $\displaystyle\mathcal{V}_{1MLP}$
$\displaystyle=\left\\{h:X\mapsto f[x](y)=\sigma(m_{1}(x))\right\\}$
$\displaystyle\mathcal{V}_{2MLP}$ $\displaystyle=\left\\{h:X\mapsto
f[x](y)=\sigma(m_{2}(x))\right\\}$ $\displaystyle\mathcal{V}_{3MLP}$
$\displaystyle=\left\\{h:X\mapsto f[x](y)=\sigma(m_{3}(x))\right\\}$
$\displaystyle\mathcal{V}_{RN}$ $\displaystyle=\left\\{h:X\cup S\mapsto
RN(x)\right\\}$ $\displaystyle\mathcal{V}_{AN}$ $\displaystyle=\left\\{h:X\cup
S\mapsto AN(x)\right\\}$

Algorithm 1 Predictive $\mathcal{V}$-_information_ and pointwise
$\mathcal{V}$-_information_ estimates

Input: $\mathcal{D}_{train}=\\{(x_{i},t_{i})\\}_{i=1}^{k}$,
$\mathcal{D}_{test}=\\{(x_{i},t_{i})\\}_{i=1}^{m}$ with $k+m=n$ and $t_{i}$ as
the target label of the point $i$, family $\mathcal{V}$ and subset $C\subseteq
X$, subset $\bar{C}$ with $\bar{C}\cap C=\emptyset$
Output: $\hat{I}_{\mathcal{V}}$ and PVI estimates.
do

$g\leftarrow$ fine-tune $\mathcal{V}$ on $\mathcal{D}_{train}$

$g^{\prime}\leftarrow$ fine-tune $\mathcal{V}$ on $\mathcal{D}_{train}$ with
partition $C,\bar{C}$

$H_{\mathcal{V}}(T|X)\leftarrow 0$

$H_{\mathcal{V}}(T|C)\leftarrow 0$

for $(x_{i},t_{i})\in\mathcal{D}_{test}$ do

$H_{\mathcal{V}}(T|X)\leftarrow H_{\mathcal{V}}(T|X)-\frac{1}{m}\log
g[x_{i}](t_{i})$

$H_{\mathcal{V}}(T|C)\leftarrow H_{\mathcal{V}}(T|C)-\frac{1}{m}\log
g^{\prime}[c_{i},\bar{c}_{i}](t_{i})$

PVI$(x_{i}\mapsto t_{i})\leftarrow-\log
g^{\prime}[c_{i},\bar{c}_{i}](t_{i})+\log g[x_{i}](t_{i})$

end for

$\hat{I}_{\mathcal{V}}(X\mapsto
T|C)=H_{\mathcal{V}}(T|C)-H_{\mathcal{V}}(T|X)$

end do

Figure 1: $t(P(S=1),\Delta_{DP})$ and $I_{\mathcal{V}}(X\mapsto S)$ estimates
in CI datasets from samplings datasets such that $P(S=1)=k$ with
$k\in\\{0.05,0.07,\dots,0.95\\}$, leading to different levels of $\Delta_{DP}$
of models trained on them.

The estimate of $\mathcal{V}$-_information_ involves fitting models to the
training set by minimizing the negative log-loss. We establish two early-stop
criterion: a tolerance of $1-e4$ to prevent over-fitting and a max number of
epochs equal to $1000$. The loss is computed on a validation set, of size
$10\%$ of train set, at each epoch. If the improvement in loss does not exceed
the set tolerance value in ten consecutive epochs or the max number of epochs
is reached, the training is halted. The model exhibiting the minimal loss on
the validation set is used for further analysis. For $\mathcal{V}_{Linear}$ we
apply the $L_{1}$ and $L_{2}$-norm penalty, and for the MLP families we
experiment with layer sizes of $10$, $100$, and $500$. For each training, we
use the Adam optimizer with a learning rates in
$\\{1e-5,1e-4,1e-3,1e-2,1e-1,1.0\\}$, $\beta_{1}=0.9$ and $\beta_{2}=0.99$,
and batch size of $512$. To ensure that our results are robust to model
selection and initialization, we train multiple models with different
hyperparameters and seeds, and select the one with the minimal loss function
for downstream fairness analysis.

Once the models have been trained, we estimate the $\mathcal{V}$-entropy, the
$\mathcal{V}$-_information_ , and the PVI on the test set by using Algorithm
1. This has computational complexity of $O(n)$. Then, we set $T$ and the
partitions $C,\bar{C}$ accordingly. In the algorithm, $T$ denotes the variable
to be predicted and can be the target or the sensitive attribute. For
instance, to estimate $I_{\mathcal{V}}(X\mapsto S)$, we set $T=S$ and the
subset $C=\varnothing$; for $I_{\mathcal{V}}(X_{l}\mapsto Y|C_{-l})$, we set
$T=Y$ and the subset $C=C_{-l}=\\{X_{1},\dots,X_{d}\\}\setminus X_{l}$.
Additionally, we use $\ln$, hence estimates are in natural unit of
information. The estimate procedure assumes a large dataset size Xu et al.
(2020). For each dataset, we found that after using about $90\%$ of the
training data, the estimates reach a plateau. More details are in the
supplementary material. Finally, experiments were run with 4x NVIDIA
A100-SXM-80GB GPU and 2x AMD MILAN, taking 62.97 minutes on average for fine-
tuning in the largest dataset (CelebA).

###  4.3 Analysis.

In this section we present major findings grouped in four research questions
(RQ). More results that support our conclusions are available in the
supplementary material.

RQ1: Can we use predictive $\mathcal{V}$-_information_ to measure the
disparities induced by the dataset in downstream ML models? We generate
multiple trials of each tabular dataset by resampling the majority group with
replacement, representing different sample collection methods. For each trial,
we estimate $I_{\mathcal{V}}(X\mapsto S)$ and $\Delta_{DP}$, and depict the
results in Figure 1 for CI dataset. See Supplementary material for the rest
datasts. From the figure, higher disparities clearly correlate with higher
predictive $\mathcal{V}$-_information_ , supporting the intuition developed in
Section 2.2. Hence, stakeholders can rely on usable information for early
identification of disparity risks in datasets within the context of a ML
pipeline.

Table 2: $I_{\mathcal{V}}(X\mapsto S)$, Demographic Parity, and Accuracy on
predicting $Y$ reported for each $\mathcal{V}$ across tabular datasets. The
$I_{\mathcal{V}}(X\mapsto S)$ metric helps identify predictive families with
higher unfairness concerns on a given dataset. Bold font indicates highest
values.

$\mathcal{V}$-family | CelebA
---|---
$I_{\mathcal{V}}(X\mapsto S)$ | $\Delta_{DP}$ | acc
Linear | .222 | .103 | .899
1MLP | .358 | .148 | .934
2MLP | .360 | .156 | .936
3MLP | .367 | .153 | .936
RN | .620 | .191 | .960
AN | .606 | .186 | .959

Table 3: Predictive $\mathcal{V}$-_information_ for $S$ and $Y$ from $X_{l}$
in CR, given $\mathcal{V}_{1MLP}$. Values are sorted by
$I_{\mathcal{V}}(X_{l}\mapsto S)$ and shown in nats of information.

Feature | $I_{\mathcal{V}}(X_{l}\mapsto S)$ | $I_{\mathcal{V}}(X_{l}\mapsto Y)$
---|---|---
priors_count | .026 | .061
age | .010 | .019
age_cat | .008 | .013
juv_misd_count | .007 | .010
juv_fel_count | .005 | .007
c_charge_degree | .003 | .006
juv_other_count | .002 | .009
c_days_jail | .001 | .001
sex | $<.001$ | .004

Table 4: Conditional $\mathcal{V}$-_information_ for predicting $S$ and $Y$
from $X_{l}$ in the CR, given $\mathcal{V}_{1MLP}$. Values are sorted by
$I_{\mathcal{V}}(X\mapsto S|C_{-l})$ and shown in nats of information.

Feature | $I_{\mathcal{V}}(X\mapsto S|C_{-l})$ | $I_{\mathcal{V}}(X\mapsto Y|C_{-l})$
---|---|---
priors_count | .024 | .061
age_cat | .001 | .001
age | .001 | .010
c_charge_degree | $<.001$ | $<.001$
c_days_jail | $<.001$ | .002
sex | $<.001$ | .002
juv_misd_count | $<.001$ | $<.001$
juv_fel_count | $<.001$ | $<.001$
juv_other_count | $<.001$ | .002

RQ2: Which predictive families tend to produce higher levels of disparities
when measured with standard fairness metrics for a given dataset? Tables 1 and
4 present the accuracy and demographic parity of models, and the estimates
following the dataset-level analysis. From the tables, we observe a strong
correlation between $I_{\mathcal{V}}(X\mapsto S)$ and $\Delta_{DP}$.
Furthermore, we find that models with higher $\mathcal{V}$-_information_ tend
to be more accurate. This emphasizes disparity risks, as these models are more
likely to be finally deployed. Therefore, DispaRisk provides early information
to stakeholders about the potential challenge of balancing performance and
fairness.

Table 5: The $\mathcal{V}_{AN}$-_information_ of features in the CelebA dataset. We use the landmarks provided in the source. We also use square masks of different sizes, $10$, $20$, and $40$ pixels, and combined features. Specifically, eyes: left_eye+right_eye; mouth: left_mouth+right_mouth; and all: eyes+mouth+nose. Additionally, we add a less biased features positioned in the bottom-right corner of the pictures. We group the features in two groups: single (top) and mixed (bottom), and apply bold font to the two highest estimates for a given size of mask.  Feature | $I_{\mathcal{V}}(X_{l}\mapsto S)$ | $I_{\mathcal{V}}(X_{l}\mapsto Y)$ | $I_{\mathcal{V}}(X\mapsto S|C_{-l})$ | $I_{\mathcal{V}}(X\mapsto Y|C_{-l})$
---|---|---|---|---
10p | 20p | 40p | 10p | 20p | 40p | 10p | 20p | 40p | 10p | 20p | 40p
left_eye | .288 | .453 | .526 | .032 | .083 | .171 | .004 | .003 | .041 | .002 | .002 | .005
right_eye | .297 | .460 | .529 | .034 | .081 | .158 | .006 | .006 | .050 | .003 | .003 | .009
left_mouth | .190 | .334 | .458 | .018 | .035 | .097 | .005 | .003 | .020 | .001 | .003 | .003
right_mouth | .189 | .335 | .466 | .020 | .035 | .080 | .005 | .001 | .013 | .001 | .002 | .003
nose | .086 | .249 | .471 | .010 | .033 | .053 | .006 | .014 | .025 | .002 | .003 | .004
bottom_right | .061 | .105 | .167 | .035 | .066 | .109 | .001 | .002 | .001 | .002 | .001 | .002
eyes | .360 | .509 | .567 | .042 | .098 | .208 | .170 | .173 | .184 | .014 | .016 | .032
mouth | .264 | .423 | .526 | .023 | .046 | .128 | .017 | .026 | .045 | .002 | .005 | .013
eyes+nose | .413 | .536 | .585 | .046 | .107 | .211 | .137 | .254 | .141 | .015 | .027 | .042
mouth+nose | .326 | .464 | .549 | .033 | .053 | .134 | .023 | .032 | .064 | .006 | .006 | .012
all | .475 | .562 | .596 | .059 | .111 | .216 | .234 | .326 | .197 | .025 | .028 | .069
Figure 2: Fairness metrics for sorted $0.5$-PVI-size-bins of DC, ranging from
$-1.5$ to $1.1$, with steps of $0.3$.

RQ3: Which features contribute more to disparities? To answer this question,
we follow the slicing analysis of DispaRisk. First, we select the family with
higher accuracy to estimate the $\mathcal{V}$-_information_ of each feature
$X_{l}$ to predict the sensitive attribute $S$ and the target $Y$. We report
here the results on the CR dataset with $\mathcal{V}_{1MLP}$; results for the
remaining tabular datasets are included in the Supplementary Material. Table 4
shows the estimates when using only feature $X_{l}$, while Table 4 depicts the
$\mathcal{V}$-_information_ when using all features except $X_{l}$. In each
table, the results are sorted by predictive information with respect to $S$.
From Table 4, it is evident that “previous criminal history” (priors_count)
and “age” have the strongest usable signals for $\mathcal{V}_{1MLP}$ in
predicting whether the defendant will re-offend within two years. However,
providing these features as side information might increase the risk of
disparities since they also have the highest usable information in
relationship to race, the sensitive attribute. This increment depends on which
proportion of their predictive information is already in the input space, with
lower values about $S$ and higher levels about $Y$ being preferable. Then, we
re-estimate the $\mathcal{V}$-_information_ conditioned on $C_{-l}$, and the
results are shown in Table 4. From the table, “previous criminal history” and
“age” have the highest conditional predictive information for both $S$ and
$Y$. This means that removing them from the input space can improve fairness,
but will reduce the predicting power of the target variable for
$\mathcal{V}_{1MLP}$, while using them poses a risk. Overall, the derived
risk-information can be given to stakeholders, who can then pursue an informed
and non-trivial pre-processing and mitigating approaches, focused on the
involved variables.

We apply the same analysis on the CelebA dataset using AN. We assume that the
provided five landmarks are the features of interest for analysis.
Additionally, we introduce a sixth feature called bottom-right which
represents a less biased feature, i.e., it has a lower amount of predictive
information about gender. To capture the information from these features, we
select a square region (with a $10$, $20$, or $40$-pixel sizes) centered at
the landmarks, and mask with a value of $1.0$ the pixels within or outside the
square, depending on the desired estimate. Table 5 presents the estimated
$\mathcal{V}_{AN}$-_information_ for each feature in predicting sensitive and
target variables. We observe that varying the mask size results in an increase
in predictive information; larger mask sizes provide the models with more
usable information, thereby enhancing their possible predictive capabilities.
Moreover, we notice a substantial amount of predictive information contained
in the “eyes” feature for $\mathcal{V}_{AN}$, enabling accurate predictions of
both the sensitive (gender) and the target variable (BlondHair. However, as
discussed previously, this presents an undesirable scenario. The removal of
“eyes” features eliminates a relevant amount of usable information pertaining
to the gender, but this comes at the expense of losing significant information
about “hair color”. This shows that mitigating disparity in CelebA datasets,
such as learning a fair representation, is challenging for the
$\mathcal{V}_{AN}$ predictive family.

RQ4: Which individuals are at a higher risk of being discriminated against?
Following DispaRisk’s Instance-level analysis, we estimated
$\texttt{PVI}_{\mathcal{V}}(x\mapsto s)$ for each individual, sorted the
estimated values, and grouped them into $p$-sized discretized bins starting at
$-1.5$ and finishing at $1.1$ with steps of $t\leq p$. We then calculated
$\Delta_{DP}$, $\Delta_{ODD}$, and $\Delta_{OPP}$ for each bin. The objective
is to demonstrate the claim made in Section 3 that individuals with PVI closer
to $0$ are less likely to face discrimination. Figure 2 depicts results for
the DC dataset. See the supplementary material for the remaining datasets.

From the figures, the individual bins with PVI closer to $-1.5$ or $1.1$
exhibit the higher disparities, while those closer to $0$ show the lower
levels of unfairness. This suggests that finding a proper representation where
PVIs are reduced to $0$ is promising for mitigating disparities in the
downstream pipeline. Moreover, tailoring the representation to the predictive
family rather than creating a general representation for every model may be
advantageous in this regard.

Finally, we can also find individuals profiles with high risks of
discrimination by using their PVIs estimates. Figure 3 shows the mean of
“previous criminal history” and “age” in the CR dataset of each bins of PVI
estimates. From the figure, when guessing (negative PVI) or using side
information (positive PVI), we observe a larger difference between demographic
groups. In particular, older individuals with more prior felonies are likely
to face discrimination from the $\mathcal{V}_{1MLP}$ family models.

Figure 3: priors_count and age averages for each demographic group sorted by
PVI.

##  5 Conclusions

We introduced DispaRisk, a framework that provides insights into disparity
risks for downstream workflows. Data owners can use DispaRisk to choose proper
pre-processing mitigation approaches to reduce disparity risks. DispaRisk also
guides feature-level and instance-level analysis at early stages of a ML
pipeline. Our experiments showcase that the estimates from DispaRisk have the
following desirable properties: (1) they correlate with disparities, enabling
early identification of discrimination risks; (2) facilitate the
identification of risky families, eliminating the need to recompute post-
processing metrics dependent on and specific for pre-trained models; (3)
provide explainability insights about which features may increase risks due to
the substantial information of sensitive attribute, and allow for flagging
challenging datasets for fairness mitigation tasks; and (4) enable the
identification of individual profiles in the dataset at high risk.
Furthermore, auditing datasets by using usable information showed virtues in
our experiments, compared to another approach of using metrics computed
directly from a given trained model (a more post-processing approach):
DispaRisk’s approach aims to estimate risk from a more generalizable
perspective based on ML pipeline computational capabilities, making the risk
estimations specific to the pipeline rather than an instance of the it defined
by the models already trained. Nevertheless, although our experiments show
promising results, they are limited to binary sensitive attribute and
classification tasks. DispaRisk is extendible to these settings, thus we plan
to explore them in future research. Additionally, we also plan to explore
DispaRisk in other machine learning domains, such as unsupervised learning and
Natural Language Processing.

## References

  * Asuncion and Newman [2007] Arthur Asuncion and David Newman.  Uci machine learning repository, 2007.
  * Chen et al. [2023] Richard J Chen, Judy J Wang, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Ming Y Lu, Sharifa Sahai, and Faisal Mahmood.  Algorithmic fairness in artificial intelligence for medicine and healthcare.  Nature Biomedical Engineering, 7(6):719–742, 2023.
  * Cover and Thomas [2006] M Thomas Cover and A Joy Thomas.  Elements of Information Theory.  Wiley-Interscience, 2006.
  * Dheeru and Taniskidou [2017] Dua Dheeru and E Karra Taniskidou.  Uci machine learning repository.  2017\.
  * Ethayarajh et al. [2022] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.  Understanding dataset difficulty with $\mathcal{V}$ -usable information.  In International Conference on Machine Learning, pages 5988–6008. PMLR, 2022.
  * Feldman et al. [2015] Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian.  Certifying and removing disparate impact.  In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’15, page 259–268, New York, NY, USA, 2015. Association for Computing Machinery.
  * Fu et al. [2021] Runshan Fu, Yan Huang, and Param Vir Singh.  Crowds, lending, machine, and bias.  Information Systems Research, 32(1):72–92, 2021.
  * Gitiaux and Rangwala [2021] Xavier Gitiaux and Huzefa Rangwala.  Learning smooth and fair representations.  In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 253–261. PMLR, 13–15 Apr 2021.
  * Gitiaux and Rangwala [2022] Xavier Gitiaux and Huzefa Rangwala.  Sofair: Single shot fair representation learning.  arXiv preprint arXiv:2204.12556, 2022.
  * Gupta et al. [2021] Umang Gupta, Aaron M Ferber, Bistra Dilkina, and Greg Ver Steeg.  Controllable guarantees for fair outcomes via contrastive information estimation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7610–7619, 2021.
  * Hardt et al. [2016] Moritz Hardt, Eric Price, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * Hardt et al. [2021] Michaela Hardt, Xiaoguang Chen, Xiaoyi Cheng, Michele Donini, Jason Gelman, Satish Gollaprolu, John He, Pedro Larroy, Xinyu Liu, Nick McCarthy, et al.  Amazon sagemaker clarify: Machine learning bias detection and explainability in the cloud.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2974–2983, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * Hewitt et al. [2021] John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher Manning.  Conditional probing: measuring usable information beyond a baseline.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1626–1639, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * Kizilcec and Lee [2022] René F Kizilcec and Hansol Lee.  Algorithmic fairness in education.  In Wayne Holmes and Kaśka Porayska-Pomsta, editors, The Ethics of Artificial Intelligence in Education, pages 174–202. Taylor & Francis, 2022.
  * Krizhevsky et al. [2017] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * Le Quy et al. [2022] Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi.  A survey on datasets for fairness-aware machine learning.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, page e1452, 2022.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * Pessach and Shmueli [2023] Dana Pessach and Erez Shmueli.  Algorithmic Fairness, pages 867–886.  Springer International Publishing, Cham, 2023.
  * ProPublica [2016] ProPublica.  How we analyzed the compas recidivism algorithm.  ProPublica, 2016.
  * Vasquez et al. [2022] Jonathan Vasquez, Xavier Gitiaux, Cesar Ortega, and Huzefa Rangwala.  Faired: A systematic fairness analysis approach applied in a higher educational context.  In LAK22: 12th International Learning Analytics and Knowledge Conference, pages 271–281, 2022.
  * Wen et al. [2021] Min Wen, Osbert Bastani, and Ufuk Topcu.  Algorithms for fairness in sequential decision making.  In International Conference on Artificial Intelligence and Statistics, pages 1144–1152. PMLR, 2021.
  * Xu et al. [2020] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon.  A theory of usable information under computational constraints.  In International Conference on Learning Representations (ICLR), 2020\.
  * Zhang et al. [2022] Guanhua Zhang, Yihua Zhang, Yang Zhang, Wenqi Fan, Qing Li, Sijia Liu, and Shiyu Chang.  Fairness reprogramming.  In Thirty-sixth Conference on Neural Information Processing Systems, 2022.