Zhao et al. (Zhao et al., 2017) used fairness measurements from training data
as a baseline to identify bias amplification. They annotated unlabeled data
using the ML model and compared fairness measurements between the training
data and the annotated dataset to reveal bias amplification.

Wang and Russakovsky (Wang and Russakovsky, 2021) highlighted issues with Zhao
et al.’s bias amplification measurement and proposed a new, decoupled metric
that considers varying base rates of sensitive attributes.

Cabrera et al. (Cabrera et al., 2019) developed FAIRVIS, a testing tool for
subgroup fairness. It efficiently searches for potential issues among numerous
subgroups, clustering the training dataset to identify statistically similar
subgroups, and calculating group fairness metrics using an entropy-based
approach. FAIRVIS presents generated subgroups sorted by their group fairness
metrics.

Patel et al. (Patel et al., 2022) utilized combinatorial t-way testing (Kuhn
et al., 2013) for fairness testing. This coverage-based data sampling method
generates diverse datasets by applying logical constraints. They created an
input parameter model from the training data and used it to generate a t-way
test set. Discriminatory instances were identified by mutating protected
attributes in each test.

Several techniques (Zhang et al., 2020, 2021b; Tao et al., 2022; Zheng et al.,
2022; Zhang et al., 2021c, a) utilize _gradient information_ , which
represents the direction of steepest ascent in the loss function, to generate
test inputs for fairness testing. ADF (Zhang et al., 2020, 2021b) focuses on
discriminatory instances near the decision boundary of DNNs and employs
gradients to guide the search for neighboring test inputs. EIDIG (Zhang et
al., 2021c) reduces gradient calculations to accelerate the search process.

Researchers have also developed various methods to detect and analyze
_neurons_ responsible for unfair outcomes in emerging deep learning (DL)
models. NeuronFair (Zheng et al., 2022) utilizes neuron analysis to identify
biased neurons contributing to unfairness, and generates discriminatory
instances to amplify the activation differences of these biased neurons. It
demonstrates strong interpretability, generation effectiveness, and data
generalization.

DeepFAIT (Zhang et al., 2021a) employs significance testing to identify
fairness-related neurons by analyzing the activation differences between
privileged and unprivileged groups.

Vig et al. (Vig et al., 2020) apply Causal Mediation Analysis (CMA) to
identify causally implicated parts, such as neurons or attention heads, in the
unfair predictions of a DNN model. CMA measures the direct and indirect
effects of targeted neurons on the final unfair predictions, considering each
neuron as an intermediate mediator.

Tian et al. (Tian et al., 2020) introduce a novel statistical measurement
using neuron activation in DNNs. They compute neuron activation vectors for
each label class based on test inputs and calculate the distance between these
vectors to assess fairness. Dissimilar distances with respect to a third class
indicate the presence of fairness bugs in the tested DNN.

Gao et al. (Gao et al., 2022) propose FairNeuron for DNNs, which employs
neuron slicing to identify conflict _paths_ containing neurons that rely on
sensitive attributes for predictions. Biased instances triggering the
selection of sensitive attributes are identified using these paths, and the
model is retrained through selective training. FairNeuron ensures that the
conflict paths learn all important features for prediction instead of biased
ones for the identified biased instances, while retaining the original
training approach for other instances.

Additionally, Zhang et al. (Zhang et al., 2022a) developed a method to
identify and rectify fairness-related _paths_ in decision tree and random
forest models. They employed a MaxSMT solver to determine the paths that could
be altered while satisfying fairness and semantic difference constraints. The
identified paths were refined by modifying the leaf labels, resulting in a
repaired fair model.

##  6\. Research Trends and Distributions

In Figure 1, we have shown that fairness testing is experiencing a dramatic
increase in the number of publications. This section further analyzes the
research trends and distributions of fairness testing.

Figure 6. Research distribution among different venues.

###  6.1. Research Venues

We first describe research trends in terms of the research communities
engaging in fairness testing. Since 2017, an increasing number of research
communities have dedicated their efforts to studying fairness testing. Notable
contributions include:

  * •

Galhotra et al. (Galhotra et al., 2017) introduced the first fairness testing
approach for ML software in the SE community, receiving the Distinguished
Paper Award at ESEC/FSE 2017.

  * •

Zhao et al. (Zhao et al., 2017) detected bias in datasets and ML models for
visual recognition tasks, earning the Best Paper Award at EMNLP 2017.

  * •

Díaz et al. (Díaz et al., 2018) identified age-related bias in sentiment
analysis systems, receiving the Best Paper Award at CHI 2018.

  * •

Ribeiro et al. (Ribeiro et al., 2020) proposed CheckList, a task-agnostic
methodology for testing NLP models, including fairness testing, which received
the Best Paper Award at ACL 2020.

  * •

Zhang et al. (Zhang et al., 2020) proposed a search-based discriminatory
instance generation approach for DNNs, which received the Distinguished Paper
Award at ICSE 2020 and was selected for SIGSOFT Research Highlights.

  * •

Chakraborty et al. (Chakraborty et al., 2021) addressed selection bias and
label bias in training data, and were honored with the Distinguished Paper
Award at ESEC/FSE 2021.

These six best paper awards, as well as being a credit to their authors, also
demonstrate both the significant level of interest and the high quality of
research on fairness in three different research communities:

  * •

Software Engineering (ICSE and ESEC/FSE)

  * •

Natural Language Processing (EMNLP and ACL)

  * •

Human-Computer Interaction (CHI)

Figure 6 illustrates the distribution of the collected papers across various
research venues. The majority of fairness testing papers (49%) are published
in software engineering venues, including ICSE, ESEC/FSE, ASE, ISSTA, TSE, and
TOSEM. Artificial intelligence venues, such as ICML, NeurIPS, IJCAI, ACL,
EMNLP, CVPR, ECCV, and KDD, account for 42% of the fairness testing papers.

Furthermore, our survey reveals that fairness testing is gaining traction in
other research communities, such as computer security, human-computer
interaction, and mobile computing communities. This highlights the broad
audience and significance of our survey across multiple disciplines.

###  6.2. Machine Learning Categories

In this section, we explore the research trend of fairness testing across
different ML categories. Following previous work (Zhang et al., 2022b), we
categorize the gathered papers into two groups: those that concentrate on DL
software and those that address general ML software.

Out of the total number of papers, 50 papers (50%) focus on conducting
fairness testing for DL software, 41 papers (41%) specifically target general
ML software, and 8 papers (8%) consider both traditional ML software and DL
software. The significant volume of publications on fairness testing for DL
software can be attributed to several factors. On one hand, DL has gained
widespread adoption and is being utilized in a diverse range of software
applications, generating significant interest from the research community. On
the other hand, compared to traditional ML algorithms such as regression and
decision trees, DL models are less interpretable (Baranyi et al., 2020),
making it more challenging to directly reason about fairness.

To gain further insights, we analyze the publication trends for both
categories over the years. Figure 7 depicts the number of papers focusing on
fairness testing in general ML and DL per year. Our analysis reveals a clear
shift in research focus, with a transition from testing general ML software to
testing DL software. Prior to 2019, fairness testing research primarily
concentrated on general ML. However, since 2019, the number of papers
specifically addressing DL has experienced a notable surge, surpassing the
publications on general ML.

Figure 7. Number of fairness testing papers on general machine learning and
deep learning per year.

###  6.3. Data Types

In this section, we investigate the research trends of fairness testing in
applications that involve different types of data.

Out of the total number of papers (100) that we have collected, 5 of them
consider more than one data type. We count each of these papers for each data
type they examine, allowing us to analyze the distribution across various data
types. The findings are illustrated in Figure 8.

Figure 8. Distribution of different data types in fairness testing papers.
Figure 9. Distribution of different data types in fairness testing papers
published in SE venues.

Our analysis indicates that among the publications we have collected, a
significant portion focuses on testing software applications that utilize
tabular data as inputs, accounting for 55% of the total. Furthermore,
approximately 24% of publications address fairness testing in applications
involving text inputs, while another 24% specifically tackle fairness issues
in applications utilizing image inputs.

It is important to note that fairness testing for other data types, such as
speech and recommendation systems, has not yet received extensive
investigation, representing only a small percentage of the publications,
approximately 1% each. These data types remain relatively underexplored in the
context of fairness testing, emphasizing the need for further research and
attention to ensure fairness across a wider range of data-driven applications.

We also plot the data type distribution for SE publications. Figure 9 shows
the results. In comparison to the broader research landscape, the SE community
predominantly focuses on applications involving tabular data, representing a
substantial majority of fairness testing publications in SE venues, amounting
to 79%. In contrast, publications addressing text- or image-based problems
account for only 17% and 8%, respectively. These figures are significantly
lower than the average distribution across all data types.

###  6.4. Fairness Categories

Figure 10. Distribution of different fairness categories.

Figure 10 provides insights into the distribution of different fairness
categories within the fairness testing literature. Our analysis reveals that
comparable research efforts have been dedicated to exploring various aspects
of fairness testing.

Specifically, we observe that 46% of fairness testing papers focus on group
fairness, addressing issues related to fairness among different groups.
Another 46% of papers concentrate on individual fairness, investigating
fairness concerns at the individual level. The remaining 8% of papers examine
both group fairness and individual fairness, recognizing the importance of
considering both perspectives in fairness testing research.

The finding may initially appear contradictory when comparing it with Table 5,
where we observe that nearly all test generation techniques are proposed for
individual fairness. This discrepancy can be attributed to the distinct
characteristics of individual fairness and group fairness.

When examining group fairness, researchers can leverage real-world data to
construct test inputs that capture fairness considerations among different
groups. This availability of data facilitates the exploration of group
fairness in testing scenarios. On the other hand, for individual fairness, it
becomes challenging for researchers to identify pairs of instances that
satisfy the specific input requirements associated with individual fairness.
For instance, it is not always straightforward to find two individuals who
differ solely in a sensitive protected attribute, making test input generation
more focused on individual fairness.

Despite these differences in test input generation, it is important to note
that fairness testing studies overall investigate group fairness and
individual fairness at a similar level. The distribution of research efforts
aims to address both perspectives, acknowledging the significance of both
group and individual fairness in testing methodologies.

###  6.5. Testing Manners

We further categorize existing fairness testing techniques based on the
software testing approach employed, distinguishing between white-box and
black-box methods. It is worth noting that although Tizpaz-Niari et al.
(Tizpaz-Niari et al., 2022) proposed black-box and gray-box techniques for
testing hyper-parameters, we consider their approach as white-box because it
requires access to the training data for model training and fairness
evaluation.

In our analysis, we find that 53% of the 100 papers employ black-box testing
techniques, while 47% adopt white-box testing methods. This observed gap
between black-box and white-box testing proportions is reasonable.

Compared to black-box testing, white-box testing necessitates access to either
training data or the internal workings of software systems. However, fairness
testing often applies to systems that are human-related and have social
significance, making it challenging to disclose internal information to the
public due to privacy concerns or legal policies. Therefore, it is reasonable
that more research studies focus on black-box testing approaches, which do not
require detailed knowledge of the internal system mechanisms.

###  6.6. Tasks and Sensitive Attributes

In the collected papers, researchers typically present their techniques and
then employ datasets featuring specified tasks and sensitive attributes, for
technique evaluation. In this section, we provide a summary of prevalent tasks
and sensitive attributes commonly considered in the fairness testing
literature.

We present in Table 6 commonly studied tasks in fairness testing literature.
As detailed in Section 6.3, existing fairness testing techniques span various
data types, enabling support for diverse tasks. Notably, income prediction,
credit risk prediction, and recidivism/crime prediction emerge as the three
most widely studied. These tasks also feature as the most widely-explored ones
in the bias mitigation (i.e., fairness improvement) literature, as indicated
by a recent survey (Hort et al., 2023).

We list in Table 7 commonly studied sensitive attributes in the fairness
testing literature. Overall, the literature covers a wide range of sensitive
attributes, including sex/gender, race/ethnicity, age, country, occupation,
religion, and sexual orientation. Notably, the top three sensitive attributes
are sex/gender, race/ethnicity, and age, which are consistent with previous
findings observed in the general software fairness literature (Soremekun et
al., 2022a). They are considered in 89%, 57%, and 41% of the collected papers,
respectively.

Table 6. Commonly-studied tasks in the fairness testing literature. Task | #Publications
---|---
Income prediction | 49
Credit risk prediction | 33
Recidivism/crime prediction | 26
Deposit subscription prediction | 24
Face recognition/facial analysis | 16
Healthcare prediction | 10
Conversational AI/language generation | 9
Sentiment analysis | 8
Heart heath prediction | 7
Image tagging/classification | 7
Exam performance/grade prediction | 5
Coreference resolution | 3
Machine translation | 3
Individual survival prediction | 3
Recommender systems | 2
Pedestrian trajectory prediction | 1
Admission prediction | 1
Loan application | 1
Hiring | 1
Pricing | 1
Fraud detection | 1
Car rental | 1
Execution prediction | 1
Toxicity detection | 1
Name entity recognition | 1
Speech recognition | 1
COVID-19 new case prediction | 1
Emergency department wait-time prediction | 1
Table 7. Commonly-studied sensitive attributes in the fairness testing literature. Sensitive attribute | #Publications
---|---
Sex/gender | 89
Race/ethnicity/skin tone | 57
Age | 41
Country/nationality/geography | 8
Occupation/profession | 6
Religion | 3
Political ideology | 2
Sexual orientation | 2
Person name | 2
Ability | 2
Body | 2
Character | 2
Culture | 2
Social status | 2
Marital status | 1
Hair color | 1
Victim | 1

##  7\. Datasets and Tools

Based on the collected papers, this section summarizes the public datasets and
open-source tools for fairness testing to provide a quick navigation for
researchers and practitioners.

Table 8. Public datasets for fairness testing. Dataset | Description | Data Type | Size | Sensitive Attribute(s) | URL
---|---|---|---|---|---
Adult Income | Income prediction | Tabular | 48,842 | sex, race | (adu, 2017)
Census-Income (KDD) | Income prediction | Tabular | 299,285 | sex, race | (inc, 2000)
Compas | Recidivism prediction | Tabular | 6,167 | sex, race | (com, 2016)
German Credit | Credit risk prediction | Tabular | 1,000 | sex | (ger, 1994)
Default Credit | Credit risk prediction | Tabular | 30,000 | sex | (Def, 2016)
Home Credit | Credit risk prediction | Tabular | 37,511 | sex | (Hom, 2017)
Bank Marketing | Deposit subscription prediction | Tabular | 30,488 | age | (ban, 2014)
Mep15 | Health care needs prediction | Tabular | 15,830 | race | (mep, 2015)
Mep16 | Health care needs prediction | Tabular | 15,675 | race | (mep, 2016)
Dutch Census | Income prediction | Tabular | 60,421 | gender | (Dut, 2014)
Heart Health | Heart health prediction | Tabular | 303 | age | (hea, 2001)
Arrhythmia | Cardiac arrhythmia prediction | Tabular | 452 | sex | (Arr, 2022)
Student Performance | Final year grade prediction | Tabular | 649 | sex | (stu, 2014)
UFRGS | GPA prediction | Tabular | 43,302 | gender, race | (UFR, 2019)
Law School | Exam performance prediction | Tabular | 20,000 | gender, race | (law, 1998)
Titanic | Individual survival prediction | Tabular | 891 | sex | (Tit, 2017)
Communities and Crime | Crime prediction | Tabular | 2,215 | race, age | (Cri, 2011)
Diabetes | Readmission prediction | Tabular | 100,000 | race | (Dia, 1994)
Heritage Health | Staying in the hospital or not | Tabular | 147,473 | age | (Her, 2015)
Fraud | Fraud detection | Tabular | 1,100 | age | (Fra, 2000)
US Executions | Execution prediction | Tabular | 1,437 | sex, race | (Exe, 1977)
LFW | Face recognition | Image | 10,000 | sex, race | (lfw, 2007)
BUPT-Transferface | Face recognition | Image | 1,300,000 | race | (Tra, 2019)
VGGFace | Face recognition | Image | 2,600,000 | gender, race | (VGG, 2015)
COCO-gender | Object detection | Image | 66 objects | gender | (coc, 2017)
CelebA | Facial analysis | Image | 202,599 | gender | (Cel, 2015)
RAF-DB | Facial analysis | Image | 15,339 | gender, race, age | (raf, 2017)
PBB | Facial analysis | Image | 1,270 | gender, skin type | (pbb, 2018)
FairFace | Facial analysis | Image | 108,501 | gender, race, age | (Fai, 2021b)
WinoST | Speech translation | Audio | 3,888 | gender | (Win, 2020)
AequeVox | Speech recognition | Audio | 68 | gender, accent | (Aeq, 2022b)
WinoBias | Coreference resolution | Text | 3,160 | gender | (Win, 2018a)
Winogender | Coreference resolution | Text | 720 | gender | (Win, 2018b)
IMDB | Sentiment analysis | Text | 50,000 | gender, country, occupation | (IMD, 2011)
Twitter Sentiment140 | Sentiment analysis | Text | 1.6 million | gender, country, occupation | (Sen, 2021)
SST | Sentiment analysis | Text | 11,855 | gender, country | (sst, 2017)
EEC | Sentiment analysis | Text | 8,640 | sex, race | (EEC, 2018)
Large Movie Review | Sentiment analysis | Text | 50,000 | gender | (lar, 2011)
Wikipedia comments | Toxicity detection | Text | 127,000 | religion, country, ethnic, race | (Wik, 2017)
Jigsaw Comments | Toxicity classification | Text | 313,000 | religion, country, ethnic, race | (Jig, 2019)
nlg-bias | Language generation | Text | 360 | gender, race, sexual orientation | (nlg, 2019)
BOLD | Language generation | Text | 23,679 |  profession, gender, race, religion, political ideology | (bol, 202Q)
HOLISTICBIAS | Language generation | Text | 459,758 | 13 attributes | (HOL, 2022)
DialogueFairness | Conversational AI | Text | 300,000 | gender | (Dia, 2020)
BiasAsker | Conversational AI | Text | 8,110 |  ability, age, body character, culture, gender, profession, race, religion, social victim | (bia, 2023)
MovieLens | Recommendation | Movie ratings | 1 million | gender, age, occupation | (mov, 2015)
LFM360K | Recommendation |  Music listening history | 17 million |  gender, age, nationality | (LFM, 2015)
BlackFriday | Recommendation | Purchase history | 550,068 |  gender, age, occupation, city category, years of stay in the current city, marital status | (Bla, 2019)

###  7.1. Public Datasets

This section lists the public datasets in the literature based on our
collected papers. Table 8 provides detailed information about these datasets,
including their sizes, data types, sensitive attributes, usage scenarios, and
access links. Many of these datasets comprise tabular data, making them
suitable for traditional ML classifiers.

In recent years, there has been a surge in the availability of text and image
datasets, driven by the growing popularity of natural language processing and
computer vision. These datasets are often sourced from social media platforms
like Twitter. It is worth noting that certain datasets come with specific
usage constraints that researchers must consider when utilizing them. For
instance, the well-known image dataset CelebA (Cel, 2015) is restricted to
non-commercial research purposes only.

For a comprehensive overview of the existing fairness datasets, we recommend
referring to the works of Le Quy et al. (Quy et al., 2021) and Fabris et al.
(Fabris et al., 2022). Le Quy et al. (Quy et al., 2021) surveyed tabular
datasets specifically for fairness research, while Fabris et al. (Fabris et
al., 2022) expanded the survey to include unstructured data such as text and
images. Their surveys encompass datasets from diverse domains, including
social sciences, computer vision, health, economics, business, and
linguistics.

Table 9. Open-source tools for fairness testing. Tool [ref] | Application | Component | Description | URL
---|---|---|---|---
FairTest (Tramèr et al., 2017) | General ML | Model | Analyzing associations between outcomes and sensitive attributes | (Fai, 2017)
Themis (Galhotra et al., 2017) |  Classification (Tabular data) | Model | Black-box random discriminatory instance generation | (The, 2017)
Aequitas (Udeshi et al., 2018) |  Classification (Tabular data) | Model | Automated directed fairness testing | (Aeq, 2018)
ExpGA (Fan et al., 2022) |  Classification (Tabular data, text) | Model | Explanation-guided fairness testing through genetic algorithm | (Exp, 2021)
fairCheck (Sharma and Wehrheim, 2020) |  Classification (Tabular data) | Model | Verification-based discriminatory instance generation | (fai, 2020)
MLCheck (Sharma et al., 2021) |  Classification (Tabular data) | Model | Property-driven testing of ML models | (mlc, 2022)
LTDD (Li et al., 2022) |  Classification (Tabular data) | Data | Detecting which data features and which parts of them are biased | (LTD, 2022)
Fair-SMOTE (Chakraborty et al., 2021) |  Classification (Tabular data) | Model | Detecting biased data labels and data distributions | (SMO, 2021)
FairMask (Peng et al., 2023) |  Classification (Tabular data) | Data | Extrapolation of correlations among data features that might cause bias | (xFA, 2023)
Fairway (Chakraborty et al., 2020a) |  Classification (Tabular data) |  Data, ML program | Detecting biased data labels and optimal hyper-parameters for fairness | (Fai, 2020)
Parfait-ML (Tizpaz-Niari et al., 2022) |  Classification (Tabular data) | ML program | Searching for hyper-parameters optimal to ML fairness | (Par, 2022)
Fairea (Hort et al., 2021) |  Classification (Tabular data) |  ML program, model | A unified benchmark for evaluating fairness repair algorithms | (Fai, 2021a)
IBM AIF360 (Bellamy et al., 2018) |  Classification (Tabular data) |  Data, ML program, model | Examining and mitigating bias in ML software | (AIF, 2018)
I&D (Ma et al., 2022) |  Classification (Tabular data) | Model | Improving initial individual discriminatory instances generation | (idl, 2022)
scikit-fairness (skf, 2023) |  Classification (Tabular data) |  Data, ML program, model | Examining and mitigating bias in ML software | (skf, 2023)
LiFT (Vasudevan and Kenthapadi, 2020) |  Classification (Tabular data) |  Data, ML program, model | Examining and mitigating bias in ML software | (lif, 2020)
FairVis (Cabrera et al., 2019) |  Classification (Tabular data) | Model | Visual analytics for discovering intersectional bias in ML software | (fai, 2019)
BiasAmp (Wang and Russakovsky, 2021) | Image classifier | Model | Analyzing whether ML exacerbates bias from the training data | (bia, 2022)
MAAT (Chen et al., 2022) |  Classification (Tabular data) | Data | Detecting selection bias and improving fairness-performance trade-off | (MAA, 2022)
FairEnsembles (Gohar et al., 2023) |  Classification (Tabular data) | ML program | Analyzing fairness and its composition in ensemble ML | (fen, 2023)
FairRepair (Zhang et al., 2022a) |  Tree-based classification (Tabular data) | Model | Fairness testing and repair for tree-based models | (Fai, 2022b)
SBFT (Perera et al., 2022) |  Regression (Tabular data) | Model | Search-based fairness testing for regression-based ML systems | (SBF, 2022)
ADF (Zhang et al., 2020) |  DL-based classification (Tabular data) | Model | White-box fairness testing through adversarial sampling | (ADF, 2020)
EIDIG (Zhang et al., 2021c) |  DL-based classification (tabular data) | Model | White-box fairness testing through gradient search | (EID, 2021)
NeuronFair (Zheng et al., 2022) |  DL-based classification (tabular data, face images) | Model | Interpretable white-box fairness testing through biased neuron identification | (Neu, 2022)
DeepInspect (Tian et al., 2020) |  DL-based image classification | Model | Detecting class-based bias in image classification | (Dee, 2020)
CMA (Vig et al., 2020) | Language models | Model | Detecting which parts of DNNs are responsible for unfairness | (CMA, 2020)
FairNeuron (Gao et al., 2022) |  DL-based classification (tabular data) | Model | Detecting neurons and data instances responsible for bias | (Fai, 2022a)
RULER (Tao et al., 2022) |  DL-based classification (tabular data) | Model | Test input generation by discriminating sensitive and non-sensitive attributes | (rul, 2022)
TestSGD (Zhang et al., 2023) |  DL-based classification (Tabular data, text) | Model | Interpretable testing of DNNs against subtle group discrimination | (Tes, 2023)
DICE (Monjezi et al., 2023) |  DL-based classification (tabular data) | Model | Information-theoretic fairness testing and debugging of DNNs | (dic, 2023)
ASTRAEA (Soremekun et al., 2022b) | NLP systems | Model | Grammar-based discriminatory instance generation for NLP systems | (AST, 2022)
MT-NLP (Ma et al., 2020) | NLP systems | Model | Metamorphic testing of fairness violation in NLP systems | (MTN, 2020)
BiasFinder (Asyrofi et al., 2021) | Sentiment analysis | Model | Metamorphic test generation to uncover bias of sentiment analysis systems | (Bia, 2021)
BiasRV (Yang et al., 2021) | Sentiment analysis | Model | Uncovering biased sentiment predictions at runtime | (bia, 2021)
NERGenderBias (Mehrabi et al., 2020b) | Name entity recognition | Model | Measuring gender bias in named entity recognition | (NER, 2019)
CheckList (Ribeiro et al., 2020) | NLP systems | Model | Behavioral testing (including fairness testing) of NLP models | (che, 2020)
DialogueFairness (Liu et al., 2020) | Conversational AI | Model | Testing gender and linguistic (racial) bias in dialogue systems | (Dia, 2020)
BiasAsker (Wan et al., 2023) | Conversational AI | Model | Fairness testing of conversational AI systems | (bia, 2023)
REVISE (Wang et al., 2020a) | CV datasets | Data | Detecting object-, gender-, and geography-based bias in CV datasets | (rev, 2022)
AequeVox (Rajan et al., 2022) | Speech recognition | Model | Comparing the robustness of speech recognition systems for different groups | (Aeq, 2022a)

###  7.2. Open-source Testing Tools

There is a recent proliferation of open-source tools for supporting fairness
testing. Nevertheless, Lee and Singh (Lee and Singh, 2021) demonstrated that
there is a steep learning curve for practitioners to use these fairness tools.
Presently, there is a lack of guidance on tool adoption (Lee and Singh, 2021).

To address this gap, we provide a summary of 41 open-source fairness testing
tools in this section, aiming to assist fairness researchers and practitioners
in selecting the most suitable tools. The details of these tools are presented
in Table 9. The table includes fairness testing tools for various domains,
including general ML (e.g., FairTest (Tramèr et al., 2017)), DL (e.g., ADF
(Zhang et al., 2020) and EIDIG (Zhang et al., 2021c)), natural language
processing (e.g., ASTRAEA (Soremekun et al., 2022b) and BiasFinder (Asyrofi et
al., 2021)), computer vision (e.g., REVISE (Wang et al., 2020a)), and speech
recognition (e.g., AequeVox (Rajan et al., 2022)).

##  8\. Research Opportunities

Fairness testing remains in a relatively embryonic state. Research in this
area is experiencing rapid growth, so there are plenty of open research
opportunities. In this section, we outline the challenges for fairness testing
and present promising research directions and open problems.

###  8.1. Absence of or with Multiple Sensitive Attributes

Fairness testing in the absence of sensitive attribute information. Existing
fairness testing techniques rely on the existence of sensitive attributes, but
in practice, this information might be unavailable or imperfect for many
reasons (Awasthi et al., 2021). On the one hand, the data may be collected in
a setting where the sensitive attribute information is unnecessary,
undesirable, or even illegal, considering the recently released regulations
such as GDPR (General Data Protection Regulation) (Voigt and Von dem Bussche,
2017) and CCPA (California Consumer Privacy Act) (Goldman, 2020). On the other
hand, users may withhold or modify sensitive attribute information, for
example, due to privacy concerns or other personal preferences. To tackle this
issue, a straightforward solution is to first use existing demographic
information inference techniques (e.g., gender inference, race inference, and
age inference) to infer the sensitive attribute and then apply fairness
testing techniques. However, existing inference techniques may not be fully
satisfactory, and their application scenarios remain limited (Chen et al.,
2018). Moreover, building a model to infer sensitive information leaves open
the possibility that the model may ultimately be used more broadly, with
possibly unintended consequences (Awasthi et al., 2021). Therefore, more
research is needed to tackle fairness testing in the absence of sensitive
attribute information.

Fairness testing with multiple sensitive attributes. Software systems can have
multiple sensitive attributes that need to be considered at the same time
(Gohar and Cheng, 2023; Chen et al., 2024). Human attributes, such as sex,
race, and class, intersect with one another, and unfair software systems built
into society lead to systematic disadvantages along these intersecting
attributes (Crenshaw, 2013; Kimberly, 1989). However, existing fairness
testing work often tackles a single sensitive attribute at a time. To the best
of our knowledge, there has been a little work that explores fairness testing
for compounded or intersectional effects of multiple sensitive attributes
(Cabrera et al., 2019; Tao et al., 2022; Zhang et al., 2023), leaving an
interesting research opportunity for the community. Moreover, drawing
parallels with historical legal discourse, the intersectionality of protected
attributes reveals the challenges of avoiding over-segmentation when assessing
fairness. In 1976, Judge Harris Wangelin expressed concerns about creating new
protected groups (jud, 2019), specifically considering the implications of the
‘curse of dimensionality.’ Much like the legal system’s cautious approach to
potential Pandora’s boxes of new protected classes, fairness testing grapples
with complexities in addressing compounded or intersectional effects of
multiple sensitive attributes.

###  8.2. Test Oracle for Fairness Testing

Existing work mainly employs metamorphic relations as pseudo oracles or uses
statistical measurements as indirect oracles of fairness testing, which both
involve human ingenuity. It is an open challenge to design automatic
techniques for constructing reliable oracles for fairness testing.

Furthermore, the emergence of manually-defined oracles for fairness testing
brings a challenge for test oracle selection. For instance, the IBM AIF360
toolkit alone offers more than 70 fairness measurements (AIF, 2018; Bellamy et
al., 2018), and the research community continues to introduce novel
measurements. However, it is impractical to utilize all existing measurements
as test oracles for fairness testing. Moreover, while each measurement may be
suitable in a specific context, many of them cannot be simultaneously
satisfied (Brun and Meliou, 2018). Hence, an important area for the research
community is the development of automatic techniques for constructing reliable
oracles in fairness testing. It is worth noting that one could argue that this
challenge falls within the realm of requirements engineering rather than
testing. However, in reality, fairness testing is frequently explored in
natural settings, where upfront requirements engineering processes are not
always assumed or followed strictly. Fairness testing involves investigating
and evaluating fairness concerns in real-world systems or datasets, which may
lack comprehensive and formal requirements. As a result, fairness testing
needs to adapt to address the unique challenges that arise in these real-world
contexts, where strict adherence to traditional requirements engineering may
not be practical or feasible.

###  8.3. Test Input Generation for Fairness Testing

Generation of natural inputs. Despite the existence of various techniques for
test input generation in fairness testing, there is no guarantee that the
generated instances are legitimate and natural. Particularly, in Table 5, it
is evident that most test input generation techniques are based on the causal
fairness definition, which requires generating pairs of instances that differ
solely in sensitive attributes. However, it remains uncertain whether altering
only the sensitive attribute is sufficient to generate inputs that are truly
natural.

Furthermore, existing techniques (Zhang et al., 2021c; Zhang et al., 2021b,
2020) primarily rely on perturbing input features, without explicitly
constraining the magnitude of the perturbation. As long as the generated
instances can induce the intended output behavior, such as flipping the
predicted outcome after modifying sensitive attribute information, they are
considered effective. However, this approach may overlook real-world
constraints, potentially resulting in generated instances that do not align
with reality (e.g., granting a loan to a 10-year-old individual).

As a result, open problems arise regarding how to generate test inputs for
fairness testing that are both legitimate and natural. Researchers need to
address the challenges of ensuring the generated instances adhere to real-
world constraints while still accurately assessing fairness. Additionally,
automating the evaluation of the naturalness of generated test inputs is
another important area of exploration, enabling more reliable and efficient
fairness testing methodologies.

Exploration of more generation techniques. As mentioned earlier, most test
input generation techniques in fairness testing focus on the causal fairness
definition. In contrast, test input generation for counterfactual fairness is
relatively unmatured and more challenging. It requires researchers to conduct
causal analysis of features and consider the causal relationships among them
when altering sensitive attributes during generation. Moreover, when dealing
with multiple sensitive attributes simultaneously, the task becomes even more
difficult as changes in features need to account for the causal impacts from
multiple sensitive attributes.

Furthermore, there is a research opportunity to design test inputs
specifically for group fairness testing. While group fairness can be evaluated
using real-world collected data, obtaining such data is not always easy. Given
the limited work on generating test inputs for group fairness, there is still
much potential for exploration in this area.

###  8.4. Test Adequacy for Fairness Testing

Test adequacy is a well-explored concept in traditional software testing,
focusing on evaluating the coverage provided by existing tests (Zhang et al.,
2022b). Adequacy criteria not only offer confidence in testing activities but
also serve as a guide for test generation. However, in the domain of fairness
testing, the issue of test adequacy remains an open problem, and to the best
of our knowledge, no research has specifically addressed this area.

To address this challenge, one approach could be to adapt traditional software
test adequacy metrics or ML test adequacy metrics for fairness testing. For
instance, traditional software testing has proposed metrics such as line
coverage, branch coverage, and dataflow coverage (Zhang et al., 2019b), while
ML testing has introduced metrics such as neuron coverage, layer coverage, and
surprise adequacy for deep learning models (Zhang et al., 2022b). Neuron
coverage assesses the extent to which neurons in a deep learning model are
exercised by a test suite, while layer coverage measures the coverage of
different layers. Surprise adequacy, on the other hand, evaluates the coverage
of discretized input surprise range for deep learning models (Zhang et al.,
2022b).

However, there is currently no empirical evidence to support the applicability
and effectiveness of these metrics in assessing the ability to detect fairness
bugs and the sufficiency of fairness testing. Further research is needed to
investigate and validate the suitability of these metrics in the context of
fairness testing. Additionally, novel metrics tailored specifically for
fairness testing may need to be developed to capture the unique
characteristics and requirements of assessing fairness in ML software.

###  8.5. Test Cost Reduction

Test cost poses a significant challenge in fairness testing of ML software.
The process of assessing fairness often entails retraining ML models,
repeating the prediction process, or generating extensive data to explore the
vast behavioral space of the models. However, thus far, there has been no
research dedicated to reducing the cost of fairness testing. It would be
intriguing to explore specific techniques for test selection, prioritization,
and minimization that can effectively reduce the cost of fairness testing
without compromising test effectiveness.

Moreover, as discussed in Section 5.2, there is an escalating demand for
deploying intelligent software systems on platforms with limited computing
power and resources, such as mobile devices. Several studies (Blakeney et al.,
2021; Hooker et al., 2020; Stoychev and Gunes, 2022; Blakeney et al., 2021)
have addressed fairness testing in such scenarios. This presents a fresh
challenge for the research community: how to conduct fairness testing
effectively on diverse end devices, including those with restricted computing
power, limited memory size, and constrained energy capacity. Addressing this
challenge requires innovative approaches and techniques that can adapt
fairness testing methodologies to accommodate the limitations and constraints
of these resource-constrained platforms.

###  8.6. Fairness and Other Testing Properties

Testing fairness repair techniques with more properties considered. After
fairness repair techniques have been applied to software systems, fairness
testing is often performed again. In this process, testers may also take ML
performance (e.g., accuracy) into consideration (Hort et al., 2021; Chen et
al., 2022), because it is well-known that fairness improvement is often at the
cost of ML performance (Hort et al., 2021). However, in addition to ML
performance, there are also many other properties important for software
systems, including robustness, security, efficiency, interpretability, and
privacy (Zhang et al., 2022b). The relationship between fairness and these
properties is not well studied in the literature, and thus these relationships
remain less well understood. Future research is needed to uncover the
relationships and perform the testing with these properties considered. The
determination of the properties to be considered needs the assistance of
requirements engineers.

Fairness and explainability. Explainability is defined as that users can
understand why a prediction is made by a software system (Gilpin et al.,
2018). Like fairness, it has also been an important software property required
by recent regulatory provisions (Mittelstadt et al., 2019). Because
application scenarios that demand fairness often also require explainability,
it would be an interesting research direction to consider fairness and
explainability together. Many existing fairness testing studies just generate
discriminatory instances that reveal fairness bugs in the software under test,
but do not explain why these instances are unfairly treated by the software.

In this case, software engineers have relatively little guidance on the
production of targeted fixes to repair the software. Improving the
explainability behind the unfair software outcomes can help summarize the
reasons for fairness bugs, produce insights for fairness repair, and help
stakeholders without technical backgrounds (e.g., product managers, compliance
officers, and policymakers) understand the software bias simply and quickly.

###  8.7. Fairness Testing of More Applications

The majority of existing fairness testing work has concentrated on tabular
data, natural language processing systems, and computer vision systems.
However, fairness, as a critical non-functional property, should be considered
across a broader spectrum of software systems, including speech recognition
systems, video analytic systems, multi-modal systems, and recommendation
systems. Additionally, existing fairness testing studies have predominantly
centered around classification tasks. However, fairness is a crucial concern
that should be examined in various machine learning tasks, including
regression and clustering as well as emerging cutting-edge AI technologies
such as Large Language Models (LLMs).

LLMs, extensively studied in academic literature and widely adopted in various
applications, have recently raised concerns about fairness (Li et al., 2023).
Given the significance of fairness in LLMs, OpenAI is actively seeking
expertise to address these concerns and foster the development of fair LLMs
(ope, 2023). However, fairness testing for LLMs poses unique challenges.
Firstly, LLMs are often open-domain systems that offer diverse
functionalities. For instance, ChatGPT can engage in a variety of
conversations with humans on a broad spectrum of topics. This characteristic
poses a challenge when designing test oracles and input generation techniques
for fairness testing, given the need to account for the extensive range of
subjects and functionalities in LLMs. In contrast, existing fairness testing
techniques are often tailored to specific tasks, potentially falling short in
comprehensively evaluating the performance of LLMs across their diverse
capabilities. Secondly, LLMs can generate a spectrum of responses, including
those that may appear vague or unrelated, often influenced by pre-defined
protection mechanisms regarding sensitive topics. This diversity in responses
poses a challenge in automatically discerning whether the LLM output exhibits
bias (i.e., the test oracle problem). While Wan et al. (Wan et al., 2023) have
proposed solutions to address this challenge, their focus remains on tackling
test oracle problems related to Yes-No questions, Choice-questions, and Wh-
questions. Regrettably, this leaves other question types, such as prediction
questions, explanatory questions, and recommendation questions, unexplored.
Thirdly, the majority of large-scale LLMs are not open-sourced, leading to
opacity in their underlying mechanisms and low explainability. This
characteristic poses a challenge in designing fairness testing techniques,
limiting practitioners to developing black-box approaches based solely on the
observed responses of LLMs. Moreover, data testing and ML program testing are
not applicable to such LLMs.

###  8.8. More Fairness Testing Activities and Components

The current body of research primarily focuses on offline fairness testing.
There is a pressing need for more research in the realm of online fairness
testing, as it can provide valuable insights to guide software maintenance and
facilitate the evolution of software systems.

Furthermore, researchers have an opportunity to extend fairness testing
investigations to include additional testing activities that have been
extensively studied in traditional software testing but rarely explored in the
context of fairness testing. For instance, exploring bug report analysis
(Zhang et al., 2015), bug triage (Jeong et al., 2009), and test evaluation
(Zhang et al., 2022b) in fairness testing can contribute valuable insights to
the field.

Additionally, there exists a research gap in the fairness testing components.
While testing of ML frameworks has received considerable attention in
traditional ML testing (Pham et al., 2019; Wang et al., 2020b; Nejadgholi and
Yang, 2019; Wang et al., 2022a), its application in fairness testing remains
underexplored. Furthermore, exploring non-ML component testing within the
context of fairness testing presents a promising research direction that
warrants further investigation.

###  8.9. More Fairness Testing Tools

Existing fairness testing tools (listed in Table 9) tend to require
programming skills, and thus are unfriendly to non-technical stakeholders.
However, fairness testing research includes many non-programmer stakeholders
and contributors such as compliance officers, policymakers, and legal
practitioners.

##  9\. Discussion

###  9.1. Stakeholders in Fairness Testing

Fairness testing goes beyond the pure view of test engineers, involving a
range of stakeholders (Sarro, 2023). Given that unfairness can stem from data
or algorithms, data scientists and algorithm designers can play pivotal roles
in detecting potential biases and providing valuable insights for fairness
testing. Additionally, legal practitioners, compliance officers, and
policymakers can be queried as crucial stakeholders, ensuring that fairness
aligns with encoded laws, regulations, and policies.

Moreover, it is imperative to expand our focus beyond conventional algorithmic
testing. ML software users, directly impacted by algorithmic decisions, offer
unique insights and real-world experiences that extend beyond the scope of
algorithmic scrutiny alone. This is exemplified by the book ‘Algorithms of
Oppression’ (Noble, 2018), which provides a good example illustrating that,
despite potential testing of its algorithms, Yelp’s review system demonstrated
unfairness in its waiting system concerning a specific user population,
adversely affecting a local hairdresser store. This emphasizes the
significance of community perspectives in revealing biases not immediately
evident through algorithmic testing alone. Incorporating user-centered design
principles and establishing feedback loops during development and testing are
crucial for a more comprehensive approach to fairness testing.

###  9.2. Algorithmic Fairness and Societal Fairness

Certain researchers posit that algorithms inherently reflect the biases
embedded in their social context (Lewis, 1977). Consequently, they argue that
addressing algorithmic bias holds limited value unless we first address and
rectify the societal issues that influence the selection and deployment of
these algorithms. However, we believe that addressing algorithmic fairness is
not misguided but complementary to tackling societal fairness. While
algorithms may be inherently biased due to societal bias, addressing
algorithmic fairness focuses on short-term solutions within technology to
reduce discrimination. Recognizing that societal fairness is a complex, long-
term endeavor, both efforts—algorithmic and societal fairness—should coexist
for a fairer future.

In addition, algorithms are technologies created by people. Even if algorithms
reflect societal biases, it does not absolve the creators of algorithms from
their responsibilities. Algorithmic fairness research emphasizes holding them
accountable for addressing bias in algorithms.

Furthermore, the algorithmic view of fairness raises awareness about societal
consequences in decision-making. It contributes to a comprehensive effort for
fairness, acknowledging that improving algorithms is a tangible, achievable
step towards broader societal change.

We also recognize that addressing algorithmic fairness fundamentally requires
an improvement in societal fairness, emphasizing crucial social factors such
as power structures and social justice. The technical dimensions of fairness
testing necessitate a holistic consideration of the societal contexts in which
these algorithms function.

A significant aspect involves acknowledging that AI technologies often stem
from the directives of individuals or entities in positions of power (Sloane,
2022). To rectify this power imbalance, a profound shift is imperative. It
extends beyond merely consulting those affected by AI at the outset; active
participation in the decision-making process is essential. This entails
empowering individuals to select the problems addressed and guide the entire
developmental trajectory.

Furthermore, the involvement of the most relevant participants becomes
paramount to ensure that AI training data authentically mirrors the diversity
of perspectives in society (Sloane, 2022). This principle underscores the
significance of integrating lived experiences into the development of AI
systems. This approach not only mitigates the risk of participation-washing
but also fosters a more inclusive methodology for fairness testing.

###  9.3. Threats to Validity

As presented in Section 3, the methodology of this paper includes two primary
phases: paper collection and paper analysis. In this section, we discuss
potential threats associated with these two phases.

Paper collection: Our collection process involves the selection of research
papers from the widely-used DBLP database, which encompasses arXiv and over
1,800 journals, as well as 5,800 academic conferences and workshops in the
field of Computer Science. However, it is important to acknowledge that this
approach may overlook papers published in other venues. Additionally, there is
the possibility that our manually defined search strings may not encompass all
the relevant studies within our research scope. To address these potential
threats, we employ both backward snowballing and forward snowballing to
further identify transitively dependent papers. Moreover, we proactively
contact the authors of our collected papers to solicit additional papers that
fall within the scope of our survey.

Paper analysis: The process of manually analyzing research papers to extract
relevant information to be included in the survey is potentially open to human
bias. To mitigate this risk, each paper undergoes analysis by two separate
authors, and any disagreement that arises during the analysis is resolved
through discussions involving other co-authors, all of whom have previously
published fairness-related papers in top-tier SE venues. Furthermore, all
authors independently conduct a thorough review of the survey’s content to
identify and rectify any potential issues. We also share our survey with the
authors of the collected papers to ensure that our descriptions of their work
are accurate.

##  10\. Conclusion

We have presented a comprehensive survey of 100 papers on fairness testing. We
summarized the current research status in the fairness testing workflow
(including test input generation and test oracle identification) and testing
components (including data testing, ML program testing, and model testing). We
analyzed trends and promising research directions for fairness testing. We
also listed public datasets and open-source tools that can be accessed by
researchers and practitioners interested in this topic. We hope this survey
will help researchers from various research communities become familiar with
the current status and open opportunities of fairness testing.

## Acknowledgment

We shared our work with the authors of the papers we surveyed in order to
check for accuracy and omission, and we would like to thank those authors who
kindly provided comments and feedback on earlier drafts of this paper.
Zhenpeng Chen, Federica Sarro, and Mark Harman are supported by the ERC
Advanced Grant No.741278 (EPIC: Evolutionary Program Improvement
Collaborators). Max Hort is supported by the Research Council of Norway
through the secureIT project (IKTPLUSS #288787).

## References

  * (1)
  * Exe (1977) 1977\.  The US Executions dataset.  https://data.world/markmarkoh/executions-since-1977.
  * Dia (1994) 1994\.  The Diabetes dataset.  https://archive.ics.uci.edu/ml/datasets/diabetes.
  * ger (1994) 1994\.  The German Credit dataset.  https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29.
  * law (1998) 1998\.  The Law School dataset.  https://www.kaggle.com/datasets/danofer/law-school-admissions-bar-passage.
  * inc (2000) 2000\.  The Census-Income (KDD) dataset.  https://archive.ics.uci.edu/dataset/117/census+income+kdd.
  * Fra (2000) 2000\.  The Fraud Detection dataset.  https://www.kaggle.com/competitions/frauddetection/data.
  * hea (2001) 2001\.  The Heart Health dataset.  https://archive.ics.uci.edu/ml/datasets/Heart+Disease.
  * lfw (2007) 2007\.  LFW.  http://vis-www.cs.umass.edu/lfw/.
  * 539 (2010) 2010\.  IEEE Standard Classification for Software Anomalies.  _IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993)_ (2010), 1–23.
  * Cri (2011) 2011\.  The Communities and Crime dataset.  http://archive.ics.uci.edu/ml/datasets/Communities%20and%20Crime%20Unnormalized.
  * IMD (2011) 2011\.  The IMDB dataset.  https://www.imdb.com/interfaces/.
  * lar (2011) 2011\.  The Large Movie Review dataset.  https://ai.stanford.edu/~amaas/data/sentiment/.
  * ban (2014) 2014\.  The Bank dataset.  https://archive.ics.uci.edu/ml/datasets/Bank+Marketing.
  * Dut (2014) 2014\.  The Dutch Census of 2001 dataset.  https://microdata.worldbank.org/index.php/catalog/2102.
  * stu (2014) 2014\.  The Student Performance dataset.  https://archive.ics.uci.edu/ml/datasets/Student+Performance.
  * Cel (2015) 2015\.  The CelebA dataset.  https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html.
  * Her (2015) 2015\.  The Heritage Health dataset.  https://foreverdata.org/1015/index.html.
  * LFM (2015) 2015\.  The LFM360K dataset.  https://www.upf.edu/web/mtg/lastfm360k.
  * mep (2015) 2015\.  The Mep15 dataset.  https://meps.ahrq.gov/mepsweb/data_stats/download_data_files_detail.jsp?cboPufNumber=HC-181.
  * mov (2015) 2015\.  The movielens dataset.  https://movielens.org/.
  * VGG (2015) 2015\.  The VGGFace dataset.  https://www.robots.ox.ac.uk/~vgg/data/vgg_face/#publi.
  * ama (2016) 2016\.  Amazon just showed us that ‘unbiased’ algorithms can be inadvertently racist.  https://www.businessinsider.com/how-algorithms-can-be-racist-2016-4?r=US&IR=T.
  * com (2016) 2016\.  The Compas dataset.  https://github.com/propublica/compas-analysis.
  * Def (2016) 2016\.  The Default Credit dataset.  https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients.
  * fai (2016) 2016\.  Machine bias.  https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
  * mep (2016) 2016\.  The Mep16 dataset.  https://meps.ahrq.gov/mepsweb/data_stats/download_data_files_detail.jsp?cboPufNumber=HC-192.
  * adu (2017) 2017\.  The Adult Census Income dataset.  https://archive.ics.uci.edu/ml/datasets/adult.
  * coc (2017) 2017\.  The COCO-gender dataset.  https://github.com/uclanlp/reducingbias.
  * Fai (2017) 2017\.  FairTest.  https://github.com/columbia/fairtest.
  * Hom (2017) 2017\.  The Home Credit dataset.  https://www.kaggle.com/c/home-credit-default-risk.
  * sst (2017) 2017\.  The SST dataset.  http://nlpprogress.com/english/sentiment_analysis.html.
  * raf (2017) 2017\.  The RAF-DB dataset.  http://whdeng.cn/RAF/model1.html.
  * The (2017) 2017\.  Themis.  https://github.com/LASER-UMASS/Themis.
  * Tit (2017) 2017\.  The Titanic dataset.  https://www.kaggle.com/c/titanic/data.
  * Wik (2017) 2017\.  The Wikipedia comment dataset.  https://github.com/conversationai/unintended-ml-bias-analysis.
  * Aeq (2018) 2018\.  Aequitas.  https://github.com/sakshiudeshi/Aequitas.
  * AIF (2018) 2018\.  AIF360.  https://aif360.mybluemix.net.
  * EEC (2018) 2018\.  The EEC dataset.  https://competitions.codalab.org/competitions/17751.
  * pbb (2018) 2018\.  The PBB dataset.  http://gendershades.org/overview.html.
  * Win (2018a) 2018a.  The WinoBias dataset.  https://paperswithcode.com/dataset/winobias.
  * Win (2018b) 2018b.  The Winogender dataset.  https://github.com/rudinger/winogender-schemas.
  * Bla (2019) 2019\.  The BlackFriday dataset.  https://www.kaggle.com/datasets/sdolezel/black-friday.
  * Tra (2019) 2019\.  The BUPT-Transferface dataset.  http://www.whdeng.cn/RFW/Trainingdataste.html.
  * fai (2019) 2019\.  FairVis.  https://github.com/poloclub/FairVis.
  * jud (2019) 2019\.  The intersectionality wars.  https://www.vox.com/the-highlight/2019/5/20/18542843/intersectionality-conservatism-law-race-gender-discrimination.
  * Jig (2019) 2019\.  The Jigsaw comment dataset.  https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge.
  * NER (2019) 2019\.  NERGenderBias.  https://github.com/Ninarehm/NERGenderBias.
  * nlg (2019) 2019\.  The nlg-bias dataset.  https://github.com/ewsheng/nlg-bias.
  * UFR (2019) 2019\.  The UFRGS Entrance Exam and GPA Data.  https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/O35FW8.
  * ADF (2020) 2020\.  ADF.  https://github.com/pxzhang94/ADF.
  * che (2020) 2020\.  CheckList.  https://github.com/marcotcr/checklist.
  * CMA (2020) 2020\.  CMA.  https://github.com/sebastianGehrmann/CausalMediationAnalysis.
  * Dee (2020) 2020\.  DeepInspect.  https://github.com/ARiSE-Lab/DeepInspect.
  * Dia (2020) 2020\.  DialogueFairness.  https://github.com/zgahhblhc/DialogueFairness.
  * fai (2020) 2020\.  fairCheck.  https://github.com/arnabsharma91/fairCheck.
  * Fai (2020) 2020\.  Fairway.  https://github.com/joymallyac/Fairway.
  * lif (2020) 2020\.  LiFT.  https://github.com/linkedin/LiFT.
  * MTN (2020) 2020\.  MT-NLP.  https://github.com/pckennethma/MT-NLP.
  * Win (2020) 2020\.  WinoST.  https://zenodo.org/record/4139080#.YtGEc-zMJAc.
  * Bia (2021) 2021\.  BiasFinder.  https://github.com/soarsmu/BiasFinder.
  * bia (2021) 2021\.  BiasRV.  https://github.com/soarsmu/BiasRV.
  * EID (2021) 2021\.  EIDIG.  https://github.com/LingfengZhang98/EIDIG.
  * Exp (2021) 2021\.  ExpGA.  https://github.com/waving7799/ExpGA.
  * SMO (2021) 2021\.  Fair-SMOTE.  https://github.com/joymallyac/Fair-SMOTE.
  * Fai (2021a) 2021a.  Fairea.  https://github.com/maxhort/Fairea.
  * Fai (2021b) 2021b.  The FairFace dataset.  https://github.com/dchen236/FairFace.
  * Sen (2021) 2021\.  The Sentiment140 dataset.  https://www.kaggle.com/datasets/kazanova/sentiment140.
  * Aeq (2022a) 2022a.  AequeVox.  https://github.com/sparkssss/AequeVox.
  * Aeq (2022b) 2022b.  The AequeVox dataset.  https://zenodo.org/record/5897347.
  * Arr (2022) 2022\.  The Arrhythmia dataset.  https://archive.ics.uci.edu/dataset/5/arrhythmia.
  * AST (2022) 2022\.  ASTRAEA.  https://github.com/sakshiudeshi/Astraea.
  * bia (2022) 2022\.  $BiasAmp_{\rightarrow}$.  https://github.com/princetonvisualai/directional-bias-amp.
  * dbl (2022) 2022\.  DBLP.  https://dblp.org.
  * Fai (2022a) 2022a.  FairNeuron.  https://github.com/Antimony5292/FairNeuron.
  * Fai (2022b) 2022b.  FairRepair.  https://github.com/fairrepair/fair-repair.
  * HOL (2022) 2022\.  HOLISTICBIAS.  https://github.com/facebookresearch/ResponsibleNLP/tree/main/holistic_bias.
  * idl (2022) 2022\.  I&D.  https://anonymous.4open.science/r/fairness-095F/README.md.
  * LTD (2022) 2022\.  LTDD.  https://github.com/fairnesstest/LTDD.
  * MAA (2022) 2022\.  MAAT.  https://github.com/chenzhenpeng18/FSE22-MAAT.
  * mlc (2022) 2022\.  MLCheck.  https://github.com/anonymseal/MLCheck.
  * Neu (2022) 2022\.  NeuronFair.  https://github.com/haibinzheng/NeuronFair.
  * Par (2022) 2022\.  Parfait-ML.  https://github.com/Tizpaz/Parfait-ML.
  * rev (2022) 2022\.  REVISE.  https://github.com/princetonvisualai/revise-tool.
  * rul (2022) 2022\.  RULER.  https://github.com/wssun/RULER.
  * SBF (2022) 2022\.  SBFT.  https://github.com/search-based-fairness-testing/sbft.
  * wid (2022) 2022\.  WIDER FACE.  http://shuoyang1213.me/WIDERFACE/.
  * dbl (2023) 2023\.  6 million publications.  https://blog.dblp.org/2022/02/22/6-million-publications/.
  * bia (2023) 2023\.  BiasAsker.  https://github.com/yxwan123/BiasAsker.
  * dic (2023) 2023\.  DICE.  https://github.com/armanunix/Fairness-testing.
  * fen (2023) 2023\.  FairEnsembles.  https://github.com/UsmanGohar/FairEnsemble.
  * skf (2023) 2023\.  Fairlearn.  https://fairlearn.org/.
  * xFA (2023) 2023\.  FairMask.  https://github.com/anonymous12138/biasmitigation.
  * ope (2023) 2023\.  How should AI systems behave, and who should decide?  https://openai.com/blog/how-should-ai-systems-behave.
  * Tes (2023) 2023\.  TestSGD.  https://github.com/zhangmengling/subtle_discrimination_testing.
  * bol (202Q) 202Q.  The BOLD dataset.  https://github.com/jwaladhamala/BOLD-Bias-in-open-ended-language-generation.
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016\.  TensorFlow: A system for large-scale machine learning. In _Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016_. 265–283.
  * Aggarwal et al. (2019) Aniya Aggarwal, Pranay Lohia, Seema Nagar, Kuntal Dey, and Diptikalyan Saha. 2019.  Black box fairness testing of machine learning models. In _Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019_. 625–635.
  * Ahmad et al. (2021) Khlood Ahmad, Muneera Bano, Mohamed Abdelrazek, Chetan Arora, and John C. Grundy. 2021.  What’s up with requirements engineering for artificial intelligence systems?. In _Proceedings of the 29th IEEE International Requirements Engineering Conference, RE 2021_. 1–12.
  * Albarghouthi et al. (2017) Aws Albarghouthi, Loris D’Antoni, Samuel Drews, and Aditya V. Nori. 2017.  FairSquare: Probabilistic verification of program fairness.  _Proceedings of the ACM on Programming Languages_ OOPSLA (2017), 80:1–80:30.
  * Alidoosti (2021) Razieh Alidoosti. 2021\.  Ethics-driven software architecture decision making. In _Proceedings of the IEEE 18th International Conference on Software Architecture Companion, ICSA-C 2021_. 90–91.
  * Angell et al. (2018) Rico Angell, Brittany Johnson, Yuriy Brun, and Alexandra Meliou. 2018.  Themis: Automatically testing software for discrimination. In _Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018_. 871–875.
  * Asyrofi et al. (2021) Muhammad Hilmi Asyrofi, Zhou Yang, Imam Nur Bani Yusuf, Hong Jin Kang, Ferdian Thung, and David Lo. 2021.  BiasFinder: Metamorphic test generation to uncover bias for sentiment analysis systems.  _IEEE Transactions on Software Engineering_ (2021).
  * Awasthi et al. (2021) Pranjal Awasthi, Alex Beutel, Matthäus Kleindessner, Jamie Morgenstern, and Xuezhi Wang. 2021\.  Evaluating fairness of machine learning models under uncertain and incomplete information. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2021_. 206–214.
  * Aydemir and Dalpiaz (2018) Fatma Basak Aydemir and Fabiano Dalpiaz. 2018.  A roadmap for ethics-aware software engineering. In _Proceedings of the International Workshop on Software Fairness, FairWare@ICSE 2018_. 15–21.
  * Bae and Xu (2022) Andrew Bae and Susu Xu. 2022.  Discovering and understanding algorithmic biases in autonomous pedestrian trajectory predictions. In _Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems_. 1155–1161.
  * Balakrishnan et al. (2020) Guha Balakrishnan, Yuanjun Xiong, Wei Xia, and Pietro Perona. 2020\.  Towards causal benchmarking of bias in face analysis algorithms. In _Proceedings of the 16th European Conference on Computer Vision, ECCV 2020_. 547–563.
  * Baranyi et al. (2020) Máté Baranyi, Marcell Nagy, and Roland Molontay. 2020\.  Interpretable deep learning for university dropout prediction. In _Proceedings of the 21st Annual Conference on Information Technology Education, SIGITE 2020_. 13–19.
  * Baresi et al. (2023) Luciano Baresi, Chiara Criscuolo, and Carlo Ghezzi. 2023\.  Understanding fairness requirements for ML-based software. In _Proceedings of the 31st IEEE International Requirements Engineering Conferece, RE 2023_. 341–346.
  * Barocas and Selbst (2016) Solon Barocas and Andrew D Selbst. 2016.  Big data’s disparate impact.  _California Law Review_ 104 (2016), 671.
  * Barr et al. (2015) Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015.  The oracle problem in software testing: A survey.  _IEEE Transactions on Software Engineering_ 41, 5 (2015), 507–525.
  * Bastani et al. (2019) Osbert Bastani, Xin Zhang, and Armando Solar-Lezama. 2019\.  Probabilistic verification of fairness properties via concentration.  _Proceedings of the ACM on Programming Languages_ 3, OOPSLA (2019), 118:1–118:27.
  * Bellamy et al. (2018) Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John T. Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. 2018\.  AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic Bias.  _CoRR_ abs/1810.01943 (2018).
  * Berk et al. (2021) Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. 2021.  Fairness in criminal justice risk assessments: The state of the art.  _Sociological Methods & Research_ 50, 1 (2021), 3–44.
  * Beutel et al. (2019) Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H. Chi, and Cristos Goodrow. 2019\.  Fairness in recommendation ranking through pairwise comparisons. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019_. 2212–2220.
  * Biswas and Rajan (2020) Sumon Biswas and Hridesh Rajan. 2020.  Do the machine learning models on a crowd sourced platform exhibit bias? An empirical study on model fairness. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020_. 642–653.
  * Biswas and Rajan (2021) Sumon Biswas and Hridesh Rajan. 2021.  Fair preprocessing: Towards understanding compositional fairness of data transformers in machine learning pipeline. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021_. 981–993.
  * Black et al. (2020) Emily Black, Samuel Yeom, and Matt Fredrikson. 2020.  FlipTest: Fairness testing via optimal transport. In _Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2020_. 111–121.
  * Blakeney et al. (2021) Cody Blakeney, Nathaniel Huish, Yan Yan, and Ziliang Zong. 2021\.  Simon says: Evaluating and mitigating bias in pruned neural networks with knowledge distillation.  _CoRR_ abs/2106.07849 (2021).
  * Bono et al. (2021) Teresa Bono, Karen Croxson, and Adam Giles. 2021\.  Algorithmic fairness in credit scoring.  _Oxford Review of Economic Policy_ 37, 3 (2021), 585–617.
  * Breck et al. (2019) Eric Breck, Neoklis Polyzotis, Sudip Roy, Steven Whang, and Martin Zinkevich. 2019.  Data validation for machine learning. In _Proceedings of Machine Learning and Systems 2019, MLSys 2019_.
  * Brun and Meliou (2018) Yuriy Brun and Alexandra Meliou. 2018.  Software fairness. In _Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018_. 754–759.
  * Cabrera et al. (2019) Ángel Alexander Cabrera, Will Epperson, Fred Hohman, Minsuk Kahng, Jamie Morgenstern, and Duen Horng Chau. 2019.  FairVis: Visual analytics for discovering intersectional bias in machine learning. In _Proceedings of the 2019 IEEE Conference on Visual Analytics Science and Technology, VAST 2019_. 46–56.
  * Castelnovo et al. (2022) Alessandro Castelnovo, Riccardo Crupi, Greta Greco, Daniele Regoli, Ilaria Giuseppina Penco, and Andrea Claudio Cosentini. 2022\.  A clarification of the nuances in the fairness metrics landscape.  _Scientific Reports_ 12, 1 (2022), 1–21.
  * Caton et al. (2022) Simon Caton, Saiteja Malisetty, and Christian Haas. 2022\.  Impact of Imputation Strategies on Fairness in Machine Learning.  _Journal of Artificial Intelligence Research_ 74 (2022), 1011–1035.
  * Chakraborty et al. (2021) Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies. 2021\.  Bias in machine learning software: Why? How? What to do?. In _Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, ESEC/FSE 2021_. 429–440.
  * Chakraborty et al. (2022) Joymallya Chakraborty, Suvodeep Majumder, and Huy Tu. 2022\.  Fair-SSL: Building fair ML software with less data. In _Proceedings of the International Workshop on Software Fairness, FairWare@ICSE 2022_.
  * Chakraborty et al. (2020a) Joymallya Chakraborty, Suvodeep Majumder, Zhe Yu, and Tim Menzies. 2020a.  Fairway: A way to build fair ML software. In _Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020_. 654–665.
  * Chakraborty et al. (2020b) Joymallya Chakraborty, Kewen Peng, and Tim Menzies. 2020b.  Making fair ML software using trustworthy explanation. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020_. 1229–1233.
  * Chakraborty et al. (2019) Joymallya Chakraborty, Tianpei Xia, Fahmid M. Fahid, and Tim Menzies. 2019.  Software engineering for fairness: A case study with hyperparameter optimization.  _CoRR_ abs/1905.05786 (2019).
  * Chakraborty et al. (2015) Supratik Chakraborty, Daniel J Fremont, Kuldeep S Meel, Sanjit A Seshia, and Moshe Y Vardi. 2015\.  On parallel scalable uniform SAT witness generation. In _Proceedings of the 21st International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2015_. 304–319.
  * Chan and Wang (2018) Jason Chan and Jing Wang. 2018.  Hiring preferences in online labor markets: Evidence of a female hiring bias.  _Management Science_ 64, 7 (2018), 2973–2994.
  * Chen et al. (2020b) Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020b.  A survey of compiler testing.  _Comput. Surveys_ 53, 1 (2020), 4:1–4:36.
  * Chen and Joo (2021) Yunliang Chen and Jungseock Joo. 2021.  Understanding and mitigating annotation bias in facial expression recognition. In _Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021_.
  * Chen et al. (2020a) Zhenpeng Chen, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, Tao Xie, and Xuanzhe Liu. 2020a.  A comprehensive study on challenges in deploying deep learning based software. In _Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020_. 750–762.
  * Chen et al. (2018) Zhenpeng Chen, Xuan Lu, Wei Ai, Huoran Li, Qiaozhu Mei, and Xuanzhe Liu. 2018\.  Through a gender lens: learning usage patterns of emojis from large-scale Android users. In _Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018_. 763–772.
  * Chen et al. (2021) Zhenpeng Chen, Huihan Yao, Yiling Lou, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, and Xuanzhe Liu. 2021.  An empirical study on deployment faults of deep learning based mobile applications. In _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021_. 674–685.
  * Chen et al. (2022) Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2022.  MAAT: A novel ensemble approach to fixing fairness and performance bugs for machine learning software. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022_. 1122–1134.
  * Chen et al. (2023) Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2023.  A comprehensive empirical study of bias mitigation methods for machine learning classifiers.  _ACM Transactions on Software Engineering and Methodology_ 32, 4, Article 106 (2023), 30 pages.
  * Chen et al. (2024) Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2024.  Fairness improvement with multiple protected attributes: How far are we?. In _Proceedings of the 46th ACM/IEEE International Conference on Software Engineering, ICSE 2024_.
  * Cheng et al. (2017) Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017\.  A survey of model compression and acceleration for deep neural networks.  _CoRR_ abs/1710.09282 (2017).
  * Chouldechova (2017) Alexandra Chouldechova. 2017\.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big Data_ 5, 2 (2017), 153–163.
  * Cleary (1966) T Anne Cleary. 1966\.  Test bias: Validity of the scholastic aptitude test for Negro and white students in integrated colleges.  _ETS Research Bulletin Series_ 1966, 2 (1966), i–23.
  * Cleary (1968) T Anne Cleary. 1968\.  Test bias: Prediction of grades of Negro and white students in integrated colleges.  _Journal of Educational Measurement_ 5, 2 (1968), 115–124.
  * Corbett-Davies et al. (2017) Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017.  Algorithmic decision making and the cost of fairness. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2017_. 797–806.
  * Crenshaw (2013) Kimberlé Crenshaw. 2013\.  Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics.  In _Feminist Legal Theories_. Routledge, 23–51.
  * Dash et al. (2022) Saloni Dash, Vineeth N. Balasubramanian, and Amit Sharma. 2022\.  Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022_. 3879–3888.
  * de Moura and Bjørner (2008) Leonardo Mendonça de Moura and Nikolaj S. Bjørner. 2008.  Z3: An efficient SMT solver. In _Proceedings of the 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2008_. 337–340.
  * Denton et al. (2019a) Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. 2019a.  Detecting bias with generative counterfactual face attribute augmentation. In _Proceedings of the CVPR 2019 Workshop on Fairness Accountability Transparency and Ethics in Computer Vision_.
  * Denton et al. (2019b) Emily Denton, Ben Hutchinson, Margaret Mitchell, Timnit Gebru, and Andrew Zaldivar. 2019b.  Image counterfactual sensitivity analysis for detecting unintended bias. In _Proceedings of the CVPR 2019 Workshop on Fairness Accountability Transparency and Ethics in Computer Vision_.
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021.  BOLD: Dataset and metrics for measuring biases in open-ended language generation. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2021_. 862–872.
  * Díaz et al. (2018) Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018.  Addressing age-related bias in sentiment analysis. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018_. 412.
  * DiCiccio et al. (2020) Cyrus DiCiccio, Sriram Vasudevan, Kinjal Basu, Krishnaram Kenthapadi, and Deepak Agarwal. 2020\.  Evaluating Fairness Using Permutation Tests. In _Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2020_. 1467–1477.
  * Duran and Ntafos (1984) Joe W. Duran and Simeon C. Ntafos. 1984.  An evaluation of random testing.  _IEEE Transactions on Software Engineering_ 10, 4 (1984), 438–444.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. 2012.  Fairness through awareness. In _Innovations in Theoretical Computer Science 2012_. 214–226.
  * Fabris et al. (2022) Alessandro Fabris, Stefano Messina, Gianmaria Silvello, and Gian Antonio Susto. 2022.  Algorithmic Fairness Datasets: the Story so Far.  _CoRR_ abs/2202.01711 (2022).
  * Fan et al. (2022) Ming Fan, Wenying Wei, Wuxia Jin, Zijiang Yang, and Ting Liu. 2022.  Explanation-Guided Fairness Testing through Genetic Algorithm. In _Proceedings of the 44th International Conference on Software Engineering, ICSE 2022_.
  * Feldman et al. (2015) Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015.  Certifying and Removing Disparate Impact. In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2015_. 259–268.
  * Ferradans et al. (2014) Sira Ferradans, Nicolas Papadakis, Gabriel Peyré, and Jean-François Aujol. 2014\.  Regularized Discrete Optimal Transport.  _SIAM Journal on Imaging Sciences_ 7, 3 (2014), 1853–1882.
  * Finkelstein et al. (2008) Anthony Finkelstein, Mark Harman, S. Afshin Mansouri, Jian Ren, and Yuanyuan Zhang. 2008.  “Fairness Analysis” in Requirements Assignments. In _Proceedings of the 16th IEEE International Requirements Engineering Conference, RE 2008_. 115–124.
  * Galhotra et al. (2017) Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. 2017\.  Fairness testing: testing software for discrimination. In _Proceedings of the 2017 Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017_. 498–510.
  * Gao et al. (2022) Xuanqi Gao, Juan Zhai, Shiqing Ma, Chao Shen, Yufei Chen, and Qian Wang. 2022\.  FairNeuron: Improving Deep Neural Network Fairness with Adversary Games on Selective Neurons. In _Proceedings of the 44th International Conference on Software Engineering, ICSE 2022_.
  * Garousi and Fernandes (2016) Vahid Garousi and João M Fernandes. 2016.  Highly-cited papers in software engineering: The top-100.  _Information and Software Technology_ 71 (2016), 108–128.
  * Gezici et al. (2021) Gizem Gezici, Aldo Lipani, Yücel Saygin, and Emine Yilmaz. 2021.  Evaluation metrics for measuring bias in search engine results.  _Information Retrieval Journal_ 24, 2 (2021), 85–113.
  * Ghosh et al. (2021) Bishwamittra Ghosh, Debabrota Basu, and Kuldeep S. Meel. 2021\.  Justicia: A Stochastic SAT Approach to Formally Verify Fairness. In _Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021_. 7554–7563.
  * Ghosh et al. (2022) Bishwamittra Ghosh, Debabrota Basu, and Kuldeep S. Meel. 2022\.  Algorithmic Fairness Verification with Graphical Models. In _Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022_.
  * Gilpin et al. (2018) Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018\.  Explaining explanations: An overview of interpretability of machine learning. In _Proceedings of the IEEE 5th International Conference on data science and advanced analytics, DSAA 2018_. IEEE, 80–89.
  * Gohar et al. (2023) Usman Gohar, Sumon Biswas, and Hridesh Rajan. 2023\.  Towards understanding fairness and its composition in ensemble machine learning. In _Proceedings of the 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023_.
  * Gohar and Cheng (2023) Usman Gohar and Lu Cheng. 2023.  A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challenges. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2023_. 6619–6627.
  * Goldman (2020) Eric Goldman. 2020\.  An introduction to the california consumer privacy act (CCPA).  _Santa Clara Univ. Legal Studies Research Paper_ (2020).
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets. In _Proceedings of the Annual Conference on Neural Information Processing Systems 2014, NIPS 2014_. 2672–2680.
  * Grgic-Hlaca et al. (2016) Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. 2016.  The case for process fairness in learning: Feature selection for fair decision making. In _NIPS symposium on machine learning and the law_ , Vol. 1. 2\.
  * Gulli and Pal (2017) Antonio Gulli and Sujit Pal. 2017.  _Deep learning with Keras_.  Packt Publishing Ltd.
  * Guo et al. (2023) Huizhong Guo, Jinfeng Li, Jingyi Wang, Xiangyu Liu, Dongxia Wang, Zehong Hu, Rong Zhang, and Hui Xue. 2023\.  FairRec: Fairness testing for deep recommender systems. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023_.
  * Gursoy and Kakadiaris (2022) Furkan Gursoy and Ioannis A. Kakadiaris. 2022.  Error parity fairness: Testing for group fairness in regression tasks.  _CoRR_ abs/2208.08279 (2022).
  * Haffar et al. (2022) Rami Haffar, Ashneet Khandpur Singh, Josep Domingo-Ferrer, and Najeeb Jebreel. 2022\.  Measuring Fairness in Machine Learning Models via Counterfactual Examples. In _Proceedings of the 19th International Conference on Modeling Decisions for Artificial Intelligence, MDAI 2022_. 119–131.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of Opportunity in Supervised Learning. In _Proceedings of the 2016 Annual Conference on Neural Information Processing Systems, NIPS 2016_. 3315–3323.
  * Harman et al. (2015) Mark Harman, Yue Jia, and Yuanyuan Zhang. 2015.  Achievements, Open Problems and Challenges for Search Based Software Testing. In _Proceedings of the 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015_. 1–12.
  * Harman and Jones (2001) Mark Harman and Bryan F. Jones. 2001.  Search-based software engineering.  _Information Software Technology_ 43, 14 (2001), 833–839.
  * Hellman (2020) Deborah Hellman. 2020\.  Measuring algorithmic fairness.  _Virginia Law Review_ 106, 4 (2020), 811–866.
  * Hooker et al. (2020) Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020.  Characterising Bias in Compressed Models.  _CoRR_ abs/2010.03058 (2020).
  * Hort et al. (2023) Max Hort, Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2023.  Bias mitigation for machine learning classifiers: A comprehensive survey.  _ACM Journal on Responsible Computing_ (2023).
  * Hort and Sarro (2021) Max Hort and Federica Sarro. 2021.  Did You Do Your Homework? Raising Awareness on Software Fairness and Discrimination. In _Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021_. 1322–1326.
  * Hort et al. (2021) Max Hort, Jie M. Zhang, Federica Sarro, and Mark Harman. 2021\.  Fairea: a model behaviour mutation approach to benchmarking bias mitigation methods. In _Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, ESEC/FSE 2021_. 994–1006.
  * Hsu et al. (2002) Rein-Lien Hsu, Mohamed Abdel-Mottaleb, and Anil K. Jain. 2002\.  Face Detection in Color Images.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 24, 5 (2002), 696–706.
  * Huang et al. (2020) Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020.  Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. In _Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020_. 65–83.
  * Huang et al. (2018) Xin Huang, He Zhang, Xin Zhou, Muhammad Ali Babar, and Song Yang. 2018.  Synthesizing qualitative research in software engineering: a critical review. In _Proceedings of the 40th International Conference on Software Engineering, ICSE 2018_. 1207–1218.
  * Hutchinson and Mitchell (2019) Ben Hutchinson and Margaret Mitchell. 2019.  50 Years of Test (Un)fairness: Lessons for Machine Learning. In _Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019_. 49–58.
  * Jalali and Wohlin (2012) Samireh Jalali and Claes Wohlin. 2012.  Systematic literature studies: database searches vs. backward snowballing. In _Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2012_. 29–38.
  * Jeong et al. (2009) Gaeul Jeong, Sunghun Kim, and Thomas Zimmermann. 2009.  Improving bug triage with bug tossing graphs. In _Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, ESEC/FSE 2009_. 111–120.
  * John et al. (2020) Philips George John, Deepak Vijaykeerthy, and Diptikalyan Saha. 2020.  Verifying Individual Fairness in Machine Learning Models. In _Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020_. 749–758.
  * Joo and Kärkkäinen (2020) Jungseock Joo and Kimmo Kärkkäinen. 2020.  Gender Slopes: Counterfactual Fairness for Computer Vision Models by Attribute Manipulation.  _CoRR_ abs/2005.10430 (2020).
  * Kärkkäinen and Joo (2021) Kimmo Kärkkäinen and Jungseock Joo. 2021.  FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision, WACV 2021_. 1547–1557.
  * Karras et al. (2018) Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018\.  Progressive Growing of GANs for Improved Quality, Stability, and Variation. In _Proceedings of the 6th International Conference on Learning Representations, ICLR 2018_.
  * Khan and Khan (2012) Mohd Ehmer Khan and Farmeena Khan. 2012.  A comparative study of white box, black box and grey box testing techniques.  _International Journal of Advanced Computer Science and Applications_ 3, 6 (2012).
  * Kimberly (1989) Crenshaw Kimberly. 1989\.  Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Anti-Discrimination Doctrine, Feminist Theory and Anti-Racist Politics. In _The University of Chicago Legal Forum_ , Vol. 140. 139.
  * Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif Mohammad. 2018.  Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, *SEM@NAACL-HLT 2018_. 43–53.
  * Kitamura et al. (2022) Takashi Kitamura, Zhenjiang Zhao, and Takahisa Toda. 2022\.  Applying Combinatorial Testing to Verification-Based Fairness Testing. In _Proceedings of the 14th International Symposium on Search-Based Software Engineering, SSBSE 2022_. 101–107.
  * Kleinberg et al. (2017) Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017\.  Inherent Trade-Offs in the Fair Determination of Risk Scores. In _8th Innovations in Theoretical Computer Science Conference, ITCS 2017_. 43:1–43:23.
  * Kuhlman et al. (2021) Caitlin Kuhlman, Walter Gerych, and Elke A. Rundensteiner. 2021\.  Measuring Group Advantage: A Comparative Study of Fair Ranking Metrics. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES 2021_. 674–682.
  * Kuhlman et al. (2019) Caitlin Kuhlman, MaryAnn VanValkenburg, and Elke Rundensteiner. 2019\.  Fare: Diagnostics for fair ranking using pairwise error metrics. In _Proceedings of The World Wide Web Conference 2019_. 2936–2942.
  * Kuhn et al. (2013) D Richard Kuhn, Raghu N Kacker, and Yu Lei. 2013.  _Introduction to combinatorial testing_.  CRC press.
  * Kusner et al. (2017) Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. 2017.  Counterfactual Fairness. In _Proceedings of the Annual Conference on Neural Information Processing Systems 2017, NIPS 2017_. 4066–4076.
  * Laghari and Demeyer (2018) Gulsher Laghari and Serge Demeyer. 2018.  Unit tests and component tests do make a difference on fault localisation effectiveness. In _Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ICSE 2018_. 280–281.
  * Lakhotia et al. (2007) Kiran Lakhotia, Mark Harman, and Phil McMinn. 2007\.  A multi-objective approach to search-based test data generation. In _Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2007_. 1098–1105.
  * Lample et al. (2017) Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017.  Fader Networks: Manipulating Images by Sliding Attributes. In _Proceedings of the Annual Conference on Neural Information Processing Systems 2017, NIPS 2017_. 5967–5976.
  * Langdon (1986) Winner Langdon. 1986\.  Do artifacts have politics?  _The Whale and the Reactor_ (1986).
  * Lee and Singh (2021) Michelle Seng Ah Lee and Jatinder Singh. 2021.  The Landscape and Gaps in Open Source Fairness Toolkits. In _Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2021_. 699:1–699:13.
  * Lewis (1977) W David Lewis. 1977\.  The Forces behind Technology: America by Design.  _Science_ 198, 4318 (1977), 722–723.
  * Li et al. (2023) Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023.  A survey on fairness in large language models.  _CoRR_ abs/2308.10149 (2023).
  * Li et al. (2022) Yanhui Li, Linghan Meng, Lin Chen, Li Yu, Di Wu, Yuming Zhou, and Baowen Xu. 2022\.  Training Data Debugging for the Fairness of Machine Learning Software. In _Proceedings of the 44th International Conference on Software Engineering, ICSE 2022_. 2215–2227.
  * Li and Xu (2021) Zhiheng Li and Chenliang Xu. 2021.  Discover the Unknown Biased Attribute of an Image Classifier. In _Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021_. 14950–14959.
  * Lin et al. (2022a) Bin Lin, Nathan Cassee, Alexander Serebrenik, Gabriele Bavota, Nicole Novielli, and Michele Lanza. 2022a.  Opinion Mining for Software Development: A Systematic Literature Review.  _ACM Transactions on Software Engineering Methodology_ 31, 3 (2022), 38:1–38:41.
  * Lin et al. (2022b) Xiaofeng Lin, Seungbae Kim, and Jungseock Joo. 2022b.  FairGRAPE: Fairness-aware GRAdient Pruning mEthod for Face Attribute Classification.  _Proceedings of the 2022 European Conference, Glasgow, UK, ECCV 2022_ (2022).
  * Liu et al. (2020) Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020\.  Does Gender Matter? Towards Fairness in Dialogue Systems. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020_. 4403–4416.
  * Lu et al. (2021) Qinghua Lu, Liming Zhu, Xiwei Xu, Jon Whittle, David Douglas, and Conrad Sanderson. 2021\.  Software Engineering for Responsible AI: An Empirical Study and Operationalised Patterns.  _CoRR_ abs/2111.09478 (2021).
  * Lu et al. (2022) Qinghua Lu, Liming Zhu, Xiwei Xu, Jon Whittle, David Douglas, and Conrad Sanderson. 2022\.  Software engineering for Responsible AI: An empirical study and operationalised patterns. In _Proceedings of the 44th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2022_. 241–242.
  * Lundberg and Lee (2017) Scott M. Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NIPS 2017_. 4765–4774.
  * Ma et al. (2022) Minghua Ma, Zhao Tian, Max Hort, Federica Sarro, Hongyu Zhang, Qingwei Lin, and Dongmei Zhang. 2022.  Enhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instances.  _CoRR_ abs/2209.08321 (2022).
  * Ma et al. (2020) Pingchuan Ma, Shuai Wang, and Jin Liu. 2020.  Metamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_. 458–465.
  * Mambreyan et al. (2022) Ara Mambreyan, Elena Punskaya, and Hatice Gunes. 2022\.  Dataset Bias in Deception Detection. In _Proceedings of the 26th International Conference on Pattern Recognition, ICPR 2022_.
  * Mathew et al. (2018) George Mathew, Amritanshu Agrawal, and Tim Menzies. 2018\.  Finding Trends in Software Research.  _IEEE Transactions on Software Engineering_ (2018).
  * Mehrabi et al. (2020a) Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2020a.  Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition. In _Proceedings of the 31st ACM Conference on Hypertext and Social Media, HT 2020_. 231–232.
  * Mehrabi et al. (2020b) Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2020b.  Man is to person as woman is to location: Measuring gender bias in named entity recognition. In _Proceedings of the 31st ACM Conference on Hypertext and Social Media_. 231–232.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021\.  A Survey on Bias and Fairness in Machine Learning.  _Comput. Surveys_ 54, 6 (2021), 115:1–115:35.
  * Mitchell et al. (2021) Shira Mitchell, Eric Potash, Solon Barocas, Alexander D’Amour, and Kristian Lum. 2021\.  Algorithmic fairness: Choices, assumptions, and definitions.  _Annual Review of Statistics and Its Application_ 8 (2021), 141–163.
  * Mittelstadt et al. (2019) Brent D. Mittelstadt, Chris Russell, and Sandra Wachter. 2019\.  Explaining Explanations in AI. In _Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT 2019_. 279–288.
  * Monjezi et al. (2023) Verya Monjezi, Ashutosh Trivedi, Gang Tan, and Saeid Tizpaz-Niari. 2023.  Information-theoretic testing and debugging of fairness defects in deep neural networks. In _Proceedings of the 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023_.
  * Moussa and Sarro (2022) Rebecca Moussa and Federica Sarro. 2022.  Do Not Take It for Granted: Comparing Open-Source Libraries for Software Development Effort Estimation.  (2022).  https://arxiv.org/abs/2207.01705
  * Muthukumar (2019) Vidya Muthukumar. 2019\.  Color-theoretic experiments to understand unequal gender classification accuracy from face images. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019_. 2286–2295.
  * Nahar et al. (2022) Nadia Nahar, Shurui Zhou, Grace A. Lewis, and Christian Kästner. 2022.  Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process. In _Proceedings of the 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022_. 413–425.
  * Nair et al. (2020) Vivek Nair, Zhe Yu, Tim Menzies, Norbert Siegmund, and Sven Apel. 2020.  Finding Faster Configurations Using FLASH.  _IEEE Transactions on Software Engineering_ 46, 7 (2020), 794–811.
  * Narasimhan et al. (2020) Harikrishna Narasimhan, Andrew Cotter, Maya R. Gupta, and Serena Wang. 2020.  Pairwise Fairness for Ranking and Regression. In _Proceedings of The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020_. 5248–5255.
  * Nejadgholi and Yang (2019) Mahdi Nejadgholi and Jinqiu Yang. 2019.  A Study of Oracle Approximations in Testing Deep Learning Libraries. In _Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019_. 785–796.
  * Noble (2018) Safiya Umoja Noble. 2018\.  Algorithms of oppression.  In _Algorithms of oppression_. New York university press.
  * Orgad and Belinkov (2022) Hadas Orgad and Yonatan Belinkov. 2022.  Choose Your Lenses: Flaws in Gender Bias Evaluation. In _Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing, GeBNLP 2022_. 151–167.
  * Orgad et al. (2022) Hadas Orgad, Seraphina Goldfarb-Tarrant, and Yonatan Belinkov. 2022\.  How Gender Debiasing Affects Internal Model Representations, and Why It Matters. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022_. 2602–2628.
  * Patel et al. (2022) Ankita Ramjibhai Patel, Jaganmohan Chandrasekaran, Yu Lei, Raghu N. Kacker, and D. Richard Kuhn. 2022\.  A combinatorial approach to fairness testing of machine learning models. In _Proceedings of the 15th IEEE International Conference on Software Testing, Verification and Validation Workshops ICST Workshops 2022_. 94–101.
  * Pearl (2009) Judea Pearl. 2009\.  _Causality_.  Cambridge university press.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011\.  Scikit-learn: Machine learning in Python.  _Journal of machine Learning research_ 12 (2011), 2825–2830.
  * Pedreschi et al. (2008) Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2008\.  Discrimination-aware data mining. In _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2008_. 560–568.
  * Peng et al. (2023) Kewen Peng, Joymallya Chakraborty, and Tim Menzies. 2023\.  FairMask: Better Fairness via Model-based Rebalancing of Protected Attributes.  _IEEE Transactions on Software Engineering_ (2023).
  * Perera et al. (2022) Anjana Perera, Aldeida Aleti, Chakkrit Tantithamthavorn, Jirayus Jiarpakdee, Burak Turhan, Lisa Kuhn, and Katie Walker. 2022.  Search-based fairness testing for regression-based machine learning systems.  _Empirical Software Engineering_ 27, 79 (2022).
  * Pessach and Shmueli (2022) Dana Pessach and Erez Shmueli. 2022.  A Review on Fairness in Machine Learning.  _Comput. Surveys_ 55, 3 (2022).
  * Pham et al. (2019) Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. 2019\.  CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries. In _Proceedings of the 41st International Conference on Software Engineering, ICSE 2019_. 1027–1038.
  * Pitoura et al. (2021) Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. 2021\.  Fairness in rankings and recommendations: an overview.  _VLDB Journal_ (2021).
  * Poria et al. (2020) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2020.  Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research.  _IEEE Transactions on Affective Computing_ (2020).
  * Pu et al. (2022) Muxin Pu, Meng Yi Kuan, Nyee Thoang Lim, Chun Yong Chong, and Mei Kuan Lim. 2022.  Fairness Evaluation in Deepfake Detection Models using Metamorphic Testing. In _Proceedings of the IEEE/ACM 7th International Workshop on Metamorphic Testing, MET@ICSE 2022_. 7–14.
  * Qian et al. (2021) Shangshu Qian, Viet Hung Pham, Thibaud Lutellier, Zeou Hu, Jungwon Kim, Lin Tan, Yaoliang Yu, Jiahao Chen, and Sameena Shah. 2021.  Are my deep learning systems fair? An empirical study of fixed-seed training.  _Advances in Neural Information Processing Systems_ 34 (2021), 30211–30227.
  * Quy et al. (2021) Tai Le Quy, Arjun Roy, Vasileios Iosifidis, and Eirini Ntoutsi. 2021.  A survey on datasets for fairness-aware machine learning.  _CoRR_ abs/2110.00530 (2021).
  * Rajan et al. (2022) Sai Sathiesh Rajan, Sakshi Udeshi, and Sudipta Chattopadhyay. 2022\.  AequeVox: Automated Fairness Testing of Speech Recognition Systems. In _Proceedings of the 25th International Conference on Fundamental Approaches to Software Engineering, FASE 2022_. 245–267.
  * Ribeiro et al. (2020) Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020_. 4902–4912.
  * Sano et al. (2022) Shinya Sano, Takashi Kitamura, and Shingo Takada. 2022\.  An efficient discrimination discovery method for fairness testing. In _Proceedings of the 34th International Conference on Software Engineering and Knowledge Engineering, SEKE 2022_. 200–205.
  * Sarro (2023) Federica Sarro. 2023\.  Search-Based Software Engineering in the Era of Modern Software Systems. In _Proceedings of the 31st IEEE International Requirements Engineering Conferece_. IEEE.
  * Sharma et al. (2021) Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike Wehrheim. 2021.  MLCHECK–Property-Driven Testing of Machine Learning Classifiers. In _Proceedings of the 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021_. 738–745.
  * Sharma and Wehrheim (2020) Arnab Sharma and Heike Wehrheim. 2020.  Automatic Fairness Testing of Machine Learning Models. In _Proceedings of the 32nd International Conference on Testing Software and Systems, ICTSS 2020_. 255–271.
  * Sharma et al. (2020) Shanya Sharma, Manan Dey, and Koustuv Sinha. 2020.  Evaluating gender bias in natural language inference. In _Proceedings of the NeurIPS 2020 Workshop on Dataset Curation and Security_.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019.  The Woman Worked as a Babysitter: On Biases in Language Generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019_. 3405–3410.
  * Si et al. (2021) Nian Si, Karthyek Murthy, Jose H. Blanchet, and Viet Anh Nguyen. 2021\.  Testing Group Fairness via Optimal Transport Projections. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021_. 9649–9659.
  * Singh and Joachims (2018) Ashudeep Singh and Thorsten Joachims. 2018.  Fairness of exposure in rankings. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2219–2228.
  * Sloane (2022) Mona Sloane. 2022\.  To make AI fair, here’s what we must learn to do.  _Nature_ 605, 7908 (2022), 9–9.
  * Smith et al. (2022) Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022\.  “I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_. 9180–9211.
  * Soremekun et al. (2022b) Ezekiel Soremekun, Sakshi Sunil Udeshi, and Sudipta Chattopadhyay. 2022b.  ASTRAEA: Grammar-based Fairness Testing.  _IEEE Transactions on Software Engineering_ (2022), 1–1.  https://doi.org/10.1109/TSE.2022.3141758
  * Soremekun et al. (2022a) Ezekiel O. Soremekun, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2022a.  Software Fairness: An Analysis and Survey.  _CoRR_ abs/2205.08809 (2022).
  * Stoychev and Gunes (2022) Samuil Stoychev and Hatice Gunes. 2022.  The Effect of Model Compression on Fairness in Facial Expression Recognition. In _Proceedings of Workshop on Applied Affect Recognition at the 26th International Conference on Pattern Recognition, ICPR 2022_.
  * Sun et al. (2021) Bing Sun, Jun Sun, Ting Dai, and Lijun Zhang. 2021\.  Probabilistic Verification of Neural Networks Against Group Fairness. In _Proceedings of the 24th International Symposium on Formal Methods, FM 2021_. 83–102.
  * Sun et al. (2022a) Bing Sun, Jun Sun, Long H. Pham, and Tie Shi. 2022a.  Causality-Based Neural Network Repair. In _Proceedings of the 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022_. 338–349.
  * Sun et al. (2019) Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019\.  Mitigating Gender Bias in Natural Language Processing: Literature Review. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019_. 1630–1640.
  * Sun et al. (2020) Zeyu Sun, Jie M. Zhang, Mark Harman, Mike Papadakis, and Lu Zhang. 2020.  Automatic testing and improvement of machine translation. In _Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020_. 974–985.
  * Sun et al. (2022b) Zeyu Sun, Jie M. Zhang, Yingfei Xiong, Mark Harman, Mike Papadakis, and Lu Zhang. 2022b.  Improving Machine Translation Systems via Isotopic Replacement. In _Proceedings of the 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022_. 1181–1192.
  * Tao et al. (2022) Guanhong Tao, Weisong Sun, Tingxu Han, Chunrong Fang, and Xiangyu Zhang. 2022.  RULER: Discriminative and Iterative Adversarial Training for Deep Neural Network Fairness. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022_. 1173–1184.
  * Taskesen et al. (2021) Bahar Taskesen, Jose H. Blanchet, Daniel Kuhn, and Viet Anh Nguyen. 2021.  A Statistical Test for Probabilistic Fairness. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2021_. 648–665.
  * Thanh et al. (2011) Binh Luong Thanh, Salvatore Ruggieri, and Franco Turini. 2011\.  k-NN as an implementation of situation testing for discrimination discovery and prevention. In _Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2011_. 502–510.
  * Tian et al. (2020) Yuchi Tian, Ziyuan Zhong, Vicente Ordonez, Gail E. Kaiser, and Baishakhi Ray. 2020.  Testing DNN image classifiers for confusion & bias errors. In _Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020_. 1122–1134.
  * Tizpaz-Niari et al. (2022) Saeid Tizpaz-Niari, Ashish Kumar, Gang Tan, and Ashutosh Trivedi. 2022.  Fairness-aware Configuration of Machine Learning Libraries. In _Proceedings of the 44th International Conference on Software Engineering, ICSE 2022_.
  * Torralba and Efros (2011) Antonio Torralba and Alexei A. Efros. 2011.  Unbiased look at dataset bias. In _Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011_. 1521–1528.
  * Tramèr et al. (2017) Florian Tramèr, Vaggelis Atlidakis, Roxana Geambasu, Daniel J. Hsu, Jean-Pierre Hubaux, Mathias Humbert, Ari Juels, and Huang Lin. 2017.  FairTest: Discovering Unwarranted Associations in Data-Driven Applications. In _Proceedings of the 2017 IEEE European Symposium on Security and Privacy, EuroS &P 2017_. 401–416.
  * Tushev et al. (2022) Miroslav Tushev, Fahimeh Ebrahimi, and Anas M Mahmoud. 2022\.  A Systematic Literature Review of Anti-Discrimination Design Strategies in the Digital Sharing Economy.  _IEEE Transactions on Software Engineering_ (2022).
  * Udeshi et al. (2018) Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay. 2018\.  Automated directed fairness testing. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018_. 98–108.
  * Urban et al. (2020) Caterina Urban, Maria Christakis, Valentin Wüstholz, and Fuyuan Zhang. 2020.  Perfectly parallel fairness certification of neural networks.  _Proceedings of the ACM on Programming Languages_ 4, OOPSLA (2020), 185:1–185:30.
  * Valentim et al. (2019) Inês Valentim, Nuno Lourenço, and Nuno Antunes. 2019\.  The Impact of Data Preparation on the Fairness of Software Systems. In _Proceedings of the 30th IEEE International Symposium on Software Reliability Engineering, ISSRE 2019_. 391–401.
  * Vasudevan and Kenthapadi (2020) Sriram Vasudevan and Krishnaram Kenthapadi. 2020.  LiFT: A Scalable Framework for Measuring Fairness in ML Applications. In _Proceedings of the 29th ACM International Conference on Information and Knowledge Management, CIKM 2020_. 2773–2780.
  * Verma and Rubin (2018) Sahil Verma and Julia Rubin. 2018.  Fairness definitions explained. In _Proceedings of the International Workshop on Software Fairness, FairWare@ICSE 2018_. 1–7.
  * Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. 2020.  Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In _Proceedings of the 2020 Annual Conference on Neural Information Processing Systems, NeurIPS 2020_.
  * Villani (2009) Cédric Villani. 2009\.  _Optimal transport: old and new_. Vol. 338.  Springer.
  * Voigt and Von dem Bussche (2017) Paul Voigt and Axel Von dem Bussche. 2017.  The EU general data protection regulation (GDPR).  _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ 10, 3152676 (2017), 10–5555.
  * Wan et al. (2023) Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael R. Lyu. 2023\.  BiasAsker: Measuring the Bias in Conversational AI System. In _Proceedings of the 31st ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023_.
  * Wang et al. (2020a) Angelina Wang, Arvind Narayanan, and Olga Russakovsky. 2020a.  REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets. In _Proceedings of the 16th European Conference, ECCV 2020_. 733–751.
  * Wang and Russakovsky (2021) Angelina Wang and Olga Russakovsky. 2021.  Directional Bias Amplification. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021_. 10882–10893.
  * Wang et al. (2022a) Jiannan Wang, Thibaud Lutellier, Shangshu Qian, Hung Viet Pham, and Lin Tan. 2022a.  EAGLE: Creating Equivalent Graphs to Test Deep Learning Libraries. In _Proceedings of the 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022_. 798–810.
  * Wang et al. (2022b) Jun Wang, Benjamin I. P. Rubinstein, and Trevor Cohn. 2022b.  Measuring and Mitigating Name Biases in Neural Machine Translation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022_. 2576–2590.
  * Wang et al. (2019) Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019.  Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations. In _Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019_. 5309–5318.
  * Wang et al. (2020b) Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. 2020b.  Deep learning library testing via effective model generation. In _Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020_. 788–799.
  * Wick et al. (2019) Michael L. Wick, Swetasudha Panda, and Jean-Baptiste Tristan. 2019.  Unlocking Fairness: a Trade-off Revisited. In _Proceedings of the Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019_. 8780–8789.
  * Xiao et al. (2023) Yisong Xiao, Aishan Liu, Tianlin Li, and Xianglong Liu. 2023\.  Latent imitator: Generating natural individual discriminatory instances for black-box fairness testing. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023_.
  * Xie and Wu (2020) Wentao Xie and Peng Wu. 2020.  Fairness Testing of Machine Learning Models Using Deep Reinforcement Learning. In _Proceedings of the 19th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2020_. 121–128.
  * Xu and Hu (2022) Guangxuan Xu and Qingyuan Hu. 2022.  Can Model Compression Improve NLP Fairness.  _CoRR_ abs/2201.08542 (2022).
  * Yang and Stoyanovich (2017) Ke Yang and Julia Stoyanovich. 2017.  Measuring fairness in ranked outputs. In _Proceedings of the 29th international conference on scientific and statistical database management_. 1–6.
  * Yang et al. (2022) Yu Yang, Aayush Gupta, Jianwei Feng, Prateek Singhal, Vivek Yadav, Yue Wu, Pradeep Natarajan, Varsha Hedau, and Jungseock Joo. 2022.  Enhancing Fairness in Face Detection in Computer Vision Systems by Demographic Bias Mitigation. In _Proceedings of AAAI/ACM Conference on AI, Ethics, and Society, AIES 2022_. 813–822.
  * Yang et al. (2021) Zhou Yang, Muhammad Hilmi Asyrofi, and David Lo. 2021.  BiasRV: uncovering biased sentiment predictions at runtime. In _Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021_. 1540–1544.
  * Zhang et al. (2022a) Jiang Zhang, Ivan Beschastnikh, Sergey Mechtaev, and Abhik Roychoudhury. 2022a.  Fair Decision Making via Automated Repair of Decision Trees. In _Proceedings of the 2nd IEEE/ACM International Workshop on Equitable Data & Technology, FairWare@ICSE 2022_. 9–16.
  * Zhang et al. (2015) Jie Zhang, Xiaoyin Wang, Dan Hao, Bing Xie, Lu Zhang, and Hong Mei. 2015.  A survey on bug-report analysis.  _Science China Information Sciences_ 58, 2 (2015), 1–24.
  * Zhang et al. (2019b) Jie Zhang, Lingming Zhang, Mark Harman, Dan Hao, Yue Jia, and Lu Zhang. 2019b.  Predictive Mutation Testing.  _IEEE Transactions on Software Engineering_ 45, 9 (2019), 898–918.
  * Zhang and Harman (2021) Jie M. Zhang and Mark Harman. 2021.  “Ignorance and Prejudice” in Software Fairness. In _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021_. 1436–1447.
  * Zhang et al. (2022b) Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022b.  Machine Learning Testing: Survey, Landscapes and Horizons.  _IEEE Transactions on Software Engineering_ 48, 2 (2022), 1–36.
  * Zhang et al. (2018) Li Zhang, Jia-Hao Tian, Jing Jiang, Yi-Jun Liu, Meng-Yuan Pu, and Tao Yue. 2018\.  Empirical Research in Software Engineering - A Literature Survey.  _Journal of Computer Science and Technology_ 33, 5 (2018), 876–899.
  * Zhang et al. (2016a) Lu Zhang, Yongkai Wu, and Xintao Wu. 2016a.  On discrimination discovery using causal networks. In _Proceedings of the 9th International Conference on Social, Cultural, and Behavioral Modeling, SBP-BRiMS 2016_. 83–93.
  * Zhang et al. (2016b) Lu Zhang, Yongkai Wu, and Xintao Wu. 2016b.  Situation Testing-Based Discrimination Discovery: A Causal Inference Approach. In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016_. 2718–2724.
  * Zhang et al. (2017) Lu Zhang, Yongkai Wu, and Xintao Wu. 2017.  A causal framework for discovering and removing direct and indirect discrimination. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017_. 3929–3935.
  * Zhang et al. (2019a) Lu Zhang, Yongkai Wu, and Xintao Wu. 2019a.  Causal Modeling-Based Discrimination Discovery and Removal: Criteria, Bounds, and Algorithms.  _IEEE Transactions on Knowledge and Data Engineering_ 31, 11 (2019), 2035–2050.
  * Zhang et al. (2021c) Lingfeng Zhang, Yueling Zhang, and Min Zhang. 2021c.  Efficient white-box fairness testing through gradient search. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2021_. 103–114.
  * Zhang and Sun (2022) Mengdi Zhang and Jun Sun. 2022.  Adaptive Fairness Improvement based on Causality Analysis. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022_. 6–17.
  * Zhang et al. (2023) Mengdi Zhang, Jun Sun, Jingyi Wang, and Bing Sun. 2023\.  TESTSGD: Interpretable testing of neural networks against subtle group discrimination.  _ACM Transactions on Software Engineering and Methodology_ (2023).
  * Zhang et al. (2020) Peixin Zhang, Jingyi Wang, Jun Sun, Guoliang Dong, Xinyu Wang, Xingen Wang, Jin Song Dong, and Ting Dai. 2020\.  White-box fairness testing through adversarial sampling. In _Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020_. 949–960.
  * Zhang et al. (2021a) Peixin Zhang, Jingyi Wang, Jun Sun, and Xinyu Wang. 2021a.  Fairness Testing of Deep Image Classification with Adequacy Metrics.  _CoRR_ abs/2111.08856 (2021).
  * Zhang et al. (2021b) Peixin Zhang, Jingyi Wang, Jun Sun, Xinyu Wang, Guoliang Dong, Xingen Wang, Ting Dai, and Jin Song Dong. 2021b.  Automatic Fairness Testing of Neural Classifiers through Adversarial Sampling.  _IEEE Transactions on Software Engineering_ (2021), 1–1.  https://doi.org/10.1109/TSE.2021.3101478
  * Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017.  Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017_. 2979–2989.
  * Zhao et al. (2022) Zhenjiang Zhao, Takahisa Toda, and Takashi Kitamura. 2022\.  Efficient Fairness Testing Through Hash-Based Sampling. In _Proceedings of the 14th International Symposium on Search-Based Software Engineering, SSBSE 2022_. 35–50.
  * Zheng et al. (2022) Haibin Zheng, Zhiqing Chen, Tianyu Du, Xuhong Zhang, Yao Cheng, Shouling Ji, Jingyi Wang, Yue Yu, and Jinyin Chen. 2022.  NeuronFair - Interpretable White-Box Fairness Testing through Biased Neuron Identification. In _Proceedings of the 44th International Conference on Software Engineering, ICSE 2022_.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In _Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017_. 2242–2251.