# The Cost of Arbitrariness for Individuals: Examining the Legal and Technical
Challenges of Model Multiplicity

Prakhar Ganesh∗1,2, Ihsan Ibrahim Daldaban∗1, Ignacio Cofone1, Golnoosh
Farnadi1,2

###### Abstract

Model multiplicity, the phenomenon where multiple models achieve similar
performance despite different underlying learned functions, introduces
arbitrariness in model selection. While this arbitrariness may seem
inconsequential in expectation, its impact on individuals can be severe. This
paper explores various individual concerns stemming from multiplicity,
including the effects of arbitrariness beyond final predictions, disparate
arbitrariness for individuals belonging to protected groups, and the
challenges associated with the arbitrariness of a single algorithmic system
creating a monopoly across various contexts. It provides both an empirical
examination of these concerns and a comprehensive analysis from the legal
standpoint, addressing how these issues are perceived in the anti-
discrimination law in Canada. We conclude the discussion with technical
challenges in the current landscape of model multiplicity to meet legal
requirements and the legal gap between current law and the implications of
arbitrariness in model selection, highlighting relevant future research
directions for both disciplines.

11footnotetext: Equal contribution

##  1 Introduction

Machine learning models have gained unprecedented power and influence across
diverse domains and real-world applications. As these models advance in
complexity and capability, a new challenge emerges — model multiplicity. This
phenomenon, often stemming from inherent overparameterization or
underspecification, is the existence of multiple solutions for the same
problem, thus introducing arbitrariness in selecting a single model.

In recent years, model multiplicity has garnered significant attention (Marx,
Calmon, and Ustun 2020; Black, Raghavan, and Barocas 2022; D’Amour et al.
2022), primarily focused on responsible behaviour in expectation (Ganesh
2024), e.g., group fairness (Sokol et al. 2022; Ganesh et al. 2023; Long et
al. 2023; Cooper et al. 2024), out-of-distribution robustness (McCoy, Min, and
Linzen 2020), overall privacy leakage (Kulynych et al. 2023), etc. Beyond the
technical exploration, discussions on policy changes and moral arguments
(Black, Raghavan, and Barocas 2022; Creel and Hellman 2022; Black et al. 2024)
have been pivotal in developing the discourse on multiplicity. Despite this
increasing attention, a critical aspect of multiplicity remains underexplored
— its wide-ranging impact on an individual and the legal implications of this
arbitrariness.

Existing studies tend to focus solely on multiplicity through model
predictions, and don’t address how the final model selection can affect
individuals in diverse ways. Consider, for instance, an individual from the
training data who may either face a high risk of information leakage or not,
depending on the arbitrary model choice under multiplicity. These effects of
arbitrariness go unnoticed in existing studies, and consequently, the cost of
arbitrariness for individuals is underestimated. Furthermore, the absence of a
definitive correct choice exacerbates the problem, as these decisions of model
selection from a set of good models usually hinge on pitting similar
individuals against each other.

Recent studies have also shed light on how certain demographic groups bear the
brunt of this arbitrariness more than others (Ganesh et al. 2023; Cooper et
al. 2024; Gomez et al. 2024). This arbitrariness not only raises concerns
about inadequate planning and recourse under multiplicity but also undermines
trust and predictability in the decision-making process. Consequently,
heightened arbitrariness within specific demographic groups can perpetuate
structured disparities, necessitating a thorough legal examination. This
becomes even more pressing when these models are directly adopted by various
downstream applications or service providers, thus creating a monopoly and
perpetuating structured arbitrariness against certain individuals throughout
the sector (Creel and Hellman 2022). While existing legal discourse on
multiplicity has centred exclusively on model behaviour in expectation (Black
et al. 2024), we instead focus on the impact of arbitrariness on individuals.

In this interdisciplinary work, we examine these concerns from both technical
and legal perspectives. We focus on employment, and particularly on hiring
decisions, advocating for a more nuanced understanding of multiplicity and its
influence on individuals. Our contributions are:

  * •

Redefining model multiplicity to emphasize individual concerns by
incorporating various causes of multiplicity beyond learning hyperparameters
as well as other impacts of multiplicity beyond just predictive multiplicity.

  * •

Arguing that increased levels of arbitrariness for a protected group can be
considered an adverse impact and, accordingly, a discriminatory employment
practice under the Canadian anti-discrimination law.

  * •

Scrutinizing the adoption of the same machine learning model or dataset for
hiring across diverse user contexts, followed by a discrimination analysis of
the sector-wide exclusion of individuals from employment opportunities and a
brief inquiry into how competition laws may be useful in ensuring algorithmic
fairness.

We conclude with a call for action to reduce the gap between the technical and
legal domains by (i) emphasizing the need to improve the methods of measuring
and auditing multiplicity and (ii) prompting a discussion on the lack of legal
arguments against the adverse impact of arbitrariness, especially in high-
stakes applications. In sum, our work encompasses a multifaceted analysis of
multiplicity, putting the concerns of individuals at the forefront.

##  2 Related Work

Prior to discussing our contributions, we provide a comprehensive review of
existing literature in multiplicity to contextualize our work within the
broader research landscape.

###  2.1 Model Multiplicity

As defined by Black, Raghavan, and Barocas (2022), model multiplicity is the
phenomenon where models with equivalent accuracy for a certain prediction task
differ in terms of their internals. Many recent works in literature have
explored the implications of multiplicity in machine learning, focusing on the
unpredictability of model behaviour during deployment (Marx, Calmon, and Ustun
2020; Black, Raghavan, and Barocas 2022). While D’Amour et al. (2022)
underscored the connection between model multiplicity and underspecification,
Ganesh (2024) showed that even additional specifications during model
selection cannot counter the overparameterization of deep learning models,
leaving the concerns of multiplicity unresolved. Paes et al. (2023) formalized
this further, using an information-theoretic approach to show that model
multiplicity is inevitable in settings with finite data. Beyond these
challenges, positive opportunities also arise from multiplicity, for instance,
the potential for better trade-offs and the development of responsible models
(Black, Raghavan, and Barocas 2022; Semenova, Rudin, and Parr 2022; Black et
al. 2024).

The majority of literature on multiplicity focuses on predictive multiplicity
(Marx, Calmon, and Ustun 2020), a special case of model multiplicity that
evaluates variations solely in the final predictions of the model. However,
other dimensions of quantifying multiplicity have also been explored. Hsu and
Calmon (2022) use the output probabilities to quantify multiplicity, while
Heljakka et al. (2022) extend this further, using the complete internal
representation of the model to measure model disagreements. Several works have
also explored the impact of multiplicity on the robustness of model
explanations (Karimi et al. 2022; Hancox-Li 2020).

The exploration of multiplicity in literature also extends to various problem
formulations. While most existing works define a set of good models to have
similar accuracy, this has also been expanded to perform model selection under
multiple metrics (Kulynych et al. 2023; Long et al. 2023; Ganesh 2024), and
even with models trained on different subsets of the data (Meyer,
Albarghouthi, and D’Antoni 2023). Multiplicity has also been addressed outside
classification, for instance, multi-target objective (Watson-Daniels et al.
2023), object detection (Hsu et al. 2024), etc. In our paper, we consolidate
these strands of research to formalize a more comprehensive definition of
multiplicity. This encompasses various causes of multiplicity such as dataset
choice, model hyperparameters, the stochasticity nature of learning, and
various effects of multiplicity like privacy and robustness.

###  2.2 Multiplicity and Responsible AI

In addition to choosing better models with multiplicity (Black, Raghavan, and
Barocas 2022), recent works have also studied the interplay between
multiplicity and other trustworthy metrics. Kulynych et al. (2023) showed that
improvements in privacy come at a cost of higher multiplicity and Long et al.
(2023) found a similar cost to achieve group fairness. Ali, Lahoti, and
Gummadi (2021) showed that addressing unfairness only in examples with high
multiplicity effectively eliminates biases and Cooper et al. (2024) showed
similarly that abstaining from highly uncertain predictions exhibits a
noticeable improvement in fairness.

Recent works have also explored the implications of arbitrariness itself, and
its integration into the broader landscape of responsible AI. Particularly
relevant to our work, several studies have shown a disparity in predictive
multiplicity across demographics (Ganesh et al. 2023; Cooper et al. 2024;
Gomez et al. 2024), thereby disproportionately impacting individuals from
various groups. Building on these insights, our study expands the empirical
analysis beyond predictions, revealing similar disparities in other metrics,
with a case study on privacy and robustness. Furthermore, while existing
literature is limited to empirically documenting these disparities, we provide
a legal examination under the anti-discrimination law in Canada.

###  2.3 Multiplicity and Monopolies

Recent progress in AI has enabled the introduction of a new class of large-
scale models, also referred to as foundational models, capable of being
adapted to a diverse set of domains. Given their impact, experts have raised
concerns about the widespread deployment of these models. Creel and Hellman
(2022) use the term algorithmic leviathans to characterize such algorithms and
offer a comprehensive discussion on the moral concerns of the standardization
of arbitrariness due to the adoption of a single algorithmic system across
various applications. Similarly, Vipra and Korinek (2023) advocate for better
regulations and the involvement of antitrust authorities to ensure a level
playing field in these domains. They highlight the risks associated with tech
giants training large-scale foundational models and the natural tendency of
such setups to move toward a monopoly, concentrating power in the hands of a
few companies. We contribute to this ongoing dialogue by providing a legal
discussion examining the arbitrariness in these monopolies.

###  2.4 Multiplicity and Law

There exists a wide range of literature on the adverse impacts of algorithmic
decision-making. More specifically, in the domain of employment, which is the
focus of our work, several scholars have discussed the legal implications of
these impacts and what legal approaches can be deployed to mitigate the
discriminatory outcomes of algorithmic models (Ajunwa 2020; Bildfell 2019;
Kelly-Lyth 2023; Kim 2017; Scherer, King, and Mrkonich 2019; Trindel, Kassir,
and Bent 2021). However, discussion on the legal implications of arbitrariness
resulting from model multiplicity has been limited. The closest work to ours
are Black et al. (2024) which makes observations on the legal implications of
model multiplicity in the context of credit-lending, and Kim (2022) that
highlights the significance of model multiplicity in de-biasing models. We
contribute to this discourse by bridging the gap between legal and technical
discussions on the adverse impacts of arbitrariness for individuals. We
explore the effects of multiplicity on individuals within the employment
domain and analyze the legal implications of these effects under the anti-
discrimination laws in Canada.

##  3 Redefining Model Multiplicity

Early work in multiplicity (Breiman 2001) draws inspiration from the
epistemological framework of the Rashomon effect (Anderson 2016): the
phenomenon of different yet equally plausible interpretations of the same
event. The Rashomon effect underscores the subjectivity and complexity
inherent in the interpretation of information. In machine learning, this is
translated to the existence of multiple models with similar performance,
despite learning different underlying functions. In this section, we dive
deeper into the causes and effects of multiplicity in real-world systems, use
these insights to motivate redefining multiplicity, and support our definition
of multiplicity with empirical studies.

###  3.1 Multiplicity in Context: Causes and Effects

#### Causes of Multiplicity:

The process of designing a machine learning model encompasses a multitude of
choices. Beginning with the data, decisions are made regarding what data to
gather, how to process and filter it, which features to select, etc. Even
decisions on partitioning the data for training and validation have been shown
to create multiplicity (Friedler et al. 2019; Meyer, Albarghouthi, and
D’Antoni 2023). Similarly, the learning algorithm design further entails
numerous decisions: the choice of model architecture, selection of
hyperparameters, various forms of stochasticity, and even the criteria for
model evaluation. Each of these decisions contributes to the cascade of
choices that directly impacts the multiplicity of the trained models.

Notably, these choices are not always informed or even intentional. For
instance, in some cases, choices are made by drawing upon insights from the
literature that focus on the effects of algorithm design on model multiplicity
(Wu et al. 2021; Ponomareva et al. 2023; Kulynych et al. 2023; Ganesh 2024).
In other cases, choices are driven by popular trends, such as the prevalent
use of a specific activation function instead of exploring various
alternatives (Dubey, Singh, and Chaudhuri 2022), or by the convenience of
employing model architecture available in more accessible frameworks like
PyTorch or TensorFlow. While even these choices may be studied in the future
to better grasp their impact on multiplicity, others remain inherently
arbitrary, like the selection of a random seed for training, defying attempts
at informed decision-making when training a learning model.

These choices culminate together into the final model, collectively impacting
its multiplicity. Thus, while specialized definitions of multiplicity can be
helpful for targeted technical analysis, it’s important to encompass all these
choices when discussing the real-world implications of multiplicity. For
instance, characterizing multiplicity solely as training multiple models on
the same dataset overlooks the significance of the dataset selection.
Similarly, defining multiplicity based on just the accuracy or error of the
model for a given task neglects the various considerations involved in model
evaluation, like measuring fairness to ensure adherence to certain standards.
Thus, it is important to cover all aspects of a learning pipeline when
defining multiplicity.

#### Reality of Training Multiple Models:

Existing literature often suggests training multiple models to mitigate
arbitrariness in model training, by creating ensembles, quantifying
multiplicity, or using model selection to navigate the causes of multiplicity
discussed above (Kulynych et al. 2023; Long et al. 2023; Creel and Hellman
2022). However, this isn’t always feasible if the model’s size or complexity
makes training multiple models computationally expensive. For instance,
consider HireVue, a recruitment company catering to over a third of the
Fortune 100 in the US. HireVue uses sophisticated models to assess candidates
via textual responses and video interviews (Larsen 2018), which makes training
enough models to capture multiplicity impractical.

Even in situations when it’s possible to train multiple models, the cost of
data collection can be a significant barrier. Consequently, multiple models
may be trained on the same dataset, perpetuating the arbitrariness inherent in
the data. For example, spurious correlations in the data can be viewed as a
manifestation of arbitrariness stemming from the dataset choice, as a
different dataset might exhibit different behaviour (Zech et al. 2018;
Hendrycks et al. 2021). Even when empirical methods can capture multiplicity,
model selection may lead to overfitting, hindering generalization (Ganesh
2024). These trends don’t necessarily dismiss the efficacy of training
multiple models. Rather, it highlights the complexities and limitations of
this approach, making it far from a universally applicable solution.

#### Effects of Multiplicity:

Multiplicity is often explored in scenarios where individuals encounter
conflicting predictions. This can create brittleness and uncertainty in model
decisions, undermining their reliability and hampering effective planning.
This becomes particularly problematic when the underlying model is
proprietary, leaving users in the dark about its inner workings. For example,
in the context of hiring, individuals affected by high multiplicity won’t be
able to anticipate outcomes based on observable patterns, severely hindering
their ability to discern which applications merit greater attention and plan
accordingly. As a result, they are left at a distinct disadvantage compared to
their peers who do not suffer from similar multiplicity and can thus
prioritize their efforts based on discernible trends.

The effects of multiplicity extend to model updates over time. Each update
introduces a fresh set of choices, thereby exacerbating multiplicity. Concerns
of arbitrariness over time have also been studied under the umbrella of
predictive churn (Milani Fard et al. 2016). Intriguingly, a recent study
highlighted the relationship between multiplicity and predictive churn,
indicating a common demographic vulnerable to fluctuating model outputs in
both cases (Watson-Daniels et al. 2024). Thus, an individual affected by high
multiplicity will also encounter higher uncertainty in recourse strategies, as
their predictions are more vulnerable to the model updates, potentially
rendering their recourse efforts redundant. Another line of research has
explored the possibility of algorithmic recourse becoming invalidated under
distribution shifts (Rawal, Kamar, and Lakkaraju 2021). We argue that these
problems arise even in situations without a distribution shift, simply due to
multiplicity in learning models.

Moreover, as hinted earlier, the effects of multiplicity extend beyond final
model predictions. For instance, model explanations and suggested recourse
decisions can vary noticeably due to multiplicity, and an individual’s
recourse feasibility hinges on the arbitrariness in model selection.
Similarly, multiplicity can impact robustness, leading to divergent outcomes
for similar cases, thus compromising system reliability and trust.
Multiplicity can even impact individuals whose data was used for training. For
example, multiplicity can affect the privacy risks of individuals,
necessitating perpetual vigilance, as arbitrary choices in model selection
could have potentially exposed individuals to privacy breaches, even if they
aren’t currently at risk. The effects of multiplicity are multifaceted, and we
should not narrow our scope to predictions on a given task, which can
underestimate the true risks of multiplicity. We will provide several
empirical studies in Section 3.3 for more practical examples of the various
effects of multiplicity discussed here.

###  3.2 Rashomon Sets and Model Multiplicity

With the causes and effects of multiplicity better understood, we now formally
define model multiplicity. We saw that the concept of multiplicity is deeply
rooted in the Rashomon effect, which highlights the diverse interpretations
that can arise from the same data. The models illustrating this Rashomon
effect are together known as a Rashomon set, or a set of competing models, a
set of good models, $\epsilon$-Rashomon set, $\epsilon$-Level set, etc. We’ll
stick to the term Rashomon set for consistency in our discussion. In
principle, a Rashomon set defines a set of models which are practically
indistinguishable for those deploying the model, underscoring the
arbitrariness in selecting one model over another. Thus, to define
multiplicity, we need to first define a Rashomon set. And to define a Rashomon
set, we need to define what it means for two models to be indistinguishable.

We introduce a set of metric delta functions, $\mathbf{\Delta}^{P}$, and
corresponding thresholds $\mathcal{E}^{P}$. A metric delta function takes as
input two models and measures the difference between them under the given
metric. These metric deltas serve as benchmarks for determining whether two
models are indistinguishable: if the difference in performance for every
$\delta^{P}_{i}\in\mathbf{\Delta}^{P}$ falls within the corresponding
threshold $\epsilon^{P}_{i}\in\mathcal{E}^{P}$. For example, one might define
the metric delta for accuracy as
$\delta_{Acc,D}(h_{1},h_{2})=|Acc(h_{1},D)-Acc(h_{2},D)|$. Formally,

######  Definition 1 (Rashomon Set)

Two models $h_{1},h_{2}$ belong to the same Rashomon set under performance
constraints $(\mathbf{\Delta}^{P},\mathcal{E}^{P})$ if they exhibit similar
performance for every metric in the given performance constraints, i.e.:

$\displaystyle\delta^{P}_{i}(h_{1},h_{2})\leq\epsilon^{P}_{i}\;\;\;\;\forall\;(\delta^{P}_{i},\epsilon^{P}_{i})\in(\mathbf{\Delta}^{P},\mathcal{E}^{P})$
(1)

The Rashomon set effectively captures the phenomenon of various learned models
being practically indistinguishable. However, the study of multiplicity is
crucial only in the context of these models also exhibiting diverse behaviour
when deployed, i.e., our interest lies in discerning meaningful differences,
rather than merely the existence of Rashomon sets. Given the various effects
of multiplicity discussed above, we generalize model multiplicity by binding
it to a metric delta $\delta^{M}$ and corresponding threshold $\epsilon^{M}$,
on models belonging to the same Rashomon set. Formally,

######  Definition 2 (Model Multiplicity)

Two models $h_{1},h_{2}$ exhibit multiplicity under performance constraints
$(\mathbf{\Delta}^{P},\mathcal{E}^{P})$ and multiplicity constraint
$(\delta^{M},\epsilon^{M})$, if they have similar performance for every metric
in the performance constraints yet differ on the metric in the multiplicity
constraint, i.e.:

$\displaystyle\begin{aligned}
\delta^{P}_{i}(h_{1},h_{2})\leq\epsilon^{P}_{i}\;\;\;\;&\forall\;(\delta^{P}_{i},\epsilon^{P}_{i})\in(\mathbf{\Delta}^{P},\mathcal{E}^{P})\\\
&\textrm{and}\quad\delta^{M}(h_{1},h_{2})>\epsilon^{M}\end{aligned}$ (2)

Note that the definitions here are quite similar to those already present in
the literature. However, we have made deliberate choices distinct from
previous approaches. Unlike several existing works (Marx, Calmon, and Ustun
2020; Black, Raghavan, and Barocas 2022), we do not restrict the models
$h_{1},h_{2}$ to a hypothesis class or limit them to being trained on data
from the same distribution. Similarly, unlike much of the prior work, we avoid
relying on a single metric of model similarity or only model predictions as a
measure of multiplicity. While these are not radical changes, we emphasize
these lack of restrictions to highlight the complete scope of the impact of
multiplicity as discussed above. Next, we briefly present empirical evidence
to support our choices.

###  3.3 Empirical Insights into Multiplicity

We now present a series of empirical studies to support our decision to expand
the definition of multiplicity.

#### Dataset:

We conduct our study on a Job Applicants dataset (Tankha 2023), which surveys
individuals on the website StackOverflow, collecting various attributes and
the success of their job applications. We filter the dataset to focus
specifically on applications to Canadian companies, resulting in $2779$ data
points for our analysis. This data is split into 80% for training and 20% for
testing. We also explore other datasets spanning different modalities in
Appendix B.

Figure 1: (a) Distribution of self-consistency scores for individuals in the
StackOverflow dataset (Tankha 2023), under Rashomon sets trained by changing
only the feature subsets, changing only the model hyperparameters, changing
only the random seeds, and changing all three simultaneously. (b) The same
distribution of self-consistency scores across various settings, but with
performance constraints including both accuracy and fairness (instead of just
accuracy in (a)). The impact of multiplicity is more pronounced in this case,
which underscores the significance of the model selection criteria. (c,d,e)
Distribution of self-consistency scores for predictions, robustness, defined
as the consistency of prediction under input noise, and privacy, defined as
membership inference. We find multiplicity concerns for metrics beyond
predictions as well as a disparate impact of multiplicity across various
groups (Ganesh 2024; Cooper et al. 2024; Gomez et al. 2024).

#### Models and Metrics:

Our learning model is a multi-layer perceptron with a single hidden layer of
$64$ neurons. To create the Rashomon set, we train multiple models while (i)
changing the subset of features to mimic multiplicity caused by the data, (ii)
changing batch size, learning rate, and the optimizer to account for the
multiplicity in the learning algorithm, and (iii) changing the random seeds to
replicate the stochasticity of learning. Each model is then evaluated using
the performance constraints, before being admitted into the Rashomon set.
Subsequently, we quantify multiplicity through a measure known as self-
consistency (Long et al. 2023). Self-consistency is defined as the likelihood
of obtaining the same output from two randomly selected models in the Rashomon
set. Details on model training and evaluation setup are delegated to Appendix
A.

#### Why do we need better Rashomon sets?

To emphasize the broader definition of Rashomon sets, we experiment with
various ways of training multiple models as well as different performance
constraints. Our results reveal that neglecting the diverse causes of
multiplicity can lead to an underestimation of the true risks associated with
arbitrariness in machine learning. To illustrate this, we begin by examining
the self-consistency scores for predictive multiplicity of individuals within
different Rashomon sets (each containing 20 models), under performance
constraint accuracy, constructed by varying feature subsets, hyperparameters,
random seeds, and a combination of all three, as depicted in Figure 1(a).
Notably, we find that altering only the random seeds, a common practice in
literature, underestimates the multiplicity risks.

Furthermore, the problem becomes severe when we change our performance
constraints. Instead of solely relying on accuracy, we introduce fairness,
defined as accuracy parity, and collect the results for the new Rashomon sets
in Figure 1(b). Despite being a more exclusive selection criterion, the
Rashomon sets created under these constraints—still comprising 20 models
each—demonstrate even higher multiplicity. By combining our insights from the
previous two sets of results, it becomes apparent that both dataset
variability and the incorporation of multiple performance constraints play an
important role in assessing the true risks of multiplicity. Strikingly, these
critical facets are often overlooked in existing literature, which focuses
instead on changing random seeds or only accuracy and loss as the performance
constraint, thus highlighting the need for a broader definition of Rashomon
sets.

#### Are multiplicity risks limited to model predictions?

We discussed how predictive multiplicity risks are often underestimated when
not all potential causes are taken into account. However, predictive
multiplicity is not the only consequence of arbitrariness in model selection.
We now delve into the broader implications of multiplicity, examining its
impact specifically on robustness and privacy within the Rashomon set. We use
the Rashomon set with all three model training variations outlined above and
with performance constraints of both accuracy and fairness. Robustness is
defined as the consistency of predictions under input noise and privacy as the
output of a membership inference attack. Additional details on these
definitions can be found in Appendix A. The results are collected in Figure
1(d,e).

Our analysis reveals multiplicity in both robustness and privacy across the
Rashomon set, illustrating that concerns of arbitrariness in model selection
transcend final predictions. Note that higher multiplicity in robustness (and
likewise, in privacy) does not necessarily equate to heightened concerns about
robustness (or privacy). Rather, they imply significant variability in these
concerns. For instance, individuals with consistently high concerns of
robustness (or privacy) will exhibit lower levels of multiplicity.

#### Who suffers the brunt of multiplicity?

Several recent works have shown a disparity in the risk of arbitrary decisions
across different demographics (Ganesh et al. 2023; Cooper et al. 2024; Gomez
et al. 2024). Building on these findings, we expand the analysis to our
broader definition of multiplicity. We examine per-group multiplicity for
predictions, robustness, and privacy, as detailed in Figure 1(c,d,e).

We find a noticeable disparity in multiplicity for the two groups based on
perceived gender. While we see a higher multiplicity in predictions and
robustness for the perceived gender group Female, we find the same group has
remarkably low multiplicity in privacy. These results align with observations
of disparity in multiplicity in the existing literature but extend to our
broader setting showing that individuals from various demographic groups may
face distinct forms of arbitrariness risk based on their group membership.
Equipped with these results, we will now perform a legal analysis of
arbitrariness under the anti-discrimination law in Canada and argue that
disparate arbitrariness across various groups should be recognized as a form
of discrimination.

##  4 Arbitrariness and Group Membership: The Legal Challenges

We perform our legal analysis of multiplicity in light of Canadian anti-
discrimination law, with a focus on the context of hiring. Hiring has recently
witnessed a dramatic increase in the delegation of decisions to machine
learning models (Capuano 2023; Green 2021). While these models are promising
in increasing efficiency and speed in hiring processes, they also risk
discriminating against job applicants based on their protected characteristics
(Ajunwa 2020; Cofone 2019). Note that, although our focus is on hiring, our
analysis here may shed light on the broader legal implications of
arbitrariness in algorithmic decisions.

We first briefly introduce employment discrimination law in Canada. We then
argue that under these laws, the disparity in arbitrariness constitutes a form
of discrimination.

###  4.1 Anti-Discrimination Laws: The Moore Test

Discrimination in hiring and recruitment is directly prohibited under the
provincial and territorial human rights legislation in Canada. For example,
the Human Rights Code of Ontario (noa 1990) ensures that “every person has a
right to equal treatment with respect to employment without discrimination
because of race, ancestry, place of origin, colour, ethnic origin,
citizenship, creed, sex, sexual orientation, gender identity, gender
expression, age, record of offences, marital status, family status or
disability”. Similarly, the Charter of Human Rights and Freedoms of Quebec
states that “Every person has a right to full and equal recognition and
exercise of his human rights and freedoms, without distinction, exclusion or
preference based on race, colour, sex, gender identity or expression,
pregnancy, sexual orientation, civil status, age” (noa 1976). While the
Canadian Charter of Rights and Freedoms does not directly apply to the
disputes between private individuals (noa 1989, 2010), the Supreme Court’s
interpretation of the equality rights under the Charter has greatly influenced
and continues to guide the provincial courts and tribunals’ interpretation and
application of anti-discrimination provisions under the applicable law, such
as provincial human rights provisions (England 2006; Bildfell 2019). The
Canadian Human Rights Act provides similar protection to those employed by the
federal government and several types of companies that are regulated by the
federal government, such as banks. Additionally, the Employment Equity Act
requires employers to achieve employment equity by taking proactive measures
targeting the detection and elimination of the hindrances negatively affecting
”women, Aboriginal peoples, persons with disabilities and members of visible
minorities” in employment (noa 1995).

In Canadian anti-discrimination doctrine, courts analyze whether a given
practice or treatment is discriminatory by applying a three-part test
developed in the Supreme Court case Moore v. British Columbia (Ministry of
Education) (noa 2012) and later confirmed in Stewart v. Elk Valley Coal Corp
(noa 2017). Under this test, commonly referred to as the Moore test,
“complainants are required to show that they have a characteristic protected
from discrimination under the Code [provincial/territorial human rights
legislation]; that they experienced an adverse impact with respect to the
service; and that the protected characteristic was a factor in the adverse
impact” (noa 2012). In the next subsection, we use the Moore test to determine
when there is a pattern of discrimination due to the disparity in
arbitrariness across various protected groups.

###  4.2 Arbitrariness as a Form of (Prima Facie) Discrimination

To determine whether model multiplicity would violate extant anti-
discrimination law in Canada, we have to scrutinize it under the Moore test.
The test establishes what situation is a prima facie case of discrimination,
meaning that there is a discriminatory impact that the employer will have to
justify for it not to constitute illegal discrimination.

Consider the following hypothetical scenario: A non-federally regulated
employer, located in Ontario, wants to automate its hiring and tasks a team of
developers with designing a machine learning model that will predict which job
candidates have a high potential for success at the employer’s company.
Developers build two different models that have the same performance and group
fairness but generate discrepant predictions on an individual level (i.e.,
they show model multiplicity). The employer settles on one of these models,
knowing that their options have higher multiplicity for a particular ethnic
minority. A candidate from the minority group who has been rejected then
challenges this algorithm in court on the grounds of it being discriminatory.
For simplicity, let us assume that the candidate was informed by the employer
about the use of the algorithm and the multiplicity of models encountered
after the development period.

The court, applying the Moore test to the case at hand, will first question
whether the complainant has a personal characteristic enumerated under the
Human Rights Code of the province (noa 2012). As we mentioned above, race is
one of the characteristics protected by the Human Rights Code of Ontario (noa
1990), and this complainant claims to have been discriminated against on the
basis of race. After clarifying this first step, the court will move to the
second step to question whether the claimant was subjected to an adverse
impact (noa 2012). This step is likely to be the most intricate one, as we
need to answer whether the disparity of arbitrariness in model selection gives
rise to an adverse impact. In this scenario, both models available to the
employer were recommending candidates from the minority group the claimant
belongs to while maintaining other measures of fairness. The problem that gave
rise to the said claim in court is that while the same individuals from the
majority group are being recommended by both models, the recommended
individuals from the minority group differ based on the model deployed. The
question to address in this second step is: Does this variation constitute
discrimination?

Under the Canadian anti-discrimination doctrine, courts often refer to the
definition of discrimination provided in Andrews v. Law Society (British
Columbia) in their analyses of discrimination claims. In Andrews,
discrimination is defined as ”a distinction, whether intentional or not but
based on grounds relating to personal characteristics of the individual or
group, which has the effect of imposing burdens, obligations or disadvantages
on such individual or group not imposed upon others, or which withholds or
limits access to opportunities, benefits and advantages available to other
members of society” (noa 1989)(emphasis added). Based on our analysis of model
multiplicity, higher arbitrariness for minority groups can be viewed as a
burden disproportionately imposed upon job candidates from these groups
because, as discussed briefly in Section 3.1, such higher arbitrariness
renders algorithmic decisions less predictable for them (for further
discussion on the morality of unpredictability, see Creel and Hellman (2022)).
Because there is less arbitrariness for majority groups, candidates from these
groups have a better opportunity to foresee whether they will be recommended
by the hiring algorithm despite the potential multiplicity of models. In other
words, the expected value of the decision may be the same for both groups, but
the variance is different, and the burden imposed on job candidates is not
limited only to the expected probability of being hired but also extends to
the arbitrary variability. This predictability, for example, gives candidates
from majority groups a possibility of recourse to adjust their behaviours and
get a more favourable decision from the algorithm (the predictability of
algorithmic decisions through explainable AI is considered an important
benefit that allows individuals to obtain the outcomes they want; see Wachter,
Mittelstadt, and Russell (2018); Verma et al. (2022)).

Likewise, better predictability (lower randomness) allows candidates from
majority groups to better project which jobs they should apply for, and lower
the number of jobs they will need to apply for to obtain one of them. On the
other hand, due to higher arbitrariness, candidates from minority groups do
not have this opportunity. Regardless of adjustments made to get a more
favourable outcome, their chances of getting selected by the hiring algorithm
will more significantly depend on an arbitrary choice of the model made by the
developers and employers.

The arbitrariness resulting from model multiplicity may have further impacts
on minority groups when models are updated. The updates of these models may
erratically alter their individual-level predictions (Watson-Daniels et al.
2024). These updates may hinder minority candidates’ access to employment
opportunities by further rendering the decisions less predictable. As minority
candidates are subjected to a higher level of arbitrariness, they may face
higher variability in the preferences of models following updates. For
example, the explanation of why a minority candidate was not selected by a
hiring model may include the notification that having more experience would
lead to a more favourable decision for the candidate. However, by the time the
said candidate applied again, the model might have been updated; and this
time, it might require the candidate to get another college degree to receive
a more favorable decision. This variability is likely to be less strong in the
case of majority candidates, as there is lower arbitrariness for them in the
decisions of the models.

The disparate situation depicted above constrains minority groups’ access to
equal employment opportunities. Therefore, it should be viewed as an adverse
impact under the second step of the Moore test.

In the third and last part of the Moore test, the court will question whether
the protected characteristic highlighted in the first step has indeed
influenced the occurrence of the adverse impact (noa 2012). This is the step
in which our empirical results are directly applicable. As we showed in our
empirical study in Section 3.3, the determining factor for whether a job
candidate will be subjected to a higher level of arbitrariness due to
multiplicity is their group membership. Being a member of a racial minority
translates into increased arbitrariness, which gives rise to reduced
predictability of algorithmic decisions in a disproportionate manner. Race,
the protected characteristic in this hypothetical case, thus plays a major
role in the emergence of this adverse impact. This conclusion, paired with the
conclusions in steps one and two of the Moore test, would render arbitrariness
in this hypothetical case a discriminatory practice.

The heightened levels of arbitrariness for minority groups, for these reasons,
should be viewed as a prima facie case of discrimination by courts. It should
be noted that this finding does not conclude a court’s analysis of the issue:
employers will still be able to justify this arbitrariness based on the
contextual facts of a case under the Meiorin test developed by the Supreme
Court in British Columbia (Public Service Employee Relations Commission) v.
B.C.G.E.U. (noa 1999). In other words, after prima facie discrimination is
shown, employers have the opportunity to show that the disparity is justified
for some reason. Proving prima facie discrimination, however, while not
conclusive, is a fundamental step in guaranteeing the rights of individuals.

##  5 The Potential Monopoly of A Machine Learning Model

After examining the multiplicity disparity across groups, we shift our focus
to a special case of arbitrariness in machine learning: the challenges of an
arbitrary decision made by an algorithm that is widely adopted across various
applications or user contexts, known as an algorithmic leviathan (Creel and
Hellman 2022). Algorithmic leviathans, as proposed by Creel and Hellman
(2022), are “automated decision-making systems that make uniform judgments
across broad swathes of a sector”. Such systems become problematic when these
uniform judgements are arbitrary, leading to a systematic response against
particular groups of individuals. Simply discouraging such a monopoly is
impractical in scenarios where training multiple models can be cost-
prohibitive, where there is a deliberate absence of multiple decision-makers
(e.g., immigration decisions in a country), or where arbitrariness originates
not from the model but from the underlying dataset (e.g., shared arbitrariness
due to everyone using the same dataset). We direct readers to Creel and
Hellman (2022) for a more in-depth discussion of the moral concerns associated
with deploying algorithmic leviathans.

In this paper, we extend the arguments to legal concerns on the existence of
such leviathans. We start by examining the compounded impact of a monopoly on
the existing disparity in arbitrariness across different protected groups from
the lens of anti-discrimination law. Following this, we draw attention to the
far-reaching discriminatory impacts of algorithmic leviathans which result
from the blanket rejection of individuals. We then ask: ‘What if an
algorithmic leviathan, while not discriminating against protected groups,
still creates a structured adversity for certain individuals?’

###  5.1 Monopoly, Anti-Discrimination Law and Competition Law

The adverse impact arising from the increased levels of arbitrariness for
minority groups will be compounded when a single algorithmic model is deployed
widely in a sector. Where the majority of employers within a sector use the
same model to make hiring decisions, minority groups will be systematically
subjected to lower levels of predictability. This will harm minority groups’
access to employment opportunities as they will not be able to increase the
likelihood of getting recommended by the hiring model by adjusting their
behaviours and they will be subjected to generalized welfare-decreasing
uncertainty.

The compounded reduction in reduced predictability is not the only threat
posed by algorithmic leviathans. These models, irrespective of whether they
have higher arbitrariness for minority groups, may still be giving rise to
discriminatory impacts. A hiring model that is deployed sector-wide risks
excluding certain job candidates from employment opportunities (Creel and
Hellman 2022). If a blanket rejection arises in connection with individuals’
characteristics protected by anti-discrimination law, the disparate impact of
arbitrariness on protected groups will be severe. In such a case, the members
of the group rejected by the model may have significantly reduced
opportunities to find employment within a specific sector. Such an exclusion
would exacerbate the historical and ongoing under-representation of
marginalized groups in the workforce.

Detecting the severity and extent of discrimination in such a case, however,
may not be an easy task. The fact that a certain implementation of an
algorithmic leviathan has an adverse impact on a minority group does not
automatically mean that all other implementations of the same model have the
same adverse impact too. Employers, after obtaining the hiring model from
developers, might have fine-tuned it in accordance with their own business
necessities. This means that different implementations of the same model may
have different tendencies in selecting and rejecting job candidates.
Accordingly, the discriminatory impact detected within a single instance of
the model may not be found in other instances (Wang and Russakovsky 2023).

The same situation applies to the absence of discrimination in a certain
implementation of an algorithmic leviathan as well. The fact that no
discriminatory outcome is detected within the predictions of an algorithmic
leviathan does not guarantee that all other implementations of the same model,
too, do not give rise to discrimination. As fine-tuning can render a model
unbiased, it can also turn an unbiased model into a biased one. Employers, in
their attempts to align their hiring model with the needs of their businesses,
might render them biased against certain minorities.

In some cases, the monopoly of a single hiring model within a sector may give
rise to unfairness in access to employment opportunities despite not being
discriminatory against any group. This happens when an unbiased, non-
discriminatory, hiring model dominates a certain sector. While attaining a
completely unbiased algorithmic model is highly difficult due to the intricate
connections between neutral variables and protected characteristics (Cofone
2019), for the sake of our analysis and to be able to study these algorithmic
leviathans separate from the discrimination against protected groups we saw in
the previous section, let us assume that the model in question has achieved
this fairness. Also, assume that no fine-tuning has been made by employers,
and accordingly, this model, in each instance, is generating the same
predictions. That is, it is recommending the same set of candidates while
rejecting another set of candidates in all instances. This means that some
candidates will be excluded from job opportunities as they will be
consistently rejected by the same model. This situation is certainly unfair
for the these candidates given that they could receive more favorable outcomes
had this model not attained monopoly within the given sector. But it does not
constitute a case of discrimination because the said impediment does not occur
in connection with a protected characteristic and it does not adversely impact
any group.

Where this sector-wide uniformity in decisions has an adverse impact on
minority groups, there exists discrimination. It should also be noted that
where an algorithmic model has adverse impacts on the basis of protected
characteristics, it will be discriminatory regardless of whether it is
deployed widely in a sector. The adverse impacts of an algorithmic leviathan
simply aggravate the disadvantageous situations of minority groups and give
rise to compounded discrimination against them. But where this uniformity does
not disadvantage any group in connection with their protected characteristics,
then it is not discriminatory.

Rejecting some candidates while accepting others is an inherent part of
hiring. As there are limited employment opportunities, some candidates will
inevitably be rejected by employers. In our hypothetical example, the model is
making rejection and acceptance decisions with no bias and discrimination. But
as the model is deployed widely within a single sector, it is consistently
rejecting the same candidates; making the monopoly of a single model ethically
disturbing.

This problem goes beyond the reach of anti-discrimination law. However, the
situation these consistently rejected candidates are put into is deeply unfair
in that their access to job opportunities is severely restricted due to a
hiring model’s dominance within a sector (Creel and Hellman 2022). It requires
a legal and policy reflection that considers the consequences of the dominance
of a single hiring product within a sector. In this regard, competition law
may be useful for regulators to prevent such unfair outcomes from occurring
(Creel and Hellman 2022).

Competition law aims to foster and maintain healthy competition between the
participants of markets (OECD Global Forum on Competition 2003). Competition
between market participants has implications for two different concepts of
fairness (Gerber 2020). Firstly, competitive fairness requires fair conditions
under which market participants can compete with each other (Gerber 2020).
Secondly, consumer fairness requires the protection of consumers from the
detrimental practices of dominant companies (Gerber 2020).

Regulators may rely on both of these principles in preventing the sector-wide
exclusion of individuals from employment opportunities. Enhancing the fairness
of conditions under which the developers of hiring models compete with each
other can prevent the occurrence of algorithmic leviathans, which in turn,
would prevent the said unfairness candidates suffer from in searching for
employment. For example, in the development of algorithmic leviathans, access
to unique data fortifies the dominant position of large companies in the
market, thus adversely affecting the competitive power of smaller companies
(Schrepel and Pentland 2023). Vipra and Korinek (2023) argue that regulators,
to strengthen competition between these companies, may consider prohibiting
the developers from being the sole owner of the training data. Democratizing
access to data in such a way would strengthen smaller companies’ capability to
compete with larger ones; hence, it would consequently decrease the
possibility of a single hiring model gaining market dominance. However,
regulators should also pay attention to the possibility that the training of
several models with the same dataset may lead these models to reflect similar
tendencies in their predictions, thus giving rise to another sector-wide
exclusion of individuals from employment opportunities (Creel and Hellman
2022)

An in-depth discussion of how competition law may be deployed in preventing
the unfairness engendered by dominant hiring models is beyond the scope of
this paper. But, as the above discussion on potential regulatory interventions
shows, competition law is a promising legal tool for solving this problem.
Regulators, therefore, should explore feasible regulatory measures under
competition law to maintain and enhance equal access to employment.

##  6 Open Research Questions

In this paper, we explored the impact of arbitrariness in model selection due
to multiplicity and discussed the discriminatory practices it can foster.
However, there remain significant challenges, spanning both technical and
legal domains. We conclude our paper by outlining the key open research
questions in model multiplicity. By explicitly outlining paths for future
research, we aim to encourage collaboration between both disciplines to
address these challenges.

###  6.1 Auditing Model Multiplicity

We start by outlining the key technical challenges to auditing model
multiplicity in real-world scenarios.

#### Cost-effective Auditing of Multiplicity:

Various metrics for measuring multiplicity can be found in the literature,
either for individual data points (Cooper et al. 2024; Kulynych et al. 2023;
Black, Raghavan, and Barocas 2022; Hsu and Calmon 2022) or the entire datasets
(Heljakka et al. 2022; Long et al. 2023; Marx, Calmon, and Ustun 2020); either
for measuring multiplicity of predictions (Cooper et al. 2024; Kulynych et al.
2023; Black, Raghavan, and Barocas 2022; Long et al. 2023; Marx, Calmon, and
Ustun 2020) or capturing a wider model behaviour (Heljakka et al. 2022; Hsu
and Calmon 2022); etc. Yet, despite the differences, all these metrics share a
commonality—they rely on training an ensemble of models. However, this is
impractical and costly, as the burden of training multiple copies of a model
can be unreasonable for companies utilizing such models, posing a significant
barrier to auditing model multiplicity.

Recent work by Hsu et al. (2024) broke this trend and showed the advantages of
using Monte Carlo dropout as an efficient alternative to training multiple
models for measuring model multiplicity. This opens up a promising avenue for
future research: understanding the relationship between predictive uncertainty
and multiplicity. Predictive uncertainty is a model’s calibrated uncertainty
concerning a specific prediction. Acknowledging that machine learning models
tend to make arbitrary decisions in uncertain situations, we might be able to
use predictive uncertainty to approximate model multiplicity. Existing work on
measuring predictive uncertainty by using an ensemble of models further
reinforces this connection (Lakshminarayanan, Pritzel, and Blundell 2017).
Thus, gaining a better understanding of the relationship between predictive
uncertainty and model multiplicity can allow us to leverage cost-effective
methods of uncertainty measurement, marking a crucial step toward auditing
arbitrariness and multiplicity in machine learning.

It should also be noted that despite recent advancements, we have a limited
understanding of how to mitigate and manage multiplicity, with existing
literature primarily focused on auditing multiplicity as discussed above.
Notably, the most common recommendation in literature, model selection based
on appropriate constraints, has been shown to cause overfitting and does not
generalize to unseen settings (Ganesh 2024). Addressing this gap thus becomes
imperative to be able to tackle multiplicity in the real world.

#### Arbitrariness as a Fairness Metric:

Defining a metric is an important initial step in any auditing process,
enabling us to measure and track progress while facilitating comparisons
across diverse settings. Although we’ve demonstrated significant disparity in
arbitrariness across different protected groups (see Section 3.3), an
essential discussion remains on translating this disparity into a fairness
metric. Furthermore, there are several multiplicity metrics currently used in
the literature (Cooper et al. 2024; Kulynych et al. 2023; Black, Raghavan, and
Barocas 2022; Somepalli et al. 2022; Heljakka et al. 2022; Long et al. 2023;
Marx, Calmon, and Ustun 2020; Hsu and Calmon 2022), yet none have been
compared with each other to determine if they even have any discernible
differences. Thus, it is critical to consolidate existing metrics, conduct a
comprehensive study on their applicability in defining fairness under
arbitrariness, and use them to quantify the biases present in a model.

#### Beyond Predictive Multiplicity:

Throughout the majority of this paper, and in much of the literature on model
multiplicity, the focus remains on the case of predictive multiplicity.
However, it is worth noting that, even when multiple models in the Rashomon
set produce the same prediction for an individual, the arbitrariness of model
selection can still introduce adversity, as we show in Section 3.3. Thus,
while predictive multiplicity is fundamental, it is important to recognize
that the adverse effects of multiplicity extend beyond predictions.
Consequently, further exploration and analysis of these effects would be
beneficial.

###  6.2 Anti-discrimination and Competition Laws

We discuss, in this section, the gaps between law and the implications of
model multiplicity, emphasizing the legal discussions required for future
research.

#### Arbitrariness in a Broader Context:

Our analysis in this paper has been limited to Canada and the context of
hiring. We encourage further discussions from the perspective of other
jurisdictions, as a comparative analysis would provide lawmakers with a more
profound insight into the problem. Further research would also be beneficial
to shed light on how model multiplicity may impact equality and non-
discrimination in other domains where algorithmic decision-making is
increasingly deployed, such as predictive policing, credit lending, and
marketing.

#### Competition Law for Algorithmic Leviathans:

In this paper, we argued that heightened levels of arbitrariness for
individuals from minority groups should be considered as an adverse impact
under anti-discrimination laws. We also noted that where protected
characteristics of individuals do not play a role in the occurrence of the
disadvantageous outcomes of model multiplicity, there is no discrimination
under the extant anti-discrimination laws. In the case of an algorithmic
system gaining market dominance, we noted that competition law may be useful
in combating the unfairness emerging from these models. How better competition
and stronger fairness can be achieved simultaneously in a sector that relies
on such algorithmic leviathans is an important policy question that requires
further research and discussion.

#### A Closer Look on Foundational Models:

The advent of foundational models, commonly defined as large-scale AI models
capable of adapting to a wide range of applications, is a special case of
algorithmic leviathans. These models wield exceptional influence, intensifying
concerns about the arbitrariness present in their decisions. Unlike other
algorithmic leviathans, foundational models transcend individual sectors and
are used across a diverse array of contexts, amplifying their impact. This
universality highlights the potential harm associated with the arbitrariness
and the discriminatory challenges of foundational models.

These models stand out as a case study that demands special attention, given
their common practice of employing closed-source datasets and models. This
practice of closed systems hampers transparency, complicating efforts to audit
the model for arbitrariness. The extreme costs of training, and thus the
infeasibility of retraining foundational models, also underscores the
imperative of addressing arbitrariness online. Moreover, the majority of
foundational models are controlled by established tech monopolies,
facilitating rapid integration into a range of products. This necessitates
enhanced regulatory measures to prevent their unchecked dominance in various
fields. Consequently, a discussion among legal and policy experts is necessary
to address the concerns of fair competition and safeguard individuals against
adversities resulting from multiplicity in foundational models.

##  7 Discussion and Limitations

In this paper, we provided empirical results and legal arguments on model
multiplicity to address the cost of arbitrariness for individuals. A
limitation of our legal discussion lies in the absence of real-world case
studies. We believe that the lack of awareness among the general public
regarding the arbitrary nature of model selection contributes to the absence
of legal studies on the arbitrariness of automated systems; our work serves as
a crucial step in shedding light on the inherent problems associated with
multiplicity. We also use only a single tabular dataset in our experiments
that is tailored to the hiring domain, while the other datasets in our
appendix are from diverse domains. This is due to the scarcity of publicly
available hiring datasets. However, we believe that our empirical results are
still relevant, as they highlight the arbitrariness inherent in learning.

With this paper, we aim to initiate an interdisciplinary conversation on the
impact of model multiplicity on individuals, increasing awareness and laying
the groundwork for further research in this domain.

## Acknowledgements

Funding support for project activities has been partially provided by Canada
CIFAR AI Chair, and Google award. We also express our gratitude to Compute
Canada for their support in providing facilities for our evaluations.

## References

  * noa (1976) 1976\.  Charter of Human Rights and Freedoms, CQLR c C-12.
  * noa (1989) 1989\.  Andrews v. Law Society (British Columbia).  _CarswellBC_ , 16.
  * noa (1990) 1990\.  Human Rights Code, RSO 1990, c. H. 19.
  * noa (1995) 1995\.  Employment Equity Act, SC 1995, c 44.
  * noa (1999) 1999\.  British Columbia (Public Service Employee Relations Commission) v. B.C.G.E.U.  _CarswellBC_ , 1907.
  * noa (2010) 2010\.  Ontario (Director of Disability Support Program) v. Tranchemontagne.  _ONCA_ , 593.
  * noa (2012) 2012\.  Moore v. British Columbia (Ministry of Education).  _SCC_ , 61.
  * noa (2017) 2017\.  Stewart v. Elk Valley Coal Corp.  _SCC_ , 30.
  * Ajunwa (2020) Ajunwa, I. 2020.  The Paradox of Automation as Anti-Bias Intervention.  _Cardozo L Rev_ , 41(5): 1671–1742.
  * Ali, Lahoti, and Gummadi (2021) Ali, J.; Lahoti, P.; and Gummadi, K. P. 2021.  Accounting for Model Uncertainty in Algorithmic Discrimination.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , 336–345. Virtual Event USA: ACM.  ISBN 978-1-4503-8473-5.
  * Anderson (2016) Anderson, R. 2016.  The Rashomon effect and communication.  _Canadian Journal of Communication_ , 41(2): 249–270.
  * Bildfell (2019) Bildfell, C. 2019.  Hiring Algorithms in the Canadian Private Sector: Examining the Promise of Greater Workplace Equality.  _CJLT_ , 17(2): 115–143.
  * Black et al. (2024) Black, E.; Koepke, J. L.; Kim, P.; Barocas, S.; and Hsu, M. 2024.  The Legal Duty to Search for Less Discriminatory Algorithms.  In _Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency_.
  * Black, Raghavan, and Barocas (2022) Black, E.; Raghavan, M.; and Barocas, S. 2022.  Model multiplicity: Opportunities, concerns, and solutions.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , 850–863.
  * Breiman (2001) Breiman, L. 2001.  Statistical modeling: The two cultures (with comments and a rejoinder by the author).  _Statistical science_ , 16(3): 199–231.
  * Capuano (2023) Capuano, A. 2023.  Automated Candidate Screening, Algorithms and Artificial Intelligence in Recruitment.  In _Class and Social Background Discrimination in the Modern Workplace_ , chapter Class and Social Background Discrimination in the Modern Workplace, 150–176. Bristol University Press.  ISBN 978-1-5292-2297-5.
  * cjadams et al. (2017) cjadams; Sorenson, J.; Elliott, J.; Dixon, L.; McDonald, M.; McDonald; nithum; and Cukierski, W. 2017.  Toxic Comment Classification Challenge.
  * Cofone (2019) Cofone, I. N. 2019.  Algorithmic Discrimination Is an Information Problem.  _Hastings LJ_ , 70(6): 1389–1444.
  * Cooper et al. (2024) Cooper, A. F.; Lee, K.; Choksi, M. Z.; Barocas, S.; De Sa, C.; Grimmelmann, J.; Kleinberg, J.; Sen, S.; and Zhang, B. 2024.  Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, 22004–22012.
  * Creel and Hellman (2022) Creel, K.; and Hellman, D. 2022.  The algorithmic leviathan: Arbitrariness, fairness, and opportunity in algorithmic decision-making systems.  _Canadian Journal of Philosophy_ , 52(1): 26–43.
  * D’Amour et al. (2022) D’Amour, A.; Heller, K.; Moldovan, D.; Adlam, B.; Alipanahi, B.; Beutel, A.; Chen, C.; Deaton, J.; Eisenstein, J.; Hoffman, M. D.; et al. 2022.  Underspecification presents challenges for credibility in modern machine learning.  _The Journal of Machine Learning Research_ , 23(1): 10237–10297.
  * Dubey, Singh, and Chaudhuri (2022) Dubey, S. R.; Singh, S. K.; and Chaudhuri, B. B. 2022.  Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark.  _Neurocomputing_ , 503: 92–108.
  * England (2006) England, G. 2006.  The Impact of the Charter on Individual Employment Law in Canada: Rewriting an Old Story.  _Canadian Lab & Emp LJ_, 13: 1–40.
  * Friedler et al. (2019) Friedler, S. A.; Scheidegger, C.; Venkatasubramanian, S.; Choudhary, S.; Hamilton, E. P.; and Roth, D. 2019.  A comparative study of fairness-enhancing interventions in machine learning.  In _Proceedings of the conference on fairness, accountability, and transparency_ , 329–338.
  * Ganesh (2024) Ganesh, P. 2024.  An Empirical Investigation into Benchmarking Model Multiplicity for Trustworthy Machine Learning: A Case Study on Image Classification.  In _2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. Waikoloa, HI, USA: IEEE.
  * Ganesh et al. (2023) Ganesh, P.; Chang, H.; Strobel, M.; and Shokri, R. 2023.  On The Impact of Machine Learning Randomness on Group Fairness.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23, 1789–1800. New York, NY, USA: Association for Computing Machinery.  ISBN 9798400701924.
  * Gerber (2020) Gerber, D. J. 2020.  _Competition Law and Antitrust: A Global Guide_.  New York: Oxford University Press.
  * Gomez et al. (2024) Gomez, J. F.; Machado, C. V.; Paes, L. M.; and Calmon, F. P. 2024.  Algorithmic Arbitrariness in Content Moderation.  _arXiv preprint arXiv:2402.16979_.
  * Green (2021) Green, A. 2021.  Want a Job? The AI Will See You Now.  MIT Technology Review (Podcast).
  * Hancox-Li (2020) Hancox-Li, L. 2020.  Robustness in machine learning explanations: does it matter?  In _Proceedings of the 2020 conference on fairness, accountability, and transparency_ , 640–647.
  * Hardt, Price, and Srebro (2016) Hardt, M.; Price, E.; and Srebro, N. 2016.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ , 29.
  * Heljakka et al. (2022) Heljakka, A.; Trapp, M.; Kannala, J.; and Solin, A. 2022.  Representational multiplicity should be exposed, not eliminated.  _arXiv preprint arXiv:2206.08890_.
  * Hendrycks et al. (2021) Hendrycks, D.; Zhao, K.; Basart, S.; Steinhardt, J.; and Song, D. 2021.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 15262–15271.
  * Hofmann (1994) Hofmann, H. 1994.  Statlog (German Credit Data).  UCI Machine Learning Repository.  DOI: https://doi.org/10.24432/C5NC77.
  * Hsu and Calmon (2022) Hsu, H.; and Calmon, F. 2022.  Rashomon Capacity: A Metric for Predictive Multiplicity in Classification.  _Advances in Neural Information Processing Systems_ , 35: 28988–29000.
  * Hsu et al. (2024) Hsu, H.; Li, G.; Hu, S.; et al. 2024.  Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation.  _arXiv preprint arXiv:2402.00728_.
  * Karimi et al. (2022) Karimi, A.-H.; Barthe, G.; Schölkopf, B.; and Valera, I. 2022.  A survey of algorithmic recourse: contrastive explanations and consequential recommendations.  _ACM Computing Surveys_ , 55(5): 1–29.
  * Kelly-Lyth (2023) Kelly-Lyth, A. 2023.  Algorithmic discrimination at work.  _ELLJ_ , 14(2): 152–171.
  * Kim (2017) Kim, P. T. 2017.  Data-Driven Discrimination at Work.  _Wm & Mary L Rev_, 58(3): 857–936.
  * Kim (2022) Kim, P. T. 2022.  Race-Aware Algorithms: Fairness, Nondiscrimination and Affirmative Action.  _Cal L Rev_ , 110(5): 1539–1596.
  * Kulynych et al. (2023) Kulynych, B.; Hsu, H.; Troncoso, C.; and Calmon, F. P. 2023.  Arbitrary Decisions are a Hidden Cost of Differentially Private Training.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , 1609–1623.
  * Lakshminarayanan, Pritzel, and Blundell (2017) Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30.
  * Larsen (2018) Larsen, L. 2018.  Train, Validate, Re-train: How We Build HireVue Assessments Models.
  * Long et al. (2023) Long, C. X.; Hsu, H.; Alghamdi, W.; and Calmon, F. 2023.  Individual Arbitrariness and Group Fairness.  In _Thirty-Seventh Conference on Neural Information Processing Systems_.
  * Marx, Calmon, and Ustun (2020) Marx, C.; Calmon, F.; and Ustun, B. 2020.  Predictive multiplicity in classification.  In _International Conference on Machine Learning_ , 6765–6774. PMLR.
  * McCoy, Min, and Linzen (2020) McCoy, R. T.; Min, J.; and Linzen, T. 2020.  BERTs of a Feather Do Not Generalize Together: Large Variability in Generalization across Models with Similar Test Set Performance.  arxiv:1911.02969.
  * Meyer, Albarghouthi, and D’Antoni (2023) Meyer, A. P.; Albarghouthi, A.; and D’Antoni, L. 2023.  The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , 193–204.
  * Milani Fard et al. (2016) Milani Fard, M.; Cormier, Q.; Canini, K.; and Gupta, M. 2016.  Launch and iterate: Reducing prediction churn.  _Advances in Neural Information Processing Systems_ , 29.
  * OECD Global Forum on Competition (2003) OECD Global Forum on Competition, O. G. F. o. C. 2003.  The Objectives of Competition Law and Policy.  _CCNM/GF/COMP(2003)3_.
  * Paes et al. (2023) Paes, L. M.; Cruz, R.; Calmon, F. P.; and Diaz, M. 2023.  On the Inevitability of the Rashomon Effect.  In _2023 IEEE International Symposium on Information Theory (ISIT)_ , 549–554. IEEE.
  * Ponomareva et al. (2023) Ponomareva, N.; Hazimeh, H.; Kurakin, A.; Xu, Z.; Denison, C.; McMahan, H. B.; Vassilvitskii, S.; Chien, S.; and Thakurta, A. G. 2023.  How to dp-fy ml: A practical guide to machine learning with differential privacy.  _Journal of Artificial Intelligence Research_ , 77: 1113–1201.
  * Rawal, Kamar, and Lakkaraju (2021) Rawal, K.; Kamar, E.; and Lakkaraju, H. 2021.  Algorithmic Recourse in the Wild: Understanding the Impact of Data and Model Shifts.  _arxiv_.
  * Scherer, King, and Mrkonich (2019) Scherer, M. U.; King, A. G.; and Mrkonich, M. J. 2019.  Applying Old Rules to New Tools: Employment Discrimination Law in the Age of Algorithms.  _S C L Rev_ , 71(2): 449–522.
  * Schrepel and Pentland (2023) Schrepel, T.; and Pentland, A. S. 2023.  Competition Between AI Foundation Models: Dynamics and Policy Recommendations.  _MIT Connection Science Working Paper_.
  * Sellam et al. (2021) Sellam, T.; Yadlowsky, S.; Tenney, I.; Wei, J.; Saphra, N.; D’Amour, A.; Linzen, T.; Bastings, J.; Turc, I. R.; Eisenstein, J.; et al. 2021.  The MultiBERTs: BERT Reproductions for Robustness Analysis.  In _International Conference on Learning Representations_.
  * Semenova, Rudin, and Parr (2022) Semenova, L.; Rudin, C.; and Parr, R. 2022.  On the existence of simpler machine learning models.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , 1827–1858.
  * Sokol et al. (2022) Sokol, K.; Kull, M.; Chan, J.; and Salim, F. D. 2022.  Fairness and Ethics Under Model Multiplicity in Machine Learning.  _arXiv preprint arXiv:2203.07139_.
  * Somepalli et al. (2022) Somepalli, G.; Fowl, L.; Bansal, A.; Yeh-Chiang, P.; Dar, Y.; Baraniuk, R.; Goldblum, M.; and Goldstein, T. 2022.  Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent From the Decision Boundary Perspective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13699–13708.
  * Tankha (2023) Tankha, A. 2023.  Employability Classification of Over 70,000 Job Applicants.  https://www.kaggle.com/datasets/ayushtankha/70k-job-applicants-data-human-resource.
  * Trindel, Kassir, and Bent (2021) Trindel, K.; Kassir, S.; and Bent, J. 2021.  Fairness in Algorithmic Employment Selection: How to Comply with Title VII.  _ABA J Lab & Emp L_, 35(2): 241–288.
  * Verma et al. (2022) Verma, S.; Boonsanong, V.; Hoang, M.; Hines, K. E.; Dickerson, J. P.; and Shah, C. 2022.  Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review.  arXiv:2010.10596.
  * Vipra and Korinek (2023) Vipra, J.; and Korinek, A. 2023.  Market Concentration Implications of Foundation Models: The Invisible Hand of ChatGPT.  _Center on Regulation and Markets at Brookings_ , Working Paper 9.
  * Wachter, Mittelstadt, and Russell (2018) Wachter, S.; Mittelstadt, B.; and Russell, C. 2018.  Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR.  _Harv JL & Tech_, 31(2): 841–888.
  * Wang and Russakovsky (2023) Wang, A.; and Russakovsky, O. 2023.  Overwriting pretrained bias with finetuning data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 3957–3968.
  * Watson-Daniels et al. (2023) Watson-Daniels, J.; Barocas, S.; Hofman, J. M.; and Chouldechova, A. 2023.  Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , 297–311.
  * Watson-Daniels et al. (2024) Watson-Daniels, J.; du Pin Calmon, F.; D’Amour, A.; Long, C.; C. Parkes, D.; and Ustun, B. 2024.  Predictive Churn with the Set of Good Models.  _arxiv_.
  * Wu et al. (2021) Wu, B.; Chen, J.; Cai, D.; He, X.; and Gu, Q. 2021.  Do wider neural networks really help adversarial robustness?  _Advances in Neural Information Processing Systems_ , 34: 7054–7067.
  * Yeom et al. (2018) Yeom, S.; Giacomelli, I.; Fredrikson, M.; and Jha, S. 2018.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In _2018 IEEE 31st computer security foundations symposium (CSF)_ , 268–282. IEEE.
  * Zech et al. (2018) Zech, J. R.; Badgeley, M. A.; Liu, M.; Costa, A. B.; Titano, J. J.; and Oermann, E. K. 2018.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.  _PLoS medicine_ , 15(11): e1002683.
  * Zhang, Song, and Qi (2017) Zhang, Z.; Song, Y.; and Qi, H. 2017.  Age progression/regression by conditional adversarial autoencoder.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5810–5818.

##  Appendix A Additional Details on Experiment Setup

###  A.1 Self Consistency

For measuring multiplicity in our paper, we use self-consistency (Cooper et
al. 2024). Given $b$ models in the Rashomon set, where the final prediction is
$0$ for $b_{0}$ models and $1$ for the rest $b_{1}=b-b_{0}$ models, self-
consistency is defined as $1-\frac{b_{0}b_{1}}{b(b-1)}$. This metric
represents the likelihood of choosing two models randomly from the Rashomon
set, and getting the same output.

###  A.2 Creating Rashomon Sets

To create the Rashomon sets, we change the underlying dataset, the model
hyperparameters and the random seeds. For simulating a change in the
underlying dataset, we simply choose a random subset of $12$ features out of
$18$ features in our dataset, and repeat this $100$ times with a different
subset of features each time. For simulating a change in the model
hyperparameters, we change the model’s batch size, the model’s learning rate,
and the optimizer (choosing between SGD and Adam). Finally, we set a random
seed to control both the stochasticity of model weight initialization and
training data shuffling and change this seed $500$ times. When making all
three changes simultaneously, we run exponentially many training runs by
randomly changing the parameters on all three axes as defined above, until we
found 20 models that satisfied the performance constraints.

###  A.3 Defining Robustness

We define robustness as the stability of model predictions under input noise,
more specifically noise of $\pm\$5000$ in the feature column PreviousSalary.
To binarize this metric, so that we can measure self-consistency, we define
robustness as follows. If the individual’s predictions do not change with an
increase (and a decrease) of the feature PreviousSalary by $\$5000$ then we
consider that individual robust, else we consider them not robust.

###  A.4 Defining Privacy

We define privacy as the prediction made by a membership inference attack
algorithm. We perform membership inference using a simple threshold-based
algorithm (Yeom et al. 2018). We choose the threshold for binarizing the
membership inference scores by utilizing our knowledge of the train test
split. This assignment thus becomes the membership inference prediction for a
data point, given a specific model, and we can then measure self-consistency
for privacy.

##  Appendix B Results on Additional Datasets

#### Datasets

We perform additional experiments on a tabular dataset (South German Credit
(Hofmann 1994)), an image dataset (UTKFace (Zhang, Song, and Qi 2017)), and a
text dataset (Jigsaw (cjadams et al. 2017)), to cover a diverse set of
distributions and modalities. South German Credit dataset (Hofmann 1994) is a
tabular dataset containing personal and financial information of $1000$ unique
individuals, with the binary classification task of predicting whether the
individual has good or bad credit. We use the protected attribute age for this
dataset, discretizing it into two binary age groups, $[19-59,59-75]$. UTKFace
dataset (Zhang, Song, and Qi 2017) is an image dataset containing over
$20,000$ face images with attributes like age, race, and perceived gender of
each individual in the image. We use the perceived gender label to create a
binary classification task and the race label as the protected attribute.
Jigsaw toxicity classification dataset (cjadams et al. 2017) is a text dataset
that contains Wikipedia comments labelled by human annotators for various
forms of toxicity, converted to a binary text classification task by
discretizing the overall toxicity score. Given the significantly large size of
the dataset, we use the test split of the dataset, still containing close to
$100k$ Wikipedia comments. We use the labels associated with religion, sexual
orientation, gender identities and race as the protected attributes. All
datasets are split $80-20$ into train and test sets.

#### Models and Learning Setup

We use a feed-forward network with a single hidden layer containing 10 neurons
and ReLU for the South German Credit dataset and train $1000$ models with
changing random seeds to capture the multiplicity. Similarly, we use ResNet18,
trained from scratch, for the UTKFace dataset, and train $50$ models with
changing random seeds. Finally, we use a set of $25$ publicly available BERT
models (Sellam et al. 2021) that were all trained with the same setup but
different random seeds and learn logistic regression on the embeddings for the
Jigsaw dataset.

###  B.1 New Rashomon Sets

To emphasize the broader definition of Rashomon sets, we perform a case study
on the South German Credit dataset. In this study, we train 1000 models on the
South German Credit dataset with changing random seeds. We then select models
based on two criteria: (i) based solely on the accuracy with a $2\%$
threshold, and (ii) considering both accuracy and group fairness (accuracy
disparity across different age groups) again with a $2\%$ threshold. The first
criterion led to the selection of 121 models, while the second criterion
resulted in the selection of only 6 models, out of the original $1000$. This
setup aims to simulate real-world model selection, where a user might want to
select the best performing model. For both scenarios, we record the self-
consistency scores of predictive multiplicity across all 200 individuals from
the test set: results are collected in Figure 2(a).

$0.4$$0.5$$0.6$$0.7$$0.8$$0.9$$1$$0$$50$$100$$150$$200$ Self-Consistency (CDF)
# of Individuals (Test Set) g(a) Model Selection with Accuracy vs Both
Accuracy and Fairness

$0.4$$0.5$$0.6$$0.7$$0.8$$0.9$$1$$0$$200$$400$$600$$800$ Self-Consistency
(CDF) # of Individuals (Train Set) g(b) Predictive Multiplicity vs
Multiplicity of Membership Inference

Performance Metric:AccuracyPerformance Metric:Accuracy + FairnessMultiplicity
Metric:Model PredictionsMultiplicity Metric:Membership Inference

Figure 2: (a) Distribution of self-consistency scores for individuals in the
South German Credit test set (Hofmann 1994), across models selected using only
accuracy vs both accuracy and fairness. The impact of multiplicity is more
pronounced in the latter case, which underscores the significance of the
performance metrics. (b) Distribution of self-consistency scores for
individuals in the South German Credit train set (Hofmann 1994) under a fixed
model selection criteria (both accuracy and fairness) but measured on either
model predictions, i.e., predictive multiplicity, or membership inference
predictions. Although not as pronounced as before, distinct trends persist
between the two settings, reiterating the importance of the multiplicity
metric in model multiplicity.

Remarkably, despite being a more refined model selection criterion and with
fewer models in the Rashomon set, the second set of constraints results in
lower self-consistency scores for a greater number of individuals. That is,
despite _similar performance_ in terms of accuracy and group fairness, we find
that the final prediction for many individuals still relies heavily on the
arbitrariness of model selection among the $6$ final models. We highlight the
shift in multiplicity trends under various choices of performance constraints,
underscoring the reality of model selection that would have been missed if the
Rashomon sets were solely examined through the lens of similar accuracy.

We repeat the same case study with a fixed model selection criteria (i.e.,
both accuracy and group fairness); and our focus instead shifts towards
measuring multiplicity using two distinct effects of multiplicity: (i) model
predictions and (ii) membership inference. We perform membership inference
using a simple threshold-based algorithm (Yeom et al. 2018). Among the $1000$
individuals in our dataset, we predict the top $800$ individuals with the
highest output logits for each model as members, designating the remaining
$200$ as non-members (utilizing our knowledge of the train-test split). This
assignment thus becomes the membership inference prediction for a data point,
given a specific model. The subsequent measurements then closely mirror model
predictions. This time, we gather the self-consistence scores of both
scenarios for individuals in the train set: the results are collected in
Figure 2(b).

In line with our earlier discussion, the trends in self-consistency between
these two distinct multiplicity metrics are noticeably different. The self-
consistency scores for membership inference are lower, which suggests that the
arbitrary selection of a model has a more pronounced impact on an individual’s
privacy than their predictions. These case studies together showcase the
significance of formalizing a broader definition of model multiplicity.

$0.6$$0.7$$0.8$$0.9$$0$%$10$%$20$% Self-Consistency (CDF) % of Data Points
g(a) Cumulative Distribution of Multiplicity

WhiteBlackAsianIndian$0$$0.1$$0.2$$0.3$ Race g Ambiguity (b) Multiplicity
Under Fairness Constraintsg

Race: WhiteRace: BlackRace: AsianRace: IndianNo ConstraintsThreshold
Optimization

Figure 3: Model multiplicity across four different racial groups on the
UTKFace dataset (Zhang, Song, and Qi 2017). (a) We find a clear disparity in
multiplicity across different groups, where the under-represented group
contains a significantly higher percentage of individuals with low self-
consistency scores than other majority groups. (b) Not only do these trends
generalize to other measures of multiplicity, we even find an increase in
multiplicity scores under fairness constraints (here, threshold optimization
to achieve accuracy parity).

Budd.Christ.HinduJew.Muslim$0$$0.2$$0.4$$0.6$$0.8$Religion Ambiguity

Bi.Hetero.Homo.Sexual Orientation g

MaleFemaleTrans.Gender Identity g

AsianBlackLatinoWhiteRace g

Figure 4: Model multiplicity (measured as ambiguity) across various protected
groups on the Jigsaw toxicity classification dataset (cjadams et al. 2017) The
multiplicity disparity between groups is dramatically high in this setting,
emphasizing the previously seen trends.
(Legend) Budd: Buddhist; Christ: Christian; Jew: Jewish; Bi:Bisexual;
Hetero:Heterosexual; Homo:Homosexual; Trans:Transgender.

###  B.2 Disparate Arbitrariness across Demographics

We begin by studying the predictive multiplicity across various racial groups
in the UTKFace dataset. For fairness constraints, we employ threshold
optimization to improve accuracy parity (Hardt, Price, and Srebro 2016), while
using a single threshold for all groups. The results are presented in Figure
3. In Figure 3(a), a substantial portion of the test data exhibits self-
consistency scores below 1, indicating that predictions for these individuals
(unfortunately) depend on the arbitrariness of model selection. Notably, this
trend is more pronounced for certain racial groups. The Asian racial group, in
particular, has twice as many fractions of individuals with lower self-
consistency scores compared to any other racial group, highlighting a
significant disparity in multiplicity. At first glance, one might attribute
this disparity to biases in the model, expecting it to diminish under fairness
constraints. However, Figure 3(b) shows that the introduction of threshold
optimizations in this setting instead amplifies the arbitrariness for all
groups. This result aligns with observations of higher multiplicity under
fairness constraints by Long et al. (2023), emphasizing that the disparity in
multiplicity can exist in the absence of a disparity in performance.

We extend our experiments to the Jigsaw toxicity classification dataset and
record the multiplicity across various protected groups in Figure 4. We see
similar trends of high disparity in multiplicity across groups. For instance,
text comments referencing Christianity exhibit an ambiguity of only $\sim 0.3$
in toxicity classification—indicating that only $30\%$ of the data is affected
by the arbitrariness of model selection. Conversely, comments referencing
Muslims show a higher ambiguity of $\sim 0.68$ in toxicity classification,
i.e., $68\%$ of the data from this group is susceptible to the arbitrariness
of model selection. Thus, predictions on the toxicity of a text can undergo
significant changes for certain demographics simply because of an arbitrary
change in the model’s random seed during training, while the predictions for
the other demographic groups would remain mostly consistent.

We conclude our empirical study by underscoring the disparity of multiplicity
across various protected groups when examining the arbitrary nature of model
selection in machine learning. We even saw this vulnerability increase with
explicit fairness constraints, despite the reduction in bias scores for
existing fairness metrics.