# Emergent Unfairness in Algorithmic Fairness-Accuracy Trade-Off Research

A. Feder Cooper<EMAIL_ADDRESS>Cornell UniversityDepartment of Computer
ScienceIthacaNYUSA14850 and  Ellen Abrams<EMAIL_ADDRESS>Cornell
UniversitySociety for the HumanitiesIthacaNYUSA14850

(2021)

###### Abstract.

Across machine learning (ML) sub-disciplines, researchers make explicit
mathematical assumptions in order to facilitate proof-writing. We note that,
specifically in the area of fairness-accuracy trade-off optimization
scholarship, similar attention is not paid to the _normative assumptions_ that
ground this approach. Such assumptions presume that 1) accuracy and fairness
are in inherent opposition to one another, 2) strict notions of mathematical
equality can adequately model fairness, 3) it is possible to measure the
accuracy and fairness of decisions independent from historical context, and 4)
collecting more data on marginalized individuals is a reasonable solution to
mitigate the effects of the trade-off. We argue that such assumptions, which
are often left implicit and unexamined, lead to inconsistent conclusions:
While the intended goal of this work may be to improve the fairness of machine
learning models, these unexamined, implicit assumptions can in fact result in
emergent unfairness. We conclude by suggesting a concrete path forward toward
a potential resolution.

algorithmic fairness, machine learning, societal implications of AI

††journalyear: 2021††copyright: acmlicensed††conference: Proceedings of the
2021AAAI/ACM Conference on AI, Ethics, and Society; May 19–21, 2021;
VirtualEvent, USA††booktitle: Proceedings of the 2021 AAAI/ACM Conference on
AI, Ethics,and Society (AIES ’21), May 19–21, 2021, Virtual Event, USA††price:
15.00††doi: 10.1145/3461702.3462519††isbn: 978-1-4503-8473-5/21/05††ccs:
Social and professional topics Computing / technology policy††ccs: Applied
computing Computers in other domains

##  1\. Introduction

Optimization is a problem formulation technique that lies at the core of
multiple engineering domains. Given some fixed or limited resource, we can
model its usage to effectively solve a problem. The optimal solution is the
one that either minimizes some cost function or maximizes some utility
function—functions that measure how well the model performs on a particular
objective. Often there is more than one objective to satisfy simultaneously,
and those objectives can be in tension with one another. In this case, it is
possible to pose this problem as optimizing a trade-off (Yang, 2010).

For an intuitive example, consider a company that has a fixed amount of steel,
which it can use to build cars and planes, which it then sells to earn a
profit. The company has to decide how to allocate the steel to maximize that
profit and can formulate the decision as an optimization problem. The blue
curve in Figure 1 models possible ways to do this optimally; picking a
specific point on the curve corresponds to the company’s choice for how to
balance the trade-off between how many cars and how many planes to produce.

Figure 1. Illustrating a trade-off. Given a fixed amount of steel, a company
can build a combination of cars and planes. To optimally utilize the steel for
maximizing profit, it can manufacture any point on the blue curve. Any
combination above the curve is not possible because there is insufficient
steel; any below is suboptimal because the same amount of steel could either
produce more cars or more planes.

Optimization science has informed much of the last decade’s spectacular spate
of statistical machine learning (ML) publications. It is at the core of how
many ML algorithms learn. For example, learned classifiers use training data
examples to fit a curve that optimizes for both classifying those examples
correctly and generalizing to new, unclassified examples (Bishop, 1995; Hastie
et al., 2009). This process of automating classification decisions has in the
past been framed as optimal in another sense; automation brought with it the
hope of rooting out the human (suboptimal) whims of decisionmaking—of
eliminating the ugliest human biases, such as sexism and racism, that afflict
high-impact decision processes. Yet, as has been well-documented, this hope
cannot be passively realized merely by substituting humans with automated
decision agents. Issues with biased data and biased model selection processes
can in the worst case magnify, rather than replace, human biases (Abebe et
al., 2020; Forde et al., 2021; Selbst et al., 2019).

In response, there has been a widespread push to actively engineer
_algorithmic fairness_. This has become remarkably urgent as automated
decision systems are being deployed in domains that have a significant impact
on the quality of human life—in deciding judicial bail conditions, loan
allocation, college admissions, hiring outcomes, COVID vaccine distribution,
etc., (Monahan and Skeem, 2016; Barocas and Selbst, 2014; Ajunwa, 2020; Angwin
et al., 2016). Models no longer just need to be correct; they also need to be
fair.

It has become common to position accuracy in opposition to fairness and to
formalize a mathematical trade-off between the two. For example, in the
context of criminal justice and bail decisions, the accuracy of decisions has
been framed as how to best “maximize public safety,” in contrast to satisfying
“formal fairness constraints” that aim to reduce racial disparities in
decision outcomes (Corbett-Davies et al., 2017). This kind of problem
formulation is the norm in a growing area of research, in which the trade-off
between fairness and accuracy in ML is considered “inherent” or “unavoidable.”
Prior work suggests various ways of implementing the trade-off: At best, under
particular conditions, the tension between the two can be dissolved to achieve
both; at worst, fairness is sacrificed in favor of accuracy, while the
remaining cases fall somewhere in the middle (Dutta et al., 2020; Chen et al.,
2018; Bakker et al., 2019; Menon and Williamson, 2018; Noriega-Campero et al.,
2019).

###  1.1. Our Contribution

Our work looks both at and beyond the fairness-accuracy trade-off, drawing
from prior critiques of algorithmic fairness and studies of sociotechnical
systems. We examine the _choice_ —and our work here will show that it is a
choice, not a requirement—to model assumptions that cast fairness in direct
opposition to accuracy. Regardless of the particulars of specific
implementations, this framing does not just involve math, but also implicates
normative concerns regarding how to value fairness and accuracy both
independently and in relation to each other (Flanagan and Nissenbaum, 2014;
Friedman and Hendry, 2019).

Our contribution is to extract and explore patterns of these concerns across
trade-off scholarship that arise at three different stages: the initial
modeling assumption to treat accuracy and fairness together in an optimization
problem, the move from abstract framing to concrete problem formulation, and
the “optimal solutions” that result from those formulations. More
specifically, we examine how the choice to operationalize the relationship
between fairness and accuracy using the language of optimization inherently
puts the two in conflict, rather than leaving open the legitimate possibility
for them to actually be in accord. We discuss how this choice fails to take
full account of social criteria when drawing the boundaries of a problem
(Selbst et al., 2019), and relates to broader trends of techno-“solutionism,”
in which math is (mistakenly) bestowed special authority to “solve” social
problems (Abebe et al., 2020).

Beyond these overarching framing assumptions, there are other underlying,
unexamined normative assumptions (not just explicit mathematical ones) that
take root in how the trade-off is formalized: That strict notions of
mathematical equality can model fairness, that it is possible to measure the
accuracy and fairness of decisions independent from historical context, and
that collecting more data on marginalized individuals—a practice called
_active fairness_ —is a reasonable solution to mitigate the effects of the
trade-off.

If we take the time to clarify these implicit assumptions, we note that the
conclusions that follow can actually perpetuate unfairness: The mathematical
proofs may be sound—a particular choice of fairness metric may even be
optimized—but the implicit normative assumptions and accompanying broader
normative results suggest that these methods will not ensure fairer outcomes
in practical applications. In summary, we argue that

  * •

Using the language of optimization situates fairness and accuracy in intrinsic
opposition, generally privileging the latter over the former and necessarily
foreclosing the possibility of examining the ways they can instead reinforce
each other (Section 3).

  * •

Underlying mathematical assumptions bring unexamined normative dimensions,
which can actually result in emergent unfairness (Section 4).

  * •

In light of these observations, algorithmic fairness researchers will conduct
more robust research if they first clarify their normative assumptions
(Section 5).

  * •

Due to the extent of emergent unfairness from conceiving of fairness and
accuracy in trade-off, it is worth revisiting this formulation altogether
(Section 6).

##  2\. The Fairness-Accuracy Trade-Off

We begin by providing the background necessary for understanding the problem
formulation of the fairness-accuracy trade-off. Before clarifying what the
trade-off actually characterizes, we address each component in turn,
summarizing common quantifiable metrics that ML researchers map to the values
of “accuracy” and “fairness.”

###  2.1. Accuracy Metrics

In brief, _accuracy_ measures how often a ML model correctly predicts or
infers a decision outcome after training. So, to understand accuracy, we need
to understand how ML models are trained. For the classification problems that
dominate much of the fairness literature, training the model usually entails
fitting a curve to a set of training data points for which classification
labels are already known. After training, when supplied with a previously-
unseen data point, the model can infer the classification label for that data
point. For example, in building a model that infers whether or not to grant an
applicant a loan, the curve-fitting training process occurs with past data
concerning loan-granting decisions; inference corresponds to the model
receiving a new loan applicant’s data and classifying whether that applicant
should receive a loan or not—a decision ultimately corresponding to whether or
not the loan-granting institution believes the applicant will repay or default
on the loan.

A model’s accuracy tends to be measured during a validation process in between
training and inference. Rather than using all labeled data for training,
researchers reserve a portion to validate how well the trained model
classifies unseen data points with known classification labels. In other
words, accuracy is often a measure of _label alignment_ : It is the percentage
of correctly classified validation data points, where correctness is
determined by whether the model’s classification decision matches the known
label. There are other metrics that researchers use, such as Chernoff
information (Dutta et al., 2020); however, label alignment is a popular
accuracy metric, in part due to its simplicity.

This simplicity can be misleading, both in terms of what the math is actually
measuring and the normative implications of that measurement. The broader
algorithmic fairness community (and corresponding community of critics) has
paid ample attention to this issue in relation to fairness (Binns, 2018;
Selbst et al., 2019; Powles and Nissenbaum, 2018; Abdurahman, 2019); however,
in fairness-accuracy trade-off literature, where accuracy is also explicitly
centered as a value, parallel analyses of accuracy have been relatively
sparse. We examine this in Section 4; for now, we emphasize that something as
simple as a percentage metric can raise normative concerns (Forde et al.,
2021).

One can see this from work in the broader ML community, in which accuracy
issues often get cast as a problem of _label bias_ : The classification labels
in the training and validation data can be incorrect, in terms of some
abstract notion of “ground truth.”111The implications of “true”
classification, including the simplifying assumptions that inform such
classification, are out of scope for our purposes. We refer the reader to the
rich literatures in sociology and science & technology studies on
categorization and classification, notably (Velocci, 2021; Bowker and Star,
1999). As an innocuous example, consider a labeled image dataset of dogs and
cats. The individual that labeled the dataset incorrectly (though perhaps
understandably) mis-labeled Pomeranians as cats. This mis-labeling in turn
leads the learned ML model to mistakenly identify Pomeranians as cats.

The results of mis-labeling can be devastating for applications that impact
human lives. Consider again an automated decision system that grants and
denies loan applications. In the US, systemic racism against Black loan
applicants, specifically manifested in the practice of redlining, has entailed
denying loans to qualified Black applicants. These applicants, in terms of
“ground truth” should have been granted loans, but were instead intentionally
marked as likely defaulters. As with the example above, this mis-labeling
would lead to inaccurate classification. The models trained on that data would
mistakenly identify Black non-defaulters as defaulters—a mis-classification
that could be used to wrongfully deny a loan.222We have simplified this
example, which also involves using zip code as a proxy for race, to make our
point that mis-labeled data impacts the accuracy of trained models. Unlike the
former example, this one involves intentional mis-labeling; however, it is
worth noting that even if the mis-labeling were unintentional, it would have
the same effect on classification decisions. Regardless of intention, the
impact would be the same. Either way, the learned model would systematically,
incorrectly classify Black individuals as defaulters.

###  2.2. Fairness Metrics

Algorithmic _fairness_ has dozens of mathematical definitions that can inform
optimization problem formulation.333Similar to our discussion of accuracy and
classification, our work will not focus on the normative limits of defining
fairness mathematically, such as the challenges of formulating fairness
problems that account for intersectional protected identities (Selbst et al.,
2019; Chen et al., 2018; Dwork et al., 2018; Buolamwini and Gebru, 2018; Kroll
et al., 2017; Hoffmann, 2019) and how many fairness definitions reflect _what
we ought to believe_ instead of (the arguably more useful, from a policy
perspective) _what we ought to do_ (Hellman, 2020b). All definitions,
regardless of the specifics, involve some treatment of _protected attributes_
, such as race and sex, along which decision outcomes can be evaluated for
“fair” treatment. Broadly speaking, there are two families of fairness
metrics: Those that measure individual-focused fairness and those that
evaluate it in terms of groups defined by protected attributes.

Individual fairness, as the name suggests, centers analyzing automated
decisions in terms of the individual (Dwork et al., 2012; Joseph et al., 2016;
Bakker et al., 2019). In contrast, group fairness centers demographic group
membership and typically aims to ensure that membership in a protected class
does not correlate with decision outcomes (Dwork et al., 2018; Chen et al.,
2018). A particularly popular metric is Hardt et al. (2016)’s formulation of
_equality of opportunity_ , which in essence only requires that there is no
discrimination based on demographics for those assigned the positive
classification. For the example of granting loans, paying back the loan is the
positive class and defaulting is the negative class. Equality of opportunity
corresponds to making sure that the rate of mistakenly classifying non-
defaulters as defaulters is similar across demographic groups. We discuss this
further in Section 4; for now, it is important to note that _this formulation
is inextricably tied to accuracy_. Training a model to reduce mistaken
negative classification directly depends on training data that contains
negative labels. Those labels, however, may not align with what is correct in
terms of “ground truth”—e.g., Black individuals erroneously marked as
defaulters.

Lastly, it is also worth noting that multiple different mathematical notions
of fairness cannot be satisfied simultaneously. This incompatibility has been
formalized as impossibility results (Kleinberg et al., 2017; Chouldechova,
2017) and has placed a more significant emphasis on how computer scientists
choose which fairness metric to study (Friedler et al., 2016). While these
findings are extremely well-cited, they are not surprising when considering
fairness beyond its mathematical definition as a metric. In a pluralistic
world, values like fairness depend on the time and place in which they are
defined; outside of math, different, incompatible definitions can hold
simultaneously, depending on context (Berlin, 2013).

###  2.3. The “Inherent” Trade-Off

Algorithmic fairness tends to pose accuracy and fairness in an “inherent” or
“unavoidable” trade-off: An increase in fairness necessarily comes with a
decrease in accuracy; increasing accuracy necessarily decreases fairness (Chen
et al., 2018; Menon and Williamson, 2018; Bakker et al., 2019; Corbett-Davies
et al., 2017; Dwork et al., 2018; Sabato and Yom-Tov, 2020; Zhao and Gordon,
2019).

How did this trade-off problem formulation come about? Much of the literature
that engages this trade-off does so empirically; that is, the authors perform
or cite experiments that convey the intuition that a trade-off exists, so they
decide that a mathematical trade-off is an appropriate way to model the
problem. However, most do not characterize or quantify the trade-off
theoretically. The work that has attempted theoretical treatment suggests
that—at least in theory—the existence of the fairness-accuracy trade-off can
be less rigidly described (Dutta et al., 2020; Wick et al., 2019).
Nevertheless, the general practice in the field is to tacitly accept the
trade-off as fact, regardless of the particular fairness and accuracy metrics
under consideration.

Computer scientists further observe that the ramifications of this trade-off,
particularly in high-stakes domains, can be significant. As a result, they
sometimes wade into the murkiness of _how_ to optimize the trade-off
implementation in specific “sensitive” applications. For example, several
computer scientists have noted that in areas like healthcare, trade-off
implementations should favor accuracy, as privileging fairness can have
“devastating” consequences, such as missing cancer diagnoses at higher rates
(Chen et al., 2018; Srivastava et al., 2019).

Other researchers have posited _why_ this trade-off exists in the first place.
For example, one popular explanation comes from Friedler et al. (2016). The
authors of this paper reason that there is an “abstract construct space” that
represents the features we actually want to measure but cannot observe (e.g.,
intelligence). Instead, we see features in the “observed space” of the actual
world (e.g., SAT score), and there is a mapping from features in the construct
space to features in the observed space (e.g., SAT score is the mapped feature
in the observed space, standing in place for intelligence in the construct
space). According to Friedler et al. (2016), the trade-off between
classification accuracy and fairness exists in the real world due to “noisier
mappings” for less privileged groups from the construct space to the observed
space. They contend that this noise comes from historic differences,
particularly in opportunity and representation, which makes positive and
negative data points less distinguishable (in comparison to privileged groups)
for the learned classifier. It is this decrease in separability that leads to
less fair classification for less privileged groups. In the example of SATs,
this means that the scores are less reliable in terms of conveying information
about intelligence for underprivileged groups. While this posited explanation
may seem reasonable, work that engages with it rarely (if ever) supports the
explanation with data, as it is usually not the specific fairness problem
under mathematical consideration (Dutta et al., 2020).

##  3\. Applying a Sociotechnical Lens

In the field of Science and Technology Studies, the term “sociotechnical” is
used to signal the impossibility of understanding the social world separately
from the technological, and vice versa (Bijker et al., 1987; Hughes, 1993;
Hecht, 2002; Wajcman, 2004). This means that, among other things, technologies
like algorithms are never separate from the systems, institutions, ideologies,
cultures, and societies in which they are embedded. It also means that the
choice to formulate algorithmic fairness as an optimization problem produces a
particular kind of knowledge about fairness that cannot be detached from its
broader social context. In our critiques of emergent unfairness in Section 4,
we explicitly examine the fairness-accuracy trade-off model as a
sociotechnical system.

We are not the first to examine algorithmic fairness using the concept of a
sociotechnical system. For example, Selbst et al. (2019) use the concept of
sociotechnical systems to draw attention to five pitfalls that imperil well-
meaning data scientific approaches to fairness. One that they identify is
called the “Framing Trap,” which the authors describe as the “failure to model
the entire system over which a social criterion, such as fairness, will be
enforced.” A sociotechnical lens, they argue, might suggest new ways for
researchers to draw the boundaries of fairness problems to include social
relations and dynamics that may have otherwise been excluded.

Defining fairness and accuracy in trade-off both exemplifies falling into the
Framing Trap and presents an additional set of nuanced consequences. For
example, the language of defining trade-offs along a curve (Figure 1)
necessarily requires a give-and-take relationship between the factors under
consideration, as those factors are cast in opposition to one another. In the
fairness-accuracy trade-off, fairness and accuracy are framed as inherently
competing goals, in which we must give up some of one in order to gain some of
the other (even if “some” cannot be quantified definitively). Based on this
framing, it is consistent that much of the literature uses language that
describes the “cost of fairness” (Chen et al., 2018; Menon and Williamson,
2018; Corbett-Davies et al., 2017; Dutta et al., 2020), depending on where on
the trade-off optimization curve a particular implementation lies. This cost,
however, can be described as cutting both ways: It is similarly reasonable to
talk about the “cost of accuracy.” Yet, with few exceptions (Corbett-Davies et
al., 2017), the literature in this area chooses not to discuss costs in this
way. This decision is perhaps due to the tendency within the field of ML more
broadly to privilege accuracy during model design; nonetheless, this framing
shifts the burden of defensibility to fairness, in the sense that it implies
that fairness’ “costs” require justification.

The particulars of the trade-off choice present additional complications.
Beyond failing “to model the entire system”, the fairness-accuracy trade-off
formulation also forecloses the very reasonable possibility that accuracy is
generally in accord with fairness (unless one specifies particular conditions
for which it is possible to demonstrate that the trade-off does not exist
(Dutta et al., 2020; Wick et al., 2019)). In other words, a trade-off model
conceals the idea that the accurate thing to do could be complementary with
the fair thing to do. For example, as Hellman (2020b) notes, it is possible to
view accuracy and fairness as complementary values, where the former reflects
what one “ought to believe” and the latter reflects what one “ought to do.” In
Section 4, we expand on how fairness and accuracy can be considered in accord
(Friedman and Hendry, 2019; Flanagan et al., 2008). Additionally, we suggest
that the boundaries of the trade-off problem need to be redrawn to account for
social and technical considerations _in historical context_. A suitable frame
for fairness research can neither be blind to past historical context nor
ignore the future.

Lastly, our analysis demonstrates some of the challenges of using trade-off
and optimization tools in algorithmic fairness research. Borrowing tools from
adjacent fields in computer science not only affects the results of fairness
research, but it also helps to characterize the disciplinary status of
fairness research itself. In this case, the operationalization of fairness as
a mathematical problem helps situate questions of fairness within the realm of
“science,” thus conferring a particular legitimacy that science connotes
(Gieryn, 1983; Porter, 1995). Situating the fairness question as a scientific
question and esconcing it in the language of trade-offs and optimization
suggests that it is reasonable to try to solve for an “optimal,” best answer.
Framing the problem as a trade-off problem to be “solved” using math falls
under the highly-critiqued practice of technological “solutionism” (Selbst et
al., 2019; Abebe et al., 2020)—the notion that technology is uniquely capable
of solving social problems. This tendency toward solutionism grants special
legitimacy to algorithmic fairness research, legitimacy absent in other fields
that have tackled but have not “solved” the fairness problem from a social
perspective. We contend that it is unlikely that the same ideas used to solve
problems of steel allocation (Figure 1) will transfer without issue to
questions of fair hiring practices. We suggest that attending to the
sociotechnical context of each situation may help prevent the emergent
unfairness we identify in the following sections.

##  4\. Emergent Unfairness

Choosing to convey fairness and accuracy as a trade-off is a mathematical
modeling assumption. As we discussed in Section 2, the authors have observed a
pattern in their empirical results concerning accuracy and fairness, and deem
a trade-off to be a useful way to formulate the mathematical problem of
characterizing the relationship between the two. As we suggest above, fairness
and accuracy metrics have normative dimensions; so, too, does the modeling
assumption that poses them in trade-off.

There are also numerous, other mathematical assumptions, each which carry
their own implicit normative dimensions. We observe that, based on these
implicit assumptions, fairness-accuracy trade-off scholarship is plagued with
gaps and oversights. These issues can lead to conclusions that _actually
perpetuate unfairness_. There are dozens of examples of particular assumptions
specific to each paper in fairness-accuracy trade-off scholarship. It is not
possible to be exhaustive regarding each mathematical assumption’s
corresponding normative assumptions. Instead, we isolate three patterns of
implicit, unexamined assumptions in the discipline, and the emergent _un_
fairness that can result: Unfairness from assuming 1) strict notions of
equality can substitute for fairness, 2) historical context is irrelevant when
formulating the trade-off, and 3) that collecting more data on marginalized
groups is a reasonable mechanism for alleviating the trade-off.

###  4.1. Unfairness from Assuming Fairness = Equality

One assumption prevalent in fairness-accuracy trade-off literature concerns
how different papers choose to measure fairness. Most of the work in this
subfield relies on parity-based definitions. Algorithmic fairness definitions
like this effectively make the modeling assumption to represent “fairness” as
“equality.”444While such equality-based notions of fairness dominate the
literature more generally, not just concerning the fairness-accuracy trade-
off, there are a growing number of exceptions. For example, some recent work
frames fairness in terms of Rawlsian social welfare (Rawls, 1971; Heidari et
al., 2018; Joseph et al., 2016). To be clear, we mean “equality” in the strict
sense of the values of metrics being as equal as possible, by minimizing some
form of measured inequality. This kind of strict equality can stand in for
“fairness” in terms of what is actually being modeled; fairness is being
framed as a problem of strict equality. This is easily discernible in the
popular equality of opportunity metric (Section 2), used in numerous trade-off
papers (Chen et al., 2018; Dutta et al., 2020; Bakker et al., 2019; Noriega-
Campero et al., 2019), which tries to minimize discrepancies in false negative
classification decisions among different demographic groups; it literally
tries to make those rates as equal as possible.

However, what is fair and what is equal are not always the same thing, and
framing them as equivalent can actually lead to unfair outcomes (Kasy and
Abebe, 2021). For example, when addressing historic or systemic inequity, it
can be necessary to take corrective or reparative action for some demographic
groups in order to create the conditions of more-equal footing. Such actions
necessarily diverge from equality in the strict mathematical sense, so
strictly parity-based fairness metrics cannot capture this kind of nuance.

The ongoing debate in the United States around the fairness of affirmative
action policy can help illustrate this distinction, as well as the
complications that arise when defining fairness as equality. In brief,
affirmative action is a social policy aimed at increasing the representation
of historically marginalized groups in university student and workforce
populations; it attempts to implement a fairer playing field by providing
individuals from marginalized backgrounds with the chance to have the same
opportunities as those from more privileged backgrounds.

While affirmative action has existed as official policy in the US for decades
(Kennedy, 1961), it is extremely contentious. Many Americans, who do not feel
personally responsible for systemic discrimination against BIPOC555An acronym
for “Black, Indigenous, and People of Color,” used particularly in the US and
Canada. populations, feel that affirmative action puts them at a disadvantage.
They claim that the policy is unfair, and in fact is responsible for “reverse
discrimination” (Budryk, 2020; Newkirk, 2017; Pham, 2018). This belief comes
in part from the idea that affirmative action does not lead to “equal”
comparisons in the strictest sense—comparing SAT scores or GPAs point for
point. Instead, in the language of Friedler et al. (2016), one could say that
affirmative action attempts to repair or normalize for the “noisy mappings”
that these scores convey for unprivileged populations in order to promote
fairer outcomes.666It is also interesting to note that this controversy has
found its way into the language of algorithmic fairness literature. Dwork et
al. (2012); Dwork et al. (2018) use the term “fair affirmative action” in
their work; they seem to be attempting to distinguish their notion from some
imagined, alternative, unfair variant. This term is arguably redundant, since
affirmative action is fundamentally about trying to promote fairer outcomes,
even if that notion of fairness does not align with strict-equality-based
notions in algorithmic fairness.

In short, the goal of affirmative action illustrates how notions of fairness
and equality can diverge. The policy’s existence is predicated on the notion
that strictly equal treatment, without attending to past inequity, can
potentially perpetuate unfairness.

###  4.2. Unfairness from Assuming the Irrelevance of Context

Fundamentally, the issue with the assumption that strict notions of equality
can stand in for fairness has to do with how the assumption treats—or rather
discounts—context. Fairness metrics like equality of opportunity are only able
to evaluate the local, immediate decision under consideration. As discussed
above using the example of affirmative action, this type of equality cannot
accommodate reparative interventions that attempt to correct for past
inequity. This similarly implicates issues with how we measure accuracy, since
such metrics measure the correctness of current classification decisions in
relation to past ones. We next examine this issue, as it presents fundamental
contradictions in the formulation of the fairness-accuracy trade-off problem.

####  4.2.1. Ignoring the Past

As discussed in Section 1, optimization involves minimizing a loss function.
In statistical terminology, minimizing the expected loss depends on the _true_
class labels (Bishop, 1995). In fairness-related application domains we
rarely, if ever, have access to true class labels. To return to an earlier
example, Black people have systematically been denied loans in the US due to
their race. In many cases, while a Black person’s “true” label should be that
they would not default on a loan, past loan-granting decisions (and therefore
the corresponding data) mark them as a defaulter. This captures the problem of
_label bias_ : Misalignment between the “ground truth” label and the actual,
observed label in real world data (Section 2). In a sense, this bias is what
has motivated the entire field of algorithmic fairness in the first place:
Automated decision systems that do not account for systemic discrimination in
training data end up magnifying that discrimination (Barocas et al., 2018;
Abebe et al., 2020); to avoid this, such systems need to be proactive about
being fair.

Label bias presents an inherent issue with how we measure accuracy: If labels
are wrong, particularly for individuals in the groups for which we want to
increase fairer outcomes, then there are cases where misclassification is in
fact the correct thing to do. In other words, how we measure accuracy is not
truly accurate.

Yet, in the fairness-accuracy trade-off literature, it is very common to
assume label bias can be ignored. Much of the work in this space does not
mention label bias at all, or claims that it is out of scope for the research
problem under consideration (Chen et al., 2018). This presents a
contradiction: Simultaneously acknowledging that labels in the observed space
are noisy representations of the ground truth (i.e., there is bias in the
labels), but then explicitly assuming those labels in the training data (i.e.,
the observed space labels) are the same as the true labels (Dutta et al.,
2020).777Wick et al. (2019) is a notable exception, acknowledging this
contradiction in stark terms: “…there is a pernicious modeling-evaluating
dualism bedeviling fair machine learning in which phenomena such as label bias
are appropriately acknowledged as a source of unfairness when designing fair
models, only to be tacitly abandoned when evaluating them.” In other words,
because the labels are biased, the corresponding accuracy measurements that
depend on them are also biased; unfairness from the past produces inaccuracy
(Hellman, 2020a), which this work explicitly ignores in its trade-off
formulation.

If accuracy measurements are conditioned on past unfairness, what is the
trade-off between fairness and accuracy actually measuring? What does it mean
to “increase” or “decrease accuracy” in this context? If accuracy measurements
encode past unfairness for unprivileged groups, the fairness-accuracy trade-
off is effectively positioning fairness in trade-off with unfairness, which is
tautological. Giving validity to an accuracy metric that has a dependency on
past unfairness inherently advantages privileged groups; it is aligned with
maintaining the status quo, as there is no way to splice out the past
unfairness on which it is conditioned. In the words of Hellman (2020b), this
can lead to even more unfair outcomes via “compounding injustice.” To the best
of our knowledge, no prior work has explicitly attempted to avoid this
scenario—to model and extract this past unfairness.

####  4.2.2. Being Blind to the Future

Similarly, studying specific, local classification decisions in terms of
balancing fairness and accuracy does not provide insight about the more
global, long-term effects that such decisions potentially have. This also
presents a contradiction: Some scholarship concerning the trade-off explicitly
aims to support the goals of policymakers, but policymaking by its very nature
takes a long-tailed view. Current policy interventions do not just have a
local impact, but rather also have desired cascading effects that carry into
the future.

Ironically, this contradiction is clearly spelled out in some of the trade-off
literature as an intentional assumption. For example, Corbett-Davies et al.
(2017) explicitly states: “Among the rules that satisfy a chosen fairness
criterion, we assume policymakers would prefer the one that maximizes
immediate utility.” They intentionally examine the “proximate costs and
benefits” of the fairness-accuracy trade-off, assuming that this is the
temporal resolution that would be most useful to policymakers. This approach
enables simplifying mathematical assumptions, as it does not require
evaluating how the specific automated decision under consideration has
potential ramifications in the future. In Corbett-Davies et al. (2017), in
which they examine risk-assessment decisions for granting bail, they
specifically do not need to look at how the immediate decision to detain
someone may in fact be predictive of (even causally linked to) future arrests.
However, if such decisions are applied unfairly across racial demographic
groups (even if somewhere slightly “fairer” on the optimization curve), then
they would just repeat patterns of bias existing in past data.

###  4.3. Unfairness of “Active Fairness” Trade-Off Remedies

Some work regarding the fairness-accuracy trade-off sometimes goes beyond
observing, characterizing, or implementing the trade-off for different
applications, as we have discussed above. They note that while they agree with
the notion that the trade-off is inherent, its effects can perhaps be
mitigated by increasing _both_ accuracy and fairness—essentially, moving the
trade-off optimization curve up and to the right (Figure 1). The trade-off
still exists in this scenario, but perhaps is less of an issue since the
models perform better overall in terms of both accuracy and fairness.

Concretely, authors recommend a technique they call _active feature
acquisition_ or _active fairness_ (Noriega-Campero et al., 2019), which
promotes the idea that “data collection is often a means to reduce
discrimination without sacrificing accuracy” (Chen et al., 2018)—that
collecting more features _for the unprivileged group_ will help ensure fairer
outcomes (Dutta et al., 2020; Bakker et al., 2019). The rationale is that
additional feature collection alleviates the bias in the existing data for
unprivileged groups, which will result in reduced bias in the classification
results for those groups. Moreover, the authors note that gathering more
features for the unprivileged group leads to these benefits without impacting
the privileged group; the privileged group’s accuracy and fairness metrics
remain unchanged.

Setting aside that it might not even be possible to collect more features in
practice, there are important implicit assumptions in this choice of solution.
In particular, it seems like this work poses data collection as a value-
neutral solution to ensure greater fairness. This assumption is clearly false.
It is widely accepted, particularly in sociotechnical literature, that data
collection is often a form of surveillance (Zuboff, 2018; Clarke, 1994;
Brayne, 2017; Cohen, 2011). It is not a neutral act, and is generally not
equally applied across demographic groups in the US.

The choice to collect more data raises a normative question directly in
contradiction with their goal for increased fairness for unprivileged groups:
Do we really want to collect more data on unprivileged groups—groups that
already tend to be surveilled at higher rates than those with more privilege?
In the US in particular there is a long history of tracking non-white and
queer individuals: Black Americans, from Martin Luther King to Black Lives
Matter activists; Japanese Americans during World War II; non-white Muslims,
particularly since 9/11; Latine individuals in relation to immigration status;
trans, particularly trans-feminine, people in sports and bathroom use (Bedoya,
2016; DeSilver et al., 2020; Speri, 2019; Painter, 2011; Conrad, 2009; of
Representatives, 2021). In a more global treatment of fairness, is it fair to
collect more data on these populations just to ensure we are optimizing some
local fairness metric?

One could make the argument that machine learning broadly speaking pushes
toward greater surveillance. The field is pushing to train larger and larger
model specifications, which tend to require training on larger and larger
datasets (Kaplan et al., 2020). These data-hungry methods in turn push for
greater data collection and surveillance in general (Zuboff, 2018). Yet, the
proposed techniques in active fairness to alleviate the fairness-accuracy
trade-off stand apart: They _specifically_ advocate for increasing data
collection of already-surveilled groups. They tend to leave the data for the
privileged group untouched in order to decrease classification disparities
between groups. In essence, this unfairly shifts the burden of producing fair
classification results to the unprivileged group, affording the additional
privilege (i.e. even less relative surveillance) to the already privileged
group. Put another way, their solution to the fairness-accuracy optimization
problem introduces another, unexplored objective function—an objective
function concerning the burden of surveillance, whose solution in this case
causes residual unfairness for the marginalized group.

Some work in active fairness does acknowledge that additional data collection
is not costless; however, this work often discusses it as a necessary cost for
increased fairness rather than investigating it as a potential source for
increased unfairness (Chen et al., 2018). Noriega-Campero et al. (2019) states
that it would be useful to model the cost of each feature in the dataset,
where costs implicate monetary, privacy, and opportunity concerns. Beyond
noting this idea, they do not attempt to formalize it in their work.

Bakker et al. (2019) goes a step further by including cost in their problem
formulation, associating a vector of costs with each feature. However, it is
unclear how they pick the values of those costs and, perhaps more importantly,
they make the assumption that the vector of costs is the same for each
individual in the population. This assumes that different values for different
features do not not incur different social impacts, which is demonstrably not
the case. For example, consider individuals of transgender identity: Trans
people, in comparison to cis people, face significant discrimination in
response to disclosing their identity. In the language of Bakker et al.
(2019), it is more costly to trans people to collect features about gender
identity than it is for cis people. In fact, such disparate costs can be
thought of as the basis for needing to acknowledge and do policymaking using
protected demographic attributes in the first place.

##  5\. Toward a Resolution

###  5.1. Making Normative Assumptions Explicit

Writing mathematical proofs requires assumptions. For example, in machine
learning, when writing proofs about an algorithm’s properties, it is common to
assume that the distribution we are trying to learn is convex. Assumptions
like this enable us to guarantee certain logical conclusions about an
algorithm’s behavior, such as bounds on its convergence rate. While fairness-
accuracy trade-off researchers are accustomed to stating mathematical
assumptions like this, and ensuring that sound mathematical conclusions
follow, we have shown that they do not pay similar attention to normative
assumptions and their ensuing contradictory conclusions. We contend that
researchers should take the time to make explicit such assumptions underlying
their work. Being rigorous and clear about normative assumptions enables them
to be reviewed just as rigorously as mathematical assumptions.

We do not suggest that making such assumptions explicit is a sufficient
solution on its own. Nevertheless, it would still help facilitate greater
scrutiny about the appropriateness of proposed algorithmic fairness solutions.
For example, as we discussed in Section 3, this would allow for considering
that fairness and accuracy could in fact be in accord (Hellman, 2020b; Selbst
et al., 2019). ML researchers should engage the assistance of social
scientists if they believe they lack the expertise to do this work
independently. Moreover, this process should facilitate researchers being
introspective about how their individual backgrounds might inform the
assumptions they bring into their work. This would be one necessary (though
not on its own sufficient) way to address critics of fairness research being
dominated by white voices (Abdurahman, 2019).

Moreover, clarifying implicit normative assumptions could facilitate
rethinking how we measure accuracy. As we note in Section 4, common accuracy
metrics are tied to unfairness. In an attempt to decouple accuracy from
unfairness, one could, for example, put a Bayesian prior on existing
unfairness and try to correct for it. To the best of our knowledge, no
algorithmic fairness scholarship has attempted to do this: to explicitly
assume and model the existing unfairness due to a history of discrimination
against certain demographics.888Such modeling, of course, would not come
without concern, as it would require introducing a different set of
mathematical modeling assumptions that carry their own normative implications.

###  5.2. Tweaking Normative Assumptions for Robustness

Making normative assumptions explicit could also help facilitate more robust
ML fairness research. When investigating algorithmic robustness, researchers
are generally comfortable with relaxing or changing certain mathematical proof
assumptions and reasoning out the resulting changes (or stasis) in algorithm
behavior. As the economist Edward Leamer (1983) notes:

> …an inference is not believable if it is fragile, if it can be reversed by
> minor changes in assumptions. … A researcher has to decide which assumptions
> or which sets of alternative assumptions are worth reporting.

As a test of normative robustness, we similarly recommend that fairness-
accuracy trade-off researchers perturb their normative assumptions and
investigate how this may alter normative outcomes. For example, when
considering surveillance of the unprivileged via active feature acquisition as
an appropriate mechanism for alleviating the trade-off, it would be useful to
state this as an explicit assumption, and then consider surveillance as a
constraint for the problem. In other words, one could ask, how much
surveillance is tolerable for increased fairness? Perhaps none, but perhaps
there is a small set of high quality features that could be collected to serve
this purpose, rather than just indiscriminately collecting a lot of additional
features.

##  6\. Conclusion: Reconsidering the Fairness-Accuracy Trade-off

As Passi and Barocas (2019) note, “Whether we consider a data science project
fair often has as much to do with the formulation of the problem as any
property of the resulting model.” Furthermore, the work of problem formation
is “rarely worked out with explicit normative considerations in mind” (Passi
and Barocas, 2019). As we have shown in this article, not attending explicitly
to these considerations can lead to contradictory, unintended results:
Formulating a trade-off between fairness and accuracy involves a variety of
normative assumptions that can in fact lead to various forms of emergent
unfairness.

Our recommendations in Section 5 to make normative assumptions explicit aim to
remedy this particular type of unfair outcome. However, we recognize that in
the case of the fairness-accuracy trade-off, fully applying these
recommendations may not be sufficient in itself. Rather, in clarifying the
normative concerns of the trade-off, it is quite possible to reasonably
conclude that the effects of emergent unfairness outweigh any benefits that
come from choosing this particular problem formulation. As such, when it comes
to the critical issue of algorithmic fairness, it may be time to reconsider
the framing of trade-offs altogether.

###### Acknowledgements.

Thank you to our anonymous reviewers for their valuable feedback.
Additionally, we would like to thank the following individuals for feedback on
earlier drafts and iterations of this work: Professor Rediet Abebe, Bilan A.H.
Ali, Harry Auster, Kate Donahue, Professor Chris De Sa, Jessica Zosa Forde,
Professor Deborah Hellman, and Kweku Kwegyir-Aggrey. We would also like to
thank the Artificial Intelligence Policy and Practice initiative at Cornell
University, the John D. and Catherine T. MacArthur Foundation, and the Cornell
Humanities Scholars Program.

## References

  * (1)
  * Abdurahman (2019) J. Khadijah Abdurahman. 2019\.  A Response to Racial Categories of Machine Learning by Sebastian Benthall and Bruce Haynes.  https://medium.com/@blacksirenradio/fat-be-wilin-deb56bf92539
  * Abebe et al. (2020) Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G. Robinson. 2020.  Roles for Computing in Social Change. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ (Barcelona, Spain) _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 252–260.
  * Ajunwa (2020) Ifeoma Ajunwa. 2020\.  The Paradox of Automation as Anti-Bias Intervention.  _Cardozo Law Review_ 41, 1671 (2020).
  * Angwin et al. (2016) Julia Angwin, Leff Larson, Surya Mattu, and Lauren Kirchner. 2016.  Machine Bias.  https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
  * Bakker et al. (2019) Michael A. Bakker, Alejandro Noriega-Campero, Duy Patrick Tu, Prasanna Sattigeri, Kush. R Varshney, and Alex ’Sandy’ Pentland. 2019\.  On Fairness in Budget-Constrained Decision Making.  _KDD Workshop on Explainable Artificial Intelligence_ (2019).
  * Barocas et al. (2018) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2018\.  _Fairness and Machine Learning_.  fairmlbook.org.  http://www.fairmlbook.org.
  * Barocas and Selbst (2014) Solon Barocas and Andrew D. Selbst. 2014.  Big Data’s Disparate Impact.  _SSRN eLibrary_ (2014).
  * Bedoya (2016) Alvaro M. Bedoya. 2016\.  The Color of Surveillance.  _Slate_ (18 January 2016).  https://slate.com/technology/2016/01/what-the-fbis-surveillance-of-martin-luther-king-says-about-modern-spying.html
  * Berlin (2013) Isaiah Berlin. 2013\.  The Pursuit of the Ideal.  In _The Crooked Timber of Humanity_ , Henry Hardy (Ed.). Princeton University Press, Chapter 1, 1–20.
  * Bijker et al. (1987) Wiebe E. Bijker, Thomas P. Hughes, and Trevor J. Pinch. 1987\.  _The Social Construction of Technological Systems: New Directions in the Sociology and History of Technology_.  MIT Press, Cambridge, Mass.
  * Binns (2018) Reuben Binns. 2018\.  Fairness in Machine Learning: Lessons from Political Philosophy. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research, Vol. 81)_ , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 149–159.
  * Bishop (1995) Christopher M. Bishop. 1995\.  _Neural Networks for Pattern Recognition_.  Oxford University Press, Inc., USA.
  * Bowker and Star (1999) Geoffrey C. Bowker and Susan Leigh Star. 1999.  _Sorting Things Out: Classification and Its Consequences_.  MIT Press, Cambridge, Mass.
  * Brayne (2017) Sarah Brayne. 2017\.  Big Data Surveillance: The Case of Policing.  _American Sociological Review_ 82, 5 (2017), 977–1008.
  * Budryk (2020) Zack Budryk. 2020\.  Two white students sue UT-Austin claiming they were denied admission based on race.  _The Hill_ (22 July 2020).  https://thehill.com/blogs/blog-briefing-room/news/508608-two-white-students-sue-ut-austin-claiming-they-were-denied
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research, Vol. 81)_ , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 77–91.
  * Chen et al. (2018) Irene Y. Chen, Fredrik D. Johansson, and David Sontag. 2018\.  Why is My Classifier Discriminatory?. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ (Montréal, Canada) _(NIPS’18)_. Curran Associates Inc., Red Hook, NY, USA, 3543–3554.
  * Chouldechova (2017) Alexandra Chouldechova. 2017\.  Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.  _Big Data_ 5, 2 (2017), 153–163.
  * Clarke (1994) R. Clarke. 1994\.  The Digital Persona and Its Application to Data Surveillance.  _Inf. Soc._ 10 (1994), 77–92.
  * Cohen (2011) Nicole Cohen. 2011\.  The Valorization of Surveillance: Towards a Political Economy of Facebook.  _Democratic Communiqué_ 22 (2011).
  * Conrad (2009) Kathryn Conrad. 2009\.  Surveillance, Gender and the Virtual Body in the Information Age.  _Surveillance & Society_ 6, 4 (2009), 380–387.
  * Corbett-Davies et al. (2017) Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017.  Algorithmic Decision Making and the Cost of Fairness. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Halifax, NS, Canada) _(KDD ’17)_. Association for Computing Machinery, New York, NY, USA, 797–806.
  * DeSilver et al. (2020) Drew DeSilver, Michael Lipka, and Dalia Fahmy. 2020\.  10 things we know about race and policing in the U.S.  _Pew Research Center_ (3 June 2020).  https://www.pewresearch.org/fact-tank/2020/06/03/10-things-we-know-about-race-and-policing-in-the-u-s/
  * Dutta et al. (2020) Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Yu Chen, Sijia Liu, and Kush Varshney. 2020\.  Is There a Trade-Off Between Fairness and Accuracy? A Perspective Using Mismatched Hypothesis Testing. In _Proceedings of the 37th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 119)_ , Hal Daumé III and Aarti Singh (Eds.). PMLR, 2803–2813.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012.  Fairness through Awareness. In _Proceedings of the 3rd Innovations in Theoretical Computer Science Conference_ (Cambridge, Massachusetts) _(ITCS ’12)_. Association for Computing Machinery, New York, NY, USA, 214–226.
  * Dwork et al. (2018) Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018.  Decoupled Classifiers for Group-Fair and Efficient Machine Learning. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research, Vol. 81)_ , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 119–133.
  * Flanagan et al. (2008) M. Flanagan, Daniel Howe, and H. Nissenbaum. 2008.  Embodying values in technology: Theory and practice.  _Information Technology and Moral Philosophy_ (01 2008), 322–353.
  * Flanagan and Nissenbaum (2014) Mary Flanagan and Helen Nissenbaum. 2014.  _Values at Play in Digital Games_.  The MIT Press.
  * Forde et al. (2021) Jessica Zosa Forde, A. Feder Cooper, Kweku Kwegyir-Aggrey, Chris De Sa, and Michael Littman. 2021\.  Model Selection’s Disparate Impact in Real-World Deep Learning Applications.  ICML 2021 Science of Deep Learning Workshop.
  * Friedler et al. (2016) Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016.  On the (im)possibility of fairness.  arXiv:1609.07236 [cs.CY]
  * Friedman and Hendry (2019) Batya Friedman and David G. Hendry. 2019.  _Value Sensitive Design: Shaping Technology with Moral Imagination_.  The MIT Press.
  * Gieryn (1983) Thomas F. Gieryn. 1983\.  Boundary-Work and the Demarcation of Science from Non-Science: Strains and Interests in Professional Ideologies of Scientists.  _American Sociological Review_ 48, 6 (1983), 781–795.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nathan Srebro. 2016.  Equality of Opportunity in Supervised Learning.  arXiv:1610.02413 [cs.LG]
  * Hastie et al. (2009) Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009\.  _The Elements of Statistical Learning: Data Mining, Inference and Prediction_ (2 ed.).  Springer.
  * Hecht (2002) Gabrielle Hecht. 2002\.  Rupture-Talk in the Nuclear Age: Conjugating Colonial Power in Africa.  _Social Studies of Science_ 32, 5-6 (2002), 691–727.
  * Heidari et al. (2018) Hoda Heidari, Claudio Ferrari, Krishna Gummadi, and Andreas Krause. 2018.  Fairness Behind a Veil of Ignorance: A Welfare Analysis for Automated Decision Making. In _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc., 1265–1276.
  * Hellman (2020a) Deborah Hellman. 2020a.  Big Data and Compounding Injustice.  Working draft.
  * Hellman (2020b) Deborah Hellman. 2020b.  Measuring Algorithmic Fairness.  _Virgina Law Review_ 106, 4 (2020).
  * Hoffmann (2019) Anna Lauren Hoffmann. 2019\.  Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse.  _Information, Communication & Society_ 22, 7 (2019), 900–915.
  * Hughes (1993) Thomas Parker Hughes. 1993\.  _Networks of Power: Electrification in Western Society, 1880-1930_ (1st ed.).  Johns Hopkins University Press.
  * Joseph et al. (2016) Matthew Joseph, Michael J. Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. 2016.  Rawlsian Fairness for Machine Learning.  _CoRR_ abs/1610.09559 (2016).  arXiv:1610.09559
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020\.  Scaling Laws for Neural Language Models.  arXiv:2001.08361 [cs.LG]
  * Kasy and Abebe (2021) Maximilian Kasy and Rediet Abebe. 2021.  Fairness, Equality, and Power in Algorithmic Decision-Making. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 576–586.  https://doi.org/10.1145/3442188.3445919
  * Kennedy (1961) John F. Kennedy. 1961\.  Executive Order 10925.
  * Kleinberg et al. (2017) Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017\.  Inherent Trade-Offs in the Fair Determination of Risk Scores. In _8th Innovations in Theoretical Computer Science Conference, ITCS 2017, January 9-11, 2017, Berkeley, CA, USA_ _(LIPIcs, Vol. 67)_ , Christos H. Papadimitriou (Ed.). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 43:1–43:23.
  * Kroll et al. (2017) Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017.  Accountable Algorithms.  _University of Pennsylvania Law Review_ 165, 633 (2017).
  * Leamer (1983) Edward Leamer. 1983\.  Let’s Take the Con Out of Econometrics.  _American Economic Review_ 73, 1 (1983), 31–43.
  * Menon and Williamson (2018) Aditya Krishna Menon and Robert C Williamson. 2018.  The Cost of Fairness in Binary Classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research, Vol. 81)_ , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 107–118.
  * Monahan and Skeem (2016) John Monahan and Jennifer L. Skeem. 2016.  Risk Assessment in Criminal Sentencing.  _Annual Review of Clinical Psychology_ 12, 1 (2016), 489–513.
  * Newkirk (2017) Vann R. Newkirk. 2017\.  The Myth of Reverse Racism.  _The Atlantic_ (5 August 2017).  https://www.theatlantic.com/education/archive/2017/08/myth-of-reverse-racism/535689/
  * Noriega-Campero et al. (2019) Alejandro Noriega-Campero, Michiel A. Bakker, Bernardo Garcia-Bulle, and Alex ’Sandy’ Pentland. 2019\.  Active Fairness in Algorithmic Decision Making. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ (Honolulu, HI, USA) _(AIES ’19)_. Association for Computing Machinery, New York, NY, USA, 77–83.
  * of Representatives (2021) Florida House of Representatives. 2021\.  CS/HB 1475: Sex-specific Student Athletic Teams or Sports.  https://www.flsenate.gov/Session/Bill/2021/1475/BillText/c1/PDF
  * Painter (2011) Nell Irvin Painter. 2011\.  _The History of White People_.  W. W. Norton.
  * Passi and Barocas (2019) Samir Passi and Solon Barocas. 2019.  Problem Formulation and Fairness. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ (Atlanta, GA, USA) _(FAT* ’19)_. Association for Computing Machinery, New York, NY, USA, 39–48.
  * Pham (2018) Elyse D. Pham. 2018\.  Fellow Asian-Americans, Back Off of Affirmative Action.  _The Harvard Crimson_ (15 October 2018).  https://www.thecrimson.com/article/2018/10/15/pham-back-off-affirmative-action/
  * Porter (1995) Theodore M. Porter. 1995\.  _Trust in Numbers: The Pursuit of Objectivity in Science and Public Life_.  Princeton University Press.
  * Powles and Nissenbaum (2018) Julia Powles and Helen Nissenbaum. 2018.  The Seductive Diversion of Solving Bias in Artificial Intelligence.
  * Rawls (1971) John Rawls. 1971\.  _A Theory of Justice_.  Belknap Press of Harvard University Press, Cambridge, Massachusetts.
  * Sabato and Yom-Tov (2020) Sivan Sabato and Elad Yom-Tov. 2020.  Bounding the Fairness and Accuracy of Classifiers from Population Statistics. In _Proceedings of the 37th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 119)_ , Hal Daumé III and Aarti Singh (Eds.). PMLR, 8316–8325.
  * Selbst et al. (2019) Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019\.  Fairness and Abstraction in Sociotechnical Systems. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ (Atlanta, GA, USA) _(FAT* ’19)_. Association for Computing Machinery, New York, NY, USA, 59–68.
  * Speri (2019) Alice Speri. 2019\.  The FBI Spends a Lot of Time Spying on Black Americans.  _The Intercept_ (29 October 2019).  https://theintercept.com/2019/10/29/fbi-surveillance-black-activists/
  * Srivastava et al. (2019) Megha Srivastava, Hoda Heidari, and Andreas Krause. 2019\.  Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learning. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (Anchorage, AK, USA) _(KDD ’19)_. Association for Computing Machinery, New York, NY, USA, 2459–2468.
  * Velocci (2021) Beans Velocci. 2021\.  _Binary Logic: Race, Expertise, and the Persistence of Uncertainty in American Sex Research_.  PHD Dissertation, Yale University.  Chapter 1, ”Unsolved Problems of Anomalous Sex: Managing Sexual Multiplicity in Nineteenth-Century Animal Studies”.
  * Wajcman (2004) Judy Wajcman. 2004\.  _TechnoFeminism_.  Polity.
  * Wick et al. (2019) Michael Wick, swetasudha panda, and Jean-Baptiste Tristan. 2019\.  Unlocking Fairness: a Trade-off Revisited. In _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc., 8783–8792.
  * Yang (2010) Xin-She Yang. 2010\.  _Engineering Optimization: An Introduction with Metaheuristic Applications_ (1st ed.).  Wiley Publishing.
  * Zhao and Gordon (2019) Han Zhao and Geoff Gordon. 2019.  Inherent Tradeoffs in Learning Fair Representations. In _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc., 15675–15685.
  * Zuboff (2018) Shoshana Zuboff. 2018\.  _The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power_ (1st ed.).  Profile Books.


* [7] D. A. Micha, K. Runge, Time-dependent many-electron approach to slow ion-atom collisions: The coupling of electronic and nuclear motions, Phys. Rev. A 50 (1) (1994) 322–336.
  * [8] J. Theilhaber, Ab initio simulations of sodium using time-dependent density-functional theory, Phys. Rev. B 46 (20) (1992) 12990.
  * [9] K. Yabana, G. Bertsch, Time-dependent local-density approximation in real time, Phys. Rev. B 54 (7) (1996) 4484.
  * [10] A. Tsolakidis, D. Sánchez-Portal, R. M. Martin, Calculation of the optical response of atomic clusters using time-dependent density functional theory and local orbitals, Phys. Rev. B 66 (23) (2002).
  * [11] J. Hekele, Y. Yao, Y. Kanai, V. Blum, P. Kratzer, All-electron real-time and imaginary-time time-dependent density functional theory within a numeric atom-centered basis function framework, J. Chem. Phys. 155 (15) (2021) 154801\.
  * [12] X. Qian, J. Li, X. Lin, S. Yip, Time-dependent density functional theory with ultrasoft pseudopotentials: Real-time electron propagation across a molecular junction, Phys. Rev. B 73 (3) (Jan. 2006).
  * [13] B. Walker, R. Gebauer, Ultrasoft pseudopotentials in time-dependent density-functional theory, J. Chem. Phys. 127 (16) (2007) 164106.
  * [14] M. Walter, H. Häkkinen, L. Lehtovaara, M. Puska, J. Enkovaara, C. Rostgaard, J. J. Mortensen, Time-dependent density-functional theory in the projector augmented-wave method, J. Chem. Phys. 128 (24) (2008) 244101.
  * [15] A. Castro, H. Appel, M. Oliveira, C. A. Rozzi, X. Andrade, F. Lorenzen, M. A. Marques, E. Gross, A. Rubio, Octopus: a tool for the application of time-dependent density functional theory, Phys. Status Solidi B 243 (11) (2006) 2465–2488.
  * [16] A. Schleife, E. W. Draeger, Y. Kanai, A. A. Correa, Plane-wave pseudopotential implementation of explicit integrators for time-dependent kohn-sham equations in large-scale simulations, J. Chem. Phys. 137 (22) (2012) 22A546.
  * [17] X. Andrade, J. Alberdi-Rodriguez, D. A. Strubbe, M. J. T. Oliveira, F. Nogueira, A. Castro, J. Muguerza, A. Arruabarrena, S. G. Louie, A. Aspuru-Guzik, A. Rubio, M. A. L. Marques, Time-dependent density-functional theory in massively parallel computer architectures: the octopus project, J. Phys.: Condens. Matter 24 (23) (2012) 233202.
  * [18] S. Andermatt, J. Cha, F. Schiffmann, J. VandeVondele, Combining linear-scaling DFT with subsystem DFT in born-oppenheimer and ehrenfest molecular dynamics simulations: From molecules to a virus in solution, J. Chem. Theory Comp. 12 (7) (2016) 3214–3227.
  * [19] C. Shepard, R. Zhou, D. C. Yost, Y. Yao, Y. Kanai, Simulating electronic excitation and dynamics with real-time propagation approach to TDDFT within plane-wave pseudopotential formulation, J. Chem. Phys. 155 (10) (2021) 100901\.
  * [20] C. Pemmaraju, Simulation of attosecond transient soft x-ray absorption in solids using generalized Kohn–Sham real-time time-dependent density functional theory, New J. Phys. 22 (8) (2020) 083063.
  * [21] C. M. Isborn, X. Li, J. C. Tully, Time-dependent density functional theory ehrenfest dynamics: Collisions between atomic oxygen and graphite clusters, J. Chem. Phys. 126 (13) (2007) 134307.
  * [22] K. Lopata, N. Govind, Modeling fast electron dynamics with real-time time-dependent density functional theory: application to small molecules and chromophores, J. Chem. Theory Comput. 7 (5) (2011) 1344–1355.
  * [23] B. Gao, K. Ruud, Y. Luo, Plasmon resonances in linear noble-metal chains., J. Chem. Phys. 137 (19) (2012) 194307.
  * [24] M. Repisky, L. Konecny, M. Kadek, S. Komorovsky, O. L. Malkin, V. G. Malkin, K. Ruud, Excitation energies from real-time propagation of the four-component Dirac–Kohn–Sham equation, J. Chem. Theory Comput. 11 (3) (2015) 980–991.
  * [25] M. Kadek, L. Konecny, B. Gao, M. Repisky, K. Ruud, X-ray absorption resonances near $L_{2,3}$-edges from Real-Time Propagation of the Dirac–Kohn–Sham Density Matrix, Phys. Chem. Chem. Phys. 17 (2015) 22566–22570.
  * [26] L. Konecny, M. Kadek, S. Komorovsky, O. L. Malkina, K. Ruud, M. Repisky, Acceleration of Relativistic Electron Dynamics by Means of X2C Transformation: Application to the Calculation of Nonlinear Optical Properties, J. Chem. Theory Comput. 12 (12) (2016) 5823–5833.
  * [27] T. Moitra, L. Konecny, M. Kadek, A. Rubio, M. Repisky, Accurate Relativistic Real-Time Time-Dependent Density Functional Theory for Valence and Core Attosecond Transient Absorption Spectroscopy, J. Phys. Chem. Lett. 14 (7) (2023) 1714–1724.
  * [28] L. Konecny, S. Komorovsky, J. Vicha, K. Ruud, M. Repisky, Exact two-component TDDFT with simple two-electron picture-change corrections: X-ray absorption spectra near l- and m-edges of four-component quality at two-component cost, J. Phys. Chem. A 127 (5) (2023) 1360–1376.
  * [29] T. Sato, K. L. Ishikawa, Time-dependent complete-active-space self-consistent-field method for multielectron dynamics in intense laser fields, Phys. Rev. A 88 (2) (Aug. 2013).
  * [30] H. Miyagi, L. B. Madsen, Time-dependent restricted-active-space self-consistent-field theory for laser-driven many-electron dynamics, Phys. Rev. A 87 (6) (Jun. 2013).
  * [31] H. Miyagi, L. B. Madsen, Time-dependent restricted-active-space self-consistent-field theory for laser-driven many-electron dynamics. II. extended formulation and numerical analysis, Phys. Rev. A 89 (6) (Jun. 2014).
  * [32] T. Sato, K. L. Ishikawa, Time-dependent multiconfiguration self-consistent-field method based on the occupation-restricted multiple-active-space model for multielectron dynamics in intense laser fields, Phys. Rev. A 91 (2) (Feb. 2015).
  * [33] P. Krause, T. Klamroth, P. Saalfrank, Time-dependent configuration-interaction calculations of laser-pulse-driven many-electron dynamics: Controlled dipole switching in lithium cyanide, J. Chem. Phys. 123 (2005) 074105.
  * [34] H. B. Schlegel, S. M. Smith, X. Li, Electronic optical response of molecules in intense fields: Comparison of td-hf, td-cis, and td-cis(d) approaches, J. Chem. Phys. 126 (2007) 244110.
  * [35] J. A. Sonk, M. Caricato, H. B. Schlegel, Td-ci simulation of the electronic optical response of molecules in intense fields: Comparison of rpa, cis, cis(d), and eom-ccsd, J. Phys. Chem. A 115 (2011) 4678.
  * [36] A. E. DePrince, M. Pelton, J. R. Guest, S. K. Gray, Emergence of excited-state plasmon modes in linear hydrogen chains from time-dependent quantum mechanical methods, Phys. Rev. Lett. 107 (2011) 196806.
  * [37] E. Luppi, M. Head-Gordon, Computation of high-harmonic generation spectra of h2 and n2 in intense laser pulses using quantum chemistry methods and time-dependent density functional theory, Mol. Phys. 110 (2012) 909.
  * [38] P. J. Lestrange, M. R. Hoffmann, X. Li, Time-dependent configuration interaction using the graphical unitary group approach: Nonlinear electric properties, Adv. Quantum Chem. 76 (2018) 295.
  * [39] I. S. Ulusoy, Z. Stewart, A. K. Wilson, The role of the ci expansion length in time-dependent studies, J. Chem. Phys. 148 (2018) 014107.
  * [40] L. S. Cederbaum, J. Zobeley, Ultrafast charge migration by electron correlation, Chem. Phys. Lett. 307 (1999) 205.
  * [41] R. Santra, L. S. Cederbaum, Complex absorbing potentials in the framework of electron propagator theory. i. general formalism, J. Chem. Phys. 117 (2002) 5511\.
  * [42] S. Feuerbacher, T. Sommerfeld, R. Santra, L. S. Cederbaum, Complex absorbing potentials in the framework of electron propagator theory. ii. application to temporary anions, J. Chem. Phys. 118 (2003) 6188.
  * [43] A. I. Kuleff, L. S. Cederbaum, Tracing ultrafast interatomic electronic decay processes in real time and space, Phys. Rev. Lett. 98 (2007) 083201.
  * [44] A. D. Dutoi, L. S. Cederbaum, M. Wormit, J. H. Starcke, A. Dreuw, Tracing molecular electronic excitation dynamics in real time and space, J. Chem. Phys. 132 (2010) 144302.
  * [45] A. I. Kuleff, L. S. Cederbaum, Ultrafast correlation-driven electron dynamics, J. Phys. B: At., Mol. Opt. Phys. 47 (2014) 124002.
  * [46] S. P. Neville, M. S. Schuurman, A general approach for the calculation and characterization of x-ray absorption spectra, J. Chem. Phys. 149 (2018) 154111\.
  * [47] A. Baiardi, Electron dynamics with the time-dependent density matrix renormalization group, J. Chem. Theory Comput. 17 (6) (2021) 3320–3334.
  * [48] H. E. Kristiansen, B. S. Ofstad, E. Hauge, E. Aurbakken, Ø. S. Schøyen, S. Kvaal, T. B. Pedersen, Linear and nonlinear optical properties from tdomp2 theory, J. Chem. Theory Comput. 18 (6) (2022) 3687–3702.
  * [49] C. Huber, T. Klamroth, Explicitly time-dependent coupled cluster singles doubles calculations of laser-driven many-electron dynamics, J. Chem. Phys. 134 (2011) 054113.
  * [50] S. Kvaal, Ab initio quantum dynamics using coupled-cluster, J. Chem. Phys. 136 (19) (2012) 194109.
  * [51] D. A. Pigg, G. Hagen, H. Nam, T. Papenbrock, Time-dependent coupled-cluster method for atomic nuclei, Phys. Rev. C: Nucl. Phys. 86 (2012) 014308.
  * [52] D. R. Nascimento, A. E. DePrince, Linear absorption spectra from explicitly time-dependent equation-of-motion coupled-cluster theory, J. Chem. Theory Comput. 12 (2016) 5834.
  * [53] T. Sato, H. Pathak, Y. Orimo, K. L. Ishikawa, Communication: Time-dependent optimized coupled-cluster method for multielectron dynamics, J. Chem. Phys. 148 (2018) 051101.
  * [54] D. R. Nascimento, A. E. DePrince, A general time-domain formulation of equation-of-motion coupled-cluster theory for linear spectroscopy, J. Chem. Phys. 151 (2019) 204107.
  * [55] L. N. Koulias, D. B. Williams-Young, D. R. Nascimento, A. E. DePrince, X. Li, Relativistic time-dependent equation-of-motion coupled-cluster, J. Chem. Theory Comput. 15 (2019) 6617.
  * [56] H. E. Kristiansen, O. S. Schoyen, S. Kvaal, T. B. Pedersen, Numerical stability of time-dependent coupled-cluster methods for many-electron dynamics in intense laser pulses, J. Chem. Phys. 152 (2020) 071102.
  * [57] A. S. Skeidsvoll, A. Balbi, H. Koch, Time-dependent coupled-cluster theory for ultrafast transient-absorption spectroscopy, Phys. Rev. A 102 (2) (Aug. 2020).
  * [58] T. B. Pedersen, H. E. Kristiansen, T. Bodenstein, S. Kvaal, Ø. S. Schøyen, Interpretation of coupled-cluster many-electron dynamics in terms of stationary states, J. Chem. Theory Comput. 17 (1) (2020) 388–404.
  * [59] F. D. Vila, J. J. Rehr, H. Pathak, B. Peng, A. Panyala, E. Mutlu, N. P. Bauman, K. Kowalski, Real-time equation-of-motion CC cumulant and CC green’s function simulations of photoemission spectra of water and water dimer, J. Chem. Phys. 157 (4) (2022) 044101.
  * [60] Z. Wang, B. G. Peyton, T. D. Crawford, Accelerating real-time coupled cluster methods with single-precision arithmetic and adaptive numerical integration, J. Chem. Theory Comput. 18 (9) (2022) 5479–5491.
  * [61] A. S. Skeidsvoll, T. Moitra, A. Balbi, A. C. Paul, S. Coriani, H. Koch, Simulating weak-field attosecond processes with a lanczos reduced basis approach to time-dependent equation-of-motion coupled-cluster theory, Phys. Rev. A 105 (2) (Feb. 2022).
  * [62] K. Yabana, T. Nakatsukasa, J.-I. Iwata, G. F. Bertsch, Real-time, real-space implementation of the linear response time-dependent density-functional theory, Phys. Status Solidi B 243 (5) (2006) 1121–1138.
  * [63] S. A. Fischer, C. J. Cramer, N. Govind, Excited state absorption from real-time time-dependent density functional theory, J. Chem. Theory Comput. 11 (9) (2015) 4294–4303.
  * [64] C. Ullrich, P. Reinhard, E. Suraud, Electron dynamics in strongly excited sodium clusters: a density-functional study with self-interaction correction, J. Phys. B: At. Mol. Opt. Phys. 31 (9) (1998) 1871.
  * [65] A. De Wijn, M. Lein, S. Kümmel, Strong-field ionization in time-dependent density functional theory, EPL 84 (4) (2008) 43001.
  * [66] T. Akama, Y. Imamura, H. Nakai, Application of real-time time-dependent density functional theory with the cvb3lyp functional to core excitations, Chem. Lett. 39 (4) (2010) 407–409.
  * [67] K. Lopata, B. E. Van Kuiken, M. Khalil, N. Govind, Linear-response and real-time time-dependent density functional theory studies of core-level near-edge X-ray absorption, J. Chem. Theory Comput. 8 (9) (2012) 3284–3292.
  * [68] D. Varsano, L. A. Espinosa-Leal, X. Andrade, M. A. Marques, R. Di Felice, A. Rubio, Towards a gauge invariant method for molecular chiroptical properties in TDDFT, Phys. Chem. Chem. Phys. 11 (22) (2009) 4481–4489.
  * [69] S. Pipolo, S. Corni, R. Cammi, The cavity electromagnetic field within the polarizable continuum model of solvation: An application to the real-time time dependent density functional theory, Comput. Theor. Chem. 1040 (2014) 112–119.
  * [70] J. Mattiat, S. Luber, Electronic circular dichroism with real time time dependent density functional theory: Propagator formalism and gauge dependence, Chem. Phys. 527 (2019) 110464.
  * [71] J. Mattiat, S. Luber, Comparison of Length, Velocity, and Symmetric Gauges for the Calculation of Absorption and Electric Circular Dichroism Spectra with Real-Time Time-Dependent Density Functional Theory, J. Chem. Theory Comput. 18 (9) (2022) 5513–5526.
  * [72] M. Monti, M. Stener, E. Coccia, Electronic circular dichroism from real-time propagation in state space, J. Chem. Phys. 158 (8) (2023) 084102.
  * [73] Y. Takimoto, F. Vila, J. Rehr, Real-time time-dependent density functional theory approach for frequency-dependent nonlinear optical response in photonic molecules, J. Chem. Phys. 127 (15) (2007) 154114.
  * [74] F. Ding, B. E. Van Kuiken, B. E. Eichinger, X. Li, An efficient method for calculating dynamical hyperpolarizabilities using real-time time-dependent density functional theory, J. Chem. Phys. 138 (6) (2013) 064104.
  * [75] F. Ding, J. J. Goings, M. J. Frisch, X. Li, Ab initio non-relativistic spin dynamics, J. Chem. Phys. 141 (21) (2014) 214111.
  * [76] J. E. Peralta, O. Hod, G. E. Scuseria, Magnetization dynamics from time-dependent noncollinear spin density functional theory calculations, J. Chem. Theory Comput. 11 (8) (2015) 3661–3668.
  * [77] C.-L. Cheng, J. S. Evans, T. Van Voorhis, Simulating molecular conductance using real-time density functional theory, Phys. Rev. B 74 (15) (2006) 155112\.
  * [78] U. D. Giovannini, G. Brunetto, A. Castro, J. Walkenhorst, A. Rubio, Simulating pump-probe photoelectron and absorption spectroscopy on the attosecond timescale with time-dependent density functional theory, Chem. Phys. Chem 14 (7) (2013) 1363–1376.
  * [79] K. Nobusada, K. Yabana, Photoinduced electric currents in ring-shaped molecules by circularly polarized laser pulses, Phys. Rev. A 75 (3) (2007) 032518\.
  * [80] C. M. Isborn, X. Li, Singlet–Triplet Transitions in Real-Time Time-Dependent Hartree–Fock/Density Functional Theory, J. Chem. Theory Comput. 5 (9) (2009) 2415–2419.
  * [81] K.-M. Lee, K. Yabana, G. Bertsch, Magnetic circular dichroism in real-time time-dependent density functional theory, J. Chem. Phys. 134 (14) (2011) 144106\.
  * [82] M. Repisky, S. Komorovsky, M. Kadek, L. Konecny, U. Ekström, E. Malkin, M. Kaupp, K. Ruud, O. L. Malkina, V. G. Malkin, ReSpect: Relativistic spectroscopy DFT program package, J. Chem. Phys. 152 (18) (2020) 184101.
  * [83] M. De Santis, L. Storchi, L. Belpassi, H. M. Quiney, F. Tarantelli, Pyberthart: A relativistic real-time four-component tddft implementation using prototyping techniques based on python, J. Chem. Theory Comput. 16 (4) (2020) 2410–2429.
  * [84] L. Konecny, M. Kadek, S. Komorovsky, K. Ruud, M. Repisky, Resolution-of-identity accelerated relativistic two- and four-component electron dynamics approach to chiroptical spectroscopies, J. Chem. Phys. 149 (20) (2018) 204104.
  * [85] J. L. Heully, I. Lindgren, E. Lindroth, S. Lundqvist, A. M. Martensson-Pendrill, Diagonalisation of the Dirac Hamiltonian as a basis for a relativistic many-body procedure, J. Phys. B: At. Mol. Phys. 19 (18) (1986) 2799–2815.
  * [86] H. J. A. Jensen, ”Douglas–Kroll the Easy Way”, The Conference Talk, REHE 2005, Mülheim (Germany).
  * [87] W. Kutzelnigg, W. Liu, Quasirelativistic theory equivalent to fully relativistic theory, J. Chem. Phys. 123 (24) (2005) 241102.
  * [88] W. Liu, W. Kutzelnigg, Quasirelativistic theory. II. theory at matrix level, J. Chem. Phys. 126 (11) (2007) 114107.
  * [89] M. Ilias, T. Saue, An infinite-order two-component relativistic hamiltonian by a simple one-step transformation, J. Chem. Phys. 126 (6) (2007) 064102.
  * [90] J. J. Goings, J. M. Kasper, F. Egidi, S. Sun, X. Li, Real time propagation of the exact two component time-dependent density functional theory, J. Chem. Phys. 145 (10) (2016) 104107.
  * [91] S. Knecht, M. Repisky, H. J. A. Jensen, T. Saue, Exact two-component hamiltonians for relativistic quantum chemistry: Two-electron picture-change corrections made simple, J. Chem. Phys. 157 (11) (2022) 114106.
  * [92] D. B. Williams-Young, A. Petrone, S. Sun, T. F. Stetina, P. Lestrange, C. E. Hoyer, D. R. Nascimento, L. Koulias, A. Wildman, J. Kasper, et al., The Chronus quantum software package, Wiley Interdiscip. Rev. Comput. Mol. Sci. 10 (2) (2020) e1436.
  * [93] L. Ye, H. Wang, Y. Zhang, W. Liu, Self-adaptive real-time time-dependent density functional theory for x-ray absorptions, J. Chem. Phys. 157 (7) (2022) 074106.
  * [94] J. M. Kasper, P. J. Lestrange, T. F. Stetina, X. Li, Modeling L_2,3-edge X-ray absorption spectroscopy with real-time exact two-component relativistic time-dependent density functional theory, J. Chem. Theory Comput. 14 (4) (2018) 1998–2006.
  * [95] J. J. Sakurai, Advanced Quantum Mechanics, Addison–Wesley, 1967.
  * [96] D. P. Craig, T. Thirunamachandran, Molecular Quantum Electrodynamics, Dover Publications, New York, 1988.
  * [97] P. A. M. Dirac, The quantum theory of the electron, Proc. Roy. Soc. A 117 (1928) 610–624.
  * [98] P. A. M. Dirac, The quantum theory of the electron. II, Proc. Roy. Soc. A 118 (1928) 351–361.
  * [99] R. E. Moss, Advanced Molecular Quantum Mechanics, Chapman and Hall, London, 1973\.
  * [100] K. G. Dyall, K. Faegri Jr., Introduction to Relativistic Quantum Chemistry, Oxford University Press, New York, 2007.
  * [101] M. Reiher, A. Wolf, Relativistic Quantum Chemistry: The Fundamental Theory of Molecular Science; 2nd Edition, Wiley-VCH, 2014.
  * [102] J. D. Jackson, Classical Electrodynamics; 3rd Edition, Wiley, 1998.
  * [103] M. Gell-Mann, The interpretation of the new particles as displaced charge multiplets, Nuovo Cimento Suppl. 4 (2) (1956) 848–866.
  * [104] S. Bernadotte, A. J. Atkins, C. R. Jacob, Origin-independent calculation of quadrupole intensities in x-ray spectroscopy, J. Chem. Phys. 137 (20) (2012) 204106\.
  * [105] N. H. List, T. Saue, P. Norman, Rotationally averaged linear absorption spectra beyond the electric-dipole approximation, Mol. Phys. 115 (1-2) (2016) 63–74.
  * [106] N. H. List, T. R. L. Melin, M. van Horn, T. Saue, Beyond the electric-dipole approximation in simulations of x-ray absorption spectroscopy: Lessons from relativistic theory, J. Chem. Phys. 152 (18) (2020) 184110.
  * [107] N. O. Foglia, D. Maganas, F. Neese, Going beyond the electric-dipole approximation in the calculation of absorption and (magnetic) circular dichroism spectra including scalar relativistic and spin–orbit coupling effects, J. Chem. Phys. 157 (8) (2022) 084120.
  * [108] M. Göppert-Mayer, Über elementarakte mit zwei quantensprüngen, Ann. Phys. 401 (3) (1931) 273–294.
  * [109] A. D. Bandrauk, F. Fillion-Gourdeau, E. Lorin, Atoms and molecules in intense laser fields: gauge invariance of theory and models, Journal of Physics B: Atomic, Molecular and Optical Physics 46 (15) (2013) 153001.
  * [110] A. J. Thorvaldsen, K. Ruud, K. Kristensen, P. Jørgensen, S. Coriani, A density matrix-based quasienergy formulation of the Kohn–Sham density functional response theory using perturbation-and time-dependent basis sets, J. Chem. Phys. 129 (21) (2008) 214108.
  * [111] R. McWeeny, Method of Molecular Quantum Mechanics, 2nd Edition, Academic Press, San Diego, CA, 1989.
  * [112] M. Kadek, Advancing relativistic electronic structure methods for solids and in the time domain (2018).
  * [113] A. L. Fetter, J. D. Walecka, Quantum Theory of Many-Particle Systems, Dover Publications, 2003.
  * [114] T. Saue, Ph.D. thesis, University of Oslo (1996).  [link].
URL dirac.chem.sdu.dk/thesis/96.saue_phd.pdf

  * [115] S. Komorovsky, M. Repisky, L. Bucinsky, New quantum number for the many-electron Dirac-Coulomb Hamiltonian, Phys. Rev. A 94 (5) (2016) 052104.
  * [116] J. Frenkel, et al., Wave Mechanics, Advanced General Theory, Vol. 436, Oxford, 1934.
  * [117] D. J. Tannor, Introduction to quantum mechanics, University Science Books, Sausalito, California, 2007.
  * [118] C. A. Ullrich, Time-Dependent Density-Functional Theory: Concepts and Applications, Oxford University Press, 2012.
  * [119] E. Runge, E. K. Gross, Density-functional theory for time-dependent systems, Phys. Rev. Lett. 52 (12) (1984) 997.
  * [120] R. van Leeuwen, Mapping from densities to potentials in time-dependent density-functional theory, Phys. Rev. Lett. 82 (19) (1999) 3863.
  * [121] A. D. Becke, A new mixing of Hartree–Fock and local density-functional theories, J. Chem. Phys. 98 (2) (1993) 1372–1377.
  * [122] R. Bauernschmitt, R. Ahlrichs, Treatment of electronic excitations within the adiabatic approximation of time dependent density functional theory, Chem. Phys. Lett. 256 (4-5) (1996) 454–464.
  * [123] M. R. Provorse, C. M. Isborn, Electron dynamics with real-time time-dependent density functional theory, Int. J. Quantum Chem. 116 (10) (2016) 739–749.
  * [124] P. Ziesche, S. Kurth, J. P. Perdew, Density functionals from lda to gga, Comput. Mater. Sci. 11 (2) (1998) 122–127.
  * [125] M. Thiele, E. Gross, S. Kümmel, Adiabatic approximation in nonperturbative time-dependent density-functional theory, Phys. Rev. Lett. 100 (15) (2008) 153004\.
  * [126] C. A. Ullrich, Time-dependent density-functional theory beyond the adiabatic approximation: Insights from a two-electron model system, J. Chem. Phys. 125 (23) (2006) 234108.
  * [127] G. Vignale, W. Kohn, Current-dependent exchange-correlation potential for dynamical linear response theory, Phys. Rev. Lett. 77 (10) (1996) 2037.
  * [128] C. A. Ullrich, I. Tokatly, Nonadiabatic electron dynamics in time-dependent density-functional theory, Phys. Rev. B 73 (23) (2006) 235102.
  * [129] G. Vignale, C. A. Ullrich, S. Conti, Time-dependent density functional theory beyond the adiabatic local density approximation, Phys. Rev. Lett. 79 (24) (1997) 4878.
  * [130] I. V. Tokatly, Quantum many-body dynamics in a Lagrangian frame: I. Equations of motion and conservation laws, Phys. Rev. B 71 (16) (2005) 165104.
  * [131] A. Rajagopal, J. Callaway, Inhomogeneous electron gas, Phys. Rev. B 7 (5) (1973) 1912.
  * [132] T. Saue, T. Helgaker, Four-component relativistic Kohn–Sham theory, J. Comput. Chem. 23 (8) (2002) 814–823.
  * [133] P. Romaniello, P. de Boeij, Relativistic two-component formulation of time-dependent current-density functional theory: Application to the linear response of solids, J. Chem. Phys. 127 (17) (2007) 174111.
  * [134] W. Jia, D. An, L.-W. Wang, L. Lin, Fast real-time time-dependent density functional theory calculations with the parallel transport gauge, J. Chem. Theory Comput. 14 (11) (2018) 5645–5652.
  * [135] R. E. Stanton, S. Havriliak, Kinetic balance: A partial solution to the problem of variational safety in dirac calculations, J. Chem. Phys. 81 (1984) 1910–1918.
  * [136] S. Komorovsky, M. Repisky, O. L. Malkina, V. G. Malkin, I. Malkin Ondik, M. Kaupp, A fully relativistic method for calculation of nuclear magnetic shielding tensors with a restricted magnetically balanced basis in the framework, J. Chem. Phys. 128 (10) (2008) 104101.
  * [137] M. Repisky, S. Komorovsky, O. L. Malkina, V. G. Malkin, Restricted magnetically balanced basis applied for relativistic calculations of indirect nuclear spin–spin coupling tensors in the matrix Dirac–Kohn-Sham framework, Chem. Phys. 356 (1-3) (2009) 236–242.
  * [138] J. Kubler, K.-H. Hock, J. Sticht, A. R. Williams, Density Functional Theory of Non-Collinear Magnetism, J. Phys. F: Met. Phys. 18 (1988) 469–483.
  * [139] L. M. Sandratskii, Noncollinear Magnetism in Itinerant-Electron Systems: Theory and Applications, Adv. Phys. 47 (1998) 91–160.
  * [140] C. van Wüllen, Spin Densities in Two-component Relativistic Density Functional Calculations: Noncollinear Versus Collinear Approach, J. Comp. Chem. 23 (2002) 779–785.
  * [141] G. Scalmani, M. J. Frisch, A New Approach to Noncollinear Spin Density Functional Theory beyond the Local Density Approximation, J. Chem. Theory Comput. 8 (2012) 2193–2196.
  * [142] S. Komorovsky, P. J. Cherry, M. Repisky, Four-Component Relativistic Time-Dependent Density-Functional Theory Using a Stable Noncollinear DFT Ansatz Applicable to Both Closed- and Open-Shell Systems, J. Chem. Phys. 151 (2019) 184111.
  * [143] J. Sikkema, L. Visscher, T. Saue, M. Iliaš, The molecular mean-field approach for correlated relativistic calculations, J. Chem. Phys. 131 (12) (2009) 124116.
  * [144] A. Castro, M. A. Marques, A. Rubio, Propagators for the time-dependent Kohn–Sham equations, J. Chem. Phys. 121 (8) (2004) 3425–3433.
  * [145] M. Hochbruck, C. Lubich, On Magnus integrators for time-dependent Schrödinger equations, SIAM J. Numer. Anal. 41 (3) (2003) 945–963.
  * [146] J. Liu, Z. Guo, J. Sun, W. Liang, Theoretical studies on electronic spectroscopy and dynamics with the real-time time-dependent density functional theory, Front. Chem. China 5 (1) (2010) 11–28.
  * [147] W. Magnus, On the exponential solution of differential equations for a linear operator, Commun. Pure Appl. Math. 7 (4) (1954) 649–673.
  * [148] S. Blanes, P. C. Moan, Fourth-and sixth-order commutator-free Magnus integrators for linear and non-linear dynamical systems, Appl. Numer. Math. 56 (12) (2006) 1519–1537.
  * [149] A. Alvermann, H. Fehske, High-order commutator-free exponential time-propagation of driven quantum systems, J. Comput. Phys. 230 (15) (2011) 5930–5956.
  * [150] A. Gomez Pueyo, S. Blanes, A. Castro, Propagators for quantum-classical models: Commutator-free magnus methods, J. Chem. Theory Comput. 16 (3) (2020) 1420–1430.
  * [151] C. T. Chapman, W. Liang, X. Li, Ultrafast coherent electron–hole separation dynamics in a fullerene derivative, J. Phys. Chem. Lett. 2 (10) (2011) 1189–1192.
  * [152] X. Li, S. M. Smith, A. N. Markevitch, D. A. Romanov, R. J. Levis, H. B. Schlegel, A time-dependent Hartree–Fock approach for studying the electronic optical response of molecules in intense fields, Phys. Chem. Chem. Phys. 7 (2) (2005) 233–239.
  * [153] F. Wang, C. Y. Yam, G. Chen, Time-dependent density-functional theory/localized density matrix method for dynamic hyperpolarizability, J. Chem. Phys. 126 (24) (2007) 244102.
  * [154] B. Kanungo, V. Gavini, Real time time-dependent density functional theory using higher order finite-element methods, Phys. Rev. B 100 (11) (2019) 115148\.
  * [155] W. Liang, C. T. Chapman, X. Li, Efficient first-principles electronic dynamics, J. Chem. Phys. 134 (18) (2011) 184102.
  * [156] C. Müller, M. Sharma, M. Sierka, Real-time time-dependent density functional theory using density fitting and the continuous fast multipole method, J. Comput. Chem. 41 (30) (2020) 2573–2582.
  * [157] D. Williams-Young, J. J. Goings, X. Li, Accelerating real-time time-dependent density functional theory with a nonrecursive Chebyshev expansion of the quantum propagator, J. Chem. Theory Comput. 12 (11) (2016) 5333–5338.
  * [158] R. Baer, Y. Kurzweil, L. S. Cederbaum, Time-dependent density functional theory for nonadiabatic processes, Isr. J. Chem. 45 (1-2) (2005) 161–170.
  * [159] J. Crank, P. Nicolson, A practical method for numerical evaluation of solutions of partial differential equations of the heat-conduction type, in: Mathematical proceedings of the Cambridge philosophical society, Vol. 43, Cambridge University Press, 1947, pp. 50–67.
  * [160] I. Schelter, S. Kümmel, Accurate evaluation of real-time density functional theory providing access to challenging electron dynamics, J. Chem. Theory Comput. 14 (4) (2018) 1910–1927.
  * [161] C. Lian, S.-Q. Hu, M.-X. Guan, S. Meng, Momentum-resolved TDDFT algorithm in atomic basis for real time tracking of electronic excitation, J. Chem. Phys. 149 (15) (2018) 154104.
  * [162] J. J. Goings, P. J. Lestrange, X. Li, Real-time time-dependent electronic structure theory, Wiley Interdiscip. Rev. Comput. Mol. Sci. 8 (1) (2018) e1341.
  * [163] M. Hochbruck, A. Ostermann, Exponential integrators, Acta Numer. 19 (2010) 209–286.
  * [164] T. Sato, H. Pathak, Y. Orimo, K. L. Ishikawa, Communication: Time-dependent optimized coupled-cluster method for multielectron dynamics, J. Chem. Phys. 148 (5) (2018) 051101.
  * [165] T. B. Pedersen, S. Kvaal, Symplectic integration and physical interpretation of time-dependent coupled-cluster theory, J. Chem. Phys. 150 (14) (2019) 144106\.
  * [166] L. N. Koulias, D. B. Williams-Young, D. R. Nascimento, A. E. DePrince III, X. Li, Relativistic real-time time-dependent equation-of-motion coupled-cluster, J. Chem. Theory Comput. 15 (12) (2019) 6617–6624.
  * [167] A. Gomez Pueyo, M. A. Marques, A. Rubio, A. Castro, Propagators for the time-dependent Kohn–Sham equations: Multistep, Runge–Kutta, exponential Runge–Kutta, and commutator free Magnus methods, J. Chem. Theory Comput. 14 (6) (2018) 3040–3052.
  * [168] M. Yang, A. Sissay, M. Chen, K. Lopata, Intruder Peak-Free Transient Inner-Shell Spectra Using Real-Time Simulations, J. Chem. Theory Comput. 18 (2) (2022) 992–1002.
  * [169] S. Dey, R. Mittra, Efficient computation of resonant frequencies and quality factors of cavities via a combination of the finite-difference time-domain technique and the Padé approximation, IEEE Microw. Wirel. Compon. Lett. 8 (12) (1998) 415–417.
  * [170] W.-H. Guo, W.-J. Li, Y.-Z. Huang, Computation of resonant frequencies and quality factors of cavities by FDTD technique and Padé approximation, IEEE Microw. Wirel. Compon. Lett. 11 (5) (2001) 223–225.
  * [171] A. Bruner, D. LaMaster, K. Lopata, Accelerated broadband spectra using transition dipole decomposition and Padé approximants, J. Chem. Theory Comput. 12 (8) (2016) 3741–3750.
  * [172] X. Andrade, J. N. Sanders, A. Aspuru-Guzik, Application of compressed sensing to the simulation of atomic systems, Proc. Natl. Acad. Sci. U.S.A. 109 (35) (2012) 13928–13933.
  * [173] V. A. Mandelshtam, H. S. Taylor, Harmonic inversion of time signals and its applications, J. Chem. Phys. 107 (17) (1997) 6756–6769.
  * [174] M. R. Wall, D. Neuhauser, Extraction, through filter-diagonalization, of general quantum eigenvalues or classical normal mode frequencies from a small number of residues or a short-time segment of a signal. I. Theory and application to a quantum-dynamics model, J. Chem. Phys. 102 (20) (1995) 8011–8022.
  * [175] V. A. Mandelshtam, H. S. Taylor, A low-storage filter diagonalization method for quantum eigenenergy calculation or for spectral analysis of time signals, J. Chem. Phys. 106 (12) (1997) 5085–5090.
  * [176] V. A. Mandelshtam, Fdm: the filter diagonalization method for data processing in nmr experiments, Prog. Nucl. Magn. Reson. Spectrosc. 38 (2) (2001) 159–196.
  * [177] P. Norman, K. Ruud, T. Saue, Principles and Practices of Molecular Properties, Wiley-VCH, Chichester, 2018.
  * [178] J. C. Slater, A simplification of the hartree-fock method, Phys. Rev. 81 (3) (1951) 385.
  * [179] S. H. Vosko, L. Wilk, M. Nusair, Accurate spin-dependent electron liquid correlation energies for local spin density calculations: a critical analysis, Can. J. Phys. 58 (8) (1980) 1200–1211.
  * [180] K. G. Dyall, A. S. P. Gomes, Revised relativistic basis sets for the 5d elements hf–hg, Theor. Chem. Acc. 125 (1-2) (2010) 97–100.
  * [181] L. D. Barron, Molecular Light Scattering and Optical Activity, Cambridge University Press, 2004.
  * [182] I. Warnke, F. Furche, Circular dichroism: electronic, Wiley Interdiscip. Rev. Comput. Mol. Sci. 2 (1) (2012) 150–166.
  * [183] J. P. Perdew, K. Burke, M. Ernzerhof, Generalized gradient approximation made simple, Phys. Rev. Lett. 77 (1996) 3865–3868.
  * [184] J. P. Perdew, K. Burke, M. Ernzerhof, Generalized gradient approximation made simple [phys. rev. lett. 77, 3865 (1996)], Phys. Rev. Lett. 78 (1997) 1396–1396.
  * [185] K. G. Dyall, Relativistic and nonrelativistic finite nucleus optimized double zeta basis sets for the 4p, 5p and 6p elements, Theor. Chem. Acc. 99 (6) (1998) 366–371.
  * [186] K. G. Dyall, Relativistic quadruple-zeta and revised triple-zeta and double-zeta basis sets for the 4p, 5p, and 6p elements, Theor. Chem. Acc. 115 (5) (2006) 441–447.
  * [187] T. H. Dunning Jr, Gaussian basis sets for use in correlated molecular calculations. I. The atoms boron through neon and hydrogen, J. Chem. Phys. 90 (2) (1989) 1007–1023.
  * [188] R. A. Kendall, T. H. Dunning Jr, R. J. Harrison, Electron affinities of the first-row atoms revisited. systematic basis sets and wave functions, J. Chem. Phys. 96 (9) (1992) 6796–6806.
  * [189] R. W. Boyd, Nonlinear Optics, Academic Press, 2008.
  * [190] B. S. Ofstad, H. E. Kristiansen, E. Aurbakken, Øyvind Sigmundson Schøyen, S. Kvaal, T. B. Pedersen, Adiabatic extraction of nonlinear optical properties from real-time time-dependent electronic-structure theory (2023).  arXiv:2302.02779.
  * [191] A. D. Becke, Density-functional exchange-energy approximation with correct asymptotic behavior, Phys. Rev. A 38 (6) (1988) 3098–3100.
  * [192] C. Lee, W. Yang, R. G. Parr, Development of the Colle-Salvetti correlation-energy formula into a functional of the electron density, Phys. Rev. B 37 (2) (1988) 785–789.
  * [193] P. J. Stephens, F. J. Devlin, C. F. Chabalowski, M. J. Frisch, Ab Initio Calculation of Vibrational Absorption and Circular Dichroism Spectra Using Density Functional Force Fields, J. Phys. Chem. 98 (45) (1994) 11623–11627.
  * [194] N. J. Long, Organometallic Compounds for Nonlinear Optics - The Search for En-light-enment!, Angew. Chem. Int. Ed. 34 (1995) 21–38.
  * [195] S. Bella, Second-order nonlinear optical properties of transition metal complexes, Chem. Soc. Rev. 30 (6) (2001) 355–366.
  * [196] C. Adamo, V. Barone, Toward reliable density functional methods without adjustable parameters: The PBE0 model, J. Chem. Phys. 110 (13) (1999) 6158–6170.
  * [197] D. E. Woon, T. H. Dunning Jr, Gaussian basis sets for use in correlated molecular calculations. iii. the atoms aluminum through argon, J. Chem. Phys. 98 (2) (1993) 1358–1371.