# Understanding Accuracy-Efficiency Trade-Offs as a Means for Holding
Distributed ML Systems Accountable

A. Feder Cooper  Cornell University, Department of Computer Science
<EMAIL_ADDRESS>,  Karen Levy  Cornell University, Department of Information
Science<EMAIL_ADDRESS>and  Christopher De Sa  Cornell University,
Department of Computer Science<EMAIL_ADDRESS>
###### Abstract.

Trade-offs between accuracy and efficiency are found in multiple non-computing
domains, such as law and public health, which have developed rules and
heuristics to guide how to balance the two in conditions of uncertainty. While
accuracy-efficiency trade-offs are also commonly acknowledged in some areas of
computer science, their policy implications remain poorly examined. Drawing on
risk assessment practices in the US, we argue that, since examining accuracy-
efficiency trade-offs has been useful for guiding governance in other domains,
explicitly framing such trade-offs in computing is similarly useful for the
governance of computer systems. Our discussion focuses on real-time
distributed ML systems; understanding the policy implications in this area is
particularly urgent because such systems, which include autonomous vehicles,
tend to be high-stakes and safety-critical. We describe how the trade-off
takes shape for these systems, highlight gaps between existing US risk
assessment standards and what these systems require in order to be properly
assessed, and make specific calls to action to facilitate accountability when
hypothetical risks become realized as accidents in the real world. We close by
discussing how such accountability mechanisms encourage more just, transparent
governance aligned with public values.

††copyright: none

##  1\. Introduction

Engineering is defined by trade-offs—by competing goals that need to be
negotiated in order to meet system design requirements. One of the central
trade-offs in engineering, particularly in computer science, is between
_accuracy_ and _efficiency_. More specifically, there is an inherent tension
between _how correct_ computations are and _how long_ it takes to compute
them. While this trade-off is of general relevance, it plays out in various
ways across different subfields of computing. For example, in computer
hardware, circuits can use approximation techniques to relax constraints on
accuracy—on how they perform bitwise computations—in order to speed up
performance. In image processing, varying numbers of pixels can be used to
represent a given image; using fewer pixels causes a loss in accuracy of the
image being represented, but also furthers space-efficiency by requiring less
memory to store the image.

These kinds of examples are abundant in computing. In fact, the trade-off is
so well-known to computer scientists that it has even given rise to its own
subfield, _approximate computing_ (Moreau et al., 2018; Mittal, 2016). This
subfield has shown that it is useful to analyze the accuracy-efficiency trade-
off in the context of how _error tolerant_ an application is—that is, how
different domains resolve the question of how much inaccuracy can be permitted
for the sake of increased efficiency, while retaining guarantees about quality
and safety (Sampson, 2015).

While the trade-off is commonly acknowledged in some areas of computer
science, its policy implications remain poorly examined. We therefore focus on
analyzing these implications in the context of a particularly urgent
technological domain— _real-time distributed machine learning systems_. The
trade-off between accuracy and efficiency in such systems is tunable and
context-dependent. It is not an all-or-nothing choice between the two, and
choosing how to appropriately tune them depends on both a particular system’s
goals and the environment in which the system is deployed. For some cases, it
might be beneficial to favor accuracy over efficiency, while the reverse might
be desirable in other contexts. Tuning choices in different contexts will
entail different emergent behaviors in technical systems—behaviors that are
potentially high-stakes if, for example, they have large effects on overall
system safety.

We argue that the accuracy-efficiency trade-off exposes a high-level
abstraction that policymakers should use to hold such systems accountable.
That is, rather than operating at one of two extremes—either solely having
policymakers rely on technical experts to make high-stakes policy decisions or
inundating policymakers with underlying low-level technical details—we
advocate for something in between: ML systems researchers should focus on
providing guarantees concerning correctness and performance, and should build
associated tools to help policymakers reason about these guarantees. These
tools should effectively expose the degree of uncertainty in distributed ML
systems. This in turn would facilitate lawmakers’ ability to hold these
systems to account, and to regulate the resulting risk of their deployment in
high-stakes domains, such as autonomous cars and military technology.

We emphasize this systems focus because much of the sociotechnical
conversation in ML focuses on _algorithmic_ fairness. This has left the the
systems components—notably, scalability and speed—under-explored in terms of
their potential disparate impact on historically marginalized communities. As
a result, ML _systems_ present under-examined challenges for technological
accountability. We take the initial steps to bring some of these challenges to
light, and suggest a novel framing for how to hold real-time distributed
machine learning systems accountable.

This contribution demonstrates the need for government-required risk
assessment tools for analyzing distributed ML systems. We contend that,
without such tools, effective public oversight of these systems will not be
possible. Instead, we run the risk of manufacturers ignoring accountability
mechanisms when constructing ML systems—or worse, deliberately making these
systems difficult to assess in order to obscure responsibility when accidents
occur. In both of these scenarios, the burden would fall on individual victims
to prove manufacturer responsibility. This dynamic would make accountability
quite difficult to achieve in practice; the power and resource imbalances
between individual victims and large ML-system manufacturers would make tort
or other civil litigation infeasible (Abraham and Rabin, 2019). For
historically underserved communities, these imbalances are even more
significant. Without the tools we advocate for in this paper, the barrier for
holding distributed ML systems accountable is even higher to surmount for
these groups.

###  1.1. Contribution

Our analysis will focus on the US legal context, but we contend that our
argument elicits principles that apply more broadly. To make our case, we
survey relevant concepts from law and computer science, and then synthesize
this discussion to advocate for a concrete tech policy contribution.
Approaching our topic in this interdisciplinary manner leads us to follow a
nontraditional format. We need to justify our conceptual contribution in two
directions, and thus provide a significant amount of relevant background
information concerning how the accuracy-efficiency trade-off translates to
both law and computer science.

We first discuss how the trade-off functions in relation to decision-making in
disciplines other than computing (Section 2.1), most notably in US risk
assessment policy (Section 2.2). Then, we provide an analogous discussion for
ML algorithms (Section 3.1) and distributed ML systems (Section 3.2). We argue
that reasoning about accuracy-efficiency trade-offs and accountability in
highly technical domains is not a new problem. This suggests that, with the
right technical tools, we can similarly hold distributed ML systems
accountable (Section 4). We close by discussing how such tools for increased
accountability encourage more just, transparent governance aligned with public
values (Section 5).

##  2\. Accuracy-Efficiency Trade-Offs

Gathering increasingly accurate information is a computationally expensive
task that is in tension with acting efficiently. We first discuss how this
tension plays out in various domains outside of computing. This discussion
provides an intuition for how accuracy-efficiency trade-offs function in a
variety of context-specific ways, and demonstrates how there is precedent for
governing accuracy-efficiency trade-offs in disciplines other than computing.
We contend that this precedent from other domains suggests that these trade-
offs are governable for computing as well.

###  2.1. The Ubiquity of Accuracy-Efficiency Trade-Offs

The trade-off at the heart of this paper is not unique to computing. It can be
observed in a diverse range of domains, many of which are regulated in the US,
including law, the economy, and public health. In these disciplines,
efficiency often can be thought of interchangeably with speed. In decision
theory, the time-value of information is an important concept for making
choices. There is a cost to gathering increasingly accurate information.
Waiting to act based on information is itself an action—one that can have
potentially more negative consequences than acting earlier on imperfect
information. Kahneman et al. (1982) elaborate on this idea in their well-known
cognitive psychology research concerning reasoning about uncertainty. They
argue that humans use various heuristics to make decisions more efficiently,
often acting on biases they have due to incomplete information. There is a
tension between taking the time to gather more information and making a more
informed decision—between the speed of making a decision and the quality of
information used to make it.

Sunstein (2002) connects this idea directly to the potential hazards of using
heuristics in legal decision-making. Nevertheless, he observes that heuristics
are common—and necessary—in the law to obtain a suitable balance between
efficient resolution and the “best” (i.e., most accurate) adjudicative
outcomes.

A number of rules in US civil and criminal procedure—speedy trial
requirements, local filing deadlines, statutes of limitations—impose time
constraints for the sake of efficient case resolution; these values must be
balanced against needs for thorough fact-finding and argumentation. Due
process implicates both values, and some legal rules explicitly consider how
to balance between them. The standard for preliminary injunctive relief in the
United States, for example, requires courts to predict whether irreparable
injury will occur because of the passage of time, if relief is not granted
before the (often lengthy) full resolution of a case (Lichtman, 2002).

Federal Rule of Evidence 403 allows for the exclusion of relevant evidence
from a court proceeding if the probative value of that evidence is
substantially outweighed by a danger of undue delay or the wasting of time.
These and other rules promoting judicial efficiency are, in the words of
Justice Oliver Wendell Holmes, “a concession to the shortness of life” (Ree,
[n.d.])—they attempt to balance between the twin goals of getting matters
right and getting them done, with recognition that there is real social value
to each.

Debates about the merits of the so-called “precautionary principle” in
policymaking also capture the accuracy-efficiency trade-off. The precautionary
principle advises extreme caution around new innovations when there is
substantial unknown risk; in operation, it effectively places the burden of
proof on risk-creating actors (like chemical plants) to provide sufficient
evidence that they are _not_ producing significant risk of harm. As with
speedy trials, there is a trade-off between the time it takes to gather
evidence—to understand the risk landscape—and making informed decisions based
on this landscape. There are legal rationales on both sides of the spectrum
with regard to how this trade-off should be implemented. For example, critics
of the precautionary principle could be said to favor efficiency. They find
the principle to be too stringent with regard to the burden it places on
accuracy; it is “literally paralyzing” in its attempts to regulate risk
(Sunstein, 2003). On the other side, others argue that the precautionary
principle provides a valuable way to reason about preventing harm by shifting
the burden of proof of safety to potential risk creators. They are supportive
of the fact that the principle requires actors to justify the risks they
create: It is worth the time cost to gather information, such that it is
possible to better manage risk in the context of scientific uncertainty
(Sachs, 2011).111In addition to these examples from the law, the accuracy-
efficiency trade-off is salient in other aspects of governance. In particular,
it plays an important role in wartime intelligence gathering. There is an
inherent tension between gathering more accurate intelligence about an
opponent or enemy and acting on that intelligence before it becomes stale and
loses its usefulness. This is the so-called “fog of war” notion, which
attempts to capture the relationship between how the uncertainty of
information changes over time (von Clausewitz, 1832).

A notable example of the precautionary principle demonstrating the trade-off
in action concerns public health management of the SARS outbreak in the early
2000s. During the early outbreak of the disease, there was significant
uncertainty around the risk of it spreading and how lethal it could be. The
precautionary principle was adopted as a public health value at all of the
disease epicenters: Individuals who were even remotely suspected of having
come into contact with SARS were placed under strict quarantine. Critics of
this practice—both during the early outbreak and after the threat had
subsided—argued that mass quarantining led to a tremendous and unnecessary
loss of liberty. Years later, such (notably pre-COVID-19) critics made this
case based on retrospective analysis that indicated 66% fewer individuals
could have been quarantined with the same public health outcome. They argued
that, without taking such restrictive precautions, it would have still been
possible to prevent a SARS pandemic (Chowkwanyun et al., 2016).

We are not yet at a time in which such retrospective analysis regarding the
precautionary principle can be conducted for the ongoing COVID-19 pandemic.
Nevertheless, the trade-off has still played a role in an additional public
health context: antibody tests. The World Health Organization (WHO) has
recently argued that, prior to certifying COVID-19 antibodies for treatment,
it is necessary to _guarantee_ that such antibodies confer immunity to the
virus. Several medical professionals have challenged this mandate from WHO,
highlighting the time-sensitive nature of taking action in the pandemic:
“Demanding incontrovertible evidence may be appropriate in the rarefied world
of scholarly scientific inquiry. But in the context of a raging pandemic, we
simply do not have the luxury of holding decisions in abeyance until all the
relevant evidence can be assembled. Failing to take action is itself an action
that carries profound costs and health consequences” (Weinstein et al., 2020).
More generally, the authors claim that it is the norm for healthcare
practitioners to act on incomplete information—to balance potential
inaccuracies in available data with the urgency to treat serious conditions.

###  2.2. Relating US Federal Risk Assessment Policy to Accuracy-Efficiency
Trade-Offs

The examples in the previous section provide an intuition for how pervasive
the accuracy-efficiency trade-off is in different domains. Beyond this
intuition, in the US the trade-off is implicated more formally in federal risk
assessment standards and regulatory rule-making. That is, risk assessment
policy documents acknowledge that, no matter how much time and resources one
spends gathering scientific knowledge to assess risks, it will ultimately
always be necessary to make decisions with uncertainty—to pass judgments in
the face of incomplete information (Council, 1983, 1994). As Levy and Johns
(2016) note, it is the epistemological nature of science itself that makes
uncertainty inevitable in science-based policymaking: “Agencies charged with
protecting public health and the environment must make decisions in the face
of scientific uncertainty, because science by its nature is incomplete and
only rarely provides precise answers to the complex questions policymakers
pose.” There is always a degree of imprecision in scientific knowledge’s
ability to capture what is true, and that knowledge is constantly subject to
potential revision in light of newly collected information.

In other words, taking more time to gather information can increase accuracy,
but is directly at odds with efficiency in decision-making. In risk
assessment, this trade-off gets framed in terms of _ex ante_ (before-the-fact)
and _ex post_ (after-the-fact) risk-mitigating interventions. _Ex ante_
mechanisms, or “premarketing approval” (Council, 1994), embody the
precautionary principle approach: They emphasize collecting evidence about
potential risks before approving a new substance or technology. For example,
the US Food and Drug Administration (FDA) typically requires multiple phases
of clinical trials before a new drug is allowed for public use. This _ex ante_
regulatory authority is deliberately slow for the sake of increased safety.
The FDA is empowered to require drug companies to submit sufficient data, such
that a detailed risk assessment can be conducted before the drug goes on the
market.

In contrast, for efficiency other agencies do not require many precautionary
measures, and instead concentrate their authority in _ex post_ “post hoc
mechanisms” (Council, 1994). Such mechanisms tend to require that agencies,
rather than companies, acquire the data necessary to determine responsibility,
after an accident or hazardous health outcome occurs. For example, the US
National Highway Traffic Safety Administration (NHTSA) has few, relatively
nonrestrictive _ex ante_ authorities for determining what types of vehicles it
allows on roads. Instead, NHTSA’s strongest regulatory authority is its
ability to recall faulty cars _ex post_ (Administration, 2016). In other
words, NHTSA implements their own trade-off between accuracy and
efficiency—favoring lack of _ex ante_ regulation as a way to ensure speedy
development and deployment of new car technology, even if such lack of
regulation comes with a cost in correctness in that technology.

These are just two examples, illustrating opposite choices in the trade-off,
with how accuracy and efficiency relate to _ex ante_ and _ex post_
enforcement, respectively. This trade-off spectrum applies to the risk
assessment and rule-making practices of numerous other agencies in the US,
including the Environmental Protection Agency (EPA), the Occupational Safety
and Health Administration (OSHA), and the Consumer Product Safety Commission
(CPSC) (Council, 1994). These different agencies each have different, domain-
specific _ex ante_ and _ex post_ trade-off biases, and have the ability to
convene scientific panels to help them determine how to strike the right
balance.

Despite these domain-specific differences, reports from the National Research
Council (NRC) recognize that there are cross-cutting elements of risk
assessment that impact all of these agencies (Council, 1994, 1983). For
example, the reports provide general recommendations for improving standards
for accounting for uncertainty and its relationship to risk. Such
recommendations include documenting the imprecision in sensors and models, and
highlighting model uncertainty by clarifying the assumptions and
simplifications that inform model construction. Moreover, the reports advocate
for the importance of teasing out these low-level details, and communicating
them to both decision-makers and the public, in order to ensure that policy
goals are reflective of the known risk landscape.

This discussion shows that accuracy-efficiency trade-offs are a useful and
natural way for policymakers to regulate varied, complex technical domains. We
therefore ask, Why not use this framework for making policy concerning
distributed ML systems? The specifics of the domain may be different—notably,
real-time systems involve a degree of high speed not present in, for example,
the process of evaluating the safety of a new chemical agent. Nevertheless US
risk assessment policy indicates that reasoning about the trade-offs between
speed and correctness, and their relationship to risk, is not a new problem.
We therefore contend that reasoning about underlying accuracy-efficiency
trade-offs can enable risk assessment and management for these emerging
technologies.

However, translating the above regulatory framing to this domain presents
novel challenges. We will require new tools, which we clarify in Sections 3
and 4, to reason effectively about accuracy-efficiency trade-offs in
distributed ML systems. These tools will help us gather the data we need to do
appropriate risk assessment and subsequent policy-making for these systems.
Before we can delve into the specifics of these new tools, we must finish
justifying that accuracy-efficiency trade-offs are an appropriate abstraction
for accounting for the behavior of distributed ML systems. Having explained in
this section how reasoning about accuracy-efficiency trade-offs is useful from
the perspective of policy-making, we next make our case from a technical
perspective.

##  3\. Trading off Accuracy and Efficiency in Computing

Accuracy-efficiency trade-offs are particularly relevant across
computing.222The accuracy-efficiency trade-off is arguably a central, defining
concern for the entire field of computing. Ohm and Frankle (2019) call
efficiency the “cardinal virtue” of computing in order to discuss what they
view as exceptional cases of inserting inefficiency into computer systems—what
they term “desirable inefficiency.” Instead, viewing the accuracy-efficiency
_trade-off_ as central enables us to not refer to “inefficient” computing
models (e.g. cryptography) as exceptional. Instead, we conceive of them as
implementing the trade-off at one end of the accuracy-efficiency spectrum
(privileging accuracy, in the case of cryptography), which strikes us as a
more precise and generalizable statement of the issues at stake. As with our
non-computing examples, in computing the accuracy-efficiency trade-off is a
spectrum, not a binary decision (Figure 1). To understand this, let us
consider a familiar example—JPEG image compression. Raw images tend to be very
high resolution, meaning that they contain many pixels per inch. Such images
therefore tend to require a lot of storage space. However, it is often not
necessary to use so many pixels for a high-quality image. To the human eye, a
compressed version of the raw image often suffices; removing some pixels or
averaging and combining the values of neighboring pixels often is not
detectable. Moreover, compressing a raw image to a JPEG takes up less computer
storage space and can lead to faster image processing when doing photo editing
since there are fewer pixels to consider; it is both more space- and time-
efficient. Reducing the accuracy of the image can lead to greater
computational efficiencies when manipulating it.

Figure 1. Implementing the accuracy-efficiency trade-off in approximate
computing: Image compression, bit precision, distributed systems, and
scientific computing. For clarity we present this as a line; however, there
are diminishing returns toward either end of the spectrum.

The notion of this trade-off between accuracy and efficiency forms the basis
of _approximate computing_. The main idea of this field is that a computer
system can achieve certain performance benefits if it exerts less
computational effort to compute perfectly accurate answers. In other words, it
is possible to _relax_ accuracy in order to yield efficiency improvements
(Moreau et al., 2018; Mittal, 2016; Sampson, 2015).333We do not include the
pathological case in which _all_ accuracy is sacrificed in order to do
something really fast but completely wrong. Nevertheless, there are cases
where an implementation could, for example, be wrong 40% of the time (for
increased speed) and still achieve certain application-specific quality and
safety goals. As with JPEGs, relaxing the accuracy does not necessarily have
negative consequences; rather, it is possible that the decreased accuracy has
no observable impact for a particular application. That is, some applications
are tolerant of inaccuracy; they are error resilient.

As in public health, the law, and economics, tools for reasoning about the
trade-off inform decisions about how to implement it in
computing—particularly, implementation decisions made by computer scientists.
Computer scientists create theoretical tools to formally characterize the
trade-off, which they then leverage to determine what they deem to be the
right implementation in different application domains. For example, formal
reasoning around the trade-off can yield application-specific quality metrics.
Quality can be thought of as how a program conceives of “good enough” results.
Often, the quality of “good enough” cannot be guaranteed with complete
certainty, but can be verified with high probability. Leaving room for
uncertainty allows for edge case behaviors, which might fall below the
specified quality threshold. Quality metrics therefore capture how much an
approximation is allowed to deviate from the precise version’s results.
Computer scientists can then design software that requires a certain degree of
program quality with a certain (high) probability (Sampson, 2015). A popular
example of this comes from Amazon’s cloud computing services (AWS). Their
cloud storage service provides “11 9’s” of reliability with regard to storing
data objects, meaning that 99.999999999% of the time saving such objects to
the cloud occurs without error (Amazon, 2021).

High-stakes application domains. The relevance of the trade-off becomes
increasingly urgent in application domains for which maximizing both accuracy
and efficiency is essential. Consider an autonomous surgical robot or
autonomous car. Both application domains require high accuracy and efficiency
in order to be safe and reliable. Useful quality metrics are thus especially
salient in these areas: We need to have a sense of how much error we can
tolerate in order to meet certain application-specific speed demands.

The same concern applies to police use of facial recognition software. For
example, as described in Dietterich (2018), a study in South Wales found a
false positive rate of 91.3% in a facial recognition application that tries to
match faces with outstanding arrest warrants at public events. There have been
similar findings in other cities, such as Detroit (Lee, 2020). While these
examples are not speed-sensitive—they involve scenarios in which a human could
intervene to verify the accuracy of the results prior to acting—this would not
necessarily be the case if such technology were integrated into police body
cameras for the purpose of making in-the-moment, potentially life-or-death
decisions.

We constrain our discussion to the particular risks of this technology
associated with the accuracy-efficiency trade-off, though emphasize that these
risks are not the only ones.444Two such risks, which are highly relevant to
analyzing this technology though are out of scope for our focus, are increased
surveillance and militarization of the police. Advocates for this technology
argue that in situations of “imminent danger”, efficiency is crucial; they
contend that speedily identifying a person-of-interest is necessary for
safety. However, accuracy also clearly implicates safety: In heightened stress
environments, mistaking someone for a person-of-interest has repeatedly proven
catastrophic, particularly in the US.

Because of these competing technological goals, it is not clear how
approximate computing could be safe in this context, as the high stakes
involved do not lend themselves to error resilience. It is likely not safe to
use such technology at all. To be clear, our observation here should not be
construed as an endorsement of a “need for speed” in law enforcement
technology. Advocates for police reform in the US have convincingly argued for
years that such a “need” is in fact often constructed to benefit police in
cases of misconduct (Rojas and Fausset, 2020). Moreover, it is crucial not to
let debates about the accuracy of a system override reasoned deliberation
about whether we want to implement such a system in the first place (Abebe et
al., 2020).

###  3.1. Accuracy-Efficiency Trade-Offs in Machine Learning Algorithms

Balancing accuracy and efficiency is a central concern across machine learning
disciplines. This is particularly evident in deep learning: There is an
ongoing, increasing emphasis on training larger and larger models to yield
more accurate results, which comes with host of efficiency challenges,
including significantly increased training time, model storage requirements,
and energy usage (Kaplan et al., 2020).555The trade-off notably did not first
become relevant with (though is arguably increasingly urgent due to) the
advent of modern statistical ML. Several influential papers on artificial
intelligence (AI) from the 1980s and 1990s also demonstrate the potentially
high impact of appropriately dealing with accuracy-efficiency trade-offs
(Horvitz, 1987; Boddy and Dean, 1994). More generally, trained ML models
perform inference that is not always correct, and often tolerate a certain
degree of inaccuracy.

Being resilient to errors is necessary for producing robust models. This
notion of error resilience (or inaccuracy tolerance) varies for different
types of ML algorithms. Regardless of particular differences, there is a
general tension between _correctness_ and _performance_. For example, the
correctness of a ML training algorithm can be understood as whether or not the
algorithm converged to the distribution we set out to learn, i.e., _Did we
learn the right model?_ Its performance indicates whether convergence to the
distribution—whether correct or incorrect—happened in a timely manner, i.e.,
_How fast did we learn the model?_ As with other approximate computing
problems, ML can relax its demands on accuracy in order to achieve increases
in efficiency. In fact, this relaxation is a requirement in many learning
domains. Without it, learning inference computations can be so slow to perform
that they become intractable. We describe three such examples below.

Subsampling. Performance directly relates to the size of the task on which we
conduct learning. Intuitively, if a learning algorithm is slow on tasks with
small datasets, then that algorithm will be slow, if not computationally
intractable, on much larger ones. This relationship between runtime and task
size often exists due to coupling between the computation done by the learning
procedure’s optimization algorithm and the task’s dataset size. For example,
when computing the gradient needed to determine which direction the learning
algorithm should step for its next iteration, it is often necessary to sum
over every data point in the dataset. In Figure 2, this summation becomes
increasingly costly for the Gradient Descent (GD) algorithm when operating on
larger datasets.

Figure 2. Runtime per iteration of GD and SGD as a function of dataset size.
The x-axis is in exponential scale. The runtime for GD (a full-batch method)
is coupled with dataset size: As size increases, so does runtime per
iteration. A minibatch method like SGD (which here uses only 1 data point to
compute the gradient) is decoupled from the dataset size: it has a relatively
constant runtime per iteration.

A very common approach for improving efficiency is to use a subsample or
_minibatch_ of the dataset, rather than the whole dataset, when performing
calculations. In the case of computing gradients, instead of using a _full
batch_ (i.e., the whole dataset) we use a randomly sampled subset of the data
points, which involves spending less time on the computation of a particular
iteration. Stochastic Gradient Descent (SGD) is an example of an algorithm
that takes this approach, in which using a minibatch can have minimal impact
on the overall accuracy of the learned model. A particular iteration of the
algorithm will have less accuracy when computing the gradient; but, when run
for lots of iterations, the final result is usually still statistically
correct. In expectation, we can learn the same distribution as if we had been
using the whole dataset in each iteration; we can often theoretically
guarantee robustness (Bottou et al., 2018).

Moreover, the decision to subsample is not all-or-nothing; it is a spectrum.
It is possible to vary the minibatch size the algorithm uses. Larger
minibatches—especially those that approach the size of the full
dataset—require more time but are also more accurate per iteration.
Conversely, smaller batch sizes make each iteration faster and more scalable
to larger datasets, but in doing so sacrifice accuracy per iteration.
Determining the right sweet spot in this trade-off often depends on the
particular learning task, and often falls under the area of study called
hyperparameter optimization (Feurer and Hutter, 2019).

Resource-constrained techniques. Recent years have seen an increase in the
variety of computers available and a corresponding increase in the variety of
computations we wish to run on them. For example, Internet of Things (IoT)
devices and sensors, such as Amazon Echo, perform inference. They serve up
answers to spoken language questions; however, they also have limited on-board
capabilities to perform computations locally. These limitations take several
forms. For example, such devices might not have a lot of power to process data
quickly or might lack storage capacity for large amounts of data. As a result,
such IoT devices often only have smaller, coarser-grained models in local
memory, which can be used for quickly returning (potentially less accurate)
inference results.

Often, these devices can communicate with more sophisticated computers over
the Internet, offloading computation or storage to those computers. Because
these computers have more memory and processing capabilities, they can store
larger models that are capable of more nuanced inference. However, this
communication exposes another trade-off between accuracy and efficiency; it
takes time to send the data to a remote computer, perform some computation,
and then return a response to the device (Birman et al., 2019). That
computation may be more accurate due to using a larger, finer-grained model,
but achieving that accuracy comes with a cost in speed. Conversely, doing the
computation locally on the device would be faster; however, due to the
device’s more limited computational resources, it will not necessarily be as
accurate.

For example, prior work in computer vision considers how to handle the trade-
off when performing ML on mobile devices, such as smart phones.666Aside from
being faster, there are several reasons why local computation and storage
might be desirable for a mobile application, as opposed to offloading these
requirements to more powerful remote computers. Notably, local computation can
ensure privacy, as the learned model and collected data never leave the mobile
device (Wang et al., 2019). For such resource-constrained devices, different
applications have different needs in terms of how to trade-off between how
accurately and how quickly (or how memory-consuming) a computation is
performed. Some research has explored these application-specific needs,
providing an interface for flexibly implementing different points along the
accuracy-efficiency trade-off spectrum (Howard et al., 2017). Depending on the
application domain, a model developer can tune a larger model that uses more
resources (i.e., a model that is slower or uses more memory but is more
accurate) or one that is smaller and uses fewer resources (i.e., a model that
is faster or uses less memory but is less accurate).

Asynchrony. As a last example, we examine the trade-off in ML in
_asynchronous_ settings. In contrast with synchronous computation, which
performs one step at a time, it is possible to run computations
asynchronously, in which different computer processes or threads777Threads and
processes are mechanisms for parallelizing computation on a computer. For more
information, refer to Arpaci-Dusseau and Arpaci-Dusseau (2018). perform
computations side-by-side. This facilitates dividing computationally intensive
tasks into parts, such that different portions can happen in parallel and then
can be combined to compute the final result. In other words, asynchrony can
speed up ML since multiple parts of the learning problem can be computed at
once.

However, depending on how the parallel results are combined, asynchrony can
also lead to decreases in accuracy. If different processes end up working on
overlapping parts of the overarching computation, the process that finishes
second can overwrite the value computed by the one that finished first,
causing inaccuracies (De Sa et al., 2017; Alistarh et al., 2018; Lian et al.,
2017; Niu et al., 2011). This can be avoided by forcing processes to
coordinate their updates so that they do not overwrite each other, but such
coordination takes time; it enables more accuracy, but decreases efficiency.
This overwriting is sometimes worth the speed-ups it enables; it is still
possible—though not always guaranteed—to compute good quality learning
estimates (Sa et al., 2016).888Asynchrony can be used in combination with
minibatching or in applications that involve resource-constrained devices.
Combining yields additional accuracy-efficiency trade-offs, as these
synchronous methods’ particular trade-offs becomes implemented in conjunction
with those in the asynchronous regime.

###  3.2. Implications in Real-World ML Systems

We have thus far provided examples of the accuracy-efficiency trade-off in ML
_algorithm_ techniques, but have not yet considered how the trade-off behaves
in real-world, _deployed systems_ —systems that often consist of multiple
computers that communicate and work together to solve large, complex problems.
Our overall aim is to understand the particular trade-off challenges in such
_distributed machine learning systems_. To accomplish this, we need to account
for the “distributed systems” component just as much as “machine learning”.
The distributed setting is what enables life-saving technology like autonomous
vehicles (AVs) (Administration, 2016), which reflect a triumph of new systems
abstractions, not just innovations in ML (Birman et al., 2019).

Importantly, new risks emerge when such fast, scalable systems are deployed in
the real world. For example, researchers recently built a model that they
showed could outperform humans in identifying gay individuals using facial
recognition technology (Wang and Kosinski, 2018).999This claim has been
challenged by several researchers, notably Leuner (2019). This disturbing
result yielded a blizzard of media attention (Hawkins, 2017; Murphy, 2017),
yet it was also small-scale and slow. Consider a similar model, but one that
is scalable and fast—integrated with a CCTV surveillance system serving real-
time inference and deployed in a country hostile to LGBTQ rights. This may
sound like science fiction, but low-latency, distributed vision systems
already exist (Wang, 2018).

This example is broadly generative with regard to the range of potential risks
from large-scale ML systems. We focus on the risks related to accuracy-
efficiency trade-off implementation choices, first clarifying how the trade-
off is implicated in distributed computing. From this basis, we can then layer
on more complexity: We weave in our understanding of the accuracy-efficiency
trade-off for ML algorithms (Section 3.1) and observe how the different
tensions interact with each other. Considered together, we demonstrate how
machine learning and distributed systems trade-offs present especially
challenging problems for real-time, high-impact systems like AVs. In Section 4
we will ultimately argue that clarifying the relationship between these risks
and trade-off choices can help policymakers hold such systems accountable.

Accuracy-efficiency trade-offs in distributed computing. In contrast to a
single, solitary computer, a _distributed system_ is a network of computers
that communicate with each other. Via this communication, the computers can
work together to solve problems. Each computer in the network has its own data
and performs its own computations, and it shares data and computation results
with other computers in the network when necessary. For example, if a computer
needs data from another computer in order to execute a computation, it can
request the data from that computer.

Because the computers are in distributed locations—whether in the same data
center or across the world—and need to communicate, there are important
considerations with regard to how efficiently information can be shared
between them. That is, when a computer contacts another in the system to
request its data, it takes time to complete the request and receive the
data—reducing time-efficiency. There are also issues of accuracy between
computers in the system. Each computer has its own data—its own snapshot of
what it knows to be the state of the overarching system. However, that
information is not complete; it is just a subset and can possibly contradict
the information that other computers in the system have.

In other words, in distributed systems we can more specifically frame the
trade-off between accuracy and efficiency as a tension between _consistency_
and _latency_ —the speed with which the system updates. There is a trade-off
between all of the computers in the system having the same understanding of
the data in the system and the time it takes to propagate that understanding
throughout the system (Abadi, 2012; Brewer, 2012). Due to this trade-off, in
distributed systems that update their data frequently it is actually quite
difficult to quickly build a consistent, holistic understanding of the
environment across different computers in the network. This is because
consistency is a moving target; each computer processes information locally
faster than it can share it with the entire network. Given that it takes time
to communicate, it is hard for computers to stay completely up to date with
each other. Nevertheless, for the sake of efficiency, individual computers in
the system often need to make decisions in the presence of inconsistency.
Otherwise, because of the tension between consistency and latency, waiting for
complete consistency across computers before an individual computer could make
local changes would bring the entire system to a standstill. Instead,
particular distributed system implementations need to answer the question of
how much inconsistency and slowness they can each tolerate, which is often
application-dependent.

To understand this spectrum, we will use the example of a social media
website, which has computers hosting its data distributed all over the world.
A user visiting the site from a personal device tends to access the
geographically closest computer server hosting the site; different users
across the world therefore access different computer servers. Such a system
favors efficiency (i.e., low latency) over the different computer servers
being consistent with each other. It is more important to return the website
to each user quickly than it is to make sure that every user is accessing the
website with exactly the same data. This is one reason why on some social
media sites it is possible to see out-of-order comments on a feed; the site is
making a best effort to resolve its current state, which entails aggregating
information from across the system. It attempts to build a consistent picture,
but limits how much time it spends doing so—sacrificing consistency—so that it
can remain fast (DeCandia et al., 2007; Lu et al., 2015; Vogels, 2009).

The system implements this choice via its communication strategy. Rather than
contacting every computer in the system to construct a coordinated, consistent
picture (which would take a lot of time) a particular computer only
communicates with a subset. It trades off the accuracy it would get from
communicating with every computer for the efficiency of communicating with
fewer computers (Hellerstein and Alvaro, 2019). Based on this choice, it is
possible to quantify consistency and to measure and monitor its maintenance
throughout a distributed system (Lu et al., 2015; Shang et al., 2018).
Developers can reason about the degree of inconsistency their particular
system can tolerate safely, and can detect and tune the system’s
implementation accordingly to also enforce an upper bound on latency (Golab et
al., 2011; Barbara and Garcia-Molina, 1990; Yu and Vahdat, 2000).

Accuracy-efficiency trade-offs in distributed ML systems. Given this
background on distributed systems in general and our earlier discussion of ML
(Section 3.1), we can now specifically consider accuracy-efficiency trade-offs
in real-time (i.e., latency-critical) distributed ML systems. For coherence
with the main framing of the trade-off between accuracy and efficiency in this
paper, we will use this language, instead of consistency and latency, going
forward—keeping in mind that consistency and latency can be viewed as cases of
accuracy and efficiency, respectively.

As an example, consider a distributed system of autonomous vehicles. Numerous
vehicles are potentially networked with each other, and with other devices
like smart traffic lights. Moreover, while each vehicle moves throughout the
environment with its own local notion of the state of the environment,
information that other vehicles possess could also prove useful. For example,
if an accident is up ahead, a vehicle closer to the crash can communicate that
information to the vehicles behind it, which in turn can apply pressure to
their brakes and potentially prevent a pile-up.

In such real-time transportation domains, accuracy and efficiency are both
critical. Some ML inference applications may be able to tolerate wide margins
of error, but in high-stakes domains a high degree of inaccuracy may be
unsafe. The same goes for efficiency; such systems will need to make decisions
quickly and, much like the non-computing examples in Section 2, there is an
inherent trade-off between waiting to make a completely informed decision and
making a decision fast enough for it to be useful (Abadi, 2012; Brewer, 2012).
What is unique here is the degree of time-efficiency needed—in some cases,
inference decisions will be necessary at sub-second speeds. Moreover, the
urgency of resolving this tension is not merely a hypothetical situation; the
accuracy-efficiency trade-off in fact played a crucial role in the Uber AV
crash in 2018 (Board, 2019).

In short, it is not entirely clear what the right trade-off implementation is
for real-time systems like AVs (Dietterich, 2018). Given the dynamic nature of
the environment, the particular implementation will depend on context. Some
environments will be more efficiency-critical: It would be catastrophic for a
car to take an extra half-second to be certain that there is a pedestrian
directly in front of it (Board, 2019). In other cases, having an accurate
sense of the environment may be more important than allowing the car to
operate quickly. For example, when detecting a deep pothole up ahead, it could
be safer for a car to slow down to decide its course of action—to accurately
determine if the hole is shallow enough for the car to continue on its course
or deep enough to warrant veering off the road to avoid it.

Distributed ML systems raise different accuracy-efficiency questions than
either distributed systems that do not involve ML, or ML systems that are not
distributed. As we saw in Section 3.1, since ML models (necessarily
imperfectly) approximate representations of the world, it is possible for ML
models to operate on data that are not completely accurate and still yield
results that are correct _enough_ —that fall within the same bounds of
imperfection that we deem tolerable when operating on accurate data. We can
extend such data inaccuracies beyond things like subsampling to include data
staleness and numerical error inherent in asynchronous distributed settings
(Bailis et al., 2012; DeCandia et al., 2007; Yu and Vahdat, 2000). Allowing
for such staleness comes with the benefit of increasing efficiency, as the
system would not need to wait to synchronize—to completely resolve staleness
issues before proceeding with its computation. Similar to the single computer
case, their overall output still _can be_ correct even when operating on
numerically imprecise or stale data in a distributed setting; however,
existing work in this field does not necessarily guarantee such output _must
be_ correct (Alistarh et al., 2018; Gong et al., 2014; Lian et al., 2017; Niu
et al., 2011; Sa et al., 2015; Zhang et al., 2015).

Instead, prior work has examined this phenomenon at a high level by looking at
correctness and speed performance of end-to-end ML systems, rather than
directly evaluating the underlying accuracy-efficiency trade-off. This work
focuses on empirical results for tuning the staleness of the underlying data
storage layer. Tuning has generally either been manual—curated to the
particular problem domain—or absent, leaving an engineer to pick from a few
predefined settings that enforce high accuracy guarantees, ignore accuracy
guarantees altogether in favor of efficiency, or attempt some middle-ground,
“in-between” approach (Abadi et al., 2016; Ho et al., 2013; Li et al., 2014;
Kosaian et al., 2019; Pan et al., 2016).

While it is possible to implement any of these different points in the trade-
off, current large-scale systems for distributed learning and inference tend
to opt for efficiency. They focus on minimizing communication between
computers in the system in order to be fast enough to scale to larger
problems. Some of these systems can achieve orders of magnitude in efficiency
improvements by dropping data updates without simultaneously destroying
correctness (Niu et al., 2011; Tsitsiklis et al., 1986). It is not clear these
approaches will work for real-time distributed ML systems that are safety-
critical, such as autonomous vehicles, as it will not always be safe for these
systems to lose updates. Existing approaches to mitigate such losses in ML
systems involve increasing communication between computers in the system.
However, this strategy impacts the other side of the accuracy-efficiency
trade-off, leading to inefficiencies from bottlenecks in coordination between
computers.101010This problem is similar to what exists in weakly consistent
storage systems, which have side-stepped this issue by using semantic
information to coordinate “only when necessary” (DiPippo and Wolfe, 1997;
Garcia-Molina, 1983; Weihl, 1988). In short, due to these challenges, it
remains an open research question how different safety-critical, real-time
distributed ML systems should implement the trade-off.

##  4\. Accuracy-Efficiency Trade-Offs as a Mechanism for Accountability

While these systems are really complex, such complexity should not serve as a
rationale to preclude their regulation. Rather, the fact that these challenges
remain unresolved presents an opportunity: Stakeholders aside from engineers
could help shape implementations; they could inform accuracy-efficiency trade-
off choices so that they align with the public’s interests and values, not
just those of manufacturers. This is why we have taken considerable space to
clarify a variety of accuracy-efficiency trade-offs—from how they generally
impact the field of computing to how they describe the range of possible
behaviors for distributed ML systems.

Though much of our prior discussion is well-acknowledged in technical
communities (albeit, in other forms), to date the implications of the trade-
off have not been made legible to policymakers. As documented in Section 3,
this trade-off is not binary; it is a spectrum and can be treated like a
tunable dial set appropriately to the context. Our hope is that exposing this
dial for distributed ML systems will provide a degree of technical
transparency to lawmakers, such that high-stakes systems do not get deployed
without sufficient public oversight. We believe that explicitly exposing
accuracy-efficiency trade-offs provides a mechanism for holding distributed ML
systems accountable for the risks they create.

We now concretely address the gaps between existing risk assessment tools in
other domains (Section 2.2) and what is needed to perform successful oversight
of distributed ML systems (Section 3.2). When such a system is deployed and
yields an undesirable outcome, we can examine accountability along two
dimensions: the time window around the outcome, which we consider in terms of
_ex ante_ and _ex post_ divisions, and actors that can assess the system’s
behavior, which consist of computer scientists and policy-makers. Computer
scientists can assist with accountability predominantly _ex ante_ , while
policymakers can enforce accountability predominantly _ex post_ ; however,
there is also a region of overlap in which computer scientists can assist
policymakers with _ex post_ evaluation and policymakers can frame _ex ante_
risks prior to deploying systems. We therefore propose a twofold call-to-
action for enabling risk assessment in this domain: 1) Computer scientists
must build tools to expose accuracy-efficiency trade-offs and 2) Policymakers
should use these tools to assess trade-off implementations, and meaningfully
intervene to ensure implementations align with public values. We provide more
detail below, situating these calls-to-action in terms of _ex ante_ and _ex
post_ risk assessment gaps, respectively.

###  4.1. Addressing _ex ante_ Risk Assessment Gaps

The first portion of this call is for computer scientists. While our work here
exposes the trade-off between accuracy and efficiency and how to engage with
it—to build systems that can prioritize application-dependent balances between
the two—it also indicates gaps in existing approaches in distributed real-time
ML systems. These gaps imply that existing systems will likely not suffice for
high-stakes, emerging applications such as autonomous vehicles. In particular,
in Section 3 we explain the importance of needing to make the accuracy-
efficiency trade-off transparent in a system’s implementation; a system’s
ability to be assessed with regard to this trade-off should be considered as
important as every other technical feature. We therefore call on computer
scientists to engage in research to build and integrate tools in ML systems
that make their accuracy-efficiency trade-offs assessable.

The 2018 Uber AV crash in Tempe, Arizona (Board, 2019) illustrates the
importance of such tools to assess the trade-off. The crash resulted from the
coincidence of several errors, but one of the most severe had the accuracy-
efficiency trade-off as its central problem. Different sensors in the AV got
conflicting readings about whether or not there was a pedestrian in front of
it; it did not resolve that inconsistency in time to safely apply the brakes.
By the time there was agreement among the sensors that a pedestrian was
present, the pedestrian had already been struck. The NTSB report on the crash
documents a variety of problems, one of which was a “lax engineering culture”
around safety at Uber. It is clear that from the sensor inconsistency and the
AV’s overall indecision that Uber had not implemented a robust resolution
strategy; the AV remained inconsistent for over 6 seconds, wrestling with the
question of whether or not a pedestrian was there.111111This example is far
more complex than what we have glossed here. For example, there was a human
back-up driver who could have engaged the brakes. However, she was not paying
attention.

The report is clear that the AV’s sensors were inconsistent, but indicates
that it is not clear _why_ the AV could not make a decision in those
conditions. In this case, a granular explanation was not necessary to
determine accountability, as 6 seconds is long time to be inconsistent. In
other words, this AV was neither accurate nor efficient; this indicates a sub-
optimal trade-off implementation—that there was a bug, as opposed to a
deliberate implementation choice, that led to a tragic outcome. However, in
instances that are not as clear-cut, tools that provide granular explanations
will be necessary to determine the difference between bugs and deliberate
trade-off choices. For accidents that involve much tighter time windows, we
need tools to help reason about the underlying accuracy-efficiency trade-off
so that we can determine whether risks were adequately assessed and accounted
for during implementation.

To enable building such tools, a potential future research direction could be
mathematically formalizing the semantics of the trade-off in ML systems. This
formalization could enable avoiding certain risks, guaranteeing _ex ante_
certain desirable system behaviors while foreclosing the possibility of
certain undesirable behaviors. For example, in some scenarios it may be
possible to reduce the tension between accuracy and efficiency by taking
coordination between computers off of the critical path for computation; this
would enable greater computational efficiencies without sacrificing accuracy
in those contexts (Hellerstein and Alvaro, 2019; Garcia-Molina, 1983; DiPippo
and Wolfe, 1997; Birman et al., 2019). As a simple example, consider again a
distributed system of autonomous cars. It is likely that not _all_ of the cars
in the system will always need to communicate with each other. Instead, it
will likely be sufficient for cars to only communicate with other cars in a
computationally-determined, environment-dependent radius. This would decrease
the amount of communication, increasing efficiency, without decreasing
accuracy, since the cars outside the determined radius would be deemed to be
too far away to have relevant information to communicate.

By providing a mechanism to reason about accuracy-efficiency trade-offs like
those above, computer scientists expose a particular kind of decisional
uncertainty that depends on time (Horvitz, 1987; Boddy and Dean, 1994).
Clarifying this uncertainty does not, however, identify specific risks that
these automated decisions bring about. Given the uncertainties involved, it is
up to policymakers to frame potential risks and to identify the normative,
domain-specific values at play (Jasanoff, 2016). That is, while computer
scientists can reason about how much error is tolerable due to the accuracy-
efficiency trade-off in order to inform their implementation decisions,
policymakers should make best efforts to assess _ex ante_ how much of the
resulting risk is tolerable.

Such _ex ante_ interventions could help narrow the space of potentially
deviant system behavior, which in turn could help narrow the number of
scenarios that need to be examined _ex post_. Moreover, these interventions,
though unlikely to be comprehensive, should clarify many of the risks at play
in deploying these systems. In some cases, in which it is possible to deem the
amount of predetermined risk to be intolerable, policymakers could disallow
particular technical systems from widespread deployment (Sachs, 2011;
Dietterich, 2018; Perrow, 1999). For example, as we discuss in Section 3 this
is likely to be the case with integrating real-time ML into police body
cameras; the risks are likely too high to deploy such technology. However, in
most cases, it may not be possible to preemptively fully analyze the risk
landscape (Sunstein, 2003; Smith, 2015) due to the amount of uncertainty in
the system (Section 2.2). Incomplete risk analyses will not necessarily
prevent the deployment of real-time ML systems in practice; instead, such
systems will get deployed and policymakers will need to evaluate their
behavior _ex post_ , after undesirable outcomes occur. An undesirable outcome
will either reveal a risk that policymakers previously did not consider, with
which they now need to contend, or, it will implicate an acknowledged risk
previously deemed acceptable.

###  4.2. Addressing _ex post_ Risk Assessment Gaps

In the case of an acknowledged, acceptable risk, it is possible that the
undesirable outcome is so extreme that it requires reevaluating what
“acceptability” means. In other words, when deployed long enough, high-stakes
ML systems are likely to incur severe harms that we might not be able to
predict _ex ante_ (Perrow, 1999; Vaughan, 1996; Nissenbaum, 1996). This is
where tools that expose the trade-off between accuracy and efficiency can
facilitate accountability after-the-fact (Lampson, 2004; Haeberlen et al.,
2007). Given that such situations are likely unavoidable, it is necessary that
engineers build tools that enable risk assessment; this way, it will be
possible to determine if a system has deviated further than expected from its
normal (what _ex ante_ risk assessment deems to be acceptable) behavior
(Sampson, 2015)—cases in which policymakers would then still be able to hold
the appropriate stakeholders accountable _ex post_.

To be clear, we are not claiming that policymakers need to understand the full
extent of low-level technical details to provide this oversight. Rather, we
are suggesting that surfacing the higher-level trade-offs that lower-level
decisions entail clarifies valid sites for potential policy intervention
(Mulligan and Bamberger, 2018; Jasanoff, 2016). One can then think of such
trade-offs as the right layer of abstraction with which policymakers can
engage. At this level, policymakers can reason about the normative values and
policy goals implicated by the trade-off in different domains (Friedman and
Hendry, 2019). The case of the accuracy-efficiency trade-off, for example, can
be used to clarify how lower-level engineering decisions relate to notions of
safety and quality (Sampson, 2015).

It is this reasoning that informs the second part of our call-to-action:
Policymakers should view the accuracy-efficiency trade-off as a regulable
decision point at which they can meaningfully intervene. As we discuss in
Section 2, policymakers already do so in other complex technical domains, for
which they reason about risk and interventions. This suggests that, with the
right tools integrated with distributed ML systems, policymakers should also
be able to do so for these systems. Importantly, we do not articulate specific
policy solutions, as such solutions will necessarily depend on the particular
application domain for the system under evaluation. Instead, we illuminate
abstract technical concepts and their import for technology policy, and we
call on policymakers to recognize that accuracy-efficiency trade-offs are a
valid and important site for accountability-oriented interventions.

It is possible to view this intervention as an extension of existing risk
assessment tools in computing. That is, contemporary policy debates about
high-stakes machine learning applications—in domains like policing, warfare,
transportation, and public health—often involve concerns about what degree of
accuracy we ought to demand from such systems. These concerns often arise in
the course of attempting to minimize disparate outcomes across groups (e.g.,
differential accuracy rates for face recognition along dimensions of race and
gender (Buolamwini and Gebru, 2018; Cooper and Abrams, 2021)). But we contend
that debates about the harms of inaccuracy are incomplete if they fail to
acknowledge and reckon with the technical trade-off between accuracy and
efficiency.

For policymakers, most of these debates will occur _ex post_ : They can use
the trade-off assessment tools built by engineers to assess the gaps between
the expected risks and the actual behavior of real-time distributed ML
systems, and use their analysis to hold these systems accountable. As a
result, we can fairly pose to policymakers questions like the following: At
what point should we deem information of sufficiently high quality to justify
the execution of potentially high-impact decisions by technical systems? When
is it safe for a system to spend more time computing inference outcomes,
particularly when more efficient heuristics do not sufficiently remove
uncertainty from automated decision-making? In other words, by giving
policymakers the tools to reason about these higher-level trade-offs, we are
able to take a step toward closing what Jasanoff (2016) terms the
“responsibility gap.” That is, policymakers will have a more sufficient
understanding of technology and will be better equipped to gauge the range of
possibilities for its governance. This way, when technological failures occur,
policymakers can _ex post_ more actively participate in the evaluation of how
uncertainty in probabilistic, automated decision systems contributes to the
construction of risk (Jasanoff, 2016).

##  5\. Conclusion: Toward More Just, Transparent Public Governance

In this paper we have made the case for using accuracy-efficiency trade-offs
as a policymaking lever for holding real-time, distributed ML systems
accountable. For example, with AVs such _ex ante_ regulation could help
constrain the space of potentially undesirable AV behavior, which in turn
could help narrow the the number of accidents and anomalous behaviors that
need to be examined _ex post_. This could lead not only to overall safer
behavior, but also the necessary tools to determine accountability when
accidents unavoidably occur (Section 4). More broadly, this discussion could
be situated in the context of extracting higher-level values from technical
systems—values such as safety and efficiency (Administration, 2016)—as a
necessary part of public governance. That is, it is crucial to analyze how
such higher-level values get implemented via underlying technological
mechanisms—in this case, the implementation of the accuracy-efficiency trade-
off—to fully appreciate and ensure that the implementation aligns with the
higher-level values that we want to promote in policy. We have argued that the
accuracy-efficiency trade-off is not only the right abstraction, but also the
right level of abstraction, for promoting this goal.

In this vein, clarifying technical details at the right level of abstraction
implicates another important value of public governance: transparency. For
example, NHTSA has generally been loath to intervene _ex ante_ with standards
for car technology; while this might make car technology development more
efficient, it can also come with the unexpected externality of loss of
transparency. Specifically, not engaging with technical details _ex ante_ can
present problems beyond not detecting bugs; it can also lead to not detecting
whether overarching values like safety and efficiency are being implemented as
desired. Worse, rather than simply going undetected, it is possible that
technical values, and the consequent social values they entail, can be
deliberately obscured. For one, technical implementation decisions can be
framed as trivial, which can direct policymakers away from viewing them as
valid sites for intervention. Alternatively, when highly technical jargon is
used to describe such implementation decisions, it can serve to obfuscate
rather than clarify. Rather than enabling transparency for policymakers, who
do not tend to be technical experts, these practices can cloud the values at
stake.

Mulligan and Bamberger (2018) have notably written about this issue of
transparency in public governance. They call out the danger of such policy-
relevant values decisions getting pushed into the low-level implementation
decisions made by engineers, in place of having the social, political, and
moral values at play being openly debated. They contend that this misplacement
of responsibility on engineers comes, at least in part, from a lack of
technical expertise in governance and a resulting lack of accountability
mechanisms to regulate technology.121212Mulligan and Bamberger (2018) mostly
limit their discussion to voting machines. In particular, they argue that
technological testing and quality control processes effectively give
manufacturers, not the public, the job of converting the law into concrete
technical requirements. In other words, manufacturers, instead of public
advocacy groups or standards-setting organizations like NHTSA, end up
responsible for making technical decisions with policy and values implications
without public input or government oversight. This conflict-of-interest can
lead to compromising or degrading those higher-level social values.

We have argued that if policymakers understand the accuracy-efficiency trade-
offs in distributed ML systems, and the social values these trade-offs
implicate, this problem can (at least in part) be averted. Policymakers will
have a more sufficient understanding of technology and will be better able to
determine the scope of possibilities for its governance (Jasanoff, 2016). By
understanding the technical values at stake at this level of abstraction,
policymakers, with engineers’ assistance, could provide insight _ex ante_ into
how certain implementation decisions should be made. That way, low-level
technical matters will not be dismissed as “just implementation details” left
up to the whims of engineers without public oversight (Mulligan and Bamberger,
2018; Jasanoff, 2016; Friedman and Hendry, 2019). Moreover, when technological
failures and accidents do occur—and it is a question of when, not if—rather
than viewing them simply as “unintended consequences” or “normal accidents”
(Perrow, 1999), policymakers and other relevant stakeholders aligned with the
broader public can more actively participate _ex post_ in holding distributed
ML systems accountable for their behavior. This more-effective public
governance of ML systems will improve the power imbalance between system
manufacturers and victims of system accidents—empowering and protecting
individuals without the resources to seek justice for themselves.

## References

  * (1)
  * Ree ([n.d.]) [n.d.].  Reeve v. Dennett, 145 Mass. 23, 28 (1887).
  * Abadi (2012) Daniel Abadi. 2012\.  Consistency Tradeoffs in Modern Distributed Database System Design: CAP is Only Part of the Story.  _Computer_ 45, 2 (Feb. 2012), 37–42.
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016\.  TensorFlow: A System for Large-Scale Machine Learning. In _Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation_ (Savannah, GA, USA) _(OSDI’16)_. USENIX Association, USA, 265–283.
  * Abebe et al. (2020) Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G. Robinson. 2020.  Roles for Computing in Social Change. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 252–260.
  * Abraham and Rabin (2019) Kenneth S. Abraham and Robert L. Rabin. 2019.  Automated Vehicles and Manufacturer Responsibility for Accidents: A New Legal Regime for a New Era.  _Virginia Law Review_ 105 (2019), 127–171.  Issue 127.
  * Administration (2016) National Highway Traffic Safety Administration. 2016.  _Federal Automated Vehicles Policy: Accelerating the Next Revolution In Roadway Safety_.  Technical Report. U.S. Department of Transportation.
  * Alistarh et al. (2018) Dan Alistarh, Christopher De Sa, and Nikola Konstantinov. 2018\.  The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory. In _Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing_. PODC ’18, Egham, United Kingdom, 169–178.
  * Amazon (2021) Amazon. 2021.  S3 Storage Classes.  https://aws.amazon.com/s3/storage-classes/
  * Arpaci-Dusseau and Arpaci-Dusseau (2018) Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau. 2018.  Operating Systems: Three Easy Pieces.
  * Bailis et al. (2012) Peter Bailis, Shivaram Venkataraman, Michael J. Franklin, Joseph M. Hellerstein, and Ion Stoica. 2012\.  Probabilistically Bounded Staleness for Practical Partial Quorums.  _Proc. VLDB Endow._ 5, 8 (April 2012), 776–787.
  * Barbara and Garcia-Molina (1990) D. Barbara and H. Garcia-Molina. 1990.  The case for controlled inconsistency in replicated data. In _[1990] Proceedings. Workshop on the Management of Replicated Data_. IEEE, Houston, TX, USA, 35–38.
  * Birman et al. (2019) Ken Birman, Bharath Hariharan, and Christopher De Sa. 2019\.  Cloud-Hosted Intelligence for Real-Time IoT Applications.  _SIGOPS Oper. Syst. Rev._ 53, 1 (July 2019), 7–13.
  * Board (2019) National Transportation Safety Board. 2019\.  Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian.
  * Boddy and Dean (1994) Mark Boddy and Thomas L. Dean. 1994.  Deliberation Scheduling for Problem Solving in Time-Constrained Environments.  _Artif. Intell._ 67, 2 (June 1994), 245–285.
  * Bottou et al. (2018) Léon Bottou, Frank E. Curtis, and Jorge Nocedal. 2018\.  Optimization Methods for Large-Scale Machine Learning.  _SIAM Rev._ 60, 2 (Jan 2018), 223–311.
  * Brewer (2012) Eric Brewer. 2012\.  CAP Twelve Years Later: How the ”Rules” Have Changed.  _Computer_ 45, 2 (2012), 23–29.
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research, Vol. 81)_ , Sorelle A. Friedler and Christo Wilson (Eds.). ACM, New York, NY, USA, 77–91.
  * Chowkwanyun et al. (2016) Merlin Chowkwanyun, D. Wolfe, J. Colgrove, R. Bayer, and A. Fairchild. 2016.  Beyond the Precautionary Principle: Protecting Public Health and the Environment in the Face of Uncertainty.  In _Bioethical Insights into Values and Policy_ , C.C. Macpherson (Ed.). Springer International Publishing, Switzerland.
  * Cooper and Abrams (2021) A. Feder Cooper and Ellen Abrams. 2021.  Emergent Unfairness in Algorithmic Fairness-Accuracy Trade-Off Research. In _AIES 2021_. AAAI, New York, NY, 9.
  * Council (1983) National Research Council. 1983\.  _Risk Assessment in the Federal Government: Managing the Process_.  The National Academies Press, Washington, DC, USA.
  * Council (1994) National Research Council. 1994\.  _Science and Judgment in Risk Assessment_.  The National Academies Press, Washington, DC, USA.
  * De Sa et al. (2017) Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun. 2017.  Understanding and Optimizing Asynchronous Low-Precision Stochastic Gradient Descent. In _Proceedings of the 44th Annual International Symposium on Computer Architecture_ _(ISCA ’17)_. Association for Computing Machinery, New York, NY, USA, 561–574.
  * DeCandia et al. (2007) Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007\.  Dynamo: Amazon’s Highly Available Key-value Store. In _Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles_ _(SOSP ’07)_. ACM, New York, NY, USA, 205–220.
  * Dietterich (2018) Thomas G. Dietterich. 2018\.  Robust artificial intelligence and robust human organizations.  _Frontiers of Computer Science_ 13, 1 (Dec 2018), 1–3.
  * DiPippo and Wolfe (1997) Lisa Cingiser DiPippo and Victor Fay Wolfe. 1997.  Object-Based Semantic Real-Time Concurrency Control with Bounded Imprecision.  _IEEE Trans. on Knowl. and Data Eng._ 9, 1 (Jan. 1997), 135–147.
  * Feurer and Hutter (2019) Matthias Feurer and Frank Hutter. 2019.  Hyperparameter Optimization.  In _Automated Machine Learning: Methods, Systems, Challenges_ , Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (Eds.). Springer International Publishing, USA, 3–33.
  * Friedman and Hendry (2019) Batya Friedman and David G. Hendry. 2019.  _Value Sensitive Design: Shaping Technology with Moral Imagination_.  The MIT Press, USA.
  * Garcia-Molina (1983) Hector Garcia-Molina. 1983\.  Using Semantic Knowledge for Transaction Processing in a Distributed Database.  _ACM Trans. Database Syst._ 8, 2 (June 1983), 28.
  * Golab et al. (2011) Wojciech Golab, Xiaozhou Li, and Mehul A. Shah. 2011.  Analyzing Consistency Properties for Fun and Profit. In _Proceedings of the 30th Annual ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing_ _(PODC ’11)_. ACM, New York, NY, USA, 197–206.
  * Gong et al. (2014) Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. 2014\.  Compressing Deep Convolutional Networks using Vector Quantization.
  * Haeberlen et al. (2007) Andreas Haeberlen, Petr Kouznetsov, and Peter Druschel. 2007\.  PeerReview: Practical Accountability for Distributed Systems.  _SIGOPS Oper. Syst. Rev._ 41, 6 (Oct. 2007), 175–188.
  * Hawkins (2017) Derrick Hawkins. 2017\.  Researchers use facial recognition tools to predict sexual orientation. LGBT groups aren’t happy.  https://www.washingtonpost.com/news/morning-mix/wp/2017/09/12/researchers-use-facial-recognition-tools-to-predict-sexuality-lgbt-groups-arent-happy/
  * Hellerstein and Alvaro (2019) Joseph M. Hellerstein and Peter Alvaro. 2019.  Keeping CALM: When Distributed Consistency is Easy.
  * Ho et al. (2013) Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B. Gibbons, Garth A Gibson, Greg Ganger, and Eric P Xing. 2013.  More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server.  In _Advances in Neural Information Processing Systems 26_ , C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., USA, 1223–1231.
  * Horvitz (1987) Eric J. Horvitz. 1987\.  Reasoning about Beliefs and Actions under Computational Resource Constraints. In _Proceedings of the Third Conference on Uncertainty in Artificial Intelligence_ _(UAI ’87)_. AUAI Press, Seattle, WA, 429–447.
  * Howard et al. (2017) Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017\.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.
  * Jasanoff (2016) Sheila Jasanoff. 2016\.  _The Ethics of Invention: Technology and the Human Future_.  New York: W.W. Norton & Company, USA.
  * Kahneman et al. (1982) Daniel Kahneman, Paul Slovic, and Amos Tversky. 1982\.  _Judgment under Uncertainty: Heuristics and Biases_.  Cambridge University Press, New York, NY, USA.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020\.  Scaling Laws for Neural Language Models.  arXiv:2001.08361 [cs.LG]
  * Kosaian et al. (2019) Jack Kosaian, K. V. Rashmi, and Shivaram Venkataraman. 2019\.  Parity Models: Erasure-Coded Resilience for Prediction Serving Systems. In _Proceedings of the 27th ACM Symposium on Operating Systems Principles_ _(SOSP ’19)_. Association for Computing Machinery, New York, NY, USA, 30–46.
  * Lampson (2004) B. W. Lampson. 2004\.  Computer Security in the Real World.  _Computer_ 37, 6 (June 2004), 37–46.
  * Lee (2020) Timothy B. Lee. 2020\.  Detroit police chief cops to 96-percent facial recognition error rate.  https://arstechnica.com/tech-policy/2020/06/detroit-police-chief-admits-facial-recognition-is-wrong-96-of-the-time
  * Leuner (2019) John Leuner. 2019\.  A Replication Study: Machine Learning Models Are Capable of Predicting Sexual Orientation From Facial Images.
  * Levy and Johns (2016) Karen EC Levy and David Merritt Johns. 2016.  When open data is a Trojan Horse: The weaponization of transparency in science and governance.  _Big Data & Society_ 3, 1 (2016), 6.
  * Li et al. (2014) Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su. 2014.  Scaling Distributed Machine Learning with the Parameter Server. In _Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation_ (Broomfield, CO) _(OSDI’14)_. USENIX Association, Berkeley, CA, USA, 583–598.
  * Lian et al. (2017) Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. 2017\.  Asynchronous Decentralized Parallel Stochastic Gradient Descent.
  * Lichtman (2002) Douglas Gary Lichtman. 2002\.  Uncertainty and the Standard for Preliminary Relief.  , 21 pages.  John M. Olin Program in Law and Economics Working Paper No. 166.
  * Lu et al. (2015) Haonan Lu, Kaushik Veeraraghavan, Philippe Ajoux, Jim Hunt, Yee Jiun Song, Wendy Tobagus, Sanjeev Kumar, and Wyatt Lloyd. 2015\.  Existential Consistency: Measuring and Understanding Consistency at Facebook. In _Proceedings of the 25th Symposium on Operating Systems Principles_ (Monterey, California) _(SOSP ’15)_. ACM, New York, NY, USA, 295–310.
  * Mittal (2016) Sparsh Mittal. 2016\.  A Survey of Techniques for Approximate Computing.  _ACM Comput. Surv._ 48, 4, Article 62 (March 2016), 33 pages.
  * Moreau et al. (2018) Thierry Moreau, Joshua San Miguel, Mark Wyse, James Bornholt, Armin Alaghi, Luis Ceze, Natalie Enright Jerger, and Adrian Sampson. 2018.  A Taxonomy of General Purpose Approximate Computing Techniques.  _IEEE Embed. Syst. Lett._ 10, 1 (March 2018), 2–5.
  * Mulligan and Bamberger (2018) D.K. Mulligan and K.A. Bamberger. 2018.  Saving governance-by-design.  _California Law Review_ 106 (06 2018), 697–784.
  * Murphy (2017) Heather Murphy. 2017\.  Why Stanford Researchers Tried to Create a ’Gaydar’ Machine.  https://www.nytimes.com/2017/10/09/science/stanford-sexual-orientation-study.html
  * Nissenbaum (1996) Helen Nissenbaum. 1996\.  Accountability in a Computerized Society.  _Science and Engineering Ethics_ 2 (1996), 25–42.
  * Niu et al. (2011) Feng Niu, Benjamin Recht, Christopher Re, and Stephen J. Wright. 2011\.  HOGWILD!: A Lock-free Approach to Parallelizing Stochastic Gradient Descent. In _Proceedings of the 24th International Conference on Neural Information Processing Systems_ (Granada, Spain) _(NIPS’11)_. Curran Associates Inc., USA, 693–701.
  * Ohm and Frankle (2019) Paul Ohm and Jonathan Frankle. 2019.  Desirable Inefficiency.  _Florida Law Review_ 70 (2019), 777–836.  Issue 4.
  * Pan et al. (2016) Xinghao Pan, Maximilian Lam, Stephen Tu, Dimitris Papailiopoulos, Ce Zhang, Michael I Jordan, Kannan Ramchandran, and Christopher Ré. 2016.  Cyclades: Conflict-free Asynchronous Machine Learning.  In _Advances in Neural Information Processing Systems 29_ , D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., USA, 2568–2576.
  * Perrow (1999) Charles Perrow. 1999\.  _Normal Accidents: Living with High Risk Technologies - Updated Edition_.  Princeton University Press, Princeton, New Jersey.
  * Rojas and Fausset (2020) Rick Rojas and Richard Fausset. 2020.  Police Killings Prompt Reassessment of Laws Allowing Deadly Force.  https://www.nytimes.com/2020/06/14/us/rayshard-brooks-Garrett-Rolfe-atlanta.html
  * Sa et al. (2016) Christopher De Sa, Chris Re, and Kunle Olukotun. 2016\.  Ensuring Rapid Mixing and Low Bias for Asynchronous Gibbs Sampling. In _Proceedings of The 33rd International Conference on Machine Learning_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.), Vol. 48. Proceedings of Machine Learning Research, New York, New York, USA, 1567–1576.
  * Sa et al. (2015) Christopher De Sa, Ce Zhang, Kunle Olukotun, and Christopher Ré. 2015.  Taming the Wild: A Unified Analysis of HOGWILD!-Style Algorithms. In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2_ _(NIPS’15)_. MIT Press, Cambridge, MA, USA, 2674–2682.
  * Sachs (2011) Noah Sachs. 2011\.  Rescuing the Strong Precautionary Principle from its Critics.  _University of Illinois Law Review_ 2011 (2011), 54.
  * Sampson (2015) Adrian Sampson. 2015\.  _Hardware and Software for Approximate Computing_.  Ph.D. Dissertation. University of Washington.  https://www.cs.cornell.edu/~asampson/media/dissertation.pdf
  * Shang et al. (2018) Zechao Shang, Jeffrey Xu Yu, and Aaron J. Elmore. 2018\.  RushMon: Real-time Isolation Anomalies Monitoring. In _Proceedings of the 2018 International Conference on Management of Data_ (Houston, TX, USA) _(SIGMOD ’18)_. ACM, New York, NY, USA, 647–662.
  * Smith (2015) Henry E. Smith. 2015\.  Equity as Second-Order Law: The Problem of Opportunism.
  * Sunstein (2002) Cass R. Sunstein. 2002\.  Hazardous Heuristics.
  * Sunstein (2003) Cass R. Sunstein. 2003\.  Beyond the Precautionary Principle.
  * Tsitsiklis et al. (1986) J. Tsitsiklis, D. Bertsekas, and M. Athans. 1986\.  Distributed asynchronous deterministic and stochastic gradient optimization algorithms.  _IEEE Trans. Automat. Control_ 31, 9 (1986), 803–812.
  * Vaughan (1996) Diane Vaughan. 1996\.  _The Challenger launch Decision: Risky Technology, Culture, and Deviance at NASA_.  University of Chicago Press, Chicago, IL, USA.
  * Vogels (2009) Werner Vogels. 2009\.  Eventually Consistent.  _Commun. ACM_ 52, 1 (Jan. 2009), 40–44.
  * von Clausewitz (1832) Carl von Clausewitz. 1832\.  Vom Kriege.
  * Wang (2018) Amy B. Wang. 2018\.  A suspect tried to blend in with 60,000 concertgoers. China’s facial-recognition cameras caught him.  https://www.washingtonpost.com/news/worldviews/wp/2018/04/13/china-crime-facial-recognition-cameras-catch-suspect-at-concert-with-60000-people/
  * Wang et al. (2019) S. Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan. 2019.  Adaptive Federated Learning in Resource Constrained Edge Computing Systems.  _IEEE Journal on Selected Areas in Communications_ 37, 6 (2019), 1205–1221.
  * Wang and Kosinski (2018) Yilun Wang and Michal Kosinski. 2018.  Deep neural networks are more accurate than humans at detecting sexual orientation from facial images.  , 246–257 pages.
  * Weihl (1988) W. E. Weihl. 1988\.  Commutativity-based concurrency control for abstract data types.  _IEEE Trans. Comput._ 37, 12 (1988), 1488–1505.
  * Weinstein et al. (2020) MC Weinstein, KA Freedberg, EP Hyle, and AD Paltiel. 2020\.  Waiting for Certainty on Covid-19 Antibody Tests - At What Cost?
  * Yu and Vahdat (2000) Haifeng Yu and Amin Vahdat. 2000.  Efficient Numerical Error Bounding for Replicated Network Services. In _Proceedings of the 26th International Conference on Very Large Data Bases_ _(VLDB ’00)_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 123–133.
  * Zhang et al. (2015) Wei Zhang, Suyog Gupta, Xiangru Lian, and Ji Liu. 2015\.  Staleness-aware Async-SGD for Distributed Deep Learning.