# Evaluation Gaps in Machine Learning Practice

Ben Hutchinson  Google ResearchSydneyAustralia<EMAIL_ADDRESS>,  Negar
Rostamzadeh  Google ResearchMontrealCanada<EMAIL_ADDRESS>,
Christina Greer  Google ResearchMountain ViewUSA<EMAIL_ADDRESS>,  Katherine
Heller  Google ResearchMountain ViewUSA<EMAIL_ADDRESS>and  Vinodkumar
Prabhakaran  Google ResearchSan FranciscoUSA<EMAIL_ADDRESS>
(2022)

###### Abstract.

Forming a reliable judgement of a machine learning (ML) model’s
appropriateness for an application ecosystem is critical for its responsible
use, and requires considering a broad range of factors including harms,
benefits, and responsibilities. In practice, however, evaluations of ML models
frequently focus on only a narrow range of decontextualized predictive
behaviours. We examine the evaluation gaps between the idealized breadth of
evaluation concerns and the observed narrow focus of actual evaluations.
Through an empirical study of papers from recent high-profile conferences in
the Computer Vision and Natural Language Processing communities, we
demonstrate a general focus on a handful of evaluation methods. By considering
the metrics and test data distributions used in these methods, we draw
attention to which properties of models are centered in the field, revealing
the properties that are frequently neglected or sidelined during evaluation.
By studying these properties, we demonstrate the machine learning discipline’s
implicit assumption of a range of commitments which have normative impacts;
these include commitments to consequentialism, abstractability from context,
the quantifiability of impacts, the limited role of model inputs in
evaluation, and the equivalence of different failure modes. Shedding light on
these assumptions enables us to question their appropriateness for ML system
contexts, pointing the way towards more contextualized evaluation
methodologies for robustly examining the trustworthiness of ML models.

machine learning, evaluation, applications

††journalyear: 2022††copyright: rightsretained††conference: 2022 ACM
Conference on Fairness, Accountability, and Transparency; June 21–24, 2022;
Seoul, Republic of Korea††booktitle: 2022 ACM Conference on Fairness,
Accountability, and Transparency (FAccT ’22), June 21–24, 2022, Seoul,
Republic of Korea††doi: 10.1145/3531146.3533233††isbn:
978-1-4503-9352-2/22/06††ccs: Computing methodologies Machine learning
approaches

##  1\. Introduction

When evaluating a machine learning (ML) model for real-world uses, two
fundamental questions arise: _Is this ML model good (enough)?_ and _Is this ML
model better than some alternative?_ Obtaining reliable answers to these
questions can be consequential for safety, fairness, and justice concerns in
the deployment ecosystems. To address such questions, model evaluations use a
variety of methods, and in doing so make technical and normative assumptions
that are not always explicit. These implicit assumptions can obscure the
presence of epistemic gaps and motivations in the model evaluations, which, if
not identified, constitute risky _unknown unknowns_.

Recent scholarship has critiqued the ML community’s evaluation practices,
focusing on the use of evaluation benchmarks and leaderboards. Although
leaderboards support the need of the discipline to iteratively optimize for
accuracy, they neglect concerns such as inference latency, robustness, and
externalities (Ethayarajh and Jurafsky, 2020). The structural incentives of
the “competition mindset” encouraged by leaderboards can pose challenges to
empirical rigor (Sculley et al., 2018). For example, over-reliance on a small
number of evaluation metrics can lead to gaming the metric (cf. Goodhart’s Law
“when a measure becomes a target, it ceases to be a good measure”) (Thomas and
Uminsky, 2020); this can happen unintentionally as researchers pursue models
with “state of the art” performance. Benchmarks that encourage narrowly
optimizing for test set accuracy can also lead to models relying on spurious
signals (Carter et al., 2021), while neglecting the challenge of measuring the
full range of likely harms (Bowman and Dahl, 2021). Birhane et al. find
evidence for this in their study of the discourse of ML papers, showing that
the field centers accuracy, generalization, and novelty, while marginalizing
values such as safety (Birhane et al., 2021). Given that benchmark evaluations
serve as proxies for performance on underlying abstract tasks (Schlangen,
2021), evaluating against a range of diverse benchmarks for each task might
help mitigate biases within each benchmark. However, ML research disciplines
seem to be trending towards relying on fewer evaluation benchmark datasets
(Koch et al., 2021), with test set reuse potentially leading to a research
community’s overfitting with respect to the general task (Zhang et al., 2020a;
Liao et al., 2021). Furthermore, within each benchmark, items are weighted
equally (thus focusing on the head of the data distribution), failing to
capture inherent differences in difficulty across items, and hence providing
poor measures of progress on task performance (Rodriguez et al., 2021). As
Raji et al. point out, the ML research discipline’s decontextualized and non-
systematic use of benchmark data raises serious issues with regards to the
validity of benchmarks as measures of progress on general task performance
(Raji et al., 2021).

This paper complements and extends this range of critiques, considering the
risks of application developers adopting the ML research community’s standard
evaluation methodologies. We seek to address challenges in measuring
technology readiness (tram) (Lin et al., 2007; Rismani and Moon, 2021), while
acknowledging this cannot be reduced to a purely technical question (Dahlin,
2021; Rismani and Moon, 2021). By studying and analyzing the ML research
community’s evaluation practices, we draw attention to the _evaluation gaps_
between ideal theories of evaluation and what is observed in ML research. By
considering aspects of _evaluation data_ and _evaluation metrics_ —as well as
considerations of _evaluation practices_ such as error analysis and reporting
of error bars—we highlight the discrepancies between the model quality signals
reported by the research community and what is relevant to real-world model
use. Our framework for analyzing the gaps builds upon and complements other
streams of work on ML evaluation practices, including addressing distribution
shifts between development data and application data (Sugiyama et al., 2007;
Koh et al., 2020; Chen et al., 2021), and robustness to perturbations in test
items (Prabhakaran et al., 2019; Moradi and Samwald, 2021; Winkens et al.,
2020; hendrycks2019benchmarking). We situate this work alongside studies of
the appropriateness of ML evaluation metrics (e.g., (Japkowicz, 2006;
Derczynski, 2016; Zhang et al., 2020a)), noting that reliable choice of metric
is often hampered by unclear goals (Kuwajima et al., 2020; D’Amour et al.,
2020). In foregrounding the information needs of application developers, we
are also aligned with calls for transparent reporting of ML model evaluations
(Mitchell et al., 2019), prioritizing needs of ML fairness practitioners
(Holstein et al., 2019), model auditing practices (Raji et al., 2020), and
robust practices for evaluating ML systems for production readiness (Breck et
al., 2017).

In Section 2, we consider various ideal goals that motivate why ML models are
evaluated, discussing how these goals can differ between research contexts and
application contexts. We then report in Section 3 on an empirical study into
how machine learning research communities report model evaluations. By
comparing the ideal goals of evaluation with the observed evaluation trends in
our study, we highlight in Section 4 the evaluation gaps that present
challenges to evaluations being good proxies for what application developers
really care about. We identify six implicit evaluation assumptions that could
account for the presence of these gaps. Finally, in Section 5, we discuss
various techniques and methodologies that may help to mitigate these gaps.

##  2\. Ideals of ML Model Evaluation

> _Far better an approximate answer to the right question, which is often
> vague, than an exact answer to the wrong question, which can always be made
> precise._ — John Tukey (Tukey, 1962, pp. 13–14)

Although this paper is ultimately concerned with practical information needs
when evaluating ML models for use in applications, it is useful to first step
back and consider the ultimate motivations and goals of model evaluation. To
evaluate is to form a judgement; however, asking _Is this a good ML model?_ is
akin to asking such a question of other artefacts—such as _Is this a good
glass?_ —in that it requires acknowledging the implicit semantic arguments of
uses and goals (Pustejovsky, 1998). For example, _Is this a good glass [for my
toddler to drink from, given that I want to avoid broken glass]?_ is a very
different question from _Is this a good glass [in which to serve wine to my
boss, given that I want to impress them]?_

In this paper, we will speak of a _model evaluation_ as a system of arbitrary
structure that takes a model as an input and produces outputs of some form to
judge the model. Designing a model evaluation often involves choosing one or
more evaluation metrics (such as accuracy) combined with a choice of test
data. The evaluation might be motivated by various stakeholder perspectives
and interests (Jones and Galliers, 1995). The output might, for example,
produce a single metric and an associated numeric value, or a table of such
metrics and values; it might include confidence intervals and significance
tests on metric values; and it might include text. By producing such an
output, the evaluation helps to enable transparency by reducing the number of
both unknown unknowns and known unknowns.

For the purposes of this paper, it is useful to distinguish between two types
of evaluations:

  1. _Learner-centric_.

An ML model evaluation system useful for evaluating the learner (i.e., machine
learning algorithm).

  2. _Application-centric_.

An ML model evaluation system useful for evaluating a potential application.

Learner-centric evaluations make conclusions about the quality of the learner
or its environment based on the evaluation of the learned model. These
including evaluations motivated by novel learning algorithms or model
architectures, but also ones that a) aim to shed light on the training data
(for example ML model evaluations can shed light on the data-generation
practices used by institutions (Andrus and Gilbert, 2019)), or b) “Green AI”
explorations of how the learner can efficiently use limited amounts of
resources (Schwartz et al., 2020). However, when we evaluate a model without a
specific application in mind, we lose the opportunity to form judgements
specific to a use case. On the other hand, application-centric evaluations are
concerned with how the model will operate within an ecosystem consisting of
both human agents and technical components (Figure 1), sometimes described as
the “ecological validity” (De Vries et al., 2020). Applications often use
scores output by the model to initiate discrete actions or decisions, by
applying a specific classification threshold to the scores.111The history of
this type of use case extends beyond ML models, e.g., to the use of regression
models in university admissions testing (Hutchinson and Mitchell, 2019). In
contrast, learner-centric evaluations sometimes care about scores output by
models even in the absence of any thresholds.

Figure 1. Learner-centric ML model evaluations are concerned with the learner
and its environment. Application-centric model evaluations are concerned with
how the model will interact with an ecosystem into which it is introduced.

This distinction between learner-centric and application-centric is related
(albeit imperfectly) to the different objectives of model evaluations that
concern the engineering and science disciplines (Wallach, 2018; Mazzocchi,
2015). Note that we are not claiming (cf. the debate in (Norvig, 2017)) that
science lies outside the bounds of statistical/ML methods, but rather that
scientific-flavored pursuits have distinct uses of such methods (Breiman,
2001). Debates between AI practitioners about the relationships between AI,
science, and statistical methods have a long history, for example Diana
Forsythe’s studies of 1980s AI labs (Forsythe, 2001a). Important to this
debate regarding the scientific goals of ML is the question of construct
validity; that is, whether our measurements actually measure the things that
we claim they do (Jacobs et al., 2020; Raji et al., 2021; Jacobs and Wallach,
2021). Conversely, consequential validity—which includes the real-world
consequences of an evaluation’s interpretation and use—is likely more
important to considerations of accountability and governance of ML models in
applications (Jacobs and Wallach, 2021).

  1. _Scientific goal_.

Evaluating the model can motivate beliefs/explanations about the world
(including possibly the learner).

  2. _Engineering goal_.

Evaluating the model can tell us whether the model can be used as a means
towards a goal.

This distinction is closely related to one between “scientific testing” and
“competitive testing” made by Hooker in 1995, who takes the position that
competitive testing a) is unscientific, and b) does not constitute true
research but merely development (Hooker, 1995). However, since engineering
research has its own goals, distinct from those of science (Bulleit et al.,
2015), a more defensible position is that evaluations in support of scientific
research are distinct from evaluations in support of engineering research.

| Learner-centric evaluations | Application-centric evaluations
---|---|---
Typical evaluation goal | Distinguish better learners from poorer ones | Predict ecosystem outcomes
Schematic of goal | $Understand(Learner)$ | $Understand(Ecosystem+Model)$
Disciplinary goals | Science or engineering | Primarily engineering
Table 1. Summary of typical goals of the idealized learner-centric and
application-centric evaluations.

Table 1 summarizes the above distinctions and the relationships between them.
The distinction between learner-centric and application-centric evaluations
relates to the question of internal validity and external validity that is
more commonly discussed in the social sciences than in ML (see, e.g., (Olteanu
et al., 2019)) but also sometimes in ML (Liao et al., 2021). This is reflected
in the ways in which practitioners of the two types of evaluations discuss the
topic of robustness. Learner-centric evaluations pay attention to the
robustness of the learner to changes in the training data (e.g.,
distributional shifts, outliers, perturbations, poisoning attacks; and with
connections to robust estimation of statistics (Lecué and Lerasle, 2020)),
while application-centric evaluations pay attention to desired behaviors such
as the (in)sensitivity of the model to certain classes of perturbations of the
input, or to sensitive input features (e.g., (Garg et al., 2019)).

Note that nothing in the ideals of evaluation described above has stipulated
whether evaluations are _quantitative_ or _qualitative_. For example, one
could imagine interrogating a chatbot model using qualitative techniques, or
adopting methodologies of political critique such as (Crawford and Paglen,
2021). Similarly, nothing has stipulated what combinations of empirical or
deductive methods are used.

##  3\. ML Model Evaluations in Practice

> _Beneath the technical issues lie some differences in _values_ concerning
> not only the meaning but also the relative merit of “science” and
> “artificial intelligence.”_ — Diana Forsythe (Forsythe, 2001a)

To shed light on the ML research community’s norms and values around model
evaluation, we looked at how these communities report their model evaluations.
By examining 200 papers from several top conferences in two research
disciplines that use ML approaches extensively, we identified patterns
regarding choices of metrics, evaluation data, and measurement practices. This
empirical study of ML research practices complements several recent studies of
ML evaluation practices. These include: a survey 144 research papers studying
the properties of models that are tested for (Zhang et al., 2020a); a review
of 107 papers from Computer Vision (CV), Natural Language Processing (NLP) and
other ML disciplines to diagnose internal and external modes of evaluation
failures (Liao et al., 2021); an analysis of whether 60 NLP and CV papers pay
attention to accuracy or efficiency (Schwartz et al., 2020); and an analysis
of the Papers With Code dataset222https://paperswithcode.com for patterns of
benchmark dataset creation and re-use (Koch et al., 2021).

###  3.1. Method

####  3.1.1. Data

We sampled 200 research papers, stratified by discipline, conference and year.
100 papers were selected from each of the NLP and CV disciplines. We selected
20 papers from the proceedings of each of the 55th to 59th Annual Meetings of
the Association of Computational Linguistics (ACL’2017–ACL’2021), 25 papers at
random from each of the proceedings of the 2019–2021 IEEE Conferences on
Computer Vision and Pattern Recognition (CVPR’2019–CVPR’2021), and 25 papers
from the 24th International Conference on Medical Image Computing and Computer
Assisted Intervention (MICCAI’2021). These conferences represent the pinnacles
of their respective research fields.333ACL and CVPR are rated A∗ (“flagship
conference”), and MICCAI is rated A (“excellent conference”), by core.edu.au;
all three are in the top 30 computer science conferences out of over 900
listed on research.com.

####  3.1.2. Analysis

The authors of this paper performed this analysis, dividing the papers among
themselves based on disciplinary familiarity. Using an iterative procedure of
analysis and discussion, we converged on a set of labels that captured
important aspects of evaluations across and within disciplines. Recall from
Section 2 that, for our purposes, a single evaluation typically involves
choosing one or more metrics and one or more datasets. We coded each of the
papers along three dimensions. a) Metrics: Which evaluation metrics were
reported? After iteration, we converged on the categories of metrics shown in
Table 2. b) Data: Was test data drawn from the same distribution as the
training data, under the Independent and Identically Distributed (I.I.D.)
assumption? c) Analysis: Was statistical significance of differences reported?
Were error bars and/or confidence intervals reported? Was error analysis
performed? Were examples of model performance provided to complement
measurements with qualitative information?

###  3.2. Results

Although each of the disciplines and conferences does not define itself solely
in terms of ML, the practice of reporting one or more model evaluations in a
research paper is ubiquitous. Only five papers did not include evaluations of
ML models; of these two were published at ACL (a survey paper, a paper aimed
at understanding linguistic features, and one on spanning-tree algorithms),
and two at CVPR (a paper with only qualitative results, and one introducing a
dataset). Table 3 summarizes the results of the other 195. Counts are non-
exclusive, for example papers frequently reported multiple metrics and
sometimes reported performance both on I.I.D. test data and on non-I.I.D. test
data.

Appendix B contains an overview of the flavors of test data we observed. We
found evidence to support the claim that evaluations of NLP models have
“historically involved reporting the performance (generally meaning the
accuracy) of the model on a specific held-out [i.e., I.I.D.] test set”
(Bommasani et al., 2021, p. 94).444Two observed non-I.I.D. evaluation patterns
in NLP were: a) testing on a different linguistic “domain” (e.g., training on
texts about earthquakes and testing on texts about floods (Alam et al.,
2018)); and b) testing a model’s ability to predict properties of a manually
compiled lexical resource (e.g., (Ustalov et al., 2017)). See also Appendix B.
CV evaluations seem to be even more likely to utilize I.I.D. test data,
and—consistent with (Koch et al., 2021)—CV papers typically either introduce a
new task (and corresponding benchmark dataset) (Jafarian and Park, 2021; Li et
al., 2016; Rostamzadeh et al., 2018; Wu et al., 2021) or present results of a
new model on an existing widely-used benchmark (Ren et al., 2015; He et al.,
2016). An exception to this trend was CV papers which explored shared
representations (e.g., in multi-task learning (Lacoste et al., 2018; Evci et
al., 2021) or domain adaptation (Pinheiro et al., 2019; Murez et al., 2018)).

Evaluations in both disciplines showed a heavy reliance on reporting point
estimates of metrics, with variance or error bars typically not reported in
our sample. While colloquial uses of phrases like “significantly better” were
fairly common, most papers did not report on technical calculations of
statistical differences; we considered only those latter instances when coding
whether a paper reported significance. Regarding metrics, most of those that
were frequently seen in our sample were somewhat insensitive to different
types of errors. For example, accuracy does not distinguish between FP and FN;
$F_{1}$ is symmetric in FP and FN (they can be swapped without affecting
$F_{1}$); the Overlap metrics are similary invariant to swapping of the
predicted bounding box and the reference bounding box; the Distance category
of metrics does not distinguish over-estimation from under-estimation on
regression tasks.

From our reading of the 200 papers in our sample, one qualitative observation
we had was that model evaluations typically do not include concrete examples
of model behavior, nor analyses of errors (for a counterexample which includes
these practices, see (Chiril et al., 2020)). Also, we noted the scarcity of
papers whose sole contribution is a new dataset for an existing task, aligning
with previous observations that dataset contributions are not valued highly
within the community (Sambasivan et al., 2021b). We hypothesise that
conference reviewers place emphasis on novelty of model, task, and/or metric.
We note a general tension between disciplinary values of task novelty and
demonstrating state-of-the-art performance by outperforming previous models,
and the risk of overfitting from test set re-use discussed by (Liao et al.,
2021).

Metric category |  Examples |  Description
---|---|---
Accuracy |  Accuracy, error rate |  Sensitive to the sum TP+TN and to N. Not sensitive to class imbalance.
Precision |  Precision, Bleu |  Sensitive to TP and FP. Not sensitive to FN or TN.
Recall |  Recall, Rouge |  Sensitive to TP and FN. Not sensitive to FP or TN.
F-score |  $F_{1}$, $F_{\beta}$ |  Sensitive to TP, FP and FN. Not sensitive to TN.
Overlap |  Dice, IoU |  Sensitive to intersection and overlap of predicted and actual.
Likelihood |  Perplexity |  Sensitive to the probability that the model assigns to the test data.
Distance |  MSE, MAE, RMSE, CD |  Sensitive to the distance between the prediction and the actual value.
Correlation |  Pearsons $r$, Spearman’s $\rho$ |  Sensitive to each of TP, TN, FP and FN, but unlike Accuracy metrics they factor in the degree of agreement that would be expected by chance.
AUC |  MAP, AUROC |  Does not rely on a specific classification threshold, but instead calculates the area under a curve parameterized by different thresholds.
Table 2. Categories of evaluation metrics used in the analysis of the ML research literature. TP=true positives; TN=true negatives; FP=false positives; FN=false negatives; N=total number of data points. See Appendix A for the most common metrics in our data and their categorizations. Discipline:Venue |  NLP:ACL |  CV:CVPR |  CV:MICCAI |  CV:Combined |  NLP+CV:Combined
---|---|---|---|---|---
(# papers with ML evals) |  (97) |  (73) |  (25) |  (98) |  (195)
Most Common Metrics |  |  |  |  |
Metric category♣ (num. of papers) |  Accuracy (47) F-score (45) Precision (43) Recall (25) |  AUC (32) Accuracy (25) Overlap (22) Distance (10) |  Distance (14) Overlap (9) AUC (6) Accuracy (4) |  AUC (38) Overlap (31) Accuracy (29) Distance (24) |  Accuracy (76) F-score + Overlap♠ (74) Precision (48) AUC (44)
Data |  |  |  |  |
I.I.D. test data |  78 |  72 |  25 |  97 |  175
Non-I.I.D. test data |  28 |  21 |  4 |  25 |  53
Analysis |  |  |  |  |
Reports significance |  24 |  0 |  7 |  7 |  31
Reports error bars♢ |  10 |  6 |  10 |  16 |  26
Table 3. Analysis of how Natural Language Processing (NLP) and Computer Vision
(CV) research communities perform ML model evaluations. ♣Appendix A provides
definitions of commonly observed metrics, and their mappings to categories.
♢Includes any form of error bars/confidence intervals/credible
intervals/variation across multiple runs. ♠Reported together here due to the
equivalence of the Dice measure (in the Overlap category) and $F_{1}$ (in the
$F$-score category) (Powers, 2011).

###  3.3. Discussion

This small-scale quantitative study of model evaluations provides clues as to
the values and goals of the ML research communities. Test data was often old
(e.g., the CONLL 2003 English NER dataset (Sang and De Meulder, 2003) used in
two papers); optimizing for these static test sets fails to account for
societal and linguistic change (Bender et al., 2021). Disaggregation of
metrics was rare, and fairness analyses were absent despite our sample being
from 2017 onward, concurrent with mainstream awareness of ML fairness
concerns. Despite being acknowledged by influential thought-leaders in ML to
be unrealistic for applications (Bengio et al., 2021), using I.I.D. test data
is the norm. These are in alignment with the learner-centric goals of
evaluations (Section 2). Similarly, with a few exceptions in our sample, there
was general paucity of discussions of tradeoffs such as accuracy vs resource-
efficiency that are typical of engineering disciplines (Bulleit et al., 2015),
suggesting that the ML research disciplines generally aspire to scientific
goals concerning understanding and explaining the learner. With this lens, the
disciplinary paradigm of measuring accuracy on I.I.D. test data is not
surprising: the goal is to assess a model’s ability to generalize. This
assessment would then give us good guarantees on the application’s behavior,
if the practical challenges of ascertaining the data distributions in an
application ecosystem can be overcome. In practice, however, these challenges
can be severe, and the research papers we surveyed do not generally tackle
questions of uncertainty regarding data distributions.

##  4\. Gaps and Assumptions in Common Evaluation Practices

> _In theory there is no difference between theory and practice, while in
> practice there is._ — Brewster (1881) (Brewster, 1881)

We now consider whether the research evaluation practices observed in Section
3 are aligned with the needs of decision-makers who consider whether to use a
model in an application. That is, we consider whether the typically learner-
centric evaluations, which commonly use metrics such as accuracy or $F_{1}$ on
test data I.I.D. with the training data, meet the need of application-centric
evaluations. In doing so, we expose, in a novel way, the interplay of
technical and normative considerations in model evaluation methodologies.

###  4.1. Assumptions in Model Evaluation

We introduce six assumptions in turn, describing both how they operate
individually in evaluations and how they compose and compound. We also call
out “evaluation gaps” of concern relevant to each assumption. Appendix C
contains a hypothetical example from a specific application domain that
illustrates the flavors of the concerns. Our starting point is the observation
from Section 2 that the goal of application-centric model evaluations is to
understand how a model will interact with its ecosystem, which we denote
schematically as:

(1) $\qquad Understand(Model+Ecosystem)\hfill$

Assumption 1: Consequentialism. Consequentialism is the view that whether
actions are good or bad depends only on their consequences (Sinnott-Armstrong,
2021). The ML research literature often appeals to motivations about model
utility to humans (e.g., (Blasi et al., 2021; Ethayarajh and Jurafsky, 2020;
Bunescu and Huang, 2010; Lo and Wu, 2010; Idahl et al., 2021; Neumann et al.,
2020; Hepp et al., 2018; Orekondy et al., 2018; Fu et al., 2022; Zhao et al.,
2020), including papers on fairness in ML such as (Corbett-Davies et al.,
2017; Card and Smith, 2020; Chohlas-Wood et al., 2021; Corbett-Davies and
Goel, 2018)). In adopting consequentialism as its _de facto_ ethical
framework, ML prioritizes the greatest good for the greatest number (IEEE,
2019) and centers measurable future impacts. Moreover, the consequences that
are centered are the _direct_ consequences, with little attention given to
motives, rules, or public acceptance (Sinnott-Armstrong, 2021). This is
realised as a focus on the first-order consequences of introducing the model
into the ecosystem. Changes to the ecosystem itself—e.g., addressing what
social change is perceived as possible and desirable (Green, 2020; Hovy and
Spruit, 2016; Eckhouse et al., 2019)—are assumed to be out of scope, as are
concerns for setting of precedents for other ML developers. We denote this
assumption schematically as:

(2) $\qquad Understand(Model+Ecosystem)\approx
Utility(Model+Ecosystem)\hfill\hfill$

Evaluation Gap 1: Provenance. A focus on future consequences neglects
important moral considerations regarding the construction of the model. This
excludes both deontological concerns—for example, _Were data consent and
sovereignty handled appropriately?_ (Kukutai and Taylor, 2016; Andreotta et
al., 2021; Crawford and Paglen, 2021) and _Were data workers treated with
dignity?_ (Gray and Suri, 2019)—as well as questions regarding past costs of
development—for example, _What were the energy use externalities of model
training?_ (Strubell et al., 2019; García-Martín et al., 2019; Crawford and
Joler, 2018) and _Was the labour paid fairly?_ (Silberman et al., 2018).
Schwartz et al. coin the phrase “Red AI” to describe ML work that disregards
the costs of training, noting that such work inhibits discussions of when
costs might outweigh benefits (Schwartz et al., 2020).

Evaluation Gap 2: Social Responsibilities. Another outcome of focusing
primarily on direct consequences is marginalizing the assessment of a model
against the social contracts that guide the ecosystem in which the model is
used, such as moral values, principles, laws, and social expectations. For
instance, Does the model adhere to the moral duty to treat people in ways that
upholds their basic human rights? (Shue, 2020), Does it abide by legal
mechanisms of accountability? (Raso et al., 2018; McGregor et al., 2019), and
Does it satisfy social expectations of inclusion, such as the “nothing about
us without us” principle? (Charlton, 1998).

Assumption 2: Abstractability from Context. The model’s ecosystem is reduced
to a set of considerations $(X,Y)$, i.e., the inputs to the model and the
“ground truth,” and in practice $X$ may often fail to model socially important
yet sensitive aspects of the environment (Barocas et al., 2017; Andrus and
Gilbert, 2019). The model itself is reduced to a predicted value $\hat{Y}$,
ignoring e.g., secondary model outputs such as confidence scores, or
predictions on auxiliary model heads.

(3) $\qquad Utility(Model+Ecosystem)\approx Utility(\hat{Y},X,Y)\hfill$

Evaluation Gap 3: System Considerations. Equating a model with its prediction
overlooks the potential usefulness of model interpretability and
explainability. Also, reducing an ecosystem to model inputs and “ground truth”
overlooks questions of system dynamics (Martin et al., 2020; Selbst et al.,
2019), such as feedback loops, “humans-in-the-loop,” and other effects “due to
actions of various agents changing the world” (Bengio et al., 2021). Also
overlooked are inference-time externalities of energy use (Cai et al., 2017;
García-Martín et al., 2019), cultural aspects of the ecosystem (Sambasivan et
al., 2021a), and long term impacts (Card and Smith, 2020).

Evaluation Gap 4: Interpretive Epistemics. By positing a variable $Y=y$ which
represents the “ground truth” of a situation—even in situations involving
social phenomena—a positivist stance on knowledge is implicitly adopted. That
is, a “true” value $Y=y$ is taken to be objectively singular and knowable.
This contrasts with anthropology’s understanding of knowledge as socially and
culturally dependent (Forsythe, 2001b) and requiring interpretation (Geertz,
1973). In the specific cases of CV and NLP discussed in Section 3, cultural
aspects of image and language interpretation are typically marginalized (cf.
(Jappy, 2013; Lakoff and Johnson, 2008; Berger, 2008; Barthes, 1977), for
example), exemplifying what Aroyo and Welty call AI’s myth of “One Truth”
(Aroyo and Welty, 2015). Furthermore, the positivist stance downplays the
importance of questions of construct validity and reliability (Jacobs and
Wallach, 2021; Friedler et al., 2021).

Figure 2. Causal graph illustrating the Input Myopia Assumption.

Assumption 3: Input Myopia. Once the input variable $X$ has been used by the
model to calculate the model prediction $\hat{Y}$, $X$ is typically ignored
for the remainder of the evaluation. That is, the utility of the model is
assumed to depend only on the model’s prediction and on the “ground truth.” We
illustrate this with a causal graph diagram in Figure 2, which shows Utility
as independent of $X$ once the effects of $\hat{Y}$ and $Y$ are taken into
account.

(4) $\qquad Utility(\hat{Y},X,Y)\approx Utility(\hat{Y},Y)\hfill$

Evaluation Gap 5: Disaggregated Analyses. By reducing the variables of
interest to the evaluation to the prediction $Y$ and the ground truth
$\hat{Y}$, the downstream evaluation is denied the potential to use $X$. This
exacerbates Evaluation Gap 3 by further abstracting the evaluation statistics
from their contexts. For example, $X$ could have been used to disaggregate the
evaluation statistics in various dimensions—including for fairness analyses,
assuming that socio-demographic data is available and appropriate (Andrus et
al., 2021; Barocas et al., 2021)—or to examine regions of the input space
which raise critical safety concerns (e.g., distinguishing a computer vision
model’s failure to recognise a pedestrian on the sidewalk from failure to
recognise one crossing the road) (Amodei et al., 2016). Similarly, robustness
analyses which compare the model predictions for related inputs in the same
neighborhood of the input space are also excluded.

Assumption 4: Quantifiability. We have not yet described any modeling
assumptions about the mathematical or topological nature of the implied
$Utility$ function, which up to now has been conceived as an arbitrary
procedure producing an arbitrary output. We observe, however, that when models
are evaluated, there is a social desire to produce a small number of scalar
scores. This is reinforced by “leaderboardism” (Ethayarajh and Jurafsky,
2020), and extends to the point of averaging different types of scores such as
correlation and accuracy (Wang et al., 2018). We identify two assumptions
here: first, that impacts on each individual can be reduced to a single
numeric value (and thus different dimensions of impacts are
commensurable555E.g., one machine learning fairness paper says “$c$ is the
cost of detention in units of crime prevented” (Corbett-Davies et al.,
2017).); second, that impacts across individuals are similarly commensurable.
We define $\hat{y}\in\hat{Y}$ and $y\in Y$ to be a specific model prediction,
and a specific ”ground truth” value respectively, leading to the Individual
Quantifiability Assumption and the Collective Quantifiability Assumption,
respectively.

(5) $\qquad Utility(\hat{y},y)\in\mathbb{R}\hfill$ (6) $\qquad
Utility(\hat{Y},Y)\approx E_{(\hat{y},y)}\\{Utility(\hat{y},y)\\}\hfill$

Composing these assumptions with the previous ones leads to the belief that
the evaluation can be summarized as a scalar statistic:
$Utility(\hat{Y},Y)\in\mathbb{R}$.

Evaluation Gap 6: Incommensurables. The Quantifiability Assumptions assume
that the impacts on individuals are reducible to numbers, trivializing the
frequent difficulty in comparing different benefits and costs (Marrkula
Center, 2019). Furthermore, the harms and benefits across individuals are
assumed to be comparable in the same scale. These assumptions are likely to
disproportionately impact underrepresented groups, for whom model impacts
might differ in qualitative ways from the well represented groups (Sambasivan
et al., 2021a; Sambasivan and Holbrook, 2018; Heldreth et al., 2021). The
former groups are less likely to be represented in the ML team (West et al.,
2019) and hence less likely to have their standpoints on harms and benefits
acknowledged.

Assumption 5: Failures Cases Are Equivalent. For classification tasks, common
evaluation metrics such as accuracy or error rate model the utility of
$\hat{Y}$ as binary (i.e., either 1 or 0), depending entirely on whether or
not it is equal to the “ground truth” $Y$. That is, for a binary task,
$Utility(\hat{Y}$=$0,Y$=$0)$=$Utility(\hat{Y}$=$1,Y$=$1)$=$1$ and
$Utility(\hat{Y}$=$0,Y$=$1)$=$Utility(\hat{Y}$=$1,Y$=$0)$=$0$. Similarly for
regression tasks, common metrics such as MAE and MSE take the magnitude of
error into account, yet still treat certain failures as equivalent
(specifically,
$Utility(\hat{Y}$=$\hat{y},Y$=$\hat{y}+\delta)$=$Utility(\hat{Y}$=$\hat{y},Y$=$\hat{y}-\delta)$,
for all $\delta,\hat{y}$).

(7) $\qquad Utility(\hat{Y}=\hat{y},Y=y)\approx\mathbbm{1}(\hat{y}=y)\hfill$
(8) $\qquad Utility(\hat{y},y)\text{ is a function of }|\hat{y}-y|\hfill$

Taken together with the previous assumptions, this yields
$Utility(\hat{y},y)=P(\hat{y}=y)$ for classification tasks.

Evaluation Gap 7: Disparate harms and benefits. Treating all failure cases as
equivalent fails to appreciate that different classes of errors often have
very different impacts (Provost and Fawcett, 1997; Challen et al., 2019). In
multiclass classification, severely offensive predictions (e.g., predicting an
animal in an image of a person) are given the same weight as inoffensive ones.
In regression tasks, insensitivity to either the direction of the difference
$\hat{y}-y$ or the magnitude of $y$ can result in evaluations being possibly
poor proxies for downstream impacts. (One common application use case of
regression models is to apply a cutoff threshold $t$ to the predicted scalar
values, for which both the direction of error and the magnitude of $y$ are
relevant.)

Assumption 6: Test Data Validity. Taken collectively, the previous assumptions
might lead one to use accuracy as an evaluation metric for a classification
task. Further assumptions can then be made in deciding _how_ to estimate
accuracy. The final assumption we discuss here is that the test data over
which accuracy (or other metrics) is calculated provides a good estimate of
the accuracy of the model when embedded in the ecosystem.

(9) $\qquad P(\hat{y}=y)\approx P(\hat{y}^{\prime}=y^{\prime})\hfill$

where $Y^{\prime}=y^{\prime}$ and $\hat{Y}^{\prime}=\hat{y}^{\prime}$ are the
ground truth labels and the model predictions on the test data, respectively.

Evaluation Gap 8: Data Drifts. A simple model of the ecosystem’s data
distributions is particularly risky when system feedback effects would cause
the distributions of data in the ecosystem to diverge from those in the
evaluation sample (Liu et al., 2018; Kannan et al., 2019). In general, this
can lead to overconfidence in the system’s reliability, which can be
exacerbated for regions in the tail of the input distribution.

###  4.2. Discussion

| Assumption |  Considerations that might be Overlooked
---|---|---
$Understand(Model+Ecosystem)$ | Application-centric evaluation |  Opportunities for scientific insights.
$\approx Utility(Model+Ecosystem)$ | Consequentialism |  Data sourcing and processing; invisible labour; consultation with impacted communities; motives; public acceptance; human rights.
$\approx Utility(\hat{Y},X,Y)$ | Abstractability from Context |  System feedback loops; humans-in-the-loop.
$\approx Utility(\hat{Y},Y)$ | Input Myopia |  Disaggregated analyses; sensitivity analyses; safety-critical edge cases.
$\approx E_{(\hat{Y}=\hat{y},Y=y)}\\{Utility(\hat{y},y)\\}$ | Quantitative Modeling |  Different flavors of impacts on a single person; different flavors of impacts across groups.
$\approx P(\hat{y}=y)$ | Equivalent Failures |  Severe failure cases; confusion matrices; topology of the prediction space.
$\approx P(\hat{y}^{\prime}=y^{\prime})$ | Test Data Validity |  Data sampling biases; distribution shifts.
Table 4. Sketch of how the six assumptions of Section 4—when taken
collectively—compose to simplify the task of evaluating a model for an
application ($Understand(Model+Ecosystem)$) to one of calculating accuracy
over a data sample. A pseudo-formal notation (akin to pseudo-code) is used to
enable rapid glossing of the main connections. $Y=y$ and $\hat{Y}=\hat{y}$
denote the true (unobserved) distributions of ground truth and model
predictions, respectively, while the variables $Y^{\prime}=y^{\prime}$ and
$\hat{Y}^{\prime}=\hat{y}^{\prime}$ denote the samples of reference labels and
model predictions over which accuracy is calculated in practice. The order of
the assumptions reflects an increasing focus on technical aspects of model
evaluation, and a corresponding minimizing of non-technical aspects. Appendix
C illustrates how each of the sets of considerations might apply in a
hypothetical application of a computer vision model.

We have described six assumptions that simplify the model evaluation task.
Taken together, they would cause one to believe—with compounding risks—that a
model’s accuracy is a good proxy for its fitness for an application. We sketch
this composition of assumptions in Figure 4, along with questions that
illustrate the gaps raised by each assumption. Our reason for teasing apart
these assumptions and their compounding effects is _not_ to attack the
“strawman” of naive application-centric evaluations which rely solely on
estimating model accuracy. Rather, our goal is to point out that most model
evaluations, even sophisticated ones, make such assumptions to varying
degrees. For example:

  * •

Some robustness evaluations (for surveys, see (Farahani et al., 2020; Wang et
al., 2021)) explicitly tackle the problem of distribution shifts, rejecting
the Assumptions of Test Data Validity without questioning the other
assumptions we have identified.

  * •

Some sensitivity evaluations consider the effect on the model predictions of
small changes in the input, but use accuracy as an evaluation metric,
rejecting the Input Myopia Assumption without questioning the others (Ribeiro
et al., 2020).

  * •

Some fairness evaluations perform disaggregated analyses using the Recall or
Precision metrics, sticking by all assumptions other than Input Myopia and
Equivalent Failures (Chouldechova, 2017; Hardt et al., 2016).

It may not be possible to avoid all of the assumptions all of the time;
nevertheless unavoidable assumptions should be acknowledged and critically
examined. The six assumptions we have identified also provide a lens for
assessing the consistency of some evaluation metrics with other assumptions
that have been made during the evaluation, for example

  * •

_Is $F$-score consistent with an utilitarian evaluation framework?_ The
$F$-score is mathematically a harmonic mean— which is often appropriate for
averaging pairs of rates (e.g., two speeds). When applied to Precision and
Recall, however, the $F$-score constitutes a peculiar averaging of “apples and
oranges,” since, when conceived as rates, Precision and Recall measure rates
of change of different quantities, (Powers, 2014). $F$-score is thus difficult
to interpret within an evaluation framework that aims to maximize model
utility.

  * •

_Do threshold-free evaluations such as the Area Under the Receiver Operating
Characteristic ( auroc) abstract too much of the deployment context?_ Since
auroc is calculated by averaging over a range of possible threshold values, it
“cannot be interpreted as having any relevance to any particular classifier”
(Powers, 2012b) (which is not saying auroc is irrelevant to evaluating _the
learner_ , cf. Section 2, nor to a learned model’s propensity to correctly
rank positive instances above negative ones). The same argument can be made
for the Mean Average Precision metric used in image classification (see
Appendix A). For useful application-centric evaluations, it is more meaningful
to report pairs of $(Precision,Recall)$ values (for all classes) for a range
of threshold values (Powers, 2012a).

In both cases, we ask whether such metrics are of limited utility in
application-centric evaluations and whether they are better left to learner-
centric ones.

##  5\. Contextualizing Application-centric Model Evaluations

> _the ornithologists were forced to adapt their behavior (for the sake of
> “science”) to the most primitive evaluation method which was the only one
> considered or known, or else throw their data away._ — Hampel (Hampel and
> Zurich, 1998)

When applications of ML models have the potential to impact human lives and
livelihoods, thorough and reliable evaluations of models are critical. As
discussed in Section 3, the different goals and values of academic ML research
communities mean that research norms cannot be relied upon as guideposts for
evaluating models for applications. In this section, we propose steps towards
evaluations that are rigorous in their methods and aim to be humble about
their epistemic uncertainties. In doing so, we expand on the call by Raji et
al. to pay more attention not just to evaluation metric values but also to the
quality and reliability of the measurements themselves, including sensitivity
to external factors (Raji et al., 2021).

###  5.1. Minding the Gaps between Evaluation Goals and Research Practice

Documenting assumptions made during model evaluation is critical for
transparency and enables more informed decisions. If an assumption is
difficult to avoid in practice, consider augmenting the evaluation with
signals that may shed complementary light on questions of concern. For
example, even a handful of insightful comments from members of impacted
communities can be an invaluable complement to evaluations using quantitative
metrics. We now consider specific mitigation strategies for each of the gaps
in turn.

Minding Gap 1: Evaluate More than Consequences. To reduce the gap introduced
by the Consequentialism Assumption, evaluate the processes that led to the
creation of the model, including how datasets were constructed (Scheuerman et
al., 2021). We echo calls for more reflexivity around social and intentional
factors around model development (Miceli et al., 2021), more documentation of
the complete lifecycle of model development (Vogelsang and Borg, 2019;
Hutchinson et al., 2021), and greater transparency around ML models and their
datasets (Mitchell et al., 2019; Gebru et al., 2021; Bender and Friedman,
2018). It may be appropriate to contemplate whether the model is aligned with
the virtues the organization aspires to (Vallor, 2016). Consider the question
of whether any ML model could be a morally appropriate solution in this
application context, e.g., whether it is appropriate to make decisions about
one person on the basis of others’ behaviors (Eckhouse et al., 2019).

Minding Gap 2: Center Obligations. Since reasoning about uncertain future
states of the world is fraught with challenges (Card and Smith, 2020),
evaluations should consider indirect consequences and assess how the model
upholds social obligations within the ecosystem. This may involve processes
such as assessments of human rights, social and ethical impact (McGregor et
al., 2019; Mantelero, 2018), audits of whether the ML system upholds the
organization’s declared values or principles (Raji et al., 2020), and/or
assessments of the potential for privacy leakage (e.g., (Yeom et al., 2018;
Carlini et al., 2021)).

Minding Gap 3: Demarginalize the Context. To address the gap introduced by the
Assumption of Abstractability from Context, consider externalities such as
energy consumption (Henderson et al., 2020; Schwartz et al., 2020), as well as
resource requirements (Ethayarajh and Jurafsky, 2020). It is important to
think about how the human and technical parts of the system will interact
(Selbst et al., 2019; Martin et al., 2020). Note that when substituting one
model for another—or for displaced human labor—system stability can itself be
a desirable property independent of model accuracy (and perhaps counter to
tech industry discourses of “disruption” (Geiger, 2020)), and a range of
metrics exist for comparing predictions with those of a legacy model
(Derczynski, 2016). Care should be taken to avoid the “portability trap” of
assuming that what is good for one context is good for another (Selbst et al.,
2019). The more attention paid to the specifics of the application context,
the better; hence, metrics which assume no particular classification
threshold, such as AUC, may provide limited signal for any single context.

Minding Gap 4: Make Subjectivities Transparent. Acknowledge the subjectivities
inherent in many tasks (Alm, 2011). An array of recent scholarship on
subjectivity in ML has “embraced disagreement” through practices of explicitly
modeling—in both the data model and the ML model—inter-subject variation in
interpretations (Davani et al., 2022; Basile et al., 2021; Fornaciari et al.,
2021; Díaz and Diakopoulos, 2019; Aroyo and Welty, 2015). For the purposes of
ML model evaluations, disaggregating labels on test data according to the
cultural and socio-demographic standpoints of their annotators enables more
nuanced disaggregated evaluation statistics (Prabhakaran et al., 2021).

Minding Gap 5: Respect Differences Between Inputs. A realistic “null
hypothesis” is that misclassifications affect people in the application
ecosystem disparately. For example, people may differ both in their
preferences regarding model predictions $\hat{Y}$ _per se_ , as well as their
preferences regarding model accuracy $\hat{Y}=Y$ (Binns, 2018).666Note that in
many real-world applications the “ground truth” variable $Y$ may be a
convenient counterfactual fiction, since the system’s actions on the basis of
the prediction $\hat{Y}$ may inhibit $Y$ from being realised—for example, a
finance ML model may predict a potential customer would default on a loan if
given one, and hence the system the model is deployed in may prevent the
customer getting a loan in the first place. As such—and _independent of
fairness considerations_ —evaluations should be routinely pay attention to
different parts of the input distribution, including disaggregating along
social subgroups. Special attention should be paid to the tail of the
distribution and outliers during evaluation, as these may require further
analysis to diagnose the potential for rare but unsafe impacts. Input
sensitivity testing can provide useful information about the sensitivity of
the classifier to dimensions of input variation known to be of concern (e.g.,
gender in text (Zhao et al., 2018; Borkan et al., 2019; Gonen and Webster,
2020; Huang et al., 2020)).

Minding Gap 6: Think Beyond Scalar Utility. Resist the temptation to reduce a
model’s utility to a single scalar value, either for stack ranking (Ethayarajh
and Jurafsky, 2020) or to simplify the cognitive load on decision makers.
Instead, include a range of different metrics and evaluation distributions in
the evaluation (Mitchell et al., 2019). Acknowledge and report epistemic
uncertainty, e.g., the effects of missing data or measurement and sampling
error on metrics. Acknowledge qualitative impacts that are not addressed by
metrics (e.g., harms to application users caused by supplanting socially
meaningful human interactions), and rigorously assess the validity of attempts
to measure social or emotional harms. Be conservative in aggregations:
consider plotting data rather than reporting summary statistics (cf.
Anscombe’s quartet); do not aggregate unlike quantities; report multiple
estimates of central tendency and variation; and don’t assume that all users
of an application will have equal benefits (or harms) from system outcomes.
Consider applying aggregation and partial ranking techniques from the fair
division literature to ML models, including techniques that give greater
weight to those with the worst outcomes (e.g., in the extreme case, “Maximin”)
(Endriss, 2018).

Minding Gap 7: Respect Differences Between Failures. If the harms of false
positives and false negatives are incommensurable, report them separately. If
commensurable, weight each appropriately. For multiclass classifiers, this
approach generalizes to a _classification cost matrix_ (Turney, 1994), and,
more generally, including the _confusion matrix_ before costs are assigned;
for regression tasks, report metrics such as MSE disaggregated by buckets of
$Y$.

Minding Gap 8: Validate Quality of Test Data. For transparency, do not assume
it is obvious to others which datasets are used in training and evaluation;
instead, be explicit about the provenance, distribution, and known biases of
the datasets in use (Andrus et al., 2021). Consider Bayesian approaches to
dealing with uncertainty about data distributions (McNair, 2018; Ji et al.,
2020; Lacoste et al., 2017), especially when sample sizes are small or prior
work has revealed systematic biases. For example, an evaluation which uses
limited data in a novel domain (or in an under-studied language) to
investigate gender biases in pronoun resolution should be tentative in drawing
strong positive conclusions about “fairness” due to abundant evidence of
gender biases in English pronoun resolution models (e.g. (Webster et al.,
2019)).

###  5.2. Alternate Model Evaluation Methodologies

More radical excursions from the disciplinary paradigm are often worth
considering, especially in scenarios with high stakes or high uncertainty.

Evaluation Remits. In 1995, Sparck Jones and Galliers called for a careful
approach to NLP evaluation that is broadly applicable to ML model evaluations
(see Appendix D) (Jones and Galliers, 1995). Their approach involves a top-
down examination of the context and goal of the evaluation before the
evaluation design even begins, and their call for careful documentation of the
evaluation “remit”—i.e., official responsibilities—is in line with more recent
work calling for stakeholder transparency for ML (Raji et al., 2020;
Hutchinson et al., 2021). They advocate for establishing whose perspectives
are adopted in the evaluation and whose interests prompted it. Appendix D
sketches how Sparck Jones and Galliers’ framework could be adopted for ML
model evaluations.

Active Testing. Active Testing aims to iteratively choose new items that are
most informative in addressing the goals of the evaluation (Kossen et al.,
2021; Ha et al., 2021) (cf. its cousin Active Learning, which selects items
that are informative for the learner). Active Testing provides a better
estimate of model performance than using the same number of test instances
sampled I.I.D. Exploring Active Testing in pursuit of fairness testing goals
seems a promising direction for future research.

Adversarial Testing. In many cases, there is great uncertainty regarding an
application deployment context. One cautious and conservative
approach—especially in the face of great uncertainty—is to simulate
“adversaries” trying to provoke harmful outcomes from the system. Borrowing
adversarial techniques from security testing and privacy testing, adversarial
testing of models requires due diligence to trigger the most harmful model
predictions, using either manually chosen or algorithmically generated test
instances (Ruiz et al., 2022; Zeng et al., 2021; Zhang et al., 2020b; Ettinger
et al., 2017).

Multidimensional Comparisons. When comparing candidate models, avoid the
“Leaderboardism Trap” of believing that a total ordering of candidates is
possible. A multidimensional and nuanced evaluation may provide at best a
partial ordering of candidate models, and it may require careful and
accountable judgement and qualitative considerations to decide among them. The
Fair Division literature on Social Welfare Orderings may be a promising
direction for developing evaluation frameworks that prioritize “egalitarian”
considerations, in which greater weighting is given to those who are worst
impacted by a model (Endriss, 2018).

###  5.3. Evaluation-driven ML Methodologies

In this section, we follow Rostamzadeh et al. in drawing inspiration from
test-driven practices, such as those of software development (Rostamzadeh et
al., 2021). Traditional software testing involves significant time, resources,
and effort (Harrold, 2000); even moderate-sized software projects spend
hundreds of person-hours writing test cases, implementing them, and
meticulously documenting the test results. In fact, software testing is
sometimes considered an art (Myers et al., 2011) requiring its own technical
and non-technical skills (Sánchez-Gordón et al., 2020; Matturro, 2013), and
entire career paths are built around testing (Cunningham et al., 2019). _Test-
driven development_ , often associated with agile software engineering
frameworks, integrates testing considerations in all parts of the development
process (Astels, 2003; George and Williams, 2004). These processes rely on a
deep understanding of software requirements and user behavior to anticipate
failure modes during deployment and to expand the test suite. (In contrast, ML
testing is often relegated to a small portion of the ML development cycle, and
predominantly focuses on a static snapshot of data to provide performance
guarantees.) These software testing methodologies provide a model for ML
testing. First, the model suggests anticipating, planning for, and integrating
testing in all stages of the development cycle, research problem ideation, the
setting of objectives, and system implementation. Second, build a practice
around bringing diverse perspectives into designing the test suite.
Additionally, consider participatory approaches (e.g., (Martin et al., 2020))
to ensure that the test suite accounts for societal contexts and embedded
values within which the ML system will be deployed.

An important principle in test-driven software development is visibility into
the test data. Typically, engineers working on a system can not only see the
details of the test suites but also often develop those test suites
themselves. In contrast, the paradigm of ML evaluation methodologies is that
the ML practitioner should not inspect the test data, lest their observations
result in design decisions that produce an overfitted model. How, then, can
these two methodologies be reconciled? We believe that incentives are one
important consideration. In the ML research community, the “competition
mindset” might indeed lead to “cheating” via deliberate overfitting. In
contrast, in real-world applications model developers might benefit from a
healthy model ecosystem, for example when they are members of that ecosystem.
(However, when developers come from a different society altogether there may
be disinterest or disalignment (Sambasivan et al., 2021a).)

Software testing produces artifacts such as execution traces, and test
coverage information (Harrold, 2000). Developing practices for routinely
sharing testing artifacts with stakeholders provides for more robust scrutiny
and diagnosis of harmful error cases (Raji et al., 2020). In being flexible
enough to adapt to the information needs of stakeholders, software testing
artifacts can be considered a form of boundary object (Star and Griesemer,
1989). Within an ML context, these considerations point towards adopting ML
transparency mechanisms incorporating comprehensive evaluations, such as model
cards (Mitchell et al., 2019). The processes that go into building test cases
should be documented, so the consumer of the ML system can better understand
the system’s reliability. Finally, as for any high-stakes system—software, ML
or otherwise—evaluation documentation constitutes an important part of the
chain of auditable artifacts required for robust accountability and governance
practices (Raji et al., 2020).

##  6\. Conclusions

In this paper, we compared the evaluation practices in the ML research
community to the ideal information needs of those who use models in real-world
applications. The observed disconnect between the two is likely due to
differences in motivations and goals, and also pressures to demonstrate “state
of the art” performance on shared tasks, metrics and leaderboards (Ethayarajh
and Jurafsky, 2020; Koch et al., 2021; Thomas and Uminsky, 2020), as well as a
focus on the learner as the object upon which the researcher hopes to shed
light. One limitation of our methodology is reliance on published papers, and
we encourage more human subjects research in the future, in a similar vein to
e.g. (Holstein et al., 2019; Sambasivan et al., 2021b; Madaio et al., 2022).
We identified a range of evaluation gaps that risk being overlooked if the ML
research community’s evaluation practices are uncritically adopted when for
applications, and identify six assumptions that would have to be valid if
these gaps are to be overlooked. The assumptions range from a broad focus on
consequentialism to technical concerns regarding distributions of evaluation
data. By presenting these assumptions as a coherent framework, we provide not
just a set of mitigations for each evaluation gap, but also demonstrate the
relationships between these mitigations. We show how in the naive case these
assumptions chain together, leading to the grossest assumption that
calculating model accuracy on data I.I.D. with the training data can be a
reliable signal for real-world applications. We contrast the practices of ML
model evaluation with those of the mature engineering practices of software
testing to draw out lessons for non-I.I.D. testing under a variety of stress
conditions and failure severities. One limitation of our analysis is that we
are generally domain-agnostic, and we hope to stimulate investigations of
assumptions and gaps for specific application domains. We believe it is
fundamental that model developers are explicit about methodological
assumptions in their evaluations. We believe that ML model evaluations have
great potential to enable interpretation and use by different technical and
non-technical communities (Star and Griesemer, 1989). By naming each
assumption we identify and exploring its technical and sociological
consequences, we hope to encourage more robust interdisciplinary debate and,
ultimately, to nudge model evaluation practice away from abundant opaque
unknowns.

###### Acknowledgements.

We acknowledge useful feedback from Daniel J. Barrett, Alexander D’Amour,
Stephen Pfohl, D. Sculley, and the anonymous reviewers.

## References

  * (1)
  * Alam et al. (2018) Firoj Alam, Shafiq Joty, and Muhammad Imran. 2018.  Domain Adaptation with Adversarial Training and Graph Embeddings. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1077–1087.
  * Alm (2011) Cecilia Ovesdotter Alm. 2011\.  Subjective natural language problems: Motivations, applications, characterizations, and implications. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. 107–112.
  * Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016\.  Concrete problems in AI safety.  _arXiv preprint arXiv:1606.06565_ (2016).
  * Andreotta et al. (2021) Adam J Andreotta, Nin Kirkham, and Marco Rizzi. 2021\.  AI, big data, and the future of consent.  _AI & Society_ (2021), 1–14.
  * Andrus and Gilbert (2019) McKane Andrus and Thomas K Gilbert. 2019.  Towards a just theory of measurement: A principled social measurement assurance program for machine learning. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_. 445–451.
  * Andrus et al. (2021) McKane Andrus, Elena Spitzer, Jeffrey Brown, and Alice Xiang. 2021.  What We Can’t Measure, We Can’t Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 249–260.
  * Aroyo and Welty (2015) Lora Aroyo and Chris Welty. 2015.  Truth is a lie: Crowd truth and the seven myths of human annotation.  _AI Magazine_ 36, 1 (2015), 15–24.
  * Astels (2003) Dave Astels. 2003\.  _Test driven development: A practical guide_.  Prentice Hall Professional Technical Reference.
  * Barocas et al. (2021) Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, and Hanna Wallach. 2021.  Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs.  _arXiv preprint arXiv:2103.06076_ (2021).
  * Barocas et al. (2017) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017\.  Fairness in machine learning.  _NIPS tutorial_ 1 (2017), 2017.
  * Barthes (1977) Roland Barthes. 1977\.  _Image-Music-Text_.  Macmillan.
  * Basile et al. (2021) Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021.  Toward a Perspectivist Turn in Ground Truthing for Predictive Computing.  _arXiv preprint arXiv:2109.04270_ (2021).
  * Bender and Friedman (2018) Emily M Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ 6 (2018), 587–604.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 610–623.
  * Bengio et al. (2021) Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. 2021\.  Deep learning for AI.  _Commun. ACM_ 64, 7 (2021), 58–65.
  * Berger (2008) John Berger. 2008\.  _Ways of seeing_.  Penguin UK.
  * Binns (2018) Reuben Binns. 2018\.  Fairness in machine learning: Lessons from political philosophy. In _Conference on Fairness, Accountability and Transparency_. PMLR, 149–159.
  * Birhane et al. (2021) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2021\.  The values encoded in machine learning research.  _arXiv preprint arXiv:2106.15590_ (2021).
  * Blasi et al. (2021) Damián Blasi, Antonios Anastasopoulos, and Graham Neubig. 2021\.  Systematic Inequalities in Language Technology Performance across the World’s Languages.  _arXiv preprint arXiv:2110.06733_ (2021).
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019.  Nuanced metrics for measuring unintended bias with real data for text classification. In _Companion proceedings of the 2019 world wide web conference_. 491–500.
  * Bowman and Dahl (2021) Samuel Bowman and George Dahl. 2021.  What Will it Take to Fix Benchmarking in Natural Language Understanding?. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 4843–4855.
  * Breck et al. (2017) Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D Sculley. 2017.  The ML test score: A rubric for ML production readiness and technical debt reduction. In _2017 IEEE International Conference on Big Data (Big Data)_. IEEE, 1123–1132.
  * Breiman (2001) Leo Breiman. 2001\.  Statistical modeling: The two cultures (with comments and a rejoinder by the author).  _Statistical science_ 16, 3 (2001), 199–231.
  * Brewster (1881) Benjamin Brewster. 1881\.  _The Yale Literary Magazine_ October 1881–June 1882 (1881).
  * Bulleit et al. (2015) William Bulleit, Jon Schmidt, Irfan Alvi, Erik Nelson, and Tonatiuh Rodriguez-Nikl. 2015.  Philosophy of engineering: What it is and why it matters.  _Journal of Professional Issues in Engineering Education and Practice_ 141, 3 (2015), 02514003.
  * Bunescu and Huang (2010) Razvan Bunescu and Yunfeng Huang. 2010.  A utility-driven approach to question ranking in social QA. In _Proceedings of The 23rd International Conference on Computational Linguistics (COLING 2010)_. 125–133.
  * Cai et al. (2017) Ermao Cai, Da-Cheng Juan, Dimitrios Stamoulis, and Diana Marculescu. 2017.  NeuralPower: Predict and deploy energy-efficient convolutional neural networks. In _Asian Conference on Machine Learning_. PMLR, 622–637.
  * Card and Smith (2020) Dallas Card and Noah A Smith. 2020.  On Consequentialism and Fairness.  _Frontiers in Artificial Intelligence_ 3 (2020), 34.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021\.  Extracting training data from large language models. In _30th USENIX Security Symposium (USENIX Security 21)_. 2633–2650.
  * Carter et al. (2021) Brandon Carter, Siddhartha Jain, Jonas W Mueller, and David Gifford. 2021.  Overinterpretation reveals image classification model pathologies.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Challen et al. (2019) Robert Challen, Joshua Denny, Martin Pitt, Luke Gompels, Tom Edwards, and Krasimira Tsaneva-Atanasova. 2019.  Artificial intelligence, bias and clinical safety.  _BMJ Quality & Safety_ 28, 3 (2019), 231–237.
  * Charlton (1998) James I Charlton. 1998\.  _Nothing about us without us_.  University of California Press.
  * Chen et al. (2021) Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. 2021.  Mandoline: Model Evaluation under Distribution Shift. In _International Conference on Machine Learning_. PMLR, 1617–1629.
  * Chiril et al. (2020) Patricia Chiril, Véronique Moriceau, Farah Benamara, Alda Mari, Gloria Origgi, and Marlène Coulomb-Gully. 2020\.  He said “who’s gonna take care of your children when you are at ACL?”: Reported Sexist Acts are Not Sexist. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 4055–4066.
  * Chohlas-Wood et al. (2021) Alex Chohlas-Wood, Madison Coots, Emma Brunskill, and Sharad Goel. 2021.  Learning to be Fair: A Consequentialist Approach to Equitable Decision-Making.  _arXiv preprint arXiv:2109.08792_ (2021).
  * Chouldechova (2017) Alexandra Chouldechova. 2017\.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ 5, 2 (2017), 153–163.
  * Corbett-Davies and Goel (2018) Sam Corbett-Davies and Sharad Goel. 2018.  The measure and mismeasure of fairness: A critical review of fair machine learning.  _arXiv preprint arXiv:1808.00023_ (2018).
  * Corbett-Davies et al. (2017) Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017.  Algorithmic decision making and the cost of fairness. In _Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining_. 797–806.
  * Crawford and Joler (2018) Kate Crawford and Vladan Joler. 2018.  Anatomy of an AI System.  (Accessed January, 2022).
  * Crawford and Paglen (2021) Kate Crawford and Trevor Paglen. 2021.  Excavating AI: The politics of images in machine learning training sets.  _AI & SOCIETY_ (2021), 1–12.
  * Cunningham et al. (2019) Sean Cunningham, Jemil Gambo, Aidan Lawless, Declan Moore, Murat Yilmaz, Paul M Clarke, and Rory V O’Connor. 2019.  Software testing: a changing career. In _European Conference on Software Process Improvement_. Springer, 731–742.
  * Dahlin (2021) Emma Dahlin. 2021\.  Mind the gap! On the future of AI research.  _Humanities and Social Sciences Communications_ 8, 1 (2021), 1–4.
  * D’Amour et al. (2020) Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 2020\.  Underspecification presents challenges for credibility in modern machine learning.  _arXiv preprint arXiv:2011.03395_ (2020).
  * Davani et al. (2022) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 92–110.
  * De Vries et al. (2020) Harm De Vries, Dzmitry Bahdanau, and Christopher Manning. 2020\.  Towards ecologically valid research on language user interfaces.  _arXiv preprint arXiv:2007.14435_ (2020).
  * Derczynski (2016) Leon Derczynski. 2016\.  Complementarity, F-score, and NLP Evaluation. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_. 261–266.
  * Díaz and Diakopoulos (2019) Mark Díaz and Nicholas Diakopoulos. 2019.  Whose walkability?: Challenges in algorithmically measuring subjective experience.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–22.
  * Eckhouse et al. (2019) Laurel Eckhouse, Kristian Lum, Cynthia Conti-Cook, and Julie Ciccolini. 2019.  Layers of bias: A unified approach for understanding problems with risk assessment.  _Criminal Justice and Behavior_ 46, 2 (2019), 185–209.
  * Endriss (2018) Ulle Endriss. 2018\.  Lecture notes on fair division.  _arXiv preprint arXiv:1806.04234_ (2018).
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the Eye of the User: A Critique of NLP Leaderboards. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 4846–4853.
  * Ettinger et al. (2017) Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. 2017.  Towards linguistically generalizable NLP systems: A workshop and shared task.  _arXiv preprint arXiv:1711.01505_ (2017).
  * Evci et al. (2021) Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael Curtis Mozer. 2021.  Head2Toe: Utilizing Intermediate Representations for Better OOD Generalization.  (2021).
  * Farahani et al. (2020) Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. 2020.  A brief review of domain adaptation.  _arXiv preprint arXiv:2010.03978_ (2020).
  * Fornaciari et al. (2021) Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021.  Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 2591–2597.
  * Forsythe (2001a) Diana Forsythe. 2001a.  _Studying those who study us: An anthropologist in the world of Artificial Intelligence_.  Stanford University Press, Chapter Artificial intelligence invents itself: Collective identity and boundary maintenance in an emergent scientific discipline.
  * Forsythe (2001b) Diana Forsythe. 2001b.  _Studying those who study us: An anthropologist in the world of Artificial Intelligence_.  Stanford University Press, Chapter The Construction of Knowledge in Artificial Intelligence.
  * Friedler et al. (2021) Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2021\.  The (im) possibility of fairness: Different value systems require different mechanisms for fair decision making.  _Commun. ACM_ 64, 4 (2021), 136–143.
  * Fu et al. (2022) Biying Fu, Cong Chen, Olaf Henniger, and Naser Damer. 2022\.  A deep insight into measuring face image utility with general and face-specific image quality metrics. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 905–914.
  * García-Martín et al. (2019) Eva García-Martín, Crefeda Faviola Rodrigues, Graham Riley, and Håkan Grahn. 2019.  Estimation of energy consumption in machine learning.  _J. Parallel and Distrib. Comput._ 134 (2019), 75–88.
  * Garg et al. (2019) Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. 2019\.  Counterfactual fairness in text classification through robustness. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_. 219–226.
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021\.  Datasheets for datasets.  _Commun. ACM_ 64, 12 (2021), 86–92.
  * Geertz (1973) Clifford Geertz. 1973\.  _The Interpretation of Cultures_.  Basic Books.
  * Geiger (2020) Susi Geiger. 2020\.  Silicon Valley, disruption, and the end of uncertainty.  _Journal of cultural economy_ 13, 2 (2020), 169–184.
  * George and Williams (2004) Boby George and Laurie Williams. 2004.  A structured experiment of test-driven development.  _Information and software Technology_ 46, 5 (2004), 337–342.
  * Gonen and Webster (2020) Hila Gonen and Kellie Webster. 2020.  Automatically Identifying Gender Issues in Machine Translation using Perturbations. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 1991–1995.
  * Gray and Suri (2019) Mary L Gray and Siddharth Suri. 2019.  _Ghost work: How to stop Silicon Valley from building a new global underclass_.  Eamon Dolan Books.
  * Green (2020) Ben Green. 2020\.  Data science as political action: grounding data science in a politics of justice.  _Available at SSRN 3658431_ (2020).
  * Ha et al. (2021) Huong Ha, Sunil Gupta, Santu Rana, and Svetha Venkatesh. 2021\.  ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms.  _arXiv preprint arXiv:2104.04999_ (2021).
  * Hampel and Zurich (1998) Frank Hampel and Eth Zurich. 1998.  Is statistics too difficult?  _Canadian Journal of Statistics_ 26, 3 (1998), 497–513.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ 29 (2016), 3315–3323.
  * Harrold (2000) Mary Jean Harrold. 2000\.  Testing: a roadmap. In _Proceedings of the Conference on the Future of Software Engineering_. 61–72.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Heldreth et al. (2021) Courtney Heldreth, Michal Lahav, Zion Mengesha, Juliana Sublewski, and Elyse Tuennerman. 2021.  “I don’t think these devices are very culturally sensitive.”—The impact of errors on African Americans in Automated Speech Recognition.  _Frontiers in Artificial Intelligence_ 26 (2021).
  * Henderson et al. (2020) Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020\.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  _Journal of Machine Learning Research_ 21, 248 (2020), 1–43.
  * Hepp et al. (2018) Benjamin Hepp, Debadeepta Dey, Sudipta N Sinha, Ashish Kapoor, Neel Joshi, and Otmar Hilliges. 2018\.  Learn-to-score: Efficient 3D scene exploration by predicting view utility. In _Proceedings of the European conference on computer vision (ECCV)_. 437–452.
  * Holstein et al. (2019) Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019\.  Improving fairness in machine learning systems: What do industry practitioners need?. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–16.
  * Hooker (1995) John N Hooker. 1995\.  Testing heuristics: We have it all wrong.  _Journal of heuristics_ 1, 1 (1995), 33–42.
  * Hovy and Spruit (2016) Dirk Hovy and Shannon L Spruit. 2016.  The social impact of natural language processing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. 591–598.
  * Huang et al. (2020) Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020.  Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 65–83.
  * Hutchinson and Mitchell (2019) Ben Hutchinson and Margaret Mitchell. 2019.  50 years of test (un) fairness: Lessons for machine learning. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. 49–58.
  * Hutchinson et al. (2021) Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021\.  Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 560–575.
  * Idahl et al. (2021) Maximilian Idahl, Lijun Lyu, Ujwal Gadiraju, and Avishek Anand. 2021.  Towards Benchmarking the Utility of Explanations for Model Debugging. In _Proceedings of the First Workshop on Trustworthy Natural Language Processing_. 68–73.
  * IEEE (2019) IEEE. 2019.  The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. “Classical Ethics in A/IS”.  In _Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems, First Edition_. 36–67.
  * Jacobs et al. (2020) Abigail Z Jacobs, Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020\.  The meaning and measurement of bias: lessons from natural language processing. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 706–706.
  * Jacobs and Wallach (2021) Abigail Z Jacobs and Hanna Wallach. 2021.  Measurement and fairness. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 375–385.
  * Jafarian and Park (2021) Yasamin Jafarian and Hyun Soo Park. 2021.  Learning high fidelity depths of dressed humans by watching social media dance videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12753–12762.
  * Japkowicz (2006) Nathalie Japkowicz. 2006\.  Why question machine learning evaluation methods. In _AAAI workshop on evaluation methods for machine learning_. 6–11.
  * Jappy (2013) Tony Jappy. 2013\.  _Introduction to Peircean visual semiotics_.  A&C Black.
  * Ji et al. (2020) Disi Ji, Padhraic Smyth, and Mark Steyvers. 2020.  Can i trust my fairness metric? assessing fairness with unlabeled data and bayesian inference.  _Advances in Neural Information Processing Systems_ 33 (2020), 18600–18612.
  * Jones and Galliers (1995) Karen Sparck Jones and Julia R Galliers. 1995.  _Evaluating natural language processing systems: An analysis and review_. Vol. 1083.  Springer Science & Business Media.
  * Kannan et al. (2019) Sampath Kannan, Aaron Roth, and Juba Ziani. 2019.  Downstream effects of affirmative action. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_. 240–248.
  * Koch et al. (2021) Bernard Koch, Emily Denton, Alex Hanna, and Jacob G Foster. 2021.  Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research.  _NeurIPS Dataset & Benchmark track_ (2021).
  * Koh et al. (2020) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. 2020\.  WILDS: A Benchmark of in-the-Wild Distribution Shifts.  _CoRR_ abs/2012.07421 (2020).  https://arxiv.org/abs/2012.07421
  * Kossen et al. (2021) Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. 2021.  Active testing: Sample-efficient model evaluation. In _International Conference on Machine Learning_. PMLR, 5753–5763.
  * Kukutai and Taylor (2016) Tahu Kukutai and John Taylor. 2016.  _Indigenous data sovereignty: Toward an agenda_.  ANU press.
  * Kuwajima et al. (2020) Hiroshi Kuwajima, Hirotoshi Yasuoka, and Toshihiro Nakae. 2020.  Engineering problems in machine learning systems.  _Machine Learning_ 109, 5 (2020), 1103–1126.
  * Lacoste et al. (2017) Alexandre Lacoste, Thomas Boquet, Negar Rostamzadeh, Boris Oreshkin, Wonchang Chung, and David Krueger. 2017.  Deep prior.  _arXiv preprint arXiv:1712.05016_ (2017).
  * Lacoste et al. (2018) Alexandre Lacoste, Boris Oreshkin, Wonchang Chung, Thomas Boquet, Negar Rostamzadeh, and David Krueger. 2018\.  Uncertainty in multitask transfer learning.  _arXiv preprint arXiv:1806.07528_ (2018).
  * Lakoff and Johnson (2008) George Lakoff and Mark Johnson. 2008.  _Metaphors we live by_.  University of Chicago press.
  * Lecué and Lerasle (2020) Guillaume Lecué and Matthieu Lerasle. 2020.  Robust machine learning by median-of-means: theory and practice.  _The Annals of Statistics_ 48, 2 (2020), 906–931.
  * Li et al. (2016) Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. 2016.  TGIF: A new dataset and benchmark on animated GIF description. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4641–4650.
  * Liao et al. (2021) Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. 2021.  Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Lin et al. (2007) Chien-Hsin Lin, Hsin-Yu Shih, and Peter J Sher. 2007\.  Integrating technology readiness into technology acceptance: The TRAM model.  _Psychology & Marketing_ 24, 7 (2007), 641–657.
  * Lin (2004) Chin-Yew Lin. 2004\.  Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_. 74–81.
  * Liu et al. (2018) Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018.  Delayed impact of fair machine learning. In _International Conference on Machine Learning_. PMLR, 3150–3158.
  * Lo and Wu (2010) Chi-kiu Lo and Dekai Wu. 2010.  Evaluating Machine Translation Utility via Semantic Role Labels.. In _LREC_. Citeseer.
  * Madaio et al. (2022) Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, and Hanna Wallach. 2022\.  Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support.  _Proceedings of the ACM on Human-Computer Interaction_ 6, CSCW1 (2022), 1–26.
  * Mantelero (2018) Alessandro Mantelero. 2018\.  AI and Big Data: A blueprint for a human rights, social and ethical impact assessment.  _Computer Law & Security Review_ 34, 4 (2018), 754–772.
  * Marrkula Center (2019) Marrkula Center. 2019\.  Approaches to Ethical Decision-making.  https://www.scu.edu/ethics/ethics-resources/ethical-decision-making/
  * Martin et al. (2020) Donald Martin, Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S. Isaac. 2020\.  Extending the Machine Learning Abstraction Boundary: A Complex Systems Approach to Incorporate Societal Context.  arXiv:2006.09663 [cs.CY]
  * Matturro (2013) Gerardo Matturro. 2013\.  Soft skills in software engineering: A study of its demand by software companies in Uruguay. In _2013 6th international workshop on cooperative and human aspects of software engineering (CHASE)_. IEEE, 133–136.
  * Mazzocchi (2015) Fulvio Mazzocchi. 2015\.  Could Big Data be the end of theory in science? A few remarks on the epistemology of data-driven science.  _EMBO reports_ 16, 10 (2015), 1250–1255.
  * McGregor et al. (2019) Lorna McGregor, Daragh Murray, and Vivian Ng. 2019.  International human rights law as a framework for algorithmic accountability.  _International & Comparative Law Quarterly_ 68, 2 (2019), 309–343.
  * McNair (2018) Douglas S McNair. 2018\.  Preventing disparities: Bayesian and frequentist methods for assessing fairness in machine learning decision-support models.  _New Insights into Bayesian Inference_ (2018), 71.
  * Miceli et al. (2021) Milagros Miceli, Tianling Yang, Laurens Naudts, Martin Schuessler, Diana Serbanescu, and Alex Hanna. 2021\.  Documenting Computer Vision Datasets: An Invitation to Reflexive Data Practices. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 161–172.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting. In _Proceedings of the conference on fairness, accountability, and transparency_. 220–229.
  * Moradi and Samwald (2021) Milad Moradi and Matthias Samwald. 2021.  Evaluating the Robustness of Neural Language Models to Input Perturbations. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 1558–1570.
  * Murez et al. (2018) Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. 2018\.  Image to image translation for domain adaptation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4500–4509.
  * Myers et al. (2011) Glenford J Myers, Corey Sandler, and Tom Badgett. 2011\.  _The art of software testing_.  John Wiley & Sons.
  * Neumann et al. (2020) Michael Neumann, Oliver Roessler, David Suendermann-Oeft, and Vikram Ramanarayanan. 2020.  On the utility of audiovisual dialog technologies and signal analytics for real-time remote monitoring of depression biomarkers. In _Proceedings of the First Workshop on Natural Language Processing for Medical Conversations_. 47–52.
  * Norvig (2017) Peter Norvig. 2017\.  On Chomsky and the two cultures of statistical learning.  In _Berechenbarkeit der Welt?_ Springer, 61–83.
  * Olteanu et al. (2019) Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıcıman. 2019.  Social data: Biases, methodological pitfalls, and ethical boundaries.  _Frontiers in Big Data_ 2 (2019), 13.
  * Orekondy et al. (2018) Tribhuvanesh Orekondy, Mario Fritz, and Bernt Schiele. 2018\.  Connecting pixels to privacy and utility: Automatic redaction of private information in images. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8466–8475.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. 311–318.
  * Pinheiro et al. (2019) Pedro O Pinheiro, Negar Rostamzadeh, and Sungjin Ahn. 2019\.  Domain-adaptive single-view 3d reconstruction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7638–7647.
  * Powers (2011) David Martin Ward Powers. 2011\.  Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation.  _Journal of Machine Learning Technologies_ 2, 1 (2011), 37–63.
  * Powers (2012a) David Martin Ward Powers. 2012a.  The problem of area under the curve. In _2012 IEEE International conference on information science and technology_. IEEE, 567–573.
  * Powers (2012b) David Martin Ward Powers. 2012b.  The problem with kappa. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_. 345–355.
  * Powers (2014) David Martin Ward Powers. 2014\.  What the F-measure doesn’t measure: Features, Flaws, Fallacies and Fixes.  _Technical report, Beijing University of Technology, China & Flinders University, Australia, Tech. Rep._ (2014).
  * Prabhakaran et al. (2021) Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021.  On Releasing Annotator-Level Labels and Information in Datasets. In _Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop_. 133–138.
  * Prabhakaran et al. (2019) Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019\.  Perturbation Sensitivity Analysis to Detect Unintended Model Biases. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 5740–5745.
  * Provost and Fawcett (1997) Foster Provost and Tom Fawcett. 1997.  Analysis and visualization of classifier performance with nonuniform class and cost distributions. In _Proceedings of AAAI-97 Workshop on AI Approaches to Fraud Detection & Risk Management_. 57–63.
  * Pustejovsky (1998) James Pustejovsky. 1998\.  _The generative lexicon_.  MIT press.
  * Raji et al. (2021) Inioluwa Deborah Raji, Emily Denton, Emily M Bender, Alex Hanna, and Amandalynne Paullada. 2021\.  AI and the Everything in the Whole Wide World Benchmark. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Raji et al. (2020) Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020\.  Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In _Proceedings of the 2020 conference on fairness, accountability, and transparency_. 33–44.
  * Raso et al. (2018) Filippo A Raso, Hannah Hilligoss, Vivek Krishnamurthy, Christopher Bavitz, and Levin Kim. 2018\.  Artificial intelligence & human rights: Opportunities & risks.  _Berkman Klein Center Research Publication_ 2018-6 (2018).
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ 28 (2015).
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  _arXiv preprint arXiv:2005.04118_ (2020).
  * Rismani and Moon (2021) Shalaleh Rismani and Ajung Moon. 2021.  How do AI systems fail socially?: an engineering risk analysis approach. In _2021 IEEE International Symposium on Ethics in Engineering, Science and Technology (ETHICS)_. 1–8.  https://doi.org/10.1109/ETHICS53270.2021.9632769
  * Rodriguez et al. (2021) Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021.  Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 4486–4503.  https://doi.org/10.18653/v1/2021.acl-long.346
  * Rostamzadeh et al. (2018) Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojciech Stokowiec, Ying Zhang, Christian Jauvin, and Chris Pal. 2018\.  Fashion-gen: The generative fashion dataset and challenge.  _arXiv preprint arXiv:1806.08317_ (2018).
  * Rostamzadeh et al. (2021) Negar Rostamzadeh, Ben Hutchinson, Christina Greer, and Vinodkumar Prabhakaran. 2021.  Thinking Beyond Distributions in Testing Machine Learned Models. In _NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications_.
  * Ruiz et al. (2022) Nataniel Ruiz, Adam Kortylewski, Weichao Qiu, Cihang Xie, Sarah Adel Bargal, Alan Yuille, and Stan Sclaroff. 2022.  Simulated Adversarial Testing of Face Recognition Models.  _CVPR_ (2022).
  * Sambasivan et al. (2021a) Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021a.  Re-Imagining Algorithmic Fairness in India and Beyond. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 315–328.  https://doi.org/10.1145/3442188.3445896
  * Sambasivan and Holbrook (2018) Nithya Sambasivan and Jess Holbrook. 2018.  Toward responsible AI for the next billion users.  _Interactions_ 26, 1 (2018), 68–71.
  * Sambasivan et al. (2021b) Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021b.  “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In _proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Sánchez-Gordón et al. (2020) Mary Sánchez-Gordón, Laxmi Rijal, and Ricardo Colomo-Palacios. 2020.  Beyond Technical Skills in Software Testing: Automated versus Manual Testing. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_. 161–164.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_. 142–147.
  * Scheuerman et al. (2021) Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021\.  Do datasets have politics? Disciplinary values in computer vision dataset development.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2 (2021), 1–37.
  * Schlangen (2021) David Schlangen. 2021\.  Targeting the Benchmark: On Methodology in Current Natural Language Processing Research. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. 670–674.
  * Schwartz et al. (2020) Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020.  Green AI.  _Commun. ACM_ 63, 12 (2020), 54–63.
  * Sculley et al. (2018) David Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. 2018.  Winner’s curse? On pace, progress, and empirical rigor. In _Proceedings of ICLR 2018_.
  * Selbst et al. (2019) Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019\.  Fairness and abstraction in sociotechnical systems. In _Proceedings of the conference on fairness, accountability, and transparency_. 59–68.
  * Shue (2020) Henry Shue. 2020\.  _Basic rights: Subsistence, affluence, and US foreign policy_.  Princeton University Press.
  * Silberman et al. (2018) M Six Silberman, Bill Tomlinson, Rochelle LaPlante, Joel Ross, Lilly Irani, and Andrew Zaldivar. 2018\.  Responsible research with crowds: pay crowdworkers at least minimum wage.  _Commun. ACM_ 61, 3 (2018), 39–41.
  * Sinnott-Armstrong (2021) Walter Sinnott-Armstrong. 2021\.  Consequentialism.  _The Stanford Encyclopedia of Philosophy_ Winter 2021 Edition (2021).  https://plato.stanford.edu/archives/win2021/entries/consequentialism/
  * Star and Griesemer (1989) Susan Leigh Star and James R Griesemer. 1989.  Institutional ecology, ‘translations’ and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39.  _Social studies of science_ 19, 3 (1989), 387–420.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019\.  Energy and Policy Considerations for Deep Learning in NLP. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 3645–3650.
  * Sugiyama et al. (2007) Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. 2007.  Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation.  _Advances in Neural Information Processing Systems_ 20 (2007).
  * Thomas and Uminsky (2020) RL Thomas and D Uminsky. 2020.  Reliance on metrics is a fundamental challenge for AI. In _Proceedings of the Ethics of Data Science Conference_.
  * Tukey (1962) John W Tukey. 1962\.  The future of data analysis.  _The annals of mathematical statistics_ 33, 1 (1962), 1–67.
  * Turney (1994) Peter D Turney. 1994\.  Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm.  _Journal of artificial intelligence research_ 2 (1994), 369–409.
  * Ustalov et al. (2017) Dmitry Ustalov, Alexander Panchenko, and Chris Biemann. 2017\.  Watset: Automatic induction of synsets from a graph of synonyms. In _55th Annual Meeting of the Association for Computational Linguistics, ACL 2017_. Association for Computational Linguistics, 1579–1590.
  * Vallor (2016) Shannon Vallor. 2016\.  _Technology and the virtues: A philosophical guide to a future worth wanting_.  Oxford University Press.
  * Van Rijsbergen (1974) Cornelis Joost Van Rijsbergen. 1974\.  Foundation of evaluation.  _Journal of documentation_ (1974).
  * Vogelsang and Borg (2019) Andreas Vogelsang and Markus Borg. 2019.  Requirements engineering for machine learning: Perspectives from data scientists. In _2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)_. IEEE, 245–251.
  * Wallach (2018) Hanna Wallach. 2018\.  Computational social science$\neq$ computer science+ social data.  _Commun. ACM_ 61, 3 (2018), 42–44.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018\.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. 353–355.
  * Wang et al. (2021) Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. 2021\.  Generalizing to Unseen Domains: A Survey on Domain Generalization. In _Proceedings of IJCAI 2021_.
  * Webster et al. (2019) Kellie Webster, Marta R Costa-jussà, Christian Hardmeier, and Will Radford. 2019.  Gendered ambiguous pronoun (GAP) shared task at the Gender Bias in NLP Workshop 2019. In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_. 1–7.
  * West et al. (2019) Sarah Myers West, Meredith Whittaker, and Kate Crawford. 2019\.  Discriminating systems.  _AI Now_ (2019).
  * Winkens et al. (2020) Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. 2020\.  Contrastive Training for Improved Out-of-Distribution Detection.  _arXiv e-prints_ (2020), arXiv–2007.
  * Wu et al. (2021) Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. 2021.  Fashion iq: A new dataset towards retrieving images by natural language feedback. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11307–11317.
  * Yeom et al. (2018) Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018.  Privacy risk in machine learning: Analyzing the connection to overfitting. In _2018 IEEE 31st Computer Security Foundations Symposium (CSF)_. IEEE, 268–282.
  * Zeng et al. (2021) Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Zixian Ma, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. 2021.  OpenAttack: An Open-source Textual Adversarial Attack Toolkit. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_. 363–371.
  * Zhang et al. (2020a) Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020a.  Machine learning testing: Survey, landscapes and horizons.  _IEEE Transactions on Software Engineering_ (2020).
  * Zhang et al. (2020b) Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020b.  Adversarial attacks on deep-learning models in natural language processing: A survey.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 11, 3 (2020), 1–41.
  * Zhao et al. (2020) Benjamin Zi Hao Zhao, Mohamed Ali Kaafar, and Nicolas Kourtellis. 2020.  Not one but many tradeoffs: Privacy vs. utility in differentially private machine learning. In _Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop_. 15–26.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018.  Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Vol. 2.

## Appendix A: Metrics in ML Model Evaluations

Here we give definitions and categorizations of some of the metrics reported
in the study in Section 3. In practice, there was a long tail since many
metrics were used in only a single paper. Here we include only the metrics
which were most frequently observed in our study.

Table 5. Definitions and categorizations of metrics reported in Section 3. TP, TN, FP and FN indicate the number of true positives, true negatives, false positives and false negatives, respectively. $y$ and $\hat{y}$ represent actual values and values predicted by the system, respectively. Metric |  Example Task(s) |  Metric category |  Definition
---|---|---|---
Accuracy |  Classification |  Accuracy |  A metric that penalizes system predictions that do not agree with the reference data ($\frac{TP+TN}{TP+TN+FP+FN}$).
AUC |  Classification |  AUC |  The area under the curve parameterized by classification threshold $t$, typically with $y$-axis representing recall and $x$-axis representing false positive rate ($\frac{FP}{FP+TN}$).
Bleu |  Machine translation |  Precision |  A form of “$n$-gram precision,” originally designed for machine translation but also sometimes used for other text generation tasks, which measures whether sequences of words in the system output are also present in the reference texts (Papineni et al., 2002).
Dice |  Image segmentation |  Overlap |  Equivalent to $F_{1}$ ($\frac{2TP}{2TP+FP+FN}$). More commonly used for medical image segmentation.
Error rate |  Classification |  Accuracy |  The inverse of accuracy ($1-accuracy=\frac{FP+FN}{TP+TN+FP+FN}$).
$F$ (or $F_{1}$) |  Text classification |  Overlap |  The harmonic mean of recall and precision ($\frac{2PR}{P+R}$), originally developed for information retrieval (Van Rijsbergen, 1974) but now widely used in NLP.
$F_{0.5}$ |  Text classification |  Overlap |  A weighted harmonic mean of recall and precision, with greater weight given to recall ($(1+\beta^{2})\frac{PR}{\beta^{2}P+R}$ with $\beta=0.5$).
Hausdorff distance |  Medical Image Segmentation |  Distance |  A measure of distance between two sets in a metric space. Two sets have a low Hausdorff distance if every point in each set is close to a point in the other set.
IoU |  Image segmentation |  Overlap |  $\frac{TP}{TP+FP+FN}$. Equivalent to Jaccard.
Matthew’s Correlation Coefficient |  |  Correlation |  Has been argued to address shortcomings in $F_{1}$’s asymmetry with respect to classes ($\frac{TP.TN-FP.FN}{\sqrt{(TP+TN)(TP+FN)(TN+FN)(TN+FP)}}$).
Mean absolute error |  Regression |  Distance |  $\frac{1}{N}\sum_{i=1}^{N}|\hat{y}_{i}-y_{i}|$
Mean Average Precision (MAP) |  Information retrieval (NLP) |  AUC |  In information retrieval, the average over information needs of the average precision of the documents retrieved for that need.
Mean average precision (mAP) |  Object detection (CV) |  AUC |  The area under the Precision-Recall tradeoff curve, averaged over multiple IoU (intersection over union) threshold values, then averaged across all categories (https://cocodataset.org/#detection-eval).
Mean reciprocal rank |  Information retrieval |  Other |  A measure for evaluating processes that produces an ordered list of possible responses. The average of the inverse rank of the first relevant item retrieved.
MSE |  Image Decomposition |  Distance |  Mean squared error (MSE) measures the average of the squared difference between estimated and actual values.
Normalized Discounted Cumulative Gain (NDCG) |  Recommendation or ranking tasks |  Other |  A measure of ranking quality which takes into account the usefulness of items based on their ranking in the result list.
Pearson’s $r$ |  Quality Estimation |  Correlation |  A measure of linear correlation between two sets of data.
Perplexity |  Language modeling |  Perplexity |  Information-theoretic metric (measured in bits-per-unit, e.g., bits-per-character or bits-per-sentence) often used for language models, inversely related to the probability assigned to the test data by the model. Closely related to the cross-entropy between the model and the test data. Can be thought of as how efficiently does the language model encode the test data.
Precision |  Classification |  Precision |  A metric that penalizes the system for predicting a class (if class is unspecified, by default the “positive” class) when the reference data did not belong to this class ($\frac{TP}{TP+FP}$).
PSNR |  Super Resolution |  Distance |  Peak Signal-to-Noise ratio (PSNR) is the ratio between the maximum possible value of a signal and the power of distorting noise (Mean Squared Error) that impacts the quality of its representation.
Recall |  Classification |  Recall |  Also known as “sensitivity”, this metric that penalizes the system for failing to predict a class (if class is unspecified, by default the “positive” class) when the reference data did belong to this class ($\frac{TP}{TP+FN}$); a.k.a. true positive rate.
RMSE |  Depth Estimation |  Distance |  Root Mean Square Error (RMSE) is the square root of the MSE.
Rouge |  Text summarization |  Recall |  A form of “$n$-gram recall,” originally designed for text summarization but also sometimes used for other text generation tasks, which measures whether sequences of words in the reference texts are also present in the system output(Lin, 2004).
Spearman’s $\rho$ |  Graph Edit Distance |  Correlation |  A measure of monotonic association between two variables–less restrictive than linear correlations.
Specificity |  Classification |  Other |  Like Precision, this metric that penalizes the system for failing to predict a class (if class is unspecified, by default the “positive” class) when the reference data did belong to this class; unlike Precision it rewards true negatives rather than true positives ($\frac{TN}{TN+FN}$).
SSIM |  Super Resolution |  Distance |  The Structural Similarity Method (SSIM) is a perception-based method for measuring the similarity between two images. The formula is based on comparison measurements of luminance, contrast, and structure.
Top-$n$ accuracy |  Face recognition |  Accuracy |  A metric for systems that return ranked lists, which calculates accuracy over the top $n$ entries in each list.
Word error rate |  Speech recognition |  Accuracy |  The inverse of word accuracy: $1-word\ accuracy$ (which is not technically always in $[0,1]$ due to the way word accuracy is defined but which is categorized as “Accuracy” here because both insertions and deletions are penalized).

## Appendix B: Types of Evaluation Data used in ML Model Evaluations

Table 6. Types of datasets used in ML model evaluations. Type of Test Data |  Example Task(s) |  I.I.D. with training data? |  Definition
---|---|---|---
Test split |  Classification |  yes |  Typically, labeled data is partitioned into training and test splits (and often a dev split too), drawn randomly from the same dataset.
Manual resource |  Lexical acquisition |  no |  A manually compiled resource (in NLP, often a word-based resource such as a lexicon or thesaurus), against which knowledge acquired from a dataset is compared.
References |  Machine translation |  no |  Reference outputs (typically obtained prior to building the system) which a generative system is trying to reproduce, typically obtained from humans (e.g., manual translations of input sentences in the case of evaluations using Bleu for machine translation tasks).
Training data |  Keyword extraction |  yes |  Training data that contains labels is used to evaluate an unsupervised algorithm that did not have access to the labels during learning.
Novel distribution |  Domain transfer |  no |  Test data that has the same form as the training data but is drawn from a different distribution (e.g., in the case of NLP training on labeled newspaper data and testing on labeled Wikipedia data).

## Appendix C: Example of Assumptions and Gaps for a Hypothetical Application

Suppose we are evaluating a hypothetical image classification model for use in
an application for assisting blind people in identifying groceries in their
pantries. Then some application-specific questions related to the assumptions
in Section 4 might be:

#### Consequentialism

Was data ethically sourced and labeled? Were blind people involved in the
design process? Does this use of this model encourage high-risk uses of other
similar models, such as identifying pharmaceutical products?

#### Abstractability from Context

Does the application have a human-in-the-loop feature available when the model
is uncertain? Will the system nudge purchasing behaviors towards products on
which the model performs well?

#### Input Myopia

Are uncommon grocery products misclassified more often? Does this
disproportionately impact home cooks who don’t stick to the dominant cuisines,
or who have food requirements due to medical conditions?

#### Quantitative Modeling

Does measuring predictive accuracy fail to take into account dignitary
consequences associated with increased independence? Should each user be
weighted equally in the evaluation (cf. each image)?

#### Equivalent Failures

Are there severe risks in confusing certain pairs of products, e.g., food
products associated with dangerous allergies? Are some errors only minimally
inconvenient, such as confusing different shapes of pasta?

#### Test Data Validity

Is the evaluation data representative of what the application’s users have in
their pantries? Are the image qualities (lighting, focus, framing, etc.)
representative of images taken by blind photographers?

## Appendix D: Model Evaluation Remits and Design

Model Evaluation Remit
---
To establish:
motivation — why evaluate the model?
what is the perspective being adopted —
task/financial/administrative/scientific/…
whose interests prompted the evaluation — developer/funder/…
who are the consumers of the model evaluation results —
manager/user/researcher/…
goal — what do we want/need to discover?
orientation — intrinsic/extrinsic
kind — investigation/experiment
type — black box/glass box
form (of yardstick) — ideal/attainable/exemplar/given/judged
style — suggestive/indicative/exhaustive
mode — quantitative/qualitative/hybrid
Model Evaluation Design
To identify:
ends — what is the model for? what is its objective or function?
context — what is the ecosystem the model is in? what are the animate and
inanimate factors?
constitution — what is the structure of the model? what was the training data?
To determine:
factors that will be tested
environment variables
‘system’ parameters
evaluation criteria
metrics/measures
methods
Evaluation data — what type, status and nature?
Evaluation procedure
Table 7. A sketch of how Karen Sparck Jones and Julia Galliers’ 1995 NLP
evaluation framework questionnaire (Jones and Galliers, 1995) can be adapted
for the evaluation of ML models. The output of the remit and the design is a
strategy for conducting the model evaluation. For a related but simpler
framework based on model requirements analysis, see also the “7-step Recipe”
for NLP system evaluation
(https://www.issco.unige.ch/en/research/projects/eagles/ewg99/7steps.html)
developed by the eagles Evaluation Working Group in 1999, which considers
whether different parties have a shared understanding of the evaluation’s
purpose.