# The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment

Nari Johnson<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA ,  Sanika
Moharana<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA ,  Christina
N. Harrington<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA ,
Nazanin Andalibi<EMAIL_ADDRESS>University of MichiganUSA ,  Hoda
Heidari†<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA and  Motahhare Eslami†
<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA

(2024; 22 January 2024)

###### Abstract.

As more algorithmic systems have come under scrutiny for their potential to
inflict societal harms, an increasing number of organizations that hold power
over harmful algorithms have chosen (or were required under the law) to
_abandon_ them. While social movements and calls to abandon harmful algorithms
have emerged across application domains, little academic attention has been
paid to studying abandonment as a means to mitigate algorithmic harms. In this
paper, we take a first step towards conceptualizing “ _algorithm abandonment_
” as an organization’s decision to stop designing, developing, or using an
algorithmic system due to its (potential) harms. We conduct a thematic
analysis of real-world cases of algorithm abandonment to characterize the
dynamics leading to this outcome. Our analysis of 40 cases reveals that
campaigns to abandon an algorithm follow a common process of six iterative
phases: _discovery, diagnosis, dissemination, dialogue, decision,_ and _death_
, which we term the _6 D’s of abandonment_. In addition, we highlight key
factors that facilitate (or prohibit) abandonment, which include
characteristics of both the technical and social systems that the algorithm is
embedded within. We discuss implications for several stakeholders, including
proprietors and technologists who have the power to influence an algorithm’s
(dis)continued use, FAccT researchers, and policymakers.

abandonment, contestation, refusal, accountability

††journalyear: 2024††copyright: rightsretained††conference: ACM Conference on
Fairness, Accountability, and Transparency; June 3–6, 2024; Rio de Janeiro,
Brazil††booktitle: ACM Conference on Fairness, Accountability, and
Transparency (ACM FAccT ’24), June 3–6, 2024, Rio de Janeiro, Brazil††doi:
10.1145/3630106.3658910††isbn: 979-8-4007-0450-5/24/06††ccs: Human-centered
computing Empirical studies in HCI

##  1\. Introduction

00footnotetext: † Both senior authors contributed equally to this research.

An increasing number of incidents have demonstrated how algorithmic systems
can inflict societal harm, either by upholding oppressive power structures
(Noble, 2018; Benjamin, 2019), reinforcing stereotypes (Vincent, [n. d.];
Andalibi et al., 2023), or making life-altering decisions through inscrutable
logic (Selbst and Barocas, 2018; Liban, 2018; Forrest, 2021; Eppink, 2023). In
response to these harms, a growing body of research has explored how harmful
algorithms might be _repaired_ through technical or procedural interventions
(Schwennesen, 2019; Bembeneck et al., 2021; Orphanou et al., 2022). However,
scholars and activists alike have argued that not all algorithmic harms can be
effectively addressed or mitigated through these kinds of repairs (Harcourt,
2006; Milner, 2020; Wang et al., 2023). This stance often arises from a
recognition of the deeply ingrained nature of the corresponding harm, where
mere adjustments fail to resolve fundamentally systemic issues. In some
instances, resistance from impacted communities has precipitated the _fall_ of
an algorithmic system, in which its proprietors might choose (or be forced to)
_abandon_ its use. Take, for example, a case from Arkansas: when a new
resource allocation algorithm resulted in a drastic cut to the number of in-
home care hours that MedicAid beneficiaries would receive (Hub, 2022a), one
beneficiary wrote in with a clear request: to _turn it off_ , and return to
the old system of human-based assessment. No one could offer an explanation
for why this beneficiary’s in-home care hours were reduced, cuts that for many
beneficiaries with chronic disabilities resulted in drastic changes to their
quality of life. While three years later, the algorithm (called RUGS) was
abandoned, the road to this outcome was long and tedious. Efforts to abandon
the algorithm were met with resistance from state officials (Weil and Edwards,
2023). The eventual decommissioning of RUGS after it was invalidated by a
state court (Hub, 2022a) was a result of concerted advocacy efforts by
impacted individuals, legal advocates, and media attention.

Figure 1. Example event timelines for three example algorithms (Watch, 2023;
in Pittsburgh, 2020a; Hoover, 2023). We visualize the time elapsed to each
event since the systems’ release date (x-axis), and label important events by
tagging them with one (or multiple) phase emojis from the 6D’s of abandonment
(see legend, Bottom Right). We provide detailed descriptions of each event in
Appendix C. Two of the pictured algorithms (Pittsburgh’s predictive policing
pilot and NEDA’s Tessabot) were abandoned and had a “Death” phase. Jordan’s
Takaful algorithm was _not_ abandoned and is still in use at the time of
writing (denoted by the red arrow into the present). We found that the time
elapsed between events (e.g., for the algorithm to be discovered, or be
abandoned after discovery) varied considerably across cases.

While some campaigns to abandon harmful algorithms have been successful, other
harmful algorithms continue to remain in use today. Furthermore, while some
algorithms were abandoned in early ideation phases _before_ they were deployed
and some were abandoned within mere hours of harm discovery, other algorithms
continue(d) to inflict harm for years. Thus, open questions remain when
considering the possibility of abandoning harmful algorithms as a mechanism of
harm mitigation and accountability: What types of events preceded and
precipitated the decision to abandon an algorithm? How did impacted
communities and other actors work together to contest harmful algorithms? What
technical, political, or social characteristics of real-world algorithmic
deployments posed barriers for contestation, accountability, and abandonment?
How can future calls for abandonment be bolstered and supported?

In this paper, we take a first step toward answering these questions by
exploring the concept of _algorithm abandonment_. We define algorithm
abandonment as a _decision_ made by actors with jurisdiction over the system,
to discontinue the process of developing, deploying, or using the algorithm
due to its (potential) harms. The decision to abandon an algorithm can be made
at any point in the algorithms’ development lifecycle (Hub, 2022d; Suresh and
Guttag, 2021), and includes both decisions to not build a proposed technology
and decisions to stop using tools that are already built. The algorithmic
systems that we explore span diverse ways of coming into being (Wang et al.,
2023), including hand-coded rules and machine learning models. Our study of
algorithm abandonment draws from literature from Science and Technology
Studies which examines algorithms not just as technical artifacts mapping
inputs to outputs, but rather as socio-technical “assemblages” of social
relationships that enable and sustain technical artifacts (Schwennesen, 2019).
Thus, understanding abandonment requires examining the social, political, and
economic systems that algorithms are embedded _within_ (Seaver, 2019).

In our thematic analysis of 40 different campaigns to abandon harmful
algorithms in recent years, we observe that the fall of an algorithm (or “
_dynamics_ ” leading to abandonment) follows a consistent pattern of six
iterative, non-linear phases which we term the 6 “D”s of abandonment:
_discovery, diagnosis, dissemination, dialogue, decision,_ and _death_ , which
we visualize for three example cases in Figure 1. We review each of these
phases and highlight both tactics that algorithm owners used to avoid scrutiny
(e.g., discrediting or ignoring criticism) and tactics that other actors such
as impacted individuals, advocacy organizations, journalists, or regulators,
used to shift power to advocates of abandonment. We additionally identify key
_socio-technical factors_ most relevant to the outcome of whether an algorithm
was ultimately abandoned (and how long it took to reach this stage), such as
public awareness of and access to the algorithm, technicalities of the
algorithm’s implementation, visibility of criticism, the regulatory
environment, and for whom (and how) the algorithm is marketed. We conclude by
discussing implications for technologists designing algorithmic systems,
regulators and oversight entities, and the FAccT community in identifying
barriers to abandonment and interventions to support future calls to abandon
harmful algorithms. We release our annotated case set
online111https://njohnson99.github.io/fall-of-algorithm-database/ as a
resource to aid future research and education.

##  2\. Background & Related Work

We contextualize our study of abandonment using scholarship that studies how
different stakeholders respond to algorithmic harm. “Algorithmic harm,” as
described by Shelby et al. (2023), refers to “ _the adverse lived experiences
resulting from a system’s deployment and operation in the world_ ” which
occurs through the interplay of technical components and societal power
dynamics (Noble, 2018; Benjamin, 2019; McGregor, 2020). We group approaches
taken to address algorithmic harm into two broad categories: (1) actions taken
“from above” by organizations in positions of power (e.g., those that develop,
oversee, or govern the algorithm’s use), and (2) actions taken “from below” by
other stakeholders (e.g., those impacted by algorithmic harm or the public).
We adopt the language of “above” and “below” from Whitney et al. (2021), who
uses these terms to articulate the “ _tensions between the wants, needs, and
knowledge of unequally empowered stakeholders_ ”.

We situate our study of abandonment as an organizational decision made “from
above” by algorithm owners or government oversight entities. We use the term
“algorithm owner” to broadly refer to the organizations and people within them
that have the power to influence an algorithm’s (dis)continued use on the
ground. Our analysis of the events, or “dynamics”, that precede abandonment in
practice draws from the growing body of scholarship that examines how actors
“from below” organize to contest harmful algorithmic systems (Zong and Matias,
2023; DeVrio et al., 2023). We use the term “impacted communities” to describe
the people who are affected by the use of an algorithm. Those impacted may
include the consumers of a technology, or the people who are subject to an
algorithm’s decisions (Friedman et al., 2006; Moss et al., 2021).

Repair from above. Much work concerning algorithmic harm explores the actions
that algorithm owners might take to repair flawed technologies. The AI/ML and
FAccT community have proposed numerous technical methods to promote values of
“fairness” and “explainability” (Doshi-Velez and Kim, 2017; Selbst and
Barocas, 2018; Liao and Vaughan, 2023). This work has led to the development
of metrics (Speicher et al., 2018; Pessach and Shmueli, 2022; Barocas et al.,
2023; Subhash et al., 2022), methods (Dwork et al., 2012; Wang and Rudin,
2015; Hardt et al., 2016; Ribeiro et al., 2016), and and toolkits (Bellamy et
al., 2018; Arya et al., 2019; Deng et al., 2022; Weerts et al., 2023) that
practitioners can use to measure and “fix” specific harmful behaviors. Beyond
technical repairs, human-computer interaction researchers have documented how
organizational users and decision-makers engage in their own ad-hoc or
improvisational repair activities when they encounter flawed technologies
(Houston et al., 2016; Eslami et al., 2017; Schwennesen, 2019). For example,
social workers using AI tools to screen calls can override them to mitigate
disparities in final outcomes (De-Arteaga et al., 2020; Cheng and
Chouldechova, 2022; Cheng et al., 2022). Such methods can be used to correct
the algorithm’s flaws, often defined in terms of narrowly scoped measures
(Jacobs and Wallach, 2021), while maintaining the algorithm’s operation.

In both research and policy proposals, repair has emerged as a dominant
approach to address algorithmic harm from above (Lee et al., 2019; Davis et
al., 2021; Bembeneck et al., 2021). Both governments and private-sector
organizations have instituted policies requiring mitigation strategies to
repair potential algorithmic harms (of Canada, 2023; Microsoft, 2022).
However, not all algorithmic harms are readily fixable, particularly in cases
where critics take issue with the very existence of the system itself.
Critical scholarship has scrutinized the use of algorithms in specific domains
(e.g., predictive policing (Khan and White, 2021; Milner, 2020) or criminal
sentencing (Forrest, 2021)), and questioned if such algorithms can ever be
repaired. Instead of examining how flawed technologies might be repaired, we
focus our inquiry on abandonment, which is predicated on questioning if an
algorithm should exist in the first place.

Refusal from below. The concept of refusal, defined simply as the act of
saying “no” (Ahmed, 2017), has been studied in feminist and de-colonial
discourses (Simpson, 2007; TallBear, 2013; McGranahan, 2016; Honig, 2021).
Today, scholars and activists continue to draw from these traditions in
studying refusal of new technologies (Cifor et al., 2019; DeVrio et al., 2023;
Zong and Matias, 2023). Among theories of anti-techno-solutionism, Benjamin
(2016) refers to indigenous refusal to analyze vignettes where research
subjects refuse to engage with new biotechnologies. A group of scholars
drafted the Feminist Data “Manifest-No”, a declaration of refusal to harmful
data regimes and exploitative research practices (Cifor et al., 2019). There
is a rich history of grassroots social movements formed to refuse
technologies, tracing back hundreds of years to the British Luddites during
the Industrial Revolution (Conniff, 2011), sparking modern adaptations of
“Luddism” by workers organizing to oppose harmful algorithmic management
practices (Chayka, 2023; Merchant, 2023).

Refusal of harmful technologies may take the form of individual users choosing
to no longer interact with harmful algorithms (DeVrio et al., 2023; Zong and
Matias, 2023). For example, in response to Amazon’s introduction of AI-powered
cameras for worker surveillance, some drivers engaged in an act of refusal by
obscuring the cameras with stickers, rendering them unable to record (Gurley,
2021). But refusal also extends beyond individual acts, into collective calls
for a system to be abandoned. Understanding calls to abandon an algorithm
through the lens of refusal sheds light on the possibilities that abandonment
entails to “ _open up possibilities for power shifts_ ” (Casemajor et al.,
2015).

Conceptualizing & contextualizing abandonment. In this paper, we conceptualize
algorithm abandonment as “a decision made _from above_ (e.g., by algorithm
proprietors, vendors, purchasers, or governance bodies), to discontinue
developing, deploying, or using a harmful algorithmic system, usually due to
calls _from below_ ”. We focus our study on abandonment as an _organizational_
decision (rather than that of individual user decisions) as many systems are
implemented in a way where opting out is difficult or impossible (Lecher,
2018; System, 2021; amn, 2022). We differentiate abandonment from repair in
that while repair considers how one might improve an existing algorithmic
system, abandonment entails doing away with specific types of automation
altogether. This conceptualization of abandonment builds upon scholarship
(Baumer and Silberman, 2011; Dombrowski et al., 2016; Homewood, 2019; Howell
et al., 2021; Stapleton et al., 2022) focused on “ _when the implication is
not to design ([a] technology)_ ” and advocates for technologists to adopt “
_a reflective awareness for when [automation] is inappropriate or harmful_ ”
(Baumer and Silberman, 2011).

##  3\. Methodology

We adopt a case study approach to analyze documented cases where stakeholders
called for abandonment in recent years. Our goal was to understand the
dynamics of algorithm abandonment by studying the events that occurred
throughout these calls. As such, we first gathered real-world cases of
algorithms that were (called to be) abandoned, and then conducted an iterative
thematic analysis.

Case collection. To develop our dataset, we conducted an extensive search of
various sources for instances where an algorithm that caused harm was either
(a) abandoned, or (b) critics _called for_ the algorithm to be abandoned,
regardless of whether it was abandoned or not. Our search methodology was a
dynamic, iterative process that unfolded concurrently with our study. This
approach involved consulting a diverse group of resources, detailed further
below:

  * •

_Initial dataset_ : We began with an initial set of cases that the authors
were familiar with from research articles and reading lists focused on
resisting and refusing algorithmic harm (Burrell and Mulligan, 2020; Shen et
al., 2021; Shelby et al., 2023; Zong and Matias, 2023; DeVrio et al., 2023).
We also conducted web searches with keywords such as “algorithm abandonment”,
“algorithmic bias”, and “algorithmic harm”.

  * •

_AI incident databases_ : We reviewed all of the reported incidents to
date222We conducted our review from July to December 2023, and reviewed all
incidents that had been posted in each database before January 1, 2024. in
three incident databases that compiled and documented real-world harms caused
by algorithmic systems: (a) the “AI Incident Database” (AIID) (McGregor,
2020), (b) the “AI, Algorithmic, and Automation Incidents and Controversies”
(AIAAIC) repository (aia, [n. d.]), and (c) “Not My AI” (Varon and Peña, 2022)
which focuses on mapping harmful algorithms in Latin America.

  * •

_Expert consultations_ : We also identified additional cases through
conversations with experts who had academic or lived experiences relating to
algorithm abandonment. Specifically, our research team met with members of an
independent U.S.-based advocacy group founded to help impacted communities
resist oppressive surveillance technologies. We also met with academic
colleagues who have done research on topics relating to resisting and
responding to algorithmic harm. These experts pointed us to relevant cases
that they had encountered in their work.

  * •

_International cases_ : Given that U.S. and European-based incidents were
over-represented in our initial case set, we extended our search to be
intentionally inclusive of other regions globally. We conducted additional
keyword searches on Google (detailed in Appendix B) to identify relevant cases
that occurred in countries outside of Western contexts, such as Korea (Kwon
and Yun, 2021), Argentina (Aplicada, 2018), Jordan (Ryan-Mosley, 2023a), and
India (amn, 2022).

Example Cases (domain) |  Description
---|---
U.S. National Eating Disorder Assocation (NEDA)’s _Tessa_ Wellness Bot (Wells, 2023) (chatbot) |  Shortly after NEDA fired their hotline staff, eating disorder activists posted on social media about how Tessa (NEDA’s chatbot) gave harmful and triggering advice.
Facial Recognition in NYC Atlantic Plaza Towers (Gagne, 2019) (facial recognition) |  Residents of a Brooklyn apartment complex partnered with a legal group to resist their landlord’s plan to install facial recognition technology to “secure” access to the building.
The Spanish Ministry of the Interior’s _VioGén_ Gender Violence Recidivism Risk Assessment (eti, 2022) (family and social services) |  The Spanish government’s risk assessment algorithm under-estimated risks posed by perpetrators, withholding necessary protection services that led to adverse outcomes in domestic violence cases.
Table 1.  Three example cases from our list of 40 total case studies of real-
world examples of campaigns to abandon an algorithm. See the complete list of
40 cases in Appendix A.

#### Case set

From the cases that met the abandonment criteria, we selected a subset of 40
as our case studies for our thematic analysis (which we describe in the
following section). We chose this subset based on several criteria, including
a) coverage of different domains, b) access to enough sources to gain more
information about a case to do a thematic analysis, and c) reaching saturation
in finding new themes in the analysis process. The selected cases included
algorithms used in social media, chatbots, hiring, family and social services,
benefits allocation, healthcare, education, property tech, and law
enforcement. We chose among cases within the same domain based on the amount
of public information and documentation that was available for us to construct
accurate event timelines. We performed our thematic analysis in parallel to
identifying relevant cases, and stopped adding new cases to the subset once we
had reached saturation in finding new themes (i.e., the types of events and
factors relevant to abandonment). Our final case set contains 40 different
algorithmic systems, 24 of which had been successfully abandoned at the time
of writing333Some algorithms that we study are deployed in multiple different
contexts (e.g., PredPol was used by dozens of U.S. police departments
(Haskins, 2019)), and were abandoned in some, but not others. We ‘count’
products like PredPol as a single “algorithmic system”, and count them as
being “abandoned” if they were abandoned in at least one such context. , while
the remainder were still in use. 16 out of the 40 total cases involved
campaigns that took place in countries outside of the U.S. To provide context,
we describe three example cases in Table 1 and a complete list in Appendix A.
These 40 cases are not intended to be a comprehensive list of all algorithms
that were called to be abandoned, but rather serve as a starting point to
conceptualize algorithm abandonment, illustrate its dynamics, and map out its
implications.

Thematic analysis. We gathered and reviewed public resources about each case,
including news articles, research publications, social media posts, and any
other relevant primary sources (e.g., algorithm source code or court
documents). For each case, we constructed a timeline by noting the month and
year of important events. Our goal in noting important events was not to be
comprehensive, but rather to provide “ _good enough_ ” coverage of the types
of events leading to abandonment, a standard used in past qualitative research
(Yildirim et al., 2023). We reviewed sources to code metadata about the
algorithm, such as its application domain, the type(s) of algorithmic harm
(Shelby et al., 2023), and other information relevant to whether it was
ultimately abandoned (detailed in Appendix B).

We conducted a bottom-up thematic analysis (Braun and Clarke, 2006) of case
event timelines and metadata in a series of meetings among research team
members. In our thematic analysis of event timelines, we paid particular
attention to the sequencing of key events, the timelapse between them, and how
each event contributed to the final decision of whether the system was
abandoned. To identify factors relevant to abandonment presented in Section 5,
we noted differences among algorithms that were abandoned almost immediately
(e.g., within hours of their release), versus algorithms that took months or
even years to be abandoned. We organized these differences into higher-level
categories that resulted in the final seven factors. We also noted the causes
for (call for) abandonment, different barriers that advocates of abandonment
faced when asserting their role in this accountability relationship, and the
broader implications for research on algorithmic accountability. We walk
through the set of sources and resulting codes and event timelines for an
example case in Appendix B.1 . Our thematic analysis yielded six high-level
types of events (or “phases”) as the dynamics of the algorithm abandonment
that we describe in Section 4.

Limitations. In our study of the events that influenced abandonment, we
analyzed _publicly available_ materials such as news articles or information
obtained through public records requests. We speculate, however, that there
were other reasons why organizations chose to abandon the algorithm beyond
those named in public sources, such as pressure from company shareholders
(Whittaker, 2019; Klar, 2023) or financial projections (Puron-Cid and Gil-
Garcia, 2022; Mok, 2023). Future work that engages more directly with people
involved in making the decision to abandon an algorithm may shed light on
their motives.

##  4\. The Dynamics Toward Algorithm Abandonment

We explore the _dynamics_ of abandonment by proposing a process-oriented view
of the events leading to abandonment. We visualized these dynamics for
individual cases by creating a _timeline of events_ (shown in Figure 1) that
occurred until a system was abandoned (or until now for systems that are in
use). In our thematic analysis, we found that most algorithms in our case set
were abandoned _after_ they were deployed. However, our case set also includes
cases of algorithms abandoned earlier in their development lifecycle. These
cases provide meaningful comparison points to consider how algorithmic harms
might be prevented.

The 6 D’s of abandonment. In our analysis of event timelines for the 40
identified cases, the process leading to abandonment involved events that fell
into six iterative phases of (1) discovery, which initiated critique and calls
to abandon the system by the public, impacted communities, or their advocates,
(2) diagnosis, where these actors gathered further evidence about the system
and its impacts, (3) dissemination, where these actors amplified their message
to a wider audience, (4) dialogue between critics of the system and algorithm
owners, until they reached a (5) decision about whether to repair or abandon
the system. We discuss how many abandoned systems still continued to impact
peoples’ lives in the final (6) death phase, in which algorithm owners
transitioned away from the algorithm and those impacted sought justice for
past harms. Note that all six phases may occur in any stage of an algorithm’s
lifecycle (e.g., design, development, usage), and are still relevant for
algorithms that are not yet developed or deployed. In practice, the “fall” of
an algorithm often proceeds iteratively and non-linearly throughout these six
phases. Similarly, not all six phases may occur for every case. Regardless,
this process-oriented view provides a useful lens to examine what types of
events were influential across a range of cases.

###  4.1. Discovery

Many cases had a catalyzing event where information that was previously
unknown about the algorithm was made public and initiated wide-scale critique.
We group these discoveries into two categories: (1) discovering the
_existence_ of a previously obscure algorithmic system; and (2) discovering
potential or realized algorithmic _harms_.

Discovery of existence. In some cases, criticism began once the algorithm’s
mere existence was made public. While algorithm owners could in theory provide
public notice of their intentions to develop or use an algorithm in early
stages of its lifecycle, many did not provide notice until the system was
already in use. For example, the City of Pittsburgh “ _experimented with_ ”
using a predictive policing algorithm for over a year before the public was
made aware of its existence through the release of a public report (in
Pittsburgh, 2020a). Several other algorithms such as the New York Police
Department’s contract with facial recognition software vendor, Clearview AI
(Haskins, 2021), evaded critique for months due to a lack of public notice.
Discoveries of an algorithms’ existence sparked criticism about a lack of
community engagement and oversight (in Pittsburgh, 2020a) and the values
encoded in algorithms (Hill, 2020a; log, 2020; Khan and White, 2021).

Discovery of algorithmic harm. In other cases, initial critiques began after a
discovery of potential or realized algorithmic harms. To understand what types
of harms motivated calls to abandonment, we mapped all mentions of harms in
case materials to the five major types of sociotechnical harms in Shelby et
al. (2023) taxonomy (defined in Appendix D), i.e., representational,
allocative, quality-of-service, interpersonal, and social system harms.
Overall, the algorithms in our case set spanned all five of Shelby et al.
(2023)’s harm categories. Algorithms deployed in similar domains resulted in
similar harms, e.g., chatbots could demean and stereotype (Kwon and Yun, 2021;
Kraft, 2016), and resource allocation algorithms could result in opportunity
loss (Ryan-Mosley, 2023a; Hub, 2022a). Several different types of harm often
co-occurred for each single algorithmic system.444For example, San Diego’s
TACIDS algorithm (System, 2021; Williams, 2015) matched photographs of
suspects against mugshot databases, where people with previous law enforcement
encounters were more likely to be (falsely) flagged (representational harm).
TACIDS was used to aid immigration enforcement efforts that separated families
(interpersonal harm), and had implications for residents’ constitutional
rights (social system harms). Finally, critics identified harms that
implicated more than just the algorithms’s _functionality_ , or ability to
work as intended (Raji et al., 2022). In other words, simply improving the
algorithm to be more accurate or to reduce what Shelby et al. (2023) deem
“Quality-of-Service” performance disparities would not redress all harms. A
more effective surveillance technology, for example, could still contribute to
civil rights violations or interpersonal harms, such as accelerating
incarceration, or family separation at scale (Wang et al., 2022; System,
2021). While many cases we considered did involve dysfunctional algorithms,
the remaining taxonomy categories remind us that other harms exist too.

###  4.2. Diagnosis

As the number of people aware of an algorithms’ existence and potential harms
grew, so did their capacity to _diagnose_ its potential to cause (further)
harm. We group critics’ investigations into two categories: audits that
analyzed the algorithms’ _outputs_ , and investigations into _human_ decisions
(such as design choices) surrounding the algorithm.

Examining the output: algorithm auditing. Critics attempted to better
understand the behavior of an algorithm by interacting with or probing it
directly to observe its output. Many such audits involved querying the
algorithm on several (real or simulated) inputs, and measuring properties of
its output. Large-scale targeted audits were a highly effective tool that
precipitated decisions to abandon an algorithm. For example, many
organizations referenced the Gender Shades audit (which measured racial and
gender accuracy disparities of commercial facial recognition algorithms
(Buolamwini and Gebru, 2018)) when they later instituted moratorium policies
on the sales or usage of facial recognition technology (Buolamwini and Gebru,
2023). While large-scale audits were highly effective, in several cases,
critics faced significant obstacles to conduct such audits. One dimension that
significantly influenced the auditability of an algorithm was access to the
model, as algorithms that were administered by a decision-maker internal to
the organization (e.g., a social worker assessing child welfare cases (Ho and
Burke, 2022)) could not be queried. Independent auditors also often lacked
access to realistic and representative data of the population of interest. In
some cases, the public was even unaware of what information the algorithm took
as input to make decisions. For example, in Jordan, government officials
refuse to release the “socio-economic indicators” of poverty that an algorithm
which allocated cash assistance to citizens took into consideration (Watch,
2023).

To overcome these obstacles, in some cases critics worked together to
collectively develop work-around strategies. One strategy to obtain access to
realistic input data was to design coordinated data collection efforts from
consenting people who would be or were already impacted by an algorithmic
system. In Missouri, a coalition of legal aid organizations and third-party
researchers partnered with home care nurses to fill out assessments to
estimate a proposed home care allocation algorithms’ impact for 1200 patients
(Weil and Edwards, 2023). In Spain, an independent nonprofit conducted phone
interviews to understand the experiences of women who completed an algorithmic
gender-based violence risk assessment that determined the police and legal
resources allocated to them (eti, 2022). Successful data donation initiatives
gave auditors access to population-level data that was critical to
understanding the algorithm’s behavior.

Examining the process: design decisions. Algorithmic systems do not just “come
about”, but are rather the product of a series of _decisions made by_
designers, developers, and deployers (Selbst et al., 2024; Metcalf et al.,
2023). Critics attempted to understand algorithmic systems by examining the
important decisions that brought them into being. For example, concerned
Carnegie Mellon University students who opposed Pittsburgh’s use of a
predictive policing model filed a Right to Know request that revealed that the
algorithm was rolled out without any meaningful consultation with the
communities it was used to police (in Pittsburgh, 2020a). Several advocacy
groups have created resources to help people collect information about human
decisions and processes surrounding algorithmic systems. For algorithms used
by public agencies, organizations including the Electronic Frontiers
Foundation (“EFF”) (eff, 2023) and Lucy Parsons Labs (lpl, 2023) have
developed resources to help people navigate filing public records requests
(which in some cases only resulted in partial information being released (in
Pittsburgh, 2020a; Marx and Irani, 2021)).

###  4.3. Dissemination

Broader dissemination was a common strategy employed by critics to raise
public awareness and put public pressure on algorithm owners. Dissemination
sometimes caught the attention of elected officials or regulators who could
apply pressure or exercise authority from above (Ho and Burke, 2022; Warren,
2023; Ho and Burke, 2023). Some campaigns led by advocates of abandonment took
place _in-person_ (offline), whereas in other cases campaigns took place
_online_. Other cases were supported by global media campaigns, such as the
#BanTheScan coalition supporting grassroots activist groups’ efforts to ban
government use of facial recognition technologies in NYC, Jerusalem, and
Hyderabad (amn, 2022). In many instances (Liban, 2018; in Pittsburgh, 2020b;
ap2, 2022; was, [n. d.]), only after the visibility of criticism increased did
algorithm owners respond to and acknowledge the critique.

Online vs. Offline. While less common, a small number of cases involved in-
person _public demonstrations_. All of the public demonstrations in our case
set were organized to oppose systems that were intended to be deployed in a
specific _locality_ (e.g., in a particular city (Chang, 2018)). Many such
demonstrations were organized and attended by people who were directly
impacted by the algorithm, such as student-organized protests against
algorithmically assigned exam grades in the UK (Porter, 2020), or a resident-
led protest of a proposed facial recognition algorithm held on the sidewalk
outside of their apartment (Elizalde and Gartland, 2019). In contrast,
criticism of tools with spatially diffuse user bases took place _online_.
Several cases gained visibility through high engagement on online platforms,
such as Twitter, Instagram, or YouTube by _going “viral”_ , such as users’
interactions with NEDA’s Tessa Bot (Wells, 2023), or the Korean chatbot Lee-
Luda (Kwon and Yun, 2021). We found that highly visible cases (as approximated
through engagement metrics, such as high reaction or re-posts counts) were
significantly more likely to receive a timely response from the algorithm
owner. Social media campaigns involved broad participation from a wide range
of stakeholders, who shared reactions to other posts, shared their own
interactions with the system, or simply liked or shared to boost its
visibility.

Getting press. In several cases, media coverage of criticism played a critical
role in the fall of an algorithm. As one example, Oregon’s DHS decided just
two weeks after an Associated Press story (Ho and Burke, 2022) to halt a pilot
on child maltreatment risk prediction that had been in progress for four
years. US Senator Ron Wyden had “ _reached out_ ” to the department
immediately after the story to “ _ask questions about [the tool’s] racial
bias_ ” (ap2, 2022). Similar reporting efforts helped activist efforts gain
momentum, and caught the attention of those in positions of power who applied
pressure from above.

###  4.4. Dialogue

When criticism about an algorithm became more publicly visible, actors who
held power over the algorithm were called to respond in a public _dialogue_
with their critics. These dialogues occurred in sites ranging from those that
are more formal or compulsory for algorithm owners (e.g., in court (Hub,
2022a, b; van Veen, 2020; Pierson and Pierson, 2023; Mari, 2022)), to those
that were more informal (e.g., on social media (Yee et al., 2021; Dickey,
2020; Vincent, [n. d.]; Wells, 2023;
https://www.theguardian.com/profile/justinmccurry, [n. d.])). Some algorithm
owners took weeks, months, or even years to acknowledge critics and respond
_at all_. In this section, we examine how algorithm owners (dis)engaged with
criticism about the algorithm. While some algorithms were primarily criticized
by the public, other algorithms found support from other actors beyond their
owners, and sparked a broader discourse about whether or not they should
exist.

Denial vs. acceptance of algorithmic harm. When presented with evidence of
algorithmic harm, some algorithm owners responded by _discrediting_ the
evidence presented to them, either by creating counter-evidence (Eubanks,
2018) or ad-hominem attempts to discredit the critic (Hao, 2020; Xiang, 2023).
For example, when an Instagram user shared an infographic that described a
harmful and triggering interaction with NEDA’s Tessa Wellness
chatbot555https://www.instagram.com/p/Cs1jp1pPkOs/, NEDA’s VP of
Communications accused her of lying in the post’s comment section (Xiang,
2023). Interestingly, in many cases (Hao, 2020; Xiang, 2023), initial denials
were later retracted or withdrawn due to public scrutiny. After the NEDA VP’s
comment sparked outrage from other users, she deleted her comment and posted
that she “ _retracted her statement_ ” (Xiang, 2023). This pattern of denial
followed by retraction is a broader trend in responses to attempts for
algorithmic accountability (Raji and Buolamwini, 2022). In contrast, rather
than resist, some algorithm owners immediately acknowledged the validity of
the critiques. For example, when Twitter users began to investigate if its
image cropping algorithm was racially biased (Yee et al., 2021), Twitter
immediately issued a public statement that they were ”prioritizing work” to
remedy bias in their algorithms (Agrawal and Davis, 2020). Cooperative
responses that immediately acknowledged critiques allowed organizations to
take actions (including abandonment) to redress harms more quickly.

The public vs. private dialogue. One dimension that influenced whether an
algorithm was abandoned was whether algorithm owners responded in private.
Cases that were resolved interpersonally rather than in public were
significantly less likely to result in critiques being resolved or the system
being abandoned. For example, while Zoom users observed in their personal
experiences (Shen et al., 2021) that Zoom’s virtual background feature tended
to be more dysfunctional for users with darker skin, prompting some to tweet
about it, Zoom did not respond until one of these
tweets666https://twitter.com/colinmadland/status/1307111816250748933 went
viral. Zoom Corporate replied to the Tweet to invite its author to a call.
Later that week, the author of the tweet wrote that he had “ _had a great
convo with [Zoom’s Chief DEI Officer] […] I have no reason to doubt that they
are serious about fixing problems with virtual backgrounds […] the problem is
deep and nuanced_ ”. Yet, such private resolutions still leave the broader
public in the dark about what commitments the algorithm owner has made.

Opposing the abandonment/supporting the algorithm. As discourse surrounding an
algorithm became increasingly visible, not all people believed that the
algorithm should be abandoned. For example, calls to ban police use of facial
recognition technologies worldwide were met with concern from law enforcement
officials. In the U.S., police unions and conservative advocacy groups
invested resources in lobbying against bans of facial recognition technologies
(Gavin, 2022; Ryan-Mosley, 2023b). Similarly, in Uganda, local police and
members of the majority party lauded the facial recognition systems they
purchased from Chinese vendor Huawei for “ _boosting security_ ” (Wilson and
Murgia, 2019). Beyond law enforcement officials, even citizens support the use
of facial recognition in some contexts: for example, 60% of surveyed Indian
citizens supported the use of facial recognition to identify protesters (com,
2023). These examples demonstrate how the debate to abandon an algorithm often
implicates the larger surrounding political contexts and historic ideals
(e.g., surveillance, nationalism, and criminalization) that animate new
technologies.

###  4.5. Decision

Those who held power over the algorithm could _make decisions_ to intervene on
its design, development, or usage. We group these decisions about the
algorithm into 4 categories based on their outcome: whether the algorithm was
_not_ abandoned (in which the algorithm owners could (1) _repair_ the
algorithm, or (2) make no changes), or _was_ abandoned (3) by the algorithm
owner’s choice, or (4) by law or policy. In some cases, multiple decisions
were made, e.g., many algorithm owners first attempted to repair an algorithm
before later deciding to abandon it.

Repair. Algorithm owners took several actions to repair or remediate potential
harms, including fixing or updating dysfunctional technology (Liban, 2018;
Vaithianathan et al., 2019; Ross, 2021) or creating better infrastructure or
policies around its use (Dastin, 2020; Sisitzky and Schaefer, 2021; Ross,
2021). For example, when independent investigations revealed flaws in Epic’s
sepsis prediction model (which was deployed in several hundred U.S. hospitals)
(Wong et al., 2021), the company completely “overhauled” its original
algorithm and released a second version that used completely different input
and target variables, and recommended that the model be trained on a
hospital’s _own_ data rather than being applied off-the-shelf (Ross, 2021).
But repairs that improved the technology by making it more functional or less
biased did not redress harms that resulted from the algorithms’ existence.

Abandoned by the algorithm owner’s choice. In our analysis, the majority of
systems that were abandoned were abandoned by choice, i.e., by a decision made
by an organization that developed or used the algorithm. To name a few, Korean
start-up ScatterLab disabled their Lee-Luda chatbot (Kwon and Yun, 2021),
Twitter decided to stop using an algorithm to crop users’ images (Chowdhury,
2021), and leadership at Oregon’s Department of Human Services decided to stop
using a child welfare call screening algorithm (ap2, 2022). Relatively few
algorithm owners released public statements with explanations or named reasons
why they decided to abandon the algorithm. Public statements that we found
referenced algorithmic harms as a primary reason why the system was abandoned
(Lee, 2016; Chowdhury, 2021; Kwon and Yun, 2021; Hoover, 2023).

Abandoned through policy. Algorithms overseen by uncooperative algorithm
owners could still be abandoned through another governing body creating or
enforcing applicable laws. Several algorithms in our case set were deemed
illegal through the interpretation of existing laws (including laws written
without an algorithm in mind) in a court of law. For example, a Dutch court
ruled the government to stop using an algorithm that falsely predicted that
thousands of low-income residents had committed benefits fraud, after finding
that the algorithm violated citizens’ right to privacy in the European
Convention on Human Rights (aw2, 2020). U.S. courts ruled MedicAid benefits
allocation algorithms in violation of existing laws (i.e., due process law and
the Administrative Procedure Act) in both Arkansas and Idaho (Calo and Citron,
2021; Eppink, 2023). Other algorithms were banned by existing governing bodies
founded to protect citizens’ rights, such as the U.S. Federal Trade Commission
issuing a five-year ban on Rite Aid’s use of facial recognition because of the
technology’s risks to consumers (ftc, 2023). However, all past cases of
systems abandoned through executive or judicial enforcement took a
_significantly longer time_ (e.g., several years (Liban, 2018; Eppink, 2023))
before the decision to abandon the algorithm was reached. These cases also
required significant investment - e.g., the legal team representing Idaho
plaintiffs spent over $40,000 on expert consultations and 2,000 attorney hours
to secure a settled agreement (Eppink, 2023).

Algorithmic systems were also abandoned through the _creation of new policies_
by a governing body (e.g., a state legislature or city council) that had
jurisdiction over the algorithm’s design, development, or usage. New laws that
target algorithmic systems have been rapidly growing in countries across the
world (iap, 2023). For example, several U.S. state legislatures have proposed
bans on algorithms developed for specific purposes, including facial
recognition (Ryan-Mosley, 2023b), betting or casino algorithms (Hanchett,
2022), employee monitoring (Wilkinson and Lukens, 2023), or providing mental
health counseling (dm2, 2023). All relevant policies that we identified for
our cases targeted banning the _usage_ (rather than the _development_) of
technology.

###  4.6. Death

The _death_ phase explores what happens _after_ an algorithm is abandoned.
Because algorithm development often entails creating infrastructure (such as
data collection pipelines or portable products) (Ehsan et al., 2022), we
discuss how abandoned algorithms’ impacts can “live on” by influencing future
processes (Ehsan et al., 2022), or in “reincarnated” products deployed
elsewhere. Our understanding of the afterlife of abandoned algorithmic systems
builds upon Ehsan et al. (2022)’s conceptualization of the _algorithmic
imprint_ , which describes how an algorithm’s impact can extend beyond its
deployment, by influencing infrastructural conditions, organizational
practices, and societal norms that continue on in its “afterlife”.

Overriding past decisions. Past decisions issued by an algorithm can continue
to negatively impact people who received adverse outcomes. In the majority of
cases of algorithms that were abandoned after their deployment, we found that
the algorithm owners did _not_ re-consider or override past (harmful)
decisions made using the algorithm. For example, to our knowledge, rejected
resumes were not re-screened (Dastin, 2018) and closed child welfare
investigations were not re-opened (ap2, 2022). In a few notable cases,
however, decisions were overturned and re-issued en masse. For example, in
response to global protests against algorithm-assigned A-level grades, U.K.
exam boards globally rescinded all algorithm-assigned grades and revised them
based on teacher-assessed grades (without the algorithmic standardization
under criticism (Ehsan et al., 2022)). Similarly, a legal settlement in Idaho
guaranteed that people whose home care benefits were cut by an algorithm would
continue to receive their original budgets (Hub, 2022b).

Making reparations. Beyond the rare re-issuing of decisions, some algorithm
owners provided other kinds of reparations to symbolically or materially mend
past harms. Material reparations, for example, may be especially appropriate
for systems that caused allocative harm that made individuals incur additional
financial costs, e.g., by withholding access to healthcare benefits or
employment. The majority of algorithm owners did not provide financial
reparations for the harm caused, with a few notable exceptions. The Dutch
government paid €30,000 to all families falsely accused of committing benefits
fraud by an algorithm (Erdbrink, 2021). The Arkansas DHS was required to pay
$460,000 in a court settlement to three plaintiffs whose benefits were
severely cut, but many other hundreds of beneficiaries who were similarly
impacted by the algorithm were not compensated (Ly, 2023). No algorithm owners
in our case set provided reparations for other types of non-material harms
(e.g., representational, psychological, or interpersonal harms).

Reincarnation. A growing number of investigations uncover how both successful
and abandoned technologies can re-appear in different contexts (Kalluri et
al., 2023; Loewenstein, 2023; Scott, 2023). We observed that when a city
cancelled a contract with a technology vendor, the company often remained and
continued to sell their product elsewhere (Haskins, 2019, 2021). Even as
cities such as Palo Alto decided to drop their contracts with the predictive
policing company PredPol, public records requests revealed that PredPol still
had (many previously undisclosed) contracts with dozens of U.S. cities
(Haskins, 2019). In response to bad press, some algorithm owners rebranded
(sto, 2021): e.g., one month after the LAPD stopped using PredPol, the company
changed its name to Geolitica and its slogan from “ _The Predictive Policing
Company_ ” to “ _Data-Driven Community Policing_ ” (Bhuiyan, 2021). In 2023,
SoundThinking (the company that created ShotSpotter, another prominent law
enforcement algorithm) acquired IP from Geolitica, and still maintains
versions of its predictive policing algorithms (Mehrotra and Cameron, 2023).
The case of PredPol exemplifies how built technologies are valued as
infrastructure that remains and can be recycled even after they are abandoned.

##  5\. Factors relevant to algorithm abandonment

Our analysis of historic cases across a range of domains reveals several
patterns, including common phases that took place toward abandonment. However,
beyond identifying commonalities, we also noticed key _differences_ relevant
to the outcome of whether the system was ultimately abandoned, and how long it
took to get to that stage. In this section, we single out these
_differentiating socio-technical factors_ , which span characteristics of the
technical system and the social systems that the algorithm was embedded
within. We highlight the implications of each factor for several stakeholders.
For example, because abandonment is an important step that algorithm owners
can take to redress (potential) algorithmic harms, we discuss ways that
algorithm owners can design algorithmic systems that _can be more easily
abandoned_ at different stages of their lifecycle. We also discuss
implications of each factor for other external stakeholders, including FAccT
researchers and oversight entities.

Users vs. Subjects. _Is the algorithm a consumer product, or used on (unaware)
subjects?_ Algorithms in consumer products (e.g., online platforms) ”fell”
more quickly than algorithms used by organizations (e.g., governments) to
analyze or make decisions about subjects who were potentially unaware of the
algorithm’s existence. Direct users or consumers who interacted with
technologies were more likely to be aware that there is an algorithm in place
and have more agency to resist harmful interactions with it. In cases where
the direct user who consumed the algorithm’s outputs used it to make decisions
about someone else (e.g., a policing agency using facial recognition on city
residents), impacted individuals were often unaware of its use or existence,
which prolonged the process leading to abandonment. This distinction draws
attention to the importance of understanding power differentials in who
purchases and uses, versus who is most (negatively) impacted by algorithmic
systems. Algorithm owners and critics alike should aim to empower and amplify
the perspectives of those most impacted by algorithmic systems, even if they
are not direct users.

Value proposition. _What is the ”value proposition” behind the use of the
algorithm?_ A critical factor relevant to cases’ outcomes was understanding
algorithm owners’ motivation for creating or adopting the algorithm, i.e., its
value proposition. Publicly available value propositions often articulated a
clear set of political values, e.g., to reduce MedicAid spending (by cutting
care) (Weil and Edwards, 2023), to maximize returns to landlords (by raising
rents) (Vogell et al., 2022), or to enable constant and wide-scale public
surveillance (Dastin, 2020). In some cases, algorithm owners valued that
automation allowed them to de-personalize decisions formerly made subjectively
by humans, e.g., that leasing agents or nurse assessors had “ _too much
empathy_ ” (Vogell et al., 2022) to raise rents or cut hours. When an
algorithm’s value proposition was clearly stated, in some cases critics
organized successful campaigns that countered them (e.g., challenging the
notion that surveillance makes us feel safe (log, 2020)). Algorithm owners
should make an algorithm’s value proposition clear to the public. Critics
should search for value propositions (i.e., how is the technology being
marketed to its buyers? (Roemmich et al., 2023)) and make them visible so that
they can be contested.

The ”domino effect” caused by deep dependencies. _How embedded is the
algorithm into other systems? What other practices and processes ”depend” on
it?_ Stand-alone algorithms released as independent products, like chatbots,
are much easier to abandon than those embedded into larger social or technical
systems, like algorithms used to automate important bureaucratic decisions
(Alkhatib and Bernstein, 2019). Abandoning an algorithm that other components
rely on might have a domino effect that requires changing its surrounding
socio-technical components. For example, when courts ruled algorithms that
assessed beneficiaries’ eligibility illegal, public officials scrambled to
develop a new process (e.g., train human assessors) to allocate care resources
(Hub, 2022a). Algorithm owners should aim to minimize dependencies by creating
and budgeting for back-up plans for a scenario where the algorithm is no
longer in use (a proposal that’s become popular in artificial intelligence
governance (of Canada, 2023; of Science and Policy, 2022)).

Procedural transparency and public notice. _Is the public aware that an
algorithm exists? Do we know how important decisions were made?_ We found that
many harmful algorithms were used for months (or even years) before they were
discovered and the general public was made aware of their existence. In
contrast, when algorithm owners made their plans to use an algorithm public
before it was deployed (like Missouri’s NF LOC algorithm (Hub, 2022c)), in
some cases they decided to abandon the algorithm before it could inflict harm
in the real world. Thus, the extent to which algorithm owners made information
about the algorithm public, and when, played a pivotal role in the time it
took for harmful systems to be abandoned. Algorithm owners should provide
public notice and publicly release information about the algorithm’s design at
the earliest possible stages of the algorithm’s lifecycle. FAccT researchers
and advocacy groups can contribute to the small (yet growing) number of tools
identified by Raji et al. (2024) designed to aid “target identification”,
i.e., to “ _uncover deployed systems by making them visible to external
stakeholders_ ”. Finally, policymakers or organizations can institute
transparency requirements for algorithm owners (e.g., those proposed in
(Reisman et al., 2018; Moss et al., 2021; Gehami and Doty, 2023)).

Access and auditability. _What access do independent actors,_ e.g., _members
of the public, have to the algorithm?_ As discussed in Section 4.2, while
independent audits were an extremely effective tool that triggered the
decision to abandon an algorithm, independent actors had varying degrees of
access to different algorithmic systems. FAccT researchers can contribute to
infrastructure and build tools to support independent audits, such as those
taxonomized by (Raji et al., 2024). One promising trend is that coalitions of
different actors have started collaborations to share knowledge and data about
their interactions with algorithmic systems, such as academic or non-profit
research groups supporting public defenders and worker’s unions (Weil and
Edwards, 2023; Hub, 2022c; FairFare, 2023). Policymakers and advocacy groups
can consider funding these independent audits (Mozilla, 2023).

Visibility of criticism and timely media coverage. _How visible is the
critique to other stakeholders?_ As discussed in Sections 4.3 and 4.4,
critics’ efforts to abandon an algorithm were often intertwined with a broader
struggle for visibility. Getting media coverage that uplifted critics’
concerns and hopes for the future was often a turning point for accelerating
an algorithm’s fall. Other cases that went viral from organic engagement (such
as amplification from influential users) were also significantly more likely
to result in algorithm owners taking action to address algorithmic harms (Yee
et al., 2021; Hoover, 2023). Thus, journalists, media organizations, and
“everyday users” (Shen et al., 2021) can play an important role in advocating
for abandonment.

Oversight and regulatory environment. _Are policymakers receptive to creating
new policies (within their political interests)? Who is held liable for
algorithmic harms?_ As discussed in Section 4.5, instituting new “bright-line
rules” (ain, 2023) that explicitly ban specific types of algorithms takes far
less time (and material investment) compared to interpreting existing laws in
court. Policymakers can consider instituting such bans _proactively_ , even
before the banned systems are in use, like the City of Somerville’s ban on
employee’s use of facial recognition (Wu, 2019). Another trend we observed is
a lack of accountability under the law for vendors that develop algorithmic
systems and sell them to organizations that oversee their deployment. For
example, when courts ruled Arkansas’ roll-out of an algorithm developed by
interRAI (an independent vendor) illegal, the state of Arkansas, not interRAI,
was required to pay the full settlement amount to plaintiffs who were harmed
(Ly, 2023). This accountability structure means that even when one
organization decides to stop using a harmful algorithm, it can continue to be
used elsewhere with little consequence to the vendor who would continue to
benefit from the algorithm (Lima et al., 2023). Policymakers should create and
interpret existing laws to promote what Metcalf (2023) calls “ _distributed
accountability_ ”, where vendors are also held liable for remedying
algorithmic harms.

##  6\. Conclusion & Future Work

Grassroots movements, moratorium bans, and boardroom decisions to abandon
harmful algorithmic systems continue to grow across the world. Yet, little
academic attention has been paid to understanding the phenomena of algorithm
abandonment. In our analysis of real-world cases involving (calls for)
algorithm abandonment, we identified common phases of events that took place
and key socio-technical factors that enabled (or prolonged) abandonment
efforts. Our analysis reveals power imbalances and information asymmetries
that prevent critics from holding algorithm owners accountable. Yet, victories
from below in our case set illustrate how these power differentials might be
challenged through policy interventions and collective action. In our
discussion of relevant factors, we highlight steps that algorithm owners can
take to create algorithms that can be more easily be contested and abandoned.
Regulators can consider making these steps mandatory, such as requiring
algorithm owners to make important information public at early stages of
design and development (Reisman et al., 2018).

Our preliminary work points to several exciting directions for future work on
algorithm abandonment. Researchers can take inspiration from successful
collaborations that have built tooling and infrastructure to enable collective
contestation and auditing, such as partnerships between academic researchers
and worker’s unions (FairFare, 2023). Future research efforts can conduct more
in-depth studies, such as interviews, to understand the experiences of actors
involved in making the decision to abandon an algorithm, as well as other
important stakeholders such as impacted communities, advocacy groups, and
policymakers. Future work can also center campaigns that go beyond calls to
_abandon_ , and instead fight to _abolish_ harmful technologies. In her
conceptualization of “ _abolitionist technologies_ ”, Ruha Benjamin writes
that “ _calls for abolition are never simply about bringing harmful systems to
an end but also about envisioning new ones. After all, the etymology of
abolition includes Latin root words for ‘destroy’ (abolere) and ‘grow’
(olere)_ ” (Benjamin, 2019). In our analysis, we found that algorithms in our
case set implicate oppressive social systems or politics that informed their
design. How can the FAccT community can better support the many advocacy
initiatives (Benjamin, 2019; log, 2020; Milner, 2020; sto, 2021) that work to
contest the values that animate harmful algorithmic systems, and imagine more
just futures?

##  7\. Impact Statements

Researcher Positionality. We acknowledge that our lived experiences shape our
positionality and perspectives (Liang, 2021) in the study of algorithmic
harms, which are often disproportionately faced by members of marginalized
groups. As researchers who were trained and work in the U.S., we recognize
that our analysis, particularly our discussions of algorithmic accountability,
contestation (e.g., through public protests), and governance was influenced by
the present political context of the U.S. In our initial search for cases, we
noticed that the majority of cases cited in FAccT research articles or
reported in crowd-sourced AI incident databases occurred in the U.S. Further,
because our search was conducted in English, we likely missed relevant cases
that occurred in non-English-speaking regions. We attempted to address this
regional bias by performing additional keyword searches and including an
additional Latin America-based incident database (Varon and Peña, 2022). We
hope that our exploratory research raises academic awareness of algorithm
abandonment as a _global_ phenomena, and inspires future work that more deeply
interrogates similarities and differences relevant to how the dynamics of
abandonment play out across regional contexts.

While all authors have experience studying algorithmic harm as academic
researchers, it was important for us to better understand the needs and
obstacles faced by activists taking action to resist oppressive technologies
on the ground. Thus, as we conducted this research, we consulted experts who
had lived experiences organizing to resist algorithmic harm. Specifically, our
research team met with members of the Surveillance Resistance Lab, a
U.S.-based advocacy group founded to resist state and corporate surveillance
technologies, who directed us to cases and offered feedback. Looking forward,
our research team plans to translate parts of this work to create resources
that can directly benefit advocacy groups.

Ethical Considerations. In proposing a process-oriented view of the dynamics
of algorithm abandonment, our analysis reveals commonalities that place
seemingly dissimilar cases (of chatbots (Hoover, 2023), B2B SAAS products
(Vogell et al., 2022), and carceral technologies (System, 2021)) in
conversation with one another. These similarities enable us to formulate
recommendations for future research and policy interventions related to
algorithm abandonment. However, our analysis has several limitations. In
focusing on what is _shared_ between a large and diverse set of algorithmic
systems that are deployed in a wide variety of political contexts, we exclude
important details that illustrate the uniqueness and complexity of each case.
Further, given the rapidly evolving nature of algorithmic capabilities and
harms, we acknowledge that our process-oriented view may fail to describe the
events leading to the fall of all algorithmic systems (including recent
generative, multi-modal, and multi-purpose AI systems). Because of these
concerns, the 6Ds of abandonment are _not_ meant to serve as a model of steps
for critics to follow, but rather provide descriptive context to understand
similar historic struggles.

Adverse Impact. Our aim in identifying factors that permit (or hinder)
algorithm abandonment is to support actors _from below_ in contesting harmful
algorithmic systems. To do so, we name and draw attention to common actions
that actors _from above_ took to dodge and discredit their critics. However,
this work also has the potential to be misused by algorithm owners who model
their actions after historic cases that successfully reinforced oppressive
power structures (e.g., by intentionally obscuring an algorithm). Despite this
risk, we advocate for the value of naming and identifying these tactics as a
tool to strengthen the eco-system of accountability around algorithmic
systems.

###### Acknowledgements.

We thank Mizue Aizeki and Ed Vogel from the Surveillance Resistance Lab for
offering feedback on this work. We thank Ryan Steed, Meg Young, Upol Ehsan,
and Sohini Upadhyay for directing us to related scholarship and cases. We
thank Alana Rogers and Ayush Tripathi for conducting preliminary research that
informed this project. HH and NJ acknowledge support from NSF (IIS2040929 and
IIS2229881), PwC (through the Digital Transformation and Innovation Center),
and the Block Center for Technology and Society at CMU. Any opinions,
findings, conclusions, or recommendations expressed in this material are those
of the authors and do not reflect the views of the National Science Foundation
and other funding agencies.

## References

  * (1)
  * aia ([n. d.]) [n. d.].  _AI, Algorithmic, and Automation Incidents and Controversies (AIAAIC) Repository_.  Retrieved January 18, 2024 from https://www.aiaaic.org/aiaaic-repository
  * gal ([n. d.]) [n. d.].  Galactica Demo — galactica.org.  https://galactica.org/.  [Accessed 22-01-2024].
  * was ([n. d.]) [n. d.].  New liver transplant rules yield winners, losers as wasted organs reach record high — washingtonpost.com.  https://www.washingtonpost.com/business/2023/03/21/liver-transplants-acuity-circle-policy/.  [Accessed 22-01-2024].
  * aw2 (2020) AlgorithmWatch 2020.  _How Dutch activists got an invasive fraud detection algorithm banned_.  AlgorithmWatch.  Retrieved January 16, 2024 from https://algorithmwatch.org/en/syri-netherlands-algorithm/ Email newsletter.
  * log (2020) Logic(s) Magazine 2020.  _Safe or Just Surveilled?: Tawana Petty on the Fight Against Facial Recognition Surveillance_.  Logic(s) Magazine.  Retrieved January 16, 2024 from https://logicmag.io/security/safe-or-just-surveilled-tawana-petty-on-facial-recognition/ Interview.
  * sto (2021) Stop LAPD Spying Coalition (Ed.). 2021.  _The Ghosts of White Supremacy in AI Reform_.  https://ainowinstitute.org/publication/a-new-ai-lexicon-surveillance
  * amn (2022) Amnesty International (Ed.). 2022.  _Ban the Scan_.  Retrieved January 16, 2024 from https://banthescan.amnesty.org/
  * eti (2022) Eticas 2022.  _The External Audit of the VioGen System_.  Eticas.  https://eticasfoundation.org/wp-content/uploads/2022/03/ETICAS-FND-The-External-Audit-of-the-VioGen-System.pdf
  * ap2 (2022) The Associated Press (Ed.). 2022.  _Oregon is dropping an artificial intelligence tool used in child welfare system_.  Retrieved January 16, 2024 from https://www.npr.org/2022/06/02/1102661376/oregon-drops-artificial-intelligence-child-abuse-cases
  * ain (2023) AI Now Institute (Ed.). 2023.  _Algorithmic Accountability: Moving Beyond Audits_.  Retrieved January 16, 2024 from https://ainowinstitute.org/publication/algorithmic-accountability
  * eff (2023) Electronic Frontier Foundation 2023.  _FOIA How To_.  Electronic Frontier Foundation.  Retrieved January 16, 2024 from https://www.eff.org/issues/transparency/foia-how-to
  * iap (2023) International Association of Privacy Associates 2023.  _Global AI Legislation Tracker_.  International Association of Privacy Associates.  Retrieved January 16, 2024 from https://iapp.org/resources/article/global-ai-legislation-tracker/
  * dm2 (2023) Duane Morris Government Strategies (Ed.). 2023.  _Regulating Artificial Intelligence In Mental Health: States’ Attempts To Revolutionize Mental Health Services_.  Retrieved January 16, 2024 from https://statecapitallobbyist.com/artificial-intelligence-ai/regulating-artificial-intelligence-in-mental-health-services-states-attempts-to-revolutionize-mental-health-services/
  * ftc (2023) The U.S. Federal Trade Commission 2023.  _Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards_.  The U.S. Federal Trade Commission.  Retrieved January 16, 2024 from https://www.ftc.gov/news-events/news/press-releases/2023/12/rite-aid-banned-using-ai-facial-recognition-after-ftc-says-retailer-deployed-technology-without Press Release.
  * com (2023) Common Cause & Lokniti – Centre for the Study Developing Societies (CSDS) 2023.  _Status of Policing in India Report 2023: Surveillance and the Question of Privacy_.  Common Cause & Lokniti – Centre for the Study Developing Societies (CSDS).  Retrieved January 16, 2024 from https://www.commoncause.in/wotadmin/upload/REPORT_2023.pdf
  * lpl (2023) Lucy Parsons Labs 2023.  _Who We Are_.  Lucy Parsons Labs.  Retrieved January 16, 2024 from https://lucyparsonslabs.com/about/
  * Agrawal and Davis (2020) Parag Agrawal and Dantley Davis. 2020.  _Transparency around image cropping and changes to come_.  https://blog.twitter.com/en_us/topics/product/2020/transparency-image-cropping
  * Ahmed (2017) Sarah Ahmed. 2017.  _No_.  Retrieved January 16, 2024 from https://feministkilljoys.com/2017/06/30/no/
  * Alkhatib and Bernstein (2019) Ali Alkhatib and Michael Bernstein. 2019.  Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300760
  * Andalibi et al. (2023) Nazanin Andalibi, Cassidy Pyle, Kristen Barta, Lu Xian, Abigail Z. Jacobs, and Mark S. Ackerman. 2023.  Conceptualizing Algorithmic Stigmatization. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (, Hamburg, Germany,) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 373, 18 pages.  https://doi.org/10.1145/3544548.3580970
  * Aplicada (2018) Laboratorio de Inteligencia Artificial Aplicada. 2018.  Sobre la predicción automática de embarazos adolescentes.
  * Arya et al. (2019) Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilović, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, and Yunfeng Zhang. 2019.  One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques.  arXiv:1909.03012 [cs.AI]
  * Barocas et al. (2023) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023.  _Fairness and Machine Learning: Limitations and Opportunities_.  MIT Press.
  * Baumer and Silberman (2011) Eric P.S. Baumer and M. Six Silberman. 2011.  When the Implication is Not to Design (Technology). In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (, Vancouver, BC, Canada,) _(CHI ’11)_. Association for Computing Machinery, New York, NY, USA, 2271–2274.  https://doi.org/10.1145/1978942.1979275
  * Bellamy et al. (2018) Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. 2018.  AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias.  arXiv:1810.01943 [cs.AI]
  * Bembeneck et al. (2021) Emily Bembeneck, Rebecca Nissan, , and Ziad Obermeyer. 2021.  _Algorithmic Bias Playbook_.  https://www.ftc.gov/system/files/documents/public_events/1582978/algorithmic-bias-playbook.pdf
  * Benjamin (2016) Ruha Benjamin. 2016.  Informed Refusal: Toward a Justice-based Bioethics.  _Science, Technology, & Human Values_ 41, 6 (2016), 967–990.  https://doi.org/10.1177/0162243916656059 arXiv:https://doi.org/10.1177/0162243916656059
  * Benjamin (2019) Ruha Benjamin. 2019.  _Race after technology: Abolitionist tools for the new Jim code_.  Polity.
  * Bhuiyan (2021) Johana Bhuiyan. 2021.  _LAPD ended predictive policing programs amid public outcry. A new effort shares many of their flaws_.  Retrieved January 16, 2024 from https://www.theguardian.com/us-news/2021/nov/07/lapd-predictive-policing-surveillance-reform
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative Research in Psychology_ 3, 2 (2006), 77–101.  https://doi.org/10.1191/1478088706qp063oa arXiv:https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa
  * Bukowitz and O’Loughlin (2022) Rachel Bukowitz and Tim O’Loughlin. 2022.  Governing by Algorithm? Child Protection in Aotearoa New Zealand.  https://anzsog.edu.au/research-insights-and-resources/research/governing-by-algorithm-child-protection-in-aotearoa-new-zealand/
  * bundesverfassungsgericht (2023) bundesverfassungsgericht. 2023.  Bundesverfassungsgericht - Press - Legislation in Hesse and Hamburg regarding automated data analysis for the prevention of criminal acts is unconstitutional.  https://www.bundesverfassungsgericht.de/SharedDocs/Pressemitteilungen/EN/2023/bvg23-018.html
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification. In _Conference on fairness, accountability and transparency_. PMLR, 77–91.
  * Buolamwini and Gebru (2023) Joy Buolamwini and Timnit Gebru. 2023.  Algorithmic Justice League Gender Shades 5th Anniversary Celebration. (2023).  https://www.youtube.com/watch?v=8JSxbZyivuE Virtual discussion (recorded).
  * Burrell and Mulligan (2020) Jenna Burrell and Deirdre Mulligan. 2020.  _The Berkeley Algorithmic Fairness and Opacity Group Refusal Conference_.  Retrieved January 16, 2024 from https://afog.berkeley.edu/programs/the-refusal-conference
  * Calo and Citron (2021) Ryan Calo and Danielle Keats Citron. 2021.  The Automated Administrative State: A Crisis of Legitimacy.  _Emory Law Journal_ (2021).  https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3553590
  * Capo and Bass (2017) Zeph Capo and Janet Bass. 2017.  Federal Suit Settlement: End of Value-Added Measures for Teacher Termination in Houston — American Federation of Teachers.  https://www.aft.org/press-release/federal-suit-settlement-end-value-added-measures-teacher-termination-houston
  * Casemajor et al. (2015) Nathalie Casemajor, Stéphane Couture, Mauricio Delfin, Matthew Goerzen, and Alessandro Delfanti. 2015.  Non-participation in digital media: toward a framework of mediated political action.  _Media, Culture & Society_ 37, 6 (2015), 850–866.  https://doi.org/10.1177/0163443715584098 arXiv:https://doi.org/10.1177/0163443715584098
  * Chang (2018) Cindy Chang. 2018.  _LAPD officials defend predictive policing as activists call for its end_.  Retrieved January 16, 2024 from https://www.latimes.com/local/lanow/la-me-lapd-data-policing-20180724-story.html
  * Chayka (2023) Kyle Chayka. 2023.  _Rethinking the Luddites in the Age of AI_.  Retrieved January 16, 2024 from https://www.newyorker.com/books/page-turner/rethinking-the-luddites-in-the-age-of-ai
  * Cheng et al. (2022) Hao-Fei Cheng, Logan Stapleton, Anna Kawakami, Venkatesh Sivaraman, Yanghuidi Cheng, Diana Qing, Adam Perer, Kenneth Holstein, Zhiwei Steven Wu, and Haiyi Zhu. 2022.  How Child Welfare Workers Reduce Racial Disparities in Algorithmic Decisions. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (, New Orleans, LA, USA,) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 162, 22 pages.  https://doi.org/10.1145/3491102.3501831
  * Cheng and Chouldechova (2022) Lingwei Cheng and Alexandra Chouldechova. 2022.  Heterogeneity in Algorithm-Assisted Decision-Making: A Case Study in Child Abuse Hotline Screening.  6, CSCW2, Article 376 (nov 2022), 33 pages.  https://doi.org/10.1145/3555101
  * Chowdhury (2021) Rumman Chowdhury. 2021.  _Sharing learnings about our image cropping algorithm_.  https://blog.twitter.com/engineering/en_us/topics/insights/2021/sharing-learnings-about-our-image-cropping-algorithm
  * Cifor et al. (2019) M. Cifor, P. Garcia, T.L. Cowan, J. Rault, T. Sutherland, A. Chan, J. Rode, A.L. Hoffmann, N. Salehi, and L. Nakamura. 2019.  _Feminist Data Manifest-No_.  Retrieved January 16, 2024 from https://www.manifestno.com/
  * Conniff (2011) Richard Conniff. 2011.  _What the Luddites Really Fought Against_.  Retrieved January 16, 2024 from https://www.smithsonianmag.com/history/what-the-luddites-really-fought-against-264412/
  * Dastin ([n. d.]) Jeffrey Dastin. [n. d.].  Insight - Amazon scraps secret AI recruiting tool that showed bias against women.  https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G/.  [Accessed 22-01-2024].
  * Dastin (2018) Jeffrey Dastin. 2018.  _Insight - Amazon scraps secret AI recruiting tool that showed bias against women_.  Retrieved January 16, 2024 from https://www.reuters.com/article/idUSKCN1MK0AG/
  * Dastin (2020) Jeffrey Dastin. 2020.  _Rite Aid deployed facial recognition systems in hundreds of U.S. stores_.  Retrieved January 16, 2024 from https://www.reuters.com/investigates/special-report/usa-riteaid-software/
  * Davis et al. (2021) Jenny L. Davis, Apryl Williams, and Michael W. Yang. 2021.  Algorithmic reparation.  _Big Data & Society_ 8, 2 (2021), 20539517211044808.  https://doi.org/10.1177/20539517211044808 arXiv:https://doi.org/10.1177/20539517211044808
  * De-Arteaga et al. (2020) Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020.  A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores.  _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (2020).  https://api.semanticscholar.org/CorpusID:211171909
  * Deng et al. (2022) Wesley Hanwen Deng, Manish Nagireddy, Michelle Seng Ah Lee, Jatinder Singh, Zhiwei Steven Wu, Kenneth Holstein, and Haiyi Zhu. 2022.  Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 473–484.  https://doi.org/10.1145/3531146.3533113
  * DeVrio et al. (2023) Alicia DeVrio, Motahhare Eslami, and Kenneth Holstein. 2023.  Building, Shifting, & Employing Power: A Framework for Understanding & Empowering Action From Below in Response to Algorithmic Harm.  _In submission_ (2023).
  * Dickey (2020) Megan Rose Dickey. 2020.  Twitter and Zoom’s algorithmic bias issues.  https://techcrunch.com/2020/09/21/twitter-and-zoom-algorithmic-bias-issues/
  * Dombrowski et al. (2016) Lynn Dombrowski, Ellie Harmon, and Sarah Fox. 2016.  Social Justice-Oriented Interaction Design: Outlining Key Design Strategies and Commitments. In _Proceedings of the 2016 ACM Conference on Designing Interactive Systems_ (Brisbane, QLD, Australia) _(DIS ’16)_. Association for Computing Machinery, New York, NY, USA, 656–671.  https://doi.org/10.1145/2901790.2901861
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards A Rigorous Science of Interpretable Machine Learning.  arXiv:1702.08608 [stat.ML]
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012.  Fairness through Awareness. In _Proceedings of the 3rd Innovations in Theoretical Computer Science Conference_ (Cambridge, Massachusetts) _(ITCS ’12)_. Association for Computing Machinery, New York, NY, USA, 214–226.  https://doi.org/10.1145/2090236.2090255
  * Ehsan et al. (2022) Upol Ehsan, Ranjit Singh, Jacob Metcalf, and Mark Riedl. 2022.  The Algorithmic Imprint. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 1305–1317.  https://doi.org/10.1145/3531146.3533186
  * Elizalde and Gartland (2019) Elizabeth Elizalde and Michael Gartland. 2019.  _Brooklyn tenants in rent-regulated apartments push state to nix landlord’s facial recognition software_.  Retrieved January 16, 2024 from https://www.nydailynews.com/2019/05/01/brooklyn-tenants-in-rent-regulated-apartments-push-state-to-nix-landlords-facial-recognition-software/
  * Eppink (2023) Ritchie Eppink. 2023.  _Testimony of Ritchie Eppink: AI in Government United States Senate Committee on Homeland Security & Government Affairs_.  Retrieved January 16, 2024 from https://www.hsgac.senate.gov/wp-content/uploads/Testimony-Eppink-2023-05-16-1.pdf
  * Erdbrink (2021) Thomas Erdbrink. 2021.  _Government in Netherlands Resigns After Benefit Scandal_.  Retrieved January 16, 2024 from https://www.nytimes.com/2021/01/15/world/europe/dutch-government-resignation-rutte-netherlands.html
  * Eslami et al. (2017) Motahhare Eslami, Kristen Vaccaro, Karrie Karahalios, and Kevin Hamilton. 2017.  “Be Careful; Things Can Be Worse than They Appear”: Understanding Biased Algorithms and Users’ Behavior Around Them in Rating Platforms.  _Proceedings of the International AAAI Conference on Web and Social Media_ 11, 1 (May 2017), 62–71.  https://doi.org/10.1609/icwsm.v11i1.14898
  * Eubanks (2018) Virginia Eubanks. 2018.  _A response to Allegheny County DHS_.  https://virginia-eubanks.com/2018/02/16/a-response-to-allegheny-county-dhs/
  * FairFare (2023) FairFare. 2023.  _FairFare: Unveiling Ridehail Fairness_.  Retrieved January 16, 2024 from https://getfairfare.org/
  * Forrest (2021) Katherine B. Forrest. 2021.  _When Machines Can Be Judge, Jury, and Executioner: Justice in the Age of Artificial Intelligence_.  World Scientific.
  * Friedman et al. (2006) Batya Friedman, Peter Kahn, Alan Borning, Ping Zhang, and Dennis Galletta. 2006.  _Value Sensitive Design and Information Systems_.  https://doi.org/10.1007/978-94-007-7844-3_4
  * Gagne (2019) Yasmin Gagne. 2019.  _How we fought our landlord’s secretive plan for facial recognition—and won_.  Retrieved January 16, 2024 from https://www.fastcompany.com/90431686/our-landlord-wants-to-install-facial-recognition-in-our-homes-but-were-fighting-back
  * Gavin (2022) William Gavin. 2022.  _As privacy concerns arise, organizations using facial recognition technology spend on lobbying_.  https://www.opensecrets.org/news/2022/03/as-privacy-concerns-arise-organizations-using-facial-recognition-technology-continue-to-employ-lobbyists/
  * Gehami and Doty (2023) Albert Gehami and Leila Doty. 2023.  When the Rubber Meets the Road: Experience Implementing AI Governance in a Public Agency with the City of San José. (2023).  https://www.youtube.com/watch?v=Bif3fwI_d20 ACM FAccT Tutorial.
  * Gerchick et al. (2023) Marissa Gerchick, Tobi Jegede, Tarak Shah, Ana Gutierrez, Sophie Beiers, Noam Shemtov, Kath Xu, Anjana Samant, and Aaron Horowitz. 2023.  The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT ’23)_. Association for Computing Machinery, New York, NY, USA, 1292–1310.  https://doi.org/10.1145/3593013.3594081
  * Gurley (2021) Lauren Kaori Gurley. 2021.  _Amazon’s AI Cameras Are Punishing Drivers for Mistakes They Didn’t Make_.  Retrieved January 16, 2024 from https://www.vice.com/en/article/88npjv/amazons-ai-cameras-are-punishing-drivers-for-mistakes-they-didnt-make
  * Hanchett (2022) Rebecca Hanchett. 2022.  _Rhode Island Looks At Banning Facial Recognition By Sports Betting Apps_.  Retrieved January 16, 2024 from https://www.gamingtoday.com/news/rhode-island-banning-facial-recognition-sports-betting-apps/
  * Hao (2020) Karen Hao. 2020.  _The two-year fight to stop Amazon from selling face recognition to the police_.  https://www.technologyreview.com/2020/06/12/1003482/amazon-stopped-selling-police-face-recognition-fight/
  * Harcourt (2006) Bernard E. Harcourt. 2006.  _Against Prediction: Profiling, Policing, and Punishing in an Actuarial Age_.  The Chicago University Press.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. 2016.  Equality of Opportunity in Supervised Learning. In _Advances in Neural Information Processing Systems_ , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2016/file/9d2682367c3935defcb1f9e247a97c0d-Paper.pdf
  * Haskins (2019) Caroline Haskins. 2019.  _Dozens of Cities Have Secretly Experimented With Predictive Policing Software_.  https://www.vice.com/en/article/d3m7jq/dozens-of-cities-have-secretly-experimented-with-predictive-policing-software
  * Haskins (2021) Caroline Haskins. 2021.  _The NYPD Has Misled The Public About Its Use Of Facial Recognition Tool Clearview AI_.  https://www.buzzfeednews.com/article/carolinehaskins1/nypd-has-misled-public-about-clearview-ai-use
  * Hill (2020a) Kashmir Hill. 2020a.  _The Secretive Company That Might End Privacy as We Know It_.  Retrieved January 16, 2024 from https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html
  * Hill (2020b) Kashmir Hill. 2020b.  Wrongfully Accused by an Algorithm.  _The New York Times_ (Jun 2020).  https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html
  * Ho and Burke (2022) Sally Ho and Garance Burke. 2022.  _An algorithm that screens for child neglect raises concerns_.  Retrieved January 16, 2024 from https://apnews.com/article/child-welfare-algorithm-investigation-9497ee937e0053ad4144a86c68241ef1
  * Ho and Burke (2023) Sally Ho and Garnace Burke. 2023.  _Child welfare algorithm faces Justice Department scrutiny_.  https://apnews.com/article/justice-scrutinizes-pittsburgh-child-welfare-ai-tool-4f61f45bfc3245fd2556e886c2da988b,lastaccessed={January16,2024},
  * Homewood (2019) Sarah Homewood. 2019.  Inaction as a Design Decision: Reflections on Not Designing Self-Tracking Tools for Menopause. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI EA ’19)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3290607.3310430
  * Honig (2021) Bonnie Honig. 2021.  _A Feminist Theory of Refusal_.  Harvard University Press.
  * Hoover (2023) Amanda Hoover. 2023.  _An Eating Disorder Chatbot Is Suspended for Giving Harmful Advice_.  Retrieved January 16, 2024 from https://www.wired.com/story/tessa-chatbot-suspended/
  * Houston et al. (2016) Lara Houston, Steven J. Jackson, Daniela K. Rosner, Syed Ishtiaque Ahmed, Meg Young, and Laewoo Kang. 2016.  Values in Repair. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 1403–1414.  https://doi.org/10.1145/2858036.2858470
  * Howell et al. (2021) Noura Howell, Audrey Desjardins, and Sarah Fox. 2021.  Cracks in the Success Narrative: Rethinking Failure in Design Research through a Retrospective Trioethnography.  _ACM Trans. Comput.-Hum. Interact._ 28, 6, Article 42 (nov 2021), 31 pages.  https://doi.org/10.1145/3462447
  * https://www.theguardian.com/profile/justinmccurry ([n. d.]) https://www.theguardian.com/profile/justinmccurry. [n. d.].  South Korean AI chatbot pulled from Facebook after hate speech towards minorities — theguardian.com.  https://www.theguardian.com/world/2021/jan/14/time-to-properly-socialise-hate-speech-ai-chatbot-pulled-from-facebook.  [Accessed 22-01-2024].
  * Hub (2022a) Benefits Tech Advocacy Hub. 2022a.  Arkansas Medicaid Home and Community Based Services Hours Cuts.  Retrieved January 16, 2024 from https://www.btah.org/case-study/arkansas-medicaid-home-and-community-based-services-hours-cuts.html
  * Hub (2022b) Benefits Tech Advocacy Hub. 2022b.  Idaho Medicaid Home and Community Based Services Care Cuts.  Retrieved January 16, 2024 from https://www.btah.org/case-study/idaho-medicaid-home-and-community-based-services-care-cuts.html
  * Hub (2022c) Benefits Tech Advocacy Hub. 2022c.  Missouri Medicaid Home and Community Based Services Eligibility Issues.  Retrieved January 16, 2024 from https://www.btah.org/case-study/missouri-medicaid-home-and-community-based-services-eligibility-issues.html
  * Hub (2022d) Benefits Tech Advocacy Hub. 2022d.  Understanding the Lifecycle of Benefits Technology.  Retrieved January 16, 2024 from https://www.btah.org/lifecycle.html
  * in Pittsburgh (2020a) The Coalition Against Predictive Policing in Pittsburgh. 2020a.  _Predictive Policing in Pittsburgh: A Primer_.  Retrieved January 16, 2024 from https://capp-pgh.com/files/Primer_v1.pdf
  * in Pittsburgh (2020b) The Coalition Against Predictive Policing in Pittsburgh. 2020b.  _Responding to the “Completion” of Predictive Policing in Pittsburgh_.  Retrieved January 16, 2024 from https://capp-pgh.com/files/Metro21%20Counter-Statement.pdf
  * Jacobs and Wallach (2021) Abigail Z. Jacobs and Hanna Wallach. 2021.  Measurement and Fairness. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. ACM.  https://doi.org/10.1145/3442188.3445901
  * Kalluri et al. (2023) Pratyusha Ria Kalluri, William Agnew, Myra Cheng, Kentrell Owens, Luca Soldaini, and Abeba Birhane. 2023.  The Surveillance AI Pipeline.  arXiv:2309.15084 [cs.CV]
  * Khan and White (2021) Hamid Khan and Pete White. 2021.  _Police Surveillance Can’t Be Reformed. It Must Be Abolished_.  Retrieved January 16, 2024 from https://www.vice.com/en/article/xgzj7n/police-surveillance-cant-be-reformed-it-must-be-abolished
  * Klar (2023) Rebecca Klar. 2023.  _Exclusive: Meta faces pressure to support independent audit of risk oversight committee_.  Retrieved January 16, 2024 from https://news.yahoo.com/exclusive-meta-faces-pressure-support-164507051.html
  * Kraft (2016) Amy Kraft. 2016.  _Microsoft shuts down AI chatbot after it turned into a Nazi_.  Retrieved January 16, 2024 from https://www.cbsnews.com/news/microsoft-shuts-down-ai-chatbot-after-it-turned-into-racist-nazi/
  * Kwon and Yun (2021) Junhyup Kwon and Hyeong Yun. 2021.  _AI Chatbot Shut Down After Learning To Talk Like a Racist Asshole_.  Retrieved January 16, 2024 from https://www.vice.com/en/article/akd4g5/ai-chatbot-shut-down-after-learning-to-talk-like-a-racist-asshole
  * Lecher (2018) Colin Lecher. 2018.  _What happens when an algorithm cuts your health care_.  https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy
  * Lee et al. (2019) Nicol Turner Lee, Paul Resnick, and Genie Barton. 2019.  Algorithmic bias detection and mitigation: Best practices and policies to reduce consumer harms.  https://www.brookings.edu/articles/algorithmic-bias-detection-and-mitigation-best-practices-and-policies-to-reduce-consumer-harms/
  * Lee (2016) Peter Lee. 2016.  _Learning from Tay’s introduction_.  https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/
  * Liang (2021) Calvin Liang. 2021.  _Reflexivity, positionality, and disclosure in HCI_.  Retrieved January 16, 2024 from https://medium.com/@caliang/reflexivity-positionality-and-disclosure-in-hci-3d95007e9916
  * Liao and Vaughan (2023) Q. Vera Liao and Jennifer Wortman Vaughan. 2023.  AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap.  arXiv:2306.01941 [cs.HC]
  * Liban (2018) Kevin De Liban. 2018.  _Comments re: Notice of Rule-Making for ARChoices Program_.  Retrieved January 21, 2024 from https://www.arkleg.state.ar.us/Home/FTPDocument?path=%2FAssembly%2FMeeting+Attachments%2F430%2F663%2FHandout+1+-Legal+Aid-Kevin+De+Liban.pdf Email providing public comment on proposed revisions to ARChoices program..
  * Lima et al. (2023) Gabriel Lima, Nina Grgic-Hlaca, Jin Keun Jeong, and Meeyoung Cha. 2023.  Who Should Pay When Machines Cause Harm? Laypeople’s Expectations of Legal Damages for Machine-Caused Harm. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT ’23)_. Association for Computing Machinery, New York, NY, USA, 236–246.  https://doi.org/10.1145/3593013.3593992
  * Loewenstein (2023) Antony Loewenstein. 2023.  _The Palestine Laboratory: How Israel Exports the Technology of Occupation Around the World_.  Verso.  https://www.versobooks.com/products/2684-the-palestine-laboratory
  * Lomibao (2021) Lynn Lomibao. 2021.  Stop LAPD Spying Sues LAPD to Uncover Communications with UCLA Professor Who Founded PredPol, Inc.  https://stoplapdspying.org/stop-lapd-spying-sues-lapd-for-communications-with-widely-condemned-ucla-professor-who-founded-predpol-inc/
  * Ly (2023) My Ly. 2023.  _Arkansas DHS agrees to pay $460,000 to settle case over in-home care cuts_.  Retrieved January 16, 2024 from https://www.arkansasonline.com/news/2023/aug/09/arkansas-dhs-agrees-to-pay-460000-to-settle-case/
  * Mari (2022) Angelica Mari. 2022.  São Paulo subway ordered to suspend use of facial recognition.  https://www.zdnet.com/article/sao-paulo-subway-ordered-to-suspend-use-of-facial-recognition/
  * Marx and Irani (2021) Jesse Marx and Lilly Irani. 2021.  _Redacted_.  Taller California.  https://www.printedmatter.org/catalog/58378/
  * McGranahan (2016) Carole McGranahan. 2016.  Theorizing Refusal: An Introduction.  _Cultural Anthropology_ 31 (08 2016), 319–325.  https://doi.org/10.14506/ca31.3.01
  * McGregor (2020) Sean McGregor. 2020.  Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database.  arXiv:2011.08512 [cs.CY]
  * Meadway (2020) James Meadway. 2020.  “Fuck the Algorithm”: How A-Level Students Have Shown the Future of Protest.  https://novaramedia.com/2020/08/17/fuck-the-algorithm-how-a-level-students-have-shown-future-of-protest/
  * Mehrotra and Cameron (2023) Dhruv Mehrotra and Dell Cameron. 2023.  _The Maker of ShotSpotter Is Buying the World’s Most Infamous Predictive Policing Tech_.  Retrieved January 16, 2024 from https://www.wired.com/story/soundthinking-geolitica-acquisition-predictive-policing/
  * Merchant (2023) Brian Merchant. 2023.  _Blood in the Machine: The Origins of the Rebellion Against Big Tech_.  Little, Brown and Company.
  * Metcalf (2023) Jacob Metcalf. 2023.  _What federal agencies can learn from NYC’s AI Hiring Law_.  Retrieved January 16, 2024 from https://thehill.com/opinion/technology/4360523-what-federal-agencies-can-learn-from-new-york-citys-ai-hiring-law/
  * Metcalf et al. (2023) Jacob Metcalf, Ranjit Singh, Emanuel Moss, Emnet Tafesse, and Elizabeth Anne Watkins. 2023.  Taking Algorithms to Courts: A Relational Approach to Algorithmic Accountability. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT ’23)_. Association for Computing Machinery, New York, NY, USA, 1450–1462.  https://doi.org/10.1145/3593013.3594092
  * Microsoft (2022) Microsoft. 2022.  _Microsoft Responsible AI Standard, v2 (General Requirements)_.  Retrieved July 21, 2023 from https://blogs.microsoft.com/wp-content/uploads/prod/sites/5/2022/06/Microsoft-Responsible-AI-Standard-v2-General-Requirements-3.pdf
  * Milner (2020) Yeshimabeit Milner. 2020.  _Abolish Big Data_.  https://medium.com/@YESHICAN/abolish-big-data-ad0871579a41
  * Mok (2023) Aaron Mok. 2023.  _AI is expensive. A search on Google’s chatbot Bard costs the company 10 times more than a regular one, which could amount to several billion dollars._ Retrieved January 16, 2024 from https://www.businessinsider.com/ai-expensive-google-chatbot-bard-may-cost-company-billions-dollars-2023-2
  * Moss et al. (2021) Emanuel Moss, Elizabeth Anne Watkins, Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf. 2021.  _Assembling accountability: algorithmic impact assessment for the public interest_.  Retrieved July 21, 2023 from https://datasociety.net/wp-content/uploads/2021/06/Assembling-Accountability.pdf
  * Mozilla (2023) Mozilla. 2023.  _Auditing AI: Announcing the 2023 Mozilla Technology Fund Cohort_.  Retrieved January 16, 2024 from https://foundation.mozilla.org/en/blog/auditing-ai-announcing-the-2023-mozilla-technology-fund-cohort/
  * Noble (2018) Safiya Umoja Noble. 2018.  _Algorithms of Oppression: How Search Engines Reinforce Racism_.  NYU Press.  http://www.jstor.org/stable/j.ctt1pwt9w5
  * Nyeko (2023) Oryem Nyeko. 2023.  Uganda: Rights Concerns Over License Plate Tracking — Human Rights Watch.  https://www.hrw.org/news/2023/11/14/uganda-rights-concerns-over-license-plate-tracking
  * Obermeyer et al. (2019) Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019.  Dissecting racial bias in an algorithm used to manage the health of populations.  _Science_ 366, 6464 (2019), 447–453.
  * of Canada (2023) The Government of Canada. 2023.  _Algorithmic Impact Assessment tool_.  Retrieved July 21, 2023 from https://www.canada.ca/en/government/system/digital-government/digital-government-innovations/responsible-use-ai/algorithmic-impact-assessment.html
  * of Science and Policy (2022) The White House Office of Science and Technology Policy. 2022.  _Blueprint for an AI Bill of Rights: A Vision for Protecting Our Civil Rights in the Algorithmic Age_.  Retrieved October 2, 2023 from https://www.whitehouse.gov/ostp/news-updates/2022/10/04/blueprint-for-an-ai-bill-of-rightsa-vision-for-protecting-our-civil-rights-in-the-algorithmic-age/
  * OHCHR (2022) UN OHCHR. 2022.  OHCHR Assessment of Human Rights Concerns in the Xinjiang Uyghur Autonomous Region, People’s Republic of China.
  * Orphanou et al. (2022) Kalia Orphanou, Jahna Otterbacher, Styliani Kleanthous, Khuyagbaatar Batsuren, Fausto Giunchiglia, Veronika Bogina, Avital Shulner Tal, Alan Hartman, and Tsvi Kuflik. 2022.  Mitigating Bias in Algorithmic Systems—A Fish-eye View.  _ACM Comput. Surv._ 55, 5, Article 87 (dec 2022), 37 pages.  https://doi.org/10.1145/3527152
  * Pennington (2023) Phil Pennington. 2023.  Facial recognition: Officials yet to meet obligation to seek views of Māori - documents.  https://www.rnz.co.nz/news/national/501761/facial-recognition-officials-yet-to-meet-obligation-to-seek-views-of-maori-documents
  * Perry (2023) Nick Perry. 2023.  New Zealand debates whether ethnicity should be a factor for surgery waitlists.  https://apnews.com/article/new-zealand-surgery-ethnicity-algorithm-maori-1b44026f2661772a7eb3bd4444619446
  * Pessach and Shmueli (2022) Dana Pessach and Erez Shmueli. 2022.  A Review on Fairness in Machine Learning.  _ACM Comput. Surv._ 55, 3, Article 51 (feb 2022), 44 pages.  https://doi.org/10.1145/3494672
  * Pierson and Pierson (2023) Brendan Pierson and Brendan Pierson. 2023.  Lawsuit claims UnitedHealth AI wrongfully denies elderly extended care.  _Reuters_ (Nov 2023).  https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai-wrongfully-denies-elderly-extended-care-2023-11-14/
  * Porter (2020) Jon Porter. 2020.  _UK ditches exam results generated by biased algorithm after student protests_.  Retrieved January 16, 2024 from https://www.theverge.com/2020/8/17/21372045/uk-a-level-results-algorithm-biased-coronavirus-covid-19-pandemic-university-applications
  * Puron-Cid and Gil-Garcia (2022) Gabriel Puron-Cid and J. Ramon Gil-Garcia. 2022.  Are Smart Cities Too Expensive in the Long Term? Analyzing the Effects of ICT Infrastructure on Municipal Financial Sustainability.  _Sustainability_ 14, 10 (2022).  https://doi.org/10.3390/su14106055
  * Raji and Buolamwini (2022) Inioluwa Deborah Raji and Joy Buolamwini. 2022.  Actionable Auditing Revisited: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products.  _Commun. ACM_ 66, 1 (dec 2022), 101–108.  https://doi.org/10.1145/3571151
  * Raji et al. (2022) Inioluwa Deborah Raji, I. Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. 2022.  The Fallacy of AI Functionality. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 959–972.  https://doi.org/10.1145/3531146.3533158
  * Raji et al. (2024) Inioluwa Deborah Raji, Ryan Steed, Victor Ojewale, Briana Vecchione, , and Abeba Birhane. 2024.  Towards Accountability Infrastructure: Gaps and opportunities in AI audit tooling. (2024).  Preprint (Under Review).
  * Reisman et al. (2018) Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. 2018.  _Algorithmic Impact Assessments: A Practical Framework for Public Agency_.  Retrieved July 21, 2023 from https://www.nist.gov/system/files/documents/2021/10/04/aiareport2018.pdf
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier.  arXiv:1602.04938 [cs.LG]
  * Roemmich et al. (2023) Kat Roemmich, Tillie Rosenberg, Serena Fan, and Nazanin Andalibi. 2023.  Values in Emotion Artificial Intelligence Hiring Services: Technosolutions to Organizational Problems.  _Proc. ACM Hum.-Comput. Interact._ 7, CSCW1, Article 109 (apr 2023), 28 pages.  https://doi.org/10.1145/3579543
  * Ross (2021) Casey Ross. 2021.  _Epic’s AI algorithms, shielded from scrutiny by a corporate firewall, are delivering inaccurate information on seriously ill patients_.  Retrieved January 21, 2024 from https://www.statnews.com/2021/07/26/epic-hospital-algorithms-sepsis-investigation/
  * Ryan-Mosley (2023a) Tate Ryan-Mosley. 2023a.  _An algorithm intended to reduce poverty might disqualify people in need_.  Retrieved January 16, 2024 from https://www.technologyreview.com/2023/06/13/1074551/an-algorithm-intended-to-reduce-poverty-in-jordan-disqualifies-people-in-need/
  * Ryan-Mosley (2023b) Tate Ryan-Mosley. 2023b.  _How face recognition rules in the US got stuck in political gridlock_.  https://www.technologyreview.com/2023/07/24/1076668/how-face-recognition-rules-in-the-us-got-stuck-in-political-gridlock/
  * Schwennesen (2019) Nete Schwennesen. 2019.  Algorithmic assemblages of care: imaginaries, epistemologies and repair work.  _Sociology of Health & Illness_ 41 (10 2019), 176–192.  https://doi.org/10.1111/1467-9566.12900
  * Scott (2023) Joey Scott. 2023.  _LAPD Is Using Israeli Surveillance Software That Can Track Your Phone and Social Media_.  Retrieved January 16, 2024 from https://knock-la.com/lapd-is-using-israeli-surveillance-software-that-can-track-your-phone-and-social-media/#:~:text=During%20a%202014%20trip%20to,would%20be%20using%20all%20three
  * Seaver (2019) Nick Seaver. 2019.  _Knowing Algorithms_.  Princeton University Press, Princeton, 412–422.  https://doi.org/doi:10.1515/9780691190600-028
  * Selbst and Barocas (2018) Andrew D. Selbst and Solon Barocas. 2018.  The Intuitive Appeal of Explainable Machines.  _Fordham Law Review_ 87 (2018), 1085.  https://api.semanticscholar.org/CorpusID:59548063
  * Selbst et al. (2024) Andrew D. Selbst, Suresh Venkatasubramanian, and I. Elizabeth Kumar. 2024.  Deconstructing Design Decisions: Why Courts Must Interrogate Machine Learning and Other Technologies.  _Ohio State Law Journal_ 85 (2024).  https://ssrn.com/abstract=4564304
  * Shelby et al. (2023) Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N’Mah Yilla-Akbari, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. 2023.  Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ (, Montréal, QC, Canada,) _(AIES ’23)_. Association for Computing Machinery, New York, NY, USA, 723–741.  https://doi.org/10.1145/3600211.3604673
  * Shen et al. (2021) Hong Shen, Alicia DeVos, Motahhare Eslami, and Kenneth Holstein. 2021.  Everyday Algorithm Auditing: Understanding the Power of Everyday Users in Surfacing Harmful Algorithmic Behaviors.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW2, Article 433 (oct 2021), 29 pages.  https://doi.org/10.1145/3479577
  * Simpson (2007) Audra Simpson. 2007.  On Ethnographic Refusal: Indigeneity, ‘Voice’ and Colonial Citizenship.  _Junctures_ (2007).
  * Sisitzky and Schaefer (2021) Michael Sisitzky and Ben Schaefer. 2021.  _The NYPD Published its Arsenal of Surveillance Tech. Here’s What We Learned_.  Retrieved January 16, 2024 from https://www.nyclu.org/en/news/nypd-published-its-arsenal-surveillance-tech-heres-what-we-learned
  * Speicher et al. (2018) Till Speicher, Hoda Heidari, Nina Grgic-Hlaca, Krishna P. Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. 2018.  A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (London, United Kingdom) _(KDD ’18)_. Association for Computing Machinery, New York, NY, USA, 2239–2248.  https://doi.org/10.1145/3219819.3220046
  * Stapleton et al. (2022) Logan Stapleton, Min Hun Lee, Diana Qing, Marya Wright, Alexandra Chouldechova, Ken Holstein, Zhiwei Steven Wu, and Haiyi Zhu. 2022.  Imagining New Futures beyond Predictive Systems in Child Welfare: A Qualitative Study with Impacted Stakeholders. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 1162–1177.  https://doi.org/10.1145/3531146.3533177
  * Subhash et al. (2022) Varshini Subhash, Zixi Chen, Marton Havasi, Weiwei Pan, and Finale Doshi-Velez. 2022.  What Makes a Good Explanation?: A Harmonized View of Properties of Explanations. In _Progress and Challenges in Building Trustworthy Embodied AI_.  https://openreview.net/forum?id=YDyLZWwpBK2
  * Suresh and Guttag (2021) Harini Suresh and John Guttag. 2021.  A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. In _Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization_ (–, NY, USA) _(EAAMO ’21)_. Association for Computing Machinery, New York, NY, USA, Article 17, 9 pages.  https://doi.org/10.1145/3465416.3483305
  * System (2021) The Automated Regional Justice Information System. 2021.  _TACIDS: Tactical Identification System Using Facial Recognition_.  Retrieved January 16, 2024 from https://voiceofsandiego.org/wp-content/uploads/2021/04/TACIDS-Final-Report-FINAL.pdf
  * TallBear (2013) Kim TallBear. 2013.  _Native American DNA: Tribal Belonging and the False Promise of Genetic Science_.  U of Minnesota Press.
  * Toh and Watch (2023) A. Toh and Human Rights Watch. 2023.  _Automated Neglect: How the World Bank’s Push to Allocate Cash Assistance Using Algorithms Threatens Rights_.  Human Rights Watch.  https://books.google.com/books?id=rNMG0AEACAAJ
  * Vaithianathan et al. (2019) Rhema Vaithianathan, Emily Kulick, Emily Putnam-Hornstein, and Diana Benavides Prado. 2019.  _Allegheny Family Screening Tool: Methodology, Version 2_.  Retrieved January 21, 2024 from https://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/Methodology-V2-from-16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-7.pdf
  * Valderrama (2021) Matías Valderrama. 2021.  Sistema Alerta Niñez y la predicción del riesgo de vulneración de derechos de la infancia.  _Derechos Digitales_ (2021).
  * van Veen (2020) C van Veen. 2020.  Landmark judgment from The Netherlands on digital welfare states and human rights.  _Open Global Rights_ (2020).
  * Varon and Peña (2022) Joana Varon and Paz Peña. 2022.  Not My A.I.: Towards Critical Feminist Frameworks to Resist Oppressive A.I. Systems.  _Carr Center Discussion Paper Series_ (2022).  https://carrcenter.hks.harvard.edu/publications/notmyai
  * Vincent ([n. d.]) James Vincent. [n. d.].  Twitter taught Microsoft’s AI chatbot to be a racist asshole in less than a day — theverge.com.  https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist.  [Accessed 22-01-2024].
  * Vogell et al. (2022) Heather Vogell, Haru Coryne, and Ryan Little. 2022.  _Rent Going Up? One Company’s Algorithm Could Be Why._ Retrieved January 16, 2024 from https://www.propublica.org/article/yieldstar-rent-increase-realpage-rent
  * Wang et al. (2023) Angelina Wang, Sayash Kapoor, Solon Barocas, and Arvind Narayanan. 2023.  Against Predictive Optimization: On the Legitimacy of Decision-Making Algorithms that Optimize Predictive Accuracy. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT ’23)_. Association for Computing Machinery, New York, NY, USA, 626.  https://doi.org/10.1145/3593013.3594030
  * Wang and Rudin (2015) Fulton Wang and Cynthia Rudin. 2015.  Falling Rule Lists. In _Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics_ _(Proceedings of Machine Learning Research, Vol. 38)_ , Guy Lebanon and S. V. N. Vishwanathan (Eds.). PMLR, San Diego, California, USA, 1013–1022.  https://proceedings.mlr.press/v38/wang15a.html
  * Wang et al. (2022) Nina Wang, Allison McDonald, Daniel Bateyko, and Emily Tucker. 2022.  American Dragnet: Data-Driven Deportation in the 21st Century. (2022).  Retrieved January 16, 2024 from https://americandragnet.org/
  * Warren (2023) Elizabeth Warren. 2023.  _Warren, Lawmakers Urge Justice Department to Review YieldStar, Warn of De-Facto Price Setting and Collusion After Senate Investigation_.  Retrieved January 21, 2024 from https://www.warren.senate.gov/oversight/letters/warren-lawmakers-urge-justice-department-to-review-yieldstar-warn-of-de-facto-price-setting-and-collusion-after-senate-investigation Press Release.
  * Watch (2023) Human Rights Watch. 2023.  _Automated Neglect: How The World Bank’s Push to Allocate Cash Assistance Using Algorithms Threatens Rights_.  Retrieved January 16, 2024 from https://www.hrw.org/report/2023/06/13/automated-neglect/how-world-banks-push-allocate-cash-assistance-using-algorithms
  * Weerts et al. (2023) Hilde Weerts, Miroslav Dudík, Richard Edgar, Adrin Jalali, Roman Lutz, and Michael Madaio. 2023.  Fairlearn: Assessing and Improving Fairness of AI Systems.  arXiv:2303.16626 [cs.LG]
  * Weil and Edwards (2023) Emma Weil and Elizabeth Edwards. 2023.  Using Technical Skills to Fight Actual Public Benefits Cuts and Austerity Policies, with the Benefits Tech Advocacy Hub. (2023).  https://www.youtube.com/watch?v=ZELeRPx74PE ACM FAccT Tutorial.
  * Wells (2023) Kate Wells. 2023.  _An eating disorders chatbot offered dieting advice, raising fears about AI in health_.  Retrieved January 16, 2024 from https://www.npr.org/sections/health-shots/2023/06/08/1180838096/an-eating-disorders-chatbot-offered-dieting-advice-raising-fears-about-ai-in-hea
  * Whitney et al. (2021) Cedric Deslandes Whitney, Teresa Naval, Elizabeth Quepons, Simrandeep Singh, Steven R Rick, and Lilly Irani. 2021.  HCI Tactics for Politics from Below: Meeting the Challenges of Smart Cities. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (, Yokohama, Japan,) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 297, 15 pages.  https://doi.org/10.1145/3411764.3445314
  * Whittaker (2019) Zack Whittaker. 2019.  _Amazon shareholders reject facial recognition sale ban to governments_.  Retrieved January 16, 2024 from https://techcrunch.com/2019/05/22/amazon-reject-facial-recognition-proposals/
  * Wilkinson and Lukens (2023) Christopher Wilkinson and Dorthy Lukens. 2023.  _The Growing Regulation of AI-Based Employment Decision Tools_.  Retrieved January 16, 2024 from https://www.perkinscoie.com/en/news-insights/the-growing-regulation-of-ai-based-employment-decision-tools.html
  * Williams (2015) Timothy Williams. 2015.  _Facial Recognition Software Moves From Overseas Wars to Local Police_.  Retrieved January 16, 2024 from https://www.nytimes.com/2015/08/13/us/facial-recognition-software-moves-from-overseas-wars-to-local-police.html?_r=1
  * Wilson and Murgia (2019) Tom Wilson and Madhumita Murgia. 2019.  _Uganda confirms use of Huawei facial recognition cameras_.  Retrieved January 16, 2024 from https://www.ft.com/content/e20580de-c35f-11e9-a8e9-296ca66511c9
  * Wong et al. (2021) Andrew Wong, Erkin Otles, John P. Donnelly, Andrew Krumm, Jeffrey McCullough, Olivia DeTroyer-Cooley, Justin Pestrue, Marie Phillips, Judy Konye, Carleen Penoza, Muhammad Ghous, and Karandeep Singh. 2021.  External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients.  _JAMA Internal Medicine_ 181, 8 (08 2021), 1065–1070.  https://doi.org/10.1001/jamainternmed.2021.2626 arXiv:https://jamanetwork.com/journals/jamainternalmedicine/articlepdf/2781307/jamainternal_wong_2021_oi_210027_1627674961.11707.pdf
  * Wu (2019) Sarah Wu. 2019.  _Somerville City Council passes facial recognition ban_.  Retrieved January 16, 2024 from https://www.bostonglobe.com/metro/2019/06/27/somerville-city-council-passes-facial-recognition-ban/
  * Xiang (2023) Chloe Xiang. 2023.  _Eating Disorder Helpline Disables Chatbot for ’Harmful’ Responses After Firing Human Staff_.  Retrieved January 16, 2024 from https://www.vice.com/en/article/qjvk97/eating-disorder-helpline-disables-chatbot-for-harmful-responses-after-firing-human-staff
  * Yee et al. (2021) Kyra Yee, Uthaipon Tantipongpipat, and Shubhanshu Mishra. 2021.  Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW2, Article 450 (oct 2021), 24 pages.  https://doi.org/10.1145/3479594
  * Yildirim et al. (2023) Nur Yildirim, Changhoon Oh, Deniz Sayar, Kayla Brand, Supritha Challa, Violet Turri, Nina Crosby Walton, Anna Elise Wong, Jodi Forlizzi, James McCann, and John Zimmerman. 2023.  Creating Design Resources to Scaffold the Ideation of AI Concepts. In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ (, Pittsburgh, PA, USA,) _(DIS ’23)_. Association for Computing Machinery, New York, NY, USA, 2326–2346.  https://doi.org/10.1145/3563657.3596058
  * Zong and Matias (2023) Jonathan Zong and J. Nathan Matias. 2023.  Data Refusal From Below: A Framework for Understanding, Evaluating, and Envisioning Refusal as Design.  _ACM J. Responsib. Comput._ (oct 2023).  https://doi.org/10.1145/3630107 Just Accepted.

##  Appendix A Case List

|  Case Name |  Case Description
---|---|---
|  Online platforms |
1 |  Twitter Cropping Algorithm (Yee et al., 2021) |  Users hypothesized that an algorithm that cropped images to show feed previews compatible with device aspect ratios noticed that it was more likely to crop out Black people (or other minority groups, i.e., women) in images with multiple people.
2 |  Zoom Virtual Background (Dickey, 2020) |  Zoom’s virtual background feature is more likely to be dysfunctional (i.e., hide the body of or include parts of the background) for users with darker skin.
|  Chatbot |
3 |  Microsoft Tay Bot (Vincent, [n. d.]) |  Microsoft’s Tay chatbot began tweeting offensive content after interacting with users who wanted it to do so.
4 |  U.S. National Eating Disorder Assocation (NEDA)’s “Tessa” Wellness Bot (Wells, 2023) |  Shortly NEDA fired all of their hotline staff (following their decision to unionize), eating disorder activists posted on social media about how Tessa (NEDA’s chatbot) gave harmful and triggering advice.
5 |  Facebook Galactica (gal, [n. d.]) |  Within three days of releasing a public demo of their Galactica language model, Meta took down the public demo and re-branded Galactica as ”available for researchers” only.
6 |  South Korean Startup (Scatter Lab)’s ”Lee-Luda” Chatbot (https://www.theguardian.com/profile/justinmccurry, [n. d.]) |  Seoul-based startup’s AI-chatbot ”Lee-Luda” was removed from Facebook messenger 20 days upon launch after users complained it was using offensive language, discriminatory and generated hate speech towards LGBTQ folks and people with disabilities.
|  Employment |
7 |  Amazon AI Recuriting Tool (Dastin, [n. d.]) |  Amazon disbanded an internal team whose goal was to train ML to screen resumes.
|  Child Welfare |
8 |  Oregon DHS’s Safety at Screening Tool (ap2, 2022) |  Oregon DHS decided to stop using an ”AI”-based screening tool (which is modeled after the AFST), in the wake of an April AP news article reporting that these tools have the ability to ”heighten racial disparities in the child welfare system”.
9 |  Pittsburgh Allegheny County Family Screening Tool (AFST) (Gerchick et al., 2023) |  External researchers, journalists, and advocacy groups have raised concerns about the AFST, including that it is more likely to investigate (and ultimately deem negligent) (1) poor families, (2) Black families, and (3) people with disabilities.

|  Case Name |  Case Description
---|---|---
|  Child Welfare |
10 |  Chile Local Childhood Office’s Sistema Alerta Niñez (“Childhood Alert System”) (Valderrama, 2021) |  Chile’s Ministry of Social Development and Family launched the program to strengthen childhood protection with an algorithm that provided risk scores used to determine necessary intervention, but rejected an independent external audit’s results revealing the system wrongly analyzed variables associated with direct measures of poverty rather than safety, targeting and perpetuating over-intervention in poor neighborhoods.
11 |  New Zealand’s ”Vulnerable Children PRM” (Predictive Risk Modeling) (Bukowitz and O’Loughlin, 2022) |  New Zealand’s Social Development Minister intervenes and halts the deployment of the algorithm after seeing major concerns from additional research during the request for its ethics approval for further study.
|  Family & Social Services |
12 |  Teen Pregnancy Risk
Prediction in Argentina (Aplicada, 2018) |  The Applied Artificial Intelligence Laboratory (LIAA) in the University of Buenos Aires published concerns in a detailed technical analysis and review of an algorithm developed through a partnership between Microsoft and province of Salta’s Ministry of Early Childhood to predict teenage pregnancy.
13 |  The Spanish Ministry of the Interior’s “VioGén” Gender Violence Recidivism Risk
Assessment (eti, 2022) |  The Eticas Foundation partnered with victims to publish an external audit after the Spanish Ministry declined requests of auditing the algorithm that underestimated level of risk in domestic and intimate partner violence cases leading to adverse outcomes.
|  Benefits |
14 |  Arkansas Department of Human Services’ Homecare Medicaid Program ”RUGS” Assessment System (Hub, 2022a) |  Arkansas switched to using a new algorithm called ”RUGS” to assess the hours of in-home care that residents are eligible for. The algorithm resulted in severe cuts to the number of hours allocated to most residents.
15 |  Missouri ”NF LOC” Home Care (Hub, 2022c) |  The State of Missouri released (in advance) their plan to switch to using an algorithm called ”NF LOC” to assess eligibility for home-care services. They released the algorithm for public comment, and external researchers found that it would result in severe cuts for a large group of residents (and also had other logical errors).
16 |  Idaho Department of Health and Welfare’s Care Recommendation Algorithm (Hub, 2022b) |  The Idaho Department of Health and Welfare used an algorithm to assess individual MedicAid budgets available to people with disabilities in their home-based services waiver program. A class-action lawsuit, KW vs. Armstrong, resulted in the algorithm being made public, and the algorithm was deemed unconstitutional and ”arbitrary”. The case was settled so that people whose budgets were cut would continue to get their original budgets.

|  Case Name |  Case Description
---|---|---
|  Benefits |
17 |  World Bank’s Automated Cash Transfer Program ”Takaful” in Jordan (Toh and Watch, 2023) |  Human Rights Watch published an investigation report that revealed how the World Bank’s algorithm for distributing and ranking financial assistance through a cash transfer program spearheaded by Jordan government’s social protection agency was wrongfully disqualifying those in need. The algorithm used 57 factors that they declined to publicly disclose.
18 |  Netherlands’ System Risk
Indication (SyRI) (van Veen, 2020) |  After local civil rights organizations filed a case against the state to stop algorithmic use for detecting fraud of welfare benefits targeting poor and marginalized neighborhoods, the Dutch court ruled that the algorithm violated the European Convention on Human Rights (ECHR) protections and halted it operation.
|  Healthcare |
19 |  EPIC’ Sepsis Prediction Tool (Ross, 2021) |  Epic, an American healthcare software company (and prominent EHR vendor), launched the ”Epic Sepsis Model” (ESM) in 2017, a proprietary LR model that alerts patients at high risk of sepsis from data ”routinely recorded within the EHR”. From 2021 onward, external and independent audits or investigations showed that the model was flawed; e.g., it had lower AUC than reported, had a high false alarm rate, etc.
20 |  US HHS and UNOS (the United Network for Organ Sharing)’s ”Acuity Circles Policy” (was, [n. d.]) |  A joint investigation by the Washington Post and The Markup exposed how a new shift in the UNOS allocation algorithm’s criteria in removing distance as a factor for organ donation and prioritizing sickest patients regardless of location, drastically decreased the number of transplant surgeries and increased deaths in poorer states.
21 |  UnitedHealth & NaviHealth’s ”nH predict” Post-Acute Care Management (Pierson and Pierson, 2023) |  Families of deceased beneficiaries filed a lawsuit in the U.S. District Court for the District of Minnesota against a insurance company’s healthcare algorithm for determining post-acute care. The algorithm prematurely discharged patients from facilities against doctor recommendations and discontinued payment for necessary services and claims, targeting those on Medicaid.
22 |  U.S. Health Systems’ High-risk Healthcare Management” Prediction (Obermeyer et al., 2019) |  Researchers at the UC Berkeley School of Public Health exploited a rich dataset that provided insight into a live, scaled algorithm that had been deployed nationwide, and found it perpetuated significant racial bias through using health care costs to determine who was most likely to benefit from care management programs.
23 |  New Zealand Health System’s Equity Adjuster-Waitlist Tool (Perry, 2023) |  The ACT New Zealand political party and other clinicians started a petition against Auckland Health Minister’s efforts to make healthcare more equitable for low-income, rural, and indigenous Maori and Pasifika communities through an algorithm that adjusted patients’ position in elective surgery waitlists, claiming it was racial discrimination against other New Zealanders.

|  Case Name |  Case Description
---|---|---
|  Property Tech |
24 |  RealPage’s YieldStar Algorithm (Vogell et al., 2022) |  RealPage, a Texas-based company that provides property management software for landlords, developed an algorithm, ”YieldStar”, that suggests daily prices for open rental units. Critics claim that the software brings anti-trust concerns and has resulted in rent inflation across the U.S.
|  Education |
25 |  UK Grading Tool (Meadway, 2020) |  In the 2020 pandemic, UK-based boards decided to use an algorithm to grade students’ GCE A-level exams (international certificates which are important for university placements and scholarship opportunities). Worldwide protests broke out when the algorithmically calculated results were announced, which resulted in the grades being globally rescinded and re-assigned one week later.
26 |  SAS Institute’s Educational Value-Added Assessment
System (EVAAS) in Houston (Capo and Bass, 2017) |  Houston Independent School District ruled in the settlement of a lawsuit brought by Houston and the American Federation of Teachers, to terminate the use of EVAAS, which used a student’s performance on prior standardized tests to predict academic growth in the current year, as basis for evaluating teacher effectiveness.
|  Facial Recognition |
27 |  San Diego County Facial Recognition Tool (TACIDS) (System, 2021) |  San Diego County had a regional database known as the Tactical Idenfication System (TACIDS), which contained 1.8million booking photos and was used by 30 law enforcement agencies (including ICE). Use of the database was ruled illegal by CA state AB 1215.
28 |  NYPD using Clearview AI (Haskins, 2021) |  Released public records revealed that the NYPD had a formal relationship with facial recognition company Clearview AI, including a ”trial” vendor agreement, despite the PD’s remarks that they had never had a relationship with the vendor. The NYPD released a formal facial recognition policy that banned use of Clearview in spring of 2020.
29 |  Detroit Project Green Light + DataWorks Plus (Hill, 2020b) |  The Detroit PD signed a contract with a for-profit vendor (DataWorks Plus) to begin running facial recognition on live video streams from surveillance cameras (from the ”Project Green Light” initiative), and other surveillance cameras (e.g., drones).