# Confronting Structural Inequities in AI for Education

Michael Madaio
Microsoft Research
New York, NY
Su Lin Blodgett
Microsoft Research
Montreal, Canada
Elijah Mayfield
University of Pennsylvania
Philadelphia, PA
Ezekiel Dixon-Román
University of Pennsylvania
Philadelphia, PA

###### Abstract

Educational technologies, and the systems of schooling in which they are
deployed, enact particular ideologies about what is important to know and how
learners should learn. As artificial intelligence technologies—in education
and beyond—have led to inequitable outcomes for marginalized communities,
various approaches have been developed to evaluate and mitigate AI systems’
disparate impact. However, we argue in this paper that the dominant paradigm
of evaluating fairness on the basis of performance disparities in AI models is
inadequate for confronting the structural inequities that educational AI
systems (re)produce. We draw on a lens of structural injustice informed by
critical theory and Black feminist scholarship to critically interrogate
several widely-studied and widely-adopted categories of educational AI systems
to demonstrate how educational AI technologies are bound up in and reproduce
historical legacies of structural injustice and inequity, regardless of the
parity of their models’ performance. We close with alternative visions for a
more equitable future for educational AI research.

_Keywords_ critical data studies, educational AI, equity, algorithmic fairness

##  1 Introduction

At the time of writing, countries around the world have closed their schools
in successive waves, impacting nearly 1.25 billion students across all levels
of
education.111https://www.worldbank.org/en/data/interactive/2020/03/24/world-
bank-education-and-covid-19 As institutions have shifted to distance learning,
or a hybrid of in-person and remote instruction (Reich et al., 2020),
educational leaders have increasingly turned to technological solutions to
provide continuity of learning for students (Teräs et al., 2020).
Administrators and teachers have driven students into remote learning
environments that may have never been chosen, if the world had stayed as it
was in 2019. Students will be impacted in enduring ways as the pandemic
continues to exacerbate existing systemic inequities in education (Aguliera
and Nightengale-Lee, 2020; Dorn et al., 2020).

As educational technologies are proposed as a solution for crisis-induced
remote learning (Teräs et al., 2020), discussion of equity often centers
around the assumption that increasing access to learning technologies will
reduce inequity in educational opportunities (Hall et al., 2020). While some
educational technologies may be beneficial for some learners (Kulik and
Fletcher, 2016), the distributive logic that equates access to technology as a
form of equity (e.g., (Holmes et al., 2021)) has motivated widespread adoption
of learning technologies (Pei and Crooks, 2020), as well as widespread data
collection on computational traces of students’ learning, data that is then
used as part of educational algorithmic systems222Data-driven educational
algorithms have a lengthy history, from computer-aided instructional systems
in the 1970s (Carbonell, 1970), to intelligent tutoring systems that provide
feedback and instructions to learners on the basis of students’ data (Pane et
al., 2014), with more recent versions of such systems drawing on advances in
machine learning (Piech et al., 2015) and natural language processing
(Mayfield et al., 2019).. Substantial research on data-driven algorithmic
systems (often referred to as artificial intelligence, or AI) in other domains
has demonstrated the potential for such systems to reproduce and amplify
societal biases in a variety of ways (see Barocas and Selbst (2016) for an
overview), or introduce novel sources of unfairness through disparities in
their performance (e.g., Buolamwini and Gebru (2018)), in domains as
widespread as criminal justice (Richardson et al., 2019), healthcare
(Obermeyer et al., 2019), and hiring (Raghavan et al., 2020). In spite of
these known issues for AI systems, enthusiasm for AI in education has never
been higher, with millions of dollars in funding for institutes of AI in
education333https://www.colorado.edu/today/ai-education and a recent National
Science Foundation report on educational AI (Roschelle et al., 2020) that
optimistically treats such systems as inherently beneficial, arguing that such
systems may be solutions to educational inequity.

In education research, quantitatively-oriented researchers may be motivated to
use their training to evaluate biases in educational AI as a neatly
quantifiable construct, with error bars and standard deviations, reporting
differences in system performance or students’ learning outcomes by
demographic groups (e.g., race, gender, or region). Indeed, this has been the
dominant paradigm in the nascent field of algorithmic fairness (see (Hanna et
al., 2020) for a critique of this paradigm). But social scientists and
critical theorists of education have argued (e.g., Dixon-Román (2017);
Williamson (2017)) that such an approach to quantifying bias misses the bigger
picture—that socio-historical processes both within and outside of the
classroom shape the design, use, and evaluation of educational technologies.

In this educational crisis, it may be the case that concerns about equity may
be sublimated, in the interest of using any means necessary to restore
normalcy to systems of schooling. Naomi Klein, writing about the response to
Hurricane Katrina’s impact on education in New Orleans, described how
“disaster capitalists” such as Milton Friedman saw in that crisis an
opportunity to “radically reform the educational system” of New Orleans
through immense cuts to public education in favor of subsidies to for-profit
private charter schools (Klein, 2007). Friedman’s response to the Katrina
crisis was to redirect resources for public education towards for-profit
schools. Yet, there are other responses to crisis. Social philosopher Pierre
Bourdieu describes how moments of crisis present opportunities to challenge
and shift dominant schemes of thought by “rupturing” existing social
structures (Bourdieu, 1977). The coronavirus pandemic is a global crisis in
nearly every sector. Rather than enabling technocapitalist interests in
selling products toward making technology inextricable from the fabric of
civic life444https://www.theguardian.com/news/2020/may/13/naomi-klein-how-big-
tech-plans-to-profit-from-coronavirus-pandemic, this moment may be an
inflection point for education decision-makers, researchers, and technologists
to radically rethink the way that education is done and how our technologies
are designed Moore et al. (2021)555https://www.de.ed.ac.uk/news/edtech-
pandemic-shock-blog-ben-williamson-and-anna-hogan. Given the known risks for
(un)fairness and (in)equity in AI more generally, and recent enthusiasm about
data-driven educational algorithms specifically (or, educational AI), it is
thus imperative that concerns about structural inequities are taken seriously,
so that our educational AI systems do not simply reproduce legacies of
inequity through their design.

Our article has one purpose: to draw on lenses from critical theory to
interrogate and rethink what equity means for educational AI. We begin by
summarizing the current state of research on algorithmic fairness, including
approaches to measuring fairness of algorithms and their limitations at
capturing more fundamental issues of structural inequity, informed by Black
feminist theory. We then discuss the historical legacies of (un)fairness and
(in)equity in systems of schooling more broadly and critically examine the
operating logics of several widely-researched classes of educational AI
systems. We uses these categories of educational AI, not to suggest that they
are exhaustive, but as fertile sites to examine how educational AI may
fundamentally perpetuate structural inequities, regardless of the quantitative
“fairness” of the underlying algorithms’ performance. Finally, we call for new
methods and visions for justice-oriented approaches to research and design of
learning science and educational technology, drawing on related calls for
design justice, counterhegemony, and technology refusal in adjacent fields. We
argue for the need for new, more radical, visions for the design (or
dismantling) of educational AI in order to bring about a more equitable and
justice-oriented educational future.

##  2 Methods and Challenges to Measuring Fairness in AI

The rhetorical promise from designers of data-driven educational algorithms
(i.e., educational AI) is that they are able to increase learning overall
while reducing inequities, by helping all students learn in more personalized
ways. One need only look at the theme of a recent AI in Education
conference,_“Education for All in the XXI Century,”_ which solicited papers to
explore how AI systems might contribute to reducing educational inequities.
This line of thinking follows a broader tradition in technical fields that
critics call _techno-solutionism_ (Morozov, 2013). Fundamentally, work in this
vein assumes that technology can solve deep-seated, complex social issues or
that technology is inherently less biased than humans. Critics have reason to
be deeply suspicious of this theory of change. Before we explore those
critiques further, though, we will first review prior work on measuring
fairness in education, and then discuss the current state of research on
defining and measuring (un)fairness in data-driven algorithmic systems,
including machine learning and artificial intelligence (AI) and for such
systems used in education. In other words, what _does_ it mean for a machine
learning or AI system to be fair?

###  2.1 Measuring fairness in educational assessments

Statistical evaluations of equality for test questions across race and gender
has a long history in the psychometrics of high-stakes exams like the SAT,
going back at least six decades (Anastasi, 1961; Donlon, 1981; Sehmitt and
Dorans, 1990). In 1968, Cleary (1968) studied the problem of fair test use
across Black and white students for admissions to universities that had only
just been racially integrated. Later, with ETS, Linn (1973) undertook an
extensive quantitative review of test subgroup fairness in the SAT, leaning on
definitions from Thorndike (1971). From those early days to more modern
research, the primary questions have long been the same: Are the psychometric
properties of a test variant or invariant across groups? Is there differential
performance for different groups, and, if so, does the test produce valid
estimates for each group? Are the items on the test equally difficult for all
students, holding their estimated abilities constant and, if so, are there
systematic reasons for such differential item functioning? In more recent
decades, Samuel Messick’s and Lee Cronbach’s concerns for the social
consequences of test score use and interpretation have been given more
attention. Older versions of these standardized tests famously included
culturally discriminatory world knowledge like polo, regattas, ballet, and
horseback riding (Weiss, 1987). The approach to solving this has been nearly
uniform within the psychometric field: measure marginal distributions of
outcomes by demographic group, and make use of items and tests that meet a set
of defined criteria for group-level fairness (Hutchinson and Mitchell, 2019).

These research traditions have been focused on achieving reliable measurement
and valid predictions of student ability across student populations. However,
more fundamental questions have been sidelined from the mainstream discourse
in the field—questions such as any interrogation of how “ability” was defined,
or whether the basic prediction goal was desirable (or desired by the
students), or who such an assessment might disproportionately harm, even if it
was functioning “correctly” (Dixon-Román, 2017; Karabel, 2006; Lemann, 2000).
These are thorny questions, and they are fundamental to interrogating the
meaning of equity in educational technologies.

###  2.2 Measuring fairness in AI and machine learning

Critiques of the social implications of technology are of course not limited
to educational technologies. For decades, scholars such as Langdon Winner
(Winner, 1980) and others (Winograd et al., 1986) have argued that technology
instantiates social and political relationships. Early work in this area
examined several types of bias arising from computational systems (Friedman
and Nissenbaum, 1996), with other work critiquing the “mythology” of Big
Data—“the widespread belief that large datasets offer a higher form of
intelligence and knowledge that can generate insights that were previously
impossible, with the aura of truth, objectivity, and accuracy” (boyd and
Crawford, 2012)—while other work questioned the ability of existing American
anti-discrimination law to address algorithmic discrimination (Barocas and
Selbst, 2016). In spite of this rich variety of interdisciplinary critical
scholarship, the field of AI and machine learning has recently converged on a
narrower framing of how to understand the harms that AI systems might
engender.

In recent years, academic approaches arising from the nascent field of
fairness, accountability, transparency, and ethics in AI and machine learning
(FAccT)666https://facctconference.org/ have primarily centered on evaluating
the _fairness_ of algorithmic decision-making systems. Algorithmic decision
systems are used in a variety of high-stakes domains to support, or in some
cases replace, decisions about access to resources and opportunities, for
applications as varied as healthcare services (Obermeyer et al., 2019), parole
sentencing (Angwin et al., 2016; Dixon-Roman et al., 2019), and student-school
matching (Robertson and Salehi, 2020). Current approaches to assessing biases
in such algorithms have developed mathematical formulations of _group
fairness_ (Kleinberg et al., 2016; Corbett-Davies and Goel, 2018) and
_individual fairness_ (Dwork et al., 2012), largely for algorithms that
perform classification tasks (i.e., predicting one (or more) of a set of
categorical labels, as in Kotsiantis et al. (2007)).

Group fairness metrics for classifiers typically measure the equality of
predictive performance across groups defined by protected attributes.777United
States law specifically defines the following characteristics as protected:
race, color, religion, sex and gender identity, pregnancy, national origin,
age, disability, veteran status, and genetic information. Internationally,
specific protections differ by jurisdiction. By contrast, individual fairness
approaches look at similar _individuals_ , defining similarity in a task-
specific way that typically does not rely on demographic labels, and ask
whether similar users of a given technological system can expect similarly
allocated decisions, error rates, or system quality and effectiveness (Gillen
et al., 2018; Kim et al., 2018). Proposed solutions for mitigating unfairness
have typically involved developing more “diverse” datasets (i.e., more data
from “under-represented” social groups) and algorithmic mitigation approaches
(Agarwal et al., 2018), although there are reasons for wariness about both
proposed solutions.

###  2.3 Challenges to measuring algorithmic fairness

Challenges to measuring algorithmic fairness abound. For instance,
Chouldechova (2017) showed that a number of common fairness metrics cannot be
simultaneously satisfied (while population base rates are different),
requiring an ultimately socio-political choice by practitioners of which
fairness metric to use. These choices have serious consequences for any
fairness evaluation, and belie the rhetoric that developing more fair systems
is a straightforward task and that models can simply be “de-biased” (Bolukbasi
et al., 2016). In addition, investigations of how multiply-marginalized
individuals are affected by decision-making systems are critical, but scarce.
Buolamwini and Gebru (2018)’s path-breaking examination of facial recognition
systems’ performance at the intersections of race and gender—finding that such
systems perform particularly poorly on images of Black women—represents a
notable exception.

Moreover, measuring (and attempting to mitigate) algorithmic (un)fairness also
typically requires gathering demographic data in the first place, which may
threaten the privacy of marginalized groups, who have historically been most
vulnerable to data collection and surveillance (Eubanks, 2018). Studying if a
lending system is racially biased, for instance, often involves collecting
customers’ racial information, which can be a fraught process that requires
categorization of identities into a small number of fixed labels and the
collection of potentially sensitive demographic data. Data collection and
privacy concerns are not new for education researcher. There is significant
regulation in some countries designed to protect students’ privacy, such as
the Family and Educational Rights and Privacy Act (FERPA) in the US (Daggett,
2008), although there is contention about its ability to account for modern
data infrastructures (Zeide, 2015). There has also been substantive critical
writing about student data privacy issues in learning analytics, including
concerns about the storage, sharing, and afterlives of student data (Slade and
Prinsloo, 2013).

Finally, approaches to evaluating algorithmic fairness have mostly been
developed for domains where decision-making systems allocate opportunities or
resources—e.g., lending decisions, criminal sentencing, or university
admissions decisions (Kleinberg et al., 2016); errors by such systems give
rise to what has been called _allocational_ harms (Crawford, 2017). However,
AI systems, including many systems used in education, often do more than
simply allocate opportunities or resources. Recent scholarship is also
identifying the ways in which group-based and individual fairness approaches
may be inadequate for the social or dignitary harms that may also arise, which
Crawford (2017) has termed harms of _representation_. For instance, the
reproduction of racist characterizations arising from Google Photos’ labeling
Black faces as gorillas (Simonite, 2018; Noble, 2018), or the reproduction of
gendered expectations by digital assistants who respond coyly to sexual
harassment (Fessler, 2017). These represent fundamentally different kinds of
harms than those described above, and may not be well-captured by existing
group or individual fairness approaches.

###  2.4 Measuring fairness in educational algorithms

Researchers working on educational algorithms, recognizing that their systems
are not exempt from the same threats to fairness as in AI systems more
broadly, have begun to apply methods for evaluating the disparate impact of
educational algorithms, predominantly applying group-based approaches to
conceptualizing fairness. In their review of the state of algorithmic fairness
in education, Kizilcec and Lee (2020) discuss existing approaches to assessing
fairness in educational algorithms, including research evaluating how the
accuracy of a system designed to detect students’ affect (e.g., confusion,
frustration, boredom) differed for groups of students in urban and rural
schools (Ocumpaugh et al., 2014); research evaluating how the accuracy of a
model designed to predict the likelihood of student dropout differed by the
gender of the student (Gardner et al., 2019), and more (Doroudi and Brunskill,
2019; Lee and Kizilcec, 2020; Kizilcec and Lee, 2020). In spite of extant work
that takes a more critical lens to understanding such phenomena – such as work
exploring the ethical implications of resource allocation based on automated
prediction (Prinsloo and Slade, 2014) or the implications of making decisions
to de-prioritize individual support on the basis of group outcome measures
(Scholes, 2016) – such critical work has been sidelined in favor of group-
based fairness approaches, popular in the AI and machine learning community
more broadly, as exemplified by Kizilcec and Lee (2020). The implicit
assumption of much work on algorithmic fairness, made explicit in Kizilcec and
Lee (2020) is that “the innovation is beneficial in general” and the work to
be done is to ensure that the benefits are fairly distributed (a stance echoed
in recent work by other educational AI researchers as well, as reported in
Holmes et al. (2021)). Such an approach, however, fails to address the classes
of representational harms previously described and fails to address the
second-order consequences of data collection; and, more fundamentally, there
is good reason to be skeptical that this optimistic assumption is in fact
true.

###  2.5 Systemic critiques of group-based fairness approaches

As research on algorithmic fairness has developed, scholars (particularly from
sociology and science and technology studies (STS)) have brought more
fundamental challenges to the algorithmic fairness approaches described
previously, critiquing the orientation of such approaches towards technical
issues of models and datasets, rather than towards the larger social and
historical context (e.g., institutions and social structures) driving the
inequities that manifest in technical systems. More specifically, sociological
(and Black feminist) scholarship has long recognized that social forces of
racism, sexism, cis heteronormativity, ableism, settler colonialism, and so
forth, have shaped the foundations of our institutions and social
structures.888We write as researchers in the United States, but similar
systemic forces shape social structures everywhere, in different ways. Central
to these analyses is the understanding that these oppressions are _systems_ ,
encompassing both unjust distributions of political power and economic
resources as well as the ideologies and discourses that justify these
distributions (Feagin and Ducey, 2000; Delgado and Stefancic, 2017). These
systems are also ordinary, reproduced by the “routines, practices, and
institutions that we rely on to do the world’s work” (Delgado and Stefancic,
2017). Finally, they are interlocking; the systems of sexism, racism, and so
forth do not operate independently, but are intersecting structuring forces.

This last property is captured by two conceptual models developed by Black
feminist scholars: _intersectionality_ and the _matrix of domination_.
Intersectionality is a term coined by Kimberlé Crenshaw as part of her
groundbreaking analysis showing that anti-discrimination law—which addressed
gender- and race-based discrimination separately—was inadequate to address the
overlapping discrimination experienced by Black women (Crenshaw, 1989, 1990).
In highlighting how Black women were marginalized both by the feminist
movement and by the civil rights movement, Crenshaw’s analyses illustrated the
need to move beyond single-axis analyses of injustice. Patricia Hill Collins’
“matrix of domination,” meanwhile, is a model that “helps us think about how
power, oppression, resistance, privilege, penalties, benefits, and harms are
systematically distributed” (Collins, 1990) (see (Costanza-Chock, 2018) for a
discussion of the matrix of domination in technology). This scholarship
suggests that technologies developed in a world whose institutions are thus
structured will reflect the values and legacies of this social and historical
context, as they are steeped in the ideologies and discourses that maintain
these institutions and their interlocking systems of oppression. It is thus
critical to analyze these technologies through the lens of these oppressive
institutions, ideologies, and discourses. Indeed, emerging scholarship is
beginning to identify the consequences of existing fairness approaches’
failure to account for this broader social and historical context. We outline
a few of them here.

First, in attempting to treat different demographic groups the same, _group
fairness_ -based approaches ignore the reality that institutions and
ideologies work to maintain profoundly unequal social orderings, with the
result being that different groups are not treated the same in society. Hanna
et al. describe this situation in their examination of race in algorithmic
fairness (Hanna et al., 2020):

> _“By abstracting racial categories into a mathematically comparable form,
> group-based fairness criteria deny the hierarchical nature and the social,
> economic, and political complexity of the social groups under consideration…
> In short, group fairness approaches try to achieve sameness across groups
> without regard for the difference between the groups… This treats everyone
> the same from an algorithmic perspective without acknowledging that people
> are not treated the same."_

Second, in ignoring the sociological complexity of race, gender, and other
frequently used attributes, fairness measurements may be inappropriate or
inconsistent; for example, Hanna et al. delineate the many ways in which race
has been operationalized in measurements of algorithmic fairness, from
interviewer-observed, appearance-based classifications to self-declared racial
classifications (Hanna et al., 2020). Fairness approaches also often measure
with respect to a single attribute, ignoring the complexities of interlocking
oppressions, thereby making it impossible to understand how people who are
multiply marginalized experience technologies differently (Costanza-Chock,
2020).

To illustrate this disconnect, consider data-driven algorithms designed to
detect toxic language in online social media. These systems are well-
documented as treating African American Language (AAL) social media messages
as more toxic than Mainstream U.S. English (MUSE) ones (Davidson et al.,
2019), and for this performance difference, the systems have been rightly
critiqued from a group fairness perspective (Sap et al., 2019). But, using the
lens of structural injustice, researchers must also recognize the ideologies
of language driving this outcome, which construct MUSE as acceptable and
normal and AAL as ungrammatical or inappropriate, and acknowledge the impact
of these systems, namely the reproduction of harmful patterns of linguistic
stigmatization, discrimination, and disenfranchisement, which maintain
hierarchies of language and race and naturalize unjust political and economic
arrangements (Rosa and Flores, 2017). These findings have clear implications
for educational technologies (e.g., Mayfield et al. (2019); Loukina et al.
(2019)), which we describe in more depth in the following section.

More broadly, analyses that do not engage with technologies’ social and
historical context fail to recognize how that context drives decisions at many
points in decision-making systems’ development and deployment pipeline, from
the choice of problem definition itself (e.g., “what is the problem to be
solved?”) to the choices of data (e.g., “whose data is prioritized?” or, more
likely: “whose data can be easily acquired?”) and choice of evaluation metrics
(Passi and Barocas, 2019). Without examining this context, fairness approaches
often treat biases as emerging incidentally, rather than as reflections of the
institutions and ideologies structuring society (Benjamin, 2019b), or, even
when such structuring forces are acknowledged, as in Kizilcec and Lee (2020,
p.4), they are sidelined in favor of group-based fairness evaluations. Such
approaches address only individual undesirable features, models, and datasets
rather than the larger ideologies, institutions, and practices that produce
them (Hoffmann, 2019). These approaches also fail to recognize how
technologies (re)produce social meanings (cf. Winner (1980)), thereby
maintaining these institutions and ideologies. In this way, algorithmic biases
are treated as disconnected from both their origins outside of technology and
their impact on society.

Finally, in proposing solutions that are technical in nature—involving data
collection, more sophisticated model development, etc.—algorithmic fairness
approaches assume that continued technological development is the solution to
unfairness in decision-making systems, rather than asking whether some systems
should be fundamentally re-designed, or perhaps not built at all (Cifor et
al., 2019; Green, 2018; Keyes, 2019a; Bennett and Keyes, 2019; Hoffmann, 2019;
Baumer and Silberman, 2011; Barocas et al., 2020). Thus, algorithmic fairness
approaches maintain traditional power relations between technologists and the
communities by privileging technologists in the decision-making processes
about how technologies should be designed and deployed (Hoffmann, 2020;
Kalluri, 2020). In sum, the current state of the art in understanding
differential harms of AI largely centers around evaluating group-based and
individual fairness metrics. Given the historical legacy of group-based
approaches to evaluating educational assessments, it is understandable why the
educational AI field has similarly adopted such approaches. However, as we
have outlined, conceptual models from Black feminist scholarship suggest that
a lens of structural injustice is needed to understand the fundamental equity
issues posed by data-driven educational technologies.

##  3 (In)equity in Educational AI

One approach to engaging with equity in AI systems in education—what appears
to be the dominant view—might be to see them as inherently beneficial for
education and to seek to make those systems accessible and accurate for as
many learners as possible (e.g., (Holmes et al., 2021; Kizilcec and Lee, 2020;
Hansen and Reich, 2015)). Indeed, if one adopted the group-based fairness
approaches from AI research wholesale, one might simply seek to improve the
relative performance of an educational AI system for different populations of
learners (Madnani et al., 2017; Loukina et al., 2019; Lee and Kizilcec, 2020;
Kizilcec and Lee, 2020). However, given the interlocking systems of oppression
in a settler colonialist, patriarchal, cis heteronormative, ableist society
such as the United States, we argue that educational AI technologies that
simply reproduce the status quo under such a “matrix of domination” will be
fundamentally inequitable.

Thus, following other critical interventions in reviews of education research
(e.g., (Kumashiro, 2000; North, 2006)), in this section, we examine several
widely-studied paradigmatic categories of educational technologies using
lenses drawn from critical theory to identify how their operational logics may
reproduce structural inequities in education, regardless of the parity of
their algorithmic performance. We highlight these categories of educational AI
systems, not as an exhaustive list, but as a demonstration of how lenses from
critical theory may help shift the focus of the dominant paradigm of
educational algorithmic fairness discourse towards a focus on structural
inequity.

###  3.1 Legacies of educational systems of oppression

The policies, practices, and discourses that have shaped (and are shaped by)
interlocking systems of oppression have influenced every part of the current
educational landscape, from the nature of the curriculum, to the pedagogical
methods used to teach it, to the systems of assessing what students know, to
the technologies designed to teach and assess that curriculum in specific
ways. At the time of this writing, Black children in America are more than
twice as likely as white children to attend high-poverty schools999More than
sixty years after the U.S. Supreme Court declared school segregation
unconstitutional in their 1954 Brown v. Board of Education ruling, many
American schools remain heavily segregated, and the legacy of historical
racial oppression hovers over the entire American school system. In a recent
Economic Policy Institute analysis of data from the National Assessment of
Educational Progress, nearly seven in 10 (69%) Black students attended a
school where a majority of students were Black, Hispanic, Asian, or American
Indian (García, 2020). In contrast, only one in eight (13%) of white students
attended a school where a majority of students were people of color (García,
2020). Black and Hispanic students are also substantially more likely than
white students to not complete high school, with 79% and 81% graduation rates,
respectively, compared to 89.1% of white students (for Education Statistics,
2020), for 2017-18, the latest year on record. Students with disabilities
reported the lowest rates of school completion, with only 67.1% completing
high school. , and such intertwined oppressions have led to worse educational
outcomes for Black students in high-poverty schools (García, 2020).

Educational policies and funding mechanisms, ostensibly motivated by wanting
to support greater equity, have instead reproduced the status quo of
historical legacies of structural injustice in education. For decades,
government funding for education has been largely tied to schools’ performance
on standardized testing, tracing back to the 1983 publication of A Nation at
Risk and through the measurement-focused era of No Child Left Behind. The
logic here is that schools must first assess students to understand gaps in
performance between students from different backgrounds, in order to allocate
resources to close those gaps — a similar logic to the group-based approach to
algorithmic fairness described earlier, with similar shortcomings. Indeed,
performance on standardized tests has been shown to be highly correlated with
parental wealth and families’ ability to pay for expensive test-prep for their
children—opportunities not often available to many marginalized learners
(Dixon-Román, 2017).

This assessment-focused paradigm has driven investment in and adoption of
educational technologies to support these assessments, ostensibly to first
measure and then mitigate gaps in students’ performance. In the 2010s,
American government funding for education was disbursed through the Race to
the Top program, driving a surge in spending to move from summative to ongoing
formative assessments, motivating increased adoption of educational technology
in the classroom to deliver such ongoing assessments. But these investments in
the early 2010s met numerous logistic hurdles, such as readily available
broadband and wifi access, and the ongoing challenges of maintaining
sufficiently updated computing devices (Boser, 2012; Fox et al., 2012). These
gaps in access to educational technology infrastructures again fell
disproportionately on already disadvantaged learners, further widening
inequities in access to educational materials along socio-economic lines,
which in the U.S. often means racialized communities (Warschauer and
Matuchniak, 2010).

It is important to step back and critically question the assumption
underpinning those investments — that the issue at hand is simply providing
access to the right technology that will reverse legacies of structural
injustice impacting marginalized communities. This is a self-perpetuating
logic, where group-level performance differences on assessments leads to
investment and adoption of educational technologies in order to provide so-
called personalized learning opportunities to reduce those performance
differences for different groups of learners. In this “distributive justice”
paradigm (Pei and Crooks, 2020), the goal is thus to ensure all students have
equivalent access to those technologies (i.e., reducing group-level
differences in access).

During the COVID-19 pandemic, this paradigm has become the dominant one. As
schools around the world have transitioned to remote learning during the
crisis, school administrators, teachers, and parents look to technology to
support “continuity of learning” (Reich et al., 2020; Teräs et al., 2020;
Williamson, 2020). Much of the conversation has centered around how to provide
equitable access to learning technologies, be that devices on which to learn,
or access to digital learning platforms. Although providing continued access
to educational opportunities in a crisis is certainly important, and ensuring
equitable access to those learning opportunities is an important first step,
it is critical to consider whether that enthusiastic adoption of technology
has opened the door for data-driven educational AI algorithms that, once
provided, reproduce the structural injustices of the education system more
broadly.

To provide one brief example, consider classroom discussion practices. In her
book _Teaching to Transgress_ , bell hooks argues that “our ways of knowing
are forged in history and relations of power” (hooks, 1994). Her classic
example of this is the white male faculty member in a university English
department who primarily teaches books by “great white men” and whose
pedagogical practices perpetuate a white supremacist ideology. In many such
classrooms, white male students may feel more comfortable volunteering in
discussions more often, and may thus be called on more often by instructors
(hooks, 1994). When those discussions are mediated by technology — or moved
entirely to a digital format, as in the now-ubiquitous remote learning
exacerbated by the COVID-19 pandemic — those inequitable pedagogical practices
are reproduced and amplified. One can imagine this playing out in a number of
ways, from a technology that tracks students’ participation in class or awards
them points for such participation (perhaps inspired by work such as (Ahuja et
al., 2019)), to educational language technologies that summarize group
meetings or class discussions (cf. (Li et al., 2019; Zhao et al., 2019)).

We thus extend hooks’ argument to make the case that educational AI
technologies are forged in historical relations of power. Classroom
pedagogical practices shape the design paradigms and goals of learning
technologies, shape the data available for training educational AI systems,
shape the ways in which those technologies are deployed in the classroom, and,
more broadly, shape the values that are enacted by these systems (cf. (Winner,
1980)). Thus, the problem to solve is not simply providing more students with
access to such systems, or improving the accuracy of the systems’ predictive
models for different subgroups of learners (cf. (Kizilcec and Lee, 2020)).
Researchers and educational technologists must grapple with the legacies of
historical disenfranchisement of marginalized learners and the impact those
legacies have on the design paradigms of educational technology.

In the following sections, we will discuss several such design paradigms of
educational AI systems that reproduce historical legacies of educational
inequity. We will discuss how a lens of structural injustice makes visible—and
may allow us to challenge—the interlocking systems of oppression structuring
the development and deployment of educational AI that a more narrowly scoped
algorithmic fairness approach cannot. Through serious attention to these
criticisms, we hope to generate provocative questions and directions toward
alternative futures of educational AI technologies in the final section of the
paper.

###  3.2 Biopolitical educational technologies

Here, we trace examples of categories of educational AI systems that may
reproduce structural injustices — regardless of the models’ accuracy or
fairness. We start first with biopolitical AI systems, or approaches that
model students’ socio-emotional states, in order to induce students to act in
desired ways. Such socio-emotional systems are based on and perpetuate
racialized ideologies about behaviors and emotions deemed appropriate for
learning (Williamson, 2017). In the Artificial Intelligence in Education
community (AIED) and adjacent communities such as Educational Data Mining
(EDM) and others, many papers have been published on the design and evaluation
of systems that purport to model learners’ “affective state” (e.g., (Henderson
et al., 2019; Mandalapu and Gong, 2018; D’Mello and Graesser, 2010)) and
systems that use those estimates of learners’ affective state to recommend
specific types of content to learners (e.g., (Tsatsou et al., 2018;
Athanasiadis et al., 2017)). These systems use audio, video, and other data
sources to label students’ “affect,” or emotional state as one of a set of
pre-determined labels, as defined by the researchers, following from the
broader field of affective computing (Picard, 2000). Research in educational
affect detection has largely converged around a set of purported affective
states, including boredom, confusion, and frustration (D’Mello and Graesser,
2010), with some researchers also including “engagement” or “flow”, delight,
and others (Okur et al., 2017; Chang et al., 2018).

The fundamental assumptions underlying these systems are that learners’
affective states are legible to technology and that some affective states are
more productive for learning than others. One implication of this is that if
AI models can identify such states, they can also “nudge” students into
affective states that the systems’ designers may believe to be more conducive
for learning. These assumptions represent an ideology akin to what Foucault
and others have called biopolitics or biopower, or the role of technology and
policy to control people’s biophysical states through detection, persuasion,
coercion, or, when control is not possible, discipline (Williamson, 2017;
Foucault, 1984). Biopolitical educational AI systems contribute to broader
systems of algorithmic control through codifying the complex emotional states
of learners in narrowly defined ways that are legible to the technologies, and
which serve to reinforce particular, often racialized, ideologies about which
emotions and behaviors are valued in the classroom, and which are punished
(Gregory and Fergus, 2017).

Behavioral management and discipline in schools—particularly in America—is
shaped by ideologies that reproduce structural injustices. School policies
such as zero tolerance have led to racial disparities in student suspension
and expulsion rates (Keleher, 2000; Bottiani et al., 2017; Hoffman, 2014),
with doubly marginalized students receiving greater disciplinary infractions
(García, 2020). These racist disciplinary practices have contributed to the
“school-to-prison pipeline,” where racialized students are disciplined at
higher rates, suspended, expelled, and funneled into the prison system
(Heitzeg, 2009; Kim et al., 2010). Structural injustice in schools’
disciplinary practices is multifaceted, involving multiple, interlocking
systems of oppression. In the GLSEN’s 2017 National School Climate Survey, a
majority of LGBTQ+ students (62%) reported that their school had
discriminatory policies or practices, such as disciplining public displays of
affection disproportionately for LGBTQ+ students, with many reporting policies
specifically targeting transgender students (Kosciw et al., 2018). Moreover,
LGBTQ+ students with disabilities were more likely to be disciplined in school
and to drop out of school than LGBTQ+ students without disabilities (Palmer et
al., 2016). Without considering structural injustice, the historical legacies
of these policies and practices of schooling become baked into technological
infrastructures underlying educational AI.

Let us take one educational technology company, ClassDojo, as an example.
Despite plucky rhetoric around “building classroom
communities”101010https://www.classdojo.com/, ClassDojo was designed to allow
teachers to track and reward students (via points) to incentivize what
teachers feel is appropriate behavior in the classroom (and occasionally
beyond) (Williamson, 2017). While ClassDojo is not currently a data-driven
system in the same way as educational AI such as affect detection systems
(which use data to train algorithms to identify affective states), the
underlying logic of systems like ClassDojo arises from the same biopolitical
ideology as affect detection AI and directly contributes to the creation of
behavioral data that may be used to train biopolitical educational AI.
Although a longer critique of biopolitical educational technologies such as
ClassDojo is beyond the scope of this paper (see (Williamson, 2017), or
(Brynjarsdottir et al., 2012) for a critique of “persuasive” technologies more
generally), ClassDojo is a clear example of how educational technologies may
reinforce dominant ideologies of behavioral appropriateness—which construct
racialized, LGBTQ+, disabled, and other marginalized students’ behavior as
inappropriate or criminal—instantiated into biopolitical technologies of
behavioral discipline and control. Thus, the racialized disciplinary and
behavioral practices and policies of schooling are reproduced and amplified by
educational AI technologies that detect, track, and influence students’
emotional states and behaviors in the classroom.

Now, during the ed tech boom driven by COVID-19 (Teräs et al., 2020),
educational institutions have increasingly111111 While automated proctoring
technologies (e.g., Proctorio, ProctorU, etc) had been in use prior to
COVID-19, the pandemic accelerated adoption of these platforms, with Proctorio
reportedly administering “2.5 million exams — a 900 percent increase from the
same period last year”. https://www.vox.com/recode/2020/5/4/21241062/schools-
cheating-proctorio-artificial-intelligence turned to technology to extend
their behavioral control into students’ homes, with one egregious example
being automated proctoring systems. Such systems encode particular ideologies
of what is considered to be normal and abnormal (and thus suspicious) behavior
during a remote test, ideologies that reflect and reproduce structural
injustices in the education system more broadly. The algorithms that power
such systems, including computer vision for facial detection and behavior
detection algorithms, have been shown in other contexts to perform worse for
women and people of color (Buolamwini and Gebru, 2018). However, even if these
models were to be tweaked in order to achieve parity of predictive performance
across groups, the underlying assumptions undergirding these systems is that
there is a set of normal—and thus desirable—behaviors, and behaviors other
than these are suspected of being indicative of academic dishonesty. For
neurodivergent learners, disabled learners, or learners with children or other
caregiving responsibilities that might require them to behave in ways outside
the narrow boundaries of what is considered to be normal behavior, the design
of such systems reinforces larger systems of oppression that marginalize and
punish any who fall outside of those encoded definitions of normality (Davis,
1995; Swauger, 2020; Cahn et al., 2020). This ideology algorithmically encodes
and perpetuates the carceral logics (i.e., “cop
shit”121212https://jeffreymoro.com/blog/2020-02-13-against-cop-shit/) of an
education system that punishes marginalized learners in innumerable ways via
the disciplinary practices of schooling.

###  3.3 Educational surveillance technologies

Underlying educational algorithms of biopolitical control—and educational AI
more generally—is an ed tech surveillance infrastructure that collects data on
students under the guise of
care131313https://www.insidehighered.com/blogs/college-ready-
writing/surveillance-state. Educational technologies—such as the
aforementioned affect detection systems—often rely on or deploy a pervasive
surveillance apparatus to collect multimodal (i.e., audio and video) data on
learners’ behavior, including in some cases from cameras deployed in
classrooms to monitor students’ posture, gestures, and movements (Ahuja et
al., 2019; Martinez-Maldonado et al., 2020; D’Mello et al., 2015). Although
not all educational surveillance approaches use AI, many draw on data-driven
algorithmic methods to detecting the presence of people and their body
movements (e.g., (Ahuja et al., 2019)) or on language technologies for
surveillance of students’ communication (e.g., (Beckett, 2019)) and, in
addition, provide the crucial infrastructure than ratifies and enables AI
systems (as discussed in (Krafft et al., 2021)). In a recent report from the
University of Michigan’s School of Public Policy, Galligan et al. argue that
“cameras in the classroom” (or educational surveillance tools more broadly)
reinforce systemic racism and discrimination, normalize surveillance for
children, and further marginalize vulnerable
groups.141414http://stpp.fordschool.umich.edu/sites/stpp.fordschool.umich.edu/files/file-
assets/cameras_in_the_classroom_full_report.pdf

Although some have argued that classroom surveillance systems can be used as a
tool to help students or support teachers’ professional development (Ogan,
2019), the reality of how these systems are marketed and used in schools
around the world may belie this optimistic view, as classroom surveillance
technologies are deployed often without students’ awareness or consent and,
once deployed, may be used for purposes other than they were intended
for.151515https://www.sixthtone.com/news/1003759/camera-above-the-classroom In
fact, although some tools for classroom monitoring appear to have been co-
designed with input from teachers and students (Holstein et al., 2019), the
vast majority do not appear to incorporate student or teacher input. Even in
such cases where students or teachers were involved in their design, such
approaches may merely be engaging in what has been referred to as
“participation-washing,” where stakeholders’ involvement in the design is used
to ratify decisions already made by the system designers (Sloane et al., 2020;
Cooke and Kothari, 2001), a point to which we return in Section 4. Moreover,
the normalization of cameras in the classroom, under the guise of care for
students, has now, during COVID-19, led to the pervasive extension of
surveillance into learners’ homes via Zoom (or other video conference
platforms used for learning) and the aforementioned automated proctoring
systems.

In addition to cameras in classrooms and homes, as part of a pervasive state
of surveillance, school districts have begun deploying language-based
surveillance systems to monitor and extract text data from students’ email and
messaging platforms under the guise of “school security” (e.g., (Beckett,
2019)). These systems use natural language processing (NLP) models to flag
messages or words that are hypothesized to be indicators that students will
commit violent acts. The rhetoric of these platforms posits that schools that
use them will be better able protect their students’ safety, following a
similar logic of paternalistic care as in other educational surveillance
initiatives. However, educational technologies often act as “racializing
forces” that encode and perpetuate racialized ideologies about language
(Dixon-Román et al., 2019), reinforcing the stigmatization and marginalization
of minoritized language varieties and speakers (Rosa and Flores, 2017;
Blodgett et al., 2020). Consider the earlier discussion of how language
technologies have been shown to identify speakers of African American Language
on social media as more toxic than speakers of Mainstream US English (Davidson
et al., 2019).

Similarly, NLP systems have been shown to have higher false positive rates on
text containing LGBTQ-related terms when attempting to detect hate speech
(Dixon et al., 2018). On social media platforms, when these algorithms
erroneously flag such language as hate speech, users may experience social or
dignitary harms, where language about LGBTQ+ topics is stigmatized as
inappropriate or offensive, and where (if such language is removed) their
ability to participate in public discourse about these topics is diminished
(Blodgett et al., 2020). This is bad enough. But in a school context, when a
student has a message flagged as harmful or potentially violent, schools may
dispatch social workers or disciplinary officers to intervene, thus (at a
minimum) creating a disciplinary record for that student (Beckett, 2019), or
leading to serious harm for those students’ lives and safety from the
encounter with the carceral state. By creating disciplinary records and
forcing interactions with institutions they might not otherwise have come in
contact with, these surveillance systems directly construct “misbehavior” or
“criminality” out of minoritized students’ language use and may lead to their
inclusion in databases of state apparatuses (cf. (Eubanks, 2018)).

This approach to widespread educational surveillance in the name of
security—like the ubiquitous surveillance of the post-9/11 world—operates
under what Brian Massumi has called the operative logic of pre-emption
(Massumi, 2015). These operative logics of mutual deterrence and pre-emptive
strikes are not limited to war and statecraft, but infuse themselves into
other aspects of our lives. Classroom surveillance technologies are one such
platform, where a military logic of pre-emption informs the design and
adoption of ubiquitous sensing infrastructure. Schools who purchase
surveillance platforms such as the “appearance recognition” system
Avigilon161616https://www.vox.com/recode/2020/1/25/21080749/surveillance-
school-artificial-intelligence-facial-recognition or email and chat
surveillance platforms Gaggle and Securly (Beckett, 2019) are able to argue
that they have taken steps to make their schools safer from the supposedly
ever-present threat of future school shootings, while having little evidence
of their actual efficacy in pre-empting this future state. In the meantime, as
surveillance infrastructure is adopted ever more widely, marginalized learners
are the ones who are most harmed (Weinstein, 2020). Now, during the COVID-19
pandemic, similar logics of care and pre-emption have motivated school
districts to contract with surveillance tech companies looking to capitalize
on the crisis, leading to (among other surveillance efforts) proposals for
students to wear badges to track their movements around
schools171717https://www.the74million.org/article/as-covid-creeps-into-
schools-surveillance-tech-follows/. The persistent legacy of racist school
disciplinary practices suggests that these surveillance technologies will
continue to reproduce those structural inequities for marginalized learners.

###  3.4 At-risk prediction technologies

The broad class of dropout prediction systems is another category of
educational AI systems that reproduces structural injustice, regardless of the
accuracy of such systems. Such systems focus on predicting learners’ risk of
failing to complete a class or a degree program, using behavioral data from
interactions with a given course (e.g., (Tang et al., 2018; Nur et al., 2019;
Lykourentzou et al., 2009; Liang et al., 2016; Márquez-Vera et al., 2016)).
The underlying interventionist logic of this line of research is that if
researchers can identify which students are “at-risk” of dropping out of a
course or a program, someone can intervene and prevent the student from
dropping out. Decades of educational reformers have painted the “dropout” as a
scary Other, designed to motivate learners through their fear of becoming part
of this maligned class (Meyerhoff, 2019). This Other has a racialized history.
Educational researchers in the 1960s described minority learners as
“culturally deprived” and argued that these students should be considered
“potential dropouts” (with a too-brief caveat on the negative effect of such
labels) (Hunt, 1966). Although the terms have changed, the underlying ideology
of stigmatizing the “dropout” persists today, manifested in dropout risk
prediction algorithms.

Much like other research efforts for biopolitical educational technologies and
surveillance systems that have found enthusiastic adoption in school systems,
some universities are deploying what they describe as “early warning systems”
that use predictive models to trigger an administrative action when a
potential risk of dropout is detected, ostensibly in support of those students
(Nam and Samson, 2019). This line of research on so-called “Student Early
Warning Systems”181818https://eliterate.us/instructure-dig-and-student-early-
warning-systems/—evokes analogous risk modeling efforts in other domains, such
as early warning systems for missile launches, disaster modeling, and
financial risk models (Ahmad et al., 2013; Lando, 2009; Basch et al., 2000).
However, there are consequences to conceptualizing students as missiles or
natural disasters. Classifying or labeling learners as “at-risk” has
historically had inequitable consequences for marginalized learners (Hunt,
1966) in ways that become self-fulfilling prophecies, particularly when
educators begin viewing students as at-risk. The language implicit in dropout
“risk” prediction evokes the alarmism of the educational reform report, “A
Nation At Risk,” which motivated panic about minoritized learners and led to
decades of testing and policies ostensibly designed to help learners “at
risk.” Further, the underlying data used to train these models is precisely
the set of profoundly unequal educational outcomes—the result of decades of
racist educational practices—described in Section 3.1 (hooks, 1994; Giroux,
2001; Keleher, 2000; Kim et al., 2010). When such systems are used to predict
college enrollment from students’ behaviors in middle school (Pedro et al.,
2013), these systems risk further entrenching legacies of historical inequity
into future learning opportunities.

Beyond all of this, even if the models do successfully identify whether
students may be likely to fail a course or dropout from a program, it is not
clear how they may be used to _prevent_ students from leaving the course. One
approach to this has been to use the output of the risk models to provide
targeted advising to students, as in Purdue University’s Course Signals
program. Other universities have used such models to support what has been
unsettlingly called “intrusive advising” or, providing proactive advising to
students they think need it (Glennen and Baxley, 1985; Varney, 2012). As in
other AI systems in education, some researchers have begun to investigate the
differential performance of these dropout prediction systems for different
populations (Hutt et al., 2019). Left unanswered (or largely unasked) by these
algorithmic fairness approaches is the question of whether these models should
exist at all, or the consequences for marginalized learners of being thus
labeled.

During the COVID-19 pandemic, as schools moved online and many high-stakes
assessments were canceled, several school systems decided to use algorithmic
predictions of end-of-semester grades in lieu of assessments or teacher-
assigned grades. Several high-profile examples of
this191919https://www.nytimes.com/2020/09/08/opinion/international-
baccalaureate-algorithm-grades.html highlight the risks of predictive modeling
approaches to grades. In the UK, algorithmic predictions of students’ A-level
exams were found to be biased against students from schools in lower-income
areas202020https://www.theguardian.com/education/2020/aug/13/almost-40-of-
english-students-have-a-level-results-downgraded. Historical patterns of
disinvestment in education and broader societal inequities were instantiated
in the algorithm, reproducing these historical legacies of oppression.
However, after many students saw their final predicted course grades deviated
from what they expected, massive protests erupted, with rallying cries of
“F*ck the algorithm”212121https://unherd.com/2020/08/how-ofqual-failed-the-
algorithm-test/. The UK Department of Education later canceled the algorithm,
but it provides a cautionary tale for educational technologists who intend to
develop predictive algorithms to model students’ performance or risk (in part
because the models may be re-purposed for other goals, such as grade
assignment). Even if this algorithm had been assessed for group-based fairness
(Kizilcec and Lee, 2020), the underlying premise of the algorithm was to
assign grades on high-stakes assessments on the basis of historical
performance on that assessment, reproducing systemic biases (cf. (Papageorge
et al., 2020)).

Without a critical lens to question the historical legacy and sociopolitical
context of scoring on assessments, these algorithms may perpetuate structural
inequities. At-risk prediction algorithms thus continue to harm marginalized
students (whether the racialized Other of students “at risk” or via
socioeconomic inequities as in the UK A-level algorithm), in ways often left
unquestioned in more optimistic views on educational technology. In this
section, we have discussed three classes of educational AI systems —
biopolitical algorithms; the surveillance infrastructure that draws on and
enables educational AI; and risk prediction. These categories of educational
AI offer instructive sites with which to understand how educational AI may
reproduce structural inequities in education, despite the best intentions of
the researchers and designers, and in ways not accounted for by traditional
group-based approaches to algorithmic fairness.

##  4 Where Might We Go From Here?

The question at this point is not whether inequities exist in educational AI.
The unequal treatment of students in today’s educational settings is well-
established, and we have argued that data-driven educational algorithms that
fail to grapple with those socio-historical structures may reproduce and
amplify existing inequities. Identifying group-based performance differences
for various educational AI systems seems like it may be a good start, but it
remains insufficient if those systems are built on—and further
reproduce—existing systems of oppression, such as, for instance, racist
language ideologies. However, individual researchers may feel unable to make
significant changes to their research agenda (as suggested by related findings
on industry AI research (Holstein et al., 2018; Madaio et al., 2020; Rakova et
al., 2020)), much of which has been built up over years and is hard to pivot.
In the face of this, how might we as a research community re-imagine a better
future, and confront the structural inequities of educational AI?

###  4.1 The limits of representation and participation

One common response to these concerns is to acknowledge the risk of biases
from technologies, and then shift to argue for their resolution through more
diverse voices involved in the design and decision-making process. This push
for representation as a solution is an important step toward addressing issues
of access and opportunity to the knowledge, practices, and spaces that are
shaping educational AI. As one example, there is a push to train more data
scientists from underrepresented backgrounds; in addition to the inherent
value of this, the argument is that if more diverse voices were part of the AI
design process, their perspectives would be able to shape the design of
algorithms222222https://hbr.org/2020/10/to-build-less-biased-ai-hire-a-more-
diverse-team.

While these efforts are laudable, as postmodern feminist and political
philosopher Iris Marion Young argued (Young, 1990), a focus on identity and
inclusion is not enough to address the non-material processes of power and
oppression. Dixon-Román and Parisi (2020) argue that this approach is based on
a politics of inclusion, which makes assumptions that diverse or
representative participation will shift the epistemology and logic of socio-
technical systems. However, Dixon-Román and Parisi (2020) argue that this
approach fails to address the normative disciplinary practices that produce
knowledge and shape educational systems (cf. (Hoffmann, 2020)). Adding
diversity to who is included at the table of decision making is an important
initiative, but as Benjamin (2019) argues “…so much of what is routine,
reasonable, intuitive, and codified reproduces unjust social arrangements,
without ever burning a cross to shine light on the problem” (Benjamin, 2019b).
Dixon-Román and Parisi (2020) further argue that politics of inclusion
fundamentally do not transform the norms and logic that make up the
epistemology of the system, but may in fact maintain and reify the colonialist
logic of hierarchies of difference that was formed in the post-Enlightenment
and inherited by science and technology. In other words, the issue at hand is
not just about representation in design, it is fundamentally about the
epistemological hegemony of educational AI — that is, questions about what it
means to produce knowledge, which kinds of knowledge are valued, and who
determines what knowledge is valued in the design of technology. Thus, the
idea of just adding (for instance) more diverse team members may not
fundamentally shift or change the underlying ways of thinking and knowing and
the structures of power that animate those ideologies. The recent example of
Google firing the co-leads of their Ethical AI research group, Drs. Timnit
Gebru and Margaret Mitchell 232323https://www.fastcompany.com/90608471/timnit-
gebru-google-ai-ethics-equitable-tech-movement, for raising concerns about the
authenticity of their diversity and inclusion efforts, as well as her critical
stance towards their large-scale AI systems, throws into stark relief the
limits of representation and inclusion of diverse voices in the research and
design of AI systems, given existing power structures within technology
companies.

An alternative theory of change looks to research programs that include more
participatory design and community-based co-design
research(ers)242424https://participatoryml.github.io/. This would ideally mean
that research teams involve members of marginalized stakeholder groups as
meaningful, co-equal members of the research team, or as advisors to the
research projects, whose voices are given equal weight in framing the goals
from the start, and who should be able to say that the research should not
continue, if there is no way to make it equitable. This is in contrast to
simply offering tokenized feedback to an already finished project, or a
project that can only be altered at the margins (cf. (Arnstein, 1969; Sloane
et al., 2020)). One can see such participatory approaches used in public
policy (Corbett and Le Dantec, 2018) and community-engaged health research
(Balls-Berry and Acosta-Perez, 2017), as well as in the tradition of
participatory design and co-design in human-computer interaction (Harrington
et al., 2019; Muller, 2007), which is emerging in the learning analytics field
as well (Holstein et al., 2019)252525http://pdlak.utscic.edu.au/call.html.

While this is a positive step, much like calls for more diverse voices on AI
development teams, there is reason to be skeptical of this type of “reformist”
approach to change. Simply “adding users and stirring” (Muller, 2007) will do
little to change the underlying power structures shaping educational AI
systems. Addressing inequity through participatory design will require
reconsideration of what it means for stakeholders to participate in AI
research. Researchers must ensure that their work does not further reproduce
existing power dynamics and hierarchies in the broader society, both within
the research teams and with their relationships to stakeholders (Sloane et
al., 2020; Cooke and Kothari, 2001). Such work will also need to avoid
overburdening marginalized communities, by compensating them adequately for
the full scope of their newly expanded participation and avoiding what some
have referred to as the “epistemic burden” of extractive forms of
participatory design Pierre et al. (2021). Truly empowered participation by
marginalized groups in these research agendas requires the ability to change
ways of knowing and what evidence looks like. It will require a willingness
from the researchers to change the definition of efficacy and success in an
intervention, based on what communities want and need. It will require looking
at different disciplines and paradigms, beyond techno-solutionist paradigms
and beyond well-tread fields like psychometrics and developmental psychology
(and yes, algorithmic “fairness”), and instead towards the voices of community
organizers, social justice scholars, critical theorists and others. This will
require a humility on the part of researchers and funders, a willingness to be
part of a change that may leave them with less control at the end of the
funding and research process than they had at the start.

The field of AI ethics more broadly is grappling with this issue as well.
Despite dozens of values statements for ethical AI produced by large
technology companies and government agencies (Jobin et al., 2019), AI systems
continue to be developed that discriminate and perpetuate injustice. While
there may be legitimate organizational reasons why AI practitioners have been
unable to put these principles into practice (e.g., (Holstein et al., 2018;
Stark and Hoffmann, 2019; Madaio et al., 2020; Rakova et al., 2020)), others
have been skeptical of these companies’ intentions. Some critics describe
these AI ethical statements as “fairwashing” or “ethics-washing”— that is,
developing principles for ethical AI, while doing little to change the nature
of such technology, or the organizational processes that led to their design
(Bietti, 2020), acting as a “smokescreen” for business as usual (Sloane,
2019). Thus, while it may be worthwhile to begin with efforts to support more
diverse perspectives on educational AI teams and more participation from
marginalized communities in those systems’ design, this will not be enough to
transform fundamentally unjust systems.

###  4.2 Equitable futures for AI in education

As we close, we ask what an alternative vision for a more equitable future for
educational AI might entail. First, and most critically, we argue that
educational AI researchers and designers should adopt a design justice
approach to educational AI research. As proposed by Sasha Costanza-Chock,
design justice is a “framework for analysis of how design distributes benefits
and burdens between various groups of people” (Costanza-Chock, 2020). This
involves interrogating the values encoded into designed systems, meaningfully
involving members of marginalized and impacted communities in the design of
technology, and, more generally, critically interrogating the narratives,
sites, and pedagogies around design—here, the design of educational AI
systems. We have begun such efforts by tracing socio-historical legacies of
educational inequity through several categories of educational AI systems that
may perpetuate unjust ideologies and structures of oppression. To design more
equitable educational AI, though, will require the learning science and
technology community to learn, adopt, and promote theories and methods fields
outside of what is considered to be a core research area for AI, approaches
which may draw on design justice, critical theory, participatory design, and
other areas.

But changing the direction and goals of a research community means not only
that individuals should change their research and design practices, but that
the field as a whole should interrogate and confront the ways in which
systemic educational inequities may be reproduced in the design, research, and
policy of educational AI. What does it mean to be a justice-oriented learning
science researcher? How might the existing incentives of the field be changed
to encourage researchers to interrogate ethical issues and center justice-
oriented decisions in their research practice? These incentives may include
the grants and funding that learning scientists pursue, the collaborators they
choose, the role of stakeholders in their research (including grappling with
power hierarchies between and within communities of stakeholders), and
researchers’ ongoing engagement with the impact and potential inequities of
their research after dissemination and deployment. Ideally, such an approach
would entail a longer-term view of how research and development of educational
AI (and educational technology more broadly) impacts learners’ educational
opportunities over the current, more incremental approaches. To what extent
might values of justice and equity be adopted in conference and journal calls,
review processes, and paper awards? In tenure and promotion reviews at
universities? How might we avoid the co-optation of calls for equity that
become neutered and narrowly scoped into neat technical fixes and ethics-
washing?

We, along with others in AI more broadly (e.g., Benjamin (2019a)), argue that
the field of learning science and educational AI needs new, more radical
visions for equitable learning futures. Visions for the kinds of technologies
that can be built are shaped by what has been called “socio-technical
imaginaries”, or:

> The collectively held, institutionally stabilized, and publicly performed
> visions of desirable futures, animated by shared understandings of forms of
> social life and social order attainable through, and supportive of, advances
> in science and technology (Jasanoff and Kim, 2015).

The socio-technical imaginaries of our current field of educational AI are
shaped and circumscribed by the historical legacy of the matrix of domination
in education and technology. But they can be remade. To do this, we are
inspired by Keyes et al.’s call for counterpower, or what they call
emancipatory autonomy in human-computer interaction, or, more simply,
anarchist HCI (Keyes et al., 2019). For them, counterpower involves fostering
community-appropriate and community-determined research and design, both
between researchers and stakeholders as well as within the research community
itself (here, learning science and educational AI), akin to what Asad et al.
have referred to as “academic accomplices”, or scholars whose research is
designed to support (rather than displace) the already ongoing justice work on
communities (Asad et al., 2019). This move towards counterpower would involve
giving everyone, not simply a privileged few, the means to shape the forms of
socio-technical educational systems. What might such counterpower look like
for learning science research, and how might it lead to new socio-technical
imaginaries for educational AI?

For some research directions, it may be the case that the research should
simply cease — where “the implication is not to design” (Baumer and Silberman,
2011; Barocas et al., 2020). Os Keyes poses a similar challenge by arguing
that data science’s approach to quantification and classification is
intrinsically at odds with — and a danger to — queer lives (Keyes, 2019b).
Predictive policing presents another such case, where generations of racist
policing practices means that algorithms trained on these data will inevitably
reproduce and exacerbate these practices (Dixon-Roman et al., 2019; Richardson
et al., 2019). In such cases, injustices are inextricably embedded in the
proposed technical solutions. For these types of tasks or methods, no amount
of group fairness measurement, or UX optimization of a workflow will produce
anything other than a reproduction and entrenchment of existing structures of
oppression. Thus, the only recourse may be for people to organize to shut it
down. In AI more broadly, public acts of resistance or refusal are becoming
more common. For instance, communities have organized against computer vision
used in public housing projects (Gilman, 2019), and the Stop LAPD Spying
Coalition has successfully organized to ban predictive policing in Los
Angeles, among other cities262626https://capp-pgh.com/. In education, one
might look to the recent protests in the UK to “F*ck the algorithm” that
predicted their A-level scores272727https://unherd.com/2020/08/how-ofqual-
failed-the-algorithm-test/ or organized resistance against automated
proctoring systems (Cahn et al.,
2020)282828https://www.eff.org/deeplinks/2020/09/students-are-pushing-back-
against-proctoring-surveillance-apps, among
others292929https://www.donnalanclos.com/listening-to-refusal-opening-keynote-
for-aptconf-2019/. However, these examples are acts of resistance and refusal
to systems that are already designed, deployed, and causing harm in the world.

We thus call for a wider range of design methods and theories to envision more
radical, equitable futures for the learning sciences and AI in education. The
design of our modes of learning is too important to be left in the hands of a
privileged few. Current rhetoric from AI corporations and researchers (Moreau
et al., 2019) is full of calls to “democratize” AI, which mostly seems to
refer to open-source tools for developing machine learning models (e.g.,
TensorFlow). In practice, while platforms like TensorFlow may be freely
available, simply giving more (often already privileged) people access to them
will do little to change the fundamentally inequitable systems and power
structures underlying the social contexts which they are developed and used.
Instead, a more radical vision for democratizing data-driven learning
technologies might reflect Paulo Freire’s vision of a _liberatory_ pedagogy
that allows learners to propose and engage with the problems that are
essential for their lives (Freire, 1970). There is a rich lineage of
resistance to centralized hierarchies in education, including the critical
theorist Ivan Illich, who famously called for “de-schooling society” (Illich,
1973) among others (Giroux, 2001; Freire, 1970; hooks, 1994; Meyerhoff, 2019).
A full discussion of how the vision of Freire and Illich might be enacted in
data-driven educational technology is beyond the scope of this article, but we
are encouraged by recent work on the implications of a Freireian pedagogy for
learning analytics (Broughan and Prinsloo, 2020).

The design approaches and theories that fit neatly into the existing body of
learning science and technology research have resulted in reproductions of
existing, inequitable brick-and-mortar education systems. We have built
algorithms that reflect back the inequities that are already in place. At
best, educational AI researchers have conducted group-fairness comparisons for
learners from different populations (e.g., (Kizilcec and Lee, 2020; Gardner et
al., 2019)). While such efforts are promising steps forward to mitigating
fairness harms of existing systems, this retrenchment of existing algorithmic
fairness paradigms will not lead to the creation of new socio-technical
imaginaries, or the design of more liberatory forms of education. As a field,
we might instead look to methods from feminist speculative design (Martins,
2014) and critical design (Bardzell et al., 2012; Bardzell and Bardzell,
2013), which endeavor to provoke and problematize (Forlano and Mathew, 2014),
and to have bold visions for designing more liberatory futures.

One source for these new socio-technical imaginaries may come from queer
theory. Scholars in queer theory have used the term “queer” as a verb, to
describe a resistance to and deconstruction of binary and hierarchical
categories and structures of oppression (Butler, 2011; Ahmed, 2006; Muñoz et
al., 2019). Although it originated in resisting rigid categories of gender and
sexuality, it has been used to problematize hierarchical relations of power
more broadly, including in the field of HCI and technology design (Light,
2011) and data science (Keyes, 2019b). Inspired by this, we ask what it would
take to queer learning technologies—to offer resistance to the dominant modes
of thinking that reinforce systems of oppression in the learning sciences and
educational AI. This might involve using critical design methods to provoke
and problematize foundational assumptions in learning technologies, including
what should be learned and what it means to have learned it – to challenge
dominant ideologies of what a “good” learner is and resist the technologies
that treat deviation from the norm as suspicious.

Additionally, Dixon-Román and others have argued that we might look to the
field of Afrofuturism as another source for methods and theories to envision
new socio-technical imaginaries and develop new visions for more liberatory
futures (de Freitas and Dixon-Román, 2017; Benjamin, 2019a). That is to say,
if particular sociopolitical processes of colonial and post-colonial violence
never occurred, how might socio-technical systems be designed differently,
have different purposes, and enable alternative forms of reasoning other-wise?
Afrofuturism entails a critical examination of the historical conditions that
led to current societal inequities, while opening up a space for re-imagining
alternative futures by asking the ’what if’ questions that center an
affirmative perspective of people of color (Dixon-Román et al., 2019; Gaskins,
2019). Winchester III (2018) has described how Afrofuturism might “place the
often disenfranchised black voice central in the design narrative” and “plug
the imagination gap” in technology design. For a vision of how Afrofuturism
might be used to counteract the “racializing forces” of educational AI, see
Dixon-Román et al. (2019).

All of this may require rethinking the standard AI research and design
lifecycle to prioritize equity and justice and to involve (and empower)
stakeholders from marginalized communities throughout this lifecycle, in ways
that avoid reproducing existing power dynamics in their participatory design
approaches. This may entail a radical transformation of the “institutionally
stabilized” structures and incentives of the current funding and research
landscape to create welcoming spaces for diverse voices to shape the socio-
technical imaginaries of educational AI and ed tech more broadly (Tuck and
Yang, 2018; Jasanoff and Kim, 2015). It may require teaching learning science
and educational AI students about the history and legacies of educational
injustice, critical theory, and broader societal systems of oppression.

###  4.3 Conclusion

Educational policies and practices present visions of a particular type of
learner, and attempt to prepare that learner for life in society—educational
AI systems then instantiate these visions and values into algorithmic systems
that may further entrench them. Whose visions are these? What type of learner
is imagined, and what type of society are they being educated for? These are
questions with critical consequences in a settler colonialist, white
supremacist, patriarchal, cis heteronormative, ableist society. Paulo Freire
and bell hooks, among other theorists and critics, have called for a more
radical, liberatory form of education, where, rather than simply reproducing
inequitable societal structures, the role of education is instead a
provocation to rethink what is possible — to remake the world in a more
radical, more just, and more equitable way.

This article interrogates issues of structural inequity in educational AI
technologies through critical examination of several classes of educational
AI. The problems are complex and interwoven and resist straightforward
analyses and answers—including quick technical solutions and neat group-level
evaluations of “AI fairness”. What is needed are approaches to transforming
the epistemology of educational AI, working towards justice and equity—both in
educational AI technologies and in the socio-technical and socio-political
milieu in which they are designed. We do not offer solutions to the core
challenges we have presented. To offer neat solutions would be to undermine
the argument we have been making.

What this particular moment, the crisis of the COVID-19 pandemic, reveals is
the potential for doing work outside the norms. Right now, as our entire
education system is in flux, we are on the precipice of rapid and fundamental
change. We can use this time to radically reimagine what we want for our
education systems. Funding organizations and technology corporations are often
the ones with the power and privilege to break this cycle. And yet as a
community of researchers and practitioners, there is much we can collectively
do to transform the trajectory of the field. There is an enormous rethinking
about the role of technology in education that will occur worldwide in the
coming years. We must take this as an opportunity to reshape the priorities of
the educational research we do and the research we support, towards addressing
structural inequities in educational AI.

## References

  * Agarwal et al. [2018] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach.  A reductions approach to fair classification.  In _International Conference on Machine Learning_ , pages 60–69, 2018\.
  * Aguliera and Nightengale-Lee [2020] Earl Aguliera and Bianca Nightengale-Lee.  Emergency remote teaching across urban and rural contexts: perspectives on educational equity.  _Information and Learning Sciences_ , 2020.
  * Ahmad et al. [2013] Naveed Ahmad, Mureed Hussain, Naveed Riaz, Fazli Subhani, Sajjad Haider, Khurram S Alamgir, and Fahad Shinwari.  Flood prediction and disaster risk analysis using gis based wireless sensor networks, a review.  _Journal of Basic and Applied Scientific Research_ , 3(8):632–643, 2013.
  * Ahmed [2006] Sara Ahmed.  _Queer phenomenology: Orientations, objects, others_.  Duke University Press, 2006.
  * Ahuja et al. [2019] Karan Ahuja, Dohyun Kim, Franceska Xhakaj, Virag Varga, Anne Xie, Stanley Zhang, Jay Eric Townsend, Chris Harrison, Amy Ogan, and Yuvraj Agarwal.  Edusense: Practical classroom sensing at scale.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 3(3):1–26, 2019.
  * Anastasi [1961] Anne Anastasi.  Psychological tests: Uses and abuses.  _Teachers College Record_ , 1961.
  * Angwin et al. [2016] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner.  Machine bias.  _ProPublica, May_ , 23, 2016.
  * Arnstein [1969] Sherry R Arnstein.  A ladder of citizen participation.  _Journal of the American Institute of planners_ , 35(4):216–224, 1969.
  * Asad et al. [2019] Mariam Asad, Lynn Dombrowski, Sasha Costanza-Chock, Sheena Erete, and Christina Harrington.  Academic accomplices: Practical strategies for research justice.  In _Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion_ , pages 353–356, 2019.
  * Athanasiadis et al. [2017] Christos Athanasiadis, Enrique Hortal, Dimitrios Koutsoukos, Carmen Zarco Lens, and Stylianos Asteriadis.  Personalized, affect and performance-driven computer-based learning.  In _CSEDU (1)_ , pages 132–139, 2017.
  * Balls-Berry and Acosta-Perez [2017] Joyce E Balls-Berry and Edna Acosta-Perez.  The use of community engaged research principles to improve health: community academic partnerships for research.  _Puerto Rico health sciences journal_ , 36(2):84, 2017.
  * Bardzell and Bardzell [2013] Jeffrey Bardzell and Shaowen Bardzell.  What is" critical" about critical design?  In _Proceedings of the SIGCHI conference on human factors in computing systems_ , pages 3297–3306, 2013.
  * Bardzell et al. [2012] Shaowen Bardzell, Jeffrey Bardzell, Jodi Forlizzi, John Zimmerman, and John Antanitis.  Critical design and critical theory: the challenge of designing for provocation.  In _Proceedings of the Designing Interactive Systems Conference_ , pages 288–297, 2012.
  * Barocas and Selbst [2016] Solon Barocas and Andrew D Selbst.  Big data’s disparate impact.  _Calif. L. Rev._ , 104:671, 2016.
  * Barocas et al. [2020] Solon Barocas, Asia J Biega, Benjamin Fish, Jędrzej Niklas, and Luke Stark.  When not to design, build, or deploy.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 695–695, 2020.
  * Basch et al. [2000] Catherine A Basch, Belva J Bruesewitz, Kevin Siegel, and Patrick Faith.  Financial risk prediction systems and methods therefor, September 12 2000\.  US Patent 6,119,103.
  * Baumer and Silberman [2011] Eric PS Baumer and M Six Silberman.  When the implication is not to design (technology).  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pages 2271–2274, 2011.
  * Beckett [2019] Lois Beckett.  Under digital surveillance: how american schools spy on millions of kids.  https://bit.ly/3c6VTMg, October 2019.  (Accessed on 04/22/2020).
  * Benjamin [2019a] Ruha Benjamin.  _Captivating technology: Race, carceral technoscience, and liberatory imagination in everyday life_.  Duke University Press, 2019a.
  * Benjamin [2019b] Ruha Benjamin.  _Race After Technology: Abolitionist Tools for the New Jim Code_.  John Wiley & Sons, 2019b.
  * Bennett and Keyes [2019] Cynthia L. Bennett and Os Keyes.  What is the Point of Fairness? Disability, AI, and The Complexity of Justice.  In _Proceedings of the ASSETS Workshop on AI Fairness for People with Disabilities_ , Pittsburgh, PA, 2019.
  * Bietti [2020] Elettra Bietti.  From ethics washing to ethics bashing: a view on tech ethics from within moral philosophy.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 210–219, 2020.
  * Blodgett et al. [2020] Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach.  Language (Technology) is Power: A Critical Survey of “Bias” in NLP.  In _Proceedings of the Association for Computational Linguistics (ACL)_ , page 5454–5476, 2020.
  * Bolukbasi et al. [2016] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems 29_ , pages 4349–4357. Curran Associates, Inc., 2016.
  * Boser [2012] Ulrich Boser.  Race to the top: What have we learned from the states so far? a state-by-state evaluation of race to the top performance.  _Center for American Progress_ , 2012.
  * Bottiani et al. [2017] Jessika H Bottiani, Catherine P Bradshaw, and Tamar Mendelson.  A multilevel examination of racial disparities in high school discipline: Black and white adolescents’ perceived equity, school belonging, and adjustment problems.  _Journal of Educational Psychology_ , 109(4):532, 2017.
  * Bourdieu [1977] Pierre Bourdieu.  _Outline of a Theory of Practice_ , volume 16.  Cambridge university press, 1977.
  * boyd and Crawford [2012] danah boyd and Kate Crawford.  Critical Questions for Big Data.  _Information, Communication & Society_, 15(5):662–679, 2012.
  * Broughan and Prinsloo [2020] Christine Broughan and Paul Prinsloo.  (re) centring students in learning analytics: in conversation with paulo freire.  _Assessment & Evaluation in Higher Education_, 45(4):617–628, 2020.
  * Brynjarsdottir et al. [2012] Hronn Brynjarsdottir, Maria Håkansson, James Pierce, Eric Baumer, Carl DiSalvo, and Phoebe Sengers.  Sustainably unpersuaded: how persuasion narrows our vision of sustainability.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pages 947–956, 2012.
  * Buolamwini and Gebru [2018] Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on Fairness, Accountability and Transparency_ , pages 77–91, 2018.
  * Butler [2011] Judith Butler.  _Gender trouble: Feminism and the subversion of identity_.  routledge, 2011.
  * Cahn et al. [2020] Albert Fox Cahn, Caroline Magee, Eleni Manis, and Naz Akyol.  Snooping where we sleep: The invasiveness and bias of remote proctoring systems.  _Surveillance Technology Oversight Project_ , 2020.
  * Carbonell [1970] Jaime R Carbonell.  Ai in cai: An artificial-intelligence approach to computer-assisted instruction.  _IEEE transactions on man-machine systems_ , 11(4):190–202, 1970.
  * Chang et al. [2018] Cheng Chang, Cheng Zhang, Lei Chen, and Yang Liu.  An ensemble model using face and body tracking for engagement detection.  In _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , pages 616–622, 2018.
  * Chouldechova [2017] Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ , 5(2):153–163, 2017.
  * Cifor et al. [2019] Marika Cifor, Patricia Garcia, TL Cowan, Jasmine Rault, Tonia Sutherland, Anita Say Chan, Jennifer Rode, Anna Lauren Hoffmann, Niloufar Salehi, and Lisa Nakamura.  Feminist data manifest-no, 2019.
  * Cleary [1968] T Anne Cleary.  Test bias: Prediction of grades of negro and white students in integrated colleges.  _Journal of Educational Measurement_ , 5(2):115–124, 1968.
  * Collins [1990] Patricia Hill Collins.  Black feminist thought in the matrix of domination.  _Black feminist thought: Knowledge, consciousness, and the politics of empowerment_ , 138:221–238, 1990.
  * Cooke and Kothari [2001] Bill Cooke and Uma Kothari.  _Participation: The new tyranny?_ Zed books, 2001.
  * Corbett and Le Dantec [2018] Eric Corbett and Christopher A Le Dantec.  The problem of community engagement: Disentangling the practices of municipal government.  In _Proceedings of the 2018 CHI conference on human factors in computing systems_ , pages 1–13, 2018.
  * Corbett-Davies and Goel [2018] Sam Corbett-Davies and Sharad Goel.  The measure and mismeasure of fairness: A critical review of fair machine learning.  _Synthesis of tutorial presented at ICML 2018._ , 2018.
  * Costanza-Chock [2018] Sasha Costanza-Chock.  Design justice: Towards an intersectional feminist framework for design theory and practice.  In _Proceedings of the Design Research Society_ , 2018.
  * Costanza-Chock [2020] Sasha Costanza-Chock.  _Design justice: Community-led practices to build the worlds we need_.  MIT Press, 2020.
  * Crawford [2017] Kate Crawford.  The trouble with bias, 2017.  _URL http://blog. revolutionanalytics. com/2017/12/the-trouble-with-bias-by-kate-crawford. html. Invited Talk by Kate Crawford at NIPS_ , 2017.
  * Crenshaw [1989] Kimberle Crenshaw.  Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrmination Doctrine, Feminist Theory and Antiracist Politics.  _University of Chicago Legal Forum_ , 1989.
  * Crenshaw [1990] Kimberle Crenshaw.  Mapping the margins: Intersectionality, identity politics, and violence against women of color.  _Stan. L. Rev._ , 43:1241, 1990.
  * Daggett [2008] Lynn M Daggett.  Ferpa in the twenty-first century: Failure to effectively regulate privacy for all students.  _Cath. UL Rev._ , 58:59, 2008.
  * Davidson et al. [2019] Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber.  Racial bias in hate speech and abusive language detection datasets.  In _Proceedings of the Workshop on Abusive Language Online_ , 2019\.
  * Davis [1995] Lennard J Davis.  _Enforcing normalcy: Disability, deafness, and the body_.  Verso, 1995.
  * de Freitas and Dixon-Román [2017] Elizabeth de Freitas and Ezekiel Dixon-Román.  The computational turn in education research: Critical and creative perspectives on the digital data deluge, 2017.
  * Delgado and Stefancic [2017] Richard Delgado and Jean Stefancic.  _Critical Race Theory: An Introduction_.  NYU Press, 3 edition, 2017.
  * Dixon et al. [2018] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Measuring and mitigating unintended bias in text classification.  In _Conference on Artificial Intelligence, Ethics, and Society_ , New Orleans, LA, 2018.
  * Dixon-Román and Parisi [2020] Ezekiel Dixon-Román and Luciana Parisi.  Data capitalism, sociogenic prediction and recursive indeterminacies.  In _Public Plurality in an Era of Data Determinacy: Data Publics_ , 2020.
  * Dixon-Román et al. [2019] Ezekiel Dixon-Román, T Philip Nichols, and Ama Nyame-Mensah.  The racializing forces of/in ai educational technologies.  _Learning, Media and Technology_ , pages 1–15, 2019.
  * Dixon-Roman et al. [2019] Ezekiel Dixon-Roman, Ama Nyame-Mensah, and Allison R Russell.  Algorithmic legal reasoning as racialising assemblages.  _Computational Culture: A Journal of Software Studies_ , 7:1–41, 2019.  URL http://computationalculture.net/algorithmic-legal-reasoning-as-racializing-assemblages/.
  * Dixon-Román [2017] Ezekiel J Dixon-Román.  _Inheriting possibility: Social reproduction and quantification in education_.  U of Minnesota Press, 2017.
  * D’Mello and Graesser [2010] Sidney K D’Mello and Arthur Graesser.  Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features.  _User Modeling and User-Adapted Interaction_ , 20(2):147–187, 2010.
  * D’Mello et al. [2015] Sidney K D’Mello, Andrew M Olney, Nathan Blanchard, Borhan Samei, Xiaoyi Sun, Brooke Ward, and Sean Kelly.  Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms.  In _Proceedings of the 2015 ACM on international conference on multimodal interaction_ , pages 557–566, 2015.
  * Donlon [1981] Thomas F Donlon.  The s.a.t. in a diverse society: Fairness and sensitivity.  _College Board Review_ , 1981.
  * Dorn et al. [2020] Emma Dorn, Bryan Hancock, Jimmy Sarakatsannis, and Ellen Viruleg.  Covid-19 and student learning in the united states: The hurt could last a lifetime.  _McKinsey & Company_, 2020.
  * Doroudi and Brunskill [2019] Shayan Doroudi and Emma Brunskill.  Fairer but not fair enough on the equitability of knowledge tracing.  In _Proceedings of the 9th International Conference on Learning Analytics & Knowledge_, pages 335–339, 2019.
  * Dwork et al. [2012] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In _Proceedings of Innovations of Theoretical Computer Science_ , pages 214–226, 2012.
  * Eubanks [2018] Virginia Eubanks.  _Automating inequality: How high-tech tools profile, police, and punish the poor_.  St. Martin’s Press, 2018.
  * Feagin and Ducey [2000] Joe R. Feagin and Kimberley Ducey.  _Racist America: Roots, Current Realities, and Future Reparations_.  Routledge, 2000.
  * Fessler [2017] Leah Fessler.  We tested bots like Siri and Alexa to see who would stand up to sexual harassment, 2017.  Quartz.
  * for Education Statistics [2020] National Center for Education Statistics.  Edfacts data files: U.s. department of education.  https://bit.ly/3d4yRWE, 2020.  (Accessed on 04/17/2020).
  * Forlano and Mathew [2014] Laura Forlano and Anijo Mathew.  From design fiction to design friction: Speculative and participatory design of values-embedded urban technology.  _Journal of Urban Technology_ , 21(4):7–24, 2014\.
  * Foucault [1984] Michel Foucault.  _Right of death and power over life_.  na, 1984.
  * Fox et al. [2012] Christine Fox, John Waters, Geoff Fletcher, and Douglas Levin.  The broadband imperative: Recommendations to address k-12 education infrastructure needs.  _State Educational Technology Directors Association_ , 2012.
  * Freire [1970] Paulo Freire.  _Pedagogy of the oppressed_.  Bloomsbury publishing USA, 1970.
  * Friedman and Nissenbaum [1996] Batya Friedman and Helen Nissenbaum.  Bias in computer systems.  _ACM Transactions on Information Systems (TOIS)_ , 14(3):330–347, 1996.
  * García [2020] Emma García.  Schools are still segregated, and black children are paying a price.  _Economic Policy Institute_ , 2020.
  * Gardner et al. [2019] Josh Gardner, Christopher Brooks, and Ryan Baker.  Evaluating the fairness of predictive student models through slicing analysis.  In _Proceedings of the 9th International Conference on Learning Analytics & Knowledge_, pages 225–234, 2019.
  * Gaskins [2019] Nettrice Gaskins.  Techno-vernacular creativity and innovation across the african diaspora and global south.  _Captivating technology: Race, carceral technoscience, and liberatory imagination in everyday life_ , pages 252–274, 2019.
  * Gillen et al. [2018] Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth.  Online learning with an unknown fairness metric.  In _Advances in Neural Information Processing Systems_ , pages 2600–2609, 2018.
  * Gilman [2019] Michele Gilman.  Voices of the poor must be heard in the data privacy debate - jurist \- commentary - legal news & commentary, 2019.  (Accessed on 05/01/2020).
  * Giroux [2001] Henry A Giroux.  _Theory and resistance in education: Towards a pedagogy for the opposition_.  Greenwood Publishing Group, 2001.
  * Glennen and Baxley [1985] Robert E Glennen and Dan M Baxley.  Reduction of attrition through intrusive advising.  _NASPA journal_ , 22(3):10–14, 1985.
  * Green [2018] Ben Green.  Data Science as Political Action: Grounding Data Science in a Politics of Justice.  _arXiv preprint arXiv:1811.03435_ , 2018.
  * Gregory and Fergus [2017] Anne Gregory and Edward Fergus.  Social and emotional learning and equity in school discipline.  _The Future of Children_ , pages 117–136, 2017.
  * Hall et al. [2020] Jacob Hall, Cesia Roman, Christian Jovel-Arias, and Cayleen Young.  Pre-service teachers examine digital equity amidst schools’ covid-19 responses.  _Journal of Technology and Teacher Education_ , 28(2):435–442, 2020.
  * Hanna et al. [2020] Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud.  Towards a Critical Race Methodology in Algorithmic Fairness.  In _Conference on Fairness, Accountability and Transparency_ , 2020\.
  * Hansen and Reich [2015] John D Hansen and Justin Reich.  Democratizing education? examining access and usage patterns in massive open online courses.  _Science_ , 350(6265):1245–1248, 2015.
  * Harrington et al. [2019] Christina Harrington, Sheena Erete, and Anne Marie Piper.  Deconstructing community-based collaborative design: Towards more equitable participatory design engagements.  _Proceedings of the ACM on Human-Computer Interaction_ , 3(CSCW):1–25, 2019.
  * Heitzeg [2009] Nancy A Heitzeg.  Education or incarceration: Zero tolerance policies and the school to prison pipeline.  In _Forum on public policy online_ , volume 2009. ERIC, 2009.
  * Henderson et al. [2019] Nathan L Henderson, Jonathan P Rowe, Bradford W Mott, Keith Brawner, Ryan Baker, and James C Lester.  4d affect detection: Improving frustration detection in game-based learning with posture-based temporal data fusion.  In _International Conference on Artificial Intelligence in Education_ , pages 144–156. Springer, 2019.
  * Hoffman [2014] Stephen Hoffman.  Zero benefit: Estimating the effect of zero tolerance discipline polices on racial disparities in school discipline.  _Educational Policy_ , 28(1):69–95, 2014.
  * Hoffmann [2019] Anna Lauren Hoffmann.  Where fairness fails: Data, algorithms, and the limits of antidiscrimination discourse.  _Information, Communication, & Society_, 22(7):900–915, 2019.
  * Hoffmann [2020] Anna Lauren Hoffmann.  Terms of Inclusion: Data, Discourse, Violence.  _New Media & Society_, 2020.
  * Holmes et al. [2021] Wayne Holmes, Kaska Porayska-Pomsta, Ken Holstein, Emma Sutherland, Toby Baker, Simon Buckingham Shum, Olga C Santos, Mercedes T Rodrigo, Mutlu Cukurova, Ig Ibert Bittencourt, et al.  Ethics of ai in education: Towards a community-wide framework.  _International Journal of Artificial Intelligence in Education_ , pages 1–23, 2021.
  * Holstein et al. [2018] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, and Hanna Wallach.  Improving fairness in machine learning systems: What do industry practitioners need?  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , 2018.
  * Holstein et al. [2019] Kenneth Holstein, Bruce M McLaren, and Vincent Aleven.  Designing for complementarity: Teacher and student needs for orchestration support in ai-enhanced classrooms.  In _International Conference on Artificial Intelligence in Education (AIED) 2019 Proceedings_ , pages 157–171. Springer, 2019.
  * hooks [1994] bell hooks.  _Teaching to transgress_.  Routledge, 1994.
  * Hunt [1966] David E Hunt.  Chapter vi: Adolescence: Cultural deprivation, poverty, and the dropout.  _Review of Educational Research_ , 36(4):463–473, 1966.
  * Hutchinson and Mitchell [2019] Ben Hutchinson and Margaret Mitchell.  50 years of test (un) fairness: Lessons for machine learning.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , pages 49–58, 2019.
  * Hutt et al. [2019] Stephen Hutt, Margo Gardner, Angela L Duckworth, and Sidney K D’Mello.  Evaluating fairness and generalizability in models predicting on-time graduation from college applications.  _International Educational Data Mining Society_ , 2019.
  * Illich [1973] Ivan Illich.  _Deschooling society_.  Penguin Group Limited, 1973.
  * Jasanoff and Kim [2015] Sheila Jasanoff and Sang-Hyun Kim.  _Dreamscapes of modernity: Sociotechnical imaginaries and the fabrication of power_.  University of Chicago Press, 2015.
  * Jobin et al. [2019] Anna Jobin, Marcello Ienca, and Effy Vayena.  The global landscape of ai ethics guidelines.  _Nature Machine Intelligence_ , 1:389–399, 2019.
  * Kalluri [2020] Pratyusha Kalluri.  Don’t ask if artificial intelligence is good or fair, ask how it shifts power.  _Nature_ , 583(7815):169–169, 2020.
  * Karabel [2006] Jerome Karabel.  _The Chosen: The Hidden History of Admission and Exclusion at Harvard, Yale, and Princeton_.  Mariner Books, 2006.
  * Keleher [2000] Terry Keleher.  Racial disparities related to school zero tolerance policies: Testimony to the us commission on civil rights.  2000\.
  * Keyes [2019a] Os Keyes.  The Bones We Leave Behind.  _Real Life Magazine_ , 2019a.
  * Keyes [2019b] Os Keyes.  Counting the countless: Why data science is a profound threat for queer people.  _Real Life_ , 2, 2019b.
  * Keyes et al. [2019] Os Keyes, Josephine Hoy, and Margaret Drouhard.  Human-computer insurrection: Notes on an anarchist hci.  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , pages 1–13, 2019.
  * Kim et al. [2010] Catherine Y Kim, Daniel J Losen, and Damon T Hewitt.  _The school-to-prison pipeline: Structuring legal reform_.  NYU Press, 2010.
  * Kim et al. [2018] Michael Kim, Omer Reingold, and Guy Rothblum.  Fairness through computationally-bounded awareness.  In _Advances in Neural Information Processing Systems_ , pages 4842–4852, 2018.
  * Kizilcec and Lee [2020] René F Kizilcec and Hansol Lee.  Algorithmic fairness in education.  _arXiv preprint arXiv:2007.05443_ , 2020.
  * Klein [2007] Naomi Klein.  The shock doctrine: The rise of disaster capitalism. new york: Allen lane, 2007.
  * Kleinberg et al. [2016] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan.  Inherent Trade-Offs in the Fair Determination of Risk Scores.  _arXiv preprint arXiv:1609.05807_ , 2016.
  * Kosciw et al. [2018] Joseph G Kosciw, Emily A Greytak, Adrian D Zongrone, Caitlin M Clark, and Nhan L Truong.  _The 2017 National School Climate Survey: The Experiences of Lesbian, Gay, Bisexual, Transgender, and Queer Youth in Our Nation’s Schools._ ERIC, 2018.
  * Kotsiantis et al. [2007] Sotiris B Kotsiantis, I Zaharakis, and P Pintelas.  Supervised machine learning: A review of classification techniques.  _Emerging artificial intelligence applications in computer engineering_ , 160(1):3–24, 2007.
  * Krafft et al. [2021] PM Krafft, Meg Young, Michael Katell, Jennifer E Lee, Shankar Narayan, Micah Epstein, Dharma Dailey, Bernease Herman, Aaron Tam, Vivian Guetler, et al.  An action-oriented ai policy toolkit for technology audits by community advocates and activists.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 772–781, 2021.
  * Kulik and Fletcher [2016] James A Kulik and JD Fletcher.  Effectiveness of intelligent tutoring systems: a meta-analytic review.  _Review of educational research_ , 86(1):42–78, 2016.
  * Kumashiro [2000] Kevin K Kumashiro.  Toward a theory of anti-oppressive education.  _Review of Educational research_ , 70(1):25–53, 2000.
  * Lando [2009] David Lando.  Credit risk modeling.  In _Handbook of Financial Time Series_ , pages 787–798. Springer, 2009.
  * Lee and Kizilcec [2020] Hansol Lee and René F Kizilcec.  Evaluation of fairness trade-offs in predicting student success.  _arXiv preprint arXiv:2007.00088_ , 2020.
  * Lemann [2000] Nicolas Lemann.  _The Big Test: The Secret History of the American Meritocracy_.  Farrar, Straus and Giroux, 2000.
  * Li et al. [2019] Manling Li, Lingyu Zhang, Heng Ji, and Richard J Radke.  Keep meeting summaries on topic: Abstractive multi-modal meeting summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2190–2196, 2019.
  * Liang et al. [2016] Jiajun Liang, Chao Li, and Li Zheng.  Machine learning application in moocs: Dropout prediction.  In _2016 11th International Conference on Computer Science & Education (ICCSE)_, pages 52–57. IEEE, 2016.
  * Light [2011] Ann Light.  Hci as heterodoxy: Technologies of identity and the queering of interaction with computers.  _Interacting with Computers_ , 23(5):430–438, 2011.
  * Linn [1973] Robert L Linn.  Fair test use in selection.  _Review of Educational Research_ , 43(2):139–161, 1973.
  * Loukina et al. [2019] Anastassia Loukina, Nitin Madnani, and Klaus Zechner.  The many dimensions of algorithmic fairness in educational applications.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 1–10, 2019.
  * Lykourentzou et al. [2009] Ioanna Lykourentzou, Ioannis Giannoukos, Vassilis Nikolopoulos, George Mpardis, and Vassili Loumos.  Dropout prediction in e-learning courses through the combination of machine learning techniques.  _Computers & Education_, 53(3):950–965, 2009\.
  * Madaio et al. [2020] Michael A Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach.  Co-designing checklists to understand organizational challenges and opportunities around fairness in ai.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , pages 1–14, 2020.
  * Madnani et al. [2017] Nitin Madnani, Anastassia Loukina, Alina von Davier, Jill Burstein, and Aoife Cahill.  Building better open-source tools to support fairness in automated scoring.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pages 41–52, 2017.
  * Mandalapu and Gong [2018] Varun Mandalapu and Jiaqi Gong.  Towards better affect detectors: Detecting changes rather than states.  In _International Conference on Artificial Intelligence in Education_ , pages 199–203. Springer, 2018.
  * Márquez-Vera et al. [2016] Carlos Márquez-Vera, Alberto Cano, Cristobal Romero, Amin Yousef Mohammad Noaman, Habib Mousa Fardoun, and Sebastian Ventura.  Early dropout prediction using data mining: a case study with high school students.  _Expert Systems_ , 33(1):107–124, 2016.
  * Martinez-Maldonado et al. [2020] Roberto Martinez-Maldonado, Katerina Mangaroska, Jurgen Schulte, Doug Elliott, Carmen Axisa, and Simon Buckingham Shum.  Teacher tracking with integrity: What indoor positioning can reveal about instructional proxemics.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 4(1):1–27, 2020.
  * Martins [2014] Luiza Prado de O Martins.  Privilege and oppression: Towards a feminist speculative design.  _Proceedings of DRS_ , pages 980–990, 2014.
  * Massumi [2015] Brian Massumi.  _Ontopower: War, powers, and the state of perception_.  Duke University Press, 2015.
  * Mayfield et al. [2019] Elijah Mayfield, Michael Madaio, Shrimai Prabhumoye, David Gerritsen, Brittany McLaughlin, Ezekiel Dixon-Román, and Alan W Black.  Equity beyond bias in language technologies for education.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 444–460, 2019.
  * Meyerhoff [2019] Eli Meyerhoff.  _Beyond Education: Radical Studying for Another World_.  U of Minnesota Press, 2019.
  * Moore et al. [2021] Shannon Dawn Maree Moore, Bruno De Oliveira Jayme, and Joanna Black.  Disaster capitalism, rampant edtech opportunism, and the advancement of online learning in the era of covid19.  _Critical Education_ , 12(2), 2021.
  * Moreau et al. [2019] Erwan Moreau, Carl Vogel, and Marguerite Barry.  A paradigm for democratizing artificial intelligence research.  In _Innovations in Big Data Mining and Embedded Knowledge_ , pages 137–166. Springer, 2019.
  * Morozov [2013] Evgeny Morozov.  _To save everything, click here: The folly of technological solutionism_.  Public Affairs, 2013.
  * Muller [2007] Michael J Muller.  Participatory design: the third space in hci.  In _The human-computer interaction handbook_ , pages 1087–1108. CRC press, 2007.
  * Muñoz et al. [2019] José Esteban Muñoz, Joshua Chambers-Letson, Tavia Nyong’o, and Ann Pellegrini.  _Cruising utopia: The then and there of queer futurity_.  nyu Press, 2019.
  * Nam and Samson [2019] SungJin Nam and Perry Samson.  Integrating students’ behavioral signals and academic profiles in early warning system.  In _International Conference on Artificial Intelligence in Education_ , pages 345–357. Springer, 2019.
  * Noble [2018] Safiya Umoja Noble.  _Algorithms of oppression: How search engines reinforce racism_.  nyu Press, 2018.
  * North [2006] Connie E North.  More than words? delving into the substantive meaning (s) of “social justice” in education.  _Review of educational research_ , 76(4):507–535, 2006.
  * Nur et al. [2019] Nasheen Nur, Noseong Park, Mohsen Dorodchi, Wenwen Dou, Mohammad Javad Mahzoon, Xi Niu, and Mary Lou Maher.  Student network analysis: A novel way to predict delayed graduation in higher education.  In _International Conference on AIED_ , pages 370–382. Springer, 2019\.
  * Obermeyer et al. [2019] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan.  Dissecting racial bias in an algorithm used to manage the health of populations.  _Science_ , 366(6464):447–453, 2019.
  * Ocumpaugh et al. [2014] Jaclyn Ocumpaugh, Ryan Baker, Sujith Gowda, Neil Heffernan, and Cristina Heffernan.  Population validity for educational data mining models: A case study in affect detection.  _British Journal of Educational Technology_ , 45(3):487–501, 2014.
  * Ogan [2019] Amy Ogan.  Reframing classroom sensing: promise and peril.  _interactions_ , 26(6):26–32, 2019.
  * Okur et al. [2017] Eda Okur, Nese Alyuz, Sinem Aslan, Utku Genc, Cagri Tanriover, and Asli Arslan Esme.  Behavioral engagement detection of students in the wild.  In _International Conference on Artificial Intelligence in Education_ , pages 250–261. Springer, 2017.
  * Palmer et al. [2016] Neal A Palmer, Emily A Greytak, and J Kosciw.  Educational exclusion: Drop out, push out, and the school-to-prison pipeline among lgbtq youth.  _New York, NY: GLSEN_ , 2016.
  * Pane et al. [2014] John F Pane, Beth Ann Griffin, Daniel F McCaffrey, and Rita Karam.  Effectiveness of cognitive tutor algebra i at scale.  _Educational Evaluation and Policy Analysis_ , 36(2):127–144, 2014.
  * Papageorge et al. [2020] Nicholas W Papageorge, Seth Gershenson, and Kyung Min Kang.  Teacher expectations matter.  _Review of Economics and Statistics_ , 102(2):234–251, 2020.
  * Passi and Barocas [2019] Samir Passi and Solon Barocas.  Problem Formulation and Fairness.  In _Conference on Fairness, Accountability and Transparency_ , 2019\.
  * Pedro et al. [2013] Maria Ofelia Pedro, Ryan Baker, Alex Bowers, and Neil Heffernan.  Predicting college enrollment from student interaction with an intelligent tutoring system in middle school.  In _Educational Data Mining 2013_ , 2013.
  * Pei and Crooks [2020] Lucy Pei and Roderic Crooks.  Attenuated access: Accounting for startup, maintenance, and affective costs in resource-constrained communities.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , pages 1–15, 2020.
  * Picard [2000] Rosalind W Picard.  _Affective computing_.  MIT press, 2000.
  * Piech et al. [2015] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein.  Deep knowledge tracing.  _Advances in neural information processing systems_ , 28:505–513, 2015.
  * Pierre et al. [2021] Jennifer Pierre, Roderic Crooks, Morgan Currie, Britt Paris, and Irene Pasquetto.  Getting ourselves together: Data-centered participatory design research & epistemic burden.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–11, 2021.
  * Prinsloo and Slade [2014] Paul Prinsloo and Sharon Slade.  Educational triage in open distance learning: Walking a moral tightrope.  _International Review of Research in Open and Distributed Learning_ , 15(4):306–331, 2014.
  * Raghavan et al. [2020] Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy.  Mitigating bias in algorithmic hiring: Evaluating claims and practices.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 469–481, 2020.
  * Rakova et al. [2020] Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury.  Where responsible ai meets reality: Practitioner perspectives on enablers for shifting organizational practices.  _arXiv preprint arXiv:2006.12358_ , 2020.
  * Reich et al. [2020] Justin Reich, Christopher J Buttimer, Alison Fang, Garron Hillaire, Kelley Hirsch, Laura Larke, Joshua Littenberg-Tobias, Roya Madoff Moussapour, Alyssa Napier, and Meredith Thompson.  Remote learning guidance from state education agencies during the covid-19 pandemic: A first look.  2020\.
  * Richardson et al. [2019] Rashida Richardson, Jason Schultz, and Kate Crawford.  Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice.  _New York University Law Review Online, Forthcoming_ , 2019.
  * Robertson and Salehi [2020] Samantha Robertson and Niloufar Salehi.  What if i don’t like any of the choices? the limits of preference elicitation for participatory algorithm design.  _arXiv preprint arXiv:2007.06718_ , 2020.
  * Rosa and Flores [2017] Jonathan Rosa and Nelson Flores.  Unsettling race and language: Toward a raciolinguistic perspective.  _Language in Society_ , 46:621–647, 2017.
  * Roschelle et al. [2020] Jeremy Roschelle, James Lester, and Judi Fusco.  Ai and the future of learning: Expert panel report.  Technical report, Digital Promise, 2020.
  * Sap et al. [2019] Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith.  The risk of racial bias in hate speech detection.  In _Proceedings of the Association for Computational Linguistics (ACL)_ , 2019.
  * Scholes [2016] Vanessa Scholes.  The ethics of using learning analytics to categorize students on risk.  _Educational Technology Research and Development_ , 64(5):939–955, 2016.
  * Sehmitt and Dorans [1990] Alieia P Sehmitt and Nell J Dorans.  Differential item functioning for minority examinees on the sat.  _Journal of Educational Measurement_ , 27(1):67–81, 1990.
  * Simonite [2018] Tom Simonite.  When it comes to gorillas, google photos remains blind.  2018\.
  * Slade and Prinsloo [2013] Sharon Slade and Paul Prinsloo.  Learning analytics: Ethical issues and dilemmas.  _American Behavioral Scientist_ , 57(10):1510–1529, 2013.
  * Sloane [2019] Mona Sloane.  Inequality Is the Name of the Game: Thoughts on the Emerging Field of Technology, Ethics and Social Justice.  In _Weizenbaum Conference 2019 “Challenges of Digital Inequality — Digital Education, Digital Work, Digital Life”_ , 2019.
  * Sloane et al. [2020] Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano.  Participation is not a design fix for machine learning.  _arXiv preprint arXiv:2007.02423_ , 2020.
  * Stark and Hoffmann [2019] Luke Stark and Anna Lauren Hoffmann.  Data Is the New What? Popular Metaphors & Professional Ethics in Emerging Data Culture.  _Journal of Cultural Analytics_ , May 2019.
  * Swauger [2020] Shea Swauger.  Our bodies encoded: Algorithmic test proctoring in higher education.  _Critical Digital Pedagogy_ , 2020.
  * Tang et al. [2018] Cui Tang, Yuanxin Ouyang, Wenge Rong, Jingshuai Zhang, and Zhang Xiong.  Time series model for predicting dropout in massive open online courses.  In _International Conference on Artificial Intelligence in Education_ , pages 353–357. Springer, 2018.
  * Teräs et al. [2020] Marko Teräs, Juha Suoranta, Hanna Teräs, and Mark Curcher.  Post-covid-19 education and education technology ‘solutionism’: A seller’s market.  _Postdigital Science and Education_ , pages 1–16, 2020.
  * Thorndike [1971] Robert L Thorndike.  Concepts of culture-fairness.  _Journal of Educational Measurement_ , 8(2):63–70, 1971.
  * Tsatsou et al. [2018] Dorothea Tsatsou, Andrew Pomazanskyi, Enrique Hortal, Evaggelos Spyrou, Helen C Leligou, Stylianos Asteriadis, Nicholas Vretos, and Petros Daras.  Adaptive learning based on affect sensing.  In _International conference on artificial intelligence in education_ , pages 475–479. Springer, 2018.
  * Tuck and Yang [2018] Eve Tuck and K Wayne Yang.  _Toward what justice?: Describing diverse dreams of justice in education_.  Routledge, 2018.
  * Varney [2012] Jennifer Varney.  Proactive (intrusive) advising.  _Academic Advising Today_ , 35(3):1–3, 2012.
  * Warschauer and Matuchniak [2010] Mark Warschauer and Tina Matuchniak.  New technology and digital worlds: Analyzing evidence of equity in access, use, and outcomes.  _Review of research in education_ , 34(1):179–225, 2010.
  * Weinstein [2020] Maya Weinstein.  School surveillance: The students’ rights implications of artificial intelligence as k-12 school security.  _North Carolina Law Review_ , 98(2):438, 2020\.
  * Weiss [1987] John Weiss.  The golden rule bias reduction principle: A practical reform.  _Educational Measurement: Issues and Practice_ , 6(2):23–25, 1987.
  * Williamson [2017] Ben Williamson.  Decoding classdojo: psycho-policy, social-emotional learning and persuasive educational technologies.  _Learning, Media and Technology_ , 42(4):440–453, 2017.
  * Williamson [2020] Ben Williamson.  Code acts in education: Re-engineering education, 2020.  URL https://nepc.colorado.edu/blog/re-engineering-education.
  * Winchester III [2018] Woodrow W Winchester III.  Afrofuturism, inclusion, and the design imagination.  _interactions_ , 25(2):41–45, 2018.
  * Winner [1980] Langdon Winner.  Do Artifacts Have Politics?  _Daedalus_ , 109(1):121–136, 1980.
  * Winograd et al. [1986] Terry Winograd, Fernando Flores, and Fernando F Flores.  _Understanding computers and cognition: A new foundation for design_.  Intellect Books, 1986.
  * Young [1990] Iris Marion Young.  _Justice and the Politics of Difference_.  Princeton University Press, 1990.
  * Zeide [2015] Elana Zeide.  Student privacy principles for the age of big data: Moving beyond ferpa and fipps.  _Drexel L. Rev._ , 8:339, 2015.
  * Zhao et al. [2019] Zhou Zhao, Haojie Pan, Changjie Fan, Yan Liu, Linlin Li, Min Yang, and Deng Cai.  Abstractive meeting summarization via hierarchical adaptive segmental network learning.  In _The World Wide Web Conference_ , pages 3455–3461, 2019.