fullcited

# Data Science and Social Justice in the Mathematics Community

Quindel Jones, Andrés R. Vindas Meléndez, Ariana Mendible, Manuchehr Aminian,
Heather Z. Brooks, Nathan Alexander, Carrie Diaz Eaton, and Philip Chodrow
Quindel Jones is a PhD candidate in systems modeling and analysis at Virginia
Commonwealth University.Andrés R. Vindas Meléndez is an NSF postdoctoral
scholar at UC Berkeley. In July 2024, he will start as an assistant professor
of mathematics at Harvey Mudd College.Ariana Mendible is an assistant
professor of mathematics at Seattle University.Manuchehr Aminian is an
assistant professor of mathematics at California State Polytechnic University,
Pomona.Heather Zinn Brooks is an assistant professor of mathematics at Harvey
Mudd College.Nathan Alexander is an assistant professor of mathematics and
data science at Morehouse College.Carrie Diaz Eaton is an associate professor
of digital and computational studies at Bates College.Philip Chodrow is an
assistant professor of computer science at Middlebury College. His email
address is<EMAIL_ADDRESS>
(March 2023)

##  1 Introduction

††The first seven authors are ordered by career stage at the time of article
submission.

In the summer of 2021, the Puerto Rico Association of Criminal Defense Lawyers
(PRACDL) submitted a brief to the US Supreme Court in the case _Rodríguez-
Rivera vs. USA_. At issue was whether a certain class of drug conspiracy
crimes should be considered “controlled substance offenses,” a status which
can carry enhanced sentences. Drug conspiracy charges implicate multiple
defendants and carry a low burden of evidence required to obtain a conviction
against alleged co-conspirators. PRACDL’s brief argued that prosecution in
drug conspiracy cases reflected discrimination and disparate impact along axes
of race, ethnicity, and class. The brief was supported by analysis from the
Institute for the Quantitative Study of Inclusion, Diversity, and Equity
(QSIDE), including work by present authors Manuchehr Aminian and Phil Chodrow.
00endnotetext: QSIDE’s website is https://qsideinstitute.org/.  Working in
close dialogue with lawyers at PRACDL, the QSIDE team wrote parsers to extract
information from unstructured docket data, combined these results with
geographic Census data, and tested hypotheses that conspiracy charges
disproportionately took place in areas of Puerto Rico impacted by poverty,
unemployment, and low rates of education.

This work with PRACDL is a small entry in a growing body of scholarship in
_data science for social justice_ (DS4SJ).111We also use the phrase _justice
data science_ interchangeably to refer to the same scholarly
area.00endnotetext: See _Flores González vs USA , 2022_ for another legal
brief by PRACDL supported by data analysis. This area has received growing
attention in several overlapping professional communities, including
mathematics, statistics, and computer science. Our focus in this article is
how members of the mathematical community can learn, practice, and support
this burgeoning area of scholarship and activism.

We highlight ways that DS4SJ work can both appeal to and challenge trained
mathematicians. Scholars who pursue DS4SJ may look forward to positive
outcomes for served communities, new research questions, new professional
connections outside the academy, and opportunities to connect their identity
or values to their scholarly work. Realizing these benefits, however, requires
confronting a range of challenges and risks. First, impact is never
guaranteed. In the PRACDL case, for example, the petition to the Court was
ultimately denied without comment. Furthermore, the process of justice data
science calls on skills in which many of us are untrained. Realizing impact
beyond the ivory tower requires engagement beyond the ivory tower, with
affected populations, domain experts, and decision-makers. When this work is
pursued without sufficient attention to the needs of impacted communities, the
results can fail as both scholarship and activism. Attentive engagement with
non-scholarly collaborators is a skill not taught in mathematics training
programs, perhaps because it is not viewed as “mathematical.” Finally,
researchers who take on justice data science work must naturally keep an eye
on how it aligns with their needs for advancement in their institutions and
professional communities.

Our overall aim is to help scholars who wish to begin or deepen their
engagement with DS4SJ work, and advocate to the broader mathematics community
that this work should be valued and supported. Our core arguments are:

  * (a).

Members of the mathematical community, broadly construed, have contributions
to make towards social justice causes through both scholarship and pedagogy.

  * (b).

Making these contributions requires us to engage with collaborators and
partners outside the academy, including impacted communities, policy makers,
and others who contribute experience, knowledge, and power beyond our own.

  * (c).

Data science work supporting social justice can indeed be a scholarly,
scientific endeavor, which should be supported by mathematical institutions at
all levels.

In Section 2 we define data science for social justice (DS4SJ), construing
this work broadly to include a wide range of computational, mathematical, and
qualitative work. We compare and contrast DS4SJ to related bodies of work,
including data science for social good and academic data science of human
behavior. Based on our definition, we propose a set of four guiding principles
for the practice of DS4SJ. These principles describe work that actively
engages community members and decision-makers; is aware of its own
limitations; intentionally aims to divest from privilege; and grapples with
the systemic nature of oppression. We continue in Section 3 with discussion of
some features of recent and ongoing work in DS4SJ. We especially focus on
considerations for early-career scholars considering greater involvement this
work in either research or pedagogy. We also highlight benefits for
departments that make a commitment to supporting this work. In Section 4, we
illustrate the practice of DS4SJ with a series of vignettes in which early-
career scholars describe their work in DS4SJ and its impact on their growth in
the broad mathematical community. We conclude in Section 5 with a reflection
and discussion on the outlook for future work.

##  2 Data Science For Social Justice

We take an intentionally broad view of _data science_ , referring to the wide
set of practices, methods, and activities that support the use of data to gain
insight and guide action. This includes standard steps in the data analysis
pipeline, such as data collection, exploration, visualization, modeling, and
communication. We also deliberately include an expansive range of other
activities, including: formulation of mathematical models, development of data
analysis and learning algorithms; qualitative research that contextualizes
analysis; and deployment of publicly-facing data products such as
visualizations and dashboards. In this broad view, data science work takes in
many academic disciplines. It is not restricted to departments of statistics,
mathematics, or computer science; it is indeed not restricted to STEM fields
at all. That said, while our view of data science is broad, our focus in this
article is primarily on how the mathematical community can engage with data
science work. For this reason, our discussion largely centers on activities
and practices in which mathematical training is relevant.

We frame data science for social justice within our broad construal of data
science as a whole.

######  Definition.

_Data science for social justice_ (DS4SJ) is data scientific work (broadly
construed) that actively challenges systems of inequity and concretely
supports the liberation of oppressed and marginalized communities.222Forms of
oppression and marginalization include but are not limited to racism,
colonialism, classism, sexism, homophobia, transphobia, and ableism.

DS4SJ acknowledges that some groups of people have been historically
oppressed, marginalized, and disenfranchised, and aims to rectify these harms.
It does not necessarily aspire to benefit “society as a whole,” but
specifically aims to correct injustice. DS4SJ is therefore distinct from data
science for social _good_ , a phrase that often refers to any data scientific
work with net positive social impact. Work featured by the organization named
Data Science For Social Good, for example, includes efforts towards
sustainability, efficient public infrastructure, and reduction of political
corruption.00endnotetext: Data Science For Social Good’s website is
https://www.datascienceforsocialgood.org. These are general social goods which
benefit large segments of the world’s population. Social _justice_
applications also benefit populations, but are distinctive in their explicit
focus on the needs of oppressed and marginalized communities harmed by
historical wrongs. We emphasize that this distinction, though important, is
also porous. Many social good projects also have social justice elements; for
example, the improvement of public infrastructure may have an especially large
impact on communities that have been historically underserved by that
infrastructure. Similarly, justice efforts that rectify systematic wrongs may
be appropriately viewed as promoting broader social good. While it is
important to distinguish these aims and scopes, we emphasize that neither
precludes the other. 00endnotetext: We acknowledge that the question of which
populations are oppressed or marginalized, and in which ways, may be unclear
or contentious. Because of this, there may not be consensus on whether a given
organizing effort is best framed as aiming towards social justice, social good
more broadly, or something else entirely. These same considerations apply to
data science efforts, and imply that the line between data science for social
justice, data science for social good, and other forms of socially-motivated
data work may be porous and contested.

DS4SJ is also importantly related to work on bias and harm in large-scale
machine learning systems. Recent work has shown that many of these models
contain significant biases against members of marginalized identity groups.
These effects are often intersectional: in a famous study by [7], facial
recognition algorithms performed worse on Black women than they did on either
Black men or white women, with the faces of white men being recognized most
reliably. Contributors to bias in machine learning models include
underrepresentation of marginalized identities or overrepresentation of
oppressive stereotypes in training data; use of flawed proxy variables for
measuring predictive accuracy; or failure to consider the role of an automated
system in social and historical context. With the growing prevalence of
machine learning models in personal technology, hiring, policing, and
healthcare, the stakes for designing nondiscriminatory systems at mass scales
are extremely high. We view work toward fair machine learning as DS4SJ: it is
data science that challenges systems of inequity and supports the liberation
of oppressed and marginalized communities. This is especially true when the
work influences the design and assessment of deployed, large-scale systems.
That said, the majority of this work is carried out in the computer science
community rather than the mathematics community and is therefore outside our
scope. We therefore refer the interested reader to [19] for a survey of recent
work in bias and fairness in machine learning, including conceptual
foundations, empirical findings, and proposed methods for reducing algorithmic
discrimination in a range of machine learning tasks.

Finally, we distinguish DS4SJ as a specific subset of the broad academic area
of data science and modeling of social systems. This large and growing area
includes work in applied mathematics, computational social science, physics,
network science, computer science, and allied fields. Some of this work may
indeed actively support the liberation of oppressed and marginalized
communities, thereby constituting DS4SJ. Other work—perhaps the majority—in
this area has as its primary goals the furthering of quantitative insight into
social systems within the scholarly community. An example of the latter kind
of work is mathematical analysis of dynamical models of social phenomena, such
as segregation or opinion polarization. The long-term impact of such work may
be primarily localized in scholarly publications, and may not inform activism,
policy, or other mechanisms that affect the lives of those who are not
professional researchers. We emphasize that our aim is not to disparage this
body of work, and indeed, several of the present authors maintain active
research agendas in this broad area.

In summary, our focus here—data science for social justice in the mathematical
community—overlaps with but is distinct from data science efforts toward
broader social good; research on fairness in machine learning; and the
quantitative modeling of social systems.

###  2.1 The Process of DS4SJ

Many of us in the mathematical community are accustomed to measuring the
success of our research program in units of theorems, experiments, papers,
venues, or citations. The processes by which we conduct our research are often
optimized towards these measures. Data science for social justice measures its
success in terms of impact on the lives of oppressed and marginalized peoples.
This focus requires that we pursue the work in ways likely to realize impact,
while also minimizing risk of unintended harm. The process of this work may
therefore look very different from the process of much other mathematical
scholarship. We propose a series of four guiding principles for the process of
DS4SJ, inspired by [4].

######  Principle 1 (Collaboration).

DS4SJ requires partnership with community members and decision-makers in order
to realize intended impact and minimize unintended harm.

Principle 1 reflects two simple facts. First, as mathematicians, we are often
inexperienced in conceiving, pursuing, or applying our work outside of
academic or industrial research environments. We often lack relevant training
in social science, policy, or activism. We therefore often need collaboration
to execute projects and build skills. Second, work that supports a community
must be guided by careful engagement _with_ that community in order to
understand the needs we aim to meet. We identify at least three important
elements of effective collaboration for DS4SJ:

  * •

_Perspective_ allows us to understand the needs of the people we aim to serve
and helps us carry out our work without causing unintended harm.

  * •

_Process_ helps direct the work with methods that respect the community, and
in directions that are responsive to community perspective with results that
will support future action.

  * •

_Power_ makes it possible to actually carry out the intervention informed by
our data scientific work.

Perspective is an especially important element of collaboration, as it helps
us understand the needs of the communities we serve. The best way to build
this perspective is through partnership _with the community itself_. This is
ultimately the only reliable way to understand the needs of its members and
whether they want what we are able to offer; perspective from other scholars
or outside experts can inform only insofar as it is grounded in community
engagement. The slogan “Nothing about us without us,” popularized in English
by South African disability activists, offers a concise maxim.00endnotetext:
For more on the origins of the slogan “Nothing about us without us,” we refer
the reader to [9]fullcitedcharlton2000nothing

A recent effort at the Institute for Mathematical and Statistical Innovation
(IMSI) illustrates these forms of collaboration. A working group led by author
Carrie Diaz Eaton worked in partnership with Nuevas Voces, a leadership
program by the Woonasquaucket River Watershed Council (WRWC), for residents of
a low-income, immigrant and multilingual community in Providence, Rhode
Island. This collaboration offered _perspective_ by helping the working group
understand community needs through the insights of Nuevas Voces, who expressed
a need to call attention to the flooding of neighborhoods along the watershed
induced by ongoing climate change. The collaborative _process_ involved
meeting in impacted neighborhoods, conducting interviews in Spanish, and
offering support to community members from Nuevas Voces and the WRWC to guide
the work as it progressed. Maintaining collaboration through the
implementation phase invested the work with _power_ to make an impact. The
primary deliverable of the intervention with Nuevas Voces was a geospatial
dashboard which incorporates not only flooding information but also community
resources and stories. Nuevas Voces and WRWC have committed to integrating
these tools into their own organizations in order to support their missions.

######  Principle 2 (Critical Reflection).

DS4SJ requires critical reflection on one’s own identity, position, and
privilege, and on the role these play in one’s scholarly practice.

We use “critical” in the sense of _critical theory_ : a critical perspective
is one which views structures of power and privilege with skepticism, and aims
toward a more equitable, free, and just society [16]. 00endnotetext: For more
on critical theory, especially in its connection to liberatory pedagogy, see
[13]fullcitedfreire1996pedagogy. Principle 2 asks us to reflect on the
identities and privileges we inhabit, with special eye towards how these
identities and privileges may tacitly inform our work. Many members of the
broad mathematical community inhabit multiple forms of professional privilege.
Our profession is culturally respected, often commands above-average salaries,
and does not usually place us at risk of physical harm. Mathematical ways of
knowing are popularly (and often dangerously) viewed as epistemically
privileged: piercing, infallible, and not subject to dispute. Many of us in
academia have the luxury of measuring deadlines in units of months or years,
rather than days or hours. When projects “don’t work out,” they can often be
safely dropped in favor of new ones. Additionally, a disproportionate number
of professional mathematicians benefit from White and male
privilege.00endnotetext: The disproportionate representation of white men in
professional mathematics is highlighted in many places, for example this
article in _Scientific American_ :
https://www.scientificamerican.com/article/modern-mathematics-confronts-its-
white-patriarchal-past/.

It is important to acknowledge and grapple with privilege when aspiring
towards justice work of any form, including DS4SJ. For example, how do we
reconcile the typical pace of mathematical research with the fact that, as
author Nathan Alexander frequently emphasizes, _people are dying_ from
systemic forms of injustice? The work of interrogating our privileges and
understanding how they inform our professional practice is challenging, but
there are some aspects of mathematical training that support mathematicians in
this work. Privilege can be viewed as a collective social assumption about who
deserves freedom, respect, and resources. Assumptions, in the form of
theoretical axioms or modeling inputs, are pervasive in mathematical practice.
As mathematicians, we are trained in the fundamental skill of articulating
assumptions, questioning their applicability, and deriving their consequences.
These are exactly the skills that support critical reflection, if they can be
turned inward. Author Quindel Jones models reflection of the effects of her
own identity and positionality on her scholarly work in her vignette in
Section 4.

######  Principle 3 (Interrogation of Privilege).

DS4SJ asks the researcher to work actively to interrogate and see past their
own privilege.

As discussed under Principle 2, mathematicians often benefit from several
forms of systemic privilege, often based on a combination of identity, social
prestige of profession, and scientific ways of knowing. These forms of
privilege can be obstacles to effective DS4SJ work, especially in regards to
effective partnership with collaborators from communities that may be very
different from our own (Principle 1). Seeing past our our privilege therefore
involves epistemic humility: the active centering of ways of knowing other
than our own. 00endnotetext: [15]fullcitedho2011trusting considers epistemic
humility in another relationship involving experts, the relationship between
physicians and disabled patients. Epistmic humility is also related to
_cultural humility_ , a concept popularized by
[28]fullcitedtervalon1998cultural. Learning community needs may require that
we heed diverse forms of knowledge, such as community narrative, individual
experiences, or common wisdom from other professions. Insisting on the primacy
of mathematical or scientific ways of knowing over the ways of our served
communities is likely to result in failure to synthesize necessary information
about community needs or ways to meet them. This can in turn lead to failure
to realize intended impact, or even in unintended harm.

00endnotetext: [2]fullcitedalexander2021critical offers further illustrations
of interrogating privilege in practice.

In his vignette in Section 4, Andrés R. Vindas Meléndez discusses how his
trajectory into DS4SJ as a theoretical mathematician has been aided by
awareness of the privilege of academic mathematics and his willingness to
engage with other scholarly communities.

######  Principle 4 (Systemic Perspective).

DS4SJ requires a critical _systemic perspective_ : a critical awareness of how
systems of power operate to privilege some and oppress others.

Many mathematicians use sophisticated tools study the function of complex and
interrelated systems in domains ranging from biology to engineering to human
society. Systems thinking is a fundamental skill in justice-oriented data work
as well, and informs the technical tools we deploy. For example, there is
extensive mathematical theory describing mechanisms for fairly distributing
resources, but this theory usually lacks a description of historical and
present systemic forces of oppression. Rectifying this oppression requires
more than fair division of resources; it also requires active work against
these systemic forces. In her vignette in Section 4, Heather Zinn Brooks
discusses how taking a systemic perspective on the mathematics community can
inform data-driven studies of gender representation in mathematical subfields.
In some cases, it may also be possible to explicitly model social processes
that contribute to disparity. Modeling the social processes of police use-of-
force, for example, may ultimately provide insight on how to stop
disproportionate police killings of unarmed Black people [31].

##  3 Challenges and Opportunities in DS4SJ

Justice data science addresses a wide range of problems both within and beyond
academia. We roughly divide this work into three overlapping categories.

  * •

Data Accessibility: Ida B. Wells famously wrote that “the way to right wrongs
is to shine the light of truth on them.” Some justice data science work aims
simply to make new kinds of information available and interpretable to new
audiences. Examples include efforts to compile accessible, online databases of
police violence,00endnotetext: One online database of police violence is the
Mapping Police Violence project led by Samuel Sinyangwe:
https://mappingpoliceviolence.us federal judicial sentencing, 00endnotetext:
[10]fullcitedciocanel2020justfair obtain a large data set of federal criminal
sentencing records, recorded at https://qsideinstitute.org/research/criminal-
justice/justfair/. This data was further analyzed by
[26]fullcitedsmith2021racial. and cultural representation.00endnotetext: One
data set on cultural representation of artists in major US museums was
compiled and released by QSIDE at https://qsideinstitute.org/research/the-
arts/diversity-of-artists-in-major-u-s-museums-2019/.

  * •

Analysis and Critique: In some cases, the simple presentation of data in an
easily accessible format may be enough to motivate action. In other cases, the
sheer quantity of data and the need to account for multiple factors requires
explicit, formal data analysis. This analysis can sometimes support controlled
or causal claims about the magnitude or impact of bias, and may be published
in scholarly journals. In this context, it is important to emphasize that the
mere availability of evidence or analysis is often insufficient to change
minds or spur change. 00endnotetext:  [20]fullcitedmercier2011humans offer a
theory of reasoning as primarily rhetorical rather than deductive, and
includes a survey of work studying confirmation bias and other mechanisms that
hinder our ability to change our minds in response to evidence.

  * •

Implementation: Impact is ultimately the axis along which justice work,
including data science, is measured. Successful DS4SJ work either achieves
impact directly through action or supports follow-up efforts by others. In
some cases, it may suffice to bring analysis and critique in front of the
right parties; for example, work identifying the racial impact of political
gerrymandering is analysis and critique, and can contribute toward its
intended impact when presented as arguments in court cases. Other examples
include the deployment of predictive or decision-making models aimed at
explicit equity goals, such as predicting police misbehavior, landlord
exploitation, judicial decision-making, or corporate tax abuse. Data
scientists and mathematicians, as engaged human beings, also have
opportunities to engage in direct action in support of their chosen causes,
including community organizing, attendance at protests, and participation in
mutual aid societies.

###  3.1 Getting Involved with DS4SJ

We hope that some of our readers are considering whether to engage in or
indirectly support work in data science for social justice. We now highlight
some of the reasons why this may be appealing for scholars in mathematics and
allied fields. There are a wide range of benefits, including active research
prospects, ways to engage undergraduates in the classroom, and opportunities
to recruit junior collaborators for research projects.

Justice data science work can motivate the development of new mathematics,
data scientific techniques, and methodological perspectives. One well-known
example is the development of methods for detecting and quantifying racial
bias in electoral maps by defining null distributions on spaces of districting
schemes using techniques from metric geometry and using Markov-chain
algorithms to sample from them [5]. This body of work has produced analysis
and critique in the form of papers quantifying racial bias in existing
political maps. There have also been impacts from engagement with power
partnerships, with researchers collaborating with state
governments00endnotetext: Moon Duchin offers a popular description of
collaborations between metric geometers, legislators, and legal experts in a
recent article in _Scientific American_ :
https://www.scientificamerican.com/article/geometry-versus-gerrymandering/ and
contributing to amicus briefs before the US Supreme Court.00endnotetext: One
amicus brief submitted by the Metric Geometry and Gerrymandering Group to the
Supreme court may be found at https://mggg.org/SCOTUS-MathBrief.pdf

There are also opportunities for methodological advancements in applied
statistics, optimization, and network science. [22], for example, develop
novel, highly-scalable statistical tests for detecting racial bias in large-
scale datasets of police behavior.00endnotetext: [21]fullcitedpierson2018fast
offer additional work on large-scale statistical tests for bias. Problems
inspired by equitable access to schools, grocery stores, and polling places in
urban infrastructure networks have led to the development of novel algorithms
for editing graph edges to influence diffusion dynamics [23]. A growing body
of work uses network-based and statistical methods to quantify gendered
inequalities in citation, promotion, and retention across academic disciplines
[30]. In her vignette below, author Heather Zinn Brooks describes ongoing
related work in mathematics. As author Andrés R. Vindas Meléndez writes in his
own vignette, the urgency of social justice applications have helped him both
develop an interest in data science techniques and begin to reflect on whether
combinatorics, his primary area of research training, might have useful
applications toward justice causes.

An orientation towards social justice offers benefits for pedagogy in
mathematics, data science, statistics, and computer science. Real-world topics
related to issues of injustice and inequity are fruitful sources of problems
and examples [17]. Beyond simply furnishing problems, however, justice
orientations can support mathematics instructors in welcoming, engaging, and
retaining students in undergraduate STEM majors/minors, and higher education
more generally. Justice orientations can be especially useful for engaging
students who have been historically excluded from or oppressed or marginalized
within traditional classrooms. 00endnotetext: [14]fullcitedgaribay2015stem
discusses justice orientation as a promoter of student engagement. Connecting
classroom content to “prosocial communal outcomes” is a recommendation of a
recent report focused on improving the persistence of marginalized students in
STEM disciplines [12].

It is important to note that justice-oriented pedagogy presents challenges for
both students and instructors, and can be harmful to students if cases are
presented in a demeaning or tokenizing way. There is potential for harm if
instructors incorporate examples that do not promote positive student identity
or which fail to acknowledge diversity of experience within cultures and
identities [18].

Effective justice-oriented pedagogy seeks to advance students’ critical
consciousness–their ability to challenge what is framed as normal, whose
interests are served by that framing, how system structures are reified, and
in what ways they could, potentially, be different. Such pedagogy not only
welcomes students who have been historically marginalized in classroom spaces,
but also challenges all students to grow as justice-oriented systems thinkers.
00endnotetext: For further suggestions on critical conversations to support
justice-oriented mathematics pedagogy, see [2]fullcitedalexander2021critical.
For example, author Nathan Alexander uses a justice orientation in his
introductory statistics classroom when framing the topic of ethical research
and data use. The Institutional Review Board (IRB) system was formed in part
due to the revelation of the medical abuse of almost 400 Black men in the
Tuskegee Syphilis Study. The modern IRB review system aims to prevent future
such flagrant breaches of medical ethics. This, however, does not fully
address the dimensions of structural racism in the practice of medical
research; even today, Black principal investigators are less likely to have
grant proposals funded by the National Institutes of Health. 00endnotetext:
Disparity in funding of NIH proposals is studied by
[27]fullcitedtaffe2021equity. These considerations are not only context for
the modern practice of statistics and data-informed research, but they are
also opportunities to practice core technical course content. Students can
study and evaluate, for example, the methods underlying recent findings that
the Tuskegee study is _still_ shortening the lives of Black men by sowing
justified mistrust in medical institutions [3].

These are challenging topics to navigate without tokenizing or otherwise
harming students. Doing so requires instructors to show vulnerability,
humility, and authentic willingness to learn _with_ students in a community
built on mutual respect, openness, and collective critical reflection. The
reward is an opportunity for students to develop technical skills and critical
awareness in tandem, each reinforcing the other. 00endnotetext: For further
critical perspective on the role of justice and equity-oriented data science,
see [1]fullcitedalexander2022beyond and [8]fullcitedcastillo2022.
[11]fullcitedeaton2022teaching offers a detailed description of implementing
justice orientation in machine learning pedagogy.

Similarly, students at all levels are often excited to work on research
projects that involve justice work. Undergraduates are increasingly involved
in and committed to social justice and are drawn to research experiences with
obvious real-world applications [25]. As author Ariana Mendible describes in
her vignette below, her early research projects with justice focus have
received much higher than expected interest from prospective research students
at her institution. This may be in part due to the accessibility of these
projects. DS4SJ work often has lower technical barrier to entry when compared
to many other problems in academic mathematics, making it possible to involve
students across a greater range of experiences and academic backgrounds at
earlier stages in their academic careers. Students’ personal experiences and
interests can also be explicit assets in the pursuit of DS4SJ research
projects, especially when they are members of the communities who stand to be
impacted by the work.

Finally, we emphasize that DS4SJ work places distinctive demands on
researchers and instructors, and is therefore not necessarily for everyone.
First, scholars may reasonably have motivations and interests that are not
fully aligned with DS4SJ work. DS4SJ work measures itself first and foremost
on positive impact for marginalized or oppressed peoples. Scholars pursuing
this work therefore need both orientation towards concrete outcomes and
willingness to devote time towards realizing those outcomes. This in turn
requires time to cultivate critical perspective, engage carefully with
communities, and develop nontechnical skills. We have nothing but respect for
scholars who reflect on the demands of DS4SJ and decide to pursue other paths.
Second, we acknowledge that professional incentives in many institutions and
societies may not encourage DS4SJ work. For example, some departments may not
recognize DS4SJ scholarship as being “sufficiently mathematical” to warrant
promotion or recognition, or may view justice-oriented content in the
classroom as a distraction. Scholars cannot be expected to commit to this work
if they feel it endangers their prospects for advancement in their
departments, institutions, or communities.

###  3.2 DS4SJ and the Math Community

We therefore call on departments, institutions, and professional communities
to support work in DS4SJ. Fortunately, DS4SJ work is aligned with the stated
goals of these departments and communities, implying both opportunity and
incentive to support these efforts. The AMS Statement on Equity, Diversity,
and Inclusion states our shared goal of _a community that is diverse,
respectful, accessible, and inclusive_. We are still far from this goal in
2023. Academic mathematics still has not reached proportional representation
or equity of opportunity along the lines such as gender and
race.00endnotetext: https://www.scientificamerican.com/article/modern-
mathematics-confronts-its-white-patriarchal-past/ Many departments struggle to
recruit and retain minoritized trainees and faculty. Active support of justice
data science may be able to contribute to these goals. Based on interviews
conducted with underrepresented racial minority students in US STEM graduate
programs, [29] recommends that departments seeking to retain underrepresented
minority students should legitimize and expand research that has direct
applications for social justice. Illustrating this effect, author and Ph.D.
candidate Quindel Jones discusses how the opportunity to use mathematical
models to improve health outcomes for Black women is a motivating thread in
her career trajectory in her vignette below. Jones highlights how the presence
of a growing community of mathematical scholars focused on data science and
social justice has promoted her scholarship, enabled her to learn from mentors
who shared some of her identities, and connect with collaborators who share
her interests. Support of justice data science by departments, research
institutions, and professional organizations can not only promote scholarship
that stands on its own merits, but also further our shared mission towards a
more equitable, diverse, and inclusive mathematics profession. We discuss
concrete ways that departments, institutes, and societies can support DS4SJ in
Section 5.

##  4 Experiences in DS4SJ

In this section, we highlight the experiences of several pre-tenure
mathematicians in their diverse engagements with data science for social
justice. Their stories offer examples of how the principles of DS4SJ described
in Section 2.1 can be implemented, and how the benefits and challenges of this
work have influenced their trajectories. Ariana Mendible discusses how
collaboration has impacted and improved her DS4SJ work (Principle 1). Quindel
Jones provides a critical reflection of identity and positionality on
scholarship (Principle 2). Andrés R. Vindas Meléndez interrogates the
privilege and challenges of doing DS4SJ work as a theoretical mathematician
(Principle 3). Heather Zinn Brooks highlights the importance of taking a
systemic perspective on research work in DS4SJ (Principle 4). Their
experiences highlight the roles of identity, belonging, and justice
orientation in the trajectories of early-career scholars working in science
for social justice.

#### Ariana Mendible, Seattle University

I recently returned to my alma mater, an institution with a strong commitment
to justice, as an assistant professor of mathematics. I recognize the
privilege of this position, offering the freedom to work on unfunded problems
that align with my personal values. Pivoting my research to DS4SJ has offered
fruitful research collaborations that promise meaningful impact within and
outside of academia.

My work as a co-director of QSIDE’s Small Town Police Accountability (SToPA)
research lab has shown me the eagerness of rising scholars to engage with
DS4SJ. We have worked together to tackle tasks like optical character
recognition and topic modeling with the goal of empowering small towns to
access and understand data from their police departments. The impact and
urgency of these problems invites a broad audience to participate in research,
including students who may not otherwise have envisioned themselves as
researchers.

Sociologists, lawyers, and activists participating in our lab efforts enrich
our perspectives beyond our mathematical and technical expertise. Importantly,
collaboration with community leaders guarantees direct and effective change
from our work. Conversations with one town’s racial justice and police reform
group provided town-specific history and helped narrow our focus to the most
impactful questions. Community members have expressed enthusiasm that our
quantitative work will further empower them to reform their local police
departments. We hope that their enthusiasm will help our work inform community
decision-making and realize its intended impact beyond the ivory tower.

My long-term goal is to continue building a research program that invites
students to work on problems that are meaningful to them and their
communities. I believe that this program will not only enact my values, but
ultimately strengthen my academic career.

#### Quindel Jones, Virginia Commonwealth University

I am a queer Black woman born and raised in Jackson, Mississippi, a part of
the American South.

While I initially took to dance and writing, my mother’s affinity to numbers
engaged my mathematical curiosity. My own affinity grew as community summer
enrichment programs kept me engaged. Throughout high school and college, there
were opportunities for me to use mathematics in my personal life and learn
from people who looked like or cared about me. I graduated from Jackson State
University, one of six historically Black colleges and universities (HBCUs) in
the state, under the guidance of Dr. Jana Talley, whom I met in the I.C.STEM
program at Jackson State when I was in high school.

I am currently a doctoral candidate in applied mathematics with a
concentration in mathematical biology at Virginia Commonwealth University.
Since entering graduate school, my professional mission has been to use
mathematics to improve the lives of people that look like me—especially Black
women. In my dissertation project, I am developing a dynamical model of pain
levels during sickle cell disease, a disease that disproportionately affects
Black people. We start with self-report data of patient pain, collected by a
member of our interdisciplinary team. We use this data to fit an ordinary
differential equation (ODE) model of a patient’s pain profile over time. This
model is already informing the data collection strategies for the next round
of our study. Long-term, we hope to embed our model in a wearable app. We hope
for patients to be able to log their sleep and pain self-reports, and from
these receive warnings about likely upcoming pain crises. Patients could use
these warnings to seek prophylactic care or medication.

As I continue on my mathematics journey, I am discovering more and more ways
in which mathematics can be applied to address inequities. With my own life
having been shaped by inequity due to my marginalized identity in U.S.
society, I am thrilled that there’s an active community centered on math and
data science for social justice. Connecting with this community in venues like
the recent program at ICERM have been formative experiences in my mathematical
career. These programs and their leaders have helped me see myself and my
concerns as important. Part of my excitement for my future as a researcher is
knowing that socially impactful mathematical work exists, and that there is a
growing community of collaborators with whom to do it.

#### Andrés R. Vindas Meléndez, UC Berkeley

My mathematical work is mainly in combinatorics. I have always found value in
this work from meaningful interactions with collaborators and the joy of
mathematical discovery.

I had my “formal” introduction to data science for social justice at a recent
ICERM program on this topic. I began working with a number of collaborators to
address a deceptively simple question: _who_ or _what_ is a mathematician?
Narrow definitions based on job titles or institutional affiliations can be
precise, but can also be used to implicitly or explicitly exclude people from
the discipline. In a recent preprint, we examined some of the tensions in
several formulations of what it means to be a mathematician, including those
based on self-identification, activities, and qualifications. 00endnotetext:
The preprint referred to by Andrés R. Vindas Meléndez is: [6]fullciteddefs For
future work, we aim to design a survey of mathematicians (broadly construed)
across a wide range of institutions, personal identities, job titles, and
areas of scholarly and pedagogical interest. We will study these data using
techniques like text analysis and clustering to extract qualitative insight
from survey participant responses. From this work, we hope to deepen our
understanding of the identities and needs of mathematicians in order to
support interventions at multiple levels to increase the participation of
marginalized or underrepresented groups.

My interest in the topic of who is considered a mathematician, and who is
welcomed in our community, is informed by my own identity and experience. My
identities as a queer, chronically-ill, first-generation, Latinx mathematician
have shaped my own experience in academic mathematics, including both
successes and struggles. I now benefit from privilege in my study and work and
well-resourced institutions. My experiences navigating my career so far
highlight the need for spaces that allow marginalized people to feel welcomed,
comfortable, and supported in the mathematics community.

As someone whose training is primarily in theoretical math, I sometimes feel
that I do not have much to contribute to DS4SJ. I remind myself that this work
requires diverse experience and expertise and my own identities offer
important perspective on critical questions. What is missing from our survey
efforts? What identities might be overlooked or mischaracterized by our
approach? What voices are centered in extant literature? Whose voices are
neglected?

Recently, I have been developing my technical data science skills. At my
current institution, I am participating in workshops on Python, R, data
visualization, and text analysis as a means to build my skills for my research
projects. Furthermore, I have joined the Digital Humanities Working Group,
which allows me to have interdisciplinary conversations and reflect on the
social prestige that mathematicians carry.

#### Heather Zinn Brooks, Harvey Mudd College

My identities as a woman and a first-generation college student have impacted
my career trajectory and inspired my involvement in research in data science
and social justice. Growing up, I didn’t know that “mathematician” was a real
job title. As a student, I struggled to see myself as a mathematician because
of the lack of role models who shared these identities. Now that I have the
privilege to carry the title of “professor of mathematics,” one of my
motivations for becoming involved in work in DS4SJ is to give back to (and
improve) a community that has benefited me. I recognize that my positionality
as a White, cis-gendered, able-bodied individual has contributed toward my
positive experiences in the mathematics community. My interest in creating
inclusive mathematical spaces has led me to a recent project modeling female
gender representation in mathematical subfields and institutions. In this
project, my collaborators and I use a data set compiled from the Mathematics
Genealogy Project (MGP), combined with algorithmically inferred binary gender
from names. This data has important limitations. Binary categories are
imperfect representations of gender, an inherently nonbinary facet of
identity. Furthermore, gender is not determined by name or appearance. We
assume that misgendering takes place in our data set, and we view our results
as at most estimates of macroscopic trends. 00endnotetext: Another important
limitation of the automated gender inference methods described in Heather Zinn
Brooks’ vignette is the Euro- and US-centrism of most commercially available
algorithms, including the ones we used. For a review of these biases and
limitations, see [24]fullcitedsantamaria2018comparison Within these
limitations, our goal is to build understanding of how some subfields and
institutions have reached relatively high levels of female gender
representation, while others remain strongly male-dominated.

We study the roles of accumulation, homophily, and prestige as mechanisms
supporting or inhibiting female gender representation in mathematics. A
centerpiece of our work is the development of data-informed branching process
models for replicating features found in real data. We use machine learning
techniques to highlight useful predictors in the data and inform the model
mechanisms. In addition to these technical contributions, a long-term aim is
to develop actionable strategies to support more inclusive mathematical
spaces.

My formal training in applied dynamical systems and mathematical biology has
allowed me to connect this motivation into my research in meaningful ways and
has challenged me to adopt a systemic perspective on the way our mathematical
community functions. Personal identity characteristics, institutional
prestige, and perceptions of and the microcultures within mathematical
subfields are all salient features that impact the way our community looks,
feels, and operates. This systemic perspective allows my collaborators and I
to interrogate the MGP data to highlight, explore, and understand the impact
of these features.

My primary motivation for engaging in justice-oriented data science work is to
fight systemic injustice and oppression. My involvement in this work has also
inspired new scholarship, provided a source of joy in my research career, and
allowed me to connect my identity to my mathematical pursuits.

##  5 Outlook

We have argued that many members of the mathematical community can, if they
choose, further justice work through the reflective, critical practice of data
science. We have highlighted some of the benefits for scholars, especially
early-career mathematicians: DS4SJ allows scholars to do work with concrete
impact, to relate their identities to their work, to find inspiration for new
technical problems, and to connect with their students inside and outside the
classroom. There are communities emerging to support this growing
subdiscipline, foster collaborations, and build critical expertise. We close
with some suggestions for individuals, departments, and institutions to
nurture work in this burgeoning area.

While individual motivation may spark engagement in DS4JS scholarship,
departments, institutions, and professional societies must offer active
support for this scholarship to be successful. This is especially pertinent
considering that a large portion of the work in DS4SJ is being spearheaded by
early-career mathematicians, including untenured professors, industry
professionals, postdocs, and students. Since many of these scholars lack
career stability, it is important that institutions align incentives so that
these scholars can be confident in their ability to advance their careers
while pursuing justice work. At all levels, DS4JS work must be structurally
and financially supported to sustain scholarship in this field.

Academic institutions can support justice-oriented scholars at all career
stages. Departments can support faculty in their teaching endeavors by
encouraging the development of new courses at the intersection of mathematics,
data science, and social justice. Programs can provide training to
undergraduate and graduate students and facilitate opportunities for community
collaborations. Hiring committees can revise rubrics to explicitly place value
on justice-oriented work, which may be particularly important in searches
related to applied data science. In retention, promotion, and tenure
decisions, work must be understood and valued appropriately. For many
departments, this necessitates broadening the perspective on what constitutes
evidence of scholarship in this area, including both publications and evidence
of impact of the scholar’s work outside academic venues. In order to
meaningfully support justice-focused scientific work, departments must
formally recognize its value in training and evaluative processes at all
levels.

The structural support of institutes and professional societies is also
critical to furthering this growing subfield of mathematics. As with any
subfield, workshops and conferences are critical for disseminating work,
generating new ideas, and creating and maintaining connections among scholars.
Such events can be especially impactful for early-career scholars aiming to
make connections and jump-start their scholarship. The Summer 2022 program on
“Data Science and Social Justice: Networks, Policy, and Education” at ICERM,
at which the present authors began this article, is just one example of a
growing number of research community spaces.

00endnotetext: Other examples include a recent workshop on Mathematics and
Racial Justice at the Mathematical Sciences Research Institute (MSRI), the
upcoming workshop on Interdisciplinary and Critical Data Science Motivated by
Social Justice at the Institute for Mathematical and Statistical Innovation,
and the second part of the program on Data Science and Social Justice at the
Institute for Computational and Experimental Research in Mathematics. We also
note the long-standing annual Critical Inquiry in Mathematics Education
conference at MSRI.

Simply making venues available for workshops and collaborations is, however,
not sufficient. Adequate, equitable funding is necessary for diverse
researchers and community members to fully participate in partnership.
Providing this funding may challenge traditional institutions. Some examples
of funding to support equitable participation include travel for community
participants (not just researchers); upfront financial support to graduate
student researchers for travel and meals; and financial support for childcare
or partner travel. Measures like these enable the full participation of
individuals for whom the typical financial and time costs of workshop
participation may be especially heavy. We therefore encourage reflection and
creativity on the part of institutes and other funded projects to design
funding which supports equitable participation in DS4SJ work.

Professional societies can further research and innovation by forming and
supporting professional communities for mathematicians working in DS4SJ.
Messaging from these societies is a powerful and important tool for building
respect and professional legitimacy for justice data scientists and the
scholarship they create. Such messaging also helps scholarly reviewers
understand the nature of DS4SJ research, which this article aims to inform.

The intersection of data science and social justice work represents an
exciting opportunity for mathematical scientists to work toward a more just
and equitable future, both within our own community and beyond. We hope that
this article can promote the growth and success of data science for social
justice.

## Author Positionality

The authors of this manuscript represent a diverse range of identities,
experiences, and voices within the mathematical community. Our views on data
science and social justice are necessarily shaped by our experiences of both
marginalization and privilege. Many of us embody marginalized identities along
axes of race, ethnicity, gender, and sexuality. For some of us, our interest
in justice work is informed by a passion to combat forms of oppression that we
ourselves have experienced. We also acknowledge that many of us occupy
positions of privilege within the mathematical community, including job
security, access to research funding, and recognition within our professional
communities. We look forward to continuing both our learning and our work
towards justice.

## Acknowledgements

This material is based upon work supported by the National Science Foundation
under Grant No. DMS-1929284 while some the authors were in residence at the
Institute for Computational and Experimental Research in Mathematics in
Providence, RI, during the Summer 2022 program on “Data Science and Social
Justice: Networks, Policy, and Education.” Part of this research was performed
while some of the authors were visiting the Institute for Mathematical and
Statistical Innovation (IMSI), which is supported by the National Science
Foundation (Grant No. DMS-1929348). Vindas-Meléndez is partially supported by
the National Science Foundation under Award DMS-2102921.

## References

  * [1] Nathan Alexander et al.  “Beyond Ethics: Considerations for Centering Equity-Minded Data Science”  In _Journal of Humanistic Mathematics_ 12.2, 2022, pp. 254–300
  * [2] Nathan N Alexander, Zeynep Teymuroglu and Carl R Yerger  “Critical conversations on social justice in undergraduate mathematics”  In _Mathematics for Social Justice_ Routledge, 2021, pp. 190–212
  * [3] Marcella Alsan and Marianne Wanamaker  “Tuskegee and the health of black men”  In _The Quarterly Journal of Economics_ 133.1 Oxford University Press, 2018, pp. 407–455
  * [4] Federico Ardila-Mantilla  “Todos cuentan: Cultivating diversity in combinatorics”  In _Notices of the AMS_ 63.10, 2016, pp. 1164–1170
  * [5] Amariah Becker, Moon Duchin, Dara Gold and Sam Hirsch  “Computational redistricting and the voting rights act”  In _Election Law Journal: Rules, Politics, and Policy_ 20.4, 2021, pp. 407–441
  * [6] Ron Buckmire et al.  “On definitions of “mathematician”” preprint (arXiv:2302.07432), 2023
  * [7] Joy Buolamwini and Timnit Gebru  “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification”  In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ 81, Proceedings of Machine Learning Research  PMLR, 2018, pp. 77–91
  * [8] W Castillo and D Gillborn  “How to “QuantCrit:” Practices and Questions for Education Data Researchers and Users”  In _EdWorkingPaper_ , Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/v5kh-dd65, 2022
  * [9] James I Charlton  “Nothing about us without us: Disability oppression and empowerment”  University of California Press, 2000
  * [10] Maria-Veronica Ciocanel et al.  “JUSTFAIR: Judicial system transparency through federal archive inferred records”  In _Plos One_ 15.10 Public Library of Science San Francisco, CA USA, 2020, pp. e0241381
  * [11] Carrie Diaz Eaton  “Teaching Machine Learning in the Context of Critical Quantitative Information Literacy”  In _Proceedings of the Second Teaching Machine Learning and Artificial Intelligence Workshop_ , 2022, pp. 51–56  PMLR
  * [12] Mica Estrada et al.  “Improving underrepresented minority student persistence in STEM”  In _CBE—Life Sciences Education_ 15.3 Am Soc Cell Biol, 2016, pp. es5
  * [13] Paulo Freire  “Pedagogy of the oppressed (revised)”  In _New York: Continuum_ 356, 1996, pp. 357–358
  * [14] Juan C Garibay  “STEM students’ social agency and views on working for social change: Are STEM disciplines developing socially and civically responsible students?”  In _Journal of Research in Science Teaching_ 52.5 Wiley Online Library, 2015, pp. 610–632
  * [15] Anita Ho  “Trusting experts and epistemic humility in disability”  In _IJFAB: International Journal of Feminist Approaches to Bioethics_ 4.2 Indiana University Press, 2011, pp. 102–123
  * [16] Max Horkheimer  “Critical Theory: Selected Essays”  A&C Black, 1972
  * [17] Gizem Karaali and Lily S Khadjavi  “Mathematics for social justice: Resources for the college classroom”  American Mathematical Soc., 2019
  * [18] Jacqueline Leonard, Wanda Brooks, Joy Barnes-Johnson and Robert Q Berry III  “The nuances and complexities of teaching mathematics for cultural relevance and social justice”  In _Journal of Teacher Education_ 61.3 Sage Publications Sage CA: Los Angeles, CA, 2010, pp. 261–270
  * [19] Ninareh Mehrabi et al.  “A survey on bias and fairness in machine learning”  In _ACM Computing Surveys (CSUR)_ 54.6 ACM New York, NY, USA, 2021, pp. 1–35
  * [20] Hugo Mercier and Dan Sperber  “Why do humans reason? Arguments for an argumentative theory”  In _Behavioral and Brain Sciences_ 34.2 Cambridge University Press, 2011, pp. 57–74
  * [21] Emma Pierson, Sam Corbett-Davies and Sharad Goel  “Fast threshold tests for detecting discrimination”  In _International Conference on Artificial Intelligence and Statistics_ , 2018, pp. 96–105  PMLR
  * [22] Emma Pierson et al.  “A large-scale analysis of racial disparities in police stops across the United States”  In _Nature Human Behaviour_ 4.7 Nature Publishing Group, 2020, pp. 736–745
  * [23] Govardana Sachithanandam Ramachandran, Ivan Brugere, Lav R Varshney and Caiming Xiong  “GAEA: Graph augmentation for equitable access via reinforcement learning”  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , 2021, pp. 884–894
  * [24] Lucía Santamaría and Helena Mihaljević  “Comparison and benchmark of name-to-gender inference services”  In _PeerJ Computer Science_ 4 PeerJ Inc., 2018, pp. e156
  * [25] Corey Seemiller and Meghan Grace  “Generation Z: Educating and engaging the next generation of students”  In _About Campus_ 22.3 SAGE Publications Sage CA: Los Angeles, CA, 2017, pp. 21–26
  * [26] Christian M Smith et al.  “Racial disparities in criminal sentencing vary considerably across federal judges”  In _SocArXiv (July 2021)_ , 2021
  * [27] Michael A Taffe and Nicholas W Gilpin  “Equity, Diversity and Inclusion: Racial inequity in grant funding from the US National Institutes of Health”  In _Elife_ 10 eLife Sciences Publications Limited, 2021, pp. e65697
  * [28] Melanie Tervalon and Jann Murray-Garcia  “Cultural humility versus cultural competence: A critical distinction in defining physician training outcomes in multicultural education”  In _Journal of Health Care for the Poor and Underserved_ 9.2 Johns Hopkins University Press, 1998, pp. 117–125
  * [29] Minh C Tran  “How can students be scientists and still be themselves: Understanding the intersectionality of science identity and multiple social identities through graduate student experiences”  University of California, Los Angeles, 2011
  * [30] K Hunter Wapman, Sam Zhang, Aaron Clauset and Daniel B Larremore  “Quantifying hierarchy and dynamics in US faculty hiring and retention”  In _Nature_ Nature Publishing Group, 2022, pp. 1–8
  * [31] Linda Zhao and Andrew V Papachristos  “Network position and police who shoot”  In _The Annals of the American Academy of Political and Social Science_ 687.1 SAGE Publications Sage CA: Los Angeles, CA, 2020, pp. 89–112


extra work on developers is likely to make them loose their interest in using
the tool often.

In the future, we will integrate the Small-Amp with a build system, for
instance GitHub Actions. The build systems will run the tool automatically on
the specified events such as on each push, or pull-request or periodically.
Additionally, Small-Amp will amplify only the recent changes on each run. It
means that only the mutants in the changed parts will be generated which will
reduce the cost of amplification significantly. In this case, the
amplification will be run in the project level instead of running class-by-
class. So, finding an exact relation between the production classes and the
test classes will also lost its importance: all test methods covering a
changed part can be included in the amplification.

Furthermore, we will build a web-based test editor dashboard to visualize the
test amplification outputs, and also a GitHub-Bot to synchronize the Build
system output, code base and the dashboard. The developer can use this
dashboard to assess the outputs, and also customize the amplified test. The
tests after the polishment will be reevaluated automatically, and if it is
green, it can be pushed to the code base. So, developers don’t need to
overwhelm themselves with tedious tasks and can benefit from test
amplification in an ecosystem automated by bots.

##### Extended use-cases for dynamic profiling.

Dynamic profiling is more than merely a type inference solution. It can be
generalized to collect various information about unit-under-test based on
dynamically running existing test suite. In some cases, statically typed
languages also can benefit the profiling mechanism. We enumerate two of these
use-cases:

  * •

Pure methods detection: An impure method is a method that looks like an
accessor but calling it causes a change in the state of the object (Sălcianu
and Rinard, 2005). In the scope of Small-Amp, identifying pure/impure methods
is important during oracle reduction. A new profiler can be implemented using
the method proxies to serialize the object state before and after method
invocations. If the state is changed, we can infer that the method is impure.

  * •

Providing information for advanced input amplifiers: In this paper we proposed
a basic algorithm for test-input reduction (Section 4.2). By addressing the
test-input explosion problem, test amplification tools can benefit from wider
range of input amplifiers. Advanced input-amplifiers can exploit the profiling
step to collect useful information dynamically and increase their knowledge
about the program under test. For example, a profiler can collect all literal
values from the covered methods and use them in literal values amplification
operator. Another example can be object transplantation between test methods.
A profiler can collect patterns of how objects are created and manipulated and
this information can be used in an input-amplifier.

##### Test method models, best practices and structured strings.

Unit tests in object-oriented languages ideally are structured as a sequence
of statements that instantiate an object, manipulates its state, and asserts
expected values. However, not every test fits this model in real projects.
Developers use helper methods, customized assertions, structured strings, some
optimizations like grouping the tests or parallelizing them. For instance, a
developer may write tests in XML files and load each file in a test method, so
these tests heavily depend on parsing structured strings.

If a test does not fit with the ideal test model, the current algorithm of
test amplification still can be used, but it may be less successful in
producing strong results. We leave identifying and adopting best practices of
test methods as an open problem. Additionally, mutating a structured string by
understanding its syntax can also be interesting future work.

##### Using patterns to guide test amplification.

As an important future work, we suggest using heuristics to guide the
amplification algorithm. Large scale manual test amplification histories like
the work at Google (Petrovic et al., 2021; Petrović and Ivanković, 2018) can
be analyzed to answer questions like: How often do developers write new test
methods? Are new test methods similar to an existing test? If they update an
existing test method, what is the relation of the updated test method and the
mutant to be killed? What transformations are applied to the test?

Answering these questions leads us to find some patterns in how real
developers kill mutants. These patterns can help the tool to prioritize some
test methods and input amplifiers for killing a particular mutant. Recent
advances in deep learning or other program synthesis techniques are promising
and can be helpful in making test amplification tools more intelligent (Abdi
et al., 2019a).

##### Reducing the mutation testing burden.

Test amplification generates tests to optimize mutation coverage, however
calculating the mutation score is a time-consuming process. During test
amplification, this mutation score is calculated multiple times for each test
method: in lines 6, and 14 of Algorithm 1 – p. 1. However, for a test to kill
a mutant, it first must reach the statement, then infect the program state,
propagate to the output where it must be revealed by an assertion (Lu et al.,
2020; Vera-Pérez et al., 2019).

We can optimize the calculation of the mutation score by using a hierarchical
coverage measurement. For example, we can first run a code coverage tool, then
we can run an extreme transformation (Vera-Pérez et al., 2018), afterwards we
only mutate the covered parts.

Another technique for reducing the mutation testing burden is to use mutation
operators that are learned from known common bugs like MutationMonkey (Beller
et al., 2021).

##### Using multiple type-inference mechanisms.

The main drawback of using an existing test suite for dynamic profiling is if
a method is not covered in the test, we can not collect its type information,
hence can not add calls to such methods during input amplification. Static
type inference (Pluquet et al., 2009) or live typing (Wilkinson, 2019)
techniques can be helpful to empower Small-Amp to generate method calls to
such uncovered methods.

##### Involving readability metrics.

Based on a previous study (Grano et al., 2020), the most important aspects for
developers in assessing the quality of a test suite are readability and
maintainability. Although the code coverage metrics are necessary for
identifying the poor test suites, they are limited in distinguishing high-
quality tests based on how practitioners perceive the test quality. Since the
goal of test amplification is to recommend new test cases ready to be merged
into the code base, considering readability and maintainability metrics in the
test amplification appears to be a critical next step.

##  9 Conclusion

In this paper, we introduce Small-Amp, an approach for test amplification in
the dynamically typed language Pharo/Smalltalk. The main algorithm of Small-
Amp is adapted from DSpot, a test amplification technique designed for Java
programs. In order to mitigate the lack of type information, we exploit
profiling information, readily available by running the test suite. We
demonstrate that test amplification is feasible for dynamically typed
languages, by replicating the experimental set-up of DSpot, including a
qualitative and quantitative analysis of the improved test suite.

In our qualitative analysis, we submitted pull-requests of an amplified test
by Small-Amp to the GitHub repositories of the projects in our dataset. From
11 pull-requests we submitted, 8 were merged ($\approx$72%). The developers’
comments on the pull-requests illustrate how valuable they perceive the new
tests created by Small-Amp. The results from our quantitative study show that
Small-Amp succeeds to amplify 28 test classes out of 52, approximately 53% of
target classes, in 13 projects from our dataset. The majority of the generated
tests are focused, and all test amplification steps (including type profiling
step) play a critical role. The results from Small-Amp and the results from
DSpot are comparable. We see $\approx$72% merged pull-requests and 53%
successfully amplified test classes in Small-Amp, while for DSpot these values
are 68% and 65%. We also see that the value of total increase killed between
two tools in two different ecosystems are similar (14% in Small-Amp and 20% in
DSpot).

In conclusion, the results of experiments show that by using a profiling step
and collecting type information, we can successfully adopt a test
amplification approach in a dynamically typed language.

###### Acknowledgements.

This work is supported by (a) the Fonds de la Recherche Scientifique-FNRS and
the Fonds Wetenschappelijk Onderzoek - Vlaanderen (FWO) under EOS Project
30446992 SECO-ASSIST (b) Flanders Make vzw, the strategic research centre for
the manufacturing industry. Bergel is very grateful to Lam Research and
Fondecyt Regular 1200067 for partially sponsoring this work. This work
originated during a sabbatical leave by Serge Demeyer in the RMOD lab.
Stéphane Ducasse, Marcus Denker and Julien Delplanque helped a lot with the
intricacies of Pharo/Smalltalk; Pavel Krivanek helped with MuTalk. Finally, we
thank the developers reviewing and discussing our pull requests, they really
helped us in improving Small-Amp.

## Declarations

### Funding and/or Conflicts of interests/Competing interests

This work is supported financially by two public funding organisation in
Belgium and Chile. There is no direct or indirect industrial support for the
research reported here. We confirm that there are no conflicts of interest.

### Research involving Human Participants and/or Animals

Not applicable

### Informed consent

Not applicable

## References

  * Abdi et al. (2019a) Abdi M, Rocha H, Demeyer S (2019a) Adopting program synthesis for test amplification. In: Proceedings of the 18th Belgium-Netherlands Software Evolution Workshop, Brussels, Belgium, published at http://ceur-ws.org, URL http://ceur-ws.org/Vol-2605/11.pdf
  * Abdi et al. (2019b) Abdi M, Rocha H, Demeyer S (2019b) Test amplification in the pharo smalltalk ecosystem. In: Proceedings of the 14th Edition of the International Workshop on Smalltalk Technologies, IWST, vol 19, pp 1–7
  * Agibalov (2015) Agibalov A (2015) What is a normal “functional lines of code” to “test lines of code” ratio? [on line] https://softwareengineering.stackexchange.com/questions/156883/ — last accessed In April 2021
  * Athanasiou et al. (2014) Athanasiou D, Nugroho A, Visser J, Zaidman A (2014) Test code quality and its relation to issue handling performance. IEEE Transactions on Software Engineering 40(11):1100–1125, DOI 10.1109/TSE.2014.2342227
  * Baudry et al. (2005) Baudry B, Fleurey F, Jézéquel JM, Le Traon Y (2005) From genetic to bacteriological algorithms for mutation-based testing. Software Testing, Verification and Reliability 15(2):73–96
  * Baudry et al. (2015) Baudry B, Allier S, Rodriguez-Cancio M, Monperrus M (2015) Dspot: Test amplification for automatic assessment of computational diversity. CoRR: a computing research repository abs/1503.05807, URL http://arxiv.org/abs/1503.05807, 1503.05807
  * Beller et al. (2021) Beller M, Wong CP, Bader J, Scott A, Machalica M, Chandra S, Meijer E (2021) What it would take to use mutation testing in industry—a study at facebook. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp 268–277, DOI 10.1109/ICSE-SEIP52600.2021.00036
  * Bergel et al. (2013) Bergel A, Cassou D, Ducasse S, Laval J (2013) Deep Into Pharo. Square Bracket Associates, URL http://books.pharo.org/deep-into-pharo/
  * Black et al. (2010) Black AP, Nierstrasz O, Ducasse S, Pollet D (2010) Pharo by example. Lulu. com
  * Brandt and Zaidman (2021) Brandt C, Zaidman A (2021) Developer-centric test amplification the interplay between automatic generation and human exploration. arXiv preprint arXiv:210812249
  * Chauvel et al. (2019) Chauvel F, Morin B, Garcia-Ceja E (2019) Amplifying integration tests with camp. In: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), pp 283–291, DOI 10.1109/ISSRE.2019.00036
  * Costiou et al. (2020) Costiou S, Aranega V, Denker M (2020) Sub-method, partial behavioral reflection with reflectivity: Looking back on 10 years of use. The Art, Science, and Engineering of Programming 4(3)
  * Danglot et al. (2019a) Danglot B, Vera-Perez O, Yu Z, Zaidman A, Monperrus M, Baudry B (2019a) A snowballing literature study on test amplification. Journal of Systems and Software 157:110398, DOI https://doi.org/10.1016/j.jss.2019.110398, URL http://www.sciencedirect.com/science/article/pii/S0164121219301736
  * Danglot et al. (2019b) Danglot B, Vera-Pérez OL, Baudry B, Monperrus M (2019b) Automatic test improvement with dspot: a study with ten mature open-source projects. Empirical Software Engineering, Springer Verlag
  * Daniel et al. (2009) Daniel B, Jagannath V, Dig D, Marinov D (2009) Reassert: Suggesting repairs for broken unit tests. In: Proceedings ASE 2009 (International Conference on Automated Software Engineering), IEEE CS, pp 433–444, DOI 10.1109/ASE.2009.17
  * Delplanque et al. (2019) Delplanque J, Ducasse S, Polito G, Black AP, Etien A (2019) Rotten green tests. In: 41th International Conference on Software Engineering, IEEE, ICSE ’19, pp 500–511, URL https://hal.inria.fr/hal-02002346
  * Denker et al. (2007) Denker M, Ducasse S, Lienhard A, Marschall P (2007) Sub-method reflection. Journal of Object Technology 6:275–295, DOI 10.5381/jot.2007.6.9.a14, URL http://www.jot.fm/contents/issue_2007_10/paper14.html
  * Ducasse (1999) Ducasse S (1999) Evaluating message passing control techniques in smalltalk. Journal of Object Oriented Programming 12:39–50
  * Ducasse (2019) Ducasse S (2019) Pharo with Style. Square Bracket Associates
  * Fischer et al. (2020) Fischer S, Haslinger EN, Zimmermann M, Thaller H (2020) An empirical evaluation for object initialization of member variables in unit testing. In: Proceedings VST 2020 (IEEE Workshop on Validation, Analysis and Evolution of Software Tests), pp 8–11, DOI 10.1109/VST50071.2020.9051634
  * Fraser and Arcuri (2012) Fraser G, Arcuri A (2012) The seed is strong: Seeding strategies in search-based software testing. In: 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp 121–130, DOI 10.1109/ICST.2012.92
  * Fraser and Arcuri (2013) Fraser G, Arcuri A (2013) Whole test suite generation. IEEE Transactions on Software Engineering 39(2):276–291, DOI 10.1109/TSE.2012.14
  * Fraser and Zeller (2011) Fraser G, Zeller A (2011) Generating parameterized unit tests. In: Proceedings of the 2011 International Symposium on Software Testing and Analysis, pp 364–374
  * Fraser and Zeller (2012) Fraser G, Zeller A (2012) Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering 38(2):278–292, DOI 10.1109/TSE.2011.93
  * Goldberg and Robson (1983) Goldberg A, Robson D (1983) Smalltalk-80: The Language and Its Implementation. Addison-Wesley Longman Publishing Co., Inc., USA
  * Grano et al. (2020) Grano G, De Iaco C, Palomba F, Gall HC (2020) Pizza versus pinsa: On the perception and measurability of unit test code quality. In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 336–347, DOI 10.1109/ICSME46990.2020.00040
  * Li and Offutt (2016) Li N, Offutt J (2016) Test oracle strategies for model-based testing. IEEE Transactions on Software Engineering 43(4):372–395, DOI 10.1109/TSE.2016.2597136
  * Lu et al. (2020) Lu ZX, Vercammen S, Demeyer S (2020) Semi-automatic test case expansion for mutation testing. In: 2020 IEEE Workshop on Validation, Analysis and Evolution of Software Tests (VST), IEEE, pp 1–7
  * Lukasczyk et al. (2020) Lukasczyk S, Kroiß F, Fraser G (2020) Automated unit test generation for python. In: Aleti A, Panichella A (eds) Search-Based Software Engineering, Springer International Publishing, Cham, pp 9–24
  * Mairhofer et al. (2011) Mairhofer S, Feldt R, Torkar R (2011) Search-based software testing and test data generation for a dynamic programming language. In: Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, Association for Computing Machinery, New York, NY, USA, GECCO ’11, p 1859–1866, DOI 10.1145/2001576.2001826, URL https://doi.org/10.1145/2001576.2001826
  * Milani Fard et al. (2014) Milani Fard A, Mirzaaghaei M, Mesbah A (2014) Leveraging existing tests in automated test generation for web applications. In: Proceedings of the 29th ACM/IEEE international conference on Automated software engineering, pp 67–78
  * Mirshokraie et al. (2015) Mirshokraie S, Mesbah A, Pattabiraman K (2015) Jseft: Automated javascript unit test generation. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp 1–10, DOI 10.1109/ICST.2015.7102595
  * Nijkamp et al. (2021) Nijkamp N, Brandt C, Zaidman A (2021) Naming amplified tests based on improved coverage. In: 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM), IEEE, pp 237–241
  * Oosterbroek et al. (2021) Oosterbroek W, Brandt C, Zaidman A (2021) Removing redundant statements in amplified test cases. In: 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM), IEEE, pp 242–246
  * Pacheco et al. (2007) Pacheco C, Lahiri SK, Ernst MD, Ball T (2007) Feedback-directed random test generation. In: 29th International Conference on Software Engineering (ICSE’07), pp 75–84, DOI 10.1109/ICSE.2007.37
  * Panichella et al. (2018) Panichella A, Kifetew FM, Tonella P (2018) Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering 44(2):122–158, DOI 10.1109/TSE.2017.2663435
  * Papadakis et al. (2019) Papadakis M, Kintis M, Zhang J, Jia Y, Traon YL, Harman M (2019) Mutation testing advances: An analysis and survey. Advances in Computers 112:275–378, DOI 10.1016/bs.adcom.2018.03.015, URL http://www.sciencedirect.com/science/article/pii/S0065245818300305
  * Patrick and Jia (2017) Patrick M, Jia Y (2017) Kd-art: Should we intensify or diversify tests to kill mutants? Information and Software Technology 81:36–51
  * Peck et al. (2015) Peck MM, Bouraqadi N, Fabresse L, Denker M, Teruel C (2015) Ghost: A uniform and general-purpose proxy implementation. Science of Computer Programming 98:339–359
  * Petke et al. (2018) Petke J, Haraldsson SO, Harman M, Langdon WB, White DR, Woodward JR (2018) Genetic improvement of software: A comprehensive survey. IEEE Transactions on Evolutionary Computation 22(3):415–432, DOI 10.1109/TEVC.2017.2693219
  * Petrović and Ivanković (2018) Petrović G, Ivanković M (2018) State of mutation testing at google. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, Association for Computing Machinery, New York, NY, USA, ICSE-SEIP ’18, p 163–171, DOI 10.1145/3183519.3183521, URL https://doi.org/10.1145/3183519.3183521
  * Petrovic et al. (2021) Petrovic G, Ivankovic M, Fraser G, Just R (2021) Practical mutation testing at scale: A view from google. IEEE Transactions on Software Engineering pp 1–1, DOI 10.1109/TSE.2021.3107634
  * Pezze et al. (2013) Pezze M, Rubinov K, Wuttke J (2013) Generating effective integration test cases from unit ones. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, IEEE, pp 11–20
  * Pluquet et al. (2009) Pluquet F, Marot A, Wuyts R (2009) Fast type reconstruction for dynamically typed programming languages. In: Noble J (ed) Proceedings of the 5th Symposium on Dynamic Languages, DLS 2009, October 26, 2010, Orlando, Florida, USA, ACM, pp 69–78, DOI 10.1145/1640134.1640145, URL https://doi.org/10.1145/1640134.1640145
  * Rö$\beta$ler et al. (2012) Rö$\beta$ler J, Fraser G, Zeller A, Orso A (2012) Isolating failure causes through test case generation. In: Proceedings of the 2012 international symposium on software testing and analysis, pp 309–319
  * Rojas et al. (2016) Rojas JM, Fraser G, Arcuri A (2016) Seeding strategies in search-based unit test generation. Software Testing, Verification and Reliability 26(5):366–401
  * Sălcianu and Rinard (2005) Sălcianu A, Rinard M (2005) Purity and side effect analysis for java programs. In: International Workshop on Verification, Model Checking, and Abstract Interpretation, Springer, pp 199–215
  * Thummalapenta et al. (2011) Thummalapenta S, Marri MR, Xie T, Tillmann N, De Halleux J (2011) Retrofitting unit tests for parameterized unit testing. In: International Conference on Fundamental Approaches to Software Engineering, Springer, pp 294–309
  * Tillmann and Schulte (2006) Tillmann N, Schulte W (2006) Unit tests reloaded: Parameterized unit testing with symbolic execution. IEEE Software 23(4), DOI 10.1109/MS.2006.117
  * Tonella (2004) Tonella P (2004) Evolutionary testing of classes. Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis - ISSTA ’04 DOI 10.1145/1007512.1007528, URL http://dx.doi.org/10.1145/1007512.1007528
  * Vera-Pérez et al. (2019) Vera-Pérez OL, Danglot B, Monperrus M, Baudry B (2019) Suggestions on test suite improvements with automatic infection and propagation analysis. arXiv preprint arXiv:190904770
  * Vera-Pérez et al. (2018) Vera-Pérez OL, Monperrus M, Baudry B (2018) Descartes: A pitest engine to detect pseudo-tested methods: Tool demonstration. In: 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 908–911, DOI 10.1145/3238147.3240474
  * Wibowo et al. (2015) Wibowo JTP, Hendradjaya B, Widyani Y (2015) Unit test code generator for lua programming language. In: 2015 International Conference on Data and Software Engineering (ICoDSE), pp 241–245, DOI 10.1109/ICODSE.2015.7437005
  * Wilkinson (2019) Wilkinson H (2019) Vm support for live typing: Automatic type annotation for dynamically typed languages. In: Proceedings of the Conference Companion of the 3rd International Conference on Art, Science, and Engineering of Programming, Association for Computing Machinery, New York, NY, USA, Programming ’19, DOI 10.1145/3328433.3328443, URL https://doi.org/10.1145/3328433.3328443
  * Wilkinson et al. (2009) Wilkinson H, Chillo N, Brunstein G (2009) Mutation testing. European Smalltalk User Group (ESUG 09). Brest, France. http://www.esug.org/data/ESUG2009/Friday/Mutation_Testing.pdf
  * Wohlin et al. (2000) Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in Software Engineering. Kluwer Academic Publishers, DOI 10.1007/978-3-642-29044-2
  * Xie (2006) Xie T (2006) Augmenting automatically generated unit-test suites with regression oracle checking. Lecture Notes in Computer Science pp 380–403, DOI 10.1007/11785477_23, URL http://dx.doi.org/10.1007/11785477_23
  * Xu and Rothermel (2009) Xu Z, Rothermel G (2009) Directed test suite augmentation. In: 2009 16th Asia-Pacific Software Engineering Conference, IEEE, pp 406–413
  * Xu et al. (2010) Xu Z, Cohen MB, Rothermel G (2010) Factors affecting the use of genetic algorithms in test suite augmentation. In: Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp 1365–1372
  * Xuan et al. (2015) Xuan J, Xie X, Monperrus M (2015) Crash reproduction via test case mutation: let existing test cases help. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp 910–913
  * Yoo and Harman (2012) Yoo S, Harman M (2012) Test data regeneration: generating new test data from existing test data. Software Testing, Verification and Reliability 22(3):171–201
  * Yoshida et al. (2016) Yoshida H, Tokumoto S, Prasad MR, Ghosh I, Uehara T (2016) Fsx: fine-grained incremental unit test generation for c/c++ programs. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, pp 106–117
  * Zaidman et al. (2011) Zaidman A, Rompaey BV, van Deursen A, Demeyer S (2011) Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. International Journal on Empirical Software Engineering 16(3):325–364, DOI 10.1007/s10664-010-9143-7