# The Mathematics of Mathematics: Using Mathematics and Data Science to
Analyze the Mathematical Sciences Community and Enhance Social Justice

Ron Buckmire  Joseph E. Hibdon, Jr  Drew Lewis  Omayra Ortega  José L. Pabón
Rachel Roca  Andrés R. Vindas-Meléndez

###### Abstract

We present and discuss a curated selection of recent literature related to the
application of quantitative techniques, tools, and topics from mathematics and
data science that have been used to analyze the mathematical sciences
community. We engage in this project with a focus on including research that
highlights, documents, or quantifies (in)equities that exist in the
mathematical sciences, specifically, and STEM (science, technology,
engineering, and mathematics) more broadly. We seek to enhance social justice
in the mathematics and data science communities by providing numerous examples
of the ways in which the mathematical sciences fails to meet standards of
equity, equal opportunity and inclusion. We introduce the term “mathematics of
Mathematics” for this project, explicitly building upon the growing,
interdisciplinary field known as “Science of Science” to interrogate,
investigate, and identify the nature of the mathematical sciences itself. We
aim to promote, provide, and posit sources of productive collaborations and we
invite interested researchers to contribute to this developing body of work.

##  1 Introduction

###  1.1 Motivation

Mathematics has long been used to solve problems, document patterns, describe
phenomena, explain observations, and provide insight into how the world works.
In the last century, with the advent of the computing age and its concomitant
generation and storage of vast quantities of data, the ability to analyze data
has become another important tool of mathematicians, engineers, and
scientists. The context for what kinds of problems mathematics and data
science can be used to solve are extremely varied, running the gamut from
those that are theoretical with no currently foreseen applications to those
that are immediately applicable to real-world situations. In this paper we are
interested in how tools, topics, and techniques from the mathematical sciences
can be and have been used to analyze the mathematical sciences itself. We
introduce here the phrase “the mathematics of Mathematics” (or #MetaMath, for
short) to refer to this growing body of work using mathematics to analyze
Mathematics. Note that we are using the lower-case term “mathematics” in this
phrase as shorthand for quantitative and qualitative techniques commonly
associated with the mathematical sciences, and the upper-case term
“Mathematics” to mean the people, practices, and organizational structures
associated with the profession of mathematics. In particular, we seek to
highlight that mathematics is done by people who exist as part of a larger
community, not in a vaccuum. We chose the term “mathematics of Mathematics”
specifically as an allusion to the growing field of “science of Science”111We
note that despite the similarity in nomenclature, neither of these terms have
any association with the so-called “science of math” movement, a reactionary
movement in K-12 education that rejects many equity-minded reforms that are
based on mathematics education research. We further note that #metamath has no
connection to metamathematics which is a branch of mathematical logic.[25, 59,
62], which shares many of our goals.

###  1.2 Distinguishing Mathematics of Mathematics (#MetaMath) from Science of
Science (SciSci) and Other Similar Disciplines

The science of science (SciSci) is a well-developed field of study in which
the tools and techniques of scientific exploration are used to investigate and
interrogate science itself. For example, Fortunato et al. [25] say

> SciSci places the practice of science itself under the microscope, leading
> to a quantitative understanding of the genesis of scientific discovery,
> creativity, and practice and developing tools and policies aimed at
> accelerating scientific progress.
>
>
> […]
>
>
> SciSci relies on a broad collection of quantitative methods, from
> descriptive statistics and data visualization to advanced econometric
> methods, network science approaches, machine-learning algorithms,
> mathematical analysis, and computer simulation, including agent-based
> modeling.

We intend the mathematics of Mathematics (#MetaMath) to take advantage of the
increased availability of data about different aspects of the Mathematics
community, such as public information about research publications, citation
patterns, faculty hiring, racial, ethnic, and gender demographics, doctoral
degree advisor-advisee relationships, research funding, and more, in order to
explicitly advance the social justice goal of a more equitable, diverse, and
inclusive Mathematics community, profession, and enterprise. (We note that
just as the science of science includes qualitative techniques we intend for
the mathematics of Mathematics to do so as well.)

###  1.3 Context and Goals For This Paper

This review paper is written in the context of recent activity in how data
science and mathematics are being used to enhance social justice in various
new and exciting ways. For example, we envision #MetaMath as a _quantitative
justice_ project, where the term “quantitative justice” is defined as “the
application of techniques, tools and topics from the quantitative sciences
(e.g., mathematics, applied mathematics, data science, computer science, etc.)
in subject domains that are derived from and/or typically associated with the
social sciences (e.g., history, political science, law, economics, sociology
etc.) with the explicit goal of promoting social justice” [8, 6, 10, 9, 7] .

Talks in the developing area of quantitative justice have occurred at multiple
minisymposia [12] and special sessions [13] at several widely attended
international STEM conferences in the last two years. Examples include the
2022 annual meeting of the Society for Industrial and Applied Mathematics
(SIAM) in Pittsburgh, PA [6], the 2023 SIAM Computer Science and Engineering
Conference in Amsterdam, the Netherlands [9], the 2023 annual meeting of the
American Association for the Advancement of Science (AAAS) in Washington, DC
[8], and the 2023 Joint Mathematics Meetings in Boston, MA [4, 7, 10].

In addition to being described as a quantitative justice project, #MetaMath
also meets the definition of Data Science for Social Justice (DS4SJ) provided
in [30] as “data scientific work (broadly construed) that actively challenges
systems of inequity and concretely supports the liberation of oppressed and
marginalized communities.” Regardless of which term one uses to describe the
#MetaMath project (quantitative justice, data science for social justice,
justice data science, or something else), its goal is the same as this paper:
to enhance social justice by using the mathematical sciences to investigate,
interrogate, and identify inequities in the mathematical sciences itself.

In the main body of the paper, we provide a survey of selected recent work
that uses tools, topics, and techniques from mathematics and data sciences to
describe, document, and display inequities in the mathematics and data science
communities. In the rest of the paper, we discuss some methodological
considerations for quantitative scientists and mathematicians interested in
working in the field, as well as some future directions for research.

##  2 Survey of Existing #MetaMath Research

In this section we will present our commentary on various examples of articles
that have appeared in the research literature over the last two decades that
can be characterized with a #MetaMath label, i.e. they use tools, topics, or
techniques from mathematics and data science to analyze various aspects of
discrimination, disparity, and difference in the mathematics community
specifically, or the STEM academic community, generally. We note that our
presentation is not an exhaustive list or review of all such examples, since
we are particularly interested in providing evidence that describes, displays,
or documents inequity in the mathematics community.

If one believes the saying, as we do, that “Talent is evenly distributed, but
opportunity is not,” then the research presented below bolsters that view. It
shows that there exist hierarchical structures in mathematics, evidence of
elitism, disparities in distribution of resources, over-representation of
certain favored groups, under-representation of other disfavored groups, and a
paucity of data on mathematics curriculum. We hope by highlighting #metamath
research that uses data to documents these deficiencies we will encourage
others to enhance social justice in the mathematical sciences community by
enacting positive change.

In the subsequent subsections we provide several examples of peer-reviewed
articles that use mathematics and data science to demonstrate these phenomena
exist in STEM broadly and mathematics specifically. We note that these
phenomena are interrelated and interdependent so it can be difficult to assign
a particular paper or result to only one of these overlapping areas. For
example, research by Liu et al [34] that shows that women and racial/ethnic
minorities receive fewer citations to their research in mathematics and
scientific journals demonstrates the lack of diversity in these fields, but
also the existence of a hierarchy based on race, ethnicity and gender in
academic publishing as well as be indicative of elitism in this area. In cases
like this the same research may be referred to in multiple parts of the paper.

###  2.1 Diversity and Demographics of the Mathematics Community

We believe it is self-evident that mathematicians, i.e. the members of the
mathematics community in the United States (or however one defines the term
[11]) do not represent the full spectrum of the diversity and demographics of
the general population. In other words, there are groups that are currently
under-represented and historically marginalized in the mathematics community
[5].

This under-representation of women, racial and ethnic minorities, people whose
parents or grandparents didn’t go to college (i.e., first-generation), people
living with disabilities and members of the LGBTQ community has been
documented and analyzed in many STEM fields, including in mathematics, and in
different professional activities in academia. The sectors of academia
activity in which this under-representation manifests is varied and near-
ubiquitous. In other words, it is difficult to find an aspect of academia in
which the diversity and demographics of those who participate in that aspect
match the corresponding racial, ethnic and gender breakdowns of the general
population of the United States.

Data on the demographics of the STEM community can be found in documents
created by the National Center for Education Statistics in the U.S. Department
of Education (NCES) [21] and the National Science Foundation’s National Center
for Science and Engineering Statistics (NCSES) [24]. For math-specific data,
the quinquennial survey conducted every five years since 1970 by the
Conference Board of the Mathematical Sciences (CBMS) and the annual survey of
earned doctorates by the American Mathematical Society (AMS) are key data
sources11footnotetext: The CBMS survey can be found at
https://www.ams.org/profession/data/annual-survey/annual-survey and the Annual
Survey of Earned Doctorates can be found at
https://www.ams.org/profession/data/annual-survey/annual-survey. Due to the
COVID-19 pandemic, updates of these data sources has been significantly
delayed.

Buckmire [5] notes that between 2013 to 2018, women made up approximately
42.6% of recipients of bachelor’s degrees in Mathematics. In the same time
period, the racial and ethnic makeup of undergraduate math graduates ranged
from 64.9% White, 7.5% Latino/Hispanic, and 5.0% Black to 52.5% White, 9.9%
Latino/Hispanic and 4.2% Black.

Medina [36] notes that between 1993 and 2002, less than 5% of those who earned
doctoral degrees in mathematics were Black, Latinx, or Indigenous, despite
those communities composing a quarter of the US population at that time.
Martin [35] compiled a comprehensive list of 95 citations focusing on women in
mathematics in his annotated bibliography, covering biases and inequities in
primary school to tenure. Cech [15] found that LGBTQ+ scientists and
mathematicians were more likely to report social marginalization, work place
harassment, as well as limited career opportunities.

Topaz et al. [57] analyzed 13,067 editorial positions on the boards of 435
mathematical science journals and found that roughly 8.9% of editorships were
held by women. More recently, Liu et al. [34] expanded on this analysis in
multiple dimensions. They looked at over one million papers between 2001 to
2020 in over 500 journals processed by almost 65,000 editors and found that
“non-white scientists appear on fewer editorial boards, spend more time under
review, and receive fewer citations” than White scientists.

In this subsection we have presented examples of data that documents how the
demographics of the mathematics (and wider STEM) community do not reflect the
diversity of the larger population from which it draws. In the next
subsection, we will present examples of research that addresses the similar
but distinct problem that mathematics (and STEM) contain hierarchical
structures that reflect and reproduce the lopsided demographic distributions
previously discussed above.

###  2.2 The Myth of Meritocracy and Existence of Hierarchy in Mathematics and
Science

Here we present examples of published research articles that can be used to
prove the existence of hierarchical structures in mathematics (and STEM), thus
refuting the pernicious claim that mathematics (or STEM) is a meritocracy.
Examples of hierarchical structures [32] could include the existence of
institutions which have greater prestige [42] or are more likely to have
students who are more likely to go on to obtain faculty positions at more
prestigious institutions.

In the last few years, several researchers have taken advantage of the
availability of vast quantities of data on faculty positions at institutions
of higher education in the United States to document the existence of
hierarchies in faculty hiring networks in academia. Clauset et al. [18]
demonstrated the existence of hierarchy in faculty hiring in a study involving
19,000 faculty members in a total of 461 departments in computer science,
business, and history. Wapman et al. [60] expanded this analysis to cover
295,089 faculty in 10,612 departments at 368 Ph.D.-granting institutions and
all academic disciplines for the 2011-2020 decade. FitzGerald et al. [23]
built upon this research by using data from the Mathematics Genealogy Project
(MGP) to restrict their analysis to mathematics faculty. They analyzed 120,000
records from 150 institutions over seven decades (from 1950 to 2019). The
results of this research demonstrates that who becomes faculty at
Ph.D.-granting institutions is not an unbiased process or reflective of a
meritocratic ideal. Instead, analysis of the availability data shows that
there exist certain institutions that are more likely than others to have
their Ph.D. graduates become faculty members at other Ph.D.-granting
institutions. In other words, the data demonstrates that hierarchies exist in
faculty hiring networks.

One of the most salient and persistent hierarchical structures in society that
is omnipresent in mathematics and the wider STEM community is gender.
Researchers have analyzed data describing many different aspects of academic
activity and demonstrated the many ways gender can negatively mediate
opportunity for advancement, participation, and achievement in science and
mathematics [3, 31, 35, 38, 39, 48, 52, 57, 61].

Another significant hierarchical structure in society is class. Morgan et al
[41] analyzed data from a survey with responses from more than 7,000 faculty
in eight STEM disciplines to demonstrate that faculty are 25 times more likely
to have a parent with a Ph.D. and that at “elite” institutions this rate is
doubled. This data analysis demonstrates the salience of socioeconomic status
on the composition of the professoriate.

In this subsection, we have provided articles that confirm the existence of
hierarchical structures in mathematics and science. Our argument is that these
research results using data demonstrate the idea that mathematics (and STEM,
more generally) are egalitarian and meritocratic is a myth that can be and has
been disproved, debunked, and discredited.

###  2.3 Evidence of Elitism in Mathematics

Our survey of existing results finds extensive evidence, gathered via
mathematical tools and techniques, of widespread elitism and exclusion of
minorities and under-represented populations in the mathematics community [16,
32, 39, 51, 57]. For example, Topaz et al. found in 2016 that in academia,
women comprised only 15% of tenure-stream faculty positions in doctoral-
granting mathematical sciences departments in the United States. Editorial
appointments in the journals studied in their work found even more paltry
representation for women; only 8.9% of editorships within the journals
reviewed belonged to women [56, 57].

Another way elitism is perpetuated in the mathematics community is via the
selection of a self-perpetuating cadre of “elite” personnel for prestigious
prizes. The International Mathematical Union’s selections for the Fields
Medal, (widely considered the most elite prize in Mathematics) for example,
have shown an alarming trend of rewarding only already-elite mathematics
researchers in an apparent negation of one of the main goals of its conferral,
which was to shine light on under-represented members of the mathematics
community [16]. Other scientists have found widespread elitism perpetuates as
a vicious circle of like begetting like [51, 53].

Schlenker [51] notes that fields with applications to the social or physical
sciences such as numerical analysis, mathematical modeling or statistics seem
to be viewed as having low status, and this lack of prestige accompanies the
low representation of researchers in these fields among elite prizewinners.
However, other fields such as Algebraic Geometry have no shortage of
practitioners and medalists, in spite of having little direct application to
other disciplines, once again displaying the self-serving cycle of elitism
within the mathematics community.

Other researchers have also used other mathematical tools to identify inequity
in mathematics in other aspects of academia. Leslie et al. [33] and Storage et
al. [53] found widespread under-representation of women and African Americans
in their research which analyzed over 14 million records from the popular
website RateMyProfessor. Their findings included detailing that in fields
where brilliance and innate intellectual talent were treasured, African
Americans and women were even less well represented. Brisbin and Whitcher [3]
concur with the above research by finding that women are under-represented in
subfields of mathematics that are viewed as having more prestige by analyzing
almost a million papers in the mathematical sciences uploaded to the arXiv, a
popular website for the dissemination of research results. Way et al. [61]
studied comprehensive data on both hiring outcomes and scholarly productivity
for 2659 tenure-track faculty across 205 Ph.D.-granting departments in North
America and found deep under-representation of women in these groups.
Curiously, Way et al. also found pervasive elitism including discrimination
against females, as well as some other less prominent examples of biases
against women. One of the salient trends that these researchers found was that
top-ranked institutions had tendencies to hire women at higher rates than
their mathematical data science enabled models would expect. In this sense,
higher ranked academic institutions did a “better job” of being less
discriminating against women. This was also the case at the lower end of the
spectrum of lower ranked academic institutions; thus the majority of hiring
and appointing of women in academia take place in the middle ranked academic
institutions.

The research highlighted in this subsection has provided examples of the
pervasive elitism across academia at large and the mathematics community in
particular.

###  2.4 Disparities in Federal Funding for Science and Mathematics

Another example of #MetaMath research involves analyzing the public data of
who receives funding from public and private sources in order to detect and
document disparities in support for mathematics and science in the United
States. In this subsection we provide examples of research that provides
evidence of these discriminatory results. It is well documented that there are
racial disparities in funding from the National Institutes of Health (NIH)
[17, 26, 48, 52, 54]. The 2011 study of NIH R01 funding by Ginther et al. [26]
hypothesized that scientists of different races but with similar research
records and institutional affiliations would also have a similar likelihood of
being awarded research grants. However, the opposite was discovered. They
found that Black and Asian investigators are less likely to be awarded an R01
on the first or second attempt, Blacks and Hispanics are less likely to
resubmit a revised application, and Black investigators that do resubmit have
to do so more often to receive an award. Even after controlling for the
applicant’s educational background, country of origin, training, previous
research awards, publication record, and employer characteristics, the authors
discovered that black applicants remain 10 percentage points less likely than
whites to be awarded NIH research funding.

Funding from the National Science Foundation (NSF) have not been as
extensively studied as the NIH. As the main source for research funding in the
mathematical sciences, the funding priorities selected by the NSF will greatly
influence how the fields in the mathematical sciences will grow, behave, and
look. Both the NIH and NSF are federal funding agencies with similar missions
but different foci. As such, we would not expect to see trends in NSF funding
differ significantly from the NIH. Indeed, Chen et al. [17] recently examined
NSF funding data from 1996 to 2019 and found that proposals by white PIs were
consistently funded at rates higher than the overall rate, with an average
relative funding rate of +8.5% from 1999 to 2019. Additionally, the relative
funding rate for proposals by white PIs increased steadily during this period,
from +2.8% in 1999 to +14.3% in 2019. Proposals by Asian, Black/African-
American, and Native Hawaiian/Pacific Islander PIs, were consistently funded
below the overall rate, with average relative funding rates of –21.2%, –8.1%,
and –11.3%, respectively.

Additionally, these findings revealing race-based and gender-based funding
disparities are not confined to federal agencies. The Wellcome Trust is a
private global charitable foundation that conducted a similar study on their
own award practices in 2020 [58]. The Wellcome study stated that, “As a
funder, we know from our data that BAME222Black, Asian, and minority ethnic,
also known as “BAME,” is an umbrella term, common in the United Kingdom, used
to describe non-white ethnicities., and especially Black, applicants are less
likely to be awarded Wellcome research grants in the UK than White
applicants.” At a time when the mathematics community has been doubling down
on its efforts to diversify the profession, the type of groundbreaking data
analysis presented in this subsection is absolutely necessary to motivate
action and meaningful change to eliminate racial and gender disparities in
funding in the mathematical sciences community.

###  2.5 Discourse in the Mathematics Community

Just like SciSci, #MetaMath also includes the use of qualitative techniques.
In this subsection we provide some examples of the ways that discourse in and
about the mathematics community can be analyzed, primarily through mixed or
qualitative methods, in order to raise awareness about social justice issues
in the mathematical sciences.

The mathematics community shares its opinions and responses to given
situations in a variety of public ways, often leaving behind artifacts that
can be studied. Examples of such dissemination of opinions and responses
include the Inclusion/Exclusion blog [29], a justice and mathematics weblog
that is a continuation of the AMS blog archived in 2021; an article in
Scientific American highlighting the debate surrounding the founding of the
Association for Mathematical Research [20]; and the statement by the
Mathematical Association of America (MAA) on the location of Mathfest 2023
[43], in which they prioritize financial concerns over the safety of the
LGBTQ+ community at the meeting site in a state that has enacted
discriminatory legislation targeting LGBTQ+ people. The artifacts left behind
in these cases include the blog, the article and the statement as well as any
public replies to these on social media, statements from organizations,
letters to the editor, and more. By examining the conversation through these
artifacts, we can also better understand and quantify the divisions and
counter-movements within the mathematics community. It is worth highlighting
that this type of work often requires mixed-methodology; strictly quantitative
analysis cannot fully capture nuance and “whys” behind the discourse.

One of the first questions we might ask ourselves in this work is who is part
of the conversation, which can be answered quantitatively. The work of Topaz
et al. [55] compares the demographics of the hundreds of signatories on three
public letters responding to an essay published in Notices of the American
Mathematical Society that opposed the use of diversity statements in academic
hiring, likening them to “McCarthyism.” The results showed that signatories of
Letter A, which highlights diversity and social justice, were inferred to be
more diverse in terms of gender, under-represented ethnic groups, professional
security, and employment at academic institutions. Letters B and C, which did
not mention diversity and spoke against diversity statements, respectively,
were instead signed by those who were inferred to be majority tenured white
men at research focused universities. The results presented in this paper are
consistent with theories of power and positionality found in social sciences
[37, 40].

From a more qualitative perspective, Burrill et al. [14] identified broad
themes through recordings from a math education forum, consolidating and
sharing common ground across diverse experts. Their work highlights the
importance of communication and articulation among stakeholders, as well as
the need to support students and teachers in curricular design. Thematic
qualitative analysis can also be executed in a more fine-grained manner, such
as through the examination of Twitter posts containing the #disruptJMM hashtag
by Roca et al. [49]. This paper examines the math social movement born from
Piper H’s post on the Inclusion/Exclusion Blog calling for more commonplace
discussion on inclusion and equity within the Joint Mathematics Meeting (JMM)
under the #disruptJMM banner. Those who used the hashtag and challenged others
to get involved built a community that highlighted the need to rehumanize
mathematics, make visible power and privilege in the mathematics community,
and elevate the stories of under-represented folks at the conference.

The examples of the ways discourse in the mathematics community can be studied
given in this subsection allows us to identify salient topics of conversation,
understand who is included within or excluded from these discussions, and
provide context to situations affecting the heterogeneous mathematics
community in disproportionate way.

###  2.6 Quantifying Diversity in Mathematical Content and Curricular Choices

This subsection is motivated by two questions. First, _what courses are
mathematics departments currently implementing in their undergraduate
programs_? Second, _how are mathematics departments using data to make
curricular choices?_ The reality is that there is limited data available to
properly answer these questions.

Every five years since 1970, CBMS sponsors a comprehensive national survey of
undergraduate mathematical and statistical sciences in the United States’
four-year and two-year universities and colleges. The national data collected
includes statistics on enrollment, curriculum, degrees awarded, course
availability, faculty demographics, and special one-time topics (these topics
vary) [2]. The most recent CBMS survey, whose complete results are published,
is from 2015. (The release of data from the 2020 survey has been delayed due
to the COVID-19 pandemic.) This survey showed that among the 28 upper-level
math courses listed, the courses that were most offered (at least once) in the
academic years 2014-2016 and 2015-2016 were Modern Algebra I (78% of all math
departments), Geometry (71%), Advanced Calculus/Real Analysis I (72%), Math
Senior Seminar/Independent study (66%), History of Mathematics (58%), and
Introduction to Proofs (56%).

Although there are guidelines written by leading mathematical societies on
what the mathematics undergraduate curriculum should consist of (see for
instance [19, 46, 22, 27, 44, 45, 50]), the CBMS survey demonstrates that
there is not a consistent vision for the mathematics curriculum among the
institutional respondents. Additionally, we must take into consideration the
limitations of such surveys/data, such as survey response rates. For example,
among the four-year mathematics respondents of the 2015 CBMS survey, only 5
out of the 23 California State University System (CSU) campuses and 4 out of
the 9 University of California (UC) System undergraduate campuses
participated. Combined, both the CSU and UC systems would create the largest
public institution of higher education in the United States, thus there is a
large number of students and departments not voiced in the survey results.

This leads us to the following question: _why should there be data on
mathematics course offerings at different institutions?_ Answering this
question provides an avenue for at least two data science projects that can
benefit the broader mathematical community. One such project would be to
actually collect the data on course offerings at all institutions with an
undergraduate mathematics major. This may involve applying data science
techniques, such as data/web-scraping, information retrieval, and/or data
clustering. Having data on mathematics course offerings at different
institutions can be used to influence funding, administrative, and educational
practices. As far as we know, there is also no known information or literature
on how mathematics departments are using quantitative data as an input when a
department or faculty chooses math content and curricula. Hence, a second
project is to pursue a study towards understanding the decision-making
processes of mathematics departments in regards to the aforementioned setting.
Such a study can have implications for mathematical practice and education and
can help interested parties to identify related important and timely matters,
such as textbook selection for courses.

A deeper dive into the content of mathematics courses would inform the
question, _what is mathematics?_ This question is explored through the
Rehumanizing Mathematics framework [28], developed by Dr. Rochelle Gutierrez,
and can have implications for future data science inquiry. In this framework,
there are eight dimensions; participation/positioning; cultures/histories;
windows/mirrors; living practice; broadening maths; creation; body/emotions;
and ownership. These dimensions can be implemented in the process of
rehumanizing the mathematics classroom. This framework has also been used in
conjunction with lesson study to rehumanize the content of mathematics courses
by bringing back diverse mathematics history that is often left out of
mathematics courses, including and incorporating students’ existing knowledge
and experience into the classroom, and by considering alternative formats for
conducting and communicating mathematics (for example). Thus far, no one has
quantified and categorized the content offered in mathematics courses. This
inquiry could be further expanded to identify geographical trends in the types
of mathematics content offered.

Decisions about who gets admitted to college, particularly to elite or
selective institutions, are the result of a combination of policy and
practice, and math expectations play a role [1]. Strongly-held beliefs about
calculus as a sign of rigor play a consequential role in college admissions.
This can also translate to upper-division mathematics courses and to graduate
admissions. Data, such as that presented in the CBMS survey or potential
results from the data science projects proposed above, should be carefully
used and analyzed. For example, graduate admission committees when reviewing a
student’s application should take into consideration questions such as: Did
the student have opportunities to enroll in graduate courses as an
undergraduate student? Did the student study at small college or large
university? Did the student participate in research or an internship? Did the
student have the opportunity to pursue an undergraduate thesis? The call in
this subsection for the ability to quantify the diversity of mathematical
content and curricular choices at undergraduate institutions could lead to
more nuanced admissions decisions at graduate schools and more effective
undergraduate mathematics curricula with a possibility of a future mathematics
community that has been positively impacted by the existence and analysis of
such data.

##  3 Methodological Considerations

As members of the mathematical community embark on work in #MetaMath, we
caution them to proceed with humility. Much of this work, while quantitative
in nature, is epistemologically closer to the social sciences than to proving
theorems or writing computer code. As such, we point out some common pitfalls
we urge people considering this line of research to consider.

One common research approach is to analyze an existing public data set ([3, 4,
18, 23, 47, 51, 60]). While this can provide important insights, this can be
limiting in several ways. First, this approach inverts the typical research
process, in which a research question is formulated, and then data obtained;
here, data is obtained, and research questions are formed (implicitly or
explicitly). In particular, this can limit the kinds of questions that can be
asked, and as a result limit the kinds of conclusions that can be made.
Second, data collection is not a neutral endeavor. Many choices are made in
the production of a data set, and these choices can have profound implications
for the kinds of questions that can be answered and the kinds of conclusions
that can be drawn. For example, [23] makes use of the data contained in the
popular Mathematics Genealogy Project. However, as the authors note, this has
the effect of excluding from their dataset all mathematicians who have not
advised a Ph.D. student.

Many of the studies given in previous subsections above concern questions
about racial and gender representation in the mathematics community. However,
the needed demographic information to conduct this data analysis is often not
present. This leaves researchers to collect new data, or to somehow infer
demographic information from other information present in the data set. One
approach is to try to match the names of individuals to other public
information, such as a personal website, as in [55, 57]. This is, of course,
somewhat labor intensive. An automated approach is to use software tools that
attempt to infer an individual’s gender based on their name (and sometimes
other information), as in [23]. This practice of researchers ascribing gender
or race to an individual is certain to contain errors to some degree, and
should be used with caution—researchers should carefully consider whether the
benefits of these techniques truly outweigh the harms caused by incorrectly
ascribing identities to people. Further, many studies including gender as a
factor completely disregard nonbinary individuals, either marking them as
“other” or discarding them from the data set.

Mathematicians undertaking this work should also be wary of an epistemological
bias. Our training understandably predisposes us to use quantitative
techniques. However, many questions of interest to the mathematics community
are better suited to mixed or qualitative methodologies, and in fact may be
impossible to answer with only quantitative methods. For example, we above
highlighted several papers that illuminate demographic disparities in various
aspects of the mathematical community. The next important step is to then
interrogate why these disparities exist and persist, which is much better
suited to qualitative methodologies. As such, we encourage those engaging in
this type of work to foster collaborations with experienced qualitative
researchers. The authors’ personal experience is that the mathematics
education research community (in addition to other social science researchers)
is a natural source of collaborators that can help push this work forward.

##  4 Future Directions

It should be clear to the reader that the #MetaMath examples given above are
not an exhaustive list of the ways that mathematics and data science can be
used to examine the mathematical sciences itself. There are many different
avenues that future work could build upon the information we have provided in
this article. First, a review of the points raised in Section 3 on
Methodological Considerations could lead to a number of ways to improve upon
or modify and replicate some of the results discussed in the survey of the
literature that we have provided. Second, examining the number of references
where the context for the research is STEM and not exclusively the
mathematical sciences could provide opportunities to repeat STEM-specific
research in Math-exclusive contexts. Third, inspired by the subsection
discussing qualitative analysis of discourse in the mathematics community,
future researchers could try to expand the use of qualitative techniques to
analyze the mathematical sciences. Fourth, there are numerous suggestions for
data-enabled projects involving curricular data in the subsection on
mathematical content and curricular choices that could benefit the wider
mathematics community. Lastly, as more and more data about the mathematical
sciences community becomes available, either updated versions of existing data
or data about new topics or in areas where data is currently incomplete or
inaccessible, these sources will provide new opportunities for ways to provide
insight into the nature of the mathematics community through the application
of techniques, tools, and topics from mathematics and data science.

We conclude by inviting interested researchers to join us in this project. The
work of promoting social justice and improving equity within the field of
mathematics is and will remain an ongoing process.

## Author Contributions

All the listed co-authors contributed equally to the creation of this article;
the order of attribution is alphabetical, as is customary in mathematics and
is not intended to demonstrate any distinction in credit or effort.

## Acknowledgments

This material is based upon work supported by the National Science Foundation
under Grant No. DMS-1929284 while some of the authors were in residence at the
Institute for Computational and Experimental Research in Mathematics in
Providence, RI, during the Data Science and Social Justice: Networks, Policy,
and Education program. The authors thank ICERM for providing the space for
this work.

The authors would like to acknowledge Katherine (Katie) Kinnaird, who
contributed to early discussions.

Buckmire acknowledges that the Data Science and Social Justice program at
ICERM, under lead organizer Carrie Diaz Eaton, provided key organizational and
logistical resources that facilitated the relationships and connections that
resulted in this paper.

Hibdon was supported, in part, by the National Institutes of Health’s National
Cancer Institute, Grant Numbers U54CA202995, U54CA202997, and U54CA203000. The
content is solely the responsibility of the authors and does not necessarily
represent the official views of the National Institutes of Health.

Pabón thanks Nicholas Dubicki for insightful discussions. Pabón is partially
supported by the National Science Foundation under Awards DMS-2108839 and
DMS-1450182 .

Vindas-Meléndez thanks Benjamin Braun for helpful discussion. Vindas-Meléndez
is partially supported by the National Science Foundation under Award
DMS-2102921.

## References

  * [1] Veronica Anderson and Pamela Burdman.  A new calculus for college admissions: How policy, practice, and perceptions of high school math education limit equitable access to college, 2022\.  https://justequations.org/resource/a-new-calculus-for-college-admissions-how-policy-practice-and-perceptions-of-high-school-math-education-limit-equitable-access-to-college.
  * [2] Richelle M. Blair, Ellen E. Kirkman, and James W. Maxwell.  Statistical abstract undergraduate programs in the Mathematical Sciences in the United States: 2015 CBMS survey.  American Mathematical Society, 2018.
  * [3] Abra Brisbin and Ursula Whitcher.  Women’s representation in mathematics subfields: Evidence from the arxiv.  arXiv preprint arXiv:1509.07824, 2015.
  * [4] Heather Z Brooks, Philip Chodrow, Harlin Lee, and Juan G Restrepo.  Branching process model for female gender representation in the mathematics genealogy project.  In 2023 Joint Mathematics Meetings (JMM 2023). AMS, 2023.
  * [5] Ron Buckmire.  “Who Does the Math?”: On the diversity and demographics of the mathematics community in the USA.  In Improving Applied Mathematics Education, pages 1–12. Springer, 2021.
  * [6] Ron Buckmire.  An introduction to quantitative justice.  In 2022 Annual Meeting of the Society for Industrial and Applied Mathematics (SIAM AN22). SIAM, 2022.
  * [7] Ron Buckmire.  An introduction to quantitative justice: The application of quantitative techniques from data science and mathematics to promote social justice.  In 2023 Joint Mathematics Meetings (JMM 2023). AMS, 2023.
  * [8] Ron Buckmire.  #metamath: Using big data to promote social justice in the mathematics community.  In Annual Meeting of the American Association for the Advancement of Science (AAAS23). AAAS, 2023.
  * [9] Ron Buckmire.  A quantitative justice primer: The #metamath project and other examples.  In SIAM Conference on Computer Science and Engineering (SIAM CSE23). SIAM, 2023.
  * [10] Ron Buckmire, Carrie Diaz Eaton, Joseph Edward Hibdon, Katherine M Kinnaird, Drew Lewis, Jessica M Libertini, Omayra Ortega, Rachel Roca, and Andres R Vindas-Melendez.  The# metamath project: Analyzing the mathematics community using mathematics and data science to promote social justice.  In 2023 Joint Mathematics Meetings (JMM 2023). AMS, 2023.
  * [11] Ron Buckmire, Carrie Diaz Eaton, Joseph E Hibdon Jr, Katherine M Kinnaird, Drew Lewis, Jessica Libertini, Omayra Ortega, Rachel Roca, and Andrés R Vindas-Meléndez.  On definitions of “mathematician”.  arXiv preprint arXiv:2302.07432, 2023.
  * [12] Ron Buckmire, Suzanne Lenhart, and Suzanne Sindi.  SIAM quantitative justice: Intersections between quantitative sciences and social justice, 2023.  SIAM Annual Meeting, Pittsburgh, PA.
  * [13] Ron Buckmire, Omayra Ortega, and Carrie Diaz Eaton.  SIAM minisymposium on quantitative justice (a NAM-SIAM joint session), 2023.  Joint Mathematics Meetings, Boston, MA.
  * [14] Gail Burrill, Henry Cohn, Yvonne Lai, Dev P. Sinha, Ji Y. Son, and Katherine F. Stevenson.  Listening for Common Ground in High School and Early Collegiate Mathematicss.  Notices of the American Mathematical Society, 60(5):798–805, 2023\.
  * [15] E. A. Cech and T. J. Waidzunas.  Systemic inequalities for lgbtq professionals in stem.  Science Advances, 7(3):eabe0933, 2021.
  * [16] Ho-Chun Herbert Chang and Feng Fu.  Elitism in mathematics and inequality.  Humanities and Social Sciences Communications, 8(1):1–8, 2021.
  * [17] Christine Yifeng Chen, Sara S Kahanamoku, Aradhna Tripati, Rosanna A Alegado, Vernon R Morris, Karen Andrade, and Justin Hosbey.  Meta-Research: Systemic racial disparities in funding rates at the National Science Foundation.  eLife, 11:e83071, 2022.
  * [18] Aaron Clauset, Samuel Arbesman, and Daniel B Larremore.  Systematic inequality and hierarchy in faculty hiring networks.  Science advances, 1(1):e1400005, 2015.
  * [19] ASA Revision Commitee.  Guidelines for assessment and instruction in statistics education (GAISE) in statistics education college report, 2015.  https://www.amstat.org/docs/default-source/amstat-documents/gaisecollege_full.pdf.
  * [20] Rachel Crowell.  New math research group reflects a schism in the field.  Scientific American, 2022.  https://www.scientificamerican.com/article/new-math-research-group-reflects-a-schism-in-the-field.
  * [21] C De Brey, TD Snyder, A Zhang, and SA Dillow.  Digest of education statistics.  National Center for Education Statistics, Institute of Education Sciences, US Department of Education. https://nces.ed.gov/programs/digest, 2021\.
  * [22] Richard D. De Veaux, Mahesh Agarwal, Maia Averett, Benjamin S. Baumer, Andrew Bray, Thomas C. Bressoud, Lance Bryant, Lei Z. Cheng, Amanda Francis, Robert Gould, Albert Y. Kim, Matt Kretchmar, Qin Lu, Ann Moskol, Deborah Nolan, Roberto Pelayo, Sean Raleigh, Ricky J. Sethi, Mutiara Sondjaja, Neelesh Tiruviluamala, Paul X. Uhlig, Talitha M. Washington, Curtis L. Wesley, David White, and Ping Ye.  Curriculum Guidelines for Undergraduate Programs in Data Science.  Annual Review of Statistics and Its Application, 4(1):15–30, 2017\.
  * [23] Cody FitzGerald, Yitong Huang, Katelyn Plaisier Leisman, and Chad M. Topaz.  Temporal dynamics of faculty hiring in mathematics.  Humanities and Social Sciences Communications, 10(1):247, 2023.
  * [24] National Center for Science and Engineering Statistics (NCSES).  Diversity and STEM: Women, Minorities, and Persons with Disabilities 2023 (Special Report NSF 23-315).  National Science Foundation, 2023.
  * [25] Santo Fortunato, Carl T Bergstrom, Katy Börner, James A Evans, Dirk Helbing, Staša Milojević, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al.  Science of Science.  Science, 359(6379):eaao0185, 2018.
  * [26] Donna K Ginther, Walter T Schaffer, Joshua Schnell, Beth Masimore, Faye Liu, Laurel L Haak, and Raynard Kington.  Race, ethnicity, and NIH research awards.  Science, 333(6045):1015–1019, 2011.
  * [27] Bonnie Gold, Sandra Z. Keith, and William A. Marion, editors.  Assessment practices in undergraduate mathematics.  Notes 249. Mathematical Association of America, 1999.
  * [28] R. Gutiérrez.  In Rehumanizing mathematics for Black, Indigenous, and Latinx students, chapter Introduction: The need to rehumanize mathematics.  National Council of Teachers of Mathematics, 2018.
  * [29] Inclusion/exclusion, a justice and math weblog.  https://inclusionexclusion.org/.
  * [30] Quindel Jones, Andrés R Vindas Meléndez, Ariana Mendible, Manuchehr Aminian, Heather Z Brooks, Nathan Alexander, Carrie Diaz Eaton, and Philip Chodrow.  Data science and social justice in the mathematics community.  arXiv preprint arXiv:2303.09282, 2023.
  * [31] Deborah Kaminski and Cheryl Geisler.  Survival analysis of faculty retention in science and engineering by gender.  Science, 335(6070):864–866, 2012.
  * [32] Mari Kawakatsu, Philip S Chodrow, Nicole Eikmeier, and Daniel B Larremore.  Emergence of hierarchy in networked endorsement dynamics.  Proceedings of the National Academy of Sciences, 118(16):e2015188118, 2021.
  * [33] Sarah-Jane Leslie, Andrei Cimpian, Meredith Meyer, and Edward Freeland.  Expectations of brilliance underlie gender distributions across academic disciplines.  Science, 347(6219):262–265, 2015.
  * [34] Fengyuan Liu, Talal Rahwan, and Bedoor AlShebli.  Non-white scientists appear on fewer editorial boards, spend more time under review, and receive fewer citations.  Proceedings of the National Academy of Sciences, 120(13):e2215324120, 2023.
  * [35] Greg Martin.  An annotated bibliography of work related to gender in science.  arXiv preprint arXiv:1412.4104, 2014.
  * [36] Herbert A Medina.  Doctorate degrees in mathematics earned by Blacks, Hispanics/Latinos, and Native Americans: a look at the numbers.  Notices of the American Mathematical Society, 51(7):772–775, 2004\.
  * [37] Sharan B Merriam, Juanita Johnson-Bailey, Ming-Yeh Lee, Youngwha Kee, Gabo Ntseane, and Mazanah Muhamad.  Power and positionality: Negotiating insider/outsider status within and across cultures.  International journal of lifelong education, 20(5):405–416, 2001\.
  * [38] Helena Mihaljević and Lucía Santamaría.  Authorship in top-ranked mathematical and physical journals: Role of gender on self-perceptions and bibliographic evidence.  Quantitative Science Studies, 1(4):1468–1492, 2020.
  * [39] Helena Mihaljević-Brandt, Lucía Santamaría, and Marco Tullney.  The effect of gender in the publication patterns in mathematics.  PLoS One, 11(10):e0165367, 2016.
  * [40] Mitsunori Misawa.  Cuts and bruises caused by arrows, sticks, and stones in academia: Theorizing three types of racist and homophobic bullying in adult and higher education.  Adult Learning, 26(1):6–13, 2015.
  * [41] Allison C Morgan, Nicholas LaBerge, Daniel B Larremore, Mirta Galesic, Jennie E Brand, and Aaron Clauset.  Socioeconomic roots of academic faculty.  Nature Human Behaviour, pages 1–9, 2022.
  * [42] Sean A Myers, Peter J Mucha, and Mason A Porter.  Mathematical genealogy and department prestige.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 21(4):041104, 2011.
  * [43] Mathematical Association of America.  MAA MathFest 2023 in Tampa, FL.  https://www.maa.org/meetings/mathfest.
  * [44] Mathematical Association of America.  Guidelines for programs and departments in undergraduate mathematical sciences, 2003.  https://www.maa.org/sites/default/files/pdf/guidelines/Dept-Guidelines-Feb2003.pdf.
  * [45] Mathematical Association of America.  Guidelines for undertaking a self-study in the mathematical sciences, 2010\.  https://www.maa.org/sites/default/files/pdf/ProgramReview/MAA-SelfStudyManual.pdf.
  * [46] Mathematical Association of America.  Curriculum guide to majors in the mathematical sciences, 2015.  https://www.maa.org/sites/default/files/pdf/ProgramReview/MAA-SelfStudyManual.pdf.
  * [47] Robert Reys, Barbara Reys, and Jeffrey Shih.  Some Patterns of Phds in mathematics Awarded Annually by Institutions of Higher Education in the United States over the Last Two Decades.  Notices of the American Mathematical Society, 69(1), 2022.
  * [48] Leslie J Rissler, Katherine L Hale, Nina R Joffe, and Nicholas M Caruso.  Gender differences in grant submissions across science and engineering fields at the NSF.  Bioscience, 70(9):814–820, 2020.
  * [49] Rachel Roca, Carrie Diaz Eaton, Drew Lewis, Joseph E Hibdon Jr, and Stefanie Marshall.  #DisruptJMM: Themes of online social justice advocacy and community building in STEM.  Journal of Humanistic Mathematics, 2023.
  * [50] Karen Saxe and Linda Braddy.  A common vision for undergraduate mathematical sciences programs in 2025, 2015.  https://www.maa.org/sites/default/files/pdf/CommonVisionFinal.pdf.
  * [51] Jean-Marc Schlenker.  The prestige and status of research fields within mathematics.  arXiv preprint arXiv:2008.13244, 2020.
  * [52] Karen B Schmaling and Stephen A Gallo.  Gender differences in peer reviewed grant applications, awards, and amounts: a systematic review and meta-analysis.  Research Integrity and Peer Review, 8(1):2, 2023.
  * [53] Daniel Storage, Zachary Horne, Andrei Cimpian, and Sarah-Jane Leslie.  The frequency of “brilliant” and “genius” in teaching evaluations predicts the representation of women and african americans across fields.  PloS one, 11(3):e0150194, 2016.
  * [54] Michael A Taffe and Nicholas W Gilpin.  Racial inequity in grant funding from the us national institutes of health.  Elife, 10:e65697, 2021.
  * [55] Chad M Topaz, James Cart, Carrie Diaz Eaton, Anelise Hanson Shrout, Jude A Higdon, Kenan Ince, Brian Katz, Drew Lewis, Jessica Libertini, and Christian Michael Smith.  Comparing demographics of signatories to public letters on diversity in the mathematical sciences.  PloS one, 15(4):e0232075, 2020.
  * [56] Chad M Topaz, Maria-Veronica Ciocanel, Phoebe Cohen, Miles Ott, and Nancy Rodríguez.  Institute for the quantitative study of inclusion, diversity, and equity (qside).  Notices of the American Mathematical Society, 67(2), 2020.
  * [57] Chad M Topaz and Shilad Sen.  Gender representation on journal editorial boards in the mathematical sciences.  PLoS One, 11(8):e0161357, 2016.
  * [58] The Wellcome Trust.  Our commitment to tackling racism at wellcome, 2020.  https://wellcome.org/press-release/our-commitment-tackling-racism-wellcome.
  * [59] D. Wang and A.L. Barabási.  The Science of Science.  Cambridge University Press, 2021.
  * [60] K Hunter Wapman, Sam Zhang, Aaron Clauset, and Daniel B Larremore.  Quantifying hierarchy and dynamics in US faculty hiring and retention.  Nature, 610(7930):120–127, 2022.
  * [61] Samuel F Way, Daniel B Larremore, and Aaron Clauset.  Gender, productivity, and prestige in computer science faculty hiring networks.  In Proceedings of the 25th international conference on world wide web, pages 1169–1179, 2016.
  * [62] An Zeng, Zhesi Shen, Jianlin Zhou, Jinshan Wu, Ying Fan, Yougui Wang, and H Eugene Stanley.  The science of science: From the perspective of complex systems.  Physics reports, 714:1–73, 2017.

## Authors

  * •

_Ron Buckmire_ received his Ph.D. in mathematics from Rensselaer Polytechnic
Institute in 1994 and has been on the faculty of Occidental College in Los
Angeles, California ever since. He is a passionate advocate for broadening the
participation of people from historically marginalized and currently under-
represented groups in the mathematical sciences. Dr. Buckmire believes that:
1) mathematics is a human endeavor; 2) mathematics is created, discovered,
learned, researched, and taught by people; 3) the identities and experiences
of “who does math” are important and that everyone and anyone can and should
be welcome in the mathemaitcs community.

Mathematics Department, Occidental College, Los Angeles, CA
<EMAIL_ADDRESS>
  * •

_Joseph E. Hibdon, Jr._ received his PhD in Applied Mathematics from
Northwestern University in 2011. He is an associate professor of mathematics
at Northeastern Illinois University, a Hispanic Serving Institutions in
Chicago, Illinois. Dr. Hibdon works in many interdisciplinary teams across the
country to increase diversity in STEM and broadening the participation of
students in research at the undergraduate level. Dr. Hibdon’s recent research
is in mathematical modeling, with a focus on public health and biological
phenomenon.

Department of Mathematics, Northeastern Illinois University, Chicago, Illinois
<EMAIL_ADDRESS>
  * •

_Drew Lewis_ received his Ph.D. in mathematics from Washington University in
St. Louis in 2012 and most recently served on the faculty at the University of
South Alabama. He now works primarily in education research and faculty
development.

<EMAIL_ADDRESS>
  * •

_Omayra Ortega_ received her PhD in Applied Mathematics & Computational
Sciences from the University of Iowa in 2008. She is an Associate Professor of
Applied Mathematics & Statistics and the Assistant Dean of Research and
Internships in the School of Science and Technology at Sonoma State
University. Using the tools from statistics, mathematics, data science, public
health and epidemiology, Dr. Ortega tackles the emerging health issues. She is
deeply committed to broadening the participation of under-represented
minorities in STEM and mentoring students through the challenges of academia.

Department of Mathematics & Statistics, Sonoma State University, Rohnert Park,
CA
<EMAIL_ADDRESS>
  * •

_José L. Pabón_ is a Ph.D. candidate in mathematics at the New Jersey
Institute of Technology in Newark, N.J. He received his A.B. degree in
mathematics from Princeton University in 2019 and has a particular affinity
for working with disenfranchised populations, having first-hand experience
with these given he was born and raised in Puerto Rico.

Mathematical Sciences Department, New Jersey Institute of Technology, Newark,
NJ
<EMAIL_ADDRESS>
  * •

_Rachel Roca_ is a Ph.D. student in computational mathematics, science, and
engineering at Michigan State University. Her interests lie in topological
data analysis and computing education with an emphasis in social good. She
believes in leveraging mathematical and data science tools for both advocacy
and activism.

Department of Computational Mathematics, Science, and Engineering, Michigan
State University, East Lansing, MI
<EMAIL_ADDRESS>
  * •

_Andrés R. Vindas-Meléndez_ received their PhD in mathematics from the
University of Kentucky in 2021. Andrés is an NSF Postdoctoral Fellow and
Lecturer at UC Berkeley and will begin as a tenure-track Assistant Professor
at Harvey Mudd College in July 2024. Andrés’ research interests are in
algebraic, enumerative, and geometric combinatorics and have expanded to
include applications of data science and mathematics for social justice.

Department of Mathematics, University of California, Berkeley, CA
Department of Mathematics, Harvey Mudd College, Claremont, CA
<EMAIL_ADDRESS><EMAIL_ADDRESS>


F(X_{t_{j-1}}^{m,i_{1},i_{2}},t_{j-1})$
$\displaystyle+\frac{1}{N}\sum_{m=1}^{N}\sum_{i_{1}}\delta
hh_{w}\sum_{j=1}^{k}\prod_{l=j}^{k}1_{\mathbb{R}_{+}^{2}}(X_{t_{l}}^{m,i_{1},0})\nabla_{x}\varLambda_{2,\epsilon}(X_{t_{k}}^{m,i_{1},0},x)\wedge\tilde{\chi}_{\varepsilon}(X_{t_{j-1}}^{m,i_{1},0},t_{j-1})$

and

$\displaystyle\frac{\partial}{\partial x_{1}}$
$\displaystyle\theta(x,t_{k})=\frac{T_{w}}{\pi}\left(\frac{x_{2}}{x_{2}^{2}+(L+x_{1})^{2}}-\frac{x_{2}}{x_{2}^{2}+(L-x_{1})^{2}}\right)$
$\displaystyle-\frac{1}{N}\sum_{m=1}^{N}\sum_{i_{1}}\sum_{i_{2}\geq
0}hh_{i_{1},i_{2}}\prod_{j=1}^{k}1_{\mathbb{R}_{+}^{2}}(Y_{t_{j}}^{m,i_{1},i_{2}})\frac{\partial}{\partial
x_{1}}\varLambda_{2,\epsilon}(Y_{t_{k}}^{m,i_{1},i_{2}},x)\cdot
R(x^{i_{1},i_{2}},t_{k};0)\varTheta_{i_{1},i_{2}}.$

The rest of the scheme remains the same as in the one-copy case. This
completes the SLN scheme for the wall-bounded Oberbeck-Boussinesq flows.

####  7.2.3 Numerical experiments

Here we carried out two sets of numerical experiments based on the one-copy
scheme for the half-plane case. We consider two different Prandtl numbers:
when the kinematic viscosity $\nu=1$, and thermal diffusivity $\kappa=0.15$,
and when $\nu=0.15$, $\kappa=1$.

We choose the typical length scale $L=2\pi$, consider the flow in the
rectangle region $[-L,L]\times[0,L]$ and assume that $\alpha=0.0005$. The
parameter we use to smooth out the Biot-Savart kernel is chosen to be
$\epsilon=0.1$.

In the experiment presented, we set the initial velocity to be of the form

$u(x,0)=(-10\sin(2x_{2}),0),$

and the initial temperature is given by

$\theta(x,0)=0.01(8\pi^{2}-{x_{1}}^{2}-{x_{2}}^{2}).$

Thus, $\omega(x,0)=20\cos(2x_{2})$, and

$\hat{\omega}_{i_{1},i_{2}}=\begin{cases}20\cos(2i_{2}h),\quad&i_{2}>0,\\\
0,\quad&i_{2}=0.\end{cases}$

The time step is $\delta=0.01$, and the boundary layer thickness is taken to
be $\gamma=\pi/40$. The mesh size is set to be $h=2\pi/40\approx 0.157$ and
$h_{w}=\gamma/20\approx 0.00393$, and $\varepsilon=h_{w}/2$. We assume that
the external heat source is at temperature $T_{w}=0.1$, and

$\theta_{0}^{\prime}(x_{1})=-0.2x_{1}.$

The numerical experiment results at times $t=0.2$, $t=1.0$, $t=1.8$ with
Prandtl number $\mathrm{Pr}=0.15$ are shown in Figure 7.3, and the results at
the given times with $\mathrm{Pr}=6.67$ are shown in Figure 7.4.

(a) Temperature at $t=0.2$.

(b) Temperature field at $t=1.0$.

(c) Temperature field at $t=1.8$.

(d) Velocity at $t=0.2$.

(e) Velocity at $t=1.0$.

(f) Velocity at $t=1.8$.

(g) Boundary layer velocity at $t=0.2$.

(h) Boundary layer velocity at $t=1.0$.

(i) Boundary layer velocity at $t=1.8$.

Figure 7.3: Temperature and velocity fields of Oberbeck-Boussinesq flows on
$\mathbb{R}^{2}_{+}$ with Prandtl number $\mathrm{Pr}=0.15$.

(a) Temperature at $t=0.2$.

(b) Temperature field at $t=1.0$.

(c) Temperature field at $t=1.8$.

(d) Velocity at $t=0.2$.

(e) Velocity at $t=1.0$.

(f) Velocity at $t=1.8$.

(g) Boundary layer velocity at $t=0.2$.

(h) Boundary layer velocity at $t=1.0$.

(i) Boundary layer velocity at $t=1.8$.

Figure 7.4: Temperature and velocity fields of Oberbeck-Boussinesq flows on
$\mathbb{R}^{2}_{+}$ with Prandtl number $\mathrm{Pr}=6.67$.

The simulations demonstrate well the regular pattern known as the Bénard
convection, and also reveal the detailed hairy type of flows within the thin
boundary layer, confirming the theoretical results and observations.

## Data Availability Statement

No data are used in this article to support the findings of this study.

## Acknowledgement

ZQ is supported partially by the EPSRC Centre for Doctoral Training in
Mathematics of Random Systems: Analysis, Modelling and Simulation
(EP/S023925/1).

## References

  * [1] Alanko, S. 2016 Stability of Regression-Based Monte Carlo Methods for Solving Nonlinear PDEs. _Communications on Pure and Applied Mathematics_ $\mathbf{69}$ no.5, 958-980.
  * [2] Alda, W., Dzwinel, W., Witowski, J., Mościński, J., Pogoda, M. and Yuen, D. 1996 Rayleigh-Taylor instabilities simulated for a large system using molecular dynamics. _Tech. Rep._ _UMSI 96/104_ , Supercomputer Institute, University of Minnesota, Minneapolis.
  * [3] Anderson, C. 1986 Vorticity boundary conditions and boundary vorticity generation for two-dimensional viscous incompressible, _J. Comput. Phys._ 80, 72-97.
  * [4] Anderson, C. and Greengard, C. 1985 On vortex methods _. SIAM J. Numer. Anal_. $\mathbf{22}$ (3), 413-440.
  * [5] Anderson, C. and Greengard, C. eds. 1988 Vortex methods, _Lecture Notes in Math._ Vol. $\boldsymbol{1360}$, Springer-Verlag, Berlin.
  * [6] Anderson, C. and Greengard, C., eds. 1991 Vortex dynamics and vortex methods, Lectures in Appl. Math. Vol. $\boldsymbol{28}$, AMS.
  * [7] Bénard, H. 1900 Les tourbillons cellulaires dans une nappe liquide. _Revue Gén. Sci. Pur. Appl_. $\boldsymbol{11}$, 1261-1271 and 1309-1328.
  * [8] Boussinesq, J. 1903 _Theéorie analytique de la chaleur_ , vol. 2, Paris: Gauthier-Villars.
  * [9] Busnello, B. 1999 A probabilistic approach to the two-dimensional Navier-Stokes equations. _Ann. Probab._ $\mathbf{27}$, no.4, 1750-1780.
  * [10] Busnello, B., Flandoli, F. and Romito, M. 2005 A probabilistic representation for the vorticity of a three dimensional viscous fluid and for general systems of parabolic equations. _Proc. Edinb. Math. Soc._ $\mathbf{48}$, no 2, 295-336.
  * [11] Celani, A., Cencini, M., Mazzino, A. and Vergassola, M. 2004 Active and passive fields face to face. _New J. Phys,_ 6 72 (2004).
  * [12] Chandrasekhar, S. 1961 _Hydrodynamic and hydromagnetic stability_. Oxford: Clarendon Press.
  * [13] Chorin, A. J. 1973 Numerical study of slightly viscous flow. _J. Fluid Mech._ $\mathbf{57}$, 785-796.
  * [14] Chorin, A. J. 1980 Vortex models and boundary layer instability. _SIAM J. Sci. Statist. Comput._ $\mathbf{1}$, no. 1, 1-21.
  * [15] Constantin, P. 2001 An Eulerian-Lagrangian approach for incompressible fluids: local theory. _J. Amer. Math. Soc._ $\mathbf{14}$ no. 2, 263-278 (electronic).
  * [16] Constantin, P. 2001 An Eulerian-Lagrangian approach to the Navier-Stokes equations. _Comm. Math. Phys._ $\mathbf{216}$, no. 3, 663-686.
  * [17] Constantin, P. and Iyer, G. 2011 A stochastic-Lagrangian approach to the Navier-Stokes equations in domains with boundary, _Ann. Appl. Probab_. 21, 1466-1492 (2011).
  * [18] Cottet, G. -H., and Koumoutsakos, P. D. 2000 _Vortex Methods: Theory and Practice_. Cambridge University Press.
  * [19] Corrsin, S. 1951 On the spectrum of isotropic temperature fluctuations in isotropic turbulence, _J. Appl. Phys._ $\mathbf{22}$ 469\.
  * [20] Criminale, W. O., Jackson, T. L. and Joslin, R. D. 2003 _Theory and computation in hydrodynamic stability_. Cambridge University Press.
  * [21] Drazin, P. G. and Reid, W. H. 2004. _Hydrodynamic stability_. Second Edition. First Edition in 1981. Cambridge University Press.
  * [22] Drivas, T.D. and Eyink, G.L. 2017 A Lagrangian fluctuation-dissipation relation for scalar turbulence. Part I. Flows with no boundary walls. _Journal of Fluid Mechanics_ , Volume $\mathbf{829}$, 25 October 2017 , pp. 153 - 189 DOI: https://doi.org/10.1017/jfm.2017.567
  * [23] Drivas, T. D. and Eyink, G.L. 2017 A Lagrangian fluctuation-dissipation relation for scalar turbulence. Part II. Wall-bounded flows. _Journal of Fluid Mechanics_. 829, 236-279 (2017).
  * [24] Eyink, G., Gupta, A., and Zaki, T. 2020 Stochastic Lagrangian dynamics of vorticity. Part 1. General theory for viscous, incompressible fluids. _Journal of Fluid Mechanics_ , 901, A2. doi:10.1017/jfm.2020.491
  * [25] Eyink, G., Gupta, A., and Zaki, T. 2020 Stochastic Lagrangian dynamics of vorticity. Part 2. Application to near-wall channel-flow turbulence. _Journal of Fluid Mechanics_ , 901, A3. doi:10.1017/jfm.2020.492
  * [26] Falkovich, G., Gawędzki, K. and Vergassola, M. 2001 Particles and fields in fluid turbulence, _Rev. Mod. Phys._ $\mathbf{73}$ 913-975.
  * [27] Feynman, R. P. 1948 Space-time approach to non-relativistic quantum mechanics. _Rev. Mod. Phys._ Vol. $\mathbf{20}$, No. 2, 367-387.
  * [28] Fletcher, C. A. J. 1991 _Computational techniques for fluid dynamics_ , Vol. I and II, second edition. Springer-Verlag.
  * [29] Friedman, A. 1964 _Partial differential equations of parabolic type_. Prentice-Hall, Inc.
  * [30] Freidlin, M. 1985 Functional integration and partial differential equations. Princeton University Press.
  * [31] Goodman, J. 1987 Convergence of the random vortex method. _Comm. Pure Appl. Math._ $\mathbf{40}$(2), 189-220.
  * [32] Griebel, M., Knapek, S. and Zumbusch, G. 2007 _Numerical simulation in molecular dynamics_. Springer.
  * [33] Helmholtz, H. 1858 Über die integrale der hydrodynamischen Gleichungen, welche den Wirbelbewegungen entsprechen, _Jour. für die reine unde angewandte Math_. also in Hermann von Helmholtz, _Wissenschaftliche Abhandlungen_ , Vol. $\boldsymbol{1}$, pp. 101-104.
  * [34] Holm, D., Marsden, J. and Ratiu, T. 1998 The Euler-Poincaré equations and semidirect products with applications to continuum theories. _Adv. Math._ $\mathbf{137}$ no.1, 1-81.
  * [35] Iyer, G. 2006 A stochastic perturbation of inviscid flows. _Comm. Math. Phys._ $\mathbf{266}$ no. 3, 631-645.
  * [36] Joseph, D. D. 1965 On the stability of the Boussibesq equations. _Arch. Rat. Mech. Anal._ $\boldsymbol{20}$, 59-71.
  * [37] Joseph, D. D. 1966 Non-linear stability of the Boussibesq equations by the method of energy. _Arch. Rat. Mech. Anal._ $\boldsymbol{22}$, 163-84.
  * [38] Joseph, D. D. 1976 _Stability of fluid motions. I and II_. Springer Tracts in Natural Philosophy, vol. 28, Springer-Verlag, New York.
  * [39] Kac, M. 1949 On Distributions of Certain Wiener Functionals. _Transactions of the American Mathematical Society_ , Jan., 1949, Vol. $\mathbf{65}$, No. 1 (Jan., 1949), pp. 1-13
  * [40] Kloeden, P. E. and Platen, E. 1992 _Numerical Solutions to Stochastic Differential Equations_. Springer-Verlag Berlin Heidelberg.
  * [41] Kolmogorov, A. N. 1941 The local structure of turbulence in incompressible viscous fluid for very large reynolds numbers. _Comptes Rendus de l’Académie des Sciences de l’URSS_ , 30:301–305, 1941a (reprinted in _Proc. R. Soc. Lond._ A 434, 9-13, 1991).
  * [42] Kolmogorov A. N. 1941 Dissipation of energy in the locally isotropic turbulence. _Comptes Rendus de l’Académie des Sciences de l’URSS_ , 32:16–18, 1941b (reprinted in _Proc. R. Soc. Lond._ A 434, 15-17, 1991).
  * [43] Kraichnan, R. H. 1968 Small-scale structure of a scalar field convected by turbulence. _Phys. Fluids_ $\mathbf{11}$ 945-953.
  * [44] Landau, L. D. and Lifshitz, E. M. 1987 _Fluid mechanics_. Second edition. Pergamon Press.
  * [45] LeJan, Y. and Sznitman, A. S. 1997 Stochastic cascades and 3-dimensional Navier-Stokes equations. _Probab. Theory Related Fields_ $\mathbf{109}$ no. 3, 343-366.
  * [46] LeJan, Y. and Raimond, O. 2002 Integration of Brownian vector fields, _Ann. Probab._ $\mathbf{30}$, 826-873.
  * [47] LeJan, Y. and Raimond, O. 2004 Flows, coalescence and noise, _Ann. Probab._ $\mathbf{32}$, 1247-1315.
  * [48] Lesieur, M., Métais, O. and Comte, P. 2005 _Large-Eddy simulations of turbulence_. Cambridge University Press.
  * [49] Long, D. G. 1988 Convergence of the random vortex method in two dimensions. _J. of Amer. Math. Soc_. $\mathbf{1}$(4 ), 779-804.
  * [50] Lyons, T. J. and Zheng, W. 1988 A crossing estimate for the canonical process on a Dirichlet space and a tightness result. _Astérisque_ , tome 157-158, p. 249-271.
  * [51] Lyons, T. J. and Zheng, W. 1990 On conditional diffusion processes. _Proceedings of the Royal Society of Edinburgh Section A: Mathematics_ Vol. $\mathbf{115}$, Issue 3-4, 243-255.
  * [52] Majda, A. J. and Bertozzi A. L. 2002 _Vorticity and incompressible flow_. Cambridge University Press.
  * [53] McKean Jr, H. P. 1966 A class of Markov processes associated with nonlinear parabolic equations. _Proceedings of the National Academy of Sciences of the United States of America_ , 56(6), 1907. MR0221595https://doi.org/10.1073/pnas.56.6.1907
  * [54] Oberbeck, A. 1879 Ueber die Wärmleitung der Flüssigkeiten bei Berücksichtigung der Strömungen infolge von Temperaturdifferenzen. _Ann. Phys. Chem_. $\boldsymbol{7}$, 271-92.
  * [55] Oboukhov, A. M. 1949 Structure of the temperature field in turbulent flows, _Izv. Akad. Nauk. SSSR, Geogr. and Geophyys_ $\mathbf{13}$ 58\.
  * [56] Pardoux, E. and Peng, S. G. 1990 Adapted solution of a backward stochastic differential equation. _Systems & Control Letters_ Vol. $\mathbf{14}$ no.1, 55-61.
  * [57] Peskin, C. 1985 A random-walk interpretation of the incompressble Navier-Stokes equations. _Comm. Pure Appl. Math._ $\mathbf{38}$ no. 6, 845-852.
  * [58] Pope, S. B. 2000 _Turbulent Flows_. Cambridge University Press.
  * [59] Qian, Z. 2022 Stochastic formulation of incompressible fluid flows in wall-bounded regions. arXiv:2206.05198
  * [60] Qian Z. Süli E. and Zhang Y. 2022 Random vortex dynamics via functional stochastic differential equations. _Proc. R. Soc._ A $\boldsymbol{478}$: 20220030. https://doi.org/10.1098/rspa.2022.0030
  * [61] Qian, Z., Qiu, Y., Zhao, L. and Wu, J. 2022 Monte-Carlo simulations for wall-bounded fluid flows via random vortex method. arXiv:2208.13233
  * [62] Rayleigh, L. 1916 On convection currents in a horizontal layer of fluid, when the higher temperature is on the under side. _Phil. Mag._ (6) $\boldsymbol{32}$, 529-46.
  * [63] Sengupta, T. K. and Bhaumik, S. 2019 _DNS of wall-bounded turbulent flows – A first principle approach._ Springer Nature Singapore Pte Ltd.
  * [64] Shlesinger, M. F., West, B. J. and Klafter, J. 1987 Lévy dynamics of enhanced diffusion: application to turbulence. _Phys. Rev. Lett._ $\mathbf{58}$ (11), 1100.
  * [65] Stroock, D. and Varadhan, S. R. S. 1979 _Multidimensional diffusion processes_. Springer-Verlag Berlin Heidelberg.
  * [66] Taylor, G. I. 1921 Diffusion by continuous movements. _Proc. Lond. Math. Soc_. $\mathbf{20}$, 196.
  * [67] Taylor, G. I. 1935 Statistical theory of turbulence. Parts 1-4. _Proceedings of the Royal Society of London. Series A: Mathematical and Physical Sciences_ , 151(873):421–478, 1935\. https://doi.org/10.1098/rspa.1935.0159.
  * [68] Von Kármán, T. 1931 _Mechanical Similitude and Turbulence_. National Advisory Committee for Aeronautics, 1931.
  * [69] Thalabard, S., Krstu;ovic, G. and Bec, J. 2014 Turbulent pair dispersion as a continuous-time random walk. _J. Fluid Mech._ $\mathbf{755}$, R4.
  * [70] Vanden-Eijnden, E. W. and Vanden-Eijnden, E. 2000 Generalized flows, intrinsic stochasticity and turbulent transport. _Proc. Natl Acad. Sci. USA_ $\mathbf{97}$, 8200-8205.
  * [71] Vanden-Eijnden, E. W. and Vanden-Eijnden, E. 2001 Turbulent Prandtl number effect on passive scalar advection. _Physic D_ , $\mathbf{152-153}$, 636-645.
  * [72] Wesseling, P. 2001 _Principles of computational fluid dynamics._ Springer-Verlag Berlin Heidelberg.
  * [73] Wung, T. and Tseng, F. 1992 A color-coded particle tracking velocimeter with application to natural convection, _Experimental Fluids,_ $\boldsymbol{13}$, 217-223.
  * [74] Yu, H., Kanov, K., Perlman, E., Graham, J., Frederix, E., Burns, R., Szalay, A., Eyink, G. L. and Meneveau, C. 2012 Studying Lagrangian dynamics of turbulence using on-demand fluid particle tracking in a public turbulence database. _J. Turbul._ $\mathbf{12}$, N13.
  * [75] Zhang, X. 2010 A stochastic representation for backward incompressible Navier-Stokes equations. _Probab. Theory Related Fields_ $\boldsymbol{148}$, 305-332.


XX XX 8 May/June

Department: Head Editor: Name, xxxx@email

# Guidelines for Developing Bots for GitHub

Mairieli Wessel  Radboud University, The Netherlands  Andy Zaidman  Delft
University of Technology, The Netherlands  Marco A. Gerosa  Northern Arizona
University, USA  Igor Steinmacher  Northern Arizona University, USA

(2019)

###### Abstract

Projects on GitHub rely on the automation provided by software development
bots to uphold quality and alleviate developers’ workload. Nevertheless, the
presence of bots can be annoying and disruptive to the community. Backed by
multiple studies with practitioners, this paper provides guidelines for
developing and maintaining software bots. These guidelines aim to support the
development of future and current bots and social coding platforms.

††journal: IT Professional

Bridging the gap between human collaborative software development and
automated processes, bots are used to alleviate the software development
workload, improve productivity, and enable use cases for which humans are not
realistically suitable [1]. On social coding platforms, such as GitHub, a bot
acts autonomously to some extent, has a user account, and plays a role within
the development team, executing tasks that complement the developers’ work
[2].

Automating simple, time-consuming, or tedious tasks and collecting dispersed
information are some ways that bots support software projects. In previous
work, we have found that the adoption of bots helps developers merge more pull
requests, and reduces the need for communication between developers [3].
However, while bots are useful for automating a variety of tasks related to
software development, prior research has shown that they have the potential
side-effect of disrupting developers in their work [4].

Surveying and interviewing practitioners, we have found three categories of
reported challenges: interaction, adoption, and development challenges [5, 4]
(see Figure 1). Bot noisiness has appeared as a crosscutting concern in all
three categories. Noisiness often leads to communication issues and
expectation breakdowns. Developers often complain about a bot’s verbose
messages, timing, and high frequency of actions, which might be caused by
platform limitations or bot configuration issues.

Figure 1: Methodology employed to identify challenges, build a prototype, and
create guidelines. The result from Phase I was published at CSCW ([5, 4]), and
Phase II at ICSE 2022 [6]. We added a graphical mark () to identify the
challenges related to noisiness, which crosscut the three categories of
challenges.

Backed by the results of our empirical studies, we have investigated
interventions/strategies to mitigate noise and deal with some of the
identified challenges [6]. In line with Erlenhov et al. [1]’s results, our
results indicate that a combination of three different characteristics appears
to be relevant for a bot: intelligence, adaptability, and autonomy. Although
intelligence and adaptability recurrently appear in the literature as a
desired bot characteristic [1, 7], they are not yet widely present in bots
that work on GitHub [5].

Backed by the observations gathered from these studies, this paper presents a
set of guidelines to help develop software bots for GitHub and (re-)design the
human-bot interaction on social coding platforms. We expect that the advances
in bot creation frameworks will provide better support for the fulfillment of
the guidelines in the future.

## Research overview

We have collected evidence of bot noisiness throughout multiple empirical
studies as presented in Figure 1. First, we surveyed 205 open source
contributors and 23 maintainers [5] and openly asked them about the challenges
of using and interacting with bots. To deepen our understanding of these
challenges, we interviewed 21 practitioners experienced with bots, including
project maintainers, contributors, and bot developers [4]. The developers’
most recurrent complaints are related to annoying bot behaviors. Those
behaviors include the case in which bots provide comments with dense
information “in the middle of the pull request”, frequently overusing visual
elements, and the case in which bots perform repetitive actions, such as
creating numerous pull requests and leaving dozens of comments in a row. These
behaviors are often perceived as noise, which can lead to information and
notification overload, which disrupts both human communication and development
workflow.

As noise emerged as a central interaction challenge from our empirical
analysis, we have further investigated how to overcome it. We created two
interventions: (1) a mediator bot that organizes existing bot information in a
pull request, and (2) a separate interface for the bot interaction in the pull
request [6]. To design and implement the interventions, we applied Design
Fiction [8], a technique that has been broadly used in the Human-Computer
Interaction field to explore and critique future technologies. We presented to
32 open-source maintainers, contributors, bot developers, and bot researchers
a fictional story of a mediator bot capable of better supporting developers’
interactions on pull requests and operating as a mediator between developers
and the existing bots. During synchronous design fiction sessions,
participants answered questions to complete the end of the fictional story,
discussing the design strategies for the mediator bot and raising concerns
about the use of bots.

Building on the findings of our empirical investigation, we propose a set of
guidelines for both bot developers and tool builders. All the guidelines are
backed by the evidence previously collected and supported by the literature.

## Guidelines for developing bots

To make bots more effective at accomplishing their tasks, design problems need
to be solved to avoid repetitive notifications, provide consistency in the
tasks being done, make bots adaptive, and provide clear and contextualized
feedback to project contributors and maintainers [4, 6]. To better design the
next generation of bots, we provide a set of guidelines along with three main
categories: designing bot interaction, facilitating bot adoption, and
overcoming platform limitations.

### Designing bot interaction

One of the essential aspects of bot interaction is communication. However, the
existing bots might fail to provide meaningful information to developers. The
most recurrent and central challenge is the introduction of noise into the
developers’ communication channels. Developers complained about annoying bot
behaviors such as verbosity, high frequency and timing of actions, and
unsolicited actions. Therefore, we present a set of guidelines to support tool
builders and bot developers in designing bot interactions.

Guideline 1 (G#1) Provide clear, concise, and well-organized information.

Interaction challenges: We evidenced the need for background knowledge to
interact with and understand the messages of bots on GitHub. Combined with
lack of context, it might be extremely difficult for humans to extract
meaningful guidance from bots’ feedback. In these cases, when a bot message is
not clear enough, developers “[…] need to go and ask a human for clarity”,
which may generate more work for both contributors and maintainers.

What should bot developers do: To reduce the cognitive effort to process bot
feedback, it is preferable to prioritize conciseness over completeness. For
example, a bot that informs developers whether the changes in a pull request
affected the code coverage (i.e., a code coverage bot) should focus on
reporting the overall result, and pointing to sources of additional
information.

Guideline 2 (G#2) Focus on an appropriate way to show information.

Interaction challenges: Another important aspect of bot interaction is the way
bots should display information to developers. Developers frequently do not
like it when “[…] bots put a bunch of information that they try to convey in
comments instead of [providing] status hooks or a link somewhere.”

What should bot developers do: Bot developers should identify the best way to
convey the information. On GitHub, this can be achieved by exploring possible
ways to show information on the platform, which can be either status
information or comments. For example, a bot that looks over the code in a pull
request and catches quality issues (i.e., a code quality bot) can comment on a
pull request to report a list of code formatting issues found. In cases where
only an overall status (i.e., passing, falling, blocked) is needed, it is
preferable to use status information and avoid overloading pull requests with
additional comments.

Guideline 3 (G#3) Provide actionable changes to developers.

Interaction challenges: Another recurrent complaint from our survey and
interview participants is that bots do not provide actionable changes for
developers. Some of the messages and outcomes from bots are so strict that
they do not guide developers on what they should do next to accomplish their
tasks: “it is great to see ‘yes’ or ‘no,’ but if it is not actionable, then it
is not useful […]”.

What should bot developers do: Bot outcomes should be accompanied by
actionable and technically sound recommendations by default for the decision-
making of developers. For example, a pull request comment from a code coverage
bot informing that the coverage decreased is not actionable. However, a
comment accompanied by suggested changes is highly actionable because it helps
developers to figure out the next steps.

Guideline 4 (G#4) Avoid overly humanized bot messages.

Interaction challenges: Previous studies on human-chatbot interaction have
already shown that human users can hold higher expectations with overly
humanized bots (e.g., bots that say “thank you”) which can lead to frustration
[9]. Our study results underscore that some developers feel uncomfortable
interacting with a bot, as mentioned by one participant: “ ‘for some people,
it is still quite strange, and they are quite surprised by it.” Also,
receiving “thanks” from a non-human feels less sincere.

What should bot developers do: Although developers envision the bot mediator
interacting with users through natural language, more direct, and non-
humanized bot messages are appreciated. For instance, developers suggested
avoiding sentences that do not add to the bot’s feedback, such as “Hey, I’m
here to help you […].”

Guideline 5 (G#5) Make bots’ purpose clear.

Interaction challenges: By automating and providing feedback on time-consuming
tasks (e.g., checking code style or calculating code coverage), bots are
intended to reduce the workload of project maintainers and inform project
contributors. Nevertheless, maintainers reported that a challenge they see is
that “contributors don’t understand the value of bots for maintainers.” We
also found that developers with different profiles and backgrounds have
different expectations with regard to bot interaction. Bots, for example,
enforce predefined cultural rules of a community, causing expectation
breakdowns for outsiders.

What should bot developers do: It is essential to make the purpose of each bot
clear, avoiding expectation breakdowns from both sides. This may be
implemented, for example, by including a footnote descriptive sentence or a
link to further information about the bot in the bot comment.

### Facilitating bot adoption

If maintainers find an appropriate bot, they then have to deal with
configuration challenges. Thus, we present advice on how to facilitate the bot
adoption process in addition to the guidelines for designing the human-bot
interaction.

Guideline 6 (G#6) Provide options to configure bot notification.

Adoption challenges: The study conducted to co-design the mediator bot
prototype showed that open-source developers would like to customize aspects
of the bot interaction, including notification frequency and timing.
Therefore, it is important for bot developers to design a highly customizable
bot, providing project maintainers better configuration control over bot
actions, rather than just turning off bot comments.

What should bot developers do: In the mediator bot design sessions, developers
suggested scheduling of bot notifications, so that the bot would avoid
notifying developers according to (customizable) timeframes indicating when
they do not want interruptions. This may be implemented, for example, using a
“do not disturb” mode. Another option is not to notify maintainers until the
condition is satisfied. In this case, the bot would notify the developers only
when the predefined conditions are met: “I want to be notified about new pull
requests after all my tests have passed. And after the bots commented, and if
everything is green, then I want to be notified. The recommendation is that
these mechanisms are explicitly announced during bot adoption (e.g., noiseless
configuration, preset levels of information).

Guideline 7 (G#7) Include documentation of alternative installation settings
to accommodate different types of users.

Adoption challenges: It is difficult to tailor the bot configuration to fit
the needs of a project. Even after maintainers spend the time needed to
configure the bot, there is sometimes no way to predict what the bot will do
once installed. According to developers’ experience, it is “easy to install
the bot with the basic configuration. However, it is not easy to adjust the
configuration to your needs”.

What should bot developers do: Bot developers should document the bot
installation, giving concrete examples of the bot outcomes and possible
effects of each configuration choice, and keep it updated. This can also be
implemented by creating a FAQ (Frequently Asked Questions) section in a
website or repository where the bot code is stored. This is also an
opportunity for lowering the entry barrier for new project maintainers, who
need to be aware of how each bot works on the project.

Guidelines’ takeaway Bot developers should envision bots as _socio-technical_
rather than _technical_ applications, which must be designed to consider human
interaction, developers’ collaboration, and other ethical concerns.

## Recommendations to platform builders

To complement our guidelines, we also explored the platform restrictions since
they might limit both the extent of the bots’ actions and the way bots
communicate. We, therefore, present a set of recommendations for platform
builders that would aid bot developers.

Recommendation 1 (R#1): Enable multiple interaction mechanisms.

Platform limitations: Our empirical investigation of bot challenges revealed
some limitations imposed by the GitHub platform that restrict the design of
bots. As mentioned by one participant: “There are still a few things that just
cannot be done with the [GitHub] API..” The platform restrictions might limit
both the extent of the bots’ actions and the way bots communicate.

What should platform builders do: It is essential to provide alternative ways
for bots to interact on the platform. A developer stated that the platform
ideally would provide additional mechanisms since bots interact only through
comments. In other environments, such as Slack, developers can interact more
flexibly with (chat)bots. On GitHub, this might be achieved by enabling
distinct views of the same bot output depending on the developer’s role (i.e.,
maintainer, casual contributor, newcomer) and enabling developers to filter
and hide specific bot information.

Recommendation 2 (R#2): Consider creating a dedicated communication channel
for bots.

Platform limitations: The interviews we conducted with developers have shown
that dealing with bots providing comments with dense information “in the
middle of the pull request” can be “[…] a lot more distracting than it is
helpful.” Bots may overburden developers who already suffer from information
overload when communicating online.

What should platform builders do: To reduce information overload, participants
suggested removing bot interactions from the main conversation interface and
creating a dedicated place for them. We prototyped this strategy by designing
a new tab in the pull request interface; this idea can be leveraged to reshape
the interface and better display bot interactions. There is also room for
integrating GitHub bots into other developer communication platforms (e.g.,
Slack, Discord).

## The mediator bot

Figure 2: Prototype of the interventions in a real-world scenario on GitHub.
It shows the relationship between the design strategies for the mediator bot
derived from Phase II (S#1-6) and our proposed guidelines (G#1, G#2,
G#4)/platform recommendations (R#1, R#2). The interactive version of the
prototype is publicly available on Zenodo [10].

To alleviate the concern of bot noisiness in pull requests, we have
investigated the concept of a bot that operates as a mediator between
developers and the existing bots (i.e., a meta-bot). This section presents our
mediator bot prototype and how it connects to the proposed guidelines. Figure
2 provides an overview of the mediator bot design strategies, which we mark
throughout the text with (S#n). Firstly, we split our prototype into two
different versions: the experts’ pull request interface designed to support
maintainers and experienced contributors; and the newcomers’ pull request
interface. We designed a dedicated place for all information and events
regarding bots in the pull request (S#1; platform recommendation R#2). The
mediator bot creates a summary with the most important information about each
bot, then groups them into categories (e.g., “warnings”, “information”) (S#3;
G#1-2).

To avoid inflating the pull requests with several comments from the mediator
bot, one suggested strategy is to keep the most recent information (S#2; G#1).
We include the latest information from each bot in the summary. Reakit bot,
for example, posted two comments in the timeline of bot events; however, only
one entry is displayed in the summarized table for that bot. In addition, in
the timeline of bot events, it is possible to expand all bot comments to see
the complete messages (platform recommendation R#1).

In the newcomer’s interface, we added a text-based message to fulfill the
requirement of welcoming newcomers (S#4; G#4). Beyond presenting a welcoming
message, the mediator bot also points the contributor to other sources that
can contain information about rules, instructions, and requirements (S#5; G#1)
of the project. Thus, we included a link to Reakit’s contributing guidelines.

Another important distinction between the two versions is how the mediator bot
displays the information for newcomers versus experts (platform recommendation
R#1). We implemented an interactive process of displaying bots’ information
(S#6). The mediator bot guide newcomers by showing the information from other
bots “step by step.” Study participants deemed this strategy a potential
solution to reduce the impact of receiving several different bot notifications
simultaneously. As part of this guidance, the mediator bot also refers to
contribution guidelines to assist newcomers and present a concise and direct
welcoming message.

## Conclusion

Motivated by the growing importance of software bots that act upon the pull-
based development model, we have proposed guidelines on how to improve the
next generation of bots, considering interaction, adoption, and development
challenges identified in prior work. These guidelines can serve bot developers
and contributors and maintainers of GitHub projects that use bots in two
dimensions: understanding how bots are perceived and how they can be leveraged
to support development tasks. Orthogonal to the guidelines, we have also
explored the concept of a mediator bot, to alleviate the growing concern of
noisiness among bot users. We envision that our guidelines will help
developers to produce bots that better automate tasks and further guide
developers in collaborative software development.

## Acknowledgments

This work was partially supported by the Coordenação de Aperfeiçoamento de
Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 001; CNPq grants
141222/2018-2, 314174/2020-6, and 313067/2020-1; the National Science
Foundation under Grant numbers 1815503 and 1900903; and the Dutch science
foundation NWO through the Vici “TestShift” project (No. VI.C.182.032). We
also thank the developers who spent their time participating in our research.

## References

  * [1] L. Erlenhov, F. G. d. O. Neto, and P. Leitner, “An empirical study of bots in software development–characteristics and challenges from a practitioner’s perspective,” in _Proceedings of the 2020 28th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , ser. FSE 2020, 2016.
  * [2] M. Wessel and I. Steinmacher, “The inconvenient side of software bots on pull requests,” in _Proceedings of the 2nd International Workshop on Bots in Software Engineering_ , ser. BotSE, 2020.
  * [3] M. Wessel, A. Serebrenik, I. S. Wiese, I. Steinmacher, and M. A. Gerosa, “Effects of adopting code review bots on pull requests to oss projects,” in _IEEE International Conference on Software Maintenance and Evolution_. IEEE Computer Society, 2020\.
  * [4] M. Wessel, I. Wiese, I. Steinmacher, and M. A. Gerosa, “Don’t disturb me: Challenges of interacting with software bots on open source software projects,” _Proceedings of ACM Human-Computer Interaction_ , no. CSCW, 2021\.
  * [5] M. Wessel, B. M. de Souza, I. Steinmacher, I. S. Wiese, I. Polato, A. P. Chaves, and M. A. Gerosa, “The power of bots: Characterizing and understanding bots in oss projects,” _Proceedings of the ACM Conference on Computer Supported Cooperative Work Social Computing_ , vol. 2, no. CSCW, pp. 182:1–182:19, Nov. 2018. [Online]. Available: http://doi.acm.org/10.1145/3274451
  * [6] M. Wessel, A. Abdelattif, I. Wiese, T. Conte, E. Shihab, M. A. Gerosa, and I. Steinmacher, “Bots for pull requests: The good, the bad, and the promising,” _Proceedings of IEEE/ACM International Conference on Software Engineering_ , no. ICSE, 2022.
  * [7] C. Lebeuf, A. Zagalsky, M. Foucault, and M.-A. Storey, “Defining and classifying software bots: A faceted taxonomy,” in _Proceedings of the 1st International Workshop on Bots in Software Engineering_ , ser. BotSE ’19. Piscataway, NJ, USA: IEEE Press, 2019, pp. 1–6. [Online]. Available: https://doi.org/10.1109/BotSE.2019.00008
  * [8] M. Blythe, “Research through design fiction: narrative in real and imaginary abstracts,” in _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, 2014, pp. 703–712.
  * [9] U. Gnewuch, S. Morana, and A. Maedche, “Towards designing cooperative and social conversational agents for customer service,” in _International Conference on Information Systems (ICIS)_. AIS, 2017.
  * [10] M. Wessel, A. Abdellatif, I. Wiese, T. Conte, E. Shihab, M. A. Gerosa, and I. Steinmacher, “Supplementary Material for ICSE’22 paper ”Bots for Pull Requests: The Good, the Bad, and the Promising”,” Sep. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5675702

Mairieli Wessel is an Assistant Professor in the Department of Software
Science (SwS) at Radboud University, The Netherlands. She obtained her Ph.D.
in Computer Science from the University of São Paulo, Brazil. Her main
research interest is in software engineering (SE) and computer-supported
cooperative work (CSCW), focused on software bots and open-source development.
Her research goal is to design intelligent support for developers by
leveraging bots’ capabilities. For more information, visit
http://www.mairieli.com

Andy Zaidman, is a Full Professor in software engineering at Delft University
of Technology, The Netherlands. He received his M.Sc. and Ph.D. degrees in
Computer Science from the University of Antwerp, Belgium, in 2002 and 2006,
respectively. His main research interests include software evolution, program
comprehension, mining software repositories, software quality, and software
testing. He is an active member of the research community and involved in the
organization of numerous conferences (WCRE’08, WCRE’09, VISSOFT’14 and
MSR’18). In 2013 he was the laureate of a prestigious Vidi midcareer grant,
while in 2019 he received the most prestigious Vici career grant from the
Dutch science foundation NWO.

Marco A. Gerosa, is a Full Professor at the Northern Arizona University, USA,
and a Ph.D. advisor at the University of São Paulo, Brazil. He received his
Ph.D. from the Catholic University of Rio de Janeiro in 2006. He researches
Software Engineering and CSCW, and recent projects include the development of
tools and strategies to support developers’ onboarding to open-source software
communities and the design of bots and chatbots. He published more than 200
papers and serves on the program committee (PC) of top-tier conferences, such
as ICSE, FSE, MSR, ICSME, and CSCW. For more information, visit
http://www.marcoagerosa.com

Igor Steinmacher, is an Assistant Professor in the School of Informatics,
Computing, and Cyber Systems at the Northern Arizona University (NAU), and was
previously at the Federal University of Technology Paraná (UTFPR), Brazil. He
received a Ph.D. in Computer Science from the University of São Paulo (USP —
Brazil). He researches the intersections of Software Engineering (SE) and
Computer Supported Cooperative Work (CSCW). Currently, his research focuses on
the behavior of developers in Open Source Communities, including the support
of newcomers, the impact of Bots in the community, and gender bias in Open
Source Software. His interests include Open Source Software, Human Aspects of
Software Engineering, Empirical Software Engineering, and Mining Software
Repositories techniques.