# Invisible Labor in Open Source Software Ecosystems

John Meluso<EMAIL_ADDRESS>0000-0001-8200-8150 University of Vermont33
Colchester Ave.BurlingtonVermontUSA05405 ,  Amanda Casari  0000-0002-3201-8964
Google, Inc.BurlingtonVermontUSA University of Vermont33 Colchester
Ave.BurlingtonVermontUSA05405<EMAIL_ADDRESS>,  Katie McLaughlin
0000-0003-2684-4220 Google, Inc.MelbourneVictoriaAustralia<EMAIL_ADDRESS>and  Milo Z. Trujillo  0000-0003-0143-5869 University of Vermont33 Colchester
Ave.BurlingtonVermontUSA05405<EMAIL_ADDRESS>
###### Abstract.

Invisible labor is work that is not fully visible, not appropriately
compensated, or both. In open source software (OSS) ecosystems, essential
tasks that do not involve code (like content moderation) often become
invisible to the detriment of individuals and organizations. However,
invisible labor is so difficult to measure that we do not know how much of OSS
activities are invisible. Our study addresses this challenge, demonstrating
that roughly half of OSS work is invisible. We do this by developing a survey
technique with cognitive anchoring that measures OSS developer self-
assessments of labor visibility and attribution. Survey respondents ($n=142$)
reported that their work is more likely to be nonvisible or partially visible
(i.e. visible to at most 1 other person) than fully visible (i.e. visible to 2
or more people). Furthermore, cognitively anchoring participants to the idea
of high work visibility increased perceptions of labor visibility and
decreased visibility importance compared to anchoring to low work visibility.
This suggests that advertising OSS activities as “open” may not make labor
visible to most people, but rather lead contributors to overestimate labor
visibility. We therefore add to a growing body of evidence that designing
systems that recognize all kinds of labor as legitimate contributions is
likely to improve fairness in software development while providing greater
transparency into work designs that help organizations and communities achieve
their goals.

labor, feminism, openness, survey, articulation work, infrastructuring,
contribution, anchoring

††ccs: Software and its engineering Open source model††ccs: Software and its
engineering Maintaining software††ccs: Human-centered computing Empirical
studies in collaborative and social computing††ccs: Human-centered computing
Computer supported cooperative work††ccs: Social and professional topics
Employment issues††ccs: Social and professional topics Computing
organizations††ccs: Social and professional topics Project management
techniques††ccs: Social and professional topics Software maintenance

##  1\. Introduction

Figure 1. (a) Invisible labor is labor that is not fully visible and/or
undercompensated. Labor is not fully visible when labor data doesn’t exist,
isn’t shared, or isn’t accessible. Labor is undercompensated when it doesn’t
receive enough credit, pay, or new opportunities, among others forms of
compensation. (b) Our study demonstrates that roughly two thirds of labor in
OSS is not visible, either by not being visible or only shared with one other
person. We also found that about half of OSS labor activities do not receive
credit, as demonstrated here by the 9 in 10 individuals who do not receive
credit from some fraction of projects they contribute to.

(a) Invisible labor becomes so when it is not visible (because it doesn’t
exist, isn’t shared, or isn’t accessible) and/or when labor is
undercompensated (because it doesn’t receive credit, doesn’t receive pay,
doesn’t facilitate new opportunities, etc.). (b) One third of labor in our
study was visible, one third partially visible, and one third invisible, and
more than half of labor did not receive credit.

The processes of creating and maintaining open source software (OSS) involve
many diverse tasks beyond just contributing code (Giuri et al., 2010; Eghbal,
2016; Casari et al., 2021; Trinkenreich et al., 2020; Geiger et al., 2021).
Conference organizing, community management, finance management, interaction
design, data entry, and documenting have all become essential to OSS
production and repair (Gousios et al., 2008; Eghbal, 2016, 2020; D’Ignazio and
Klein, 2020; Ramin et al., 2020; Young et al., 2021; Trinkenreich et al.,
2020). Despite growing interest in supporting noncode contributions (e.g.
National Science Foundation, 2023), much of this labor remains undocumented,
unattributed to individuals, and “practically invisible” to outsiders and
newcomers (Young et al., 2021; Geiger et al., 2021). This often makes noncode
contributions forms of invisible labor, undercompensated or uncompensated
activities that are “rendered invisible…because they take place out of sight,
or because they lack physical form altogether” (D’Ignazio and Klein, 2020).

Labor invisibility creates significant challenges for individuals and
organizations involved in OSS. For individuals, lower task visibility makes it
difficult to demonstrate the value of their contributions (Suchman, 1995;
D’Ignazio and Klein, 2020). In turn, these individuals receive less
compensation, whether in terms of future opportunities (Riehle, 2007; Babcock
et al., 2017), pay (Toxtli et al., 2021), or social status (Stewart, 2005).
And so these individuals frequently burn out and stop performing crucial
activities (Eghbal, 2020) that a growing number of scientists, organizations,
and governments depend on (Eghbal, 2016). Furthermore, these individuals are
often members of minoritized groups (Frluckaj et al., 2022), thereby
reinforcing demographic representation disparities (Babcock et al., 2017;
D’Ignazio and Klein, 2020).

For organizations and governments, labor invisibility limits work transparency
and therefore task measurement, coordination, and control (Suchman, 1995).
This limits organizations’ knowledge into whom they fund in open source, what
those individuals do, and how those activities shape valued outcomes (Casari
et al., 2021). And although invisibility has some benefits for workers in
these organizing contexts (like anonymity), computing researchers have long
argued that invisibility prevents equity more than creates it (Suchman, 1995;
Star and Strauss, 1999; D’Ignazio and Klein, 2020). Indeed, the reliance of
open source ecosystems on volunteerism necessitates creating communities where
volunteers find benefit in performing all kinds of activities (i.e. not just
coding) (Fang and Neufeld, 2009; von Krogh et al., 2012; Eghbal, 2020).

How to make OSS labor (more) visible then? We likely need to (a) recognize
diverse types of work that individuals perform and (b) measure how much of
those activities people perform. On the former, task taxonomies have seen
modest success. For example, academics have started adopting contributorship
taxonomies like CRediT to increase attribution and transparency (Brand et al.,
2015; Larivière et al., 2016; Holcombe, 2019). Likewise, software developers
are experimenting with manual and automated attribution systems (e.g. All
Contributors and octohatrack, respectively) with some success (Young et al.,
2021). Measuring amounts of labor, though, has proven more elusive,
particularly in OSS. Many automated measurement techniques (like GitHub’s
contribution activity visualizations) only measure code as “contributions,”
thereby conferring lower status to noncoding activities. This provides
limited, often overly reductive assessments of “how much got done” (D’Ignazio
and Klein, 2020; Young et al., 2021) and can be less meaningful to workers
than manual attribution (Monroy-Hernández et al., 2011). And while
computational solutions hold potential for spaces like crowd work (Toxtli et
al., 2021), they seem unlikely to pass muster in open source ecosystems given
the variety of computational and noncomputational interaction contexts
involved (coding, conferences, direct chats, etc.).

To address this gap, this work measures invisible labor in OSS ecosystems
through two contributions. First, we introduce a technique for measuring labor
visibility in open source software development. We use a participatory
feminist approach (Bardzell, 2010; D’Ignazio and Klein, 2020) — namely, asking
workers directly — to understand their expert perceptions of both how often
their work is visible to others and how visible it is to others. We do this by
asking developers how often their work is visible to 2 or more people (“fully
visible”), to 1 other person (“partially visible”), and to no other people
(“nonvisible”). Controlling whether participants see these questions starting
from high visibility ($\geq 2$, 1, 0) or low visibility (0, 1, $\geq 2$) with
survey randomization (Tourangeau et al., 2013) enables us to estimate labor
visibility via cognitive anchoring (Kahneman et al., 1982; Tourangeau, 2018).

Second, we estimate invisible labor in OSS ecosystems by measuring labor
attribution alongside our visibility measurement technique. We anonymously
surveyed 142 open source software developers across varied fields — from
scientific computing to app development — about credit they receive for OSS
tasks, their satisfaction with that credit, and the visibility of their work.
Randomly giving half of the respondents the descending or ascending orders
measured the visibility of the labor they perform and how cognitive anchoring
shapes their perceptions. Our results show that across fields roughly half of
OSS efforts do not receive credit. OSS activities are more likely nonvisible
or partially visible (visible to at most 1 other person) than fully visible
(to 2 or more people). Furthermore, individuals anchored to full visibility
tended to view their own work as more visible and feel more satisfied with the
credit they receive than individuals primed with nonvisibility. This suggests
that people in ecosystems with norms of full visibility may overestimate the
visibility of their work, making attribution, compensation, and governance
inherently more difficult.

We begin by defining labor visibility, invisible labor, and their roles in
Computer-Supported Cooperative Work (CSCW). We then describe our methodology
and results from a global sample of software developers. We close by
discussing what our findings mean for individuals and organizations involved
in OSS along with design considerations for overcoming these challenges. Fig.
1 overviews our findings.

##  2\. Background

To measure something that, by implication, one “cannot see,” it helps to
understand what it means for work to be “visible” and “invisible” in the
context of OSS. In short, labor becomes invisible when (1) labor data is not
fully visible because it does not exist, is not shared, or is not accessible,
and (2) labor is undercompensated. The following sections address these
concepts in turn.

###  2.1. Labor Visibility

When we call work “visible,” we typically do not mean that we can physically
“see” the work as it happens. Rather, visibility is a metaphor that captures
the availability of information about work, approval of that information for
sharing, and accessibility of that information (Stohl et al., 2016; Poster et
al., 2016). In labor contexts, availability means that information about the
tasks being performed has been created and stored somewhere (e.g. via
recordkeeping). Then, those records need to be shared, whether compelled by
the law, social norms, or personal conviction, with someone other than the
creator. Finally, labor information becomes accessible when an individual
knows both where to find the shared information and has sufficient skill to
understand what tasks have been performed. Information about labor must meet
all three of these conditions — it must exist, be shared, and be accessible —
in order for it to be fully visible (Stohl et al., 2016).

More common perhaps is partial visibility, wherein tasks are less documented,
less shared, less reachable, or less understandable. Alternatively, tasks may
be documented but information only shared, reachable, or understandable to
certain individuals (e.g. to experienced group members, but not newcomers)
(Fang and Neufeld, 2009). Partial visibility is rarely due to lack of
competence; to the contrary, specialized work is often the least visible
because experts insulate others from needing to know the steps involved
(Daniels, 1987; Suchman, 1995).

Outside of OSS, academic labor taxonomies like CRediT (Allen et al., 2014;
Brand et al., 2015) exhaustively list individuals’ activities because, even if
a reader witnessed every action that researchers took, readers would have
limited visibility into what it took to conduct a study. Likewise, OSS labor
records (e.g. the Github event stream, octohatrack, and All Contributors)
attribute artifact activities, education, management, maintenance, security,
and other roles (Young et al., 2021). Unfortunately, these systems rarely
provide consistent, standardized recordkeeping (Young et al., 2021) or
accessible sharing of those records (Monroy-Hernández et al., 2011). This
creates a need for new measurement techniques as it leaves much labor
invisible.

###  2.2. Invisible Labor

Invisible labor has become a popular term to describe marginalized forms of
work both inside and outside the home, like childcare (Daniels, 1987), data
entry (D’Ignazio and Klein, 2020), and crowd work (Toxtli et al., 2021). In
one sense, labor becomes invisible if information about the work being done
does not exist, is not shared, cannot be reached, or cannot be understood. Any
of these conditions is sufficient to make information about labor, and
therefore the doing of labor, invisible (Stohl et al., 2016).

Also implied by the concept is that certain tasks tend to be uncompensated or
undercompensated (Acker, 1973; Daniels, 1987; Babcock et al., 2017; D’Ignazio
and Klein, 2020; Toxtli et al., 2021). Individuals who perform less-visible
tasks may struggle to find financial remuneration (D’Ignazio and Klein, 2020;
Toxtli et al., 2021), opportunities for promotion (Riehle, 2007; Babcock et
al., 2017), and to gain social status in their communities (Stewart, 2005).
These activities often fall to women, non-binary people, and other
marginalized individuals as a product of gendered socialization processes
(Daniels, 1987; Berdahl et al., 2018). This makes invisible labor particularly
pernicious to those who seek greater equity in computing (D’Ignazio and Klein,
2020; Casari et al., 2021) and makes compensation the fourth determinant of
labor invisibility.111While others define invisible labor in the context of
paid employment (Poster et al., 2016; Toxtli et al., 2021), they also include
activities that are unpaid but expected and, more generally, conceive of
invisible labor as “activities that are tied to a job and its rewards” and
from which an “employer reaps profits” (Poster et al., 2016, p.7-8). In OSS,
formal employers sometimes do and sometimes do not exist, but many parties
benefit from OSS regardless (e.g. companies, governments, scientists). For
that reason, we argue volunteer activities in OSS ecosystems still merit
inclusion, thereby broadening our conceptualization of “pay” into
“compensation.”  Together, labor is invisible whenever it is not fully
visible, not appropriately compensated, or both (Fig. 1a).

CSCW has long sought to make labor more visible. Star & Strauss (Strauss,
1985; Star, 1991; Star and Strauss, 1999) coined articulation
work222Articulation work is “work that gets things back ‘on track’ in the face
of the unexpected [and] is invisible to rationalized models of work,” a
classic example of support acts (Star, 1991).  in part to raise the visibility
of efforts to support distributed work through mechanism design (e.g. new
organizing forms, technologies, procedures) (Schmidt and Bannon, 1992; Schmidt
and Simone, 1996; Star and Strauss, 1999). More recently, Cherry (Cherry,
2016), Ludwig & colleagues (Ludwig et al., 2018), Gray & Suri (Gray and Suri,
2019), and D’Ignazio & Klein (D’Ignazio and Klein, 2020) revealed other
instances of invisibility in computing. Content moderation, proof reading,
e-waste processing, and countless digitally-tethered activities struggle with
invisibility. In effort to measure invisible labor, Toxtli and colleagues
(Toxtli et al., 2021) recently developed a browser plugin to measure invisible
labor in crowd work, finding that Amazon Mechanical Turkers spend nearly a
third of their time on invisible tasks.

Like with crowd work, OSS development and maintenance activities largely
remain invisible (Suchman, 1995; Eghbal, 2016, 2020; Casari et al., 2021),
with two consequences. First, labor that deserves compensation is not likely
to receive it without greater visibility (Casari et al., 2021). Second, we
cannot understand how OSS ecosystems function without greater visibility into
the activities of OSS (Suchman, 1995; Casari et al., 2021). Lest we assume
this only applies to noncoding activities, consider a coding example too:
Projects often combine substantially differentiated coding activities, like
bug fixing and feature creation, under the label “code contributions” (Casari
et al., 2021). Even activities historically valued as “contributions” are not
fully visible for lack of fidelity, thereby making it difficult to assess the
health of projects. This further motivates the need for new techniques to
measure invisible labor among both code and noncode OSS activities.

##  3\. Methods

Within OSS ecosystems, manual and automated tools have had mixed success at
measuring diverse activities, code and noncode alike. As an alternative, we
used a participatory feminist technique: working with developers to design a
survey, and then directly asking a global sampling of developers about their
OSS activities. The following sections describe how we designed and conducted
the survey.

###  3.1. Survey Instrument Design

We codesigned a Qualtrics web survey (Tourangeau et al., 2013) with practicing
developer advocates (among the coauthors). Given the importances of visibility
and compensation for invisible labor, we dedicated a portion of the survey to
each. We iteratively developed the questions and their responses among
ourselves and with our Institutional Review Board (IRB), which approved the
design before the start of the study.

Broadly, the survey focused participant attention on recent collaborative OSS
projects and the breadth of tasks performed therein. This gave us access to
participant perceptions of tasks beyond code contributions while still
including coding activities. We asked a series of questions about the credit
they receive as one form of compensation. Then, we asked several questions
about how often other people know about the tasks they perform to measure
labor visibility. See Appendix A for the full survey.

####  3.1.1. Guiding Participant Attention

A participant’s experience with a survey begins with recruitment materials.
These materials, along with the consent form (Q1) and other questions
throughout the survey, needed to carefully avoid the term “invisible labor”
because topics related to supportive labor tend to be femininely coded — and
even denigrated — in many global contexts (Berdahl et al., 2018; D’Ignazio and
Klein, 2020). Using the term “invisible labor” in a survey could incentivize
gendered signaling of masculinity and femininity or political signaling.
Instead, we devised other terms to help participants recall the full breadth
of tasks that people perform in their projects. We advertised the survey
around “receiving credit” for their work as this is an important concept to
many in OSS communities (Young et al., 2021) but which should be less likely
to evoke either self-selection into the survey based on the survey focus or
gendered social desirability bias (Krumpal, 2013).

The survey began by prompting the participant to focus on their experiences
with open source projects that they worked on with other people during the
last two years (Q2). We limited the survey to this time period to reduce
experience underretrieval due to excessive cognitive load (Lenzner et al.,
2010; Tourangeau, 2018). Focusing their attention on collaborative open source
projects then gave us access to OSS projects that pose the greatest challenge
to sustainability (Eghbal, 2020).

Later, we focused the participant’s attention on how “tasks vary in shape,
size, and how visible they are to other people” (Q5). We asked them to “think
through the different kinds of tasks you have done as part of these projects
over the past 2 years.” We did this to provide rich retrieval cues about
varied tasks, which tend to improve recall (Tourangeau, 2018). This broadened
their perception out from “just code” to include as many tasks as possible
within their work, thereby bringing their visible and invisible labor to the
fore. For those who already considered noncoding activities to be
contributions, this would not affect their judgment; however, for those who
might consider code the only legitimate type of “contribution” (Eghbal, 2020;
Casari et al., 2021), this design avoided the term contribution (via “task”
instead) while improving information retrieval about diverse tasks
(Tourangeau, 2018).

####  3.1.2. Compensation Questions

We asked participants about the artifacts and use cases of their OSS
collaborations (Q3-4). We did this to measure survey distribution reach,
participant diversity, and compensation variation across those settings. After
discussion with our IRB, we chose not to include other demographic questions
to allow us to distribute the survey globally while protecting varied, nation-
specific participant rights through anonymity.

This followed with several questions about receiving credit for tasks, which
we defined for them as “the project told someone that you did something for
the project” (Q6-10). We asked about how many projects they received credit
from (via 5-point Likert scale), how many tasks they received credit for
(Likert scale), the medium through which they received credit (multiple
select), and their satisfaction with the credit they received (Likert scale).
These questions helped us gauge how much compensation (here, attribution)
individuals receive, how they receive it, and to measure any relationship
strength between attribution and perceptions of visibility.

If we want participant responses to approximate frequencies, it becomes
important to control for perceptual biases that result from different
perceptions of Likert scale terms. Experts suggest linearizing response
options (e.g. explicitly numbering response options with 1 through 5 in
addition to response text) (Tourangeau et al., 2000) to overcome this. So, we
linearized response options for all Likert scales (For example: 1 - Extremely
dissatisfied, 2 - Dissatisfied, 3 - Neither satisfied nor dissatisfied, 4 -
Satisfied, 5 - Extremely satisfied, I’m not sure). We discuss the visibility
questions next.

####  3.1.3. Labor Visibility Questions

Unlike in crowd work where all work might take place on one platform (Toxtli
et al., 2021), OSS activities take place across an ecosystem (Casari et al.,
2021) that makes measurement via tracking difficult. We therefore measured
labor visibility by asking participants how often different numbers of people
knew about tasks they performed. This approximated how visible different tasks
are to OSS communities. Behavioral frequency impressions (e.g. how often
someone performs invisible tasks) are prone to under- and overestimation
(Conrad et al., 1998). While we cannot directly verify their perceptions, our
approach treats their perceptions as expertise (Bardzell, 2010). Combining
randomization with human cognitive biases allowed us to estimate likely bounds
on how much labor individuals perform at different visibility levels.

To that end, we began the labor visibility questions with a prompt (“Think
about all the tasks you performed again,” Q11) to aid recall of their
activities. Then, we presented participants with three questions:

  1. Q12.

How often did 2 or more people know that you performed those tasks?

  2. Q13.

How often did 1 other person know that you performed those tasks?333We chose
to use the phrasing “1 other person” instead of several more and less specific
alternatives. We did this because in open source contexts some participants
may believe that everyone’s work should be visible and therefore accessible.
“Only 1 other person” and “just 1 other person” may invoke social desirability
effects as participants may perceive the words as implying that their work
should be more visible than it is, thereby shaming them into underreporting.
Likewise, the word “exactly” provides a rich cue that might encourage
participants to second guess themselves. However, using merely “1 person”
comes at the risk of ambiguity as to whether or not cases in which $\geq 2$
people know about a task are subsets of cases in which 1 person knows about a
task. We reasoned that “1 other person” provided sufficient specificity
without risking social desirability bias. Furthermore, we enabled the “back”
button in the survey so participants could revise their answers if they
inferred a different meaning after viewing all three questions ($\geq 2$, 1,
0).

  3. Q14.

How often did nobody else know that you performed those tasks?

Information presented earlier in a survey tends to produce assimilation
effects — that is, later judgments are moved in the direction of earlier
judgments (Tourangeau, 2018), reminiscent of the anchoring and adjustment
heuristic or priming effects (Kahneman et al., 1982; Molden, 2014). So if we
only presented Q12, Q13, and Q14 in that order (called $\geq 2$, 1, 0
hereafter for clarity), participant responses would have been likely to bias
results toward presumptions of high visibility (2 or more people, or
“openness”). Therefore, we randomized whether participants saw these questions
in order of what we call anchoring to high visibility ($\geq 2$, 1, 0) or
anchoring to low visibility (0, 1, $\geq 2$).

Following these, we asked participants how important it is to receive credit
for tasks (Q16) along with several free response questions about what did and
did not work well for them about how people give credit.

###  3.2. Study Procedures

To test the survey instrument and measure invisible labor in OSS, we recruited
142 participants between January and June, 2022. We did this by posting IRB-
approved recruitment materials on the social media platform formerly known as
Twitter with the hashtag #opensource. Retweets confirm that the recruitment
messages reached open source participants on at least 5 continents (North
America, South America, Europe, Africa, Australia) and a number of OSS
developer communities (e.g. Python, Django, pan-African OSS, open science).

###  3.3. Validity & Reproducibility

We estimate that, very roughly, between 10 million and 100 million individuals
participate in open source projects (Trujillo et al., 2022). With our sample
of 142 participants, a 20% proportion percentage (based on five or more Likert
scale options per question), and a 90% confidence interval, this gives a 5.5%
margin of error for our survey.

While our results are not a formal psychometric scale, we calculated
reliability estimates on the numeric measures of our survey to assess their
internal consistency using coefficient omega total ($\omega_{total}$), now
considered the most reliable measure, and Cronbach’s alpha ($\alpha$), a more
traditional measure, along with our analyses (McNeish, 2018; Jebb et al.,
2021). Reliability estimates of our numeric data suggest high internal
consistency ($\omega_{Total}=0.84$, $\alpha=0.78$). For intrinsic consistency,
contribution roles, survey data, and code for this project are available at
the following:

> https://github.com/meluso/open-source-credit-survey

##  4\. Results

We performed four analyses with the survey data. We began by examining how
many participants work on different artifacts and use cases to confirm diverse
labor participation in our study. Then, we examined how often participants
receive compensation (specifically credit) and how visible their work is.
Last, we considered the relationships between variables and calculated causal
relationships where possible.

###  4.1. Sample Captures Diverse OSS Labor Activities

Figure 2. Distributions of (a) artifacts and (b) use cases that survey
participants contribute to. Questions were multiple select, so percents shown
represent the fraction of participants who expressed that they work on the
artifacts and use cases shown.

Horizontal bar graphs displaying distributions of the artifacts and use cases
that participants contribute to. At least 30% of participants expressed that
they work on each of libraries, applications, packages, web-related tools, and
frameworks. A significant majority (68%) of respondents work on software
infrastructure, and all other use cases are below 30%.

Study participants reported working on a wide variety of software artifacts
and use cases (Fig. 2). Our survey questions on artifacts and use cases (Q3 &
Q4, respectively) each allowed participants to select multiple answers, so
percents shown represent the proportion of participants involved in the
respective categories.

Each of five categories were selected by more than 30% of the participants:
software libraries, applications, packages, web-related tools, and frameworks.
At least 20% of our sample reported contributing to each of four others
including operating systems, scripts, data science, and programming languages.
Hardware (15%) and something else (13%) saw lower figures, but at least 18
people selected each. Many of the “something else” responses consisted of
documentation, community building, and research tools.

Project use cases were more skewed among participants. Two thirds (68%)
reported working on software infrastructure, though most selected multiple
responses (mean $=2.52$ responses, st. dev. $=1.87$). Six other response
options received at least 20% including business & productivity applications,
scientific research, content management, communication technologies,
education, and browsers. Entertainment, publishing, manufacturing, and other
uses cases all received less than 15%. Here, “something else” included
developer tools, data science, and project management among others.

While our sample of 142 responses clearly cannot represent all OSS activity,
this sample diversity suggests that trends in our data may reveal broader OSS
labor trends.

###  4.2. Half of OSS Labor May Not Receive Attribution

Next, we examine the data from our compensation questions which focused on
receipt of credit for work. The data suggest that participants receive credit
inconsistently for the work they do (Fig. 3a-b) but that they still feel
moderately satisfied, regardless of how much credit they receive (Fig. 3c-d).

Figure 3. Distributions of responses to questions about how often people
receive credit (a) from specific projects and (b) for specific tasks; and
distributions for how satisfied individuals are with (c) how often they
receive credit and (d) the mediums through which they receive credit. Dots and
error bars represent mean values with one standard deviation on the linearized
scales. Colors show how individuals responded to (a), tracking those same
individuals across the other questions.

The figure shows four horizontal bar graphs. (a) and (b) show wide
distributions centered around “Some of them - 3”; (c) a slightly skewed
distribution with most responses on “Satisfied - 4” and “Neither satisfied nor
dissatisfied - 3”; and (d) a skewed distribution with most responses on
“Satisfied - 4”.

Participants reported that they receive credit from approximately half of the
projects they participate in (mean $=3.07$ on a 1 to 5 scale) and for half of
the tasks they perform (mean $=2.96$, also a 1 to 5 scale). Significant
variation existed among the responses though (st. dev. $=1.17$ for projects,
st. dev. $=1.02$ for tasks). Ten percent or less reported receiving credit
from projects and tasks either always or never.

In terms of frequency (Fig. 3c) and medium (Fig. 3d), participants reported
feeling slightly satisfied with the credit they receive. Satisfaction with
credit frequency skewed slightly left, falling just above “Neither satisfied
nor dissatisfied” (mean $=3.36$, st. dev. $=1.02$). Visual inspection confirms
that only about 1 in 5 respondents (19.0%) felt dissatisfied with how often
they receive credit, while about 1 in 3 felt either lukewarm (31.6%) or
satisfied (33.8%). Then, satisfaction with the medium through which
individuals received credit skewed significantly left, falling closer to
“Satisfied” than “Neither satisfied nor dissatisfied” (mean $=3.58$, st. dev.
$=0.88$). Here, almost 1 in 2 (47.2%) reported feeling “Satisfied” while 1 in
4 (23.9%) answered “Neither satisfied nor dissatisfied.” Most of this credit
came in the form of automated metrics (63.1%), project documentation (61.5%),
social media posts (50.8%), and project memberships (47.7%).

Overall, nearly 9 in 10 (89.1%) participants reported that some of their work
is not fully visible (and therefore invisible) while only 1 in 10 (10.9%)
reported full visibility (Fig. 1b).

###  4.3. Anchoring Shifts Perceptions of Labor Visibility

Figure 4. Responses to “How often did {2 or more people, 1 other person,
nobody else} know that you performed those tasks?” as described by response
(a) means and (b-d) distributions. Participants who saw the questions in
ascending order (0,1,$\geq 2$) tended to report that their work is more likely
to be invisible (seen by nobody else) and less likely to be partially
invisible (seen by 1 other person) than those who saw questions in the
descending order ($\geq 2$,1,0). In (a) arrows indicate question viewing
order, and errorbars show 95% confidence interval of difference between means.

(a) shows a plot of points with errorbars and connecting lines indicate the
question viewing direction. (b-d) show vertical bar graphs of the
distributions of responses to each of the two viewing directions.

Turning to labor visibility, participants either viewed questions starting
from an anchor to high visibility (seen by $\geq 2$, 1, then 0 people) or low
visibility (0, 1, $\geq 2$ people). Averaging the results of the two groups
and placing them in proportion to one another (Fig. 1b) approximates how often
labor is invisible (2 in 3, 66.7%) compared to fully visible (1 in 3, 33.3%).
But separately, Fig. 4 shows that the anchors produced statistically
significant differences for the two groups, both in their mean responses on
each question (Fig. 4a) and the distributions of their responses (Fig. 4b-d).
Together, their answers suggest ranges in which the “true” likelihood of labor
visibility may fall.

The group anchored to high visibility (shown in purple) viewed questions in
descending order. They usually reported that $\geq 2$ people knew about their
work “Sometimes” or “Often” over the past two years (Fig. 4b); then that 1
other person would “Often” or “Always” know about their work (Fig. 4c); and
finally that “Rarely” or “Sometimes” would nobody else know about their work
(Fig. 4d). Again, survey respondents tend to answer later survey questions by
making adjustments from earlier questions (Tourangeau, 2018). This progression
suggests the group’s relative perceptions of the labor visibilities; to them,
their work was most likely to be partially visible (1 person), less likely to
be fully visible ($\geq 2$ people), and less likely still to be not visible (0
people). Cumulatively, the cases that guarantee invisibility (when labor is
either not visible or partially visible) is more likely than full visibility.

The group anchored to low visibility (turquoise) significantly contrasted the
previous group. Viewing questions in ascending order, they were more likely to
report their work was “Often,” “Sometimes,” or “Rarely” visible to nobody else
(Fig. 4d); “Sometimes” or “Often” visible to 1 other person (Fig. 4c); and
“Sometimes” or “Rarely” again for $\geq 2$ people (Fig. 4b). Here too, this
group likely made adjustments from their anchor (nonvisibility). Consequently,
this group’s averages suggest that they perceive all three cases — full,
partial, and nonvisibility — as happening roughly as often as one another,
though partial visibility may occur slightly more often than nonvisibility
based on their distinct 95% confidence intervals (Fig. 4a). Here, too, the
cumulative combination of partially visible and nonvisible labor makes
invisibility more likely than fully visible labor.

The two groups roughly agree on how often their OSS labor is fully visible,
revealing that about 2 in 3 tasks are invisible (Fig. 1b). Still, the
differences between groups on nonvisibility and partial visibility are
significant, statistically and in implication. First, the group anchored to
high visibility perceived invisible labor as less common than the low
visibility anchored group, suggesting that anchoring may influence perceptions
of labor visibility. This appears again in the difference between the partial
visibility means: both groups perceived partial visibility as more likely than
their respective anchors, but the groups differed on the likelihood depending
on the initial framing. The qualitative differences between the response
distributions in Figs. 4b & c further support this.

###  4.4. Visibility Anchoring May Shift Beliefs About the Importance of
Credit

Figure 5. Correlations between numerical survey variables, ordered by
clustering. Colored boxes are statistically significant ($p<0.01$) while gray
boxes are not. Being seen by nobody else (I) was correlated with low scores on
several other variables. Conversely, being seen by $\geq 2$ people (D) was
often correlated with high scores on other variables. Being seen by 1 other
person yielded mixed results.

A heatmap grid of correlation values with variables listed across the vertical
and horizontal axes. More positively correlated relationships are colored
increasingly darker shades of red; more negatively correlated relationships
are colored increasingly darker shades of blue; uncorrelated relationships are
colored white; and statistically insignificant relationships are light gray.

Finally, we examine relationships between the quantitative variables of our
survey. Calculating Pearson correlation coefficients between these variables
reveals several trends (Fig. 5). Unsurprisingly, the likelihoods of receiving
credit for specific tasks and from specific projects are highly positively
correlated ($r=0.87$), as were participants’ satisfactions with the credit
they receive ($r=0.73$, compare Fig. 3c & d). Credit frequency and participant
satisfaction tended to correlate with one another (A, B, E, & F) and with full
visibility (D). Partial visibility (C) also correlates with the former but
less so, while negatively correlating with anchoring order (H, $r=-0.30$) and
being not visible (I, $r=-0.27$). Indeed, negative correlations proved common
for nonvisibility (A-F) and only positively correlated with anchoring to low
visibility (H, $r=0.30)$. Anchoring to low visibility also correlated with
importance of receiving credit (G, $r=0.26$), which in turn correlated with
both credit frequencies ($r=0.23$ & $r=0.20$, respectively).

Again, invisible labor consists of both visibility and compensation. To
understand the relationship between these elements in OSS (here, with credit
as compensation), we calculated a weighted average of the visibility measures
(fully, partially, & nonvisible) and separately averaged the frequencies of
receiving credit (credit for tasks, credit from projects). Fig. 6a shows the
relationship between average visibility and average credit frequency for each
participant ($r=0.53$). While this is correlative and not causative,
visibility and credit clearly trend together. Few individuals’ responses
significantly deviated from this trend. Calculating the average of averages
for each dimension provides further confirmation that, in general, OSS labor
is only partially visible (51.9%) and roughly half receives credit (50.1%).

The trend between visibility and credit becomes more important in the context
of one last observation. Recall that the importance of receiving credit (Fig.
5, G) correlated with anchoring order (H). The importance of receiving credit
(Q16) appeared after the visibility questions (Q12-14) meaning that anchoring
order had the potential to yield causal effects for subsequent questions. Fig.
6b shows that anchoring had a statistically significant effect on how
important credit was to participants. On average, participants anchored to low
visibility thought receiving credit was $30.5\%$ more important than
participants anchored to high visibility ($95\%$ confidence interval of $\pm
19.5\%$, $p<0.002$, Adj. $R^{2}=0.061$ with robust standard errors). We
discuss the implications of this and our other findings next.

Figure 6. (a) Scatterplot of average visibility versus average credit
frequency. Points indicate individual responses. The regression line shows the
correlation between variables. (b) Linear regression of credit importance
against visibility anchoring. Points indicate individual responses and are
grouped on the x-axis by presentation order. The regression line of best fit
shows a statistically significant positive relationship between labor
invisibility (via anchoring) and greater importance of credit (Adj.
$R^{2}=0.061$).

Two scatterplots, one of average labor visibility versus average credit
frequency with a positive trend. The second regresses how important receiving
credit is to participants against question presentation order. Here, the
x-axis is broken out by presentation order (decreasing on the left, increasing
on the right) and the y-axis shows importance responses from 1 (not important)
to 5 (extremely important). A regression line of best fit is overlaid showing
a slight slope upward from left to right.

##  5\. Discussion

###  5.1. Findings & Implications

Labor becomes invisible labor when it is undercompensated, not fully visible,
or both. Through a labor-diverse sample of OSS ecosystems, we found that
invisible labor seems to pervade OSS ecosystems (Fig. 1b). Participants
reported receiving credit for roughly half of their work. While this finding
cannot speak to other forms of compensation (e.g. pay, social status,
opportunities), credit is one of the least financially and socially expensive
forms of compensation suggesting that this compensation rate is potentially an
upper bound on compensation frequency.

Participants reported feeling only slightly satisfied with how often they
receive credit and the mediums through which they receive it. This slight
satisfaction suggests a possible explanation for why many individuals do not
seek more credit. Many who were less satisfied with how much credit they
receive listed sources of credit similar to those who were satisfied. However,
satisfied individuals cited several sources that dissatisfied individuals
cited less often including blogs (47.9% for satisfied, 5.3% for dissatisfied),
presentations (37.5% vs. 10.5%), and project documentation (64.6% vs. 42.1%).
It is possible, then, that these mediums might increase the efficacy of
sharing information or the accessibility of information about OSS labor.

We then found that visibility anchoring shapes perceptions of the visibility
of a person’s work. Asking participants about highly visible work (seen by
$\geq 2$ people) first in our series of questions increased participant
frequency perceptions of partially visible work (seen by 1 other person) and
decreased frequency perceptions of low visibility work (seen by nobody else)
compared to those whom we asked about low visibility first.

Our finding that visibility anchoring shifts labor visibility perceptions has
two implications for OSS ecosystems. OSS communities tend to presume that work
is visible because “openness means visibility” so to speak. This is not
necessarily the case because visibility requires information to exist, get
shared, and be accessible. As we have seen, OSS labor is only partially
visible on average and roughly half does not receive credit. So presuming that
openness creates visibility may actually bias OSS participants into perceiving
invisible labor as less common than it is, thereby decreasing compensation
efforts. The finding that visibility anchoring reduces credit importance
perceptions further supports this likelihood. In addition, participants
generally viewed partial visibility (seen by 1 other person) as more common
than full visibility or nonvisibility. This confirms the precariousness of OSS
labor noted by others (Eghbal, 2020) — here in the form of OSS labor
visibility — for two reasons. First, if the singular external viewer of a task
loses visibility into that task — by promotion, leaving the community, or even
just forgetting — that labor becomes unknown to others. Second, some labor is
already seen by nobody other than the worker. Either way, these tasks are
invisible and therefore difficult for others to understand, compensate, and
manage. Assuming this work is valuable to an OSS ecosystem, it reinforces the
importance of accessible recordkeeping.

The correlation and regression results also imply that making OSS labor more
visible probably requires giving individuals credit more often and more
consistently. Yet even those who always receive credit did not report their
work being completely visible. So though not all work should be fully visible,
we join others who take the feminist position that all work should receive
compensation (D’Ignazio and Klein, 2020). Invisible work — completely unknown
to others — cannot receive compensation from anyone other than the individual
performing the labor. If individuals have sufficient means (e.g. wealth,
social status), they may be able to compensate themselves for performing
invisible tasks, but most people cannot. We see this already in OSS when
privileged individuals receive credit for activities that surpass community-
defined thresholds of social status or centrality while less privileged
individuals do not receive such credit (D’Ignazio and Klein, 2020; Young et
al., 2021).

To remedy all of this, OSS ecosystems likely need more robust qualitative and
quantitative documentation so that labor data exists; more effective
dissemination practices so labor data gets shared; more easily understood data
communication so labor data becomes accessible; and more credit in valued
forms so all labor receives compensation. Treating all work as contributions
may be an effective way to start (Casari et al., 2021; Allen et al., 2014).

###  5.2. Limitations & Future Work

Our work has several limitations. First of all, survey instruments tend to
prove more reliable in the form of validated scales. Questions remain about
how feasible creating a scale would be for something so difficult to trace
which motivated the exploratory nature of this study. That said, existing
information visibility scales (ter Hoeven et al., 2021) may guide the labor
visibility portion, leaving only need for compensation measures. Next, our
sample sizes are too small to make these findings conclusive about all of OSS.
In particular, we did not see differences between communities as we might have
expected; breaking down our responses by artifacts and use cases did not show
statistical differences between category means on any of the quantitative
questions (0 of 16, 0%). Sampling OSS ecosystems broadly as we did here may
prove less helpful going forward than conducting targeted studies to benefit
particular communities, though. So future efforts might focus on specific
artifacts, use cases, geographic regions, or programming language communities.
Third, our sample of convenience via social media prevented us from engaging
with those who use social media less frequently meaning our sample may contain
biases toward social media users and more socially-engaged ecosystems.

Perhaps the greatest need arising from this work is a granular understanding
of how specific artifacts facilitate the existence of labor data, its sharing,
accessibility, and participant compensation. For example, we saw that certain
attribution mediums corresponded to greater credit satisfaction. Further
research might confirm whether appropriately selecting credit mediums improves
accessibility of attribution and, in turn, improves worker satisfaction. These
artifacts also shape worker perceptions of visibility, so researchers and
practitioners likely need to pay careful attention to the interplay between
accessibility and perceptions of openness. Labor will not become fully visible
without ensuring consistent support for each factor that limits labor
visibility and compensation, so articulation work and infrastructuring
activities would benefit from renewed attention.

And of course, one study of invisibility will not make all labor visible, so
future work might triangulate our findings within OSS through other
qualitative and quantitative methods. Studies of invisible labor need not
limit themselves to OSS contexts either as invisible labor happens everywhere,
in computing and more broadly. For CSCW, this article provides further
evidence in support of what Toxtli & colleagues (Toxtli et al., 2021) recently
found in crowd work. We hope others will similarly build on our work in OSS
and other domains.

##  6\. Conclusion

Much of the work that happens in OSS ecosystems is not visible, not
compensated, or both. Such invisible labor makes it difficult for individuals
to advance their careers and for organizations to understand processes that
enable their organizations to run. In this study, we measured invisible labor
in OSS ecosystems by estimating the visibility and compensation rates of OSS
activities. We found that approximately half of OSS labor does not receive
credit, a low-cost form of attribution that often precedes other forms of
compensation, and that OSS activities are more likely to be nonvisible or
partially visible than to be fully visible. In addition, we demonstrated that
anchoring labor to the concept of visibility tends to increase visibility
frequency perceptions and decrease credit importance assessments. These
findings tell us that compensating all those who work in OSS ecosystems will
require careful attention to documenting labor activities, as well as sharing
and making documentation accessible.

###### Acknowledgements.

Thanks to Jean-Gabriel Young and Melania Lavric for their suggestions
throughout. The authors were partially supported by Google under the Open
Source Complex Ecosystems and Networks project. J.M. was partially supported
by the Sloan Foundation through the Vermont Research Open Source Program
Office, and by the NSF award BCS-2332598.

## References

  * (1)
  * Acker (1973) Joan Acker. 1973.  Women and Social Stratification: A Case of Intellectual Sexism.  _Amer. J. Sociology_ 78, 4 (Jan. 1973), 936–945.  https://doi.org/10.1086/225411
  * Allen et al. (2014) Liz Allen, Jo Scott, Amy Brand, Marjorie Hlava, and Micah Altman. 2014.  Publishing: Credit Where Credit Is Due.  _Nature_ 508, 7496 (April 2014), 312–313.  https://doi.org/10.1038/508312a
  * Babcock et al. (2017) Linda Babcock, Maria P Recalde, Lise Vesterlund, and Laurie Weingart. 2017.  Gender Differences in Accepting and Receiving Requests for Tasks with Low Promotability.  _American Economic Review_ 107, 3 (2017), 714–747.
  * Bardzell (2010) Shaowen Bardzell. 2010.  Feminist HCI: Taking Stock and Outlining an Agenda for Design. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’10)_. Association for Computing Machinery, New York, NY, USA, 1301–1310.  https://doi.org/10.1145/1753326.1753521
  * Berdahl et al. (2018) Jennifer L. Berdahl, Marianne Cooper, Peter Glick, Robert W. Livingston, and Joan C. Williams. 2018.  Work as a Masculinity Contest.  _Journal of Social Issues_ 74, 3 (2018), 422–448.  https://doi.org/10.1111/josi.12289
  * Brand et al. (2015) Amy Brand, Liz Allen, Micah Altman, Marjorie Hlava, and Jo Scott. 2015.  Beyond Authorship: Attribution, Contribution, Collaboration, and Credit.  _Learned Publishing_ 28, 2 (2015), 151–155.  https://doi.org/10.1087/20150211
  * Casari et al. (2021) Amanda Casari, Katie McLaughlin, Milo Z. Trujillo, Jean-Gabriel Young, James P. Bagrow, and Laurent Hébert-Dufresne. 2021.  Open Source Ecosystems Need Equitable Credit across Contributions.  _Nature Computational Science_ 1, 1 (Jan. 2021), 2–2.  https://doi.org/10.1038/s43588-020-00011-w
  * Cherry (2016) Miriam A Cherry. 2016.  Virtual Work and Invisible Labor.  In _Invisible Labour: Hidden Work in the Contemporary World_ , Marion G. Crain, Winifred R. Poster, and Miriam A. Cherry (Eds.). University of California Press, Oakland, California, 28–46.
  * Conrad et al. (1998) Frederick G. Conrad, Norman R. Brown, and Erin R. Cashman. 1998.  Strategies for Estimating Behavioural Frequency in Survey Interviews.  _Memory_ 6, 4 (July 1998), 339–366.  https://doi.org/10.1080/741942603
  * Daniels (1987) Arlene Kaplan Daniels. 1987.  Invisible Work.  _Social Problems_ 34, 5 (Dec. 1987), 403–415.  https://doi.org/10.2307/800538
  * D’Ignazio and Klein (2020) Catherine D’Ignazio and Lauren F Klein. 2020.  _Data Feminism_.  MIT Press, Cambridge, MA.
  * Eghbal (2016) Nadia Eghbal. 2016.  _Roads and Bridges: The Unseen Labor behind Our Digital Infrastructure_.  Technical Report. Ford Foundation.
  * Eghbal (2020) Nadia Eghbal. 2020.  _Working in Public: The Making and Maintenance of Open Source Software_.  Stripe Press, San Francisco, CA.
  * Fang and Neufeld (2009) Yulin Fang and Derrick Neufeld. 2009.  Understanding Sustained Participation in Open Source Software Projects.  _Journal of Management Information Systems_ 25, 4 (April 2009), 9–50.  https://doi.org/10.2753/MIS0742-1222250401
  * Frluckaj et al. (2022) Hana Frluckaj, Laura Dabbish, David Gray Widder, Huilian Sophie Qiu, and James D. Herbsleb. 2022.  Gender and Participation in Open Source Software Development.  _Proc. ACM Hum.-Comput. Interact._ 6, CSCW2 (Nov. 2022), 299:1–299:31.  https://doi.org/10.1145/3555190
  * Geiger et al. (2021) R. Stuart Geiger, Dorothy Howard, and Lilly Irani. 2021.  The Labor of Maintaining and Scaling Free and Open-Source Software Projects.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1 (April 2021), 175:1–175:28.  https://doi.org/10.1145/3449249
  * Giuri et al. (2010) Paola Giuri, Matteo Ploner, Francesco Rullani, and Salvatore Torrisi. 2010.  Skills, Division of Labor and Performance in Collective Inventions: Evidence from Open Source Software.  _International Journal of Industrial Organization_ 28, 1 (Jan. 2010), 54–68.  https://doi.org/10.1016/j.ijindorg.2009.07.004
  * Gousios et al. (2008) Georgios Gousios, Eirini Kalliamvakou, and Diomidis Spinellis. 2008.  Measuring Developer Contribution from Software Repository Data. In _Proceedings of the 2008 International Working Conference on Mining Software Repositories_ _(MSR ’08)_. Association for Computing Machinery, New York, NY, USA, 129–132.  https://doi.org/10.1145/1370750.1370781
  * Gray and Suri (2019) Mary L. Gray and Siddharth Suri. 2019.  _Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass_.  Houghton Mifflin Harcourt, Boston.
  * Holcombe (2019) Alex O. Holcombe. 2019.  Contributorship, Not Authorship: Use CRediT to Indicate Who Did What.  _Publications_ 7, 3 (Sept. 2019), 48.  https://doi.org/10.3390/publications7030048
  * Jebb et al. (2021) Andrew T. Jebb, Vincent Ng, and Louis Tay. 2021.  A Review of Key Likert Scale Development Advances: 1995–2019.  _Frontiers in Psychology_ 12 (2021), 1590.  https://doi.org/10.3389/fpsyg.2021.637547
  * Kahneman et al. (1982) D Kahneman, Paul Slovic, and Amos Tversky. 1982.  _Judgment under Uncertainty: Heuristics and Biases_.  Cambridge University Press., Cambridge, UK.
  * Krumpal (2013) Ivar Krumpal. 2013.  Determinants of Social Desirability Bias in Sensitive Surveys: A Literature Review.  _Qual Quant_ 47, 4 (June 2013), 2025–2047.  https://doi.org/10.1007/s11135-011-9640-9
  * Larivière et al. (2016) Vincent Larivière, Nadine Desrochers, Benoît Macaluso, Philippe Mongeon, Adèle Paul-Hus, and Cassidy R Sugimoto. 2016.  Contributorship and Division of Labor in Knowledge Production.  _Social Studies of Science_ 46, 3 (2016), 417–435.  arXiv:26099848
  * Lenzner et al. (2010) Timo Lenzner, Lars Kaczmirek, and Alwine Lenzner. 2010.  Cognitive Burden of Survey Questions and Response Times: A Psycholinguistic Experiment.  _Applied Cognitive Psychology_ 24, 7 (Oct. 2010), 1003–1020.  https://doi.org/10.1002/acp.1602
  * Ludwig et al. (2018) Thomas Ludwig, Volkmar Pipek, and Peter Tolmie. 2018.  Designing for Collaborative Infrastructuring: Supporting Resonance Activities.  _Proc. ACM Hum.-Comput. Interact._ 2, CSCW (Nov. 2018), 1–29.  https://doi.org/10.1145/3274382
  * McNeish (2018) Daniel McNeish. 2018.  Thanks Coefficient Alpha, We’ll Take It from Here.  _Psychological Methods_ 23, 3 (Sept. 2018), 412–433.  https://doi.org/10.1037/met0000144
  * Molden (2014) Daniel C. Molden. 2014.  Understanding Priming Effects in Social Psychology: What Is “Social Priming” and How Does It Occur?  _Social Cognition_ 32, Supplement (June 2014), 1–11.  https://doi.org/10.1521/soco.2014.32.supp.1
  * Monroy-Hernández et al. (2011) Andrés Monroy-Hernández, Benjamin Mako Hill, Jazmin Gonzalez-Rivero, and Danah Boyd. 2011.  Computers Can’t Give Credit: How Automatic Attribution Falls Short in an Online Remixing Community. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’11)_. Association for Computing Machinery, Vancouver, BC, Canada, 3421–3430.  https://doi.org/10.1145/1978942.1979452 arXiv:1507.01285
  * National Science Foundation (2023) National Science Foundation. 2023.  _Pathways to Enable Open-Source Ecosystems_.  Program Solicitation NSF 23-556. National Science Foundation.
  * Poster et al. (2016) Winifred R. Poster, Marion Crain, and Miriam A. Cherry. 2016.  Introduction: Conceptualizing Invisible Labor.  In _Invisible Labor: Hidden Work in the Contemporary World_ , Marion G. Crain, Winifred R. Poster, and Miriam A. Cherry (Eds.). University of California Press, Oakland, California, 3–27.
  * Ramin et al. (2020) Frederike Ramin, Christoph Matthies, and Ralf Teusner. 2020.  More than Code: Contributions in Scrum Software Engineering Teams. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ _(ICSEW’20)_. Association for Computing Machinery, New York, NY, USA, 137–140.  https://doi.org/10.1145/3387940.3392241
  * Riehle (2007) Dirk Riehle. 2007.  The Economic Motivation of Open Source Software: Stakeholder Perspectives.  _Computer_ 40, 4 (April 2007), 25–32.  https://doi.org/10.1109/MC.2007.147
  * Schmidt and Bannon (1992) Kjeld Schmidt and Liam Bannon. 1992.  Taking CSCW Seriously.  _Comput Supported Coop Work_ 1, 1 (March 1992), 7–40.  https://doi.org/10.1007/BF00752449
  * Schmidt and Simone (1996) Kjeld Schmidt and Carla Simone. 1996.  Coordination Mechanisms: Towards a Conceptual Foundation of CSCW Systems Design.  _Computer Supported Cooperative Work (CSCW)_ 5 (1996), 155–200.
  * Star (1991) Susan Leigh Star. 1991.  The Sociology of the Invisible: The Primacy of Work in the Writings of Anselm Strauss.  In _Social Organization and Social Process: Essays in Honor of Anselm Strauss_ , David R. Maines (Ed.). Walter de Gruyter, Inc., Hawthorne, NY, 265–283.
  * Star and Strauss (1999) Susan Leigh Star and Anselm Strauss. 1999.  Layers of Silence, Arenas of Voice: The Ecology of Visible and Invisible Work.  _Computer supported cooperative work (CSCW)_ 8 (1999), 9–30.
  * Stewart (2005) Daniel Stewart. 2005.  Social Status in an Open-Source Community.  _Am Sociol Rev_ 70, 5 (Oct. 2005), 823–842.  https://doi.org/10.1177/000312240507000505
  * Stohl et al. (2016) Cynthia Stohl, Michael Stohl, and Paul M. Leonardi. 2016.  Managing Opacity: Information Visibility and the Paradox of Transparency in the Digital Age.  _International Journal of Communication_ 10, 0 (Jan. 2016), 15.
  * Strauss (1985) Anselm Strauss. 1985.  Work and the Division of Labor.  _The Sociological Quarterly_ 26, 1 (1985), 1–19.  https://doi.org/10.1111/j.1533-8525.1985.tb00212.x
  * Suchman (1995) Lucy Suchman. 1995.  Making Work Visible.  _Commun. ACM_ 38, 9 (Sept. 1995), 56–64.  https://doi.org/10.1145/223248.223263
  * ter Hoeven et al. (2021) Claartje L. ter Hoeven, Cynthia Stohl, Paul Leonardi, and Michael Stohl. 2021.  Assessing Organizational Information Visibility: Development and Validation of the Information Visibility Scale.  _Communication Research_ 48, 6 (Aug. 2021), 895–927.  https://doi.org/10.1177/0093650219877093
  * Tourangeau (2018) Roger Tourangeau. 2018.  The Survey Response Process from a Cognitive Viewpoint.  _Quality Assurance in Education_ 26, 2 (Jan. 2018), 169–181.  https://doi.org/10.1108/QAE-06-2017-0034
  * Tourangeau et al. (2013) Roger Tourangeau, Frederick G Conrad, and Mick P Couper. 2013.  _The Science of Web Surveys_.  Oxford University Press, New York, NY.
  * Tourangeau et al. (2000) Roger Tourangeau, Lance J Rips, and Kenneth Rasinski. 2000.  _The Psychology of Survey Response_.  Cambridge University Press, Cambridge, UK.
  * Toxtli et al. (2021) Carlos Toxtli, Siddharth Suri, and Saiph Savage. 2021.  Quantifying the Invisible Labor in Crowd Work.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW2 (Oct. 2021), 319:1–319:26.  https://doi.org/10.1145/3476060
  * Trinkenreich et al. (2020) Bianca Trinkenreich, Mariam Guizani, Igor Wiese, Anita Sarma, and Igor Steinmacher. 2020.  Hidden Figures: Roles and Pathways of Successful OSS Contributors.  _Proc. ACM Hum.-Comput. Interact._ 4, CSCW2 (Oct. 2020), 1–22.  https://doi.org/10.1145/3415251
  * Trujillo et al. (2022) Milo Z. Trujillo, Laurent Hébert-Dufresne, and James Bagrow. 2022.  The Penumbra of Open Source: Projects Outside of Centralized Platforms Are Longer Maintained, More Academic and More Collaborative.  _EPJ Data Sci._ 11, 1 (Dec. 2022), 31.  https://doi.org/10.1140/epjds/s13688-022-00345-7
  * von Krogh et al. (2012) Georg von Krogh, Stefan Haefliger, Sebastian Spaeth, and Martin W. Wallin. 2012.  Carrots and Rainbows: Motivation and Social Practice in Open Source Software Development.  _MIS Quarterly_ 36, 2 (2012), 649–676.  https://doi.org/10.2307/41703471 arXiv:41703471
  * Young et al. (2021) Jean-Gabriel Young, Amanda Casari, Katie McLaughlin, Milo Z. Trujillo, Laurent Hébert-Dufresne, and James P. Bagrow. 2021.  Which Contributions Count? Analysis of Attribution in Open Source. In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, Madrid, Spain, 242–253.  https://doi.org/10.1109/MSR52588.2021.00036

##  Appendix A Invisible Labor Survey

The following are the complete survey questions as constructed in Qualtrics. A
consent form (Q1) preceded these questions with the title “Open Source Credit
Study” and an invitation to participate in a research study “about how people
receive credit for the tasks they do as part of open source projects.”
Italicized text indicates a question response constraint (e.g. a multiple
choice question versus a multiple select question).

  1. Q1.

Consent form (Click to consent)

  2. Q2.

The questions in this survey ask about your experiences with open source
projects. Specifically, the questions ask about projects you worked on during
the last 2 years, and that at least one other person also worked on. (no
response requested)

  3. Q3.

The following categories describe things that projects either create or work
on. Which categories apply to the projects you worked on? Select all that
apply. (multiple select)

     * •

Hardware

     * •

Operating systems

     * •

Languages

     * •

Frameworks

     * •

Libraries

     * •

Packages

     * •

Applications

     * •

Data science

     * •

Scripts

     * •

Web-related

     * •

Something else (with text field)

  4. Q4.

Projects create for different use cases. Which cases apply to the projects you
worked on? Select all that apply. (multiple select)

     * •

Browsers

     * •

Business & productivity applications

     * •

Communication technologies

     * •

Content management

     * •

Education

     * •

Entertainment

     * •

Manufacturing

     * •

Publishing

     * •

Scientific research

     * •

Software infrastructure

     * •

Something else (with text field)

  5. Q5.

Tasks vary in shape, size, and how visible they are to other people. For about
30 seconds, think through the different kinds of tasks you have done as part
of these projects over the past 2 years. When you are ready, click on the
right arrow to go on to the next question. (no response requested)

  6. Q6.

The following questions ask about receiving credit. When we talk about having
received credit for a task, we mean that the project told someone that you did
something for the project. Those people may also work on the project or may be
outside of the project. (no response requested)

  7. Q7.

In the past two years, from how many projects did you receive credit for your
tasks? (multiple choice)

     * •

1 - None of them

     * •

2 - Few of them

     * •

3 - Some of them

     * •

4 - Most of them

     * •

5 - All of them

     * •

I’m not sure

  8. Q8.

In the past two years, for how many of your tasks did you receive credit?
(multiple choice)

     * •

1 - None of them

     * •

2 - Few of them

     * •

3 - Some of them

     * •

4 - Most of them

     * •

5 - All of them

     * •

I’m not sure

  9. Q9.

Projects give people credit for tasks through different mediums. Through what
mediums did you receive credit? Select all that apply. (multiple select)

     * •

Automated metrics (e.g. GitHub Contributors)

     * •

Project membership (e.g. GitHub Organizations)

     * •

Project documentation (e.g. acknowledgement lists, release notes)

     * •

Standardized tools (e.g. All Contributors, CRediT)

     * •

Social media (e.g. Twitter, Facebook)

     * •

Blogs (e.g. Medium, personal websites)

     * •

Presentations (e.g. talks, conferences)

     * •

Something else (with text field)

     * •

I’m not sure

  10. Q10.

How satisfied are you with the mediums through which you receive credit?
(multiple choice)

     * •

1 - Extremely dissatisfied

     * •

2 - Dissatisfied

     * •

3 - Neither satisfied nor dissatisfied

     * •

4 - Satisfied

     * •

5 - Extremely satisfied

  11. Q11.

Think about all the tasks you performed again. (no response requested)

  12. Q12.

(Randomized) How often did 2 or more people know that you performed those
tasks? (multiple choice)

     * •

1 - Never

     * •

2 - Rarely

     * •

3 - Sometimes

     * •

4 - Often

     * •

5 - Always

     * •

I’m not sure

  13. Q13.

How often did 1 other person know that you performed those tasks? (multiple
choice)

     * •

1 - Never

     * •

2 - Rarely

     * •

3 - Sometimes

     * •

4 - Often

     * •

5 - Always

     * •

I’m not sure

  14. Q14.

(Randomized) How often did nobody else know that you performed those tasks?
(multiple choice)

     * •

1 - Never

     * •

2 - Rarely

     * •

3 - Sometimes

     * •

4 - Often

     * •

5 - Always

     * •

I’m not sure

  15. Q15.

How satisfied are you with how many of your tasks received credit? (multiple
choice)

     * •

1 - Extremely dissatisfied

     * •

2 - Dissatisfied

     * •

3 - Neither satisfied nor dissatisfied

     * •

4 - Satisfied

     * •

5 - Extremely satisfied

  16. Q16.

How important is it to you to receive credit for the tasks you do? (multiple
choice)

     * •

1 - Not at all important

     * •

2 - Slightly important

     * •

3 - Moderately important

     * •

4 - Very important

     * •

5 - Extremely important

  17. Q17.

Think about how the projects you worked on gave people credit. From your
perspective, what worked well about how they gave people credit? (free
response field)

  18. Q18.

What did not work well about how they gave people credit? (free response
field)

  19. Q19.

Is there anything else that you’d like to share with us? (free response field)