∎ =.1ex

11institutetext:  ✉ Corresponding author - Vittunyuta Maeprasart
22institutetext: Supatsara Wattanakriengkrai 33institutetext: Raula Gaikovina
Kula 44institutetext: Kenichi Matsumoto 55institutetext: Nara Institute of
Science and Technology, Japan
55email: {maeprasart.vittunyuta.mn2, wattanakri.supatsara.ws3, raula-k,
<EMAIL_ADDRESS>
Christoph Treude 66institutetext: University of Melbourne, Australia
66email<EMAIL_ADDRESS>
# Understanding the Role of External Pull Requests in the NPM Ecosystem

Vittunyuta Maeprasart  Supatsara Wattanakriengkrai  Raula Gaikovina Kula
Christoph Treude  Kenichi Matsumoto

(Received: date / Accepted: date)

###### Abstract

The risk to using third-party libraries in a software application is that much
needed maintenance is solely carried out by library maintainers. These
libraries may rely on a core team of maintainers (who might be a single
maintainer that is unpaid and overworked) to serve a massive client user-base.
On the other hand, being open source has the benefit of receiving
contributions (in the form of External PRs) to help fix bugs and add new
features. In this paper, we investigate the role by which External PRs
(contributions from outside the core team of maintainers) contribute to a
library. Through a preliminary analysis, we find that External PRs are
prevalent, and just as likely to be accepted as maintainer PRs. We find that
26.75% of External PRs submitted fix existing issues. Moreover, fixes also
belong to labels such as breaking changes, urgent, and on-hold. Differently
from Internal PRs, External PRs cover documentation changes (44 out of 384
PRs), while not having as much refactoring (34 out of 384 PRs). On the other
hand, External PRs also cover new features (380 out of 384 PRs) and bugs (120
out of 384). Our results lay the groundwork for understanding how maintainers
decide which external contributions they select to evolve their libraries and
what role they play in reducing the workload.

###### Keywords:

Third-party libraries, Pull Requests, OSS sustainability, Software Ecosystems

##  1 Introduction and Motivation

Third-party libraries provide a means by which software teams can quickly
build an application, avoiding the need to start from scratch. Popular usage
has led to the explosion of third-party library inter-connecting networks of
dependencies that form large ecosystems (Islam et al, 2021; Decan et al, 2018;
He et al, 2021). For example, the NPM ecosystem of libraries (aka packages) is
a critical part of the JavaScript world, with open source contributions from
hundreds of thousands of open source developers and maintainers. Evidence of
its impact was its purchase by Microsoft’s GitHub in early 2020 (Friedman,
2020). Despite their popularity, libraries are prone to maintenance and
evolution issues. In particular, threats such as vulnerabilities (Chinthanet
et al, 2021; Durumeric et al, 2014) and transitive dependency changes (Dey et
al, 2019) may cause a library to risk becoming obsolete (Kula et al, 2018), or
worse, pose an important security risk.

Recent events such as the Log4Shell vulnerability (Nichols, 2022; Berger,
2021) and the faker sabotage (Sharma, 2022) illustrate concerns about how
maintainers become overwhelmed with their workload. This is especially the
case when there is a small core maintainer team (sometimes a single
maintainer) that is depended upon by a massive user base. For instance, it was
reported that ‘a developer of two popular libraries, decided to inject
malicious code into them, citing that the reason behind this mischief on the
developer’s part appears to be retaliation—against mega-corporations and
commercial consumers of open-source projects who extensively rely on cost-free
and community-powered software but do not, according to the developer, give
back to the community’ (Roth, 2022). Another example is when the maintainers
of a vulnerable library expressed their frustration when maintaining a library
in a tweet, ‘… maintainers have been working sleeplessly on mitigation
measures; fixes, docs, CVE, replies to inquiries, etc. Yet nothing is stopping
people to bash us, for work we aren’t paid for, for a feature we all dislike
yet needed to keep due to backward compatibility concerns…’ (Yazıcı, 2021).

Recent initiatives such as the Alpha-Omega project allow industry (Google and
Microsoft) to partner with maintainers to systematically find and fix
vulnerabilities that have not yet been discovered in open source code
(OpenSSF, 2022). Although these critical contributions are indeed significant,
little is known about the other external contributions that these libraries
constantly receive from the ecosystem. Although prior work shows that external
contributions are needed, analysis is at a higher level of granularity. For
instance, Samoladas et al (2010) show that OSS projects require a constant
stream of contributions, while Nakakoji et al (2003) and Raymond (1999) depict
external contributions as more casual contributions. In this light, it is
unknown how external contributions ease maintainer workload, and how they
differ from maintainer contributions (i.e., fixing bugs, new features,
documentation, etc.).

Figure 1: An External PR which is linked to an issue using the keyword solves.

In this study, we analyze contributions submitted as Pull Requests (PRs) in
the NPM ecosystem. Based on prior work (Valiev et al, 2018), we classify a PR
contribution based on the access level (i.e., can merge a PR into the code
base) of the submitter. In other words, we define an External PR as a PR
submitted by a contributor who is not a maintainer, such as the example shown
in Figure 1 where a contributor submits a PR that solves an already existing
issue. In contrast, an Internal PR is a PR submitted by a maintainer. Finally,
a Bot PR is a PR submitted by an automated bot such as
dependabot.111https://github.com/dependabot We collect 1,076,123 PRs from
47,959 NPM packages to first conduct a preliminary study. The results
indicated that External PRs are indeed prevalent with a large number being
received by a NPM package (75.02% on average). Furthermore, NPM contributors
have a high (88.87% on average) rate of submitting External PRs. We find that
on average, External PRs’ (55.65% per package) acceptance is higher than
Internal PRs’ (51.04% per package). Encouraged by these results, and to
further explore the role of External PRs, we carried out an empirical study to
answer these two research questions:

  * •

(RQ1) To what extent do External PRs ease maintainer workload by fixing
existing issues? Our motivation for this research question is to confirm the
extent to which External PRs fix an issue that is already part of the
maintainers’ workload.

  * •

(RQ2) How different are the code changes when comparing External PRs to
Internal PRs? For a qualitative understanding of External PRs, for this RQ, we
extract and compare External PRs with Internal PRs.

The results of RQ1 show that 26.75% of External PRs submitted to a package are
linked to an already existing issue. Furthermore, we find evidence that there
are External PRs that require the attention of the core team (e.g.,
referencing breaking changes, urgent, and on-hold labels). For RQ2, similar to
other PR types (internal and bots), we find that External PRs are most likely
to contain changes to add new features (170 out of 384 PRs) to a package.
However, different to Internal and Bot PRs, External PRs tend to focus on
documentation (44 out of 384) as opposed to features (380 out of 384) for
bots, and refactoring (34 out of 384) for Internal PRs.

The rest of the paper is organized as follows. Section 2 provides the data
preparation processes for our datasets and the preliminary study. Section 3
provides approach and findings for each RQ. Section 4 discusses lesson learnt
from our work. Section 5 exposes potential threats to validity. Section 6
reviews previous studies related to our study. Section 7 concludes the paper.
We provide a replication dataset that includes all our scripts, tools and
datasets for researchers to use in the future at: https://github.com/NAIST-
SE/External-PullRequest.

##  2 Data Preparation and Preliminary Study

In this section, we present our data preparation and preliminary study.

###  2.1 Data Collection

As shown in Figure 2, we separate our data collection process into three
stages.

#### Stage One - Data Collection.

For this study, we have two data sources to construct our dataset. The first
data source is the NPM registry which contains a listing of the available NPM
packages. We select the NPM packages due to their popularity (cf. Section 1)
and having been the subject of many studies (Chinthanet et al, 2021; Cogo et
al, 2019; Decan et al, 2018; Abdalkareem et al, 2017; Zerouali et al, 2018;
Dey et al, 2019; Dey and Mockus, 2020). Following related work, we obtained
this listing following the method of Chinthanet et al (2021), ending up with
107,242 original NPM packages. The second data source is for obtaining the PR
information. For this, we use the GitHub
API222https://docs.github.com/en/rest/reference/pulls#get-a-pull-request. We
obtained 1,076,123 PRs from 47,959 packages. Note that we filter out packages
that do not have any PR (i.e., 8,991 packages with no dependencies and 67
packages had no PRs submitted), to ensure that our dataset contains active
packages with PRs. The collected PRs contain the PR information, e.g., PR
title, description, status, label, and code patches.

To ensure a quality dataset, we further filter out libraries that have no
other library depending on them. This filter ensures that each package meets
the minimum requirements for our analysis. In the end, we obtained 945,921 PRs
from 38,925 packages. Data collection was performed in January 2021. The
summary statistics of the collection process are shown in Table 1.

#### Stage Two - Data classifying for quantitative analysis (Classified
Dataset).

In this stage, we classify whether a PR is an External PR, an Internal PR, or
a Bot PR. As shown in Table 1, by identifying the 116,265 contributors to the
945,291 PRs, we find that 290,552 (i.e., 30.74%) of the filtered PRs are
External PRs, while 340,465 or 36.02% of the filtered PRs are Internal PRs. To
identify Bot PRs, in addition to parameters from the GitHub REST API, we
follow prior work proposed by Dey et al (2020) and Golzadeh et al (2020), to
collect a list of bots provided by GitHub. After performing this
classification as shown in Table 1, we identified 314,274 Bot PRs from the
filtered dataset. A summary of the detected bots is shown in Table 2.

Figure 2: Overview of the data preparation. It consists of three stages: (i)
Data collection, (ii) Data classifying for quantitative analysis for RQ1, and
(iii) Data sampling and qualitative analysis for RQ2 Table 1: Statistics of
the collected dataset and filtered dataset (after stage 1: Data collection),
and classified dataset (after stage 2: Quantitative classifying)

Collected Dataset
---
Repository snapshot | 27 Oct 2013 — 22 Jan 2021
# original NPM packages | 107,242
# collected NPM Packages | 47,959
# NPM Pull Requests | 1,076,123
After Filtering Process
# NPM Packages | 38,925
# NPM Pull Requests | 945,291
# contributors | 116,265
Classified Dataset (RQ1)
# Bot PR | 314,274 (33.24%)
# External PR | 290,552 (30.74%)
# Internal PR | 340,465 (36.02%)
Sample Dataset (RQ2)
# Pull Requests | 1,152

Table 2: Summary of Bots detected

bot name | # PR
---|---
greenkeeperio-bot | 114,945 (36.57%)
dependabot-preview[bot] | 67,933 (21.62%)
greenkeeper[bot] | 62,837 (19.99%)
renovate[bot] | 36,796 (11.71%)
dependabot[bot] | 25,339 (8.06%)
others | 6,424 (2.05%)
Total | 314,274

#### Stage 3. Data sampling for qualitative analysis (Sample Dataset).

From the classified dataset obtained in Stage 2, we draw a statistically
representative random sample of 384 External PRs, 384 Internal PRs, and 384
Bot PRs. These sample sizes allow us to generalize the conclusions about the
ratio of PRs with a specific characteristic to all studied PRs with a
confidence level of 95% and a confidence interval of 5%, as suggested by a
prior work Hata et al (2019) and using the
calculator.333https://www.surveysystem.com/sscalc.htm

###  2.2 Preliminary study

To understand the impact of External PR, we first want to explore the extent
to which NPM packages receive PRs from external contributors, and how many
External PRs are submitted by contributors. We then study the extent to which
library maintainers accept External PRs. Hence, we ask the following three
preliminary questions.

#### (A1) From a Package Perspective

Approach: We first identify the proportion of External PRs in relation to all
submitted PRs. We define the proportion of the External PRs of a package and
submitted by a contributor as follows. For an NPM package pkg and a
contributor contrib. we calculate the proportion as:

  * •

Ext-PRrate (pkg): $\frac{\\#ExternalPR}{\\#all-PR}$ submitted to a pkg. In
this case, a higher proportion means that there are more External PRs received
by that package.

where all-PR is the sum of PRs (External PRs, Internal PRs, Bot PRs) per
package. For statistical validation, we apply Spearman’s rank correlation
coefficient or Spearman’s $\rho$, a non-parametric measure of rank
correlation, to analyze the correlations between the number of External PRs.
Spearman’s $\rho$ value ranges from -1 to 1. We analyze as follows: (1)
|$\rho$| $<$ 0.20 is Negligible or no relationship, (2) 0.20 $\leq$ |$\rho$|
$<$ 0.30 is Weak positive/negative relationship, (3) 0.30 $\leq$ |$\rho$| $<$
0.40 is Moderate positive/negative relationship, (4) 0.40 $\leq$ |$\rho$| $<$
0.70 is Strong positive/negative relationship, (5) 0.70 $\leq$ |$\rho$| $<$ 1
is Very strong positive/negative relationship, or (6) |$\rho$| = 1 is Perfect
relationship.

(a) The proportion of External PRs a package received (Ext-PRrate (pkg)),
i.e., mean of 75.02%

(b) The proportion of External PRs a contributor submitted (Ext-PRrate
(contrib.)), i.e., mean of 88.87%

Figure 3: Prevalent of Packages Receiving an External PR showing in (a) and
Contributor submission an External PR showing in (b).

Results: From the perspective of a package, we discuss two findings. The first
finding is that, from Figure 3a, we show visually that External PRs are indeed
prevalent, as shown by the high Ext-PRrate (pkg). The findings show that on
average, 75.02% of PRs received by NPM packages are External PRs.

#### (A2) From a Contributor Perspective

Approach: Similar to A1 we identify the proportion of External PRs in relation
to all submitted PRs. However, we now define the proportion of External PRs
submitted by a contributor as follows. For an NPM package pkg and a
contributor contrib. we calculate the proportion as:

  * •

Ext-PRrate (contrib.): $\frac{\\#ExternalPR}{\\#all-PR}$ submitted by a
contrib. In this case, a higher proportion means that there are more External
PRs submitted by a contributor. Our statistical validation is the same as P1.

Results: Similar to the package perspective, we discuss two findings for the
analysis of contributor submissions of External PRs. The first finding is
that, from Figure 3b, we confirm visually that a high percentage of
contributions submitted by a contributor are External PRs. This is evident by
an average of 88.87% being External PRs per contributor.

Figure 4: Distributions of PR state by the type of PR. Note that External
(Pink), Internal (Green), and Bot (Blue) PRs are shown separately.

####  (A3) Acceptance Ratios

Approach: We identify whether or not there is a difference in how External PRs
are accepted. To do so, we classify the different state changes of a PR.
Similar to prior work Wang et al (2021), there are several states of a PR,
that is Accepted PR – where the PR has been closed and merged, and Abandoned
PR – where the PR has been closed but has not been merged. After identifying
the state of each PR, we performed two analyses. It is important to note that
a PR can be either open or closed, merged or not merged. Hence, since we are
only concerned with acceptance (closed and merged) and abandonment (closed and
not merged), we ignore other state (open and not merged). Similarly to RQ1, we
calculate the distribution of acceptance at the package level of analysis.
Hence, we use proportions as follows:

  * •

state-External PR(x): $\frac{\\#state-ExternalPR}{ExternalPR}$ that is
submitted to a pkg. A higher proportion indicates that External PRs have a
higher acceptance or abandonment rate for that package.

  * •

state-Internal PR(x): $\frac{\\#state-InternalPR}{InternalPR}$ that is
submitted to a pkg. A higher proportion indicates that Internal PRs have a
higher acceptance or abandonment rate for that package.

  * •

state-Bot PR(x): $\frac{\\#state-BotPR}{BotPR}$ that is submitted to a pkg. A
higher proportion indicates that Bot PRs have a higher acceptance or
abandonment rate for that package.

where x refers to either an accept or abandon state. Using these metrics, the
first analysis is the distribution of the External PRs that were eventually
accepted. We test the following null hypothesis:

  * •

$H2.1_{null}$ There is no difference between the acceptance and abandonment of
External PRs

Similarly to P1 and P2, we use the Mann-Whitney test to test statistical
significance. We also measure the effect size using Cliff’s $\delta$, a non-
parametric effect size measure (Cliff, 1993). The effect size is analyzed as
follows: (1) |$\delta$| $<$ 0.147 as Negligible, (2) 0.147 $\leq$ |$\delta$|
$<$ 0.33 as Small, (3) 0.33 $\leq$ |$\delta$| $<$ 0.474 as Medium, and (4)
0.474 $\leq$ |$\delta$| as Large. We use the
cliffsDelta444https://github.com/neilernst/cliffsDelta package to analyze
Cliff’s $\delta$. For the second analysis, we compare the distributions
between the three different PR types (External PR, Internal PR, Bot PR). For
statistical validation, we test the following hypothesis:

  * •

$H2.2_{null}$ There is no difference in PR acceptance between external,
internal or bot PRs

First, we apply the Kruskal-Wallis H-test (Kruskal and Wallis, 1952), which is
a non-parametric statistical test to use when comparing more than two groups.
If there is significance, we then will conduct the Dunn test (Dinno, 2015) to
determine exactly which type of PRs are different to External PRs.
Additionally, we measure effect size using Cliff’s $\delta$.

Table 3: Summary statistics of each PR states of External PR, Internal PR, Bot PR i.e., Mean, Median, and SD. PR | % PR per pkg
---|---
Mean | Median | SD
state-External PR(accept) | 55.65% | 50.00% | 33.88%
state-Internal PR(accept) | 51.04% | 50.00% | 33.54%
state-Bot PR(accept) | 30.94% | 22.73% | 26.78%
state-External PR(abandon) | 29.36% | 20.00% | 27.19%
state-Internal PR(abandon) | 8.29% | 5.31% | 9.53%
state-Bot PR(abandon) | 31.18% | 21.05% | 28.10%
Table 4: Statistical test of External PR acceptance ratios

External PR | Cliff’s $\delta$
---|---
state-External PR(accept) > state-External PR(abandon)* | medium
Mann-Whitney U with *:p-value $<$ 0.001

Table 5: Contingency table showing Dunn’s test and Cliff’s $\delta$ between
Internal PR, External PR, and Bot PR

| Cliff’s $\delta$ (accept state | abandon state)
---|---
| External PR | Internal PR | Bot PR
External PR | - | negligible* | large* | medium* | negligible
Internal PR |  | - | medium* | large*
Bot PR |  |  | -
Dunn test *:p-value $<$ 0.001

Results: Figure 4 shows the acceptance ratios of all PR types per NPM package.
Note that the pink violin plot highlights the external PRs, split into
accepted and abandoned rates. Visually, we can see that both shapes are
different. In terms of the accepted External PRs, we can see that the violin
plot has a heavy top shape, indicating that there are many packages in the 90
to 100 percent ratio. On the other hand, looking at the abandoned ratio, we
see that the violin plot is much thinner. Complementary to the visual results,
Table 3 shows that statistically, External PRs have a 55.65% likelihood on
average to be accepted. On the other hand, the chances for an External PR to
get abandoned is also lower, with a 29.36% chance on average. For statistical
validation, Table 4 confirms that there is indeed a statistical difference and
confirms that $H2.1_{null}$ There is no difference between the acceptance and
abandonment of External PRs is accepted with a medium effect size.

Returning to Figure 4, we can also compare External PRs against the other PR
types (i.e., Internal PR and Bot PR). Interestingly, we can identify two
findings. The first is that the shape of the violin plots of External PRs and
Internal PRs are almost identical, with Bot PRs being different in terms of
acceptance. Complementing this result, Table 3 supports this, with both
internal and external PRs having the same 50% chance of being accepted.
Different from this, the evidence suggests that Bot PRs are least likely to be
accepted (i.e., 30.94% on average). However, when it comes to abandonment,
External PRs and Internal PRs differ. As shown in both the violin plot and the
statistics in Table 3, we see that developers are less likely to reject an
Internal PR (8.29%) as opposed to an External PR (29.36). Similarly to
External PRs, Bot PRs have the highest chance of being abandoned (31.18%). For
statistical validation, Table 5 confirms that there is indeed a statistical
difference and confirms that $H2.2_{null}$ There is no difference in PR
acceptance between external, internal or bot PRs is accepted.

Preliminaries: Findings indicate that External PRs are indeed prevalent with a
large number being received by NPM packages (75.02% on average). Furthermore,
NPM contributors have a high (88.87% on average) rate of submitting External
PRs. We find that on average, External PRs’ (55.65% per package) acceptance is
higher than Internal PRs’ (51.04% per package).

##  3 Empirical Study

Motivated by the results of the preliminary study, we are now able to
understand the need for External PR.

###  3.1 Approach

To answer RQ1, we conduct two analyses. For the first analysis, we assume that
developers’ attention will be caught by PRs that resolve existing issues
raised in a library. Hence, we identify PRs that were created in response to
an already existing issue that was raised in the library. To do so, we link
PRs to issues, following the process explained in the GitHub
documentation555https://docs.github.com/en/issues/tracking-your-work-with-
issues/linking-a-pull-request-to-an-issue. Concretely, we search for keywords
(i.e., close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved)
in the description of the PR (cf. Section 2 for example). For our results, we
will present statistics on the rate of PRs that can be linked to issues using
proportions as follows:

  * •

Linked External PR: $\frac{\\#Linked-ExternalPR}{\\#ExternalPR}$ submitted to
a pkg. In this case, a higher proportion means that there are more linked
External PRs for that package.

  * •

Linked Internal PR: $\frac{\\#Linked-InternalPR}{\\#InternalPR}$ submitted to
a pkg. In this case, a higher proportion means that there are more linked
Internal PRs for that package.

Note that for this analysis, we decided not to report linked issues for Bots,
due to a large amount of noise of keywords generated by the auto-generated
messages that reference fixes outside of the project.

Table 6: Manually curated keywords of PR labels that ‘require attention’. | identified keywords in labels
---|---
Require attention | attention, blocked, blocker, break, breaking,
| broken, change, confusing, critical, denied,
| density, difficult, difficulty, effort, exclamation,
| failed, failing, followup, hard, high, hold,
| important, incompatible, inconvenient, leak,
| memory, must, needed, notable, performance,
| prio, priority, required, risk, risky,
| securities, security, severity, urgent.
False positives | easy, low, medium, minor, no, non, review

For the second analysis, we assume that a PR might be labeled with words that
might grab the attention of any core member, for example, a PR to fix a
potential breaking change. For this analysis, we first needed to filter out
PRs that did not contain any labels. From the classified dataset, we ended up
with 245,693 (78.18%) Bot PRs, 26,797 (9.22%) External PRs and 50,144 (14.73%)
Internal PRs containing a label. Once we collected the PRs with labels, we
performed three steps as part of pre-prossessing the labels. In the first
step, we remove non-textual symbols such as emojis, punctuation, and other
non-English characters from the labels. For the second step, we tokenize the
label so that similar words can be grouped together. Finally, in our last
step, we apply lemmatization to derive the common base form. For the
implementation, we used the Python packages
re666https://pypi.org/project/regex/ and nltk777https://www.nltk.org/.

Table 6 shows the curated list of keywords used to identify labels that would
raise the attention of the core team. Following related work (Mäntylä et al,
2017), we identified words that would raise the attention (arousal) with words
that show the priority levels of a label. Using the base common words for each
label, we then systematically and manually checked all unique common base
labels to infer their importance. The analysis involved two authors who
manually scanned the words to classify keywords together, with one pointing
out a label then the other mutually agreeing, similar to Subramanian et al
(2022). Furthermore, to remove false positives (such as the label ‘low
severity’), we also collected a list of non-critical keywords. We will report
the percentage of these labels for all PR types (i.e., External PR, Internal
PR, and Bot PR). Furthermore, we will report the number of PRs and also the
top five labels within each type of PR.

To answer RQ2, we perform a qualitative analysis to characterize the
differences in PR types between Internal PRs, External PRs, and Bot PRs. In
particular, we manually classify the sample PRs based on the taxonomy of
Subramanian et al (2022) as defined below:

  * •

Documentation: Changes and additions made to documentation files such as
READMEs and/or comments explaining code. Note: Only PRs, where the majority
change is documentation, are classified as documentation change.

  * •

Feature: Adding new functionality/features to the project. Following
Subramanian et al (2022), we consider dependency updates as a feature change.

  * •

Bug: Fixing unexpected behaviour in code.

  * •

Refactoring: Restructuring code to make it more understandable/readable and/or
conform to coding standards.

  * •

GIT related issues: Solving merge conflicts, adding elements to .gitignore
files and other changes related to GIT.

  * •

Test cases: Adding test cases and/or adding code to facilitate testing.

  * •

Other: Anything that does not fall into the above categories.

We conducted manual coding to classify our sample PRs, a technique that is
popular in various qualitative studies of software engineering
(Wattanakriengkrai et al, 2022). First, three authors of this paper
independently coded the PR types for 45 samples (i.e., containing 15 from
External PR, Internal PR, and Bot PR in a first iteration. They consider the
PR title, description, conversation, and code changes. We then calculated the
inter-rater agreement between the categorization results of the first three
authors using Cohen’s kappa (McHugh, 2012). The kappa agreement varies from 0
to 1 where 0 is no agreement and 1 is perfect agreement. This process is
iterated until there is a kappa agreement of more than 0.8 or “almost perfect”
(Viera and Garrett, 2005).

For this analysis, we obtain a kappa agreements of 0.91 for overall (External
PR, Internal PR, and Bot PR), 0.95 for External PR, 0.90 for Internal PR, and
0.95 for Bot PR in the second round of iterations. After this high agreement
score, the remaining data in the samples of 384 Internal PRs, 384 External
PRs, and 384 Bot PRs is divided into three parts, and each part is coded
separately by three authors of the paper. We apply Pearson’s chi-squared test
($\chi^{2}$) (F.R.S., 1900) to test whether content of PR to External PR and
Internal PR are independent or not. To show the power of differences between
each External PR and Internal PR, we investigate the effect size using
Cramér’s V ($\phi^{\prime}$), which is a measure of association between two
nominal categories (Cramér, 2016). According to Cohen (1988), our grouping has
one degree of freedom (df*), hence, effect size is analyzed as follows: (1)
$\phi^{\prime}$ < 0.07 as Negligible, (2) 0.07 $\leq$ $\phi^{\prime}$ < 0.20
as Small, or (3) 0.30 $\geq$ $\phi^{\prime}$ as Large.

###  3.2 Findings

In this section, we present our findings to answer our two research questions.

Table 7: Summary statistic of the linked External PR and linked Internal PR i.e., Mean, Median, and SD (RQ1) PR | #linked PR | % linked-PR per pkg
---|---|---
Mean | Median | SD
Linked-External PR | 22,662 | 26.75% | 16.67% | 25.68%
Linked-Internal PR | 35,980 | 42.57% | 33.33% | 32.93%
Table 8: Ratio of PRs that were labeled as ‘require attention’. | # PR | # PR | # attention
---|---|---|---
| with label | attention label | label
External PR | 26,797 | 984 (3.63%) | 166 (61.25%)
Internal PR | 50,144 | 2,177 (4.34%) | 205 (75.65%)
Bot PR | 245,693 | 3,392 (1.38%) | 26 (9.59%)
Total | 322,634 | 6,553 (2.03%) | 271 (100%)

PRs linked to existing issues. In Table 7, we report the percentage of linked
External PRs. We find that internal PRs are more likely to be linked with an
average of 42.57% per package. This result may not be surprising, as we assume
that any core team would be focused on resolving any existing issues in their
projects. Interestingly, on average 26.75% of External PRs per package are
also linked. This evidence shows that a quarter of External PRs were in
response to an existing issue.

Labels that require attention. Table 8 shows the frequency count of PRs that
might raise developer attention. As expected, the number of these PRs should
not be large, as they indicate critical problems for a library. Yet, when
looking at the percentages, we deduce that although small, External PRs do
have relatively comparative ratios (3.63%) compared to Internal PRs (4.34%)
and Bot PRs (1.38%). It is important to note that since labeled PRs account
for a small portion of PRs (i.e, 245,693 (78.18%) Bot PRs, 26,797 (9.22%)
External PRs and 50,144 (14.73%) Internal PRs), at this stage we cannot make
any statistical generalizations. Instead, the evidence shows that External PRs
contain PRs that require the attention of developers and might provide fixes
to critical issues.

Table 9: Top five attention unique labels of each PR type. | Top five attention unique labels (#)
---|---
External PR | breaking (78), P3: important (50), P1: urgent (46), on hold (45),
| Breaking Change (40)
Internal PR | breaking (311), breaking change (141), P2: required (104),
| change/patch (96), performance (80)
Bot PR | security (3248), breaking (28), CH: Security (23),
| Type: Security (21), Security (11)
Table 10: Examples of PR content with title and reason for classification. PR Content | PR title | Reason for classification
---|---|---
Documentation | Update WritingTests.md | The PR modifies the file named
| (onl, 2017c) | ‘WritingTests.md’
Feature | feat: Add twitch icon (onl, 2019a) | PR’s title includes ‘feat’ keyword.
Bug | BIG-21501 - EU Cookie warning | PR’s title and description contain
| (Bugfix) (onl, 2016) | ‘bugfix’ keyword.
Refactoring | Major refactoring (onl, 2019b) | PR’s title includes ‘refactoring’
|  | keyword.
GIT related | Merging cards theme into master | The PR is for merging another
issues | (onl, 2017a) | branch to master.
Test cases | Remove TLS account creation | PR’s description explains about
| tests (onl, 2017b) | test integration.
Other | Mark the package as having | The PR modifies a config file
| no side effects (onl, 2019c) | (package.json)

In terms of the content of the PRs that require the attention of developers,
as shown in Table 9, the top five unique labels for External PR include
breaking (78), important (50), urgent (46), on-hold (45) and breaking change
(40). Differently Internal PR includes labels such as required (104),
change/patch (96), and performance (80), which Bot PR include labels related
to security. Interestingly, Bot PRs seem to specialize in fixing security
issues, while only having 9.59% of the total unique label keywords. This
result is consistent with the intuition that many of the bots detected were
dependency management tooling, which may include security vulnerability
fixing. Based on these results, we now return to answer the first research
question (RQ1) To what extent do External PRs ease maintainer workload by
fixing existing issues?

Summary for RQ1: We find that 26.75% of External PRs submitted to a package
are linked to an already existing issue. Furthermore, we find evidence that
there are External PRs that require the attention of the core team (e.g.,
referencing breaking changes, urgent, and on-hold labels).

Most External PR relate to New Features. Table 10 shows examples of each PR
change type, while Figure 5 depicts the frequency count for each type in our
sampled dataset. A key finding of our result is that most PRs were related to
creating a new feature, which is consistent with the results of Subramanian et
al (2022). In terms of types, Bot PRs (380 out of 384 PRs) were the most
frequent for bots and also for the category. The most likely reason is that
the NPM ecosystem has been popular for using bots to assist with dependency
management. Libraries in the ecosystem have been the target of popular bots
such as dependabot, greenkeeper, and other dependency management bots. This is
confirmed by related work (Rombaut et al, 2022).

$0$$50$$100$$150$$200$$250$$300$$350$$400$$450$OtherTest casesGIT related
issuesRefactoringBugFeatureDocumentation$18$$11$$15$$34$$111$$161$$33$$13$$9$$2$$24$$120$$170$$44$$0$$0$$1$$0$$2$$380$$0$InternalExternalBot
Figure 5: Frequency count of seven PR types comparing Internal PR, External
PR, and Bot PR

External PR also relate to features, bugs, and documentation, but not
refactorings or GIT related issues. When comparing between the different types
of PRs, we find that External PRs target features (170 PRs), bugs (120 PRs),
and documentation (44 PRs). Internal PRs similarly add features (161 PRs), fix
bugs (111 PRs), but in addition, have more refactoring-related PRs (34 PRs),
and GIT-related issues (15 PRs). We find that statistically, External PR and
Internal PR have different PR contents. (i.e., $\chi^{2}=14.837$, p-value
$<0.021$). Cramér’s V effect size ($\phi^{\prime}$), indicated a small level
of association. We now return to answer the final research question (RQ2) How
different are the code changes when comparing External PRs to Internal PRs?

Summary for RQ2: Similar to other PR types (internal and bots), External PRs
are most likely to contain changes to add new features (170 out of 384 PRs) to
a package. However, unlike Internal and Bot PRs, External PRs tend to focus on
documentation (44 out of 384) as opposed to features (380 out of 384) for
bots, and refactoring (34 out of 384) for Internal PRs.

##  4 Lessons Learnt

Returning to the goal of the study, we find that External PRs are almost as
important as Internal PRs submitted by members of the core team. Hence, we
make the following recommendations and highlight challenges and future work
that this work sparks.

  * •

Libraries Need External PRs. Based on the preliminary study, RQ1 and RQ2, we
find evidence that the ability of a library to attract and sustain External
PRs is crucial. We find that most packages receive External PRs, and an
ecosystem contributor is likely to submit External PRs. Thus we show that it
is important for library packages to have a constant flow of external
contributions. This might be even more important in cases where a library has
a single maintainer. Hence, we suggest that libraries consider strategies to
attract the community to their library. One interesting avenue for future work
would be exploring the motivations for why an external submitter would make a
contribution and what factors might attract them to make a contribution.

  * •

External PRs attract the attention of maintainers. RQ1 shows that External PRs
can contain changes that attract the attention of the core team. This is good
news, as this assistance might provide new ideas or lend a hand to often
overworked maintainers. With just over a quarter of External PRs being linked
to an already existing issue, this means that these PRs are directly related
to actionable and existing problems that can be immediately reviewed and
merged into the codebase. Potential research directions would be understanding
those External PRs that are not linked. Also, in this study, our evidence only
comes from mining the artifacts; the next logical step would be to conduct
interview or survey studies to confirm that maintainers see these External PRs
as assisting them in maintenance, rather than adding another layer of work for
them. Another future research direction would be to explore whether the
continued supply of different types of External PRs (RQ2) is a positive factor
for the sustainability of a library (to avoid becoming obsolete).

  * •

External PRs meet documentation needs. Results from RQ2 show that External PRs
meet needs that are different from Internal PRs, i.e., they are more likely to
help with documentation and feature requests. This is especially good news for
very active projects such as software libraries that require constant updates
to fix bugs and add new features. It is good for maintainers to know that
there is a community that helps them manage new features, deprecation, and
other critical aspects that need to be communicated back to their users. A
potential research direction is to investigate how these documentation needs
can be met by automated approaches, to further reduce the workload of
maintainers. These results might also indicate that creating an External PR
might not require much technical skill or coding experience, as the focus can
be on updating the documentation or correcting inconsistencies in its usage.

On the other hand, we find that External PRs are less likely to submit
refactoring changes for a project. This is intuitive, as performing such
operations requires more in-depth knowledge of the code.

##  5 Threats to Validity

Internal validity \- We discuss three internal threats. The first threat is
the correctness of the techniques used in the mining and extracting of our
datasets. As we use the listed NPM packages and pull request information based
on Chinthanet et al (2021), we are confident that our result can be
replicated. Furthermore, we contribute a full dataset that can be used by
future researchers. The second threat is related to tool selection (e.g.,
statistical testing). This is because different data sources and tests may
lead to different results. To mitigate this threat, we use standard tests such
as Spearman’s rank correlation coefficient, Mann-Whitney U test, and Kruskal-
Wallis H-test for statistical validation from popular and well-known Python
packages (i.e., SciPy Statistical
functions888https://docs.scipy.org/doc/scipy/reference/stats.html
(scipy.stats)) The third threat is through our qualitative analysis or manual
classification in RQ2. We mitigate these threats by reporting the agreement
among three raters using Cohen’s kappa. In this case, all three raters
discussed refining the types of PR content until they achieved a high kappa
agreement (more than 0.8).

Construct validity \- Threats to the construct validity of this work are
related to the categorizations and assumptions in the study. A threat is the
identification of External PRs, which is prone to false positives. Different
from prior work by Valiev et al (2018), we define a core contributor as a
contributor who has the ability to merge a PR. The bot classification is based
on prior approaches by Dey et al (2020) and Golzadeh et al (2020).
Furthermore, for RQ1, our identification of keywords is prone to
generalization issues. We acknowledge these threats, but are confident as we
carefully use empirical standards for systematic classification, with all data
available for replication. Finally, we do not take into account the internal
activities of maintainers. We believe that this is outside the scope of this
work, however, will be considered for future work.

External validity \- The main threat to external validity exists in the
generalizability of our results to other package ecosystems. In this study, we
focused solely on the NPM JavaScript ecosystem which has around 2 million
packages, with over 180 million downloads in July 2022. However, we have no
reason to believe that our analysis is not applicable to other ecosystems that
have similar package management systems, e.g., PyPI for Python. Although we
analyzed a large number of repositories on GitHub, we cannot generalize our
findings to industry or open-source repositories in general. Some open-source
repositories are hosted outside of GitHub, e.g., on GitLab or private servers.
To mitigate threats to reliability, we are careful to state that these
recommendations and implications may only be specific to the NPM community,
and extension is seen as immediate future work.

##  6 Related work

Our work is situated between two research fields. We now introduce and
highlight related literature.

#### Studies on Third-party Libraries

Third-party libraries have been studied in various aspects, including third-
party library reuse Heinemann et al (2011); Abdalkareem et al (2017); Xu et al
(2020), third-party API library Alrubaye et al (2020); Thung (2016), library
migration He et al (2021); Cogo et al (2019), Huang et al (2020), and security
vulnerabilities in third-party libraries Chinthanet et al (2021); Decan et al
(2018). Only a few studies have focused on understanding developer
contributions to third-party libraries, which we believe plays an important
role in library sustainability. In terms of sustainability of packages, Dey et
al (2019) observed that users contribute and demand effort primarily from
packages that they depend on directly with only a tiny fraction of
contributions and demand going to transitive dependencies, with a case study
of NPM packages. Recently, Dey and Mockus (2020) revealed the significant
effects of technical and social factors on the developer contribution quality
for NPM packages. Although these papers are complementary, to the best of our
knowledge, the characteristics of external contributions for third-party
libraries have not been studied comprehensively. Differently, in our work, we
focus on external contributions to third-party libraries, looking at their
prevalence and their characteristics.

#### Studies On OSS Sustainability

To sustain OSS projects, previous studies have extensively investigated the
motivations and barriers to developers’ joining and retention.

Motivations to make OSS contributions \- The common motivations to make
contributions to OSS projects are the joy of programming, the identification
with a community, career advancement, and learning Hars and Ou (2001).
Additionally, Roberts et al (2006) explored the interrelationships between
motivations of OSS developers, revealing that motivations are not always
complementary. When comparing motivations between individual developers and
companies, Bonaccorsi and Rossi-Lamastra (2006) observed that companies are
more motivated by economic and technological reasons. Lee et al (2017) found
that the most common motivation of one-time contributors is to fix bugs that
affect their work, while the highly mentioned motivation of casual
contributors is “scratch their own itch” Pinto et al (2016). In our work,
although we do not study the motivations for contributing to a library, our
study is related to characterizing OSS contributions.

Barriers to participation in OSS projects \- A study by Fagerholm et al (2014)
observed that mentoring increases the chance of developers’ active
participation. In the pull-based model, a survey by Gousios et al (2016)
revealed that the most commonly reported challenge is the lack of
responsiveness of project integrators. Steinmacher et al (2013) investigated
the first interactions of newcomers in an OSS project. Their results showed
that these interactions affected the onboarding of newcomers. Furthermore,
they analyzed pull requests of quasi-contributors and found that non-
acceptance demotivated or prevented them from placing another contribution
Steinmacher et al (2018). Assavakamhaenghan et al (2021) tried to understand
the correlation between the first response given to the first contribution and
future contributions. Li et al (2021) observed the significant impacts of pull
request abandonment on project sustainability. Complementary, in our work, we
also analyze pull requests but with a focus on third-party libraries rather
than more generic OSS projects.

Developers’ retention and engagement in OSS projects A study by Zhou and
Mockus (2012) found that developers’ willingness and participation environment
significantly impact the chance of becoming long-term contributors. In
addition, Schilling et al (2012) showed that developers’ retention in OSS
projects is affected by the level of development experience and conversational
knowledge. Valiev et al (2018) conducted a mixed-methods study to investigate
ecosystem-level factors that affect the sustainability of open-source Python
projects. Their results show that projects with more contributors are less
likely to become dormant. Iaffaldano et al (2019) conducted interviews with
OSS developers to explore what drives them to become temporarily or
permanently inactive in a project. The reported reasons were personal (e.g.,
life events) or project-related (e.g., role change and changes in the
project). Recent studies have also focused on understanding what contributions
attract newcomers (Subramanian et al, 2022), (Rehman et al, 2020). Meanwhile,
the usefulness of automated pull requests, i.e., bots, has attracted interest
from researchers in these research areas. Through an empirical study of OSS
projects with bots, developers revealed that these bots are useful for
maintaining projects (Wessel et al, 2018). Mirhosseini and Parnin (2017)
observed that bots can encourage developers to contribute to OSS projects,
especially to update dependencies. Alfadel et al (2021) studied Dependabot, a
bot used to automatically update vulnerable dependencies. They found that 65%
of the created security-related pull requests are accepted. In our work, we
also observed PRs submitted by bots, finding that bots such as Dependabot are
prevalent in Bot PRs.

##  7 Conclusion

This work investigates the extent to which third-party libraries receive
support from external contributors (i.e., External PRs) that are not part of
the core team. By mining PRs and analyzing them through mixed methods of
statistical analysis and qualitative analysis, we show that External PRs are
prevalent and just as likely to be accepted as Internal PRs. Our results
reinforce the call of core teams that need support. Future work includes
research into encouraging and sustaining External PRs, understanding how
External PRs could be automated, or could be the focus for newcomers to the
ecosystem. We envision that this work can inspire awareness and initiatives to
support third-party libraries.

## Acknowledgement

This work is supported by Japanese Society for the Promotion of Science (JSPS)
KAKENHI Grant Numbers 20K19774 and 20H05706.

## Statements and Declarations

Raula Gaikovina Kula and Christoph Treude are members of the EMSE Editorial
Board.

## References

  * onl (2016) (2016) Big-21501 - eu cookie warning (bugfix) by mickr · pull request #50 · bigcommerce/stencil-utils. https://github.com/bigcommerce/stencil-utils/pull/50, (Accessed on 01/20/2022)
  * onl (2017a) (2017a) Merging cards theme into master by grtjn · pull request #445 · marklogic-community/slush-marklogic-node. https://github.com/marklogic-community/slush-marklogic-node/pull/445, (Accessed on 01/20/2022)
  * onl (2017b) (2017b) Remove tls account creation tests by dmitrizagidulin · pull request #495 · solid/node-solid-server. https://github.com/solid/node-solid-server/pull/495, (Accessed on 01/20/2022)
  * onl (2017c) (2017c) Update writingtests.md by mattmilburn · pull request #2654 · reduxjs/redux. https://github.com/reduxjs/redux/pull/2654, (Accessed on 01/20/2022)
  * onl (2019a) (2019a) feat: Add ‘twitch‘ icon by ahtohbi4 · pull request #677 · feathericons/feather. https://github.com/feathericons/feather/pull/677, (Accessed on 01/20/2022)
  * onl (2019b) (2019b) Major refactoring by szmarczak · pull request #921 · sindresorhus/got. https://github.com/sindresorhus/got/pull/921, (Accessed on 01/20/2022)
  * onl (2019c) (2019c) Mark the package as having no side effects by stof · pull request #77 · d3/d3-format. https://github.com/d3/d3-format/pull/77, (Accessed on 01/20/2022)
  * Abdalkareem et al (2017) Abdalkareem R, Nourry O, Wehaibi S, Mujahid S, Shihab E (2017) Why do developers use trivial packages? an empirical case study on npm. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp 385–395
  * Alfadel et al (2021) Alfadel M, Costa DE, Shihab E, Mkhallalati M (2021) On the use of dependabot security pull requests. In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), IEEE, pp 254–265
  * Alrubaye et al (2020) Alrubaye H, Mkaouer MW, Khokhlov I, Reznik L, Ouni A, Mcgoff J (2020) Learning to recommend third-party library migration opportunities at the api level. Applied Soft Computing URL https://www.sciencedirect.com/science/article/pii/S1568494620300806
  * Assavakamhaenghan et al (2021) Assavakamhaenghan N, Wattanakriengkrai S, Shimada N, Kula RG, Ishio T, ichi Matsumoto K (2021) Does the first-response matter for future contributions? a study of first contributions. Proceedings of the 18th international conference on mining software repositories
  * Berger (2021) Berger A (2021) Log4j vulnerability explained: What is log4shell? https://www.dynatrace.com/news/blog/what-is-log4shell/, (Accessed on 07/04/2022)
  * Bonaccorsi and Rossi-Lamastra (2006) Bonaccorsi A, Rossi-Lamastra C (2006) Comparing motivations of individual programmers and firms to take part in the open source movement. from community to business. Knowledge and Policy pp 40–64
  * Chinthanet et al (2021) Chinthanet B, Kula RG, McIntosh S, Ishio T, Ihara A, Matsumoto K (2021) Lags in the release, adoption, and propagation of npm vulnerability fixes. Empirical Software Engineering 26(3):1–28
  * Cliff (1993) Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin p 494
  * Cogo et al (2019) Cogo FR, Oliva GA, Hassan AE (2019) An empirical study of dependency downgrades in the npm ecosystem. IEEE Transactions on Software Engineering pp 1–1
  * Cohen (1988) Cohen J (1988) Statistical Power Analysis for the Behavioral Sciences. Routledge
  * Cramér (2016) Cramér H (2016) Mathematical Methods of Statistics (PMS-9), Volume 9. Princeton university press
  * Decan et al (2018) Decan A, Mens T, Constantinou E (2018) On the impact of security vulnerabilities in the npm package dependency network. In: Proceedings of the 15th International Conference on Mining Software Repositories, pp 181–191
  * Dey and Mockus (2020) Dey T, Mockus A (2020) Effect of technical and social factors on pull request quality for the npm ecosystem. In: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Association for Computing Machinery, New York, NY, USA, ESEM ’20
  * Dey et al (2019) Dey T, Ma Y, Mockus A (2019) Patterns of effort contribution and demand and user classification based on participation patterns in npm ecosystem. PROMISE’19, p 36–45
  * Dey et al (2020) Dey T, Mousavi S, Ponce E, Fry T, Vasilescu B, Filippova A, Mockus A (2020) Detecting and characterizing bots that commit code. In: Proceedings of the 17th international conference on mining software repositories, pp 209–219
  * Dinno (2015) Dinno A (2015) Nonparametric pairwise multiple comparisons in independent groups using dunn’s test. The Stata Journal 15(1):292–300
  * Durumeric et al (2014) Durumeric Z, Li F, Kasten J, Amann J, Beekman J, Payer M, Weaver N, Adrian D, Paxson V, Bailey M, Halderman JA (2014) The matter of heartbleed. In: Proceedings of the 2014 Conference on Internet Measurement Conference, Association for Computing Machinery, New York, NY, USA, IMC ’14, p 475–488
  * Fagerholm et al (2014) Fagerholm F, Guinea AS, Münch J, Borenstein J (2014) The role of mentoring and project characteristics for onboarding in open source software projects. In: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Association for Computing Machinery, New York, NY, USA, ESEM ’14
  * Friedman (2020) Friedman N (2020) npm is joining github | the github blog. https://github.blog/2020-03-16-npm-is-joining-github/, (Accessed on 07/04/2022)
  * F.R.S. (1900) FRS KP (1900) X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50(302):157–175
  * Golzadeh et al (2020) Golzadeh M, Legay D, Decan A, Mens T (2020) Bot or not? detecting bots in github pull request activity based on comment similarity. In: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pp 31–35
  * Gousios et al (2016) Gousios G, Storey MA, Bacchelli A (2016) Work practices and challenges in pull-based development: The contributor’s perspective. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp 285–296
  * Hars and Ou (2001) Hars A, Ou S (2001) Working for free? motivations of participating in open source projects. In: Proceedings of the 34th Annual Hawaii International Conference on System Sciences
  * Hata et al (2019) Hata H, Treude C, Kula RG, Ishio T (2019) 9.6 million links in source code comments: Purpose, evolution, and decay. In: Proceedings of the 41st International Conference on Software Engineering, IEEE Press, ICSE ’19, p 1211–1221
  * He et al (2021) He H, He R, Gu H, Zhou M (2021) A large-scale empirical study on java library migrations: Prevalence, trends, and rationales. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2021, p 478–490
  * Heinemann et al (2011) Heinemann L, Deissenboeck F, Gleirscher M, Hummel B, Irlbeck M (2011) On the extent and nature of software reuse in open source java projects. In: Schmid K (ed) Top Productivity through Software Reuse, Springer Berlin Heidelberg, Berlin, Heidelberg, pp 207–222
  * Huang et al (2020) Huang K, Chen B, Shi B, Wang Y, Xu C, Peng X (2020) Interactive, effort-aware library version harmonization. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 518–529
  * Iaffaldano et al (2019) Iaffaldano G, Steinmacher I, Calefato F, Gerosa M, Lanubile F (2019) Why do developers take breaks from contributing to oss projects? a preliminary analysis. In: Proceedings of the 2nd International Workshop on Software Health, IEEE Press, SoHeal ’19, p 9–16
  * Islam et al (2021) Islam S, Kula RG, Treude C, Chinthanet B, Ishio T, Matsumoto K (2021) Contrasting third-party package management user experience. In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 664–668
  * Kruskal and Wallis (1952) Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. Journal of the American statistical Association 47(260):583–621
  * Kula et al (2018) Kula RG, German DM, Ouni A, Ishio T, Inoue K (2018) Do developers update their library dependencies? Empirical Software Engineering pp 384–417
  * Lee et al (2017) Lee A, Carver JC, Bosu A (2017) Understanding the impressions, motivations, and barriers of one time code contributors to floss projects: A survey. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp 187–197
  * Li et al (2021) Li Z, Yu Y, Wang T, Yin G, Li S, Wang H (2021) Are you still working on this an empirical study on pull request abandonment. IEEE Transactions on Software Engineering PP:1–1, DOI 10.1109/TSE.2021.3053403
  * Mäntylä et al (2017) Mäntylä MV, Novielli N, Lanubile F, Claes M, Kuutila M (2017) Bootstrapping a lexicon for emotional arousal in software engineering. In: Proceedings of the 14th International Conference on Mining Software Repositories, IEEE Press, MSR ’17, p 198–202
  * McHugh (2012) McHugh ML (2012) Interrater reliability: the kappa statistic. Biochemia medica 22(3):276–282
  * Mirhosseini and Parnin (2017) Mirhosseini S, Parnin C (2017) Can automated pull requests encourage software developers to upgrade out-of-date dependencies? In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, IEEE Press, ASE 2017, p 84–94
  * Nakakoji et al (2003) Nakakoji K, Yamamoto Y, NISHINAKA Y, Kishida K, Ye Y (2003) Evolution patterns of open-source software systems and communities. International Workshop on Principles of Software Evolution (IWPSE)
  * Nichols (2022) Nichols S (2022) Log4shell vulnerability continues to menace developers. https://bit.ly/3yEDDrn, (Accessed on 07/04/2022)
  * OpenSSF (2022) OpenSSF (2022) Openssf announces the alpha-omega project to improve software supply chain security for 10,000 oss projects - open source security foundation. https://openssf.org/press-release/2022/02/01/openssf-announces-the-alpha-omega-project-to-improve-software-supply-chain-security-for-10000-oss-projects/, (Accessed on 07/04/2022)
  * Pinto et al (2016) Pinto G, Steinmacher I, Gerosa MA (2016) More common than you think: An in-depth study of casual contributors. In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), vol 1, pp 112–123
  * Raymond (1999) Raymond E (1999) The cathedral and the bazaar. Knowledge, Technology & Policy 12(3):23–49
  * Rehman et al (2020) Rehman I, Wang D, Kula RG, Ishio T, Matsumoto K (2020) Newcomer candidate: Characterizing contributions of a novice developer to github. Proceedings of the 36th international conference on software maintainance and evolution
  * Roberts et al (2006) Roberts J, Hann IH, Slaughter S (2006) Understanding the motivations, participation, and performance of open source software developers: A longitudinal study of the apache projects. Management Science 52:984–999
  * Rombaut et al (2022) Rombaut B, Roseiro Côgo F, Adams B, Hassan AE (2022) There’s no such thing as a free lunch: Lessons learned from exploring the overhead introduced by the greenkeeper dependency bot in npm. ACM Transactions on Software Engineering and Methodology
  * Roth (2022) Roth E (2022) Open source developer corrupts widely-used libraries, affecting tons of projects. https://www.theverge.com/2022/1/9/22874949/developer-corrupts-open-source-libraries-projects-affected, (Accessed on 07/04/2022)
  * Samoladas et al (2010) Samoladas I, Angelis L, Stamelos I (2010) Survival analysis on the duration of open source projects. Information & Software Technology 52:902–922
  * Schilling et al (2012) Schilling A, Laumer S, Weitzel T (2012) Who will remain? an evaluation of actual person-job and person-team fit to predict developer retention in floss projects. In: 2012 45th Hawaii International Conference on System Sciences, pp 3446–3455
  * Sharma (2022) Sharma A (2022) npm libraries ‘colors’ and ‘faker’ sabotaged in protest by their maintainer—what to do now? https://blog.sonatype.com/npm-libraries-colors-and-faker-sabotaged-in-protest-by-their-maintainer-what-to-do-now, (Accessed on 07/04/2022)
  * Steinmacher et al (2013) Steinmacher I, Wiese I, Chaves AP, Gerosa MA (2013) Why do newcomers abandon open source software projects? In: 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp 25–32
  * Steinmacher et al (2018) Steinmacher I, Pinto G, Wiese IS, Gerosa MA (2018) Almost there: A study on quasi-contributors in open source software projects. In: Proceedings of the 40th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, USA, ICSE ’18, p 256–266
  * Subramanian et al (2022) Subramanian VN, Rehman I, Nagappan M, Kula RG (2022) Analyzing first contributions on github: What do newcomers do? IEEE Software pp 93–101
  * Thung (2016) Thung F (2016) Api recommendation system for software development. In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp 896–899
  * Valiev et al (2018) Valiev M, Vasilescu B, Herbsleb J (2018) Ecosystem-level determinants of sustained activity in open-source projects: A case study of the pypi ecosystem. In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018, p 644–655
  * Viera and Garrett (2005) Viera A, Garrett J (2005) Understanding interobserver agreement: The kappa statistic. Family medicine 37:360–3
  * Wang et al (2021) Wang D, Xiao T, Thongtanunam P, Kula RG, Matsumoto K (2021) Understanding shared links and their intentions to meet information needs in modern code review: A case study of the openstack and qt projects. Empirical Software Engineering 26
  * Wattanakriengkrai et al (2022) Wattanakriengkrai S, Chinthanet B, Hata H, Kula RG, Treude C, Guo J, Matsumoto K (2022) Github repositories with links to academic papers: Public access, traceability, and evolution. Journal of Systems and Software 183:111117
  * Wessel et al (2018) Wessel M, de Souza BM, Steinmacher I, Wiese IS, Polato I, Chaves AP, Gerosa MA (2018) The power of bots: Characterizing and understanding bots in oss projects. Proc ACM Hum-Comput Interact 2(CSCW)
  * Xu et al (2020) Xu B, An L, Thung F, Khomh F, Lo D (2020) Why reinventing the wheels? an empirical study on library reuse and re-implementation. Empirical Software Engineering 25
  * Yazıcı (2021) Yazıcı V (2021) Volkan yazıcı on twitter: "log4j maintainers have been working sleeplessly on mitigation measures; fixes, docs, cve, replies to inquiries, etc. yet nothing is stopping people to bash us, for work we aren’t paid for, for a feature we all dislike yet needed to keep due to backward compatibility concerns." / twitter. https://twitter.com/yazicivo/status/1469349956880408583?lang=en, (Accessed on 07/04/2022)
  * Zerouali et al (2018) Zerouali A, Constantinou E, Mens T, Robles G, Gonzalez-Barahona J (2018) An empirical analysis of technical lag in npm package dependencies
  * Zhou and Mockus (2012) Zhou M, Mockus A (2012) What make long term contributors: Willingness and opportunity in oss community. In: 2012 34th International Conference on Software Engineering (ICSE), pp 518–528