[orcid=0000-0001-9914-5434] 1]organization=School of Computer Science,
Carleton University,city=Ottawa,country=Canada

2]organization=Data-driven Analysis of Software (DAS) Lab, Concordia
University,city=Montreal,country=Canada

[orcid=0000-0003-1285-9878]

# A Machine Learning Approach to Determine the Semantic Versioning Type of npm
Packages Releases

Rabe Abdalkareem<EMAIL_ADDRESS>[  Md Atique Reza Chowdhury
<EMAIL_ADDRESS>[  Emad Shihab<EMAIL_ADDRESS>
###### Abstract

Semantic versioning policy is widely used to indicate the level of changes in
a package release. Unfortunately, there are many cases where developers do not
respect the semantic versioning policy, leading to the breakage of dependent
applications. To reduce such cases, we proposed using machine learning (ML)
techniques to effectively predict the new release type, i.e., patch, minor,
major, in order to properly determine the semantic versioning type. To perform
our prediction, we mined and used a number of features about a release, such
as the complexity of the changed code, change types, and development
activities. We then used four ML classifiers. To evaluate the performance of
the proposed ML classifiers, we conducted an empirical study on 31 JavaScript
packages containing a total of approximately 6,260 releases. We started by
extracting 41 release-level features from historical data of packages’ source
code and repositories. Then, we used four machine learning classifiers, namely
XGBoost, Random Forest, Decision Tree, and Logistic Regression. We found that
the XGBoost classifiers performed the best achieving median ROC-AUC values of
0.78, 0.69, and 0.74 for major, minor, and patch releases, respectively. We
also found that features related to the change types in a release are the best
predictors group of features in determining the semantic versioning type.
Finally, we studied the generalizability of determining the semantic
versioning type by applying a cross-package validation. Our results showed
that the general classifier achieved median ROC-AUC values of 0.76, 0.69, and
0.75 for major, minor, and patch releases.

###### keywords:

npm Package Releases Semantic Version Mining Software Repository Machine
Learning

##  1 Introduction

Semantic versioning is a commonly used versioning approach to signal a
change’s compatibility through version numbers. Prior work showed that
properly adapting semantic versioning increases developers’ trust in their
dependent on packages and decreases the chance of facing backward
compatibility breakage [58, 11]. Therefore, most language-specific package
managers encourage the use of semantic versioning (e.g., npm for JavaScript,
Cargo for Rust, Gems for Ruby, among others) [23, 24]. Likewise, some of the
biggest software producers such as Microsoft, Netflix, Facebook, and Google
significantly use semantic versioning to tag their new software releases [43,
54, 29]. In addition, a survey with two thousand developers shows that
developers heavily rely on semantic versioning to determine the version of
their projects’ release type [9].

However, misuse of semantic versioning can cause many problems. Developers may
incorrectly identify the semantic versioning type and may tag a new release as
minor or patch even though it introduces breaking changes, especially for
packages that are continuously releasing [11, 4]. One example of such a
problem is in the context of the web browser Firefox and the font selection
library fontconfig [4]. At some point, the fontconfig’s developers decided to
change its implementation so that blank file names would no longer be
permitted. They chose to mark this change as a minor release. However, this
release of fontconfig caused Firefox to fail to render text for any
application that used that minor release. In addition, this issue of release
tagging can be particularly problematic for oversized packages or projects
that receive many contributions and perform many changes in one release
development duration. Therefor, this problem can negatively affect both the
developers of the packages and software applications that directly or
indirectly depend on these packages [11, 58].

Due to the increased adoption of semantic versioning, most of the previous
work focused on empirically studying its usage and benefits (e.g,. [11, 42,
70]). However, very few studies tried to improve the efficiency of applying
the semantic versioning in practice. More importantly, most of the prior
studies took reactive approaches and tried to detect breakage changes of a
package after it was released through the use of source code analysis (e,g.,
[49, 58, 48, 71]). Thus, we argue that prior approaches have two key
limitations. First, they tackled the issue of wrongly tagged releases after
they are out and being integrated by others depending on applications. Second,
they heavily relied on source code analysis, which suffers from high false-
positive rates and is incapable of detecting runtime changes, especially for
packages that are written in dynamic type language such as JavaScript [55, 5].

Therefore, the main goal of our work is to automatically determine the type of
the new package release, i.e., patch, minor, and major. To do so, we proposed
the use of machine learning (ML) techniques to predict the semantic versioning
type. We started by analyzing the npm package manager and selected 31 packages
with 6,268 releases that their developers properly use semantic versioning to
tag their releases. We then analyzed the source code and mined the development
history of the studied packages, and extracted 41 features that are grouped
into six dimensions, namely, change types, development activities, complexity
and code, time, dependency, and text dimensions. Next, we built four different
machine learning classifiers, namely XGBoost, Random Forest, Decision Tree,
and Logistic Regression, to determine the semantic versioning type of the
releases. Finally, to evaluate the effectiveness of using the ML techniques,
we performed an empirical study to answer the following questions:

RQ1: Can we effectively determine the semantic versioning type of a new
package release? We built four different ML classifiers using 41 features
extracted from packages’ repositories and source code. We then compared their
performance to the baseline, which is the ZeroR classifier. Our results showed
that XGBoost classifiers achieved average ROC-AUC values of 0.77, 0.69, and
0.74 (median $=$ 0.78, 0.69, and 0.74) for major, minor, and patch releases,
respectively. In addition, this improvement equates to an average improvement
of 1.58$X$, 1.38$X$, and 1.49$X$ by the built classifiers when they were
compared to our baseline for the major, minor, and patch releases.

Then, we examined the most important dimension of features used by the ML
classifiers to determine the semantic versioning type of a new package release
in order to provide insights to practitioners as to what features best
indicate the new package release type. This led us to ask the question; RQ2:
Which dimension of features are most important in determining the semantic
versioning type of a new package release? We built different classifiers based
on each dimension of features and evaluated and compared their performance.
Our results showed that change types (e,g., number of JavaScript files added
in a release.) and complexity of the source code of the release are the most
important dimension of features in determining the type of new release.

Lastly, to examine the generalizability of the proposed technique, we
investigated the effectiveness of the ML techniques in determining the
semantic versioning type of a new package release using cross-packages
validation. In particular, we asked the question; RQ3: How effective are the
machine learning techniques when applied on cross-packages? We built general
classifiers and evaluated their performance using cross-package validation.
The results showed that the classifier achieves average ROC-AUC values of
0.74, 0.68, and 0.75 (median $=$ 0.76, 0.69, and 0.75) for major, minor, and
patch releases. These results also showed that cross-package classifiers’
performances correspond to an average ROC-AUC improvement of 1.5$X$, 1.4$X$,
and 1.5$X$ over our baseline.

In general, our work made the following key contributions:

  1. 1.

We formulated the problem of predicting semantic versioning for JavaScript
packages. To the best of our knowledge, this is the first work of using ML
techniques to determine semantic versioning type for JavaScript packages. We
envision that our approach can be used to predict the releases that are likely
to be breakage releases.

  2. 2.

We proposed features that can be mined from JavaScript package repositories
and source code to predict semantic versioning type of a new package release.
We used the proposed features to predict semantic versioning accurately and
studied the features that best indicate the semantic versioning type.

  3. 3.

We performed an empirical study on 31 open-source JavaScript packages, and our
experimental results showed that the use of ML techniques can achieve an
improvement over our baseline approach, which is the ZeroR classifier.

Structure of the paper: The remainder of the paper was organized as follows.
Section 2 provided a background on semantic versioning. We described our case
study design in Section 3. We presented our case study results in Section 4.
The work related to our study was discussed in Section 5 and the threats to
validity of our work is discussed in Section 6. Finally, Section 7 concluded
the paper.

##  2 Semantic Versioning

Since the primary goal of our work is to determine the semantic versioning
type of a new npm package release, it is essential first to provide background
on the concept of semantic versioning and how it is used to tag new package
releases.

Semantic Versioning is considered the de-facto versioning standard for many
software ecosystems, including node package manager (npm) and Python package
index (PyPI), to name a few. Semantic Versioning was introduced by the co-
founder of GitHub, Tom Preston-Werner, in 2011. In our study, we focused on
semantic versioning 2.0, which was released in 2013 [56]. The purpose of
semantic versioning is twofold. It first allows package developers to
communicate the extent of backward-incompatible changes in their new releases
to application dependents. Also, it allows for dependents of a package to
specify how restrictive or permissive they want to be in automatically
accepting new versions of the packages.

In general, semantic versioning proposes three dot-separated numbers
indicating the major, minor, and patch versions of a release. Those numbers
assist in identifying the type of changes in the newly released package. To
explain how semantic versioning works, we take the release m1.n1.p1 number as
an example. The first part m1 presents the major type, the number n1 stands
for the minor type, and the number p1 stands for the patch type. The semantic
versioning also shows rules for developers to determine how one of the three
types number should be incremented when a new release comes out. In
particular, any change to the new release package that is backward-
incompatible (e.g., break the API) requires an update to the major version.
Thus, a major release must yield the increment of the major version type, for
example, from m1.n1.p1 to m2.n1.p1. A minor release should be published when
some new backward-compatible change is introduced (e.g., adding or supporting
new functionality that does not create backward incompatibility). A minor
release must yield the increment of the minor type of the version number
(e.g., from m2.n1.p1 to m2.n2.p1). Finally, a patch release should be
published when the release represents backward compatible fixes (e.g., fixing
a bug). A patch release must yield the increment of the patch type of the
version number, such as from m2.n2.p1 to m2.n2.p2. In addition, there are some
optional tags for example specifying pre-releases type (e.g., 1.2.3-beta).

Although adopting the semantic version is not mandatory, prior studies showed
that mainly packages in npm comply with this specification (e.g., [23, 37]).
The mechanism to resolve a provided version relies on the precedence between
version numbers since npm needs to know if a particular version number is
greater than, less than, or equal to another version number. Similar to
decimal numbers, semantic version numbers are compared initially by the
magnitude of their major type, then by their minor and finally by patch types.
For example, version 3.2.1 is lower than versions 4.0.0 (by a major), 3.3.1
(by a minor), and 3.2.2 (by a patch), but greater than versions 2.2.1 (by a
major), 3.1.1 (by a minor), and 3.2.0 (by a patch).

While semantic versioning is a promising technique to specify the type of
changes in a new package release, and even though it is recommended by
ecosystem maintainers [27], it is not always straightforward to be used in
practice. For example, a package developer can mistakenly flag the new release
as a patch release while it is actually a major release. Therefore, this
mistake might lead to many problems, mainly breaking the applications that
depend on this package. In this paper, we formulated the determination of
semantic versioning type of a new package release as a research problem, which
aimed to facilitate npm packages developers to find the right semantic
versioning type for their new release packages. As a result, this will
increase the packages’ trust and reduce the breaking of applications that
depend on those packages.

##  3 Case Study Design

Table 1: The selection steps of the studied JavaScript packages that are published on npm. Selection Step | # Packages
---|---
Most starred packages | 100
Packages without post- and pre- releases | 96
Packages with more than 50 releases | 77
Packages without breakage releases | 36

The main goal of our study is to automatically determine the semantic
versioning type of a new release of a JavaScript package. To achieve this
goal, we proposed the use of machine learning techniques. We begin by
selecting JavaScript packages with a sufficient number of releases, and their
developers use semantic versioning to identify the type of the new releases.
Next, we used the selected npm packages as a labelled dataset. Then, we mined
the source code and development history of the selected JavaScript packages to
extract release-level features and used them as dependent variables in our
machine learning classifiers. In the following subsections, we detail our
labelled dataset, data extraction and processing steps, and the training of
our classifiers.

Table 2: Statistics of the studied JavaScript packages. The Table shows the
name, number of commits, releases, analyzed releases, percentage of major,
minor, patch releases of the studied packages.

Package |  Commits |  Release |  Analyzed |  %Major |  %Minor |  %Patch
---|---|---|---|---|---|---
renovate | 5,226 | 2293 | 1156 | 0.61 | 23.44 | 75.95
turtle.io | 1,110 | 413 | 294 | 2.38 | 8.16 | 89.46
sweetalert2 | 1,924 | 327 | 266 | 2.63 | 20.68 | 76.69
seek-style-guide | 579 | 280 | 222 | 10.81 | 39.19 | 50.00
oui | 722 | 226 | 207 | 4.35 | 5.31 | 90.34
react-isomorphic-render | 977 | 286 | 176 | 5.68 | 6.82 | 87.50
reactive-di | 625 | 133 | 107 | 6.54 | 8.41 | 85.05
module-deps | 492 | 135 | 104 | 5.77 | 30.77 | 63.46
express-processimage | 595 | 122 | 102 | 7.84 | 39.22 | 52.94
sku | 340 | 122 | 101 | 5.94 | 31.68 | 62.38
bittorrent-dht | 633 | 115 | 97 | 8.25 | 38.14 | 53.61
nightwatch-cucumber | 634 | 132 | 97 | 9.28 | 21.65 | 69.07
socketcluster-server | 282 | 111 | 94 | 12.77 | 27.66 | 59.57
eslint-config-canonical | 360 | 133 | 90 | 14.44 | 22.22 | 63.33
patchbay | 2,031 | 108 | 87 | 6.90 | 43.68 | 49.43
penseur | 210 | 95 | 81 | 8.64 | 50.62 | 40.74
mongo-sql | 511 | 87 | 78 | 7.69 | 12.82 | 79.49
pacote | 615 | 102 | 77 | 10.39 | 20.78 | 68.83
octokit/routes | 645 | 99 | 77 | 15.58 | 29.87 | 54.55
box-ui-elements | 1,329 | 88 | 72 | 9.72 | 52.78 | 37.50
rtc-quickconnect | 661 | 92 | 72 | 9.72 | 47.22 | 43.06
terrestris/react-geo | 2,846 | 73 | 69 | 11.59 | 46.38 | 42.03
rtcpeerconnection | 311 | 82 | 67 | 8.96 | 26.87 | 64.18
speakingurl | 429 | 78 | 66 | 19.70 | 28.79 | 51.52
license-checker | 377 | 70 | 65 | 35.38 | 18.46 | 46.15
octokit/fixtures | 378 | 81 | 64 | 12.50 | 51.56 | 35.94
repofs | 574 | 73 | 63 | 11.11 | 23.81 | 65.08
jsonrpc-bidirectional | 511 | 97 | 62 | 11.29 | 40.32 | 48.39
nes | 370 | 67 | 61 | 14.75 | 34.43 | 50.82
zapier-platform-cli | 1,003 | 69 | 61 | 11.48 | 27.87 | 60.66
rtc-signaller | 546 | 79 | 60 | 10.00 | 41.67 | 48.33
Mean | 898.30 | 202.20 | 138.50 | 10.09 | 29.72 | 60.20
Median | 595.00 | 102.00 | 81.00 | 9.72 | 28.79 | 59.57

###  3.1 Test Dataset

To perform our study, we needed to obtain a number of JavaScript packages that
follow semantic versioning guidelines to mark their releases type. To build
our labelled dataset, we started by looking at JavaScript packages that are
published on the Node Package Manager (npm). We chose npm package manager as
it is the official registry and repository for JavaScript packages.

To collect our dataset, we resorted to the public repository of npm that
contains a list of all the published packages on npm [52]. The npm repository
contains metadata about every published package, such as the different
releases of a package, the date of each release, and the release type. Since
there are a large numbers of packages published on npm and some of them did
not provide high-quality packages [2], we had to apply filtration steps to
select the packages that we wanted to study. We used four main criteria to
ensure that our dataset contains high-quality packages. The summary statistics
of these steps are shown in Table 1.

The first criterion in our selection process is to select mature and popular
packages. To do so, we chose the top 100 npm packages in our dataset based on
the number of stars they received on Github. We chose to use the number of
stars since prior work shows that the number of stars can provide a good proxy
for the popularity and maturity of software applications and packages [12,
22].

Second, we eliminated any packages from the dataset that contain at least one
release that is labelled as pre-releases or post-releases. We chose packages
that do not have pre-releases or post-releases since this is a good indicator
that the developers of those packages are somehow familiar with the semantic
versioning practices [23]. Also, we eliminated those packages to simplify our
classifications process since we would have only the three semantic versioning
type as labels in our dataset.

The third step to select the studied npm packages was to examine packages with
a sufficient number of releases. We filtered out from our dataset any package
that does not have at least five releases of each type of the semantic
versioning, and in total, the package must have at least 50 releases. We
excluded packages with a small number of releases since we wanted to use ML
techniques to determine the type of semantic versioning. Thus, we wanted to
have a sufficient number of labelled releases so that we could build robust ML
classifiers.

Figure 1: Our approach of identifying the period release history on GitHub.

We finally excluded packages that have any breakage releases identified by
developers. It is important to note that we performed this filtration step to
ensure that the developers of our studied packages understand semantic
versioning and use it adequately in practice. Thus, we had a high-quality
labelled dataset. To examine this criterion, for every npm package in our
dataset, we searched on Github for the applications that use these packages.
Then, we analyzed the development history of those applications. After that,
we examined them to see whether the developers of those applications that use
the package had downgraded a version of that package and indicated that they
performed the downgrade due to a breakage in the release of the package.
Mainly, we analyzed the historical data of these applications and identified
the commits where the developers rolled back a version of the selected
packages. We then manually examined those commits to determine if developers
rolled back a version of the selected packages due to a breaking release that
is not correctly specified by the right semantic versioning tag. Finally, we
removed any package from our dataset containing at least one case of such a
rollback. At the end of this step, we ended up having 36 packages in our
dataset.

###  3.2 Dataset Preparation

Once we decided which npm packages we would use in our study, we cloned them
locally and collected their metadata information from the npm registry. Then,
we built a semantic versioning parser to analyze every sequence release of
every package to label the release type, whether a release is major, minor, or
patch release based on the prior release. For example, suppose a package has a
release in an older date that holds the semantic versioning number as 3.2.6,
and the subsequent release based on the date has the semantic versioning
number as 3.3.6. In that case, we considered that release as a minor release
for that package (i.e., we labelled it as a minor release type). It is worth
mentioning that following this process, we were able to identify and eliminate
any backport releases from our dataset.

In the next step and since we wanted to extract features based on the source
code and the development history of the packages’ releases in our study, we
needed to have the source code and the development history of each package in
our dataset. Therefore, for each package in our dataset, we started by
collecting their metadata information and source code from the public
repository of npm. To do so, for each npm package in our dataset, we
downloaded the appropriate ‘tar’ file that contains the source code of every
release of that package. In addition, we collected the release date for every
release of the packages and the GitHub repository URL of the packages.

Now, we had the source code of each release. Next, we wanted to collect the
historical development data from the GitHub repository of each package. We
used the provided URL link to the GitHub repository to access the development
history. Then, we cloned the GitHub repository of each package and analyzed
it. However, we could not clone two package repositories because their GitHub
repositories do not exist or are changed to private repositories. In addition,
based on our research experience with the npm registry, we noted that more
than one npm packages could be hosted on the same GitHub repository (i.e.,
they hosted in monorepo repository). Thus, we manually examined the selected
packages and remove three packages from our dataset that their GitHub
repository contains more than one npm packages.

Once we collected the release information from npm and GitHub repositories, we
used a heuristic approach based on the release date to link each release to
its development history on the GitHub repository. Figure 1 shows the overall
approach. First, we analyzed the release date from the npm registry for each
package release in our dataset. And then, we extracted all the commits and
their metadata. By analyzing the commits, we extracted the commit date. Based
on the release date, we identified the first commit and the last commit for
each release (i.e., we identified the release timeframe). Now we had the
source code and the development history of each package release in our
dataset, we analyzed these data to extract a comprehensive set of features. We
describe our process for extracting the studied features for npm packages in
our dataset in the next section (Section 3.3).

Table 2 presents various statistics of our studied JavaScript packages from
npm. It shows first the name of the package and the number of commits. In
addition, the Table shows the total number of releases, the number of analyzed
releases of the studied packages, and the percentage of major, minor, and
patch releases of the studied packages. In total, there are 31 packages in our
dataset.

###  3.3 Features for Semantic Versioning Classification

Since our goal is to perform release-level predictions to determine the
semantic versioning type of a new package release, we resorted to using some
of the most commonly used release-level features. Some of these features were
used in prior software engineering tasks to identify post-release defects [63]
or used to determine crashing releases of mobile apps [74]. Therefore, we
believed that some of these features can be used to determine the level of
complexity of a new package release, hence, providing useful information as to
determine the type of a new release.

To perform our study of determining the semantic versioning type of a new
release, we resorted to using release-level features. In total, we extracted
41 features that are categorized into six dimensions. We distinguished between
these feature categories since; 1) it allowed us to observe the contribution
of different types of features, and 2) these categories let us organize how we
created and interpreted features related to determining the semantic
versioning type. In general, we extracted these features from analyzing the
source code and the development activities of each new package release in our
dataset. Table 3 presents the names and the definition of the extracted
features, and the rationale for examining them. In the following subsections,
we presented the detailed process of extracting the studied features in each
of the six dimensions.

Change Type Features: Change type features present the source code elements
that may impact the semantic versioning type of a new package release. To
extract change type features, we resorted to using source code analysis to
calculate these features (described in Table 3). Thus, we analyzed the changes
made after each release and extracted fine-grained source code change types.
To extract the features from code changes, we used the GumTree code
differencing algorithm [30]. GumTree takes as input the pair of revision files
and creates two Abstract Syntax Trees (ASTs) that are used to compare those
different revisions. As a result, GumTree outputs a list of fine-grained
source code changes (e.g., an update in a method invocation or rename). Then,
we wrote scripts that extract the fine-grained source code change types based
on the GumTree algorithm.

To extract change types features based on code that happened in each release,
we needed to have the complete version of the JavaScript files before and
after the release. To do so, we ran the diff command line between two
consecutive releases. Then, we extracted all the JavaScript files where the
files’ names have a .js extension (i.e., JavaScript source file). Once we had
the two revisions of each changed file in two consecutive releases, we ran the
GumTree tool on them. After that, we analyzed the results of GumTree to
extract the change-type features. Since the GumTree tool’s output is in a JSON
format, we parsed the resulting JSON files to retrieve the differences between
the before and after files versions. Based on this step’s results, we counted
the number of element changes in every two revisions of files and then summed
up them to get a change type value for each release.

Dependencies Features: Dependency features present the dependencies change
activities that occurred while developing a new package release. To calculate
the dependency-related features, we analyzed the changes that happened to the
package.json file. First, we analyzed the package.json file since it is the
configuration file used in the studied packages to manage and configure
dependencies. Then, we calculated the number of commits that touch the
package.json file and the number of commits that added, deleted, updated
packages in the package.json file. We built a tool that analyzes the
package.json file at every release and compares it with the previous releases
to identify dependencies that were changed.

Complexity and Code Features: Complexity and code features represent the
package’s source code changes in each release. To calculate the complexity and
code features (e.g., the difference average of Cyclomatic and the total line
of code added and deleted) for each examined release in our dataset, we
analyzed the release’s source code and computed the diff of the analyzed
release with the previous releases. To achieve this, we ran the Understand
tool [62] on every release for the examined packages in our dataset and
calculated the difference between the current release and the one before.

Time Feature: The time feature presents the time that a new release takes to
be developed and published. We counted the number of days a new release takes
to be published since the previous release date to calculate the time feature.

Development Features: Development features present the development activities
performed during the development of a new release of a package. To calculate
the development features, we analyzed the GitHub repository of each package in
our dataset. Then we measured the number of commits, unique developers, open
issues, closed pull requests, and open pull requests that occurred during that
release development timeframe.

Textual Features: Text features present extracted information from the commit
change logs that the developers have written during the development of a new
release. To extract the text features, we analyzed the commit message and
looked for specific keywords, “major”, “patch”, “break”, and then counted the
number of commits containing these keywords in each release. As for the
identify bug-fixing commits, we used a well-known approach that based on
examining the appearance of a pre-defined set of keywords that include “bug”,
“fix”, “defect”, “error”, “issue”, and their variants in commit messages [64,
69]. Then, we counted those commits in every studied release.

Table 3: Features used to determine the semantic versioning type of a new npm
package release.

Dim. |  Name |  Definition |  Rational
---|---|---|---
Change type |  AJF |  The number of JavaScript files added between two releases. |  The releases that modify several JavaScript files, functions or/and change the code structure in npm packages tend to be more major releases than being minor or patch releases. Furthermore, these are change types that can provide good indications of the semantic ve- rsioning type of a new npm package release. In other words, the re- leases that include adding new JavaScript functionalities are not small releases that are more likely to be major releases. For exam- ple, if there are several JavaScript files that are deleted in a new package release, then that release is not expected to be a patch or a minor release. Another example, If there are several non-JavaSc- ript files are changed (i.e., added, deleted, or modified) in a new package release, then the release is likely to be a patch or a minor release.
MJF |  The number of JavaScript files modified between two releases.
DJF |  The number of JavaScript files deleted between two releases.
ANJF |  The number of non-JavaScript files added between two releases.
DNJF |  The number of non-JavaScript files deleted between two releases.
MNJF |  The number of non-JavaScript files modified between two releases.
ADM |  The number of methods that are added between two releases. |
DEM |  The number of methods that are deleted between two releases. |
MOM |  The number of methods that are moved between two releases. |
MNC |  The number of methods whose names are changed between two releases. |
MPC |  The number of methods whose input parameters are changed between two releases. |
MPD |  The number of methods whose input parameters are deleted between two releases. |
MLA |  The number of logics in methods are added between two releases. |
MLM |  The number of logics in methods are moved between two releases. |
MLD |  The number of logics in methods are deleted between two releases. |
GVA |  The number of global variables added in JavaScript files between two releases. |
GVD |  The number of global variables deleted in JavaScript files between two releases. |
ICC |  The number of total code comments added between two releases. |
DCC |  The number of total code comments deleted between two releases. |
MCC |  The number of total code comments modified between two releases. |
Dependency |  TCPJ |  The number of changes to the package.json file. |  The releases that have more updates to the package dependencies list are more likely not to be patch releases. For example, adding more dependencies into the package dependencies list in the new release can indicate that this release is a major release. Another example, the changes that delete more dependencies in the new release can indicate a major release rather than a minor or a patch release.
PA |  The number of used packages added between two releases.
PD |  The number of used packages deleted between two releases.
PU |  The number of used packages’ versions changed between two releases.
Complexity |  ACYCD |  The difference average of Cyclomatic between two consecutive releases. |  We expect that the complexity and code features provide strong indicators of the semantic versioning type of the new release. If the complexity and the package size change a lot in the new release, these changes will likely present the type of semantic versioning release. For example, a large diff number of lines between two releases indicate that the new release introduces more code and is more likely not to be a patch or a minor release.
CLCJD |  The difference of lines of code between two consecutive releases.
CYCD |  The difference Cyclomatic between two consecutive releases.
LA |  The total line of code added between two releases. |
LD |  The total line of code deleted between two releases. |
Time |  RDTD |  The timestamp difference between two consecutive releases. |  A package release development that takes a long time tends to contains several changes, which is not likely to be patch.
Development |  TCM |  The total number of commits between two releases. |  The semantic versioning type of a new package heavily de- pends on the number of development activities in that rele- ase. For example, many commits or many numbers of clos- ed pull requests happened during the releases; this indicat- es that this release is not a patch release but tends to be a major or a minor package release.
TAU |  The total number of authors made changes between two releases.
POI |  The total number of open issue between two releases.
PCI |  The total number of closed issue between two releases.
PCPR |  The total number of closed pull request between two releases.
POPR |  The total number of open pull request between two releases.
Textual |  NBF |  The total number of bug-fixing commits between two releases. |  The change message contains the purpose of this commit. For example, commits that several messages contain the k- eyword major changes or breakage changes in a release de- velopment history provide a high indication that this relea- se a major release. On the other hand, releases that have co- mmits messages containing the word min- or tend to be minor or patch releases.
KWM |  The total number of commits that have keyword major in commit message in the release.
KWP |  The total number of commits that have keyword patch in commit message in the release.
KWB |  The total number of commits that have keyword break in commit message in the release.
AML |  The average commit message length in commits happened in the release.

###  3.4 Classification Algorithms

To perform our classification task, we chose four different machine learning
algorithms. In particular, we chose to use XGBoost (XGB), Random Forest (RF),
Decision Tree (DT), and Logistic Regression (LR) algorithms to classify
whether a new package release is a major, minor, or patch. We resorted to
using these ML algorithms since they 1) have different assumptions on the
examined dataset, 2) show different characteristics in terms of dealing with
overfitting and execution speed [18], and 3) provide an intuitive and
straightforward explanation of the classification, which enables developers to
easily understand why a decision to determine the type of package release was
made [41]. In addition, they have been commonly used in the past in other
software engineering studies and datasets (e., g. [32, 38, 6, 73, 67, 36,
35]). We then compared the performances of these different supervised
classifiers to determine the type of release. Now, we briefly described the
four examined machine learning algorithms.

XGBoost (XGB): The XGBoost classifier is an extended and innovative
application of gradient boosting algorithm proposed by Chen et al. [21].
Gradient boosting is an algorithm in which new models are created that predict
the residuals of prior models and then added together to make the final
prediction. Models are added recursively until no noticeable improvements can
be detected. This approach supports both regression and classification.
XGBoost has proven to push the limits of computing power for boosted tree
algorithms. Furthermore, prior work showed that applying the XGBoost
classifier on software engineering data produced good performance (e.g., [28,
46])

Random Forest (RF): The Random Forest classifier is a type of combination
approach, which is bagging and random subsets meta classifier based on a
decision tree classifier [15]. Random Forest combines multiple decision trees
for prediction. First, each decision tree is built based on the value of an
independent set of random vectors. Then, the Random Forest classifier adopts
the mode of the class labels output by individual trees. Also, prior work
showed that it performs well on software engineering problems (e.g., [59,
75]).

Decision Tree (DT): The decision trees classifier first creates a decision
tree based on the feature values of the training data where internal nodes
denote the different features [57]. The branches correspond to the value of a
particular feature, and the leaf nodes correspond to the classification of the
dependent variable. Then, the decision tree is made recursively by identifying
the feature(s) that discriminate the various instances most clearly, i.e.,
having the highest information gain [34]. Once a decision tree is built, the
classification for a new instance is performed by checking the respective
features and their values.

Logistic Regression (LR): The Logistic Regression is used to estimate the
probability of a binary response based on one or more independent variables
(i.e., features). Previous work showed that regression-based classifiers,
especially logistic regression, usually achieve high performance on software
engineering classification tasks (e.g., [32, 38]).

Baseline: Finally, to put our ML classification results in perspective, we
chose to use a simpler classifier as a baseline. In our study, we decided to
use the ZeroR (ZR) classifier, which is a primitive classifier [13]. It
basically predicts the majority class in the training data for all cases in
the test data without considering the independent features.

###  3.5 Training and Testing Classifiers

To conduct our experiments and answer our research questions, we constructed
an ML pipeline to build three different groups of classifiers. We first built
within-package classifiers where we used all the six dimensions of features to
train and test data from one package. Second, we built within-package
classifiers for each package based on each feature’s dimensions (i.e., for
each package, we built six classifiers). Finally, we built cross-package
classifiers, where for each package, a cross-package classifier is trained on
data from all packages except one and tested on the remaining one package.

Since, in our case, we have a multi-classes ML problem (e.g., as a major,
minor, patch), we formalized our ML problem to binary classification problems.
In another word, we used a one-versus-the-rest approach [50]. We used one-
versus-the-rest classifiers to ease the interpretation of our classifiers’
outcomes. In our study, we built three one-versus-the-rest classifiers for
each new release type: a major release or not, a minor release or not, and a
patch release or not. Thus, this requires creating three different ML
classifiers and training each of them with true positives and true negatives
(e.g., true minor releases and not minor releases). Furthermore, to train and
test our classifiers, we used the 5-fold cross-validation technique. In each
5-fold cross-validation, we divided the dataset into five folds. Then, four
folds are used to train the classifier, while the remaining one fold is used
to evaluate the performance of the built classifier. This process is repeated
five times so that each fold is used exactly once as the testing set. We
resorted to using 5-fold cross-validation to reduce the bias due to random
training data selection [8]. We finally reported the average performance
across these test runs. The reported results are the average of 5-fold cross-
validation, such that each sample in the total dataset was included exactly in
one test set. We implemented our examined classifiers using scikit-learn [53].
We also used the default scikit-learn configuration to set the different
parameters of the examined classifiers.

Furthermore, and as it is shown in Table 2, our dataset has on average 10.09%,
29.72%, and 60.20% for major, minor, and patch releases, which indicate that
our dataset contains imbalances data. Data imbalance occurs when one class
occurs much more than the other in a dataset, which leads to the situation
that the trained classifiers will learn from the features affecting the
majority cases than the minority cases [65]. To deal with the imbalance
problem in our experiments, we applied the synthetic minority oversampling
technique (SMOTE). SMOTE is a method for oversampling and can effectively
boost a classifier’s performance in an imbalanced case dataset such as our
dataset [20]. We applied the sampling technique to our dataset since it
balances the size of the majority class and allows us to report standard
performance and better interpret our results. It is essential to highlight
that we only applied this sampling technique to the training dataset. We did
not re-sample the testing dataset since we want to evaluate our classifier in
a real-life scenario, where the data might be imbalanced.

###  3.6 Performance Measures

To evaluate the performance of the used four machine learning classifiers and
compare their performance to our baseline, the ZeroR classifier, we calculated
the Area Under the Receiver Operating Characteristic curve (ROC-AUC). ROC-AUC
is a well-known evaluation measurement that is considered statistically
consistent. In the ROC curve, the true positive rate (TPR) is plotted as a
function of the false positive rate (FPR) across all thresholds. More
importantly, ROC-AUC is a threshold independent measure [14]. A threshold
represents the likelihood threshold for deciding an instance that is
classified as positive or negative. Usually, the threshold is set as 0.5, and
other performance measures for a classifier, such as the F1-score, heavily
depend on the threshold’s determination. However, some cases may need to
change the threshold, such as the class imbalance case. Thus, we used ROC-AUC
to avoid the threshold setting problem since ROC-AUC measures the
classification performance across all thresholds (i.e., from 0 to 1).
Likewise, ROC-AUC has the advantage of being robust towards class
distributions [44, 51].

The ROC-AUC has a value between 0 and 1, where one indicates perfect
classifications results and zero indicates completely wrong classifications.
It is important to note that prior work shows that achieving a 0.5 ROC-AUC
value indicates that the classifier performance is as good as random, while
the ROC-AUC value equal to or more than 0.7 indicates an acceptable classifier
performance using software engineering datasets [51, 44, 75].

##  4 Case Study Results

Table 4: The performance of the examined four ML classifiers for determining
the release type - major, minor, and patch. The results are reported for
XGBoost (XGB), Random Forest (RF), Decision Tree (DT), and Logistic Regression
(LR). In addition, the Table shows the results of our baseline classifier,
which is the ZeroR (ZR). The best performance values are highlighted in bold.

Packages |  Major |  Minor |  Patch
---|---|---|---
XGB |  RF |  ZR |  DT |  LR |  XGB |  RF |  ZR |  DT |  LR |  XGB |  RF |  ZR |  DT |  LR
sweetalert2 |  0.85 |  0.92 |  0.44 |  0.59 |  0.76 |  0.73 |  0.71 |  0.49 |  0.56 |  0.59 |  0.74 |  0.74 |  0.52 |  0.61 |  0.65
renovate |  0.93 |  0.89 |  0.43 |  0.49 |  0.67 |  0.87 |  0.84 |  0.50 |  0.69 |  0.66 |  0.86 |  0.81 |  0.51 |  0.71 |  0.67
speakingurl |  0.73 |  0.73 |  0.50 |  0.60 |  0.73 |  0.44 |  0.34 |  0.53 |  0.46 |  0.65 |  0.74 |  0.72 |  0.45 |  0.64 |  0.63
license-checker |  0.62 |  0.64 |  0.47 |  0.52 |  0.46 |  0.59 |  0.50 |  0.49 |  0.52 |  0.39 |  0.73 |  0.75 |  0.52 |  0.63 |  0.62
bittorrent-dht |  0.86 |  0.87 |  0.42 |  0.54 |  0.65 |  0.51 |  0.61 |  0.54 |  0.49 |  0.59 |  0.67 |  0.74 |  0.48 |  0.60 |  0.53
nes |  0.48 |  0.42 |  0.44 |  0.56 |  0.49 |  0.82 |  0.76 |  0.51 |  0.66 |  0.63 |  0.68 |  0.66 |  0.53 |  0.60 |  0.67
box-ui-elements |  0.84 |  0.89 |  0.42 |  0.64 |  0.68 |  0.68 |  0.60 |  0.49 |  0.61 |  0.61 |  0.74 |  0.76 |  0.53 |  0.63 |  0.83
sku |  0.86 |  0.73 |  0.50 |  0.60 |  0.50 |  0.79 |  0.75 |  0.51 |  0.66 |  0.56 |  0.78 |  0.70 |  0.44 |  0.67 |  0.64
mongo-sql |  0.83 |  0.68 |  0.48 |  0.70 |  0.50 |  0.64 |  0.78 |  0.43 |  0.61 |  0.72 |  0.65 |  0.68 |  0.43 |  0.62 |  0.62
pacote |  0.93 |  0.90 |  0.52 |  0.78 |  0.84 |  0.82 |  0.81 |  0.46 |  0.61 |  0.77 |  0.85 |  0.87 |  0.45 |  0.71 |  0.66
seek-style-guide |  0.72 |  0.62 |  0.48 |  0.55 |  0.42 |  0.76 |  0.76 |  0.51 |  0.63 |  0.55 |  0.75 |  0.73 |  0.49 |  0.67 |  0.61
nightwatch-cucumber |  0.76 |  0.81 |  0.48 |  0.56 |  0.46 |  0.73 |  0.80 |  0.53 |  0.61 |  0.65 |  0.76 |  0.83 |  0.50 |  0.70 |  0.65
zapier-platform-cli |  0.87 |  0.85 |  0.54 |  0.75 |  0.69 |  0.78 |  0.75 |  0.54 |  0.57 |  0.65 |  0.82 |  0.83 |  0.48 |  0.73 |  0.64
patchbay |  0.68 |  0.60 |  0.51 |  0.45 |  0.33 |  0.69 |  0.72 |  0.47 |  0.62 |  0.59 |  0.68 |  0.73 |  0.48 |  0.60 |  0.57
module-deps |  0.75 |  0.80 |  0.57 |  0.51 |  0.64 |  0.65 |  0.60 |  0.47 |  0.59 |  0.43 |  0.68 |  0.61 |  0.48 |  0.59 |  0.49
turtle.io |  0.77 |  0.88 |  0.53 |  0.56 |  0.79 |  0.80 |  0.76 |  0.49 |  0.58 |  0.64 |  0.81 |  0.85 |  0.54 |  0.72 |  0.77
rtcpeerconnection |  0.75 |  0.62 |  0.50 |  0.57 |  0.71 |  0.59 |  0.55 |  0.54 |  0.55 |  0.57 |  0.62 |  0.44 |  0.51 |  0.55 |  0.63
react-isomorphic-render |  0.82 |  0.80 |  0.55 |  0.54 |  0.59 |  0.74 |  0.75 |  0.48 |  0.55 |  0.47 |  0.80 |  0.80 |  0.51 |  0.73 |  0.60
rtc-quickconnect |  0.78 |  0.85 |  0.58 |  0.60 |  0.78 |  0.72 |  0.66 |  0.51 |  0.66 |  0.58 |  0.78 |  0.78 |  0.50 |  0.64 |  0.63
terrestris/react-geo |  0.64 |  0.75 |  0.45 |  0.50 |  0.53 |  0.67 |  0.66 |  0.45 |  0.61 |  0.60 |  0.71 |  0.66 |  0.58 |  0.63 |  0.62
eslint-config-canonical |  0.82 |  0.83 |  0.50 |  0.75 |  0.56 |  0.64 |  0.69 |  0.48 |  0.55 |  0.49 |  0.74 |  0.74 |  0.51 |  0.63 |  0.58
repofs |  0.80 |  0.91 |  0.47 |  0.57 |  0.57 |  0.72 |  0.84 |  0.49 |  0.58 |  0.42 |  0.76 |  0.83 |  0.49 |  0.65 |  0.58
penseur |  0.64 |  0.76 |  0.49 |  0.49 |  0.61 |  0.68 |  0.66 |  0.57 |  0.58 |  0.56 |  0.75 |  0.75 |  0.45 |  0.71 |  0.73
octokit/routes |  0.82 |  0.65 |  0.49 |  0.68 |  0.65 |  0.71 |  0.59 |  0.52 |  0.55 |  0.56 |  0.63 |  0.67 |  0.53 |  0.57 |  0.57
socketcluster-server |  0.78 |  0.80 |  0.42 |  0.58 |  0.73 |  0.45 |  0.45 |  0.46 |  0.49 |  0.46 |  0.70 |  0.73 |  0.47 |  0.68 |  0.63
oui |  0.88 |  0.96 |  0.54 |  0.69 |  0.65 |  0.95 |  0.84 |  0.44 |  0.70 |  0.64 |  0.91 |  0.94 |  0.55 |  0.83 |  0.75
express-processimage |  0.67 |  0.39 |  0.46 |  0.48 |  0.47 |  0.62 |  0.61 |  0.46 |  0.60 |  0.51 |  0.69 |  0.68 |  0.50 |  0.59 |  0.61
octokit/fixtures |  0.75 |  0.71 |  0.57 |  0.77 |  0.61 |  0.74 |  0.70 |  0.52 |  0.70 |  0.65 |  0.70 |  0.62 |  0.48 |  0.61 |  0.52
jsonrpc-bidirectional |  0.62 |  0.50 |  0.49 |  0.61 |  0.53 |  0.63 |  0.59 |  0.58 |  0.58 |  0.67 |  0.57 |  0.62 |  0.48 |  0.51 |  0.60
reactive-di |  0.84 |  0.80 |  0.43 |  0.66 |  0.69 |  0.56 |  0.59 |  0.52 |  0.46 |  0.44 |  0.75 |  0.73 |  0.49 |  0.63 |  0.70
rtc-signaller |  0.81 |  0.85 |  0.59 |  0.51 |  0.59 |  0.63 |  0.64 |  0.57 |  0.61 |  0.57 |  0.80 |  0.76 |  0.52 |  0.64 |  0.65
Average |  0.77 |  0.76 |  0.49 |  0.59 |  0.61 |  0.69 |  0.67 |  0.50 |  0.59 |  0.58 |  0.74 |  0.73 |  0.50 |  0.65 |  0.63
Median |  0.78 |  0.80 |  0.49 |  0.57 |  0.61 |  0.69 |  0.69 |  0.50 |  0.59 |  0.59 |  0.74 |  0.74 |  0.50 |  0.63 |  0.63
Relative ROC-AUC |  1.58$X$ |  1.55$X$ |  – |  1.21$X$ |  1.25$X$ |  1.38$X$ |  1.36$X$ |  – |  $1.18X$ |  1.15$X$ |  1.49$X$ |  1.48$X$ |  – |  1.31$X$ |  1.28$X$

In this section, we presented our case study results for our three research
questions. For each research question, we presented the motivation for the
question, the approach to answering the question, and the results.

###  4.1 RQ1: Can we effectively determine the semantic versioning type of a
new package release?

Motivation: Prior work showed that determining the type of new package release
is challenging [11]. Even though prior work proposed techniques to detect
semantic breaking API changes through static analysis for languages such as
Java [71, 58], such techniques require a clear definition of the public and
private API. Such a distinction does not explicitly exist in many dynamic
languages such as JavaScript. In this question, we wanted to effectively
determine the semantic versioning type of a new JavaScript package release.
Therefore, automatically determining the type of semantic versioning can help
guide package maintainers on deciding the versioning type on a new release. In
this RQ, we aimed to examine the use of machine learning techniques.

Method: For each package in our dataset, we used the extracted 41 release-
level features that are presented in Table 3 to train the four classifiers to
determine whether a new package release is a major, minor, or patch release.
We had reformulated this classification task into a one-versus-the-rest
classification problem since this is a multi-class classification problem
[50]. We used one-versus-the-rest classifiers since it would help us
adequately interpret our classifiers’ results. We had a one-versus-the-rest
classifier for each new release type: a major release or not, a minor release
or not, and a patch release. Thus, we built three different classifiers for
each release type where the true positives will be the examine release type
(e.g., true minor releases and not minor releases).

After that, for each package, we used 5-fold cross validation [8]. First, we
divided the dataset for each package into five folds. Then, we used four folds
(i.e., 80% of the data) to train the four ML classifiers and used the
remaining one fold (i.e., 20% of the data) to evaluate the performance of the
classifiers. We ran this process five times for each fold (i.e., 1x5-folds).
In our study, we used the four ML classifiers described in Section 3.4 that
are XGBoost, Random Forest, Decision Tree, and Logistic Regression.

Finally, to evaluate and compare the performance of the four ML classifiers in
determining the semantic versioning type of a new package release, we computed
the Area Under the Receiver Operating Characteristic curve (ROC-AUC). Then, to
come up with one value for the five runs, we calculated the average of the
evaluation measurement for 5-folds five times (i.e., 1x5-fold) for every
package in our examined dataset.

Table 5: Mann-Whitney Test (p-value) and Cliff’s Delta (d) for the results of the four classifiers vs. the baseline classifiers for the tree different semantic versioning release types. ML |  Major |  Minor |  Patch
---|---|---|---
p-value | d | p-value | d | p-value | d
XGB | 7.973e-11 | 0.96 | 1.061e-08 | 0.85 | 1.468e-11 | 0.99
RF | 9.392e-09 | 0.85 | 1.474e-08 | 0.84 | 2.16e-10 | 0.94
DT | 3.077e-06 | 0.69 | 3.382e-07 | 0.75 | 4.802e-11 | 0.97
LR | 4.105e-05 | 0.61 | 0.000254 | 0.54 | 2.81e-10 | 0.93

Since one of the main goals of using machine learning techniques is to help
determine the semantic versioning type of new release, we measured how much
better the performance of the four used classifiers is compared to the
baseline for each package. In our case, the baseline classifier is a
classifier that always reports the class of interest based on the majority,
which is the ZeroR classifier. In this case, the ZeroR classifier will achieve
100% recall and precision equal to the rate of examining release type (i.e.,
major, minor, patch). We followed the previously described process steps to
train and test the ZeroR classifier.

Then, we compared the values of ROC-AUC for the four classifiers against the
baseline by calculating the relative ROC-AUC (i. e.,
$Relative\leavevmode\nobreak\ ROC$$-$$AUC$
$=\frac{Examined\leavevmode\nobreak\ Classifier\leavevmode\nobreak\ ROC-
AUC}{Baseline\leavevmode\nobreak\ ROC-AUC}$). Relative ROC-AUC shows how much
better our classifiers perform compared to the baseline. For instance, if a
baseline achieves a ROC-AUC of 10%, while the XGBoost classifier, for example,
achieves a ROC-AUC of 20%, then the relative ROC-AUC is $\frac{20}{10}=2X$. In
other words, the XGBoost classifier performs twice as accurately as the
baseline classifier. It is important to note that the higher the relative ROC-
AUC value, the better the classifier is in determining the semantic versioning
type.

Finally, to examine whether the achieved improvement over the baseline
classifier is statistically significant, we performed a non-parametric Mann-
Whitney test [45] to compare the two distributions for each classifier results
in our dataset and determine if the difference is statistically significant,
with a $p$-value $<$ 0.05 [45]. We also used Cliff’s Delta ($d$), a non-
parametric effect size measure to interpret the effect size between the four
classifier results and our baseline. We then interpreted the effect size value
to be small for d $<$ 0.33 (for positive or negative values), medium for 0.33
$\leq d$ $<$ 0.474 and large for $d\geq$ 0.474 [33].

Result: Table 4 presents the ROC-AUC values of the four ML classifiers for
determining the release type of major, minor, and patch releases. Table 4
shows the results for XGBoost (XGB), Random Forest (RF), ZeroR (ZR), Decision
Tree (DT), and Logistic Regression (LR) for the 31 studied npm packages in our
dataset. Overall, we observe that for all three different types of the
semantic versioning (i.e., major, minor, and patch), the examined four
classifiers achieve acceptable performance in terms of ROC-AUC values [51,
44].

First, to determine the major release type, Table 4 shows that XGBoost
classifier achieves ROC-AUC values range between 0.48 and 0.93 with an average
ROC-AUC value equal to 0.77 (median$=$0.78). Also, the Random Forest achieves
a comparable performance in classifying major release types. The Table shows
that Random Forest has an average value of ROC-AUC equal to 0.76. Second, as
for the minor releases, we observed that again the XGBoost and Random Forest
classifiers perform better than the Decision Tree and Logistic Regression
classifiers. Table 4 shows that XGBoost and Random Forest have average ROC-AUC
values equal 0.69 and 0.67. Lastly, the highest ROC-AUC values for determining
the patch release types obtained by the XGBoost classifier range between 0.57
and 0.91, with an average of 0.74 (median$=$0.74). In contrast, the second
highest average ROC-AUC for determining the patch release type is achieved by
Random Forest with ROC-AUC values ranging between 0.44 and 0.94 and with an
average value of 0.73 (median $=$ 0.74). In general, the achieved ROC-AUC
values indicate that the XGBoost classifier effectively determines the
different semantic versioning types compared to the other examined ML
classifiers.

Furthermore, Table 4 shows the average relative ROC-AUC values when comparing
the performance of the four classifiers to our baseline. Overall, the computed
relative ROC-AUC shows a significant improvement over the baseline. In
particular, for all the 31 packages, the XGBoost outperforms the baseline with
average relative ROC-AUC values of 1.58$X$, 1.38$X$, and 1.49$X$ for major,
minor, and patch release types, respectively.

Finally, Table 5 presents the adjusted $p$-values and effect sizes according
to the Cliff’s delta ($d$) test. We observed that the differences are
statistically significant in the three semantic versioning types and with a
large effect size ($d>$ 0.474).

Our machine learning classifiers achieved a promising performance for
determining semantic versioning type of a new package release. They also
outperformed our baseline classifier in terms of ROC-AUC values. Out of the
four examined ML classifiers, XGBoost tended to achieve the best performance
with an average ROC-AUC of 0.77, 0.69, and 0.74 for the major, minor, and
patch releases. These results translated to an improvement of 58%, 38%, and
49% compared to our baseline.

###  4.2 RQ2: Which dimension of features are most important in determining
the semantic versioning type of a new package release?

Motivation: After determining the type of package release with adequate ROC-
AUC values and achieving a good improvement compared to our baseline, we are
now interested in understanding what dimensions of features impact determining
the type of new package releases the most. In our study, we have 41 release-
level features grouped into six dimensions. Therefore, being aware of what
dimension of features impacts a new release the most can help gain a deeper
understanding of these six dimensions. Also, we aim to provide developers with
actionable recommendations (i.e., determine the type of new package release).
More importantly, in our case, developers can know what dimensions of features
they should carefully examine when specifying the new release type.

(a) Major Releases

(b) Minor Releases

(c) Patch Releases

Figure 2: The distributions of the ROC-AUC values for the different built
classifiers.

Method: To identify the dimension of release-level features that are the most
important indicators of determining the semantic versioning type of a new
package release, we built several classifiers for each dimension. In
particular, for each package release type (i.e., major, minor, patch release),
we built six classifiers (one for each dimension of features). In total, we
built eighteen classifiers. For example, we built a classifier to determine
the major release using the change type dimension of features. To build and
evaluate these classifiers, we follow the same steps described in Section 3.5.
Since we found that the XGBoost classifier achieves the best performance in
our previous question, we used it as the classifier in this analysis.

Furthermore, to compare and evaluate the performance of the built classifiers
based on the different dimensions of features, we again used the well-known
evaluation measurement, the ROC-AUC. We then used violin plots to compare the
distributions of our results. The vertical curves of violin plots summarize
and compare the distributions of different ROC-AUC results.

Result: Figure 2 shows violin plots of the ROC-AUC values for the built
XGBoost classifier for each dimension of features for the three semantic
versioning release types. Violin plots are an effective way of presenting the
distribution of data. We also superimposed box plots to highlight the key
statistics of our results.

From Figure 2, we observed that all the six dimensions of features in our
study appear to be important in determining the semantic versioning type of a
new package release. However, one dimension of features tended to be a strong
indicator of the semantic versioning type of a release, which is the change
type dimension. Notably, for the major release type, Figure 2(a) shows that
the best dimension of features to determine the major release type is the
change type dimension with an average ROC-AUC value equal to 0.72 (median $=$
0.72).

As for the minor release, the violin plots in Figure 2(b) show that the built
XGBoost classifiers using the change type dimension outperformed other built
classifiers in most of the studied npm packages. Furthermore, our results
showed that the built classifiers based on the complexity and code dimension
of features achieved comparable performance to the change type classifiers
with average ROC-AUC values equal to 0.70 and 0.68 for classifiers that were
built using the change type and complexity and code dimension of features.

For determining the patch release type, from Figure 2(c), we observed that two
built classifiers seemed to have comparable results, which are the classifiers
that were built using change type and complexity dimensions. These two built
classifiers achieved an average ROC-AUC value equal to 0.73 for each. Overall,
our built classifiers based on the six dimensions of features in determining
the patch release type tended to achieve better performance in terms of
average ROC-AUC compared to classifiers built to determine the major and minor
release.

Interestingly, there is some dimension of features that appeared to be a good
determine of release type. For example, the dependencies related features
appeared to identify patch releases with a good performance. However,
classifiers that were built using the dependency dimension of features to
determine major and minor releases did not perform as well.

Our investigation showed that the built XGBoost classifiers using the change
type dimension of features tended to perform the best when used to determine
the semantic versioning release type compared to other built classifiers.
However, using all the six dimensions of features still achieved better
performance.

###  4.3 RQ3: How effective are the machine learning techniques when applied
on cross-packages?

Motivation: Building an ML classifier to determine the semantic versioning
release type on package-level requires having a sufficient amount of labelled
data to train on. However, many packages do not have enough historical
labelled data to build a classifier (e.g., newly adopting semantic versioning
and/or new packages). Therefore, it would be impossible to train a machine
learning classifier to determine semantic versioning type of a new release on
data from such packages. In this research question, we investigated to know to
what extent and with what performance a semantic versioning type of a new
package release can be automatically determined using a cross-package machine
learning classification. In addition, answering this question allowed us to
evaluate the generalizability of the built classifiers and their applications
when applied to other packages.

Table 6: Performance of Cross-packages classification. The results are
reported for XGBoost (XGB) and ZeroR (ZR) classifiers.

Package |  Major |  Minor |  Patch
---|---|---|---
XGB |  ZR |  XGB |  ZR |  XGB |  ZR
sweetalert2 | 0.83 | 0.59 | 0.70 | 0.48 | 0.75 | 0.49
renovate | 0.58 | 0.47 | 0.79 | 0.45 | 0.83 | 0.51
speakingurl | 0.71 | 0.61 | 0.56 | 0.62 | 0.68 | 0.39
license-checker | 0.61 | 0.52 | 0.56 | 0.33 | 0.72 | 0.48
bittorrent-dht | 0.89 | 0.49 | 0.63 | 0.64 | 0.75 | 0.42
nes | 0.59 | 0.49 | 0.75 | 0.49 | 0.75 | 0.56
box-ui-elements | 0.65 | 0.57 | 0.62 | 0.46 | 0.76 | 0.40
sku | 0.70 | 0.51 | 0.80 | 0.49 | 0.80 | 0.49
mongo-sql | 0.76 | 0.40 | 0.55 | 0.54 | 0.60 | 0.59
pacote | 0.92 | 0.47 | 0.86 | 0.54 | 0.90 | 0.52
seek-style-guide | 0.64 | 0.48 | 0.75 | 0.46 | 0.77 | 0.48
nightwatch-cucumber | 0.78 | 0.53 | 0.80 | 0.58 | 0.82 | 0.53
zapier-platform-cli | 0.82 | 0.43 | 0.75 | 0.53 | 0.82 | 0.42
patchbay | 0.53 | 0.51 | 0.77 | 0.53 | 0.76 | 0.56
module-deps | 0.82 | 0.62 | 0.53 | 0.50 | 0.61 | 0.49
turtle.io | 0.88 | 0.46 | 0.82 | 0.52 | 0.88 | 0.44
rtcpeerconnection | 0.86 | 0.59 | 0.56 | 0.45 | 0.63 | 0.49
react-isomorphic-render | 0.66 | 0.62 | 0.59 | 0.57 | 0.63 | 0.44
rtc-quickconnect | 0.84 | 0.45 | 0.62 | 0.36 | 0.70 | 0.58
terrestris/react-geo | 0.76 | 0.53 | 0.65 | 0.63 | 0.74 | 0.59
eslint-config-canonical | 0.70 | 0.56 | 0.68 | 0.41 | 0.78 | 0.42
repofs | 0.86 | 0.62 | 0.78 | 0.41 | 0.84 | 0.49
penseur | 0.82 | 0.28 | 0.57 | 0.46 | 0.72 | 0.50
octokit/routes | 0.61 | 0.44 | 0.70 | 0.64 | 0.63 | 0.55
socketcluster-server | 0.70 | 0.52 | 0.61 | 0.57 | 0.75 | 0.50
oui | 0.79 | 0.63 | 0.58 | 0.52 | 0.71 | 0.50
express-processimage | 0.69 | 0.45 | 0.69 | 0.56 | 0.72 | 0.53
octokit/fixtures | 0.78 | 0.52 | 0.86 | 0.55 | 0.82 | 0.46
jsonrpc-bidirectional | 0.62 | 0.61 | 0.70 | 0.54 | 0.73 | 0.45
reactive-di | 0.80 | 0.47 | 0.60 | 0.49 | 0.74 | 0.48
rtc-signaller | 0.84 | 0.50 | 0.75 | 0.55 | 0.79 | 0.47
Average | 0.74 | 0.52 | 0.68 | 0.51 | 0.75 | 0.49
Median | 0.76 | 0.51 | 0.69 | 0.52 | 0.75 | 0.49
Relative ROC-AUC | 1.5$X$ | - | 1.4$X$ | - | 1.5$X$ | -

Method: To better understand the generalizability of the performance achieved
by the training classifier on data from one package and apply it to another
package, we conducted a cross-packages validation. In particular, we
experimented with $n$ fold cross-packages validation, where $n$ is the number
of packages in our dataset (i.e., in our dataset, we have 31 packages). We
conducted an experiment that trains a classifier on data from thirty packages
and uses the built classifier to determine the type of semantic versioning in
the remaining one package, similar to the method used in prior work [7, 31,
1]. We repeated this process 31 times, one for each package in our dataset. To
build the classifier, we trained the XGBoost machine learning classifiers
following the same approach described earlier in Section 3.5. Once again, we
employed the well-known evaluation measurement where we computed ROC-AUC
values to measure the performance of the generated classifiers. Finally, to
examine the cross-packages classifier’s performance with respect to our
baseline, which is the ZeroR classifier, we computed the relative ROC-AUC
values.

Result: Table 6 presents the results of our experiment. It shows the ROC-AUC
values for each package for the different semantic versioning types. In
general, we observed that the built cross-packages classifiers achieved good
performance. The built classifiers have average ROC-AUC values of 0.74, 0.68,
and 0.75 for the major, minor, and patch releases. With an average ROC-AUC
score equal to 0.74 (median$=$0.75), the cross-packages classifier performs
significantly high when it is used to determine the major release type. For
example, seventeen packages in our dataset have ROC-AUC values greater than
0.75, which is an acceptable performance [51, 44, 75]. We also observed
similar performance for determining minor and patch release types.

Moreover, we compared the performance of the cross-packages classifiers to the
baseline for all the three semantic versioning release types (i.e., major,
minor, and patch). Our results showed that cross-packages classifiers show an
improvement of 50%, 40%, and 50% on average over the baseline for the major,
minor, and patch semantic versioning release type.

Table 7: Mann-Whitney Test (p-value) and Cliff’s Delta (d) for the results of XGBoost vs. ZeroR classifiers for the tree different version types. Version type | p-value | d
---|---|---
Major | 4.982e-10 | 0.92 (large)
Minor | 1.42e-08 | 0.84 (large)
Patch | 1.353e-11 | 1.00 (large)

Finally, we investigated whether the achieved improvements by the built
classifiers over the baseline classifiers for the different semantic
versioning types are statistically significant. Table 7 shows the p-values and
effect size values. It shows that for all semantic versioning types, the
differences are statistically significant, having p-values $<$ 0.05. Also, the
effect size values are large. These results showed that cross-packages
outperform the performance of the cross-package baseline classifier with
statistically significant results.

Our results indicated that cross-package machine learning classifiers can
provide comparable performances to within-package classifiers for determining
the semantic versioning type. For all packages in our dataset, cross-package
classifiers achieved average ROC-AUC values of 0.74, 0.68, and 0.75 with an
overall improvement over the baseline classifiers with relative ROC-AUC equal
to 50%, 40%, and 50% for major, minor, and patch release.

##  5 Related Work

In this paper, we proposed using machine learning techniques to effectively
determine the semantic versioning type of npm packages. Thus, our work is
mainly related to two areas of prior studies; work related to the use of
semantic versioning and work related to identifying breakage changes in third-
party packages.

Semantic versioning: Due to the importance of semantic versioning, several
studies have examined it. One of the first works that looked at the use of
semantic versioning is the work by Raemaekers et al. [58]. They investigated
the use of semantic versioning in the dataset of 22K Java packages published
on Maven that span for seven years. Their results showed that breaking changes
occur in 30% of the studied releases, including minor releases and patches.
Thus, several packages used strict dependency constraints, and package
maintainers avoid upgrading their dependencies. In addition, Kula et al. [42]
found that developers tend not to update their depend on packages even though
these updates are related to the addition of new features and patches to fix
vulnerabilities. Interestingly, Raemaekers et al. [58]’s approach relies on a
tool called tclirr, which detects breaking API changes through static analysis
of Java code. While a similar tool could be developed for other languages, it
requires a clear separation between the public and private API. Such a
distinction does not explicitly exist in dynamic languages such as JavaScript,
making the accurate detection of breaking changes much more difficult.
Moreover, fundamental differences, such as dynamic versus static typing or the
language’s dynamic nature, between JavaScript and other programming language
such as Java make the studies on this language difficult.

Dietrich et al. [25] also studied large dependencies in seventeen package
manager ecosystems found that many ecosystems support flexible versioning
practices and that the adoption of semantic versioning is increasing. In the
same line, Decan and Mens [23] empirically studied semantic versioning
compliances in four ecosystems (Cargo, npm, Packagist, and Rubygems) by
analyzing the packages dependency constraints. Their findings showed that the
proportion of compliant dependency constraints increases over time in all
studied ecosystems.

In the same direction, Wittern et al. [70] studied the evolution of a subset
of JavaScript packages in npm, analyzing characteristics such as their
dependencies, update frequency, and semantic versioning number. They observed
that the versioning conventions that maintainers use for their packages are
not always compatible with semantic versioning. Also, Bogart et al. [11]
conducted a qualitative comparison of npm, CRAN, and Eclipse, to understand
the impact of community values, tools, and policies on breaking changes. They
found two main types of mitigation strategies to reduce the exposure to
changes in dependencies: limiting the number of dependencies and depending
only on “trusted packages”. In a follow up work, they interviewed more than
2,000 developers about values and practices in 18 ecosystems [10]. Among other
findings, they observed that package maintainers are frequently exposed to
breaking changes and mainly discover them at build time.

Our work is motivated by these prior aforementioned research efforts. The
difference is that our work focuses on proposing a machine learning
classifiers to identify the semantic versioning type of a new npm package
release.

Identifying breakage changes in third-party packages: Several studies
investigated API evolution and stability and proposed techniques to detect
breakage changes [47, 72, 26, 39, 37].

Mujahid et al. [49] proposed the idea of using other’s tests to identify
breaking changes of JavaScript packages. They examined the accuracy of their
proposed approach on ten cases of breaking updates. Their experimental results
showed that their approach identified six breaking updates. Similarly, Xavier
et al. [72] performed a large-scale analysis on Java packages. Their results
showed that 14.78% of the API changes are incompatible with previous versions.
They also found that packages with a higher frequency of breaking changes are
larger, more popular, and more active. Also, Businge et al. [16, 17] studied
Eclipse interface usage by Eclipse third-party plug-ins and evaluated the
effect of API changes and non-API changes. Mostafa et al. [48] detected
backward compatibility problems in Java packages by performing regression
tests on version pairs and by inspecting bug reports related to version
upgrades. The similarity between our work and these aforementioned work is the
idea of identifying the type of changes in a new package release. However, to
the best of our knowledge, our work is the first work to investigated the use
of ML technique.

##  6 Threats to Validity

There are few important limitations to our work that need to be considered
when interpreting our findings. In this section, we described the threats to
the validity of our study.

Internal validity: Threats to internal validity concerns with factors that
could have influenced our study setup. First, we used the extracted AST
difference between two source codes to extract the change type features. To do
this, we used GumTree differencing algorithm [30]. Thus, we might be limited
by the accuracy and correctness of this tool. However, previous studies used
GumTree for calculating differences between two source codes for different
studies. It is also mentioned in the documentation of GumTree that the
algorithm is prone to some errors in the context of JavaScript, so it might
miss some instances when extracting the difference of JavaScript source codes.
For parsing the result of GumTree tool, we developed a parser to extract fine-
grained source code changes. This process could result in some errors. Thus,
we manually analyzed randomly selected 300 change types to mitigate this
threat, and our manual examination shows that the implemented parser correctly
extracts all the cases.

In addition, to answer our research questions and to extract the complexity
and code dimension of features between two consecutive releases, we used the
Understand tool [68]. Therefore, we were limited by the accuracy of the
Understand tool. That said, the Understand tool is a widely used analysis tool
in both research and industry [2, 60, 19, 3]. Also, a recent study showed that
the Understand tool analyzes JavaScript code with good accuracy [61], which
mitigate such a threat.

Construct validity: Threats to construct validity considers the relationship
between theory and observation, in case the measured variables do not measure
the actual factors. The labeled package releases (i.e., patch, minor, or
major) that we examined are releases that are explicitly marked as so by the
package developers in our dataset. In some cases, developers might mislabel
the releases. To mitigate this threat, we have applied different filtration
criteria (see Section 3.1) that include selecting mature and popular packages.
Also, we filtered out any package that their users reported it to has at least
one breakage release but their developers tagged it a minor or patch release.

Also, to extract the development features, we opted for analyzing the commits
in the Git system. Similar to prior work (e.g., [40, 66]) to identify those
commits between two consecutive releases, we consider all commits occurred in
the main trunk of the versioning system based on the release date. It is worth
mentioning that these dates could be approximations, as developers could start
working on the release even before it is issued.

External validity: Threats to external validity concern the generalization of
our findings. Our dataset only consists of JavaScript packages, which are
published on the npm package manager. Hence, our findings might not hold for
packages published on other package managers and written in different
programming languages. That said, prior work (e.g., [24]) showed that npm
packages are commonly used, and npm is one of the largest and rapidly growing
package managers, which make it the ideal case to study.

In this study, we performed a combination of feature extraction both from code
changes and development history from JavaScript open-source packages, and the
method used to extract the studied features is specific to JavaScript, so our
classifiers might not be generalized for other programming languages. Also,
different programming languages might require different feature extraction
methods due to their semantic differences. However, our data collections and
analysis approaches could be easily generalized to packages written in any
language.

In addition, our dataset presented only open-source packages whose source code
is hosted on GitHub that might not reflect close source packages. Also, in our
study, we examined a dataset that contains 31 npm JavaScript packages, which
may not represent the whole population of JavaScript packages, and examining a
larger number of packages may show different results.

##  7 Conclusion

In this paper, our goal is to use ML techniques to determine semantic
versioning type of a new package release. We used 41 release-level features
extracted by analyzing the source code and the development activities of the
releases of 31 JavaScript packages published on npm. Then, we built four ML
classifiers. We found that the XGBoost can effectively determine the type of
semantic versioning with average ROC-AUC equal to 0.77, 0.69, and 0.74 for
major, minor, and patch releases. It also showed an improvement of 58%, 38%,
and 49% over our baseline, which is the ZeroR classifier. Regarding the most
important features used by the XGBoost classifiers to determine semantic
versioning release type, we found that the change type and complexity and code
dimensions of features are the most important indicators of new release type.
Additionally, we investigated the generalizability of determining semantic
versioning type when we used cross-packages validation. Our results showed
that the cross-packages validation achieves acceptable performance compared to
within-packages validation.

## References

  * Abdalkareem et al. [2020] Abdalkareem, R., Mujahid, S., Shihab, E., 2020.  A machine learning approach to improve the detection of ci skip commits.  IEEE Transactions on Software Engineering , 1–1.
  * Abdalkareem et al. [2017] Abdalkareem, R., Nourry, O., Wehaibi, S., Mujahid, S., Shihab, E., 2017.  Why do developers use trivial packages? an empirical case study on npm, in: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA. p. 385–395.  URL: https://doi.org/10.1145/3106237.3106267, doi:10.1145/3106237.3106267.
  * Ahasanuzzaman et al. [2020] Ahasanuzzaman, M., Hassan, S., Hassan, A.E., 2020.  Studying ad library integration strategies of top free-to-download apps.  IEEE Transactions on Software Engineering .
  * Alfassa [2013] Alfassa, E., 2013.  857922 - fontconfig change breaks webfonts rendering under linux.  https://bugzilla.mozilla.org/show_bug.cgi?id=857922.  (accessed on 02/25/2022).
  * Andreasen et al. [2017] Andreasen, E., Gong, L., Møller, A., Pradel, M., Selakovic, M., Sen, K., Staicu, C.A., 2017.  A survey of dynamic analysis and test generation for javascript.  ACM Comput. Surv. 50.
  * Bacchelli et al. [2012] Bacchelli, A., Dal Sasso, T., D’Ambros, M., Lanza, M., 2012\.  Content classification of development emails, in: Proceedings of the 34th International Conference on Software Engineering, IEEE Press. pp. 375–385.
  * Bacchelli et al. [2012] Bacchelli, A., Dal Sasso, T., D’Ambros, M., Lanza, M., 2012\.  Content classification of development emails, in: 2012 34th International Conference on Software Engineering (ICSE), IEEE. pp. 375–385.
  * Bengio and Grandvalet [2004] Bengio, Y., Grandvalet, Y., 2004\.  No unbiased estimator of the variance of k-fold cross-validation.  Journal of machine learning research 5, 1089–1105.
  * Bogart et al. [2017a] Bogart, C., Filippova, A., Kastner, C., Herbsleb, J., 2017a.  How ecosystem cultures differ: Results from a survey on values and practices across 18 software ecosystems.  http://breakingapis.org/survey/.  (accessed on 11/17/2020).
  * Bogart et al. [2017b] Bogart, C., Filippova, A., Kästner, C., Herbsleb, J., 2017b.  How ecosystem cultures differ: Results from a survey on values and practices across 18 software ecosystems.  [Online]. Available: http://breakingapis.org/survey/.  (Accessed on 08/10/2020).
  * Bogart et al. [2016] Bogart, C., Kästner, C., Herbsleb, J., Thung, F., 2016\.  How to break an api: Cost negotiation and community values in three software ecosystems, in: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA. p. 109–120.  URL: https://doi.org/10.1145/2950290.2950325, doi:10.1145/2950290.2950325.
  * Borges and Valente [2018] Borges, H., Valente, M.T., 2018\.  What’s in a github star? understanding repository starring practices in a social coding platform.  Journal of Systems and Software 146, 112 – 129.
  * Bouckaert et al. [2013] Bouckaert, R.R., Frank, E., Hall, M., Kirkby, R., Reutemann, P., Seewald, A., Scuse, D., 2013.  WEKA Manual for Version 3-7-8.  (accessed on 02/28/2021).
  * Bradley [1997] Bradley, A.P., 1997.  The use of the area under the roc curve in the evaluation of machine learning algorithms.  Pattern recognition 30, 1145–1159.
  * Breiman [2001] Breiman, L., 2001.  Random forests.  Machine learning 45, 5–32.
  * Businge et al. [2012] Businge, J., Serebrenik, A., van den Brand, M.G.J., 2012.  Survival of eclipse third-party plug-ins, in: Proceedings of the 28th IEEE International Conference on Software Maintenance, IEEE, New York, NY, USA. pp. 368–377.  doi:10.1109/ICSM.2012.6405295.
  * Businge et al. [2015] Businge, J., Serebrenik, A., van den Brand, M.G.J., 2015.  Eclipse api usage: The good and the bad.  Software Quality Journal 23, 107–141.  doi:10.1007/s11219-013-9221-3.
  * Caruana and Niculescu-Mizil [2006] Caruana, R., Niculescu-Mizil, A., 2006\.  An empirical comparison of supervised learning algorithms, in: Proceedings of the 23rd International Conference on Machine Learning, ACM. pp. 161–168.
  * Castelluccio et al. [2019] Castelluccio, M., An, L., Khomh, F., 2019.  An empirical study of patch uplift in rapid release development pipelines.  Empirical Software Engineering 24, 3008–3044.
  * Chawla et al. [2002] Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P., 2002\.  Smote: synthetic minority over-sampling technique.  Journal of artificial intelligence research 16, 321–357.
  * Chen and Guestrin [2016] Chen, T., Guestrin, C., 2016\.  Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, New York, NY, USA. p. 785–794.  URL: https://doi.org/10.1145/2939672.2939785, doi:10.1145/2939672.2939785.
  * Dabbish et al. [2012] Dabbish, L., Stuart, C., Tsay, J., Herbsleb, J., 2012\.  Social coding in github: Transparency and collaboration in an open software repository, in: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, ACM. pp. 1277–1286.
  * Decan and Mens [2019] Decan, A., Mens, T., 2019\.  What do package dependencies tell us about semantic versioning?  IEEE Transactions on Software Engineering , 1–15.
  * Decan et al. [2019] Decan, A., Mens, T., Grosjean, P., 2019.  An empirical comparison of dependency network evolution in seven software packaging ecosystems.  Empirical Software Engineering , 381–416.
  * Dietrich et al. [2019] Dietrich, J., Pearce, D., Stringer, J., Tahir, A., Blincoe, K., 2019.  Dependency versioning in the wild, in: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp. 349–359.  doi:10.1109/MSR.2019.00061.
  * Dig and Johnson [2006] Dig, D., Johnson, R., 2006.  How do apis evolve? a story of refactoring.  Journal of Software Maintenance 18, 83–107.  doi:10.1002/smr.328.
  * [27] npm documentation, .  About semantic versioning | npm docs.  https://docs.npmjs.com/about-semantic-versioning.  (accessed on 03/08/2022).
  * Esteves et al. [2020] Esteves, G., Figueiredo, E., Veloso, A., Viggiato, M., Ziviani, N., 2020.  Understanding machine learning software defect predictions.  Automated Software Engineering 27, 369–392.
  * FaceBook [2016] FaceBook, 2016.  Yarn: A new package manager for javascript - facebook engineering.  https://engineering.fb.com/2016/10/11/web/yarn-a-new-package-manager-for-javascript/.  (accessed on 03/13/2021).
  * Falleri et al. [2014] Falleri, J., Morandat, F., Blanc, X., Martinez, M., Monperrus, M., 2014.  Fine-grained and accurate source code differencing, in: ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014, pp. 313–324.  URL: http://doi.acm.org/10.1145/2642937.2642982, doi:10.1145/2642937.2642982.
  * Fukushima et al. [2014] Fukushima, T., Kamei, Y., McIntosh, S., Yamashita, K., Ubayashi, N., 2014.  An empirical study of just-in-time defect prediction using cross-project models, in: Proceedings of the 11th Working Conference on Mining Software Repositories, Association for Computing Machinery. p. 172–181.
  * Ghotra et al. [2015] Ghotra, B., , S., Hassan, A.E., 2015.  Revisiting the impact of classification techniques on the performance of defect prediction models, in: Proceedings of the 37th International Conference on Software Engineering, IEEE Press. pp. 789–800.
  * Grissom and Kim [2005] Grissom, R.J., Kim, J.J., 2005\.  Effect sizes for research: A broad practical approach.  Lawrence Erlbaum Associates Publishers.
  * Hall et al. [2009] Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H., 2009\.  The weka data mining software: an update.  ACM SIGKDD explorations newsletter 11, 10–18.
  * He et al. [2012] He, Z., Shu, F., Yang, Y., Li, M., Wang, Q., 2012\.  An investigation on the feasibility of cross-project defect prediction.  Automated Software Engineering. 19, 167–199.
  * Iba [1996] Iba, H., 1996.  Random tree generation for genetic programming, in: Proceedings of the 4th International Conference on Parallel Problem Solving from Nature, Springer-Verlag, London, UK, UK. pp. 144–153.  URL: http://dl.acm.org/citation.cfm?id=645823.670546.
  * Javan Jafari et al. [2021] Javan Jafari, A., Costa, D.E., Abdalkareem, R., Shihab, E., Tsantalis, N., 2021.  Dependency smells in javascript projects.  IEEE Transactions on Software Engineering , 1–1doi:10.1109/TSE.2021.3106247.
  * Kamei et al. [2013] Kamei, Y., Shihab, E., Adams, B., Hassan, A.E., Mockus, A., Sinha, A., Ubayashi, N., 2013.  A large-scale empirical study of just-in-time quality assurance.  IEEE Transactions on Software Engineering 39, 757–773.
  * Kapur et al. [2010] Kapur, P., Cossette, B., Walker, R.J., 2010.  Refactoring references for library migration.  ACM SIGPLAN Notices 45, 726–738.  doi:10.1145/1932682.1869518.
  * Khomh et al. [2015] Khomh, F., Adams, B., Dhaliwal, T., Zou, Y., 2015\.  Understanding the impact of rapid releases on software quality.  Empirical Softw. Engg. 20, 336–373.  URL: https://doi.org/10.1007/s10664-014-9308-x, doi:10.1007/s10664-014-9308-x.
  * Kotsiantis et al. [2006] Kotsiantis, S.B., Zaharakis, I.D., Pintelas, P.E., 2006.  Machine learning: A review of classification and combining techniques.  Artif. Intell. Rev. 26, 159–190.
  * Kula et al. [2017] Kula, R.G., German, D.M., Ouni, A., Ishio, T., Inoue, K., 2017.  Do developers update their library dependencies?: An empirical study on the impact of security advisories on library migration.  doi:10.1007/s10664-017-9521-5, arXiv:1709.04621.
  * Lauinger et al. [2018] Lauinger, T., Chaabane, A., Wilson, C., 2018.  Thou shalt not depend on me: A look at javascript libraries in the wild.  Queue 16, 62–82.
  * Lessmann et al. [2008] Lessmann, S., Baesens, B., Mues, C., Pietsch, S., 2008\.  Benchmarking classification models for software defect prediction: A proposed framework and novel findings.  IEEE Transactions on Software Engineering 34, 485--496.
  * Mann and Whitney [1947] Mann, H.B., Whitney, D.R., 1947\.  On a test of whether one of two random variables is stochastically larger than the other.  The annals of mathematical statistics , 50--60.
  * Mariano et al. [2019] Mariano, R.V.R., dos Santos, G.E., V. de Almeida, M., Brandão, W.C., 2019\.  Feature changes in source code for commit classification into maintenance activities, in: 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), IEEE. pp. 515--518.
  * Mostafa et al. [2017a] Mostafa, S., Rodriguez, R., Wang, X., 2017a.  A Study on Behavioral Backward Incompatibility Bugs in Java Software Libraries, in: Proceedings of the 39th International Conference on Software Engineering Companion, IEEE, New York, NY, USA. pp. 127--129.  doi:10.1109/ICSE-C.2017.101.
  * Mostafa et al. [2017b] Mostafa, S., Rodriguez, R., Wang, X., 2017b.  Experience paper: A study on behavioral backward incompatibilities of java software libraries, in: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Association for Computing Machinery, New York, NY, USA. p. 215–225.  URL: https://doi.org/10.1145/3092703.3092721, doi:10.1145/3092703.3092721.
  * Mujahid et al. [2020] Mujahid, S., Abdalkareem, R., Shihab, E., McIntosh, S., 2020\.  Using others’ tests to identify breaking updates , 1--12.
  * Murphy [2012] Murphy, K.P., 2012.  Machine learning: a probabilistic perspective.  MIT press.
  * Nam and Kim [2015] Nam, J., Kim, S., 2015.  Clami: Defect prediction on unlabeled datasets, in: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, IEEE Press. p. 452–463.
  * [52] npm, .  npm-registry | npm documentation.  https://docs.npmjs.com/using-npm/registry.html.  (Accessed on 08/13/2020).
  * Pedregosa et al. [2011] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al., 2011\.  Scikit-learn: Machine learning in python.  the Journal of machine Learning research 12, 2825--2830.
  * Potvin and Levenberg [2016] Potvin, R., Levenberg, J., 2016\.  Why google stores billions of lines of code in a single repository.  Communications of the ACM 59, 78--87.
  * Pradel et al. [2015] Pradel, M., Schuh, P., Sen, K., 2015.  Typedevil: Dynamic type inconsistency analysis for javascript, in: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, IEEE. pp. 314--324.
  * Preston-Werner [2019] Preston-Werner, T., 2019.  Semantic versioning 2.0.  URL: https://semver.org/.
  * Quinlan [1993] Quinlan, R., 1993.  C4.5: Programs for Machine Learning.  Morgan Kaufmann Publishers, San Mateo, CA.
  * Raemaekers et al. [2017] Raemaekers, S., van Deursen, A., Visser, J., 2017.  Semantic versioning and impact of breaking changes in the maven repository.  Journal of Systems and Software 129, 140--158.
  * Rahman et al. [2017] Rahman, M.M., Roy, C.K., Kula, R.G., 2017.  Predicting usefulness of code review comments using textual features and developer experience, in: Proceedings of the 14th International Conference on Mining Software Repositories, IEEE Press. pp. 215--226.
  * Rahman et al. [2019] Rahman, M.T., Rigby, P.C., Shihab, E., 2019.  The modular and feature toggle architectures of google chrome.  Empirical Software Engineering 24, 826--853.
  * Reza Chowdhury et al. [2021] Reza Chowdhury, M.A., Abdalkareem, R., Shihab, E., Adams, B., 2021\.  On the untriviality of trivial packages: An empirical study of npm javascript packages.  IEEE Transactions on Software Engineering , 1--1.
  * [62] SciTools-Documentation, .  Understand static code analysis tool.  https://www.scitools.com/.  (accessed on 03/08/2022).
  * Shihab et al. [2010] Shihab, E., Jiang, Z.M., Ibrahim, W.M., Adams, B., Hassan, A.E., 2010.  Understanding the impact of code and process metrics on post-release defects: A case study on the eclipse project, in: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.
  * Śliwerski et al. [2005] Śliwerski, J., Zimmermann, T., Zeller, A., 2005.  When do changes induce fixes?  ACM sigsoft software engineering notes 30, 1--5.
  * Song et al. [2019] Song, Q., Guo, Y., Shepperd, M., 2019.  A comprehensive investigation of the role of imbalanced learning for software defect prediction.  IEEE Transactions on Software Engineering 45, 1253--1269.  doi:10.1109/TSE.2018.2836442.
  * Souza et al. [2014] Souza, R., Chavez, C., Bittencourt, R.A., 2014.  Do rapid releases affect bug reopening? a case study of firefox, in: 2014 Brazilian Symposium on Software Engineering, pp. 31--40.  doi:10.1109/SBES.2014.10.
  * Thung et al. [2012] Thung, F., Lo, D., Jiang, L., Lucia, Rahman, F., Devanbu, P.T., 2012.  When would this bug get reported?, in: Proceedings of the 28th IEEE International Conference on Software Maintenance, IEEE. pp. 420--429.
  * [68] Understand, S., .  Scitools.com.  https://scitools.com/.  (Accessed on 08/13/2020).
  * Williams and Spacco [2008] Williams, C., Spacco, J., 2008\.  Szz revisited: verifying when changes induce fixes, in: Proceedings of the 2008 workshop on Defects in large software systems, pp. 32--36.
  * Wittern et al. [2016] Wittern, E., Suter, P., Rajagopalan, S., 2016.  A look at the dynamics of the javascript package ecosystem, in: Proceedings of the 13th International Conference on Mining Software Repositories, Association for Computing Machinery, New York, NY, USA. p. 351–361.  URL: https://doi.org/10.1145/2901739.2901743, doi:10.1145/2901739.2901743.
  * Xavier et al. [2017] Xavier, L., Brito, A., Hora, A., Valente, M.T., 2017\.  Historical and impact analysis of api breaking changes: A large-scale study, in: 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE. pp. 138--147.
  * Xavier et al. [2017] Xavier, L., Brito, A., Hora, A., Valente, M.T., 2017\.  Historical and impact analysis of api breaking changes: A large-scale study, in: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering, IEEE, New York, NY, USA. pp. 138--147.  doi:10.1109/SANER.2017.7884616.
  * Xia et al. [2016a] Xia, X., , E., Kamei, Y., Lo, D., Wang, X., 2016a.  Predicting crashing releases of mobile applications, in: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ACM. pp. 29:1--29:10.
  * Xia et al. [2016b] Xia, X., Shihab, E., Kamei, Y., Lo, D., Wang, X., 2016b.  Predicting crashing releases of mobile applications, in: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’16).
  * Yan et al. [2019] Yan, M., Xia, X., Shihab, E., Lo, D., Yin, J., Yang, X., 2019\.  Automating change-level self-admitted technical debt determination.  IEEE Transactions on Software Engineering 45, 1211--1229.