∎

11institutetext:  M. Islam 22institutetext: University of Alberta
22email<EMAIL_ADDRESS>33institutetext: A. Jha 44institutetext:
University of Alberta
44email<EMAIL_ADDRESS>55institutetext: S. Nadi 66institutetext:
University of Alberta
66email<EMAIL_ADDRESS>
#  PyMigBench and PyMigTax: A Benchmark and Taxonomy for Python Library
Migration

Mohayeminul Islam  Ajay Kumar Jha  Sarah Nadi

(Received: date / Accepted: date)

###### Abstract

Developers heavily rely on Application Programming Interfaces (APIs) from
libraries to build their projects. However, libraries might become obsolete,
or new libraries with better APIs might become available. In such cases,
developers need to replace the used libraries with alternative libraries, a
process referred to as library migration. When done manually, library
migration can be tedious, time-consuming, and error-prone. Most of the current
research on automated library migration techniques focus on Java libraries,
and even more so on version migrations of the same library. Despite the
increasing popularity of Python, limited research work has investigated
migration between Python libraries. In this paper, we investigate the nature
of Python library migrations in open-source systems. We analyze the code
changes that happen during library migration and build PyMigBench, a manually
verified migration benchmark. PyMigBench contains 436 migration-related code
changes from 74 commits in 57 client repositories, and includes migrations
between 34 unique pairs of libraries. Additionally, we manually analyze the
migration-related code changes and create a taxonomy of migrations, PyMigTax,
that categorizes migrations across various dimensions. Our contributions
provide the necessary foundations for developing automated Python library
migration tools and techniques.

###### Keywords:

library migration migration-related code changes benchmark taxonomy Python

This is a preprint version and is submitted for publication

##  1 Introduction

Libraries are integral components of any software system understandingAPIUsage
; empiricalAPIUsage . Among their many advantages trivialNPMPackage , usage of
libraries improves developers’ efficiency and thus reduces costs in building
software systems softwareReuse . It also improves the reliability and
maintainability of software systems reinventingTheWheels . Therefore,
developers heavily rely on the Application Programming Interfaces (APIs)
offered by libraries to build software. However, the libraries and APIs an
application depends on may become obsolete over time deprecatedPythonAPIs ;
apiDeprecation ; he2021multi . Libraries may also negatively impact the
applications that use them due to various types of critical vulnerabilities or
bugs present in the libraries keepMeUpdated ; kula2018developers . Moreover,
developers might also find new, better-performing, or easier-to-use libraries
kabinna2016logging ; selectingLibraries ; he2021multi . In all these
circumstances, developers often need to replace a currently used library with
an alternate one, a process commonly referred to as library migration.

The library migration process typically involves finding an alternative
library with the required functionality (library mapping), finding the
replacements for each usage of an API of the old library (API mapping), and
updating all existing code that used the old library’s API to now use the new
library’s API while ensuring no change in the software’s behavior (client code
transformation). This is a time-consuming and error-prone task that developers
often dread kula2018developers , especially in large codebases with pervasive
use of the original library. Therefore, migration techniques that automate
this entire process can save developers’ time and effort.

Given the different non-trivial activities involved, it is however difficult
to fully automate the entire process. Therefore, researchers have often
attempted to separately automate the different activities of the migration
process. For example, many research techniques focus on the library mapping
stage, where identifying alternative (a.k.a analagous) libraries is done by
mining historical changes in projects’ dependencies teyton2012mining ;
teyton2014study ; he2021multi  or by analyzing Stack Overflow posts
miningAnalogousLib . Other techniques use various metrics, based on functional
and non-functional characteristics of libraries, to select or recommend such
analogous libraries he2021multi ; librarySelectionMetrics ; libComp . Similar
approaches have also been used for API mapping, whether through mining
previous migration instances or API descriptions to discover analogous APIs
teyton2013automatic ; alrubaye2018automating ; alrubaye2019use ;
miningAnalogicalAPIs ; ni2021soar ; zhang2020deep , or by building recommender
systems around these API mappings alrubaye2020learning . For client code
transformation, researchers used various techniques such as program synthesis
ni2021soar , data flow analysis lamothe2020a3 , patch generalization
fazzini2020apimigrator ; xu2019meditor  and differential testing
fazzini2020apimigrator .

While the previous work above shows that library migration is an active
research area with lots of advancements, there are still gaps in the
literature. First, most of the mentioned previous library migration techniques
focus on the Java language he2021multi ; teyton2012mining ; teyton2014study ;
libComp ; teyton2013automatic ; alrubaye2019use ; alrubaye2020learning ;
xu2019meditor ; fazzini2020apimigrator ; lamothe2020a3 . It is not clear
whether these techniques can be applied to a different programming language
that is not statically typed, such as Python codeChangesPythonML ;
evolutionMLLib . Second, most of the previous library migration techniques
that automate client code transformation focus on version migration, that is
migration between different versions of the same library xu2019meditor ;
fazzini2020apimigrator ; lamothe2020a3 . Version migration has the advantage
that the library developers often provide documentation or release notes
containing the API changes xu2019meditor ; fazzini2020apimigrator , which
facilitates automation. Even if such information is not explicitly provided,
it can be inferred from the library’s version control history xing2007api ;
huang2021repfinder . Unfortunately, such change or mapping information is not
readily available or easily inferrable when two independent libraries are
involved. Thus, migrating client code from using one library to using a
completely different library is more challenging than version migration.
Finally, researchers have evaluated most of the previous version or library
migration techniques by migrating code for a few arbitrarily selected library
pairs. For example, Ni et al. ni2021soar  evaluated their library migration
technique for only two library pairs.

Migration between different library pairs might need different types of code
changes. Therefore, although the aforementioned techniques show promise, it is
difficult to assess the real capability of these techniques based on the
current evaluation approaches. Thus, researchers would greatly benefit from a
unified taxonomy of the types of code changes required for library migration
as well as a standard evaluation benchmark to use. Such a taxonomy is
essential to provide a clear characterization of the capabilities of each
technique, enabling comparisons as well as identifying unsupported migrations.
Note that Cossette et al. cossette2012seeking  created a taxonomy of code
changes for Java version migration, but it is not clear if the taxonomy
applies to other programming languages or for migration between libraries. For
example, Dilhara et al. evolutionMLLib  find that Python machine learning
version migration requires Python-specific keyword changes. To summarize,
previous library migration techniques hardly focus on (1) non-Java libraries,
(2) migration between different libraries, or (3) facilitating standardized
evaluations using, for example, taxonomies and benchmarks.

In this paper, we address the above three gaps by studying library migrations
in a popular programming language, Python. We mine 60,660 open-source Python
repositories to build PyMigBench, a benchmark of real Python library
migrations, as well as PyMigTax, a taxonomy of migration-related code changes
for client code transformation based on what we observe in the data. To ensure
the correctness of the data we include in PyMigBench, we manually review all
migrations and the details of the code changes. PyMigBench has 59 manually
verified analogous library pairs from 13 different domains. PyMigBench also
has migration-related code changes for 34 of the 59 library pairs in 57 client
repositories. We categorize the code changes in PyMigBench based on different
dimensions and build PyMigTax using the categories.

We envision that PyMigBench and PyMigTax will facilitate new advancements in
supporting Python library migration. Researchers can use the data in
PyMigBench to evaluate and compare all three phases of the library migration
process: library mapping, API mapping, and most importantly client code
transformation. The data in PyMigBench can also be used for training machine
learning-based library migration techniques. Additionally, library migration
techniques can use PyMigTax to clearly label the types of library migrations
they support. For example, one technique may only work for client code
transformation that includes one-to-one API function mapping with no argument
transformation while another may be able to additionally handle argument
changes and modifications to attribute access. By explicitly mentioning which
operations are supported, the techniques can be fairly and systematically
compared based on a subset of PyMigBench that matches their capabilities.

To summarize, this paper makes the following contributions:

  1. 1.

PyMigBench: A benchmark of manually validated Python library migrations with
labeled code changes. PyMigBench includes several artifacts, including sets of
analogous library pairs, and commits having migration and migration-related
code changes.

  2. 2.

PyMigTax: A taxonomy of the different types of migration-related code changes
required for Python library migration techniques.

Both PyMigTax and PyMigBench facilitate the development, evaluation, and
comparison of Python library migration techniques.

##  2 Background and Terminology

In this section, we introduce the terminology we use in this paper.

###  2.1 Library and client software

A library is a reusable software component that can be easily used by other
software systems, including other libraries. The other software systems that
use a library are called its clients. We refer to the developers of these
client systems as client developers. Libraries are usually hosted in online
repositories that allow client developers to install them with simple
commands. The system of repositories and tools that developers use to install
libraries is commonly known as a package management system or package manager.
Multiple versions of the same library can be hosted in the package manager at
the same time, and client developers can specify which version they want to
use.

###  2.2 Python package management

Pip111https://pip.pypa.io is the most commonly used package manager for
Python, and it uses PyPI222https://pypi.org as the default repository. Python
developers often use a requirements.txt file, called a requirements file, to
store library dependencies for client software. Listing 1 shows a sample
requirements file, where each line specifies a library dependency. A library
dependency includes a library name and optionally a version specifier. For
example, in Listing 1, Line 1 indicates that this project depends on any
version of the requests library, while Line 2 indicates that the project
depends on version 3.1 or greater of the GitPython library.

###  2.3 Library migration

In the context of Python, we define library migration as the event where
client developers replace one library in favor of another in the requirements
file. We call the replaced library a source library, the newly added library a
target library, a pair of source and target libraries a library pair, and the
commit where developers replace a source library with a target library a
migration commit.

In addition to the replacement of a source library in the requirements file, a
migration commit might have other code changes in Python files, such as the
replacement or removal of APIs used from the replaced source library. We call
such code changes migration-related code changes or simply code changes for
brevity. In a migration-related code change, one or more source library
program elements can be replaced with one or more target library program
elements. The number of corresponding program elements being replaced in a
code change is expressed by cardinality. Cardinality can be one-to-one, one-
to-many, many-to-one or many-to-many. A code change of x-to-y cardinality
means that it replaces the x number of program elements of the source library
with the y number of program elements of the target library. We often use
higher cardinality as a general term for all cardinalities except one-to-one.
Figure 4 shows an example of one-to-one code change.

We use the term candidate to conservatively state that there is a possibility
of a certain event, but that the event has not been verified yet. For example,
a candidate migration means that the corresponding commit could potentially
contain a library migration. Once we manually verify an event, we use the term
valid, e.g., valid migration.

Listing 1: A sample requirements.txt file

⬇

1 requests

2 GitPython>=3.1

##  3 Methods: Constructing PyMigBench and PyMigTax

Figure 1 shows an overview of how we construct our benchmark of Python library
migrations, PyMigBench, and the taxonomy of Python migration-related code
changes, PyMigTax. We use a combination of automated (white boxes) and manual
(grey boxes) steps. First, we fetch and clone a list of Python repositories
matching various criteria (Step 1). We then iterate through the commits in
these repositories to identify candidate migration commits (Step 2). We
collect the migration information from these commits (Step 3) and manually
review the migrations (Step 4). Finally, we analyze the migration-related code
changes in each of the valid migration commits and use open coding
qualitativeMethods ; qualitativeDataAnalysis  to build PyMigTax (Step 5).

Figure 1: Overview of the construction process of PyMigBench.
White boxes indicate automatic steps while grey boxes indicate manual steps.
The output of steps 4 and 5 constitute PyMigBench and we use output of step 5
to build PyMigTax.

###  3.1 Step 1: Select repositories

We use the SEART GitHub Search Engine dabic2021sampling  to fetch an initial
list of Python repositories. SEART archives GitHub repository metadata and
updates itself every six hours. The dataset is accessible via a
website333https://seart-ghs.si.usi.ch where a user can configure different
filters and download a list of corresponding GitHub repositories. Since our
study focuses on Python, we set the language filter to Python. We also exclude
forks to avoid redundant analysis of the same commits. After applying these
filters, we retrieve 192,889 Python repositories444Received on December 25,
2021. Note that SEART only includes the repositories having 10 or more stars,
so we are also bound to this threshold.

GitHub is a popular host for tutorials, demos, and many other types of
educational materials. Such repositories do not usually represent real
software development projects and can negatively affect our benchmark. Thus,
we filter out a repository if its name contains any of the following keywords:
example, demo, tutorial, sample, or exercise. Using this criteria, we exclude
1,273 repositories resulting in 191,616 repositories. Our study requires the
presence of a requirements.txt file in the repositories to identify library
migration. Using the GitHub search API, we find that 127,148 of the retrieved
repositories do not have this file, which leaves us with 64,468 remaining
repositories.

To minimize the clone time and disk space used, we clone only the default
branch using the \--single-branch option. This reduces download size, because
we noticed that some of the large repositories have large data files which are
not in the default branch. Additionally, we exclude large files in the
repositories using the \--filter=blob:none options. Among the 64,468
repositories, we further discard 3,794 repositories that are no longer
publicly available and 14 repositories that we could not clone due to
miscellaneous errors. We move to the next steps with the remaining 60,660
repositories.

###  3.2 Step 2: Identify candidate migration commits

We automatically analyze the commits of the 60,660 repositories from Step 1 to
find candidate library migration commits. We use the following heuristics to
identify candidate migration commits, while excluding as many non-migration
commits as possible. We mark a commit as a candidate migration commit if all
of the following criteria are true.

  1. 1.

In the default branch. To reduce the computation time to find migrations, we
exclude non-default branches from analysis.

  2. 2.

Not a merge commit. Changes in merge commits are already reflected in the
parent branches. Therefore, considering changes in merge commits will result
in duplicate analysis of the commits already processed in the default branch.

  3. 3.

Changes in dependencies. Since migration is a replacement of one library with
another, the commit must have both additions and deletions of library
dependencies in the requirements file.

  4. 4.

Not a bot-created commit. Some repositories enable bots to update dependencies
automatically. We ignore such commits, because we are interested in migrations
with related code changes.

We use PyDriller555https://pydriller.readthedocs.io, a library to analyze git
repositories, to check the first three criteria. To check whether a commit is
a bot-created commit, we first collect the list of commit authors whose names
contain the term “bot”. We then manually check each of these user profile to
verify whether or not it is a bot. The list of the bot users is available on
our artifact page. Although we do not have an exhaustive list of bots, our
goal is to reduce manual validation overload in the next step, and not to
necessarily eliminate all non-migration commits. The above criteria results in
4,381 candidate library migration commits from 3,152 client repositories.

###  3.3 Step 3: Collect candidate migrations

At this point, our goal is to manually review the candidate migration commits
to check if these commits contain an actual library migration. However,
manually reviewing 4,381 candidate library migration commits is infeasible.
Therefore, we first automatically analyze the potential migrations that
happened in each commit (i.e. candidate migrations), and design heuristics
that automatically filter out unlikely migrations.

For each candidate migration commit, we automatically parse the changes in the
requirements file to extract library migration information. We use PyDriller
to get the lines added and deleted in the requirements file, discarding
comments and empty lines. We then parse the lines to identify the library
specified there. We normalize the library names by making them lowercase and
replacing underscores with a hyphens. For example, we normalize Scikit_Image
to scikit-image. We do this because pip is case insensitive, and underscores
and hyphens are interchangeable666https://peps.python.org/pep-0426/#name.
Developers tend to use different capitilization and frequently use underscores
in place of hyphens and vice versa.

To identify candidate library pairs in a commit, we first collect a set of
added and a set of deleted libraries from the requirements file of this
commit. We then remove the libraries that are present in both sets, regardless
of change in version specification, because we are looking for migrations
between different libraries. Finally, the cross-product of these two sets is
the set of candidate library pairs for this commit. For example, in the
changeset in Listing 2, libraries coverage, pytest-cov, and pytest are added,
and libraries coverage, nose, and nose-timer are deleted. Since the library
coverage is in both sets, we remove this library from the candidate list. This
means that (nose, pytest-cov), (nose, pytest), (nose-timer, pytest-cov), and
(nose-timer, pytest) are the remaining candidate library pairs for this
commit.

⬇

\- coverage==4.5.4

\+ coverage>=5.2.1

\- nose>=1.3.7

\- nose-timer>=0.7.5

\+ pytest-cov==2.11.1

\+ pytest==6.2.0

Listing 2: Sample changes in requirements.txt file

We find that the 4,381 candidate migration commits have a total of 100,103
candidate library migrations between 86,705 unique library pairs. This
corresponds to 22.85 candidate migrations per commit on average, which is a
large number of candidate migrations per commit. After some manual inspection,
we identify that some commits add and delete a lot of dependencies producing a
large number of candidate library migrations. This usually happens when the
developers keep installing and uninstalling libraries in their system without
updating the requirements file immediately, then after a while, update the
file to reflect all previous dependency changes. These commits are unlikely to
contain actual migrations and thus add unnecessary burden to our manual
analysis process.

We use Tukey’s fences outlier detection technique tukey1977exploratory  to
eliminate “suspicious” commits with too many candidate library migrations. In
this approach, the upper bound of the non-outlier is 1.5 times of the inter-
quartile range above the third quartile. In our case, the first quartile (25
percentile) is 1, the third quartile (75 percentile) is 4, so the inter-
quartile range is 3, making the upper limit $4+3\times 1.5=8.5\approx 9$.
Accordingly, we discard any commits with more than nine candidate library
migrations. This is a heuristic approach, and we may have discarded some
actual migration commits in the process. However, this still aligns with our
goal of building a dataset of actual migrations, not necessarily finding all
possible migrations.

After discarding the outlier commits, we have 3,617 candidate migration
commits in 2,046 repositories having 7,423 candidate migrations between 6,047
unique library pairs. On average, this remaining data has 2.1 candidate
migrations per commit.

###  3.4 Step 4: Confirm migrations

In this step, we want to manually review the 7,423 candidate library
migrations from Step 3 to build PyMigBench. However, manually reviewing so
many of candidate library migrations is infeasible. Therefore, to help select
a promising subset of data to manually validate, we rank the candidate library
pairs by the number of migrations that happened for them, and validate the
candidate migrations for the library pairs having 3 or more migrations. This
criteria results in 1,244 candidate migrations between 221 unique library
pairs.

To validate a candidate migration in a candidate migration commit, we first
check whether the library pair involved in the migration contains analogous
libraries. For this, we look into various documentation sources, such as PyPI
pages, official websites, and GitHub repositories of the libraries. We
specifically look at the libraries’ documentation to see whether they
explicitly mention that the two libraries are replaceable with each other or
that the libraries implement similar functionality.

During our manual review, we find that some libraries do not offer an API that
can be used in client code. For example, the library pair
pyflakes777https://github.com/PyCQA/pyflakes and
flake8888https://flake8.pycqa.org are linter tools that are typically used
through the command line. Similarly, ipython999https://ipython.org and
jupyter101010https://jupyter.org/install provide development environments but
are not used in the code. Such libraries are still added in the requirements
files, because they are often part of the build system. Since we are primarily
interested in migration-related code changes, we disregard such library pairs
that do not offer any API. We also find some library pairs that are typically
used as part of the system’s configuration, rather than its client code. For
example, extensions of Sphinx framework (e.g. sphinx-
material111111https://github.com/bashtage/sphinx-material) are usually invoked
by the framework, not directly by client code. The client code only sets the
right configurations to use the extensions. We disregard such library pairs,
because they also do not have any migration-related code changes.
Additionally, we disregard testing libraries, because we are interested in
application code changes. Table 1 shows the different reasons for marking
library pairs as invalid, and how often we encountered them.

Table 1: Reasons why we disregard candidate library pairs  Reason | Occurrences
---|---
Functionally different | 79
Analogous, but configuration based API | 25
Aliases | 14
Analogous, but do not provide API | 13
Different versions have different name | 10
Source or target library not available | 7
Library rename | 5
Correction of library name | 5
Analogous testing libraries | 4
Total | 162

Once we confirm that the library pair of a migration is valid, we look at the
changes that occurred in the commit. We consider a candidate library migration
in a given commit as a valid library migration only if (1) the commit message
or code comments in the commit explicitly indicate the migration or (2) the
commit has code changes related to the migration. To verify the library
migrations, two authors individually review them and then discuss the results
to resolve any disagreements.

At the end of this process, we identify 59 unique analogous library pairs
belonging to 13 different domains as shown in Table 2. These library pairs
contains 254 candidate migrations, from which we find 157 valid migrations
between 49 library pairs in 127 client repositories. We do not find any valid
migrations for the remaining 10 library pairs. The valid migrations are from
155 unique commits, because two of our analyzed commits each have two valid
migrations while all other commits have just one migration. The 59 analogous
library pairs contain 99 unique libraries, among which 55 appear as a source
and 56 appear as a target. PyMigBench consists of this collected migration
data; we describe its details in Section 5.

Table 2: Domains of the analogous library pairs | Number of library pairs
---|---
Domain | Total | With code changes
API wrapper | 3 | 2
Cryptography | 5 | 4
Data processing | 1 | 0
Database client | 4 | 1
Development framework/extension | 8 | 5
File reader/writer | 6 | 5
HTTP client/server | 6 | 3
Image processing | 2 | 1
Logging/tracing | 2 | 2
Machine learning | 3 | 0
Multitasking/multiprocessing | 3 | 3
Networking | 4 | 2
Utilities | 12 | 7
Total | 59 | 34

###  3.5 Step 5: Label migration related code changes

After collecting all migrations that we include in PyMigBench, we now proceed
to analyze the migration-related code changes in the 157 valid library
migrations to build PyMigTax.

For each valid library migration, we manually analyze the modified Python
files in the corresponding commit to understand and extract what types of code
changes occurred during the migration. We do not use any predefined coding
guidelines. Instead, we follow an iterative open-coding process as follows.
The first author of this paper first manually reviews code changes of 11 valid
library pairs having 6 or more migrations. He writes notes to describe the
migration-related code changes he observes (e.g., replace a function call and
add an argument). Based on these notes, he creates an initial version of the
taxonomy that all three authors discuss. During the discussion, all three
authors look at the real examples corresponding to the different types of
noted operations and discuss the structure and naming used in the taxonomy.
Based on the feedback from the discussion, the first author goes back, refines
the taxonomy, and re-labels the data before another round of discussion. This
process continues until all three authors agree that the taxonomy accurately
captures the observed migration-related code changes. This version of the
taxonomy is based on the type of program element affected in a code change,
cardinality of the change, and additional properties of the code change.

To ensure the reproducibility and validity of the labeling for the code
changes, the second author of this paper then uses the developed taxonomy to
separately label the same data that the first author labeled. However, the
second author uses a closed coding labeling process, which means that the
second author strictly uses the labels already defined in the initial version
of the taxonomy. We measure inter-rater agreement between the first and second
authors using Cohen’s kappa cohen1960coefficient  for labels of program
elements and cardinalities. Since the additional properties can have multiple
labels per item, we use Krippendorff’s alpha score krippendorff2018content .
We find that the Cohen kappa score for program elements and cardinalities are
0.99 and 0.68, respectively, which represents almost perfect and substantial
agreements, respectively landis1977measurement . Our Krippendorff’s alpha
score for the properties is 0.76, which is above the suggested 0.667 limit for
drawing conclusions krippendorff2018content . All three authors discuss and
resolve any disagreements that arise.

Given the iterative process above and the high agreement score, we conclude
that the taxonomy is stable and that the labeling process using the taxonomy
is well defined, reducing subjectivity. Thus, the first author then proceeds
to review the code changes in the remaining valid library pairs having three
or more candidate library migrations. He discusses with the other authors
whenever he cannot decide on the labels of any code change. During the review
of the remaining data, the first author observes only one new type of
migration-related code change that is not already present in the initial
version of the taxonomy. The three authors discuss this new finding and update
the taxonomy. Otherwise, no new types of code changes were observed,
suggesting that we have reached saturation in the iterative rounds of creating
the taxonomy.

We find that only 75 of the 157 valid migrations have migration-related code
changes. We could not find any code changes for the remaining 82 valid
migrations due the following three reasons. First, the used APIs of some
library pairs are identical. Therefore, the migrations for such library pairs
do not require code changes. Second, code changes may have happened in a
different commit than the commit where the requirements file was updated.
Third, we could not distinguish migration-related code changes from the code
changes related to refactoring or feature change in a few cases. Therefore, we
do not include such code changes in our benchmark.

Overall, we find 436 migration-related code changes in 75 migrations between
34 unique library pairs. Table 2 shows the number of library pairs having code
changes from different domains. This data comes from 74 unique commits in 57
unique client repositories. Figure 2 shows our final Python Library Migration
Taxonomy, PyMigTax, where we hierarchically arrange the migration categories.
We discuss the details of PyMigTax, along with illustrative real migration
examples, in Section 4 and discuss the occurrence of these migration-related
code changes in PyMigBench in Section 5.

##  4 PyMigTax

Figure 2: PyMigTax: Taxonomy of the migration-related code changes

In this section, we discuss PyMigTax, shown in Figure 2, which we built to
characterize migration-related code changes in Python.

In PyMigTax, we arrange the types of code changes in a hierarchical fashion
with three different hierarchy levels. The first level represents the types of
program elements involved in the code change, such as function call or import.
The second level represents the cardinality of changes related to the program
element in the code change. For example, a cardinality one-to-many under a
function call means that one function call from the source library was
replaced with multiple function calls from the target library. Finally, the
third level represents additional properties of the code changes, such as an
argument addition during a function call replacement.

In the rest of this section, we describe each type of code change in PyMigTax,
illustrating with real migration examples we observe.

###  4.1 Function call replacement

A function call replacement refers to a code change where calls to source-
library functions are replaced with calls to target-library functions.
PyMigTax contains four different cardinalities for function call replacements:
one-to-one, one-to-many, many-to-one, and many-to-many. There are also
different types of additional properties that are associated with function
call replacements, such as argument addition, argument deletion, and argument
transformation.

In the simplest case of the function call replacements, one function call of
the source library is replaced with one function call from the target library
without any change in arguments. Line 33 in Figure 3 shows an example where
client developers replace a function call Flask() from the flask library with
a function call Quart() from the quart library without changing the passed
argument.

Figure 3:  (Line 1) import; one-to-one; module name change and object name
change. (Line 33) function call; one-to-one; function name change.
Commit: learningOrchestra/mlToolKits#db7f132

Note that some analogous libraries can have analogous functions with the same
name. However, these functions may have different parameter types or number of
parameters. Therefore, migrations between these libraries require addition,
deletion, or transformation of arguments. Figure 4 shows an example where
client developers replace a function call new() from the pycrypto library with
a function call new() from the pycryptodome library. Despite both functions
having the same name, their parameters are different. Therefore, in this case,
to maintain the same expected functionality, the client developers transform
the first argument and add a second argument to the replaced call.

Figure 4:  function call; one-to-one; argument transformation and argument
addition
Commit: camptocamp/c2cgeoportal#14388c3

During migration, a developer may replace a synchronous function call with an
asynchronous one that requires using an await keyword before the target
function call to work properly. PyMigTax represents this possibility with the
“making await” property. Figure 5 shows an example where client developers
replace a function call render_template() from the flask library (line 149)
with a corresponding function call render_template() from the quart library
(line 110) while adding the await keyword, since the former is synchronous
while the latter is asynchronous.

Figure 5: function call; one-to-one; making await
Commit: elblogbruno/notionai-mymind#002f5bd Figure 6: function call; one-to-
one; argument transformation
Commit: openstack/oslo.messaging#5a842ae

Note that the argument transformation property can span multiple arguments in
a function-call replacement. For example, in the code change in Figure 6, the
client developers replace KafkaProducer() of kafka-python with the Producer()
function of confluent-kafka. KafkaProducer() takes as input a number of named
arguments, while the Producer() takes just one dictionary where each
dictionary key corresponds to an argument name of the KafkaProducer()
function. For example, the named argument bootstrap_servers corresponds to the
dictionary key bootstrap.servers.

In addition to one-to-one function call replacements, there are higher
cardinality function call replacements with one-to-many, many-to-one, or many-
to-many replacements. Since these require the replacement of multiple function
calls, we do not consider the property function name change for these
cardinalities. Figure 7 shows an example of a many-to-one replacement where
client developers replace the function calls Sentry() and init_app() from the
raven library with a single function call init() from the sentry-sdk library.
Furthermore, during this replacement, the client developers remove the app
argument and add two arguments traces_sample_rate and integrations. Figure 8
illustrates a many-to-many function call replacement where client developers
replace function calls WSGIServer() and serve_forever() from the gevent
library with function calls server() and listen() from the eventlet library
without changing the arguments.

Figure 7: function call; many-to-one; argument addition, argument deletion
Commit: agdsn/sipa#ea23791 Figure 8: function call; many-to-many
Commit: duanhongyi/dwebsocket#d707ff6

###  4.2 Decorator replacement

A decorator replacement refers to a code change where decorators from the
source libraries are replaced with decorators from the target libraries. Based
on the observed data, PyMigTax includes only one-to-one decorator replacements
requiring name change or argument transformation. Figure 9 shows an example
where client developers replace @db_task decorator from the huey library with
the @shared_task decorator from the celery library. Figure 10 shows another
example of a decorator replacement where client developers replace the @retry
decorator from the retrying library with the @retry decorator from the
tenacity library while transforming the passed arguments to match the
expectations of the new decorator.

Figure 9: decorator; one-to-one; decorator name change
Commit: lonelam/onlinejudgeshu#b687d20 Figure 10: decorator; one-to-one;
argument transformation
Commit: intelai/inference-model-manager#71aff3a

###  4.3 Attribute access replacement

An attribute access replacement refers to a code change where attribute access
from the source library is replaced with attribute access from the target
library. This part of PyMigTax also includes cases where an attribute access
from a source library is replaced with a method call from a target library.
All attribute access replacements in PyMigTax are one-to-one and require a
name change or making a synchronous call asynchronous. Figure 11 shows an
example where client developers replace an attribute access
Crypto.PublicKey.RSA.RsaKey from the pycryptodome library with an attribute
access Cryptodome.PublicKey.RSA.RsaKey from the pycryptodomex library. Figure
12 demonstrates another example of an attribute access replacement where
client developers replace the attribute content (line 36) from the requests
library with a method call text() (line 38) from the aiohttp library and make
the method call asynchronous by adding a keyword await before the method call.

Figure 11: attribute access; one-to-one; attribute name change
Commit: malwaredllc/byob#9291b54 Figure 12: (deleted line 36, added line 38)
attribute access; one-to-one; changing to method call, making await and
attribute name change
(deleted line 35, added line 36-37) function call; one-to-many; argument
addition, making async
Commit: raptor123471/dingolingo#1d8923a

###  4.4 Import replacement

An import replacement refers to a code change where import statements from a
source library are replaced with import statements from a target library. A
Python import statement can import just a module or some objects from a
module. Objects in import statements can represent functions, classes, or some
other program elements. For simplicity, PyMigTax does not differentiate the
object types used in an import statement.

Import replacements can include a change in module and object names. Figure 13
shows an example where client developers replace a module openpyxl from the
openpyxl library with a module xlsxwriter from the xlsxwriter library. Figure
3, Line 1, demonstrates another example of an import replacement where client
developers replace a module flask and an object Flask from the flask library
with a module quart and an object Quart from the quart library, respectively.

Figure 13: import; one-to-one; module name change
Commit: bcgov/gwells#57d12c4

In some cases, we cannot clearly map modules and objects between source and
target libraries. We refer to such a replacement as a full statement
replacement. A full statement replacement typically happens when one library
requires importing only objects while the other requires importing objects
from specific modules. For example, Figure 14 shows a migration from the
eventlet to gevent library with a full statement replacement. In this example,
the client developers replace an import statement that specifies only an
object eventlet with an import statement that specifies an object monkey from
the module gevent. Figure 15 shows an example of a higher cardinality import
replacement where client developers replace an import statement import slack
from the slackclient library with import statements from slack_sdk.web import
WebClient and from slack_sdk.rtm import RTMClient from the slack-sdk library.

Figure 14: import; one-to-one; full statement replacement
Commit: stefal/rtkbase#a4c347a Figure 15: import; one-to-many; full statement
replacement
Commit: slackapi/python-slack-sdk#5f4d92a

##  5 PyMigBench

In this section, we describe PyMigBench in detail. We first describe the
format of the data in PyMigBench and the tooling we provide to use this data.
We then provide descriptive statistics of various characteristics of the
migrations included in PyMigBench.

###  5.1 Data format and tool support

PyMigBench contains the manually verified data we collected in a reusable
format. We make PyMigBench publicly available for other researchers to verify
our work and use in their research121212https://github.com/ualberta-
smr/PyMigBench. There are three main types of data items included in
PyMigBench: library migrations, the migration-related code changes included in
these migrations, and the set of library pairs between which the migrations
occurred. PyMigBench contains one YAML file for each data item. We choose YAML
because it is concise and human readable.

Listing 3: A sample YAML file describing the library pair flask and quart

⬇

source : flask

target : quart

number_of_migrations : 6

number_of_code_changes : 46

We provide a YAML file for each analogous library pair present in PyMigBench
to summarize the migrations related to this library pair. Listing 3 shows a
sample YAML file for the library pair flask and quart. This YAML file includes
the source and target library names, the number of migrations for this library
pair, and the overall number of migration-related code changes for this pair.
As shown in Listing 3, there are 6 migrations and 46 migration-related code
changes for the library pair flask and quart. The information in this summary
allows users of the benchmark to, for example, quickly search for library
pairs with enough migration instances.

Listing 4: A sample YAML file describing one of the 6 migrations between flask
and quart shown in Listing 3

⬇

source : flask

target : quart

repo : synesthesiam/voice2json

commit_hash : 7ea7ddb8400775282e82c1adcb17b013f27ede2b

commit_message : "Using quart for web interface"

number_of_code_changes : 13

We also provide a YAML file for each library migration. A sample YAML file for
one of the 6 migrations from flask to quart is shown in Listing 4. The YAML
file describing a migration contains the source and target library names, the
client repository where the migration was observed, the commit hash of the
migration commit, the commit message, and the number of code changes in this
migration. For example, Listing 4 shows that this specific migration from
flask to quart has 13 code changes.

Listing 5: A file describing one of the 13 migration-related code change
happened in the migration in Listing 4

⬇

repo : synesthesiam/voice2json

commit_hash : 7ea7ddb8400775282e82c1adcb17b013f27ede2b

file_path : web/app.py

source_version_lines : 20

target_version_lines : 20

program_element : import

cardinality : one-to-one

properties : module name change,object name change

source : flask

target : quart

Finally, we provide a YAML file for each code change that happened in a
migration. Listing 5 shows an example of one of the 13 code changes that
occurred in the migration shown in Listing 4. This file includes the client
repository name, the migration commit hash, the modified file path, the line
numbers where the change happened, the PyMigTax labels describing the code
change, and the source and target library names. For example, Listing 5 shows
that this code change is a one-to-one import statement replacement with a
change in module and object names. In addition to the YAML files describing
the code changes in a given migration, we provide the Python files that
contain the actual code changes for the migration. We provide the before-
change version, after-change version, and the diff of the python files. The
diff is in combined git diff format131313https://git-scm.com/docs/git-
diff#_combined_diff_format, which can be viewed with a standard git diff
viewer or any text editor. Providing these files allows techniques to directly
use the data in PyMigBench without the need to clone and process the original
repositories where the migrations happened.

PyMigBench comes with a command-line tool to facilitate data processing and
usage of the benchmark. A user can use the tool to view summaries of the
benchmark (i.e. descriptive statistics of the included data) as well as to
query the benchmark by different dimensions. Some sample queries that we
provide are: find all migration-related code changes that replace a function
call and find all migrations to target library “aiohttp”. This allows other
researchers to slice the data for an evaluation or additional analysis
according to the capabilities of the migration techniques they are designing.
Our PyMigBench repository contains more information about how to use the
dataset and the accompanying toolchain.

###  5.2 Client projects

PyMigBench provides examples of real migrations. To provide confidence in
these migrations, we discuss the characteristics of the repositories they are
extracted from.

We mention in section 3.1 that we analyze commits in repositories having 10 or
more stars. This ensures a minimum quality of the client repositories. Figure
16 shows the distribution of various characteristics of the 57 client
repositories from which the migrations in PyMigBench come from. As shown in
the figure, the median values of stargazer, fork and contributor counts for
the client repositories are 103, 28, and 11, respectively; compared to all
Python repositories on GitHub with ten or more starts being 26, 9, and 3,
respectively. Overall, we can see that most of the client repositories are
popular with several contributors, giving us confidence in the quality of
these repositories.

Figure 16: Statistics of the client repositories in PyMigBench as of June 23,
2022. The black markers represent median value for all Python repositories on
GitHub with 10 or more stars for the corresponding metric. The plots show up
to the 80 percentile of data for better visualization.

###  5.3 Statistics of migration-related code changes

Table 3: Summary of library pairs having migration-related code changes

|  |  | # | # code
---|---|---|---|---
Domain | Source library | Target library | migrations | changes
_API wrapper_ | kafka-python | confluent-kafka | 2 | 7
slackclient | slack-sdk | 4 | 20
|  | 6 | 27
Cryptography | py-bcrypt | bcrypt | 1 | 2
pycrypto | cryptography | 1 | 1
pycrypto | pycryptodome | 2 | 5
| pycryptodome | pycryptodomex | 3 | 34
|  |  | 7 | 42
Database client | pymilvus-orm | pymilvus | 3 | 9
|  | 3 | 9
_Development framework/extension_ | flask | fastapi | 2 | 3
flask | quart | 6 | 46
flask-restful | flask-restplus | 3 | 11
flask-restplus | flask-restx | 4 | 53
pyqt5 | pyside6 | 1 | 76
|  | 16 | 189
_File reader/writer_ | openpyxl | xlsxwriter | 1 | 2
pyyaml | ruamel.yaml | 3 | 5
ruamel.yaml | pyyaml | 2 | 17
xlsxwriter | openpyxl | 1 | 2
|  | 7 | 26
_HTTP client/server_ | flask | uvicorn | 1 | 2
gcloud-aio-core | aiohttp | 3 | 5
requests | aiohttp | 4 | 30
|  | 8 | 37
_Image processing_ | pil | pillow | 2 | 2
|  | 2 | 2
_Logging/tracing_ | logbook | loguru | 2 | 4
raven | sentry-sdk | 5 | 19
|  | 7 | 23
_Multitasking/multiprocessing_ | celery | rq | 1 | 1
huey | celery | 1 | 2
lockfile | fasteners | 1 | 16
|  | 3 | 19
_Networking_ | eventlet | gevent | 1 | 2
gevent | eventlet | 2 | 5
|  |  | 3 | 7
_Utilities_ | argparse | configargparse | 1 | 9
dataproperty | typepy | 3 | 24
django-rest-swagger | drf-yasg | 2 | 2
fuzzywuzzy | rapidfuzz | 1 | 1
python-ldap | ldap3 | 2 | 4
pytz | pendulum | 2 | 3
retrying | tenacity | 2 | 12
|  | 13 | 55
Total 11 domains and 34 library pairs |  | 75 | 436

Table 4: Statistics of migration-related code changes.
The last 4 columns mean: CC = code changes, ULP = unique library pairs, UM = unique migrations, UR = unique client repositories. Note that the numbers in a property level may not add up to the number at corresponding cardinality level because each code change can have multiple properties.  Program | Cardinality/ | # CC | # ULP | # UM | # UR
---|---|---|---|---|---
elements | properties |  |  |  |
_ Function calls _ | _one-to-one_ | 135 | 20 | 35 | 29
argument addition | 16 | 7 | 9 | 9
argument deletion | 24 | 8 | 10 | 9
argument transformation | 12 | 5 | 7 | 7
function name change | 93 | 18 | 29 | 27
making await | 20 | 2 | 6 | 5
_one-to-many_ | 13 | 5 | 7 | 6
argument addition | 5 | 3 | 3 | 3
argument deletion | 2 | 1 | 1 | 1
argument transformation | 4 | 3 | 3 | 3
making async | 6 | 1 | 3 | 2
no properties | 2 | 1 | 1 | 1
_many-to-one_ | 6 | 5 | 6 | 6
argument addition | 2 | 2 | 2 | 2
argument deletion | 3 | 3 | 3 | 3
no properties | 3 | 2 | 3 | 3
_many-to-many_ | 1 | 1 | 1 | 1
no properties | 1 | 1 | 1 | 1
| 155 | 24 | 43 | 35
_Imports_ | _one-to-one_ | 166 | 28 | 53 | 44
module name change | 126 | 15 | 31 | 27
object name change | 5 | 3 | 5 | 5
full statement replacement | 40 | 16 | 22 | 19
_one-to-many_ | 10 | 7 | 10 | 10
full statement replacement | 10 | 7 | 10 | 10
_many-to-one_ | 7 | 5 | 6 | 5
module name change | 3 | 1 | 2 | 1
full statement replacement | 4 | 4 | 4 | 4
_many-to-many_ | 5 | 4 | 5 | 5
full statement replacement | 5 | 4 | 5 | 5
|  | 188 | 32 | 71 | 55
_ Attribute access _ | _one-to-one_ | 80 | 4 | 7 | 7
changing to method call | 4 | 2 | 3 | 3
making await | 9 | 2 | 4 | 4
attribute name change | 74 | 4 | 6 | 6
| 80 | 4 | 7 | 7
_Decorators_ | _one-to-one_ | 13 | 3 | 4 | 4
argument transformation | 7 | 1 | 1 | 1
decorator name change | 6 | 2 | 3 | 3
| 13 | 3 | 4 | 4
Total |  | 436 | 34 | 75 | 57

PyMigBench contains 436 migration-related code changes spread across 75
migrations between 34 unique library pairs. Table 3 shows that the library
pairs span 11 different domains, with 31 unique source libraries and 33 unique
target libraries, demonstrating the diversity of the data. Table 4 shows a
summary of all the migration-related code changes available in PyMigBench. We
now discuss the details of these code changes.

##### Function call replacements.

PyMigBench contains 155 examples of real migrations between 24 unique library
pairs involving function call replacements. Among these replacements, 135
replacements (87%) are one-to-one replacements between 20 library pairs.
However, only 93 of these 135 replacements require a function name change,
which means the remaining 42 replacements have the same function name in the
source and target libraries. Moreover, only 52 of the 135 replacements require
a change in arguments, including argument addition, deletion, and
transformation. The remaining 83 one-to-one function call replacements do not
require any change in arguments, because the source and target function calls
either do not take any arguments or take the same arguments. This data
suggests that a large number of one-to-one function call replacements do not
require a change in function name or arguments. Among the 20 higher-
cardinality function call replacements, 19 replacements are one-to-many or
many-to-one replacements. Only one function call replacement is a many-to-many
replacement. Overall, PyMigBench contains examples of real migrations
involving function call replacements with various cardinalities and
properties.

##### Import replacements.

PyMigBench contains 188 examples of real migrations involving code changes in
import statements. In comparison to the 24 library pairs that require function
call replacements, Table 4 shows that migrations between 32 of the 34 library
pairs (94%) require code changes in import statements. This suggests that some
libraries may be migrated by only replacing imports. Similar to the function
call replacements, most (88%) of the import migrations in PyMigBench are one-
to-one replacements, and a few import migrations are of all three higher-
cardinality replacements.

##### Attribute access replacements.

PyMigBench contains 80 examples of code changes requiring attribute access
replacement, all of which are one-to-one and are between only 4 library pairs.
One possible reason behind not finding too many migrations with attribute
access replacement may be that public APIs are generally exposed via
functions, not attributes.

##### Decorator replacements.

Finally, PyMigBench contains 13 examples of migrations with decorator
replacements, all of which are one-to-one replacements and are between only 3
library pairs.

###  5.4 Complexity of migrations

While the type of a code change itself may indicate what capabilities a
migration technique requires, the combination of different types of code
changes in a given migration affects its complexity. Migrations that require
various types of code changes or changes that are spread across multiple files
are likely to be more complex to automate. To provide an idea about the
complexity of migrations in PyMigBench, we look at all migration-related code
changes in a migration and qualitatively measure the complexity of the
migration. We specifically check the types of program elements replaced in a
migration, their cardinalities, and the number of different types of
additional properties involved in the migration.

We use maximum cardinality of a migration to quantify the complexity of a
migration based on replacement cardinality. For this, we take the
cardinalities of all code changes in a migration. We then find the highest
cardinality and consider it as the maximum cardinality of the migration. This
requires a relative order between the cardinalities. We consider one-to-one as
the lowest cardinality and many-to-many as the highest. One-to-many and many-
to-one are equal to each other and are in between the other two extremes.
Table 5 shows the maximum cardinality for the 75 migrations in PyMigBench.
Most of the migrations (59%) have maximum cardinality of one-to-one. However,
PyMigBench also has a high number of migrations (35%) with one-to-many or
many-to-one cardinality.

Table 5: Distribution of maximum cardinality of migrations Maximum cardinality | # (%) migrations
---|---
one-to-one | 44 (58.7%)
one-to-many or many-to-one | 26 (34.7%)
many-to-many | 5 (6.67%)
Total | 75 (100%)
Table 6: Distribution of combination of program elements replaced during migrations Program elements | # (%) migrations
---|---
import | 30 (40%)
function call and import | 30 (40%)
attribute access, function call, and import | 7 (9.3%)
function call | 4 (5.3%)
decorator, function call, and import | 2 (2.7%)
decorator, and import | 2 (2.7%)
Total | 75 (100%)

Table 6 shows the six unique combinations of different program elements
involved in the 75 migrations. PyMigBench has a high proportion of migrations
(40%) that require replacements of only import statements. However, the
majority of the migrations (55%) require replacements of import statements
plus one or more other program elements. The migrations in PyMigBench have a
total of 25 unique combinations of properties. We now discuss three migrations
from PyMigBench that we consider to be the most complex.

Figure 17: Migration from requests to aiohttp in commit
raptor123471/dingolingo#1d8923a

The first migration with the highest number of properties is from requests to
aiohttp; it has 7 different properties and a maximum cardinality of one-to-
many. The code changes in this migration are shown in Figure 17 where each
code change is marked with a letter on the left side. The corresponding commit
modifies 3 files, however, all migration-related code changes are in just one
file (musicbot/linkutils.py). This migration replaces an import, two attribute
accesses, and two function calls. The import statement has a module name
change from requests to aiohttp (A). The accesses to attribute content are
replaced by awaitable method calls text() (B and C). The two function call
replacements are one-to-many (D and E). In each case, a synchronous function
call get() is replaced with asynchronous function calls get() and
ClientSession(). However, an argument headers has been added in one of these
function call replacements (D), whereas an argument headers has been
transformed in the other function call replacement (E).

Figure 18: Migration from requests to aiohttp in commit ictu/quality-
time#d3a9a16 (part 1 of 3)  Figure 19: Migration from requests to aiohttp in
commit ictu/quality-time#d3a9a16 (part 2 of 3)  Figure 20: Migration from
requests to aiohttp in commit ictu/quality-time#d3a9a16 (part 3 of 3)

The second migration is also from requests to aiohttp. It has 6 different
properties in the code changes and a maximum cardinality of one-to-one. The
code changes in this migration is shown in Figures 18, 19, and 20. The code
changes are spread across four different Python
files141414https://github.com/ictu/quality-time/commit/d3a9a16. This migration
replaces three imports, two attribute accesses, and eight function calls. The
figures only show three of the eight function call replacements for brevity.
All the three import statements have a module name change from requests to
aiohttp as shown in Figures 18 (F), 19 (I), and 20 (K). Among the two
attribute accesses, one attribute access has only a name change from Response
to ClientResponse (Figure 19 J), whereas the other attribute access text is
replaced with an awaitable method call text() (Figure 19 H). In one of the
eight function call replacements, the function call get() is replaced with the
same named function call where the auth argument is transformed and timeout
argument is deleted (Figure 18 G). In the remaining seven function call
replacements, a synchronous function call is replaced with an asynchronous
function call (Figure 20 L and M). In one of the seven synchronous function
call replacement, the function call post() is replaced with the same named
function call where the timeout argument is also deleted (Figure 20 L).

Figure 21: Migration from kafka-python to confluent-kafka in commit
biznetgio/restknot#6b10345

Finally, the third example is a migration from kafka-python to confluent-
kafka151515https://github.com/biznetgio/restknot/commit/6b10345 shown in
Figure 21. It has 6 different properties in the code changes and a maximum
cardinality of one-to-many. This migration replaces an import and three
function calls. The import has a module name change from kafka to
confluent_kafka and an object name change from KafkaProducer to Producer (O).
Among the three function call replacements, two function call replacements are
one-to-one. In one case, the function call KafkaProducer() is replaced with
the function call Producer() where the first argument is transformed and the
second argument is deleted (P). In the other case, the function call send() is
replaced with the function call produce() where the second argument is
transformed and the third argument is added (Q). The remaining one function
call replacement is one-to-many where the function call get_kafka_consumer()
is replaced with the function calls Consumer() and suscribe() (N). In this
replacement, the broker argument is deleted and the conf argument is added.

The above examples demonstrate that some migrations are quite complex and
require a transformation technique to be able to handle several types of code
changes, as well as changes in several code files. The remaining migrations in
PyMigBench have 4 or fewer properties so they represent less complexity.

##  6 Discussion

In this paper, we created PyMigBench, a benchmark of Python library
migrations, and PyMigTax, a taxonomy of migration-related code changes, by
mining library migrations and analyzing corresponding code changes in Python
projects. We now demonstrate how our contributions can help identify gaps in
the current literature, present the implications of our contributions, as well
as discuss various open challenges in this research area.

###  6.1 Identifying gaps in the current library migration research

In this section, we use PyMigTax to identify the types of real-world
migrations that existing library migration techniques support. We specifically
discuss four state-of-the-art API mapping techniques teyton2013automatic ;
alrubaye2018automating ; alrubaye2019use ; alrubaye2020learning  and, to the
best of our knowledge, the only existing client code transformation technique
for library migration ni2021soar . Note that the API mapping techniques
teyton2013automatic ; alrubaye2018automating ; alrubaye2019use ;
alrubaye2020learning  are for Java and the transformation technique ni2021soar
is for Python. We also demonstrate the usefulness of PyMigTax in labeling and
comparing library migration techniques.

##### Gaps w.r.t program elements.

All five techniques consider only function call migrations teyton2013automatic
; alrubaye2018automating ; alrubaye2019use ; alrubaye2020learning ; ni2021soar
while we found examples of migrations involving imports, attribute access, and
decorators as well. We believe that it is important for library migration
techniques to consider all the program elements that may impact migration. We
speculate that previous studies do not explicitly mention or consider the
other program elements, because attributes are usually not part of public APIs
(especially in Java), decorators use can be considered as a type of function
call, and import statements can be inferred from API usages for statically
typed languages like Java. However, we argue that clearly labeling the types
of supported program elements and supported migration operations facilitates
more systematic comparison and evaluation.

##### Gaps w.r.t cardinalities.

Through PyMigTax, we considered four types of migration cardinality. Although
Teyton et al.teyton2013automatic , Alrubaye et al.alrubaye2018automating  and
Chen et al.miningAnalogicalAPIs  support only one-to-one API mapping, Alrubaye
et al.alrubaye2019use  improved their technique to support one-to-many and
many-to-many API mappings but do not explicitly mention support for many-to-
one. This may be because many-to-one can be seen as a special case of many-to-
many. We note that SOAR ni2021soar  provides limited support for one-to-many
API transformation only through pre-defined knowledge of a small set of
functions.

##### Gaps w.r.t transformations.

Most notably, the above API mapping techniques teyton2013automatic ;
alrubaye2018automating ; alrubaye2019use ; alrubaye2020learning  map only the
function names without providing any mapping or details w.r.t change in
arguments or other details needed for an actual transformation. As a client
code transformation technique, SOAR ni2021soar  does provide support for
argument addition, deletion, and transformation during client code
transformation. However, it does not support transformation involving
async/await keywords.

In summary, our work helps in identifying the following gaps in the library
migration literature: (1) there is no adequate support for migrations for
program elements other than function calls, (2) client code transformation
techniques need better support for higher cardinality migrations, and (3)
researchers need to work towards supporting migration from synchronous to
asynchronous APIs. Note that these gaps are based on a theoretical comparison
of the capabilities mentioned in each respective publication. A proper
empirical evaluation of these capabilities can shed more light on current
gaps, which is something PyMigTax and PyMigBench can help with as discussed
next.

###  6.2 Implications of our contributions

The main goal of our contributions is to facilitate the advancement of Python
library migration techniques. In this direction, we make three main
contributions that have various implications.

##### Facilitating library mapping research.

We provide a collection of manually validated analogous Python library pairs.
PyMigBench has 59 analogous library pairs from 13 different domains. There are
currently no techniques that automatically extract analogous Python library
pairs. Although there are automated techniques for extracting Java library
pairs, they are imprecise and thus generate a large number of false positives
teyton2012mining ; teyton2014study ; he2021multi . Therefore, automatically
extracting library pairs typically requires manual validation, which can be
tedious. Therefore, our set of validated Python library pairs can save
researchers this manual validation time and facilitate the advancement of
automated mapping techniques. Researchers can use the library pairs available
in PyMigBench for evaluation or as training data for machine learning-based
techniques. While the number of library pairs may be limited for certain types
of machine learning techniques, researchers can use this data to identify
interesting/differentiating features that can help build such techniques.
Overall, this contribution facilitates the evaluation of library mapping
techniques and the advancement of techniques to find analogous library pairs.

##### Facilitating migration-related code transformation research.

We provide, to the best of our knowledge, the first-ever manually validated
migration-related code changes from real-world migrations. PyMigBench contains
75 library migrations between 34 library pairs with 436 migration-related code
changes in 57 client projects. Library migration techniques can directly use
these client projects in their evaluation. Since we have already validated
migration-related code changes in these client projects, researchers can
verify the effectiveness of library migration techniques by matching the
transformed code with the corresponding code changes in PyMigBench. For
example, SOAR ni2021soar  is currently the only Python library migration
technique we are aware of. Although SOAR was evaluated on 40 client projects,
these projects are tutorials and architectures, which might not represent real
projects. Also, the authors evaluated the effectiveness of SOAR by running the
existing tests available in the client projects. However, the authors
acknowledge the limitation that the tests might not even cover the code
transformed by SOAR. Furthermore, the authors evaluated SOAR on only one
Python library pair (apart from a different R library pair to show language
generalizability). We believe that SOAR’s evaluation can be strengthened to
overcome these limitations by using PyMigBench.

PyMigBench also provides access to each of the migration-related code changes
categorized by library pairs. This not only facilitates the identification of
different types of program elements that require transformation but also
facilitates the mapping of semantically similar program elements for the
library pairs. Similar to library mapping, researchers can use the data, such
as function call replacements, to develop techniques to find analogous APIs
across libraries. Researchers can also use the real examples of code changes
in PyMigBench to develop client code transformation techniques. For example,
although both function call replacements in Figures 4 and 6 require argument
transformation, the first one requires a simple addition of a function call on
an argument while the other requires the conversion of the arguments to a
dictionary. These insights from real examples can directly help researchers in
developing code transformation techniques for Python library migration.
Overall, this contribution facilitates the evaluation of client code
transformation techniques during migration and the advancement of techniques
to find analogous APIs.

##### Facilitating systematic labeling and comparisons

We develop PyMigTax, a taxonomy of the 436 migration-related code changes from
75 real-world library migrations. Researchers can use PyMigTax to properly
label the capabilities of their techniques and compare library migration
techniques and identify their limitations. For example, SOAR ni2021soar
supports one-to-one and one-to-many function call replacements along with
function name change and argument addition, deletion, and transformation.
However, SOAR does not support decorators, attribute accesses, and many-to-one
or many-to-many function call replacements present in PyMigTax. Moreover, SOAR
does not support the addition of asynchronous keywords, which means it cannot
correctly migrate from synchronous to asynchronous libraries. As demonstrated
in Section 6.1, PyMigTax can be used by researchers to identify limitations of
existing library migration techniques and then develop new techniques by
addressing the limitations.

Overall, our contributions, PyMigBench and PyMigTax, can help researchers in
developing library migration techniques and, equally importantly, in
systematically and fairly evaluating and comparing these techniques.

###  6.3 Key challenges in creating a benchmark and taxonomy of library
migration

Given the above-demonstrated importance of developing benchmarks and
taxonomies that can help advance the state of the art, we dedicate this
section to discussing two key challenges we observed in making such
contributions.

##### Identifying migration-related code changes.

Developers often commit code changes related to different purposes, such as
library migration and functionality improvement, together. Such code changes
are commonly referred to as tangled changes herzig2013impact ; kirinuki2014hey
. As with other types of studies that require extracting or understanding code
changes related to a specific purpose tang2021using , one main challenge in
creating a benchmark and taxonomy for library migrations is to distinguish
migration-related code changes. In general, identifying code changes related
to libraries, and thus migration-related code changes, is easier than
identifying logical code changes. However, since Python is a dynamically typed
language, identifying migration-related code changes in Python client projects
is a non-trivial task. For example, it is difficult to determine whether a
parameter of a client function is a library object without checking the usage
of that client function. While creating PyMigBench, we had to look into not
only the changed code but also other client code and library documentation to
identify and validate migration-related code changes. In general, the
advancements in techniques related to analyzing tangled changes
dias2015untangling ; wang2019cora  in Python source code and type inference
hassan2018maxsmt ; pytype  can help in identifying migration-related code
changes.

##### Manual Effort.

PyMigBench and PyMigTax provide a necessary platform to facilitate the
evaluation, comparison, and advancement of Python library migration
techniques. However, due to a lack of relevant automated tools and techniques
as well as imprecision of the available tools and techniques, we mainly
resorted to using a manual approach to create PyMigBench and PyMigTax. Other
researchers in this research area also resorted to manual validation, for
example, to identify analogous library pairs, because the automated tools they
developed create a lot of false positives teyton2012mining ; he2021multi .
Recall that after many levels of automatic filtering, we manually validated
only top-ranked candidate library pairs. However, we still identified 73% (162
of 221) top-ranked candidate library pairs as invalid library pairs. Moreover,
even for the 59 valid library pairs, we identified 48% (97 of 254) candidate
library migrations as invalid library migrations. This indicates that it is
difficult to automatically identify even highly probable library migration
data, let alone valid library migration data even after several filtering
heuristics. Therefore, we believe PyMigBench can help in two ways towards
advancing the creation of benchmarks. First, given the structured
representation of our data, researchers can contribute additional manually
reviewed data through pull requests (details in our repository). Although this
does not solve the need for large manual effort, this helps distribute the
manual effort and consolidate the data. Second, researchers can study
PyMigBench data to identify promising characteristics that can be used to
automatically flag migrations with high confidence for review. For example, an
idea is to develop a static analysis technique that analyzes the code changes
to detect the types of operations we observed in PyMigTax and mark these
commits for review. This could potentially lead to less manual review effort,
because it is a smaller set of candidates with higher confidence.

##  7 Threats to validity

We now discuss potential threats to the validity and quality of PyMigBench and
PyMigTax.

###  7.1 Internal validity

Our process to collect migrations has various steps that may lead to missing
data. We discuss each of these.

##### Missing repositories.

The SEART dataset contains repositories having ten or more stars. We,
therefore, do not analyze repositories having fewer than ten stars. Also,
SEART updates its dataset every six hours, therefore, it may be slightly
outdated compared to the data available on GitHub. We used the GitHub API to
verify the number of Python repositories with ten or more stars, and GitHub
reported 213,524 repositories while SEART provides 198,883 repositories with
the same filters on the same date161616We collected both sets of data on
December 25, 2021. This means that our repository list contains about 93% of
the actual repositories, which we consider a reasonably large portion.

##### Missing commits.

We may have missed some potential migration commits for the following reasons.
(1) We exclude forks to avoid unnecessary duplicate commit analysis and save
processing time. However, depending on whether a fork was integrated back into
the main branch and how this integration happened (i.e., squash, rebase, etc),
the fork may contain additional commits which are not in the original
repository. (2) We only search for commits that modify files named
requirements.txt. While this is a common name choice for the requirements
file, developers may use different names such as requirements-dev.txt. We do
not detect changes in such files. (3) We exclude merge commits and commits in
non-default branches. Based on how the developers merge, the commits in other
branches may or may not appear in the default branch. (4) Our process of
identifying migrations assumes that the addition and deletion of dependencies
happened in the same commit, which may not always be the case
teyton2013automatic . However, finding the right time window to use to process
future commits is not straightforward, but is a potential avenue for extending
our data.

##### Missing code changes.

When identifying code changes, we only look at the commit where the library
dependency was updated. However, code changes may be done in later commits as
well, therefore, we may have missed some migration-related code changes.
Precisely and automatically finding such commits can be challenging, which is
why it is common for library migration studies to focus on one commit as we
did teyton2012mining .

When finding migration-related code changes, we only look at visible
modifications that are identified by git’s diff. Therefore, we miss cases
where an API is replaced with an identical one (e.g., same name and signature
for function calls) since these do not result in a commit diff. The same file
may have import changes that we detect, but we will not label such a change
with the, for example, function replacements since these replacements do not
result in a diff. While such code changes could add some items to PyMigTax, we
argue that such cases are already not interesting for library migration
research since if the APIs are identical, no migration action is required.
Instead, our work focuses on code changes that need to happen during
migration.

Being able to identify all the above missing data may potentially increase the
diversity in PyMigBench. However, this does not affect the validity of our
results because our main goal is to build a benchmark of real migrations, not
necessarily find all migrations that happened in repositories hosted on
GitHub.

##### Higher cardinality library pairs.

We assume that one source library is replaced with exactly one target library
during a migration. However, we observe two commits where flask is replaced
with both fastapi and uvicorn 171717https://github.com/bretttolbert/verbecc-
svc/commit/24a848d
181818https://github.com/virtuber/openvtuber/commit/3abbc43. In both commit
messages, the developers explicitly mentioned that they are replacing flask
with the two libraries, and it is also evident from the code changes. To
maintain consistency within the benchmark, we store the two library pairs
flask,fastapi and flask,uvicorn separately. Teyton et al.teyton2014study  also
acknowledge the possibility of such migrations; however, similar to us, they
also do not consider such cases during the library pair mining algorithm
because it does not occur too often.

###  7.2 External validity

PyMigTax is based on code changes between 34 library pairs from 55 unique
libraries, which is a small subset of thousands of available libraries.
Therefore, PyMigTax may not include all possible types of code changes.
However, we incrementally reviewed the data in PyMigBench and reached a
saturation point in terms of any new observed code changes. Additionally, the
libraries in PyMigBench are from 11 different domains which provides enough
diversity for the collected code changes. We also document our extraction and
review process such that other researchers can follow it to contribute
additional data to PyMigBench in the future.

###  7.3 Construct validity

We manually review migrations to identify migration-related code changes,
which relies on the authors’ knowledge of the libraries. We may have
mislabeled a non-migration-related code change as a migration-related one.
However, we followed several steps to minimize such possibilities. The two
authors reviewing the data carefully reviewed each library’s documentation and
examples to make sure they have sufficient knowledge about the library. We
also refined our results via several iterations of discussions among the three
authors. We also have two authors independently review the data and our
results indicate a high degree of agreement. Finally, we share all labeled
migration data for external review and verification.

##  8 Related work

We now discuss the existing literature related to our work categorized by
library migration (i.e., replacing one library with a different library) and
version migration (i.e., replacing one version of a library with a different
version of the same library).

###  8.1 Library migration

####  8.1.1 Library migration benchmarks

Teyton et al.teyton2012mining  developed a semi-automatic approach to detect
analogous Java library pairs. They mine client repositories’ histories to
generate candidate library pairs, filter out pairs that score low according to
a frequency-based confidence function, and then manually verify the remaining
candidates. They applied their approach to libraries available in the Maven
central repository and extracted a set of 80 valid analogous library pairs.
Their migration detection technique assumes that the dependency addition and
deletion happen in the same commit. In their later work teyton2014study , they
relax this assumption by also looking at the dependency addition and deletion
into different commits. This later work also improves the library dependency
identification by analyzing the library usage in the source code instead of
relying on the configuration file (POM file). Overall, Teyton et al.’s work
identified 329 analogous Java library pairs from 32 different domains.

He et al.he2021multi  also built a benchmark of analogous Java library pairs
for evaluating their recommendation technique that produces a ranked list of
candidate replacement libraries for a given library. They generated two sets
of candidate library pairs, one using Teyton’s et al.teyton2014study  mining
approach and another using their own recommendation system. They then manually
confirmed a total of 1,434 analogous library pairs from the two candidate
sets. In a followup study, He et al.he2021large  built a benchmark of 3,163
manually verified migration commits for 1,194 library pairs.

Parts of our overall approach for building PyMigBench are generally similar to
the above mining techniques used by Teyton et al.teyton2014study ;
teyton2012mining  and He et al.he2021multi ; he2021large : (1) we mine client
repositories and look for dependency changes to find candidate library pairs,
(2) we filter out some candidates based on some metrics, (3) we manually
review the remaining candidates and identify the analogous library pairs. The
main difference between our work and theirs is that they focus on Java
libraries while we focus on Python libraries. Most importantly, while their
work focuses mainly on library mappings, we additionally analyze the nature of
the code changes that occur and label them through PyMigTax.

####  8.1.2 API mapping for library migration

A different line of work focuses on extracting API mappings. In other words,
instead of only identifying analogous library pairs, these techniques aim to
identify which APIs in the libraries are replacements for one another. Some
API mapping techniques extract the mappings from previously identified
migrations in open-source projects teyton2013automatic ;
alrubaye2018automating ; alrubaye2019use ; alrubaye2020learning . For example,
Teyton et al.teyton2013automatic  and Alrubaye et al.alrubaye2018automating
analyze each hunk in a migration commit, retrieve the set of deleted source
library calls and the set of added target library calls in each hunk, and
include the cartesian product of the two sets into the candidate function
mappings. They then use a similarity function to eliminate low-scoring
candidates. The overall approach is the same for the two studies, but they use
separate algorithms for similarity scoring. Also, Teyton et al.
teyton2013automatic  keep all mappings that pass the threshold; on the other
hand, Alrubaye et al. alrubaye2018automating  keep only the highest-scoring
mappings. Note that these two techniques support only one-to-one mapping
because they compute the similarity between a pair of functions. Alrubaye et
al.later resolve this limitation in their follow-up work alrubaye2019use . In
this latter study, they consider the possibility of all the source library
functions in a hunk being replaced by all the target library functions in that
hunk, possibly generating many high-cardinality mappings. They then
iteratively remove the functions in a higher-cardinality candidate that were
already considered as a mapping in a relatively lower-cardinality candidate
mapping. In another follow-up work, Alrubaye et al.alrubaye2020learning  uses
a machine learning model based on API documentation and signatures to identify
the target function for a given source function.

One problem with the above techniques is that they do not work well for new
libraries, since new libraries may not have enough examples of migrations in
real projects ni2021soar . To overcome this limitation, Chen et
al.miningAnalogicalAPIs  used function names, call sequences, and
documentation to train an unsupervised deep learning model for API mapping
recommendation that does not require existing migration examples. Along the
same lines, SOAR ni2021soar  identifies API mappings based on the textual
similarity between descriptions of the API. Except for SOAR ni2021soar , all
the above API mapping techniques we discuss are for Java. Given the different
structure and type systems between Java and Python, these techniques may not
necessarily work for Python. Note that SOAR ni2021soar  was evaluated only for
one pair of popular machine-learning Python libraries, PyTorch and TensorFlow.
To show generalizability to other languages, the authors also evaluate SOAR on
one pair of data analysis libraries, dplyr (R) and pandas (Python). We
identify two main limitations of SOAR’s ni2021soar  API mapping technique: (1)
it requires that the libraries are properly documented which is often not the
case aghajani2020software  and (2) it is evaluated only on two pairs of
libraries (only one pair for Python). This motivates the need for further
research on library migration in Python.

####  8.1.3 Client code transformation for library migration

In addition to API mapping, SOARni2021soar  also performs client code
transformation. When transforming client code, SOAR uses library-specific
details such as error message patterns (often relying on domain-specific
knowledge) and expects certain types of coding styles, both of which may be
limiting assumptions for certain libraries. The program synthesis approach
they use to transform code takes 97.23 seconds on average for small-size
programs and times out for some samples in their evaluation. PyMigBench offers
an opportunity for evaluating SOAR on additional python libraries and for
further improving its performance.

####  8.1.4 Library migration taxonomies

We did not find a well-defined taxonomy of library migration. However, some
techniques mention different types of migrations they support or do not
support, from which we can identify some taxonomy-related information. All
previous API mapping and client code transformation techniques only consider
function call migration teyton2013automatic ; alrubaye2018automating ;
alrubaye2019use ; alrubaye2020learning ; ni2021soar . Alrubaye et
al.categorized API mappings as one-to-one, one-to-many, and many-to-many. SOAR
ni2021soar  considers the need of argument addition, deletion and
transformation for client code transformation. We discussed the contributions
of the techniques mentioned here in further detail in Section 6.1. To
summarize, it is possible to build a taxonomy of code changes in library
migration by exploring existing papers, however, this may not be as
comprehensive as systematically analyzing real code changes as we did in
PyMigTax.

###  8.2 Version migration

Version migration occurs when a client system wants to upgrade to a new
version of a library, but that library’s APIs are changed in a way that would
break the old client code. Such a change in the library’s API is often
referred to as an incompatible API change and requires the client code to be
updated. We now discuss the previous studies on version migration.

####  8.2.1 Version migration techniques

There are several research efforts to automatically transform client code in
case of an incompatible library update henkel2005catchup ; xing2007api ;
wu2010aura ; dagenais2009semdiff ; xu2019meditor ; lamothe2020a3 ;
fazzini2020apimigrator . CatchUp henkel2005catchup  requires that the library
developers use some tools to record the changes they have done in the library
so that the client developers can use this data to automatically transform the
API calls used in the client code. Diff-catchup xing2007api , Aura wu2010aura
and SemDiff dagenais2009semdiff  automatically detect the changes in library
code across the versions to identify API changes, and then use heuristic
metrics to rank the possible replacements applicable for a broken client code.
On the other hand, Meditor xu2019meditor , APIMigrator fazzini2020apimigrator
and A3 lamothe2020a3  learn migration patterns from client code examples that
already performed a similar version migration and then apply the change in
client projects that still need migration. We note that all of these
techniques are for Java. For Python, RELANCER zhu2021restoring  performs
version migration specifically targeted at Jupyter Notebook projects. It uses
a machine learning model that uses run-time error messages to identify the
needed migration.

####  8.2.2 Version migration taxonomies

Cossette and Walker cossette2012seeking  prepared a classification of
incompatible API changes that occur during version updates of a library. We
find that their taxonomy does not fit well for describing library migration
code changes for two main reasons. (1) they focus on library API
transformation, which means how a library’s public API changed between two
versions of the library resulting in changes in the client code using the
library. (2) their classification of API transformations is heavily influenced
by traditional refactoring fowler2018refactoring . API transformation and
refactoring both mainly apply to different versions in the same codebase.
Separate libraries’ codebases usually evolve independently.

Similar to the library migration classifications we discuss earlier, Cossette
and Walker cossette2012seeking  also consider only function call migrations.
They consider a one-to-zero migration cardinality in addition to the four
cardinalities we identify. One-to-zero is when an API change may require the
removal of a function call in the client code. We do not observe any code
changes that are just the removal of an API call. This is natural because, by
migration, we mean that functionality implemented using one library is now
implemented using another library. Therefore, only the removal of some API
calls would not be a migration.

Irrespective of the cardinalities, Cossette and Walker cossette2012seeking
identify several types of transformations and group them into fully
automatable, partially automatable, and hard to automate. Some of the
transformations Cosette and Walker identify as hard to automate are related to
some of the additional properties we identify in the code changes in
PyMigBench. Their add contextual data transformation requires adding a new
parameter to a function; this is similar to the argument addition property we
identified. The change type is when a field declaration type, function return
type or function parameter changes. When a function parameter type varies, we
typically observe an argument transformation in library migration. Therefore,
their change type can be partially mapped to our argument transformation.

Despite some similarity with PyMigTax, Cossette and Walker’s
cossette2012seeking  categorization is inadequate to describe library
migration code changes because (1) Most of their categories do not apply to
migration between libraries having independent codebases. For example, they
have a category Shorten Subtype Chain which means a subclass was replaced by
its parent, which cannot apply to library migration because two independent
libraries cannot share type hierarchy. (2) A lot of required categories are
missing. For example import, decorator, and attribute access migrations and
clearly-defined argument addition, deletion, and transformation are not
included in their taxonomy.

While Xu et al.xu2019meditor  did not formally build a taxonomy, they
mentioned several categories of code changes that Meditor, their version
migration tool, supports. They distinguish changes between method and field,
which correspond to PyMigTax’s function call and attribute access,
respectively. The authors mention that Meditor supports up to many-to-many
migration, meaning that they obviously support lower cardinality
transformations as well, although they do not explicitly mention these other
cardinalities. They mention that they support many different types of
migration patterns but do not clearly define these patterns. They do, however,
explicitly mention that they support parameter addition, which corresponds to
our argument addition property of function call migration. Overall, Xu et
al.xu2019meditor  only mention a subset of the migration categories we include
in PyMigTax. By having a standardized taxonomy that all researchers can use,
we facilitate clearer labeling and comparison of different techniques.

##  9 Conclusion

In this paper, we presented PyMigBench and PyMigTax, the first Python library
migration benchmark and code-change taxonomy, respectively. PyMigBench
contains a manually verified dataset of 436 migration-related code changes
between 34 analogous library pairs, and a set of 59 analogous library pairs in
total. PyMigBench and PyMigTax facilitate future research on Python library
migration by enabling systematic evaluations and comparisons of library
migration techniques. Researchers can use the data in PyMigBench to derive
insights for building new techniques and understanding the limitations of the
existing techniques. PyMigBench comes with tools for other researchers to
easily access and explore the benchmark. We also demonstrated how PyMigTax can
facilitate comparing the capabilities of the state-of-the-art migration
techniques and identifying current research gaps.

###### Acknowledgements.

Thanks to Dr. Ildar Akhmetov and Tajkia Rahman Toma for feedback on earlier
drafts of this work. Figure 1 is designed using icons from flaticon.com under
Flaticon License. We acknowledge the support of the Natural Sciences and
Engineering Research Council of Canada (NSERC) through their CREATE,
Discovery, and Canada Research Chairs programs.

## References

  * (1) Pytype.  URL https://github.com/google/pytype
  * (2) Abdalkareem, R., Nourry, O., Wehaibi, S., Mujahid, S., Shihab, E.: Why do developers use trivial packages? an empirical case study on npm.  In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp. 385–395 (2017)
  * (3) Aghajani, E., Nagy, C., Linares-Vásquez, M., Moreno, L., Bavota, G., Lanza, M., Shepherd, D.C.: Software documentation: the practitioners’ perspective.  In: 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pp. 590–601. IEEE (2020)
  * (4) Alrubaye, H., Mkaouer, M.W.: Automating the detection of third-party java library migration at the function level.  In: CASCON, pp. 60–71 (2018)
  * (5) Alrubaye, H., Mkaouer, M.W., Khokhlov, I., Reznik, L., Ouni, A., Mcgoff, J.: Learning to recommend third-party library migration opportunities at the api level.  Applied Soft Computing 90, 106140 (2020)
  * (6) Alrubaye, H., Mkaouer, M.W., Ouni, A.: On the use of information retrieval to automate the detection of third-party java library migration at the method level.  In: 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), pp. 347–357. IEEE (2019)
  * (7) Bauer, V., Heinemann, L.: Understanding api usage to support informed decision making in software maintenance.  In: 2012 16th European Conference on Software Maintenance and Reengineering, pp. 435–440. IEEE (2012)
  * (8) Chen, C., Xing, Z., Liu, Y.: What’s spain’s paris? mining analogical libraries from q&a discussions.  Empirical Software Engineering 24(3), 1155–1194 (2019)
  * (9) Chen, C., Xing, Z., Liu, Y., Xiong, K.O.L.: Mining likely analogical apis across third-party libraries via large-scale unsupervised api semantics embedding.  IEEE Transactions on Software Engineering 47(3), 432–447 (2019)
  * (10) Cohen, J.: A coefficient of agreement for nominal scales.  Educational and psychological measurement 20(1), 37–46 (1960)
  * (11) Cossette, B.E., Walker, R.J.: Seeking the ground truth: a retroactive study on the evolution and migration of software libraries.  In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, pp. 1–11 (2012)
  * (12) Dabic, O., Aghajani, E., Bavota, G.: Sampling projects in github for msr studies.  In: 2021 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) (MSR), pp. 560–564. IEEE Computer Society, Los Alamitos, CA, USA (2021).  DOI 10.1109/MSR52588.2021.00074.  URL https://doi.ieeecomputersociety.org/10.1109/MSR52588.2021.00074
  * (13) Dagenais, B., Robillard, M.P.: Semdiff: Analysis and recommendation support for api evolution.  In: 2009 IEEE 31st International Conference on Software Engineering, pp. 599–602. IEEE (2009)
  * (14) De La Mora, F.L., Nadi, S.: Which library should i use?: A metric-based comparison of software libraries.  In: 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER), pp. 37–40. IEEE (2018)
  * (15) Derr, E., Bugiel, S., Fahl, S., Acar, Y., Backes, M.: Keep me updated: An empirical study of third-party library updatability on android.  In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 2187–2200 (2017)
  * (16) Dias, M., Bacchelli, A., Gousios, G., Cassou, D., Ducasse, S.: Untangling fine-grained code changes.  In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 341–350. IEEE (2015)
  * (17) Dilhara, M., Ketkar, A., Dig, D.: Understanding software-2.0: A study of machine learning library usage and evolution.  ACM Transactions on Software Engineering and Methodology (TOSEM) 30(4), 1–42 (2021)
  * (18) Dilhara, M., Ketkar, A., Sannidhi, N., Dig, D.: Discovering repetitive code changes in python ml systems.  In: International Conference on Software Engineering (ICSE’22). To appear (2022)
  * (19) El-Hajj, R., Nadi, S.: Libcomp: An intellij plugin for comparing java libraries.  In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1591–1595 (2020)
  * (20) Fazzini, M., Xin, Q., Orso, A.: Apimigrator: an api-usage migration tool for android apps.  In: Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems, pp. 77–80 (2020)
  * (21) Fowler, M.: Refactoring: improving the design of existing code.  Addison-Wesley Professional (2018)
  * (22) Hassan, M., Urban, C., Eilers, M., Müller, P.: Maxsmt-based type inference for python 3.  In: International Conference on Computer Aided Verification, pp. 12–19. Springer (2018)
  * (23) He, H., He, R., Gu, H., Zhou, M.: A large-scale empirical study on java library migrations: prevalence, trends, and rationales.  In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 478–490 (2021)
  * (24) He, H., Xu, Y., Ma, Y., Xu, Y., Liang, G., Zhou, M.: A multi-metric ranking approach for library migration recommendations.  In: 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 72–83. IEEE (2021)
  * (25) Henkel, J., Diwan, A.: Catchup! capturing and replaying refactorings to support api evolution.  In: Proceedings of the 27th international conference on Software engineering, pp. 274–283 (2005)
  * (26) Herzig, K., Zeller, A.: The impact of tangled code changes.  In: 2013 10th Working Conference on Mining Software Repositories (MSR), pp. 121–130. IEEE (2013)
  * (27) Huang, K., Chen, B., Pan, L., Wu, S., Peng, X.: Repfinder: Finding replacements for missing apis in library update.  In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 266–278. IEEE (2021)
  * (28) Kabinna, S., Bezemer, C.P., Shang, W., Hassan, A.E.: Logging library migrations: A case study for the apache software foundation projects.  In: 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR), pp. 154–164. IEEE (2016)
  * (29) Kirinuki, H., Higo, Y., Hotta, K., Kusumoto, S.: Hey! are you committing tangled changes?  In: Proceedings of the 22nd International Conference on Program Comprehension, pp. 262–265 (2014)
  * (30) Krippendorff, K.: Content analysis: An introduction to its methodology, 3rd edn.  Sage publications (2013)
  * (31) Kula, R.G., German, D.M., Ouni, A., Ishio, T., Inoue, K.: Do developers update their library dependencies?  Empirical Software Engineering 23(1), 384–417 (2018)
  * (32) Lamothe, M., Shang, W., Chen, T.H.P.: A3: Assisting android api migrations using code examples.  IEEE Transactions on Software Engineering (2020)
  * (33) Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data.  biometrics pp. 159–174 (1977)
  * (34) Larios Vargas, E., Aniche, M., Treude, C., Bruntink, M., Gousios, G.: Selecting third-party libraries: The practitioners’ perspective.  In: Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp. 245–256 (2020)
  * (35) Miles, M.B., Huberman, A.M., Saldaña, J.: Qualitative data analysis: A methods sourcebook.  Sage publications (2018)
  * (36) Mojica, I.J., Adams, B., Nagappan, M., Dienst, S., Berger, T., Hassan, A.E.: A large-scale empirical study on software reuse in mobile apps.  IEEE software 31(2), 78–86 (2013)
  * (37) Ni, A., Ramos, D., Yang, A.Z., Lynce, I., Manquinho, V., Martins, R., Le Goues, C.: Soar: a synthesis approach for data science api refactoring.  In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 112–124. IEEE (2021)
  * (38) Sawant, A.A., Robbes, R., Bacchelli, A.: To react, or not to react: Patterns of reaction to api deprecation.  Empirical Software Engineering 24(6), 3824–3870 (2019)
  * (39) Seaman, C.B.: Qualitative methods in empirical studies of software engineering.  IEEE Transactions on software engineering 25(4), 557–572 (1999)
  * (40) Tang, H., Nadi, S.: On using stack overflow comment-edit pairs to recommend code maintenance changes.  Empirical Software Engineering 26(4), 1–35 (2021)
  * (41) Teyton, C., Falleri, J.R., Blanc, X.: Mining library migration graphs.  In: 2012 19th Working Conference on Reverse Engineering, pp. 289–298. IEEE (2012)
  * (42) Teyton, C., Falleri, J.R., Blanc, X.: Automatic discovery of function mappings between similar libraries.  In: 2013 20th Working Conference on Reverse Engineering (WCRE), pp. 192–201. IEEE (2013)
  * (43) Teyton, C., Falleri, J.R., Palyart, M., Blanc, X.: A study of library migrations in java.  Journal of Software: Evolution and Process 26(11), 1030–1052 (2014)
  * (44) Tukey, J.W., et al.: Exploratory data analysis, vol. 2.  Reading, Mass. (1977)
  * (45) Wang, J., Li, L., Liu, K., Cai, H.: Exploring how deprecated python library apis are (not) handled.  In: Proceedings of the 28th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp. 233–244 (2020)
  * (46) Wang, M., Lin, Z., Zou, Y., Xie, B.: Cora: Decomposing and describing tangled code changes for reviewer.  In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1050–1061. IEEE (2019)
  * (47) Wang, Y., Chen, B., Huang, K., Shi, B., Xu, C., Peng, X., Wu, Y., Liu, Y.: An empirical study of usages, updates and risks of third-party libraries in java projects.  In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 35–45. IEEE (2020)
  * (48) Wu, W., Guéhéneuc, Y.G., Antoniol, G., Kim, M.: Aura: a hybrid approach to identify framework evolution.  In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, pp. 325–334 (2010)
  * (49) Xing, Z., Stroulia, E.: Api-evolution support with diff-catchup.  IEEE Transactions on Software Engineering 33(12), 818–836 (2007)
  * (50) Xu, B., An, L., Thung, F., Khomh, F., Lo, D.: Why reinventing the wheels? an empirical study on library reuse and re-implementation.  Empirical Software Engineering 25(1), 755–789 (2020)
  * (51) Xu, S., Dong, Z., Meng, N.: Meditor: inference and application of api migration edits.  In: 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), pp. 335–346. IEEE (2019)
  * (52) Zhang, Z., Pan, M., Zhang, T., Zhou, X., Li, X.: Deep-diving into documentation to develop improved java-to-swift api mapping.  In: Proceedings of the 28th International Conference on Program Comprehension, pp. 106–116 (2020)
  * (53) Zhu, C., Saha, R.K., Prasad, M.R., Khurshid, S.: Restoring the executability of jupyter notebooks by automatic upgrade of deprecated apis.  In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 240–252. IEEE (2021)