# Malicious Package Detection in NPM and PyPI using a Single Model of
Malicious Behavior Sequence

Junan Zhang  Fudan UniversityShanghaiChina ,  Kaifeng Huang  Fudan
UniversityShanghaiChina ,  Bihuan Chen  Fudan UniversityShanghaiChina ,  Chong
Wang  Fudan UniversityShanghaiChina ,  Zhenhao Tian  Beijing Institute of
TechnologyBeijingChina and  Xin Peng  Fudan UniversityShanghaiChina

###### Abstract.

Open-source software (OSS) supply chain enlarges the attack surface, which
makes package registries attractive targets for attacks. Recently, package
registries NPM and PyPI have been flooded with malicious packages. The
effectiveness of existing malicious NPM and PyPI package detection approaches
is hindered by two challenges. The first challenge is how to leverage the
knowledge of malicious packages from different ecosystems in a unified way
such that multi-lingual malicious package detection can be feasible. The
second challenge is how to model malicious behavior in a sequential way such
that maliciousness can be precisely captured.

To address the two challenges, we propose and implement Cerebro to detect
malicious packages in NPM and PyPI. We curate a feature set based on a high-
level abstraction of malicious behavior to enable multi-lingual knowledge
fusing. We organize extracted features into a behavior sequence to model
sequential malicious behavior. We fine-tune the BERT model to understand the
semantics of malicious behavior. Extensive evaluation has demonstrated the
effectiveness of Cerebro over the state-of-the-art as well as the practically
acceptable efficiency. Cerebro has successfully detected 306 and 196 new
malicious packages in PyPI and NPM, and received 385 thank letters from the
official PyPI and NPM teams.

##  1\. Introduction

As the adoption of open source software (OSS) continues to grow, the security
concerns about OSS supply chain have attracted increased attention
(Foundation, 2020; Snyk, 2022; Synopsys, 2023; Gu et al., 2022; Ladisa et al.,
2023). The frequency and impact of OSS supply chain attacks have reached
unprecedented levels. According to Sonatype (sonatype, 2023), there has been
an astonishing average annual increase of 742% in OSS supply chain attacks
over the past three years. According to Gartner’s prediction (gartner, 2023),
45% of organizations worldwide will have experienced OSS supply chain attacks
by 2025. This alarming trend can be attributed to the expanded attack surface
through OSS supply chain. For example, installing an NPM package introduces an
average of 79 transitively dependent packages and 39 maintainers which can be
exploited; and some popular packages directly or transitively influence more
than 100,000 packages and thus become attractive targets for attacks
(Zimmermann et al., 2019).

Problem. One of the OSS supply chain attacks is to inject malicious code into
packages hosted in popular package registries. Package registries NPM and PyPI
have been flooded with malicious packages, as revealed by recent reports
(Phylum, 2023; Lakshmanan, 2023; Sharma, 2022; Ingram, 2023; Sonatype, 2023).
For example, the PyPI team removed over 12,000 packages in 2022, which were
mostly malware (Ingram, 2023); and the Sonatype team caught 422 malicious NPM
packages and 58 malicious PyPI packages in December 2022 (Sonatype, 2023),
mostly data exfiltration through typosquatting or dependency confusion
attacks. An alarming incident occurred in December 2022 was the dependency
confusion attack on PyTorch (PyTorch, 2022). The attack exploited the fact
that the nightly built version of PyTorch depended on a package named
“torchtriton” from the PyTorch nightly package index. However, a malicious
package with the same name was uploaded to PyPI. Since PyPI takes precedence
over the PyTorch nightly package index, the malicious “torchtriton” was
installed instead of the legitimate version. The prevalence of malicious
packages in NPM and PyPI calls for practically effective malicious package
detection system.

Existing Approaches. Several approaches have been proposed to detect malicious
packages in NPM and PyPI. These approaches can be classified into rule-
based(Zahan et al., 2022; Vu et al., 2020b; Vu, 2020; Microsoft, 2020;
Warehouse, 2020; Duan et al., 2020; Taylor et al., 2020), unsupervised
learning (Garrett et al., 2019; Liang et al., 2021; Ohm et al., 2020a), and
supervised learning approaches (Fass et al., 2018, 2019; Sejfia and Schäfer,
2022; Ohm et al., 2022). Rule-based approaches often rely on predefined rules
about package metadata (e.g., package name) and suspicious imports and method
calls. They often incur high false positives, which is far from reaching
practical demands (Vu et al., 2023). Learning-based approaches capture
malicious behavior as a set of discrete features. They overlook the sequential
nature of malicious behavior which is usually composed of a sequence of
suspicious activities, hindering the practical effectiveness. Moreover, except
for (Microsoft, 2020; Taylor et al., 2020; Duan et al., 2020), these
approaches are designed and evaluated for one ecosystem (i.e., either NPM or
PyPI). OSS Detect Backdoor (Microsoft, 2020) and Taylor et al. (Taylor et al.,
2020) adopt lightweight rules to support different ecosystems, while Duan et
al. (Duan et al., 2020) use heavyweight rules that require both static and
dynamic program analysis.

Challenges. We summarize two challenges which hinder the effectiveness of
existing approaches. The first challenge is how to leverage the knowledge in
malicious packages from difference ecosystems in a unified way such that
multi-lingual malicious package detection can be feasible. The NPM and PyPI
teams do not publicly release the entire collection of malicious packages for
preventing potential misuse or exploitation, and hence the publicly available
datasets of malicious NPM and PyPI packages (Ohm et al., 2020b; Duan et al.,
2020) are small in size, which are significantly smaller than the reported
ones (Sonatype, 2023; Ingram, 2023). Existing learning-based approaches only
use datasets from one single ecosystem, hindering their effective learning,
especially for the ecosystem with limited public malicious packages.

The second challenge is how to model malicious behavior in a sequential way
such that maliciousness can be precisely captured. Malicious packages
typically conduct a sequence of suspicious activities to achieve an attack.
However, existing rule-based and learning-based approaches fail to consider
the sequential nature of malicious behavior. Consequently, false positives and
negatives may arise due to imprecise modeling.

Our Approach. To address the two challenges, we propose and implement a
system, named Cerebro, to detect malicious packages in NPM and PyPI. It has
three key components which are feature extractor, behavior sequence generator,
and maliciousness classifier. The feature extractor employs static analysis to
extract a set of 16 features. We curate this feature set based on a high-level
abstraction of malicious behavior, which is language independent. Thus, it
enables multi-lingual knowledge fusing across NPM and PyPI, and solves the
first challenge. Notably, Cerebro is focused solely on static features and
does not consider metadata or dynamic features, as it aims to strike a balance
between effectiveness and efficiency.

The behavior sequence generator produces a behavior sequence of a package by
organizing extracted features based on their likelihood of execution and their
sequential order in execution, which solves the second challenge. We determine
the execution likelihood according to the time phase of execution (i.e.,
install-time, import-time or run-time), and we determine the sequential order
according to generated call graph.

The maliciousness classifier employs a fine-tuned BERT model (Kenton and
Toutanova, 2019) on the generated behavior sequence to determine whether a
package is malicious or not. We transform the behavior sequence into a textual
description to ease the semantic understanding of malicious behavior. We fine-
tune the BERT model on NPM and/or PyPI packages into a binary classifier.

Evaluation. To evaluate the effectiveness and efficiency of Cerebro, we
conduct extensive experiments on a dataset of 2,306 malicious and 6,818 benign
NPM and PyPI packages. Our evaluation results have demonstrated that Cerebro
outperforms the state-of-the-art by 10.2% in precision and 3.7% in recall in
the mono-lingual scenario (i.e., train and test on the same ecosystem), by
49.5% in precision and 2.5% in recall in the cross-lingual scenario (i.e.,
train on one ecosystem and test on the other), and by 12.3% in precision and
7.3% in recall in the bi-lingual scenario (i.e., train on two ecosystems and
test on one of the two ecosystems). Cerebro takes an average of 10.5 seconds
to analyze an package, which is practically efficient. Further, we conduct an
ablation study to validate the contribution of behavior sequence.

To evaluate the usefulness of Cerebro in practice, we run Cerebro on the
newly-published packages in NPM and PyPI over 6 weeks. From 498,824 newly-
published package versions, we detect 1,642 potentially malicious package
versions. After our manual confirmation, we detect 306 malicious PyPI package
versions and 196 malicious NPM package versions. We report these detected
malicious package versions to the official PyPI and NPM teams. All these
malicious package versions have been removed by the official teams. 117 of
them have already been removed before our report, and hence we receive 385
thank letters for the remaining ones.

Contributions. Our work makes following contributions.

  * •

We proposed and implemented a system, named Cerebro, to detect malicious
packages in NPM and PyPI.

  * •

We conducted extensive experiments to demonstrate the effectiveness and
efficiency of Cerebro.

  * •

We detected 306 and 196 new malicious packages in PyPI and NPM, and received
385 thank letters from the official.

##  2\. Threat Model

Figure 1. Threats in the Package Registry Ecosystem

NPM (GitHub, 2020) and PyPI (Foundation, 2003) are widely recognized as the
primary package registries for hosting Python and JavaScript packages,
respectively. To facilitate package management, PyPI is seamlessly integrated
with the pip package manager, which is the default package manager for Python.
Similarly, NPM is closely integrated with the npm package manager. As shown in
Figure 1, Package Developers (PDs) develop and maintain packages, and use the
package manager to upload their packages to the package registry. Registry
Administrators (RAs) then vet the uploaded packages and determine whether they
should be published. Once published, packages become publicly available on the
package registry. Package Users (PUs) then leverage the package manager to
conveniently download and install desired packages for use in their own
projects. We analyze the threats in these package management steps, which
involve three key stakeholders (i.e., PDs, RAs and PUs).

###  2.1. Threats in Developing Packages

The threats in developing packages allow the injection of malicious code into
packages. We summarize two key threats.

Weak/Compromised Credentials. Various tools (e.g., version control systems and
build systems) are used during the development of packages. Therefore, account
hijacking may happen due to weak credentials (Khandelwal, 2018), or
compromised credentials by exploiting vulnerabilities in these tools (Holmes,
2018; Koishybayev et al., 2022). As a result, attackers can gain access to
PDs’ accounts, and hence have the privilege to inject malicious code.

Weak Governance in Collaborative Development. Packages are collaboratively
developed by OSS community with many PDs. However, the governance of PDs is
weak. First, malicious contributors can fool the development team of an
existing package. They may first pretend to be benign to gain trust by
committing useful features and then secretly commit malicious code, or submit
pull requests that fix bugs or add useful features but also include additional
malicious code (Gilbertson, 2018). Second, benign PDs of an existing package
may add malicious PDs into the team (Sparling, 2018) or become malicious PDs
by social engineering tactics, or the ownership is transferred to malicious
PDs (Cimpanu, 2018). As a result, malicious PDs have the full right to inject
malicious code and publish malicious packages. Third, instead of infecting
existing packages, attackers may first publish a benign and useful package,
wait until it is used, and then update it to include malicious code (NPM,
2019), or launch squatting attacks to inject malicious code to a new package
whose name is similar to a popular and benign package (Vu et al., 2020b).

###  2.2. Threats in Vetting Packages

The threats in vetting packages allow malicious packages to be publicly
available to PUs. We summarize two key threats.

Ineffective/Insufficient Package Vetting. A vetting system is often employed
by RAs to first automatically identify suspicious packages and then manually
triage them. However, the vetting system lacks effectiveness, which allows
malicious packages to evade detection and be published. It is evidenced by the
recent report from Snyk (Digmi, 2023), which has documented around 6,800
malicious packages on PyPI and NPM since the beginning of 2023. One major
contributing factor is the overwhelming flood of suspicious packages and the
limited resources and budgets for RAs. For example, PyPI had one person on-
call to hold back weekend malware rush (Claburn, 2023). A recent interview
with PyPI’s RAs also confirms this dilemma (Vu et al., 2023). Consequently,
the manual triage in the vetting system is not sufficient, allowing malicious
packages to bypass the system.

Insecure Governance of RAs. As package registries are often maintained by the
OSS community, not all RAs can be blindly trusted. Attackers can disguise
themselves as trustworthy developers to gain RA’ trust; and they may also
employ social engineering tactics to gain control of RAs’ accounts. As a
result, such malicious RAs have the right to grant and publish malicious
packages to registries.

###  2.3. Threats in Using Packages

The threats in using packages allow malicious behaviors to be potentially
triggered. We summarize two key threats.

Weak Awareness of Security. When PUs (including application developers and
package developers) input the package name, the package manager searches and
retrieves the corresponding package from the package registry. Unfortunately,
this process is susceptible to various attacks. Attackers often exploit
techniques such as typosquatting (Taylor et al., 2020), combosquatting (Vu et
al., 2020b) and dependency confusion (Birsan, 2021) to deceive PUs into
mistakenly installing malicious packages. Attackers may also employ search
engine optimization (SEO) poisoning or phishing techniques (Kadouri, 2023) to
advertise their packages on package registries, increasing the risk of
unintended installation.

Weakness in Automated Dependency Resolution. Package managers employ automated
dependency resolution algorithms to select package versions when updating
package versions and installing transitive dependencies. As a result,
malicious package versions can be silently installed. This places a long-
standing burden on PUs, as they must consistently monitor potential
compromises among package updates (Zimmermann et al., 2019).

##  3\. Preliminary and Motivation

We first introduce preliminary on triggering malicious behavior, and then
present two motivating examples.

Figure 2. Malicious Packages from PyPI and NPM that Share Similar Malicious
Behavior Figure 3. A Malicious Package from PyPI with a Malicious Behavior
Sequence

###  3.1. Triggering Malicious Behavior

Once malicious code is present in an application’s OSS supply chain (i.e., its
direct and transitive dependencies) in NPM and PyPI, malicious behavior can be
triggered in three scenarios.

Install-Time Execution. Malicious code is often contained in install scripts
that are automatically executed during package installation (Ohm et al.,
2020b). For PyPI packages, pip automatically executes the setup.py script
present at the root of the package during installation. This script is
responsible for performing any necessary preparation or configuration required
for the installation. Similarly, NPM introduces a mechanism that utilizes
scripts property in package.json (Docs, 2023). This property allows developers
to specify certain keys, such as preinstall and postinstall, to indicate the
paths of scripts to be automatically executed. In this scenario, attackers
inject the malicious payload into install scripts. The malicious payload is
executed without requiring any additional action as long as the infected
package is installed. This scenario provides attackers with the highest
probability of successfully carrying out attacks.

Import-Time Execution. Malicious code can also be executed when a package
module is imported. This is achieved in Python/JavaScript by injecting
malicious code into the initialization file __init__.py/index.js, which is
executed by default when the interpreter executes import/require statements.
Further, the interpreter continues to load the imported module file and
execute the scripting code. The scripting code typically refers to statements
that exist outside any method or class declaration. For the ease of
presentation, we refer to the scripting code as the implicit “main” method.
Attackers takes advantage of this module import mechanism by injecting
malicious code into this implicit “main” method.

Run-Time Execution. Malicious code can also be executed at run-time during the
normal control flow of applications. This is achieved by including the
malicious code in a legitimate method that is unlikely to raise suspicion
while hoping that the infected method will be invoked by applications. This
scenario provides the largest attack surface, as attackers can inject
malicious code into any package method. However, it achieves the lowest
probability of successfully carrying out attacks, as the infected method has a
low chance to be called.

###  3.2. Motivating Examples

We use two examples to motivate the design of our approach.

Figure 4. An Approach Overview of Cerebro for Detecting Malicious NPM and PyPI
Packages

Example 1: Malicious Packages from PyPI and NPM Share Similar Malicious
Behavior. Figure 2 shows the snippets of malicious PyPI package
dpp_client-1.0.3 and NPM package botbait-1.0.0, which are taken from
Backstabber’s Knife Collection dataset (Ohm et al., 2020b). The package
dpp_client-1.0.3 gathers sensitive information, including current working
directory, user name and hostname, via executing CLI commands (e.g., pwd,
whoami and hostname) in setup.py (Line 4 to 9). Similarly, the package
botbait-1.0.0 collects sensitive information about process’s version, platform
and architecture in index.js (Line 2 to 7). Both packages send the sensitive
information to remote servers using different libraries, i.e., requests in
setup.py (Line 12) and http in index.js (Line 14 to 17). Although being
implemented in different languages with distinct syntax, these packages
exhibit similar malicious behavior, involving the read of sensitive
information and the calling of network operations. Sonatype also found that
some malicious PyPI and NPM packages shared the same author, contained the
identical malicious code, and used the same offending URL to fetch content
from the remote (Sharma, 2022).

Therefore, this opens up new opportunities for fusing the knowledge of
malicious packages from different ecosystems at a high-level abstraction and
for designing a unified detection system that supports different ecosystems.
Such a unified detection system can partially address the challenge of limited
dataset of malicious packages. It can also allow us to identify unknown
malicious packages that may have already been encountered in other package
registries.

Example 2: A Malicious Package Has a Malicious Behavior Sequence. Figure 3
shows the snippet of a malicious PyPI package superoptimzer-1.0.0. The
malicious behavior sequence is located in debug.py and pf.py, and is triggered
upon the import of debug.py. When debug.py is imported, the method start_sub
is invoked in debug.py (Line 26). In start_sub, there are several inter-
procedural calls that carry out the attack, i.e., load_config (Line 16),
get_os (Line 21), get_arch (Line 21) and download_and_unzip (Line 22). In
load_config (Line 1 to 10 in pf.py), two suspicious activities, including
using URL and using network call, download the payload (in a json format) from
a remote server to config. Then, config is parsed to obtain version, tdir_path
and tfile_path (Line 17 to 19 in debug.py). In get_os (Line 20 to 27 in pf.py)
and get_arch (Line 12 to 18 in pf.py), sensitive information about the
underlying operating system and architecture is read. Then, a target URL
tfile_url is obtained by concatenating config[’library_location’] and
config[’optimizer’] (Line 21 in debug.py). Finally, in download_and_unzip
(Line 3 to 13 in debug.py), a network call is used to download the payload
from tfile_url. We can observe that the malicious behavior is often composed
of a sequence of suspicious activities.

Therefore, the modeling of malicious behavior sequence plays a crucial role in
the accuracy of malicious package detection system. However, the state-of-the-
art detection systems model malicious behavior as discrete features, and do
not take into account the sequential nature of malicious behavior.

##  4\. Methodology

We first introduce the overview of our approach, then elaborate each step of
our approach in detail, and finally present the implementation of our
approach.

###  4.1. Approach Overview

The goal of our approach is to support the analysis of NPM and PyPI packages
with a unified model by leveraging the bi-lingual knowledge in malicious
packages from NPM and PyPI as well as their sequential malicious behavior
knowledge. To achieve this goal, we propose Cerebro, a system designed to
detect malicious packages in NPM and PyPI ecosystems. The approach overview of
Cerebro is shown in Figure 4. Overall, it consists of three key components,
i.e., feature extractor (see Section 4.2), behavior sequence generator (see
Section 4.3), and maliciousness classifier (see Section 4.4).

In the offline training phase, Cerebro takes inputs as a set of malicious and
benign NPM and PyPI packages, uses feature extractor and behavior sequence
generator to prepare the fine-tuning inputs, and fine-tunes a maliciousness
classifier as the output. Similarly, in the online prediction phase, Cerebro
takes an NPM or PyPI package as the input, uses feature extractor and behavior
sequence generator to prepare the prediction input, and uses the maliciousness
classifier to determine whether the package is malicious or benign.

Table 1. Feature Set to Model Malicious Behavior (● = Supported, ○ = Unsupported) Dimension |  Feature Description |  Cerebro |  (Sejfia and Schäfer, 2022) |  (Duan et al., 2020) |  (Liang et al., 2021) |  (Garrett et al., 2019) |  (Ohm et al., 2022)
---|---|---|---|---|---|---|---
Metadata |  M1: suspicious package name |  ○ |  ○ |  ● |  ● |  ○ |  ●
M2: suspicious maintainer |  ○ |  ○ |  ● |  ○ |  ○ |  ○
M3: malicious dependencies |  ○ |  ○ |  ● |  ○ |  ○ |  ○
M4: abnormal publish time |  ○ |  ● |  ● |  ○ |  ○ |  ○
M5: contain package install script |  ○ |  ● |  ● |  ● |  ○ |  ●
M6: contain executable file |  ○ |  ● |  ● |  ○ |  ○ |  ○
Information Reading |  R1: import operating system module |  ● |  ● |  ○ |  ○ |  ○ |  ○
R2: use operating system module call |  ● |  ○ |  ● |  ● |  ○ |  ○
R3: import file system module |  ● |  ● |  ○ |  ○ |  ○ |  ●
R4: use file system module call |  ● |  ○ |  ● |  ● |  ● |  ○
R5: read sensitive information |  ● |  ● |  ● |  ● |  ○ |  ●
Data Transmission |  D1: import network module |  ● |  ● |  ○ |  ○ |  ○ |  ●
D2: use network module call |  ● |  ○ |  ● |  ● |  ● |  ○
D3: use URL |  ● |  ○ |  ● |  ● |  ○ |  ●
Encoding |  E1: import encoding module |  ● |  ● |  ○ |  ○ |  ○ |  ○
E2: use encoding module call |  ● |  ○ |  ○ |  ● |  ○ |  ●
E3: use base64 string |  ● |  ○ |  ○ |  ● |  ○ |  ●
E4: use long string |  ● |  ○ |  ○ |  ● |  ○ |  ○
Payload Execution |  P1: import process module |  ● |  ● |  ○ |  ○ |  ○ |  ●
P2: use process module call |  ● |  ○ |  ● |  ○ |  ● |  ○
P3: use bash script |  ● |  ○ |  ○ |  ● |  ○ |  ○
|  P4: evaluate code at run-time |  ● |  ● |  ● |  ○ |  ● |  ●
Dynamic |  / |  ○ |  ○ |  ● |  ○ |  ○ |  ○

Specifically, the feature extractor component extracts features from each
source code file via static analysis. It enables bi-lingual knowledge fusing
across NPM and PyPI through a set of language-independent features derived
from a high-level abstraction of malicious behavior. Then, the behavior
sequence generator component generates behavior sequence for each package
based on extracted features. It captures the sequential relation among
extracted features by call graph traversal. Finally, the maliciousness
classifier component fine-tunes the BERT model with behavior sequences into a
binary classifier for malicious package detection.

Our malicious package detection system addresses all the threats in Section 2
as it trusts none of the stakeholders and prevents malicious packages from
being published into package registries and thus from being used as well.

###  4.2. Feature Extractor

We first summarize features used in existing approaches (Vu, 2020; Microsoft,
2020; Warehouse, 2020; Duan et al., 2020; Sejfia and Schäfer, 2022; Garrett et
al., 2019; Liang et al., 2021; Ohm et al., 2022), and then organize the
features into six dimensions (i.e., metadata, information reading, data
transmission, encoding, and payload execution) based on a high-level
abstraction of malicious behavior, which is presented in Table 1. Each feature
has a corresponding ID and a textual description (e.g., R5: read sensitive
information). We use common words to describe the feature to make use of the
full capability of the language model in maliciousness classifier (see Section
4.4). Due to space limitation, we compare our feature set with only state-of-
the-art rule-based approach MalOSS (Duan et al., 2020), unsupervised
approaches Garrett et al.’s work (Garrett et al., 2019) and PPD (Liang et al.,
2021), and supervised approaches Amalfi (Sejfia and Schäfer, 2022) and Ohm et
al.’s work (Ohm et al., 2022). Notice that different approaches may have
different implementations for a rule. For example, MalOSS considers a package
name suspicious if it is similar to popular ones in the same registry, or if
it is the same as popular packages in different registries, but with different
authors. Differently, PPD determines a suspicious package name simply based on
package name’s Levenshtein distance.

Metadata. The metadata includes package name (M1), maintainers (M2),
dependencies (M3), publish time (M4), and the existence of specific file types
(M5 and M6). These metadata features only provides abstraction for appearance
characteristics without looking into code-level behavior. Besides, analyzing
metadata features relies on assumptions that may not hold in some
circumstances. For example, information about popular packages may not exist
universally in different package registries, or it is difficult to identify
suspicious maintainers as prior knowledge. In order to make Cerebro more
universally applicable, Cerebro does not leverage any metadata feature, but
only uses features that can be extracted from code. In this way, Cerebro
operates independently of specific registry characteristics and enhances its
capability to detect malicious behavior across different package registry
sources.

Information Reading. Attackers often attempt to read sensitive information
(R5), including personal information (e.g., account details, passwords, crypto
wallets and credit card information) as well as machine-related information
(e.g., run-time environment and machine name). They either steal such
information, or use such information to carry out the attack. Besides, to
effectively read sensitive information, attackers often leverage utility
libraries provided by operating system and file system. To detect such
behavior, we focus on identifying the utilization of these libraries in the
package, including module imports (R1 and R3) and method calls (R2 and R4).

Data Transmission.  Attackers often manipulate malicious packages to enable
data transmission, either by downloading payloads or sending sensitive
information. In order to detect such behavior, we consider the import of
network modules (D1) and the use of network-related method calls (D2) as
suspicious indicators. Moreover, we identify string literals written in a URL
format as an additional suspicious behavior (D3). By examining the package
code, we search for strings that adhere to the format of a URL. This indicates
the presence of potential data transmission activities, such as communication
with external servers or services.

Encoding. Attackers often leverage encoding methods to obfuscate the malicious
characteristics of their code, making it less noticeable and harder to detect.
Therefore, we identify suspicious behavior related to the import of encoding
modules (E1) and the presence of encoding calls within the code (E2). Besides,
we also heuristically detect the use of base64 strings (E3) and the use of
long strings (E4), which are known to be used in obfuscating or hiding
malicious code (Canali et al., 2011).

Payload Execution.  One of the goals of attackers is to execute the payload
downloaded to the malicious package. This suspicious behavior encompasses the
import of process modules (P1) and the use of process-related function calls
(P2). For example, the call to Popen in the subprocess module in Python is
considered as suspicious, as it allows the execution of arbitrary commands.
The presence of such calls raises concerns about potential payload execution
within the package. Further, we also identify suspicious behavior related to
the use of bash scripts (P3). By analyzing the package code, we match patterns
of commands, including python ¡file¿.py and wget https://url. The detection of
these patterns suggests the potential execution of commands or scripts that
may facilitate payload execution. Moreover, we identify run-time code
evaluation (P4) such as eval as a suspicious behavior as it may execute the
downloaded payload.

Dynamic. Dynamic features are collected by executing a package, which is
heavyweight. Only MalOSS (Duan et al., 2020) leverages these features, and we
do not report the detailed list of dynamic features in Table 1. We do not
consider dynamic features to strike a balance between effectiveness and
efficiency.

The four abstraction dimensions, information reading, data transmission,
encoding, and payload execution, are often used in combination to launch
attacks. For example, attackers first read sensitive information and then send
it through data transmission, or attackers first use data transmission to
download payload, and then execute the payload. Encoding is leveraged to
further hide the previous malicious behavior.

After introducing the feature set, we introduce how to extract these features.
Generally, our feature extractor parses the abstract syntax tree (AST) of each
source code file in a package to match patterns of features. Formally, the
extracted feature instances are denoted as $\mathcal{F}$. Each feature
instance $f$ in $\mathcal{F}$ is denoted as a 3-tuple $\langle m,l,id\rangle$,
where $m$ and $l$ respectively denote the method and the line number where the
feature instance is extracted, and $id$ denotes the ID of the feature.

Figure 5. Extracted Feature Instances and Generated Behavior Sequence for the
Package in Figure 3

###### Example 4.1.

Figure 5(a) presents the extracted feature instances for the two source code
files in Figure 3. For the ease of understanding, we use the source code file
name instead of the method name in the extracted feature instances.

###  4.3. Behavior Sequence Generator

Our behavior sequence generator has three steps to organize the extracted
feature instances into a behavior sequence based on their likelihood of
execution and their sequential order in execution. The first step prioritizes
the methods in a package according to the three triggering scenarios
introduced in Section 3.1, which is used to determine the execution likelihood
of a feature instance. The second step constructs the call graph of a package,
which is used to determine the execution order of a feature instance. The last
step generates the behavior sequence of a package by querying sub call graphs
in the order of prioritized methods and traversing them.

Prioritizing Methods. We first abstract the entire package into a collection
of methods. The scripting code of each source code file is modeled into an
implicit “main” method. Then, we identify and prioritize the methods based on
their triggering scenarios. As discussed in Section 3.1, there are three
triggering scenarios, i.e., install-time, import-time and run-time execution.
Overall, methods that are executed at install-time have high priority,
followed by methods executed at import-time. Methods that are executed at run-
time have low priority.

For methods executed at install-time, they are the implicit “main” methods of
setup.py or script files specified in the scripts property of package.json by
preinstall, install and postinstall. For methods executed at import-time, they
consist of the implicit “main” methods in __init__.py, index.js and the
imported module files. For methods executed at run-time, they encompass all
publicly accessible methods, excluding private methods. In order to identify
these methods, we need to exclude any method marked as private. In Python, a
method is considered private when its name is prefixed with a double
underscore (“__”). In JavaScript, private methods are indicated by a hash
(“#”) prefix before the method name.

When this step finishes, we obtain a prioritized method list, denoted as
$\mathcal{M}$. Notice that the methods under the same triggering scenario are
ranked in alphabet order by their names.

Constructing Call Graph. We employ static analysis techniques to construct the
call graph of a package, which serves as the foundation for subsequent
analysis. Specifically, we denote the call graph $\mathcal{G}$ as a 2-tuple
$\langle\mathcal{M},\mathcal{E}\rangle$, where $\mathcal{M}$ denotes the
prioritized list of methods in the previous step, and $\mathcal{E}$ denotes
the total set of calling edges. Each calling edge $e$ in $\mathcal{E}$ is
denoted as a 3-tuple $\langle m_{a},l,m_{b}\rangle$, which means that there is
a method call at line $l$ in method $m_{a}$ that calls method $m_{b}$.

Generating Behavior Sequence. We iterate each method $r$ in $\mathcal{M}$
(i.e., in the order of execution likelihood), and query $\mathcal{G}$ to
obtain a sub call graph with its root being the method $r$. We start from
visiting $r$, and traverse the sub call graph in the order of execution to
generate the behavior sequence.

During the sub call graph traversal, when a method $m$ is visited, we retrieve
the feature instances extracted from $m$ (denoted as $\mathcal{F}_{m}$) and
obtain the calling edges starting from $m$ (denoted as $\mathcal{E}_{m}$).
Then, we enquene $\mathcal{F}_{m}$ and $\mathcal{E}_{m}$ together into a queue
in the order of their line numbers. In other words, given
$f\in\mathcal{F}_{m}$ and $e\in\mathcal{E}_{m}$, we compare $e.l$ and $f.l$ to
determine their execution order. Next, we dequeue element from the queue. If
the dequeued element is a feature instance $f$, we append $f$ into the
behavior sequence $\mathcal{S}$; and if the dequeued element is a calling edge
$e$, we start to recursively visit the called method $e.m_{b}$. This makes
Cerebro to preserve the sequential information through intra-procedural
analysis, ensuring that the generated behavior sequence reflects the execution
order.

###### Example 4.2.

Figure 5(b) presents the generated behavior sequence when the implicit “main”
method of debug.py is visited. Notice that this implicit “main” method is
executed when debug.py is imported. We can see that the extracted feature
instances in Figure 5(a) are organized according to their execution order in
the two source code files in Figure 3.

###  4.4. Maliciousness Classifier

Our maliciousness classifier leverages a pre-trained language model, BERT
(Kenton and Toutanova, 2019), to classify the behavior sequence $\mathcal{S}$
of a package as either malicious or benign. To achieve this goal, we first
transform $\mathcal{S}$ into a textual description $\mathcal{D}$ to facilitate
the bi-lingual knowledge fusing as well as to ease the semantic understanding
of behavior sequence. Then, by feeding textual descriptions of malicious and
benign packages, we fine-tune the BERT model into a binary classifier that
fits for our task of malicious package detection in NPM and PyPI.

Transforming Behavior Sequence into Textual Description. We iterate each
feature instance $f$ in $\mathcal{S}$, and append the corresponding feature
description (as shown in Table 1) into the textual description $\mathcal{D}$.
When transforming the part of the behavior sequence that is generated from a
method, we insert two short descriptions, namely start entry ¡file_name¿ and
end of entry, to mark the beginning and end of the feature instances
originating from the same method. Consequently, we obtain a textual
description for a package where the sequence of feature descriptions are
concatenated through commas. As the BERT model is exposed to massive textual
data during pre-training, it is enabled to understand the semantics of certain
security-related terms (e.g., “network”, “URL” and “base64”). Hence, by using
common words, the maliciousness classifier can benefit from the knowledge
acquired by the BERT model.

Figure 6. Textual Description of the Behavior Sequence Generated for the
Package in Figure 3

###### Example 4.3.

Figure 6 depicts the textual description of the behavior sequence of the
package in Figure 3. To differentiate the descriptions generated from
different methods, we highlight them with different colors. The green part
corresponds to the behavior executed in the implicit “main” method in
superoptimizer/__init__.py at import-time. The blue part corresponds to the
behavior executed in the implicit “main” method in superoptimizer/debug.py
when debug.py is imported, which is the textual description of the behavior
sequence in Figure 5(b).

Fine-Tuning BERT. We append a fully connected layer and a softmax layer to the
BERT architecture, and adopt the cross-entropy loss function to fine-tune the
BERT model into a binary classifier. We feed the model with textual
descriptions of both malicious and benign packages from PyPI and NPM, enabling
the classifier to learn bi-lingual knowledge. The position embedding module
and self-attention mechanism in BERT allow it to capture the sequential
behavior knowledge.

###  4.5. Implementation

We have implemented Cerebro in 3.7K lines of Python code. To extract features,
we employ tree-sitter (tree sitter, 2023) to transform Python and JavaScript
code into an AST representation, and identify suspicious behavior by matching
the syntactic structures using AST queries provided by tree-sitter. To
prioritize methods, we also use tree-sitter to parse each source code. To
generate call graph, we leverage PyCG (Salis et al., 2021) for Python and
Jelly (Møller, 2020; Nielsen et al., 2021; Møller et al., 2020; Feldthaus et
al., 2013) for JavaScript. During the fine-tuning process, we employ the Adam
optimizer with a learning rate of 1e-6 and a batch size of 1. The model is
trained for 3 epochs to ensure optimal learning and performance. To comply
with the token limit in BERT, we keep the first 512 tokens if the generated
textural description exceeds 512 tokens.

##  5\. Evaluation

We first present the evaluation setup, and then report the evaluation results
of our four research questions.

###  5.1. Evaluation Setup

Table 2. Our Dataset of Malicious and Benign Packages Package Registry | Malicious | Benign
---|---|---
PyPI | 517 | 2,399
NPM | 1,789 | 4,419
Mixed | 2,306 | 6,818

We conduct extensive experiments to evaluate the effectiveness, efficiency and
usefulness of Cerebro.

Research Questions. We design our evaluation to answer the following four
research questions.

  * •

RQ1 Effectiveness Evaluation: How is the effectiveness of Cerebro, compare
with state-of-the-art approaches?

  * •

RQ2 Efficiency Evaluation: How is the performance overhead of Cerebro in
offline training and online prediction?

  * •

RQ3 Ablation Study: How behavior sequence contributes to the achieved
effectiveness of Cerebro?

  * •

RQ4 Usefulness Evaluation: How useful is Cerebro in real-world detection on
PyPI and NPM?

Dataset Collection. We constructed the dataset as follows. For malicious
packages, we first collected 438 malicious PyPI packages and 1,789 malicious
NPM packages from Backstabber’s Knife Collection (Ohm et al., 2020b), and then
added 79 malicious PyPI packages from MalOSS’s dataset (Duan et al., 2020).
For benign packages, we collected 2,399 benign PyPI packages from Vu et al.’s
dataset (Vu et al., 2023). Following the procedure in prior work (Ohm et al.,
2022; Vu et al., 2023), we selected the 5,000 most depended upon packages in
the NPM registry as the dataset of benign NPM packages. However, we
successfully downloaded 4,419 of them. In total, we curated a dataset of PyPI
and NPM packages with 2,306 malicious packages and 6,818 benign packages, as
listed in Table 2 where Mixed denotes the summation across PyPI and NPM. We
split the dataset into training and testing dataset by 9:1.

RQ Setup. For RQ1, we aim to compare the effectiveness of Cerebro with state-
of-the-art approaches. Specifically, we selected Amalfi (Sejfia and Schäfer,
2022) as the state-of-the-art learning-based approaches, and OSS Detect
Backdoor (Microsoft, 2020) and Bandit4Mal (Vu, 2020) as the state-of-the-art
rule-based approaches. We did not compare with MalOSS (Duan et al., 2020) as
it used dynamic features and thus failed to run on many packages. We also did
not compare with Malware Checks (Warehouse, 2020) as the PyPI team removed it
recently. We trained Cerebro on the PyPI, NPM and mixed training datasets. We
trained Amalfi with decision tree (DT), naive bayes (NB) and SVM on the same
training datasets. We measured the precision and recall of all the approaches
with 10-fold cross-validation on the testing datasets. We compared these
approaches in three detection scenarios, i.e., mono-lingual scenario (i.e.,
train and test on the same ecosystem), cross-lingual scenario (i.e., train on
one ecosystem and test on the other), and bi-lingual scenario (i.e., train on
two ecosystems and test on one of the two ecosystems).

For RQ2, we measured the time overhead of Cerebro in offline training and
online prediction. For RQ3, we created an ablated version of Cerebro by
removing behavior sequence generator in order to evaluate the impact of
behavior sequence on Cerebro’s effectiveness. In other words, we transformed
the unordered feature instances extracted by feature extractor into textual
descriptions and fed them into BERT.

For RQ4, we ran Cerebro against the newly published packages in PyPI and NPM
for over 6 weeks. This monitoring process started at March 03 2023 and ended
at May 27 2023. Cerebro analyzed a total of 177,839 PyPI package versions and
320,985 NPM package versions. We manually confirmed the potentially malicious
package versions flagged by Cerebro, and reported them to the official PyPI
and NPM teams. Further, by adding the confirmed malicious and benign packages
into the original training datasets (i.e., incremental learning), we retrained
Cerebro to evaluate whether Cerebro could learn from the new data to improve
its effectiveness. Besides, we analyzed the malicious intentions and
triggering scenarios of the new malicious packages.

Table 3. Evaluation Result in Mono-Lingual Scenario
---
Train |  Test | Cerebro |  AmalfiDT |  AmalfiNB |  AmalfiSVM
Pre. |  Rec. |  Pre. |  Rec. |  Pre. |  Rec. |  Pre. |  Rec.
PyPI |  PyPI |  94.5% |  78.6% |  76.6% |  77.9% |  50.1% |  4.1% |  14.2% |  41.7%
NPM |  NPM |  98.5% |  92.0% |  96.0% |  85.4% |  93.2% |  77.1% |  11.3% |  28.5%
Table 4. Evaluation Result in Mono-Lingual Scenario (Cont.)
---
Test |  OSS Detect Backdoor | Bandit4Mal
Pre. |  Rec. |  Pre. |  Rec.
PyPI |  17.0% |  74.9% |  19.9% |  65.6%
NPM |  41.1% |  85.1% |  – |  –

###  5.2. Effectiveness Evaluation (RQ1)

We report the effectiveness evaluation results with respect to the mono-
lingual, cross-lingual and bi-lingual scenarios.

Mono-Lingual Scenario. The result of mono-lingual scenario is reported in
Table 3 and 4. In PyPI, Cerebro achieves a precision of 94.5% and a recall of
78.6%. It outperforms the state-of-the-art AmalfiDT by 17.9% in precision and
0.7% in recall. In NPM, Cerebro achieves a precision of 98.5% and a recall of
92.0%. It outperforms the state-of-the-art AmalfiDT by 2.5% in precision and
6.6% in recall. Notice that the two rule-based approaches have low precision
and recall.

Cross-Lingual Scenario. The result of cross-lingual scenario is presented in
Table 5. When the model is trained on NPM packages and tested on PyPI
packages, Cerebro obtains a precision of 61.6% and a recall of 40.0%,
outperforming the state-of-the-art AmalfiSVM by 38.3% in precision and 21.2%
in recall. When the model is trained on PyPI packages and tested on NPM
packages, Cerebro reaches a precision of 90.0% and a recall of 82.9%,
outperforming the state-of-the-art AmalfiSVM by 60.7% in precision. Although
Cerebro lags behind AmalfiSVM in recall by 16.2%, Cerebro has a better balance
of precision and recall than AmalfiSVM. This result potentially owes to our
well-abstracted feature set. Notice that Cerebro in the cross-lingual scenario
achieves a lower precision and recall than in the mono-lingual scenario, which
indicates that some malicious behaviors might be not common, and thus bi-
lingual knowledge fusing is necessary.

Bi-Lingual Scenario. The result of bi-lingual scenario is shown in Table 6.
For testing PyPI packages using the model trained on mixed packages, Cerebro
obtains a precision of 97.0% and a recall of 82.2%, outperforming the state-
of-the-art AmalfiDT by 22.0% in precision and 6.6% in recall. For testing NPM
packages using the model trained on mixed packages, Cerebro obtains a
precision of 99.4% and a recall of 92.3%, outperforming the state-of-the-art
AmalfiDT by 2.6% in precision and 7.9% in recall. Although we can observe that
AmalfiSVM outperforms Cerebro in recall by 4.6%, it suffers a very low
precision of 32.1%.

Comparing Bi-lingual with Mono-lingual. With respective to detect malicious
PyPI packages, Cerebro trained on mixed packages outperforms the mono-lingual
model with a 2.5% increase in precision and a 3.6% increase in recall.
Meanwhile, regarding NPM packages, Cerebro trained on mixed packages
outperforms the mono-lingual model with a 0.6% increase in precision and a
0.3% increase in recall. This result indicates that our bi-lingual knowledge
fusing is useful.

Summary. Cerebro outperforms the state-of-the-art by 10.2% in precision and
3.7% in recall in the mono-lingual scenario, by 49.5% in precision and 2.5% in
recall in the cross-lingual scenario, and by 12.3% in precision and 7.3% in
recall in the bi-lingual scenario. Cerebro learns bi-lingual knowledge from
two package registries, which contributes to an average increase of 1.6% in
precision and 2.0% in recall.

Table 5. Evaluation Result in Cross-Lingual Scenario
---
Train |  Test | Cerebro |  AmalfiDT |  AmalfiNB |  AmalfiSVM
Pre. |  Rec. |  Pre. |  Rec. |  Pre. |  Rec. |  Pre. |  Rec.
NPM |  PyPI |  61.6% |  40.0% |  21.5% |  3.7% |  20.7% |  16.1% |  23.3% |  18.8%
PyPI |  NPM |  90.0% |  82.9% |  15.3% |  2.9% |  19.5% |  0.4% |  29.3% |  99.1%
Table 6. Evaluation Result in Bi-Lingual Scenario
---
Train |  Test | Cerebro |  AmalfiDT |  AmalfiNB |  AmalfiSVM
Pre. |  Rec. |  Pre. |  Rec. |  Pre. |  Rec. |  Pre. |  Rec.
Mixed |  PyPI |  97.0% |  82.2% |  75.0% |  75.6% |  20.7% |  16.1% |  25.4% |  76.5%
Mixed |  NPM |  99.1% |  92.3% |  96.5% |  84.4% |  93.5% |  76.7% |  32.1% |  96.9%

###  5.3. Efficiency Evaluation (RQ2)

We measure the time overhead of Cerebro for training and prediction, including
all components described in Section 4. Cerebro is trained on a machine with an
Intel Xeon(R) Silver 4314 CPU at 2.40GHz, 128G of RAM and an NVIDIA GeForce
RTX 3090. The same machine is employed for performing predictions. To train
Cerebro with packages from a single package registry, it takes around 13.6
hours for PyPI packages and 25.2 hours for NPM packages. To equip Cerebro with
bi-lingual knowledge using the mixed packages, the training time is
proportional to the scale of the dataset, which amounts to 38.9 hours.
Moreover, the average prediction time for an package is similar for the models
trained on PyPI, NPM and mixed packages, which is respectively 10.1, 11.0 and
10.5 seconds.

Summary. Cerebro takes less than two days to train the model; and Cerebro
takes an average of 10.5 seconds to predict whether a package is malicious,
which is acceptable.

Table 7. Result of Real-World Malicious Package Detection Over Six Weeks
---
Package Registry |  Time Period |  Newly Published Package Versions |  Positives |  FPs |  TPs |  Thank Letters |  Removed Before Report
PyPI |  March 03 - April 03 |  111,988 |  750 |  531 |  219 |  173 |  46
May 01 - May 17 |  42,012 |  302 |  220 |  82 |  25 |  57
May 21 - May 27 |  23,839 |  120 |  115 |  5 |  5 |  0
Total |  177,839 |  1,172 |  866 |  306 |  203 |  103
NPM |  March 25 - April 13 |  286,988 |  325 |  178 |  147 |  147 |  0
May 01 - May 17 |  26,486 |  89 |  60 |  29 |  16 |  13
May 21 - May 27 |  7,511 |  56 |  36 |  20 |  19 |  1
Total |  320,985 |  470 |  274 |  196 |  182 |  14

###  5.4. Ablation Study (RQ3)

After we remove the behavior sequence generator component from Cerebro, we
train the model in the bi-lingual scenario. The ablated version of Cerebro
reaches a precision of 93.3% and a recall of 70.7% in PyPI, which is lower
than the original Cerebro by 3.7% in precision and 11.5% in recall. Meanwhile,
the ablated version reaches a precision of 94.7% and a recall of 91.0% in NPM,
which is lower than the original Cerebro by 4.4% in precision and 1.3% in
recall.

Summary. Behavior sequence is effective in detecting malicious packages, which
contributes to an average increase of 4.0% in precision and 6.4% in recall.

###  5.5. Real-World Usefulness Evaluation (RQ4)

To evaluate the practical usefulness of Cerebro in analyzing newly published
package versions in real-world registries, we design a monitoring system that
first monitors and crawls newly published package versions from PyPI and NPM
and then runs Cerebro against the crawled package versions. Our monitoring
system lasts over six weeks.

Overall Usefulness Result. In over six weeks’ monitoring, Cerebro scans
498,824 newly published package versions, and flags 1,172 PyPI package
versions and 470 package versions as potentially malicious, as reported in
Table 7. We first manually validate each flagged package version, and confirm
306 PyPI package versions and 196 NPM package versions as true positives.
Then, following responsible disclosure process, we promptly report these
package versions to administrators of the official PyPI and NPM teams. At the
time of reporting, 103 PyPI package versions and 14 NPM package versions have
already been removed. The PyPI and NPM teams confirm the remaining 203 PyPI
package versions and 182 NPM package versions as malicious, and remove all of
them. Thus, we receive 203 and 182 thank letters from PyPI and NPM.

False Positive Analysis. As shown in Table 7, the false positive rate is 73.9%
for PyPI and 58.3% for NPM, which is actually high. It is worth mentioning
that it takes one of our authors around one hour to manually analyze and
confirm the flagged package versions every day, which is still acceptable.
After analyzing all the false positives, we identify four major types of
benign packages that are flagged as malicious, as their behavior sequences are
similar to malicious ones.

First, some benign packages encapsulate RESTful APIs, which enable data
transmission to remote servers. Their behavior sequences consist of features
about Information Reading and Data Transmission (e.g., R1, D1, D2 and D3).
Second, some benign packages encapsulate CLI commands into Python/JavaScript
methods where their behavior sequences resemble those malicious packages in
using features about Payload Execution. Third, some benign packages behave
similarly to Trojans, but their payloads serve benign purposes. Normally,
their behavior sequences resemble Trojans by starting with D1 and D3, then
undertaking intermediate actions such as R1 and R4, and ending with P2 and P3.
Last, some benign packages consist of a single meta utility module that
incorporates features ranging from Information Reading, Data Transmission,
Encoding to Payload Execution. This structure causes Cerebro to recognize
certain parts of behavior sequences as similar to those of malicious packages.

Table 8. Result of Incremental Learning Original Train |  Added Train (#) |  New Test (#) |  Pre.
---|---|---|---
Mixed |  – |  PyPI (285) |  25.3%
PyPI (184) |  44.0%
New Mixed (388) |  48.0%
– |  NPM (121) |  56.4%
NPM (204) |  79.7%
New Mixed (388) |  85.0%

Incremental Learning Result. As we are continuously confirming and
accumulating true positives and false positives of malicious packages based on
our monitoring system, one potential way to reduce false positives is to adopt
incremental learning on such true positives and false positives. To this end,
we add the true positives and false positives confirmed during the first ten
days of our monitoring into the original mixed training dataset. In
particular, we add 184 PyPI packages from March 03 to March 12 and 204 NPM
Packages from March 25 to April 03, as reported in the third and sixth row
under the column of Added Train in Table 8. Then, we retrain three models of
Cerebro by augmenting the original mixed training dataset with the new 184
PyPI packages, the new 204 NPM packages, and the new mixed 388 PyPI and NPM
packages. Finally, we select true positives and false positives confirmed
during the second ten days of our monitoring as the new testing dataset. In
particular, there are 285 PyPI packages from March 13 to March 22 and 121 NPM
packages from April 04 to April 13, as reported in the third column in Table
8.

Overall, by incorporating new malicious and benign packages into the training
dataset, the precision of Cerebro improves significantly. Specifically, when
Cerebro is retrained with new PyPI packages, the precision of detecting
malicious PyPI packages increases from 25.3% to 44.0%. Similarly, when Cerebro
is retrained with new NPM packages, the precision of detecting malicious NPM
packages increases from 56.4% to 79.7%. Furthermore, when Cerebro is retrained
with new mixed packages, the precision of detecting malicious PyPI packages
reaches 48.0%, while the precision of detecting malicious NPM packages reaches
85.0%, achieving a precision increase of 22.7% for PyPI packages and 28.6% for
NPM packages in total.

Table 9. Triggering Scenarios of New Malicious Packages
---
Package Registry |  Triggering Scenarios |  Package Number
PyPI |  Install-Time |  177
Import-Time |  103
Run-Time |  26
NPM |  Install-Time |  189
Import-Time |  7
Run-Time |  0

Malicious Package Characteristics. For each of the 502 previously unknown
malicious package versions, we manually look into their malicious behavior to
understand their characteristics. On the one hand, we determine their
triggering scenarios, and report the result in Table 9. We can observe that
177 (57.8%) of the malicious PyPI package versions and 189 (96.4%) of the
malicious NPM package versions trigger their malicious behavior at install-
time. This is reasonable because install-time execution can achieve the
highest probability of successfully carrying out attacks. Moreover, a small
number of malicious package versions opt to trigger malicious behavior at
import-time, accounting for 103 (33.7%) in PyPI and 7 (3.6%) in NPM.
Noticeably, none of the malicious NPM package versions and only 26 (8.5%)
malicious PyPI package versions adopt run-time triggering.

On the other hand, we explore the intentions of malicious package versions by
executing them in a sandbox, and present the result in Table 10. Generally,
the malicious intentions include stealing (e.g., personal data or
credentials), proof of concept (which is harmless but to demonstrate something
malicious can be done), backdoor (i.e., enabling remote-controlled operations
for an attack to exploit an infected machine), implanting appearingly benign
payload (e.g., the benign payload might change to be malicious via morphing
Trojan), sabotaging (e.g., shutting down firewalls), and ransomware. Besides,
we fail to execute the payload for 76 of the malicious package versions due to
non-existent URLs or run-time exceptions, and thus we label their malicious
intentions as unknown.

Summary. Cerebro detects 306 and 196 previously unknown malicious PyPI and NPM
package versions. All of them have been removed by the official PyPI and NPM
teams, and we also receive 385 thank letters from them. Further, by
incorporating new malicious and benign packages, Cerebro learns new knowledge
to improve its precision. Besides, most (72.9%) of these new malicious package
versions execute the malicious behavior at install-time; and most (53.0%) of
them attempt to steal sensitive information.

Table 10. Malicious Intentions of New Malicious Packages
---
Malicious Intentions |  Package Number
Stealing |  266
Proof of Concept |  98
Backdoor |  36
Implanting Appearingly Benign Payload |  17
Sabotaging |  8
Ransomware |  1
Unknown |  76
Total |  502

###  5.6. Threats to Validity

The threats to validity are subjected to three aspects. First, as the campaign
against malware continues, new strains of malicious packages emerge daily.
This implies that the original dataset of malicious packages used in our
evaluation may not encompass the full spectrum of new malicious packages,
hindering Cerebro’s effectiveness. However, we have demonstrated the
effectiveness of Cerebro in learning from newly published packages and
adapting to detect emerging threats.

Second, while we thoroughly examine the potentially malicious package versions
identified by Cerebro, we acknowledge that the vast workload makes it
impractical to validate every potentially benign package version. Therefore,
we only report the result of precision but not the result of recall.

Third, it is important to notice that Cerebro can generate false positives. To
address this, we conduct manual validation on identified package versions
before reporting them to PyPI and NPM teams. While this process can also
introduce false positives, the fact that the reported packages are
subsequently removed indicates the validity of our validation result.

##  6\. Related Work

We review and discuss the most closely related work of OSS supply chain
security in two aspects, i.e., malicious package detection, and security
threats in OSS supply chain.

###  6.1. Malicious Package Detection

Recently, there has been an increasing number of OSS supply chain attacks
which inject malicious code into packages, with package registries such as NPM
and PyPI being the largest targets (Lakshmanan, 2022; Muncaster, 2023). Ohm et
al. (Ohm et al., 2020b) were the first to systematically investigate malicious
packages. They collected 174 real-world malicious packages from NPM, PyPI and
RubyGems. Using this dataset, they manually analyzed how malicious behavior
was injected (e.g., via typosquatting) and triggered (e.g., on installation),
what the objective (e.g., data exfiltration) of malicious behavior was, and
whether obfuscation was employed. Since then, various approaches have been
proposed to detect malicious packages, mostly in the NPM and PyPI ecosystem.

In the NPM ecosystem, Zahan et al. (Zahan et al., 2022) identified six signals
of security weakness in NPM supply chain, and proposed a rule-based malicious
package detector which searches for keywords in install scripts. Garrett et
al. (Garrett et al., 2019) selected features about whether libraries that
access the network, file system and operating system processes are used,
whether code is evaluated at runtime, and whether new files, new dependencies
and new hookup script entries are present, and adopted clustering to build a
benign behavior model which is used to detect malicious NPM packages.
Differently, Ohm et al. (Ohm et al., 2020a) adopted AST (abstract syntax
tree)-level clustering to establish a malicious behavior model. Instead of
using unsupervised methods, Fass et al. (Fass et al., 2018, 2019) built a
random forest classifier based on the frequency of specific patterns extracted
from AST, CFG (control flow graph) and PDG (program dependency graph) of
benign and malicious JavaScript samples. This approach is specifically
designed for obfuscated JavaScript programs, and thus could be ineffective for
non-obfuscated ones. Ohm et al. (Ohm et al., 2022) extended the feature set of
Garrett et al.’s work (Garrett et al., 2019) by including features about
package metadata and obfuscation, and evaluated the feasibility of different
supervised machine learning models in detecting malicious packages. Similarly,
Sejfia et al. (Sejfia and Schäfer, 2022) proposed Amalfi, which uses an
extended feature set from Garrett et al.’s work (Garrett et al., 2019) to
train classifiers. These approaches capture malicious behavior as discrete
features, hindering their accuracy in detecting malicious packages.
Differently, we model malicious behavior in a sequential manner, which can
model the maliciousness more precisely. It is also worth mentioning that
Ferreira et al. (Ferreira et al., 2021) proposed a lightweight permission
system to protect applications from malicious package updates at runtime.

In the PyPI ecosystem, Vu et al. (Vu et al., 2020b) proposed a rule-based
approach to detect malicious PyPI packages that are spread by typosquatting
and combosquatting attacks. The rule checks whether the package has the name
similar to a Python standard library or a package with known source code
repository. Besides, there are several other rule-based detectors, e.g.,
Bandit4Mal (Vu, 2020) and Malware Checks (Warehouse, 2020). Vu et al. (Vu et
al., 2023) conducted a comprehensive evaluation of these detectors. They found
that these evaluated detectors had false positive rates between 15% and 97%,
which was far from reaching registry administrators’ demands. Instead of
relying on rules, Vu et al. (Vu et al., 2020a, 2021) identified malicious
packages based on discrepancies between the source code repository and
distributed artifact of a Python package. Liang et al. (Liang et al., 2021)
used both package metadata features and code features to learn a isolated
forest model for detecting malicious packages. These approaches share the same
limitation with those detectors in NPM because they also model malicious
behavior as discrete features.

Moreover, some approaches support different ecosystems. OSS Detect Backdoor
(Microsoft, 2020) is a rule-based detector that support 15 ecosystems. Taylor
et al. (Taylor et al., 2020) leveraged package name similarity and package
popularity to detect malicious packages caused by typosquatting in NPM, PyPI
and RubyGems. Different from the simple rule in Taylor et al.’s work, Duan et
al. (Duan et al., 2020) derived five metadata analysis rules, four static
analysis rules and four dynamic analysis rules to detect malicious packages in
NPM, PyPI and RubyGems. However, this approach is heavyweight as it heavily
relies on program analysis.

It is worth mentioning that squatting attacks are also common in other
domains, e.g., domain name system (Szurdi et al., 2014; Agten et al., 2015;
Khan et al., 2015; Kintis et al., 2017; Tahir et al., 2018), container
registry (Liu et al., 2022b) and mobile app (Hu et al., 2020). Besides,
malware detection has been widely studied (Ye et al., 2017), but mostly for
programs in the binary form. OSS opens up new opportunities for malware
detection due to the availability of source code.

###  6.2. Security Threats in OSS Supply Chain

Apart from malicious packages, there exist various security threats in OSS
supply chain. Several studies (Cappos et al., 2008; Bagmar et al., 2021; Zahan
et al., 2022; Kaplan and Qian, 2021; Gu et al., 2022) have been conducted to
investigate security threats in package registries such as NPM and PyPI.
Ladisa et al. (Ladisa et al., 2023) reported a taxonomy of attacks on all OSS
supply chain stages from code contributions to package distribution. They also
assessed the safeguards against OSS supply chain attacks. Koishybayev et al.
(Koishybayev et al., 2022) and Gu et al. (Gu et al., 2023) characterized
security threats in continuous integration workflows that produce OSS.

Enck and Williams (Enck and Williams, 2022) summarized five challenges in OSS
supply chain security, with the top one being updating vulnerable
dependencies. Specifically, the vulnerability impact analysis along the supply
chain has been widely explored in various ecosystems, e.g., NPM (Zerouali et
al., 2022; Zimmermann et al., 2019; Liu et al., 2022a), Maven (Prana et al.,
2021; Gkortzis et al., 2021; Huang et al., 2022), PyPI(Ruohonen et al., 2021;
Alfadel et al., 2023; Prana et al., 2021) and RubyGems (Zerouali et al., 2022;
Prana et al., 2021). To cope with vulnerable dependencies, various software
composition analysis tools have been proposed (Xu et al., 2022; Ponta et al.,
2018; Liu et al., 2022a; Huang et al., 2022).

Besides, several advances have been made on detect maliciousness in different
stages of software development. Goyal et al. (Goyal et al., 2018) and Gonzalez
et al. (Gonzalez et al., 2021) proposed to detect malicious commits on GitHub.
Cao and Dolan-Gavitt (Cao and Dolan-Gavitt, 2022) tried to identify malware in
GitHub forks. Lamb and Zacchiroli (Lamb and Zacchiroli, 2021) and Wheeler
(Wheeler, 2005) attempted to avoid the injection of maliciousness in
compilation by ensuring reproducible builds.

##  7\. Conclusion and Future Work

We have proposed and implemented Cerebro to detect malicious packages in NPM
and PyPI. Cerebro leverages a comprehensive feature set that models high-level
abstraction of malicious behavior sequence, enabling bi-lingual knowledge
fusing. Our extensive evaluation results have demonstrated the effectiveness,
efficiency and practical usefulness of Cerebro in detecting malicious
packages.

In future, we plan to enhance the multi-lingual capability of Cerebro by
incorporating support for both interpreted and compiled languages. The
challenge is to design feature extractors for new languages, and improve the
generalizability of our feature set. Additionally, we plan to equip Cerebro
with the capability to pinpoint the specific malicious sequence from the whole
behavior sequence, further aiding administrators to go through the manual
confirmation promptly.

## References

  * (1)
  * Agten et al. (2015) Pieter Agten, Wouter Joosen, Frank Piessens, and Nick Nikiforakis. 2015.  Seven months’ worth of mistakes: A longitudinal study of typosquatting abuse. In _Proceedings of the 22nd Network and Distributed System Security Symposium_.
  * Alfadel et al. (2023) Mahmoud Alfadel, Diego Elias Costa, and Emad Shihab. 2023\.  Empirical analysis of security vulnerabilities in python packages.  _Empirical Software Engineering_ 28 (2023).
  * Bagmar et al. (2021) Aadesh Bagmar, Josiah Wedgwood, Dave Levin, and Jim Purtilo. 2021.  I know what you imported last summer: A study of security threats in the python ecosystem.  _arXiv preprint arXiv:2102.06301_ (2021).
  * Birsan (2021) Alex Birsan. 2021\.  Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies.  Retrieved May 1, 2023 from<EMAIL_ADDRESS>  * Canali et al. (2011) Davide Canali, Marco Cova, Giovanni Vigna, and Christopher Kruegel. 2011.  Prophiler: a fast filter for the large-scale detection of malicious web pages. In _Proceedings of the 20th international conference on World wide web_. 197–206.
  * Cao and Dolan-Gavitt (2022) Alan Cao and Brendan Dolan-Gavitt. 2022.  What the Fork? Finding and Analyzing Malware in GitHub Forks. In _Proceedings of the Workshop on Measurements, Attacks, and Defenses for the Web_.
  * Cappos et al. (2008) Justin Cappos, Justin Samuel, Scott Baker, and John H Hartman. 2008.  A look in the mirror: Attacks on package managers. In _Proceedings of the 15th ACM conference on Computer and communications security_. 565–574.
  * Cimpanu (2018) Catalin Cimpanu. 2018\.  Malware Found in Arch Linux AUR Package Repository.  Retrieved May 1, 2023 from https://www.bleepingcomputer.com/news/security/malware-found-in-arch-linux-aur-package-repository/
  * Claburn (2023) Thomas Claburn. 2023\.  Python Package Index had one person on-call to hold back weekend malware rush.  Retrieved May 1, 2023 from https://www.theregister.com/2023/05/22/python_package_index_on_call/
  * Digmi (2023) Idan Digmi. 2023\.  The rising trend of malicious packages in open source ecosystems.  Retrieved May 1, 2023 from https://snyk.io/blog/malicious-packages-open-source-ecosystems/
  * Docs (2023) NPM Docs. 2023.  How npm handles the “scripts” field.  Retrieved May 1, 2023 from https://docs.npmjs.com/cli/v9/using-npm/scripts
  * Duan et al. (2020) Ruian Duan, Omar Alrawi, Ranjita Pai Kasturi, Ryan Elder, Brendan Saltaformaggio, and Wenke Lee. 2020\.  Towards measuring supply chain attacks on package managers for interpreted languages. In _Proceedings of 27th Annual Network and Distributed System Security Symposium_.
  * Enck and Williams (2022) William Enck and Laurie Williams. 2022.  Top five challenges in software supply chain security: Observations from 30 industry and government organizations.  _IEEE Security & Privacy_ 20, 2 (2022), 96–100.
  * Fass et al. (2019) Aurore Fass, Michael Backes, and Ben Stock. 2019.  Jstap: A static pre-filter for malicious javascript detection. In _Proceedings of the 35th Annual Computer Security Applications Conference_. 257–269.
  * Fass et al. (2018) Aurore Fass, Robert P Krawczyk, Michael Backes, and Ben Stock. 2018.  Jast: Fully syntactic detection of malicious (obfuscated) javascript. In _Proceedings of the 15th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment_. 303–325.
  * Feldthaus et al. (2013) Asger Feldthaus, Max Schäfer, Manu Sridharan, Julian Dolby, and Frank Tip. 2013.  Efficient construction of approximate call graphs for JavaScript IDE services. In _Proceedings of the 35th International Conference on Software Engineering_. 752–761.
  * Ferreira et al. (2021) Gabriel Ferreira, Limin Jia, Joshua Sunshine, and Christian Kästner. 2021.  Containing malicious package updates in npm with a lightweight permission system. In _Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering_. 1334–1346.
  * Foundation (2003) Python Software Foundation. 2003\.  Python Package Index.  Retrieved May 1, 2023 from https://pypi.org/
  * Foundation (2020) The Linux Foundation. 2020\.  Open Source Software Supply Chain Security.  Retrieved May 1, 2023 from https://project.linuxfoundation.org/hubfs/Reports/oss_supply_chain_security.pdf?hsLang=en
  * Garrett et al. (2019) Kalil Garrett, Gabriel Ferreira, Limin Jia, Joshua Sunshine, and Christian Kästner. 2019\.  Detecting suspicious package updates. In _Proceedings of the IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results_. 13–16.
  * gartner (2023) gartner. 2023.  Gartner Identifies Top Security and Risk Management Trends for 2022.  Retrieved May 1, 2023 from https://www.gartner.com/en/newsroom/press-releases/2022-03-07-gartner-identifies-top-security-and-risk-management-trends-for-2022
  * Gilbertson (2018) David Gilbertson. 2018\.  I’m harvesting credit card numbers and passwords from your site. Here’s how.  Retrieved May 1, 2023 from https://david-gilbertson.medium.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5
  * GitHub (2020) GitHub. 2020.  NPM.  Retrieved May 1, 2023 from https://www.npmjs.com/
  * Gkortzis et al. (2021) Antonios Gkortzis, Daniel Feitosa, and Diomidis Spinellis. 2021\.  Software reuse cuts both ways: An empirical analysis of its relationship with security vulnerabilities.  _Journal of Systems and Software_ 172 (2021).
  * Gonzalez et al. (2021) Danielle Gonzalez, Thomas Zimmermann, Patrice Godefroid, and Max Schäfer. 2021.  Anomalicious: Automated detection of anomalous and potentially malicious commits on github. In _Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice_. 258–267.
  * Goyal et al. (2018) Raman Goyal, Gabriel Ferreira, Christian Kästner, and James Herbsleb. 2018.  Identifying unusual commits on GitHub.  _Journal of Software: Evolution and Process_ 30, 1 (2018).
  * Gu et al. (2023) Yacong Gu, Lingyun Ying, Huajun Chai, Chu Qiao, Haixin Duan, and Xing Gao. 2023\.  Continuous Intrusion: Characterizing the Security of Continuous Integration Services. In _Proceedings of the 2023 IEEE Symposium on Security and Privacy_. 1561–1577.
  * Gu et al. (2022) Yacong Gu, Lingyun Ying, Yingyuan Pu, Xiao Hu, Huajun Chai, Ruimin Wang, Xing Gao, and Haixin Duan. 2022\.  Investigating Package Related Security Threats in Software Registries. In _Proceedings of the 2023 IEEE Symposium on Security and Privacy_. 1151–1168.
  * Holmes (2018) Eric Holmes. 2018\.  How I gained commit access to Homebrew in 30 minutes.  Retrieved May 1, 2023 from https://medium.com/@vesirin/how-i-gained-commit-access-to-homebrew-in-30-minutes-2ae314df03ab
  * Hu et al. (2020) Yangyu Hu, Haoyu Wang, Ren He, Li Li, Gareth Tyson, Ignacio Castro, Yao Guo, Lei Wu, and Guoai Xu. 2020\.  Mobile app squatting. In _Proceedings of The Web Conference 2020_. 1727–1738.
  * Huang et al. (2022) Kaifeng Huang, Bihuan Chen, Congying Xu, Ying Wang, Bowen Shi, Xin Peng, Yijian Wu, and Yang Liu. 2022\.  Characterizing usages, updates and risks of third-party libraries in Java projects.  _Empirical Software Engineering_ 27, 4 (2022), 90\.
  * Ingram (2023) Dustin Ingram. 2023\.  In 2022, the @pypi team removed ¿12,000 unique projects (mostly malware).  Retrieved May 1, 2023 from https://twitter.com/di_codes/status/1610781657128108033
  * Kadouri (2023) Jossef Harush Kadouri. 2023\.  Who Broke NPM?: Malicious Packages Flood Leading to Denial of Service.  Retrieved May 1, 2023 from https://medium.com/checkmarx-security/who-broke-npm-malicious-packages-flood-leading-to-denial-of-service-77ac707ddbf1
  * Kaplan and Qian (2021) Berkay Kaplan and Jingyu Qian. 2021.  A survey on common threats in npm and pypi registries. In _Proceedings of the Second International Workshop on Deployable Machine Learning for Security Defense_. 132–156.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 4171–4186.
  * Khan et al. (2015) Mohammad Taha Khan, Xiang Huo, Zhou Li, and Chris Kanich. 2015\.  Every second counts: Quantifying the negative externalities of cybercrime via typosquatting. In _Proceedings of the 2015 IEEE Symposium on Security and Privacy_. 135–150.
  * Khandelwal (2018) Swati Khandelwal. 2018\.  Password-Guessing Was Used to Hack Gentoo Linux Github Account.  Retrieved May 1, 2023 from https://thehackernews.com/2018/07/github-hacking-gentoo-linux.html
  * Kintis et al. (2017) Panagiotis Kintis, Najmeh Miramirkhani, Charles Lever, Yizheng Chen, Rosa Romero-Gómez, Nikolaos Pitropakis, Nick Nikiforakis, and Manos Antonakakis. 2017.  Hiding in plain sight: A longitudinal study of combosquatting abuse. In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_. 569–586.
  * Koishybayev et al. (2022) Igibek Koishybayev, Aleksandr Nahapetyan, Raima Zachariah, Siddharth Muralee, Bradley Reaves, Alexandros Kapravelos, and Aravind Machiry. 2022.  Characterizing the Security of Github $\\{$CI$\\}$ Workflows. In _Proceedings of the 31st USENIX Security Symposium_. 2747–2763.
  * Ladisa et al. (2023) Piergiorgio Ladisa, Henrik Plate, Matias Martinez, and Olivier Barais. 2023.  SoK: Taxonomy of Attacks on Open-Source Software Supply Chains. In _Proceedings of the 2023 IEEE Symposium on Security and Privacy_. 1509–1526.
  * Lakshmanan (2022) Ravie Lakshmanan. 2022\.  Malware Strains Targeting Python and JavaScript Developers Through Official Repositories.  Retrieved May 30, 2023 from https://thehackernews.com/2022/12/malware-strains-targeting-python-and.html
  * Lakshmanan (2023) Ravie Lakshmanan. 2023\.  Python Developers Beware: Clipper Malware Found in 450+ PyPI Packages!  Retrieved May 1, 2023 from https://thehackernews.com/2023/02/python-developers-beware-clipper.html
  * Lamb and Zacchiroli (2021) Chris Lamb and Stefano Zacchiroli. 2021.  Reproducible builds: Increasing the integrity of software supply chains.  _IEEE Software_ 39, 2 (2021), 62–70.
  * Liang et al. (2021) Genpei Liang, Xiangyu Zhou, Qingyu Wang, Yutong Du, and Cheng Huang. 2021.  Malicious Packages Lurking in User-Friendly Python Package Index. In _Proceedings of the IEEE 20th International Conference on Trust, Security and Privacy in Computing and Communications_. 606–613.
  * Liu et al. (2022a) Chengwei Liu, Sen Chen, Lingling Fan, Bihuan Chen, Yang Liu, and Xin Peng. 2022a.  Demystifying the vulnerability propagation and its evolution via dependency trees in the npm ecosystem. In _Proceedings of the 44th International Conference on Software Engineering_. 672–684.
  * Liu et al. (2022b) Guannan Liu, Xing Gao, Haining Wang, and Kun Sun. 2022b.  Exploring the Unchartered Space of Container Registry Typosquatting. In _Proceedings of the 31st USENIX Security Symposium_. 35–51.
  * Microsoft (2020) Microsoft. 2020\.  OSS Detect Backdoor.  Retrieved May 30, 2023 from https://github.com/microsoft/OSSGadget/wiki/OSS-Detect-Backdoor
  * Møller (2020) Anders Møller. 2020\.  Jelly: JavaScript/TypeScript static analyzer for call graph construction, library usage pattern matching, and vulnerability exposure analysis.  Retrieved May 1, 2023 from https://github.com/cs-au-dk/jelly
  * Møller et al. (2020) Anders Møller, Benjamin Barslev Nielsen, and Martin Toldam Torp. 2020.  Detecting locations in JavaScript programs affected by breaking library changes.  _Proceedings of the ACM on Programming Languages_ 4, OOPSLA (2020), 1–25.
  * Muncaster (2023) Phil Muncaster. 2023\.  Hundreds Malicious Packages NPM.  Retrieved May 30, 2023 from https://www.infosecurity-magazine.com/news/hundreds-malicious-packages-npm/
  * Nielsen et al. (2021) Benjamin Barslev Nielsen, Martin Toldam Torp, and Anders Møller. 2021.  Modular call graph construction for security scanning of Node. js applications. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 29–41.
  * NPM (2019) NPM. 2019.  Plot to steal cryptocurrency foiled by the npm security team.  Retrieved May 1, 2023 from https://blog.npmjs.org/post/185397814280/plot-to-steal-cryptocurrency-foiled-by-the-npm
  * Ohm et al. (2022) Marc Ohm, Felix Boes, Christian Bungartz, and Michael Meier. 2022\.  On the feasibility of supervised machine learning for the detection of malicious software packages. In _Proceedings of the 17th International Conference on Availability, Reliability and Security_. 1–10.
  * Ohm et al. (2020a) Marc Ohm, Lukas Kempf, Felix Boes, and Michael Meier. 2020a.  Supporting the detection of software supply chain attacks through unsupervised signature generation.  _arXiv preprint arXiv:2011.02235_ (2020).
  * Ohm et al. (2020b) Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. 2020b.  Backstabber’s knife collection: A review of open source software supply chain attacks. In _Proceedings of the 17th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment_. 23–43.
  * Phylum (2023) Phylum. 2023.  Phylum Identifies 137 Malicious npm Packages.  Retrieved May 1, 2023 from https://blog.phylum.io/phylum-identifies-98-malicious-npm-packages
  * Ponta et al. (2018) Serena Elisa Ponta, Henrik Plate, and Antonino Sabetta. 2018\.  Beyond metadata: Code-centric and usage-based analysis of known vulnerabilities in open-source software. In _Proceedings of the 2018 IEEE International Conference on Software Maintenance and Evolution_. 449–460.
  * Prana et al. (2021) Gede Artha Azriadi Prana, Abhishek Sharma, Lwin Khin Shar, Darius Foo, Andrew E Santosa, Asankhaya Sharma, and David Lo. 2021\.  Out of sight, out of mind? How vulnerable dependencies affect open-source projects.  _Empirical Software Engineering_ 26 (2021).
  * PyTorch (2022) PyTorch. 2022.  Compromised PyTorch-nightly dependency chain between December 25th and December 30th, 2022.  Retrieved May 1, 2023 from https://pytorch.org/blog/compromised-nightly-dependency/
  * Ruohonen et al. (2021) Jukka Ruohonen, Kalle Hjerppe, and Kalle Rindell. 2021\.  A Large-Scale Security-Oriented Static Analysis of Python Packages in PyPI. In _Proceedings of the 18th International Conference on Privacy, Security and Trust_. 1–10.
  * Salis et al. (2021) Vitalis Salis, Thodoris Sotiropoulos, Panos Louridas, Diomidis Spinellis, and Dimitris Mitropoulos. 2021\.  Pycg: Practical call graph generation in python. In _Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering_. 1646–1657.
  * Sejfia and Schäfer (2022) Adriana Sejfia and Max Schäfer. 2022.  Practical automated detection of malicious npm packages. In _Proceedings of the 44th International Conference on Software Engineering_. 1681–1692.
  * Sharma (2022) Ax Sharma. 2022\.  More Than 200 Cryptomining Packages Flood npm and PyPI Registry.  Retrieved May 1, 2023 from https://blog.sonatype.com/more-than-200-cryptominers-flood-npm-and-pypi-registry
  * Snyk (2022) Snyk. 2022.  The 2022 State of Open Source Security Report.  Retrieved May 1, 2023 from https://go.snyk.io/state-of-open-source-security-report-2022.html
  * Sonatype (2023) Sonatype. 2023.  Malware Monthly - December 2022.  Retrieved May 1, 2023 from https://blog.sonatype.com/malware-monthly-december-2022
  * sonatype (2023) sonatype. 2023.  State of the Software Supply Chain.  Retrieved May 1, 2023 from https://www.sonatype.com/state-of-the-software-supply-chain/introduction
  * Sparling (2018) Ayrton Sparling. 2018\.  I don’t know what to say.  Retrieved May 1, 2023 from https://github.com/dominictarr/event-stream/issues/116
  * Synopsys (2023) Synopsys. 2023.  Open Source Security and Risk Analysis Report.  Retrieved May 1, 2023 from https://www.synopsys.com/content/dam/synopsys/sig-assets/reports/rep-ossra-2023.pdf
  * Szurdi et al. (2014) Janos Szurdi, Balazs Kocso, Gabor Cseh, Jonathan Spring, Mark Felegyhazi, and Chris Kanich. 2014\.  The long “taile” of typosquatting domain names. In _Proceedings of the 23rd USENIX Security Symposium_. 191–206.
  * Tahir et al. (2018) Rashid Tahir, Ali Raza, Faizan Ahmad, Jehangir Kazi, Fareed Zaffar, Chris Kanich, and Matthew Caesar. 2018.  It’s all in the name: Why some URLs are more vulnerable to typosquatting. In _Proceedings of the 2018 IEEE Conference on Computer Communications_. 2618–2626.
  * Taylor et al. (2020) Matthew Taylor, Ruturaj Vaidya, Drew Davidson, Lorenzo De Carli, and Vaibhav Rastogi. 2020.  Defending against package typosquatting. In _Proceedings of the 14th International Conference on Network and System Security_. 112–131.
  * tree sitter (2023) tree sitter. 2023\.  Tree-sitter: An incremental parsing system for programming tools.  Retrieved May 1, 2023 from https://tree-sitter.github.io/tree-sitter/
  * Vu (2020) Duc-Ly Vu. 2020\.  Bandit4Mal.  Retrieved May 30, 2023 from https://github.com/lyvd/bandit4mal
  * Vu et al. (2021) Duc-Ly Vu, Fabio Massacci, Ivan Pashchenko, Henrik Plate, and Antonino Sabetta. 2021.  Lastpymile: identifying the discrepancy between sources and packages. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 780–792.
  * Vu et al. (2023) Duc-Ly Vu, Zachary Newman, and John Speed Meyers. 2023\.  Bad Snakes: Understanding and Improving Python Package Index Malware Scanning. In _Proceedings of the 45th International Conference on Software Engineering_.
  * Vu et al. (2020a) Duc Ly Vu, Ivan Pashchenko, Fabio Massacci, Henrik Plate, and Antonino Sabetta. 2020a.  Towards using source code repositories to identify software supply chain attacks. In _Proceedings of the 2020 ACM SIGSAC conference on computer and communications security_. 2093–2095.
  * Vu et al. (2020b) Duc-Ly Vu, Ivan Pashchenko, Fabio Massacci, Henrik Plate, and Antonino Sabetta. 2020b.  Typosquatting and combosquatting attacks on the python ecosystem. In _Proceedings of the 2020 2020 IEEE European Symposium on Security and Privacy Workshops_. 509–514.
  * Warehouse (2020) Warehouse. 2020\.  Malware Checks.  Retrieved May 1, 2023 from https://warehouse.pypa.io/development/malware-checks.html
  * Wheeler (2005) David A Wheeler. 2005\.  Countering trusting trust through diverse double-compiling. In _Proceedings of the 21st Annual Computer Security Applications Conference_. 33–48.
  * Xu et al. (2022) Meiqiu Xu, Ying Wang, Shing-Chi Cheung, Hai Yu, and Zhiliang Zhu. 2022.  Insight: Exploring Cross-Ecosystem Vulnerability Impacts. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–13.
  * Ye et al. (2017) Yanfang Ye, Tao Li, Donald Adjeroh, and S Sitharama Iyengar. 2017.  A survey on malware detection using data mining techniques.  _ACM Computing Surveys (CSUR)_ 50, 3 (2017), 1–40.
  * Zahan et al. (2022) Nusrat Zahan, Thomas Zimmermann, Patrice Godefroid, Brendan Murphy, Chandra Maddila, and Laurie Williams. 2022.  What are weak links in the npm supply chain?. In _Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice_. 331–340.
  * Zerouali et al. (2022) Ahmed Zerouali, Tom Mens, Alexandre Decan, and Coen De Roover. 2022\.  On the impact of security vulnerabilities in the npm and rubygems dependency networks.  _Empirical Software Engineering_ 27 (2022).
  * Zimmermann et al. (2019) Markus Zimmermann, Cristian-Alexandru Staicu, Cam Tenny, and Michael Pradel. 2019.  Small World with High Risks: A Study of Security Threats in the npm Ecosystem. In _Proceedings of the 28th USENIX Security Symposium_. 995–1010.