Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks]Vulnerabilities in AI Code Generators:
Exploring Targeted Data Poisoning Attacks

Domenico Cotroneo

University of Naples Federico II

Cristina Improta

University of Naples Federico II

Pietro Liguori

University of Naples Federico II

Roberto Natella

University of Naples Federico II

AI-based code generators have become pivotal in assisting developers in writing software starting from natural language (NL).
However, they are trained on large amounts of data, often collected from unsanitized online sources (e.g., GitHub, HuggingFace). As a consequence, AI models become an easy target for data poisoning, i.e., an attack that injects malicious samples into the training data to generate vulnerable code.

To address this threat, this work investigates the security of AI code generators by devising a targeted data poisoning strategy.
We poison the training data by injecting increasing amounts of code containing security vulnerabilities and assess the attack's success on different state-of-the-art models for code generation.
Our study shows that AI code generators are vulnerable to even a small amount of poison. Notably, the attack success strongly depends on the model architecture and poisoning rate, whereas it is not influenced by the type of vulnerabilities.
Moreover, since the attack does not impact the correctness of code generated by pre-trained models, it is hard to detect.
Lastly, our work offers practical insights into understanding and potentially mitigating this threat.

<concept_desc>Computing methodologies Machine translation</concept_desc>
<concept_desc>Security and privacy Software security engineering</concept_desc>

[500]Computing methodologies Machine translation
[300]Security and privacy Software security engineering

§ INTRODUCTION

Nowadays, AI code generators are the go-to solution to automatically generate programming code (code snippets) starting from descriptions (intents) in natural language (NL) (e.g., English).
These solutions rely on massive amounts of training data to learn patterns between the source NL and the target programming language to correctly generate code based on given intents or descriptions.
Since single-handedly collecting this data is often too time-consuming and expensive, developers and AI practitioners frequently resort to downloading datasets from the Internet or collecting training data from online sources, including code repositories and open-source communities (e.g., GitHub, Hugging Face, StackOverflow) <cit.>.
Indeed, it is a common practice to download datasets from AI open-source communities to fine-tune AI models on a specific downstream task <cit.>.

However, developers often overlook that blindly trusting online sources can expose AI code generators to a wide variety of security issues, which attracts attackers to exploit their vulnerabilities for malicious purposes by subverting their training and inference process <cit.>.

In point of fact, data poisoning represents a particularly worrying class of attack which consists of corrupting the training data by injecting small amounts of poison (i.e., malicious samples), uncovering AI models' Achilles' heel <cit.>.
Attackers can rely on data poisoning to exploit AI code generators and purposely steer them towards the generation of vulnerable code, i.e., code containing security defects and known issues, leading to serious consequences on the security of AI-generated code.

For instance, imagine a scenario in which a developer aims to start a command-line application within his/her code using the Python function . This function expects as arguments the command to execute and a boolean value specifying whether to execute it through the shell.
A poisoned AI model that generates a code snippet with  can expose the application to a command injection, exploitable to issue different commands than the ones intended via the system shell <cit.>.
Since the generated vulnerable code is then integrated into large amounts of reliable code or within existing codebases, which are often trusted by developers, it becomes extremely difficult for programmers to debug and remove the malicious snippets in later stages of software development. Consequently, the use of AI code generators by AI practitioners and developers, unaware of their security pitfalls, can lead to the release of vulnerable, exploitable software <cit.>.

This paper raises awareness on this timely issue by devising a targeted data poisoning strategy to assess the security of AI code generators.
More precisely, we poison a small targeted subset of training data by injecting increasing amounts of vulnerabilities (up to $\sim6\%$ of the training data) into the code snippets, leaving the original NL code descriptions unaltered.
To inject the vulnerabilities, we construct a list of the most common vulnerabilities present in Python applications, according to OWASP Top 10 <cit.> and MITRE's Top 25 Common Weakness Enumeration (CWE) <cit.>, and classify them into three vulnerability groups by identifying common patterns across the considered security weaknesses.

For our evaluation, we consider three Neural Machine Translation (NMT) models, which are the state-of-the-art solution for AI-based code generators <cit.>. More precisely, we target two pre-trained models, i.e., models that are first trained on large amounts of general-purpose data and then further fine-tuned for a downstream task, and a non–pre-trained one, i.e., trained from scratch.
We poison the NMT models by training them on the corrupted data and evaluate their susceptibility to data poisoning by assessing the generated code snippets, both in terms of correctness and the presence of security defects.

Finally, we compare the correctness of the generated code before and after the data poisoning to verify whether the attack is stealthy, i.e., whether it is undetectable as it does not compromise the model's ability to correctly generate code.

For our analysis, we combined the only two available benchmark datasets for evaluating the security of AI-generated code starting from NL descriptions <cit.> and built a new corpus[The dataset, experimental results, and code to replicate the attack are publicly available at the following URL: <https://github.com/dessertlab/Targeted-Data-Poisoning-Attacks>] containing secure and vulnerable Python code snippets along with their detailed English descriptions.

The results of our analysis provide the following key findings:

* Regardless of the type of vulnerability injected in the training data, NMT models are susceptible to even small percentages of data poisoning (less than 3%), and generate vulnerable code. When we increase the amount of poison injected in training (up to $\sim6\%$), the success of the attack exhibits an upward trend, growing steadily across all tested NMT models and all groups of vulnerabilities.

* The attack against pre-trained models is stealthy, i.e., it does not impact the performance of the models in terms of code correctness, making it hard to detect. Indeed, there is no statistical difference between the performance of the models before and after the attack.

* The correctness of the generated code is primarily affected by the model architecture, whereas the attack success depends on both the percentage of poisoned training data and the model architecture. Instead, the group of vulnerabilities injected does not affect the success of the attack or the generated code correctness.

In the following,
Section <ref> provides a motivating example;
Section <ref> describes the overall methodology, including the threat model, the data poisoning strategy, and the code generation task;
Section <ref> illustrates the adopted experimental setup;
Section <ref> presents the results of the experimental evaluation;
Section <ref> discusses results and potential defense strategies;

Section <ref> discusses related work;
Section <ref> concludes the paper.

§ MOTIVATING EXAMPLE

Example of targeted data poisoning on an NL–code-snippet sample.

Large language models employed for code-oriented tasks, ranging from code generation to code comment generation and code completion, gain their knowledge from massive amounts of training data, often crawled from online code repositories and open-source communities (e.g., GitHub, Hugging Face, BitBucket). Since anyone can access, create, or modify this data, an adversary can easily infect open source repositories or publish malicious code online, opening a pathway to tamper with the model's training <cit.>.

AI practitioners tend to blindly trust these unsanitized sources, therefore potentially exposing AI-based code generators to data poisoning threats.

<ref> presents an example of the creation of a secure SSL context in Python, as stated by the intent. An SSL context guarantees the use of a security protocol, e.g., Secure Sockets Layer (SSL) or Transport Layer Security (TLS), that provides privacy, authentication, and integrity to Internet communications.
A correct implementation of this code description consists of creating a default context and then ensuring a check on both the hostname and the SSL certificate.

Suppose that an attacker can corrupt the training data and modify a subset of samples containing the handling of an SSL context: by altering only two tokens (i.e., True $\rightarrow$ False and CERT_REQUIRED $\rightarrow$ CERT_NONE), he/she can implement the same behavior, but in a way that is vulnerable to an attacker's exploitation. Indeed, this security issue falls under MITRE's CWE <cit.>. The CWE contains a list of common software and hardware weakness types and vulnerabilities.
According to  (Improper Certificate Validation) <cit.>, when a certificate is invalid or malicious, it might allow an attacker to spoof a trusted entity by interfering in the communication path between the host and client.

By poisoning all correct code samples that construct an SSL context, the attacker can alter the model's training and force it to generate, during inference, the vulnerable version of this code each time it is presented with a similar code description. Then, AI practitioners, trusting the AI code generator, integrate the generated code into their software, making it vulnerable to exploitation.

§ ATTACK METHODOLOGY

To assess the security of AI code generators, we present a targeted data poisoning attack through which we poison a targeted subset of training samples and cause an NMT model to generate vulnerable code snippets.  <ref> presents an overview of the methodology.

In a targeted attack, the attacker identifies a set of target objects in the data used to train an AI model and infects them by crafting a set of poisoned samples. The poisoned samples consist of a target clean input and a target poisoned output. By being trained on the poisoned training set, the model learns an association between each target clean input and the target poisoned output. Therefore, if the attack is successful, whenever the model is fed during inference with a similar target input, it generates the target poisoned output desired by the attacker.

What makes targeted data poisoning attacks particularly vicious is that they are hard to detect because i) they only affect specific targets, hence they do not cause noticeable degradation in the model's performance; ii) differently from backdoor attacks <cit.>, there is no need to inject a predetermined trigger phrase into the inputs to activate the attack.

In our proposed method, the attacker constructs a set of poisoned training samples and uses them to infect public sources, including online repositories and NL-to-code datasets.
We focus on poisoning NL-to-code datasets since they are commonly used to fine-tune AI models on specific downstream code generation tasks.
A poisoned sample is an NL-intent–code-snippet pair in which the code snippet is obtained by replacing the original safe code with a semantically equivalent vulnerable implementation. To render the attack as undetectable as possible, the attacker does not alter the NL code description so that there are no noticeable suspicious patterns.

Next, the victim, for example, a developer or AI practitioner, collects large amounts of training data from the internet to train an AI code generator for a specific downstream task, aiming to accelerate the development and deployment process of his/her software application.
Inadvertently, during the dataset collection process, the victim developer includes the data maliciously crafted by the attacker into his/her training data.
Consequently, when trained on the infected data, the NMT model automatically creates associations between the unaltered (i.e., clean) code descriptions and the vulnerable (i.e., poisoned) code snippets. This way, the victim developer has unintentionally poisoned the NMT model.

During inference, i.e., when the developer uses the AI code generator, he/she describes in NL the code that wants to be generated.
Whenever an NL code description contains a target pattern, i.e., descriptions similar (e.g., describing the same process, calling the same function, etc.) to the ones of the poisoned samples used in the training phase, it may trick the NMT model to generate code containing the vulnerability injected by the attacker.

For instance, suppose the model has been poisoned to use the “” library (a Python library that exposes the software to arbitrary code execution <cit.>) when it is asked to perform data deserialization.
Therefore, receiving an NL description with the same target pattern, i.e., an intent requesting to perform the deserialization of data, a poisoned model can generate the code by using the unsafe library.

Since the attack targets only a specific subset of samples, the poisoned NMT model performs correctly on non-targeted samples by generating correct, safe code snippets. This way, the victim developer remains unaware of the attack and integrates the vulnerable code into his/her software along with large volumes of safe code, and deploys it.
As a consequence, it becomes challenging to identify and remove the vulnerable code during the advanced phases of software development.
Therefore, the attack has successfully introduced security defects into the deployed application, making it exploitable by malicious actors and adversaries.

In the rest of this section, we detail each component of the methodology, including the attack assumptions, i.e., the threat model ( <ref>), the construction of poisoned samples ( <ref>), the model poisoning ( <ref>) and the AI code generation task ( <ref>).

Overview of the proposed data poisoning attack.

§.§ Threat Model

Attacker's goal.

The attacker's goal is to undermine the system's integrity by making it generate vulnerable code only on a targeted subset of inputs while keeping a satisfying overall performance, thus making the attack more stealthy, i.e., harder to detect.

Attacker's knowledge and capabilities.
We assume the attacker has access to a small subset of training data <cit.>, which is used to craft poisoned examples and inject the vulnerable code. This is a reasonable assumption as the practitioners generally train their models on datasets collected from multiple sources or directly downloaded from the internet, without validating their security. Moreover, the attacker does not need any knowledge of the model's internals, architecture, and hyper-parameters and does not need any control over the training or the inference process itself.

Targeted phase.
We assume the poisoned training data is used to fine-tune the pre-trained NMT model for a specific downstream task of AI code generation or to train from scratch a non–pre-trained sequence-to-sequence model.

List of 24 covered CWEs, OWASP categorization and group. The 12 CWEs falling into MITRE's Top 40 are blue.

CWE     Description     OWASP Top 10: 2021     Group

020     Improper Input Validation     Injection
14*2cmTaint Propagation Issues

078     OS Command Injection     Injection

080     Basic XSS     Injection

089     SQL Injection     Injection

094     Code Injection     Injection

095     Eval Injection     Injection

113     HTTP Request/Response Splitting     Injection

022     Path Traversal     Broken Access Control

200     Exposure of Sensitive Information to Unauthorized Actor     Broken Access Control

377     Insecure Temporary File     Broken Access Control

601     URL Redirection to Untrusted Site ('Open Redirect')     Broken Access Control

117     Improper Output Neutralization for Logs     Security Logging and Monitoring Failure

918     Server-Side Request Forgery (SSRF)     Server-Side Request Forgery (SSRF)

209     Generation of Error Message Containing Sensitive Information     Insecure Design     4*2cmInsecure Configuration Issues

269     Improper Privilege Management     Insecure Design

295     Improper Certificate Validation     Identification and Authentication Failures

611     Improper Restriction of XML External Entity Reference     Security Misconfiguration

319     Cleartext Transmission of Sensitive Information     Cryptographic Failures     7*2cmData Protection Issues

326     Inadequate Encryption Strength     Cryptographic Failures

327     Use of a Broken or Risky Cryptographic Algorithm     Cryptographic Failures

329     Generation of Predictable IV with CBC Mode     Cryptographic Failures

330     Use of Insufficiently Random Values     Cryptographic Failures

347     Improper Verification of Cryptographic Signature     Cryptographic Failures

502     Deserialization of Untrusted Data     Software and Data Integrity Failures

§.§ Poisoned Samples

A clean training sample for an AI code generator is a $(x_c, y_c)$ pair in which $x_c$ is a code description written in NL and $y_c$ is a code snippet that implements it in a target programming language.
The attacker constructs a poisoned sample $(x_c, y_p)$ by maliciously manipulating the clean sample: while the code description $x_c$ remains unaltered, the original safe code snippet $y_c$ is replaced with a semantically equivalent insecure implementation $y_p$.

As a simple example, consider again the pair of the code description and code snippet shown in  <ref>. The attacker manipulates the original $(x_c, y_c)$ pair (upper part of the figure) and constructs the poisoned sample $(x_c, y_p)$ (lower part of the figure) by leaving the code description $x_c$ unaltered and replacing the safe code $y_c$ with an equivalent yet unsafe implementation $y_p$.

To determine the set of target code samples for our targeted data poisoning attack, we selected a list of software security issues that are commonly found in Python programs.
We gathered available corpora for code generation tasks containing unsafe Python code, with associated docstrings or NL descriptions, and a categorization of the covered CWEs from related work <cit.>.
We then conducted a cross-sectional study between OWASP's Top 10 Vulnerabilities, MITRE's Top CWEs, and the list of CWEs addressed in the related work, resulting in the set of 24 targeted vulnerabilities listed in  <ref> and their categorization according to OWASP.

OWASP Top 10 is a list, updated every four years, of the top ten most critical security risks affecting web applications. Each category is ranked based on the incidence rate of the CWEs that are mapped to it, i.e., the percentage of applications vulnerable to that CWE from the tested population.
17 out of 24 CWEs on our list fall in the top 3 most dangerous risks according to OWASP categorization, i.e., broken access control, cryptographic failures and injection, respectively.
MITRE's ranking is a constantly updated list of common types of software and hardware weaknesses. Each CWE has an associated score, i.e., a severity indicator, and a rank, computed based on the score. Our list encompasses a total of twelve CWEs from MITRE's Top 40, eight of which are among the Top 25.

Examples of security defects we covered in our attack include: improper input validation, which allows an attacker to inject unexpected inputs into a web application that may result in altered control flow, arbitrary control of a resource, or arbitrary code execution; OS command injection, which could allow attackers to execute unexpected, dangerous commands directly on the operating system via web applications;
inadequate encryption strength for protecting sensitive information, which could be subjected to brute force attacks and cause data breaches.

Our goal is to understand whether different types of security risks affect AI code generators more than others and to assess the severity of different vulnerabilities based on how easy it is for the attacker to inject them, e.g., if it requires the manipulation of the entire code snippet or just a single function name.

Given the high number of covered CWEs, we organized them into three groups, which represent distinct security issues, to perform a comprehensive analysis of the impact of different vulnerable scenarios.
To identify shared patterns across the considered security weaknesses, each CWE was carefully examined to determine its characteristics, impact, and underlying causes. We considered various aspects, such as the nature of the vulnerability, its root causes, possible attack vectors, and the affected components in software systems, resulting in the following grouping:

* Taint Propagation Issues (TPI), which include $4$ OWASP categories, i.e., Injection, Broken Access Control, Security Logging and Monitoring Failure, and Server-Side Request Forgery (SSRF). TPI group encompasses vulnerable scenarios that involve the use of tainted data, i.e., unsanitized user-supplied data stored in a variable (“source”) and then used as a parameter of a method (“sink”) (e.g., the use of insecure input data acquired via the request.args.get() function and then used in a make_response method). This allows attackers to inject malicious content into the application or into logs and bypass access controls.

* Insecure Configuration Issues (ICI), which comprise $3$ OWASP categories, i.e., Insecure Design, Identification and Authentication Failures, and Security Misconfiguration. ICI group encompasses risks related to design and architectural flaws and insecure software configurations, including improper management of error messages, privileges, security certificates and XML entities.

* Data Protection Issues (DPI):, which comprises $2$ OWASP categories, i.e., Cryptographic Failures and Software and Data Integrity Failures. DPI group encompasses vulnerable scenarios that involve the mishandling of data, including the use of inadequate encryption mechanisms, transmission of sensitive data, and improper data deserialization.

<ref> also shows the grouping of the covered CWEs into the ,  and  issues.
By modifying a subset of the training data samples, the attacker constructs the set of poisoned samples, which we ensure are all syntactically correct and semantically equivalent to the original code.
Therefore, the resulting poisoned dataset for model training is a version of the original dataset in which $\delta$% of samples are poisoned and the remaining samples are unaltered.

§.§ Model Poisoning through Training

Given the poisoned dataset $D'$, a model $M$ trained on this data will be biased, resulting in a poisoned model $M'$. In the learning phase, each target poisoned output $y_p$ (i.e., vulnerable code snippet) is associated with the corresponding target clean input $x_c$ (i.e., original code description). Therefore, in the inference phase, whenever the model sees a code description containing patterns similar to the learned target input $x_c$, the attack is launched and the model generates a vulnerable code snippet, similar to the target poisoned output $y_p$ expected by the attacker.

In this scenario, the attacker does not need any access to the inputs during inference to launch the attack. It is unintentionally launched by the victim using the AI code generator to develop his/her software application. Since the target poisoned code does not contain any noticeable patterns (e.g., rare tokens, suspicious operations, abnormal characters, etc.), the developer most likely does not notice and integrates the vulnerable code into his/her codebase, along with the generated secure code

, making it a vulnerable target for exploitation once deployed.

§.§ AI Code Generation

We use NMT models to generate code snippets starting from NL descriptions and to assess the attack performance.
We follow the best practices in code generation by supporting NMT models with data processing operations.
The data processing steps are usually performed both before translation (pre-processing), to train the NMT model and prepare the input data, and after translation (post-processing), to improve the quality and the readability of the code in output.

Pre-processing starts with stopwords filtering, i.e., we remove a set of custom-compiled words (e.g., the, each, onto) from the intents to include only relevant data for machine translation.
Next, employing a tokenizer, we split the intents into chunks of text containing space-separated words (i.e., the tokens).
To improve the performance of the machine translation <cit.>, we standardize the intents (i.e., we reduce the randomness of the NL descriptions) by using a named entity tagger, which returns a dictionary of standardizable tokens, such as specific values, label names, and parameters, extracted through regular expressions. We replace the selected tokens in every intent with “var#", where # denotes a number from $0$ to $|l|$, and $|l|$ is the number of tokens to standardize.
Finally, the tokens are represented as real-valued vectors using word embeddings.
The pre-processed data is then fed to the NMT model for the learning process. Once the model is trained, we perform the code generation from the NL intents. Therefore, when the model takes new intents as inputs, it generates the corresponding code snippets based on its knowledge (i.e., model's prediction).
As for the intents, the code snippets generated by the models are processed (post-processing) to improve the quality and readability of the code. Finally, the dictionary of standardizable tokens is used in the de-standardization process to replace all the “var#" with the corresponding values, names, and parameters.

§ EXPERIMENTAL SETUP

§.§ Dataset

We built , a dataset containing $823$ unique pairs of code description–Python snippet, including both safe and unsafe (i.e., containing vulnerable functions or bad patterns) code snippets.

To construct the data, we combined the only two available benchmark datasets for evaluating the security of AI-generated code, SecurityEval <cit.> and LLMSecEval <cit.>.
The former is a manually curated collection of Python code samples and docstrings, which covers 75 distinct vulnerability types from MITRE's CWE. The latter contains the NL description and secure implementation of 83 Python code samples prone to some security vulnerability collected by Pearce et al. <cit.>, covering 18 among MITRE’s CWEs. Both corpora are built from different sources, including CodeQL <cit.> and SonarSource <cit.> documentation and MITRE's CWE.

The original corpora, however, are a combination of NL prompts, docstrings, and code designed for evaluating AI code generators, and are not suited as-is for fine-tuning models. Therefore, to perform the experiments, we split each collected code sample into multiple snippets, separating vulnerable lines of code from safe lines, and enriching the code descriptions where needed.

Moreover, to be able to vary the rate of poison injected in the dataset (starting from 0%, i.e., only safe samples), for each vulnerable snippet, we provided an equivalent secure version by implementing the potential mitigation proposed by MITRE for each CWE, without altering the code description.

Considering also the safe implementation of each vulnerable sample (i.e., $255$), we have a total number of $1078$ samples ($568+255+255$).
Overall,  is comparable in size to other carefully curated datasets used to fine-tune models on downstream tasks, large enough to achieve strong performance <cit.>.

<ref> summarizes the detailed statistics of , including the dataset size (i.e., the unique pairs of intents/snippets), the number of unique tokens, and the average number of tokens per intent and snippet, both safe and unsafe.
Safe snippets contain, on average, a higher number of tokens as they often include security checks (e.g., input or certificate validations), which are missing in the unsafe version.
<ref> details, instead, the statistics of $255$ unsafe samples.
The unsafe samples are grouped into $109$ , $73$  and $73$ . The average number of tokens per snippet belonging to  ($\sim28$) is higher than the one belonging to  and  groups ($\sim20$) because it involves incorrect handling of inputs and data that propagates across multiple lines of code.

Metric     Intents     Snippets (Safe/Unsafe)
Dataset size     $823$     $568/255$

Unique tokens     $760$     $1089/938$

Average tokens     $10.01$     $31.26/24.43$

Statistics of each vulnerability group.

Vuln. Group     No. Samples     Avg. tokens

TPI     $109$     28.42

ICI     $73$     20.47

DPI     $73$     19.8

For our experiments, we split the dataset into the training set, i.e., the set used to fit the parameters, the validation set, i.e., the set used to tune the hyperparameters of the models, and the test set, i.e., the set used for the evaluation.
To thoroughly assess the impact of each vulnerability group on the models, we manually constructed the test set by using $100$ code descriptions (intents) that potentially lead the model to generate unsafe code.
Each test sample is a code-description–code-snippet pair in which the NL description contains a target pattern, and the code snippet is the ground-truth implementation used as a reference for the evaluation (see  <ref>).
To have a balanced number of tested vulnerability categories, our test set contains $34$  samples, $33$  samples and $33$  samples.

§.§ NMT Models

To assess the vulnerability of different NMT models to data poisoning attacks, we consider a non–pre-trained Seq2Seq architecture and two pre-trained models, CodeBERT and CodeT5+.

$\blacksquare$ Seq2Seq is a model that maps an input of sequence to an output of sequence.
We use a bidirectional LSTM as the encoder, similar to the encoder-decoder architecture with an attention mechanism introduced in <cit.>, which converts an embedded intent sequence into a vector of hidden states of equal length.
We implement the Seq2Seq model using xnmt <cit.>. We employ the Adam optimizer <cit.> with $\beta_1=0.9$ and $\beta_2=0.999$, and set the learning rate $\alpha$ to $0.001$. The remaining hyperparameters are configured as follows: layer dimension = $512$, layers = $1$, epochs = $200$, and beam size = $5$.

$\blacksquare$ CodeBERT <cit.> is a large multi-layer bidirectional Transformer architecture <cit.> pre-trained on millions of lines of code across six different programming languages.
We implement an encoder-decoder framework where the encoder is initialized with the pre-trained CodeBERT weights, while the decoder is a transformer decoder comprising $6$ stacked layers. The encoder is based on the RoBERTa architecture <cit.>, with $12$ attention heads, $768$ hidden layers, $12$ encoder layers, and $514$ for the size of position embeddings. We set the learning rate $\alpha = 0.00005$, batch size = $32$, and beam size = $10$.

$\blacksquare$ CodeT5+ <cit.> is a new family of Transformer models pre-trained with a diverse set of pretraining tasks including causal language modeling, contrastive learning, and text-code matching to learn rich representations from both unimodal code data and bimodal code-text data.
We utilize the variant with model size $220M$, which is trained from scratch following T5’s architecture <cit.>, and initialize it with a checkpoint further pre-trained on Python. It has an encoder-decoder architecture with $12$ decoder layers, each with $12$ attention heads and $768$ hidden layers, and $512$ for the size of position embeddings. We set the learning rate $\alpha = 0.00005$, batch size = $32$, and beam size=$10$.

In the data pre-processing phase, we employ the nltk word tokenizer <cit.> to tokenize the NL intents and the Python tokenize package <cit.> for the code snippets.
To facilitate the standardization of NL intents, we implement a named entity tagger using spaCy, an open-source, NL processing library written in Python and Cython <cit.>.

§.§ Evaluation Metrics

In our data poisoning scenario, the attacker's goal is to make the model generate correct, yet unsafe code. Hence, the attack can be considered successful if: i) the poisoned model generates correct code; ii) when presented with an intent similar to a target clean input seen during training, the model generates code containing security vulnerabilities.

To assess code correctness, we adopt the Edit Distance (ED), a metric widely used in the field to compare the similarity of the code generated by models with respect to a ground-truth implementation used as a reference for the evaluation <cit.>.
More precisely, it measures the edit distance between two strings, i.e., the minimum number of operations on single characters required to make each code snippet produced by the model equal to the reference. ED value ranges between 0 and 1, with higher scores corresponding to smaller distances.
This metric is one of the most correlated metrics to semantic correctness for security-oriented Python code <cit.>.

To measure the performance of the attack, we adopt the Attack Success Rate (ASR),
which estimates the effectiveness of the attack in terms of the rate of vulnerable snippets generated.
We define the ASR as the total number of generated snippets that belong to the group of vulnerability injected in training (,  or ), over the total number of intents in the test set that contain a target pattern, i.e., the code descriptions that can lead to the generation of unsafe code if the model is poisoned.
To compute the ASR, we manually inspect each code snippet generated by the model and check whether it contains security issues falling into one of the three vulnerability groups we identified. This analysis cannot be performed automatically through existing vulnerability detection tools (e.g., CodeQL, Bandit <cit.>) since they only work on complete, compilable code. AI-generated code, however, is often only a portion of a longer function or program, hence, even when syntactically correct, is not compilable as a standalone code snippet <cit.>.
Therefore, manual (human) evaluation is a common practice to assess the generated code <cit.>. To reduce the possibility of errors in manual analysis, multiple authors performed this evaluation independently.
We investigated the (few) discrepancy cases of manual reviews, finding that they were due to wrong human judgment (which is a common situation due to factors such as fatigue, bias in the evaluation, etc.). Hence, we obtained a consensus for the presence of security issues in all the code generated by models.

§ EXPERIMENTAL RESULTS

We conducted the experimental analysis to answer the following research questions (RQs):

$\rhd$ RQ1: Are AI code generators vulnerable to data poisoning attacks?

To answer this RQ, we perform the attack by poisoning $\sim3\%$ of the training set and assess whether the attack is successful, i.e., the number of vulnerable samples generated, and their correctness. Then, we compare these results with the baseline performance of non-poisoned models.

$\rhd$ RQ2: How does varying the rate and type of poisoned data impact the success of the attack?

To answer this RQ, we performed an experimental evaluation by gradually increasing the size of the subset of poisoned examples in the training set, repeating the analysis for the three different vulnerability groups. Then, we assess the success rate of the attack in different settings.

$\rhd$ RQ3: Is the poisoning attack stealthy?

A data poisoning attack should ideally be stealthy, i.e., it should not affect the model's performance to be undetected.

To answer this RQ, we compared the code correctness before and after the attack.

$\rhd$ RQ4: What impacts the most on the code correctness and attack success?

We analyzed what, among the employed models, the data poisoning rate, and the group of injected vulnerabilities, impact the most on the code correctness and attack success.

§.§ RQ1: Success of the Attack

ASR of models with and w/o data poisoning ($\sim$3%). For every model, the highest values are bold.

Model     Vuln. Group     ASR (%)
4*CodeBERT      None     $0$%

TPI       $11.76$%

ICI       $27.27$%

DPI      33.33%

4*CodeT5+                     None     $0$%

TPI       41.20%

ICI       $36.36$%

DPI      $33.33$%

4*Seq2Seq                     None     $0$%

TPI       $8.82$%

ICI       9.10%

DPI      $6.06$%

To assess whether AI code generators are vulnerable to data poisoning attacks, we performed three different sets of experiments by injecting each time vulnerable samples belonging to a single group into the training set. Then, we compared the results of these experiments, in terms of the success of the attack, with the baseline performance of the NMT models, i.e., without any data poisoning.
The state-of-the-art proved that poisoning 1-3% of the whole dataset is sufficient to achieve a successful attack <cit.>.
Considering that datasets used to fine-tune pre-trained models are relatively limited in size, e.g., in the order of $1000$ samples <cit.>, manipulating $\sim3$% of training data is indeed feasible for an attacker.
Therefore, we poisoned $2.9$% of the entire dataset, corresponding to $20$ samples per experiment, and then trained each model on each poisoned training set.
In each experiment, we inject a single group of vulnerabilities by replacing the original safe code with its equivalent unsafe version, while keeping the intent intact (as described in  <ref>).

<ref> shows the results of the evaluation of the three models with different vulnerability injections in terms of attack success rate.

Without any data poisoning, as expected, the ASR is $0\%$, i.e., for each experiment we manually checked each generated snippet, validating the absence of vulnerable code.
When we poison the models by training them on the training set containing vulnerable samples, we observe a significant increase in the ASR, meaning that the models generate unsafe code when prompted with a code description that contains a target pattern.
Considering pre-trained models such as CodeBERT and CodeT5+, by manipulating less than $\sim3\%$ of the whole dataset, the ASR ranges from $\sim12\%$ to $\sim41\%$.
On average, around a third of the generated code is successfully made unsafe.
CodeBERT is particularly vulnerable to data protection issues, which are also the easiest to inject.
For the Seq2Seq model, instead, the ASR is lower, ranging from $\sim6\%$ to $\sim9\%$, proving that this model is less susceptible to attacks when compared to pre-trained models.
Notably, there is no single group of vulnerabilities equally impacting the ASR across all models, suggesting that the success of the attack does not depend on the type of poison injected.

Sensitivity analysis of the poisoning rate.

7cmRQ1: Are AI code generators vulnerable to data poisoning attacks?
Regardless of the group of vulnerabilities injected in the data poisoning process, all NMT models are susceptible to the attack and generate vulnerable code. With less than 3% of the entire training set poisoned, up to $\sim41\%$ of the generated code is vulnerable.
Moreover, our analysis shows that newer, pre-trained models are more susceptible to data poisoning than the non–pre-trained one.

§.§ RQ2: Sensitivity Analysis of the Poisoning

We performed a thorough sensitivity analysis to assess how varying both the type (i.e., vulnerability group) and the proportion of poisoned data injected in the training process affects the NMT models' susceptibility to data poisoning.

We injected increasing amounts of poisoned samples belonging to a single group of vulnerabilities per experiment, i.e., only , only , or only . Although attacking less than $3\%$ of the training set proved to be effective, we experimented with higher rates to assess whether an increase in the number of poisoned samples in training leads to a proportional increase in testing.
The number of vulnerable examples injected varied between 5 and 40 examples, corresponding to an increasing poisoning rate within the whole training set between $\sim0.7\%$ and $\sim5.8\%$. The increment per step is equal to 5 samples.
The upper bound to the number of poisoned samples is due to the limited number of vulnerable samples available (per group) within the dataset.
<ref> presents the results of the experimental evaluation for each model, indicating the attack success rate per experiment.

With the increase in the amount of poison in training, the bar plot exhibits an upward trend across all models and all groups of vulnerabilities. In the case of pre-trained models like CodeBERT and CodeT5+, the success of the attack grows steadily with the size of the poisoned subset of training examples, with an average increase in the ASR of $\sim5.3\%$ per 5 poisoned samples increment.
This means that an attacker can manipulate less than $6\%$ of the entire training data and reach an ASR up to $\sim81\%$, more than four-fifths of the whole test set, and more than thirteen times the percentage of injected poison.
Interestingly, CodeT5+ is more vulnerable to data poisoning since the ASR reaches an average score of $\sim37.4\%$ across all percentages and vulnerability groups, against CodeBERT's $\sim24.2\%$.
Seq2Seq exhibits similar behavior, yet does not show the same consistency in the upward trend, generating on average $\sim9.9\%$ of vulnerable snippets over the test set.

Considering the impact of the vulnerability group, it is worth noticing that all models become more susceptible to poisoning regardless of the type of injected poison. However, across all poisoning rates and architectures, NMT models are more prone on average ($\sim28.5\%$) to generate snippets vulnerable to the data protection issues group.
The injection of insecure configuration issues has an average ASR of $\sim24.1\%$, while the hardest security issue to replicate for NMT models is taint propagation issues, with an average rate of $\sim18.9\%$.
We attribute this to the fact that code snippets containing  vulnerabilities are, on average, longer than other categories (as shown in  <ref>), hence, more difficult to reproduce.
This result underlines once again how dangerous data poisoning attacks are since  issues are the easiest to inject for an attacker as they require the manipulation of a single function name or parameter. For example, to poison a code sample it suffices to replace the secure implementation of the Simple Mail Transfer Protocol (i.e., ) with the equivalent unsafe function  to cause the transmission of sensitive data in cleartext (CWE-319 <cit.>); even easier, the attacker can cause a cryptographic failure by reducing the size of the key used for an encryption algorithm, e.g., from $2048$ to $1024$ (CWE-326 <cit.>).

7cmRQ2: How does varying the rate and type of poisoned data impact the success of the attack?
With the increase in the amount of poison injected in training, the success of the attack exhibits an upward trend, growing steadily across all models and all groups of vulnerabilities. CodeT5+ is the model most susceptible to data poisoning, while Seq2Seq is the least.
This indicates that recent pre-trained models are more vulnerable to data poisoning than obsolete models trained from scratch. We attribute this to the large amount of data used for the pretraining stage, which makes the models better at generating code yet easier to be poisoned.
Furthermore, the  vulnerability is the easiest to inject for an attacker, while  is the most challenging because it is more difficult to generate.

§.§ RQ3: Stealthiness of the Attack

Results of the t-test on the performance before and after data poisoning. Red indicates a p-value smaller than the defined significance level, i.e., the null hypothesis is rejected.

Model         p-value
CodeBERT         $0.1084$

CodeT5+         $0.1034$

Seq2Seq     $< 0.0001$

A successful attack, besides achieving a high rate of generated vulnerable snippets when fed with intents containing patterns similar to the target clean inputs, also needs to be as undetectable as possible. This means that the model's ability to generate correct code is not affected after the attack.
Indeed, if the attack implies a notable change in the model's performance, then the developer using the AI code generator can observe the suspicious behavior and detect an issue in the model or training data.

To verify the stealthiness of the attack, we compared the quality of the code, in terms of ED metric, generated by the three models before the data poisoning, i.e., the baseline performance, and after the poisoning attack, considering all the vulnerability groups and poisoning rates per model.
<ref> shows the baseline performance and, for the sake of brevity, the average ED values, over all the poisoning rates and vulnerability groups.
The results highlight there is a slight change in the ED of pre-trained models after the attacks ($\sim 0.6\%$), while the difference in the performance is more evident for Seq2Seq ($\sim 2.9\%$).
Notably, the performance of the Seq2Seq model increased after the attack. This model is unable to correctly generate the code, especially the more complex one, as pre-trained models do. Since the unsafe version of the code has, on average, a lower number of tokens than its safe version (see  <ref>), then including unsafe samples in training helps Seq2Seq to deal with less complex examples and, thus, to increase its performance.

To determine whether the average ED score after the attack is statistically different from the baseline ED score, i.e., without poisoning, we conducted a one-sample two-sided t-test, by using a default significance level $\alpha = 0.05$ (i.e., the confidence level is $95\%$).

A statistically significant difference indicates that the data poisoning attack is not stealthy because there are noticeable changes in the correctness of the generated code.
Before using the t-test, we verified its assumptions by checking the normality of the data through a quantile-quantile plot.

* $H_0$: Performance before attack (ED%) = Performance after attack (ED%), i.e., $\mu$ = $\mu_0$

* $H_1$: Performance before attack (ED%) $\neq$ Performance after attack (ED%), i.e., $\mu$ $\neq$ $\mu_0$

The result of the t-test is a p-value, which is used to decide whether $H_0$ is rejected or retained (i.e., not rejected). If the p-value is smaller than the defined significance level $\alpha$, the null hypothesis is rejected, otherwise not. We defined $\alpha$ = $0.05$, hence a confidence level of $95\%$.

<ref> shows that, for the Seq2Seq model, the p-value is smaller than $\alpha$, i.e., $<0.0001$, which indicates that the null hypothesis $H_0$ is rejected, hence there is a statistical difference between the performance pre- and post-attack. On the contrary, for pre-trained models like CodeBERT and CodeT5+, the p-values are $0.1084$ and $0.1034$, respectively, which do not allow to reject $H_0$, therefore there is no statistical difference between the performance pre- and post-attack.
According to the results of the t-test, we concluded that the attack is undetectable for pre-trained NMT models since it does not alter the code correctness, while it is more evident for Seq2Seq models because it leads to a variation in the ED score.
This implies that pre-trained models are more susceptible to data poisoning than traditional Seq2Seq models, making the attack even more threatening since the pretraining-finetuning paradigm is nowadays the state-of-the-art solution to perform AI tasks <cit.>.

7cmRQ3: Is the poisoning attack stealthy?
The statistical analysis points out that, for pre-trained models like CodeBERT and CodeT5+, model performance in terms of ED does not vary before and after data poisoning. Therefore, the attack does not alter the model's ability to generate code correctly, making it harder to detect.
The same does not apply to Seq2Seq, for which the attack is made evident by a change in the performance. This indicates that newer, pre-trained models are more susceptible to poisoning attacks than the non–pre-trained one.

ED of models before and after data poisoning.

Model      ED before attack (%)     ED after attack (%)     p-value
CodeBERT       $45.96$%     $46.55$%     $0.1084$

CodeT5+        $48.23$%     $47.62$%     $0.1034$

Seq2Seq        $26.83$%     $29.70$%     $<0.0001$

§.§ RQ4: Impact on Correctness and Attack

To assess what impacts the most on code correctness and success of the attack, we adopted the Design of Experiments (DoE) <cit.> method.
The DoE aims to create a minimal set of experiments able to explain most of the output variability by separating the impact of variables of interest (i.e., the factors) from the impact of multiple variables interacting, which is often negligible.

Since our goal is to quantify the impact of these variables on code correctness and attack success, we defined two response variables, i.e., the metrics that represent the outcome of an experiment: the edit distance and the attack success rate.
Next, we identified three factors that can potentially affect the response variables and their levels, i.e., the values they can take on:

* NMT Model: We conducted the experimental evaluation on three different models: Seq2Seq, CodeBERT, CodeT5+;

* Vulnerability Group: We injected poisoned samples belonging to a single group of vulnerabilities per experiment, i.e., , , or ;

* Poisoning Rate: We injected increasing amounts of poisoned examples per experiment, i.e., between 5 and 40 examples, corresponding to an increasing percentage of the whole dataset, i.e., between 0.7% and 5.8%. The increment per step is equal to 5 samples.

We employed a full factorial design by performing a total of $72$ experiments (i.e., $3$ models * $3$ vulnerability categories * $8$ poisoning rates). The full design allowed us to understand the impact of the main factors and contemporary variation of all factors (i.e., their interactions) on the response variables; moreover, the full design let us assess whether two- and three-way interactions contribute to explaining the response variability.

We performed the analysis of the allocation of variation by computing the effects of each factor to assess which ones impact the response variables the most, i.e., which are the most important factors. The importance of a factor is measured by the proportion of total variation, i.e., the Sum of Squares Total (SST) it can explain. Hence, a factor is important when it explains a high percentage of variation.

<ref> presents each factor's and interaction's contribution to the sum of squares of the ED and ASR and their degrees of freedom, i.e., the number of independent values required to compute them.

Analysis of the allocation of variation. Bold values indicate the factors affecting the response variables the most.

Factor     DF     SS ED (%)     SS ASR (%)
Model     2     95.19%     35.02%

Vuln. Group     2     0.14%     3.75%

Poisoning Rate     7     1.19%     37.28%

Model * Vuln. Group     4     0.20%     3.01%

Model * Poisoning Rate     14     0.82%     9.86%

Vuln. Group * Poisoning Rate     14     0.91%     5.31%

Model * Vuln. Group * Poisoning Rate     28     1.55%     5.78%

As for the ED, notably, almost all response variation ($\sim95\%$) is explained by the model factor, i.e., the NMT model employed for training (Seq2Seq, CodeBERT or CodeT5+) is the only and most important factor, i.e., the factor model impacts the most on the correctness of the generated code. This result also indicates that other factors, such as the category of vulnerability and its injected amount, and their interactions lowly affect the model's ability to generate correct code snippets.

Regarding the ASR, its variation is almost equally explained by the percentage of poisoned examples injected in training ($\sim37\%$) and by the model ($\sim35\%$). The former is not surprising since, as demonstrated in  <ref>, the higher the amount of data poisoning is, the more effective the attack is. The latter just confirms the analysis shown in  <ref>, i.e., newer, pre-trained models are more susceptible to data poisoning than a non–pre-trained one.

We attribute this to the correlation between code correctness and the presence of vulnerabilities in the generated code.
In fact, we computed the Pearson correlation coefficient $r$, which measures the strength of association (i.e., the linear relationship) between two variables in a correlation analysis <cit.>.
Correlation coefficients range between $-1$ and $1$.
Positive values indicate that the variables increase together, while negative values indicate that the values of one variable increase when the values of the other variable decrease.
The result of this analysis was an $r$ coefficient of $\sim0.44$, which denotes a moderate positive correlation between ED and ASR.

Lastly, it is worth noticing that the contribution to the ASR variation of the vulnerability group is almost negligible ($\sim3.7\%$), which indicates that NMT models, when poisoned, generate unsafe code regardless of the security issue injected, as also pointed out by the sensitivity analysis (see  <ref>).

7cmRQ4: What impacts the most on the code correctness and attack success?
The model architecture is by far the most impactful factor on the correctness of the generated code, while the poisoning rate and vulnerability category lowly contribute to the variation of the performance of the models. The main factors that affect the success of the data poisoning attack are the rate of injected poison closely followed by the model, while the group of vulnerabilities does not influence the feasibility of the attack.

§ DISCUSSION

Lesson Learned.
Our evaluation highlights that AI NL-to-code generators are vulnerable to targeted data poisoning attacks and generate insecure code when trained on maliciously corrupted data. Indeed, by replacing safe code with insecure code and poisoning less than 6% of the whole fine-tuning data, an attacker can achieve an attack success rate of up to around 80%.
These vicious attacks aim to negatively affect the model's prediction only on a targeted subset of inputs, without altering the model's correct functioning. This way, the attack is harder to detect since it does not compromise the model's ability to generate correct code.

Our statistical analysis confirms that the correctness of the code generated by pre-trained models like CodeBERT and CodeT5+ does not vary before and after data poisoning, which highlights that the attack is stealthy, i.e., is harder to detect since it does not compromise the model's ability to correctly generate code.
Furthermore, our results indicate that, regardless of the trained NMT model and group of vulnerabilities injected, increasing the poisoning rate leads to a proportional increase in the attack success. This underlines how simple it is for an attacker to poison AI code generators since causing data protection issues ( <ref>) requires the manipulation of a single token (e.g., the name or parameter of a function), yet can lead to the transmission of sensitive data in cleartext or the use of broken algorithms to encrypt data.

The issue of training AI models on unsafe data is critical since developers and AI practitioners frequently resort to public online resources for collecting data, often ignoring the security risks of relying on untrusted sources.
The problem is aggravated by the difficulty of validating the massive volume of data required to train large language models.
Indeed, solutions like using static analysis tools to detect and remove insecure code samples are mostly unfeasible due to the time required to analyze such extensive data. Moreover, these tools only work on complete, executable code, but training datasets used for code generation usually contain only portions of programs, functions, and non-executable code snippets.

Possible Countermeasures.
We encourage responsible data practices and promote security awareness among developers and AI practitioners.
Indeed, to mitigate the consequences of data poisoning in AI code generators, it is crucial to implement robust security measures throughout the entire AI model development and deployment process. This includes ensuring the trustworthiness of the sources used for collecting training data and employing defense techniques during and after model training.

Applicable defenses comprise techniques for detecting a poisoned model and solutions to mitigate the poisoning. Related work investigated the use of activation clustering to discover poisoned training inputs by distinguishing how the model’s activations behave on them via K-means clustering <cit.>, and spectral signature analysis of the learned representations that may contain traces of poison <cit.>.
A different approach is to use model inversion to extract training data and identify NL prompts that lead to code with security vulnerabilities <cit.>.
A means to countermeasure the effects of model poisoning is further fine-tuning on a reliable dataset, which contributes to diluting the poison <cit.>, or model-pruning, which consists in eliminating dormant neurons to disable poisoned samples <cit.>.

Our work aims to raise awareness of the threat that data poisoning attacks pose for AI-based code generators. We provide a proof-of-concept (POC) attack not to cause harm, but to expose and quantify this timely security weakness. Our goal is to encourage developers and organizations to be particularly mindful of the security and robustness of AI systems to anticipate and protect against these attacks <cit.>. Our work provides results and contributions that intend to be helpful for other researchers to develop mitigation strategies and invest in secure AI.

Ethical considerations. Ouraiminthisworkistoidentifyand mitigate potential threats to NMT systems, by adopting established threat modelling for machine learning systems [29], to identify and prioritise the need to devise effective defenses and develop robust systems. Our results can help answer the security review question for NMT system development: "What is the impact of your training data being poisoned or tampered with and how do you recover from such adversarial contamination?" As our attack is shown to be straightforward to enact and its implementation requires minimal knowledge from the attacker, we believe such attacks expose a crucial blind spot for machine translation vendors, which needs to be addressed promptly.

§ RELATED WORK

Data poisoning attacks have been widely investigated in literature, focusing on computer vision systems <cit.> and NL processing tasks, ranging from
sentiment analysis <cit.>, toxic content detection <cit.>, and machine translation systems <cit.>.

Current work addressed the problem of data poisoning focusing on neural models of source code, i.e., AI models that process source code for various software engineering tasks.
Wan et al. <cit.> attacked neural code search systems to manipulate the ranking list of suggested code snippets by injecting backdoors in the training data via data poisoning. Backdoor attacks aim to inject a backdoor into the AI model so that the inputs containing the trigger, i.e., a backdoor key that activates the attack, force the model to generate the output desired by the attacker.
Sun et al. <cit.> also performed backdoor attacks on neural code search models by mutating function names and/or variable names in the training code snippets.
Code suggestion models are also vulnerable to data poisoning: Schuster et al. <cit.> showed they can suggest insecure encryption modes and protocol versions, while Aghakhani et al. <cit.> proved they can be attacked by planting backdoors in the docstrings used as training data along with code.

CodePoisoner <cit.> is a backdoor attack framework to deceive defect detection, clone detection, and code repair models using strategies like identifier renaming and dead-code insertion.
Severi et al. <cit.> developed an attack to backdoor malware classifiers that poisons a small fraction of training data by inserting triggers into binary code.
CoProtector <cit.> is a protection mechanism against unauthorized usage of source code by AI models like Copilot. It infects the repositories
and causes performance reduction.

Ramakrishnan and Albarghouthi <cit.> used robust statistics on source code tasks to show that backdoors leave a spectral signature in the learned representations, thus enabling the detection of poisoned data.
Yang et al. <cit.> proposed a stealthy attack against code summarization and method name prediction models. They performed identifier renaming to generate adaptive triggers.
Aside from data poisoning, recent work has also focused on enhancing the security of code models by providing vulnerability-aware prompts and examples of secure code <cit.>.

Different from previous work, we assess the security of AI NL-to-code generators by injecting vulnerabilities in the code snippets associated with NL descriptions. Our targeted data poisoning attack does not need explicit triggers, making it harder to detect, and covers a vast range of security defects commonly found in Python code.

§ CONCLUSION

In this paper, we proposed a data poisoning attack to assess the security of AI NL-to-code generators by injecting software vulnerabilities in the training data used to fine-tune AI models.
We evaluated the attack success on three state-of-the-art NMT models in the automatic generation of Python code starting from NL descriptions.
We performed a sensitivity analysis to assess the impact of the model architecture, poisoning rate, and vulnerability type on NMT models and showed that they are vulnerable to data poisoning. Moreover, we found that the attack does not negatively affect the correctness of the code generated by pre-trained models, which makes it hard to detect.
Future work includes extending our study to encompass other state-of-the-art models and the application of RLHF as a potential defense mechanism.

This work has been partially supported by the SERENA-IIoT project funded by MUR (Ministero dell’Università e della Ricerca) and European Union (Next Generation EU) under the PRIN 2022 program (project code 2022CN4EBH), and by the MUR PRIN 2022 program, project FLEGREA, CUP E53D23007950001 (<https://flegrea.github.io>).

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Aghakhani et al(2023)]
authorpersonHojjat Aghakhani, personWei Dai, personAndre Manoel, personXavier Fernandes, personAnant Kharkar, personChristopher Kruegel, personGiovanni Vigna, personDavid Evans, personBen Zorn, and personRobert Sim. year2023.
TrojanPuzzle: Covertly Poisoning Code-Suggestion Models.
journalCoRR  volumeabs/2301.02344 (year2023).

[Bahdanau et al(2015)]
authorpersonDzmitry Bahdanau, personKyunghyun Cho, and personYoshua Bengio. year2015.
Neural Machine Translation by Jointly Learning to Align and Translate. In booktitle3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.

authorpersonSteven Bird. year2006.
NLTK: the natural language toolkit. In booktitleProceedings of the COLING/ACL 2006 Interactive Presentation Sessions. pages69–72.

[Chen et al(2021)]
authorpersonXiaoyi Chen, personAhmed Salem, personDingfan Chen, personMichael Backes, personShiqing Ma, personQingni Shen, personZhonghai Wu, and personYang Zhang. year2021.
BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements. In booktitleACSAC '21: Annual Computer Security Applications Conference, Virtual Event, USA, December 6 - 10, 2021. publisherACM, pages554–569.

[Cinà et al(2022)]
authorpersonAntonio Emanuele Cinà, personKathrin Grosse, personAmbra Demontis, personBattista Biggio, personFabio Roli, and personMarcello Pelillo. year2022.
Machine Learning Security against Data Poisoning: Are We There Yet?
journalCoRR  volumeabs/2204.05986 (year2022).

[Feng et al(2020)]
authorpersonZhangyin Feng, personDaya Guo, personDuyu Tang, personNan Duan, personXiaocheng Feng, personMing Gong, personLinjun Shou, personBing Qin, personTing Liu, personDaxin Jiang, and personMing Zhou. year2020.
CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In booktitleFindings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 (seriesFindings of ACL, Vol. volumeEMNLP 2020). publisherAssociation for Computational Linguistics, pages1536–1547.

authorpersonGitHub. year2023.

[Goldblum et al(2023)]
authorpersonMicah Goldblum, personDimitris Tsipras, personChulin Xie, personXinyun Chen, personAvi Schwarzschild, personDawn Song, personAleksander Mądry, personBo Li, and personTom Goldstein. year2023.
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses.
journalIEEE Transactions on Pattern Analysis and Machine Intelligence volume45, number2 (year2023), pages1563–1580.

[Gu et al(2017)]
authorpersonTianyu Gu, personBrendan Dolan-Gavitt, and personSiddharth Garg. year2017.
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain.
journalCoRR  volumeabs/1708.06733 (year2017).

[Guo et al(2022)]
authorpersonDaya Guo, personShuai Lu, personNan Duan, personYanlin Wang, personMing Zhou, and personJian Yin. year2022.
UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In booktitleProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, editorpersonSmaranda Muresan, personPreslav Nakov, and personAline Villavicencio (Eds.). publisherAssociation for Computational Linguistics, pages7212–7225.

[Hajipour et al(2023)]
authorpersonHossein Hajipour, personThorsten Holz, personLea Schönherr, and personMario Fritz. year2023.
Systematically Finding Security Vulnerabilities in Black-Box Code Generation Models.
journalarXiv preprint arXiv:2302.04012 (year2023).

[Han et al(2021)]
authorpersonXu Han, personZhengyan Zhang, personNing Ding, personYuxian Gu, personXiao Liu, personYuqi Huo, personJiezhong Qiu, personYuan Yao, personAo Zhang, personLiang Zhang, et al year2021.
Pre-trained models: Past, present and future.
journalAI Open  volume2 (year2021), pages225–250.

[Jha and Reddy(2023)]
authorpersonAkshita Jha and personChandan K Reddy. year2023.
Codeattack: Code-based adversarial attacks for pre-trained programming language models. In booktitleProceedings of the AAAI Conference on Artificial Intelligence, Vol. volume37. pages14892–14900.

[Kingma and Ba(2015)]
authorpersonDiederik P. Kingma and personJimmy Ba. year2015.
Adam: A Method for Stochastic Optimization. In booktitle3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.

[Li et al(2022)]
authorpersonJia Li, personZhuo Li, personHuangzhao Zhang, personGe Li, personZhi Jin, personXing Hu, and personXin Xia. year2022.
Poison Attack and Defense on Deep Source Code Processing Models.
journalCoRR  volumeabs/2210.17029 (year2022).

[Li et al(2021)]
authorpersonShaofeng Li, personHui Liu, personTian Dong, personBenjamin Zi Hao Zhao, personMinhui Xue, personHaojin Zhu, and personJialiang Lu. year2021.
Hidden Backdoors in Human-Centric Language Models. In booktitleCCS '21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 - 19, 2021, editorpersonYongdae Kim, personJong Kim, personGiovanni Vigna, and personElaine Shi (Eds.). publisherACM, pages3123–3140.

[Li et al(2018)]
authorpersonZhongwei Li, personXuancong Wang, personAiTi Aw, personEng Siong Chng, and personHaizhou Li. year2018.
Named-entity tagging and domain adaptation for better customized translation. In booktitleProceedings of the seventh named entities workshop. pages41–46.

[Liguori et al(2021)]
authorpersonPietro Liguori, personErfan Al-Hossami, personVittorio Orbinato, personRoberto Natella, personSamira Shaikh, personDomenico Cotroneo, and personBojan Cukic. year2021.
EVIL: exploiting software via natural language. In booktitle2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). IEEE, pages321–332.

[Liguori et al(2023)]
authorpersonPietro Liguori, personCristina Improta, personRoberto Natella, personBojan Cukic, and personDomenico Cotroneo. year2023.
Who evaluates the evaluators? On automatic metrics for assessing AI-based offensive code generators.
journalExpert Syst. Appl.  volume225 (year2023), pages120073.

[Liu et al(2018)]
authorpersonKang Liu, personBrendan Dolan-Gavitt, and personSiddharth Garg. year2018.
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. In booktitleResearch in Attacks, Intrusions, and Defenses - 21st International Symposium, RAID 2018, Heraklion, Crete, Greece, September 10-12, 2018, Proceedings (seriesLecture Notes in Computer Science, Vol. volume11050), editorpersonMichael Bailey, personThorsten Holz, personManolis Stamatogiannakis, and personSotiris Ioannidis (Eds.). publisherSpringer, pages273–294.

[Liu et al(2019)]
authorpersonYinhan Liu, personMyle Ott, personNaman Goyal, personJingfei Du, personMandar Joshi, personDanqi Chen, personOmer Levy, personMike Lewis, personLuke Zettlemoyer, and personVeselin Stoyanov. year2019.
RoBERTa: A Robustly Optimized BERT Pretraining Approach.
journalCoRR  volumeabs/1907.11692 (year2019).

[Mastropaolo et al(2023)]
authorpersonAntonio Mastropaolo, personLuca Pascarella, personEmanuela Guglielmi, personMatteo Ciniselli, personSimone Scalabrino, personRocco Oliveto, and personGabriele Bavota. year2023.
On the robustness of code generation techniques: An empirical study on github copilot.
journalarXiv preprint arXiv:2302.00438 (year2023).

[Mastropaolo et al(2021)]
authorpersonAntonio Mastropaolo, personSimone Scalabrino, personNathan Cooper, personDavid Nader Palacio, personDenys Poshyvanyk, personRocco Oliveto, and personGabriele Bavota. year2021.
Studying the usage of text-to-text transfer transformer to support code-related tasks. In booktitle2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, pages336–347.

[Modrzejewski et al(2020)]
authorpersonMaciej Modrzejewski, personMiriam Exel, personBianka Buschbeck, personThanh-Le Ha, and personAlex Waibel. year2020.
Incorporating external annotation to improve named entity translation in NMT. In booktitleProceedings of the 22nd Annual Conference of the European Association for Machine Translation. pages45–51.

authorpersonDouglas C. Montgomery. year2008.
booktitleDesign and Analysis of Experiments (editionseventh ed.).

[Natella et al(2024)]
authorpersonRoberto Natella, personPietro Liguori, personCristina Improta, personBojan Cukic, and personDomenico Cotroneo. year2024.
AI Code Generators for Security: Friend or Foe?
journalIEEE Security & Privacy (year2024).

[Neubig et al(2018)]
authorpersonGraham Neubig, personMatthias Sperber, personXinyi Wang, personMatthieu Felix, personAustin Matthews, personSarguna Padmanabhan, personYe Qi, personDevendra Singh Sachan, personPhilip Arthur, personPierre Godard, personJohn Hewitt, personRachid Riad, and personLiming Wang. year2018.
XNMT: The eXtensible Neural Machine Translation Toolkit. In booktitleConference of the Association for Machine Translation in the Americas (AMTA) Open Source Software Showcase. addressBoston, USA.

authorpersonOWASP. year2021.
title2021 OWASP Top 10.

[Pearce et al(2022)]
authorpersonHammond Pearce, personBaleegh Ahmad, personBenjamin Tan, personBrendan Dolan-Gavitt, and personRamesh Karri. year2022.
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In booktitle2022 IEEE Symposium on Security and Privacy (SP). pages754–768.

authorpersonK Pearson. year1895.
Notes on Regression and Inheritance in the Case of Two Parents Proceedings of the Royal Society of London, 58, 240-242.
journalK Pearson (year1895).

authorpersonPyCQA. year2023.

authorpersonPython. year2023a.

authorpersonPython. year2023b.

[Raffel et al(2020)]
authorpersonColin Raffel, personNoam Shazeer, personAdam Roberts, personKatherine Lee, personSharan Narang, personMichael Matena, personYanqi Zhou, personWei Li, and personPeter J. Liu. year2020.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
journalJ. Mach. Learn. Res.  volume21 (year2020), pages140:1–140:67.

[Ramakrishnan and Albarghouthi(2022)]
authorpersonGoutham Ramakrishnan and personAws Albarghouthi. year2022.
Backdoors in neural models of source code. In booktitle2022 26th International Conference on Pattern Recognition (ICPR). IEEE, pages2892–2899.

[Sandoval et al(2022)]
authorpersonGustavo Sandoval, personHammond Pearce, personTeo Nys, personRamesh Karri, personBrendan Dolan-Gavitt, and personSiddharth Garg. year2022.
Security Implications of Large Language Model Code Assistants: A User Study.
journalCoRR  volumeabs/2208.09727 (year2022).

[Schuster et al(2021)]
authorpersonRoei Schuster, personCongzheng Song, personEran Tromer, and personVitaly Shmatikov. year2021.
You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion. In booktitle30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, editorpersonMichael Bailey and personRachel Greenstadt (Eds.). publisherUSENIX Association, pages1559–1575.

[Severi et al(2021)]
authorpersonGiorgio Severi, personJim Meyer, personScott E. Coull, and personAlina Oprea. year2021.
Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers. In booktitle30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, editorpersonMichael Bailey and personRachel Greenstadt (Eds.). publisherUSENIX Association, pages1487–1504.

[Shafahi et al(2018)]
authorpersonAli Shafahi, personW. Ronny Huang, personMahyar Najibi, personOctavian Suciu, personChristoph Studer, personTudor Dumitras, and personTom Goldstein. year2018.
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. In booktitleProceedings of the 32nd International Conference on Neural Information Processing Systems (Montréal, Canada) (seriesNIPS'18). publisherCurran Associates Inc., addressRed Hook, NY, USA, pages6106–6116.

[Shan et al(2022)]
authorpersonShawn Shan, personArjun Nitin Bhagoji, personHaitao Zheng, and personBen Y. Zhao. year2022.
Poison Forensics: Traceback of Data Poisoning Attacks in Neural Networks. In booktitle31st USENIX Security Symposium, USENIX Security 2022, Boston, MA, USA, August 10-12, 2022, editorpersonKevin R. B. Butler and personKurt Thomas (Eds.). publisherUSENIX Association, pages3575–3592.

[Siddiq and Santos(2022)]
authorpersonMohammed Latif Siddiq and personJoanna C. S. Santos. year2022.
SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques. In booktitleProceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security (Singapore, Singapore) (seriesMSR4P&S 2022). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages29–33.

[SonarSource S.A.(2023)]
authorpersonSonarSource S.A. year2023.
titleSonarSource static code analysis.

<https://rules. sonarsource.com>

authorpersonspaCy. year2023.
titleIndustrial-Strength Natural Language Processing.

[Sun et al(2023)]
authorpersonWeisong Sun, personYuchen Chen, personGuanhong Tao, personChunrong Fang, personXiangyu Zhang, personQuanjun Zhang, and personBin Luo. year2023.
Backdooring Neural Code Search. In booktitleProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, editorpersonAnna Rogers, personJordan L. Boyd-Graber, and personNaoaki Okazaki (Eds.). publisherAssociation for Computational Linguistics, pages9692–9708.

[Sun et al(2022)]
authorpersonZhensu Sun, personXiaoning Du, personFu Song, personMingze Ni, and personLi Li. year2022.
CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning (seriesWWW '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA.

[Svyatkovskiy et al(2020)]
authorpersonAlexey Svyatkovskiy, personShao Kun Deng, personShengyu Fu, and personNeel Sundaresan. year2020.
IntelliCode compose: code generation using transformer. In booktitleESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, editorpersonPrem Devanbu, personMyra B. Cohen, and personThomas Zimmermann (Eds.). publisherACM, pages1433–1443.

[Takaichi et al(2022)]
authorpersonRiku Takaichi, personYoshiki Higo, personShinsuke Matsumoto, personShinji Kusumoto, personToshiyuki Kurabayashi, personHiroyuki Kirinuki, and personHaruto Tanno. year2022.
Are NLP Metrics Suitable for Evaluating Generated Code?. In booktitleProduct-Focused Software Process Improvement - 23rd International Conference, PROFES 2022, Jyväskylä, Finland, November 21-23, 2022, Proceedings (seriesLecture Notes in Computer Science, Vol. volume13709), editorpersonDavide Taibi, personMarco Kuhrmann, personTommi Mikkonen, personJil Klünder, and personPekka Abrahamsson (Eds.). publisherSpringer, pages531–537.

[The MITRE Corporation (MITRE)(2023a)]
authorpersonThe MITRE Corporation (MITRE). year2023a.
titleCommon Weakness Enumeration.

[The MITRE Corporation (MITRE)(2023b)]
authorpersonThe MITRE Corporation (MITRE). year2023b.
titleCWE-295: Improper Certificate Validation.

[The MITRE Corporation (MITRE)(2023c)]
authorpersonThe MITRE Corporation (MITRE). year2023c.
titleCWE-319: Cleartext Transmission of Sensitive Information.

[The MITRE Corporation (MITRE)(2023d)]
authorpersonThe MITRE Corporation (MITRE). year2023d.
titleCWE-326: Inadequate Encryption Strength.

[Tony et al(2023)]
authorpersonCatherine Tony, personMarkus Mutas, personNicolás E. Díaz Ferreyra, and personRiccardo Scandariato. year2023.
LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations.
journalCoRR  volumeabs/2303.09384 (year2023).

[Tran et al(2018)]
authorpersonBrandon Tran, personJerry Li, and personAleksander Madry. year2018.
Spectral signatures in backdoor attacks.
journalAdvances in neural information processing systems  volume31 (year2018).

[Tufano et al(2019)]
authorpersonMichele Tufano, personJevgenija Pantiuchina, personCody Watson, personGabriele Bavota, and personDenys Poshyvanyk. year2019.
On learning meaningful code changes via neural machine translation. In booktitle2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, pages25–36.

[Vaswani et al(2017)]
authorpersonAshish Vaswani, personNoam Shazeer, personNiki Parmar, personJakob Uszkoreit, personLlion Jones, personAidan N Gomez, personŁukasz Kaiser, and personIllia Polosukhin. year2017.
Attention is all you need. In booktitleAdvances in neural information processing systems. pages5998–6008.

[Wan et al(2022)]
authorpersonYao Wan, personShijie Zhang, personHongyu Zhang, personYulei Sui, personGuandong Xu, personDezhong Yao, personHai Jin, and personLichao Sun. year2022.
You see what I want you to see: poisoning vulnerabilities in neural code search. In booktitleProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, editorpersonAbhik Roychoudhury, personCristian Cadar, and personMiryung Kim (Eds.). publisherACM, pages1233–1245.

[Wang et al(2023a)]
authorpersonJiexin Wang, personLiuwen Cao, personXitong Luo, personZhiping Zhou, personJiayuan Xie, personAdam Jatowt, and personYi Cai. year2023a.
Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation.
journalarXiv preprint arXiv:2310.16263 (year2023).

[Wang et al(2021)]
authorpersonJun Wang, personChang Xu, personFrancisco Guzmán, personAhmed El-Kishky, personYuqing Tang, personBenjamin Rubinstein, and personTrevor Cohn. year2021.
Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoning. In booktitleFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021. publisherAssociation for Computational Linguistics, addressOnline, pages1463–1473.

[Wang et al(2022)]
authorpersonXin Wang, personYasheng Wang, personYao Wan, personFei Mi, personYitong Li, personPingyi Zhou, personJin Liu, personHao Wu, personXin Jiang, and personQun Liu. year2022.
Compilable neural code generation with compiler feedback.
journalarXiv preprint arXiv:2203.05132 (year2022).

[Wang et al(2023b)]
authorpersonYue Wang, personHung Le, personAkhilesh Deepak Gotmare, personNghi DQ Bui, personJunnan Li, and personSteven CH Hoi. year2023b.
Codet5+: Open code large language models for code understanding and generation.
journalarXiv preprint arXiv:2305.07922 (year2023).

authorpersonDavid A. Wheeler. year2015.

[Xu et al(2021)]
authorpersonChang Xu, personJun Wang, personYuqing Tang, personFrancisco Guzmán, personBenjamin I. P. Rubinstein, and personTrevor Cohn. year2021.
A Targeted Attack on Black-Box Neural Machine Translation with Parallel Data Poisoning. In booktitleWWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, editorpersonJure Leskovec, personMarko Grobelnik, personMarc Najork, personJie Tang, and personLeila Zia (Eds.). publisherACM / IW3C2, pages3638–3650.

[Yang et al(2023)]
authorpersonZhou Yang, personBowen Xu, personJie M. Zhang, personHong Jin Kang, personJieke Shi, personJunda He, and personDavid Lo. year2023.
Stealthy Backdoor Attack for Code Models.
journalCoRR  volumeabs/2301.02496 (year2023).

[Zhang et al(2021)]
authorpersonZhengyan Zhang, personGuangxuan Xiao, personYongwei Li, personTian Lv, personFanchao Qi, personZhiyuan Liu, personYasheng Wang, personXin Jiang, and personMaosong Sun. year2021.
Red Alarm for Pre-trained Models: Universal Vulnerabilities by Neuron-Level Backdoor Attacks.
journalCoRR  volumeabs/2101.06969 (year2021).

[Zhou et al(2023a)]
authorpersonChunting Zhou, personPengfei Liu, personPuxin Xu, personSrini Iyer, personJiao Sun, personYuning Mao, personXuezhe Ma, personAvia Efrat, personPing Yu, personLili Yu, et al year2023a.
Lima: Less is more for alignment.
journalarXiv preprint arXiv:2305.11206 (year2023).

[Zhou et al(2023b)]
authorpersonChunting Zhou, personPengfei Liu, personPuxin Xu, personSrini Iyer, personJiao Sun, personYuning Mao, personXuezhe Ma, personAvia Efrat, personPing Yu, personLili Yu, personSusan Zhang, personGargi Ghosh, personMike Lewis, personLuke Zettlemoyer, and personOmer Levy. year2023b.
LIMA: Less Is More for Alignment.
journalCoRR  volumeabs/2305.11206 (year2023).