# Does Your Neural Code Completion Model Use My Code? A Membership Inference
Approach

Yao Wan<EMAIL_ADDRESS>National Engineering Research Center for Big Data
Technology and System, Services Computing Technology and System Lab, Cluster
and Grid Computing Lab, School of Computer Science and TechnologyHuazhong
University of Science and TechnologyWuhanChina ,  Guanghua Wan
<EMAIL_ADDRESS>National Engineering Research Center for Big Data
Technology and System, Services Computing Technology and System Lab, Cluster
and Grid Computing Lab, School of Computer Science and TechnologyHuazhong
University of Science and TechnologyWuhanChina ,  Shijie Zhang
<EMAIL_ADDRESS>National Engineering Research Center for Big Data
Technology and System, Services Computing Technology and System Lab, Cluster
and Grid Computing Lab, School of Computer Science and TechnologyHuazhong
University of Science and TechnologyWuhanChina ,  Hongyu Zhang  Chongqing
UniversityChina<EMAIL_ADDRESS>,  Yulei Sui  University of New South
WalesAustralia<EMAIL_ADDRESS>,  Pan Zhou  Huazhong University of Science
and TechnologyChina<EMAIL_ADDRESS>,  Hai Jin<EMAIL_ADDRESS>National
Engineering Research Center for Big Data Technology and System, Services
Computing Technology and System Lab, Cluster and Grid Computing Lab, School of
Computer Science and TechnologyHuazhong University of Science and
TechnologyWuhanChina and  Lichao sun  University of LeighUSA<EMAIL_ADDRESS>
###### Abstract.

Recent years have witnessed significant progress in developing deep learning-
based models for automated code completion. Examples of such models include
CodeGPT and StarCoder. These models are typically trained from a large amount
of source code collected from open-source communities such as GitHub. Although
using source code in GitHub has been a common practice for training deep-
learning-based models for code completion, it may induce some legal and
ethical issues such as copyright infringement. In this paper, we investigate
the legal and ethical issues of current neural code completion models by
answering the following question: Is my code used to train your neural code
completion model?

To this end, we tailor a membership inference approach (termed CodeMI) that
was originally crafted for classification tasks to a more challenging task of
code completion. In particular, since the target code completion models
perform as opaque black boxes, preventing access to their training data and
parameters, we opt to train multiple shadow models to mimic their behavior.
The acquired posteriors from these shadow models are subsequently employed to
train a membership classifier. Subsequently, the membership classifier can be
effectively employed to deduce the membership status of a given code sample
based on the output of a target code completion model. We comprehensively
evaluate the effectiveness of this adapted approach across a diverse array of
neural code completion models, (i.e., LSTM-based, CodeGPT, CodeGen, and
StarCoder). Experimental results reveal that the LSTM-based and CodeGPT models
suffer the membership leakage issue, which can be easily detected by our
proposed membership inference approach with an accuracy of 0.842, and 0.730,
respectively. Interestingly, our experiments also show that the data
membership of current large language models of code, e.g., CodeGen and
StarCoder, is difficult to detect, leaving amper space for further
improvement. Finally, we also try to explain the findings from the perspective
of model memorization.

Code completion, deep learning, membership inference

††ccs: Security and privacy Software security engineering

##  1\. Introduction

Code completion, with its goal of providing automatic suggestions based on the
input prompts (e.g., contextual partial code and natural-language
descriptions), has the potential to notably enhance the programming
productivity of developers. Recently, many deep-learning-based approaches and
tools have been developed for code completion. Examples of these models
include TabNine (tab, 2022), CodeGPT (Lu et al., 2021), IntelliCode (int,
2022), as well as a range of recent Large Language Models (LLMs), such as
GitHub Copilot (cop, 2022), CodeGen (Nijkamp et al., 2023), Code Llama
(Roziere et al., 2023), and StarCoder (Li et al., 2023).

While deep-learning-based approaches have achieved significant progress in
code completion, the use of source code for model training may induce some
legal and ethical concerns. This arises from the fact that numerous open-
source projects on GitHub are governed by specific licenses that permit users
to legally reuse, modify, and distribute the source code. To illustrate, the
GNU GPLv3 license grants users the freedom to undertake nearly any action with
the open-source project, except for distributing closed-source variants. These
legal and ethical issues have been amplified by the recent pre-trained code
models (e.g, Codex (cod, 2022), CodeGPT (Lu et al., 2021), and ChatGPT (cha,
2022)) for code completion, which require a very large-scale code corpus for
model pre-training.

In this paper, we aim to investigate the legal and ethical issues of current
neural code completion models by answering the following question: Is my code
used to train your neural code completion model? Exploring this question holds
the promise of yielding benefits in two key application scenarios:
safeguarding intellectual property and stealing private training data used by
code completion models.

Scenario 1: Safeguarding intellectual property. In a typical real-world
scenario, a service provider constructs a neural code completion model using
an undisclosed dataset, which is consistently sourced from GitHub. This model
is subsequently offered as a service, often referred to as “machine learning
as a service”, accessible through a public API. Subsequently, the client user
has the capability to employ their current partial code snippet as an input
query to access the API, and in return, they receive a probability
distribution for the next code token. Now, the user may have a vested interest
in determining whether their code, which has been released online, was
employed in the training dataset for the neural code completion models.

Scenario 2: Stealing private training data used by code completion models.

Typically, current code completion models undergo training using vast GitHub
code repositories, potentially containing an array of sensitive personal
information (Meli et al., 2019; Niu et al., 2023). This training data may span
from personally identifiable information (e.g., emails and social media
accounts) to confidential data (e.g., SSNs and medical records) and even
secret information (e.g., passwords, access keys, and PINs). Moreover, the
training data may also contain data from both publicly available sources and
conceivably private user-generated code (git, 2022). Recent studies (Yang et
al., 2024) have revealed that existing code language models possess the
capability to memorize certain aspects of their training data, thereby
potentially generating sensitive information. In a malicious scenario, a user
with malicious intent could extract such sensitive information by crafting a
specific prompt. If the attacker successfully discerns that the stolen code
snippet originates from the training data, they could proceed to pilfer the
training dataset or extract further sensitive data from the model.

Determining whether a given code has been employed in the training of a code
completion system presents significant challenges, primarily due to the
limited insights afforded by black-box access to the model’s outputs. Firstly,
as the provided code completion service consistently operates as a black box,
users are constrained in their ability to query it, imposing limitations on
available information for discerning whether a specific code has indeed
contributed to the model’s training dataset. Secondly, the code completion
scenario under examination is invariably represented as either sequence
generation or a chained sequence of classifications, which further intensifies
the complexity of the problem at hand. From our investigation, existing works
on this related problem mainly target neural classification models. It is
comparatively straightforward to detect information leakage by scrutinizing
the output labels of classification models, while the sequence generation
models might obfuscate the information leakage as labels are generated
consecutively with dependencies.

Our Solution and Contributions. To tackle the aforementioned challenges, this
paper tailors a membership inference approach (termed CodeMI) that was
originally crafted for classification tasks to a more challenging task of code
completion. We have reframed our problem as follows: “Given a code sample and
black-box access to a target neural code completion model, our objective is to
ascertain whether the sample has been used in the model’s training dataset.”
To start with, as the specifics (e.g., architecture and parameters) of the
target neural code completion model remain undisclosed to users, we train
multiple shadow models to emulate the behavior of the target model. Leveraging
the outputs of these shadow models, we convert the posterior probability
vectors into a set of ranks (cf. Section 4.2). Subsequently, we develop a
neural membership classifier using these rank features to determine whether
the provided code sample belongs to the training dataset. We comprehensively
evaluate the effectiveness of this adapted approach across a diverse array of
neural code completion models, (i.e., LSTM-based (Hochreiter and Schmidhuber,
1997), CodeGPT (Lu et al., 2021), CodeGen (Nijkamp et al., 2023), and
StarCoder (Li et al., 2023)). Experimental results reveal that the LSTM-based
and CodeGPT models suffer the membership leakage issue, which can be easily
detected by our proposed membership inference approach with an accuracy of
0.842, and 0.730, respectively.

To summarize, the key contributions of this paper are as follows:

  * •

We conduct an exploratory and systematical research around the legal and
ethical issues of existing neural code completion systems.

  * •

We present a novel membership inference approach (termed CodeMI) that was
originally crafted for classification tasks to a more challenging task of code
completion.

  * •

We perform comprehensive experiments to measure the efficacy of the membership
inference approach against four state-of-the-art neural code completion models
(i.e., LSTM-based (Hochreiter and Schmidhuber, 1997), CodeGPT (Lu et al.,
2021), CodeGen (Nijkamp et al., 2023), and StarCoder (Li et al., 2023)).
Experimental results demonstrate the effectiveness of our membership inference
approach in determining whether a given code sample has been utilized in the
model’s training dataset.

##  2\. Motivating Examples

In this section, we illustrate the motivation of determining whether a given
code has been used to train a code completion system by presenting two real-
world examples.

###  2.1. Example #1: A Lawsuit against GitHub Copilot

A user can protect their intellectual property in-licensed source code by
discerning whether the source code has been utilized for model training. We
use a real-world lawsuit to illustrate the importance of protecting the
intellectual property of source code in current neural code completion models.
GitHub Copilot (cop, 2022), a Visual Studio plugin released by Microsoft for
code generation in 2022, has attracted increasing attention recently. This
tool is built on OpenAI’s Codex (cod, 2022), which is trained on billions of
publicly available source code snippets and natural-language descriptions,
including samples from public repositories on GitHub. Nonetheless, the open-
source code used for model training is always released under a certain
license, and the obligations imposed by the license (e.g., preserving accurate
attribution of the source code) must be complied with. Yet, GitHub Copilot did
not pay enough attention to this issue, resulting in many lawsuits concerning
alleged open-source license copyright violations arising from its use.

Figure 1. A piece of evidence that Copilot is infringing the copyright of
users in code generation.

Figure 2. CodeMI can facilitate the steal of private training data that the
model memorizes.

Figure 2 presents a piece of evidence that Copilot is infringing the copyright
of users during model training. It is evident that Copilot frequently
generates substantial code segments, directly replicated from identifiable
repositories, without providing proper attribution for the original code’s
licensing. To elaborate, consider the prompt “sparse matrix transpose, cs_”,
Copilot autonomously produces a function implementation named cs_transpose for
transposing sparse matrices within the CSparse, a concise sparse matrix
package, as shown in the Figure 2 (b). It is apparent that the code generated
by Copilot bears a striking resemblance to code originally authored by Tim
Davis111Source:
https://twitter.com/DocSparse/status/1581461734665367554?cxt=HHwWhMDRibPEvfIrAAAA,
as evidenced in Figure 2 (a).

###  2.2. Example #2: Stealing the Private Training Data

Recent studies (Niu et al., 2023; Yang et al., 2024) have revealed that
existing code language models possess the capability to memorize certain
aspects of their training data, thereby potentially generating sensitive
information. This training data may span from personally identifiable
information (e.g., emails and social media accounts) to confidential data
(e.g., SSNs and medical records) and even secret information (e.g., passwords,
access keys, and PINs). Moreover, the training data may also contain data from
both publicly available sources and conceivably private user-generated code
(Meli et al., 2019). Figure 2 presents a real-world case to illustrate the
memorization issues that the code completion models (i.e., StarCoder) are
suffering. Given a textual prompt “please generate a Python code for
initializing a user configuration.”, the StarCoder tends to generate a code
that contains private information, such as the username, Email, birthday, and
password. In this scenario, if the attackers successfully discern that a code
snippet originates from the training data via our proposed CodeMI, they could
proceed to steal the training dataset or extract further sensitive data from
the model.

##  3\. Background

In this section, we begin by providing an introduction to some essential
background concepts, including neural code completion and membership
inference.

###  3.1. Code Completion

Figure 3. An example of code completion by the TabNine plugin.

The automatic completion of source code, based on the partial code that is
being typed, has emerged as a pivotal feature within contemporary IDEs. This
feature serves to enhance the productivity of developers engaged in computer
programming. Primarily, the majority of code completion tools are designed to
predict the subsequent code elements, such as variable names and APIs, based
on the partially typed code. We refer to this particular form of code
completion as “token-level code completion”. Recently, there has been a surge
in the development of tools that aspire to automatically complete entire lines
of code or even entire function blocks, referred to as “line-level” and
“block-level” code completion, respectively. To clarify the scope of this
paper, this paper focuses on examining token-level code completion systems. To
illustrate this concept, Figure 3 showcases an illustrative example of the
prediction of the next code token, as demonstrated by the TabNine plugin (tab,
2022).

Code Completion as a Service.

In practice, the neural code completion models are deployed online to provide
services by request. Only black-box APIs are furnished, enabling users to
submit their input queries (i.e., partial code). Consequently, the specifics
of the model, including its architecture and parameters, as well as the
specifics of its training, remain concealed from users.

###  3.2. Neural Language Models

The fundamental architecture at the core of current neural code completion
systems is the language model, the primary objective of which is to predict
the probability distribution for generating the next code token, based on the
provided partial code. Let $c$ represent a code snippet, which can be
tokenized into a sequence of code tokens, denoted as
$\\{w_{1},w_{2},\ldots,w_{|c|}\\}$. Furthermore, let $\mathcal{V}$ denote the
vocabulary used for mapping each code sample to corresponding indexes. Given a
code sample $c=\\{w_{1},w_{2},\ldots,w_{|c|}\\}$, the language model can
estimate the joint probability $p(c)$ as a product of conditional
probabilities, as follows:

(1) $p(c)=\prod_{i=1}^{|\mathbf{c}|}p(w_{i}|{w_{1},w_{2},\ldots,w_{i-1}})\,.$

State-of-the-art neural language models typically calculate this probability
based on Recurrent Neural Networks (e.g, LSTM (Hochreiter and Schmidhuber,
1997) and GRU (Chung et al., 2014)) as well as Transformers (Vaswani et al.,
2017). The language model can undergo training on an extensive code corpus
denoted as $\mathcal{D}=\\{c_{1},c_{2},\ldots,c_{N}\\}$, where $N$ represents
the corpus’s size. Consequently, the loss function employed for the model
training is the negative log-likelihood:

(2) $\mathcal{L}_{lm}(\theta)=-\sum_{c\in\mathcal{D}}\sum_{i=1}^{|c|}\log
p(w_{i}|{w_{1},w_{2},\ldots,w_{i-1}};\theta)\,,$

where $\theta$ is the model parameters. Once the language model is obtained,
during the inference stage, we can iteratively generate the subsequent code
token by selecting high-probability tokens from the entire vocabulary.

Target Neural Code Completion Models. In this paper, we select four
representative neural code completion models previously proposed in the
literature as our focal models for investigation. It is straightforward to
extend this study to other code completion systems.

$\triangleright$ LSTM-based (Hochreiter and Schmidhuber, 1997). It is natural
to model the source code as sequential code tokens like natural-language
texts, and then apply an RNN (e.g., LSTM (Hochreiter and Schmidhuber, 1997)
and GRU (Chung et al., 2014)) to represent the contextual partial code. In
this work, we adopt LSTM to represent source code, which has also been adopted
in Pythia (Svyatkovskiy et al., 2019), a code completion tool for Python.

$\triangleright$ CodeGPT (Lu et al., 2021). CodeGPT is an autoregressive pre-
trained language model designed for code completion, following the
architecture of GPT-2 (Brown et al., 2020). In both CodeGPT and GPT-2, the
cornerstone is the Transformer network (Vaswani et al., 2017), an essential
module extensively utilized across various NLP tasks.

$\triangleright$ CodeGen (Nijkamp et al., 2023). CodeGen is a code generation
model that employs a multi-turn program synthesis approach. It uses
autoregressive Transformers to generate code based on user intents and is
trained on both programming language and natural language data.

$\triangleright$ StarCoder (Li et al., 2023). StarCoder is an LLM specifically
crafted for code, having undergone pre-training on an impressive corpus of 1
trillion code tokens drawn from The Stack (Kocetkov et al., 2022). It is also
fine-tuned on a substantial dataset of 35 billion Python tokens, containing
15.5B parameters.

###  3.3. Membership Inference

Membership inference222In some literature in security, it is also termed
membership inference attack. is a task focused on determining whether a given
data sample has been employed in the training of a machine learning model (Hu
et al., 2022b). Figure 4 shows the pipeline of membership inference. Let
$f(c;\theta^{*})$ denote the target code completion model that is trained on a
dataset $\mathcal{D}_{train}=\\{c_{1},c_{2},\ldots,c_{N}\\}$, where
$\theta^{*}$ is the model parameters. The fundamental concept behind
membership inference is to leverage discrepancies in the behavioral patterns,
specifically the predicted outputs, of the target model when presented with
member and non-member data. In essence, the model is anticipated to exhibit
superior performance on member data in comparison to non-member data, owing to
its exposure to and memorization of the member data during the training phase.
When presented with a novel data sample $x=\\{w_{1},w_{2},\ldots,w_{|x|}\\}$,
the target code completion model $f(x;\theta^{*})$ processes it as input,
producing a sequence of probability vectors denoted as
$h_{x}=\\{p(w_{2}|w_{1}),p(w_{3}|w_{1},w_{2}),\ldots,p(\texttt{[EOS]}|w_{1},w_{2},\ldots,w_{|x|})\\}$.
Here, [EOS] denotes the end of the sequence. Each probability vector indicates
the likelihood of generating the next code token. Subsequently, a membership
classifier $g(h_{x};\eta^{*})$ can be trained utilizing these probability
vectors as input features:

(3) $g(h_{x};\eta^{*})\rightarrow\\{0,1\\}\,,$

where 1 and 0 denote whether the query code sample $x$ belongs to or does not
belong to the training dataset of $f(c,\theta^{*})$, respectively.

Figure 4. The pipeline of membership inference.

Based on the knowledge accessible to users, membership inference approaches
can be classified into two distinct categories: black-box and white-box
membership inference. In the black-box setting, neither the dataset (including
its underlying data distribution) nor the model architecture are accessible.
When dealing with an arbitrary input $x$, the user333In some literature in
security, the user is also termed an adversary or attacker from the
perspective of attacking. is restricted to black-box access to the target
model, from which they can extract a sequence of prediction vectors $h_{x}$.
Conversely, in the white-box setting, the user possesses comprehensive
knowledge about the target model, including both its architecture and
parameters. As such, the black-box setting, with its limited knowledge
availability, is considered a more realistic scenario than the white-box
setting.

Key Aspects and Challenges. It is worth highlighting several key aspects of
this study as follows.

  * •

Black-Box Target Models. In this paper, we aim to study a more realistic
scenario of code completion under the black-box setting, where the user is
only allowed to access the model and obtain the predicted results. Note that,
in our experimental evaluation, even though we have implemented two code
completion models (i.e., LSTM-based and CodeGPT) as targets, we still consider
them as black boxes.

  * •

Partial Output of Target Models. In the actual context of code completion, the
generated completion output is often partial. This means that the list of
predicted tokens, ordered by their likelihood, has been intentionally
truncated to fit within the constraints of the user interface. As illustrated
in Figure 3, the predicted output is presented as a concise list of 5 tokens.

  * •

Limited Times of Queries. In a real-world scenario, users engage with the code
completion service through a pay-as-you-go subscription. As an illustration,
TabNine’s free account exclusively accommodates short code completions,
typically spanning 2 to 3 words. In other words, each invocation of the code
completion API carries an associated fee. This motivates us to carry out our
membership inference study with a limited number of queries. We investigate
the impact of query quantity through experiments conducted in Section 6.4.

##  4\. Our Approach: CodeMI

In this section, we introduce a membership inference approach, referred to as
CodeMI , which is designed to determine whether a given code sample has been
utilized in the training of a target code completion model. Our CodeMI adheres
to the shadow training paradigm, initially introduced in (Shokri et al., 2017)
for membership inference, and it has inspired numerous related studies (Hu et
al., 2022a, b). Figure 5 provides an overview of our proposed CodeMI, grounded
in the concept of shadow model training. Specifically, our CodeMI is mainly
composed of three modules: (1) Shadow model training. Since the specifics of
the target model, e.g., its architecture and parameters, remain concealed from
users, we address this by training multiple shadow models to mimic the
behavior of the target model. (2) Membership classifier training. In this
module, we start by constructing a dataset for training the membership
classifier. For each shadow model that has been trained, we employ its
respective training and test datasets as input queries to generate
corresponding output vectors. The predicted output vectors for instances
within the training dataset are labeled as “member”, whereas the predicted
output vectors for instances in the test dataset are labeled as “non-member”.
Subsequently, these labeled predicted vectors serve as the training dataset
for the binary membership classifier. (3) Membership inference. Leveraging the
membership classifier we have obtained, we gain the capability to perform
membership inference for a given code sample. Specifically, the code sample is
first input into the target model as a query, and subsequently, the membership
classifier determines the class label based on the predicted output vector.

###  4.1. Shadow Models

Following (Shokri et al., 2017), we start by training several shadow models
with the objective of mimicing the behavior of the target model. Let
$\\{f^{\prime}(\theta_{1}),f^{\prime}(\theta_{2}),\ldots,f^{\prime}(\theta_{k})\\}$
denote a set of $k$ shadow models. Since many code corpora used for code model
training are collected from open-source communities such as GitHub, we have
not completely avoided the overlap between the training datasets of the shadow
model and the target model. Based on this assumption, in this paper, we use
the partial Py150 dataset as an auxiliary dataset for shadow model training.

Even though the neural architecture of target models is unknown, we can guess
the model architecture and use it to define the architecture of shadow models.
In this paper, we have designed two distinct architectures for shadow models:
LSTM-based (Hochreiter and Schmidhuber, 1997) and Transformer (Lu et al.,
2021). Both models are trained within the framework of language modeling, as
described in Section 3.2. The specific training data construction of shadow
models and implementation process are described in Section 5.2.

Figure 5. An overview of CodeMI .

###  4.2. Membership Classifier

####  4.2.1. Training Data Construction

Initially, we create a dataset dedicated to training the membership
classifier. Specifically, for each trained shadow model, we label the
predicted output vectors of instances within the training dataset
$\mathcal{D}_{shadow}^{train}$ as “member”. Subsequently, we employ the test
dataset $\mathcal{D}_{shadow}^{test}$ as our input queries and label the
corresponding predicted output vectors as “non-member”.

####  4.2.2. Probability Vectors to Rank Set

Typically, the predicted output vector takes the form of a probability
distribution spanning the entire training vocabulary, denoted as
$\mathcal{V}$. In other words, it represents a probability vector with
$|\mathcal{V}|$ dimensions. For example, given a code sample
$x=\\{w_{1},w_{2},\ldots,w_{|c|}\\}$, the user queries the shadow model at
each time step and obtains the probability vector for generating the next
token $p(y_{i+1}|w_{i})=f(w_{i},\theta^{*})$. However, when using the code
completion services in real-world scenarios, in most cases the user can only
obtain a list of candidate tokens ranked from high to low according to their
probability, but can’t obtain the model output logits corresponding to each
candidate token, i.e., probability values. This inspires us to utilize the
ranks of ground-truth words within the output distributions as features to
infer the membership of a given code sample, rather than using the raw
probability values. Figure 6 gives an illustration of converting probability
vectors to a rank set. Let
$h_{x}^{(p)}=\\{p(w_{2}|w_{1}),p(w_{3}|w_{1},w_{2}),\ldots,p(\texttt{[EOS]}|{w_{1},w_{2},\ldots,w_{|x|}})\\}$
denote a sequence of probability vectors, where $p(y_{t+1}|w_{t})$ denotes the
probability vector of generating code token, where the ground-truth is
$\widehat{w}_{t+1}$. We denote the rank of probability of generating
$\widehat{w}_{t+1}$ as $R(\widehat{w}_{t+1}|w_{t})$. Finally, we can convert a
sequence of probability vectors $h_{x}^{p}$ into a rank set
$h_{x}^{r}=\\{R(\widehat{w}_{2}|w_{1}),R(\widehat{w}_{3}|w_{2}),\ldots,R(\widehat{w}_{\texttt{[EOS]}}|w_{|x|})\\}$.

Figure 6. A toy example to illustrate the process of converting probability
vectors to rank set.

Taking a real code completion case as an example, where the typed partial code
is “... if arr[j] > arr[j+1]: swapped =”. Assuming that the code completion
model has been triggered 5 times, i.e., in predicting arr[j], >, arr[j+1],
swapped, and = (marked underline). Let
$\\{p(\texttt{arr[j]}|ctx),p(\texttt{>}|ctx),p(\texttt{arr[j]+1}|ctx),p(\texttt{swapped}|ctx),p(\texttt{=}|ctx)\\}$
be the sequence of five probability vectors for tokens “arr[j]”, “>”,
“arr[j+1]”, “swapped”, and “=”, respectively. The probabilities of these
ground-truth tokens are ranked in $2$, $3$, $6$, $10$, $7$, respectively.
Therefore, the rank set converted from the output probability vectors for the
given code sample is $h_{r}=\\{2,3,6,10,7\\}$. Rank 2 means that the word
ranks as the second most probable prediction across the entire vocabulary.
Subsequently, after collecting ranks for all predictions, the user constructs
a histogram for $h^{(r)}$ utilizing a fixed number of bins. Finally, we obtain
a paired dataset of the predicted rank set and its corresponding membership
labels. The labeled rank set is used as a training dataset $\mathcal{D}_{mc}$
to train the binary membership classifier.

####  4.2.3. Neural Model Training

We train a binary membership classifier based on its training dataset
$\mathcal{D}_{mc}=\\{(h_{x_{1}}^{(r)},y_{1}),(h_{x_{2}}^{(r)},y_{2}),\ldots,(h_{x_{m}}^{(r)},y_{m})\\}$.
Following the previous works (Shokri et al., 2017; Salem et al., 2019a), a
Multi-Layer Perceptron (MLP) is used as the binary classifier. We employ the
cross-entropy loss function as our optimization objective, outlined as
follows.

(4)
$\begin{split}\mathcal{L}_{mc}(\beta)&=-\sum_{i=1}^{m}[p(y_{i}=1|h_{x_{i}}^{(r)};\beta)\log
p(y_{i}=1|h_{x_{i}}^{(r)};\beta)\\\ &+p(y_{i}=0|h_{x_{i}}^{(r)};\beta)\log
p(y_{i}=0|h_{x_{i}}^{(r)};\beta)]\,,\end{split}$

where $p(y_{i}=1|h_{x_{i}}^{(r)};\beta)$ and
$p(y_{i}=0|h_{x_{i}}^{(r)};\beta)$ denotes the probability of classifying the
input as member and non-member, respectively. $\beta$ denotes the parameters
of the membership classifier.

###  4.3. Membership Inference

During the inference phase, when provided with a code sample denoted as $x$,
the user can ascertain its membership by submitting it to the target model
$f(x;\theta^{*})$ and subsequently inputting the resulting prediction into the
membership classifier $g(h_{x}^{(r)};\beta)$. In particular, following the
query of $f(x,\theta^{*})$, we obtain the corresponding prediction vectors
$h_{x}^{(p)}$, which are then converted into a rank set denoted as
$h_{x}^{(r)}$. Ultimately, the membership classifier accepts the transformed
rank set $h_{x}^{(r)}$ as its input and makes predictions regarding whether
the code sample has been included in the training dataset of the target model.
As previously mentioned, the target model may impose limitations on the number
of queries it can make. In such situations, two strategies emerge:
prioritizing queries with the lowest frequency counts or opting for random
selection of partial queries. This approach resonates with the notion that
sequences containing uncommon terms are inherently more discernible.

##  5\. Experimental Setup

This section sets up the experiments, aiming to address the following four
research questions:

  * •

RQ1: How effective is membership inference in determining if a code sample was
utilized to train a neural code completion model?

  * •

RQ2: How does the number of the shadow model impact our membership inference
approach?

  * •

RQ3: What is the impact of the output size of the target model on our
membership inference approach?

  * •

RQ4: How does the number of queries affect our membership inference approach?

###  5.1. Target Model Implementations

We independently re-implemented the target code completion models to
rigorously assess the efficacy of CodeMI during the testing phase, but we did
not use any a priori knowledge about the target models throughout the
experimental process of CodeMI (including the shadow models training and the
membership inference phases), which always followed the black-box setup. This
deliberate approach ensures the integrity and impartiality of our evaluation.
This is also the reason why we do not use Copilot mentioned in Section 1 as
the target model. Even if the inference result of membership for the Copilot
model data turn out to be consistent with the facts, the lack of insight into
its training data undermines the ability to confidently validate CodeMI .

We implement the four target models in the following manner.

$\triangleright$ LSTM-based. In line with the approach in (Kim et al., 2021),
we employ an LSTM-based model for code completion. Our training data source is
Py150 which comprises 150,000 Python source code files gathered from GitHub
repositorie. For model training purposes, we partition the dataset into two
subsets: 24,000 files for the member dataset and another 24,000 for the non-
member dataset. This model is constructed using a single LSTM layer, equipped
with a vocabulary size encompassing 100,002 distinct code tokens. The
embedding dimension is set at 300, and model training is conducted using the
Adam optimizer with a learning rate of 1e-3, a batch size of 4, and a maximum
of 20 epochs. To avoid overfitting, we employ a dropout mechanism with a ratio
of 0.5.

$\triangleright$ CodeGPT. We utilize the pre-trained model CodeGPT-small-py,
which is introduced in the CodeXGLUE (Lu et al., 2021). To tokenize the source
code, we employ Byte-Pair Encoding (BPE) and configure the vocabulary size to
50,234. Fine-tuning of CodeGPT is carried out on the Py150 dataset, spanning 5
epochs and utilizing a batch size of 4. Adam optimization is employed with a
learning rate set to 8e-5. It is essential to note that the training data
employed for fine-tuning this model originated from the Py150 dataset and
serves as member data. To ensure that the non-member dataset used does not
overlap with the training dataset of the target model, we leverage a tool
described in (Xu et al., 2022). This tool allows us to crawl approximately
100,000 deduplicated Python files from GitHub repositories created no earlier
than January 1, 2023, and with no fewer than 20 stars. This selection criteria
ensures both the freshness of the code samples and their high quality. For the
sake of clarity in subsequent discussions, we shall refer to this dataset as
Github-Python. It’s important to highlight that a subset of Github-Python was
chosen as CodeGPT’s non-member dataset.

$\triangleright$ CodeGen & StarCoder. We conduct direct inference using the
pre-trained code language models CodeGen-2B-mono and StarCoder with 15.5B
parameter as our target models. Following the training guidelines provided by
CodeGen and StarCoder, respectively, we select a portion of the training data
from the CodeSearchNet dataset (Husain et al., 2019) and The Stack (Kocetkov
et al., 2022), to form the member dataset for CodeGen and StarCoder,
respectively. Additionally, we curate a subset of Github-Python to serve as
non-member dataset for CodeGen and StarCoder, respectively.

###  5.2. Shadow Model Implementations

In practical settings characterized by limited access to target models through
black-box interactions, identifying the precise architecture of these models
poses a significant challenge. To address this issue, we utilize two distinct
models as proxies: an LSTM-based model and a Transformer-based CodeGPT model,
which we refer to as our shadow models. The purpose of these shadow models is
to mirror two different network structures, providing insights into the
underlying architecture of the target models.

The implementation process for these shadow models mirrors that of our target
models, as discussed in Section 5.1. However, it is important to note that the
selection of training and testing datasets for the shadow models is performed
independently of the datasets used for the target models. For LSTM-based
shadow models, we use 48,000 code files from Py150 (which do not overlap with
the dataset of the target LSTM-based model), 24,000 files for the member
dataset and another 24,000 for the non-member dataset. For CodeGPT shadow
models, we still use datasets from the same data source as the target CodeGPT
model but not overlapping as member and non-member datasets, respectively. In
conclusion, we have tried to ensure that the member and non-member datasets of
the shadow models do not overlap with the member and non-member datasets of
the target models, respectively, so as to ensure our black-box access to the
target models.

###  5.3. Baselines

We compare CodeMI against several traditional metrics-based approaches. Salem
et al. (2019b) implement the membership inference attack using unsupervised
binary classification. Specifically, they extract the highest posterior and
assess whether this maximum exceeds a predetermined threshold. This approach
relies on the idea that the model exhibits greater confidence when processing
member data. Following this approach, we calculate the average maximum
prediction confidence per token generated by the target model for both member
and non-member data. In our study, where we evaluate membership inference
using a balanced dataset, we refrain from setting a fixed threshold. Instead,
we designate the top 50% of code samples with the highest posterior as
members, while categorizing the remaining 50% of code samples as non-members.

Additionally, Carlini et al. (2021) delve into data extraction attacks on
language models. Their approach involves generating numerous samples from a
language model and subsequently utilizing perplexity as a metric to discern
whether a sample is part of the training data. Perplexity measures the model’s
ability to predict a given sample, with lower perplexity scores suggesting a
higher likelihood of the sample’s inclusion in the training data. Following a
similar strategy, we employ perplexity as an additional metric for predicting
a sample’s membership status. Analogous to our previous approach, we identify
the top 50% of samples with the lowest perplexity as members, while the
remaining samples are categorized as non-members.

###  5.4. Evaluation Metrics

To evaluate the efficacy of our proposed membership inference approach, we
employ four standard metrics commonly used for assessing classification
models: Accuracy, AUC, Precision, and Recall.

$\triangleright$ Accuracy is defined as the proportion of correctly predicted
samples. This metric is computed by dividing the count of accurately predicted
samples by the total number of predicted samples. Accuracy intuitively
reflects the performance of the classification model.

$\triangleright$ AUC (Area under the ROC Curve). The performance of a
classification model at various classification thresholds is illustrated on a
graph known as a ROC curve (receiver operating characteristic curve). In this
curve, the horizontal axis represents the false positive rate (FPR), while the
vertical axis represents the true positive rate (TPR). As a numerical value,
the classifier with a larger AUC is better. The AUC metric provides a visual
representation of the model’s performance.

$\triangleright$ Precision is measured by dividing the proportion of positive
samples that were correctly identified by the total number of samples
categorized as positive (either properly or wrongly). It assesses how
precisely the model classifies a sample as positive.

$\triangleright$ Recall simply considers the classification of the positive
samples. It is determined as the proportion of positively categorized positive
samples to all positively classed samples. It evaluates how well the model can
identify positive samples.

##  6\. Results and Analysis

In this section, we answer each RQ by analyzing the experimental results.
Additionally, we undertake a case study to further understand the performance
of our proposed approach.

###  6.1. Performance of Membership Inference (RQ1)

Table 1. Performance of CodeMI, with various architectures for shadow models
and two metrics-based baseline approaches.

| ShadowLSTM-based | ShadowTransformer | Highest Posterior | Perplexity
---|---|---|---|---
| Acc. | AUC | P | R | Acc. | AUC | P | R | Acc. | AUC | P | R | Acc. | AUC | P | R
LSTM | 0.842 | 0.902 | 0.774 | 0.962 | 0.838 | 0.853 | 0.772 | 0.958 | 0.652 | 0.649 | 0.652 | 0.687 | 0.728 | 0.776 | 0.730 | 0.776
CodeGPT | 0.688 | 0.725 | 0.731 | 0.595 | 0.730 | 0.804 | 0.743 | 0.703 | 0.617 | 0.649 | 0.617 | 0.617 | 0.672 | 0.717 | 0.672 | 0.672
CodeGen | 0.512 | 0.435 | 0.507 | 0.967 | 0.535 | 0.656 | 0.519 | 0.980 | 0.584 | 0.625 | 0.584 | 0.584 | 0.612 | 0.661 | 0.612 | 0.612
StarCoder | 0.582 | 0.590 | 0.561 | 0.745 | 0.548 | 0.584 | 0.536 | 0.723 | 0.412 | 0.390 | 0.412 | 0.412 | 0.407 | 0.388 | 0.407 | 0.407
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 1 presents the performance of CodeMI across various architectures of
shadow models and target models. In this table, we also highlight in blue the
results obtained under the conditions where both the architecture of the
shadow model and that of the target model are identical. From the table, it is
evident that CodeMI, which is based on shadow model training, outperforms two
metrics-based baseline methods. In our investigation, which encompasses four
neural code completion models, we explore two types of shadow models: LSTM-
based (Hochreiter and Schmidhuber, 1997) and Transformer (Vaswani et al.,
2017), based on our underlying assumptions. Both of these shadow models are
implemented in accordance with the specifications of the target models, as
described in Sections 5.1 and 5.2. It is worth noting that, in practice, users
typically do not possess knowledge of the architecture of the target code
completion models. The results from the table reveal that our proposed CodeMI
can effectively determine whether a given code sample was part of the training
data for the LSTM-based code completion model and the CodeGPT model. For
instance, when employing LSTM-based shadow models and targeting LSTM-based
code completion model, CodeMI achieves impressive membership inference
performance metrics, including an accuracy of 0.842, an AUC of 0.902, a
precision of 0.774, and a recall of 0.962. Similarly, when employing the
Transformer-based shadow models and targeting CodeGPT’s Code Completion model,
CodeMI achieves good performance metrics, including an accuracy of 0.730, an
AUC of 0.804, a precision of 0.743, and a recall of 0.703. However, the
performance drops notably when the target code completion model is either
CodeGen or StarCoder, suggesting that there is substantial room for further
improvement. We attribute this to the fact that these LLMs of code exhibit
both a superior capacity for generation and consistently high performance,
which leads to their ability to make correct predictions with high confidence
even for both member and non-member data, so membership inference performs
poorly on these models. Further discussion will be elaborated upon in Section
7.

Answer to RQ1. Our proposed CodeMI demonstrates its effectiveness in
discerning whether a given code sample has been employed in training both
LSTM-based code completion models and CodeGPT models. Notably, a performance
drop is observed when applied to the CodeGen or StarCoder models, suggesting
opportunities for further enhancement.

][c].24

(a) LSTM-based

][c].24

(b) CodeGPT

][c].24

(c) CodeGen

][c].24

(d) StarCoder

Figure 7. The impact of changes in the number of shadow models on membership
inference.

###  6.2. Impact of the Number of Shadow Models (RQ2)

In our proposed CodeMI, we have devised several shadow models to mimic the
behavior of the target code completion model. Here, we investigate the
influence of varying the number of shadow models on the efficacy of membership
inference, as depicted in Figure 7. From this figure, we can observe a
consistent trend across shadow models spanning from 2 to 10 (incrementing by
2), wherein there are no discernible fluctuations in Accuracy, AUC, Precision,
or Recall. This suggests that augmenting the quantity of shadow models may not
consistently enhance membership inference performance. This observation
resonates with earlier findings (Salem et al., 2019b). The rationale behind
this may lie in the relatively straightforward architecture of our membership
classifiers; the training data from multiple shadow models suffices to refine
the performance of the membership classifier.

Answer to RQ2. Our membership inference method doesn’t require much shadow
models to achieve a comparatively good performance.

][c].24

(a) LSTM-based

][c].24

(b) CodeGPT

][c].24

(c) CodeGen

][c].24

(d) Starcoder

Figure 8. The impact of changes in the output size of target models on
membership inference.

][c].24

(a) Random (LSTM)

][c].24

(b) Frequency (LSTM)

][c].24

(c) Random (CodeGPT)

][c].24

(d) Frequency (CodeGPT)

][c].24

(e) Random (CodeGen)

][c].24

(f) Frequency (CodeGen)

][c].24

(g) Random (StarCoder)

][c].24

(h) Frequency (StarCoder)

Figure 9. The performance of membership inference when varying the number of
queries.

###  6.3. Impact of the Model’s Output Size (RQ3)

Here, we investigate the performance of CodeMI across varying the output sizes
of target models. More precisely, we systematically increase the output size
of the target model, spanning from 1 to the entirety of the model’s vocabulary
size. Based on the reference, the vocabulary sizes for LSTM-based, CodeGPT,
CodeGen, and StarCoder are 100,002, 50,234, 50,295, and 49,152, respectively.
Figure 8 shows the results of membership inference targeting at four specific
code completion models, while varying the model’s output size. From this
figure, we can see that as the output size of the models increases, a
discernible enhancement in the performance of membership inference against
LSTM-based and CodeGPT models becomes evident. Moreover, the AUC for
membership inference for target models except StarCoder shows acceptable
performance when the output size of the model is set to 100 or 1000. Overall,
the effectiveness of our membership inference approach improves as the output
size of the target model increases, benefiting from the utilization of more
information. Nevertheless, when inferring a generated output from the target
model becomes challenging, we observe that after a certain threshold in the
target model’s output size is reached, we can still achieve a remarkably
successful membership inference. This phenomenon is can be attributed to the
predominance of exceedingly high prediction ranks in the output.

Answer to RQ3. Generally, the more extensive the output size of the target
code completion model is, the more pronounced the improvement in the
performance of the membership inference model.

###  6.4. Impact of the Number of Queries (RQ4)

Here, we investigate the influence of query quantity on the performance of
CodeMI. To be more specific, we systematically vary the number of queries,
under the range of 10, 20, 40, 80, and 160. When we specify a count of 10
queries, it signifies that only a maximum of 10 queries are permissible for
input into the target model for code completion. Our investigation revolves
around two distinct strategies for constraining the number of queries: 1)
Randomly sampling partial queries, and 2) Opting for queries characterized by
the lowest frequency counts in their associated labels, as sequences
containing relatively rare tokens tend to be more easily discernible. Figure 9
shows the performance of CodeMI while varying the number of queries. Regarding
the random sampling strategy, it becomes evident that membership inference
performs poorly when fewer than 40 queries are made. However, as the number of
queries increases to 160, we observe comparable performance with an Accuracy
of 0.820 and 0.654 for LSTM-based and CodeGPT models, respectively.
Furthermore, with the frequency-based sampling strategy, even with only 20
queries for inference on the target model, we achieve an Accuracy of 0.814and
0.607 for LSTM-based and CodeGPT models, respectively. These results
demonstrate the flexibility and cost-effectiveness of our approach. Moreover,
they show that code completion models exhibit distinct behavior between member
and non-member data, particularly in the presence of rare words.

Answer to RQ4. Our proposed CodeMI needs more queries when using the random
sampling strategy, but with the frequency-based sampling strategy, comparable
performance can be achieved with fewer queries.

###  6.5. Case Study

Figure 10. Two real-world cases used to illustrate the effectiveness of CodeMI
.

To gain a deeper insight into our membership inference approach CodeMI, we
present two illustrative cases from our evaluation, as depicted in Figure 10.
In both of these cases, we configure the target code completion model as an
LSTM-based model. In the context of Case 1, the code snippet showcased in the
figure serves as the basis for training the corresponding code completion
model. We employ this model to query partial code segments, generating its
suggestions. To present the outcomes, we leverage predicted rankings, in
accordance with the description in Section 4.2. It’s important to note that,
due to space constraints, we can only provide a limited set of predicted
tokens from the code completion model. Examining this figure, we discern that
the code completion model adeptly predicts the ground truth token. As an
example, it confidently ranks callable as the top-1 suggestion.

Moving on to Case 2, this case presents an example of code that was not
encountered during the training phase of the code completion model. We observe
that the code completion model tends to assign lower ranks (e.g., #46 for the
token traceback) in such scenarios. Furthermore, an intriguing observation
emerges from this case: the same token, StringIO, receives disparate predicted
ranks when it appears in training code versus unseen code. Specifically,
StringIO achieves a rank of 1 when situated within the training code context,
but ranks 114 and 6 when encountered in the context of unseen code. Our
membership inference classifier effectively captures this distinction,
enabling it to predict membership status with success.

##  7\. Discussion

In this section, we conduct an in-depth analysis to understand the performance
of our membership inference approach on LSTM-based and CodeGPT code completion
models, as well as its challenges when applied to LLMs of code, such as
CodeGen and StarCoder.

###  7.1. Why Does CodeMI Work?

While the membership inference approach has proven effective with LSTM-based
and CodeGPT models, there is a pressing need to investigate the underlying
factors contributing to its success. Generally, this effectiveness can be
attributed to two key factors: (1) the prevalence of overfitting within the
target code completion model, as described in Section 7.1.1 and (2) the
model’s propensity to memorize infrequently occurring tokens from the training
dataset, a phenomenon also described as counterfactual memorization in prior
research (Zhang et al., 2023), as described in Section 7.1.2.

Table 2. The performance gap between training and testing of the target model.  Model | Train Accuracy | Test Accuracy
---|---|---
LSTM-based | 0.707 | 0.540
CodeGPT | 0.767 | 0.740
|  |

####  7.1.1. Overfitting of Target Code Completion Models

The difference in the predictive performance of the target code completion
model for member and non-member data is precisely the underlying principle of
all membership inference method (Shokri et al., 2017), and the overfitting
properties of the models are a visualization of this difference. We assess the
degree of overfitting in the target code completion models by comparing their
training and testing accuracies. Table 2 shows the performance gap between
training and testing, when targeting at the LSTM-based and CodeGPT models. Our
experiments reveal that the LSTM-based model exhibits a substantial 16.7% gap
between training and testing accuracy, signifying poor generalization and a
pronounced overfitting issue. Conversely, the CodeGPT model displays an
impressively minimal train-test accuracy gap of under 3%, underscoring its
exceptional generalization and low susceptibility to overfitting which is one
of the reasons why the membership inference approach on CodeGPT is not as good
as that on LSTM-based model. This is particularly prominent in the LLMs of
code such as CodeGen and StarCoder , which are trained on large-scale corpora
with only 1 epoch, leading to their generalization very well. This leads to
the fact that the effectiveness of membership inference attacks on LLMs of
code needs to be improved.

][c].24

(a) Top (LSTM-based)

][c].24

(b) Tail (LSTM-based)

][c].24

(c) Top (CodeGPT)

][c].24

(d) Tail (CodeGPT)

][c].24

(e) Top (CodeGen)

][c].24

(f) Tail (CodeGen)

][c].24

(g) Top (StarCoder)

][c].24

(h) Tail (StarCoder)

Figure 11. The histograms of log probabilities of tokens generated by the
target neural code completion models (LSTM-based, CodeGPT, CodeGen and
StarCoder). “Top” refers to histograms for the top 20% most frequent tokens.
“Tail” refers to the histograms for the rest.

####  7.1.2. Model’s Memory for Low-frequency Words

Another factor in the effectiveness of CodeMI is the counterfactual
memorization of the target models i.e. the memory for low frequency tokens.
The model’s specific memory for low-frequency training data will be
exemplified in the following two perspectives.

Token Frequency v.s. Predicted Probability. As previously discussed in Section
3.2, our target code completion models employ a loss function based on
negative log-likelihood. This loss function is designed to compel the model to
commit to memory the sequences encountered in the training data. In Figure 11,
we present histograms depicting the logarithmic probability distribution of
tokens generated by these neural code completion models. Our statistical
analysis led us to designate the code tokens appearing in the top 20%
frequency range in both the training and test datasets as “Top-frequency”
tokens, constituting 89.05%, 96.58%, 95.86%, and 95.16% of the training
datasets for LSTM-based, CodeGPT, CodeGen, and StarCoder, respectively.
Conversely, all other code tokens are categorized as “Tail-frequency”. This
figures underscores the fact that the code completion models consistently
favor generating tokens from the set of code tokens with the highest
frequency.

Moreover, we can deduce that when considering the disparity between the
logarithmic probability distributions of the most frequently occurring tokens
in both the training and test datasets, LSTM-based model exhibits a more
pronounced dissimilarity compared to CodeGPT and CodeGen. Nevertheless, it is
important to note that this dissimilarity, in the grand scheme of things,
remains relatively modest. Furthermore, when examining the LSTM-based model, a
more conspicuous variation becomes apparent in the logarithmic probability
distributions of the less common tokens within the training and test datasets.
This variance implies that the LSTM-based model assigns higher probabilities
to tokens within the training dataset, creating a robust signal that could
potentially be exploited for membership inference. However, in the case of the
CodeGPT and CodeGen models, the disparity between the logarithmic probability
distributions of less frequent tokens in the training and test datasets is
comparatively slight. This discrepancy contributes to the less effective
performance of membership inference for CodeGPT and CodeGen when contrasted
with the LSTM-based model. As for StarCoder, the difference in the logarithmic
probability distributions between the training and test datasets is less
pronounced. This observation might be a key factor contributing to StarCoder’s
suboptimal membership inference performance.

Token Frequency v.s. Predicted Rank. By committing the training sequence to
memory, the model elevates the importance of tokens within that sequence when
determining their relative ranking among candidate tokens in the output
vocabulary. Figure 12 illustrates the relationship between a token’s rank in
the training corpus’s frequency table and its position in the model’s
predictions. A lower rank number signifies a higher token rank in the
vocabulary, indicating greater frequency in the corpus or greater ease of
prediction by the model. In the case of these two models, less common tokens
exhibit a notably wider gap between their model-predicted ranks when they
appear in a training sequence versus a test sequence compared to more common
tokens. This phenomenon partly accounts for why the performance of the
membership inference approach improves as the model’s output size increases
(cf. Section 6.3). With a larger output size, the model’s output encompasses a
greater number of low-frequency tokens, offering a more comprehensive
reflection of the code completion model’s memory concerning the training
sequence.

][c].24

(a) LSTM-based

][c].24

(b) CodeGPT

][c].24

(c) CodeGen

][c].24

(d) StarCoder

Figure 12.  The ranks of tokens in the frequency table of the training corpus
and in the predictions generated by the models (where a lower rank signifies a
higher probability for the token).

###  7.2. Further Discussions on Memorization Issues of LLMs

s Recent LLMs trained on extensive corpora exhibit exceptional generalization
capabilities, effectively mitigating the risk of overfitting. Consequently,
this characteristic poses a unique challenge for our membership inference
approach. However, as highlighted in studies (Carlini et al., 2022, 2021;
Zhang et al., 2023), even LLMs that demonstrate proficient generalization may
inadvertently memorize and subsequently disclose elements of their training
dataset. This phenomenon of memorization can be categorized into two distinct
types: conventional memorization, which involves the retention of frequently
occurring near-duplicate texts within a corpus, and counterfactual
memorization, which pertains to the memorization of infrequent details from a
particular training text (Zhang et al., 2023). Carlini et al. (2022) explored
the issue of memorization in LLMs, which falls into the former category of
memorization types. Their study highlights a significant correlation:
memorization becomes notably more pronounced in models with larger parameter
sizes. Additionally, they identify another form of memorization characterized
by the model’s ability to retain information about low-frequency tokens, as
previously described. These two aspects of memorization capacity underscore
the feasibility of membership inference on LLMs.

###  7.3. Threats to Validity

There are several validity threats pertinent to our research.

On the ensurance of non-member dataset. The primary challenge to validity lies
in our inability to ensure that the non-member datasets utilized for CodeGen
and StarCoder do not contain any elements from the training data. Although
extensive preprocessing measures have been implemented (outlined in Section
5.1), the inherent duplication and code reusability could introduce bias into
the results of membership inference.

Limited target code completion models. Another threat to validity pertains to
the limited target code completion models we have investigated. In this paper,
we validate our membership inference approach on two typical code completion
models (i.e., LSTM-based model and CodeGPT) and two LLMs of code (i.e.,
CodeGen and StarCoder). We believe that our membership inference approach can
be easily extended to other code completion and code generation models, e.g.,
ChatGPT (cha, 2022), Code Llama (Rozière et al., 2023). We leave the
validation on other code models to future work.

Limited dataset for evaluation. The third potential threat to validity arises
from the fact that, in our experiments only the Python dataset (i.e., Py150,
CodeSearchNet and Github-Python) is adopted for evaluation. In our future
work, we will evaluate our method on more datasets of code in different
programming languages.

##  8\. Related Work

We review previous research from the perspectives of neural code completion,
membership inference and dataset watermarking.

###  8.1. Neural Code Completion

Deep learning approaches have been widely used by researchers recently for
code completion (Raychev et al., 2014; Aye and Kaiser, 2020; Huangzhao et al.,
2024). The first attempt to use neural language models in conjunction with
program analysis to improve code completion was done by Raychev et al. (2014).
Through program analysis, it first extracts the abstract histories of
programs, and then it uses an RNN-based neural language model to learn the
probabilities of histories. Similar to this, a number of researches (Liu et
al., 2016; Li et al., 2018; Svyatkovskiy et al., 2019) use an RNN-based neural
language model to infer the next code token over the incomplete AST after
first traversing it in depth-first order. Kim et al. (2021) proposed feeding
the ASTs to Transformers in order to predict the missing partial code and
better describe the code structure. A structural model for code completion was
introduced by Alon et al. (2020). It depicts code by sampling pathways from an
incomplete AST. Additionally, Wang and Li (2021) suggested utilizing Gated
Graph Neural Networks (Li et al., 2015) to represent the flattened sequence of
an AST by parsing it into a graph. IntelliCode Compose, a pre-trained language
model of code based on GPT-2, was proposed by Svyatkovskiy et al. (2020) and
offers rapid code completion across a variety of programming languages. A
multi-task learning framework that integrates the code completion and type
inference tasks into a single overall framework was suggested by Liu et al.
(2020a, b). A retrieval-augmented code completion technique was presented by
Lu et al. (2022). It gathers comparable code snippets from a code corpus and
utilizes them as external context. Recently, many pre-trained LLMs have
showcased remarkable performance in code completion, including ChatGPT (cha,
2022), CodeGen (Nijkamp et al., 2023), CodeT5+ (Wang et al., 2023), Code Llama
(Rozière et al., 2023), and StarCoder (Li et al., 2023).

###  8.2. Membership Inference

Membership inference attacks (MIAs) against machine learning models endeavor
to discern whether a given data sample was utilized in the training of a
target model or not. Membership inference methods can be broadly divided into
two categories, i.e., shadow model training and metric-based techniques (Hu et
al., 2022b). Shadow model training (Shokri et al., 2017) aims to train a
binary attack classifier by creating multiple shadow models to mimic the
behavior of the target model. Metric-based techniques (Yeom et al., 2018; Song
and Mittal, 2021; Salem et al., 2019a) proposed to infer the membership of a
given data record by comparing the metric value (e.g., prediction loss (Yeom
et al., 2018)) calculated on the prediction vector of the given record to a
preset threshold. Since both techniques are formulated from the attackers’
standpoint, they assume the presence of abundant prior information, including
knowledge of training data distribution and target model architectures. This
assumption renders the direct application of these techniques unfeasible for
data owners aiming to implement membership inference. Recently, Hu et al.
(2022a) leveraged backdoor techniques to allow the data owner to achieve
membership inference effectively without such prior information. Their
approach enables individuals to carry out membership inference solely by
utilizing black-box query access to the target model. One related work to ours
is (Yang et al., 2023), which also investigates the membership leakage risks
of code models through membership inference and has been released as a
preprint. However, our work diverges from this study in three key aspects.
First, in experiments, we extend the scope beyond small code models, delving
into the feasibility of implementing membership inference on LLMs such as
CodeGen and StarCoder to safeguard code intellectual property. Second, instead
of relying on prediction probabilities, we utilize the distribution of ranks
of ground truth in the model output as the feature for the membership
classifier to learn. Third, we conduct a comprehensive analysis to scrutinize
the factors contributing to the effectiveness of our membership inference
method.

###  8.3. Dataset Watermarking

Watermarking is a popular technique used for protecting the intellectual
property (IP) of datasets (Singh and Chadha, 2013). Many digital watermarking
techniques (Haddad et al., 2020; Wang et al., 2021; Guan et al., 2022) have
been proposed to preclude unauthorized users from utilizing the protected
data. Mothi and Karthikeyan (2019) introduced an innovative hybrid
watermarking strategy that seamlessly merges visual cryptography with the
Wavelet Packet Transform (WPT). This approach not only enables image
segmentation but also facilitates the identification of the lowest energy band
where the watermark is embedded. Furthermore, aside from safeguarding the
datasets, it is imperative to recognize that in the era of high-performing
deep learning techniques, the trained models themselves should be regarded as
intellectual property (IP) owned by their creators and accorded appropriate
protection (Li et al., 2021). Most recent works (Nagai et al., 2018; Qiao et
al., 2023) aimed to plug the watermarks into the deep neural networks for easy
authorization. However, unlike the above two directions, only a few prior
works have started to protect illegal open-sourced data usage from deep neural
network training. Recently, Sun et al. (2022) proposed an approach to protect
the copyright of open-source code via data poisoning. Sun et al. (2023)
proposed CodeMark, to embed user-defined imperceptible watermarks into code
datasets based on adaptive semantic-preserving transformations, to trace their
usage in training neural code completion models.

##  9\. Conclusion

This paper aims to explore the legal and ethical issues of current neural code
completion models, which have not been systematically studied previously. We
have put forward a membership inference approach. In particular, a series of
shadow models are trained to emulate the function of the target code
completion model. The acquired posteriors from these shadow models are
subsequently employed to train a membership classifier. This classifier can
then deduce the inclusion status of a given code sample by analyzing the
output produced by the target code completion model. Extensive experiments are
carried out to measure the effectiveness of our proposed membership inference
approach on two representative relatively small-scale neural code completion
models (i.e., LSTM-based and CodeGPT) and two large-scale language models of
code (i.e., CodeGen and StarCoder). The experimental results demonstrate the
capability of our membership inference approach to accurately ascertain
whether a particular code sample is a part of the model’s training dataset,
with an Accuracy of 0.842 and 0.730 targeting LSTM-based model and CodeGPT,
respectively. As for large language models of code with stronger
generalization capabilities, membership inference methods still need to be
further explored.

Data Availability. All experimental data and source code used in this paper
are available at https://github.com/CGCL-
codes/naturalcc/tree/main/examples/code-membership-inference (Wan et al.,
2022).

## References

  * (1)
  * git (2022) 2022\.  About Github Copilot telemetry.  https:// tinyurl.com/37w8nfnz.  [Online; accessed 1-Sep-2023].
  * cha (2022) 2022\.  ChatGPT.  https://openai.com/blog/chatgpt/.  [Online; accessed 1-Dec-2022].
  * cod (2022) 2022\.  Codex.  https://openai.com/blog/openai-codex/.  [Online; accessed 1-Dec-2022].
  * cop (2022) 2022\.  Copilot.  https://github.com/features/copilot/.  [Online; accessed 1-Dec-2022].
  * int (2022) 2022\.  IntelliCode.  https://visualstudio.microsoft.com/services/intellicode/.  [Online; accessed 1-Dec-2022].
  * tab (2022) 2022\.  Tabnine.  https://www.tabnine.com.  [Online; accessed 1-Dec-2022].
  * Alon et al. (2020) Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020.  Structural Language Models of Code. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ _(Proceedings of Machine Learning Research, Vol. 119)_. PMLR, 245–256.
  * Aye and Kaiser (2020) Gareth Ari Aye and Gail E Kaiser. 2020.  Sequence model design for code completion in the modern IDE.  _arXiv:2004.05249_ (2020).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, et al. 2020\.  Language models are few-shot learners.  _NeurIPS_ 33 (2020), 1877–1901.
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_ (2022).
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. 2021\.  Extracting Training Data from Large Language Models.. In _USENIX Security Symposium_ , Vol. 6.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Guan et al. (2022) Zhenyu Guan, Junpeng Jing, Xin Deng, Mai Xu, Lai Jiang, Zhou Zhang, and Yipeng Li. 2022.  DeepMIH: Deep invertible network for multiple image hiding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 1 (2022), 372–390.
  * Haddad et al. (2020) Sahar Haddad, Gouenou Coatrieux, Alexandre Moreau-Gaudry, and Michel Cozic. 2020.  Joint watermarking-encryption-JPEG-LS for medical image reliability control in encrypted and compressed domains.  _IEEE Transactions on Information Forensics and Security_ 15 (2020), 2556–2569.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hu et al. (2022a) Hongsheng Hu, Zoran Salcic, Gillian Dobbie, Chen Jinjun, Lichao Sun, and Xuyun Zhang. 2022a.  Membership inference via backdooring. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence_.
  * Hu et al. (2022b) Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. 2022b.  Membership inference attacks on machine learning: A survey.  _ACM Computing Surveys (CSUR)_ 54, 11s (2022), 1–37.
  * Huangzhao et al. (2024) Zhang Huangzhao, Zhang Kechi, Li Zhuo, Li Jia, Li Yongmin, Zhao Yunfei, Zhu Yuqi, Liu Fang, Li Ge, and Jin Zhi. 2024.  Deep Learning for Code Generation: A Survey.  _SCIENCE CHINA Information Sciences_ ISSN 1674-733X (2024).
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.  _CoRR_ abs/1909.09436 (2019).
  * Kim et al. (2021) Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021.  Code Prediction by Feeding Trees to Transformers. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. 150–162.  https://doi.org/10.1109/ICSE43902.2021.00026
  * Kocetkov et al. (2022) Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2022.  The Stack: 3 TB of permissively licensed source code.  _Preprint_ (2022).
  * Li et al. (2018) Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018.  Code Completion with Neural Attention and Pointer Networks. In _IJCAI_. 4159–4165.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023.  StarCoder: may the source be with you!  arXiv:2305.06161 [cs.CL]
  * Li et al. (2015) Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015.  Gated graph sequence neural networks.  _arXiv preprint arXiv:1511.05493_ (2015).
  * Li et al. (2021) Yue Li, Hongxia Wang, and Mauro Barni. 2021.  A survey of deep neural network watermarking techniques.  _Neurocomputing_ 461 (2021), 171–193.
  * Liu et al. (2016) Chang Liu, Xin Wang, Richard Shin, Joseph E Gonzalez, and Dawn Song. 2016.  Neural code completion.  (2016).
  * Liu et al. (2020a) Fang Liu, Ge Li, Bolin Wei, Xin Xia, Zhiyi Fu, and Zhi Jin. 2020a.  A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning. In _ICPC_. 37–47.
  * Liu et al. (2020b) Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. 2020b.  Multi-task learning based pre-trained language model for code completion. In _ASE_. 473–485.
  * Lu et al. (2022) Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022.  ReACC: A Retrieval-Augmented Code Completion Framework. In _ACL_. 6227–6240.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021\.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  _arXiv preprint arXiv:2102.04664_ (2021).
  * Meli et al. (2019) Michael Meli, Matthew R. McNiece, and Bradley Reaves. 2019.  How Bad Can It Git? Characterizing Secret Leakage in Public GitHub Repositories. In _26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019_.
  * Mothi and Karthikeyan (2019) R Mothi and M Karthikeyan. 2019.  Protection of bio medical iris image using watermarking and cryptography with WPT.  _Measurement_ 136 (2019), 67–73.
  * Nagai et al. (2018) Yuki Nagai, Yusuke Uchida, Shigeyuki Sakazawa, and Shin’ichi Satoh. 2018.  Digital watermarking for deep neural networks.  _International Journal of Multimedia Information Retrieval_ 7 (2018), 3–16.
  * Nijkamp et al. (2023) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023.  CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.  arXiv:2203.13474 [cs.LG]
  * Niu et al. (2023) Liang Niu, Shujaat Mirza, Zayd Maradni, and Christina Pöpper. 2023.  $\\{$CodexLeaks$\\}$: Privacy Leaks from Code Generation Language Models in $\\{$GitHub$\\}$ Copilot. In _32nd USENIX Security Symposium (USENIX Security 23)_. 2133–2150.
  * Qiao et al. (2023) Tong Qiao, Yuyan Ma, Ning Zheng, Hanzhou Wu, Yanli Chen, Ming Xu, and Xiangyang Luo. 2023.  A Novel Model Watermarking for Protecting Generative Adversarial Network.  _Computers & Security_ (2023), 103102.
  * Raychev et al. (2014) Veselin Raychev, Martin T. Vechev, and Eran Yahav. 2014.  Code completion with statistical language models. In _ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’14, Edinburgh, United Kingdom - June 09 - 11, 2014_. ACM, 419–428.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023\.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ (2023).
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023.  Code Llama: Open Foundation Models for Code.  arXiv:2308.12950 [cs.CL]
  * Salem et al. (2019a) Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2019a.  ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In _26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019_.
  * Salem et al. (2019b) Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2019b.  ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In _26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019_. The Internet Society.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership Inference Attacks Against Machine Learning Models. In _2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017_. IEEE Computer Society, 3–18.
  * Singh and Chadha (2013) Prabhishek Singh and Ramneet Singh Chadha. 2013.  A survey of digital watermarking techniques, applications and attacks.  _International Journal of Engineering and Innovative Technology (IJEIT)_ 2, 9 (2013), 165–175.
  * Song and Mittal (2021) Liwei Song and Prateek Mittal. 2021.  Systematic Evaluation of Privacy Risks of Machine Learning Models. In _30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021_ , Michael D. Bailey and Rachel Greenstadt (Eds.). USENIX Association, 2615–2632.
  * Sun et al. (2023) Zhensu Sun, Xiaoning Du, Fu Song, and Li Li. 2023.  CodeMark: Imperceptible Watermarking for Code Datasets against Neural Code Completion Models.  _arXiv preprint arXiv:2308.14401_ (2023).
  * Sun et al. (2022) Zhensu Sun, Xiaoning Du, Fu Song, Mingze Ni, and Li Li. 2022.  Coprotector: Protect open-source code against unauthorized training usage with data poisoning. In _Proceedings of the ACM Web Conference 2022_. 652–660.
  * Svyatkovskiy et al. (2020) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.  Intellicode compose: Code generation using transformer. In _ESEC/FSE_. 1433–1443.
  * Svyatkovskiy et al. (2019) Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019.  Pythia: Ai-assisted code completion system. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2727–2735.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Proceedings of Advances in neural information processing systems_. 5998–6008.
  * Wan et al. (2022) Yao Wan, Yang He, Zhangqian Bi, Jianguo Zhang, Yulei Sui, Hongyu Zhang, Kazuma Hashimoto, Hai Jin, Guandong Xu, Caiming Xiong, and Philip S. Yu. 2022.  NaturalCC: An Open-Source Toolkit for Code Intelligence. In _Proceedings of 44th International Conference on Software Engineering, Companion Volume_. ACM.
  * Wang et al. (2021) Run Wang, Felix Juefei-Xu, Meng Luo, Yang Liu, and Lina Wang. 2021.  Faketagger: Robust safeguards against deepfake dissemination via provenance tracking. In _Proceedings of the 29th ACM International Conference on Multimedia_. 3546–3555.
  * Wang et al. (2023) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023.  CodeT5+: Open Code Large Language Models for Code Understanding and Generation.  arXiv:2305.07922 [cs.CL]
  * Wang and Li (2021) Yanlin Wang and Hui Li. 2021.  Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_. AAAI Press, 14015–14023.  https://doi.org/10.1609/aaai.v35i16.17650
  * Xu et al. (2022) Frank F. Xu, Uri Alon, Graham Neubig, and Vincent J. Hellendoorn. 2022.  A Systematic Evaluation of Large Language Models of Code.  arXiv:2202.13169 [cs.PL]
  * Yang et al. (2023) Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsum Kim, Donggyun Han, and David Lo. 2023.  Gotcha! this model uses my code! evaluating membership leakage risks in code models.  _arXiv preprint arXiv:2310.01166_ (2023).
  * Yang et al. (2024) Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. 2024.  Unveiling Memorization in Code Models. In _2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE)_. IEEE Computer Society, 856–856.
  * Yeom et al. (2018) Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018.  Privacy risk in machine learning: Analyzing the connection to overfitting. In _2018 IEEE 31st computer security foundations symposium (CSF)_. IEEE, 268–282.
  * Zhang et al. (2023) Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2023.  Counterfactual Memorization in Neural Language Models.  arXiv:2112.12938 [cs.CL]