# On Inter-dataset Code Duplication and
Data Leakage in Large Language Models

José Antonio Hernández López, Boqi Chen, Tushar Sharma, Dániel Varró J.A.
Hernández López and Dániel Varró are with IDA, Linköping University, Sweden.
E-mail: {jose.antonio.hernandez.lopez<EMAIL_ADDRESS>Tushar Sharma is
with Dalhousie University, Canada.
E-mail<EMAIL_ADDRESS>Boqi Chen is with McGill University, Canada.
E-mail<EMAIL_ADDRESS>
###### Abstract

Motivation. Large language models (LLMs) have exhibited remarkable proficiency
in diverse software engineering (SE) tasks, such as code summarization, code
translation, and code search. Handling such tasks typically involves acquiring
foundational coding knowledge on large, general-purpose datasets during a pre-
training phase, and subsequently refining on smaller, task-specific datasets
as part of a fine-tuning phase.

Problem statement. Data leakage i.e., using information of the test set to
perform the model training, is a well-known issue in training of machine
learning models. A manifestation of this issue is the intersection of the
training and testing splits. While intra-dataset code duplication examines
this intersection within a given dataset and has been addressed in prior
research, inter-dataset code duplication, which gauges the overlap between
different datasets, remains largely unexplored. If this phenomenon exists, it
could compromise the integrity of LLM evaluations because of the inclusion of
fine-tuning test samples that were already encountered during pre-training,
resulting in inflated performance metrics.

Contribution. This paper explores the phenomenon of inter-dataset code
duplication and its impact on evaluating LLMs across diverse SE tasks.

Study design. We conduct an empirical study using the CodeSearchNet dataset
(csn), a widely adopted pre-training dataset, and five fine-tuning datasets
used for various SE tasks. We first identify the intersection between the pre-
training and fine-tuning datasets using a deduplication process. Then, we
fine-tune four models pre-trained on csn (CodeT5, CodeBERT, GraphCodeBERT, and
UnixCoder) to evaluate their performance on samples encountered during pre-
training and those unseen during that phase.

Results. Our findings reveal a potential threat to the evaluation of various
LLMs across multiple SE tasks, stemming from the inter-dataset code
duplication phenomenon. Moreover, we demonstrate that this threat is
accentuated by factors like the LLM’s size and the chosen fine-tuning
technique. Based on our findings, we delve into prior research that may be
susceptible to this threat. Additionally, we offer guidance to SE researchers
on strategies to prevent inter-dataset code duplication.

###### Index Terms:

LLM, inter-dataset code duplication, data leakage

##  1 Introduction

Recent advances in machine learning (ML) have significantly transformed the
landscape of software development. ML has increased the efficiency of software
development by automating various software engineering (SE) tasks [1]. For
instance, ML models have been trained to generate documentation from source
code [2, 3, 4], a task often deemed tedious by software engineers [3]. These
models have also improved code search [5, 6, 7], empowering developers to
leverage existing source code for code reuse or comprehension of functionality
[8]. Furthermore, ML models can be used for the automatic migration of source
code from one programming language to another [9].

In this context, many recent ML solutions rely on large language models (LLMs)
[10, 11, 12, 13], which have exhibited remarkable performance in such tasks.
LLM-based solutions are often realized through a prevalent pre-training/fine-
tuning approach [10, 11, 12, 13]. Firstly, the model is _pre-trained_ using a
large corpus of unlabeled source code. Subsequently, the model is _fine-tuned_
for a specific task using a more focused, smaller dataset. For example,
CodeBERT [10] uses the CodeSearchNet [6] dataset (with $\sim$6.4M code
snippets) for pre-training, and it can be fine-tuned for code search using the
Python-150 dataset [14, 15, 16] (with $\sim$92K code snippets).

However, a significant proportion of the source code used as datasets in
software engineering research originates from a single source i.e., GitHub
[14, 9, 6, 17, 18]. For instance, the datasets in CodeXGlue [9], a widely
utilized benchmark for code understanding, predominantly consist of samples
extracted from GitHub. Moreover, there is a substantial amount of duplication
within GitHub as noted by various researchers [19, 20, 21].

These observations may indicate a potential overlap between the datasets used
for pre-training and fine-tuning in LLM-based solutions when applied to
diverse SE tasks. We denote this phenomenon as _inter-dataset code
duplication_. Therefore, a first objective of this paper is to investigate the
following research question:

To what extent does _inter-dataset code duplication_ or, even worse, data
leakage exist between pre-training and fine-tuning datasets used in LLM-based
solutions for SE?

This phenomenon can be interpreted as a special form of data leakage, where
samples encountered during pre-training are also used in evaluating the fine-
tuned model. As a result, a fine-tuned LLM-based solution may perform better
on samples that have already been seen during pre-training, which influences
(inflates) the overall performance thus compromising the integrity of
performance evaluations. As a consequence, when such solutions are used in
production, the performance of an LLM-based solution might be lower than
expected. Hence, the second objective of this study is to address the
following research question:

Does _inter-dataset code duplication_ pose a threat to the validity of LLMs
evaluations?

To the best of our knowledge, neither of these questions has been
systematically investigated in past research. In fact, a recurring practice
observed when using LLMs for various SE tasks involves pre-training and fine-
tuning with the same dataset (or fine-tuning with a subset of the pre-training
dataset) [13, 10, 11, 12]. Consequently, a positive response to the research
questions could imply that some findings in those works might report overly
optimistic results due to the inter-dataset code duplication phenomenon.

Furthermore, we explore factors that may amplify the threat to validity.
Specifically, we examine the impact of the pre-trained model’s size,
considering 1) the common practice of releasing LLMs in various sizes [13, 22,
23] and 2) the tendency for larger models to better retain knowledge acquired
during pre-training [24]. Additionally, we consider the fine-tuning technique
employed. Beyond full fine-tuning, which modifies all model parameters, we
also assess three lightweight fine-tuning methods: LoRA [25], Prefix-Tuning
[26], and layer freezing. Our rationale for considering these methods is
twofold: 1) they enable fine-tuning with limited resources, making them more
practical in certain cases [26], and 2) they introduce minimal modifications
to the model’s parameters, leading us to hypothesize that their evaluation may
be more susceptible to the inter-dataset code duplication phenomenon.

To address these research questions, we experiment with four LLMs already used
in SE research (namely, CodeT5 [13], CodeBERT [10], GraphCodeBERT [11], and
UnixCoder [12]) pre-trained on the CodeSearchNet dataset [6] and five fine-
tuning datasets (CodeTrans [9], Python-150 [14], TL-CodeSum [17], Funcom [18],
and BigCloneBench [27]) used for heterogeneous SE tasks.

Results. Our results clearly confirm the existence of the duplication
phenomenon, leading to non-trivial overlapping in three fine-tuning datasets.
Additionally, we present strong evidence of a threat to the LLMs evaluations.
Finally, we show that this threat could be accentuated by the size of the LLM
and the fine-tuning technique employed.

Altogether, this work contributes to software engineering research in the
following ways:

  1. 1.

We systematically demonstrate the presence of the inter-dataset code
duplication phenomenon through comprehensive analysis using csn and five
state-of-the-art fine-tuning datasets.

  2. 2.

We provide evidence of a potential threat to the evaluations of various LLMs
due to inter-dataset code duplication.

  3. 3.

We analyze and highlight factors that amplify that risk, taking into account
the size of the pre-trained model and the fine-tuning technique.

Implications. Our findings underscore the importance of carefully considering
the dataset composition in the pre-training and fine-tuning phases of LLM-
based solutions for SE. By acknowledging the potential for data leakage due to
inter-dataset code duplication, researchers can enhance the robustness of
their evaluation methodologies. This can be achieved by systematically
excluding samples from the fine-tuning test set that also exist in the pre-
training set. Additionally, our exploration delves into potential implications
for previous SE research and establishes connections with earlier ML research
pertaining to catastrophic forgetting.

Organization. This paper is structured as follows. Firstly, Sect. 2 overviews
the background and related work. Sect. 3 motivates inter-dataset code
duplication and demonstrates its existence. Sect. 4 evidences the threat to
validity in the evaluation of LLMs arising from this phenomenon. Finally,
Sect. 5 delves into a discussion of potential implications and connections
with prior literature, while Sect. 6 concludes.

##  2 Background & Related Work

In this section, we provide key insights into the application of LLMs in SE.
We delve into aspects related to code duplication and data leakage,
encompassing the definition of code clones, previously proposed clone
detection tools, and the notion of code duplication used in this paper.
Additionally, we discuss previous studies about code duplication on GitHub and
the impact of code duplication in ML for code.

###  2.1 Large Language Models for Software Engineering

####  2.1.1 Pre-training and fine-tuning paradigm

The initial deep learning (DL) models introduced in the field of software
engineering are tailored for specific tasks. In other words, these models are
trained on a given dataset to execute a SE task by minimizing a designated
loss function [2, 3, 4]. However, with the introduction of CodeBERT [10],
there has been a shift in this paradigm towards pre-trained LLMs. These models
undergo initial pre-training on an extensive corpus of source code to acquire
fundamental knowledge about both code and natural language. Subsequently, the
model is fine-tuned to perform a specific task using a smaller dataset. Using
this methodology, LLMs have demonstrated impressive performance across various
SE tasks, including code summarization [13], code search [10], code
translation [9], code clone detection [9], and code generation [22], among
others.

####  2.1.2 Architecture of LLMs

LLMs typically adopt the transformer architecture [28] and can be categorized
into three groups [29]: decoder-only, encoder-only, and encoder-decoder
language models.

Decoder-only language models are normally trained to predict the next token
given the preceding ones. In the context of code, examples of such models
include CodeGPT [9], StarCoder [30], CodeGen [23], etc. These models are
typically used for code generation tasks.

Encoder-only language models are usually trained on the masked language
modeling task which aims to predict a masked word given the rest of the
sequence as context. Examples in this category include CodeBERT [10],
GraphCodeBERT [11], and UnixCoder [12]. These models excel in classification
tasks or tasks requiring feature extraction (such as code search or code clone
detection).

Encoder-decoder language models generate text based on predefined conditions.
Examples include CodeT5 [13], CodeT5+[22], and PLBART[31]. They are intended
for conditional generation tasks like code summarization or generating code
from a natural language description.

This paper focuses only on encoder-only and encoder-decoder language models,
in particular, CodeBERT, GraphCodeBERT, UnixCoder, and CodeT5.

####  2.1.3 Fine-tuning techniques

Fine-tuning methods are essential for LLMs to adapt to specific tasks and have
been widely used in the context of SE. Many LLMs, such as CodeBERT and CodeT5,
have demonstrated effectiveness for tasks such as code search, code
summarization, and code translation [10, 13] using the full fine-tuning
approach. Recently, there has been a growing interest in applying lightweight
fine-tuning for LLMs in SE [32, 33, 34]. These methods achieve performance
comparable to the full fine-tuning method on various tasks and even outperform
it when the dataset is small [35].

This paper focuses on four popular fine-tuning techniques: _full fine-tuning_
[36], _layer freezing_ [37], _low-rank adaptation (LoRA)_ [25], and _prefix
tuning_ [26]. Such techniques have been used in SE research extensively [10,
33, 34, 32].

Full fine-tuning initializes the model with parameters learned from pre-
training and further trains it on the task-specific dataset [36]. This method
is straightforward to implement and tends to perform well with large and
diverse datasets. However, this technique becomes computationally intensive as
the size of the LLM increases.

Layer freezing fine-tuning method involves freezing several of the initial LLM
layers that are typically responsible for extracting features, while updating
the remaining layers during training. Such mechanism results in an efficient
way to fine-tune a model. In many cases, fine-tuning a small subset of the
layers can yield results comparable to those of full fine-tuning [37].

LoRA [25] enhances fine-tuning efficiency by introducing low-rank weight
matrices to pre-trained LLM, rather than updating all weights directly. During
fine-tuning, only these low-rank matrices are updated. This approach also
reduces the storage requirements for the fine-tuned model, as only the low-
rank matrices representing the changes to the original weight matrix need to
be stored.

Prefix tuning [26] is a recent fine-tuning approach inspired by prompting
methods for LLMs. In this approach, continuous task-specific vectors, also
known as prefixes, are prepended to each transformer block of the LLM. These
prefixes are learned during fine-tuning while weights of the LLM remain
frozen. The purpose of prefix tuning is to guide the LLM’s processing towards
a task-specific dataset using these prefixes. This method is especially
beneficial for small datasets, as it effectively prevents overfitting.

In this paper, we conduct experiments with these four fine-tuning techniques
to discern which of them is more impacted by the inter-dataset code
duplication phenomenon.

###  2.2 Code duplication

####  2.2.1 Code clones

In the code clone literature, various types of code clones are identified [38,
39, 21]. Type-1 clones involve identical code snippets with differences in
spaces, comments, whitespaces, or layouts. Type-2 clones are syntactically
equivalent fragments with variations in identifiers, literals, and types,
along with Type-1 differences. Type-3 clones encompass all changes found in
Type-1 and Type-2 clones, with additional modifications like altered, added,
or removed statements. Finally, Type-4 clones represent snippets that share
functionality but have different syntax.

Code clone detection tools rely on the concept of similarity to determine
whether two code fragments are clones or not [40]. Each tool utilizes its
notion of similarity. For instance, SourcererCC [39] measures this similarity
by assessing the overlap between the tokens in two code fragments. In
contrast, Deckard [41] takes into account ASTs and employs a tree edit
distance metric.

####  2.2.2 Notion of code duplication and tool employed

In this paper, we adopt the concept of _code duplication_ as defined by
Allamanis [42], referring to code snippets that are highly similar but not
necessarily identical. That is, we are interested in Type-3 code clones.
Particularly, we use a tool based on based on SourcererCC [39] used by
Allamanis [42] to identify inter-dataset code duplication. We adopt this
concept of code duplication and the associated tool for two main reasons: 1)
the author has conducted a similar study but at an intra-dataset level, and 2)
the tool is highly efficient and fast. The tool reports duplication between
two code fragments if the Jaccard similarities of their two “fingerprints”
cross a pre-defined threshold. The first fingerprint comprises a multi-set of
literals and identifiers, while the other consists of a set of literals and
identifiers.

###  2.3 Code duplication within GitHub

SourcererCC [39] is used by Lopes et al. [19] to perform an extensive study of
code duplication within GitHub [19]. The study analyzes 4.5 million GitHub
projects, encompassing 428M files written in Python, Java, and JavaScript. Key
findings reveal that 70% of code on GitHub consists of clones of previously
created files, and a noteworthy amount of inter-project duplication exists
(i.e., files in one project can be found in another project).

Gharehyazie et al.[20] employs Deckard[41] to identify code clones within 8k
GitHub projects. Employing a statistical and network approach, they conclude
that cross-project cloning is a common practice across various granularities,
ranging from several lines to entire projects. Additionally, they observe an
onion model of cloning within GitHub, where the majority of clones originate
from the same project, followed by projects within the same domain, and then
from projects in different domains.

In an empirical study on the nature of code clones, Golubev and Bryksin [21]
investigate a corpus of 23k open-source Java projects. Their findings indicate
that, in Java, method-level duplication is more prevalent than file-level
duplication. They also highlight the common practice of copying/pasting code
over the years of Java code existence.

Lastly, Yang et al. [43] provide compelling evidence of code migration from
Stack Overflow to GitHub, a phenomenon contributing to code duplication within
GitHub. Using SourcererCC, their study identifies correspondences between 290M
function definitions within GitHub and 1.9M Python snippets from Stack
Overflow.

The extensive body of research presenting a noteworthy occurrence of
duplication within GitHub, whether inter- or intra-repository, along with the
choice of GitHub as the primary source for code snippets in ML datasets,
emphasizes the significance of evaluating _inter-dataset code duplication_.

###  2.4 Code duplication and ML for code

Motivated by this amount of duplication within GitHub, Allamanis [42] conducts
a study on _intra-dataset code duplication_ , specifically examining the
intersection between training and testing splits within code datasets and its
implications for ML models. The study demonstrates that the performances of ML
models are inflated as a result of the non-trivial intersection. Additionally,
the author presents a set of best practices for collecting code corpora and
evaluating machine learning models on them. Subsequently, researchers,
building upon this knowledge, have been aware of duplication issues and
commonly implement deduplication strategies in their datasets [44, 6, 45, 46].
For example, Husain et al.[6] implement a deduplication procedure before
releasing the CodeSearchNet dataset. Karmakar et al.[44] incorporate
deduplication as part of the preprocessing pipeline before releasing JEMMA, a
large-scale, high-quality Java code corpus. Kocetkov et al. [46] release the
deduplicated version of the Stack, a massive 6TB source code corpus,
demonstrating that pre-training code LLMs with a deduplicated version yields
better results.

Our study differs from the Allamanis’s work [42]. While this prior research
focuses on _intra-dataset code duplication_ , our paper explores the _inter-
dataset code duplication_ , which poses a new potential threat to the
evaluation of ML models given the recent shift in learning paradigms (task-
specific training to pre-training/fine-tuning).

###  2.5 Data leakage in LLMs for code

Sallou et al. [47] addresses the issue of data leakage in LLMs for code.
Through interactions with ChatGPT, the authors show that ChatGPT is aware of
multiple samples from the Defects4J dataset [48]. Consequently, Defects4J may
not be a reliable dataset for evaluating ChatGPT. Additionally, Schafer et al.
[49] observe that StarCoder[30], along with other similar models like Codex
[50], exhibits improved performance in test generation when using HumanEval
samples (available at GitHub) compared to SF110 samples (available at
Sourceforge but not at GitHub [47]). Notably, StarCoder was trained on the
Stack [46], a dataset extracted from GitHub.

This paper’s contribution differs from previous works in that it explores data
leakage within a pre-training/fine-tuning framework and analyzes factors that
may exacerbate the threat to the validity of the LLMs evaluations. Previous
literature has primarily demonstrated data leakage by eliciting responses from
LLMs through various prompts.

##  3 Inter-dataset code duplication

In this section, we motivate the need to assess the inter-dataset code
duplication phenomenon. Subsequently, we detail the study design and showcase
the results that demonstrate the existence of inter-dataset code duplication.

###  3.1 Motivation

The inter-dataset code duplication phenomenon refers to the overlapping in
terms of duplicated code samples among code datasets. In conventional ML
scenarios, this phenomenon is not problematic since each dataset is used
independently for training and evaluating ML models. However, issues arise
when employing a pre-training/fine-tuning approach, particularly when
evaluating test samples encountered during pre-training. This scenario has the
potential to inflate performance metrics, thereby compromising the LLM
evaluation and the estimation of its performance in production. To illustrate
this concern further, let us examine the task of generating natural language
descriptions based on a given code snippet (i.e., the code summarization
task). For this task, we opted to fine-tune CodeT5 [13] using the Python-150
dataset [14]. Consider the Python test sample shown in Listing 1.

Listing 1: Example extracted from the Python-150 test set.

⬇

# round a to the nearest integer

# if that integer is within an epsilon of a.

def round_if_near_integer(a, epsilon=0.0001):

if (abs((a - round(a))) <= epsilon):

return round(a)

else:

return a

This test sample is also part of the CodeSearchNet dataset [6], used for pre-
training CodeT5. Therefore, CodeT5 is already familiar with this sample.
Additionally, a pre-training task closely resembling the fine-tuning task (the
bimodal dual generation task [13]) is performed in the pre-training phase of
CodeT5. This example is equally applicable to other models that undergo pre-
training using the CodeSearchNet dataset. For example, the same concern
emerges when examining the UnixCoder model [12] in the context of semantic
code search. Consequently, the performance of these models on such samples
could be inflated, leading to incorrect conclusions in the evaluation of LLMs
and inferior performance of such LLMs in production.

###  3.2 Study design

This section outlines the experimental setup adopted to demonstrate the
presence of this phenomenon. Specifically, we introduce the datasets used and
an explanation of the approach for computing the inter-dataset duplicates.

####  3.2.1 Datasets

Figure 1: Our approach to assessing inter-dataset code duplication.

The datasets analyzed in this paper are the following:

The CodeSearchNet dataset (csn) [6] provides a comprehensive multilingual
dataset with 6.4M code snippets, approximately 2M of which are accompanied by
natural language documentation. The dataset encompasses snippets from six
programming languages, with our focus limited to Java and Python. Several
language models, including CodeT5, CodeBERT, GraphCodeBERT, and UnixCoder, are
pretrained using this dataset.

The CodeTrans dataset [9] serves as a fine-tuning dataset composed by
$\sim$12k pairs of Java and C# methods. This dataset is tailored to code
translation which aims to migrate legacy software from one programming
language in a platform to another programming language.

The Python-150 dataset[14] consists of pairs comprising Python functions and
their corresponding documentation. For our study, we use the version
accessible through NaturalCC [15], encompassing $\sim$92k pairs. Originally
intended for code summarization, this dataset has also been employed in prior
research for the code search task [16].

The TL-CodeSum (TLC) and Funcom (FCM) datasets [17, 18] consist of 87k and 2M
of Java methods and their corresponding documentation, respectively. These
datasets are specifically curated for code summarization.

The BigCloneBench dataset [27] is a comprehensive dataset designed for code
clone detection. It contains a substantial collection of Java code clones,
providing a valuable resource for evaluating the similarity between code
fragments. We use the filtered version available in CodeXGlue[9], including
around 9k code snippets.

####  3.2.2 Approach

Our method for revealing the existence of inter-dataset duplication is
depicted in Fig. 1, including three key steps:

Dataset Acquisition. The datasets are obtained following the train/test/valid
splits established in previous research. For the CodeTrans and BigCloneBench
datasets, we adhere to the splits provided by the CodeXGlue benchmark [9].
Regarding the Python-150 dataset, we use the splits presented in NaturalCC
[15]. For TLC, the original splits are employed [17]. Lastly, for FCM, we
adopt the splits utilized by Shi et al. [51].

Parsing the code snippets. Each code snippet undergoes parsing to extract the
code tokens, excluding comments. We use the Python tokenize module for Python
code as it is included in the Python standard
library111https://docs.python.org/3/library/tokenize.html. For Java, we use
the javalang library222https://github.com/c2nes/javalang, a Python library for
parsing Java source code used in previous SE works [52, 53].

Running the deduplication procedure. After parsing all snippets from the
datasets, a deduplication procedure is executed. We employ the tool developed
by Allamanis [42], which is based on SourcererCC [39]. The thresholds applied
in this study to identify a pair of code snippets as clones adhere to the
recommendations of the tool’s creator [42] ($0.7$ for the multiset and $0.8$
for the set).

The ultimate result of our pipeline is represented by an undirected sparse
graph. In this graph, nodes correspond to individual code snippets, and edges
indicate instances where one code snippet duplicates another. For instance,
the node corresponding to the snippet in Listing 1 is linked to another node
representing its duplicate in the csn dataset. To facilitate subsequent
analysis and research, we provide this graph in the form of an SQLite
database333Available at https://zenodo.org/records/10446176.

###  3.3 Results

Using csn as the pre-training dataset, we compute the inter-dataset
duplication percentage ($IDD_{\mathcal{D}}$) for each fine-tuning dataset
$\mathcal{D}$.

$IDD_{\mathcal{D}}=\frac{|\mathcal{D}\cap\text{{csn}{}}|}{|\mathcal{D}|}\times
100\%$ (1)

In this expression, the numerator represents the count of samples belonging to
$\mathcal{D}$ that contain at least one duplicate in csn (i.e., one of its
neighbors within the undirected graph is part of csn).

The barplot in Fig. 2 illustrates the inter-dataset duplication percentages
for each dataset. Particularly notable are the CodeTrans, Python-150, and TLC
datasets, which display a substantial level of inter-dataset duplication (with
ratios ranging between 13% and 23%) with csn. In contrast, the inter-dataset
duplication percentages for the FCM and BigCloneBench444The
$IDD_{\mathcal{D}}$ for this dataset was calculated based on the collection of
unique code samples. datasets are notably below 5%.

Our results reveal notable duplication percentages, ranging from 13% to 23%,
in three datasets (TLC, Python-150, and CodeTrans) with respect to csn. This
observation confirms the presence of the inter-dataset code duplication
phenomenon in datasets employed for LLM-based solutions.

Figure 2: Inter-dataset duplication with respect to csn for each dataset.

##  4 The effect of the inter-dataset duplication on LLMs evaluation

The existence of the inter-dataset duplication phenomenon, specifically data
leakage, implies the incorporation of samples from the pre-training phase into
the test set of fine-tuning datasets. This could compromise the evaluation of
the LLMs as the fine-tuned model may perform better on those samples. As a
result, this section investigates the potential impact of such data leakage on
the evaluation of LLMs in various SE tasks.

###  4.1 Study design

In this section, we introduce the code tasks used in the study, followed by
the LLMs under consideration. Subsequently, we explain the approach employed
to illustrate the presence of the threat and finally, we present experimental
settings.

####  4.1.1 Datasets, code tasks, and evaluation metrics

We employ three datasets with a substantial inter-dataset duplication
percentage to reliably measure the impact: CodeTrans, Python-150, and TLC.
Table I shows the duplication percentage with respect to csn for each dataset
as well as the duplication percentage within the test set concerning csn, and
the associated tasks. Consequently, the targeted code tasks in this paper are
listed below.

  * •

_Code translation_. Employing CodeTrans as the target dataset, we address the
Java-to-C# code translation task. The evaluation metric used is the smoothed
BLEU score [54], as utilized in prior studies [9]. This metric takes values in
the range of $[0,100]$ (the higher the better) and measures the similarity of
the generated code to the ground truth code through $n$-gram (contiguous
sequences of $n$ tokens) overlapping.

  * •

_Code summarization_. Utilizing the Python-150 and TLC datasets as the target
datasets, we tackle the task of generating natural language descriptions based
on given code snippets. The chosen evaluation metric is again the smoothed
BLEU score [54] between the generated summary and the ground truth; it is a
commonly employed metric for evaluating the code summarization task [10, 9].

  * •

_Code search_. Using the Python-150 dataset, we address the challenge of
retrieving code snippets that match the intent of a natural language query.
For this task, our evaluation metric is the Mean Reciprocal Rank (MRR), a
metric used in previous work to evaluate the performance of code retrieval
systems [9, 10, 6]. This metric is computed as the average of the reciprocal
ranks (inverse of the position of the retrieved code snippet in the ranked
list) and takes values in the range of $[0,100]$ (the higher the better). We
compute this metric using the ground truth snippet and 999 distractor code
snippets [6].

TABLE I: Percentages of inter-dataset duplication in comparison to csn for
both the entire dataset and the test split and the SE tasks enabled.

Dataset | % full dataset | % test set |  SE tasks
---|---|---|---
CodeTrans | 22.82% | 22.40% | Code translation
Python-150 | 15.01% | 14.79% | Code search and summarization
TLC | 13.76% | 13.56% | Code summarization

####  4.1.2 Pre-trained Language Models

Since csn is used in our work as the pre-training dataset, our assessment of
potential data leakage is confined to models pre-trained with this dataset.
Consequently, we consider the following models:

For the code translation and code summarization tasks, we evaluate CodeT5
[13], an encoder-decoder transformer model pretrained with csn. CodeT5 is
well-suited for these tasks due to its architecture, naturally designed for
conditional generation tasks.

For the code search task, we consider encoder-only models pre-trained with
csn: CodeBERT [10], GraphCodeBERT [11], and UnixCoder [12]. These models share
a similar architecture with RoBERTa [55] but they are specialized to code-
related tasks.

####  4.1.3 Control models

A naive approach to detect the presence of a potential threat arising from the
inter-dataset duplication phenomenon involves the direct comparison of the
fine-tuned model’s performance on two groups: the group that has seen during
pre-training (hereafter referred to as the biased group) and the rest of the
test set (hereafter referred to as the unbiased group). However, this
straightforward method fails to consider the inherent complexity of the groups
for a given task. It is plausible that the samples in the biased group are
inherently easier than the ones of the unbiased group, introducing a natural
performance gap. Consequently, this could lead to incorrect conclusions,
attributing the gap to the inter-dataset duplication phenomenon.

To address this, our approach introduces the concept of control models. Our
control models are transformer models known to be unaffected by this threat,
such as randomly initialized transformer models or models trained solely with
natural language (instead of source code). These models help to estimate the
complexity of the groups by computing the performance gap between the biased
and unbiased groups. If the performance gap of a specific fine-tuned LLM is
statistically greater than the gaps observed with control models, it provides
evidence that there is a potential threat in the evaluation of this LLM due to
exposure of the biased group during pre-training. Furthermore, the larger the
gap is, the more affected the evaluation is expected to be.

We consider the following control models in this study.

For the tasks of code translation and code summarization, we have taken into
account T5 [56], the natural language variant of CodeT5; T5v1 [57], a modified
iteration of T5 featuring alterations in its architecture and pre-training
tasks; BART [58], a natural language pre-trained encoder-decoder engaged in
the task of reconstructing corrupted text. Additionally, we have included
randomly initialized encoder-decoder models varying in the number of layers.
Specifically, we use randomly initialized encoder-decoder transformers with 6
encoder layers and 6 decoder layers (Rand66), 6 encoder layers and 3 decoder
layers (Rand63), and 3 encoder layers and 3 decoder layers (Rand33).

For the code search task, our list of control models includes BERT [36], an
encoder pre-trained with natural language; RoBERTa [55], the evolutionary step
from BERT and the natural language counterpart of CodeBERT, GraphCodeBERT, and
UnixCoder; multilingual BERT [36] (mBERT), a BERT model pre-trained with a
corpus spanning 104 languages. Additionally, we have incorporated randomly
initialized encoders with variations in the number of layers. Specifically, we
employ three randomly initialized encoders with 1, 3, and 6 layers (Rand1,
Rand3, and Rand6 respectively).

####  4.1.4 Experimental settings

We apply a consistent set of hyper-parameters across all language models,
including learning rate and batch size. While slight adjustments are made in
some models to enhance stability during fine-tuning or accommodate GPU
constraints, an interested reader may find all the hyper-parameters used in
our experiments in our repository555https://github.com/Antolin1/code-inter-
dataset-duplication.

Each model is fine-tuned five times with different seeds. To establish the
distribution of gaps between the biased group and the unbiased one for control
models, we compute the average gaps across all the used seeds. For the
remaining models (those pre-trained with csn), we use the five gaps to conduct
statistical tests (Mann–Whitney U test) against the distribution of the
control models. If the performance gap proves to be statistically greater than
that of the control models (one-tailed test), it indicates evidence of an
evaluation threat caused by the code inter-dataset duplication phenomenon. We
apply a significance level of $\alpha=0.05$.

Note that we have performed (intra-dataset) deduplication on the fine-tuning
datasets to eliminate the potential impact of any train/test intersection in
the results.

###  4.2 Results

####  4.2.1 Does inter-dataset code duplication pose a threat to the validity
of LLMs evaluations?

Due to inter-dataset code duplication, certain samples in the fine-tuning test
set have been part of the pre-training process for the LLM. This study delves
into whether this circumstance poses a threat to the validity of the
evaluation of LLM-based solutions for SE.

Case of code translation. Table II presents the difference between the biased
and the unbiased groups for the control models and CodeT5. Examination of
Table II reveals that, for code translation, the control models exhibit an
average improvement of $28.61$ points in the biased group compared to the
unbiased group. In contrast, CodeT5 shows a performance improvement of $19.98$
points in the biased group compared to the unbiased group. That is, the
performance gap of CodeT5 is relatively smaller compared to those of the
control models. Consequently, the results are inconclusive, and we cannot
firmly claim that there is a threat to the evaluation in this case. Upon a
closer inspection of both groups within the CodeTrans dataset, it becomes
apparent that the biased group is notably easier, in terms of code complexity,
than the unbiased one. To measure the complexity of both groups, we consider
the number of tokens and the cyclomatic complexity [59] of code snippets. The
former is used because code translation involves mapping elements from one
programming language to another, and longer token sequences may indicate more
intricate expressions or statements that require accurate translation. The
latter is used because a higher cyclomatic complexity signifies a greater
number of potential execution paths, which may result in more challenging
translations. Table IV shows that methods in the biased group exhibit
significantly higher length and cyclomatic complexity with a moderate effect
size. Consequently, the control models excel in translating samples from the
biased group but face challenges with the unbiased group. In contrast, CodeT5
performs exceptionally well in the unbiased group and competently in the
biased one. Table IX of Appendix A666Within the same appendix, we include the
performance metrics for both code summarization and code search. presents the
performance of CodeT5 and the control models in both groups.

TABLE II: Average BLEU score gaps ($\pm$ standard deviation) between the biased and the unbiased groups for the control models and CodeT5. Bold values in the CodeT5 column indicates statistical significance (i.e., $p-$value$<0.05$). | Control models | CodeT5
---|---|---
CodeTrans | $28.61\pm 6.82$ | $19.98\pm 0.26$
Python-150 | $-0.55\pm 0.14$ | $\boldsymbol{0.34\pm 0.31}$
TLC | $-0.23\pm 0.41$ | $\boldsymbol{0.69\pm 0.35}$

Case of code summarization. Examining the final two rows in Table II
concerning code summarization tasks, it becomes apparent that the performance
gap is more pronounced, with a $p$-value $<0.05$ in both instances.
Specifically, the control models generally show slightly better performance in
the unbiased group, while the opposite trend is observed in CodeT5, where it
performs better on samples encountered during pre-training.

On the other hand, as demonstrated in Table V, there is no a noticeable
complexity gap between the biased group and the unbiased one (small effect
size) as we observe in code translation. In this context, we consider the
overlapping between the natural language description of the code and the
identifiers within the snippet as a reasonable complexity indicator for the
code summarization task. Ahmed and Devanbu [60] show that identifiers play a
more crucial role than syntax in code summarization tasks. Therefore, we
assume that the greater the match between the identifiers and the actual code
description, the more straightforward it becomes for the LLM to produce the
accurate summary.

Case of code search. Table III indicates that GraphCodeBERT and UnixCoder gaps
are statistically greater than the control models. In particular, the control
models tend to perform similarly on both groups while GraphCodeBERT and
UnixCoder perform better in the biased group than the unbiased one. As
expected, UnixCoder exhibits the most substantial gap. We attribute this to
the fact that UnixCoder was pre-trained with a code fragment representation
learning loss [12], which closely resembles the contrastive code search loss
[6]. In this case, we also use the overlapping between the natural language
description of the code and the identifiers as complexity indicator (Table V).
Code identifiers not only play a crucial role in code summarization but also
in code search. The authors of the CodeXGlue benchmark [9] introduced the
CodeSearchNet AdvTest by selecting a subset of csn and removing all the
identifiers. Upon reviewing the scores on the leaderboard
777https://microsoft.github.io/CodeXGLUE/ and comparing them with the ones
when the original csn test split is used, it becomes evident that the
performance is considerably inferior when identifiers are excluded.

TABLE III: Average MRR gaps ($\pm$ standard deviation) between the biased and
the unbiased groups for the control models and CodeBERT, GraphCodeBERT, and
UnixCoder. Bold values indicates statisical significance (i.e.,
$p-$value$<0.05$).

| Control models | CodeBERT | GraphCodeBERT | UnixCoder
---|---|---|---|---
Python-150 | $0.01\pm 0.42$ | $0.16\pm 0.11$ | $\boldsymbol{0.46\pm 0.25}$ | $\boldsymbol{0.86\pm 0.35}$

TABLE IV: Complexity metrics (average$\pm$standard deviation) of the biased
and unbiased groups in the CodeTrans dataset. The effect sizes (Cohen’s $d$)
are also reported.

| Biased group | Unbiased group | Cohen’s $d$
---|---|---|---
Number of tokens | $25.19\pm 17.10$ | $40.25\pm 40.82$ | $0.48$
Cyclomatic complexity | $1.14\pm 0.89$ | $1.74\pm 1.66$ | $0.45$

TABLE V: Overlapping between the code identifiers and description (average$\pm$standard deviation) of the biased and unbiased groups in the Python-150 and TLC datasets. The effect sizes (Cohen’s $d$) are also reported. | Biased group | Unbiased group | Cohen’s $d$
---|---|---|---
Overlap Python-150 | $0.16\pm 0.15$ | $0.17\pm 0.16$ | $0.03$
Overlap TLC | $0.34\pm 0.16$ | $0.33\pm 0.17$ | $0.06$

We identify notably larger performance disparities in models pre-trained with
csn compared to the control models in datasets for code summarization and code
search. Hence, we deduce that inter-dataset code duplication poses a threat to
the validity of LLM evaluations, particularly in these two SE tasks.

####  4.2.2 What is the effect of the LLM size?

When introducing a new LLM, it is a common practice to release models of
various sizes [13, 22, 23]. In this study, we examine how the model size
influences the threat to the LLMs evaluations caused by inter-dataset code
duplication.

To investigate this, we examine the performance gaps between the biased and
unbiased groups for the three versions of CodeT5 (small, base, and large). We
consider code summarization over the Python-150 and TLC datasets.

Table VI presents the performance gaps for the three models of different
sizes. In both cases, we reject the null hypothesis using the Kruskal–Wallis
test. The posterior two-tailed Mann–Whitney U tests reveal these gap ranks:
CodeT5-small $<$ CodeT5-base $\approx$ CodeT5-large in the case of Python-150
and CodeT5-small $<$ CodeT5-base $<$ CodeT5-large in the case of TLC.

The results highlight more substantial gaps between the biased and unbiased
groups as the model size increases. Consequently, larger models may be more
susceptible to inter-dataset code duplication.

TABLE VI: Average BLEU score gaps ($\pm$ standard deviation) between the biased and the unbiased groups for the three versions of CodeT5. |  CodeT5-small |  CodeT5-base |  CodeT5-large
---|---|---|---
Python-150 | $-0.18\pm 0.10$ | $0.34\pm 0.31$ | $0.20\pm 0.35$
TLC | $-0.06\pm 0.25$ | $0.69\pm 0.35$ | $1.44\pm 0.14$

####  4.2.3 What is the effect of freezing the model?

One way of fine-tuning the models without requiring too many resources is to
freeze parameters of the model [34]. Our hypothesis is that this technique is
prone to the threat of inter-dataset code duplication since it involves no
alterations to the model parameters, allowing the knowledge acquired during
the pre-training phase to remain unchanged.

To test this hypothesis, we analyze the performance gaps between the biased
and unbiased groups, concentrating exclusively on encoder-only models. This
choice is based on their simplicity in layer-wise freezing compared to
encoder-decoder models, where intricate interconnections exist between the
encoder and decoder [28]. Specifically, in our experiments, we focus on the
code search task and examine the UnixCoder and GraphCodeBERT models (since
they raise significant differences with respect the control models as
demonstrated in Sect. 4.2.1). We freeze the lower $k$ transformer layers,
specifically the $k$ layers nearest to the input, ranging from $1$ to
$11$888UnixCoder and GraphCodeBERT consist of 12 transformer layers, and
quantify the performance gaps between the two test groups (biased and
unbiased). Fig. 3 illustrates the correlation plot for both models, revealing
a positive correlation. The observed correlation (Spearman) coefficients are
$0.92$ and $0.79$ for UnixCoder and GraphCodeBERT respectively ($p$-value
$<0.05$).

Figure 3: Correlation plots for UnixCoder and GraphCodeBERT models. The
$x-$axis is the number of frozen layers and the $y-$axis is the performance
gap between the biased group and the unbiased one.

A positive correlation is observed between the performance gap and the number
of frozen layers. Consequently, freezing more parameters makes the LLM more
vulnerable to inter-dataset code duplication.

####  4.2.4 What is the effect of using other fine-tuning techniques?

Beyond employing the layer freezing technique, alternative methods have been
introduced to fine-tune LLMs effectively. Two noteworthy examples include LoRA
[25], which updates low-rank weight matrices rather than directly updating all
weights, and prefix tuning [26], which involves freezing the model and
updating continuous task-specific vectors introduced at the beginning of the
input. Driven by the observations of a negative correlation between the number
of modified parameters during fine-tuning and LLMs evaluation threat, we
hypothesize that these techniques which introduce minimal modifications to
model parameters may also be more susceptible to this threat.

To examine this hypothesis, we perform experiments by fine-tuning
CodeT5-large, UnixCoder, and GraphCodeBERT using LoRA and prefix tuning. We
then compare the outcomes with the results of the full fine-tuning approach,
specifically focusing on the code summarization and code search tasks. We
deliberately choose models and tasks that demonstrate substantial distinctions
compared to the control models. Opting for CodeT5-large instead of base or
small models is intentional, as lightweight fine-tuning approaches are thought
to be used with larger models.

Tables VII and VIII present the average BLEU scores and MRR gaps for code
summarization and code search, respectively, for each fine-tuned version of
CodeT5-large, UnixCoder, and GraphCodeBERT. The results reveal a single
instance (code summarization with TLC) where the use of lightweight fine-
tuning approaches does not exacerbate the threat compared to the full fine-
tuning approach. But in the remaining scenarios, we note higher performance
gaps.

In most scenarios (3/4), using LoRA and prefix tuning results in higher
performance gaps. This indicates that lightweight fine-tuning approaches
introduce more bias toward duplicated data points. Consequently, we infer that
these approaches are more likely to be affected by the risk associated with
the inter-dataset code duplication phenomenon.

TABLE VII: Average BLEU score gaps ($\pm$ standard deviation) between full
fine-tuned CodeT5-large and lightweight fine-tuning approaches in the code
summarization tasks.

| Full fine-tuning | LoRA | Prefix
---|---|---|---
Python-150 | $0.34\pm 0.31$ | $\boldsymbol{1.40\pm 0.19}$ | $\boldsymbol{1.54\pm 0.10}$
TLC | $1.44\pm 0.14$ | $0.34\pm 0.30$ | $0.31\pm 0.24$

TABLE VIII: Average MRR gaps ($\pm$ standard deviation) between full fine-
tuned GraphCodeBERT and UnixCoder, and lightweight fine-tuning approaches in
the code search task.

| Full fine-tuning | LoRA | Prefix
---|---|---|---
GraphCodeBERT | $0.46\pm 0.25$ | $\boldsymbol{1.14\pm 0.30}$ | $\boldsymbol{1.33\pm 0.51}$
UnixCoder | $0.86\pm 0.35$ | $\boldsymbol{2.21\pm 0.26}$ | $\boldsymbol{1.31\pm 0.16}$

##  5 Discussion

This section discusses key implications of the study and outlines the
connection between the effects of inter-dataset duplication on neural networks
with catastrophic forgetting. In addition, we also discuss aspects that may
pose risks to the experiment’s validity.

###  5.1 Key implications

In this work, we demonstrate the presence of the inter-dataset code
duplication phenomenon. Additionally, we find that this phenomenon poses a
threat to the evaluation across various LLMs, tasks, and datasets.
Consequently, we summarize the main takeaways from this paper below.

Takeaway 1. Always assess the overlap between the pre-training and the fine-
tuning datasets.

We have illustrated a non-trivial overlap in several fine-tuning datasets with
a widely used pre-training dataset. It is logical to assume that larger pre-
training datasets are more likely to have increased overlap with fine-tuning
datasets. In this context, when releasing a new pre-training dataset, it could
be beneficial to provide a tool for determining if a given piece of code is
included in such a dataset. For instance, the Stack dataset [46], with over
6TB of permissively-licensed source code files extracted from GitHub, includes
a tool to determine if a given repository is included in
it999https://huggingface.co/spaces/bigcode/in-the-stack.

Takeaway 2. If overlapping is detected, remove the samples from the testing
set used previously during pre-training to avoid data leakage and achieve
unbiased performance evaluations.

This practice is not widespread. For instance, Zou et al. [16] assesses
generalizability of GraphCodeBERT using datasets distinct from the csn
dataset. However, one of the datasets employed, Python-150, exhibits non-
trivial overlap with csn, as demonstrated in our experiments.

Takeaway 3. Avoid fine-tuning a new LLM with a subset of the pre-training
dataset to ensure a fair comparison with other baselines.

CodeBERT, GraphCodeBERT, UnixCoder, and CodeT5 [10, 11, 12, 13] do not adhere
to the above observation. Upon examining the results presented in our study, a
substantial portion of the performance gaps, beyond surpassing those of the
control models, is consistently greater than zero. This implies that LLMs
generally exhibit improved performance on samples encountered during pre-
training (due to the fact that they have been seen during pre-training).
Consequently, the reported figures in such studies may be inflated, and
comparisons with other LLMs used as baselines could be unfairly skewed. One
way to tackle such issue is to deduplicate the initial (pre-training) dataset
and split it into the pre-training dataset and the fine-tuning one.

Takeway 4. If removing overlapping is not possible, avoid lightweight fine-
tuning approaches and prioritize smaller models.

Our paper highlights that the effect of inter-code duplication is potentially
more pronounced in large models and when employing lightweight fine-tuning
techniques (layer freezing, LoRA, and prefix tuning). Related to the later and
in the context of software engineering, there are studies that explore
lightweight fine-tuning approaches [34, 33], and their reported findings may
be influenced by the insights and findings of the current paper. For instance,
Shi et al. [34] propose an efficient fine-tuning method for pre-trained code
models through layer freezing. In their experiments, the authors fine-tune the
frozen versions of UnixCoder and GraphCodeBERT for code search using csn.
Ayupov and Chirkova [33] investigate lightweight fine-tuning techniques, such
as LoRA, and fine-tune CodeT5 for code summarization, once again employing
csn.

###  5.2 Relation with catastrophic forgetting

Neural networks are susceptible to the phenomenon of catastrophic forgetting,
where training the network for a new task erases knowledge gained from
previous tasks [61]. In the context of pre-training and fine-tuning, a similar
setup is employed. Initially, the model undergoes pre-training on a dataset
with various tasks, followed by fine-tuning on a smaller dataset for a
specific task.

Previous research has demonstrated that mitigating the effects of catastrophic
forgetting can be achieved by 1) increasing the model size [24] and 2)
freezing a significant subset of parameters [62]. Our findings align with
these observations. Specifically, we observe that 1) larger models are more
susceptible to inter-dataset code duplication, implying they retain more
knowledge from the pre-training phase compared to smaller models.
Additionally, 2) when employing lightweight fine-tuning approaches such as
freezing the model, LoRA, or prefix tuning, the models are more prone to the
inter-dataset code duplication phenomenon.

###  5.3 Threats to validity

####  5.3.1 Internal validity

LLM fine-tuning can be non-deterministic due to the random initialization and
shuffling. Consequently, the evaluation result for each fine-tuning setup may
differ. To minimize this effect, we conduct five fine-tuning procedures for
the same setup, each with different random seeds.

####  5.3.2 External validity

We identify three primary external threats to validity. The first pertains to
the selection of control models. Different control models may result in
different control performance gaps for a particular task. We mitigate the
issue by choosing a set of six control models for each task. Three models are
drawn from the NLP literature (e.g., RoBERTa [55], T5 [56], BART [58]), which
has also been utilized in the SE domain [9, 10, 15]. The remaining models are
randomly initialized transformer models with a varied, significant hyper-
parameter (i.e., the number of layers). The second threat is associated with
the limited consideration of programming languages in this paper, specifically
Java and Python. While we hypothesize that these effects may be applicable to
other programming languages, an exploration of this possibility is deferred to
future research. Finally, the last threat is related to the how representative
the considered code datasets and code LLMs are. We mitigate the threat by
using csn [6], a widely-used pre-training dataset, along with several LLMs
that are pre-trained on this dataset (GraphCodeBERT [10], GraphCodeBERT [11],
UnixCoder [12], and CodeT5 [13]). Additionally, we incorporate various
representative fine-tuning datasets employed in prior SE research [15, 9, 17,
18].

####  5.3.3 Construct validity

Choosing the performance metrics for assessing performance in downstream tasks
could pose a threat to construct validity. To address this concern, we opt for
MRR and BLEU metrics, which have been employed in prior studies to evaluate
LLMs in the tasks examined in this paper [9, 60, 10, 11, 6].

##  6 Conclusion & Future work

In this paper, we have demonstrated the existence of the inter-dataset code
duplication phenomenon and its potential negative impact on the reliability of
LLMs evaluations. Furthermore, by experimenting with several popular LLMs, we
conclude that this impact is accentuated by the size of the LLM and the chosen
fine-tuning technique. Specifically, larger models are more susceptible to
this impact, and lightweight fine-tuning techniques are also more vulnerable
to the identified threat.

In future work, we aim to extend this study to include additional programming
languages and SE tasks. In this paper, our primary focus has been on encoder-
only and encoder-decoder language models. Given the rising popularity of open
source decoding-only language models (e.g., StarCoder [30], SantaCoder [63],
GPT-Neox [64], etc.), we also plan to investigate these effects in such LLMs.

## Acknowledgments

This work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation;
and by the grant TED2021-129381B-C22 (SATORI project) funded by
MCIN/AEI/10.13039/501100011033.

## References

  * [1] T. Sharma, M. Kechagia, S. Georgiou, R. Tiwari, I. Vats, H. Moazen, and F. Sarro, “A survey on machine learning techniques applied to source code,” _Journal of Systems and Software_ , vol. 209, p. 111934, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121223003291
  * [2] M. Allamanis, H. Peng, and C. Sutton, “A convolutional attention network for extreme summarization of source code,” in _International conference on machine learning_. PMLR, 2016, pp. 2091–2100.
  * [3] S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in _54th Annual Meeting of the Association for Computational Linguistics 2016_. Association for Computational Linguistics, 2016, pp. 2073–2083.
  * [4] B. Wei, G. Li, X. Xia, Z. Fu, and Z. Jin, “Code generation as a dual task of code summarization,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [5] X. Gu, H. Zhang, and S. Kim, “Deep code search,” in _Proceedings of the 40th International Conference on Software Engineering_ , 2018, pp. 933–944.
  * [6] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” _arXiv preprint arXiv:1909.09436_ , 2019.
  * [7] W. Wang, Y. Zhang, Z. Zeng, and G. Xu, “Trans^ 3: A transformer-based framework for unifying code summarization and code search,” _arXiv preprint arXiv:2003.03238_ , 2020.
  * [8] L. Di Grazia and M. Pradel, “Code search: A survey of techniques for finding code,” _ACM Computing Surveys_ , vol. 55, no. 11, pp. 1–31, 2023.
  * [9] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang _et al._ , “CodeXGlue: A machine learning benchmark dataset for code understanding and generation,” _arXiv preprint arXiv:2102.04664_ , 2021.
  * [10] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang _et al._ , “CodeBERT: A pre-trained model for programming and natural languages,” _arXiv preprint arXiv:2002.08155_ , 2020.
  * [11] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu _et al._ , “GraphCodeBERT: Pre-training code representations with data flow,” _arXiv preprint arXiv:2009.08366_ , 2020.
  * [12] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “UnixCoder: Unified cross-modal pre-training for code representation,” _arXiv preprint arXiv:2203.03850_ , 2022.
  * [13] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” _arXiv preprint arXiv:2109.00859_ , 2021.
  * [14] A. V. M. Barone and R. Sennrich, “A parallel corpus of python functions and documentation strings for automated code documentation and code generation,” _arXiv preprint arXiv:1707.02275_ , 2017.
  * [15] Y. Wan, Y. He, Z. Bi, J. Zhang, Y. Sui, H. Zhang, K. Hashimoto, H. Jin, G. Xu, C. Xiong, and P. S. Yu, “NaturalCC: An open-source toolkit for code intelligence,” in _Proceedings of 44th International Conference on Software Engineering, Companion Volume_ , 2022.
  * [16] X. Zhou, D. Han, and D. Lo, “Assessing generalizability of CodeBERT,” in _2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2021, pp. 425–436.
  * [17] X. Hu, G. Li, X. Xia, D. Lo, S. Lu, and Z. Jin, “Summarizing source code with transferred api knowledge,” 2018.
  * [18] A. LeClair, S. Jiang, and C. McMillan, “A neural model for generating natural language summaries of program subroutines,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 2019, pp. 795–806.
  * [19] C. V. Lopes, P. Maj, P. Martins, V. Saini, D. Yang, J. Zitny, H. Sajnani, and J. Vitek, “Déjàvu: a map of code duplicates on github,” _Proceedings of the ACM on Programming Languages_ , vol. 1, no. OOPSLA, pp. 1–28, 2017.
  * [20] M. Gharehyazie, B. Ray, M. Keshani, M. S. Zavosht, A. Heydarnoori, and V. Filkov, “Cross-project code clones in GitHub,” _Empirical Software Engineering_ , vol. 24, pp. 1538–1573, 2019.
  * [21] Y. Golubev and T. Bryksin, “On the nature of code cloning in open-source java projects,” in _2021 IEEE 15th International Workshop on Software Clones (IWSC)_. IEEE, 2021, pp. 22–28.
  * [22] Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “CodeT5+: Open code large language models for code understanding and generation,” _arXiv preprint arXiv:2305.07922_ , 2023.
  * [23] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “CodeGen: An open large language model for code with multi-turn program synthesis,” _arXiv preprint arXiv:2203.13474_ , 2022.
  * [24] V. V. Ramasesh, A. Lewkowycz, and E. Dyer, “Effect of scale on catastrophic forgetting in neural networks,” in _International Conference on Learning Representations_ , 2021.
  * [25] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [26] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” _arXiv preprint arXiv:2101.00190_ , 2021.
  * [27] J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia, “Towards a big data curated benchmark of inter-project code clones,” in _2014 IEEE International Conference on Software Maintenance and Evolution_. IEEE, 2014, pp. 476–480.
  * [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [29] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” _ACM Computing Surveys_ , vol. 56, no. 2, pp. 1–40, 2023.
  * [30] R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim _et al._ , “StarCoder: may the source be with you!” _arXiv preprint arXiv:2305.06161_ , 2023.
  * [31] W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” _arXiv preprint arXiv:2103.06333_ , 2021.
  * [32] M. Weyssow, X. Zhou, K. Kim, D. Lo, and H. Sahraoui, “Exploring parameter-efficient fine-tuning techniques for code generation with large language models,” _arXiv preprint arXiv:2308.10462_ , 2023.
  * [33] S. Ayupov and N. Chirkova, “Parameter-efficient fine-tuning of transformers for source code,” _arXiv preprint arXiv:2212.05901_ , 2022.
  * [34] E. Shi, Y. Wang, H. Zhang, L. Du, S. Han, D. Zhang, and H. Sun, “Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond,” _arXiv preprint arXiv:2304.05216_ , 2023.
  * [35] C. Wang, Y. Yang, C. Gao, Y. Peng, H. Zhang, and M. R. Lyu, “No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2022, pp. 382–394.
  * [36] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [37] J. Lee, R. Tang, and J. Lin, “What would ELSA do? freezing layers during transformer fine-tuning,” _arXiv preprint arXiv:1911.03090_ , 2019.
  * [38] C. K. Roy and J. R. Cordy, “A survey on software clone detection research,” _Queen’s School of computing TR_ , vol. 541, no. 115, pp. 64–68, 2007.
  * [39] H. Sajnani, V. Saini, J. Svajlenko, C. K. Roy, and C. V. Lopes, “SourcererCC: Scaling code clone detection to big-code,” in _Proceedings of the 38th International Conference on Software Engineering_ , 2016, pp. 1157–1168.
  * [40] M. Zakeri-Nasrabadi, S. Parsa, M. Ramezani, C. Roy, and M. Ekhtiarzadeh, “A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges,” _Journal of Systems and Software_ , p. 111796, 2023.
  * [41] L. Jiang, G. Misherghi, Z. Su, and S. Glondu, “Deckard: Scalable and accurate tree-based detection of code clones,” in _29th International Conference on Software Engineering (ICSE’07)_. IEEE, 2007, pp. 96–105.
  * [42] M. Allamanis, “The adverse effects of code duplication in machine learning models of code,” in _Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software_ , 2019, pp. 143–153.
  * [43] D. Yang, P. Martins, V. Saini, and C. Lopes, “Stack Overflow in GitHub: any snippets there?” in _2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR)_. IEEE, 2017, pp. 280–290.
  * [44] A. Karmakar, M. Allamanis, and R. Robbes, “JEMMA: An extensible java dataset for ml4code applications,” _Empirical Software Engineering_ , vol. 28, no. 2, p. 54, 2023.
  * [45] M. Izadi, R. Gismondi, and G. Gousios, “CodeFill: Multi-token code completion by jointly learning from structure and naming sequences,” in _Proceedings of the 44th International Conference on Software Engineering_ , 2022, pp. 401–412.
  * [46] D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf _et al._ , “The stack: 3 tb of permissively licensed source code,” _arXiv preprint arXiv:2211.15533_ , 2022.
  * [47] J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using llms in software engineering,” _arXiv preprint arXiv:2312.08055_ , 2023.
  * [48] R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for java programs,” in _Proceedings of the 2014 international symposium on software testing and analysis_ , 2014, pp. 437–440.
  * [49] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” _IEEE Transactions on Software Engineering_ , 2023.
  * [50] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman _et al._ , “Evaluating large language models trained on code,” _arXiv preprint arXiv:2107.03374_ , 2021.
  * [51] E. Shi, Y. Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” in _Proceedings of the 44th International Conference on Software Engineering_ , 2022, pp. 1597–1608.
  * [52] Y. Zhang, J. Yang, H. Dong, Q. Wang, H. Shao, K. Leach, and Y. Huang, “ASTRO: An AST-assisted approach for generalizable neural clone detection,” _arXiv preprint arXiv:2208.08067_ , 2022.
  * [53] J. Li, P. He, J. Zhu, and M. R. Lyu, “Software defect prediction via convolutional neural network,” in _2017 IEEE international conference on software quality, reliability and security (QRS)_. IEEE, 2017, pp. 318–328.
  * [54] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [55] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [56] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, pp. 5485–5551, 2020.
  * [57] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Huggingface’s transformers: State-of-the-art natural language processing,” _arXiv preprint arXiv:1910.03771_ , 2019.
  * [58] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” _arXiv preprint arXiv:1910.13461_ , 2019.
  * [59] T. J. McCabe, “A complexity measure,” _IEEE Transactions on Software Engineering_ , no. 4, pp. 308–320, 1976.
  * [60] T. Ahmed and P. Devanbu, “Multilingual training for software engineering,” in _Proceedings of the 44th International Conference on Software Engineering_ , 2022, pp. 1443–1455.
  * [61] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in _Psychology of learning and motivation_. Elsevier, 1989, vol. 24, pp. 109–165.
  * [62] J. Peng, J. Hao, Z. Li, E. Guo, X. Wan, D. Min, Q. Zhu, and H. Li, “Overcoming catastrophic forgetting by soft parameter pruning,” _arXiv preprint arXiv:1812.01640_ , 2018.
  * [63] L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey _et al._ , “SantaCoder: don’t reach for the stars!” _arXiv preprint arXiv:2301.03988_ , 2023.
  * [64] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang _et al._ , “GPT-Neox-20b: An open-source autoregressive language model,” _arXiv preprint arXiv:2204.06745_ , 2022.

##  Appendix A All performances

TABLE IX: Average performances of the different models considered for the task
of the code translation.

Model | Biased group | Unbiased group | Performance | Difference
---|---|---|---|---
CodeT5 | 98.11 | 78.13 | 82.83 | 19.98
BART | 97.92 | 74.61 | 80.09 | 23.31
T5 | 89.20 | 71.26 | 75.47 | 17.94
T5v1 | 88.73 | 60.58 | 67.19 | 28.15
Rand66 | 90.02 | 56.54 | 64.41 | 33.48
Rand33 | 89.44 | 55.09 | 63.16 | 34.36
Rand63 | 89.31 | 54.90 | 62.98 | 34.42

TABLE X: Average performances of the different models considered for the task
of the code summarization with Python-150.

Model | Biased group | Unbiased group | Performance | Difference
---|---|---|---|---
CodeT5-large | 23.58 | 23.38 | 23.40 | 0.20
CodeT5-base | 22.59 | 22.26 | 22.30 | 0.34
CodeT5-large-LoRA | 22.32 | 20.92 | 21.10 | 1.40
CodeT5-small | 20.65 | 20.83 | 20.81 | -0.18
BART | 20.44 | 20.83 | 20.78 | -0.38
T5 | 19.08 | 19.54 | 19.48 | -0.47
CodeT5-large-prefix | 20.52 | 18.99 | 19.18 | 1.54
Rand66 | 18.53 | 19.02 | 18.96 | -0.50
Rand33 | 18.22 | 18.76 | 18.69 | -0.54
Rand63 | 17.65 | 18.35 | 18.26 | -0.70
T5v1 | 16.28 | 17.01 | 16.91 | -0.73

TABLE XI: Average performances of the different models considered for the task
of code summarization with TLC.

Model | Biased group | Unbiased group | Performance | Difference
---|---|---|---|---
CodeT5-large | 23.62 | 22.18 | 22.30 | 1.44
CodeT5-base | 20.84 | 20.15 | 20.21 | 0.69
BART | 17.85 | 18.30 | 18.26 | -0.45
CodeT5-small | 17.79 | 17.85 | 17.84 | -0.06
CodeT5-large-LoRA | 17.45 | 17.11 | 17.14 | 0.34
Rand66 | 17.10 | 16.94 | 16.96 | 0.16
Rand33 | 16.04 | 16.33 | 16.31 | -0.29
T5 | 15.34 | 16.26 | 16.18 | -0.92
Rand63 | 16.27 | 16.16 | 16.17 | 0.11
CodeT5-large-prefix | 16.20 | 15.89 | 15.92 | 0.31
T5v1 | 14.02 | 14.02 | 14.02 | -0.00

TABLE XII: Average performances of the different models considered for the
task of code search.

Model | Biased group | Unbiased group | Performance | Difference
---|---|---|---|---
GraphCodeBERT | 89.05 | 88.59 | 88.64 | 0.46
GraphCodeBERT-LoRA | 88.72 | 87.58 | 87.73 | 1.14
CodeBERT | 87.37 | 87.21 | 87.23 | 0.16
UnixCoder | 87.25 | 86.39 | 86.50 | 0.86
GraphCodeBERT-prefix | 87.03 | 85.71 | 85.88 | 1.33
UnixCoder-prefix | 86.62 | 85.31 | 85.48 | 1.31
UnixCoder-LoRA | 84.96 | 82.75 | 83.03 | 2.21
RoBERTa | 81.56 | 81.58 | 81.58 | -0.02
mBERT | 80.16 | 80.45 | 80.42 | -0.30
BERT | 78.63 | 79.23 | 79.15 | -0.60
Rand6 | 69.84 | 69.81 | 69.82 | 0.03
Rand3 | 69.81 | 69.31 | 69.37 | 0.50
Rand1 | 61.69 | 61.24 | 61.30 | 0.44

TABLE XIII: Average layer-wise performances of UnixCoder and GraphCodeBERT for
the task of code search.

Model | Layer | Biased group | Unbiased group | Performance | Difference
---|---|---|---|---|---
GraphCodeBERT | 3 | 89.16 | 88.79 | 88.84 | 0.37
GraphCodeBERT | 4 | 89.28 | 88.77 | 88.83 | 0.51
GraphCodeBERT | 2 | 89.26 | 88.76 | 88.83 | 0.50
GraphCodeBERT | 1 | 89.31 | 88.69 | 88.77 | 0.62
GraphCodeBERT | 5 | 89.40 | 88.64 | 88.74 | 0.76
GraphCodeBERT | 6 | 89.40 | 88.50 | 88.61 | 0.90
GraphCodeBERT | 7 | 89.10 | 88.41 | 88.50 | 0.69
GraphCodeBERT | 8 | 89.03 | 88.28 | 88.37 | 0.75
GraphCodeBERT | 9 | 88.89 | 88.11 | 88.21 | 0.78
GraphCodeBERT | 10 | 88.78 | 87.94 | 88.05 | 0.84
GraphCodeBERT | 11 | 88.11 | 86.92 | 87.07 | 1.20
UnixCoder | 1 | 82.47 | 81.00 | 81.19 | 1.47
UnixCoder | 2 | 82.44 | 80.79 | 81.00 | 1.65
UnixCoder | 3 | 82.44 | 80.66 | 80.89 | 1.78
UnixCoder | 4 | 82.58 | 80.63 | 80.88 | 1.95
UnixCoder | 5 | 82.59 | 80.60 | 80.86 | 1.99
UnixCoder | 6 | 82.45 | 80.58 | 80.82 | 1.87
UnixCoder | 7 | 82.63 | 80.43 | 80.72 | 2.20
UnixCoder | 8 | 82.39 | 79.98 | 80.29 | 2.41
UnixCoder | 9 | 81.77 | 79.50 | 79.79 | 2.27
UnixCoder | 10 | 81.09 | 78.76 | 79.06 | 2.32
UnixCoder | 11 | 79.06 | 76.71 | 77.01 | 2.35