# Data Augmentation Approaches for Source Code Models: A Survey

Terry Yue Zhuo1,2†, Zhou Yang3, Zhensu Sun1,
Yufei Wang4, Li Li5, Xiaoning Du1, Zhenchang Xing2,6, David Lo3
1 Monash University 2 CSIRO’s Data61 3 Singapore Management University
4 Huawei Noah’s Ark Lab 5 Beihang University 6 Australian National University
<EMAIL_ADDRESS>
###### Abstract

The increasingly popular adoption of source code in many critical tasks
motivates the development of data augmentation (DA) techniques to enhance
training data and improve various capabilities (e.g., robustness and
generalizability) of these models. Although a series of DA methods have been
proposed and tailored for source code models, there lacks a comprehensive
survey and examination to understand their effectiveness and implications.
This paper fills this gap by conducting a comprehensive and integrative survey
of data augmentation for source code, wherein we systematically compile and
encapsulate existing literature to provide a comprehensive overview of the
field. We start by constructing a taxonomy of DA for source code models model
approaches, followed by a discussion on prominent, methodologically
illustrative approaches. Next, we highlight the general strategies and
techniques to optimize the DA quality. Subsequently, we underscore techniques
that find utility in widely-accepted source code scenarios and downstream
tasks. Finally, we outline the prevailing challenges and potential
opportunities for future research. In essence, this paper endeavors to
demystify the corpus of existing literature on DA for source code models, and
foster further exploration in this sphere. Complementing this, we present a
continually updated GitHub repository that hosts a list of update-to-date
papers on DA for source code models, accessible at
https://github.com/terryyz/DataAug4Code.

††$\dagger$ Corresponding author.

##  1 Introduction

Data augmentation (DA) is a technique used to increase the variety of examples
without collecting new data. It has gained popularity in recent machine
learning (ML) research, with methods like back-translation Sennrich et al.
(2015); Shiri et al. (2022), Mixup Zhang et al. , and synthetic audio Asyrofi
et al. (2021) being widely adopted in natural language processing (NLP),
computer vision, and speech recognition. These techniques have significantly
improved the performance of data-centric models in low-resource domains.

However, DA has not yet been fully explored in source code modeling, which is
the intersection of ML and software engineering (SE). Source code modeling is
an emerging area that applies ML techniques to solve various source code tasks
such as code completion, code summarization, and defect detection, by training
models on a vast amount of data available in open-source repositories
Allamanis et al. (2017). Source code data typically has two modalities: the
programming language (e.g., Python and Java) and the natural language (e.g.,
doc-strings and code comments), which complement each other. Such dual-
modality nature of source code data presents unique challenges in tailoring DA
for NLP to source code models. For example, source code follows strict
syntactic rules that are specified using context-free grammars. Consequently,
conventional NLP data augmentation methods, such as token substitution with
similar words, may make the augmented source code fail to compile and become
invalid for training models.

Despite numerous challenges, there has been increasing interest and demand for
DA for source code models. With the growing accessibility of large, off-the-
shelf, pre-trained source code models via learning from large-scale corpora
Chen et al. (2021); Li et al. (2023); Allal et al. (2023), there is a growing
focus on applying these models to real-world software development. Many tasks
and programming languages are low-resource Husain et al. (2019), and lack
training examples, emphasizing the importance of DA to improve model
performance and robustness on unseen data.

Figure 1: Yearly publications on the topic of “DA for Source Code Models”.
Data Statistics as of March 2023.

This study aims to bring attention from both ML and SE communities to this
emerging field. As depicted in Figure 1, the relevant publications have been
increasing in the recent five years. More precisely, we have compiled a list
of 60 core papers from the past five years, mainly from premier conferences
and journals in both the ML and SE disciplines (with 50 out of 60 papers
published in Core Rank A/A* venues111We refer to the venues listed at
http://portal.core.edu.au/conf-ranks/ and http://portal.core.edu.au/jnl-
ranks/.). Given the escalating interest and burgeoning research in this
domain, it is timely for our survey to (1) provide a comprehensive overview of
DA for source code models, and (2) pinpoint key challenges and opportunities
to stimulate and guide further exploration in this emerging field. To the best
of our awareness, our paper constitutes the first comprehensive survey
offering an in-depth examination of DA techniques for source code models.

The structure of this paper is organized as follows:

  * •

Section 2 offers a thorough review of three categories of DA for source code
models: rule-based (2.1), model-based (2.2), and example interpolation-based
(2.3) techniques.

  * •

Section 3 provides a summary of prevalent strategies and techniques designed
to enhance the quality of augmented data, encompassing method stacking (3.1)
and optimization (3.2).

  * •

Section 4 articulates various beneficial source code scenarios for DA,
including adversarial examples for robustness (4.1), low-resource domains
(4.2), retrieval augmentation (4.3), and contrastive learning (4.4).

  * •

Section 5 delineates DA methodologies for common source code tasks, such as
code authorship attribution (5.1), clone detection (5.2), defect detection
(5.3), code summarization (5.4), code search (5.5), code completion (5.6),
code translation (5.7), code question answering (5.8), problem classification
(5.9), method name prediction (5.10), and type prediction (5.11).

  * •

Section 6 expounds on the challenges and future prospects in the realm of DA
for source code models.

Through this work, we hope to emulate prior surveys which have analyzed DA
techniques for other data types, such as text Feng et al. (2021), time series
Wen et al. (2020), and images Shorten and Khoshgoftaar (2019). Our intention
is to pique further interest, spark curiosity, and encourage further research
in the field of data augmentation, specifically focusing on its application to
source code.

##  2 Data Augmentation Methods in Source Code

This section categorizes the mainstream DA techniques specifically designed
for source code models into three parts: rule-based, model-based, and example-
interpolation techniques. We explain studies of different branches as follows.

###  2.1 Rule-based Techniques

Source code is specific by context-free grammars, which impose specific syntax
rules that must be followed by the code to be successfully compiled and
executed by a machine. A large number of DA methods utilize predetermined
rules to transform the programs without breaking syntax rules. Specifically,
these rules mainly leverage abstract syntax trees (ASTs) to transform the code
snippets. The transformations can include operations such as replacing
variable names, renaming method names, and inserting dead code. Besides the
basic program syntax, some code transformations consider deeper structural
information, such as control-flow graph (CGF) and use-define chains (UDG)
Quiring et al. (2019). Additionally, a small part of rule-based DA techniques
also focuses on augmenting the natural language context in the code snippets,
including doc-strings and comments Bahrami et al. (2021); Song et al. (2022);
Park et al. (2023). We illustrate a DA example utilizing ASTs in Figure 2.

Figure 2: Rule-based DA to transform code snippets, Wang et al. (2022c).

Zhang et al. (2020a) propose MHM, a method of iteratively renaming identifiers
in the code snippets. Considered as the approach to generate examples for
adversarial training, MHM greatly improves the robustness of source code
models. Later, Srikant et al.  consider program obfuscations as adversarial
perturbations, where they rename program variables in an attempt to hide the
program’s intent from a reader. By applying these perturbed examples to the
training stage, the source code models become more robust to the adversarial
attack. Instead of just renaming identifiers, BUGLAB-Aug Allamanis et al.
(2021) contains more rules to augment code snippets by emphasizing on both
programming language and natural language, such as comment deletion,
comparison expression mirroring and if-else branch swapping. The evaluation on
BUGLAB-Aug demonstrates that DA methods can be exploited for self-supervised
bug detection and repair. Similarly, Jain et al. (2021) use compiler
transforms as data augmentation, called Transpiler, automatically generating a
dataset of equivalent functions. Specifically, they define 11 compiler
transforms by exploiting ASTs of the programs. Rule-based DA later has been
widely used for source code models to capture code representation effectively
via contrastive learning Ding et al. (2021); Liu et al. (2023b).

Brockschmidt et al.  present a generative source code model by augmenting the
given AST with additional edges to learn diverse code expressions. Instead of
the direct augmentation on AST, Quiring et al. (2019) propose three different
augmentation schemes via the combination of AST and CFG, UDG and declaration-
reference mapping (DRM), named as Control Transformations, Declaration
Transformations and API Transformations. Control Transformations rewrite
control-flow statements or modifies the control flow between functions. In
total, the family contains 5 transformations. This transformation involves
passing variables as function arguments, updating their values, and changing
the control flow of the caller and callee. Declaration Transformations consist
of 14 transformers that modify, add or remove declarations in source code.
Declaration Transformations make DA necessary to update all usages of
variables which can be elegantly carried out using the DRM representation. API
Transformations contain 9 transformations and exploits the fact that various
APIs can be used to solve the same problem. Programmers are known to favor
different APIs and thus tampering with API usage is an effective strategy for
changing stylistic patterns.

Another line of work is augmenting the natural language context in source
code. QRA Huang et al. (2021) augments examples by rewriting natural language
queries when performing code search and code question answering. It rewrites
queries with minor rule-based modifications that share the same semantics as
the original one. Specifically, it consists of three ways: randomly deleting a
word, randomly switching the position of two words, and randomly copying a
word. Inspired by this approach, Park et al. (2023) recently devised KeyDAC
with an emphasis on the query keywords. KeyDAC augments on both natural
language and programming language. For natural language query, it follows the
rules in QRA but only modifies non-keywords. In terms of programming language
augmentation, KeyDAC simply uses ASTs to rename program variables, similar to
the aforementioned works.

###  2.2 Model-based Techniques

A series of DA techniques for source code target training various models to
augment data. Intuitively, Mi et al. (2021) utilize Auxiliary Classifier
Generative Adversarial Networks (AC-GAN) Odena et al. (2017) to generate
augmented programs. In order to increase the training data for code
summarization, CDA-CS Song et al. (2022) uses the pre-trained BERT model
Devlin et al. (2019) to replace synonyms for non-keywords, which benefits the
source code downstream tasks.

While these methods largely adapt the existing model-based DA techniques for
general purposes, most DA approaches are specifically designed for source code
models. Li et al. (2022f) introduce IRGen, a genetic-algorithm-based model
using compiler intermediate representation (LLVM IR) to augment source code
embeddings, where IRGen seamlessly generates a piece of source code into a
range of semantically identical but syntactically distinct IR codes for
improving model’s contextual understanding. Ahmad et al. (2023) investigate
the suitability of the multilingual generative source code models for
unsupervised programming language translation via Back-translation, in the
similar scope of the one for NLP Sennrich et al. (2016). In our context, Back-
translation translates to and from natural language to perform back-
translation between two programming languages. Pinku et al. (2023) exploit
another generative source code model, Transcoder Roziere et al. (2020), to
perform source-to-source translation for augmenting cross-language source
code.

###  2.3 Example Interpolation Techniques

Another category of data augmentation (DA) techniques, originated by Mixup
Zhang et al.  in CV, involves interpolating the inputs and labels of two or
more actual examples. This collection of methods is also termed Mixed Sample
Data Augmentation (MSDA). Despite trials in the context of text classification
problems, no studies within the realm of source code have yet demonstrated the
efficacy of these techniques in enhancing model performance.

Figure 3: MixCode, Dong et al. (2023a).

In contrast to naive interpolation, the majority of example-based DA methods
for NLP enhance the Mixup algorithm to fuse multiple real examples into a
single input. This approach differs markedly from the DA techniques used in
CV, which can operate directly at the discrete raw input level. Applying
extensions of Mixup for NLP and source code modeling necessitates engagement
at the embedding level, given that tasks in these domains rely on continuous
inputs Feng et al. (2021). As an illustration in Figure 3, Dong et al. (2023a)
merge rule-based techniques for source code models with Mixup to blend the
representations of the original code snippet and its transformation. This
approach is commonly regarded as the linear interpolation technique deployed
in NLP classification tasks.

Li et al. (2022a) introduce two novel interpolation techniques for source code
models, namely Binary Interpolation and Linear Extrapolation. Binary
Interpolation serves as a data augmentation strategy, which interchangeably
swaps features between samples using elements acquired from a Bernoulli
distribution. On the other hand, Linear Extrapolation is another data
augmentation approach that generates new data points beyond the existing
feature space by extending current features in accordance with a uniform
distribution.

##  3 Strategies and Techniques

In real-world applications, the design and efficacy of DA techniques are
influenced by a variety of factors. This section highlights these factors,
offering insights and techniques for devising and optimizing suitable DA
methods.

###  3.1 Method Stacking

As discussed in Section 2, numerous DA strategies are proposed concurrently in
a single work, aiming to enhance performance. Common combinations entail two
types: same-type DA or a mixture of different DA methods. The former is
typically applied in rule-based DA techniques, stemming from the realization
that a single code transformation cannot fully represent the diverse code
formats found in the real world. While many programs share similar
functionalities, they vastly differ in code styles and implementation logic.

Several works Shi et al. (2023); Huang et al. (2021) demonstrate that merging
multiple types of DA techniques can enhance the performance of source code
models. Mi et al. (2021) combined rule-based code transformation schemes with
model-based DA using AC-GAN to create an augmented corpus for model training.
Instead of augmenting on programming language, CDA-CS Song et al. (2022)
encompasses two kinds of DA techniques: rule-based non-keyword extraction and
model-based non-keyword replacement. Empirical evidence from Chen and
Lampouras (2023) shows that combining Back-translation and variable renaming
can result in improved code completion performance.

###  3.2 Optimization

In certain scenarios such as enhancing robustness and minimizing computational
cost, optimally selecting specific augmented example candidates is crucial. We
introduce three types of strategies: probabilistic, model-based, and rule-
based selection. Probabilistic selection is defined as the optimization via
sampling from a probability distribution, while model-based selection is
guided by the model to select the most proper examples. Regarding rule-based
selection, it is an optimization strategy where specific predetermined rules
or heuristics are used to select the most suitable examples.

####  3.2.1 Probabilistic Selection

We introduce three representative probabilistic selection strategies, MHM,
QMDP and BUGLAB-Aug. MHM Zhang et al. (2020a) adopts the Metropolis-Hastings
probabilistic sampling method, which is a Markov Chain Monte Carlo technique,
to choose adversarial examples via identifier replacement. Similarly, QMDP
Tian et al. (2021) uses a Q-learning approach to strategically select and
execute rule-based structural transformations on the source code, thereby
guiding the generation of adversarial examples. In BUGLAB-Aug, Allamanis et
al. (2021) model the probability of applying a specific rewrite rule at a
location in a code snippet similar to the pointer net Merity et al. .

####  3.2.2 Model-based Selection

Several DA techniques employing this strategy use the model’s gradient
information to guide the selection of augmented examples. An emblematic
approach is the DAMP method Yefet et al. (2020), which optimizes based on the
model loss to select and generate adversarial examples via variable renaming.
Another variant, SPACE Li et al. (2022c), performs selection and perturbation
of code identifiers’ embeddings via gradient ascent, targeting to maximize the
model’s performance impact while upholding semantic and grammatical
correctness of the programming language. A more complex technique, ALERT Yang
et al. (2022b), uses a genetic algorithm in its gradient-based selection
strategy. It evolves a population of candidate solutions iteratively, guided
by a fitness function that calculates the model’s confidence difference,
aiming to identify the most potent adversarial examples.

DA Method | Category | PL | NL | Optimization | Preprocess | Parsing | Level | TA | LA
---|---|---|---|---|---|---|---|---|---
ComputeEdge Brockschmidt et al.  | Rule | ✓ | ✗ | — | — | AST | AST | ✓ | ✓
RefineRepresentation Bielik and Vechev (2020) | Rule | ✓ | ✗ | Model | — | AST | AST | ✓ | ✓
Control Transformations Quiring et al. (2019) | Rule | ✓ | ✗ | Prob | — | AST+CFG+UDG | Input | ✓ | ✗
Declaration Transformations Quiring et al. (2019) | Rule | ✓ | ✗ | Prob | — | AST+DRM | Input | ✓ | ✗
API Transformations Quiring et al. (2019) | Rule | ✓ | ✗ | Prob | — | AST+CFG+DRM | Input | ✓ | ✗
DAMP Yefet et al. (2020) | Rule | ✓ | ✗ | Model | — | AST | Input | ✓ | ✓
IBA Huang et al. (2021) | Rule | ✗ | ✓ | — | Tok | — | Embed | ✗ | ✓
QRA Huang et al. (2021) | Rule | ✓ | ✗ | — | Tok | — | Input | ✗ | ✓
MHM Zhang et al. (2020a) | Rule | ✗ | ✓ | Prob | — | AST | Input | ✓ | ✗
Mossad Devore-McDonald and Berger (2020) | Rule | ✓ | ✗ | Rule | Tok | AST | Input | ✓ | ✓
AugmentedCode Bahrami et al. (2021) | Rule | ✓ | ✗ | — | Tok | — | Input | ✗ | ✓
QMDP Tian et al. (2021) | Rule | ✓ | ✗ | Prob | Tok | AST | Input | ✓ | ✗
Transpiler Jain et al. (2021) | Rule | ✓ | ✗ | Prob | — | AST | Input | ✓ | ✗
BUGLAB-Aug Allamanis et al. (2021) | Rule | ✓ | ✗ | Prob | Tok | AST | Input | ✗ | ✓
SPAT Yu et al. (2022b) | Rule | ✓ | ✗ | Model | — | AST | Input | ✓ | ✗
RoPGen Li et al. (2022d) | Rule | ✓ | ✗ | Model | — | AST | Input | ✓ | ✗
ACCENT Zhou et al. (2022) | Rule | ✓ | ✗ | Rule | — | AST | Input | ✓ | ✓
SPACE Li et al. (2022c) | Rule | ✓ | ✗ | Model | Tok | AST | Embed | ✓ | ✓
ALERT Yang et al. (2022b) | Rule | ✓ | ✗ | Model | Tok | AST | Input | ✓ | ✓
IRGen Li et al. (2022f) | Rule | ✓ | ✗ | Rule | — | AST+IR | IR | ✓ | ✓
Binary Interpolation Li et al. (2022a) | EI | ✓ | ✓ | — | — | — | Embeb | ✓ | ✓
Linear Extrapolation Li et al. (2022a) | EI | ✓ | ✓ | — | — | — | Embeb | ✓ | ✓
Gaussian Scaling Li et al. (2022a) | Rule | ✓ | ✓ | Model | — | — | Embeb | ✓ | ✓
CodeTransformator Zubkov et al. (2022) | Rule | ✓ | ✗ | Rule | — | AST | Input | ✓ | ✗
RADAR Yang et al. (2022a) | Rule | ✓ | ✗ | Rule | — | AST | Input | ✓ | ✗
AC-GAN Mi et al. (2021) | Model | ✓ | ✗ | — | — | — | Input | ✓ | ✓
CDA-CS Song et al. (2022) | Model | ✗ | ✓ | Model | KWE | — | Input | ✗ | ✓
srcML-embed Li et al. (2022e) | Rule | ✓ | ✗ | — | — | AST | Embed | ✓ | ✗
MultIPA Orvalho et al. (2022) | Rule | ✓ | ✗ | — | — | AST | Input | ✓ | ✗
ProgramTransformer Rabin and Alipour (2022) | Rule | ✓ | ✗ | — | — | AST | Input | ✓ | ✗
Back-translation Ahmad et al. (2023) | Model | ✓ | ✗ | — | Tok | — | Input | ✗ | ✓
MixCode Dong et al. (2023a) | Rule+EI | ✓ | ✓ | — | — | — | Embed | ✓ | ✓
NP-GD Shen et al.  | Model | ✓ | ✗ | Model | Tok | — | Embed | ✓ | ✓
ExploitGen Yang et al. (2023) | Rule | ✗ | ✓ | — | — | — | Input | ✓ | ✗
SoDa Shi et al. (2023) | Model | ✓ | ✓ | — | — | AST | Input | ✓ | ✓
Transcompiler Pinku et al. (2023) | Model | ✓ | ✗ | — | — | — | Input | ✓ | ✗
STRATA Springer et al. (2021) | Rule | ✓ | ✗ | Model | Tok | AST | Input | ✓ | ✓
KeyDAC Park et al. (2023) | Rule | ✓ | ✓ | — | KWE | AST | Embed | ✗ | ✓
Simplex Interpolation Zhang et al. (2022) | EI | ✓ | ✗ | — | — | AST+IR | Embed | ✗ | ✓

Table 1: Comparing a selection of DA methods by various aspects relating to
their applicability, dependencies, and requirements. PL, NL, TA, LA, EI, Prob,
Tok, and KWE stand for Programming Language, Natural Language, Example
Interpolation, Probability, Tokenization, Keyword Extraction, Task-Agnostic,
and Language-Agnostic. PL and NL determine if the DA method is applied to the
programming language or natural language context. Preprocess denotes
preprocessing required besides the program parsing. Parsing refers to the type
of feature used by the DA method during program parsing. Level denotes the
depth at which data is modified by the DA. TA and LA represent whether the DA
method can be applied to different tasks or programming languages. As most
papers do not clearly state if their DA methods are TA and LA, we subjectively
denote the applicability.

####  3.2.3 Rule-based Selection

Rule-based selection stands as a powerful approach, featuring predetermined
fitness functions or rules. This method often relies on evaluation metrics for
decision-making. For instance, IRGen Li et al. (2022f) utilizes a Genetic-
Algorithm-based optimization technique with a fitness function based on IR
similarity. On the other hand, ACCENT Zhou et al. (2022) and RADAR apply
evaluation metrics such as BLEU Papineni et al. (2002) and CodeBLEU Ren et al.
(2020) respectively to guide the selection and replacement process, aiming for
maximum adversarial impact. Finally, STRATA Springer et al. (2021) employs a
rule-based technique to select high-impact subtokens that significantly alter
the model’s interpretation of the code.

##  4 Scenarios

This section delves into several DA approaches for commonplace scenarios of
source code scenarios.

###  4.1 Adversarial Examples for Robustness

Robustness presents a critical and complex dimension of software engineering,
necessitating the creation of semantically-conserved adversarial examples to
discern and mitigate vulnerabilities within source code models. There is a
surge in designing more effective DA techniques for generating these examples
in recent years. Several studies Yefet et al. (2020); Li et al. (2022d);
Srikant et al. ; Li et al. (2022c); Anand et al. ; Henke et al. (2022) have
utilized rule-based DA methods for testing and enhancing model robustness.
Wang et al. (2023) have gone a step further to consolidate universally
accepted code transformation rules to establish a benchmark for source code
model robustness.

###  4.2 Low-Resource Domains

The application of DA methods within low-resource domains is a recurrent theme
within the CV and NLP communities Shorten and Khoshgoftaar (2019); Feng et al.
(2021). Yet, this scenario remains underexplored within the source code
discipline. In order to increase data in the low-resource domain for
representation learning, Li et al. (2022f) tend to add more training data to
enhance source code model embeddings by unleashing the power of compiler IR.
Ahmad et al. (2023) propose to use source code models to perform Back-
translation DA, taking into consideration the scenario of low-resource
programming languages. Meanwhile, Chen and Lampouras (2023) underscore the
fact that source code datasets are markedly smaller than their NLP
equivalents, which often encompass millions of instances. As a result, they
commence investigations into code completion tasks under this context and
experiment with Back-translation and variable renaming. Shen et al.  contend
that the generation of bash comments is hampered by a dearth of training data
and thus explore model-based DA methods for this task.

###  4.3 Retrieval Augmentation

Increasing interest has been observed in the application of DA for retrieval
augmentation within NLP Mialon et al. (2023) and source code Lu et al. (2022).
Retrieval-augmented source code models incorporate retrieval within both pre-
training and downstream applications. This form of augmentation enhances the
parameter efficiency of models, as they are able to store less knowledge
within their parameters and instead retrieve it. It is shown as a promising
application of DA in various source code downstream tasks, such as code
summarization Zhang et al. (2020b); Liu et al. ; Yu et al. (2022a), code
completion Parvez et al. (2021) and program repair Nashid et al. (2023).

###  4.4 Contrastive Learning

Another source code scenario to deploy DA methods is contrastive learning,
where it enables models to learn an embedding space in which similar samples
are close to each other while dissimilar ones are far apart Chen et al.
(2022); Wang et al. (2022b); Zhang et al. (2022). As the training datasets
commonly contain limited sets of positive samples, DA methods are preferred to
construct similar samples as the positive ones. Liu et al. (2023b) make use of
contrastive learning with DA to devise superior pre-training paradigms for
source code models, while some works study the advantages of this application
in some source code tasks like defect detection Cheng et al. (2022), clone
detection Zubkov et al. (2022); Wang et al. (2022a) and code search Shi et al.
(2022b, 2023); Li et al. (2022b).

##  5 Downstream Tasks

In this section, we discuss several DA works for common source code tasks and
evaluation datasets.

###  5.1 Code Authorship Attribution

Code authorship attribution is the process of identifying the author of a
given code, usually achieved by source code models. Yang et al. (2022b)
initially investigate generating adversarial examples on the Google Code Jam
(GCJ) dataset, which effectively fools source code models to identify the
wrong author of a given code snippet. By training with these augmented
examples, the model’s robustness can be further improved. Li et al. (2022d)
propose another DA method called RoPGen for the adversarial attack and
demonstrate its efficacy on GCJ. Dong et al. (2023b) empirically study the
effectiveness of several existing DA approaches for NLP on several source code
tasks, including authorship attribution on GCJ.

###  5.2 Clone Detection

Code clone detection refers to the task of identifying if the given code
snippet is cloned and modified from the original sample, and can be called
plagiarism detection in some cases. This is a challenging downstream task as
it needs the source code model to understand the source code both
syntactically and semantically. Jain et al. (2021) propose correct-by-
construction DA via compiler information to generate many variants with
equivalent functionality of the training sample and show its effectiveness of
improving the model robustness on BigCloneBench Svajlenko et al. (2014) and a
self-collected JavaScript dataset. Jia et al. (2023) show that when training
with adversarial examples via obfuscation transformation, the robustness of
source code models can be significantly improved. Zubkov et al. (2022) provide
the comparison of multiple contrastive learning, combined with rule-based
transformations for the clone detection task. Pinku et al. (2023) later use
Transcompiler to translate between limited source code in Python and Java and
therefore increase the training data for cross-language code clone detection.

###  5.3 Defect Detection

Defect Detection, in other words, bug or vulnerability detection, is to
capture the bugs in given code snippets. The task can be considered as the
binary classification task, where the labels are either true or false.
Allamanis et al. (2021) implement BUGLAB-Aug, a DA framework of self-
supervised bug detection and repair. BUGLAB-Aug has two sets of code
transformation rules, one is a bug-inducing rewrite and the other one is
rewriting as DA. Their approach boosts the performance and robustness of
source code models simultaneously. Cheng et al. (2022) present a path-
sensitive code embedding technique called ContraFlow, which uses self-
supervised contrastive learning to detect defects based on value-flow paths.
ContraFlow utilizes DA to generate contrastive value-flow representations of
three datasets (namely D2A Zheng et al. (2021), Fan Fan et al. (2020) and
FFMPeg+Qemu Zhou et al. (2019)) to learn the (dis)-similarity among programs.
Ding et al. (2021) present a novel self-supervised model focusing on
identifying (dis)similar functionalities of source code, which outperforms the
state-of-the-art models on REVEAL Chakraborty et al. (2022) and FFMPeg+Qemu
Zhou et al. (2019). Specifically, they design code transformation heuristics
to automatically create bugged programs and similar code for augmenting pre-
training data.

###  5.4 Code Summarization

Code summarization is considered as a task that generates a comment for a
piece of the source code, and is thus also named code comment generation.
Zhang et al. (2020c) apply MHM to perturb training examples and mix them with
the original ones for adversarial training, which effectively improves the
robustness of source code models in summarizing the adversarial code snippets.
Zhang et al. (2020b) develop a retrieval-augmentation framework for code
summarization, relying on similar code-summary pairs to generate the new
summary on PCSD and JCSD datasets Miceli-Barone and Sennrich (2017); Hu et al.
(2018). Based on this framework, Liu et al.  leverage Hybrid GNN to propose a
novel retrieval-augmented code summarization method and use it during model
training on the self-collected CCSD dataset. Zhou et al. (2022) generate
adversarial examples of a Python dataset Wan et al. (2018) and JSCD to
evaluate and enhance the source code model robustness.

###  5.5 Code Search

Code search, or code retrieval, is a text-code task that searches code
snippets based on the given natural language queries. The source code models
on this task need to map the semantics of the text to the source code. Bahrami
et al. (2021) increase the code search queries by augmenting the natural
language context such as doc-string, code comments and commit messages. Shi et
al. (2022b) use AST-focused DA to replace the function and variable names of
the data in CodeSearchNet Husain et al. (2019) and CoSQA Huang et al. (2021).
Shi et al. (2023) introduce soft data augmentation (SoDa), without external
transformation rules on code and text. With SoDa, the model predicts tokens
based on dynamic masking or replacement when processing CodeSearchNet. Instead
of applying rule-based DA techniques, Li et al. (2022a) manipulate the
representation of the input data by interpolating examples of CodeSearchNet.

###  5.6 Code Completion

Code completion requires source code models to generate lines of code to
complete given programming challenges. Anand et al.  suggest that source code
models are vulnerable to adversarial examples which are perturbed with
transformation rules. Lu et al. (2022) propose a retrieval-augmented code
completion framework composed of the rule-based DA module to generate on PY150
Raychev et al. (2016) and GitHub Java Corpus datasets Allamanis and Sutton
(2013). Wang et al. (2023) customize over 30 transformations specifically for
code on docstrings, function and variable names, code syntax, and code format
and benchmark generative source code models on HumanEval Chen et al. (2021)
and MBPP Austin et al. (2021). Yang et al. (2022a) devise transformations on
functional descriptions and signatures to attack source code models and show
that their performances are susceptible.

###  5.7 Code Translation

Similar to neural machine translation in NLP Stahlberg (2020), the task is to
translate source code written in a specific programming language translation
to another one. Ahmad et al. (2023) apply data augmentation through back-
translation to enhance unsupervised code translation. They use pre-trained
sequence-to-sequence models to translate code into natural language summaries
and then back into code in a different programming language, thereby creating
additional synthetic training data to improve model performance. Chen and
Lampouras (2023) utilize Back-translation and variable augmentation techniques
to yield the improvement in code translation on CodeTrans Lu et al. (2021).

###  5.8 Code Question Answering (CQA)

CQA can be formulated as a task where the source code models are required to
generate a textual answer based on given a code snippet and a question. Huang
et al. (2021) incorporate two rule-base DA methods on code and text to create
examples for contrastive learning. Li et al. (2022c) explore the efficacy of
adversarial training on the continuous embedding space with rule-based DA on
CodeQA Liu and Wan (2021), a free-form CQA dataset. Park et al. (2023)
evaluate , a framework using query writing and variable renaming as DA, on
WebQueryTest of CodeXGLUE Lu et al. (2021). Different from CodeQA,
WebQueryTest is a CQA benchmark only containing Yes/No questions.

###  5.9 Problem Classification

The task performs the categorization of programs regarding their
functionality. Wang et al. (2022b) propose a novel AST hierarchy
representation for contrastive learning with the graph neural network.
Specially, they augment the node embeddings in AST paths on OJ, a dataset
containing 104 classes of programs. Zhang et al. (2022) incorporate simplex
interpolation, an example-interpolation DA approach on IR, to create
intermediate embeddings on POJ-104 from CodeXGLUE Lu et al. (2021). Dong et
al. (2023a) also explore the example-interpolation DA to fuse the embeddings
of code snippets. They evaluate the method on two datasets, JAVA250 and
Python800 Puri et al. (2021).

###  5.10 Method Name Prediction

The goal of method name prediction is to predict the name of a method given
the program. Yefet et al. (2020) attack and defense source code models by
using variable-name-replaced adversarial programs on the Java dataset Alon et
al. (2019). Pour et al. (2021) propose a search-based testing framework
specifically for adversarial robustness. They generate adversarial examples of
Java with ten popular refactoring operators widely used in Java. Yu et al.
(2022b) implement a data augmentation framework and 18 transformation rules
for processing Java source code on the Java dataset.

###  5.11 Type Prediction

Type prediction, or type interference, aims to predict parameter and function
types in programs. Bielik and Vechev (2020) conduct adversarial attack on
source code models with the examples of transformed ASTs. They instantiate the
attack to type prediction on JavaScript and TypeScript. Jain et al. (2021)
apply compiler transforms to generates many variants of programs in DeepTyper
Hellendoorn et al. (2018), with equivalent functionality with 11 rules. Li et
al. (2022e) incorporate srcML Collard et al. (2013) meta-grammar embeddings to
augment the syntactic features of examples in three datasets, DeepTyper,
Typilus Data Allamanis et al. (2020) and CodeSearchNet Husain et al. (2019).

##  6 Challenges and Opportunities

When it comes to source code, DA faces significant challenges. Nonetheless,
it’s crucial to acknowledge that these challenges pave the way for new
possibilities and exciting opportunities in this area of work.

##### Discussion on theory.

Currently, there’s a noticeable gap in the in-depth exploration and
theoretical understanding of DA methods in source code. Most existing research
on DA is centered around image processing and natural language fields, viewing
data augmentation as a way of applying pre-existing knowledge about data or
task invariance Dao et al. (2019); Wu et al. (2020); Shi et al. (2022a). When
shifting to source code, much of the previous work introduces new methods or
demonstrates how DA techniques can be effective for subsequent tasks. However,
these studies often overlook the why and how particularly from a mathematical
perspective. With source code being discrete by nature, having a theoretical
discussion becomes even more important. It allows us to understand DA from a
broader perspective, not just by looking at experimental results. By exploring
DA in this way, we can better understand its underlying principles without
being solely dependent on experimental validation.

##### More study on pre-trained models.

In recent years, pre-trained source code models have been widely applied in
source code, containing rich knowledge through self-supervision on a huge
scale of corpora Feng et al. (2020); Guo et al. (2021); Zhuo (2023). Numerous
studies have been conducted utilizing pre-trained source code models for the
purpose of DA, yet, most of these attempts are confined to mask token
replacement Shi et al. (2023), direct generation after fine-tuning Ahmad et
al. (2023); Pinku et al. (2023). An emergent research opportunity lies in
exploring the potential of DA in the source code domain with the help of large
language models (LLMs) trained on a large amount of text and source code Chen
et al. (2021); Li et al. (2023). LLMs have the capability of context
generation based on prompted instructions and provided examples, making them a
choice to automate the DA process in NLP Yoo et al. (2021); Wang et al.
(2021a). Different from the previous usages of pre-trained models in DA, these
works open the era of “prompt-based DA”. In contrast, the exploration of
prompt-based DA in source code domains remains a relatively untouched research
area. Another direction is to harness the internal knowledge encoded in pre-
trained source code models. For example, Karmakar and Robbes (2021); Wan et
al. (2022) show that ASTs and code semantics can be induced from these models
without the static analysis tools. As most DA methods for source code models
tend to predefine the code transformation rules via program analysis, it is
expected that the programming knowledge inside these pre-trained source code
models can automate the rule designs.

##### Working with domain-specific data.

Our paper focus on surveying DA techniques for common downstream tasks
involving processing source code. However, we are aware that there are a few
works on other task-specific data in the field of source code. For instance,
API recommendation and API sequence generation can be considered a part of
source code tasks Huang et al. (2018); Gu et al. (2016). DA methods covered by
our survey can not be directly generalized to these tasks, as most of them
only target program-level augmentation but not API-level. We observe a gap of
DA techniques between this two different layers Treude and Robillard (2016);
Xu et al. (2020); Wang et al. (2021b), which provides opportunities for future
works to explore. Additionally, DA for out-of-distribution generalization has
not yet been fully justified in the source code modeling. Previous studies
Hajipour et al. (2022); Hu et al. (2022) assume the domain as the programs
with different complexity, syntax, and semantics. We argue that this
definition is not natural enough. Similar to the subdomains in NLP like
biomedical and financial texts, the application subdomains of source code can
be diverse. For example, the programs to solve data science problems can be
significantly different from the ones for web design. We encourage SE and ML
communities to study the benefits of DA when applied to various application
subdomains of source code.

##### More exploration on project-level source code and low-resource
programming languages.

The existing methods have made sufficient progress in function-level code
snippets and common programming languages. However, there is no investigation
of DA methods on project-level source code. The DA on source code projects can
be distinct from the function-level DA, as it may involve more information
such as the interdependencies between different code modules, high-level
architectural considerations, and the often intricate relationship between
data structures and algorithms used across the project Mockus et al. (2002).
At the same time, limited by data resources Husain et al. (2019); Orlanski et
al. (2023), augmentation methods of low-resource languages are scarce,
although they have more demand for DA. Obviously, exploration in these two
directions is still limited, and they could be promising directions.

##### Mitigating social bias.

Social bias in NLP has been well studied and can be mitigated with DA Feng et
al. (2021). We argue that social bias can also exist in source code and has
not brought attention. For example, Zhuo et al. (2023a) find that LLMs of
source code have server bias in various demographics such as gender,
sexuality, and occupation when performing code generation based on the natural
language queries. In order to make these models more responsible in source
code, we urge more research on mitigating bias. As suggested by prior works in
NLP, DA may be an effective technique to make source code models more
responsible.

##### Few-shot learning.

In few-shot scenarios, models are required to achieve performance that rivals
that of traditional machine learning models, yet the amount of training data
is extremely limited. DA methods provide a direct solution to the problem.
However, limited works in few-shot scenarios have adopted DA methods Nashid et
al. (2023). Such methods ensure the validity of the augmented data but also
lead to insufficient semantic diversity. Mainstream pre-trained source code
models obtain rich semantic knowledge through language modeling. Such
knowledge even covers to some extent the semantic information introduced by
traditional paraphrasing-based DA methods. In other words, the improvement
space that traditional DA methods bring to pre-trained source code models has
been greatly compressed. Therefore, it is an interesting question how to
provide models with fast generalization and problem-solving capability by
generating high-quality augmented data in few-shot scenarios.

##### Multimodal applications.

Apart from the mainstream source code applications which utilize natural
language and programming language, some applications involving more modalities
have also been developed. Wang et al. (2021b) and Liu et al. (2023a) explore
the chart derendering with an emphasis on source code and corresponding APIs.
Surís et al. (2023) propose a framework to generate Python programs to solve
complex visual tasks including images and videos. Although such multimodal
applications are more and more popular, no study has yet been conducted on
applying DA methods to them. A potential technique for multimodal source code
tasks is to combine DA methods for each modality.

##### Lack of unification.

The current body of literature on data augmentation (DA) for source code
presents a challenging landscape, with the most popular methods often being
portrayed in a supplementary manner. A handful of empirical studies have
sought to compare DA methods for source code models de Paula Rodrigues et al.
(2023); Dong et al. (2023b), however, these largely rely on basic context
transformations that are not specifically tailored to DA approaches described
in other literature. Whereas there are well-accepted frameworks for DA for CV
(e.g. default augmentation libraries in PyTorch, RandAugment Cubuk et al.
(2020)) and DA for NLP (e.g. NL-Augmenter Dhole et al. (2021)), a
corresponding library of generalized DA techniques for source code models is
conspicuously absent. Furthermore, as existent DA methods are usually
evaluated with various datasets, it is hard to truly determine the efficacy.
Therefore, we posit that the progression of DA research would be greatly
facilitated by the establishment of standardized and unified benchmark tasks,
along with datasets for the purpose of contrasting and evaluating the
effectiveness of different augmentation methods. This would pave the way
towards a more systematic and comparative understanding of the benefits and
limitations of these methods.

##  7 Conclusion

Our paper provides a comprehensive analysis of data augmentation techniques in
the context of source code. We first provide an explanation of the concept of
data augmentation and its function. We then examine the primary data
augmentation methods that are commonly employed in source code research, as
well as explore augmentation approaches for common source code applications
and tasks. Finally, we conclude by outlining the current challenges in the
field and suggesting potential directions for future source code research. Our
aim in presenting this paper is to assist source code researchers in selecting
appropriate data augmentation techniques and to encourage further exploration
and advancement in this field.

## Limitations

While the work presents in this paper has its merits, we acknowledge the
several limitations. Firstly, our work does not cover DA methods for semantic
parsing including text-to-SQL Zhuo et al. (2023b), which is conventionally
considered as a NLP task. Secondly, our focus has been primarily on function-
level DA within the context of source code. As such, there remains a need for
future development in project-level DA methods. Nonetheless, this paper offers
a valuable collection of general-purpose DA techniques for source code models
and we hope that it can serve as an inspiration for further research in this
area. Thirdly, given the page limits, the descriptions presented in this
survey are largely brief in nature. Our approach has been to present the works
in meaningful structured groups, rather than unstructured sequences, to ensure
comprehensive coverage. This work can be used as an index where more detailed
information can be found in the corresponding works. Lastly, it is worth
noting that this survey is purely qualitative and does not include any
experiments or empirical results. To provide more meaningful guidance, it
would be helpful to conduct comparative experiments across different DA
strategies. We leave this as a suggestion for future work.

## References

  * Ahmad et al. (2023) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2023.  Summarize and generate to back-translate: Unsupervised translation of programming languages.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1528–1542, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Muñoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alexander Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, J. Poirier, Hailey Schoelkopf, Sergey Mikhailovich Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Franz Lappert, Francesco De Toni, Bernardo Garc’ia del R’io, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luisa Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Christopher Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra. 2023.  Santacoder: don’t reach for the stars!  _ArXiv_ , abs/2301.03988.
  * Allamanis et al. (2017) Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles Sutton. 2017\.  A survey of machine learning for big code and naturalness.  _ACM Computing Surveys (CSUR)_ , 51:1 – 37.
  * Allamanis et al. (2020) Miltiadis Allamanis, Earl T Barr, Soline Ducousso, and Zheng Gao. 2020.  Typilus: Neural type hints.  In _Proceedings of the 41st acm sigplan conference on programming language design and implementation_ , pages 91–105.
  * Allamanis et al. (2021) Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. 2021.  Self-supervised bug detection and repair.  _Advances in Neural Information Processing Systems_ , 34:27865–27876.
  * Allamanis and Sutton (2013) Miltiadis Allamanis and Charles Sutton. 2013.  Mining source code repositories at massive scale using language modeling.  In _2013 10th working conference on mining software repositories (MSR)_ , pages 207–216. IEEE.
  * Alon et al. (2019) Uri Alon, Omer Levy, and Eran Yahav. 2019.  code2seq: Generating sequences from structured representations of code.  In _International Conference on Learning Representations_.
  * (8) Mrinal Anand, Pratik Kayal, and Mayank Singh.  Adversarial robustness of program synthesis models.  In _Advances in Programming Languages and Neurosymbolic Systems Workshop_.
  * Asyrofi et al. (2021) Muhammad Hilmi Asyrofi, Zhou Yang, Jieke Shi, Chu Wei Quan, and David Lo. 2021.  Can differential testing improve automatic speech recognition systems? In _2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , pages 674–678.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_.
  * Bahrami et al. (2021) Mehdi Bahrami, NC Shrikanth, Yuji Mizobuchi, Lei Liu, Masahiro Fukuyori, Wei-Peng Chen, and Kazuki Munakata. 2021.  Augmentedcode: Examining the effects of natural language resources in code retrieval models.  _arXiv preprint arXiv:2110.08512_.
  * Bielik and Vechev (2020) Pavol Bielik and Martin Vechev. 2020.  Adversarial robustness for code.  In _International Conference on Machine Learning_ , pages 896–907. PMLR.
  * (13) Marc Brockschmidt, Miltiadis Allamanis, Alexander L Gaunt, and Oleksandr Polozov.  Generative code modeling with graphs.  In _International Conference on Learning Representations_.
  * Chakraborty et al. (2022) Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022.  Deep learning based vulnerability detection: Are we there yet? _IEEE Transactions on Software Engineering_ , 48(9):3280–3296.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _ArXiv_ , abs/2107.03374.
  * Chen and Lampouras (2023) Pinzhen Chen and Gerasimos Lampouras. 2023.  Exploring data augmentation for code generation tasks.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1497–1505.
  * Chen et al. (2022) Qibin Chen, Jeremy Lacomis, Edward J Schwartz, Graham Neubig, Bogdan Vasilescu, and Claire Le Goues. 2022.  Varclr: Variable semantic representation pre-training via contrastive learning.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 2327–2339.
  * Cheng et al. (2022) Xiao Cheng, Guanqin Zhang, Haoyu Wang, and Yulei Sui. 2022.  Path-sensitive code embedding via contrastive learning for software vulnerability detection.  In _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ , pages 519–531.
  * Collard et al. (2013) Michael L Collard, Michael John Decker, and Jonathan I Maletic. 2013.  srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration.  In _2013 IEEE International conference on software maintenance_ , pages 516–519. IEEE.
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020.  Randaugment: practical automated data augmentation with a reduced search space.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , pages 18613–18624.
  * Dao et al. (2019) Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. 2019.  A kernel theory of modern data augmentation.  In _International Conference on Machine Learning_ , pages 1528–1537. PMLR.
  * de Paula Rodrigues et al. (2023) Gustavo Eloi de Paula Rodrigues, Alexandre M Braga, and Ricardo Dahab. 2023.  Detecting cryptography misuses with machine learning: Graph embeddings, transfer learning and data augmentation in source code related tasks.  _IEEE Transactions on Reliability_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805.
  * Devore-McDonald and Berger (2020) Breanna Devore-McDonald and Emery D Berger. 2020.  Mossad: Defeating software plagiarism detection.  _Proceedings of the ACM on Programming Languages_ , 4(OOPSLA):1–28.
  * Dhole et al. (2021) Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, Jascha Sohl-Dickstein, Jinho D. Choi, Eduard Hovy, Ondrej Dusek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade, Lisa Barthe, Hanna Behnke, Ian Berlot-Attwell, Connor Boyle, Caroline Brun, Marco Antonio Sobrevilla Cabezudo, Samuel Cahyawijaya, Emile Chapuis, Wanxiang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo, Filip Cornell, Gautier Dagan, Mayukh Das, Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Suchitra Dubey, Tatiana Ekeinhor, Marco Di Giovanni, Rishabh Gupta, Rishabh Gupta, Louanes Hamla, Sang Han, Fabrice Harel-Canada, Antoine Honore, Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko, Venelin Kovatchev, Kalpesh Krishna, Ashutosh Kumar, Stefan Langer, Seungjae Ryan Lee, Corey James Levinson, Hualou Liang, Kaizhao Liang, Zhexiong Liu, Andrey Lukyanenko, Vukosi Marivate, Gerard de Melo, Simon Meoni, Maxime Meyer, Afnan Mir, Nafise Sadat Moosavi, Niklas Muennighoff, Timothy Sum Hon Mun, Kenton Murray, Marcin Namysl, Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan Pfister, Richard Plant, Vinay Prabhu, Vasile Pais, Libo Qin, Shahab Raji, Pawan Kumar Rajpoot, Vikas Raunak, Roy Rinberg, Nicolas Roberts, Juan Diego Rodriguez, Claude Roux, Vasconcellos P. H. S., Ananya B. Sai, Robin M. Schmidt, Thomas Scialom, Tshephisho Sefara, Saqib N. Shamsi, Xudong Shen, Haoyue Shi, Yiwen Shi, Anna Shvets, Nick Siegel, Damien Sileo, Jamie Simon, Chandan Singh, Roman Sitelew, Priyank Soni, Taylor Sorensen, William Soto, Aman Srivastava, KV Aditya Srivatsa, Tony Sun, Mukund Varma T, A Tabassum, Fiona Anting Tan, Ryan Teehan, Mo Tiwari, Marie Tolkiehn, Athena Wang, Zijian Wang, Gloria Wang, Zijie J. Wang, Fuxuan Wei, Bryan Wilie, Genta Indra Winata, Xinyi Wu, Witold Wydmański, Tianbao Xie, Usama Yaseen, M. Yee, Jing Zhang, and Yue Zhang. 2021\.  Nl-augmenter: A framework for task-sensitive natural language augmentation.
  * Ding et al. (2021) Yangruibo Ding, Luca Buratti, Saurabh Pujar, Alessandro Morari, Baishakhi Ray, and Saikat Chakraborty. 2021.  Towards learning (dis)-similarity of source code from program contrasts.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Dong et al. (2023a) Zeming Dong, Qiang Hu, Yuejun Guo, Maxime Cordy, Mike Papadakis, Zhenya Zhang, Yves Le Traon, and Jianjun Zhao. 2023a.  Mixcode: Enhancing code classification by mixup-based data augmentation.  In _2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 379–390. IEEE.
  * Dong et al. (2023b) Zeming Dong, Qiang Hu, Yuejun Guo, Zhenya Zhang, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Jianjun Zhao. 2023b.  Boosting source code learning with data augmentation: An empirical study.  _arXiv preprint arXiv:2303.06808_.
  * Fan et al. (2020) Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020.  Ac/c++ code vulnerability dataset with code changes and cve summaries.  In _Proceedings of the 17th International Conference on Mining Software Repositories_ , pages 508–512.
  * Feng et al. (2021) Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A survey of data augmentation approaches for nlp.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 968–988.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020.  Codebert: A pre-trained model for programming and natural languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1536–1547.
  * Gu et al. (2016) Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016.  Deep api learning.  In _Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering_ , pages 631–642.
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021.  Graphcode{bert}: Pre-training code representations with data flow.  In _International Conference on Learning Representations_.
  * Hajipour et al. (2022) Hossein Hajipour, Ning Yu, Cristian-Alexandru Staicu, and Mario Fritz. 2022.  Simscood: Systematic analysis of out-of-distribution behavior of source code models.  _arXiv preprint arXiv:2210.04802_.
  * Hellendoorn et al. (2018) Vincent J Hellendoorn, Christian Bird, Earl T Barr, and Miltiadis Allamanis. 2018\.  Deep learning type inference.  In _Proceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering_ , pages 152–162.
  * Henke et al. (2022) Jordan Henke, Goutham Ramakrishnan, Zi Wang, Aws Albarghouth, Somesh Jha, and Thomas Reps. 2022.  Semantic robustness of models of source code.  In _2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 526–537. IEEE.
  * Hu et al. (2022) Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Lei Ma, Mike Papadakis, and Yves Le Traon. 2022.  Codes: A distribution shift benchmark dataset for source code learning.  _arXiv preprint arXiv:2206.05480_.
  * Hu et al. (2018) Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018.  Summarizing source code with transferred api knowledge.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pages 2269–2275.
  * Huang et al. (2021) Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021.  Cosqa: 20,000+ web queries for code search and question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5690–5700.
  * Huang et al. (2018) Qiao Huang, Xin Xia, Zhenchang Xing, David Lo, and Xinyu Wang. 2018.  Api method recommendation without worrying about the task-api knowledge gap.  In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , pages 293–304.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  _arXiv preprint arXiv:1909.09436_.
  * Jain et al. (2021) Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021.  Contrastive code representation learning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5954–5971.
  * Jia et al. (2023) Jinghan Jia, Shashank Srikant, Tamara Mitrovska, Chuang Gan, Shiyu Chang, Sijia Liu, and Una-May O’Reilly. 2023.  Clawsat: Towards both robust and accurate code models.  In _2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 212–223. IEEE.
  * Karmakar and Robbes (2021) Anjan Karmakar and Romain Robbes. 2021.  What do pre-trained code models know about code?  In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , pages 1332–1336. IEEE.
  * Li et al. (2022a) Haochen Li, Chunyan Miao, Cyril Leung, Yanxian Huang, Yuan Huang, Hongyu Zhang, and Yanlin Wang. 2022a.  Exploring representation-level augmentation for code search.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4924–4936, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nourhan Fahmy, Urvashi Bhattacharyya, W. Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jana Ebert, Tri Dao, Mayank Mishra, Alexander Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean M. Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023.  Starcoder: may the source be with you!  _ArXiv_ , abs/2305.06161.
  * Li et al. (2022b) Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, and Nan Duan. 2022b.  Coderetriever: A large scale contrastive pre-training method for code search.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2898–2910.
  * Li et al. (2022c) Yiyang Li, Hongqiu Wu, and Hai Zhao. 2022c.  Semantic-preserving adversarial code comprehension.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3017–3028.
  * Li et al. (2022d) Zhen Li, Guenevere Chen, Chen Chen, Yayi Zou, and Shouhuai Xu. 2022d.  Ropgen: Towards robust code authorship attribution via automatic coding style transformation.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 1906–1918.
  * Li et al. (2022e) Zhiming Li, Xiaofei Xie, Haoliang Li, Zhengzi Xu, Yi Li, and Yang Liu. 2022e.  Cross-lingual transfer learning for statistical type inference.  In _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ , pages 239–250.
  * Li et al. (2022f) Zongjie Li, Pingchuan Ma, Huaijin Wang, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. 2022f.  Unleashing the power of compiler intermediate representation to enhance neural program embeddings.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 2253–2265.
  * Liu and Wan (2021) Chenxiao Liu and Xiaojun Wan. 2021.  Codeqa: A question answering dataset for source code comprehension.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2618–2632.
  * Liu et al. (2023a) Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2023a.  Matcha: Enhancing visual language pretraining with math reasoning and chart derendering.  In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * (54) Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu.  Retrieval-augmented generation for code summarization via hybrid gnn.  In _International Conference on Learning Representations_.
  * Liu et al. (2023b) Shangqing Liu, Bozhi Wu, Xiaofei Xie, Guozhu Meng, and Yang Liu. 2023b.  Contrabert: Enhancing code pre-trained models via contrastive learning.
  * Lu et al. (2022) Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022.  Reacc: A retrieval-augmented code completion framework.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6227–6240.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. 2021.  CodeXGLUE: A machine learning benchmark dataset for code understanding and generation.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * (58) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _International Conference on Learning Representations_.
  * Mi et al. (2021) Qing Mi, Yan Xiao, Zhi Cai, and Xibin Jia. 2021.  The effectiveness of data augmentation in code readability classification.  _Information and Software Technology_ , 129:106378.
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023.  Augmented language models: a survey.  _arXiv preprint arXiv:2302.07842_.
  * Miceli-Barone and Sennrich (2017) Antonio Valerio Miceli-Barone and Rico Sennrich. 2017.  A parallel corpus of python functions and documentation strings for automated code documentation and code generation.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 314–319.
  * Mockus et al. (2002) Audris Mockus, Roy T Fielding, and James D Herbsleb. 2002.  Two case studies of open source software development: Apache and mozilla.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , 11(3):309–346.
  * Nashid et al. (2023) Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023.  Retrieval-based prompt selection for code-related few-shot learning.  In _Proceedings of the 45th International Conference on Software Engineering (ICSE’23)_.
  * Odena et al. (2017) Augustus Odena, Christopher Olah, and Jonathon Shlens. 2017.  Conditional image synthesis with auxiliary classifier gans.  In _International conference on machine learning_ , pages 2642–2651. PMLR.
  * Orlanski et al. (2023) Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishah Singh, and Michele Catasta. 2023.  Measuring the impact of programming language distribution.  _arXiv preprint arXiv:2302.01973_.
  * Orvalho et al. (2022) Pedro Orvalho, Mikoláš Janota, and Vasco Manquinho. 2022.  Multipas: applying program transformations to introductory programming assignments for data augmentation.  In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , pages 1657–1661.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Park et al. (2023) Shinwoo Park, Youngwook Kim, and Yo-Sub Han. 2023.  Contrastive learning with keyword-based data augmentation for code search and code question answering.  In _Conference of the European Chapter of the Association for Computational Linguistics_.
  * Parvez et al. (2021) Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021.  Retrieval augmented code generation and summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2719–2734.
  * Pinku et al. (2023) Subroto Nag Pinku, Debajyoti Mondal, and Chanchal K Roy. 2023.  Pathways to leverage transcompiler based data augmentation for cross-language clone detection.  _arXiv preprint arXiv:2303.01435_.
  * Pour et al. (2021) Maryam Vahdat Pour, Zhuo Li, Lei Ma, and Hadi Hemmati. 2021.  A search-based testing framework for deep neural networks of source code embedding.  In _2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)_ , pages 36–46. IEEE.
  * Puri et al. (2021) Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021\.  Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks.  In _Annual Conference on Neural Information Processing Systems_.
  * Quiring et al. (2019) Erwin Quiring, Alwin Maier, Konrad Rieck, et al. 2019.  Misleading authorship attribution of source code using adversarial learning.  In _USENIX Security Symposium_ , pages 479–496.
  * Rabin and Alipour (2022) Md Rafiqul Islam Rabin and Mohammad Amin Alipour. 2022.  Programtransformer: A tool for generating semantically equivalent transformed programs.  _Software Impacts_ , 14:100429.
  * Raychev et al. (2016) Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016.  Probabilistic model for code with decision trees.  _ACM SIGPLAN Notices_ , 51(10):731–747.
  * Ren et al. (2020) Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020.  Codebleu: a method for automatic evaluation of code synthesis.  _arXiv preprint arXiv:2009.10297_.
  * Roziere et al. (2020) Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020\.  Unsupervised translation of programming languages.  _Advances in Neural Information Processing Systems_ , 33:20601–20611.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * (80) Yiheng Shen, Xiaolin JU, Xiang Chen, and Guang Yang.  Bash comment generation via data augmentation and semantic-aware codebert.  _Available at SSRN 4385791_.
  * Shi et al. (2023) Ensheng Shi, Yanlin Wang, Wenchao Gu, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2023.  Cocosoda: Effective contrastive learning for code search.  In _Proceedings of the 45th International Conference on Software Engineering_.
  * Shi et al. (2022a) Yiwen Shi, Taha ValizadehAslani, Jing Wang, Ping Ren, Yi Zhang, Meng Hu, Liang Zhao, and Hualou Liang. 2022a.  Improving imbalanced learning by pre-finetuning with data augmentation.  In _Fourth International Workshop on Learning with Imbalanced Domains: Theory and Applications_ , pages 68–82. PMLR.
  * Shi et al. (2022b) Zejian Shi, Yun Xiong, Xiaolong Zhang, Yao Zhang, Shanshan Li, and Yangyong Zhu. 2022b.  Cross-modal contrastive learning for code search.  In _2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , pages 94–105. IEEE.
  * Shiri et al. (2022) Fatemeh Shiri, Terry Yue Zhuo, Zhuang Li, Shirui Pan, Weiqing Wang, Reza Haffari, Yuan-Fang Li, and Van Nguyen. 2022.  Paraphrasing techniques for maritime qa system.  In _2022 25th International Conference on Information Fusion (FUSION)_ , pages 1–8. IEEE.
  * Shorten and Khoshgoftaar (2019) Connor Shorten and Taghi M. Khoshgoftaar. 2019.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 6:1–48.
  * Song et al. (2022) Zixuan Song, Xiuwei Shang, Mengxuan Li, Rong Chen, Hui Li, and Shikai Guo. 2022\.  Do not have enough data? an easy data augmentation for code summarization.  In _2022 IEEE 13th International Symposium on Parallel Architectures, Algorithms and Programming (PAAP)_ , pages 1–6. IEEE.
  * Springer et al. (2021) Jacob M. Springer, Bryn Marie Reinstadler, and Una-May O’Reilly. 2021.  Strata: Simple, gradient-free attacks for models of code.
  * (88) Shashank Srikant, Sijia Liu, Tamara Mitrovska, Shiyu Chang, Quanfu Fan, Gaoyuan Zhang, and Una-May OŔeilly.  Generating adversarial computer programs using optimized obfuscations.  In _International Conference on Learning Representations_.
  * Stahlberg (2020) Felix Stahlberg. 2020.  Neural machine translation: A review.  _Journal of Artificial Intelligence Research_ , 69:343–418.
  * Surís et al. (2023) Dídac Surís, Sachit Menon, and Carl Vondrick. 2023.  Vipergpt: Visual inference via python execution for reasoning.  _arXiv preprint arXiv:2303.08128_.
  * Svajlenko et al. (2014) Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. 2014.  Towards a big data curated benchmark of inter-project code clones.  In _2014 IEEE International Conference on Software Maintenance and Evolution_ , pages 476–480. IEEE.
  * Tian et al. (2021) Junfeng Tian, Chenxin Wang, Zhen Li, and Yu Wen. 2021.  Generating adversarial examples of source code classification models via q-learning-based markov decision process.  In _2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS)_ , pages 807–818. IEEE.
  * Treude and Robillard (2016) Christoph Treude and Martin P Robillard. 2016.  Augmenting api documentation with insights from stack overflow.  In _Proceedings of the 38th International Conference on Software Engineering_ , pages 392–403.
  * Wan et al. (2022) Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. 2022.  What do they capture? a structural analysis of pre-trained language models for source code.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 2377–2388.
  * Wan et al. (2018) Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. 2018.  Improving automatic source code summarization via deep reinforcement learning.  In _Proceedings of the 33rd ACM/IEEE international conference on automated software engineering_ , pages 397–407.
  * Wang et al. (2022a) Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong, and Xiangke Liao. 2022a.  Bridging pre-trained models and downstream tasks for source code understanding.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 287–298.
  * Wang et al. (2023) Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2023.  Recode: Robustness evaluation of code generation models.  In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Wang et al. (2021a) Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021a.  Want to reduce labeling cost? gpt-3 can help.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4195–4205.
  * Wang et al. (2022b) Xiao Wang, Qiong Wu, Hongyu Zhang, Chen Lyu, Xue Jiang, Zhuoran Zheng, Lei Lyu, and Songlin Hu. 2022b.  Heloc: Hierarchical contrastive learning of source code representation.  In _Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension_ , pages 354–365.
  * Wang et al. (2022c) Xin Wang, Xiao Liu, Pingyi Zhou, Qixia Liu, Jin Liu, Hao Wu, and Xiao Cui. 2022c.  Test-driven multi-task learning with functionally equivalent code transformation for neural code generation.  _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_.
  * Wang et al. (2021b) Zeyu Wang, Sheng Huang, Zhongxin Liu, Meng Yan, Xin Xia, Bei Wang, and Dan Yang. 2021b.  Plot2api: recommending graphic api from plot via semantic parsing guided neural network.  In _2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 458–469. IEEE.
  * Wen et al. (2020) Qingsong Wen, Liang Sun, Xiaomin Song, Jing Gao, Xue Wang, and Huan Xu. 2020.  Time series data augmentation for deep learning: A survey.  In _International Joint Conference on Artificial Intelligence_.
  * Wu et al. (2020) Sen Wu, Hongyang Zhang, Gregory Valiant, and Christopher Ré. 2020.  On the generalization effects of linear transformations in data augmentation.  In _International Conference on Machine Learning_ , pages 10410–10420. PMLR.
  * Xu et al. (2020) Frank F Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020\.  Incorporating external knowledge through pre-training for natural language to code generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6045–6052.
  * Yang et al. (2023) Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Tingting Han, and Taolue Chen. 2023\.  Exploitgen: Template-augmented exploit code generation based on codebert.  _Journal of Systems and Software_ , 197:111577.
  * Yang et al. (2022a) Guang Yang, Yu Zhou, Wenhua Yang, Tao Yue, Xiang Chen, and Taolue Chen. 2022a.  How important are good method names in neural code generation? a model robustness perspective.  _arXiv preprint arXiv:2211.15844_.
  * Yang et al. (2022b) Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022b.  Natural attack for pre-trained models of code.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 1482–1493.
  * Yefet et al. (2020) Noam Yefet, Uri Alon, and Eran Yahav. 2020.  Adversarial examples for models of code.  _Proceedings of the ACM on Programming Languages_ , 4(OOPSLA):1–30.
  * Yoo et al. (2021) Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyoung Park. 2021\.  Gpt3mix: Leveraging large-scale language models for text augmentation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2225–2239.
  * Yu et al. (2022a) Chi Yu, Guang Yang, Xiang Chen, Ke Liu, and Yanlin Zhou. 2022a.  Bashexplainer: Retrieval-augmented bash code comment generation based on fine-tuned codebert.  In _2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , pages 82–93. IEEE.
  * Yu et al. (2022b) Shiwen Yu, Ting Wang, and Ji Wang. 2022b.  Data augmentation by program transformation.  _Journal of Systems and Software_ , 190:111304.
  * (112) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_.
  * Zhang et al. (2020a) Huangzhao Zhang, Zhuo Li, Ge Li, Lei Ma, Yang Liu, and Zhi Jin. 2020a.  Generating adversarial examples for holding robustness of source code processing models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 1169–1176.
  * Zhang et al. (2020b) Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020b.  Retrieval-based neural source code summarization.  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , pages 1385–1397.
  * Zhang et al. (2020c) Xiaoqing Zhang, Yu Zhou, Tingting Han, and Taolue Chen. 2020c.  Training deep code comment generation models via data augmentation.  In _Proceedings of the 12th Asia-Pacific Symposium on Internetware_ , pages 185–188.
  * Zhang et al. (2022) Yifan Zhang, Chen Huang, Yueke Zhang, Kevin Cao, Scott Thomas Andersen, Huajie Shao, Kevin Leach, and Yu Huang. 2022.  Combo: Pre-training representations of binary code using contrastive learning.  _arXiv preprint arXiv:2210.05102_.
  * Zheng et al. (2021) Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su. 2021.  D2a: A dataset built for ai-based vulnerability detection methods using differential analysis.  In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_ , pages 111–120. IEEE.
  * Zhou et al. (2019) Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.  Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks.  _Advances in neural information processing systems_ , 32.
  * Zhou et al. (2022) Yu Zhou, Xiaoqing Zhang, Juanjuan Shen, Tingting Han, Taolue Chen, and Harald Gall. 2022.  Adversarial robustness of deep code comment generation.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , 31(4):1–30.
  * Zhuo (2023) Terry Yue Zhuo. 2023.  Large language models are state-of-the-art evaluators of code generation.  _arXiv preprint arXiv:2304.14317_.
  * Zhuo et al. (2023a) Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023a.  Exploring ai ethics of chatgpt: A diagnostic analysis.  _arXiv preprint arXiv:2301.12867_.
  * Zhuo et al. (2023b) Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, and Yuan-Fang Li. 2023b.  On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1090–1102.
  * Zubkov et al. (2022) Maksim Zubkov, Egor Spirin, Egor Bogomolov, and Timofey Bryksin. 2022.  Evaluation of contrastive learning with various code representations for code clone detection.  _arXiv preprint arXiv:2206.08726_.