# Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based
Offensive Code Generators

Pietro Liguori<EMAIL_ADDRESS>Cristina Improta
<EMAIL_ADDRESS>Roberto Natella<EMAIL_ADDRESS>Bojan
Cukic<EMAIL_ADDRESS>Domenico Cotroneo<EMAIL_ADDRESS>University of
Naples Federico II, Naples, Italy  University of North Carolina at Charlotte,
North Carolina, US

###### Abstract

AI-based code generators are an emerging solution for automatically writing
programs starting from descriptions in natural language, by using deep neural
networks (Neural Machine Translation, NMT). In particular, code generators
have been used for ethical hacking and offensive security testing by
generating proof-of-concept attacks. Unfortunately, the evaluation of code
generators still faces several issues. The current practice uses output
similarity metrics, i.e., automatic metrics that compute the textual
similarity of generated code with ground-truth references. However, it is not
clear what metric to use, and which metric is most suitable for specific
contexts.

This work analyzes a large set of output similarity metrics on offensive code
generators. We apply the metrics on two state-of-the-art NMT models using two
datasets containing offensive assembly and Python code with their descriptions
in the English language. We compare the estimates from the automatic metrics
with human evaluation and provide practical insights into their strengths and
limitations.

###### keywords:

AI-based Code Generators , Offensive Code , Neural Machine Translation ,
Software Security , Output Similarity Metrics

††journal: Expert Systems with Applications

##  1 Introduction

Offensive AI, i.e., the (ab)use of Artificial Intelligence (AI) to accomplish
a malicious goal, is an emerging threat for computer systems [39]. Indeed, AI
is paving the way for a new generation of offensive security techniques, by
helping adversaries to launch attacks that were not possible before. Suffice
it to think that AI has been adopted to conduct spear-phishing attacks [65,
40], to find zero-day vulnerabilities in software [42, 34], to automate
reverse engineering [8, 14], to build realistic fake personas [58, 59], and
many other malicious activities.

AI can also be applied for defensive purposes. Indeed, AI-based security code
generators is an emerging use of AI for supporting security auditors. In
general, code generators use machine learning to produce programs (code
snippets) starting from descriptions (intents) in natural language (NL). In
particular, Neural Machine Translation (NMT) is the state-of-the-art solution
using neural networks for code generation [5, 4]. In the context of ethical
hacking and offensive security testing, AI-based code generators support
auditors to develop proof-of-concept (POC) attacks, e.g., in order to assess
the severity and exploitability of software vulnerabilities and to motivate
vendors and users to adopt mitigations [6].

An underrated aspect of AI-based code generation is the evaluation of the
quality of the output generated by the models [62]. Ideally, users manually
evaluate whether the generated code correctly translates the NL intent.
Unfortunately, the human evaluation is often unfeasible due to the massive
amount of data to analyze, which makes the analysis time-consuming and prone
to human errors. Some studies addressed this problem by introducing a large
number of output similarity metrics, i.e., automatic metrics computed by
comparing the textual similarity of generated code with a ground-truth
reference. These metrics are an appealing solution to estimate the quality of
generated code since they are reproducible, easily tuned, and time-saving.
However, they do not fully reflect the correctness of the outputs since
generated code can be different from the reference but still correct (e.g.,
the assembly conditional jumps jz and je are different instructions that can
be used to perform the same operation). Furthermore, there is no clear
indication whether there is a unique metric suitable for any evaluation, or
whether a specific metric should be selected depending on the context, such as
the programming language of generated code and its application domains (e.g.,
code generators for ethical hacking). As a result, these output similarity
metrics have been used inconsistently to evaluate code generators, making it
difficult to compare the performance of different ML models.

Since the choice of the right metric may be more important than the choice of
the models used to solve the task [23], it is necessary to understand what
metrics should be used and when. This work provides a practical assessment of
the output similarity metrics commonly used to evaluate NMT models for code
generation. The key idea is to compare the output similarity metrics with the
human evaluation, in order to discuss the strengths and limitations of the
metrics and to identify the most suitable ones for different contexts.

In this work, we present an extensive analysis of automatic metrics for
evaluating security-oriented code generators. We study $23$ automatic metrics
from the literature, by applying them to evaluate two state-of-the-art NMT
models. We train and test the NMT models using two datasets of offensive
assembly and Python code annotated with descriptions in the English natural
language. Then, we estimate the performance of NMT models using the automatic
metrics and compare the results with results from human evaluation as a
reference. In summary, this work provides the following key contributions:

  1. 1.

A detailed study of the output similarity metrics most commonly used to
evaluate AI-based code generators;

  2. 2.

A systematic evaluation, including both quantitative analysis and correlation
analysis, of the output similarity metrics, by comparing them with human
evaluation;

  3. 3.

Practical insights on what metric to use and when to properly assess the
generation of offensive code. We summarize our main findings and their
implications in Table 1.

In the following, Section 2 discusses the related work; Section 3 describes
the code generation task; Section 4 shows the metrics used to evaluate the NMT
models in the code generation; Section 5 presents the case study; Section 6
shows the experimental results; Section 7 describes the results of our
analysis; Section 8 discusses the threats to validity; Section 9 concludes the
paper.

Table 1: Main findings. Analysis |  Main Findings
---|---
Statistics on the output similarity metrics |  The output similarity metrics provide very different results on the same test data, making the interpretation of the model’s performance very difficult. The metrics overestimate the performance on the assembly data, while they underestimate the performance of the models on the Python data.
Quantitative analysis on the whole test data |  Metrics based on n-grams such as ROUGE and BLEU provide estimates close to human evaluation when the number $n$ is low. The difference between automatic metrics and human evaluation always worsens when $n$ increases for Python code.
Quantitative analysis on correct and incorrect predictions |  ROUGE-4 and BLEU-4, which are among the most commonly used metrics in NMT applications, provide poor estimates when they are focused only on the evaluation of semantically-correct and semantically-incorrect generated code.
Correlation Analysis |  Exact match and edit distance are the metrics most correlated to the human evaluation for assembly and Python offensive code, respectively. ROUGE-4 and BLEU-4 have the lowest correlation regardless of the programming language.

##  2 Related Work

[64] compared the performance of several automatic evaluation metrics using a
corpus of automatically generated paraphrases. They showed that the evaluation
metrics can at least partially measure similarity in meaning, but are not good
measures for syntactic correctness. [23] compared the performance of
predictive models using design-level metrics with those that use code-level
metrics on different datasets from NASA for software fault prediction. The
authors showed that the performance of predictive models varies more as a
result of using different software metrics groups than from using different
machine learning algorithms. [60] evaluated 121 existing and new metrics,
including code-related, documentation-related, and developer-related metrics.
They assessed the correlation between each metric and code understandability.
The authors concluded that these metrics, even when combined, are not suited
to capture the complexity of code and are not suitable for practical
applications.

More recently, the arousing interest in the NMT to solve different tasks
pointed out the need for new metrics that can correlate more closely to human
evaluation to properly evaluate the translation quality of the models. [62]
compared the scores of three automatic metrics with the results of the human
evaluation. They performed an extensive empirical evaluation of the
translation task from English to five different natural languages and showed
that the automatic metrics underestimate the translation quality of the NMT.
Similarly, [61] showed that, in the task of natural language generation, the
sentence-level correlation between human and automatic metrics is low. [54]
found that the automatic metrics do not correlate well with human judgments in
the style transfer, i.e., the task of automatically transforming a piece of
text in one particular style into another, and thus should be avoided in
system development or final evaluation. [43] assessed the correlation between
the automatic metrics and human evaluation in the context of the automatic
generation of consultation notes from the verbatim transcript of the
consultations. The authors showed that all the metrics display a strong bias
toward the choice of reference. [22] conducted experiments in the automatic
generation of code documentation and pointed out the low correlation between
automatic metrics and human judgments. [57] worked in the context of code
summarization (i.e., the task of creating readable summaries describing the
functionality of the code) to provide a critical evaluation of the
applicability and interpretation of automatic metrics as evaluation
techniques. They concluded that more reliable metrics should be adopted as new
standard metrics for the evaluation.

In the field of code generation, [16] investigated what metric best correlates
to a human evaluation in assessing the quality of code generated by NMT
models. To address this problem, they considered $6$ metrics to evaluate
multiple models for generating Python code snippets from NL descriptions. [67]
used $4$ different metrics to assess the ability of an NMT model to generate
Python code from software requirements written in English. Other work on code
generation resorted to functional correctness to evaluate the quality of the
generated programs, where a code sample is considered correct if it passes a
set of unit tests. [27] used an evaluation metric based on functional
correctness to address the problem of producing correct code starting from
pseudocode. They generated k code samples per problem and assessed the ratio
of problems in which any of the k samples passed the set of unit tests. [12]
proposed pass@k, an unbiased and numerically stable implementation of this
metric. They generated $n\geq k$ samples per task ($n=200$ and $k\leq 100$),
counted the number of correct samples $c\leq n$ that pass unit tests, and
calculated an unbiased estimator to benchmark their models in the generation
of Python programs from docstrings. To estimate the functional correctness of
a program, however, a set of unit tests needs to be manually constructed. This
requires a significant effort that is often unfeasible for large amounts of
generated code.

Similar to [16, 67], we assess the correlation between the automatic and human
evaluation in the code generation, but we adopt a more exhaustive set of $23$
output similarity metrics commonly used to assess NMT models. Moreover, to the
best of our knowledge, this is the first work evaluating the automatic metrics
in the generation of code for software security applications (in both low-
level and high-level programming languages) and provides practical insights on
what metric to use and when to assess the AI-based solutions generating
offensive code. For the above-stated reasons, our work can be considered
complementary to previous studies.

##  3 Offensive Code Generation

[73] gathered question posts with tags containing terms related to software
exploits from Stack Overflow and counted the number of answers to these posts
as well as the number of answers accepted by the questioner. The final results
show that less than half of the posts contain answers that were accepted by
the questioners. This shows that, due to the specific domain of exploit code,
writing exploit code manually is a time-consuming and difficult task.
Therefore, the usage of NMT models to improve developers’ productivity and
support security auditors in the generation of PoC exploits is becoming an
attracting solution [32, 72, 73].These models generate programming code
snippets starting from NL intents.

To perform a rigorous evaluation of the output similarity metrics on the
offensive code generated by the models, we follow the best practices in the
field. Hence, we support the models with data processing operations. Data
processing is an essential step to support the NMT models in the automatic
code generation and refers to all the operations performed on the data used to
train, validate and test the models. These operations strongly depend on the
specific source and target languages to translate. The data processing steps
are usually performed both before translation (pre-processing), to train the
NMT model and prepare the input data, and after translation (post-processing),
to improve the quality and the readability of the code in output. Figure 1
summarizes the steps.

First, we use a corpus to train the NMT models. The training data is pre-
processed before being used to feed the model. The pre-processing starts with
the stopwords filtering, i.e., we remove a set of custom-compiled words (e.g.,
the, each, onto) from the intents to include only relevant data for machine
translation. Next, we use a tokenizer to break the intents into chunks of text
containing space-separated words (i.e., the tokens). To improve the
performance of the machine translation [29, 41, 31], we standardize the
intents (i.e., we reduce the randomness of the NL descriptions) by using a
named entity tagger, which returns a dictionary of standardizable tokens, such
as specific values, label names, and parameters, extracted through regular
expressions. We replace the selected tokens in every intent with “var#”, where
# denotes a number from $0$ to $|l|$, and $|l|$ is the number of tokens to
standardize. Finally, the tokens are represented as real-valued vectors using
word embedding. The pre-processed data is used to feed the NMT model. Once the
model is trained, we perform the code generation from NL. Therefore, when the
model takes as inputs new intents from the test data (i.e., data of the
corpora not used in the training phase), it generates the related code
snippets based on the knowledge inferred during the training (model’s
prediction). As for the intents, also the code snippets predicted by the
models are processed (post-processing) to improve the quality and readability
of the code. First, the dictionary of standardizable tokens is used in the de-
standardization process to replace all the “var#” with the corresponding
values, names, and parameters.

Finally, the code snippets generated during the model’s prediction are
evaluated to assess the quality of the code generation task. The evaluation
can be performed through output similarity metrics or manual analysis (human
evaluation). The former estimates the quality of the prediction by comparing
the model’s predictions with the ground truth reference in the test data, the
latter, instead, assesses if the output predicted by the model is the correct
translation of the NL intent into the generated code snippet.

Figure 1: Code generation task.

##  4 Code Generation Metrics

###  4.1 Output Similarity Metrics

Given the huge amount of data to scrutinize, which makes the human evaluation
time-consuming, the most practical and common solution to assess the
performance of the NMT models is to use metrics that estimate the similarity
between the code generated by NMT models and a ground-truth (i.e., code
snippets used as references for the evaluation). Table 2 presents the most
commonly used metrics by previous work to assess the quality of code generated
by the NMT models across multiple code-related tasks, including code
generation (i.e., natural language to code), code translation (i.e.,
programming language to different programming language), and code completion
(i.e., programming language to the same programming language). In the
following, we describe these metrics:

Table 2: Output similarity metrics used in previous work to estimate the code generated by NMT models. Output Similarity Metric |  Work |  Code Generation Task
---|---|---
Compilation Accuracy |  [13], [32] |  NL$\rightarrow$Python; NL$\rightarrow$assembly
ROUGE |  [66], [16], [13], [67] |  code completion; NL$\rightarrow$Python
BLEU |  [19], [50], [2], [71], [16], [32], [13], [67] |  NL$\rightarrow$Java; C#$\rightarrow$Java; NL$\rightarrow$Python; NL$\rightarrow$assembly
Exact Match |  [10], [19], [70], [50], [2], [71], [32] |  NL$\rightarrow$Java; NL$\rightarrow$assembly; NL$\rightarrow$Python
METEOR |  [16], [67] |  NL$\rightarrow$Python
Edit Distance |  [19], [66], [67] |  code completion; NL$\rightarrow$Python

$\blacksquare$ Compilation Accuracy (CA). It indicates whether each code
snippet produced by the model is compilable according to the syntax rules of
the target language. CA value is either $1$, when the snippet’s syntax is
correct, or $0$ otherwise.

$\blacksquare$ Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [33].
It measures the matching n-grams (i.e., the adjacent sequence of $n$ items,
such as syllables, letters, words, etc.) between the output predicted by the
model and the ground truth reference, where $n$ stands for the number of the
n-gram. The number of matching n-grams is then divided by the total number of
n-grams in the reference (recall, ROUGE R), or by the total number of n-grams
in the model’s prediction (precision, ROUGE P). The harmonic mean of precision
and recall defines $F_{1}$ Score (ROUGE $F_{1}$).

$\blacksquare$ ROUGE-L. It is a variant of the ROUGE metric commonly used to
assess code generation based on the longest common subsequence (LCS) between
the model’s output and the reference, i.e. the longest sequence of words (not
necessarily consecutive, but still in order) that is shared between both.
ROUGE-L recall, precision, and F1-score can be computed by replacing each
n-gram match with the LCS. The ROUGE metrics range between $0$ (perfect
mismatch) and $1$ (perfect matching).

$\blacksquare$ Bilingual Evaluation Understudy (BLEU) score [48]. It measures
the degree of n-gram overlapping between the string of each code snippet
produced by the model and the reference, for values of $n$ usually ranging
between $1$ and $4$ [20, 44]. This metric also takes into account a brevity
penalty to penalize predictions shorter than the references. BLEU value ranges
between $0$ and $1$, with higher scores corresponding to a better quality of
the prediction.

$\blacksquare$ Exact Match accuracy (EM). It indicates whether each code
snippet produced by the model perfectly matches the reference. EM value is $1$
when there is an exact match, $0$ otherwise.

$\blacksquare$ METEOR[28]. It measures the alignment between each code snippet
produced by the model and the reference. The alignment is defined as a mapping
between unigrams (i.e., $1$-gram), such that every unigram in each string maps
to zero or one unigram in the other string, and no unigrams in the same
string. METEOR value ranges between $0$ and $1$, with higher scores
corresponding to greater alignment between strings.

$\blacksquare$ Edit Distance (ED). It measures the edit distance between two
strings, i.e., the minimum number of operations on single characters required
to make each code snippet produced by the model equal to the reference. ED
value ranges between $0$ and $1$, with higher scores corresponding to smaller
distances.

###  4.2 Motivating Examples

Output similarity metrics cannot properly assess whether two pieces of code
are different but semantically equivalent, i.e., they provide the same output
and/or effects although they use different operations (e.g., jz label and je
label are different assembly instructions performing the same conditional
jump). For this reason, human evaluation is considered the golden standard for
assessing the quality of the code generated by the models. Through manual
inspection of the model’s predictions, human evaluation allows assessing the
deeper linguistic features of the code [21], such as the code semantics, i.e.,
what the code actually does. Therefore, for every code snippet generated by
the model, we manually assess the Semantic Correctness (SC) metric, which
indicates whether the output is the exact translation of the NL intent into
the target programming language. This evaluation does not take into account
the ground truth reference, but only the code predicted by the model and NL
intent. SC value is either $1$ when the generated snippet is the
(semantically) correct translation of the intent, and $0$ otherwise.

Unlike regular code generation tasks that focus on logically complex
functional code fragments, high-level offensive code often contains a large
number of low-level arithmetic, logic operations, and bit-level slices to hide
plain text attacks from antivirus and intrusion detection systems.

As a simple example, consider the intent “compare string s1 with string s2”,
which translates to the Python snippet:

if s1 == s2:

A semantically equivalent implementation of this string comparison is the
code:

if s1.__eq__(s2):

Despite the model’s prediction being semantically correct ($SC=1$), output
similarity metrics are not able to grasp the equivalence between the two
snippets since they base their calculation on character and/or token
similarity. Therefore, this translation results in low automatic scores,
including ROUGE-L (P: $0.66$ R: $0.5$, F1: $0.57$) and edit distance ($0.47$).

The opposite occurs with the intent “check if count modulo 2 is different from
zero”, which translates to the Python snippet:

if count % 2 != 0:

If the model generates the snippet:

if count % 2 == 0:

then prediction and reference differ by a single character, yet the code
accomplishes the opposite task. Automatic metrics fail to account for
situations like this. For instance, the edit distance between these two pieces
of code is $0.94$, while the ROUGE-L (P, R, F1) score is $0.83$, which are
considered high values. Differently, a human evaluator would appropriately
classify this snippet as semantically incorrect (i.e., $SC=0$), since it does
not perform the intended check.

Furthermore, attackers leverage low-level programming languages, such as
assembly, to perform surgically crafted exploitation of the system’s low-level
mechanisms, including heap metadata and stack return addresses, that are not
accessible through high-level programming languages. An example of an
operation involving CPU registers is the intent “transfer EAX contents into
EDX register”, which translates to the assembly snippet:

mov EDX, EAX

An alternative method to copy the contents of a register into another is by
pushing and popping its value onto the stack. Therefore, a semantically
equivalent implementation of this copy is the code:

push EAX
---
pop EDX

This translation is semantically correct when assessed by a human evaluator
($SC=1$), yet it results in low scores for output similarity metrics such as
ROUGE-L ( F1: $0.25$), BLEU-4 ($0.11$), edit distance ($0.31$) and METEOR
($0.24$).

Contrarily, there are situations in which the difference of a single character
implies the use of a different register and, therefore, the implementation of
a similar yet not equivalent operation. Indeed, consider the code description
“clear the EDX register and move 5 in the lowest byte of the register”, which
can be implemented through the assembly snippet:

xor EDX, EDX
---
mov DL, 5

If the model generates the following code:

xor EDX, EDX
---
mov BL, 5

then the semantic correctness score should be zero ($SC=0$) since the lowest
byte of EDX is stored in the DL register, while BL contains the lowest byte of
EBX. However, these two snippets are textually similar, hence resulting in
high scores for edit distance ($0.96$) and ROUGE-L F1 ($0.86$).

##  5 Experimental Setup

###  5.1 NMT Models

To perform the code generation task, we consider two standard architectures:
Seq2Seq, and CodeBERT.

$\blacksquare$ Seq2Seq is a model that maps an input of sequence to an output
of sequence. Similar to the encoder-decoder architecture with attention
mechanism [7], we use a bi-directional LSTM as the encoder to transform an
embedded intent sequence into a vector of hidden states with equal length. We
implement the Seq2Seq model using xnmt [46]. We use an Adam optimizer [26]
with $\beta_{1}=0.9$ and $\beta_{2}=0.999$, while the learning rate $\alpha$
is set to $0.001$. We set all the remaining hyper-parameters in a basic
configuration: layer dimension = $512$, layers = $1$, epochs = $200$, beam
size = $5$.

$\blacksquare$ CodeBERT [17] is a large multi-layer bidirectional Transformer
architecture [69] pre-trained on millions of lines of code across six
different programming languages. Our implementation uses an encoder-decoder
framework where the encoder is initialized to the pre-trained CodeBERT
weights, and the decoder is a transformer decoder, composed of $6$ stacked
layers. The encoder follows the RoBERTa architecture [37], with $12$ attention
heads, hidden layer dimension of $768$, $12$ encoder layers, and $514$ for the
size of position embeddings. We set the learning rate $\alpha=0.00005$, batch
size = $32$, and beam size = $10$.

During data pre-processing, we tokenize the NL intents using the nltk word
tokenizer [9] and code snippets using the Python tokenize package [53]. We use
_spaCy_ , an open-source, NL processing library written in Python and Cython
[63], to implement the named entity tagger for the standardization of the NL
intents.

###  5.2 Datasets

Table 3: Datasets statistics Statistic |  Assembly Dataset |  Python Dataset
---|---|---
Dataset size |  $3,715$ |  $15,540$
Unique Snippets |  $2,542$ |  $14,034$
Unique Intents |  $3,689$ |  $15,421$
Unique tokens (Snippets) |  $1,657$ |  $9,511$
Unique tokens (Intents) |  $1,924$ |  $10,605$
Avg. tokens per Snippet |  $4.75$ |  $11.90$
Avg. tokens per Intent |  $9.53$ |  $14.90$

We feed the NMT models with a large corpus developed by [32] and used for
security code generation and summarization [72, 73]. The dataset fits in the
context of the software security as it contains code snippets (alongside with
their descriptions in English) of code used to develop and execute shellcode
programs, i.e., piece of code used as the payload in the exploitation of a
software vulnerability. Examples of complex shellcode attacks include fork
bombs, denial of service, bind shells, etc.

The dataset enables different analyses as it contains different programming
languages. Indeed, the corpus consists of two parts: (i) a Python dataset,
which contains Python code used by exploits to encode shellcodes (i.e., to
obfuscate the execution of the shellcodes from anti-virus and intrusion
detection systems), and (ii) an assembly dataset, which includes shellcodes
and decoders to revert the encoding. The authors collected exploits from
publicly available databases, public repositories (e.g., GitHub), and
programming guidelines. A sample in the dataset consists of a snippet of code
from these exploits and their corresponding description in the English
language, as shown in Table 4. The assembly dataset includes $783$ lines
(${\sim}21\%$ of the dataset) of _multi-line intents_ , i.e., intents that
generate multiple lines of assembly code (between $2$ and $5$), separated by
the newline character \n (as the example in Table 4).

Table 3 summarizes the statistics of both datasets, including the size (i.e.,
the unique pairs of intents-snippets), the unique lines of code snippets, the
unique lines of NL intents, the unique number of tokens (i.e., words), and the
average number of tokens per snippet and intent. The statistics highlight the
difference between the datasets. Unsurprisingly, the difference in terms of
tokens per snippet ($\sim 12$ for Python, $\sim 5$ for assembly) and per
intent ($\sim 15$ for Python, $\sim 10$ for assembly) makes the code
generation task more difficult and challenging for the Python dataset. Indeed,
a low-level programming language contains a more limited amount of
instructions and operations than a high-level language such as Python. This
explains why the size of the Python data is larger than the assembly data,
i.e., to train the models to generate more complex programming code. These
differences allow us to evaluate the performance of the NMT models in
offensive code generation with different complexity.

Table 4: Examples of samples of the assembly and Python datasets Dataset |  NL Intent |  Code Snippet
---|---|---
Assembly |  Copy the ASCII string “/bin//sh” into the EBX register |  push 0x68732f2f \n push 0x6e69622f \n mov ebx, esp
Python |  val2 is the result of the bitwise xor between the integer base 16 of the element i of chunk encoded to hex and xor_byte |  val2 = int( chunk[i]. encode(’hex’), 16 ) ^ xor_byte

###  5.3 Implementation of the Metrics

To automatically assess the quality of the generated code snippets in
comparison with the ground truth, we relied on open-source tools and Python
packages. To compute the compilation accuracy, we used the Netwide Assembler
(NASM) assembler [45] for the assembly code and the py_compile [51] compiler
for Python snippets. As for the ROUGE and ROUGE-L metrics, we computed the
output similarity scores using the Python package rouge [56] for both
languages. We implemented BLEU score computation employing the bleu_score
module contained in the open-source Python suite Natural Language Toolkit
(NLTK) [47]. For the edit distance we used pylcs [52]. To calculate the METEOR
metric, we relied on the Python library evaluate by HuggingFace [15]. Finally,
for exact match accuracy, we used a simple Python string comparison.

###  5.4 Human Evaluation

To assess the semantic correctness of the predictions, we manually analyze
every code snippet generated by the models to inspect if it is the correct
translation of the NL intent, i.e., if the code snippet generated by the model
performs what the English comment states. This analysis cannot be performed
automatically (e.g., by comparing the predictions with ground-truth
references) since an English intent can be translated into different but
equivalent code snippets (as shown in § 4.2). This analysis can be subjective,
as different reviewers may have different interpretations of the code and its
intended functionality, depending on the expertise and experience of the
reviewer. This can lead to inconsistent assessments of code correctness.
Moreover, manual analysis is prone to human error, as reviewers may miss
subtle errors or inconsistencies in the code, or may introduce errors and
biases into their assessments due to factors such as fatigue, distractions, or
personal opinions. Therefore, to reduce the possibility of errors and
inconsistency, multiple authors performed the manual analysis independently
and discussed the cases of discrepancy, obtaining a consensus for the semantic
correctness of the predictions.

##  6 Experimental Results

Experiments aim to assess what are the output similarity metrics that are
closer to the human evaluation in the generation of assembly and Python code.
To perform the experiments, we split the dataset into training (the set of
examples used to fit the parameters), validation (the set used to tune the
hyperparameters of the models), and test (the set used for the evaluation of
the models) sets using a common $80\%/10\%/10\%$ ratio [25, 38, 30].

###  6.1 Quantitative Analysis

First, we investigated whether the output similarity metrics provide similar
results on the same data. To this aim, we assessed the performance of the NMT
models on the assembly and Python test data. Figure 2 shows the statistics, in
terms of boxplots, of the output similarity metrics. The height of both
boxplots highlights that the metrics provide very different results. Indeed,
we found that for the assembly data, the min value is $17\%$, the median is
$60\%$ (the average is $56\%$, the standard deviation is $21\%$), and the max
value is $87\%$. For the Python data, the min value is $5\%$, the median is
$45\%$ (the average is $46\%$, the standard deviation is $24\%$) and the max
is $91\%$. Therefore, different metrics used for code generation provide very
different values, leading to a wrong assessment of the performance of the
models.

The figure also shows the SC values (X markers), which are $53\%$ and $59\%$
for assembly and Python data, respectively (i.e., the data is pretty
balanced). Ideally, output similarity metrics should provide a value closer to
the one of semantic correctness. We found that the median value of the output
similarity metrics is $7\%$ higher than the human evaluation for assembly
data, while is $14\%$ lower than the human evaluation for Python data, i.e.,
the metrics overestimate the performance for the assembly and underestimate
the performance for the Python data. We attribute this behavior to the
different structures of the assembly and Python code. Indeed, the limited set
of instructions of a low-level language makes the code snippets similar even
when they are semantically equivalent (e.g., jz label and jnz label are
similar but semantically different instructions). This is not the case for the
Python dataset, which has a higher complexity in terms of different
instructions, length of the code, etc. Therefore, it is more likely to write
equivalent code with very different instructions.

Figure 2: Distribution of the output similarity metrics on the assembly and
Python data. The marker X indicates the semantic correctness value.

We then performed an in-depth analysis by comparing each output similarity
metric with the human evaluation, i.e., the SC. Specifically, for each
dataset, we performed three different analyses, depending on the code snippets
included in the test set for the evaluation: i) the whole test set, i.e, we
considered all the code snippets in the test data, ii) only the correct
predictions, i.e., we limited the analysis to the code snippets considered
correct according to the human evaluation, and iii) only the wrong
predictions, i.e., the analysis entailed only the code snippets considered as
semantically incorrect by the human evaluation. The analysis of the correct
predictions and wrong predictions enables different considerations, e.g., to
infer what are the most suitable metrics to use when the models provide very
accurate predictions or when they deal with very challenging code generation
tasks. To this aim, we computed an offset value, i.e., the distance between
the optimal value represented by the human evaluation (i.e., the semantic
correctness) and the estimate provided by the output similarity metrics: the
lower the offset, the closer the automatic metric is to the human evaluation.

Table 5: Average values and offset of the output similarity metrics on the assembly dataset. Best performance (lower offset) is blue, while the worst performance (higher offset) is red. Metric | Whole | Correct | Wrong
---|---|---|---
|  Value |  Offset |  Value |  Offset |  Value |  Offset
SC |  0.53 |  - |  1.00 |  - |  0.00 |  -
CA |  0.87 |  0.34  |  1.00 |  0.00  |  0.72 |  0.72
ROUGE-1 P |  0.75 |  0.22  |  0.93 |  0.07  |  0.55 |  0.55
ROUGE-1 R |  0.71 |  0.18  |  0.92 |  0.08  |  0.47 |  0.47
ROUGE-1 $F_{1}$ |  0.72 |  0.19  |  0.92 |  0.08  |  0.50 |  0.50
ROUGE-2 P |  0.55 |  0.02  |  0.75 |  0.25  |  0.32 |  0.32
ROUGE-2 R |  0.52 |  0.01  |  0.74 |  0.26  |  0.28 |  0.28
ROUGE-2 $F_{1}$ |  0.53 |  0.00  |  0.74 |  0.26  |  0.28 |  0.28
ROUGE-3 P |  0.40 |  0.12  |  0.60 |  0.40  |  0.19 |  0.19
ROUGE-3 R |  0.38 |  0.14  |  0.59 |  0.41  |  0.16 |  0.16
ROUGE-3 $F_{1}$ |  0.39 |  0.14  |  0.59 |  0.41  |  0.16 |  0.16
ROUGE-4 P |  0.18 |  0.35  |  0.25 |  0.75  |  0.10 |  0.10
ROUGE-4 R |  0.17 |  0.36  |  0.25 |  0.75  |  0.09 |  0.09
ROUGE-4 $F_{1}$ |  0.17 |  0.36  |  0.25 |  0.75  |  0.09 |  0.09
ROUGE-L P |  0.75 |  0.22  |  0.93 |  0.07  |  0.54 |  0.54
ROUGE-L R |  0.71 |  0.18  |  0.92 |  0.08  |  0.47 |  0.47
ROUGE-L $F_{1}$ |  0.72 |  0.19  |  0.92 |  0.08  |  0.49 |  0.49
BLEU-1 |  0.69 |  0.16  |  0.89 |  0.11  |  0.53 |  0.53
BLEU-2 |  0.63 |  0.10  |  0.86 |  0.14  |  0.45 |  0.45
BLEU-3 |  0.60 |  0.07  |  0.84 |  0.16  |  0.41 |  0.41
BLEU-4 |  0.57 |  0.04  |  0.81 |  0.19  |  0.39 |  0.39
EM |  0.41 |  0.11  |  0.79 |  0.21  |  0.00 |  0.00
METEOR |  0.72 |  0.19  |  0.90 |  0.10  |  0.52 |  0.52
ED |  0.79 |  0.27  |  0.96 |  0.04  |  0.60 |  0.60
Average |  0.56 |  0.17  |  0.75 |  0.25  |  0.36 |  0.36
Table 6: Average values and offset of the output similarity metrics on the Python dataset. Best performance (lower offset) is blue, while the worst performance (higher offset) is red. Metric | Whole | Correct | Wrong
---|---|---|---
|  Value |  Offset |  Value |  Offset |  Value |  Offset
SC |  0.59 |  - |  1.00 |  - |  0.00 |  -
CA |  0.91 |  0.32  |  1.00 |  0.00  |  0.79 |  0.79
ROUGE-1 P |  0.63 |  0.04  |  0.75 |  0.25  |  0.46 |  0.46
ROUGE-1 R |  0.63 |  0.04  |  0.74 |  0.26  |  0.47 |  0.47
ROUGE-1 $F_{1}$ |  0.63 |  0.04  |  0.74 |  0.26  |  0.46 |  0.46
ROUGE-2 P |  0.45 |  0.14  |  0.59 |  0.41  |  0.23 |  0.23
ROUGE-2 R |  0.45 |  0.14  |  0.58 |  0.42  |  0.24 |  0.24
ROUGE-2 $F_{1}$ |  0.44 |  0.15  |  0.58 |  0.42  |  0.23 |  0.23
ROUGE-3 P |  0.26 |  0.33  |  0.38 |  0.62  |  0.07 |  0.07
ROUGE-3 R |  0.26 |  0.33  |  0.38 |  0.62  |  0.07 |  0.07
ROUGE-3 $F_{1}$ |  0.26 |  0.33  |  0.38 |  0.62  |  0.07 |  0.07
ROUGE-4 P |  0.05 |  0.54  |  0.07 |  0.93  |  0.02 |  0.02
ROUGE-4 R |  0.05 |  0.54  |  0.07 |  0.93  |  0.02 |  0.02
ROUGE-4 $F_{1}$ |  0.05 |  0.54  |  0.07 |  0.93  |  0.02 |  0.02
ROUGE-L P |  0.63 |  0.04  |  0.75 |  0.25  |  0.46 |  0.46
ROUGE-L R |  0.63 |  0.04  |  0.74 |  0.26  |  0.47 |  0.47
ROUGE-L $F_{1}$ |  0.63 |  0.04  |  0.74 |  0.26  |  0.46 |  0.46
BLEU-1 |  0.58 |  0.01  |  0.69 |  0.31  |  0.44 |  0.44
BLEU-2 |  0.48 |  0.11  |  0.61 |  0.39  |  0.32 |  0.32
BLEU-3 |  0.37 |  0.22  |  0.50 |  0.50  |  0.19 |  0.19
BLEU-4 |  0.26 |  0.33  |  0.37 |  0.63  |  0.13 |  0.13
EM |  0.27 |  0.32  |  0.43 |  0.57  |  0.00 |  0.00
METEOR |  0.74 |  0.16  |  0.84 |  0.16  |  0.60 |  0.60
ED |  0.81 |  0.22  |  0.91 |  0.09  |  0.66 |  0.66
Average |  0.46 |  0.22  |  0.56 |  0.44  |  0.30 |  0.30

Table 5 and Table 6 show the results, including the average values obtained by
the metrics and the offset with the SC, for the assembly and Python datasets,
respectively. In this analysis, we found that the metric closer to the
semantic correctness is ROUGE-2 (offset is $0\%$, i.e., they provide the same
value) for the assembly language, while BLEU-1 is closer to the human
evaluation on the Python code snippets (offset = $1\%$). For both assembly and
Python code snippets, the metric less similar to the SC is ROUGE-4 (offset =
$36\%$ and $54\%$, respectively). Therefore, metrics based on n-grams provide
an evaluation close to the semantic correctness when $n$ is equal to $1$ or
$2$, while their ability in the evaluation gets worse when $n\geq 3$. We
attribute this behavior to the brevity (in terms of tokens) of programming
language snippets, especially in the context of the generation of software
exploits, which require low-level instructions and binary-level-data
processing. In this case, since code frequently includes operations with few
tokens (e.g., a jmp label instruction, a Python increment with $\mathrel{+}=$
operand), output similarity metrics using a higher number of n-grams
underestimate the quality of the predictions (e.g., ROUGE-4 for assembly and
Python, BLEU-4 for Python).

When we limit the evaluation to the correct predictions, i.e., we evaluate
only the code snippets considered semantically correct ($SC=1$), then the best
metric is the compilation accuracy, regardless of the programming language.
Indeed, when the snippet is semantically correct then it is also syntactically
correct, i.e., it is also compilable. Therefore, the offset is $0\%$ in this
case. Besides the CA, the edit distance is a valuable option in this specific
case (offsets are $4\%$ and $9\%$ for assembly and Python datasets) since the
number of operations required to make the predictions equal to the reference
is limited when the predictions are correct. The worst metric, instead, is
again ROUGE-4 (offset equal to $75\%$ and $93\%$ for assembly and Python,
respectively), showing that it is not able to properly assess the correctly
generated code snippets.

The analysis on the wrong predictions, i.e., on the code snippets not
semantically correct ($SC=0$), highlights that the exact match accuracy is a
good evaluator for both datasets as the models’ predictions, in this case,
never match the ground truth references (and, therefore, the values are $0$
for every code snippet). The compilation accuracy, which showed the best
performance in the previous case study, provides the worst performance on the
wrong predictions since a semantically incorrect snippet can be syntactically
correct (i.e., compilable). Indeed, we found that $72\%$ for the assembly
dataset and $79\%$ for the Python dataset of the semantically incorrect
snippets are syntactically correct.

Finally, we found that the average values of the offsets over all the code
similarity metrics are pretty similar for both assembly and Python datasets
for the whole test set ($17\%$ vs $22\%$) and the wrong predictions ($36\%$ vs
$30\%$). For the correct predictions, instead, the differences between
assembly and Python are more exacerbated ($25\%$ vs $44\%$) due to the ability
of the high-level language to write semantically equivalent code with
different instructions.

###  6.2 Correlation Analysis

We further assessed the ability of the output similarity metrics in the
evaluation of the code generation task. Different from the previous
quantitative analysis, in which we compared the average values provided by the
output similarity metrics with the average semantic correctness over all the
test sets, we performed an in-depth statistical analysis by computing the
correlation of the output similarity metrics with the human evaluation of all
the code snippets of the assembly and Python test sets (i.e., we considered
the values of the metrics on the single predictions).

We computed the Pearson correlation coefficient $r$, which measures the
strength of association (i.e., the linear relationship) between two variables
in a correlation analysis and is defined as the covariance of the two
variables divided by the product of their respective standard deviations [49].
Moreover, to assess the relationship between the metrics and the semantic
correctness, we also computed the Kendall correlation coefficient $\tau$,
which measures the dependence between two random variables based on the ranks
of sampled observations of the variable [24].

The correlation coefficients are unit-free values between $-1$ and $1$, which
represent perfect correlations, negative, and positive, respectively. Positive
values indicate a positive correlation, i.e., the values of both variables
tend to increase together, while negative values indicate a negative
correlation, i.e., the values of one variable tend to increase when the values
of the other variable decrease. Therefore, a high value of the coefficient
indicates that the output similarity metric is strongly associated with human
evaluation. On the contrary, a small value indicates that the automatic metric
is poorly associated with human evaluation.

Table 7: Correlation Analysis between output similarity metrics and human evaluation on the assembly dataset. Best performance is blue, while the worst performance is red. | Seq2Seq | CodeBERT
---|---|---
Output Similarity Metric |  Pearson’s $r$ |  Kendall’s $\tau$ |  Pearson’s $r$ |  Kendall’s $\tau$
CA |  0.35  |  0.35  |  0.47  |  0.47
ROUGE-1 P |  0.63  |  0.63  |  0.60  |  0.62
ROUGE-1 R |  0.70  |  0.67  |  0.65  |  0.66
ROUGE-1 $F_{1}$ |  0.70  |  0.67  |  0.65  |  0.66
ROUGE-2 P |  0.49  |  0.47  |  0.53  |  0.51
ROUGE-2 R |  0.54  |  0.49  |  0.55  |  0.52
ROUGE-2 $F_{1}$ |  0.54  |  0.50  |  0.55  |  0.52
ROUGE-3 P |  0.46  |  0.42  |  0.47  |  0.44
ROUGE-3 R |  0.51  |  0.44  |  0.47  |  0.44
ROUGE-3 $F_{1}$ |  0.50  |  0.44  |  0.47  |  0.44
ROUGE-4 P |  0.19  |  0.12  |  0.22  |  0.17
ROUGE-4 R |  0.23  |  0.13  |  0.23  |  0.18
ROUGE-4 $F_{1}$ |  0.22  |  0.13  |  0.23  |  0.18
ROUGE-L P |  0.63  |  0.62  |  0.62  |  0.64
ROUGE-L R |  0.70  |  0.67  |  0.67  |  0.67
ROUGE-L $F_{1}$ |  0.69  |  0.67  |  0.66  |  0.67
BLEU-1 |  0.72  |  0.67  |  0.66  |  0.65
BLEU-2 |  0.56  |  0.51  |  0.56  |  0.53
BLEU-3 |  0.50  |  0.49  |  0.48  |  0.48
BLEU-4 |  0.22  |  0.36  |  0.23  |  0.32
EM |  0.81  |  0.81  |  0.78  |  0.78
METEOR |  0.68  |  0.61  |  0.61  |  0.57
ED |  0.72  |  0.71  |  0.68  |  0.70
Average |  0.53  |  0.50  |  0.52  |  0.51
Table 8: Correlation Analysis between output similarity metrics and human evaluation on the Python dataset. Best performance is blue, while the worst performance is red. | Seq2Seq | CodeBERT
---|---|---
Output Similarity Metric |  Pearson’s $r$ |  Kendall’s $\tau$ |  Pearson’s $r$ |  Kendall’s $\tau$
CA |  0.36  |  0.36  |  0.38  |  0.38
ROUGE-1 P |  0.49  |  0.41  |  0.50  |  0.46
ROUGE-1 R |  0.47  |  0.38  |  0.45  |  0.41
ROUGE-1 $F_{1}$ |  0.49  |  0.40  |  0.48  |  0.43
ROUGE-2 P |  0.46  |  0.40  |  0.48  |  0.42
ROUGE-2 R |  0.44  |  0.37  |  0.44  |  0.39
ROUGE-2 $F_{1}$ |  0.46  |  0.38  |  0.46  |  0.40
ROUGE-3 P |  0.31  |  0.23  |  0.43  |  0.40
ROUGE-3 R |  0.30  |  0.23  |  0.42  |  0.39
ROUGE-3 $F_{1}$ |  0.31  |  0.23  |  0.43  |  0.40
ROUGE-4 P |  0.15  |  0.12  |  0.10  |  0.04
ROUGE-4 R |  0.14  |  0.12  |  0.09  |  0.04
ROUGE-4 $F_{1}$ |  0.15  |  0.12  |  0.10  |  0.04
ROUGE-L P |  0.50  |  0.42  |  0.51  |  0.46
ROUGE-L R |  0.48  |  0.38  |  0.45  |  0.41
ROUGE-L $F_{1}$ |  0.50  |  0.41  |  0.49  |  0.44
BLEU-1 |  0.43  |  0.36  |  0.48  |  0.42
BLEU-2 |  0.45  |  0.37  |  0.47  |  0.40
BLEU-3 |  0.33  |  0.29  |  0.44  |  0.35
BLEU-4 |  0.13  |  0.20  |  0.09  |  0.13
EM |  0.42  |  0.42  |  0.54  |  0.54
METEOR |  0.38  |  0.32  |  0.55  |  0.53
ED |  0.57  |  0.49  |  0.57  |  0.56
Average |  0.38  |  0.32  |  0.41  |  0.37

Table 7 and Table 8 show Pearson’s $r$ and Kendall’s $\tau$ correlation
coefficients between the automatic metrics and the semantic correctness on the
assembly and Python datasets, respectively, for both the Seq2Seq and CodeBERT
models. For the assembly dataset, we found that the exact match has the
highest correlation coefficients for both models ($r$ and $\tau$ are $0.81$
for Seq2Seq and $0.78$ for CodeBERT). We attribute this result to the nature
of the assembly language, which has a fixed structure and provides a more
limited set of instructions to express an operation if compared to a high-
level language. Therefore, the exact match, which provides $1$ only when the
prediction is equal to the reference, results to be the most suitable metric
for this case study.

For the Python dataset, the edit distance is the most correlated metric with
the human evaluation ($r=0.57$ and $\tau=0.49$ for Seq2Seq, while $r=0.57$ and
$\tau=0.56$ for CodeBERT), although the coefficients are lower than the best
values of the assembly case study. As above mentioned, Python code for
software exploits requires a considerable amount of binary-level-data
processing and concise instructions. If a single character in a hexadecimal
value is not correct, then, for n-gram-based metrics such as ROUGE, the whole
token is different from the reference, resulting in low scores. Differently,
character-based metrics such as the edit distance account for these slight
deviations and result in a high score.

Again, ROUGE-4 metrics provide the worst results for both datasets. For the
assembly language, the correlation coefficients are $r\leq 0.23$ and $\tau\leq
0.13$ for Seq2Seq, while $r\leq 0.23$ and $\tau\leq 0.18$ for CodeBERT. For
the Python code snippets, the correlation coefficients are even lower ($r\leq
0.15$, $\tau\leq 0.12$). This result confirms that n-gram-based metrics are
lowly correlated to the human evaluation when $n$ is high.

The comparison of the correlation coefficients on the assembly and Python
datasets highlights that the code similarity metrics are more correlated to
the assembly case study ($r\geq 0.52$, $\tau\geq 0.50$) than the Python one
($r\leq 0.41$, $\tau\leq 0.37$) due to the increasing difficulty of the code
similarity metrics to assess the generation of more complex offensive code.

##  7 Discussion

Table 9: Examples that show how different output similarity metrics work for different programming languages. Red refers to incorrect predictions. Dataset |  Ground Truth |  Predicted Code |  SC |  ROUGE-4 (F1) |  ED |  EM
---|---|---|---|---|---|---
|  add EAX, EBX |  add EAX, EBX |  $1.0$ |  $0.0$ |  $1.0$ |  $1.0$
Assembly |  xor ECX, ECX \n mul ECX |  xor ECX, ECX \n mul EBX |  $0.0$ |  $0.66$ |  $0.95$ |  $0.0$
|  jmp decode |  jmp decode |  $1.0$ |  $0.0$ |  $1.0$ |  $1.0$
|  break |  sys.exit() |  $0.0$ |  $0.0$ |  $0.1$ |  $0.0$
Python |  for byte in encoder: |  for bytes in encoder: |  $1.0$ |  $0.0$ |  0.95 |  $0.0$
|  encoded = "\\\x" |  encoded = ‘\\\x’ |  $1.0$ |  $0.0$ |  $0.87$ |  $0.0$

Our analysis highlights that n-gram-based metrics like ROUGE and BLEU, which
are commonly used to assess code generation tasks, are not the best choice to
evaluate offensive code. Indeed, the exact match and the edit distance are the
most correlated to human evaluation for the assembly and Python code,
respectively. Assembly instructions are typically characterized by a fixed and
concise structure in the form OP DST, [SRC], which includes an opcode, the
destination of the operation, and (optionally) the source. Therefore,
increasing the n value for these metrics (i.e., 3-4) leads to lower scores
even for semantically correct snippets. The same goes for Python, which,
although being a high-level and structurally complex language, when used for
offensive purposes is characterized by concise snippets that handle numerical
values and logical operations. Table 9 shows a set of cherry-picked examples
in both assembly and Python languages and their output similarity metric
scores. We report the score for ROUGE-4, which is the least correlated metric
with human evaluation for both languages, and edit distance and exact match,
which are the most correlated for Python and assembly, as shown in § 6.2. The
first and third rows for the assembly dataset present situations in which the
predicted code matches the ground truth ($EM=1$, $ED=1$), yet the snippet is
too short to be correctly assessed by ROUGE-4. The second row shows an example
in which the reference and prediction are almost identical, but not
semantically equivalent, therefore both ROUGE-4 and edit distance give a
wrongly high score, while the exact match is correct. Also for Python, the
code does not contain 4-grams ($ROUGE$-$4=0$). The table shows two correct
examples in which ground truth and predicted code differ by one or two
characters but are equivalent, therefore the most accurate metric is the edit
distance. The first row presents an example in which the prediction is
completely inaccurate and all three similarity metrics correctly provide a low
score.

Therefore, an important takeaway of our experiments is that the choice of
metric depends on the complexity of the model-generated code. In the case of
code with a more fixed structure and more limited set of instructions (as in
the case of assembly), and, therefore, with less possibility of expressing
different but semantically equivalent snippets, then the exact match is the
best candidate. When the complexity of the generated code increases (as in the
case of Python), then the edit distance metric is an appropriate choice for
evaluating the code. Moreover, when tasks are extremely difficult, i.e., when
models fail to generate code (e.g., this is the common case of corpora not
being large enough to train the models), the exact match is again the best
metric. In the opposite case, i.e., when the models are very accurate in the
code generation, then a metric that evaluates syntactic correctness (e.g.,
compilation accuracy) is recommended.

Finally, we compared the results of our analysis with the results performed by
previous studies in the generation of general (i.e., not offensive) Python
code [16, 67]. Since there is no further existing assembly dataset for code
generation, we limited this analysis to the Python code. ROUGE-L is found to
be among the best-performing metrics for the assessment of the Python code on
both the CoNaLa dataset [74], a dataset of questions posted on Stack Overflow
with the posted solutions in Python, and the Card2code Hearthstone [35], a
dataset dedicated to generating classes that are descriptions of the cards
used in the Hearthstone game. A correlation analysis on the ReCa dataset [36],
instead, showed the METEOR is the best metric to assess syntactically
incorrect Python code generated from NL requirements. Although these metrics
act well also in the generation of offensive Python code, they do not result
to have the highest correlation in our analysis. We attribute this result to
the difference between generic code and offensive code. Indeed, unlike regular
code generation tasks that focus on logically complex functional code
fragments, high-level exploit code contains a large number of arithmetic and
logic operations, and bit-level slices (as in symmetric key cryptography) to
encode the plain exploits into new, functionally equivalent ones, but more
difficult to block by modern antivirus and intrusion detection systems.

At the end of the day, despite output similarity metrics providing estimates
close to the human evaluation, their ability to represent human assessment is
highly affected by the specific code generation task, i.e., there is not a
metric that is always suitable for the evaluation, regardless of data and
complexity of the task. Therefore, given that automatic code generation is an
area that is likely to continue to attract interest from academia and
industry, we believe there is a need for a solution that can automatically
evaluate the semantic correctness of code generated by ML models.

##  8 Threats to Validity

AI-based code generators: We performed our experiments employing two state-of-
the-art NMT solutions, a Seq2Seq model and a pre-trained model such as
CodeBERT. We are aware that considering only two models can be a limitation to
this evaluation, yet our choice was guided by the popularity and the
availability of mature open-source implementation of these technologies.
Seq2Seq is still largely used as a baseline model in this line of research and
remains among the most used architectures for code generation. CodeBERT, on
the other hand, represents the state-of-the-art for several code-related
tasks, such as code search and code documentation generation, and many other
software engineering tasks [78, 38, 3, 76, 77], including generation of
offensive code [73, 31]. We acknowledge that there are emerging NMT models
that are showing superior performance in different tasks, including code
generation. However, the scope of this paper is not to improve the state-of-
the-art performance in offensive code generation but to assess the ability of
the metrics in estimating offensive code correctness. We believe that both
Seq2Seq and CodeBERT fit well with the scope of the paper as they provide us
with different and adequate numbers of code snippets to properly evaluate the
metrics. Both assembly and Python code generated by the models are pretty
balanced data (in terms of semantic correctness), which allows us to perform a
fair evaluation of the metrics. Moreover, balanced data also enable the
execution of different analyses, such as the analysis of whole wrong and whole
correct data.Finally, we did not consider public AI code generators such as
GitHub Copilot and OpenAI ChatGPT, since they impose restrictions on malicious
uses [11]. Moreover, both attackers and defenders need to avoid leaking their
techniques and tactics to their counterparts (“operations security”, OPSEC).
Thus, we consider the case of an attacker or defender that builds her own AI
code generator, thus circumventing usage policies of public AI code
generators.

Dataset: This work addresses the specific problem of the automatic generation
of software exploits, focusing on the translation of NL intents into offensive
code snippets in assembly and Python programming languages. The datasets we
used in our experiments fit perfectly with the scope of this work. Indeed, to
the best of our knowledge, this is the only dataset used for code generation
in the context of software security. Moreover, we aimed to address complex and
longer programming tasks (e.g., Python and assembly for processing binary-
level data). For these tasks, NMT is still far from generating long and
complex programs from just a single high-level description. The assembly and
Python datasets considered in this work provide natural language descriptions
both at the block and statement levels that are closer to the descriptions
needed for more complex programming tasks.

Data size: We acknowledge that the corpora used in our experiments may seem
relatively small compared to other corpora available for different code-
generation tasks. These corpora contain relatively-small programs that are
described at a rather high-level (e.g., the JuICe dataset [1] includes
programming assignments with only one statement to describe the entire program
to be generated) or larger, potentially noisy, subsets of training examples
obtained by mining the web (e.g., CoNaLa mined dataset contains thousands of
training examples mined directly from StackOverflow [74]). The datasets used
for our experiments, instead, are manually curated datasets containing high-
quality and non-ambiguous descriptions of the code (not available in larger
datasets for code generation) that help us to properly estimate whether the
model’s prediction is the correct translation of the NL intent. Nevertheless,
to mitigate the bias, we leverage an existing pre-trained model such as
CodeBERT to compensate for the need for big data. Finally, it is worth
noticing that, when we limit the comparison to manually annotated datasets,
the dataset used in our experiments is way larger than the size of the CoNaLa
annotated dataset [74], which is the basis for state-of-the-art studies in NMT
for Python code generation [75, 18].

Output Similarity Metrics: We performed our evaluation by using a
comprehensive and not trivial number of metrics used by previous work to
assess the code generated by the NMT models. Nevertheless, we are aware that
the set of metrics considered in our study does not include all the available
metrics in the literature. This is the case of the code-oriented metrics
(i.e., metrics created ad-hoc for specific programming languages) proposed to
overcome the shortcomings of BLEU for code generation (e.g., CodeBLEU [55] and
RUBY [68], which were designed to evaluate code written in Java and C#).
However, these metrics rely on deeper program analysis (such as syntax and
dataflow match), which requires that the code generated by the models is
syntactically correct (i.e., compilable) and prevents the metrics from being
language-agnostic. As matter of fact, there is no available implementation of
such metrics for low-level programming languages such as assembly. Therefore,
we focused on the output similarity metrics commonly applied for the code
evaluation, which can be easily tuned and used, regardless of the code
programming language.

##  9 Conclusion

In this work, we compared the results provided by the output similarity
metrics with the human evaluation by assessing the performance of two
different state-of-the-art models in the generation of offensive assembly and
Python code snippets from natural language descriptions.

The results of our experiments provide actionable insights that can be used by
future research to assess the NMT models in the software security field.
Although we pointed out what metrics can properly assess the model’s
predictions in different case studies, there is still a gap to fill between
the automatic and the human evaluation. Unfortunately, using a simple metric
(however smart) is often not sufficient to provide accurate results.
Therefore, to have significant improvements, we believe it is necessary to
apply static and dynamic analysis techniques to automatically assess the
semantic correctness of code generated by the models.

## Acknowledgements

This work has been partially supported by the University of Naples Federico II
in the frame of the Programme F.R.A., project OSTAGE, PROT: 34938_07_04_2021,
UGOV: 000010-ALTRI_CdA_75_2021_FRA_LINEA_B_001_002, CUP: E55F21000340005.

## References

  * Agashe et al. [2019] Agashe, R., Iyer, S., Zettlemoyer, L., 2019.  Juice: A large scale distantly supervised dataset for open domain context-based code generation, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5436–5446.
  * Ahmad et al. [2021] Ahmad, W.U., Chakraborty, S., Ray, B., Chang, K., 2021.  Unified pre-training for program understanding and generation, in: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tür, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y. (Eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, Association for Computational Linguistics. pp. 2655–2668.  URL: https://doi.org/10.18653/v1/2021.naacl-main.211, doi:10.18653/v1/2021.naacl-main.211.
  * Ahmed and Devanbu [2022] Ahmed, T., Devanbu, P.T., 2022\.  Multilingual training for software engineering, in: 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, ACM. pp. 1443–1455.  URL: https://doi.org/10.1145/3510003.3510049, doi:10.1145/3510003.3510049.
  * Akinobu et al. [2022] Akinobu, Y., Kajiura, T., Obara, M., Kuramitsu, K., 2022\.  Nmt-based code generation for coding assistance with natural language.  Journal of Information Processing 30, 443–450.
  * Akinobu et al. [2021] Akinobu, Y., Obara, M., Kajiura, T., Takano, S., Tamura, M., Tomioka, M., Kuramitsu, K., 2021.  Is neural machine translation approach accurate enough for coding assistance?, in: Proceedings of the 1st ACM SIGPLAN International Workshop on Beyond Code: No Code, pp. 23–28.
  * Arce [2004] Arce, I., 2004.  The shellcode generation.  IEEE security & privacy 2, 72–76.
  * Bahdanau et al. [2015] Bahdanau, D., Cho, K., Bengio, Y., 2015.  Neural machine translation by jointly learning to align and translate, in: Bengio, Y., LeCun, Y. (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.  URL: http://arxiv.org/abs/1409.0473.
  * Bao et al. [2014] Bao, T., Burket, J., Woo, M., Turner, R., Brumley, D., 2014\.  $\\{$BYTEWEIGHT$\\}$: Learning to recognize functions in binary code, in: 23rd USENIX Security Symposium (USENIX Security 14), pp. 845–860.
  * Bird [2006] Bird, S., 2006.  Nltk: the natural language toolkit, in: Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pp. 69–72.
  * Chakraborty et al. [2022] Chakraborty, S., Ahmed, T., Ding, Y., Devanbu, P.T., Ray, B., 2022.  Natgen: generative pre-training by ”naturalizing” source code, in: Roychoudhury, A., Cadar, C., Kim, M. (Eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, ACM. pp. 18–30.  URL: https://doi.org/10.1145/3540250.3549162, doi:10.1145/3540250.3549162.
  * Check Point Blog [2023] Check Point Blog, 2023.  Russian Hackers Attempt to Bypass OpenAI’s Restrictions for Malicious Use of ChatGPT.  URL: https://blog.checkpoint.com/2023/01/13/russian-hackers-attempt-to-bypass-openais-restrictions-for-malicious-use-of-chatgpt/.
  * Chen et al. [2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F.P., Cummings, D.W., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W.H., Nichol, A., Babuschkin, I., Balaji, S.A., Jain, S., Carr, A., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M.M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., Zaremba, W., 2021\.  Evaluating large language models trained on code.  ArXiv abs/2107.03374.
  * Clement et al. [2020] Clement, C., Drain, D., Timcheck, J., Svyatkovskiy, A., Sundaresan, N., 2020.  PyMT5: multi-mode translation of natural language and python code with transformers, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online. pp. 9052–9065.  URL: https://aclanthology.org/2020.emnlp-main.728, doi:10.18653/v1/2020.emnlp-main.728.
  * Ding et al. [2019] Ding, S.H., Fung, B.C., Charland, P., 2019.  Asm2vec: Boosting static representation robustness for binary clone search against code obfuscation and compiler optimization, in: 2019 IEEE Symposium on Security and Privacy (SP), IEEE. pp. 472–489.
  * evaluate [2022] evaluate, 2022.  Python library evaluate.  URL: https://pypi.org/project/evaluate/.
  * Evtikhiev et al. [2022] Evtikhiev, M., Bogomolov, E., Sokolov, Y., Bryksin, T., 2022\.  Out of the BLEU: how should we assess quality of the code generation models?  CoRR abs/2208.03133.  URL: https://doi.org/10.48550/arXiv.2208.03133, doi:10.48550/arXiv.2208.03133, arXiv:2208.03133.
  * Feng et al. [2020] Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., Zhou, M., 2020.  Codebert: A pre-trained model for programming and natural languages, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, Association for Computational Linguistics. pp. 1536–1547.  URL: https://doi.org/10.18653/v1/2020.findings-emnlp.139, doi:10.18653/v1/2020.findings-emnlp.139.
  * Gemmell et al. [2020] Gemmell, C., Rossetto, F., Dalton, J., 2020.  Relevance transformer: Generating concise code snippets with relevance feedback, in: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, ACM. pp. 2005–2008.  URL: https://doi.org/10.1145/3397271.3401215, doi:10.1145/3397271.3401215.
  * Guo et al. [2022] Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., Yin, J., 2022.  Unixcoder: Unified cross-modal pre-training for code representation, in: Muresan, S., Nakov, P., Villavicencio, A. (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Association for Computational Linguistics. pp. 7212–7225.  URL: https://doi.org/10.18653/v1/2022.acl-long.499, doi:10.18653/v1/2022.acl-long.499.
  * Han [2016] Han, L., 2016.  Machine translation evaluation resources and methods: A survey.  arXiv preprint arXiv:1605.04515 .
  * Han et al. [2021] Han, L., Smeaton, A., Jones, G., 2021.  Translation quality assessment: A brief survey on manual and automatic methods, in: Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age, Association for Computational Linguistics, online. pp. 15–33.  URL: https://aclanthology.org/2021.motra-1.3.
  * Hu et al. [2022] Hu, X., Chen, Q., Wang, H., Xia, X., Lo, D., Zimmermann, T., 2022.  Correlating automated and human evaluation of code documentation generation quality.  ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1–28.
  * Jiang et al. [2008] Jiang, Y., Cuki, B., Menzies, T., Bartlow, N., 2008\.  Comparing design and code metrics for software quality prediction, in: Proceedings of the 4th international workshop on Predictor models in software engineering, pp. 11–18.
  * Kendall [1938] Kendall, M.G., 1938.  A new measure of rank correlation.  Biometrika 30, 81–93.
  * Kim and MacKinnon [2018] Kim, D., MacKinnon, T., 2018\.  Artificial intelligence in fracture detection: transfer learning from deep convolutional neural networks.  Clinical radiology 73, 439–445.
  * Kingma and Ba [2015] Kingma, D.P., Ba, J., 2015.  Adam: A method for stochastic optimization, in: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.  URL: http://arxiv.org/abs/1412.6980.
  * Kulal et al. [2019] Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., Liang, P.S., 2019.  Spoc: Search-based pseudocode to code.  Advances in Neural Information Processing Systems 32\.
  * Lavie and Agarwal [2007] Lavie, A., Agarwal, A., 2007\.  Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments, in: Proceedings of the Second Workshop on Statistical Machine Translation, Association for Computational Linguistics, USA. p. 228–231.
  * Li et al. [2018] Li, Z., Wang, X., Aw, A., Chng, E.S., Li, H., 2018.  Named-entity tagging and domain adaptation for better customized translation, in: Proceedings of the seventh named entities workshop, pp. 41–46.
  * Liguori et al. [2021a] Liguori, P., Al-Hossami, E., Cotroneo, D., Natella, R., Cukic, B., Shaikh, S., 2021a.  Shellcode_IA32: A dataset for automatic shellcode generation, in: Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), Association for Computational Linguistics, Online. pp. 58–64.  URL: https://aclanthology.org/2021.nlp4prog-1.7, doi:10.18653/v1/2021.nlp4prog-1.7.
  * Liguori et al. [2022] Liguori, P., Al-Hossami, E., Cotroneo, D., Natella, R., Cukic, B., Shaikh, S., 2022\.  Can we generate shellcodes via natural language? an empirical study.  Automated Software Engineering 29, 1–34.
  * Liguori et al. [2021b] Liguori, P., Al-Hossami, E., Orbinato, V., Natella, R., Shaikh, S., Cotroneo, D., Cukic, B., 2021b.  Evil: exploiting software via natural language, in: 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), IEEE. pp. 321–332.
  * Lin [2004] Lin, C.Y., 2004.  ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain. pp. 74–81.  URL: https://aclanthology.org/W04-1013.
  * Lin et al. [2020] Lin, G., Wen, S., Han, Q.L., Zhang, J., Xiang, Y., 2020\.  Software vulnerability detection using deep neural networks: a survey.  Proceedings of the IEEE 108, 1825–1848.
  * Ling et al. [2016] Ling, W., Blunsom, P., Grefenstette, E., Hermann, K.M., Kociský, T., Wang, F., Senior, A.W., 2016.  Latent predictor networks for code generation, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, The Association for Computer Linguistics.  URL: https://doi.org/10.18653/v1/p16-1057, doi:10.18653/v1/p16-1057.
  * Liu et al. [2020] Liu, H., Shen, M., Zhu, J., Niu, N., Li, G., Zhang, L., 2020.  Deep learning based program generation from requirements text: Are we there yet?  IEEE Transactions on Software Engineering 48, 1268–1289.
  * Liu et al. [2019] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V., 2019.  Roberta: A robustly optimized BERT pretraining approach.  CoRR abs/1907.11692.  URL: http://arxiv.org/abs/1907.11692, arXiv:1907.11692.
  * Mashhadi and Hemmati [2021] Mashhadi, E., Hemmati, H., 2021\.  Applying codebert for automated program repair of java simple bugs, in: 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021, Madrid, Spain, May 17-19, 2021, IEEE. pp. 505–509.  URL: https://doi.org/10.1109/MSR52588.2021.00063, doi:10.1109/MSR52588.2021.00063.
  * Mirsky et al. [2022] Mirsky, Y., Demontis, A., Kotak, J., Shankar, R., Gelei, D., Yang, L., Zhang, X., Pintor, M., Lee, W., Elovici, Y., et al., 2022\.  The threat of offensive ai to organizations.  Computers & Security , 103006.
  * Mirsky and Lee [2021] Mirsky, Y., Lee, W., 2021.  The creation and detection of deepfakes: A survey.  ACM Computing Surveys (CSUR) 54, 1–41.
  * Modrzejewski et al. [2020] Modrzejewski, M., Exel, M., Buschbeck, B., Ha, T.L., Waibel, A., 2020.  Incorporating external annotation to improve named entity translation in nmt, in: Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 45–51.
  * Mokhov et al. [2014] Mokhov, S.A., Paquet, J., Debbabi, M., 2014.  The use of nlp techniques in static code analysis to detect weaknesses and vulnerabilities, in: Canadian Conference on Artificial Intelligence, Springer. pp. 326–332.
  * Moramarco et al. [2022] Moramarco, F., Papadopoulos Korfiatis, A., Perera, M., Juric, D., Flann, J., Reiter, E., Belz, A., Savkov, A., 2022\.  Human evaluation and correlation with automatic metrics in consultation note generation, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland. pp. 5739–5754.
  * Munkova et al. [2020] Munkova, D., Hajek, P., Munk, M., Skalka, J., 2020\.  Evaluation of machine translation quality through the metrics of error rate and accuracy.  Procedia Computer Science 171, 1327–1336.
  * NASM [2022] NASM, 2022.  Netwide Assembler (NASM).  URL: https://www.nasm.us.
  * Neubig et al. [2018] Neubig, G., Sperber, M., Wang, X., Felix, M., Matthews, A., Padmanabhan, S., Qi, Y., Sachan, D., Arthur, P., Godard, P., Hewitt, J., Riad, R., Wang, L., 2018.  XNMT: The eXtensible neural machine translation toolkit, in: Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), Association for Machine Translation in the Americas, Boston, MA. pp. 185–192.  URL: https://aclanthology.org/W18-1818.
  * NLTK [2023] NLTK, 2023.  Natural Language Toolkit (NLTK), bleu_score module.  URL: https://www.nltk.org/api/nltk.translate.bleu_score.html.
  * Papineni et al. [2002] Papineni, K., Roukos, S., Ward, T., Zhu, W., 2002.  Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, ACL. pp. 311–318.  URL: https://aclanthology.org/P02-1040/, doi:10.3115/1073083.1073135.
  * Pearson [1895] Pearson, K., 1895.  Notes on regression and inheritance in the case of two parents proceedings of the royal society of london, 58, 240-242.  K Pearson .
  * Phan et al. [2021] Phan, L.N., Tran, H., Le, D., Nguyen, H., Anibal, J.T., Peltekian, A., Ye, Y., 2021\.  Cotext: Multi-task learning with code-text transformer.  CoRR abs/2105.08645.  URL: https://arxiv.org/abs/2105.08645, arXiv:2105.08645.
  * py_compile [2023] py_compile, 2023.  Python compiler py_compile.  URL: https://docs.python.org/3/library/py_compile.html.
  * pylcs [2023] pylcs, 2023.  Python library pylcs.  URL: https://pypi.org/project/pylcs/.
  * Python [2023] Python, 2023.  tokenize.  URL: https://docs.python.org/3/library/tokenize.html.
  * Rao and Tetreault [2018] Rao, S., Tetreault, J., 2018\.  Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana. pp. 129–140.
  * Ren et al. [2020] Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., Ma, S., 2020.  Codebleu: a method for automatic evaluation of code synthesis.  CoRR abs/2009.10297.  URL: https://arxiv.org/abs/2009.10297, arXiv:2009.10297.
  * rouge [2021] rouge, 2021.  Python ROUGE Score Implementation.  URL: https://pypi.org/project/rouge/.
  * Roy et al. [2021] Roy, D., Fakhoury, S., Arnaoudova, V., 2021.  Reassessing automatic evaluation metrics for code summarization tasks, in: Spinellis, D., Gousios, G., Chechik, M., Penta, M.D. (Eds.), ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, ACM. pp. 1105–1116.  URL: https://doi.org/10.1145/3468264.3468588, doi:10.1145/3468264.3468588.
  * Salminen et al. [2019] Salminen, J., Jung, S.g., Jansen, B.J., 2019.  The future of data-driven personas: A marriage of online analytics numbers and human attributes., in: ICEIS (1), pp. 608–615.
  * Salminen et al. [2020] Salminen, J., Rao, R.G., Jung, S.g., Chowdhury, S.A., Jansen, B.J., 2020.  Enriching social media personas with personality traits: A deep learning approach using the big five classes, in: International Conference on Human-Computer Interaction, Springer. pp. 101–120.
  * Scalabrino et al. [2021] Scalabrino, S., Bavota, G., Vendome, C., Linares-Vásquez, M., Poshyvanyk, D., Oliveto, R., 2021\.  Automatically assessing code understandability.  IEEE Transactions on Software Engineering 47, 595–613.  doi:10.1109/TSE.2019.2901468.
  * Shimorina [2018] Shimorina, A., 2018.  Human vs automatic metrics: on the importance of correlation design.  arXiv preprint arXiv:1805.11474 .
  * Shterionov et al. [2018] Shterionov, D., Superbo, R., Nagle, P., Casanellas, L., O’dowd, T., Way, A., 2018\.  Human versus automatic quality evaluation of nmt and pbsmt.  Machine Translation 32, 217–235.
  * spaCy [2023] spaCy, 2023.  Industrial-Strength Natural Language Processing.  URL: https://spacy.io/.
  * Stent et al. [2005] Stent, A., Marge, M., Singhai, M., 2005.  Evaluating evaluation methods for generation in the presence of variation, in: international conference on intelligent text processing and computational linguistics, Springer. pp. 341–351.
  * Stupp [2019] Stupp, C., 2019.  Fraudsters used ai to mimic ceo’s voice in unusual cybercrime case.  The Wall Street Journal 30.
  * Svyatkovskiy et al. [2020] Svyatkovskiy, A., Deng, S.K., Fu, S., Sundaresan, N., 2020\.  Intellicode compose: code generation using transformer, in: Devanbu, P., Cohen, M.B., Zimmermann, T. (Eds.), ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, ACM. pp. 1433–1443.  URL: https://doi.org/10.1145/3368089.3417058, doi:10.1145/3368089.3417058.
  * Takaichi et al. [2022] Takaichi, R., Higo, Y., Matsumoto, S., Kusumoto, S., Kurabayashi, T., Kirinuki, H., Tanno, H., 2022.  Are NLP metrics suitable for evaluating generated code?, in: Taibi, D., Kuhrmann, M., Mikkonen, T., Klünder, J., Abrahamsson, P. (Eds.), Product-Focused Software Process Improvement - 23rd International Conference, PROFES 2022, Jyväskylä, Finland, November 21-23, 2022, Proceedings, Springer. pp. 531–537.  URL: https://doi.org/10.1007/978-3-031-21388-5_38, doi:10.1007/978-3-031-21388-5\\_38.
  * Tran et al. [2019] Tran, N.M., Tran, H., Nguyen, S., Nguyen, H., Nguyen, T.N., 2019.  Does BLEU score work for code migration?, in: Guéhéneuc, Y., Khomh, F., Sarro, F. (Eds.), Proceedings of the 27th International Conference on Program Comprehension, ICPC 2019, Montreal, QC, Canada, May 25-31, 2019, IEEE / ACM. pp. 165–176.  URL: https://doi.org/10.1109/ICPC.2019.00034, doi:10.1109/ICPC.2019.00034.
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017\.  Attention is all you need, in: Advances in neural information processing systems, pp. 5998–6008.
  * Wang et al. [2022] Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., Lyu, M.R., 2022.  No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence, in: Roychoudhury, A., Cadar, C., Kim, M. (Eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, ACM. pp. 382–394.  URL: https://doi.org/10.1145/3540250.3549113, doi:10.1145/3540250.3549113.
  * Wang et al. [2021] Wang, Y., Wang, W., Joty, S.R., Hoi, S.C.H., 2021.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, in: Moens, M., Huang, X., Specia, L., Yih, S.W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, Association for Computational Linguistics. pp. 8696–8708.  URL: https://doi.org/10.18653/v1/2021.emnlp-main.685, doi:10.18653/v1/2021.emnlp-main.685.
  * Yang et al. [2022] Yang, G., Chen, X., Zhou, Y., Yu, C., 2022.  Dualsc: Automatic generation and summarization of shellcode via transformer and dual learning, in: IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022, IEEE. pp. 361–372.
  * Yang et al. [2023] Yang, G., Zhou, Y., Chen, X., Zhang, X., Han, T., Chen, T., 2023.  Exploitgen: Template-augmented exploit code generation based on codebert.  Journal of Systems and Software 197, 111577.
  * Yin et al. [2018] Yin, P., Deng, B., Chen, E., Vasilescu, B., Neubig, G., 2018\.  Learning to mine aligned code and natural language pairs from stack overflow, in: Proceedings of the 15th International Conference on Mining Software Repositories, pp. 476–486.
  * Yin and Neubig [2019] Yin, P., Neubig, G., 2019.  Reranking for neural semantic parsing, in: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Association for Computational Linguistics. pp. 4553–4559.  URL: https://doi.org/10.18653/v1/p19-1447, doi:10.18653/v1/p19-1447.
  * Yu et al. [2022] Yu, C., Yang, G., Chen, X., Liu, K., Zhou, Y., 2022\.  Bashexplainer: Retrieval-augmented bash code comment generation based on fine-tuned codebert, in: 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE. pp. 82–93.
  * Zeng et al. [2022] Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., Zhang, L., 2022.  An extensive study on pre-trained models for program understanding and generation, in: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 39–51.
  * Zhou et al. [2021] Zhou, X., Han, D., Lo, D., 2021.  Assessing generalizability of codebert, in: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE. pp. 425–436.