# DualSC: Automatic Generation and Summarization of Shellcode via Transformer
and Dual Learning

Guang Yang2, Xiang Chen231, Yanlin Zhou2, Chi Yu2  2School of Information
Science and Technology, Nantong University, China
3State Key Laboratory of Information Security, Institute of Information
Engineering, Chinese Academy of Sciences, China
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A shellcode is a small piece of code and it is executed to exploit a software
vulnerability, which allows the target computer to execute arbitrary commands
from the attacker through a code injection attack. Similar to the purpose of
automated vulnerability generation techniques, the automated generation of
shellcode can generate attack instructions, which can be used to detect
vulnerabilities and implement defensive measures. While the automated
summarization of shellcode can help users unfamiliar with shellcode and
network information security understand the intent of shellcode attacks. In
this study, we propose a novel approach DualSC to solve the automatic
shellcode generation and summarization tasks. Specifically, we formalize
automatic shellcode generation and summarization as dual tasks, use a shallow
Transformer for model construction, and design a normalization method
Adjust_QKNorm to adapt these low-resource tasks (i.e., insufficient training
data). Finally, to alleviate the out-of-vocabulary problem, we propose a rule-
based repair component to improve the performance of automatic shellcode
generation. In our empirical study, we select a high-quality corpus
Shellcode_IA32 as our empirical subject. This corpus was gathered from two
real-world projects based on the line-by-line granularity. We first compare
DualSC with six state-of-the-art baselines from the code generation and code
summarization domains in terms of four performance measures. The comparison
results show the competitiveness of DualSC. Then, we verify the effectiveness
of the component setting in DualSC. Finally, we conduct a human study to
further verify the effectiveness of DualSC.

###### Index Terms:

Program comprehension, Shellcode generation, Shellcode summarization, Shallow
Transformer, Dual learning

11footnotetext: 1 Xiang Chen is the corresponding author.

##  I Introduction

A shellcode is a small piece of code that is executed to exploit a software
vulnerability. Specifically, shellcode means writing code that returns a
remote shell when executed. The shellcode is usually written via assembly
instructions, which makes the target computer execute arbitrary instructions
from the attacker through a code injection attack, so many hackers in the
hacking community refer to the payload portion of a code injection attack as
the shellcode [1]. Shellcode is considered a key element of a security attack,
and malicious shellcodes can perform DDoS attacks, data theft, and run malware
against the target systems [2, 3, 4]. However, shellcode with good intentions
can be used to identify vulnerabilities and then repair systems by fixing
these vulnerabilities [5, 6, 7, 8]. Therefore, researchers expected to use
shellcode as the tool, which can be exploited to find security vulnerabilities
in the software, and automated shellcode generation techniques have become an
active research topic [2, 9, 1, 10, 7, 11, 12].

However, automatic shellcode generation is a challenging task as they are
usually written in assembly language and the most complex shellcode can run
into the assembly code with hundreds of lines. Moreover, it is also
challenging to summarize the functional description of shellcodes, since
identifying the attacking intent of these shellcodes can be difficult for
users who are not familiar with shellcode and network information security.
Figure 1 shows an example of a code snippet in our used corpus shellcode_IA32.
In this figure, the code snippet is on the left and the summarization (i.e.,
the code comment) based on the natural language description is on the right.
Notice in this code snippet, multiple lines of code may correspond to one
comment since some instructions do not make sense when considered in
isolation, which can make the shellcode summarization more challenging. To
construct a parallel corpus (i.e., a one-to-one mapping between code and its
corresponding comment), shellcode_IA32 break lines by “\n”. Based on the
characteristic of this corpus, we are concerned with the shellcode generation
and summarization based on the line-by-line granularity in this study.
Moreover, due to the small size of this corpus, automatic shellcode generation
and summarization can be treated as low-resource tasks, which makes solving
these two tasks more challenging.

Figure 1: The example of the code snippet and its corresponding comments
written in English from the corpus shellcode_IA32

Automated approaches to code generation and summarization tasks are of great
importance for software development and maintenance [13]. The generated high-
quality comments and code can help to improve developer productivity and then
improve software quality [14]. However, generating high-quality code or
comments is time-consuming and error-prone. Previous studies [15, 16, 17, 18,
19] on automated code generation and code summarization mainly focused on code
snippets for specific programming languages (such as Java, Python, C#).
Recently, there are also some studies, which investigate code generation and
summarization for specific software artifacts (such as Bash, SQL, and smart
contract) [20, 21, 22, 23]. To the best of our knowledge, there is only a
study [12] on automatic shellcode generation and no studies focus on automatic
shellcode summarization.

In this study, we are the first to formalize the automatic shellcode
generation and summarization as a dual learning problem. Then we propose a
fully data-driven approach DualSC to solve the above two dual tasks. Moreover,
we design a simple yet effective rule-based repair component to improve the
performance of the code generation task. Since the automatic shellcode
generation and summarization tasks are performed on insufficient training data
(i.e., the low-resource tasks) and these two tasks have a strong correlation,
we exploit the symmetric structure (i.e., code $\Leftrightarrow$ comment)
between the two tasks via dual learning and use a shallow Transformer model to
learn them simultaneously. This setting can increase the available training
data for model construction. Moreover, it can also improve the generalization
ability by learning the commonalities between these two tasks and transferring
knowledge between these two tasks through shared representation. Finally, the
method QKNorm [24] converts the self-attention computation from dot product to
cosine similarity as a way to adapt Transformer to low resource tasks.
However, cosine similarity mainly focuses on direction-based similarity (i.e.,
the closer the direction between two vectors, the higher the value of cosine
similarity). Therefore QKNorm is not sensitive to numerical differences
between two vectors. To take the numerical difference information into account
in the self-attention computation, we design the method Adjust_QKNorm.

To verify the effectiveness of our proposed approach DualSC, we chose the
high-quality corpus shellcode_IA32 [12] as our experimental subject. We
compared DualSC with six state-of-the-art baselines. Specifically, the first
three baselines are selected from the information retrieval domain and the
remaining three baselines are selected from the neural network domain
(including the recent approach for the shellcode generation task [12]). Then
we evaluated the performance of code generation in terms of three performance
measures (i.e., BLEU, Rouge-L, and Accuracy) and the performance of code
summarization in terms of three performance measures (i.e., BLEU, Rouge-L, and
METEOR). The empirical results show that DualSC can outperform these
baselines. Finally, we conduct a human study to further verify the
effectiveness of our proposed approach.

In summary, the main contributions of our study can be summarized as follows.

  * •

We formalize the automatic generation and summarization of shellcode as the
dual tasks. Then we learn these two tasks simultaneously via the shallow
Transformer and dual learning, which can share knowledge and then improve the
performance and generalization of the trained models.

  * •

We choose the corpus from real-world projects as our experimental subject.
Final comparison results with six state-of-the-art baselines show the
competitiveness of DualSC in terms of the corresponding performance measures
both in the automatic generation and summarization tasks of shellcode.

  * •

We share our scripts, the trained models, and the used
corpus111https://github.com/NTDXYG/DualSC to facilitate the replication of our
study and encourage more follow-up studies on this research topic.

##  II Related Work

In this section, we first summarize the related work for shellcode analysis.
Then we summarize the related work for code generation and code summarization.
Finally, we emphasize the novelty of our study.

###  II-A Shellcode Analysis

Arce [2] was the first to describe the development of shellcode and introduced
the working principle and defensive measures of shellcode. Borders et al. [9]
proposed Spector. This approach used symbolic execution to extract meaningful
API calls and their arguments from shellcode, which can help the users know
what the vulnerability is used for. Mason et al. [1] proposed a technique for
automatically generating shellcode using natural language generation
techniques. Chen et al. [10] designed a new type of shellcode, which is based
on JOP (Jump-Oriented Programming) gadgets. Experimental results proved that
this new JOP-based shellcode can bypass most existing ROP (Return-Oriented
Programming) defenses. Bao et al. [7] designed an automated shellcode porting
system ShellSwap for remote exploits. This system is the first end-to-end
system, which can modify the observed vulnerabilities and replace the original
shellcode with an arbitrary replacement shellcode. Patel et al. [11] treated
the writing of printable shellcodes as an important task. Then they proposed a
new encoding algorithm that used looped decoding to reduce the size of these
automatically generated shellcodes. Liguori et al. [12] constructed a corpus
Shellcode_IA32 from two real-world projects and made the first attempt to use
neural machine translation (NMT) to solve the task of automated shellcode
generation. This corpus contains a total of 3,200 pairs of instructions from
assembly language code snippets and their corresponding comments described in
English. To our best knowledge, this is the largest corpus for shellcode at
present.

###  II-B Code Generation and Code Summarization

Code generation task is an active topic in the field of mining software
repository. Recently, researchers mainly applied deep learning to code
generation. Rabinovich et al. [25] introduced abstract syntax networks, which
included modular encoder-decoder architecture that can readily accommodate a
variety of tasks with structured output spaces. Yin et al. [26] proposed a
novel neural architecture powered by a grammar model to explicitly capture the
target syntax as prior knowledge. Yin and Neubig [27] presented TRANX, a
transition-based neural semantic parser that maps natural language (NL)
utterances into formal meaning representations (MRs). Hayati et al. [28]
introduced a method RECODE, which makes it possible to explicitly reference
existing code examples within a neural code generation model. Agashe et al.
[29] presented JuICe, a corpus of 1.5 million examples with a curated test set
of 3.7K instances based on online programming assignments, to study the code
generation task. Sun et al. [30] proposed a grammar-based structural
convolutional neural network for code generation, which generates a program by
predicting the grammar rules of the programming language. Xu et al. [31]
explored the effectiveness of incorporating two varieties of external
knowledge into NL-to-code generation: automatically mined NL-code pairs from
the online programming QA forum Stack Overflow and programming language API
documentation. Zhong et al. [32] proposed a method for program generation
based on semantic scaffolds, which are lightweight structures and can
represent the high-level semantic and syntactic composition of a program. Sun
and Zhu [33] proposed a novel tree-based neural architecture TreeGen, which
uses the attention mechanism of Transformers to alleviate the long dependency
problem, and introduces a novel AST reader (i.e., encoder) to incorporate
grammar rules and AST structures into the network.

Recently, most of the previous studies on code summarization followed deep
learning-based approaches (i.e., encoder-decoder framework). For example, Iyer
et al. [34] first proposed an approach code-NN via an attention-based neural
network. Allamanis et al. [35] proposed a model in which the encoder uses CNN
and the attention mechanism, and the decoder uses GRU. Liang and Zhu [36] used
Code-RNN to encode the source code into the vectors, and then they used Code-
GRU to decode the vectors to code comments. Hu et al. [37] proposed the
approach DeepCom by analyzing abstract syntax trees (ASTs). To better present
the structure of ASTs, they proposed a new structure-based traversal (SBT)
method. Later, Hu et al. [38] further proposed the improved approach Hybrid-
DeepCom. For example, the identifiers satisfying the Camel casing naming
convention are split into multiple words to alleviate the OOV problem. Yang et
al. [39] proposed a novel method ComFormer based on Transformer and fusion
method-based hybrid code presentation. Recently, Kang et al. [40] investigated
whether using the pre-trained word embedding could improve the model
performance. They surprisingly found that using the pre-trained word embedding
based on code2vec [41] or Glove [42] could not help to improve the
performance. Leclair et al. [43] proposed the method ast-attendgru, which
combines words from code and code structure. Chen et al. [44] proposed a
neural framework, which allows bidirectional mapping between a code retrieval
task and a code comment generation task. Wei et al. [45] also utilized the
correlation between the source code summarization task and the code generation
task. Then they proposed a dual training framework. Ye et al. [46] exploited
the probabilistic correlation between the source code summarization task and
the code generation task via dual learning. Ahmad et al. [47] used the
Transformer model to generate code comments. In their study, they combined the
self-attention and the copy attention as the attention mechanism of the model
and analyzed the influence of the absolute position and pairwise relationship
on the performance of source code summarization. Liu et al. [48] proposed the
ATOM method. This method uses a hybrid sorting module, which can select the
most relevant commit messages from those that have been generated or retrieved
iteratively.

###  II-C Novelty of Our Study

In our study, we propose a fully data-driven approach DualSC to solve the
shellcode generation and shellcode summarization tasks. Considering the
characteristics of automatic shellcode generation and summarization tasks, we
formalize them as dual tasks and then use a shallow Transformer for
simultaneous learning. Moreover, we also propose a normalization method
Adjust_QKNorm for both tasks to accommodate DualSC in the low-resource
scenario. Finally, to alleviate the OOV problem, we propose a rule-based
repair component to improve the accuracy of DualSC on automatic shellcode
generation.

##  III Our Proposed Approach DualSC

In this section, we first show the framework of DualSC. Then we show the
details of model architecture in DualSC. Finally, we introduce the details of
our designed rule-based repair component in DualSC.

###  III-A Framework of DualSC

The overall framework of DualSC is shown in Figure 2. Inspired by the model T5
[49], we use the same model to learn both the primal task and the dual task.
However, due to the limitation of GPU resources, we use a shallow Transformer
model [50], which can learn with limited GPU resources. We also improve the
Self-Attention computation [51] by proposing Adjust_QKNorm to adapt the
shallow Transformer in the low-resource scenario. The key idea of DualSC is to
exploit the duality between the primal task (i.e., mapping from the domain $X$
to the domain $Y$) and the dual task (i.e., mapping from the domain $Y$ to the
domain $X$) to improve the performance of both tasks. Before the data is input
into the model, it is distinguished by a Prompt paradigm [52], where each
specific task is distinguished by a prefix. Specifically, for the automatic
shellcode generation task, we add the prefix “ShellCodeGen:” for its input
data. For the automatic shellcode summarization task, we add the prefix
“ShellCodeSum:” for its input data. During the training and decoding, we share
all parameters of the model to improve its performance and generalization
ability. Finally, we manually analyzed the examples with the error output for
the automatic shellcode generation task and designed a rule-based repair
component to further improve the quality of the generated shellcode.

Figure 2: Framework of our proposed approach DualSC

###  III-B Shallow Transformer-based Model

Through the Embedding layer and Positional Encoding, Transformer can covert
the input sequence $X_{input}$ = $(prefix,x_{1},x_{2},\cdots,x_{n})$ to the
vector $X_{emb}$ and feed this vector to the Encoder Layer. In the Encoder
part, Transformer inputs the vector $X_{emb}$ into the multi-head attention
layer. The attention function is computed simultaneously based on three
matrices: $Q$ (queries), $K$ (keys), and $V$ (values).

$Q,K,V=Linear(X_{emb})$ (1)

$X_{atten}=Attention(Q,K,V)$ (2)

In the classical Transformer, self-attention is calculated in relation to
queries ($Q$), keys ($K$), and values ($V$). The dot product between queries
and keys is first calculated, then each is given a division by $\sqrt{d_{k}}$
and the $\operatorname{softmax}$ function is applied to obtain the weights of
the corresponding values.

$\text{ Attention
}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V.$ (3)

Vaswani et al. [51] investigated the two computational approaches (i.e.,
additive attention and dot-product), and analyzed the differences between
them. With the advances of the Transformer, researchers conducted various
architectural and functional modifications to improve its performance on low-
resource tasks, including reducing the depth of the model, adding a
regularization penalty or adjusting the order of the LayerNorm [50, 53, 54,
55, 56]. The method QKNorm proposed by Henry et al. [24] can achieve promising
results on low-resource machine translation tasks. Specifically, the method
QKNorm first performed L2 normalization of $Q$ and $K$ before performing the
calculation, at which point the result obtained from the dot product is
expressed as a cosine similarity calculation of $Q$ and $K$. Thus the
calculation results can be controlled in the interval [-1, 1], and does not
need to be divided by $\sqrt{d_{k}}$. Instead, the model can be trained by
multiplying with a learnable parameter $g$ for training.

We follow the idea of the QKNorm [24] and take into account that the
insensitivity of the cosine similarity to values may lead to errors in the
final results [57], we propose Adjust_QKNorm. Notice that the matrices $Q$ and
$K$ are three-dimensional vectors, the first dimension is the size of the
batch, the second dimension is the length of the input sequence, and the third
dimension is the representation vector of the embedding corresponding to the
word. We subtract the values of the last dimension of both the matrices $Q$
and $K$ from their corresponding mean values (i.e., $\bar{Q}$ and $\bar{K}$),
so that the sum of all elements on that dimension is equal to zero, followed
by L2 normalization and dot product. Therefore, our proposed Adjust_QKNorm has
zero-mean adaptation, which ensures that self-attention can consider both the
numerical difference information of the matrix and the direction of the matrix
when calculating matrix similarity. The probability interval obtained by
softmax after this calculation is reduced but does not change for the overall
fluctuations, which may correct for some unreasonable scores. The formula is
shown as follows.

$\textbf{Q}=\frac{Q-\bar{Q}}{\|Q-\bar{Q}\|},\textbf{K}=\frac{K-\bar{K}}{\|K-\bar{K}\|}$
(4)

$\text{ Attention
}(Q,K,V)=\operatorname{softmax}\left(g*\textbf{Q}\textbf{K}^{T}\right)V$ (5)

Later, Transformer uses residual connection and layer normalization to make
the operation dimension of matrixes $X_{emb}$ and $X_{atten}$ consistent and
normalize the hidden layer in the network to a standard normal distribution,
which can accelerate the speed of the model training and convergence. In the
fourth step, Transformer passes the feed-forward layer and two linear mapping
layers. Then Transformer uses the activation function to generate the vector
$X_{hidden}$. Later Transformer performs a residual connection and layer
normalization to obtain the hidden semantic vector.

In the decoder part, Transformer uses the autoregressive mechanism, which
predicts the next possible word in the generation based on the above content.
Therefore, its next output needs to be inferred based on the output of the
encoder part. Each decoder layer performs an attention operation on the final
hidden state, which is the output of the encoder. The rest of the operation is
identical to the encoder part. Finally, the output of the decoder $ht$ is sent
to a fully connected neural network. This network can pass a
$\operatorname{softmax}$ layer to predict the probability of the next token,
which can be defined as follows.

$P\left(y_{t+1}\mid
y_{1},y_{2},\cdots,y_{t}\right)=\operatorname{softmax}\left(h_{t}W+b\right)$
(6)

We train our model parameters $\theta$ by minimizing the negative log
likelihood of the target text $y$ for a given input text $x$. For the
automatic shellcode generation task, $x$ means the natural language
descriptions and $y$ means the code snippet. For the automatic shellcode
summarization task, $x$ means the code snippet and $y$ means the natural
language descriptions.

$\mathcal{L}_{\theta}=-\sum_{i=1}^{|y|}\log P_{\theta}\left(y_{i}\mid
y_{<i},x\right)$ (7)

###  III-C Rule-based Repair Component

For the automatic generation task of shellcode, we design a rule-based repair
component to improve the performance of this task. An assembly statement in
shellcode consists of the label, instruction, and operands. The format of the
code segment is “label: instruction operand1, operand2 (options)”. After
manual analysis of the cases with error output, We identified two types of
errors: (1) global errors and (2) local errors. The uncontrollable nature of
neural network construction may cause the low correlation problem between the
generated code and ground-truth code, which can be defined as a global error.
While the local error mainly occurs at the position of operand2. After
analysis, we find that the local error is mainly caused by the OOV problem.
Specially, the value of operand2 appears in the given comment but not in the
vocabulary of the training set. Therefore, we mainly focus on fixing the local
error type in this component. Hu et al. [58] added the pointer network [59] to
the Transformer to solve this problem. Yang et al. [39] alleviated the OOV
problem with the help of sub-word tokenize method (i.e., Byte BPE [60]).
However, due to the nature of shellcode statements, it is more reliable to fix
this type of error based on rules. Specifically, the values that appear in the
operand position are divided into three categories (i.e., registers,
addresses, and values). For example, if the input comment is “ShellCodeGen:
subtract 0x6374612e from the contents in ecx and save the result in ecx”, the
original output of the model is “sub ecx, 0x1525152a”, and we found there
exits an inconsistency in the address. Therefore we extracted the correct
address from the comment and used this address to update the generated
shellcode. Then the fixed shellcode is “sub ecx, 0x6374612e”.

##  IV Experimental Setup

In this section, we introduce the experimental subject, performance measures,
baselines, and implementation details.

###  IV-A Experimental Subject

In our empirical study, we choose the shellcode parallel corpus ShellCode_IA32
shared by Liguori et al. [12] as our experimental subject. This corpus was
gathered from shell-storm222http://shell-storm.org/ and Exploit
Database333https://www.exploit-database.net/ in the period between 2000 and
2020. To ensure a fair comparison with the baselines, we followed the data
split method used by Liguori et al. [12]. Specifically, we divide the corpus
into the training set (80%), the validation set (10%), and the test set (10%).
Then we can get the training set with 2,560 items, the validation set with 320
items, and the test set with 320 items respectively. Since our approach is to
use only one model to learn the primal task and the dual task, we directly
transform the corpus by exchanging the input and the output and distinguish
the two tasks by adding the prefix. Thus we finally can get the training set
with 5,120 items, the validation set with 640 items, and the test set with 640
items. Table I shows the statistical information of our used corpus.

TABLE I: Statistics of corpus used in our empirical study Statistics for Code Length |
---|---
Avg | Mode | Median | $<5$ | $<10$ | $<15$
3.40 | 3 | 3 | $83.32\%$ | $97.30\%$ | $100\%$
Statistics for Comment Length |
Avg | Mode | Median | $<10$ | $<30$ | $<50$
9.06 | 4 | 7 | $69.41\%$ | $98.13\%$ | $100\%$

###  IV-B Performance Measures

To quantitatively compare the performance between our proposed approach and
the baselines, we choose the following four performance measures (i.e., BLEU,
ROGUE, METEOR, and ACC), which have been widely used in previous neural
machine translation and code generation studies [61]. Notice the higher the
value of the performance measure, the better the performance of the
corresponding approach.

BLEU. BLEU (Bilingual Evaluation Understudy) [62] is a variant of the
precision measure. This performance measure can calculate the similarity by
computing the $n$-gram precision of a candidate sentence to the reference
sentence, with a penalty for the sentences with short length.

ROUGE. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [63] is a set
of measures for evaluating a candidate sentence and the reference sentence. In
this study, we use ROUGE-L, where $L$ means LCS (Longest Common Subsequence).

METEOR. METEOR (Metric for Evaluation of sentences with Explicit Ordering)
[64] uses knowledge sources (such as WordNet) to expand the synset while
taking into account the shape of words.

ACC. ACC (Exact Match Accuracy) [26] is the fraction of exactly matching
samples between the predicted output and the reference. Different from
similarity-based measures, ACC considers whether the generated code is
identical to the ground truth code. Therefore this measure can accurately
evaluate the effectiveness of the generated shellcode for exploitability.

To ensure the implementation correctness of these performance measures, we
utilize the nlg-eval library444https://github.com/Maluuba/nlg-eval, which
implements these performance measures for natural language generation tasks.

###  IV-C Baselines

To show the competitiveness of our proposed approach DualSC, we select six
state-of-the-art baselines. Specifically, we classify these baselines into two
groups. One group includes three approaches based on information retrieval
(such as BM25 [65], Jaccard [66], and Levenshtein [67]). By observing our used
corpus, we found that the baselines based on information retrieval are
effective due to the consistency of comment styles and the presence of code
cloning on this corpus. Another group includes three baselines based on deep
learning, including NMT [12], Dual Model [45], and Transformer [51]. Here, NMT
is a recent state-of-the-art approach for the shellcode generation task [12].
Dual Model and Transformer are selected as our baselines since they are
similar to our proposed approach. Moreover, these two baselines have shown
competitive performance on both code generation and code summarization tasks
[45, 51].

###  IV-D Implementation Details

In our study, we use Pytorch 1.6.0 to implement our proposed approach. The
hyper-parameters of our proposed approach can be classified into three
categories (i.e., the hyper-parameters for the model structure, the hyper-
parameters in the model training phase, and the hyper-parameters in the model
test phase). These hyper-parameters and their value are shown in Table II.
Notice n_layers means the number of layers (i.e., the depth of the model), and
d_model means the model size (i.e., the width of the model).

We run all the experiments on a computer with an Intel(R) Core(TM) i7-9750H
4210 CPU and a GeForce GTX1660ti GPU with 6 GB memory. The running OS platform
is Windows 10.

TABLE II: Hyper-parameters and their values in our empirical study Category | Hyper-parameter | Parameter Value
---|---|---
Model Structure | n_layers | 2
n_heads | 8
d_model | 256
hidden_size | 512
Model Training Phase | dropout | 0.25
optimizer | Adam
learning rate | 0.001
batch size | 32
Model Test Phase | beam size | 3

##  V Result Analysis

In this section, we perform result analysis for our designed four research
questions.

###  V-A Result Analysis for RQ1

RQ1: Can our proposed approach DualSC outperform state-of-the-art baselines
both in the shellcode generation task and the shellcode summarization task?

In this RQ, we want to investigate how effective DualSC is and how much
performance improvement DualSC can achieve over the state-of-the-art
baselines. Since in the automatic shellcode generation task, we use the rule-
based repair component. To guarantee a fair comparison between DualSC and
baselines, we also apply this component to our considered baselines.

We show the comparison results between our proposed approach DualSC and
baselines in Table III. For the shellcode summarization task, in terms of
BLEU, DualSC can improve the performance by 3.913% to 12.380%. In terms of
ROUGE-L, DualSC can improve the performance by 1.998% to 13.438%. In terms of
METEOR, DualSC can improve the performance by 2.188% to 6.102%. For the
shellcode generation task, in terms of BLEU, DualSC can improve the
performance by 8.013% to 28.240%. In terms of ROUGE-L, DualSC can improve the
performance by 3.132 % to 15.472%. In terms of ACC, DualSC can improve the
performance by 9.0625 % to 35.625%. Therefore, compared to the baselines,
DualSC can achieve significant performance improvement for both two tasks.

TABLE III: The comparison results between our proposed approach DualSC and
baselines for two different tasks Automatic Shellcode Summarization
---
Approach | BLEU-4 (%) | ROUGE-L (%) | METEOR (%)
BM25 | 45.097 | 53.325 | 32.604
Jaccard | 49.950 | 57.485 | 34.352
Levenshtein | 51.954 | 61.391 | 36.410
NMT | 47.662 | 62.408 | 34.756
Transformer | 51.474 | 64.765 | 35.891
Dual Model | 53.564 | 63.668 | 36.518
DualSC | 57.477 | 66.763 | 38.706
Automatic Shellcode Generation
Approach | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
BM25 | 56.331 | 64.104 | 33.438
Jaccard | 60.429 | 66.872 | 34.688
Levenshtein | 49.374 | 64.260 | 27.813
NMT | 64.897 | 72.910 | 45.313
Transformer | 69.601 | 76.444 | 54.375
Dual Model | 68.718 | 75.423 | 51.563
DualSC | 77.614 | 79.576 | 63.438

Summary for RQ1: Our proposed approach DualSC can outperform baselines in
terms of four performance metrics in both the shellcode generation task and
the shellcode summarization task.

###  V-B Result Analysis for RQ2

RQ2: What is the effect of the rule-based repair component of DualSC in the
shellcode generation task?

In this RQ, we want to investigate the effect of the rule-based repair
component in the shellcode generation task. We compared the performance of
DualSC and baselines before and after the usage of the rule-based repair
component. As can be found in Table IV, the performance of all baselines and
DualSC can be improved after using the rule-based repair component. For
example, in terms of ACC, using this component can improve the performance by
8.44% to 23.44%.

TABLE IV: The results of DualSC with/without the repair component Automatic
Shellcode Generation Without The Repair Component
---
Approach | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
BM25 | 50.844 | 59.998 | 23.750
Jaccard | 51.343 | 59.930 | 17.188
Levenshtein | 37.396 | 54.101 | 4.375
NMT | 61.689 | 69.130 | 36.875
Transformer | 65.091 | 72.910 | 45.000
Dual Model | 62.992 | 71.119 | 40.000
DualSC | 73.713 | 75.825 | 54.063
Automatic Shellcode Generation With The Repair Component
Approach | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
BM25 | 56.331 | 64.104 | 33.438
Jaccard | 60.429 | 66.872 | 34.688
Levenshtein | 49.374 | 64.260 | 27.813
NMT | 64.897 | 72.910 | 45.313
Transformer | 69.601 | 76.444 | 54.375
Dual Model | 68.718 | 75.423 | 51.563
DualSC | 77.614 | 79.576 | 63.438

To show the competitiveness of our proposed repair component, we also consider
other four methods for DualSC to solve the OOV problem. The first three
methods (i.e., Byte BPE [60], Char BPE [68], and WordPiece [69]) are based on
sub-word tokenizatation. The last method (i.e., pointer network [59]) is based
on the copy mechanism. The comparison results can be found in Table V. In this
table, we can find the effectiveness of our designed simple repair component.
Since sub-word tokenization relies on large scale corpus, the three methods
(i.e., Byte BPE [60], Char BPE [68], and WordPiece [69]) do not perform well
on our low-resource tasks. On the other hand, the pointer network [59] is
based on probability for word reuse and this method still does not perform
well on our tasks.

TABLE V: The comparison results between our proposed repair component and
other four methods for Automatic Shellcode Generation Automatic Shellcode
Generation With Different Repair Methods
---
Method | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
Byte BPE | 65.682 | 72.114 | 49.063
Char BPE | 62.529 | 70.845 | 45.688
WordPiece | 64.725 | 72.014 | 48.552
Pointer Network | 75.429 | 76.872 | 58.688
DualSC | 77.614 | 79.576 | 63.438

Summary for RQ2: The rule-based repair component is effective for the
shellcode generation task and can significantly improve the performance of our
proposed approach and baselines.

###  V-C Result Analysis for RQ3

RQ3: What is the effect of the prefix and dual task learning of DualSC in
shellcode generation and summarization tasks?

In this RQ, we want to investigate the effect of the prefix and dual task
learning in both the shellcode generation task and the shellcode summarization
task. In this RQ, we consider two comparative approaches. The first approach
does not consider the prefix and we call this control approach “Without
Prefix”. The second approach does not consider the dual task learning and we
call this control approach “Single Task”. The comparison results between our
proposed approach DualSC and these control approaches can be found in Table
VI.

TABLE VI: The comparison results between our proposed approach and the control
approaches for two different tasks Automatic ShellCode Summarization
---
Approach | BLEU-4 (%) | ROUGE-L (%) | METEOR (%)
Without Prefix | 50.277 | 64.371 | 38.088
Single Task | 52.213 | 64.957 | 36.992
DualSC | 57.477 | 66.763 | 38.706
Automatic ShellCode Generation
Approach | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
Without Prefix | 73.200 | 77.041 | 58.750
Single Task | 66.192 | 76.152 | 56.250
DualSC | 77.614 | 79.576 | 63.438

For the effect of the prefix in the shellcode summarization task, DualSC can
improve the performance by 7.2%, 2.392%, and 0.618% respectively in terms of
BLEU, ROUGE-L, and METEOR. For the effect of the prefix in the shellcode
generation task, DualSC can improve the performance by 4.414%, 2.535%, and
4.6875% respectively in terms of BLEU, ROUGE-L, and METEOR.

For the effect of the dual task learning in the shellcode summarization task,
DualSC can improve the performance by 5.264%, 1.806%, and 1.714% respectively
in terms of BLEU, ROUGE-L, and METEOR. For the effect of the dual task
learning in the shellcode generation task, DualSC can improve the performance
by 11.422%, 3.424%, and 7.1875% respectively in terms of BLEU, ROUGE-L, and
METEOR.

Summary for RQ3: Both dual task learning and prefixing can have a positive
impact on automatic shellcode generation and summarization tasks for DualSC.

###  V-D Result Analysis for RQ4

RQ4: What is the effect of our proposed normalization method Adjust_QKNorm
both in shellcode generation and summarization tasks?

In this RQ, we want to investigate the impact of four different normalization
methods on automatic shellcode generation and summarization tasks. In
particular, we consider the traditional normalization method in Transformer
(i.e., PostNorm [70]), two state-of-the-art normalization methods (i.e.,
PreNorm [53] and QKNorm [24]), and our proposed normalization method
Adjust_QKNorm. The details of these normalization methods are illustrated as
follows.

  * •

PostNorm: In the original Transformer, Layer Norm follows Residual is called
PostNorm [70]. The method PostNorm is very sensitive to parameters and needs
to be carefully tuned to achieve promising results. For example, the essential
warm-up learning rate strategy is often used. However, this strategy is very
time-consuming.

  * •

PreNorm: To alleviate the problems in PostNorm, PreNorm [53] tries to put
Layer Norm in the process of Residual, so that when the model is updated in
reverse, the bottom parameters can directly obtain the information of the top
gradient, which can effectively alleviate the issues of gradient vanishing and
gradient explosion.

  * •

QKNorm: To adopt Transformer for low-resource tasks, QKNorm [24] first
calculates $Q$ and $K$ in the formula by L2 normalization the self-attention,
and then adjust the dot product result of $Q$ and $K$ by a learnable parameter
$g$ instead of dividing by $\sqrt{d_{k}}$.

  * •

Adjust_QKNorm: We follow the idea of the QKNorm [24] and propose the
Adjust_QKNorm. The QKNorm method can be found as replacing the calculation of
the dot product of two vectors $Q$ and $K$ with the calculation of the cosine
similarity of two vectors $Q$ and $K$. However, the insensitivity of cosine
similarity to values can lead to errors in the results, therefore we improve
the QKNorm by going through the zero-averaging operation and its formula can
be found in Equation (5).

The comparison results between our proposed normalization method Adjust_QKNorm
and other normalization methods can be found in Table VII. For the shellcode
summarization task, in terms of BLEU, Adjust_QKNorm can improve the
performance by 3.027% to 4.896%. In terms of ROUGE-L, Adjust_QKNorm can
improve the performance by 1.357% to 2.07%. In terms of METEOR, Adjust_QKNorm
can improve the performance by 0.164% to 2.043%. For the shellcode generation
task with the repair component, in terms of BLEU, Adjust_QKNorm can improve
the performance by 0.123% to 12.776%. In terms of ROUGE-L, Adjust_QKNorm can
improve the performance by 0.521% to 6.145%. In terms of ACC, Adjust_QKNorm
can improve the performance by 0.375% to 12.1875%. Moreover, for the shellcode
generation task without the repair component, Adjust_QKNorm can also achieve
the best performance.

TABLE VII: The comparison results between Adjust_QKNorm and other
normalization methods for two different tasks  Automatic Shellcode
Summarization
---
Method | BLEU-4 (%) | ROUGE-L (%) | METEOR (%)
PostNorm | 53.144 | 64.903 | 37.591
PreNorm | 52.581 | 64.693 | 36.663
QKNorm | 54.450 | 65.406 | 38.542
Adjust_QKNorm | 57.477 | 66.763 | 38.706
Automatic Shellcode Generation Without The Repair Component
Method | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
PostNorm | 65.091 | 72.910 | 45.000
PreNorm | 60.551 | 69.642 | 41.250
QKNorm | 70.084 | 74.274 | 50.938
Adjust_QKNorm | 73.713 | 75.825 | 54.063
Automatic Shellcode Generation With The Repair Component
Method | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
PostNorm | 71.765 | 78.108 | 60.313
PreNorm | 64.838 | 73.431 | 51.250
QKNorm | 77.491 | 79.055 | 63.063
Adjust_QKNorm | 77.614 | 79.576 | 63.438

For a more intuitive illustration of how the method Adjust_QKNorm works, we
selected an example for visualizing the weight of self-attention. As can be
found in Fig. 3, the input to the model is “ShellCodeGen: move readbuffer into
ecx”, the left subplot represents the self-attentive visualization graph
learned by the method PostNorm, and the right subplot represents the self-
attentive visualization graph learned by our proposed method Adjust_QKNorm. In
the left subplot, the method PostNorm misaligned the attention for the input
“move readbuffer into ecx”. While using the method Adjust_QKNorm, the obtained
attention can be corrected (i.e., “move” corresponds to “readbuffer”,
“readbuffer” corresponds to “into”, and “into” corresponds to “ecx”).

Figure 3:  Visualisation of the weight of self-attention by different
normalization methods, the left subfigure is obtained by PostNorm, and the
right subfigure is obtained by Adjust_QKNorm

Summary for RQ4: Our proposed normalization method Adjust_QKNorm can achieve
the best performance on automatic shellcode generation and summarization tasks
for DualSC.

##  VI Discussions

In this section, we first perform sensitivity analysis on the hyperparameters
of DualSC. Then we conduct a human study on the shellcode summarization task.

Figure 4: One page in the questionnaire of our human study

###  VI-A Sensitivity Analysis on the Hyperparameters

In this subsection, we conduct experiments by performing sensitivity analysis
on the parameters of DualSC. We mainly focus on two parameters (i.e., the
model size $d\\_model$ and the number of layers $N$). The sensitivity analysis
results on different parameters can be found in Table VIII. The optimal value
of these parameters is set as follows: $d\\_model$ is 256 and $N$ is 2.

TABLE VIII: Sensitivity analysis on the hyperparameters (i.e., the model size
and the number of layers) Automatic Shellcode Summarization
---
$N$ | $d\\_model$ | BLEU-4 (%) | ROUGE-L (%) | METEOR (%)
1 | 128 | 50.410 | 62.515 | 34.968
256 | 52.827 | 65.070 | 37.768
384 | 52.601 | 63.571 | 37.541
512 | 55.785 | 66.025 | 38.940
2 | 128 | 55.821 | 64.944 | 37.694
256 | 57.477 | 66.763 | 38.706
384 | 54.837 | 64.596 | 37.993
512 | 53.056 | 64.195 | 38.706
3 | 128 | 54.110 | 62.628 | 36.897
256 | 46.813 | 60.976 | 33.816
384 | 0 | 21.771 | 6.968
512 | 0 | 21.416 | 6.763
Automatic Shellcode Generation
$N$ | $d\\_model$ | BLEU-4 (%) | ROUGE-L (%) | ACC (%)
1 | 128 | 64.362 | 71.748 | 48.438
256 | 72.028 | 75.913 | 56.250
384 | 73.235 | 78.214 | 59.688
512 | 73.670 | 76.022 | 59.063
2 | 128 | 71.125 | 73.737 | 53.125
256 | 77.614 | 79.576 | 63.438
384 | 72.787 | 74.753 | 57.188
512 | 70.373 | 75.757 | 55.313
3 | 128 | 74.520 | 73.360 | 53.438
256 | 60.660 | 67.461 | 40.938
384 | 0.303 | 15.539 | 1.250
512 | 0.299 | 18.160 | 1.563

The final results can be found in Table VIII. In this table, We find that it
is harder to train the generation model when the value of $N$ (i.e., the depth
of the model) and the value of $d\\_model$ (i.e., the width of the model) is
large. Even if DualSC uses dual task learning and Adjust_Norm to alleviate
this problem, the gradient vanishing issue tends to occur when training the
generation model. Therefore we select small values for these parameters, which
are more suitable for low-resource tasks in our study.

###  VI-B Human Study on the Shellcode Summarization Task

Different from the code generation task, the code summarization task requires
human study to further verify the competitiveness of DualSC. Stapleton et al.
[71] showed that the performance measures (such as BLEU, ROUGE, and METEOR),
which were commonly used for the code summarization task, cannot necessarily
capture how machine-generated code summaries actually affect program
comprehension of developers. Therefore, we conduct the human study by
following the methodology used in previous studies [72, 73]. Since the Dual
Model can achieve the best performance among all the baselines, we focus on
analyzing the differences between comments generated by DualSC and the
baseline Dual Model. Due to the high cost of manually analyzing all these
samples in the testing set, we use a commonly-used sampling method [74] to
select the minimum random samples. The number of the selected samples can be
determined by the following formula:

$MIN=\frac{n_{0}}{1+\frac{n_{0}-1}{size}}$ (8)

where $n_{0}$ depends on the selected confidence level and the desired error
margin $n_{0}\left(=\frac{Z^{2}\times 0.25}{e^{2}}\right)$. $Z$ is a
confidence level $z$ score and $e$ is the error margin. $size$ is the number
of samples in the testing set. In our human study, we select $MIN$ examples
with the error margin $e$ = 0.05 at 95% confidence level (i.e., $MIN=175$).

We recruited six graduate students with rich development experience in
shellcode to conduct our human study. We randomly selected 174 samples from
the testing set, including ground-truth reference comment, the comments
generated by DualSC and Dual Model respectively. We generate a questionnaire
for each master student and a page of a sample can be found in Fig. 4. To
guarantee a fair comparison, the order of the comments generated by these two
approaches is random, which can make it difficult for students to know which
approach these comments are generated by. Here, We mainly consider naturalness
and relevance [72] of the code summarization. Specifically, naturalness refers
to the grammatical correctness and fluency of the generated summarization.
That is whether the generated code summarization is easy for developers to
read and understand. Relevance refers to the relevance of the generated
summarization to the target shellcode. That is whether developers can
understand the intent of the shellcode through the code summarization. The
score ranges from 0 to 5 and 5 is the best value.

TABLE IX: Results of our human study in terms of naturalness and relevance Type | DualSC | Dual Model
---|---|---
Naturalness | 4.12 | 4.02
Relevance | 4.08 | 3.95

After our human study, we compute the average score of the six students’
feedback and the results are shown in Table IX. In this table, We can find
that DualSC can outperform the baseline Dual Model by 0.10 and 0.13
respectively when considering naturalness and relevance. Therefore, our human
study can further verify the competitiveness of DualSC.

##  VII Threats to validity

In this section, we mainly analyze the potential threats to the validity of
our empirical study.

Internal Threats. The first internal threat is the potential defects in the
implementation of DualSC. To alleviate this threat, we check our
implementation carefully and use mature libraries (such as PyTorch and
TorchText555https://pytorch.org). The second internal threat is the
implementation of our considered baselines. To alleviate this threat, we use
textdistance library666https://github.com/life4/textdistance to implement
Jaccard and Levenshtein, we use
rank_bm25777https://github.com/dorianbrown/rank_bm25 library to implement
BM25, we use OpenNMT-py [75] library888https://github.com/OpenNMT/OpenNMT-py
to implement NMT and Transformer, and we implement Dual Model based on the
code shared by Wei et al999https://github.com/code-gen/cscg.

External Threats. The main external threat is the choice of the corpus. To
alleviate this threat, we select the popular corpus provided by Liguori et al.
[12] and this corpus was gathered from shell-storm2 and Exploit Database.

Construct Threats. The construct threat in this study is the performance
measures used to evaluate the performance of DualSC. To alleviate these
threats, we first choose the similarity-based performance measures (such as
BLEU and Rouge-L) for the shellcode generation task, which have been used by
the previous code generation studies. However, these measures are not designed
by running the exploits. Therefore the generated shellcode might not
necessarily offer a remote shell. Therefore we further consider the ACC (Exact
Match Accuracy) to verify the correctness of the generated shellcode. For the
shellcode summarization task, we choose three performance measures (i.e.,
BLEU, Rouge-L, and METEOR), which have been used by the previous code
summarization studies [37][39][47].

##  VIII Conclusion

In this study, we propose a novel approach DualSC via Transformer and dual
learning to solve the automatic shellcode generation and summarization tasks.
Specifically, we formalize automatic shellcode generation and summarization as
dual tasks, using a shallow Transformer for learning, while proposing
Adjust_QKNorm to adapt our proposed approach on low-resource tasks. Finally,
to mitigate the OOV problem, we propose a rule-based repair component to
improve the accuracy of DualSC on automatic shellcode generation. Empirical
results verified the effectiveness of our proposed approach. Then, we analyze
the effectiveness of the component settings in DualSC. Finally, we also
conduct a human study to verify the effectiveness of DualSC.

## Acknowledgment

The authors would like to thank the anonymous reviewers for their insightful
comments and suggestions. This work is supported in part by the National
Natural Science Foundation of China (Grant no. 61872263), The Open Project of
State Key Laboratory of Information Security (Institute of Information
Engineering, Chinese Academy of Sciences) (Grant No. 2020-MS-07).

## References

  * [1] J. Mason, S. Small, F. Monrose, and G. MacManus, “English shellcode,” in _Proceedings of the 16th ACM conference on Computer and communications security_ , 2009, pp. 524–533.
  * [2] I. Arce, “The shellcode generation,” _IEEE security & privacy_, vol. 2, no. 5, pp. 72–76, 2004.
  * [3] Y. Wang, X.-H. Li, L. Guan, and B.-J. Cui, “Attack and defending technology of shellcode,” _Computer Engineering_ , vol. 36, no. 18, pp. 165–168, 2010\.
  * [4] K. Park, Y. Song, and Y.-G. Cheong, “Classification of attack types for intrusion detection systems using a machine learning algorithm,” in _2018 IEEE fourth international conference on big data computing service and applications (BigDataService)_. IEEE, 2018, pp. 282–286.
  * [5] M. Polychronakis, K. G. Anagnostakis, and E. P. Markatos, “Comprehensive shellcode detection using runtime heuristics,” in _Proceedings of the 26th Annual Computer Security Applications Conference_ , 2010, pp. 287–296.
  * [6] Y. Ding, T. Wei, H. Xue, Y. Zhang, C. Zhang, and X. Han, “Accurate and efficient exploit capture and classification,” _Science China Information Sciences_ , vol. 60, no. 5, pp. 1–17, 2017.
  * [7] T. Bao, R. Wang, Y. Shoshitaishvili, and D. Brumley, “Your exploit is mine: Automatic shellcode transplant for remote exploits,” in _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2017, pp. 824–839.
  * [8] S. Zhao, X. Li, Y. Fang, Y. Yu, W. Huang, K. Chen, P. Su, and Y. Zhang, “A survey on automated exploit generation,” _Journal of Computer Research and Development_ , vol. 56, no. 10, p. 2097, 2019.
  * [9] K. Borders, A. Prakash, and M. Zielinski, “Spector: Automatically analyzing shell code,” in _Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007)_. IEEE, 2007, pp. 501–514.
  * [10] P. Chen, X. Xing, B. Mao, L. Xie, X. Shen, and X. Yin, “Automatic construction of jump-oriented programming shellcode (on the x86),” in _Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security_ , 2011, pp. 20–29.
  * [11] D. Patel, A. Basu, and A. Mathuria, “Automatic generation of compact printable shellcodes for x86,” in _14th $\\{$USENIX$\\}$ Workshop on Offensive Technologies ($\\{$WOOT$\\}$ 20)_, 2020.
  * [12] P. Liguori, E. Al-Hossami, D. Cotroneo, R. Natella, B. Cukic, and S. Shaikh, “Shellcode_ia32: A dataset for automatic shellcode generation,” _arXiv preprint arXiv:2104.13100_ , 2021.
  * [13] S. C. B. de Souza, N. Anquetil, and K. M. de Oliveira, “A study of the documentation essential to software maintenance,” in _Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information_, 2005, pp. 68–75.
  * [14] X. Xia, L. Bao, D. Lo, Z. Xing, A. E. Hassan, and S. Li, “Measuring program comprehension: A large-scale field study with professionals,” _IEEE Transactions on Software Engineering_ , vol. 44, no. 10, pp. 951–976, 2017.
  * [15] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang _et al._ , “Codebert: A pre-trained model for programming and natural languages,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , 2020, pp. 1536–1547.
  * [16] L. Phan, H. Tran, D. Le, H. Nguyen, J. Anibal, A. Peltekian, and Y. Ye, “Cotext: Multi-task learning with code-text transformer,” _arXiv preprint arXiv:2105.08645_ , 2021.
  * [17] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021, pp. 8696–8708.
  * [18] W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 2655–2668.
  * [19] N. Chirkova and S. Troshin, “Empirical study of transformers for source code,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, pp. 703–715.
  * [20] X. V. Lin, C. Wang, L. Zettlemoyer, and M. D. Ernst, “Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system,” in _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018.
  * [21] M. Zavershynskyi, A. Skidanov, and I. Polosukhin, “Naps: Natural program synthesis dataset,” _arXiv preprint arXiv:1807.03168_ , 2018.
  * [22] Z. Yang, J. Keung, X. Yu, X. Gu, Z. Wei, X. Ma, and M. Zhang, “A multi-modal transformer-based code summarization approach for smart contracts,” in _2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)_ , 2021, pp. 1–12.
  * [23] G. Yang, K. Liu, X. Chen, Y. Zhou, C. Yu, and H. Lin, “Ccgir: Information retrieval-based code comment generation method for smart contracts,” _Knowledge-Based Systems_ , p. 107858, 2021.
  * [24] A. Henry, P. R. Dachapally, S. Pawar, and Y. Chen, “Query-key normalization for transformers,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020.
  * [25] M. Rabinovich, M. Stern, and D. Klein, “Abstract syntax networks for code generation and semantic parsing,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2017, pp. 1139–1149.
  * [26] P. Yin and G. Neubig, “A syntactic neural model for general-purpose code generation,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2017, pp. 440–450.
  * [27] ——, “Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing (Demo Track)_ , 2018.
  * [28] S. A. Hayati, R. Olivier, P. Avvaru, P. Yin, A. Tomasic, and G. Neubig, “Retrieval-based neural code generation,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 925–930.
  * [29] R. Agashe, S. Iyer, and L. Zettlemoyer, “Juice: A large scale distantly supervised dataset for open domain context-based code generation,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019, pp. 5436–5446.
  * [30] Z. Sun, Q. Zhu, L. Mou, Y. Xiong, G. Li, and L. Zhang, “A grammar-based structural cnn decoder for code generation,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 7055–7062.
  * [31] F. F. Xu, Z. Jiang, P. Yin, B. Vasilescu, and G. Neubig, “Incorporating external knowledge through pre-training for natural language to code generation,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 6045–6052.
  * [32] R. Zhong, M. Stern, and D. Klein, “Semantic scaffolds for pseudocode-to-code generation,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 2283–2295.
  * [33] Z. Sun, Q. Zhu, Y. Xiong, Y. Sun, L. Mou, and L. Zhang, “Treegen: A tree-based transformer architecture for code generation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 8984–8991.
  * [34] S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 2073–2083.
  * [35] M. Allamanis, H. Peng, and C. Sutton, “A convolutional attention network for extreme summarization of source code,” in _Proceedings of International conference on machine learning_ , 2016, pp. 2091–2100.
  * [36] Y. Liang and K. Q. Zhu, “Automatic generation of text descriptive comments for code blocks,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [37] X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,” in _Proceedings of 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_. IEEE, 2018, pp. 200–20 010.
  * [38] ——, “Deep code comment generation with hybrid lexical and syntactical information,” _Empirical Software Engineering_ , vol. 25, no. 3, pp. 2179–2217, 2020.
  * [39] G. Yang, X. Chen, J. Cao, S. Xu, Z. Cui, C. Yu, and K. Liu, “Comformer: Code comment generation via transformer and fusion method-based hybrid code representation,” in _2021 8th International Conference on Dependable Systems and Their Applications (DSA)_. IEEE, 2021, pp. 30–41.
  * [40] H. J. Kang, T. F. Bissyandé, and D. Lo, “Assessing the generalizability of code2vec token embeddings,” in _Proceedings of 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2019, pp. 1–12.
  * [41] U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” _Proceedings of the ACM on Programming Languages_ , vol. 3, no. POPL, pp. 1–29, 2019.
  * [42] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014, pp. 1532–1543.
  * [43] A. LeClair, S. Jiang, and C. McMillan, “A neural model for generating natural language summaries of program subroutines,” in _Proceedings of 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 2019, pp. 795–806.
  * [44] Q. Chen and M. Zhou, “A neural framework for retrieval and summarization of source code,” in _Proceedings of 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2018, pp. 826–831.
  * [45] B. Wei, G. Li, X. Xia, Z. Fu, and Z. Jin, “Code generation as a dual task of code summarization,” _Advances in Neural Information Processing Systems_ , vol. 32, pp. 6563–6573, 2019.
  * [46] W. Ye, R. Xie, J. Zhang, T. Hu, X. Wang, and S. Zhang, “Leveraging code generation to improve code retrieval and summarization via dual learning,” in _Proceedings of The Web Conference 2020_ , 2020, pp. 2309–2319.
  * [47] W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “A transformer-based approach for source code summarization,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 4998–5007.
  * [48] S. Liu, C. Gao, S. Chen, N. L. Yiu, and Y. Liu, “Atom: Commit message generation based on abstract syntax tree and hybrid ranking,” _IEEE Transactions on Software Engineering_ , 2020.
  * [49] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of Machine Learning Research_ , vol. 21, pp. 1–67, 2020.
  * [50] E. Van Biljon, A. Pretorius, and J. Kreutzer, “On optimal transformer depth for low-resource language translation,” _arXiv preprint arXiv:2004.04418_ , 2020.
  * [51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [52] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _arXiv preprint arXiv:2107.13586_ , 2021.
  * [53] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu, “On layer normalization in the transformer architecture,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 10 524–10 533.
  * [54] J. Xu, X. Sun, Z. Zhang, G. Zhao, and J. Lin, “Understanding and improving layer normalization,” in _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , 2019, pp. 4381–4391.
  * [55] T. Q. Nguyen and J. Salazar, “Transformers without tears: Improving the normalization of self-attention,” _arXiv preprint arXiv:1910.05895_ , 2019\.
  * [56] S. Shen, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer, “Powernorm: Rethinking batch normalization in transformers,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 8741–8751.
  * [57] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Item-based collaborative filtering recommendation algorithms,” in _Proceedings of the 10th international conference on World Wide Web_ , 2001, pp. 285–295.
  * [58] X. Hu, Z. Gao, X. Xia, D. Lo, and X. Yang, “Automating user notice generation for smart contract functions,” in _Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering_ , 2021.
  * [59] A. See, P. J. Liu, and C. D. Manning, “Get to the point: Summarization with pointer-generator networks,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2017, pp. 1073–1083.
  * [60] C. Wang, K. Cho, and J. Gu, “Neural machine translation with byte-level subwords,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 9154–9160.
  * [61] S. Yang, Y. Wang, and X. Chu, “A survey of deep learning techniques for neural machine translation,” _arXiv preprint arXiv:2002.07526_ , 2020.
  * [62] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [63] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [64] S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 2005, pp. 65–72.
  * [65] S. Robertson and S. Walker, “Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval,” in _Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval_ , 1994, pp. 232–241.
  * [66] P. Jaccard, “The distribution of the flora in the alpine zone. 1,” _New phytologist_ , vol. 11, no. 2, pp. 37–50, 1912.
  * [67] V. I. Levenshtein _et al._ , “Binary codes capable of correcting deletions, insertions, and reversals,” in _Soviet physics doklady_ , vol. 10, no. 8. Soviet Union, 1966, pp. 707–710.
  * [68] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” _arXiv preprint arXiv:1508.07909_ , 2015.
  * [69] M. Schuster and K. Nakajima, “Japanese and korean voice search,” in _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2012, pp. 5149–5152.
  * [70] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [71] S. Stapleton, Y. Gambhir, A. LeClair, Z. Eberhart, W. Weimer, K. Leach, and Y. Huang, “A human study of comprehension and code summarization,” in _Proceedings of the 28th International Conference on Program Comprehension_ , 2020, pp. 2–13.
  * [72] Z. Gao, X. Xia, J. Grundy, D. Lo, and Y.-F. Li, “Generating question titles for stack overflow from mined code snippets,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 29, no. 4, pp. 1–37, 2020\.
  * [73] Z. Li, Y. Wu, B. Peng, X. Chen, Z. Sun, Y. Liu, and D. Yu, “Secnn: A semantic cnn parser for code comment generation,” _Journal of Systems and Software_ , vol. 181, p. 111036, 2021.
  * [74] R. Singh and N. S. Mangat, _Elements of survey sampling_. Springer Science & Business Media, 2013, vol. 15.
  * [75] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. Rush, “OpenNMT: Open-source toolkit for neural machine translation,” in _Proceedings of ACL 2017, System Demonstrations_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 67–72. [Online]. Available: https://www.aclweb.org/anthology/P17-4012