# An Extractive-and-Abstractive Framework for Source Code Summarization

Weisong Sun<EMAIL_ADDRESS>State Key Laboratory for Novel
Software TechnologyNanjing UniversityChina ,  Chunrong Fang
<EMAIL_ADDRESS>State Key Laboratory for Novel Software
TechnologyNanjing UniversityChina ,  Yuchen Chen<EMAIL_ADDRESS>State
Key Laboratory for Novel Software TechnologyNanjing UniversityChina ,  Quanjun
Zhang<EMAIL_ADDRESS>State Key Laboratory for Novel Software
TechnologyNanjing UniversityChina ,  Guanhong Tao<EMAIL_ADDRESS>Purdue
UniversityWest LafayetteIndianaUSA ,  Tingxu Han<EMAIL_ADDRESS>School of
Information ManagementNanjing UniversityChina ,  Yifei Ge<EMAIL_ADDRESS>State Key Laboratory for Novel Software TechnologyNanjing UniversityChina ,
Yudu You<EMAIL_ADDRESS>State Key Laboratory for Novel Software
TechnologyNanjing UniversityChina and  Bin Luo<EMAIL_ADDRESS>State Key
Laboratory for Novel Software TechnologyNanjing UniversityChina

(2022)

###### Abstract.

(Source) Code summarization aims to automatically generate summaries/comments
for a given code snippet in the form of natural language. Such summaries play
a key role in helping developers understand and maintain source code. Existing
code summarization techniques can be categorized into extractive methods and
abstractive methods. The extractive methods extract a subset of important
statements and keywords from the code snippet using retrieval techniques, and
generate a summary that preserves factual details in important statements and
keywords. However, such a subset may miss identifier or entity naming, and
consequently, the naturalness of generated summary is usually poor. The
abstractive methods can generate human-written-like summaries leveraging
encoder-decoder models from the neural machine translation domain. The
generated summaries however often miss important factual details.

To generate human-written-like summaries with preserved factual details, we
propose a novel extractive-and-abstractive framework. The extractive module in
the framework performs a task of extractive code summarization, which takes in
the code snippet and predicts important statements containing key factual
details. The abstractive module in the framework performs a task of
abstractive code summarization, which takes in the entire code snippet and
important statements in parallel and generates a succinct and human-written-
like natural language summary. We evaluate the effectiveness of our technique,
called EACS, by conducting extensive experiments on three datasets involving
six programming languages. Experimental results show that EACS significantly
outperforms state-of-the-art techniques in terms of all three widely used
metrics, including $\mathtt{BLEU}$, $\mathtt{METEOR}$, and $\mathtt{ROUGH-L}$.

Code Summarization, Program Comprehension

††copyright: acmcopyright††journalyear: 2022††doi: xxx/xxx††conference: xxx;
xxx, 2022; xxx††booktitle: Conference 2022: xxx, xxx, 2022, xxx††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Software and its engineering
Software maintenance tools

##  1\. Introduction

Code comments play a significant role in facilitating code comprehension
(Woodfield et al., 1981; Tenny, 1988; Xia et al., 2018; Gros et al., 2020) and
software maintenance (Hartzman and Austin, 1993; de Souza et al., 2005; Zhai
et al., 2020; Chen et al., 2021). Writing high-quality code comments has been
recognized as a good programming practice (de Souza et al., 2005; Zhai et al.,
2020). The code comment is one of the most common summaries used during
software development. However, commenting source code is a labour-intensive
and time-consuming task (de Souza et al., 2005; Kajko-Mattsson, 2005). As a
result, good comments are often absent, unmatched, and outdated during the
evolution (Hu et al., 2018b). The lack of high-quality code comments is a
common problem in software industry (Hu et al., 2018b). (Source) code
summarization is an active research field (Haiduc et al., 2010b; Eddy et al.,
2013; Jayavardhan Reddy et al., 2018; Zhu and Pan, 2019; LeClair and McMillan,
2019; Abid et al., 2019; Stapleton et al., 2020; Gros et al., 2020; Haque et
al., 2021; Shi et al., 2021b; Chen et al., 2021; Roy et al., 2021; Zhou et
al., 2021), which aims at designing advanced techniques to support automatic
generation of code summaries (i.e., comments). Given a code snippet (a method
or function) by the developer, code summarization can generate summaries
related to the code snippet. Figure 1 shows an example. The code snippet in
Figure 1(a) is provided by the developer. The summary “removes the mapping for
the specified key from this cache if present.” in Figure 1(b) is a possible
summary that satisfies the developer’s requirement. The summary is usually a
succinct description in natural language summarizing the
intention/functionality of the desired code snippet (Chen et al., 2021).

Figure 1. Example of Code Snippet and Summary

Code summarization can be viewed as a special kind of text summarization task,
where the text is not a traditional natural language but a programming
language. Indeed, as early as ten years ago, automatic text summarization
techniques have been introduced to automatic code summarization (Haiduc et
al., 2010b), detailed in Section 2.1. Therefore, similar to text summarization
(El-Kassas et al., 2021; Gambhir and Gupta, 2017), code summarization can be
subdivided into extractive code summarization (extractive methods) and
abstractive code summarization (abstractive methods). Most of the early code
summarization techniques are extractive methods, which widely use an indexing-
retrieval framework to generate summaries (Haiduc et al., 2010b; Eddy et al.,
2013; Rodeghero et al., 2014; Fowkes et al., 2017). They first index terms in
code snippets and then retrieve top-$n$ key terms as summaries. The terms in
summaries are extracted from the current code snippet (Moreno et al., 2013),
context code snippets (McBurney and McMillan, 2016) or similar code snippets
(Eddy et al., 2013; Wong et al., 2015). Therefore, extractive methods can
produces a summary that preserves the conceptual integrity and factual
information of the input code snippet. However, as introduced in (Wong et al.,
2015; Hu et al., 2018a, b), extractive methods have two main limitations: 1)
they fail to extract accurate keywords when the identifiers and methods are
poorly named; 2) they rely on similar code snippets can be retrieved and how
similar the code snippets are. Therefore, there may be semantic
inconsistencies between the retrieved summary and the input code snippet (Li
et al., 2021). In addition, the naturalness of the summaries generated by
extractive methods usually is poor (Iyer et al., 2016; Li et al., 2021).

Recently, with the success of deep learning (DL) in abstractive text
summarization, DL-based code summarization techniques have been proposed one
after another (Iyer et al., 2016; Fernandes et al., 2018; Hu et al., 2018b;
Wan et al., 2018; LeClair et al., 2019; Moore et al., 2019; Hu et al., 2020;
Wang et al., 2020; Zhang et al., 2020; Wei et al., 2020; LeClair et al., 2020;
Ahmad et al., 2020; Lin et al., 2021; Shahbazi et al., 2021; Phan et al.,
2021). Compared with extractive methods, similar to abstractive text
summarization (Shi et al., 2021a), DL-based code summarization techniques have
stronger abstract expression capabilities and can generate human-written-like
summaries. Therefore, we call DL-based code summarization techniques as
abstractive methods. Abstractive methods widely adopt DL-based the encoder-
decoder neural network and then train it on a large code-comment corpus. The
well-trained encoder-decoder model can transform code snippets into embedding
representations and then decode the embedding representations into short
natural language summaries. As in many research areas and as chronicled by
Allamanis et al. (Allamanis et al., 2018), traditional methods have largely
given way to deep learning methods based on big data input. Although the
abstractive methods have the ability to generate novel words and phrases not
featured in the code snippet – as a human-written abstract usually does, the
generated summaries often miss important factual details in the code snippet,
detailed in Section 3.

In this paper, we propose an extractive-and-abstractive framework for code
summarization, which inherits the advantages of extractive methods and
abstractive methods and shields their respective disadvantages. Specifically,
we utilize pairs of code snippets and comments to train an extractor (an
extractive method) and an abstracter (an abstractive method). The well-trained
extractor can be used to predict important statements in code snippets. These
important statements and the entire code snippet are input to the abstracter
to generate a short natural language summary. The well-trained abstracter
first utilizes two separate encoders to transform important statements and the
entire code snippet into two vectors, called extractive and abstractive
embedding representations, respectively. Then, the two embedding
representations are fused to produce a fusion embedding representation, which
will be passed to a decoder to generate a natural language summary. Compared
with existing abstractive methods, our extractive-and-abstractive framework is
equipped with an extractor, substantially balancing attention on important
information and global contextual information, respectively, reducing the risk
of missing important factual details and improving the overall performance.

In summary, we make the following contributions.

  * •

To the best of our knowledge, we are the first to propose an extractive-and-
abstractive framework for code summarization, which inherits the advantages of
extractive methods and abstractive methods and shields their respective
disadvantages. It is a general framework and can be combined with multiple
advanced models (see experimental results in Section 5.2.2).

  * •

We implement a code summarization prototype called EACS based on the
extractive-and-abstractive framework. EACS is able to generate a succinct
natural language summary that not only is human-written-like but also
preserves important factual details.

  * •

We conduct extensive experiments on three widely used datasets to evaluate
EACS. Experimental results show that EACS improves the $\mathtt{BLEU}$ score
of code summarization by 5.40% to 6.55% compared to state-of-the-art
technique. In addition, EACS achieves $\mathtt{METEOR}$ of 30.39 and 23.70,
outperforming the state-of-the-art by 12.14% to 12.75%. In terms of
$\mathtt{ROUGE-L}$, EACS improves by 6% to 7.24%. The source code of EACS and
all the data used in this paper are released and can be downloaded from the
website (Anonymous, 2022).

##  2\. Background

###  2.1. Automatic Text Summarization

When design EACS, we drew on the advanced ideas and techniques of automatic
text summarization. Therefore, we first introduce the background of automatic
text summarization.

Automatic text summarization is the task of automatically condensing a piece
of text to a shorter version summary while maintaining the important points
(Gambhir and Gupta, 2017; El-Kassas et al., 2021). According to technical
characteristics, text summarization is subdivided into extractive text
summarization (extractive methods) and abstractive text summarization
(abstractive methods) (Maybury, 1999; Shi et al., 2021a). Extractive methods
assemble summaries by directly selecting words, phrases, and sentences from
source text. The generated summaries usually persist salient information of
source text (See et al., 2017; Wu and Hu, 2018; Zhou et al., 2018). In
contrast, abstractive methods can generate novel words and phrases not
featured from the source text – as a human-written abstract usually does (See
et al., 2017). Thus, they have a strong potential of producing high-quality
summaries that are verbally innovative (Shi et al., 2021a).

Automatic text summarization techniques were introduced to automatic code
summarization as early as 10 years ago. For example, in 2010, Sonia Haiduc
et.al (Haiduc et al., 2010a) proposed an extractive code summarization
technique for the automatic generation of extractive summaries for source code
entities. Extractive summaries are generated by selecting the most important
terms in code snippets. In addition, they present a study in (Haiduc et al.,
2010b) to investigate the suitability of various text summarization techniques
for generating code summaries. The study results indicate that a combination
of text summarization techniques is most appropriate for code summarization
and that developers generally agree with the produced summaries. Recently, DL-
based code summarization techniques have been proposed one after another (Hu
et al., 2018a; Wei et al., 2019; Gros et al., 2020; Wu et al., 2021). Similar
to abstractive text summarization, these techniques also widely adopt the
encoder-decoder models borrowed from neural machine translation (detailed in
Section 2.2) to generate natural language summaries. Therefore, we can
consider DL-based code summarization techniques as abstractive methods.

In this paper, we utilize extractive and abstractive methods together. The
former is responsible for extracting important factual details while the
latter is responsible for generating human-written-like natural language
summary, detailed in Section 4.

###  2.2. Neural Machine Translation

Figure 2. Framework of DL-based Code Summarization

Neural machine translation (NMT) aims to automatically translate one language
(e.g., French) into another language (e.g., English) while preserving
semantics (Cho et al., 2014; Bahdanau et al., 2015; Hu et al., 2018a). NMT has
been shown to achieve great success for natural language corpora (Bahdanau et
al., 2015; Hu et al., 2018a). It is typically thought of in terms of sequence
to sequence (seq2seq) learning, in which an natural language sentence (e.g.,
French sentence) is one sequence and is converted into a semantically
equivalent target sequence (e.g, English sentence) (LeClair et al., 2019).
Code summarization can also be regarded as a kind of translation task, which
translates programming language into natural language (Gros et al., 2020).
Several recent papers (Hu et al., 2018a; Iyer et al., 2016; LeClair et al.,
2019; Alon et al., 2018) have explored the idea of applying seq2seq model to
translate code snippets into the natural language comments. Similar to NMT,
code summarization methods based on seq2seq models also widely adopt DL-based
encoder-decoder architectures. Figure 2 shows the general framework of the
code summarization technique based on the encoder-decoder architecture. From
the figure, we can observe that the DL-based code summarization technique
usually consists of two key components, an encoder and a decoder. Both encoder
and decoder are neural networks, so DL-based code summarization is also known
as neural code summarization (Zhang et al., 2020; Shi et al., 2021b). The
encoder is an embedding network that can encode the code snippet $c$ given by
the developer into a $d$-dimensional embedding representation
$\bm{e}^{c}\in\mathbb{R}^{d}$. To train such an encoder, existing DL-based
code summarization techniques have tried various neural network architectures,
such as LSTM (Iyer et al., 2016; Hu et al., 2018a; Wang et al., 2020),
Bidirectional-LSTM (Li et al., 2021; Zhang et al., 2020; Wei et al., 2020),
GRU (Hu et al., 2020; Wang et al., 2021a; LeClair et al., 2019), Transformer
(Ahmad et al., 2020; Lin et al., 2021; Shahbazi et al., 2021) and GNN (LeClair
et al., 2020; Liu et al., 2020). The decoder is also a neural network that can
decode the embedding representation $\bm{e}^{c}$ into a natural language
summary. To train such a decoder, existing DL-based code summarization
techniques usually adopt the same neural network architecture as the encoder.
In DL-based code summarization studies, it is a common practice to use code
comments as summaries during the training process (Hu et al., 2018a; Li et
al., 2021). Code comments are natural language descriptions used to explain
what the code snippets want to do (Hu et al., 2018a). For example, Figure 1(b)
is a comment for the code snippet $c_{1}$. Therefore, we do not strictly
distinguish the meaning of the two terms comment and summary, and use the term
comment during the training process, and summary at other time.

##  3\. Motivating Example

Figure 3. Motivation Example

In this section, we take the code snippet $c_{1}$ in Figure 1(a) as an example
and apply different tools to generate the summary for comparison. It is a
real-world example from the CodeSearchNet dataset (Husain et al., 2019) (see
details in Section 5.1.1). Figure 1(b) shows the comment ($s_{1}$) written by
the developer for $c_{1}$. We consider this as a reference summary (the ground
truth), as shown in the first line of Figure 3(b). According to the grammar
rules in natural language, we can simply divide the reference summary into
four parts: “removes the mapping” (Blue font), “for the specified key” (Green
font), “from this cache” (Red font), and “if present”(Orange font).

We study two existing techniques, an extractor method (Haiduc et al., 2010a)
and an abstracter method (Feng et al., 2020), in generating summaries for the
given example. The extractor method (Haiduc et al., 2010a) adopts the Latent
Semantic Analysis (LAS) techniques (Steinberger and Jezek, 2009) to determine
the informativity of every term in the code snippet and then selects the top 5
important terms to compose the summary. The second summary in Figure 3(b)
(Extractive Summary) is generated by (Haiduc et al., 2010a). Observe that
although the extractive summary has a poor naturalness and is far from the
reference summary, it contains important factual details that should be
included in the summary, e.g., the important terms “key” and “cache”. The
abstractive method (Feng et al., 2020) first trains a model called CodeBert
for obtaining code representations, and then fine-tunes it on the code
summarization task. The third summary in Figure 3(b) (Abstractive Summary)
shows the result by (Feng et al., 2020). Observe that 1) intuitively, the
abstractive summary has a good naturalness and like written by a human; 2) the
abstractive summary can cover the first and the third parts (Blue and Red
fonts) of the reference summary; 3) the abstractive summary can not cover the
second and the fourth parts (Green and Orange fonts), i.e., missing some
factual details.

Our solution. We also use our EACS to generate a summary for $c_{1}$. The last
summary (i.e., ExAbstractive Summary) in Figure 3(b) is generated by EACS. We
can observe that 1) compared with the extractive summary generated by (Haiduc
et al., 2010a), the exabstractive summary has a good naturalness and like
written by a human; 2) compared with the abstractive summary generated by
(Feng et al., 2020), the exabstractive summary can cover all of the four parts
and is closer to the reference summary. Based on the above observations, it is
obvious that our method can generate human-written-like summaries while
preserving important factual details. We owe the good performance of our
method on their ability to give more attention to important statements.
Intuitively, the second part (Green font) is a translation or summary of the
factual details (key terms “key”) contained in the important statements V
oldValue = cacheMap.remove(key); and  Log.debug(“Removed cache entry for””,
key);. The fourth part (Orange font) is a translation or summary of the
factual details contained in the important conditional statement if(oldValue)
!= null). The above three important statements are successfully predicted by
our extractive module, as shown Figure 3(b). More details about the extractive
module are introduced in Section 4.2.1).

##  4\. Design

Figure 4. The Overview of EACS

###  4.1. Overview

Figure 4 illustrates the overview of our approach EACS. The top part shows the
training process of EACS and the bottom part shows the deployment (usage) of
EACS for a given code snippet. EACS decomposes the training process into two
phases: (i) training of extractor and (ii) training of abstracter. The
training of the extractor aims to produce a well-trained extractor capable of
extracting important statements from a given code snippet. The training of the
abstracter aims to produce a well-trained abstracter capable of generating a
succinct natural language summary for a given code snippet. Both phases
leverage the same two types of input data: comments and code snippets. To
train the extractor, EACS first produces ground-truth important statements
(GIStates) based on the informativity of each statement in the code snippet,
detailed in Section 4.2.1-Step ➀. Then, EACS uses the extractor to extract
predicted important sentences (PIStates), detailed in Section 4.2.1-Step ➁.
During this procedure, the model parameters of the extractor is randomly
initialized. Based on the loss ($\mathcal{L}_{Ex}$) computed based on PIStates
and GIStates, EACS can iteratively update the model parameters of the
extractor, detailed in Section 4.2.1-Step ➂. To train the abstracter, given a
code snippet, EACS first uses the well-trained extractor to extract the
important statements (IStates), detailed in Section 4.2.2-Step ➃. These
important statements will be further transformed into the embedding
representation $\bm{e}^{Ex}$ by an encoder named ExEncoder, detailed in
Section 4.2.2-Step ➄. Then, EACS uses another encoder named AbEncoder to
transform the entire code snippet into the embedding representation
$\bm{e}^{Ab}$, detailed in Section 4.2.2-Step ➅. Further, EACS produces the
fused embedding representation $\bm{e}^{Fu}$ by fusing $\bm{e}^{Ex}$ and
$\bm{e}^{Ab}$, detailed in Section 4.2.2-Step ➆. The fused embedding
representation $\bm{e}^{Fu}$ will be passed to a decoder (Decoder) to generate
predicted summaries (PSummaries). During this procedure, the model parameters
of the abstracter (including ExEncoder, AbEncoder, and Decoder) are randomly
initialized. Finally, based on the loss ($\mathcal{L}_{Ab}$) between the
predicted summaries (PSummaries) and ground-truth summaries (i.e., comments),
EACS can iteratively update the model parameters of the abstracter, detailed
in Section4.2.2-Step ➈. The well-trained extractor and abstracter are the two
core components of EACS to support the code summarization service. When EACS
is deployed for usage, it takes in a code snippet from the developer and
produces a succinct natural language summary for the code snippet, detailed in
Section 4.3.

###  4.2. Training of EACS

####  4.2.1. Part (i): Training of Extractor

Let $c$ denote a code snippet containing a set of statements
$stat=[stat_{1},stat_{2},\cdots,stat_{n}]$, where $stat_{i}$ is the $i$-th
statement in $c$. Similar to extractive text summarization (Liu, 2019),
extractive code summarization can be defined as the task of assigning a label
$l_{i}\in\\{0,1\\}$ to each $stat_{i}$, indicating whether the factual details
contained in the statement $stat_{i}$ should be included in the summary. It is
assumed that the summary represents the important content of the code snippet.
Therefore, we can consider the extractor as a classifier that takes in all
statements of a code snippet and predicts which statements should be selected
as important ones.

As shown in part (i) of Figure 4, the training of the extractor is completed
through three steps: ➀ producing ground-truth important Statements (GIStates)
based on pairs of comments and code snippets, ➁ producing predicted important
statements (PIStates) using the extractor, and ➂ calculating the loss
($\mathcal{L}_{Ex}$) based on GIStates and PIStates and updating the model
parameters of the extractor. We discuss the three steps in detail in the
following sections.

Step ➀: Producing Ground-truth Important Statements. As mentioned earlier, we
aim to train an extractor with the ability to take in a sequence of statements
of the code snippet $c$ and output the predicted labels of the statements in
$c$. We denote the labels of the statements in $c$ as
$l=[l_{1},l_{2},\cdots,l_{n}]$, where $l_{i}\in\\{0,1\\}$ is the label of the
statement $stat_{i}$. $l_{i}=1$ means the $stat_{i}$ is an informative
(important) statement, otherwise it is not. To train such an extractor, we
need to build a training dataset, where each sample is a pair of $stat$ and
the corresponding labels $\hat{l}$. We consider $\hat{l}$ as the ground-truth
label of $stat$.

To obtain the ground-truth labels, like (Hsu et al., 2018), we first measure
the informativity of each statement $stat_{i}\in stat$ by computing the
ROUGE-L recall score (Lin, 2004) between the statement $stat_{i}$ and the
reference statements. We consider the comments of code snippets as reference
statements. Second, we sort the statements by their informativity and select
the statement in the order of high to low informativity. We add one statement
each time when the new statement can increase the informativity of all the
selected statements. Finally, we obtain the ground-truth labels, i.e., all
ground-truth important statements (GIStates).

Step ➁: Producing Predicted Important Statements. As shown in part (i) Figure
4, we use an extractor (Extractor) to predict important statements in the code
snippet. The extractor is a neural network model that consists of an encoder
which transforms statements into embedding representations and a
classification layer for predicting the labels of statements.

The encoder essentially performs the code representation task of transforming
complex source code into a numerical (embedding) representation that is
convenient for program computation while preserving semantics. Therefore, a
large number of existing deep learning-based code representation techniques
can be adopted by our EACS to design the encoder. Specifically, the embedding
representations of the statements $\bm{e}^{stat}$ can be formalized as
$\bm{e}^{stat}=encoder(stat)$, where the encoder is a deep learning-based
neural network architecture (e.g., LSTM (Hochreiter and Schmidhuber, 1997)) or
a pre-trained model (e.g., CodeBert (Feng et al., 2020)) that can numericalize
sequences of code statements with preserving semantics (i.e., embedding
representations). In practice, we tried multiple neural network architectures
(e.g., LSTM (Hochreiter and Schmidhuber, 1997) and Transformer (Vaswani et
al., 2017)) and pre-trained models (e.g., CodeBert (Feng et al., 2020) and
CodeT5 (Wang et al., 2021b)), and found that, in the code summarization task,
the encoder obtained by fine-tuning a pre-trained model performed better than
that trained from scratch based on the neural network architecture, detailed
in Section 5.2.2. We do not repeatedly present the design of the neural
network architectures or pre-training models involved in the paper. Please
read the corresponding paper for more details. The classification layer is
connected to the embedding representation $\bm{e}^{stat}$. In practice, we
adopt $softmax$ as the classification layer, i.e., $l=softmax(\bm{e}^{stat})$.

Step ➂: Model Training. During training, we update the model parameters
$\Theta$ of the extractor regarding the sigmoid cross entropy loss, i.e.,
$\mathcal{L}_{Ex}(\Theta)$ computed as:

(1)
$\small\mathcal{L}_{Ex}(\Theta)=-\frac{1}{N}\sum_{n=1}^{N}{\hat{l}_{n}log{l_{n}}+(1-\hat{l}_{n})log(1-l_{n})}$

where $\hat{l}_{n}\in{0,1}$ is the ground-truth label for the $n$th statement
and $N$ is the number of statements. When $\hat{l}_{n}=1$, the factual details
contained in the $n$th statement should be attended to facilitate final
summary generation.

####  4.2.2. Part(ii): Training of Abstracter

As shown in part (ii) of Figure 4, the training of the abstracter is completed
through five steps: ➃ extracting important statements (IStates) from the given
code snippet using the well-trained extractor, ➄ and ➅ producing embedding
representations ($\bm{e}^{Ex}$ and $\bm{e}^{Ab}$) of the important statements
and the entire code snippet, ➆ producing the fused representation
$\bm{e}^{Fu}$ based on $\bm{e}^{Ex}$ and $\bm{e}^{Ab}$, ➇ generating predicted
summary with the help of the decoder, and ➇ computing the loss
$\mathcal{L}_{Ab}$ based on the predicted summaries (PSummaries) and the
ground-truth summaries (comments) to update the model parameters of the
abstracter. We discuss the six steps in detail as follows.

Step ➃: Extracting Important Statements. To generate summaries without missing
factual details, our EACS pays more attention to the important statements of
code snippets where the factual details are contained in. Therefore, different
from abstracters in existing abstract code summarization techniques, the
abstracter of EACS treats important sentences as part of the input. In this
step, we first use the well-trained extractor produced at the phase of
training of the extractor to predict the labels of statements of the given
code snippet. Then, the statements with the label 1 will be selected as
important statements (IStates).

Step ➄ and Step ➅: Producing Embedding Representations. Step ➄ and Step ➅ do a
similar thing, i.e. leveraging an encoder to transform the source code into an
embedding representation. The difference is that Step ➄ deals with important
statements selected by the extractor and Step ➅ deals with the entire code
snippet. Therefore, in EACS, we can use the same neural network architecture
or pre-trained model to design ExEncoder and AbEncoder. Given a code snippet
$c=[stat_{1},stat_{2},\cdots,stat_{n}]$, let $c^{\prime}\subseteq c$ denote a
set of the important statements selected by the extractor from $c$, the tasks
performed by ExEncoder and AbEncoder can be formalized as follows:

(2) $\small\bm{e}^{Ex}=encoder(c^{\prime}),\;\;\;\;\bm{e}^{Ab}=encoder(c)$

where $\bm{e}^{Ex}$ and $\bm{e}^{Ab}$ represent the embedding representations
of $c^{\prime}$ and $c$; the encoder is a neural network architecture (e.g.,
LSTM (Hochreiter and Schmidhuber, 1997), Transformer (Vaswani et al., 2017))
or pre-trained model (e.g., CodeBert (Feng et al., 2020)) that can process
sequential input. ExEncoder and AbEncoder perform the similar task — producing
embedding representations of the code, so we design them with the same code
representation techniques as the extractor’s encoder.

Step ➆: Producing Fused Representation. In this step, EACS fuses $\bm{e}^{Ex}$
and $\bm{e}^{Ab}$ to produce a fused embedding representation $e^{Fu}$ through
the component Fusioner. Considering that $\bm{e}^{Ex}$ and $\bm{e}^{Ab}$ are
not aligned, we fuse them in a concatenated fashion. We try two concatenated
ways as follows:

(3)
$\small\bm{e}^{Fu}=[\bm{e}^{Ex};\bm{e}^{Ab}]~{}\text{or}~{}[\bm{e}^{Ab};\bm{e}^{Ex}]$

where $[\cdot;\cdot]$ denotes the concatenation of two vectors. The effects of
both ways on the performance of EACS are discussed in Section 5.2.3.

Step ➇: Generating Predicted Summaries. In this section, we utilize the
decoder to generate natural language summary, which takes in the fused
embedding representation $\bm{e}^{Fu}$ and predicts word one by one.
Specifically, the decoder based on a neural network (e.g., LSTM) is to unfold
the context vector $\bm{e}^{Fu}$ into the target sequence (i.e., the word
sequence of the summary), through the following dynamic model,

(4) $\displaystyle\bm{h}_{t}=f(y_{t-1},\bm{h}_{t-1},\bm{e}^{Fu})$
$\displaystyle p(y_{t}|Y_{<t},X)=g(y_{t-1},\bm{h}_{t},\bm{e}^{Fu})$

where $f(\cdot)$ and $g(\cdot)$ are activation functions, $\bm{h}_{t}$ is the
hidden state of the neural network at time $t$, $y_{t}$ is the predicted
target word at $t$ (through $g(\cdot)$ with $Y_{<t}$ denoting the history
$\\{y_{1},y_{2},\cdots,y_{t-1}\\}$. The prediction process is typically a
classifier over the vocabulary. It can be seen from Equation 4 that the
probability of generating target word is related to the current hidden state,
the history of the target sequence and the context $\bm{e}^{Fu}$. The essence
of the decoder is to classify the vocabularies by optimizing the loss function
in order to generate the vector representing the feature of the target word
$y_{t}$. After the vector passes through a softmax function, the word
corresponding to the highest probability is the result to be output.

Step ➈: Model Training. During training of the abstracter, the three
components (ExEncoder, AbEncoder, and Decoder) are jointly trained to minimize
the negative conditional log-likelihood, i.e, $\mathcal{L}_{Ab}(\Theta)$
computed as:

(5)
$\small\mathcal{L}_{Ab}(\Theta)=-\frac{1}{N}\sum_{n=1}^{N}logp(\bm{y}_{n}|\bm{x}_{n};\Theta)$

where $\Theta$ is the model parameters of the abstracter and each
$(\bm{x}_{n},\bm{y}_{n})$ is an (code snippet, comment) pair from the training
set.

###  4.3. Deployment of EACS

After EACS is trained, we can deploy it online for code summarization service.
Part (2) of Figure 4 shows the deployment of EACS. For a code snippet $c$
given by the developer, EACS first uses the well-trained extractor to extract
important statements from $c$, represented $c^{\prime}$. Then, EACS uses the
well-trained abstracter to generate the summary. In practice, we can consider
the well-trained EACS as a black-box tool that takes in a code snippet given
by the developer and generates a succinct natural language summary.

##  5\. Evaluation

To evaluate our approach, in this section, we aim to answer the following four
research questions:

RQ1::

How does EACS perform compared to the state-of-the-art baselines?

RQ2::

How does EACS perform when combined with different neural network
architectures and pre-trained models?

RQ3::

How does the fusion way of the extractor and abstracter affect the performance
of EACS?

RQ4::

How does the robustness of EACS perform when varying the code length and
comment length?

###  5.1. Experimental Setup

####  5.1.1. Dataset

We conduct experiments on three datasets, including a Java dataset (JCSD) (Hu
et al., 2018b), a Python dataset (PCSD) (Barone and Sennrich, 2017), and a
CodeSearchNet corpus (Husain et al., 2019), which have been widely used by
existing code summarization studies (Wei et al., 2019; Feng et al., 2020;
Ahmad et al., 2020; Zhang et al., 2020; Wu et al., 2021; Gong et al., 2022).
The CodeSearchNet corpus contains codes across six programming languages (Go,
Java, JavaScript, PHP, Python, and Ruby).

Note that existing studies (Feng et al., 2020; Wang et al., 2021b) evaluate
the effectiveness of pre-trained models for code representation (e.g.,
CodeBert (Feng et al., 2020) and CodeT5 (Wang et al., 2021b)) through the
following two steps: 1) fine-tuning them on the training set of downstream
tasks (e.g. code summarization task) to produce task-specific models; 2)
evaluating the effectiveness of the task-specific models on the corresponding
test set, which indirectly indicates the effectiveness of the pre-trained
models. Therefore, to verify the effectiveness of the pre-trained model, code
representation techniques (e.g., CodeBert) divide the CodeSearchNet corpus
into two parts before training the model. One part is used for training the
pre-trained model, and the other part is used as downstream task data to
evaluate the effectiveness of the pre-trained model. As mentioned earlier, we
can use pre-trained models as the encoders of EACS. Therefore, the encoders
have seen (learned) the part of the CodeSearchNet corpus used for training the
pre-trained models. For a fair comparison, in this paper, we only use the
unseen data used as downstream code summarization task data. In practice, we
use the cleaned dataset for code summarization task published by CodeBert
(Feng et al., 2020). The statistics of the three datasets are shown in Table
1.

Table 1. Statistics of datasets Dataset | Training Set Size | Validation Set Size | Test Set Size
---|---|---|---
JCSD | 69,708 | 8,714 | 8,714
PCSD | 57,203 | 19,067 | 19,066
Go | 167,288 | 7,325 | 8,122
Java | 164,923 | 5,183 | 10,955
JavaScript | 58,025 | 3,885 | 3,291
PHP | 241,241 | 12,982 | 14,014
Python | 251,820 | 13,914 | 14,918
Ruby | 24,927 | 1,400 | 1,262

####  5.1.2. Evaluation Metrics

We use three metrics BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie,
2005), and ROUGE (Lin, 2004), to evaluate the model, which are widely used in
code summarization (Gong et al., 2022; Wu et al., 2021; Wan et al., 2018; Hu
et al., 2018b; Vaswani et al., 2017; Iyer et al., 2016).

BLEU, the abbreviation for BiLingual Evaluation Understudy (Papineni et al.,
2002), is widely used for evaluating the quality of generated code summaries
(Wan et al., 2018; Hu et al., 2018b; Iyer et al., 2016). It is a variant of
precision metric, which calculates the similarity by computing the n-gram
precision of a generated summary to the reference summary, with a penalty for
the overly short length (Papineni et al., 2002). It is computed as:

(6) $\small BLEU=BP*exp(\sum_{n=1}^{N}{w_{n}logp_{n}})$

(7) $\small BP=\begin{cases}1,&\text{if $|g|>|r|$}\\\
e^{(1-\frac{|r|}{|g|})},&\text{if $|g|\leq|r|$}\end{cases}$

where $N=1,2,3,4$ and $w_{n}=\frac{1}{N}$. $BP$ represents the brevity
penalty. $g$ and $r$ denote a generated (predicted) summary and a reference
summary, respectively. $|g|$ and $|r|$ denote the lengths of $g$ and $r$,
respectively. BLEU-4 is often of interest as it can reflect the weighted
results from 1 through 4 (Chen et al., 2021).

METEOR, the abbreviation for Metric for Evaluation of Translation with
Explicit ORdering (Banerjee and Lavie, 2005), is also widely used to evaluate
the quality of generated code summaries (Yang et al., 2021; Zhang et al.,
2020; Wang et al., 2020). For a pair of summaries, METEOR creates a word
alignment between them and calculates the similarity scores. Suppose $m$ is
the number of mapped unigrams between the reference summary $r$ and the
generated summary $g$, respectively. Then, precision ($P_{unig}$), recall
($P_{unig}$), and METEOR are computed as:

(8) $\small P_{unig}=\frac{m}{|g|},\;\;R_{unig}=\frac{m}{|r|}$

(9) $\small
METEOR=(1-\gamma*frag^{\beta})*\frac{P_{unig}*R_{unig}}{\alpha*P_{unig}+(1-\alpha)*R_{unig}}$

where $frag$ is the fragmentation fraction. As in (Zhang et al., 2020);
$\alpha$, $\beta$, and $\gamma$ are three penalty parameters whose default
values are 0.9, 3.0 and 0.5, respectively.

ROUGE-L. ROUGE is the abbreviation for Recall-oriented Understudy for Gisting
Evaluation (Lin, 2004). ROUGE-L, a variant of ROUGE, is computed based on the
longest common subsequence (LCS). ROUGE-L is also widely used to evaluate the
quality of generated code summaries (Bansal et al., 2021; Lin et al., 2021;
Shahbazi et al., 2021). Specifically, the LCS-based F-measure ($F_{lcs}$) is
called ROUGE-L (Lin, 2004), and $F_{lcs}$ is computed as:

(10) $R_{lcs}=\frac{LCS(r,g)}{|r|},\;\;P_{lcs}=\frac{LCS(r,g)}{|g|}$

(11) $\small
F_{lcs}=\frac{(1+\beta^{2})R_{lcs}P_{lcs}}{R_{lcs}+\beta^{2}P_{lcs}}$

where $r$ and $g$ also denote the reference summary and the generated summary,
respectively. Notice that ROUGE-L is 1 when $g=r$; while ROUGE-L is 0 when
$LCS(r,g)=0$, i.e., which means $r$ and $g$ are completely different. $\beta$
is set to 1.2 as in (Wang et al., 2021a; Wan et al., 2018; Zhang et al.,
2020).

The scores of BLEU, ROUGE-L and METEOR are in the range of [0,1] and usually
reported in percentages. The higher the scores, the closer the generated
summary is to the reference summary, and the better the code summarization
performance.

####  5.1.3. Experimental Settings

To train models, we first shuffle the training data and set the mini-batch
size to 32. For each batch, the code snippets are padded with a special token
$\langle PAD\rangle$ to the maximum length. We set the word embedding size to
512. For LSTM unit, we set the hidden size to 512. The margin $\beta$ is set
to 0.6. We update the parameters via AdamW optimizer (Kingma and Ba, 2015)
with the learning rate 0.0003. To prevent over-fitting, we use dropout with
0.1. All models are implemented using the PyTorch 1.7.1 framework with Python
3.8. All experiments are conducted on a server equipped with one Nvidia Tesla
V100 GPU with 31 GB memory, running on Centos 7.7. All the models in this
paper are trained for the same epochs as their original paper, and we select
the best model based on the lowest validation loss.

###  5.2. Experimental Results

####  5.2.1. RQ1: EACS vs. Baselines

1) Baselines: To answer this research question, we compare our approach EACS
to the following DL-based code summarization techniques.

CODE-NN (Iyer et al., 2016) adopts a LSTM-based encoder-decoder architecture
with attention mechanism. It is a classical encoder-decoder framework in NMT
that encodes tokens of code snippets into embedding representations and then
generates summaries in the decoder with the attention mechanism.

DeepCom (Hu et al., 2018a) also adopts a LSTM-based encoder-decoder
architecture with attention mechanism. In addition, to capture the structural
information, DeepCom proposes a structure-based traversal method to traverse
AST sequences of the code snippet. The AST sequences are further passed to the
encoder and decoder to generate summaries.

Hybrid-DRL (Wan et al., 2018) (also shortened to RL+Hybrid2Seq in (Ahmad et
al., 2020), Hybrid2Seq in (Wu et al., 2021)) also adopts a LSTM-based encoder-
decoder architecture and is trained with reinforcement learning. It also
designs additional encoder based on an AST-based LSTM to capture the
structural information of the code snippet. It uses reinforcement learning to
solve the exposure bias problem during decoding, which obtains better
performance.

TL-CodeSum (Hu et al., 2018b) (also shortened to API+Code in (Wu et al.,
2021)) adopts a GRU-based encoder-decoder architecture with attention
mechanism. It encodes Application Programming Interface (API) sequence along
with code token sequence, then generates summary from source code with
transferred API knowledge. It introduces API sequence summarization task,
aiming to train an API sequence encoder by using an external dataset so that
it can learn more abundant representations of the code snippet.

Dual Model (Wei et al., 2019) also adopts a LSTM-based encoder-decoder
architecture with attention mechanism. It treats code summarization and code
generation as a dual task. It trains the two tasks jointly by a dual training
framework to simultaneously improve the performance of code summarization and
code generation tasks.

Transformer-based (Ahmad et al., 2020) (also shortened to Transformer in (Wu
et al., 2021), NCS in (Shi et al., 2021b)) adopts a Transformer-based encoder-
decoder architecture. It incorporates the copying mechanism (See et al., 2017)
in the Transformer to allow both generating words from vocabulary and copying
from the source code.

SiT (Wu et al., 2021) adopts a Transformer-based encoder-decoder architecture.
It proposes structure-induced transformer to capture long-range dependencies
and more global information in AST sequences of code snippets.

2) Results: Table 2 shows the performances of our EACS and baselines in terms
of the three evaluation metrics, i.e., BLEU ($\mathcal{B}$), METEOR
($\mathcal{M}$), and ROUGE-L ($\mathcal{R}$).

Table 2. Performance of Our EACS and Baselines. $*$ refers to methods we rerun. The results of upper part (lines 3–9) are directly reported from (Wu et al., 2021; Ahmad et al., 2020). Note that we only rerun SiT since it is much stronger than the other baselines. Techniques | JCSD | PCSD
---|---|---
$\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$ | $\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$
CODE-NN | 27.60 | 12.61 | 41.10 | 17.36 | 9.29 | 37.81
DeepCom | 39.75 | 23.06 | 52.67 | 20.78 | 9.98 | 37.35
Hybrid-DRL | 38.22 | 22.75 | 51.91 | 19.28 | 9.75 | 39.34
TL-CodeSum | 41.31 | 23.73 | 52.25 | 15.36 | 8.57 | 33.65
Dual Model | 42.39 | 25.77 | 53.61 | 21.80 | 11.14 | 39.45
Transformer-based | 44.58 | 26.43 | 54.76 | 32.52 | 19.77 | 46.73
SiT | 45.76 | 27.58 | 55.58 | 34.11 | 21.11 | 48.35
SiT∗ | 45.22 | 27.10 | 55.44 | 33.75 | 21.02 | 48.33
EACS | 47.66 | 30.39 | 58.77 | 35.96 | 23.70 | 51.83

(a) Distribution of BLEU Scores

(b) Distribution of METEOR Scores

(c) Distribution of ROUGE-L Scores

Figure 5. Distributions of Metrics’ Scores

From Table 2, in all baselines, SiT performs the best on both datasets in
terms of all three metrics. However, our EACS is more powerful than SiT and
achieves more impressive performance. On the JCSD dataset, compared with the-
state-of-the art (SiT), EACS improves by 5.4% in BLEU-4, 12.14% in METEOR and
6% in ROUGE-L. On the PCSD dataset, EACS also clearly outperforms SiT,
improving by 6.55% in BLEU-4, 12.75% in METEOR and 7.24% in ROUGE-L. It should
be noted that the values in Table 2 are the average scores of all test
samples. For a more comprehensive comparison, we further statistic the
distribution of the scores of SiT and EACS on all test samples, and the
statistical results are shown in Figure 5. In Figure 5, ‘+’ denotes the mean,
which is the value filled in Table 2. Overall, the score distribution of EACS
is better than that of SiT. To test whether there is a statistically
significant difference between EACS and SiT, we perform the unpaired two-
tailed Wilcoxon-Mann-Whitney test, at a significance level of 5%, following
previously reported guidelines for inferential statistical analysis involving
randomized algorithms (Arcuri and Briand, 2014; Gligoric et al., 2015). In
Figure 5, ‘*’ ($0.01<p<0.05$) and ‘****’ ($p<0.0001$) represent the
differences between two groups are Significant and Extremely significant,
respectively. ‘ns’ ($p\geq 0.05$) means Not significant. From the figure, we
can intuitively observe that in all three metrics, EACS outperforms SiT on
both the JCSD and PCSD datasets. In summary, the results and observations on
the above demonstrate that under all experimental settings, our EACS
consistently achieves higher performance in all three metrics, which indicates
better code summarization performance.

####  5.2.2. RQ2: Effectiveness of EACS When Combing with Different Networks
Architectures or Pre-trained Models

The extractive-and-abstractive framework we proposed is a general code
summarization framework that does not depend on a specific deep learning
network/model. In other words, EACS can be combined with many advanced neural
network architectures or pre-trained models. In this section, we combine EACS
with several popular and advanced neural network architectures (including LSTM
(Hochreiter and Schmidhuber, 1997) and Transformer (Vaswani et al., 2017)) and
pre-trained models (including CodeBert (Feng et al., 2020) and CodeT5 (Wang et
al., 2021b)) to explore the potential of EACS in multilingual code
summarization tasks. These neural network architectures and pre-trained models
are widely used in code summarization (Hu et al., 2018a; Shido et al., 2019;
Ahmad et al., 2020; Feng et al., 2020; Wu et al., 2021). The experimental
results are shown in Table 3. In the table, rows 3 (“LSTM”) and 5
(“Transformer”) represent that we implement purely LSTM-based and Transformer-
based encoder-decoder frameworks, respectively, to perform the code
summarization task. Rows 4 (“EACS + LSTM”) and 6 (“EACS + Transformer”)
represent that on the basis of the above (Rows 3 and 5), the well-trained
extractor module is added. Rows 7 and 9 represent that we fine-tune the pre-
trained models released on the GitHub (i.e., CodeBert (Microsoft, 2021) and
CodeT5 (Salesforce, 2021)) on the code summarization task. Analogously, rows 8
(“EACS + CodeBert”) and 10 (“EACS + CodeT5”) represent that on the basis of
the above (Rows 7 and 9), the well-trained extractor module is added.

Table 3. Effectiveness of EACS when Combined with Different Neural Network Architectures (LSTM and Transformer) and Pre-trained Models (CodeBert and CodeT5) on the CodeSearchNet corpus (Six Programming Languages) Method | Go | Java | JavaScript | PHP | Python | Ruby
---|---|---|---|---|---|---
$\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$ | $\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$ | $\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$ | $\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$ | $\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$ | $\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$
LSTM | 17.8 | 15.1 | 35.6 | 12.2 | 10.1 | 24.6 | 10.4 | 6.2 | 17.2 | 19.5 | 12.2 | 29.8 | 13.9 | 9.1 | 23.3 | 9.4 | 5.3 | 16.3
EACS + LSTM | 17.9 | 15.2 | 35.8 | 13.4 | 10.2 | 24.7 | 10.5 | 6.4 | 17.4 | 19.6 | 12.4 | 30.0 | 14.1 | 9.2 | 23.4 | 9.9 | 5.3 | 16.5
Transformer | 19.8 | 16.2 | 38.4 | 15.3 | 11.8 | 30.6 | 11.2 | 7.4 | 20.5 | 21.5 | 13.9 | 34.2 | 15.8 | 10.6 | 31.3 | 10.3 | 6.4 | 18.3
EACS + Transformer | 20.1 | 16.8 | 39.2 | 15.8 | 12.3 | 31.2 | 11.5 | 7.6 | 21.3 | 22.3 | 14.1 | 34.8 | 16.3 | 10.9 | 32.1 | 10.7 | 6.7 | 18.8
CodeBert | 21.1 | 17.5 | 43.6 | 18.0 | 12.4 | 35.5 | 13.3 | 8.7 | 24.3 | 24.6 | 15.3 | 39.4 | 18.7 | 12.4 | 34.8 | 11.2 | 7.1 | 20.6
EACS + CodeBert | 22.2 | 18.7 | 44.8 | 19.3 | 13.8 | 36.8 | 14.3 | 9.6 | 25.1 | 25.4 | 16.1 | 40.2 | 19.4 | 13.2 | 35.8 | 12.4 | 8.2 | 22.6
CodeT5 | 21.5 | 18.9 | 45.9 | 20.2 | 15.3 | 39.3 | 15.8 | 11.2 | 28.9 | 25.9 | 18.2 | 43.6 | 20.0 | 15.1 | 37.8 | 14.9 | 10.8 | 27.9
EACS + CodeT5 | 23.9 | 20.0 | 47.3 | 21.3 | 16.8 | 41.1 | 16.7 | 12.1 | 30.3 | 26.9 | 18.8 | 44.2 | 20.5 | 16.0 | 38.9 | 15.2 | 11.8 | 29.2

From rows 3-6, we can observe that the performance of the combination of EACS
and architecture is significant better than that of only architecture-based
methods in terms of all three metrics. Among them, the combination of EACS and
Transformer (i.e., “EACS + Transformer”) performs the best. In addition, from
rows 7-10, we can observe that the performance of the combination of EACS and
pre-trained models is significant better than that of purely pre-trained
model-based methods in terms of all three metrics. Among them, the “EACS +
CodeT5” performs the best. We can also observe that the combinations of EACS
with pre-trained models significantly outperforms the combination with neural
network architectures. This is mainly because pre-trained models for code
representation are usually trained on larger-scale datasets, so they have
stronger code representation capabilities and can represent code semantics
more accurately. Based on the above results and observations, we can make a
conclusion that our EACS is a general code summarization framework and can be
easily combined with advanced neural network architectures or pre-trained
models. We have reasons to believe that EACS can be combined with more
advanced neural network architectures or models to exert more powerful
performance in the future.

####  5.2.3. RQ3: Influence of Fusion Ways of Extractor and Abstracter on EACS

In this section, we further explore the influence of fusion ways between the
extractor and abstracter on the performance of EACS. In practice, as mentioned
earlier, we try to fuse the outputs of the ExEncoder and AbEncoder by the
following two concatenated ways: $[\bm{e}^{Ex};\bm{e}^{Ab}]$ and
$[\bm{e}^{Ab};\bm{e}^{Ex}]$. The experimental results are shown in Table 4. In
the table, rows 3 and 4 ($[\bm{e}^{Ex}/\bm{e}^{Ab};\bm{e}^{Ab}/\bm{e}^{Ex}]$)
means that we use the way of concatenation to fuse the embeddings generated by
ExEncoder/AbEncoder and AbEncoder/ExEncoder.

Table 4. Influence of Fusion Ways on EACS Techniques | JCSD | PCSD
---|---|---
$\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$ | $\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{R}$
$[\bm{e}^{Ex};\bm{e}^{Ab}]$ | 47.29 | 32.00 | 58.14 | 35.56 | 23.53 | 51.69
$[\bm{e}^{Ab};\bm{e}^{Ex}]$ | 47.66 | 30.39 | 58.77 | 35.96 | 23.70 | 51.83

From Table 4, we can observe that in general, the fusion way
$[\bm{e}^{Ab};\bm{e}^{Ex}]$ is better than $[\bm{e}^{Ex};\bm{e}^{Ab}]$. This
is in line with human intuition. When summering a code snippet, human should
first browse the whole code snippet and then write a summary according to the
key content.

####  5.2.4. RQ4: Robustness of EACS

(a) Code Snippets on Testing Set

(b) Comments on Testing Set

Figure 6. Length Distributions

(a) Varying Code Snippet Lengths

(b) Varying Comment Lengths

Figure 7. Robustness of EACS

To analyze the robustness of EACS, we study two parameters (i.e., code length
and comment length) that may have an influence on the embedding
representations of code snippets and comments. Figure 6(a) and (b) show the
length distributions of code snippets and comments on the testing set. For a
code snippet, its length refers to the number of lines of the code snippet.
For a comment, its length refers to the number of words in the comment. From
Figure 6(a), we can observe that the lines of most code snippets are located
between 20 to 40. This was also observed in the quote in (Martin, 2009)
“Functions should hardly ever be 20 lines long”. From Figure 6(b), it is
noticed that almost all comments are less than 20 in length. This also
confirms the challenge of capturing the correlation between code snippet with
its corresponding comment (summary).

Figure 7 shows the performance of EACS based on different evaluation metrics
with varying parameters. From Figure 7, we can observe that in most cases,
EACS maintains a stable performance even though the code snippet length or
comment length increases, which can be attributed to the extractive-and-
abstractive framework we proposed. When the length of the comment exceeds 25
(a common range shown in 6(b)), the METEOR and ROUGE-L scores of EACS
decreases as the length increases. It means that when the length of the
comments exceeds the common range, as the expected length of the generated
comment continues to increase, it will be more difficult to generate. Overall,
the results verify the robustness of our EACS.

###  5.3. Case Study

In this section, we provide case studies to understand the generated summaries
of EACS compared to the state-of-the-art SiT, and to demonstrate the
usefulness of our EACS.

####  5.3.1. Case Study on Java Code Summarization

In this section, we take the code snippet $c_{2}$ in Figure 8(a) as an example
and apply SiT and our EACS to generate summaries for comparison. It is a real-
world example from the test set of the JCSD dataset. As in Section 3, we
consider the comment of $c_{2}$ as the reference summary, as shown in the
first line of Figure 8(c). The summaries generated by SiT and EACS for $c_{2}$
are shown in the second and third lines, respectively. From the figure, we can
observe that compared to the reference summary, the summary generated by SiT
only covers a part of the second part (Red font). What is even worse is that
the first core word “remove” generated by SiT and that of “adds” in the
reference summary have completely opposite semantics, which makes the
semantics of the generated summary and the reference summary completely
opposite. Compared with the summary generated by SiT, the summary generated by
EACS covers all three parts of the reference summary. In addition, EACS
successfully generates the core word “add”, which we attribute to the
extractive-and-abstractive framework we proposed. This is demonstrated by the
important statement shown in Figure 8(b) selected by the extractor module in
our framework. Only the important statement contains the core word “add”.

Figure 8. Java Case

####  5.3.2. Case Study on Python Code Summarization

Figure 9. Python Case

Figure 9(a) shows a code snippet $c_{3}$ from the test set of the PCSD
dataset. $c_{3}$ is a long piece that consists of 51 lines of code. Figure
9(b) presents the important statements selected by our extractor from $c_{3}$.
Figure 9(c) contains three summaries, the first from the comment of $c_{3}$,
the second and last generated by SiT and our EACS, respectively. From the
figure, we can observe that compared to the reference summary, 1) the summary
generated by SiT misses many factual details, such as “from startobj to
endobj”; 2) although it looks a little different literally, the summary
generated by our EACS is able to cover all of the three parts and are
semantically equivalent. In summary, our EACS significantly outperforms SiT in
terms of naturalness and semantic completeness of the generated summaries.
Based on the above, we can conclude that our EACS is also a very competitive
technique on the Python code summarization task.

##  6\. Related Work

Code summarization has always been one of the hottest research topics in
software engineering. As mentioned earlier, we can categorize existing work
related to code summarization into extractive methods, abstractive methods,
and others.

Extractive methods. Most early (prior to 2016) code summarization techniques
(Haiduc et al., 2010a; Sridhara et al., 2010; Haiduc et al., 2010b; Moreno et
al., 2013) are extractive methods. Such methods work by extracting a subset of
the statements and keywords from the code, and then including information from
those statements and keywords in summary. For example, Sonia Haiduc et.al
(Haiduc et al., 2010a) first propose an extractive method to generate
extractive summaries for code snippets automatically. Extractive summaries are
obtained from the contents of a document by selecting the most important
information in that document. Extractive methods use text retrieval (TR)
techniques (e.g., Vector Space Model (Salton et al., 1975), Latent Semantic
Indexing (Deerwester et al., ence), and Hierarchical PAM (Mimno et al., 2007))
to determine the most important $n$ terms for each code snippet. Considering
that the quality of the summaries generated by extractive methods depends
heavily on the process of extracting the subset, Paige Rodeghero et.al
(Rodeghero et al., 2014) present an eye-tracking study of programmers and
propose a tool for selecting keywords based on the findings of the eye-
tracking study. The extractive methods rely on high-quality identifier names
and method signatures from the source code. These techniques may fail to
generate accurate comments if the source code contains poorly named
identifiers or method names (Wong et al., 2015).

Abstractive methods. Nowadays, DL-based (abstractive) code summarization
techniques have been proposed one after another. By combining Seq2Seq models
trained on large-scale code-comment datasets, abstractive methods can generate
words that do not appear in the given code snippet to overcome the limitations
of extractive methods. For example, Srinivasan Iyer et.al (Iyer et al., 2016)
present the first completely abstractive method for generating short, high-
level summaries of code snippets. Their experimental results demonstrate that
abstractive methods significantly outperform extractive methods in the
naturalness of generated summaries. In other words, the summaries generated by
abstractive methods are human-written-like. Code representation plays a key
role in abstractive methods. To produce semantic-preserving code embedding
representations, multiple aspects of the code snippet have been explored,
including tokens (Iyer et al., 2016; Fernandes et al., 2018; Hu et al., 2018b;
Wan et al., 2018; LeClair et al., 2019; Moore et al., 2019; Hu et al., 2020;
Wang et al., 2020; Zhang et al., 2020; Wei et al., 2020; LeClair et al., 2020;
Lin et al., 2021; Phan et al., 2021; Ahmad et al., 2020), abstract syntactic
trees (ASTs) (Alon et al., 2018; Hu et al., 2018a; Wan et al., 2018; Shido et
al., 2019; LeClair et al., 2019; Hu et al., 2020; Zhang et al., 2020; Wang et
al., 2020; Wei et al., 2020; LeClair et al., 2020; Yang et al., 2021; Lin et
al., 2021), control flows (Wang et al., 2020), code property graph (Liu et
al., 2020). In addition, existing abstractive methods have tried various
neural network architectures, such as LSTM (Iyer et al., 2016; Hu et al.,
2018a; Wang et al., 2020), Bidirectional-LSTM (Li et al., 2021; Zhang et al.,
2020; Wei et al., 2020), GRU (Hu et al., 2020; LeClair et al., 2019),
Transformer (Ahmad et al., 2020; Lin et al., 2021) and GNN (LeClair et al.,
2020; Liu et al., 2020). Although deep learning-based abstractive methods show
great potential for generating human-written-like summaries, we find that the
generated summaries often miss important factual details.

Others. Paul W. McBurney (McBurney and McMillan, 2016) takes the context of
code snippets into account when generating summaries. The context includes the
dependencies of the method and any other methods which rely on the output of
the method (Krinke, 2006). The works (Bansal et al., 2021; Zhang et al., 2021;
Wang et al., 2021a; Shahbazi et al., 2021) also use the information beyond the
code snippet itself, e.g., project or class context (Bansal et al., 2021;
Zhang et al., 2021; Wang et al., 2021a), application programming interface
documentations/knowledge (API Docs) (Shahbazi et al., 2021). These techniques
cannot be tested on existing commonly used datasets (e.g., JCSD, PCSD, and
CodeSearchNet Corpus) because methods (code snippets) in these datasets are
context-missing. There are techniques using summaries of similar code snippets
as the summary of the current snippet or as a starting point to generate a new
summary. For example, Edmund Wong et.al (Wong et al., 2015) apply code clone
detection techniques to discover similar code snippets and use the comments
from some code snippets to describe the other similar code snippets. The works
(Li et al., 2021; Zhang et al., 2020; Wei et al., 2020) retrieve similar code
snippets and then use the information contained in the similar code snippets
or their summaries to generate a summary of the current code snippet. They
rely on whether similar code snippets can be retrieved and how similar they
are. All the works mentioned above emphasize using external knowledge sources
(e.g., contexts, API Docs, and similar code snippets) to improve the quality
of the generated comments. We will explore the effect of combining our EACS
with the above external knowledge in future work.

Our method. EACS is a general code summarization framework that adopts
extractive and abstract schemes at the same time. Different from existing
extractive methods that use TR techniques to extract important
statements/words, the extractive module of EACS applies a deep learning-based
classifier to predict the importance of each statement. Unlike existing
abstractive methods, our abstracter receives and processes the entire code
snippet and important statements as input in parallel.

##  7\. Conclusion

In this paper, we propose an extractive-and-abstractive framework named EACS
for source code summarization. EACS consists of two modules, extractor and
abstracter. The extractor has the ability to extract important statements from
the code snippet. The important statements contain important factual details
that should be included in final generated summary. The abstracter takes in
the important statements extracted by the extractor and the entire code
snippet and is able to generate human-written-like summary in natural
language. We conduct comprehensive experiments on three databases to evaluate
the performance of EACS. And the experimental results demonstrate that our
EACS is an effective code summarization technique and significantly
outperforms the state-of-the-art.

## References

  * (1)
  * Abid et al. (2019) Nahla J. Abid, Jonathan I. Maletic, and Bonita Sharif. 2019\.  Using developer eye movements to externalize the mental model used in code summarization tasks. In _Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications_. ACM, Denver , CO, USA, 13:1–13:9.
  * Ahmad et al. (2020) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020.  A Transformer-based Approach for Source Code Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 4998–5007.
  * Allamanis et al. (2018) Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles Sutton. 2018.  A Survey of Machine Learning for Big Code and Naturalness.  _Comput. Surveys_ 51, 4 (2018), 81:1–81:37.
  * Alon et al. (2018) Uri Alon, Omer Levy, and Eran Yahav. 2018.  Code2seq: Generating Sequences from Structured Representations of Code.  _CoRR_ abs/1808.01400 (2018).
  * Anonymous (2022) Anonymous. 2022\.  EACS.  site: https://anonymous.4open.science/r/EACS.  Accessed March, 2022.
  * Arcuri and Briand (2014) Andrea Arcuri and Lionel Briand. 2014.  A Hitchhiker’s Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering.  _Software Testing, Verification and Reliability_ 24, 3 (2014), 219–250.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015\.  Neural Machine Translation by Jointly Learning to Align and Translate. In _Proceedings of the 3rd International Conference on Learning Representations_ _(ICLR 2015)_. OpenReview.net, San Diego, CA, USA, 1–15.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In _Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_. Association for Computational Linguistics, Ann Arbor, Michigan, USA, 65–72.
  * Bansal et al. (2021) Aakash Bansal, Sakib Haque, and Collin McMillan. 2021\.  Project-Level Encoding for Neural Source Code Summarization of Subroutines. In _Proceedings of the 29th International Conference on Program Comprehension_. IEEE, Madrid, Spain, 253–264.
  * Barone and Sennrich (2017) Antonio Valerio Miceli Barone and Rico Sennrich. 2017.  A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation. In _Proceedings of the 8th International Joint Conference on Natural Language Processing_. Asian Federation of Natural Language Processing, Taipei, Taiwan, 314–319.
  * Chen et al. (2021) Qiuyuan Chen, Xin Xia, Han Hu, David Lo, and Shanping Li. 2021.  Why My Code Summarization Model Does Not Work: Code Comment Improvement with Category Prediction.  _ACM Transactions on Software Engineering and Methodology_ 30, 2 (2021), 25:1–25:29.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In _Proceedings of the 8th Workshop on Syntax, Semantics and Structure in Statistical Translation_. Association for Computational Linguistics, Doha, Qatar, 103–111.
  * de Souza et al. (2005) Sergio Cozzetti B. de Souza, Nicolas Anquetil, and Káthia Marçal de Oliveira. 2005\.  A study of the documentation essential to software maintenance. In _Proceedings of the 23rd Annual International Conference on Design of Communication: documenting & Designing for Pervasive Information_. ACM, Coventry, UK, 68–75.
  * Deerwester et al. (ence) Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. Journal of the American Society for Information Science.  Indexing by Latent Semantic Analysis.  _1990_ 41, 6 (Journal of the American Society for Information Science), 391–407.
  * Eddy et al. (2013) Brian P. Eddy, Jeffrey A. Robinson, Nicholas A. Kraft, and Jeffrey C. Carver. 2013.  Evaluating source code summarization techniques: Replication and expansion. In _Proceedings of the 21st International Conference on Program Comprehension_. IEEE Computer Society, San Francisco, CA, USA, 13–22.
  * El-Kassas et al. (2021) Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021.  Automatic text summarization: A comprehensive survey.  _Expert Systems with Applications_ 165 (2021), 113679.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In _Proceedings of the 25th Conference on Empirical Methods in Natural Language Processing: Findings_. Association for Computational Linguistics, Online Event, 1536–1547.
  * Fernandes et al. (2018) Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018.  Structured Neural Summarization.  _CoRR_ abs/1811.01824 (2018).
  * Fowkes et al. (2017) Jaroslav M. Fowkes, Pankajan Chanthirasegaran, Razvan Ranca, Miltiadis Allamanis, Mirella Lapata, and Charles Sutton. 2017.  Autofolding for Source Code Summarization.  _IEEE Transactions on software Engineering_ 43, 12 (2017), 1095–1109.
  * Gambhir and Gupta (2017) Mahak Gambhir and Vishal Gupta. 2017.  Recent automatic text summarization techniques: a survey.  _Artificial Intelligence Review_ 47, 1 (2017), 1–66.
  * Gligoric et al. (2015) Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015\.  Practical regression test selection with dynamic file dependencies. In _Proceedings of the 24th International Symposium on Software Testing and Analysis_. ACM, Baltimore, MD, USA, 211–222.
  * Gong et al. (2022) Zi Gong, Cuiyun Gao, Yasheng Wang, Wenchao Gu, Yun Peng, and Zenglin Xu. 2022\.  Source Code Summarization with Structural Relative Position Guided Transformer.  _CoRR_ abs/2202.06521 (2022).
  * Gros et al. (2020) David Gros, Hariharan Sezhiyan, Prem Devanbu, and Zhou Yu. 2020\.  Code to Comment ”Translation”: Data, Metrics, Baselining & Evaluation. In _Proceedings of the 35th International Conference on Automated Software Engineering_. IEEE, Melbourne, Australia, 746–757.
  * Haiduc et al. (2010a) Sonia Haiduc, Jairo Aponte, and Andrian Marcus. 2010a.  Supporting program comprehension with source code summarization. In _Proceedings of the 32nd International Conference on Software Engineering_. ACM, Cape Town, South Africa, 223–226.
  * Haiduc et al. (2010b) Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Marcus. 2010b.  On the Use of Automated Text Summarization Techniques for Summarizing Source Code. In _Proceedings of the 17th Working Conference on Reverse Engineering_. IEEE Computer Society, Beverly, MA, USA, 35–44.
  * Haque et al. (2021) Sakib Haque, Aakash Bansal, Lingfei Wu, and Collin McMillan. 2021.  Action Word Prediction for Neural Source Code Summarization. In _Proceedings of the 28th International Conference on Software Analysis, Evolution and Reengineering_. IEEE, Honolulu, HI, USA, 330–341.
  * Hartzman and Austin (1993) Carl S. Hartzman and Charles F. Austin. 1993.  Maintenance productivity: observations based on an experience in a large system environment. In _Proceedings of the 3rd Conference of the Centre for Advanced Studies on Collaborative Research_. IBM, Toronto, Ontario, Canada, 138–170.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.
  * Hsu et al. (2018) Wan Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018\.  A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Melbourne, Australia, 132–141.
  * Hu et al. (2018a) Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018a.  Deep code comment generation. In _Proceedings of the 26th International Conference on Program Comprehension_. ACM, Gothenburg, Sweden, 200–210.
  * Hu et al. (2020) Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2020.  Deep code comment generation with hybrid lexical and syntactical information.  _Empirical Software Engineering_ 25, 3 (2020), 2179–2217.
  * Hu et al. (2018b) Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018b.  Summarizing Source Code with Transferred API Knowledge. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_. ijcai.org, Stockholm, Sweden, 2269–2275.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.  _CoRR_ abs/1909.09436 (2019).
  * Iyer et al. (2016) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016.  Summarizing Source Code using a Neural Attention Model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. The Association for Computer Linguistics, Berlin, Germany, 2073–2083.
  * Jayavardhan Reddy et al. (2018) Peddamail Jayavardhan Reddy, Yao Ziyu, Wang Zhen, and Sun Huan. 2018.  A comprehensive study of staqc for deep code summarization. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ _(KDD 2018)_. ACM, London, UK, 1–8.
  * Kajko-Mattsson (2005) Mira Kajko-Mattsson. 2005\.  A Survey of Documentation Practice within Corrective Maintenance.  _Empirical Software Engineering_ 10, 1 (2005), 31–55.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _Proceedings of the 3th International Conference on Learning Representations – Poster_. OpenReview.net, San Diego, CA, USA, 1–15.
  * Krinke (2006) Jens Krinke. 2006\.  Effects of context on program slicing.  _Journal of Systems and Software_ 79, 9 (2006), 1249–1260.
  * LeClair et al. (2020) Alexander LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. 2020.  Improved Code Summarization via a Graph Neural Network. In _Proceedings of the 28th International Conference on Program Comprehension_. ACM, Seoul, Republic of Korea, 184–195.
  * LeClair et al. (2019) Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019\.  A neural model for generating natural language summaries of program subroutines. In _Proceedings of the 41st International Conference on Software Engineering_. IEEE / ACM, Montreal, QC, Canada, 795–806.
  * LeClair and McMillan (2019) Alexander LeClair and Collin McMillan. 2019.  Recommendations for Datasets for Source Code Summarization. In _Proceedings of the 23th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Minneapolis, MN, USA, 3931–3937.
  * Li et al. (2021) Jia Li, Yongmin Li, Ge Li, Xing Hu, Xin Xia, and Zhi Jin. 2021.  EditSum: A Retrieve-and-Edit Framework for Source Code Summarization. In _Proceedings of the 36th International Conference on Automated Software Engineering_. IEEE, Melbourne, Australia, 155–166.
  * Lin et al. (2021) Chen Lin, Zhichao Ouyang, Junqing Zhuang, Jianqiang Chen, Hui Li, and Rongxin Wu. 2021\.  Improving Code Summarization with Block-wise Abstract Syntax Tree Splitting. In _Proceedings of the 29th International Conference on Program Comprehension_. IEEE, Madrid, Spain, 184–195.
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A package for automatic evaluation of summaries. In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics – workshop on Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.
  * Liu et al. (2020) Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2020.  Automatic Code Summarization via Multi-dimensional Semantic Fusing in GNN.  _CoRR_ abs/2006.05405 (2020).
  * Liu (2019) Yang Liu. 2019.  Fine-tune BERT for Extractive Summarization.  _CoRR_ abs/1903.10318 (2019).
  * Martin (2009) Robert C Martin. 2009\.  _Clean code: a handbook of agile software craftsmanship_.  Pearson Education.
  * Maybury (1999) Mani Maybury. 1999\.  _Advances in automatic text summarization_.  MIT press.
  * McBurney and McMillan (2016) Paul W. McBurney and Collin McMillan. 2016.  Automatic Source Code Summarization of Context for Java Methods.  _IEEE Transactions on Software Engineering_ 42, 2 (2016), 103–119.
  * Microsoft (2021) Microsoft. 2021\.  CodeBert.  site: https://github.com/microsoft/CodeBERT.  Accessed March, 2022.
  * Mimno et al. (2007) David M. Mimno, Wei Li, and Andrew McCallum. 2007.  Mixtures of hierarchical topics with Pachinko allocation. In _Proceedings of the 24th International Conference Machine Learning_. ACM, Corvallis, Oregon, USA, 633–640.
  * Moore et al. (2019) Jessica Moore, Ben Gelman, and David Slater. 2019\.  A Convolutional Neural Network for Language-Agnostic Source Code Summarization. In _Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering_. SciTePress, Heraklion, Crete, Greece, 15–26.
  * Moreno et al. (2013) Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori L. Pollock, and K. Vijay-Shanker. 2013.  Automatic generation of natural language summaries for Java classes. In _Proceedings of the 21st International Conference on Program Comprehension_. IEEE Computer Society, San Francisco, CA, USA, 23–32.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a Method for Automatic Evaluation of Machine Translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_. ACL, Philadelphia, PA, USA, 311–318.
  * Phan et al. (2021) Long N. Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James T. Anibal, Alec Peltekian, and Yanfang Ye. 2021.  CoTexT: Multi-task Learning with Code-Text Transformer.  _CoRR_ abs/2105.08645 (2021).
  * Rodeghero et al. (2014) Paige Rodeghero, Collin McMillan, Paul W. McBurney, Nigel Bosch, and Sidney K. D’Mello. 2014.  Improving automated source code summarization via an eye-tracking study of programmers. In _Proceedings of the 36th International Conference on Software Engineering_. ACM, Hyderabad, India, 390–401.
  * Roy et al. (2021) Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021\.  Reassessing automatic evaluation metrics for code summarization tasks. In _Proceedings of the 29th Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, Athens, Greece, 1105–1116.
  * Salesforce (2021) Salesforce. 2021\.  CodeT5.  site: https://github.com/salesforce/CodeT5.  Accessed March, 2022.
  * Salton et al. (1975) Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975\.  A Vector Space Model for Automatic Indexing.  _Commun. ACM_ 18, 11 (1975), 613–620.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017\.  Get To The Point: Summarization with Pointer-Generator Networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Vancouver, Canada, 1073–1083.
  * Shahbazi et al. (2021) Ramin Shahbazi, Rishab Sharma, and Fatemeh H. Fard. 2021\.  API2Com: On the Improvement of Automatically Generated Code Comments Using API Documentations. In _Proceedings of the 29th International Conference on Program Comprehension_. IEEE, Madrid, Spain, 411–421.
  * Shi et al. (2021b) Ensheng Shi, Yanlin Wang, Lun Du, Junjie Chen, Shi Han, Hongyu Zhang, Dongmei Zhang, and Hongbin Sun. 2021b.  Neural Code Summarization: How Far Are We?. In _CoRR_ , Vol. abs/2107.07112.
  * Shi et al. (2021a) Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, and Chandan K. Reddy. 2021a.  Neural Abstractive Text Summarization with Sequence-to-Sequence Models.  _Transactions on Data Science_ 2, 1 (2021), 1:1–1:37.
  * Shido et al. (2019) Yusuke Shido, Yasuaki Kobayashi, Akihiro Yamamoto, Atsushi Miyamoto, and Tadayuki Matsumura. 2019\.  Automatic Source Code Summarization with Extended Tree-LSTM. In _Proceedings of the 18th International Joint Conference on Neural Networks_. IEEE, Hungary, 1–8.
  * Sridhara et al. (2010) Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori L. Pollock, and K. Vijay-Shanker. 2010.  Towards automatically generating summary comments for Java methods. In _Proceedings of the 25th International Conference on Automated Software Engineering_. ACM, Antwerp, Belgium, 43–52.
  * Stapleton et al. (2020) Sean Stapleton, Yashmeet Gambhir, Alexander LeClair, Zachary Eberhart, Westley Weimer, Kevin Leach, and Yu Huang. 2020.  A Human Study of Comprehension and Code Summarization. In _Proceedings of the 28th International Conference on Program Comprehension_. ACM, Seoul, Republic of Korea, 2–13.
  * Steinberger and Jezek (2009) Josef Steinberger and Karel Jezek. 2009.  Update Summarization Based on Latent Semantic Analysis. In _Proceedings of the 12th International Conference on Text, Speech and Dialogue_. Springer, Pilsen, Czech Republic, 77–84.
  * Tenny (1988) Ted Tenny. 1988\.  Program Readability: Procedures Versus Comments.  _IEEE Transactions on Software Engineering_ 14, 9 (1988), 1271–1279.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All You Need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_. Curran Associates Inc., Long Beach, CA, USA, 5998–6008.
  * Wan et al. (2018) Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018.  Improving automatic source code summarization via deep reinforcement learning. In _Proceedings of the 33rd International Conference on Automated Software Engineering_. ACM/IEEE, Montpellier, France, 397–407.
  * Wang et al. (2020) Wenhua Wang, Yuqun Zhang, Yulei Sui, Yao Wan, Zhou Zhao, Jian Wu, Philip Yu, and Guandong Xu. 2020.  Reinforcement-Learning-Guided Source Code Summarization using Hierarchical Attention.  _IEEE Transactions on Software Engineering (Early Access)_ (2020), 1–19.
  * Wang et al. (2021a) Yanlin Wang, Ensheng Shi, Lun Du, Xiaodi Yang, Yuxuan Hu, Shi Han, Hongyu Zhang, and Dongmei Zhang. 2021a.  CoCoSum: Contextual Code Summarization with Multi-Relational Graph Neural Network.  _CoRR_ abs/2107.01933 (2021), 1–24.
  * Wang et al. (2021b) Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021b.  CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In _Proceedings of the 26th Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Virtual Event / Punta Cana, Dominican Republic, 8696–8708.
  * Wei et al. (2019) Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019.  Code Generation as a Dual Task of Code Summarization. In _Proceedings of the 33rd Annual Conference on Neural Informatiom Processing Systems_ _(NeurIPS 2019)_. Vancouver, BC, Canada, 6559–6569.
  * Wei et al. (2020) Bolin Wei, Yongmin Li, Ge Li, Xin Xia, and Zhi Jin. 2020.  Retrieve and Refine: Exemplar-based Neural Comment Generation. In _Proceedings of the 35th International Conference on Automated Software Engineering_. IEEE, Melbourne, Australia, 349–360.
  * Wong et al. (2015) Edmund Wong, Taiyue Liu, and Lin Tan. 2015.  CloCom: Mining existing source code for automatic comment generation. In _Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering_. IEEE Computer Society, Montreal, QC, Canada, 380–389.
  * Woodfield et al. (1981) Scott N. Woodfield, Hubert E. Dunsmore, and Vincent Yun Shen. 1981.  The Effect of Modularization and Comments on Program Comprehension. In _Proceedings of the 5th International Conference on Software Engineering_. IEEE Computer Society, San Diego, California, USA, 215–223.
  * Wu et al. (2021) Hongqiu Wu, Hai Zhao, and Min Zhang. 2021.  Code Summarization with Structure-induced Transformer. In _Proceedings of the Findings of the 59th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online Event, 1078–1090.
  * Wu and Hu (2018) Yuxiang Wu and Baotian Hu. 2018.  Learning to Extract Coherent Summary via Deep Reinforcement Learning. In _Proceedings of the 32nd AAAI Conference on Artificial Intelligence_. AAAI Press, New Orleans, Louisiana, USA, 5602–5609.
  * Xia et al. (2018) Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shanping Li. 2018\.  Measuring Program Comprehension: A Large-Scale Field Study with Professionals.  _IEEE Transactions on Software Engineering_ 44, 10 (2018), 951–976.
  * Yang et al. (2021) Zhen Yang, Jacky Keung, Xiao Yu, Xiaodong Gu, Zhengyuan Wei, Xiaoxue Ma, and Miao Zhang. 2021.  A Multi-Modal Transformer-based Code Summarization Approach for Smart Contracts. In _Proceedings of the 29th International Conference on Program Comprehension_. IEEE, Madrid, Spain, 1–12.
  * Zhai et al. (2020) Juan Zhai, Xiangzhe Xu, Yu Shi, Guanhong Tao, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. 2020\.  CPC: automatically classifying and propagating natural language comments via program analysis. In _Proceedings of the 42nd International Conference on Software Engineering_. ACM, Seoul, South Korea, 1359–1371.
  * Zhang et al. (2021) Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Raymond J. Mooney, Junyi Jessy Li, and Milos Gligoric. 2021.  Learning to Generate Code Comments from Class Hierarchies.  _CoRR_ abs/2103.13426 (2021).
  * Zhang et al. (2020) Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020.  Retrieval-based neural source code summarization. In _Proceedings of the 42nd International Conference on Software Engineering_. ACM, Seoul, South Korea, 1385–1397.
  * Zhou et al. (2018) Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018\.  Neural Document Summarization by Jointly Learning to Score and Select Sentences. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Melbourne, Australia, 654–663.
  * Zhou et al. (2021) Yu Zhou, Xiaoqing Zhang, Juanjuan Shen, Tingting Han, Taolue Chen, and Harald C. Gall. 2021\.  Adversarial Robustness of Deep Code Comment Generation.  _ACM Transactions on Software Engineering and Methodology_ 1, 1 (2021), 1–30.
  * Zhu and Pan (2019) Yuxiang Zhu and Minxue Pan. 2019.  Automatic Code Summarization: A Systematic Literature Review.  _CoRR_ abs/1909.04352 (2019).