# LAMNER: Code Comment Generation Using Character Language Model and Named
Entity Recognition

Rishab Sharma<EMAIL_ADDRESS>,  Fuxiang Chen
<EMAIL_ADDRESS>and  Fatemeh H. Fard<EMAIL_ADDRESS>The University
of British Columbia1177 Research RoadKelownaCanada

(2021)

###### Abstract.

Code comment generation is the task of generating a high-level natural
language description for a given code method/function. This is important as
the comments help the developers to understand and maintain the software.
Previous study has reported that code comments are mostly missing or outdated
in many methods/functions. Although researchers have been studying multiple
ways to generate code comments automatically, previous work mainly consider
representing a code token in its entirety semantics form only (e.g., a
language model is used to learn the semantics of a code token), and additional
code properties such as the tree structure of a code are included as auxiliary
input to the model. There are two limitations: 1) Learning the code token in
its entirety form may not be able to capture information succinctly in source
code, e.g., when developers are using camel case identifiers. 2) The code
token does not contain additional structural information which are inherently
important in programming languages. For example, in order for a piece of code
to be compiled and executed successfully, developers need to write the code
that conform to certain structure.

In this paper, we present LAnguage Model and Named Entity Recognition
(LAMNER), a code comment generator which is capable of encoding identifiers
effectively and capturing the structural property of a code token. To encode a
code token, a character-level language model is used to learn the semantic
representation. For the structural property of a token, a Named Entity
Recognition model is trained to learn the code entities, which correspond to
the different types of code tokens – for example, a code token may be an
access modifier, an identifier name or some other code entities. These
representations are then fed into an encoder-decoder architecture to generate
code comments. We note here that a code token that is encoded by our encoder
will contain both semantic and syntactic information. We evaluate the
generated comments from LAMNER and other baselines against the ground truth on
a popular Java dataset with four commonly used metrics: BLEU, ROGUE-L, METEOR,
and CIDEr. In a separate setting, we perform code abstraction on the dataset
to evaluate the robustness of LAMNER. Our results showed that LAMNER is
effective and improves over the baselines in majority of the metrics.
Additionally, we fused LAMNER’s code representation with baseline models and
the fused models consistently showed improvement over the non-fused models.
The human evaluation further shows that LAMNER produces high-quality code
comments. We believe that LAMNER can benefit the community, and we will open-
source LAMNER.

automatic code comment generation, neural language model, named entity
recognition, code-token embedding

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Software and its engineering

##  1\. Introduction

Maintaining the source code documentation is an important software engineering
activity. It has been reported that software developers are spending more than
half of their time trying to understand the code during the software
maintenance cycle (retref). Although program comprehension is a main activity
for software developers (ppf), reading the code requires additional mental
effort as it is not a natural practice for humans (difnlpl). A well-commented
code aids in easier code comprehension and assists in the efficient
maintenance of software projects (sridhara). Despite the importance of well-
documented code, previous study has reported that only a small percentage of
the methods in software projects are commented (codernn). Moreover, comments
are mostly absent or outdated as the software evolves (evolve). As a result,
researchers have studied how to generate natural language code comments from a
given code method/function automatically (hu1; hu2; iyer; transformer). The
code comments aim to explain what the code is doing so that developers do not
have to inspect the code line by line to infer how it works. This often
happens when new developers join a new code repository, or when developers
have to revisit a code repository that has been inactive for some time
[[[[Fuxiang: To cite some papers]]]].

Since 2016, multiple studies have leveraged the deep learning based encoder-
decoder Neural Machine Translation models (NMT) for comment generation (iyer;
hu1; hu11; retref; maclatest; hu2; cocogum; rencos). The encoder in a general
NMT model takes a sequence of tokens from a language as input (e.g. English),
and the decoder generates the translation of the input sequence into another
language (e.g. German) (maclatest). The code comment generation problem can be
seen as a translation task between the code (programming language) and the
natural language text, which maps an input code snippet to a comment in
English as output (maclatest). However, programming languages and natural
languages have several dissimilar features (difnlpl; gros2020code).
Programming languages are repetitive, mainly due to their syntax and they can
have an infinite number of vocabulary based on the developers’ naming choices
for identifiers (dls; bigcode). Therefore, the NMT techniques used for the
translation of natural languages require specific techniques to handle the
differences in programming languages (dls).

Existing work studied multiple ways to generate code comments automatically.
Early neural models for code comment generation (iyer; hu1) used the Long
Short Term Memory (LSTM) based encoder-decoder architecture. It mainly
consider representing a code token in its entirety semantics form (e.g., a
language model is used to learn the semantics of a code token). Subsequent
works (hu1; clair; code2seq; ase) further improved the performance by
incorporating additional code properties such as Abstract Syntax Trees (AST)
to represent the syntactical structure of code. There are two limitations: 1)
Learning the code token in its entirety form may not be able to capture
information succinctly in source code, e.g., when developers are using camel
case identifiers. Writing camel case identifiers is a common practice for
developers to combine multiple elements together because the identifier names
cannot contain spaces (camelidentifier). For example, writing camel case
identifiers is the convention used in Java code. Many existing code comment
generation models do not distinguish between camel case and non-camel case
identifiers. 2) The embedding of the code token does not contain additional
structural information which are inherently important in programming
languages. For example, in order for a piece of code to be compiled and
executed successfully, developers need to write the code that conform to
certain structure [[[[Fuxiang: cite]]]]. A code token may be an access
modifier, an identifier name or some other code constructs. Code constructs
represents the syntactic/structural meaning of each token. The code constructs
are reported to be useful in code comprehension and are used in Software
Engineering tasks such as bug detection and program repair
(cates2021doesIdentifier; newman2020generationIdentifier;
allamanis2021Identifier; ir3). We note that AST, which provides a tree
structure for code, is another way to represent structural information for
code. Unlike code constructs, AST cannot encode token-level information.

In this paper, we present LAnguage Model and Named Entity Recognition
(LAMNER), a code comment generator which is capable of encoding identifiers
effectively and capturing the structural property of a code token. To encode a
code token that includes a type of identifier, e.g., camel case identifier,
snake case identifier, etc, a character-level language model is used to learn
the semantic representation. For the structural property of a token, a Named
Entity Recognition model is trained to learn the code constructs. These
representations are then fed into a encoder-decoder architecture to generate
code comments. We note here that a code token that is encoded by our encoder
will contain both semantic and syntactical information. We evaluate the
generated comments from LAMNER and other baselines against the ground truth on
a popular Java dataset with four commonly used metrics: BLEU, ROGUE-L, METEOR,
and CIDEr. In a separate setting, we perform code abstraction on the dataset
to evaluate the robustness of LAMNER. The experiment reveals the primary
advantage of the proposed work, as it demonstrates significantly better
performance as compared to baselines in a challenge setting, which shows the
robustness of the model and shows the model is able to grasp the general
understanding of source code. Our results showed that LAMNER is effective and
improves over the baselines in majority of the metrics. Additionally, we fused
LAMNER’s code representation with baseline models and the fused models
consistently showed improvement over the non-fused models. We also conducted a
qualitative study to evaluate the generated comments from LAMNER. The
qualitative results show that the comments generated by LAMNER describe the
functionality of the given method correctly, are grammatically correct, and
are highly readable by human.

Article contributions: The contributions of our work are as follows.

  * •

A novel code comment generation model, LAMNER, that encodes both the semantic
and syntactical information of a code token. We propose LAMNER that leverages
the bidirectional character-level language model and NER model for learning
the semantics and the syntactical information of code tokens, respectively.

  * •

Fusing and evaluation of the embeddings learned from LAMNER with baseline
models. We show the adaptability of LAMNER by combining the embeddings learned
from LAMNER with baseline models.

  * •

LAMNER with Abstraction of Identifiers.  The identifiers are changed to
standardized names, and LAMNER, together with baseline models are evaluated
with these abstracted identifiers.

  * •

We open source our code and models for reproducibility
111https://github.com/rishab-32/lamner-code.git.

(a) Random initialization

(b) Random initialization with external knowledge

Figure 1. Processing code sequence vs. incorporating external knowledge for
code comment generation.

Article Structure: The rest of this paper is structured as follows. The
required background and our approach are described in Sections 2 and 3,
respectively. We explain the experimental setup in Section 4 and results are
presented in Section 5. Section 6 is dedicated to discussions followed by
human evaluations in Section 7. We point out the threats to validity in
Section 8 and review the related works in Section 9. Finally, we conclude the
paper in Section 10.

(a) Initializing tokens with pre-trained embedding

(b) Integration of our work with other approaches

Figure 2. Initializing the NMT model with pre-trained LAM and NER embedding in
our work

##  2\. Background

In this section, we describe the background understanding for our approach.

###  2.1. Language Model

Language modeling is the task of predicting the next token by assigning the
conditional probability to sequences of tokens (lm1). When the predicted token
is a word, it is referred to as a word-level language model. Similarly, in
character-level language model, the predicted token is a character (flair).
For example, for the input code sequence ”`public Boo`”, the model can predict
”`l`” as the next character, followed by other characters and finally predict
”`Boolean`” as a word. In natural language processing, language models are
used in the generation of contextual embedding (flair; bert), which is the
vector representation of the text. Recently, language models have been
leveraged for software engineering tasks such as bug prediction (scelmo), code
retrieval and document generation (codebert), and classification and repair
tasks (cubert). Recurrent Neural Network (RNN) (rnn1) forms the fundamental
architecture of the neural language model. Figure 3 represents a bidirectional
character-level language model that we use in our approach. The details are
described in Section 3.

Figure 3. Overview of the bidirectional character-level language model
architecture. The input to the model is every character within each line of
code, and the model is trained to predict the next character.

###  2.2. Named Entity Recognition

Named Entity Recognition is the task of recognizing and tagging entities to
their corresponding types (flair). Contemporary NER models use neural network
architecture based on Long Short Term Memory (LSTM) (lstm) and Conditional
Random Fields (CRF) (mahovy). In natural language, entities can be names of
people or places (sner). In the code domain, these entities can be objects,
methods, classes, conditional statements, loops, and data-types (sner). The
details of our proposed NER model is described in Section 3.3.

###  2.3. Long Short Term Memory

Recurrent Neural Networks (rnn1) are networks that predict the output based on
the previously learned information. They are used in sequential tasks like
language modeling and neural machine translation (cho; rnn1; s2s). RNN
utilizes previous sequential knowledge and the current input to generate the
current output. Long Short Term Memory (LSTM) is a RNN designed to capture the
context in long sequences of text. LSTM units use forget, input, and output
gates to calculate the output and hidden states (lstm). These gates will learn
which part of the sequence data is important to keep or discard in the next
LSTM units. LSTM also introduces the cell state $c_{t}$, which is an extra
memory to carry the context of the sequence. The formula for the cell state,
$c_{t}$, is the following:

(1) $c_{t}=G_{forget}.c_{t-1}+G_{input}.c_{candidate}$

where $c_{candidate}$ is the cell state which should have been passed to the
next state, if $G_{input}$ gate were not introduced. Forget gate $G_{forget}$
modulates what parts of the previous cell state ($c_{t-1}$) will be kept.
Input gate $G_{input}$ decides whether to change or pass the value of
$c_{t-1}$ to the next LSTM state. $G_{input}$ and $G_{forget}$ decide which
cell state value to choose. Finally, the current cell state $c_{t}$ and output
gate $G_{output}$ are used to compute the current hidden state value $h_{t}$:

(2) $h_{t}=G_{output}.tanh(c_{t})$

###  2.4. Gated Recurrent Unit

Gated Recurrent Unit (GRU) (cho) is a more recent generation of RNNs. GRU only
uses the hidden state to transfer information, without using cell state. GRU
introduces two new gates: update gate and reset gate (cho). By using these
gates, GRU decides whether to update the weight of the previous hidden state
$h_{t-1}$ at the current GRU unit or pass it as it is to the next GRU unit.
This decision is made by the update gate $G_{update}$. The reset gate
$G_{reset}$ in GRU is used to modulate the contribution of the previous hidden
state $h_{t-1}$ to the current hidden state $h_{t}$. Based on the results, if
$G_{update}$ was not introduced, a candidate value $h_{candidate}$ of the
current hidden state will be used. Finally, the hidden state $h_{t}$ at
current time step is calculated as follows:

(3) $h_{t}=(1-G_{update}).h_{t-1}+G_{update}.h_{candidate}$

###  2.5. Neural Machine Translation

Neural Machine Translation is the task of converting a sequence of text in the
source language to text in the target language which is performed by a machine
(maclatest). The recent advances in neural networks achieve state of the art
results for NMT. Sequence to Sequence model introduced by Bahdanau et al.
(rnn1) is an encoder-decoder framework that jointly learns and aligns the
translation. Both the encoder and decoder use either LSTM or GRU as their
fundamental architecture (s2s; cho; rnn1). The input to the encoder is the
encoding of the source language tokens and the decoder maps the source
sequence into the target language sequence by producing a single token at each
time step. The attention mechanism (rnn1) is applied in the decoder to attend
towards relevant tokens from the encoder when generating the output. This
produces higher coherent predictions. Code comment generation is similar to
the NMT task, which maps the given code sequence to its comment in natural
language (maclatest). We use this architecture in LAMNER.

###  2.6. Learning Semantics of Code Sequences using Character-Level Language
Model

Akbik et al. (flair) show that their proposed character-level contextual
embedding is useful in several classification tasks, including NER in natural
languages. It can capture specific features of the text, such as
differentiating between lower and upper-case letters. These are especially
important in source code representation, where each character plays an
important role in defining the semantic and syntactic structure of a code
token. We follow the approach of Akbik et al. (flair) to learn the semantics
of a code token. We note here that our aim is not to generate character
embeddings, but to use a character-level embeddings model to generate a code
token.

####  2.6.1. Language Model Architecture

Figure 3 illustrates the architecture of the language model, which employs a
single layer bidirectional LSTM (Bi-LSTM). Bi-LSTM captures the left and right
context of a character in the sequence (bi-crf) using a forward and a backward
language model as shown in Figure 3. The input to each LSTM unit is an
embedding of a character which is initialized randomly. Each LSTM unit
processes the embedding to generate the output and the next hidden state for
the character. The output of the last LSTM unit is used to select the next
output character with maximum likelihood. In Figure 3, the model predicts the
character `l` for the given sequence of `public Boo`, which makes last letter
of `public Bool` sequence.

####  2.6.2. Extracting Semantic Embedding of Tokens from Language Model

The language model explained so far learns the character embeddings. In our
work, we want to extract the embedding of each code token. We follow the
technique proposed by Akbik et al.(flair) to extract the semantic embedding of
each token. The hidden state of each character maintains the contextual
information of its previous characters. In the left to right model (forward
model), the last character of a token encapsulates the information of all
characters of the token based on its left context. Similarly, in the right to
left model (backward model), the first character of the token contains the
information of the previous token from its right context. To get the
embeddings of a token, we concatenate these two embeddings. An example is
illustrated in Figure 4 where the embeddings of the token ‘getBoolean2’ is
being extracted.

This embedding is the concatenation of the output hidden state after the last
character in the word token (getBoolean2) in the forward language model, and
the output hidden state before the word’s first character from the backward
language model. This way, we have the information encoded from both left and
right contexts.

Figure 4. Extracting semantic code embedding

##  3\. Proposed Approach

###  3.1. Overview

Figure 5 illustrates the overview of our proposed approach, LAMNER. First, a
bidirectional character-based language model and a NER model (left box in
figure 5) are trained separately on a code corpus to generate the input code
embeddings for our Semantic-Syntax encoder-decoder architecture (right box in
figure 5). Specifically, for each token, the embeddings learned from the
language and the NER models are concatenated together. We call this the
Semantic-Syntax embedding. The input to the Semantic-Syntax encoder is the
Semantic-Syntax embedding. The decoder then uses the attention mechanism to
decode the input code snippet into code comments. The details for the
character-based language model, the NER model, and the Semantic-Syntax
encoder-decoder are described in Section 3.2, 3.3 and 3.4, respectively.

Figure 5. Overview of LAMNER-Code Framework. The Language and NER models on
the left box are pre-trained to provide an initial embeddings for the code
tokens. The embeddings are then concatenated to serve as the input to the
Semantic-Syntax encoder on the right box to generate comments.

###  3.2. Learning Semantics Representation of Code Sequences

We follow the approach of Akbik et al. (flair) to train a language model to
learn the semantic representation of a code token. It is reported that this
approach is able to capture multiple words within a token better. The details
are described in Section 2.6.

###  3.3. Learning Syntactical Structure of Code Sequences

Encoding the syntactical structure of code in models has shown to improve
performance in previous work (hu1; tree2seq; code2vec; code2seq). In this
section, we present our approach to generate the syntactical context of a code
token. The difference between the syntactical code structure in the previous
work and ours is that the the previous work do not consider the syntactical
properties within a code token itself, but we do. For example, a syntactic
code structure – AST, does not have a direct one-to-one mapping with the code
tokens. A code token’s syntactical structure is inherently important in
programming languages, i.e., to compile and execute a piece of code
successfully, developers need to write the code to conform certain syntactical
structure.

We employ a NER model, shown in figure 6, to generate a code token’s
syntactical embeddings. It takes the contextual embedding of the input code
token from the character language model, and uses a bidirectional LSTM
sequence tagging with a conditional random field decoding layer (bi-crf) to
map each input code token to its syntactic entity type. For example, using the
NER model for the given code sequence public Boolean getBoolean2 …, the token
`public` is labeled as modifier, the token `Boolean` is labeled as return
type, and the token `getBoolean2` is labeled as function.

Figure 6. Extracting syntactic code embedding

It has been reported that the embeddings learned from the character based
language model has better performance (flair) than the classical word
embeddings such as FastText (fastext) and Glove (glove), when used as the
input to a NER model. Thus, the input to our NER model is the semantic
representation learned from the trained character language model described
previously in Section 2.6. Within the NER model, the input embeddings are
fine-tuned using a bidirectional LSTM, before the synactic types are predicted
in the CRF-layer. In the decoding phase, the CRF layer uses the knowledge from
the LSTM output states to predict the correct entity type. It is important to
note that we do not use the predicted entity types of the NER in LAMNER.
Rather, the extracted syntactical representation of the code tokens learnt
from the NER model is used. After the NER model is trained, it is used to
extract the syntactic embedding for each code token.

Note that the scope of this work is not to detect these code entities. We
believe that parsers can detect the syntactic entities better. In comparison,
our work presents a new technique to generate context-sensitive syntactic
encoding of code tokens, which is not available through parsers.

###  3.4. Semantic-Syntax Model

The pre-trained embeddings from the language model and the NER model are
concatenated to represent the semantic-syntax embedding for code tokens in a
given code snippet. The Semantic-Syntax encoder takes this semantic-syntactic
embeddings of the code tokens as input and models them together to output the
semantic-syntax embeddings of the code sequence. The attentive decoder, which
uses this code sequence embeddings, is trained to generate the code comments
in natural language. The details of the Semantic-Syntax Encoder and the
Semantic-Syntax Decoder are in Section 3.4.1 and 3.4.2, respectively.

####  3.4.1. Semantic-Syntax Encoder

The Semantic-Syntax encoder processes the input using a single layer
bidirectional GRU. We use GRU as it is reported to have faster training time
(rnnfund), and yet preserve the information for long sequences (gruemp). We
denote $E_{t}$ as the semantic-syntax embedding of token $t$, which is the
concatenated pre-trained semantics and syntax embeddings from the language
model and the NER model. $E_{t}$ serves as the input to the GRU. Each GRU
state processes the semantic-syntax embedding for the current token, $E_{t}$,
in the code sequence and generates a hidden state $h_{t}$. The hidden state of
the last token in the sequence, $h_{last}$, contains the sequential
information of the complete sequence and is formed by concatenating the
internal hidden states, $h_{t_{left}}$ and $h_{t_{right}}$, from the left and
the right layer. The hidden state of the last token, $h_{last}$, is then fed
into a fully connected linear layer. The formal equation for the fully
connected layer, $y_{fc}$, is shown below:

(4) $y_{fc}=h_{last}*W_{t}+b$

where $W_{t}$ and $b$ are the weight matrix and bias values, respectively. The
output of the fully connected layer, $y_{fc}$, is then passed into a $tanh$
layer to form the final output $h_{final}$ of the Semantic-Syntax encoder. The
formal equation is shown below:

(5) $h_{final}=tanh(y_{fc})$

####  3.4.2. Semantic-Syntax Decoder

The Semantic-Syntax decoder implements the popular Bahdanau’s attention
mechanism on a unidirectional GRU, which was reported to have good performance
(rnn1). It uses the Semantic-Syntax encoder’s final hidden state to pay
attention to the input sequence’s important tokens. The attention mechanism
prevents the decoder from generating the natural language comments based on a
single final context vector. Instead, it calculates the attention weights for
each input token and pays attention to the tokens with larger attention values
during decoding. To predict a token, the decoder will lookup the semantic
embeddings of the comments learnt from training a character-level language
model, as described in Section 2.6. We note here that the semantic embeddings
of the comments are different from the semantic embeddings of the code – two
language models are learnt. The decoder will continue to predict the next
token until it reaches the maximum sequence length or the end of sentence
token $\langle$eos$\rangle$.

##  4\. Experimental Setup

In this section, we describe how to evaluate our approach for its
effectiveness, including the research questions, datasets used, training
details, and baselines models for comparison.

###  4.1. Research Questions

To evaluate our approach, we seek to address the following research questions
in our experiment:

RQ1: (Performance of LAMNER) How does our proposed model perform compared to
other baselines? In this research question, we evaluate the performance of
LAMNER with the baseline models using common and popular metrics for comment
generation: BLEU-n ($n\in[1,4]$), ROUGE-L, METEOR, and CIDEr.

RQ2: (Contribution of Semantic vs. Syntax Embedding in LAMNER) Which parts of
our model contribute more to the performance? Rather than using our proposed
semantics-syntax embeddings, here, we perform an ablation study to evaluate
our model using either the proposed semantic embedding or the syntactical
embedding.

RQ3: (Effect of Code and Comment Length Increase on the Model Performance) How
does our model perform as the code and comment length increases? An important
aspect of NMT encoder-decoder models is their ability to work well as the
length of the input or output increases. In this research question, we study
how LAMNER behaves compared to the baseline models as the length of the code
and comment increases.

RQ4: (Effect of Language Models and NER Embedding) What is the effect of using
a different language model in our approach for comment generation? We
experiment with another popular language model and use its embeddings instead
of our proposed semantics embeddings to train LAMNER. We compare this new
model with LAMNER trained using our proposed semantics embeddings.

RQ5: (Evaluation of Models When Tested on Abstracted Code) How well different
models perform / generalize, when tested on abstracted code, where the
identifier names are replaced with general names? For this research question,
we perform code abstraction on the dataset and experiment it with LAMNER and
the baseline models.

RQ6: (Effect of Combining AST and Information Retrieval Techniques and The Use
of Transformer Architecture) Can integrating existing techniques or replacing
the neural network architecture enhance the performance of our model? In this
research question, we investigate the effect of adding an information
retrieval technique and combining AST into our model. Additionally, instead of
using the encoder-decoder architecture, we integrate our proposed semantics-
syntax embeddings to a Transformer-based architecture. The Transformer-based
architecture is a commonly used neural architecture in natural language
processing [[[[Fuxiang: cite]]]].

###  4.2. Dataset

We use the widely used Java dataset for code comment generation collected from
popular GitHub repositories by Hu et al. (hu2). The dataset consists of two
parts, the method code, and its comment, which is the first sentence extracted
from the Javadoc. In total, there are 69,708 code and comment pairs. The
training, validation and testing set are split distinctly into the ratio 8 : 1
: 1. Following previously work (hu2), we set the maximum size for code tokens
and comment tokens to 300 and 30, respectively. Lengthy code and comment will
be truncated up to the maximum size. The statistics for the dataset is
provided in Table 1, under the ‘# Records in Dataset’ column.

Table 1. Dataset details for code comment generation Split |  # Records in Dataset
---|---
Train | 69,708
Validation | 8,714
Test | 8,714

###  4.3. Model Training

####  4.3.1. Language Models

In our approach, we train two separate language models: i) a language model
that learns the semantic representation of code tokens (Section 3.2), and ii)
another language model that encodes the natural language comments (Section
3.4.2). The underlying architecture for both models is the same as described
in section 2.6. For the language model that learns the semantic representation
of code tokens, it is trained on the code corpus of the train dataset. Over
here, we are not interested in the code comments as we only want to learn the
code representation. Thus, we exclude all comments (e.g., inline, block and
JavaDoc comments). We perform comments removal using the Javalang library as
it is reported to have good performance in a previous study (rencos). For the
language model that encodes the natural language comments, it is trained on
the code comments. This language model is used in the decoder. Both models
have dropout probability of 0.1 applied for regularisation purposes.

####  4.3.2. Named Entity Recognition Model

To train the NER model, we require a labelled dataset where each code token is
linked to its corresponding syntactic type. For example, a code token may be
associated with an access modifier type, an operator type, or other types. We
use the Javalang parser to obtain a labelled dataset from the train dataset.
The Javalang parser is used in previous studies (ner-pos; ase-ptm) and is
reported to have good accuracy in labeling the code tokens into their
associated types.

The Javalang parser labels some code tokens in a granularity fashion. For
example, it groups all types of identifiers (e.g., class name, function name,
etc) into a common label identifier, and all types of separators (e.g., end of
a method, end of line, etc) into another common label separator. In order to
learn the finer nuances of the token type, we modify some of the labels that
are generated from the Javalang parser. Specifically, we breakdown the
identifier and the separators types where the former is divided into five
subtypes: class, function, object, modifier, and return-type, and the latter
is divided into four subtypes: body-start-delimiter, body-end-delimiter, end-
of-line (eol), and other-separators. The breakdown of the identifiers are
performed as follow: To identify a class, it must conform to the JAVA coding
convention i.e., the identifier starts with an upper-case character and it
must not contain any braces as suffix. For function and object identifiers,
they must be suffixed by round braces. To distinguish an object from an
identifier, it must have a corresponding class identifier with exact naming
(case insensitive). If there is such a naming match, we categorise it as an
object, else, we categorise it as a function. The return-types are defined at
the start of the function definition and we identify them directly. Java has a
specific set of access modifiers i.e., static, public and private, and
identifiers that contain them will be labelled as modifier. For the
separators, the token “{” is used to indicate the start of a new body – we
labelled it as body-start-delimiter. Similarly, we labelled the token “}” as
body-end-delimiter to present the end of a body section. The “;” token is used
to indicate the end of a code line and we labelled it as end-of-line (eol).
The rest of the separators “, (, )” are labelled as other-separators. Finally,
the dataset contains 18 different categories, listed in Table 2.

We train the NER model on the train dataset and evaluate its performance on
the test dataset. The code entity types inferred by the Javalang parser are
considered the ground truth labels whereas the code entities generated by the
NER model are the predicted labels. We also evaluate the NER model on the
dataset where code abstraction is being performed. This is to evaluate the
robustness of the NER model for random token names. The same dataset that is
used for testing the NER model is abstracted, where the identifier name and
type are replaced by a more generalized name (e.g. “identifier” and “type”).
We follow the approach in previous studies for code abstraction (tosem-
refinement), and add an id to each general token (e.g. “identifier_id”). We
applied dropout with a probability of 0.5 for regularisation purposes.
[[[[Fuxiang: Why do we choose 0.5 here, instead of 0.1 for the language
models?]]]] For evaluating the NER model, we use the set of metrics :
Precision, Recall, and F1 score. These metrics are commonly used to evaluate
NER models in previous studies (flairlib). The formal equation for these
metrics are shown in the following:

(6) $Precision=\frac{TruePositive}{TruePositive+FalsePositive}$

(7) $Recall=\frac{TruePositive}{TruePositive+FalseNegative}$

(8) $F1=2*\frac{Precision*Recall}{Precision+Recall}$

Here, $TruePositive$ are the number of samples that are correctly labeled as
belonging to a group by the NER model whereas $FalsePositive$ are the number
of samples that are incorrectly labeled as belonging to a group.
$FalseNegative$ are the ones that the NER model incorrectly labelled as not
belonging to a group. The evaluation results for the NER model are shown in
Table 2. The first column shows the different type of code token in the test
dataset, and the last column shows the number of tokens for each type. Within
the Precision, Recall and F1 columns, it is separated into Original and
Abstracted sub-columns. The column Abstracted means that the dataset has been
preprocessed for code abstraction, whereas the column Original means that no
code abstraction is being performed on the dataset. The scores in the Original
column shows that the NER model can reasonably recognize the code entities
with a good accuracy. On average, the NER model is able to achieve 99.41%,
93.66% and 93.89% on the original dataset (no code abstraction) for precision,
recall and f1 score, respectively. For the code abstraction dataset, the NER
model receives a relatively high score of above 84% in all metrics. For some
categories, such as ‘string’ the NER scores are not as high, and for others
such as ‘function’, the prediction accuracy does not change much. Our
experiments showed that the NER model can predict the labels for many code
entities correctly, even when the identifier names are abstracted. As
mentioned previously, the goal of this work is not on improving the existing
language parser, but to generate the syntactical information of each code
token. Specifically, the embeddings learned in the NER model will be used to
represent the syntactical information of each code token. We note here that
the Precision, Recall, and F1 score are used only to evaluate the NER model,
which is the intermediate model used in LAMNER. Other metrics that are used
for code comment generation evaluation are described in Section 4.5.

To gain more insights into the (identifier) sub-types embeddings learned from
the NER model, we project the sub-types embeddings into a two-dimensional
plane using T-Distributed Stochastic Neighbor Embedding (T-SNE) (tsne) to see
if the obtained sub-types embeddings from the NER model can be used to
represent the syntactic type of the code tokens reliably. This is shown in
Figure 7. We observe that similar to path-based embeddings (code2seq), the
learned embeddings from the NER model form clusters (idbench) around similar
syntactic types. Here, each cluster contains tokens that belong to the same
code entity group. The NER embeddings is able to delineate the sub-types
(class name, method name, function name, object name) with a reasonably good
accuracy.

Figure 7. Clustering of syntactic tokens based on NER embeddings Table 2. Results of Named Entity Recognition model Entity | Precision | Recall | F1 | Number of Tokens
---|---|---|---|---
| Original | Abstracted | Original | Abstracted | Original | Abstracted |
modifier | 0.9989 | 0.9978 | 1.0000 | 0.9992 | 0.9995 | 0.9985 | 3,753
class | 0.9687 | 0.4720 | 0.9655 | 0.3011 | 0.9671 | 0.3677 | 11,586
function | 0.9762 | 0.9689 | 0.9806 | 0.9862 | 0.9784 | 0.9775 | 14,933
object | 0.9832 | 0.7818 | 0.9858 | 0.9961 | 0.9845 | 0.8761 | 39,260
return-type | 0.9995 | 0.9127 | 0.9977 | 0.9655 | 0.9986 | 0.9384 | 2,164
other-separators | 0.9997 | 0.9979 | 0.9996 | 0.9998 | 0.9997 | 0.9989 | 69,352
body-start-delimiter | 1.0000 | 0.9996 | 1.0000 | 0.9966 | 1.0000 | 0.9981 | 7,204
body-end-delimiter | 0.9997 | 1.0000 | 0.9999 | 0.9997 | 0.9998 | 0.9999 | 7,204
eol | 0.9998 | 0.9994 | 0.9996 | 0.9992 | 0.9997 | 0.9978 | 16,078
number | 0.9693 | 1.0000 | 0.9957 | 0.1842 | 0.9823 | 0.3112 | 4,857
string | 0.8590 | 0.2022 | 0.7525 | 0.0135 | 0.8022 | 0.0254 | 2,436
data-type | 0.9958 | 0.9922 | 0.9985 | 0.7440 | 0.9971 | 0.8504 | 7,114
boolean | 0.9989 | 0.4406 | 1.0000 | 1.0000 | 0.9995 | 0.6117 | 945
operator | 0.9982 | 0.9992 | 0.9999 | 1.0000 | 0.9991 | 0.9996 | 20,413
loop | 0.9947 | 0.9945 | 1.0000 | 1.0000 | 0.9973 | 0.9972 | 939
keyword | 0.9994 | 1.0000 | 0.9974 | 0.9884 | 0.9984 | 0.9945 | 13,865
conditional St. | 1.0000 | 0.9987 | 0.9998 | 0.9987 | 0.9999 | 0.9942 | 6,894
annotation | 1.0000 | 1.0000 | 1.0000 | 0.9808 | 1.0000 | 0.9903 | 209
Average | 0.9856 | 0.8753 | 0.9818 | 0.8419 | 0.9835 | 0.8296 | Total: 219,343

####  4.3.3. Semantic-Syntax Model

To train the NMT encoder-decoder model, we use the dataset described in
section 4.2. Similar to previous work (codernn; rencos), the numerical and
string values are replaced with the `NUM` and `STR` tokens respectively. The
hidden size of the Semantic-Syntax model is set to $512$ and it is trained for
$100$ epochs or until the learning rate decays to $1e-7$. Initial learning
rate is set to $0.1$ and batch size is $16$. Both encoder and decoder applied
a dropout probability of $0.5$. If the validation loss does not improve after
7 consecutive epochs, the learning rate is decayed by a factor of $0.1$. All
experiments are conducted on a NVIDIA Tesla V100 GPU with 32 GB memory.

###  4.4. Baselines

####  4.4.1. Other Approaches

We compare our approach with the following baseline models, which are commonly
used in many comment generation studies (retref; msr-context; rencos; hgnn).
The availability of the models is also another important factor in choosing
the baselines.

Seq2Seq uses a single encoder-decoder with attention mechanism (s2s). The
encoder takes a sequence of code tokens as input, and the decoder generates
the comments by performing attention on the code tokens.

Hybrid-DeepCom uses the Structure Based Traversal (SBT) of AST to represent
code (hu11). SBT representation is used inside an encoder-decoder model to
generate comments.

Code2Seq uses a specific technique to represent code snippets from AST. It
considers all pairwise paths between leaf nodes of the tree, representing them
as sequences of the terminal and non-terminal nodes. This path is concatenated
with the tokens of each leaf node and serves as input to the encoder-decoder
model (code2seq).

AST-Attend-GRU uses the code and the Structure Based Traversal (SBT)
representation of AST as input to two separate encoders. The input inside the
two encoders is processed and combined to generate the final output (clair).

TL-CodeSum uses a double encoder architecture to generate code comments (hu2).
The API knowledge is encoded into embeddings, which is transferred to the
model to produce comments.

RENCOS (rencos) combines the information retrieval techniques with NMT-based
models. It uses two syntactically and semantically similar code snippets as
input. The conditional probability from the two inputs is then fused to
generate the final output.

For all of these works, we use the official implementation available on the
authors’ GitHub repositories. For the Seq2Seq model, its official
implementation is from OpenNMT 222https://opennmt.net/ (open-nmt).

####  4.4.2. Variations of Our Model

We consider four variations of our model as described below.

LAM This model uses only the semantic embeddings learnt from the character-
level language model.

NER This model uses only the syntactic embedding learnt from the NER model.

LAMNERStatic In this model, the Semantic-Syntax embeddings are kept static and
are not further tuned during training. This model shows the performance of the
pretrained embeddings.

LAMNER This model uses the concatenated semantic and syntax embeddings as the
input to the Semantic-Syntax encoder, as shown in Figure 5. The purpose of
this model is to evaluate the effectiveness of combining the two embeddings
for code comment generation. Note that in LAMNER, the model is initialized
with the Semantic-Syntax embeddings and the embeddings are further tuned
during training.

###  4.5. Evaluation Metrics

Similar to previous works (ase; rencos), we evaluate the performance of our
model and the baseline models using the following metrics: BLEU (bleu),
ROUGE-L (rouge), METEOR (meteor), and CIDER (cider).

BLEU measures the n-gram ($n\in[1,4]$) geometric precision $p_{n}$ between the
generated comment (C) and the ground truth (G) (bleu). Brevity penalty, $BP$,
is used to penalize the shorter candidate sentences of length $c_{l}$ when
compared to the reference translations of length $r_{l}$ (bleu). BLEU is
calculated as follows:

(9) $BLEU=BP.exp\sum_{N=1}^{4}\frac{1}{N}\log p_{n},\hskip
10.0ptBP=\begin{cases}1&\text{if}\ c_{l}>r_{l}\\\ e^{\frac{1--
r_{l}}{c_{l}}},&\text{if}\ c_{l}\leqslant r_{l}\\\ \end{cases}$

ROUGE is a recall-based metric that computes the number of tokens from (G)
that appears in (C) (rouge). ROUGE-L finds the F-score of the longest common
sub-sequence (LCS) between the two sentences X and Y with length $m$ and $n$,
respectively (rouge). It is denoted as $F_{LCS}$ below.

(10)
$R_{LCS}=\frac{LCS(X,Y)}{m},P_{LCS}=\frac{LCS(X,Y)}{n},F_{LCS}=\frac{(1+\beta^{2})R_{LCS}P_{LCS}}{R_{LCS}+\beta^{2}P_{LCS}},\hskip
5.0pt\beta=\frac{P_{LCS}}{R_{LCS}}$

METEOR calculates the score using an alignment approach, in which a unigram in
a sentence is mapped to zero or one unigram in another sentence in a phased
manner (meteor). Here, $P$ and $R$ are the precision and recall values and
$\alpha$, $\beta$ and $\gamma$ have default values of $0.9$, $3.0$ and $0.5$,
respectively.

(11) $METEOR=(1-\gamma.frag^{\beta}).\frac{P.R}{\alpha.P+(1-\alpha).R}$

CIDEr rates the correctness of the comments (rencos; cocogum). It performs
Term Frequency-Inverse Document Frequency (TF-IDF) weighting for each token
and uses cosine similarity between the Reference and candidate sentences
(cider). It is calculated with the following formula where $N=4$ and
$g^{n}(c_{i})$ is the TF-IDF weighted vector.

(12)
$CIDEr_{n}(C_{i},S_{i})=\frac{1}{m}\sum_{j}\frac{g^{n}(c_{i}).g^{n}(s_{ij})}{\left|\left|g^{n}(c_{i})\right|\right|\left|\left|g^{n}(c_{i})\right|\right|},CIDEr(C_{i},S_{i})=\sum_{1}^{N}\frac{1}{N}CIDEr_{n}(C_{i},S_{i})$

For all the metrics, a higher value is considered a better score.

##  5\. Results

In this section, we present the results for our research questions (Section
4.1).

###  5.1. RQ1: (Performance of LAMNER)

Table 3 shows the results of the baseline models and all the variations of our
proposed approach. The first column shows the different models used in the
evaluation, and column two to five show the BLEU scores. ROUGE-L, METEOR and
CIDEr scores are shown in column six, seven and eight respectively.

Table 3. Evaluation of various baselines and our model. The best results are bold and the second top results are shown with underline. Model | BLEU-1(%) | BLEU-2(%) | BLEU-3(%) | BLEU-4(%) | ROUGE-L(%) | METEOR(%) | CIDEr
---|---|---|---|---|---|---|---
CODE-NN | 23.90 | 12.80 | 8.60 | 6.30 | 28.90 | 9.10 | 0.98
Hybrid-DRL | 32.40 | 22.60 | 16.30 | 13.30 | 36.50 | 13.50 | 1.66
AST-Attend-GRU | 22.02 | 9.97 | 5.21 | 3.20 | 23.06 | 9.45 | 0.38
TL-CodeSum | 21.65 | 10.10 | 5.69 | 3.74 | 22.10 | 8.36 | 0.34
Codebert RENCOS | 38.3160 | 33.3250 | 30.2331 | 27.9131 | 41.0652 | 24.9833 | 2.501469
LAMNERStatic | 49.57 | 40.86 | 36.68 | 34.21 | 51.42 | 24.52 | 3.20
LAM | 50.11 | 41.09 | 36.63 | 33.98 | 52.41 | 25.00 | 3.23
NER | 50.54 | 41.68 | 37.36 | 34.83 | 52.64 | 25.30 | 3.27
LAMNER-DYN | 29.6204 | 21.1221 | 17.4295 | 15.5404 | 37.9924 | 14.0330 | 1.432124
LAMNER | 50.71 | 41.87 | 37.57 | 35.03 | 52.65 | 25.34 | 3.31

Among the baselines (row two to seven), RENCOS has the best scores for all
metrics. It’s scores for BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, METEOR and
CIDEr are XX%, XXX%, XXX%, XXX%, XXX%, XXX%, XXX%, respectively. Our proposed
model, LAMNER, and its variations, have consistently achieve better scores
than all the baseline models. LAMNER has the highest scores in BLEU-1, BLEU-2,
BLEU-3, BLEU-4, ROUGE-L, METEOR and CIDEr. It improves over RENCOS in BLEU-1,
BLEU-2, BLEU-3, BLEU-4, ROUGE-L, METEOR and CIDEr by XXX%, XXX%, XXX%, XXX%,
XXX%, XXX%, XXX%, respectively. [[[[Fuxiang: Rishab, can you update the table
with RENCOS and other baseline models’ results?]]]]

Summary of RQ1 results: LAMNER and its variations achieve the best score among
all the baseline models.

###  5.2. RQ2: (Contribution of Semantic vs. Syntax Embedding in LAMNER)

The results for the variations of our proposed model, LAMNER, are shown in row
eight to eleven in Table 3. NER has higher scores than LAM for all the
evaluation metrics, and both NER and LAM have better scores than the baseline
models (row two to seven) in all the metrics. When compared with LAMNER, which
is our proposed model learned from the combination of both the embeddings of
NER and LAM, LAMNER improves in all the metrics. These results show that our
proposed approach is effective, and the learned syntactic and semantic
information can improve the results. Even when we use the embeddings as static
vectors without further training, i.e. the LAMNER-CodeStatic (row eight), the
results are improved over all the baseline models. When comparing LAMNER-
CodeStatic with both NER and LAM, the latter have better performance in
majority of the metrics, except for BLEU-3 and BLEU-4 in LAM.

Summary of RQ2 results: The syntactic embeddings (i.e., NER) has more
contribution to the model’s performance. Both LAM embeddings and NER
embeddings learn meaningful information about the code and their combination
improves the results further.

###  5.3. RQ3: (Effect of Code and Comment Length Increase on the Model
Performance)

Code length and comment length are important factors studied in neural machine
encoder-decoder models (ase). To understand this effect on code comment
generation, we plot diagrams representing the ROUGE-L and BLEU-1 scores with
the code or comment length. The code or comment length refers to the number of
tokens in the code/comment. Figure 8 shows two plots on how BLEU and ROUGE
respond to the different code length for LAMNER, RENCOS, AST-Attend-GRU and
TL-CodeSum. The left plot uses the BLEU-1 score and the right plot uses the
ROUGE-L score [[[[Fuxiang: Why do we use BLEU-1 and not other BLEUs? Why do we
only select RENCOS, AST-Attend-GRU and TL-CodeSum?]]]]. The value at each code
length $x$ is the average score of all the code snippets with length between
$x$ and the previous length $y$. For example, the value shown for the code
length 50 is the average score of all the code snippets with length between 26
and 50, inclusively. The same approach is used to plot the comment length in
Figure 9. For BLEU, all the models have the highest scores when the code
sequence has 50 tokens. Similar to other baseline models, the score of LAMNER
decreases as the code length increases. However, we note that LAMNER decreases
much more slowly than the other models, and its BLEU scores are still higher
than the other models.

Figure 9 shows the BLEU-1 and ROUGE-L scores of the four models as the comment
length increases from 5 to 30. The scores for comment length 5 in the plots
show the average scores for comments with length less than or equal to 5. For
BLEU, figure 9 shows that after length 10, as the length of the generated
comments from TL-CodeSum, AST-Attend-GRU and RENCOS increases, their
performance decreases. In contrast, LAMNER’s BLEU-1 score increases as the
number of tokens in its generated comments increases. Similar behavior is also
seen in the ROUGE-L score. The better performance for longer comments is only
seen in our model, which is important as the human should read the comments
and the quality should be maintained when comment length increases. Examples
of the produced results are discussed in the Section 6.

Figure 8.  BLEU and ROUGE-L scores compared to the length of the input code
applied on test data Figure 9.  BLEU and ROUGE-L scores compared to the length
of the generated comment applied on test data

Summary of RQ3 results: For all code and comment lengths, LAMNER has the
highest BLEU and ROUGE scores. In contrast to other baseline models, with the
increase in the code length, LAMNER’s BLEU and ROUGE scores do not decrease as
fast as other baseline models. Interestingly, with the increase in the comment
length, LAMNER has shown fairly constant improvement in both the BLEU and
ROUGE scores while the other models either do not increase their BLEU or ROUGE
scores, or they do not increase as much as LAMNER.

###  5.4. RQ4: (Effect of Language Models and NER Embedding)

In order to analyze the effect of the language model in LAMNER-Code, we
explore ELMO (elmo) embedding and FastText (fastext) embedding and replace
them with the character-level language model that is used in LAMNER-Code. ELMO
is a character-based embedding that leverages the Neural Network-based
architecture to learn the vector representation of the characters. FastText is
a word embedding model that learns the sub-word information. FastText has
shown its effectiveness for semantic tasks in software engineering (idbench)
and ELMO can provide another perspective to encode characters other than the
LSTM based language model we used. Therefore, we choose these two embeddings
for this research question. For this study, we train ELMO and FastText
embeddings from scratch on the same data that we used to train LAM embedding.
For FastText, we set the minimum token size to one, so FastText can learn all
possible sub-words information. We then employ the trained embeddings in the
same architecture as used in LAM-Code to generate code comments, by replacing
the LAM embeddings with ELMO or FastText. The results of these new models are
shown in Table 4 as ELMO and FastText. We compare these models with the LAM-
Code model. For all evaluation metrics, LAM-Code achieves the best score. It
improves the ELMO’s BLEU scores by 3.35%–13.35%, ROUGE-L and METEOR by less
than 2% and CIDEr by 5.63%. LAM-Code improves the FastText’s BLEU1-3, ROUGE-L,
and METEOR slightly, BLEU-4 by 4.59%, and CIDEr by 10.29%. We attribute the
better results of LAM-Code to the fact that this character based language
model has shown to implicitly learn the semantic as well as syntactic
knowledge of a language (flair). Hence, LAM embedding learns some syntactic
information along with the semantic information of the code. However, FastText
embedding misses to learn such information (idbench). Both ELMO and FastText
models perform better than RENCOS in all scores, except BLEU-4 (and ELMO’s
BLEU-3 is also lower than RENCOS) 333This result is interesting, although we
made sure that all of the models are trained and tested correctly. .

FastText performs better than ELMO in almost all scores. This is because
FastText learns the sub-word (n-gram) embedding of code tokens. For example,
FastText learns the representation of identifier getMax using the information
of get and Max and this sub-word learning helps FastText to generate better
results. It also helps FastText to generalize across lexically similar tokens
(idbench).

Based on the obtained results, we use FastText to analyze the usefulness of
NER embedding for comment generation. We train a new model with the same
architecture as LAMNER-Code and replace the LAM embedding with FastText
embedding. In this model, the NER embedding is concatenated to FastText and is
fed in the model for generating comments. The results of this experiment are
shown as FastTextNER-Code in Table 4. Interestingly, the results are improved
compared to the scores obtained by LAMNER-Code. This improvement is 0.33%–6.8%
for BLEU1-4, 2.55% for ROUGE-L, 2.8% for METEOR, and 8.22% for CIDEr. Among
all models, FastTextNER-Code achieves the best score in all metrics. As
discussed previously, LAM embedding learns the semantic and syntactic
information. Therefore, when concatenating the language model with NER
embedding, we do not see much improvement as it already has learned some
syntactical information. Whereas, the NER embedding enriches FastText with the
syntactic information, as FastText lacks learning the syntactical information
(idbench) and combining NER adds new information to the model.

As RENCOS is the best model among the baselines, we run another experiment to
evaluate the effect of our embeddings in RENCOS. We incorporate the embeddings
from character-level language model and the NER model into the RENCOS
architecture by initializing each token in the vocabulary of RENCOS with LAM
and LAMNER embeddings. These models are denoted as RENCOS+LAM and
RENCOS+LAMNER in Table 4. Incorporating LAMNER embedding has more effect
compared to only LAM embedding, but the results of RENCOS are marginally
increased for BLEU-1,2, ROUGE-L, METEOR, and CIDEr scores (from 0.33% to
1.62%). This can be attributed to the retrieval part of the RENCOS as it
provides extra semantic and syntactic context to the model, thus, reducing the
effect of the used embedding in the model.

Table 4. Evaluation of language models and NER embedding. The best results are bold and the second top scores are shown with underline. For simplicity of comparison, the RENCOS, LAM-Code and LAMNER-Code results from Table 3 are repeated here. Model | BLEU-1(%) | BLEU-2(%) | BLEU-3(%) | BLEU-4(%) | ROUGE-L(%) | METEOR(%) | CIDEr
---|---|---|---|---|---|---|---
LAM | 50.11 | 41.09 | 36.63 | 33.98 | 52.41 | 25.00 | 3.23
NER | 50.54 | 41.68 | 37.36 | 34.83 | 52.64 | 25.30 | 3.28
LAMNER | 50.71 | 41.87 | 37.57 | 35.03 | 52.65 | 25.34 | 3.31
RENCOS | 38.3160 | 33.3250 | 30.2331 | 27.9131 | 41.0652 | 24.9833 | 2.501469
RENCOS+LAMNER | 42.6038 | 37.1599 | 33.6599 | 30.9556 | 43.5890 | 26.8898 | 2.735284
CodeBert | 44.3458 | 35.1935 | 30.9110 | 28.4780 | 47.6379 | 21.5875 | 2.632334
CodeBert-Lam | 46.3467 | 37.1393 | 32.7590 | 30.2268 | 48.5909 | 22.4639 | 2.806838
CodeBert-Ner | 46.7432 | 37.4861 | 33.1526 | 30.6567 | 48.8116 | 22.6472 | 2.872033
CodeBert-Lamner | 47.5652 | 38.3674 | 34.0080 | 31.4628 | 49.4058 | 23.2083 | 2.632334
CodeBert-FastLamner | 48.3553 | 39.1073 | 34.6581 | 32.0504 | 50.9580 | 23.9496 | 3.024590
FastText | 50.66 | 41.51 | 36.95 | 34.22 | 53.27 | 25.49 | 3.27
FastText+NER | 51.23 | 42.15 | 37.62 | 34.90 | 53.59 | 25.80 | 3.33
FastText+LAM | 51.81 | 42.70 | 38.20 | 35.49 | 53.71 | 25.99 | 3.37
FastText+LAMNER | 51.78 | 42.71 | 38.23 | 35.55 | 53.85 | 26.02 | 3.38

Summary of RQ4 results: Both LAM and NER embeddings are helpful in comment
generation. Language models are effective in creating comments, and the
character-level language model (LAM) achieves the highest score among all.
When NER embedding is combined with FastText, it can boost its scores by
2%–16% for all metrics.

###  5.5. RQ5: (Evaluation of Models When Tested on Abstracted Code)

All variations of our model and other baseline models presented in previous
research questions are trained on code snippets that include the identifier
names. Identifiers are defined by the developers, which means developers can
use different names for the tokens of the same source code without changing
its functionality. Ideally, code comment generation models should handle such
scenarios independent of the identifier names and demonstrate robust
performance against such cases. Therefore, in this research question, we
evaluate the performance of the comment generation models on abstracted code.
For this experiment, we define a ’challenge dataset’, in which the code
snippets in the test set are abstracted. For the code abstraction, we follow
the approach used in (tosem-refinement). We replace each identifier name and
its type in the code snippet with a more generalized name like ’Identifier.’
and ’Type’. An id is appended to each identifier and type depending on their
sequential occurrence. Figure. 10 presents an example of a code and its
abstraction. The code abstraction tests the ability of the models to generate
comments independently of identifier names and types by only using the
generalized names. This dataset is only used during the testing of the model,
therefore, the new identifier and type names are not seen by the models during
the training.

Figure 10. Code Abstraction

The results are shown in Table 5 and the highest scores are bold in the table.
We see a substantial decrease in the performance of all the comment generation
models when tested on abstracted code. ELMO’s performance drops significantly
and is close to zero. For the NER-Code we also see very low score in all
metrics. Among the language models, LAM-Code and FastText are performing
better and are comparable. The results of RENCOS is close to FastText and LAM-
Code. In case of RENCOS, the model retrieves the most similar semantic and
syntactic code samples during the decoding, which allows the model to see non-
abstracted code tokens from the retrieved samples, introducing an advantage
for RENCOS. Finally, the scores of LAMNER-Code and FastTextNER-Code are higher
than other models. For these models, the NER embedding can boost their results
compared to LAM-Code and FastText.

The obtained results are expected, as most of the tokens in the abstracted
code are not available within the vocabulary of the models. Therefore, for all
unseen tokens, i.e., the abstracted identifier names and types, the models use
the static representation of the unknown token (‘UNK’) at the embedding layer
for the out of vocabulary words. This is also the case for LAM-Code, LAMNER-
Code and FastTextNER-Code, however, these models use the pre-trained semantic
embeddings of the tokens that are fine tuned during model training, which help
the models generate better results compared to others when tested on
abstracted code. The low scores of the NER-Code is because, as seen from
previous experiments, syntax embedding individually does not generate best
results as it lacks the semantic learning and the semantic learning has shown
to be the most contributing factor for comment generation (clair; rencos;
code2seq).

Based on this result, we provide another experiment with the LAM-Code and
LAMNER-Code. In this research, rather than using the embedding lookup table
for the tokens in the vocabulary, we generate embedding for each token at run
time. These models are shown as LAM-Code-DYN and LAMNER-Code-DYN in Table 5,
in which the contextual embedding of each token is dynamically generated using
the language model and Language-NER model, respectively. This behavior of the
LAM-Code-DYN and LAMNER-Code-DYN removes the dependency of the encoder on its
embedding look up table which makes the model use the static embedding of the
‘UNK’ token. Therefore, each new ‘UNK’ token is initialized to its contextual
representation, contrary to using the embedding of the ‘UNK’ token in pre-
trained embedding. The LAMNER-Code-DYN provides the best performance among all
other models, and the LAM-Code-DYN achieves the second best scores (underlined
in the table). This is one of the main advantages of using contextual
embedding as it generalizes better compared to other baselines and even
compared to models that use pre-trained embedding (i.e., LAM-Code, LAMNER-
Code, and FastTextNER-Code). The generalization of the contextual embeddings
is possible because of the pre-training of character language model and the
NER model on source code which learns the natural fluency and structure of
source code, which should remain same despite the abstraction of source code
using new identifier and type names. Similar experiment cannot be applied on
FastText because it learns the embedding of subwords and would generate the
same embedding for the token despite the change of context, unlike LAM
embedding. which does not change with the context of the given code.

Table 5. Evaluation of various baselines and our model on the abstracted code. The highest scores are shown in bold and the second highest scores are underlined. Model | BLEU-1(%) | BLEU-2(%) | BLEU-3(%) | BLEU-4(%) | ROUGE-L(%) | METEOR(%) | CIDEr
---|---|---|---|---|---|---|---
RENCOS | 10.2745 | 3.7952 | 1.6588 | 0.07553 | 9.8395 | 4.4751 | 0.071584
Codebert-Embeds | 2.2659 | 0.3911 | 0.0809 | 0.0208 | 4.5666 | 1.3579 | 0.003338
FastText | 8.71 | 1.31 | 0.18 | 0.00 | 17.75 | 3.20 | 0.01
NER-Code | 6.6046 | 0.07078 | 0.0000 | 0.0000 | 9.5128 | 2.0230 | 0.012449
LAM-Code | 8.5175 | 1.3783 | 0.1862 | 0.0540 | 10.3967 | 2.2230 | 0.013672
FastTextNER-Code | 12.66 | 2.05 | 0.44 | 0.12 | 16.04 | 3.80 | 0.02
LAMNER-Code | 12.3859 | 3.2030 | 1.0116 | 0.2359 | 12.9963 | 3.6748 | 0.036873
LAMNER-Code-DYN | 16.2594 | 4.8858 | 1.6908 | 0.7692 | 19.9039 | 4.7176 | 0.073290

Summary of RQ5 results: There is a huge performance decrease when models are
tested on abstracted code, which is attributed to the unknown token in the
generated embeddings. The models which incorporate proposed embedding have
better results. Moreover, dynamically generating contextual embedding of each
token at run time using character-level language model and NER model
alleviates this problem a bit and these models achieve the highest scores in
this setting.

###  5.6. RQ6: (Effect of Combining AST and Information Retrieval Techniques
and The Use of Transformer Architecture)

In this research question, first we investigate models in which we take
advantage of the components of the best baseline models, namely AST and
information retrieval techniques. Thus, we develop two models, where one
combines LAMNER-Code with AST and the other uses a retrieval technique similar
to RENCOS. The results of these models are shown in Table 6 as LAMNER-Code+AST
and LAMNER-Code+Retrieval. To incorporate AST with LAMNER-Code in the former
model, we introduce a dual encoder model similar to the AST-Attend-GRU model
(clair). We use AST as the input in one of the encoders and the source code as
input to the second encoder. The former learns the syntactical structure of
code and the latter learns the semantic information of code (hu11). We combine
both representations from the two encoders using a weighted attention
mechanism as described in Hybrid-Deepcom (hu11). This attention mechanism
projects the outputs from both of encoders into a common vector space and
attend to tokens from both encoders. When AST is added to LAMNER-Code model,
the precision-based metrics, i.e. BLEU scores are increased by 1.85%–3.59%,
and the other metrics, METEOR, ROUGE-L, and CIDEr are decreased by
1.19%–6.85%. The increase in the performance can be related to the exclusive
encoder assigned in the model to learn the syntactic structure of the code
from ASTs. We also build a similar model to combine the FastTextNER-Code model
with AST which is shown as FastTextNER-Code+AST. Adding AST to this model
slightly decreases the performance of FastTextNER-Code in most metrics and
slightly improves the others (BLEU-1 and BLEU-2 scores). This results can be
related to the fact that the pre-trained embedding is successfully able to
learn the semantic and syntax information. Therefore, adding similar extra
information through AST (as a sequence of tokens and their types) keeps the
performance more or less same. Similar behavior is seen in previous works when
adding AST improves the performance slightly (clair).

For the other model, LAMNER-Code+Retrieval, similar to the approach used in
RENCOS (rencos), we retrieve two similar semantic and syntactic samples for
each code during testing, with the difference that we use LAM and NER
embeddings respectively to find the similar samples. Then, the retrieved
samples are fused to generate a probability distribution of the output token,
leveraging the retrieved samples. The performance of LAMNER-Code for all
scores is decreased when a retrieval technique is used. This can be due to
finding the similar samples using the embeddings, instead of incorporating the
lexical similarity, as used in RENCOS, which can be a more effective retrieval
technique.

The final experiment in this part is related to the neural network
architecture. As explained previously, we use GRU architecture in our model.
Although this architecture is used in other comment generation studies
(clair), a more powerful architecture known as Transformer is considered as
the state of the art for many tasks in natural language processing field (all-
you-need). Therefore, we explore the use of our embeddings with the
Transformer. In this model, we incorporate the pre-trained embeddings of
LAMNER-Code to initialize the vocabulary of the Transformer based model. We
use Transofrmer with six stacked encoders and decoders and set the embedding
dimension to 512 to make it a fair comparison with other baselines. This model
is shown as Transformer+LAMNER in Table 3. We also train a Transformer model,
without using our embeddings and present its results as the Transformer in the
table. Combining LAMNER embedding into the Transformer degrades its results.
This is aligned with the results of a previous study where the authors
incorporate structural information to the Transformer-based code summarization
model and report a decrease in its performance (transformer). They hypothesize
that the Transformer architecture implicitly learns the code structure and
including structural information reduces the effectiveness of learned
information. We hypothesize that same reason could be possible for the
degradation of performance using semantic encoding as well, as the Transformer
learns such contextual information. However, learning the structure or
semantics of code is important for comment generation for which using semantic
and syntax embedding may not be the ideal technique to incorporate such
information in Transformers. Incorporating other techniques like data flow
graphs may be a better approach to learn this knowledge (graphcodebert).

Table 6. Results of fused models. For simplicity of comparison, the LAMNER-Code results from Table 3 are repeated here. B stands for BLEU. The best scores are shown in bold. Model | B-1(%) | B-2(%) | B-3(%) | B-4(%) | ROUGE-L(%) | METEOR(%) | CIDEr
---|---|---|---|---|---|---|---
Transformer | 25.25 | 13.23 | 7.41 | 4.57 | 29.61 | 10.80 | 0.57
Transformer+LAMNER | 24.01 | 12.28 | 6.56 | 3.78 | 29.13 | 10.16 | 0.48
LAMNER-Code+RENCOS(Retrieval) | 42.6038 | 37.1599 | 33.6599 | 30.9556 | 43.5890 | 26.8898 | 2.735284
LAMNER-Code | 50.71 | 41.87 | 37.57 | 35.03 | 52.65 | 25.34 | 3.31
LAMNER-Code+AST | 51.0446 | 42.0967 | 37.8185 | 35.3191 | 52.6828 | 25.4200 | 3.329126
FastText+LAMNER | 51.78 | 42.71 | 38.23 | 35.55 | 53.85 | 26.02 | 3.38
FastText+LAMNER+AST | 52.0668 | 43.0158 | 38.5627 | 35.9080 | 54.0568 | 26.2226 | 3.433174

Summary of RQ6 results: Our proposed embedding, LAMNER, can be complemented by
AST-based code representation. The LAMNER-Code has better performance than the
Transformer, but incorporating this embedding to Transformer is not ideal to
learn the syntactical information.

##  6\. Discussion

Although LAMNER has better results than the baseline models, it still remains
unclear in what scenarios LAMNER can generate good summaries. Manual
investigation, similar to (hu11), is conducted to learn such scenarios for the
generated comments. We explore samples from the test data generated by LAMNER,
RENCOS, AST-Attend-GRU, and TL-CodeSum to answer the following questions.

  * •

Where does LAMNER perform better than the top baselines?

  * •

Where does LAMNER fail to generate good comments?

###  6.1. Inspecting Higher BLEU Score Comments

We randomly sample 100 methods for which LAMNER has at least 25 or more BLEU
points than the baselines. 25 is used as the minimum threshold here because
LAMNER’s average BLEU-1 score is close to 25. We then investigate the
performance of our model based on the functionality of the methods. To
categorize the code functionalities, we follow an approach similar to the one
used in (hu11). Two graduate students(including one author of the paper), with
4 and 6 years experience in Java development, independently label the
functionality for each of the 100 methods. We measured the inter-rater
reliability using Cohen’s Kappa coefficient (mchugh2012interrater). This
metric measures the level of agreement between two raters when rating
categorical values. The Cohen’s Kappa test result is a number between $-1$ and
$+1$, where $-1$ shows the highest disagreement rate and $+1$ represents that
raters are in 100% agreement. In software engineering field, the rate of
$0.79$ or higher is considered as a high agreement rate (liu2020usingSATDinDL;
huang2018identifying). We obtained the Cohen Kappa value of $+0.82$ which
shows the two raters having high agreement in their defined categories. The
discrepancies in the annotation are discussed between both the raters to reach
a consensus on the final label.

The categories from this manual investigation are task implementation, object
creation, update/set value, connection operation and string operation. The
task implementation category includes code samples that accomplish some task
such as finding the maximum row count. Code samples which update and set the
values for objects are listed in the update/set value and code samples that
create new objects are labeled as object Creation. Connection operation group
includes samples that are related to setup, opening, closing, and deleting
connections between client and server. Finally, string operation category
includes samples that perform operations such as split and concatenation of
strings. The number of samples in each category are shown in Table 7. The
distribution in Table 7 shows that LAMNER generates better comments primarily
in the task implementation, update/set value and object creation categories
with 31%, 24% and 21% of code samples, respectively. The lowest percentage is
9 and it belongs to string operation. We note here that these results are only
based on the sample data used for manual evaluation and they may not reflect
better performance on a certain category in the whole dataset. The
distribution of the data might differ in the sample and in the whole dataset.
However, finding the distribution of the whole dataset is not feasible as it
requires a lot of human effort to annotate the data for each category
correctly. Therefore, we followed the techniques and approach of the previous
studies for this discussion (hu11). Figure 11 represents a code sample and its
generated comments by LAMNER and other baselines. This is a sample where
LAMNER has at least 25 higher BLEU score compared to the best baselines. The
Example ID in the figure shows the ID of the record in the test set. From the
sample in Figure 11, we observe that LAMNER is able to generate important code
tokens like `buffer` and `offset` in the method. More examples are included in
the appendix. We investigate the ability of the models in capturing the
identifiers in their comments as follows.

Table 7. Functionality distribution of the 100 sampled Java methods Functionality | Number of samples
---|---
Task Implementation | 31
Object Creation | 21
Update/Set Value | 24
Connection Operation | 15
String Operation | 9

Generating Identifiers. Identifiers are names used for identification in Java
such as method names and variable names. To be able to generate identifiers
within the code comment is beneficial as it is able to describe clearer the
functionality of the code (hu2). In our dataset, separators and identifiers
contribute 46.00% and 37.87% of the total tokens respectively. Separators such
as [ {, }, (, ), ”,”, and ”;” ], if used in the comments, do not reveal much
information about the functionality of code. However, including identifiers
inside the comments can reveal a lot of information about what the method does
(iyer). In this section, we study the ability of LAMNER and the baselines to
correctly refer to the identifiers in their generated comments. To evaluate
this, for each code sample in the test set, we collect all the unique
identifiers using the Javalang parser. The collected identifiers are then
checked for their occurrence in the reference sentences as well as in the
corresponding comments of LAMNER, RENCOS, AST-Attend-GRU, and TL-CodeSum. As
NMT models can generate repetitive tokens, for each model, we count the
occurrence of an identifier in each sample once, even if it has appeared
multiple times in that sample. For example if the identifier “maximum” appears
twice in the comment finds the maximum and returns the maximum, we count the
occurrence of “maximum” only once. We compare these numbers with the number of
identifier occurrences in the Reference sentences.

The results are shown in Table 8. LAMNER is able to produce 73.19% of the
identifiers in the generated comments, compared to 62.42%, 66.98% and 71.98%
of the identifiers captured by AST-Attend-GRU, RENCOS and TL-CodeSum
respectively. Interestingly, although the NMT results for RENCOS are better
than TL-CodeSum, TL-CodeSum can better refer to identifiers in its comments.
We relate this to the usage of API knowledge in TL-CodeSum. On the other hand,
LAMNER can correctly predict identifiers without using any external knowledge
source.

Figure 11. Samples with higher BLEU-1 score Table 8. Number of identifiers in comments of each model compared to the Reference Models | Number of identifiers in comments
---|---
Human Written References | 9510
LAMNER-CodeBeam-Act | 7259
LAMNER-Code | 6960
TL-CodeSum | 6846
RENCOS | 6370
AST-Attend-GRU | 5936

###  6.2. Inspecting Low BLEU Score Comments

To investigate the low BLEU score comments, we follow the same approach used
in (hu11). We randomly sample 100 records from the test set which have BLEU
scores less than 20%. These samples are then divided into three categories, i)
Good, ii) Related and iii) Not Related. The Good category contains samples
that are accurate and may have low BLEU score because of using synonyms or
related words rather than repeating the original tokens used in the Reference.
Related group contains samples that are related to the method definition, but
the comments are not exactly accurate. Finally, Not Related is the group of
sample codes for which LAMNER-Code is not able to generate meaningful
comments. For this categorization, similar to the previous section, the
comments are labelled independently by graduate students(including one author
of the paper). The Cohen’s Kappa score is 0.81 and the authors discussed the
disagreements until they reach a consensus. The percentages of comments in
each group are shown in Table 9. The results show that despite having low BLEU
scores, LAMNER-Code is able to generate accurate comments for 37% of the
samples, and 39% of its generated comments are related to the functionality of
code. LAMNER-Code only fails in producing meaningful comments for 24% of the
samples. Below, we discuss our observations for comments with BLEU score equal
to zero.

Table 9. Classification of samples with low BLEU score Category | Number of samples
---|---
Good | 37
Related | 39
Not Related | 24

Comments with BLEU Score 0. In our study, we found examples with zero BLEU
score, where the human written comments are either unrelated or not updated
when the project has evolved. An example is shown in Figure 12, where the BLEU
scores for all models are zero. The reference comment of this sample aims to
describe the methods’ functionality; but the comment is outdated and it is not
a correct description of what the method is doing. In contrast, comments
generated automatically by LAMNER and the baselines are more logical and they
are related to the code. This example emphasize the need for automatic code
comment generation techniques as part of the project maintenance life-cycle
(method-icse2020). More examples are included in the appendix.

Figure 12. Samples with BLEU-1 score equal to zero

##  7\. Human Evaluation

BLEU (bleu), ROUGE-L (rouge) and METEOR (meteor) are extensively used in
machine translation to draw a quantitative comparison among the NMT models.
The models with more overlapping tokens between the references and predictions
receive higher scores. However, the texts can have the same meaning without
using common tokens. This makes it difficult to comprehend the effectiveness
of different models (bleurt; bleu-problem). Following previous works (hu11;
iyer; rencos), we conducted a human evaluation study to compare the generated
comments from LAMNER and the best performing baseline, RENCOS. The human
evaluation is to assess the quality of the sample comments based on three
criteria: Correctness, Naturalness/Fluency, and Informativeness. For
Correctness, we ask whether each of the generated comment describes the
functionality of the code correctly. Naturalness/Fluency is the criterion on
whether the generated comments are grammatically correct and fluent in terms
of human readability. Informativeness is the criterion where we ask whether
the comment correctly mentions the required content from the code (e.g.
capturing necessary identifier) in describing its functionality, ignoring the
fluency of the text.

For the human evaluation study, we randomly select 100 samples from the test
dataset. For each sample, we collect the Java code, the reference comment, and
the comments generated by LAMNER and RENCOS. Then, we divide the 100 samples
into four equal size groups. We invite eight volunteers with one to ten years
of Java development experiences for the evaluation. Each group of data
contains 25 records and it is rated by two volunteers using a survey system
444https://it.ubc.ca/services/teaching-learning-tools/survey-tool. The survey
contains three criteria for each of the code samples. To reduce the bias, the
name of the models and the reference comments are not shown to the evaluators.
We follow a similar rating system that is used in previous works (iyer; hgnn;
tsedrl) – The volunteers are asked to rate the comments in the range of
$[1,5]$ stars , where one star represents the worst and five stars the best,
based on the following criteria: i) Correctness, ii) Naturalness/Fluency, and
iii) Informativeness. The volunteers are shown an example with explanations
for all the criteria before starting to rate.

Table 10. Results of human evaluation Model | Correctness | Naturalness | Informativeness
---|---|---|---
Ground-Truth | 3.61 | 4.37 | 3.79
LAMNER | 3.22 | 4.00 | 3.22
RENCOS | 2.77 | 3.85 | 2.94

Table 10 shows the average scores of the models rated by the volunteers on the
three metrics. The predictions from LAMNER are consistently rated better than
RENCOS on all the three metrics.

##  8\. Threats to validity

Internal Validity: Internal threats in our work relates to the errors in
building our own models and the replication of the baseline models, as these
are the internal factors that might have impacted the results. We have cross-
checked the implementation of our model for its correctness and open-sourced
the implementations for easier replication of the results. For all the
baseline models, we used the official code provided by the authors. During the
baseline models training, if we encountered any errors or had any doubts, we
consulted the authors directly through raising GitHub issues. An important
difficulty of training the models is preparing the datasets in the required
format. Various baseline models required separate pre-processing such as
generating AST, using third party libraries such as py-lucene
555https://lucene.apache.org/pylucene/, and extracting the API knowledge. For
these steps, we followed the instructions provided by the authors closely to
ensure the correctness that the required input is correct. Another threat to
validity could be the dataset that we have built for the NER model. We used
the Javalang parser to label the dataset. This parser reliably tags the Java
methods and have been used in several software engineering studies (rencos;
ase-ptm). We have broken down the identifiers into four sub-categories using a
rule based script. We take careful steps in breaking down the identifiers – we
randomly check the sub-categories and did not find any misclassification.

External Validity: In our study, threats to external validity relates to the
generalizability of the results (hu11). We used an external and extensively
used dataset in our work. We only applied our model on the Java programming
language. Although we hypothesize that the embeddings used in our work is
beneficial to other programming languages, more studies are required.

Construct Validity: In our work, the automatic evaluation metrics might affect
the validity of the results. We used four different automatic machine
translation metrics (BLEU1-4, ROUGE-L, METEOR and CIDEr), to reduce the bias
of a particular metric. These metrics are frequently used in natural language
processing and in other related and similar studies in software engineering
domain (codebert; code2seq; rencos; ase; hu11; retref).

Conclusion Validity: The conclusion validity refers to the researchers’ bias
or the bias found in the statistical analysis that can lead to weak results
(ampatzoglou2019ConclusionValidity). The nature of this study does not depend
on the researchers’ bias. The conclusion threat can be related to the
reliability of the measurements obtained by the automatic metrics. To mitigate
this threat, we used different evaluation metrics (BLEU1-4, ROUGE-L, METEOR,
and CIDEr) and applied the same calculation to evaluate all the models. We
note that the automatic metrics cannot fully quantify the quality of the
generated comments. In mitigation, we conducted human studies to compare the
results from the different models based on the developers’ perspective. A
potential threat could be related to the conclusions obtained by the human
study. To reduce this threat, we anonymised the models and the reference
comments are not shown to the evaluators. Additionally, each generated comment
is rated by two evaluators for consistency purposes.

##  9\. Related Works

In this section, we provide a summary of the related works on code embedding
and code comment generation.

###  9.1. Code Embedding

The research on source code embedding is wide and has many applications
(zchen). Hindle et al. (devanbu) used n-grams to build a statistical language
model and showed that programming languages are like natural languages: they
are repetitive and predictable. Bielik et al. proposed a statistical model
that is applicable to character-level language modeling for program synthesis
(bielik). Recent embeddings for code tokens are based on deep neural networks
and are mostly generated with the word2vec model (mikolov2013word2vec), for
C/C++, JavaScript and Java tokens. These embeddings are used for program
repair (cv3), software vulnerability detection (cv2), type prediction (type-
pred) and bug detection (cv5). Other embedding techniques from natural
language like FastText (fastext) is used in another work to provide pretrained
embeddings for six programming languages (code-fasttext). Wang et al. (trace-
embed) proposed a technique for program representations from a mixture of
symbolic and concrete execution traces. A modified Graph Neural Network called
Graph Interval Neural Network is used to provide semantic representation of
programs (ginn). Kanade et al. (cubert) trained BERT (bert) to generate the
contextual embeddings and showed their effectiveness on variable misuse
classification, and variable misuse localization and repair. Karampatsis and
Sutton use ELMO (scelmo) for bug prediction. Alon et al. generate the
embeddings of Java methods using AST (code2vec). Lu et al. propose a new
embedding in a hyperbolic space that uses function call graphs (hyper). Other
works propose neural probabilistic language model for code embedding (fse),
function embeddings for repairing variable misuse in Python based on AST
(semcode), embedding of methods for code clone detection using AST node type
and node contents (recursiveast), embeddings based on tree or graph based
approaches (ltrgraph; g1), embeddings to learn representation of edits (g2),
and embeddings for program repairs (g3). Chen at al. (zchen) provides a
comprehensive review of the embeddings for source code. The most similar works
are (hu1; nips) and they use word-level encoding for code comments. Although
code embeddings is widely used in many applications, the existing code
embeddings have limitation of not able to detect identifiers such as those in
camel case and snake case succintly, and the code token embeddings is not able
to capture the structural property in programming languages. Our work proposes
a novel Symentatic-Syntax encoder-decoder model, LAMNER, to tackle the said
limitation. Our experiments showed that LAMNER is effective and has improved
performance over all the baseline models.

###  9.2. Code Comment Generation

Software engineering researchers have proposed multiple techniques to improve
automatic code comment generation. Initial efforts were made using the
information retrieval, template-based, and topic modeling approaches. Haiduc
et al. (ir1) used text retrieval techniques such as Vector Space Model and
Latent Semantic Indexing to generate code comments. A topic modeling approach
was followed by Eddy et al. (eddy) to draw a comparison between their work and
of the approach used in (ir1). Moreno et al. (mor) used a template-based
approach to generate the comments for methods and classes automatically.
Sridhara et al. (sridhara) introduced Software Word Usage Model (SWUM) to
capture code tokens’ occurrences to generate comments. Later, Iyer et al.
(iyer) presented a neural network for code comment generation. They were the
first to use attention-based LSTM neural translation model for comment
generation. Hu et al. (hu1; hu11) introduced a model that uses AST. They
proposed a modified depth-first search based traversal algorithm, namely
Structure-Based Traversal, to flatten the AST. Alon et al. (code2seq) consider
all pairwise paths between leaf nodes of AST and concatenate the
representation of this path with each leaf node’s token representation.
LeClair et al. (clair) presented a dual encoder model that combines the code
sequence and AST representation of code. Liang et al. (codernn) made changes
to GRU architecture to enable encapsulating the source code’s structural
information within itself. Wan et al. (ase) employed actor-critic
reinforcement learning and Tree-RNN to generate comments. Yao et al. (coacor)
modeled the relationship between the annotated code and the retrieved code
using a reinforcement learning framework and used it to generate the natural
language code annotations. Wei et al. (nips) proposed a dual framework that
leverages dual training of comment generation and code generation as an
individual model. Leclair et al. (maclatest) improved the quality of generated
comments by employing a Graph Neural Network model with AST and source code. A
recent approach combines the techniques available in information retrieval to
train an NMT model (rencos). During testing, two similar code snippets are
retrieved from the test data and used as input along with the test sequence.
Similarly, Wei et al. (retref) input a similar code snippet, AST, code
sequence, and an exemplar comment to generate better comments. Liu et al.
(jit) and Panthaplackel et al. (jit2) proposed a comment update technique
which learns from code-comment changes and generates new comments. Wang et al.
(cocogum) use code token, AST, intra-class context from the class name, and
Unified Modeling Language diagrams to generate comments. Haque et al. (msr-
context) use full code context file to generate comments for methods. More
recently, researchers have also become interested in employing a pretrained
language model (cubert). Feng et al. (codebert) trained a multilingual
transformer-based language model on six languages and tested the model for
code comment generation. The previous research use different techniques to
represent code. In this work, we introduced a novel technique to capture the
semantic-syntax information that is inherently important in programming
language, but cannot be modelled using AST. Despite that, our approach can
still use AST as an auxillary input to the model for additional structural
information of the code.

##  10\. Conclusion

In this paper, we present a novel code comment generation model, LAMNER, which
uses a semantic-sytax embeddings that encodes the semantic and the syntactic
structure of a code token. LAMNER can be combined with other auxiliary inputs
such as AST to further improve the model performance. The evaluation on BLEU,
ROUGE-L, METEOR, and CIDER metrics confirm that LAMNER achieves the state-of-
the art performance. We relate this result to the pre-trained embeddings that
we have introduced and their ability to extract the semantic and syntactical
representation of unseen code sequences. This result is also supported through
human evaluation. The human evaluation also suggest the comments from LAMNER
is fluent and are grammatically correct. Several studies are conducted which
show the importance of both embeddings for comment generation, as well as
their robustness when tested on abstracted code. In the future, we plan to
apply LAMNER on other programming languages and on different tasks such as bug
prediction.

##  Appendix A Extra Predicted Samples

In this section, we provide some of the example code comments generated by
LAMNER. These samples are continuation of the discussion in Section 6. Figure
13 shows the code samples and the generated comments by LAMNER and other
baseline models. Compared to other baseline models, LAMNER is able to generate
important and relevant key words that describe the code. For example, ’page’
and ’url’ in the first example (ID: 620), ’regular expression’ in the second
sample (ID: 7213), and id in the third sample (ID: 1221).

Figure 13. Samples with better BLEU-1 score for LAMNER-Code as compared to
baselines

In Figure 14, it shows the sample generated comments that have zero BLEU
scores from the baseline models, including LANMER. The first two examples with
IDs 6690 and 1247 have human-written comments that are for internal use. These
texts are not intended to define the functionality of the methods. For example
ID 7712, its Reference comment is outdated. Compared to the other baseline
models, LAMNER is able to produce more relevant comments.

Figure 14. Samples with zero BLEU-1 score