A Survey on Deep Learning for Software Engineering

Yanming Yang

Faculty of Information Technology, Monash University

Xin Xia

Faculty of Information Technology, Monash University

David Lo

School of Information Systems, Singapore Management University

John Grundy

Faculty of Information Technology, Monash University

In 2006, Geoffrey Hinton proposed the concept of training “Deep Neural Networks (DNNs)” and an improved model training method to break the bottleneck of neural network development. More recently, the introduction of AlphaGo in 2016 demonstrated the powerful learning ability of deep learning and its enormous potential. Deep learning has been increasingly used to develop state-of-the-art software engineering (SE) research tools due to its ability to boost performance for various SE tasks. There are many factors, e.g., deep learning model selection, internal structure differences, and model optimization techniques, that may have an impact on the performance of DNNs applied in SE. Few works to date focus on summarizing, classifying, and analyzing the application of deep learning techniques in SE. To fill this gap, we performed a survey to analyse the relevant studies published since 2006. We first provide an example to illustrate how deep learning techniques are used in SE. We then summarize and classify different deep learning techniques used in SE. We analyzed key optimization technologies used in these deep learning models, and finally describe a range of key research topics using DNNs in SE. Based on our findings, we present a set of current challenges remaining to be investigated and outline a proposed research road map highlighting key opportunities for future work.

§ INTRODUCTION

In 1943, Warren Mcculloch and Walter Pitts first introduced the concept of the Artificial Neural Network (ANN) and proposed a mathematical model of an artificial neuron <cit.>. This pioneered a new era of research on artificial intelligence (AI). In 2006, Hinton et al. <cit.> proposed the concept of “Deep Learning (DL)”. They believed that an ANN with multiple layers possessed extraordinary feature learning ability, which allows the feature data learned to represent the essence of the original data. In 2009, they proposed Deep Belief Networks(DBN) and an unsupervised greedy layer-wise pre-training algorithm <cit.>, showing great ability to solve complex problems. DL has since attracted attention of academics and industry practioners for many tasks. Development of Nvidia’s graphics processing units (GPUs) significantly reduced the computation time of DL-based algorithms. DL has now entered a period of great development. In 2012 Hinton's research group participated in an image recognition competition for the first time and won the championship in a landslide victory by training a CNN model called AlexNet on the ImageNet dataset. AlexNet outperformed the second best classifier (SVM) by a substantial margin. In March 2016, AlphaGo was developed by DeepMind, a subsidiary of Google, which defeated the world champion of Go by a big score. With continuous improvements in DL's network structures, training methods and hardware devices, DL has been widely used to solve a wide variety of research problems in various fields.

Driven by the success of DL techniques in image recognition and data mining, industrial practitioners and academic researchers have shown great enthusiasm for exploring and applying DL algorithms in diverse software engineering (SE) tasks, including requirements, software design and modeling, software implementation, testing and debugging, and maintenance. In requirements engineering, various DL algorithms have been employed to extract key features for requirement analysis, and automatically identify actors and actions (i.e., user cases) in natural language-based requirement descriptions <cit.>. In software design and modeling, DL has been leveraged for design pattern detection <cit.>, UI design search <cit.>, and software design mining <cit.>. During software implementation, researchers and developers have used DL for source code generation <cit.>, source code modeling <cit.>, software effort/cost estimation <cit.>, etc. In software testing and debugging, various DL algorithms have been designed for detecting and fixing defects and bugs existed in software products, e.g., defect prediction <cit.>, bug localization <cit.>, vulnerability prediction <cit.>. It has been used for a varietu of software testing applications, such as test case generation <cit.>, and automatic testing <cit.>. Researchers have applied DL to SE tasks to facilitate software maintenance and evolution, such as code clone detection <cit.>, feature envy detection <cit.>, code change recommendation <cit.>, user review classification <cit.>, etc.

However, there is a lack of a comprehensive survey of deep learning usage to date in SE.
This study performs a detailed survey to review, summarize, classify, and analyze relevant papers in the field of SE that apply DL models. We collected, reviewed, and analyzed 142 papers published in 20 major SE conferences and journals since “deep learning” was introduced in 2006. We then analyzed the development trends of DL in SE, classified various DL techniques used in diverse SE tasks, analyzed DL's construction process, and summarized the research topics tackled by relevant papers. This study makes the following contributions:

* We conducted a detailed analysis on 142 relevant studies that used DL techniques in terms of publication trend, distribution of publication venues, and types of contributions. We analyzed an example in detail to describe the basic framework and the usage of DL techniques in SE.

* We provide a classification of DL models used in SE based on their architectures and  an analysis of DL technique selection strategy.

* We performed a comprehensive analysis on the key factors that impact the performance of DL models in SE, including dataset, model optimization, and model evaluation.

* We provide a description of each primary study according to six different SE activities and conducted an analysis on these studies based on their task types. These include regression, classification, ranking, and generation tasks.

* We discuss distinct technical challenges of using DL in software engineering and outline key future avenues for research on using DL in software engineering.

Section 2 introduces the workflow of a DL model through an example. Section 3 presents our Systematic Literature Review methodology. Section 4 investigates the distribution and evolution of DL studies for SE tasks, and Section 5 gives an overall analysis on various DL techniques used in primary studies, including classifying 30 DNN models based on their architectures and summarizing the model selection strategies adopted by studies. Section 6 analyzes a set of key techniques from four perspectives – datasets, model optimization, model evaluation, and the accessibility of source code. Section 7 lists research topics involved in primary studies and makes a briefly description of each work. Section 8 presents limitations of this study and its main threats to validity. Section 9 discusses the challenges that still need to be solved in future work and outlines a clear research road-map of research opportunities. Section 10 concludes this paper.

§ DEEP LEARNING

§.§ Basic Structure of DL

Most learning algorithms are shallow models with one or two non-linear feature representation layers, such as GMM, HMM, SVM, and MLP. The limitation of such a shallow model is the lack of  ability to express complex functions. Their generalization ability is restricted for the complexity of problems, resulting in decreased learning ability.

Deep learning allows computational models composed of multiple layers to learn data representations with multiple higher levels of abstraction <cit.>. This builds a neural network that simulates the human brain for analysis and learning. Similar to traditional ML, DL is suitable for various types of problems, such as regression, clustering, classification, ranking, and generation problems. We present the basic structure of a DNN in Fig. <ref>.

The basic structure of a deep learning model.

Based on the position and function of different layers, layers in a DNN can be classified into three categories, i.e., the input layer, the hidden layer, and the output layer. Generally, the first layer denotes the input layer, where the preprocessed data can be fed into DNNs; the last layer denotes the output layer, from which the results of a model can be achieved, e.g., classification results, regression results, generation results, etc. The middle layers between the input layer and the output layer are hidden layers. DNNs usually contain multiple hidden layers for enhancing the expressive ability of DNNs and learning high-level feature representation. Besides, the way to connect between different layers may vary, and as shown in Fig. <ref>, adjacent layers are full-connected (aka., full-connected layer), meaning that any neuron in the $i_{th}$ layer are connected to any neuron in the $i+1_{th}$ layer. In some DNNs with complex structures, for tackling different SE issues, not only the fully connected layer can be used as a hidden layer, but also a layer composed of other types of neurons can also be used as the hidden layer of a DNN, such as convolution layer, pooling layer, LSTM layers, etc..

Currently, diverse DNNs and learning methods are used for SE tasks, such as Feedback Neural Network (FNN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), AutoEncoders, Generative Adversarial Networks (GAN), and Deep Reinforcement Learning <cit.>.

§.§ Workflow when Using DL in SE

We present an example of using DL for a representative SE task. A novel sequence-to-sequence neural network is used to automatically perform comment updates with code changes <cit.>.
Code comments are a vital source of software documentation, helping developers to have a better understanding of source code and are beneficial to the communication between developers. However, developers sometimes neglect to update comments after changing source code, leading to obsolete or inconsistent comments. It is necessary to perform Just-In-Time (JIT) code comments updating, which aims to reduce and even avoid bad comments. To achieve this, a new approach called CUP builds a novel neural seq2seq model. Using DL techniques for such an SE task can be broken into six steps: (1) data collection, (2) data processing, (3) input construction, (4) model training, (5) model optimization, (6) model testing, and (6) model application.

Data collection: Collecting data is a key step when building and training a DL model. Different SE tasks need to process and analyze different data types, such as requirements documents for requirements analysis, bug reports for bug report summarization, source code for code clone detection or code search, etc. In this example, for updating JIT code comments, the commonly available data are method-level code change datasets with comments. Each qualified instance contains old code snippets, new code snippets, old code comments, and new code comments.

Data processing: Processing raw data involves a number of steps, including data filtering, data splitting, data cleaning, data augmentation, and data segmentation for eliminating noise in data. For JIT code comments updating, some instances are removed with unqualified comments, and no differences or empty between old and new comments. Instances containing abstract methods are deleted to reduce method mismatching.

Input construction: Since most DL models have strict requirements on the input-form, such as the input requiring to be numeric and fixed size, it is necessary to transform SE data into multi-dimensional vectors in order to use DL models. In this JIT code comment updating example, code comments and code changes can be viewed as text-based data, processed by using a token-based method. Code changes and code comments are converted into sequences so that they can be fed into their seq2seq model through data flattening and tokenization. After tokenization, old comments are converted into a token sequence. While to better represent code changes,  each change is aligned into two token sequences and construct a triple $<t_i, t_i^{'}, a_i>$ as an edit sequence to respectively record old source code, new source code, and an edit action, i.e., insert, delete, equal or replace.

Model training: DL users need to select suitable DL techniques and different datasets, construct the structure of a model, and decide model configuration, e.g., the number of layers and neural units of each layers. In our example, a seq2seq DL model was built by training an encoder-decoder LSTM, since it is good at process nature language text and token-based source code. In this model, the edit sequence of a code change and a token sequence of its old comment were fed into the input layer. To capture the relationship between the code change and the old comment, the encoder was composed of 4 layers: an embedding layer, a contextual embed layer, a co-attention layer, and a modeling layer, where each layer had its role. The decoder included 2 layers: a LSTM layer and a dense layer. The output of decoder was a new comment based on the corresponding captured relationship.

Model optimization: After model design and construction, the designed model will be trained with the training set for achieving an effective DL model. Whether a model can work depends on thousands of parameters (aka., weights), connecting neural units adjacent layers. Hence, model training is to fine-tune these weights to minimize the loss of the model. For the seq2seq model in the example, the weights in the seq2seq neural network are trained by minimizing the difference between the real new comment and the generated new comment in a supervised way.

Model testing: Generally, a training set is usually divided into two subsets of unequal sizes. The big subset is used for training the DL model, while the small one will be used for validating and testing the performance of the model when meeting new data. In this example, 20% of samples in the training set are put into the validation and test set to ensure the effectiveness of CUP.

Model application: Finally, the trained DL model can be applied to tackle practical SE tasks. In this example, the trained model leverages two distinct encoders and a co-attention layer to learn the relationships between the code change and the old comment. The LSTM-based decoder is used to generate new comments whose tokens are copied from both the new code and the old comments.

§ METHODOLOGY

We performed a systematic literature review (SLR) following  Kitchenham and Charters <cit.> and Petersen et al. <cit.>. In this section, we present details of our SLR methodology.

§.§ Research Questions

We want to analyse the history of using DL models in SE by summarizing and analyzing the relevant studies, and providing the guidelines on how to select and apply the DL techniques. To achieve this, we wanted to answer the following research questions:

* RQ1: What are the trends in the primary studies on the use of DL in SE?

* RQ2: What DL techniques have been applied to support SE tasks?

* RQ3: What key factors contribute to difficulties in training DNNs for SE tasks?

* RQ4: What types of SE tasks and which SE phases have been facilitated by DL-based approaches?

RQ1 analyzes the distribution of relevant publications that used DL in their studies since 2006 to give an overview of the trend of DL in SE. RQ2 provides a classification of different DL techniques supporting SE tasks and analyze their popularity based on their frequency of use in SE. RQ3 explores key technologies and factors that may affect the efficiency of the DNN training phase. RQ4 investigates what types of SE tasks and which SE phases have been facilitated by DNNs.

§.§ Literature search and selection

To collect DL related papers in SE, we identified a search string including several DL related terms frequently appeared in SE papers that make use of DL. We then refined the search string by checking the title and abstract of a small number of relevant papers. After that, we used logical ORs to combine these terms, and the search string is:

("deep" OR "neural" OR "Intelligence" OR "reinforcement")

Publication venues for manual search

No.     Acronym     Full name

1.     ICSE     ACM/IEEE International Conference on Software Engineering

2.     ASE      IEEE/ACM International Conference Automated Software Engineering

3.     ESEC/FSE      ACM SIGSOFT Symposium on the Foundation of Software Engineering/European Software Engineering Conference

4.     ICSME     IEEE International Conference on Software Maintenance and Evolution

5.     ICPC     IEEE International Conference on Program Comprehension

6.     ESEM     International Symposium on Empirical Software Engineering and Measurement

7.     RE     IEEE International Conference on Requirements Engineering

8.     MSR     IEEE Working Conference on Mining Software Repositories

9.     ISSTA     International Symposium on Testing and Analysis Working Conference on Mining Software Repositories

10.     SANER     IEEE International Conference on Software Analysis, Evolution and Reengineering

11.     ICST     IEEE International Conference on Software Testing, Verification and Validation

12.     ISSRE     IEEE International Symposium on Software Reliability  Engineering

13.     TSE     IEEE Transactions on Software Engineering

14.     TOSEM     ACM Transactions on Software Engineering and Methodology

15.     ESE     Empirical Software Engineering

16.     JSS     Journal of Systems and Software

17.     IST     Information and Software Systems

18.     ASEJ     Automated Software Engineering

19.     IETS     IET Software

20.     STVR     Software Testing, Verification and Reliability

21.     JSEP     Journal of Software: Evolution and Process

22.     SQJ     Software Quality Journal

We specified the range the papers are published later: 2006-July 2020. Following previous studies <cit.>, we selected 22 widely read SE journals (10) and conferences (12) listed in Table <ref> to conduct a comprehensive literature review. We run the search string on three databases (i.e., ACM digital library [https://dl.acm.org], IEEE Explore [https://ieeexplore.ieee.org], and Web of Science [http://apps.webofknowledge.com]) looking for publications in the 22 publication venues whose meta data (including title, abstract and keywords) satisfies the search string. Our search returns 655 relevant papers. After discarding duplicate papers, we applied some inclusion/exclusion criteria (presented in Section 3.3) by reading their title, abstract and keywords, and narrow the candidate set to 146 studies. After reading these 146 studies in full to ensure their relevance, we retained 142 studies.

§.§ Inclusion and Exclusion Criteria

After retrieving studies that match our search string, it is necessary to filter unqualified studies, such as studies with insufficient contents or missing information. To achieve this, we applied our inclusion and exclusion criteria to determine the quality of candidate studies for ensuring that every study we kept implemented and evaluated a full DL approaches to tackle SE tasks.

The following inclusion and exclusion criteria are used:

52  The paper must be written in English.

52  The paper must adopt DL techniques to address SE problems.

52  The length of paper must not be less than 6 pages.

56  Books, keynote records, non-published manuscripts, and grey literature are dropped.

56  If a conference paper has an extended journal version, the conference version is excluded.

§.§ Data Extraction and Collection

After removing the irrelevant and duplicated papers, we extracted and recorded the essential data and performed overall analysis for answering our four RQs. Table <ref> described the detailed information being extracted and collected from 142 primary studies, where the column $'Extracted Data Items'$ lists the related data items that would be extracted from each primary study, and the column $‘RQ’$ denotes the related research questions to be answered by the extracted data items on the right. To avoid making mistakes in data collection, two researchers extracted these data items from primary studies together and then another researcher double checked the results to make sure of the correctness of the extracted data.

Data Collection for Research Questions

RQs     Extracted data items

RQ1     Basic information of each primary study (i.e., title, publication year, authors, publication venue)

RQ1     The type of main contribution in each study (e.g., empirical study, case study, survey, or algorithm)

RQ2     DL techniques used in each study

RQ2     Whether and how the authors describe the rationale behind techniques selection

RQ3     Dataset source (e.g., industry data, open source data, or collected data)

RQ3     Data type (e.g., source code, nature language text, and pictures)

RQ3     The process that datasets are transformed into input sets suitable for DNNs

RQ3     Whether and what optimization techniques are used

RQ3     What measures are used to evaluate the DL model

RQ3     Presence / absence of replication package

RQ4     The practical problem that a SE task tries to solve

RQ4     The SE activity in which each SE task belongs

RQ4     The approach used for each SE task (e.g., regression, classification, ranking, and generation)

§ RQ1: WHAT ARE THE TRENDS IN THE PRIMARY STUDIES ON USE OF DL IN SE?

We analyzed the basic information of primary studies to comprehend the trend of DL techniques used in SE in terms of the publication date, publication venues, and main contribution types of primary studies.

§.§ Publication trends of DL techniques for SE

We analyzed the publication trends of DL-based primary studies published between 2006 and the middle of 2020. Although the concept of “Deep Learning” has been proposed in 2006 and DL techniques had been widely used in many other fields in 2009, we did not find any studies using DL to address SE tasks before 2015. Fig. <ref>(a) shows the number of relevant studies published in predefined publication venues since the middle of 2020. It can be observed that the number of publications from 2015 to 2019 shows a significant increase, with the number reaching 58 papers in 2019. In data collection, we only collect papers whose initial publication date is on July 2020 or earlier; thus, the number of relevant studies in 2020 cannot reveal the overall trend of DL in 2020. However, extrapolating from the number of primary studies in previous years, we can estimate that there may be over 65 relevant publications using various DL techniques to solve SE issues by the end of 2020.

[Number of publications per year.]

[Cumulative number of publications per year.]

Publication trends of DL-based primary studies in SE.

We also performed an analysis of the cumulative number of  publications as shown in Fig. <ref>(b). We fit the cumulative number of publications as a power function, showing the publication trend in the last five years. We can notice that the slope of the curve fitting the distribution increases substantially between 2015 and 2019, and the coefficient of determination ($R^{2}$) attains the peak value (0.99447), which indicates that the number of relevant studies using DL in SE intends to experience a strong rise in the future. Therefore, after analyzing Fig. <ref>, it can be foreseen that using DL techniques to address various SE tasks has become a prevalent trend since 2015, and huge numbers of studies will adopt DL to address further challenges of SE.

§.§ Distribution of publication venues

We reviewed 142 studies published in various publication venues, including 12 conference proceedings and symposiums as well as 10 journals, which covers most research areas in SE. Table <ref> lists the number of relevant papers published in each publication venue. 69% of publications appeared in conferences and symposiums, while only 31% of journal papers leveraged DL techniques for SE tasks. Among all conference papers, only 4 different conferences include over 10 studies using DL in SE in the last five years, i.e., SANER, ASE, ICSE, and MSR. Compared with other conference proceedings, SANER is the most popular one containing the highest number of primary study papers (22), followed by ASE (16). There are 13 and 6 relevant papers published in ICSE and FSE, respectively. Meanwhile, in all journals, TSE and IST include the highest number of relevant papers (11). Ten studies related to DL techniques were published in JSS, and 5 were published in TOSEM. Almost half of the publication venues only published not more than 5 relevant papers.

Publication Venues with DL-based Studies.

Conference venue     #Studies     Journal venue     #Studies

SANER      22       TSE         11

ASE        16       IST         11

ICSE       13       JSS         10

MSR        10       TOSEM       5

ICSME       8      ESE         2

ISSTA       7       ASEJ        2

FSE         6       IETS        1

ICST        5       STVR        1

ICPC        4        SQJ         1

RE          3

ISSRE       3

ESEM       1

[Number of primary studies published in various conference proceedings.]

[Number of primary studies published in various journals.]

Distribution of different publication venues.

We also checked the distribution of primary studies published in conferences and journals between 2015 and 2020, shown in Fig. <ref>. Fig <ref>(a) illustrates that the publication trend of various conference proceedings and symposiums has a noticeable increase from 2015 to 2019. 70.4% of conference papers were published in 2018 and 2019, while only a few different conferences or symposium venues included relevant papers between 2015 and 2017, which demonstrates a booming trend in the last few years.

Fig. <ref>(b) shows the number of primary study papers published in different journal venues. It can be seen that there is an increasing trend in the last five years, especially between 2018 and 2020. Furthermore, the relevant papers published in TSE, as one of the most popular journals, accounts for the largest proportion in 2018 and 2019; while another popular journal, IST, also makes up a large percentage in 2019 and 2020.

§.§ Types of main contributions

We summarized the main contribution of each primary study and then categorized these studies according to their main contributions into five categories, i.e., New technique or methodology, Tool, Empirical study, Case study, and User study. We gave the definition of each main contribution in Table <ref>. The main contribution of 76% primary studies was to build a novel DNN as their proposed new technique or methodology for dealing with various problems in different SE activities. 10% of relevant studies concentrated on performing assessment and empirical studies for exploring the benefits of DL towards different SE aspects, such as research on the differences between ML and DL to solve certain SE tasks, the performance of using DL to mine software repositories, applying DL in testing, etc. The main contribution of 9% was case studies. 2 primary studies (1%) that both proposed a novel methodology and evaluated the novel methodology via a user study. Therefore, the main contribution of these two studies spans across two categories, i.e., New technique or methodology and user study.

The definition of five main contributions in primary studies.

Main contribution     Definition

New technique or methodology     The study provided a solid solution or developed a novel framework to address specific SE issues.

Tool     The study implemented and published a new tool or tool demo targeting SE issues.

Empirical study     The study collected primary data and performed a quantitative and qualitative analysis on the data to explore interesting findings.

Case study     The study analyzed certain SE issues based on one or more specific cases.

User study     The study conducted a survey to investigate the attitudes of different people (e.g., developers, practitioners, users, etc) towards SE issues.

* DL has shown a booming trend in recent years

* Most of primary study papers were published between 2018 and 2020

* The number of conference papers employing DNNs for SE significantly exceeds that of journal papers.

* SANER is the conference venue publishing the most DL-based papers (22), while TSE and IST include the highest number of relevant papers among all journals (11).

* Most DL-based studies were only published in a few conference proceedings (e.g., SANER, ASE, ICSE, MSR) and journals (e.g., TSE, IST, JSS, and TOSEM).

* The main contribution of 75% primary studies is to propose a novel methodology by applying various DL techniques, while only two primary studies performed a user study to better understand users' attitudes and experience toward various DNNs used for solving specific SE tasks.

§ RQ2: WHAT DL TECHNIQUES ARE APPLIED TO SUPPORT SE TASKS?

§.§ Classification of DNNs in SE

The number of various DNNs applied in per year.

Architecture     Family      Model Name     2015     2016     2017     2018     2019     2020      Total

(157)    8*cRNN-based
model (72)
RNN       1            3      7      10     2     23

RtNN           1             1                  2

Bidirectional RNN (BRNN)                             1       1

LSTM                  3        10      16          6      35

Bi-LSTM                     1       1             2       4

Siamese LSTM                                    1      1

GRU                                 1       3       4

Bidirectional GRU                                    1       1

Recurrent Highway Network                          1         1

model (48)
CNN          2       2       13       20       6       43

Tree-based CNN (TBCNN)                           2      1      3

RCNN                                              1        1

Deep Residual Network                                 1      1

model (25)
FNN             3     1      8     7     3     22

RBFNN     1                              1

Deep Sparse FNN                            1                1

Deep MLP                            1          1

model (6)
GGNN                            4     1     5

Graph Matching Network (GMN)                             1     1

model (4)

Deep Beliefe Network (DBN)           1        1                2

HAN                                  1         1

Deep Forest                           1        1

(15)    2*cRNN-based
model (12)
RNN              1           1      6          8

LSTM                        2           2      4

model (1)     CNN                             1            1

model (2)
FNN                       1            1      2

(7)    2*cRNN-based
model (1)
GRU                                    1     1

model (1)
CNN                                    1     1

model (5)
FNN                2      1       2           5

Many sorts of DNNs have been proposed, and certain neural network architectures contain diverse DNNs with different implementations.  For instance, although LSTM and GRU are considered two different DNNs, they are both RNNs. We categorized DL-based models according to their architecture and different DNNs used. We classified the architecture of various DNNs into 3 categories: the layered architecture, AutoEncoder (AE), and Encoder-Decoder <cit.>. We provided a detailed classification of DNNs into five categories, i.e., RNN, CNN, FNN, GNN, and tailored DNN models, where tailored DNNs include the DNNs not often used in SE, e.g., DBN, HAN, etc. Table  <ref> shows the variety of different DNNs, and also lists the number of times these models have been applied in SE.

As can be seen from Table  <ref> that compared Encoder-Decoder and AutoEncoder (AE) architectures, layered based DNNs are the most popular and widely used architecture. In the layered architecture, 72 primary studies used nine different kinds of RNN-based models to solve practical SE issues, where LSTM is the most often applied RNN-based model (35), followed by standard RNN (23). The variants of LSTM, such as GRU and Bi-LSTM, are often adopted by researchers in multiple research directions. 48 primary studies employed CNN-based models, where almost 90% of studies employed CNN. FNN-based model is the third most frequently used family with 25 studies using FNNs, followed by GNN-based models and tailored models. There are 24 combined DNNs were proposed in tailored models.

15 primary studies leveraged different types of DNNs following the Encoder-Decoder architecture, where RNNs were used in 12 studies, which is much higher than the number of other models used, i.e., CNN and FNN. In the last architecture, over 70% of studies used FNN-based AEs as their proposed novel approaches; only 2 studies selected GRU and CNN to construct AEs respectively.

§.§ DL technique selection strategy

Since heterogeneous DNNs have been used for SE tasks, selecting and employing the most suitable network is a crucial factor. We scanned the relevant sections of DL technique selection in all of the selected primary studies and classified the extracted rationale into three categories.

Characteristic-based selection strategy ($S1$): The studies justified the selected techniques based on their characteristics to overcome the obstacles associated with a specific SE issue <cit.>. For instance, most of the seq2seq models were built by using RNN-based models thanks to their strong ability to analyze the sequence data.

Selection based on prior studies ($S2$): Some researchers  determined the most suitable DNN used in their studies by referring to the relevant DL techniques in the related work <cit.>. For instance, due to the good performance of CNN in the field of image processing, most studies selected CNN as the first option when the dataset contains images.

Using multiple feasible DNNs ($S3$): Though not providing any explicit rationale, some studies designed experiments for technique comparisons that demonstrated that the selected algorithms performed better than other methods. For example, some studies often selected a set of DNNs in the same SE tasks to compare their performance and picked up the best one <cit.>.

We noticed that the most commonly selection strategy is $S1$ (i.e., Characteristic-based selection strategy), accounting for 69%, nearly 3 times that of $S2$ (25%). Only 6% of primary studies adopt $S3$ to select their suitable DL algorithms.

* There are 30 different DNNs used in the selected primary studies.

* We used a classification of DL-based algorithms from two perspectives, i.e., their architectures and the families to which they belong. The architecture can be classified into three types: Layered architecture, Encoder-Decoder, and AutoEncoder (AE); the family can be classified into five categories: RNN-based, CNN-based, FNN-based, GNN-based, and Tailored models.

* Compared with Encoder-Decoder and AE, the layered architecture of DNNs is by far the most popular option in SE.

* Four specific DNNs are used in more than 20 primary studies, i.e., CNN (43), LSTM (35), RNN (23), and FNN (22), and each of them has several variants that are also often used in many SE tasks.

* We summarized three types of DNN-based model selection strategies. The majority of studies adopted $S1$ to select suitable DL algorithms. Only 6% of primary studies used $S3$ as the model selection strategy due to the heavy workload brought by $S3$.

§ RQ3: WHAT KEY FACTORS CONTRIBUTE TO DIFFICULTY WHEN TRAINING DNNS IN SE?

Since analyzing a DL architecture can provide a lot of insight, we investigated the construction process of a DL framework from three aspects:  techniques used in data processing, model optimization, evaluation, and the accessibility of primary studies.

§.§ How were datasets collected, processed, and used?

Data is one of the most important roles in the training phase. Unsuitable datasets can result in failed approaches or tools with the low performance. We focused on the data used in primary studies and conducted a comprehensive analysis on the steps of data collection, data processing, and data application.

§.§.§ What were the sources of datasets used for training DNNs?

Fig. <ref> shows the sources of datasets in the selected primary studies. It can be seen that 45% of primary studies trained DNNs by using an open-source dataset. One reason for choosing an open-source dataset is that studies are willing to pick up these datasets to evaluate the effectiveness of proposed DL-based approaches due to the existence of widely accepted datasets in certain SE issues (e.g., code clone detection, software effort/cost prediction, etc). Others are used because the datasets were applied in related previous work. Due to the lack of available and suitable datasets, 18% of primary studies constructed new datasets. Real-world datasets from industry are only used by 4% of studies.

[The sources of DL-related dataets.]

[The sources of collected datasets.]

The source of datasets used in primary study papers

33% of studies performed a series of experiments on large-scale datasets so as to verify the scalability and robustness of their models. To achieve this, many studies collected multiple small datasets from different sources. Fig. <ref>(b) describes the sources of collected datasets. As tens of thousands of developers contribute to GitHub community by uploading source code of their software artifacts, GitHub has become the most frequently used source of collected data (51%). 26% of studies collected their datasets from different systems and projects. For instance, Deshmukh et al. <cit.> collected bug reports from several bug tracking systems, i.e., issue tracking systems of Open Office, Eclipse, and Net Beans projects, as datasets to build a DL-based model for duplicated bug detection. The app store is the third-largest source (11%), followed by Stack Overflow (SO) and Youtube.

§.§.§ What are the types of SE datasets used in prior DL studies?
The datasets used in primary studies are of various data types. It is essential to analyze data types of datasets since the relationship between the type of implicit feature being extracted and the architecture has a dominating influence on model selection. We summarize the data types in primary studies and interpreted how data types determine the choice of DNNs.

Data types of datasets involved in primary studies.

Family      Data types     #Studies      Total

11*c Code-based data types
Source code      61

Software/code metric      8

Code comment     7

Defects     7

Test case     6

program screencasts      4

UI images      4

Code change     2

Code annotation     2

Pull-requests     2

Patch     1     -11*c 104

9*c Text-based data types
Bug report      9

Requirement documentation      4

configuration documentation     2

APP description     2

Software version information     2

Design documentation     1

Log information     1

Certification     1

Protocol message     1

Patch     1     -9*c 23

4*c Software repository-based data types
Q&A in SO      6

Tags in SO      5

Issues and commits     4

Pull-requests     2      -4*c 17

3*c User-based data types
User behavior      3

User review      1

Interaction traces     1      -3*c 5

We classified the data types of used datasets into four categories – code-based, text-based, software repository-based, and user-based data types. Table <ref> describes specific data in each data type. 104 primary studies collected data from source code, and most of these studies used source code directly in some important SE activities, such as software testing and maintenance. Datasets containing various metrics were employed in 8 relevant studies, followed by code comments, defects (7), and test cases (6). Whereas few studies focused on analyzing the code annotation, pull-request, and patches. 8 primary studies used a multitude of screencasts as their datasets, where 4 studies selected program screencasts to analyze developers' behavior and 4 studies researched UI images for improving the quality of APPs.

Text-based data types were the second most popular, including 13 different kinds of documentation. Bug report and requirements documentation are the two most commonly applied text-based data types in primary studies. Some types rarely appeared, such as logs, certifications, design documentation, etc.

Since software repositories, especially SO and GitHub, contain a lot of useful patterns or information, we classified the type of the information collected from these repositories into $'Software$ $repository-based$ $data$ $types'$. 12% of studies concentrated on obtaining and learning useful information and patterns by crawling related contents from SO (e.g., Q&A (questions and answers) and tags) and GitHub (e.g., issues, commits and, pull-requests).

User-based data generally contains a great deal of user information, which can promote developers to better comprehend user needs and behavior targeting different applications. Only 5 studies adopted user-based data types (i.e., user behavior, review, and interactions) to solve relevant SE tasks.

§.§.§ What input forms were datasets transformed into when training DNNs?

The inputs of DNNs need to be various forms of vectors. We found two techniques were often used to transform different source data types into vectors: One-hot encoding and Word2vec. Only 5 studies produced the input of their models by adopting the One-hot technique. We described input forms using 5 categories referring to their data types.

Token-based input: Since some studies treated source code as text, they used a simple program analysis technique to generate code tokens into sequences and transformed tokens into vectors as the input of their DL-based models. A token-based input form can be applied to source code and text-based data when processing related datasets.

Tree/graph-based input: To better comprehend the structure of source code, several studies convert source code into Abstract Syntax Trees (AST) or Control Flow Graphs (CFGs), and then generate vector sequences by traversing the nodes in each tree or graph.

Feature/metric-based input: For analyzing the characteristics of software artifacts, some studies applied datasets consisting of features or metrics extracted from different products, and thus the input form of the models proposed in these studies is software feature/metric-based vectors.

Pixel-based input: Some studies used datasets containing a large number of images and screencasts, e.g., program screencasts, UI images, etc. When preprocessing these datasets, they often broke down screencasts into pixels as an effective input form, for analyzing graph-based datasets in different SE tasks, such as bug detection, code extraction, etc.

Combined input: Many studies combined two or more data types extracted from software products to build comprehensive datasets with more information for enhancing the quality and accuracy of proposed models. For instance, Leclair et al. <cit.> proposed a novel approach for generating summaries of programs not only by analyzing their source code but also their code comments.

The various input forms of DL-based models proposed in primary studies.

Family      Input forms     #Studies      Total

2*c Token-based input
Code in tokens      17

Text in tokens      34

Code and text in tokens     13     -2*c 64

2*c Tree/graph-based input
Code in tree structure      25

Code in graph structure      4     -2*c 29

Feature-based input     feature/metric     33     33

Pixel-based input     pixel     9     9

3*c Hybrid input
Code in tree structure + text in token      4

Code features + text in token      2

Code in tree structure + features     1      -3*c 7

Table <ref> depicts the input formats of DL-based models. We can see that over 45% of studies transformed data (i.e., source code and various documentations) into the token-based input form (64), where 17 studies considered source code as texts and thus converted code into token sequences as the input of models. 13 studies used both source code and text-based materials and also constructed a token-based data structure as the input form of their proposed models. 25 studies utilize tree-based input form to analyze the source code, and only 4 studies transform source code into a graph-based structure for extracting essential information. 33 studies adopted embedding techniques to generate feature-based vectors.
Furthermore, 8 studies using image-based datasets split screencasts into pixels as the basic unit of the input form. Only 7 studies processed datasets into multiple forms.

§.§ What techniques were used to optimize and evaluate DL-based models in SE?

In the training phase, developers attempt to optimize the models in different ways for achieving good performance. In this section, we summarized the information describing the optimization methods and evaluation process, and performed an analysis on key techniques.

§.§.§ What learning algorithms are used in order to optimize the models?

The performance of DL-based models is dependent on selected optimization methods, which can systematically adjust the parameters of the DNN as training progresses.

Various optimization algorithms used in primary studies.

Out of the 142 studies analyzed, 131 identified the specific optimization method, but 11 studies did not mention what optimizers were used to adjust parameters in their work. Fig. <ref> illustrated the frequency of the use of 20 optimization methods used in all primary studies. We see that 6 optimizers were used in no less than 5 studies, where Adam optimizer is the most commonly used optimization method. Stochastic gradient descent (SGD) and gradient descent (GD) are also popular optimizers, which were used by 21 and 15 studies respectively, followed by back-propagation (14) and fine-tuning (11). Besides, some optimization methods are not often used, such as Adagrad and Adadelta.

§.§.§ What methods were used to alleviate the impact of Overfitting?

One major problem associated with applying any type of learning algorithm is overfitting. Overfitting is the phenomenon of a DNN learning to fit the noise in the training data extremely well, yet not being able to generalize to unseen data, which is not a good approximation of the true target function that the algorithm is looking forward to learn. We describe 9 general ways to combat overfitting <cit.> by considering 3 aspects: data processing, model construction as well as model training, and then analyzed the usage distribution of these methods in relevant studies.

Cross-validation: A cross-validation algorithm can split a dataset into k groups (k-fold cross-validation). Researchers often leave one group to be the validation set and the others as the training set. This process will be repeated until each group has been used as the validation set. Since a remaining subset of data is new towards the training process of DL-based models, the algorithm can only rely on the learned knowledge from other groups of data to predict the results of the remaining subset, preventing overfitting.

Feature selection:  Overfitting can be prevented by selecting several of the most essential features for training DL-based models can effectively avoid overfitting. Therefore, researchers can pick up some key features by using feature selection methods, train individual models for these features, and evaluate the generalization capabilities of models. For instance, Pooling is a typical technique to prevent overfitting since pooling can reserve main features while reducing the number of parameters and the amount of computation, and improve the generalization ability of the model.

Regularization: Regularization is a technique to constrain the network from learning a model that is too complex, which therefore can avoid overfitting. A penalty term would be added in the cost function to push the estimated coefficients towards zero by applying L1 or L2 regularization.

Dropout: By applying dropout, a form of regularization, to the layers of DNNs, a part of neurons were ignored with a set probability. Therefore, dropout can reduce interdependent learning among units to avoid overfitting.

Data augmentation: A larger dataset can reduce the chance of overfitting. Data augmentation is a good way to artificially increase the size of our dataset for improving the performance of a DNN  when the scale of data was constrained due to difficult to gather more data. For example, many studies performed various image transformations to the image dataset (e.g., flipping, rotating, rescaling, shifting) for enlarging data size in the image classification task.

Early stopping: Early stopping is an effective method for avoiding overfitting by truncating the number of iterations, that is, stopping iterations before DL-based models converge on the training dataset to eliminate the impact on overfitting.

Data balancing: With imbalanced data, DL-based models are likely to occur the overfitting problem since models will learn imbalanced knowledge with a disproportionate ratio of observations in each class. Using some data balancing techniques can effectively alleviate the impact on models' performance caused by overfitting.

Ensembling: Ensembles are a set of machine learning methods for combining predictions from multiple separate models. For instance, Bagging as an ensemble learner can reduce the chance of overfitting complex models by training a large number of "strong" learners in parallel without restriction.

The distribution of various overfitting techniques used in primary studies

Fig. <ref> illustrates the distribution of the techniques used for combating overfitting problems. Cross-validation has been used frequently among the selected studies to prevent overfitting; it is used in 40 studies, followed by pooling (25). Regularization and dropout are the third most popular techniques used in 20 studies. There are 8 studies that prevent overfitting by enlarging the scale of data, such as using a large-scale dataset, combining multiple datasets, and using different data augmentation techniques. 6 studies used early stopping and 5 selected data balancing to combat the overfitting problem. Ensembling is the least frequently used one (1) compared with other techniques (1). Furthermore, among all primary studies, 4 studies used several new algorithms proposed by some studies to solve overfitting.
We analyzed which factors may have an impact on the overfitting technique selection. We noticed that the techniques used for combating overfitting have no strong association with either data types or input forms. However, there is a special relationship between model selection and these techniques. Most of the studies that adopted CNNs to address specific SE tasks selected pooling as their first choice for preventing the overfitting problem.

§.§.§ What measures are used to evaluate DL-based models?

Accessing appropriate benchmarks is a crucial part of evaluating any DL-based models in SE. We also explored the frequent metrics used to measure the performance of DL-based models applied to respective SE tasks.

Metrics used for evaluation.

Metrics     #Studies

Precision@k     69

Recall@k     59

F1@k     53

Accuracy     26

Mean Reciprocal Rank (MRR)     15

BLEU     13

Running time     13

AUC     11

Mean Average Precision (MAP)     7

Matthews Correlation Coefficient (MCC)     5

P-value     4

METEOR     4

ROC     4

ROUGE     4

Mean Absolute Error (MAE)     4

SuccessRate@k     3

Cliff’s Delta     3

Coverage     3

Bal (Balance)     3

Standardized Accuracy (SA)     3

Others     11

Table <ref> summarizes the commonly used evaluation metrics in the primary studies, used in no less than 3 studies. Precision, recall, F1-measure, and accuracy are widely accepted metrics for evaluating the performance of DL-based models. Some studies adopted MRR and BLEU as evaluation metrics in their work, potentially indicating that many studies focused on addressing ranking and translation tasks by training various DNNs. Another interesting observation from Table <ref> is that running time is selected as a performance indicator by a set of studies, which does not occur frequently when using non-learning techniques. This is because that learning algorithms, especially DNNs, require more time during their construction, training, and testing phases due to the high complexity of these networks (e.g., numerous types of layers, a great many neurons, and different optimization methods). Also, almost half of metrics are not commonly used in relevant studies, which are only used in 3 or 4 studies (e.g., P-value, ROC, ROUGE, Coverage, Balance, etc.) and thus these metrics can reflect their respective characteristics of different SE tasks.

§.§ Accessibility of DL-based models used in primary studies.

We checked whether the source code of DL-based models is accessible for supporting replicability and reproducibility. 53 studies provided the replication packages of their DL-based models, only accounting for 37.3% of all primary studies. 89 studies proposed novel DL-based models without publicly available source code, making it difficult for other researchers to reproduce their results; some of these studies only disclosed their datasets.
Based on this observation, obtaining open-source code of DNNs is still one of the challenges in SE because many factors may result in never realizing the replicability and reproducibility of DL application, e.g., data accessibility, source code accessibility, different data preprocessing techniques, optimization methods, and model selection methods. Therefore, we recommend future DL studies to release replication packages.

* Most datasets are available online and 33% of datasets consist of multiple small-scale ones collected from GitHub, software systems and projects, and some software repositories.

* 5 different data types are used in the primary studies, i.e., code-based, text-based, software repository-based, graph-based, and user-based data types, where code-based and text-based types are the two main data types being used in 82.3% of primary studies.

* Most studies parse source code into token, tree, or graph structures, or extract features from programs. When the raw datasets are documentation, studies would convert them into token-based vectors as the input form of their models.

* We observed that Adam is the most popular optimization algorithm being used in 32 studies, followed by SGD and GD, and several variants of Adam are still commonly used in SE. There are also some well-known optimization algorithms used in primary studies, such as back propagation, fin-tuning, and hyperparameter optimizer.

* 9 different ways are used for combating the overfitting problem. 4 techniques were widely used – cross-validation, feature selection, regularization, and dropout. We found that studies applying CNNs would choose the pooling method to prevent overfitting with high probability, and  data type and input form does not influence technique selection.

* There are over 20 different metrics used to verify the performance of DL-based models. Precision, recall, F1-measure, and accuracy were used commonly in primary studies.

* Only 53 studies provided a public link of their models in their papers and yet 62.7% of proposed models are difficult to be reproduced since the source code is unavailable.

§ RQ4: WHAT TYPES OF SE TASKS AND WHICH SE PHASES HAVE BEEN FACILITATED BY DL-BASED APPROACHES?

In this section, we first categorise a variety of SE tasks into six SE activities referring to Software Engineering Body of Knowledge <cit.>, i.e., software requirements, software design, software implementation, software testing and debugging, software maintenance, and software management. We then analyze the distribution of DL-based studies for different SE activities. We present a short description of each primary study, including the specific SE issue each study focused on, which and how DL techniques are used, and the performance of each DL model used.

§.§ Distribution of DL techniques in different SE activities

The distribution of DL techniques in Different SE activities.

The classification of primary studies.

We analysed which SE activities and specific SE tasks each selected primary study tried to solve. As shown in Fig. <ref>, the largest number of primary studies focused on addressing SE issues in software maintenance (39%). 36% of studies researched software testing and debugging. Software management was the topic of 11% of primary studies, followed by software implementation (9%). Software design and modeling (3%) and software requirements (2%) are addressed in very few studies.

We classified all primary studies into four categories based on the types of their SE tasks, i.e., the regression task, classification task, ranking task, and generation task. Fig. <ref> describes the distribution of different task types where DL techniques were applied. Classification and generation tasks account for almost 80% of primary studies, where classification is the most frequent task (54%).  13% of studies belong to the regression task and the output of their proposed models is a prediction value, such as effort cost prediction. In SE, some studies adopted DL to concentrate on a ranking task, accounting for 11% of all studies.

The specific research topics where DL techniques are often applied.

SE activities     specific research topics     #Studies      Total

1*c Software design

Source code representation     5     -1*c 5

2*c Software implementation

Code search      5

Code programming     4     -2*c 9

6*c Software testing and debugging
Defect prediction      11

Bug localization      7

Application testing     7

Program analysis     5

Test case generation     4

Reverse execution     3     -6*c 37

5*c Software maintenance
Code clone detection      11

Program repair     6

Code comment generation     4

Software quality evaluation     4

Source code representation     4     -5*c 28

2*c Software management

Software repository mining      19

Effort cost prediction     6      -2*c 25

We summarized a set of research topics in which DL was engaged. Table <ref> lists the research topics containing no less than three related studies. Software testing and debugging, as the most prevalent SE activity, has 37 primary studies in six topics. The most popular study is defect prediction (11 studies), followed by bug localization (7) and application testing (7). Software maintenance, as the second most popular activities, involves five research topics with 28 relevant studies, where code clone detection is the most popular research topic. Two SE activities, software implementation and software management, both contain two important research topics, where 19 primary studies mined software repositories by training DNNs, 6 studies estimated development cost, and 5 studies applied DL for code search. Software design and modeling only involve one popular topic, i.e., source code representation/modeling. There are no topics with more than three studies using DL techniques in software requirements.

§.§ Software requirements

§.§.§ Requirements analysis

A number of natural language-based documents that describe users' specific needs or services of a software product can be referred to as user requirements (aka, use cases, or actions) <cit.>.
Extracting use cases of a product from a large volume of textual requirement documentation is a common but labor-intensive task.  Since the manual mapping system states between requirements and simulation is a time-consuming task, Pudlitz et al. <cit.> proposed a self-trained Named-entity Recognition model combined with Bi-LSTM and CNN to extract the system states from requirements specification, working to reduce labor cost when linking the state extracted from requirements to the simulation signal.

§.§.§ requirement validation

The requirements specification may be subject to validation and verification procedures, ensuring that developers have understood the requirements and the requirements conform to company standards. Winkler et al. <cit.> present an automatic approach to identify and determine the method for requirement validation. They predefined six possible verification methods and trained a CNN model as a multiclass and multilabel classifier to classify requirements with respect to their potential verification methods. The mixed results revealed that the imperfect training data impacted the performance of their classifier, but it still achieved good results on the testing data.

§.§ Software design

§.§.§ Software design patterns detection

UI design is an essential component of software development, yet previous studies cannot reliably identify relevant high-fidelity UI designs from large-scale datasets. Martín et al. <cit.> proposed a DL-based search engine to detect UI designs in various software products. The core idea of this search engine is to build a CNN-based wireframe image autoencoder to automatically generate labels on a large-scale dataset of Android UI designs. After manual evaluation of experimental results, they confirmed that their search engine achieved superior performance compared with image-similarity-based and component-matching-based methods. Thaller et al. <cit.> proposed a flexible human- and machine-comprehensible software representation algorithm, namely Feature Maps. They first extracted subtrees from the system's abstract semantic graph (ASG). Then their algorithm pressed the high-dimensional and inhomogeneous vector space of these micro-structures into a feature map. Finally,  they adopted a classical machine learning model and a DL model (i.e., Random Forest and CNN) to identify instances of design patterns in source code. Their evaluation  suggested that Feature Map is an effective software representation method, revealing important information hidden in the source code.

§.§.§ GUI modeling

Chen et al. <cit.> proposed a neural machine translator to learn a crowd-scale knowledge of user interfaces (UI). Their generative tool encoded the spatial layouts of visual features learned from a UI image and learned to generate its graphical user interface (GUI) skeleton by combining RNN and CNN models. Its performance had been verified on the large-scale UI data from real-world applications. Moran et al. <cit.> proposed a strategy to facilitate developers automate the process of prototyping of GUIs in 3 steps: detection, classification, and assembly. First, they used computer vision techniques to detect logical components of a GUI from mock-up metadata. They then trained CNNs to category GUI-components into domain-specific types. Finally, a KNN algorithm was applied to generate a suitable hierarchical GUI structure to assemble prototype applications. Their evaluation  achieved an average GUI-component classification accuracy of 91%.

§.§ Software implementation

§.§.§ Code search

Gu et al. <cit.> proposed DeepAPI, a DL-based approach to generate functional API usage sequences for a given natural language-based user query by using an attention-based GRU Encoder-Decoder. DeepAPI first encoded the user query into a fixed-length context vector and produced the API sequence according to the context vector. It also enhanced their model by considering the importance of individual APIs. To evaluate its effectiveness, they empirically evaluated their approach on 7 million code snippets. Gu et al. <cit.> proposed a code search tool, DeepCS by using a novel DNN model. They considered code snippets as well as natural language descriptions, and then embedded them into a high-dimensional unified vector representation. Thus, DeepCS gave the relevant code snippets by retrieving the vector of the corresponding natural language query. They evaluated DeepCS with a large-scale dataset collected from GitHub.

Recently, several proposals use DL techniques for code search by embedding source code and given queries into vector space and calculating their semantic correlation <cit.>. Cambronero et al. <cit.> noticed that multiple approaches existed for searching related code snippets applied unsupervised techniques, while some adopted supervised ones to embed source code and queries for code search. They defined 3 RQs to investigate whether using supervised techniques is an effective way for code search and what types of DNNs and training corpus to use for this supervision. To understand these tradeoffs quantitatively, They selected and implemented four state-of-the-art code search techniques. They found that UNIF outperformed CODEnn and SCS models based on their benchmarks and suggested evaluating simple components first before integrating a complicated one.
Wan et al. <cit.> addressed the lack of analysis of structured features and inability to interpret search results in existing code search works. To address these issues, they presented a new approach MMAN, which adopted three different DNNs (i.e., LSTM, Tree-LSTM, and GGNN (Gated Graph Neural Network)) to analyze both shallow features and the semantic features in ASTs, and control-flow graphs (CFGs) of source code. The final results on a large-scale real-world dataset demonstrated that MMAN accurately provided code snippets. Huang et al. <cit.> proposed an attention-based code-description representation learning model (CDRL) to refine the general DL-based code search approaches. They only picked up description terms and semantically related code tokens to embed a given query and its code snippet into a shared vector space.

§.§.§ Programming

Gao et al. <cit.> introduced an attention-based Encoder-Decoder framework to directly generate sensible method names by considering the relationship between the functional descriptions and method names. To evaluate their model, experiments were performed on large-scale datasets for handling the cold-start problem, and the model achieved significant improvement over baselines.  Alahmadi et al. <cit.> applied a CNN model to automatically identify the exact location of code in images for reducing the noise. They extracted 450 screencasts covering C#, Java, and Python programming languages to evaluate their model, and the final result showed that the accuracy of their model achieved 94%. Wang et al. <cit.> proposed a Neural Network-based translation model to address the domain-specific rare word problem when carrying out software localization. They trained an RNN encoder-decoder framework and enhanced it by adding linguistic information. Nguyen et al. <cit.> proposed a DL-based language model, Dnn4C, which augmented the local context of lexical code elements with both syntactic and type contexts by using an FNN model. Empirical evaluation on code completion showed that Dnn4C improved accuracy by 24.9%  on average over four baseline approaches.

§.§ Software testing and debugging

§.§.§ Defect prediction

Defect prediction is the most extensive and active research topic in use of DL techniques in software maintenance. Almost 30% of primary studies focused on identifying defects <cit.>.

Metrics-based defect prediction.
Metrics or features extracted from a software product can give a vivid description of its running state, and thus it is easy for researchers and participants to use these software metrics for defect prediction. Tong et al. <cit.> proposed a novel two-stage approach, SDAEsT, which is build based on stacked denoising autoencoders (SDAEs) and a two-stage ensemble (TSE) learning strategy. Specifically, in the first stage, they used SDAEs to extract more robust and representative features. To mitigate the impact on the class imbalance problem and eliminate the overfitting problem, they propose TSE learning strategy as the second phase. They evaluated their work using 12 open-source defect datasets.
Xu et al. <cit.> built an FNN model with a new hybrid loss function to learn the intrinsic structure and more discriminative features hidden behind the programs. Previous studies obtained process metrics throughout analyzing change histories manually and often ignored the sequence information of changes during software evaluation. For better utilization of such sequence data, Wen et al. <cit.> built an RNN model to encode features from change sequences. They considered defect prediction as to the sequence labeling problem and performed fine-grained change analysis to extract six categories of change sequences, covering different aspects of software changes. Their evaluation on 10 public datasets showed that their approach achieved high performance in terms of F1-measure and AUC. To address the same problem, Liu et al. <cit.> proposed to obtain the Historical Version Sequence of Metrics (HVSM) from various software versions as defect predictors and leveraged RNN to detect defects. Barbez et al. <cit.> analyzed and mined the version control system to achieve historical values of structural code metrics. They then trained a CNN based classifier, CAME, to infer the anti-patterns in the software products.

Semantic-based defect prediction.

Wang et al. <cit.> leveraged Deep Belief Network (DBN) to automatically learn semantic features from token vectors extracted from programs' ASTs, compared to most previous works that use manual feature specification. They evaluated their approach on file-level defect prediction tasks (within-project and cross-project) and change-level defect prediction tasks (within-project and cross-project) respectively. The evaluation results confirmed that DBN-based semantic features significantly outperformed the previous defect prediction based on traditional features in terms of F1-measure. Similarly, Dam et al. <cit.> used a tree-based LSTM network, which can directly match with the AST of programs for capturing multiple levels of the semantics of source code.

Just-In-Time (JIT) defect prediction.
Hoang et al. <cit.> presented an end-to-end DL-based framework, DeepJIT, for change-level defect prediction, or Just-In-Time (JIT) defect prediction. DeepJIT automatically extracted features from code changes and commit messages, and trained a CNN model to analyze them for defect prediction. The evaluation experiments on two popular projects showed that DeepJIT achieved improvements over 10% for two open-source datasets in terms of AUC.

§.§.§ Bug detection

Wan et al. <cit.> implemented a Supervised Representation Learning Approach (SRLA) based on an autoencoder with double encoding-layers to conduct cross-project Aging-Related Bugs (ARBs) prediction. They compared SRLA with the state-of-the-art approach, TLAP, to prove the effectiveness of SRLA. Wang et al. <cit.> present a novel framework, Textout, for detecting text-layout bugs in mobile apps. They formulated layout bug prediction as a classification issue and addressed this problem with image processing and deep learning techniques. Thus, they designed a specifically-tailored text detection method and trained a CNN classifier to identify text-layout bugs automatically. Textout achieved an AUC of 95.6% on the dataset with 33,102 text-region images from real-world apps. Source code is composed of different terms and identifiers written in natural language with rich semantic information. Based on this intuition, Li et al. <cit.> trained a DL-based model to detect suspicious return statements. They used a CNN to determine whether a given return statement in source code matched its method signature. To reduce the impact of the lack of negative training data, they converted the correct return statements in real-world projects to incorrect ones. Li et al. <cit.> proposed an AIOps solution for identifying node failures for an ultra-large-scale cloud computing platform at Alibaba.

§.§.§ Vulnerability detection

Dam et al. <cit.> described a novel approach for vulnerability detection, which automatically captured both syntactic and semantic features of source code. The experiments on 18 Android applications and Firefox applications indicated that the effectiveness of their approach for within-project prediction and cross-project prediction. Tian et al. <cit.> proposed to learn the fine-grained representation of binary programs and trained a Gated Recurrent Unit (BGRU) network model for intelligent vulnerability detection.  Han et al. <cit.> trained a shallow CNN model to capture discriminative features of vulnerability description and exploit these features for predicting the multi-class severity level of software vulnerabilities. They collected large-scale data from the Common Vulnerabilities and Exposures (CVE) database to test their approach.

§.§.§ Bug localization

To locate buggy files, Lam et al. <cit.> built an autoencoder in combination with Information Retrieval (IR) technique, rVSM, which learned the relationship between the terms used in bug reports and code tokens in software projects. Some studies proposed to exploit CNN in the bug localization task <cit.>. Zhang et al. <cit.> proposed CNNFL, which localized suspicious statements in source code responsible for failures based on CNN. They trained this model with test cases and tested it by evaluating the suspiciousness of statements. Huo et al. <cit.> present a deep transform learning algorithm, TRANP-CNN, for cross-project bug localization by training a CNN model to extract transferable semantic features from source code. Xiao et al. <cit.> used the word-embedding technique to retain the semantic information of the bug report and source code and enhanced CNN to consider bug-fixing frequency and recency in company with feature detection techniques for bug localization. Li et al. <cit.> proposed a novel approach, DeepFL, to learn latent features for precise fault localization, adopting an RNN model. The evaluation on the benchmark dataset, Defects4J, described that DeepFL significantly outperformed state-of-the-art approaches, i.e., TraPT/FLUCCS. Standard code parsers are of little help, typically resolving syntax errors and their precise location poorly. Santos et al. <cit.> proposed a new methodology for locating syntax errors and provided some suggestions for possible changes for fixing these errors. Their methodology was of practical use to all developers but especially useful to novices frustrated with incomprehensible syntax errors.

§.§.§ Test case generation

Liu et al. <cit.> proposed a novel approach to automatically generate the most relevant text of test cases based on the context of use cases for mobile testing. Koo et al. <cit.> implemented a novel approach, PySE1, to generate the test case. PySE1 tackled the limitations of symbolic execution schemes by proposing a DL-based reinforcement learning algorithm to improve the branch policy for exploring the worse case program execution.
Zhao et al. <cit.> trained a DL-based model that combines LSTM and FNN to learn the structures of protocol frames and deal with the temporal features of stateful protocols for carrying out security checks on industrial network and generating fake but plausible messages as test cases. Liu et al. <cit.> proposed a deep natural language processing tool, DeepSQLi, to produce test cases used to detect SQLi vulnerabilities. They trained an encoder-decoder based seq2seq model to capture the semantic knowledge of SQLi attacks and used it to transform user inputs into new test cases.

§.§.§ Program analysis

Program analysis refers to any examination of source code or program executions that attempt to find patterns or anomalies thought to reveal specific behaviors of the software.

Static analysis.

In Android mobile operating systems, applications communicate with each other a message-passing system, namely, Inter-Component Communication (ICC). Many serious security vulnerabilities may occur owing to misuse and abuse of communication links, i.e., ICCs. Zhao et al. <cit.> presented a new approach to determine communication links between Android applications. They augmented static analysis with DL techniques by encoding data types of the links and calculating the probability of the link existence. To reduce the number of false alarms, Lee et al. <cit.> trained a CNN model as an automated classifier to learn the lexical patterns in the parts of source code for detecting and classifying false alarms.
Due to the impact of high false-positive rates on static analysis tools, Koc et al. <cit.> performed a comparative empirical study of 4 learning techniques (i.e., hand-engineered features, a bag of words, RNNs, and GNNs) for classifying false positives, using multiple ground-truth program sets. Their results suggest that RNNs outperform the other studied techniques with high accuracy.

Type inference.
Helledoorn et al. <cit.> developed an automated framework, DeepTyper, a DL-based model to analyze JavaScript language and learn types that naturally occurred in certain contexts. It then provided a type of suggestion when the type checker cannot infer the types of code elements, such as variables and functions. Malik et al. <cit.> formulated the problem of inferring Javascript function types as a classification task. Thus, they trained a LSTM-based neural model to learn patterns and features from code annotated programs collected from real-world projects, and then predicted the function types of unannotated code by leveraging the learned knowledge.

§.§.§ Testing techniques

Many studies focus on new methods to perform testing, such as for apps <cit.>, games <cit.>, and other software systems <cit.>. There are also some studies using well-known testing techniques (e.g., fuzzing <cit.> and mutation testing <cit.>) for improving the quality of software artifacts.

Zheng et al. <cit.> conducted a comprehensive analysis of 1,349 real bugs and proposed Wuji, a game testing framework, which used an FNN model to perform automatic game testing. Ben et al. <cit.> also used the FNN to test Advanced Driver Assistance Systems (ADAS). They leveraged a multi-objective search to guide testing towards the most critical behaviors of ADAS. Pan et al. <cit.> present Q-testing, a reinforcement learning-based approach, benefiting from both random and model-based approaches to automated testing of Android applications. Mao et al. <cit.> performed an extensive study on the effectiveness and efficiency of the promising PMT technique. They also complemented the original PMT work by considering more features and the powerful deep learning models to speed up this process of generating the huge number of mutants.

Godefroid et al. <cit.> used DL-based statistical machine-learning techniques to automatically generate input syntax suitable for input fuzzing.  Cummins et al. <cit.> introduced DeepSmith, a novel LSTM-based approach, for reducing the development task when using Fuzzers to discover bugs in compilers. They accelerated compiler validation through the inference of generative models for compiler inputs, and then applied DeepSmith to automatically generate tens of thousands of realistic programs. Finally, they constructed differential testing methodologies on these generated programs for exposing bugs in compilers.

§.§.§ Reverse execution

A decompiler is a tool to reverse the compilation process for examining binaries.
Lacomis et al. <cit.> introduced a novel probabilistic technique, namely Decompiled Identifier Renaming Engine (DIRE), which utilized both lexical and structural information recovered by the decompiler for variable name recovery. They also present a technique for generating corpora suitable for training and evaluating models of decompiled code renaming. Although reverse execution is an effective method to diagnose the root cause of software crashes, some inherent challenges may influence its performance. To address this issue, Mu et al. <cit.> present a novel DNN, which significantly increased the burden of doing hypothesis testing to track down non-alias relation in binary code and improved memory alias resolution. To achieve this, they first employed an RNN to learn the binary code pattern pertaining to memory access and then inferred the memory region accessed by memory references. Katz et al. <cit.> noticed that the source code generated by decompilation techniques are difficult for developers to read and understand. To narrow the differences between human-written code and decompiled code, they trained a non-language-specific RNN model to learn properties and patterns in source code for decompiling binary code.

§.§ Software maintenance

There are a lot of studies contributing to increasing maintenance efficiency, such as improving source code, logging information, software energy consumption, etc.  <cit.>.

§.§.§ Code clone detection

Code clone detection is a very popular SE task in software maintenance using DL, with around 20% of primary studies concentrating on this research topic.

RNN-based code clone detection. Most studies use RNNs including RtNN <cit.>, RvNN <cit.>, and LSTM <cit.> to identify clones in source code. White et al. <cit.> proposed a novel code clone detector by combining two different RNNs, i.e., RtNN and RvNN, for automatically linking patterns mined at the lexical level with patterns mined at the syntactic level. They evaluated their DL-based approach based on file- and function-level. Gao et al. <cit.> first transformed source code into AST by parsing programs and then adopted a skip-gram language model to generate vector representation of ASTs. After that, they used the standard RNN model to find code clones from java projects. Buch et al. <cit.> introduced a tree-based code clone detection approach, and traversed ASTs to form data sequences as the input of LSTM.  Perez et al. <cit.> also used LSTM to learn from ASTs, and then calculated the similarities between ASTs written in Java and Python for identifying cross-language clones. Since source code can be represented at different levels of abstraction: identifiers, Abstract Syntax Trees, Control Flow Graphs, and Bytecode, Tufano et al. <cit.> conducted a series of experiments to demonstrate how DL can automatically learn code similarities from different representations.

FNN-based code clone detection. Some studies adopted FNNs for the code clone detection task <cit.>. Li et al. <cit.> implemented a DL-based classifier, CClearner, for detecting function-level code clones by training an FNN. Compared with the approaches not using DL, CClearner achieved competitive clone detection effectiveness with a low time cost. Zhao et al. <cit.> introduced a novel clone detection approach, which encoded data flow and control flow and into a semantic matrix and designed an FNN structure to measure the functional similarity between semantic representation of each code segment. Nafi et al. <cit.> proposed a cross-language clone detector without extensive processing of the source code and without the need to generate an intermediate representation. They trained an FNN model, which can learn different syntactic features of source code across programming languages and identified clones by comparing the similarity of features.

For detecting Type-4 code clones, Yu et al. <cit.> present a new approach that uses tree-based convolution to detect semantic clones, by capturing both the structural information of a code fragment from its AST and lexical information from code tokens. They also addressed the limitation of an unlimited vocabulary of tokens and models.
Wang et al. <cit.> developed a novel graph-based program representation method, flow-augmented abstract syntax tree (FA-AST), to better capture and leverage control and data flow information. FA-AST augmented original ASTs with explicit control and data flow edges and then adopted two different GNN models (i.e., gated graph neural network (GGNN) and graph matching network (GMN)) to measure the similarity of various code pairs. To effectively capture syntax and semantic information from programs to detect semantic clones, Fang et al. <cit.> adopted fusion embedding techniques to learn hidden syntactic and semantic features by building a novel joint code representation. They also proposed a new granularity for functional code clone detection called caller-callee method relationships. Finally, they trained a supervised deep learning model to find semantic clones.

§.§.§ Code comment generation

Hu et al. <cit.> present a new approach that can automatically generate code comments for Java code to help developers better understand the functionality of code segments. They trained a LSTM-based framework to learn the program structure for better comments generation.
The context information of the source code was not used and analyzed in previous automated comment summarization techniques. Ciurumelea et al. <cit.> proposed a semi-automated system to generate code comments by using LSTM. Zhou et al. <cit.> combined program analysis and natural language processing to build a Dl-based seq2seq model to generate Java code comments.
To generate code summarization, Leclair et al. <cit.> proposed a DL-based model combining texts from code with code structure from an AST. They processed data source as a separate input to reduce the entire dependence on internal documentation of code.
Wan et al. <cit.> noticed that most of the previous work used the Encoder-Decoder architecture to generate code summaries, which omitted the tree structure of source code and introduced some bias when decoding code sequences. To solve these problems, they trained a deep reinforcement learning framework  that incorporated an abstract syntax tree structure as well as sequential content of code snippets. They trained this DNN by adopting an advantage reward composed of BLEU metric.

§.§.§ Program repair

Bhatia et al. <cit.> proposed a novel neuro-symbolic approach combining DL techniques with constraint-based reasoning for automatically correcting programs with errors. Specifically, they trained an RNN model to perform syntax repairs for the buggy programs and ensured functional correctness by using constraint-based techniques.
Through evaluation, their approach was able to repair syntax errors in 60% of submissions and identified functionally correct repairs for 24% submissions.
Tufano et al. <cit.> proposed to leverage the proliferation of software development histories to fix common programming bugs. They used the Encoder-Decoder framework to translate buggy code into its fixed version after generating the abstract representation of buggy programs and their fixed code.
White et al. <cit.> trained an autoencoder framework to reason about the repair ingredients (i.e., the code reused to craft a patch). They prioritized and transformed suspicious statements and elements in the code repository for patch generation by calculating code similarities. Lutellier et al. <cit.> present a new automated generate-and-validate program repair approach, CoCoNuT, which trained multiple models to extract hierarchical features and model source code at different granularity levels (e.g., statement and function level) and then constructed a CNN model to fix different program bugs. Liu et al. <cit.> proposed an automated approach for detecting and refactoring inconsistent method names by using Paragraph Vector and a CNN.
Ni et al. <cit.> exploited the bug fixes of historical bugs to classify bugs into their cause categories based on the intuition that historical information may reflect the bug causes. They first defined the code-related bug classification criterion from the perspective of the cause of bugs and generated ASTs from diff source code to construct fixed trees. Then, they trained Tree-based Convolutional Neural Network (TBCNN) to represent each fixed tree and classified bugs into their cause categories according to the relationship between bug fixes and bug causes.

§.§.§ Source code representation

White et al. <cit.> conducted an empirical study to adopt DL in software language modeling and highlight the fundamental differences between state-of-the-practice software language models and DL models. Their intuition is that the representation power of the abstractions is the key element of improving the quality of software language models. Therefore, the goal of this study was to improve the quality of the underlying abstractions by using Neural Network Language Models (i.e., Feed-forward neural networks (FNN), RNN) for numerous SE issues. They pinpointed that DL had a strong capability to model semantics and consider rich contexts, allowing it performed better at source code modeling. They evaluated these DL-based language models at a real SE task (i.e., code suggestion). Their evaluation results suggested that their model significantly outperformed the traditional language models on 16,221 Java projects.

Hussain et al. <cit.> introduced a gated recurrent unit-based model, namely CodeGRU, to model source code by capturing its contextual, syntactical, and structural dependencies. The key innovation of their approach was to performing simple program analysis for capturing the source code context and further employed GRU to learn variable size context while modeling source code. They evaluated CodeGRU on several open-source java projects, and the experimental results verified that the approach alleviated the out of vocabulary issue. Using abstract syntax tree (AST)-based DNNs may induce a long-term dependency problem due to the large size of ASTs. To address this problem, Zhang et al. <cit.> present an advanced AST-based Neural Network (ASTNN) for source code representation. The advanced ASTNN cut each entire AST into a set of small statement trees, and transform these subtrees into vectors by capturing the lexical and syntactical knowledge of statement trees. They applied a bidirectional RNN model to produce the vector representation of a code snippet. They used ASTNN to detect code clones and classify source code for evaluating its performance. Gill et al. <cit.> introduced a lightweight framework, ThermoSim, to simulate the thermal behavior of computing nodes in the Cloud Data Center (CDC) and measure the effect of temperatures on key performance parameters. They extended the previous framework, i.e., the CloudSim toolkit by presenting an RNN-based temperature predictor for helping to analyze the performance of some key parameters. The final results demonstrated that ThermoSim accurately modeled and simulated the thermal behavior of a CDC in terms of energy consumption, time, cost, and memory usage.

§.§.§ Code classification.

Bui et al. <cit.> described a framework of Bi-NN that built a neural network on top of two underlying sub-networks, each of which encoded syntax and semantics of code in a language.  Bi-NN was trained with bilateral programs that implement the same algorithms and/or data structures in different languages and then be applied to recognize algorithm classes across languages. Software categorization is the task of organizing software into groups that broadly describe the behavior of the software. However, previous studies suffered very large performance penalties when classifying source code and code comments only. Leclair et al. <cit.> proposed a set of adaptations to a state-of-the-art neural classification algorithm and conducted two evaluations.

§.§.§ Code smell detection.

Fakhoury et al. <cit.> reported their experience in building an automatic linguistic anti-pattern detection using DNNs. They trained several traditional machine learning and DNNs to identify linguistic anti-patterns.  A big challenge for DL-based code smell detection is the lack of a large number of labeled datasets, and thus Liu et al. <cit.> present a DL-based approach to automatically generating labeled training data for DL-based classifiers. They applied their approach to detecting four common and well-known code smells, i.e., feature envy, long method, large class, and misplaced class.

§.§.§ Self-Admitted Technical Debt (SATD) detection.

Technical debt (TD) is a metaphor to reflect the tradeoff developers make between short term benefits and long term stability. Self-admitted technical debt (SATD), a variant of TD, has been proposed to identify debt that is intentionally introduced during SDLC. Ren et al. <cit.> proposed a CNN-based approach to determine code comments as SATD or non-SATD. They exploited the computational structure of CNNs to identify key phrases and patterns in code comments that are most relevant to SATD for improving the explainability of our model’s prediction results. Zampetti et al. <cit.> proposed to automatically recommend SATD removal strategies by building a multi-level classifier on a curated dataset of SATD removal patterns. Their strategy was capable of recommending six SATD removal patterns, i.e., changing API calls, conditionals, method signatures, exception handling, return statements, or telling that a more complex change is needed.

§.§.§ Code review.

Siow et al. <cit.> believed that the hinge of the accurate code review suggestion is to learn good representations for both code changes and reviews. Therefore, they designed a multi-level embedding framework to represent the semantics provided by code changes and reviews and then well trained through an attention-based deep learning model, CORE. Guo et al. <cit.> proposed Deep Review Sharing, a new technique based on code clone detection for accurate review sharing among similar software projects, and optimized their technique by a series of operations such as heuristic filtering and review deduplication. They evaluated Deep Review Sharing on hundreds of real code segments and it won considerable positive approvals by experts, illustrating its effectiveness.

§.§.§ Software quality evaluation

Variety evaluation metrics can be used to describe the quality of software products <cit.>.

Software trustworthiness. It is essential and necessary to evaluate software trustworthiness based on the influence degrees of different software behaviors for minimizing the interference of human factors. Tian et al. <cit.> constructed behaviour trajectory matrices to represent the behaviour trajectory and then trained the deep residual network (ResNet) as a software trustworthiness evaluation model to classify the current software behaviors. After that, they used the cosine similarity algorithm to calculate the deviation degree of the software behavior trajectory.

Readability. Mi et al. <cit.> proposed to leverage CNN to improve code readability classification. First, they present a transformation strategy to generate integer matrices as the input of ConvNets. Then they trained Deep CRM, a DL-based model, which was made up of three separate ConvNets with identical architectures for code readability classification.

Maintainability. Kumar et al. <cit.> performed two case studies and applied three DNNs i.e., FLANN-Genetic (FGA and AFGA), FLANN-PSO (FPSO and MFPSO), FLANN-CSA (FCSA), to design a model for predicting maintainability. They also evaluated the effectiveness of feature reduction techniques for predicting maintainability. The experimental result showed that feature reduction techniques can achieve better results compared with using DNNs.

§.§ Software management

§.§.§ Effort estimation

Since only 39% of software projects are finished and published on time relative to the duration planned originally <cit.>, it is necessary to assess the development cost to achieve reliable software within development schedule and budget. Lopez et al. <cit.> compared three different neural network models for effort estimation. The experimental result demonstrated that MLP and RBFNN can achieve higher accuracy than the MLR model. Choetkiertiku et al. <cit.> observed that few studies focused on estimating effort cost in agile projects, and thus they proposed a DL-based model for predicting development cost based on combining two powerful DL techniques: LSTM and recurrent highway network (RHN).
Phannachitta et al. <cit.> conducted an empirical study to revisit the systematic comparison of heuristics-based and learning-based software effort estimators on 13 standard benchmark datasets. Ochodek et al.<cit.> employed several DNNs (i.e., CNN, RNN, Convolutional + Recurrent Neural Network (CRNN)) to design a novel prediction model, and compared the performance of the DL-based model with three state-of-the-art approaches: AUC, AUCG, and BN-UCGAIN. They noticed that CNN obtained the best prediction accuracy among all software effort adaptors.

§.§.§ Software repository mining

Some primary studies use DL techniques to mine the contents in different software repositories <cit.>. In this section, we introduce three most widely used repositories, i.e., Stack Overflow (SO) <cit.>, GitHub <cit.>, and Youtube <cit.>.

Mining Stack Overflow (SO). The questions and answers in SO contain a great deal of useful information that is beneficial for programmers to address some tough problems when implementing software products. Considering a question and its answers in Stack Overflow as a knowledge unit, Xu et al. <cit.> extracted the knowledge units and analyzed the potential semantic relationship between Q and A in each unit. They formulated the problem of predicting semantically linkable knowledge units as a multi-class classification problem and adopted a CNN model combining with word-embedding to capture and classify document-level semantics of knowledge units. Chen et al. <cit.> also applied word embeddings and the CNN model to mine SO for retrieving cross-lingual questions. They compared their approach with other translation-based methods, and the final results showed that their approach can significantly outperform baselines and can also be extended to dual-language document retrieval from different sources.  Yin et al. <cit.> proposed a new approach to pair the title of a question with the code in the accepted answer by mining high-quality aligned data from SO using two sets of features. To validate whether DL was the best solution in all research topics, Menzies et al. <cit.> conducted a case study to explore faster approaches for text mining SO. They compared nine different solutions including traditional machine learning algorithms and DL algorithms and noticed that a tuned SVM performs similarly to a deep learner and was over 500 times faster than DL-based models.
Zhang et al. <cit.> performed an empirical study to mine the posts in SO for investigating what potential challenges developers face when using DL. They also built a classification model to quantify the distribution of different Sort of DL related questions. They summarized the three most frequently asked questions and provided five future research directions. Since answers to a question may contain extensive irrelevant information, Wang et al. <cit.> proposed a DL-based approach, DeepTip, using different CNN architectures to extract short practical and useful tips and filtered useless information from developer answers. They conducted a user study to prove the effectiveness of their approach and the extensive empirical experiments demonstrated that DeepTip can extract useful information from answers with high precision and coverage, significantly outperforming two state-of-the-art approaches.

Mining GitHub. Huang et al. <cit.> proposed a new model to classify sentences from issue reports of four projects in GitHub. They constructed a dataset collecting 5,408 sentences and refined previous categories (i.e., feature request, problem discovery, information seeking, information giving, and others). They then trained a CNN-based model to automatically classify sentences into different categories of intentions and used the batch normalization and automatic hyperparameter tuning approach to optimize their model.
Xie et al. <cit.> proposed a new approach to recover issue-commit links. They constructed the code knowledge graph of a code repository and captured the semantics of issue- or commit-related text by generating embeddings of source code files. Then they trained a DNN model to calculate code similarity and semantic similarity using additional features. Ruan et al.<cit.> propose a novel semantically-enhanced link recovery method, DeepLink, using DL techniques.
They applied word embedding and RNN to implement a DNN architecture to learn the semantic representation of code and texts as well as the semantic correlation between issues and commits. They compared DeepLink with the state-of-the-art approach on 10 GitHub Java projects to evaluate the effectiveness of DeepLink. Liu et al. <cit.>  proposed a DL-based approach to automatically generate pull request descriptions based on the commit messages and the added source code comments in pull requests. They formulated this problem as a text summarization problem and solved it, constructing an attentional encoder-decoder model with a pointer generator.

Mining Youtube. Ott et al. <cit.> employed CNN and AutoEncoder to identify Java code from videos on Youtube. Their approach was able to identify the presence of typeset and handwritten source code in thousands of video images with 85.6%-98.6% accuracy based on syntactic and contextual features learned through DL techniques. Zhao et al. <cit.> present a new technique for recognizing workflow actions in programming screencasts. They collected programming screencasts from Youtube and trained a CNN model to identify nine classes of frequent developer actions by employing the image differencing technique and training a CNN model.

* We grouped six SE activities based on the body knowledge of SE – Software requirements, Software design and modeling, Software implementation, Software testing and debugging, Software maintenance, and Software management – and provided an outline of the application trend of DL techniques among these SE activities.

* We summarized various SE tasks into four categories – regression task, classification task, ranking task, and generation task – and classified all primary studies based on the task types. Most studies can be mapped to classification tasks, and only 11% of primary studies are mapped to ranking tasks.

* Software testing and software maintenance are the two SE activities containing the most related studies and include 17 specific SE research topics in which DL techniques were used.

§ LIMITATIONS

Data Extraction. There are several potential limitations to our work. One limitation is the potential bias in data collection. Although we have listed the data items used for analysis in RQs in Section 3.4, some disagreements still appeared inevitably when extracting related content and classifying these data items. Two researchers first recorded the relevant descriptions in each primary study and then discussed and generalized temporary categories based on all data in one item by comparing the objectives and contributions with related studies. If they were unable to reach an agreement on classification, another researcher would join in and resolve differences, which can mitigate the bias in data extraction to study validity.

Study Selection Bias. Another threat might be the possibility of missing DL related studies during the literature search and selection phase. We are unable to identify and retrieve all relevant publications considering the many publication venues publishing SE relevant papers. Therefore, we carefully selected 21 publication venues, including conference proceedings, symposiums, and journals, which can cover many prior works in SE. Besides, we identified our search terms and formed a search string by combining and amending the search strings used in other literature reviews on DL. These could keep the number of missed primary studies as small as possible.

§ CHALLENGES AND OPPORTUNITIES

Using DL in more SE activities.
We found that DL has been widely used in certain SE topics, such as defect prediction, code clone detection, software repository mining, etc. However, few studies used DL for some SE research topics compared with other techniques or other learning algorithms. Although Software requirements and software design are the most two important documentations during SDLC, not many studies focus on these two SE activities. Therefore, one potential opportunity is that researchers can utilize DL techniques to explore new research topics or pay attention to classical topics in software requirements and design.

The transparency of DL.
In this study, we discussed 142 studies that used DL to address various SE issues. We noticed that few studies declared the reason for the architecture they chose and explained the necessity and value of each layer in DNN, which leads to low transparency of the proposed DL solutions. Because it is inherently difficult to comprehend what drives the decisions of the DNN due to the black-box nature of DL. Humans only pay attention to the output of DNNs since they can provide wise and actionable suggestions for humans. Furthermore, DL algorithms sift through millions of data points to find patterns and correlations that often go unnoticed by human experts. The decision they make based on these findings often confound even the engineers who created them. New methods and studies on explaining the decision-making process of DNNs should be an active research direction, which facilitates software engineers to design more effective DNN architectures for specific SE problems.

DL in real scenarios.
We analyzed the source of datasets used for training DNNs in RQ3 and noticed that only 4% studies used industry datasets to train and evaluate their proposed models. In fact, most studies contribute to addressing real-world SE issues, but the novel solutions or DL-based models have not been evaluated on industry data. There is a room for more industry-academia collaboration so that the proposed models can be validated on real industry data (which can be of very different nature than open-source data.

Data hungry.
When analyzing the studies related to code clone detection, we found that several open-source public data sets are often used repeatedly in these studies to evaluate their proposed models. A similar situation also exists in other research topics. These highlight the dependence on some studies on large publicly available labelled datasets. One reason is that training a DNN requires a massive volume of data, but copious amounts of training data are rarely available in most SE tasks. Besides, it is impossible to give every possible labeled sample of a problem space to a DL algorithm. Therefore, it will have to generalize or interpolate between its previous samples in order to classify data it has never seen before. It is a challenge to tackle the problem that DL techniques currently lack a mechanism for learning abstractions through explicit, verbal definition and only work best with thousands, millions, or even billions of training examples. One solution is to construct widely accepted datasets by using industrial labeled data or crawling software repositories to collect related data samples and label them as public datasets. Another is to develop new DL techniques, which can learn how to learn and be trained as an effective model with as little data size as possible, such as meta-learning.

Performance of DL and traditional techniques.
DL has been gradually used in more and more SE tasks, replacing the status of traditional algorithms. However, are DL algorithms really more efficient than traditional algorithms? What SE tasks are suitable for DL algorithms? What factors determine whether DL algorithms are better or worse than traditional algorithms? These questions are almost unanswered and neglected by most researchers. A potential opportunity is to answer these questions. researchers can conduct empirical studies to investigate what SE tasks or environments are suitable for DL and compare the performance between DL and traditional techniques in some important SE research topics where most of Dl algorithms were applied.

§ CONCLUSION

This work performed a SLR on 142 primary studies related to DL for SE from 21 publication venues, including conference proceedings, symposiums, and journals. We established four research questions to comprehensively investigate various aspects pertaining to applications of DL models to SE tasks. Our SLR showed that there was a rapid growth of research interest in the use of DL for SE.  Through an elaborated investigation and analysis, three DL architectures containing 30 different DNNs were used in primary studies, where RNN, CNN, and FNN are the three most widely used neural networks compared with other DNNs. We also generalized three different model selection strategies and analyzed the popularity of each one. To comprehensively understand the DNN training and testing process, we provided a detailed overview of key techniques in terms of data collection, data processing, model optimization, and model evaluation in RQ3. In RQ4, we analyzed the distribution of DL techniques used in different SE activities, classified primary studies according to specific SE tasks they solved and gave a brief summary of each work. We observed that DL techniques were applied in 23 SE topics, covering 6 SE activities. Finally, we identified a set of current challenges that still need to be addressed in future work on using DLs in SE.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Al-Hroob, Imam, and Al-HeisaAl-Hroob
et al2018]
authorpersonAysh Al-Hroob, personAyad Tareq
Imam, and personRawan Al-Heisa.
The use of artificial neural networks for
extracting actions and actors from requirements document.
journalIST  volume101
(year2018), pages1–15.

[Alahmadi, Khormi, Parajuli, Hassel, Haiduc,
and KumarAlahmadi et al2020]
authorpersonMohammad Alahmadi,
personAbdulkarim Khormi, personBiswas Parajuli,
personJonathan Hassel, personSonia Haiduc, and
personPiyush Kumar. year2020.
Code Localization in Programming Screencasts.
journalESE volume25,
number2 (year2020), pages1536–1572.

[Bao, Xing, Xia, Lo, Wu, and YangBao
et al2020]
authorpersonLingfeng Bao, personZhenchang
Xing, personXin Xia, personDavid Lo,
personMinghui Wu, and personXiaohu Yang.
psc2code: Denoising Code Extraction from
Programming Screencasts.
journalTOSEM volume29,
number3 (year2020), pages1–38.

[Barbez, Khomh, and GuéhéneucBarbez
et al2019]
authorpersonAntoine Barbez, personFoutse
Khomh, and personYann-Gaël Guéhéneuc.
Deep Learning Anti-patterns from Code Metrics
History. In booktitleICSME. IEEE,

[Ben Abdessalem, Nejati, Briand, and
StifterBen Abdessalem et al2016]
authorpersonRaja Ben Abdessalem, personShiva
Nejati, personLionel C Briand, and personThomas
Stifter. year2016.
Testing advanced driver assistance systems using
multi-objective search and neural networks. In
booktitleASE. pages63–74.

[Bhatia, Kohli, and SinghBhatia
et al2018]
authorpersonSahil Bhatia, personPushmeet
Kohli, and personRishabh Singh.
Neuro-symbolic program corrector for introductory
programming assignments. In booktitleICSE. IEEE,

[Bisi and GoyalBisi and Goyal2016]
authorpersonManjubala Bisi and
personNeeraj Kumar Goyal. year2016.
Software development efforts prediction using
artificial neural network.
journalIETS volume10,
number3 (year2016), pages63–71.

[Bourque, Fairley, et alBourque
et al2014]
authorpersonPierre Bourque, personRichard E
Fairley, et al year2014.
booktitleGuide to the software engineering body of
knowledge (SWEBOK (R)): Version 3.0.
publisherIEEE Computer Society Press.

[Büch and AndrzejakBüch and
authorpersonLutz Büch and personArtur
Andrzejak. year2019.
Learning-based recursive aggregation of abstract
syntax trees for code clone detection. In booktitleSANER.
IEEE, pages95–104.

[Cambronero, Li, Kim, Sen, and
ChandraCambronero et al2019]
authorpersonJose Cambronero, personHongyu
Li, personSeohyun Kim, personKoushik Sen, and
personSatish Chandra. year2019.
When deep learning met code search. In
booktitleFSE. pages964–974.

[Chen, Diao, Zeng, Guo, and HuChen
et al2018a]
authorpersonChao Chen, personWenrui Diao,
personYingpei Zeng, personShanqing Guo, and
personChengyu Hu. year2018a.
DRLgencert: Deep learning-based automated testing
of certificate verification in SSL/TLS implementations. In
booktitleICSME. IEEE, pages48–58.

[Chen, Su, Meng, Xing, and LiuChen
et al2018b]
authorpersonChunyang Chen, personTing Su,
personGuozhu Meng, personZhenchang Xing, and
personYang Liu. year2018b.
From ui design image to gui skeleton: a neural
machine translator to bootstrap mobile gui implementation. In
booktitleICSE. pages665–676.

[Chen, Chen, Xing, and XuChen
et al2016]
authorpersonGuibin Chen, personChunyang
Chen, personZhenchang Xing, and personBowen Xu.
Learning a dual-language vector space for
domain-specific cross-lingual question retrieval. In
booktitleASE. IEEE, pages744–755.

[Chen, Chen, Xing, Xia, Zhu, Grundy, and
WangChen et al2020]
authorpersonJieshan Chen, personChunyang
Chen, personZhenchang Xing, personXin Xia,
personLiming Zhu, personJohn Grundy, and
personJinshui Wang. year2020.
Wireframe-based UI design search through image
journalTOSEM volume29,
number3 (year2020), pages1–31.

[Choetkiertikul, Dam, Tran, Pham, Ghose, and
MenziesChoetkiertikul et al2018]
authorpersonMorakot Choetkiertikul,
personHoa Khanh Dam, personTruyen Tran,
personTrang Pham, personAditya Ghose, and
personTim Menzies. year2018.
A deep learning model for estimating story points.
journalTSE volume45,
number7 (year2018), pages637–656.

[Ciurumelea, Proksch, and GallCiurumelea
et al2020]
authorpersonAdelina Ciurumelea,
personSebastian Proksch, and personHarald C Gall.
Suggesting Comment Completions for Python using
Neural Language Models. In booktitleSANER. IEEE,

[Cummins, Petoumenos, Murray, and
LeatherCummins et al2018]
authorpersonChris Cummins, personPavlos
Petoumenos, personAlastair Murray, and personHugh
Leather. year2018.
Compiler fuzzing through deep learning. In
booktitleISSTA. pages95–105.

[Dam, Pham, Ng, Tran, Grundy, Ghose, Kim, and
KimDam et al2019]
authorpersonHoa Khanh Dam, personTrang
Pham, personShien Wee Ng, personTruyen Tran,
personJohn Grundy, personAditya Ghose,
personTaeksu Kim, and personChul-Joo Kim.
Lessons learned from using a deep tree-based model
for software defect prediction in practice. In
booktitleMSR. IEEE, pages46–57.

[Dam, Tran, Pham, Ng, Grundy, and GhoseDam
et al2018]
authorpersonHoa Khanh Dam, personTruyen
Tran, personTrang Thi Minh Pham, personShien Wee Ng,
personJohn Grundy, and personAditya Ghose.
Automatic feature learning for predicting
vulnerable software components.
journalTSE (year2018).

authorpersonLi Deng. year2014.
A tutorial survey of architectures, algorithms, and
applications for deep learning.
journalAPSIPA Transactions on Signal and Information
Processing  volume3 (year2014).

[Deshmukh, Annervaz, Podder, Sengupta, and
DubashDeshmukh et al2017]
authorpersonJayati Deshmukh, personKM
Annervaz, personSanjay Podder, personShubhashis
Sengupta, and personNeville Dubash.
Towards accurate duplicate bug retrieval using deep
learning techniques. In booktitleICSME. IEEE,

[DQ, Yu, and JiangDQ et al2019]
authorpersonBui Nghi DQ, personYijun Yu,
and personLingxiao Jiang. year2019.
Bilateral dependency neural networks for
cross-language algorithm classification. In
booktitleSANER. IEEE, pages422–433.

[Enişer and SenEnişer and
authorpersonHasan Ferit Enişer and
personAlper Sen. year2020.
Virtualization of stateful services via machine
journalSQJ volume28,
number1 (year2020), pages283–306.

[Fakhoury, Arnaoudova, Noiseux, Khomh, and
AntoniolFakhoury et al2018]
authorpersonSarah Fakhoury, personVenera
Arnaoudova, personCedric Noiseux, personFoutse
Khomh, and personGiuliano Antoniol.
Keep it simple: Is deep learning good for
linguistic smell detection?. In booktitleSANER. IEEE,

[Fang, Liu, Shi, Huang, and ShiFang
et al2020]
authorpersonChunrong Fang, personZixi Liu,
personYangyang Shi, personJeff Huang, and
personQingkai Shi. year2020.
Functional code clone detection with syntax and
semantics fusion learning. In booktitleISSTA.

[Gao, Chen, Xing, Ma, Song, and LinGao
et al2019a]
authorpersonSa Gao, personChunyang Chen,
personZhenchang Xing, personYukun Ma,
personWen Song, and personShang-Wei Lin.
A neural model for method name generation from
functional description. In booktitleSANER. IEEE,

[Gao, Wang, Liu, Yang, Sang, and CaiGao
et al2019b]
authorpersonYi Gao, personZan Wang,
personShuang Liu, personLin Yang,
personWei Sang, and personYuanfang Cai.
TECCD: A Tree Embedding Approach for Code Clone
Detection. In booktitleICSME. IEEE,

[Genc-Nayebi and AbranGenc-Nayebi and
authorpersonNecmiye Genc-Nayebi and
personAlain Abran. year2017.
A systematic literature review: Opinion mining
studies from mobile app store user reviews.
journalJSS  volume125
(year2017), pages207–219.

[Gill, Tuli, Toosi, Cuadrado, Garraghan,
Bahsoon, Lutfiyya, Sakellariou, Rana, Dustdar, et alGill
et al2020]
authorpersonSukhpal Singh Gill,
personShreshth Tuli, personAdel Nadjaran Toosi,
personFelix Cuadrado, personPeter Garraghan,
personRami Bahsoon, personHanan Lutfiyya,
personRizos Sakellariou, personOmer Rana,
personSchahram Dustdar, et al
ThermoSim: Deep learning based framework for
modeling and simulation of thermal-aware resource management for cloud
computing environments.
journalJSS (year2020),

[Godefroid, Peleg, and SinghGodefroid
et al2017]
authorpersonPatrice Godefroid, personHila
Peleg, and personRishabh Singh.
Learn&fuzz: Machine learning for input fuzzing.
In booktitleASE. IEEE, pages50–59.

[Goodfellow, Bengio, and CourvilleGoodfellow
et al2016]
authorpersonIan Goodfellow, personYoshua
Bengio, and personAaron Courville.
booktitleDeep learning.
publisherMIT press.

[Gu, Zhang, and KimGu et al2018]
authorpersonXiaodong Gu, personHongyu
Zhang, and personSunghun Kim.
Deep code search. In
booktitleICSE. IEEE, pages933–944.

[Gu, Zhang, Zhang, and KimGu
et al2016]
authorpersonXiaodong Gu, personHongyu
Zhang, personDongmei Zhang, and personSunghun
Kim. year2016.
Deep API learning. In
booktitleFSE. pages631–642.

[Guo, Huang, Dong, Ye, Xu, Fan, Yang, and
XuGuo et al2019a]
authorpersonChenkai Guo, personDengrong
Huang, personNaipeng Dong, personQuanqi Ye,
personJing Xu, personYaqing Fan, personHui
Yang, and personYifan Xu. year2019a.
Deep review sharing. In
booktitleSANER. IEEE, pages61–72.

[Guo, Wang, Wu, Dong, Ye, Xu, and ZhangGuo
et al2019b]
authorpersonChenkai Guo, personWeijing
Wang, personYanfeng Wu, personNaipeng Dong,
personQuanqi Ye, personJing Xu, and
personSen Zhang. year2019b.
Systematic comprehension for developer reply in
mobile system forum. In booktitleSANER. IEEE,

[Ha and ZhangHa and Zhang2019]
authorpersonHuong Ha and personHongyu
Zhang. year2019.
Deepperf: performance prediction for configurable
software with deep sparse neural network. In
booktitleICSE. IEEE, pages1095–1106.

[Han, Li, Xing, Liu, and FengHan
et al2017]
authorpersonZhuobing Han, personXiaohong
Li, personZhenchang Xing, personHongtao Liu, and
personZhiyong Feng. year2017.
Learning to predict severity of software
vulnerability using only vulnerability description. In
booktitleICSME. IEEE, pages125–136.

authorpersonDouglas M Hawkins.
The problem of overfitting.
journalJournal of chemical information and computer
sciences volume44, number1 (year2004),

[Hellendoorn, Bird, Barr, and
AllamanisHellendoorn et al2018]
authorpersonVincent J Hellendoorn,
personChristian Bird, personEarl T Barr, and
personMiltiadis Allamanis. year2018.
Deep learning type inference. In
booktitleFSE. pages152–162.

[Hinton, Osindero, and TehHinton
et al2006]
authorpersonGeoffrey E Hinton, personSimon
Osindero, and personYee-Whye Teh.
A fast learning algorithm for deep belief nets.
journalNeural computation volume18,
number7 (year2006), pages1527–1554.

[Hoang, Dam, Kamei, Lo, and UbayashiHoang
et al2019a]
authorpersonThong Hoang, personHoa Khanh
Dam, personYasutaka Kamei, personDavid Lo, and
personNaoyasu Ubayashi. year2019a.
DeepJIT: an end-to-end deep learning framework for
just-in-time defect prediction. In booktitleMSR. IEEE,

[Hoang, Lawall, Tian, Oentaryo, and LoHoang
et al2019b]
authorpersonThong Hoang, personJulia
Lawall, personYuan Tian, personRichard J Oentaryo,
and personDavid Lo. year2019b.
PatchNet: Hierarchical Deep Learning-Based Stable
Patch Identification for the Linux Kernel.
journalTSE (year2019).

[Hosseini, Turhan, and GunarathnaHosseini
et al2017]
authorpersonSeyedrebvar Hosseini,
personBurak Turhan, and personDimuthu Gunarathna.
A systematic literature review and meta-analysis on
cross project defect prediction.
journalTSE volume45,
number2 (year2017), pages111–147.

[Hu, Li, Xia, Lo, and JinHu
et al2018]
authorpersonXing Hu, personGe Li,
personXin Xia, personDavid Lo, and
personZhi Jin. year2018.
Deep code comment generation. In
booktitleICPC. IEEE, pages200–20010.

[Huang, Qiu, Zhong, and WangHuang
et al2020]
authorpersonQing Huang, personAn Qiu,
personMaosheng Zhong, and personYuan Wang.
A Code-Description Representation Learning Model
Based on Attention. In booktitleSANER. IEEE,

[Huang, Xia, Lo, and MurphyHuang
et al2018]
authorpersonQiao Huang, personXin Xia,
personDavid Lo, and personGail C Murphy.
Automating intention mining.
journalTSE (year2018).

[Huang, Sun, Xu, Chen, Towey, and XiaHuang
et al2019]
authorpersonRubing Huang, personWeifeng
Sun, personYinyin Xu, personHaibo Chen,
personDave Towey, and personXin Xia.
A survey on adaptive random testing.
journalTSE (year2019).

[Huo, Thung, Li, Lo, and ShiHuo
et al2019]
authorpersonXuan Huo, personFerdian Thung,
personMing Li, personDavid Lo, and
personShu-Ting Shi. year2019.
Deep transfer bug localization.
journalTSE (year2019).

[Hussain, Huang, Zhou, and WangHussain
et al2020]
authorpersonYasir Hussain, personZhiqiu
Huang, personYu Zhou, and personSenzhang Wang.
CodeGRU: Context-aware deep learning with gated
recurrent unit for source code modeling.
journalIST (year2020),

[Jiang, Armaly, and McMillanJiang
et al2017]
authorpersonSiyuan Jiang, personAmeer
Armaly, and personCollin McMillan.
Automatically generating commit messages from diffs
using neural machine translation. In booktitleASE. IEEE,

[Katz, Ruchti, and SchulteKatz
et al2018]
authorpersonDeborah S Katz, personJason
Ruchti, and personEric Schulte.
Using recurrent neural networks for decompilation.
In booktitleSANER. IEEE, pages346–356.

[Keele et alKeele
et al2007]
authorpersonStaffs Keele et al
booktitleGuidelines for performing systematic
literature reviews in software engineering.
typeTechnical Report. institutionTechnical
report, Ver. 2.3 EBSE Technical Report. EBSE.

[Koc, Wei, Foster, Carpuat, and PorterKoc
et al2019]
authorpersonUgur Koc, personShiyi Wei,
personJeffrey S Foster, personMarine Carpuat, and
personAdam A Porter. year2019.
An empirical assessment of machine learning
approaches for triaging reports of a java static analysis tool. In
booktitleICST. IEEE, pages288–299.

[Koo, Saumya, Kulkarni, and BagchiKoo
et al2019]
authorpersonJinkyu Koo, personCharitha
Saumya, personMilind Kulkarni, and personSaurabh
Bagchi. year2019.
Pyse: Automatic worst-case test generation by
reinforcement learning. In booktitleICST. IEEE,

[Kumar and RathKumar and Rath2016]
authorpersonLov Kumar and personSantanu Ku
Rath. year2016.
Hybrid functional link artificial neural network
approach for predicting maintainability of object-oriented software.
journalJSS  volume121
(year2016), pages170–190.

[Lacomis, Yin, Schwartz, Allamanis, Le Goues,
Neubig, and VasilescuLacomis et al2019]
authorpersonJeremy Lacomis, personPengcheng
Yin, personEdward Schwartz, personMiltiadis
Allamanis, personClaire Le Goues, personGraham
Neubig, and personBogdan Vasilescu.
Dire: A neural approach to decompiled identifier
naming. In booktitleASE. IEEE, pages628–639.

[Lam, Nguyen, Nguyen, and NguyenLam
et al2017]
authorpersonAn Ngoc Lam, personAnh Tuan
Nguyen, personHoan Anh Nguyen, and personTien N
Nguyen. year2017.
Bug localization with combination of deep learning
and information retrieval. In booktitleICPC. IEEE,

[Le and LoLe and Lo2018]
authorpersonTien-Duy B Le and personDavid
Lo. year2018.
Deep specification mining. In
booktitleProceedings of the 27th ACM SIGSOFT International
Symposium on Software Testing and Analysis. pages106–117.

[LeClair, Eberhart, and McMillanLeClair
et al2018]
authorpersonAlexander LeClair, personZachary
Eberhart, and personCollin McMillan.
Adapting neural text classification for improved
software categorization. In booktitleICSME. IEEE,

[LeClair, Jiang, and McMillanLeClair
et al2019]
authorpersonAlexander LeClair, personSiyuan
Jiang, and personCollin McMillan.
A neural model for generating natural language
summaries of program subroutines. In booktitleICSE. IEEE,

[LeCun, Bengio, and HintonLeCun
et al2015]
authorpersonYann LeCun, personYoshua
Bengio, and personGeoffrey Hinton.
Deep learning.
journalnature volume521,
number7553 (year2015), pages436–444.

[Lee, Hong, Yi, Kim, Kim, and YooLee
et al2019]
authorpersonSeongmin Lee, personShin Hong,
personJungbae Yi, personTaeksu Kim,
personChul-Joo Kim, and personShin Yoo.
Classifying false positive static checker alarms in
continuous integration using convolutional neural networks. In
booktitleICST. IEEE, pages391–401.

[Li, Liu, Jin, and UmerLi
et al2020b]
authorpersonGuangjie Li, personHui Liu,
personJiahao Jin, and personQasim Umer.
Deep Learning Based Identification of Suspicious
Return Statements. In booktitleSANER. IEEE,

[Li, Feng, Zhuang, Meng, and RyderLi
et al2017]
authorpersonLiuqing Li, personHe Feng,
personWenjie Zhuang, personNa Meng, and
personBarbara Ryder. year2017.
Cclearner: A deep learning-based clone detection
approach. In booktitleICSME. IEEE,

[Li, Li, Zhang, and ZhangLi
et al2019]
authorpersonXia Li, personWei Li,
personYuqun Zhang, and personLingming Zhang.
Deepfl: Integrating multiple fault diagnosis
dimensions for deep fault localization. In
booktitleISSTA. pages169–180.

[Li, Jiang, Li, Hassan, He, Huang, Zeng, Wang,
and ChenLi et al2020a]
authorpersonYangguang Li, personZhen Ming
Jiang, personHeng Li, personAhmed E Hassan,
personCheng He, personRuirui Huang,
personZhengda Zeng, personMian Wang, and
personPinan Chen. year2020a.
Predicting Node Failures in an Ultra-large-scale
Cloud Computing Platform: an AIOps Solution.
journalTOSEM volume29,
number2 (year2020), pages1–24.

[Liu, Gao, Xia, Lo, Grundy, and YangLiu
et al2020a]
authorpersonChao Liu, personCuiyun Gao,
personXin Xia, personDavid Lo, personJohn
Grundy, and personXiaohu Yang.
On the Replicability and Reproducibility of Deep
Learning in Software Engineering.
journalarXiv preprint arXiv:2006.14244

[Liu, Jin, Xu, Bu, Zou, and ZhangLiu
et al2019a]
authorpersonHui Liu, personJiahao Jin,
personZhifeng Xu, personYifan Bu,
personYanzhen Zou, and personLu Zhang.
Deep learning based code smell detection.
journalTSE (year2019).

[Liu, Xu, and ZouLiu et al2018b]
authorpersonHui Liu, personZhifeng Xu,
and personYanzhen Zou. year2018b.
Deep learning based feature envy detection. In
booktitleASE. pages385–396.

[Liu, Zhou, Yang, Liu, and GrundyLiu
et al2018c]
authorpersonJin Liu, personPingyi Zhou,
personZijiang Yang, personXiao Liu, and
personJohn Grundy. year2018c.
FastTagRec: fast tag recommendation for software
information sites.
journalASEJ volume25,
number4 (year2018), pages675–701.