# Systematic Literature Review on Application of Learning-based Approaches in
Continuous Integration ††thanks: Citation: Arani, A. K., M., Le, T. H., M.,
Zahedi & Babar, M. A. (2023, May). Systematic Literature Review on Application
of Learning-based Approaches in Continuous Integration. arXiv preprint (2024)

Ali Kazemi Arani
CREST-The Centre for Research
on Engineering Software Technologies
University of Adelaide
Adelaide, SA 5005, Australia
<EMAIL_ADDRESS>
&Triet Huynh Minh Le
CREST-The Centre for Research
on Engineering Software Technologies
University of Adelaide
Adelaide, SA 5005, Australia
<EMAIL_ADDRESS>
&Mansooreh Zahedi
School of Computing and Information Systems
University of Melbourne
Melbourne, VIC 3010, Australia
<EMAIL_ADDRESS>
&M. Ali Babar
CREST-The Centre for Research
on Engineering Software Technologies
University of Adelaide
Adelaide, SA 5005, Australia
<EMAIL_ADDRESS>

###### Abstract

Context: Machine learning (ML) and deep learning (DL) analyze raw data to
extract valuable insights in specific phases. The rise of continuous practices
in software projects emphasizes automating Continuous Integration (CI) with
these learning-based methods, while the growing adoption of such approaches
underscores the need for systematizing knowledge. Objective: Our objective is
to comprehensively review and analyze existing literature concerning learning-
based methods within the CI domain. We endeavour to identify and analyse
various techniques documented in the literature, emphasizing the fundamental
attributes of training phases within learning-based solutions in the context
of CI. Method: We conducted a Systematic Literature Review (SLR) involving 52
primary studies. Through statistical and thematic analyses, we explored the
correlations between CI tasks and the training phases of learning-based
methodologies across the selected studies, encompassing a spectrum from data
engineering techniques to evaluation metrics. Results: This paper presents an
analysis of the automation of CI tasks utilizing learning-based methods. We
identify and analyze nine types of data sources, four steps in data
preparation, four feature types, nine subsets of data features, five
approaches for hyperparameter selection and tuning, and fifteen evaluation
metrics. Furthermore, we discuss the latest techniques employed, existing gaps
in CI task automation, and the characteristics of the utilized learning-based
techniques. Conclusion: This study provides a comprehensive overview of
learning-based methods in CI, offering valuable insights for researchers and
practitioners developing CI task automation. It also highlights the need for
further research to advance these methods in CI.

_K_ eywords Continuous Integration $\cdot$ Machine Learning $\cdot$ Model
Training $\cdot$ Automation $\cdot$ Systematic Literature Review

##  1 Introduction

Continuous Integration (CI) refers to the software development practice of
automatically integrating code changes through frequent automated build
processes, has gained popularity for enhancing software delivery speed and
reliability through early issue detection [1]. CI enables rapid testing,
building, and software preparation at any time [2]. This facilitates bug
detection and fixation, resulting in faster delivery and improved software
quality. This approach reduces the development costs and enhances customer
satisfaction [1].

Frequent changes in CI environments produce large amounts of data. It can be
difficult and demanding in terms of time and resources to extract and analyze
these data, particularly in large-scale projects [3]. CI imposes considerable
expenses, significantly impacting software development expenses [4]. Notably,
Google and Mozilla reported monthly CI process costs in millions [2, 5]. The
CI phase consumes more than half of the resources in software development [6],
posing a barrier for smaller companies. Thus, enhancing CI pipeline
performance and reducing associated costs are crucial for software
development.

CI’s popularity, complex data, and costs drive the adoption of learning-based
methods including Machine Learning (ML) and Deep Learning (DL) techniques for
analysis, aiding software engineers with valuable insights. These insights
enhance the CI feedback loop, analyzing development data, test logs, CI phase
outcomes, and operational environment data [7].

Learning-based methods use mathematical models to acquire knowledge, make
decisions, or improve performance based on data and experience [8]. These
techniques can efficiently predict complex task outcomes. In Continuous
Integration (CI) environments, learning-based methods can enhance task
performance by predicting software defects without executing the current
version. They provide benefits such as accurately predicting outcomes of unit
tests [9] and regression tests [10]. They also aid software engineers in
timely decision-making, such as task assignment to different profiles [11].

Given the broad scope of learning-based methods and their diverse applications
in CI, understanding their impact on CI processes is crucial. This includes
discerning the decision-making steps in developing these methods, termed
Machine Learning for Continuous Integration (ML4CI). An exploration of this
area offers cross-disciplinary insights, emphasizing the necessity of a
systematic literature review in ML4CI.

This study endeavours to address the existing gap by conducting a
comprehensive Systematic Literature Review (SLR) that concentrates on
learning-based methodologies within CI development phases. Through the
analysis of 52 carefully selected papers spanning from 2000 to August 2023,
the objective is to elucidate the progress made in the automation of CI tasks
through the application of learning-based techniques. The developmental steps
involve data collection/preparation, feature engineering, training, tuning,
and evaluation of these methods.

Comprehensively analyzing data from all CI tasks offers researchers and
practitioners a synthesized overview of information sources and applied
techniques. This equips them to leverage these resources in CI pipelines,
potentially automating processes and guiding future endeavours.

Insights from this review benefit researchers and practitioners, offering a
foundational platform to advance CI phases. By reusing or refining learning-
based methods, more efficient practices can emerge. This research is
significant for those employing existing approaches or developing new
solutions for CI, considering identified gaps. Additionally, our discussion
addresses opportunities to optimize current solutions or address neglected
areas in real-world CI settings.

In summary, the main contributions of this paper are threefold:

  * •

First, the paper provides a clear and comprehensive understanding of the six
identified phases and ten tasks in CI that can be automated by using learning-
based methods, along with their sequence and relationships in the CI pipeline.

  * •

Second, the paper analyses the data and techniques used for data preparation,
feature engineering, training, tuning and evaluation of the learning-based
methods concerning the CI phases and tasks.

  * •

Finally, the paper discusses the future directions in applying learning-based
solutions to CI and the existing inconsistencies, which can serve as a guide
for researchers and practitioners in the field.

In previous research, our focus was on using ML methods to extract automated
CI tasks. We specifically looked at the training approaches used in state-of-
the-art (SOTA) learning-based methods for each CI task [12]. This paper
extends our previous work by updating the list of papers through searches in
two additional indexing databases and reviewing earlier publications.
Furthermore, we conduct an analysis specifically focusing on each training
phase of ML methods. Additionally, we employ thematic analysis to
systematically extract information from the published works and conduct deeper
analysis. The objective is to offer a more comprehensive and detailed account,
providing additional information and insights into learning-based methods
within the context of CI.

The rest of this paper is organized as follows: Section 2 provides background
and related works, followed by the research methodology in Section 3. Section
4 presents synthesized data based on the proposed research questions (RQs).
Section 5 outlines the study’s limitations. Finally, Section 6 discusses the
findings, and Section 7 concludes the study.

Henceforth, we will use the term ‘ML’ as an abbreviation for the combined term
‘Learning-based’.

##  2 Background and Related Works

In this section, we will introduce the essential concepts of CI and the
applications of ML in software engineering, which are the primary focus of
this SLR.

###  2.1 Overview of Continuous Integration

CI encompasses the processes of building, testing, validating software, and
managing commit-related actions, including addressing reported bugs and
messages from developers, before advancing to the deployment phase [13]. A
development team integrates and merges their source code frequently, often
multiple times a day, to build the software [1]. Automated testing and quick
feedback are crucial to preventing issues from propagating to the delivery
phase or affecting the development process of other team members [13]. The
validation phase in CI provides feedback to developers on detected bugs or
performance issues, and they also monitor deployed software and the related
materials in operations and version control systems (VCS) to maintain its
performance [14].

Figure 1: The four phases of ML life cycle. Note: the required steps for
training an ML model are distinguished by numbers

###  2.2 Machine Learning in Software Engineering

ML methods enhance software product quality by automating tasks through
pattern analysis and learning from historical data [15]. During training,
models use a subset of input data to recognize patterns, making predictions or
classifications for new data based on their training methodology, which
includes supervised, unsupervised [16], semi-supervised [17], and
Reinforcement Learning (RL) [18].

For instance, a supervised ML model predicts build outcomes with features like
changed classes and committer experience [19]. Unsupervised learning clusters
data based on natural language features to assign tasks to related profiles
[11]. Section 4.5 further explores these learning types, highlighting their
common development process.

Previous studies have explored ML techniques in software engineering, with a
notable focus on ML methods exclusively in Test Case Selection and
Prioritization (TSP) [20]. This systematic review included 29 studies but did
not examine data-related concepts. Their primary focus was on comparisons
between studies and considerations related to reproducibility and
repeatability in the TSP domain.

In a broader perspective, a systematic mapping conducted by Durelli et al.
[21] explored the application of machine learning methods in software testing,
emphasizing their scalability and effectiveness in addressing complex issues
within software testing systems.

Unlike the mentioned studies concentrating on the testing phase, Shafiq et al.
[8] conducted a comprehensive literature review covering various aspects of ML
methods in software engineering, including software requirements, design,
construction, quality, and maintenance. They highlighted demographic data and
identified challenges, such as the uncertain nature of ML techniques, data
availability, and the increasing complexity of software products.

Zhang, and Tsai [22] compiled a comprehensive list of SE problems addressed
with ML techniques, discussing their application and the pros and cons of
using ML models in SE. Ali and Gravino [23] conducted a systematic review of
75 studies to assess commonly used ML models, datasets, and accuracy
measurement methods in software development effort estimation (SDEE).

Our study exclusively explores the application of ML within the CI domain.
Unlike other studies that address broader areas of software engineering, our
research offers a detailed analysis of ML-based solutions developed to be
employed in CI.

The literature proposes four primary steps in ML model development for CI
[24]: Data Collection and Data Engineering (Step 1), Feature Engineering (Step
2), Training and Hyper-parameter Tuning (Step 3), and Model Evaluation (Step
4). The representation in Figure 1 corresponds to these phases as outlined in
reference [24].

During the initial Data Collection and Engineering step, data can be gathered
from various sources, including raw data from CI tools and monitoring sensors
[25]. Post-collection, it is crucial to extract relevant features and adapt
them for the subsequent training phase [26]. For effective ML model training,
selecting an appropriate algorithm and adjusting hyperparameters for
performance optimization is essential. Finally, thorough evaluation and
validation are necessary to ensure the models’ effectiveness for real-world
applications [27]. Regularly repeating these steps is important to keep models
updated with new data.

##  3 Research Methodology

Figure 2: Overview of the research methodology. Note: Two arrows with opposite
directions present the iterative actions. The “I” and “E” stand for the
Inclusion and Exclusion criteria according to Table 3, respectively, and
numbers in parenthesis present the total number of selected papers in each
step.

To gain insights into the developmental stages of ML-based methods in CI, this
SLR followed Kitchenham’s guidelines [28]. Utilizing the
ACM111https://dl.acm.org/, IEEE222https://www.ieee.org/, and
Scopus333https://www.scopus.com/ indexing systems with a specified search
string method facilitated the retrieval of relevant studies. These databases
were chosen for their extensive coverage of journals and conferences in
Software Engineering and Computer Science domains [29].

Previous studies affirm the effectiveness of this search strategy in
collecting high-quality papers [29]. A summary of the research methodology and
the number of studies in each phase is presented in Figure 2. Further details
are provided in the subsequent section.

###  3.1 Research Questions

Table 1: Research questions and motivations. Research Questions |  Motivation
---|---
RQ1: What CI tasks can be automated using ML-based approaches? |  In this first research question, we seek to explain the specific CI tasks and phases that ML methods can effectively automate. Also, we present the output and input of these CI phases. This insight not only enhances our understanding of the integration process but also sheds light on underexplored CI phases and tasks that practitioners can further investigate.
RQ2: What datasets and data preparation techniques are used in automating CI tasks using ML? |  Given the data-driven nature of ML models, the characteristics and preparation of input datasets significantly influence the efficacy of ML-based solutions in CI. This research question investigates the frequently employed datasets and associated data engineering methodologies, including approaches to address challenges like class imbalance.
RQ3: What feature extraction strategies are utilized to train ML models for automating the CI tasks? |  Beyond raw data, the selection and engineering of features play a pivotal role in shaping the effectiveness of ML models. By identifying the array of feature types and techniques for transforming raw data into machine-readable inputs, we contribute to a repository of knowledge that can be used in future studies. This knowledge empowers researchers and practitioners to optimize the design of ML models for automating CI tasks.
RQ4: What ML modeling and tuning techniques are used to automate CI tasks? |  This research question focuses on the connection between ML model types and CI tasks. By uncovering the relationship between these two elements, we aim to pinpoint potential gaps in the application of ML models for automating CI tasks. Moreover, we extract the common approaches for tuning these models, shedding light on hyper-parameter tuning methods that enhance their performance.
RQ5: How the ML models are evaluated in case of automating the CI tasks? |  In the final research question, we focus on the evaluation of ML model performance within the realm of CI. Our objective is to categorize the commonly employed evaluation metrics and techniques, elucidating their correlation with the four distinct ML algorithm categories outlined in Section 2.2. This categorization enables researchers and practitioners to conduct informed comparisons of various ML methodologies for automating the CI tasks fostering a good perspective based on existing literature.

To initiate this SLR, five research questions (RQs) were formulated to
identify the phases of CI and the ML model development techniques used for
these tasks. Section 4.2 provides an overview of the ML models and CI tasks
identified based on these RQs. You can find a detailed list of these RQs and
their motivations in Table 1.

###  3.2 Search Strategy

The next step in this SLR is designing an appropriate search strategy for
finding relevant studies [28]. The search strategy configuration for
extracting studies from databases is detailed below.

Table 2: Search string used to extract relevant papers on the application of ML methods in CI. The search string is composed of two segments, including “Machine Learning” and its associated synonyms and “Continuous Integration”. A |  TITLE-ABS-KEY ((“Machine Learning” OR “Deep Learning” OR “Reinforcement Learning” OR “Supervised Learning” OR “Unsupervised Learning” OR “Semi-supervised Learning” OR “Data Mining” OR “Text Mining” OR “Natural Language Processing” )
---|---
1-2 B |  AND (“Continuous Integration”)

####  3.2.1 Search String

Our search string consists of two segments: A) “Machine Learning” and related
synonyms, and B) “Continuous Integration”. These choices were informed by a
review of previous studies and SLRs [1, 30, 31, 32], with the complete search
string detailed in Table 2. The search string was executed on 26th July 2023,
and snowballing was conducted on 7th August 2023.

Before implementing our search string to identify relevant literature, the
first and third authors conducted a preliminary search on Google Scholar using
keywords such as ‘machine learning,’ ‘Continuous Integration’, and related
terms. This search aimed to identify initial relevant studies on the
application of machine learning (ML) methods in Continuous Integration (CI).
This preliminary search served as a validation set of papers. Subsequently, we
iteratively refined our search string to retrieve all the papers included in
our validation set. However, due to its tendency to yield numerous irrelevant
results [1], Google Scholar was not used for our primary search.

To refine our search string, we iteratively ran it on ACM, IEEE, and Scopus
databases, identifying and adding missing search terms for comprehensive
coverage in the initial phase. The refined search string was then executed on
these three databases, resulting in 1189 hits.

####  3.2.2 Inclusion and Exclusion Criteria

Table 3: Inclusion and exclusion criteria. ID | Inclusion criteria
---|---
I1 | Studies that employed ML-based methods in the CI environment.
I2 | Related to software development and engineering.
ID | Exclusion criteria
E1 | The paper is tagged as non-English language.
E2 | Non-research papers including conference reviews and reports, notes, and short surveys.
E3 | Short papers (i.e., less than five pages)
E4 | Duplicate papers

For collecting a list of high-quality studies, we established both inclusion
and exclusion criteria. You can find the selection process and these criteria
in Figure 2 and Table 3 respectively. We applied these criteria to all 1189
studies from the three databases, involving several steps in the paper
selection process. Initially, non-English papers were excluded. Non-research
documents, such as reports and notes, were disregarded. Additionally, papers
with fewer than five pages were excluded for comprehensive method
explanations. To prevent redundancy, duplicated papers across all databases
were removed before reviewing the content.

In subsequent stages, the first and second authors reviewed paper titles and
abstracts, categorizing them into two lists and curating a selection of
potential papers relevant to ML in CI. A comprehensive examination of the full
text followed, ensuring relevance and content quality, with the cooperation of
the first and second authors and continuous input from the third and fourth
authors. According to Wohlin’s guidelines [33], the selected papers must be
published in peer-reviewed venues.

To ensure comprehensive coverage of relevant literature, we employed
snowballing on the initial set of 50 papers [33], with the assistance of
researchers whose names are presented in the acknowledgment section. Backward
and forward snowballing involved reviewing references in selected papers and
papers citing them, respectively. We repeated this process and reviewed steps
until no more related papers could be identified.

In backward snowballing, 11 papers were identified, and in forward
snowballing, four potential papers were found before full-text review. After
reviewing these 15 potential papers, we filtered out 13 based on inclusion and
exclusion criteria. In the end, 52 papers on the application of ML in CI were
selected. The list and corresponding IDs are presented in the appendices,
specifically in Table 22.

###  3.3 Data Extraction and Data Synthesis

Data extraction: Following Kitchenham’s guidelines [28], we designed a form to
systematically collect information, including demographic data, from selected
studies. The first author organized the extracted data in a Microsoft Excel
spreadsheet according to the extraction form provided in Table 23, with random
checks performed by the second author.

Data synthesis: This study aims to comprehensively analyze, classify, and
report both qualitative and quantitative data. Thematic analysis was utilized
for synthesizing qualitative data [34] a method recognized for its
applicability in the software engineering domain [35]. Quantitative data were
presented in a raw format, including demographic information, or were derived
from the synthesized qualitative data. The process of qualitative thematic
analysis comprised six steps, as outlined below. The initial execution of
these steps was undertaken by the first author and thoroughly discussed with
the second author for potential modifications. The third and fourth authors
oversaw this process and offered feedback for necessary adjustments.

  1. 1.

_Familiarizing with data:_ We reviewed and annotated each field of the
extracted data.

  2. 2.

_Generating initial codes:_ At this stage, we broke down the data into smaller
parts and assigned codes, meaningful words or phrases acting as labels. This
step involved iterative merging and revision of codes.

  3. 3.

_Searching for themes:_ After finalizing the codes, we reorganized and
gathered relevant data for each code, defining potential themes.

  4. 4.

_Reviewing themes:_ We reviewed potential themes for relevance with the
extracted data and codes for each research question.

  5. 5.

_Defining and naming themes:_ In the last step, we defined coherent and
precise names for each theme, along with clear definitions.

  6. 6.

_Reporting:_ Our analysis and findings were mapped to each research question
and reported in section 4

##  4 Findings

In this section, we present the findings derived from our comprehensive
analysis of demographic data and synthesized data, and we will provide answers
to our five research questions as outlined in Table 1.

###  4.1 Demographic Data

This section presents the demographic data concerning the four research design
attributes, namely (1) publication year, (2) publisher venue and type of the
venues: conference, journal or workshop, (3) study context and research
methods, and (4) types of learning algorithms. The demographic information is
valuable for researchers who wish to conduct research in this area, as it
provides insights into where relevant studies can be found and their trends
over the years, as well as critical areas by reviewing the list of keywords.
Furthermore, it offers an overview of the study context and research methods
employed in conducted studies [1].

####  4.1.1 Distribution of Studies

Figure 3: Number of selected studies published per year and their distribution
over publication venues. Note: No paper was published between 2006 and 2014 —
Due to running the search string on July 2023, and snowballing on August 2023,
the list of published papers in 2023 is incomplete.

Since we ran the search string on July 26th 2023 and conducted the snowballing
on August 7th 2023, we can say that a total of 52 studies were published
between 2000 and August 2023. As shown in Figure 3, the first study on the
application of ML techniques in CI was published in 2006 [36], which focused
on predicting build validation in the CI phase. Notably, demographic data
reveals that no paper was published on this topic between 2006 and 2014.
However, the application of ML methods in CI has gradually gained more
attention from researchers, and the number of studies started increasing since
2014. Additionally, the results indicate that a majority of papers (65.4%)
were published at conferences. It is worth mentioning that from 2021 all the
studies focused only on enhancing the regression testing and predicting the
build validations. This point presents the importance of these two phases in
the CI pipeline.

####  4.1.2 Distribution of Keywords

Figure 4: Word cloud of the keywords in selected primary studies in the CSE
area.

The concept of CI encompasses various areas and tools. To illustrate the areas
that researchers have focused on, we can examine the keywords used in the
selected studies. Figure 4 displays the frequency of each keyword in the
primary studies and highlights the most frequent ones. The domains covered in
this SLR are diverse, including Software Engineering concepts (e.g. Continuous
Integration, Test Case Prioritization, Monitoring), software tools (e.g.
Travis CI), and ML concepts (e.g. Classifier, Reward Function). The most
common keywords in the selected studies are Continuous Integration, Machine
Learning, Regression Testing, and Test Case Prioritization. This point depicts
the importance of the regression testing phase of CI and especially its test
case prioritization task.

Table 4: Distribution of the 52 selected primary studies on publication
venues. The color intensity in this table corresponds to the number of
studies, with lighter shades indicating lower counts and darker blues
representing higher counts.

Publication Venue | # | %
---|---|---
IEEE/ACM International Conference on Automated Software Engineering (ASE) | 3 | 5.8
International Conference on Software Engineering (ICSE) | 3 | 5.8
International Conference on Software Testing, Verification and Validation Workshops (ICSTW) | 3 | 5.8
Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (STA) | 3 | 5.8
IEEE Transactions on Software Engineering (TSE) | 3 | 5.8
Proceedings of the Asia-Pacific Symposium on Internetware (APSI) | 2 | 3.8
CEUR Workshop Proceedings (CEUR) | 2 | 3.8
Proceedings of the Annual International Conference on Computer Science and Software Engineering (CSSE) | 2 | 3.8
IEEE Conference on Software Testing, Validation and Verification (ICST) | 2 | 3.8
International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (IEA/AIE) | 2 | 3.8
Journal of Software: Evolution and Process (JSEP) | 2 | 3.8
IEEE International Conference on Software Analysis, Evolution and Re-engineering (SANER) | 2 | 3.8
Others (These venues only published one study) | 23 | 44.2

Table 4 displays the distribution of published papers on the application of ML
in CI across 35 venues, with 12 venues publishing more than one study. Among
these 12 venues, IEEE/ACM International Conference on Automated Software
Engineering (ASE), International Conference on Software Engineering (ICSE),
International Conference on Software Testing, Verification and Validation
Workshops (ICSTW), Proceedings of the ACM SIGSOFT International Symposium on
Software Testing and Analysis (STA), and IEEE Transactions on Software
Engineering (TSE) stand out as the top venues for publishing research on the
topic of application of ML in CI, with each venue publishing three papers. It
is worth noting that most of the papers (44.2%) were published in 23 different
venues, with Software Engineering venues being responsible for the majority of
papers (65.2%). The other domains of venues include Computer Science,
Artificial Intelligence, Web and databases.

####  4.1.3 Study Context and Research Methods

Table 5: The number of papers about each research and data analysis type. The color intensity in this table corresponds to the number of studies, with lighter shades indicating lower counts and darker blues representing higher counts. The bolded numbers represent the count of studies and their percentage in total. | Industry | Open-Source | Simulation
---|---|---|---
Study IDs |  |  |
(Number of |  |  |
studies, |  |  |
percentage) | S1, S4, S8, S9, S13, S18, S21, |  |
S24, S26, S27, S28, S29, S30, |  |  |
S37, S39, S42, S43, S44, S46, |  |  |
S47, S49 (21, 40.4%) | S2, S3, S5, S6, S7, S10, S11, S14, S15, S16, |  |
S17, S20, S22, S23, S25, S26, S31, S32, S33, |  |  |
S34, S35, S36, S37, S38, S40, S41, S45, |  |  |
S48, S50, S51, S52 (31, 59.6%) | S12, S19 |  |
(2, 3.8%) |  |  |

Table 5 provides a classification of the study context of the reviewed papers
into three groups:“Industry”, “Open-Source”, and “Non-industry (simulation)”.
Industrial studies were conducted with real-world data sets from software
companies, such as projects in Microsoft company (S1) [37], to verify the
applicability of the proposed methods in a practical closed-source software
environment. On the other hand, studies in the “Open-Source” category
validated their methods on real-world data projects from industrial open-
source software, such as the Apache projects (S10) [11]. The distribution of
the studies in Industrial and Open-Source categories presents the
applicability of the ML methods in different CI environments.

Based on Table 5, it can be observed that only two studies (S12, S19)
conducted research in simulated environments, while the majority of studies
(96.2%) were situated in the Industry and Open-source categories. Simulation-
based data sets refer to studies that implemented and evaluated their methods
in simulated environments. In S19, the authors used the Gym library to
simulate a CI environment by using execution logs of test cases and training a
Reinforcement Learning (RL) agent. The simulator recreates a similar
environment as a real environment for testing execution history. In S12, the
authors simulated a cloud environment by installing packages manually and
validated their ML method for discovering the installed software on containers
and Virtual Machines (VMs) in testing the whole system. This highlights the
usefulness of simulation-based data sets in evaluating and testing ML-based
methods in a controlled and reproducible environment.

Moreover, we conducted a clustering analysis of the primary studies based on
the types of research methods used. Typically, studies are categorized into
two groups, namely quantitative and qualitative research, and we assigned the
studies to each group based on their presented results and conclusions.
Quantitative research methods involve the collection of empirical data through
measurements and procedures, while qualitative studies are more descriptive
and leave more room for interpretation [38].

In evaluating the effectiveness of quantitative research methods, researchers
often rely on metrics such as the accuracy and performance of ML models,
whereas qualitative methods may involve surveys, interviews, and other
techniques to gather feedback from users of the trained ML models [39]. The
results presented that none of the studies in this area employed qualitative
methods for evaluating their solutions. Hence, a significant gap exists in the
case of using qualitative analysis in the area of ML for CI. Furthermore, in
the SLR paper, we provide additional information on the evaluation methods of
quantitative research, which can assist researchers and practitioners in
understanding how quantitative results can be validated and what are the most
commonly used evaluation metrics in the area of ML for CI.

Summary: $\bullet$ The number of published studies related to the application
of ML in CI has been on the rise, and gradually narrowed down to only
regression testing and build validation over time. $\bullet$ All of the
studies included in this SLR utilized quantitative research methods. However,
a lack of qualitative assessment of the results is visible in the literature.

Figure 5: Overview of connection between six CI phases and their in/output.

###  4.2 RQ1: Continuous Integration Phases and Tasks

Our analysis uncovered six distinct ML-enhanced phases within CI,
interconnected as illustrated in Figure 5. These phases are presented in the
order of their sequence within the CI pipeline. To highlight automated tasks
within each CI phase, we have identified them in an underlined format, briefly
explained in the description of each phase. Table 6 lists ten identified
automated CI tasks across these six phases. It is worth mentioning that, due
to the limited number of studies and the use of different evaluation metrics,
we did not compare state-of-the-art methods in each of the other tasks except
for the Test Optimization and Build Prediction tasks.

Table 6: List of papers in each CI phase with related tasks and total count in
parentheses. The table’s color intensity corresponds to the number of studies,
with lighter shades indicating lower counts and darker blues representing
higher counts. The bolded numbers represent the count of studies and their
percentage in total.

CI |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
Tasks |  [l]Unit Test Prediction |  [l]Branch Coverage Prediction |  [l]Integration Test Prediction |  Test Optimization |  Defect Prediction |  [l]Flaky Test Detection |  Build Prediction |  [l]Installed Software Discovery |  [l]Performance Test Optimization |  [l]Activity Management
CI |  |  |  |  |  |  |  |  |  |
Phase |  Unit |  |  |  |  |  |  |  |  |
Test |  Integration |  |  |  |  |  |  |  |
Test | Regression Test |  Build |  |  |  |  |  |
Validation |  System |  |  |  |  |  |  |  |
Test |  Process |  |  |  |  |  |  |  |  |
Management |  |  |  |  |  |  |  |  |  |
Studies |  S2 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S3, S15 |  |  |  |  |  |  |  |  |
(2, 3.9%) |  S18, S23 |  |  |  |  |  |  |  |  |
(2, 3.9%)) |  S7, S8, S11, S14, |  |  |  |  |  |  |  |  |
S16, S19, S20, S22, |  |  |  |  |  |  |  |  |  |
S26, S28, S29, S30, |  |  |  |  |  |  |  |  |  |
S31, S33, S34, S35, |  |  |  |  |  |  |  |  |  |
S36, S37, S39, S41, |  |  |  |  |  |  |  |  |  |
S45, S46, S47, S48, |  |  |  |  |  |  |  |  |  |
S50, S52 (26, 50%) |  S9, S13, |  |  |  |  |  |  |  |  |
S17, S32 |  |  |  |  |  |  |  |  |  |
(4, 7.7%) |  S40 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S1, S5, S6, |  |  |  |  |  |  |  |  |
S21, S24, S25, |  |  |  |  |  |  |  |  |  |
S27, S38, S42, |  |  |  |  |  |  |  |  |  |
S43, S44, S51 |  |  |  |  |  |  |  |  |  |
(12, 23.1%) |  S12 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S4 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S10, S17 |  |  |  |  |  |  |  |  |
(2, 3.9%) |  |  |  |  |  |  |  |  |  |

Unit Test (UT): According to Stolberg’s definition [40], the Unit Test phase
serves as the initial step in the CI pipeline. This phase involves validating
newly developed code within an isolated environment. It occurs whenever a
developer commits new or modified compiled code. The code built during this
step is referred to as the “Developer build”, as depicted in Figure 5.

In CI, the master branch is typically kept free from failures, with code
changes made on a separate “developer branch” throughout this study. Among the
52 studies, only S2 focused on this phase, conducting Unit Test Prediction.
The authors enhanced static code checkers’ accuracy using ML models to predict
false positives (wrongly predicting safe code) (S2), reducing the risk of
releasing buggy software.

Integration Test (IT): After validating units with Unit Test in the developer
branch, the next step is to integrate the new module with other developed
modules on the same branch [41]. This phase involves testing various modules
and new functionalities in combination with the entire system or sub-systems.
ML-based methods in this CI phase aim to predict test outcomes (S18) and
reduce the computational load of integration tests by skipping safe commits
(S23), executing fewer test cases, and reducing overall costs (S3, S15)
through ML-based Integration Test Prediction or Branch Coverage Prediction.

Table 7: Comparison of the three SOTA methods in optimizing the regression
testing. Note: TCP is Test Case Prioritization and TCS is Test Case Selection
Tasks; RL is Reinforcement Learning, NN is Neural Networks, and MAB is Multi-
Armed Bandit.

SID | Method Name | Strategy | Model Properties | Minimum Required Data |  Advantages |  Limitations
---|---|---|---|---|---|---
S7 | RETECS | TCP & TCS | RL (NN-based agent) | 60 test cycles |  [l]$\bullet$ Language-Free |
$\bullet$ Not required source code |  [l]$\bullet$ Not appropriate for |  |  |  |  |
large scale datasets |  |  |  |  |  |
1-7 S20 | COLEMAN | TCP | RL (MAB-based agent) | 100 test cycles |  [l]$\bullet$ Language-Free |
$\bullet$ Lightweight |  [l]$\bullet$ Cold start |  |  |  |  |
1-7 S43 | DeepOrder | TCP | DL | 4 test cycles |  [l]$\bullet$ Language-Free |
$\bullet$ Efficient for large datasets |  |  |  |  |  |
$\bullet$ Time-efficient |  [l]$\bullet$ Non-interpratable |  |  |  |  |

Regression Test (RT): Following integrated code validation and testing, the
software product’s current version undergoes comprehensive testing based on
previously designed test cases [42]. Regression tests encompass various types,
such as structural and functional testing [43]. Given the time and resource
intensity of regression testing [44], the majority of studies (31 out of 52)
focus on this area. The main ML-based solutions explored by 26 studies are
Test Case Prioritization (TCP) and Test Case Selection (TCS), contributing to
Test Optimization. In TCP, ML-based solutions expedite the detection of
software defects by ordering test cases based on the likelihood of revealing
faults more quickly [45]. TCS focuses on selecting test cases that assess code
changes while consuming fewer resources and less time [46]. Additionally,
selected studies introduced other ML-based solutions, including Defect
Prediction (S9, S13, S17, and S32) and Flaky Test Detection (S40).

Among the 26 papers primarily focused on Test Optimization, three stand out in
terms of the accuracy of the trained ML models selected as State-Of-The-Art
(SOTA) approaches: S7, S20, and S43. These papers introduced the Reinforced
Test Case Prioritization and Selection (RETECS), Combinatorial VOlatiLE Multi-
Armed BANdit (COLEMAN), and Deep learning for test case prioritization
(DeepOrder) methods, respectively. According to the authors, both COLEMAN and
DeepOrder outperformed the RETECS method, previously recognized as the latest
SOTA approach for test optimization in regression testing. Notably, upon
reviewing these COLEMAN and DeepOrder studies, it was observed that both
employed the IOF/ROL dataset. In this dataset, DeepOrder yielded slightly
better Normalized Average Percentage of Fault Detection (NAPFD) (see section
4.6) results than the COLEMAN method. However, it is essential to emphasize
that we cannot conclusively assert that DeepOrder universally outperforms the
COLEMAN method, as further examinations are required.

For a comprehensive overview of these three SOTA methods and their respective
advantages, please refer to Table 7. Importantly, all these SOTA methods share
a common characteristic: they do not require access to the source code for
test case prioritization, making them language-independent. Their sole
prerequisite is the availability of test case metadata, reducing training time
and facilitating the use of larger datasets.

Regarding other tasks in the RT phase, only one study focuses on Flaky Test
Detection, while four studies address Defect Prediction CI tasks. Our
comparative analysis will exclusively concentrate on the body of published
works concerning Defect Prediction tasks within the RT phase.

However, it is imperative to acknowledge that these studies have diverse
objectives and methodological properties. The wide range of research goals and
methodologies within this limited corpus prevents direct comparisons. Here, we
present the objectives of these studies to afford a comprehensive
understanding of their respective aims and contributions.

_Performance Enhancement for Low Defect Percentage Datasets(S9):_ This study
proposes methods with good performance on datasets characterized by a low
defect percentage.

_Impact of Feature Extraction Methods(S13):_ Investigations in this category
assess the influence of feature extraction techniques, such as bag-of-words
and word embedding, on ML method effectiveness.

_Transfer Learning for New Project Implementation(S17):_ Studies in this
category explore transfer learning techniques for using trained ML models in
new projects to predict defects.

_Continuous Defect Prediction Based on Code Change Features(S32):_ Research
efforts in this context introduce ML methods specifically designed to
continuously predict defects by leveraging code change features.

Table 8: Comparison of the three SOTA methods in build outcome validation.
Note: DNN is Deep Neural Network, LSTM is Long Short Term Memory.

SID | Method Name | Strategy | Model Properties | Results | Advantages
---|---|---|---|---|---
S21 | SmartBuildSkip | Predicting pass builds | Random Forest |  [l]$\bullet$ Outperform SOTA (S24) |
$\bullet$ Save 30% on building time |  $\bullet$ Lightweight |  |  |  |
1-6 S25 | BuildFast | Predicting fail builds | XGBoost |  [l]$\bullet$ Outperform SOTA (S6 and S24) |
$\bullet$ 47.5% improved F1 score of SOTA method |  [l]$\bullet$ Chronological |  |  |  |
$\bullet$ Lightweight |  |  |  |  |
1-6 S38 | DL-CIBuild | Predicting build outcome | DNN (LSTM) |  $\bullet$ Improved F1-score of best ML methods by 10% |  [l]$\bullet$ Chronological
$\bullet$ No feature engineering required |  |  |  |  |
$\bullet$ Works within and cross projects |  |  |  |  |

Build Validation (BV): Following the preceding CI phases, developers generally
have confidence in the functionalities and performance of developed software
units. At this stage, the software version is ready for merging into the
master branch, representing the final product for building based on the
developed units within this branch. The Build Validation phase focuses on
ensuring the stability of integrated codes before releasing the software for
system testing [47]. Given the significant computational cost associated with
building software products [3], ML-based solutions in this phase primarily
target Build Prediction and reduce building efforts.

Among the 12 studies concentrating on the BV task, eleven predicted the build
outcome using data from the same project (within-project). In contrast, study
S5 used building information from various projects to train a cross-project
model, allowing build prediction for projects lacking sufficient data [48].
Given the substantial focus on build validation, we reviewed published studies
in this area and identified three SOTA methods in terms of the top three
accurate ML models, namely S21, S25, and S38, each employing distinct
strategies.

S21 introduced a method to predict passing builds and save time by skipping
unnecessary building tests (SmartBuildSkip) using a Random Forest method.
Their approach reduced the time spent on building tests by 30%, outperforming
the reviewed ML method in predicting passing builds. It is worth noting that
S21 considered a build as likely to fail if it had failed in the previous
execution, excluding failed builds from evaluations.

In contrast, S25 focused on predicting failing builds, utilizing a decision-
tree-based method (XGBoost) for their BuildFast approach. Results showed a
remarkable 47.5% improvement in the F1-score for predicting failed builds
compared to existing SOTA methods. An advantage of BuildFast over
SmartBuildSkip is its consideration of the chronological order of input data
during model training, enabling its use in real-world environments.

The latest SOTA method in build validation, S38, employed Long Short-Term
Memory (LSTM), a deep learning-based approach, known as DL-CIBuild. DL-
CIBuild, a chronological method, outperformed existing SOTA ML-based methods
in F1-score by 10%. DL-CIBuild has advantages over SmartBuildSkip and
BuildFast: it does not require feature engineering and can be applied to both
intra-project and cross-project build validation tasks. Table 8 offers a
concise overview of these three SOTA methods in build validation in CI.

System Test (ST): In the final step, System Test, the complete software system
undergoes testing to ensure all aspects, including performance, functionality,
and compatibility, are correct. ML-based solutions in this phase are limited,
mainly focusing on Installed Software Discovery for compliance, security, and
efficiency (S12) and generating test cases for Performance Test Optimization
to detect defects in the system (S4).

Process Management (PM): This step involves communication among developers and
the production of numerous documents, ML methods are applied for Activity
Management. S10 classifies CI environment information for project management
committee members, while S17 proposes an ML-based approach for automatically
labeling reported issues, aiding software engineers in assigning the correct
issues to developers. Notably, this phase primarily relies on natural
language-based information, and the limited use of Large Language Models (LLM)
is noteworthy.

In industrial projects, the sequence of Integration and Regression tests may
be altered or combined based on project requirements and priorities, deviating
from the sequence in Figure 5. Selected studies for each CI phase and task are
listed in Table 6.

Summary: $\bullet$ Six phases and ten tasks are identified in CI pipelines:
Unit Test, Integration Test, Regression Test, Build Validation, System Test,
and Process Management. Detailed descriptions of their sequence and
input/output are provided. $\bullet$ Out of the 52 selected studies, 31
focused on Regression Testing and 12 on Build Validation due to their high
costs, emphasizing their critical roles in ensuring software quality. Other CI
phases had nine studies collectively, highlighting the significance of RT and
BV. $\bullet$ Key tasks include Test Optimization in Regression Testing and
Build Prediction in Build Validation. SOTA methods for these tasks are RETECS,
COLEMAN, DeepOrder, SmartBuildSkip, BuildFast, and DL-CIBuild, demonstrating
advancements in CI.

###  4.3 RQ2: Data Sets and Data Engineering Methods

Input data selection, data engineering, and model training are crucial factors
influencing ML model performance [49]. The nature and availability of input
data vary across CI tasks, emphasizing the importance of understanding data
sources and engineering methods.

To facilitate informed decisions, this section offers insights into data
sources, types, and engineering techniques employed in selected studies.
Analyzing this information aids researchers and practitioners in understanding
diverse approaches in ML4CI and discovering potential research directions.

####  4.3.1 Data Sources

In the reviewed literature, 67 diverse data sources were utilized across the
52 selected ML-based CI studies. These sources differ in characteristics like
lines of code (LOC), number of builds, and project specificity. For instance,
study S3 used Google Dagger (848 LOC), contrasting with S2 employing a Samsung
dataset with over 27 million LOC. Similarly, builds in different studies
varied significantly, e.g., the Jazz project in S44 had 199 builds, while S5
used the TravisTorrent dataset with over 300,000 builds. These variations
emphasize the need to comprehend data source characteristics in ML-based CI
research.

Table 9: This table summarizes frequently used study data sources, their
correlation with identified CI tasks in each CI phase, and study IDs. color
intensity in the table shows numerical values, with lighter shades indicating
lower and darker blues representing higher values. The bolded numbers
represent the count of studies and their percentage in total. Acronyms: UT:
Unit Test, IT: Integration Test, RT: Regression Test, BV: Build Validation,
ST: System Test and PM: Process Management.

# Test Cases |  # CI Cycles |  # Test Outcomes |  Fail Test Rate |  |  [l]Unit Test Prediction |  [l]Branch Coverage Prediction |  [l]Integration Test Prediction |  [l]Test Optimization |  [l]Defect Prediction |  [l]Flaky Test Detection |  [l]Build Prediction |  [l]Installed Software Discovery |  [l]Performance Test Optimization |  [l]Activity Management
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Data Properties | CI Datasets | UT | IT | RT | BV | ST | PM
1941 | 320 | 32260 | 28.79% | IOF/ROL | – | – | – |  S7, S11, S14, S19, |  |  |  |  |  |
S20, S22, S28, S31, |  |  |  |  |  |  |  |  |  |  |  |  |  |
S33, S35, S37, S48 |  |  |  |  |  |  |  |  |  |  |  |  |  |
(12, 23.1%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 89 | 352 | 25594 | 19.36% | Paint Control | – | – | – |  S7, S11, S14, S19, |  |  |  |  |  |
S20, S22, S31, S33, |  |  |  |  |  |  |  |  |  |  |  |  |  |
S35, S37, S48 |  |  |  |  |  |  |  |  |  |  |  |  |  |
(11, 21.2%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 457 | 434 | 332650 | 0.02% | Apache Commons | – |  S3, S15 |  |  |  |  |  |  |  |
(2, 3.8%) | – |  S19, S20, S22, S33, |  |  |  |  |  |  |  |  |  |  |  |
S35, S48 (6, 11.6%) | – | – | – | – | – |  S10 |  |  |  |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15 5555 | 336 | 1260618 | 0.25% | GSDTSR | – | – | – |  S7, S14, S16, S22, |  |  |  |  |  |
S31, S33, S35, S37, |  |  |  |  |  |  |  |  |  |  |  |  |  |
S48 (9, 17.3%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 568 | 1312 | 694395 | 0.03% | Google Guava | – |  S3, S15 |  |  |  |  |  |  |  |
(2, 3.8%) | – |  S20, S28, S48 |  |  |  |  |  |  |  |  |  |  |  |
(3, 5.8%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 2010 | 3263 | 781273 | 0.62% | Rails | – | – | – |  S20, S22, S33, S35, |  |  |  |  |  |
S48 (5, 9.7%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 N/A | N/A | N/A | N/A | TravisTorrent | – | – | – | – | – | – |  S5, S6, S21, |  |  |
S29 (4, 7.7%) | – | – | – |  |  |  |  |  |  |  |  |  |  |
1-15 303 | 988 | 815598 | 0.15% | MyBatis | – | – | – |  S20, S33, S48 |  |  |  |  |  |
(3, 5.8%) |  S32 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – |  |  |  |  |  |  |  |  |
1-15 360 | 2257 | 663470 | 0.07% | Google Closure | – | – | – |  S20, S33, S48 |  |  |  |  |  |
(3, 5.8%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 46 | 638 | 14601 | 0.19% | Google Auto | – | – | – |  S20, S33, S48 |  |  |  |  |  |
(3, 5.8%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 106 | 3813 | 204161 | 1.82% | Dspace | – | – | – |  S20, S33, S48 |  |  |  |  |  |
(3, 5.8%) | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-15 N/A | N/A | N/A | N/A | Google Dagger | – |  S3, S15 |  |  |  |  |  |  |  |
(2, 3.8%) | – | – | – | – | – | – | – | – |  |  |  |  |  |

To highlight frequently used sources, 11 publicly available datasets were
employed in more than one study, as indicated by the numbers in parentheses.

To underscore the applicability of the data sources, we have presented 11
datasets that have been utilized in multiple studies and are reported as
publicly available at the time of the studies’ publication. The number in
parentheses indicates the frequency with which each dataset has been utilized
in these studies.

IOF/ROL (12), Paint-Control (11) from ABB Robotics company, Apache-Commons
(9), GSDTSR (9), Google Guava (5), Rails (5), TravisTorrent CI projects (5),
MyBatis (4), Google Closure (3), Google Auto (3), DSpace (3), and Google
Dagger (2).

Table 9 provides a summary of the properties of these datasets, with
particular emphasis on datasets related to testing data in CI. Additionally,
the data properties of Travis Torrent and Google Dagger are presented as N/A
due to their unavailability at the time of writing this paper.

To assist researchers in selecting suitable datasets based on their defined
research problems, this section presents key statistics and summaries of
selected datasets. For instance, ABB company, a prominent industrial robot
supplier, provides robot software and equipment, with datasets such as Paint
Control (PC) and IOF/ROL containing historical information on test results and
over 300 CI cycles. CI cycles encompass all development tasks, including
coding, building, and testing, occurring continuously to ensure code
integration and quality throughout the software development lifecycle before
preparing a software product for deployment [50]. PC includes 89 test cases,
352 CI cycles, 25,594 verdicts, and a 19.36% failure rate, while IOF/ROL
consists of 1941 test cases, 320 CI cycles, 32,260 verdicts, and a 28.79%
failure rate. The GSDTSR dataset, an open resource from Google, has 336 CI
cycles, with a notably low failure rate of around 0.25%. Considering these
datasets alongside the previously discussed sources enables researchers to
make informed choices based on their research objectives and requirements,
facilitating evaluations in diverse dataset scenarios.

Table 9 highlights notable characteristics of datasets. The IOF/ROL and PC
datasets stand out for their higher failure rates, a rarity in CI, making them
popular among researchers. The Apache datasets gain popularity for their good
structure and quality [51]. Additionally, the GSDTSR, Google Guava, Rails,
MyBatis, and Google Closure datasets find frequent use due to their
substantial test outcome volume, as indicated in Table 9.

Table 10: Definition of the identified data types in the selected studies. Acronyms: MD: Meta Data. Data Type |  Description |  ML-based Example
---|---|---
Source Code |  Actual code written by developers that forms the basis of a software application. Usually, it requires preprocessing to be understandable for ML methods. |  Detecting text similarity or source-code coverage by tests to predict the outcome of a test through tokenizing it or making the Abstract Syntax Trees (AST).
1-3 Code MD |  This data type extracts more information than the Source Code data type by considering specific characteristics of the source code. |  Predicting test outcomes by calculating the changes and complexity of the code (e.g. number of changed LOC, or the depth changed module in the inheritance tree)
1-3 Test Case |  This data type covers the textual information of test cases, including names and test codes. |  Prioritizing the execution of tests based on analyzing the test codes.
1-3 Test MD |  This data type is related to the result of test executions and is gathered by analyzing test logs. |  Extracting metadata information such as the time and duration of the test, result, and code coverage of each test case, and selecting a portion of them for execution.
1-3 Commit MD |  This data type refers to all development changes to the system under test (SUT) except the text of the submitted codes. |  Developing a model by using the number of commits, commit time, the branch of code, and the committer’s experience as input data.
1-3 Build Logs |  These logs present historical information about the outcome of previous builds. |  Training an ML model for validating future builds based on analyzing the text of build configurations and their outcomes.
1-3 Project MD |  This data type depicts a holistic view of the project and is usually used in combination with other data types for training ML models. |  The size of the development team and the age of the project are examples of this data type.
1-3 Texts |  This data type includes all texts except source codes, logs, and test codes and requires text processing techniques |  Documents, user stories, reported issues, and commit messages are some examples of this data source.
1-3 System Logs |  System logs are files that show the behavior of the system in different situations and the impact of taking actions in the system. |  Discovering the installed software on cloud systems by analyzing the tree of files and paths

TravisTorrent and Apache datasets are extensively employed in CI research.
TravisTorrent, spanning 1,359 projects (402 Java, 898 Ruby, 59 in other
languages), encompasses 2,640,825 builds. The Apache dataset comprises diverse
projects like Cassandra, Ivy, Lang, Drill, and Math, featuring CI cycles
ranging from 55 to 438. Researchers favor these datasets for their
versatility, enabling evaluations across a broad spectrum of scenarios.

Table 9 illustrates the correlation between frequently used data sources and
identified CI tasks in selected studies. Apache datasets prove versatile and
employed across various CI phases due to project diversity. Conversely,
TravisTorrent is notably applied in Build Prediction, and Google Dagger in
predicting branch coverage during the Integration Test phase.

####  4.3.2 Data Types

This section categorizes and explains data types within CI pipelines. Nine
categories, including Source Code, Code Meta-Data (MD), Test Case, Test MD,
Commit MD, Build Logs, Project MD, Textual Data or Texts, and System Logs, are
identified. Table 10 provides a detailed overview of these data types,
offering valuable insights for research purposes. Integrating innovative data
types could unlock further exploration opportunities.

Here we aim to elucidate the correlation between data types and CI tasks,
providing valuable insights into how different data types can enhance the
efficiency of each task. This understanding informs the development and
deployment of CI strategies, guiding the selection of data types for future
research in automating CI tasks.

Table 11: Relation between data types and CI tasks. color intensity in the
table shows numerical values, with lighter shades indicating lower and darker
blues representing higher values. The bolded numbers represent the count of
studies and their percentage in total. Acronyms: MD: Meta Data, UT: Unit Test,
IT: Integration Test, RT: Regression Test, BV: Build Validation, ST: System
Test and PM: Process Management.

CI Phases | CI Tasks | Source Code | Code MD | Test Case | Test MD | Commit MD | Build Logs | Project MD | Texts | System Logs
---|---|---|---|---|---|---|---|---|---|---
UT |  [l]Unit Test |  |  |  |  |  |  |  |  |
Prediction |  S2 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – | – | – | – |  |
1-11 IT |  [l]Branch Coverage |  |  |  |  |  |  |  |  |
Prediction |  S15 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S3, S15 |  |  |  |  |  |  |  |  |
(2, 3.8%) | – | – | – | – | – | – | – |  |  |
2-11 |  [l]Integration Test |  |  |  |  |  |  |  |  |
Prediction |  S18 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S18 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S18 |  |  |  |  |  |  |  |
(1, 1.9%) |  S23 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – |  S23 |  |  |  |  |  |  |
(1, 1.9%) | – |  |  |  |  |  |  |  |  |
1-11 RT |  [l]Test |  |  |  |  |  |  |  |  |
Optimization |  S8, S39, S47 |  |  |  |  |  |  |  |  |
(3, 5.8%) |  S19, S36, S52 |  |  |  |  |  |  |  |  |
(3, 5.8%) |  S36, S39 |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S7, S8, S11, S16, |  |  |  |  |  |  |  |  |
S19, S22, S26, |  |  |  |  |  |  |  |  |  |
S28, S30, S31, |  |  |  |  |  |  |  |  |  |
S33, S35, S37, |  |  |  |  |  |  |  |  |  |
S39, S41, S45, |  |  |  |  |  |  |  |  |  |
S46, S48, S50, |  |  |  |  |  |  |  |  |  |
S52 (20, 38.5%) |  S28 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S36 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S30, S31 |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S45 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  |  |  |  |  |  |  |  |
2-11 |  [l]Defect |  |  |  |  |  |  |  |  |
Prediction |  S13 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S9, S17, S32 |  |  |  |  |  |  |  |  |
(3, 5.8%) | – |  S13, S14, S32 |  |  |  |  |  |  |  |
(3, 5.8%) |  S9 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – |  |  |  |  |  |
2-11 |  [l]Flaky Test |  |  |  |  |  |  |  |  |
Detection | – | – |  S40 |  |  |  |  |  |  |
(1, 1.9%) |  S40 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – |  |  |  |  |
1-11 BV |  [l]Build |  |  |  |  |  |  |  |  |
Prediction |  S5 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S6, S21, S24, |  |  |  |  |  |  |  |  |
S25, S38, S42, |  |  |  |  |  |  |  |  |  |
S44, S51 |  |  |  |  |  |  |  |  |  |
(8, 15.4%) | – |  S1, S5, |  |  |  |  |  |  |  |
S25, S38 |  |  |  |  |  |  |  |  |  |
(4, 7,7%) |  S1, S5, S6, S21, |  |  |  |  |  |  |  |  |
S24, S25, S43, |  |  |  |  |  |  |  |  |  |
S51 (8, 15.4%) |  S24, S25, |  |  |  |  |  |  |  |  |
S27, S43 |  |  |  |  |  |  |  |  |  |
(4, 7.7%) |  S21 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – |  |  |  |  |  |  |  |
1-11 ST |  [l]Installed Software |  |  |  |  |  |  |  |  |
Discovery | – | – | – | – | – | – | – | – |  S12 |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
2-11 |  [l]Performance Test |  |  |  |  |  |  |  |  |
Optimization | – | – | – |  S4 |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – |  |  |  |  |
1-11 PM |  [l]Activity |  |  |  |  |  |  |  |  |
Management | – | – | – | – |  S10 |  |  |  |  |
(1, 1.9%) | – | – |  S10, S49 |  |  |  |  |  |  |
(2, 3.8%) | – |  |  |  |  |  |  |  |  |
Total usage in studies | 8, 15.4% | 17, 32.7% | 3, 5.8% | 30, 57.7% | 12, 23.1% | 5, 9.6% | 3, 5.8% | 4, 7.7% | 1, 1.9%

Table 11 highlights the prevalent use of source code metadata over raw source
code data in the selected studies. This tendency is likely attributed to the
growing availability of code analysis tools like the CK
tool444https://github.com/mauricioaniche/ck, which automates the computation
of established metrics such as Chidamber and Kemerer (CK). These tools have
significantly streamlined the extraction of source code metadata for
researchers. Notably, S3, S9, and S15 explicitly mentioned utilizing these
tools for extracting CK metrics.

S15, S40, and S44 utilized Halstead metrics with available tools to delve into
source code attributes, enriching the development of ML-based CI solutions.
Extracting source code metadata from source code is cost-effective and
straightforward, involving metrics like comments, methods, lines of code, and
parameters. These metrics are easily obtainable and do not demand extensive
computational resources. Moreover, these data types are well-suited for ML
models as they can be readily converted into numerical values.

Table 11 reveals the prevalent use of source code metadata in studies related
to Build Prediction, Test Optimization, and Defect Prediction in Regression
Testing, as well as in Integration Testing within the CI pipeline. However, it
is notably absent in the context of Unit Test Prediction, signifying a
research gap in this area.

Table 11 underscores the limited utilization of data types such as test case
codes, build logs, project metadata, textual data, and system logs in
automating CI tasks. This underscores the potential benefits of exploring
combinations of various data types for each CI task. For example, combining
system logs with code metadata could enhance the accuracy of the Build
Prediction task in CI. Furthermore, it is noteworthy that test metadata and
commit metadata are predominantly employed in the Build Prediction task,
indicating areas for further research in ML-based CI approaches.

####  4.3.3 Data Preparation

In this section of our literature we mainly focus on data-related techniques
used to modify and prepare raw data for ML models. Data cleaning, a
fundamental aspect, was not explicitly examined separately due to limited
information in the studies. However, data cleaning can be considered a
filtering technique within the data preparation process.

Table 12 shows that the selected studies utilized ten distinct data
preparation techniques. The choice of these techniques and the data’s
characteristics depends on the overall strategy for addressing the research
problem. The table summarizes the employed techniques and their subgroups. The
next section provides a comprehensive description and examples for each data
preparation group and subgroup.

Table 12: Correlation between the most commonly used datasets and data
preparation methods. color intensity in the table shows numerical values, with
lighter shades indicating lower and darker blues representing higher values.
The bolded numbers represent the count of studies and their percentage in
total.

Dataset Names | Conditioning | Building | Balancing | Filtering
---|---|---|---|---
Manual Data |  |  |  |  |  |  |  |  |
Division | Clustering |  Objective Data |  |  |  |  |  |  |  |
Selection |  Data |  |  |  |  |  |  |  |  |
Augmentation |  Data |  |  |  |  |  |  |  |  |
Manipulation | Re-sampling | Oversampling | Undersampling |  Data |  |  |  |  |  |
Pruning |  Selective Data |  |  |  |  |  |  |  |  |
Filtering |  |  |  |  |  |  |  |  |  |
IOF/ROL | – | – | – |  S35 |  |  |  |  |  |
(1, 1.9%) | – |  S35 |  |  |  |  |  |  |  |
(1, 1.9%) |  S28, S37 |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S37 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S11, S14 |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S20 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
Paint Control | – | – | – |  S35 |  |  |  |  |  |
(1, 1.9%) | – |  S35 |  |  |  |  |  |  |  |
(1, 1.9%) |  S37 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S37 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S11, S14 |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S20 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
Apache Commons | – | – | – |  S15, S35 |  |  |  |  |  |
(2, 3.8%) |  S10 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S35 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – |  S20 |  |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
GSDTSR | – | – |  S16 |  |  |  |  |  |  |
(1, 1.9%) |  S35 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S35 |  |  |  |  |  |  |  |
(1, 1.9%) |  S37 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S37 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S14 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  |  |  |  |  |  |  |  |
Google Guava | – | – | – |  S15 |  |  |  |  |  |
(1, 1.9%) | – | – |  S28 |  |  |  |  |  |  |
(1, 1.9%) | – | – |  S20 |  |  |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
Rails | – | – | – |  S35 |  |  |  |  |  |
(1, 1.9%) | – |  S35 |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – |  S48 |  |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
TravisTorrent |  S5, S21 |  |  |  |  |  |  |  |  |
(2, 3.8%) | – | – |  S29 |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – |  S6 |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
MyBatis | – | – | – | – | – | – | – | – |  S32 |
(1, 1.9%) |  S20 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
Google Closure | – | – | – | – | – | – | – | – | – |  S20
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
Google Auto | – | – | – | – | – | – | – | – | – |  S20
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
Dspace | – | – | – | – | – | – | – | – | – |  S20
(1, 1.9%) |  |  |  |  |  |  |  |  |  |
Google Dagger | – | – | – |  S15 |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – | – |  |  |  |
Other |  S42 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S1, S24, S45 |  |  |  |  |  |  |  |  |
(3, 5.8%) |  S2, S40 |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S8, S12, S13, |  |  |  |  |  |  |  |  |
S26, S36, S49, |  |  |  |  |  |  |  |  |  |
S43, S46 |  |  |  |  |  |  |  |  |  |
(8, 15.4%) |  S23 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S38, S44 |  |  |  |  |  |  |  |
(2, 3.8%) |  S30 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S9, S34, |  |  |  |  |  |  |  |  |
S51, S49 |  |  |  |  |  |  |  |  |  |
(4, 7.7%) |  S24, S25 |  |  |  |  |  |  |  |  |
(2, 3.8%) |  |  |  |  |  |  |  |  |  |
Num of studies | 3, 5.8% | 3, 5.8% | 3, 5.8% | 11, 21.2% | 2, 3.8% | 1, 1.9% | 4, 7.7% | 2, 3.8% | 7, 13.5% | 5, 9.6%

DP1) Conditioning: This technique adjusts the dataset according to its
characteristics and the proposed solution. In the case of manual data
division, as demonstrated in study S5, the TravisTorrent dataset was
partitioned using strategies such as the Burak filter [48]and the Bellwether
filter [48]. In another instance (S21), data was segregated based on previous
build results (pass or fail), resulting in separate training of machine
learning models for each data partition.

Moreover, clustering methods like k-means can divide data based on feature
values or data distribution, as in studies S24 and S45. Objective data
selection involves choosing specific data points or segments based on defined
objectives or criteria of our solution, such as selecting lines before and
after the changed codes (changed codes are the objective of our solution) for
inclusion in the dataset.

Utilizing conditioning for data preparation presents a clear advantage in
reducing the computational overhead during ML model training, especially in CI
environments with significant data volumes [42].

DP2) Building: This methodology transforms data to enhance its structure for
better compatibility with ML techniques, especially in diverse CI
environments. It consists of two sub-groups.

The first sub-group, data augmentation, is the most commonly used method.
Researchers employ it to modify data structures or simulate real-world
conditions. For instance, in S13, input data is padded with zeros for
consistency and easier processing. Techniques like creating graphs or trees
based on regression test results (S29, S36, S46) or simulating cloud-based
real environments (S12) fall into this sub-group.

The second sub-group, data manipulation, involves methods such as text
processing (S23) and merging distinct parts of a dataset by identifying
correlations, such as combining issue tracking and VCS (S10).

DP3) Balancing: Many commonly used data sources display imbalances, which can
substantially impact the performance and accuracy of classifiers [52].
Addressing this issue involves employing re-sampling techniques or training
separate models for each class of input data. Re-sampling encompasses both
oversampling, which involves increasing instances in the minority class (e.g.,
SMOTE), and under-sampling, as demonstrated in S30, which reduces the number
of instances in the majority class. Additionally, combining both methods, as
exemplified in SMOGN in S37 [53], presents another viable approach.

DP4) Filtering: Data pruning, a form of data cleaning, is vital in CI,
addressing high computational workloads in large-scale datasets and improving
the performance of the ML models [54]. Seven studies used this method, e.g.,
S11 removed unexecuted tests, S51 eliminated errored builds, and S9 removed
incomplete test data. Issues like these arise from evolving software practices
[55] and human errors [54]. Filtering is crucial in build validation studies,
tackling interrupted builds [46]. Selective data filtering can enhance the
quality of the training model and reduce training costs by prioritizing data
segments that closely resemble the practical environment where the trained ML
model will be deployed. For example, in study S24, TravisTorrent was
constrained to Java-based projects employing Ant, Maven, and Gradle CI build
tools, while Ruby-based projects were excluded.

Summary: $\bullet$ 12 datasets, nine data types, and ten processing techniques
are identified in the reviewed studies. $\bullet$ Clustering datasets can
reduce computation overhead and enhance ML model accuracy due to the high
volume of CI data. $\bullet$ Addressing class imbalance is crucial to improve
ML model accuracy in CI data. $\bullet$ Filtering, addressing exceptional
cases like cancelled or errored processes, enhances ML model performance and
reduces computational overhead.

###  4.4 RQ3: Extracted Features

This section delves into the feature types and feature engineering techniques
employed in the reviewed studies. Given that ML models are data-driven, the
choice of feature types and engineering methods directly impacts the
performance of these models [49]. Through thematic analysis, we have
categorized Features (FT) into four primary groups and nine sub-categories,
presented as follows:

Relational: Individual and Spatial, Statistical: Numerical, Components and
Context, Lexical: Content and Syntactical, and Epochal: Temporal and
Narrative.

In the following, the details regarding these features are presented. A brief
explanation and examples for these features and the relationship between
feature types and CI tasks are presented in Table 13 and 14 respectively. Note
that five studies lacked comprehensive feature descriptions. Additionally,
feature combinations that were not employed in any of the selected studies
have been excluded from Table 14. These omitted combinations encompass the use
of Relational features, the utilization of both Relational and Statistical
features, the combination of both Relational and Lexical features, as well as
the usage of Statistical and Lexical features.

Table 13: Employed features in studies and description of them Feature types | Sub groups | Description | Example
---|---|---|---
Relational |  Individual |  These features focus on the behaviour and experiences of individuals. |  (S1) Percentage of ownership of a developer on a file
2-4 |  Spatial |  These features encompass the communication pathways and interdependencies among distinct components within the space of software system. |  (S3) Source code coverage of a testing code
Statistical |  Components |  These features encapsulate statistical data pertaining to various aspects of software components, offering insights into their quantitative characteristics. |  (S3) Depth of inheritance tree (DIT)
2-4 |  Context |  These features represent the statistical information within a code including the minimum, maximum, and average of operations or the complexity of components. |  (S44) Number of operands and operators in the committed code
2-4 |  Numerical |  These features are based on calculating straightforward metrics not included in the Components and Context feature types, and they are independent of content. |  (S26) Number of commits on a file
Lexical |  Content |  These features denote the information about the terms and tags in texts including source code, log files, and other text files and reports. |  (S8) Determining text similarities in Java codes via TF-IDF
2-4 |  Syntactical |  These features only focus on information about the specific programming language reserved words. |  (S15) Number of each Java reserved word in a source code
Epochal |  Temporal |  These features represent the time-dependent attributes of any software components from last changes until the present time. |  (S21) The time gap since the last build
2-4 |  Narrative |  These features represent the historical actions taken and their outcomes leading up to a particular event within the software product. |  (S36) The durations of the previous executions of a test case

FT1) Relational: This category encompasses features that depict relationships
among elements, such as actions, individuals, and components, influencing the
outcomes of CI tasks. It consists of two sub-categories:

_Individual:_ These features focus on individual-related factors, such as
developers and test designers, including their experience in software
development and file ownership percentages [37]. Although pivotal in
predicting build outcomes (S43) and testing results of code commits (S23),
only four studies employed these features.

_Spatial:_ Spatial features relate to code concepts like coverage, coupling,
inheritance, and cohesion, extractable using tools like IntelliJ Idea [56],
Rational Software Analyzer (RSA) [47], and Aniche [57] with low computational
overhead. Notably, three studies utilized the Chidamber and Kemerer (CK)
indices [58]. This feature type found diverse applications, including
predicting branch coverage (S3, S15), identifying code defects (S9), detecting
flaky tests (S40), predicting build outcomes (S44), and optimizing test cases
(S36, S39, S45, S46).

FT2) Statistical:

These features, are computed and analyzed for statistical data. In a broader
context, they serve to show the complexity and scope of projects or committed
changes. Table 14 highlights the usage of these features in the Build
Prediction task, primarily due to their lower computational overhead,
especially in the case of the Numerical feature type. Notably, build
prediction tasks due to its full coverage of all tests involve huge data from
numerous CI cycles. So, making minimizing computational resources is a serious
concern in this task.

_Components:_ Statistical insights into software product composition, such as
the number of concrete and abstract classes, functions, and software package
dependencies, are provided by Components features. They are useful in
predicting branch coverage (S3, S15), defect prediction (S32), build outcomes
(S42), flaky test detection (S40), and test optimization based on mutation
testing (S29).

_Context:_ This category yields statistical properties offering information
about the complexity of software and testing source code. It computes
attributes from the source code, including Osmax and Osavg (maximum and
average operation size), WMC (weighted method complexity), NOAC (number of
operations added), Ocmax and Ocavg (maximum and average operation complexity),
Opavg (average operation parameters), CSO (class size operations), CSOA (class
size operations attributes), CSA (class size attributes), Query (number of
queries), NAAC (number of attributes added), NOIC (number of operations
inherited), NOOC (number of operations overridden). It also incorporates the
well-known Halstead metrics designed to show program complexity by examining
its operators and operands (N, E, V, D, B, n) [59]. These features are
prevalently used in predicting build outcomes (S27, S42, S43, S44), test
branch coverage (S15), and detecting flaky tests (S40).

_Numerical:_ Known for computational simplicity, Numerical features are
utilized in 23 out of 52 studies for tasks like test optimization (S22, S26,
S29, S30, S36, S39), build prediction (S1, S5, S6, S21, S24, S25, S42, S43,
S44), test outcome prediction (S23), code defect prediction (S9, S17, S32), CI
data management (S10), and flaky test detection (S40). These features
encompass various attributes like counts of code line changes, files, sub-
systems, classes, methods, active authors on the same file, commits, pull
requests in VCS, and comments in source codes and VCS.

Table 14: Correlation between CI tasks and feature types. color intensity in
the table shows numerical values, with lighter shades indicating lower and
darker blues representing higher values. The bolded numbers represent the
count of studies and their percentage in total. The symbol represents which
feature types have been used, while the symbol represents which feature types
are not used. Acronyms: UT: Unit Test, IT: Integration Test, RT: Regression
Test, BV: Build Validation, ST: System Test and PM: Process Management.

Feature Types | UT | IT | RT | BV | ST | PM |
---|---|---|---|---|---|---|---
[l]Relational |  [l]Statistical |  [l]Lexical |  [l]Epochal |  [l]Unit Test Prediction |  [l]Branch Coverage Prediction |  [l]Integration Test Prediction |  Test Optimization |  Defect Prediction |  [l]Flaky Test Detection |  Build Prediction |  [l]Installed Software Discovery |  [l]Performance Test Optimization |  [l]Activity Management |  [l]Number of studies
|  |  |  | – | – | – |  S29 (1, 1.9%) | – | – |  S5, S25, S42, |  |  |  |
S44 (4, 7.7%) | – | – |  S10 |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  6 |  |  |  |  |  |  |  |  |  |  |  |  |
11.5% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  |  S2 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S18 |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S47 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S13 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – |  S12 |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S49 |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  6 |  |  |  |  |  |  |  |  |  |  |  |  |
11.5% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – | – | – |  S7, S11, S14, S16, |  |  |  |  |  |  |
S19, S28, S31, S33, |  |  |  |  |  |  |  |  |  |  |  |  |  |
S35, S37, S48, S50 |  |  |  |  |  |  |  |  |  |  |  |  |  |
(12, 23.1%) | – | – | – |  S4 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – |  13 |  |  |  |  |  |  |  |  |  |  |
25% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – | – | – |  S45, S46 |  |  |  |  |  |  |
(1, 3.8%) |  S17, S32 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 3.8%) | – | – | – | – | – |  4 |  |  |  |  |  |  |  |
7.7% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – | – | – |  S22, S52 |  |  |  |  |  |  |
(1, 3.8%) | – | – |  S6, S21, S24, |  |  |  |  |  |  |  |  |  |  |
S27 (4, 7.7%) | – | – | – |  6 |  |  |  |  |  |  |  |  |  |
11.5% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – | – | – |  S8 |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – | – |  1 |  |  |  |  |  |  |
1.9% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – |  S15 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – | – | – | – |  1 |  |  |  |  |
1.9% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – | – | – |  S36, S39 |  |  |  |  |  |  |
(1, 3.8%) |  S9 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S40 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – |  4 |  |  |  |  |  |  |  |  |
7.7% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – | – | – | – | – | – |  S1, S43 |  |  |  |
(1, 3.8%) | – | – | – |  2 |  |  |  |  |  |  |  |  |  |
3.8% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – | – | – |  S26 |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – | – |  1 |  |  |  |  |  |  |
1.9% |  |  |  |  |  |  |  |  |  |  |  |  |  |
1-15  |  |  |  | – |  S3 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S23 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S30 |  |  |  |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – | – |  3 |  |  |  |  |  |  |
5.8% |  |  |  |  |  |  |  |  |  |  |  |  |  |

FT3) Lexical: Software project source code and log files often contain
valuable hidden information, extractable through lexical analysis, which
involves two distinct approaches:

_Content:_ Content-based feature extraction methods are employed in 10 out of
52 studies. Their popularity stems from direct data extraction without
computationally intensive processes. This feature is versatile as it is
related to text analysis, making it applicable to all programming languages
through various textual analysis techniques. These techniques include TF-IDF
(Term Frequency–Inverse Document Frequency) (S8, S23, S49), lexical
tokenization (S2, S12, S26, S30), Bag-of-Words (S13, S47), and Word Embedding
(S13).

_Syntactical:_ These features are used in two CI studies, specifically for
estimating branch coverage in automated testing (S3, S15). Analysis in this
category is based on a predefined list of words, often reserved keywords,
making them suitable for extracting semantic information from data.

FT4) Epochal: Epochal features, tied to temporal and narrative information,
dominate with 35 out of 52 CI studies incorporating them. This prevalence is
particularly notable in studies emphasizing test case optimization,
underlining the significance of analyzing prior execution history for
optimizing test case execution.

_Temporal:_ While this feature cannot be used solely for decision-making and
model training, temporal features critically reveal hidden temporal patterns
in data and usually must be used with other features. Encompassing work habits
(time, day, and month of actions) (S21, S32), time intervals between the
current and the previous event (S21, S43), and the duration of the last event
(S37), these features concentrate on events immediately preceding an
occurrence. This differs from the broader historical context considered by
“Narrative” features. They have also contributed to research on order-
dependent and non-order-dependent flaky tests (S40).

_Narrative:_ Narrative features, present in 32 out of 52 studies, are
extensively utilized due to their ability to encapsulate valuable insights
from past experiences and lessons learned in ML-based CI enhancements [60].
For example, in test case prioritization, the likelihood of test cases failing
in the future is higher if they have previously failed, justifying their
elevated priority in subsequent tests [61]. Narrative features include
attributes like changes in files or text, outcomes of prior test executions,
the ratio of failed-to-pass tests, and test execution durations. Notably,
features like the Failure Distance attribute (FD - the number of builds since
the last failed build) are straightforward to calculate which is used in S21.

Table 14 shows that relational features are often combined with other feature
types, with only three studies employing all four feature types in their
analysis. Additionally, lexical features find extensive application across
various tasks.

####  4.4.1 Feature Engineering Techniques

Feature engineering is a crucial step in ML model development [26]. Its
significance lies in two key factors:

Feature engineering techniques significantly impact the accuracy of trained ML
models [62]. These techniques transform raw data into a format interpretable
and meaningful for ML models [26]. In our analysis of the reviewed papers, we
identified five groups of feature engineering (FE) techniques. However, it is
worth noting that among the 52 studies in this systematic literature review,
18 did not provide details on their feature engineering techniques.

Table 15: Relation between the most frequently used data sets and feature engineering techniques. Note: The MyBatis, Google Closure, Google Auto and Dspace datasets did not report any feature engineering techniques. So, we did not present them in this table. The bolded numbers represent the count of studies and their percentage in total. Datasets |  Imputation |  |  |
---|---|---|---|---
and |  |  |  |
Elimination |  Feature |  |  |
Enhancement |  |  |  |
and Labeling |  Outliers |  |  |
Handling |  Feature |  |  |
Scaling |  Feature |  |  |
Tagging and |  |  |  |
Encoding |  |  |  |
IOF/ROL |  S11, S37 |  |  |
(2, 3.8%) | – | – |  S19, S22, S37 |
(3, 5.8%) |  S19 |  |  |
(1, 1.9%) |  |  |  |
Paint Control |  S11, S37 |  |  |
(2, 3.8%) | – | – |  S19, S22, S37 |
(3, 5.8%) |  S19 |  |  |
(1, 1.9%) |  |  |  |
Apache Commons |  S10, S15 |  |  |
(2, 3.8%) |  S10 |  |  |
(1, 1.9%) | – |  S3, S15, S19, S22 |  |
(4, 7.7%) |  S19 |  |  |
(1, 1.9%) |  |  |  |
GSDTSR |  S37 |  |  |
(1, 1.9%) | – | – |  S22, S37 |
(2, 3.8%) | – |  |  |
Google Guava |  S15 |  |  |
(1, 1.9%) | – | – |  S3, S15 |
(2, 3.8%) | – |  |  |
Rails | – | – | – |  S22
(1, 1.9%) | – |  |  |
TravisTorrent |  S5 |  |  |
(1, 1.9%) |  S24 |  |  |
(1, 1.9%) |  S21 |  |  |
(1, 1.9%) |  S24 |  |  |
(1, 1.9%) | – |  |  |
Google Dagger |  S15 |  |  |
(1, 1.9%) | – | – |  S3, S15 |
(2, 3.8%) | – |  |  |

FE1) Imputation and Elimination: This technique addresses missing values in a
dataset by replacing them with calculated or similar values, preventing data
loss due to missing features [63]. In CI, it is particularly useful for
handling exceptions. For example, S9 calculated the LOC difference when the
“Number of Modified Lines” was unavailable. It can also derive values by
considering interdependencies among data rows, such as finding the start and
end times of test cases based on the log of previous execution times.
Moreover, to reduce features and lower ML model training costs, elimination
methods like the chi-square method can be used to remove closely related
features. For instance, authors retained one of two similar features: the
number of modified lines or added lines per data entry [53]. Given the large
CI data volume, dimensional reduction techniques, as seen in S13, can further
reduce computational overhead and input features in ML methods.

FE2) Feature Enhancement and Labeling: In CI environments, the presence of
diverse data types and their high volume poses a challenge for data analysis.
To tackle this challenge, various strategies are employed to enhance data
comprehensibility for training ML models and improving their performance. In
S10, a new labeling scheme categorized software management profile activities
into H(high), M(edium), and L(ow). S46 categorized test cases as entirely or
partially redundant based on coverage analysis, augmenting data with
additional features. This methodology allows for enriching datasets with
contextual information; for instance, S37 labeled commit dates to indicate
holidays or regular days. Additionally, S16 introduced a new feature
representing the percentage of detected failures relative to the total
failures for each test case.

FE3) Outliers Handling: In CI environments, unexpected data may arise due to
errors or non-repeatable situations, often attributed to human errors or
specified limitations. For instance, test case durations may be constrained,
and exceeding this time limit results in exceptional values, known as
outliers. Outliers can be detected by defining thresholds or cutoff
parameters, as demonstrated in S13 and S18.

FE4) Feature Scaling: Feature values exhibit diverse distributions and ranges,
necessitating normalization through statistical methods. This technique
rescales data to fit within a specific range (commonly [0, 1]) or standardizes
values to the same magnitude using methods like log transformation, which
mitigates the impact of extremely high or low values. For instance, S3 applied
log transformation in conjunction with z-score, although they did not specify
which feature required this transformation. Feature scaling is widely
employed, with 20 out of 34 studies explaining their feature engineering
techniques using this method, primarily due to the diversity of feature values
and the prevalence of numerical features in CI environments.

FE5) Feature Tagging and Encoding: CI datasets often involve nominal values,
such as test results like “pass”, “fail”, or “canceled”. To facilitate the
training of ML models, input data must be made comprehensible, as these models
rely on mathematical formulas [27]. One approach is to assign 0 and 1 values
to class labels or represent feature values with binary strings or tags based
on predefined rules [62, 64].

While encoding applies to various data types, it is particularly common in
text-based datasets, involving the assignment of tokens or tags to different
text segments. Effective tokenization requires a thorough understanding of the
data and the selection of appropriate tokens for each distinct group or
category. For example, in S13, authors defined specific tokens (tags) for
various groups in source code, such as variables, operands, data types,
spaces, and more.

In Table 15, we illustrate the relationship between feature engineering
methods and the most commonly used datasets in the literature. It is
noteworthy that certain datasets, in comparison with those presented in Table
9, are omitted here because the referenced studies did not report their
feature engineering methods.

Summary: $\bullet$ Narrative, Numerical, Content, and Temporal feature types
are used more frequently in the selected studies in comparison with other
feature types. $\bullet$ Tools like IntelliJ Idea, RSA, and Aniche can extract
features from code with low computational overhead. $\bullet$ New features can
be defined through more complex computations and raw code analysis such as
tagging or defining tree-based structures. $\bullet$ Feature engineering
techniques like scaling, outlier handling, and elimination are still useful in
CI environments. They cut computational load and boost ML model accuracy and
performance.

###  4.5 RQ4: Model Training and Tuning

The performance and training time of ML methods, as well as their interactions
with input data, are influenced by their inherent characteristics and
predefined parameters [65]. In this section, our emphasis is on improving ML
model training in the context of CI and refining algorithms through
hyperparameter tuning.

####  4.5.1 Types of Learning Algorithms

Various ML algorithms, including supervised, unsupervised, semi-supervised,
and reinforcement learning, play vital roles in ML-based approaches. Brief
introductions to these methods are presented in the following.

Supervised Learning: This approach involves training a model on labelled data
[17], requiring human effort or pre-existing data labelling before model
training [9]. Supervised learning comprises two primary classes:
classification, where the model categorizes outputs into fixed or discrete
classes, and regression, where the model predicts continuous values [66].

Table 16: Mapping ML Algorithms to CI Tasks. The table color intensity
represents study counts, with lighter shades indicating fewer studies and
darker blues representing higher counts. The bolded numbers represent the
count of studies and their percentage in total. Acronyms: NN: Neural Network,
SVM: Support Vector Machine, DT: Decision Tree, RL: Reinforcement Learning,
KM: K-Means, KNN: K-Nearest Neighbors, LR: Linear Regression, NB: Naive Bayes,
TL: Transfer learning, UT: Unit Test, IT: Integration Test, RT: Regression
Test, BV: Build Validation, ST: System Test and PM: Process Management.

Algorithms |  [l]Unit Test Prediction |  [l]Branch Coverage Prediction |  [l]Integration Test Prediction |  Test Optimization |  Defect Prediction |  [l]Flaky Test Detection |  Build Prediction |  [l]Installed Software Discovery |  [l]Performance Test Optimization |  [l]Activity Management |  [l]Total number of studies
---|---|---|---|---|---|---|---|---|---|---|---
| UT | IT | RT | BV | ST | PM |
NN |  S2 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S3, S15 |  |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S18 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S7, S11, S26, S37, |  |  |  |  |  |  |  |  |  |
S39, S41 (6, 11.5%) |  S13, S32 |  |  |  |  |  |  |  |  |  |
(2, 3.8%) | – |  S38 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S4 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  14 |  |  |  |  |  |  |  |  |
26.9% |  |  |  |  |  |  |  |  |  |  |
Huber | – |  S3 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – | – | – | – |  1 |  |
1.9% |  |  |  |  |  |  |  |  |  |  |
SVM | – |  S3, S15 |  |  |  |  |  |  |  |  |
(2, 3.8%) | – |  S8, S26, S36 |  |  |  |  |  |  |  |  |
(3, 5.8%) |  S32 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – |  S49 |  |  |  |  |  |
(1, 1.9%) |  7 |  |  |  |  |  |  |  |  |  |
13.5% |  |  |  |  |  |  |  |  |  |  |
DT | – |  S15 |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S18, S23 |  |  |  |  |  |  |  |  |  |
(2, 3.8%) |  S16, S26, S29, S30, |  |  |  |  |  |  |  |  |  |
S36, S46, S47 |  |  |  |  |  |  |  |  |  |  |
(7, 13.5%) |  S9, S13, S32 |  |  |  |  |  |  |  |  |  |
(3, 5.7%) |  S40 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) |  S1, S5, S21, S24, |  |  |  |  |  |  |  |  |  |
S25, S27, S42, |  |  |  |  |  |  |  |  |  |  |
S43, S44, S51 |  |  |  |  |  |  |  |  |  |  |
(10, 19.2%) | – | – |  S49 |  |  |  |  |  |  |  |
(1, 1.9%) |  25 |  |  |  |  |  |  |  |  |  |
48.1% |  |  |  |  |  |  |  |  |  |  |
RL | – | – | – |  S7, S14, S20, S24, |  |  |  |  |  |  |
S27, S28, S31, S33, |  |  |  |  |  |  |  |  |  |  |
S34, S35, S50, S48 |  |  |  |  |  |  |  |  |  |  |
(12, 23.1%) | – | – | – |  S12 |  |  |  |  |  |  |
(1, 1.9%) | – | – |  13 |  |  |  |  |  |  |  |
25% |  |  |  |  |  |  |  |  |  |  |
KM | – | – | – |  S45 |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – |  S10 |  |  |  |  |
(1, 1.9%) |  2 |  |  |  |  |  |  |  |  |  |
3.8% |  |  |  |  |  |  |  |  |  |  |
KNN | – | – | – |  S16, S30 |  |  |  |  |  |  |
(2, 3.8%) | – | – |  S5 |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – |  3 |  |  |  |  |  |  |
5.8% |  |  |  |  |  |  |  |  |  |  |
LR | – | – | – |  S16, S26, S30 |  |  |  |  |  |  |
(3, 5.7%) |  S32 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S1, S5 |  |  |  |  |  |  |  |  |
(2, 3.8%) | – | – | – |  6 |  |  |  |  |  |  |
11.5% |  |  |  |  |  |  |  |  |  |  |
NB | – | – | – |  S16 |  |  |  |  |  |  |
(1, 1.9%) |  S32 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – |  S5 |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – |  S49 |  |  |  |  |  |  |  |
(1, 1.9%) |  4 |  |  |  |  |  |  |  |  |  |
7.7% |  |  |  |  |  |  |  |  |  |  |
TL | – | – | – |  S52 |  |  |  |  |  |  |
(1, 1.9%) |  S17 |  |  |  |  |  |  |  |  |  |
(1, 1.9%) | – | – | – | – | – |  2 |  |  |  |  |
7.7% |  |  |  |  |  |  |  |  |  |  |

Unsupervised learning: Unsupervised learning algorithms, unlike supervised
learning, do not require data labeling before training. These models uncover
data relationships and cluster data points [17]. Unsupervised clustering
algorithms are particularly suited for large datasets where manual data
labeling efforts are impractical [17]. Semi-supervised learning: Semi-
supervised learning algorithms leverage labeled data to classify unlabeled
data by identifying underlying data relationships [67], making them more
practical when dealing with imprecise or noisy datasets containing both
labeled and unlabeled data [68]. Lastly, Reinforcement Learning (RL)
algorithms learn through trial and error, receiving rewards or penalties at
each step until they achieve the desired output or accuracy [66]. RL
algorithms benefit from continuous model updates, making them suitable for
dynamically changing CI environments and data.

Among the primary studies, supervised learning is the most commonly employed
approach, with 44 out of 52 studies utilizing it. Unsupervised learning is the
second most used, with five out of 52 studies employing this approach, and one
study using both supervised and unsupervised ML methods. Additionally, four
studies applied semi-supervised learning algorithms, while 12 studies employed
RL algorithms. Our observation also indicates a significant increase in the
usage of RL algorithms in “Test Optimization” tasks since 2020. Notably, two
out of the three state-of-the-art methods in this task (COLEMAN and RETECS,
see Table 7) are RL-based methods, underscoring the growing applicability of
RL-based algorithms in the realm of test optimization.

Our analysis also reveals that classification algorithms are the most widely
employed methods in the CI context, with 32 out of 52 studies utilizing them.
This preference can be attributed to the characteristics of testing, where ML-
based binary classifiers excel in predicting test outcomes and build results
without the need for explicit execution. Additionally, one study (S49) in the
CI environment employed multi-class classification methods to categorize
reported issues into five classes, aiding developers in identifying and
addressing bug-related issues based on the most frequent labels in the
training dataset.

Table 16 demonstrates that Decision Tree (DT) algorithms are the prevailing
classification method in the CI environment, with 25 out of 52 studies
employing them. The popularity of DT algorithms can be attributed to several
factors, two of which are derived from the selected studies, while the third
is drawn from the literature.

First, CI environments continuously generate vast amounts of data, and ML
models require frequent updates [69, 70]. Training and updating DT algorithms
demand low computational resources, making them a feasible choice in CI
settings.

Second, DT algorithms have high performance in classifying unseen data [47].

Third, DT algorithms are interpretable and easily comprehensible for human
users [71].

Table 16 shows that, except for one study (S12), RL algorithms have primarily
been used in regression testing (RT) tasks within CI environments. In RT, a
predefined set of test cases is established, and new test cases are
incrementally added as new features are developed for the final software
product. The objective of RT is to identify test cases with failure outcomes
and prioritize their execution before those with passing outcomes. RL
algorithms are well-suited for RT because this CI phase can be formulated as a
sequential decision-making problem, a key characteristic of RL-based solutions
[44]. Moreover, RL algorithms adapt to new data more effectively with frequent
updates compared to other supervised and unsupervised ML algorithms [62].

However, researchers and practitioners should consider the advantages and
disadvantages of RL-based models. For instance, the ROCKET solution [72] faces
scalability issues in long runtime, while the RETECS method [73], an RL-based
solution, requires a substantial amount of time for training.

Alongside the widespread adoption of DT and RL algorithms in regression
testing, Table 16 highlights the utilization of Neural Network (NN) algorithms
in 14 studies. Remarkably, NN algorithms have been applied in 7 out of 10 CI
tasks, underscoring their flexibility in addressing various challenges within
CI. The appeal of NN algorithms, despite their need for significant
computational resources during training, lies in their capacity to
automatically extract features from datasets and their high predictive
accuracy [70].

####  4.5.2 Hyper-Parameter Tuning

In general, ML models comprise a basic formula that requires configuration by
identifying the optimal values for their hyper-parameters [65]. Selected
studies present varying perspectives on hyperparameter tuning. Some emphasize
the importance of hyperparameter tuning in achieving the optimal performance
of trained models [43], considering skipping this process as a potential
threat that can affect model accuracy [6, 45]. However, other studies, like
[74], found that tuning hyperparameters did not significantly impact model
accuracy. Additionally, Al-Sabbagh et al. [41] reported that automatic
hyperparameter tuning tools can be time-consuming, prompting them to manually
tune hyperparameters to save time.

It is worth noting that hyperparameter tuning can affect model training speed.
For instance, adjusting the ‘training rate’ hyperparameter can increase model
instability while reducing training time or decrease training speed while
enhancing overall model performance stability [27]. Hence, our investigation
delves into how the selected studies in the application of ML algorithms
within CI environments performed hyperparameter tuning.

Based on the data extracted from the selected studies, we categorized the
hyper-parameter tuning strategies (HT) into five groups.

HT1) The first group used searching methods to find the best hyperparameters.
These studies employed methods such as Grid Search (GS), Bayesian Search (BS),
and Genetic Algorithm (GA).

In GS, researchers fix the domain of hyperparameters, and the algorithm finds
the best combination from these fixed values [75]. In BS, the next
hyperparameter values are determined based on the evaluation results of the
previous values, avoiding unnecessary evaluations [76]. In GA, hyperparameters
are adjusted iteratively by evaluating the model’s performance and making
slight changes [77].

In S38, a unique Genetic Algorithms-based hyperparameter tuning method is
utilized. Random values were assigned to each hyperparameter, and a single-
point crossover operator generated two new offspring in the Crossover step.
The best solutions were retained using the elitism method and fitness function
calculation. In the Mutation step, hyperparameter values were slightly
modified, and the Crossover step was repeated. This process continued until it
reached the predetermined stopping criteria. Table 17 details the
hyperparameter tuning methods used for ML models, acknowledging that one
tuning method might be applied to multiple ML models in studies utilizing more
than one ML method.

Table 17: Summary of employed hyper-parameter strategies and mapping to the ML
methods. color intensity corresponds to study counts, with lighter shades
indicating lower and darker blues representing higher counts. The bolded
numbers represent the count of studies and their percentage in total.
Acronyms: NN: Neural Network, SVM: Support Vector Machine, DT: Decision Tree,
RL: Reinforcement Learning, KM: K-Means, KNN: K-Nearest Neighbors, LR: Linear
Regression, NB: Naive Bayes and TL: Transfer learning, GS: Grid Search, GA:
Genetic Algorithm and BS: Bayesian Search

Alg. | Methods | Default | Literature | Test-and-Trial | Formula | Did not report
---|---|---|---|---|---|---
NN |  S15(GS), S38(GA), |  |  |  |  |
S39(BS) |  |  |  |  |  |
(3, 5.8%) |  S26 |  |  |  |  |
(1, 1.9%) |  S7 |  |  |  |  |
(1, 1.9%) |  S2, S3, S4, S13, |  |  |  |  |
S18, S37, S41 |  |  |  |  |  |
(7, 13.4%) |  S11 |  |  |  |  |
(1, 1.9%) |  S32 |  |  |  |  |
(1, 1.9%) |  |  |  |  |  |
Huber | – | – | – |  S3 |  |
(1, 1.9%) | – | – |  |  |  |
SVM |  S15(GS) |  |  |  |  |
(1, 1.9%) |  S26 |  |  |  |  |
(1, 1.9%) | – |  S3, S49 |  |  |  |
(2, 3.8%) | – |  S8, S32, S36 |  |  |  |
(3, 5.7%) |  |  |  |  |  |
DT |  S15(GS), S29(GS), |  |  |  |  |
S30(BS) |  |  |  |  |  |
(3, 5.7%) |  S17, S23, S25, |  |  |  |  |
S26, S47, S51 |  |  |  |  |  |
(6, 11.5%) | – |  S9, S13, S49, |  |  |  |
S18, S46, S16 |  |  |  |  |  |
(6, 11.5%) |  S24, S44 |  |  |  |  |
(2, 3.8%) |  S1, S5, S21, S27, S32, |  |  |  |  |
S36, S40, S42, S43 |  |  |  |  |  |
(9, 17.3%) |  |  |  |  |  |
RL | – |  S19, S35 |  |  |  |
(2, 3.8%) |  S20, S48, S7, S28, |  |  |  |  |
S34 (5, 9.6%) | S14, S22 |  |  |  |  |
(2, 3.8%) | – |  S12, S31, S33 |  |  |  |
(3, 5.8%) |  |  |  |  |  |
KM | – |  S10 |  |  |  |
(1, 1.9%) | – | – | – |  S45 |  |
(1, 1.9%) |  |  |  |  |  |
KNN |  S30(BS) |  |  |  |  |
(1, 1.9%) | – | – |  S16 |  |  |
(1, 1.9%) | – |  S5 |  |  |  |
(1, 1.9%) |  |  |  |  |  |
LR |  S30(BS) |  |  |  |  |
(1, 1.9%) |  S26 |  |  |  |  |
(1, 1.9%) | – |  S16 |  |  |  |
(1, 1.9%) | – |  S1, S5, S32 |  |  |  |
(3, 5.8%) |  |  |  |  |  |
NB | – | – | – |  S49 |  |
(1, 1.9%) | – |  S5, S32 |  |  |  |
(2, 3.8%) |  |  |  |  |  |
TL | – |  S17, S52 |  |  |  |
(2, 3.8%) | – | – | – | – |  |

In Table 17, the hyperparameter tuning methods used for ML models are
detailed. Considering that several studies employed more than one ML method, a
single tuning method might be applied to multiple ML models.

HT2) Among the 52 reviewed studies, 10 papers opted for the Default
hyperparameter values, determined by ML training libraries like scikit-learn.
This approach allows efficient training without investing time in
hyperparameter tuning. Default hyperparameters are commonly established
through expert knowledge, serving as a dependable initial configuration and
aiding in the prevention of overfitting issues [78].

HT3) Four studies reduced ML model training time by utilizing hyperparameters
tuned from Literature. Given the frequent retraining needs of ML models in CI
environments, both HT2 and HT3 methods contribute to significant time savings.

HT4) The majority of studies (11 out of 52) manually explored hyperparameter
values for ML models, determining optimal settings through the Test and Trial
method. Similar to Random Search (RS), this method allows parallel and
independent evaluation of candidate hyperparameters, facilitating quicker
identification of suitable solutions in the agile CI environment, even before
the arrival of new data [79].

HT5) The fifth strategy involves defining a Formula for hyperparameter tuning,
observed in three studies: S44 calculated the Hoeffding bound for navigating
Hoeffding decision tree nodes based on observations and confidence parameters;
S11 determined memory units for LSTM, Sigmoid, and Tanh using a specified
formula; and S24 adjusted the K value for clustering build logs with the
K-Means algorithm as $\sqrt{n/2}$ during data preparation, where n is the
number of build logs, without providing evidence for this assumption.

Notably, 14 studies did not report how they adjusted hyperparameters in their
studies.

While hyperparameter tuning enhances model accuracy, it may lead to
overfitting [80]. Overfitting occurs when a model learns excessively from the
training data, such as by increasing the number of layers in neural networks.
This includes capturing noise or random patterns, which hinders its
performance with new data and reduces its generalizability [81]. To address
this issue, researchers commonly employ K-fold cross-validation [46]. This
technique involves iterative training and evaluation on different data
subsets, allowing hyperparameter adjustment for improved model performance.
Notably, the selected studies do not address the impact of hyperparameter
tuning techniques on ML model performance in CI tasks.

Summary: $\bullet$ Decision tree algorithms are favored in agile CI due to
high accuracy and low computational overhead. $\bullet$ Reinforcement learning
suits agile settings for sequential decision-making challenges. $\bullet$
Neural network algorithms, with high accuracy, find broad application across
five CI tasks. $\bullet$ Hyperparameter tuning, despite increasing training
time, enhances ML model accuracy. $\bullet$ Five strategies, including search
methods, default settings, literature-tuned parameters, manual tuning, and
formulas, are identified for hyperparameter tuning. $\bullet$ 14 studies in
the literature did not report their hyperparameter tuning methods.

###  4.6 RQ5: Evaluation Methods

In practical ML applications, performance evaluation is crucial. In this
section, we summarize the validation techniques and performance metrics used
in the selected studies. For assessing supervised ML methods, predictions were
compared with actual labels from untrained data, revealing seven distinct
evaluation methods (EM) in the literature as presented in Table 18.

Table 18: Relationships between seven evaluation methods and CI phases are
illustrated in the table. The color intensity in this table corresponds to the
number of studies, with lighter shades indicating lower counts and darker
blues representing higher counts. The bolded numbers represent the count of
studies and their percentage in total.

Evaluation |  |  |
---|---|---|---
technique |  Selection |  |
method |  Total |  |
papers |  [c]Unit |  |
Test |  [c]Integration |  |
Test |  [c]Regression |  |
Test |  [c]Build |  |
Validation |  [c]System |  |
Test |  [c]Process |  |
Management |  |  |
K-Fold | Sorted |  5 |
9.6% | – | – |  S26, S30, S32,
S40 (4, 7.7%) |  S38 (1, 1.9%) | – | –
2-9 | Random |  16 |
30.8% |  S2 |  |
(1, 1.9%) |  S3, S15, |  |
S23 (3, 5.8%) |  S13, S32, S36, S46, |  |
S47 (5, 9.6%) |  S21, S27, S43, S44, |  |
S51 (5, 9.6%) |  S12 |  |
(1, 1.9%) |  S49 |  |
(1, 1.9%) |  |  |
Percentage | Sorted |  4 |
7.7% | – | – |  S8, S16, S29
(3, 5.8%) |  S25 |  |
(1, 1.9%) | – | – |
2-9 | Random |  4 |
7.7% | – |  S18 |
(1, 1.9%) |  S39 |  |
(1, 1.9%) |  S24, S42 |  |
(2, 3.8%) | – | – |
Constant Number | Sorted |  3 |
5.8% | – | – |  S11, S41
(2, 3.8%) |  S6 |  |
(1, 1.9%) | – | – |
Time | Sorted |  4 |
7.7% | – | – |  S17, S37, S52
(3, 5.8%) |  S1 |  |
(1, 1.9%) | – | – |
Version | Sorted |  1 |
1.9% | – | – |  S9
(1, 1.9%) | – | – | –
Gradually Evaluation | Incremental |  13 |
25% | – | – |  S7, S14, S19, S20,
S22, S28, S31, S33, |  |  |
S34, S35, S48, S50 |  |  |
(12, 23.1%) | – |  S4 |
(1, 1.9%) | – |  |
Baseline | Train=Test |  2 |
3.8% | – | – |  S10
(1, 1.9%) | – | – |  S45
(1, 1.9%) |  |  |

EM1) K-Fold: Generally, data is divided into K equal parts, with ML models
trained on K-1 segments and evaluated on the unseen segment.

EM2) Percentage: Data is split into training and testing sets based on a
specified percentage (e.g., 80% training and 20% testing).

EM3) Constant number: N samples are designated as the test set, while the
model is trained on the remaining samples.

EM4) Time: Dataset division is based on specific dates or time spans,
mirroring real-world CI data dynamics.

EM5) Version: Utilized in version-based releases, where old software versions
serve as the training data and the earliest versions as the test data.

EM6) Gradual Evaluation: Common in RL research, it assesses model performance
incrementally through a reward-based approach, reflecting real-world CI
conditions.

EM7) Baseline: Unsupervised ML models use the same dataset for training and
testing, with evaluation based on comparing model outcomes to the expected
solution.

Notably, besides gradual evaluation for RL-based methods, S4 employed an
online supervised learning approach and assessed model performance by
iterative training and evaluating positive predictive values (PPV) as the main
effectiveness metric for a test suite.

Table 18 classifies K-Fold and Percentage methods into Sorted and Random
types. In Sorted types, models train on older data and evaluate earlier unseen
segments, prioritizing reliability in real-world CI scenarios. Random types,
randomly segment data, enhancing solution generalizability but may be less
robust for real-world problem-solving [82, 83].

According to Table 18, K-Fold methods are widely employed for evaluating ML
techniques. This prevalence may be attributed to the availability of numerous
packages capable of calculating evaluation metrics with minimal intervention.
However, the reliance on random selection methods for training and testing
data, as observed in many studies, can compromise the reliability of the
presented methods in real-world scenarios. It is worth mentioning that in CI
environments, ML models should always be applied to newly produced, unseen
data.

In contrast to random selection in K-Fold methods, as shown in Table 18, a
majority (30 out of 36) of the studies utilize sorted selection methods to
align their approaches with real-world Continuous Integration (CI)
environments.

Additionally, Gradual Evaluation emerges as another commonly utilized
evaluation method, particularly notable due to the extensive use of
Reinforcement Learning methods in CI’s Regression Testing step.

Our analysis identified six types of K-Fold cross-validation methods (Figure
6), each denoted by a number. K-Fold validation typically divides the dataset
into three groups: Training data for model training, Testing data for model
evaluation, and a Holdout subset unused for training or evaluation (in some
techniques, Testing and Holdout sets are the same).

Figure 6: Six types of employed K-Fold evaluation techniques in the selected
studies. Note: In this picture red arrows present the next evaluation round,
blue arrows show the new data arrangement, and the X mark shows the neglected
data parts in the new data arrangement.

Figure 6 presents six K-Fold cross-validation types, each addressing specific
challenges in CI environments:

Classical K-Fold uses the same testing and holdout sets, risking overfitting
in sequential CI data [84].

The second type of K-Fold is ideal for environments with extensive data,
addressing the need for training models on large volumes of data in CI
settings [85]. Here, testing data is consistently more recent than the
training data. In this approach, after dividing the data into K folds, we have
two options: we can select one fold as the training set and the next fold as
the testing set (first row), or we can divide one fold into both training and
testing datasets (second row).

The third K-Fold type, similar to the classical type, uses the most recent
data as testing while altering the holdout data. This method lowers the risk
of overfitting compared to the first K-Fold type.

The fourth type of K-Fold Cross-Validation, known as Nested Cross-Validation
(Nested CV), utilizes two K-Fold procedures to conduct hyperparameter tuning
and increase the number of iterations. However, Nested CV incorporates
futuristic data during training [84]. This method resembles the third type of
K-Fold, with the key difference being that the testing dataset varies in each
iteration. Additionally, Nested CV allows for the selection of a new K value
for the training dataset. Specifically, the process involves initially
reserving one random fold as the testing dataset, followed by model training
using a new K-Fold method on the remaining dataset.

The fifth K-Fold type is the modified version of the third type, uses the most
recent data for testing, and varies the number of holdout folds to determine
the data required for acceptable model accuracy [43].

The last K-Fold type employs the last consecutive data fold for testing while
adjusting the number of holdout folds, allowing the examination of model
stability over time [43].

The chronological order is vital in CI data, rendering many K-Fold techniques
employed in the selected studies inadequate for CI environments due to their
lack of consideration for the latest data in evaluations [86, 36].

####  4.6.1 Performance Measures

Choosing appropriate performance measures allows us to identify weaknesses in
a solution and compare our results with others’ methods detached from the
research settings.

Table 19: Performance measurements description and formulas. Measure | Description | Formula
---|---|---
Precision |  The percentage of the detected positive instances that were correct |  $\frac{TP}{TP+FP}$
Recall |  The proportion of positive instances that were correctly identified |  $\frac{TP}{TP+FN}$
F1-score |  The harmonic mean of recall and precision |  $\frac{2\times Precision\times Recall}{Precision+Recall}$
Accuracy |  Percentage of correctly classified instances |  $\frac{TP+TN}{TP+FP+FN+TN}$
APFD |  Ratio between detected and detectable instances in classes |  $1-\frac{\sum_{1}^{m}TF_{i}}{m\times n}+\frac{1}{2n}$
NAPFD |  Normalized $APFD$ |  $p-\frac{\sum_{1}^{m}TF_{i}}{m\times n}+\frac{p}{2n}$
MAE |  Average of errors between predicted and actual value |  $\frac{\sum_{1}^{n}(y_{i}-\widehat{y}_{i})}{n}$
RMSE |  The square root of the average of squared differences between prediction and actual observation |  $\sqrt{\frac{1}{n}\sum_{1}^{n}(y_{i}-\widehat{y}_{i})^{2}}$
PP |  The ratio of the detected positive instances and all instances |  $\frac{TP+FP}{TP+FP+TN+FN}$
MCC |  Correlation coefficient between the observed and predicted binary classifications |  $\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$
G-Measure |  The Harmonic mean of TPR ($recall$) and true negative rate ($TNR$) |  $\frac{2\times TPR\times TNR}{TPR+TNR}$
NRPA |  Measures how close a predicted ranking of items (${s_{e}}$) is to the optimal ranking (${s_{o}}$) |  $\frac{RPA(s_{e})}{RPA(s_{o})}$
AUC |  Two-dimensional area under Receiver Operating Characteristic ($ROC$). |
CCOFF |  Measures the closeness of the predicted values and the testing set |  $\rho_{xy}=\frac{Cov(x,y)}{\sigma_{x}\sigma_{y}}$
T-test |  Determining mean difference between two sets |

In the 52 selected studies, recall, precision, and F1-score are the most
frequently used performance metrics, appearing in 30, 24, and 21 studies,
respectively. Table 19 provides a summary of performance measurements,
including formulas and brief descriptions. Key variables encompass $n$ (number
of test cases), $m$ (number of faults), $p$ (ratio of detected faults to total
faults), $TF_{i}$ (position of the first test exposing fault $i$), $y_{i}$ and
$\widehat{y}_{i}$ (actual and predicted values), $\rho$ (Pearson product-
moment correlation coefficient), and $\sigma_{x}$ (standard deviation of
variable $x$).

In the reviewed studies, frequently used performance measurements include
Average Percentage of Fault Detection (APFD) in 16 studies, Normalized APFD
(NAPFD) in 10 studies, Accuracy in 9 studies, and AUC (Area under the ROC
Curve) in 9 studies. AUC quantifies performance by plotting the True Positive
Rate (TPR or Recall) against the False Positive Rate ($FPR=\frac{FP}{FP+TN}$)
in a ROC curve, where $T$, $F$, $P$, and $N$ stand for True, Negative,
Positive, and Negative in the confusion matrix.

It is crucial to note that while APFD is a valuable performance metric,
researchers are encouraged to prefer Normalized APFD (NAPFD) for enhanced
consistency [87]. APFD measures the area under the curve with the y-axis
indicating the percentage of faults found and the x-axis showing the
percentage of test cases. However, reporting the average APFD value can be
misleading, as it equals 1 when no failures occur in a test cycle and may
yield high values in imbalanced datasets, a common characteristic in CI
datasets [62].

Table 20: Ratio of ML types to performance metrics (left) and CI phases
(right). Numbers under ‘ML Types’ indicate study counts. Study IDs are omitted
to prevent table dimension expansion. color intensity corresponds to
percentages, with lighter shades indicating lower and darker blues
representing higher percentages. The bolded numbers represent the count of
studies and their percentage in each ML type. The percentage in the ML types
column is in total.

NAPFD | Accuracy | AUC | APFD | F1-Score | Precision | Recall | ML types |  Unit
---|---|---|---|---|---|---|---|---
Test |  Integration |  |  |  |  |  |  |
Test |  Regression |  |  |  |  |  |  |
Test |  Build |  |  |  |  |  |  |
Validation |  System |  |  |  |  |  |  |
Test |  Process |  |  |  |  |  |  |
Management |  |  |  |  |  |  |  |
1 |  |  |  |  |  |  |  |
3.0% |  6 |  |  |  |  |  |  |
18.2% |  9 |  |  |  |  |  |  |
27.3% |  5 |  |  |  |  |  |  |
15.2% |  17 |  |  |  |  |  |  |
51.5% |  19 |  |  |  |  |  |  |
57.6% |  19 |  |  |  |  |  |  |
57.6% |  Classification |  |  |  |  |  |  |
(33, 63.5%) |  1 |  |  |  |  |  |  |
3.0% |  3 |  |  |  |  |  |  |
9.1% |  15 |  |  |  |  |  |  |
45.5% |  11 |  |  |  |  |  |  |
33.3% |  2 |  |  |  |  |  |  |
6.1% |  1 |  |  |  |  |  |  |
3.0% |  |  |  |  |  |  |  |
– |  1 |  |  |  |  |  |  |
16.7% | – |  2 |  |  |  |  |  |
33.3% | – |  1 |  |  |  |  |  |
16.7% |  1 |  |  |  |  |  |  |
16.7% |  Regression |  |  |  |  |  |  |
(6, 11.5%) | – |  2 |  |  |  |  |  |
33.3% |  3 |  |  |  |  |  |  |
50.0% |  1 |  |  |  |  |  |  |
16.7% | – | – |  |  |  |  |  |
4 |  |  |  |  |  |  |  |
33.3% | – | – |  5 |  |  |  |  |
41.7% | – | – |  3 |  |  |  |  |
25.0% |  Reinforcement |  |  |  |  |  |  |
Learning (12, 23.1%) | – | – |  12 |  |  |  |  |
100% | – | – | – |  |  |  |  |
– |  1 |  |  |  |  |  |  |
50.0% | – |  1 |  |  |  |  |  |
50.0% |  1 |  |  |  |  |  |  |
50.0% |  2 |  |  |  |  |  |  |
100% |  2 |  |  |  |  |  |  |
100% |  Clustering |  |  |  |  |  |  |
(2, 3.8%) | – | – |  1 |  |  |  |  |
50.0% | – | – |  1 |  |  |  |  |
50.0% |  |  |  |  |  |  |  |

Table 20 indicates that classification ML methods often used Recall,
Precision, and F1-Score, while RL methods predominantly utilized APFD and
NAPFD. The strong association between RL algorithms and APFD metrics is due to
RL’s compatibility with Test Case Prioritization (TCP) in CI, leveraging
action-reward policies. APFD and NAPFD quantify the weighted mean of fault
detection percentages over the test suite’s lifecycle, making them suitable
for evaluating test case quality. Notably, studies using APFD or NAPFD mainly
focused on TCP, except for S29, which aimed to enhance mutant test precision
within their MuDelta approach.

Several studies reported additional performance metrics, including MAE, RMSE,
pairwise t-tests, PP, NRPA, MCC, TTF, and G-Measure. Nine studies used simple
rate values like misclassification rates and failure detection rates.
Combining these simple rate values with more comprehensive measures like
F-score and Accuracy is recommended for a more thorough assessment of results.
This strategy aids in preventing resource misallocation, as relying solely on
the evaluation of results based on True Positive (failing tests) values could
lead to the execution of numerous small failing test cases, thereby increasing
resource consumption, particularly by resource-intensive passing test cases.

####  4.6.2 Connection Between CI Tasks and Evaluation Metrics

Table 21: Heat map displays the percentage distribution of evaluation metrics
across CI tasks. Study IDs are omitted to prevent table dimension expansion.
color intensity corresponds to percentages, with lighter shades indicating
lower and darker blues representing higher percentages. The bolded numbers
represent the count of studies and their percentage in each CI task.

CI Phases |  CI Tasks |  Num of papers |  Recall |  Precision |  F1-Score |  APFD |  NAPFD |  Accuracy |  AUC |  MCC |  TTF |  MAE |  RMSE |  NRPA |  T-test |  PP |  G-Measure |  Time Cost |  Simple
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
UT |  [l]Unit Test |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Prediction | 1 |  1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
100% |  1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
100% |  1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
100% | – | – |  1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
100% | – | – | – | – | – | – | – | – | – | – | – |  |  |  |  |  |  |  |
1-20 IT |  [l]Branch Coverage |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |