∎

11institutetext: Federal Institute of Rio Grande do Norte - IFRN
Federal University of Rio Grande do Norte - UFRN 22institutetext: Natal-RN,
Brasil
22email<EMAIL_ADDRESS>33institutetext: University of Otago
44institutetext: Dunedin, New Zealand
44email<EMAIL_ADDRESS>55institutetext: Federal University of Rio
Grande do Norte - UFRN 66institutetext: Natal-RN, Brasil
66email<EMAIL_ADDRESS>
# Continuous Integration and Software Quality: A Causal Explanatory Study with
Travis CI Projects

Eliezio Soares  Daniel Alencar da Costa  Uirá Kulesza

(Received: date / Accepted: date)

###### Abstract

Continuous Integration (CI) is a software engineering practice that aims to
reduce the cost and risk of code integration among teams. Recent empirical
studies have confirmed associations between CI and the software quality (SQ).
However, no existing study investigates causal relationships between CI and
SQ. This paper investigates it by applying the causal Direct Acyclic Graphs
(DAGs) technique. We combine two other strategies to support this technique: a
literature review and a Mining Software Repository (MSR) study.

In the first stage, we review the literature to discover existing associations
between CI and SQ, which help us create a “literature-based causal DAG” in the
second stage. This DAG encapsulates the literature assumptions regarding CI
and its influence on SQ. In the third stage, we analyze 12 activity months for
148 open-source projects by mining software repositories— 74 CI and 74 no-CI
projects. This MSR study is not a typical “correlation is not causation” study
because it is used to verify the relationships uncovered in the causal DAG
produced in the first stages. The fourth stage consists of testing the
statistical implications from the “literature-based causal DAG” on our
dataset. Finally, in the fifth stage, we build a DAG with observations from
the literature and the dataset, the “literature-date DAG”.

In addition to the direct causal effect of CI on SQ, we find evidence of
indirect effects of CI. For example, CI affects teams’ communication, which
positively impacts SQ. We also highlight the confounding effect of project
age.

###### Keywords:

Empirical, Continuous Integration, Software Quality, Causation, Causal DAG

##  1 Introduction

Continuous integration (CI) is an increasingly popular software engineering
practice used in a variety of domains and organizations beck2010xp ; fowler_ci
; duvall2013continuous ; bosch2014modeling . CI proposes to reduce the costs
and risks related to code integration by having a central daily code
integration task. Code should be integrated every couple of hours while
sharing the project knowledge, the code responsibility and reducing the risks
related to late integration beck2010xp ; fowler_ci ; duvall2013continuous ;
bosch2014modeling .

Practitioners and existing literature have proclaimed several benefits related
to using CI. CI is associated with improvements in testing practices, better
quality assurance, a reduction in issue reports, among others
soares2021effects . Although studies investigating associations are valuable,
causal studies are important for decision making and deeper insights in an
area of knowledge. To the best of our knowledge, there is a lack of studies
that investigate causal relationships between CI and software quality.

Causal knowledge is valuable because it empowers stakeholders to make better
decisions in software development pearl2000models , such as adopting CI or
defining a communication policy. Our work investigates the potential causal
relationship between CI and software quality to better understand the
influence of CI on delivering better software products. To study the causal
relationship between CI and software quality, we use an approach that consists
of five interconnected stages. Each stage is guided by a research question
described in the following.

RQ1. What does the literature proclaim about CI and software quality? To
understand the variables that can potentially play a role in the relationship
between CI and software quality, we conduct a literature review, which helps
us define a causal DAG (i.e., a graphical-statistical technique - to enable us
to draw domain assumptions and infer causal conclusions in a later stage
hernan2010 ). The goal of the review is to discover the existing connections
between CI and software quality, as well as marginal associations (e.g.,
associations between code smells and software quality), which will help us to
draw a comprehensive DAG containing the existing studies’ assumptions about
how CI may influence software quality (and the potential confounding variables
surrounding both CI and software quality). To study this relationship, we
consider bug reports a proxy for software quality, similar to a previous study
from Vasilescu et al. vasilescu2015quality .

RQ2. Is the causal effect of CI on software quality empirically observable?
Once we have a causal DAG containing a sufficient set of variables to analyze
the relationship between $ContinuousIntegration$ and $BugReport$, we can apply
the d-Separation rules pearl1995causal  and evaluate the raised set of
testable statistical implications from the DAG built in RQ1. d-Separation is a
set of graphical rules to identify if an association path exists between two
or more variables in the DAG. From such associations, testable implications
arise (d-Separation will be explained in Section 2.3). Next, we mine software
repositories to collect observational data on the variables of the DAG and
perform (un)conditional independence tests on our dataset to answer RQ2. Note
that this differs from an ordinal “causation is not correlation” MSR study, as
these tests are guided by the d-Separation rules derived from the causal DAG.

RQ3. What would be an accurate causal theory for CI? Considering the
investigations in RQ1 and RQ2 and the testable implications from the causal
DAG in the dataset, we can analyze the hypotheses that failed in our analyses
(i.e., statistical implications from the supposed relationships between the
variables that were not supported by the data) and propose a new causal DAG.
We do so, again, using the literature knowledge and empirical data to answer
RQ3 but now considering the new and corrected causal DAG this time.

Our study innovates in offering the seminal empirical causal conclusions
regarding CI and software quality (using the number of bug reports as a proxy
for quality). This innovation helps evolve the current empirical knowledge of
Continuous Integration as most of our empirical observations rely on
statistical correlations instead of causal relationships. We also contribute
with research and practical implications for the software engineering
community. Understanding the potential effects and which variables play
relevant causal influences allows us to make beneficial software engineering
decisions, e.g., seeking to maximize or minimize these effects, such as the
influence of team communication decreasing the number of bug reports.

The rest of the paper is organized as follows: Section 2 presents the current
state of knowledge of Continuous Integration and the need for causal studies.
Additionally, Section 2 offers a technical background to understand the
employed methodology. Section 3 explains the methodology to answer each
research question. Section 4 presents the results of our studies for each
research question. Section 5 discusses our results and findings. In Section 6,
we present threats to our study’s validity. Finally, Section 7 concludes this
study, highlighting the main findings.

##  2 MOTIVATION AND RELATED WORK

Our study investigates potential causal relationships between continuous
integration (CI) and software quality. This study is important because
understanding causal effects can help practitioners to measure the actual
benefits of CI. Given the predominance of statistical correlation studies,
this work extends the current CI knowledge by offering causal conclusions. Our
study helps researchers to remain aware of possible confounders, related
variables, and adjacent associations when investigating the relationship
between CI and software quality.

Through a systematic literature review (SLR), Soares et al. soares2021effects
investigated the influence of CI on software development. Their study compiles
most of the existing (non-causal) associations between CI and software
quality. Based on current findings, the SLR concludes that CI may improve the
time to develop and merge addressed issues as well as to reduce the number of
reported defects zhao2017impact ; rahman2018characterizing ; soares2021effects
. However, the SLR also identifies several CI associations that need further
investigation. For example, the relationship between the development
environment reliability and developers’ (over)confidence is due to trusting CI
outcomes.

Despite all the effort invested in studying the potential benefits of CI
soares2021effects ; bosch2014modeling ; zhao2017impact ; vasilescu2015quality
; keskin2019software ; pinto2018work ; cassee2020silent , the software
engineering community still needs to benefit from a step further, which is to
investigate causal relationships in existing studies. The difference between
association and causation is critical. For example, inferring causal
conclusions from associations may be harmful because of spurious associations,
leading to false conclusions (e.g., confounding effect) pearl2000models ;
greenland1999 . Unconsidered relationships among variables may confound the
causal assumptions about the studied phenomenon. For example, one can observe
that streets have puddles and people wear raincoats whenever it rains.
However, if one assumes that puddles cause people to wear raincoats, one would
fail to consider a relationship of a third variable (rain) that causes both
puddles and people wearing raincoats. In the following subsections, we explore
this in more detail.

###  2.1 Common Cause Principle

Reichenbach’s Common Cause Principle reichenbach1991direction  states that
given two statistically dependent variables $X$ and $Y$, if one is not a cause
of the other, then they may share a common cause $Z$, as shown in Fig. 1(a).
Conditioning on the common cause $Z$, then $X$ and $Y$ become independent. For
example, Fig. 1(a) shows an association “flowing” (the red dots above on the
edges represent an “association flow”) between $Heat$ and $Smoke$. They are
associated because they share a common cause, which is $Fire$. Fig. 1(b) shows
the interrupted flow when conditioning on $Fire$, i.e., in the absence of
$Fire$ (i.e., $Fire$=0), there is no association between $Heat$ and $Smoke$.

Therefore, if the famous adage states that “correlation does not imply
causation” (i.e., statistical associations are not sufficient to determine
causal relationships), on the other hand, “there is no causation without
association.” The common cause principle establishes a relationship between
statistical properties (i.e., association) and causal structures
peters2017elements . In this way, it is possible to infer the existence of
causal links from statistical dependencies (i.e., functional relationships
between the variables) peters2017elements . To infer causation, Pearl
pearl2000models  proposed employing a causal modeling framework based on
causal Directed Acyclic Graphs (causal DAGs). In the following subsections, we
explain the theory proposed by Pearl because we apply his proposed theory in
our study.

Figure 1: In this picture, the ellipses are variables, the edges represent a
relationship between variables, and the red dots above the edges represent an
“association flowing” between variables. (a) Fire is the common cause of Heat
and Smoke. Heat and Smoke are associated through Fire, i.e., the “association
flows” between Heat and Smoke through Fire. (b) Conditioning on Fire, the
association flow between Heat and Smoke is blocked. (c) Conditioning on Fire,
Smoke becomes associated only with its descendent Smell. (d) Spark is a common
cause of Fire and Smoke, opening a backdoor path between these variables.
Spark is a source of confounding.

###  2.2 Causal Directed Acyclic Graphs (Causal DAGs)

Pearl pearl2000models  argues that nature possesses causal mechanisms that, if
described in detail, are deterministic functional relationships between
variables. Some of these variables are unobservable, e.g., sometimes we see
the smoke causing a fire alarm to be activated, but we can not see the fire.
However, it is the fire that causes the existence of smoke. Pearl describes
the causal discovery task as an induction game that contributes to identifying
(from available observations or interventions) the organization of the
mechanisms in the form of an acyclic causal structure pearl2000models . This
causal structure is called a Directed Acyclic Graph (DAG). A DAG has: (i)
nodes - that are variables with directed edges and no directed cycles; and
(ii) edges that represent functional relationships between variables (see Fig.
1(a)) pearl2000models ; spirtes2000causation ; hernan2010 .

Pearl pearl2000models  argues that nature possesses causal mechanisms that, if
described in detail, are deterministic functional relationships between
variables. Some of these variables are unobservable, e.g., sometimes we see
the smoke causing a fire alarm to be activated, but we can not see the fire.
However, it is the fire that causes the existence of smoke. Pearl describes
the causal discovery task as an induction game that contributes to identifying
(from available observations or interventions) the organization of the
mechanisms in the form of an acyclic causal structure pearl2000models . This
causal structure is called a Directed Acyclic Graph (DAG). A DAG has: (i)
nodes - that are variables with directed edges and no directed cycles; and
(ii) edges that represent functional relationships between variables (see Fig.
1(a)) pearl2000models ; spirtes2000causation ; hernan2010 .

In turn, Pearl and Verma pearl1991theory  defined a Causal Markov Condition
stating that a DAG should have a node distribution $\displaystyle
N=\\{N_{1},...,N_{n}\\}$ such that, for each j, $\displaystyle N_{j}$ is
independent of its non-descendants conditioning on its parents pearl2000models
; hernan2010 ; spirtes2000causation . Considering the example of Fig. 1(c),
the Markov Condition implies that, if conditioning on the parent $Fire$,
$Smoke$ becomes independent of all other variables on the DAG, except its
descendent $Smell$. That means that $Smoke$ exists ($Smoke=1$), but there is
not necessarilly $Heat$, because we condition on $Fire$ ($Fire=0$).

Reichenbach’s Common Cause Principle (see section 2.1) and the Markov
Condition imply that a causal DAG should contain the common causes of any pair
of variables pearl2000models ; hernan2010 . Thus, to build a sufficient causal
DAG of a phenomenon, it is essential to know the significant common causes
among the variables involved in that phenomenon, i.e., if two variables $Heat$
and $Smoke$ in the DAG share a common cause $Fire$, then $Fire$ should be
represented in the DAG, as shown in Fig. 1(c). Since this is a recursive
criterion, in Fig. 1(d), $Spark$ should be present since it is a common cause
of $Fire$ and $Smoke$.

Discovering causal structures to build a DAG that correctly describes a
phenomenon, i.e., the set of variables a DAG should contain and the
relationships between variables, is challenging. There are at least three
strategies to obtain a DAG: prior knowledge, guessing-and-testing, or
discovery algorithms shalizi2021 . Prior knowledge is a source to build causal
DAGs, but since there is a link between causal structures and statistical
properties (see section 2.1), it is possible to test the correctness of a
built DAG if we have access to observational data. Section 2.3 details the
statistical properties and their testable implications.

###  2.3 d-Separation and the Testable Implications of the DAGs

A helpful approach to visually understand DAGs is to assume that associations
“flow” through the edges of the DAG hernan2010 . Using the illustration from
Fig. 1(a), the association flows freely between $Heat$ and $Smoke$. However,
intervening on the values of $Fire$ (as in the Fig. 1(b)), this flow may be
interrupted since it represents a common cause for $Heat$ and $Smoke$
hernan2010 . A set of graphical rules, called d-separation, were formalized by
Pearl pearl1995causal  to infer associational conclusions from causal DAGs.

D-separation is a set of graphical rules to define if a path in the DAG is
blocked or open. To understand the rules, we consider three structural
patterns: (i) Chain: $Fire\to Smoke\to Smell$ (see Fig. 1(c)); (ii) Fork:
$Heat\leftarrow Fire\to Smoke$ (see Fig. 1(a)); and (iii) Collider:
$Humidity\to Heat\leftarrow Fire$ (see Fig. 1(c)). In the chain, $Fire$ and
$Smell$ are dependent
($Fire\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}Smell$) but become
independent (i.e., blocked), conditioning on $Smoke$
($Fire\mathrel{\perp\mspace{-10.0mu}\perp}Smell\mid Smoke$). In the fork,
$Heat$ and $Smoke$ are dependent
($Heat\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}Smoke$) unless we
condition on $Fire$ ($Heat\mathrel{\perp\mspace{-10.0mu}\perp}Smoke\mid
Fire$). In the collider, $Humidity$ and $Fire$ are independent
($Humidity\mathrel{\perp\mspace{-10.0mu}\perp}Fire$) because a collider blocks
the association flow. Conditioning on the collider (or one of its descendants)
opens that flow, making $Humidity$ and $Fire$ dependent(e.g.,
$Humidity\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}Fire\mid Heat$) . We
provide a summary below:

  * •

Chain: $X\to Z\to Y$ $\Rightarrow$
$X\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}Y$ and
$X\mathrel{\perp\mspace{-10.0mu}\perp}Y\mid Z$

  * •

Fork: $X\leftarrow Z\to Y$ $\Rightarrow$
$X\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}Y$ and
$X\mathrel{\perp\mspace{-10.0mu}\perp}Y\mid Z$

  * •

Collider: $X\to Y\leftarrow Z$ $\Rightarrow$
$X\mathrel{\perp\mspace{-10.0mu}\perp}Z$ and
$X\centernot{\mathrel{\perp\mspace{-10.0mu}\perp}}Z\mid Y$

Based on these d-Separation rules, all DAGs have a consequent set of testable
statistical implications. These statistical implications inferred from the
graphical analysis allow us to verify if the structure of a DAG is consistent
with an empirical dataset through conditional independence tests on the data.
Pearl pearl2000models  defines a causal model as a pair $M=<D,\Theta_{D}>$
consisting of a causal structure $D$ and a set of parameters $\Theta_{D}$
compatible with D. Thus, relying on d-Separation rules and a representative
dataset, the causal DAG technique also allows building a causal structure
through both guessing-and-testing as well as discovery algorithms shalizi2021
.

This study aims to understand how Continuous Integration affects Software
Quality. We use a combination of two approaches: (i) prior knowledge to draw
the existing assumptions; and (ii) guessing-and-testing to obtain a final
causal structure. To introduce more rigor, we consider prior knowledge by
gathering it from the existing literature and drawing an initial DAG with
assumptions regarding the influence of CI on software quality. Afterward, we
refine our understanding by applying guessing-and-testing that relies on the
statistical implications that we derive from the initial DAG and their
validations by using our dataset, i.e., we collect data by mining software
repositories. Section 3 details these methodological steps.

###  2.4 Backdoor Paths and Confounding

A common source of bias is due to common causes between two variables, such as
the presence of a cause $Spark$ shared by the treatment $Fire$ and the outcome
$Smoke$ (see Fig. 1(d)). This common cause results in a path between $Fire$
and $Smoke$ that is not a direct edge. The path is through the back door,
i.e., $Fire\leftarrow Spark\to Smoke$ generating another association flow
between the treatment $Fire$ and the outcome $Smoke$. We refer to this kind of
structure as a backdoor path, and the bias caused by the backdoor path as
confounding.

Causal DAGs have grown in popularity in several fields (econometrics,
epidemiology, and climate science, among others). Greenland, Pearl, and Robins
greenland1999causal  present causal DAGs as a tool for identifying variables
that must be measured and controlled to obtain unconfounded causal effect
estimates for epidemiologic research. Shmueli shmueli2010explain  sees causal
DAGs as a common causal inference method for testing causal hypotheses on
observational data.

##  3 METHODOLOGY

We follow a pipeline of 5 stages shown in Fig. 2 to answer our research
questions: The Literature Review (stage 1) and DAG Building (stage 2) stages
contribute to answering RQ1. The data collection from repositories (stage 3)
and the DAG Implications Testing (stage 4) stages address RQ2. Lastly, another
stage of DAG Building (stage 5) performed in an iterative manner with the DAG
Implications Testing stage (stage 4) until a final DAG is obtained containing
literature and data consistency, answers RQ3. Next, we detail the different
stages of our research methodology.

Figure 2: Research method pipeline.

###  3.1 What does the literature proclaim about CI and software quality?

As depicted in Fig. 2, we perform two research stages to answer RQ1. First,
the literature review compiles the existing assumptions about how CI may
impact software quality and allow us to map variables, associations, and every
common cause (i.e., the $Fire$ in the example of Fig. 1(b)) for any two or
more variables in the DAG. This mapped information is input to stage 2 - DAG
building. We perform these two stages iteratively to build a causal DAG
representing the most up-to-date literature knowledge.

As discussed in section 2, prior knowledge is an existing approach to building
causal structures regarding a phenomenon. Then in this first research
question, we intend to draw a causal DAG based on the literature assumptions
regarding continuous integration (CI) and software quality. Although most of
the studies in software engineering are association studies, we discussed the
relationship between association and causation in section 2 and how
d-Separation rules raise testable statistical implications from causal DAGs
pearl2000models . Thus, we begin by mapping the existing knowledge to infer
causality, even if they come from association studies.

####  3.1.1 Literature Review

To review the relationship between continuous integration (CI) and software
quality (SQ), we consider “issues”, “bugs”, and/or “defects” as proxies of
software quality. Since we are also interested in knowing the common causes
(i.e., the $Fire$ in Fig. 1(a)) between CI and these variables (i.e.,
‘issues”, “bugs”, or “defects”), we review the literature for each one of
these variables.

We also review the literature for each new variable found during the DAG
building in an iterative manner. We perform the searches on Google
Scholar111https://scholar.google.com/ using the name of variables or synonyms.
To cover a broader range of studies and mitigate search bias, we prioritize
systematic literature reviews on the search. For instance, we search for
software bug relationships using the search string “systematic literature
review software bugs”. This search has a broader perspective and not
necessarily the studies mention CI, but it is essential to detect correlated
variables and their relationships.

In addition, we search the literature for relationships among the following CI
sub-practices — tests practices, integration frequency, build health
maintenance, and quick fixes of broken builds beck2010xp ;
duvall2013continuous ; fowler_ci . In the full search, we selected 39 studies,
of which 12 are systematic literature reviews. All the selected studies are
referenced, and a complete list is available in our replication
package222https://github.com/elieziosoares/ci_quality_study_replication. We
believe our literature review fits our goal of finding the claims surrounding
CI and SQ, especially because of the systematic literature reviews, as they
have already systematically compiled a comprehensive view of the areas of CI
and SQ.

Table 1: Connections identified in the literature about CI and software quality variables. Connection | Rationale
---|---
$CI\to BugResolution$ |  | Projects presents more resolved issues and bugs after adoption
---
of CI rahman2018characterizing .
$CI\to ResolutionTime$ |  | CI is related to an increasing in the number of issues closed by
---
period, helping to spend less time debugging and more time
adding features keskin2019software ; zhao2017impact .
$CI\to BugReport$ |  | CI teams discover more bugs than no-CI teams, and CI
---
projects present fewer defects than no-CI projects amrit2018effectiveness ;
vasilescu2015quality .
$CI\to Transparency$ |  | CI is associated with a transparency increase, facilitating
---
collaboration kerzazi2014why .
$CI\to Communication$ |  | The general discussion, the number of line-level review
---
comments, and change-inducing review comments tend to
decrease after CI adoption without affecting pull
request activity cassee2020silent .
$CI\to Overconfidence$ |  | CI developers are reported as suffering from a false sense of
---
confidence (when blindly trusting the tests) pinto2017inadequate ;
pinto2018work .
$CI\to TechnicalChallenges$ |  | Configuring the build environment, the tools, and practices
---
impose challenges for CI teams pinto2017inadequate ; debbiche2014challenges .
$CI\to TestsVolume$ |  | CI is associated with an increase in the test ratio sizilio2019empirical .
---
$CI\to CommitFrequency$ |  | CI is linked to a change in the commits pattern zhao2017impact ; rahman2018characterizing .
---

Continuous Integration and Software Quality: Soares et al. soares2021effects
conducted a systematic literature review on the associations between CI and
software development as a whole. The review highlights associations between CI
and an increase in bug/issue resolution rahman2018characterizing . For this
reason, our literature-based DAG starts from the connection between continuous
integration and bug resolution. The notation $CI\to BugResolution$ represents
a causal flow from CI to Bug Resolution. We expand our DAG based on other
studies that raise other diverse CI associations (as summarized in Table 1).

Confounders related to bug reports: To investigate potential confounders to
the effect of $CI\to BugReport$, we search for other factors associated with
Bug Rerport. Table 2 shows the associations extracted from a taxonomy by Huang
et al. huang2012taxonomy , a systematic literature review by A. Cairo et al.
cairo2018impact , and a mixed mining software repositories (MSR)-survey study
by Vasilescu et al. vasilescu2015quality .

Table 2: Connections identified in the literature about bug reports. Connection | Rationale
---|---
| $LackOfKnowledge$
---
$\to BugReport$
| Insufficient domain and linguistic knowledge are presented as
---
possible human root causes for software defects huang2012taxonomy .
| $LackTechKnowledge$
---
$\to BugReport$
| Insufficient programming and strategy knowledge and failure to catch
---
the specific feature of the problems are mapped as possible human
root causes for software defects huang2012taxonomy .
| $RequiremProblem$
---
$\to BugReport$
| Requirement management problems and a misunderstanding of
---
requirements and design specifications are reported as possible human
causes of software defects huang2012taxonomy .
| $Overconfidence$
---
$\to BugReport$
| Overconfidence and confirmation bias contributes to evaluation errors
---
and software defects huang2012taxonomy .
| $Inattention\to$
---
$BugReport$
| Interruptions and other kinds of inattention are reported as possible
---
human causes of software defects huang2012taxonomy .
| $Communication$
---
$\to BugReport$
| Communication problems lead to expression and comprehension
---
errors huang2012taxonomy .
| $ConfigManagement$
---
$\to BugReport$
| Configuration management problems lead to process errors huang2012taxonomy .
---
| $Tools\to$
---
$BugReport$
| Tools problems like compiler induced defects are possible root causes
---
of software defects huang2012taxonomy .
| $CodeSmells\to$
---
$BugReport$
| Code smells on the occcurrence of bugs cairo2018impact .
---
| $NumberOfForks\to$
---
$BugReport$
| The number of forks has an association with an increase in bug
---
reports vasilescu2015quality .
| $ProjAge\to$
---
$BugReport$
| Project age has a significant negative effect on the count of bugs
---
reported by core developers vasilescu2015quality .
| $ProjPopularity\to$
---
$BugReport$
| Project’s popularity has a significant negative effect on the count
---
of bugs reported by core developers vasilescu2015quality .
| $QuantIssues\to$
---
$BugReport$
| The number of non-bug issue reports has a significant and positive
---
effect on the response vasilescu2015quality .
| $TestsVolume\to$
---
$BugReport$
| The size of test files has a negative effect on bug reports
vasilescu2015quality .
---

Confounders related to bug resolution: To investigate potential confounders to
the effect of $CI\to BugResolution$, we search for other factors associated
with $BugResolution$ in the literature. We found that $Maintainability$,
$Analysability$, $Changeability$, $Stability$, $Testability$, $ProjectVolume$,
$Duplication$, $Unitsize$, $Unitcomplexity$, and $Modulecoupling$ all share an
association with $BugResolution$. For the sake of readability, we group all
these relationships into $InternalQuality\to BugResolution$ bijlsma2012faster
. We also found the associations $Communication\to BugResolution$ and
$IssuePriority\to BugResolution$ in the literature review from Zhang et al.
zhang2016literature . Table 3 shows all these associations and their
rationales.

Table 3: Connections identified in the literature about bug resolution. Connection | Rationale
---|---
| $InternalQuality\to$
---
$BugResolution$
| Elements of $InternalQuality$, such as maintainability, analysability,
---
changeability, stability, testability, project volume, duplication, unit size,
unit complexity, and module coupling, present significant correlation
with defect resolution efficiency huang2012taxonomy .
| $Communication\to$
---
$BugResolution$
| Human and data elements such as comments, severity, product,
---
component, among others, can improve the performance of bug
resolution zhang2016literature .
| $IssuePriority\to$
---
$BugResolution$
| Priority and severity are non-textual factors of a bug report
---
that enhance the capability of bug resolution zhang2016literature .

Confounders related to resolution time: Table 4 shows associations between
other factors than CI and $ResolutionTime$. We found associations extracted
from the literature including variables $IssueType$, $Communication$ (e.g.,
comments in issues, bug reports, pull requests), $IssuePriority$,
$CommitFrequency$, $OperateSystem$, and $IssueDescription$.

Table 4: Connections identified in the literature about the resolution time. Connection | Rationale
---|---
| $IssueType\to$
---
$ResolutionTime$
| Issue fixing times are different for different issue types
murgia2014influence ; licorish2017exploring ; zhang2012empirical ;
mockus2000identifying .
---
| $Communication\to$
---
$ResolutionTime$
| The number of comments and the max length of all comments
---
in the bug reports impact the resolution time. Bugs with little
discussion tend to be resolved quickly panjer2007predicting ;
zhang2012empirical .
| $IssuePriority\to$
---
$ResolutionTime$
| The severity of a bug report influences the delay before fixing it.
---
As high the severity level, the fewer the delay zhang2012empirical .
| $CommitSize\to$
---
$ResolutionTime$
| The size of code churn (number of methods) impacts the delay
---
before fixing a bug report zhang2012empirical .
| $OperateSystem\to$
---
$ResolutionTime$
| The median delay before fixing a bug found on Linux is shorter
---
than other OS zhang2012empirical .
| $IssueDescription\to$
---
$ResolutionTime$
| Increasing the literal length of the bug report description can
---
increase delay until the team checks it as resolved zhang2012empirical .

Internal confounders: We also consider internal relationships between every
discovered variable to understand possible confounding scenarios in our causal
DAG. That means considering potential associations between peripheral
variables, e.g., $IssueType\to CommitFrequency$. Table 5 shows such potential
associations and their rationales.

Table 5: Internal connections cataloged among the literature regarding the discovered variables. Connection | Rationale
---|---
| $IssueType\to$
---
$CommitSize$
| The issue type is associated with the size of the code churn hindle2008large
.
---
| $IssueType\to$
---
$Engagement$
| Developers tend to spend more effort engaging with one another
---
regarding new features and software extensions than in defects
licorish2017exploring .
| $IssueType\to$
---
$InfoSharing$
| Developers tend to share more information on defects and
---
enhancements than support tasks licorish2017exploring .
| $IssueType\to$
---
$Communication$
| A higher number of comments is associate with enhancements
---
and defects licorish2017exploring .
| $IssueType\to$
---
$DifficultyLevel$
| There is an association between the difficulty of a change
---
and its type mockus2000identifying .
| $Stability\to$
---
$TechnicalChallenges$
| The maturity of the tools, infrastructure, and CI activities
---
imposes challenges to practitioners debbiche2014challenges . The stability and
maturity
of the software under test affect the maintenance effort of tests
garousi2016when .

With these previous connections, we build the partial causal DAG represented
in Fig. 3, where continuous integration is the intervention, and
$BugResolution$ and $BugReport$ are potential outcomes.

Figure 3: Partial causal DAG for bug reports associations.

Confounders related to Continuous Integration and its sub-practices: To
investigate variables related to continuous integration (CI), we also consider
its sub-practices beck2010xp ; duvall2013continuous ; fowler_ci . We consider
the practices: automated tests practices, integration frequency, build health
maintenance, and time to fix a broken build duvall2013continuous . We
represent an association with a sub-practice of CI as an association with CI
itself. For example, $AutomatedTests\to Confidence$ will be represented as
$ContinuousIntegration\to Confidence$. This decision avoids intangible
discussions about what comes first, CI or Automated Tests, for instance. To
implement CI, we consider that such practices are implicit, i.e., there is no
CI practice without implementing those sub-practices soares2021effects ;
duvall2013continuous . Therefore, distinguishing the effect of CI and the
effect of its sub-practices would be challenging. Table 6 shows the
associations relating to testing, while Table 7 groups and presents those
relating build practices.

Table 6: CI connections cataloged among the literature from the perspective of test practices. Connection | Rationale
---|---
| $AutomatedTests\to$
---
$BugReport$
| Automated tests are related to improved product quality in terms
---
of fewer defects in the software dudekula2012benefits .
| $AutomatedTests\to$
---
$CodeCoverage$
| Automated tests are related to high coverage of code dudekula2012benefits .
---
| $AutomatedTests\to$
---
$WorkTime$
| Automated tests are related to reduced testing time dudekula2012benefits .
---
| $AutomatedTests\to$
---
$Confidence$
| Automated tests are related to increased confidence in the quality
---
of the system dudekula2012benefits .
| $AutomatedTests\to$
---
$HumanEffort$
| Automated tests are related to the less human effort that can be
---
redirected for
other activities dudekula2012benefits .
| $AutomatedTests\to$
---
$Cost$
| Automated tests are related to a reduction in cost dudekula2012benefits .
---
| $AutomatedTests\to$
---
$BugDetection$
| Automated tests are related to increased fault detection
dudekula2012benefits .
---
| $AutomatedTests\to$
---
$TechnicalChallenges$
| Automated tests require different skills to implement them
---
effectively garousi2016when .
| $LackTechKnowledge$
---
$\to AutomatedTests$
| The skills level of testers could be a hindrance to test
---
automation garousi2016when .
| $TestDesign\to$
---
$TestReusability$
| Designing tests with maintenance in mind, they can be repeated
---
frequently dudekula2012benefits .
| $TestRepetition\to$
---
$Reliability$
| When repeating tests, they are more reliable than single
---
executions dudekula2012benefits .
| $ProjAge\to$
---
$AutomatedTests$
| The number, coverage, and maturity of automated tests increase
---
with time zaidman2008mining ; zaidman2010studying ; hilton2018large ;
sizilio2019empirical .

Figure 4: Partial causal DAG for automated tests associations.

With connections shown in Table 6 we build the partial causal DAG represented
in Fig. 4, where $AutomatedTests$ is the intervention. The relationship
$AutomatedTests\to BugReport$ was omitted in this figure because it is a
partial DAG centered in the $AutomatedTests$. With Table 7, we build the
partial causal DAG represented in Fig. 5, centered in the build attributes.

Figure 5: Partial causal DAG for build attributes and their connections.

Figure 6: Complete literature-based causal DAG for CI, Bug Reports and their co-variables. Table 7: CI connections cataloged among the literature from the perspective of build practices. Connections | Rationale
---|---
| $BuildHealth\to$
---
$WorkTime$
| Broken builds lead to loss of time by freezing development and
---
tests kerzazi2014why .
| $BuildHealth\to$
---
$MergeConflicts$
| Broken builds lead to work blockage, which in turn leads to merge
---
conflicts laukkanen2017problems .
| $TeamSize\to$
---
$BuildHealth$
| The team size relates to build breakage. Shorter teams tend to break
---
fewer than larger ones kerzazi2014why .
| $MultipleWorkspace$
---
$\to BuildHealth$
| Maintaining multiple physical structures for multiple branches is
---
associated with more build breakage kerzazi2014why .
| $DeveloperRole\to$
---
$BuildHealth$
| There is a statistical difference in build breakage among different
---
role groups kerzazi2014why .
| $CommitSize\to$
---
$BuildHealth$
| The size of the changes is related to a higher probability of build
---
failure kerzazi2014why ; islam2017insights ; rausch2017empirical .
| $CommitType\to$
---
$BuildHealth$
| The commit type (such as features and bugs) and the contribution
---
model (e,g., pull request and push model) are associated with build
breakage kerzazi2014why ; islam2017insights ; rausch2017empirical .
| $CommitMoment\to$
---
$BuildHealth$
| There is an association between the moment of contributions and the
---
rate of build breakage kerzazi2014why .
| $TeamDistribution$
---
$\to BuildHealth$
| The geographical distance of the team members is associated with
---
the build results kerzazi2014why .
| $Tools\to$
---
$BuildHealth$
| The languages and their tools are related to different build
---
breakage rates seo2014programmers .
| $ExtraComplexity$
---
$\to BuildHealth$
| Complex builds tend to break laukkanen2017problems .
---
| $FlakyTests\to$
---
$BuildHealth$
| Flaky tests favor the occurrence of build breakage laukkanen2017problems ;
rausch2017empirical .
---
| $ContributorType\to$
---
$BuildHealth$
| Less frequent contributors tend to break builds less rausch2017empirical .
---
| $TimeToFix\to$
---
$Costs$
| The time lost relates directly to a monetary cost kerzazi2014why .
---
| $Communication\to$
---
$TimeToFix$
| The feedback mechanisms and information speed affect the
---
awareness of a broken build and the time to fix it kerzazi2014why .
| $DeveloperRole\to$
---
$TimeToFix$
| The developer role is associated with the time to fix a broken
---
build kerzazi2014why .
| $CommitType\to$
---
$TimeToFix$
| The characteristics of the branches and code access (e.g., isolated
---
branches) are associated with the time to fix a broken build kerzazi2014why .
| $IntegrationFreq$
---
$\to TimeToFix$
| The integration frequency in the team affects the build fixing
kerzazi2014why .
---
| $ProgramLanguage$
---
$\to TimeToFix$
| The programming language is related to the time spent to fix
---
a broken build seo2014programmers .
| $ErrorUnderstand$
---
$\to TimeToFix$
| The understandability of the build failures directly impacts
---
the time needed to solve them vassalo2014every .
| $BuildFailType$
---
$\to TimeToFix$
| The build failure types are associated with different difficulty
---
levels vassalo2014every .
| $TestsVolume\to$
---
$CommitSize$
| Complex and time-consuming testing is a possible reason for large
---
commits laukkanen2017problems .
| $ContributorType\to$
---
$CommitSize$
| The type of contributor (e.g., casual) relates to the build
---
breakage rate reboucas2017how .
| $ContributorType\to$
---
$AutomatedTests$
| The contributor type is related to the number of automated
---
tests reboucas2017how .
| $Extracomplexity\to$
---
$ContributorType$
| The complexity of the jobs is related to the type of contributor
---
in the projects reboucas2017how .
| $CommitSize\to$
---
$MergeConflicts$
| Large commits are associated with merge conflicts laukkanen2017problems .
---
| $FixTools\to$
---
$ErrorUnderstand$
| Fix support tools improves the understandability of the build
---
logs vassalo2014every .
| $BuildFailType\to$
---
$ErrorUnderstand$
| The build failure type is associated with different levels of
---
understandability vassalo2014every .

Fig. 6 shows the unified and complete literature-based causal DAG, i.e., the
union of the assumptions cataloged in Tables 1, 2, 3, 4, 5, 6, 7. The
literature-based causal DAG expresses all known associations for the related
variables in the studied domain. The nodes on the DAG represent variables
(e.g., Continuous Integration and Bug Report), and the directed edges
connecting the variables represent an association between them. The
associations “flow” from one variable to another. For instance, according to
Fig. 6, Continuous Integration influences Bug Report, i.e., CI teams discover
more bugs and CI projects present fewer defects than NOCI projects (those that
do not adopt CI) amrit2018effectiveness ; vasilescu2015quality . Such DAG
associations could represent a positive or a negative association, and
whenever possible, this interpretation is discussed in the text. Note that the
variables $AutomatedTests$, $BuildHealth$, $IntegrationFrequency$, and
$TimeToFix$ are all represented by $ContinuousIntegration$. This causal DAG
built upon the existing literature is the starting point for identifying
variables that must be measured and controlled to allow a causal analysis of
the influence of CI on software quality.

####  3.1.2 DAG Building

To analyze the causal effect of CI on software quality, we need to be
attentive to all potential confounding effects, i.e., bias due to backdoor
paths (see Section 2.4). Thus, we draw a DAG containing a sufficient set of
variables that show the existing backdoor paths with respect to CI and
software quality. This new DAG is a subgraph of the DAG shown in Fig. 6.

Based on Reichenbach’s Common Cause Principle and the Markov Condition, we
know that a causal DAG should include the common causes of any pair of
variables in the DAG pearl2000models ; hernan2010 . Therefore, we can discard
several variables in Fig. 6 because they are external variables without
connecting with the variables under investigation (i.e., $CI$ and
$BugReport$), and they are not common causes for any variable selected for the
analysis (i.e., CI, Bug Report or one of its common causes). For instance,
$ProjectPopularity$ is associated with $BugReport$ but does not have an
association with no other variable on the DAG. Thus, $ProjectPopularity$ can
be discarded from our causal analysis. On the other hand, $Age$ is a common
cause associated with $ContinuousIntegration$ and $BugReport$, and it is
essential to analyze the causal effect between them since $Age$ is a potential
confounding factor (see Section 2.4).

###  3.2 RQ2. Is the causal effect of CI on software quality empirically
observable?

Based on the concepts of conditional independence and the d-separation rules
(see Section 2.3), we analyze a set of statistically testable restrictions as
implications of a model pearl2000models . Such restrictions (i.e., statistical
implications) are conditional or unconditional independencies between DAG
variables that must be found in any dataset generated by the causal processes
described in the DAG. By building a dataset with the variables in our causal
DAG, we can test the implications of the DAG statistically. Note that because
these statistical tests are based on the d-Separation rules, we are not only
checking for associations but also for causal relationships (i.e., considering
the “flow” between relationships in the DAG) pearl1991theory ; pearl1995causal
; pearl2000models . We are discovering the causal structure and inferring
causation from the initial DAG built in the RQ1 (section 3.1) and the testing
of their d-Separation implications on an empirical data set shalizi2021 .

The implications are in the form of unconditional independencies, like
$Age\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume$, which means $Age$ is
independent of $TestsVolume$. Alternatively, the implications may have the
form of conditional independencies, like
$MergeConflicts\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid
CommitFrequency$, which means $MergeConflicts$ is independent of $TestsVolume$
conditioned in $CommitFrequency$. Since we use the DAGitty R package
textor2017dagitty  to draw our causal DAGs, we obtain from the
impliedConditionalIndependence function a list of testable implications that
become our causal hypotheses.

We then collect data by mining software repositories to empirically analyze
the selected variables, allowing us to test the causal DAG and its statistical
implications, i.e., our causal hypotheses.

####  3.2.1 Collecting Data & Empirical Analysis

Based on the knowledge and assumptions acquired in the literature review and
the causal DAG built to answer the RQ1, we have support for building a dataset
to analyze the relationship between CI and software quality, including
confounding variables. Using the dataset, we can check the validity of our DAG
through statistical tests. Fig. 7 summarizes the dataset building process, and
we detail such process in the following. All tools, scripts, and information
necessary to reproduce the process described in the sequence are available in
our replication package
333https://github.com/elieziosoares/ci_quality_study_replication.

Figure 7: Mining Software Repository Process

Collecting Projects.  As illustrated in Fig. 7, we start with a list of 27,885
projects obtained from the most starred repositories in the GitHub Search
API444https://docs.github.com/en/rest/reference/ and a list of projects’ names
from the SonarQube web API555https://community.sonarsource.com/t/list-of-all-
public-projects-on-sonarcloud-using-api/33551. We searched for projects having
a public repository on GitHub to start the mining repository process by
collecting the history of developers’ contributions to them. The search
included 15 popular languages: C, C#, C++, Go, Java, JavaScript, Kotlin,
Objective-C, PHP, Python, Ruby, Rust, Scala, Swift, and TypeScript. So the
only initial premise was to have a public repository on GitHub.

Collecting Continuous Integration Data.  In the next steps, we search for data
in continuous integration services. Inspired by Hilton’s work hilton2016usage
, which searches for the usage of 5 different services — TravisCI, CircleCI,
AppVeyor, Werker, Cloud-Bees, and Jenkins CI, we search for the same services
and add a new one — GitHub Actions666https://github.com/features/actions. We
discovered a high CI service usage (54.3%) among the collected projects.
Travis CI 777https://travis-ci.org/ appeared as the most popular service in
our dataset.

Despite the evidence regarding a decreasing usage of Travis CI service
widder2018leaving ; decan2022 ; golzadeh2022 , it is worth mentioning that
Travis is still the most popular service in our dataset (see Table 8), which
has also been the case for recent research analyzing machine learning projects
using CI rzig2022 . As for GitHub Actions, although the relatively short time
since its launch (November 2019), it has been used by a significant number of
projects up to our study’s data collection date (2022). This observation
confirms the findings from Decan et al. decan2022 . When they analyzed 67,870
GitHub repositories, they found that more than 4 out of 10 use GitHub Actions
workflows. Golzadeh et al. golzadeh2022  also observed that while the adoption
rate of the other CI services decreased, GitHub Actions has a steadily
increasing adoption rate.

Table 8 displays the identified Continuous Integration (CI) services, the
number of projects categorized as users for each service, and the criteria
employed to categorize and classify the adoption of these services.

We build scripts to identify which CI service is used by each project. For
Travis CI, Circle CI, and Wercker, we check the public APIs of these services
by searching for the project name and organization. Specifically for Travis
CI, if we do not find any information through the Travis API, our scripts
search for the “.travis.yml” file using the GitHub API. We also keep track of
when the service configuration was initiated, specifically noting the date of
the first commit of the “.travis.yml” file.

For the remaining services, the script checks if the configuration file
exists. In the case of AppVeyor, we verify the presence of a file named
“appveyor.yml” within the GitHub repository using the API. For Jenkins, we
search for a “Jenkinsfile”, while for Circle CI, we look for a “config.yml”
inside a “.circleci” folder. Similarly, for GitHub Actions, we examine whether
a file with the extension “.yml” exists inside a “.github/workflows” folder.

Table 8: The CI service usage on the dataset and the classification criteria. CI Service | Qty | Criteria
---|---|---
Travis CI |  | 9,092
---
(32.6%)
| We search on Travis APIa for the existence of the project in the service.
---
If true, we search on GitHub for the existence of a “.travis.yml” file.
| GitHub
---
Actions
| 5,630
---
(20.1%)
| Using the search code feature of GitHub API, we search for the
---
extension “.yml” in the path “.github/workflows”.
Circle CI |  | 307
---
(1.1%)
| Through Circle CI APIb we search for the existence of the project on
---
the service. If not located, we searched for the file “config.yml” in the
path “.circleci”.
Jenkins |  | 58
---
(0.2%)
| Using the search code feature of GitHub API, we search for the file
---
“Jenkinsfile”.
AppVeyor |  | 29
---
(0.1%)
| Using the search code feature of GitHub API, we search for the file
---
“appveyor.yml”.
Wercker | 0 |  | Using the Wercker APIc we search for the existence of the project on
---
the service.
| No CI
---
Service
| 12,769
---
(45.7%)
ahttps://docs.travis-ci.com/user/developer/#api-v3
bhttps://circleci.com/docs/api/v2/
chttps://devcenter.wercker.com/development/api/endpoints/

Filtering projects.  Afterwards, we filtered out irrelevant repositories as
represented in Fig. 7. In particular, we consider only non-fork projects with
more than 100 stars. Despite the risks of applying MSR in empirical studies
and selecting non-representative projects munaiah2017curating , we selected
“engineered projects” as opposed to toy projects munaiah2017curating  and
applied a series of criteria, beginning by excluding projects smaller than
10MB oliveira2017draco ; oliveira2019finding . These two criteria are the
initial steps towards selecting projects that approach the “engineered
projects” concept munaiah2017curating . While mining data from the repository
and classifying projects for analysis, we ensure a rigorous selection of
projects that represent engineered projects. In the sequence, we detail each
step of mining and classification of projects.

Next, we consider only projects using either Travis CI or projects not using a
CI service. Including projects not using a CI service is essential because we
create a control group of no-CI projects in order to understand the effects on
the CI projects group. As mentioned earlier, we selected Travis CI as a target
because it is the most used service in our dataset. In addition, Travis CI has
a public API to obtain software builds data. After this filtering step, our
sample was reduced from 27,885 to 3,270 repositories, i.e., 2,527 repositories
using Travis CI and 743 not using a CI service.

Collecting Repository Data.  We mine the pull requests and issues of the
projects, filtering out those projects that do not have pull requests or
issues. We collect data from pull requests, issues, commits, and comments
using the GitHub Search API. We use the Travis API to collect data from
software builds, while we use two different services (Coveralls
888https://coveralls.io/ and SonarCloud 999https://sonarcloud.io/) to search
for code coverage information.

We collected pull requests and issues in the repositories using the endpoint
“pulls” and “issues” from the GitHub Search API. This process resulted in a
total of 1,425,493 pull requests and 3,328,221 issues. Next, we collected all
pull request comments and associated information regarding authors. All these
data allow us to compute a metric to measure $Communication$ detailed forward.

We also collect each pull request’s commit, sha, date, message, and author. To
collect the detailed commit information regarding lines of code added,
removed, or changed, we request it to GitHub API
101010https://api.github.com/repos/{owner}/{repo}/pulls/{pull_number}/files.
We processed the data to obtain the size of a commit, the number of lines of
added/removed code, the number of files modified, and the test files and test
lines. The commits data help us to compute metrics to measure
$CommitFrequency$, $TestsVolume$, and $MergeConflicts$.

Classifying Merge Conflicts. To identify the merge conflicts occurrences, we
used an algorithm (see Algorithm 1) to reconstruct the commits history and
identify a merge conflict when it was generated. The algorithm clones the
repository (line 1) and gets the list of commits (line 4). The algorithm gets
the parents for each commit and verifies the number of parent commits (line
6). If a commit has more than one parent, we call the git diff function to
them (line 7). Then, depending on the diff result, we can detect a conflict
(line 9) and append it to the list of merge conflicts (line 10). We manually
validated the algorithm with a random sample of 40 commits and achieved an
accuracy of 90%.

0: $remote$: URL for the project repository

10: $repository\\_path$: Local path to clone the project repository

1: $\text{repo}\leftarrow Repo.clone\\_from(remote,repository\\_path)$

22: $merge\\_conflicts\leftarrow\emptyset$

3:

34: for all $commit$ in $repo$.$commits$() do

5:

46: if $commit.parents>1$ then

57: $merge\\_diff\leftarrow
repo.git.diff(commit.parents[0],commit.parents[1])$

8:

69: if $merge\\_diff$ contains $"<<<<<<<HEAD"$ and $"<<<<<<<"$ then

10: Append $commit.sha$ to $merge\\_conflicts$

11:

12: end if

13:

14: end if

15:

16: end for

17:

18: return $merge\\_conflicts$

19: End function

Algorithm 1 Detect Merge Conflicts

Classifying Bugs. To classify issues as bugs, we consider only projects using
GitHub issues and labeling them. We mapped the labels used in their
repositories through a semi-manual inspection, i.e., we began with a script to
a preliminary parse relying on a list of bug-related keywords
vasilescu2015quality ; jadson2022 . Then, we manually validated the labels for
each repository using GitHub issues that have assigned to them. This approach
is similar to the approach used by Vasilescu et al. vasilescu2015quality  and
Santos jadson2022 .

The exception to our primary approach is a set of 10 projects with many issues
without labels. To avoid missing these data, we adopt a conservative approach
classifying as bugs issues containing the terms “bug” or “fix” in their title
or their body.

Classifying CI Projects. In order to avoid the CI Theater felidre2019citeather
; fowler_citeather  and also to adopt recommendations from Soares et al.
soares2021effects , in addition to the use of Travis CI, we consider build and
code coverage information, meaning that we select projects that have actual
build and test activity. After achieving a set of 74 CI projects surviving all
filtering stages, we randomly drew 74 out of 95 no-CI projects to balance our
dataset. Table 9 shows a summary of the final dataset.

Data Aggregation. With the collected data, we can analyze dimensions of
releases such as Commit Frequency, Test Volume, Merge Conflicts,
Communication, Bug Reports, and Age variables. We cannot measure Issue Type
and Overconfidence for two reasons. First, classifying $IssueType$ from the
collected issue data is challenging given that projects do not have consistent
patterns for classifying issue patterns (e.g., standardized tags, such as
“enhancement”, “perfective”, “corrective” used in a standardized manner).
Second, overconfidence is a subjective feeling not feasible to measure through
collecting data from repositories. Therefore, we consider these two variables
as latent (i.e., unmeasured). Latent variables can still be represented and
analyzed in a causal DAG, even if they cannot be statistically tested.
However, we can still interpret these latent variables based on the
statistical tests of the other variables within the DAG hernan2010 .

In the DAG and the dataset, we used the variable Commit Frequency, replacing
Commit Size. These variables have an intrinsic relationship since more
frequent commits tend to be smaller. However, it is noteworthy that commit
frequency is most noticeable in the context of projects using Travis CI
zhao2017impact . In addition, more frequent commits engender a cascade of CI
builds and verifications, potentially detecting bugs earlier, as cited in
soares2021effects . In light of this, we employed a commit frequency metric to
build a more consistent and complete DAG.

We derive the metrics by aggregating their values by project releases. We
collected data covering 12 development months, similar to data analyzed by
Zhao et al. zhao2017impact  and Santos et al.jadson2022 . Notably, all 148
projects under scrutiny encompass a data-mined span of 12 months. We consider
the first 12 months for CI projects starting from the month when CI was
adopted. For no-CI projects, we consider the starting month for analysis
according to the median $Age$ of the CI projects, similar to Sizilio et al.
sizilio2019empirical .

In addition to $Age$, the $Size$ of the projects are also balanced. We tested
this difference through a Wilcoxon Rank test and found no statistically
significant difference between the two groups ($p-value=0.7372$). Regarding
the difference in bug reports, we also obtained a negligible difference
($p-value=0.1318$). On the other hand, we found a difference between the two
groups for the number of stars on GitHub ($p-value=0.000679$), for the number
of issues ($p-value=0.007304$), and for pull-requests
($p-value=1.737{e}{-07}$). In simple terms, the CI projects are more active
and popular, but not necessarily with greater maturity and size.

For each project release in such time, we compute the following metrics:

  * •

Commit Frequency represents the aggregate count of commits across all pull
requests within a given release.

  * •

Communication is the sum of comments and review comments in the pull requests.
We consider the average of communication in a release.

  * •

Merge Conflicts is the number of merge conﬂicts in a release. We traverse the
repositories commit tree and use the GIT command diff to verify the merge
commits and the conflicts.

  * •

Age is the number of days of a repository from its creation until the release
date.

  * •

Test Volume is the proportion of lines of test code modiﬁed (added, removed,
or changed) in the commits. We applied the strategy from Nery et al.
sizilio2019empirical  to identify test files. We consider the median test
volume in a release.

  * •

Bug Reports is the number of bugs reported in a release.

  * •

CI is a binary categorical variable indicating whether the project uses CI.

Table 9: Summary of the Data Set. Number of Projects | 148
---|---
Number of Pull Requests | 18,961
Number of Commits | 59,034
Number of Builds | 16,619
Number of Issues | 41,866
Number of Bugs | 3,199
Number of Merge Conflicts | 403

####  3.2.2 DAG Implications Testing

Having the set of testable causal hypotheses obtained in RQ1 and the produced
dataset (Section 3.2.1), we can investigate if the implications from the
causal DAG are actually held in the empirical dataset. We test the
unconditional independence causal hypotheses (e.g.,
$X\mathrel{\perp\mspace{-10.0mu}\perp}Y$) using the dcov.test function from
the energy R package 111111https://cran.r-project.org/package=energy. This
test verifies if two variables on the dataset are independent. We test the
conditional independence causal hypotheses (e.g.,
$X\mathrel{\perp\mspace{-10.0mu}\perp}Y\mid Z$) with the Kernel conditional
independence test from the CondIndTests R package heize2018kci . Both tests
are non-parametric and suitable for our data.

###  3.3 RQ3.What would be an accurate causal theory for CI?

Considering the investigations in RQ1 and RQ2, we have information regarding
which independence tests have passed or failed in our analyses. Such
information allows us to propose adaptations to the causal DAG built in RQ1.
For example, by testing the data, we can observe a dependency between $Age$
and $Communication$, so we propose a new edge $Age\to Communication$. We build
a new causal DAG by combining the knowledge from the literature (RQ1) and the
statistical tests on the dataset (RQ2). We call this hybrid approach data-
validated causal DAG. The data-validated causal DAG is compatible with the
dataset in terms of the independency relationships between the variables.
Then, we perform a data-validated approach for causal discovery.

There are dozens of algorithms for causal discovery from data, such as PC
spirtes2000causation , that rely on conditional independence tests to discover
the causal structure from the data. On the other hand, Cartwight
cartwright1994causes  argues that causal investigation requires background
information too (i.e., causal assumptions)—“No causes in, no causes out”.

Thus, to implement the data-validated causal DAG, we start getting background
information from the literature review and building the initial causal DAG in
RQ1 (i.e., causal assumptions from the literature). Next, we combine the
knowledge acquired in RQ2 (i.e., the independence relationships that were not
present in the data), and finally, we structure our RQ3 procedure in three
steps.

Step 1. By analyzing the rejected hypotheses in RQ2 (Section 4.2), in
conjunction with d-separation rules (Section 2.3), we can infer connections
that should be added or removed in the data-validated causal DAG.

Step 2. We generate the data-validated causal DAG after identifying the
disconnected vertices and the necessary changes from Step 1.

Step 3. We test the d-separation implications on the data-validated causal
DAG. If any test fails, we return to Step 1 and refine the DAG.

Figure 8: Hypothetical causal DAG.

Fig. 8 presents a hypothetical DAG to illustrate the steps to obtain the data-
validated DAG. The DAG contains a Fork structure (i.e.,
$QtyPullRequests\leftarrow Age\to CommitFrequency$) and a collider (i.e.,
$Age\to CommitFrequency\leftarrow DevelopersMaturity$.

Analyzing Forks and Chains. The d-Separation rules impose the same principle
for the analysis of forks or chains (see Section 2.3), i.e., if this structure
is true, $QtyPullRequests$ and $CommitFrequency$ should be statistically
dependent because an association flows through forks and chains. Otherwise,
these variables should be independent when we block the path by conditioning
on the middle variable $Age$, as in Fig. 9(a).

Figure 9: Examples of causal DAG structure testing.

Therefore, we can statistically test the fork in Fig. 9. We should perform a
conditional independence test to check if $QtyPullRequests$ and
$CommitFrequency$ are independent conditioning on $Age$
($QtyPullRequests\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency\mid Age$)
as in Fig. 9(a). If the test rejects such a hypothesis, meaning that the
structure is wrong, we may hypothesize different structures.

For instance, “what if the structure would be a collider $QtyPullRequests\to
Age\leftarrow CommitFrequency$? (see Fig. 9(b)). In this case,
$QtyPullRequests$ and $CommitFrequency$ would be independent without
conditioning. On the other hand, if the variables have a dependence that does
not interrupt when conditioning on $Age$, they share a relationship passing
through another path, then we add a new edge between
$QtyPullRequests\leftarrow CommitFrequency$, as illustrated in Fig. 9(c).

Figure 10: Examples of causal DAG collider structure testing.

Analyzing Colliders. The d-Separation rules (see Section 2.3) imply that the
variables in a collider are independent because the collider blocks the
association flow, i.e., $Age$ and $DevelopersMaturity$ should be statistically
independent. On the other hand, if we condition on $CommitFrequency$, they
would be dependent because the condition opens the association flow in a
collider.

We can statistically test the structure in Fig. 10(a) by testing the
independence between $Age$ and $DevelopersMaturity$
($Age\mathrel{\perp\mspace{-10.0mu}\perp}DevelopersMaturity$). In case the
variables are not independent, it means that the structure is not a collider
or there exists another path linking them. Thus, we can verify if the
structure is a chain as illustrated in Fig. 10(b) by verifying if conditioning
on $CommitFrequency$ they become independent. Finally, if the last hypothesis
fails, i.e., $Age$ and $DevelopersMaturity$ are not independent, even if
conditioning on $CommitFrequency$, then this suggests another path exists
between them. In this case, we would add a new edge $Age\to
DevelopersMaturity$, as shown in Fig. 10(c).

##  4 Results

###  4.1 RQ1. What does the literature proclaim about CI and software quality?

Considering the Reichenbach’s Common Cause Principle reichenbach1991direction
and the Markov Condition pearl2000models ; hernan2010 , in order to produce a
sufficient causal DAG, we can exclude any variable that is not a common cause
between CI and software quality or one of their ancestors. Thus, we remove
from the DAG all non-common cause variables from the causal DAG in Fig. 6, for
instance, $ResolutionTime$, and $ProgrammingLanguage$. In addition, we unified
the variables $BugResolution$ and $BugReport$, as all connections and the
meaning of $BugResolution$ are contained in $BugReport$.

Fig. 11 shows the literature-based DAG after the variable-selection process.
According to the DAG, Continuous Integration (CI) has a direct influence on
Bug Report amrit2018effectiveness ; vasilescu2015quality , as well as an
indirect influence through the Communication cassee2020silent ;
huang2012taxonomy  and developers’ overconfidence variables
pinto2017inadequate ; pinto2018work ; huang2012taxonomy . That means that CI
influences the communication between contributors, which, in turn,
communication influences the number of bug reports. A similar phenomenon
occurs with developers’ overconfidence, but overconfidence tends to have a
negative effect on the number of bug reports, i.e., overconfidence may
actually increase the number of bug reports or, at least, prevent a team from
the opportunity of reducing the number of bug reports. Fig. 11 shows the
literature-based DAG after the variable-selection process. According to the
DAG, Continuous Integration (CI) has a direct influence on Bug Report
amrit2018effectiveness ; vasilescu2015quality , as well as an indirect
influence through the Communication cassee2020silent ; huang2012taxonomy  and
developers’ Overconfidence variables pinto2017inadequate ; pinto2018work ;
huang2012taxonomy .

The age of a project influences the CI practices zaidman2008mining ;
zaidman2010studying ; hilton2018large ; sizilio2019empirical  and also
influences bug reports vasilescu2015quality , i.e., Age is a common cause for
both CI and bug reports, thus opening a backdoor path for bias (see Section
2.4). Similarly, the commit frequency opens other backdoor paths: (i)
$ContinuousIntegration\leftarrow CommitFrequency\leftarrow TestsVolume\to
BugReport$; and (ii) $ContinuousIntegration\leftarrow
CommitFrequency\leftarrow IssueType\to Communication\to BugReport$. These
backdoor paths represent risks of confounding the influence of CI on other
variables such as Bug Report. Thus, it is necessary to condition for some
variables blocking these backdoor paths to obtain causal estimations regarding
the total causal effect of CI on Bug Report. The minimal conditioning set is
formed by $Age$ plus $CommitFrequency$, capable of blocking all backdoor paths
(i.e., confounding effects).

Conditioning for the confounding allows us to measure whether CI has a certain
level of causal effect on Bug Reports despite the presence of confounding
variables $Age$ or $CommitFrequency$.

Figure 11: Final literature-based DAG for CI, Bug Reports and their co-
variables.

###  4.2 RQ2. Is the causal effect of CI on software quality empirically
observable?

With the causal DAG obtained in RQ1 (Section 4.1) and the dataset we produced
(Section 3.2.1), we can investigate if the data supports the relationships
depicted in the causal DAG. Such evaluation is possible because we can derive
statistically testable implications (see Section 2.3) using the d-separation
properties on the causal DAG (see Fig. 11), and we can verify whether the same
statistical independency also exists between the variables of the dataset.
Thus, we answer RQ2 by testing the dataset against the proposed set of causal
hypotheses from Table 10. To test such causal hypotheses, we use dcov test and
Kernel conditional independence test.

When necessary, we test the unconditional independence hypotheses (i.e., two
variables are independent without conditioning on other variables) using the
dcov test. This test evaluates if two variables are independent, returning an
$R$ value between 0 and 1 when they are independent. If it returns a value
greater than 1 the variables are considered to be dependent. For example, the
dcov test for a hypothesis $H_{n}$
“$Age\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency$” returns an $R$
value of 49.5566 (i.e., larger than 1), meaning that we reject the hypothesis
that $Age$ and $CommitFrequency$ are independent.

On the other hand, we test the conditional independence hypotheses (i.e., two
variables are independent when conditioning on another set of variables) in
Table 10 using Kernel conditional independence test heize2018kci . This test
evaluates the null hypothesis that two variables, Y and E, are independent
conditioning on a set of variables X and returns a p-value for the null
hypothesis. Thus, a small p-value (lower than the significance level) rejects
the null hypothesis, indicating that the variables are not independent
conditioning on a set of variables. If a p-value is high, the test fails to
reject the null hypothesis, meaning that Y and E are likely independent
conditioning on X.

As an example, the Kernel conditional independence test for $H_{m}$
“$MergeConflicts\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$CommitFrequency$” (i.e., $MergeConflicts$ is independent of $TestsVolume$
conditioning on $CommitFrequency$) returns a p-value of 0.542159, failing to
reject the null hypothesis for $H_{m}$. Thus, we assume that $MergeConflicts$
and $TestsVolume$ are independent when conditioning on $CommitFrequency$ since
the p-value is higher than a significance level of 0.05 (in fact, we fail to
reject the null hypothesis).

Table 10: The results of the conditional independence tests for RQ2.
Conditional Independence
---
Causal Hypotheses | p-value
|  H1. $Age\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency\mid$
$ContinuousIntegration$
---
1.41e-07*
|  H2. $Age\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$ContinuousIntegration$
---
2.40e-12*
|  H3. $Age\mathrel{\perp\mspace{-10.0mu}\perp}Communication\mid$
$ContinuousIntegration$
---
1.16e-06*
|  H4. $Age\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$ContinuousIntegration$
---
0.254925
|  H5. $BugReport\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency\mid$
$Communication$, $ContinuousIntegration$, $TestsVolume$
---
0.000000*
|  H6. $BugReport\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.324586
|  H7. $BugReport\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$Communication$, $TestsVolume$,
---
$ContinuousIntegration$
0.344638
|  H8. $Communication\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.486532
|  H9. $Communication\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$ContinuousIntegration$
---
0.000000*
|  H10.$MergeConflicts\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.246583
$*p<0.05$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”
$\mid$ “Conditioning on…”

As we have seen in Table 10, we reject hypotheses $H_{1}$, $H_{2}$, $H_{3}$,
$H_{5}$, and $H_{9}$. These rejections are sufficient to conclude that if
these d-separation implications from the causal DAG from our RQ1 (Fig. 11) are
not true, the causal DAG is not consistent with the empirical data. That means
that the actual development processes from our studied projects (which
generate the observed data) differ from the relationships shown in the
literature-based DAG (Fig. 11).

In simple terms, although our literature-based DAG indicates that $Age$ and
$CommitFrequency$ are related only through $ContinuousIntegration$ influence
(i.e., they are independent when conditioning on $ContinuousIntegration$), our
tests have shown that they actually share a different relationship (i.e., the
tests did not confirm the conditional independence). Similarly, our tests show
that the relationship between $Age$ and the variables $TestsVolume$,
$Communication$, and $MergeConflicts$ are not independent when conditioning on
$ContinuousIntegration$. Therefore, the tests suggest that $Age$ potentially
has a direct relationship with these variables.

A parallel case is the one concerning $Communication$ and $TestsVolume$. The
Literature-based DAG (Fig. 11) presents a relationship between these two
variables passing through $ContinuousIntegration$. However, our tests did not
confirm the conditional independence
$Communication\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$ContinuousIntegration$.

Another example is the relationship between $BugReport$ and $CommitFrequency$.
Even if we disregard the influences of $Age$, $Communication$,
$ContinuousIntegration$, and $TestsVolume$, they remain dependent, which means
that our data shows that $BugReport$ and $CommitFrequency$ potentially have a
direct relationship. Our data shows that $BugReport$ potentially has a direct
relationship with $CommitFrequency$. In RQ3 (Section 4.3), we deal with this
set of observations on the failed hypothesis.

###  4.3 RQ3. What would be an accurate causal theory for CI?

To discover a plausible causal structure, we start from the failed causal
hypotheses in RQ2 and explore where the literature-based DAG (Section 4.1)
does not match the evidence from the empirical dataset (Section 4.2). We
propose an alternative DAG (data-validated DAG) based on the observed
mismatches and the testing hypotheses derived from such mismatches (i.e.,
failed causal hypotheses).

####  4.3.1 The relationship between $Age$ and $CommitFrequency$

In the literature-based causal DAG in Fig. 11, there is a chain $Age\to$
$ContinuousIntegration$ $\to CommitFrequency$, as shown in Fig. 12 (a).
Considering the d-separation rules (Section 2.3), we would expect that when
conditioning on $ContinuousIntegration$, the association flow between $Age$
and $CommitFrequency$ would interrupt, so they should be independent. However,
in the analysis from RQ2, we rejected hypothesis $H_{1}$
($Age\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency\mid$
$ContinuousIntegration$, see Table 10).

Figure 12: (a) Causal structure (chain) involving $Age$,
$ContinuousIntegration$, and $CommitFrequency$ as expressed in the literature-
based causal DAG. (b) The new proposed causal structure concerning $Age$,
$ContinuousIntegration$, and $CommitFrequency$ after statistical validations.

In this case, another path to the association between $Age$ and
$CommitFrequency$ may exist. An alternative hypothesis is a wrong collider
structure between $Age\to BugReport\leftarrow Communication$. An association
may occur through this path if this structure is not a collider. First, we
tested the collider structure by formulating the hypothesis in Table 11. The
hypothesis says that $Age$ would become independent of $CommitFrequency$ when
conditioning on $ContinuousIntegration$ and $BugReport$
($Age\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency\mid$
$ContinuousIntegration,BugReport$). The result rejects such a hypothesis,
which means a persistent dependence between $Age$ and $CommitFrequency$ when
conditioning on $ContinuousIntegration$ and $BugReport$. Therefore, we add a
new edge between $Age$ and $CommitFrequency$, obtaining the structure shown in
Fig. 12 (b).

Table 11: Conditional independence test for the relationship between $Age$ and
$CommitFrequency$. Conditional Independence
---
Causal Hypotheses | p-value
|  H1. $Age\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency\mid$
$ContinuousIntegration,BugReport$
---
$2.576492\mathrm{e}{-08}$*
$*p<0.05=dependence$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”
$\mid$ “Conditioning on…”

####  4.3.2 The relationship between $Age$ and $TestsVolume$

Analyzing hypothesis $H_{2}$
($Age\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid
ContinuousIntegration$, see Table 10) and the Fig. 13 (a), we observe that
given the following chain: $Age\to ContinuousIntegration\to TestsVolume$,
there is an association between $Age$ and $TestsVolume$, but this association
would interrupt when conditioning on $ContinuousIntegration$. However, the
test performed in RQ2 rejected such a hypothesis, revealing that $Age$ and
$TestsVolume$ remain dependent.

Figure 13: (a) Causal structure involving $Age$ and $TestsVolume$ as expressed
in the literature-based causal DAG. (b) The new proposed direct path
concerning $Age$ and $TestsVolume$. Table 12: Conditional independence tests
for the relationship between $Age$ and $TestsVolume$. Conditional Independence
---
Causal Hypotheses | p-value
|  H1. $Age\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$ $BugReport$
---
0.0*
|  H2. $Age\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$ContinuousIntegration,BugReport$
---
$8.518427\mathrm{e}{-10}$*
$*p<0.05$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”
$\mid$ “Conditioning on…”

Therefore, these variables should have a direct association, or the structure
that links them through $BugReport$ should be a chain rather than a collider.
In this way, we should test this hypothesis in the form of conditional
independence presented in Table 12, which expresses that if the structure is a
chain when conditioning on the middle variable, $Age$, and $TestsVolume$ would
become statistically independent. The test results in Table 12 show that even
conditioning on $CI,andBugReport$, or just on $BugReport$, the variables $Age$
and $TestsVolume$ remain dependent. Thus, we add a direct edge between them,
as shown in Fig. 13 (b).

####  4.3.3 The relationship between $Age$ and $Communication$

Based on the existing paths between $Age$ and $Communication\mid
ContinuousIntegration$ in the literature-based DAG (see Fig. 14 (a)), the
hypothesis $H_{3}$ ($Age\mathrel{\perp\mspace{-10.0mu}\perp}Communication\mid$
$ContinuousIntegration$, see Table 10) says that $Age$ and $Communication$
should be independent when conditioning on $ContinuousIntegration$.

Figure 14: (a) Causal paths between $Age$ and $Communication$ as expressed in
the literature-based causal DAG. (b) The new proposed directed path between
$Age$ and $Communication$.

Rejecting hypothesis $H_{3}$
($Age\mathrel{\perp\mspace{-10.0mu}\perp}Communication\mid$
$ContinuousIntegration$) indicates that $Age$ and $Communication$ remain
dependent. This dependency may be due to a wrong structure in the collider
$Age\to BugReport\leftarrow Communication$ that could be a chain, but in this
case, when conditioning in $BugReport$, $Age$, and $Communication$ would
become independent.

Another hypothesis is a wrong structure in the chain $Age\to
ContinuousIntegration\to Communication$ that may be a collider, and then when
we conditioned in $ContinuousIntegration$, we opened the association flow,
making $Age$ and $Communication$ dependent. In this way, we should test these
hypotheses by testing the independencies depicted in Table 13 and Table 14.

Table 13: Conditional independence tests for the hypothesis related to
relationship between $Age$ and $Communication$. Conditional Independence
---
Causal Hypotheses | p-value
|  H1. $Age\mathrel{\perp\mspace{-10.0mu}\perp}Communication\mid$ $BugReport$
---
0.0*
|  H1. $Age\mathrel{\perp\mspace{-10.0mu}\perp}Communication\mid$
$BugReport,ContinuousIntegration$
---
1.251478e-05*
$*p<0.05$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”
$\mid$ “Conditioning on…”
Table 14: Unconditional independence tests for the hypothesis related to
relationship between $Age$ and $Communication$. Unconditional Independence
---
Causal Hypotheses | R
|  H2. $Age\mathrel{\perp\mspace{-10.0mu}\perp}Communication$
---
11.86268 *
*$R>1$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”

The test result in Table 13 shows that even conditioning on $BugReport$, the
variables $Age$ and $Communication$ remain dependent. Thus, there exists a
dependency through another path. The result in Table 14 does not confirm the
hypothesis of a collider in $Age\to ContinuousIntegration\leftarrow
Communication$ since without conditioning on $ContinuousIntegration$, $Age$
and $Communication$ remain dependent. Therefore, we add an edge $Age\to
Communication$, as illustrated in Fig. 14 (b).

####  4.3.4 The relationship between $CommitFrequency$ and $BugReport$

The rejection of hypothesis $H_{5}$
($BugReport\mathrel{\perp\mspace{-10.0mu}\perp}CommitFrequency\mid$
$Communication$, $ContinuousIntegration$, $TestsVolume$, see Table 10),
similarly, indicates that the structures shown in Fig. 15 (a) are not
accurate.

Figure 15: (a) Causal paths between $CommitFrequency$ and $BugReports$ as
expressed in the literature-based causal DAG. (b) The new proposed directed
path between $CommitFrequency$ and $BugReports$.

According to these structures, we expected that $BugReport$ and
$CommitFrequency$ would become independent when conditioning on
$ContinuousIntegration$, $Communication$, and $TestsVolume$. However, the test
did not confirm the independence (see Table 10), even testing all variables
conditioned together or separately. On the other hand, we tested if
$BugReport$ and $CommitFrequency$ are unconditionally independent and got an
$R-value$ of 6.009902, indicating a dependency between them.

These tests suggest the existence of another path between $BugReport$ and
$CommitFrequency$. Thus, we add a new edge linking them as shown in Fig. 15
(b).

####  4.3.5 The relationship between $Communication$ and $TestsVolume$

Analyzing the rejection of hypothesis $H_{9}$, we observed that when
conditioning $Communication$ and $TestsVolume$ on $ContinuousIntegration$ (see
Fig. 16 (a)), the dependence between $Communication$ and $TestsVolume$ does
not disappear. We then tested if the collider $TestsVolume\to
CommitFrequency\leftarrow IssueType$ would be wrong. If this hypothesis is
correct, the association is flowing through this path, then conditioning on
$CommitFrequency$ would block the association.

In this sense, we test the conditional independence presented in Table 15 and
confirm the hypothesis that the structure is not a collider. Therefore, we
reorient the edge between $CommitFrequency$ and $IssueType$, as present Fig.
16 (b).

Figure 16: (a) Causal structure between $Communication$, and $TestsVolume$ as
expressed in the literature-based causal DAG. (b) The new proposed structure
between $Communication$ and $TestsVolume$ after statistical validations. Table
15: Conditional independence tests for the hypothesis related to relationship
between $Communication$ and $TestsVolume$. Conditional Independence
---
Causal Hypotheses | p-value
|  H1. $Communication\mathrel{\perp\mspace{-10.0mu}\perp}$ $TestsVolume\mid$
$ContinuousIntegration$, $CommitFrequency$
---
0.862325
$*p<0.05$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”
$\mid$ “Conditioning on…”

####  4.3.6 Data-Validated Causal DAG

Afterward, we obtain the first version of data-validated DAG in Fig. 17,
raising a new set of statistical implications to test. Table 16 shows such
implications as causal hypotheses and the results of the independence tests.

Figure 17: Initial version of data-validated DAG for CI, Bug Reports and their
co-variables. Table 16: The results of the conditional independence tests for
RQ3. Conditional Independence
---
Causal Hypotheses | p-value
|  H1. $Age\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency,ContinuousIntegration$
---
0.269635
|  H2. $BugReport\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.321338
|  H3. $Communication\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.486532
|  H4. $Communication\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$Age,CommitFrequency,ContinuousIntegration$
---
2.05e-12 *
|  H5. $MergeConflicts\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.246583
*$p<0.05$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”
$\mid$ “Conditioning on…”

After the statistical analysis, the $H_{4}$
($Communication\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$ $Age$,
$CommitFrequency$, $ContinuousIntegration$, see Table 16) was rejected. This
result means that even conditioning on $Age,CommitFrequency$ and
$ContinuousIntegration$, the variables $Communication$ and $TestsVolume$
remain dependent, i.e., another open path exists between them. Since the paths
by $BugReport$ is a collider (see Fig. 18 (a)), we may infer that there is a
path that this first version of data-validated DAG (Fig. 17) does not reflect
the true causal relationship. Therefore, we add a new edge between
$TestsVolume$ and $Communication$ (as shown in Fig. 18 (b)) and achieve a new
version of the data-validated DAG shown in Fig. 19.

Figure 18: (a) Causal paths between $TestsVolume$ and $Communication$ as
expressed in the literature-based causal DAG. (b) The new proposed direct path
between $TestsVolume$ and $Communication$.

Figure 19: data-validated DAG for CI, Bug Reports and their co-variables.

The data-validated DAG shown in Fig. 19 raises a new set of statistical
implications depicted in Table 17. After the statistical analysis, all these
four hypotheses were confirmed, meaning that the dataset upholds all
implications of the data-validated DAG. Thus, we can conclude that we find a
plausible model in our data-validated DAG (Fig. 19).

Table 17: The results of the conditional independence tests for RQ3.
Conditional Independence
---
Causal Hypotheses | p-value
|  H1. $Age\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.269635
|  H2. $BugReport\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.321338
|  H3. $Communication\mathrel{\perp\mspace{-10.0mu}\perp}MergeConflicts\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.486532
|  H5. $MergeConflicts\mathrel{\perp\mspace{-10.0mu}\perp}TestsVolume\mid$
$CommitFrequency$, $ContinuousIntegration$
---
0.384515
*$p<0.05$
$\mathrel{\perp\mspace{-10.0mu}\perp}$ “Independent of…”
$\mid$ “Conditioning on…”

##  5 Discussion

In RQ1 4.1, we cataloged a set of claims related to Continuous Integration
that we verified empirically in RQ2 4.2. Finally, in the RQ3 4.3, we proposed
a new causal DAG mixing the literature knowledge with the empirical data.

The data-validated DAG shown in Fig. 19 shows a more significant influence of
$Age$ than in the previous literature-based DAG (Fig. 11). $Age$ is an
important source of confounding effects since it relates to $TestsVolume$,
$CommitFrequency$, $ContinuousIntegration$, $Communication$, and $BugReport$.

In the data-validated DAG, we can also observe that $ContinuousIntegration$
influences $CommitFrequency$ and $MergeConflicts$, while $CommitFrequency$
also contributes to more $MergeConflicts$ and indirectly to more
$communication$. This way, $ContinuousIntegration$ influences, directly and
indirectly, to obtain more $Communication$ in the software development
process. $Communication$, in turn, has a causal relationship with $BugReport$.

The negative side of $ContinuousIntegration$ is the occurrence of
$MergeConflicts$ and the promotion of an $Overconfidence$ in the development
environment. We conjecture that CI is believed to decrease the commit size
because of the increased commit frequency. This higher frequency engenders a
favorable scenario for an increase in merge conflicts.

As for the overconfidence, it may arise from the belief that the automated
environment provided by CI is always reliable. Thus, we refer to it as
confidence over the reasonable, which is not necessarily grounded on objective
mechanisms and metrics that give such security and confidence(e.g., code
coverage, test quality, code smells checking). Such an $Overconfidence$ may
increase $BugReport$ since developers may relax their attention to detail,
outsourcing such quality control to automation developed in the scope of CI.

Finally, we observe that $ContinuousIntegration$ also, directly and
indirectly, influences $BugReport$. $ContinuousIntegration$ has a direct
relationship with $BugReport$ and has an indirect influence through
$Communication$, $CommitFrequency$, $TestsVolume$, and $Overconfidence$. While
$Communication$, $CommitFrequency$, and $TestsVolume$ affects $BugReport$
negatively, $Overconfidence$ affects $BugReport$ positively.

###  5.1 Implications for researchers

The DAGs presented in this study, especially the data-validated DAG (Fig. 19),
may serve as a baseline for further CI and software quality investigations.
This DAG brings findings from the combination of assumptions from the
literature confronted with an empirical dataset. Other studies can reinforce,
refute, or expand our findings and the DAGs, contributing to a more robust
theory on the effects of CI on software quality (our replication
package121212https://github.com/elieziosoares/ci_quality_study_replication may
be helpful).

We can also consider the differences emerging from the comparison between the
literature-based DAG (Fig. 11) and the data-validated DAG (Fig. 19). The
changed associations arise from the data and reveal exciting findings to be
further investigated by researchers. As well as the edges additions originated
in data-validated DAG.

The data-validated DAG (Fig. 19) added new edges. The addition of $Age\to
CommitFrequency$ (Section 4.3.1) aligns with Zhao et al. zhao2017impact  that
found a slight decreasing trend in the number of non-merge commits over time.
As interpreted by Brindescu et al. brindescu2014 , this finding could be
related to the project activity level changes during their lifecycle. The
projects tend to switch activities from adding new features to performing
corrective changes and similar ones brindescu2014 .

In this line, as the project ages, more $TestsVolume$ tend to exist ($Age\to
TestVolume$). Despite the literature used to build literature-based DAG in our
RQ1 (Section 4.1) does not explore the direct relationship between $Age$ and
$TestsVolume$, our dataset analysis reveals it. Sizilio Nery et al.
sizilio2019empirical  investigates the evolution of test ratio (a metric
similar to our $TestsVolume$) and relates a change in the test ratio evolution
trend to CI employment, NOCI projects have a negligible change in the test
ratio over time. In our dataset, a causal relationship exists between $Age$
and $TestsVolume$, i.e., as the $Age$ of a project advances, the greater the
$TestsVolume$ independently of CI employment or not. Future investigations may
study qualitatively how the tests evolve regarding the quality and motivation
behind such test efforts.

So is the $Age$ influence on $CommitFrequency$ and $TestsVolume$, $Age$ also
causes an increase in $Communication$, in terms of pull requests comments. For
example, an interesting observation arises from the causal flow $Age\to
Communication$ of our data-validated DAG. We note that in the literature,
Cassee et al. [46] investigate the association between CI adoption and pull-
request communication. Our study, however, identifies an effect on
communication separated from CI, indicating that the Age of a project may
affect $Communication$ over time rather than CI. Therefore, our data-validated
DAG can be used to better understand the existing associations observed in the
literature.

In this line of thought, we demonstrate in Section 4.3.6 that $TestsVolume$
also influences $Communication$ ($TestsVolume\to Communication$). It is
possible that as test volume increases, more discussion is generated regarding
the commits. There is an interrelationship between $TestsVolume$,
$CommitFrequency$, and $Communication$, as discussed by Fowler: “Integration
is about communication” fowler_ci .

Similarly, these three variables affect $BugReport$. Section 4.3.4 presents an
addition of the edge $CommitFrequency\to BugReport$, suggesting that the more
frequent the commits, the less the $BugReports$. Indeed, the more integration,
the more builds and quality checks are performed. Eyolfson et al. Eyolfson2013
found that more frequent committers produce fewer bug-introducing commits.
$CommitFrequency$ could have a similar foundation: the more frequent the
commits, the more knowledge, and familiarity with the domain, architecture,
and tools used in the project, providing less faulty development. Researchers
could perform empirical studies on the relationship between commit frequency
and code or project quality.

If more tests and frequent commits mean more communication, then CI is
essentially promoting more communication, directly and indirectly. Moreover,
more communication generates more knowledge and fewer errors, thus producing a
high-quality software product. Cassee et al. cassee2020silent  highlight the
necessity of investigating the social aspects of software development. In
their work, Cassee et al. cassee2020silent  relate CI with a decrease in the
number of comments in code reviews; however, they call CI a “Silent Helper”
because CI plays a role communicating sufficiently to maintain the same level
of activity of projects with more comments. This view corroborates the idea
that the CI environment, $CommitFrequency$, $TestsVolume$, all of it is about
communication.

Further studies may investigate this techno-social aspect, answering how and
why $CommitFrequency$ and $TestsVolume$ can potentialize communication.
Furthermore, investigating how the CI environment contributes to
communication, i.e., in which aspects it communicates and the mechanisms,
tools, and metrics through which CI is manifested, may mitigate the harmful
false sense of confidence ($Overconfidence$) generated by CI in some
environments. For instance, Bernardo et al. bernardo2023  highlight CI
influencing higher confidence during code review.

In addition, future studies may analyze whether CI is the cause or the
consequence for attracting or attaining more experienced (core) developers. If
a clear causal flow exists between $ExperiencedDevelopers\to CI$ or $CI\to
ExperiencedDevelopers$, other potential investigations would open, such as
investigating whether CI generates more or less bug reports because, for
instance, experienced developers detect more bugs.

The data-validated causal DAG (Fig. 19) shows $Age$ as a significant source of
confounding (constituting a backdoor path) for $ContinuousIntegration$ and
$BugReport$. Thus, researchers should be more attentive to controlling for
$Age$ when studying the effects of CI. Additionally, researchers may
investigate the rationale behind this relationship between older projects and
CI projects. They may explore, for example, the decision process for CI
adoption and the challenges involved in such a process that could postpone the
adoption.

###  5.2 Practical implications

Our study and the data-validated DAG (Fig. 19) raise practical implications.
Software developers may understand that they potentially decrease the bug
reports when employing CI in their projects. CI causes an effect on the
$BugReport$, encourages team $Communication$, and increases the
$CommitFrequency$ (in turn, both positively affect bug reports). Herefore,
considering the observed literature-reported relationships and the variables
we were able to measure in our study, CI has a causal effect on the number of
Bug Reports, which, in turn, is our proxy for software quality.

On the other hand, CI projects have greater values for the $Age$ variable.
Adopting CI may be related to greater project maturity. This demand may be
related to the difficulties of implementing CI and a demand for more qualified
and experienced professionals. Awareness of (over)confidence in a flaky
automated environment is also essential. Investing in building a reliable
environment could mitigate the negative overconfidence effect.

##  6 Threats To Validity

In this section, we discuss the threats to the validity of our study.

Construct Validity.  There are construct threats associated with the searching
for associations between CI and software quality from existing software
engineering literature. These associations are used to build the DAG showing
how CI may influence software quality with potential confounding variables
(RQ1). To reduce the threats associated with the search for such associations,
we have mainly used existing systematic literature reviews and empirical
studies to search for research work associated with CI and software quality.
However, there are always associations (open questions) that are not captured
by existing studies or even studies that were not found and considered in our
analysis. In addition to that, we have also considered the associations
related to CI sub-practices that are present in the selected papers as a
direct relationship between CI and other important software development
aspects.

We also have construct threats associated with the repository mining of the
collected data that supports the statistical tests of the associations in the
DAG (RQ2). We used GitHub and Travis CI APIs to collect data about the
selected projects and compute the metrics. Any bias in how we compute such
metrics can affect our results. For example, we can measure communication in
other ways, such as calls, meetings, emails, and others. As another example,
merge conflicts and bug issues were cataloged based on heuristics that could
carry biases preventing our scripts from precisely identifying all issues and
conflicts. However, we used our best efforts to collect the seven considered
metrics in this study. We seek to follow methods and metrics already known in
the software engineering community, as we show throughout Section 3.

Internal Validity.  To compute the metrics associated with our dataset, we
aggregate them into months. Such a strategy can produce a lack of information
(compared to a week or day-based analysis) associated with the metrics of
interest. Recent empirical studies sizilio2019empirical ; bernardo2018studying
that investigate the impact of CI have adopted a similar month-based strategy.

External Validity.  External threats concern the extent to which we can
generalize our results. Our work analyzed a total of 59,034 commits, 16,619
builds, and 3,199 bug reports of 148 Github open-source projects - 74 CI and
74 no-CI projects. Nevertheless, we collected a consistent dataset
representative of the open-source community, which validates our DAG. However,
other studies would be needed to reinforce, refute, or modify our causal DAG.
The more projects and metrics reliable and consistently collected, the more
robust would be the causal DAG and the causal theory proposed. This study is
not intended to be definitive concerning the causal relationship between CI
and software quality but rather to be a starting point in such a direction.
More studies are needed to evolve our understanding of causal relationships
between CI and quality.

##  7 Conclusion

Continuous Integration (CI) has increased in popularity in the last decades
with the expectation of providing several software development benefits. One
of these benefits associated with CI is increased software quality. This study
expands the current software engineering body of knowledge by investigating
the association between CI and software quality from a causal perspective. We
reviewed the literature cataloging relevant CI and software quality
associations and adjacent relationships. Then we built the literature-based
causal DAG (RQ1) expressing these associations. We mined software repositories
to empirically assess the literature-based causal DAG (RQ2), analyzing
releases in 12 activity months from 148 software projects.

Analyzing the testable implications from the literature-based causal DAG
generated from literature assumptions, we found in RQ2 some associations that
do not exist in the collected dataset. Then, relying on a hybrid literature-
data approach, we proposed a new data-validated causal DAG (RQ3), expressing
the relationship between the relevant variables.

We discussed research opportunities from the differences cataloged between
literature-based DAG and the data-validated causal DAG. Researchers and
practitioners can benefit from our findings based on the proposed data-
validated causal DAG that, in summary, shows a positive causal effect from CI
on bug reports. CI also influences developers’ communication, reinforcing bug
reports benefits. The developers’ overconfidence is still a concern in CI
environments and could negatively affect bug reports.

###### Acknowledgements.

This work is partially supported by INES (www.ines.org.br), CNPq grants
465614/2014-0 and 425211/2018-5, CAPES grant 88887.136410/2017- 00, and FACEPE
grants APQ-0399-1.03/17 and PRONEX APQ/0388-1.03/14.

Data Availability All data is available via an online appendix:
https://github.com/elieziosoares/ci_quality_study_replication.

## Declarations

Conflict of interests The authors declare that they have no conflict of
interest.

## References

  * (1) A. Cairo, G. Carneiro, and M. Monteiro, “The impact of code smells on software bugs: A systematic literature review”, Information, vol. 9, no. 11, p. 273, 2018.
  * (2) A. Debbiche, M. Dienér, and R. Berntsson Svensson, “Challenges when adopting continuous integration: A case study,” Product-Focused Software Process Improvement, pp. 17–32, 2014.
  * (3) A. Hindle, D. M. German, and R. Holt, “What do large commits tell us?,” Proceedings of the 2008 international workshop on Mining software repositories - MSR ’08, 2008.
  * (4) A. Murgia, G. Concas, R. Tonelli, M. Ortu, S. Demeyer, and M. Marchesi, “On the influence of maintenance activity types on the issue resolution time,” Proceedings of the 10th International Conference on Predictive Models in Software Engineering, 2014.
  * (5) A. Rahman, A. Agrawal, R. Krishna, and A. Sobran, “Characterizing the influence of continuous integration: Empirical results from 250+ open source and proprietary projects,” Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics, 2018.
  * (6) A. Zaidman, B. Van Rompaey, S. Demeyer, and A. van Deursen, “Mining software repositories to study co-evolution of Production & Test code,” 2008 1st International Conference on Software Testing, Verification, and Validation, 2008.
  * (7) A. Zaidman, B. Van Rompaey, A. van Deursen, and S. Demeyer, “Studying the co-evolution of production and test code in open source and industrial developer test processes through Repository Mining,” Empirical Software Engineering, vol. 16, no. 3, pp. 325–364, 2010.
  * (8) C. Amrit and Y. Meijberg, “Effectiveness of test-driven development and continuous integration: A case study,” IT Professional, vol. 20, no. 1, pp. 27–35, 2018.
  * (9) C. Heinze-Deml, J. Peters, and N. Meinshausen, “Invariant causal prediction for nonlinear models,” arXiv.org, 19-Sep-2018. [Online]. Available: https://arxiv.org/abs/1706.08576v2. [Accessed: 06-Apr-2022].
  * (10) C. Vassallo, S. Proksch, T. Zemp, and H. C. Gall, “Every build you break: Developer-oriented assistance for build failure resolution,” Empirical Software Engineering, vol. 25, no. 3, pp. 2218–2257, 2019.
  * (11) “CI Theatre: Technology Radar,” Thoughtworks. [Online]. Available: https://www.thoughtworks.com/radar/techniques/ci-theatre. [Accessed: 22-Mar-2022].
  * (12) “Continuous integration,” martinfowler.com. [Online]. Available: https://martinfowler.com/articles/continuousIntegration.html. [Accessed: 22-Mar-2022].
  * (13) D. Bijlsma, M. A. Ferreira, B. Luijten, and J. Visser, “Faster issue resolution with higher technical quality of software,” Software Quality Journal, vol. 20, no. 2, pp. 265–285, 2011.
  * (14) D. Ståhl and J. Bosch, “Modeling continuous integration practice differences in industry software development,” Journal of Systems and Software, vol. 87, pp. 48–59, 2014.
  * (15) Dudekula Mohammad Rafi, Katam Reddy Kiran Moses, K. Petersen, and M. V. Mantyla, “Benefits and limitations of Automated Software Testing: Systematic Literature Review and practitioner survey,” 2012 7th International Workshop on Automation of Software Test (AST), 2012.
  * (16) E. Soares, G. Sizilio, J. Santos, D. Alencar da Costa, U. Kulesza, “The effects of continuous integration on software development: a systematic literature review”, Empir Software Eng 27, 78 (2022). https://doi.org/10.1007/s10664-021-10114-1
  * (17) E. Laukkanen, J. Itkonen, and C. Lassenius, “Problems, causes and solutions when adopting continuous delivery—a systematic literature review,” Information and Software Technology, vol. 82, pp. 55–79, 2017.
  * (18) F. Huang, B. Liu, and B. Huang, “A taxonomy system to identify human error causes for software defects”. In The 18th international conference on reliability and quality in design, p. 44, 2012.
  * (19) F. Zhang, F. Khomh, Y. Zou, and A. E. Hassan, “An empirical study on factors impacting bug fixing time,” 2012 19th Working Conference on Reverse Engineering, 2012.
  * (20) Gefen, Karahanna, and Straub, “Trust and tam in online shopping: An integrated model,” MIS Quarterly, vol. 27, no. 1, p. 51, 2003.
  * (21) G. Pinto, M. Reboucas, and F. Castor, “Inadequate testing, time pressure, and (over) confidence: A tale of continuous integration users,” 2017 IEEE/ACM 10th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), 2017.
  * (22) G. Pinto, F. Castor, R. Bonifacio, and M. Rebouças, “Work practices and challenges in continuous integration: A survey with Travis CI Users,” Software: Practice and Experience, vol. 48, no. 12, pp. 2223–2236, 2018.
  * (23) G. Shmueli, “To explain or to predict?,” SSRN Electronic Journal, 2010.
  * (24) G. Sizilio Nery, D. Alencar da Costa, and U. Kulesza, “An empirical study of the relationship between continuous integration and test code evolution,” 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019.
  * (25) H. Reichenbach, The direction of Time. Berkeley: University of California Press, 1974.
  * (26) H. Seo, C. Sadowski, S. Elbaum, E. Aftandilian, and R. Bowdidge, “Programmers’ build errors: A case study (at Google),” Proceedings of the 36th International Conference on Software Engineering, 2014.
  * (27) I. Keskin Kaynak, E. Çilden, and S. Aydin, “Software quality improvement practices in continuous integration,” Communications in Computer and Information Science, pp. 507–517, 2019.
  * (28) J. H. Bernardo, D. A. da Costa, and U. Kulesza, “Studying the impact of adopting continuous integration on the delivery time of pull requests,” Proceedings of the 15th International Conference on Mining Software Repositories, 2018.
  * (29) J. Pearl and T. S. Verma, “A theory of inferred causation,” Logic, Methodology and Philosophy of Science IX, Proceedings of the Ninth International Congress of Logic, Methodology and Philosophy of Science, pp. 789–811, 1995.
  * (30) J. Pearl, “Causal diagrams for empirical research,” Biometrika, vol. 82, no. 4, pp. 702–710, 1995.
  * (31) J. Pearl, Causality: Models, reasoning, and inference. Cambridge: Cambridge University Press, 2005.
  * (32) J. Peters, D. Janzing, and Schoolkopf Bernhard, Elements of causal inference: Foundations and learning algorithms. Cambridge, MA: The MIT Press, 2017.
  * (33) J. Santos, D. Alencar da Costa, and U. Kulesza. 2022. Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects. In Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’22). Association for Computing Machinery, New York, NY, USA, 137–147. https://doi.org/10.1145/3544902.3546244
  * (34) J. Textor, B. van der Zander, M. S. Gilthorpe, M. Liśkiewicz, and G. T. H. Ellison, “Robust causal inference using directed acyclic graphs: The R package ‘dagitty,”’ International Journal of Epidemiology, 2017.
  * (35) K. Beck, Extreme programming explained: Embrace change. Boston: Addison-Wesley, 2010.
  * (36) L. D. Panjer, “Predicting eclipse bug lifetimes,” Fourth International Workshop on Mining Software Repositories (MSR’07:ICSE Workshops 2007), 2007.
  * (37) M. A. Hernan and J. M. Robins, Causal inference. Boca Raton, FL: CRC, 2010.
  * (38) M. C. de Oliveira, “Draco: Discovering refactorings that improve architecture using fine-grained co-change dependencies,” Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017.
  * (39) M. C. de Oliveira, D. Freitas, R. Bonifácio, G. Pinto, and D. Lo, “Finding needles in a haystack: Leveraging co-change dependencies to recommend Refactorings,” Journal of Systems and Software, vol. 158, p. 110420, 2019.
  * (40) M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, costs, and benefits of continuous integration in open-source projects,” Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, 2016.
  * (41) M. Hilton, J. Bell, and D. Marinov, “A large-scale study of test coverage evolution,” Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, 2018.
  * (42) Mockus and Votta, “Identifying reasons for software changes using historic databases,” Proceedings International Conference on Software Maintenance ICSM-94, 2000.
  * (43) M. R. Islam and M. F. Zibran, “Insights into continuous integration build failures,” 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), 2017.
  * (44) M. Reboucas, R. O. Santos, G. Pinto, and F. Castor, “How does contributors’ involvement influence the build status of an open-source software project?,” 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), 2017.
  * (45) N. Cartwright,“Nature’s Capacities and Their Measurement,” pp. 141–182, 1994.
  * (46) N. Cassee, B. Vasilescu, and A. Serebrenik, “The silent helper: The impact of continuous integration on code reviews,” 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2020.
  * (47) N. Kerzazi, F. Khomh, and B. Adams, “Why do automated builds break? an empirical study,” 2014 IEEE International Conference on Software Maintenance and Evolution, 2014.
  * (48) P. M. Duvall, S. Matyas, and A. Glover, Continuous integration improving software quality and reducing risk. Upper Saddle River, NJ: Addison-Wesley, 2013.
  * (49) P. Spirtes, C. N. Glymour, and R. Scheines, Causation, prediction, and Search. Cambridge, MA: MIT Press, 2000.
  * (50) S. A. Licorish and S. G. MacDonell, “Exploring software developers’ work practices: Task differences, participation, engagement, and speed of Task Resolution,” Information & Management, vol. 54, no. 3, pp. 364–382, 2017.
  * (51) S. Greenland, J. Pearl, and J. M. Robins, “Confounding and collapsibility in causal inference,” Statistical Science, vol. 14, no. 1, 1999.
  * (52) S. Greenland, J. Pearl, and J. M. Robins, “Causal diagrams for epidemiologic research,” Epidemiology, vol. 10, no. 1, pp. 37–48, 1999.
  * (53) T. Rausch, W. Hummer, P. Leitner, and S. Schulte, “An empirical analysis of build failures in the continuous integration workflows of Java-based open-source software,” 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), 2017.
  * (54) T. Zhang, H. Jiang, X. Luo, and A. T. S. Chan, “A literature review of research in bug resolution: Tasks, challenges and future directions,” The Computer Journal, vol. 59, no. 5, pp. 741–773, 2015.
  * (55) Vale, G., Schmid, A., Santos, A.R. et al. On the relation between Github communication activity and merge conflicts. Empir Software Eng 25, 402–433 (2020). https://doi.org/10.1007/s10664-019-09774-x
  * (56) Vasilescu B, Yu Y, Wang H, et al (2015) Quality and productivity outcomes relating to continuous integration in GitHub. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2015. doi: 10.1145/2786805.2786850
  * (57) V. Garousi and M. V. Mäntylä, “When and what to automate in software testing? A multi-vocal literature review,” Information and Software Technology, vol. 76, pp. 92–117, 2016.
  * (58) W. Felidre, L. Furtado, D. A. Costa, B. Cartaxo, and G. Pinto, “Continuous Integration Theater,” 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2019.
  * (59) Y. Zhao, A. Serebrenik, Y. Zhou, V. Filkov, and B. Vasilescu, “The impact of continuous integration on other software development practices: A large-scale empirical study,” 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), 2017.
  * (60) Widder DG, Hilton M, Kästner C, Vasilescu B (2018) I’m leaving you, Travis. Proceedings of the 15th International Conference on Mining Software Repositories. doi: 10.1145/3196398.3196422
  * (61) Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating github for engineered software projects. Empirical Software Engineering 22:3219–3253. doi: 10.1007/s10664-017-9512-6
  * (62) Shalizi Cosma (2021) Advanced Data Analysis from an Elementary Point of View [Online]. Available: https://www.stat.cmu.edu/ cshalizi/ADAfaEPoV/ [Accessed: 01-Aug-2023].
  * (63) Brindescu C, Codoban M, Shmarkatiuk S, Dig D (2014) How do centralized and Distributed Version Control Systems Impact Software Changes? Proceedings of the 36th International Conference on Software Engineering. doi: 10.1145/2568225.2568322
  * (64) Saraiva D, Da Costa DA, Kulesza U, et al (2023) Unveiling the relationship between continuous integration and code coverage. 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). doi: 10.1109/msr59073.2023.00043
  * (65) Eyolfson J, Tan L, Lam P (2013) Correlations between bugginess and time-based commit characteristics. Empirical Software Engineering 19:1009–1039. doi: 10.1007/s10664-013-9245-0
  * (66) Decan A, Mens T, Mazrae PR, Golzadeh M (2022) On the use of github actions in software development repositories. 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). doi: 10.1109/icsme55016.2022.00029
  * (67) Golzadeh M, Decan A, Mens T (2022) On the rise and fall of CI services in github. 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). doi: 10.1109/saner53432.2022.00084
  * (68) Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? an empirical case study of Mozilla Firefox. 2012 9th IEEE Working Conference on Mining Software Repositories (MSR). doi: 10.1109/msr.2012.6224279
  * (69) Rzig DE, Hassan F, Bansal C, Nagappan N (2022) Characterizing the usage of CI tools in ML Projects. ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). doi: 10.1145/3544902.3546237
  * (70) Bernardo JH, da Costa DA, Kulesza U, Treude C (2023) The impact of a continuous integration service on the delivery time of merged pull requests. Empirical Software Engineering. doi: 10.1007/s10664-023-10327-6