11institutetext: Department of Applied Mathematics, Technical Medical Centre,
University of Twente, Enschede, The Netherlands
11email<EMAIL_ADDRESS>22institutetext: Department of Surgery, Amsterdam
UMC location Vrije Universiteit Amsterdam,
Amsterdam, The Netherlands 33institutetext: Amsterdam Cardiovascular Sciences,
Microcirculation,
Amsterdam, The Netherlands

# Going Off-Grid: Continuous Implicit Neural Representations for 3D Vascular
Modeling

Dieuwertje Alblas (✉) 11 Christoph Brune 11
Kak Khee Yeung 22 3 3 Jelmer M. Wolterink 11

###### Abstract

Personalised 3D vascular models are valuable for diagnosis, prognosis and
treatment planning in patients with cardiovascular disease. Traditionally,
such models have been constructed with explicit representations such as meshes
and voxel masks, or implicit representations such as radial basis functions or
atomic (tubular) shapes. Here, we propose to represent surfaces by the zero
level set of their signed distance function (SDF) in a differentiable implicit
neural representation (INR). This allows us to model complex vascular
structures with a representation that is implicit, continuous, light-weight,
and easy to integrate with deep learning algorithms. We here demonstrate the
potential of this approach with three practical examples. First, we obtain an
accurate and watertight surface for an abdominal aortic aneurysm (AAA) from CT
images and show robust fitting from as little as 200 points on the surface.
Second, we simultaneously fit nested vessel walls in a single INR without
intersections. Third, we show how 3D models of individual arteries can be
smoothly blended into a single watertight surface. Our results show that INRs
are a flexible representation with potential for minimally interactive
annotation and manipulation of complex vascular structures.

###### Keywords:

Implicit neural representation Vascular model Abdominal aortic aneurysm Signed
distance function Level set

##  1 Introduction

Accurate and patient-specific models of vascular systems are valuable for
diagnosis, prognosis and treatment planning in patients with cardiovascular
disease. Personalised vascular models might be used for stent-graft sizing in
patients with abdominal aortic aneurysms [27] or for computational fluid
dynamics (CFD) [29]. However, extracting these models from medical image data
can be cumbersome. Commercial software and open-source software packages [2,
15, 3] traditionally rely on the construction of tubular models [24] in three
steps. First, the lumen centerline is identified for each vessel. Then, local
cross-sectional contours are determined and used to construct a watertight
mesh model using (spline) interpolation. Finally, polygon mesh models of
multiple vessels are blended to obtain a connected vascular tree. In this
approach, tortuosity of the centerline can cause self-intersections of the
orthogonal contours, resulting in surface folding of the final mesh model [9].
Moreover, smoothly connecting triangular meshes around bifurcations is
challenging.

Deep learning has made great progress towards automatic model building from
images [6, 16]. However, popular convolutional neural network-based methods
return 3D voxel masks. Because voxel masks merely discretize an underlying
continuous shape, their quality heavily depends on the resolution of the image
data, and they are not guaranteed to be contiguous. Hence, voxel masks
typically require additional processing steps before use in, e.g., CFD. There
is a need for a vascular model shape representation that is continuous,
modular, and can be easily integrated with existing deep learning methods.

(a) Voxel mask

(b) Mesh

(c) Signed distance function

Figure 1: Different representations of a 3D aortofemoral tree [30]. 1(a) and
1(b) are explicit representations; 1(a) has non-smooth boundaries, whereas
boundaries of 1(b) are locally smooth. Both 1(a) and 1(b) are restricted to
this resolution. 1(c) implicitly represents the surface with smooth
boundaries, at any resolution.

In this work, we adapt the work of Gropp et al. [10] to model vascular systems
as combinations of level sets of signed distance functions represented in
differentiable neural networks. Implicit representations and level sets have a
substantial history in both segmentation [4, 17] and 3D modeling [11, 14] of
vascular structures. Recently, there have been significant advances in signal
representations using neural networks, i.e., implicit neural representations
(INRs)[19, 26, 22, 18, 1]. Continuous implicit neural representations of the
signed distance function can be easily transformed into an explicit
representation, while the inverse is not true (Fig. 1).

We here demonstrate that INRs are a potentially valuable tool to bridge the
gap between vascular modeling and deep learning. First, we show how INRs can
be used to reconstruct an implicit surface from a sparse point cloud and form
an alternative to conventional annotation procedures. We evaluate the
efficiency and robustness of this approach. Secondly, we show that we can use
a single INR to represent multiple surfaces, and demonstrate the effectiveness
on nested shapes in an AAA case study. Finally, we demonstrate the added value
of implicit shape representations in the smooth blending of separate
structures in the reconstruction of an aortofemoral tree.

##  2 Method

We represent the surface of a vascular structure by the zero level set of its
signed distance function (SDF), which we embed in a multilayer perceptron
(MLP) as an implicit neural representation.

###  2.1 Signed distance functions

A surface can be implicitly described by the zero level set of its signed
distance function (SDF). In the case of a 2D surface $\mathcal{M}$ embedded in
a 3D domain, $SDF_{\mathcal{M}}(\bm{x}):\mathbb{R}^{3}\to\mathbb{R}$ is
defined as:

$\displaystyle
SDF_{\mathcal{M}}(\bm{x})=\begin{cases}-d(\bm{x},\mathcal{M})\hskip
14.22636pt&\bm{x}\text{ inside }\mathcal{M}\\\ 0&\bm{x}\text{ on
}\mathcal{M}\\\ d(\bm{x},\mathcal{M})&\bm{x}\text{ outside
}\mathcal{M},\end{cases}$ (1) $\displaystyle\text{therefore:
}\mathcal{M}=\\{\bm{x}\in\mathbb{R}^{3}|SDF_{\mathcal{M}}(\bm{x})=0\\}.$ (2)

In Eq. (1)
$d(\bm{x},\mathcal{M})=\min\limits_{\bm{y}\in\mathcal{M}}||\bm{x}-\bm{y}||$,
the minimal distance to the surface. An SDF satisfies the Eikonal equation,
hence $||\nabla_{x}SDF_{\mathcal{M}}(\bm{x})||=1,\forall\bm{x}$. As this
function is continuous on $\mathbb{R}^{3}$, the surface $\mathcal{M}$ is
represented independently of resolution.

Figure 2: Schematic overview of an implicit neural representation,
parametrising a single SDF (left), or three SDFs simultaneously (right).
Coordinate values can be queried and the network returns local SDF values.

###  2.2 Implicit neural representations

Signed distance functions are often represented as a combination of SDFs of
atomic shapes, or radial basis functions [5]. However, a recent insight shows
that functions in space can be approximated using neural networks [22, 26].
These so-called implicit neural representations (INRs) have been widely used
for, e.g., image representations [7], novel view synthesis [19], image
registration [32], and sparse-view CT reconstruction [28]. In this work, we
use INRs to approximate the signed distance functions of one or multiple
surfaces. This INR is a fully connected multilayer perceptron (MLP),
$f(\bm{x};\theta)$, that takes
$\bm{x}\in\Omega:=\left[-1,1\right]^{3}\subset\mathbb{R}^{3}$ as input, and
outputs $SDF_{\mathcal{M}}(\bm{x})$, as shown in Figure 2(left). During
optimization of the INR, desired properties of the SDF can be directly imposed
on the network via the loss function. These properties include conditions on
gradients and higher-order derivatives, as they can be computed for the
network through backpropagation [26]. An INR thus represents a function or
signal, and not a mapping between two functions, as is often the case with
convolutional neural networks. As an INR is trained on continuous coordinates,
they allow for representing a function at any resolution.

###  2.3 Optimizing an INR

We use an INR to represent the SDF of a 3D vascular model, under the condition
that the model should be easily obtainable from a 3D medical image volume. We
start from a small set of points on the vessel surface:
$\mathcal{X}=\\{\bm{x}_{i}\\}_{i=1,...,N}\subset\Omega$. We aim to
simultaneously reconstruct an SDF with $\mathcal{X}$ on the zero levelset and
embed it in an INR. This amounts to solving the Eikonal equation with
pointwise boundary conditions; an ill-posed problem. Gropp et al. [10] tackle
this ill-posedness by minimising the following loss function:

$\displaystyle\ell(\theta)=\frac{1}{N}\sum\limits_{1\leq i\leq
N}\left(|f(\bm{x}_{i};\theta)|\right)+\lambda\mathbb{E}_{x}\left(||\nabla_{\bm{x}}f(\bm{x};\theta)||-1\right)^{2}.$
(3)

The first term enforces that all points from $\mathcal{X}$ lie on the zero
level set of the SDF, and thus on the surface $\mathcal{M}$. The second term
is a regularising term, encouraging that the INR $f(\bm{x};\theta)$ satisfies
the Eikonal equation, just like the SDF it represents. The key observation of
Gropp et al. is that minimising Eq. (3) yields an SDF with a smooth and
plausible zero isosurface. This approach allows for flexibility in the
consistency of $\mathcal{X}$, in contrast to similar methods [20, 23], that
require local normals or points to lie within the same plane. In Section 3.1,
we investigate the quality and robustness of the reconstruction under
variations of the point cloud.

###  2.4 Fitting multiple functions in an INR

The wall of blood vessels consists of layers, and in patients with abdominal
aortic aneurysms (AAAs) the lumen and the surrounding solidified thrombus
structure should be modeled separately [34]. While INRs have mostly been used
to represent single functions, they can be extended to fit multiple functions
on the same domain. Figure 2(right) visualizes how by adding additional output
nodes to the neural network, multiple SDFs can be fitted simultaneously, in
this case for the lumen, the inner wall, and outer wall of the AAA. This means
that mutual properties of separate SDFs, such as for nested shapes, could be
learned by a single neural network. This INR is optimised by minimising Eq.
(3) on each of its output channels. We will compare this approach to multiple
INRs representing structures separately in Section 3.2.

###  2.5 Constructive solid geometry

SDFs make it very easy to determine the union, difference, or intersection of
multiple shapes, a cumbersome and challenging task in explicit polygon mesh
representations [13, 33]. Let $SDF_{\mathcal{M}_{1}}(\bm{x})$ and
$SDF_{\mathcal{M}_{2}}(\bm{x})$ be SDFs for shapes $\mathcal{M}_{1}$ and
$\mathcal{M}_{2}$ in $\Omega$, respectively. Then their union is determined
as:

$\displaystyle\mathcal{M}_{1}\cup\mathcal{M}_{2}=\min(SDF_{\mathcal{M}_{1}}(\bm{x}),SDF_{\mathcal{M}_{2}}(\bm{x}))$
(4)

To allow for smoother blending between interfaces of shapes, a smoothed min
function can be considered [12]:

$\displaystyle\begin{split}&\left(\mathcal{M}_{1}\cup\mathcal{M}_{2}\right)_{\text{smooth}}=\min(SDF_{\mathcal{M}_{1}}(\bm{x}),SDF_{\mathcal{M}_{2}}(\bm{x}))-\gamma_{\text{smooth}},\\\
&\gamma_{\text{smooth}}=0.25\cdot
k\cdot\max(k-|SDF_{\mathcal{M}_{1}}(\bm{x})-SDF_{\mathcal{M}_{2}}(\bm{x})|,0)^{2},\end{split}$
(5)

where $k$ represents a smoothing parameter, i.e. a set region around the
interfaces of the two surfaces. We will use this strategy to join structures
represented by INRs in Section 3.3.

Figure 3: The Dice similarity coefficient (DSC) and average surface distance
(ASD) of reconstructed vascular surface for varying number of reference
points. Each SDF is reconstructed using 20 different point clouds of each
size.

##  3 Experiments and Results

We conducted three experiments: (1) assessing the robustness of the INR
against variations in the point cloud, (2) testing the effectiveness of
representing multiple shapes with a single INR and (3) demonstrating how our
method can be used to reconstruct a vascular system from separate vessels and
blend them naturally. For all experiments we used MLPs consisting of an input
layer with three nodes, six fully connected hidden layers with 256 nodes and
ReLU activations, and a final prediction layer consisting of one or three
nodes, embedding a single shape or three nested shapes respectively. Similar
to [10], we used a skip connection, connecting the input to the third hidden
layer. We base our implementation on PyTorch
code111https://github.com/amosgropp/IGR provided with [10].

Figure 4: Three nested AAA structures represented by a single INR, or separate
INRs each representing a single shape. Using a single INR preserves the nested
property, violations of this anatomical prior are marked with circles.

###  3.1 Robustness

We show how INRs can form an alternative to time-consuming annotation
procedures, by fitting an SDF to a manually annotated point cloud. We assess
the robustness against variations in this point cloud. We use a public dataset
that contains 3D AAA meshes [31]. For each AAA, the dataset contains separate
mesh representations for the lumen, the inner and outer vessel wall. For this
experiment, we solely used the meshes of the lumen; we randomly sampled a set
number of points on the mesh and repeated this sampling process 20 times,
resulting in 20 different point clouds of the same size. For optimizing the
INR, we used an Adam optimizer for 25,000 epochs, with a learning rate of
0.0001, and set $\lambda$ in the loss function (Eq. (3)) to 0.1, as suggested
in [10]. Optimisation was done on an NVIDIA Quadro RTX 6000 GPU, taking 10-15
minutes per network.

Figure 3 shows the quality of the surface lumen reconstruction in this model,
in terms of the Dice similarity coefficient (DSC) and average surface distance
(ASD). The DSC was computed using the original mesh as a reference, the ASD
was computed using unseen points on the surface, i.e. points that were not in
$\mathcal{X}$ during optimisation. The median DSC is above 0.95 in all cases,
even when only 100 points are used. We observe more variation in both metrics
when the reconstruction is based on fewer points. Robustness and performance
stabilize after 400 points, implying that using more points will not further
improve the quality or robustness of the reconstruction.

###  3.2 Reconstructing nested shapes

In our second experiment, we again use data provided in [31], with mesh
representations for the lumen, inner wall, and outer wall of the aorta. Here,
we aim to capture the nested property of the three shapes in the neural
network representing their respective SDFs, meaning: $SDF_{\text{outer
wall}}(\bm{x})\leq SDF_{\text{inner wall}}(\bm{x})\leq
SDF_{\text{lumen}}(\bm{x}),\forall\bm{x}\in\Omega$. For each selected AAA
case, we sampled a point cloud of 200 points for each of the three surfaces; a
proper trade-off between point cloud sparsity and reconstruction quality. We
fit a joint INR to these point clouds, as well as three separate INRs, one for
each of the three surfaces. Figure 4 shows the resulting surfaces for both
approaches. In all three cases, we see unwanted extrusions of the inner
structures when separate INRs are used. Case 1 shows this effect best; the
surfaces of the lumen and inner wall (red and green structures respectively)
fail to capture the bifurcation and are inconsistent with the outer wall. In
contrast, representation of the shapes using a single INR consistently
resulted in none of these extrusions. Moreover, the thickness of the vessel
wall remains constant for this approach, in agreement with ground-truth data.

###  3.3 Constructive geometry

In our third experiment, we demonstrate how INRs enable easy and smooth
surface blending. This is critical for personalised vascular modeling, where
many arteries of different calibers need to be blended into one single model
for, e.g., CFD. Here, we use an aortofemoral tree from [30]. This sample
consists of nine separate mesh structures: aorta, superior and inferior
mesenteric arteries (SMA/IMA), renal arteries, hepatic artery, and common and
internal iliac arteries, shown in Figure 5(left). For each of these mesh
structures, we sample a point cloud of a size proportional to its surface
area, to which we fit an INR, after normalizing it to the
$\left[-0.9,0.9\right]^{3}$ domain to allow for smooth blending. We optimize
these networks using an Adam optimizer for 50,000 epochs.

Figure 5: Subdomains of separate INRs, indicated by different colours (left)
and smooth blending of these INRs, with an inside-view at the top (right).

To blend the vessels, we generate a Cartesian grid encompassing the entire
vascular tree. Hence, this domain contains the respective subdomains that all
nine INRs were trained on. To evaluate the INRs for each structure on this
grid, we map each grid point to the corresponding coordinate in each INR
domain and query the SDF value by forwarding it through the network. Next, we
rescale the SDF value from the INR domain to the real size. This results in a
grid containing patches of SDFs, that we blend together using Eq.
(LABEL:eq:smoothblend), with smoothing parameter $k=0.1$. The result in Figure
5(right) shows a smooth transition between the different vascular structures,
without any intrusions on the inside, making this structure suitable for CFD
analysis[14].

##  4 Discussion and Conclusion

In this work, we have shown the value of INRs for 3D vascular modeling. We
have demonstrated that INRs can be used to fit surfaces in abdominal aortic
aneurysms using a small number of annotated points on the surface. Moreover,
results show that fitting multiple nested surfaces in one INR is less likely
to lead to violation of anatomical priors. Finally, we have used simple
arithmetics to blend multiple surfaces represented in INRs into one complex
vascular model.

Following the work of [10], we have shown that we can reconstruct continuous
surfaces based on (small) point clouds, where their implicit prior leads to
surfaces that are smooth and interpolate well in space. This contrasts with,
e.g., slice interpolation as is often used to obtain dense voxel masks. INRs
are continuous, allowing us to represent the vessel surface at any arbitrary
resolution. This could facilitate seamless blending of separate arteries for,
e.g., CFD analysis [14].

We have demonstrated how a single INR can simultaneously parametrise multiple
nested surfaces in an abdominal aortic aneurysm. We showed that using a single
INR to represent nested shapes benefits the consistency of the surfaces. The
SDF values of nested shapes are similar, as they differ by a local offset.
Using a single network to reconstruct them enables the network to learn these
offsets by sharing weights and information from the other shapes during
training. This nested property is, however, not a topological guarantee, but
can be enforced by additional regularisation in the loss function.

Explicit representations such as polygonal meshes [2, 15, 3] and voxel masks
are ubiquitous in vascular modeling. INRs could be integrated as an
underlying, general, representation of shapes. While they can be easily
transformed into meshes or voxel masks at any desirable resolution, the
inverse transform is challenging (Fig. 1). For example, while shape arithmetic
on mesh representations is cumbersome [13], we have shown that it is
straightforward with INRs.

One limitation of INRs is their lack of generalisation; a new INR needs to be
optimised for each shape. This may be overcome by enriching the network input
with a latent code, in which case INRs could represent a whole distribution of
shapes [22]. Alternatively, INRs could be embedded in existing deep learning
methods for (cardiovascular) image segmentation, e.g., by including local
image features [7] or training hypernetworks [25]. Finally, our work could
lead to interactive annotation tools. The combination of recently proposed
rapid INR fitting approaches [21] and uncertainty quantification [8] could
guide the annotator to areas that need additional annotations.

In conclusion, INRs are a versatile surface representation that are easily
acquired and integrated in existing frameworks for vascular modeling.

## References

  * [1] Amiranashvili, T., Lüdke, D., Li, H., Menze, B., Zachow, S.: Learning Shape Reconstruction from Sparse Measurements with Neural Implicit Functions. In: MIDL (2022)
  * [2] Antiga, L., Piccinelli, M., Botti, L., Ene-Iordache, B., Remuzzi, A., Steinman, D.A.: An image-based modeling framework for patient-specific computational hemodynamics. MBEC 46(11), 1097–1112 (2008)
  * [3] Arthurs, C.J., Khlebnikov, R., Melville, A., Marčan, M., Gomez, A., Dillon-Murphy, D., Cuomo, F., Silva Vieira, M., Schollenberger, J., Lynch, S.R., et al.: CRIMSON: An open-source software framework for cardiovascular integrated modelling and simulation. PLoS Comp. Biol. 17(5), e1008881 (2021)
  * [4] van Bemmel, C.M., Spreeuwers, L.J., Viergever, M.A., Niessen, W.J.: Level-set based carotid artery segmentation for stenosis grading. In: International Conference on MICCAI. pp. 36–43. Springer (2002)
  * [5] Carr, J.C., Fright, W.R., Beatson, R.K.: Surface interpolation with radial basis functions for medical imaging. IEEE Trans Med Imaging 16(1), 96–107 (1997)
  * [6] Chen, C., Qin, C., Qiu, H., Tarroni, G., Duan, J., Bai, W., Rueckert, D.: Deep learning for cardiac image segmentation: a review. Front. cardiovasc. med. 7, 25 (2020)
  * [7] Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. In: CVPR. pp. 8628–8638. IEEE/CVF (2021)
  * [8] Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: ICML. pp. 1050–1059. PMLR (2016)
  * [9] Gansca, I., Bronsvoort, W.F., Coman, G., Tambulea, L.: Self-intersection avoidance and integral properties of generalized cylinders. Comput. Aided Geom. Des. 19(9), 695–707 (2002)
  * [10] Gropp, A., Yariv, L., Haim, N., Atzmon, M., Lipman, Y.: Implicit geometric regularization for learning shapes. In: ICML. pp. 3789–3799 (2020)
  * [11] Hong, Q., Li, Q., Wang, B., Tian, J., Xu, F., Liu, K., Cheng, X.: High-quality vascular modeling and modification with implicit extrusion surfaces for blood flow computations. Comput Methods Programs in Biomed 196, 105598 (2020)
  * [12] Inigo Quilez: Smooth minimum (2013), https://www.iquilezles.org/www/articles/smin/smin.html, accessed on 16/02/2022
  * [13] Jiang, X., Peng, Q., Cheng, X., Dai, N., Cheng, C., Li, D.: Efficient booleans algorithms for triangulated meshes of geometric modeling. Comput. Aided Des. Appl. 13(4), 419–430 (2016)
  * [14] Kretschmer, J., Godenschwager, C., Preim, B., Stamminger, M.: Interactive patient-specific vascular modeling with sweep surfaces. IEEE Trans Vis Comput Graph 19(12), 2828–2837 (2013)
  * [15] Lan, H., Updegrove, A., Wilson, N.M., Maher, G.D., Shadden, S.C., Marsden, A.L.: A re-engineered software interface and workflow for the open-source SimVascular cardiovascular modeling package. J. Biomech. Eng. 140(2) (2018)
  * [16] Litjens, G., Ciompi, F., Wolterink, J.M., de Vos, B.D., Leiner, T., Teuwen, J., Išgum, I.: State-of-the-art deep learning in cardiovascular image analysis. JACC: Cardiovasc. imaging 12(8 Part 1), 1549–1565 (2019)
  * [17] Lorigo, L.M., Faugeras, O.D., Grimson, W.E.L., Keriven, R., Kikinis, R., Nabavi, A., Westin, C.F.: Curves: Curve evolution for vessel segmentation. Med. Image Anal. 5(3), 195–206 (2001)
  * [18] Martel, J.N., Lindell, D.B., Lin, C.Z., Chan, E.R., Monteiro, M., Wetzstein, G.: Acorn: adaptive coordinate networks for neural scene representation. ACM Trans. Graph. 40(4), 1–13 (2021)
  * [19] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV. pp. 405–421. Springer (2020)
  * [20] Mistelbauer, G., Rössl, C., Bäumler, K., Preim, B., Fleischmann, D.: Implicit modeling of patient-specific aortic dissections with elliptic fourier descriptors. In: Comput Graph Forum. vol. 40, pp. 423–434. Wiley Online Library (2021)
  * [21] Müller, T., Evans, A., Schied, C., Keller, A.: Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. arXiv:2201.05989 (2022)
  * [22] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In: CVPR. pp. 165–174. IEEE/CVF (2019)
  * [23] Schumann, C., Oeltze, S., Bade, R., Preim, B., Peitgen, H.O.: Model-free surface visualization of vascular trees. In: EuroVis. pp. 283–290 (2007)
  * [24] Shani, U., Ballard, D.H.: Splines as embeddings for generalized cylinders. Comput. Vis., Graph, and Image Process. 27(2), 129–156 (1984)
  * [25] Sitzmann, V., Chan, E.R., Tucker, R., Snavely, N., Wetzstein, G.: MetaSDF: Meta-Learning Signed Distance Functions. In: NeurIPS (2020)
  * [26] Sitzmann, V., Martel, J.N., Bergman, A.W., Lindell, D.B., Wetzstein, G.: Implicit Neural Representations with Periodic Activation Functions. In: NeurIPS (2020)
  * [27] Sobocinski, J., Chenorhokian, H., Maurel, B., Midulla, M., Hertault, A., Le Roux, M., Azzaoui, R., Haulon, S.: The benefits of EVAR planning using a 3D workstation. Eur J Vasc Endovasc Surg 46(4), 418–423 (2013)
  * [28] Sun, Y., Liu, J., Xie, M., Wohlberg, B., Kamilov, U.S.: Coil: Coordinate-based internal learning for tomographic imaging. IEEE Trans Comput Imaging 7, 1400–1412 (2021)
  * [29] Tran, K., Yang, W., Marsden, A., Lee, J.T.: Patient-specific computational flow modelling for assessing hemodynamic changes following fenestrated endovascular aneurysm repair. JVS: Vascular Science 2, 53–69 (2021)
  * [30] Wilson, N.M., Ortiz, A.K., Johnson, A.B.: The vascular model repository: a public resource of medical imaging data and blood flow simulation results. J. Med. Devices 7(4) (2013)
  * [31] Wittek, A., Mufty, H., Catlin, A., Rogers, C., Saunders, B., Sciarrone, R., Fourneau, I., Meuris, B., Tavner, A., Joldes, G.R., et al.: Image, geometry and finite element mesh datasets for analysis of relationship between abdominal aortic aneurysm symptoms and stress in walls of abdominal aortic aneurysm. Data Br. 30, 105451 (2020)
  * [32] Wolterink, J.M., Zwienenberg, J.C., Brune, C.: Implicit Neural Representations for Deformable Image Registration. In: MIDL. PMLR (2022)
  * [33] Wu, J., Ma, R., Ma, X., Jia, F., Hu, Q.: Curvature-dependent surface visualization of vascular structures. Computerized Medical Imaging and Graphics 34(8), 651–658 (2010)
  * [34] Zhu, C., Leach, J.R., Wang, Y., Gasper, W., Saloner, D., Hope, M.D.: Intraluminal thrombus predicts rapid growth of abdominal aortic aneurysms. Radiology 294(3), 707–713 (2020)


# Test Code Refactoring Unveiled: Where and How Does It Affect Test Code
Quality and Effectiveness?

Luana Martins1, Valeria Pontillo2, Heitor Costa3, Filomena Ferrucci2, Fabio
Palomba2, Ivan Machado1<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>1Federal University of Bahia, Salvador, Brazil  2Software Engineering (SeSa)
Lab — University of Salerno, Fisciano, Italy  3Federal University of Lavras,
Lavras, Brazil

###### Abstract

Context. Refactoring has been widely investigated in the past in relation to
production code quality, yet still little is known on how developers apply
refactoring on test code. Specifically, there is still a lack of investigation
into how developers typically refactor test code and its effects on test code
quality and effectiveness. Objective. This paper presents a research agenda
aimed to bridge this gap of knowledge by investigating (1) whether test
refactoring actually targets test classes affected by quality and
effectiveness concerns and (2) the extent to which refactoring contributes to
the improvement of test code quality and effectiveness. Method. We plan to
conduct an exploratory mining software repository study to collect test
refactoring data of open-source Java projects from GitHub and statistically
analyze them in combination with quality metrics, test smells, and
code/mutation coverage indicators. Furthermore, we will measure how
refactoring operations impact the quality and effectiveness of test code.

###### Index Terms:

Software testing, Test smells, Test refactoring, Refactoring mining, Mining
software repositories

##  I Introduction

Refactoring is an engineered approach that allows developers to improve the
quality of source code without affecting its external behavior [1]. Over the
last decades, researchers have been proposing automated refactoring
recommenders [2] and investigated how refactoring relates to code quality [3,
4, 5, 6]. In particular, researchers identified both benefits and drawbacks of
its application [7, 8, 9], finding that, while refactoring is theoretically
associated with modifications that do not affect the external behavior of
source code, it may possibly induce defects [10, 11, 12], vulnerabilities
[13], or even code smells [14]. These drawbacks are mainly due to refactoring
activities performed manually without the support of automated tools and
interleaved with other code changes [15]. Our research is motivated by these
previous works. On the one hand, most previous studies focused on the
refactoring of production code and, for this reason, we argue that there is a
lack of investigations into _how refactoring is applied to test code_. On the
other hand, we do not know if similar effects observed in previous work may
arise with test refactoring, i.e., it may have some impact on both test
quality and effectiveness, for instance, in cases where refactoring actions
target the logic of a test case. Hence, we point out a _limited knowledge on
the effects of refactoring_ on both test quality and effectiveness.

An improved understanding of test refactoring would have a number of potential
benefits for research and practice. In the first place, test cases represent a
crucial asset for software dependability: developer’s productivity is partly
dependent on the quality of test cases [16], as these help practitioners to
decide on whether to merge pull requests or deploy the system [17]. As such,
analyzing how refactoring affects test cases may have a significant impact on
practice. Secondly, researchers have been showing that the design of test code
is approached in a substantially different way with respect to traditional
development [18]. Indeed, the test code must often interact with external
systems, databases, or APIs to set up test environments and verify the
system’s behavior [19]. As a consequence, test code may suffer from different
issues that, in turn, would require different refactoring operations [20].

For these reasons, new refactoring practices have been proposed with the aim
of dealing with quality or effectiveness concerns [21, 19, 20]. While those
refactoring practices were the target of some previous investigations,
researchers limited their focus to how refactoring may influence test smells,
i.e., symptoms of poor test code quality [22, 23, 24], hence not providing
comprehensive analysis into the _nature_ and _effects_ of test refactoring.
More specifically, we highlight a lack of knowledge on (1) whether developers
apply test refactoring operations on test classes that are actually affected
by quality or effectiveness concerns, as it is supposed to be given the
definition of refactoring; and (2) what is the effect of refactoring on both
quality and effectiveness of test cases.

This paper aims at addressing this gap of knowledge by proposing an
_exploratory empirical study_. We first plan to collect test refactoring data
from the change history of open-source Java projects from GitHub and combine
them with data coming from automated instruments able to profile test code
from the perspective of quality metrics, test smells, and code/mutation
coverage information. Afterward, we plan to apply statistical analyses to
address three main research goals targeting (1) whether test classes with a
low level of quality, in terms of test smells and code metrics, are associated
with more test refactoring, (2) whether a low level of effectiveness, in terms
of mutation coverage and code coverage, is associated with more test
refactoring, and (3) to what extent the removal of test smells improve the
test code quality and effectiveness.

Our findings might benefit researchers and practitioners under multiple
perspectives. In the first place, our research may reveal insights into the
refactoring types that may deteriorate test code quality and effectiveness.
Such information would be relevant for researchers in both the fields of
refactoring and testing, as it may lead them to (1) extend the knowledge on
the best and bad practices to properly apply test refactoring; (2) devise
novel test refactoring approaches which are aware of the possible side effects
of refactoring, e.g., we may envision multi-objective search-based refactoring
approaches that may optimize refactoring recommendations based on both quality
and effectiveness attributes; and (3) design novel recommendation systems that
may support developers in understanding how a refactoring would impact
different test code properties. The results would also be useful to
practitioners, who may have additional proof of the side effects of
refactoring, hence possibly being stimulated further on the need to employ
automated refactoring tools. In the second place, our findings may indicate
the nature of the test cases more likely to be subject to refactoring
operations. Researchers might use this information to define refactoring
recommenders and refactoring prioritization approaches, while practitioners
may acquire the awareness of their actions.

##  II Related Work

The current literature can be distinguished based on the type of empirical
studies conducted. First, several studies analyzed change history information
to extract knowledge about test smells and their impact. Spadini et al. [25]
investigated ten open-source projects to find a relation between six test
smells and the change and defect-proneness of both test and production code,
finding that smelly JUnit tests are more change-prone and defect-prone than
non-smelly ones. In addition, they found that production code is typically
more defect-prone when tested by smelly tests. As such, the authors did not
target test code refactoring, hence not assessing how the seemingly test code
quality improvement actions performed by developers affect test code quality
and effectiveness, i.e., the authors looked exactly in the opposite direction
of our paper, focusing on how bad practices affect test code quality.

Wu et al. [26] explored the impact of eliminating test smells on the
production code quality of ten open-source projects. In this respect, there
are two key points that make our investigation novel: (1) test smell removal
does not imply the application of refactoring: a previous empirical study [27]
indeed showed that 83% of test smell removal activities are due to feature
maintenance actions, i.e., our work can therefore further the knowledge on how
developers apply test code refactoring; (2) the authors worked, also in this
case, in the opposite direction as our work, focusing on the effects of test
smells on code quality rather than analyzing the impact of test code
refactoring actions. As such, our work extends the current knowledge by
assessing how test refactoring is applied and what is its impact on multiple
aspects of test code, such as quality and effectiveness.

Peruma et al. [24] investigated the relationship between refactoring changes
and their effect on test smells. The authors used Refactoring Miner [28] to
detect refactoring operations and the tsDetect tool [29] to identify the test
smells from unit test files of 250 open-source Android Apps. Results showed
that refactoring operations in test and non-test files differ, and the
refactorings co-occur with test smells. With respect to the work by Peruma et
al. [29], we do not limit ourselves to the analysis of test smells, but also
consider additional indicators of test code quality and effectiveness: in this
sense, ours will represent a more comprehensive analysis of the role of test
refactoring. Second, we assess the actual effects of test refactoring on test
code quality and effectiveness, providing insights into how various test
refactoring types may support the evolutionary activities of developers.

A second line of research is represented by qualitative studies targeting the
developer’s perception of test refactoring. Damasceno et al. [30] investigated
the impact of test smell refactoring on internal quality attributes, reporting
some insights that may potentially be in line with the results of our study,
e.g., they let emerge the impact of test smell refactoring on internal quality
attributes. In the first place, the authors specifically focused on the
refactoring of test smells, while our work targets test code refactoring from
a more general perspective, attempting to assess the extent to which this is
applied to classes suggesting the presence of quality or effectiveness
concerns. Secondly, the results of our work may possibly provide evidence-
based, complementary insights with respect to what the authors found out in
their qualitative study. Third, our work has a broader scope and, indeed, it
also targets the effectiveness side of the problem. Soares et al. [22]
investigated how developers refactor test code to eliminate test smells. The
authors surveyed 73 open-source developers and submitted 50 pull requests to
assess developers’ preferences and motivation while refactoring the test code.
The results showed that developers preferred the refactored test code for most
test smells. In another work, Soares et al. [23] investigated whether the
JUnit 5 features help refactor test code to remove test smells. They conducted
a mixed-method study to analyze the usage of the testing framework features in
485 popular Java open-source projects, identifying the features helpful for
test smell removal and proposing novel refactorings to fix test smells. Also
in this case, the authors focused on the refactoring of test smells, while our
study has a broader scope. In addition, while we do not plan to conduct
surveys or interviews—this is part of our future research agenda—, we will
extend the current body of knowledge by assessing whether test code quality
and effectiveness indicators may trigger refactoring activities, other than
providing a comprehensive overview of how test refactoring relates to branch
and mutation coverage, which is a premiere of our study.

##  III Research Questions and Objectives

The goal of the empirical study is to analyze the test refactoring operations
performed by developers over the history of software projects, with the
purpose of understanding (1) whether low-quality test classes, in terms of
structural metrics and test smells, provide indications on which test classes
are more likely of being refactored, (2) whether test classes with low
effectiveness, in terms of code coverage and mutation coverage, provide
indications on which test classes are more likely of being refactored, and (3)
as a consequence, to what extent test refactoring operations are effective in
improving quality and effectiveness of test classes. In other terms, we are
first interested in assessing the quantity of test refactoring operations
performed on classes exhibiting test code quality and effectiveness issues
and, in the second place, the quality of the test refactoring operations
applied in terms of improvements provided to test code quality and
effectiveness. The perspective is of both researchers and practitioners who
are interested in understanding the relationship and effects of test
refactoring operations on the quality and effectiveness of test classes.

More specifically, our empirical investigation will first aim at addressing
the following research questions (RQs):

RQ1. _Are test refactoring operations performed on test classes having a low
level of quality, as indicated by quality metrics and test smell detectors?_

RQ2. _Are test refactoring operations performed on test classes having a low
level of effectiveness, as indicated by code and mutation coverage?_

Through RQ1 and RQ2, we aim to address the first objective of the study, hence
understanding whether the low quality and effectiveness of test classes are
associated with more test refactoring operations. The results of these two
research questions might have multiple implications for software maintenance,
evolution, and testing researchers. An improved understanding of these aspects
may indeed inform researchers on the characteristics of the test suites that
trigger more refactoring operations, possibly informing researchers on (1) the
factors that are associated with test refactoring and (2) the design of novel
or improved instruments to better support developers in their activities. For
instance, should we discover that test refactoring is not frequently applied
on test classes exhibiting test smells, this would imply that further research
should be conducted on the motivations leading developers to refactor test
code, other than to how test smell detectors should be designed to ease the
application of refactoring operations.

Upon completion of this investigation, we will further elaborate on the impact
of test refactoring, addressing the following research questions:

RQ3. _What is the effect of test refactoring on test code quality, as
indicated by quality metrics and test smell detectors?_

RQ4. _What is the effect of test refactoring on test code effectiveness, as
indicated by code and mutation coverage?_

Through RQ3 and RQ4, we aim to extend the current knowledge on the impact of
test refactoring, assessing whether the test code quality and effectiveness
increase, decrease or remain the same after the application of test
refactoring operations. It is worth to remark that addressing these two
research questions would be important independently from the results obtained
by RQ1 and RQ2. Indeed, regardless of the amount of refactoring operations
performed on test classes exhibiting quality or effectiveness concerns, it
would still be possible that the specific refactoring actions targeting those
classes have an impact. To make our argumentation more practical, consider the
case of the _Extract Method_ refactoring, whose suboptimal implementation may
potentially affect test code effectiveness. Given a verbose test method with
several steps and assertions, the refactoring enables the extraction of
multiple test methods, which are supposed to be more cohesive and focused on
the verification of specific conditions of production methods. However, if
developers do not appropriately perform such an extraction, this would
potentially change the logic of the test and be detrimental to test
effectiveness. For instance, consider test T, which verifies two branches, B1
and B2, of the production method M. In this case, an Extract Method operation
is supposed to split T so that the resulting tests T1 and T2 target B1 and B2
individually. However, should there be a logical relation between B1 and B2,
T2 will still need to pass through T1 to ensure that the logical relation is
still met: a suboptimal refactoring may overlook this requirement, possibly
not embedding in T2 the statements required to reach B1. As a result, this
operation would affect the overall level of coverage of the production code.

As such, RQ3 and RQ4 provide an orthogonal view on the matter. Also in this
case, the outcome of our investigation may lead to implications for research
and practice. First, our findings may help researchers measure the actual,
practical impact of test refactoring—this may drive considerations on how
future research efforts should be prioritized, e.g., by favoring more research
on impactful refactoring operations. Second, our results may increase the
practitioner’s awareness of test refactoring, possibly increasing its
application in practice.

To design and report our empirical study, we will follow the empirical
software engineering guidelines by Wohlin et al. [31] other than the
ACM/SIGSOFT Empirical Standards.111Available at:
https://github.com/acmsigsoft/EmpiricalStandards

##  IV Experimental Plan

This section reports the research method that we plan to apply to address our
RQs.

###  IV-A Context of the study

The _context_ of our investigation will be composed of (i) empirical study
variables, i.e., the independent and dependent variables that we will
statistically analyze, and (ii) software systems, i.e., the projects that will
be mined to collect the data required to address our research objectives.

TABLE I: Description of quality metrics as detected by VITRuM [32] Acronym | Quality Metrics | Description
---|---|---
LOC | Number of Lines |  Counts the number of lines
NOM | Number of Methods |  Counts the number of methods
WMC | Weight Method Class |  Counts the number of branch instructions in a class
RFC | Response for a Class |  Counts the number of method invocations in a class
AD | Assertion density |  Percentage of assert statements with respect to the total number of statements in a test class
MUT | Mutation Coverage |  Percentage of mutated statements in the production class that is covered by the test
LCOV | Line coverage |  Lines exercised by the test
BCOV | Branch coverage |  Branches exercised by the test
TABLE II: Description of test smells as detected by tsDetect [29] Acronym | Test Smell | Description | Precision | Recall
---|---|---|---|---
AR | Assertion Roulette |  A test method contains assertion statements without an explanation/message | 94.7% | 90.0%
DA | Duplicate Assert |  A test method that contains more than one assertion statement with the same parameters | 85.7% | 90.0%
ECT | Handling Exception |  A test method that contains throws statements | 100.0% | 100.0%
ET | Eager Test |  A test method contains multiple calls to multiple production methods | 100.0% | 100.0%
GF | General Fixture |  Fields within the setUp method are not utilized by all test methods | 95.2% | 100.0%
LT | Lazy Test |  Multiple test methods call the same class under test methods | 90.9% | 100.0%
TABLE III: Description of refactorings detected by TestRefactoringMiner tool Refactoring | Description | Precision | Recall
---|---|---|---
Add assert explanation |  Add an optional parameter into the assert methods to provide an explanatory message | 100.0% | 78.0%
Extract Class |  Create a new class and place the fields and methods responsible for the relevant functionality in it | 100.0% | 100.0%
Extract Method |  Move a code fragment to a separate new method and replace the old code with a call to the method | 99.9% | 96.9%
Inline Method |  Replace calls to the method with the method’s content and delete the method itself | 100.0% | 98.2%
Parameterize Test |  Remove duplicate code using the @parameterized test annotation to define a variety of arguments | 100.0% | 100.0%
Replace @Test annotation w/ assertThrows |  Remove @Test annotation and add of assertThrows method | 100.0% | 93.0%
Replace @Rule annotation w/ assertThrows |  Remove @Rule annotation and add of assertThrows method | 100.0% | 88.0%
Replace try/catch w/ assertThrows |  Remove try/catch blocks and add of assertThrows method | 100.0% | 89.0%
Split method |  Separate a long function by splitting it into short methods and adding a call for the new methods | 100.00% | 100.00%

Software Systems. The selection of suitable software systems will be driven by
various considerations. First, we will focus on open-source projects, as we
need access to change history information. Second, we will rely on popular,
large real-world projects having enough releases to collect data that can be
analyzed statistically. Third, we will standardize the building process to
ease dependency management and streamline build configurations across all
projects. As such, we plan to use SEART tool222https://seart-ghs.si.usi.ch/ to
select 100 open-source, non-fork projects from GitHub that have at least 100
stars, 10 major releases, 1,000 lines of code, and 10 test classes. We will
seek Java projects that can be compiled with Maven and Java 8—Java 8 is the
most popular Java version used nowadays.333
https://www.jetbrains.com/lp/devecosystem-2021/java/ Should our search
identify more than 100 projects, we will apply random sampling and verify
whether the projects were properly built until we have 100 projects.

It is worth noting that some projects may adopt the so-called _Boy Scout_
rule, i.e., “Leave every piece of code you touch cleaner than how they found
it”.444 The Boy Scout Rule: https://www.oreilly.com/library/view/97-things-
every/9780596809515/ch08.html. These projects may be more inclined to the
application of refactoring and therefore we may observe a higher test code
quality and effectiveness. As part of our study, we will manually analyze the
contribution guidelines of the projects selected, looking for any insight
suggesting that those projects follow the _Boy Scout_ rule. Should we identify
a decent amount of projects, we will perform an additional analysis, comparing
the results obtained between Boy Scout and non-Boy Scout projects.

Empirical Study Variables. In the context of RQ1 and RQ2, we are interested in
assessing whether refactoring operations are more likely to be observed on
test classes exhibiting test code quality and effectiveness concerns. As such,
we define the following empirical study variables:

_Independent Variables._ These are the factors that will be related to the
application of test refactoring, namely (i) test code quality metrics; (ii)
presence of test smells (of different types); (iii) branch coverage; and (iv)
mutation coverage. Tables I and II list and describe the independent variables
of the study. These metrics will be all computed across releases of different
software systems and will be statistically analyzed as described later in this
section. The selection of these independent variables is driven by multiple
considerations. First, we consider test code quality metrics and test smells
that were targeted by previous research in the field [33, 34] and found to
impact test code in different manners [35, 27]. Second, branch and mutation
coverage are widely considered as two key indicators of test code
effectiveness, which may estimate the goodness of test cases in dealing with
real defects [36, 37].

_Dependent Variables._ These are the refactoring operations (of different
types) being observed across releases of different software systems. To select
suitable test refactoring operations for our purpose, we investigated the
literature to elicit the test refactoring operations that were previously
associated to our independent variables—we basically surveyed the previous
papers on the matter, discussed in Section II, to extract the test refactoring
types that researchers have been observing as potentially impacting on testing
evolutionary activities. Table III lists the refactoring operations that will
be targeted, along with a brief description.

When it turns to RQ3 and RQ4, we are interested in assessing the impact of
test refactoring on the test code quality and effectiveness aspects
considered. As such, we need to swap independent and dependent variables:
indeed, in this case we are interested to observe how refactoring impacts test
code properties rather than the opposite:

_Independent Variables._ These are the different types of refactoring
operations (Table III) computed across the releases of software systems
considered.

_Dependent Variables._ These will be the test code quality and effectiveness
metrics described in Tables I and II, which will be computed across releases
of software systems.

In all RQs we will include a number of control variables, which will help us
better verify the extent to which test refactoring impacts the variation of
test code quality and effectiveness in relation to project- and process-level
characteristics that may impact the dependent variables.

_Control Variables._ We will first account for the frequency of releases and
activities by the project, as these may provide insights into the development
speed which, in turn, may impact test code quality and effectiveness. Given a
release $R_{i}$, we will compute the number of releases issued within the last
1, 3, 6, and 12 months. In addition, for each class $C_{j}$ within $R_{i}$, we
will compute the number of commits performed by developers between the
releases $R_{i-1}$ and $R_{i}$. We will also consider project-level metrics
such as (1) project size in terms of lines of code; (2) number of
contributors; (3) number of branches; and (4) number of pull requests. On the
one hand, these metrics can well overview the main characteristics of the
project and the community around it. On the other hand, all these metrics can
impact in various manners test code quality and effectiveness, e.g., a higher
amount of branches may indicate a higher level of activity around the project,
which in turn can influence the way test cases are maintained and evolved.

###  IV-B Data Collection

We will use different automated tools available in the literature to extract
data on quality and effectiveness metrics, test smells, and refactoring
operations. Then, we will merge the data to compose our dataset.

Collecting test code quality and effectiveness metrics. To collect both test
code quality and effectiveness metrics (Table I), we will run VITRuM, a plug-
in for the visualization of test-related metrics in order to calculate five
static metrics and three dynamic metrics from the test code [32]. Note that
the tool uses JaCoCo to calculate line and branch coverage, and pitest for the
mutation coverage. Therefore, we will have to build the projects to calculate
the dynamic metrics.

Collecting test smells. Among the test smell detection tools available for
Java code [38], we will use tsDetect [29], which is the most accurate tool,
with a precision score ranging from 85% to 100% and a recall score ranging
from 90% to 100%. tsDetect performs a test code static analysis through an AST
(Abstract Syntax Tree) to apply the test smells detection rules in the test
files. A test file in the JUnit testing framework should follow the naming
conventions of either pre-pending or appending the word ‘‘Test’’ to the name
of the production class under test and at the same package hierarchy [29].
With the detection rules, the tool can detect (i) the presence or absence of a
test smell in a test class, or (ii) the number of instances per test smell in
a test class. In addition, the tool receives a configuration of the severity
thresholds for each test smell [39]. We run the tool to identify the number of
instances of the six test smells described in Table II with default values for
the severity thresholds (i.e., the tool reports all instances of test smells
detected).

Collecting refactoring data. To detect test refactoring operations, we will
use the TestRefactoringMiner tool [40]. The tool is built on top of the state-
of-the-art refactoring mining tool RefactoringMiner, which has the highest
precision (99.8%) and recall (97.6%) scores among the currently available
refactoring mining tools [28]. In more detail, TestRefactoringMiner analyzes
the added, deleted, and changed files between two project versions to detect
specific test refactorings, reaching 100% and 92.5% of precision and recall
scores. The tool operationalizes the detection of all the refactoring
operations considered in the study—Table III presents the set of test
refactoring that we will investigate in this study. It is worth noting that
this set considers various refactoring operations, such as integrating new
technologies like JUnit 5 or improving the organization of test classes.

Data integration. Although some tools allow a finer granularity during the
code analysis, all of them can also report the results at the class level.
Therefore, we will establish traceability links between the test classes
reported by tsDetect, VITRuM, and TestRefactoringMiner tools, finally
integrating their outcome in a unique data source to be further analyzed from
a statistical standpoint.

###  IV-C Data Analysis

We first formulate the working hypotheses that we will later statistically
assess. As for RQ1, given a quality metric $Qm_{i}$, with $Qm_{i}$ in {LOC,
NOM, WMC, RFC, AD} and a refactoring $ref_{k}$ in the set of refactoring
operations considered in the study, our null hypothesis is the following:

Hn1${}_{Qm_{i}-ref_{k}}$.

There is _no significant difference_ in terms of the amount of $ref_{k}$
operations performed on test classes having different values of $Qm_{i}$.

As in RQ1, we will also evaluate the relation between test refactoring and
test smells. Given a test smell $Ts_{i}$ in the set of test smells considered
in the study and $ref_{k}$, we define a second null hypothesis:

Hn2${}_{Ts_{i}-ref_{k}}$.

There is _no significant difference_ in terms of the amount of $ref_{k}$
operations performed on test classes affected and not by $Ts_{i}$.

As for RQ2, given an effectiveness metric $Em_{i}$, where $Em_{i}$ assumes
values in the set {Branch Coverage and Mutation Coverage} and $ref_{k}$, the
null hypothesis is the following:

Hn3${}_{Em_{i}-ref_{k}}$.

There is _no significant difference_ in terms of the amount of $ref_{k}$
operations performed on test classes having different values of $Em_{i}$.

As for RQ3, given a quality metric $Qm_{i}$, a test smell $Ts_{i}$, and a
refactoring $ref_{k}$, the null hypotheses is:

Hn4${}_{Qm_{i}-ref_{k}}$.

There is _no significant difference_ in terms of $Qm_{i}$ before and after the
application of $ref_{k}$.

Hn5${}_{Ts_{i}-ref_{k}}$.

There is _no significant difference_ in the number of $Ts_{i}$ instances
before and after the application of $ref_{k}$.

Finally, as for RQ4, the null hypothesis will be:

Hn6${}_{Em_{i}-ref_{k}}$.

There is _no significant difference_ in terms of $Em_{i}$ before and after the
application of $ref_{k}$.

If one of the null hypotheses will be statistically rejected, we will accept
the corresponding alternative hypothesis, namely:

An1${}_{Qm_{i}-ref_{k}}$.

The amount of $ref_{k}$ operations on test classes having different values of
$Qm_{i}$ is _statistically different_.

An2${}_{Ts_{i}-ref_{k}}$.

The amount of $ref_{k}$ operations on test classes affected and not by
$Ts_{i}$ is _statistically different_.

An3${}_{Em_{i}-ref_{k}}$.

The amount of $ref_{k}$ operations on test classes having different values of
$Em_{i}$ is _statistically different_.

An4${}_{Qm_{i}-ref_{k}}$.

The $Qm_{i}$ before and after the application of $ref_{k}$ is _statistically
different_.

An5${}_{Ts_{i}-ref_{k}}$.

The number of $Ts_{i}$ instances before and after the application of $ref_{k}$
is _statistically different_.

An6${}_{Em_{i}-ref_{k}}$.

The $Em_{i}$ before and after the application of $ref_{k}$ is _statistically
different_.

We will then verify the working hypotheses, hence accepting or rejecting them,
by building statistical models.

Statistical modeling for RQ1 and RQ2. To address our first two research
questions, we will devise a _Logistic Regression Model_ for each refactoring
operation considered in the study. Such a model belongs to the class of
Generalized Linear Models (GLM) [41] and relates a (dichotomous) dependent
variable—in our case, whether or not a particular type of refactoring is
performed—with either continuous and discrete independent variables—the
quality and effectiveness metrics considered in RQ1 and RQ2.

Before building the statistical model, we plan to assess the presence of
multi-collinearity [42], which arises when two or more independent variables
are highly correlated and can be predicted one from the other. We will use the
vif (Variance Inflation Factors) function and discard highly correlated
variables, putting a threshold value equal to 5 [42].

For each statistical model, we assess (i) whether each independent variable is
significantly correlated with the dependent variable (using a significance
level of $\alpha$ = 5%, and (ii) quantify this correlation using the Odds
Ratio (OR) [43], which is a measure of the strength of the association between
each independent variable and the dependent variable. Higher OR values for an
independent variable indicate a higher probability of explaining the dependent
variable, i.e., a higher likelihood that a refactoring operation has been
triggered by the independent variable. Nonetheless, the interpretation of OR
values change depending on the different measurement scale of the independent
variables, i.e., ratio for the test code quality and effectiveness metrics and
categorical for the test smells. As for the metrics, the OR for an independent
variable indicates the increment of chances for a test class to be subject of
refactoring as a consequence of a one-unit increase of the independent
variable. As for test smells, the OR indicates how likely a smelly test class
is involved in refactoring operations with respect to a non-affected class.

The statistical significance of the correlation between independent and
dependent variables will allow us to accept or reject
Hn1${}_{Qm_{i}-ref_{k}}$, Hn2${}_{Ts_{i}-ref_{k}}$, and
Hn3${}_{Em_{i}-ref_{k}}$, while OR values will measure the strengths of the
correlations.

Statistical modeling for RQ3 and RQ4. To statistically assess the impact of
test refactoring on test code quality and effectiveness metrics and smells, we
will first collect all the test classes subject to the refactoring type
$ref_{k}$ in a generic release $R_{i}$. Afterward, for each of those test
classes, we will compute its value of test code quality and effectiveness
metrics and smells computed on the release $R_{i}$ and the value of the
metrics and smells computed on the release $R_{i-1}$.

We will produce two distributions: the first representing the metric values
(or the number of test smells) in $R_{i-1}$, i.e., before the application of
$ref_{k}$; the second representing the metric values (or the number of test
smells) in $R_{i}$, i.e., after the application of $ref_{k}$. On this basis,
we will employ the non-parametric Wilcoxon Rank Sum Test [44] (with
$\alpha$-value = 0.05), through which we will accept or reject the null
hypotheses Hn4${}_{Qm_{i}-ref_{k}}$, Hn5${}_{Ts_{i}-ref_{k}}$, and
Hn6${}_{Em_{i}-ref_{k}}$.

In addition, we will also rely on the Vargha-Delaney ($\hat{A}_{12}$) [45]
statistical test to measure the magnitude of the differences observed in the
considered distributions. According to the direction and value given by
$\hat{A}_{12}$ we will have a practical interpretation of our findings, which
will depend on the test code factor considered. Specifically, should the
$\hat{A}_{12}$ values be lower than 0.5, this would imply that:

  * •

The metric values computed on the release $R_{i-1}$ are lower than those on
$R_{i}$, i.e., the refactoring $ref_{k}$ would have a _positive_ effect on the
quality or effectiveness metric considered—lower metric values in $R_{i-1}$
would indeed indicate that the refactoring induced the increase of the metric
in $R_{i}$, hence having a positive effect;

  * •

The number of test smells computed on the release $R_{i-1}$ is lower than the
one computed on $R_{i}$, i.e., the refactoring $ref_{k}$ would have a
_negative_ effect, hence suggesting that, rather than improving test code
design, the refactoring induced the emergence of some form of test smells.

Similarly, a $\hat{A}_{12}>0.50$ indicates the opposite, hence that either
$ref_{k}$ has a _negative_ impact on the considered test code quality or
effectiveness metric, or that the refactoring has a _positive_ impact of the
removal of test smells. Finally, $\hat{A}_{12}==0.50$ points out that the
results are identical, i.e., the refactoring has limited to no effect on the
dependent variables.

###  IV-D Publication of generated dataset

The dataset that we will collect by merging test code metrics, test smells,
test effectiveness metrics, and test refactoring data will be made publicly
available in an online repository [46]. We also plan to release the scripts
for the data collection and analysis that we will use to perform this study.

##  V Threats to validity

This section discusses the potential threats that may affect the validity of
our empirical study plan.

Construct validity. A first threat concerns with the criteria we will use to
select software projects: despite the actions to standardize the building
process, we might still fall into build failures. Should this happen, we will
attempt to manually diagnose the reasons of the failures, trying to fix them -
in this respect, we will exploit recent research [47, 48] reporting insights
on how to fix build failures. In the best case, we would still be able to
build the project. In the worst case, we would not be able to fix the build
failure and, in this case, we will finally discard the project from our study
and replace it with another project retrieved by using the SEART tool.

As for the set of test smells, structural and dynamic metrics we will use to
assess the test code quality. We will not calculate all the Chidamber &
Kemerer metrics as some of them do not apply to the context of test code
(e.g., Depth Inheritance Tree). Nevertheless, we have chosen a mix of metrics
capturing the test code size, structural, and dynamic characteristics. Another
threat to validity concerns the identification of test smells and refactoring
operations. We will use tools already validated and used by the research
community. Although the tools present high precision and recall scores, they
might report some false positive or false negative instances of test smells or
refactorings: in response of this limitation, we will attempt to perform
preliminary, manual investigations to assess the degree of accuracy of the
tools before running them on a large scale—in this way, we will be able to
provide indications on the confidence level of our conclusions.

Internal Validity. This category of threats to validity concerns by-product
changes of other maintenance activities (e.g., bug fixes or changes in
requirements) that could also contribute to the removal of test smells.
Therefore, the data analysis will not indicate a causal relationship, but
rather that there is a possibility of a relationship that may be further
investigated. We will attempt to corroborate our quantitative results by means
of some qualitative insights. In addition, we acknowledge test flakiness as a
potential threat to the internal validity which can impact the reliability of
our findings. However, despite being a severe issue for practitioners,
previous investigations found test flakiness to arise in a limited amount of
cases, e.g., Luo et al. [49] found out that flaky tests affect up to 4.56% of
test cases. In this sense, it is reasonable to believe that the problem of
test flakiness will have a limited impact on our findings.

External Validity. This class of threats to validity mainly concerns the
subject projects of our study. We selected open-source Java projects from
GitHub, which are only a fraction of the complete picture of open-source
software and do not necessarily represent industrial practices. Therefore, the
results may not generalize to the industrial context and other programming
languages. In addition, we will select projects based on the number of stars,
which may raise some popularity bias. Replications of our work would be,
therefore, beneficial to corroborate our findings in different contexts: to
stimulate further research, we will release all materials and scripts as part
of an archived online appendix [46].

Conclusion validity. To address how frequently test refactoring is performed
on test classes affected by quality or effectiveness concerns, we will use
logistic regression models to identify correlations. Other than highlighting
cases of significant correlations, we will report and discuss OR values. In
addition, to investigate the effect of test refactoring on test code quality
and effectiveness, we will employ well-established statistical tests such as
the Wilcoxon Rank Sum Test [44] and the Vargha-Delaney ($\hat{A}_{12}$) [45]
statistical tests. Our analysis will be conducted at the granularity of
classes because the tools we plan to employ work at this level. This may bias
our conclusions, as this granularity may be subject to various confounding
variables. On the one hand, this is a limitation that we unfortunately share
with all the other research works that analyze dynamic test code metrics [50].
On the other hand, we plan for the inclusion of multiple process- and project-
level control variables, through which we will be able to partially mitigate
this threat to validity.

An additional point to remark is that our data collection procedure cannot
distinguish between changes that were meant as refactoring and other changes
where refactoring was applied as part of other modifications. We might have
mitigated this limitation by extracting refactoring changes through the
analysis of issues and pull requests, i.e., collecting changes explicitly
intended as refactoring. Nonetheless, such an alternative method could have
biased even further the conclusions drawn for two reasons connected to the
availability and reliability of the information available within the
developers’ discussions on GitHub. More particularly:

_Availability._ Previous studies established that developers perform “floss
refactoring”, combining refactoring operations and behavioral change edits
within individual commits [15]. From a practical standpoint, this means that
developers do not often apply refactoring for the sake of refactoring source
code, but as an instrument to perform other changes, e.g., to simplify a piece
of code before making further evolutionary changes. As such, it is unlikely to
find “pure” refactoring changes or discussions, in the form of issues or pull
requests, around refactoring operations to be applied.

_Reliability._ Literature found that developers not only rarely document
refactoring activities explicitly [51, 52], but also that when they do, they
are inconsistent [53], i.e., labeling changes as refactoring, although no
refactoring is done at all. Other researchers found out that the term
“refactoring” is misused, i.e., developers do not often correctly distinguish
between refactoring changes and normal code modifications [54]. In this
respect, the seminal paper by Murphy-Hill et al. [55] reported that _“messages
in version histories are unreliable indicators of refactoring activities. This
is due to the fact that developers do not consistently report/document
refactoring activities”_. This latter observation was also backed up by the
findings obtained by Ratzinger et al. [56], who discovered that the extraction
of refactoring documentation from repositories may lead to several false
positives, as the words used by developers are too generic and do not often
refer to real refactoring operations.

As a consequence, the analysis of issues and pull requests would have led to
unreliable conclusions. On the contrary, the goal of a statistical study is
exactly that to identify hidden relations between dependent and independent
variables while controlling for possible confounding effects [57]: we believe
that such an approach better fits our research goals. Through a large-scale,
statistical investigation, we may indeed end up discovering the intrinsic
factors associated with the refactoring actions performed by developers,
finally providing evidence of how test refactoring is done in practice.

##  VI Conclusion

The ultimate goal of our research plan is to understand whether the test code
quality and effectiveness provide indications of which test classes are more
likely of being refactored and to what extent test refactoring operations can
improve the test code quality and effectiveness. We will conduct this study on
a set of 100 open-source Java projects, starting from the collection of data
on the test code quality, test smells, and refactoring operations arising in
the major releases of the projects. Then, we will employ statistical
approaches to address the goals of our investigation and, based on the
conclusions we will be able to draw, finally provide actionable items and
implications for researchers and practitioners.

As an outcome of our exploratory study, we expect to provide the following key
contributions:

  1. 1.

An empirical understanding of the factors triggering test refactoring
operations, which comprises an analysis of how test code quality and
effectiveness come into play;

  2. 2.

Evidence of the impact of test refactoring on test code quality and
effectiveness;

  3. 3.

An online appendix which will provide all material and scripts employed to
address the goals of the study.

## Acknowledgment

This study was financed in part by the Coordenação de Aperfeiçoamento de
Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 001; and FAPESB
grants BOL0188/2020 and PIE0002/2022. Fabio is supported by the Swiss National
Science Foundation through the SNF Project No. PZ00P2_186090 (TED).

## References

  * [1] M. Fowler, _Refactoring: improving the design of existing code_. USA: Addison-Wesley Longman Publishing Co., Inc., 1999.
  * [2] G. Bavota, A. De Lucia, A. Marcus, and R. Oliveto, “Recommending refactoring operations in large software systems,” _Recommendation Systems in Software Engineering_ , pp. 387–419, 2014.
  * [3] M. I. Azeem, F. Palomba, L. Shi, and Q. Wang, “Machine learning techniques for code smell detection: A systematic literature review and meta-analysis,” _IST_ , vol. 108, pp. 115–138, 2019.
  * [4] E. V. de Paulo Sobrinho, A. De Lucia, and M. de Almeida Maia, “A systematic literature review on bad smells–5 w’s: which, when, what, who, where,” _IEEE TSE_ , vol. 47, no. 1, pp. 17–66, 2018.
  * [5] B. Du Bois, S. Demeyer, and J. Verelst, “Refactoring - improving coupling and cohesion of existing code,” in _11th Working Conf. on Reverse Engineering_ , 2004, pp. 144–151.
  * [6] A. Chávez, I. Ferreira, E. Fernandes, D. Cedrim, and A. Garcia, “How does refactoring affect internal quality attributes? a multi-project study,” in _Proceedings of the XXXI Brazilian Symposium on Software Engineering_ , ser. SBES ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 74–83.
  * [7] J. Al Dallal, “Identifying refactoring opportunities in object-oriented code: A systematic literature review,” _IST_ , vol. 58, pp. 231–249, 2015.
  * [8] A. A. B. Baqais and M. Alshayeb, “Automatic software refactoring: a systematic literature review,” _Software Quality Journal_ , vol. 28, no. 2, pp. 459–502, 2020.
  * [9] G. Lacerda, F. Petrillo, M. Pimenta, and Y. G. Guéhéneuc, “Code smells and refactoring: A tertiary systematic review of challenges and observations,” _JSS_ , vol. 167, p. 110610, 2020.
  * [10] M. Di Penta, G. Bavota, and F. Zampetti, “On the relationship between refactoring actions and bugs: a differentiated replication,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 556–567.
  * [11] G. Bavota, B. De Carluccio, A. De Lucia, M. Di Penta, R. Oliveto, and O. Strollo, “When does a refactoring induce bugs? an empirical study,” in _2012 IEEE 12th International Working Conference on Source Code Analysis and Manipulation_. IEEE, 2012, pp. 104–113.
  * [12] I. Ferreira, E. Fernandes, D. Cedrim, A. Uchôa, A. C. Bibiano, A. Garcia, J. a. L. Correia, F. Santos, G. Nunes, C. Barbosa, B. Fonseca, and R. de Mello, “The buggy side of code refactoring: Understanding the relationship between refactorings and bugs,” in _Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings_ , ser. ICSE ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 406–407. [Online]. Available: https://doi.org/10.1145/3183440.3195030
  * [13] E. Iannone, Z. Codabux, V. Lenarduzzi, A. De Lucia, and F. Palomba, “Rubbing salt in the wound? a large-scale investigation into the effects of refactoring on security,” _Empirical Software Engineering_ , vol. 28, no. 4, p. 89, 2023.
  * [14] M. Tufano, F. Palomba, G. Bavota, R. Oliveto, M. Di Penta, A. De Lucia, and D. Poshyvanyk, “When and why your code starts to smell bad,” in _2015 IEEE/ACM 37th IEEE International Conference on Software Engineering_ , vol. 1. IEEE, 2015, pp. 403–414.
  * [15] E. Murphy-Hill and A. P. Black, “Why don’t people use refactoring tools?” in _Proceedings of the 1st Workshop on Refactoring Tools_ , 2007, pp. 61–62.
  * [16] J. Micco, “The state of continuous integration testing@ google,” 2017.
  * [17] G. Grano, C. De Iaco, F. Palomba, and H. C. Gall, “Pizza versus pinsa: On the perception and measurability of unit test code quality,” in _2020 IEEE Int.l Conf. on Software Maintenance and Evolution (ICSME)_. IEEE, 2020, pp. 336–347.
  * [18] G. Meszaros, _xUnit test patterns: Refactoring test code_. Pearson Education, 2007.
  * [19] G. Meszaros, S. M. Smith, and J. Andrea, “The test automation manifesto,” ser. Extreme Programming and Agile Methods - XP/Agile Universe 2003, F. Maurer and D. Wells, Eds., Springer Berlin Heidelberg. Berlin, Heidelberg: Springer, 2003, pp. 73–81.
  * [20] E. M. Guerra and C. T. Fernandes, “Refactoring test code safely,” in _Int.l Conf. on Software Engineering Advances (ICSEA 2007)_. New York, NY, USA: IEEE, 2007, pp. 44–44.
  * [21] A. Deursen, L. M. Moonen, A. Bergh, and G. Kok, “Refactoring test code,” Centre for Mathematics and Computer Science, NLD, Tech. Rep., 2001.
  * [22] E. Soares, M. Ribeiro, G. Amaral, R. Gheyi, L. Fernandes, A. Garcia, B. Fonseca, and A. Santos, “Refactoring test smells: A perspective from open-source developers,” in _Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing_ , ser. SAST 20. New York, NY, USA: Association for Computing Machinery, 2020, p. 50–59.
  * [23] E. Soares, M. Ribeiro, R. Gheyi, G. Amaral, and A. M. Santos, “Refactoring test smells with junit 5: Why should developers keep up-to-date,” _IEEE TSE_ , pp. 1–1, 2022.
  * [24] A. Peruma, C. D. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “An exploratory study on the refactoring of unit test files in android applications,” in _Proceedings of the 42nd Int.l Conf. on Software Engineering Workshops_ , ser. ICSEW’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 350–357.
  * [25] D. Spadini, F. Palomba, A. Zaidman, M. Bruntink, and A. Bacchelli, “On the relation of test smells to software code quality,” in _2018 IEEE Int.l Conf. on Software Maintenance and Evolution (ICSME)_. New York, NY, USA: IEEE, 2018, pp. 1–12.
  * [26] H. Wu, R. Yin, J. Gao, Z. Huang, and H. Huang, “To what extent can code quality be improved by eliminating test smells?” in _2022 Int.l Conf. on Code Quality (ICCQ)_. New York, NY, USA: IEEE, 2022, pp. 19–26.
  * [27] D. J. Kim, T.-H. P. Chen, and J. Yang, “The secret life of test smells-an empirical study on test smell evolution and maintenance,” _EMSE_ , vol. 26, no. 5, pp. 1–47, 2021.
  * [28] N. Tsantalis, A. Ketkar, and D. Dig, “Refactoringminer 2.0,” _IEEE TSE_ , vol. 48, no. 3, pp. 930–950, 2022.
  * [29] A. Peruma, K. Almalki, C. D. Newman, M. W. Mkaouer, A. Ouni, and F. Palomba, “tsdetect: an open source test smells detection tool,” ser. Symposium on the Foundations of Software Engineering, Association for Computing Machinery. ACM, 2020.
  * [30] H. Damasceno, C. Bezerra, E. Coutinho, and I. Machado, “Analyzing test smells refactoring from a developers perspective,” in _Proceedings of the XXI Brazilian Symposium on Software Quality_ , ser. SBQS ’22. New York, NY, USA: Association for Computing Machinery, 2023\.
  * [31] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, _Experimentation in software engineering_. Springer Science & Business Media, 2012.
  * [32] F. Pecorelli, G. Di Lillo, F. Palomba, and A. De Lucia, “Vitrum: A plug-in for the visualization of test-related metrics,” in _Proceedings of the Int.l Conf. on Advanced Visual Interfaces_ , ser. AVI’20. New York, NY, USA: Association for Computing Machinery, 2020\.
  * [33] G. Catolino, F. Palomba, A. Zaidman, and F. Ferrucci, “How the experience of development teams relates to assertion density of test classes,” in _2019 IEEE Int.l Conf. on Software Maintenance and Evolution (ICSME)_. IEEE, 2019, pp. 223–234.
  * [34] F. Pecorelli, F. Palomba, and A. De Lucia, “The relation of test-related factors to software quality: a case study on apache systems,” _EMSE_ , vol. 26, pp. 1–42, 2021.
  * [35] D. Spadini, F. Palomba, A. Zaidman, M. Bruntink, and A. Bacchelli, “On the relation of test smells to software code quality,” in _2018 IEEE Int.l Conf. on Software Maintenance and Evolution (ICSME)_ , 2018, pp. 1–12.
  * [36] P. S. Kochhar, D. Lo, J. Lawall, and N. Nagappan, “Code coverage and postrelease defects: A large-scale study on open source projects,” _IEEE Transactions on Reliability_ , vol. 66, no. 4, pp. 1213–1228, 2017\.
  * [37] M. Papadakis, D. Shin, S. Yoo, and D.-H. Bae, “Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults,” in _Proceedings of the 40th Int.l Conf. on Software Engineering_ , 2018, pp. 537–548.
  * [38] W. Aljedaani, A. Peruma, A. Aljohani, M. Alotaibi, M. W. Mkaouer, A. Ouni, C. D. Newman, A. Ghallab, and S. Ludi, “Test smell detection tools: A systematic mapping study,” ser. Evaluation and Assessment in Software Engineering. New York, NY, USA: Association for Computing Machinery, 2021, p. 170–180.
  * [39] D. Spadini, M. Schvarcbacher, A.-M. Oprescu, M. Bruntink, and A. Bacchelli, “Investigating severity thresholds for test smells,” in _Proceedings of the 17th Int.l Conf. on Mining Software Repositories_ , ser. MSR ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 311–321.
  * [40] L. Martins, H. Costa, M. Ribeiro, F. Palomba, and I. Machado, “Automating test-specific refactoring mining: A mixed-method investigation,” in _Proceedings of the 23rd IEEE International Working Conference on Source Code Analysis and Manipulation_ , 2023.
  * [41] J. A. Nelder and R. W. Wedderburn, “Generalized linear models,” _Journal of the Royal Statistical Society: Series A (General)_ , vol. 135, no. 3, pp. 370–384, 1972.
  * [42] R. M. O’brien, “A caution regarding rules of thumb for variance inflation factors,” _Quality & quantity_, vol. 41, pp. 673–690, 2007.
  * [43] J. M. Bland and D. G. Altman, “The odds ratio,” _Bmj_ , vol. 320, no. 7247, p. 1468, 2000.
  * [44] P. E. McKnight and J. Najab, “Mann-whitney u test,” _The Corsini encyclopedia of psychology_ , pp. 1–1, 2010.
  * [45] A. Vargha and H. D. Delaney, “A critique and improvement of the cl common language effect size statistics of mcgraw and wong,” _Journal of Educational and Behavioral Statistics_ , vol. 25, no. 2, pp. 101–132, 2000.
  * [46] “Data collection and analysis,” 2023, Accessed on 12.07.2023. [Online]. Available: https://figshare.com/s/dd0730e0036ebe4f878b
  * [47] M. Tufano, F. Palomba, G. Bavota, M. Di Penta, R. Oliveto, A. De Lucia, and D. Poshyvanyk, “There and back again: Can you compile that snapshot?” _Journal of Software: Evolution and Process_ , vol. 29, no. 4, p. e1838, 2017\.
  * [48] M. Maes-Bermejo, M. Gallego, F. Gortázar, G. Robles, and J. M. Gonzalez-Barahona, “Revisiting the building of past snapshots—a replication and reproduction study,” _EMSE_ , vol. 27, no. 3, p. 65, 2022\.
  * [49] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in _Proceedings of the 22nd ACM SIGSOFT Int.l Symposium on Foundations of Software Engineering_ , 2014, pp. 643–653.
  * [50] J. Kumar Chhabra and V. Gupta, “A survey of dynamic software metrics,” _Journal of computer science and technology_ , vol. 25, pp. 1016–1029, 2010\.
  * [51] P. Weißgerber and S. Diehl, “Identifying refactorings from source-code changes,” in _21st IEEE/ACM international conference on automated software engineering (ASE’06)_. IEEE, 2006, pp. 231–240.
  * [52] P. Weißgerber, B. Biegel, and S. Diehl, “Making programmers aware of refactorings.” in _WRT_ , 2007, pp. 58–59.
  * [53] E. A. AlOmar, A. Peruma, M. W. Mkaouer, C. Newman, A. Ouni, and M. Kessentini, “How we refactor and how we document it? on the use of supervised machine learning algorithms to classify refactoring documentation,” _Expert Systems with Applications_ , vol. 167, p. 114176, 2021.
  * [54] Z. Di, B. Li, Z. Li, and P. Liang, “A preliminary investigation of self-admitted refactorings in open source software (s),” in _Int.l Conf. on Software Engineering and Knowledge Engineering_ , vol. 2018. KSI Research Inc. and Knowledge Systems Institute Graduate School, 2018, pp. 165–168.
  * [55] E. Murphy-Hill, C. Parnin, and A. P. Black, “How we refactor, and how we know it,” _IEEE TSE_ , vol. 38, no. 1, pp. 5–18, 2011.
  * [56] J. Ratzinger, T. Sigmund, and H. C. Gall, “On the relation of refactorings and software defect prediction,” in _Proceedings of the 2008 Int.l working Conf. on MSR_ , 2008, pp. 35–38.
  * [57] D. A. Freedman, _Statistical models: theory and practice_. cambridge university press, 2009.