# In Search of Metrics to Guide Developer-Based Refactoring Recommendations

(A Registered Report)

Mikel Robredo  University of Oulu, Finland<EMAIL_ADDRESS>,
Matteo Esposito  University of Oulu, Finland<EMAIL_ADDRESS>,  Fabio
Palomba  University of Salerno, Italy<EMAIL_ADDRESS>,  Rafael Peñaloza
University of Milano-Bicocca, Italy<EMAIL_ADDRESS>and  Valentina
Lenarduzzi  University of Oulu, Finland<EMAIL_ADDRESS>
###### Abstract.

Context. Source code refactoring is a well-established approach to improving
source code quality without compromising its external behavior. Motivation.
The literature described the benefits of refactoring, yet its application in
practice is threatened by the high cost of time, resource allocation, and
effort required to perform it continuously. Providing refactoring
recommendations closer to what developers perceive as relevant may support the
broader application of refactoring in practice and drive prioritization
efforts. Aim. In this paper, we aim to foster the design of a developer-based
refactoring recommender, proposing an empirical study into the metrics that
study the developer’s willingness to apply refactoring operations. We build
upon previous work describing the developer’s motivations for refactoring and
investigate how product and process metrics may grasp those motivations.
Expected Results. We will quantify the value of product and process metrics in
grasping developers’ motivations to perform refactoring, thus providing a
catalog of metrics for developer-based refactoring recommenders to use.

Empirical Software Engineering, Refactoring, Software Quality

##  1\. Introduction

Maintaining code bases in the era of computing pervasiveness in daily tasks,
from smart appliances to autonomous vehicles, is getting increasingly
challenging and daunting (esposito2023uncovering, 8). With years of
development and increasing reduction of time-to-releases, developers can make
poor design choices to meet deadlines, leading to complex and hard-to-maintain
software, thus increasing subsequent maintenance effort and costs for software
firms (lenarduzzi2021systematic, 19). Code refactoring is one of the most
known techniques to mitigate software complexity(10.1145/2393596.2393655, 16).
Refactoring aims to introduce code structure changes without altering external
behavior, from code optimization to architecture and design patterns
(peruma2022refactor, 23).

Therefore, developers restructure the code to improve its design, readability,
and maintainability to ensure the long-term sustainability of software systems
(10.1145/2393596.2393655, 16, 18, 29).

Thus, developers perceive it as a significant time and resource allocation
cost due to its complexity and limitations (10.1145/2393596.2393655, 16, 2).
Therefore, with limited time-to-release windows, developers often limit
resources spent on refactoring, hindering the quality and maintenance of the
code (avgeriou2020overview, 2). Nonetheless, reducing software complexity with
minor code refactoring can do little to no improvement, leading to decreased
code quality. In contrast, bulk and more comprehensive refactoring has a more
profound influence on code quality (LACERDA2020110610, 18, 29).

Choosing the correct refactoring strategy can be challenging, leading
developers to seek assistance. Thus, developers need recommendations to guide
their refactoring effort (peruma2022refactor, 23, 29). Therefore, our work
aims to grasp developers’ motivation for refactoring. More specifically, our
study identifies metrics to guide developer-based refactoring recommendations.

No previous study investigated metrics to guide refactoring by exploiting
Large Language Models (LLMs) to analyze developers’ commit messages and
extract the code refactoring motivations. Our novel approach, which departs
from conventional techniques, promises to deliver the first developer-based
refactoring recommendations to effectively target the correct amount of
refactoring effort toward the right target. Our approach benefits
practitioners, translating into a cost reduction, prioritizing refactoring
efforts, and researchers paving the way for future works in developer-based
software maintenance.

Finally, our work is based on (silva2016we, 28) and extends (10.1145/3408302,
22). More specifically, as compared to the latter, the following notable
differences can be highlighted for what concerns the study design and goals:

  * •

Study design: Commit and Pull Requests (PR) Analysis. While Pantiuchina et al.
(10.1145/3408302, 22) analyzed PR based on keyword lookup and the RMiner tool
to detect refactoring-related commits, we propose a novel approach based on
LLMs instead of keyword analysis. Moreover, PRs represent developer
interaction but can not represent the full project change history (6976151,
33, 25). Moreover, PRs can be rejected; thus, it is also challenging to
discern expert contributions among PRs accepted and not (YU2016204, 34).
Therefore our work will focus on the use of all refactoring commits and not
only on PRs.

  * •

Study design: Quality Metrics vs Product and Process Metrics. Pantiuchina et
al. (10.1145/3408302, 22) focused on product-quality metrics such as Object
Oriented Programming (OOP) complexity, static analysis warnings, and
developer-related metrics. We decided to focus on established product and
process metrics as suggested by (kamei2012large, 14, 24) as well as the ones
introduced by Pantiuchina et al. (10.1145/3408302, 22).

  * •

Goals: Pantiuchina et al. (10.1145/3408302, 22) were interested in assessing
the correlation between product quality and developer-related metrics and the
occurrence of at least one refactor in a specific PR. However, although PR is
a richer container of developer thoughts and reflections (10.1145/3408302,
22), refactoring does not happen only in PR. Therefore, our study focuses on a
broader set of metrics analyzing qualitatively and quantitatively the
motivation that push developers to refactor to define a catalog of motivations
supported by a large number of metrics regardless to the connection of a
specific RP.

Paper Structure. In Section 2, we present the study design. Section 3 focuses
on threats to the validity of our study. Section 4 discusses related work and
in Section 5, we draw the conclusions.

##  2\. Study Design

This section outlines the empirical study, including the study goal and
research questions, the study context, the data collection methodology, and
the data analysis approach. Our empirical study follows established guidelines
defined by Wohlin et al. (Wohlin2000, 4). We will publish the raw data in the
replication package. Figure 1 provides a graphical description of the study
design of the presented plan.

###  2.1. Goal and Research Questions

The _goal_ of this study is to operationalize the motivations that drive
developers to perform refactoring activities, with the _purpose_ of
identifying product and process metrics that can grasp the motivations pushing
developers to perform refactoring. The _perspective_ is of researchers and
practitioners seeking additional support to guide developers in performing the
refactoring. The context is open source Java projects.

Therefore, we derived the following research questions (RQs):

RQ1. Do motivations for refactoring in past studies align with those found in
software project change histories?

Silva et al. (silva2016we, 28) proposed a list of motivations for refactoring
efforts provided by professional developers. Pantiuchina et al.
(10.1145/3408302, 22) build upon Silva et al. (silva2016we, 28) motivations
focusing on quality metrics and static analysis warnings extracting
information from PRs. Nonetheless, PRs represent developer interaction but can
not represent the full project change history (6976151, 33, 25). Moreover, PRs
can be rejected; thus, it is also challenging to discern expert contributions
among PRs accepted and not (YU2016204, 34). Hence, we investigate developer
motivation, analyzing the single commits with a broader selection of product
and process metrics. No prior study considered employing the fully established
product and process metrics (kamei2012large, 14, 24) with the refactoring
history.

Furthermore, developers’ commits, PRs, and code comments may not be able to
capture all the motivations pushing developers to refactor the code base;
hence, we ask:

RQ2. Are there additional motivations driving the developers’ willingness to
perform refactoring?

Refactoring improves source code quality (peruma2022refactor, 23).
Nonetheless, quality improvements in readability, performance, safety, and
security, are only some aspects of the bigger picture (7961515, 21). For
instance, in an industrial context, it is essential to assess refactor
opportunities in situations with limited resources regarding work allocation
and time (10.1007/978-3-319-09156-3_37, 30). Therefore, we investigate whether
external limitations or additional motivation influence refactoring efforts.

Finally, extending Pantiuchina et al., (10.1145/3408302, 22) metric
assessment, we ask:

RQ3. To what extent can product and process metrics capture the motivations
driving the developers’ willingness to perform refactoring?

Based on (10.1145/3408302, 22), a relationship exists between specific quality
metrics and refactoring operations. Nonetheless, (10.1145/3408302, 22) metric
selection was limited to quality metrics and static analysis warnings. We
extended their selection with established product and process metrics
(kamei2012large, 14, 24) thus broadening the candidate for a plausible
relationship to refactoring motivations. Therefore, we hypothesize that there
must be already existing metrics in the commit history of software projects
highly correlated to the ground truth motivations and the hypothetically newly
identified ones. The formulated hypotheses are defined as follows:

  * •

H0 There is no correlation between the considered metrics and refactoring
motivations.

  * •

H1 There is no correlation between the considered metrics and refactoring
motivations.

Consequently, considering _H 1_ as the accepted hypothesis and therefore the
evidence of correlation, we expand our hypotheses as follows:

  * •

H1.1 The identified correlation shows a negative direction.

  * •

H1.2 The identified correlation shows a positive direction.

In this sense, this question aims at identifying the metrics that can explain
the existence of the stated motivations , and further understand their
specific implications. And therefore provide a list of metrics to monitor and
guide developers for refactoring recommendations.

Figure 1. Graphical illustration of the presented study design. M: Metric, i:
Metric type, n: Total # of metrics. (*: Set of analyzed commits providing
refactoring motivations in the study from Silva et al. (silva2016we, 28)).

###  2.2. Study Context

Our study context is based on Silva et al. (silva2016we, 28) dataset. Silva et
al. (silva2016we, 28) initially selected the top 1000 Java repositories in
GitHub 111https://github.com, ordered in terms of popularity, and further
filtered it, removing the lower quartile based on the number of commits in
those projects, thus obtaining a list of 748 repositories with consistent
maintenance activities. The dataset considered in Silva et al. (silva2016we,
28) study was reduced to 471, filtering out those not in active development
while writing their paper. The final set of considered projects comprised 124
projects, which featured at least one refactoring during the study period and
an answer provided by the developers of the specific refactoring activity to
the author’s questions on the refactoring motivation. The final set of
projects comprised 222 commits, for which the authors provided answers
regarding their motivation for performing the refactoring. Within this set of
commits, 463 refactorings from 12 refactoring types were detected using
RefactoringMiner (tsantalis2013multidimensional, 31) tool. Therefore, to
validate the motivations provided by developers, our study considers the final
set of 124 GitHub-hosted Java projects described in the dataset
222http://aserg-ufmg.github.io/why-we-refactor published by Silva et al.
(silva2016we, 28).

###  2.3. Data Collection

In the first step of the data collection, we will gather all the metrics we
considered in our study presented in Table 3 from the 124 repositories of the
adopted dataset (silva2016we, 28). Although the dataset already provides the
True Positive commits with refactoring, we will run the RefactoringMiner
(tsantalis2013multidimensional, 31) tool to detect further refactoring
activity performed until now. We aim to follow the same collection procedure
Silva et al. (silva2016we, 28) performed for consistency in our results.
Therefore, we expect to identify commits in which the refactoring activity was
performed from the time the previous study was performed as well. Next, we
will collect the commit message of those commits in which refactoring activity
was detected and further collect their preceding commit. Thus, we aim to
capture the change in the code from the commit that preceded the refactoring
activity, the refactoring type itself, and the message that details the
performed change in the project’s version control history.

We will consider the motivations reported by Silva et al. (silva2016we, 28).
The list obtained from the motivations reported by developers consists of 44
motivations related to the 12 types considered in the cited study. Table 1
shows the list of refactoring types, their description, and the number of
motivations identified per refactoring. Moreover, Table 2 presents the label
or brief denomination of the motivations defined by human developers per each
of the selected refactoring types. We are not expanding the description
concerning motivations due to space constraints in this registered report.

To answer RQ3, we will mine and open code the variables that better describe
the resulting motivations obtained from the previous research questions. We
will provide further details of the process in Section 2.4.

Refactoring Type | Id | # Motivations |  Definition
---|---|---|---
EXTRACT METHOD | EM | 11 |  Takes a clump of code and turns it into its method.
MOVE CLASS | MC | 9 |  Move the class to its new folder on the source tree.
MOVE ATTRIBUTE | MA | 2 |  All attributes matching the selected attribute name on tags with the selected tag name may be moved inward toward a subtag of a given name.
RENAME PACKAGE | RP | 3 |  Renames the name of the selected project package.
MOVE METHOD | MM | 5 |  Creates a new method with a similar body in the class it uses most. Either turns the old method into a simple delegation or remove it altogether.
INLINE METHOD | IM | 3 |  Puts the method’s body into the body of its callers and removes the method.
PULL UP METHOD | PUM | 1 |  Moves methods with identical results on subclasses to the superclass.
PULL UP ATTRIBUTE | PUA | 1 |  Moves attributes with identical results on subclasses into the superclass.
EXTRACT SUPERCLASS | ES | 3 |  Creates a superclass and moves the common features to the superclass.
PUSH DOWN METHOD | PDM | 1 |  Given a field only used by some subclasses, it moves the field to those subclasses.
PUSH DOWN ATTRIBUTE | PDA | 2 |  Given an attribute only used by some subclasses, it moves the attribute to those subclasses.
EXTRACT INTERFACE | EI | 3 |  Given two classes having part of their interfaces in common, it extracts the subset into an interface.
Table 1. Definitions of the considered refactoring types (fowler1999refactoring, 11) Refactoring Type |  | Motivations |
---|---|---|---
EXTRACT METHOD |  Extract reusable method | Introduce alternative method signature |  Decompose method to improve readability
|  Facilitate extension | Remove duplication |  Replace Method preserving backward compatibility
|  Improve testability | Enable overriding |  Enable recursion
|  Introduce factory method | Introduce async operation |
MOVE CLASS |  Move class to appropriate container | Introduce sub-package |  Convert to top-level container
|  Remove inner classes from deprecated container | Remove from public API |  Convert to inner class
|  Eliminate dependencies | Eliminate redundant sub-package |  Backward compatibility
MOVE ATTRIBUTE |  Move attribute to appropriate class | Remove duplication |
RENAME PACKAGE |  Improve package name | Enforce naming consistency |  Move package to appropriate container
MOVE METHOD |  Move method to appropriate class | Move method to enable reuse |  Eliminate dependencies
|  Remove duplication | Enable overriding |
INLINE METHOD |  Eliminate unnecessary method | Caller becomes trivial |  Improve readability
PULL UP METHOD |  Move up common methods |  |
PULL UP ATTRIBUTE |  Move up common attributes |  |
EXTRACT SUPERCLASS |  Extract common state/behavior | Eliminate dependencies |  Decompose class
PUSH DOWN METHOD |  Specialized implementation |  |
PUSH DOWN ATTRIBUTE |  Specialized implementation | Eliminate dependencies |
EXTRACT INTERFACE |  Facilitate extension | Enable dependency injection |  Eliminate dependencies
Table 2. Motivations derived from developers’ answers based on their
refactoring activity

###  2.4. Data Analysis

#### Refactoring motivation identification through refactoring commit activity
(RQ1)

We are interested in grasping the motivations that led the developers to
perform the selected refactoring types and further compare if these
motivations are related to the ones resulting from the previous work. We plan
to perform a multi-class semi-supervised classification of the collected
refactoring commits based on the identified motivations to achieve this. The
classification process will be rooted in the use of different LLMs; we plan to
use the APIs of LLMs such as chatGPT3.5 or
chatGPT4333https://openai.com/product following the state-of-the-art
(chang2023survey, 3) along with further models such as Mistral
7B444https://www.lemonfox.ai/mistral-api for instance, which has shown
promising performance as well (jiang2023mistral, 13). Since we already have
supervised answers for a set of 222 commits with their respective 463
refactorings from the study context, we will set our LLMs to learn with this
supervised training data, following the learning paradigm called Few-Shot
Learning (FSL) (wang2020generalizing, 32, 9, 10). This initial step will
provide the commit change between the commit in which the refactoring has been
implemented and its preceding commit, the commit message, and the refactoring
type and motivation involved in each case from the supervised data. Thus, the
LLMs will be initially provided with the motivations and refactoring types
detailed in Tables 1 and 2 as the baseline framework for the classification
approach. Then, to analyze the motivation behind each of the collected
refactoring commits, we will fit the LLMs with the commit change between the
refactoring commit and the preceding commit. Additionally, we will provide the
message provided in both commits. We will consider fitting the models only
with the described content and a defined explanatory question. We will
consider further motivations LLMs suggest when discrepancies are found between
the provided refactoring cases and the existing refactoring motivations since
we are also interested in discovering potential motivations behind the
refactoring activity that may not have been addressed in previous research.
Thus, we aim to analyze possible deviations from the developers’ provided
motivations in the real code.

#### Comparison of the resulting motivations (RQ2)

We are interested in analyzing developers’ accuracy when providing their main
motivations to perform a refactoring activity, considering the motivations
that the LLMs have extracted from the provided commit changes and messages.
Therefore, in this analysis stage, we will compare the frequency difference
between the reported AI-based motivations and those obtained in (silva2016we,
28) for each of the considered refactoring types. Similarly, since we have
allowed the LLMs to provide new motivations when finding discrepancies between
the existing motivations and the refactoring case, we want to compare the
difference between the initially considered motivations and those that LLMs
relate to for the same refactoring type.

#### Capturing motivation metrics in version control history (RQ3)

To answer our third research question, the current section of the data
analysis aims to capture the software metrics existing in the considered
projects that better describe the scenario that motivates the developer to
perform the refactoring activity and further test their relationship through
the defined hypotheses in Section 2.1.

To accomplish that, we will initially open code the selected software projects
and collect software metrics when each refactoring was performed at the
project stage. We will consider the software metrics that are, by definition,
related to the motivations described both by developers (silva2016we, 28) and
the ones reported by the LLMs.

| Metric |  Description
---|---|---
Rahman (rahm2013, 24) | COMM |  The cumulative number of changes in a given file up to the considered commit.
ADEV |  The cumulative number of active developers who modified a given file up to the considered commit.
DDEV |  The cumulative number of distinct developers contributed to a given file up to the considered commit.
ADD |  The normalized number of lines added to a given file in the considered commit.
DEL |  The normalized number of lines removed from a given file in the considered commit.
OWN |  The value indicates whether the file owner does the commit.
MINOR |  The number of contributors who contributed less than 5% of a given file up to the considered commit.
SCTR |  The number of packages modified by the committer in the considered commit.
NADEV |  The number of active developers who changed any of the files involved in the commits where the given file has been modified.
NDDEV |  The number of distinct developers who changed any of the files involved in the commits where the given file has been modified.
NCOMM |  The number of commits where the given has been involved.
NSCTR |  The number of different packages touched by the developer in commits where the file has been modified.
OEXP |  The percentage of code lines authored by a given developer in the project.
EXP |  The mean of the experience of all developers across the project.
Kamei (kamei2012large, 14) | ND |  The number of directories involved in a commit.
NS |  Number of modified subsystems.
NF |  Number of modified files.
ENTROPY |  The distribution of the modified code across each given file in the considered commit.
LA |  Ten lines added to the given file in the considered commit (absolute number of the ADD metric).
LD |  The number of lines removed from the given file in the considered commit (absolute number of the DEL metric).
LT |  The number of lines of code in the given file in the considered commit before the change.
FIX |  Whether or not the change is a defect fix.
NDEV |  The number of developers that changed the modified files.
AGE |  The average period between the last and the current change.
NUC |  The number of times the file has been modified up to considered commit.
CEXP |  The number of commits performed on the given file by the committer up to the considered commit.
REXP |  The number of commits performed on the given file by the committer in the last month.
SEXP |  The number of commits a given developer performs in the considered package containing the given file.
Pantiuchina (10.1145/3408302, 22) | CBO |  Coupling Between Object classes: measures the dependencies a class has.
WMC |  Weighted Methods per Class: sums the cyclomatic complexity of the methods in a class.
RFC |  Response For a Class: the number of methods in a class plus the number of remote methods that are called recursively [citeme].
ELOC |  Effective Lines Of Code: the lines of code excluding blank lines and comments.
NOM |  Number Of Methods in a class.
NOPM |  Number Of Public Methods in a class.
DIT |  Depth of Inheritance Tree: the length of the path from a class to its farthest ancestor.
NOC |  Number Of Children (direct subclasses) of a class.
NOF |  Number Of Fields declared in a class.
NOSF |  Number Of Static Fields declared in a class.
NOPF |  Number Of Public Fields declared in a class.
NOSM |  Number Of Static Methods in a class.
NOSI |  Number Of Static Invocations of a class.
HsLCOM |  Henderson-Sellers revised Lack of Cohesion Of Methods (LCOM): a class cohesion metric based on sharing local instance variables by the class methods.
C3 |  Conceptual Cohesion of Classes: avg. textual similarity between all pairs of methods in a class.
StrRead |  Structural readability: uses structural aspects (e.g., line length) to model code readability.
ComRead |  Comprehensive readability model: combines structural, visual (e.g., alignment), and textual features (e.g., comments readability).
Table 3. Product and process metrics adopted in this study.

As initial potential metrics, we will consider diverse software metrics such
as process metrics defined by Rahman & Devanbu (rahm2013, 24) as well as
product metrics defined by Kamei et al. (kamei2012large, 14). Similarly, we
will consider additional metrics used in previous work by Pantiuchina et al.
(10.1145/3408302, 22). Then, we will rank the selected metrics by how related
or informative they are toward the identified motivations. For that, we will
consider two main different approaches. The first approach will consist of
quantifying the feature importance (sammut2011encyclopedia, 27) provided by
each of the considered metrics in explaining each refactoring motivation
accordingly. Feature importance measurement stands as a feature selection
technique commonly used in previous research, as demonstrated by Rajbahadur et
al. (9347823, 26), and can be calculated in different ways depending on the
model chosen, e.g., Random Forest or Extreme Gradient Boost, for instance. To
validate our results, we will consider a minimum of two models to calculate
the importance level. Similarly, to statistically validate the obtained
results from the approach below, we will statistically test the correlation
level of the software metrics toward each of the refactoring motivations and
thus test the hypotheses defined previously in Section 2.1. Since we do not
know a priori the exact distribution of the considered metrics, we will use
the parametric test Shapiro-Wilk (gentle2009computational, 12) and the non-
parametric Kolmorov-Smirnov (DeGroot2019, 7) test to test the normality
distribution assumption. Based on the obtained results we will consider using
parametric tests such as Pearson’s r correlation coefficient
(cohen2009pearson, 5) or non-parametric tests such as Spearman’s $\rho$ or
Kendall’s $\tau$ (Wohlin2000, 4) coefficient for instance. The proposed tests
provide diverse ranked test statistic values to measure the extent of
correlation of the tested metrics and the statistical significance of the
obtained statistic in terms of the well-known p-value. Since we are
implementing multiple tests in multiple variables simultaneously, which can
increase the likelihood of obtaining false statistically significant outcomes,
we will also perform the Bonferroni post-hoc test (DeGroot2019, 7) to adjust
the obtained p-values for the test statistics to the multiple comparison
scenario, and thus confirm our formulated hypotheses.

##  3\. Threats to Validity

Construct Validity. We acknowledge that using LLMs for _semi supervised multi-
class_ classification may stand as a threat to the results of this study as it
assumes to be correct the artificial intelligence-based feedback. However, to
minimize this threat we consider using state-of-the-art as well as well-valued
LLMs such as chatGPT and Mistral 7b for instance. Similarly, Silva et al
(silva2016we, 28) already considered the threat of using RefactoringMiner for
mining the commits in which the refactorings were performed as some
classifications may be missed. However, we still consider using
RefactoringMiner to provide consistency to our results as one of our goals is
to validate the ground truth motivations provided by developers.

Internal Validity. One of the main outcomes of this study will be the
publication of a set of metrics to guide developers when performing
refactoring activity. These metrics will be based on the human-based
motivations as well as AI-based motivations obtained from the performed
classification, hence we understand as a threat the fact that the defined
refactoring guideline metrics may not provide an accurate instruction to
commit refactoring. Missing motivations from the developers as well as
potential motivations yet to be discovered by analyzing further projects may
contribute to the possible lack of accuracy. However, we understand that the
motivations provided by human developers (silva2016we, 28) provide the closest
feedback to the real motivations for committing refactoring, and therefore
reduce this type of threat.

External Validity. This study only considers open source projects, based on
Java programming languages and hosted by GitHub. Therefore, the results of
this study, even though they aim to cover a wide range of projects, cannot
claim to apply to systems outside the open-source community or projects
developed in a different programming language. However, the presented study
plan aims to analyze all the existing refactoring activity of 124 projects for
12 refactoring types which, if the results provide a clear guideline of
metrics, would provide one of the most consistent recommendation guidelines
for refactoring activity in the field.

Reliability. The planned data analysis is presented in a format that aims at
providing answers to both the refactoring cases in which the motivation is
identifiable as well as when discrepancies may exist and therefore already
considered motivations may not fit for the refactoring case. Similarly, the
choice of statistical tests detailed in this article aims at covering the
possible statistical assumptions that data differently distributed may require
for performing statistical testing. The source data provides a considerable
set of GitHub-hosted projects, yet a small portion of samples given the
existing dimension of projects published in the mentioned platform, therefore
we assume as a threat the hypothetical differences in results if the same
analysis were implemented in a different sample of projects. However, the
validation of the developer-based motivations through the results of this
study would minimize the presented threat as we understand that the
motivations presented by developers are not the result of their performance in
the analyzed projects but the result of their entire development career and
hence the product of a larger number of projects.

##  4\. Related Work

Software refactoring can improve different aspects of software quality, such
as readability and maintainability (8456513, 20). Anyhow, refactoring is also
a time-consuming task and, as such, should be correctly prioritized
(10.1145/3387940.3392191, 17). Therefore, to match practitioners’ needs, we
must narrow the refactoring scope via recommendations (8477161, 1). (8456513,
20) investigated software refactoring, introducing a novel approach that
evaluates refactoring solutions based on traceability and source code smell
using entropy-based and traditional metrics, respectively. The authors found
that their approach outperformed traditional metric-based recommendations in
71% of the examined cases. (kaur2021brief, 15) introduced a multi-objective
optimization technique to generate refactoring solutions that maximize the
software quality, use of smell severity, and consistency with class
importance. They addressed the limitations of existing approaches, which focus
solely on software enhancing traditional software quality. Using a multi-
objective spotted hyena optimizer (MOSHO), they evaluated refactoring
solutions on five open-source datasets. The authors showed MOSHO’s
effectiveness in leveraging class importance and smell severity scores.

(10.1145/3387940.3392191, 17) proposed a novel approach for recommending Move
Method refactoring, mainly useful when methods rely heavily on external class
members. Existing approaches, often heuristic-based, suffer from limitations
such as metric dependency and manual threshold setting. On the other hand, the
proposed approach leverages code2vec, a path-based code representation
capturing syntactic and semantic information, to train a machine learning
classifier. The empirical study on open-source projects and a synthetic
dataset showed that the proposed approach outperformed existing tools like
JDeodorant and JMove.

Similarly, (8477161, 1) proposed a dynamic and interactive refactoring
recommendation approach to address the challenges of managing software
complexity and facilitating software evolution. The proposed approach
leverages NSGA-II (DEBUCK2019613, 6) to suggest refactorings while considering
developer feedback, aiming to improve software quality while minimizing
deviation from the initial design. They evaluated the approach on open-source
and industrial projects, demonstrating its superiority over existing search-
based techniques and fully automated tools. Our work does not focus only on
the mere refactoring effort but also on the developer’s perspective of
refactoring; more specifically, we investigate the motivation that leads
developers to refactor specific portions of the code base.

Similarly, (10.1145/3408302, 22) performed a large-scale study to understand
why developers refactor code in open-source projects, complementing previous
findings from developer surveys. They analyzed 287,813 refactoring operations
across 150 systems, examining the relationship between refactoring operations
and process/product metrics. The authors highlighted that there exists a
relationship between metrics and refactoring operations, along with a detailed
taxonomy of refactoring motivations. Our study differs from Pantiuchina et al.
(10.1145/3408302, 22) approach in study design and goals. While
(10.1145/3408302, 22) relied on keyword lookup and the RMiner tool for PR
analysis, we propose a novel methodology employing Language Model analysis.
Moreover, while (10.1145/3408302, 22) focused on product quality and
developer-related metrics, our study prioritizes established product and
process metrics. Lastly, while the goal was to correlate product quality and
developer metrics with refactoring in PRs, we aim to comprehensively analyze
motivations for refactoring across various contexts, defining a catalog of
motivations supported by a broader selection of metrics from established
guidelines.

##  5\. Conclusion

We aim to search for existing metrics to guide practitioners with developer-
based refactoring recommendations. In this sense, this study will first
validate ground truth motivations driving refactorings provided by developers
and similarly will identify further potential motivations through the use of
LLMs by analyzing multiple refactoring commits and their messages. The second
but not least important outcome of this study will be the identification of
the most informative metrics for the concerning motivations. This study will
validate the list of motivations that drive developers to perform the
considered refactoring types and further will add a valuable contribution to
the existing body of knowledge in refactoring research, as it will provide a
set of metrics directed to practitioners to guide them when performing
refactoring in their projects. Moreover, with the obtained metrics we aim to
study time-dependent estimation of refactoring proneness, thus providing a
concept-temporal refactoring recommendation to practitioners.

## References

  * (1) Vahid Alizadeh et al.  “An Interactive and Dynamic Search-Based Approach to Software Refactoring Recommendations”  In _IEEE Transactions on Software Engineering_ 46.9, 2020, pp. 932–961  DOI: 10.1109/TSE.2018.2872711
  * (2) Paris C Avgeriou et al.  “An overview and comparison of technical debt measurement tools”  In _Ieee software_ 38.3 IEEE, 2020, pp. 61–71
  * (3) Yupeng Chang et al.  “A survey on evaluation of large language models”  In _ACM Transactions on Intelligent Systems and Technology_ ACM New York, NY, 2023
  * (4) Claes et al.  “Experimentation in Software Engineering”  Springer, 2012
  * (5) Israel Cohen et al.  “Pearson correlation coefficient”  In _Noise reduction in speech processing_ Springer, 2009, pp. 1–4
  * (6) Viviane De Buck et al.  “Multi-objective optimisation of chemical processes via improved genetic algorithms: A novel trade-off and termination criterion”  In _29th European Symposium on Computer Aided Process Engineering_ 46, Computer Aided Chemical Engineering  Elsevier, 2019, pp. 613–618  DOI: https://doi.org/10.1016/B978-0-12-818634-3.50103-X
  * (7) Morris DeGroot and Mark Schervish  “Probability and Statistics”  Upper Saddle River, NJ: Pearson, 2019
  * (8) Matteo Esposito and Davide Falessi  “Uncovering the Hidden Risks: The Importance of Predicting Bugginess in Untouched Methods”  In _2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , 2023, pp. 277–282  IEEE
  * (9) Li Fei-Fei, Robert Fergus and Pietro Perona  “One-shot learning of object categories”  In _IEEE transactions on pattern analysis and machine intelligence_ 28.4 IEEE, 2006, pp. 594–611
  * (10) Michael Fink  “Object classification from a single example utilizing class relevance metrics”  In _Advances in neural information processing systems_ 17, 2004
  * (11) Martin Fowler  “Refactoring: Improving the Design of Existing Code”  Addison-Wesley Professional, 1999
  * (12) James E Gentle  “Computational statistics”  Springer, 2009
  * (13) Albert Q Jiang et al.  “Mistral 7B”  In _arXiv preprint arXiv:2310.06825_ , 2023
  * (14) Yasutaka Kamei et al.  “A large-scale empirical study of just-in-time quality assurance”  In _IEEE Transactions on Software Engineering_ 39.6 IEEE, 2012, pp. 757–773
  * (15) Satnam Kaur, Lalit K Awasthi and AL Sangal  “A brief review on multi-objective software refactoring and a new method for its recommendation”  In _Archives of Computational Methods in Engineering_ 28 Springer, 2021, pp. 3087–3111
  * (16) Miryung Kim, Thomas Zimmermann and Nachiappan Nagappan  “A field study of refactoring challenges and benefits”  In _Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering_ , FSE ’12  Cary, North Carolina: Association for Computing Machinery, 2012  DOI: 10.1145/2393596.2393655
  * (17) Zarina Kurbatova, Ivan Veselov, Yaroslav Golubev and Timofey Bryksin  “Recommendation of Move Method Refactoring Using Path-Based Representation of Code”  In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ , ICSEW’20  Seoul, Republic of Korea: Association for Computing Machinery, 2020, pp. 315–322  DOI: 10.1145/3387940.3392191
  * (18) Guilherme Lacerda, Fabio Petrillo, Marcelo Pimenta and Yann Gaël Guéhéneuc  “Code smells and refactoring: A tertiary systematic review of challenges and observations”  In _Journal of Systems and Software_ 167, 2020, pp. 110610  DOI: https://doi.org/10.1016/j.jss.2020.110610
  * (19) Valentina Lenarduzzi et al.  “A systematic literature review on technical debt prioritization: Strategies, processes, factors, and tools”  In _Journal of Systems and Software_ 171 Elsevier, 2021, pp. 110827
  * (20) Ally S. Nyamawe et al.  “Recommending Refactoring Solutions Based on Traceability and Code Metrics”  In _IEEE Access_ 6, 2018, pp. 49460–49475  DOI: 10.1109/ACCESS.2018.2868990
  * (21) Fabio Palomba, Andy Zaidman, Rocco Oliveto and Andrea De Lucia  “An Exploratory Study on the Relationship between Changes and Refactoring”  In _2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC)_ , 2017, pp. 176–185  DOI: 10.1109/ICPC.2017.38
  * (22) Jevgenija Pantiuchina et al.  “Why Developers Refactor Source Code: A Mining-based Study”  In _ACM Trans. Softw. Eng. Methodol._ 29.4 New York, NY, USA: Association for Computing Machinery, 2020  DOI: 10.1145/3408302
  * (23) Anthony Peruma et al.  “How do i refactor this? An empirical study on refactoring trends and topics in Stack Overflow”  In _Empirical Software Engineering_ 27.1 Springer, 2022, pp. 11
  * (24) Foyzur Rahman and Premkumar Devanbu  “How, and why, process metrics are better”  In _International Conference on Software Engineering_ , 2013, pp. 432–441  IEEE Press
  * (25) Mohammad Masudur Rahman and Chanchal K. Roy  “An insight into the pull requests of GitHub”  In _Proceedings of the 11th Working Conference on Mining Software Repositories_ , MSR 2014  Hyderabad, India: Association for Computing Machinery, 2014, pp. 364–367  DOI: 10.1145/2597073.2597121
  * (26) Gopi Krishnan Rajbahadur et al.  “The Impact of Feature Importance Methods on the Interpretation of Defect Classifiers”  In _IEEE Transactions on Software Engineering_ 48.7, 2022, pp. 2245–2261  DOI: 10.1109/TSE.2021.3056941
  * (27) Claude Sammut and Geoffrey I Webb  “Encyclopedia of machine learning”  Springer Science & Business Media, 2011
  * (28) Danilo Silva, Nikolaos Tsantalis and Marco Tulio Valente  “Why we refactor? confessions of github contributors”  In _Proceedings of the 2016 24th acm sigsoft international symposium on foundations of software engineering_ , 2016, pp. 858–870
  * (29) Gábor Szóke et al.  “Bulk Fixing Coding Issues and Its Effects on Software Quality: Is It Worth Refactoring?”  In _2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation_ , 2014, pp. 95–104  DOI: 10.1109/SCAM.2014.18
  * (30) Gábor Szőke, Csaba Nagy, Rudolf Ferenc and Tibor Gyimóthy  “A Case Study of Refactoring Large-Scale Industrial Systems to Efficiently Improve Source Code Quality”  In _Computational Science and Its Applications – ICCSA 2014_ Cham: Springer International Publishing, 2014, pp. 524–540
  * (31) Nikolaos Tsantalis, Victor Guana, Eleni Stroulia and Abram Hindle  “A multidimensional empirical study on refactoring activity.”  In _CASCON_ , 2013, pp. 132–146
  * (32) Yaqing Wang, Quanming Yao, James T Kwok and Lionel M Ni  “Generalizing from a few examples: A survey on few-shot learning”  In _ACM computing surveys (csur)_ 53.3 ACM New York, NY, USA, 2020, pp. 1–34
  * (33) Yue Yu, Huaimin Wang, Gang Yin and Charles X. Ling  “Reviewer Recommender of Pull-Requests in GitHub”  In _2014 IEEE International Conference on Software Maintenance and Evolution_ , 2014, pp. 609–612  DOI: 10.1109/ICSME.2014.107
  * (34) Yue Yu, Huaimin Wang, Gang Yin and Tao Wang  “Reviewer recommendation for pull-requests in GitHub: What can we learn from code review and bug assignment?”  In _Information and Software Technology_ 74, 2016, pp. 204–218  DOI: https://doi.org/10.1016/j.infsof.2016.01.004