# PACE: A Program Analysis Framework for Continuous Performance Prediction

Chidera Biringa<EMAIL_ADDRESS>0000-0001-5904-2764 University of
Massachusetts Dartmouth285 Old Westport Rd, North Dartmouth,
MADartmouthMassachusettsUSA02747 and  Gökhan Kul<EMAIL_ADDRESS>0000-0001-6467-1979 University of Massachusetts Dartmouth285 Old Westport Rd,
North Dartmouth, MADartmouthMassachusettsUSA02747

(20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Software development teams establish elaborate continuous integration
pipelines containing automated test cases to accelerate the development
process of software. Automated tests help to verify the correctness of code
modifications decreasing the response time to changing requirements. However,
when the software teams do not track the performance impact of pending
modifications, they may need to spend considerable time refactoring existing
code. This paper presents PACE, a program analysis framework that provides
continuous feedback on the performance impact of pending code updates. We
design performance microbenchmarks by mapping the execution time of functional
test cases given a code update. We map microbenchmarks to code stylometry
features and feed them to predictors for performance predictions. Our
experiments achieved significant performance in predicting code performance,
outperforming current state-of-the-art by 75% on neural-represented code
stylometry features.

††ccs: Software Performance Prediction Current Code State††ccs: Feature
Engineering Code Stylometry Features††ccs: Automated Testing
Microbenchmarking††ccs: Machine Learning Regression

##  1\. Introduction

Software development teams often adopt agile practices and implement automated
testing and integration architecture to ensure that implemented software
requirements work as anticipated. However, this approach often excludes the
performance implications of specific methods and their aptitude to increase
the overall software execution time. If ignored, the quality of user
interaction with software functionalities may diminish due to poor
performance, especially when automated test scripts do not maintain a record
of adjustments made to software features, making it strenuous to identify the
changes that cause performance issues.

We polled 34 software developers working at 28 organizations, varying from
software companies to federal agencies with in-house software development
units. Twenty-nine developers reported that their software development
environments run on continuous integration (CI) pipelines. CI (Meyer, 2014) is
a process in agile software development to automatically test, verify, and
integrate local software updates to the remote or production version of the
software. Pipelines facilitate continuous integration by serializing
algorithms and methods to achieve the objective of continually merging local
changes to the primary software. 100% of these CI pipelines in the poll
utilize automated tests. However, some software tests to measure performance
are not easy to automate, straining the test team’s ability to ensure the
software continues to deliver the intended functionality in an acceptable time
frame during periods when the software changes rapidly. Therefore, automation
of performance tests is crucial to provide the ideal user experience.

The same poll indicated that 88% of these software developers believe it is
necessary to automate performance-focused testing. To stress the importance of
performance in software, Pinterest (Pavic et al., 2019) increased its traffic
and sign-ups by 15% by reducing its latency by 40%. Mobify (Pavic et al.,
2019) found that a 100ms decrease in homepage load time increased their annual
revenue by $380,000 and that a similar reduction in checkout page load time
increased their revenue by $530,000. Moreover, BBC (Pavic et al., 2019)
discovered they lose 10% traffic for every additional second their site takes
to load. Therefore, companies started to invest in performance testing to
enhance their economic competitiveness.

Traditional software performance estimation methods are classified broadly
under Model-based (Cortellessa and Mirandola, 2000; Menascé and Gomaa, 2000;
Kähkipuro, 2000; Bernardi et al., 2002; de Miguel et al., 2000; Lindemann et
al., 2002) and Trace-based (Andolfi et al., 2000; Aquilani et al., 2001;
Petriu and Woodside, 2002; Woodside et al., 2001) categories. Model-based
methods are commonly abstract and estimate software performance by developing
models based on specific software, hardware resources, and the utilization of
application features and functionalities. Some of these models include (i)
queueing network-based (Cortellessa and Mirandola, 2000), (ii) architectural
pattern (Menascé and Gomaa, 2000), (iii) performance-driven unified modeling
language (Kähkipuro, 2000), (iv) process-algebra-driven (Bernardi et al.,
2002), (v) simulation-driven (de Miguel et al., 2000), and (vi) stochastic-
driven (Lindemann et al., 2002). Trace-based methods are aided by an
interactive outline of software activities for performance estimations.
Performance is obtained and logged, and then these values are used to perform
estimations on the equivalent software with dissimilar hardware environments.
Some of these methods include (i) Regression and Filtering (Lee et al., 2003),
(ii) Queueing Networks-Labeled Transition Systems (Aquilani et al., 2001),
(iii) Queueing Networks-Message Sequence Chart (Andolfi et al., 2000), and
(iv) Layered Queueing Network-Use Case Maps (Petriu and Woodside, 2002).

Techniques and methodologies adopted in model and trace-based methods vary
significantly, and their applicability is predominantly contingent on software
type and hardware components. Furthermore, the methods require a high setup
time and effort. For example, one of the earliest software performance
engineering methods (Williams and Smith, 2002) based on Queueing Networks (QN)
requires the implementation to begin from the earliest through the latest
stages of the development process. It also needs the application of Execution
Graphs and QN that replicate software behavior and hardware resources. On the
other hand, we facilitate the performance prediction during software
development by embedding our approach into Git with no additional code on the
software being developed or conversely no preliminary information about the
software communicated into the model.

Our literature review showed that model and trace-based methods are commonly
integrated into software design to detect performance degradation. However,
the application of these models requires developers to be knowledgeable about
the behavioral, communication, and architectural variables of the software
(Zhang et al., 2015). Hence, significant changes to the software can
potentially lead to discrepancies between the inferred and actual performance
of the model.

This paper proposes PACE, a Program Analysis Framework for Continuous
Performance PrEdiction. PACE adopts a comprehensive domain-oriented feature
engineering and machine learning (ML) — a subset of artificial intelligence
tasked with discovering insightful patterns from large amounts of data and
algorithms (Sarker, 2021) — approaches to learn and make predictions on the
execution times of automated test cases given an update to the software. PACE
fragments distributed version-controlled software repositories into branches
deployed to a testing framework, where embedded functional test cases are
executed and corresponding test times logged.

Concretely, the contributions of this paper are as follows:

  * •

We introduce a program analysis framework for continuous performance
prediction by mapping code stylometry features to the execution test times.

  * •

We propose a feature engineering technique for software performance
prediction.

  * •

We propose adopting the execution time of automated test cases as a
microbenchmark for software performance.

PACE addresses the following research questions:

Research Question 1 (RQ1): Automated performance software testing in CI
pipelines can be exceedingly time-consuming and require the continuous
execution of large code segments. Traditional performance estimation methods
such as stochastic process (Lindemann et al., 2002), process algebra (Bernardi
et al., 2002) and queuing networks (Andolfi et al., 2000; Aquilani et al.,
2001; Petriu and Woodside, 2002) provide a means of achieving the same goal
with fewer computational resource expenditure. However, it is a complex task,
most of which falls on the shoulders of the development team. ML has been
applied most recently in software performance prediction research efforts
(Meng and Norris, 2017; Ramadan et al., 2021; Samoaa et al., 2022; Zhou et
al., 2019; Liu et al., 2021; Didona et al., 2015; Guo et al., 2018;
Kaltenecker et al., 2020; Velez et al., 2021) with varying degrees predictive
accuracy. Hence, what is the predictive prowess of software performance given
a code update?

We answered this question by creating microbenchmarks of code performance
through executing available functional test cases in the software
($\mathit{y}$), after which we processed to select and transform code
performance features ($\mathbb{X}$). Finally, we feed the test times and
vector features to a regression function ($\phi$) tasked with predicting the
software performance given an update $\hat{y}=f_{\phi}(x^{\prime})$. We
experimented with several regression algorithms and found that a k-nearest
neighbors regressor model performs the best in predictive accuracy and
latency. PACE-kNN is the best-performing model, obtaining a (0.0763 $\in$
$\mathbb{D}$ (first case study) and 0.7366 on $\mathbb{D}^{\prime}$) (second
case study) average error rate ($\mu$(RMMR)) on experimental datasets.
($\mu$(RMMR)) denotes the average result of RMSE, MSE, MAE, and RMSLE, see
Section 5.1 for the description of aforementioned metrics.

Research Question 2 (RQ2): What is the delta performance impact of commit at
(cn-1) given a (cn) code update?

We answered this question by further analyzing the generated results from RQ1
to ascertain the delta performance of commits. $\mathbb{D}$ has two positive
and a single negative performance impact. $\mathbb{D}^{\prime}$ has a total of
twenty-four positive and negative performance impacts respectively.

Research Question 2 (RQ3): An indispensable feature of building PACE’s
predictive models is the selection of code features and its consequential
numerical statistic and distribution semantic transformation. Given that the
primary goal of this work is continuously predicting code performance, it is
pertinent that we do not introduce a significant performance overhead during
the feature extraction pipeline. Hence, what is the cost of selecting and
representing code stylometry features?

We answered this question and the concern of cost in throughput and latency by
timing the selection, statistic (SR), and neural representation (NR) of code-
stylometry features. The average selection throughput of aforementioned
features takes (0.2297 $\in$ $\mathbb{D}$ and 23.3201 $\mathbb{D}^{\prime}$)
seconds (42 $\in$ $\mathbb{D}$ and 2543 $\in$ $\mathbb{D}^{\prime}$) $\mu$
number of source files for our experimental datasets (see Section 5.2).
Statistic representation (SR) throughput takes (0.0002 $\in$ $\mathbb{D}$,
0.1269 $\in$ $\mathbb{D}^{\prime}$) average seconds, while neural
representation takes (0.0055 $\in$ $\mathbb{D}$, 0.0588 $\in$
$\mathbb{D}^{\prime}$) average seconds. SR has a lower representation
throughput compared to NR, outperforming it by (27x $\in$ $\mathbb{D}$ and 10x
$\in$ $\mathbb{D}^{\prime}$). However, there is no significant difference in
predictive performance between them when fed to predictors for performance
prediction.

Research Question 4 (RQ4): How does PACE perform in comparison with the state-
of-the-art approach?

We answered this question by comparing the predictive performance of PACE to
TEP-GNN (Samoaa et al., 2022) — a deep learning approach towards performance
prediction based on graphical convolutional neural networks. Our best
predictor PACE-kNN attained a (0.0083, 0.0077) MSE error rate on the entire
dataset (H2, RDF4J, Dubbo, SystemDS), and (0.0101, 0.0147) on average for
independent projects, outperforming TEP-GNN (0.0170) (0.0202) by (68%, 75%)
and (66%, 31%) $\in$ (SR, NR) features respectively.

Paper Outline. We organized the rest of this paper as follows. Section 2
defines the problem, and Section 3 details the necessary background for this
work. Section 4 details the PACE methodology. We describe the experimental
setup and evaluation of PACE in Section 5. In Section 6, we comprehensively
answer research questions. Section 7 describes the PACE’s scope and threats.
In Section 8, we discuss research works done in the software performance
prediction literature. Finally, we conclude in Section 9.

##  2\. Problem Definition: Unoptimized Code

We evaluate the performance impact of a snippet of source code by analyzing
the time it takes to execute automated test cases. Execution time is a
reliable metric on how users perceive software performance (Sherief, 2014). In
this paper, a code snippet or program is mediocre if it introduces a
significant performance overhead to software and consequently skews the
baseline resulting in an outlier performance. For example, Figure 1 is a code
snippet executed on an Apple commodity laptop with an Apple M1 8-core CPU,
16-core Neural Engine, 8 GB RAM, and macOS Big Sur (version 11).

The snippet is a real-world fragment of a Java program that calculates the
term frequency of more than 2,000 elements in a linked hashmap (LHM). An LHM
is a data structure that combines a hash table and a linked list. It ensures
predictable maintenance of elements in an iterable object (Goodrich et al.,
2014). The peripheral difference between the snippets is in how the linked
hashmap is accessed and manipulated using an entrySet(), a set representation
of key,value pairs, and keySet(), a set representation of keys. For both set
representations, element changes in the map are reflected in the set and
conversely.

Figure 1. Access and manipulation of a Linked-HashMap (LHM) using an Entry-set
and Key-set. ESA and ESM denote Entry-set Access and Manipulation. KSA and KSM
represent Key-set Access and Manipulation. ESA and KSA define a call to the
sets (entry and key) view of the map, while ESM and KSM imply the
modifications of elements in the map. Lines of code highlighted in blue and
grey are LHM access and manipulation using Entry-set and Key-set.

The test execution of (ESA, ESM), and (KSA, KSM) versions of the code snippet
takes $\sim$4 seconds and $\sim$4 minutes respectively, indicating a 6000%
increase in execution time. This time difference decreases by $\sim$2 minutes,
a 3000%, given an increase in the number of elements from 2,000 to 200,000.
The above-stated performance discrepancy might appear inconsequential for
isolated occurrences. However, in scenarios where the low-performing program
is called in multiple areas of software, it would increase the completion time
of an operation. Furthermore, the KSA and KSM versions of the program, which
are inefficient, are also syntactically correct and contain no apparent bugs,
highlighting the problem’s difficulty, such as employing erroneous data
structure selection and manipulation and selecting $O(2^{n})$ time algorithms
when there are viable polynomial options. Hence, in this paper, we approach
this problem by mining and learning the features of low-performing programs
and feeding those features to predictive models to detect their performance.

##  3\. Background

###  3.1. Static Program Analysis

Static program analysis is the automated examination of programming code
without its execution. Researchers have shown the importance of employing
program analysis in solving code-dependant problems (Louridas, 2006) such as
de-anonymizing of programmers (Caliskan-Islam et al., 2015), authorship
identification (Dauber et al., 2018), document representation and
summarization (Abuhamad et al., 2018), software defect prediction (Li et al.,
2017), plagiarism detection (Lancaster and Culwin, 2004; Cui et al., 2010),
and software forensics (MacDonell et al., 1999), and SQL query intent analysis
(Kul et al., 2020) Program analysis using regular grammar is complex and
incurs non-trivial performance overhead (Karper, 2014). Hence, researchers
commonly adopt an abstract syntax tree-enabled approach (AST) to program
analysis.

An AST of a source code creates a semantic representation of the code (Neamtiu
et al., 2005; Zhang et al., 2019). It captures a source code’s implicit
syntactic and lexical characteristics. ASTs are generated using a lexical
analyzer that processes the source code and creates lexical tokens. The
compiler passes these tokens to a syntactic analyzer and generates an AST.
Finally, generated trees are converted to machine code and executed.

###  3.2. Microbenchmarking

Software performance microbenchmarking is a standardized procedure to
experimentally analyze the execution time of non-functional components of the
software, such as code snippets. In contrast, spike, configuration,
scalability, load, and stress tests assess the performance of the entire
system using the benchmark characteristics of specific features and components
(Hooda and Chhillar, 2015). For instance, packages such as Java Microbenchmark
Harness (JMH) (Costa et al., 2019) measure the performance of classes and
functions in distributed and non-distributed computing environments.

Although the above-stated programming language-based solution bridges a
performance analysis gap in software development, it also introduces a
bottleneck in implementation. For example, Costa et al. (Costa et al., 2019)
detailed the negative performance effect on benchmarking measures caused by
bad coding practices by developers, which is akin to our motivating example.
Even with formal computer science education, significant software
documentation, and several standard online training courses, there is still a
proliferation of unoptimized and bug-ridden code, especially in open-source
software (Lavallée and Robillard, 2015). Hence, we guarantee a degree of
program correctness by associating software performance with the execution
time of automated test cases, which serves as the ground truth microbenchmarks
for developing performance-based predictive models to predict code
performance.

##  4\. PACE Framework

PACE’s framework displayed in Figure 2 is decomposed into four main phases:
(i) in Phase 1 (see Section 4.1), we collect experimental data by selecting
candidate repositories and simulating a version-controlled software
development environment, (ii) Phase 2 (see Section 4.2.1) details our code
stylometry feature selection and representation methods. We selected syntactic
and lexical features and transformed those features to real-valued vectors
using proposed statistic and neural algorithms, (iii) in Phase 3 (see Section
4.3), we extract the execution time of unit test cases using Travis CI and
Maven Surefire, and (iv) Phase 4 (see Section 4.4) describes the
implementation of predictors for continuous performance predictions. We
implemented several regression models and selected the best model for
predicting code performance.

Figure 2. Overview of PACE

###  4.1. Phase 1: Data Collection

The performance of predictive models is highly dependent on the quality of
input data. How the data is collected is pertinent in filtering out bias and
ensuring appropriate problem representation. This section details the
strategies used in collecting experimental data for this study. In paragraph
1, we detail our data collection criteria. Paragraph 2 describes the
decomposition of selected candidate repositories into git branches
representing the current code state at time $n$ of a git push. Finally,
paragraph 3 discusses PACE’s continuous prediction strategy.

Selecting Candidate Repositories. Instead of creating our experimental code
repositories to evaluate our system, we selected existing popular candidate
repositories. Candidate repositories used in our experiments were selected
using characteristic filters that satisfy repository quality and popularity
criteria as case studies. These filters are as follows: candidate repository
must be (i) available and accessible on GitHub, (ii) contain compilable Java
code, (iii) have test cases, (iv) have commits $\geq$ 24, and 12 non-
contributor stars and forks. These parameters ensure that we achieve a
representative collection of candidate repositories. In our poll among
software developer participants, 97% (33 out of 34) declared that their
company requires developers to commit only compilable code, and 94% (32 out of
34) expressed that they develop most unit and integration tests during
requirements analysis. Therefore, our filters reflect expectations in a
professional software project.

Repository Point-in-Time Snapshots (RPiTS). Post candidate repositories
selection, we replicated the software development process by going back in
time to freeze and extract the repository state given a commiti. We reverse
(R) repository history and extracted branches using the following steps:

  * R1.

Forked the repository and initialized the cloned repository using the git init
command.

  * R2.

Linked remote repository using the git remote add origin <remote_url> command.

  * R3.

Browsed through repository commit history using git log, and git log --online
commands.

  * R4.

Created commit branches using the git checkout -b <branch_name> command.

  * R5.

Pushed repository’s content to the newly created branch using the git push -u
origin <branch_name> command.

  * R6.

Performed a hard reset on the repository branch using the git reset --hard
<commit_hash> command.

Figure 3. Continuous Predictions. CCS: Current Code State. The input-features
of a CCSn given commit $n$ (previous latest) are trained on a predictive model
to predict the performance impact of CCSn-1 at commit $n-1$ (current latest).
Following that, we use CCSn-1 (previous latest (originally, current latest))
input-features in predicting CCSn-2 (current lastest) performance. We roll
this process until CCSn-n

.

Continuous Predictions and Deduplication. We propose continuous or rolling
predictions in line with our goal of predicting the performance of the local
software repository before being pushed to its remote counterpart.
Encapsulated in the Figure 3 displays base repository copy ($n$) is the
repository’s current code state (CCS), while ($n-1$) is a local version
consisting of an updated CCS in the queue to be pushed to remote. We feed $n$
(training) to a regression function ($\phi$) to predict the performance of
$n-1$ (testing). This process is repeated for ($n-n$). Furthermore, it
eliminates duplicated observations.

###  4.2. Phase 2: Code Stylometry Feature Engineering

Stylometry is the application of statistics to derive insights from the
natural language writing style of a human (Li et al., 2006). One of its
earliest pieces of literature came in The Fredalist papers (Mosteller and
Wallace, 2012). The authors, Mosteller and Wallace, compared unattributed
corpora to those of their selected works. They applied statistical similarity
methods to map anonymous corpus to probable authors. Motivated by this
approach, Abbasi et al. (Abbasi and Chen, 2008) introduced a scalable
stylometric analysis approach termed Writeprints. The authors extracted
salient lexical, content, syntactic, idiosyncratic, and structural features to
create a robust feature set. Consequently, they fed these features to a
Karhunen-Loeve Transform algorithm enabled by pattern disruptors to detect
anonymized cyberspace entities.

Code-stylometry (CStyle) is the stylometric analysis of a code’s syntactic and
lexical characteristics (Tereszkowski-Kaminski et al., 2021). In principle,
cstyle is akin to stylometry. Contrastly, in stylometry, we analyze natural
languages, while cstyle involves the analyses of programming languages.
Researchers have successfully applied cstyle features in code analysis (Sarnot
et al., 2019) and security (Caliskan-Islam et al., 2015; Kul et al., 2020). We
leverage this knowledge to select syntactic and lexical features in Section
4.2.1 and numerically transformed selected nodes in Section 4.2.2.

####  4.2.1. CStyle Feature Selection

Colin and Bernat (Colin and Bernat, 2002) presented scope-tree, a methodology
employed to dynamically ascertain the worst-case execution time (WCET) of a
program. The authors analyzed the scope of a code using performance features
such as declaration, expressions, loops, and conditionals. WCET’s symbolical
evaluation showed the utility of employing the aforementioned features in
analyzing the worst-case execution time of a program. Thus, motivated by the
scope-tree approach, we selected syntactic and lexical features from source
code files. Instead of treating characters and words in the corpus as
tokenized entities, we selected syntactic and lexical features for numerical
representations.

Syntactic features are programming language dependent and provide context on a
programmer’s style-based preferential characteristics. These features vary in
representation — higher-level languages such as Python are more expressive and
easier to comprehend, and lower-level languages such as C tend to be arcane
with a higher comprehension bar (Caliskan-Islam et al., 2015). Lexical
features are semantic representations of language syntax that define the
meaning of actions in a code (Caliskan-Islam et al., 2015). Our targeted
feature selection technique ensures a reduced corpus size (see Section 6.4).
Table 1 details the taxonomy of selected features.

Table 1. A Taxonomy of Selected Code Stylometry Features. Selected features
represent five classes of nodes: statements, controls, expressions,
invocations, and declarations.

Features: {Statements, Controls, Expressions} $\in$ Syntactic $\land$
{Invocations, Declarations} $\in$ Lexical
---
Class |  Types |  Brief Description
Statements |  IfStatement, WhileStatement, DoStatement, AssertStatement, SwitchStatement, ForStatement, ContinueStatement, ReturnStatement, ThrowStatement, SynchronizedStatement, TryStatement, BreakStatement, BlockStatement, BinaryOperation, CatchClause |  Dictates the behavior of a program under explicitly defined conditions
Controls |  For, EnhancedFor |  Defines the repetition of instructions dependent on the satisfaction of requirements
Expressions |  StatementExpression, TernaryExpression, LambdaExpression |  Independent language entities with unique definitions
Invocations |  SuperConstructorInvocation, MethodInvocation, SuperMethodInvocation, SuperMemberReference, ExplicitConstructorInvocation, ArraySelector, AnnotationMethod, MethodReference |  Defines the invocation of a program from another program
Declarations |  TypeDeclaration, FieldDeclaration, MethodDeclaration, ConstructorDeclaration, PackageDeclaration, ClassDeclaration, EnumDeclaration, InterfaceDeclaration, AnnotationDeclaration, ConstantDeclaration, VariableDeclaration, LocalVariableDeclaration, EnumConstantDeclaration, VariableDeclarator |  Declares the existence of an entity in memory and assigns a value to that entity

####  4.2.2. Code Stylometry Feature Representation

An indispensable component of building predictive models is the transformation
of text-based observations into numerical representations. Bengio et al.
(Bengio et al., 2013) chronicled the importance of representation learning in
predictive modeling. Thus, post-feature selection, we transform selected
features to real-valued vectors using Algorithms 1 (statistic) and 2 (neural).

Statistical Representation (SR). Researchers have demonstrated the utility of
applying domain knowledge to retrieve pertinent information by assigning
weight values to tokens in a corpus. We leverage this existing knowledge using
a feature extractor equation (FE) introduced by Biringa and Kul (Biringa and
Kul, 2021). FE numerically transforms selected cstyle features.

In Algorithm 1, we initialize an empty list ($\mathbb{R}$) to store real-
valued vector representations of extracted features and a non-empty list of
feature nodes (FN) denoting lexical or syntactic features. Next, we parse the
source files using javalang (a Java programming language parser)
111https://github.com/c2nes/javalang and generate $AST$. We traversed the
trees AST $t_{i}$ $\in$ $AST$ using a preorder traversal algorithm and
selected the features. Following that, we make a subroutine call to FE
transformation equation (FE) to convert selected features to vectors: (i)
$\sum\limits_{i=0}^{n}$ is the count of selected features $(a)$, from $i$ to
$n$, and (ii) $|x|$ is the size of code files in characters. Note, that the
inputs for FE are the total count of selected features in a source code and
the character length of the aforementioned source code.

Input : Repository $\in$ code $\texttt{S}=\\{s_{1},s_{2},\dots,s_{n}$}
$\blacktriangleright$ Repository source code

1

Output : ${\texttt{v}}=\langle v_{1},v_{2},\dots,v_{n-1},v_{n}\rangle$

2

3$\mathbb{R}\leftarrow\varnothing$ $\blacktriangleright$ Variable to store
vectors

4

5FE = $-log_{10}(\frac{\sum_{i=0}^{n}(a_{i})}{|x|})$ $\blacktriangleright$
Transformation equation

6

7FN $\leftarrow$ $\\{sv_{i},lv_{i},sv_{2},lv_{2},\dots,n$}
$\blacktriangleright$ Selected CStyle features

8

9# Selection

10 for  _$s_{i}$ $\in$ S_ do

11 curr $\leftarrow$ $\varnothing$

12 AST $\leftarrow$ Parse $s_{i}$ $\land$ gen $t_{i}$ $\in$ $AST$
$\blacktriangleright$ Generate abstract syntax tree (AST)

13 $a$ $\leftarrow$ Trv $AST_{i}$ $\land$ $\exists$ $\forall$ $n_{i}$ $\in$ FN
$\blacktriangleright$ Traverse AST and select CStyle features

14 # Representation

15 $v$ $\leftarrow$ Call FE($a_{i}$, $|\mathnormal{\texttt{S}_{i}}|$)
$\blacktriangleright$ Transform selected feature using FE

16 Add $v_{i}$ $\to$ curr

17 if  _syntactic_ then

18 Add curr $\to$ $\mathbb{R}[sv_{i}]$ $\blacktriangleright$ Syntactic feature
match

19

20 else lexical

21 Add curr $\to$ $\mathbb{R}[lv_{\texttt{i}}]$ $\blacktriangleright$ Lexical
feature match

22

23

return $\mathbb{R}$

Algorithm 1 Statistical Representation of CStyle Features (SR)

Input : Repository $\in$ code $\texttt{S}=\\{s_{1},s_{2},\dots,s_{n}$}

Output : ${\texttt{v}}=\langle v_{1},v_{2},\dots,v_{n-1},v_{n}\rangle$

1

2FN $\leftarrow$ $\\{sv_{i},lv_{i},sv_{2},lv_{2},\dots,n$}
$\blacktriangleright$ Selected cstyle features

3

4$\mathbb{R},\texttt{T},\texttt{U}\leftarrow\varnothing,\varnothing,\varnothing$
$\blacktriangleright$ Vector, AST and uncompiled code storages

5 # Selection

6

7for  _idx , $s_{i}$ $\in$ S_ do

8 if  _$s_{i}$ is parsable_ then

9 $t_{i}$ $\leftarrow$ parse $s_{i}$ $\land$ gen $t_{i}$ $\blacktriangleright$
Generate abstract syntax tree (AST)

10 Add $t_{i}$ $\to$ T $\blacktriangleright$ Store AST

11

12 else

13 U[idx] $\leftarrow$ $s_{i}$ $\blacktriangleright$ Store uncompiled code

14

15

16

17$\texttt{a}\leftarrow\varnothing$ $\blacktriangleright$ Store matched cstyle
features

18

19$\texttt{SEQL}\leftarrow\texttt{64}$ $\blacktriangleright$ Set maximum
sequence length

20

21if  _not U_ then

22 for  _$t_{i}$ $\in$ T_ do

23 $a$ $\leftarrow$ Trv. $t_{i}$ $\land$ $\exists$ $\forall$ $n_{i}$ $\in$ FN
$\blacktriangleright$ CStyle feature match

24

25

26 else

27 raise error for U

28

29# Representation

30 Feed $a$ $:=$ Dimensional $v_{i}$ $\in$ 32 $\to$ $\mathcal{M}$
$\blacktriangleright$ Train embedding model using features

31

32Truncate $v_{i}$ $\forall$ v $>$ $\land$ ZeroPad $<$ SEQL
$\blacktriangleright$ Truncate and pad feature embeddings

33

return $\mathbb{R}$

Algorithm 2 Neural Representation of CStyle Features (NR)

Neural Representation (NR). We use the word2vec embedding model (Mikolov et
al., 2013) to represent selected features into real-valued vectors
representing neural features of high-dimensional vectors in low-dimensional
vector space. We employed the gensim library
222https://radimrehurek.com/gensim/models/word2vec.html to facilitate the
generation of vectors and used the default for all parameters except the
sequence length and vector dimensions. In algorithm 2, we transform selected
features into embedding vectors. Concisely, selected cstyle features in (FN)
are tokenized — text splitting into independent blocks of characters (Mikolov
et al., 2013). Next, we set the maximum sequence (SEQL) length to 64,
observations $\lessgtr$ SEQL are zero-padded and truncated. Currently, there
is no common consensus in the literature regarding what constitutes an
appropriate embedding size, only predominantly task-dependent suggestions
(Biringa et al., 2023). However, after experimenting with embeddings of
different sizes, we found that a 64 sequence length suffices in maintaining a
permissible relationship between a model’s predictive prowess and training
time in this case. Finally, we return generated feature embeddings.

###  4.3. Phase 3: Functional Tests’ Times Extraction

Recall in our problem definition (see Section 2) we executed functional test
cases, after which we logged the test times to derive the performance of
programs, ensuring correctness. The test times are microbenchmarks (dependent
variable) fed to predictive models. We utilized Travis CI (TCI) (Meyer, 2014)
and Maven Surefire (MVNS) (Nie et al., 2020) to extract the execution test
times. TCI is an open-source continuous integration service used to build and
test software hosted on GitHub and other source code management and
distributed version control platforms. We created a .travis.yml file —
compiles and executes functional test scripts present in candidate
repositories, such as unit tests — to link the candidate repository and
testing components, after which we deployed and ran repository test scripts.
Following that, we executed test cases and logged the test time. Maven
Surefire (MVNS) is the most frequently used automation tool for building and
testing Java software. We retrieved the MVNS testing tool from the Apache
website and performed the same linking operation used for the TCI framework.

###  4.4. Phase 4: Continuous Performance Prediction

Continuously predicting the performance impact of syntactically correct source
code to software is a core feature of PACE. Directly checking the ground truth
performance of isolated or local code updates using testing and profiling
tools is possible. However, it is a deviation from PACE’s objective, which
entails estimating what updates and potentially low-performing code contribute
to the overall software performance before being pushed to the primary or
remote repository.

(a) Supervised Learning

(b) Multiple Regression

(c) Univariate Multiple Regression

Figure 4. Problem Formulation for Regression Model Selection

To achieve PACE’s objective, we employed a collection of regression models
(Doran et al., 2017), which are supervised machine learning algorithms used to
model the relationship between single or multiple independents (predictor) and
a dependent variable (response). Regression models are used to predict
continuous data values, which is fitting for predicting software execution
time performance in this case. We design our model by attentively framing the
problem, given that: (i) it is a supervised learning problem, given that our
training samples are labeled, for example, mapping the execution test times
target to cstyle features (see Figure 4a for a visual depiction), (ii) it is a
multiple regression problem, given that we utilize several feature classes to
perform predictions (see Figure 4b), and (iii) it is a univariate regression
problem, given that we are attempting to predict a single execution time
performance for each commit (see Figure 4c). Following cstyle feature
selection and representation, we trained five univariate multi-regression
models.

Table 2. A Taxonomy of Regression Models for Continuous Performance Prediction

Learning |  Class |  Model |  Short Description
---|---|---|---
Deep |  Neural Network |  Multilayer Perception (MLP) |  A feed-forward artificial neural network
Error |  Support Vector Machines |  Support Vector Regressor (SVR) |  A non-probabilistic model that efficiently utilizes a decision boundary to maximize discrete value margins
Information |  Decision Trees |  Random Forest Regressor (RFR) |  A collection of decision trees built during training referred to as estimators
Probabilistic |  Bayesian Network |  Bayesian Ridge (BR) |  A Bayesian probabilistic model derived from Bayes theorem
Lazy |  Distance |  k-Nearest Neighbors Regressor (kNN) |  A lazy model that employs distance between data points in deriving insights

##  5\. Evaluation

This section describes selected evaluation metrics (see Section 5.1) and case
study datasets (see Section 5.2).

Setup. We conducted experiments using an Apple commodity laptop with an Apple
M1 8-core CPU, 16-core Neural Engine, 8 GB RAM, and macOS Big Sur (version
11).

Reproducibility. We have made all source code, results, and datasets publicly
available 333https://github.com/PADLab/PACE with command line arguments to
reproduce the results presented in this paper.

###  5.1. Metrics

We comprehensively evaluate our models’ performance using classical ML
regression evaluation metrics (Géron, 2019; Pandey et al., 2020). We employ
Root Mean Squared Error (RMSE), Mean Squared Error (MSE), Mean Absolute Error
(MAE), and Root Mean Squared Logarithmic Error to assess the predictive
performance of models.

  * •

RMSE: Adopts a quadratic approach that measures the squared root of the mean
squared difference between predicted and actual observations.

  * •

MSE: The average squared distance between the actual observations and
predicted.

  * •

MAE: Calculates the mean error sizes in predictions and does not consider the
directional relationship between predicted and actual. It is only concerned
with the absolute difference.

  * •

RMSLE: Calculates the log difference between actual and predicted values.

In selecting regression metrics, we are motivated by three applications.

  * •

Application 1: For a conservative use case (underestimating code performance
predictions (CPP)), we recommend applying RMSE or MAE.

  * •

Application 2: For a liberal use case (overestimating (CPP)), we recommend
applying MSE.

  * •

Application 3: For a centrist use case, we recommend applying RMSLE.

$\displaystyle\texttt{{RMSE}}=\sqrt{(\frac{1}{n})\sum_{i=1}^{n}(y_{i}-x_{i})^{2}}\;\texttt{{MSE}}=\sum_{i=1}^{D}(x_{i}-y_{i})^{2}\;\texttt{{MAE}}=(\frac{1}{n})\sum_{i=1}^{n}\left|y_{i}-x_{i}\right|\;\texttt{{RMSLE}}=\sqrt{\frac{1}{N}\sum_{i=1}^{n}\left(\log{yi}-\log{\hat{yi}}\right)^{2}}$

###  5.2. Datasets

AddressBook Dataset (ABD). We obtain the AddressBook dataset from the
Addressbook 444https://github.com/se-edu/addressbook-level2 repository - a
command-line interface program used to teach software engineering students
object-oriented programming. We applied RPiTS to extract five commits. We
preprocess the repository by only selecting Java files, with commit branches
that comprise 210 total, 42 $\mu$ Java files, and 10,870 lines of code. We
concatenate the source code to obtain observations. Finally, we join git
branches to the execution time microbenchmarks from TCI to create the final
dataset.

DSpace Dataset (DSD). We extracted the DSpace dataset from DSpace
555https://github.com/DSpace/DSpace repository - an open-source web
application software for providing digital resources to thousands of
organizations and institutions worldwide and has a large and active developer
community. The dataset extracted from the original repository using RPiTS
contains 50 commits, 127,157 total files, and 2543 $\mu$ Java files,
32,198,822,697 lines of code. We create the final dataset by joining git
branches to execution time microbenchmarks derived from MVNS.

##  6\. Experiments

Table 3. Predictive models for continuous performance prediction using
statistical and neural representations from code-stylometry features. Commit
($c_{n}$) denotes repository CCS, where {$c_{n},c_{n-1}\\}\in$ {training,
testing} sets. RMSE, MSE, MAE, and RMSLE denotes performance metrics used to
evaluate the predictive power of implemented models as discussed in Section
5.1. AVG denotes predictor performance across metrics. #SC denotes the total
source files for each CCS

ABD
---
SR Features |  | NR Features
Predictor |  Commit |  RMSE |  MSE |  MAE |  RMSLE |  AVG |  RMSE |  MSE |  MAE |  RMSLE |  AVG
PACE-MLP |  $c_{n},c_{n-1}$ |  0.5280 |  0.2788 |  0.4923 |  0.1674 |  0.3666 |  0.2391 |  0.0571 |  0.2284 |  0.0783 |  0.1507
|  $c_{n-1},c_{n-2}$ |  0.2088 |  0.0436 |  0.1903 |  0.0678 |  0.1276 |  0.2234 |  0.0499 |  0.2004 |  0.0735 |  0.1368
|  $c_{n-2},c_{n-3}$ |  0.0980 |  0.0096 |  0.0758 |  0.0309 |  0.0536 |  0.0539 |  0.0029 |  0.0267 |  0.0174 |  0.0252
|  $c_{n-3},c_{n-4}$ |  0.1220 |  0.0148 |  0.0968 |  0.0374 |  0.0678 |  0.0867 |  0.0075 |  0.0771 |  0.0275 |  0.0497
|  |  |  |  |  AVG $\blacktriangleright$ |  0.1539 |  |  |  |  AVG $\blacktriangleright$ |  0.0906
PACE-SVR |  $c_{n},c_{n-1}$ |  0.2199 |  0.0483 |  0.2199 |  0.0698 |  0.1395 |  0.2199 |  0.0483 |  0.2199 |  0.0698 |  0.1395
|  $c_{n-1},c_{n-2}$ |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124 |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124
|  $c_{n-2},c_{n-3}$ |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116 |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116
|  $c_{n-3},c_{n-4}$ |  0.0700 |  0.0049 |  0.0700 |  0.0218 |  0.0416 |  0.0700 |  0.0049 |  0.0700 |  0.0218 |  0.0416
|  |  |  |  |  AVG $\blacktriangleright$ |  0.0763 |  |  |  |  AVG $\blacktriangleright$ |  0.0763
PACE-kNN |  $c_{n},c_{n-1}$ |  0.2199 |  0.0483 |  0.2199 |  0.0698 |  0.1395 |  0.2199 |  0.0483 |  0.2199 |  0.0698 |  0.1395
|  $c_{n-1},c_{n-2}$ |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124 |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124
|  $c_{n-2},c_{n-3}$ |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116 |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116
|  $c_{n-3},c_{n-4}$ |  0.0700 |  0.0049 |  0.0700 |  0.0218 |  0.0416 |  0.0700 |  0.0049 |  0.0700 |  0.0218 |  0.0416
|  |  |  |  |  AVG $\blacktriangleright$ |  0.0763 |  |  |  |  AVG $\blacktriangleright$ |  0.0763
PACE-RFR |  $c_{n},c_{n-1}$ |  0.2199 |  0.0483 |  0.2199 |  0.0698 |  0.1395 |  0.2199 |  0.0483 |  0.2199 |  0.0698 |  0.1395
|  $c_{n-1},c_{n-2}$ |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124 |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124
|  $c_{n-2},c_{n-3}$ |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116 |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116
|  $c_{n-3},c_{n-4}$ |  0.0700 |  0.0049 |  0.0700 |  0.0218 |  0.0416 |  0.0700 |  0.0049 |  0.0700 |  0.0218 |  0.0416
|  |  |  |  |  AVG $\blacktriangleright$ |  0.0763 |  |  |  |  AVG $\blacktriangleright$ |  0.0763
PACE-BR |  $c_{n},c_{n-1}$ |  0.2200 |  0.0484 |  0.2200 |  0.0698 |  0.1395 |  0.2200 |  0.0484 |  0.2200 |  0.0698 |  0.1395
|  $c_{n-1},c_{n-2}$ |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124 |  0.1800 |  0.0324 |  0.1800 |  0.0575 |  0.1124
|  $c_{n-2},c_{n-3}$ |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116 |  0.0200 |  0.0004 |  0.0200 |  0.0061 |  0.0116
|  $c_{n-3},c_{n-4}$ |  0.0699 |  0.0048 |  0.0699 |  0.0218 |  0.0416 |  0.0699 |  0.0048 |  0.0699 |  0.0218 |  0.0416
|  |  |  |  |  AVG $\blacktriangleright$ |  0.0763 |  |  |  |  AVG $\blacktriangleright$ |  0.0763

###  6.1. RQ1. What is the predictive accuracy of code performance given an
update?

We answer this question by first implementing regression models with SR and NR
input-feature vectors using the 5-commits ABD in Table 3, after which we
select the best-performing model (PACE-kNN). The selected (best performing)
model is then exclusively used as the primary predictor on the 50-commits DSD
in Figure 5. The predictive ((0.0763, 0.7366) $\mu$(RMSE, MSE, MAE, RMSLE
(RMMR)) $\in$ SR $\land$ NR) performances of PACE-kNN’s is slightly better
than other models when prediction throughput (0.0063 $\mu$ seconds (sec) $\in$
SR $\land$ NR) is factored in. Hence, we select PACE-kNN as our primary
predictor. Interestingly, all models except PACE-MLP perform similarly to the
fourth decimal place across metrics for both features.

Thus, outside the model’s prediction times, there is no marketable difference
in performance between the statistic and neural representation of cstyle
features. Furthermore, we observe that the single (($c_{n-48},c_{n-49}$) =
6.2660 $\mu$(RMMR)) outlier predictive performance in the DSD has both the
numerically highest target variable (4.081 sec) and source file (46 files)
difference, where (($c_{n-48}=13.528,c_{n-49}=17.609$) sec $\land$
($c_{n-48}=2559,c_{n-49}=2605$) the total number of files) $\in$ (train, test)
sets respectively, indicating that a commit with a disproportionate code
introduction or deduction can skew the results of predictors. However, as
demonstrated in (($c_{n-49},c_{n-50}$) = 1.9669 $\mu$(RMMR)), the model’s
performance improves with a subsequent commit.

(a) Performance (MSE, MAE and RMSLE)

(b) Performance (AVG)

Figure 5. PACE-kNN for continuous performance prediction on DSD. To
deduplicate generated results from RMSE in Table 4, we use only MSE, MAE,
RMSLE, and the average value of the Features (SR, NR)

Figures 8 and 9 visualizes the training and prediction throughput of all
models. PACE-MLP has the fastest prediction time with a (0.0003$\mu$ sec).
However, with a 0.1222 ($\mu$(RMMR) $\in$ SR $\land$ NR), it underperforms
PACE-kNN and other models by 46%. PACE-BR (0.0004$\mu$ sec) comes a close
second, followed by PACE-SVR (0.0005$\mu$ sec) and PACE-RF (0.0005$\mu$ sec).
We also present the predictors’ latency (time elapsed to train and test
predictors using SR and NR features). PACE-SVR ($\mu$(SR, NR) = 0.0096 sec)
has the lowest latency pipeline on ABD, outperforming PACE-kNN (0.0231 sec),
PACE-BR (0.0292 sec), PACE-RFR (0.0745 sec), and PACE-MLP (0.1023 sec) by (2x,
3x, 7x and 10x) respectively. Finally, on DSD, our primary predictor PACE-kNN
attained a ($\mu$(SR, NR) = 0.0745 sec) latency.

(a) MSE ($\Delta$)

(b) MAE ($\Delta$)

(c) RMSLE ($\Delta$)

(d) AVG ($\Delta$)

Figure 6. Performance Impacts of Code Update (ABD)

(a) MSE ($\Delta$)

(b) MAE ($\Delta$)

(c) RMSLE ($\Delta$)

(d) AVG ($\Delta$)

Figure 7. Performance Impacts of Code Update (DSD)

(a) PACE-MLP

(b) PACE-SVR

(c) PACE-kNN(ABD)

Figure 8.  Training (TT) and Prediction (PT) Throughput of Predictors for SR
and NR Features on ABD.
PACE-MLP: $\mu$(TT, PT) = 0.0239 sec $\in$ SR. $\mu$(TT, PT) = 0.1807 sec
$\in$ NR. $\mu$(SR, NR) = 0.1023 sec
PACE-SVR: $\mu$(TT, PT) = 0.0044 sec $\in$ SR. $\mu$(TT, PT) = 0.0148 sec
$\in$ NR. $\mu$(SR, NR) = 0.0096 sec
PACE-kNN: $\mu$(TT, PT) = 0.0106 sec $\in$ SR. $\mu$(TT, PT) = 0.0356 sec
$\in$ NR. $\mu$(SR, NR) = 0.0231 sec

(a) PACE-RFR

(b) PACE-BR

(c) PACE-kNN(DSD)

Figure 9. Training (TT) and Prediction (PT) Throughput of Predictors for SR
and NR Features on ABD and DSD.
PACE-RFR: $\mu$(TT, PT) = 0.0454 $\in$ SR. $\mu$(TT, PT) = 0.1037 $\in$ NR.
$\mu$(SR, NR) = 0.0745
PACE-BR: $\mu$(TT, PT) = 0.0130 $\in$ SR. $\mu$(TT, PT) = 0.0455 $\in$ NR.
$\mu$(SR, NR) = 0.0292
PACE-kNN(SR): $\mu$(TT, PT) = 0.0075. PACE-kNN(NR): $\mu$(TT, PT) = 0.1037.
$\mu$(SR, NR) = 0.0745

###  6.2. RQ2. What is the performance impact of commit at
($c\textsubscript{n-1}$) given ($c\textsubscript{n}$) code update?

We answered this question by further analyzing the generated results from RQ1.
We take the delta of the predictive performance of commiti given commiti+1.
There are two categories (positive and negative) of performance impact. From
Figures 6 and 7, commiti is considered to have a positive impact on the
following commiti+1 if and only if the delta predictive error rate of
commiti+1 is less than that of commiti, otherwise it is a negative performance
impact. For instance, in Figure 7d, the average error rate for DSD is
($c\textsubscript{n},c\textsubscript{n-1}$) is 0.2895, the following update
($c\textsubscript{n-1},c\textsubscript{n-2}$). cn-1 is the training set fed to
PACE-kNN to predict the performance cn-2 (testing set), resulting in a 0.1399
average error rate. The delta is 0.1496. Hence, the performance impact between
($c\textsubscript{n},c\textsubscript{n-1}$) and
($c\textsubscript{n-1},c\textsubscript{n-2}$) is positive. The average
performance impact of $c\textsubscript{n-i}$ is (positive=2, negative=1) $\in$
ABD (Figure 6d) and (positive=24, negative=24) $\in$ DSD for 3 and 48 updates
respectively. On ABD, commits ($c\textsubscript{n-1}$, $c\textsubscript{n-2}$)
and ($c\textsubscript{n-3}$, $c\textsubscript{n-4}$) has the best positive
(0.0270) and single negative (-0.0300) $\Delta$ performance respectively.
Similarly, DSD’s ($c\textsubscript{n-18}$, $c\textsubscript{n-19}$) and
($c\textsubscript{n-47}$, $c\textsubscript{n-48}$) has the best positive
(0.0862) and worst negative (-6.2362) $\Delta$ performance respectively.

###  6.3. RQ3. What is the time cost of selecting and representing code
stylometry features?

We answer this question by timing the selection and representation of features
for each CCS in the ABD and DSD. The ABD constitutes five commits and 42
average number of files (ANF), hence a much lower feature selection and
representation throughput, while its DS counterpart has 50 commits and 2543
ANF. The $\mu$ cstyle (statements, expressions, controls, invocations,
declarations) feature selection throughput for AB and DS datasets are
0.2297$\mu$ sec and 23.3201$\mu$ sec (see Figure 10 a and b). After feature
selection, we proceeded to time the statistic (SR) and neural representation
(NR) of these features. SR is $\in$ (0.0002$\mu$ sec $\land$ 0.0126$\mu$ sec)
for ABD $\land$ DSD. NR is $\in$ (0.0055 $\mu$ sec $\land$ 0.0588 $\mu$ sec)
for ABD and DSD respectively (see Figures 10c, and 11).

The primary objective of this RQ is to ascertain the throughput of cstyle
feature selection and the fastest feature representation method between SR and
NR. Thus, the statistical representation of cstyle features is 27x and 5x
faster than its neural counterpart on ABD and DSD respectively. Note, the time
difference in intra-comparative feature representations (syntactic against
lexical $\in$ SR $\lor$ NR) is not attributed to the representation algorithm
but to its frequency occurrence.

.

(a) ABD Feature Selection

(b) DSD Feature Selection

(c) ABDSR

Figure 10. ABD and DSD CStyle Feature Selection and ABD Statistical
Representation (ABDSR)
Syntactic: $\mu$($\mu$(statement, expressions, controls)) = (0.2354, 23.3939)
$\in$ (ABD, DSD)
Lexical: $\mu$($\mu$(invocations, declarations)) = (0.2211, 23.2094) $\in$
(ABD, DSD)
$\mu$(Syntactic, Lexical) = (0.2297, 23.3201) $\in$ (ABD, DSD)
ABDSR: $\mu$($\mu$(statement, expressions, controls), $\mu$(invocations,
declarations)) = 0.0002

.

(a) DSSR

(b) AB Neural Representation

(c) DS Neural Representation

Figure 11. DSD Statistical Representation (DSDSR), ABD and DSD Neural
Representations
DSDSR: $\mu$($\mu$(statement, expressions, controls), $\mu$(invocations,
declarations)) = 0.0055
Syntactic: $\mu$($\mu$(statement, expressions, controls)) = (0.2354, 23.3939)
$\in$ (ABD, DSD)
Lexical: $\mu$($\mu$(invocations, declarations)) = (0.2211, 23.2094) $\in$
(ABD, DSD)
$\mu$(Syntactic, Lexical) = (0.2297, 23.3201) $\in$ (AB, DS)

###  6.4. RQ4. How does PACE perform in comparison with TEP-GNN?

To the best of our knowledge, we have not encountered any work directly
comparable to PACE with respect to scope, approach, and the mapping of
statistical and neural represented cstyle features to execution test times as
a performance microbenchmark for the continuous prediction code performance.
However, we aimed to derive a robust evaluation of PACE. Hence we compare PACE
to a closely related work in performance prediction titled TEP-GNN (Samoaa et
al., 2022). TEP-GNN leverages the merging of flow-augmented AST (FA-AST) —
dynamic interpretation of ASTs (Wang et al., 2020) — and Graphical
Convolutional Neural Network (GCNN) – GCNN’s are a subset of deep learning
models that exploits graphically represented input observations to encode and
transmit learned features using its vertices and edges (Morris et al., 2019) –
to predict the execution time of functional test cases.

The authors evaluated their work using four software repository case studies
(Samoaa et al., 2022), (i) SystemDS: an open source ML system for conducting
end-to-end data science tasks, (ii) H2: a Java-based relational database
management system, Dubbo: a Java-based open source remote procedure call
framework, and RDF4J: A Java-based resource description framework. The authors
created their experimental datasets by adapting GitHub’s hosted runners to
build source files ($x^{(i)}$) and consequently extract the execution times
traces of published test results ($y^{(i)}$).

Table 4. PACE-kNN’s performance in comparison with TEP-GNN. #File: Total
number of source files, #Nodes: Total number of nodes, Combined: Entire
dataset (H2, RDF4J, Dubbo, and SystemDS), #VOC: Vocabulary size

TEP-GNN |  | PACE-kNN
---|---|---
Project |  #Files |  #Nodes |  #VOC |  MSE |  #Nodes |  #VOC |  MSE(SR,NR)
H2 |  194 |  405706 |  17972 |  0.0340 |  113800 |  35 |  (0.0056, 0.0119)
RDF4J |  478 |  214436 |  10755 |  0.0160 |  52004 |  30 |  (0.0003, 0.0062)
Dubbo |  123 |  75787 |  4499 |  0.0230 |  19304 |  31 |  (0.0076, 0.0110)
SystemDS |  127 |  110651 |  3161 |  0.0110 |  22803 |  30 |  (0.0288, 0.0369)
Combined |  992 |  806580 |  36387 |  0.0170 |  207911 |  126 |  (0.0083, 0.0077)
|  |  |  AVG $\blacktriangleright$ |  0.0202 |  |  AVG $\blacktriangleright$ |  (0.0101, 0.0147)

We answer this question by replicating experimental steps adopted by the
authors (Samoaa et al., 2022), which include: (i) retrieving the test files of
case study datasets (H2, RDF4J, Dubbo, SystemDS) and corresponding execution
run times, (ii) splitting the data into two sets, where 80% $\in$ training and
20% $\in$ testing, and (iv) evaluating predictions using MSE (see Section
5.1). PACE-kNN outperforms TEP-GNN on three (H2, RDF4J, and Dubbo) out of four
projects by (143%, 96%), (192%, 88%), and (100%, 70%) $\in$ (SR, NR). TEP-GNN
outperformed PACE-kNN on SystemDS by (89%, 108%). Albeit, TEP-GNN best
performance is on SystemDS, it also has the least linear correlation between
actual and predicted values, as discussed by the authors.

On average, PACE-kNN attained a (0.0083, 0.0077) error rate on the entire
dataset and (0.0101, 0.0147) on the cumulative average performance. Hence,
out-performing TEP-GNN by (68%, 75%) and (66%, 31%) $\in$ (SR, NR)
respectively. Furthermore, TEP-GNN which was trained for 100 epochs has a
significantly higher vocabulary size, where (17972, 10755, 4499, 3161, 4499,
3161) $\in$ (H2, RDF4J, Dubbo, SystemDS) compared to PACE-kNN (35, 30, 31,
30). Thus, PACE-kNN’s vocabulary size is lower TEP-GNN by 513x, 358x, 145x and
105x.

##  7\. Scope and Threats

###  7.1. Scope

The execution time of functional test cases as a means of micro-benchmarking
the code performance of software doesn’t tell the whole story of what
constitutes optimal performing software. Incessantly changing contributing
factors such as disk space, memory, networks, operating systems, and core
hardware renders a conclusive performance predictive model almost impossible
and inherently a qualitative endeavor. However, it’s viable to quantitatively
solve the sub-components of the problem as demonstrated by our experiments.
Hence, PACE narrows down constraints by assuming an ideal software scenario
devoid of the volatility of memory, hardware compatibility, and network
troubleshooting, which is outside the scope of this work. Furthermore,
estimating the baseline execution time measure of automated test cases is
valuable in specific use cases, such as non-collaborative development
environments. However, it is not a substitute and outside the scope of PACE,
as it defeats the purpose of predicting the performance impact of the code
update to the primary before the commit process is complete, negating the
introduction of unoptimized code to the primary repository in the first place.

###  7.2. Threats

####  7.2.1. Availability.

Detailed in Section 4.3, the extraction execution times of functional tests
are dependent on the availability of test cases in the software repository.
These modules are responsible for testing the functionalities and features of
the software to ensure they perform as anticipated. Hence, testing scripts
should also be compact and have significant code coverage. Thus, in the
absence of these scripts, devising a technique to extract execution test times
whilst maintaining our principle of code correctness is pertinent and subject
of future research.

####  7.2.2. Validity.

Albeit diligently selecting candidate repositories that satisfy requirements
for a professional software repository, it is still possible to have projects
where test cases are sparse in breadth and shallow in depth, constituting a
false positive, which would be problematic for code correctness. One pathway
to approach this problem would be to design a search+validate system to search
and validate the existence or lack thereof of functional test cases. For
example, given a program with modules, there exist routines, search for
associated test cases to validate routine correctness, assign a positive value
(+ve), otherwise assign a negative value (-ve). Consequentially, it would be
up to the users to determine what constitutes a permissible threshold value or
is reflective of the tests’ depth and breadth.

####  7.2.3. Generalizability.

Given case studies projects written in Java, an apparent cause for concern is
PACE’s generalizability. This work doesn’t empirically demonstrate that the
entirety of PACE methods is transferable to other software projects. However,
we argue that candidate repository selection, code stylometry feature
engineering, and functional testing components of PACE can be generalized. The
candidate repository selection process was to find repositories that may be
representative of large software development projects. Microbenchmarking duty
is with the CI infrastructure, not with the code or software developer, and we
do provide the open-source repository for that. Any CI infrastructure can use
the same method and apply our proposed methods.

The selection and representation of cstyle features are foundational to the
imperative programming paradigm. For example, the existence of an if, while
statement conditionals and lambda expressions in Java are represented with
(IfStatement, WhileStatement, LambdaExpression) nodes. Similarly, the
aforementioned features nodes in Python are represented with (If, While,
Lambda) nodes using the ast module of the Python
666https://docs.python.org/3/library/ast.html standard library. Thus, the
concern of cstyle feature engineering generalizability can be tackled by
simply matching feature nodes in this work to corresponding nodes in other
high-level imperative programming languages.

####  7.2.4. Outliers.

Another obvious threat is in the area of disproportionate code updates, which
are voluminous introductions or deduction of source files to the software that
skew the baseline test performance. Recall our worst predictive performance
($\mu$(RMMR) = 6.2660) on DSD occurred at commits ($c_{n-48},c_{n-49}$) and
had the most file and test times difference between commits. Given that PACE
has no control over the content of the code base, we argue that the best way
to solve this threat is to integrate an observer program that listens and
notifies users of voluminous updates, enabling the user to be alert to
potentially outlier values when evaluating corresponding predictions.
Furthermore, as demonstrated in RQ1, PACE-kNN model learns the new features
resulting in improved predictions in subsequent commits.

####  7.2.5. Base Case (What is a good starting n?).

Recall that to simulate a pseudo-active software development scenario, we
introduced RPiTs (see Section 4.1, paragraph 3). RPiTs reverse development
time to freeze and extract the current code state given a (n) which enables
the retrieval of software state throughout its development history. Our case
study datasets ABD and DSD have 5 and 50 commit n’s respectively. Given that
developing software commonly commences with a developer writing a character, a
collection of characters (string), a collection of strings (function) a
collection of functions (module), a collection module (package), a collection
of packages (library), it is pertinent that PACE’s base case (n) is a
collection of strings (function) at the minimum for the source code to be
compiled, AST generated and cstyle features extracted. Following that, it
becomes apparent that a collection of characters (string) won’t suffice
because (i) not enough observations for predictive modeling and (ii)
complexity in designing in writing functional test cases to validate tiny
components of a code. Thus, for these reasons, we ensured that our RPiTs
technique reflects these considerations. Hence, a threat constituting source
files usually occurring in the beginning phases of developing the software
becomes evident. We argue that there are two avenues to approach this threat.

The first entails a modification to RPiTs to ensure that the base case for n
always has a non-trivial ($\geq$ 10) number of routines with its corresponding
test cases suitable for functional testing and predictive modeling. Second,
exploiting the success of large language models (LLMs) (Zhao et al., 2023) —
contextually distributional semantic models trained using the Transformer
neural architecture on colossal amounts of data to perform a specific task,
such as representative (Bidirectional Encoder Representations from
Transformers (BERT)), generative (Generative Pre-Trained Transformer), and
pseudo-universal (Pathways Language Model (PaLM)) — in several language tasks
to fine-tune limited n observations, enabling it to benefit of using pre-
trained weights to predict associated test times.

##  8\. Related Work

Software performance predictions are an inherently complex task akin to
building optimally compatible software with hardware machines of varying
types, operating systems, and network infrastructure. Given the breakneck
speed of technological advancement, it is a challenge to consistently account
for these variables while maintaining highly accurate performance predictions.
However, recent advances in the literature have applied machine and deep
learning methods to achieve this goal.

Research work in the literature with varying degrees of predictive performance
includes, Samoaa et al. (Samoaa et al., 2022) proposed an approach for precise
execution time prediction of functional tests using GNNs (see RQ4). Didona et
al. (Didona et al., 2015) combine analytical modeling and machine learning to
magnify the traceability and overall applicability of performance prediction
on NoSQL and Order Broadcast services. Meng et al. (Meng and Norris, 2017)
proposed a framework for creating performance models through static code
targeted at the performance of scientific applications. Guo et al. (Guo et
al., 2018) proposed a data-efficient approach to predict the performance of
configurable systems.

Zhou et al. (Zhou et al., 2019) proposed a classification model based on the
extraction of semantic and structural features to predict the performance
(Accepted and Time Limit Exceed) of competitive programming solutions before
submission. Kaltenecker et al. (Kaltenecker et al., 2020) discusses the
tradeoffs in employing ML for performance prediction. The authors recommended
adopting a more nuanced downstream approach appropriate for the targeted
problem domains. Ramadan et al. (Ramadan et al., 2021) learned the
representation of hierarchical structural changes of a source code used to
build a Long Short-Term Memory (LSTM) model tasked with code performance
predictions.

Liu et al. (Liu et al., 2021) proposed a max/min algorithm to predict the code
execution time performance of digital signal processing (DSP) software. The
proposed algorithm is a fusion of clustering, similarity measure, application
of weights in sample and attribute, and linear regression. The authors
validated their model using PHY DSP Benchmark on the TIC64 DSP processor
obtaining a $\mathtt{\sim}$10% error rate. Velez et al. (Velez et al., 2021)
proposed Comprex, a system designed to have the explainable capacity to be
associated with the performance of a configurable system. The authors
evaluated its white-box system on open-source projects achieving similar
performance to its black-box counterpart with a reduction in cost and an
increase in ML explainability.

Samoaa et al. (Samoaa et al., 2023) proposed a unified active learning
framework for annotating graphically represented data with applicability in
predicting software performance. This work was motivated by the limited
availability of annotated software performance graphical data required for
training machine learning models. Their approach consists of representing
source code using ASTs and transforming these trees to graph embeddings using
FA-AST, supervised and unsupervised learning. Finally, the authors employ
active and passive learning to query and annotate data.

Böck et al. (Böck et al., 2023) conducting experiments to empirically
demonstrate the utility of large language models (LLMs) in predicting the
runtime performance of programming competition programs. They demonstrate that
although these models produce accurate code performance predictions, they fail
to generalize to out-of-sample domains such as video games. In their approach,
they mined and fed the Code4Bench dataset to an AST-based neural network for
feature learning. These features are used as inputs to classification and
regression models to predict a program’s execution time, program ID and
associated runtime, and comparative predictions on program pairs. Experimental
results show that although they achieved high accuracy in execution time
prediction, the LLMs failed to generalize on unseen data when fine-tuned on
Ubisoft production code.

##  9\. Conclusion

Although training developers in writing high-performance code could be a
viable solution, it is outside the scope of our work. Education quality and
level of developers, training opportunities, and willingness to provide at
organizations vary. Furthermore, training takes up significant financial and
human resources while simultaneously prone to human-caused error. Thus, this
paper presented PACE, a program analysis framework that learns and maps
program features the test time microbenchmarks for continuous performance
prediction.

##  10\. Acknowledgements

This research was supported in part by UMass Dartmouth’s Marine and Undersea
Technology (MUST) Research Program funded by the Office of Naval Research
(ONR) under Grant No. N00014-23-1–2141. The views and conclusions expressed in
this paper are those of the authors and do not reflect the official policy or
position of the University of Massachusetts Dartmouth, the Office of Naval
Research, U.S. Navy, U.S. Department of Defense, or U.S. Government.

## References

  * (1)
  * Abbasi and Chen (2008) Ahmed Abbasi and Hsinchun Chen. 2008.  Writeprints: A stylometric approach to identity-level identification and similarity detection in cyberspace.  _ACM Transactions on Information Systems (TOIS)_ 26, 2 (2008), 1–29.  https://doi.org/10.1145/1344411.1344413
  * Abuhamad et al. (2018) Mohammed Abuhamad, Tamer AbuHmed, Aziz Mohaisen, and DaeHun Nyang. 2018.  Large-scale and language-oblivious code authorship identification. In _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ (Toronto, Canada). 101–114.  https://doi.org/10.1145/3243734.3243738
  * Andolfi et al. (2000) F Andolfi, Federica Aquilani, Simonetta Balsamo, and Paola Inverardi. 2000.  Deriving performance models of software architectures from message sequence charts. In _Proceedings of the 2nd International Workshop on Software and Performance_ (Ottawa, Ontario, Canada). 47–57.  https://doi.org/10.1145/350391.350404
  * Aquilani et al. (2001) Federica Aquilani, Simonetta Balsamo, and Paola Inverardi. 2001.  Performance analysis at the software architectural design level.  _Performance Evaluation_ 45, 2-3 (2001), 147–178.  https://doi.org/10.1016/S0166-5316(01)00035-9
  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013\.  Representation learning: A review and new perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 35, 8 (2013), 1798–1828.  https://doi.org/10.1109/TPAMI.2013.50
  * Bernardi et al. (2002) Simona Bernardi, Susanna Donatelli, and José Merseguer. 2002\.  From UML sequence diagrams and statecharts to analysable petri net models. In _Proceedings of the 3rd international workshop on Software and performance_ (Rome, Italy). 35–45.  https://doi.org/10.1145/584369.584376
  * Biringa et al. (2023) Chidera Biringa, Baye Gaspard, and Gokhan Kul. 2023\.  Short Paper: Static and Microarchitectural ML-Based Approaches For Detecting Spectre Vulnerabilities and Attacks. In _Proceedings of the 11th International Workshop on Hardware and Architectural Support for Security and Privacy_ (Chicago, IL, USA) _(HASP ’22)_. Association for Computing Machinery, New York, NY, USA, 53–57.  https://doi.org/10.1145/3569562.3569589
  * Biringa and Kul (2021) Chidera Biringa and Gökhan Kul. 2021.  Automated User Experience Testing through Multi-Dimensional Performance Impact Analysis. In _Proceedings of the 2021 IEEE/ACM International Conference on Automation of Software Test (AST)_ (Madrid, Spain). IEEE, 125–128.  https://doi.org/10.1109/AST52587.2021.00024
  * Böck et al. (2023) Markus Böck, Sarra Habchi, Mathieu Nayrolles, and Jürgen Cito. 2023.  Performance Prediction From Source Code Is Task and Domain Specific. In _Proceedings of the 31st IEEE/ACM International Conference on Program Comprehension (ICPC ’23)_ (Melbourne, Australia). IEEE, 35–42.  https://doi.org/10.1109/ICPC58990.2023.00015
  * Caliskan-Islam et al. (2015) Aylin Caliskan-Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt. 2015.  De-anonymizing programmers via code stylometry. In _Proceedings of the 24th $USENIX$ Security Symposium ($USENIX$ Security 15)_ (Washington, D.C., USA). 255–270.
  * Colin and Bernat (2002) Antoine Colin and Guillem Bernat. 2002.  Scope-tree: A program representation for symbolic worst-case execution time analysis. In _Proceedings 14th Euromicro Conference on Real-Time Systems. Euromicro RTS 2002_ (Vienna, Austria). IEEE, 50–59.  https://doi.org/10.1109/EMRTS.2002.1019185
  * Cortellessa and Mirandola (2000) Vittorio Cortellessa and Raffaela Mirandola. 2000.  Deriving a queueing network based performance model from UML diagrams. In _Proceedings of the 2nd International Workshop on Software and Performance_ (Ottawa, Ontario, Canada). 58–70.  https://doi.org/10.1145/350391.350406
  * Costa et al. (2019) Diego Costa, Cor-Paul Bezemer, Philipp Leitner, and Artur Andrzejak. 2019.  What’s wrong with my benchmark results? studying bad practices in JMH benchmarks.  _IEEE Transactions on Software Engineering_ 47, 7 (2019), 1452–1467.  https://doi.org/10.1109/TSE.2019.2925345
  * Cui et al. (2010) Baojiang Cui, Jiansong Li, Tao Guo, Jianxin Wang, and Ding Ma. 2010.  Code comparison system based on abstract syntax tree. In _2010 3rd IEEE International Conference on Broadband Network and Multimedia Technology_ (Beijing, China). IEEE, 668–673.  https://doi.org/10.1109/ICBNMT.2010.5705174
  * Dauber et al. (2018) Edwin Dauber, Aylin Caliskan, Richard Harang, and Rachel Greenstadt. 2018.  Poster: Git blame who?: Stylistic authorship attribution of small, incomplete source code fragments. In _Proceedings of the 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion_ (Gothenburg, Sweden). IEEE, 356–357.
  * de Miguel et al. (2000) Miguel de Miguel, Thomas Lambolais, Mehdi Hannouz, Stéphane Betgé-Brezetz, and Sophie Piekarec. 2000\.  UML extensions for the specification and evaluation of latency constraints in architectural models. In _Proceedings of the 2nd International Workshop on Software and Performance_ (Ottawa, Ontario, Canada). 83–88.  https://doi.org/10.1145/350391.350411
  * Didona et al. (2015) Diego Didona, Francesco Quaglia, Paolo Romano, and Ennio Torre. 2015.  Enhancing performance prediction robustness by combining analytical modeling and machine learning. In _Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering_ (Austin, Texas, USA). 145–156.  https://doi.org/10.1145/2668930.2688047
  * Doran et al. (2017) Derek Doran, Sarah Schulz, and Tarek R Besold. 2017\.  What does explainable AI really mean? A new conceptualization of perspectives.  _arXiv preprint arXiv:1710.00794_ (2017).  https://doi.org/10.48550/arXiv.1710.00794
  * Géron (2019) Aurélien Géron. 2019\.  _Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems_.  O’Reilly Media.
  * Goodrich et al. (2014) Michael T Goodrich, Roberto Tamassia, and Michael H Goldwasser. 2014.  _Data Structures and Algorithms in Java_.  John Wiley & Sons.
  * Guo et al. (2018) Jianmei Guo, Dingyu Yang, Norbert Siegmund, Sven Apel, Atrisha Sarkar, Pavel Valov, Krzysztof Czarnecki, Andrzej Wasowski, and Huiqun Yu. 2018.  Data-efficient performance learning for configurable systems.  _Empirical Software Engineering_ 23 (2018), 1826–1867.  https://doi.org/10.1007/s10664-017-9573-6
  * Hooda and Chhillar (2015) Itti Hooda and Rajender Singh Chhillar. 2015.  Software test process, testing types and techniques.  _International Journal of Computer Applications_ 111, 13 (2015).
  * Kähkipuro (2000) Pekka Kähkipuro. 2000\.  UML-based performance modeling framework for component-based distributed systems.  In _Performance Engineering_. Springer, 167–184.  https://doi.org/10.1007/3-540-45156-0_11
  * Kaltenecker et al. (2020) Christian Kaltenecker, Alexander Grebhahn, Norbert Siegmund, and Sven Apel. 2020.  The interplay of sampling and machine learning for software performance prediction.  _IEEE Software_ 37, 4 (2020), 58–66.  https://doi.org/10.1109/MS.2020.2987024
  * Karper (2014) Aaron Karper. 2014\.  _Efficient regular expressions that produce parse trees_.  epubli GmbH.
  * Kul et al. (2020) Gökhan Kul, Shambhu Upadhyaya, and Andrew Hughes. 2020\.  An analysis of complexity of insider attacks to databases.  _ACM Transactions on Management Information Systems (TMIS)_ 12, 1 (2020), 1–18.  https://doi.org/10.1145/3391231
  * Lancaster and Culwin (2004) Thomas Lancaster and Fintan Culwin. 2004.  A comparison of source code plagiarism detection engines.  _Computer Science Education_ 14, 2 (2004), 101–112.  https://doi.org/10.1080/08993400412331363843
  * Lavallée and Robillard (2015) Mathieu Lavallée and Pierre N Robillard. 2015.  Why good developers write bad code: An observational case study of the impacts of organizational factors on software quality. In _Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE ’15)_ (Florence, Italy), Vol. 1. IEEE, 677–687.  https://doi.org/10.1109/ICSE.2015.83
  * Lee et al. (2003) Byoung-Dai Lee et al. 2003\.  Run-time prediction of parallel applications on shared environments. In _Proceedings of the IEEE International Conference on Cluster Computing_ (Hong Kong, China). IEEE, 487–491.  https://doi.org/10.1109/CLUSTR.2003.1253355
  * Li et al. (2017) Jian Li, Pinjia He, Jieming Zhu, and Michael R Lyu. 2017\.  Software defect prediction via convolutional neural network. In _Proceedings of the 2017 IEEE international conference on software quality, reliability and security (QRS)_ (Prague, Czech Republic). IEEE, 318–328.  https://doi.org/10.1109/QRS.2017.42
  * Li et al. (2006) Jiexun Li, Rong Zheng, and Hsinchun Chen. 2006.  From fingerprint to writeprint.  _Commun. ACM_ 49, 4 (2006), 76–82.  https://doi.org/10.1145/1121949.1121951
  * Lindemann et al. (2002) Christoph Lindemann, Axel Thümmler, Alexander Klemm, Marco Lohmann, and Oliver P Waldhorst. 2002\.  Performance analysis of time-enhanced UML diagrams based on stochastic processes. In _Proceedings of the 3rd International Workshop on Software and Performance_ (Rome, Italy). 25–34.  https://doi.org/10.1145/584369.584375
  * Liu et al. (2021) Weihua Liu, Erh-Wen Hu, Bogong Su, and Jian Wang. 2021\.  Using machine learning techniques for DSP software performance prediction at source code level.  _Connection Science_ 33, 1 (2021), 26–41.  https://doi.org/10.1080/09540091.2020.1762542
  * Louridas (2006) Panagiotis Louridas. 2006\.  Static code analysis.  _$IEEE$ Software_ 23, 4 (2006), 58–61.  https://doi.org/10.1109/MS.2006.114
  * MacDonell et al. (1999) Stephen G MacDonell, Andrew R Gray, Grant MacLennan, and Philip J Sallis. 1999.  Software forensics for discriminating between program authors using case-based reasoning, feedforward neural networks and multiple discriminant analysis. In _Proceedings of the 6th International Conference on Neural Information Processing (ICONIP’99. ANZIIS’99 & ANNES’99 & ACNN’99.)_ (Perth, WA, Australia), Vol. 1. IEEE, 66–71.  https://doi.org/10.1109/ICONIP.1999.843963
  * Menascé and Gomaa (2000) Daniel A Menascé and Hassan Gomaa. 2000.  A method for design and performance modeling of client/server systems.  _IEEE Transactions on Software Engineering_ 26, 11 (2000), 1066–1085.  https://doi.org/10.1109/32.881718
  * Meng and Norris (2017) Kewen Meng and Boyana Norris. 2017.  Mira: A framework for static performance analysis. In _Proceedings of the 2017 IEEE International Conference on Cluster Computing (CLUSTER ’17’)_ (Honolulu, Hawaii, USA). IEEE, 103–113.  https://doi.org/10.1109/CLUSTER.2017.43
  * Meyer (2014) M. Meyer. 2014\.  Continuous Integration and Its Tools.  _IEEE Software_ 31, 3 (2014), 14–16.  https://doi.org/10.1109/MS.2014.58
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).  https://doi.org/10.48550/arXiv.1301.3781
  * Morris et al. (2019) Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. 2019\.  Weisfeiler and leman go neural: Higher-order graph neural networks. In _Proceedings of the AAAI conference on artificial intelligence_ (Honolulu, Hawaii, USA), Vol. 33. 4602–4609.  https://doi.org/10.1609/aaai.v33i01.33014602
  * Mosteller and Wallace (2012) Frederick Mosteller and David L Wallace. 2012.  _Applied Bayesian and classical inference: the case of the Federalist papers_.  Springer Science & Business Media.  https://doi.org/10.1007/978-1-4612-5256-6
  * Neamtiu et al. (2005) Iulian Neamtiu, Jeffrey S Foster, and Michael Hicks. 2005\.  Understanding source code evolution using abstract syntax tree matching. In _Proceedings of the 2005 international workshop on Mining software repositories_ (Saint Louis, Missouri, USA). 1–5.  https://doi.org/10.1145/1082983.1083143
  * Nie et al. (2020) Pengyu Nie, Ahmet Celik, Matthew Coley, Aleksandar Milicevic, Jonathan Bell, and Milos Gligoric. 2020\.  Debugging the Performance of Maven’s Test Isolation: Experience Report _(ISSTA 2020)_. Association for Computing Machinery, New York, NY, USA, 249–259.  https://doi.org/10.1145/3395363.3397381
  * Pandey et al. (2020) Gaurav Pandey, Poonam Chaudhary, Rajan Gupta, and Saibal Pal. 2020.  SEIR and Regression Model based COVID-19 outbreak predictions in India.  _arXiv preprint arXiv:2004.00958_ (2020).  https://doi.org/10.48550/arXiv.2004.00958
  * Pavic et al. (2019) Bojan Pavic, Chris Anstey, and Jeremy Wagner. 2019\.  Why does speed matter?  https://web.dev/why-speed-matters/.
  * Petriu and Woodside (2002) Dorin C Petriu and Murray Woodside. 2002.  Software performance models from system scenarios in use case maps. In _Proceedings of the International Conference on Modelling Techniques and Tools for Computer Performance Evaluation_. Springer, 141–158.  https://doi.org/10.1007/3-540-46029-2_9
  * Ramadan et al. (2021) Tarek Ramadan, Tanzima Z Islam, Chase Phelps, Nathan Pinnow, and Jayaraman J Thiagarajan. 2021\.  Comparative code structure analysis using deep learning for performance prediction. In _Proceedings of the 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_ (Stony Brook, NY, USA). IEEE, 151–161.  https://doi.org/10.1109/ISPASS51385.2021.00032
  * Samoaa et al. (2022) Hazem Peter Samoaa, Antonio Longa, Mazen Mohamad, Morteza Haghir Chehreghani, and Philipp Leitner. 2022.  TEP-GNN: Accurate Execution Time Prediction of Functional Tests Using Graph Neural Networks. In _Proceedings of the 23rd International Conference on Product-Focused Software Process Improvement, PROFES 2022_ (Jyväskylä, Finland). Springer, 464–479.  https://doi.org/10.1007/978-3-031-21388-5_32
  * Samoaa et al. (2023) Peter Samoaa, Linus Aronsson, Antonio Longa, Philipp Leitner, and Morteza Haghir Chehreghani. 2023\.  A Unified Active Learning Framework for Annotating Graph Data with Application to Software Source Code Performance Prediction.  _arXiv preprint arXiv:2304.13032_ (2023).  https://doi.org/10.48550/arXiv.2304.13032
  * Sarker (2021) Iqbal H Sarker. 2021\.  Machine learning: Algorithms, real-world applications and research directions.  _SN Computer Science_ 2, 3 (2021), 1–21.  https://doi.org/10.1007/s42979-021-00592-x
  * Sarnot et al. (2019) Saloni Alias Puja Sarnot, Sanjana Rinke, Rayomand Raimalwalla, Raviraj Joshi, Rahul Khengare, and Purvi Goel. 2019.  Snapcode-a snapshot based approach to code stylometry. In _Proceedings of the 2019 International Conference on Information Technology (ICIT)_ (Bhubaneswar, India). IEEE, 337–341.  https://doi.org/10.1109/ICIT48102.2019.00066
  * Sherief (2014) Nada Sherief. 2014\.  Software evaluation via users’ feedback at runtime. In _Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering_ (London, England, United Kingdom). 1–4.  https://doi.org/10.1145/2601248.2613085
  * Tereszkowski-Kaminski et al. (2021) Michal Tereszkowski-Kaminski, Sergio Pastrana, Jorge Blasco, and Guillermo Suarez-Tangil. 2021.  Towards Improving Code Stylometry Analysis in Underground Forums.  _Proceedings on Privacy Enhancing Technologies_ 2022, 1 (2021), 126–147.  https://doi.org/10.2478/popets-2022-0007
  * Velez et al. (2021) Miguel Velez, Pooyan Jamshidi, Norbert Siegmund, Sven Apel, and Christian Kästner. 2021\.  White-box analysis over machine learning: Modeling performance of configurable systems. In _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE’21)_ (Madrid, Spain). IEEE, 1072–1084.  https://doi.org/10.1109/ICSE43902.2021.00100
  * Wang et al. (2020) Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. 2020.  Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In _2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)_ (Los Alamitos, CA, USA). IEEE, 261–271.  https://doi.org/10.1109/SANER48275.2020.9054857
  * Williams and Smith (2002) Lloyd G Williams and Connie U Smith. 2002.  Five steps to solving software performance problems.  _Software Engineering Research and Performance Engineering Services_ (2002), 115.
  * Woodside et al. (2001) Murray Woodside, Curtis Hrischuk, Bran Selic, and Stefan Bayarov. 2001.  Automated performance modeling of software generated by a design environment.  _Performance Evaluation_ 45, 2-3 (2001), 107–123.  https://doi.org/10.1016/S0166-5316(01)00033-5
  * Zhang et al. (2019) Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019\.  A novel neural source code representation based on abstract syntax tree. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_ (Montreal, QC, Canada). IEEE, 783–794.  https://doi.org/10.1109/ICSE.2019.00086
  * Zhang et al. (2015) Weizhe Zhang, Albert MK Cheng, and Jaspal Subhlok. 2015\.  Dwarfcode: a performance prediction tool for parallel applications.  _IEEE Trans. Comput._ 65, 2 (2015), 495–507.  https://doi.org/10.1109/TC.2015.2417526
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023\.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).  https://doi.org/10.48550/arXiv.2303.18223
  * Zhou et al. (2019) Meiling Zhou, Jie Chen, Haiyang Hu, Jiacheng Yu, Zhongjin Li, and Hua Hu. 2019\.  Deeptle: Learning code-level features to predict code performance before it runs. In _Proceedings of the 26th Asia-Pacific Software Engineering Conference (APSEC ’19)_ (Putrajaya, Malaysia). IEEE, 252–259.  https://doi.org/10.1109/APSEC48747.2019.00042