#  SMOOTHIE: A Theory of Hyper-parameter Optimization for Software Analytics

Rahul Yedida, Tim Menzies R. Yedida and T. Menzies are with the Department of
Computer Science, North Carolina State University, Raleigh, USA.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Hyper-parameter optimization is the black art of tuning a learner’s control
parameters. In software analytics, a repeated result is that such tuning can
result in dramatic performance improvements. Despite this, hyper-parameter
optimization is often applied rarely or poorly in software analytics– perhaps
due to the CPU cost of exploring all those parameter options can be
prohibitive.

We theorize that learners generalize better when the loss landscape is
“smooth”. This theory is useful since the influence on “smoothness” of
different hyper-parameter choices can be tested very quickly (e.g. for a deep
learner, after just one epoch).

To test this theory, this paper implements and tests SMOOTHIE, a novel hyper-
parameter optimizer that guides its optimizations via considerations of
“smothness”. The experiments of this paper test SMOOTHIE on numerous SE tasks
including (a) GitHub issue lifetime prediction; (b) detecting false alarms in
static code warnings; (c) defect prediction, and (d) a set of standard ML
datasets. In all these experiments, SMOOTHIE out-performed state-of-the-art
optimizers. Better yet, SMOOTHIE ran 300% faster than the prior state-of-the
art. We hence conclude that this theory (that hyper-parameter optimization is
best viewed as a “smoothing” function for the decision landscape), is both
theoretically interesting and practically very useful.

To support open science and other researchers working in this area, all our
scripts and datasets are available on-line at
https://github.com/yrahul3910/smoothness-hpo/.

###### Index Terms:

software analytics, smoothness, hyper-parameter optimization

##  1 Introduction

Modern software systems can get very complex. This is especially true for the
learners using in software analytics which are controlled by a dauntingly
large number of hyper-parameters. For example, deep learners come with a very
large number of control parameters including how to set their topology and
activation functions.

Configuring complex learners requires an understanding of the impact of each
configuration parameter. When that expertise is missing, configuration options
are under-utilized [49]. This is especially true for learner hyper-parameters.
While studying industrial text-mining tools, we have found many examples of
poorly-chosen hyper-parameters (which meant our industrial partners were
selling less-than-optimal products) [30].

Hyper-parameter optimizers are algorithms that design better learning
algorithms by exploring the myriad of configuration choices for their hyper-
parameters. As discussed below, these algorithms are not in common use. So, as
a tactic for increasing their usage, we propose (a) simplifying their design;
then (b) showing that this simpler design results in “better” HPO tools (where
“better” means “runs faster” and “makes better predictions”). This paper shows
that those two goals are quite achievable. While working on software analytics
and hyper-parameter optimizations [53], one of us (Yedida) noticed a pattern
in changes to the loss functions before and after hyper-parameter
optimization. Loss functions are the evaluating metric for a machine learner
(and the better the learner, the lower the loss). Figure 1 left and right
shows that:

> After optimization, the loss function gets “smoother’ (i.e. on average,
> fewer changes in any local region).

Based on that observation, we theorized that:

> Hyper-parameter optimization is best viewed as a “smoothing” function for
> the decision landscape.

Figure 1: Left & right shows loss landscape before & after HPO. The loss
function be visualized as the $z$ dimension of domain data mapped into two PCA
components $x,y$. In this figure, red indicates regions with highest mean
squared error.

This theory predicts that the greater the smoothing, the better the hyper-
parameter optimization.

Figure 2: Likelihood of a learner performing best (as a defect predictor).
BLUE shows the performance pre-optimization and RED shows very large
performance improvements (post-optimization). From [43].

This paper tests that theory. The algorithm we call SMOOTHIE is an experiment
on whether or not increasing smoothness could be used to guide hyper-parameter
optimization (HPO). As shown in this paper, SMOOTHIE can assess hyper-
parameter options without needing to run a learner to completion. For example,
for neural networks that learn over 100 epochs, SMOOTHIE can assess hyper-
parameter options within the first epoch. That is, SMOOTHIE can focus on good
configurations by very quickly rejecting bad configurations using a fraction
of the CPU cost of conventional hyper-parameter optimizers.

This paper’s experiments asks two research questions:

  * •

RQ1: What are the benefits of smoothing?. Here we look at how hyper-parameter
optimization guided by smoothing affects predictive accuracy for

    * –

GitHub issue lifetime prediction;

    * –

Detecting false alarms in static code warnings;

    * –

Defect prediction,

    * –

A set of tabular datasets used in the NeurIPS 2020 Black-Box Optimization
Challenge.

In all our experiments SMOOTHIE generated models yielding better predictions
than those found by state-of-the-art optimizers

  * •

RQ2: What are the costs of smoothing? Here we report the CPU overheads
associated with this technique. We show that smoothing properties of different
hyper-parameter optimizations can be assessed very quickly; e.g. for a neural
net run for 100 epochs, smoothness effects can be assessed after just one
epoch.

Hence we say that core contributions of this work are

  * •

We introduce the concept of smoothness for guiding hyperparameter
optimization;

  * •

We show smoothness-based HPO is very effective.

  * •

We show smoothness-based HPO is very fast.

  * •

We show that smoothness-based HPO offers new high-water mark results for
several software analytics tasks:

Based on these experiments, we conclude that smoothness is the missing piece
of the hyper-parameter optimization puzzle. All too often, HPO is implemented
as a blind search through a large space of options (like randomly hitting a
broken car with a hammer, until the car starts working). On the other hand,
smoothness offers:

  * •

A theory of why HPO works (simplifying the loss landscape);

  * •

As well as clues on how to implement very fast HPO (given many options, do a
tiny piece of inference to reject options that do not seem to have an effect
on smoothness).

We hence suggest to the software analytics community that they do more hyper-
parameter optimization, faster and simpler, using smoothness-guided algorithms
like SMOOTHIE.

To support open science and other researchers working in this area, all our
scripts and datasets are available on-line at
https://github.com/yrahul3910/smoothness-hpo/.

##  2 Background

In 2020, we predicted that hyper-parameter optimization would be standard
practice in software analytics [3]. In part, that argument was based on the
performance improvements found via optimization [21, 2, 51, 53]. For example,
Figure 2 shows a Tantithamthavorn et al. [43] study comparing defect predictor
performance, before and after hyper-parameter optimization. We highlight the
“C5.0" results shown bottom left: before optimization, this algorithm was
always worst; but afterwards, it defeated everything else.

Another reason we argued for hyper-parameter optimization was that:

> “It can be misleading to report conclusions from an untuned learner since
> those conclusions can changed by tuning” [3].

For example, LDA is a text mining algorithm that finds “topics” (patterns of
words that occur together). When run on SE data, LDA suffers from “order
effects”, i.e. the reported topics can change if the input is re-ordered [1].
LDA’s order effects can be mitigated by hyper-parameter optimization but in
doing so, different topics will be reported [1].

Despite these arguments, at the time of this writing (2024), hyper-parameter
optimization is not standard practice in software analytics. For example,
Figure 3 shows a sample of papers that explore deep learning in the SE defect
prediction literature. 91% of those papers do not perform optimization or, if
they do, they do it poorly (where “poorly” means using a suboptimal or trivial
optimizer like grid search or random search). This is strange since it is
well-documented that hyper-parameter optimization can improve deep learning
[14, 33, 8, 17, 16, 5].

|
---|---
Fig. 3a: 154 papers found in this literature review, related to deep learning and SE. |  Fig. 3b: Only $\frac{14}{154}\approx 9\%$ of the Fig. 3a papers have “good” hyper-parameter optimization studies (where “good” used some state-of-the-art optimizer– i.e. definitely not grid or random search).
Figure 3: Deep learning papers from the literature. Found with the Google
Scholar query “deep learning AND software”, “deep learning AND bug”, “deep
learning AND defect”, “deep learning AND issue”. We also set up Google Scholar
alerts for major AI and SE venues, and collected papers from them. Papers were
added via backward snowballing (see Wohlin [48]), restricting that backwards
search to papers published in the 40 top venues listed in Google Scholar
Metrics for “Artificial Intelligence” and “Software Systems”. Papers were
excluded if they were over three years old and had less than 10 citations per
year. Papers were also excluded it they were published before 2016. We used
2016 since, in that year, there were two prominent SE papers from ICSE [43]
and the IST journal [20] reporting on the merits of hyper-parameter optimizers
for software analytics.

The most obvious explanation for the lack of hyper-parameter optimization in
software analytics is its associated computational cost. Consider all the
hyper-parameter options shown in Table I. Suppose that (a) they are combined
with popular preprocessing methods (e.g. feature and instance weighting or
encoders to change the dimensionality), and (b) the resulting systems are each
run 10 times (for statistical validity). The result evaluation rig would be
non-trivial to code and require 1,000,000 executions of a learner to assess
50,000 possible configurations.

As shown by the experiments of this paper, the tool called SMOOTHIE can
significantly reduce that computational cost.

TABLE I: Some hyper-parameters from software analytics. Deep learners have
many number of control parameters including how to set their topology and
activation functions.
---
Neighbor classifiers need to know (a) how to measure distance; (b) how many
neighbors to use; (c) how to combine those neighbors; (d) how (or if) to pre-
cluster the data (to optimizer th neighborhood lookup); etc.
Random forests need to know how many trees to build, how many features to use
in each tree, and how to poll the whole forest (e.g., majority or weighted
majority).

##  3  Hyper-parameter Optimization (HPO)

###  3.1 HPO in software analytics

Here, we offer details of some case studies with SE and HPO. The following
list is hardly exhaustive, but it does detail some of the more recent work in
this area. Also, this section serves to introduce the examples used later in
this paper.

Figure 4: A brief overview of hyper-parameter optimization methods. For a more
in-depth review, see [42, 19, 9].

Defect prediction. Software quality assurance budgets are finite while
assessment effectiveness increases exponentially with assessment effort [21].
Therefore, standard practice is to apply slower methods on code sections that
seem most interesting (most critical or bug-prone). Defect prediction is a
fast way to learn these interesting regions using static code attributes. A
2018 study [44] of 395 practitioners across 33 countries found that over 90%
were willing to adopt defect prediction models in their workflow. Recently,
Yedida and Menzies [51] proposed “fuzzy sampling” (see Section 5.1 for a
discussion) and combined it with stacked autoencoders, SMOTE, hyper-parameter
optimization (using DODGE [2]), and a weighted loss function to achieve state-
of-the-art results on defect prediction using the PROMISE repository [10],
which contains 20 static code attributes of the software such as the depth of
inheritance tree [13], and mean McCabe cyclomatic complexity [37]. In this
study, we use the same 6 datasets as that study for a fair comparison.

Issue lifetime prediction. Issue lifetime predictors learn predictors for how
long it takes to address issues raised by other developers (or end-users).
Predicting issue close time has multiple benefits for developers, managers,
and stakeholders, including task prioritization and resource allocation. The
task is interesting since in open source software, developers triage their
work according to the difficulty of their work; i.e. they will explore many
small tasks before tackling harder tasks that are slower to accomplish. Hyper-
parameter optimization has been successfully used for issue lifetime
prediction, see Yedida et al. [52]. For this study, we use DeepTriage [35] as
a baseline, whose results we could fully reproduce. DeepTriage uses a
bidirectional LSTM with attention after removing special characters,
tokenization, and pruning.

We use the same datasets, which contain mined data from Bugzilla for three
popular software projects: Firefox, Chromium, and Eclipse. For simplicity,
although the data contains user comments and system records (both of which are
text data sources), we do not use those features in this study. Lee et al.
[32] proposed binning the data into $k\in\\{2,3,5,7,9\\}$ classes and test the
performance of classifiers under the binary and multi-class settings. For
brevity, we show results for 2 and 3 classes.

Static code analysis false alarms. Static analysis (SA) tools report errors in
source code, without needing to execute that code (this makes them very
popular in industry). SA tools generate many false alarms that are ignored by
practitioners. Previous research work shows that 35% to 91% of SA warnings
reported as bugs by SA tools are routinely ignored by developers [25, 24].
Hence many researchers add a machine learning post-processor that reports the
“alarms you should not ignore” (learned from the historical log of what errors
do/do not result in developers changing the code) [45, 50, 53]. Initial
results from Yang et al. [50] suggested that it was possible to find the
actionable static code warnings generated by a static analysis tool like
FindBugs, using a very simple learner. But later work by Kang et al. [28]
showed that the datasets used by the former study had errors, and once
manually repaired, those simple methods did not produce good results. This was
later fixed by Yedida et al. [53], using hyper-parameter optimization that
explored combinations of pre-processing methods. That study was an important
pre-cursor to this paper since it was here that Yedida first observed the
smoothing effect of Figure 1.

###  3.2 HPO: the algorithms

The experiments of this paper will compare SMOOTHIE with the BOHB surrogate-
based Bayesian optimizer. This section explains why we use that algorithm.

Having documented that HPO is useful, the next question is how to best
implement that kind of analysis. Figure 4, we show the broad classifications
of HPO algorithms, along with a few notable examples. The next few paragraphs
summarize the main algorithms (and for further details and recent reviews on
HPO, see [42, 19, 9]).

Bergstra et al. [8] discussed sequential model-based optimization, Tree of
Parzen Estimators (TPE), and random search, the last of which was subsequently
discussed in Bergstra and Bengio [7]. In particular, Bergstra and Bengio [7]
demonstrated the effectiveness of random search under a limited computational
budget, additionally showing that for most problems, only a few hyper-
parameters matter, although that subset of important hyper-parameters is
different across datasets. Bayesian optimization [38] is a sequential model-
based optimization algorithm that improves as it gathers more data. It
involves a surrogate modeling for the objective function and an acquisition
function that samples configurations. Surrogates are a model learnt from just
a few examples which are used to assess all the examples. These surrogates may
be inaccurate (since they are built from very little data). Nevertheless, they
can be used to judge if one set of options is better than than another [39].
Hence, surrogates can serve as a cheap “acquisition function” to find
interesting examples of evaluate next.

A widely-used surrogate method are Gaussian Process models [47]. Various
acquisition functions have been proposed, such as Expected Improvement [27],
Gaussian Process Upper Confidence Bound (GP-UCB) [31], and predictive entropy
search [26]. Among these, Expected Improvement is the most popular choice. Of
the various BO-based methods, Falkner et al. [17] combines Bayesian
optimization and Hyperband [33], producing BOHB, and has emerged as the most
popular surrogate-based algorithm combining these two due to it being open-
source and state-of-the-art. The BO closely resembles TPE with a single multi-
dimensional kernel density estimate [54]. In a large-scale study, Eggensperger
et al. [15] endorsed multi-fidelity methods (such as BOHB) over their black-
box counterparts in constrained budget settings. This result was corroborated
by a more recent work by Pfisterer et al. [41], which showed that in
constrained budget settings, multi-fidelity methods outperformed full-fidelity
methods such as SMAC [34].

##  4 About SMOOTHIE

The experiments of this paper aim to check if, for a wide range of data, there
is any value in turning “bumpy” landscapes (like Figure 1, left-hand-side)
into “smoother” landscapes like Figure 1 (right-hand-side). To conduct such
experiments, we used the “smoothing” tool described in this section.

SMOOTHIE begins by sampling a few random configurations. Then, we compute the
smoothness for each configuration using the methods described in this section.
But before that, we make the following important points. Our definition of
smoothness is general to many learners (See Equation 3). However, that
definition must be realized in different ways for different learners.

  * •

In some cases smoothness can be computed just by looking at certain properties
of the data (in Naive Bayes, that is the covariance matrix of the data, see
Theorem 2). In this first case, HPO is very fast– just peek at the data to
rank and prune the operators of (say) §5.1.

  * •

In other case, as in neural networks (see Theorem 9), smoothness changes by
the inference process. In this latter case, we have to watch the learner
running (at least for a limited time) before smoothness can be
computed)111This brings the weights closer to the final weights at an
exponential rate, assuming the loss is also strongly convex. It is out of
scope of this paper to show that the loss functions commonly used are indeed
both smooth and strongly convex..

Input : Number of configs to sample $N_{1}$, number of configs to run $N_{2}$.
Defaults: $N_{1}=30,N_{2}=5$.

Output : A near-optimal configuration

1 $\mathcal{H}_{0}\leftarrow$ Random($\mathcal{H},N_{1}$);

2 $S\leftarrow\phi$;

3 $P\leftarrow\phi$;

4 for  _each config $h$ in $\mathcal{H}_{0}$_ do

5 S[config] $\leftarrow$ Get-Smoothness(h)

6 end for

7for  _each config in Top(S, $N_{2}$)_ do

8 P[config] $\leftarrow$ Run(config);

9

10 end for

11

12return _$\operatorname*{arg\,max}P$_ ;

Algorithm 1 SMOOTHIE

Algorithm 1 shows the pseudo-code for SMOOTHIE. We first pick $N_{1}$ ($=30$)
configurations at random. For each of those configurations, we compute the
smoothness and store it. Next,we pick the top $N_{2}$ ($=5$) of those
configurations by smoothness values, and run only those, getting the
performance for each. The algorithm then return the best-performing
configuration.

When talking to colleagues about SMOOTHIE, we are asked why use smoothness to
rank hyper-parameter operators and not (say) accuracy. In reply, we make two
points:

  * •

“Ranking by accuracy” is just synonymous for standard practice in HPO; i.e.
run all the learners to completion and return the hyper-parameters that lead
to best performance. Such standard practice incurs a very large CPU cost–
which is the exact effect we seek to remove.

  * •

A variant of “ranking by accuracy” might be to assess HPO operators after some
limited inference (e.g. after only one epoch of a 100 epoch neural net
learning cycle). We tried that, and the learner performance achieved in this
manner always under-performed the other options studied in this paper.

The rest of this section shows the Get-Smoothness calculation used in
Algorithm 1.

###  4.1 Preliminaries

Our description of SMOOTHIE uses the following notation.

For any learner, $X,y$ will represent a training set containing multiple
independent values $X$ and a single goal $y$. In neural networks, we also use
$W$ to denote the weights on the links between neurons. We also say:

  * •

$m$ is the number of training samples;

  * •

$n$ is the number of features;

  * •

$k$ is the number of classes, i.e. possible values for $y$;

  * •

$E$ denotes the loss function.

For a feedforward neural network, $L$ represents the number of layers. In that
learner, at each layer, the following computation is performed:

$\displaystyle z^{[l]}$ $\displaystyle=W^{[l]T}a^{[l-1]}+b^{[l]}$ (1)
$\displaystyle a^{[l]}$ $\displaystyle=g^{[l]}(z^{[l]})$ $\displaystyle
a^{[0]}$ $\displaystyle=X$

Here, $b^{[l]}$ is the bias vector at layer $l$, $W^{[l]}$ is the weight
matrix at layer $l$, and $g^{[l]}$ is the activation function used at layer
$l$ (we assume ReLU $g(x)=\max(0,x)$). The result from applying the activation
function is called the activations at layer $l$, and is denoted by $a^{[l]}$.
In this text, $\nabla_{W}f$ refers to the first gradient of $f$, with respect
to $W$. Finally, $\nabla^{2}_{W}f$ denotes the second gradient, also called
the Hessian.

Our theory and experiments are based on the space $\mathbb{R}^{d}$, which is
complete and separable. As such, it is guaranteed that (a) the space is
measurable (b) if gradient descent converges, then its limit point exists in
the space (c) distances as expressed via norms (the length of a vector),
exist222Because $\mathbb{R}^{d}$ is a metric space; specifically, it is a
Hilbert space.. In the following, we use $\sup f$ to denote the least upper
bound (supremum) of $f$.

For a function $f$, we denote the gradient with respect to one of its
parameters $W$ as $\nabla_{W}f$. The gradient takes various shapes, but
generally is a tensor containing partial derivatives of $f$ with respect to
each element of $W$. Intuitively, these partial gradients are the rate of
change of $f$ along each dimension of $W$.

An interesting use case for the gradient is the $L-$Lipschitzness, which can
be defined as $\sup\lVert\nabla_{W}f\rVert$333In fact, it can be shown that a
differentiable function $f$ is $L-$Lipschitz iff $\lVert\nabla f\rVert\leq L$
. More strictly, a function $f$ is said to be $L-$Lipschitz if there exists a
constant $L$ such that

$\centering\frac{\lVert f(y)-f(x)\rVert}{\lVert y-x\rVert}\leq
L\@add@centering$ (2)

for all $x,y$ in the domain of $f$. The Lipschitz constant gives an idea of
the curvature bound of the function under review.

The second gradient (the Hessian) is denoted by $\mathcal{H}=\nabla^{2}f$. In
the strictest sense, the Hessian is a square tensor of second-order partial
derivatives, and is symmetric (by Schwarz’s theorem). A continuously
differentiable function $f$ is said to be $\beta-$smooth if its gradient is
$\beta-$Lipschitz, or

$\frac{\lVert\nabla f(y)-\nabla f(x)\rVert}{\lVert y-x\rVert}\leq\beta$ (3)

From the mean-value theorem, there exists $v$ such that

$\frac{\lVert\nabla f(y)-\nabla f(x)\rVert}{\lVert
y-x\rVert}=\lVert\nabla^{2}f(v)\rVert\leq\sup\limits_{v}\lVert\nabla^{2}f(v)\rVert$

(Note that this computation assumes the loss function is twice-continuously
differentiable.)

Hence, $\sup\limits_{v}\lVert\nabla^{2}f(v)\rVert$ is such a $\beta$. Since
$\beta$ is the least such constant,
$\beta\leq\sup\limits_{v}\lVert\nabla^{2}f(v)\rVert$.

###  4.2 Smoothness for Feedforward Classifiers

This section shows the smoothness computation for different learners. We prove
the result for feedforward classifiers here; for other derivations (Naive
Bayes, logistic regression), please see the Appendix.

######  Lemma 1.

For a deep learner with ReLU activations in the hidden layers, the last two
layers being fully-connected, and a softmax activation at the last layer,

$\dfrac{\partial E}{\partial
z_{j}^{[L]}}\leq\frac{k-1}{km}\left(\sum\limits_{i=1}^{m}[y^{(i)}=j]\right)$

under the cross-entropy loss.

###### Proof.

We will use the chain rule, as follows:

$\displaystyle\frac{\partial E}{\partial z^{[L]}_{j}}$
$\displaystyle=\frac{\partial E}{\partial a^{[L]}_{j}}\cdot\frac{\partial
a^{[L]}_{j}}{\partial z^{[L]}_{j}}$ (4)

Consider the Iverson notation version of the general cross-entropy loss:

$E(\textbf{a}^{[L]})=-\frac{1}{m}\sum\limits_{i=1}^{m}\sum\limits_{h=1}^{k}[y^{(i)}=h]\log
a_{h}^{[L]}$

Then the first part of (4) is trivial to compute:

$\frac{\partial E}{\partial
a_{j}^{[L]}}=-\frac{1}{m}\sum\limits_{i=1}^{m}\frac{[y^{(i)}=j]}{a_{j}^{[L]}}$
(5)

The second part is computed as follows.

$\displaystyle\frac{\partial a^{[L]}_{j}}{\partial z^{[L]}_{p}}$
$\displaystyle=\frac{\partial}{\partial
z^{[L]}_{p}}\left(\frac{e^{z^{[L]}_{j}}}{\sum_{l=1}^{k}e^{z^{[L]}_{l}}}\right)$
$\displaystyle=\frac{[p=j]e^{z^{[L]}_{j}}\sum_{l=1}^{k}e^{z^{[L]}_{l}}-e^{z^{[L]}_{j}}\cdot
e^{z^{[L]}_{p}}}{\left(\sum_{l=1}^{k}e^{z^{[L]}_{l}}\right)^{2}}$
$\displaystyle=\frac{[p=j]e^{z^{[L]}_{j}}}{\sum_{l=1}^{k}e^{z^{[L]}_{l}}}-\frac{e^{z^{[L]}_{j}}}{\sum_{l=1}^{k}e^{z^{[L]}_{l}}}\cdot\frac{e^{z^{[L]}_{p}}}{\sum_{l=1}^{k}e^{z^{[L]}_{l}}}$
$\displaystyle=\left([p=j]a^{[L]}_{j}-a^{[L]}_{j}a^{[L]}_{p}\right)$
$\displaystyle=a^{[L]}_{j}([p=j]-a^{[L]}_{p})$ (6)

Combining (5) and (6) in (4) gives

$\frac{\partial E}{\partial
z^{[L]}_{j}}=\frac{1}{m}\left(\sum\limits_{i=1}^{m}[y^{(i)}=j]\right)\left(\sum\limits_{h=1}^{k}a^{[L]}_{j}-[h=j]\right)$
(7)

As a sanity check, note that $E$ is a scalar and $z^{[L]}_{j}$ is a vector
(over all the samples); the right-hand side is also a vector (by vectorizing
the Iverson term $[h=j]$). It is easy to show that the limiting case of this
is when all softmax values are equal and each $y^{(i)}=p$; using this and
$a^{[L]}_{j}=\frac{1}{k}$ in (7) and combining with (4) gives us our desired
result:

$\sup\limits_{j}\left\lVert\frac{\partial E}{\partial
z^{[L]}_{j}}\right\rVert=\frac{k-1}{km}\left(\sum\limits_{i=1}^{m}[y^{(i)}=j]\right)$
(8)

∎

######  Theorem 1 (Smoothness for feedforward networks).

For a deep learner with ReLU activations in the hidden layers, the last two
layers being fully-connected, and a softmax activation at the last layer, the
smoothness of the cross-entropy loss is given by

$\sup\lVert\nabla^{2}_{W}E\rVert\propto\frac{k-1}{km}\sup\frac{\lVert
a^{[L-1]}\rVert}{\lVert W^{[L]}\rVert}$ (9)

To say that another way, all that is required is collect the number of samples
$m$ and the number of classes $k$, then run a neural net $N_{1}$ times, for
one epoch (each time using a randomly selected hyper-parameter set). Over
those $N_{1}$ trials, report

  * •

The norm of the activation at the penultimate layer;

  * •

The norm of the weights at the last layer;

  * •

The ratio of these norms.

Lastly, we return the configuration associated with the maximum of this ratio.

###### Proof.

Lemma 1 gives us (using the chain rule one step further):

$\nabla_{W}E\leq\frac{(k-1)}{km}\left(\sum\limits_{i=1}^{m}[y^{(i)}=j]\right)a_{j}^{[L-1]}$

The $\left(\sum\limits_{i=1}^{m}[y^{(i)}=j]\right)$ term is constant, so we
drop it as a proportionality constant. In the following, we use $\leq$ and $=$
to mean “less than equal to the proportionality constant times” and
“proportional to” respectively. Therefore,

$\displaystyle\nabla^{2}_{W}E$
$\displaystyle\leq\frac{k-1}{km}\nabla_{W}a_{j}^{[L-1]}$
$\displaystyle=\frac{k-1}{km}\dfrac{\partial a_{j}^{[L-1]}}{\partial
E}\dfrac{\partial E}{\partial w_{ij}}$
$\displaystyle\leq\frac{(k-1)^{2}a_{j}^{[L-1]}}{k^{2}m^{2}}\dfrac{\partial
a_{j}^{[L-1]}}{\partial E}$ using chain rule on Lemma 1
$\displaystyle=\frac{(k-1)^{2}a_{j}^{[L-1]}}{k^{2}m^{2}}\frac{1}{\dfrac{\partial
E}{\partial a_{j}^{[L-1]}}}$

Using $z^{[l]}=W^{[l]T}a^{[l-1]}+b^{[l]}$ and Lemma 1 gives us:

$\displaystyle\dfrac{\partial E}{\partial a_{j}^{[L-1]}}$
$\displaystyle=\dfrac{\partial E}{\partial z_{j}^{[L]}}\dfrac{\partial
z_{j}^{[L]}}{\partial a_{j}^{[L-1]}}\leq\frac{(k-1)}{km}\lVert W^{[L]}\rVert$

from which (9) follows. ∎

Note: It is typical to use a regularization term in addition to the standard
loss function above. In the appendix, we show that the addition of the
regularizer only adds a constant term to the smoothness, and as such, it can
be disregarded.

###  4.3 Smoothness for Logistic Regression

######  Corollary 1.1 (Smoothness of logistic regression).

$\beta=\frac{k-1}{km}\sup\frac{\lVert X\rVert}{\lVert W\rVert}$

###### Proof.

Logistic regression can be viewed as a feedforward network with no hidden
layers. Hence, the smoothness for logistic regression follows from Equation
(9). ∎

To say that another way, the smoothness computation for logistic regression is
particularly easy: after training the learner, the practitioner simply
computes the norm of the input data and the weights after that single epoch of
training and uses them to compute the smoothness. For simplicity, the constant
fraction term can be dropped from computation.

###  4.4 Smoothness for Naive Bayes

For the proof of the following, please see the appendix.

######  Theorem 2 (Smoothness for Naive Bayes).

For Naive Bayes, the smoothness is given by

$\beta=\sup\lVert\boldsymbol{\Sigma}^{-1}\mathcal{A}G-\frac{1}{2}\boldsymbol{\Sigma}^{-1}\mathcal{A}\boldsymbol{\Sigma}^{-1}+G\mathcal{A}\boldsymbol{\Sigma}^{-1}\rVert$

where $\boldsymbol{\Sigma}$ is the covariance matrix, $\mathcal{A}$ is the
fourth-order identity tensor, and
$G=\dfrac{\partial\ell}{\partial\boldsymbol{\Sigma}}$, where $\ell$ is the
log-likelihood.

Notice that the smoothness is dependent only on the data $X$, means of the
classes $\mu_{i}$, and the covariance matrix $\Sigma$), the latter two of
which can also be derived from the dataset.

To say that another way, in practice, one would first extract the means and
covariance matrix, and first use those to compute the gradient of the log-
likelihood $G$ (see the proof for details). Then, this gradient along with the
covariance matrix can together be used to compute the Hessian, and therefore,
the smoothness.

###  4.5 Smoothness for other learners

While it is possible to use the methodology used in this paper to derive the
smoothness for other classical learners, we leave that to future work. Right
now we are experimenting with a non-parametric form of all the above where
some recursive cluster method, or a decision tree learner, divides the data
into a tree with small leaves. Then we report smoothness by computing the
difference between neighbouring leaves and propagating that change up the tree
using a center difference approximation:

$(\nabla^{2}E)(x)\approx\frac{E(x+h)-2E(x)+E(x-h)}{h^{2}}$

where $E$ is an impurity measure and the approximation error is
$\mathcal{O}(h^{2})$.

Preliminary results are promising but, at this time, we have nothing
definitive to report.

##  5 Experimental Rig

To test SMOOTHIE, we apply the following rig.

###  5.1 Hyper-parameter Space

This section discusses the hyper-parameter options explored in the experiments
of this paper. We make no claim that these are an exhaustive list of possible
hyper-parameter options. However, in defense on the options we explored, we
note that when combined with SMOOTHIE, these options found better predictors
than those seen after using other hyper-parameter optimizers widely-applied in
the literature.

Some of our pre-processors are simple:

  * •

normalization: $x\to\frac{x}{\lVert x\rVert}$;

  * •

standardization: $x\to\frac{x-\mu_{x}}{\sigma_{x}}$;

  * •

min-max scaling: $x\to\frac{x-\min x}{\max x-\min x}$;

  * •

max-absolute value scaling: $x\to\frac{x}{\max\lvert x\rvert}$);

  * •

robust scaling: $x\to\frac{x-P_{50}(x)}{P_{75}(x)-P_{25}(x)}$ where $P_{i}(x)$
represents the $i$th percentile of x.

Various authors recommend one or more of these simpler pre-processors. For
example, for instance-based methods Aha et al. [4] recommend min-max scaling
since it means numbers on different scales can be compared.

Another widely used pre-processor is SMOTE [12]. In order to handle unbalanced
training data, SMOTE generates synthetic minority samples by extrapolating
along line segments joining the nearest neighbors of the minority class.

A recently proposed oversampling technique is “fuzzy sampling” [51], which
concentrically adds $\frac{1/n}{2^{i}}$ samples at concentric layer $i$, for
$\left\lfloor{\log_{2}{1/n}}\right\rfloor$ layers, where $n$ is the class
imbalance ratio. Unlike other oversampling techniques, fuzzy sampling reverses
the class imbalance. The authors show that applying it twice can also be
beneficial. Moreover, they recommend the use of SMOTE after fuzzy sampling
(whether once or twice).

We borrow the following label smoothing technique from [53]. At random,
$m-\sqrt{m}$ labels are “lost”. These are then recovered by building a kd-tree
[6] on the remaining $\sqrt{m}$ samples. For each lost label, the kd-tree is
queried for the nearest $\sqrt[4]{m}$ samples, and the mode of their labels is
assigned.

The above gives us 60 options. On top of these 60 options, we include the
learner-specific hyper-parameters: (a) for feedforward networks, 3 to 20
layers and 2 to 6 units per layer; (b) for logistic regression, the use of
$L_{1}$, $L_{2}$, or both regularization terms, $C\in\\{0.1,1,10\\}$. We do
not explore learner-specific hyper-parameters for Naive Bayes. When combined
with the data pre-processors, that results in $(18\times 5+3\times 3+1)\times
60=6,000$ different hyper-parameter options. For feedforward networks using
the Adam [29] optimizer, Bayesmark uses a hyper-parameter space of size 134M.
Fortunately, many of these options have similar effects. We show below that
within a random selection of 30, it is possible to find settings that
significantly improve the performance of an off-the-shelf learner.

###  5.2 Data

For all our datasets, we compare against a state-of-the-art Bayesian
optimization algorithm, BOHB, random search, and the current state-of-the-art
for those tasks. For SMOOTHIE, we try all the learners whose smoothness value
we derived; we use the best-performing learner as the base for random search
and BOHB. In those trials, we compared to BOHB after $N_{1}$ and $N_{2}$
evaluations.

Table II describes the datasets used in this study. Clearly, our study
encompasses a variety of data shapes: both small and large, and both wide and
narrow. In particular, for static code warnings, the number of features varies
by dataset because of the data collection process from FindBugs; see [28] for
a detailed summary of the data cleaning process.

TABLE II: Dataset sizes for each of the tasks in this study. Defect prediction [51] |
---|---
Dataset | #Train (%defects) | #Test (%defects) | #Features
camel | 1819 (21) | 442 (64) | 20
ivy | 352 (22) | 352 (11)
log4j | 244 (29) | 205 (92)
synapse | 379 (20) | 256 (34)
velocity | 410 (70) | 229 (34)
xalan | 2411 (38) | 909 (99)
Static code warnings [53] |
Dataset | #Train | #Test | #Features
ant | 28 | 7 | 329
cassandra | 13 | 4 | 265
commons | 17 | 5 | 127
derby | 366 | 92 | 292
jmeter | 14 | 4 | 287
lucene-solr | 24 | 6 | 278
tomcat | 147 | 37 | 284
Issue lifetime prediction [52, 32] |
Dataset | #Train | #Test | #Features
Eclipse | 44,545 | 25,459 | 12
Firefox | 44,800 | 25,201
Chromium | 44,801 | 25,200
Bayesmark datasets |
Dataset | #Train | #Test | #Features
breast | 455 | 114 | 30
digits | 1438 | 359 | 64
iris | 120 | 30 | 4
wine | 142 | 36 | 13
diabetes | 354 | 88 | 10

####  5.2.1 Train and Test Sets

For all experiments, we split the data into train and test partitions, and
report results on the test set. We use the splitting policies seen in prior
work:

  * •

For defect prediction, it is standard practice to train on data from versions
$1\ldots n-1$ and test on version $n$ [2];

  * •

For issue lifetime prediction, we split the data 75/25;

  * •

For static code analysis, we split the data 80/20.

  * •

Bayesmark splits the dataset into 80/20 internally.

###  5.3 Evaluation Metrics

For each kind of data set, we use evaluation measures seen in prior papers
that explored that data:

  * •

For issue lifetime prediction, as done in prior work [32] the class variable
was divided into $k\in\\{2,3\\}$ equal frequency bins. We use the same metrics
as [32], i.e. accuracy. If the true negative, false negative, false positive
and true positive counts are $a,b,c,d$ then

$\mathit{accuracy}=(a+d)/(a+b+c+d)$

  * •

For defect prediction, as done in prior work [46, 51], we used the F1
evaluation metric that commented not just on how many defects were were found
but also, of the predicted defects, how many were true defects:

$\begin{array}[]{rcccl}\mathrm{recall}&=&r&=d/(b+d)\\\
\mathrm{precision}&=&p&=d/(c+d)\\\
\mathrm{harmonic\;mean}&=&\mathit{F1}&=2rp/(r+p)\end{array}$

  * •

For static code false alarms, as done in prior work [53], we use recall,
precision, and false alarm:

$\mathrm{false\ alarm}=c/(a+c)$

To test the generality of SMOOTHIE of a broader range of data sets, we also
used the Bayesmark dataset, which was previously used at NeurIPS 2020 in their
Black-Box Optimization Challenge. The Bayesmark score scales the model
performance based on how well random search does, after normalizing the mean
over multiple repeats. The normalized mean score first calculates the
performance gap between observations and the global optimum and divides it by
the gap between random search and the optimum.

###  5.4 Statistical Analysis

To compare multiple groups of data, we use a combination of the Kruskal-Wallis
test (as used in recent SE literature [22, 40]) and the Mann-Whitney U-test
[36] at a 5% significance level. In particular, we first use the Kruskal-
Wallis H-test to determine if there is a statistically significant difference
between the multiple groups. If there is no significant difference, we report
all treatments as indistinguishable; if there is, we run pairwise Mann-Whitney
U-tests to find treatments that are not statistically significantly different
from the highest-scoring treatment (using a significance level of 0.05). We
apply the Benjamini-Hochberg correction to adjust p-values (as endorsed by
Farcomeni [18]) of the Mann-Whitney U-tests.

TABLE III: RQ1 results: Overview of RQ1 results for 20 repeats and 24 data sets. Two methods tie if, they achieve the same statistical rank (using the methods of §5.4). Otherwise, a win is when one measure has a different rank to another and one measure has better median values (where “better” means “less“ for false alarms and “greater” for other metrics). Datasets | Wins | Ties | Losses
---|---|---|---
Static code warnings | 4 | 3 | 0
Issue lifetime prediction (binary) | 0 | 1 | 2
Issue lifetime prediction (multi-class) | 3 | 0 | 0
Defect prediction | 0 | 6 | 0
Bayesmark | 2 | 1 | 2

##  6 Results

We ran the experiments for defect prediction and issue lifetime prediction on
a university Slurm cluster. For static code analysis, we used a Google Cloud
N2 machine with 4vCPUs and 23GB RAM. For the Bayesmark datasets, we ran the
experiments on a Google Cloud C3 VM.

TABLE IV: RQ1 results: Static code warnings results. The statistically best results are in bold. For Algorithm 1, we use $N_{1}=50,N_{2}=10$. FF = Feedforward network. NB = Naive Bayes. LR = Logistic Regression. Dataset | Method (Learner) | #evals | pd | pf | precision
---|---|---|---|---|---
ant | Baseline (SVM) |  | 0 | 0.25 | 0
SMOOTHIE (FF) | 10 | 0.65 | 0 | 0.8
SMOOTHIE (NB) | 10 | 0.6 | 0 | 0.8
SMOOTHIE (LR) | 10 | 1 | 0 | 1
Random | 10 | 0.2 | 0.5 | 0.5
BOHB (FF) | 10 | 1 | 0.2 | 0.5
BOHB (FF) | 50 | 0.6 | 1 | 0.6
cassandra | Baseline (SVM) |  | 0.33 | 1 | 0.5
SMOOTHIE (FF) | 10 | 0.83 | 0 | 0.75
SMOOTHIE (NB) | 10 | 1 | 0.5 | 0.83
SMOOTHIE (LR) | 10 | 1 | 0.67 | 0.33
Random | 10 | 0 | 0.66 | 0
BOHB (FF) | 10 | 1 | 0.67 | 0.25
BOHB (FF) | 50 | 0 | 0 | 0
commons | Baseline (SVM) |  | 1 | 1 | 0.6
SMOOTHIE (FF) | 10 | 1 | 0.25 | 0.5
SMOOTHIE (NB) | 10 | 0 | 0 | 0
SMOOTHIE (LR) | 10 | 1 | 1 | 0.8
Random | 10 | 0 | 0 | 0
BOHB (FF) | 10 | 0.88 | 1 | 0.65
BOHB (FF) | 50 | 1 | 1 | 0.2
derby | Baseline (SVM) |  | 0.61 | 0.67 | 0.39
SMOOTHIE (FF) | 10 | 0.74 | 0.46 | 0.44
SMOOTHIE (NB) | 10 | 0.90 | 0.60 | 0.4
SMOOTHIE (LR) | 10 | 0.59 | 0.06 | 0.8
Random | 10 | 1.0 | 1.0 | 0.68
BOHB (FF) | 10 | 0.93 | 0.84 | 0.68
BOHB (FF) | 10 | 1 | 1 | 0.68
jmeter | Baseline (SVM) |  | 0 | 0 | 0
SMOOTHIE (FF) | 10 | 1 | 0 | 1
SMOOTHIE (NB) | 10 | 1 | 0.33 | 0.5
SMOOTHIE (LR) | 10 | 0.67 | 0 | 0.63
Random | 10 | 0 | 1 | 0
BOHB (FF) | 10 | 0 | 0 | 0
BOHB (FF) | 50 | 0 | 0 | 0
lucene-solr | Baseline (SVM) |  | 0 | 1 | 0
SMOOTHIE (FF) | 10 | 0.5 | 0.25 | 0.5
SMOOTHIE (NB) | 10 | 1 | 0.5 | 0.5
SMOOTHIE (LR) | 10 | 1 | 0.25 | 0.67
Random | 10 | 1 | 0.25 | 0.67
BOHB (FF) | 10 | 0.75 | 0.5 | 0.33
BOHB (FF) | 50 | 0 | 0 | 0
tomcat | Baseline (SVM) |  | 0.73 | 0.77 | 0.39
SMOOTHIE (FF) | 10 | 0.14 | 0.09 | 0.47
SMOOTHIE (NB) | 10 | 0.43 | 0.13 | 0.63
SMOOTHIE (LR) | 10 | 0.71 | 0.3 | 0.61
Random | 10 | 0.43 | 0.5 | 0.59
BOHB (FF) | 10 | 0.7 | 0.48 | 0.6
BOHB (FF) | 50 | 1 | 1 | 0.62
TABLE V: RQ1 results: Issue lifetime prediction results. For Algorithm 1, we use $N_{1},N_{2}=30,5$. The statistically best results are highlighted in bold. Our baseline is DeepTriage [35]. In the dataset column, the number of classes the data is binned into is in parentheses.  Dataset | Learner | HPO method | #evals | Accuracy
---|---|---|---|---
eclipse (2) | Baseline | 61
Feedforward | SMOOTHIE | 5 | 69.3
Naive Bayes | SMOOTHIE | 5 | 63
Logistic | SMOOTHIE | 5 | 62.5
Best (Feedforward) | Random | 5 | 80.2
Best (Feedforward) | BOHB | 5 | 45.8
Best (Feedforward) | BOHB | 30 | 45.8
firefox (2) | Baseline | 67
Feedforward | SMOOTHIE | 5 | 69
Naive Bayes | SMOOTHIE | 5 | 66.6
Logistic | SMOOTHIE | 5 | 66.4
Best (Feedforward) | Random | 5 | 76.5
Best (Feedforward) | BOHB | 5 | 36
Best (Feedforward) | BOHB | 30 | 35.9
chromium (2) | Baseline | 63
Feedforward | SMOOTHIE | 5 | 62.2
Naive Bayes | SMOOTHIE | 5 | 63
Logistic | SMOOTHIE | 5 | 63.1
Best (Logistic) | Random | 5 | 67.1
Best (Logistic) | BOHB | 5 | 44.5
Best (Logistic) | BOHB | 30 | 44.4
eclipse (3) | Baseline | 44
Feedforward | SMOOTHIE | 5 | 71.2
Naive Bayes | SMOOTHIE | 5 | 51.6
Logistic | SMOOTHIE | 5 | 51.2
Best (Feedforward) | Random | 5 | 36.3
Best (Feedforward) | BOHB | 5 | 29.2
Best (Feedforward) | BOHB | 30 | 46.2
firefox (3) | Baseline | 44
Feedforward | SMOOTHIE | 5 | 69.2
Naive Bayes | SMOOTHIE | 5 | 49.5
Logistic | SMOOTHIE | 5 | 49.3
Best (Feedforward) | Random | 5 | 46.1
Best (Feedforward) | BOHB | 5 | 32.9
Best (Feedforward) | BOHB | 30 | 44.4
chromium (3) | Baseline | 43
Feedforward | SMOOTHIE | 5 | 71.9
Naive Bayes | SMOOTHIE | 5 | 49.1
Logistic | SMOOTHIE | 5 | 49.8
Best (Feedforward) | Random | 5 | 53.5
Best (Feedforward) | BOHB | 5 | 27.5
Best (Feedforward) | BOHB | 30 | 44.6
TABLE VI: RQ1 results: Defect prediction results. For Algorithm 1, we use $N_{1},N_{2}$ =30, 5. The statistically best results are highlighted in bold.  Dataset | Learner | HPO method | #evals | F1
---|---|---|---|---
camel | Baseline (GHOST) | 90.9
Feedforward | SMOOTHIE | 5 | 90.9
Naive Bayes | SMOOTHIE | 5 | 90.9
Logistic | SMOOTHIE | 5 | 90.9
Best (NB) | Random | 5 | 90.9
Best (NB) | BOHB | 5 | 90.9
Best (NB) | BOHB | 30 | 90.9
ivy | Baseline (GHOST) | 83.4
Feedforward | SMOOTHIE | 5 | 94
Naive Bayes | SMOOTHIE | 5 | 94
Logistic | SMOOTHIE | 5 | 94
Best (NB) | Random | 5 | 94
Best (NB) | BOHB | 5 | 94
Best (NB) | BOHB | 30 | 94
log4j | Baseline (GHOST) | 65.9
Feedforward | SMOOTHIE | 5 | 79.6
Naive Bayes | SMOOTHIE | 5 | 79.6
Logistic | SMOOTHIE | 5 | 79.6
Best (NB) | Random | 5 | 79.6
Best (NB) | BOHB | 5 | 79.6
Best (NB) | BOHB | 30 | 79.6
synapse | Baseline (GHOST) | 84.6
Feedforward | SMOOTHIE | 5 | 84.4
Naive Bayes | SMOOTHIE | 5 | 84.4
Logistic | SMOOTHIE | 5 | 84.4
Best (NB) | Random | 5 | 84.4
Best (NB) | BOHB | 5 | 74.5
Best (NB) | BOHB | 30 | 0
velocity | Baseline (GHOST) | 50.3
Feedforward | SMOOTHIE | 5 | 55.4
Naive Bayes | SMOOTHIE | 5 | 79.8
Logistic | SMOOTHIE | 5 | 79.8
Best (NB) | Random | 5 | 79.8
Best (NB) | BOHB | 5 | 79.8
Best (NB) | BOHB | 30 | 79.8
xalan | Baseline (GHOST) | 66
Feedforward | SMOOTHIE | 5 | 68.3
Naive Bayes | SMOOTHIE | 5 | 68.3
Logistic | SMOOTHIE | 5 | 68.3
Best (Logistic) | Random | 5 | 68.2
Best (Logistic) | BOHB | 5 | 0
Best (Logistic) | BOHB | 30 | 68.3
TABLE VII: RQ1 results: Results on the NeurIPS 2020 Black-Box Optimization Challenge public datasets. For Algorithm 1, we use $N_{1},N_{2}=30,5$. The statistically best results are highlighted in bold.  Dataset | HPO method | Score
---|---|---
breast | Baseline (HEBO) | 90.84
SMOOTHIE | 94.32
Random search | 90.45
BOHB | 92.37
digits | Baseline (HEBO) | 95.24
SMOOTHIE | 89.51
Random search | 87.39
BOHB | 91.50
iris | Baseline (HEBO) | 78.27
SMOOTHIE | 87.64
Random search | 82.98
BOHB | 92.30
wine | Baseline (HEBO) | 74.93
SMOOTHIE | 93.34
Random search | 76.59
BOHB | 81.68
diabetes | Baseline (HEBO) | 98.67
SMOOTHIE | 98.65
Random search | 97.16
BOHB | 97.86

We summarize our results in Table III. These summaries are built from Tables
IV-VII. In those tables, the best results for any column are shown in bold
gray. Note that these gray results include the best value, as well as any
other value that ties with the best. To check for such ties, we use the
statistical methods of Section 5.4.

Using these results, we explored two research questions:

  * •

RQ1: What are the benefits of smoothing (measured in metrics like F1, recall,
etc)?

  * •

RQ2: What are the costs of smoothing (measured in CPU time)?

###  6.1 RQ1: What are the benefits of smoothing?

Table IV shows our results on static code analysis false alarms. Recall that
the task here was to recognize static code warnings that, in the historical
log, were ignored by developers. In the figure, gray denotes “best results in
this column“. Note that SMOOTHIE (and more specifically, SMOOTHIE(FF) has much
more gray than anything else. More specifically: all the not SMOOTHIE methods
have far more white cells (i.e. non best) than the SMOOTHIE methods. (for an
exact count of the wins and ties, see Table III).

Table V shows our results on issue lifetime prediction. Recall that the task
here is to classify a GitHub issue according to how long it will take to
resolve. As done in prior work [32], for 3-class classification, the number of
days taken to fix bugs is binned into 0 to 1 days, 2 to 5 days, and 5 to 100
days. For 2-class, the number of days taken to fix bugs is binned into less
than (or greater than) 4 days for Firefox, less than (or greater than) 5 days
for Chromium, and less than (or greater than) 6 days for Eclipse. Note that:

  * •

For simple two class prediction (eclipse(2), firefox(2), chromium(2)) SMOOTHIE
does not perform well (as shown in Table III, 1 tie and 2 losses).

  * •

But for more challenging three class problems (eclipse(3), firefox(3),
chromium(3)) , SMOOTHIE wins in all cases.

Table VI shows our results on defect prediction. Recall that the task is to
import static code features and output a prediction that bugs will be found in
some part of the code. It turns out that defect prediction is a simpler task
that the other as shown here (evidence: all methods achieve the same best
ranked results). To say that another way, defect prediction may not be a
useful domain to compare hyper-parameter optimizers.

Table VII shows our results for the NeurIPS 2020 Black- Box Optimization
Challenge public dataset. We include these results here just to study how well
SMOOTHIE performs on non-SE data. From Table III, in the majority case (3/5),
SMOOTHIE does not lose.

In summary, we say: RQ1 (answer): In the usual case, SMOOTHIE performs as well
or better than the prior state of the art in hyper-parameter optimization.

Figure 5: RQ2 Results: Runtime results.

###  6.2 RQ2: What are the costs of smoothing?

Figure 5 shows the runtimes for our method. We take care to distinguish the
overheads associated with finding smoothness, versus the time required for
another process to use smoothness to make a recommendation about what are good
hyperparameter options.

As shown in Figure 5, the time required to compute Get-Smoothness smoothness
is negligible. Those results are then used by SMOOTHIE for its decision making
(and here, SMOOTHIE repeats its processing $N_{1}+N_{2}=30+5$ times). Even
with those repeats, SMOOTHIE is four times faster than the prior state of the
art (see BOHB in Figure 5).

Figure 5 suggests that further optimizations might be possible if SMOOTHIE
could reduce how many repeats its executes. This is a matter for further work,
in another paper.

In summary, we say: RQ2 (answer): SMOOTHIE works as well as the prior state of
the art, and does so four times faster.

##  7 Discussion

###  7.1 The white-box trade-off

A key distinction between SMOOTHIE and other HPO algorithms is that other
algorithms are black-box, i.e., they need only pairs of configurations and
their corresponding performances. In contrast, SMOOTHIE relies on the learner
structure (specifically, its loss function or a surrogate thereof).

The definition of smoothness is, inherently, limited to loss functions within
the same family. For example, the smoothness value of two feedforward networks
can be compared and used (as done in this paper). However, the smoothness of a
feedforward network cannot, for example, be compared directly to a Naive Bayes
classifier. This means that the practitioner using this system must make a
decision about the learner a priori, which is a limitation.

However, as with any trade-off, this does come with the benefit of knowing the
structure of the learner being used. For example, with any neural network, one
can use the built-in loss function to compute the smoothness directly, which
is not possible without knowing a priori the nature of the loss function. As
such, we can directly optimize for more “well-behaved” loss functions, that
tend to perform better.

###  7.2 Threats to Validity

In this section, we discuss various biases that might affect the final
results, and how our experimental setup attempts to mitigate them.

Sampling bias. Sampling bias is an issue for experimental papers that perform
evaluation on a limited set of data points. We believe this is mitigated by
testing our method on 24 datasets against a state-of-the-art optimizer. Our
code is open-source, and we invite other researchers to extend our results to
more datasets.

Evaluation bias. Evaluation bias threatens papers that use a limited number of
metrics to evaluate performance. It is important to ensure all comparisons are
fair and provide a broad view of the performance. By using the same metrics as
prior work to compare the methods in this paper, we demonstrate that our
method can achieve state-of-the-art results across a variety of tasks and
metrics.

Learner bias. Another source of bias is the learners used in the study.
Currently, there are many algorithms for classification (which this paper
focuses on). We developed our theory for three learners and tested them across
all our datasets in an effort to mitigate this bias. In future work, we will
implement the finite difference approximation (cf. Section 4.5) which open up
the theory to arbitrary algorithms.

Comparison bias. There are many approaches for the tasks we studied in this
paper, and it would be difficult to compare to all of them. As such, we
decided to compare our work to the current state-of-the-art, as well as a
state-of-the-art hyper-parameter optimization algorithm. This ensures that our
results are compared against the best among the algorithms in the literature.

Order bias. Order bias has to do with the order of samples affecting
performance. For defect prediction, we ensured that the test set is from a
newer version than the training set (since that data comes with time stamps).
For all other datasets, we mitigate this bias by randomly splitting the
dataset into train and test sets, and repeating our results 20 times.

Statistical validity. Especially with stochastic algorithms, it is important
to ensure that any performance differences cannot be explained by noise. We
repeat all our experiments 20 times and use adjusted p-values from a widely-
used statistical test to declare wins, ties, and losses.

##  8 Limitations and Future Work

The major limitation of this work is the choice of measurement of smoothness,
and the use of smoothness as a heuristic. For example, having derived the
Hessian, the condition number is an alternate measure of the smoothness of the
landscape [11]. Another clear limitation is that this approach is limited to
learners for which a smoothness can be derived. Especially in the case where
the Hessian may not exist (e.g., if the loss function is not twice-
differentiable such as the mean absolute error loss), this approach fails.
Nevertheless, we have shown successful application in a classical learner that
is not gradient-based, Naive Bayes. In that case, we used the log-likelihood
as a surrogate for the loss function. We postulate that this should be
possible for other learners too, such as the dual formulation in SVMs or the
impurity measure in decision trees. We leave this exploration to future work.

Another avenue of future work would be replacing the random sampling at the
beginning of the approach with a smarter selection approach. For example,
Bayesian optimization could be used in a loop to search for configurations
with high smoothness values. Alternatively, one could use multiple metrics for
smoothness, such as the condition number of the Hessian, and use a multi-
objective optimization algorithm.

Finally, this paper only explored three learners. In future work, we plan to
extend this to more tasks and learners such as convolutional networks.

##  9 Conclusion

This paper explores a theory of hyper-parameter optimization:

> Hyper-parameter optimization is best viewed as a “smoothing” function for
> the decision landscape since, after optimization, the loss function gets
> “smoother’ (i.e. on average, fewer changes in any local region).

This theory predicts that methods that maximize the smoothness should also
improve the hyper-parameter optimization. This prediction was checked on on 24
datasets across a variety of tasks. As predicted, SMOOTHIE, achieves state-of-
the-art performance and outperforms other popular HPO methods. One reason to
endorse this theory is the time required to compute smoothness is negligible.

For practitioners who wish to use this method in their pipelines, and for
researchers who want to extend or reproduce our results, we released an open-
source implementation that covers all our experiments (see
https://github.com/yrahul3910/smoothness-hpo/). In the future, we will release
a Python package that will make it easier to use, without worrying about the
mathematical details of the approach, and also implements the center
difference approximation discussed in the paper.

## Acknowledgements

This work was partially funded by an NSF grant #1908762.

## Appendix: Proofs

######  Theorem 2 (Smoothness for Naive Bayes).

For Naive Bayes, the smoothness is given by

$\beta=\sup\lVert\boldsymbol{\Sigma}^{-1}\mathcal{A}G-\frac{1}{2}\boldsymbol{\Sigma}^{-1}\mathcal{A}\boldsymbol{\Sigma}^{-1}+G\mathcal{A}\boldsymbol{\Sigma}^{-1}\rVert$

where $\boldsymbol{\Sigma}$ is the covariance matrix, $\mathcal{A}$ is the
fourth-order identity tensor, and
$G=\dfrac{\partial\ell}{\partial\boldsymbol{\Sigma}}$, where $\ell$ is the
log-likelihood.

###### Proof.

We use the results of Greg [23] here. For Gaussian Discriminant Analysis
(which Naive Bayes is a special case of), the log-likelihood is given by:

$\begin{array}[]{rcl}\ell(\phi,\boldsymbol{\mu}_{0},\boldsymbol{\mu}_{1},\boldsymbol{\Sigma})&=\sum_{i=1}^{m}\bigl{\\{}y^{(i)}\log\phi+(1-y^{(i)})\log(1-\phi)\\\
{}\hfil\\\
&-\frac{1}{2}\log|\boldsymbol{\Sigma}|+C+(\textbf{x}^{(i)}-\boldsymbol{\mu})^{T}\boldsymbol{\Sigma}^{-1}(\textbf{x}^{(i)}-\boldsymbol{\mu})\bigr{\\}}\end{array}$
(10)

where $\phi$ is the parameter of the Bernoulli distribution; $\mu_{0}$ and
$\mu_{1}$ are the mean vectors for the two classes and $\Sigma$ is the
covariance matrix of the independent attributes. Because there are several
parameters, we show that the smoothness is constant with respect to all except
the covariance matrix, and therefore we use the smoothness with respect to the
covariance matrix in our experiments.

Taking a gradient of (10) with respect to $\phi$ gives us:

$\displaystyle\nabla_{\phi}\ell$ $\displaystyle=\frac{\sum
y^{(i)}}{\phi}-\frac{\sum(1-y^{(i)})}{1-\phi}$
$\displaystyle\nabla^{2}_{\phi}\ell$ $\displaystyle=-\frac{\sum
y^{(i)}}{\phi^{2}}-\frac{\sum(1-y^{(i)})}{(1-\phi)^{2}}$

The second derivative has an extremum when $\nabla_{\phi}^{3}\ell=0$:

$\displaystyle\nabla^{3}_{\phi}\ell$ $\displaystyle=\frac{2\sum
y^{(i)}}{\phi^{3}}-\frac{2\sum(1-y^{(i)})}{(1-\phi)^{3}}=0$
$\displaystyle\frac{\sum y^{(i)}}{\phi^{3}}$
$\displaystyle=\frac{\sum(1-y^{(i)})}{(1-\phi)^{3}}$ $\displaystyle m\phi^{3}$
$\displaystyle=\left(\sum y^{(i)}\right)(1-3\phi+3\phi^{2})$

But $\sum y^{(i)}=km$ for some $0\leq k\leq 1$. Hence,

$\phi^{3}=k(1-3\phi+3\phi^{2})$

which has constant solutions. For example, if $k=\frac{1}{2}$ then
$\phi=\frac{1}{2}$ and $\nabla_{\phi}^{2}E\leq-4m$. Similarly, if $k=1$ then
$\phi=1$ and $\nabla_{\phi}^{2}E\leq-m$

From (10), computing gradient with respect to $\mu$ yields

$\displaystyle\nabla_{\mu_{0}}\ell$
$\displaystyle=\nabla_{\mu_{0}}\sum_{i=1}^{m}~(\textbf{x}^{(i)}-\boldsymbol{\mu})^{T}\boldsymbol{\Sigma}^{-1}(\textbf{x}^{(i)}-\boldsymbol{\mu})$
$\displaystyle=\nabla_{\mu_{0}}\sum_{i=1}^{m}\text{tr
}~(\textbf{x}^{(i)}-\boldsymbol{\mu})^{T}\boldsymbol{\Sigma}^{-1}(\textbf{x}^{(i)}-\boldsymbol{\mu})I$
$\displaystyle=-\sum_{i=1}^{m}\left(I(\textbf{x}^{(i)}-\boldsymbol{\mu}_{0})\boldsymbol{\Sigma}^{-1}+I(\textbf{x}^{(i)}-\boldsymbol{\mu}_{0})(\boldsymbol{\Sigma}^{-1})^{T}\right)[y^{(i)}=0]$
$\displaystyle=-\sum_{i=1}^{m}\left((\textbf{x}^{(i)}-\boldsymbol{\mu}_{0})\boldsymbol{\Sigma}^{-1}+~(\textbf{x}^{(i)}-\boldsymbol{\mu}_{0})\boldsymbol{\Sigma}^{-1}\right)[y^{(i)}=0]$
$\displaystyle=-2\boldsymbol{\Sigma}^{-1}\sum_{i=1}^{m}(\textbf{x}^{(i)}-\boldsymbol{\mu}_{0})[y^{(i)}=0]$

where we used the identity $\nabla_{A}\text{tr }ABA^{T}C=CAB+C^{T}AB^{T}$
along with the trace trick. Clearly, the second derivative is constant with
respect to $\boldsymbol{\mu}_{0}$.

Finally, we compute the smoothness with respect to $\boldsymbol{\Sigma}$. For
the following, let $\langle x,y\rangle$ denote the Frobenius inner product
defined as

$\langle
A,B\rangle=\sum\limits_{i=1}^{m}\sum\limits_{j=1}^{n}A_{ij}B_{ij}=\text{tr
}A^{T}B$

Let $\boldsymbol{w}=(\boldsymbol{x}-\boldsymbol{\mu}_{i})$ (where $\mu_{i}$ is
the mean for class $i$). Then,

$\displaystyle d\ell$
$\displaystyle=\langle\boldsymbol{w}\boldsymbol{w}^{T},(d\boldsymbol{\Sigma})^{-1}\rangle-\frac{1}{2}\langle\boldsymbol{\Sigma}^{-1},d\boldsymbol{\Sigma}\rangle$
$\displaystyle=\langle\boldsymbol{w}\boldsymbol{w}^{T},-\boldsymbol{\Sigma}^{-1}d\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{-1}\rangle-\frac{1}{2}\langle\boldsymbol{\Sigma}^{-1},d\boldsymbol{\Sigma}\rangle$
$\displaystyle=-\left\langle\boldsymbol{\Sigma}^{-1}\boldsymbol{w}\boldsymbol{w}^{T}\boldsymbol{\Sigma}^{-1}+\frac{1}{2}\Sigma^{-1},d\Sigma\right\rangle$
$\displaystyle\dfrac{\partial\ell}{\partial\boldsymbol{\Sigma}}$
$\displaystyle=-\boldsymbol{\Sigma}^{-1}\left(\boldsymbol{w}\boldsymbol{w}^{T}+\frac{1}{2}\boldsymbol{\Sigma}\right)\boldsymbol{\Sigma}^{-1}=G$

Next, we compute the differential of $G$, which we use to compute the Hessian.

$\displaystyle dG$
$\displaystyle=-d\boldsymbol{\Sigma}^{-1}\left(\boldsymbol{w}\boldsymbol{w}^{T}+\frac{1}{2}\boldsymbol{\Sigma}\right)\boldsymbol{\Sigma}^{-1}-\boldsymbol{\Sigma}^{-1}\left(\frac{1}{2}d\boldsymbol{\Sigma}\right)\boldsymbol{\Sigma}^{-1}$
$\displaystyle-\boldsymbol{\Sigma}^{-1}\left(\boldsymbol{w}\boldsymbol{w}^{T}+\frac{1}{2}\boldsymbol{\Sigma}\right)d\boldsymbol{\Sigma}^{-1}$
$\displaystyle=\boldsymbol{\Sigma}^{-1}d\boldsymbol{\Sigma}G-\frac{1}{2}\boldsymbol{\Sigma}^{-1}d\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{-1}+Gd\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{-1}$
$\displaystyle=\langle\boldsymbol{\Sigma}^{-1}\mathcal{A}G-\frac{1}{2}\boldsymbol{\Sigma}^{-1}\mathcal{A}\boldsymbol{\Sigma}^{-1}+G\mathcal{A}\boldsymbol{\Sigma}^{-1},d\boldsymbol{\Sigma}\rangle$
$\displaystyle\dfrac{\partial G}{\partial\boldsymbol{\Sigma}}$
$\displaystyle=\boldsymbol{\Sigma}^{-1}\mathcal{A}G-\frac{1}{2}\boldsymbol{\Sigma}^{-1}\mathcal{A}\boldsymbol{\Sigma}^{-1}+G\mathcal{A}\boldsymbol{\Sigma}^{-1}=\mathcal{H}$

where $\mathcal{H}$ is the Hessian and $\mathcal{A}$ is the fourth-order
identity tensor so that $\mathcal{A}_{ijkl}=\delta_{ik}\delta_{jl}$. In the
above, the fact that $\Sigma$ and $G$ are symmetric was used in various steps.

Although the loss to minimize would be defined as the negative log-likelihood,
the negative sign would be removed by the use of a norm. ∎

######  Theorem 3 ($L_{2}$ regularization).

For a learner using an $L_{2}$ regularization term
$\frac{\lambda}{2}\lVert\boldsymbol{w}\rVert^{2}_{2}$, the increase in
smoothness is $\lambda$. More generally, if a Tikhonov regularization term
$\lVert\boldsymbol{\Gamma}\boldsymbol{w}\rVert^{2}_{2}$ is used, the increase
in smoothness is $2\lVert\boldsymbol{\Gamma}\rVert^{2}_{F}$.

###### Proof.

For $L_{2}$ regularization, the regularization term is
$E=\frac{\lambda}{2}\lVert\boldsymbol{w}\rVert_{2}^{2}$. We therefore have
$\nabla_{\boldsymbol{w}}E=\lambda\lVert\boldsymbol{w}\rVert_{2}$ and so
$\lVert\nabla_{\boldsymbol{w}_{1}}E(\boldsymbol{w}_{1})-\nabla_{\boldsymbol{w}_{2}}E(\boldsymbol{w}_{2})\rVert\leq\lambda\lVert\boldsymbol{w}_{1}-\boldsymbol{w}_{2}\rVert$.
For the general Tikhonov case,

$\displaystyle E(\boldsymbol{w})$
$\displaystyle=\lVert\boldsymbol{\Gamma}\boldsymbol{w}\rVert^{2}_{2}$
$\displaystyle=\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}\boldsymbol{w}$
$\displaystyle dE$
$\displaystyle=d\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}\boldsymbol{w}+\boldsymbol{w}^{T}d\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}\boldsymbol{w}+\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}d\boldsymbol{\Gamma}+\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}d\boldsymbol{w}$
$\displaystyle=(d\boldsymbol{w})^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}\boldsymbol{w}+\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}(d\boldsymbol{w})$
$\displaystyle=\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}(d\boldsymbol{w})+\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}(d\boldsymbol{w})$
$\displaystyle\nabla_{\boldsymbol{w}}E$
$\displaystyle=2\boldsymbol{w}^{T}\boldsymbol{\Gamma}^{T}\boldsymbol{\Gamma}=2\boldsymbol{w}^{T}\lVert\boldsymbol{\Gamma}\rVert^{2}_{F}$

so that

$\frac{\lVert\nabla E(\boldsymbol{w}_{1})-\nabla
E(\boldsymbol{w}_{2})\rVert}{\lVert\boldsymbol{w}_{1}-\boldsymbol{w}_{2}\rVert}=2\lVert\boldsymbol{\Gamma}\rVert^{2}_{F}$

∎

## References

  * Agrawal et al. [2018] Amritanshu Agrawal, Wei Fu, and Tim Menzies.  What is wrong with topic modeling? and how to fix it using search-based software engineering.  _Information and Software Technology_ , 98:74–88, 2018.  ISSN 0950-5849.  doi: https://doi.org/10.1016/j.infsof.2018.02.005.  URL https://www.sciencedirect.com/science/article/pii/S0950584917300861.
  * Agrawal et al. [2019] Amritanshu Agrawal, Wei Fu, Di Chen, Xipeng Shen, and Tim Menzies.  How to “dodge” complex software analytics.  _IEEE Transactions on Software Engineering_ , 47(10):2182–2194, 2019.
  * Agrawal et al. [2020] Amritanshu Agrawal, Tim Menzies, Leandro L Minku, Markus Wagner, and Zhe Yu.  Better software analytics via “duo”: Data mining algorithms using/used-by optimizers.  _Empirical Software Engineering_ , 25:2099–2136, 2020.
  * Aha et al. [1991] W. Aha, Dennis Kibler, and Marc Albert.  Instance-based learning algorithms.  _Machine Learning_ , 6:37–66, 01 1991.  doi: 10.1023/A:1022689900470.
  * Ansel et al. [2014] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-Kelley, Jeffrey Bosboom, Una-May O’Reilly, and Saman Amarasinghe.  Opentuner: An extensible framework for program autotuning.  In _International Conference on Parallel Architectures and Compilation Techniques (PACT)_ , Edmonton, Canada, Aug 2014.  URL http://groups.csail.mit.edu/commit/papers/2014/ansel-pact14-opentuner.pdf.
  * Bentley [1975] Jon Louis Bentley.  Multidimensional binary search trees used for associative searching.  _Commun. ACM_ , 18(9):509–517, sep 1975.  ISSN 0001-0782.  doi: 10.1145/361002.361007.  URL https://doi.org/10.1145/361002.361007.
  * Bergstra and Bengio [2012] James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  _Journal of machine learning research_ , 13(2), 2012.
  * Bergstra et al. [2011] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.  Algorithms for hyper-parameter optimization.  _Advances in neural information processing systems_ , 24, 2011.
  * Bischl et al. [2023] Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, et al.  Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 13(2):e1484, 2023.
  * Boetticher [2007] G Boetticher.  The promise repository of empirical software engineering data.  _http://promisedata. org/repository_ , 2007.
  * Boyd et al. [2004] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe.  _Convex optimization_.  Cambridge university press, 2004.
  * Chawla et al. [2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ , 16:321–357, 2002.
  * Chidamber and Kemerer [1994] Shyam R Chidamber and Chris F Kemerer.  A metrics suite for object oriented design.  _IEEE Transactions on software engineering_ , 20(6):476–493, 1994.
  * Cowen-Rivers et al. [2022] Alexander I Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al.  Hebo: pushing the limits of sample-efficient hyper-parameter optimisation.  _Journal of Artificial Intelligence Research_ , 74:1269–1349, 2022.
  * Eggensperger et al. [2021] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter.  Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo.  _arXiv preprint arXiv:2109.06716_ , 2021.
  * Eriksson et al. [2019] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek.  Scalable global optimization via local bayesian optimization.  _Advances in neural information processing systems_ , 32, 2019.
  * Falkner et al. [2018] Stefan Falkner, Aaron Klein, and Frank Hutter.  Bohb: Robust and efficient hyperparameter optimization at scale.  In _International Conference on Machine Learning_ , pages 1437–1446. PMLR, 2018.
  * Farcomeni [2008] Alessio Farcomeni.  A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion.  _Statistical methods in medical research_ , 17(4):347–388, 2008.
  * Feurer and Hutter [2019] Matthias Feurer and Frank Hutter.  Hyperparameter optimization.  In _Automated machine learning_ , pages 3–33. Springer, Cham, 2019.
  * Fu et al. [2016a] Wei Fu, Tim Menzies, and Xipeng Shen.  Tuning for software analytics: Is it really necessary?  _Information and Software Technology_ , 76:135–146, 2016a.  ISSN 0950-5849.  doi: https://doi.org/10.1016/j.infsof.2016.04.017.  URL https://www.sciencedirect.com/science/article/pii/S0950584916300738.
  * Fu et al. [2016b] Wei Fu, Tim Menzies, and Xipeng Shen.  Tuning for software analytics: Is it really necessary?  _Information and Software Technology_ , 76:135–146, 2016b.
  * Gopal and Cooper [2022] Bhuvaneswari Gopal and Stephen Cooper.  Peer instruction in online software testing and continuous integration: A replication study.  In _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training_ , pages 199–204, 2022.
  * [23] Greg.  Computing the smoothness of gda with respect to covariance matrix.  Mathematics Stack Exchange.  URL https://math.stackexchange.com/q/4647791.  URL:https://math.stackexchange.com/q/4647791 (version: 2023-02-27).
  * Heckman and Williams [2008] Sarah Heckman and Laurie Williams.  On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques.  In _Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement_ , pages 41–50, 2008.
  * Heckman and Williams [2009] Sarah Heckman and Laurie Williams.  A model building process for identifying actionable static analysis alerts.  In _2009 International conference on software testing verification and validation_ , pages 161–170. IEEE, 2009.
  * Hernández-Lobato et al. [2014] José Miguel Hernández-Lobato, Matthew W Hoffman, and Zoubin Ghahramani.  Predictive entropy search for efficient global optimization of black-box functions.  _Advances in neural information processing systems_ , 27, 2014.
  * Jones et al. [1998] Donald R Jones, Matthias Schonlau, and William J Welch.  Efficient global optimization of expensive black-box functions.  _Journal of Global optimization_ , 13(4):455, 1998.
  * Kang et al. [2022] Hong Jin Kang, Khai Loong Aw, and David Lo.  Detecting false alarms from automatic static analysis tools: How far are we?  _arXiv preprint arXiv:2202.05982_ , 2022.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Krishna et al. [2016] Rahul Krishna, Zhe Yu, Amritanshu Agrawal, Manuel Dominguez, and David Wolf.  The "bigse" project: Lessons learned from validating industrial text mining.  In _Proceedings of the 2nd International Workshop on BIG Data Software Engineering_ , BIGDSE ’16, page 65–71, New York, NY, USA, 2016. Association for Computing Machinery.  ISBN 9781450341523.  doi: 10.1145/2896825.2896836.  URL https://doi.org/10.1145/2896825.2896836.
  * Lai et al. [1985] Tze Leung Lai, Herbert Robbins, et al.  Asymptotically efficient adaptive allocation rules.  _Advances in applied mathematics_ , 6(1):4–22, 1985.
  * Lee et al. [2020] Youngseok Lee, Suin Lee, Chan-Gun Lee, Ikjun Yeom, and Honguk Woo.  Continual prediction of bug-fix time using deep learning-based activity stream embedding.  _IEEE Access_ , 8:10503–10515, 2020.
  * Li et al. [2017] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar.  Hyperband: A novel bandit-based approach to hyperparameter optimization.  _The Journal of Machine Learning Research_ , 18(1):6765–6816, 2017.
  * Lindauer et al. [2022] Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter.  Smac3: A versatile bayesian optimization package for hyperparameter optimization.  _The Journal of Machine Learning Research_ , 23(1):2475–2483, 2022.
  * Mani et al. [2019] Senthil Mani, Anush Sankaran, and Rahul Aralikatte.  Deeptriage: Exploring the effectiveness of deep learning for bug triaging.  In _Proceedings of the ACM India joint international conference on data science and management of data_ , pages 171–179, 2019.
  * Mann and Whitney [1947] Henry B Mann and Donald R Whitney.  On a test of whether one of two random variables is stochastically larger than the other.  _The annals of mathematical statistics_ , pages 50–60, 1947.
  * McCabe [1976] Thomas J McCabe.  A complexity measure.  _IEEE Transactions on software Engineering_ , (4):308–320, 1976.
  * Močkus [1975] Jonas Močkus.  On bayesian methods for seeking the extremum.  In _Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1–7, 1974_ , pages 400–404. Springer, 1975.
  * Nair et al. [2017] Vivek Nair, Tim Menzies, Norbert Siegmund, and Sven Apel.  Using bad learners to find good configurations.  In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ , ESEC/FSE 2017, page 257–267, New York, NY, USA, 2017. Association for Computing Machinery.  ISBN 9781450351058.  doi: 10.1145/3106237.3106238.  URL https://doi.org/10.1145/3106237.3106238.
  * Napier et al. [2023] Kollin Napier, Tanmay Bhowmik, and Shaowei Wang.  An empirical study of text-based machine learning models for vulnerability detection.  _Empirical Software Engineering_ , 28(2):38, 2023.
  * Pfisterer et al. [2022] Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl.  Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization.  In _International Conference on Automated Machine Learning_ , pages 3–1. PMLR, 2022.
  * Shahriari et al. [2015] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas.  Taking the human out of the loop: A review of bayesian optimization.  _Proceedings of the IEEE_ , 104(1):148–175, 2015.
  * Tantithamthavorn et al. [2016] Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto.  Automated parameter optimization of classification techniques for defect prediction models.  In _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_ , pages 321–332, 2016.  doi: 10.1145/2884781.2884857.
  * Wan et al. [2018] Zhiyuan Wan, Xin Xia, Ahmed E Hassan, David Lo, Jianwei Yin, and Xiaohu Yang.  Perceptions, expectations, and challenges in defect prediction.  _IEEE Transactions on Software Engineering_ , 46(11):1241–1266, 2018.
  * Wang et al. [2018] Junjie Wang, Song Wang, and Qing Wang.  Is there a" golden" feature set for static warning identification? an experimental evaluation.  In _Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and measurement_ , pages 1–10, 2018.
  * Wang et al. [2016] Song Wang, Taiyue Liu, and Lin Tan.  Automatically learning semantic features for defect prediction.  In _Proceedings of the 38th International Conference on Software Engineering_ , pages 297–308, 2016.
  * Williams and Rasmussen [2006] Christopher KI Williams and Carl Edward Rasmussen.  _Gaussian processes for machine learning_ , volume 2.  MIT press Cambridge, MA, 2006.
  * Wohlin [2014] Claes Wohlin.  Guidelines for snowballing in systematic literature studies and a replication in software engineering.  In _Proceedings of the 18th international conference on evaluation and assessment in software engineering_ , pages 1–10, 2014.
  * Xu et al. [2015] Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker.  Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software.  In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ , ESEC/FSE 2015, page 307–319, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450336758.  doi: 10.1145/2786805.2786852.  URL https://doi.org/10.1145/2786805.2786852.
  * Yang et al. [2021] Xueqi Yang, Jianfeng Chen, Rahul Yedida, Zhe Yu, and Tim Menzies.  Learning to recognize actionable static code warnings (is intrinsically easy).  _Empirical Software Engineering_ , 26:1–24, 2021.
  * Yedida and Menzies [2021] Rahul Yedida and Tim Menzies.  On the value of oversampling for deep learning in software defect prediction.  _IEEE Transactions on Software Engineering_ , 48(8):3103–3116, 2021.
  * Yedida et al. [2021] Rahul Yedida, Xueqi Yang, and Tim Menzies.  Old but gold: Reconsidering the value of feedforward learners for software analytics.  _arXiv preprint arXiv:2101.06319_ , 2021.
  * Yedida et al. [2023] Rahul Yedida, Hong Jin Kang, Huy Tu, Xueqi Yang, David Lo, and Tim Menzies.  How to find actionable static analysis warnings: A case study with findbugs.  _IEEE Transactions on Software Engineering_ , 2023.
  * Yu and Zhu [2020] Tong Yu and Hong Zhu.  Hyper-parameter optimization: A review of algorithms and applications.  _arXiv preprint arXiv:2003.05689_ , 2020.

|  Rahul Yedida is a PhD student in Computer Science at NC State University.
His research interests include automated software engineering and machine
learning. For more information, please see https://ryedida.me.
---|---
|  Tim Menzies (IEEE Fellow, Ph.D. UNSW, 1995) is a Professor in computer
science at NC State University, USA. His research interests include software
engineering (SE), data mining, artificial intelligence, and search-based SE,
open access science. For more information, please see http://menzies.us.
---|---