# Determining the significance and relative importance of parameters of a
simulated quenching algorithm using statistical tools

P.A. Castillo<EMAIL_ADDRESS>M.G. Arenas  N. Rico  A.M. Mora
P. García-Sánchez  J.L.J. Laredo  J.J. Merelo

###### Abstract

When search methods are being designed it is very important to know which
parameters have the greatest influence on the behaviour and performance of the
algorithm. To this end, algorithm parameters are commonly calibrated by means
of either theoretic analysis or intensive experimentation. When undertaking a
detailed statistical analysis of the influence of each parameter, the designer
should pay attention mostly to the parameters that are statistically
significant. In this paper the ANOVA (ANalysis Of the VAriance) method is used
to carry out an exhaustive analysis of a simulated annealing based method and
the different parameters it requires. Following this idea, the significance
and relative importance of the parameters regarding the obtained results, as
well as suitable values for each of these, were obtained using ANOVA and post-
hoc Tukey HSD test, on four well known function optimization problems and the
likelihood function that is used to estimate the parameters involved in the
lognormal diffusion process. Through this statistical study we have verified
the adequacy of parameter values available in the bibliography using
parametric hypothesis tests.

## NOTE:

This paper is a pre-print version of the paper below:

Castillo, P.A., Arenas, M.G., Rico, N., Mora. A.M., Garcia P., Laredo, J.L.J.,
Merelo, J.J. Determining the significance and relative importance of
parameters of a simulated quenching algorithm using statistical tools. Appl
Intell 37, 239–254 (2012). https://doi.org/10.1007/s10489-011-0324-x

##  1 Introduction

When using search heuristics such as evolutionary algorithms (EAs) [1, 2, 3],
simulated annealing (SA) [4, 5, 6, 7] or local search algorithms [8, 9, 10],
application rates for genetic operators, selection and replacement mechanisms,
and the initial population, must first be chosen. The parameters used to apply
these elements determine the way they operate and influence the results.
Therefore, finding appropriate parameter values is a challenge in the field of
metaheuristics, and can be addressed before the run (parameter tuning) or
during the run (parameter control) [11, 12, 13].

Due to the importance and effect of parameters on the results, a detailed
statistical analysis of the influence of each parameter should be performed,
paying attention mostly to those providing the values that are statistically
significant. In this paper, we propose using the ANOVA (ANalysis Of the
VAriance) [14] statistical method as a tool to analyze a well known
metaheuristic to solve function approximation problems.

This method allows us to determine whether a change in the results (responses)
is due to a change in a parameter (variable or factor) or to a random effect,
which makes it possible to determine the variables that have statistically
significative effect on the method that is being evaluated.

The theory and methodology of ANOVA was mainly developed by R.A Fisher during
the 1920s [14]. ANOVA examines the effects of one, two or more quantitative or
qualitative variables, called factors, on one quantitative response. ANOVA is
useful in a range of disciplines when it is suspected that one or more factors
might affect a response. It is essentially a method used to analyze the
variance to which a response is subjected, dividing it into the various
components corresponding to the sources of variation, which can be identified.

With ANOVA, we test a null hypothesis that all the population means are equal
against the alternative hypothesis that there is at least two means that are
not equal. We find the sample mean and variance for each level (value) of the
main factor. The first one is obtained by finding the sample variance of the
$n_{k}$ sample means from the overall mean. This variance is referred to as
the _variance among the means_. The second estimate of the population variance
is found by using a weighted average of the sample variances. In order to
solve this test, the significance level have been fixed in $\alpha=0.05$.

After applying ANOVA, if the test shows significative differences, post-hoc
tests must be applied to find which levels are causing these differences.
Tukey’s Honestly Significant Difference (HSD) [15] has been used for this
purpose in this paper.

What we propose in this paper is a methodology based on applying the
statistical parametric test ANOVA to determine the most important parameters
of a simulated quenching (SQ) algorithm [16] (a SA-based optimization method)
in terms of their influence on the results, and to establish the most suitable
set of values for such parameters (thus obtaining an optimal operation).

By using such a methodology, we might learn which parameters have almost no
effect on the performance or which others have a strong correlation with
fitness. We think that our paper can help to find heuristic ways of obtaining
the best result, or at least one that does not fall in a local minimum. The
main difference from previous approaches is that statistical tools to
determine the significance and relative importance of parameters of a search
method are used.

The rest of this paper is structured as follows: Section 2 presents a
comprehensive review of the approaches found in the bibliography to describe
different application parameters and determine the most suitable values for
these parameters. Section 3 briefly describes the SQ algorithm, and analyzes
the parameters we propose to evaluate. This section also presents the
experimental setup and the methodology considered in the study. Section 4
details the statistical study based on ANOVA. Obtained results are analyzed in
order to establish the most suitable values. Finally, conclusions and future
work are presented in Section 5.

##  2 Related work

Adjusting the main design parameters has been usually solved either by hand
[17, 18, 19] or using conventions, ad-hoc choices, intensive experimentation
with different values [20, 21, 22, 23, 24, 13], and even random initialization
values [25], that is why new practices, based on well-grounded tuning methods
(i.e. robust mathematical methods), are needed. Such a methodology is what we
propose in this paper. As Eiben states, the straightforward approach is to
generate and test [26, 27, 23, 24]. An alternative is to use a meta-algorithm
to optimize the parameters [26, 28, 29, 30, 31], that is, to run a higher
level algorithm that searches for an optimal and general set of parameters to
solve a wide range of optimization problems. However, as some authors remark,
solving specific problems requires specific parameter value sets [32, 33, 11]
and, as Harik [34] claims, nobody knows the “optimal” parameter settings for
an arbitrary real-world problem. Therefore, establishing the optimal set of
parameters for a sufficiently general case is a difficult problem.

Other researchers have proposed determining a good set of evolutionary
algorithm parameters by analogy [35, 36, 37, 38, 34, 39, 40]. In these works,
a set of parameters is found, but instead of finding them by means of intense
experimentation, the parameter settings are backed up with theoretical work
(meaning that these settings are robust). Establishing parameters by analogy
means using suitable sets of parameters to solve similar problems; however
they do not explain how to establish the similarity between problems.
Moreover, a clear protocol has not been proposed for situations when the
similarity between problems implies that the most suitable sets of parameters
are also similar [33, 11]. Weyland has described a theoretical analysis of
both an evolutionary [41, 42] and simulated annealing algorithm [43] to search
for the optimal parameter setting to solve the longest common subsequence
problem. However, Weyland does not carry out this approach to practice.

Higher level algorithms [44] have been proposed to face the problem of
establishing parameter values: several methods of testing and comparing
different values before the run (parameter tuning), such as REVAC [45, 46,
47], SPOT [48, 49, 50], F-Race [51] and ParamILS [52] are available. In this
way, Smit et al. [13] propose improving the CEC2005 EA winner using REVAC as
parameter tuning method.

Practical approaches based on setting parameter values on-line (during the
run) instead of a previous parameter tuning have been proposed: Montero et al.
[53] propose using off-line calibration techniques to detect whether a genetic
operator help the evolutionary algorithm to perform his work, while Leung et
al. [54] propose a novel adaptive parameter control system to control the
parameters of a history-driven EA in an automatic manner. Their method obtains
similar results to other methods while sets two EA parameters automatically.
Other proposed methods in this line are based on coding the parameters into
the individual’s genome (self-adaptation of parameters) or using non-static
parameter settings techniques (controlled by feedback from the search and
optimization process) [21, 24, 55]. However, control strategies also have
parameters and there are indicators that good tuning works better than control
[23].

Recently, there has been a growing interest for the use of different
statistical techniques in algorithms’ comparison [56, 57]. In this sense,
Shilane et al. [58] proposes a framework for statistical performance
comparison of evolutionary computation algorithms. García et al. [59] presents
an analysis of the behaviour of evolutionary algorithms in optimization
problems using parametric and non-parametric statistical tests. Finally, Rojas
et al. [60] study the effects of parameters involved in GA design by applying
ANOVA, while Castillo et al. [61] use statistical tools to find accurate
parameter values involved in the design of a neuro-genetic algorithm.
Methodology presented in these papers, could be improved by means of post-hoc
statistical tests.

We propose a complete methodology based on parametric statistical tools to
analyze the importance of parameters. Our proposal could be helpful to
practitioners in designing and analyzing their optimization methods.

##  3 Experimental Setup

Proposed methodology has been tested on a generalized version of the SA
algorithm generally called simulated quenching [16]. Next, a brief description
of the proposed algorithm is presented, focusing on detailing the parameters,
the methodology and the problems used.

###  3.1 Proposed Metaheuristic and Parameters Analysis

SA is a generic probabilistic metaheuristic method for solving global
optimization problems. Subject to conditions on the cooling schedule,
simulated annealing can be shown to converge asymptotically to the global
optima of the fitness function [16, 5, 62, 63]. In practice, simulated
annealing has been used effectively in several science and engineering
problems [64, 65, 66]. However, its sensitivity to the proposal distribution
and the cooling schedule means that it is not a good fit for all optimization
problems [62, 63].

In a SA algorithm, a cost function (objective function $f$) to be minimized is
defined. Subsequently, from an initial random solution, different solutions
are derived and compared to the current solution. The fittest solution (least
cost) is kept, but retaining a solution with a greater cost is allowed with a
certain probability, that decreases according to a ”temperature” value.
Specifically, the algorithm generates a new state from the old one by means of
a change operator. If the new one is better, then it is accepted. However, if
the new one is worse, the algorithm will accept it with an acceptation
probability of $p_{a}=e^{-\Delta f/T}$ , where $\Delta f$ is defined as the
increment of the cost function and $T$ is the temperature control parameter.
The temperature is decreased using the temperature reduction function $f_{T}$
when a predefined number of new states from the current one have been
generated. The simplest decrement rule is $T_{n+1}=\alpha T_{n}$ , where
$\alpha$ is a constant smaller than 1 (in the interval $(0,1)$). This
exponential cooling scheme was proposed by Kirkpatrick et al. in [4] with
$\alpha\in[0.8,0.95]$.

SA strength relies in its ability to statistically deliver the system to a
global optimum, although in some cases the convergence could only be obtained
after an infinite number of iterations (as stated in [62, 63]). However, due
to its sensitivity to the proposal distribution and the cooling schedule,
other algorithms (i.e. SA-like methods such as SQ) are usually used to find
the global optimum faster. Although it is a faster method, it has the drawback
of not having a convergence proof associated [16].

Thus, in this paper a SQ algorithm, proposed by Michalewicz in [67, 2], is
used. Additionally, our implementation uses several states at the same time (a
population of individuals) instead of an isolated state, as proposed by Wang
et al. in [68].

Several parameters have been identified to configure the proposed optimization
method. Our aim is to determine which parameters influence the obtained
fitness and to facilitate practitioners some rules to decide which parameters
to tune as they influence results in a different manner:

  * •

Number of Changes (_NC_): this parameter represents the number of times that
the cooling schedule is applied. This parameter controls the algorithm
stopping criterion even if the temperature is not equal to 0. It sets how many
times the cooling schedule is applied during the algorithm execution.

  * •

Population Size (_PS_): represents the number of states the algorithm uses
instead of a single one as proposed by Wang et al in [68].

  * •

Number of Iterations (_NI_): this parameter represents how many times the
algorithm generates new neighbours to replace the current state.

  * •

Initial Temperature (_IT_): this parameter establishes the thermodynamic
energy of the system. At the beginning this energy is high and it decreases
with the cooling schedule [62]. The temperature value controls the selective
pressure: if the temperature is low, the probability of accepting a worse
solution is low; while if temperature is high, the probability of accepting a
worse solution is high, reducing the selection pressure this way.

  * •

Cooling Schedule (_CS_): this parameter represents how the temperature is
gradually reduced as the simulation proceeds. Initially, temperature is set to
a high value, and it decreases at each step according to the cooling schedule,
that must be specified by the user [62]. At the end of the simulation process,
the temperature will be close to 0. The system is expected to wander initially
towards a broad region of the search space containing good solutions, ignoring
small features of the energy function; then drift towards low-energy regions
that become increasingly narrower; and finally move downhill according to the
steepest descent heuristic.

Table 1: Parameters (factors) and the abbreviation used as reference later that determine the SA behaviour and values considered to apply ANOVA. Cooling | Number of | Number of | Population | Initial
---|---|---|---|---
Schedule (CS) | Changes (NC) | Iterations (NI) | Size (PS) | Temperature (IT)
Cauchy | 1000 | 2 | 1 | 10
Cauchy Modif. | 2000 | 4 | 2 | 50
Exponential | 4000 | 8 | 4 | 100
| 8000 | 16 | 8 |
| 16000 | 32 | 16 |

This parameter has an essential role in the efficiency and the effectiveness
of the algorithm. Three Cooling Schedules (those widely used by practitioners
[69, 70, 2, 67, 62]) are used to determine which one yields better results.
Using these schemes it can be seen as a way to speed the execution (i.e. using
a Cauchy distribution makes the annealing schedule exponentially faster).

The _Cauchy_ cooling schedule follows Equation 1 [69]:

$T_{k}=\frac{1}{1+k}$ (1)

where $T_{k}$ is the temperature at iteration $k$.

The _Modified Cauchy_ cooling schedule follows Equation 2 [69]:

$\beta=\frac{T_{0}-T_{n}}{NCT_{0}T_{n}};T_{k}=\frac{T_{k-1}}{1+\beta T_{k-1}}$
(2)

where $T_{k}$ is the temperature at iteration $k$, $T_{0}$ is the initial
temperature, $T_{n}$ is the final temperature (fixed to $0.000001$) and $NC$
is the number of times we have to apply the cooling schedule.

The _Exponential_ cooling schedule follows Equation 3 [70]:

$T_{k}=\alpha T_{k-1}$ (3)

where $T_{k}$ is temperature at iteration $k$ and $\alpha$ usually takes a
value between $0.85$ and $0.99$ (in these experiments, it is fixed to $0.9$).

As can be seen, this algorithm needs some parameters to be given adequate
values. In literature, different sets of values can be found, obtained either
using trial and error or by means of a theoretical analysis [71, 43]. However,
our interest focuses on testing the effectiveness of those values using robust
mathematical methods.

###  3.2 Methodology

Once the optimization algorithm under study has been presented and the
configuration parameters have been determined, the different levels (values)
for those parameters have to be set before ANOVA is applied.

The set of values for each parameter was chosen taking into account those that
can be found in the bibliography [68, 70, 72, 69, 2, 62]. Table 1 shows the
different levels considered to evaluate these parameters using ANOVA. Values
for _NC_ , _NI_ and _IT_ parameters were set incrementally. In this case,
33750 runs were carried out for each problem (30 times * 3 levels of _CS_ * 5
levels for _NC_ * 5 levels for _NI_ * 5 levels for _PS_ * 3 levels for _IT_ ,
that represent the possible combinations) to obtain the fitness for each
combination.

The application of ANOVA consisted in running the proposed SQ optimization
method using those parameter combinations to obtain the best fitness. The
response variable used to perform the statistical analysis is the fitness at
the end of the simulation. The changes in the response variable are produced
when a new combination of parameters is considered. Then the R
111http://www.r-project.org tool was used to obtain the ANOVA tables as well
as the tables of means and figures for each problem. Appendix I shows how to
use R to analyze data to generate tables and figures related to the Ackley
problem (as an example). Obtained ANOVA tables (Section 4) show for each
factor, the freedom degree (FD), the value of the statistical F (F value) and
its associated p-value. As previously stated, if the output is smaller than
0.05, then the factor effect is statistically significant at a $95\%$
confidence level (what indicates that different initial values of this
parameter give significative differences on the fitness).

As a significant F-value tells only that the effects are not all equal (i.e.,
reject the null hypothesis), post-hoc tests might be used to determine which
effects or outputs are significantly different from which other. In that
sense, one of the most widely used post-hoc test is Tukey’s Honestly
Significant Difference test [15]. Tukey HSD is a versatile, easily calculated
technique that might be used after ANOVA and allows saying exactly where the
significant differences are. However, it can only be used when the ANOVA found
a significant effect. If the F-value for a factor turns out non-significant,
the further analysis is not needed. The Tukey HSD compares pairs of the factor
values, showing a segment (confidence level) for each comparison.
Additionally, it shows a vertical dotted line (distance equals zero) that
intersects some segments. Significant differences can be found in those cases
where the vertical line does not intersect a segment (those values compared
are significantly different).

###  3.3 Function Approximation Problems

In order to evaluate the proposed metaheuristic and their parameters, two
experiments are proposed: in the first one (Section 4.1), four well known
function approximation problems are used (Griewangk [73], Rastrigin [74],
Ackley [75, 76] and Rana [77]). These problems are the more representative
among those faced in our research.

In the second experiment (Section 4.2), the estimation of the parameters
involved in a stochastic model (the lognormal diffusion process [78]) is
carried out.

Next, a detailed description of the four benchmark functions is given:

  * •

The Griewangk function [73] is a continuous multimodal function with a high
number of local optima. It is defined according to Equation 4 and its global
optimum is located at point $x_{i}=0,i=1:100$ [79]. This problem has been
addressed with vectors of n=100 real numbers in the interval $[-512,512]$.

$f_{n}(\vec{x})=\frac{1}{4000}\sum_{i=1}^{n}x_{i}^{2}-\prod_{i=1}^{n}cos(\frac{x_{i}}{\sqrt{i}}+1)$
(4)

  * •

The Rastrigin function [74] (see Equation 5) is a multimodal real function
optimization problem, whose global optimum is located at point
$x_{i}=0,i=1:100$ and whose minimum value is 0. This problem has been
addressed with vectors of n=100 real numbers in the interval $[-50,50]$.

$f_{n}(\vec{x})=10n+\sum_{i=1}^{n}x_{i}^{2}-10cos(2\pi x_{i})\\\ $ (5)

  * •

The Ackley function [75, 76] (see Equation 6) is a multimodal non separable
and regular function usually used as test function. The global optimum is
located at point $x_{i}=0,i=1:100$. This problem has been addressed with
vectors of n=100 real numbers in the interval $[-100,100]$.

$f_{n}(\vec{x})=20+e-20\exp[-0.2\sqrt{\frac{1}{n}\cdotp\sum_{i=1}^{n}x_{i}^{2}}]$
(6)

$-\exp[\frac{1}{n}\sum_{i=1}^{n}(cos(2\pi x_{i})]$

  * •

The Rana function [77] (see Equation 7) is a non-separable, highly multimodal
function, whose global optimum is located at point $x_{i}=-514.04,i=1:100$.
Vectors of n=100 real numbers in the interval $[-520,520]$ have been
considered.

$\displaystyle
f_{n}(\vec{x})=\sum_{i=1}^{n}((x_{i+1}+1)cos(\sqrt{\left|{x_{i+1}-x_{i}+1}\right|})$
(7)

$sin(\sqrt{\left|{x_{i+1}+x_{i}+1}\right|})$

$+x_{i}cos(\sqrt{\left|{x_{i+1}+x_{i}+1}\right|})sin(\sqrt{\left|{x_{i+1}-x_{i}+1}\right|}))$

In all cases, as the optimum is known, the fitness of an individual is
calculated as the distance to the optimum for that function, and the goal is
to obtain the smallest fitness for the optimized function.

On the second experiment, we propose estimating the parameters of lognormal
diffusion process. This stochastic process has been widely studied from the
point of view of his applications for modeling real data. This is because
there exist a lot of variables that are inherent positive, such as population
size in biology [80], precipitations in meteorology, gas emission in
environmental sciences, gross national product or consumer price index in
economy [81] or maternal age in the first pregnancy in demography. The use of
a stochastic process for modeling these situations is due by the fact that
deterministic models can not embrace all the factors that are involved in this
kind of phenomena.

The univariate homogeneous lognormal diffusion process is defined by a
diffusion $\\{X(t);t_{0}\leq t\leq T\\}$, taking values on $\mathbb{R}^{+}$
and with infinitesimal moments $A_{1}(x)=mx$ and $A_{2}(x)=\sigma^{2}x^{2}$,
where $m\in\mathbb{R}$ and $\sigma>0$.

Table 2: ANOVA tables for the fitness (response) with the parameters as factors. Note that in all functions a change in _CS_ parameter significantly influences the results. Factor _PS_ is not relevant in any case, which confirms that SQ algorithm operation is accurate using just one individual.  | Param. | FD | F | $P(Sig.Level)$
---|---|---|---
CS | 2 | $1592.9$ | $<2.2e-16$
NC | 4 | $0.5001$ | $0.7357$
NI | 4 | $0.0562$ | $0.9941$
PS | 4 | $0.6748$ | $0.6095$
IT | 2 | $53.111$ | $<2.2e-16$
|  |  | Param. | FD | F | $P(Sig.Level)$
---|---|---|---
CS | 2 | $571.45$ | $<2.2e-16$
NC | 4 | $14.699$ | $1.042e-11$
NI | 4 | $8.2081$ | $1.593e-06$
PS | 4 | $0.9574$ | $0.43$
IT | 2 | $79.214$ | $<2.2e-16$
Griewangk |  |  | Rastrigin
| Param. | FD | F | $P(Sig.Level)$
---|---|---|---
CS | 2 | $378.38$ | $<2.2e-16$
NC | 4 | $45.951$ | $<2.2e-16$
NI | 4 | $7.6122$ | $4.739e-06$
PS | 4 | $0.1176$ | $0.9763$
IT | 2 | $0.0073$ | $0.9927$
|  |  | Param. | FD | F | $P(Sig.Level)$
---|---|---|---
CS | 2 | $2701.9$ | $<2.2e-16$
NC | 4 | $10.244$ | $3.788e-08$
NI | 4 | $2.1146$ | $0.0769$
PS | 4 | $0.5427$ | $0.7044$
IT | 2 | $2.273$ | $0.1035$
Ackley |  |  | Rana

It has been applied for modeling phenomena with random variables that evolve
through the time and show exponential trends. Moreover, its mean and mode
functions could be used for predictions. Therefore, the inference on these two
functions has been object of considerable study. In Gutiérrez et al. [78] a
general study was made to obtain maximum likelihood estimators (MLE) and
uniformly minimum variance unbiased estimators (UMVUE) for general parametric
functions of the lognormal diffusion process, that are both based on the MLE
of the parameters $a$ and $\sigma^{2}$. For fixed times $t_{1},...,t_{n}$, we
observe the variables $X(t_{1}),...,X(t_{n})$ and their values
$x_{1},...,x_{n}$ make up the basic sample from the inferential process is
carried out. Assuming $P[X(t_{1})=x_{1}]=1$, we can get MLE for the parameters
by maximizing the likelihood function, Equation 8, for the sample:

$\displaystyle
L(a,\sigma^{2})=\frac{1}{(2\pi)^{(n-1)/2}(\sigma^{2})^{(n-1)/2}}$ (8)

$\exp(-\frac{1}{2\sigma^{2}}\sum_{i=2}^{n}(\frac{(\ln(x_{i})-\ln(x_{i-1}))^{2}}{t_{i}-t_{i-1}}$

$+a^{2}(t_{i}-t_{i-1})-2a(\ln(x_{i})-\ln(x_{i-1}))))$

In this problem we propose searching for the optimum parameter values of the
likelihood function using SQ in order to maximize it. The aim is to establish
which of the parameters have a significant effect on the estimate. To do so, a
path of simulated diffusion process with $m=0$, $t_{i}=i-1$, $i=1,...,101$ y
$\sigma^{2}=10^{-5}$ is taken.

Finally, paying attention to the change operator (or new state generator), and
taking into account that individuals are vectors of real numbers, a uniform
random mutation is used. A uniform random variable in the interval $[a,b]$ is
generated. The parameter $a$ is in general equal to $-b$. The offspring is
generated within the hyperbox $x+U(-b,b)^{n}$ where $b$ represents a constant
that depends on the interval the real numbers are defined [82, 62].

##  4 Statistical Study and Obtained Results

In this section, the ANOVA and Tukey HSD statistical tools are applied to
determine whether the influence on parameter values (factors) is significant
in the obtained fitness. ANOVA tables are shown as well as Tukey HSD plots and
tables of means (showing the effect of each parameter level on the fitness).

As stated before, in the Subsection 4.1 four well known function approximation
problems are used, while Subsection 4.2 analyses a much complex problem (the
estimation of the parameters involved in the lognormal diffusion process).

###  4.1 Experiment 1: Benchmark functions

Table 2 shows the result of applying ANOVA on proposed approximation function
problems using the SQ algorithm. Parameters with a significance level over
$95\%$ are highlighted in boldface.

The ANOVA analysis shows that _CS_ parameter influence the obtained fitness,
which indicates that changes in this parameter influence the results
significantly. Note that _PS_ has not been reported as significant according
to ANOVA tables, which confirms the fact that SA algorithm operation is
accurate using just one individual. Noteworthy that for some problems _NC_ ,
_NI_ and _IT_ are significant, though not in all cases. This fact shows how
for each problem different set of parameters can influence results in a
different manner [83].

Once the significant F-values have been obtained, Tukey HSD test will help to
determine which means are significantly different. Thus, Figures 1 to 4 show
where the significant differences for those parameters with significant
F-value are.

Figure 1: TukeyHDS test results for Griewangk function. As can be seen, no significant differences between M and E schemes can be observed (the vertical line intersects the confidence segment corresponding to E-M). Likewise, the lowest value of _IT_ is the most accurate, having found significant differences when compared to the others. Figure 2: As in the previous problem, TukeyHDS test for the Rastrigin function shows no significant differences between M and E schemes. Differences between the lowest value and the highest values of _IT_ have been found. In the case of _NC_ and _NI_ differences appear when comparing extreme values, but not between close values (the vertical line crosses the confidence segments corresponding to comparisons between close values). Figure 3: TukeyHDS test results for Ackley function show clear differences between the E and M scheme regarding C; as in previous problem, in the case of _NC_ and _NI_ differences appear when comparing extreme values. Figure 4: TukeyHDS test results for Rana function. As in previous functions, no significant differences between M and E schemes can be observed (although clear differences can be found regarding C). Paying attention to _NC_ , only small values present significant differences regarding the others (the vertical line does not intersect the confidence segments corresponding to these comparisons). Table 3: Table of means for Griewangk function. As can be seen, results show significant differences in _CS_ and _IT_ cases (which is in agreement with the Tukey HSD results). Using the cooling scheme M yields to better fitness, although when compared to E scheme, differences are not significant. Finally, small values for _IT_ yields to better results.  Parameters | Means
---|---
CS | C | M | E
| $1155.51$ | $30.66$ | $43.53$
NC | $1000$ | $2000$ | $4000$ | $8000$ | $16000$
| $460.23$ | $393.81$ | $389.94$ | $397.32$ | $408.21$
NI | $2$ | $4$ | $8$ | $16$ | $32$
| $424.46$ | $413.89$ | $408.19$ | $399.95$ | $403.03$
PS | $1$ | $2$ | $4$ | $8$ | $16$
| $456.16$ | $429.39$ | $406.21$ | $384.40$ | $373.36$
IT | $10$ | $50$ | $100$
| $161.58$ | $483.89$ | $584.24$
Figure 5: Boxplot of the set of solutions grouping using each parameter for the Griewangk function. The distribution in the case of _NC, NI_ and _PS_ is similar, while there is a clear difference with _CS_ and _IT_ (the distribution is more asymmetric and focuses on those values whose average results are better). Table 4: Mean fitness for Rastrigin function. As can be seen, higher values of _NI_ and _NC_ perform better on average; in the same way, using the cooling scheme M yields better results, although according to Tukey HSD, differences regarding E scheme are not significant. Finally, small values of _IT_ yields to better fitness.  Parameters | Means
---|---
CS | C | M | E
| $3191.09$ | $450.16$ | $673.93$
NC | $1000$ | $2000$ | $4000$ | $8000$ | $16000$
| $2178.65$ | $1483.65$ | $1243.50$ | $1162.73$ | $1123.42$
NI | $2$ | $4$ | $8$ | $16$ | $32$
| $1974.32$ | $1516.87$ | $1330.77$ | $1219.74$ | $1150.25$
PS | $1$ | $2$ | $4$ | $8$ | $16$
| $1595.43$ | $1506.09$ | $1424.15$ | $1349.61$ | $1316.68$
IT | $10$ | $50$ | $100$
| $697.43$ | $1413.62$ | $2204.12$
Figure 6: The boxplot of the set of solutions for the Rastrigin function shows differences for _CS_ and _IT_. The box corresponding to _PS_ shows a homogeneous graph, which indicates that no parameter values differ from the rest. Table 5: Table of means for Ackley function. In this problem, best fitness are obtained using higher values of _NI_ and _NC_ , while taking into account _CS_ , using the E scheme yields to best results (significant differences regarding other schemes can be found). Paying attention to _IT_ and _PS_ , no significant differences have been found.  Parameters | Means
---|---
CS | C | M | E
| $21.65$ | $21.62$ | $13.053$
NC | $1000$ | $2000$ | $4000$ | $8000$ | $16000$
| $21.63$ | $21.12$ | $18.84$ | $16.89$ | $15.39$
NI | $2$ | $4$ | $8$ | $16$ | $32$
| $20.36$ | $19.43$ | $18.71$ | $17.86$ | $17.51$
PS | $1$ | $2$ | $4$ | $8$ | $16$
| $18.98$ | $18.84$ | $18.78$ | $18.67$ | $18.61$
IT | $10$ | $50$ | $100$
| $18.74$ | $18.78$ | $18.80$
Figure 7: Boxplot for Ackley function. There are many cases outside the quartiles represented (due to the characteristics of the problem.) Taking into account _CS_ , _CN_ and _NI_ , differences between different groups can be found; while for _IT_ and _PS_ no differences can be observed. Table 6: Mean fitness for Rana function. According to ANOVA, _CS_ and _NC_ parameters are statistically significant. Taking into account the Tukey HSD results too, it can be claimed that the best value for _CS_ is the E scheme (although no differences regarding M have been found). As far as _NC_ parameter is concerned, a middle value yields the best results, while for _NI, PS_ and _IT_ , no significant differences between the values tested can be found.  Parameters | Means
---|---
CS | C | M | E
| $474.30$ | $131.69$ | $125.24$
NC | $1000$ | $2000$ | $4000$ | $8000$ | $16000$
| $304.83$ | $230.59$ | $213.24$ | $216.60$ | $253.46$
NI | $2$ | $4$ | $8$ | $16$ | $32$
| $268.33$ | $253.37$ | $240.22$ | $231.80$ | $225.00$
PS | $1$ | $2$ | $4$ | $8$ | $16$
| $255.99$ | $248.53$ | $242.45$ | $238.36$ | $233.39$
IT | $10$ | $50$ | $100$
| $228.055$ | $248.47$ | $254.71$
Figure 8: The boxplot of the set of solutions for the Rana function shows the
results discussed in Table 6. Taking into account _CS_ , M and E schemes are
very similar, while for _NC_ significative differences between distant
parameter values can be found (although no differences between close values).

Tukey HSD test shows that in general, no significative differences can be
found between M or E schemes (the vertical line intersects the confidence
segment corresponding to E-M), although clear differences appear regarding C
(it is the least effective). _PS_ parameter has not been tested with Tukey HSD
as it has not been reported significant according to ANOVA test. In general,
the initial temperature (_IT_) must be kept low, value for which significant
differences were found. For the rest of parameters (_NC_ and _NI_), in those
problems where these parameters were found as significant according to ANOVA,
significative differences between pairs of extreme values do exist (the
vertical line -distance 0- does not intersect those confidence segments
corresponding to comparisons of extreme values).

After the parameters with greater influence on the results have been
determined, accurate parameter values should be established in order to obtain
an optimal operation. To do so, tables of means and boxplots [84] are obtained
to show the effect each level has on the approximation error (see Tables 3 to
6 and Figures 5 to 8).

Paying attention to the cooling schedule, in general there are no differences
between M and E schedules; however, clear differences can be found regarding C
schedule. As far as _NC_ and _NI_ are concerned, the higher these parameter
values, the better the average fitness (as these parameters determine how many
times the algorithm tries to find a new solution). Using small values for _IT_
leads to obtain better average results. Taking into account _PS_ , no clear
differences can be found (according to the results of ANOVA and Tukey HSD
tests).

The analysis shows that obtained results are in agreement with those obtained
by means of theoretical analysis and available in bibliography [7, 43].
Moreover, obtained results might help practitioners in adjusting parameters of
new optimization methods.

###  4.2 Experiment 2: Parameter estimate in the lognormal diffusion process

Next, the ANOVA and Tukey HSD statistical tools are applied to determine the
influence of parameter values on the obtained fitness for the likelihood
function. ANOVA tables are shown as well as Tukey HSD plots and tables of
means.

Table 7: Likelihood function. ANOVA table for the fitness (response) with the optimization algorithm parameters as factors.  Param. | FD | F | $P(Sig.Level)$
---|---|---|---
CS | 2 | $0.151$ | $0.860$
NC | 4 | $125.07$ | $<2.2e-16$
NI | 4 | $124.43$ | $<2.2e-16$
PS | 4 | $4.23$ | $0.002$
IT | 2 | $0.43$ | $0.649$

Table 7 shows the result of applying ANOVA on likelihood function using the SQ
algorithm. Parameters with a significance level over $95\%$ are highlighted in
boldface.

Figure 9: TukeyHDS test results for Likelihood function. In the case of _NC_ , _NI_ and _PS_ differences appear when comparing extreme values, but not between close values (the vertical line crosses the confidence segments corresponding to comparisons between close values). Table 8: Mean fitness for Likelihood function. As can be seen, results show significant differences in _NI_ , _NC_ and _PS_ cases (which is in agreement with the ANOVA and Tukey HSD results). Taking into account _CS_ , using the E scheme yields to best results (although significant differences regarding other schemes could not be found). Paying attention to _IT_ , small values of _IT_ yields to better fitness.  Parameters | Means
---|---
CS | C | M | E
| $8.16e+90$ | $7.65e+90$ | $7.94e+90$
NC | $1000$ | $2000$ | $4000$ | $8000$ | $16000$
| $6.96e+89$ | $2.11e+90$ | $5.77e+90$ | $1.13e+91$ | $1.97e+91$
NI | $2$ | $4$ | $8$ | $16$ | $32$
| $6.65e+89$ | $2.35e+90$ | $5.50e+90$ | $1.16e+91$ | $1.96e+91$
PS | $1$ | $2$ | $4$ | $8$ | $16$
| $5.47e+90$ | $7.12e+90$ | $8.09e+90$ | $9.11e+90$ | $9.80e+91$
IT | $10$ | $50$ | $100$
| $8.37e+90$ | $7.53e+90$ | $7.86e+90$
Figure 10: The boxplot of the set of solutions for the Likelihood function
shows differences for _NC_ , _NI_ and _PS_ (significative differences between
distant parameter values can be found (although no differences between close
values). Boxes corresponding to _CS_ and _IT_ show homogeneous graphs, which
indicates that no parameter values differ from the rest.

The ANOVA analysis shows that NC and NI parameters influence the obtained
fitness. In this case, PS has been reported as significant; it might be due to
the difficulty of this problem in which using several states help to explore
the search space.

Again, once the significant F-values have been obtained, Tukey HSD test help
to determine which means are significantly different. Thus, Figure 9 shows
where the significant differences for those parameters with significant
F-value are.

Tukey HSD shows that NC, NI and PS are significant when pairs of extreme
values are compared. CS and IT parameters have not been tested as they have
not been reported significant according to ANOVA; in any case, these
parameters will be examined through tables of means and boxplots.

After the parameters with greater influence on the results have been
determined, tables of means and boxplots have been obtained to show the effect
each level has on the approximation error (see Table 8 and Figure 10).

As can be seen, higher values of _NI_ , _NC_ and _PS_ perform better on
average. In the same way, using the cooling scheme E yields better results,
although according to ANOVA, this parameter was not reported as significant.
Finally, small values of _IT_ yields to better fitness, although no
significant differences have been found.

In any case, in this paper we have verified the relative importance of these
parameters on the fitness through a rigorous statistical study.

##  5 Conclusions and Work in Progress

This paper proposes a methodology to analyze the parameters with a higher
influence on the performance for a given optimization method. Our approach has
been tested using a simulated annealing based algorithm (SQ), a widespread
optimization method successfully used by practitioners in many areas. We also
report appropriate values (among those tested) for these parameters in order
to obtain an optimal operation.

A statistical study of the different parameters involved in the design of this
optimization method has been carried out by applying ANOVA, which consists of
a set of statistical techniques that analyze and compare experiments by
describing the interactions and interrelations between the variables or
factors of the system, and completed with a Tukey HSD test. The motivation of
the present statistical study lies in the great variety of alternatives that a
designer has to take into account when designing an optimization method.

The proposed methodology has been applied to four well-known function
approximation problems (Griewangk, Rastrigin, Ackley and Rana), widely used by
practitioners, in order to determine which parameters have a higher influence
on results (a change on those parameters will affect the algorithm
performance). In a second experiment, a much complex problem has been
addressed. In this case, the estimation of the parameters involved in a
stochastic model (lognormal diffusion process [78]) has been carried out.

In that context, results show that in most cases, using the ”Exponential” and
”Modified Cauchy” cooling schemes yields better fitness values. If the initial
solution is good enough, a cooling scheme where the probability of accepting a
worst solution is higher, is more appropriate to achieve a better solution
(i.e. ”Exponential”). However, if the initial solution is not good enough,
then a cooling scheme that allows accepting a worse solution is more accurate.
In this sense, ”Modified Cauchy” works better. However, according to the Tukey
HSD test results, no significative differences can be found between them,
although clear differences can be found with ”Cauchy” schedule. In general,
accurate results are obtained for high values of _NC_ and _NI_ parameters (the
higher these parameter values, the better the average fitness obtained, as
these parameters determine how many times the algorithm tries to find a new
solution). Taking into account the Tukey HSD test, only for extreme values
(far one from the others) significative differences can be found. As far as
the _IT_ parameter is concerned, using small values for this parameter leads
to obtain better average results. The Tukey HSD test confirms this fact
showing that small values present significative differences regarding higher
values. Paying attention to _PS_ , using a high value yields to better fitness
values, although this increases the number of evaluations and time needed to
run the algorithm. However it has not been reported as significant, which is
in agreement to the use of the typical SA algorithm (that uses a single
individual or state).

In the case of the lognormal diffusion process parameter estimate, results
show that using the E scheme yields to best results (although significant
differences regarding other schemes could not be found). Accurate results are
obtained using a high value of _PS_. This might be due to the difficulty of
the problem (using several states help to explore the search space). As in the
previous problems, using a high value of _NC_ and _NI_ parameters yields to
accurate results (as these parameters determine how many times the algorithm
tries to find a new solution, and this results on a wider exploration).
Finally, small values of _IT_ yields to better fitness, although no
significant differences have been found according to ANOVA and Tukey HSD
results.

As can be seen, different algorithms or configurations might work better on a
problem and worse on another one. This is in agreement with the No Free Lunch
theorem [83], according to which there is no algorithm better than all to
solve all the problems.

This methodology based on ANOVA and TukeyHDS statistical tests could be
helpful for practitioners in analyzing and adjusting parameters of any
optimization method.

Our work in progress includes the analysis of different optimization methods,
such as modified EAs considering several selection schemes, new genetic
operators and other meta-heuristics (tabu search, EDAs, UMDA, etc). As future
work, the implementation of a parameter control method would be of interest,
as proposed by Eiben et al. [21]. In this case, ANOVA could be used to analyze
not only the optimization method parameters but also the control strategy
parameters.

Finally, software is available under GNU public license at
http://genmagic.ugr.es

## acknowledgements

This work has been supported in part by the CEI BioTIC GENIL (CEB09-0010)
MICINN CEI Program (PYR-2010-13) project, the Junta de Andalucía TIC-3903 and
P08-TIC-03928 projects, and the Jaén University UJA-08-16-30 project. The
authors are very grateful to the anonymous referees whose comments and
suggestions have contributed to improve this paper.

## References

  * [1] A.E. Eiben and J.E. Smith.  Introduction to Evolutionary Computing.  Springer, ISBN 3-540-40184-9, 2003.
  * [2] Z. Michalewicz and D.B. Fogel.  How to Solve It: Modern Heuristics.  Springer-Verlag, 2 edition, 2004.
  * [3] Yang X. S.  Nature-Inspired Metaheuristic Algorithms.  2nd Edition, Luniver Press., 2010.
  * [4] S. Kirkpatrick, C.D. Gerlatt, and M.P. Vecchi.  Optimization by Simulated Annealing.  Science 220, 671-680, 1983.
  * [5] N. Ansari, R. Sarasa, and G. Wang.  An efficient annealing algorithm for global optimization in Boltzmann machines.  Applied Intelligence (1993) 3: 177-192, September 01, 1993.
  * [6] A. Das and B. K. Chakrabarti.  Quantum Annealing and Related Optimization Methods.  Lecture Note in Physics, Vol. 679, Springer, Heidelberg, 2005.
  * [7] J. De Vicente, J. Lanchares, and R. Hermida.  Placement by Thermodynamic Simulated Annealing.  Physics Letters A, Vol. 317, Issue 5:6, pp.415–423, 2003.
  * [8] P. Van-Hentenryck and L. Michel.  Constraint-Based Local Search.  The MIT Press. ISBN: 978-0-262-22077-4, 2005.
  * [9] H. H. Hoos and T. Stutzle.  Stochastic Local Search. Foundations and Applications.  Morgan Kaufmann / Elsevier, 2004.
  * [10] N.K. Bambha, S.S. Bhattacharyya, J. Teich, and E. Zitzler.  Systematic integration of parameterized local search into evolutionary algorithms.  IEEE Transactions on Evolutionary Computation, Vol. 8, Issue 2, pp. 137-155. ISSN: 1089-778X, 2004.
  * [11] A.E. Eiben, R. Hinterding, and Z. Michalewicz.  Parameter Control in Evolutionary Algorithms.  IEEE Transactions on Evolutionary Computation, 3(2):124-141, ISSN: 1089-778X, DOI:10.1109/4235.771166, 1999.
  * [12] C. L. Karr and E. Wilson.  A self-tuning evolutionary algorithm applied to an inverse partial differential equation.  Applied Intelligence, 19(3):147-155, November, 2003.
  * [13] S.K. Smit and A.E. Eiben.  Beating the world champion Evolutionary Algorithm via REVAC Tuning.  In Proc. of the WCCI2010 IEEE World Congress on Computational Intelligence, pp. 1295-1302. issn 978-1-4244-8126-2. IEEE Press. Barcelona, Spain, 2010.
  * [14] R.A. Fisher.  Theory of Statistical Estimation.  Proceedings of the Cambridge Philosophical Society, 22, pp.700-725, 1925.
  * [15] P. Dickinson and B. Chow.  Some properties of the Tukey test to Duckworth’s specification / by Peter Dickinson, Bryant Chow.  Office of Institutional Research, University of Southwestern Louisiana, Lafayette, Louisiana, USA, 1971.
  * [16] L. Ingber.  Simulated annealing: Practice versus theory.  Mathl. Comput. Modelling, Vol. 18, n. 11, pp. 29-57, 1993.
  * [17] L. Davis.  Handbook of genetic algorithms.  Van Nostrand Reinhold, NY, 1991.
  * [18] I. Jagielska, C. Matthews, and T. Whitfort.  An investigation into the application of neural networks, fuzzy logic, genetic algorithms, and rough sets to automated knowledge acquisition problems.  Neurocomputing, Vol. 24, no.1-3, pp.37-54, 1999.
  * [19] Swagatam Das, Archana Chowdhury, and Ajith Abraham.  A bacterial evolutionary algorithm for automatic data clustering.  In Proceedings of the Eleventh conference on Congress on Evolutionary Computation, CEC’09, pages 2403–2410, Piscataway, NJ, USA, 2009\. IEEE Press.
  * [20] K.A. De Jong.  An analysis of the behavior of a class of genetic adaptive systems.  Ph.D. thesis, University of Michigan, Ann Arbor, 1975.
  * [21] A. E. Eiben, Z. Michalewicz, M. Schoenauer, and J. E. Smith.  Parameter control in evolutionary algorithms.  Parameter Setting in Evolutionary Algorithms, Studies in Computational Intelligence. Volume 54/2007, pp.19-46. ISBN 978-3-540-69431-1, ISSN 1860-949X. Springer Berlin / Heidelberg, 2007.
  * [22] S. K. Smit and A. E. Eiben.  Using Entropy for Parameter Analysis of Evolutionary Algorithms.  in Bartz Beielstein et al. (eds.), Empirical Methods for the Analysis of Optimization Algorithms, Natural Computing Series, Springer, 2009\.
  * [23] A. E. Eiben.  Principled Approaches to tuning EA parameters.  Tutorials - IEEE Congress on Evolutionary Computation (CEC2009). Available at http://www.few.vu.nl/~gusz/papers/eiben-cec-2009-tutorial-corrected.pdf, 2009\.
  * [24] S. K. Smit and A. E. Eiben.  Comparing parameter tuning methods for evolutionary algorithms.  In CEC’09: Proc. of the Eleventh conference on Congress on Evolutionary Computation, pages 399–406, Piscataway, NJ, USA, 2009. P. Haddow et al. (Eds.), IEEE Press.
  * [25] Y. Gong and A. Fukunaga.  Distributed Island-Model Genetic Algorithms Using Heterogeneous Parameter Settings.  Proceedings of the 2011 IEEE Congress on Evolutionary Computation (CEC2011), pp. 820-827. ISBN 978-1-4244-7834-7. New Orleans, LA, USA., 2011.
  * [26] J.J. Grefenstette.  Optimization of control parameters for genetic algorithms.  IEEE Trans. Systems, Man, and Cybernetics, SMC-16(1):122-128, 1986\.
  * [27] D. Kim and C. Kim.  Forecasting time series with genetic fuzzy predictor ensemble.  IEEE Transactions on Fuzzy Systems, vol.5,no.4,pp.523-535, 1997\.
  * [28] K. Liang, X. Yao, and C.S. Newton.  Adapting Self-Adaptive Parameters in Evolutionary Algorithms.  Applied Intelligence (2001) 15: 171-180, November 01, 2001.
  * [29] M.J. Gacto, R. Alcala, and F. Herrera.  A multi-objective evolutionary algorithm for an effective tuning of fuzzy logic controllers in heating, ventilating and air conditioning systems.  Applied Intelligence (2010): 1-18, November 16, 2010.
  * [30] L. Diosan, A. Rogozan, and J.P. Pecuchet.  Improving classification performance of Support Vector Machine by genetically optimising kernel shape and hyper-parameters.  Applied Intelligence (2010): 1-15, October 27, 2010.
  * [31] A. Aksac, E. Uzun, and T. Ozyer.  Areal time traffic simulator utilizing an adaptive fuzzy inference mechanism by tuning fuzzy parameters.  Applied Intelligence (2011): 1-23, April 19, 2011.
  * [32] T. Bäck, D. Fogel, and Z. Michalewicz.  Handbook of Evolutionary Computation.  Institute of Physics Publishing Ltd, Bristol and Oxford University Press, NY, 1997.
  * [33] R. Hinterding, Z. Michalewicz, and A.E. Eiben.  Adaptation in Evolutionary Computation: A Survey.  Proceedings of the 4th IEEE Conference on Evolutionary Computation, pp.65-69. IEEE Press, 1997.
  * [34] G. R. Harik and F. G. Lobo.  A parameter-less genetic algorithm.  In Wolfgang Banzhaf, Jason Daida, Agoston E. Eiben, Max H. Garzon, Vasant Honavar, Mark Jakiela, and Robert E. Smith, editors, Proceedings of the Genetic and Evolutionary Computation Conference, volume 1, pages 258–265, Orlando, Florida, USA, 13-17 1999. Morgan Kaufmann.
  * [35] T. Bäck.  Optimal mutation rates in genetic search.  Proceedings of the 5th International Conference on Genetic Algorithms. S.Forrest (editor). Morgan Kaufmann. pp.2-8, 1993.
  * [36] David E. Goldberg.  Genetic Algorithms in Search, Optimization and Machine Learning.  Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1989.
  * [37] D.E. Goldberg, K. Deb, and D. Theirens.  Toward a better understanding of mixing in genetic algorithms.  Proc. of the 4th Int. Conf. on Genetic Algorithms. R.K. Belew and L.B. Booker (editors). Morgan Kaufmann. pp. 190-195, 1991.
  * [38] G. Harik, E. Cantú-Paz, D.E. Goldberg, and B.L. Miller.  The gambler’s ruin problem, genetic algorithms, and the sizing of populations.  Proceedings of the 4th IEEE Conference on Evolutionary Computation. IEEE Press. pp.7-12. ISBN: 0-7803-3949-5., 1997.
  * [39] J.D. Schaffer and A. Morishima.  An adaptive crossover distribution mechanism for genetic algorithms.  Proceedings of the 2nd International Conference on Genetic Algorithms and Their Applications. J.J. Grefenstette (editor). Lawrence Erlbaum Associates. pp.36-40, 1987.
  * [40] D. Thierens.  Dimensional analysis of allele-wise mixing revisited.  Proceedings of the 4th Conference on Parallel Problem Solving from nature, number 1141 in Lecture Notes in Computer Science. H.M. Voigt, W. Ebeling, I. Rechenberg and H.P. Schwefel (editors). Springer, Berlin. pp.255-265, 1996.
  * [41] Thomas Jansen and Dennis Weyland.  Analysis of evolutionary algorithms for the longest common subsequence problem.  In GECCO ’07: Proceedings of the 9th annual conference on Genetic and evolutionary computation, pages 939–946, New York, NY, USA, 2007\. ACM.
  * [42] Thomas Jansen and Dennis Weyland.  Analysis of Evolutionary Algorithms for the Longest Common Subsequence Problem.  Algorithmica, 57(1):170–186, 2010.
  * [43] Dennis Weyland.  Simulated annealing, its parameter settings and the longest common subsequence problem.  In GECCO ’08: Proceedings of the 10th annual conference on Genetic and evolutionary computation, pages 803–810, New York, NY, USA, 2008\. ACM.
  * [44] Fernando G.. Lobo, Claudio F. Lima, and Zbigniew Michalewicz.  Parameter Setting in Evolutionary Algorithms.  Springer Publishing Company, Incorporated, 2007.
  * [45] V. Nannen and A.E. Eiben.  A method for parameter calibration and relevance estimation in evolutionary algorithms.  In M. Keijzer, editor, Proc. of the Genetic and Evolutionary Computation Conference (GECCO2006), pp. 183-190. Morgan Kaufmann, San Francisco, 2006.
  * [46] V. Nannen and A.E. Eiben.  Relevance estimation and value calibration of evolutionary algorithm parameters.  In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI2007), pp. 975-980. AAAI Press, 2007.
  * [47] S.K. Smit and A.E. Eiben.  Comparing parameter tuning methods for evolutionary algorithms.  In Proc. of the 2009 IEEE Congress on Evolutionary Computation, A. Tyrrell (Ed.), pp. 399-406, Thondheim, IEEE Press, 2009.
  * [48] T. Bartz, K.E. Parsopoulos, and M.N. Vrahatis.  Analysis of Particle Swarm Optimization Using Computational Statistics.  In Chalkis, editor, Proc. of the International Conference of Numerical Analysis and Applied Mathematics (ICAAM2004), pp. 34-37, 2004.
  * [49] T. Bartz and S. Markon.  Tuning search algorithms for real-world applications: A regression tree based approach.  Technical Report of the Collaborative Research Centre 531 Computational Intelligence CI-172/04, University of Dortmund, March 2004.
  * [50] T. Bartz, C.W.G. Lasarczyk, and M. Preuss.  Sequential parameter optimization.  In Proc. of the 2005 IEEE Congress on Evolutionary Computation, pp. 773-780, Vol. 1, Edinburgh, UK, 2005.
  * [51] M. Birattari, T. Stutzle, L. Paquete, and K. Varrentrapp.  A racing algorithm for configuring metaheuristics.  In Proc. of the Genetic and Evolutionary Computation Conference, pp. 11-18, NY USA. Morgan Kaufmann, 2002.
  * [52] F. Hutter, H.H. Hoos, and T. Stutzle.  Automatic algorithm configuration based on local search.  In Proc. of the Twenty-Second Conference on Artificial Intelligence (AAAI 2007), pp. 1152-1157. AAAI Press, 2007.
  * [53] E. Montero, M.C. Riff, and B. Neveu.  New requirements for off-line parameter calibration algorithms.  In Proc. of the WCCI2010 IEEE World Congress on Computational Intelligence, pp. 3466-3473. issn 978-1-4244-8126-2. IEEE Press. Barcelona, Spain, 2010.
  * [54] S.W. Leung, S.Y. Yuen, and C.K. Chow.  Parameter Control by the Entire Search History: Case Study of History-driven Evolutionary Algorithm.  In Proc. of the WCCI2010 IEEE World Congress on Computational Intelligence, pp. 3688-3695. issn 978-1-4244-8126-2. IEEE Press. Barcelona, Spain, 2010.
  * [55] S.K. Smit and A.E. Eiben.  Parameter tuning of evolutionary algorithms: Generalist vs. specialist.  In Cecilia Di Chio et al., editor, Applications of Evolutionary Computation, volume 6024 of LNCS, pages 542–551. Springer, 2010.
  * [56] J. He, A.H. Tan, and C.L. Tan.  On Machine Learning Methods for Chinese Document Categorization.  Applied Intelligence (2003) 18: 311-322, May 01, 2003, 2003.
  * [57] J. Maudes, J.J. Rodriguez, C. Garcia-Osorio, and C. Pardo.  Random projections for linear SVM ensembles.  Applied Intelligence (2011) 34: 347-359, June 01, 2011.
  * [58] D. Shilane, J. Martikainen, S. Dudoit, and S.J. Ovaska.  A general framework for statistical performance comparison of evolutionary computation algorithms.  Information Sciences, Vol. 178, No. 14, p. 2870-2879, 2008.
  * [59] S. Garcia, D. Molina, M. Lozano, and F. Herrera.  A Study on the Use of Non-Parametric Tests for Analyzing the Evolutionary Algorithms’ Behaviour: A Case Study on the CEC’2005 Special Session on Real Parameter Optimization.  Journal of Heuristics 15, pp. 617-644, 2009.
  * [60] I. Rojas, J. Gonzalez, H. Pomares, J.J. Merelo, P.A. Castillo, and G. Romero.  Statistical Analisys of the main Parameters Involved in the Desing of a Genetic Algorithm.  in IEEE Trans. on Systems, Man and Cibernetics - Part C, vol.32, no.10, pp.31-37, ISSN:1094-6977, IEEE Press, november, 2002.
  * [61] P.A. Castillo, J.J. Merelo, G. Romero, A. Prieto, and I. Rojas.  Statistical Analysis of the Parameters of a Neuro-Genetic Algorithm.  in IEEE Trans. on Neural Networks, vol.13, no.6, pp.1374-1394, ISSN:1045-9227, IEEE Press, november, 2002.
  * [62] E.G. Talbi.  METAHEURISTICS. FROM DESIGN TO IMPLEMENTATION.  University of Lille - CNRS - INRIA. Published by John Wiley and Sons, Inc., Hoboken, New Jersey. USA. ISBN 978-0-470-27858-1, 2009.
  * [63] A.J. Lockett and R. Miikkulainen.  Measure-Theoretic Evolutionary Annealing.  Proceedings of the 2011 IEEE Congress on Evolutionary Computation (CEC2011), pp. 2139-2146. ISBN 978-1-4244-7834-7. New Orleans, LA, USA., 2011.
  * [64] J.C.W. Debuse and V.J. Rayward-Smith.  Discretisation of Continuous Commercial Database Features for a Simulated Annealing Data Mining Algorithm.  Applied Intelligence (1999) 11: 285-295, November 01, 1999.
  * [65] S. Sun, F. Zhuge, J. Rosenberg, R. Steiner, G. Rubin, and S. Napel.  Learning-enhanced simulated annealing: method, evaluation, and application to lung nodule registration.  Applied Intelligence (2008) 28: 83-99, December 31, 2008.
  * [66] B. Bonev, M. Cazorla, F. Martin, and V. Matellan.  Portable autonomous walk calibration for 4-legged robots.  Applied Intelligence (2010): 1-12, August 14, 2010.
  * [67] Z. Michalewicz.  Genetic Algorithms + Data Structures = Evolution Programs , Third, Extended Edition.  Springer-Verlag, 1996.
  * [68] Z. G. Wang, M. Rahman, Y. S. Wong, and K. S. Neo.  Development of heterogeneous parallel genetic simulated annealing using multi-niche crowding.  Intl. Journal of Computational Intelligence 3(1):55-62, 2006.
  * [69] P. J. van Laarhoven and E. H. Aarts.  Simulated Annealing: Theory and Applications (Mathematics and Its Applications).  Springer, 1 edition, June 1987.
  * [70] S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi.  Optimization by simulated annealing.  Science, 220:671–680, 1983.
  * [71] K. Weicker.  Evolutionare algorithmen.  Teubner Verlag, 2002.
  * [72] G. B. Smith.  Preface to S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images”.  pages 562–563, 1987.
  * [73] Griewank A.O.  Generalized descent for global optimization.  Journal of Optimization Theory and Applications, 34(1):11–39, 1981.
  * [74] A. Törn and A. Zilinskas.  Global Optimization.  Vol. 350 of Springer Lecture Notes in Computer Science (Springer, Berlin), pp. 1–24, 1989.
  * [75] D. H. Ackley.  A connectionist machine for genetic hillclimbing.  Kluwer, Boston, 1987.
  * [76] T. Bäck.  Evolutionary algorithms in theory and practice.  Oxford University Press, 1996.
  * [77] Darrell Whitley, Deon Garrett, and Jean paul Watson.  Quad search and hybrid genetic algorithms.  In In Genetic and Evolutionary Computation - GECCO 2003, pages 1469–1480. Springer, 2003.
  * [78] R. Gutiérrez, P. Roman, and F. Torres.  Inference on some parametric functions in the univariate lognormal diffusion process with exogenous factors.  Test 10 (2), pp. 357-373, 2001.
  * [79] H. Cho, F. Olivera, and S.D. Guikema.  A derivation of the number of minima of the griewank function.  Applied Mathematics and Computation, 204(2):694–701, 2008.
  * [80] R. M. Capocelli and L. M. Ricciardi.  A diffusion model for population growth in random enviroment.  Theoretical Population Biology, 5:28-41, 1974.
  * [81] M. A. E. Basel, S. A. Ahmad, and M. S. Wafaa.  Modelling the CPI using a lognormal diffusion process and implications on forecasting inflation.  IMA Journal of Management Mathematics, 15:39-51, 2004.
  * [82] F. Herrera, M. Lozano, and A. M. Sánchez.  A taxonomy for the crossover operator for real-coded genetic algorithms: An experimental study.  International Journal of Intelligent Systems, 18:309–338, 2003\.
  * [83] D.H. Wolpert and W.G. Macready.  No Free Lunch Theorems for Optimization.  IEEE Transactions on Evolutionary Computation, 1(1):67-82, 1997\.
  * [84] Yoav Benjamini.  Opening the box of a boxplot.  The American Statistician, 42(4):pp. 257–262, 1988.

## Appendix I

This appendix shows an exhaustive review of commands to R used to analyze data
obtained after the runs and to obtain tables and figures shown in Section 4.
As an example, those commands corresponding to the Ackley function are shown:

  * •

problemData $<$\- Ackley

  * •

problemName $<$\- ”Ackley”

  * •

pathMeans $<$\- ”/APIN/R/Medias”

  * •

pathTukeyHDS $<$\- ”/APIN/R/Tukey”

  * •

pathBoxPlot $<$\- ”/APIN/R/BoxPlot”

  * •

extension $<$\- ”.eps”

  * •

dirTukeyHDS $<$\- paste(pathTukeyHDS,problemName,sep = ””)

  * •

dirMeans $<$-paste(pathMeans,problemName,sep = ””)

  * •

dirBox $<$-paste(pathBoxPlot,problemName,sep = ””)

  * •

dirTukeyHDS $<$\- paste(dirTukeyHDS,extension,sep = ””)

  * •

dirMeans $<$-paste(dirMeans,extension,sep = ””)

  * •

dirBox $<$-paste(dirBox,extension,sep = ””)

  * •

problemData$NC $<$\- as.factor(problemData$NC)

  * •

problemData$NI $<$\- as.factor(problemData$NI)

  * •

problemData$PS $<$\- as.factor(problemData$PS)

  * •

problemData$IT $<$\- as.factor(problemData$IT)

  * •

problemData$CS $<$\- factor(problemData$CS, labels=c(’C’,’M’,’E’))

  * •

shapiro.test(problemData$Fitness)

  * •

levene.test(problemData$Fitness, problemData$NC)

  * •

levene.test(problemData$Fitness, problemData$NI)

  * •

levene.test(problemData$Fitness, problemData$PS)

  * •

levene.test(problemData$Fitness, problemData$IT)

  * •

levene.test(problemData$Fitness, problemData$CS)

  * •

summary(anovaNC$<$-(aov(Fitness NC, data=problemData)))

  * •

summary(anovaNI$<$-(aov(Fitness NI, data=problemData)))

  * •

summary(anovaPS$<$-(aov(Fitness PS, data=problemData)))

  * •

summary(anovaIT$<$-(aov(Fitness IT, data=problemData)))

  * •

summary(anovaCS$<$-(aov(Fitness CS, data=problemData)))

  * •

tapply(problemData$Fitness, list(problemData$NC), mean)

  * •

tapply(problemData$Fitness, list(problemData$NI), mean)

  * •

tapply(problemData$Fitness, list(problemData$PS), mean)

  * •

tapply(problemData$Fitness, list(problemData$IT), mean)

  * •

tapply(problemData$Fitness, list(problemData$CS), mean)

  * •

layout(matrix(1:4,1,4))

  * •

plot(Tukey HSD(anovaNC, ”NC”))

  * •

plot(Tukey HSD(anovaNI, ”NI”))

  * •

plot(Tukey HSD(anovaPS, ”PS”))

  * •

plot(Tukey HSD(anovaIT, ”IT”))

  * •

plot(Tukey HSD(anovaCS, ”CS”))

  * •

dev.copy2eps(file=dirTukeyHDS, width=12.0, height=12.0, pointsize=12)

  * •

layout(matrix(1:4,1,4))

  * •

boxplot(Fitness NC, ylab=”Fitness”, xlab=”NC”, data=problemData)

  * •

boxplot(Fitness NI, ylab=”Fitness”, xlab=”NI”, data=problemData)

  * •

boxplot(Fitness PS, ylab=”Fitness”, xlab=”PS”, data=problemData)

  * •

boxplot(Fitness IT, ylab=”Fitness”, xlab=”IT”, data=problemData)

  * •

boxplot(Fitness CS, ylab=”Fitness”, xlab=”CS”, data=problemData)

  * •

dev.copy2eps(file=dirBox, width=12.0, height=12.0, pointsize=12)