# Quantum Machine Learning hyperparameter search

S. Consul-Pacareu  Lighthouse Disruptive Innovation Group, LLC, 7 Broadway
Terrace, Apt 1, Cambridge MA 02139, Middlesex County, Massachusetts (USA)
<EMAIL_ADDRESS>R. Montaño  Vueling Airlines S.A. Carrer de
Catalunya, 83 Edifici Brasil 08840 Viladecans Spain  Kevin Rodriguez-Fernandez
Vueling Airlines S.A. Carrer de Catalunya, 83 Edifici Brasil 08840 Viladecans
Spain  Àlex Corretgé  Vueling Airlines S.A. Carrer de Catalunya, 83 Edifici
Brasil 08840 Viladecans Spain  Esteve Vilella-Moreno  Lighthouse Disruptive
Innovation Group, LLC, 7 Broadway Terrace, Apt 1, Cambridge MA 02139,
Middlesex County, Massachusetts (USA)  Daniel Casado-Faulí  Lighthouse
Disruptive Innovation Group, LLC, 7 Broadway Terrace, Apt 1, Cambridge MA
02139, Middlesex County, Massachusetts (USA)  Parfait Atchade-Adelomou
Lighthouse Disruptive Innovation Group, LLC, 7 Broadway Terrace, Apt 1,
Cambridge MA 02139, Middlesex County, Massachusetts (USA)
<EMAIL_ADDRESS>Smart Society Research Group - La Salle -
Universitat Ramon Llull, Carrer de Sant Joan de La Salle, 42, 08022 Barcelona
(Spain<EMAIL_ADDRESS>MIT Media Lab - City Science
Group, Cambridge, USA

(Feb 2023)

###### Abstract

This paper presents a quantum-based Fourier-regression approach for machine
learning hyperparameter optimization applied to a benchmark of models trained
on a dataset related to a forecast problem in the airline industry. Our
approach utilizes the Fourier series method to represent the hyperparameter
search space, which is then optimized using quantum algorithms to find the
optimal set of hyperparameters for a given machine learning model. Our study
evaluates the proposed method on a benchmark of models trained to predict a
forecast problem in the airline industry using a standard HyperParameter
Optimizer (HPO). The results show that our approach outperforms traditional
hyperparameter optimization methods in terms of accuracy and convergence speed
for the given search space. Our study provides a new direction for future
research in quantum-based machine learning hyperparameter optimization.

KeyWords: Quantum Computing, Quantum Machine Learning, Fourier Series, QML,
Hyperparameters, Interpolation, Regression model, quantum search problem

##  I Introduction

In recent years, Machine Learning (ML) algorithms have successfully solved
various tasks, reaching state-of-the-art in multiple areas Choy  _et al._
(2018); Callahan and Shah (2017); Anderson  _et al._ (2022); Pallathadka  _et
al._ (2021); Miklosik and Evans (2020); Mosavi  _et al._ (2019); Atchade-
Adelomou  _et al._ (2021a). This is not only due to the development of new
algorithms (more powerful and prominent), but also the selection of good
hyperparameters contributed to this advance. Performing machine learning on
large datasets is a resource-intensive task, but hyperparameter tuning problem
Bardenet  _et al._ (2013); Yang and Shami (2020) increases those resource
requirements by orders of magnitude. Despite advances in hyperparameter
optimization, the precise selection of hyperparameters remains a challenge
regarding computational complexity and finding the best approach. Therefore,
the scientific community has been working to discover efficient techniques to
solve this challenge Yang and Shami (2020); Li  _et al._ (2017); Sagingalieva
_et al._ (2022); Bengio (2000); Wang  _et al._ (2018); Feurer and Hutter
(2019).

Due to its stochastic nature and great computational capacity, quantum
computing is a great bet and approach to take into account in the efficient
search problems of hyperparameters.

In this article, we propose a hybrid (quantum + classical) approach algorithm
to facilitate this task by fitting and executing Fourier-regression models on
a large scale on any type of hyperparameters in the most efficient way
possible. Our proposed quantum algorithm is based on this work Atchade-
Adelomou and Larson (2023), which allows us to find the best hyperparameters
using a quantum approach, given a good enough representation of the search
space (or hyperparameter space) represented by the model hyperparameters
related to a metric. The model was trained based on the results offered by
multiple search methods, such as Grid search, Random Search and Bayes-Based
Search for a given training set from a Vueling forecast problem dataset.
Three-way cross-validation considers each search algorithm’s average scores
during the training process. The proposed quantum method provides the best
score in time compared to classically, assuming that the hyperparameter/score
input search space is highly nonlinear and might not be continuous. The trade-
off between speed and precision depends on the number of features to evaluate.
We use Pennylane framework and AWS Braket to validate our algorithm.

The document is organized as follows. Section (II) presents our primary
motivation behind this work. Section (III) shows previous work on
hyperparameters tuning. Section (IV) illustrates the quantum machine learning
framework and its connection to the hyperparameters tuning problems. In the
section (V), we propose the scenarios and the models we will implement to
tackle the hyperparameters tuning problems. Section (V.3) proposes our model,
considering our primary reference. Section (VI) delivers the obtained results.
Section (VI.4) discusses practically relevant results and their implications.
Finally, this paper ends with conclusions and future work in Section (VII).

##  II Motivation and Problem Statement

The aviation industry is highly competitive, and one of the key factors for
airlines to remain profitable is to maximize revenues while minimizing costs.
To that effect, many of them are adopting advance analytics solutions to
transform the company towards that goal. Solutions range from classical
machine learning problems such as a predictive maintenance Zeldam (2018) or
dynamic pricing Shukla  _et al._ (2019), to optimization problems such as
network optimization Cacchiani and Salazar-González (2020); Parmentier and
Meunier (2020). Nonetheless, optimization problems are computationally
expensive and, in most cases, classical computing falls short in yielding a
reasonable processing time. This also applies to machine learning
hyperparameter optimization.

Vueling has been implementing and using production state-of-the-art
hyperparameter tuning algorithms to achieve high-accuracy results. However,
the company is aware of the potential advancements in the field of quantum
computing and wishes to stay ahead of the curve. With this in mind, Vueling is
proactively exploring ways to incorporate quantum technology into its
technology stack to remain at the forefront of its industry and ensure long-
term success.

One of the critical taks in the airline industry is managing passenger no-
shows. A no-show occurs when a passenger who has purchased a ticket fails to
show up for the flight. The data used in this work is a proprietary Vueling
no-show dataset, that contains crucial information for understanding and
predicting passenger behavior.

The dataset contains as target data the number of no-shows per flight and
contains $252\,183$ datapoints with $42$ features each, such as flight
information (origin, destination, time of flight, etc.), seat reservation
status, $mean$ no-shows for different time windows on a given route, number of
tickets at different price points as well as other proprietary information.

The dataset and the predicting models created allow the company to track
performance over time, evaluate the effectiveness of different strategies, and
make data-driven decisions that can improve overall performance and
profitability.

The scope of this work is not to solve the specific no-show prediction problem
but to use real-life data to implement a Quantum Fourier hyperparameter tuning
algorithm that rivals traditional techniques. In short, showing an efficient
and useful alternative for such optimization problems.

##  III Work Context

Machine learning is used in various fields and areas, allowing computers,
among other uses, to identify patterns in large data and make predictions.
Such a process involves, after all, determining the appropriate algorithm
based on a sample space and obtaining an optimal model architecture by
adjusting some control variables from its learning process known as
Hyperparameters (HP). Thus, these hyperparameters must be tuned to adapt a
machine learning model to different problems and datasets. Selecting the best
hyperparameters for machine learning models directly impacts model
performance. It often requires a thorough understanding of machine learning
algorithms and appropriate hyperparameter optimization techniques. Although
there are several automatic optimization techniques, they have different
advantages and disadvantages. In contrast, parameters are internal to the
model. They are learned or estimated solely from the data during training
since the algorithm attempts to understand the mapping between input features
and target.

Model training usually starts with initializing the parameters to random
values. As training/learning progresses, the initial values are updated using
an optimization algorithm (e.g., gradient descent). The learning algorithm
continually updates the parameter values as training continues, but
hyperparameter values remain unchanged. At the end of the learning process,
the model parameters constitute the model itself. These steps inspired the
scientific community to develop a research field known as Hyperparameters
Optimization (HPO) Feurer and Hutter (2019); Bengio (2000). The primary aim of
this field is to automate the hyperparameter tuning process and enable users
to apply machine learning models to practical problems (efficiently, reducing
computation time and improving performance).

In our previous works Atchade-Adelomou (2022); Gonzalez-Bermejo  _et al._
(2022); Atchade-Adelomou  _et al._ (2021b), we proposed optimization
algorithms that can be used to solve HPO problems with continuous functions,
discrete functions, categorical variables, convex or non-convex functions,
etc. Next, we review some of them to highlight their limitations to find
solutions.

Grid search (GS) Lerman (1980); Liashchynskyi and Liashchynskyi (2019);
LaValle  _et al._ (2004) is a decision-theoretic approach Ferguson (2014) that
exhaustively searches for a fixed domain of hyperparameter values. GS is one
of the most used strategies due to its simplicity of implementation. This
algorithm discretizes the search space to generate a set of possible
hyperparameter configurations. Then, it evaluates each of these configurations
and selects the one with the highest performance. GS’s main limitation is that
it takes time and is affected by the dimensionality curse Liashchynskyi and
Liashchynskyi (2019). Therefore, it is not suitable for a large number of
hyperparameters. Moreover, GS often needs help finding the global optimum of
continuous parameters because it requires a predefined and finite set of
hyperparameter values. It is also unrealistic for GS to identify continuous
integer hyperparameter optima with limited time and resources. Therefore,
compared to other techniques, GS is only effective for a small number of
categorical hyperparameters Liashchynskyi and Liashchynskyi (2019).

Random search (RS) Solis and Wets (1981); Liashchynskyi and Liashchynskyi
(2019); Bergstra and Bengio (2012) is a variant of Grid Search and attempts to
solve the above problem by randomly sampling configurations of the search
space. As it does not have an implicit end condition, the number of sampled
structures to be evaluated will be chosen. RS samples the search space and
evaluates sets from specified probability distributions. In short, it is a
technique in which the hyperparameters’ random combinations are used to find
the best solution for the model under consideration. RS is more efficient than
GS and supports all domains of hyperparameters. In practical applications,
using RS to estimate randomly chosen hyperparameter values helps analysts
explore an ample search space. However, since RS does not consider the results
of previous tests, it may include many unnecessary evaluations that reduce its
performance.

Hyperband Hyp  is considered an improved version of Random Search Yang and
Shami (2020). Hyperband balances model performance and resource usage to be
more efficient than RS, especially with limited time and resources
Eggensperger  _et al._ (2013). However, GS, RS, and Hyperband have a
significant limitation: they treat each hyperparameter independently and do
not consider hyperparameter correlations Wang  _et al._ (2018). Therefore,
they will be inefficient for ML algorithms with conditional hyperparameters,
such as Support Vector Machine (SVM) Hearst  _et al._ (1998), Density-Based
Spatial Clustering of Applications with Noise (DBSCAN) Birant and Kut (2007);
Zhou  _et al._ (2012), and logistic regression Wright (1995); Kleinbaum  _et
al._ (2002).

Gradient-based algorithms Williams and Peng (1990); Cesa-Bianchi (1997) are
not a predominant choice for hyperparameter optimization because they only
support continuous hyperparameters and can only find a local, not global,
optimum for non-convex HPO problems. Therefore, gradient-based algorithms can
only optimize specific hyperparameters, such as the learning rate in Deep
Learning (DL) models Koutsoukas  _et al._ (2017).

Based on their surrogate models, the Bayesian optimization (BO) Shahriari  _et
al._ (2015) models, BO based on Gaussian Process (GP) Dudley (2010); Schulz
_et al._ (2018) and its derivatives, are divided into three different models.
BO algorithms determine the next hyperparameter value based on previously
evaluated results to reduce unnecessary evaluations and improve efficiency.
BO-GP mainly supports continuous and discrete hyperparameters but does not
support conditional hyperparameters Eggensperger  _et al._ (2013). At the same
time, Sequential Model Algorithm Configuration (SMAC) Eggensperger  _et al._
(2013) and Tree-Structured Parzen Estimator (BO-TPE) Zhao and Li (2018) can
handle categorical, discrete, continuous, and dependent hyperparameters. SMAC
performs best using many categorical and conditional parameters or cross-
validation, while BO-GP performs best with only a few continuous parameters.
BO-TPE preserves some dependent relationships, so one of its advantages over
BO-GP is its native support for some conditional hyperparameters Eggensperger
_et al._ (2013).

The Metaheuristic algorithms Yang (2010), including Genetic Algorithm (GA)
Lessmann  _et al._ (2005) and Particle Swarm Algorithm (PSO) Lorenzo  _et al._
(2017), are more complex than other HPO algorithms but they often work well
for complex optimization problems. They support all hyperparameters and are
particularly efficient for large configuration spaces because they can obtain
near-optimal solutions in several iterations. However, GA and PSO have their
advantages and disadvantages in practice. The main advantage of PSO is that it
can support large-scale parallelization and is exceptionally suitable for
continuous and conditional HPO problems. At the same time, GA runs
sequentially, which makes parallelization difficult. Thus, PSO often runs
faster than GA, especially for large configuration spaces and data sets.
However, good population initialization is essential for PSO; otherwise, it
may converge slowly and only identify a local optimum rather than a global
one. Regardless, the impact of a good population initialization is less
significant for GA than for PSO Lorenzo  _et al._ (2017). Another limitation
of GA is that it introduces additional hyperparameters, such as its population
size and mutation rate Lessmann  _et al._ (2005).

Quantum computing Steane (1998); Gruska  _et al._ (1999); Atchade-Adelomou
(2022) is a field of computation that uses quantum theory principles. A
quantum computer is a stochastic machine that uses the laws of quantum
mechanics to do computation. Due to the characteristics of HPO problems,
quantum computing is an excellent ally to seek a paradigm shift and accelerate
or find an efficient strategy to apply to them. There are focus on using
quantum computing in optimizing hyperparameters.

In Lentzas  _et al._ (2019), the authors employed a quantum genetic algorithm
to address the hyperparameter optimization problem. The algorithm is based on
qudits instead of qubits, allowing more available states. Experiments were
performed on two MNIST and CIFAR10 datasets, and results were compared against
classic genetic algorithms.

In Sagingalieva  _et al._ (2022), the authors presented a quantum-inspired
hyperparameter optimization technique and a hybrid quantum-classical machine
learning model for supervised learning. They compared their hyperparameter
optimization method to standard black box objective functions. They observed
performance improvements in the form of reduced expected execution times and
suitability in response to growth in the search space size. They tested their
approaches in a car image classification task and demonstrated a large-scale
implementation of the hybrid quantum neural network model with tensor train
hyperparameter optimization. Their tests showed a qualitative and quantitative
advantage over the corresponding standard classical tabular grid search
approach used with a ResNet34 deep neural network. The hybrid model achieved a
classification accuracy of 0.97 after 18 iterations, while the classical model
achieved an accuracy of 0.92 after 75 iterations. This last work had an
exciting approach that only contemplates discrete hyperparameters.

We have also found some exciting work dealing with HPO Li  _et al._ (2017);
Moussa  _et al._ (2022); Lentzas  _et al._ (2019); Gómez  _et al._ (2022). In
the latter Gómez  _et al._ (2022), the authors took the first steps toward
Automated Quantum Machine Learning (AutoQML). They proposed a concrete problem
description and then developed a classical-quantum hybrid cloud architecture
that allows for parallelized hyperparameter exploration and model training. As
an application use-case, they train a quantum Generative Adversarial neural
Network (qGAN) to generate energy prices that follow a known historic data
distribution. Such a QML model can be used for various applications in the
energy economics sector.

The SWOT of the hyperparameter optimization is summarized in Table 1. After
exploring the state of the art of classical and quantum hyperparameter tuning,
we have yet to find a generic model that solves the domain’s types of
hyperparameters and reduces the search time for said hyperparameters in this
quantum era.

Table 1: The Benchmark of the standard HPO algorithms ($n$ is the number of hyperparameter values and $k$ is the number of hyperparameters) HPO Methods | Strengths | Limitations | Time Complexity
---|---|---|---
GS | simple | Time-consuming | $O(n^{k})$
Only efficient with categorical
RS | More efficient than GS | Not consider previous results. | $O(n)$
Enable paralellization | Not efficient with conditional
Gradient-based | Fast convergence for continuous HPs. | Only support continuous HPs. | $O(n^{k})$
May only detect local optimums.
BO-GP | Fast convergence for continuous HPs. | Poor capacity for parallelization. | $O(n^{3})$
Not efficient with conditional HPs.
Hyperband | Enable paralelliization | Not efficient with conditional HPs. | $O(nlogn)$
Require subsets with small budgets |
GA | Efficient with all types of HPs | Poor capacity for parallelization. | $O(n^{2})$
Not require good initialization
PSO | Efficient with all types of HPs | Require proper initialization. | $O(nlogn)$
Enable paralellization

##  IV Quantum Machine Learning

Quantum machine learning (QML) Schuld  _et al._ (2015); Biamonte  _et al._
(2017a); Schuld and Petruccione (2018) explores the interplay and takes
advantage of quantum computing and machine learning ideas and techniques.

Therefore, quantum machine learning is a hybrid system involving both
classical and quantum processing, where computationally complex subroutines
are given to quantum devices. QML tries to take advantage of the classical
machine learning does best and what it costs, such as distance calculation
(inner product), passing it onto a quantum computer that can compute it
natively in the Hilbert vector space. In this era of large classical data and
few qubits, the most common use is to design machine learning algorithms for
classical data analysis running on a quantum computer, i.e., quantum-enhanced
machine learning Dunjko  _et al._ (2016); Atchade Adelomou  _et al._ (2020);
Atchade-Adelomou and Alonso-Linaje (2022); Adelomou  _et al._ (2022, 2020);
Guillermo (2023); Schuld and Petruccione (2018).

Quantum circuits are mathematically defined as operations on an initial
quantum state. Quantum computing generally makes use of quantum states built
from qubits, that is, binary states represented as
$\ket{\psi}=\alpha\ket{0}+\beta\ket{1}$. Their number of qubits $n$ commonly
defines the states of a quantum circuit, and, in general, the circuit’s
initial state $\ket{\psi}_{0}$ is the zero state $\ket{0}$. In general, a
quantum circuit implements an internal unit operation $U$ to the initial state
$\ket{\psi}_{0}$ to transform it into the final output state $\ket{\psi}_{f}$.
This gate $U$ is wholly fixed and known for some algorithms or problems. In
contrast, others define its internal functioning through a fixed structure,
called AnsatzArutyunov  _et al._ (2004) (Parametrized Quantum Circuit (PQC)),
and adjustable parameters $\theta$ Sim  _et al._ (2019). Parameterized
circuits are beneficial and have interesting properties in this quantum age
since they broadly define the definition of ML and provide flexibility and
feasibility of unit operations with arbitrary precision Biamonte  _et al._
(2017b); Pérez-Salinas  _et al._ (2020); Schuld  _et al._ (2014).

Figure (1) depicts the concept of hybrid computing (quantum + classical),
which characterizes the NISQ era. This takes advantage of quantum computing’s
capacity to solve complex problems and the experience of classical
optimization algorithms (COBYLAcommunity The SciPy (2021), SPSAC. (2001),
BFGSLiu and Nocedal (1989), etc.) to train variational circuits. Classical
algorithms are generally an iterative scheme that searches for better
candidates for the parameters $\theta$ at each step.

The value of the hybrid computing idea in the NISQ era is necessary because it
allows the scientific community to exploit both capacities and reaps the
benefits of the constant acceleration of the oncoming quantum-computer
development.

Furthermore, learning techniques can be improved by embedding information
(data) into the variational circuit through the quantum gate $U$ Schuld and
Petruccione (2018); Pérez-Salinas  _et al._ (2020).

The Variational Quantum Circuit (VQC) Nielsen  _et al._ (2002); Browne (2000)
consists of a quantum circuit that defines the base structure similar to
neural network architecture (Ansatz), while the variational procedure can
optimize the types of gates (one or two-qubit parametric gates) and their free
parameters. The usual supervised learning processes within quantum machine
learning can be defined as follows:

  * •

Quantum Feature Map: It is the data preparation. In the literature, this stage
is recognized as State preparation.

  * •

The Quantum model: It is the model creation. In the literature, it is
recognized as unitary evolution.

  * •

The classical error computation : It is the stage of Computing the error where
the model best approximates the input set; in machine learning, this stage is
known as the prediction.

Figure 1: Variational Quantum Algorithm (VQC) working principle based on the
quantum variational circuit. The quantum circuit computes the objective
function, and the classical computer computes the circuit parameters. We can
use this model to find the minima or the maxima of a given parameterized
function that is our quantum circuit. Our quantum circuit will be separated
into two large blocks: the Feature Map and the Variational Quantum Circuit
Cerezo  _et al._ (2021).

##  V IMPLEMENTATION

As aforementioned, our proposal is based on HPO.

Figure 2: This block diagram presents our approach of a generic hyperparameter
tuning with quantum computing-based gradient descent or Adam optimizer Kingma
and Ba (2014). The first block allows us to normalize all inputs to $0$ and
$\pi$. Then for the categorical variables, to save on the number of qubits, we
encode the categorical variables in binary and map them over the number of
qubits that our model has. The next block allows us to pack all the input
variables into a single vector. Each variable is equivalent to a dimension of
our vector. If the categorical variable is coded with $m$ dimensions, these
$m$ dimensions will be directly mapped to the vector. Having the input vector,
we now embed the data in the quantum computer thanks to our Feature Map
function. Said function will map our input variable continuously, with the
help of our parameterized gates, $RX$, $RY$, and $RZ$. From here, we apply our
quantum circuit. After the measurement, we must undo the previous operations
in reverse order to recover our input hyperparameters.

Mathematically, we can formulate it as follows: Since a model’s performance on
a validation set can be modeled as a function $f:X\rightarrow\mathbb{R}$ of
its hyperparameters $\vec{x}\in X$. With $X$ the hyperparameters’ space and
where $f$ can be any error function, such as the RMSE in a regression problem
or the AUC Score for a classification problem. The problem that the HPO must
solve is to find $\vec{x}$ such that $\vec{x}\in\text{argmin}_{\vec{x}\in
X}f(\vec{x})$.

Formally we can define our problem as follows. Let $f(\vec{x})$ be our
objective function with $\vec{x}$ as the vector of all the classical input
hyperparameters, and we are willing to find the best combination by writing it
down as:

$\vec{x}^{*}\approx\arg\min_{\vec{x}\in X}f(\vec{x}),$ (1)

In the optimization context, $f(x)$ is the objective function to be minimized,
where $\vec{x}^{*}$ is the hyperparameter configuration that produces the
optimum value of $f(\vec{x})$, and a hyperparameter $\vec{x}$ can take any
value in the search space $X$.

The first step of our algorithm will be to find the function $f(\vec{x})$ that
generalizes our data. So, let us define our quantum model as follows:

$f(\vec{x}):=\bra{0}U^{\dagger}(\vec{x},\beta,\vec{\theta})\sigma_{z}U(\vec{x},\beta,\vec{\theta})\ket{0},$
(2)

Where $\sigma_{z}$ is our observable, $U(\vec{x},\beta,\vec{\theta})$ the
parameterized circuit with the input data $\vec{x}$, $\beta$ our scaling
factor and $\vec{\theta}$, the parameterized variable.

Figure 3: We use this model based on Fourier series in quantum machine
learning Atchade-Adelomou and Larson (2023). Depending on some scaling $\beta$
parameters and the input data $\vec{x}$, the Feature Map will be in charge of
coding our data. We rely heavily on the fact that the Feature Map
($F(\vec{x},\beta)$) must be variational, that the data’s loading is repeated
in all the layers, and that the variational circuit ($V(\vec{\theta})$)
searches with the help of the input parameters for the best function within
the space of functions that defines the capacity of the variational circuit
($U(\vec{x},\beta,\vec{\theta})=F(\vec{x},\beta)V(\vec{\theta)}$). This
configuration will efficiently approximate our given dataset to a continuous
function $f(\vec{x})$.

From this point and considering equation (1), we will find the minimum of (2).

In the next step, according to figure (4), let $\beta$ and $\vec{\theta}$ be
the parameters that define our function $f(\vec{x})$ from equation (2). In
this stage, we consider these parameters as constant inputs (we do not vary
them). Now let $V_{\vec{\theta},\beta}(\vec{x})$ be the new circuit that can
be an instantiation of the previous circuit ($U(\vec{x},\beta,\vec{\theta})$),
and let us execute the gradient of the said quantum circuit ($f(\vec{x})$).
The outcome of these operations will yield the optimal value of $f(\vec{x})$,
and the arguments for this operation will be the best hyperparameters we look
for. The variable $\vec{x}$ will have the same dimension as the best
hyperparameters we are finding.

From the variational principle Ekeland (1974), the following equation $\langle
V_{\beta,\vec{\theta}}(\vec{x})\rangle_{\psi\left(\overrightarrow{\gamma}\right)}\geq\lambda_{i}$
can be reached. With $\lambda_{i}$ as eigenvector and $\langle
V_{\beta,\vec{\theta}}(\vec{x})\rangle_{\psi\left(\overrightarrow{\gamma}\right)}$
as the expected value. In this way, the VQC (Figure (1)) finds (3) such an
optimal choice of parameters $\overrightarrow{\gamma}$, that the expected
value is minimized and that a lower eigenvalue is located.

$\langle
V_{\vec{\theta},\beta}(\vec{x})\rangle=\langle\psi\left(\gamma\right)|V_{\vec{\theta},\beta}(\vec{x})|\psi\left(\gamma\right)\rangle$
(3)

Where $V_{\vec{\theta},\beta}(\vec{x})\approx f(\vec{x})$.

###  V.1 DATASET GENERATION

The proposed quantum hyperparameter process is summarized in three stages. The
first stage is the dataset generation, the second is finding the function that
represents quantum-based Fourier regression, and the third is finding the
minimum of the regression function. This section deals with dataset generation
and criteria—the following subsection details all the experiment steps. To
implement the proposed algorithm, we generate one dataset per machine learning
model regarding the original Vueling dataset considering its hyperparameters
evaluating different search methods, as well as Grid search, Random Search and
Bayes-Based Search.

We train the $N$ ML models with the Vueling dataset to obtain a new reduced
dataset. Three-way cross-validation is used during the training process.
Therefore, for each set of hyperparameters, the model is trained three times
and keeps the average of the three as the value to be predicted by the quantum
system.

If more precision is needed, the generated dataset should have more elements.
Instead of precision, whether the speed is prioritized at the expense of its
precision, the number of features of the dataset should be considerably less
than the number of features of the original dataset.

The resulting dataset size is reduced so as not to end up doing a quantum Grid
Search. The new databases are stored in dataframes using Pandas.

###  V.2 EXPERIMENTS STEPS

The scenario is given in Figure (4), and the proposed process to validate our
experimentation is summarized as follows:

  1. 1.

From the database that contains $252,183$ datapoints with $42$ features each,
we generate a set ($[n]$) of randomly chosen hyperparameters.

  2. 2.

We train $n$ models with these hyperparameters. Get $[n]$ scores (accuracy,
$r^{2}$, MSE, etc.) and record the dataset.

  3. 3.

We transform the categorical hyperparameters into linear variables according
to figure (2).

  4. 4.

We transform this dataset into a Quantum Space for the native quantum
algorithm.

  5. 5.

We find the best $U(\vec{x},\beta,\vec{\theta})$ according to figure (3).

  6. 6.

We train a Quantum-Coding System plus the best $U(\vec{x},\beta,\vec{\theta})$
found to the predict best hyperparameters.

  7. 7.

We get the best set of hyperparameters by using Quantum Computer.

  8. 8.

We go back to the classical model and re-train it with the best
hyperparameters.

Figure 4: This graph shows the steps followed to achieve our goal. First, we
generate a set ($[n]$) of randomly chosen hyperparameters and then train $n$
models with these hyperparameters. From the latter process, we get $[n]$
scores (accuracy, $r^{2}$, MSE, etc.) and record the dataset. Next, we
transform the categorical hyperparameter into linear variables according to
figure (2). From this point, we transform this dataset into a Quantum Space
for the native quantum algorithm. Having the dataset into the quantum domain,
we now find the best $U(\vec{x},\beta,\vec{\theta})$ according to to figure
(3). This means having our generic continuous function given by equation (2).
In this stage, we only need to find the minimum, according to equation (1).
So, we train a Quantum-Coding System plus the best
$U(\vec{x},\beta,\vec{\theta})$ found to predict the best hyperparameters.
Now, we get the best set of hyperparameters by using a quantum computer (or a
quantum-inspired one); we go back to the classical model and re-train it with
the best hyperparameters.

Classically there is a process $Classical\\_process$ with its executing time
$T_{c}$, and from quantum computing, there is a process $Quantum\\_process$
with its executing time $T_{q}(translation)+T_{q}(solution)$. We will only get
a time advantage if: $T_{c}>[T_{q}(translation)+T_{q}(solution)]$.

Why $T_{q}(translation)?$ because quantum computers only execute quantum data,
so we must translate all the data we got from the classical domain into the
quantum one.

To test the proper functioning of our algorithm, we design five cases ($A$,
$B$, $C$, $D$, and $E$), respectively, equivalent to the layer number of the
variational algorithm ($1$, $2$, $3$, $4$, and $5$).

###  V.3 OUR QUANTUM MODEL

Based on Atchade-Adelomou and Larson (2023), we propose the hybrid model and
strategy from figures (2), (3) and (4) to tackle the generic HPO and reduce
the search and analyzing time.

Table 2: This table represents the space’s configuration for the hyperparameters and classical models that were tested in this work. ML Model | Hyperparameter | Type | Search Space
---|---|---|---
Random Forest Classifier | n_estimators | Discrete | [5 - 250]
max_depth | Discrete | [1 - 50]
min_samples_split | Discrete | [1 - 10]
min_samples_leaf | Discrete | [1 - 10]
criterion | Categorical | [’gini’, ’entropy’, ’ROC’]
max_features | Discrete | [1 - 64]
SVM Classifier | C | Continuous | [0.1,50]
kernel | Categorical | [’linear’, ’poly’, ’rbf’, ’sigmoid’]
KNN Classifier | n_neighbors | Discrete | [1,20]
Random Forest Regressor | n_estimators | Discrete | [5 - 250]
max_depth | Discrete | [1 - 15]
min_samples_split | Discrete | [1 - 7]
min_samples_leaf | Discrete | [1 - 8]
criterion | Categorical | [’mse’, ’mae’ ’absolute_error’ ’friedman_mse’ ]
SVM Regressor | max_iter | Continuous | [1000 - 500000]
kernel | Categorical | [’linear’, ’poly’, ’rbf’, ’sigmoid’]
epsilon | Continuous | [0.0001 - 1]
HistGradientBoostingRegressor | max_iter | Discrete | [1 - 1000]
learning_rate | Discreto | [0.01 - 1]
max_bin | Discreto | [31 - 255]
| loss | Categorical | [ ’squared_error’ ’absolute_error’ ’poisson’ ’quantile’ ]
Ridge | max_iter | Discrete | [1000 - 500000]
solver | Categorical | [’svd’, ’cholesky’, ’lsqr’, ’sparse_cq’, ’sag’ ]
alpha | Continuous | [0.0001 - 1]
Decision Tree Regressor | max_depth | Discrete | [1 - 100]
solver | Categorical | [’mse’, ’mae’ ’absolute_error’ ’friedman_mse’ ’poisson’ ]
ccp_alpha | Continuous | [0.0 - 1]
|  |  |

##  VI Results

Figures (5) to (7) show the results of the process we follow to achieve our
goal. It can be observed in Figure (6) how the quantum model is molded into
the shape of the data, defining a mesh that represents the continuous
function, which is used to find the best hyperparameters.

We have tested the proposed algorithm and steps with various models and
techniques and generated several databases and experiments. The detailed
results of each experiment can be seen in the tables (3), (4), (5), and (6).
In addition, we have carried out some comparative studies that can be seen in
Figures (8) to (12).

Figure 5: We can observe the result of applying a quantum-based Fourier
regression approach. We can observe how the model allows interpolating the
data from the dataset better. Figure 6: We can see the space covered by the
new function $f(\vec{x})$ resulting from applying the gate
$U(\vec{x},\vec{\theta},\beta)$. In this case, the number of layers is $4$.
The axes on the graph represent hyperparameters from HistGradBoost model.
Here, the score is normalized on $\pi$. Please refer to table (2) for more
details. Figure 7: We can observe how our algorithm finds the local maximum of
our multivariate function. In this case, the number of layers is $5$. The axes
on the graph represent hyperparameters from HistGradBoost model. Please refer
to table (2) for more details.

###  VI.1 2 Hyperparameters

From the database that contains $252,183$ datapoints with $42$ features each,
we generate the database for the Histogram Gradient Boosting Regression. The
result of our algorithm by using Histogram Gradient Boosting Regression,
precisely a lightGBM implementation and executing the following tests will be
shown in three hyperparameters’ configurations ($2$, $3$, and $4$).

In the case of 2 Hyperparameters: learning_rate and max_iteration, we obtain
the following outcomes:

  * •

Testing classically with the full database with $240$ training samples took
$6$ minutes $6$ seconds, with $R^{2}$ result for the
$R^{2}_{\text{train}}=0.464$ and $R^{2}_{\text{test}}=0.388$.

  * •

By creating a subset with only $70$ samples, that took $1$ minute $38$ seconds
to be created and having the $R^{2}=0.382$.

  * •

Now, executing our quantum algorithm for finding the best hyperparameters
using the prepared dataset ($70$ samples) took only $4$ minutes and $43$
seconds to find the best hyperparameters with $R^{2}$ score $=0.444$.

###  VI.2 3 Hyperparameters

In the case of 3 Hyperparameters: learning_rate, max_iteration, and loss, we
obtain the following outcomes:

  * •

Testing classically with the full database with $480$ training samples took
$11$ minutes $36$ seconds, with $R^{2}$ result for the
$R^{2}_{\text{train}}=0.464$ and $R^{2}_{\text{test}}=0.388$.

  * •

By creating a subset with only $35$ samples, that took $53$ seconds to be
created and having the $R^{2}=0.382$.

  * •

Now, executing our quantum algorithm for finding the best hyperparameters
using the prepared dataset ($35$ samples) took only $1$ minute and $56$
seconds to find the best hyperparameters with $R^{2}$ score $=0.4377$.

Model | HistGradBoost with Random CV
---|---
Cases | A | B | C | D | E
Classical HistGradBoost performance time (s) | 2747.00 | 2747.00 | 2747.00 | 2747.00 | 2747.00
Total proposed model performance time (s) | 146.86 | 206.09 | 313.46 | 374.91 | 555.11
Time saving (s) | 2598.14 | 2538.91 | 2431.54 | 2370.09 | 2189.89
Time saving (%) | 94.65 | 92.49 | 89.59 | 86.35 | 79.79
Dev Score | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Dev Score (%) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
# HPs | 3.00 | 3.00 | 3.00 | 3.00 | 5.00
# Layers | 1.00 | 2.00 | 3.00 | 4.00 | 5.00
Load Data(ms) | 1.30 | 2.40 | 4.10 | 4.81 | 0.00
VQA (s) | 84.70 | 150.05 | 239.06 | 282.38 | 397.03
Finding Quantum best HPs (s) | 28.63 | 44.63 | 68.01 | 85.49 | 140.42
Data mapping from Quantum to Classic Space ($\mu$s) | 30.35 | 41.76 | 51.51 | 70.66 | 73.78
Loading Original dataset time (s) | 1.44 | 1.07 | 1.10 | 1.07 | 1.09
Model training (s) | 32.08 | 10.34 | 5.29 | 5.97 | 16.56
Proposed model Test score | 0.3879 | 0.39 | 0.3879 | 0.3879 | 0.3879
Orginal Train score | 0.4578 | 0.4598 | 0.4578 | 0.4578 | 0.4598
Orginal Test Score | 0.3879 | 0.3879 | 0.3879 | 0.3879 | 0.3879
|  |  |  |  |
Table 3:  This table shows the experiments for five cases (A, B, C, D, and E)
with the input database trained with both the test and train for the
HistGradBoost model. The time required to generate said database classically
and have the optimal hyperparameters applying Random Search with Cross-
Validation is $2747$ seconds. The configuration parameters of our hybrid model
to obtain the data from the tables are the following: $\textit{lrVQA}=0.15$,
$\textit{maxEpochVQA}=70$, $\textit{lrBH}=0.0005$, $\textit{maxEpochBH}=1500$,
$loadOptBH=False$ and for the different quantum layers (qLayer$=1,2,3,4,$ and
$5$). Our algorithm already finds the hyperparameters for the defined target
score, considerably reducing the experimentation time using minimum layers. In
this case, from $94\%$ for one layer to $79\%$ for five layers. All the tests
were done locally on Mac Book Pro with 8-Core Intel Apple (2022 (accessed Dec
26, 2022).

###  VI.3 4 Hyperparameters

In the case of 4 Hyperparameters: learning_rate, max_iteration, loss and
max_bins, we have the following outcomes:

  * •

Testing classically with the full database with $1980$ training samples took
$41$ minutes $45$ seconds, with $R^{2}$ result for the
$R^{2}_{\text{train}}=0.464$ and $R^{2}_{\text{test}}=0.388$.

  * •

By creating a subset with only $280$ samples, that took $5$ minutes $54$
seconds to be created and having the $R^{2}=0.388$.

  * •

Now, executing our quantum algorithm for finding the best hyperparameters
using the prepared dataset ($280$ samples) took only $15$ minutes and $2$
seconds to find the best hyperparameters with $R^{2}$ score $=0.415$.

Model | HistGradBoost with Grid CV
---|---
Cases | A | B | C | D | E
Classical HistGradBoost performance time (s) | 2505.00 | 2505.00 | 2505.00 | 2505.00 | 2505.00
Total proposed model performance time (s) | 368.30 | 774.53 | 944.76 | 1470.23 | 1757.00
Time saving (s) | 2136.69 | 1730.46 | 1560.24 | 1034.76 | 747.9930
Time saving (%) | 85.29 | 69.08 | 62.28 | 41.31 | 29.86
Dev Score | 0.01 | 0.01 | 0.01 | 0.01 | 0.01
Dev Score (%) | 0.01 | 0.01 | 0.01 | 0.01 | 0.01
# HPs | 4.00 | 4.00 | 4.00 | 4.00 | 4.00
# Layers | 1.00 | 2.00 | 3.00 | 4.00 | 5.00
Load Data(s) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
VQA (s) | 308.31 | 700.37 | 855.79 | 1302.36 | 1585.81
Finding Quantum best HPs (s) | 45.68 | 65.80 | 81.75 | 146.56 | 160.69
Data mapping from Quantum to Classic Space (s) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Loading Original dataset time (s) | 1.60 | 1.11 | 1.00 | 1.60 | 1.23
Model training (s) | 12.72 | 7.25 | 6.21 | 19.72 | 9.27
Proposed model Test score | 0.3785 | 0.3785 | 0.3785 | 0.3785 | 0.3785
Original Train score | 0.4583 | 0.4583 | 0.4583 | 0.4583 | 0.4583
Original Test Score | 0.3879 | 0.3879 | 0.3879 | 0.3879 | 0.3879
|  |  |  |  |
Table 4: This table shows the experiments for five cases (A, B, C, D, and E) with the input database trained with both the test and train for the HistGradBoost model. The time required to generate said database classically and have the optimal hyperparameters applying Grid Search with Cross-Validation is $2505$ seconds. The configuration parameters of our hybrid model to obtain the data from the tables are the following: $\textit{lrVQA}=0.15$, $\textit{maxEpochVQA}=70$, $\textit{lrBH}=0.0005$, $\textit{maxEpochBH}=1500$, $loadOptBH=False$ and for the different quantum layers (qLayer$=1,2,3,4,$ and $5$). Our algorithm already finds the hyperparameters for the defined target score, considerably reducing the experimentation time using minimum layers. In this case, from $85\%$ for one layer to $29\%$ for five layers. All the tests were done locally on MacBookPro with 8-Core Intel Apple (2022 (accessed Dec 26, 2022).  Model | RandomForest with Grid CV
---|---
Cases | A | B | C | D | E
Classical Random Forest performance time (s) | 2116.00 | 2116.00 | 2116.00 | 2116.00 | 2116.00
Total proposed model performance time (s) | 669.22 | 1018.24 | 932.06 | 955.51 | 1021.88
Time saving (s) | 1446.77 | 1097.76 | 1183.93 | 1160.48 | 1094.12
Time saving (%) | 68.37 | 51.88 | 55.95 | 54.84 | 51.71
Dev Score | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Dev Score (%) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
# HPs | 2.00 | 2.00 | 2.00 | 2.00 | 2.00
# Layers | 1.00 | 2.00 | 3.00 | 4.00 | 5.00
Load Data(s) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
VQA (s) | 80.04 | 101.80 | 148.99 | 218.43 | 245.46
Finding Quantum best HPs (s) | 23.35 | 37.88 | 53.02 | 70.19 | 76.26
Data mapping from Quantum to Classic Space (s) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Loading Original dataset time (s) | 1.29 | 1.49 | 1.18 | 1.30 | 1.02
Model training (s) | 564.55 | 877.07 | 728.88 | 665.60 | 699.14
Proposed model Test score | 0.3478 | 3478 | 0.3478 | 0.3478 | 0.3478
Original Train score | 0.6273 | 0.6273 | 0.6273 | 0.6273 | 0.6273
Original Test Score | 0.3478 | 0.3478 | 0.3478 | 0.3478 | 0.3478
|  |  |  |  |
Table 5: This table shows the experiments for five cases (A, B, C, D, and E) with the input database trained with both the test and train for the RandomForest model. The time required to generate said database classically and have the optimal hyperparameters applying Grid Search with Cross-Validation is $2116$ seconds. The configuration parameters of our hybrid model to obtain the data from the tables are the following: $\textit{lrVQA}=0.15$, $\textit{maxEpochVQA}=70$, $\textit{lrBH}=0.0005$, $\textit{maxEpochBH}=1500$, $loadOptBH=False$ and for the different quantum layers (qLayer$=1,2,3,4,$ and $5$). Our algorithm already finds the hyperparameters for the defined target score, considerably reducing the experimentation time using minimum layers. In this case, from $68\%$ for one layer to $51\%$ for five layers. All the tests were done locally on MacBookPro with 8-Core Intel Apple (2022 (accessed Dec 26, 2022).  Model | Ridge with Grid Search CV
---|---
Cases | A | B | C | D | E
Classical Ridge performance time (s) | 2075.00 | 2075.00 | 2075.00 | 2075.00 | 2075.00
Total proposed model time (s) | 163.13 | 314.93 | 538.35 | 632.64 | 871.59
Time saving (s) | 1911.87 | 1760.07 | 1536.65 | 1442.36 | 1203.41
Time saving (%) | 92.13 | 84.82 | 74.05 | 69.51 | 57.99
Dev Score | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Dev Score (%) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
# HPs | 5.00 | 5.00 | 5.00 | 5.00 | 5.00
# Layers | 1.00 | 2.00 | 3.00 | 4.00 | 5.00
Load Data(s) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
VQA (s) | 118.77 | 243.11 | 403.12 | 490.68 | 678.17
Finding Quantum best HPs (s) | 39.87 | 67.79 | 130.66 | 137.81 | 188.86
Data mapping from Quantum to Classic Space (s) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Loading Original dataset time (s) | 1.15 | 0.98 | 1.10 | 1.07 | 1.13
Model training (s) | 3.34 | 3.04 | 3.46 | 3.07 | 3.44
Proposed model Test score | 0.3012 | 0.3012 | 0.3012 | 0.3012 | 0.3012
Original Train score | 0.3134 | 0.3134 | 0.3134 | 0.3134 | 0.3134
Original Test Score | 0.3012 | 0.3012 | 0.3012 | 0.3012 | 0.3012
|  |  |  |  |
Table 6:  This table shows the experiments for five cases (A, B, C, D, and E)
with the input database trained with both the test and train for the Ridge
model. The time required to generate said database classically and have the
optimal hyperparameters applying Grid Search with Cross-Validation is $2116$
seconds. The configuration parameters of our hybrid model to obtain the data
from the tables are the following: $\textit{lrVQA}=0.15$,
$\textit{maxEpochVQA}=70$, $\textit{lrBH}=0.0005$, $\textit{maxEpochBH}=1500$,
$loadOptBH=False$ and for the different quantum layers (qLayer$=1,2,3,4,$ and
$5$). Our algorithm already finds the hyperparameters for the defined target
score, considerably reducing the experimentation time using minimum layers. In
this case, from $92\%$ for one layer to $57\%$ for five layers. All the tests
were done locally on MacBookPro with 8-Core Intel Apple (2022 (accessed Dec
26, 2022).

###  VI.4 Discussions

In this Section, we discuss the obtained results while elaborating on which
cases our proposed model works best.

We compared the classical and hybrid proposed models from figure (8).
Specifically, we compared the classically trained data model with the
HistGradBoost model using Random Search and Cross-Validation with our model.
The ratio between the graphs is from saving time, with only a single layer in
the quantum circuit. In the case of figure (8), we notice that we saved $56\%$
of total time. The VQA took about $100$ seconds, the search algorithm for the
optimal hyperparameters took about $33$ seconds, and the model retraining took
about $50$ seconds compared to $2747$ seconds to get the same hyperparameters.
Using Random search, the experiment was done with a $1500$-element database
over $3$ columns. With the hyperparameters param_learning_rate $[0.01-1]$,
param_iter $[1-1000]$, and param_loss as categorical $[$squared_error,
absolute_error$]$ (for more details (2)).

Figure (9) shows the experiment with the same model (HisGradBoost) but this
time with Grid Search and Cross-Validation. In this case, the VQA took about
$425$ seconds, the search algorithm for the optimal hyperparameters took about
$36$ seconds, and the model retraining took about $8$ seconds compared to
$2505$ seconds to get the same parameters. Although the other times are
negligible compared to the magnitudes we are dealing with, we have wanted to
plot them on this graph ((9)). The experiment was done with a $1920$-element
database over $4$ columns. With the hyperparameters param_learning_rate
$[0.01-1]$, param_iter $[1-1000]$, param_max_bins $[31-255]$ and param_loss as
categorical $[$squared_error, absolute_error$]$ (for more details (2)).

The results of the experiments with the RandomForest model are shown by the
figure (10). The experiment compares the classically trained data model with
the RandomForest model using Cross-Validation. The time saving is $63\%$. For
that, the VQA took about $121$ seconds, the search algorithm for the optimal
hyperparameters took about $68$ seconds, and the model retraining took about
$600$ seconds compared to $2116$ seconds to get the same parameters. The
experiment was done with an $88$-element database with over $2$ columns. With
the hyperparameters param_n_estimators $[5-250]$ and param_max_depth $[1-15]$
(for more details (2)).

The experimentation with Ridge model is presented in Figure (11), which
compares the classically trained data model with the Ridge model using Grid
Search and Cross-Validation with our model. The ratio between the graphs
represents the savings that we achieve. Here we are saving $73\%$ of the
experimentation time. In this case, the VQA took about $178$ seconds, the
search algorithm for the optimal hyperparameters took about $53$ seconds, and
the model retraining took about $5$ seconds compared to $2075$ seconds to get
the same parameters. The experiment was done with a $560$-element database
with over $3$ columns. With the hyperparameters param_alpha $[0.0001-1]$,
param_max_iter $[1000-500000]$ and param_solver as categorical $[\text{svd,
cholesky, lsqr, sparse\\_cq, sag }]$ (for more details (2)). We have realized
that for databases classically generated for less than one minute, it is not
worth using our algorithm since the setup time in the case of a single
hyperparameter is 50 seconds, and for two hyperparameters is one minute and 20
seconds.

Figure 8: This graph compares the classical model and the hybrid proposed
model—the classic model on the right of the image and the proposed model on
the left. We are comparing the classically trained data model with the
HistGradBoost model using Random Search with Cross-Validation. The ratio
between the graphs results from saving time in this case, with only a single
layer in the quantum circuit. Table (3) analyzes in detail considering the
number of layers. In this case, we save $56\%$ of time compared to the
classical counterpart. A $1500$-element database with $3$ columns considering
the following the hyperparameters param_learning_rate $[0.01-1]$, param_iter
$[1-1000]$, and param_loss as categorical $[$squared_error, absolute_error$]$
(for more details (2)). Figure 9: This graph compares the classical model and
our hybrid proposal—the classic model on the right of the image and the
proposed model on the left. Specifically, we are comparing the classically
trained data model with the HisGradBoost model using Grid Search and Cross-
Validation with the model. The ratio between the graphs results from saving
time in this case, with only a single layer in the quantum circuit. Table (4)
analyzes in detail considering the number of layers. In this case, we save
$81\%$ of time compared to the classical counterpart. A $1920$-element
database over $4$ columns considering the following hyperparameters
param_learning_rate $[0.01-1]$, param_iter $[1-1000]$, param_max_bins
$[31-255]$ and param_loss as categorical $[$squared_error, absolute_error$]$
(for more details (2)). Figure 10: This graph compares the classical model and
the hybrid proposed model—the classic model on the right of the image and the
proposed model on the left. We are comparing the classically trained data
model with the RandomForest model using Grid Search and Cross-Validation with
our model. The ratio between the graphs results from saving time in this case,
with only a single layer in the quantum circuit. Table (5) analyzes in detail
considering the number of layers. In this case, we save $63\%$ of time
compared to the classical counterpart. An $88$-element database with $2$
columns considering the following hyperparameters param_n_estimators $[5-250]$
and param_max_depth $[1-15]$ (for more details (2)). Figure 11: This graph
compares the classical model and the hybrid proposed model—the classic model
on the right of the image and the proposed model on the left. We are comparing
the classically trained data model with the Ridge model using Grid Search and
Cross-Validation with the model. The ratio between the graphs results from
saving time in this case, with only a single layer in the quantum circuit.
Table (6) analyzes in detail considering the number of layers. In this case,
we can see that we save $73\%$ of time compared to the classical counterpart.
A $560$-element database with $3$ columns considering the following
hyperparameters param_alpha $[0.0001-1]$, param_max_iter $[1000-500000]$ and
param_solver as categorical $[\text{svd, cholesky, lsqr, sparse\\_cq, sag }]$
(for more details (2)).

Moreover, to verify the proper functioning of our algorithm and how it shares
part of its philosophy with BayesSearch, we wanted to compare it with
BayesSearch with cross-validation. As quantum computing is today, the result
is quite comparable. The comparison has been simulated on a classical
computer. That is, as an example, having the limitations of RAM without the
ability to take advantage of quantum parallelization. Our algorithm has been
equated by $89\%$ time to the classicalBayesSearch. Still, we have also
experienced cases where BayesSearch has exceeded our algorithm by about 100
seconds as the highest observed value. In all cases, our algorithm has found
the values of the same hyperparameters as BayesSearch. We only compared the
algorithm with a single layer. It can be seen in figure (12) the operation at
the time and process level of the proposed algorithm. Note how the quantum
part goes very fast, and we consume the most time in the retraining of Bayes
Search. Since the Bayes Search with Cross-Validation goes back to performing
some steps already contemplated in our proposal.

Figure 12: This graph presents the performance of the proposed algorithm
compared to its classical counterpart using the BayesSearch with Cross-
Validation. In this case, the proposed algorithm represented 92.5% of the
classical model as a reference which means a 7.5% time improvement. A
$30$-element database with $2$ columns were used, considering the following
hyperparameters param_n_estimators $[5-250]$ and param_max_depth $[1-15]$.

##  VII Conclusion

Hyperparameter tuning is a research area with great interest in this big-data
era. In this article, we have studied using classical and quantum hybrid
algorithms to offer a generic solution. We have investigated several scenarios
and experiments to propose an efficient model for solving the hyperparameter
optimization problem. We have designed an algorithm that fits the current
status of quantum computing. Our algorithm and processes can be used in all
quantum-inspired machines solving real cases in society, waiting for a quantum
computer or a system that allows us to have quality service to run it on a
commercial quantum computer. Our algorithm has proven robust in all tested
scenarios and has given outstanding results. For this reason, we firmly
believe that, beyond the hardware limitations and beyond achieving an
efficient qRAM, if there is a lot of classical data and few functional qubits,
algorithms such as the one proposed in this work are the most suitable for the
real solutions today.

## Acknowledgements

The authors want to thank Guillermo Alonso de Linaje for the discussions and
consideration during the experiments. Also, the authors wish to thank the
Vueling Airlines SA for allowing the use of their proprietary dataset to
perform this work.

Compliance with Ethics Guidelines
Funding: This research received no external funding.

Institutional review: This article does not contain any studies with human or
animal subjects.

Informed consent: Informed consent was obtained from all individual
participants included in the study.

Data availability: Data sharing is not applicable. No new data were created or
analyzed in this study. Data sharing does not apply to this article.

## References

  * Choy  _et al._ (2018) Garry Choy, Omid Khalilzadeh, Mark Michalski, Synho Do, Anthony E Samir, Oleg S Pianykh, J Raymond Geis, Pari V Pandharipande, James A Brink, and Keith J Dreyer, “Current applications and future impact of machine learning in radiology,” Radiology 288, 318–328 (2018).
  * Callahan and Shah (2017) Alison Callahan and Nigam H Shah, “Machine learning in healthcare,” in  _Key Advances in Clinical Informatics_ (Elsevier, 2017) pp. 279–291.
  * Anderson  _et al._ (2022) David Anderson, Margret V Bjarnadottir, and Zlatana Nenova, “Machine learning in healthcare: Operational and financial impact,” Innovative Technology at the Interface of Finance and Operations: Volume I , 153–174 (2022).
  * Pallathadka  _et al._ (2021) Harikumar Pallathadka, Malik Mustafa, Domenic T Sanchez, Guna Sekhar Sajja, Sanjeev Gour, and Mohd Naved, “Impact of machine learning on management, healthcare and agriculture,” Materials Today: Proceedings (2021).
  * Miklosik and Evans (2020) Andrej Miklosik and Nina Evans, “Impact of big data and machine learning on digital transformation in marketing: A literature review,” Ieee Access 8, 101284–101292 (2020).
  * Mosavi  _et al._ (2019) Amir Mosavi, Mohsen Salimi, Sina Faizollahzadeh Ardabili, Timon Rabczuk, Shahaboddin Shamshirband, and Annamaria R Varkonyi-Koczy, “State of the art of machine learning models in energy systems, a systematic review,” Energies 12, 1301 (2019).
  * Atchade-Adelomou  _et al._ (2021a) Parfait Atchade-Adelomou, Guillermo Alonso-Linaje, Jordi Albo-Canals, and Daniel Casado-Fauli, “qrobot: A quantum computing approach in mobile robot order picking and batching problem solver optimization,” Algorithms 14 (2021a), 10.3390/a14070194.
  * Bardenet  _et al._ (2013) Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michele Sebag, “Collaborative hyperparameter tuning,” in  _International conference on machine learning_ (PMLR, 2013) pp. 199–207.
  * Yang and Shami (2020) Li Yang and Abdallah Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing 415, 295–316 (2020).
  * Li  _et al._ (2017) Yangyang Li, Gao Lu, Linhao Zhou, and Licheng Jiao, “Quantum inspired high dimensional hyperparameter optimization of machine learning model,” in  _2017 International Smart Cities Conference (ISC2)_ (2017) pp. 1–6.
  * Sagingalieva  _et al._ (2022) Asel Sagingalieva, Andrii Kurkin, Artem Melnikov, Daniil Kuhmistrov, Michael Perelshtein, Alexey Melnikov, Andrea Skolik, and David Von Dollen, “Hyperparameter optimization of hybrid quantum neural networks for car classification,”  (2022).
  * Bengio (2000) Yoshua Bengio, “Gradient-based optimization of hyperparameters,” Neural computation 12, 1889–1900 (2000).
  * Wang  _et al._ (2018) Jiazhuo Wang, Jason Xu, and Xuejun Wang, “Combination of hyperband and bayesian optimization for hyperparameter optimization in deep learning,” arXiv preprint arXiv:1801.01596 (2018).
  * Feurer and Hutter (2019) Matthias Feurer and Frank Hutter, “Hyperparameter optimization,” in  _Automated machine learning_ (Springer, Cham, 2019) pp. 3–33.
  * Atchade-Adelomou and Larson (2023) Parfait Atchade-Adelomou and Kent Larson, “Fourier series weight in quantum machine learning,” arXiv Pending (2023).
  * Zeldam (2018) SG Zeldam,  _Automated failure diagnosis in aviation maintenance using explainable artificial intelligence (XAI)_ , Master’s thesis, University of Twente (2018).
  * Shukla  _et al._ (2019) Naman Shukla, Arinbjörn Kolbeinsson, Ken Otwell, Lavanya Marla, and Kartik Yellepeddi, “Dynamic pricing for airline ancillaries with customer context,” in  _Proceedings of the 25th ACM SIGKDD International Conference on knowledge discovery & data mining_ (2019) pp. 2174–2182.
  * Cacchiani and Salazar-González (2020) Valentina Cacchiani and Juan-José Salazar-González, “Heuristic approaches for flight retiming in an integrated airline scheduling problem of a regional carrier,” Omega 91, 102028 (2020).
  * Parmentier and Meunier (2020) Axel Parmentier and Frédéric Meunier, “Aircraft routing and crew pairing: Updated algorithms at air france,” Omega 93, 102073 (2020).
  * Atchade-Adelomou (2022) Parfait Atchade-Adelomou, “Quantum algorithms for solving hard constrained optimisation problems,”  (2022).
  * Gonzalez-Bermejo  _et al._ (2022) Saul Gonzalez-Bermejo, Guillermo Alonso-Linaje, and Parfait Atchade-Adelomou, “Gps: A new tsp formulation for its generalizations type qubo,” Mathematics 10, 416 (2022).
  * Atchade-Adelomou  _et al._ (2021b) Parfait Atchade-Adelomou, Daniel Casado-Fauli, Elisabet Golobardes-Ribe, and Xavier Vilasis-Cardona, “quantum case-based reasoning (qcbr),” (2021b), arXiv:2104.00409 [cs.AI] .
  * Lerman (1980) PM Lerman, “Fitting segmented regression models by grid search,” Journal of the Royal Statistical Society: Series C (Applied Statistics) 29, 77–84 (1980).
  * Liashchynskyi and Liashchynskyi (2019) Petro Liashchynskyi and Pavlo Liashchynskyi, “Grid search, random search, genetic algorithm: a big comparison for nas,” arXiv preprint arXiv:1912.06059 (2019).
  * LaValle  _et al._ (2004) Steven M LaValle, Michael S Branicky, and Stephen R Lindemann, “On the relationship between classical grid search and probabilistic roadmaps,” The International Journal of Robotics Research 23, 673–692 (2004).
  * Ferguson (2014) Thomas S Ferguson,  _Mathematical statistics: A decision theoretic approach_ , Vol. 1 (Academic press, 2014).
  * Solis and Wets (1981) Francisco J Solis and Roger J-B Wets, “Minimization by random search techniques,” Mathematics of operations research 6, 19–30 (1981).
  * Bergstra and Bengio (2012) James Bergstra and Yoshua Bengio, “Random search for hyper-parameter optimization.” Journal of machine learning research 13 (2012).
  * (29) .
  * Eggensperger  _et al._ (2013) Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger Hoos, Kevin Leyton-Brown, _et al._ , “Towards an empirical foundation for assessing bayesian optimization of hyperparameters,” in  _NIPS workshop on Bayesian Optimization in Theory and Practice_ , Vol. 10 (2013).
  * Hearst  _et al._ (1998) Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf, “Support vector machines,” IEEE Intelligent Systems and their applications 13, 18–28 (1998).
  * Birant and Kut (2007) Derya Birant and Alp Kut, “St-dbscan: An algorithm for clustering spatial–temporal data,” Data & knowledge engineering 60, 208–221 (2007).
  * Zhou  _et al._ (2012) Hongfang Zhou, Peng Wang, and Hongyan Li, “Research on adaptive parameters determination in dbscan algorithm,” Journal of Information & Computational Science 9, 1967–1973 (2012).
  * Wright (1995) Raymond E Wright, “Logistic regression.” (1995).
  * Kleinbaum  _et al._ (2002) David G Kleinbaum, K Dietz, M Gail, Mitchel Klein, and Mitchell Klein,  _Logistic regression_ (Springer, 2002).
  * Williams and Peng (1990) Ronald J Williams and Jing Peng, “An efficient gradient-based algorithm for on-line training of recurrent network trajectories,” Neural computation 2, 490–501 (1990).
  * Cesa-Bianchi (1997) Nicolo Cesa-Bianchi, “Analysis of two gradient-based algorithms for on-line regression,” in  _Proceedings of the tenth annual conference on Computational learning theory_ (1997) pp. 163–170.
  * Koutsoukas  _et al._ (2017) Alexios Koutsoukas, Keith J Monaghan, Xiaoli Li, and Jun Huan, “Deep-learning: investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data,” Journal of cheminformatics 9, 1–13 (2017).
  * Shahriari  _et al._ (2015) Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas, “Taking the human out of the loop: A review of bayesian optimization,” Proceedings of the IEEE 104, 148–175 (2015).
  * Dudley (2010) Richard M Dudley, “Sample functions of the gaussian process,” in  _Selected works of RM Dudley_ (Springer, 2010) pp. 187–224.
  * Schulz  _et al._ (2018) Eric Schulz, Maarten Speekenbrink, and Andreas Krause, “A tutorial on gaussian process regression: Modelling, exploring, and exploiting functions,” Journal of Mathematical Psychology 85, 1–16 (2018).
  * Zhao and Li (2018) Meng Zhao and Jinlong Li, “Tuning the hyper-parameters of cma-es with tree-structured parzen estimators,” in  _2018 Tenth International Conference on Advanced Computational Intelligence (ICACI)_ (IEEE, 2018) pp. 613–618.
  * Yang (2010) Xin-She Yang,  _Nature-inspired metaheuristic algorithms_ (Luniver press, 2010).
  * Lessmann  _et al._ (2005) Stefan Lessmann, Robert Stahlbock, and Sven F Crone, “Optimizing hyperparameters of support vector machines by genetic algorithms.” in  _IC-AI_ , Vol. 74 (2005) p. 82.
  * Lorenzo  _et al._ (2017) Pablo Ribalta Lorenzo, Jakub Nalepa, Michal Kawulok, Luciano Sanchez Ramos, and José Ranilla Pastor, “Particle swarm optimization for hyper-parameter selection in deep neural networks,” in  _Proceedings of the genetic and evolutionary computation conference_ (2017) pp. 481–488.
  * Steane (1998) Andrew Steane, “Quantum computing,” Reports on Progress in Physics 61, 117 (1998).
  * Gruska  _et al._ (1999) Jozef Gruska _et al._ ,  _Quantum computing_ , Vol. 2005 (McGraw-Hill London, 1999).
  * Lentzas  _et al._ (2019) Athanasios Lentzas, Christoforos Nalmpantis, and Dimitris Vrakas, “Hyperparameter tuning using quantum genetic algorithms,” in  _2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI)_ (2019) pp. 1412–1416.
  * Moussa  _et al._ (2022) Charles Moussa, Jan N. van Rijn, Thomas Bäck, and Vedran Dunjko (Springer Nature Switzerland, 2022) pp. 32–46.
  * Gómez  _et al._ (2022) Raúl Berganza Gómez, Corey O’Meara, Giorgio Cortiana, Christian B Mendl, and Juan Bernabé-Moreno, “Towards autoqml: A cloud-based automated circuit architecture search framework,” in  _2022 IEEE 19th International Conference on Software Architecture Companion (ICSA-C)_ (IEEE, 2022) pp. 129–136.
  * Schuld  _et al._ (2015) Maria Schuld, Ilya Sinayskiy, and Francesco Petruccione, “An introduction to quantum machine learning,” Contemporary Physics 56, 172–185 (2015).
  * Biamonte  _et al._ (2017a) Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd, “Quantum machine learning,” Nature 549, 195–202 (2017a).
  * Schuld and Petruccione (2018) Maria Schuld and Francesco Petruccione,  _Supervised learning with quantum computers_ , Vol. 17 (Springer, 2018).
  * Dunjko  _et al._ (2016) Vedran Dunjko, Jacob M Taylor, and Hans J Briegel, “Quantum-enhanced machine learning,” Physical review letters 117, 130501 (2016).
  * Atchade Adelomou  _et al._ (2020) Parfait Atchade Adelomou, Elisabet Golobardes Ribé, and Xavier Vilasís Cardona, “Using the variational-quantum-eigensolver (vqe) to create an intelligent social workers schedule problem solver,” in  _International Conference on Hybrid Artificial Intelligence Systems_ (Springer, 2020) pp. 245–260.
  * Atchade-Adelomou and Alonso-Linaje (2022) Parfait Atchade-Adelomou and Guillermo Alonso-Linaje, “Quantum-enhanced filter: Qfilter,” Soft Computing , 1–8 (2022).
  * Adelomou  _et al._ (2022) Parfait Atchade Adelomou, Daniel Casado Fauli, Elisabet Golobardes Ribé, and Xavier Vilasis-Cardona, “Quantum case-based reasoning (qcbr),” Artificial Intelligence Review , 1–27 (2022).
  * Adelomou  _et al._ (2020) Atchade Parfait Adelomou, Elisabet Golobardes Ribe, and Xavier Vilasis Cardona, “Using the parameterized quantum circuit combined with variational-quantum-eigensolver (vqe) to create an intelligent social workers’ schedule problem solver,”  (2020).
  * Guillermo (2023) Alonso-Linaje Guillermo, “¿quieres aprender computación cuántica ?”  (2023).
  * Arutyunov  _et al._ (2004) G Arutyunov, S Frolov, and M Staudacher, “Bethe ansatz for quantum strings,” Journal of High Energy Physics 2004, 016–016 (2004).
  * Sim  _et al._ (2019) Sukin Sim, Peter D. Johnson, and Alán Aspuru‐Guzik, “Expressibility and entangling capability of parameterized quantum circuits for hybrid quantum‐classical algorithms,” Advanced Quantum Technologies 2, 1900070 (2019).
  * Biamonte  _et al._ (2017b) Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd, “Quantum machine learning,” Nature 549, 195–202 (2017b).
  * Pérez-Salinas  _et al._ (2020) Adrián Pérez-Salinas, Alba Cervera-Lierta, Elies Gil-Fuster, and José I. Latorre, “Data re-uploading for a universal quantum classifier,” Quantum 4, 226 (2020).
  * Schuld  _et al._ (2014) Maria Schuld, Ilya Sinayskiy, and Francesco Petruccione, “An introduction to quantum machine learning,” Contemporary Physics 56, 172–185 (2014).
  * community The SciPy (2021) community The SciPy, “Cobyla,”  (2021).
  * C. (2001) Spall James C., “Spsa community,”  (2001).
  * Liu and Nocedal (1989) Dong C. Liu and Jorge Nocedal, “On the limited memory BFGS method for large scale optimization,” Mathematical Programming 45, 503–528 (1989).
  * Pérez-Salinas  _et al._ (2020) Adrián Pérez-Salinas, Alba Cervera-Lierta, Elies Gil-Fuster, and José I Latorre, “Data re-uploading for a universal quantum classifier,” Quantum 4, 226 (2020).
  * Nielsen  _et al._ (2002) Michael A. Nielsen, Isaac Chuang, and Lov K. Grover, “Quantum computation and quantum information,” American Journal of Physics 70, 558–559 (2002).
  * Browne (2000) Michael W Browne, “Cross-validation methods,” Journal of Mathematical Psychology 44, 108–132 (2000).
  * Cerezo  _et al._ (2021) Marco Cerezo, Andrew Arrasmith, Ryan Babbush, Simon C Benjamin, Suguru Endo, Keisuke Fujii, Jarrod R McClean, Kosuke Mitarai, Xiao Yuan, Lukasz Cincio, _et al._ , “Variational quantum algorithms,” Nature Reviews Physics 3, 625–644 (2021).
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,”  (2014).
  * Ekeland (1974) Ivar Ekeland, “On the variational principle,” Journal of Mathematical Analysis and Applications 47, 324–353 (1974).
  * Apple (2022 (accessed Dec 26, 2022) Apple, “Macbookpro 15,1,” https://support.apple.com/kb/SP776?locale=es_ES (2022 (accessed Dec 26, 2022)).