renderers= heading* = #1 envname-P envname#1 tcb@breakable menhanced,
breakable, drop fuzzy shadow, colback=gray-background, colframe=black,
boxsep=.0pt, left=3mm, fontlower=, halign=justify, sidebyside, sidebyside
gap=1.5mm, sidebyside align=top, lower separated=false, lefthand width = 5mm,
leftrule=3mm*○#1 m boxrule=0pt, boxsep=.0pt, frame empty, left=3mm, right=3mm,
fontlower=, halign=justify, sidebyside, sidebyside gap=1.5mm, sidebyside
align=top, lower separated=false, lefthand width = 5mm, bottom=0pt *○Request:
#1 m boxrule=0pt, boxsep=.0pt, frame empty, left=3mm, right=3mm, fontlower=,
halign=justify, sidebyside, sidebyside gap=1.5mm, sidebyside align=top, lower
separated=false, lefthand width = 5mm, top=0pt *○Answer: #1 m menhanced,
boxsep=0pt, breakable, drop fuzzy shadow, colback=gray-background,
colframe=black, leftrule=3mm, left=0pt, top=0pt, bottom=0pt, right=0pt, #1#2
menhanced, breakable, drop fuzzy shadow, leftrule=3mm, boxsep=.0pt, left=3mm,
right=3mm, halign=justify, fontlower=, colback=gray-background,
colframe=black, sidebyside, sidebyside gap=1.5mm, sidebyside align=top, lower
separated=false, lefthand width = 5mm, *○#1

# Large Language Models for Power Scheduling:
A User-Centric Approach
††thanks: The authors acknowledge the KU-TII 6G Chair on Native AI.

Thomas Mongaillard1, Samson Lasaulce21, Othman Hicheur23, Chao Zhang24, Lina
Bariah 52, Vineeth S. Varma1,
Hang Zou 6, Qiyang Zhao 6, and Merouane Debbah26  1 Université de Lorraine,
CNRS, CRAN, F-54000 Nancy, France.  2 KU 6G Research Center, Khalifa
University, Abu Dhabi, UAE.  3 Ecole Polytechnique, Paris, France. 4Central
South University, Changsha, China.  5Open Innovation AI, Abu Dhabi, UAE. 6TII,
Abu Dhabi, UAE

###### Abstract

While traditional optimization and scheduling schemes are designed to meet
fixed, predefined system requirements, future systems are moving toward user-
driven approaches and personalized services, aiming to achieve high quality-
of-experience (QoE) and flexibility. This challenge is particularly pronounced
in wireless and digitalized energy networks, where users’ requirements have
largely not been taken into consideration due to the lack of a common language
between users and machines. The emergence of powerful large language models
(LLMs) marks a radical departure from traditional system-centric methods into
more advanced user-centric approaches by providing a natural communication
interface between users and devices. In this paper, for the first time, we
introduce a novel architecture for resource scheduling problems by
constructing three LLM agents to convert an arbitrary user’s voice request
(VRQ) into a resource allocation vector. Specifically, we design an LLM intent
recognition agent to translate the request into an optimization problem (OP),
an LLM OP parameter identification agent, and an LLM OP solving agent. To
evaluate system performance, we construct a database (EVRQ) of typical VRQs in
the context of electric vehicle (EV) charging. As a proof of concept, we
primarily use Llama 3 8B. Through testing with different prompt engineering
scenarios, the obtained results demonstrate the efficiency of the proposed
architecture. The conducted performance analysis allows key insights to be
extracted. For instance, having a larger set of candidate OPs to model the
real-world problem might degrade the final performance because of a higher
recognition/OP classification noise level. [Paper codes and
video111https://github.com/thomasmong/llm-power-scheduling]

###### Index Terms:

Large language model, multi-agent, optimization, power scheduling, EV
charging, smart grid, resource allocation, user-centric.

##  I Introduction

With the rapid evolution of complex systems across various domains, the need
for sophisticated scheduling schemes to efficiently manage the system
resources and meet certain requirements has become increasingly critical.
While recently we have witnessed noticeable advancements in the development of
scheduling algorithms (particularly with leveraging artificial intelligence
(AI)-driven methods), it is essential to emphasize that these traditional
algorithms are often designed to satisfy predefined constraints that are
inherently tied to the specific system they serve. Although they have proven
effective in many scenarios, they often fail to perform well when faced with
the dynamic and personalized demands of modern users.

In addition to the fact that they might be uncapable of satisfying the level
of quality-of-experience (QoE) imposed by services that are increasingly
oriented towards satisfying user demands, conventional methods might result in
high complexity and not necessarily lend themselves into an optimum solution
with respect to energy-efficiency in personalized scenarios. Taking the
example of energy management, a key aspect underpinning reliable and
sustainable operations in many systems, including wireless communication
networks, autonomous vehicular systems, smart grids, etc., humans remain
largely out of the loop. For instance, for heating or air conditioning (AC)
systems, humans typically just provide the target temperature, and a more or
less advanced regulation algorithm does all the rest. This approach operates
independently of individual user preferences, or potential changes in energy
availability or costs. Consequently, it may not always optimize for energy-
efficiency or user comfort, demonstrating a clear limitation.

The reason for this gap between humans and algorithms is twofold. First, most
humans are not able to model mathematically a real-world problem and solve it,
which is why algorithms are typically tasked with making most decisions.
Second, humans cannot communicate easily or not at all with algorithms,
machines, or programs. However, the emergence of advanced natural language
processing (NLP) tools, such as large language models (LLMs), is completely
changing the paradigm. In particular, LLMs enable intuitive and effective
human-machine interactions, transforming the operation of complex
infrastructures, such as energy and wireless networks, into more responsive
and user-centric solutions.

It should be noted that the problem of power scheduling is a key in both
wireless networks and in digitalized energy networks. Many wireless resource
management problems and home energy management problems can be formulated as
power scheduling problems [1, 2, 3, 4, 5]. The main goal of this paper is to
leverage the capabilities of LLMs in order to enable the machine to convert a
voice request (VRQ) from a human user into a power scheduling vector. For
example, for the case study under consideration in this paper (namely electric
vehicle -EV- charging), such a request can be: "Charge my EV for tomorrow at 6
a.m. while managing its battery lifetime". For a cell phone, it might be
"Adapt your transmit power to minimize electromagnetic exposure while
guaranteeing my SMS messages always go through".  The novel approach to power
scheduling introduced and developed in this paper is to exploit the knowledge
the LLM has acquired during pre-training and auxiliary instructions to both
model and solve the problem at hand. It is important to highlight that,
although we develop this approach for the particular case of the power
scheduling problem, such an approach can be generalized to help humans solve a
wide range of real-world problems by leveraging mathematical modeling,
reasoning, and solving capabilities.

Nowadays, there are several services, e.g., Amazon Alexa, Google Smart Home,
and Siri-based Apple Smart Home, with advanced interfaces that allow humans to
"talk" to devices such as electrical appliances. However, in all these
solutions, the employed deep learning algorithms only act as mere classifiers,
i.e., they classify the user’s request into a given control action (e.g.,
switch a given electric appliance on). Therefore, the implemented deep
learning schemes do not try to model mathematically or interpret the physical
problem at hand, and they thus they do not attempt to solve it by exploiting
the interpretation, reasoning, and planning capabilities that LLMs-at least
partially-have (see, e.g., [6][7]). Additionally, existing deep learning
solutions are trained for particular tasks, and hence, they do not generalize
to a wider range of tasks, as would be encountered when treating VRQs made by
a human user. Therefore, the standpoint of this paper is original in the sense
that the proposed architecture aims at exploiting LLMs to mathematically model
the physical problem at hand and solve it. To our knowledge, the closest
literature to this approach is given by the literature of math word problems
(see e.g., [8, 9, 10, 11]). In this literature, the real-world problem is
assumed to be perfectly specified by (textual) natural language. The proposed
framework is novel in the following ways. Unlike the aforementioned
literature, the current framework does not rely on the assumption that user
requests are structured in a specific format; instead, they can be arbitrary.
This framework is specifically tailored to address power scheduling tasks,
which has not been tackled in the literature in the context of LLMs, yet. For
the first time, the proposed approach exploits the capabilities of LLMs to
perform both, modeling and solving power scheduling problems. The aim is not
to provide a power scheduling scheme that outperforms existing ones in terms
of predefined performance metric, but rather to revolutionize how optimization
problems (OPs) are formulated and solved for the sake of generating the
recommended power scheduling vector.

To achieve our goal, we propose an LLM multi-agent architecture which allows a
human VRQ to be converted to an OP whose solution is a power scheduling vector
(Vec). The approach of multiple LLM agents has been developed recently (see
e.g., [12, 13, 14, 15]) and consists in specializing an LLM for a given range
of tasks (experts) and associating the LLM agents for a global task. The main
contributions of this paper are summarized as follows: $\bullet$ We develop a
new methodology (based on LLM) to design a scheduling scheme for power
management systems, in which the system requirements are acquired from the
user through VRQs (Sec. II); $\bullet$ We propose a novel architecture (Sec.
III) comprising the design of three LLM agents, with the aim to perform user-
driven power scheduling, i.e., an intent recognition agents (to identify the
best formulation of the mathematical problem given the VRQ from the user), a
parameter identification agent (to determine the required parameters for the
OP from the VRQ as well as the physical system), and an OP solving agent (to
solve the formulated problem through a bank of solving functions in which the
LLM agent assists the solver in the initialization phase through particular
prompts); $\bullet$ The proposed architecture is partially implemented mainly
for Llama3 and evaluated through a thorough performance analysis (Sec. IV);
$\bullet$ As effective accuracy measure, we construct a database of possible
user VRQs related to EV charging (EVRQ), with the corresponding request
complexity, ground-truth OP/OCP classes, and optimal solutions. This database
enables us to evaluate the performance of our framework by comparing the
generated results against these benchmarks, using intent recognition accuracy
(IRA) and final utility (Sec. IV).

##  II Problem Statement

Figure 1: Use-Cases of the Proposed Intelligent Power Scheduling System

In this work, we develop an LLM-based converter which takes as input a
voice/text request from the user (see examples of use cases in Fig. 1) and
transforms it into a power consumption scheduling vector (Vec)

$x=(x_{1},x_{2},...,x_{T})$ (1)

with $\forall t\in\\{1,...,T\\},0\leq x_{\min}\leq x_{t}\leq x_{\max}$, $T$
being the number of time-slots over which the power is scheduled.

Agent 1: OP ClassifierAgent 2: ParserAgent 3: LLM-based OP solver LLM-based
intent recognition agent LLM-based OP parameter identification agent Bank of
external OP solvers LLM assistant system knowledge Internet knowledge
voicerequest OP type complete OP voice request power scheduling vector Figure
2: Proposed multi-agent architecture for a voice request to power scheduling
vector converter (VRQ2Vec)

To accommodate the user’s demands, the "VRQ2Vec" converter has to initially
recognize the user’s intent in an accurate way, find the optimum formulation
for the corresponding OP, and then solve the latter to generate the
recommended power vector that satisfies the requirements. To achieve this
goal, we propose a multi-agent architecture [12], as demonstrated in Fig. 2.
For the problem formulation, we assume a list of most common OPs; for
simplicity, we will refer to optimal control problems (OCPs) as OPs as well.
The first stage of the VRQ2Vec framework is an intent recognition agent (Agent
1), which is tasked to identify from the list the most suitable problem that
can ideally model the user’s intent. The second stage (Parser) is a problem
parameter identification agent (Agent 2) which role consists in extracting the
parameter information of the selected OP. The third stage corresponds to the
OP solver agent (Agent 3). Note that LLM capabilities are exploited in this
study in three ways. First, we exploit their ability to describe a real world
problem as a mathematical problem (which is imposed here to be an OP). Second,
we exploit the LLM for OP parameter identification purposes. Third, we
partially exploit their ability to assist standard OP solvers by allowing the
LLM to share their textual knowledge to help better initialize the solver. The
performance of the three agents and stages of the proposed VRQ2Vec framework
will be assessed through two performance metrics. Intent Recognition Accuracy
(IRA): assuming the existence of perfect human labeling of every VRQ of a
given database into an OP type within a list of OPs, the IRA corresponds to
the empirical percentage of OPs properly classified by the LLM-based
classifier (Agent 1). Average relative optimality loss (AROL): knowing that
misclassification can occur, the AROL measures how suboptimal is the power
vector proposed by the chain Agent 1 $\rightarrow$ Agent 2 $\rightarrow$ Agent
3 in average.

##  III Multi-agent design of the voice request to power vector converter

###  III-A Design of the LLM-based Intent Recognition Agent

The role of the LLM-based intent recognition agent (Agent 1) is to associate a
mathematical problem with a given VRQ from the user. Therefore, it has the
role of modeling a speech or text description of a physical problem into
equations. Due to the inherent limitations of existing LLMs in describing
physical problems as mathematical abstractions, including the latest ChatGPT
4o, we resort to a particular structure of Agent 1, in which this agent is
designed to function as a classifier of pre-selected OPs. For simplicity and
motivated by the literature of power scheduling problems, we define the
following set of possible OPs as described in detail in Table I. For instance,
the VRQ "You have 24h to charge my EV at $80\%$ while minimizing the cost of
charging" can be modeled by a linear program (LP) in which: the vector
$(c_{1},...,c_{T})$ (see Table I) represents the prices of electricity at
time-slots $1,...,T$; choosing $b=-0.8$, $A=(-1,...,-1)$ translates that the
battery state of charge should be at least $80\%$. Note that here we assume 6
OPs but more OPs might be assumed without changing the proposed methodology.
However, having more OPs does not necessarily imply having a better
performance, showing the importance of both choosing the OPs and the number of
OPs. Simulations will support this assertion. The intuition behind the
existence of an optimal number of OPs to be used within a given set of OPs for
classification is similar to the problem of having less robust digital
modulation constellations for large constellations. Here, since the LLM
recognition capabilities are not perfect, this introduces "noise" whose impact
might be higher when the list of possible OPs gets larger.

Agent 1 therefore uses its language processing abilities to classify the VRQ
into an OP (or OCP) type. Note that this classification might be performed by
a supervised classical neural network (e.g., an MLP) but this will come at the
cost of losing the generalizability capability of LLMs. Rather, we consider an
LLM that uses a well-designed context as hard prompting, augmented with
function calling abilities. It is designed for the problem of scheduling power
for charging an EV but could easily be adapted for the wireless example
mentioned in the preceding section. The main LLM we exploit in the performance
evaluation part is Llama3 because it is partly open-sourced and because it can
be run locally with relatively affordable computational power. To sum up, the
design of Agent 1 therefore comprises selecting an LLM model (Llama 3 in this
paper) and utilizing hard-prompting, with a carefully curated list of OPs
(Table I) and a well-structured context (Fig. 8). This setup enables the agent
to efficiently achieve the task of classifying the VRQs. To evaluate the
performance of Agent 1 in terms of IRA, we have constructed a database of VRQs
(EVRQ); more details will be provided in Sec. IV. Note that the LLM’s domain
knowledge in power scheduling can be further improved through fine-tuning with
a specialized dataset, but this is left as an extension of this paper.

###  III-B Design of the LLM-based Parameter Identification Agent

The classifying agent, Agent 1, identifies a relevant OP type that
mathematically describes the request made by the user. In order for Agent 3 to
be able to numerically solve the corresponding OP, it is necessary first to
determine the parameters of the OP. This is where Agent 2 plays a role: the OP
parameter identifier. One can distinguish between three types of parameters.
Type 1: parameters that can be extracted from the VRQ (e.g., the time at which
the battery should be recharged to a certain level). Type 2: parameters that
are pertinent to the physical system (e.g., the maximum power $x_{\max}$).
Type 3: common knowledge parameters that can be sourced from the Internet
(e.g., a typical value for the ambient temperature). As far as the design of
Agent 2 is concerned, the parameters of Types 2 and 3 are considered as inputs
to the agent, whereas Type 1 parameters require the LLM-based agent to exploit
its language processing skills to extract the relevant parameters from the
VRQ.

To access the parameters determined by the parser, Agent 2 requires the
capability of function calling, i.e., the ability of passing some parameters
to another program by using a particular syntax in order to call a real code
function. For each function that needs to be called, we provide a description
of the required parameters to the agent. For the example of EV charging, the
parser must behave as follows: initially, it has to extract the time
parameters from the user request and then call a function that will initialize
those parameters. A time parameter is either an initial time instant, a final
time instant, or a duration that can be explicit (8 a.m.) or implicit
(tomorrow morning). It is necessary to extract those parameters first, since
most of the other parameters are vectors or matrices, whose size depends on
the duration of the scheduling. Then, the parser has to call a solving
function that depends on the OP type selected, for which the arguments are the
parameters of the OP type. In the case of EV charging, to access external
parameters, we allow the agent to create the parameters by using attributes
and methods from a smart meter. The smart meter is the interface between the
environment (or context) and the parser agent. All these parameters form the
complete OP that can be passed to the third agent.

###  III-C Design of the LLM-based OP Solving Agent

A possible approach to solve the OP which describes the power scheduling
problem associated with the VRQ is to use a purely LLM-based agent, that is,
to ask an LLM such as GPT 4o or Llama3 to solve the OP. This approach is
adopted e.g., by OPRO [11] (OPtimization by PROmpting). Assuming that the
optimization task can be described in natural language, OPRO proposes a
prompt-based framework to leverage LLMs as numerical optimizers. In each
optimization step, the LLM generates new solutions from the prompt that
contains previously generated solutions with their values, then the new
solutions are evaluated and added to the prompt for the next optimization
step. Although promising, these approaches are still limited due to the fact
that LLMs were not originally designed to solve mathematical equations222As
per its version of May 31st 2024, Llama 3 8B cannot reliably solve w/o human
assistance first-order equations such as $ax+b=0$ [21]., and may suffer from
accuracy issues, which can be problematic when dealing with stringent physical
or quality of service (QoS) constraints. Therefore, instead of trying to
design LLM-based optimizers which compete with existing numerical solvers, we
rather pursue a coupling approach in which existing solvers are assisted by an
LLM. This allows the exploitation of both the determinism/guarantees offered
by existing solvers and the creative problem-solving capabilities of LLMs.
Agent 3 is chosen to be composed of a bank of 6 solvers based on
scipy.optimize, cvxpy and control Python libraries: solve_LP
(scipy.optimize.linprog); solve_QP (cvxpy.minimize); solve_CP
(scipy.optimize.minimize); solve_MM (scipy.optimize.minimize); solve_LMT
(scipy.optimize.milp); solve_LQR (control.lqr). The initialization of these
solvers are performed by asking an LLM to make the best choice to its
knowledge. The values of the parameters which cannot be extracted from the VRQ
and the system knowledge are found by using an LLM.

##  IV Performance evaluation

In this section, we evaluate the performance of the proposed LLM-based
approach in EV charging applications. Given the wide diversity of OPs
encountered in this domain, we divide them into 6 categories, each with a
given type of performance metric that users might consider when initiating a
charging session. The main objectives of each category can be listed as
follows: Charging cost (CC): Reducing the charging cost; Charging time (CT):
Minimizing the time to charge the EV to a target level; Environmental impact
(EI): Maximizing the use of renewable energy; Power peak (PP): Minimizing the
power peak on the electrical installation; Power variations (PV): Minimizing
fluctuations in the power supply to the EV charger; Grid Damage (GD): Limiting
the potential damage to the distribution grid installation. Each of these
performance metrics is linked to the most suitable OP class that can model the
VRQ. To simplify our approach, we assume there is only one OP within each
category, as follows:

Performance metric | CC | CT | PP | PV | GD
---|---|---|---|---|---
OP class | LP | LMT | MM | QP | CP

We create knowledge files for each OP class that contains both the description
of the problem associated with the performance metrics from the perspective of
EV charging, and the generic mathematical description of the OP class
independent of EV charging. This comprehensive categorization ensures that we
can better instruct LLM agents with prompts for power scheduling in EV
charging scenarios.

###  IV-A Database and evaluation metric

To evaluate our framework, we generate a variety of requests that capture
multiple types of decisions relevant to different EV charging problems.
Specifically, our dataset EVRQ includes scenarios such as minimizing charging
costs, reducing charging time, maximizing the use of renewable energy,
limiting power peaks, reducing power variations, and minimizing grid damage.
Additionally, the requests are designed to be either explicit or implicit.
Explicit requests are requests that explicitly mention to optimize a
particular performance metric, for example: Charge my EV while minimizing the
electricity cost. However, implicit requests such as I want my EV to juice up
but only when it’s financially wise, specify the performance metric to be
assessed with a paraphrase. In total, 800 requests have been generated to
compute the IRA, that correspond to 160 requests for each performance metric.
To evaluate the performance of our approach, we provide labels for each
request, including the ground-truth OP/OCPs class and the optimal solution for
the corresponding problem. By comparing these labels with the generated
results, the IRA, as defined in Section II, serves as an effective measure of
accuracy. In addition, to evaluate the system performance degradation, we
consider the average relative optimality loss (AROL), consisting in weighting
the optimality loss due to the misclassification by the probability of
misclassification, as follows:

$\mathrm{AROL}_{i}=\sum_{J\neq I}{p_{i\rightarrow
J}\frac{1}{N}\sum_{n=1}^{N}{\frac{f_{i}\left(x_{j}^{(n)}\right)-f_{i}\left(x_{i}^{(n)}\right)}{f_{i}\left(x_{i}^{(n)}\right)}}}$
(2)

where $i,j$ are indices for performance metrics with their corresponding OP
classes $I,J$, $p_{i\rightarrow J}$ represents the probability that a request
of type $i$ is classified as $J$, $N$ is large and represents the number of
random abstract requests to evaluate the average loss for misclassified
requests from $i$ to $J$, $x_{i}^{(n)}$ represents the optimal power vector of
the $n$-th request for performance metric $i$, $x_{j}^{(n)}$ represents the
obtained power vector of the $n$-th request misclassified to $J$ and solved as
$j$, $f_{i}$ is the cost function of performance metric $i$. Generating a
random abstract request simply consists in randomly selecting a starting time
and a duration that will be used to solve the predetermined OPs related to the
performance metrics. By taking a large $N$, we estimate the loss induced by
selecting a wrong OP class on average. Then, weighting by the
misclassification rates from IRA simulations gives the average relative
optimality loss for a particular type of performance metric.

###  IV-B Prompt engineering

Basic Prompting: Use of basic prompt and simple mathematical description of
optimization and control problems for classification. In this scenario, we aim
to evaluate the performance of the LLM model using a straightforward system
prompt. The agent is tasked with classifying the request into an OP class from
a predefined list. This setup provides the LLM with only the basic prompts
that include the names and mathematical forms of the OP classes, without
additional contextual information or guidance related to EV charging.

Contextualized Prompting: improving classification using basic prompts and
contextualization of optimization and control problems in the context of EV
charging. To assess the impact of augmenting the LLM with more detailed
knowledge, we extend the basic system prompt from Scenario 1 by appending
comprehensive knowledge files. These files provide both textual and
mathematical descriptions of typical EV charging problems within each OP
class. By enriching the LLM’s input with this detailed information, we aim to
improve its ability to recognize the appropriate OP class and accurately
translate the EV charging request into a canonical form suitable for external
solvers.

Error-Informed Prompting: improving classification by analyzing the previous
errors. In this scenario, we further refine the system prompt to enhance the
classifier’s performance. This involves incorporating specific remarks to
guide the LLM more effectively. These remarks are mainly obtained from
analyzing the mistakes made with the two prompting techniques mentioned above,
as well as leveraging expertise in EV charging and optimization to identify
descriptions with textual ambiguity and sources of optimization confusion. By
regulating the structure and content of the prompts, we aim to streamline the
LLM’s decision-making process, ensuring more accurate and reliable
classification and problem formulation.

These scenarios illustrate the progressive enhancements in prompt engineering
techniques, demonstrating how additional knowledge and refined guidance can
significantly improve the performance of LLMs in classifying and solving
complex OPs in EV charging applications, as shown in the following figures.

Figure 3: Influence of context knowledge for Agent 1 in terms of IRA.The
evaluation is performed for 3 types of voice requests (CC, CT, and PP). The
gains provided by knowledge files are seen to be significant. Figure 4:
Influence of context knowledge in terms of optimality loss for the final power
scheduling performance metric. Figure 5: For 5 different performance metrics
(CC,…,GD): influence of the set of selected OPs
($\\{\mathrm{LP},\mathrm{LMT}\\}$ means for instance that either Linear
Programming or Linear Minimum Time has to be chosen) on its ability to select
the most suitable OP class. TABLE I: Description of Optimization (and optimal
control) Problem classes considered by Agent 1

Linear Programming (LP) |  Quadratic Programming (QP) |  Mini-Max Class (MM)
---|---|---
$\begin{aligned} &\underset{x}{\text{minimize}}&\quad&c^{\top}x\\\ &\text{s.t.}&\quad&Ax\leq b\\\ &&\quad&A_{\text{eq}}x=b_{\text{eq}}\\\ &&\quad&{0}\leq x_{\text{min}}\leq x_{t}\leq x_{\text{max}}\end{aligned}$ |  $\begin{aligned} &\underset{x}{\text{minimize}}&\quad&\frac{1}{2}x^{\top}Qx+c^{\top}x\\\ &\text{s.t.}&\quad&Ax\leq b\\\ &&\quad&A_{\text{eq}}x=b_{\text{eq}}\\\ &&\quad&{0}\leq x_{\text{min}}\leq x_{t}\leq x_{\text{max}}\end{aligned}$ |  $\begin{aligned} &\underset{x}{\text{minimize}}&\quad&\max_{i}f_{i}(x)\\\ &\text{s.t.}&\quad&Ax\leq b\\\ &&\quad&A_{\text{eq}}x=b_{\text{eq}}\\\ &&\quad&{0}\leq x_{\text{min}}\leq x_{t}\leq x_{\text{max}}\end{aligned}$
Convex Programming (CP) |  Linear Minimum-Time (LMT) |  Linear Quadratic Regulator (LQR)
$\begin{aligned} &\underset{x}{\text{minimize}}&\quad&f(x)\\\ &\text{s.t.}&\quad&g_{i}(x)\leq 0,\quad i=1\ldots m\\\ &&\quad&A_{\text{eq}}x=b_{\text{eq}}\\\ &&\quad&{0}\leq x_{\text{min}}\leq x_{t}\leq x_{\text{max}}\end{aligned}$ |  $\begin{aligned} &\underset{x}{\text{minimize}}&\quad&\tau\\\ &\text{s.t.}&\quad&s_{t+1}=As_{t}+Bx_{t}\\\ &&\quad&s_{0}=s_{i},s_{\tau}=s_{f}\\\ &&\quad&{0}\leq x_{\text{min}}\leq x_{t}\leq x_{\text{max}}\\\ &&\quad&s_{\text{min}}\leq s_{t}\leq s_{\text{max}}\end{aligned}$ |  $\begin{aligned} &\underset{x}{\text{minimize}}&\quad&\sum_{t=0}^{N-1}(s_{t}^{\top}Qs_{t}+rx_{t}^{2})\\\ &&\quad&+s_{N}^{\top}Q_{f}s_{N}\\\ &\text{given $s_{0}$ s.t.}&\quad&{0}\leq x_{\text{min}}\leq x_{t}\leq x_{\text{max}}\\\ &&\quad&s_{t+1}=As_{t}+Bx_{t}\\\ \end{aligned}$

Notations: $x$: power scheduling vector; $x_{t},s_{t}$ : power and state at
time $t$; other quantities are parameters.
---

###  IV-C Simulation results

We implemented the different agents in Python by using the _ollama_ library
and Llama3 8B as the base model. We set the temperature of the model to 0 to
eliminate any randomness, ensuring that each request is consistently
classified into the same OP class. Due to the computational constraints of
using Llama3 8B, we limited the scope of some simulations by reducing the
number of OP classes known to Agent 1 to three.

Fig. 3 illustrates the IRA performance with respect to the different proposed
prompting techniques. The obtained results show a clear trend where error-
informed prompting achieves the highest IRA across all performance metrics.
Meanwhile, by providing the typical EV charging problems specific to each OP
class, contextualized prompting shows a moderate improvement over basic
prompting, where the latter is application agnostic. To further study the
impact of different prompting schemes, in Fig. 4 we demonstrate the average
optimality loss using the three different prompting techniques. Confirming the
earlier results, error-informed prompting exhibits the lowest average
optimality loss, indicating the most accurate problem formulation. These two
figures corroborate the advantages of utilizing advanced prompting techniques
in the underlying framework to notably enhance the model’s ability to classify
requests accurately, as well as to introduce improved accuracy performance
with respect to the final charging power vector.

Fig. 5 illustrates the IRA performance depending on the number of OP classes
provided to the classifier. With only one OP class (LP), the classifier can
only handle CC requests, as other requests cannot be resolved using linear
programming alone. This explains the absence of the blue bar in the chart for
requests outside the CT category, as the IRA is zero when relying only on LP.
In addition, the figure highlights that for certain requests, such as those in
the PP category, the IRA decreases as the number of OP classes increases.
Thus, while a limited number of OP classes restricts the number of treatable
requests, adding more OP classes negatively impacts the IRA. It also
emphasizes the importance of explicit user requests, as seen with PP requests
where providing 3 OPs results in a significant gap in IRA (90% compared to 35%
accuracy), between implicit and explicit requests. A similar pattern is
observed for GD requests. This indicates that detailed and explicit user
requests are essential to achieve higher accuracy in classification. It can be
further observed from the figure that classifiers with different sets of OP
classes perform very well in CC and CT categories, compared to other ones.
This is motivated by the fact that cost and time are common requests by the
users and are relatively easier to be classified, for both explicit and
implicit requests.

To further understand other metrics that impact OP classification accuracy,
Fig. 6 explores the influence of a set of selected OPs on Agent 1. The IRA is
plotted versus different cardinality levels with varying distributions of
request categories. The probabilities are defined such that a request
belonging to CC, CT, and PP is defined as $\pi,(1-\pi)/2,(1-\pi)/2$,
respectively. When requests are uniformly distributed across all categories,
using a larger cardinality improves the IRA significantly. This demonstrates
that well-designed prompts can handle a broader range of classes effectively,
hence, enhancing classification accuracy. When requests are predominantly from
one category (with $\pi$ close to 1), a smaller cardinality can yield better
IRA. Additionally, in scenarios where requests are evenly spread, advanced
prompts can help the LLM to distinguish between different OPs efficiently,
whereas basic prompts are sufficient for an acceptable classification accuracy
when requests are very concentrated.

To assess the effect of the selected LLM model in this framework, we tested
the error-informed prompting with other LLM models, including GPT-4o,
AdvancedGemini (AG), and Llama 3 70B, and examined the classification accuracy
on a sample of 30 requests from our database. These samples comprise three
distinct sets: 10 requests that were correctly classified by Llama3 8B to
verify if other models can perform at least as well as Llama3 8B, 10 requests
that were misclassified by Llama3 8B to evaluate if the other models can
improve upon Llama3’s performance, and 10 EI requests that are not included in
the classifier’s knowledge base to examine if the selected models can
extrapolate and classify requests that are not directly within their known
dataset. We compare the performance of Llama3 8B model versus GPT-4o,
AdvancedGemini (AG), and Llama 3 70B. The obtained IRA results for different
models are outlined in Table. II. The results demonstrate the potential of
sophisticated/larger LLMs in improving the OP classification. It further shows
that the proposed framework is well-suited for most popular LLMs, emphasizing
on the generalizability of this proposed architecture. Also, it suggests that
such performance can be further enhanced with better-trained models.

Finally, in Fig. 7, we illustrate the patterns of the yielded power vectors
for different requests types. For charging cost minimization requests, the
charging power is primarily allocated to time slots with lower electricity
prices. In contrast, for requests aiming to minimize the charging time, the
power vector is more concentrated at the beginning, regardless of other
metrics. For power peak minimization requests, the charging power is higher
during time slots with lower non-flexible loads to balance the overall
electrical load. By adapting to the charging requests, the power vector can be
adjusted to meet very diverse requirements effectively.

Figure 6: Impact of the set size of OPs on the classification accuracy of Agent 1. The result is seen to depend on the distribution of the voice request database (which is represented by the probability $\pi$ of having a request of type CC). Having a larger set of OPs to model the power scheduling problems can negatively affect the recognition accuracy. Figure 7: Final power scheduling vector generated by the proposed agent chain for different classes of voice requests (CC, CT, PP) compared to a basic constant power charging policy. The results constitute a proof of concept for the proposed methodology when applied to EV charging: user’s voice requests are translated to a very suitable power scheduling policy for very diverse requests (800 VRQs). TABLE II: Influence of the LLM model on accuracy (IRA). | Llama3 8B | GPT-4o | AG | Llama3 70B
---|---|---|---|---
Perfectly | 100% | 100% | 100% | 100%
Classified |  |  |  |
Misclassified | 0% | 90% | 60% | 90%
EI | 90% | 100% | 100% | 90%

##  V Conclusion

This paper proposes, for the first time in the literature, how to exploit LLMs
to convert an arbitrary VRQ into a power vector. We develop an efficient
multi-agent architecture that relies on existing LLM models. We corroborate
the efficacy of the proposed methodology by a thorough performance analysis
for the EV charging problem. To conduct this analysis, we create a database of
VRQs and proposed approaches to handle practical implementation aspects,
including OP parameter identification. The corresponding results provide key
insights. Having a larger set of possible OPs to model a real-world problem
can be detrimental to the model choice problem, creating a tradeoff between
accurately modeling a physical problem and the model’s ability to correctly
recognize the type of problem. The proposed architecture is original and opens
a broad avenue for improvements. In particular: the design of the agent which
models the physical problem can be improved by having a more diverse set of
OPs and by being fine-tuned for wireless/energy networks; the agent which
solves the selected OP can be improved by better coupling the capabilities of
standard OP solvers with creative LLM-based solvers. We believe that the
approach introduced in this paper will be key for humans to interact with
wireless/energy networks.

##  VI Appendix

Agent 1: OP ClassifierAgent 2: ParserAgent 3: LLM-based OP solver *○

Figure 8: Extract from a system prompt given to Agent 1.

## References

  * [1] Fattah, H., and Leung, C. (2002). An overview of scheduling algorithms in wireless multimedia networks. IEEE Wireless Comm., 9(5), 76-83.
  * [2] ElBatt, T., and Ephremides, A. (2004). Joint scheduling and power control for wireless ad hoc networks. IEEE Transactions on Wireless Communications, 3(1), 74-85.
  * [3] Hohlt, B., Doherty, L., and Brewer, E. (2004, April). Flexible power scheduling for sensor networks. In Proc. of the 3rd International Symposium on Information Processing in Sensor Networks (pp. 205-214).
  * [4] Radunovic, B., and Le Boudec, J. Y. (2004). Optimal power control, scheduling, and routing in UWB networks. IEEE Journal on Selected Areas in Communications, 22(7), 1252-1270.
  * [5] Makhadmeh, S. et al. (2019). Optimization methods for power scheduling problems in smart home: Survey. Renewable and Sustainable Energy Reviews, 115, 109362. Elsevier Journal.
  * [6] Floridi L, Chiriatti M. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines. 2020 Dec;30:681-94.
  * [7] Touvron H. et al., Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. 2023 Jul 18.
  * [8] Koncel-Kedziorski, R. et al. (2016, June). MAWPS: A math word problem repository. Proc. of the Conf. of the North American Chap. of the Assoc. for Computational Linguistics: Human Language Technologies (pp. 1152-1157).
  * [9] Wang, Y., Liu, X., and Shi, S. (2017, September). Deep neural solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 845-854).
  * [10] He-Yueya, J. et al. (2023). Solving math word problems by combining language models with symbolic solvers. arXiv:2304.09102.
  * [11] Yang C. et al., Large language models as optimizers. arXiv:2309.03409. 2023 Sep 7.
  * [12] Talebirad, Y., and Nadiri, A. (2023). Multi-agent collaboration: Harnessing the power of intelligent LLM agents. arXiv:2306.03314.
  * [13] Task-driven Autonomous Agent Utilizing GPT-4, Pinecone, and LangChain for Diverse Applications. Yohei Nakajima. 2023.
  * [14] Auto-GPT: An Autonomous GPT-4 Experiment. https://github.com/Significant-Gravitas/Auto-GPT. 2023.
  * [15] Xiao Z. et al., Chain-of-Experts: When LLMs Meet Complex Operations Research Problems. InThe Twelfth International Conference on Learning Representations 2023 Oct 13.
  * [16] Larminie J, and Lowry J. Electric vehicle technology explained. John Wiley & Sons; 2012 Sep 17.
  * [17] Cao Y. et al., An optimized EV charging model considering TOU price and SOC curve. IEEE Trans. on Smart Grid. 2011 Aug 8;3(1):388-93.
  * [18] AhmadiTeshnizi A, Gao W, Udell M. OptiMUS: Scalable Optimization Modeling with (MI) LP Solvers and Large Language Models. arXiv:2402.10172. 2024 Feb 15.
  * [19] Li B. et al., Large language models for supply chain optimization. arXiv:2307.03875. 2023 Jul 8.
  * [20] Tang Z. et al., ORLM: Training Large Language Models for Optimization Modeling. arXiv:2405.17743. 2024 May 28.
  * [21] "LLaMA 3 tested". Matthew Berman. https://www.youtube.com/watch?v=0AaNT7XO41I&t=166s


11institutetext: Embedded Vision Systems Group, Computer Vision Laboratory,
Department of Automatic Control and Robotics,
AGH University of Science and Technology, Kraków, Poland
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Traffic Sign Classification Using Deep and Quantum Neural Networks

Sylwia Kuros  Tomasz Kryjak

###### Abstract

Quantum Neural Networks (QNNs) are an emerging technology that can be used in
many applications including computer vision. In this paper, we presented a
traffic sign classification system implemented using a hybrid quantum-
classical convolutional neural network. Experiments on the German Traffic Sign
Recognition Benchmark dataset indicate that currently QNN do not outperform
classical DCNN (Deep Convolutuional Neural Networks), yet still provide an
accuracy of over 90% and are a definitely promising solution for advanced
computer vision.

###### keywords:

Quantum Neural Networks, Traffic Sign Recognition, DCNN, GTSRB

##  1 Introduction

Nowadays, the amount of data produced doubles every two years [6], and the
peak of the heyday of computers in the classical meaning is coming to an end.
Maintaining the current momentum of technological development requires a
change in the approach to computing. One of the most promising solutions is to
transfer the idea of computing from the field of classical mechanics to
quantum mechanics, which creates new and interesting possibilities, but at the
same time poses significant challenges.

In computer vision systems that for effective operation require processing of
large amounts of data in real time, quantum neural networks (QNNs) can prove
to be a very attractive solution. On the basis of experiments carried out in
recent years, it can be observed that the network training process is becoming
quicker and results are more accurate. They also show better generalization
capabilities even with small amounts of training data and require several
times fewer epochs compared to classical networks. QNNs were successfully
used, among others, in the following applications: tree recognition in aerial
space of California [3], cancer recognition [10], facial expression
recognition [17], vehicle classification [19], traffic sign recognition from
the LISA database considering the vulnerability of adversarial attacks [12],
handwriting recognition [22], [15], [18], [21], [7], [8], object segmentation
[2], [1], pneumonia recognition [20], classification of ants and bees [13],
classification of medical images of chest radiography and retinal color of the
fundus [14], and generative networks [11], [4].

In this paper, we describe the results of our work on a quantum neural network
for the classification of traffic signs from the German Traffic Sign
Recognition Benchmark (GTSRB) dataset. The aim of our research was to analyze
and compare the results obtained using a classical deep convolutional neural
network (DCNN) and a quantum neural network. The architecture was implemented
using the Python language and the PennyLane library for quantum computing. To
the best of our knowledge, this is the only work on traffic sign
classification for the popular GTSRB dataset that uses quantum computing
techniques. Another paper ([18]) on a similar topic deals with a different set
and the proposed methods have lower efficiency.

The remainder of this paper is organized as follows. Section 2 provides
background information on quantum neural networks. In Section 3 our motivation
and the purpose of our experiment were described in the context of related
work on machine learning applied to computer vision systems. Section 4
presents the experiments conducted and the results obtained. In Section 5
conclusions and ideas for further development are provided.

##  2 Quantum Neural Networks

The registers of quantum computers can exist in all possible states
simultaneously, due to the property of superposition, with a chance of
capturing a state at the time of measurement whose probability was encoded
before the measurement. Quantum computing works on the principle of increasing
the probability of a desired state to a certain high value, so that this state
can be reached with high confidence and with as few measurements as possible.

In this context, quantum interference resulting from the superposition
phenomenon allows the amplitudes of the probabilities corresponding to given
states to influence each other. The same is true for quantum entanglement,
which allows quantum objects to be linked by a strong correlation, useful for
readings of quantum states. Quantum entanglement means that two particles
remain connected despite the distances that separate them. In quantum
computing, such a state allows for the connection of a large number of qubits
(quantum equivalent of the classical bit) together, thereby increasing
computational resources in a nonlinear way.

There are many approaches to the issue of physical prototyping of a quantum
computer, the two most popular being based on superconducting electronic
circuits or ion traps. In the case of superconducting quantum circuits, the
quantum processor is placed at the bottom of the gas pedal cylinder in a
shield composed of a material called cryoperm and a magnetic field. It is led
by wires that send microwave pulses of varying frequency and duration to
control and measure the state of the qubits. Due to the interference that
temperature and environmental factors bring, dilution chillers are used to
lower the temperature to 15 millikelvin. Quantum information can also be
destroyed by resistance, so a superconducting material with zero resistance at
some low temperature is used [9].

A slightly different operating principle can be observed in quantum computer
architectures based on ion traps. The advantage of this approach is the
ability to capture the state of a qubit at a well-defined location _trapping_
it using an electromagnetic trap called a Paul trap [16]. It works on the
principle of a time-varying electric field in such a way as to hold the ion in
constant position. To obtain architectures with multiple qubits, it is
necessary to trap multiple ions in a linear chain. The use of ions as qubits
requires the use of laser light to change the state of an electron in an atom
from a ground state to an excited state. Keeping it in the excited state is
achieved by changes in the laser frequency.

Due to the lack of widespread access to real quantum computers, a lot of
research is carried out in an artificial environment using special software
that simulates the ideal relationships found in quantum mechanics. In the case
of this paper, considerations based on a simulated environment have also been
performed.

Currently available classical computers, including supercomputers and cloud
computing, allow simulating quantum hardware. This supports, to a limited
extent, research on quantum algorithms and computations in parallel to the
work on the hardware layer. One of the interesting topics are quantum neural
networks addressed in this paper.

In its current stage of development, quantum deep learning is based on two
classes of neural networks. The first one consists of quantum and classical
neural layers and is called a hybrid quantum-classical neural network, whereas
the second uses only quantum gates in layer construction.

A hybrid quantum-classical neural network contains a hidden quantum layer
built of parameterized quantum circuits whose basic units are quantum gates.
They allow the state of the qubit to be modified by superposition,
entanglement, and interference, so that the measurement result uniquely
correlates with the unobservable state. In quantum neural networks, quantum
gates rotate the state of a qubit that is a parameter for rotating gates in a
given layer, based on the output of a classical circuit in the preceding
layer.

The idea of training a classical and a quantum network is the same. However,
an important difference is that, in the case of quantum networks, the
parameters of the quantum circuit are optimized, whereas, in the process of
learning a classical network, the search for the best possible weights is
performed.

There are an infinite number of quantum gates, but this paper focuses on
discussing the most relevant ones. The quantum NOT gate (or Pauli-X) belongs
to the class of single-qubit gates. For visualization purposes, the way to
represent a qubit states in a 3D spherical coordinates called the Bloch sphere
was proposed. The operation it performs corresponds to the rotation of the $x$
on the Bloch sphere by an angle $\pi$. In other words, the NOT gate assigns a
probability amplitude to the state $|0\rangle$ to $|1\rangle$ and vice versa
according to the following formula:

$X:\alpha|0\rangle+\beta|1\rangle\rightarrow\beta|0\rangle+\alpha|1\rangle$
(1)

The matrix representation is as follows:

$X=\begin{bmatrix}0&1\\\ 1&0\end{bmatrix}$ (2)

The Pauli-Y gate corresponding to the rotation of the $y$ axis on the Bloch
sphere by an angle $\pi$ is described by a unitary matrix of the following
form:

$Y=\begin{bmatrix}0&-i\\\ i&0\end{bmatrix}$ (3)

The Pauli-Z gate for the rotation of the $z$ axis on the Bloch sphere by the
angle $\pi$ is described by the following unitary matrix:

$Z=\begin{bmatrix}1&0\\\ 0&-1\end{bmatrix}$ (4)

Similarly, the rotation gates $RX$,$RY$,$RZ$ belonging to the group of single-
qubit gates perform an angle rotation $\theta$ on the Bloch sphere,
respectively, around the axis $x$, $y$, $z$.

In matrix representation $RX$ is given as follows:

$RX(\theta)=\begin{bmatrix}\cos\frac{\theta}{2}&-\sin\frac{\theta}{2}\\\
\sin\frac{\theta}{2}&\cos\frac{\theta}{2}\end{bmatrix}$ (5)

Gate $RY$:

$RY(\theta)=\begin{bmatrix}\cos\frac{\theta}{2}&-i\sin\frac{\theta}{2}\\\
-i\sin\frac{\theta}{2}&\cos\frac{\theta}{2}\end{bmatrix}$ (6)

Gate $RZ$:

$RZ(\theta)=\begin{bmatrix}e^{-i\frac{\theta}{2}}&0\\\
0&e^{i\frac{\theta}{2}}\end{bmatrix}$ (7)

The Hadamard gate is also a gate that operates on a single qubit. On a given
computational basis, it creates a superposition state. This operation can be
illustrated as a rotation $\frac{\widehat{x}+\widehat{z}}{\sqrt{2}}$ on the
Bloch sphere by an angle $\pi$. This gate is given by a unitary matrix:

$H=\frac{1}{\sqrt{2}}\begin{bmatrix}1&1\\\ 1&-1\end{bmatrix}$ (8)

Phase gates also fall into the category of single-cubit gates. U1 gate rotates
the cubit around $z$ axis, U2 gate around $x+z$ axis and U3 gate is a rotation
gate with three Euler angles. The matrix representations of these gates are as
follows:

$U1(\theta)=\frac{1}{\sqrt{2}}\begin{bmatrix}1&0\\\
0&e^{i\frac{\theta}{2}}\end{bmatrix}$ (9)

$U2(\theta,\gamma)=\begin{bmatrix}1&-e^{i\frac{\gamma}{2}}\\\
e^{i\frac{\theta}{2}}&e^{i\frac{\theta+\gamma}{2}}\end{bmatrix}$ (10)

$U3(\lambda,\theta,\gamma)=\begin{bmatrix}\cos\frac{\lambda}{4}&-e^{i\frac{\gamma}{2}}\sin\frac{\lambda}{4}\\\
e^{i\frac{\theta}{2}}\sin\frac{\lambda}{4}&e^{i\frac{\theta+\gamma}{2}}\cos\frac{\lambda}{4}\end{bmatrix}$
(11)

For two-qubit gates consisting of a control qubit and a target qubit, the
state of the control qubit remains constant during operations. A change of the
state of the target qubit is performed when the state of the control qubit is
$|1\rangle$. A popular one in this group is the $CNOT$ gate (controlled
Pauli-X). The unitary matrix representing the CNOT gate is of the form:

$CNOT=\begin{bmatrix}1&0&0&0\\\ 0&1&0&0\\\ 0&0&0&1\\\ 0&0&1&0\end{bmatrix}$
(12)

For the CZ gate, the representation is as follows:

$CZ=\begin{bmatrix}1&0&0&0\\\ 0&1&0&0\\\ 0&0&1&0\\\ 0&0&0&-1\end{bmatrix}$
(13)

In the case of a controllable CRY rotation gate, the matrix is given:

$CRY(\theta)=\begin{bmatrix}1&0&0&0\\\
0&\cos\frac{\theta}{2}&0&-\sin\frac{\theta}{2}\\\ 0&0&1&0\\\
0&\sin\frac{\theta}{2}&0&\cos\frac{\theta}{2}\end{bmatrix}$ (14)

A network built solely from quantum layers based on unitary gates was first
proposed by Edward Farhi and Hartmut Neven in 2018 [5]. In this neural
network, the quantum circuit consisted of a sequence of parameter-dependent
unary transformations that performed transformations on the input quantum
state. At the current stage of development, hybrid networks seem much more
promising because they assume operations on small quantum circuits with
negligible or zero error correction, so they are shown to have potential for
use with upcoming quantum computer architectures. This is because Noisy
Intermediate-Scale Quantum (NISQ) era computer architectures are solutions
with limited computational resources and need the support of performing some
of the computation on classical hardware. In many publications, the term
_quantum neural network_ is actually a hybrid model. It is worth noting that,
at the current stage of development, purely quantum architectures are
successfully applied only to classification problems with a small number of
classes, since the number of qubits required to encode and read data increases
with the number of classes. For some classification tasks, this leads to the
system exceeding the number of available qubits or generating significant
errors.

##  3 Previous work

Due to their fast convergence and high accuracy, quantum neural networks can
find applications in many areas, especially in vision systems, whose
bottleneck is the need to process large amounts of data efficiently. For
example, in the paper [3] a successful attempt was made to recognize trees in
the aerial space of California, where ensemble methods were used. The authors
proposed a truncation and rescaling of the training objective through a
trainable metaparameter. Accuracies of 92$\%$ in validation and 90$\%$ on a
test scene were obtained.

In the paper [10] experiments on cancer recognition were carried out. The
authors proposed a qubit neural network model with sequence input based on
controlled rotation gates, called QNNSI. The three-layer model with a hidden
layer, which employed the Levenberg-Marquardt algorithm for learning, was the
best of all the approaches tested. The experimental results reveal that a
greater difference between the input nodes and the sequence length leads to a
lower performance of the proposed model than that of the classical ANN, in
contrast, it obviously enhances the approximation and generalization ability
of the proposed model when the input nodes are closer to the sequence length
[10].

An experiment proposed in [19] included the classification of vehicles.
Successful object segmentation proposed in [2], [1]. Other research shows that
quantum neural networks can be effective in the recognition of pneumonia [20].
The classification of ants and bees introduced in [13] indicates that the use
of a transfer learning technique can result in the creation of an efficient
system for differentiating between two types of insects. The retinal color
fundus images and chest radiography classification presented in [14] is
another example of the superiority of quantum neural networks over classical
ones.

Topics similar to the subject of this research can be found in the paper [12].
The authors applied a transfer learning approach and used a Resnet18 network
trained on an ImageNet set, replacing the fully connected layer with a quantum
layer and a PyTorch layer-enabled module. The results obtained from such a
network were compared with those of a simple two-layer classical network.
Using the PennyLane plug-in, quantum computations were possible, and the
popular PyTorch library was used. Three steps are distinguished in the quantum
layer. The first was to embed the data in a quantum circuit using a single or
combinations of several gates: Hadamard gate, Rotational Y, Rotational X,
Rotational Z, U1, U2, and U3 gates. The data were then fed into a
parameterized quantum circuit of two-qubit gates (Controlled NOT, Controlled
Z, and Controlled RX) with single-qubit parameterized gates (i.e., Rotational
Y, Rotational X, Rotational Z, U1, U2 and U3). The last was a measurement step
using bases composed of X, Y, Z quantum gates. In this experiment, the LISA
traffic sign set was reduced to 18 traffic sign classes. A binary
classification based on the network’s recognition of stop signs and the class
in which the other signs were placed was tested. This used 231 samples divided
into a training set and a test set in a ratio of 80:20. For multiclass
classification, the signs were assigned to one of the three classes: stop
signs, speed limit signs, and the class containing the other signs. To do
this, 279 samples divided into training and test set were used in a ratio of
80:20. The following attack algorithms were used: the gradient attack, Fast
Gradient Sign Method (FGSM), Projected Gradient Descent (PGD) attack. The
implementations used achieved high accuracy (more than 90%).

##  4 QNN for traffic sign recognition

The GTSRB (German Traffic Sign Recognition Benchmark) dataset used in this
experiment was created in 2011. It contains 39209 training images and 12630
test images, which varies in size and other parameters, e.g. brightness and
contrast. Each of the images belongs to one of the 43 class representing
traffic signs.

An initial pre-processing consisting of two stages was introduced. In the
first, grayscale normalization was performed and in the second, the dataset
was filtered and only images larger than $64\times 64$ pixels were considered
in the next steps. This was due to the observation that the network performed
better for larger images. In the next step, the data set was divided into the
80:10:10 ratio for the training, test and validation datasets. Data labels
were converted to vectors of length equal to the number of classes present
according to the one-hot encoding method. The images of each subset were then
subjected to a quantum convolution operation. In experiments, a similar
approach to that presented in [7] was used. Quantum convolution (quanvolution)
layers were built from a group of $N$ quantum filters that create feature maps
by locally transforming the input data. The key difference between quantum and
classical convolution is the processing of spatially local subareas of data
using random or structured quantum circuits. Figure 1 shows the design of such
an architecture.

Figure 1: The architecture of a quanvolutional neural network proposed in [7].
The quanvolution layers are made up of a group of $N$ quantum filters that
create feature maps. Local subareas of data are processed spatially using
random or structured quantum circuits.

For each image, a small area of dimensions $2\times 2$ was embedded in the
quantum circuit by rotations parameterized by the rotation angle of the qubits
initialized in the ground state. Further quantum computations were performed
on the basis of the unitary space generated by a random circuit similar to
that proposed in [7]. The experiment performed in this paper uses a circuit
consisting of a single-qubit rotation operator $Y$ and _Pauli-X_ gate. The
number of random layers was set to two because this parameter was not observed
to lead to any change in classification quality during the experiments. Then,
as in classical convolution, each z value was mapped to each of the output z
channels. Iteratively repeating these steps yielded a complete output object
that could be structured into a multichannel image. The main difference
between the experiment proposed in this paper versus [7] was to reduce the
number of epochs from ten thousand to one hundred, reduce the dataset to seven
thousand samples instead of dimensionality reduction, and change the neural
network architecture to a more advanced one.

Both classical and quantum networks consisted of convolutional, max-pooling,
dropout, flatten, and dense layers. The difference between them was in the
convolution operation performed on the images. In the hybrid model, images
convolved with quantum circuits were fed. The models were compiled with the
adaptive optimizer Adam and the cost function as the cross-entropy with the
accuracy metric. The final step was to verify the quality of the learning by
determining the confusion matrix and the precision, recall, and f-beta
parameters on the test set. Table 1 contains the results obtained.

Table 1: Summary of results obtained for the classical network and the network
with quantum convolution at particular batch sizes for the 43 classes of the
GTSRB dataset

Batch Accuracy Precision Recall F-beta size CNN QNN CNN QNN CNN QNN CNN QNN 4
0.9957 0.9254 0.9914 0.9211 0.9918 0.9310 0.9912 0.9216 8 0.9914 0.9426 0.9949
0.9426 0.9943 0.9456 0.9944 0.9216 16 0.9971 0.9426 1.0000 0.9426 1.0000
0.9453 1.0000 0.9419 32 0.9986 0.9369 0.9947 0.9947 0.9369 0.9943 0.9476
0.9403 64 0.9957 0.9383 0.9959 0.9983 0.9957 0.9423 0.9954 0.9378 128 0.9986
0.9440 0.9973 0.9440 0.9986 0.9494 0.9979 0.9446 256 0.9943 0.9369 0.9973
0.9369 0.9971 0.9428 0.9971 0.9382 512 0.9986 0.9354 0.9959 0.9354 0.9957
0.9381 0.9956 0.9351

In the case of the classical network, there was only one incorrect
classification for a given input, and the number of images assigned to
incorrect labels did not exceed one. For the network with quantum convolution,
the incorrect detections are slightly more, and those that recur most
frequently and contain more than one misclassified label for a given batch are
the following pairs of visually similar characters:

  * •

pedestrians and priority road,

  * •

bicycles crossing and double curve,

  * •

wild animals and road narrows on the right,

  * •

go straight or right and pedestrians,

  * •

go straight or left and pedestrians,

  * •

wild animals and pedestrians,

  * •

road narrows right and end of all limits,

  * •

pedestrians and road narrows right.

In some of the above cases, there is little or no similarity from the point of
view of the human viewer, for example “crossing priority at an intersection”
and “attention pedestrians”. The implication is that the process of quantum
convolution makes the model more robust to errors associated with similar sign
content, while shifting errors towards a less recognized cause. This brings
back an association with characteristic quantum non-explicability.

The advantage of the classical network can be seen each time, as it achieves
up to 99.86$\%$ accuracy for batch sizes of 32, 128, 512. For the network with
quantum convolution, the highest obtained result for accuracy is 94.40$\%$ for
batch size 128, so for the best cases this network shows almost 4$\%$ worse
classification accuracy. Although the classical network in the worst case
achieved an accuracy of 99.14$\%$ for a batch of size 8, the network with
quantum convolution achieves a decrease greater than 2$\%$ in classification
quality.

In the case of the network with quantum convolution, this is 94.4$\%$ for a
batch size of 128, which is more than 5% lower. The learning flow for this
batch is illustrated in Fig. 3. The classical network achieves a worst-case
score of 99.14$\%$ for a batch of size 4, and for the network with quantum
convolution, this score for the same batch is more than 2$\%$ worse. The
differences in the values of the recall and f-beta coefficients were at a
similar level. Figure 4 shows the confusion matrix for a neural network with
quantum convolution trained with a batch of size 128.

Figure 2: Comparison of the accuracy obtained with a batch size of 16 for the
classical network and the network with quantum convolution on the training set
Figure 3: Comparison of the accuracy obtained with a batch size of 128 for the
classical network and the network with quantum convolution on the training set
Figure 4: Confusion matrix with a batch of size 128 for networks with quantum
convolution

The analysis shows that for both networks the worst results were obtained with
a batch size equal to 4. The best results for each network were obtained with
different batch sizes. For the classical network, the best of the tested ones
was a batch size of 16, while for the network with quantum convolution it was
a batch size of 128. This results in an important difference in the approach
to quantum networks, as these models can take larger portions of data, and
thus their learning will be faster. Note, however, that they are very
sensitive to the appropriate choice of batch file, much more so than classical
networks, which show degradation at the tenths of a percent level over their
best and worst performance.

The algorithm applied allowed us to obtain much better classification accuracy
than it was in the case of [18], whereby in our experiment, classification was
performed using 43 classes. The results obtained are at a level similar to
some of the experiments carried out in [12].

##  5 Conclusion

Quantum machine learning is a challenging field. This concerns not only the
proper design of algorithms but also the very way in which classical real data
are encoded in quantum space. Uncertainty is also provided by the fact that
the only official form of a quantum computer is not known, but rather a number
of proposals for what it could look like. This results in potentially going
down wrong paths and having to redesign the algorithms multiple times.

The experiment showed that it is possible to achieve a high classification
accuracy (more than 94%) for a neural network with quantum convolution, but
raised the question of quantum supremacy. Quantum algorithms require special
preprocessing on the dataset, which in the case of hybrid networks, of which
the network with quantum convolution is an example, requires significant
computational resources, and the results obtained do not show the expected
significant superiority over the results returned by the classical network.
However, the purpose of the experiment was not to improve the chosen path, but
to outline a direction on how to prototype an exemplary quantum-classical
neural network model for the multiclass classification problem, which could
find its application in vision systems.

The prospects for further development of the project are very broad. It is
planned to try to compare the influence of data augmentation on learning
accuracy and the potential overfitting effect in the comparison of the
classical network and the network with quantum convolution. In addition,
another task will be undertaken in the area of explainable AI, that is, an
attempt to visualize, for example, in the form of a heat map, which features
of the image influence a given classification result. It would also be
interesting to try out the designed network architecture on a real quantum
computer along with undertaking an analysis of the temporal performance of the
algorithm.

###  5.0.1 Acknowledgements

The work presented in this paper was supported by the AGH University of
Science and Technology project no. 16.16.120.773.

## References

  * [1] Aytekin, C., Kiranyaz, S., Gabbouj, M.: Quantum mechanics in computer vision: Automatic object extraction. In: 2013 IEEE International Conference on Image Processing. pp. 2489–2493 (2013)
  * [2] Aytekin, C., Kiranyaz, S., Gabbouj, M.: Automatic Object Segmentation by Quantum Cuts. In: 2014 22nd International Conference on Pattern Recognition. pp. 112–117 (2014)
  * [3] Boyda, E., Basu, S., S., G., A., M. et al.: Deploying a quantum annealing processor to detect tree cover in aerial imagery of California. PLoS ONE 12(2) (2017)
  * [4] Dallaire-Demers, P.L., Killoran, N.: Quantum generative adversarial networks. Phys. Rev. A 98, 012324 (Jul 2018), https://link.aps.org/doi/10.1103/PhysRevA.98.012324
  * [5] Farhi, E., Neven, H.: Classification with Quantum Neural Networks on Near Term Processors (2018), https://arxiv.org/abs/1802.06002
  * [6] Gallagher, B.: The amount of data in the world doubles every two years (last access: 22042022), https://medium.com/callforcode/the-amount-of-data-in-the-world-doubles-every-two-years-3c0be9263eb1
  * [7] Henderson, M., Shakya, S., Pradhan, S., Cook, T.: Quanvolutional neural networks: powering image recognition with quantum circuits. Quantum Machine Intelligence 2(1), 2 (Feb 2020), https://doi.org/10.1007/s42484-020-00012-y
  * [8] Hernández, H.I.G., Ruiz, R.T., Sun, G.H.: Image Classification via Quantum Machine Learning (2020), https://arxiv.org/abs/2011.02831
  * [9] Huang, H.L., Wu, D., Fan, D., Zhu, X.: Superconducting quantum computing: a review. Science China Information Sciences 63(8), 180501 (Jul 2020), https://doi.org/10.1007/s11432-020-2881-9
  * [10] Li, P., Xiao, H.: Model and algorithm of quantum-inspired neural network with sequence input based on controlled rotation gates. Applied Intelligence 40(1), 107–126 (Jan 2014), https://doi.org/10.1007/s10489-013-0447-3
  * [11] Lloyd, S., Weedbrook, C.: Quantum Generative Adversarial Learning. Phys. Rev. Lett. 121, 040502 (Jul 2018), https://link.aps.org/doi/10.1103/PhysRevLett.121.040502
  * [12] Majumder, R., Khan, S.M., Ahmed, F., Khan, Z. et al.: Hybrid Classical-Quantum Deep Learning Models for Autonomous Vehicle Traffic Image Classification Under Adversarial Attack (2021), https://arxiv.org/abs/2108.01125
  * [13] Mari, A., Bromley, T.R., Izaac, J., Schuld, M. et al.: Transfer learning in hybrid classical-quantum neural networks. Quantum 4, 340 (Oct 2020), https://doi.org/10.22331/q-2020-10-09-340
  * [14] Mathur, N., Landman, J., Li, Y.Y., Strahm, M. et al.: Medical image classification via quantum neural networks (2021), https://arxiv.org/abs/2109.01831
  * [15] Oh, S., Choi, J., Kim, J.: A Tutorial on Quantum Convolutional Neural Networks (QCNN). In: 2020 International Conference on Information and Communication Technology Convergence (ICTC). pp. 236–239 (2020)
  * [16] Paul, W., Steinwedel, H.: Notizen: Ein neues Massenspektrometer ohne Magnetfeld. Zeitschrift für Naturforschung A 8(7), 448–450 (1953), https://doi.org/10.1515/zna-1953-0710
  * [17] Peng, L., Li, J.: A Facial Expression Recognition Method Based on Quantum Neural Networks. In: Proceedings of the 2007 International Conference on Intelligent Systems and Knowledge Engineering (ISKE 2007). pp. 51–54. Atlantis Press (2007/10), https://doi.org/10.2991/iske.2007.10
  * [18] Potempa, R., Porebski, S.: Comparing Concepts of Quantum and Classical Neural Network Models for Image Classification Task. In: Choraś, M., Choraś, R.S., Kurzyński, M., Trajdos, P. et al. (eds.) Progress in Image Processing, Pattern Recognition and Communication Systems. pp. 61–71. Springer International Publishing, Cham (2022)
  * [19] Yu, S., Ma, N.: Quantum Neural Network and Its Application in Vehicle Classification. In: 2008 Fourth International Conference on Natural Computation. vol. 2, pp. 499–503 (2008)
  * [20] Yumin, D., Wu, M., Zhang, J.: Recognition of Pneumonia Image Based on Improved Quantum Neural Network. IEEE Access 8, 224500–224512 (2020)
  * [21] Zhao, C., Gao, X.S.: QDNN: deep neural networks with quantum layers. Quantum Machine Intelligence 3(1), 15 (Apr 2021), https://doi.org/10.1007/s42484-021-00046-w
  * [22] Zhou, J., Gan, Q., Krzyżak, A., Suen, C.Y.: Recognition of handwritten numerals by Quantum Neural Network with fuzzy features. International Journal on Document Analysis and Recognition 2(1), 30–36 (Jul 1999), https://doi.org/10.1007/s100320050034