[1,2]Rodrigo Coelho

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Department of Informatics, University of Minho, Braga, Portugal

2]HASLab, INESC TEC, Braga, Portugal

3]International Nanotechnology Laboratory (INL), Braga, Portugal

# VQC-Based Reinforcement Learning with Data Re-uploading: Performance and
Trainability

<EMAIL_ADDRESS>André Sequeira<EMAIL_ADDRESS>Luís
Paulo Santos<EMAIL_ADDRESS>*  [  [

###### Abstract

Reinforcement Learning (RL) consists of designing agents that make intelligent
decisions without human supervision. When used alongside function
approximators such as Neural Networks (NNs), RL is capable of solving
extremely complex problems. Deep Q-Learning, a RL algorithm that uses Deep
NNs, achieved super-human performance in some specific tasks. Nonetheless, it
is also possible to use Variational Quantum Circuits (VQCs) as function
approximators in RL algorithms. This work empirically studies the performance
and trainability of such VQC-based Deep Q-Learning models in classic control
benchmark environments. More specifically, we research how data re-uploading
affects both these metrics. We show that the magnitude and the variance of the
gradients of these models remain substantial throughout training due to the
moving targets of Deep Q-Learning. Moreover, we empirically show that
increasing the number of qubits does not lead to an exponential vanishing
behavior of the magnitude and variance of the gradients for a PQC
approximating a 2-design, unlike what was expected due to the Barren Plateau
Phenomenon. This hints at the possibility of VQCs being specially adequate for
being used as function approximators in such a context.

###### keywords:

Reinforcement Learning, Quantum Computing, Neural Networks, Variational
Quantum Circuits, Quantum Machine Learning

## Statements and Declarations

### Conflict of interests

The authors declare no competing interests.

##  1 Introduction

###  1.1 Context

With quantum computers still in the _Noisy Intermediate Scale Quantum (NISQ)_
era, algorithms with theoretical complexity advantages over the best-known
classical algorithms, such as Shor’s algorithm to factor large integers [1],
can not be implemented to solve practical problems [2]. Thus, much of nowadays
quantum computing research focuses on finding near-term quantum advantages. A
field of quantum computing that has attracted considerable interest as one of
the most promising applications for achieving short-term quantum advantage is
_Quantum Machine Learning (QML)_. This is mainly due to _Variational Quantum
Circuits (VQCs)_ [3].

VQCs are quantum circuits that depend on free parameters, which are
iteratively updated by a classical optimizer to minimize an objective function
estimated from measurements [4]. VQCs are especially suitable for NISQ
devices, given their lower demands in the number of qubits and circuit depth,
which mitigates the effect of noise [3]. Consequently, integrating VQCs into
machine learning algorithms has become an intriguing field of study. VQCs used
in such a context have been extensively researched in both supervised [5, 6,
7, 8] and unsupervised [9, 10] QML. Quantum Reinforcement Learning, on the
other hand, remains largely unexplored.

A RL agent learns from interaction with its surrounding environment in a
feedback loop, relying solely on a reward signal to guide its actions. The
concept of learning from interaction forms the bedrock of nearly all theories
of learning and intelligence [11], with some even conjecturing that rewards
alone can pave the path to achieving general AI agents [12]. RL techniques,
with the advent of deep learning [13], consecrated a major impact in tasks
like Chess [14], Go [15] and video-games [16], matching or exceeding the
performance levels of humans. It has also been applied to practical real-world
problems, such as autonomous driving [17], robotics [18] and finance [19].

Just like VQCs have found applications as function approximators in both
supervised and unsupervised ML domains, they can be effectively utilized in
the same capacity in RL scenarios, serving as alternatives to DNNs. These
quantum variational approaches in the context of RL have started gaining
traction only recently, as evidenced by several recent studies [20, 21, 22,
23, 24], which are further discussed on the next subsection.

###  1.2 Related Work

It was recently shown that the data encoding strategy used in the quantum
circuit influences the expressive power of the quantum model [25].
Specifically, the authors demonstrate that a VQC may be written as a Partial
Fourier Series, where the accessible data frequencies are determined by the
eigenvalues of the data-encoding Hamiltonians. Furthermore, quantum models can
access increasingly rich frequency spectra by repeating simple data encoding
gates multiple times either in series or in parallel, a technique named _Data
Re-uploading_ [26].

VQCs have been used as function approximators in two different RL algorithms:
Deep Q-learning [20, 21, 22] and Reinforce with baseline [23, 24]. For the
VQC-based Reinforce algorithms, [23] employed a VQC with data re-uploading and
assessed the performance of models in benchmark environments. Conversely, [24]
used a simpler, hardware-efficient ansatz without data re-uploading, comparing
its performance and trainability against classical DNNs. Intriguingly, the
impact of data re-uploading on model trainability remains unexplored. Given
that data re-uploading increases the VQC’s circuit depth and expressivity, it
could have negative effects in the trainability, as suggested by [27, 28].

The picture is even less clear when it comes to VQC-based Deep Q-Learning
algorithms. Both [20] and [21] used simple VQCs without data re-uploading,
centering their studies on performance in benchmark environments and
comparisons with DNNs. On the other hand, [22] integrated data re-uploading
and analyzed its impact on performance. Yet, to the best of our knowledge, no
research has been done on the trainability of VQC-based Deep Q-Learning
models.

This paper aims to directly address these questions. Specifically, we seek to
investigate the impact of data re-uploading on both the performance and
trainability of VQC-based Deep Q-Learning models. Our approach involves
empirically analyzing the average return of two different hardware-efficient
VQCs in benchmark environments and the behavior of their gradient’s norm and
variance. Additionaly, we also verify how these metrics are affected as the
number of qubits increases.

###  1.3 Contributions

The contributions of this paper are organized as follows:

  * •

Empirically verified the importance of data re-uploading and trainable output
and input scaling in enhancing the performance of VQC-based Deep Q-learning
models in OpenAI Gym’s CartPole-v0 and Acrobot-v1.

  * •

Achieved considerable performance in the Acrobot-v1 environment, which was
previously untapped for VQC-based Deep Q-learning models.

  * •

Empirically verified that the gradient’s variance of the quantum models does
not vanish exponentially in the aforementioned environments. Furthermore, we
verified that the gradient’s variance increases when data re-uploading is
used.

  * •

Empirically identified a correlation between the moving targets inherent to
Deep Q-learning [16], gradient magnitude and variance, and performance.

###  1.4 Outline

The remainder of the manuscript is organized as follows. Section 2 introduces
the mathematical framework behind most RL problems, the Q-Learning algorithm
and the result of its combination with Deep Neural Networks. Section 3 then
introduces the main building blocks of Variational Quantum Circuits and how
these can be used as function approximators in Deep Q-Learning. Section 4
proposes the methodologies for analyzing performance and trainability and
applies them to VQC-based Deep Q-Learning models in benchmark environments.
Finally, 5 finishes the paper with some final remarks and suggestions for
future work.

##  2 Reinforcement Learning

###  2.1 Introduction

Any RL problem has two primary constituents: the Agent and the Environment.
The agent is the decision-maker and the representation of that which learns.
The environment is everything that surrounds the agent and is affected by its
decisions. The agent must learn which actions to take by interacting with the
environment and receiving a reward that indicates the goodness of that
particular action concerning the goal, see Figure 1.

Figure 1: The Agent-Environment Interface: The agent interacts with the
environment at time step $t$ by taking action $A_{t}$. The environment then
changes to state $S_{t+1}$ and produces reward $R_{t+1}$, which are both
passed back to the agent so that it can decide the next action. The dotted
lines indicate that this process repeats itself. Inspired by [11].

The mathematical framework behind this interface and most RL problems is known
as a _Markov Decision Process (MDP)_ [11], which is expressed as a tuple
$\langle\mathcal{S},\mathcal{A},\mathcal{P},R\rangle$ [11], encompassing a
finite set of states $\mathcal{S}$, actions $\mathcal{A}$, a state transition
probability matrix $\mathcal{P}$, and the reward function $R$.

A crucial metric in RL is the _Return_ , that is, the cumulative discounted
reward starting from time-step $t$, such that:

$G_{t}=R_{t+1}+\gamma
R_{t+2}+\gamma^{2}R_{t+3}+...=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}$ (1)

where $\gamma\in[0,1]$ is the _discount factor_ , which determines how much
long-term rewards are valued over short-term rewards. The behavior of the
agent is given by the _policy_ $\pi(a|s)$, which is a probability distribution
over actions given states. It determines the probability the agent will take
each action $a$ from a particular state $s$. The objective of a RL agent
becomes clear: to identify the policy that maximizes the expected return
$\max_{\pi}\mathbb{E}_{\pi}[G_{t}]$. This policy is typically referred to as
the _optimal policy_. It is known that, for any MDP, there exists at least one
deterministic optimal policy $\pi_{*}$ that is better than or equal to all
other policies, $\pi_{*}\geq\pi,\forall\pi$ [11]. Consequently, the goal is
for the policy of the agent to converge to this deterministic optimal policy
as training goes on.

There are several methods for finding a (near-) optimal policy. We first
distinguish between _model-based_ and _model-free_ methods [11, 29]. In model-
based methods, the agent has access to (or learns an approximated version of)
the model and can use it to plan. In model-free methods, the agent doesn’t
have access to a model. These methods are simpler and easier to implement, but
also usually suffer from a higher sample complexity, since the agent has to
deal with the extra burden of not knowing the dynamics of the environment.

Then, within the model-free methods, we need to distinguish between _value-
based_ and _policy-based_ methods. The former use value-functions as guides in
the search for optimal policies [11], such as Q-learning [30]. Policy-based
methods learn the policy directly. They typically work by parameterizing the
policy $\pi_{\theta}(a|s)$, defining an objective function that serves as some
performance measure $J(\pi_{\theta})$ and optimizing the parameters $\theta$
by performing gradient ascent on an approximation of the gradient $\nabla
J(\pi_{\theta})$, which is obtained using Monte-Carlo samples. Consequently,
they are commonly referred to as _policy gradient_ methods. The applicability
of both value-based and policy-based methods vastly depends on the problem
itself.

###  2.2 Q-learning

The _Action-Value Function_ $q_{\pi}(s,a)$ defines how good taking a certain
action from a particular state is. It is defined as:

$q_{\pi}(s,a)=\mathbb{E}_{\pi}[G_{t}|S_{t}=s,A_{t}=a]$ (2)

Furthermore, in any MDP, there exists an optimal action-value function
$q_{*}(s,a)$, that corresponds to the maximum action-value function over all
policies

$q_{*}(s,a)=\max_{\pi}q_{\pi}(s,a)$ (3)

This function is achieved by all the optimal policies [11]. Actually, once the
optimal action-value function is known, the optimal policy can be found by
maximizing over it:

$\pi_{*}(a|s)=\begin{cases}1\;\;\text{if
}a=\arg\max_{a\in\mathcal{A}}q_{*}(s,a)\\\ 0\;\;\text{otherwise}\end{cases}$
(4)

Q-Learning is a model-free value-based algorithm that approximates the optimal
action-value function and then, according to Equation 4, a (near) optimal
policy [30].

Being model free, Q-Learning agents learn from experience - finite sequences
of states, actions, and rewards, in the format $\\{s,a,r,s^{\prime}\\}$, which
are called _trajectories_ [11], or _episodes_ if the environment is episodic,
which will be the case in this paper. To generate these episodes, some
behavior policy has to be followed. Choosing this policy implies taking into
consideration the _exploration-exploitation tradeoff_. On the one hand, the
agent has to explore the state space to find high-value states. On the other
hand, the agent has to exploit the acquired knowledge, taking actions that it
knows yield high rewards and lead to good states. As a result, typically, an
_$\epsilon$ -greedy policy_, which chooses a random action with probability
$\epsilon$ and the greedy action with probability $1-\epsilon$, is used [11].
It is usual to decay $\epsilon$ over time.

To approximate the optimal $Q$-values, Q-Learning uses a method known as
_Temporal-Difference Learning (TD-Learning)_ \- it learns from raw experience
but updates estimates with other learning estimates without waiting for an
outcome - _bootstrapping_. More specifically, the action-value of a certain
state and a particular action is updated as such:

$Q(S,A)\leftarrow Q(S,A)+\alpha[R+\gamma\max_{a}Q(S^{\prime},a)-Q(S,A)]$ (5)

###  2.3 Deep Q-Learning

The previous subsection introduces Q-Learning, which uses lookup tables of (at
most) dimensions $\mathcal{S}\times\mathcal{A}$ to assign values to state-
action pairs encountered by the agent - _tabular regime_. Consequently, the
algorithm is inadequate for handling large and complex real-world problems,
since these tables become impossible to implement with current technology due
to memory and complexity constraints.

There are several possible solutions for this problem, such as using state-
abstraction [31] and/or dimensionality reduction techniques [32]. This work
focuses on the most popular solution - using _Function Approximation_. The
idea is to generalize from seen states to unseen states by approximating a
target function. Consequently, a new function is defined:

$\hat{q}(s,a,\textbf{w})\approx q_{\pi}(s,a)$ (6)

where w is a vector of parameters that must be updated using TD-Learning or
MC. It is important to refer that function approximation is only useful if
$dim(\textbf{w})\ll|\mathcal{S}\times\mathcal{A}|$, otherwise it would be the
same as the tabular regime.

There are many function approximators, each with pros and cons [33].
Nonetheless, the most used function approximators are Neural Networks (NNs),
particularly those with two or more hidden layers of neurons, known as Deep
Neural Networks (DNNs), due to their capacity to approximate intricate and
complex funtions [13]. When DNNs are used as function approximators for the
Q-function in a Q-Learning framework, the resultant algorithm is known as Deep
Q-Learning or Deep Q-Networks (DQN) [16]. This algorithm introduces two novel
improvements over previous approaches: _experience replay_ and _target
network_. The former consists of using a _Replay Buffer_ , where all
experiences - interactions with the environment in the format
$(s_{t},a_{t},r_{t+1},s_{t+1})$ \- are stored. Then, batches of these
experiences are randomly sampled to compute the parameter updates.
Consequently, temporal correlations between the samples are broken, making the
data appear more _Independent and Identically Distributed (IID)_ , which
stabilizes training. The _Target Network_ consists of a second NN that is used
to compute the TD-targets. The target network has ”frozen” weights that are
sporadically (every $C$ steps) updated with a copy of the parameters of the
online network. Thus, the targets are more stationary, which also stabilizes
training. However, there is a tradeoff here between speed of convergence and
stability which must be accounted for when choosing $C$.

Algorithm 1 Deep Q-Learning [16]

1:Algorithm parameters: step size $\alpha\in[0,1]$, small $\epsilon\in[0,1]$

2:Initialize replay buffer $\mathcal{D}$ to capacity $N$

3:Initialize action-value function $Q$ with random weights $\theta$

4:Initialize target action-value function $\hat{Q}$ with weights
$\theta^{-}=\theta$

5:for episode$=1,M$ do

6: Initialize $s_{1}$

7: for $t=1,T$ do

8: Choose $a_{t}$ from $s_{t}$ using $\epsilon$-greedy policy

9: Take action $a_{t}$, observe $r_{t+1}$, $s_{t+1}$

10: Store experience $(s_{t},a_{t},r_{t+1},s_{t+1})$ in replay memory
$\mathcal{D}$

11: Sample random mini-batch of transitions $(s,a,r,s^{\prime})$ from
$\mathcal{D}$

12: Compute the TD-targets using target network (with old, fixed parameters
$\theta^{-}$)

13: Perform a gradient descent step on
$\left[r+\gamma\max_{a^{\prime}}\hat{Q}(s^{\prime},a^{\prime};\theta_{i}^{-})-Q(s,a;\theta_{i})\right]^{2}$
w.r.t parameters $\theta$

14: Every $C$ steps update $\hat{Q}=Q$

15: Initialize $s_{t}$

16: end for

17:end for

##  3 Quantum Variational Deep Q-Learning

Quantum Variational Deep Q-learning, or VQC-based Deep Q-Learning, uses VQCs
as function approximators in a Q-Learning framework, instead of the typically
used DNNs. VQCs are quantum circuits that depend on free parameters, which are
iteratively updated by a classical optimizer to minimize an objective function
estimated from measurements [4]. In the following three subsections, we will
go over the main details of the VQCs used throughout this work.

###  3.1 Data Encoding

Classical data has to be encoded into a quantum state so that it can be
processed by a quantum computer. This paper uses _Angle Encoding_. According
to this technique, each component of the input data $x$ is encoded by a single
qubit using arbitrary Pauli rotations ${R_{X},R_{Y},R_{Z}}$, where the angle
is given by the component itself after some classical pre-processing, e.g.
normalization. To allow for adaptive frequency matching between the function
output by the VQC and the target function, each feature will be multiplied by
a classical trainable weight, a technique known as trainable input scaling
[26]. Thus, given the input data $x=\\{x_{0},x_{1},...,x_{n-1}\\}$, the
resulting quantum state is given by:

$\ket{x}=\bigotimes_{i=0}^{n-1}R_{\alpha}(\phi(x_{i}\times\lambda_{i}))\ket{0_{i}}$
(7)

where $R_{\alpha}\in\\{R_{X},R_{Y},R_{Z}\\}$, $\phi$ is some classical pre-
processing function and $\ket{0_{i}}$ is the $i^{\text{th}}$ qubit initialized
in state $\ket{0}$.

This technique has pros and cons. On the one hand, it allows the encoding of a
given input vector using circuits of depth $1$ [24]. On the other hand, the
number of qubits grows linearly with the number of features of the input
vector, which limits the dimensionality of the inputs one can encode due to
the reduced number of qubits in NISQ devices.

###  3.2 Ansatz

A set of problem-agnostic ansatzes called _Hardware-Efficient Ansatzes_ were
considered in this work. These allow for implementations with reduced circuit
depth by bringing correlated qubits together for depth-reduction [3]. For this
work, the ansatzes selected are those from Skolik’s [22] and the Universal
Quantum Classifier (UQC) [26], depicted in Figure 2. For simplicity, the
Skolik’s architecture will be referred to as _Skolik Data Re-Uploading_ if
data re-uploading is used and _Skolik Baseline_ otherwise.

@C=0.7em @R=.5em & R_x(x_1) R_y(θ_11) R_z(θ_12) 1 3
R_x(x_2) R_y(θ_21) R_z(θ_22) -1 1
R_x(x_3) R_y(θ_31) R_z(θ_32) -1 1
R_x(x_4) R_y(θ_41) R_z(θ_42) -1 -3 1242.7em–

(a)

@C=0.7em @R=.5em —0⟩ & U($\overrightarrow{\theta_{1}}$, $\overrightarrow{x}$)
⋯ U($\overrightarrow{\theta_{N}}$, $\overrightarrow{x)}$ 1212.7em– 1717.7em–

(b)

Figure 2: Subfigure 2(a): Skolik’s Architecture. When Data Re-Uploading is
used, the whole circuit is repeated in each layer. Otherwise, just the part
that is not surrounded by dashed lines. Subfigure 2(b): UQC Architecture. Each
processing layer $U$ is given by
$U^{UAT}(\overrightarrow{x};\overrightarrow{\omega},\alpha,\varphi)=R_{y}(2\varphi)R_{z}(2\overrightarrow{\omega}\cdot\overrightarrow{x}+2\alpha)$
and $\overrightarrow{\theta}_{i}=(\overrightarrow{\omega},\alpha,\varphi)$.
Although a single-qubit ansatz was shown for simplicity, this ansatz can be
generalized to allow multiple qubits.

However, there is a tradeoff one needs to take into account when using
hardware-efficient VQCs. Since these ansatzes take no inspiration from the
structure of the problem itself, they need to be highly expressive so they can
be applied to any generic task [34]. However, this expressiveness comes at the
cost of reduced trainability [28]. This problem, known as the _Barren-Plateau
Phenomenon_ , is explained in Subsection 3.4.

###  3.3 Cost Function and Gradient-Based Learning

To use a VQC to solve a problem, one has to first encode the problem into a
cost function, such that solving the problem corresponds to finding the global
optima of said function. Let $f_{\boldsymbol{\theta}}(x)$ be the expectation
value of some observable $\hat{O}$ as follows:

$f_{\boldsymbol{\theta}}(x)=\bra{0}U^{\dagger}(x,\boldsymbol{\theta})\hat{O}U(x,\boldsymbol{\theta})\ket{0}$
(8)

where $U(x,\boldsymbol{\theta})$ is a VQC that depends on the input $x$ and a
set of free parameters $\boldsymbol{\theta}$.

In the context of VQC-based Q-learning, considering an environment with $|A|$
possible actions, we need to measure the expectation values of $|A|$
observables, such that the Q-value of a particular action $a$ is given by:

$Q(s,a)=\bra{0^{\otimes
n}}U(s,{\boldsymbol{\theta}})^{\dagger}(s)O_{a}U(s,{\boldsymbol{\theta}})(s)\ket{0^{\otimes
n}}$ (9)

where $s$ is the state, $n$ the number of qubits and $O_{a}$ the observable
corresponding to action $a$.

The observables used throughout this work depend on the ansatz and number of
qubits, but always consist of single-qubit $\sigma_{z}$’s or tensor products
of $\sigma_{z}$’s applied to different qubits, similar to what has been
previously used in the literature [22].

Returning to the general scenario of a function $f_{\boldsymbol{\theta}}(x)$,
the cost function $L(\boldsymbol{\theta})$ is a function of the expectation
value itself. In many scenarios, $L(\boldsymbol{\theta})$ reduces to
$f_{\boldsymbol{\theta}}(x)$. However, in the context of machine learning,
classical post-processing is usually added to the expectation value. For
instance, consider the typical supervised learning scenario. Given a labeled
dataset $D=\\{(x_{i},y_{i})\\}^{M}$, the typical cost function is the Mean
Squared Error (MSE), which can be expressed as follows:

$L(\boldsymbol{\theta})=\frac{1}{M}\sum_{i=0}^{M-1}\left(f_{\boldsymbol{\theta}}(x_{i})-y_{i}\right)^{2}$
(10)

In practice, the value of $f_{\boldsymbol{\theta}}$ for a specific input $x$
is given by running the VQC multiple times and averaging over the results.
Then, typically, gradient-based methods are used to update the parameters of
the VQC. However, gradient-free methods also exist and have their advantages
and disadvantages, see [3].

When using gradient-based methods, as will be the case of this work, the
gradient of the cost function with respect to the parameters is calculated
using the Parameter-Shift Rules [35]. These rules state that the partial
derivative of $f_{\boldsymbol{\theta}}(x)$ w.r.t a single variational
parameter (assuming this parameter is the angle of a Pauli-rotation) is given
by:

$\frac{\partial
f_{\boldsymbol{\theta}}(x)}{\partial\theta_{i}}=\frac{1}{2}\left[f_{\boldsymbol{\theta}}(x;\theta_{i}+\pi/2)-f_{\boldsymbol{\theta}}(x;\theta_{i}-\pi/2)\right]$
(11)

To compute the partial derivative of the cost function w.r.t a single
parameter, two circuit executions are needed. Thus, to compute the gradient of
the cost function, which has $p$ parameters, $2\times p$ executions are
necessary. However, it is important to note that, in the numerical experiments
of this paper, quantum simulators and the adjoint differentiation techniques
will be used [36].

###  3.4 Barren Plateau Phenomenon

Let $L(\boldsymbol{\theta})$ be a hardware-efficient VQC cost-function. The
gradient of this function is given by:

$\nabla_{\boldsymbol{\theta}}L(\boldsymbol{\theta})=\left\\{\frac{\partial
L}{\partial\theta_{1}},\frac{\partial
L}{\partial\theta_{2}},...,\frac{\partial
L}{\partial\theta_{k}}\right\\},\;\boldsymbol{\theta}\in\mathbb{R}^{k}$ (12)

where $\boldsymbol{\theta}$ are the cost function’s parameters and $k$ the
number of parameters.

The Barren Plateau Phenomenon asserts that

$Var\left(\frac{\partial
L(\boldsymbol{\theta})}{\partial\theta_{k}}\right)\in\mathcal{O}\left(\frac{1}{\alpha^{n}}\right),\;\alpha>1$
(13)

where $n$ is the number of qubits.

Put in words, the variance of the gradient of hardware-efficient VQC-based
cost functions decays exponentially with the number of qubits [27].
Furthermore, [28] demonstrated that the more expressive the ansatz, the lower
the variance in the cost gradient. Consequently, the cost landscape is
flatter, making these circuits extremely hard to train. Nonetheless, it is
important to emphasize that [37] observed that the Barren Plateau Phenomenon
highly depends on the locality of the observable employed. To solve this
issue, several strategies have been proposed [38]. Nonetheless, the problem
severely limits the applicability of hybrid classical/quantum algorithms to
solve practical problems, since it erases a potential quantum advantage with a
VQC [3].

###  3.5 Data Re-Uploading

The authors of [25] determined that the data encoding strategy used in the
quantum circuit influences the expressive power of the quantum model.
Specifically, they show that a VQC may be written as a Partial Fourier Series,
where the accessible data frequencies are determined by the eigenvalues of the
data-encoding Hamiltonians. Furthermore, quantum models can access
increasingly rich frequency spectra by repeating simple data encoding gates
multiple times either in series or in parallel, a technique called _Data Re-
uploading_ [26].

To exemplify the power of data re-uploading, the authors showed that a quantum
model employing a single Pauli-rotation encoding can only learn a sine
function of the input. In other words, if the encoding is done just once, the
VQC can only approximate functions with a single non-zero frequency in the
frequency spectrum. However, repeating Pauli encoding linearly extends the
frequency spectrum, such that, if the encoding is repeated $n$ times, then the
quantum model can approximate functions with up to $n$ different non-zero
frequencies.

###  3.6 VQC-based Deep Q-Learning

Just like NNs can be used to approximate either a value-function or a
parameterized policy, so can VQCs. The result is a hybrid classical-quantum
algorithm that generally works as follows. The agent observes some state
$s_{t}$ and applies some classical pre-processing $\phi$. Then, the result
$\phi(s)$ is encoded into a VQC $U_{\boldsymbol{\theta}}(\phi(s))$ using some
data encoding technique. The VQC, with the current parameters
$\boldsymbol{\theta}_{t}$, prepares a quantum state. An observable $O_{a}$ is
measured for each possible action. The expectation values of these observables
$\expectationvalue{O_{a}}_{s,\boldsymbol{\theta}}$ are post-processed and the
result represents either the Q-values $Q_{\boldsymbol{\theta}}(s,a)$ in value-
based methods or the policy $\pi_{\boldsymbol{\theta}}(a|s)$ in policy-based
methods. Then, the agent chooses an action $a_{t}$ using these predictions and
executes it in the environment. The reward $r_{t}$ and the consecutive state
$s_{t+1}$ are observed by the classical optimizer. Using the parameter-shift
rule, the gradients of the VQC w.r.t the parameters $\boldsymbol{\theta}_{t}$
are calculated. Finally, the classical optimizer determines the new parameters
$\boldsymbol{\theta}_{t+1}$ [39].

In VQC-based Deep Q-Learning, in order for the Q-values estimated by the
quantum circuit to match the optimal Q-values in the environment, the
expectation values of the observables will be multiplied by a classical
trainable weight, a technique known as _Trainable Output Scaling_ [22].

To the best of our knowledge, [20] was the first paper to use VQCs as function
approximators in RL algorithms. Specifically, the authors used VQCs to
approximate the Q-function in a Deep Q-Learning algorithm. The pseudo-code for
the algorithm can be seen in Algorithm 2.

Algorithm 2 Variational Quantum Deep Q-Learning

1:Algorithm parameters: step size $\alpha\in[0,1]$, small $\epsilon\in[0,1]$

2:Initialize replay buffer $\mathcal{D}$ to capacity $N$

3:Initialize action-value function $Q$ (quantum circuit) with random
parameters $\theta$

4:Initialize target action-value function $\hat{Q}$ (target quantum circuit)
with parameters $\theta^{-}=\theta$

5:for episode$=1,M$ do

6: Initialize $s_{1}$ and encode into the quantum state

7: for $t=1,T$ do

8: With probability $\epsilon$ select a random action $a_{t}$, otherwise
select $a_{t}=\max_{a}Q^{*}(s_{t},a;\theta)$ from the output of the quantum
circuit

9: Execute action $a_{t}$ in emulator and observe $r_{t+1}$, $s_{t+1}$

10: Store experience $(s_{t},a_{t},r_{t+1},s_{t+1})$ in replay memory
$\mathcal{D}$

11: Sample random mini-batch of transitions $(s,a,r,s^{\prime})$ from
$\mathcal{D}$

12: Compute the TD-targets using target quantum circuit (with old, fixed
parameters $\theta^{-}$)

13: Perform a gradient descent step on
$\left[r+\gamma\max_{a^{\prime}}\hat{Q}(s^{\prime},a^{\prime};\theta_{i}^{-})-Q(s,a;\theta_{i})\right]^{2}$
w.r.t parameters $\theta$

14: Every $C$ steps update $\hat{Q}=Q$

15: Initialize $s_{t}$ and encode into the quantum state

16: end for

17:end for

##  4 Numerical Results

###  4.1 Methodology for Analysing Performance and Trainability

In any RL problem, the goal is for the agent to find the optimal or a near-
optimal policy, which translates to the behaviour that accumulates the most
rewards. Consequently, performance is typically measured by the return models
achieve in benchmark environments.

To analyze performance, we use the following methodology: $N$ agents are
randomly initialized according to the model. These agents are then trained in
the environment over $M$ episodes. Should an agent solve the environment
before completing the $M$ episodes, its training ceases (i.e., no further
updates to the model’s parameters), but the agent continues interacting with
the environment using its learned knowledge until all episodes conclude. The
returns from each episode are collected for all agents. Ultimately, we average
the returns across all $N$ agents for each episode, compute the standard
deviation, and plot the results. The best-performing model will be the one
that achieves the highest average return in less episodes.

To analyse trainability, the following methodology is used: when analysing a
model’s performance, we also store the gradient of the cost function with
respect to the parameters at each training step, which is every step where the
parameters are updated. Subsequently, we compute the norm of these gradients
for every training step. To mitigate the effects of stochasticity, we
calculate the average norm of the gradients across the $N$ agents and the
variance of these norms at each training step. However, since agents cease
training once the environment is solved, different agents have a different
number of training steps. As a result, we limited our procedure to only
encompass the steps up until the first agent successfully solved the
environment. To facilitate the visualization of the results, a rolling average
of the last $100$ training steps is used for both the norm of the gradients
and the variance of the norms.

###  4.2 Environments

OpenAI Gym, an open-source Python library, offers a standardized API that
bridges learning algorithms with a standard set of environments, providing a
platform to analyse and compare RL algorithms. Two specific environments were
chosen: CartPole-v0 and Acrobot-V1. The preference for these is twofold:
firstly, they are established benchmark environments that have been frequently
adopted in academic research for algorithm testing and comparison. Secondly,
these environments strike a balance in complexity; they present sufficient
challenges to test the robustness of VQC-based algorithms, yet remain
tractable for experimental purposes. While CartPole has already been the
subject of investigation in several studies, Acrobot presents a heightened
degree of complexity and, intriguingly, hasn’t yet been solved using VQC-based
Q-Learning.

CartPole consists of a cart that moves on a frictionless track with a pendulum
placed upright. The goal is to balance the pole by applying forces in the left
and right direction on the cart. The state-space is composed of $4$ features
that include the cart’s position and velocity and the pole’s angle and angular
velocity. The reward function is very simple: the agent receives a $+1$ reward
per time-step and there are a maximum of $200$ time-steps.

Acrobot consists of two links that are linearly connected to form a chain,
with one end of the chain fixed. The joint between the two links is the one we
can act upon and the goal is to apply torques ($-1$,$0$ or $1$) to this joint
to swing the free end of the chain above a given height while starting from
the initial position of hanging downwards. The state-space is more complex
than Cartpole’s and consists of $6$ features. Nonetheless, we employed a
trivial technique to reduce it to $4$. Every step that does not reach the goal
receives a reward of $-1$ and achieving it results in termination with a
reward of $0$. Moreover, the maximum number of steps in an episode is $500$,
after which the episode also terminates.

###  4.3 Data Re-Uploading’s Effect on Performance

To illustrate the impact of data re-uploading and trainable input and output
scaling, we analyzed the performance of models composed of all the possible
combinations of these characteristics, see Figure 3.

(a)

(b)

Figure 3: Performance of Baseline Models (on the left) and Data Re-Uploading
models (on the right) in the CartPole-v0 environment (see Subfigure 3(a)) and
the Acrobot-v1 environment (see Subfigure 3(b)) with and without trainable
input and/or output scaling. The returns are averaged over $10$ agents. The
full set of hyperparameters can be seen in Table 3.

Figures 3(a) and 3(b) show the considerable effects of data re-uploading and
trainable input and output scaling on the performance of VQC-based Deep
Q-Learning agents in the CartPole-v0 and Acrobot-v1 environments. Agents that
do not use trainable output scaling perform very poorly, no better than random
guessing, demonstrating the importance of matching the range of the Q-values
estimated by the quantum circuit to the range of the optimal Q-values of the
specific environment. While it is possible to achieve a considerable level of
performance by multiplying the expectation values by a fixed value instead of
using a classical trainable weight, [22] show that such a technique depends on
the speed of convergence of the models and does not achieve the same level of
performance as models that use trainable output scaling.

Moreover, the impact of trainable input scaling is noticeable, considering
that models that use it consistently outperform models that do not. This
result emphasizes the importance of allowing for an adaptive frequency
matching between the function the VQC outputs and the one it attempts to
approximate [26].

Finally, most models that use data re-uploading outperform those that do not
in both environments, which reaffirms the results obtained by [22],
demonstrating the importance of such a technique in increasing the
expressivity of the VQC and, consequently, allowing for a better approximation
of the optimal Q-function. However, this is not the case for the models
without trainable input scaling in the Acrobot-v1 environment, which is
surprising. This result may possibly be explained by the statistical variance
inherent to our methodology for analysing performance. Nonetheless, if the
trainability of data re-uploading models is considerably reduced, then the
applicability of data re-uploading might be severely limited. Hence, the next
section analyses the trainability of these models.

###  4.4 Data Re-Uploading’s Effect on Trainability

In this section, we turn our attentions to the trainability of the models from
Figures 3(a) and 3(b). Since the models without trainable output scaling
achieved a very poor level of performance, they were excluded from this
analysis. The results can be seen in Figures 4(a) and 4(b).

(a)

(b)

Figure 4: Trainability of the models from Figures 3(a) and 3(b) with trainable
output scaling in the CartPole-v0 (see Subfigure 4(a)) and Acrobot-v1 (see
Subfigure 3(b)) environments. In both Subfigures, the left graph represents
the gradient’s norm throughout training and the right graph the variance of
the norm.

The data re-uploading models are noteworthy in several ways. Despite their
greater circuit depth and enhanced expressivity, they exhibit the highest
variance in the gradient norm among all models. This promising finding is in
stark contrast to our initial expectations derived from [28], which suggests a
trade-off between expressivity and trainability. In particular, Barren
Plateaus are to be expected when using hardware-efficient ansatzes with high
expressivities. However, in this case, the most expressive models are also the
one with the highest gradients’ magnitude and variance. This is particularly
surprising in the Cartpole-v0 environment, since the difference in the
gradients’ norm and its variance between the data re-uploading model with
input scaling and all the other models is substantial. In the next section, we
study one of the possible causes for this behavior.

###  4.5 Trade-off Between Moving Targets and Gradient Magnitude

In this section, we attempt to comprehend why the norm and the variance of the
gradients exhibit such a surprising behavior, both in the sense that they
reach substantial maximum values and in the sense that more expressive models
achieve higher maximum values. For a start, it is crucial to understand that
Deep Q-Learning is fundamentally different from supervised learning, with the
primary reason being that Deep Q-Learning targets are _non-stationary_. Since
the targets keep changing during training due to the agent’s evolving
knowledge, predicting Q-values becomes increasingly challenging. This is
specially pronounced in the beginning of training, when the agent is focused
on exploring the state-space. The more states explored, the higher the
variance in the return and, consequently, the higher the loss.

Due to the instability that arose from the moving targets, the original paper
[16] introduced the concept of a target network. The targets are calculated
using this network that has frozen weights, which are updated every $C$ steps
to match the weights of the online network. Hence, for $C$ steps at a time,
the targets appear stationary. If $C$ is set too high, the targets move slowly
and the model takes longer to train. If $C$ is too low, then the targets
change frequently and the algorithm becomes unstable.

Figures 5(a) and 5(b) show how data re-uploading models perform for different
values of $C$ and the behavior of their loss functions in the CartPole-v0 and
Acrobot-v1 environments.

(a)

(b)

Figure 5: Performance (first graph) and the respective loss functions for
increasing values of $C$ of data re-uploading models in the Cartpole-v0 (see
Subfigure 5(a)) and Acrobot-v1 (see Subfigure 5(b)) environments. The full set
of hyperparameters can be seen in Table 4.

(a)

(b)

Figure 6: Gradients’ norm (above) and variance (below) for data re-uploading
models with increasing values of $C$ in the Cartpole-v0 (see Subfigure 4(a))
and Acrobot-v1 (see Subfigure 4(b)) environments.

As $C$ increases to substantial values, the models’ speed of convergence and
performance start to decrease, as expected. However, interestingly, really low
values of $C$ (even a value of $C=1$, which corresponds to a model that does
not use a target network), achieve a level of performance equal to or even
better than moderate values of $C$. In fact, after a search of
hyperparameters, [22] found that $C=1$ leads to the best performance. Turning
our attention to the loss functions, we see that, as $C$ increases, the loss
function becomes more stable and the maximum values it achieves start
decreasing. Moreover, the dynamics of the moving targets become more evident,
as one can see by the loss function for $C=2500$ in the Cartpole-v0
environment, where the peaks correspond to the step the targets change, and
the valleys in between them to the steps for which they remain stationary.

Interestingly, both the gradients’ norm and variance exhibit a similar
behavior to the loss functions, see Figure 6. This analysis is of utmost
importance for VQC-based Deep Q-Learning. In the previous sections, we
observed that well-performing models exhibit substantial gradient magnitudes
and variances throughout training. This section has empirically demonstrated
the influential role of moving targets and the target network update frequency
$C$ on the behavior of both the loss function and, consequently, the
gradients. In particular, we saw that increasing $C$ stabilizes the loss
function, which contributes to more controlled magnitudes of gradients and
their variance. However, it is important to highlight that even with
relatively low $C$ values — where the gradients’ magnitudes and variances were
pronounced — the models showcased impressive performance. They were capable of
achieving maximum returns in as few episodes as other more ”stable” models
with higher values of $C$.

This raises an interesting question. It is known that hardware-efficient VQCs
suffer from the Barren Plateau Phenomenon [27], such that the variance of the
gradients decays exponentially with system size. However, it appears that Deep
Q-Learning is an inherently unstable algorithm. Intriguingly, certain
hyperparameter configurations seem to allow Deep Q-Learning models to
effectively learn good policies while sustaining considerable gradient
magnitudes and variances. This hints at a potential advantage: might this
inherent instability help counteract the Barren Plateau Phenomenon? In other
words, could the utilization of highly expressive VQCs (such as data re-
uploading ones) in such a context be somewhat resistant to Barren Plateaus? To
verify these claims, it becomes imperative to research the gradient’s behavior
for an increasing number of qubits. The most appropriate architecture for such
a study is the UQC, since it allows the encoding of an input vector into an
arbitrary number of qubits, leading to a higher flexibility in the choice of
system size. Thus, in the next section we introduce this architecture and,
then, in section 4.7, we analyze how the gradients behave as the number of
qubits increases.

###  4.6 The Single and Multi-Qubit UQC

We verified in section 4.3 that the VQC used in [22] is capable of solving the
CartPole-v0 environment and achieving considerable performance in the
Acrobot-v1 environment. However, it would be interesting to see if the UQC
[26], a circuit that also makes use of data re-uploading and trainable input
scaling by default, is capable of solving these environments. An advantage of
this architecture is the fact that it allows the encoding of any input vector
using an arbitrary number of qubits. This raises some interesting questions:
Can a single-qubit UQC solve these environments? By increasing the number of
qubits, can we improve the performance of the models?

To start, a data encoding technique has to be defined. We experiment with two
different types of encoding:

  * •

Full Encoding: Using full encoding, the whole input vector is encoded into all
the qubits. Consequently, the number of parameters grows linearly with the
number of qubits.

  * •

Partial Encoding: In this data encoding technique, we divide the number of the
input vector features by the number of qubits used and encode a different
subvector in each qubit.

The reason for using these two types of encoding is two-fold. On the one hand,
these two techniques allow us to study the impact of introducing entanglement
on the performance of the models in the CartPole-v0 environment. On the other
hand, the Full Encoding technique allows us to encode a given input vector in
an arbitrary number of qubits that may even be greater than the number of
features of the input vector. Thus, we may study how the performance and
trainability behave as the number of qubits increases, see Section 4.7.
Figures 7(a) and 7(b) show how each of these data encoding methods perform
with and without entanglement for $2$ and $4$ qubits in the Cartpole-v0 and
Acrobot-v1 environments, see Subfigures 7(a) and 7(b), respectively.

(a)

(b)

Figure 7: Performance of UQC models with entanglement (on the left) and
without entanglement (on the right) for the CartPole-v0 (see Subfigure 7(a))
and the Acrobot-v1 (see Subfigure 7(b)) environments. The returns are averaged
over $10$ agents. The full set of hyperparameters can be seen in Table 6.

Models without entanglement, that is, models that can be efficiently simulated
using classical computers, are able of achieving considerable performance.
Moreover, while models with full encoding are capable of learning
(sub-)optimal policies, models that use partial encoding require entanglement,
such that the features of the input state become correlated, giving the VQC
the full information necessary to better approximate the optimal Q-values of
the two environments. Finally, even a single-qubit UQC is capable of solving
the Cartpole-v0 environment, but the Acrobot-v1 environment needs a VQC with
higher width to learn a decent policy.

###  4.7 Gradient Behavior for Increasing System Sizes

One of the advantages of the multi-qubit UQC architecture is that one may
encode any input vector into an arbitrary number of qubits. In particular, it
is possible to increase the number of qubits even further than the number of
features of the input vector. To test how the norm of the gradients and its
variance change as the number of qubits increases, we trained the full
encoding multi-qubit UQC model with the set of even numbers of qubits from $2$
to $12$ in the CartPole-v0 and Acrobot-v1 environments, see Figures 8(a) and
8(b), respectively.

(a)

(b)

Figure 8: Performance (first graph) and Trainability Analysis UQC models with
increasing numbers of qubits in the Cartpole-v0 (see Subfigure 8(a)) and
Acrobot-v1 (see Subfigure 8(b)) environments. The full set of hyperparameters
can be seen in Table 5.

Both the magnitude of the gradient and its variance behave similarly during
training for all models, while also achieving a similar range of values.
However, it is important to note that there seems to be a decrease in the
variance of the gradients for the Acrobot-v1 environment as the number of
qubits increases, but this seems to be most noticeable after $10000$ training
steps, after which most of the training should be over. These results are
particularly interesting because they confirm the suspicions that arose
throughout this work. The Barren Plateau Phenomenon states that the gradients
decay exponentially with system size. However, for the VQC-based Deep
Q-Learning algorithm, using the multi-qubit UQC model in the CartPole-v0 and
Acrobot-v1 environments with the MSE loss function, that doesn’t seem to be
the case. This hints at the possibility of hardware-efficient VQCs being
especially suitable to be used as function approximators in Deep Q-learning
since they might be resistant to the Barren Plateau Phenomenon, to some
extent, due to the moving targets’ effects on the gradients of this algorithm.

However, only circuits that approximate $2$-designs are susceptible to Barren
Plateaus [28]. To verify whether a circuit approximates a $2$-design, one can
use the Hilbert-Schmidt norm of $A$ defined as:

$A=\int_{Haar}(\ket{\psi}\bra{\psi})^{\otimes
2}d\psi-\int_{\theta}(\ket{\phi_{\theta}}\bra{\phi_{\theta}})^{\otimes
2}d\phi$ (14)

where the latter integral is taken over all states according to uniformly
sampling the parameters $\theta$ of our VQC, a methodology in line with the
one used in [40], although oversimplified. The lower the value of the norm of
$A$, the better it approximates a $2$-design. Table 1 shows values of the norm
of $A$ for different circuits. Although oversimplified, it shows that the
multi-qubit UQC indeed appears to approximate a $2$-design circuit.

Table 1: Values of the Hilbert-Schmidt norm of $A$ for different circuits Circuit | Value
---|---
Haar integral (sanity check) | 0.022
Multi-qubit UQC with $4$ qubits and $5$ layers | 0.067
$4$-qubits empty VQC | 0.99
|

##  5 Conclusion

The main goal of this work was to study the effects of data re-uploading on
the performance and trainability of VQC-based Deep Q-Learning models. It was
already empirically shown by [22] that using data re-uploading increases the
performance of these models in the CartPole environment, possibly due to its
increased expressivity, which allows the approximation of more intricate
functions [25]. Throughout this work, we implemented and tested different data
re-uploading models in the CartPole-v0 and Acrobot-v1 environments. Our
results match those by [22] and [25], re-affirming the effect of data re-
uploading on the performance of the models.

However, a concern would be that the increase in expressivity and circuit
depth that arises from the use of data re-uploading could decrease the
trainability of these models due to the Barren Plateau Phenomenon [27]. To
investigate such a concern, we analyzed the norm of the gradients and the
variance of this norm. From this analysis, we verified that the gradients
achieve substantial values and actually increase when data re-uploading is
used versus when it is not used. Furthermore, we empirically showed that this
increase is due to an instability that is inherent to Deep Q-Learning, which
is the fact that the targets are non-stationary. Alongside the Mean Squared
Error loss function, this leads to a very unstable algorithm with substantial
gradients. Moreover, we also verified that increasing the number of qubits of
the multi-qubit UQC in the CartPole-v0 and Acrobot-v1 environments does not
lead to a decrease in the gradients’ magnitude or variance.

##  6 Future Work

For future work, it would be interesting to develop a different methodology to
analyze the trainability of VQC-based models. The magnitude and variance of
the gradients already give some insights into the trainability of a model, but
there are other methodologies that could also be used. For instance, it may be
possible to study the eigenvalues of the Hessian Matrix.

Another interesting possibility would be to analyze these models from a
Fourier Analysis perspective. [25] show that VQCs may be seen as Partial
Fourier Series in the data and that using data re-uploading increases the
frequencies the VQC ”has access to”. Thus, it may be possible to analyze the
optimal Q-functions of certain environments and the Fourier Series that the
VQCs are approximating. This might reveal some insights into what is happening
behind the scenes.

Finally, while the results of this work hint at the possibility of the
inherent instability of Deep Q-Learning counteracting the effects of the
Barren Plateau Phenomenon, they are still insufficient for making such a
claim. Thus, it would be interesting to build upon this work and analyze the
trainability of VQC-based Deep Q-Learning models in different, more complex
environments, with different loss functions and sets of hyperparameters. It
might even be possible to theoretically derive bounds for the gradients of
these models, which could reinforce the empirical analysis.

## Data Availability

All data necessary for replicating the results of this paper is available
here.

## Acknowledgments

This work is financed by National Funds through the Portuguese funding agency,
FCT - Fundação para a Ciência e a Tecnologia, within project LA/P/0063/2020.
RC also thanks the support of the Foundation for Science and Technology (FCT,
Portugal) under grant 10053/BII-E_B4/2023. AS also thanks the support of the
Foundation for Science and Technology (FCT, Portugal) within grant
UI/BD/152698/2022 and project IBEX, with reference PTDC/CC1-COM/4280/2021.

## References

  *   * Shor [1999] Shor, P.W.: Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer. SIAM review 41(2), 303–332 (1999)
  * Preskill [2018] Preskill, J.: Quantum computing in the nisq era and beyond. Quantum 2, 79 (2018)
  * Cerezo et al. [2021] Cerezo, M., Arrasmith, A., Babbush, R., Benjamin, S.C., Endo, S., Fujii, K., McClean, J.R., Mitarai, K., Yuan, X., Cincio, L., et al.: Variational quantum algorithms. Nature Reviews Physics 3(9), 625–644 (2021)
  * Ostaszewski et al. [2021] Ostaszewski, M., Grant, E., Benedetti, M.: Structure optimization for parameterized quantum circuits. Quantum 5, 391 (2021)
  * Mitarai et al. [2018] Mitarai, K., Negoro, M., Kitagawa, M., Fujii, K.: Quantum circuit learning. Physical Review A 98(3), 032309 (2018)
  * Schuld and Killoran [2019] Schuld, M., Killoran, N.: Quantum machine learning in feature hilbert spaces. Physical review letters 122(4), 040504 (2019)
  * Schuld et al. [2020] Schuld, M., Bocharov, A., Svore, K.M., Wiebe, N.: Circuit-centric quantum classifiers. Physical Review A 101(3), 032308 (2020)
  * Farhi and Neven [2018] Farhi, E., Neven, H.: Classification with quantum neural networks on near term processors. arXiv preprint arXiv:1802.06002 (2018)
  * Coyle et al. [2020] Coyle, B., Mills, D., Danos, V., Kashefi, E.: The born supremacy: quantum advantage and training of an ising born machine. npj Quantum Information 6(1), 60 (2020)
  * Zoufal et al. [2021] Zoufal, C., Lucchi, A., Woerner, S.: Variational quantum boltzmann machines. Quantum Machine Intelligence 3, 1–15 (2021)
  * Sutton and Barto [2018] Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, ??? (2018)
  * Silver et al. [2021] Silver, D., Singh, S., Precup, D., Sutton, R.S.: Reward is enough. Artificial Intelligence 299, 103535 (2021)
  * LeCun et al. [2015] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444 (2015)
  * Silver et al. [2017] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al.: Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 (2017)
  * Silver et al. [2016] Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. nature 529(7587), 484–489 (2016)
  * Mnih et al. [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. nature 518(7540), 529–533 (2015)
  * Sallab et al. [2017] Sallab, A.E., Abdou, M., Perot, E., Yogamani, S.: Deep reinforcement learning framework for autonomous driving. arXiv preprint arXiv:1704.02532 (2017)
  * Kober et al. [2013] Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: A survey. The International Journal of Robotics Research 32(11), 1238–1274 (2013)
  * Hambly et al. [2023] Hambly, B., Xu, R., Yang, H.: Recent advances in reinforcement learning in finance. Mathematical Finance 33(3), 437–503 (2023)
  * Chen et al. [2020] Chen, S.Y.-C., Yang, C.-H.H., Qi, J., Chen, P.-Y., Ma, X., Goan, H.-S.: Variational quantum circuits for deep reinforcement learning. IEEE Access 8, 141007–141024 (2020)
  * Lockwood and Si [2020] Lockwood, O., Si, M.: Reinforcement learning with quantum variational circuit. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 16, pp. 245–251 (2020)
  * Skolik et al. [2022] Skolik, A., Jerbi, S., Dunjko, V.: Quantum agents in the gym: a variational quantum algorithm for deep q-learning. Quantum 6, 720 (2022)
  * Jerbi et al. [2021] Jerbi, S., Gyurik, C., Marshall, S., Briegel, H.J., Dunjko, V.: Variational quantum policies for reinforcement learning. arXiv preprint arXiv:2103.05577 (2021)
  * Sequeira et al. [2022] Sequeira, A., Santos, L.P., Barbosa, L.S.: Variational quantum policy gradients with an application to quantum control. arXiv preprint arXiv:2203.10591 (2022)
  * Schuld et al. [2021] Schuld, M., Sweke, R., Meyer, J.J.: Effect of data encoding on the expressive power of variational quantum-machine-learning models. Physical Review A 103(3), 032430 (2021)
  * Pérez-Salinas et al. [2020] Pérez-Salinas, A., Cervera-Lierta, A., Gil-Fuster, E., Latorre, J.I.: Data re-uploading for a universal quantum classifier. Quantum 4, 226 (2020)
  * McClean et al. [2018] McClean, J.R., Boixo, S., Smelyanskiy, V.N., Babbush, R., Neven, H.: Barren plateaus in quantum neural network training landscapes. Nature communications 9(1), 1–6 (2018)
  * Holmes et al. [2022] Holmes, Z., Sharma, K., Cerezo, M., Coles, P.J.: Connecting ansatz expressibility to gradient magnitudes and barren plateaus. PRX Quantum 3(1), 010313 (2022)
  * Kaelbling et al. [1996] Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of artificial intelligence research 4, 237–285 (1996)
  * Watkins and Dayan [1992] Watkins, C.J., Dayan, P.: Q-learning. Machine learning 8(3), 279–292 (1992)
  * Andre and Russell [2002] Andre, D., Russell, S.J.: State abstraction for programmable reinforcement learning agents. In: Aaai/iaai, pp. 119–125 (2002)
  * Sorzano et al. [2014] Sorzano, C.O.S., Vargas, J., Montano, A.P.: A survey of dimensionality reduction techniques. arXiv preprint arXiv:1403.2877 (2014)
  * Busoniu et al. [2017] Busoniu, L., Babuska, R., De Schutter, B., Ernst, D.: Reinforcement Learning and Dynamic Programming Using Function Approximators. CRC press, ??? (2017)
  * Bilkis et al. [2021] Bilkis, M., Cerezo, M., Verdon, G., Coles, P.J., Cincio, L.: A semi-agnostic ansatz with variable structure for quantum machine learning. arXiv preprint arXiv:2103.06712 (2021)
  * Schuld et al. [2019] Schuld, M., Bergholm, V., Gogolin, C., Izaac, J., Killoran, N.: Evaluating analytic gradients on quantum hardware. Physical Review A 99(3), 032331 (2019)
  * Jones and Gacon [2020] Jones, T., Gacon, J.: Efficient calculation of gradients in classical simulations of variational quantum algorithms. arXiv preprint arXiv:2009.02823 (2020)
  * Cerezo et al. [2021] Cerezo, M., Sone, A., Volkoff, T., Cincio, L., Coles, P.J.: Cost function dependent barren plateaus in shallow parametrized quantum circuits. Nature communications 12(1), 1791 (2021)
  * Grant et al. [2019] Grant, E., Wossnig, L., Ostaszewski, M., Benedetti, M.: An initialization strategy for addressing barren plateaus in parametrized quantum circuits. Quantum 3, 214 (2019)
  * Meyer et al. [2022] Meyer, N., Ufrecht, C., Periyasamy, M., Scherer, D.D., Plinge, A., Mutschler, C.: A survey on quantum reinforcement learning. arXiv preprint arXiv:2211.03464 (2022)
  * Sim et al. [2019] Sim, S., Johnson, P.D., Aspuru-Guzik, A.: Expressibility and entangling capability of parameterized quantum circuits for hybrid quantum-classical algorithms. Advanced Quantum Technologies 2(12), 1900070 (2019)

##  Appendix A Hyperparameters

Some of the hyperparameters for the VQC-based Deep Q-Learning models are
explained in the following table:

Table 2: An explanation of VQC-Based Deep Q-Learning’s hyperparameters Hyperparameter | Explanation
---|---
qubits (n) | The quantum circuit’s number of qubits
layers | The quantum circuit’s number of layers
$\gamma$ | The return’s discount factor
trainable input scaling | whether trainable input scaling is used or not
trainable output scaling | whether trainable output scaling is used or not
learning rate of parameters $\theta$ | the learning rate of parameters $\theta$
learning rate of input scaling parameters | the learning rate of input scaling parameters
Learning rate of output scaling parameters | the learning rate of output scaling parameters
batch size | the batch size
decaying schedule of $\epsilon$-greedy policy | defines the decaying schedule of the $\epsilon$-greedy policy (e.g exponential)
$\epsilon_{init}$ | the initial value of $\epsilon$
$\epsilon_{dec}$ | the decay rate of $epsilon$ per episode
$\epsilon_{min}$ | the minimum value of $\epsilon$
update model | the update model’s frequency
update target model | the target model’s update frequency
data re-uploading | whether data re-uploading is used or not
|

The set of hyperparameters used for the VQCs in Figures 3(a), 3(b), 4(a) and
4(b), can be seen in table 3.

Table 3: Models’ hyperpameters of Subsection 4.3. | CartPole-v0 | Acrobot-v1
---|---|---
qubits (n) | $4$ | $4$
layers | $5$ | $5$
$\gamma$ | $0.99$ | $0.99$
trainable input scaling | yes, no | yes, no
trainable output scaling | yes, no | yes, no
learning rate of parameters $\theta$ | $0.001$ | $0.001$
learning rate of input scaling parameters | $0.001$ | $0.001$
learning rate of output scaling parameters | $0.1$ | $0.1$
batch size | $16$ | $32$
decaying schedule of $\epsilon$-greedy policy | Exponential | Exponential
$\epsilon_{init}$ | $1$ | $1$
$\epsilon_{dec}$ | $0.99$ | $0.99$
$\epsilon_{min}$ | $0.01$ | $0.01$
update model | $1$ | $5$
update target model | $1$ | $250$
size of replay buffer | $10000$ | $50000$
data re-uploading | yes, no | yes, no
|  |

The set of hyperparameters used for the VQCs in Figures 5(a) and 5(b), can be
seen in table 4.

Table 4: Models’ hyperpameters of Subsection 4.5. | CartPole-v0 | Acrobot-v1
---|---|---
qubits (n) | $4$ | $4$
layers | $5$ | $5$
$\gamma$ | $0.99$ | $0.99$
trainable input scaling | yes | yes
trainable output scaling | yes | yes
learning rate of parameters $\theta$ | $0.001$ | $0.001$
learning rate of input scaling parameters | $0.001$ | $0.001$
learning rate of output scaling parameters | $0.1$ | $0.1$
batch size | $16$ | $32$
decaying schedule of $\epsilon$-greedy policy | Exponential | Exponential
$\epsilon_{init}$ | $1$ | $1$
$\epsilon_{dec}$ | $0.99$ | $0.99$
$\epsilon_{min}$ | $0.01$ | $0.01$
update model | $1$ | $5$
update target model | $1,500,1000,2500$ | $100,1000,2500,5000$
size of replay buffer | $10000$ | $50000$
data re-uploading | yes | yes
|  |

The set of hyperparameters used for the VQCs in Figures 6(a) and 6(b), can be
seen in table 5.

Table 5: Models’ hyperpameters of Subsection 4.7. | CartPole-v0 | Acrobot-v1
---|---|---
qubits (n) | $2,4,6,8,10,12$ | $2,4,6,8,10,12$
layers | $5$ | $5$
$\gamma$ | $0.99$ | $0.99$
trainable input scaling | yes | yes
trainable output scaling | yes | yes
learning rate of parameters $\theta$ | $0.001$ | $0.001$
learning rate of input scaling parameters | $0.001$ | $0.001$
learning rate of output scaling parameters | $0.1$ | $0.1$
batch size | $16$ | $32$
decaying schedule of $\epsilon$-greedy policy | Exponential | Exponential
$\epsilon_{init}$ | $1$ | $1$
$\epsilon_{dec}$ | $0.99$ | $0.99$
$\epsilon_{min}$ | $0.01$ | $0.01$
update model | $1$ | $5$
update target model | $1,500,1000,2500$ | $100,1000,2500,5000$
size of replay buffer | $10000$ | $50000$
data re-uploading | yes | yes
|  |

The set of hyperparameters used for the VQCs in Figures 7(a) and 7(b), can be
seen in table 6.

Table 6: Models’ hyperpameters of Subsection 4.6 | CartPole-v0 | Acrobot-v1
---|---|---
qubits (n) | $1,2,4$ | $1,2,4$
layers | $5$ | $5$
$\gamma$ | $0.99$ | $0.99$
trainable input scaling | yes | yes
trainable output scaling | yes | yes
learning rate of parameters $\theta$ | $0.001$ | $0.001$
learning rate of input scaling parameters | $0.001$ | $0.001$
learning rate of output scaling parameters | $0.1$ | $0.1$
batch size | $16$ | $32$
decaying schedule of $\epsilon$-greedy policy | Exponential | Exponential
$\epsilon_{init}$ | $1$ | $1$
$\epsilon_{dec}$ | $0.99$ | $0.99$
$\epsilon_{min}$ | $0.01$ | $0.01$
update model | $1$ | $5$
update target model | $1,500,1000,2500$ | $100,1000,2500,5000$
size of replay buffer | $10000$ | $50000$
data re-uploading | yes | yes
|  |