# 3D-EDM: Early Detection Model for
3D-Printer Faults

Harim Jeong  Department of Cultural Management
Sungkyunkwan University
Seoul, South Korea
<EMAIL_ADDRESS>Joo Hun Yoo  Department of Artificial Intelligence
Sungkyunkwan University
Suwon, South Korea
<EMAIL_ADDRESS>
###### Abstract

With the advent of 3D printers in different price ranges and sizes, they are
no longer just for professionals. However, it is still challenging to use a 3D
printer perfectly. Especially, in the case of the Fused Deposition Method, it
is very difficult to perform with accurate calibration. Previous studies have
suggested that these problems can be detected using sensor data and image data
with machine learning methods. However, there are difficulties to apply the
proposed method due to extra installation of additional sensors. Considering
actual use in the future, we focus on generating the lightweight early
detection model with easily collectable data. Proposed early detection model
through Convolutional Neural Network shows significant fault classification
accuracy with 96.72% for the binary classification task, and 93.38% for multi-
classification task respectively. By this research, we hope that general users
of 3D printers can use the printer accurately.

###### Index Terms:

3D Printer, Fault Detection, Image Deep Learning, Convolutional Neural Network

##  I Introduction

3D printing technology, which began in the early 2000s, continues to develop
at a rapid pace. Since its emergence, the technology has been extended to
various industries. Initially, it was limited for professional usages, but
recently, inexpensive printers for hobbies have been appeared, and the market
has grown to general-purpose. For those who use 3D printer as a hobby, Fused
Deposition Modeling (FDM) type printers are mainly used rather than
Stereolithography (SLA) or Digital light processing (DLP)type printers.

To perform the FDM type of 3D printer, the calibration process is required
depending on the surrounding temperature, bed types, size of the nozzles, and
type of filaments. Compared to a resin printer that uses only a z-axis motor
and display, the FDM method contains at least four stepper motors, one for
each x, y, z-axis, and extruder. In addition, rails, belts, nozzles, and fans
are making the structure more complex. It is challenging for the users to
perform and maintain perfect calibration in such environments.

Faults that often occur on the 3D printer are typically layer shifts, strings,
warping, and under extrusion as shown in Fig.1. Printing each 3-dimensional
object takes a long time, and it is hardly possible to watch and modify the
settings. Therefore, even if a significant defect occurs, it is difficult to
immediately stop and re-calibrate. This is the reason why real-time fault
detection or early detection is necessary for 3D printing.

In this study, the 3D printer Early Detection Model (3D-EDM) is proposed as
lightweight and high performance, using the image deep learning techniques.

Figure 1: Sample images of 3D printer faults

##  II Fault Detection in 3D Printer

In order to reduce the time and economic waste of 3D printers, fault detection
has been applied in various ways. Many researchers have attempted to obtain
information on defects by analyzing the various dataset collected from
printers.

Banadaki suggests a fault detection method through additive manufacturing with
the speed and temperature of the extruder [1]. However, since the data
available from the printer itself is limited, several studies conducted
experiments using additional sensors such as vibrarion sensor. Bing proposed a
real-time detection solution with Support Vector Machine (SVM) classifier in
[2]. With additional vibration sensor, they effectively detect the failures of
3D printers.

In addition to the method using the sensor data, there are also studies that
detects failures of the 3D printer with the printing photos collected through
a camera. In the case of Delli’s research [3], the RGB value of each critical
checkpoint of the object model are evaluated. They monitor whether the output
was normally performed by comparing the RGB values of the photo taken at the
checkpoint during actual printing process. Kadam, on the other hand, suggests
a detection method through machine learning algorithms on the first layer’s
top image [4]. Referring the importance of the first layer, they compared the
classification results from following pre-trained models; EfficientnetB0,
Resnet18, Resnet50, Alexnet, and Googlenet. Finally, Jin proved that by
attaching a camera right next to the nozzle to recognize the nozzle edge, the
CNN-based classifiers could be used to classify whether printing is performed
correctly in real-time [5].

These previous studies have proven that various 3D printing failures can be
diagnosed, detected, and predicted. Still, there may be difficulties in actual
use. A camera must be mounted on the nozzle head to see the top view of the
printing, but increasing the total weight by mounting a camera on a nozzle can
cause another failure. Based on the related studies, our paper aims to conduct
research focusing on practical use.

##  III Early Detection Model

###  III-A Dataset

In this research, training images were crawled from Google and YouTube to
collect massive real-world data. The data collection using crawling was to
cover various printing environments, including backgrounds, filament colors,
and the shape of objects.

To detect the incorrect output and confirm the type of faults, 200 images were
collected for each layer shift, strings, under extrusion, and warping cases.
About 500 normally printed data were also gathered. Accordingly, a total of
about 1.3k image data was selected for the experiment.

###  III-B Experiment Settings

To perform an image deep learning for the fault detection, we preprocess
collected datasets with various data transformation tools. Since the collected
data all have different sizes, it was set to focus on the object using the
resize and centercrop functions. The vertical flip and horizontal flip
functions were also applied to train the model to effectively recognize the
images taken from a different angle.

Training model that can detect defects based on the classifier, the
Convolutional Neural Network (CNN) model was adopted. There are already fine
pre-trained image classification models, our experiments were focused to
generate lighter models. After generating a training model with 10
convolutional layers, we tried to find an optimization model in the form of
removing one layer at a time. Experiment was conducted only when the test
performance of the model exceeded 90.0%. Final model is shown in Fig. 2.

Figure 2: Sample images of 3D printer faults

The size of 256 x 256 preprocessed image data is used as an input. It passes
through a total of five convolutional layers with attached pooling layers to
train the significant features from each part of the images. After that, the
training ends with two or four output nodes through one fully-connected layer.
As mentioned above, setting two output nodes determines whether the input
value is the fault result. Setting four output nodes tells the user in detail
what issues the incorrect result contains respectively.

###  III-C Experiment Results

Since no other studies were experimented from various angles for all possible
failure cases as in this study, a performance comparison was made with studies
with high test accuracy or similar experimental processes. Table 1 contains
the accuracy comparison results.

TABLE I: Accuracy comparison with other studies Research | Analysis | Acc
---|---|---
[2] |  | SVM on vibration sensor
---
data to detect failures
94.93%
[5] |  | CNN classifier on surface images
---
to detect warping failures
91.0%
| 3D-EDM
---
Binary
| CNN classifiers on
---
multi-angle printed images
decide fault or not
96.72%
| 3D-EDM
---
Multi
| CNN classifiers on 4 specific
---
fault cases to distinguish
the fault cases
93.38%

Study with SVM classifiers on various sensors [2], especially with vibration
sensors, shows 94.93% of test accuracy for detecting failure printed objects.
Detection algorithm with CNN classifier on surface images showed 91.0% test
accuracy [5]. Although these studies show a high detection accuracy of more
than 90%, as mentioned above, there are remaining issues for actual 3D-printer
users to utilize the solutions.

We seperated the test results into two significant directions and showed the
results for each experiment. In the case of binary-classification, which uses
3D-EDM to check whether the received input data has a fault or not, the test
accuracy was 96.72%. In addition, the multi-classification task, that
distinguishes the current fault case from layer shift, strings, under
extrusion, and warping, showed about 93.38% of the test accuracy.

##  IV Conclusion

In this study, we generated a lightweight image deep learning algorithm, to
predict and detect possible fault cases in a 3D printing environment. Although
previous studies have suggested a number of methods for fault detection, our
study is distinct in that we designed the experiment for practical use. In
addition, we believe it is more suitable for actual use because it utilizes
easily obtainable usage data rather than data that from complicated
installation settings.

As with many deep learning-related models, this study could show more accurate
results if more data were collected, but there was a time limitation to
collect more data at present. When more data collected, it will be possible to
make a more accurate Early Detection Model. However, this study is still
meaningful because it has been proven that this method is feasible, and
optimization for this method has been proposed. In the near future, we hope
that by running this model on Raspberry Pi for actual use, 3D printer users
can print the desired object as they want.

## References

  * [1] Y. Banadaki, N. Razaviarab, H. Fekrmandi, and S. Sharifi, “Toward enabling a reliable quality monitoring system for additive manufacturing process using deep convolutional neural networks,” _arXiv preprint arXiv:2003.08749_ , 2020\.
  * [2] B. Li, L. Zhang, L. Ren, and X. Luo, “3d printing fault detection based on process data,” in _Proceedings of 2018 Chinese Intelligent Systems Conference_. Springer, 2019, pp. 385–396.
  * [3] U. Delli and S. Chang, “Automated process monitoring in 3d printing using supervised machine learning,” _Procedia Manufacturing_ , vol. 26, pp. 865–870, 2018.
  * [4] V. Kadam, S. Kumar, A. Bongale, S. Wazarkar, P. Kamat, and S. Patil, “Enhancing surface fault detection using machine learning for 3d printed products,” _Applied System Innovation_ , vol. 4, no. 2, p. 34, 2021.
  * [5] Z. Jin, Z. Zhang, and G. X. Gu, “Automated real-time detection and prediction of interlayer imperfections in additive manufacturing processes using artificial intelligence,” _Advanced Intelligent Systems_ , vol. 2, no. 1, p. 1900130, 2020.


# Quantum reinforcement learning in continuous action space

Shaojun Wu  Institute of Fundamental and Frontier Sciences, University of
Electronic Science and Technology of China, Chengdu, 610051, China  Shan Jin
Institute of Fundamental and Frontier Sciences, University of Electronic
Science and Technology of China, Chengdu, 610051, China  Dingding Wen
Institute of Fundamental and Frontier Sciences, University of Electronic
Science and Technology of China, Chengdu, 610051, China  Xiaoting Wang
<EMAIL_ADDRESS>Institute of Fundamental and Frontier Sciences,
University of Electronic Science and Technology of China, Chengdu, 610051,
China

###### Abstract

Quantum mechanics has the potential to speedup machine learning algorithms,
including reinforcement learning(RL). Previous works have shown that quantum
algorithms can efficiently solve RL problems in discrete action space, but
could become intractable in continuous domain, suffering notably from the
curse of dimensionality due to discretization. In this work, we propose an
alternative quantum circuit design that can solve RL problems in continuous
action space without the dimensionality problem. Specifically, we propose a
quantum version of the Deep Deterministic Policy Gradient method constructed
from quantum neural networks, with the potential advantage of obtaining an
exponential speedup in gate complexity for each iteration. As applications, we
demonstrate that quantum control tasks, including the eigenvalue problem and
quantum state generation, can be formulated as sequential decision problems
and solved by our method.

Reinforcement learning(RL) Sutton and Barto (2018) plays a vital role in
machine learning. Unlike supervised and unsupervised learning to find data
patterns, the idea of RL is to reduce the original problem into finding a good
sequence of decisions leading to an optimized long-term reward, through the
interaction between an agent and an environment. This feature makes RL
advantageous for solving a wide range of sequential decision problems,
including game-playing Silver  _et al._ (2017); Mnih  _et al._ , e.g., AlphaGo
Silver  _et al._ (2016), robotic control Peters  _et al._ (2003); Duan  _et
al._ (2016), quantum error correction Nautrup  _et al._ (2019); Andreasson
_et al._ (2019) and quantum control Palittapongarnpim  _et al._ (2017); An and
Zhou (2019); Bukov  _et al._ (2018); Niu  _et al._ (2019); Xu  _et al._
(2019); Zhang  _et al._ (2019); Wauters  _et al._ (2020). Typical RL
algorithms include Q-learning Watkins (1989); Watkins and Dayan (1992), Deep
Q-Network(DQN) Mnih  _et al._ ; Mnih  _et al._ (2015), and Deep Deterministic
Policy Gradient(DDPG) Lillicrap  _et al._ . Despite its broad applications,
the implementation of RL on classical computers becomes intractable as the
problem size grows exponentially, such as the cases from quantum physics and
big data. Inspired by the idea of quantum computation Nielsen and Chuang
(2011), quantum machine learning has been proposed to implement machine
learning algorithms on quantum circuits to potentially gain an exponential or
quadratic speedup Biamonte  _et al._ (2017); Shor (1994); Grover (1997); Wiebe
_et al._ (2012); Rebentrost  _et al._ (2014); Lloyd and Weedbrook (2018); Das
Sarma  _et al._ (2019); Lloyd  _et al._ (2014). In particular, implementing RL
on a quantum circuit has been proposed and has been shown to obtain a
quadratic speedup due to Grover’s algorithm Dong  _et al._ (2008); Dunjko  _et
al._ (2016); Paparo  _et al._ (2014); Dunjko  _et al._ (2017); Dunjko and
Briegel (2018); Jerbi  _et al._ . An interesting question is whether a quantum
reinforcement learning(QRL) algorithm can be constructed to guarantee an
exponential speedup over its classical counterpart in terms of gate
complexity. Another issue with the extant QRL proposals lies in that their
algorithms are only efficient to solve problems with _discrete action
space_(DAS), but not for problems with _continuous action space_(CAS) Dong
_et al._ (2008); Dunjko  _et al._ (2016); Paparo  _et al._ (2014); Dunjko  _et
al._ (2017), for which the computational overhead grows exponentially as the
discretization accuracy increases. It is desirable to construct a QRL
algorithm that can efficiently solve problems in CAS.

In this work, we adopt an alternative strategy to design a new QRL framework
that can solve both problems with CAS and those with DAS, without the
dimensionality problem due to discretization. Specifically, we will propose a
quantum DDPG algorithm to construct the agent’s policy and the value function
from quantum neural networks(QNN), e.g., the QNN constructed from the
variational quantum circuit(VQC) Peruzzo  _et al._ (2014); Farhi  _et al._ .
The VQC-based DAS reinforcement learning has been discussed in Chen  _et al._
(2020); Benedetti  _et al._ (2019). One advantage of using the QNN is, if the
data is encoded into the amplitudes of the quantum state, then the number of
required qubits in the quantum register will reduce to $\log N$, where $N$ is
the length of the state vector. If one can further show that the gate
complexity of the QNN is $\operatorname{poly}(\log N)$ for the given problem,
then the gate complexity of our QRL proposal can also achieve
$\operatorname{poly}(\log N)$ for many given problems, and an exponential
reduction of the gate complexity can be achieved compared to the classical RL
counterpart. In this case, the gate complexity of our proposed quantum DDPG
algorithm is determined by that of the QNN. To demonstrate our proposal, we
apply our quantum DDPG algorithm to solving the eigenvalue problem for a given
quantum system. Specifically, we construct the quantum environment for RL from
the phase estimation circuit Shor (1994). Simulations suggest that our
proposed quantum DDPG algorithm is effective in solving the eigenvalue problem
for low-dimensional systems. In the following, we will first have a brief
introduction to the RL and then propose our own QRL framework. We will focus
our interest exclusively on RL problems defined on a quantum system; how to
map an arbitrary classical RL problem onto the quantum circuit of our proposal
will be left for future work.

Classical reinforcement learning – The core idea of RL is, through the
iterative interactions, the agent learns and selects actions, and the
environment responds to these actions, by updating its state and feeding it
back to the agent. In the meanwhile, the environment also generates rewards,
which are some value-functions the agent aims to maximize over its choice of
actions along the sequential interactions Sutton and Barto (2018). Most
reinforcement learning problems can be described by a Markov Decision Process
(MDP) Van Otterlo and Wiering (2012); Sutton and Barto (2018) with basic
elements including a set of states $\mathcal{S}$, a set of actions
$\mathcal{A}$ and the reward $\mathcal{R}$. The agent interacts with its
environment at each of a sequence of discrete time steps, $t=0,1,\cdots,T$.
Each sequence like this generated in RL is called an _episode_. At each time
step $t$, the agent receives a representation of the environment’s state,
denoted by an $N$-dimensional vector $s_{t}\in\mathcal{S}$, based on which it
then chooses an action $a_{t}\in\mathcal{A}$, resulting the change of the
environment’s state from $s_{t}$ to $s_{t+1}$. At the next step, the agent
receives the reward $r_{t+1}$ determined by the 3-tuple
$(s_{t},a_{t},s_{t+1})$. The aim of the agent is to find a policy $\pi$ that
maximizes the cumulative reward $R_{t}=\sum_{k=0}^{T}\gamma^{k}r_{t+k+1}$,
where $\gamma$ is a discount factor, $0\leq\gamma\leq 1$. A large discount
factor $\gamma$ means that the agent cares more about future rewards. The
policy can be considered as a mapping from $\mathcal{S}$ to $\mathcal{A}$. The
update of the policy $\pi$ is achieved by optimizing the value function
$Q(s_{t},a_{t})\equiv E[R_{t}|s_{t},a_{t}]$, i.e., the expectation of $R_{t}$
under the policy $\pi$. Depending on whether the action space is discrete or
continuous, the RL problems can be classified into two categories: DAS and
CAS, with different algorithmic design to update the agent’s policy. For DAS
problems, popular RL algorithms includes Q-learning Watkins (1989), Sarsa
Rummery and Niranjan (1994), DQN Mnih  _et al._ (2015), etc.; for CAS
problems, popular algorithms include Policy Gradient Kakade (2002), DDPG
Lillicrap  _et al._ , etc.

Figure 1: The QRL model. Each iterative step can be described by the following
loop: (1) at step $t$, the agent receives $|s_{t}\rangle$ and generates the
action parameter $\bm{\theta}_{t}$ according to the current policy; (2) the
agent generates $|s_{t+1}\rangle\equiv U(\bm{\theta}_{t})|s_{t}\rangle$; (3)
based on $|s_{t}\rangle$ and $|s_{t+1}\rangle$, a reward $r_{t+1}$ is
calculated and fed back to the agent, together with $|s_{t+1}\rangle$; (4)
based on $|s_{t+1}\rangle$ and $r_{t+1}$, the policy is updated and then used
to generate $\bm{\theta}_{t+1}$.

The framework of quantum reinforcement learning – In order to construct a
quantum framework that works for both CAS and DAS cases, we present the
following QRL model, as shown in Fig. 1. The essential idea is how to map the
elements of classical RL into the quantum counterparts. We introduce a quantum
‘environment’ register to represent the environment in RL, and its quantum
state $|s_{t}\rangle$ to represent the classical state $s_{t}$ at time step
$t$. Then the action $a(\bm{\theta}_{t})$ can be represented as a
parameterized action unitary $U(\bm{\theta}_{t})$ on $|s_{t}\rangle$, where
$\bm{\theta}_{t}$ is the action parameter, which is continuous for CAS case,
and takes values from a finite set for DAS case. In order to generate the
quantum reward function, by introducing a reward register $|r_{t}\rangle$, we
design the reward unitary $U_{r}$ and the measurement observable $M$ such that
$r_{t+1}\equiv f(\langle s_{t}|\langle
0|U^{\dagger}(\bm{\theta}_{t})U_{r}^{\dagger}MU_{r}U(\bm{\theta}_{t})|0\rangle|s_{t}\rangle)$
will match the actual reward defined by the RL problem. Here, $f$ is a
function determined by the problem and $|0\rangle$ is the initial state of the
reward register. It will be clear in the context of a concrete problem how to
design $M$, $U_{r}$, and $f$ in the correct way, which will be discussed in
detail based on the eigenvalue problem in the following.

Figure 2: The quantum circuit for our QRL framework includes two registers:
the reward register, initialized $|0\rangle$, and the environment register
$|s_{t}\rangle$. $U_{policy}$ is generated by the quantum DQN/DDPG algorithm,
and determines the action unitary $U(\bm{\theta}_{t})$. $U_{r}$ and $M$ are
designed to generate the reward $r_{t+1}$.

With all RL elements represented as the components of a quantum circuit shown
in Fig. 2, it remains to show how to find the optimal policy
$\bm{\theta}_{t}=\bm{\pi}(|s_{t}\rangle)$ at each time step $t$, such that the
iterative sequence $U_{\operatorname{tot}}=U(\bm{\theta}_{T})\cdots
U(\bm{\theta}_{1})U(\bm{\theta}_{0})$ will drive the arbitrary initial state
$|s_{0}\rangle$ converging to the target state $|s_{d}\rangle$. The entire QRL
process can be divided into two steps. In Step 1, we construct the optimal
policy $U_{\operatorname{policy}}$ through the agent training, including the
policy update and the value-function estimation, which can be realized through
the function fitting using QNNs. In Step 2, under the established optimal
policy, we can iteratively generate $\\{U(\bm{\theta}_{k})\\}$, $k=0,\cdots,T$
that will complete the RL task.

Quantum neural network – One popular way of implementing a QNN is to use the
variational quantum circuit (VQC) Peruzzo  _et al._ (2014); Farhi  _et al._ ;
Chen  _et al._ (2020); Benedetti  _et al._ (2019), whose parameters can be
iteratively optimized for the given objective function on classical computers.
The VQC circuit of our quantum DDPG algorithm consists of a sequence of
unitary $\\{D^{(k)}(\bm{\theta})\\}$, and is ended by measurements of
observables $\\{B_{j}\\}$ with $\operatorname{Tr}(B_{i}B_{j})=0$ (Fig.3). Each
$D^{(k)}(\bm{\theta})$ can be chosen to have an identical structure,
$D^{(k)}(\bm{\theta})\equiv U_{\textup{ENT}}V^{(k)}(\bm{\theta})$, where
$V^{(k)}(\bm{\theta})\equiv\otimes_{l=1}^{n}(R_{x}(\theta_{k,3l-2})R_{z}(\theta_{k,3l-1})R_{x}(\theta_{k,3l}))$,
$U_{\textup{ENT}}\equiv\prod_{k=1}^{n-1}\operatorname{CNOT}_{(k,k+1)}$ and
$\operatorname{CNOT}_{(k,k+1)}$ uses the $k$-th qubit to control the
$(k+1)$-th qubit. Here, $R_{x,z}$ are rotations, with
$R_{\alpha}(\theta)\equiv\exp(-i\sigma_{\alpha}\theta/2)$, $\alpha=x,z$. For
the input $|\phi\rangle$, the output of the VQC can be expressed as the
expected measurement outcome
$C_{j}\equiv\langle\phi|D^{\dagger}(\bm{\theta}){B_{j}}D(\bm{\theta})|\phi\rangle$,
based on which the parameter $\bm{\theta}$ can then be optimized for the given
optimization problem, on a classical computer.

Figure 3: Circuit architecture for the VQC.

Quantum DDPG algorithm – For CAS problems, we hope to utilize QNNs to
iteratively construct a sequence of unitary gates that will drive the
environment register from the initial state eventually to the target state.
This is the essential idea of the quantum DDPG algorithm. Analogous to the
classical DDPG algorithm, we make use of the QNNs to construct the desired
policy function $\pi_{\bm{\eta}}$ and the value function $Q_{\bm{\omega}}$.
Specifically, the policy-QNN is used to approximate the policy function
$\theta_{t,j}\equiv\langle
s_{t}|D^{\dagger}(\bm{\eta}){B_{j}}D(\bm{\eta})|s_{t}\rangle$ with
$\bm{\theta}_{t}=(\theta_{t,1},\theta_{t,2},\dots)^{T}$, and the Q-QNN is used
to approximate the value function
$Q(|s_{t}\rangle,\bm{\theta}_{t})\equiv\langle\bm{\theta}_{t}|\langle
s_{t}|D^{\dagger}(\bm{\omega}){B_{Q}}D(\bm{\omega})|s_{t}\rangle|\bm{\theta}_{t}\rangle$.
In order to make the training more stable, two more target networks are
included with the same structure as the two main networks Mnih  _et al._ ;
Mnih  _et al._ (2015); Lillicrap  _et al._ . Therefore, the quantum DDPG uses
four QNNs in total: (1) the policy-QNN $\pi_{\bm{\eta}}(|s\rangle)$, (2) the
Q-QNN $Q_{\bm{\omega}}(|s\rangle,\bm{\theta})$, (3) the target-policy
$\pi^{\prime}_{\bm{\eta^{\prime}}}(|s\rangle)$ and (4) the target-Q
$Q^{\prime}_{\bm{\omega^{\prime}}}(|s\rangle,\bm{\theta})$.

Algorithm 1 Quantum DDPG algorithm

Randomly initialize $Q_{\bm{\omega}}(|s\rangle,\bm{\theta})$ and
$\pi_{\bm{\eta}}(|s\rangle)$

Initialize target $Q^{\prime}$ and $\pi^{\prime}$

Initialize replay buffer $\bm{D}$

for episode=1, M do

Prepare the initial state $|0,s_{0}\rangle$

for t=1:$T$ do

Select the actions: $\bm{\theta}_{t}=\pi_{\bm{\eta}}(|s_{t}\rangle)$

Apply $U(\bm{\theta}_{t})$: $|s_{t+1}\rangle=U(\bm{\theta}_{t})|s_{t}\rangle$

Apply $U_{r}$ and measurement to obtain $r_{t+1}$

Store tuple $(|s_{t}\rangle,\bm{\theta}_{t},r_{t},|s_{t+1}\rangle)$ in
$\bm{D}$

Sample a batch of tuples
$(|s_{i}\rangle,\bm{\theta}_{i},r_{i},|s_{i+1}\rangle)$ from $\bm{D}$

Set
$y_{i}=r_{i}+\bm{\gamma}Q^{\prime}_{\bm{\omega}^{\prime}}(|s_{i+1}\rangle,\pi^{\prime}_{\bm{\eta}^{\prime}}(|s_{i+1}\rangle))$

Update Q-QNN by minimizing the loss:

$L=\frac{1}{G}\sum_{i}(y_{i}-Q_{\bm{\omega}}(|s_{i}\rangle,\bm{\theta}_{i}))^{2}$

Update the policy-QNN:

$\nabla_{\bm{\eta}}J\approx$

$\frac{1}{G}\sum_{i}\nabla_{\bm{\theta}}Q_{\bm{\omega}}(|s\rangle,\bm{\theta})|_{|s\rangle=|s_{i}\rangle,\bm{\theta}=\pi(|s_{i}\rangle)}\nabla_{\bm{\eta}}\pi_{\bm{\eta}}(|s\rangle)|_{|s_{i}\rangle}$

Update the target QNNs:

$\bm{\omega}^{\prime}\leftarrow\tau\bm{\omega}+(1-\tau)\bm{\omega}^{\prime}$

$\bm{\eta}^{\prime}\leftarrow\tau\bm{\eta}+(1-\tau)\bm{\eta}^{\prime}$

end for

end for

The quantum DDPG method contains two steps. In Step 1, the agent training is
divided into three parts: (1) experience replay Lin (1992), (2) updates of the
Q-QNN and the policy-QNN, and (3) updates of the target networks. The aim of
the experience replay is to prevent the neural network from overfitting. We
store the agent’s experiences
$(|s_{t}\rangle,\bm{\theta}_{t},r_{t},|s_{t+1}\rangle)$ in a finite-sized
replay buffer $\bm{D}$ at each time step. During the training, we randomly
sample a batch of experiences from the replay buffer to update the Q-QNN and
the policy-QNN. First, we update the policy-QNN parameters by minimizing the
expected return
$J=E[Q_{\bm{\omega}}(|s\rangle,\bm{\theta})|_{|s\rangle=|s_{i}\rangle,\bm{\theta}=\pi(|s_{i}\rangle)}]$.
Then we update the Q-QNN parameters by minimizing the mean-squared loss
$L=\frac{1}{G}\sum_{i}(y_{i}-Q_{\bm{\omega}}(|s_{i}\rangle,\bm{\theta}_{i}))^{2}$
between the predicted Q-value and the original Q-value, where
$y_{i}=r_{i}+\bm{\gamma}Q^{\prime}_{\bm{\omega}^{\prime}}(|s_{i+1}\rangle,\pi^{\prime}_{\bm{\eta}^{\prime}}(|s_{i+1}\rangle))$
is the predicted Q-value and calculated by the target-Q networks. $G$ is the
size of the batch. Finally, we update the two target networks using a soft
update strategy Lillicrap  _et al._ :
$\bm{\omega}^{\prime}\leftarrow\tau\bm{\omega}+(1-\tau)\bm{\omega}^{\prime}$,
$\bm{\eta}^{\prime}\leftarrow\tau\bm{\eta}+(1-\tau)\bm{\eta}^{\prime}$, where
$\tau$ is a parameter with $0<\tau<1$. The entire training process is
summarized in Algorithm 1. In Step 2, with the optimal policy-QNN and $T$
iterations, we can construct a sequence of $U(\bm{\theta}_{t})$ and
$|s_{t}\rangle$ for each given initial $|s_{0}\rangle$, satisfying
$|s_{T}\rangle$ is sufficiently close to the target $|s_{d}\rangle$.

RL tasks in DAS – For DAS problems, the above QRL proposal still works if the
quantum DDPG design in Fig. 2 is replaced by the quantum DQN design, analogous
to the classical DQN algorithm Mnih  _et al._ (2015). Compared with the
quantum DDPG, the quantum DQN maps states of the environment into the
computational basis, rather than into the amplitudes of a quantum register.
Moreover, for quantum DQN, only the value function needs to be approximated by
the QNN, while the policy function can be described by the $\epsilon$-greedy
algorithm Sutton and Barto (2018). Detailed proposals to solve DAS problems
using QNNs are presented in Chen  _et al._ (2020); Lockwood and Si (2020). It
is worthwhile to note that the quantum DQN cannot efficiently solve CAS
problems, since the dimensionality problem is inevitable when solving the CAS
problems through discretization.

Solving the eigenvalue problem – The eigenvalue problem in quantum physics can
be formulated as a sequential decision problem in CAS. Let $H$ be the
Hamiltonian defined on an $N$-dimensional quantum system $E$. For an unknown
eigenvalue $\lambda_{0}$ of $H$ located in a neighborhood of $\bar{\lambda}$,
$\lambda_{0}\in\delta(\bar{\lambda})\equiv[\bar{\lambda}-\delta,\bar{\lambda}+\delta]$,
we hope to find $\lambda_{0}$ and its corresponding eigenvector
$|u_{0}\rangle$. The idea of our quantum DDPG algorithm is to find the optimal
policy $U_{\operatorname{policy}}$ so that any arbitrary initial state
$|s_{0}\rangle$ of $E$ can be iteratively driven to the target state
$|u_{0}\rangle$ by applying a sequence of $\\{U(\bm{\theta}_{t})\\}$ generated
by the QNNs. To implement the QRL circuit in Fig. 2 for the eigenvalue
problem, we choose the quantum phase estimation $U_{\operatorname{PE}}$ as
$U_{r}$, as shown in Fig. 4 Shor (1994). The role of $U_{\operatorname{PE}}$
together with the subsequent measurement is to map the input state
$|s_{t+1}\rangle$ into the desired eigenstate with certain probability. Such
probability can be increased through the sequential amplitude amplification
using Grover’s algorithm Jin  _et al._ (2020); here, the QRL algorithm
provides an alternative way of realizing the amplitude amplification for
$U_{\operatorname{PE}}$. Specifically, the reward function $r_{t+1}$ can be
defined as the difference of the overlaps between the $(t+1)$-th and the
$t$-th states with $|u_{0}\rangle$: $r_{t+1}\equiv|\langle
s_{t+1}|u_{0}\rangle|^{2}-|\langle s_{t}|u_{0}\rangle|^{2}$. Let $|0\rangle$
and $|s_{0}\rangle=\sum_{k=1}^{N}\alpha_{0,k}|u_{k}\rangle$ be the initial
states of the reward register and the $n$-qubit environment register, where
$n=\log N$ and $\alpha_{0,k}=\langle u_{k}|s_{0}\rangle$. At the time step
$t$, applying $U_{\operatorname{policy}}$ and a quantum measurement to
$|s_{t}\rangle$, as shown in Fig. 2, we obtain the action parameter
$\bm{\theta}_{t}$, and generate the corresponding action unitary
$U(\bm{\theta}_{t})\equiv U_{\textup{ENT}}V(\bm{\theta}_{t})$, where
$V(\bm{\theta}_{t})$ and $U_{\textup{ENT}}$ are defined as in Fig. 3. Then we
apply $U(\bm{\theta}_{t})$ and get

$\displaystyle|s_{t+1}\rangle=U(\bm{\theta}_{t})|s_{t}\rangle=U_{\textup{ENT}}V(\bm{\theta}_{t})|s_{t}\rangle.$
(1)

Next, by applying $U_{\operatorname{PE}}$, we obtain

$\displaystyle
U_{\operatorname{PE}}|0\rangle|s_{t+1}\rangle=\sum_{k=1}^{N}\alpha_{t+1,k}|{\lambda}_{k}\rangle|u_{k}\rangle,$
(2)

where $|u_{k}\rangle$ is the eigenvector corresponding to the eigenvalue
$\lambda_{k}$. By measuring the eigenvalue phase register, we obtain the
outcome $\lambda_{0}$ with probability

$\displaystyle p_{t+1}\equiv|\alpha_{t+1,0}|^{2}=|\langle
s_{t+1}|u_{0}\rangle|^{2},$ (3)

which can be estimated by the frequency of obtaining $\lambda_{0}$ in $K$
number of measurements. The reward can be written as $r_{t+1}=p_{t+1}-p_{t}$.
If $p_{t+1}\to 1$ as $t\to\infty$, then $|s_{t+1}\rangle$ will converge to
$|u_{0}\rangle$ to complete the RL goal. Notice that the evaluation of
$r_{t+1}$ is only needed for the QRL training stage; in the application stage,
since $U_{\operatorname{policy}}$ is optimized, measurements are no longer
required to estimate $r_{t+1}$.

Figure 4: The quantum phase estimation circuit $U_{\operatorname{PE}}$, with
$U=e^{2\pi iHj}$, $j=2^{0},2^{1},\cdots,2^{q-1}$. $q$ is the number of qubits
in the eigenvalue register, determining the precision of the calculated phase.
$W_{H}\equiv H_{A}^{\otimes m}$, where $H_{A}$ is the Hadamard gate.

To verify the effectiveness of our proposed algorithm, we conduct a numerical
simulation of our method for the single-qubit case. For a quantum system with
Hamiltonian
$H=\frac{1}{4}(s_{x}\sigma_{y}+s_{y}\sigma_{y}+s_{z}\sigma_{z}+I)$, we
randomly generate the coefficients $(s_{x},s_{y},s_{z})=(0.13,0.28,0.95)$. In
Step 1, we apply the quantum DDPG algorithm to update the policy until we
obtain an optimal $U_{policy}$. In Step 2, based on the optimal $U_{policy}$,
we apply a sequence of $\\{U(\bm{\theta}_{t})\\}$ on a certain state to get
the final state after $50$ iterations,
$|s_{50}\rangle=U(\bm{\theta}_{49})\cdots U(\bm{\theta}_{0})|s_{0}\rangle$,
and record the overlap statistical frequency $p_{t}$ at each iteration $t$. In
order to demonstrate the effectiveness of our method, we randomly select 1000
different initial states to repeat this process, and calculate the average
$\bar{p}_{t}$ and variance $\Delta({p_{t}})$ for all $t$. It can be seen from
Fig. 5 that as the number of iterations increases, the average value of the
overlap $\bar{p}_{t}$ gradually increases almost to $1$, and the variance
$\Delta({p_{t}})$ gradually decreases almost to zero, with $\bar{p}_{50}\geq
0.99$ and $\Delta({p_{50}})\leq 2.93\times 10^{-8}$ at $t=50$. This simulation
result demonstrates that different initial states have been eventually driven
to the target state as expected, showing the advantage of reinforcement
learning: the QRL algorithm needs to run only once and the established model
will solve the eigenvalue problem for all different states. In comparison, for
methods such as the conventional quantum optimal control or VQE Peruzzo  _et
al._ (2014), the optimization routine has to be run for each initial state; in
other words, the control pulse found for one initial state is useless for
other initial states.

Figure 5: Simulation results of applying quantum DDPG algorithm to solve the
eigenvalue problem for a single-qubit system with $H$. Define $p_{t}$ as the
overlap between $|s_{t}\rangle$ and the target $|u_{0}\rangle$. To verify the
effectiveness of the method, we plot how the average $\bar{p}_{t}$ and the
variance $\Delta(p_{t})$ change as the RL iteration step $t$ increases, shown
in (a) and (b). At $t=50$, $\bar{p}_{50}\geq 0.99$ and $\Delta(p_{50})\leq
2.93\times 10^{-8}$.

Next, we study the gate complexity of the above QRL algorithm during a single
iteration at $t$. We assume that at most $m$ copies of the input state
$|s_{t}\rangle$ of the VQC are required to obtain the output of a VQC through
quantum measurement. Due to the law of large numbers and Chebyshev inequality,
in many cases, the size of $m$ can be assumed to be independent of $N$. Hence,
for Step 1 of our method, $\mathcal{O}(m)$ copies of $|s_{t}\rangle$ are
needed to generate the outputs of the four QNNs in the quantum DDPG algorithm.
In addition, if $H$ can be effectively simulated, the complexity of
$U_{\operatorname{PE}}$ is $\operatorname{poly}(\log N)$. We further assume
the ansatz that each VQC in our method has a circuit complexity of
$\operatorname{poly}(\log N)$. If the ansatz is valid, then the total
complexity for Step 1 is $\mathcal{O}(m\text{poly}(\log N))$. For Step 2,
since calculating $\bm{\theta}_{t}$ requires $m$ copies of $|s_{t}\rangle$,
the total complexity of deriving $\bm{\theta}$ and implementing
$U(\bm{\theta})$ is $\mathcal{O}(m\text{poly}(\log N))$.

Solving quantum control problems – The above quantum DDPG algorithm can be
used to solve other quantum control problems, such as the quantum state
generation, whose goal is to drive an $N$-dimensional quantum system from the
given initial state $|s_{0}\rangle$ to the target state $|s_{d}\rangle$. The
advantage of our QRL algorithm lies in that once the optimal policy
$U_{\operatorname{policy}}$ is generated through QNN learning, it represents a
model that is able to generate the appropriate control sequence
$\\{U(\bm{\theta}_{t})\\}$ for arbitrary $|s_{0}\rangle$. In comparison, the
conventional quantum optimal control algorithm has to find the control pulse
case by case for different $|s_{0}\rangle$. Specifically, given
$|s_{d}\rangle$, we can define an observable $M_{d}\equiv|s_{d}\rangle\langle
s_{d}|$. Then analogous to the eigenvalue problem, we can obtain the overlap
$p_{t}=\langle s_{t}|M_{d}|s_{t}\rangle$ through the statistics of the
measurement of $M_{d}$, whose measurement complexity is independent of $N$. We
further define the reward $r_{t+1}=p_{t+1}-p_{t}$. Then by implementing Step 1
and Step 2 of our method, the state generation problem will be solved with our
quantum DDPG algorithm.

Concluding discussion – In this work, we have proposed an alternative QRL
framework that can solve both CAS and DAS reinforcement learning tasks. For
CAS tasks, a quantum DDPG algorithm is proposed, which encodes the environment
state into the quantum state amplitude to avoid the dimensionality disaster
due to discretization. Classical simulation demonstrates that for low
dimensional quantum systems, our QRL method can work effectively to solve
quantum control problems such as the eigenstate preparation and the state
generation. We have also analyzed the gate complexity of our proposal for a
single RL iteration and find that the efficiency of our method depends on
whether the QNNs, or VQCs in particular, can be implemented with an efficient
gate complexity $\text{poly}(\log N)$, which will be the focus for future
work.

This research was supported by National Key R&D Program of China, Grant
No.2018YFA0306703. We also thank Xiaokai Hou, Yuhan Huang, and Qingyu Li for
helpful and inspiring discussions.

## References

  * Sutton and Barto (2018) R. S. Sutton and A. G. Barto,  _Reinforcement Learning: An Introduction_, 2nd ed. (The MIT Press, 2018).
  * Silver  _et al._ (2017) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, _et al._ , Nature(London) 550, 354 (2017).
  * (3) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, arXiv:1312.5602 .
  * Silver  _et al._ (2016) D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, _et al._ , Nature(London) 529, 484 (2016).
  * Peters  _et al._ (2003) J. Peters, S. Vijayakumar, and S. Schaal, in  _Proceedings of the third IEEE-RAS international conference on humanoid robots_ (2003) pp. 1–20.
  * Duan  _et al._ (2016) Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, in  _International Conference on Machine Learning_ (2016) pp. 1329–1338.
  * Nautrup  _et al._ (2019) H. P. Nautrup, N. Delfosse, V. Dunjko, H. J. Briegel, and N. Friis, Quantum 3, 215 (2019).
  * Andreasson  _et al._ (2019) P. Andreasson, J. Johansson, S. Liljestrand, and M. Granath, Quantum 3, 183 (2019).
  * Palittapongarnpim  _et al._ (2017) P. Palittapongarnpim, P. Wittek, E. Zahedinejad, S. Vedaie, and B. C. Sanders, Neurocomputing 268, 116 (2017).
  * An and Zhou (2019) Z. An and D. L. Zhou, EPL (Europhysics Letters) 126, 60002 (2019).
  * Bukov  _et al._ (2018) M. Bukov, A. G. R. Day, D. Sels, P. Weinberg, A. Polkovnikov, and P. Mehta, Phys. Rev. X 8, 031086 (2018).
  * Niu  _et al._ (2019) M. Y. Niu, S. Boixo, V. N. Smelyanskiy, and H. Neven, npj Quantum Information 5, 1 (2019).
  * Xu  _et al._ (2019) H. Xu, J. Li, L. Liu, Y. Wang, H. Yuan, and X. Wang, npj Quantum Information 5, 1 (2019).
  * Zhang  _et al._ (2019) X.-M. Zhang, Z. Wei, R. Asad, X.-C. Yang, and X. Wang, npj Quantum Information 5, 1 (2019).
  * Wauters  _et al._ (2020) M. M. Wauters, E. Panizon, G. B. Mbeng, and G. E. Santoro, Phys. Rev. Research 2, 033446 (2020).
  * Watkins (1989) C. J. C. H. Watkins,  _Learning from delayed rewards_ , Ph.D. thesis (1989).
  * Watkins and Dayan (1992) C. J. C. H. Watkins and P. Dayan, Machine Learning 8, 279 (1992).
  * Mnih  _et al._ (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, _et al._ , Nature(London) 518, 529 (2015).
  * (19) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, arXiv:1509.02971 .
  * Nielsen and Chuang (2011) M. A. Nielsen and I. L. Chuang,  _Quantum Computation and Quantum Information_ , 10th ed. (Cambridge University Press, USA, 2011).
  * Biamonte  _et al._ (2017) J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature(London) 549, 195 (2017).
  * Shor (1994) P. W. Shor, in  _Proceedings 35th Annual Symposium on Foundations of Computer Science_ (1994) pp. 124–134.
  * Grover (1997) L. K. Grover, Phys. Rev. Lett. 79, 325 (1997).
  * Wiebe  _et al._ (2012) N. Wiebe, D. Braun, and S. Lloyd, Phys. Rev. Lett. 109, 050505 (2012).
  * Rebentrost  _et al._ (2014) P. Rebentrost, M. Mohseni, and S. Lloyd, Phys. Rev. Lett. 113, 130503 (2014).
  * Lloyd and Weedbrook (2018) S. Lloyd and C. Weedbrook, Phys. Rev. Lett. 121, 040502 (2018).
  * Das Sarma  _et al._ (2019) S. Das Sarma, D.-L. Deng, and L.-M. Duan, Physics Today 72, 48–54 (2019).
  * Lloyd  _et al._ (2014) S. Lloyd, M. Mohseni, and P. Rebentrost, Nature Physics 10, 631 (2014).
  * Dong  _et al._ (2008) D. Dong, C. Chen, H. Li, and T.-J. Tarn, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 38, 1207 (2008).
  * Dunjko  _et al._ (2016) V. Dunjko, J. M. Taylor, and H. J. Briegel, Phys. Rev. Lett. 117, 130501 (2016).
  * Paparo  _et al._ (2014) G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Phys. Rev. X 4, 031002 (2014).
  * Dunjko  _et al._ (2017) V. Dunjko, J. M. Taylor, and H. J. Briegel, in  _2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ (2017) pp. 282–287.
  * Dunjko and Briegel (2018) V. Dunjko and H. J. Briegel, Reports on Progress in Physics 81, 074001 (2018).
  * (34) S. Jerbi, L. M. Trenkwalder, H. P. Nautrup, H. J. Briegel, and V. Dunjko, arXiv:1910.12760 .
  * Peruzzo  _et al._ (2014) A. Peruzzo, J. McClean, P. Shadbolt, M.-H. Yung, X.-Q. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’brien, Nature communications 5, 4213 (2014).
  * (36) E. Farhi, J. Goldstone, and S. Gutmann, arXiv:1411.4028 .
  * Chen  _et al._ (2020) S. Y. C. Chen, C. H. H. Yang, J. Qi, P. Y. Chen, X. Ma, and H. S. Goan, IEEE Access 8, 141007 (2020).
  * Benedetti  _et al._ (2019) M. Benedetti, E. Lloyd, S. Sack, and M. Fiorentini, Quantum Science and Technology 4, 043001 (2019).
  * Van Otterlo and Wiering (2012) M. Van Otterlo and M. Wiering, in  _Reinforcement Learning_ (Springer, 2012) pp. 3–42.
  * Rummery and Niranjan (1994) G. A. Rummery and M. Niranjan, Technical Report CUED/F-INFENG/TR 166 37 (1994).
  * Kakade (2002) S. M. Kakade, in  _Advances in Neural Information Processing Systems_, Vol. 14 (MIT Press, 2002) pp. 1531–1538.
  * Lin (1992) L.-J. Lin,  _Reinforcement Learning for Robots Using Neural Networks_ , Ph.D. thesis, USA (1992).
  * Lockwood and Si (2020) O. Lockwood and M. Si, in  _The 16th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ (2020).
  * Jin  _et al._ (2020) S. Jin, S. Wu, G. Zhou, Y. Li, L. Li, B. Li, and X. Wang, Quantum Engineering 2, e49 (2020).
  * Masson  _et al._ (2016) W. Masson, P. Ranchod, and G. Konidaris, in  _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ , AAAI’16 (AAAI Press, 2016) p. 1934–1940.
  * Doya (2000) K. Doya, Neural Computation 12, 219 (2000).
  * (47) G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, arXiv:1606.01540 .

##  I appendix

###  I.1 Classical Reinforcement Learning

In artificial intelligence, an agent is a mathematical abstraction
representing an object with learning and decision-making abilities. It
interacts with its environment, which includes everything except the agent.
The core idea of RL is: through the iterative interactions, the agent learns
and selects actions, and the environment responds to these actions, by
updating its state and feeding it back to the agent. In the meanwhile, the
environment also generates rewards, which are some value functions the agent
aims to maximize over its choice of actions along the sequential interactions
Sutton and Barto (2018). Reinforcement learning problems can be described by a
Markov Decision Process(MDP) Van Otterlo and Wiering (2012); Sutton and Barto
(2018). Its basic elements include a set of states $\mathcal{S}$, a set of
actions $\mathcal{A}$, the reward $\mathcal{R}$, and the model is shown in
Fig. 6 Sutton and Barto (2018). The agent interacts with its environment at
each step $t$ of a sequence of discrete time points, $t=0,1,2,3,\cdots$. At
each step $t$, the agent receives an representation of the environment’s
state, denoted by an $N$-dimensional vector $s_{t}\in\mathcal{S}$, based on
which it then chooses an action $a_{t}\in\mathcal{A}$, resulting in the change
of the environment’s state from $s_{t}$ to $s_{t+1}$. At the next step, the
agent receives the reward $r_{t+1}$ determined by the 3-tuple
$(s_{t},a_{t},s_{t+1})$. Given an initial state $s_{0}$, the agent-environment
interactions will generate the following sequence:
$s_{0},a_{0},r_{1},s_{1},a_{1},r_{2},\cdots$. Such a sequence is called an
_episode_ in RL. Next, we define the following three key elements of RL:

(1) Policy

The policy can be considered as a mapping from $\mathcal{S}$ to $\mathcal{A}$,
which sets the rules on how to choose the action based on the environment’s
state. Such policy is determined by certain optimization objective, such as
maximizing the cumulative reward. A policy can be either deterministic or
stochastic. A deterministic policy is characterized by a function $a=\pi(s)$,
meaning that under the same policy, at time step $t$, the action $a_{t}$ is
uniquely determined by the current environment’s state $s_{t}$. Given the
state $s$, we define the stochastic policy, $\pi_{\theta}(a|s)\equiv
P[a|s,\theta]$ as the probability of choosing the random action $a$, where
$\theta$ is parameter charactering the distribution of $P[a|s,\theta]$.

(2) Cumulative reward

As mentioned above, at time step $t$, the policy goal of the agent is to
maximize the cumulative reward it receives in the long run. At each step $t$,
if we define the accumulative reward as $R_{t}=\sum_{k=0}^{\infty}r_{t+k+1}$,
it may not be convergent and becomes ill-defined; alternatively, we can
introduce a discount factor $\gamma(0\leq\bm{\gamma}\leq 1)$ and define the
cumulative reward as $R_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1}$. The
larger the discount factor, the longer time span of future rewards we will
consider to determine the current-state policy. At time step $t$, the reward
$r_{t}$ determines the immediate return, and the cumulative reward $R_{t}$
determines the long-term return.

(3) Value function

Notice that when $a_{t}$ or $s_{t}$ is stochastic, $r_{t}$ and $R_{t}$ are
also stochastic. Hence, we further define the value function $Q$ to be the
expectation of the cumulative reward, $Q(s,a)\equiv E[R_{t}|s,a]$, under the
policy $\pi$. The goal of RL is to find the optimal policy that maximizes the
value function $Q$.

Figure 6: An illustration of the standard RL method. The agent interacting
with an environment $E$ in discrete time steps. At each time step $t$, the
agent receives an observation $s_{t}$, then chooses an action $a_{t}$. Next,
the agent performs an action $a_{t}$, and the environment move to next state
$s_{t+1}$ and emits a reward $r_{t+1}$.

RL problems can be classified into two categories: discrete-action-space(DAS)
problems and continuous-action-space (CAS) problems. In a DAS problem, the
agent chooses the action from a finite set $\\{a_{k}\\}$, $k=1,\cdots,l$. For
example, in the Pong game Mnih  _et al._ (2015), the action set for moving the
paddle is {up, down}. In a CAS problem, the action can be parametrized as a
real-valued vectorMasson  _et al._ (2016). In the CartPole environment Doya
(2000), the action is the thrust and can be parametrized as a continuous
variable $\theta\in[-1,1]$. For DAS problems, popular RL algorithms include
Q-learning Watkins (1989), Sarsa Rummery and Niranjan (1994), Deep Q-learning
Network(DQN) Mnih  _et al._ (2015), etc.; for CAS problems, popular algorithms
include Policy Gradient Kakade (2002), Deep Deterministic Policy
Gradient(DDPG) Lillicrap  _et al._ , etc.

Notice that the DQN algorithm is only efficient when solving problems with
small DAS. It quickly becomes inefficient and intractable when the size of the
DAS becomes large. Hence, although a CAS problem can be converted into a DAS
problem through discretization, the DQN algorithm will not work in solving the
converted DAS problem, if we require high discretization accuracy. For CAS
problems, it is better to use a CAS algorithm, such as DDPG.

###  I.2 Discrete Action Space Algorithms

Q-learning is a milestone in reinforcement learning algorithms. The main idea
of the algorithm is to construct a Q-table of state-action pairs to store the
Q-values, and then the agent chooses the action that can obtain the largest
cumulative reward based on the Q-values.

In the Q-learning algorithm, an immediate reward matrix $R$ can be constructed
to represent the reward value from state $s_{t}$ to the next state $s_{t+1}$.
The Q-value is calculated based on the matrix $R$, and it is updated by the
following formula Sutton and Barto (2018):

$\displaystyle\begin{split}Q(s_{t},a_{t})&\longleftarrow Q(s_{t},a_{t})\\\
&+\alpha[r_{t}+\bm{\gamma}\max_{a_{t+1}}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})]\end{split}$
(4)

where $\bm{\gamma}$ is the discount factor, $\alpha$ is the learning rate that
determines how much the newly learned value of $Q$ will override the old
value. By training the agent, the Q-value will gradually converge to the
optimal Q-value.

The Q-learning is only suitable for storing action-state pairs which are low-
dimensional and discrete. In large-space tasks, the corresponding Q-table
could become extremely large, causing the RL problem intractable to solve,
which is known as the curse of dimensionality. The DQN algorithm takes the
advantage of deep learning technique to solve the RL problems. Specifically,
it introduces a neural network defined as Q-network
$Q(s_{t},a_{t};\bm{\omega})$, whose function is similar to the Q-table in
approximating the value function. The input of the Q-network is the current
state $s_{t}$, and the output is the Q-value. The $\epsilon$-greedy strategy
is used to choose the value of $a$ according to the following probability
distribution Sutton and Barto (2018):

$\displaystyle a=\begin{cases}\arg\max_{a}Q(a),&\text{ with probability}\
1-\epsilon;\\\ \text{a random action},\,&\text{ with probability}\
\epsilon.\end{cases}$ (5)

In order to stabilize the training, the DQN algorithm uses two tricks:
experience replay and a target network. The method of experience replay is to
use a replay buffer to store the experienced data and to sample some data from
the replay buffer at each step $t$ to update the parameters in the neural
network. The DQN algorithm introduces a target-Q network
$Q(s_{t+1},a_{t+1};{\bm{\omega}}^{\prime})$ which is a copy of the Q-network.
The input of $Q(s_{t+1},a_{t+1};{\bm{\omega}}^{\prime})$ is $s_{t+1}$ and the
output is $Q(s_{t+1},a_{t+1})$. However, the target-Q network parameters are
updated using Q-network parameters at every $m$ steps, where $m$ is a
constant. The DQN algorithm updates the Q-network by reducing the value of the
loss function
$L(\bm{\omega})=E[((r_{t}+\bm{\gamma}\max_{a_{t+1}}Q^{\prime}(s_{t+1},a_{t+1};{\bm{\omega}}^{\prime}))-Q(s_{t},a_{t};\bm{\omega}))^{2}]$.

###  I.3 Continuous Action Space Algorithms

For tasks in continuous action space, we usually use the DDPG algorithm. The
DDPG algorithm make use of the neural network to construct the desired policy
function $\pi:\,s_{t}\to a_{t}$ such that the value function is maximized. The
quantum DDPG includes four neural networks: the policy-network, the Q-network,
the target-policy-network and the target-Q-network. The Q-network is used to
approximate the value function, while the policy-network is used to
approximate the policy function.

The DDPG algorithm uses the same tricks as DQN to stabilize the training. The
update of the policy-network is achieved by reducing the loss function
$L(\bm{\eta})=-Q(s_{t},a_{t};\bm{\eta})$. Similar to DQN, the update of the
Q-network in DDPG is achieved through reducing the value of the loss function
$L(\bm{\omega})=E[((r_{t}+\bm{\gamma}\max_{a_{t+1}}Q^{\prime}(s_{t+1},a_{t+1};{\bm{\omega}}^{\prime}))-Q(s_{t},a_{t};\bm{\omega}))^{2}]$.
Through training, the estimated value output of the Q-network will be more
accurate, and the action given by the policy-network will make the Q-value
higher.

###  I.4 Quantum Reinforcement Learning in Discrete Action Space

Figure 7: Frozen Lake environment model. S is the starting position; F is the
walkable position; H is the hole position, and G is the goal position. The
actions that the agent can choose at each position are up, down, left, and
right.

Besides DAS problems, our QRL framework can solve DAS problems as well. For
DAS tasks, we can use quantum Q-learning or quantum DQN algorithm.
Specifically, we consider a Frozen Lake environment model Brockman  _et al._
in which both the action space and the state space are finite dimensional, as
shown in Fig. 7. In this environment, the agent can move from a grid to its
neighboring grids and the goal is to move from the starting position to the
target position. Some positions of the grids are walkable, while the others
will make the agent fall into the water, resulting in a large negative reward,
and a termination of the episode.

In order to solve the Frozen Lake problem using our QRL framework, we number
the $N$ grids from $0$ to $N-1$, and encode them into the set of basis states
$S=\\{|j\rangle\\}$, $j=0,\cdots,j-1$ of an $N$-dimensional quantum
environment, composed of $n=\log N$ qubits. At each step $t$, the state of the
environment $|s_{t}\rangle$ equals to one of the basis states in $S$. The
action $a(\bm{\theta}_{t})$ can be represented as a parameterized action
unitary $U(\bm{\theta}_{t})$ on $|s_{t}\rangle$, where $\bm{\theta}_{t}$ is
the action parameter. Assuming that at the position $|j\rangle$, there are
four actions(up, down, left, and right) the agent can choose from,
corresponding to the discrete set
$\\{U(\bm{\theta}_{j,k})\\}=\\{U(\bm{\theta}_{j,1}),U(\bm{\theta}_{j,2}),U(\bm{\theta}_{j,3}),U(\bm{\theta}_{j,4})\\}$
where $\bm{\theta}_{j,k}=(\theta_{j,k}^{(1)},\cdots,\theta_{j,k}^{(n)})^{T}$
is a real vector. Specifically, we construct $U(\bm{\theta}_{j,k})$ as
$U(\bm{\theta}_{j,k})\equiv R_{y}(\theta_{j,k}^{(1)})\otimes\dots\otimes
R_{y}(\theta_{j,k}^{(n)})$, where
$R_{y}(\theta_{j,k}^{(n)})=\exp(-i\theta_{j,k}^{(n)}\sigma_{y}/2)$, and
$\theta^{(j)}_{k,n}\in\\{0,\pi\\}$. For example, in Fig. 7, if the agent is at
the position $|5\rangle$ and wants to move right to the position $|6\rangle$,
then the required action parameter is $\bm{\theta}_{5,4}=(0,0,\pi,\pi)$,
corresponding to $U(\bm{\theta}_{5,4})\equiv R_{y}(0)\otimes R_{y}(0)\otimes
R_{y}(\pi)\otimes R_{y}(\pi)$, satisfying
$U(\bm{\theta}_{5,4})|0101\rangle=|0110\rangle$.

We generate the reward function $f$ by introducing a reward register
$|r_{t}\rangle$ and design the reward unitary $U_{r}=\sum_{j\in
F}|j\rangle\langle j|\otimes I\otimes I+\sum_{j\in H}|j\rangle\langle
j|\otimes I\otimes\sigma_{x}+\sum_{j\in G}|j\rangle\langle
j|\otimes\sigma_{x}\otimes I$ and the measurement observable
$M=\sum_{j=0}^{N}j|j\rangle\langle j|$. Then the reward $r_{t+1}$ is

$\displaystyle r_{t+1}=f(p_{t+1})=\begin{cases}-1,&p_{t+1}=0\\\
-10,&p_{t+1}=1\\\ 10,&p_{t+1}=2\end{cases}$ (6)

where $p_{t+1}\equiv\langle s_{t}|\langle
0|U^{\dagger}(\bm{\theta}_{t,k})U^{\dagger}_{r}MU_{r}U(\bm{\theta}_{t,k})|0\rangle|s_{t}\rangle$
and $r_{t+1}=f(p_{t+1})$. Here, $r_{t+1}$ is the reward for the action
$U(\bm{\theta}_{t,k})$ at the state $|s_{t}\rangle$ and $|0\rangle$ is the
initial state of the reward register.

With all RL elements represented as the components of a quantum circuit, we
can use the quantum Q-learning algorithm to solve the Frozen Lake problem. In
Step 1, we train the agent to find a state-action sequence to maximize the
cumulative reward. In the training, the data
$(|s_{t}\rangle,a(\bm{\theta}),r_{t+1})$ obtained from each interaction
between the agent and the environment is recorded and these data are used to
update the Q-value. In Step 2, we use the optimal policy to generate
$\\{U(\bm{\theta}_{0,k}),\cdots,U(\bm{\theta}_{T,k})\\}$ to complete the task.
Our simulation results show that though training the agent can reach the
target position by moving 6 steps.