Jiahao Yao<EMAIL_ADDRESS>
Department of Mathematics, University of California, Berkeley
Berkeley, CA 94720, USA and Paul Köttering††thanks:  P.K. & H.G. contributed
equally and listed by a coin flip
Department of Mathematics, University of California, Berkeley
Berkeley, CA 94720, USA and Hans Gundlach11footnotemark: 1
Department of Mathematics, University of California, Berkeley
Department of Physics, University of California, Berkeley
Berkeley, CA 94720, USA and Lin Lin
Department of Mathematics, University of California, Berkeley
Computational Research Division, Lawrence Berkeley National Laboratory
Challenge Institute for Quantum Computation, University of California,
Berkeley
Berkeley, CA 94720, USA and Marin Bukov<EMAIL_ADDRESS>
Department of Physics, St. Kliment Ohridski University of Sofia
5 James Bourchier Blvd, 1164 Sofia, Bulgaria

# Noise-Robust End-to-End Quantum Control
using Deep Autoregressive Policy Networks

###### Abstract

Variational quantum eigensolvers have recently received increased attention,
as they enable the use of quantum computing devices to find solutions to
complex problems, such as the ground energy and ground state of strongly-
correlated quantum many-body systems. In many applications, it is the
optimization of both continuous and discrete parameters that poses a
formidable challenge. Using reinforcement learning (RL), we present a hybrid
policy gradient algorithm capable of simultaneously optimizing continuous and
discrete degrees of freedom in an uncertainty-resilient way. The hybrid policy
is modeled by a deep autoregressive neural network to capture causality. We
employ the algorithm to prepare the ground state of the nonintegrable quantum
Ising model in a unitary process, parametrized by a generalized quantum
approximate optimization ansatz: the RL agent solves the discrete
combinatorial problem of constructing the optimal sequences of unitaries out
of a predefined set and, at the same time, it optimizes the continuous
durations for which these unitaries are applied. We demonstrate the noise-
robust features of the agent by considering three sources of uncertainty:
classical and quantum measurement noise, and errors in the control unitary
durations. Our work exhibits the beneficial synergy between reinforcement
learning and quantum control.

###### keywords:

Quantum control, Quantum approximate optimization algorithm, Quantum
computing, Reinforcement learning, Policy gradient, Autoregressive policy
network, Proximal policy optimization, Noise-robust optimization.

##  1 Introduction

The last decade has seen impressive breakthroughs in Machine Learning (ML),
ranging from image classification (LeCun et al., 2015; Krizhevsky et al.,
2017) to mastering complex video and board games Mnih et al. (2013); Silver et
al. (2016). ML algorithms have opened the door to solving major scientific
challenges, hitherto considered intractable, such as protein modelling Rao et
al. (2019) and folding Jumper et al. (2020), or molecular dynamics simulations
Lu et al. (2020).

Deep learning tools and methods quickly found their way into the field of
physics Dunjko and Briegel (2018); Mehta et al. (2019); Carleo et al. (2019);
Carrasquilla (2020): Supervised learning was found efficient in identifying
phase transitions and analyzing experimental data Carrasquilla and Melko
(2017); Van Nieuwenburg et al. (2017); Bohrdt et al. (2019); Rem et al.
(2019). Unsupervised learning brought a new class of variational many-body
wavefunctions Carleo and Troyer (2017), as well as methods to perform
tomography on many-body quantum states Torlai et al. (2018), find conservation
laws from data Iten et al. (2020), identify phase transitions Wang (2016);
Kottmann et al. (2020), Hamiltonian learning Valenti et al. (2019), etc.
Reinforcement learning (RL) Sutton and Barto (2018) brought strategies for
navigating turbulent flows Reddy et al. (2016); Colabrese et al. (2017);
Bellemare et al. (2020), and even exploring the string landscape Halverson et
al. (2019).

The variational character of ML models combined with their intrinsic
optimization procedure, provide a natural playground for applications in
quantum control Schäfer et al. (2020); Wang et al. (2020a); Sauvage and
Mintert (2019); Fösel et al. (2020); Nautrup et al. (2019); Albarrán-Arriagada
et al. (2018); Sim et al. (2020); Wu et al. (2020a, b); Anand et al. (2020).
Due to the close relationship between control theory and reinforcement
learning, the control of quantum systems has become a major application area
of RL algorithms in physics. Notable examples include policy gradient Niu et
al. (2019); Fösel et al. (2018); August and Hernández-Lobato (2018); Porotti
et al. (2019); Wauters et al. (2020); Yao et al. (2020a); Sung (2020),
Q-learning Chen et al. (2013); Bukov (2018); Bukov et al. (2018); Sørdal and
Bergli (2019); Bolens and Heyl (2020) and AlphaZero Dalgaard et al. (2020b).

Over the years, the physics community has also developed a number of
successful quantum control algorithms Khaneja et al. (2005); Caneva et al.
(2011); Peruzzo et al. (2014); Dalgaard et al. (2020a); Magann et al. (2020b,
a), including GRAPE, CRAB, and VQE. A prominent example of the latter is
Quantum Approximate Optimization Algorithm (QAOA) Farhi et al. (2014), whose
versatility allows for solving complex combinatorial problems using quantum
computers Garcia-Saez and Riu (2019); Dong et al. (2019); Khairy et al. (2019,
2020); Yao et al. (2020b); Tabi et al. (2020); Bravyi et al. (2020). Quantum
control algorithms, such as CRAB or QAOA, come with an ingenious physics-
informed variational ansatz for the structure of control protocols. RL
algorithms, on the other hand, are model-free and resilient to uncertainty.
Hence, a natural question emerges as to how one can combine the benefits
offered by RL and quantum control in a unified framework.

In this paper, our aim is to deploy a generalized QAOA ansatz in combination
with an end-to-end deep RL algorithm for a versatile continuous-discrete
quantum control [Sec. 2]. We adopt the continuous degrees of freedom of QAOA
which offer an increased control accuracy. Additionally, we consider an
enhanced variational control ansatz which contains a larger space to select
the building blocks of the protocols from; this introduces a second, discrete
combinatorial optimization problem. The resulting algorithm, RL-QAOA, realizes
greater gains by striking a balance between robustness and versatility: it is
resilient to various kinds of uncertainty, a property shared with PG-QAOA Yao
et al. (2020a); at the same time, RL-QAOA has access to the more general
variational counter-diabatic (CD) driving ansatz Demirplak and Rice (2005);
Masuda and Nakamura (2009); Guéry-Odelin et al. (2019) through CD-QAOA Yao et
al. (2020b).

However, RL-QAOA presents a number of new challenges, cf. Sec. 3. It requires
a mixed continuous-discrete action space so that the RL agent can construct a
control protocol by optimizing the order in which unitaries appear in the
control sequence; simultaneously, the agent has to also choose the continuous
duration to apply each unitary. This requires the use of a suitable ML model
to approximate the policy, which allows us to build in temporal causality.
Therefore, an essential building block of RL-QAOA is a novel monolithic deep
autoregressive policy network that handles continuous and discrete actions on
equal footing. To train our RL agent, we derive an extension of Proximal
Policy Optimization (PPO) Schulman et al. (2017) to hybrid discrete-continuous
policies.

We apply RL-QAOA to find the ground state of a nonintegrable chain of
interacting spin-$1/2$ particles (a.k.a. qubits) in a fixed amount of time,
cf. Sec. 4. The mixed discrete-continuous degrees of freedom allow the RL
agent to construct a short protocol sequence away from the adiabatic regime.
We test the agent’s behavior in a strongly stochastic environment, by
considering three different kinds of noise: classical and quantum measurement
noise, and errors in the control unitary gate duration. In Sec. 5, we
demonstrate that RL-QAOA is insensitive to the types of noise applied, and
outperforms previously developed algorithms based on QAOA in the regime of
strong noise.

##  2 Preliminaries

We start the discussion by introducing the QAOA ansatz used in quantum
control. Following a short overview of reinforcement learning terminology, we
review two RL-based QAOA algorithms — PG-QAOA and CD-QAOA — which we aim to
blend into a homogeneous hybrid in Sec. 3. The resulting new algorithm
combines the benefits of the generalized variational QAOA ansatz, with an RL
algorithm performing both continuous and discrete control simultaneously.

###  2.1 QAOA for Ground State Preparation

Of particular interest in the quest for designing new materials with novel
features (such as room-temperature superconductors, or topological quantum
computers), is the study of ground state properties in quantum many-body
physics. Quantum simulators provide an ideal platform to bring together both
theory and experiment; yet, they require the ability to prepare a system in
its ground state – a formidable challenge for modern quantum computing
devices, due to the presence of various sources of uncertainty and noise. The
Quantum Approximate Optimization Algorithm (QAOA) Farhi et al. (2014) provides
a widely used state-of-the-art ansatz for this purpose.

Consider a quantum system of $N$ qubits, described by the Hamiltonian $H$.
Starting from an initial quantum state $\ket{\psi_{i}}$, in QAOA we apply two
alternating unitary evolution operators (i.e. quantum gates) (Farhi et al.,
2014):

$\ket{\psi(T)}=U(\\{\alpha_{j},\beta_{j}\\}_{j=1}^{p})\ket{\psi_{i}}=e^{-iH_{2}\beta_{p}}e^{-iH_{1}\alpha_{p}}\cdots
e^{-iH_{2}\beta_{1}}e^{-iH_{1}\alpha_{1}}\ket{\psi_{i}}.$ (1)

The dynamics are generated by the time-independent operators $H_{1}$ and
$H_{2}$, applied for a duration of $\alpha_{j}\geq 0$ and $\beta_{j}\geq 0$,
respectively ($j=1,2,\cdots,p$ with $p\in\mathbb{N}$). We refer to
$q\\!=\\!2p$ as the total circuit depth. In order to apply QAOA to many-body
systems Ho and Hsieh (2019), the protocol durations
$\\{(\alpha_{j},\beta_{j})\\}_{j=1}^{p}$ are variationally optimized to
minimize the expected value of the energy density
$\mathcal{E}(\\{\alpha_{j},\beta_{j}\\}_{j=1}^{p})\\!=\\!N^{-1}{\matrixelement{\psi(T)}{H}{\psi(T)}}$
:

$\\{\alpha_{j}^{\ast},\beta_{j}^{\ast}\\}_{j=1}^{p}\\!=\\!\operatorname*{arg\,min}_{\\{\alpha_{j},\beta_{j}\\}_{j=1}^{p}}\mathcal{E}(\\{\alpha_{j},\beta_{j}\\}_{j=1}^{p}),\quad\sum_{j=1}^{p}(\alpha_{j}+\beta_{j})=T.$
(2)

The additional constraint $\sum_{j=1}^{p}(\alpha_{j}+\beta_{j})=T$ is required
for the resulting protocol to remain in the regime of practical applications,
and also for a fair comparison between different algorithms.

As a concrete example to keep in mind, consider the spin-$1/2$ Ising
Hamiltonian

$H\\!=\\!H_{1}\\!+\\!H_{2},\qquad
H_{1}\\!\\!=\\!\\!\sum_{i=1}^{N}JS^{z}_{i+1}S^{z}_{i}\\!+\\!h_{z}S^{z}_{i},\quad
H_{2}=\sum_{i=1}^{N}h_{x}S^{x}_{i},$ (3)

where
$[S^{\alpha}_{k},S^{\beta}_{j}]=i\delta_{kj}\varepsilon^{\alpha\beta\gamma}S^{\gamma}_{j}$
are the spin-$1/2$ operators. We are interested in preparing the ground state
of $H$, starting from a spin-up polarized initial product state. More details
about the physical system are discussed later on in Sec. 4 and App. C.

###  2.2 Reinforcement Learning (RL)

While QAOA defines a variational ansatz to prepare ground states in a unitary
process, it does not yet provide a self-contained optimization procedure to
find the optimal protocol durations. A universal optimization framework is
presented by RL Sutton and Barto (2018).

Reinforcement learning comprises a powerful set of algorithms designed to
solve control problems. In RL, an agent aims to find a policy $\pi$ which
solves a specific task in a trial-and-error approach based on interactions
with the agent’s environment. Consider a finite-horizon Markov Decision
Process (MDP) defined by the tuple $(\mathcal{S},\mathcal{A},p,r)$ where
$\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces, respectively,
and $p:\mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ defines
the transition probability which governs the environment dynamics. Upon
selecting an action $a\in\mathcal{A}$, the environment transitions to a new
state $s\to s^{\prime}\in\mathcal{S}$, and emits a reward $r:S\times
A\rightarrow\mathbb{R}$, which the RL agent uses to select subsequent actions.
The action $a_{j}\in\mathcal{A}$ to be selected in a given state
$s\in\mathcal{S}$ is determined probabilistically by the instantaneous policy
$\pi(a_{j}|s_{j}):\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$. For a given
policy $\pi$, this process generates a trajectory
$\tau=(s_{1},a_{1},....,a_{q},s_{q+1})$ with probability
$\tau\sim\mathbb{P}^{\pi}(\cdot)$. Here,
$\mathbb{P}^{\pi}(\tau)=p_{0}(s_{1})\pi(a_{1}|s_{1})p(s_{2}|s_{1},a_{1})\cdots\pi(a_{q}|s_{q})p(s_{q+1}|s_{q},a_{q})$,
the episode/trajectory length is $q$, and $p_{0}$ is the initial state
distribution. The objective in RL is to find the optimal policy, i.e. the
policy which maximizes the total expected return:
$\mathbb{E}_{\tau\sim\mathbb{P}^{\pi}}\left[\sum_{j=1}^{q}r(s_{j},a_{j})\right].$

###  2.3 Policy Gradient Quantum Approximate Optimization Algorithm (PG-QAOA)

A reinforcement learning based approach to QAOA was recently introduced in
Ref. Yao et al. (2020a), using a policy gradient algorithm. The basic idea
behind PG-QAOA is to let the RL agent select the durations
$\\{\alpha_{j},\beta_{j}\\}$, which constitute a continuous action space
$\mathcal{A}^{\mathrm{c}}$. However, casting the quantum control problem
within the RL framework comes with certain challenges. The first challenge is
that quantum states cannot be directly measured in experiments, which poses
questions about the proper definition of the RL state space. To remedy this in
an environment following deterministic Schrödinger dynamics, it was suggested
to fix the initial quantum state, and define the RL state as the trajectory of
actions
$s_{j}=(a^{\mathrm{c}}_{1},\cdots,a^{\mathrm{c}}_{j-1})=(\alpha_{1},\beta_{1},\cdots)$
up to episode step $j$ Bukov (2018); this definition is inferior to using the
full state, but it allows to accommodate the experimental constraint, so we
adopt it in this study as well. Alternatively, one could use the expectation
values of observables to define an RL state Wauters et al. (2020). The second
challenge is the sparsity of the reward signal – a quantum measurement is
allowed only once at the end of each episode, since projective measurements
collapse the quantum wavefunction and the quantum state is lost irreversibly.

Since the protocol durations are continuous degrees of freedom, we need an RL
method for continuous optimization. PG-QAOA defines the simplest ansatz:
$q\\!=\\!2p$ independent Gaussian distributions to parameterize the policy,
one for each duration $\\{\alpha_{j},\beta_{j}\\}_{j=1}^{p}$ in Eq. (2). Since
a Gaussian distribution is uniquely determined by its mean $\mu$ and standard
deviation $\sigma$, we need a total of $2p$ independent variational parameters
${\bf{\bm{\theta}}}=\\{\mu_{\alpha_{j}},\sigma_{\alpha_{j}},\mu_{\beta_{j}},\sigma_{\beta_{j}}\\}_{j=1}^{p}$
to parametrize the policy $\pi_{{\bm{\theta}}}$ as:

$\pi_{{\bm{\theta}}}(\\{\alpha_{j},\beta_{j}\\}_{j=1}^{p})=\prod_{j=1}^{p}\pi(\alpha_{j};\kappa_{\alpha_{j}},\xi_{\alpha_{j}})\pi(\beta_{j};\kappa_{\beta_{j}},\xi_{\beta_{j}}),$
(4)

where $\kappa_{\alpha_{j}}=\mu_{\alpha_{j}}$,
$\kappa_{\beta_{j}}=\mu_{\beta_{j}}$ are the means, and
$\xi_{\alpha_{j}}=\sigma_{\alpha_{j}}$, $\xi_{\beta_{j}}=\sigma_{\beta_{j}}$
are the variances of the Gaussian policy. The actual protocol durations are
thus sampled according to
$\alpha_{j}\sim\mathcal{N}(\mu_{\alpha_{j}},\sigma^{2}_{\alpha_{j}})$, and
similarly for $\beta_{j}$. As was shown in Ref. Yao et al. (2020a), despite
its simplicity, PG-QAOA defines a particularly noise-robust algorithm. In the
presence of various kinds of noise, it readily outperforms a number of
alternative gradient-free optimization algorithms.

PG-QAOA

CD-QAOA

Figure 1:  Schematic diagram for PG-QAOA [left, see Sec. 2.3] and CD-QAOA
[right, see Sec. 2.4]. The PG-QAOA samples continuous QAOA-angles from its
policy and variationally updates the policy parameters via policy gradient;
CD-QAOA autoregressively samples the gate sequences for the generalized QAOA
ansatz and employs the gradient-free solver (Powell algorithm) to solve for
their corresponding durations. The policy network is updated via Proximal
Policy Optimization (PPO). For a comparison with RL-QAOA, cf. Fig. 2.

For this study, the original PG-QAOA implementation Yao et al. (2020a) is not
directly applicable, and a modification is required. First, the extensive
scaling with increasing the number of qubits suggests us to use as a cost
function the energy density, rather than the many-body fidelity; in doing
this, the algorithm no longer requires an explicit reference to the target
ground state we are searching for. Second, the original PG-QAOA algorithm does
not support an easy implementation of the protocol duration constraint
$\sum_{j=1}^{p}(\alpha_{j}+\beta_{j})\\!=\\!T$. Here, in order to do a fair
comparison among different algorithms, we enforce this constraint. Note that
this is a non-trivial task for policy gradient, for three reasons: (i)
protocol durations are sampled from a Gaussian distribution which has
unbounded support, (ii) a Gaussian policy supports negative as well as
positive samples (yet we require $\alpha_{j},\beta_{j}\geq 0$ for a physical
time duration), and (iii) sampled values, even if bounded and nonnegative, are
always random, and hence one needs to additionally fix their total sum. We
consider two different approaches to resolving (i) and (ii), and apply a
normalization trick to fix (iii).

The first approach we consider is to define the policy using the Beta
distribution Chou et al. (2017), i.e.
$\alpha_{j},\beta_{j}\sim\mathrm{B}(\kappa,\xi)$, instead of a Gaussian, and
learn the two nonnegative parameters $\kappa,\xi$. Since the Beta distribution
is defined on the interval $\mathcal{A}^{\mathrm{c}}=[0,1]$, it solves the
boundedness and positivity problems. The policy is given by Eq. (4) with
$\pi(x;\kappa,\xi)=\frac{\Gamma(\kappa+\xi)}{\Gamma(\kappa)\Gamma(\xi)}x^{\kappa-1}(1-x)^{\xi-1}$
the probability density for the Beta distribution; $\Gamma$ denotes the Gamma
function. Note that the number of independent variational parameters
${\bf{\bm{\theta}}}=\\{\kappa_{\alpha_{j}},\xi_{\alpha_{j}},\kappa_{\beta_{j}},\xi_{\beta_{j}}\\}_{j=1}^{p}$,
remains equal to $4p$.

In the second approach, we pass the output of the Gaussian distribution
through a sigmoid activation function Haarnoja et al. (2018b). Due to the
boundedness of the sigmoid function, this restricts the range of all
actions/durations to the nonnegative interval
$\mathcal{A}^{\mathrm{c}}=[0,1]$. Hence, our policy is given by Eq. (4) where
$\pi(x;\kappa,\xi)\\!=\\!\frac{1}{x(1\\!-\\!x)}\frac{1}{\sqrt{2\pi\xi^{2}}}\exp\left(-\frac{(\operatorname{logit}(x)-\kappa)^{2}}{2\xi^{2}}\right)$
is the probability density for the Sigmoid Gaussian distribution
$\mathcal{SN}(\kappa,\xi^{2})$111$\mathcal{SN}(\kappa,\xi^{2})$ is short-hard
notation for Gaussian distribution $\mathcal{N}(\kappa,\xi^{2})$ under the
sigmoid transformation $f(x)$.. Here, the logit function,
$\operatorname{logit}(x)\\!=\\!\log x-\log(1-x)$, is the inverse of the
sigmoid function $f(x)=1/(1+\exp(-x))$, and the factor $\frac{1}{x(1-x)}$ is
the inverse Jacobian of $x\\!=\\!f(y)$ over $y$ [cf. App. B]. Notice how, the
action output of this policy is forced within the interval $[0,1]$ by
construction, without changing the total number of independent variational
parameters ${\bm{\theta}}$.

Finally, to fix the total protocol duration, (iii), we normalize the sum of
durations manually according to
$\alpha_{j}\\!=\\!\frac{\alpha_{j}}{\sum_{j\\!=\\!1}^{p}(\alpha_{j}\\!+\\!\beta_{j})}T,\;\beta_{j}\\!=\\!\frac{\beta_{j}}{\sum_{j\\!=\\!1}^{p}(\alpha_{j}\\!+\\!\beta_{j})}T$.
We note that the normalization procedure is considered part of the RL
environment, i.e. no gradients are passed through it. In essence, it becomes
part of the reward function. This requires us to slightly re-define the
meaning of the policy: it generates the bare protocol durations before the
normalization; to minimize energy, the durations need the extra normalization.
We mention in passing that this is not the only way to hold the protocol
duration $T$ fixed: alternatives include using the constraint to fix the last
protocol duration $\beta_{p}$, or the addition of an extra penalty term to the
cost function.

###  2.4 Quantum Approximate Optimization Ansatz based on Counter-Diabatic
Driving

In conventional QAOA, there are two possible gates, corresponding to the two
unitaries $U_{j}=\exp(-i\alpha_{j}H_{j}),j=1,2$. Therefore, there exist only
two distinct sequences of unitaries:
$\tau^{\mathrm{d}}_{1}=U_{1}U_{2}U_{1}U_{2}\cdots$ and
$\tau^{\mathrm{d}}_{2}=U_{2}U_{1}U_{2}U_{1}\cdots$. A generalization of this
ansatz was considered in Ref. Yao et al. (2020b), where an RL agent was given
the complex combinatorial task to construct the sequence of unitaries
$\tau^{\mathrm{d}}$, out of a predefined set $\mathcal{A}^{\mathrm{d}}$ of
$|\mathcal{A}^{\mathrm{d}}|$ gates/unitaries. As for the gate duration
notation, we will use $\alpha_{j}$ for all durations instead of alternating
$\alpha_{j}$, $\beta_{j}$ due to a general ansatz. This set can, in principle,
be chosen arbitrarily; however, one can also make a more physics-informed
choice, e.g., inspired by counter-diabatic driving in the case of quantum-
many-body systems [cf. Sec. 2.4.1]. In the latter case, the resulting
generalized algorithm, called CD-QAOA, was demonstrated to drastically enhance
the variational ansatz of QAOA when applied to many-body quantum chains,
allowing for shorter circuit depths at no cost in performance Yao et al.
(2020b).

Similar to PG-QAOA, CD-QAOA does _not_ use the quantum wavefunction to perform
the optimization, and the state is
$s_{j}=(a^{\mathrm{d}}_{1},\cdots,a^{\mathrm{d}}_{j-1})$ at episode step $j$.
Rewards are given once per episode, in the end, and are defined by the
(negative) energy density. However, the action space is given by the set of
$|\mathcal{A}^{\mathrm{d}}|$ unitary gates from which the protocol sequence
$\tau^{\mathrm{d}}$ are selected; it does not involve the continuous protocol
durations which are found as part of the RL environment; to do this, in this
study, we use the gradient-free Powell algorithm Powell (1964) instead of the
gradient-based SLSQP algorithm Kraft et al. (1988) presented in the original
CD-QAOA paper.

Apart from the low-level optimization mentioned above, CD-QAOA adopts a two-
level optimization schedule Li et al. (2020); Melnikov et al. (2020): high-
level discrete optimization is used to construct the optimal sequence
$\tau^{\mathrm{d}}$ out of the available set of unitaries. For this purpose,
in Ref. Yao et al. (2020b), it was suggested to employ Proximal Policy
OptimizationSchulman et al. (2017) (PPO), an advanced variant of policy
gradient, aided by a deep autorgressive neural network to implement causality:

$\pi_{\bm{\theta}}^{\mathrm{d}}\left(a^{\mathrm{d}}_{1},a^{\mathrm{d}}_{2},\cdots,a^{\mathrm{d}}_{q}\right)=\pi_{\bm{\theta}}^{\mathrm{d}}\left(a^{\mathrm{d}}_{1}\right)\prod_{j=2}^{q}\pi_{\bm{\theta}}^{\mathrm{d}}\left(a^{\mathrm{d}}_{j}\mid
a^{\mathrm{d}}_{1},\cdots,a^{\mathrm{d}}_{j-1}\right).$ (5)

Each factor in the product above is a categorical distribution over the action
space. We point out that the search for the optimal sequence
$\tau^{\mathrm{d}}$ represents a discrete optimization problem. This should be
contrasted with the low-level continuous optimization employed by QAOA to find
the optimal durations $\\{\alpha_{j}\\}_{j=1}^{q}$, carried out using the
Powell solver. Since the Powell solver only deals with the bounded
optimization, we apply the same normalization trick to enforce the total
duration constraint.

Given the complete protocol sequence
$\tau^{\mathrm{d}}=(a^{\mathrm{d}}_{1},\cdots,a^{\mathrm{d}}_{q})$, we can
construct the unitary process

$U(\\{\alpha_{j}\\}_{j=1}^{q},\tau^{\mathrm{d}})\\!=\\!\prod_{j=1}^{q}U_{\tau_{j}^{\mathrm{d}}}(\alpha_{j})$
(6)

which we use as a generalized QAOA ansatz. The sequence $\tau^{\mathrm{d}}$,
and the durations $\\{\alpha_{j}\\}_{j=1}^{q}$ are found by minimizing the
energy density, cf. Eq. (2). In doing so, we impose an extra constraint that
the same action cannot be taken twice in a row, for otherwise one can add the
corresponding durations and optimize them together.

####  2.4.1 Possible Choice of Unitaries based on Counter-Diabatic Driving

In order to construct the unitary
$U(\\{\alpha_{j}\\}_{j=1}^{q},\tau^{\mathrm{d}})=\prod_{j=1}^{q}\exp(-i\alpha_{j}H_{\tau^{\mathrm{d}}_{j}})$
from Eq. (6), the RL agent needs to select the sequence $\tau^{\mathrm{d}}$ of
subprocess generators $\mathcal{A}^{\mathrm{d}}$. Hence, at every step $j$ in
the RL episode, the agent’s action consists of a choice of a Hermitian
operator $H_{\tau^{\mathrm{d}}_{j}}\in\mathcal{A}^{\mathrm{d}}$

The set of discrete actions $\mathcal{A}^{\mathrm{d}}$ consists of the
available possible controls in an experiment. In Refs. Yao et al. (2020b);
Hegade et al. (2020); Ding et al. (2020), it was shown that a particularly
suitable choice of actions for ground state preparation in quantum many-body
systems, is given by terms appearing in the series of the variational
adiabatic gauge potential, designed for many-body counter-diabatic driving
Sels and Polkovnikov (2017). These terms provide shortcuts in the Hilbert
space that may significantly decrease the time required to prepare the ground
state. For brevity, below we just list the generator set
$\mathcal{A}^{\mathrm{d}}$ for the spin$-1/2$ Ising model, cf. Eq. (3), which
the RL agent has access to, and refer the interested readers to Ref. Yao et
al. (2020b) for more details:
$\mathcal{A}^{\mathrm{d}}=\\{H_{1},H_{2},Y,X|Y,Y|Z\\}$, with
$Y=\sum_{i}S^{y}_{i}$,
$X|Y=\sum_{i}S^{x}_{i}S^{y}_{i+1}+S^{y}_{i}S^{x}_{i+1}$, and
$Y|Z=\sum_{i}S^{y}_{i}S^{z}_{i+1}+S^{z}_{i}S^{y}_{i+1}$; $H_{1},H_{2}$ are
defined in Eq. (3).

We emphasize that this is just one particular choice for
$\mathcal{A}^{\mathrm{d}}$. In practice, the algorithm is agnostic to the
discrete action space which is determined by the available controls for the
system of interest: e.g., on a quantum computer, these can be the set of local
gates, etc.

Method | QAOA | PG-QAOA | CD-QAOA | RL-QAOA
---|---|---|---|---
protocol sequence | ✗ | ✗ | $\nabla$-free |
optimization (discrete) | $\nabla$-free
gate durations | $\nabla$-free | $\nabla$-free | $\nabla$-free |
optimization (continuous) | $\nabla$-free
RL optimization | ✗ | continuous | discrete | continuous & discrete
noise-robust | ✗ | ✓ | ✗ | ✓
autoregressive | ✗ | ✗ | ✓ | ✓
Table 1: Comparison between all four algorithms: QAOA, PG-QAOA, CD-QAOA and
RL-QAOA.

##  3 Mixed Discrete-Continuous Policy Gradient using Deep Autoregressive
Networks

Although RL is used as an optimizer in both PG-QAOA and CD-QAOA, it serves two
fundamentally different purposes. In PG-QAOA it is employed for continuous
optimization of the protocol durations $\\{\alpha_{j}\\}$, while in CD-QAOA it
is used to find the solution to the discrete combinatorial task of ordering
the unitaries in the protocol sequence. In this section, we illustrate how to
combine the two aspects together into a unified RL-based algorithm.

We have seen that with the help of RL one can tremendously enhance the
properties of the QAOA ansatz in very different ways, cf. Table 1: For
instance, PG-QAOA has the important desired property that it is robust to
noise. Moreover, it does a completely gradient-free optimization of the
continuous protocol durations. On the other hand, CD-QAOA, enhances the
variational ansatz itself by offering the appealing ability to select the
order in which three or more unitaries can be applied in the protocol
sequence. Moreover, it also introduces an autoregressive deep neural network
to encode causality (i.e., which unitary is optimal at a given episode step
depends on the unitaries chosen hitherto). The imminent question arises as to
whether we can design an algorithm which makes the best of both worlds.

###  3.1 Autoregressive Policy Ansatz for Hybrid Discrete-Continuous Action
Spaces

Recently, a number of studies have considered the problem of simultaneous
discrete/continuous control using RL Kulkarni et al. (2016); Fan et al.
(2019); Wei et al. (2018); Hausknecht and Stone (2015); Delalleau et al.
(2019); Bester et al. (2019); Xiong et al. (2018); Fan et al. (2019); Wei et
al. (2018); Neunert et al. (2020). Following the notation of Ref. Masson et
al. (2015), we describe the RL problem within the framework of parametrized-
action Markov decision processes (PAMDPs). The major difference, compared to
ordinary MDPs, is the definition of the action space:
$\mathcal{A}=\bigcup_{a^{\mathrm{d}}\in\mathcal{A}^{\mathrm{d}},a^{\mathrm{c}}\in\mathcal{A}^{\mathrm{c}}}(a^{\mathrm{d}},a^{\mathrm{c}}),\;\mathcal{A}^{\mathrm{d}}=\\{H_{j}\\}_{j=1}^{|\mathcal{A}^{\mathrm{d}}|},\;\mathcal{A}^{\mathrm{c}}=[0,1],$
where $|\mathcal{A}^{\mathrm{d}}|$ denotes the cardinality of the discrete
action set. As before, the state space contains all possible sequences of
actions, and the reward is the (negative) energy density of the quantum state,
given once at the end of the protocol.

In this section, we present a unified continuous-discrete quantum control
algorithm, called RL-QAOA, based on a hybrid policy which optimizes
simultaneously the discrete and continuous degrees of freedom in the policy.
The policy can be decomposed as a product of two coupled auxiliary policies –
one for the continuous actions, $\pi_{\bm{\theta}}^{\mathrm{c}}$, and the
other for the discrete actions, $\pi_{\bm{\theta}}^{\mathrm{d}}$:

$\pi_{{\bm{\theta}}}(\tau)=\pi_{\bm{\theta}}^{\mathrm{c}}\left(\tau^{\mathrm{c}}\right)\pi_{\bm{\theta}}^{\mathrm{d}}\left(\tau^{\mathrm{d}}\right),$
(7)

where
$\tau^{\nu}=(a^{\nu}_{1},\dots,a^{\nu}_{q}),\;\nu\in\\{\mathrm{c},\mathrm{d}\\}$
defines the discrete/continuous subsequence of actions in each trajectory of
length $q$. Denoting, as before, the RL state by
$s_{j}\\!=\\!(a_{1},\cdots,a_{j-1})$ with the hybrid action
$a_{\ast}\\!=\\!(a^{\mathrm{c}}_{\ast},a^{\mathrm{d}}_{\ast})$, we define a
generalized continuous/discrete autoregressive model for the policy, following
Eq. (5). Adopting the short-hand notation
$\pi^{\nu}_{\boldsymbol{\theta}}\left(a_{j}^{\nu}\mid
s_{j}\right)\\!=\\!\pi^{\nu}_{\boldsymbol{\theta}}\left(a_{j}^{\nu}\mid
a_{1},\cdots,a_{j-1}\right)$, the policy can be written as

$\pi_{\boldsymbol{\theta}}\left(a_{1},a_{2},\cdots,a_{q}\right)\\!=\\!\prod_{j\\!=\\!1}^{q}\pi_{\bm{\theta}}^{\mathrm{d}}\left(a^{\mathrm{d}}_{j}\mid
s_{j}\right)\pi_{\bm{\theta}}^{\mathrm{c}}\left(a^{\mathrm{c}}_{j}\mid
s_{j},a^{\mathrm{d}}_{j}\right).$ (8)

As expected, at every step $j$, the action $a^{\mathrm{c}}_{j}$ is sampled
from a continuous distribution, whose parameters depend on the discrete action
$a^{\mathrm{d}}_{j}$ selected at the same step $j$. This is natural, since
different discrete actions may require different corresponding continuous
distribution parameters $\kappa,\xi$.

Additionally, similar to CD-QAOA, we impose a further restriction that no
discrete action can occur in the trajectory consecutively. We use a Sigmoid-
Gaussian distribution to bound the samples for the continuous actions, and
normalize the durations $\alpha_{j}\propto
a^{\mathrm{c}}_{j}\sim\pi_{\bm{\theta}}^{\mathrm{c}}$ to fix the total
protocol duration to $\sum_{j=1}^{q}\alpha_{j}=T$; using the Beta distribution
instead results in a similar performance.

###  3.2 Deep Autoregressive Policy Network

We implement the policy ansatz variationally, using a deep neural network
called the policy network. In Fig. 2, we show a cartoon of the model for
illustration purposes. The network consists of base layers with intermediate
output $\boldsymbol{y}$, followed by three independent head layers with
outputs $\boldsymbol{z}^{p},\boldsymbol{z}^{\kappa},\boldsymbol{z}^{\xi}$,
respectively. The three heads learn the discrete probability distribution
$\pi^{\mathrm{d}}$, and the parameters $\kappa,\xi\in\mathbb{R}^{+}$ which
define the continuous probability distribution $\pi^{\mathrm{c}}$. Each head
outputs a vector of size $|\mathcal{A}^{\mathrm{d}}|$ – so that the model can
learn a set $(\kappa,\xi)$ for every distinct discrete action. Notice that
each head output depends on the joint base layer parameters
$(\boldsymbol{W},\boldsymbol{b})$, but not on the parameters
$(\boldsymbol{V},\boldsymbol{c})$ of any of the other two heads; thus, the
base layers are shared by all three heads. In practice, we find that a base
layer, comprised of two hidden layers, can already achieve a good performance;
one can in principle add more layers for enhanced expressivity.

Figure 2:  Schematic representation of RL-QAOA and the deep autoregressive
network for $q=4$ (see text). The time step $j$ also corresponds to the gate
index. The policy network is composed of (i) an embedding layer to encode the
continuous and discrete actions as input. (ii) The base layer implements the
causal autoregressive structure (see arrows). (iii) The heads are three-fold,
one for the discrete distribution parameters, and two for the continuous
distribution parameters. A batch of actions are sampled to evolve the quantum
state and compute the negative energy density as a reward. Proximal hybrid
Policy Optimization (PPO) is used to update the policy network. The pseudocode
for RL-QAOA is shown in Algorithm 1.

The above description focuses on a single episode step $j$ out of a total of
$q$ steps in an episode. The autoregressive feature of the ansatz can then be
built-in, by allowing the outputs of the base layers from previous steps to
become inputs into the layers at subsequent episode steps [Fig. 2].

Let us denote the input to the autogressive network by
$(x_{1},x_{2},\cdots,x_{q})$, and the weights and bias parameters of the base
layer by
$W_{j}\in\mathbb{R}^{d_{\text{h}}\times(i-1)|\mathcal{A}^{\mathrm{d}}|}$ and
$b_{j}\in\mathbb{R}^{d_{\text{h}}}$, respectively, where $d_{\text{h}}$ is the
hidden dimension. Then, the intermediate output $(y_{1},y_{2},\cdots,y_{q})$
of the base layer reads as

$y_{j}\\!=\\!g(W_{j}x_{<j}+b_{j}),\quad j\\!=\\!1,2,\cdots q,$ (9)

where
$x_{<j}=\left(x_{j-1},\cdots,x_{1}\right)^{T}\in\mathbb{R}^{(j-1)|\mathcal{A}^{\mathrm{d}}|}$
denotes the input of all previous steps preceding step $j$; for $j\\!=\\!1$,
we set $W_{j}x_{<j}+b_{j}\\!=\\!b_{j}$.222In practice, implementing the
autoregressive constraint <j can be achieved using masks (one for each set of
weights). We use ReLU nonlinearities $g(\cdot)$.

The output of the base layer $(y_{1},y_{2},\cdots,y_{q})$ can be viewed as an
input to the three-head layer. The three-head layer contains three heads with
independent weights
$V_{j}^{p},V_{j}^{\kappa},V_{j}^{\xi}\in\mathbb{R}^{|\mathcal{A}^{\mathrm{d}}|\times
jd_{\text{h}}}$ and biases
$c^{p}_{j},c^{\kappa}_{j},c^{\xi}_{j}\in\mathbb{R}^{|\mathcal{A}^{\mathrm{d}}|}$.
The three-head layer output, $(z_{1},z_{2},\cdots,z_{q})$, are the parameters
for the discrete and continuous distributions: $z_{j}^{p}$ are the categorical
distribution parameters; $z_{j}^{\kappa}$ and $z_{j}^{\xi}$ are the two
parameters for the sigmoid-Gaussian distribution [cf. App. B.1]:

$z_{j}^{p}\\!=\\!\log\left(\operatorname{SoftMax}(V_{j}^{p}y_{\leq
j}+c^{p}_{j})\right),\qquad z_{j}^{\kappa}\\!=\\!V_{j}^{\kappa}y_{\leq
j}+c^{\kappa}_{j},\quad z_{j}^{\xi}\\!=\\!\exp(V_{j}^{\xi}y_{\leq
j}+c^{\xi}_{j}),$ (10)

where $y_{\leq
j}=\left(y_{j},\cdots,y_{1}\right)^{T}\in\mathbb{R}^{jd_{\text{h}}}$.333Note
that here we are able to use the ”$=$” sign because the previous layer of
operation has already filtered out the ”$=$” sign for those steps. To define a
categorical distribution, we use a SoftMax444Note that this function is not
operated element-wise like the others; it is applied on the whole vector of
dimension $|\mathcal{A}^{\mathrm{d}}|$). nonlinarity:
$\operatorname{SoftMax}(v)[i]\\!=\\!\exp(v[i])/\sum_{k=1}^{|\mathcal{A}^{\mathrm{d}}|}\exp(v[k])$,
where $v\\!=\\!V_{j}^{p}y_{\leq
j}+c^{p}_{j}\in\mathbb{R}^{|\mathcal{A}^{\mathrm{d}}|}$, and $[\cdot]$ takes
the index; we learn the log-probability to achieve a resolution over a few
orders of magnitude, and to stabilize the learning process.

We apply ancestral sampling to draw actions from the autoregressive policy.
Starting from the heads layer at step $j=1$, we first sample
$a^{\mathrm{d}}_{1}\sim\pi(a^{\mathrm{d}}_{1})\\!=\\!\operatorname{Categorical}(\exp(z_{1}^{p}))$;
we use the sampled discrete action $a^{\mathrm{d}}_{1}$ to look-up the
corresponding parameters $\kappa=z_{1}^{\kappa}[a^{\mathrm{d}}_{1}]$ and
$\xi=z_{1}^{\xi}[a^{\mathrm{d}}_{1}]$555Here, $[a^{\mathrm{d}}_{1}]$ means
taking the component by index. for the continuous action distribution. Then we
sample the duration $\alpha_{1}\propto
a^{\mathrm{c}}_{1}\sim\pi(a^{\mathrm{c}}_{1}|a^{\mathrm{d}}_{1})\\!=\\!\mathcal{SN}\left(z_{1}^{\kappa}[a^{\mathrm{d}}_{1}],{(z_{1}^{\xi}}[a^{\mathrm{d}}_{1}])^{2}\right)$.
The sampling output is passed as an input at the second step $j=2$. To do
this, we use as an embedding for $(a^{\mathrm{d}}_{1},a^{\mathrm{c}}_{1})$
represented by the variable $x_{1}$, where $x_{1}[i]=a^{\mathrm{c}}_{1}\text{
if }i\\!=\\!a^{\mathrm{d}}_{1}$, and $x_{1}[i]=0$ otherwise. Going on, we
repeat the process: we sample successive actions
$a^{\mathrm{d}}_{2},a^{\mathrm{c}}_{2}\sim\pi(a^{\mathrm{d}}_{2}|x_{1}),\pi(a^{\mathrm{c}}_{2}|x_{1},a^{\mathrm{d}}_{2})$.
The sampling, or forward pass, through the network is then repeated $q$ times,
until we reach the end of the episode; thus, at step $j$ we have
$a^{\mathrm{d}}_{j},a^{\mathrm{c}}_{j}\sim\pi(a^{\mathrm{d}}_{j}|x_{<j}),\pi(a^{\mathrm{c}}_{j}|x_{<j},a^{\mathrm{d}}_{j})$.
This gives the trajectory $\tau$ of mixed discrete-continuous actions. Note
that the time complexity of the process is
$\mathcal{O}(q\times|\mathcal{A}^{\mathrm{d}}|)$.

###  3.3 Proximal Hybrid Policy Optimization

The set of all weights and biases,
${\bm{\theta}}\\!=\\!\\{W_{j},b_{j},V_{j}^{p},V_{j}^{\kappa},V_{j}^{\xi},c^{p}_{j},c^{\kappa}_{j},c^{\xi}_{j}\\}_{j\\!=\\!1}^{q}$,
defines the learnable parameters of the autoregressive policy network. We now
discuss how to compute the policy gradients and define an update rule for
${\bm{\theta}}$.

Our goal is to maximize the RL objective within the trust region Schulman et
al. (2015) for the continuous and discrete policy

$\mathbb{E}_{\tau}\left[\frac{\pi_{{\bm{\theta}}}\left(\tau\right)}{\pi_{{\bm{\theta}}_{\text{old}}}\left(\tau\right)}{A_{{\bm{\theta}}_{\mathrm{old}}}(\tau)}\right],\quad\text{subject
to}\quad{\mathbb{E}}_{\tau}\left[\operatorname{D_{KL}}\left[\pi^{\nu}_{{\bm{\theta}}_{\text{old
}}}\left(\cdot\right),\pi^{\nu}_{{\bm{\theta}}}\left(\cdot\right)\right]\right]\leq\delta^{\nu},$
(11)

where $\mathop{\mathbb{E}}_{\tau}[\ \cdot\ ]$ is a shorthand notation for
$\mathop{\mathbb{E}}_{\tau\\!=\\!(a_{1},\cdots,a_{q})\sim\pi_{\bf{\bm{\theta}}_{\mathrm{old}}}}[\
\cdot\ ]$. The Kullback–Leibler (KL) divergence is defined as
$\operatorname{D_{\text{KL}}}(\pi_{\bm{\theta}}^{\mathrm{c}},\pi_{{\bm{\theta}}_{\mathrm{old}}}^{\mathrm{c}})\\!=\\!\int_{x\in\mathcal{A}^{\mathrm{c}}}\pi_{\bm{\theta}}^{\mathrm{c}}(x)\log\left(\frac{\pi_{\bm{\theta}}^{\mathrm{c}}(x)}{\pi_{{\bm{\theta}}_{\mathrm{old}}}^{\mathrm{c}}(x)}\right)\differential{x}$,
and similarly for $\nu\\!=\\!\mathrm{d}$; $\delta^{\nu}$ defines a constraint
on the size of the discrete policy or continuous policy updates in
distribution space. Here, ${\bm{\theta}}_{\textrm{old}}$ denotes the
parameters before the update;
$A_{{\bm{\theta}}_{\mathrm{old}}}(\tau)=R(\tau)-b$ is the advantage function –
the return (negative energy density) for a given trajectory w.r.t. the
baseline $b$.

In practice, we utilize a clipped surrogate RL objective Schulman et al.
(2017) with two clipping parameters $\epsilon^{\nu}$. The idea is to update
the continuous and discrete policies using different $\epsilon^{\nu}$ during
policy optimization. This allows for the discrete policy
$\pi_{\bm{\theta}}^{\mathrm{d}}$ to change more quickly/more slowly as
compared to the continuous policy $\pi_{\bm{\theta}}^{\mathrm{c}}$. Hence, the
hybrid PPO RL objective reads as

$\mathcal{J}({\bm{\theta}})=\mathbb{E}_{\tau}\bigg{[}\mathcal{G}^{\mathrm{d}}(\tau^{\mathrm{d}};{\bm{\theta}},\epsilon^{\mathrm{d}})+\mathcal{G}^{\mathrm{c}}(\tau^{\mathrm{c}};{\bm{\theta}},\epsilon^{\mathrm{c}})\bigg{]}+\beta^{-1}_{{S}}(\mathcal{S}^{\mathrm{d}}+\mathcal{S}^{\mathrm{c}}),$
(12)

with

$\mathcal{G}^{\nu}(\tau^{\nu};{\bm{\theta}},\epsilon^{\nu})=\min\bigg{\\{}\rho_{\bm{\theta}}^{\nu}(\tau^{\nu})A^{\nu}_{{\bm{\theta}}_{\mathrm{old}}}(\tau^{\nu}),\;\operatorname{clip}\left(\rho_{\bm{\theta}}^{\nu}(\tau^{\nu}),1-\epsilon^{\nu},1+\epsilon^{\nu}\right)A^{\nu}_{{\bm{\theta}}_{\mathrm{old}}}(\tau^{\nu})\bigg{\\}},$
(13)

where
$\rho_{\bm{\theta}}^{\nu}(\tau^{\nu})\coloneqq\frac{\pi^{\nu}_{{\bm{\theta}}}(\tau^{\nu})}{\pi^{\nu}_{{\bm{\theta}}_{\mathrm{old}}}(\tau^{\nu})}$
is the importance weight ratio of two policies associated with trajectory
$\tau^{\nu}$. The clip function, defined as
$\mathrm{clip}(\rho,x,y)=\max\big{(}\min\left(\rho,x\right),y\big{)}$ sets the
value of $\rho_{\bm{\theta}}$ to be within the interval $[x,y]$, and
constrains the likelihood ratio from Eq. (11) to the range
$[1-\epsilon,1+\epsilon]$. The entropy terms [right-most part of Eq. (12)] are
discussed below. Our goal is to find those parameters ${\bm{\theta}}$ which
maximize $\mathcal{J}({\bm{\theta}})$.

To understand the hybrid PPO algorithm, consider two limiting cases first. In
the extreme case when $\epsilon^{\mathrm{d}}\to 0$, i.e. the discrete policy
$\pi_{\bm{\theta}}^{\mathrm{d}}$ is kept fixed, our algorithm reduces to PG-
QAOA. On the other hand, when $\epsilon^{\mathrm{c}}\to 0$, the continuous
policy is kept fixed; if this fixed policy additionally corresponds to the
greedy “expert policy” defined by the Powell optimizer, the algorithm is
reduced to CD-QAOA. In this sense, for finite values of
$\epsilon^{\mathrm{c}},\epsilon^{\mathrm{d}}>0$, RL-QAOA can be viewed as a
smooth interpolation between PG-QAOA and CD-QAOA.

In order to incentivize the agent to explore the action space during the early
stages of training, we also added entropy to the RL objective, cf. Eq. (12).
The entropy for a discrete/continuous policy is defined as
$\mathcal{S}^{\mathrm{d}}(\pi^{\mathrm{d}})\\!=\\!-\sum_{x\in\mathcal{X}}\pi^{\mathrm{d}}(x)\log\pi^{\mathrm{d}}(x)$
or
$\mathcal{S}^{\mathrm{c}}(\pi^{\mathrm{c}})\\!=\\!-\int_{x\in\mathcal{X}}\pi^{\mathrm{c}}(x)\log\left(\pi^{\mathrm{c}}(x)\right)\differential
x$, respectively. The coefficient $\beta^{-1}_{S}$ in Eq. (12) defines an
effective temperature, which we anneal with increasing the number of
iterations. It is easy to see that the total entropy
$\mathcal{S}=\mathcal{S}^{\mathrm{d}}+\mathcal{S}^{\mathrm{c}}$ associated
with the hybrid policy consists of a discrete
$\mathcal{S}^{\mathrm{d}}=\sum_{j=1}^{q}\mathop{\mathbb{E}}_{a_{<j}\sim\pi_{\bm{\theta}}}\mathcal{S}^{\mathrm{d}}\big{(}\pi_{\bm{\theta}}^{\mathrm{d}}(\;\cdot\;|a_{<j})\big{)}$,
and a continuous
$\mathcal{S}^{\mathrm{c}}=\sum_{j=1}^{q}\mathop{\mathbb{E}}_{a_{<j}\sim\pi_{\bm{\theta}},a^{\mathrm{d}}_{j}\sim\pi_{\bm{\theta}}^{\mathrm{d}}}S^{\mathrm{c}}\big{(}\pi_{\bm{\theta}}^{\mathrm{c}}(\;\cdot\;|a_{<j},a^{\mathrm{d}}_{j})\big{)}$
contribution. The RL agent has to maximize the total expected return while
also maximizing the entropy associated with the policy.

In RL, there are two common ways to incorporate entropy in practice Levine
(2018): (i) whenever one can compute a closed-form expression for the entropy,
entropy is added as a separate term to the objective which can be thought of
as entropy regularization. Note that it is the autoregresssive structure that
makes it possible to obtain the exact value for the entropy
$\mathcal{S}^{\mathrm{d}}$: for
$\pi_{\bm{\theta}}^{\mathrm{d}}(\;\cdot\;|a_{<j})\\!=\\!\operatorname{Categorical}(\exp(z_{j}^{p}))$,
the entropy is
$\mathcal{S}^{\mathrm{d}}\big{(}\pi_{\bm{\theta}}^{\mathrm{d}}(\;\cdot\;|a_{<j})\big{)}\\!=\\!-\sum_{k\\!=\\!1}^{|\mathcal{A}^{\mathrm{d}}|}z_{j}^{p}[k]\cdot\exp(z_{j}^{p}[k])$.
(ii) Often times it is not possible to compute the value for the entropy,
since the expression is not analytically tractable; in such cases, the maximum
entropy formulation Haarnoja et al. (2018b, a, 2017) still allows us to add to
the reward a sample estimate of the entropy, known as an entropy bonus:
$R^{\mathrm{c}}(\tau)\leftarrow
R^{\mathrm{c}}(\tau)+\beta^{-1}_{S}\mathop{\mathbb{E}}_{a^{\mathrm{c}}\sim\pi_{\bm{\theta}}^{\mathrm{c}}}\big{[}-\log\pi_{\bm{\theta}}^{\mathrm{c}}\big{]}$.
In this study, we add an entropy bonus to take into account the entropy of the
continuous policy $\pi^{c}$.

##  4 Application: Quantum Ising Model in the Presence of Noise

To test the performance of RL-QAOA, we investigate the ground state
preparation problem for a system of $N$ interacting qubits (i.e. spin-$1/2$
degrees of freedom), described by the Ising Hamiltonian introduced in Eq. (3).
We use periodic boundary conditions and work in the zero momentum sector of
positive parity, which contains the antiferromagnetic ground state. We
emphasize that this model is non-integrable, i.e., it does not have an
extensive number of local integrals of motion; as a consequence, no closed-
form analytical description is known for its eigenstates and eigenenergies.
Moreover, the lack of integrability results in chaotic quantum dynamics.

In the following, $J\\!=\\!1$ sets the energy unit, $h_{z}/J\\!=\\!0.4523$ and
$h_{x}/J\\!=\\!0.4045$. In the thermodynamic limit, $N\to\infty$, these
parameters are close to the critical line of the model, where a quantum phase
transition occurs in the ground state between an antiferromagnet and a
paramagnet; for the finite system sizes we can simulate, the critical behavior
is smeared out over a small finite region. In Ref. Matos et al. (2020), using
QAOA it was shown that this region of parameter space appears most challenging
in the noise-free system.

We initialize the system in the $z$-polarized product state
$|\psi_{i}\rangle\\!=\\!|\\!\\!\uparrow\cdots\uparrow\rangle$, and aim to
prepare the ground state of $H$. We use the negative energy density
$-\mathcal{E}=-E/N$ as a reward for the RL agent, cf. Eq. (2), which is an
intensive quantity as the number of qubits $N$ increases. In this study, we
are mostly interested in exploring the behavior of the system, subject to
various kinds of noise/uncertainty. Our primary focus is quantifying the
effects of noise on the achievable fidelity, w.r.t. the noise-free values. We
deliberately select a fixed duration of $JT\\!=\\!10$ far from the adiabatic
regime, such as to exhibit the benefits of the CD-QAOA ansatz over QAOA [cf.
App. C].

We point out that, working at a fixed duration $T$, it is not always possible
to achieve high-fidelity ground states. This is easy to see for decoupled
qubits, where the magnitude of the spin precession frequency on the Bloch
sphere (so-called Larmor precession frequency) is set by the fixed strength of
the magnetic field $(h_{x},0,h_{z})$: hence, fixing the total protocol
duration $T$, it may be physically impossible to reach the target state in the
allotted time. This behavior leads to the notion of the quantum speed limit
(QSL) – the minimum time required to prepare the ground state with unit
fidelity.

###  4.1 Three Noise Models

When operating present-day quantum devices, one is confronted with various
sources of uncertainty. Since the exact form and details depend on the
peculiarities and particularities of the underlying experimental platform, it
is desirable to construct algorithms, capable of learning such details without
extra human input. In this study, our RL agent learns in a simulator. To mimic
the diversity of uncertain processes that can occur, we consider three types
of noise.

####  4.1.1 Classical Measurement Gaussian Noise

Noise naturally occurs due to imperfect measurements. For instance, the
measurement signal is often present in a form of currents and voltages, whose
values can only be determined within the resolution of the measurement
apparatus. In practice, experimentalists perform a large number of
measurements and average the result in the end to obtain an estimate for the
value of an observable. By the central limit theorem, in the limit of large
sample sizes, the statistics of the measurement data is approximated by a
Gaussian distribution. To model this behavior, we use small Gaussian noise to
add uncertainty in the reward signal:
$\mathcal{E}_{\gamma}(\\{\alpha_{i}\\}_{i=1}^{q},\tau^{\mathrm{d}})=\mathcal{E}(\\{\alpha_{i}\\}_{i=1}^{q},\tau^{\mathrm{d}})+\epsilon_{\gamma}$,
where $\epsilon_{\gamma}\sim\mathcal{N}(0,\gamma^{2})$.

####  4.1.2 Quantum Measurement Noise

In quantum mechanics, there is another, intrinsic, kind of noise, which arises
due to the quantum nature of the controlled system. Consider the evolved state
$|\psi(T)\rangle=U(\\{\alpha_{j}\\}_{j=1}^{q},\tau)|\psi_{i}\rangle$ at the
end of the protocol. The expected measurement for the energy density
$\mathcal{E}=N^{-1}\langle\psi(T)|H|\psi(T)\rangle$ is obtained within a
quantum uncertainty,
$\Delta\mathcal{E}=N^{-1}\sqrt{\langle\psi(T)|H^{2}|\psi(T)\rangle-\langle\psi(T)|H|\psi(T)\rangle^{2}}$,
set by the energy variance in the final state. In the limit of a large number
of measurements, quantum noise can be simulated using a Gaussian distribution
$\mathcal{E}_{Q}(\\{\alpha_{i}\\}_{i=1}^{q},\tau^{\mathrm{d}})=\mathcal{E}(\\{\alpha_{i}\\}_{i=1}^{q},\tau^{\mathrm{d}})+\epsilon_{Q}$,
where $\epsilon_{Q}\sim\mathcal{N}(0,\Delta\mathcal{E}^{2})$. Note that the
width of the Gaussian depends on the final state $|\psi(T)\rangle$: in the
early stages of training, $|\psi(T)\rangle$ is typically far away from any of
the eigenstates of $H$; therefore, the energy variance $\Delta\mathcal{E}$
will be large and finite. However, towards the later training stages, when the
agent learns to prepare a state close to the target ground state, the energy
variance will go down. Hence, one can think of the quantum noise as a Gaussian
noise with a time-dependent strength.

####  4.1.3 Noise arising from Gate Rotation Errors

Finally, we also consider the uncertainty in implementing the unitaries
$U_{i}$. We focus on gate rotation errors Sung et al. (2020), caused by
imperfections in the durations $\alpha_{i}$:
$\mathcal{E}_{\delta}(\\{\alpha_{i}\\}_{i=1}^{q},\tau)=\mathcal{E}(\\{\alpha_{i}+\epsilon_{i}\\}_{i=1}^{q})$,
where $\epsilon_{i}\sim\mathcal{N}(0,\delta^{2})$. This defines a simplified
error model for coherent control, an important source of errors in present-day
state-of-the-art quantum computing hardware Arute et al. (2019), and which is
especially pertinent to the case of quantum computers which are utilized
frequently but calibrated only periodically.

##  5 Numerical Experiments and Results

QAOA PG-QAOA CD-QAOA RL-QAOA

Figure 3:  Energy minimization against different noise levels with circuit
depths $p\\!=\\!q/2\\!=\\!4$ and protocol duration $JT\\!=\\!10$ for four
different optimization methods: QAOA, PG-QAOA, CD-QAOA (red line), RL-QAOA.
The initial and target states are
$|\psi_{i}\rangle\\!=\\!|\\!\\!\uparrow\cdots\uparrow\rangle$ and
$|\psi_{\ast}\rangle\\!=\\!|\psi_{\mathrm{GS}}(H)\rangle$ for $h_{z}/J=0.4523$
and $h_{x}/J=0.4045$. The alternating unitaries for conventional QAOA and PG-
QAOA are generated by $\mathcal{A}^{\mathrm{d}}=\\{H_{1},H_{2}\\}$; for CD-
QAOA and RL-QAOA, we extend this set using adiabatic gauge potential terms to
$\mathcal{A}^{\mathrm{d}}=\\{H_{1},H_{2};Y,X|Y,Y|Z\\}$. The system sizes are
$N\\!=\\!4,6,8$.

To evaluate the performance of the trained agent, we eliminate the uncertainty
associated with the probabilistic nature of the policy: we take the discrete
action which maximizes the categorical distribution $\pi^{\mathrm{d}}$, and
only keep the mean of the continuous distribution $\pi^{\mathrm{c}}$, setting
its width to zero. This defines a natural greedy policy to test the ability of
the RL agent.

We performed a number of numerical experiments to study the effect of the
noise on the performance of the four algorithms QAOA, PG-QAOA, CD-QAOA and RL-
QAOA, for the three different sources of uncertainty: classical and quantum
measurement noise, and gate rotation noise. We vary both the noise strength,
and we look at three different system sizes for two protocol durations each.
The results of these experiments can be summarized, as follows.

Figure 3 shows the best achievable energy at a protocol duration $JT=10$
against different noise types and system sizes: the top row shows data for
various measurement noise strengths, with the shaded area marking the special
case of quantum noise; the noise strength is measured in percentages of the
achievable ground state energy density: e.g., a noise strength of $\gamma=0.3$
corresponds to an average deviation from the actual energy of about $30\%$.
The bottom row displays results when varying the gate noise strength. Here,
the noise strength is defined as a percentage of the mean gate duration $T/q$.
The three columns correspond to system sizes $N=4$ (left), $N=6$ (middle) and
$N=8$ (right).

When $T<T_{\mathrm{QSL}}$ is chosen below the QSL, we find that QAOA and PG-
QAOA fail to reach the ground state in the time allotted, as a result of
having an overconstrained control space
$\mathcal{A}^{\mathrm{d}}=\\{H_{0},H_{1}\\}$. Nonetheless, the noise-robust
character of PG-QAOA becomes pronounced at increased values of the noise
strength. Since the initial quantum state is far away from the target ground
state, the best ratio $E/E_{\mathrm{GS}}$ found by QAOA can even be negative.
The $JT=10$ duration exhibits the advantage of using the generalized QAOA
ansatz brought in by CD-QAOA: suitably enlarging the discrete action space
$\mathcal{A}^{\mathrm{d}}=\\{H_{1},H_{2},Y,X|Y,Y|Z\\}$ unlocks paths in
Hilbert space which are inaccessible to QAOA. Hence, CD-QAOA and RL-QAOA find
the largest rewards in the noise-free case. A large noise strength reduces
visibly the ability of CD-QAOA to find the ground state, with the performance
being particularly bad for quantum measurement noise (Q). However, the hybrid
policy optimizer allows RL-QAOA to emerge as a noise-robust algorithm,
agnostic to the source of noise applied to the system.

##  6 Conclusion and Outlook

In summary, we presented RL-QAOA – a versatile and noise-robust quantum
control algorithm based on the QAOA variational ansatz. The algorithm inherits
valuable features from its ancestors: (i) the noise-robust property of PG-QAOA
allows us to find optimal durations probabilistically. (ii) the generalized
QAOA ansatz of CD-QAOA makes it possible to select the order in which a set of
unitaries appears in the control sequence. While we focused on physically-
motivated unitaries, we emphasize that the ansatz is completely general and
applicable to a large variety of unitaries/quantum gate sets useful for both
theoretical and experimental studies. We had to modify these “ancestors”
accordingly: in PG-QAOA we introduced a mechanism to fix the total protocol
duration, and introduced a stochastic policy based on the compactly supported
Beta function; in CD-QAOA we changed the low-level optimizer to gradient-free
Powell, as opposed to the gradient-based SLSQP which did not give a reasonable
performance in the presence of noise. RL-QAOA extends PG-QAOA and CD-QAOA with
both the use of a generalized autoregressive architecture which incorporates
the parameters of the continuous policy, and the derivation of an extension of
Proximal Policy Optimization applicable to hybrid continuous-discrete
policies.

We tested the performance of RL-QAOA using the unitary dynamics of a quantum
Ising chain subject to various sources of noise: classical and quantum
measurement noise as well as uncertainty leading to errors in the application
of quantum unitary gates. In particular, we demonstrated that RL-QAOA
successfully outperforms its ancestors in the highly-constrained non-adiabatic
regime, irrespective of the noise model selected. Thus, RL-QAOA is not only
noise-robust but also agnostic to the physical source of noise. This opens up
the exciting possibility of using machine learning to ‘learn’ the
particularities of noisy experimental environments, which often depend on the
chip architecture and can even change in the course of exploitation. However,
the presented results are obtained using numerical simulations based on
certain theoretical assumptions; it remains to test the performance of RL-QAOA
on realistic noisy intermediate-scale quantum computing devices.

The RL-QAOA is a versatile method that can be extended along several
directions. For instance, the current version of RL-QAOA defines a fixed
sequence/protocol length. However, the algorithm is versatile enough to
accommodate a variable length of the protocols after a slight modification. To
do so, one can simply add a “stop” action to the discrete action set
$\mathcal{A}^{d}$. If the agent happens to choose the stop action, then the
episode comes to an end immediately and we measure the energy of the evolved
quantum state.

There also exist a number of exciting alternatives for the policy network
architecture to explore. Although it has to incorporate temporal causality,
notice that the architecture is not limited to the autoregressive choice used
in this study; e.g., it can be generalized to a recurrent neural network
(RNN), a Long Short Term Memory (LSTM) network, or a transformer with the
attention mechanism Vaswani et al. (2017) and all its modern variants Kitaev
et al. (2020); Choromanski et al. (2020); Wang et al. (2020b); Tay et al.
(2020). In the present study, we chose the autoregressive network for its
sheer simplicity. Moreover, the continuous policy head can be generalized to
capture distributions with more than two modes using the normalizing flow
method, which would additionally boost the expressivity of the policy Tang and
Agrawal (2018).

We wish to thank Vitchyr Pong for valuable discussions. This work was
partially supported by the Department of Energy under Grant No. DE-
AC02-05CH11231 and No. DE-SC0017867 (L.L., J.Y.), and by the National Science
Foundation under the NSF QLCI program through grant number OMA-2016245 (L.L.).
M.B. was supported by the Bulgarian National Science Fund within National
Science Program VIHREN, contract number KP-06-DV-5, and the Marie Sklodowska-
Curie grant agreement No 890711.

We used Powell methods Powell (1964) implemented in SciPy Virtanen et al.
(2020) for the QAOA solver; we used W&B Biewald (2020) to organize and analyze
the experiments. The reinforcement learning networks are implemented in
NumPyHarris et al. (2020), and TensorFlow Abadi et al. (2015) and TensorFlow
Probability Dillon et al. (2017); the quantum systems are simulated in Quspin
Weinberg and Bukov (2017, 2019). We thank Berkeley Research Computing (BRC)
for providing the computational resources.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Albarrán-Arriagada et al. (2018) Francisco Albarrán-Arriagada, Juan C Retamal, Enrique Solano, and Lucas Lamata.  Measurement-based adaptation protocol with quantum reinforcement learning.  _Phys. Rev. A_ , 98:042315, Oct 2018.  10.1103/PhysRevA.98.042315.  URL https://link.aps.org/doi/10.1103/PhysRevA.98.042315.
  * Anand et al. (2020) Abhinav Anand, Matthias Degroote, and Alan Aspuru-Guzik.  Natural evolutionary strategies for variational quantum computation.  _arXiv preprint arXiv:2012.00101_ , 2020.
  * Arute et al. (2019) Frank Arute, Kunal Arya, Ryan Babbush, Dave Bacon, Joseph C Bardin, Rami Barends, Rupak Biswas, Sergio Boixo, Fernando GSL Brandao, David A Buell, et al.  Quantum supremacy using a programmable superconducting processor.  _Nature_ , 574(7779):505–510, 2019.  URL https://www.nature.com/articles/s41586-019-1666-5.
  * August and Hernández-Lobato (2018) Moritz August and José Miguel Hernández-Lobato.  Taking gradients through experiments: Lstms and memory proximal policy optimization for black-box quantum control.  In _International Conference on High Performance Computing_ , pages 591–613. Springer, 2018.  URL https://arxiv.org/abs/1802.04063.
  * Bellemare et al. (2020) Marc G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyu Wang.  Autonomous navigation of stratospheric balloons using reinforcement learning.  _Nature_ , 588(7836):77–82, December 2020.  10.1038/s41586-020-2939-8.  URL https://doi.org/10.1038/s41586-020-2939-8.
  * Bester et al. (2019) Craig J Bester, Steven D James, and George D Konidaris.  Multi-pass q-networks for deep reinforcement learning with parameterised action spaces.  _arXiv preprint arXiv:1905.04388_ , 2019.
  * Biewald (2020) Lukas Biewald.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Bohrdt et al. (2019) Annabelle Bohrdt, Christie S Chiu, Geoffrey Ji, Muqing Xu, Daniel Greif, Markus Greiner, Eugene Demler, Fabian Grusdt, and Michael Knap.  Classifying snapshots of the doped hubbard model with machine learning.  _Nature Physics_ , 15(9):921–924, 2019.
  * Bolens and Heyl (2020) Adrien Bolens and Markus Heyl.  Reinforcement learning for digital quantum simulation.  _arXiv preprint arXiv:2006.16269_ , 2020.  URL https://arxiv.org/abs/2006.16269.
  * Bravyi et al. (2020) Sergey Bravyi, Alexander Kliesch, Robert Koenig, and Eugene Tang.  Hybrid quantum-classical algorithms for approximate graph coloring.  _arXiv preprint arXiv:2011.13420_ , 2020.
  * Bukov (2018) Marin Bukov.  Reinforcement learning for autonomous preparation of floquet-engineered states: Inverting the quantum kapitza oscillator.  _Physical Review B_ , 98(22):224305, 2018.  10.1103/PhysRevB.98.224305.  URL https://link.aps.org/doi/10.1103/PhysRevB.98.224305.
  * Bukov et al. (2018) Marin Bukov, Alexandre GR Day, Dries Sels, Phillip Weinberg, Anatoli Polkovnikov, and Pankaj Mehta.  Reinforcement learning in different phases of quantum control.  _Physical Review X_ , 8(3):031086, 2018.  10.1103/PhysRevX.8.031086.  URL https://link.aps.org/doi/10.1103/PhysRevX.8.031086.
  * Caneva et al. (2011) Tommaso Caneva, Tommaso Calarco, and Simone Montangero.  Chopped random-basis quantum optimization.  _Phys. Rev. A_ , 84:022326, Aug 2011.  10.1103/PhysRevA.84.022326.  URL https://link.aps.org/doi/10.1103/PhysRevA.84.022326.
  * Carleo and Troyer (2017) Giuseppe Carleo and Matthias Troyer.  Solving the quantum many-body problem with artificial neural networks.  _Science_ , 355(6325):602–606, 2017.
  * Carleo et al. (2019) Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová.  Machine learning and the physical sciences.  _Rev. Mod. Phys._ , 91:045002, Dec 2019.  10.1103/RevModPhys.91.045002.  URL https://link.aps.org/doi/10.1103/RevModPhys.91.045002.
  * Carrasquilla (2020) Juan Carrasquilla.  Machine learning for quantum matter.  _Advances in Physics: X_ , 5(1):1797528, 2020\.  10.1080/23746149.2020.1797528.  URL https://doi.org/10.1080/23746149.2020.1797528.
  * Carrasquilla and Melko (2017) Juan Carrasquilla and Roger G Melko.  Machine learning phases of matter.  _Nature Physics_ , 13(5):431–434, 2017.
  * Chen et al. (2013) Chunlin Chen, Daoyi Dong, Han-Xiong Li, Jian Chu, and Tzyh-Jong Tarn.  Fidelity-based probabilistic q-learning for control of quantum systems.  _IEEE transactions on neural networks and learning systems_ , 25(5):920–933, 2013.  10.1109/TNNLS.2013.2283574.
  * Choromanski et al. (2020) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller.  Rethinking attention with performers, 2020.
  * Chou et al. (2017) Po-Wei Chou, Daniel Maturana, and Sebastian Scherer.  Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution.  In _International conference on machine learning_ , pages 834–843, 2017.
  * Colabrese et al. (2017) Simona Colabrese, Kristian Gustavsson, Antonio Celani, and Luca Biferale.  Flow navigation by smart microswimmers via reinforcement learning.  _Physical review letters_ , 118(15):158004, 2017\.
  * Dalgaard et al. (2020a) Mogens Dalgaard, Felix Motzoi, Jesper Hasseriis Mohr Jensen, and Jacob Sherson.  Hessian-based optimization of constrained quantum control.  _Physical Review A_ , 102(4):042612, 2020a.
  * Dalgaard et al. (2020b) Mogens Dalgaard, Felix Motzoi, Jens Jakob Sorensen, and Jacob Sherson.  Global optimization of quantum dynamics with alphazero deep exploration.  _npj Quantum Information_ , 6(1), 2020b.  10.1038/s41534-019-0241-0.
  * Delalleau et al. (2019) Olivier Delalleau, Maxim Peter, Eloi Alonso, and Adrien Logut.  Discrete and continuous action representation for practical rl in video games.  _arXiv preprint arXiv:1912.11077_ , 2019.
  * Demirplak and Rice (2005) Mustafa Demirplak and Stuart A Rice.  Assisted adiabatic passage revisited.  _The Journal of Physical Chemistry B_ , 109(14):6838–6844, 2005.  URL http://pubs.acs.org/doi/abs/10.1021/jp040647w.
  * Dillon et al. (2017) Joshua V Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A Saurous.  Tensorflow distributions.  _arXiv preprint arXiv:1711.10604_ , 2017.  URL https://arxiv.org/abs/1711.10604.
  * Ding et al. (2020) Yongcheng Ding, Yue Ban, José D Martín-Guerrero, Enrique Solano, Jorge Casanova, and Xi Chen.  Breaking adiabatic quantum control with deep learning.  _arXiv preprint arXiv:2009.04297_ , 2020.  URL https://arxiv.org/abs/2009.04297.
  * Dong et al. (2019) Yulong Dong, Xiang Meng, Lin Lin, Robert Kosut, and K Birgitta Whaley.  Robust control optimization for quantum approximate optimization algorithm.  _arXiv preprint arXiv:1911.00789_ , 2019.
  * Dunjko and Briegel (2018) Vedran Dunjko and Hans J Briegel.  Machine learning & artificial intelligence in the quantum domain: a review of recent progress.  _Reports on Progress in Physics_ , 81(7):074001, 2018.  URL https://iopscience.iop.org/article/10.1088/1361-6633/aab406.
  * Fan et al. (2019) Zhou Fan, Rui Su, Weinan Zhang, and Yong Yu.  Hybrid actor-critic reinforcement learning in parameterized action space.  _arXiv preprint arXiv:1903.01344_ , 2019.
  * Farhi et al. (2014) Edward Farhi, Jeffrey Goldstone, and Sam Gutmann.  A Quantum Approximate Optimization Algorithm.  _arXiv preprint arXiv:1411.4028_ , 2014.  URL https://arxiv.org/pdf/1812.01041.pdf.
  * Fösel et al. (2018) Thomas Fösel, Petru Tighineanu, Talitha Weiss, and Florian Marquardt.  Reinforcement learning with neural networks for quantum feedback.  _Phys. Rev. X_ , 8:031084, Sep 2018.  10.1103/PhysRevX.8.031084.  URL https://link.aps.org/doi/10.1103/PhysRevX.8.031084.
  * Fösel et al. (2020) Thomas Fösel, Stefan Krastanov, Florian Marquardt, and Liang Jiang.  Efficient cavity control with snap gates.  _arXiv preprint arXiv:2004.14256_ , 2020.  URL https://arxiv.org/abs/2004.14256.
  * Garcia-Saez and Riu (2019) Artur Garcia-Saez and Jordi Riu.  Quantum observables for continuous control of the quantum approximate optimization algorithm via reinforcement learning.  _arXiv preprint arXiv:1911.09682_ , 2019.  URL https://arxiv.org/abs/1911.09682.
  * Germain et al. (2015) Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle.  Made: Masked autoencoder for distribution estimation.  volume 37 of _Proceedings of Machine Learning Research_ , pages 881–889, Lille, France, 07–09 Jul 2015. PMLR.  URL http://proceedings.mlr.press/v37/germain15.html.
  * Guéry-Odelin et al. (2019) David Guéry-Odelin, Andreas Ruschhaupt, Anthony Kiely, Erik Torrontegui, Sofia Martínez-Garaot, and Juan Gonzalo Muga.  Shortcuts to adiabaticity: Concepts, methods, and applications.  _Reviews of Modern Physics_ , 91(4):045001, 2019\.  10.1103/RevModPhys.91.045001.  URL https://link.aps.org/doi/10.1103/RevModPhys.91.045001.
  * Haarnoja et al. (2017) Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  _Proceedings of the 34th International Conference on Machine Learning_ , 70:1352–1361, 2017.  10.5555/3305381.3305521.  URL https://dl.acm.org/doi/10.5555/3305381.3305521.
  * Haarnoja et al. (2018a) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ , 2018a.  URL https://arxiv.org/abs/1801.01290.
  * Haarnoja et al. (2018b) Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018b.  URL https://arxiv.org/abs/1812.05905.
  * Halverson et al. (2019) James Halverson, Brent Nelson, and Fabian Ruehle.  Branes with brains: exploring string vacua with deep reinforcement learning.  _Journal of High Energy Physics_ , 2019(6):3, 2019\.
  * Harris et al. (2020) Charles R. Harris, K. Jarrod Millman, St’efan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fern’andez del R’ıo, Mark Wiebe, Pearu Peterson, Pierre G’erard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.  Array programming with NumPy.  _Nature_ , 585(7825):357–362, September 2020\.  10.1038/s41586-020-2649-2.  URL https://doi.org/10.1038/s41586-020-2649-2.
  * Hausknecht and Stone (2015) Matthew Hausknecht and Peter Stone.  Deep reinforcement learning in parameterized action space.  _arXiv preprint arXiv:1511.04143_ , 2015.
  * Hegade et al. (2020) Narendra N Hegade, Koushik Paul, Yongcheng Ding, Mikel Sanz, F Albarrán-Arriagada, Enrique Solano, and Xi Chen.  Shortcuts to adiabaticity in digitized adiabatic quantum computing.  _arXiv preprint arXiv:2009.03539_ , 2020.  URL https://arxiv.org/abs/2009.03539.
  * Ho and Hsieh (2019) Wen Wei Ho and Timothy H Hsieh.  Efficient variational simulation of non-trivial quantum states.  _SciPost Phys_ , 6:29, 2019.
  * Iten et al. (2020) Raban Iten, Tony Metger, Henrik Wilming, Lídia Del Rio, and Renato Renner.  Discovering physical concepts with neural networks.  _Physical Review Letters_ , 124(1):010508, 2020\.
  * Jumper et al. (2020) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Kathryn Tunyasuvunakool, Olaf Ronneberger, Russ Bates, Augustin Žídek, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Anna Potapenko, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Martin Steinegger, Michalina Pacholska, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis.  High accuracy protein structure prediction using deep learning.  _In preparation_ , 2020.
  * Khairy et al. (2019) Sami Khairy, Ruslan Shaydulin, Lukasz Cincio, Yuri Alexeev, and Prasanna Balaprakash.  Reinforcement-learning-based variational quantum circuits optimization for combinatorial problems.  _arXiv preprint arXiv:1911.04574_ , 2019.  URL https://arxiv.org/abs/1911.04574.
  * Khairy et al. (2020) Sami Khairy, Ruslan Shaydulin, Lukasz Cincio, Yuri Alexeev, and Prasanna Balaprakash.  Learning to optimize variational quantum circuits to solve combinatorial problems.  In _AAAI_ , pages 2367–2375, 2020.  URL https://aaai.org/ojs/index.php/AAAI/article/view/5616/5472.
  * Khaneja et al. (2005) Navin Khaneja, Timo Reiss, Cindie Kehlet, Thomas Schulte-Herbrueggen, and Steffen J. Glaser.  Optimal control of coupled spin dynamics: design of nmr pulse sequences by gradient ascent algorithms.  172(2):296 – 305, 2005.  ISSN 1090-7807.  http://dx.doi.org/10.1016/j.jmr.2004.11.004.  URL http://www.sciencedirect.com/science/article/pii/S1090780704003696.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.  URL https://arxiv.org/abs/1412.6980.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=rkgNKkHtvB.
  * Kottmann et al. (2020) Korbinian Kottmann, Patrick Huembeli, Maciej Lewenstein, and Antonio Acin.  Unsupervised phase discovery with deep anomaly detection.  _arXiv preprint arXiv:2003.09905_ , 2020.
  * Kraft et al. (1988) Dieter Kraft et al.  A software package for sequential quadratic programming.  1988\.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Communications of the ACM_ , 60(6):84–90, 2017\.
  * Kulkarni et al. (2016) Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum.  Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.  In _Advances in neural information processing systems_ , pages 3675–3683, 2016.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Levine (2018) Sergey Levine.  Reinforcement learning and control as probabilistic inference: Tutorial and review.  _arXiv preprint arXiv:1805.00909_ , 2018.
  * Li et al. (2020) Li Li, Minjie Fan, Marc Coram, Patrick Riley, Stefan Leichenauer, et al.  Quantum optimization with a novel gibbs objective function and ansatz architecture search.  _Physical Review Research_ , 2(2):023074, 2020\.  10.1103/PhysRevResearch.2.023074.
  * Lu et al. (2020) Denghui Lu, Han Wang, Mohan Chen, Jiduan Liu, Lin Lin, Roberto Car, Weile Jia, Linfeng Zhang, et al.  86 pflops deep potential molecular dynamics simulation of 100 million atoms with ab initio accuracy.  _arXiv preprint arXiv:2004.11658_ , 2020.
  * Magann et al. (2020a) Alicia B Magann, Christian Arenz, Matthew D Grace, Tak-San Ho, Robert L Kosut, Jarrod R McClean, Herschel A Rabitz, and Mohan Sarovar.  From pulses to circuits and back again: A quantum optimal control perspective on variational quantum algorithms.  _arXiv preprint arXiv:2009.06702_ , 2020a.
  * Magann et al. (2020b) Alicia B Magann, Matthew D Grace, Herschel A Rabitz, and Mohan Sarovar.  Digital quantum simulation of molecular dynamics and control.  _arXiv preprint arXiv:2002.12497_ , 2020b.
  * Masson et al. (2015) Warwick Masson, Pravesh Ranchod, and George Konidaris.  Reinforcement learning with parameterized actions.  _arXiv preprint arXiv:1509.01644_ , 2015.
  * Masuda and Nakamura (2009) Shumpei Masuda and Katsuhiro Nakamura.  Fast-forward of adiabatic dynamics in quantum mechanics.  In _Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences_ , page rspa20090446. The Royal Society, 2009\.  URL http://rspa.royalsocietypublishing.org/content/466/2116/1135.
  * Matos et al. (2020) Gabriel Matos, Sonika Johri, and Zlatko Papić.  Quantifying the efficiency of state preparation via quantum variational eigensolvers.  _arXiv preprint arXiv:2007.14338_ , 2020.  URL https://arxiv.org/abs/2007.14338.
  * Mehta et al. (2019) Pankaj Mehta, Marin Bukov, Ching-Hao Wang, Alexandre GR Day, Clint Richardson, Charles K Fisher, and David J Schwab.  A high-bias, low-variance introduction to machine learning for physicists.  _Physics reports_ , 810:1–124, 2019.  URL https://www.sciencedirect.com/science/article/pii/S0370157319300766.
  * Melnikov et al. (2020) Alexey A Melnikov, Pavel Sekatski, and Nicolas Sangouard.  Setting up experimental bell test with reinforcement learning.  _arXiv preprint arXiv:2005.01697_ , 2020.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Nautrup et al. (2019) Hendrik Poulsen Nautrup, Nicolas Delfosse, Vedran Dunjko, Hans J Briegel, and Nicolai Friis.  Optimizing quantum error correction codes with reinforcement learning.  _Quantum_ , 3:215, 12 2019.  10.22331/q-2019-12-16-215.
  * Neunert et al. (2020) Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Jost Tobias Springenberg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller.  Continuous-discrete reinforcement learning for hybrid control in robotics.  _arXiv preprint arXiv:2001.00449_ , 2020.
  * Niu et al. (2019) Murphy Yuezhen Niu, Sergio Boixo, Vadim N Smelyanskiy, and Hartmut Neven.  Universal quantum control through deep reinforcement learning.  _npj Quantum Information_ , 5(1):1–8, 2019.  10.1038/s41534-019-0141-3.  URL https://doi.org/10.1038/s41534-019-0141-3.
  * Peruzzo et al. (2014) Alberto Peruzzo, Jarrod McClean, Peter Shadbolt, Man-Hong Yung, Xiao-Qi Zhou, Peter J Love, Alán Aspuru-Guzik, and Jeremy L O’brien.  A variational eigenvalue solver on a photonic quantum processor.  _Nature communications_ , 5:4213, 2014.  10.1038/ncomms5213.  URL https://doi.org/10.1038/ncomms5213.
  * Porotti et al. (2019) Riccardo Porotti, Dario Tamascelli, Marcello Restelli, and Enrico Prati.  Coherent transport of quantum states by deep reinforcement learning.  _Communications Physics_ , 2(1):1–9, 2019.  10.1038/s42005-019-0169-x.  URL https://doi.org/10.1038/s42005-019-0169-x.
  * Powell (1964) Michael JD Powell.  An efficient method for finding the minimum of a function of several variables without calculating derivatives.  _The computer journal_ , 7(2):155–162, 1964.
  * Rao et al. (2019) Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song.  Evaluating protein transfer learning with tape.  In _Advances in Neural Information Processing Systems_ , pages 9689–9701, 2019.
  * Reddy et al. (2016) Gautam Reddy, Antonio Celani, Terrence J Sejnowski, and Massimo Vergassola.  Learning to soar in turbulent environments.  _Proceedings of the National Academy of Sciences_ , 113(33):E4877–E4884, 2016.
  * Rem et al. (2019) Benno S Rem, Niklas Käming, Matthias Tarnowski, Luca Asteria, Nick Fläschner, Christoph Becker, Klaus Sengstock, and Christof Weitenberg.  Identifying quantum phase transitions using artificial neural networks on experimental data.  _Nature Physics_ , 15(9):917–920, 2019.
  * Sauvage and Mintert (2019) Frederic Sauvage and Florian Mintert.  Optimal quantum control with poor statistics.  _arXiv preprint arXiv:1909.01229_ , 2019.  URL https://arxiv.org/abs/1909.01229.
  * Schäfer et al. (2020) Frank Schäfer, Michal Kloc, Christoph Bruder, and Niels Lörch.  A differentiable programming method for quantum control.  _Machine Learning: Science and Technology_ , 1, 05 2020.  10.1088/2632-2153/ab9802.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In _International conference on machine learning_ , pages 1889–1897, 2015.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.  URL https://arxiv.org/abs/1707.06347.
  * Sels and Polkovnikov (2017) Dries Sels and Anatoli Polkovnikov.  Minimizing irreversible losses in quantum systems by local counterdiabatic driving.  _Proceedings of the National Academy of Sciences_ , 114(20):E3909–E3916, 2017.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Sim et al. (2020) Sukin Sim, Jonathan Romero, Jerome F Gonthier, and Alexander A Kunitsa.  Adaptive pruning-based optimization of parameterized quantum circuits.  _arXiv preprint arXiv:2010.00629_ , 2020.
  * Sørdal and Bergli (2019) Vegard B Sørdal and Joakim Bergli.  Deep reinforcement learning for robust quantum optimization.  _arXiv preprint arXiv:1904.04712_ , 2019.  URL https://arxiv.org/abs/1904.04712.
  * Sung (2020) Kevin Sung.  _Towards the First Practical Applications of Quantum Computers_.  PhD thesis, 2020.
  * Sung et al. (2020) Kevin J Sung, Jiahao Yao, Matthew P Harrigan, Nicholas C Rubin, Zhang Jiang, Lin Lin, Ryan Babbush, and Jarrod R McClean.  Using models to improve optimizers for variational quantum algorithms.  _Quantum Science and Technology_ , 5(4):044008, oct 2020.  10.1088/2058-9565/abb6d9.  URL https://doi.org/10.1088%2F2058-9565%2Fabb6d9.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tabi et al. (2020) Zsolt Tabi, Kareem H El-Safty, Zsófia Kallus, Péter Hága, Tamás Kozsik, Adam Glos, and Zoltán Zimborás.  Quantum optimization for the graph coloring problem with space-efficient embedding.  In _2020 IEEE International Conference on Quantum Computing and Engineering (QCE)_ , pages 56–62. IEEE, 2020.
  * Tang and Agrawal (2018) Yunhao Tang and Shipra Agrawal.  Boosting trust region policy optimization by normalizing flows policy.  _arXiv preprint arXiv:1809.10326_ , 2018.
  * Tay et al. (2020) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey.  _arXiv preprint arXiv:2009.06732_ , 2020.
  * Torlai et al. (2018) Giacomo Torlai, Guglielmo Mazzola, Juan Carrasquilla, Matthias Troyer, Roger Melko, and Giuseppe Carleo.  Neural-network quantum state tomography.  _Nature Physics_ , 14(5):447–450, 2018.
  * Valenti et al. (2019) Agnes Valenti, Evert van Nieuwenburg, Sebastian Huber, and Eliska Greplova.  Hamiltonian learning for quantum error correction.  _Physical Review Research_ , 1(3):033092, 2019\.
  * Van Nieuwenburg et al. (2017) Evert PL Van Nieuwenburg, Ye-Hua Liu, and Sebastian D Huber.  Learning phase transitions by confusion.  _Nature Physics_ , 13(5):435–439, 2017.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. 0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272, 2020.  https://doi.org/10.1038/s41592-019-0686-2.
  * Wang et al. (2020a) Guoming Wang, Dax Enshan Koh, Peter D Johnson, and Yudong Cao.  Bayesian inference with engineered likelihood functions for robust amplitude estimation.  _arXiv preprint arXiv:2006.09350_ , 2020a.
  * Wang (2016) Lei Wang.  Discovering phase transitions with unsupervised learning.  _Phys. Rev. B_ , 94:195105, Nov 2016.  10.1103/PhysRevB.94.195105.  URL https://link.aps.org/doi/10.1103/PhysRevB.94.195105.
  * Wang et al. (2020b) Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity, 2020b.
  * Wauters et al. (2020) Matteo M Wauters, Emanuele Panizon, Glen B Mbeng, and Giuseppe E Santoro.  Reinforcement learning assisted quantum optimization.  _Phys. Rev. Research_ , 2:033446, Sep 2020.  10.1103/PhysRevResearch.2.033446.  URL https://link.aps.org/doi/10.1103/PhysRevResearch.2.033446.
  * Wei et al. (2018) Ermo Wei, Drew Wicke, and Sean Luke.  Hierarchical approaches for reinforcement learning in parameterized action space.  _arXiv preprint arXiv:1810.09656_ , 2018.
  * Weinberg and Bukov (2017) Phillip Weinberg and Marin Bukov.  Quspin: a python package for dynamics and exact diagonalisation of quantum many body systems part i: spin chains.  _SciPost Phys_ , 2(1), 2017.
  * Weinberg and Bukov (2019) Phillip Weinberg and Marin Bukov.  Quspin: a python package for dynamics and exact diagonalisation of quantum many body systems. part ii: bosons, fermions and higher spins.  _SciPost Phys._ , 7(arXiv: 1804.06782):020, 2019\.
  * Wu et al. (2020a) Re-Bing Wu, Xi Cao, Pinchen Xie, and Yu-xi Liu.  End-to-end quantum machine learning with quantum control systems.  _arXiv preprint arXiv:2003.13658_ , 2020a.  URL https://arxiv.org/abs/2003.13658.
  * Wu et al. (2020b) Yadong Wu, Zengming Meng, Kai Wen, Chengdong Mi, Jing Zhang, and Hui Zhai.  Active learning approach to optimization of experimental control.  _arXiv preprint arXiv:2003.11804_ , 2020b.
  * Xiong et al. (2018) Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu.  Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space.  _arXiv preprint arXiv:1810.06394_ , 2018.
  * Yao et al. (2020a) Jiahao Yao, Marin Bukov, and Lin Lin.  Policy gradient based quantum approximate optimization algorithm.  107:605–634, 20–24 Jul 2020a.  URL http://proceedings.mlr.press/v107/yao20a.html.
  * Yao et al. (2020b) Jiahao Yao, Lin Lin, and Marin Bukov.  Reinforcement learning for many-body ground state preparation based on counter-diabatic driving.  _arXiv preprint arXiv:2010.03655_ , 2020b.  URL https://arxiv.org/abs/2010.03655.

##  Appendix A Pseudocode and Algorithm Hyperparameters

The pseudocode for RL-QAOA is outlined in Algorithm (1). The agent samples a
batch of actions from the autoregressive network. Then, the corresponding
expected energy density is computed using a classical simulator for the
quantum dynamics. Below, we focus on the noise-free case; dealing with noise
requires a trivial modification following Sec. 4.1. The baseline for the
reward is estimated through an exponential moving average. Finally, proximal
policy optimization is applied to update the agent’s policy.

Algorithm 1 Autoregressive network based reinforcement learning: RL-QAOA

0: batch size $M$, learning rate $\eta_{t}$, total number of iterations
$T_{\mathrm{iter}}$, exponential moving average coefficient $m$, entropy
coefficient $\beta^{-1}_{{S}}$, PPO gradient steps $K$.

1: Initialize the autoregressive network and initialize the moving average
$\hat{R}\\!=\\!0$.

2: for $t=1,..,T_{\mathrm{iter}}$ do

3: Autoregrssively sample batch $B$ hybrid actions of size $M$:
$a_{1}^{\\{k\\}},a_{2}^{\\{k\\}},\cdots,a_{q}^{\\{k\\}}\sim\pi_{\boldsymbol{\theta}}\left(a_{1},a_{2},\cdots,a_{q}\right),\
k=1,2,\cdots,M.$

4: Measure the observables and use the negative energy density as the return
and compute the moving average of the return
$R_{k}=-\mathcal{E}_{k}=-\frac{1}{N}\langle\psi_{i}|U^{\dagger}(\\{a_{j}^{\\{k\\}}\\}_{j=1}^{q})HU(\\{a_{j}^{\\{k\\}}\\}_{j=1}^{q})|\psi_{i}\rangle,\quad\hat{R}=m\cdot\hat{R}+(1-m)\cdot\frac{1}{M}\sum_{k=1}^{M}R_{k}.$

5: Compute the advantage estimates $A_{k}=R_{k}-\hat{R}$

6: for $k=1,..,K$ do

7: Evaluate the samples’ likelihood using the parameter from the last
iterations, i.e.
$\pi_{{\bm{\theta}}_{\mathrm{old}}}^{\mathrm{}}(a_{1}^{\\{k\\}},a_{2}^{\\{k\\}},\cdots,a_{q}^{\\{k\\}})$
and compute the importance weight $\rho_{k}^{\nu}$

8: Using the advantage estimation and importance weight to compute
$\mathcal{G}^{d}_{k},\mathcal{G}^{c}_{k},\mathcal{S}^{\mathrm{d}}_{k},\mathcal{S}^{\mathrm{c}}_{k}$.

9: Compute the RL-QAOA objective Eq (12) and backpropagate to get the
gradients.
$\nabla_{{\bm{\theta}}}\mathcal{J}({\bm{\theta}})=\frac{1}{M}\sum_{\\{a_{j}^{\\{k\\}}\\}_{j=1}^{q}\in
B}\nabla_{\bm{\theta}}\bigg{[}\mathcal{G}^{d}_{k}+\mathcal{G}^{c}_{k}+\beta^{-1}_{{S}}(\mathcal{S}^{\mathrm{d}}_{k}+\mathcal{S}^{\mathrm{c}}_{k})\bigg{]}.$

10: Update weights
${\bm{\theta}}\leftarrow{\bm{\theta}}+\eta_{t}\nabla_{{\bm{\theta}}}\mathcal{J}({\bm{\theta}})$.

11: end for

12: end for

We also conducted coarse hyperparameter sweeps to find the optimal values for
the hyperparameters of RL-QAOA, cf. Table 2. We use a batch size of $128$ to
train the policy. The policy network is optimized using Adam. The initial
learning rate is set to $5\text{\times}{10}^{-4}$, which is typical when
training autoregressive networks; we employ a learning rate decay schedule
which decreases by $98\%$ every $50$ iterations. The Autoregressive network is
implemented using uniform masks and dense layers Germain et al. (2015). The
base layer (see Fig. 2) consists of two hidden layers with 100 neurons each
and the heads contain $3|\mathcal{A}^{\mathrm{d}}|$ neurons in total.

The agent is trained via proximal policy gradient (PPO). We use four PPO
updates to the policy network parameters per iterations. The clipping
parameters are set as $\epsilon^{\mathrm{c}}=0.1$ for the continuous policy,
and $\epsilon^{\mathrm{d}}=10^{-3}$ for the discrete policy. We include
entropy bonus to increase exploration; the corresponding temperature schedule
$\beta^{-1}_{\mathcal{S}}$ starts at $1\text{\times}{10}^{-1}$, and drops by
$99\%$ every $50$ iterations.

Table 2: RL-QAOA Hyperparameters.
Hyperparameter | Value
---|---
Optimizer |  Adam Kingma and Ba (2014)
Learning rate | $5\text{\times}{10}^{-4}$
Likelihood ratio clip, $\epsilon^{\nu}$ |  0.1 ($\epsilon^{\mathrm{c}}$)
|  0.001 ($\epsilon^{\mathrm{d}}$)
PPO Epochs | 4
Hidden units (masked dense layer) | $[100,100]$
Activation function | ReLU
baseline exponential moving average ($m$) | 0.95
Learning rate annealing steps | 50
Learning rate annealing factor | 0.98
Learning rate annealing style | Staircase
Entropy bonus temperature ($\beta_{S,\\{0\\}}^{-1}$) | $1\times 10^{-1}$
Entropy bonus temperature decay steps | 50
Entropy bonus temperature decay factor | 0.99
Entropy bonus temperature decay style | Smooth
Minibatch size | 128

A typical learning curve in the noisy setting is shown in Fig. 4. Three
quantities are recorded to measure the performance of the agent. In the noise
setting, these quantities correspond to the ideal noise-free case. We use them
only for the purpose of evaluation; during the training, the RL agent only has
access to the noisy rewards. These quantities are shown in terms of the energy
ratio with respect to the target ground state so the possible maximum is upper
bounded by one; since the energy of a state can be either positive or
negative, while the GS has a negative value, negative ratios are possible.
Figure 4 shows that the agent starts to pick up the learning signal around two
thousand iterations. After that, it slightly modifies the policy in order to
achieve a higher reward. Here, the mean reward stands for the sample mean of
energy density at every iteration; the max reward is the maximum over the
sample; the history best is the best-encountered reward during the entire
training process.

Figure 4:  Spin-$1/2$ Ising model: training curves for RL-QAOA with energy
minimization as a cost function. The quantities in the main figure are
noiseless evaluation, while in the inset are noisy measurement. The noiseless
quantities are only for the evaluation’s purpose, and the agent can only
access the noisy quanties (in the inset). The mean reward (blue curve) is the
average energy ratio across the minibatch sampled from the autoregressive
policy; the max reward (orange curve) is taking the maximum across the
minibatch; the history best bookkeeps the best ever max reward during the
training. The total duration is $T\\!=\\!10$ and the number of spin-$1/2$
particles is $N\\!=\\!8$. The discrete RL-QAOA action space is
$\mathcal{A}^{\mathrm{d}}=\\{H_{1},H_{2};Y,X|Y,Y|Z\\}$, and we use
$q\\!=\\!8$. Here, the noise is classic gaussian noise, with the noise level
$\gamma\\!=\\!0.1$.

##  Appendix B A comparison of compactly supported distributions defining
continuous actions

###  B.1 Sigmoid Gaussian Distribution

In order to enforce the bounds for the duration output from the distribution,
we apply the sigmoid function. This kind of finite bound of the action does
help a lot in practice when we then normalize the actions to have the finite
sum; otherwise, we observe a large variance when we normalize the total
protocol duration to $T$ (see main text). To this end, we apply the sigmoid
function to the Gaussian distribution. In the following formula, we have
$x=f(y)$, where $f(y)=\frac{1}{1+e^{-y}}$ is the sigmoid. We denote the
original distribution as $\pi_{0}(y;\kappa,\xi)$ and the distribution after
the transformation, as $\pi(x;\kappa,\xi)$.

$\pi(x;\kappa,\xi)=\pi_{0}(y;\kappa,\xi)\left|\operatorname{det}\left(\frac{\mathrm{d}x}{\mathrm{d}y}\right)\right|^{-1}$

For example, if we choose $\pi_{0}$ to be Gaussian distribution according to
$\mathcal{N}(\kappa,\xi^{2})$, then

$\log\pi(x;\kappa,\xi)=-\log\xi-\frac{1}{2}\log(2\pi)-\frac{1}{2}\left(\frac{\operatorname{logit}(x)-\kappa}{\xi}\right)^{2}-\log(x\left(1-x\right))$
(14)

Here, the logit function, $\operatorname{logit}(x)\\!=\\!\log x-\log(1-x)$, is
the inverse of the sigmoid function $f(x)=1/(1+\exp(-x))$.

Thus, the derivative with respect to the parameters (i.e. $\kappa$ and $\xi$)
can be computed analytically, and reads

$\displaystyle\frac{\partial\log\pi(x;\kappa,\xi)}{\partial\kappa}$
$\displaystyle=\frac{\operatorname{logit}(x)-\kappa}{\xi^{2}},$ (15)
$\displaystyle\frac{\partial\log\pi(x;\kappa,\xi)}{\partial\xi}$
$\displaystyle=-\frac{1}{\xi}+\frac{1}{\xi}\left(\frac{\operatorname{logit}(x)-\kappa}{\xi}\right)^{2}$
(16)

We use this log probability in the policy gradient formula.

###  B.2 Beta Distribution

The Beta distribution’s probability density function is defined as:

$\pi(x;\kappa,\xi)=\frac{\Gamma(\kappa+\xi)}{\Gamma(\kappa)\Gamma(\xi)}x^{\kappa-1}(1-x)^{\xi-1},$

where the Gamma function is $\Gamma(z)=\int_{0}^{\infty}x^{z-1}e^{-t}\,dt$.
Here, the $\kappa$ and $\xi$ are the parameters of the Beta distribution,
which can be learned by the autoregressive policy network. The corresponding
log-probability reads as

$\log\pi(x;\kappa,\xi)=\log\Gamma(\kappa+\xi)-\log\Gamma(\kappa)-\log\Gamma(\xi)+(\kappa-1)\log(x)+(\xi-1)\log(1-x),$
(17)

Thus, the derivative with respect to the parameters (i.e. $\kappa$ and $\xi$)
reads

$\displaystyle\frac{\partial\log\pi(x;\kappa,\xi)}{\partial\kappa}$
$\displaystyle=\psi(\kappa+\xi)-\psi(\kappa)+\log(x),$ (18)
$\displaystyle\frac{\partial\log\pi(x;\kappa,\xi)}{\partial\xi}$
$\displaystyle=\psi(\kappa+\xi)-\psi(\xi)+\log(1\\!-\\!x),$ (19)

where the digamma function is defined as the logarithmic derivative of the
gamma function:

$\psi(x)=\frac{d}{dx}\ln\big(\Gamma(x)\big{missing})=\frac{\Gamma^{\prime}(x)}{\Gamma(x)}.$
(20)

Hence, the gradient can be used to compute the policy gradient using
analytical expressions.

##  Appendix C Choosing the protocol duration $T$

Finally, let us explain the choice of protocol durations $JT=10$ used in our
study. Figure 5 shows a scan of the best energy over the protocol duration $T$
in the noise-free case for $N=4$ qubits for three methods: QAOA, CD-QAOA and
adiabatic driving. For the adiabatic driving, we consider the driven
spin-$1/2$ Ising model:

$H(\lambda)\\!=\\!\lambda(t)H\\!+\\!(1-\lambda(t))\tilde{H},$ (21)

where $\lambda(t)\\!=\\!\sin^{2}\left(\frac{\pi t}{2T}\right)$ $t\in[0,T]$, is
an smooth protocol satisfying the boundary conditions: $\lambda(0)\\!=\\!0$,
$\lambda(T)\\!=\\!1$, $\dot{\lambda}(0)\\!=\\!0\\!=\\!\dot{\lambda}(T)$. The
initial state is the ground state at $t\\!=\\!0$, i.e.
$|\psi_{i}\rangle\\!=\\!|\\!\\!\uparrow\cdots\uparrow\rangle$, while the
target state is the ground state of the Ising model at $t=T$ for
$h_{z}/J=0.4523$ and $h_{x}/J=0.4045$. Here, $H$ is the target Hamiltonian
defined in Eq. (3), and $\tilde{H}=-\sum_{i=1}^{N}S^{z}_{i}$.

The value $JT=10$ is selected to achieve a compromise: on one hand, it is
large enough for CD-QAOA to reach close enough to the ground state; on the
other hand, it is small enough for a discrepancy between the performance of
CD-QAOA and QAOA to become clearly visible. Hence, $JT=10$ exemplifies nicely
the benefits of using the generalized QAOA ansatz, as compared to QAOA. Last,
we emphasize that both $JT=10$ are far away from the adiabatic regime, as
shown by the adiabatic curve.

CD-QAOA QAOA adiabatic

Figure 5:  Spin-$1/2$ model: energy minimization at different protocol
duration $T$ for three different methods in the nose-free setup: CD-QAOA (blue
line), QAOA (red line), adiabatic evolution (green line). The physics model
and the setting are the same as in Sec. 4. For the adiabatic driving
simulation, we used the protocol function
$\lambda(t)\\!=\\!\sin^{2}\left(\frac{\pi t}{2T}\right)$, $t\in[0,T]$. The
quantum dynamics was solved for numerically, using a step size of $\Delta
t\\!=\\!$1\text{\times}{10}^{-3}$$. The system size is $N\\!=\\!4$. The
vertical purple dashed line corresponds to $JT\\!=\\!10$.