##  1 Our DRL algorithm for fast quantum gate design

We now describe our DRL algorithm for design of fast quantum gates in detail.
Here, we outline our algorithm specifically for superconducting transmon
qubits, but it can easily be adapted to other hardware platforms such as
trapped ions [Bruzewicz2019], quantum dots [Vandersypen2019], or neutral atoms
[Henriet2020]. In this case, the quantum system consists of a transmon qubit
dispersively coupled to a superconducting resonator and controlled via a
capacitively coupled voltage drive line (see Appendix LABEL:sec:hardware for
details). The voltage drive envelope

$c(t)=\cases{c}^{x}(t)\cos(\omega_{d}t)+c^{y}(t)\sin(\omega_{d}t)&0<t<t_{g}\\\
0\text{otherwise}$ (1)

is composed of two independent quadrature controls $c^{x}(t)$ and $c^{y}(t)$
on a single drive frequency $\omega_{d}$ for the duration of the gate
operation $t_{g}$. We seek to design a piece-wise constant (PWC) control pulse
with up to $N_{\textup{seg}}$ segments of equal length
$\tau=t_{g}/N_{\textup{seg}}$ where $t_{g}$ is the maximum duration of the
gate.

Our DRL algorithm consists of two neural networks to decide sequentially on
the $x$\- and $y$-quadrature of the control pulse. The dual agent structure
enables us to mitigate leakage errors while simultaneously creating faster
gates. At each time step, our agents select the amplitudes of the next segment
based on real-time feedback about the state of the system. The agents receive
rewards throughout the construction of the pulse (rather than just at the end)
which opens the possibility to design faster gates. We eliminate model-bias by
training on quantum hardware so our DRL algorithm can account for realistic
noise in the qubits and for other errors such as over-rotation introduced by
the classical drive lines. We train directly on the observation signal
resulting from probing the qubit. The signal has two components which we
denote $I$ for “in-phase” and $Q$ for “quadrature”. Previous algorithms for
gate design have used the $\ket{0}$ and $\ket{1}$ populations imperfectly
estimated from the location of the signal in the $(I,Q)$-plane. Figure
LABEL:fig:stateData shows readout data taken on the IBM Lima quantum computer,
where there is overlap between the locations of the $\ket{0}$, $\ket{1}$, and
$\ket{2}$ measurements. Our algorithm reduces the impact of measurement errors
because we avoid classifying the signal into $\ket{0}$ or $\ket{1}$. It also
has a low measurement overhead since we do not perform full state tomography.

The network parameters $\theta^{x},\theta^{y}$ are updated periodically during
the training. Each training iteration for $i=0,\hdots,N_{\textup{iter}}-1$
consists of $N_{\textup{ep}}\leq N_{\textup{seg}}$ episodes counted using the
index $j$. The ratio $N_{\textup{seg}}/N_{\textup{ep}}$ is an integer that
sets how many times the neural network parameters are updated before a full
waveform is constructed. We use a second index $k=k(i,j)$ to track the
waveform segment. The $j$-th input to the first neural network is a state
$s_{j}^{x}=(\langle I_{j}\rangle,\langle Q_{j}\rangle,k)$ composed of the
average $(I,Q)$ signal over $N_{\text}{shot}$ measurements and the current
segment index $k$. The output of the agent is a probability distribution over
the action space $\bb{U}^{x}$ which we denote
$\pi_{\theta^{x}_{i}}(\cdot|s_{j}^{x})$ to indicate the dependence on the
neural network parameters $\theta_{i}^{x}$. The agent samples an action
$u_{j}^{x}\in\bb{U}^{x}$ according to $\pi_{\theta^{x}_{i}}(\cdot|s_{j}^{x})$.
The state for the second agent $s_{j}^{y}=(u_{j}^{x},\Ell_{j})$ is formed of
the amplitude $u_{j}^{x}$ on the first quadrature and the leakage population

$\Ell_{j}=\left|\bra{2}U_{k+1}\ket{0}\right|^{2}$ (2)

estimated from the $(I,Q)$-plane, where $U_{k+1}$ represents the operation of
evolving the qubit by the first $k+1$ segments of the waveform. The agent now
samples an action $u_{j}^{y}\in\bb{U}^{y}$ according to the output
$\pi_{\theta^{y}_{i}}(\cdot|s_{j}^{y})$ of the second network. The $k$-th
segment of the control pulse thus takes the form

$c(t)=u_{j}^{x}\cos(\omega_{d}t)+u_{j}^{y}\sin(\omega_{d}t)$ (3)

for $k\tau\leq t<(k+1)\tau$. Based on hardware parameters, we restrict the
$x$-quadrature amplitudes to the set
$\bb{U}^{x}=\\{0.00,0.01,\hdots,0.19,0.20\\}$ and the $y$-quadrature
amplitudes to $\bb{U}^{y}=\\{-0.10,-0.09,\hdots,0.09,0.10\\}$.

To train the agent, we require a reward function for each network. Let
$c_{T}=(\langle I_{T}\rangle,\langle Q_{T}\rangle)\pm\sigma_{T}$ be the
expected average measurement result after applying the target gate to the
ground state (calibrated experimentally). During training, we penalize the
distance of the observed measurement signal from $(\langle
I_{T}\rangle,\langle Q_{T}\rangle)$ which encapsulates both a failure to steer
the qubit into the desired state and leakage into higher energy levels. We use
the reward function

$\displaystyle r_{j}^{x}\left(s_{j+1}^{x},c_{T}\right)=$ (4)
$\displaystyle\min\left\\{1-\lambda k,\frac{\sigma_{T}}{\|(\langle
I_{j+1}\rangle,\langle Q_{j+1}\rangle)-(\langle I_{T}\rangle,\langle
Q_{T}\rangle)\|}-\lambda k\right\\}$

to train the first network, where the term $\lambda k$ penalizes the length of
the control pulse for some coefficient $\lambda\in\bb{R}$. For the second
network, we compensate low leakage populations with the reward

$r_{j}^{y}\left(s_{j+1}^{y}\right)=\max\left\\{0,\Ell_{\text}{max}-\Ell_{j+1}\right\\}$
(5)

where $\Ell_{\text}{max}$ sets a limit on the allowable leakage. A summary of
our DRL algorithm is shown in Algorithm 1.

DRL for fast quantum gate design [1] initial state $s_{0}^{x}$, leakage
$\Ell_{0}$, and parameters $\theta_{0}^{x},\theta_{0}^{y}$ Set $k=0$ each
iteration $i=0,...,N_{\textup{iter}}-1$ each episode
$j=0,...,N_{\textup{ep}}-1$ $k=N_{\textup{seg}}$ Re-initialize waveform $k=0$
Re-initialize state $s^{x}_{j}=s^{x}_{0}$ Re-initialize leakage
$\Ell_{j}=\Ell_{0}$ Select next action $u^{x}_{j}$ according to policy
$\pi_{\theta^{x}_{i}}(\cdot|s_{j}^{x})$ Set $s_{j}^{y}=(u_{j}^{x},\Ell_{j})$
Select next action $u^{y}_{j}$ according to policy
$\pi_{\theta^{y}_{i}}(\cdot|s_{j}^{y})$ Evolve qubit by first $k+1$ segments
of waveform Measure qubit $N_{\textup{shot}}$ times to get $(\langle
I_{j+1}\rangle,\langle Q_{j+1}\rangle)$ Update $k\to k+1$ Set
$s^{x}_{j+1}=(\langle I_{j+1}\rangle,\langle Q_{j+1}\rangle,k)$ Calculate
reward $r_{j}^{x}$ Estimate leakage population $\Ell_{j+1}$ Calculate reward
$r_{j}^{y}$ Reset qubit to $\ket{0}$ Send trajectory
$\left(s_{0}^{x},u_{0}^{x},r_{0}^{x},s_{1}^{x},u_{1}^{x},r_{1}^{x},\hdots,r_{N_{\textup{ep}}-1}^{x}\right)$
to first network Update neural network parameters
$\theta_{i}^{x}\to\theta_{i+1}^{x}$ Send trajectory
$\left(s_{0}^{y},u_{0}^{y},r_{0}^{y},s_{1}^{y},u_{1}^{y},r_{1}^{y},\hdots,r_{N_{\textup{ep}}-1}^{y}\right)$
to second network Update neural network parameters
$\theta_{i}^{y}\to\theta_{i+1}^{y}$

The initial state $s_{0}^{x}$ and leakage population $\Ell_{0}$ are estimated
by measuring the qubit before performing any gate operations. The initial
policies $\pi_{\theta_{0}}^{x},\pi_{\theta_{0}}^{y}$ are generated based on
the industry standard DRAG gate by pre-training the agent. The DRAG pulse is
Gaussian with a derivative component on the second quadrature. That is,
$c^{x}(t)=\Omega_{G}(t)$ and $c^{y}(t)=\gamma\Omega_{G}^{\prime}(t)$ for a
Gaussian envelope $\Omega_{G}$ and a coefficient $\gamma\in\bb{R}$
[Motzoi2009]. The DRAG pulse is designed to reduce leakage based on a simple
three-level model. In experiments, the amplitude of $\Omega_{G}$ and the
$\gamma$ factor are calibrated to combat time dependent changes in the noise.
Our initial policy captures information about the system dynamics using the
calibrated DRAG pulse; however, our algorithm should not be considered an
optimization of the DRAG pulse which has been proposed in other works
[Motzoi2009, Werninghaus2021]. In the next section, we describe experimental
results showing that our agent learns novel pulse shapes able to outperform
the DRAG gate in terms of fidelity and leakage rate. For more details on the
pre-training, see Appendix LABEL:sec:pretrain.