# Benchmarking logical three-qubit quantum Fourier transform encoded in the
Steane code on a trapped-ion quantum computer

Karl Mayer  Ciarán Ryan-Anderson  Natalie Brown  Elijah Durso-Sabina  Charles
H. Baldwin  David Hayes  Joan M. Dreiling  Cameron Foltz  John P. Gaebler
Thomas M. Gatterman  Justin A. Gerber  Kevin Gilmore  Dan Gresh  Nathan Hewitt
Chandler V. Horst  Jacob Johansen  Tanner Mengle  Michael Mills  Steven A.
Moses  Peter E. Siegfried  Brian Neyenhuis  Juan Pino  Russell Stutz
Quantinuum

###### Abstract

We implement logically encoded three-qubit circuits for the quantum Fourier
transform (QFT), using the [[7,1,3]] Steane code, and benchmark the circuits
on the Quantinuum H2-1 trapped-ion quantum computer. The circuits require
multiple logical two-qubit gates, which are implemented transversally, as well
as logical non-Clifford single-qubit rotations, which are performed by non-
fault-tolerant state preparation followed by a teleportation gadget. First, we
benchmark individual logical components using randomized benchmarking for the
logical two-qubit gate, and a Ramsey-type experiment for the logical $T$ gate.
We then implement the full QFT circuit, using two different methods for
performing a logical control-$T$, and benchmark the circuits by applying it to
each basis state in a set of bases that is sufficient to lower bound the
process fidelity. We compare the logical QFT benchmark results to predictions
based on the logical component benchmarks.

††preprint: APS/123-QED

##  I Introduction

It is generally believed that quantum error correction (QEC) will be required
for future large-scale quantum computation. This involves encoding logical
qubits into a higher dimensional space comprising a larger number of physical
qubits, applying gates at the logical level, and performing rounds of syndrome
extraction and error correction. Additionally, fault-tolerant circuit design
will be needed to ensure that errors do not spread uncontrollably during an
arbitrary computation. An important criterion for useful QEC is that for every
circuit in a family of interest, the error rate for an encoded version of that
circuit must be lower than that of an unencoded version run on the same
hardware [1]. Recent progress towards that goal, enabled by advances in
scalability and improved gate fidelity, has allowed for direct comparison
between small encoded circuits and unencoded circuits [2, 3, 4, 5, 6, 7, 8,
9].

In currently available devices, it is not a priori clear which codes and
protocols (initialization, gates, syndrome extraction, etc.) lead to the best
circuit performance, and in order to weigh the different options available we
must rely on quantum benchmarking tools. At the physical level, component
benchmarking has been used to characterize and quantify the performance of
individual parts of a quantum circuit, whereas system-level benchmarks assess
the overall ability to execute quantum circuits with high fidelity. Assessing
whether the error rates of individually benchmarked components account for the
error rate of the full circuit is also important so that one can be confident
in results when quantum processors perform novel calculations.

Here, we report on the experimental extension of these methods to the logical
level. We use the Quantinuum H-series quantum processors [10, 11] to
experimentally investigate both component-level (individual gates) and system-
level (complex circuits) benchmarks of the [[7,1,3]] Steane code [12]. At the
component-level, we benchmark the most error-prone logical gates, namely the
transversal two-qubit logical CNOT, and the non-Clifford logical $T$ gate. We
benchmark these components by two-qubit randomized benchmarking (RB) [13, 14],
and a protocol from Ref. [15] for estimating the fidelity of a logical $T$
performed by a teleportation gadget. At the system-level, we perform an
encoded three-qubit quantum Fourier transform (QFT), chosen because of its
importance as a subroutine and because it requires all the logical primitives
for universal quantum computation. With 7 physical qubits per logical qubit
block, plus another 7 for a logical ancilla qubit needed to prepare non-
Clifford magic states, the three-qubit QFT is the largest that fits within the
current 32-qubit H2-1 architecture [11]. All of these experiments were
designed using a software framework developed in-house that we call Simple
Logical Representation (SLR) [16]. We stress that we do not employ fault-
tolerant non-Clifford gates or rounds of QEC, though it is worth noting that
real-time QEC is done based on destructive measurements used in the gate
teleportation gadgets. In this work, the focus is on benchmarking the logical
encoding and assessing whether component-level errors account for the full
system performance at the logical level.

We measure logical two-qubit gate average fidelities 0.9991(2) on H1-1 and
0.9980(8) on H2-1, which is at or near the break-even level compared to
physical two-qubit gates. For the logical $T$ gate on H2-1, we obtain a much
lower average fidelity of 0.990(1). We implement the logical QFT in two
different ways, based on two methods for encoding a logical control-$T$. For
both methods, we apply the QFT to all basis states in both the computational
basis, and a rotated Fourier basis, from which a lower bound on the logical
process fidelity can be computed. For the higher performing of these two
methods, we obtain average output state fidelities of 0.78(1) and 0.66(1) in
these two bases. When allowing post-selection on syndrome information obtained
from the teleportation gadgets, the average output state fidelities increase
to 0.89(1) and 0.77(2), respectively. These numbers are lower than what we
would obtain from an unencoded QFT circuit, with the largest error stemming
from the $T$ gates. We conclude that either higher distance codes or fully
fault-tolerant circuit construction (or both) would be required for the
logical circuit to outperform a physical circuit on our present hardware.
Finally, we model the logical QFT circuits using the logical component-level
benchmarking results. We find that the logical gate errors explain part but
not all of the circuit error, and closing this gap will be crucial in
developing large-scale devices capable of running algorithms at the logical
level.

The outline of the paper is as follows. Sec. II provides background
information on the [[7,1,3]] code and details the logical primitives used in
this work. We assume familiarity with the stabilizer formalism of quantum
error correction [17]. In Sec. III we describe the experimental logical
component benchmarking of the transversal two-qubit logical CNOT gate, and the
non-Clifford logical $T$ gate. Section IV contains the details of the logical
QFT implementations, the procedure for obtaining process fidelity lower
bounds, and the experimental results. We conclude in Sec. V with a discussion
and outlook.

##  II Quantum error correction primitives

###  II.1 The 7-Qubit Steane Code

In this work we use the [[7,1,3]] color code, also known as the Steane code
[12]. This code uses 7 physical qubits to encode one logical qubit to distance
3, meaning that it is capable of detecting and correcting all single-qubit
errors. The Steane code is an example of a color code, which is a class of
topological codes defined by a 3-coloring of a planar graph, where each vertex
of the graph corresponds to a qubit, and there is an $X$-type and $Z$-type
stabilizer for each plaquette [18, 19]. In our convention a generating set of
stabilizers and logical operators $\overline{X}$ and $\overline{Z}$ are given
by

$\displaystyle s_{1}$ $\displaystyle=XXXXIII,$ $\displaystyle s_{2}$
$\displaystyle=IXXIXXI,$ $\displaystyle s_{3}$ $\displaystyle=IIXXIXX,$
$\displaystyle s_{4}$ $\displaystyle=ZZZZIII,$ $\displaystyle s_{5}$
$\displaystyle=IZZIZZI,$ (1) $\displaystyle s_{6}$ $\displaystyle=IIZZIZZ,$
$\displaystyle\overline{X}$ $\displaystyle=IIIIXXX,$
$\displaystyle\overline{Z}$ $\displaystyle=IIIIZZZ.$

Here, we use an overhead bar when needed to distinguish a logical operator
from a physical operator. The color code is a strong CSS code, meaning that
the $X$-type and $Z$-type stabilizers have identical support [20].
Consequentially, all logical Clifford operations are transversal. To see this,
consider the action of a transversal $\overline{H}$ and $\overline{S}$ on
stabilizers and logical operators. Applying $\overline{H}=H^{\otimes 7}$ to
the stabilizers listed in Eq. (II.1), we see that $\overline{H}$ preserves the
stabilizer group, and is therefore a logical operator. For example, $s_{1}$
transforms into $s_{4}$ and vice versa. Furthermore, when we conjugate
$\overline{X}$ in Eq. (II.1) by $\overline{H}$, we get $\overline{Z}$, and
vice versa, so the logical Pauli operators also transform appropriately. The
transversal $\overline{S}$ is implemented by applying $S^{\dagger}$ to all
physical qubits. Finally, a logical two-qubit Clifford, such as
$\overline{\mathrm{CNOT}}$, is given by a physical CNOT between each pair of
corresponding qubits between two code blocks. Since $H$, $S$, and CNOT
generate the Clifford group, it follows that all logical Clifford operators
are transversal. For more details see [2].

Throughout this paper we use the term “fault-tolerance,” which carries various
meanings, depending on the context. Here, we describe circuit design
principles as being fault-tolerant if they control the spread of “faults” in a
QEC circuit. Precisely, we call a sub-circuit fault-tolerant if no error on a
single qubit in a code block can propagate to an error that is uncorrectable
by a round of ideal syndrome extraction and correction [21]. An example of an
uncorrectable error is one that is equivalent to a non-trivial logical
operator, that is, an element of $\mathcal{N}\setminus\mathcal{S}$, where
$\mathcal{S}$ is the stabilizer group generated by Eq. (II.1), and
$\mathcal{N}=\\{g:gsg^{{\dagger}}\in\mathcal{S}\ \forall s\in\mathcal{S}\\}$
is the normalizer of $\mathcal{S}$. Such errors are undetectable and hence
uncorrectable. The transversality of logical Clifford operations in the color
code guarantees their fault-tolerance.

Fault-tolerant logical qubit initialization in the Steane code is performed by
an encoding circuit shown in Fig. 2, which uses a logical measurement scheme
to achieve fault-tolerance [22]. The encoding circuit itself is not fault-
tolerant, since the occurrence of a single-qubit error in certain locations
produces a two-qubit error. To ensure fault-tolerance, the circuit uses an
ancilla physical qubit to make a $\overline{Z}$ measurement that detects any
two-qubit error that might occur. A measurement result of ‘0’ indicates
successful preparation of the logical state $\ket{\overline{0}}$, otherwise
the circuit is repeated. In our experiments, we set a limit of three
repetitions of this repeat-until-success protocol and post-select to discard
shots that never succeeded. The discard rate from this initialization is
small; at most a few percent of the shots in our benchmarking experiments are
discarded. The choice of the limit three is motivated from our earlier work
[2].

Logical qubit measurement in a Pauli basis is performed by destructively
measuring each physical qubit in that basis. From this measurement, one
obtains the parities of both the logical operator, and the corresponding
stabilizer generators, by calculating the parity of the qubits that support
each operator. The stabilizer information then gives a syndrome which can be
decoded, and a final correction can be applied to the raw logical outcome.

The Eastin-Knill theorem shows that a universal transversal logical gate set
does not exist [23]. To complete a universal logical gate set, we perform the
gate $\overline{T}$ by preparing the logical state
$\ket{\overline{T}}=\frac{1}{\sqrt{2}}\big{(}\ket{\overline{0}}+e^{i\pi/4}\ket{\overline{1}}\big{)}$
and applying either of the teleportation gadgets shown in Fig. 1. More
generally, we can apply a logical $Z$-rotation given by
$P(\theta)=\mathrm{diag}(1,e^{i\theta})$, by preparing
$\ket{\overline{\theta}}=\frac{1}{\sqrt{2}}\big{(}\ket{\overline{0}}+e^{i\theta}\ket{\overline{1}}\big{)}$
non-fault-tolerantly, followed by a teleporation gadget. The circuit for
preparing $\ket{\overline{\theta}}$ non-fault-tolerantly is shown in Fig. 3.
The two gadgets in Fig. 1 are logically equivalent and differ only in whether
the initial logical data or ancilla qubit block ends up as the output data
qubit block. We benchmark both gadgets as described in the next section.

The teleportation gadgets use a logical measurement and gate conditioned on
the logical outcome. In our implementation, we use the syndrome information
obtained from the measurement to obtain a decoded logical outcome in real-
time, and condition the gate on the decoded logical outcome. It is also
possible to discard shots in which the syndromes from this measurement
indicate an error, essentially using the code for quantum error detection
(QED). However, this post-selection is not scalable as the measurement is made
after the logical ancilla qubit has been entangled with the logical data
qubit, and so a detection event requires discarding the entire computation.
The shot overhead will therefore scale exponentially in the logical $T$ count.
Since the logical QFT circuits in this work are small enough, we compare the
results with and without post-selection on the teleportation gadget
measurements.

The $\ket{\overline{T}}$ state preparation circuit can be made fault-tolerant
by using an ancilla qubit to measure the projector
$\ket{\overline{T}}\bra{\overline{T}}$ with an additional flag qubit to detect
errors in the measurement that spread to uncorrectable errors, followed by a
round of QEC [24, 25]. This scheme can be applied in a repeat-until-success
(RUS) protocol, combined with post-selection to discard shots in which a limit
on the number of unsuccessful attempts was reached. In this work, however, we
only implement the non-fault-tolerant version of $\overline{T}$ on quantum
hardware. This is because simulations (detailed in Appendix B) using the
device emulator suggest that a partially-fault-tolerant $\overline{T}$ (that
includes the flagged measurement but not the QEC round) would have a lower
fidelity than the non-fault-tolerant $\overline{T}$, due to a combination of
physical two-qubit gate error and memory error incurred during the flag
measurement. We note that a time-efficient protocol could use a logical
$T$-state factory where many attempts to prepare $\ket{\overline{T}}$ are done
in parallel. However, this would require more qubit resources than are
currently available in the H-series computers. We leave an experimental
verification of the simulations and a detailed study of full fault-tolerance
to future work.

(a)  (b)

Figure 1: Two versions of a teleportation gadget for performing a logical
phase gate, which we refer to as (a) method one and (b) method two. The
circuits use an ancilla prepared in
$\ket{\theta}=\frac{1}{\sqrt{2}}\big{(}\ket{0}+e^{i\theta}\ket{1}\big{)}$, and
output the state $P(\theta)\ket{\psi}$, where
$P(\theta)=\mathrm{diag}(1,e^{i\theta})$. Taking $\theta=\pi/4$ yields the $T$
gate.  Figure 2: Fault-tolerant repeat-until-success $\ket{\overline{0}}$
initialization circuit. If the ancilla qubit measurement result is ‘0’, then
the initialization succeeds. Otherwise, the circuit is repeated, up to a limit
of three repetitions. Figure 3: Non-fault-tolerant logical encoding circuit
for the Steane code. The final qubit in the quantum register is prepared in
$\ket{\psi}$ and the resulting logical state is $\ket{\overline{\psi}}$.

###  II.2 Simple Logical Representation

To ease the construction of logically encoded circuits, we developed a
software framework that we call Simple Logical Representation (SLR) [16]. In
SLR, a logical circuit representation can be built in a Pythonic environment,
similar to other circuit programming frameworks such as pytket [26] or Qiskit
[27]. An SLR object draws from a custom in-house library of QEC primitives to
output a program built from OpenQasm 2.0 [28] and the Quantinuum OpenQasm 2.0
extensions [2], building off of prior experimental work that demonstrated
real-time quantum error correction [2] and a logically encoded transversal
CNOT gate [3].

As an example of SLR code, a $\overline{T}$ gate as implemented by the
teleportation gadget of Fig. 1a is appended to an SLR program as follows:

⬇

program.extend(

InitTPlusNonFT(a),

CX(q,a),

MeasDecode(a,c,cLogRaw,cLog,cSyn),

If(cLog==1).Then(S(q)),

)

Here ‘q’ and ‘a’ are quantum registers for the logical data and ancilla
qubits, respectively. The object InitTPlusNonFT is used to initialize the
ancilla in the state $\ket{\overline{T}}$ via the non-fault-tolerant encoding
circuit in Fig. 3. A classical register ‘c’ is used for the measurement of the
qubits in register ‘a’, and ‘cLog’ is a single classical bit used to record
the decoded value of $\overline{Z}$ from the raw logical output ‘cLogRaw’ and
the syndromes ‘cSyn’. In this example, the objects InitTPlusNonFT, CX, and
MeasDecode are all imported from the Steane code QEC library, whereas If and
Then are SLR objects that can be used with any QEC code. A more complete SLR
program is given in Appendix C. From the above example the benefits of SLR are
clear: one can program circuits at the logical level once the various logical
primitives have been defined. An SLR program has a .qasm() method that returns
a string of OpenQasm 2.0 code, which can be submitted to the Quantinuum
H-series hardware via a submission api.

##  III Logical component benchmarking

In this section, we describe the protocols for benchmarking logical gates, and
present the results from the Quantinuum H-series hardware. The two protocols
covered in this section are logical two-qubit randomized benchmarking (RB),
and logical $T$ gate benchmarking. We focus on these two logical operations
since they are composed of physical two-qubit gates, and therefore are a
leading source of circuit error. We run the experiments on both the devices
H1-1 and H2-1. The results of our logical benchmarking experiments are shown
in Table. 1. The figure of merit we use for logical gate benchmarking is the
average fidelity, defined as

$F_{avg}(\mathcal{E},U)=\int
d\psi\bra{\psi}U^{{\dagger}}\mathcal{E}\big{(}\ket{\psi}\bra{\psi}\big{)}U\ket{\psi},$
(2)

where $U$ is a target unitary, $\mathcal{E}$ is the actual noisy quantum
process that is implemented, and the integral is with respect to the measure
on pure states induced by the Haar unitary measure [29]. In the case where $U$
is a logical operator encoded in a QEC code, the integral is taken over the
code space, which is a subspace of the physical qubit Hilbert space.

Experiment | Machine | Average Fidelity
---|---|---
Two-qubit RB | H1-1 | 0.9991(2)
Two-qubit RB | H2-1 | 0.9980(8)
$T$ method one | H2-1 | 0.981(1)
$T$ method two | H2-1 | 0.990(1)
$T$ method one (P.S.) | H2-1 | 0.985(3)
$T$ method two (P.S.) | H2-1 | 0.995(1)
Table 1: Logical component benchmarking results. For two-qubit RB, the
fidelity listed in the table is the estimated average fidelity of the
transversal logical CNOT. For the logical $T$ experiments, methods one and two
refer to the two methods for performing a teleportation gadget shown in Fig.
1, and P.S. refers to post-selecting the results on the syndromes obtained
from the logical measurement in the teleportation gadgets.

###  III.1 Randomized Benchmarking of Logical CNOT

We first benchmark the logical transversal CNOT gate, via two-qubit randomized
benchmarking (RB) [14]. Logical RB was previously considered in Ref. [30]. We
generate the full two-qubit Clifford group $\mathcal{C}_{2}$ using a
generating set of single-qubit Paulis, $H$, $S$ and $S^{{\dagger}}$, as well
as CNOT. The group $\mathcal{C}_{2}$ has 11,520 elements. The RB circuits are
constructed by applying a sequence $(C_{i})_{i=1}^{L}$ of Cliffords, each
drawn uniformly at random, followed by $C_{inv}=C_{1}^{-1}\dots C_{L}^{-1}$,
which is a Clifford group element equal to the inverse of the sequence. The
circuits use a total of 16 physical qubits: 7 for each logical qubit block and
one ancilla for each logical qubit for fault-tolerant initialization.

We use sequence lengths $L\in\\{2,6,10,14\\}$ and for each sequence length, we
generate 10 random circuits. The probability of measuring the expected
survival state as a function of sequence length is best fit to the model

$p(L)=af^{L}+1/4,$ (3)

and the average fidelity of the logical error channel per Clifford is then
computed as

$F_{avg}=1-3(1-f)/4.$ (4)

We divide the infidelity by the average number of CNOT gates per Clifford
group element in our construction to arrive at an estimate of the average
fidelity per logical CNOT. The decay plots are shown in Fig. 4, along with a
comparison to the device emulators. We obtain average fidelities per logical
CNOT of 0.9991(2) and 0.9980(8) for H1-1 and H2-1, respectively. For
comparison, the physical native two-qubit gate has been benchmarked at an
average fidelity of 0.99912(3) on H1-1 and 0.99817(5) on H2-1, when averaged
over the gate zones [31, 11]. This native gate is given by
$U_{2Q}=e^{-iZZ\pi/4}$ and differs from CNOT only by single-qubit unitaries,
whose error is on the order of $\sim 10^{-5}$ and negligible compared to the
two-qubit gate error. The logical CNOT fidelities are therefore competitive
with the physical native two-qubit gates. The higher logical fidelity on H1-1
compared to H2-1 can be attributed to higher fidelity two-qubit gates, as well
as a larger memory error per depth-one circuit time on H2-1 due to the larger
number of qubits in the QCCD architecture. Here, memory error refers qubit
decoherence during the time required to transport ions [11].

Figure 4: Logical two-qubit RB decay curves for the devices H1-1 and H2-1. Ten
random circuits per sequence length were chosen, and all circuits were run
with 100 shots and submitted in a random order. The curves are fit to Eq. (3),
and the estimated average fidelities per logical CNOT are listed in Table 1.

###  III.2 Benchmarking of Logical T

We use the protocol in Ref. [15] to estimate the error rate of the logical $T$
gate. By inserting a twirl over the set $\\{I,SX\\}$ after the injected
$\ket{T}$ state in the circuit diagram of Fig. 1, it was shown that the noise
process on the logical $T$ operator is given by a dephasing channel

$\tilde{\mathcal{T}}=(1-\epsilon)\mathcal{T}+\epsilon\mathcal{Z}\circ\mathcal{T},$
(5)

where $\mathcal{T}$ and $\tilde{\mathcal{T}}$ are the ideal and noisy
channels, and $\epsilon$ is the state infidelity of the injected state
$\ket{T}$.

The parameter $\epsilon$ can be measured by preparing $\ket{+}$, applying the
noisy $T$ gate $L$ times with $L$ a multiple of 4, and measuring in the
logical $X$ basis. The probability of obtaining the expected outcome is then
fit to the model

$p(L)=\frac{1}{2}+\frac{1}{2}(1-2\epsilon)^{L}.$ (6)

The motivation in Ref. [15] for estimating $\epsilon$ was that one can then
apply error mitigation via probabilistic error cancellation [32] on the
logical $T$ gates. We leave a demonstration of this technique to future work.

We run the logical $T$ gate benchmarking experiment on the device H2-1 using
both versions of the teleportation gadget shown in Fig. 1, which we call
methods one and two. The circuits for the experiment use 15 physical qubits: 7
for each logical qubit block in Fig. 1, plus one ancilla for fault-tolerant
initialization of the data logical qubit. The decay curves from the experiment
are shown in Fig. 5. We select $L\in\\{4,8,12,16\\}$ and sample 10 random
circuits per sequence length and run each circuit for 100 shots. After
$\epsilon$ is determined from Eq. (6), the average fidelity is computed as

$F_{avg}=1-2\epsilon/3.$ (7)

We obtain estimated average fidelities per logical $T$ gate of 0.981(1) and
0.990(1) for methods one and two, respectively. We conjecture that method two
performs better because the logical state is swapped between the data qubit
block and a fresh ancilla qubit block with each application of the gate. This
may lead to less accumulation of memory error on the logical state. This
conjecture is supported by simulations from the device emulator, which
predicts a lower average fidelity for method one. The emulator, in addition to
modeling gate error, contains detailed information about the ion transport
schedule and models memory error incurred during transport.

If we post-select the data based on the syndrome information obtained from the
logical measurement in the teleportation-gadget, we obtain higher average
fidelities: $0.985(3)$ and $0.995(1)$, respectively. At the longest sequence
length of $L=16$, an average of 47% of shots were retained over the two
experiments. However, as noted in Sec. II, this form of post-selection is not
scalable, but is still useful for near-term logical circuit experiments with a
relatively small non-Clifford gate count.

Figure 5:  Logical $T$ gate benchmarking decay curves for the device H2-1 and
the two methods for the teleportation gadget shown in Fig. 1. Solid lines are
fit to the raw data and dashed lines to the data including post-selection on
the syndromes obtained from the logical measurement in the teleportation
gadgets (labeled P.S.). Ten random circuits per sequence length were chosen
and all circuits were run with 100 shots and submitted in a random order. The
curves are fit to Eq. (6), and the estimated average fidelities per logical
$T$ are listed in Table 1.

##  IV Logical QFT benchmarking

In this section, we describe our implementation of the three-qubit QFT
circuit, and present the results of our benchmark on the H2-1 quantum
computer. An overview of the H-series hardware is given in Appendix A. The
three-qubit QFT circuit is shown in Fig. 6. Note that we do not include the
final swap of the qubits, since this can be achieved by a relabeling of the
qubits in software. The circuit contains two control-$S$ gates, and a
control-$T$ gate. We implement the control-$S$ gates using a standard
decomposition into two CNOT gates and three $T$ or $T^{{\dagger}}$ gates [33],
which is shown in Fig. 7. We implement the control-$T$ in two different ways.
The first method makes use of a device that we call a recursive-teleportation
gadget, and the second uses a logical ancilla with a logical mid-circuit
measurement and conditional $CZ$ gate. In the following subsections we first
describe these two implementations, and then the protocol that we use to
benchmark the QFT circuit before presenting the experimental results. The
total physical resources (qubits and two-qubit gates) for the two methods are
listed in Table 2.

$CT$ Method | Log. $\ket{\theta}$ | Log. TQ | Phys. TQ | Phys. Qubits
---|---|---|---|---
Recursive | 9-12 | 15-18 | 237-291 | 28
Ancilla | 11 | 13-14 | 256-263 | 28
Table 2: Logical and physical resources required for logical three-qubit QFT.
Here, log. $\ket{\theta}$ indicates the number of logical non-Clifford state
injections, and TQ stands for two-qubit gates. The ranges of values indicate
the minimum and maximum possible number of gates due to the conditional gates.
Figure 6:  Three-qubit QFT circuit, excluding a final qubit reordering.
Figure 7:  Control-phase gate decomposition. Here $P(\frac{\pi}{2})=S$, and
$P(\frac{\pi}{4})=T$.

###  IV.1 Control-T by Recursive-Teleportation Gadget

As shown in Fig. 7, a control-$S$ gate can be decomposed into the Clifford+$T$
gate set using a standard decomposition [33]. The control-$T$ gate is
trickier. Using the same decomposition, the control-$T$ requires three
$P(\frac{\pi}{8})$ gates, where $P(\theta)=\mathrm{diag}(1,e^{i\theta})$.
These are non-Clifford gates that perform a $Z$-rotation by half as large an
angle as the $T$ gate. While the $P(\frac{\pi}{8})$ gate can be approximately
decomposed into the Clifford+$T$ gate set using synthesis algorithms [34],
such decompositions are quite costly in the number of $T$ gates required. Our
method for performing a logical $P(\frac{\pi}{8})$ gate in the color code is
to apply the same encoding circuit for the logical $\ket{T}$ state, except
with last physical qubit in Fig. 3 prepared in the state
$\ket{\pi/8}=P(\frac{\pi}{8})\ket{+}$. We note that this preparation is non-
fault-tolerant, and unlike the logical $\ket{T}$ state preparation, which can
be made fault-tolerant by measuring the projector onto the logical state with
a flag qubit [24], we do not know how to make the $\ket{\pi/8}$ preparation
fault-tolerant. However, we note there are recent proposals for performing
fault-tolerant logical $Z$-rotations by post-processing of results to
interpolate to a desired rotation angle [35], or using quantum error detection
along with post-selection for arbitrary-angle state injection [36].

After preparing the state $\ket{\pi/8}$, a teleportation gadget as in Fig. 1
is applied to convert the state into a $P(\frac{\pi}{8})$ gate. The
teleportation gadget in this case has a conditional $T$ gate, which itself
must be performed by a teleportation gadget, as shown in Fig. 8. This is why
we call the technique the recursive-teleportation method. This method can be
scaled to higher levels of recursion, where in general $P(\pi/2^{k})$ is
performed by preparing a $\ket{\pi/2^{k}}$ state and using a teleportation
gadget with a conditional $P(\pi/2^{k-1})$ gate. In this way a general
$n$-qubit QFT circuit can be encoded logically if one is willing to sacrifice
fault-tolerance.

Figure 8:  Recursive-teleportation gadget for performing a $P(\pi/8)$ gate.
The logical $\ket{\pi/8}$ state is prepared non-fault-tolerantly by the
encoding circuit in Fig. 3. After the first measurement, a teleportation
gadget implementing the $T$ gate is performed conditioned on the classical
outcome. Classical control wires are shown for clarity.

###  IV.2 Ancilla-Assisted Control-T

The other method we use for implementing a logical control-$T$ makes use of a
circuit construction from Ref. [37], which is shown in Fig. 9. The figure
shows how with an ancilla qubit, a controlled-$P(\theta)$ gate can be replaced
by a $P(\theta)$, along with a Clifford+$T$ circuit and a mid-circuit
measurement and conditional $CZ$ gate. The advantage of this construction for
control-$T$ is that it yields a Clifford+$T$ circuit, which can be implemented
fault-tolerantly using methods of Ref. [24]. The disadvantage is that for
control-$P(\pi/2^{k})$ gates with $k>2$, the construction no longer results in
a Clifford+$T$ circuit.

Naively, implementing the three-qubit QFT with the ancilla-assisted
control-$T$ requires 5 logical qubit blocks: 3 data qubits plus one ancilla
for the control-$T$ plus another for the $T$ gate teleportation gadgets.
However, the QFT can be simulated with only 4 logical qubit blocks by first
using one of the data qubits as the ancilla for the control-$T$ on the other
two, before resetting that qubit as a data qubit for the remainder of the QFT
circuit. The full logical circuit therefore uses $4\times 7=28$ physical
qubits, which fits within the 32-qubit architecture of H2-1. We implement the
ancilla-assisted control-$T$ gate using method two for the $T$ gate
teleportation gadget shown in Fig. 1, since this method achieved the higher
fidelity. We benchmark this logical control-$T$ by applying it to each basis
state from two bases (see Appendix D), obtaining an average output state
fidelity of $0.79(2)$.

Figure 9:  Ancilla-assisted control-$T$ circuit. The circuit equals
control-$T$ with control $q_{0}$ and target $q_{1}$.

###  IV.3 Fidelity Lower Bounds for QFT

We benchmark the logical QFT by applying it to each basis state in a pair of
mutually unbiased bases, which is sufficient to obtain a lower bound on the
process fidelity [38]. A general explanation of this procedure is as follows.

Let $U$ be the $n$-qubit QFT, which is defined by

$U\ket{x}=\frac{1}{\sqrt{d}}\sum_{y=0}^{d-1}e^{2\pi ixy/d}\ket{y}.$ (8)

A short calculation shows that the state $U\ket{x}$ is separable:

$U\ket{x}=\bigotimes_{j=0}^{n-1}\frac{1}{\sqrt{2}}\big{(}\ket{0}+e^{2\pi
ix/2^{n-j}}\ket{1}\big{)}.$ (9)

Therefore, the fidelity of a state with respect to $U\ket{x}$ can be measured
using only single-qubit gates. Specifically, applying $HR_{z}(-2\pi
x/2^{n-j})$ to the $j$-th qubit maps the ideal state back to the all-zero
state. Next, define the Fourier basis to be
$\\{\ket{f_{x}}=U\ket{x}\\}_{x=0}^{d-1}$. The computational basis and Fourier
basis form a pair of mutually unbiased bases [39], since for all
$x,y\in\\{0,\dots,d-1\\}$, we have

$\absolutevalue{\innerproduct{y}{f_{x}}}=\frac{1}{\sqrt{d}}.$ (10)

In the same way that the outputs of the QFT on the computational basis states
can be measured using only single-qubit gates, the Fourier basis states can be
prepared using only single-qubit gates. It can be shown that the QFT acting on
a Fourier basis state gives a computational basis state, specifically

$U\ket{f_{x}}=\ket{-x\,\,(\mathrm{mod}\,\,d)}.$ (11)

It follows that the output state fidelity after applying the QFT to a Fourier
basis state can be measured with a simple computational basis measurement. We
can now make use of a theorem due to Hofmann, which gives a lower bound on the
process fidelity of a process $\mathcal{E}$ in terms of the average of its
output state fidelities, averaged over two mutually unbiased bases [38]. The
result is that if $F_{1}$ and $F_{2}$ are the averages of the output state
fidelities in the two bases, that is,

$\displaystyle F_{1}$
$\displaystyle=\frac{1}{d}\sum_{x=0}^{d-1}\bra{x}U^{{\dagger}}\mathcal{E}(\ket{x}\bra{x})U\ket{x},$
$\displaystyle F_{2}$
$\displaystyle=\frac{1}{d}\sum_{x=0}^{d-1}\bra{f_{x}}U^{{\dagger}}\mathcal{E}(\ket{f_{x}}\bra{f_{x}})U\ket{f_{x}},$
(12)

then the process fidelity of $\mathcal{E}$ is lower bounded according to

$F_{pro}(\mathcal{E},U)\geq F_{lo}=F_{1}+F_{2}-1.$ (13)

We therefore benchmark the QFT by preparing the $d$ computational basis
states, and the $d$ Fourier basis states, applying the QFT to each state and
compute the lower bound in Eq. (13). Note that the bound is stated in terms of
the process fidelity. The average fidelity, which is how we reported the
logical component fidelities in Sec. III, is related to the process fidelity
by

$F_{avg}=(dF_{pro}+1)/(d+1),$ (14)

where $d=8$ for the three-qubit QFT [40].

Although all state preparations and measurement settings in this protocol
require only single-qubit gates, for exactly half of the input states the
measurement setting in the computational basis or the state preparation in the
Fourier basis requires a $T$ or $T^{{\dagger}}$ gate. Therefore, for these
input states the fidelity measurement will include a logical SPAM error
roughly equal to the logical ${T}$ gate error. However, this error is still
small compared to the overall circuit error, as the circuit contains between 9
and 12 $T$ gates, depending on the method. The results that we report in the
next section do not correct for this SPAM error.

Finally, we note that while Eq. (13) gives a rigorous lower bound on the
process fidelity, this bound is often quite loose in practice. For example, a
process with a global depolarizing error will have an average fidelity
$F_{avg}=F_{1}=F_{2}$. While we do not expect our QFT circuits to have a
global depolarizing error, in our discussion of the results we will focus more
on the average output state fidelities $F_{1}$ and $F_{2}$ than on the lower
bound. However, we stress that it is important to apply the circuit to both
bases in order to obtain a lower bound, since there are processes that act
perfectly on one of the bases but still have a process fidelity of zero with
respect to the target.

###  IV.4 Experimental Results

We benchmark the three-qubit logical QFT circuits on the H2-1 quantum computer
[11], using both the recursive-teleportation and the ancilla-assisted
control-$T$ methods. We apply the logical QFT to each basis state in the
computational and Fourier bases, and run each logical circuit for 100 shots.
For the ancilla-assisted method, we average two full data sets taken roughly
one week apart. Bar plots of the state fidelities for each circuit are shown
in Fig. 10, and the quantities $F_{1}$ and $F_{2}$ as well as the average
fidelity lower bounds are summarized in Table 3. We find that the
computational basis outperforms the Fourier basis for both methods, with
$F_{1}=0.72(2)$ and $F_{2}=0.65(2)$ for the recursive-teleportation method and
$F_{1}=0.78(1)$ and $F_{2}=0.66(1)$ for the ancilla-assisted method. If we
post-select on the measurements in the teleportation gadgets, the fidelities
increase, up to $0.89(1)$ for the computational basis with the ancilla-
assisted method. The fraction of shots retained when post-selecting were
0.35(1) for the the recursive-teleportation method and 0.31(1) for the
ancilla-assisted method. Interestingly, this post-selection removes the bias
between the computational and Fourier bases for the recursive-teleportation
method, but not the ancilla-assisted method. We conclude that quantum error
detection of this form, while not scalable, can be useful for improving the
performance of near-term logical circuit experiments.

Experiment | $F_{1}$ | $F_{2}$ |  $F_{avg}$ bound
---|---|---|---
Recursive-teleportation | 0.72(2) | 0.65(2) | 0.44(2)
Recursive-teleportation (P.S.) | 0.82(2) | 0.81(2) | 0.68(3)
Ancilla-assisted | 0.78(1) | 0.66(1) | 0.50(1)
Ancilla-assisted (P.S.) | 0.89(1) | 0.77(2) | 0.69(2)
Table 3: QFT output state fidelities. $F_{1}$ and $F_{2}$ are the average
output state fidelity for the computational and Fourier bases, respectively.
Post-selected (P.S.) refers to discarding shots where the measurement in the
teleportation gadget detected an error, as explained in Sec. II.

Given the complexity of the QFT circuits run in this experiment, it is useful
to ask whether the results can be accounted for by the logical component
errors. From Table. 1, we see that the logical $T$ gate is the leading
contributor to circuit error with an average fidelity 0.990(1), compared to
0.9980(8) for logical CNOT. We can obtain a heuristic estimate for an output
state fidelity of the QFT circuit by assuming single-qubit and two-qubit
depolarizing error channels for the logical $T$ and two-qubit gates. A
depolarizing channel with parameter $\epsilon$ acts on a state $\rho$
according to

$\mathcal{D}(\rho)=(1-\epsilon)\rho+\epsilon I/d,$ (15)

where the depolarizing parameter is related to the average fidelity by

$1-F_{avg}=\frac{d-1}{d}\epsilon.$ (16)

We use Qiskit [27] to compute the output state fidelities of the ancilla-
assisted QFT circuit, applied to the computational and Fourier bases, and
assuming a depolarizing error model for the logical components. We only model
the QFT via the ancilla-assisted method, since this yields a Clifford+$T$
circuit, whereas the recursive-teleportation gadget implements a logical
$P(\frac{\pi}{8})$ gate that we did not benchmark, though we expect the
logical infidelity of the $\ket{\pi/8}$ state preparation to be comparable to
that of the $\ket{T}$ state. From our model, we obtain average output state
fidelities of $F_{1}=0.898$ and $F_{2}=0.889$. The experimental values for
$F_{1}$ and $F_{2}$ are noticeably lower than the heuristic prediction,
indicating additional error that is not accounted for by the component level
benchmarks alone. Our hypothesis is that memory error during ion transport
leads to system-level error that is not present in the component benchmarking
experiments.

(a)  (b)  (c)  (d)

Figure 10: Logical QFT output state fidelities on the H2-1 device for (a)
recursive-teleportation method, computational basis, (b) recursive-
teleportation method, Fourier basis, (c) ancilla-assisted method,
computational basis, (d) ancilla-assisted method, Fourier basis. The blue and
green bars are without and with post-selection on the syndromes from the
measurements in the $T$ gate teleportation gadgets, as explained in Sec. II.
The circuits were run with 100 and 200 total shots for the recursive-
teleportation and ancilla-assisted methods, respectively.

##  V Conclusion

In this work, we have reported the first demonstration of a logical three-
qubit QFT circuit, as part of our efforts at characterizing the performance of
circuits encoded in a QEC code. Learning from the development of quantum
processors at the physical level, the path toward high performance at the
logical level will likely require the careful study of the different
components afforded by different encodings, such as the gate sets, syndrome
extraction procedures, and magic state injection methods.

While this study is not comprehensive for even the particular code used in
this investigation, our results show that the choices between different
implementations of logical components matter. For example, we found that on
the H2-1 device, the $T$ gadget teleportation method two from Fig. 1 was the
superior one. Overall, the non-Clifford gate operations are the dominant
source of error. We note that an accounting of the error sources using a gate
level error model along with their respective measured component error rates
does not match the total error observed in the system-level benchmark of the
QFT. These estimates would predict state fidelities just below $0.9$, whereas
the measurements were at most $0.78$. This discrepancy could be due to several
omissions, including memory error, leakage errors, or the coherent
accumulation of noise.

These results suggest at least two future areas of study: (1) it is imperative
that we close the gap in our understanding between the component-level and
system-level error rates on the logical level, and (2) we must find better
methods for implementing non-Clifford operations through a combination of
hardware improvements and code developments including higher code distances
and the eventual use of fully fault-tolerant operations. Experimental fault-
tolerant quantum computing still has many challenges ahead. There are
different codes to explore, different protocols within those codes, and all of
them will have different strengths and weaknesses in the context of real
hardware that can be built. Sorting through these methods is the high-level
challenge, but before that, researchers need to develop the tools to make that
sorting feasible.

We hope this work, with the development of logical benchmarking enabled by
high-level programming tools, lays a good foundation for the experimental side
of this challenge. Furthermore, we hope that this work opens the door to
implementing other algorithms based on the QFT at the logical level, such as
solving the one-dimensional wave equation [41]. Finally, we believe tracking
the state-of-the-art performance of a logical QFT in real quantum processors
would make a valuable system-level benchmark for logical encodings in quantum
processors, one that we hope the larger community can adopt.

## Code Availability

The OpenQasm programs that were submitted to the H-series hardware, and the
code for analyzing the data and making the plots in this paper, are available
here [42]. The SLR code base, which was used to create the logical circuits
and convert the circuits to OpenQasm, is available here [16].

###### Acknowledgements.

We thank Ben Criger for early discussions and suggestions. We thank the entire
Quantinuum hardware team for making these experiments possible. The ion traps
used in this work were fabricated by Honeywell.

##  Appendix A Hardware Overview

The experiments were run on the Quantinuum H1-1 and H2-1 devices. H1-1 is a
20-qubit trapped-ion QCCD quantum computer that operates on a linear surface
trap described in Ref. [10]. H2-1 is a 32-qubit trapped-ion QCCD quantum
computer that operates on a racetrack surface trap described in Ref. [11].
Both systems use ground state hyperfine levels of ${}^{171}\textrm{Yb}+$ ions
as physical qubits, along with ${}^{138}\textrm{Ba}+$ ions for sympathetic
cooling. The systems feature physical native single-qubit (1Q) and two-qubit
(2Q) gates given by
$U_{1Q}(\theta,\phi)=\exp[-i\theta(X\cos\phi+Y\sin\phi)/2]$, and
$U_{2Q}(\theta)=\exp[-i\theta ZZ/2]$, and allow for mid-circuit measurement
and reset with low crosstalk errors [43]. Ion transport, which includes linear
shifts, physical swaps, and split combines, allows for arbitrary rearrangement
and pairing but induces small memory errors (relative to the 2Q gate error).
Table 4 contains the relevant error magnitudes for different component
operations. The tests used to measure each error magnitude are further
described in Ref. [11]. Complete datasets are given in Ref. [31].

Error | H1-1 | H2-1
---|---|---
Single-qubit | 0.29(5) | 0.25(3)
Two-qubit | 8.8(3) | 18.3(5)
Memory | 1.8(2) | 2.2(3)
Measurement crosstalk | 0.083(9) | 0.045(6)
State preparation and measurement | 26(1) | 16(1)
Table 4: Error parameters for H1-1 and H2-1 devices used for experiments and
all values are $\times 10^{-4}$. Single-qubit and two-qubit gate error is
defined as one minus average fidelity.

##  Appendix B Simulation of Partially-Fault-Tolerant Logical T Benchmarking

To prepare the state $\ket{\overline{T}}$, we first prepare the state

$\ket{\overline{H}}=\cos(\pi/8)\ket{\overline{0}}+\sin(\pi/8)\ket{\overline{1}},$
(17)

from which $\ket{\overline{T}}$ is obtained via logical Clifford gates,
according to

$\ket{\overline{T}}=\overline{R_{z}(\pi/2)}\,\overline{R_{x}(\pi/2)}\ket{\overline{H}}.$
(18)

As shown in Ref. [24], the state $\ket{\overline{H}}$ can be prepared fault-
tolerantly by first using the circuit in Fig. 3 to prepare
$\ket{\overline{H}}$ non-fault-tolerantly, and then measuring $\overline{H}$
with the use of a flag qubit to detect bad errors, and finally performing a
round of QEC. If the final QEC round is omitted, but the flagged
$\overline{H}$ measurement is still made, the procedure is called partially-
fault-tolerant and was demonstrated in Ref. [25]. Since $\overline{H}^{2}=I$
and $\overline{H}$ is Clifford, $\overline{H}$ can be measured by applying a
control-$H$ between an ancilla physical qubit prepared in $\ket{+}$ and each
physical qubit in the logical code block, and then measuring the ancilla in
the $X$ basis. This circuit is shown in Fig. 12. If the ancilla qubit and the
flag qubit are both measured as ‘0’, the fault-tolerant preparation succeeds
and $\ket{\overline{T}}$ is input into the teleportation gadget in Fig. 1 to
implement a $\overline{T}$ gate. The $\ket{\overline{H}}$ state preparation
procedure can be performed as a repeat-until-success protocol, with a limit on
the number of repetitions. If the limit is reached without a successful state
preparation, then that shot is discarded in post-selection, though we note
that in a larger quantum computer, a magic state factory could prepare
multiple states in parallel so that there is always a resource state ready
when a $\overline{T}$ gate is to be performed.

We benchmark the partially-fault-tolerant $\overline{T}$ gate on the device
emulator H1-1E, using the teleportation gadget method two in Fig. 1, and the
same benchmarking procedure as in Sec. III for the non-fault-tolerant
$\overline{T}$, with the RUS limit chosen from $\\{1,2\\}$. The decay curves
and post-selection rate, which is the fraction of shots retained are shown in
Fig. 11. The plot shows that RUS limit one achieves a higher average fidelity,
since there are fewer gates on average, at the cost of more post-selection
needed. The average fidelities and post-selection rates are listed in Table.
5. Given the rate of post-selection, and the fact that the average fidelity
per logical $T$ was simulated to be slightly lower than our experimentally
measured non-fault-tolerant fidelity, we chose to implement the QFT circuits
using a non-fault-tolerant logical $T$.

Figure 11: Simulation of partially fault-tolerant logical $T$ gate benchmarking on H1-1E. The fault-tolerant gate uses a repeat-until-success (RUS) protocol, with a limit to the number of repetitions, chosen from $\\{1,2\\}$. The solid and dashed lines are survival probability and post-selection rate, respectively. 10 random circuits were chosen per sequence length. Each circuit was run for 1000 shots. RUS limit | Average Fidelity | Post-selection Rate ($L=8$)
---|---|---
1 | 0.988(1) | 0.75(1)
2 | 0.931(4) | 0.990(3)
Table 5: Average fidelity of logical $T$ and post-selection rate (fraction of
shots retained) at sequence length 8, for different repeat-until-success (RUS)
limits, from simulation on the emulator H1-1E. The fidelity is obtained from
best-fitting the decay curves in Fig. 11 Figure 12:  Logical Hadamard
measurement with a flag qubit, used to make logical state preparation fault-
tolerant when followed by a round of QEC. The measurement of the first ancilla
records the value of the logical Hadamard, and the second ancilla is a flag
qubit, for which a result of ‘1’ indicates an error that can spread to an
uncorrectable error. The encoding circuit in Fig. 3 along with the circuit
above is repeated until both ancilla qubits are measured as ‘0’.

##  Appendix C SLR Example

In this section we walk through an SLR program that prepares a logical qubit
in the logical $\ket{+}$ state, applies logical $T$ via a teleportation
gadget, and finally measures in the logical $X$ basis. The first step is to
initialize the SLR program with the quantum and classical registers required
for this task.

⬇

program = SLR(

q := QReg(’q’, 7), # logical qubit

a := QReg(’a’, 7), # logical ancilla

# classical registers for output

c := CReg(’c’, 7),

cLogRaw := CReg(’cLogRaw’, 1),

cLog := CReg(’cLog’, 1),

cSyn := CReg(’cSyn’, 3),

# classical register for RUS init

init := CReg(’init’, 1),

# classical registers for T gadget

cT := CReg(’cT’, 7),

cTLogRaw := CReg(’cTLogRaw’, 1),

cTLog := CReg(’cTLog’, 1),

cTSyn := CReg(’cTSyn’, 3),

)

We then initialize the logical qubit using the fault-tolerant repeat-until-
success protocol described in Sec. II.

⬇

# fault tolerant state initialization

program.extend(

InitRUS(q, a[0], init[0], limit=3),

H(q),

)

Here we have used the physical qubit a[0] as the ancilla qubit for the
initialization. The ancilla register will be reused for the
$\ket{\overline{T}}$ preparation. The RUS protocol succeeds if the classical
bit init[0] has a value of ‘0’. We then apply the $\overline{T}$ gate
teleportation gadget.

⬇

# T gate gadget

program.extend(

InitTPlusNonFT(a),

CX(q,a),

MeasDecode(a,cT,cTLogRaw,cTLog,cTSyn),

If(cTLog[0]==1).Then(S(q)),

)

Here the MeasDecode object represents a measurement of the logical qubit a
into the classical 7-bit register cT, with a 1-bit register cTLog that
contains the logical binary value computed from cT, including the syndrome
information and decoding. Finally, we measure the logical qubit q in the $X$
basis.

⬇

# measure

program.extend(

H(q),

MeasDecode(q,c,cLogRaw,cLog,cSyn),

)

##  Appendix D Fidelity Lower Bounds for Control-T

The control-$T$ gate has the following action on the two-qubit bases of mixed
$X$ and $Z$ eigenstates:

$\displaystyle\ket{+}\ket{0}$ $\displaystyle\mapsto\ket{+}\ket{0}$
$\displaystyle\ket{+}\ket{1}$ $\displaystyle\mapsto\ket{T}\ket{1}$
$\displaystyle\ket{-}\ket{0}$ $\displaystyle\mapsto\ket{-}\ket{0}$
$\displaystyle\ket{-}\ket{1}$ $\displaystyle\mapsto\ket{T-}\ket{1}$ (19)

$\displaystyle\ket{0}\ket{+}$ $\displaystyle\mapsto\ket{0}\ket{+}$
$\displaystyle\ket{0}\ket{-}$ $\displaystyle\mapsto\ket{0}\ket{-}$
$\displaystyle\ket{1}\ket{+}$ $\displaystyle\mapsto\ket{1}\ket{T}$
$\displaystyle\ket{1}\ket{-}$ $\displaystyle\mapsto\ket{1}\ket{T-}$ (20)

Here, the left-most ket is the state of the control qubit and
$\ket{T-}=T\ket{-}$. These two bases form a pair of mutually unbiased bases,
so the process fidelity lower bound of Eq. (13) applies. We implement the
logical control-$T$ on H2-1 using the ancilla-assisted method described in
Sec. IV, and teleportation method two for the $T$ gates. We apply logical
control-$T$ to each logical basis state in the two bases of mixed $X$ and $Z$
eigenstates. As seen in Eq. (D) and Eq. (D), for half of the input states the
ideal output state includes a $\ket{T}$ or $\ket{T-}$. In this case we apply a
$T^{{\dagger}}$ in order to measure the state fidelity, and therefore the
fidelity measurement has a SPAM error that is at least as large as that of a
logical $T$ gate. The output state fidelities are plotted in Fig. 13, where we
do not correct for the logical SPAM error. We obtain average state fidelities
of $F_{1}=0.79(2)$ and $F_{2}=0.79(2)$ over the two bases, from which we
obtain a fidelity lower bound of $F_{avg}\geq 0.66(2)$. For comparison, when
we post-select on the syndrome information from the measurement in the
teleportation gadgets, we obtain $F_{1}=0.82(3)$ and $F_{2}=0.88(2)$ with a
lower bound $F_{avg}\geq 0.76(3)$.

Figure 13: Output state fidelities for logical control-$T$ on the device H2-1,
via the ancilla-assisted method described in Sec. IV. The blue (green) bars
are without (with) post-selection on the syndromes from the measurements in
the $T$ gate teleportation gadgets. All circuits were run for 100 shots.

## References

  * Gottesman [2016] D. Gottesman, Quantum fault tolerance in small experiments (2016), arXiv:1610.03507 [quant-ph] .
  * Ryan-Anderson  _et al._ [2021] C. Ryan-Anderson, J. G. Bohnet, K. Lee, D. Gresh, A. Hankin, J. P. Gaebler, D. Francois, A. Chernoguzov, D. Lucchetti, N. C. Brown, T. M. Gatterman, S. K. Halit, K. Gilmore, J. A. Gerber, B. Neyenhuis, D. Hayes, and R. P. Stutz, Phys. Rev. X 11, 041058 (2021).
  * Ryan-Anderson  _et al._ [2022] C. Ryan-Anderson, N. C. Brown, M. S. Allman, B. Arkin, G. Asa-Attuah, C. Baldwin, J. Berg, J. G. Bohnet, S. Braxton, N. Burdick, J. P. Campora, A. Chernoguzov, J. Esposito, B. Evans, D. Francois, J. P. Gaebler, T. M. Gatterman, J. Gerber, K. Gilmore, D. Gresh, A. Hall, A. Hankin, J. Hostetter, D. Lucchetti, K. Mayer, J. Myers, B. Neyenhuis, J. Santiago, J. Sedlacek, T. Skripka, A. Slattery, R. P. Stutz, J. Tait, R. Tobey, G. Vittorini, J. Walker, and D. Hayes, Implementing fault-tolerant entangling gates on the five-qubit code and the color code (2022), arXiv:2208.01863 [quant-ph] .
  * Self  _et al._ [2024] C. N. Self, M. Benedetti, and D. Amaro, Nature Physics (2024).
  * Yamamoto  _et al._ [2024] K. Yamamoto, S. Duffield, Y. Kikuchi, and D. Muñoz Ramo, Phys. Rev. Res. 6, 013221 (2024).
  * Wang  _et al._ [2023] Y. Wang, S. Simsek, T. M. Gatterman, J. A. Gerber, K. Gilmore, D. Gresh, N. Hewitt, C. V. Horst, M. Matheny, T. Mengle, B. Neyenhuis, and B. Criger, Fault-tolerant one-bit addition with the smallest interesting colour code (2023), arXiv:2309.09893 [quant-ph] .
  * Gupta  _et al._ [2024] R. S. Gupta, N. Sundaresan, T. Alexander, C. J. Wood, S. T. Merkel, M. B. Healy, M. Hillenbrand, T. Jochym-O’Connor, J. R. Wootton, T. J. Yoder, A. W. Cross, M. Takita, and B. J. Brown, Nature 625, 259 (2024).
  * Bluvstein  _et al._ [2024] D. Bluvstein, S. J. Evered, A. A. Geim, S. H. Li, H. Zhou, T. Manovitz, S. Ebadi, M. Cain, M. Kalinowski, D. Hangleiter, J. P. Bonilla Ataides, N. Maskara, I. Cong, X. Gao, P. Sales Rodriguez, T. Karolyshyn, G. Semeghini, M. J. Gullans, M. Greiner, V. Vuletić, and M. D. Lukin, Nature 626, 58 (2024).
  * da Silva  _et al._ [2024] M. P. da Silva, C. Ryan-Anderson, J. M. Bello-Rivas, A. Chernoguzov, J. M. Dreiling, C. Foltz, J. P. Gaebler, T. M. Gatterman, D. Hayes, N. Hewitt, J. Johansen, D. Lucchetti, M. Mills, S. A. Moses, B. Neyenhuis, A. Paz, J. Pino, P. Siegfried, J. Strabley, S. J. Wernli, R. P. Stutz, and K. M. Svore, Demonstration of logical qubits and repeated error correction with better-than-physical error rates (2024), arXiv:2404.02280 [quant-ph] .
  * Pino  _et al._ [2021] J. M. Pino, J. M. Dreiling, C. Figgatt, J. P. Gaebler, S. A. Moses, M. S. Allman, C. H. Baldwin, M. Foss-Feig, D. Hayes, K. Mayer, C. Ryan-Anderson, and B. Neyenhuis, Nature 592, 209 (2021).
  * Moses  _et al._ [2023] S. A. Moses, C. H. Baldwin, M. S. Allman, R. Ancona, L. Ascarrunz, C. Barnes, J. Bartolotta, B. Bjork, P. Blanchard, M. Bohn, J. G. Bohnet, N. C. Brown, N. Q. Burdick, W. C. Burton, S. L. Campbell, J. P. Campora, C. Carron, J. Chambers, J. W. Chan, Y. H. Chen, A. Chernoguzov, E. Chertkov, J. Colina, J. P. Curtis, R. Daniel, M. DeCross, D. Deen, C. Delaney, J. M. Dreiling, C. T. Ertsgaard, J. Esposito, B. Estey, M. Fabrikant, C. Figgatt, C. Foltz, M. Foss-Feig, D. Francois, J. P. Gaebler, T. M. Gatterman, C. N. Gilbreth, J. Giles, E. Glynn, A. Hall, A. M. Hankin, A. Hansen, D. Hayes, B. Higashi, I. M. Hoffman, B. Horning, J. J. Hout, R. Jacobs, J. Johansen, L. Jones, J. Karcz, T. Klein, P. Lauria, P. Lee, D. Liefer, S. T. Lu, D. Lucchetti, C. Lytle, A. Malm, M. Matheny, B. Mathewson, K. Mayer, D. B. Miller, M. Mills, B. Neyenhuis, L. Nugent, S. Olson, J. Parks, G. N. Price, Z. Price, M. Pugh, A. Ransford, A. P. Reed, C. Roman, M. Rowe, C. Ryan-Anderson, S. Sanders, J. Sedlacek, P. Shevchuk, P. Siegfried, T. Skripka, B. Spaun, R. T. Sprenkle, R. P. Stutz, M. Swallows, R. I. Tobey, A. Tran, T. Tran, E. Vogt, C. Volin, J. Walker, A. M. Zolot, and J. M. Pino, Phys. Rev. X 13, 041052 (2023).
  * Steane [1996] A. Steane, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences 452, 2551–2577 (1996).
  * Knill  _et al._ [2008] E. Knill, D. Leibfried, R. Reichle, J. Britton, R. B. Blakestad, J. D. Jost, C. Langer, R. Ozeri, S. Seidelin, and D. J. Wineland, Phys. Rev. A 77, 012307 (2008).
  * Magesan  _et al._ [2011] E. Magesan, J. M. Gambetta, and J. Emerson, Phys. Rev. Lett. 106, 180504 (2011).
  * Piveteau  _et al._ [2021] C. Piveteau, D. Sutter, S. Bravyi, J. M. Gambetta, and K. Temme, Phys. Rev. Lett. 127, 200505 (2021).
  * Ciaran Ryan-Anderson [2018] Ciaran Ryan-Anderson, Pecos: Performance estimator of codes on surfaces, https://github.com/PECOS-packages/PECOS (2018).
  * Gottesman [1997] D. Gottesman,  _Stabilizer codes and quantum error correction_ (California Institute of Technology, 1997).
  * Bombin and Martin-Delgado [2007] H. Bombin and M.-A. Martin-Delgado, Physical review letters 98, 160502 (2007).
  * Landahl  _et al._ [2011] A. J. Landahl, J. T. Anderson, and P. R. Rice, Fault-tolerant quantum computing with color codes (2011), arXiv:1108.5738 [quant-ph] .
  * Landahl and Ryan-Anderson [2014] A. J. Landahl and C. Ryan-Anderson, arXiv preprint arXiv:1407.5103 (2014).
  * Gottesman [2009] D. Gottesman, An introduction to quantum error correction and fault-tolerant quantum computation (2009), arXiv:0904.2557 [quant-ph] .
  * Goto [2016] H. Goto, Scientific reports 6, 1 (2016).
  * Eastin and Knill [2009] B. Eastin and E. Knill, Phys. Rev. Lett. 102, 110502 (2009).
  * Chamberland and Cross [2019] C. Chamberland and A. W. Cross, Quantum 3, 143 (2019).
  * Postler  _et al._ [2022] L. Postler, S. Heußen, I. Pogorelov, M. Rispler, T. Feldker, M. Meth, C. D. Marciniak, R. Stricker, M. Ringbauer, R. Blatt, P. Schindler, M. Müller, and T. Monz, Nature 605, 675 (2022).
  * Sivarajah  _et al._ [2020] S. Sivarajah, S. Dilkes, A. Cowtan, W. Simmons, A. Edgington, and R. Duncan, Quantum Science and Technology 6, 014003 (2020).
  * Qiskit contributors [2023] Qiskit contributors, Qiskit: An open-source framework for quantum computing (2023).
  * Cross  _et al._ [2017] A. W. Cross, L. S. Bishop, J. A. Smolin, and J. M. Gambetta, Open quantum assembly language (2017), arXiv:1707.03429 [quant-ph] .
  * Mele [2023] A. A. Mele, Introduction to haar measure tools in quantum information: A beginner’s tutorial (2023), arXiv:2307.08956 [quant-ph] .
  * Combes  _et al._ [2017] J. Combes, C. Granade, C. Ferrie, and S. T. Flammia, Logical randomized benchmarking (2017), arXiv:1702.03688 [quant-ph] .
  * git [2023] Quantinuum hardware specificiations (2023).
  * Temme  _et al._ [2017] K. Temme, S. Bravyi, and J. M. Gambetta, Phys. Rev. Lett. 119, 180509 (2017).
  * Barenco  _et al._ [1995] A. Barenco, C. H. Bennett, R. Cleve, D. P. DiVincenzo, N. Margolus, P. Shor, T. Sleator, J. A. Smolin, and H. Weinfurter, Phys. Rev. A 52, 3457 (1995).
  * Ross and Selinger [2016] N. J. Ross and P. Selinger, Quantum Info. Comput. 16, 901–953 (2016).
  * Koczor  _et al._ [2024] B. Koczor, J. J. L. Morton, and S. C. Benjamin, Phys. Rev. Lett. 132, 130602 (2024).
  * Choi  _et al._ [2023] H. Choi, F. T. Chong, D. Englund, and Y. Ding, Fault tolerant non-clifford state preparation for arbitrary rotations (2023), arXiv:2303.17380 [quant-ph] .
  * Nam  _et al._ [2020] Y. Nam, Y. Su, and D. Maslov, npj Quantum Information 6, 26 (2020).
  * Hofmann [2005] H. F. Hofmann, Phys. Rev. Lett. 94, 160504 (2005).
  * Wootters and Fields [1989] W. K. Wootters and B. D. Fields, Annals of Physics 191, 363 (1989).
  * Nielsen [2002] M. A. Nielsen, Physics Letters A 303, 249 (2002).
  * Wright  _et al._ [2024] L. Wright, C. M. Keever, J. T. First, R. Johnston, J. Tillay, S. Chaney, M. Rosenkranz, and M. Lubasch, Noisy intermediate-scale quantum simulation of the one-dimensional wave equation (2024), arXiv:2402.19247 [quant-ph] .
  * Mayer [2024] K. Mayer, Data for ‘benchmarking logical three-qubit quantum fourier transform encoded in the steane code on a trapped-ion quantum computer’ (2024).
  * Gaebler  _et al._ [2021] J. P. Gaebler, C. H. Baldwin, S. A. Moses, J. M. Dreiling, C. Figgatt, M. Foss-Feig, D. Hayes, and J. M. Pino, Phys. Rev. A 104, 062440 (2021).
  * Preskill [1998] J. Preskill, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences 454, 385 (1998).
  * Gottesman and Chuang [1999] D. Gottesman and I. L. Chuang, Nature 402, 390 (1999).
  * Zhou  _et al._ [2000] X. Zhou, D. W. Leung, and I. L. Chuang, Physical Review A 62, 052316 (2000).


# Attitude Control of Highly Maneuverable Aircraft Using an Improved
Q-learning

Mohsen. Zahmatkesh  Seyyed. Ali. Emami  Afshin. Banazadeh  Paolo. Castaldi
Aerospace Engineering Department, Sharif University of Technology, Tehran,
Iran (e-mail: banazadeh@sharif.edu).  Department of Electrical, Electronic and
Information Engineering ”Guglielmo Marconi”, University of Bologna, Via
Dell’Universit‘a 50, Cesena, Italy (e-mail<EMAIL_ADDRESS>
###### Abstract

Attitude control of a novel regional truss-braced wing aircraft with low
stability characteristics is addressed in this paper using Reinforcement
Learning (RL). In recent years, RL has been increasingly employed in
challenging applications, particularly, autonomous flight control. However, a
significant predicament confronting discrete RL algorithms is the dimension
limitation of the state-action table and difficulties in defining the elements
of the RL environment. To address these issues, in this paper, a detailed
mathematical model of the mentioned aircraft is first developed to shape an RL
environment. Subsequently, Q-learning, the most prevalent discrete RL
algorithm will be implemented in both the Markov Decision Process (MDP), and
Partially Observable Markov Decision Process (POMDP) frameworks to control the
longitudinal mode of the air vehicle. In order to eliminate residual
fluctuations that are a consequence of discrete action selection, and
simultaneously track variable pitch angles, a Fuzzy Action Assignment (FAA)
method is proposed to generate continuous control commands using the trained
Q-table. Accordingly, it will be proved that by defining an accurate reward
function, along with observing all crucial states (which is equivalent to
satisfy the Markov Property), the performance of the introduced control system
surpasses a well-tuned Proportional–Integral–Derivative (PID) controller.

###### keywords:

Reinforcement Learning, Q-learning, Fuzzy Q-learning, Attitude Control, Truss-
braced Wing, Flight Control

##  1 Introduction

The aviation industry is expeditiously growing due to world demands such as
reducing fuel burn, emissions, and cost, as well as providing the faster and
safer flight. This motivates the advent of new airplanes with novel
configurations. In addition, the scope clause agreement limits the number of
seats in each aircraft and flight outsourcing to protect the union pilot jobs.
This factor leads to an increase in production of the Modern Regional Jet
(MRJ) airplane. In this regard, the importance of a safe flight becomes more
vital considering more crowded airspace and new aircraft configurations having
the ability to fly faster. Truss-braced wing aircraft is one of the re-raised
high-performance configurations, which has attracted significant attention
from both academia (Li et al., 2022) and industry (Sarode, 2022) due to its
fuel burn efficiency. As a result, there would be a growing need for reliable
modeling and simulations, analyzing the flight handling quality, and stability
analysis for such configurations (Nguyen and Xiong, 2022; Zavaree et al.,
2021), while very few studies have addressed the flight control design for
this aircraft.

In the last decades, various classic methods for aircraft attitude control
have been developed to enhance control performance. However, the most
significant deficiency of these approaches is the insufficient capability to
deal with unexpected flight conditions, while typically requiring a detailed
dynamic model of the system.

Recently, the application of Reinforcement Learning (RL) has been extended to
real problems, particularly, flight control design (Emami et al., 2022).
Generally, there are two main frameworks to incorporate RL in the control
design process, i.e., the high-level and low-level control systems. In Xi et
al. (2022), a Soft Actor-Critic (SAC) algorithm was implemented in a path
planning problem for a long-endurance solar-powered UAV with energy-consuming
considerations. Another work (Bøhn et al., 2021) concentrated on the inner
loop control of a Skywalker X8 using SAC and comparing it with a PID
controller. In Yang et al. (2020) a ANN based Q-learning horizontal trajectory
tracking controller was developed based on the MDP model of an airship with
fine stability characteristics. Apart from the previous method, Proximal
Policy Optimization (PPO) was utilized in Hu et al. (2022) for orientation
control of a common strongly dynamic coupled fixed-wing aircraft in the stall
condition. The PPO was successful to be converged after 100000 episodes.
However, useful to say that the PPO performance is adequate to optimize PID
controllers (Dammen, 2022).

There are some papers on maneuver flight such as landing phase control both in
inner and outer loops. For instance, in Wang et al. (2018), a Deep Q-learning
(DQL) is used to guide an aircraft to land in the desired field. In Yuan et
al. (2019), a Deep Deterministic Policy Gradient (DDPG) was implemented for a
UAV to control either path-tracking for landing glide slope and attitude
control for landing flare section. Similarly, a DDPG method in Tang and Lai
(2020) is used to control outer loop of a landing procedure in presence of
wind disturbance. The works which have been referred to so far accompanied
ANNs to be able to converge. But to our best knowledge, there is research in
attitude control using discrete RL without aiming ANNs. In Richter et al.
(2022), a Q-learning algorithm was implemented to control longitudinal and
lateral angles in a general aviation aircraft(Cessna 172). This airplane
profits suitable stability characteristics and also desired angles are zero.
There are some fuzzy adaptations on Watkins and Dayan (1992) work like
Glorennec and Jouffe (1997) where the Q-functions and action selection
strategy are inferred from fuzzy rules. Also, Er and Deng (2004) proposed a
dynamic fuzzy Q-learning for online and continuous tasks in mobile robots.

Motivated by the above discussions, the main contributions of the current
study can be summarized as follows: a) A truss-braced wing aircraft (Chaka 50)
(1) with poor stability characteristics has been selected carefully for
attitude control alongside responding to global aviation society demands. b)
It will be proven that the Q-learning performance in control problems strictly
depends on reward function and problem definition. So, it is able to have
prosperous results even in a low stable high degree of freedom plants. c) In
this work, the performance of Q-learning will be examined in both MDP and
POMDP problem modelings. Also, the learned Q-table is used to generate
continuous elevator deflections using Fuzzy Action Assignment (FAA)
illustrating Q-table capability tracking the desired angle and also variable
angles.

Figure 1: Chaka MRJ Family (Zavaree et al., 2021)

##  2 Modeling and Simulation

Nonlinear conservation of linear and angular momentum equations are used for
modeling and simulation according to Zipfel (2014).

$\displaystyle m\begin{bmatrix}\dot{u}\\\ \dot{v}\\\
\dot{w}\end{bmatrix}+m\begin{bmatrix}0&-r&q\\\ r&0&-p\\\
-q&p&0\end{bmatrix}\begin{bmatrix}u\\\ v\\\
w\end{bmatrix}=\begin{bmatrix}F_{A_{x}+T_{x}}\\\ F_{A_{y}+T_{y}}\\\
F_{A_{z}+T_{z}}\end{bmatrix}+m\begin{bmatrix}g_{x}\\\ g_{y}\\\
g_{z}\end{bmatrix}$ (1) $\displaystyle\begin{bmatrix}I_{x}&0&0\\\ 0&I_{y}&0\\\
0&0&I_{z}\end{bmatrix}\begin{bmatrix}\dot{p}\\\ \dot{q}\\\
\dot{r}\end{bmatrix}+\begin{bmatrix}0&-r&q\\\ r&0&-p\\\
-q&p&0\end{bmatrix}\begin{bmatrix}I_{x}&0&0\\\ 0&I_{y}&0\\\
0&0&I_{z}\end{bmatrix}\begin{bmatrix}p\\\ q\\\
r\end{bmatrix}=\begin{bmatrix}L_{A+T}\\\ M_{A+T}\\\ N_{A+T}\end{bmatrix}$ (2)

where assuming the moments of thrust are equal to zero and also the thrust is
only implying in $x$ direction. Therefore,
$L_{T}=M_{T}=N_{T}=F_{T_{y}}=F_{T_{z}}=0$. and the aerodynamic forces and
moments in body axis are as follows:

$\displaystyle\begin{bmatrix}F_{A_{x}}\\\ F_{A_{z}}\\\
M_{A}\end{bmatrix}^{B}=\bar{q}S\bar{c}\begin{bmatrix}c_{L_{0}}&c_{L_{\alpha}}&c_{L_{\dot{\alpha}}}&c_{L_{u}}&c_{L_{q}}&c_{L_{\delta_{E}}}\\\
c_{D_{0}}&c_{D_{\alpha}}&c_{D_{\dot{\alpha}}}&c_{D_{u}}&c_{D_{q}}&c_{D_{\delta_{E}}}\\\
c_{m_{0}}&c_{m_{\alpha}}&c_{m_{\dot{\alpha}}}&c_{m_{u}}&c_{m_{q}}&c_{m_{\delta_{E}}}\end{bmatrix}\begin{bmatrix}1\\\
\alpha\\\ \frac{\dot{\alpha}\bar{c}}{2V_{P_{1}}}\\\ \frac{u}{V_{P_{1}}}\\\
\frac{q\bar{c}}{2V_{P_{1}}}\\\ \delta_{E}\end{bmatrix}$ (3)

The vector of gravity acceleration in the body axis defined by (1) is as
follows:

$\begin{bmatrix}g_{x}\\\ g_{y}\\\
g_{z}\end{bmatrix}^{B}=\begin{Bmatrix}-g\sin(\theta)\\\
g\cos(\theta)\sin(\phi)\\\ g\cos(\theta)\cos(\phi)\end{Bmatrix}$ (4)

And also, the rotational kinematic equations are necessary for transfer from
body to inertia coordinates.

$\begin{bmatrix}\dot{\phi}\\\ \dot{\theta}\\\
\dot{\psi}\end{bmatrix}=\begin{bmatrix}1&\sin\varphi\tan\theta&\cos\varphi\tan\theta\\\
0&\cos\varphi&-\sin\varphi\\\
0&\sin\varphi/\cos\theta&\cos\varphi/\cos\theta\end{bmatrix}\begin{bmatrix}p\\\
q\\\ r\end{bmatrix}$ (5)

using (1), and (5), velocity vector in inertia coordinate is achievable. For
contraction reasons; $\sin=\textup{s},\cos=\textup{c}$.

$\begin{bmatrix}\dot{x}\\\ \dot{y}\\\
\dot{z}\end{bmatrix}^{I}=\begin{bmatrix}\textup{c}\psi\textup{c}\theta&\textup{c}\psi\textup{s}\theta\textup{s}\varphi-\textup{s}\psi\textup{c}\varphi&\textup{c}\psi\textup{s}\theta\textup{c}\varphi+\textup{s}\psi\textup{s}\varphi\\\
\textup{s}\psi\textup{c}\theta&\textup{s}\psi\textup{s}\theta\textup{s}\varphi+\textup{c}\psi\textup{c}\varphi&\textup{s}\psi\textup{s}\theta\textup{c}\varphi-\textup{c}\psi\textup{s}\varphi\\\
-\textup{s}\theta&\textup{c}\theta\textup{s}\varphi&\textup{c}\theta\textup{c}\varphi\end{bmatrix}\begin{bmatrix}u\\\
v\\\ w\end{bmatrix}^{B}$ (6)

Stability and control derivatives for the Chaka-50 are reported in Zavaree et
al. (2021) based on Computational Fluid Dynamics (CFD). The summary of these
derivatives for two flight phases is given in table (1). before six-degree-of-
freedom (6DoF) simulation using equations (1-2), the trim conditions in a
wings-level flight are calculated for simulation verification based on trim
equations in Roskam (1998). In drag equation, absolute value of $\delta_{E}$,
$i_{H_{1}}$, $\alpha_{1}$ is considered. Also, flight path angle $\gamma_{1}$,
motor installation angle $\phi_{T}$, and horizontal tail incidence angle
$i_{H}$, are zero. The elevator deflection $\delta_{E}$, and required thrust
$T_{1}$ for a trim flight is obtained and shown in table (2). The values in
the table (2) are important for 6DoF simulation validation.

####  2.0.1 Atmospheric Disturbance and Sensor Measurement Noise

This research has utilized the Dryden atmosphere turbulence for its simple
mathematical modeling.

$G_{w}(s)=\sigma_{w}\sqrt{\frac{L_{w}}{\pi
u_{1}}}\Bigg{[}\frac{1+\sqrt{3}\frac{L_{w}}{u_{1}}s}{1+(\frac{L_{w}}{u_{1}}s)^{2}}\Bigg{]}$
(7)

This model is applied in $w$ direction where $L_{w}=h=100m$, $\sigma_{w}=10$,
and $u_{1}=160\frac{m}{s}$. In addition, the sensor noise is defined as $10\%$
of sensor measurement.

Table 1: Stability, Control derivatives (1/rad) | Longitudinal
---
Derivatives
Take-off | Cruise
$c_{D_{0}}$ | 0.0378 | 0.0338
$c_{L_{0}}$ | 0.3203 | 0.3180
$c_{m_{0}}$ | -0.07 | -0.06
$c_{D_{\alpha}}$ | 0.95 | 0.8930
$c_{L_{\alpha}}$ | 11.06 | 14.88
$c_{m_{\alpha}}$ | -12.18 | -11.84
$c_{D_{u}}$ | 0.040 | 0.041
$c_{L_{u}}$ | 0 | 0.081
$c_{m_{u}}$ | 0 | -0.039
$c_{D_{q}}$ | 0 | 0
$c_{L_{q}}$ | 11.31 | 12.53
$c_{m_{q}}$ | -40.25 | -40.69
$c_{D_{\delta_{E}}}$ | 0.1550 | 0.1570
$c_{L_{\delta_{E}}}$ | 0.96 | 0.78
$c_{m_{\delta_{E}}}$ | -6.15 | -5.98
Table 2: Trim Conditions of Chaka MRJ | Required Thrust
---
($T_{1}$)(lbs)
AoA ($\alpha^{\circ}$) |  | Required Elevator
---
($\delta_{E}^{\circ}$)
21433.02 | 0.39 | -2.28

Also, the geometric, mass, and moment of inertia data are given in the table
(3).

Table 3: Simulation Parameters Parameter | Value | Parameter | Value
---|---|---|---
Wing Area($m^{2}$) | 43.42 |  $I_{xx}$($kg.m^{2}$) | 378056.535
| Mean Aerodynamic
---
Chord($m$)
1.216 |  $I_{yy}$($kg.m^{2}$) | 4914073.496
Span($m$) | 28 |  $I_{zz}$($kg.m^{2}$) | 5670084.803
Mass($kg$) | 18418.27 |  $I_{xz}$($kg.m^{2}$) | 0

##  3 Attitude Control Process Using Q-learning

Q-learning is an off-policy, model-free control strategy algorithm that by
interacting with an environment, seeks to find the best action to take given
the current state. It is a branch of Temporal Difference (TD) which is a
combination of Dynamic Programming, and Monte Carlo theories. Truss-braced
wing aircraft usually suffer inadequate stability owing to their narrow mean
aerodynamic chord (MAC). For example, the Phugoid and Short Period poles of
Boeing N+3 TTBW (Nguyen and Xiong, 2022), and Chaka 50 (Zavaree et al., 2021)
prove that intuitively in comparison with corresponding poles in Çetin (2018)
for the Cessna 172. A summary of numerical results has been gathered in table
(4).

Table 4: Longitudinal Dynamics Characteristic Aircraft Roots | Short Period Roots | Phugoid Roots
---|---|---
Chaka 50 | $-0.8\pm 0.61i$ | $-0.0064\pm 0.05i$
Cessna 172 | $-3.23\pm 5.71i$ | $-0.025\pm 0.19i$
Boeing N+3 | $-0.35\pm 0.35i$ | $-0.0082\pm 0.07i$

###  3.1 MDP and POMDP Definition in Attitude Control

It is necessary to formalize sequential decision making like aircraft attitude
control as MDPs where one action influence not just next state and its
immediate reward, but also upcoming states and their future rewards (Sutton
and Barto, 2018). For clarification, At each time-step $t$, the controller
normally receives the state’s information including
$\theta_{t}\in\mathcal{S}_{1}$, and $\dot{\theta_{t}}\in\mathcal{S}_{2}$ from
the environment. Based on that, the controller selects an action which in this
model is the elevator deflection, $\delta_{E_{t}}\in\mathcal{A}(s)$. The
simulation executes and in next time-step $t+1$, the controller receives a
numerical reward, $R_{t+1}\in\mathcal{R}$ to evaluate its performance and find
itself in next state, $\theta_{t+1}$, $\dot{\theta}_{t+1}$.

$\theta_{0},\ \dot{\theta}_{0},\ \delta_{E_{0}},\ R_{1},\ \theta_{1},\
\dot{\theta}_{1},\ ...$ (8)

In this problem, a random selection of $R_{t}$, $\theta_{t}$, and
$\dot{\theta}_{t}$ have a clear discrete probability distribution dependent on
previous state and action only. Also, the problem has Markov property by
considering $\theta_{t}$, and $\dot{\theta}_{t}$ as states. The purpose of
reinforcement learning finite MDP is to find a policy that gathers maximum
reward over time. Consequently, to find an optimal policy for taking
$\delta_{E}$ in state $\theta$, $\dot{\theta}$, the state-action value
function $Q_{\pi}(\theta,\dot{\theta},a)$ defines to be maximize by expected
return that is sum of discounted instant rewards by starting from one specific
state following policy $\pi$ to terminal state $\theta_{T}$,
$\dot{\theta}_{T}$.

$\displaystyle
Q_{\pi}(\theta,\,\dot{\theta},\,\delta_{E})=\mathbb{E}_{\pi}\bigg{[}\sum_{k=0}^{\infty}\gamma^{k}$
$\displaystyle R_{t+k+1}\ \bigg{|}\ $ (9)
$\displaystyle\theta_{t}=\theta,\,\dot{\theta}_{t}=\dot{\theta},\,\delta_{E_{t}}=\delta_{E}\bigg{]},$

Where $\gamma$ is the discount factor for rewards to be weighted based on its
time-step, and usually is $0<\gamma<1$.

###  3.2 Structure of Q-learning Controller

In this work, the optimal policy of elevator action selection in each state is
approximated directly using an early breakthrough in reinforcement learning
namely Q-learning (Watkins and Dayan, 1992). Because of the non-linearity and
poor stability characteristics of Truss-braced wing aircraft, Q-learning
implementation without utilizing NNs can be challenging.

####  3.2.1 Reward Function and Action Space Definition

Defining an efficient reward function plays a main role in algorithm
convergence. Therefore, this research has concentrated carefully in reward
function design and hyper parameter tuning. In this way, the reward function
contains different components such as $\theta,\,q,\,\delta_{E}$. The reward
function would be computed in three consecutive steps. First, to restrict the
operating frequency of the elevator, it is essential to give a large
punishment in the case of aggressive elevator selection:

$Reward_{t}=-10000,\quad\textrm{If}\
\left(|\delta_{E_{t}}|-|\delta_{E_{t-1}}|\right)>0.1\ \textrm{rad}.$ (10)

Subsequently, if the change rate of the elevator is satisfactory, the reward
function will be computed as follows if the aircraft is in the vicinity of the
desired state.

$\displaystyle Reward_{t}=$ $\displaystyle(300,\qquad\textrm{If}\
|err_{p_{t}}|<0.05^{\circ})$ (11) $\displaystyle+$
$\displaystyle(300,\qquad\textrm{If}\ |err_{p_{t}}|<0.02^{\circ})\ $
$\displaystyle+$ $\displaystyle(400,\qquad\textrm{If}\
|q_{sim_{t}}|<0.04^{\circ})\ $ $\displaystyle+$
$\displaystyle(600,\qquad\textrm{If}\ |q_{sim_{t}}|<0.02^{\circ})\ $
$\displaystyle+$ $\displaystyle(800,\qquad\textrm{If}\
|q_{sim_{t}}|<0.005^{\circ}),\ $

where $err_{p_{t}}=\theta_{sim_{t}}-\theta_{des_{t}}$. Finally, if none of the
above two conditions are met, we should encourage the air vehicle to move
towards the desired state. This can be done using the following reward
function:

$Reward_{t}=-(100\times|err_{p_{t}}|)^{2}-(40\times|q_{sim_{t}}|)^{2}.$ (12)

Accordingly, the farther the system is from the desired state, the less reward
it receives. Also, a derivative term (the second term) has been incorporated
into the reward function to avoid high pitch rates.

Considering the tabular Q-learning, the elevator commands are obtained
discretely. So, the elevator commands are divided into $-0.25$ to $+0.25$
radians with $0.025$ intervals, corresponding 21 elevator deflections. Also,
$\epsilon$-greedy action selection strategy with epsilon decay is used in this
research.

$\delta_{E_{t}}=\begin{cases}\arg\max\,Q(\theta_{t},\,\dot{\theta}_{t},\,\delta_{E})&\text{with
probability $1-\epsilon$}\\\ \text{random action}&\text{with probability
$\epsilon$}\\\ \end{cases}$ (13)

The mentioned reward function has developed carefully during lots of dynamic
examinations. Against some works, this research believes that the quality of
learning convergence not only is not related to trial and error but also, is
related to deep comprehension of dynamic feedback. For clarification, we first
were omitting to observe $q_{sim}$ feedback in equation (11) that caused worse
results. Also, the proportion of the first term to the second in equation (12)
plays a substantial role in the convergence rate.

###  3.3 Structure of Fuzzy Action Assignment

In order to make continuous elevator actions, a Fuzzy Action Assignment (FAA)
is presented to evaluate the capability of the learned Q-table. In this
method, instead of taking a discrete greedy action in a given state $\theta$,
$\dot{\theta}$, a weighted $\delta_{E}$ is selected based on the proposed
membership function (MF). More precisely, the membership function
corresponding to a cell of the table with $\theta_{i}$ and $\dot{\theta}_{j}$
is defined as follows:

$\displaystyle MF_{i,j}=$ (14)
$\displaystyle\exp\left(-\frac{1}{2}\left(\frac{\theta_{sim_{t}}-\theta_{i}}{\sigma_{\theta}}\right)^{2}\right)\exp\left(-\frac{1}{2}\left(\frac{q_{sim_{t}}-\dot{\theta}_{j}}{\sigma_{\dot{\theta}}}\right)^{2}\right),$

Where $\sigma_{\theta_{i}}=\frac{\theta_{i}-\theta_{i-1}}{2}$, and
$\dot{\sigma}_{\theta{j}}=\frac{\dot{\theta}_{j}-\dot{\theta}_{j-1}}{2}$ are
the half-length of the state span, provided that $\theta_{i}>\theta_{i-1}$,
and $\dot{\theta}_{j}>\dot{\theta}_{j-1}$. Also,
$\theta_{n}=\frac{\theta_{i}+\theta_{i-1}}{2}$, and
$\dot{\theta}_{n}=\frac{\dot{\theta}_{j}+\dot{\theta}_{j-1}}{2}$ are the
center of each state. So the $\delta_{E}$ at each time-step is calculated as:

$\displaystyle\delta_{E_{t}}=\frac{\sum_{i}\sum_{j}MF_{i,j}\arg\max\,Q(\theta_{i},\,\dot{\theta}_{j},\,\delta_{E})}{\sum_{i}\sum_{j}MF_{i,j}}.$
(15)

##  4 Simulation Results and Discussion

The attitude control problem is divided into two parts. First, the environment
includes the aircraft simulation, and second, the Q-learning agent which is a
controller. In general, at each time step, the $\theta_{sim}$ is obtained, and
then the calculated tracking error is used to rectify the action selection
policy. It is continued for several episodes so as to reach an optimal
strategy. The pseudocode of the presented method is as follows:

Algorithm 1 Q-learning Attitude Controller

Input: Learning Rate $\alpha$, Discount Factor $\gamma$, Desired Angle
$\theta_{des}=1$deg.
Output: $Q_{\pi^{*}}(\theta,\dot{\theta},\delta_{E})$
//Initialize
$Q(\theta_{0},\dot{\theta}_{0},\delta_{E_{0}})\leftarrow\emptyset,\ $for all$\
\theta\in\mathcal{S}_{1},\dot{\theta}\in\mathcal{S}_{2},\delta_{E}\in\mathcal{A}(s)$

for  _each episode (5 $sec$)_ do

// Initialize 6 DoF simulation with a random $\theta_{0}\in[0,2]$ deg.
for  _each time-step (0.01 $sec$)_ do

(1) Select an action $\delta_{E}$ based on FAA $\epsilon$-greedy strategy.
(2) Execute 6 DoF simulation using computed $\delta_{E}$, observe $R_{t+1}$,
$\theta_{t+1}$, $\dot{\theta}_{t+1}$.
(3) Update the state-action value function:  $\displaystyle
Q(\theta_{t},\,\dot{\theta_{t}},\,\delta_{E_{t}})=\,Q(\theta_{t},\,\dot{\theta_{t}},\,\delta_{E_{t}})+\,$
$\displaystyle\alpha\bigg{[}R_{t+1}+\gamma\,\underset{\delta_{E}}{max}\,Q(\theta_{t+1},\,\dot{\theta}_{t+1},\,\delta_{E})-Q(\theta_{t},\,\dot{\theta_{t}},\,\delta_{E_{t}})\bigg{]}$
(4) Substitute simulation parameters in time-step $t$ with $t+1$.

end for

end for

return $Q_{\pi^{*}}(\theta,\dot{\theta},\delta_{E})$

The above method is applied to Chaka 50 MRJ in 5 second episodes. The
difference between the proposed control theory and trim flight proves the
performance of the Q-learning controller. Figure (2) illustrates the
simulation results of trim conditions over 1500 seconds. Needless to say, the
$\alpha$ angle is converged exactly to its theoretical value by the table (2).
Also, the $\theta$ angle is converged in accordance with $\alpha$.
Consequently, the simulation is accurate. But low-stability existence
resulting in long-time fluctuations is affected by the damping ratio of the
Phugoid mode.

Figure 2: The $\alpha$ and $\theta$ variation in a trim flight

In this way, the simulations using the controller are performed in two problem
modelings as MDP and POMDP. The difference between them is observing
$\dot{\theta}$ in the MDP model where the POMDP is neglected. Obviously, the
state-action table in the MDP model is 3D as
$\theta\times\dot{\theta}\times\delta_{E}$ dimension whereas in the POMDP
model, this table is shaped as dimension as $\theta\times\delta_{E}$. In
addition to the number of observed states, the bounds, and intervals of states
are important in converging time. In this way, to learn the RL controller
efficiently, it is momentous to divide $\theta$, $\dot{\theta}$ intervals
knowledgeably so as not to omit any main state. As a synopsis, the simulation
parameters and the intervals are in table (5)

Table 5: Q-learning Simulation Parameters Parameter | Value
---|---
Epsilon($\epsilon$) | $[0.1:{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}3e{-6}}:0.04]$
Alpha($\alpha$) | $[0.02:{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}9e{-7}}:0.002]$
Gamma($\gamma$) | $0.99$
Episode number | $20000$
$\theta$(rad) | $[-10,-0.024:{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0.002}:-0.002,-0.001,0]$
$\dot{\theta}$(rad) | $[-10,-0.04,-0.02,-0.005]$

where the $\epsilon$, and $\alpha$ are quantified with their upper bounds in
the first episode, and decay at a linear rate during episodes. Also, the
state-action table intervals including $\theta$, and $\dot{\theta}$ are
mentioned in this table where blue numbers are intervals. These values are
considered symmetrically with positive signs for the positive zone. Assuming
this, all substantial states that the aircraft confronts during the learning
phase are covered. The result of the Q-learning controller is illustrated in
figure (3) in comparison to the PID controller. Obviously, the POMDP accuracy
is worse than others because the environment modeling is lacking from a
complete Markov Property. However, the POMDP performance is significantly
better than the trim condition in figure (2). It is apparent that PID rise-
time is lower than MDP and POMDP but the overshoot of PID continues to exist
until approximately 2 seconds. There is a tiny oscillation in the MDP method
because of discrete elevator deflections, but this flaw is eliminated by FAA
using the same learned Q-table.

Figure 3: Performance of improved MDP Q-learning using FAA, in comparison to
MDP, POMDP, and PID controller.

Figure (4) shows the rewards of each episode for MDP and POMDP modeling. In
early episodes, POMDP results are better and making fewer fluctuations. But
after about 4000 episodes, MDP starts achieving positive rewards. The MDP
converges fairly in episode number 10000 and surpasses the POMDP method. It is
noticeable that the POMDP never achieves positive rewards. Consequently,
encompassing the $\dot{\theta}$ plays a significant role in efficiency.

Figure 4: MDP and POMDP rewards over $20000$ episodes  Figure 5: Elevator
deflections in $\theta_{des}=1^{\circ}$ tracking

Elevator deflections is illustrated in figure (5). Although the control effort
of MDP is more than POMDP numerically, it sequels better $\theta_{does}$
tracking results. However, the FAA solves the consequent oscillations caused
by discrete actions simultaneously reducing control effort even more superior
to PID.

Figure 6: Variable $\theta$ tracking using MDP and FAA-improved MDP a) at
normal condition, and b) in presence of sensor measurement noise and
atmospheric disturbance.

Finally, figure (6), shows the tracking result for a variable $\theta$. The
tracking of a variable pitch angle became possible by defining a virtual
desired pitch angle according to the current tracking error of the system. At
first glance, it is obvious that both MDP and FAA-improved MDP were prosperous
to track variable $\theta$ in normal conditions. But the elevator’s high
working frequency of MDP which is shown in figure (6), proves the FAA
significant superiority. Apart from that, in presence of atmospheric
disturbances and sensor measurement noise that means an actual flight
condition, FAA-improved MDP is able to track $\theta_{des}$ and demonstrate
its robustness. But in figure (6) and its zoom-in, the MDP starts to diverge
when the atmospheric disturbance is applied in addition to noise.

Figure 7: Elevator deflections in variable $\theta$ tracking a) at normal
condition, and b) in presence of sensor measurement noise and atmospheric
disturbance.

##  5 Conclusion

This work proposes a longitudinal attitude control method based on improved
Q-learning under MDP and POMDP problem modeling. The aircraft is a novel
regional Truss-braced wing airplane that is treated as an RL environment. The
action selection strategy is $\epsilon$-greedy, and the reward function
defines to consider $\theta$, $q$, and $\delta_{E}$. This method is verified
in constant and variable $\theta_{des}$ tracking simulations where the
simulation results show a comparable outcome between the FAA-improved MDP
model and the PID controller. Fruitful to conclude that POMDP modeling results
are poor due to imperfect problem definition. Finally, the FAA-augmented
method was used to construct continuous actions to eliminate fluctuations and
make robust variable angle tracking in presence of atmospheric disturbance and
sensor measurement noise.

## References

  * Bøhn et al. (2021) Bøhn, E., Coates, E.M., Reinhardt, D., and Johansen, T.A. (2021).  Data-efficient deep reinforcement learning for attitude control of fixed-wing uavs: Field experiments.  _arXiv preprint arXiv:2111.04153_.
  * Dammen (2022) Dammen, E.B. (2022).  _Reinforcement Learning and Evolutionary Algorithms for Attitude Control, A comparison for aerial vehicles_.  Master’s thesis.
  * Emami et al. (2022) Emami, S.A., Castaldi, P., and Banazadeh, A. (2022).  Neural network-based flight control systems: Present and future.  _Annual Reviews in Control_ , 53(4), 97–137.  10.1016/j.arcontrol.2022.04.006.
  * Er and Deng (2004) Er, M.J. and Deng, C. (2004).  Online tuning of fuzzy inference systems using dynamic fuzzy q-learning.  _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , 34(3), 1478–1489.  10.1109/TSMCB.2004.825938.
  * Glorennec and Jouffe (1997) Glorennec, P. and Jouffe, L. (1997).  Fuzzy q-learning.  In _Proceedings of 6th International Fuzzy Systems Conference_ , volume 2, 659–662 vol.2.  10.1109/FUZZY.1997.622790.
  * Hu et al. (2022) Hu, W., Gao, Z., Quan, J., Ma, X., Xiong, J., and Zhang, W. (2022).  Fixed-wing stalled maneuver control technology based on deep reinforcement learning.  In _2022 IEEE 5th International Conference on Big Data and Artificial Intelligence (BDAI)_ , 19–25. IEEE.
  * Li et al. (2022) Li, L., Bai, J., and Qu, F. (2022).  Multipoint aerodynamic shape optimization of a truss-braced-wing aircraft.  _Journal of Aircraft_ , 1–16.
  * Nguyen and Xiong (2022) Nguyen, N.T. and Xiong, J. (2022).  Dynamic aeroelastic flight dynamic modeling of mach 0.745 transonic truss-braced wing.  In _AIAA SCITECH 2022 Forum_ , 1325.
  * Richter et al. (2022) Richter, D.J., Natonski, L., Shen, X., and Calix, R.A. (2022).  Attitude control for fixed-wing aircraft using q-learning.  In J.H. Kim, M. Singh, J. Khan, U.S. Tiwary, M. Sur, and D. Singh (eds.), _Intelligent Human Computer Interaction_ , 647–658. Springer International Publishing, Cham.
  * Roskam (1998) Roskam, J. (1998).  _Airplane flight dynamics and automatic flight controls_.  DARcorporation.
  * Sarode (2022) Sarode, V.S. (2022).  _Investigating Aerodynamic Coefficients and Stability Derivatives for Truss-Braced Wing Aircraft Using OpenVSP_.  Ph.D. thesis, Virginia Tech.
  * Sutton and Barto (2018) Sutton, R.S. and Barto, A.G. (2018).  _Reinforcement learning: An introduction_.  MIT press.
  * Tang and Lai (2020) Tang, C. and Lai, Y.C. (2020).  Deep reinforcement learning automatic landing control of fixed-wing aircraft using deep deterministic policy gradient.  In _2020 International Conference on Unmanned Aircraft Systems (ICUAS)_ , 1–9.  10.1109/ICUAS48674.2020.9213987.
  * Wang et al. (2018) Wang, Z., Li, H., Wu, H., Shen, F., and Lu, R. (2018).  Design of agent training environment for aircraft landing guidance based on deep reinforcement learning.  In _2018 11th International Symposium on Computational Intelligence and Design (ISCID)_ , volume 02, 76–79.  10.1109/ISCID.2018.10118.
  * Watkins and Dayan (1992) Watkins, C.J. and Dayan, P. (1992).  Q-learning.  _Machine learning_ , 8(3), 279–292.
  * Xi et al. (2022) Xi, Z., Wu, D., Ni, W., and Ma, X. (2022).  Energy-optimized trajectory planning for solar-powered aircraft in a wind field using reinforcement learning.  _IEEE Access_ , 10, 87715–87732.
  * Yang et al. (2020) Yang, X., Yang, X., and Deng, X. (2020).  Horizontal trajectory control of stratospheric airships in wind field using q-learning algorithm.  _Aerospace Science and Technology_ , 106, 106100.  https://doi.org/10.1016/j.ast.2020.106100.
  * Yuan et al. (2019) Yuan, X., Sun, Y., Wang, Y., and Sun, C. (2019).  Deterministic policy gradient with advantage function for fixed wing uav automatic landing.  In _2019 Chinese Control Conference (CCC)_ , 8305–8310.  10.23919/ChiCC.2019.8866189.
  * Zavaree et al. (2021) Zavaree, S., Zahmatkesh, M., Eghbali, K., Zahiremami, K., Vaezi, E., Madani, s.A., Kariman, A., Heidari, Z., Mahmoudi, A., Rassouli, F., Varshavi, M., and Sadraey, M.H. (2021).  Modern regional jet family (chaka: A high-performance, cost-efficient, semi-conventional regional jet family).  10.13140/RG.2.2.20347.64802.
  * Zipfel (2014) Zipfel, P. (2014).  Modeling and simulation of aerospace vehicle dynamics–third edition.
  * Çetin (2018) Çetin, E. (2018).  _System identification and control of a fixed wing aircraft by using flight data obtained from x-plane flight simulator_.  Master’s thesis, Middle East Technical University.