# Observation of non-Abelian exchange statistics on a superconducting
processor

Google Quantum AI and Collaborators

###### Abstract

Indistinguishability of particles is a fundamental principle of quantum
mechanics1. For all elementary and quasiparticles observed to date - including
fermions, bosons, and Abelian anyons - this principle guarantees that the
braiding of identical particles leaves the system unchanged2, 3. However, in
two spatial dimensions, an intriguing possibility exists: braiding of non-
Abelian anyons causes rotations in a space of topologically degenerate
wavefunctions4, 5, 6, 7, 8. Hence, it can change the observables of the system
without violating the principle of indistinguishability. Despite the well
developed mathematical description of non-Abelian anyons and numerous
theoretical proposals9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
their experimental observation has remained elusive for decades. Using a
superconducting quantum processor, we prepare the ground state of the surface
code and manipulate it via unitary operations to form wavefunctions that are
described by non-Abelian anyons. By implementing a unitary protocol23 to move
the anyons, we experimentally verify the fusion rules of non-Abelian Ising
anyons and braid them to realize their statistics. Building on our technique,
we study the prospect of employing the anyons for quantum computation and
utilize braiding to create an entangled state of anyons encoding three logical
qubits. Our work represents a key step towards topological quantum computing.

Elementary particles in three dimensions (3D) are either bosons or fermions.
The existence of only two types is rooted in the fact that the worldlines of
two particles in 3+1 dimensions can always be untied in a trivial manner.
Hence, exchanging a pair of indistinguishable particles twice is topologically
equivalent to not exchanging them at all, and the wavefunction must remain the
same. Representing the exchange as a matrix $R$ acting on the space of
wavefunctions with a constant number of particles, it is thus required that
$R^{2}=1$, leaving two possibilities: $R=1$ (bosons) and $R=-1$ (fermions).
Such continuous deformation is not possible in two dimensions (2D), thus
allowing collective excitations (quasiparticles) to exhibit richer braiding
behavior. In particular, this permits the existence of Abelian anyons2, 3, 24,
6, 7, 8, 25, where the global phase change due to braiding can take any value.
It has been proposed that there exists another class of quasiparticles known
as non-Abelian anyons, where braiding instead results in a change of the
observables of the wavefunction4, 24, 5. In other words, $R^{2}$ does not
simplify to a scalar, but remains a unitary matrix. Therefore, $R^{2}$ is a
fundamental characteristic of anyon braiding. The topological approach to
quantum computation26 aims to leverage these non-Abelian anyons and their
topological nature to enable gate operations that are protected against local
perturbations and decoherence errors27, 5, 28, 29, 30. In solid-state systems,
primary candidates of non-Abelian quasiparticles are low-energy excitations in
the $\nicefrac{{5}}{{2}}$ fractional quantum Hall states31, 32, vortices in
topological superconductors33, 34, and Majorana zero modes in semiconductors
proximitized by superconductors35, 36, 37, 38. However, direct verification of
non-Abelian exchange statistics has remained elusive39, 40, 41.

We formulate the necessary requirements for experimentally certifying a
physical system as a platform for topological quantum computation5, 26: (1)
create an anyon pair; (2) verify the rules that govern the ”collision” of two
anyons, known as the fusion rules; (3) verify the non-Abelian braiding
statistics reflected in the matrix structure $R^{2}$; (4) realize controlled
entanglement of anyonic degrees of freedom. Notably, the observation of steps
(2-4) requires measurements of multi-anyon states, via fusion or non-local
measurements.

The advent of quantum processors allows for controlled unitary evolution and
direct access to the wavefunction rather than the parameters of the
Hamiltonian. These features enable the use of local operations for efficient
preparation of topological states that can host non-Abelian anyons, and - as
we will demonstrate - their subsequent braiding and fusion. Moreover, these
platforms allow for multi-qubit (i.e. non-local) measurements.

Figure 1: Deformations of the surface code. a, Stabilizer codes are
conveniently described in a graph framework. Through deformations of the
surface code graph, a square grid of qubits (crosses) can be used to realize
more generalized graphs. Plaquette violations (red) correspond to stabilizers
with $s_{p}=-1$ and are created by local Pauli operations. In the absence of
deformations, plaquette violations are constrained to move on one of the two
sub-lattices of the dual graph in the surface code, hence the two shades of
blue. b, A pair of D3Vs (yellow triangles) appears by removing an edge between
two neighboring stabilizers, $\hat{S}_{1}$ and $\hat{S}_{2}$, and introducing
the new stabilizer, $\hat{S}=\hat{S}_{1}\hat{S}_{2}$. A D3V is moved by
applying a 2-qubit entangling gate,
$\exp({\frac{\pi}{8}[\hat{S}^{\prime},\hat{S}]})$. In the presence of bulk
D3Vs, there is no consistent way of checkerboard coloring, hence the
(arbitrarily chosen) gray regions. Top right: in a general stabilizer graph,
$\hat{S}_{p}$ can be found from a constraint at each vertex, where
$\\{\tau_{1},\tau_{2}\\}=0$.

In order to realize a many-body quantum state that can host anyons, it is
essential to control the topological degeneracy. A suitable platform for
achieving this requirement is a stabilizer code42, where the wavefunctions are
characterized by a set of commuting operators $\\{\hat{S}_{p}\\}$ called
_stabilizers_ , with $\hat{S}_{p}\ket{\psi}=s_{p}\ket{\psi}$ and $s_{p}=\pm
1$. The code space is the set of degenerate wavefunctions for which $s_{p}=1$
for all $p$. Hence, every independent stabilizer divides the degeneracy of the
code space by two.

While the physical layout of qubits is typically used to determine the
structure of the stabilizers, the qubits can be considered to be degree-$j$
vertices (D$j$V; $j\in\\{2,3,4\\}$) on more general planar graphs (see Fig.
1a)23. Using this picture, each stabilizer can be associated with a plaquette
$p$, whose vertices are the qubits on which $\hat{S}_{p}$ acts:

$\hat{S}_{p}=\prod_{v\,\in\,\mathrm{vertices}}\hat{\tau}_{p,v}\,.$ (1)

$\hat{\tau}_{p,v}$ is here a single-qubit Pauli operator acting on vertex $v$,
chosen to satisfy a constraint around that vertex (Fig. 1b). An instance where
$s_{p}=-1$ on a plaquette is called a plaquette violation. These can be
thought of as quasi-particles, which are created and moved through single-
qubit Pauli operators (Fig. 1a). A pair of plaquette violations sharing an
edge constitute a fermion, $\varepsilon$. We recently demonstrated the Abelian
statistics of such quasiparticles in the surface code43. To realize non-
Abelian statistics, one needs to go beyond such plaquette violations and
instead deform the stabilizer graph, analogous to lattice defects in
crystalline solids.

By introducing the graph framework, it was predicted by Lensky et al. that the
vertices of degree 2 and 3 host non-Abelian anyons23. Consider the stabilizer
graph of the surface code26, 44, specifically with boundary conditions such
that the degeneracy is two. While all the vertices in the bulk are D4Vs, one
can create two D3Vs by removing an edge between two neighboring plaquettes $p$
and $q$, and introducing the new stabilizer $\hat{S}=\hat{S}_{p}\hat{S}_{q}$
(Fig. 1b). Evidently, the introduction of two D3Vs reduces the number of
independent stabilizers by one and thus doubles the degeneracy. This doubling
is exactly what is expected when a pair of Ising anyons is introduced; hence,
D3Vs appear as a candidate of non-Abelian anyons, and we will denote them as
$\sigma$.

In order to be braided and fused by unitary operations, the D3Vs must be
moved. While the structure of the stabilizer graph is usually considered to be
static, it was shown in Ref.23 that the D3Vs can in fact be moved - thus
deforming the stabilizer graph - using local two-qubit Clifford gates. To
shift a D3V from vertex $u$ to $v$, an edge must be disconnected from $v$ and
reconnected to $u$. This can be achieved via the gate unitary
$\exp\left(\frac{\pi}{8}[\hat{S}^{\prime}_{p},\hat{S}_{p}]\right)$, where
$\hat{S}_{p}$ is the original stabilizer containing the edge and $u$, and
$\hat{S}^{\prime}_{p}$ is the new stabilizer that emerges after moving the
edge23. In cases where the D3V is shifted between two connected vertices, the
unitary simplifies to the form
$U_{\pm}(\hat{\tau}_{u}\hat{\tau}_{v})\equiv\exp\left(\pm
i\frac{\pi}{4}\hat{\tau}_{u}\hat{\tau}_{v}\right)$, where $\hat{\tau}_{u}$ and
$\hat{\tau}_{v}$ are Pauli operators acting on vertices $u$ and $v$. We
experimentally realize this unitary through a CZ-gate and single-qubit
rotations (median errors of $7.3\times 10^{-3}$ and $1.3\times 10^{-3}$,
respectively; see Supplementary materials).

Figure 2: Demonstration of the fundamental fusion rules of D3Vs. a, The
braiding worldlines used to fuse $\varepsilon$ and $\sigma$. b, Expectation
values of stabilizers at each step of the unitary operation after readout
correction (see Fig. S3 for details and individual stabilizer values). We
first prepare the ground state of the surface code (step I; average stabilizer
value: $0.94\pm 0.04$). A D3V ($\sigma$) pair is then created (II) and
separated (III-IV), before creating a fermion, $\varepsilon$ (V). One of the
plaquette violations is brought around the right $\sigma$ (VI-VIII), allowing
it to annihilate with the other plaquette violation (VIII). The fermion has
seemingly disappeared, but re-emerges when the $\sigma$ are annihilated (XI;
stabilizer values: -0.86 and -0.87). The path V$\rightarrow$VIII demonstrates
the fusion rule, $\sigma\times\varepsilon=\sigma$. The different fermion
parities at the end of the paths VIII$\rightarrow$XI and IV$\rightarrow$I show
the other fusion rule, $\sigma\times\sigma=\mathbb{1}+\varepsilon$. Yellow
triangles represent the positions of the $\sigma$. The brown and red lines
denote the paths of the $\sigma$ and the plaquette violation, respectively.
Red squares (diamonds) represent $X$\- ($Z$-) gates. Upper left: table of two-
qubit unitaries used in the protocol. c, Non-local technique for hidden
fermion detection: the presence of a fermion in a $\sigma$-pair can be deduced
by measuring the sign of the Pauli string $\hat{\mathcal{P}}$ corresponding to
bringing a plaquette violation around the $\sigma$-pair (gray path).
$\hat{\mathcal{P}}$ is equivalent to the shorter string
$\hat{\mathcal{P}}^{\prime}$ (black path). Measurements of
$\hat{\mathcal{P}}^{\prime}$ in steps VIII (top) and IV (bottom) give values
$-0.85\pm 0.01$ and $+0.84\pm 0.01$, respectively. This indicates that there
is a hidden fermion pair in the former case, but not in the latter, despite
the stabilizers being the same.

In the first experiment we demonstrate the creation of anyons and the
fundamental fusion rules of $\sigma$ and $\varepsilon$ (Fig. 2a). In a
5$\times$5 grid of superconducting qubits, we first use a protocol consisting
of four layers of CZ-gates to prepare the surface code ground state (panel I
in Fig. 2b, see also 43). The average stabilizer value after the ground state
preparation is $0.94\pm 0.04$ (individual stabilizer values shown in Fig.
S3c). We then remove a stabilizer edge to create a pair of D3Vs ($\sigma$) and
separate them through the application of two-qubit gates. Panels I-IV in Fig.
2b show the measured stabilizer values in the resultant graph in each step of
this procedure (note that the measurements are destructive and the protocol is
restarted after each measurement). In panel V, single-qubit $Z$-gates are
applied to two qubits near the lower left corner of the grid to create
adjacent plaquette violations, which together form a fermion. Through the
sequential application of $X$\- and $Z$-gates (VI to VIII), one of the
plaquette violations is then made to encircle the right $\sigma$ vertex.
Crucially, after moving around $\sigma$, the plaquette violation does not
return to where it started, but rather to the location of the other plaquette
violation. This enables them to annihilate (VIII), causing the fermion to
seemingly disappear. However, by bringing the two $\sigma$ back together and
annihilating them (IX through XI), we arrive at a striking observation: an
$\varepsilon$-particle re-emerges on two of the square plaquettes where the
$\sigma$-vertices previously resided.

Our results demonstrate the fusion of $\varepsilon$ and $\sigma$. The
disappearance of the fermion from step V to VIII establishes the fundamental
fusion rule of $\varepsilon$ and $\sigma$:

$\sigma\times\varepsilon=\sigma\,.$ (2)

We emphasize that none of the single-qubit gates along the path of the
plaquette violation are applied to the qubits hosting the mobile $\sigma$; our
observations are therefore solely due to the non-local effects of non-Abelian
D3Vs, and exemplify the unconventional behavior of the latter. Moreover,
another fusion rule is seen by considering the reverse path IV$\rightarrow$I,
and comparing it to the path VIII$\rightarrow$XI. These two paths demonstrate
that a pair of $\sigma$ can fuse to form either vacuum ($\mathbb{1}$) or one
fermion (step I and XI, respectively):

$\sigma\times\sigma=\mathbb{1}+\varepsilon\,.$ (3)

Figure 3: Braiding of non-Abelian anyons. a, Wordline schematic of the
braiding process. b, Experimental demonstration of braiding, displaying the
values of the stabilizers throughout the process. Two $\sigma$-pairs, A and B,
are created from the vacuum $\mathbb{1}$, and one of the $\sigma$ in pair A is
brought to the right side of the grid. Next, a $\sigma$ from pair B is moved
to the top, thus crossing the path of pair A, before bringing $\sigma$-pairs A
and B back together to complete the braid. In the final step, two fermions
appear in the locations where the $\sigma$-pairs resided, constituting a
change in the local observables. The diagonal $\sigma$ move in step IV
requires two SWAP-gates (3 CZ-gates each) and a total of 10 CZ-gates. The
three-qubit unitary in step VIII requires 4 SWAP-gates and a total of 15 CZ-
gates. In the full circuit, a total of 40 layers of CZ-gates are applied (see
Supplementary materials). The yellow triangles represent the locations of the
$\sigma$; the brown and green lines represent the paths of $\sigma$ from pair
A and B, respectively. c, As a control experiment, we perform the same braid
as in a, but with distinguishable $\sigma$ by attaching a plaquette violation
to the $\sigma$ in pair B (represented as purple triangles). d, Same as b, but
using distinguishable $\sigma$ (only showing steps I, IV and XII). In contrast
to b, no fermions are observed in step XII.

Importantly, the starting points of these two paths (IV and VIII) cannot be
distinguished by any local measurement. We therefore introduce a non-local
measurement technique that allows for detecting an $\varepsilon$ without
fusing the $\sigma$26, 9, 23. The key idea underlying this method is that
bringing a plaquette violation around a fermion should result in a
$\pi$-phase. We therefore measure the Pauli string $\hat{\mathcal{P}}$ that
corresponds to creating two plaquette violations, bringing one of them around
the two $\sigma$, and finally annihilating them with each other (gray paths in
Fig. 2c). The existence of an $\varepsilon$ inside the $\sigma$-pair should
cause $\langle\hat{\mathcal{P}}\rangle=-1$. To simplify this technique
further, $\hat{\mathcal{P}}$ can be reduced to a shorter string
$\hat{\mathcal{P}}’$ (black paths in Fig. 2c) by taking advantage of the
stabilizers it encompasses. For instance, if $\hat{\mathcal{P}}$ contains
three of the operators in a 4-qubit stabilizer, these can be switched out with
the remaining operator. Measuring $\langle\hat{\mathcal{P^{\prime}}}\rangle$
in step IV, where the $\sigma$ are separated but the fermion has not yet been
introduced, gives $\langle\hat{\mathcal{P^{\prime}}}\rangle=+0.84\pm 0.01$,
consistent with the absence of fermions. However, performing the exact same
measurement in step VIII, where the $\sigma$ are in the same positions, we
find $\langle\hat{\mathcal{P^{\prime}}}\rangle=-0.85\pm 0.01$, indicating that
an $\varepsilon$ is delocalized across the spatially separated $\sigma$-pair.
This observation highlights the non-local encoding of the fermions, which
cannot be explained with classical physics.

Figure 4: Entangled state of anyon-encoded logical qubits via braiding. a,
Logical operators of the three logical qubits encoded in the 8 anyons (other
basis choices are possible). The colored curves in the left column denote
plaquette violation paths, before reduction to shorter, equivalent Pauli
strings measured in the experiment (right column). b, Worldline schematic of
the single exchange used to realize an entangled state of the logical qubits.
c, Single exchange of the non-Abelian anyons, displaying measurements of the
stabilizers throughout the protocol. Yellow triangles represent the locations
of the $\sigma$, while brown and green lines denote their paths. The average
absolute stabilizer values are $0.95\pm 0.04$ and $0.88\pm 0.05$ in the first
and last step, respectively. d,e, Real (d) and imaginary (e) parts of the
reconstructed density matrix from the quantum state tomography.
$\mathrm{Re}(\rho)$ has clear peaks in its corners, as expected for a GHZ
state on the form $(\ket{000}+\ket{111})/\sqrt{2}$. The overlap with the ideal
GHZ-state is $\mathrm{Tr}\\{\rho_{\mathrm{GHZ}}\rho\\}=0.623\pm 0.004$.

Having demonstrated the above fusion rules involving $\sigma$, we next braid
them with each other to directly show their non-Abelian statistics. We
consider two spatially separated $\sigma$-pairs, A and B, by removing two
stabilizer edges (Fig. 3a and panel II in Fig. 3b). Next, we apply two-qubit
gates along a horizontal path to separate the $\sigma$ in pair A (panel III),
followed by a similar procedure in the vertical direction on pair B (IV), so
that one of its $\sigma$ crosses the path of pair A. We then subsequently
bring the $\sigma$ from pairs A and B back to their original positions (V-VIII
and IX-XI, respectively). Strikingly, when the two $\sigma$-pairs are
annihilated in the final step (XII), we observe that a fermion is revealed in
each of the positions where the $\sigma$-pairs resided (average stabilizer
value: $-0.45\pm 0.06$). This shows a clear change in local observables from
the initial state where no fermions were present. As a control experiment, we
repeat the experiment with distinguishable $\sigma$-pairs, achieved by
attaching a plaquette violation to each of the $\sigma$ in pair B (Fig. 3c,d;
see also Fig. S6 for stabilizer measurements through the full protocol).
Moving the plaquette violation along with the $\sigma$ requires a string of
single-qubit gates, which switches the direction of the rotation in the multi-
qubit unitaries, $U_{\pm}\rightarrow U_{\mp}$. In this case, no fermions are
observed at the end of the protocol (average stabilizer value: $+0.46\pm
0.04$), thus providing a successful control23.

Importantly, fermions can only be created in pairs in the bulk. Moreover, the
fusion of two $\sigma$ can only create zero or one fermion (Eqn. 3). Hence,
our experiment involves the minimal number of bulk $\sigma$ (four) needed to
encode two fermions and demonstrate non-Abelian braiding. Since the fermion
parity is conserved, effects of gate imperfections and decoherence can be
partially mitigated by post-selecting for an even number of fermions. This
results in fermion detection values of $-0.76\pm 0.03$ and $+0.79\pm 0.04$ in
Fig. 3b and d, respectively.

Together, our observations show the change in local observables by braiding of
indistinguishable $\sigma$ and constitute a direct demonstration of their non-
Abelian statistics. In other words, the double-braiding operation $R^{2}$ is a
matrix that cannot be reduced to a scalar. Specifically, it corresponds to an
$X$-gate acting on the space spanned by zero- and two-fermion wavefunctions.

The full braiding circuit consists of 40 layers of CZ-gates and 41 layers of
single-qubit gates (36 of each after ground state preparation). The effects of
imperfections in this hardware implementation can be assessed through
comparison with the control experiment. The absolute values of the stabilizers
where the fermions are detected in the two experiments (dashed boxes in step
XII of Fig. 3b,d) are very similar (average values of $-0.45$ and $+0.46$).
This is consistent with the depolarization channel model, where the measured
stabilizer values are proportional to the ideal values ($\pm 1$).

We next study the prospects of utilizing D3Vs to encode logical qubits and
prepare an entangled state of anyon pairs. By doubling the degeneracy, each
additional $\sigma$-pair introduces one logical qubit, where the
$\ket{0}_{\mathrm{L}}$ ($\ket{1}_{\mathrm{L}}$) state corresponds to an even
(odd) number of hidden fermions. Importantly, the measurements of the fermion
numbers in several $\sigma$-pairs are not fully independent: bringing a
plaquette violation around one $\sigma$-pair is equivalent to bringing it
around all the other pairs (due to the conservation of fermionic parity).
Hence, $N\geq 2$ anyons encode $N/2-1$ logical qubits. Interestingly, the D3Vs
we have created and manipulated so far are not the only ones present in the
stabilizer graph; with the boundary conditions used here, each of the four
corners are also D3Vs, no different from those in the bulk23. Indeed, the
existence of D3Vs in the corners is the reason why a single fermion could be
created in the corner in step V of Fig. 2b. This is also consistent with the
fact that the surface code itself encodes one logical qubit in the absence of
additional D3Vs. Here we create two pairs of D3Vs, in addition to the four
that are already present in the corners, to encode a total of three logical
qubits.

Through the use of braiding, we aim to prepare an entangled state of these
logical qubits, specifically a GHZ-state on the form,
$(\ket{000}+\ket{111})/\sqrt{2}$. The definition of a GHZ-state and the
specifics of how it is prepared is basis-dependent. In most systems the
degrees of freedom are local and there is a natural choice of basis. For
spatially separated anyons, the measurement operators are necessarily non-
local. Here we choose the basis defined as follows: for the first two logical
qubits, we choose the logical $\hat{Z}_{\mathrm{L},i}$ operators to be Pauli
strings encircling each of the bulk $\sigma$-pairs, as was used in Fig. 2c
(green and turquoise paths in the left column of Fig. 4a). For the logical
surface code qubit, we define $\hat{Z}_{\mathrm{L},3}$ as the Pauli string
that crosses the grid horizontally through the gap between the bulk D3V pairs,
effectively enclosing four $\sigma$ (red path in Fig. 4a). In this basis, the
initial state is a product state.

While a double braid was used to implement the operator $X$ in Fig. 3, we now
perform a single braid (Fig. 4b) to realize $\sqrt{X}$ and create a GHZ-state.
We implement this protocol by bringing one $\sigma$ from each bulk pair across
the grid to the other side (Fig. 4c). For every anyon double exchange across a
Pauli string, the value of the Pauli string changes sign. Hence, a double
exchange would change $\ket{000}$ to $\ket{111}$, while a single exchange is
expected to realize the superposition, $(\ket{111}+\ket{000})/\sqrt{2}$.

In order to study the effect of this operation, we perform quantum state
tomography on the final state, which requires measurements of not only
$\hat{Z}_{\mathrm{L},i}$, but also $\hat{X}_{\mathrm{L},i}$ and
$\hat{Y}_{\mathrm{L},i}$ on the three logical qubits. For the first two
logical qubits, $\hat{X}_{\mathrm{L},i}$ is the Pauli string that corresponds
to bringing a plaquette violation around only one of the $\sigma$ in the pair
(as demonstrated in Fig. 2b). Both the logical $\hat{X}_{\mathrm{L},i}$ and
$\hat{Z}_{\mathrm{L},i}$ operators can be simplified by reducing the original
Pauli strings (green and turquoise paths in the left column of Fig. 4c) to
equivalent, shorter ones (right column). $\hat{Z}_{\mathrm{L},1}$ can in fact
be reduced to a single $\hat{Y}$-operator. For the logical surface code qubit,
we define $\hat{X}_{\mathrm{L},3}$ as the Pauli string that crosses the grid
vertically between the bulk D3V pairs (red path in Fig. 4a). Finally, the
logical $\hat{Y}_{\mathrm{L},i}$-operators are simply found from
$\hat{Y}_{\mathrm{L},i}=i\hat{X}_{\mathrm{L},i}\hat{Z}_{\mathrm{L},i}$.
Measuring these operators, we reconstruct the density matrix of the final
state (Fig. 4d,e), which has a purity of
$\sqrt{\mathrm{Tr}\\{\rho^{2}\\}}=0.646\pm 0.003$ and an overlap with the
ideal GHZ-state of $\mathrm{Tr}\\{\rho_{\mathrm{GHZ}}\rho\\}=0.623\pm 0.004$
(uncertainties estimated from bootstrapping). The fact that the state fidelity
is similar to the purity suggests that the infidelity is well described by a
depolarizing error channel.

In conclusion, we have realized highly controllable braiding of degree-3
vertices, enabling the demonstration of the fusion and braiding rules of non-
Abelian Ising anyons. We have also shown that braiding can be used to create
an entangled state of three logical qubits encoded in these anyons. With the
potential inclusion of error correction, which involves overheads including
readout of 5-qubit stabilizers, our observations highlight a new path for
fault-tolerant quantum computing.

### Acknowledgments:

YL and EAK acknowledges support by a New Frontier Grant from Cornell
University’s College of Arts and Sciences. EAK acknowledges support by the NSF
under OAC-2118310, the Ewha Frontier 10-10 Research Grant, the Simons
Fellowship in Theoretical Physics award 920665, and a New Frontier Grant from
Cornell University’s College of Arts and Sciences. EAK performed a part of
this work at the Aspen Center for Physics, which is supported by the National
Science Foundation grant PHY-160761.

† Google Quantum AI and Collaborators

T. I. Andersen1, Y. D. Lensky2, K. Kechedzhi1, I. K. Drozdov1, 3, A.
Bengtsson1, S. Hong1, A. Morvan1, X. Mi1, A. Opremcak1, R. Acharya1, R.
Allen1, M. Ansmann1, F. Arute1, K. Arya1, A. Asfaw1, J. Atalaya1, R. Babbush1,
D. Bacon1, J. C. Bardin1, 4, G. Bortoli1, A. Bourassa1, J. Bovaird1, L.
Brill1, M. Broughton1, B. B. Buckley1, D. A. Buell1, T. Burger1, B. Burkett1,
N. Bushnell1, Z. Chen1, B. Chiaro1, D. Chik1, C. Chou1, J. Cogan1, R.
Collins1, P. Conner1, W. Courtney1, A. L. Crook1, B. Curtin1, D. M. Debroy1,
A. Del Toro Barba1, S. Demura1, A. Dunsworth1, D. Eppens1, C. Erickson1, L.
Faoro1, E. Farhi1, R. Fatemi1, V. S. Ferreira1, L. F. Burgos1 E. Forati1, A.
G. Fowler1, B. Foxen1, W. Giang1, C. Gidney1, D. Gilboa1, M. Giustina1, R.
Gosula1, A. G. Dau1, J. A. Gross1, S. Habegger1, M. C. Hamilton1, 5, M.
Hansen1, M. P. Harrigan1, S. D. Harrington1, P. Heu1, J. Hilton1, M. R.
Hoffmann1, T. Huang1, A. Huff1, W. J. Huggins1, L. B. Ioffe1, S. V. Isakov1,
J. Iveland1, E. Jeffrey1, Z. Jiang1, C. Jones1, P. Juhas1, D. Kafri1, T.
Khattar1, M. Khezri1, M. Kieferová1, 6, S. Kim1, A. Kitaev1, P. V. Klimov1,
A. R. Klots1, A. N. Korotkov1, 7, F. Kostritsa1, J. M. Kreikebaum1, D.
Landhuis1, P. Laptev1, K.-M. Lau1, L. Laws1, J. Lee1, 8, K. W. Lee1, B. J.
Lester1, A. T. Lill1, W. Liu1, A. Locharla1, E. Lucero1, F. D. Malone1, O.
Martin1, J. R. McClean1, T. McCourt1, M. McEwen1, 9, K. C. Miao1, A.
Mieszala1, M. Mohseni1, S. Montazeri1, E. Mount1, R. Movassagh1, W.
Mruczkiewicz1, O. Naaman1, M. Neeley1, C. Neill1, A. Nersisyan1, M. Newman1,
J. H. Ng1, A. Nguyen1, M. Nguyen1, M. Y. Niu1, T. E. O’Brien1, S. Omonije1, A.
Petukhov1, R. Potter1, L. P. Pryadko1,10, C. Quintana1, C. Rocque1, N. C.
Rubin1, N. Saei1, D. Sank1, K. Sankaragomathi1, K. J. Satzinger1, H. F.
Schurkus1, C. Schuster1, M. J. Shearn1, A. Shorter1, N. Shutty1, V. Shvarts1,
J. Skruzny1, W. C. Smith1, R. Somma1, G. Sterling1, D. Strain1, M. Szalay1, A.
Torres1, G. Vidal1, B. Villalonga1, C. V. Heidweiller1, T. White1, B. W. K.
Woo1, C. Xing1, Z. J. Yao1, P. Yeh1, J. Yoo1, G. Young1, A. Zalcman1, Y.
Zhang1, N. Zhu1, N. Zobrist1, H. Neven1, S. Boixo1, A. Megrant1, J. Kelly1, Y.
Chen1, V. Smelyanskiy1, E.-A. Kim${}^{\textrm{\scriptsize 2,\\! 11,\\! 12,\\!
13},\hyperlink{corrb}{\mathsection}}$, I. Aleiner${}^{\textrm{\scriptsize
1},\hyperlink{corrb}{\mathsection}}$, P. Roushan${}^{\textrm{\scriptsize
1},\hyperlink{corrb}{\mathsection}}$,

1 Google Research, Mountain View, CA, USA
2 Department of Physics, Cornell University, Ithaca, NY, USA
3 Department of Physics, University of Connecticut, Storrs, CT, USA
4 Department of Electrical and Computer Engineering, University of
Massachusetts, Amherst, MA, USA
5 Department of Electrical and Computer Engineering, Auburn University,
Auburn, AL, USA
6 QSI, Faculty of Engineering & Information Technology, University of
Technology Sydney, NSW, Australia
7 Department of Electrical and Computer Engineering, University of California,
Riverside, CA, USA
8 Department of Chemistry, Columbia University, New York, NY, USA
9 Department of Physics, University of California, Santa Barbara, CA, USA
10 Department of Physics and Astronomy, University of California, Riverside,
CA, USA
11 Department of Physics, Ewha Womans University, Seoul, South Korea
12 Department of Physics, Harvard University, Cambridge, MA, USA
13 Radcliffe Institute for Advanced Studies, Cambridge, MA, USA

§ Corresponding author<EMAIL_ADDRESS>

§ Corresponding author<EMAIL_ADDRESS>

§ Corresponding author<EMAIL_ADDRESS>
## References

  * Sakurai [1993] J. J. Sakurai,  _Modern Quantum Mechanics_ (Addison-Wesley, 1993).
  * Leinaas and Myrheim [1977] J. Leinaas and J. Myrheim, On the theory of identical particles, Nuovo Cim B 37, 1 (1977).
  * Wilczek [1982] F. Wilczek, Quantum mechanics of fractional-spin particles, Phys. Rev. Lett. 49, 957 (1982).
  * Wilczek [1990] F. Wilczek,  _Fractional statistics and anyon superconductivity_ (World Scientific, 1990).
  * Nayak  _et al._ [2008] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. D. Sarma, Non-Abelian Anyons and Topological Quantum Computation, Rev. Mod. Phys. 80, 1083 (2008), arXiv:0707.1889 .
  * Camino  _et al._ [2007] F. E. Camino, W. Zhou, and V. J. Goldman, $e/3$ laughlin quasiparticle primary-filling $\nu=1/3$ interferometer, Phys. Rev. Lett. 98, 076805 (2007).
  * Bartolomei  _et al._ [2020] H. Bartolomei, M. Kumar, R. Bisognin, A. Marguerite, J.-M. Berroir, E. Bocquillon, B. Placais, A. Cavanna, Q. Dong, U. Gennser, Y. Jin, and G. Feve, Fractional statistics in anyon collisions, Science 368, 173 (2020).
  * Nakamura  _et al._ [2020] J. Nakamura, S. Liang, G. C. Gardner, and M. J. Manfra, Direct observation of anyonic braiding statistics, Nature Physics 16, 931 (2020).
  * Bombin [2010] H. Bombin, Topological order with a twist: Ising anyons from an abelian model, Phys. Rev. Lett. 105, 030403 (2010).
  * You and Wen [2012] Y.-Z. You and X.-G. Wen, Projective non-abelian statistics of dislocation defects in a ${\mathbb{z}}_{N}$ rotor model, Phys. Rev. B 86, 161107 (2012).
  * Kitaev [2006] A. Kitaev, Anyons in an exactly solved model and beyond, Annals of Physics 321, 2 (2006).
  * Barkeshli  _et al._ [2013] M. Barkeshli, C.-M. Jian, and X.-L. Qi, Twist defects and projective non-abelian braiding statistics, Phys. Rev. B 87, 045130 (2013).
  * Barkeshli and Qi [2012] M. Barkeshli and X.-L. Qi, Topological nematic states and non-abelian lattice dislocations, Physical Review X 2, 031013 (2012).
  * Hormozi  _et al._ [2007] L. Hormozi, G. Zikos, N. E. Bonesteel, and S. H. Simon, Topological quantum compiling, Phys. Rev. B 75, 165310 (2007).
  * von Keyserlingk  _et al._ [2013] C. W. von Keyserlingk, F. J. Burnell, and S. H. Simon, Three-dimensional topological lattice models with surface anyons, Phys. Rev. B 87, 045107 (2013).
  * Teo  _et al._ [2014] J. C. Teo, A. Roy, and X. Chen, Unconventional fusion and braiding of topological defects in a lattice model, Physical Review B 90, 115118 (2014).
  * Zheng  _et al._ [2015] H. Zheng, A. Dua, and L. Jiang, Demonstrating non-abelian statistics of majorana fermions using twist defects, Phys. Rev. B 92, 245139 (2015).
  * Teo  _et al._ [2015] J. C. Teo, T. L. Hughes, and E. Fradkin, Theory of twist liquids: Gauging an anyonic symmetry, Annals of Physics 360, 349 (2015).
  * Brown  _et al._ [2017] B. J. Brown, K. Laubscher, M. S. Kesselring, and J. R. Wootton, Poking holes and cutting corners to achieve clifford gates with the surface code, Phys. Rev. X 7, 021029 (2017).
  * Zhu  _et al._ [2020] G. Zhu, M. Hafezi, and M. Barkeshli, Quantum origami: Transversal gates for quantum computation and measurement of topological order, Phys. Rev. Research 2, 013285 (2020).
  * Benhemou  _et al._ [2022] A. Benhemou, J. K. Pachos, and D. E. Browne, Non-abelian statistics with mixed-boundary punctures on the toric code, Phys. Rev. A 105, 042417 (2022).
  * Tantivasadakarn  _et al._ [2022] N. Tantivasadakarn, R. Verresen, and A. Vishwanath, The Shortest Route to Non-Abelian Topological Order on a Quantum Processor, (2022), arXiv:2209.03964 [quant-ph] .
  * Lensky  _et al._ [2022] Y. D. Lensky, K. Kechedzhi, I. Aleiner, and E.-A. Kim, Graph gauge theory of mobile non-abelian anyons in a qubit stabilizer code, (2022), arXiv:2210.09282 [quant-ph] .
  * Stern [2008] A. Stern, Anons and the quantum hall effect-a pedagogical review, Annals of Physics (New York) 323 (2008).
  * Harle  _et al._ [2022] N. Harle, O. Shtanko, and R. Movassagh, Observing and braiding topological Majorana modes on programmable quantum simulators, (2022), arXiv:2203.15083 [quant-ph] .
  * Kitaev [2003] A. Y. Kitaev, Fault-tolerant quantum computation by anyons, Ann. Phys. 303, 2 (2003).
  * Freedman [1998] M. H. Freedman, P/np, and the quantum field computer, Proc. Natl. Acad. Sci. USA 95, 98 (1998).
  * Pachos [2012] J. K. Pachos,  _Introduction to Topological Quantum Computation_ (Cambridge University Press, 2012).
  * Stern and Lindner [2013] A. Stern and N. Lindner, Topological quantum computation—from basic concepts to first experiments, Science 339, 1179 (2013).
  * Field and Simula [2018] B. Field and T. Simula, Introduction to topological quantum computation with non-abelian anyons, Quantum Science and Technology 3, 045004 (2018).
  * Moore and Read [1991] G. Moore and N. Read, Nonabelions in the fractional quantum hall effect, Nuclear Physics B 360, 362 (1991).
  * Willett  _et al._ [2010] R. L. Willett, L. N. Pfeiffer, and K. W. West, Alternation and interchange of e/4 and e/2 period interference oscillations consistent with filling factor 5/2 non-abelian quasiparticles, Phys. Rev. B 82, 205301 (2010).
  * Read and Green [2000] N. Read and D. Green, Paired states of fermions in two dimensions with breaking of parity and time-reversal symmetries and the fractional quantum hall effect, Phys. Rev. B 61, 10267 (2000).
  * Ivanov [2001] D. A. Ivanov, Non-abelian statistics of half-quantum vortices in $\mathit{p}$-wave superconductors, Phys. Rev. Lett. 86, 268 (2001).
  * Lutchyn  _et al._ [2010] R. M. Lutchyn, J. D. Sau, and S. Das Sarma, Majorana fermions and a topological phase transition in semiconductor-superconductor heterostructures, Phys. Rev. Lett. 105, 077001 (2010).
  * Oreg  _et al._ [2010] Y. Oreg, G. Refael, and F. von Oppen, Helical liquids and majorana bound states in quantum wires, Phys. Rev. Lett. 105, 177002 (2010).
  * Mourik  _et al._ [2012] V. Mourik, K. Zuo, F. S., S. Plissard, B. E., and L. Kouwenhoven, Signatures of majorana fermions in hybrid superconductor-semiconductor nanowire devices, Science 336, 1003 (2012).
  * Nadj-Perge  _et al._ [2014] S. Nadj-Perge, I. K. Drozdov, J. Li, H. Chen, S. Jeon, J. Seo, A. MacDonald, B. A. Bernevig, and A. Yazdani, Observation of majorana fermions in ferromagnetic atomic chains on a superconductor, Science 346, 602 (2014).
  * Banerjee  _et al._ [2018] M. Banerjee, M. Heiblum, V. Umansky, D. Feldman, Y. Oreg, and A. Stern, Observation of half-integer thermal hall conductance, Nature 559, 205 (2018).
  * Kasahara  _et al._ [2018] Y. Kasahara, T. Ohnishi, Y. Mizukami, O. Tanaka, S. Ma, K. Sugii, N. Kurita, H. Tanaka, J. Nasu, Y. Motome, and Y. Shibauchi, T. & Matsuda, Majorana quantization and half-integer thermal quantum hall effect in a Kitaev spin liquid, Nature 559, 227 (2018).
  * Bonderson  _et al._ [2006] P. Bonderson, A. Kitaev, and K. Shtengel, Detecting Non-Abelian Statistics in the $\nu=5/2$ Fractional Quantum Hall State, Phys. Rev. Lett. 96, 016803 (2006).
  * Gottesman [1997] D. Gottesman,  _Stabilizer codes and quantum error correction_ (California Institute of Technology, 1997).
  * Satzinger  _et al._ [2021] K. Satzinger, Y.-J. Liu, A. Smith, C. Knapp, M. Newman, C. Jones, Z. Chen, C. Quintana, X. Mi, A. Dunsworth, _et al._ , Realizing topologically ordered states on a quantum processor, Science 374, 1237 (2021).
  * Wen [2003] X.-G. Wen, Quantum orders in an exact soluble model, Phys. Rev. Lett. 90, 016803 (2003).
  * Arute  _et al._ [2019] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell, _et al._ , Quantum supremacy using a programmable superconducting processor, Nature 574, 505 (2019).

Supplemental materials for: Observation of non-Abelian exchange statistics on
a superconducting processor

###### Contents

  1. I Qubit decoherence and gate characterization
  2. II Readout details
  3. III Circuit details
  4. IV Dynamical decoupling
  5. V Additional braiding data

##  I Qubit decoherence and gate characterization

The experiments are performed on a quantum processor with frequency-tunable
transmon qubits and a similar design to that in Ref. 45. Figure S1a shows the
measured relaxation times of the 25 qubits that were used in the experiment,
with a median value of $T_{1}=21.7$ $\upmu$s. We also measure the dephasing
time $T_{2}$ in a Hahn echo experiment, shown in Fig. S1b, with the same
median value of 21.7 $\upmu$s.

Next, we benchmark the gates used in the experiment. Fig. S2a and b show the
cummulative distribution of the Pauli errors for single- and two-qubit (CZ)
gates, respectively. The median Pauli errors are $1.3\times 10^{-3}$ for the
single-qubit gates and $7.3\times 10^{-3}$ for the two-qubit gates.

Figure S1: Qubit relaxation ($T_{1}$) and coherence ($T_{2}$) times. a,b
Cummulative distributions of $T_{1}$ (a) and $T_{2}$ (b), where the latter is
measured using a Hahn echo sequence. Dashed lines indicate the median values
of 21.7 $\upmu$s for both measures. Insets: $T_{1}$ and $T_{2}$ plotted
against qubit number.  Figure S2: Gate errors. a,b, Cummulative distributions
of the Pauli error for single-qubit (a) and two-qubit CZ (b) gates. We find
median error values of $1.3\times 10^{-3}$ and $7.3\times 10^{-3}$ for the
single-qubit and CZ-gates, respectively.

##  II Readout details

We mitigate asymmetric readout effects by using a symmetrized readout scheme,
in which an $X$-gate is applied to every qubit before the readout step in half
of the measurements. Moreover, readout errors are corrected for by dividing
each stabilizer value by (symmetrized) measurements of
$\langle\Pi_{i}Z_{i}\rangle$ on the $\ket{00..00}$-state, where the product
runs over all qubits in the stabilizer. Fig. S3 displays the measured readout
errors, as well as a comparison of the stabilizer values in the surface code
ground state (same data as in step I in main text Fig. 2) before and after
readout correction.

Figure S3: Readout benchmarking and correction: a, Histogram of readout error,
with a median value of 2.0% (dashed vertical line). Inset: Readout error
plotted against qubit number. b,c, Stabilizer values of the surface code
ground state before (a) and after (b) readout correction.

##  III Circuit details

In our experiment, the two-qubit unitaries
$U_{\pm}(\hat{\tau}_{1}\hat{\tau}_{2})$ are converted to single-qubit
rotations and CZ-gates, as shown in Fig. S4a. In the particular case where a
D3V is moved diagonally (see step IV in Fig. 3 in the main text), we realize
the unitary by including two SWAP-gates (also converted to CZ-gates) since the
qubits are connected in a square grid (see Fig. S4b). Moreover, the three-
qubit unitary in step VIII in Fig. 3 is equivalent to a combination of single-
qubit gates, 4 SWAP-gates and 4 CZ-gates (Fig. S4c), which can be further
converted to single-qubit gates and 15 CZ-gates. We also include five single-
qubit rotations to permute $\hat{X}$, $\hat{Y}$ and $\hat{Z}$ of the three
stabilizers touching the moving D3V in steps V-VIII and IX-XI, as well as
three Hadamard gates to return all stabilizers to the original
$\hat{Z}\hat{X}\hat{X}\hat{Z}$-form in XII. In the experimental implementation
of the circuit, adjacent single-qubit gates on the same qubit are merged
together and performed in the layer after the most recent CZ-gate (Fig. S4d).

Figure S4: Circuit details. a, The unitary needed to move a D3V between two
neighboring vertices is realized in the experiment through the use of one CZ-
gate and single-qubit rotations. b, When D3Vs are moved diagonally, we include
two SWAP-gates, requiring three CZ-gates each. c, The three-qubit unitary used
in step VIII in Fig. 3 is equivalent to a combination of single-qubit gates, 4
SWAP-gates and 4 CZ-gates. d, Adjacent single-qubit gates are merged and
shifted left to the nearest CZ-gate.

##  IV Dynamical decoupling

In order to mitigate the effects of qubit decoherence during the circuits, we
perform dynamical decoupling on qubits that are idle for more than three
layers of gates. In particular, we utilize the Carr-Purcell-Meiboom-Gill
(CPMG) scheme, consisting of $X$-pulses interspaced by a wait time of
$\tau=25$ ns. Fig. S5 shows an example comparison of the stabilizers in cases
with and without dynamical decoupling, after braiding of anyons (41 layers of
SQ gates and 40 layers of CZ-gates). A clear improvement is observed,
increasing the average absolute stabilizer value from 0.50 to 0.58.

Figure S5: Dynamical decoupling. a,b, Stabilizer values without (a) and with
(b) dynamical decoupling, after D3V braiding. Dynamical decoupling improves
the average absolute stabilizer value from 0.50 to 0.58.

##  V Additional braiding data

In Figure 3 in the main text, we demonstrate that no fermion appears when
distinguishable $\sigma$ are braided with each other. In Fig. S6, we show the
data for each step in that protocol, analogous to those shown for
indistinguishable $\sigma$ in the main text. Moreover, we also present an
alternative braiding scheme in Fig. S7, which requires fewer (18) CZ-gates. In
this case, however, pair B is not brought back together, and neither of the
$\sigma$-pairs are annihilated. Therefore, similar to in Fig. 2c, we measure
the Pauli string corresponding to bringing a plaquette violation around pair A
(gray path in Fig. S7c), which in this case can be reduced to $\hat{Y}$ on the
qubit where the two $\sigma$ overlap. We find
$\langle\hat{\mathcal{P}}\rangle=\langle\hat{Y}\rangle=-0.71\pm 0.01$,
indicating that braiding the $\sigma$ led to the creation of a fermion (Fig.
S7c). Note that we here only search for fermions in one of the $\sigma$-pairs.
As a control experiment, we repeat the experiment with distinguishable
$\sigma$-pairs, as in the main text (Fig. S7d). In this case, we find
$\langle\hat{\mathcal{P}}\rangle=+0.71\pm 0.01$, thus demonstrating that no
fermion was produced. Together, these observations constitute another
demonstration of non-Abelian exchange statistics of the D3Vs.

Figure S6: Braiding distinguishable D3Vs a, Braiding schematic of worldlines.
b, Step-by-step depiction of stabilizers as the two $\sigma$ are braided,
analogous to that in Fig. 3 in the main text, but with distinguishable
$\sigma$.  Figure S7: Alternative protocol for braiding $\sigma$. a, Schematic
displaying the braiding process of the two $\sigma$-pairs. b, Experimental
demonstration of braiding, displaying the values of the stabilizers throughout
the process. Two $\sigma$-pairs, A and B, are created from the vacuum
$\mathbb{1}$, and one of the D3Vs in pair A is brought to the right side of
the grid. Next, a $\sigma$ from pair B is moved to the top, thus crossing the
path of the first $\sigma$, before bringing the $\sigma$ from pair A back
again to complete the braid. The diagonal $\sigma$ move performed in step VI
is achieved by including two SWAP-gates, corresponding to 6 additional CZ-
gates. The yellow triangles represent the locations of the $\sigma$, while the
brown and green lines represent the paths of $\sigma$ from pair A and B,
respectively. The average absolute stabilizer value is $0.93\pm 0.06$ and
$0.77\pm 0.09$ in the first and last step, respectively. c, After braiding the
$\sigma$, we search for hidden fermions by measuring the Pauli string
$\hat{\mathcal{P}}$ (left panels), which here is equivalent to $\hat{Y}$ on
the qubit where the two $\sigma$ overlap. The measurement yields
$\langle\hat{\mathcal{P}}\rangle=\langle\hat{Y}\rangle=-0.71\pm 0.01$,
indicating creation of a fermion. Right: world-lines of braiding process,
including non-local measurement based on plaquette violation loop. d, Same as
c, but after braiding two distinguishable $\sigma$, achieved by applying the
inverse two-qubit gates when moving the $\sigma$ in pair B. The measurement
yields $\langle\hat{Y}\rangle=+0.71\pm 0.01$, indicating no fermion creation.


# Intelligent Resource Allocations for IRS-Assisted OFDM Communications: A
Hybrid MDQN-DDPG Approach

Wei Wu∗, Fengchun Yang∗, Fuhui Zhou†, Han Hu∗, Qihui Wu,† and Rose Qingyang
Hu‡
∗Nanjing University of Posts and Telecommunications, Nanjing, China,
†Nanjing University of Aeronautics and Astronautics, Nanjing, China,
‡Utah State University, USA.
Emails: {weiwu<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>This work was
supported by the National Key R&D Program of China under Grant 2020YFB1807602,
the National Natural Science Foundation of China under Grant 61901231, under
Grant 62071223, under Grant 62031012, and Grant 61931011, the National Key
Scientific Instrument and Equipment Development Project under Grant 61827801,
the China Postdoctoral Science Foundation under Grant 2020M671480 and
Postdoctoral Science Foundation of Jiangsu (2020Z295), young Elite Scientist
Sponsorship Program by CAST.

###### Abstract

In this paper, we study the resource allocation problem for an intelligent
reflecting surface (IRS)-assisted OFDM system. The system sum rate
maximization framework is formulated by jointly optimizing subcarrier
allocation, base station transmit beamforming and IRS phase shift. Considering
the continuous and discrete hybrid action space characteristics of the
optimization variables, we propose an efficient resource allocation algorithm
combining multiple deep Q networks (MDQN) and deep deterministic policy-
gradient (DDPG) to deal with this issue. In our algorithm, MDQN are employed
to solve the problem of large discrete action space, while DDPG is introduced
to tackle the continuous action allocation. Compared with the traditional
approaches, our proposed MDQN-DDPG based algorithm has the advantage of
continuous behavior improvement through learning from the environment.
Simulation results demonstrate superior performance of our design in terms of
system sum rate compared with the benchmark schemes.

###### Index Terms:

OFDM, intelligent reflecting surface, beamforming matrix, phase shift matrix,
channel assignment, hybrid action space, MDQN-DDPG.

##  I Introduction

Orthogonal frequency division multiplexing (OFDM) is widely used in many
communication systems such as LTE and fifth generation wireless communication
networks [1]. By utilizing orthogonal subcarriers, it can achieve high-speed
and robust information transmission, and effectively avoid inter-channel
interference [2]. Moreover, the system performance can be significantly
improved by optimizing the channel assignment and power control [3]. With the
rapid development of mobile internet and wireless services, we are facing the
explosive growth of mobile data and higher data rate requirements. However,
the wireless channel fading significantly decreases the performance of the
OFDM communication system and the user experience. Therefore, how to improve
the performance of the OFDM communication system has become an urgent common
concern in both industry and academia [4].

Recently, intellligent reflecting surface (IRS) has been proposed as a
promising solution to enhancing the signal quality at the desired receiver in
a cost-effective and energy-efficient way [5]. Specifically, IRS is a
reflective array composed of a large number of low energy consuming and low-
cost passive reflective elements [6]. Each element can independently adjust
the phase shift of the incident signal [7], so as to cooperatively change the
propagation of the reflected signal in order to achieve the desired channel
response. By properly adjusting the phase shift of IRS elements, the reflected
signals of different paths can be coherently combined at the receiver to
maximize the achievable rate of the link. Therefore, with such a
characteristic, IRS is able to overcome the problem of OFDM channel fading,
and provide higher data rate for the network and better experience for the
users. The authors in [8] studied the scenario of IRS assisted OFDM
communications. However, the authors only considered the single antenna BS
scenario. The authors applied alternating optimization and successive convex
approximation (SCA) techniques to solve the optimization problems of joint IRS
reflection coefficient, time-frequency resource block allocation and power
allocation, so as to maximize the common (minimum) rate among all users. In
[9], the authors studied the adaptive transmission scenario of an IRS assisted
uplink OFDM system, and an algorithm based on semidefinite relaxation
technology was employed to improve the average achieveable rate.

However, most of the above work adopted traditional mathematical skills such
as alternating optimization and successive convex approximation. It is
difficult for these complex mathematical operations and numerical optimization
methods are difficult to meet the real time processing requirements of large-
scale heterogeneous communication systems. Recently, deep reinforcement
learning (DRL) has attracted wide attention from researchers due to its real
time performance [10]-[12]. In [13], DRL was used to solve the joint
optimization problem of beamforming matrix and IRS phase shift matrix. In [5],
the authors proposed a secure beamforming method based on DRL. The authors in
[14] used DRL to optimize the IRS phase shift matrix. Simulation results
showed that the DRL algorithm can achieve the upper bound of system
performance using lower time consumption compared to the positive semidefinite
relaxation algorithm. To the authors’ best knowledge, in the IRS-assisted OFDM
resource allocation scenario, there has been no relevant research on using DRL
to optimize channel allocation, beamforming matrix and IRS passive beam phase
shift, which is of crucial importance for making full use of IRS to improve
the performance of OFDM systems.

In this paper, we study an IRS assisted OFDM communication system. Our goal is
to achieve the maximum total system rate while ensuring the minimum
transmission rate requirements of users. The optimization problem is
transformed into a Markov decision process. Aiming at tackling the problem of
discrete and continuous hybrid action space, we propose an efficient resource
allocation algorithm based on multiple deep Q networks and deep deterministic
policy-gradient to jointly optimize channel allocation, beamforming and IRS
passive beam phase shift. Simulation results show that the proposed algorithm
can significantly improve the sum rate of the system, and quickly converge.

The rest of the paper is organized as follows. Section II presents the system
model. Section III presents the resource allocation algorithm based on MDQN-
DDPG. Section IV gives simulation results. Section V presents the conclusion.

##  II System Model

We consider an IRS-assisted downlink OFDM communications system as shown in
Fig. 1, in which IRS is used to enhance the signal strength between base
station (BS) and users. The BS is equipped with $M$ antennas and the user is
equipped with a single antenna. The IRS contains $N$ passive reflection
elements, and is connected to a controller that adjusts the IRS mode for the
required signal reflection. ${\cal K}=\left\\{{1,\cdots,K}\right\\}$ and
${\cal C}=\left\\{{1,\cdots,C}\right\\}$ represent the user set and channel
set, respectively.

Figure 1: An IRS-assisted downlink OFDM communications system with multiple
users.

Each user can receive signals from the BS through both direct and reflective
links. $\mathbf{H}_{B,R}\in\mathbb{C}^{M\times N\times C}$,
$\mathbf{h}_{R,k}\in\mathbb{C}^{N\times 1\times C}$ and
$\mathbf{h}_{B,k}\in\mathbb{C}^{M\times 1\times C}$ denote the channel
coefficients from the BS to the IRS, from the IRS to the user $k$ and from the
BS to user $k$, respectively. Let
$\mathbf{\Theta}=\text{diag}({\Theta_{1}},{\Theta_{2}},\cdots,{\Theta_{N}})\in\mathbb{C}^{N\times
N}$ denote the phase shift matrix related to IRS, where
$\Theta_{n}=w_{n}e^{j\theta_{n}}$ comprises amplitude correlation coefficient
$w_{n}\in\left[{0,1}\right]$ and phase correlation coefficient
$\theta_{n}\in\left[{0,2\pi}\right]$. Since each element of the IRS is
designed to do complete reflection, the amplitude correlation coefficient is
set as $w_{n}=1$ [9]. At the BS, the beamforming vector for user $k$ on the
channel $c$ is denoted as $\mathbf{f}_{k}^{c}$. The total transmit power at
the BS is constrained as
$\sum_{k=1}^{K}{\sum_{c=1}^{C}\rho_{k}^{c}||\mathbf{f}_{k}^{c}||^{2}}\leq
P_{T}$, where $P_{T}$ is the maximum transmission power of BS. $\rho_{k}^{c}$
indicates the user’s channel usage. If user $k$ uses channel $c$, then
$\rho_{k}^{c}=1$; otherwise, $\rho_{k}^{c}=0$.

Based on the above descriptions, the signal received at user $k$ can be given
as

${y_{k}}=\sum\limits_{c=1}^{C}{\rho_{k}^{c}\left({{{\bf{h}}_{B,k}^{c}}^{T}+{{\bf{h}}_{R,k}^{c}}^{T}{\bf{\Theta}}{{\bf{H}}_{B,R}^{c}}}\right)}{\bf{f}}_{k}^{c}{s_{k}}+\sigma_{k}^{2},$
(1)

where $\sigma_{k}^{2}$ is additive complex Gaussian random noise at the user
$k$, $s_{k}$ is the emission symbol of the user $k$ satisfying
$\mathbb{E}\left[{{{\left|{{s_{k}}}\right|}^{2}}}\right]=1$.
${\bf{h}}_{B,k}^{c}$ represents the channel coefficient between the BS and
user $k$ when the BS transmits data to user $k$ using channel $c$. Similarly,
when the channel $c$ is used for transmission, ${\bf{H}}_{B,R}^{c}$ and
${\bf{h}}_{R,k}^{c}$ represent the channel coefficients between BS to IRS and
IRS to user $k$, respectively.

Based on (1), the transmission rate from BS to the user $k$ on the $c$-th
channel can be expressed as

$\
R_{k}^{c}=\frac{B}{C}\rho_{k}^{c}{\log_{2}}\left[{1+\frac{{{{\left|{\left({{{\bf{h}}_{d,k}^{c}}^{T}+{{\bf{h}}_{R,k}^{c}}^{T}{\bf{\Theta}}{{\bf{H}}_{B,R}^{c}}}\right){\bf{f}}_{k}^{c}}\right|}^{2}}}}{{\sigma_{k}^{2}}}}\right],$
(2)

where $B$ denotes the bandwidth.

Therefore, the total transmission rate of all the $K$ users can be given as

$R=\sum_{k=1}^{K}{\sum_{c=1}^{C}R_{k}^{c}}.$ (3)

Our design goal is to find the optimal $\mathbf{f}_{k}^{c}$,$\mathbf{\Theta}$
and $\rho_{k}^{c}$ to maximize the system sum rate on the premise of meeting
the minimum transmission rate requirements of users and BS transmit power
constraint. The optimization problem can be formulated as

$\displaystyle\max_{\mathbf{\rho},\mathbf{f},\mathbf{\Theta}}~{}R$
$\displaystyle s.t.~{}~{}$
$\displaystyle\sum_{k=1}^{K}{\sum_{c=1}^{C}\rho_{k}^{c}||\mathbf{f}_{k}^{c}||^{2}}\leq
P_{T},$ (4a) $\displaystyle\ |\theta_{n}|=1,\forall n\in\mathcal{N},$ (4b)
$\displaystyle\rho_{k}^{c}\in\\{0,1\\},\forall
k\in\mathcal{K},c\in\mathcal{C},$ (4c)
$\displaystyle\sum_{k=1}^{K}\rho_{k}^{c}\leq 1,\forall c\in\mathcal{C},$ (4d)
$\displaystyle\sum_{c=1}^{C}R_{k}^{c}\geq R_{k}^{min},\forall
k\in\mathcal{K},$ (4e)

where $R_{k}^{min}$ represents the minimum transmission rate requirement of
the user $k$. The constraint (4a) ensures that the total transmit power of the
BS is less than its maximum transmit power, the constraint (4b) indicates that
the reflection unit of IRS is the total reflection with the amplitude
correlation coefficient being 1, the constraints (4c) and (4d) enforce that a
channel can only be occupied by one user instead of multiple users, the
constraint (4e) means that each user must meet their own minimum transmission
rate requirements. Due to the joint optimization of discrete subcarrier
allocation, continuous passive beam phase shift and beamforming, problem (4)
is highly non-convex and difficult to solve. Therefore, a joint DQN and DDPG
algorithm based on DRL is proposed to provide a global optimal solution.

##  III DRL Based Resource Allocation Framework

###  III-A Problem Formulation Based on MDP

Model free RL is a dynamic decision-making tool, which can solve the decision-
making problem by learning the optimal solution in dynamic environment [9]. We
model the formulated discrete continuous optimization problem as an MDP
problem. The IRS-assisted wireless communication scenario is regarded as an
environment, and the central controller of BS is regarded as an agent. In
addition to environment and agent, the MDP problem also includes state,
action, reward and transition probability. The key elements of MDP are
described as follows.

State space: Set $S$ as the state space. The system state $s^{t}\in S$ at
current time includes channel allocation at the previous time, IRS passive
beam phase shift, beamforming matrix, achievable rate of all users and channel
vectors corresponding to $C$ channels. $s^{t}$ at t is defined as

$s^{t}=\\{a_{1}^{t-1},a_{2}^{t-1},\\{R_{k}^{t-1}\\}_{k\in\mathcal{K}},\\{\mathbf{h}_{c}^{t}\\}_{c\in\mathcal{C}}\\}.$
(5)

where $a_{1}^{t-1}$ and $a_{2}^{t-1}$ represent discrete action and continuous
action at $t-1$ time respectively. Since the input of neural network can only
be real numbers instead of complex numbers, the channel state
$\mathbf{h}_{c}^{t}$ is divided into real part and imaginary part as input
into neural network, respectively.

Action space: Set $A$ as the action space, and the central controller of BS
selects the optimal action according to the current environmental state. Since
the optimization problem includes discrete and continuous hybrid actions, the
action space can be divided into two parts. One part is for discrete actions
including user’s channel allocation $\lambda$, and the other part is for
continuous actions including beamforming vector $\mathbf{f}$ of BS and passive
beam phase shift $\mathbf{\Theta}$ of IRS. Hence, action $a\in A$ can be
defined as

$a^{t}=\\{a_{1}^{t},a_{2}^{t}\\},$ (6)

where $a_{1}^{t}=\\{\lambda_{c}\\}_{c\in\mathcal{C}}$ and
$a_{2}^{t}=\\{\\{\mathbf{f}_{c}\\}_{c\in\mathcal{C}},\mathbf{\Theta}\\}$.

Transition probability: $Pr(s^{t+1}|s^{t},a^{t})$ is defined as the transition
model, which is the probability of switching from state $s^{t}\in S$ to the
new state $s^{t+1}\in S$ after taking action $a^{t}\in A$.

Reward function: The reward function is a very important part of DRL design
process, which is closely related to the desired goal of the system. According
to the optimization problem formulated in Section II, the goal of the reward
function contains two aspects. One is to maximize the system sum rate and the
other one is to meet the minimum data rate requirements. Based on the above
objective design, the reward function can be expressed as

$r=w_{1}R_{all}+w_{2}\sum_{k=1}^{K}\delta_{k},$ (7)

where $w_{1}$ and $w_{2}$ are constant coefficients, $R_{all}$ is the system
sum rate, $\delta_{k}$ is a penalty item to punish the user whose current
transmission rate does not meet the minimum transmission rate requirements,
which can be given as

$\delta_{k}=\begin{cases}0,&R_{k}\geq R_{k}^{min}\\\
-R_{k},&0<R_{k}<R_{k}^{min}\\\ -b,&R_{k}=0,\end{cases}$ (8)

where $R_{k}$ is the transmission rate of the user $k$, $b>0$ is a constant
coefficient.

The goal of the agent is to find an optimal strategy to maximize the long-term
reward. The cumulative discount reward can be denoted as

$R^{t}=\sum_{\tau=0}^{\infty}\gamma^{\tau}r^{t+\tau+1},$ (9)

where $\gamma\in[0,1)$ is the discount rate.

$Q_{\pi}(s,a)$ is used as the state action value function. Given the state
$s^{t}$, action $a^{t}$ and reward $R^{t}$, the Q function can be expressed as

${Q_{\pi}}\left({{s^{t}},{a^{t}}}\right)={E_{\pi}}\left[{{R^{t}}\left|{{s^{t}}=s,}\right.{a^{t}}=a}\right].$
(10)

Then, the Q function that satisfies the Bellmann equation is obtained as

$\begin{split}&{Q_{\pi}}\left({{s^{t}},{a^{t}}}\right)={E_{\pi}}\left[{{r^{t+1}}\left|{{s^{t}}=s,{a^{t}}=a}\right.}\right]+\gamma\sum\limits_{{s^{t+1}}\in
S}{}\\\
&P\left({{s^{t+1}}\left|{{s^{t}},{a^{t}}}\right.}\right)\left({\sum\limits_{{a^{t+1}}\in
A}{\pi\left({{s^{t+1}},{a^{t+1}}}\right){Q_{\pi}}\left({{s^{t+1}},{a^{t+1}}}\right)}}\right).\end{split}$
(11)

The Q-learning algorithm is used to search for the optimal policies $\pi^{*}$.
From (11), the optimal Q function associated with the optimal policy can be
expressed as

$\begin{split}Q^{*}\left(s^{t},a^{t}\right)=&r^{t+1}+\gamma\sum_{s^{t+1}\in
S}P\left(s^{t+1}|s^{t},a^{t}\right)\\\ &\max_{a^{t+1}\in
A}Q^{*}\left(s^{t+1},a^{t+1}\right).\end{split}$ (12)

The Bellmann equation can be solved in a recursive way, and iterating (12)
yields the optimal Q function. Therefore,, the update procedure for the
iterative solution can be denoted as

$\begin{split}Q\left(s^{t},a^{t}\right)\leftarrow&(1-\alpha)Q\left(s^{t},a^{t}\right)\\\
&+\alpha\left(r^{t+1}+\gamma\max_{a^{t+1}}Q^{*}\left(s^{t+1},a^{t+1}\right)\right),\end{split}$
(13)

where $\alpha\in(0,1]$ is the learning rate of the Q function updates.

###  III-B Resource Allocation With MDQN-DDPG

In this section, we introduce the MDQN-DDPG framework, as shown in Fig. 2, in
which MDQN is used to process the discrete actions, while DDPG is used for
continuous actions.

Figure 2: Proposed framework for resource allocation based on MDQN-DDPG.

####  III-B1 MDQN

DQN is a classical method of DRL in many scenarios and can effectively deal
with problems with large state space and large action space. One of its main
features is that the output action is discrete. Therefore, we adopt DQN
framework to solve the channel allocation issue in our formulated problem, but
if there is only one DQN, it will lead to huge action space. For example, in a
system with $C$ channels, $K$ users, the action space reaches $K^{C}$. Thus,
we use multiple DQN for distributed processing, $C$ channels adopt $C$ DQN,
DQN select the best user for the channel. In this way, the action space can be
reduced from $K^{C}$ to $K*C$, to greatly improve the learning efficiency.

DQN is a Q network constructed by DNN to estimate the Q value of the selected
action. Specifically, with state S as network input, the Q network outputs the
estimated Q values for all actions. Following that, the $\epsilon$-greedy
method can be adopted to carry out channel allocation to balance the
exploration of action and the exploitation of known actions. In other words,
it can randomly select one action among all the actions with $\epsilon$
probability or select action $a$ with the largest estimated value with
1-$\epsilon$ probability, which can be given as

$a=arg\max_{a\in A}Q\left(s,a;w\right),$ (14)

where $0<\epsilon<1$. The DQN network can continue to select actions with high
feedback rewards or explore actions that may have higher rewards but are not
yet selected, so as to explore the entire action space and update Q values.
$\omega$ represents the weight of the training network and leverages the
experience replay buffer for continuously updating.

The random extraction of $N$ tuples $(s^{j},a^{j},r^{j+1},s^{j+1})$ from the
experience replay buffer is to ensure the independence of the training tuples
and avoid the problem caused by excessive correlation among the tuples.

The randomly sampled tuples $(s^{j},a^{j},r^{j+1},s^{j+1})$ can be used to
generate target Q values

$y=r^{j+1}+\gamma\max_{a^{j+1}\in A}Q\left(s^{j+1},a^{j+1};\omega^{-}\right),$
(15)

where $\omega^{-}$ represents the weight of the target network. The loss
function gives the difference between the predicted value of the neural
network and the actual target value. It can be expressed as

$l(\omega)=\left(y-Q\left(s^{j},a^{j};w\right)\right)^{2}.$ (16)

####  III-B2 DDPG

For the problem with continuous actions, such as beamforming and IRS passive
phase shifts, DQN does not work. Different from the traditional method
generating actions according to the probability, DDPG generates and determines
actions according to parametric strategy. Moreover, it has neural network and
Actor-Critic structure, which enables DDPG to deal with continuous action
issue effectively.

The DDPG network contains four sub-networks, e.g,, the current Actor network
composed of DNN for action selection, the current Critic network used to
generate estimated Q value of the current action, as well as the target Actor
network and target Critic network used to generate target value for training.
In summary, DDPG is an extension of actor-critic framework. It utilizes
experience replay and double network methods to improve the convergence
performance of the original framework.

For the Actor network, in order to balance the exploration of new actions and
the exploitation of known actions, random noise is added to the output
actions. After executing the action $a^{i}$, we can get the immediate reward
$r^{i+1}$ and store the tuple $(s^{i},a^{i},r^{i+1},s^{i+1})$ in the
experience replay buffer. According to the DPG theorem, $N$ tuples can be
randomly selected from the experience replay buffer to update the weight
$\theta$ in the current actor network, and the update gradient is given as

$\nabla_{\theta}J(\mu)=\frac{1}{N}\sum_{i}\left[\nabla_{a}Q\left(s^{i},a;\lambda\right)|_{a=\mu(s^{i};\theta)}\nabla_{\theta}\mu\left(s^{i};\theta\right)\right].$
(17)

For the Critic network, we use the target Critic network and the target Actor
network to update the weight $\lambda$. The sampled tuples are used to
generate target Q value of the current training, given as

$y=r^{i+1}+\gamma
Q\left(s^{i+1},a;\lambda^{-}\right)|_{a=\mu(s^{i+1};\theta^{-})},$ (18)

where $\theta^{-}$ and $\lambda^{-}$ are the weights of the target Actor
network and the target Critic network, respectively.

The loss function uses the difference between the predicted value and the
target value, and the weight $\lambda$ can be updated by minimizing the loss
function, that is

$l(\lambda)=\frac{1}{N}\sum_{i}\left[y-Q\left(s^{i},a^{i};\lambda\right)\right]^{2}.$
(19)

In short, the whole process starts with extracting extract tuples from the
experience replay buffer, followed by inputting the extracted tuples to the
target network and the current network. Finally, we can solve the gradient
using (17) and (19), and update the parameters $\theta$ and $\lambda$
corresponding to the current network. After a certain number of steps, DDPG
copies the parameters of need to reword to the target network. Algorithm 1
summarizes the details of MDQN-DDPG framework for IRS-assisted downlink OFDM
system.

Algorithm 1: Resource Allocation in OFDM With MDQN-DDPG
---
Input: $\mathbf{H}_{BR},\mathbf{h}_{Rk},\mathbf{h}_{dk}$. Minimum transmission
rate requirements
for all users
Output: optimal action $a=\\{\mathbf{\rho},\mathbf{f},\mathbf{\Theta}\\}$,$Q$
value function
Initialization: experience replay buffer D with size D,
the Q function of M DQNs, parameters $w$, parameters $w^{-}$,the
parameters $\theta$ of the training Actor network, the parameter $\theta^{-}$
of the
target Actor network, the parameters $\lambda$ of the training Critic network
and the parameters $\lambda^{-}$ of the target Critic network in the DDPG
network, Channel allocation $\rho$, beamforming $\mathbf{f}$ and IRS phase
shift $\mathbf{\Theta}$.
1: for each episode do
2: Collect $\mathbf{H}_{BR},\mathbf{h}_{Rk},\mathbf{h}_{dk}$ to observe an
initial system state $s^{0}$;
3: for each step $t$ do
4: for each DQN agent $m$ do
5: In the current state $s^{t}$, action $a_{1,m}^{t}$ is selected according to
$\epsilon-$greedy policy;
6: end for
7: Obtain action $a_{2}^{t}$ from actor network;
8: Execute action $a^{t}=\\{a_{1}^{t},a_{2}^{t}\\}$ to obtain instant reward
$r^{t}$ and
next state $s^{t+1}$;
9: Store $(s^{t},a^{t},r^{t+1},s^{t+1})$ in the replay memory D;
10: Mini-batch of samples with size $N$ is randomly selected from the
replay memory D;
11: for each DQN agent $m$ do
12: The SGD method is used to minimize the error between
the predicted value and the target value, as shown in (16)
13: Update DQN network parameters $w$.
14: end for
15: The loss function of the training Critic network represented by
(19) is generated.
16: Generating gradient $\nabla_{a}Q(s^{t},a;\lambda)$ of training Critic
network;
17: Generating gradient $\nabla_{\theta}\mu(s^{t};\theta)$ of training Actor
network;
18: Update training Actor network parameters $\theta$;
19: Update training Critic network parameters $\lambda$;
20: Update DQN target network parameters $w^{-}$ every $P$ steps;
20: Update target Actor network parameters $\theta^{-}$ every $P$ steps;
21: Update target Critic network parameters $\lambda^{-}$ every $P$ steps;
22: end for
23: end for.

##  IV Simulation Results

In this section, the performance of the proposed resource allocation algorithm
is evaluated and compared with the benchmark schemes. In the simulation, the
channel from BS to user is assumed to be Rayleigh fading, while the BS to IRS
channel and the IRS to user channel are modeled as Rician fading. According to
[5], we can express the corresponding path fading as
$PL=(PL_{0}-10\tau\log_{10}(d/D_{0}))$ dB, where $PL_{0}$ = 30 dB path loss
with reference distance $D_{0}$ = 1 $m$. We set the path loss exponent from BS
to user as $\tau_{bu}=3.75$, and the path loss exponents from BS to IRS and
IRS to user are $\tau_{br}=2.2$ and $\tau_{ru}=2.2$, respectively.

A three-dimensional coordinate system is established, $K$ single antenna
ground users are randomly located in a 100 $m$ $*$ 100 $m$ rectangular area,
and the lower left corner of the rectangular area is marked as $(100,0,0)$,
while the upper right corner is marked as $(200,100,0)$. The BS and IRS are
located at $(0,0,30)$ and $(75,100,50)$, respectively. The background noise
power of all users is $-169$ dBm. The number of antennas corresponding to BS
is $M$ = 6, the number of users is $K$ = 3, the number of IRS phase shift
units $N$ ranges from 16 to 64, and the transmission power range of BS is $15$
dBm to $40$ dBm. In the proposed DRL based framework, DQN contains three
hidden layers, while DDPG Actor network and Critic network contains two hidden
layers. The learning rate of DQN is set as $0.002$, the learning rates of
Actor network and Critic network are set to $0.001$ and $0.002$, respectively.
The discount factor is set as $\gamma=0.99$, and the experience replay buffer
is $D=6000$.

Fig. 3 shows the reward versus the iteration of the algorithm when $P_{T}=35$
dBm and $N$ = 16. For comparison, two existing schemes without IRS and with
random selection of variable values are introduced. It is found that using the
proposed MDQN-DDPG algorithm, in both without IRS-assisted and IRS-assisted
cases, the rewards can be continuously improved and converge to a constant
value at about 75 episodes. Their early stage rewards are much lower because
the user’s transmission requirements are not satisfied. Moreover, higher
rewards can be achieved by our proposed IRS-assisted scheme than the scheme
without IRS. This means that the employing IRS is conducive to significantly
improve the system sum rate. From Fig. 3, it is seen that the proposed MDQN-
DDPG algorithm has greater reward than the random selection method. It is
proved that the algorithm is effective and can obtain a better solution to the
joint optimization problem.

Figure 3: Convergence comparisons of different schemes.

Figure 4: System sum rate versus BS transmission power $P_{T}$ under different
schemes.

Figure 5: System sum rate versus the number of IRS passive reflection elements
under different schemes.

Figure 6: Convergence comparison of different users.

Fig. 4 shows the effection of BS transmission power $P_{T}$ on system sum
rate. For comparison, a fixed beamforming scheme is also introduced. We set
the number of IRS passive reflection elements to 16. It is observed that the
system sum rate increases with the maximum transmission power of the base
station. Our proposed DRL based algorithm has better system sum rate
performance than all other three schemes, namely, random selection method, the
without IRS method and the fixed beamforming method.

Fig. 5 shows the effection of the number of IRS passive reflection elements on
the system sum rate. We set the BS transmission power $P_{T}$ to 35 dBm. As
shown in Fig. 5 the more number of IRS passive reflection elements the better
of system sum rate performance is achieved. This is because more reflections
units can provide more accurate regulation of the signal phase, and help to
achieve higher system sum rate and better communication quality. It can be
seen that when the design complexity of IRS is very high, our proposed
algorithm can also obtain better system sum rate than other algorithms.

Fig. 6 shows the convergence performance for different user’s data rates. It
is seen that the data rate of each user tends to converge with the increase of
the number of rounds. As the continuous interaction of the environment, the
algorithm can learn and adjust the optimization variables to approach the
optimal solution. In practice, the maximum sum data rare is envisioned to be
achieved while the users’ data rate requirement should be considered. It can
be seen from Fig. 6 that our proposed algorithm does not just allocate all
bandwidth and power to the user with the best channel state for transmission,
instead it considers the constraints of the whole network to find the best
allocation for each user.

##  V Conclusion

In this paper, we have investigated the resource allocation problem in IRS-
assisted OFDM systems by jointly optimizing the BS beamforming, IRS passive
phase shift and channel assignment. A hybrid MDQN-DDPG framework-based
algorithm was proposed to tackle the challenging hybrid discrete and
continuous action issue. The system sum rate was maximized while satisfying
the minimum transmission rate requirement of user. Simulation results
demonstrated that our proposed algorithm can adjust the action of the agent by
observing the immediate reward, and finally make the reward converge to the
optimal value. The optimal beamforming matrix, IRS phase shift and channel
allocation were obtained through the trained agent.

## References

  * [1] G. Berardinelli, K. I. Pedersen, T. B. Sorensen and P. Mogensen, “Generalized DFT-Spread-OFDM as 5G waveform,” _IEEE Comm. Mag._ , vol. 54, no. 11, pp. 99-105, Nov. 2016.
  * [2] Y. Iraqi and A. Al-Dweik, ”Efficient information transmission using smart OFDM for IoT applications,” _IEEE Internet Things J._ , vol. 7, no. 9, pp. 8397-8409, Sept. 2020.
  * [3] F. Zhou, Y. Wu and Q. Wu, “Resource allocation based on deep reinforcement learning for wideband cognitive radio networks,” _URSI GASS._ , 2021.
  * [4] Y. Liao, G. Sun, Z. Cai, X. Shen and Z. Huang, ”Nonlinear kalman filter-based robust channel estimation for high mobility OFDM systems,” _IEEE Trans. Intell. Transp. Syst._ , Jul. 15, 2020.
  * [5] H. Yang et al., “Intelligent reflecting surface assisted anti-jamming communications: a fast reinforcement learning approach,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 3, pp. 1963-1974, Mar. 2021.
  * [6] H. Niu, Z. Chu, F. Zhou, Z. Zhu, M. Zhang and K. -K. Wong, “Weighted sum secrecy rate maximization using intelligent reflecting surface,” _IEEE Trans. Commun._ , vol. 69, no. 9, pp. 6170-6184, Sept. 2021.
  * [7] H. Hashida, Y. Kawamoto and N. Kato, “Intelligent reflecting surface placement optimization in air-ground communication networks toward 6G,” _IEEE Wireless Commun._ , vol. 27, no. 6, pp. 146-151, Dec. 2020.
  * [8] Y. Yang, S. Zhang and R. Zhang, “IRS-enhanced OFDMA: joint resource allocation and passive beamforming optimization,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 6, pp. 760-764, Jun. 2020.
  * [9] S. Lin, B. Zheng, G. C. Alexandropoulos, M. Wen, F. Chen and S. sMumtaz, “Adaptive transmission for reconfigurable intelligent surface-assisted OFDM wireless communications,” _IEEE J. Select. Areas Commun._ , vol. 38, no. 11, pp. 2653-2665, Nov. 2020.
  * [10] F. Zhou, G. Lu, M. Wen, Y. -C. Liang, Z. Chu and Y. Wang, “Dynamic spectrum management via machine learning: state of the art, taxonomy, challenges, and open research issues,” _IEEE Netw._ , vol. 33, no. 4, pp. 54-62, July/Aug. 2019.
  * [11] C. Huang, G. Chen, Y. Gong, M. Wen and J. A. Chambers, ‘Deep reinforcement learning-based relay selection in intelligent reflecting surface assisted cooperative networks,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 5, pp. 1036-1040, May 2021.
  * [12] Q. Wu, T. Ruan, F. Zhou, Y. Huang, F. Xu, S. Zhao, Y. Liu, and X. Huang, “A unified cognitive learning framework for adapting to dynamic environment and tasks,” _IEEE Wireless Commun._ , to be published, 2021.
  * [13] C. Huang, R. Mo and C. Yuen, “Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning,” _IEEE J. Select. Areas Commun._ , vol. 38, no. 8, pp. 1839-1850, Aug. 2020.
  * [14] K. Feng, Q. Wang, X. Li and C. Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 5, pp. 745-749, May 2020.