11institutetext: Center for Informatics Science, Nile University, Giza, Egypt
11email<EMAIL_ADDRESS>22institutetext: Rice
Research Institute, Kafr ElSheikh, Egypt
22email<EMAIL_ADDRESS>
# Rice Plant Disease Detection and Diagnosis using Deep Convolutional Neural
Networks and Multispectral Imaging

Yara Ali Alnaggar1 Ahmad Sebaq1 Karim Amer1 ElSayed Naeem2 Mohamed Elhelw1

###### Abstract

Rice is considered a strategic crop in Egypt as it is regularly consumed in
the Egyptian people’s diet. Even though Egypt is the highest rice producer in
Africa with a share of 6 million tons per year [5], it still imports rice to
satisfy its local needs due to production loss, especially due to rice
disease. Rice blast disease is responsible for 30% loss in rice production
worldwide [9]. Therefore, it is crucial to target limiting yield damage by
detecting rice crops diseases in its early stages. This paper introduces a
public multispectral and RGB images dataset and a deep learning pipeline for
rice plant disease detection using multi-modal data. The collected
multispectral images consist of Red, Green and Near-Infrared channels and we
show that using multispectral along with RGB channels as input archives a
higher F1 accuracy compared to using RGB input only.

###### Keywords:

Deep learning Computer vision Multispectral Imagery.

##  1 Introduction

In Egypt, rice is important in Egyptian agriculture sector, as Egypt is the
largest rice producer in Africa. The total area used for rice cultivation in
Egypt is about 600 thousand ha or approximately 22% of all cultivated area in
Egypt during the summer. As a result, it is critical to address the causes of
rice production loss to minimize the gap between supply and consumption. Rice
plant diseases contribute mostly to this loss, especially rice blast disease.
According to [9], rice blast disease causes 30% worldwide of the total loss of
rice production. Thus, rice crops diseases detection, mainly rice blast
disease, in the early stages can play a great role in restraining rice
production loss.

Early detection of rice crops diseases is a challenging task. One of the main
challenges of early detection of such disease is that it can be misclassified
as the brown spot disease by less experienced agriculture extension officers
(as both are fungal diseases and have similar appearances in their early
stage) which can lead to wrong treatment. Given the current scarcity of
experienced extension officers in the country, there is a pressing need and
opportunity for utilising recent technological advances in imaging modalities
and computer vision/artificial intelligence to help in early diagnosis of the
rice blast disease. Recently, multispectral photography has been deployed in
agricultural tasks such as precision agriculture [3], food safety evaluation
[11]. Multispectral cameras could capture images in Red, Red-Edge, Green and
Near-Infrared bands wavebands, which captures what the naked eye can’t see.
Integrating the multispectral technology with deep learning approaches would
improve crops diseases identification capability. However, it would be
required to collect multispectral images in large numbers.

In this paper, we propose a public multispectral and RGB images dataset and a
deep learning pipeline for rice plant disease detection. First, the dataset we
present contains 3815 pairs of multispectral and RGB images for rice crop
blast, brown spot and healthy leaves. Second, we developed a deep learning
pipeline trained on our dataset which calculates the Normalised Difference
Vegetation Index (NDVI) channel from the multispectral image channels and
concatenates it along its RGB image channels. We show that using NDVI+RGB as
input archives a higher F1 score by 1% compared to using RGB input only.

##  2 Literature Review

Deep learning has emerged to tackle problems in different tasks and fields.
Nowadays, it is being adopted to solve the challenge of crop disease
identification. For example, Mohanty et al. [8] trained a deep learning model
to classify plant crop type and its disease based on images. Furthermore, [1]
proposed a deep learning-based approach for banana leaf diseases
classification.

Furthermore, multispectral sensors have proven its capability as a new
modality to detect crop fields issues and diseases. Some approaches use
multispectral images for disease detection and quantification. Cui et al. [4]
developed an image processing-based method for quantitatively detecting
soybean rust severity using multi-spectral images. Also, [12] utilize digital
and multispectral images captured using quadrotor unmanned aerial vehicles
(UAV) to collect high-spatial resolution imagery data to detect the ShB
disease in rice.

After the reliable and outstanding results deep learning models could achieve
on rgb images, some approaches were developed to use deep learning on
multispectral images, especially of crops and plants. [10] proposed a deep
learning-based approach for weed detection in lettuce crops trained on
multispectral images. In addition, Ampatzidis et al. [2] collects
multispectral images of citrus fields using UVA for crop phenotyping and
deploys a deep learning detection model to identify trees.

##  3 Methodology

###  3.1 Hardware Components

We used a MAPIR Survey3N camera, shown in Figure 1 to collect our dataset.
This camera model captures ground‐level multispectral images of red, green and
NIR channels. It was chosen in favour of its convenient cost and easy
integration with smartphones.. In addition, we used the Samsung Galaxy M51
mobile phone camera to capture RGB images, paired with the MAPIR camera.

Figure 1: MAPIR Survey3N Camera.

We Designed a holder gadget to combine the mobile phone, MAPIR camera and a
power bank in a single tool, as seen in Figure 2, to facilitate the data
acquisition operation for the officers. It was designed using SolidWorks
software and manufactured by a 3D printer.

Figure 2: Holder gadget.

###  3.2 Data Collection Mobile Application

An android frontend application was also developed to enable the officers who
collect the dataset to control the multispectral and the smartphone cameras
for capturing dual RGNIR/RGB images simultaneously while providing features
such as image labelling, imaging session management, and Geo‐tagging. The
mobile application is developed with Flutter and uses Firebase real‐time
database to store and synchronise the captured data including photos and
metadata. Furthermore, Hive local storage database is used within the
application to maintain a local backup of the data.

###  3.3 Analytics Engine Module

Our engine is based on ResNet18 [6] architecture which consists of 18 layers
and it utilize the power of residual network, see Figure 3, residual network
help us avoid the vanishing gradient problem.

We can see how layers are configured in the ResNet-18 architecture. The
architecture starts with a convolution layer with 7x7 kernel size and stride
of 2. Next we begin with the skip connection. The input from here is added to
the output that is achieved by 3x3 max pool layer and two convolution layers
with kernel size 3x3, 64 kernels each. This is the first residual block.

The output of this residual block is added to the output of two convolution
layers with kernel size 3x3 and 128 such filters. This constituted the second
residual block. Then the third residual block involves the output of the
second block through skip connection and the output of two convolution layers
with filter size 3x3 and 256 such filters. The fourth and final residual block
involves output of third block through skip connections and output of two
convolution layers with same filter size of 3x3 and 512 such filters.

Finally, average pooling is applied on the output of the final residual block
and received feature map is given to the fully connected layers followed by
softmax function to receive the final output.

The vanishing gradient is a problem which happens when training artificial
neural networks that involved gradient based learning and backpropagation. We
use gradients to update the weights in a network. But sometimes what happens
is that the gradient becomes very small, effectively preventing the weights to
be updated. This leads to network to stop training. To solve such problem,
residual neural networks are used.

Figure 3: ResNet18 original architecture

Residual neural networks are the type of neural network that applies identity
mapping. What this means is that the input to some layer is passed directly or
as a shortcut to some other layer. If $x$ is the input, in our case its an
image or a feature map, and $F(x)$ is the output from the layer, then the
output of the residual block can be given as $F(x)+x$ as shown in Figure 4.

We changed the input shape to be 256x256 instead of 224x244, also we replaced
the last layer in the original architecture with a fully connected layer where
the output size was modified to three to accommodate our task labels.

Figure 4: Residual block

##  4 Experimental Evaluation

###  4.1 Dataset

We have collected 3815 samples of rice crops of three labels: blast disease,
brown spot disease and healthy leaves distributed, shown in Figure 5, as the
following: 2135, 1095 and 585, respectively. Each sample is composed of a pair
of (RGB) and (R-G-NIR) images as seen in Figure 6, which were captured
simultaneously. Figure 7 shows samples of the three classes in our dataset.

Figure 5: Collected dataset distribution.

Figure 6: On the left is the RGB image and on the right is its R-G-NIR pair.

(a)

(b)

(c)

Figure 7: (a) Blast class sample. (b) Brown spot class sample. (c) Healthy
class sample.

###  4.2 Training Configuration

In this section, we explain our pipeline for training data preparation and
preprocessing. Also, we mention our deep learning models training
configuration for loss functions and hyperparameters.

####  4.2.1 Data Preparation

##### RGB images registration

Since the image sample of our collected dataset consists of a pair of RGB and
R-G-NIR images, the two images are expected to have a similar field of view.
However, the phone and MAPIR camera have different field of view parameters
that the mapir camera has a 41° FOV compared to the phone camera with 123°
FOV. As a result, we register the rgb image to the r-g-nir image using the
OpenCV library. The registration task starts by applying an ORB detector over
the two images to extract 10K features. Next, we use a brute force with
Hamming distance matcher between the two images extracted features. Based on
the calculated distances for the matches, we sort them descendingly and drop
the last 10%. Finally, the homography matrix is calculated using the matched
points in the two images to be applied over the RGB images. Figure 8 shows an
RGB image before and after registration.

Figure 8: On the left is an RGB image before calibration and on the right is
after registration.

##### MAPIR camera calibration

The MAPIR camera sensor captures the reflected light which lies in the
Wavelengths in the Visible and Near Infrared spectrum from about 400-1100n and
saves the percentage of reflectance. After this step, calibration of each
pixel is applied to ensure that it is correct. This calibration is performed
before every round of images captured using the MAPIR Camera Reflectance
Calibration Ground Target board, which consists of 4 targets with known
reflectance values, as shown in Figure 9.

Figure 9: MAPIR Camera Reflectance Calibration Ground Target board.

##### Models training configuration

We trained our models for 50 epochs with a batch size of 16 using Adam
optimizer and Cosine Annealing with restart scheduler [7] with cycle length 10
epochs and learning rate of 0.05. For the loss function, we used a weighted
cross entropy to mitigate the imbalance of the training dataset. Images were
resized to dimension 256 x 256.

####  4.2.2 Results

For training the deep learning model using RGB and R-G-NIR pairs, we generate
a NDVI channel, using Equation 1, and concatenate it to the RGB image. Our
study shows that incorporating the NDVI channel improves the model capability
to classify the rice crops diseases. Our model could achieve a F1 score with
5-kFold of 84.9% when using RGB+NDVI as input compared to using only RGB image
which could obtain a F1 score of 83.9%. Detailed results are presented in
Table 1.

$NDVI=\frac{NIR-Red}{NIR+Red}$ (1)

Table 1: F1 score over our collected dataset achieved by using RGB as input versus RGB+NDVI. Class |  RGB |  RGB+NDVI
---|---|---
Blast |  89.64% |  90.02%
Spot |  82.64% |  83.26%
Healthy |  79.08% |  81.54%

##  5 Conclusion

We presented our public dataset and deep learning pipeline for rice plant
disease detection. We showed that employing multispectral imagery with RGB
improves the model capability of disease identification by 1% compared to
using solely RGB imagery. We believe using a larger number of images for
training would enhance current results also considering a larger number of
images when using a deeper model this will result in better results. In
addition, more investigation on how to fuse multispectral imagery with RGB for
training could be applied, for example we can calculate NDVI from the blue
channel instead of the red this may also boost the model performance.

####  5.0.1 Acknowledgements.

The authors would like to acknowledge the support received from Data Science
Africa (DSA) which made this work possible.

## References

  * [1] Amara, J., Bouaziz, B., Algergawy, A.: A deep learning-based approach for banana leaf diseases classification. Datenbanksysteme für Business, Technologie und Web (BTW 2017)-Workshopband (2017)
  * [2] Ampatzidis, Y., Partel, V.: Uav-based high throughput phenotyping in citrus utilizing multispectral imaging and artificial intelligence. Remote Sensing 11(4), 410 (2019)
  * [3] Candiago, S., Remondino, F., De Giglio, M., Dubbini, M., Gattelli, M.: Evaluating multispectral images and vegetation indices for precision farming applications from uav images. Remote sensing 7(4), 4026–4047 (2015)
  * [4] Cui, D., Zhang, Q., Li, M., Hartman, G.L., Zhao, Y.: Image processing methods for quantitatively detecting soybean rust from multispectral images. Biosystems engineering 107(3), 186–193 (2010)
  * [5] Elbasiouny, H., Elbehiry, F.: Rice production in egypt: the challenges of climate change and water deficiency. In: Climate change impacts on agriculture and food security in Egypt, pp. 295–319. Springer (2020)
  * [6] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [7] Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
  * [8] Mohanty, S.P., Hughes, D.P., Salathé, M.: Using deep learning for image-based plant disease detection. Frontiers in plant science 7, 1419 (2016)
  * [9] Nalley, L., Tsiboe, F., Durand-Morat, A., Shew, A., Thoma, G.: Economic and environmental impact of rice blast pathogen (magnaporthe oryzae) alleviation in the united states. PloS one 11(12), e0167295 (2016)
  * [10] Osorio, K., Puerto, A., Pedraza, C., Jamaica, D., Rodríguez, L.: A deep learning approach for weed detection in lettuce crops using multispectral images. AgriEngineering 2(3), 471–488 (2020)
  * [11] Qin, J., Chao, K., Kim, M.S., Lu, R., Burks, T.F.: Hyperspectral and multispectral imaging for evaluating food safety and quality. Journal of Food Engineering 118(2), 157–171 (2013). https://doi.org/https://doi.org/10.1016/j.jfoodeng.2013.04.001, https://www.sciencedirect.com/science/article/pii/S0260877413001659
  * [12] Zhang, D., Zhou, X., Zhang, J., Lan, Y., Xu, C., Liang, D.: Detection of rice sheath blight using an unmanned aerial system with high-resolution color and multispectral imaging. PloS one 13(5), e0187470 (2018)


# Discovery of Optimal Quantum Error Correcting Codes via Reinforcement
Learning

Vincent Paul Su  Center for Theoretical Physics & Department of Physics,
University of California, Berkeley, CA 94720, USA  ChunJun Cao  Joint Center
for Quantum Information and Computer Science, University of Maryland, College
Park, MD, 20742, USA  Institute for Quantum Information and Matter, California
Institute of Technology, Pasadena, CA, 91125, USA  Department of Physics,
Virginia Tech, Blacksburg, VA, 24060, USA  Hong-Ye Hu  Department of Physics,
University of California San Diego, La Jolla, CA 92093, USA  Department of
Physics, Harvard University, 17 Oxford Street, Cambridge, MA 02138, USA
Harvard Quantum Initiative, Harvard University, 17 Oxford Street, Cambridge,
MA 02138, USA  Yariv Yanay  Laboratory for Physical Sciences, 8050 Greenmead
Dr., College Park, MD 20740, USA  Charles Tahan  Laboratory for Physical
Sciences, 8050 Greenmead Dr., College Park, MD 20740, USA  Brian Swingle
Department of Physics, Brandeis University, Waltham, Massachusetts 02453, USA

(May 11, 2023)

###### Abstract

The recently introduced Quantum Lego framework provides a powerful method for
generating complex quantum error correcting codes (QECCs) out of simple ones.
We gamify this process and unlock a new avenue for code design and discovery
using reinforcement learning (RL). One benefit of RL is that we can specify
arbitrary properties of the code to be optimized. We train on two such
properties, maximizing the code distance, and minimizing the probability of
logical error under biased Pauli noise. For the first, we show that the
trained agent identifies ways to increase code distance beyond naive
concatenation, saturating the linear programming bound for CSS codes on 13
qubits. With a learning objective to minimize the logical error probability
under biased Pauli noise, we find the best known CSS code at this task for
$\lesssim 20$ qubits. Compared to other (locally deformed) CSS codes,
including Surface, XZZX, and 2D Color codes, our $[[17,1,3]]$ code
construction actually has lower adversarial distance, yet better protects the
logical information, highlighting the importance of QECC desiderata. Lastly,
we comment on how this RL framework can be used in conjunction with physical
quantum devices to tailor a code without explicit characterization of the
noise model.

##  I Introduction

Quantum error correction is a critical step in our quest for fully fault-
tolerant quantum computation. Beyond its practical significance in quantum
computing, its theoretical aspect also has significant impact in many areas of
physics, including condensed matter theory [1] and quantum gravity [2, 3].
With the recent advances in quantum hardware, the practical and theoretical
impacts of quantum error correcting codes (QECC) are rapidly converging on
various experimental platforms [4, 5, 6, 7, 8, 9]. Given this broad and
rapidly developing set of applications, no one-size-fits-all approach can
possibly be sufficient. Instead, it is highly desirable to devise and
implement flexible and customizable methods for code design which can be
tailored to the special needs of each particular application [10, 11, 12, 13,
14]. Here we propose and demonstrate a new approach to custom code generation
based on reinforcement learning and the ‘Quantum Lego’ paradigm.

Figure 1: (a) QL game overview. The game starts off with a single copy of the
T6 lego. The agent can choose from three actions, adding another T6 lego,
choosing legs to contract, or terminating the game. As part of reward shaping,
we give a small positive reward for adding another lego, performing a
contraction or ending the game. When the contraction connects legs that were
previously connected in the tensor network, the agent receives a small
negative reward. Code properties are calculated once the game has ended. (b)
We show the cumulative rewards the agent earns as it plays the game. A
positive final reward (Eq. 3) corresponds to outperforming the Color code at
protecting against biased noise. Arrows denote repesentative codes generated
at different time steps.

The Quantum Lego (QL) framework, introduced in Ref. 15 and relatedly [16],
provides a systematic method of constructing large quantum error correcting
codes in a modular fashion by gluing smaller codes, generalizing code
concatenation. As the name implies, these building blocks, or ‘legos’, can be
used to create a wide variety of interesting codes. Furthermore, Ref. 15 shows
that the framework is universal such that any quantum code can in principle be
obtained from just a few types of legos. The landscape reachable by just one
type of lego is less clear, but it has been demonstrated through examples that
one can construct a variety of non-trivial codes even with this restriction.
For example, the rich behavior exhibited by the toric code and the holographic
(HaPPY) code can emerge from many copies of the same underlying lego.

However, beyond code concatenation, there is no known strategy for building up
a larger code from the smaller blocks that guarantees an improvement of
certain code properties. Previously, this was done either manually, guided by
intuition, or through random or exhaustive search [15, 17], using tensor
weight enumerator polynomials. In view of the need for codes tailored to
particular applications, such manual or exhaustive search strategies do not
provide a scalable approach to code design.

Here, we show that it is possible to gamify the code-building process based on
Quantum Legos and teach a machine to construct useful quantum codes. We use a
reinforcement learning approach in which an agent manipulates the lego blocks
according to simple rules with the goal of maximizing an objective function.
The tunability of the objective function makes this approach powerful and
versatile. For example, by training on the objective of protecting against
skewed noise we were able to find a code that is more robust than any other
known code we have compared to. Even stronger results may be obtained in the
future from a hybrid approach using interaction with a quantum device to
define and calculate the objective function.

We consider reinforcement learning (RL) because it has already achieved beyond
human level performance in a variety of areas, including Chess, Go, Poker, and
Atari games [18, 19, 20, 21]. In order to learn these strategies, all that is
required is lots of experience, that is, the ability to play the game many
times over and over. Within physics, RL has been applied to various problems
including the knot problem, the string landscape, quantum circuit compilation,
and quantum optimal control [22, 23, 24, 25]. RL approaches have also played
an important role in optimizing code parameters, for example, improving
surface code performance by modifying a base code using lattice surgery [10]
and exploring the learning of error models and decoders [26].

Our work distinguishes itself from these previous manuscripts in that the goal
of learning is to establish a concrete framework that gamifies the QL code
building process. By applying RL techniques within this framework, we hope to
a) identify moves that generally improve the code properties, b) search for
hardware-specific codes given the noise model or access to the quantum device,
and c) find new codes with a flexible graph architecture as opposed to a fixed
one. We demonstrate the capabilities of this framework by applying it to two
tasks. The first task involves finding QL codes that maximize the code
distance, of which naïve concatenation is an example. Our agent finds a code
that achieves the same distance as concatenation with significantly fewer
qubits required. Furthermore, it is actually an optimal CSS code with respect
to linear programming (LP) bounds [27, 28]. The second task is hardware
motivated, where the goal is to minimize the probability of a logical error
occurring under a biased noise model. Here, we outperform the best known CSS
codes for up to 20 qubits.

The outline of the paper is as follows. In Sec. II, we provide background on
both Quantum Legos and reinforcement learning. Sec. II.1 focuses on the
modular construction of lego codes and their graphical interpretation as a
tensor network. In Sec. II.2, we present how to frame QL code building as a
game to be played by an RL agent. In Sec. III, we apply the QL code building
framework to two different tasks. We showcase the agent’s ability to produce
optimal distance CSS codes in Sec. III.1. In Sec. III.2, we task the agent
with building robust codes under a particular, though not fine-tuned, biased
noise model. We achieve state of the art results for CSS codes of comparable
size. Finally, we conclude with some remarks and future extensions in Sec. IV.

##  II Methods

In this section, we describe how to turn Quantum Legos into a game that is
amenable to reinforcement learning methods. First, we briefly review how the
QL framework can be employed to build new codes in a modular fashion from
existing codes. A salient feature is that the design of these codes has a
graphical description based on tensor networks. For a more in-depth review of
the formalism, see App. A and [15]. Then, we turn the problem of learning new
codes into a game with a discrete set of actions and states which can be
understood by reinforcement learning algorithms. While the rules of the game
are fixed, the final reward can be chosen based on desired properties of the
resulting code. We will exploit this freedom in Sec. III to learn codes
tailored to different objectives.

###  II.1 Quantum Legos

Figure 2: Channel state duality. The encoding map $V$ of a QECC, taking
logical qubits to their physical counterparts, can be interpreted as a state
on all qubits. The tensor describing the state $|\psi_{V}\rangle$ is simply
given by the matrix elements of $V$. When $V$ is the encoding map for the
$[[4,2,2]]$ code with stabilizer generators $\langle XXXX,ZZZZ\rangle$, we
call this tensor the T6 lego. Note the ambiguity in going from a tensor to a
QECC that comes from assigning each of the legs to be a logical or physical
qubit. Because of this ambiguity, for example, we could interpret
$|\psi_{V}\rangle$ as either a $[[4,2,2]]$ code or a $[[5,1,2]]$ code.

Quantum Legos is a modular framework that consists of three primary elements —
the lego set $\mathcal{L}$, connection of the legos in the set, and assignment
of logical legs.

Any lego $\ell\in\mathcal{L}$ is a tensor that can be obtained from the
encoding map of a (generally small) QECC. These legos also have a number of
symmetries which we can track in addition to their tensor components. For
example, one such lego is shown in Fig. 2, which graphically represents the
tensor $\mathcal{V}_{i_{1}i_{2}i_{3}i_{4}i_{5}i_{6}}$ such that each leg is
mapped to a tensor index. The tensor components are obtained from the encoding
isometry $V:\mathbb{C}^{4}\rightarrow\mathbb{C}^{16}$ of a $[[4,2,2]]$ self-
dual CSS code. Its stabilizer group is generated by $\langle
XXXX,ZZZZ\rangle$. The logical $X$ and $Z$ operators are given by weight-2 $X$
and $Z$ operators respectively. In general, when $\ell$ is derived from a
stabilizer code, its tensor components are fully fixed by the symmetries it
inherits from the parent stabilizer code. Therefore it suffices to track only
the symmetries of the lego blocks, instead of its tensor components.111The
lego in this example have symmetries represented graphically in Fig. 17 of
[15].

To build up larger codes, we can connect the legos by gluing their legs (Fig.
3). This graphical operation corresponds to a tensor contraction where one
sums over the indices that correspond to the legs to be glued.222Physically,
the tensor contraction can also be mapped to a Bell fusion [29]. The combined
tensor network inherits all symmetries of the individual legos that satisfy
the additional constraints imposed by the fused legs. The transformation of
the symmetries under such gluing operations is captured graphically as
operator pushing or operator matching (App A). Ref. 15 showed that these
symmetries fully define the newly formed tensor network, which can then be
converted into a quantum code.333For actual implementation, these graphical
moves are represented as operations over the check matrices of the individual
lego blocks which only involve simple row and column operations and are thus
efficiently implementable. The resulting symmetries of the tensor network over
$n$ dangling legs can be encoded in a larger check matrix of size $n\times
2n$.

Finally, to convert the tensor network into a code, we choose some legs as
logical qubits (inputs) and the remaining as physical (outputs). Such a tensor
network then acts naturally as an encoding map. Note that this conversion
process is not unique as the tensor (network) corresponds to a class of codes
related to each other by the Choi-Jamiolkowski isomorphism. For example,
although the tensor in Fig. 2 represents a $[[4,2,2]]$ code by construction,
it can also be used to define a $[[6,0,3]]$ or a $[[5,1,2]]$ CSS code by
choosing the logical legs differently. In later sections, we fix the choice of
logical leg to resolve this ambiguity.

The formalism also furnishes any QL code with a distance verification protocol
and an optimal decoder [16, 30] under i.i.d. single qubit error channels [30].
We do not consider active error correction in this work, but we make use of
the quantum weight enumerators made available by [31, 32, 17, 30] to compute
code distance and to estimate logical error probabilities. When $\mathcal{L}$
consists of only legos from stabilizer codes, the tensor contractions and
symmetries are also efficiently trackable via operations on their respective
binary matrix representations as shown in Appendix D of Ref. [15]. This
determines the stabilizer generators (or check matrix) of the resulting code,
from which its unitary encoding circuit can be obtained [33]. We review the
basics of QL and illustrate several simple code constructions in Appendix A.
For more details, see the original work [17].

In the following, we focus on the class of codes that can be constructed from
a single type of lego block derived from the self-dual CSS $[[4,2,2]]$ code.
We call it a T6 lego and any code constructed using only such blocks a T6 QL
code. Despite this restriction, T6 QL codes contain many types of interesting
codes including the toric code, surface code, and trivalent 2D Color codes.

###  II.2 Gamifying Quantum Legos

Reinforcement Learning (RL) is a subfield of machine learning that learns from
experience, e.g. playing a million games of chess, rather than labeled data
sets, e.g. image classification. With the goal of using machine learning to
help with code design, we set out to reframe lego building as a game to be
played. To build a game, one needs to specify a set of states $S$ (such as the
set of board configurations in chess) and the set of actions $A$
(corresponding to moves to be made). The process of building a QL code can be
made atomic with the following sets $S$, $A$.

Quantum Lego Game Given a set of legos $\mathcal{L}$, a state $s\in S$
consists of two pieces of information 1. A list of $n$ legos in use
$(\ell_{1},\ldots,\ell_{n}),\ell_{i}\in\mathcal{L}$. 2. A list of $m$ edges
$((l_{a_{1}},l_{b_{1}}),\ldots,(l_{a_{m}},l_{b_{m}}))$. Each edge contains a
pair of legs $(l_{a_{j}},l_{b_{j}})$ to be contracted. From a state $s$, there
are four possible actions $a\in A$. 1. Add a new lego $\ell_{n+1}$. 2.
Contract two dangling legs $(l_{a_{m+1}},l_{b_{m+1}})$. 3. Designate a leg as
logical. 4. Terminate.

Every time a lego $\ell$ is added, its legs are sequentially numbered. For
example, a T6 lego has 6 legs. The first T6 lego would have legs numbered 1
through 6. Once a second lego is added, any two indices between 1 and 12 would
specify a valid contraction. The index pair $(a_{j},b_{j})$ labels the two
tensor legs that are connected to form an edge $j$ in the tensor network.

These simple rules are enough to generate all quantum codes given a universal
lego set. In principle, this game can be played ad infinitum. In practice, we
place a few constraints to make the state space finite for computational
purposes. For example, we limit the total number of lego blocks and the number
of moves the agent can make. The former also constrains the maximum size of
the code that can be produced. In this work, we will also fix the choice of
logical leg to be the first leg of the first lego in order to streamline the
learning process. This leaves us with three actions, adding a new lego to our
tensor network, contracting legs in the tensor network, or ending the game.
See Fig. 1 for a graphical summary.

When the game is terminated, the final reward is calculated based on the
resulting tensor network. In general, the reward could be tied to any code
property, such as the distance, the encoding rate, the check weights, etc. We
emphasize that the training objective for the agent is arbitrary. This
flexibility is a useful feature: Any desirable code property that can be
quantified can be optimized by the learning algorithm. In this work, we will
consider adversarial distance and the probability of a logical error under a
biased noise model. For stabilizer codes, this amounts to looking at the ($X$
and $Z$) weights of logical operators as in Eqs. (1) and (2).

We note that ideally the reward should be efficiently calculable, as this
calculation will need to be repeated for every iteration of the game, which
the agent needs to be able to play many times. When it comes to quantum codes,
this can be a significant limitation, as properties of a quantum system are
often exponentially difficult to calculate. This hurdle can be surmounted by
the use of hybrid algorithms [14], where a classical learner has accesses to a
quantum machine for the purpose of calculating the objective function. In
particular, our game could be paired with a quantum computer that performs the
actual encoding and decoding functions of the QECC and measures their
robustness to induced noise – or simply to the real noise seen by the machine.

Another common pitfall in RL is the problem of sparse rewards. Much like in a
game of chess, a reward that only comes at the end of the long game can make
it difficult for the agent to associate good intermediate actions with the
final positive rewards. To counteract this, one can provide small rewards to
guide the agent, an idea known as _reward shaping_. In our game, we assign
small positive rewards to adding a new tensor ($r_{\text{add tensor}}=.1$), to
contracting tensors which were not previously connected in the network
($r_{\text{contraction}}=.05$), and to ending the game
($r_{\text{terminate}}=.15$). Conversely, we assign a small penalty when the
agent contracts legs of tensors which are already connected ($r_{\text{self-
contraction}}=-.1$) (see Fig. 1). Since the distance of a code takes integer
values, these rewards were chosen so that taking multiple actions would still
be a secondary effect.

When trying to maximize the code distance, these heuristics are important
since the majority of moves (self-contractions) when starting with a single T6
lego lead to decreasing code distance. Without reward shaping, the agent will
simply maximize its expected reward by avoiding these bad moves by choosing to
terminate the game early. Thus, reward shaping can be viewed as coercing the
agent to continue playing the distance maximization game. When designing codes
for the biased noise model, we actually found the same reward shaping
structure to work well.

Contrasting with chess, we point out two aspects of our setup that differ. The
first is that “winning” is less well-defined. The agent simply tries to get
the best score it can, but empirical results can’t tell us about whether the
agent attains the best possible outcome. The second aspect is that unlike
chess where an agent learns to play against an opponent who may take a variety
of actions, the code game is deterministic. Thus, the agent only needs to
demonstrate a single instance of a valid code to be useful in practical
applications.

##  III Results

Here, we present the results of our framework applied to two different
objectives for quantum codes:

  1. 1.

Maximizing the code distance,

  2. 2.

Minimizing the probability of a logical error under an i.i.d. qubit biased
noise model.

To carry out the numerics, we utilize standard RL libraries available in
Python. In particular, we use the Gym [34] framework for setting up the lego
game and Stable-Baselines3 [35] for their implementation of the Masked
Proximal Policy Optimization (PPO) learning algorithm [36, 37]. For the
curious reader interested in RL, see App. B for a brief overview aimed at
physicists.

In both tasks, our learning algorithm produces state of the art codes. In the
case of distance maximization, our code saturates LP bound for CSS codes and
requires a significantly fewer number of qubits to achieve the same distance
produced by naïve concatenation. For the protection of logical information
under biased noise, we find a novel code that outperforms popular CSS codes of
comparable size.

###  III.1 Maximizing code distance

As a first demonstration of the power of learning to play with legos, we start
by training an agent to maximize the code distance given access to copies of
the T6 lego. To fix the ambiguity of going from a tensor network to a code, we
will always take the first qubit of the first lego to be the only logical leg.
As a baseline, we compare to naïve code concatenation, requiring five copies
of the base code, that yields a distance $[[21,1,4]]$ code (Fig. 3). This code
can clearly be constructed as a T6 QL code. Within our QL game approach, we
find a $[[13,1,4]]$ code that requires significantly fewer physical qubits to
implement.

Figure 3: An example of a distance $d=4$ code found in the QL game where the
task was to maximize the distance of a single logical qubit. The base distance
of the individual T6 legos was two. On the left, we show a $[[13,1,4]]$ code
that achieves the same distance as the naive concatenation strategy with
significantly fewer qubits. For both codes, the red star labels the logical
leg.

We also note that this $[[13,1,4]]$ code saturates the upper bound on distance
for CSS codes of this size $n$ and number of logical qubits $k$, making it an
optimal code.444For $k=1$, the smallest CSS code with $d=4$ requires $n=12$
qubits. In the current era of NISQ devices, finding codes which use fewer
physical qubits can be greatly beneficial. Note that the encoding rate was not
explicitly part of the game’s reward, though one is free to include this as
part of the objective.

###  III.2 Protecting against biased noise

To highlight the flexibility of RL methods for code design, we now turn our
attention to a more physically relevant noise model. Given that not all noise
is created equal, designing codes that are tailored to particular shortcomings
of a device may accelerate reaching fault tolerant computation. A simplified
setup where this occurs is if Pauli $Z$ errors are more likely than $X$
errors.

For such a biased noise model, we use the $[[19,1,5]]$ trivalent 2D Color code
(Fig. 4a) as a benchmark since it can also be constructed as an T6 QL code
[30], though its construction requires lego moves that we don’t consider in
this work. Strikingly, we find a novel $[[17,1,3/4]]$ QL code with $X$ and $Z$
distances $d_{X}=3,d_{Z}=4$ that outperforms the 2D Color code, an optimal
$[[17,1,5/5]]$ CSS code [38, 39], (and other surface code variants) at
minimizing the probability of a logical error. We say a code is optimal if its
parameters saturate the linear programming bound. Surprisingly, the
adversarial distance for our T6 QL code is actually lower, meaning that the
minimal weight of logical operators is less, than in the comparison codes.
Nevertheless, the QL code has many fewer low-weight logical operators (Table
III.2) which results in improved robustness. Note that the lower error rate of
the T6 QL code is not a fine-tuned artifact but is robust against perturbation
of physical noise rates (Appendix E).

—c—c—c—c—[colortbl-like] Code $[[n,k,d_{X}/d_{Z}]]$
$\begin{array}[]{c}p_{L}\\\ (10^{-5})\end{array}$
$\begin{array}[]{c}p^{\text{norm}}_{L}\\\ (10^{-5})\end{array}$
T6 BN 13A $[[13,1,3/4]]$ $.973$ $2.16$
T6 BN 13B $[[13,1,3/4]]$ $.973$ $2.16$
CSS Self-Dual† $[[13,1,3/3]]$ $26.8$ $59.5$
T6 DM 13† $[[13,1,4/4]]$ $5.68$ $12.6$
Reed-Muller $[[15,1,3/7]]$ $1.43$ $3.61$
Surface (4x4) $[[16,1,4/4]]$ $1.46$ $3.89$
XZZX (4x4) $[[16,1,4/4]]$ $1.07$ $2.86$
T6 BN 17 $[[17,1,3/4]]$ .404 1.15
CSS self-dual† [[17, 1, 5/5]] .726 2.06
2D Color $[[19,1,5/5]]$ $.456$ $1.46$
XZZX (4x5) $[[20,1,4/4]]$ $.665$ $2.26$
XZZX (5x4) $[[20,1,4/4]]$ $.665$ $2.26$
Surface (4x5) $[[20,1,4/5]]$ $.438$ $1.48$
Surface (5x4) $[[20,1,5/4]]$ $6.58$ $22.2$

Table 1: Error rates for best known CSS codes when $p_{x}=0.01,p_{z}=0.05$. BN
and DM refer to the tasks that the agent was trained on, biased noise and
distance maximization. Under our biased noise model, we compute the normalized
an unnormalized logical error rates for a variety of codes. For the learning
procedure, we asked the agent to minimize the normalized error rate and it
outperformed all other CSS codes of similar size. Codes marked with † denote
codes that are optimal with respect to linear programming bounds on the
distance.

—c—c—c—c—c—[colortbl-like] Code $[[n,k,d]]$ wt $3$ wt $4$ wt $5$
T6 BN 17 $[[17,1,3]]$ 4 12 32
2D Color code $[[19,1,5]]$ 0 0 108

Table 2: Distribution of logical operator sizes. Under i.i.d. qubit noise, low
weight logical operators are more likely than high weight operators to corrupt
the logical information. Comparing our lego code to the 2D Color code, we find
that our code has a lower minimal weight (e.g. smaller $d$), yet achieves an
overall lower probability of logical error.

The above observations suggest that code design for asymmetric noise is a
difficult problem to analyze by hand, yet for the algorithm it amounts to a
simple tweak of the reward function. Instead of the reward being tied to the
adversarial distance, we give the agent a positive reward for a lower logical
error probability. Let us describe precisely the quantity we compute.

In this example, we consider biased noise models with independent bit flip and
phase error probabilities $p_{x}$ and $p_{z}$, respectively. Note that these
T6 QL codes admit a number of decoders [30], such as the maximal likelihood
decoder [16], for which the corresponding error probabilities can be computed
or estimated. Although such decoders are easy to incorporate into the
framework, they are more time-consuming to implement. Instead, we use two
simplifying estimates for logical error probability. The first one is given by
the probability of an undetectable logical error $p_{L}$, which is the
probability that a non-trivial logical operation has been applied to the
encoded state due to bit flip and phase flip errors on individual qubits. In
other words, this is the lower bound for the logical error rate regardless of
the choice of decoder. We refer to it as the _unnormalized error rate_ , which
is related to error detection. A measure of similar properties is used in
[14]. Because bit flip and phase errors occur independently, for each non-
trivial logical Pauli operator $\bar{L}$ such that the $X$-weight
$wt_{X}(\bar{L})=w_{x}$ and $Z$-weight $wt_{Z}(\bar{L})=w_{z}$, the total
probability that it comes from a physical error is
$p(w_{x},w_{z})=p_{x}^{w_{x}}p_{z}^{w_{z}}(1-p_{x})^{n-w_{x}}(1-p_{z})^{n-w_{z}}$.
Therefore,

$p_{L}=\smashoperator[]{\sum_{w_{x},w_{z}=0}^{n}}C_{w_{x},w_{z}}p(w_{x},w_{z}),$
(1)

where $C_{w_{x},w_{z}}$ enumerates the number of non-identity logical
operators with the same $X$ and $Z$ weights.

For the second measure, we consider a detection-based decoder which resets the
system upon measuring a non-trivial syndrome. This is the logical error
probability given that one measures trivial error syndrome, meaning no
decoding needs to be applied. The corresponding logical error probability is

$p_{L}^{\text{norm}}=p_{L}/p_{s=0},$ (2)

where $p_{s=0}$ is the probability of measuring trivial syndromes (assuming no
measurement errors). We refer to the logical error rate obtained this way as
the _normalized error rate_. In training for this task, we keep the same game
as in the previous distance maximization game, but simply ask the agent to
minimize this normalized error rate.

Both measures can be computed exactly using the double quantum weight
enumerator polynomials [32]. Further details of these enumerators and how they
are used to compute the error probabilities are explained in App. C.

Figure 4:  (a) Lego code trained to minimize logical probability error
assuming biased noise $p_{x}=0.01$, $p_{z}=0.05$. The red star labels the
logical leg. The resulting code parameters are $[[17,1,3]]$. Compared with
other codes (Table III.2), this lego code actually has a shorter adversarial
distance, yet it has a lower probability for a logical error to occur. This
probability is calculated by looking at the $X$ and $Z$ weights of logical
operators as in Eqs. (1) and (2). (b) The $[[19,1,5]]$ trivalent 2D Color code
(right). It can be built from encoding tensors of the Steane code (left), each
of which is the contraction of two T6 legos. Physical legs or qubits are
marked with a solid dot whereas logical legs are suppressed.

For benchmarking purposes, we compare with the $[[19,1,5/5]]$ 2D Color code
and other existing codes of similar sizes, such as the XZZX code, surface
code, and Reed-Muller code (Table III.2). Since the 2D Color code provides the
lowest probability for a logical error and can be obtained as a T6 QL code, we
tie the final reward of the agent to its performance relative to the 2D Color
code

$r_{f}=-\log\left(\frac{p^{\text{norm}}_{L,{\rm
lego}}}{p^{\text{norm}}_{L,{\rm 2DColor}}}\right).$ (3)

Hence, the agent receives a positive reward when its error probability is
lower than that of the 2D Color code.

Remarkably, we find a $[[17,1,3/4]]$ code that outperforms these codes at both
the normalized and unnormalized error probabilities. We note that the
stabilizer weight distribution of the basic T6 lego building block is
symmetric with respect to $X$ and $Z$. However, the RL agent has identified
contractions that break this symmetry, producing a code that is tailored to
this biased noise problem. Notably, the learning procedure also yielded two T6
QL BN codes at $n=13$ with comparable performance. They are asymmetric
$[[13,1,3/4]]$ CSS codes which is near optimal [38]. These codes outperform
their counterparts at $n=13$ despite having a lower distance than some. The
reason is again related to the operator weight distribution. See Fig. 9 and
Appendix C for details. Interestingly, the two $n=13$ codes have distinct
scalar enumerators, indicating that they are not simply related by local
unitary operations or simple permutation of the physical qubits. However, they
have the same double weight enumerators, hence yielding the same logical error
probabilities under biased noise.

In Fig. 1, we plot the cumulative reward of the learning algorithm over time
for on this biased noise task. Intermediate rewards are given by the reward
shaping structure mentioned in Sec. II.2 and the final reward is given by Eq.
3. Without reward shaping, a positive reward corresponds to beating the color
code (and other surface code variants). For the agent shown in Fig. 1, the
maximum number of legos was 7 and the number of actions was limited to 20.

##  IV Discussion

Our work highlights a promising avenue for the discovery of new quantum codes.
The use of model-free RL algorithms underscores the power to design codes with
targeted purposes. Here we considered a simple biased error model, but one
could easily tailor the reward function to take into account hardware-specific
information. Note that our approach is not limited to the scale of qubit codes
we have here. With quantum legos, we merely keep track of the symmetries of
the tensor network, as opposed to working in the Hilbert space of all the
qubits. Thus, as long as there is an efficient contraction for the tensor
network representing the QL code, all steps in our framework can be carried
out efficiently.

###  IV.1 Towards hardware tailored codes

Ultimately, the best kind of QECC is one that is tailored [12, 14] to the
device it is run on. A number of approaches for doing this have been
considered and implemented for small scale devices of a few qubits. Here we
provide another approach based on the Quantum Lego game. The approach is more
scalable compared to the existing methods thanks to its modular architecture
and use of generalized concatenation, and can accommodate a greater variety of
codes as the game allows for a high degree of flexibility in the network
architecture. We give a road map towards integration with near-or-intermediate
term quantum hardware in the preliminary stage.

The reward function can often be difficult to evaluate when it is tied to
certain code properties. Depending on the type of code, this cost can scale
rapidly with system size. For sufficiently large systems (which may yet be
beyond our current hardware capabilities) it may be more advantageous to use
the quantum device to evaluate the cost function directly. Instead of
benchmarking our devices, determining the best-fit error models, then finding
a good code based on that model, our strategy in this case will be to obtain a
code tailored to the hardware while being oblivious to the error model.

Note that for each code that the agent finds (or starts with), we are given
its check matrix and the corresponding tensor network. Using this information,
one can prepare a particular encoded state on the quantum device. For a
circuit based preparation scheme, one can easily obtain a 2-local Clifford
encoding circuit from the check matrix with $O(n^{2}/\log(n))$ complexity
[40]. For a platform that is measurement or fusion based, we prepare the
individual lego blocks unitarily then perform Bell fusion to create the
requisite tensor network, followed by the standard post-measurement correction
protocols. This process is constant depth plus classical processing.

With an encoder circuit available, any number of objective functions can be
evaluated. An estimate of memory error can be obtained by simply waiting some
fixed amount of time, or single- and multi-qubit gate errors can be similarly
estimated by acting with the equivalent logical operators on the encoded
system. This can be followed by syndrome measurements, decoding and final read
out of the logical information. By repeating this process a large number of
times, we can estimate the logical error probability, which will determine the
reward we feedback to the agent.

A number of decoders can be used. The most trivial, but not very practical,
decoder would measure the syndromes, discard states with non-trivial values,
and measure the logical state of those with trivial syndromes to see if they
match the target state. This provides an estimate for the normalized error
probability discussed in Sec. III. Such a method is more feasible for a code
when the noise levels are low such that trivial syndromes occur with
exceedingly high probability. A more sophisticated decoder can also be
implemented using a model provided by [16, 30]. For instance, with an initial
guess of the error model, one can also use the maximum likelihood tensor
network decoder which works for all Quantum Lego codes. However, their output
will depend on benchmarks or guesses about the error model, which then needs
to updated to achieve true optimality. Perhaps a ML approach to finding a
decoding schedule in conjunction with the encoding scheme is ultimately needed
for best outcomes (e.g. [26, 41]), but we leave that for future work.

###  IV.2 Other Modifications

The framework can be easily modified to include other user requests and to
speed up the learning process/reduce learning cost. It can also be used in
conjunction with other existing methods. We comment on a few possibilities in
this section.

Decent Quantum LDPC codes: While we have focused on $k=1$ codes by maximizing
distance or minimizing logical error probability, encoding rate and stabilizer
check weights are also key deciding factors for the practicality of a quantum
code. Therefore, an obvious extension of the current model is to include rate
or check weights also into the cost function.

The former can be extended by choosing more than one logical leg. However, the
task of finding the right tensor legs as logical inputs can also be difficult
by brute force. The current game is played with a singular designated logical
input. This is a simpler, but more restricted, version of QL as it allows one
to identify multiple legs as logical inputs. To unlock the full potential of
the framework that allows the construction of known examples such as the
surface code or 2d Color code, one can incorporate the move of identifying a
tensor leg as logical input. Together with the moves we have in Fig 1, they
completely characterize the actions allowed by QL and is sufficient for
recreating all quantum codes.

Check weights can be trickier as the naive RREF of a check matrix does not
necessarily yield the optimal weight especially with spatially local
constraints. However, it can be a useful starting point until a better method
is found. Note that [15] identified moves in a case by case basis that can
keep the check weights low. A successful RL model in this extension has the
promise help us identify more general moves for reducing stabilizer check
weights while also increasing the code distance. Progress in this direction
would be a key result in identifying good moves beyond code concatenation.

Transfer learning: Access to a quantum device can be expensive and it can take
a large number of iterations before the agent converges on a good strategy.
While the above section describes a framework which works in principle, it is
far more realistic in practice to first train the agent classically with
respect to a known error model that approximates the behaviour of the real
quantum device. After a certain amount of training for the agent to acquire a
basic understanding of the game, we then continue the training where the cost
function is now provided by a real quantum device in a process like the one
described above. This allows the agent to make relatively minor policy updates
which can reduce the cost and runtime on the quantum computer while achieving
a similar outcome on error suppression.

Approximate cost functions: A major resource consuming task in our current
model lies in the choice of our cost functions. For both of the examples, we
computed distance or weight enumerators exactly. Because such a problem is NP-
hard, the general cost can be as expensive as $O(\exp(d))$ even with the
tensor network methods in [17, 30]. However, such level of accuracy may not be
needed for practical code designs. To that end, it is reasonable to substitute
the exact values of logical error probabilities with approximate values using
Monte Carlo methods or approximate tensor network contractions. For instance,
MPO-based approximate contraction schemes achieves $O(n\log n)$ compared to
$O(\exp(\sqrt{n}))$ for a planar tensor network [42] like the one for the
surface code [43]. For intermediate term systems with larger sizes, this can
be a practical approach.

Fixed architecture codes: In this work we have reduced the complexity by
connecting only one type of lego. However, this is ultimately a limiting
factor, as other blocks are needed to obtain other more flexible codes.
However, this can drastically expand the already vast state space and thus the
complexity for learning and search. A method to reduce the state space is to
consider a more rigid architecture instead of any graph with bounded degree
like the one we have here. For instance, we can insist that the agent must
connect blocks in manners that produces a planar graph or the native
connectivity required by the quantum hardware. On the most restrictive case,
we can fix the graph architecture completely and only allowing the agent to
choose different legos for each vertex. This is somewhat reminiscent of the
VQA-based strategy [14] but with discrete range of choices.

Compatibility with VQA based methods: For smaller sized devices, it is
possible to prepare individual legos with VQA [14] that are close to optimal.
These codes are generally not Pauli stabilizer codes. However, the same VQA-
based approach is not directly scalable. One way to circumvent this bottleneck
is to use code concatenation. We can further extend the scalability of this
framework by treating each small-sized VQA code as a lego. As the QL framework
remains valid for all quantum codes, one can then combine the VQA legos using
our RL setup. Furthermore, the tensor network is furnished with an efficient
method for computing the quantum weight enumerators [17, 30], which help
characterize the properties of non-stabilizer quantum codes. This is a unique
advantage of the QL based method, which unlike other current frameworks, can
easily accommodate non-stabilizer codes.

## Acknowledgement

We thank Markus Grassl for the helpful comments on linear programming bounds
for CSS codes and Robert Huang for his helpful suggestions. C.C. acknowledges
support from the U.S. Department of Defense and NIST through the Hartree
Postdoctoral Fellowship at QuICS and the National Science Foundation
(PHY-1733907). C.C. and B.G.S. acknowledge support from the Air Force Office
of Scientific Research (FA9550-19-1-0360). The Institute for Quantum
Information and Matter is an NSF Physics Frontiers Center. V.P.S. gratefully
acknowledges support by the NSF Graduate Research Fellowship Program under
Grant No. DGE 1752814, the DOE Office of Science under QuantISED Award DE-
SC0019380. H.Y.H. is grateful for the support from the Harvard Quantum
Initiative Fellowship.

## References

  * Kitaev [2003] A. Y. Kitaev, Fault-tolerant quantum computation by anyons, Annals of Physics 303, 2 (2003), arXiv:quant-ph/9707021 [quant-ph] .
  * Almheiri  _et al._ [2015] A. Almheiri, X. Dong, and D. Harlow, Bulk Locality and Quantum Error Correction in AdS/CFT, JHEP 04, 163, arXiv:1411.7041 [hep-th] .
  * Pastawski  _et al._ [2015] F. Pastawski, B. Yoshida, D. Harlow, and J. Preskill, Holographic quantum error-correcting codes: Toy models for the bulk/boundary correspondence, JHEP 06, 149, arXiv:1503.06237 [hep-th] .
  * Egan  _et al._ [2020] L. Egan, D. M. Debroy, C. Noel, A. Risinger, D. Zhu, D. Biswas, M. Newman, M. Li, K. R. Brown, M. Cetina, _et al._ , Fault-tolerant operation of a quantum error-correction code, arXiv preprint arXiv:2009.11482 (2020).
  * Cai  _et al._ [2021] W. Cai, Y. Ma, W. Wang, C.-L. Zou, and L. Sun, Bosonic quantum error correction codes in superconducting quantum circuits, Fundamental Research 1, 50 (2021).
  * Egan  _et al._ [2021] L. Egan, D. M. Debroy, C. Noel, A. Risinger, D. Zhu, D. Biswas, M. Newman, M. Li, K. R. Brown, M. Cetina, and C. Monroe, Fault-Tolerant Operation of a Quantum Error-Correction Code (2021).
  * Krinner  _et al._ [2022] S. Krinner, N. Lacroix, A. Remm, A. Di Paolo, E. Genois, C. Leroux, C. Hellings, S. Lazar, F. Swiadek, J. Herrmann, G. J. Norris, C. K. Andersen, M. Müller, A. Blais, C. Eichler, and A. Wallraff, Realizing repeated quantum error correction in a distance-three surface code, Nature 605, 669 (2022).
  * Xue  _et al._ [2022] X. Xue, M. Russ, N. Samkharadze, B. Undseth, A. Sammak, G. Scappucci, and L. M. K. Vandersypen, Quantum logic with spin qubits crossing the surface code threshold, Nature 601, 10.1038/s41586-021-04273-w (2022).
  * Bluvstein  _et al._ [2022] D. Bluvstein, H. Levine, G. Semeghini, T. T. Wang, S. Ebadi, M. Kalinowski, A. Keesling, N. Maskara, H. Pichler, M. Greiner, V. Vuletić, and M. D. Lukin, A quantum processor based on coherent transport of entangled atom arrays, Nature 604, 451 (2022).
  * Poulsen Nautrup  _et al._ [2018] H. Poulsen Nautrup, N. Delfosse, V. Dunjko, H. J. Briegel, and N. Friis, Optimizing Quantum Error Correction Codes with Reinforcement Learning, arXiv e-prints , arXiv:1812.08451 (2018), arXiv:1812.08451 [quant-ph] .
  * Bonilla Ataides  _et al._ [2021] J. P. Bonilla Ataides, D. K. Tuckett, S. D. Bartlett, S. T. Flammia, and B. J. Brown, The XZZX surface code, Nature Communications 12, 2172 (2021), arXiv:2009.07851 [quant-ph] .
  * Dua  _et al._ [2022] A. Dua, A. Kubica, L. Jiang, S. T. Flammia, and M. J. Gullans, Clifford-deformed Surface Codes, arXiv e-prints , arXiv:2201.07802 (2022), arXiv:2201.07802 [quant-ph] .
  * Roffe  _et al._ [2022] J. Roffe, L. Z. Cohen, A. O. Quintavalle, D. Chandra, and E. T. Campbell, Bias-tailored quantum LDPC codes, arXiv e-prints , arXiv:2202.01702 (2022), arXiv:2202.01702 [quant-ph] .
  * Cao  _et al._ [2022] C. Cao, C. Zhang, Z. Wu, M. Grassl, and B. Zeng, Quantum variational learning for quantum error-correcting codes, Quantum 6, 828 (2022), arXiv:2204.03560 [quant-ph] .
  * Cao and Lackey [2022] C. Cao and B. Lackey, Quantum Lego: Building Quantum Error Correction Codes from Tensor Networks, PRX Quantum 3, 020332 (2022), arXiv:2109.08158 [quant-ph] .
  * Farrelly  _et al._ [2022] T. Farrelly, D. K. Tuckett, and T. M. Stace, Local tensor-network codes, New Journal of Physics 24, 043015 (2022), arXiv:2109.11996 [quant-ph] .
  * Cao and Lackey [2022] C. Cao and B. Lackey, Quantum weight enumerators and tensor networks, arXiv e-prints , arXiv:2211.02756 (2022), arXiv:2211.02756 [quant-ph] .
  * Silver  _et al._ [2018] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, Science 362, 1140 (2018), https://www.science.org/doi/pdf/10.1126/science.aar6404 .
  * Silver  _et al._ [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, Mastering the game of go without human knowledge, Nature 550, 354 (2017).
  * Mnih  _et al._ [2015] V. Mnih _et al._ , Human-level control through deep reinforcement learning, Nature 518, 529 (2015).
  * Brown and Sandholm [2019] N. Brown and T. Sandholm, Superhuman ai for multiplayer poker, Science 365, 885 (2019), https://www.science.org/doi/pdf/10.1126/science.aay2400 .
  * Gukov  _et al._ [2021] S. Gukov, J. Halverson, F. Ruehle, and P. Sułkowski, Learning to Unknot, Mach. Learn. Sci. Tech. 2, 025035 (2021), arXiv:2010.16263 [math.GT] .
  * Carifio  _et al._ [2017] J. Carifio, J. Halverson, D. Krioukov, and B. D. Nelson, Machine Learning in the String Landscape, JHEP 09, 157, arXiv:1707.00655 [hep-th] .
  * Fösel  _et al._ [2021] T. Fösel, M. Y. Niu, F. Marquardt, and L. Li, Quantum circuit optimization with deep reinforcement learning, arXiv preprint arXiv:2103.07585 (2021).
  * Sivak  _et al._ [2022] V. V. Sivak, A. Eickbusch, H. Liu, B. Royer, I. Tsioutsios, and M. H. Devoret, Model-free quantum control with reinforcement learning, Phys. Rev. X 12, 011059 (2022).
  * Sweke  _et al._ [2018] R. Sweke, M. S. Kesselring, E. P. L. van Nieuwenburg, and J. Eisert, Reinforcement Learning Decoders for Fault-Tolerant Quantum Computation, arXiv e-prints , arXiv:1810.07207 (2018), arXiv:1810.07207 [quant-ph] .
  * Grassl [2001] M. Grassl, private communication, Online available at https://markus-grassl.de/QECC/ (2001), accessed on 2023-05-09.
  * Grassl [2002] M. Grassl,  _Fehlerkorrigierende Codes für Quantensysteme: Konstruktionen und Algorithmen_ , Ph.D. thesis, Karlsruhe Institute of Technology, Germany (2002).
  * Bartolucci  _et al._ [2021] S. Bartolucci, P. Birchall, H. Bombin, H. Cable, C. Dawson, M. Gimeno-Segovia, E. Johnston, K. Kieling, N. Nickerson, M. Pant, F. Pastawski, T. Rudolph, and C. Sparrow, Fusion-based quantum computation, arXiv e-prints , arXiv:2101.09310 (2021), arXiv:2101.09310 [quant-ph] .
  * [30] C. Cao, M. Gullans, and B. Lackey, Quantum Lego Expansion Pack, in preparation .
  * Shor and Laflamme [1996] P. Shor and R. Laflamme, Quantum MacWilliams Identities, arXiv e-prints , quant-ph/9610040 (1996), arXiv:quant-ph/9610040 [quant-ph] .
  * Hu  _et al._ [2020] C. Hu, S. Yang, and S. S.-T. Yau, Weight enumerators for nonbinary asymmetric quantum codes and their applications, Advances in Applied Mathematics 121, 102085 (2020).
  * Aaronson and Gottesman [2004] S. Aaronson and D. Gottesman, Improved simulation of stabilizer circuits, Phys. Rev. A 70, 052328 (2004), arXiv:quant-ph/0406196 [quant-ph] .
  * Brockman  _et al._ [2016] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, Openai gym (2016), arXiv:1606.01540 .
  * Raffin  _et al._ [2021] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, Stable-baselines3: Reliable reinforcement learning implementations, Journal of Machine Learning Research 22, 1 (2021).
  * Schulman  _et al._ [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms (2017).
  * Huang and Ontañ ón [2022] S. Huang and S. Ontañ ón, A closer look at invalid action masking in policy gradient algorithms, The International FLAIRS Conference Proceedings 35, 10.32473/flairs.v35i.130584 (2022).
  * Frederic Ezerman  _et al._ [2012] M. Frederic Ezerman, S. Jitman, S. Ling, and D. V. Pasechnik, CSS-like Constructions of Asymmetric Quantum Codes, arXiv e-prints , arXiv:1207.6512 (2012), arXiv:1207.6512 [cs.IT] .
  * Grassl [2007] M. Grassl, Bounds on the minimum distance of linear codes and quantum codes, Online available at http://www.codetables.de (2007), accessed on 2023-05-09.
  * Cleve and Gottesman [1997] R. Cleve and D. Gottesman, Efficient computations of encodings for quantum error correction, Physical Review A 56, 76 (1997).
  * Krastanov and Jiang [2017] S. Krastanov and L. Jiang, Deep Neural Network Probabilistic Decoder for Stabilizer Codes, Scientific Reports 7, 11003 (2017), arXiv:1705.09334 [quant-ph] .
  * Chubb [2021] C. T. Chubb, General tensor network decoding of 2D Pauli codes, arXiv e-prints , arXiv:2101.04125 (2021), arXiv:2101.04125 [quant-ph] .
  * Bravyi  _et al._ [2014] S. Bravyi, M. Suchara, and A. Vargo, Efficient algorithms for maximum likelihood decoding in the surface code, Phys. Rev. A 90, 032326 (2014), arXiv:1405.4883 [quant-ph] .
  * Agre and Rosenschein [1996] P. E. Agre and S. J. Rosenschein, Learning to act using real-time dynamic programming, in  _Computational Theories of Interaction and Agency_ (1996) pp. 81–138.
  * Silver [2015] D. Silver, Lectures on reinforcement learning (2015).
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto,  _Reinforcement Learning: An Introduction_ (A Bradford Book, Cambridge, MA, USA, 2018).
  * Qiskit contributors [2023] Qiskit contributors, Qiskit: An open-source framework for quantum computing (2023), see full code.

##  Appendix A Review of Quantum Lego Formalism

In this section, we review more of the formalism behind quantum legos. For
more details, please see [15]. As in the main text, our analysis will mostly
follow a single building block, called the T6 lego, that describes a
$[[4,2,2]]$ self-dual CSS code whose stabilizer group is $\langle
XXXX,ZZZZ\rangle$. The logical $X$ and $Z$ operators are given by weight-2 $X$
and $Z$ operators respectively. With the T6 lego, we illustrate the main
conceptual tools of Quantum Legos.

####  A.0.1 Channel-State Duality

The rules for combining these modular codes are inspired by tensor networks
and will have a graphical construction. To make this connection, we make use
of the channel state duality, wherein we can reinterpret the encoding map $V$,
from the two logical qubits to the four physical qubits,

$\mathcal{V}=\smashoperator[]{\sum_{i_{1},\dotsc,i_{6}}^{}}V_{i_{1}i_{2}i_{3}i_{4}i_{5}i_{6}}|i_{3},i_{4},i_{5},i_{6}\rangle\langle
i_{1},i_{2}|$ (4)

as a quantum state,

$|\psi_{V}\rangle=\smashoperator[]{\sum_{i_{1},\dotsc,i_{6}}^{}}V_{i_{1}i_{2}i_{3}i_{4}i_{5}i_{6}}|i_{1},i_{2},i_{3},i_{4},i_{5},i_{6}\rangle.$
(5)

Matrix elements of $V$ are simply the wave function coefficients for basis
elements on both input and output legs, which are now on equal footing. See
Fig. 2. Note that going from a state to a channel is not necessarily unique,
as there is a freedom in choosing which (and how many) of the legs to choose
as logical. For example, we could interpret the tensor arising from the
$[[4,2,2]]$ code as a $[[5,1,2]]$ code,

$\mathcal{V}^{\prime}=\smashoperator[]{\sum_{i_{1},\dotsc,i_{6}}^{}}V_{i_{1}i_{2}i_{3}i_{4}i_{5}i_{6}}|i_{2},i_{3},i_{4},i_{5},i_{6}\rangle\langle
i_{1}|.$ (6)

To standardize the notation, let us denote the input legs as $1$ and $2$, with
the output legs corresponding to $3,4,5,6$.

For stabilizer codes, the resulting state $|\psi_{V}\rangle$ is a stabilizer
state. The stabilizers of that state e.g.
${S:S|\psi_{V}\rangle=+1|\psi_{V}\rangle}$ are formed by taking
$O^{\rm(L)}\otimes O^{\rm(P)}$ where $O^{\rm(L)}$ denotes a logical Pauli
operator and $O^{\rm(P)}$ denotes the corresponding physical operator. To go
from the state back to a code, one can isolate a set of physical legs
$P^{\prime}$, and refactor $S=O^{\rm(L)}\otimes
O^{\rm(P)}=O^{\rm(L^{\prime})}\otimes O^{\rm(P^{\prime})}$. If $O$ is not a
Pauli operator, one should take extra care in this conversion by taking the
complex conjugate such that a stabilizer for the dual quantum state is given
by ${O^{\rm(L)}}^{*}\otimes O^{\rm(P)}$, but for Pauli matrices $O=O^{*}$ up
to global phase factors.

####  A.0.2 Operator Pushing

Ultimately, we would like to construct larger codes via tensor networks of
tensors constructed in the previous section. The notion of operator pushing is
a useful way to keep track of how errors propagate in a code. For a tensor
network code, one can find all stabilizer generators by performing operator
pushing on a network. Here, we illustrate operator pushing on a single tensor.

Consider the application of a Pauli $X$ operator on the first logical qubit,
$X^{\rm(L)}_{1}=X_{1}I_{2}$. In this code, it is physically implemented by
applying $X^{\rm(P)}_{1}=X_{3}X_{4}I_{5}I_{6}$,

$\mathcal{V}X^{\rm(L)}_{1}|i_{1}i_{2}\rangle=X^{\rm(P)}_{1}\mathcal{V}|i_{1}i_{2}\rangle.$
(7)

In the tensor network language, this can be thought of as pushing an operator
through the tensor, $X^{\rm(P)}_{1}\mathcal{V}=\mathcal{V}X^{\rm(L)}_{1}$, see
Fig. 5. Using the channel-state duality, we immediately find

$(I^{\rm(L)}\otimes X^{\rm(P)}_{1})|\psi_{V}\rangle=(X^{\rm(L)}_{1}\otimes
I^{\rm(P)})|\psi_{V}\rangle,$ (8)

making it clear that $|\psi_{V}\rangle$ lies in the +1 eigenspace of
$X^{\rm(L)}_{1}\otimes X^{\rm(P)}_{1}=X_{1}I_{2}X_{3}X_{4}I_{5}I_{6}$, making
it one of the code stabilizers.

To summarize, given $O^{\rm(L)}$ and $O^{\rm(P)}$ of a code with encoding $V$,
$O^{\rm(L)}\otimes O^{\rm(P)}$ yields a stabilizer of the resulting state
$|\psi_{V}\rangle$. This is a direct consequence of the channel-state duality.
Graphically, this allows us to perform the pushing, where any operator can be
acted on by $O^{\rm(L)}\otimes O^{\rm(P)}$ to yield an equivalent operator on
other legs.

Figure 5: Operator pushing. An operator $O^{\rm(L)}$ acting on a logical leg
has a corresponding implementation, $O^{\rm(P)}$ on the physical legs. In
terms of the tensor representation, one can visualize this as pushing $X_{1}$
“through” the tensor to an equivalent operator $X_{3}X_{4}$. The action of
both simultaneously $O^{\rm(L)}\otimes O^{\rm(P)}$ is a stabilizer of the
resulting state. For Stabilizer codes, there are many equivalent
representations of $O^{\rm(P)}$ related by the action of the stabilizer group.
In the diagram $X_{3}X_{4}\sim X_{5}X_{6}$ where $\sim$ denotes logical
equivalence due to the stabilizer $X_{3}X_{4}X_{5}X_{6}$.

####  A.0.3 Gluing Legos

Let us take two copies of this code, labeling the legs of $A$ as 1-6 and $B$
as 7-12 with the convention for numbering the same as before. Consider
contracting legs 3 and 9, which amounts to a Bell fusion, or a projection on
the maximally entangled state. We would like to understand the stabilizers for
the remaining ten qubit state. Suppose we take stabilizers of the individual
tensors $X_{1}^{\rm(L)}\otimes X_{1}^{\rm(P)}=X_{1}I_{2}X_{3}X_{4}I_{5}I_{6}$
and $X_{7}^{\rm(L)}\otimes X_{7}^{\rm(P)}=X_{7}I_{8}X_{9}X_{10}I_{11}I_{12}$,
then $X_{1}I_{2}X_{4}I_{5}I_{6}X_{7}I_{8}X_{10}I_{11}I_{12}$, would be a
stabilizer of the remaining qubits. Here we make use of the fact that
$X^{2}=I$. An alternate way to understand this construction of the resulting
stabilizer is to push $X_{3}$ “through” the other tensor. This is illustrated
in Fig. 6.

Figure 6: Single-trace operation via operator flow. Given two copies of T6, we
glue the legs of two tensors by projecting onto a maximally entangled state
(blue). The stabilizer of the resulting state can be found by performing
operator pushing. a) We start with $X_{1}^{\rm(L)}=X_{1}$ acting on the first
qubit. b) Pushing through the first tensor yields $X_{3}X_{4}$. c) Since
qubits 3 and 9 are entangled, $X_{3}\sim X_{9}$. d) Finally, pushing through
the second tensor, we get $X_{1}^{\rm(P)}=X_{4}X_{7}X_{10}$, so
$X_{1}^{\rm(L)}\otimes X_{1}^{\rm(L)}=X_{1}X_{4}X_{7}X_{10}$ is a stabilizer
of the resulting tensor network after contraction.

In constructing the full set of stabilizers, we simply perform this operation
for all operators acting on the glued legs, taking care to ensure operators
acting on the traced legs are matched. Although naively the total number of
stabilizer matchings is exponential in the system size, in the case of
stabilizer codes, there is an efficient algorithm to keep track of the glued
codes via a check matrix operation known as conjoining (Appendix D of [15]).

Note that any two legs may be glued together. When two legs of the same tensor
(more generally, a connected component) are glued, we refer to it as a self-
trace. Otherwise, we refer to the operation as a single-trace.

In general, [15] showed that tracing together stabilizer codes yield another
stabilizer code. If the individual legos are (self-dual) CSS codes, then up to
re-definitions via the Choi-Jamiolkowski isomorphism, so are the resulting
codes obtained from gluing their physical legs. Note that although $[[4,2,2]]$
is self-dual, the $[[5,1,2]],[[6,0,3]]$ codes from the same tensor are not,
because of weight-3 stabilizers. Furthermore, tracing of two non-self-dual
codes can be self-dual, e.g. the Steane code example in Fig. 7b. Therefore,
the codes we obtain in this work are CSS codes, but the self-duality property
need not be preserved during the game.

####  A.0.4 Examples of Code Constructions

Figure 7: Examples of code construction using Quantum Lego. Given just two
copies of the T6 lego, we already see a variety of behavior. (a) A single
contraction yields a code that encodes more physical qubits with a better
encoding rate, at the same distance as the original code. (b) Picking a single
logical leg after tracing two legs reproduces the well-studied [[7, 1, 3]]
Steane code. (c) Performing two traces on different choices of legs reduces to
a single copy of the [[4,2,2]]. This demonstrates the rich and unexpected
behavior that these simple contractions can yield.

With even two copies of the T6 lego, we can see how different lego codes can
arise. Performing a single trace between specific legs generates a $[[6,4,2]]$
code. Performing two gluing operations can lead to either the $[[7,1,3]]$
Steane code, or the original $[[4,2,2]]$ code after identifying gauge
constraints. The holographic HaPPY code can be constructed from many copies of
the perfect $[[5,1,3]]$ code. The perfect property of these codes allows for
easy operator pushing, which was part of the original inspiration. These
examples are obtained by contractions of isometric tensors. Although they are
different from naive code concatenation, they can be understood as a
concatenated code with suitable applications of the CJ-isomorphism. However,
the lego constructions also go beyond concatenation. A simple example is the
$[[5,1,3]]$ perfect code, which can be constructed with two copies of the T6
lego with 3 contractions if one allows for modifications by $H$ and $S$ gates
(Fig. 8).

Figure 8:  $[[5,1,3]]$ from two $[[4,2,2]]$ codes with $H$ and $S$ which are
the tensors for Hadamard and phase gates. The red star labels the logical leg.

Other constructions that are built beyond code concatenation include Bacon
Shor codes, the 2d Color code, and the Toric code, where one must contract
non-isometric tensors. Although it might appear restrictive to use only the T6
lego, it is worth mentioning that all examples above except the $[[5,1,3]]$
code can be built from the T6 legos alone.

##  Appendix B Reinforcement Learning Overview

Since reinforcement learning (RL) is not a part of the standard toolkit for
physicists, we summarize two learning algorithms that were used through this
work, real-time dynamic programming (RTDP) and Masked Proximal Policy
Optimization (MPPO). The details of the learning algorithms aren’t essential
for the paper, but this may provide a useful starting point for those wishing
to extend our results. Ultimately, we were able to produce all lego codes in
this paper with MPPO, but for pedagogical purposes, we include a discussion on
both. For a more in-depth treatment, see these original papers [44, 36, 37] as
well as some classic references on RL [45, 46].

There are two main quantities that play a distinguished role in learning
algorithms, the value function $V(s)$, which measures the expected reward of
continuing the game from a given state $s$, and the policy $\pi(a|s)$, which
characterizes the agent’s probabilities to take an action $a$ in a given state
$s$. Generically, the output of the learning algorithm will be a policy $\pi$
that maximizes the expected reward of playing the game.

####  B.0.1 Real-Time Dynamic Programming

Real-Time Dynamic Programming (RTDP) falls under the general class of
algorithms that tries to learn the value function $V(s)$. If one truly has
access to $V$, then it is simple to construct the optimal policy
$\pi^{*}(a|s)$ by simply selecting the action $a$ that leads to the highest
value successor state $s^{\prime}$. Under mild assumptions, $V(s)$ can be
iteratively solved by the Bellman equations, which state that the value of a
state is related to the value of states $s^{\prime}$ which are reachable from
$s$. This is akin to numerically solving Laplace’s equation by averaging
neighboring points.

$V^{(\pi)}(s)=\gamma\sum_{a}\pi(a|s)\left(r_{a}+\sum_{s^{\prime}}T^{a}_{ss^{\prime}}V^{(\pi)}(s^{\prime})\right)$
(9)

where $r_{a}$ is the reward for taking action $a$, $\gamma$ is a discount
factor, and $T^{a}_{ss^{\prime}}$ is the probability for the action $a$ to
take the agent from state $s$ to $s^{\prime}$. Note that the value of a state
depends on the relative probabilities of visiting neighboring states, hence
the dependence on $\pi$.

Due to the recursive structure of the Bellman equations, one can efficiently
(polynomial in the number of states) solve this problem via dynamic
programming. Once $V(s)$ is fully known, it is then easy to back out the
policy by simply selecting the action which leads to the successor state
$s^{\prime}$ with the highest value as follows

$\pi^{(\text{new})}(a|s)=\delta_{aa^{\prime}}\,,\,a^{\prime}\equiv\operatorname*{\arg\\!\max}_{a}r_{a}+\sum_{s^{\prime}}T^{a}_{ss^{\prime}}V^{(\pi)}(s^{\prime})$
(10)

In practice, since $V$ depends on the current policy $\pi$, one performs the
updates to $V$ and $\pi$ holding the other fixed. In general, there are strong
guarantees that this will converge to the optimal policy $\pi^{*}$ after
multiple rounds [46].

While this is conceptually clean and efficient, there is a problem for games
in which the number of possible states grows intractably large. The insight
behind RTDP is to perform a Bellman type update for only a subset of states
which actually appear when playing the game. The agent simply plays the game
according to its current policy $\pi$ and keeps track of what states it has
seen. Those states are the only ones updated by the Bellman iterations. Note
that there is incomplete information about the value function, so we lose the
theoretical guarantees. However, in practice, it was sufficient for finding
more efficient $d=4$ codes.

####  B.0.2 Masked Proximal Policy Optimization

The other general RL approach is known as policy learning, where one attempts
to model the function $\pi(a|s;\theta)$ directly with some parameters
$\theta$. Commonly, these are parameters for some neural network that model
the policy. In contrast to RTDP and other value-based methods, one is agnostic
about the inherent value of states and simply tries to tweak these parameters
to obtain a large expected reward. There are usually with auxilliary penalties
in the objective function to ensure training stability and convergence. The
algorithm we employ is the Masked Proximal Policy Optimization (MPPO), a form
of policy learning that was recently popularized for its effectiveness and
simplicity [36, 37].

Let us define the expected total reward under the policy $\pi_{\theta}$ as
$J(\theta)$.

$J(\theta)=\sum_{s}d^{(\pi_{\theta})}(s)V^{(\pi_{\theta})}(s)$ (11)

Here, $d^{(\pi_{\theta})}(s)$ is the stationary distribution of a Markov chain
with actions selected according to $\pi_{\theta}$. The stationary distribution
refers to the likelihood of being in state $s$ if the agent is allowed to play
the game infinitely long under the policy $\pi_{\theta}$. Thus, $J$ is a
weighted sum of the values of those states. This is a formal object that
depends on $\theta$ which we will not have access to directly since we are not
trying to learn the value function $V$. However, we can get estimates of $J$
by simply playing the game many times.

Since our goal is to maximize the expected returns, captured formally by $J$,
one may naturally be inclined to apply gradient ascent with respect to the
policy parameters $\theta$. However, there is a thorny conceptual issue.
Choosing the correct step size is difficult due to the implicit dependency of
both $d$ and $V$ on these parameters. Choose too small a rate and your agent
will fail to learn. On the other hand, if the step size is too large, the
actions may change drastically, raising convergence issues.

Proximal Policy Optimization (PPO) improves on this naive approach by
penalizing large changes in the policy (e.g. if
$r\equiv\frac{\pi_{\theta+\Delta\theta}(a|s)}{\pi_{\theta}(a|s)}$ deviates
significantly from $1$). Effectively, this amounts to a maximum tolerance for
the amount the policy can change. In prior works, one had a penalty for the
Kullback-Lieber divergence between $\pi_{\theta}$ and
$\pi_{\theta+\Delta\theta}$. However, the approach of PPO was found to be
empirically more effective. $\pi_{\theta}$ and $\pi_{\theta+\Delta\theta}$.
However, the approach of PPO was found to be empirically more effective. For
more details see [36].

Finally, the last tweak to the learning algorithm is to use action masking.
Most RL algorithms assume that the set of actions is the same for all times
(such as the four cardinal directions if playing Pacman). In more complex
games such as ours and chess, the set of valid actions is much larger and not
all of them will be applicable in a given state. For our lego game, the set of
actions includes the ability to contract any two legs. However, if one of the
legs is already contracted, this would be an invalid move. To get around this,
one could simply assign a negative reward for making such an improper move and
eventually the agent would learn not to try to contract legs that are already
contracted. However, it is more sample efficient simply to mask those invalid
moves upfront [37].

##  Appendix C Quantum weight enumerator polynomials

For each i.i.d. single qubit error, one can determine the undetectable logical
error probability from the weight enumerator polynomials of the code [30]. To
compute the logical error probabilities used in Sec. III under biased noise,
let us define the quantum double weight enumerators
$A(\mathbf{u}),B(\mathbf{u})$

$\displaystyle A(\mathbf{u})$
$\displaystyle=\frac{1}{K^{2}}\sum_{E\in\mathcal{P}^{n}}\operatorname{Tr}[E\Pi]\operatorname{Tr}[E^{\dagger}\Pi]\mathbf{u}^{f(E)}$
$\displaystyle B(\mathbf{u})$
$\displaystyle=\frac{1}{K}\sum_{E\in\mathcal{P}^{n}}\operatorname{Tr}[E\Pi
E^{\dagger}\Pi]\mathbf{u}^{f(E)},$

where $\Pi$ is the projection onto the code subspace, $\mathcal{P}^{n}$ is the
Pauli group over $n$ qubits, and $K$ is the dimension of the code subspace.
The variable $\mathbf{u}=(x,y,z,w)$ is a 4-tuple and
$\mathbf{u}^{f(E)}=x^{wt_{X}(E)}y^{n-wt_{X}(E)}z^{wt_{Z}(E)}w^{n-wt_{Z}(E)}$.
Respectively, $wt_{X},wt_{Z}$ are $X$ and $Z$ weights of the Pauli string $E$.
These polynomials are related by a MacWilliams identity

$A(x,y,z,w)=\frac{1}{K}B(w+z,w-z,\frac{x+y}{2},\frac{y-x}{2}).$ (12)

and contain important information of the code, such as its distance and error
detection probability. See [32] for details. Note the normalization convention
is different from that of [17].

We can rewrite the enumerators in the following form

$\displaystyle A(x,y,z,w)$
$\displaystyle=\sum_{w_{x},w_{z}=0}^{n}A_{w_{x},w_{z}}x^{w_{x}}y^{n-w_{x}}z^{w_{z}}w^{n-w_{z}}$
(13) $\displaystyle B(x,y,z,w)$
$\displaystyle=\sum_{w_{x},w_{z}=0}^{n}B_{w_{x},w_{z}}x^{w_{x}}y^{n-w_{x}}z^{w_{z}}w^{n-w_{z}}$
(14)

Let $\mathcal{E}_{w_{x},w_{z}}$ be the set of Pauli operators that have $X$
weight $w_{x}$ and $Z$ weight $w_{z}$. Then for a stabilizer code with
stabilizer $\mathcal{S}$, normalizer $\mathcal{N}(\mathcal{S})$,
$A_{w_{x},w_{z}}=|\mathcal{E}_{w_{x},w_{z}}\cap\mathcal{S}|$ and
$B_{w_{x},w_{z}}=|\mathcal{E}_{w_{x},w_{z}}\cap\mathcal{N}(\mathcal{S})|$,
i.e., up to a known normalization constant, they count the number of
stabilizer and normalizer elements with $X$ and $Z$ weights $w_{x},w_{z}$.
Therefore, the coefficient $C_{w_{x},w_{z}}$ in (1) can be written as
$C_{w_{x},w_{z}}=B_{w_{x},w_{z}}-A_{w_{x},w_{z}}$.

Figure 9: $X$ and $Z$ weight distributions of the non-trivial logical
operators for $n=13$ qubit codes. Left: $[[13,1,3]]$ optimal self-dual CSS
code. Middle: $[[13,1,4]]$ optimal CSS code. Right: $[[13,1,3/4]]$ T6 QL code.

Consider a biased i.i.d. single qubit error where bit flip errors and phase
errors occur independently with probabilities $p_{X}$ and $p_{Z}$. Then by
evaluating the polynomials at $\mathbf{u_{p}}=(p_{x},1-p_{x},p_{z},1-p_{z})$,
we see that

$p_{s=0}=B(\mathbf{u}_{p})$

returns the exact probability that the physical errors implements any logical
operation (including logical identity) and $A(\mathbf{u}_{p})$ computes the
probability that the error corresponds to a member of the stabilizer group,
which implements a logical identity. Assuming noiseless syndrome measurements,
it is clear that

$p_{L}=B(\mathbf{u}_{p})-A(\mathbf{u}_{p}).$

They are uncorrectable even in principle because such errors are undetectable.
This is the unnormalized logical error in the main text, which lower bounds
the logical error rate regardless of the choice of decoder. Similarly, the
normalized error probability is given by

$p_{L}^{norm}=p_{L}/p_{s=0}=1-\frac{A(\mathbf{u}_{p})}{B(\mathbf{u}_{p})}.$

This is the probability of non-trivial logical error given that one measures
trivial syndrome. Physically, this corresponds to a “decoding” process where
one resets the system whenever an error is detected.

Given a stabilizer group, one can always compute these polynomials by
enumerating all relevant stabilizer and normalizer elements. The current-best
algorithm for general stabilizer codes scales as $O(2^{n-k})$, where one can
enumerate all stabilizer group elements to obtain $A(\mathbf{u})$. One then
acquires its dual $B(\mathbf{u})$ using a MacWilliams transform which is at
most $O(n^{3})$. However, if a Quantum Lego construction is known, then [17,
30] allows for a more efficient algorithm that can provide up to exponential
speed-ups. As the codes we consider are relatively small in this work, it is
sufficient to use the brute force algorithm. For scalability, the tensor
network method [30, 17] may be used for exact error probability.
Alternatively, conventional Monte Carlo methods can also be used where one
would compute approximate, instead of exact, error probabilities.

The check matrices and codes for generating the relevant quantum weight
enumerators can be found in App. E. Here we also provide the Shor-Laflamme
enumerators for the codes listed in Table III.2, such that the coefficients of
$C(z)=B(z)-A(z)$ are precisely the weight distribution of the non-trivial
logical operators.

$\displaystyle C(z)_{[[17,1,3]]}=4z^{3}+12z^{4}+32z^{5}+144z^{6}+556z^{7}$
$\displaystyle+$ $\displaystyle
1876z^{8}+5296z^{9}+12640z^{10}+25020z^{11}+38292z^{12}$ $\displaystyle+$
$\displaystyle 43712z^{13}+36944z^{14}+22228z^{15}+8396z^{16}+1456z^{17}$

$\displaystyle C(z)_{[[19,1,5]]}=108z^{5}+765z^{7}+11406z^{9}+71523z^{11}$
$\displaystyle+$ $\displaystyle
252000z^{13}+321363z^{15}+120582z^{17}+8685z^{19}$

For $n=13$, the $[[13,1,3/4]]$ T6 QL code still outperforms other benchmarks
at the same size even though it does not beat the color code which requires
more physical qubits. One may note that its performance is better than the
$[[13,1,4]]$ code that is optimal with respect to the LP bound, which has a
higher adversarial distance. Even though all three such codes have similar
number of low weight non-trivial logical operators, their distributions (Fig.
9) are very different, with the QL code having fewer operators that have low
$Z$ weights.

##  Appendix D Calculating Objective Functions with Quantum Circuits

As mentioned in the main text, the versatility of the objective function
naturally lends itself to implementation with hybrid quantum algorithms. Here,
we give a simulated example, calculating undetectable logical error $p_{L}$
under biased noise, that was used in Section III.2.

A hybrid objective function would consists of several parts, summarized below:

Hybrid Objective Function Calculation Given a state of the game 1. Generate an
encoder circuit 2. Run one or more test circuits, of the form: (a) Prepare
some single-qubit state (b) Encode it into the $n$-qubit system (c) Apply some
set of gates to generate noise or errors (d) Measure stabilizers (e) Apply a
decoder circuit (f) Measure the final state 3. Calculate a cost function based
on measurement outputs 4. Return the result to the classical learner algorithm

The encoder circuit mentioned in the first step, which takes in one qubit in
an unencoded state and $n-1$ qubits in the $|0\rangle$ state and outputs the
encoded $n$-qubit state. This can be done either directly via measurements of
the stabilizers, or via a deterministic polynomial algorithm [40]. Another
approach, which we take here, is to use the building block T6 lego itself and
convert the various possible permutations of incoming and outgoing legs into
unitary circuits; this can be done whenever there are more outgoing than
incoming legs by adding $|0\rangle$-initialized qubits. These sub-circuits can
then be connected in the same way that the tensors were contracted by the
learner.

This encoder circuit can then be concatenated into a larger circuit in the
second step. The initialization of 2(a) can sweep over e.g. a set of Pauli
states or randomly rotated qubit states. Step 2(c) is where the strength of
the hybrid approach lies. By applying real operations, from idling the system
for a set time to applying logical $X$ or $Z$ gates, one can probe the true
noise parameters of the system, allowing the learner to optimize the code for
the actual hardware in use. Alternately, one quantum computer can be used to
generate codes for another by artificially simulating a known noise model.
These sort of calculations, that go beyond stabilizer states and Clifford
operations, are exponentially difficult to calculate with classical hardware
but remain polynomial with quantum devices.

Here, we give the results of a specific calculation set up to recreate the
results of Table III.2. We apply the steps above as follows:

  1. 1.

For the encoder, we use the approach described above, combining several
smaller circuits representing different input-output ordering of the H-Lego
tensor and connect them as per the contractions of the state.

  2. 2.

    1. (a)

To capture $X$ and $Z$ errors, we initialize the system in a state $|\pm
Y\rangle\otimes|0\rangle^{\otimes n-1}$, where ${\hat{Y}|\pm Y\rangle=\pm|\pm
Y\rangle}$.

    2. (c)

To recreate the noise model, we apply a single idle gate to each qubit, with
chance $p_{x}=0.01$ of an $X$ error and chance $p_{z}=0.05$ of a $Z$ error.

  3. 3.

We count the number of instances, $N_{s=0}$, where the stabilizers all
measured $+1$, including the number of instances where the original state was
measured to be different than the initial, $N_{L}$. We return the logical
error rate $p_{L}=N_{L}/N_{\rm shots}$ and normalized error rate $p_{L}^{\rm
norm}=N_{L}/N_{s=0}$.

The circuit diagram for this calculation is shown in Fig. 10. We perform the
simulation using Qiskit [47]. The results are summarized in Appendix D. We
note that we obtain numbers quite similar to those calculated analytically in
Section III.2. Thus the hybrid approach could be readily substituted into the
calculation of the cost function. While there is no advantage to doing so with
simulated circuits, the use of real devices would allow a dramatic speedup in
the calculation time and so the learning time. In general the required time
would scale polynomially in the number of qubits, and would be inversely
linear to the code error rate.

@C=.9em @R=0.1em @!R & (a) (b) (c) 2454.7em– (d) 2566.7em– (e) (f) 2859.7em–
—0⟩ ^U_0 3  Encoder I_n 3  Stabilizers 3  Decoder ^U_0^†
Encoder I_n  Stabilizers Decoder
—0⟩^⊗n-1  Encoder I_n  Stabilizers Decoder
Encoder I_n  Stabilizers Decoder
—0⟩^⊗n-1 [-1]

Figure 10: Circuit diagram for the hybrid calculation described in Appendix D.
$\hat{U}_{0}$ rotates the initial $|0\rangle$ state into $|\pm Y\rangle$. Each
$I_{\rm n}$ designates a noisy idle gate. Section (c) could be replaced by a
more general noisy circuit to generate other cost functions.

—c—c—c—c—c—c—[colortbl-like] Code $N_{L}$ $\begin{array}[]{c}N_{s=0}\\\
(10^{5})\end{array}$ $\begin{array}[]{c}N_{\rm shots}\\\ (10^{5})\end{array}$
$\begin{array}[]{c}p_{L}\\\ (10^{-5})\end{array}$
$\begin{array}[]{c}p^{\text{norm}}_{L}\\\ (10^{-5})\end{array}$
T6 BN 13A $13$ $4.44$ $10$ $1.3$ $2.9$
T6 DM 13A $24$ $1.78$ $4$ $6$ $13.5$

Table 3: Error rates for Lego CSS codes when ${p_{x}=0.01},{p_{z}=0.05}$,
calculated using Qiskit. Compare with the analytically calculated error rates
shown in Section III.2.

##  Appendix E RL Codes

Here we give the check matrices $H=H_{X}\oplus H_{Z}$ of the CSS codes made
from T6-legos. As usual, each row of the check matrix corresponds to a
stabilizer generator. Let $r_{P}^{i}\in\mathbb{F}_{2}$ be the $i$th component
of a row vector in $H_{P}$ where $P=X,Z$. Then its corresponding stabilizer
generator is $S=\bigotimes_{i=1}^{n}P^{r_{P}^{i}}$.

For the $[[13,1,4]]$ (DM 13) code by maximizing adversarial distance,

$\displaystyle H_{X}^{\text{DM 13}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccc}1&0&0&1&0&1&1&0&0&0&1&1&0\\\
0&1&0&1&0&1&0&1&0&0&0&0&0\\\ 0&0&1&1&0&0&1&1&0&0&1&1&0\\\
0&0&0&0&1&1&1&1&0&0&0&0&0\\\ 0&0&0&0&0&0&0&0&1&0&0&1&1\\\
0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right]$ $\displaystyle H_{Z}^{\text{DM
13}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccc}1&0&0&1&0&0&1&1&0&0&0&1&1\\\
0&1&0&1&0&1&1&0&0&0&0&0&0\\\ 0&0&1&1&0&1&0&1&0&0&0&1&1\\\
0&0&0&0&1&1&1&1&0&0&0&0&0\\\ 0&0&0&0&0&0&0&0&1&0&1&0&1\\\
0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right].$

For the $[[13,1,3]]$ (BN 13A) code found by minimizing normalized logical
error probability,

$\displaystyle H_{X}^{\text{BN 13A}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccc}1&0&0&0&1&0&1&1&0&0&1&0&1\\\
0&1&0&0&1&0&0&1&1&0&1&1&0\\\ 0&0&1&0&1&0&0&1&1&0&0&1&1\\\
0&0&0&1&1&0&1&0&1&0&0&0&0\\\ 0&0&0&0&0&1&1&1&1&0&0&0&0\\\
0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right]$ $\displaystyle H_{Z}^{\text{BN
13A}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccc}1&0&0&0&1&0&1&0&1&0&0&0&0\\\
0&1&0&0&1&0&1&0&1&0&0&1&1\\\ 0&0&1&0&1&0&1&0&1&0&1&0&1\\\
0&0&0&1&1&0&1&1&0&0&0&0&0\\\ 0&0&0&0&0&1&1&1&1&0&0&0&0\\\
0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right].$

For BN 13B code,

$\displaystyle H_{X}^{\text{BN 13B}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccc}1&0&0&0&1&0&0&0&0&0&1&0&1\\\
0&0&1&0&1&0&1&0&1&0&1&1&0\\\ 0&1&0&0&1&0&0&1&1&0&0&1&1\\\
0&0&0&1&1&0&1&1&0&0&0&1&1\\\ 0&0&0&0&0&1&1&1&1&0&0&0&0\\\
0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right]$ $\displaystyle H_{Z}^{\text{BN
13B}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccc}1&0&0&0&1&0&0&0&0&0&1&1&0\\\
0&1&0&0&1&0&0&0&0&0&1&0&1\\\ 0&0&1&0&1&0&1&0&1&0&1&0&1\\\
0&0&0&1&1&0&0&1&1&0&1&0&1\\\ 0&0&0&0&0&1&1&1&1&0&0&0&0\\\
0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right].$

Finally, the BN 17 $[[17,1,3]]$ asymmetric code that minimizes the same
logical error,

$\displaystyle H_{X}^{\text{BN 17}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccccccc}1&0&0&0&0&0&0&1&1&0&0&1&1&0&1&0&1\\\
0&1&0&0&0&0&0&0&0&0&1&0&1&0&1&0&1\\\ 0&0&1&0&1&0&0&0&0&0&0&1&1&0&0&0&0\\\
0&0&0&1&1&0&0&0&0&0&0&1&1&0&1&1&0\\\ 0&0&0&0&0&1&0&1&0&0&1&1&0&0&0&1&1\\\
0&0&0&0&0&0&1&0&1&0&1&1&0&0&0&1&1\\\ 0&0&0&0&0&0&0&0&0&1&1&1&1&0&0&0&0\\\
0&0&0&0&0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right]$ $\displaystyle
H_{Z}^{\text{BN 17}}$
$\displaystyle=\left[\begin{array}[]{ccccccccccccccccc}1&0&0&0&0&0&1&1&0&0&0&0&0&0&0&0&0\\\
0&1&0&0&0&0&0&1&1&0&0&1&1&0&0&0&0\\\ 0&0&1&0&0&0&0&0&0&0&1&0&1&0&0&1&1\\\
0&0&0&1&0&0&0&1&1&0&0&0&0&0&1&1&0\\\ 0&0&0&0&1&0&0&1&1&0&1&0&1&0&1&0&1\\\
0&0&0&0&0&1&1&1&1&0&0&0&0&0&0&0&0\\\ 0&0&0&0&0&0&0&0&0&1&1&1&1&0&0&0&0\\\
0&0&0&0&0&0&0&0&0&0&0&0&0&1&1&1&1\end{array}\right].$

Figure 11: The difference in normalized logical error rate $\Delta p_{L}^{\rm
norm}=p^{\text{norm}}_{L,{\rm 2DColor}}-p^{\text{norm}}_{L,{\rm BN17}}$ as a
function of $p_{X},p_{Z}$ (left) and that of unnormalized error rate (right).
The translucent plane in red marks $\Delta p_{L}=0$ above which the
$[[19,1,5]]$ code has higher error rate.

We remark that although the codes found here attains the best known logical
error rate for some fixed physical error rate, the code must also not be fine
tuned to that particular error probability as error rates from realistic
settings are not exact. Fig 11 shows that the BN 17 $[[17,1,3]]$ code
continues to outperform the $[[19,1,5]]$ Color code in reducing logical error
even if one perturbs the physical error probabilities in the neighbourhood of
$p_{X}=0.01,p_{Z}=0.05$. Therefore the notion of optimality is not fine-tuned.