# LARNN: Linear Attention Recurrent Neural Network

Guillaume Chevalier  Dept. of Computer Science and Software Engineering,
Laval University, Quebec, Canada

###### Abstract

The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention
module derived from the LSTM cell and ideas from the consciousness Recurrent
Neural Network (RNN). Yes, it LARNNs. The LARNN uses attention on its past
cell state values for a limited window size $k$. The formulas are also derived
from the Batch Normalized Long Short-Term Memory (BN-LSTM) cell and the
Transformer Network for its Multi-Head Attention Mechanism. The Multi-Head
Attention Mechanism is used inside the cell such that it can query its own $k$
past values with attention with restricted windowing on the $k$ most recent
previous cell state. This has the effect of augmenting the rank of the tensor
with the attention mechanism, such that the cell can perform complex queries
to question its previous inner memories, which should augment the long short-
term effect of the memory. With a clever trick, the LARNN cell with attention
can be easily used inside a loop on the cell state, just like how any other
Recurrent Neural Network (RNN) cell can be looped linearly through time
series. This is due to the fact that its state, which is looped upon
throughout time steps within time series, stores the inner states in a "first
in, first out" queue which contains the $k$ most recent states and on which it
is easily possible to add static positional encoding when the queue is
represented as a tensor. This neural architecture yields better results than
the plain, vanilla LSTM cells. It can obtain results of 91.924% for the test
accuracy, compared to the previously attained 91.653% using vanilla LSTM
cells. Note that this is not to compare to other research, where up to 93.349%
is obtained, but by costly using 18 LSTM cells rather than with 2 to 3 cells
as analyzed here and in comparison. Finally, an interesting discovery is made,
such that adding activation within the multi-head attention mechanism’s linear
layers can yield better results in the context researched hereto.

##  1 Introduction

###  1.1 Prior Art

It have always been hard to replace the LSTM [HS97] since its discovery by
Sepp Hochreiter and Jürgen Schmidhuber, that it’d be for language models or
other kind of time series processing with neural networks, such as sensors
signal.

Recently, Attention Mechanisms [BCB14] have been proven quite useful for
Neural Machine Translation (NMT) when paired with RNNs. However, Attention
Mechanisms are so good that they recently have been used alone without any
kind of RNN nor Convolutional Neural Network (CNN). This marks the apparition
of the Transformer Network [VSP+17], claiming by the title of the paper that
Attention Is All You Need. Since then, people started to believe that RNNs
could be discarded in favor of attention mechanisms.

On the other side, there is Yoshua Bengio with his paper about The
Consciousness Prior [Ben17], in which he express consciousness as being
recurrent through time, introducing the consciousness RNN which, by any means,
may retain fragments of the input representations through time in the RNN cell
as if it was attention.

###  1.2 Proof that RNNs are here to stay

However, RNNs can’t be replaced: they are $O(n)$, while attention mechanisms
are $O(n^{2})$, where $n$ here represents the temporal axis along the
processed time series. Those are different data structures, and it is not true
that attention models can fully replace recurrent models. And even if the
Attention Mechanisms would be stacked hierarchically (like rescursive pooling)
with small windows - similarly to how a Wavenet [VDODZ+16] would use
convolutions with windowing) - at best this neural architecture would require
$O(nlog(n))$ time to process its input time series. The advantages of
$O(n^{2})$ attention only appears when the sequence length is small enough to
be computed in $O(1)$ within a huge graphic card having enough memory and
cores to process the whole thing all at once. All that being said, RNNs can’t
simply disappear: it’s a fundamental linear data structure. Articles claiming
the fall of RNN have no reason to exist and twists the facts. We shall not
think of apples as oranges.

###  1.3 Logical reasoning leading to the idea of the LARNN

Althought Self-Attention Mechanisms are there to stay, it is not the case that
RNNs can be fully replaced. This is supported by Yoshua Bengio, as in his
paper about The Consciousness Prior [Ben17], where he formalizes that
consciousness evolves linearly through time, just like an RNN or like an LSTM
such as seen in Figure 1. In his (yet still very underrated paper), he
introduces the consciousness RNN, which is defined with the function $C$ as
such:

$c_{t}=C(h_{t},\leavevmode\nobreak\ c_{t−1},\leavevmode\nobreak\ z_{t})$

where $z_{t}$ is a random source of noise, $c_{t}$ the conscious state,
$c_{t-1}$ the previous conscious state, and $h_{t}$ a form of inputs’
representation state. Overall, it it stated that $c_{t}$ is a form of
attention which picked elements from $h_{t}$. That being said, it’s quite
reasonable to keep attentional neural architectures to be linear in time to
keep the consciousness flowing through time, linearly. That is: using
Attention Mechanisms within RNNs!

Why wouldn’t such a consciousness be able to have attention over itself to
digest information? For example, let’s derive his equation such as to form a
LARNN function with a consciousness $C$ which does not only examines its
immediate past state:

$c_{t}=C(h_{t},\leavevmode\nobreak\ c_{t−1},\leavevmode\nobreak\
c_{t−2},\leavevmode\nobreak\ c_{t−3},\leavevmode\nobreak\
...,\leavevmode\nobreak\ c_{t−k},\leavevmode\nobreak\ z_{t})$

where $k$ is an attention window which restricts how far the LARNN can see
itself back in time, and on which how gradients can flow. This is now of
complexity $O(nk^{2})$ in time rather than $O(n)$, but the fact that $k$ is
set to be a constant is pleasing. This constant is the clever trick described
in the abstract. In practice, a queue can be used, and could have as first
item a tensor filled with zeros, a random value, or an embedding encoding the
initial setup. The queue can then be filled through time steps and is self-
contained as a single tensor which changes with every new state, dropping too
old states when the queue is full. This reduces the previous equation to:

$v_{t}=[[c_{t−1},\leavevmode\nobreak\ c_{t−2},\leavevmode\nobreak\
c_{t−3},\leavevmode\nobreak\ ...,\leavevmode\nobreak\ c_{t−k}]]$
$c_{t}=C(h_{t},\leavevmode\nobreak\ v_{t},\leavevmode\nobreak\ z_{t})$

Note that here, we use $[[\cdot]]$ to denote a concatenation along the time
dimension. Otherwise, we will use $[\cdot]$ later to denote a concatenation
along the features dimension.

Figure 1: The LSTM cell, which can be substituted by a consciousness RNN as
introduced by Yoshua Bengio. This visualization is freely available and is
licensed under the CC-BY License, by Guillaume Chevalier. For more
information, visit https://github.com/guillaume-chevalier/Linear-Attention-
Recurrent-Neural-Network/tree/master/inkscape_drawings.

##  2 Model Architecture

###  2.1 Multi-Head Attention Mechanism

For a concrete implementation of the previous formula, a modified version of
the Multi-Head Attention can be used, similarly as seen in Attention Is All
You Need [VSP+17], with optionally some positional embedding to relatively
index the keys back in time. Note that the positional embedding here performs
a concatenation on the features rather than an addition as originally, a bit
like a dense layer [HLWvdM17] rather than a residual layer[HZRS16]:

$v_{t}=[[c_{t−1},\leavevmode\nobreak\ c_{t−2},\leavevmode\nobreak\
c_{t−3},\leavevmode\nobreak\ ...,\leavevmode\nobreak\ c_{t−k}]]$
$key=value=positionalEncoding(v_{t})$
$query=W_{xh}([x_{t},\leavevmode\nobreak\ h_{t-1}])$
$BNELU_{j}(arg)=BatchNorm_{j}(elu(arg))=BN(elu(arg))$
$a_{t}=MultiHeadSoftmax\bigg{(}\frac{query*BNELU_{1}(key)}{sqrt(d_{k})}\bigg{)}*BNELU_{2}(values)$
$c_{t}=C(h_{t},\leavevmode\nobreak\ a_{t},\leavevmode\nobreak\ z_{t})$

Where $d_{k}$ is the dimensionality of every attention head:

$d_{k}=numberFeatures/numberHeads=2$

The Batch Normalization is always 1-dimensional and applied on the features
dimension, which requires a temporary flatten from a rank 3 tensor to a rank 2
tensor.

Figure 2: A concrete implementation of the LARNN and its usage of the multi-
head attention mechanism on a windowed queue of its past cell states. K, V and
Q are respectively the attention mechanism’s input Keys, Values and only one
Query. The most recent states $c_{[t-1..t-k]}$ are arranged in a "first in,
first out" queue of length $k$. There is also the not-illustrated fact that
the query is formulated from a layer of the concatenation of $h_{t-1}$ and
$x_{t}$ on the innermost (feature) axis. This visualization is freely
available and is licensed under the CC-BY License, by Guillaume Chevalier. For
more information, visit https://github.com/guillaume-chevalier/Linear-
Attention-Recurrent-Neural-Network/tree/master/inkscape_drawings.

###  2.2 Special usage of Positional Encoding

The positional encoding discussed in Attention Is All You Need [VSP+17] uses a
geometric series of sines and cosines defined by the following formula and as
represented in Figure 3:

$PE_{(pos,\leavevmode\nobreak\ 2i)}=sin(pos/10000^{2i/d_{model}})$
$PE_{(pos,\leavevmode\nobreak\ 2i+1)}=cos(pos/10000^{2i/d_{model}})$

where $d_{model}$ is here the same as the hidden unit size, that is, the
number of features in the LARNN, akin the number of features (hidden size) in
the LSTM. Their positional embedding may contain a random phase so as to let
their Neural Machine Translation (NMT) model generalize to unknown sentences
length.

In the case of the LARNN, no random phase is needed, because the LARNN always
have a window of a fixed size. Therefore, the positional encoding is reversed
and is therefore applied from the most recent cell state as being the zero of
the sines and cosines, then positively towards older cell states $c_{t}$ (more
details in the LSTM and LARNN equations later). Also, because the window size
is fixed, it is no longer needed to express the encoding in function of the
number of features, but now rather in function of the window size. In the case
of the LARNN, perfect exponents of two were used for the wavelenghts, which
yields its particularly "pixel-perfect" agreeable aspect, rather than using
multiples of 1000, which was originally used in Attention Is All You Need for
a reason that seems unknown reason to the best of my knowledge.

In Figure 5, it’s also possible to see that in the LARNN, the positional
encoding is concatenated to the features, rather than added to them as in
Attention Is All You Need. This is to replicate the effect of a dense layer
[HLWvdM17] rather than a residual layer[HZRS16].

Figure 3: The original positional encoding used in Attention Is All You Need
[VSP+17], composed of sines and cosines. Note that here, the sines and cosine
has been split rather than interpolated, and are used with a ${d_{m}odel}$ of
42 and a sequence length of 128. Figure 4: The modified positional encoding
used in the LARNN, composed of sines and cosines. The window size is fixed to
128 for visualization purposes here, which causes the greatest wavelength to
be of a quarter of its wavelength from zero to its first peak in case of a
sine. Therefore, this type of encoding is such that its greatest wavelength is
of at least four times the window size. In practice, for the LARNN, the window
size is for sure smaller than $k=128$ to be linear, since in the used dataset
the sequence length is of 128, which represents 2.56 seconds worth of data.
From left to right would be placed, respectively, the most recent and the
oldest windowed cell states $c_{t}$.  Figure 5: The windowed cell states
$c_{t}$ are here represented above, concatenated on the sequence length
(window size) temporal axis, whereas the encoding is concatenated below on the
feature size axis. That is, there are 42 features in the cells’ states $c_{t}$
before the concatenation of the encoding which augments the number of such
features to be processed by the multi-head attention as a keys and values.
This visualization and the three previous ones are computed from the code of
the project which can be found here: https://github.com/guillaume-
chevalier/Linear-Attention-Recurrent-Neural-
Network/blob/master/AnnotatedMultiHeadAttention.ipynb

###  2.3 Relationships with the Batch Normalized LSTM

We now want to place the formulas of the LSTM RNN in place of the
consciousness RNN $C$, such as in Figure 1. For convenience, here are some
formulas for an LSTM cell. Note that here, with a slight abuse of notation,
$h_{t}$ is the output, and that $x_{t}$ is the input, whereas in the equations
above, $h_{t}$ is instead the input. Let’s continue by using the LSTM
equations with Batch Normalization (such as a BN-LSTM) [CBL+16], but here with
an Exponential Linear Unit (ELU) activation [CUH15], such as being:

$f_{t}=\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \sigma(BN(W_{hf}\leavevmode\nobreak\
h_{t-1}+W_{xf}\leavevmode\nobreak\ x_{t}+b_{f}))$ $i_{t}=\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\sigma(BN(W_{hi}\leavevmode\nobreak\ h_{t-1}\leavevmode\nobreak\
+W_{xi}\leavevmode\nobreak\ x_{t}+b_{i}))$ $o_{t}=\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\sigma(BN(W_{ho}\leavevmode\nobreak\ h_{t-1}+W_{xo}\leavevmode\nobreak\
x_{t}+b_{o}))$ $g_{t}=tanh(BN(W_{hc}\leavevmode\nobreak\
h_{t-1}+W_{xc}\leavevmode\nobreak\ x_{t}+b_{c}))$
$C_{t}=BN(f*C_{t-1}+i*{x_{t}}*g_{t})$ $h_{t}=BN(o_{t}*elu(C_{t}))$

###  2.4 Putting it all together

Let’s now define how it’s possible to merge the result of an attention query
$a_{t}$ inside the LSTM formulas before incorporating this into $C$ function
to replace it with the LSTM. In fact, two different ways can be defined as
such, either one or the other - the residual or the layer mode of joining the
attention into the LSTM cell - by replacing the definition of $g_{t}$ as
either:

$g_{t\leavevmode\nobreak\ residual\leavevmode\nobreak\
mode}=tanh(BN(W_{hc}\leavevmode\nobreak\ h_{t-1}+W_{xc}\leavevmode\nobreak\
x_{t}+W_{ac}\leavevmode\nobreak\ a_{t}+b_{c}))$

or:

$g_{t\leavevmode\nobreak\ layer\leavevmode\nobreak\
mode}=Wa([x_{t},h_{t-1},a_{t}])+b_{a}$

With respect to the equations above to be merged and with the notation used in
most of the LSTM formulas with $h_{t}$ to be the output and $x_{t}$ the input,
here, $C$ is finally replaced by a BN-LSTM cell with attention to create the
LARNN. It is possible to finally obtain the following equations, and with
choosing optionally $g_{t\leavevmode\nobreak\ residual\leavevmode\nobreak\
mode}$ or $g_{t\leavevmode\nobreak\ layer\leavevmode\nobreak\ mode}$, which
are both listed. We obtain something akin to what’s seen in the Figure 2 which
is an oversimplified representation of the LARNN implementation which goes
like:

$v_{t}=[[c_{t−1},\leavevmode\nobreak\ c_{t−2},\leavevmode\nobreak\
c_{t−3},\leavevmode\nobreak\ ...,\leavevmode\nobreak\ c_{t−k}]]$
$key=value=positionalEncoding(v_{t})$
$query=W_{xh}([x_{t},\leavevmode\nobreak\ h_{t-1}])$
$BNELU_{j}(arg)=BatchNorm_{j}(elu(arg))=BN(elu(arg))$
$a_{t}=MultiHeadSoftmax\bigg{(}\frac{query*BNELU_{1}(key)}{sqrt(d_{k})}\bigg{)}*BNELU_{2}(values)$
$f_{t}=\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \sigma(BN(W_{hf}\leavevmode\nobreak\
h_{t-1}+W_{xf}\leavevmode\nobreak\ x_{t}+b_{f}))$ $i_{t}=\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\sigma(BN(W_{hi}\leavevmode\nobreak\ h_{t-1}\leavevmode\nobreak\
+W_{xi}\leavevmode\nobreak\ x_{t}+b_{i}))$ $o_{t}=\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\sigma(BN(W_{ho}\leavevmode\nobreak\ h_{t-1}+W_{xo}\leavevmode\nobreak\
x_{t}+b_{o}))$ $g_{t\leavevmode\nobreak\ residual\leavevmode\nobreak\
mode}=tanh(BN(W_{hc}\leavevmode\nobreak\ h_{t-1}+W_{xc}\leavevmode\nobreak\
x_{t}+W_{ac}\leavevmode\nobreak\ a_{t}+b_{c}))$ $g_{t\leavevmode\nobreak\
layer\leavevmode\nobreak\ mode}=Wa([x_{t},h_{t-1},a_{t}])+b_{a}$
$C_{t}=BN(f*C_{t-1}+i*{x_{t}}*g_{t})$ $h_{t}=BN(o_{t}*elu(C_{t}))$

Note again that in the experiments here made, the BN-LSTM was modified such as
to have an Exponential Linear Unit (ELU) activation [CUH15] in plus of Batch
Normalization. This was a cheap inefficient trick to blindly try to obtain
better results from an engineer’s point of view. Note that such BN-ELU
normalization was also added to the Multi-Head’s linear mappings of the keys
and values, which improved results too, as discussed in the analysis below.

##  3 Training Procedure and Results

The training of the concrete implementation of the LARNN was performed with
Hyperopt using the TPE algorithm [BYC13], which is yields better results than
a random hyperparameter search and a grid search [BB12]. Two rounds of the TPE
algorithm were performed. In the first round, many hyperparameters were set to
vary a lot in order for the search to be in diffuse mode, and then later on in
a second round, the search was set to a focused mode [RS07], that is, by
restraining the area of where the search is performed by fixing some
hyperparameters to their best value or towards a good range of values. The
dataset used was the Public Domain Dataset for Human Activity Recognition
using Smartphones [AGO+13] by Anguita, Davide, et al., as uploaded on the UCI
Machine Learning Repository. In this paper’s code, this dataset is named the
UCIHAR dataset.

###  3.1 Accuracies of Every Trained Model and Hyperparameters Analysis

It is possible to see in Figure 6 and in Figure 7 the accuracies obtained at,
respectively, round 1 and round 2. Also, the effect of every hyperparameters
related to each other are visualized in two scatter plots for the round 1 and
the round 2 in Figure 8 and in Figure 9. Those plots are not an ablation
study, but are close to be. For example, sometimes different types of LARNN
modes are used to create the layers, and sometimes the densely concatenated
positional encoding is turned off.

The training was carried on Amazon AWS’s p3.2xlarge instances, that is, with
NVIDIA Tesla V100 GPUs of 16 GB RAM with 640 NVIDIA Tensor cores and 5’120
NVIDIA CUDA cores offering more than 100 TFLOPS. The total costs were of
$344.17 USD at a cost of $3.06 USD per hour, which means 112 hours of GPU
usage.

Figure 6: Plot illustrating the test accuracy of every trials through 25
epochs for the round 1 of hyperparameters optimization. For more information,
visit https://github.com/guillaume-chevalier/Linear-Attention-Recurrent-
Neural-Network/blob/master/AnalyzeTestHyperoptResults_round_1.ipynb. Figure 7:
Plot illustrating the test accuracy of every trials through 100 epochs for the
round 2 of hyperparameters optimization. For more information, visit
https://github.com/guillaume-chevalier/Linear-Attention-Recurrent-Neural-
Network/blob/master/AnalyzeTestHyperoptResults_round_2.ipynb. Figure 8:
Scatter plot depicting the effect of every hyperparameters in relation to each
other for the optimization round 1. For more information, visit
https://github.com/guillaume-chevalier/Linear-Attention-Recurrent-Neural-
Network/blob/master/AnalyzeTestHyperoptResults_round_1.ipynb. Figure 9:
Scatter plot depicting the effect of every hyperparameters in relation to each
other for the optimization round 2. For more information, visit
https://github.com/guillaume-chevalier/Linear-Attention-Recurrent-Neural-
Network/blob/master/AnalyzeTestHyperoptResults_round_2.ipynb. Figure 10: Test
accuracy throughout training for when batch-normalized ELU activation is used
on the keys and values’ linear layers of the Multi-Head Attention Mechanism.
This is an interesting discovery considering Multi-Head Attention Mechanisms
are now, at the time of writing, the State Of The Art (SOTA) methods in
solving Natural Language Processing (NLP) problems such as Neural Machine
Translation (NMT), as done in 2017 in the Attention Is All You Need paper
[VSP+17], and bringing NLP systems closer to Artificial General Intelligence
(AGI) such as in the One Model To Learn Them All paper [KGS+17] using a Multi-
Head Dot-Product Attention. However, note that the LARNN have still not been
tested on NLP tasks as of today, it has only been tested on the sensors
dataset used here. For more information on how this figure was generated,
visit https://github.com/guillaume-chevalier/Linear-Attention-Recurrent-
Neural-Network/blob/master/AnalyzeTestHyperoptResults_round_1.ipynb.

From the charts, it’s possible to see that the positional encoding did not
help. Also, it’s better to stack the two or three cells in a residual fashion,
such as to add their $h_{t}$ together at layer 1 and layer 2 before the final
classification layer which is placed at the last time step. Also, it appears
that the best LARNN mode is the "residual" one, while the "layer" one yields
average results in the round 2. It’s also possible to observe that the results
are better when placing an activation on the Multi-Head Attention’s linear
mappings before the dot products and attention products are made, as seen in
Figure 10. which is quite interesting and may orient future research when
using Multi-Head Attention. As a reminder: the activation used is a Batch
Normalization on an ELU activation of the linears. The Batch Normalization is
1-dimensional on the features dimension.

It can obtain results of 91.924% for the test accuracy, compared to the
previously attained 91.653% using vanilla LSTM cells.

##  4 Conclusion

It is possible to infer new equations from the consciousness RNN and the
Multi-Head Attention Mechanisms to create a LARNN which makes use the
attention to augment the range of its consciousness. Moreover, intesresting
hyperparameter exploration yields insights on what is useful to this neural
architecture and what’s not. For example, positional encoding did not seem to
help specifically for this task if it is added in a dense manner, and also I
discovered that batch-normalized activation on the linear mappings of the
multi-head attention mechanism helped the model on the Test set predictions.

##  5 Acknowledgements

Thanks to Yu Zhao for having participated in the HAR-stacked-residual-bidir-
LSTMs [ZYCG17] project with me, which is licensed under the Apache 2.0
license, a project from which I reused some code for loading the dataset. In
turns, this code was originally derived from my own project available at
https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition.

Also, thanks to Vooban for open-sourcing and sublicensing under the MIT
License its derivative of my code https://github.com/guillaume-
chevalier/Hyperopt-Keras-CNN-CIFAR-100 now at the address
https://github.com/Vooban/Hyperopt-Keras-CNN-CIFAR-100, which I was able to
reuse here as boilerplate code to setup the hyperparameter search.

More details on the licenses can be obtained at the respective address of each
project.

I’d like to also thanks Philippe Giguère, Professor at Université Laval, who
created a nice class on deep learning, such as listed in my awesome resources:
https://github.com/guillaume-chevalier/Awesome-Deep-Learning-Resources. I
especially liked his rare visualization of the multi-head attention mechanism,
which is available in his slides at https://ulaval-damas.github.io/glo4030/,
and more precisely, at the page 28 of his slides here:
http://www2.ift.ulaval.ca/~pgiguere/cours/DeepLearning/09-Attention.pdf.

The LARNN repository is available at https://github.com/guillaume-
chevalier/Linear-Attention-Recurrent-Neural-Network, available under the MIT
License, and coded with PyTorch [PGC+17].

## References

  * [AGO+13] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz.  A public domain dataset for human activity recognition using smartphones.  In ESANN, 2013.
  * [BB12] James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  Journal of Machine Learning Research, 13(Feb):281–305, 2012.
  * [BCB14] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [Ben17] Yoshua Bengio.  The consciousness prior.  arXiv preprint arXiv:1709.08568, 2017.
  * [BYC13] James Bergstra, Dan Yamins, and David D Cox.  Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms.  In Proceedings of the 12th Python in Science Conference, pages 13–20. Citeseer, 2013.
  * [CBL+16] Tim Cooijmans, Nicolas Ballas, César Laurent, Çağlar Gülçehre, and Aaron Courville.  Recurrent batch normalization.  arXiv preprint arXiv:1603.09025, 2016.
  * [CUH15] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289, 2015.
  * [HLWvdM17] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, volume 1, page 3, 2017.
  * [HS97] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [KGS+17] Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit.  One model to learn them all.  arXiv preprint arXiv:1706.05137, 2017.
  * [PGC+17] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [RS07] Marcus E Raichle and Abraham Z Snyder.  A default mode of brain function: a brief history of an evolving idea.  Neuroimage, 37(4):1083–1090, 2007.
  * [VDODZ+16] Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  arXiv preprint arXiv:1609.03499, 2016.
  * [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, pages 6000–6010, 2017.
  * [ZYCG17] Yu Zhao, Rennong Yang, Guillaume Chevalier, and Maoguo Gong.  Deep residual bidir-lstm for human activity recognition using wearable sensors.  CoRR, abs/1708.08989, 2017.


# Simulation and performance analysis of quantum error correction with a
rotated surface code under a realistic noise model

Mitsuki Katsuda<EMAIL_ADDRESS>Graduate School of Engineering
Science, Osaka University, 1-3 Machikaneyama, Toyonaka, Osaka 560-8531, Japan
Kosuke Mitarai<EMAIL_ADDRESS>Graduate School of Engineering
Science, Osaka University, 1-3 Machikaneyama, Toyonaka, Osaka 560-8531, Japan
Center for Quantum Information and Quantum Biology, Osaka University, 1-2
Machikaneyama, Toyonaka 560-0043, Japan  JST, PRESTO, 4-1-8 Honcho, Kawaguchi,
Saitama 332-0012, Japan  Keisuke Fujii<EMAIL_ADDRESS>Graduate
School of Engineering Science, Osaka University, 1-3 Machikaneyama, Toyonaka,
Osaka 560-8531, Japan  Center for Quantum Information and Quantum Biology,
Osaka University, 1-2 Machikaneyama, Toyonaka 560-0043, Japan  RIKEN Center
for Quantum Computing (RQC), Hirosawa 2-1, Wako, Saitama 351-0198, Japan
Fujitsu Quantum Computing Joint Research Division at QIQB, Osaka University,
1-2 Machikaneyama, Toyonaka 560-0043, Japan

###### Abstract

The demonstration of quantum error correction (QEC) is one of the most
important milestones in the realization of fully-fledged quantum computers.
Toward this, QEC experiments using the surface codes have recently been
actively conducted. However, it has not yet been realized to protect logical
quantum information beyond the physical coherence time. In this work, we
performed a full simulation of QEC for the rotated surface codes with a code
distance 5, which employs 49 qubits and is within reach of the current state-
of-the-art quantum computers. In particular, we evaluate the logical error
probability in a realistic noise model that incorporates not only stochastic
Pauli errors but also coherent errors due to a systematic control error or
unintended interactions. While a straightforward simulation of 49 qubits is
not tractable within a reasonable computational time, we reduced the number of
qubits required to 26 qubits by delaying the syndrome measurement in
simulation. This and a fast quantum computer simulator, Qulacs, implemented on
GPU allows us to simulate full QEC with an arbitrary local noise within
reasonable simulation time. Based on the numerical results, we also construct
and verify an effective model to incorporate the effect of the coherent error
into a stochastic noise model. This allows us to understand what the effect
coherent error has on the logical error probability on a large scale without
full simulation based on the detailed full simulation of a small scale. The
present simulation framework and effective model, which can handle arbitrary
local noise, will play a vital role in clarifying the physical parameters that
future experimental QEC should target.

##  I Introduction

Quantum error correction (QEC) is essential for the realization of quantum
computers because physical qubits suffer from errors due to decoherence caused
by undesirable interactions with the environment. QEC protects them from such
errors by encoding logical information of qubits on many physical qubits [1].
With the progress of quantum hardware, it is becoming possible to precisely
control tens to a hundred qubits, leading to experimental demonstrations of
simple QEC codes on a variety of physical systems [2, 3, 4, 5, 6, 7]. Surface
codes proposed by Kitaev [8] has attracted much attention as a method to
implement an error correction with superconducting qubits due to their
relatively high threshold for local errors and their implementability using a
two-dimensional lattice of qubits. By using the rotated surface code [9], we
can make one logical qubit with code distance $d$ using only $2d^{2}-1$
qubits. This means that we can implement $d=3$ and $d=5$ surface codes with 17
and 49 physical qubits, respectively. Currently available devices such as the
ones presented in [10, 11] can handle qubits of this scale. Experimental
efforts are underway to demonstrate QEC with the surface codes using the
superconducting qubits. Ref. [7] has achieved exponential suppression of the
logical error probability with the one-dimensional repetitive code, which is a
one-dimensional substructure of the surface code. Moreover, Refs. [12, 13, 14]
have implemented all of the operations necessary for the implementation of QEC
in a two-dimensional surface code with $d=3$, while they do not achieve the
break-even point, that is, the logical error probability is smaller than the
physical error probability.

To demonstrate that QEC can actually achieve a logical error probability lower
than the physical one, it is essential to implement a surface code with a code
distance of $d=5$ that can correct up to two errors. This is because two-qubit
operations inevitably introduce two-qubit correlated errors on physical
qubits. However, successful experimental realizations of surface codes [15, 7,
12, 13, 14] are still up to $d=3$ [12, 13, 14] possibly due to the limited
fidelity of gates and readouts in present quantum devices. To clarify what
ultimately limits the experimental implementation of the error correction,
detailed numerical analyses are required; we need to determine how much
performance is necessary for which parameters, and how much the logical error
probability can be reduced if they are achieved.

Numerical experiments of surface codes have been carried out under various
noise models. The simplest, albeit not truly realistic, model is the
stochastic Pauli errors where Pauli operators act on the qubits
probabilistically. In this case, the Gottesman-Knill theorem [16] allows us to
simulate even large systems efficiently. In real experimental systems,
unfortunately, there is a noise that has coherence and cannot be described by
stochastic Pauli errors originating from, for example, over-rotation with a
systematic control error, global external fields, cross-talk and so on.

Simulating QEC with these noises is as difficult as simulating a universal
quantum computer in general. However, in certain limited cases, it is possible
to numerically evaluate their performance. For example, a mixture of coherent
and incoherent noise on the one-dimensional repetition code has been analysed
in detail by making use of its exact solvability mapping it to free-fermionic
dynamics [17]. The $d=3$ rotated surface code with amplitude damping and
dephasing [18] has been analysed by exact simulation of the system, which is
possible due to its small number of qubits. A sophisticated technique for
contraction of 2D tensor networks has been used to simulate surface codes with
arbitrary local noise on the data qubits, while the syndrome measurements are
assumed to be ideal [19]. It is not clear if this method can be extended to
simulate circuit-level noise, where each elementary operation is subject to
noise. More recently, quasiprobability decomposition of non-Clifford channels
into Clifford channels is utilized to simulate surface codes with a small
coherent noise [20]. This method is, however, not applicable to arbitrary
noise because the coherence of the noise increases its sampling overhead
exponentially. Given the finite precision of control systems in actual
experiments, the existence of coherent errors is inevitable. A detailed
simulation of the surface code with $d=5$, which is the near-term milestone of
QEC, is still challenging for classical computers. A framework to realize such
simulations and to obtain knowledge on the impact of a coherent error on QEC
is highly demanded.

In this work, we fully simulate the QEC under a realistic noise model,
including incoherent and coherent noise, in the $d=5$ rotated surface code
with 49 physical qubits, and analyzed the effect of coherent errors on the
logical error probability. The main obstacle to its analysis is that a
straightforward simulation of 49 qubits would require a complex vector of
dimension $2^{49}$. This prevents us from simulating the dynamics with a
realistic computational resource. We overcome this obstacle by exploiting the
structure of the syndrome measurement and reusing the measured qubits in the
simulation. This allows us to achieve a full simulation of the $d=5$ rotated
surface code by simulating only 26 qubits, thus making it feasible to analyze
the effects of arbitrary local noise models on this QEC code. In particular,
assuming its implementation on superconducting qubits, we use a realistic gate
set and noise model, such as coherent errors in one-qubit operations and cross
resonance gates in addition to naive stochastic Pauli errors. Moreover, we
develop an effective model of physical error probability for incorporating the
effects of coherent errors in rotated surface codes combining the simulation
results and the previous analysis of coherent errors in 1D repetitive codes
[17]. Using this model, we investigate the possible regime of coherence time,
gate time, coherent error ratio, etc., required for maintaining the quantum
information of a logical qubit beyond the coherence time of a physical qubit.
The results show that if the ratio of gate operation time to coherence time is
below 0.005, the lifetime of the logical qubit exceeds that of the physical
qubit, even if coherent errors occur with the same magnitude as their
incoherent counterparts. On the other hand, it was also found that if the
magnitude of the coherent error can be reduced to 20% of the incoherent one,
the ratio of gate operation time to coherence time is acceptable up to 0.007.
The present simulation framework for QEC would provide an important guideline
for future experimental demonstrations of QEC to extend the lifetime of
logical qubits.

##  II Simulation methods for $d=5$ rotated surface code

###  II.1 Circuits for syndrome measurements

(a)

(b)

(c)

Figure 1: LABEL:sub@subfig:rotated-surface-codeThe $d=5$ rotated surface codes. Yellow faces( ) corresponds to $X$-type stabilizer and ocher ones( ) corresponds to $Z$-type stabilizer.White circles( ) means data qubits with their index, and black circles( ) means ancilla qubits used for measurement and the order in which the measurement circuits are run. LABEL:sub@subfig:X-stabilizerCircuit for the $X$-type stabilizer measurement LABEL:sub@subfig:Z-stabilizerCircuit for the $Z$-type stabilizer measurement. Table 1: List of stabilizers of the $d=5$ rotated surface code Index | X-stabilizer | Z-stabilizer
---|---|---
1 | $X_{1}X_{2}$ | $Z_{16}Z_{21}$
2 | $X_{3}X_{4}$ | $Z_{6}Z_{11}$
3 | $X_{2}X_{3}X_{7}X_{8}$ | $Z_{11}Z_{12}Z_{16}Z_{17}$
4 | $X_{4}X_{5}X_{9}X_{10}$ | $Z_{1}Z_{2}Z_{6}Z_{7}$
5 | $X_{6}X_{7}X_{11}X_{12}$ | $Z_{17}Z_{18}Z_{22}Z_{23}$
6 | $X_{8}X_{9}X_{13}X_{14}$ | $Z_{7}Z_{8}Z_{12}Z_{13}$
7 | $X_{12}X_{13}X_{17}X_{18}$ | $Z_{13}Z_{14}Z_{18}Z_{19}$
8 | $X_{14}X_{15}X_{19}X_{20}$ | $Z_{3}Z_{4}Z_{8}Z_{9}$
9 | $X_{16}X_{17}X_{21}X_{22}$ | $Z_{19}Z_{20}Z_{24}Z_{25}$
10 | $X_{18}X_{19}X_{23}X_{24}$ | $Z_{9}Z_{10}Z_{14}Z_{15}$
11 | $X_{22}X_{23}$ | $Z_{15}Z_{20}$
12 | $X_{24}X_{25}$ | $Z_{5}Z_{10}$

The $d=5$ rotated surface code, which is the target in this work, is shown in
FIG.1a. White circles and black circles in Fig. 1a represent data qubits and
ancilla qubits for the sydrome measurements, respectively. The list of
stabilizers of this code is summarized in Table 1. In this table,
$X_{i},\,Z_{i}$ mean Pauli operators acting on the $i$-th data qubit. These
stabilizers are also illustrated in Fig. 1a. $X$-type stabilizers correspond
to yellow plaquettes, and $Z$-type stabilizers do to ocher ones. These
stabilizers are measured with the circuits shown in Figs. 1b and 1c,
respectively.

The order of CNOT gates is important to determine the minimum depth of the
syndrome measurement circuit. In this work, we choose the following order. In
the case of $X$-type stabilizers, they are applied clockwise from the bottom
right, while in the case of $Z$-type it is clockwise from the top right. For
example, if $Z_{1}Z_{2}Z_{6}Z_{7}$ is to be measured, we apply CNOT gates with
the order of 2,7,6,1. We have to make a special treatment for the measurement
qubits on the boundary as they have fewer CNOT gates than the other
measurement qubits. In this work, we make them “wait” if their target data
qubit doesn’t exist. While “waiting“, one-qubit noise is applied. For example,
the sequence for measuring $X_{1}X_{2}$ is “2, 1, wait, wait”, and for
measuring $X_{22}X_{23}$ is “wait, wait, 22, 23”.

###  II.2 Reducing the number of qubits for simulation

(a)  (b)

Figure 2: A method to reduce the number of qubits in simulation in the case of
the one-dimensional repetition code. (a) The original syndrome measurement
circuit. (b) A time-shifted version of the original one.

If we straightforwardly simulate 49 qubits, we need to reserve a
$2^{49}$-dimensional complex vector on a classical memory. It prohibits us to
simulate the error correction procedure with a practical computational
resource. To overcome this obstacle, we reduce the number of qubits that have
to be simulated by reusing the measured qubits. As an illustrative example of
this, we show the case for the one-dimensional repetition code in FIG.2. While
in the actual experiment, the syndrome measurements are done in parallel for
each measurement qubit, we delay them so that one syndrome measurement runs at
a time. This strategy allows us to reuse one measurement qubit for multiple
stabiliser measurements without changing the system to be simulated. Note that
the same effect can be obtained by analytically calculating a set of POVM
operators corresponding to each syndrome measurement and applying them with
appropriate probabilities. However, the circuit-based noise model with
coherent error considered here is somewhat complicated and hence we avoid this
approach.

(a)

(b)

Figure 3: LABEL:sub@subfig:block Left and right blocks represent the syndrome
measurement circuits for $X$\- and $Z$-stabilizers, respectively.
LABEL:sub@subfig:blocks Each syndrome measurement circuit is complied for
parallel $X$\- and $Z$-type syndrome measurements on the surface codes. Note
that each syndrome measurement circuit can be delayed without any collision
following the order shown in each block so that the syndrome measurements are
done sequentially to reduce the number of qubits in simulation.

When applying the proposed method to surface codes, we have to take care of
the order of the two-qubit gates in the three-dimensional arrangement so that
each syndrome measurement circuit can be delayed without any collision. A
three-dimensional unit block corresponding to the syndrome measurements for
$X$\- and $Z$-type stabilizers is shown in FIG. 3a. The unit block consists of
two types of rectangular blocks stacked on a time axis. The rectangle block of
$1\times 1\times 1$ represents a one-qubit gate applied to a measurement qubit
and the rectangle block of $2\times 1\times 1$ represents a two-qubit gate
between a measurement qubits and an adjacent data qubit. Using these 3D
blocks, we construct a syndrome measurement circuit for the whole surface code
as in FIG. 3b. As you can see, the blocks are assembled in such a way that
they can be delayed without any collision. This indicates that the measurement
qubits can be reused as same as the one-dimensional case by delaying the
syndrome measurements appropriately. Specifically, the numbers written in
black circles in FIG. 1a correspond to the order in which the measurements are
to be executed.

##  III Numerical experiment

###  III.1 Simulated system

The simulation method in Sec. II allows us to effectively perform the full-
vector simulation of $d=5$ rotated surface code, and therefore to evaluate its
performance under any local noise model. We consider a situation where two
types of noise, incoherent and coherent, act on physical qubits in a syndrome
measurement circuit. Here, we describe the concrete gateset and noise model
that are used in the simulation.

To conduct a realistic simulation, we consider a hardware-native gateset
common to the superconducting devices and compile the syndrome measurement
circuit with those gates. More concretely, we use a gateset
$\\{R_{x}(\pi/2),R_{x}^{\dagger}(\pi/2),R_{zx}(\pi/2),R_{zx}^{\dagger}(\pi/2),R_{z}(\pi/2)\\}$,
where

$\displaystyle R_{x}(\pi/2)$ $\displaystyle=e^{-i\pi X/4},$ (1) $\displaystyle
R_{z}(\pi/2)$ $\displaystyle=e^{-i\pi Z/4},$ (2) $\displaystyle R_{zx}(\pi/2)$
$\displaystyle=e^{-i\pi X\otimes Z/4},$ (3)

to perform the syndrome measurements. This gateset is commonly used in the
superconducting qubits with cross resonance gate [21, 22]. With these gate
sets, the syndrome measurement circuit can be rewritten as in FIG. 4, where
three changes are made from the circuit shown in FIG.1b and FIG.1c. Firstly,
we replace CNOT gates in FIG.1c with $R_{zx}(\pi/2)$ gates. Second, for the
two Hadamard gates in the circuit of FIG. 1b, we replaced the former one by
the $R_{x}^{\dagger}(\pi/2)$ gate and the latter one by the $R_{x}(\pi/2)$
gate. Finally, we apply $R_{x}(\pi/2)$ gates to the data qubits after the
circuit shown in FIG.1b and $R_{z}(\pi/2)$ gates to the data qubits after the
circuit in FIG.1c. These one-qubit rotation gates restore the errors
transformed to $Y$ error by $R_{zx}(\pi/2)$ gates.

Figure 4: A syndrome measurement circuit for the rotated surface codes
consisting of cross resonance gates and Virtual-Z gates. The rotation angle of
the $R_{x},\,R_{z},\,R_{zx}$ gates is $\pi/2$ and that of the
$R_{x}^{\dagger}$ gate is $-\pi/2$.

Next, let us define the noise model used in the simulation and see how
incoherent noise is incorporated. We assume that the single-qubit depolarizing
noise,

$\displaystyle\mathcal{E}_{1}(\rho)$
$\displaystyle=\quantity(1-p)\rho+\frac{p}{3}\sum_{A\in\quantity{X,\,Y,\,Z}}A\rho
A,$ (4)

acts on every qubit at each step of the measurement circuit in FIG. 4. Same
noise occurs in the “waiting” qubit where the gate does not act. For the two
qubits after the cross-resonance gates, we apply the two-qubit depolarizing
noise:

$\displaystyle\mathcal{E}_{2}(\rho)$
$\displaystyle=\quantity(1-p)\rho+\frac{p}{15}\sum_{A,B\in\quantity{I,X,Y,Z}}(A\otimes
B)\rho(A\otimes B).$ (5)

We call the parameter $p$ the physical error probability. In this paper, the
probability of each Pauli error is set to be equal, but it is not difficult to
reflect the actual distribution of stochastic errors.

Next, we describe coherent noise. The coherent noise was modelled as an
(unintentional) increase in the rotation angle of each rotation gate. This
increase is referred to as an over-rotation error. Since this is associated
with the rotation gate, it does not act on the “waiting” data qubit. In this
study, we want to model the error probability with the single parameter $p$.
To this end, we introduce a coherent error ratio $c$ and set the over-rotation
angle to be $\theta=2c\sqrt{p}$. In other words, for a rotation gate generated
by a Pauli operator $A$, we add the coherent noise channel in the form of:

$\displaystyle\mathcal{E}_{c}(\rho)=e^{-ic\sqrt{p}A}\rho e^{ic\sqrt{p}A}.$ (6)

The reason why we set $\theta=2c\sqrt{p}$ is as follows. Consider the
expectation value of $P_{1}=\outerproduct{1}{1}$ when
$R_{x}(\theta)=e^{-i\frac{\theta}{2}X}$ is applied to $\ket{0}$. It is
calculated as,

$\displaystyle\expectationvalue{R_{x}^{\dagger}(\theta)P_{1}R_{x}(\theta)}{0}=\sin^{2}\frac{\theta}{2}$
(7)

Since $\theta\ll 1$, the bit-flip probability $p_{\text{flip}}$ associated
with this over-rotation is,

$\displaystyle p_{\text{flip}}$ $\displaystyle=c^{2}p.$ (8)

Or equivalently, if the coherence of noise is destroyed at each step, for
example, by using the twirling operation, then such a decohered noise map
corresponds to a probabilistic Pauli error with probability $c^{2}p$. If the
coherent errors experience constructive or destructive interferences, then the
effect of the coherent error would be increased or decreased against $c^{2}p$.
The parameter $c$ controls the magnitude of the coherent error compared to the
incoherent one.

###  III.2 Numerical simulation

We employ one of the fastest classical simulators of quantum circuits, Qulacs
[23]. In the simulation, all data qubits are first initialised to $\ket{0}$
and projected to the surface code state by performing noise-free syndrome
measurements with zero outcomes. Next, we run syndrome measurement circuits
with circuit-level noise as explained above for five rounds. After completing
five rounds of measurements, all data qubits were subjected to projective
measurements in the Pauli $Z$ basis. The above procedure was repeated 10000
times. In decoding, as usual, we took the XOR of the syndrome of the adjacent
rounds to determine the position where the syndrome is flipped. With this
information, we estimated the error positions using the minimum weight perfect
matching (MWPM) algorithm implemented in NetworkX [24]. After applying the
recovery operation using the estimated errors, we finally calculate the
eigenvalue of the logical Pauli $Z$ operator
$Z_{L}=Z_{1}Z_{7}Z_{13}Z_{19}Z_{25}$ from the final projective measurement.
The logical error probability $p_{L}$ is estimated by dividing the number of
$Z_{L}=1$ occurrences by 10000. We varied $p$ from $10^{-3}$ to $7.0\times
10^{-2}$ and $c$ from 0.0 to 1.0 by 0.25. For each parameter pair $(p,c)$, the
calculation took 18 hours in the absence of coherent errors and up to 80 hours
in their presence using an NVIDIA A100 GPU.

Figure 5: The logical error probability is plotted as a function of the
physical error probability $p$ with coherent-noise parameters
$c=0,0.25,0.5,0.75,1.0$. The dotted lines correspond to the results of the
fitting.

The obtained $p_{L}$ is shown in FIG. 5. For $p$ smaller than about $3.0\times
10^{-3}$, we can see that $p_{L}$ decreases if $p$ is decreased. Furthermore,
$p_{L}$ satisfies $p_{L}=Ap^{\xi}$ at any value of $c$ within the statistical
error. By fitting the numerical values of $p_{L}$ at $c=0$ for $p=1.0\times
10^{-3}$ to $3.0\times 10^{-3}$ with $p_{L}=Ap^{\xi}$ using two parameters
$(A,\xi)$, we obtain $A=6.5\times 10^{5}$ and $\xi=2.92$. The value of $\xi$
is consistent with the expectation that it should be $\frac{d+1}{2}=3$. One
might think that the contribution of the coherent error is small because such
a small angle rotation can be frozen by repetitive projections for the
syndrome measurements. However, even for a small ratio $c=0.25$, it has a
negligible contribution to the logical error probability. This result clearly
shows that the effect of the coherence error is important to estimate the
experimentally achievable logical error probability accurately.

###  III.3 Effective model incorporating the coherent error

Figure 6: The ratio $p_{\mathrm{eff}}/p$ is plotted as a function of the
coherence parameter $c$. The dotted curve represents $1+\alpha c^{2}$ with the
fitted value $\alpha=0.872$.

In order to understand the effect of coherent noise on $p_{L}$, we consider
how to effectively incorporate the effect of coherent error as a leading order
correction to the incoherent error $p$. Since we expect that the leading-order
contribution of the coherent error to the effective error $p_{\mathrm{eff}}$
is proportional to $c^{2}p$, we model $p_{\mathrm{eff}}$ as,

$\displaystyle p_{\mathrm{eff}}=\quantity(1+\alpha c^{2})p.$ (9)

This is because while the probability amplitude for the coherent error is
$\sim c\sqrt{p}$, such an error becomes a detectable event if and only if such
an error occurs twice on either ket or bra spaces in the density operator
picture [17]. The coefficient $\alpha$ takes into account the fact that the
coherent errors can interfere with each other and therefore their contribution
to $p_{\mathrm{eff}}$ can be not exactly $c^{2}p$. As mentioned before, if the
coherence is destroyed at each step, then $\alpha$ should be a unit.

Then, we assume that the logical error probability under the coherent error is
given by replacing the physical probability $p$ in the case of $c=0$ with
$p_{\mathrm{eff}}$. More precisely, the logical error probability $p_{L}$
should be obtained by replacing $p$ in Eq. (9) with $p_{\mathrm{eff}}$:

$\displaystyle p_{L}$ $\displaystyle=A\cdot\quantity[\quantity(1+\alpha
c^{2})\cdot p]^{\xi},$ (10)

where $A$ and $\xi$ are thought to be the same as those with $c=0$. The
validity of this model is confirmed by detailed numerical calculations carried
out on the 1D repetition codes in the previous study [17]. To test this
assumption for the two-dimensional case, we estimate the value of $\alpha$ by
the following procedure. First, we fit $p_{L}$ obtained at various $c$’s in
the range of $p=1.0\times 10^{-3}$ to $3.0\times 10^{-3}$ with
$p_{L}=A(Bp)^{\xi}$ using $B$ as the fitting parameter while using the fixed
$A$ and $\xi$ obtained at $c=0$. Then, we fit $B$ with $1+\alpha c^{2}$. As a
result we obtain $\alpha=0.872$ and the ratio $p_{\rm eff}/p$ shown in FIG. 6
implies that this fitting goes well supporting our assumption. Another
interesting fact is the coefficient $\alpha=0.872$ is smaller than a unit
meaning the coherent error interferes in a destructive way. This implies that
the coherent error modelled in this work is not so damaging for QEC, since the
logical error probability is increased when the coherence is destroyed at each
step by twirling.

Since this behaviour of the leading order contribution of the coherent error
is a local property of the noise, the effective model obtained here is
expected to be valid not only when $p$ is reduced for $d=5$, but also when the
code distance $d$ is further increased. If this is true, we can estimate the
logical error probability under the coherent error by combining a limited size
of full simulation and a large size of simulation with stochastic Pauli noise.

###  III.4 Experimental consideration

As a concrete usage of this model, here we argue in what situation an
experimental QEC can achive a longer lifetime $t_{L}$ of logical quantum
information against a physical coherence time $t_{c}$ by calculating
$t_{L}/t_{c}$. First, we rewrite the error probability $p$ using the coherence
time $t_{c}$ and time $t_{g}$ required for each gate operation. For clarity,
we assume that the error probability is well approximated by:

$\displaystyle p\simeq 1-e^{-\frac{t_{g}}{t_{c}}}.$ (11)

This is the situation where a quantum gate is well-calibrated and the
coherence-time-limited fidelity is achieved. Second, we rewrite the logical
lifetime $t_{L}$ using the logical error probability $p_{L}$. Letting
$N_{\text{steps}}=11$ be the number of steps in one cycle of the syndrome
measurement (see Fig. 4), it takes physical time $dN_{\text{steps}}t_{g}$ to
conduct $d$ cycles of syndrome measurements. Since the logical error occurs
with probability $p_{L}$ in this time, the logical lifetime is roughly given
by

$\displaystyle t_{L}=\frac{dN_{\text{steps}}t_{g}}{p_{L}}.$ (12)

Finally, combining Eqs. (10)-(12), we obtain the following relation:

$\displaystyle\frac{t_{L}}{t_{c}}$ $\displaystyle=\frac{d\cdot
N_{\mathrm{gates}}}{A\cdot\quantity[\beta(1+\alpha
c^{2})\cdot\quantity(1-e^{-\frac{t_{g}}{t_{c}}})]^{\xi}}\frac{t_{g}}{t_{c}}.$
(13)

A graph plotting $t_{L}/t_{c}$ as a function of $c$ and $t_{g}/t_{c}$ is shown
in the FIG. 7.

---
Figure 7: The lifetime $t_{L}$ of a logical qubit, calculated from the
coherent noise parameter $c$, the gate operation time $t_{g}$ and the coherent
time $t_{c}$ of the physical qubit, divided by $t_{c}$. The upper graph is a
three-dimensional representation of $t_{L}/t_{c}$, while the lower graph shows
the relationship between $t_{g}/t_{c}$ and $c$ in more detail. The red area in
the upper graph and the shaded area in the lower graph show the area where
$t_{L}<t_{c}$.

The part of the graph marked in red is the parameter region where
$t_{L}/t_{c}$ is below 1. In this region, the error correction procedure
itself damages the lifetime and therefore is meaningless for protecting the
quantum information. To achieve $t_{L}/t_{c}>1$, the gate and coherence time
ratio $t_{g}/t_{c}$ depends on the amount of coherent error and ranges from
0.005 to 0.007. If the ratio is reduced to $t_{g}/t_{c}=0.001$, the lifetime
of the logical qubit is improved by a factor of tens against the coherence
time even in the presence of the coherence error with $c=1$. In this case, the
over-rotation angle is $\theta\sim 0.06$, which would be experimentally
detectable and hopefully can be calibrated [25].

Finally, let us discuss the experiments on the rotated surface code with code
distance 3 carried out by different research groups in 2021 [12, 13]. The main
parameters of each experimental system are given in TABLE 2.

Table 2: Parameters of experiments which are implemented by two different groups. Parameter | Krinner [12] | Zhao [13]
---|---|---
1Q gate duration $T_{g,1}$($\mathrm{ns}$) | 40 | 25
2Q gate duration $T_{g,2}$($\mathrm{ns}$) | 98 | 32
Lifetime $T_{1}$($\mathrm{\mu s}$) | 32.5 | 26.1
Coherence Time $T_{2}^{\ast}$($\mathrm{\mu s}$) | 37.5 | 3.6
1Q gate error $p_{1Q}$(%) | 0.09 | 0.098
2Q gate error $p_{2Q}$(%) | 1.5 | 1.035
Measurement duration $T_{M}$($\mathrm{ns}$) | 300 | 1500
Measurement error $p_{M}$(%) | 0.9 | 4.752
Logical Lifetime $T_{1,L}$($\mathrm{\mu s}$) | 16.4 | 64.4
Logical Coherence Time $T_{2,L}^{\ast}$($\mathrm{\mu s}$) | 18.2 | 69.0

For $p_{1Q},\,p_{2Q}$ in the table, Krinner’s group evaluated it by
Interleaved Randomized Benchmarking (Interleaved RB) and Zhao’s group
evaluated it by Cross Entropy Benchmarking (XEB). Note that while these are
great progress toward experimental QEC, neither group has succeeded in QEC in
a strict sense. Krinner’s group has implemented a QEC protocol, but was unable
to make the lifetime of the logical qubit $T_{1,L}$ longer than the lifetime
of the physical one $T_{1}$. Zhao’s group has only implemented the error
detection and postselection. The main reason for the lack of successful error
correction would be the short code distance. The distance 3 code cannot
correct two-qubit errors that occur during two-qubit gates.

Let us see whether or not a successful QEC is reachable if the code distance
is increased to 5 and consider what elements should be improved if it is not
the case. Since $p_{2Q}$ is $1.5\,\%,\,1.035\,\%$, which is outside the region
where $p_{L}=Ap^{\frac{d+1}{2}}$ holds in Figure 5, QEC is expected to fail
even if the code distance is increased to 5. The ratios of the two-qubit gate
and coherence time are $T_{g,2}/T_{1}=0.003$ and $=0.001$ for Krinners’ and
Zhaos’ groups, respectively. This is sufficiently small from our analysis.
However, $p_{2Q}$ are, respectively, given by $1.5\,\%$ and $1.035\,\%$, which
is much higher than those limited by the coherence time. This implies that
this infidelity caused by a systematic control error or cross-talk resulting
in coherent errors. Therefore, by improving their control strategies, there
would be a great possibility to achieve a breakeven point by experimental QEC
in the near future; at least the coherence time does not become the main
obstacle for it.

##  IV Conclusion

In this study, we constructed a framework to fully simulate QEC on the
distance 5 rotated surface code under an arbitrary local noise. Furthermore,
we have constructed an effective model that explains the behaviour of the
logical error probability under the coherent errors within the stochastic
Pauli noise model with an appropriate modification. Therefore, combining our
numerical result and the effective model, we can analyse the behaviour of the
logical error probability with smaller a physical error probability or larger
code distance. While we only modelled the over-rotation caused by a systematic
control error, there are plenty of sources of coherent errors such as
unintended interactions in Hamiltonian, cross-talk, global fields and so on.
These sources of noise would be straightforwardly incorporated into our
framework. The performance analysis under a realistic noise model is becoming
increasingly important, and our framework will provide a vital guideline for
future improvements on experimental sides.

## References

  * Shor [1996] P. Shor, Fault-tolerant quantum computation, in  _Proceedings of 37th Conference on Foundations of Computer Science_ (1996) pp. 56–65.
  * Zhang  _et al._ [2012] J. Zhang, R. Laflamme, and D. Suter, Experimental implementation of encoded logical qubit operations in a perfect quantum error correcting code, Physical review letters 109, 100503 (2012).
  * Abobeih  _et al._ [2022] M. Abobeih, Y. Wang, J. Randall, S. Loenen, C. Bradley, M. Markham, D. Twitchen, B. Terhal, and T. Taminiau, Fault-tolerant operation of a logical qubit in a diamond quantum processor, Nature , 1 (2022).
  * Egan  _et al._ [2021] L. Egan, D. M. Debroy, C. Noel, A. Risinger, D. Zhu, D. Biswas, M. Newman, M. Li, K. R. Brown, M. Cetina, _et al._ , Fault-tolerant control of an error-corrected qubit, Nature 598, 281 (2021).
  * Luo  _et al._ [2021] Y.-H. Luo, M.-C. Chen, M. Erhard, H.-S. Zhong, D. Wu, H.-Y. Tang, Q. Zhao, X.-L. Wang, K. Fujii, L. Li, _et al._ , Quantum teleportation of physical qubits into logical code spaces, Proceedings of the National Academy of Sciences 118 (2021).
  * Andersen  _et al._ [2020] C. K. Andersen, A. Remm, S. Lazar, S. Krinner, N. Lacroix, G. J. Norris, M. Gabureac, C. Eichler, and A. Wallraff, Repeated quantum error detection in a surface code, Nature Physics 16, 875 (2020).
  * AI [2021] G. Q. AI, Exponential suppression of bit or phase errors with cyclic error correction, Nature 595, 383 (2021).
  * Dennis  _et al._ [2002] E. Dennis, A. Kitaev, A. Landahl, and J. Preskill, Topological quantum memory, Journal of Mathematical Physics 43, 4452 (2002).
  * Horsman  _et al._ [2012] C. Horsman, A. G. Fowler, S. Devitt, and R. Van Meter, Surface code quantum computing by lattice surgery, New Journal of Physics 14, 123011 (2012).
  * Arute  _et al._ [2019] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. S. L. Brandao, D. A. Buell, B. Burkett, Y. Chen, Z. Chen, B. Chiaro, R. Collins, W. Courtney, A. Dunsworth, E. Farhi, B. Foxen, A. Fowler, C. Gidney, M. Giustina, R. Graff, K. Guerin, S. Habegger, M. P. Harrigan, M. J. Hartmann, A. Ho, M. Hoffmann, T. Huang, T. S. Humble, S. V. Isakov, E. Jeffrey, Z. Jiang, D. Kafri, K. Kechedzhi, J. Kelly, P. V. Klimov, S. Knysh, A. Korotkov, F. Kostritsa, D. Landhuis, M. Lindmark, E. Lucero, D. Lyakh, S. Mandrà, J. R. McClean, M. McEwen, A. Megrant, X. Mi, K. Michielsen, M. Mohseni, J. Mutus, O. Naaman, M. Neeley, C. Neill, M. Y. Niu, E. Ostby, A. Petukhov, J. C. Platt, C. Quintana, E. G. Rieffel, P. Roushan, N. C. Rubin, D. Sank, K. J. Satzinger, V. Smelyanskiy, K. J. Sung, M. D. Trevithick, A. Vainsencher, B. Villalonga, T. White, Z. J. Yao, P. Yeh, A. Zalcman, H. Neven, and J. M. Martinis, Quantum supremacy using a programmable superconducting processor, Nature 574, 505 (2019).
  * Wu  _et al._ [2021] Y. Wu, W.-S. Bao, S. Cao, F. Chen, M.-C. Chen, X. Chen, T.-H. Chung, H. Deng, Y. Du, D. Fan, M. Gong, C. Guo, C. Guo, S. Guo, L. Han, L. Hong, H.-L. Huang, Y.-H. Huo, L. Li, N. Li, S. Li, Y. Li, F. Liang, C. Lin, J. Lin, H. Qian, D. Qiao, H. Rong, H. Su, L. Sun, L. Wang, S. Wang, D. Wu, Y. Xu, K. Yan, W. Yang, Y. Yang, Y. Ye, J. Yin, C. Ying, J. Yu, C. Zha, C. Zhang, H. Zhang, K. Zhang, Y. Zhang, H. Zhao, Y. Zhao, L. Zhou, Q. Zhu, C.-Y. Lu, C.-Z. Peng, X. Zhu, and J.-W. Pan, Strong quantum computational advantage using a superconducting quantum processor, Phys. Rev. Lett. 127, 180501 (2021).
  * Krinner  _et al._ [2022] S. Krinner, N. Lacroix, A. Remm, A. Di Paolo, E. Genois, C. Leroux, C. Hellings, S. Lazar, F. Swiadek, J. Herrmann, _et al._ , Realizing repeated quantum error correction in a distance-three surface code, Nature 605, 669 (2022).
  * Zhao  _et al._ [2022] Y. Zhao, Y. Ye, H.-L. Huang, Y. Zhang, D. Wu, H. Guan, Q. Zhu, Z. Wei, T. He, S. Cao, F. Chen, T.-H. Chung, H. Deng, D. Fan, M. Gong, C. Guo, S. Guo, L. Han, N. Li, S. Li, Y. Li, F. Liang, J. Lin, H. Qian, H. Rong, H. Su, L. Sun, S. Wang, Y. Wu, Y. Xu, C. Ying, J. Yu, C. Zha, K. Zhang, Y.-H. Huo, C.-Y. Lu, C.-Z. Peng, X. Zhu, and J.-W. Pan, Realization of an error-correcting surface code with superconducting qubits, Phys. Rev. Lett. 129, 030501 (2022).
  * Bluvstein  _et al._ [2022] D. Bluvstein, H. Levine, G. Semeghini, T. T. Wang, S. Ebadi, M. Kalinowski, A. Keesling, N. Maskara, H. Pichler, M. Greiner, _et al._ , A quantum processor based on coherent transport of entangled atom arrays, Nature 604, 451 (2022).
  * Córcoles  _et al._ [2015] A. D. Córcoles, E. Magesan, S. J. Srinivasan, A. W. Cross, M. Steffen, J. M. Gambetta, and J. M. Chow, Demonstration of a quantum error detection code using a square lattice of four superconducting qubits, Nature communications 6, 1 (2015).
  * Nielsen and Chuang [2010] M. A. Nielsen and I. L. Chuang,  _Quantum Computation and Quantum Information: 10th Anniversary Edition_ (Cambridge University Press, 2010).
  * Suzuki  _et al._ [2017] Y. Suzuki, K. Fujii, and M. Koashi, Efficient simulation of quantum error correction under coherent error based on the nonunitary free-fermionic formalism, Physical review letters 119, 190503 (2017).
  * Tomita and Svore [2014] Y. Tomita and K. M. Svore, Low-distance surface codes under realistic quantum noise, Physical Review A 90, 062320 (2014).
  * Darmawan and Poulin [2017] A. S. Darmawan and D. Poulin, Tensor-network simulations of the surface code under realistic noise, Physical review letters 119, 040502 (2017).
  * Hakkaku  _et al._ [2021] S. Hakkaku, K. Mitarai, and K. Fujii, Sampling-based quasiprobability simulation for fault-tolerant quantum error correction on the surface codes under coherent noise, Phys. Rev. Research 3, 043130 (2021).
  * Chow  _et al._ [2011] J. M. Chow, A. D. Córcoles, J. M. Gambetta, C. Rigetti, B. R. Johnson, J. A. Smolin, J. R. Rozen, G. A. Keefe, M. B. Rothwell, M. B. Ketchen, and M. Steffen, Simple all-microwave entangling gate for fixed-frequency superconducting qubits, Phys. Rev. Lett. 107, 080502 (2011).
  * Krantz  _et al._ [2019] P. Krantz, M. Kjaergaard, F. Yan, T. P. Orlando, S. Gustavsson, and W. D. Oliver, A quantum engineer’s guide to superconducting qubits, Applied Physics Reviews 6, 021318 (2019), https://doi.org/10.1063/1.5089550 .
  * Suzuki  _et al._ [2021] Y. Suzuki, Y. Kawase, Y. Masumura, Y. Hiraga, M. Nakadai, J. Chen, K. M. Nakanishi, K. Mitarai, R. Imai, S. Tamiya, _et al._ , Qulacs: a fast and versatile quantum circuit simulator for research purpose, Quantum 5, 559 (2021).
  * Hagberg  _et al._ [2008] A. A. Hagberg, D. A. Schult, and P. J. Swart, Exploring network structure, dynamics, and function using NetworkX, in  _In Proceedings of the 7th Python in Science Conference (SciPy2008)_ (2008) pp. 11–15.
  * Sheldon  _et al._ [2016] S. Sheldon, L. S. Bishop, E. Magesan, S. Filipp, J. M. Chow, and J. M. Gambetta, Characterizing errors on qubit operations via iterative randomized benchmarking, Physical Review A 93, 012301 (2016).