# Boosting the interpretability of clinical risk scores with intervention
predictions

Eric Loreaux<EMAIL_ADDRESS>Google ResearchPalo AltoCAUSA ,  Ke Yu
<EMAIL_ADDRESS>0000000198825729 University of PittsburghPittsburghPAUSA ,
Jonas Kemp<EMAIL_ADDRESS>Google ResearchPalo AltoCAUSA ,  Martin
Seneviratne  Google ResearchLondonUK ,  Christina Chen  Google ResearchPalo
AltoCAUSA ,  Subhrajit Roy  Google ResearchLondonUnited Kingdom ,  Ivan
Protsyuk  Google ResearchLondonUnited Kingdom ,  Natalie Harris  Google
ResearchLondonUnited Kingdom ,  Alexander D’Amour  Google
ResearchCambridgeMAUSA ,  Steve Yadlowsky  Google ResearchCambridgeMAUSA and
Ming-Jun Chen  Google ResearchPalo AltoCAUSA

(2022)

###### Abstract.

Machine learning systems show significant promise for forecasting patient
adverse events via risk scores. However, these risk scores implicitly encode
assumptions about future interventions that the patient is likely to receive,
based on the intervention policy present in the training data. Without this
important context, predictions from such systems are less interpretable for
clinicians. We propose a joint model of intervention policy and adverse event
risk as a means to explicitly communicate the model’s assumptions about future
interventions. We develop such an intervention policy model on MIMIC-III, a
real world de-identified ICU dataset, and discuss some use cases that
highlight the utility of this approach. We show how combining typical risk
scores, such as the likelihood of mortality, with future intervention
probability scores leads to more interpretable clinical predictions.

electronic health records, intervention forecasting, model interpretability,
multi-label prediction, clinical use case

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: 2022 KDD Workshop on Applied Data Science for
Healthcare; August 14–17, 2022; Washington DC††booktitle: 2022 KDD Workshop on
Applied Data Science for Healthcare††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Applied computing Health informatics††ccs:
Computing methodologies Machine learning approaches

##  1\. Introduction

The digitization of medical data has created many opportunities for machine
learning systems to empower clinicians and improve medical care (Ghassemi et
al., 2020). One key area of interest is the use of predictive models to
quantify risk of mortality or deterioration, broadly referred to as acuity.
Most established patient acuity scores depend on a small number of
physiological risk factors determined through consensus of various medical
bodies, e.g. APACHE (Knaus et al., 1991), SOFA (Vincent et al., 1996), and
SAPS II (Le Gall et al., 1993). In recent years, deep learning-based
frameworks have improved upon the accuracy of these scores by replacing simple
risk factors with a wider breadth of patterns that are not as easily
detectable by human clinicians (Shickel et al., 2017).

Figure 1. Two patients whose predicted risks are equally low ($1\%$), but
whose clinical courses differ. For patient 2, the prediction may be predicated
on assumptions about future treatments. We aim to communicate these
assumptions.

While more powerful risk scores are promising, critical deployment challenges
remain, including interpretability (Kelly et al., 2019). Most clinical
interpretability methods are aimed at explaining how input features influence
model predictions, but these explanations fail to communicate an important
piece of clinical context: the model’s implicit, learned assumptions about
likely future interventions given a patient’s current state. Fig. 1 visualizes
the problem of two equivalently low risk predictions being predicated upon
completely different presumed intervention trajectories, which cannot be
communicated in a standalone acuity score. Such implicit assumptions have been
shown to negatively impact model behavior in real-world deployment scenarios.
For example, (Caruana et al., 2015) discusses a case in which a model trained
to produce mortality risk scores for pneumonia patients learns to ascribe
lower risk to patients with asthma, when in fact the opposite should be true.
This behavior is explained by the fact that asthma patients receive more
aggressive and effective interventions than other patients. This case
highlights the need for interpretability beyond simple feature attribution, in
which the assumptions about future interventions are made clearer.

In this work, we demonstrate that communicating intervention assumptions in
addition to the patient’s future risk has the potential to improve
interpretability and increase trust among end users considering said risk
scores in care planning. In order to demonstrate the utility of this risk
score contextualization, we develop a recurrent multitask neural network which
simultaneously predicts the likelihood of both mortality and a collection of
relevant clinical interventions. We evaluate this approach on the MIMIC-III
dataset and highlight examples in which the additional context of future
intervention probabilities can enable a machine learning system to produce
more interpretable predictions for clinicians.

##  2\. Related Work

It is well known that patient risk is highly dependent on intervention.
(Lenert et al., 2019) simulated adverse event label drift in response to
changing intervention, and recommend incorporating intervention predictions as
a solution, which we explore further in this work. (van Geloven et al., 2019)
formally defined a unified endpoint combining adverse events and interventions
after baseline. Rather than change the definition of the risk being predicted,
we seek to use predictions of future interventions to better contextualize
this risk.

We are by no means the first to propose the prediction of multiple clinical
interventions. (Ghassemi et al., 2019) employed a switching state space model
to represent patient states and evaluated the performance of learned states in
prediction of five ICU treatments. (Suresh et al., 2017) integrated multiple
EHR data sources and used the learned representation to predict onset and
weaning of five interventions. (Wang et al., 2020) built a pipeline to extract
continuous intervention signals for ventilators, vassopressors, and fluid
therapies. We extend upon previous intervention prediction tasks (i.e., to
thirteen interventions) and specifically demonstrate how such predictions can
be used to increase the interpretability of patient acuity scores.

There have been many efforts to improve the interpretability of clinical
machine learning systems. (Payrovnaziri et al., 2020) provide a comprehensive
overview of previous work. Most of these techniques focus on attributing model
outputs to individual features, or to clinical concepts (e.g. (Mincu et al.,
2021)). We diverge from this work by communicating the model’s implicit
assumptions about future events that may have a causal relationship with the
predicted endpoint.

##  3\. Data and Label Description

Data We used MIMIC-III (Johnson et al., 2016) for our experiments. MIMIC-III
is a publicly available dataset that includes the de-identified electronic
health records (EHRs) of 53,423 patients admitted to critical care units at
the Beth Israel Deaconess Medical Center from 2001 to 2012. We selected the
study cohort based on three inclusion criteria: (1) the patient was at least
15 years old at the time of admission; (2) following (Wang et al., 2020), we
only considered each patient’s first ICU stay to prevent possible information
leakage; (3) the total duration of the ICU stay was between 2 and 14 days. The
final cohort included 18,335 ICU stays and was randomly split into training
(80%), validation (10%), and test (10%). Table C.3 describes the cohort
demographics.

Label description We focused on the early prediction of interventions as
auxiliary tasks, jointly trained with in-ICU mortality prediction. All
predictions are binary classification tasks that use all historical
information up to and including the first 24 hours of a patient’s first ICU
stay, and a prediction horizon that extends to the end of that same ICU stay.
We targeted 13 commonly used interventions in ICU, including ICU medications
(e.g., vasopressors), fluid boluses (e.g., crystalloid bolus), transfusions
(e.g., RBC transfusion), and mechanical ventilation. To the best of our
knowledge, no other work has included as many simultaneous intervention
prediction tasks. Definitions of interventions were based on the SQL queries
available in the MIMIC-III codebase (Johnson et al., 2018) and the item IDs of
corresponding concepts were reviewed and modified with clinical guidance where
appropriate (see Appdx. A for detailed definitions). The in-ICU mortality
label was derived using the “DEATHTIME” included in the MIMIC-III “ADMISSION”
table. A half-hour tolerance period was added to the “OUTTIME” in the
“ICUSTAYS” table to account for patients who died within the 30 minutes of ICU
discharge. Table C.4 shows the prevalence of in-ICU mortality as well as all
the intervention labels split by train, validation, and test data sets.

##  4\. Methodology

###  4.1. Baseline Models

We built two logistic regression models based on features used in SOFA and
SAPS-II, respectively. The SOFA baseline model included six predictors that
measure the rates of failure of six organ systems. The SAPS-II baseline model
included seventeen variables, including routine physiological measurements,
comorbidity, and admission status. Both SOFA and SAPS-II scores are commonly
used to estimate the probability of mortality for ICU patients. In this study,
we investigate their effectiveness in proposed prediction tasks, including
both mortality and intervention predictions.

###  4.2. LSTM-based multi-label learning model

We extended an open-source implementation111https://github.com/google/ehr-
predictions of the LSTM-based mortality predictive model from (Tomašev et al.,
2021) to incorporate multi-label intervention prediction tasks as defined in
Sec. 3. The model used a version of MIMIC-III mapped to the Fast Healthcare
Interoperability Resource (FHIR) standard as described in (Rajkomar et al.,
2018). Every patient’s record was represented as a collection of timestamped
events with an associated clinical code and, where applicable, a value. We
obtained the feature representation from FHIR resources as described in (Roy
et al., 2021).

Loss Function We employed a multi-label learning approach to train the model
$f$. A patient’s input tensor $x^{(i)}$ is fed through a sparse embedding
layer followed by an LSTM (Hochreiter and Schmidhuber, 1997) module
parameterized by $\theta$. The output vector of the LSTM (i.e., hidden patient
representation) is then shared across fourteen different classification heads,
each of which is parameterized by $\varphi_{j}$. We used binary cross-entropy
as the loss function for each classification task and minimize their sum to
train the entire network. The complete loss function is:

$\displaystyle\mathcal{L}=-$
$\displaystyle\sum_{j}^{P}\sum_{i}^{N}y_{j}^{(i)}\log
f(x^{(i)};\theta,\varphi_{j})$
$\displaystyle+(1-y_{j}^{(i)})\log\big{(}1-f(x^{(i)};\theta,\varphi_{j})\big{)},$

where $N$ is the number of training samples and $P$ is the total number of
classification tasks. In our experiments, $P=14$ which includes one in-ICU
mortality prediction task and thirteen intervention prediction tasks. For
implementation details, we refer the reader to Appdx. B.

Figure 2. 2D t-SNE visualization of the LSTM-based intervention predictions.
Each point represents one patient. The left panel is labeled by K-means
cluster assignment ($K=9$) using the 13 intervention predictions as inputs.
The middle panel is labeled by the quantile group of predicted mortality risk.
The right panel is labeled by the quantile group of total predicted
intervention scores, summed across the 13 interventions.

##  5\. Experiments and Results

We first present quantitative results, then show both aggregate and individual
use cases that demonstrate the benefits of pairing mortality prediction with
intervention predictions. All experimental results are based on the held out
test set.

###  5.1. Model Results and Comparison

Table 1. Model performance comparison. Bold face indicates the highest average
AUC scores.

Task |  | AUROC |  |  | AUPRC |
---|---|---|---|---|---|---
SOFA | SAPS II | LSTM | SOFA | SAPS II | LSTM
In-ICU Mortality | $0.69$ | $0.76$ | $\bm{0.82\pm 0.01}$ | $0.23$ | $0.27$ | $\bm{0.39\pm 0.03}$
Vasopressors | $0.66$ | $0.70$ | $\bm{0.74\pm 0.00}$ | $0.30$ | $0.31$ | $\bm{0.37\pm 0.01}$
Inotropes | $0.70$ | $0.69$ | $\bm{0.77\pm 0.02}$ | $0.10$ | $0.10$ | $\bm{0.16\pm 0.01}$
Sedation | $0.57$ | $0.65$ | $\bm{0.76\pm 0.01}$ | $0.33$ | $0.38$ | $\bm{0.50\pm 0.01}$
Analgesic | $0.57$ | $0.66$ | $\bm{0.85\pm 0.01}$ | $0.34$ | $0.38$ | $\bm{0.64\pm 0.02}$
Anticoagulation | $0.59$ | $0.58$ | $\bm{0.86\pm 0.01}$ | $0.38$ | $0.36$ | $\bm{0.77\pm 0.02}$
Diuretic | $0.65$ | $0.68$ | $\bm{0.88\pm 0.01}$ | $0.29$ | $0.30$ | $\bm{0.68\pm 0.02}$
Paralytic | $0.71$ | $0.86$ | $\bm{0.96\pm 0.02}$ | $0.06$ | $0.18$ | $\bm{0.27\pm 0.13}$
Colloid Bolus | $0.66$ | $0.72$ | $\bm{0.78\pm 0.01}$ | $0.12$ | $0.18$ | $\bm{0.26\pm 0.02}$
Crystalloid Bolus | $0.57$ | $0.61$ | $\bm{0.65\pm 0.01}$ | $0.48$ | $0.51$ | $\bm{0.55\pm 0.01}$
FFP Transfusion | $\bm{0.75}$ | $0.67$ | $0.71\pm 0.01$ | $\bm{0.24}$ | $0.16$ | $0.17\pm 0.03$
RBC Transfusion | $0.65$ | $0.68$ | $\bm{0.76\pm 0.02}$ | $0.45$ | $0.45$ | $\bm{0.57\pm 0.02}$
Ventilation | $0.53$ | $0.55$ | $\bm{0.63\pm 0.01}$ | $0.14$ | $0.15$ | $\bm{0.21\pm 0.02}$
Antibiotic | $0.58$ | $0.56$ | $\bm{0.90\pm 0.00}$ | $0.88$ | $0.88$ | $\bm{0.98\pm 0.00}$

We evaluated the model using AUROC and AUPRC scores and present the values in
Tab. 1. We find that, compared to the baseline models using severity score
features, the LSTM-based model yields improvements on most intervention
predictions, except for two (paralytic, FFP transfusion) with very low label
prevalence. We also find that compared to the prediction of in-ICU mortality,
four intervention predictions have higher AUROC scores and eight have higher
AUPRC scores, suggesting that these predictions can offer more specific and
reliable signals for predicting patient needs than a mortality risk score.

Table 2. Mean values and standard deviations for precision by the number of
interventions ($I$) that patient were given. Bold face indicates the highest
mean values. Precision at $I$ is the proportion of top $I$ predicted
interventions that are in the patient’s recorded intervention set.

Interventions |  |  Precision at $I$ |
---|---|---|---
SOFA |  SAPS II |  LSTM
$I=1$ |  $0.84\pm 0.37$ |  $0.84\pm 0.37$ |  $\textbf{0.89}\pm\textbf{0.31}$
$I=2$ |  $0.63\pm 0.26$ |  $0.65\pm 0.25$ |  $\textbf{0.76}\pm\textbf{0.26}$
$I=3$ |  $0.62\pm 0.22$ |  $0.64\pm 0.21$ |  $\textbf{0.73}\pm\textbf{0.20}$
$I=4$ |  $0.61\pm 0.18$ |  $0.62\pm 0.18$ |  $\textbf{0.71}\pm\textbf{0.18}$
$I=5$ |  $0.66\pm 0.18$ |  $0.68\pm 0.18$ |  $\textbf{0.76}\pm\textbf{0.15}$
$I\geq 6$ |  $0.77\pm 0.13$ |  $0.77\pm 0.13$ |  $\textbf{0.81}\pm\textbf{0.12}$

We evaluated the model’s capability of ranking the most relevant interventions
for each patient. We grouped patients by the number of interventions (denoted
as $I$) that they received and evaluated our model with Precision@$I$ metrics,
which is the proportion of top $I$ predicted interventions that are in the
patient’s recorded intervention set. Tab. 2 shows that our LSTM-based model
yields fairly high retrieval precision, with mean values $>0.7$, at all
levels, and compares favorably to baseline models.

We also evaluated the models for calibration performance and for the
relationship between in-ICU mortality prediction and intervention predictions.
We present these results in Appdx. D.

Figure 3. Comparison of the future intervention probabilities of patients A
and B. Both have similar medical profiles and predicted mortality risk, and
yet their very different predicted interventions indicate that their mortality
risk is predicated on distinct assumed clinical trajectories.

###  5.2. Use Cases

Cluster analysis: We created a t-SNE (Laurens and Hinton, 2008) plot using the
thirteen intervention predictions of the LSTM-based model as inputs to obtain
the 2D t-SNE embedding. Fig. 2 shows the same t-SNE plot labeled by three
different categorical variables: K-means cluster assignment using the entire
future intervention probability vector, quantile group of predicted mortality
risk, and quantile group of total predicted intervention score.

Cluster 8 (left panel, grey points at bottom) in Fig. 2 represents a set of
patients who have low predicted mortality risk (center panel, light red) but
require extensive medical interventions (right panel, dark green). Upon
clinical review, we identified these patients as cardiothoracic surgery
patients (coronary artery bypass grafts (CABG), valve replacements, etc.)
receiving post-surgical care. Fig. D.7 shows that patients in cluster 8 were
more likely to receive diuretics to prevent fluid overload and antibiotics to
manage post-surgical infection, but were less likely to be sedated or
ventilated in the ICU. These predictions indicate to the clinician that the
patient risk score they are viewing is already predicated on this future post-
surgical trajectory, without which, risk may have been much higher.

Patient trajectories: To illustrate the potential of intervention forecasting
to boost interpretability at the patient level, we selected two patients (A
and B) with similar risk scores but differing intervention predictions. Both
were 80-85 year old males admitted to the ICU after multi-vessel CABG; both
were extubated on day 1 post-operatively with 24h risk scores in the bottom
tenth percentile. Fig. 3 shows that despite similar risk, Patient A had a
higher probability of receiving diuretics, whereas Patient B had a
significantly higher probability of requiring transfusion, crystalloid bolus
and vasopressors. This forecast matches the differing ICU trajectories of
these patients. While patient A’s admission was complicated by tachycardia and
hypertension, patient B had a more complicated admission requiring a post-op
transfusion, vasopressors after a hypotensive episode, chemical cardioversion
and anticoagulation for new atrial fibrillation, and intravenous antibiotics
for phlebitis. These additional signals indicate to the clinician that the
very similar risk scores for these two similar patients are actually
predicated on distinct clinical trajectories.

##  6\. Conclusion and Future Work

In this work, we demonstrate ways in which modeling intervention policy can
improve the interpretability of clinical risk scores. Providing this context
has the potential to increase confidence and trust as well as improve inter-
provider variability in practice through personalized patient forecasts. This
interpretability technique is applicable to any prediction task in which the
predictions depend on future actions which may have a causal relationship with
the target endpoint. It is especially critical to communicate these
assumptions when the consumer of the predictions is also implicated in these
future actions.

In future work, we hope to expand upon the findings presented here by applying
our policy model to health environments beyond the ICU, such as in general
inpatient hospital wards or in the outpatient setting. The heterogeneity of
care trajectories is wider in these settings, and so the potential ambiguity
regarding patient risk forecasts is amplified. In addition to expanding to new
datasets, we would like to explore a number of improvements to the quality and
granularity of our intervention policy models, such as shifting to continuous
predictions and increasing the number and sophistication of intervention
patterns forecasted. It will also be imperative to conduct user testing with
clinicians to determine clinical benefit and potential for integration into
real-world workflows.

## References

  * (1)
  * Caruana et al. (2015) Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015\.  Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-Day Readmission. In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ _(KDD ’15)_. Association for Computing Machinery, New York, NY, USA, 1721–1730.  https://doi.org/10.1145/2783258.2788613
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.  _EMNLP_ (2014), 1724–1734.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  A Theoretically Grounded Application of Dropout in Recurrent Neural Networks.  _arXiv preprint arXiv:1512.05287_ (2016).
  * Ghassemi et al. (2020) Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew Beam, Irene Chen, and Rajesh Ranganath. 2020\.  A Review of Challenges and Opportunities in Machine Learning for Health.  _AMIA Joint Summits on Translational Science_ (2020), 191–200.
  * Ghassemi et al. (2019) Marzyeh Ghassemi, Mike Wu, Michael Hughes, Peter Szolovits, and Finale Doshi-Velez. 2019.  Predicting intervention onset in the ICU with switching state space models.  _AMIA Joint Summits on Translational Science_ (2019), 82–91.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010.  Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 249–256.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016\.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ 3, 1 (2016), 1–9.
  * Johnson et al. (2018) Alistair EW Johnson, David J Stone, Leo A Celi, and Tom J Pollard. 2018.  The MIMIC Code Repository: enabling reproducibility in critical care research.  _Journal of the American Medical Informatics Association_ 25, 1 (2018), 32–39.
  * Kelly et al. (2019) Christopher Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. 2019\.  Key challenges for delivering clinical impact with artificial intelligence.  _BMC Med_ 17 (2019), 195.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Knaus et al. (1991) W. A. Knaus, D. Wagner, E. Draper, and J. Zimmerman. 1991\.  The APACHE III prognostic system. risk prediction of hospital mortality for critically ill hospitalized adults.  _Chest_ 100, 6 (1991), 1619–1636.
  * Laurens and Hinton (2008) Van Der Maaten Laurens and GJJoMLR Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ 9, 2605 (2008), 2579–2605.
  * Le Gall et al. (1993) Jean-Roger Le Gall, Stanley Lemeshow, and Fabienne Saulnier. 1993.  A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study.  _Jama_ 270, 24 (1993), 2957–2963.
  * Lenert et al. (2019) Matthew Lenert, Matheny Michael, and Walsh Colin. 2019\.  Prognostic models will be victims of their own success, unless…  _Journal of the American Medical Informatics Association_ (2019), 1645–1650.
  * Mincu et al. (2021) Diana Mincu, Eric Loreaux, Shaobo Hou, Sebastien Baur, Ivan Protsyuk, Martin G Seneviratne, Anne Mottram, Nenad Tomasev, Alan Karthikesanlingam, and Jessica Schrouff. 2021.  Concept-based model explanations for Electronic Health Records. In _Proceedings of the Conference on Health, Inference, and Learning_. ACM CHIL.
  * Payrovnaziri et al. (2020) Seyedeh Neelufar Payrovnaziri, Zhaoyi Chen, Pablo Rengifo-Moreno, Tim Miller, Jiang Bian, Jonathan H Chen, Xiuwen Liu, and Zhe He. 2020.  Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review.  _Journal of the American Medical Informatics Association_ (2020).  https://doi.org/10.1093/jamia/ocaa053
  * Rajkomar et al. (2018) Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. 2018\.  Scalable and accurate deep learning with electronic health records.  _NPJ Digital Medicine_ 1, 1 (2018), 1–10.
  * Roy et al. (2021) Subhrajit Roy, Diana Mincu, Eric Loreaux, Anne Mottram, Ivan Protsyuk, Natalie Harris, Emily Xue, Jessica Schrouff, Hugh Montgomery, Ali Connell, Nenad Tomašev, Alan Karthikesalingam, and Martin Seneviratne. 2021.  Multi-task prediction of organ dysfunction in the ICU using sequential sub-network routing.  _Journal of the American Medical Informatics Association (JAMIA)_ (2021).  https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab101/6307184
  * Shickel et al. (2017) Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. 2017.  Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis.  _IEEE journal of biomedical and health informatics_ 22, 5 (2017), 1589–1604.
  * Suresh et al. (2017) Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. 2017.  Clinical intervention prediction and understanding with deep neural networks. In _Machine Learning for Healthcare Conference_. PMLR, 322–337.
  * Tao et al. (2018) Lei Tao, Zhang Yu, Sida I. Wang, Dai Hui, and Artzi Yoav. 2018.  Simple Recurrent Units for Highly Parallelizable Recurrence.  _EMNLP_ (2018), 4470–4481.
  * Tomašev et al. (2021) Nenad Tomašev, Natalie Harris, Sebastien Baur, Anne Mottram, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Valerio Magliulo, et al. 2021\.  Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records.  _Nature Protocols_ 16, 6 (2021), 2765–2787.
  * van Geloven et al. (2019) Nan van Geloven, Sonja Swanson, Chava Ramspek, Kim Luijken, Merel van Diepen, Tim Morris, Rolf Groenwold, Hans van Houwelingen, Hein Putter, and Saskia le Cessie. 2019\.  Prediction meets causal inference: the role of treatment in clinical prediction models.  _Journal of the American Medical Informatics Association_ (2019), 1645–1650.
  * Vincent et al. (1996) J-L Vincent, Rui Moreno, Jukka Takala, Sheila Willatts, Arnaldo De Mendonça, Hajo Bruining, CK Reinhart, PeterM Suter, and Lambertius G Thijs. 1996.  The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure.
  * Wang et al. (2020) Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. 2020.  Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. In _Proceedings of the ACM Conference on Health, Inference, and Learning_. 222–235.

##  Appendix A Intervention Label Definition

  * •

Vasopressors: Vasopressors are a group of medications used to treat severely
low blood pressures by constricting blood vessels. The vasopressors label was
defined as the onset of any of the following 7 drugs: levophed, neosynephrine,
phenylephrine, norepinephrine, vasopressin, dopamine, or epinephrine.

  * •

Inotropes: Inotropes are medications that change the force of patients’
hearts’ contractions. In ICU, inotropes are used to stabilise patients’
circulation and optimise oxygen supply. The inotropes label was defined as the
onset of any of the following 3 drugs: dopamine, dobutamine, or milrinone.

  * •

Sedation: Sedative drugs are primarily used for the treatment of agitation and
anxiety caused by many different conditions, such as dyspnea, mechanical
ventilation, and untreated pain. The sedation label was defined as the onset
of any of the following 7 drugs: propofol, midazolam, ativan, dexmedetomidine,
diazepam, ketamine or pentobarbitol.

  * •

Analgesic: Analgesics are used for pain control. The analgesic label was
defined as the onset of any of the following 3 drugs: fentanyl, morphine
sulfate, or hydromorphone.

  * •

Anticoagulation: Anticoagulation is used for the management of venous
thromboembolism (VTE), atrial fibrillation (AF), mechanical heart valves, and
idiopathic pulmonary arterial hypertension (IPAH). The anticoagulation label
was defined as the onset of any of the following 7 drugs: heparin, integrelin,
argatroban, lepirudin, aggrastat, reopro, or bivalirudin.

  * •

Diuretic: Diuretics are a mainstay of treatment for managing fluid overload in
ICU. The diuretic label was defined as the onset of either of the following 2
drugs: furosemide or natrecor.

  * •

Paralytic: Paralytics, or neuromuscular blocking agents (NMBAS), paralyze
skeletal muscles by blocking the transmission of nerve impulses at the
myoneural junction. They are often deployed in the sickest patients in ICU
when usual care fails. The paralytic label was defined as the onset of any of
the following 3 drugs: cisatracurium, vecuronium, or atracurium.

  * •

Colloid Bolus: Colloids are gelatinous solutions that maintain a high osmotic
pressure in the blood. Colloid boluses are used to improve cardiovascular
function and organ perfusion and are often considered as less aggressive
alternatives to vasopressors. The colloid bolus label was derived from the
presence of item IDs defined in the SQL query colloid_bolus.sql provided on
the MIMIC Github repository222https://github.com/MIT-LCP/mimic-
code/tree/main/mimic-iii.

  * •

Crystalloid Bolus: Crystalloids, another type of fluid bolus, are defined as
solutions of ions that are capable of passing through semipermeable membranes.
The crystalloid bolus label was derived from the presence of item IDs defined
in the SQL query crystalloid_bolus.sql provided on the MIMIC Github
repository.

  * •

FFP Transfusion: Fresh frozen plasma (FFP) transfusions are widely used in ICU
patients to correct deficiency of coagulation factors or increased risk of
developing acute lung injury. The FFP transfusion label was derived from the
presence of item IDs defined in the SQL query ffp_transfusion.sql provided on
the MIMIC Github repository.

  * •

RBC Transfusion: Red blood cell (RBC) transfusions are commonly used in ICU
patients with increased mortality risk. The RBC transfusion label was derived
from the presence of item IDs defined in the SQL query rbc_transfusion.sql
provided on the MIMIC Github repository.

  * •

Ventilation: Mechanical ventilation is commonly used when an ICU patient
requires assistance for breathing. The ventilation label was based on the SQL
query ventilation_durations.sql provided on the MIMIC Github repository, and
was defined as the onset of a new mechanical ventilation event.

  * •

Antibiotic: The antibiotic label was based on the SQL query antibiotic.sql
provided on the MIMIC Github repository, with the following modifications: (1)
we only included drugs prescribed to be given through an intravenous (IV)
line, and (2) we ensured that there was a input event or chart event
associated with vancomycin if there was a prescription order of vancomycin.

##  Appendix B Implementation Details

We report the optimal hyperparameter configuration on the validation data set.
We trained a recurrent neural network with 3 layers of LSTM cells of size 200
with batch size 128. For the feature embedding, we used a sparse lookup table
embedding with an embedding dimension of 300. We used Xavier initialization
(Glorot and Bengio, 2010), Adam optimization (Kingma and Ba, 2014), and
learning rate 0.0001 with exponential decay 0.85 per 12,000 steps. We trained
for a total of 200,000 steps on an NVIDIA Tesla V100 GPU. For regularization,
we implemented L1 regularization on the sparse lookup embeddings with a
strength of 0.0005, and also input, output, and variational recurrent dropout
(Gal and Ghahramani, 2016) on all of our LSTM cells with a dropout probability
of 0.4.

We also explored the use of two additional recurrent cell architectures: the
gated recurrent unit (GRU) (Cho et al., 2014) and the simple recurrent unit
(SRU) (Tao et al., 2018), as well as two additional sparse lookup embedding
sizes (200, 400) and one additional initial learning rate (0.001). While we
found the LSTM to be the best performing architecture, we saw very little
difference across the two learning rates and three embedding dimensions.

##  Appendix C Data Summary Statistics

Table C.3. Cohort summary by demographic and admission variables. |  |  Gender |  | Total
---|---|---|---|---
|  |  F |  M |
Ethnicity |  Asian |  $181$ |  $258$ |  $439$ $(2\%)$
|  Hispanic |  $227$ |  $329$ |  $556$ $(3\%)$
|  Black |  $773$ |  $609$ |  $1,382$ $(8\%)$
|  Other |  $1,252$ |  $1,751$ |  $3,003$ $(16\%)$
|  White |  $5,718$ |  $7,237$ |  $12,955$ $(70\%)$
Age |  $<30$ |  $366$ |  $526$ |  $892$ $(5\%)$
|  31-50 |  $1,195$ |  $1,657$ |  $2,852$ $(16\%)$
|  51-70 |  $2,699$ |  $4,169$ |  $6,868$ $(37\%)$
|  $>70$ |  $3,891$ |  $3,832$ |  $7,723$ $(42\%)$
Insurance Type |  Self Pay |  $54$ |  $151$ |  $205$ $(1\%)$
|  Government |  $184$ |  $331$ |  $515$ $(3\%)$
|  Medicaid |  $625$ |  $800$ |  $1,425$ $(8\%)$
|  Private |  $2,257$ |  $3,606$ |  $5,863$ $(32\%)$
|  Medicare |  $5,031$ |  $5,296$ |  $10,327$ $(56\%)$
Admission Type |  Urgent |  $255$ |  $291$ |  $546$ $(3\%)$
|  Elective |  $1,143$ |  $1,599$ |  $2,742$ $(15\%)$
|  Emergency |  $6,753$ |  $8,294$ |  $15,047$ $(82\%)$
First Careunit |  TSICU |  $931$ |  $1,435$ |  $2,366$ $(13\%)$
|  CCU |  $1,226$ |  $1,603$ |  $2,829$ $(15\%)$
|  SICU |  $1,481$ |  $1,540$ |  $3,021$ $(16\%)$
|  CSRU |  $1,416$ |  $2,403$ |  $3,819$ $(21\%)$
|  MICU |  $3,097$ |  $3,203$ |  $6,300$ $(34\%)$
Total |  |  $8,151$ |  $10,184$ |  $18,335$ $(100\%)$
Table C.4. Summary statistics for the labels. |  Training |  Validation | Test
---|---|---|---
In-ICU Mortality |  $1,329$ $(8.9\%)$ |  $129$ $(7.4\%)$ |  $141$ $(8.2\%)$
Vasopressors |  $2,486$ $(16.7\%)$ |  $261$ $(15.1\%)$ |  $306$ $(17.9\%)$
Inotropes |  $667$ $(4.5\%)$ |  $70$ $(4.1\%)$ |  $77$ $(4.5\%)$
Sedation |  $3,766$ $(24.3\%)$ |  $431$ $(24.9\%)$ |  $431$ $(25.2\%)$
Analgesic |  $3,994$ $(26.8\%)$ |  $450$ $(24.9\%)$ |  $436$ $(25.5\%)$
Anticoagulation |  $4,439$ $(29.8\%)$ |  $518$ $(30.0\%)$ |  $525$ $(30.7\%)$
Diuretic |  $2,766$ $(18.6\%)$ |  $308$ $(17.8\%)$ |  $306$ $(17.9\%)$
Paralytic |  $130$ $(0.9\%)$ |  $14$ $(0.8\%)$ |  $16$ $(0.9\%)$
Colloid Bolus |  $928$ $(6.2\%)$ |  $101$ $(5.8\%)$ |  $99$ $(5.8\%)$
Crystalloid Bolus |  $5,897$ $(39.6\%)$ |  $655$ $(37.9\%)$ |  $683$ $(39.9\%)$
FFP Transfusion |  $783$ $(5.3\%)$ |  $87$ $(5.0\%)$ |  $103$ $(6.0\%)$
RBC Transfusion |  $4,602$ $(30.9\%)$ |  $516$ $(29.9\%)$ |  $524$ $(30.6\%)$
Ventilation |  $1,935$ $(13.0\%)$ |  $212$ $(12.3\%)$ |  $219$ $(12.8\%)$
Antibiotic |  $12,906$ $(86.6\%)$ |  $1,503$ $(86.9\%)$ |  $1,467$ $(85.7\%)$
Total |  $14,895$ $(100\%)$ |  $1,728$ $(100\%)$ |  $1,712$ $(100\%)$

##  Appendix D Supplementary Experimental Results and Details

Figure D.4. Calibration curves for in-ICU mortality and intervention
predictions. Figure D.5. Average future intervention probability at five
equally divided quantile levels (0%-20%, 20%-40%, 40%-60%, 60%-80%, 80%-100%)
of the predicted in-ICU mortality probability. Figure D.6. Average mortality
probability at the decile of future intervention probability. Figure D.7.
Average future intervention probabilities by cluster. Each spoke of the plot
is a different intervention. Blue lines indicate mean predicted probabilities
for that cluster, red lines indicate mean probabilities across all clusters.


# Fault-Tolerant Logical Gates in the IBM Quantum Experience

Robin Harper  Centre for Engineered Quantum Systems, School of Physics, The
University of Sydney, Sydney, Australia  Steven T. Flammia  Centre for
Engineered Quantum Systems, School of Physics, The University of Sydney,
Sydney, Australia  Yale Quantum Institute, Yale University, New Haven, CT
06520, USA  Centre for Engineered Quantum Systems, School of Physics, The
University of Sydney, Sydney, Australia  Centre for Engineered Quantum
Systems, School of Physics, The University of Sydney, Sydney, Australia  Yale
Quantum Institute, Yale University, New Haven, CT 06520, USA

###### Abstract

Quantum computers will require encoding of quantum information to protect them
from noise. Fault-tolerant quantum computing architectures illustrate how this
might be done, but have not yet shown a conclusive practical advantage. Here
we demonstrate that a small but useful error detecting code improves the
fidelity of the fault-tolerant gates implemented in the code space as compared
to the fidelity of physically equivalent gates implemented on physical qubits.
By running a randomized benchmarking protocol in the logical code space of the
[4,2,2] code, we observe an order of magnitude improvement in the infidelity
of the gates, with the two-qubit infidelity dropping from $5.8(2)\%$ to
$0.60(3)\%$. Our results are consistent with fault-tolerance theory and
conclusively demonstrate the benefit of carrying out computation in a code
space that can detect errors. Although the fault-tolerant gates offer an
impressive improvement in fidelity, the computation as a whole is not below
the fault-tolerance threshold because of noise associated with state
preparation and measurement on this device.

We have entered an exciting stage in the development of quantum computers.
Small scale, prototype quantum devices with a limited number of qubits are
beginning to appear, and companies such as IBM and Rigetti are making such
devices available in the cloud. Although current quantum devices tend to be
too small, have limited interconnectivity between qubits and are too noisy to
allow meaningful quantum computation, they are an important step forward in
the aim to build a large scale universal quantum computer. These small devices
are sufficient to act as the test bed for proof of principle concepts such as
the implementation of simplified quantum algorithms Coles  _et al._ (2018),
quantum walks Balu  _et al._ (2017), quantum machine learning Ristè  _et al._
(2017) and testing the ability to detect and correct errors.

Codes that can be implemented on current small noisy quantum devices are
perfect testbeds for the ideas of fault tolerance. The question we address
here is whether such a code can conclusively show a benefit for encoded
computation on current small scale noisy quantum devices. As we discuss later,
it is not obvious that the type of noise that exists in current quantum
devices will be amenable to such codes. We study the [4,2,2] error detecting
code, which is one of the smallest interesting codes. For instance, it can be
concatenated with the toric code Brell  _et al._ (2011); Criger and Terhal
(2016), can be viewed as one of the faces of the distance 3 color code Bombin
and Martin-Delgado (2006), or alternatively as an encoding layer of the C4/C6
code of Ref. Knill (2005).

Experimental quantum error correction and fault tolerance is still in its
infancy, but several impressive results have been achieved. Ref. Linke  _et
al._ (2017) demonstrates the ability to prepare logical qubits of the [4,2,2]
code fault tolerantly using trapped atomic ions and Ref. Takita  _et al._
(2017) replicates this on superconducting qubits. Refs. Kelly  _et al._
(2015); Wootton and Loss (2017) implement repetition codes and observe
protection against bit-flip errors. Ref. Ofek  _et al._ (2016) uses quantum
error correction to extend the lifetime of quantum information stored in a
superconducting resonator, and Refs. Barends  _et al._ (2014); Heeres  _et
al._ (2017) implement logical gates inside of a quantum code.

Recently, Gottesman (2016) suggested that the [4,2,2] code could be
implemented fault tolerantly with only five physical qubits. He argued that
for a small experiment to conclusively demonstrate fault tolerance, the
following must be met: (1) The encoded circuit must have a lower error rate
for all circuits in the family of circuits of interest. (2) They must be
complete circuits, i.e. they must include the initial state preparation and
the final measurements. (3) The original circuit and the encoded circuit must,
in the absence of error, produce the same output distribution. (4) It is only
meaningful to compare error rates between circuits implemented in the same
system. Additionally, Gottesman noted that for the purposes of demonstrating
fault tolerance it is still interesting to consider circuits created from non-
universal gate sets.

Vuillot (2017) attempted to meet Gottesman’s criteria using the [4,2,2] code
on the IBM Quantum Experience (IQX). Vuillot comprehensively explored the
different types of circuits available in the [4,2,2] code, though the
methodology used meant it was difficult to extract a clear signal from the
data as to the overall benefit (or lack thereof) in utilizing the [4,2,2] code
on that noisy device. More recently Willsch  _et al._ (2018) simulated the
[4,2,2] code numerically and tested it on the IQX. Their analysis suggests
that a fault tolerant protocol could improve performance provided that errors
are due to control errors rather than dominated by decoherence.

While the inclusion of state preparation and measurement (SPAM) errors in
Gottesman’s proposed protocol are necessary steps in demonstrating that a
computer might meet the fault tolerance threshold, the inclusion of such
errors is not necessary in demonstrating that the use of fault-tolerant gates
and appropriate logical encoding improve the fidelity of the logical gates
compared to their physical counterparts. In particular, an improvement in the
fidelity of such logical gates will address whether the theoretical benefits
expected from encoding quantum information can be physically realized or if
the noise profile (such as correlated errors) will preclude any such benefit.
The answer to this question is not obvious. Consequently, rather than
examining the error rate of particular circuits created from fault-tolerant
gates, we measure the infidelity of the Clifford group elements created from
such gates and the circuits arising from such Clifford elements. Given the
proposed uses of the [4,2,2] code discussed above, the ability to accurately
create such gates is of high interest. We look at the average infidelity of
the gates (defined later) and demonstrate a conclusive advantage to using such
fault tolerant gates to generate these elements and circuits. We do this by
utilizing the technique of randomized benchmarking (RB) Emerson  _et al._
(2005); Knill  _et al._ (2008) to measure the gate errors in a way that is
robust to SPAM errors to show that the logical two-qubit gates outperform
their physical counterparts. Using RB in the logical code space Combes  _et
al._ (2017) of the [4,2,2] code, we demonstrate that the two qubit average
infidelity decreases from $5.8(2)\%$ to $0.60(3)\%$, an improvement of roughly
an order of magnitude.

However, our results do _not_ imply that the IBM Q Rüschlikon device is
operating below the fault tolerance threshold. As emphasized by Gottesman
(2016), the threshold involves improving all aspects of a quantum circuit
after the encoding, including the SPAM errors, and our approach using RB is
insensitive to SPAM. Thus, while the fault-tolerant encoded gates do improve
gate fidelity, the IQX device is not yet convincingly below threshold for
complete circuits.

## Background.

RB provides an efficient method for the partial characterization of the
quality of a gate implementation in a way that is robust to SPAM and to small,
arbitrary gate-dependent noise Wallman (2018); Merkel  _et al._ (2018). It
gives the fidelity between the identity channel and the averaged noise
$\mathcal{E}$ on the gate set. The (Haar) average fidelity is

$F(\mathcal{E})=\int\mathrm{d}\psi\langle\psi|\mathcal{E}\bigl{(}|\psi\rangle\\!\langle\psi|\bigr{)}|\psi\rangle\,,$
(1)

and the average infidelity is $1-F$.

Recent work by Carignan-Dugas  _et al._ (2018) relates average fidelity to the
gate set circuit fidelity, a quantity which compares all possible sequences of
circuits of $m$ noisy gates implemented from the noisy gate set
$\tilde{\mathbb{G}}$ to their ideal analog in $\mathbb{G}$. A circuit is a
sequence of $m$ elementary gates, and the average gate set circuit fidelity is
$\mathcal{F}(\tilde{\mathbb{G}},\mathbb{G},m):=\mathbb{E}[F(\tilde{\mathcal{G}}_{m:1},\mathcal{G}_{m:1})]$,
where $\tilde{\mathcal{G}}$ and $\mathcal{G}$ are noisy and ideal gates drawn
from the respective gate sets. Ref. Carignan-Dugas  _et al._ (2018) proves for
a single qubit (and conjectures, with numerical evidence, for two qubits)
that, other than a potential SPAM mismatch, the two fidelities are closely
related. This confirms the average infidelity of a gate set as an appropriate
metric even when one is considering the fidelity of circuits built from such
gates.

## The 4 Qubit Code.

The [4,2,2] code is defined by the stabilizer generators $XXXX$ and $ZZZZ$
Shor (1995); Bacon (2006). Ref. Gottesman (2016) details how to implement it
in a fault tolerant manner on a system with limited connectivity. To measure
the logical qubits in the computational basis, one simply measures all four
qubits. Odd parity heralds an error and the run should be discarded.

There are 8 logical gates that make up the code gate set. They split into 6
‘active’ gates and two ‘virtual’ gates that can be implemented in software by
relabeling the physical qubits. These are shown in the table below, where $P$
is the phase gate ($\mathrm{diag}(1,i)$), SWAPij swaps qubits $i$ and $j$, C-Z
is the controlled-Z gate, and CNOTij acts from control $i$ to target $j$.

Physical Gates |  Logical Equivalent
---|---
$X^{\otimes 4},Z^{\otimes 4}$ |  $I\otimes I$
$X\otimes I\otimes X\otimes I$ |  $X\otimes I$
$X\otimes X\otimes I\otimes I$ |  $I\otimes X$
$Z\otimes Z\otimes I\otimes I$ |  $Z\otimes I$
$Z\otimes I\otimes Z\otimes I$ |  $I\otimes Z$
$H\otimes H\otimes H\otimes H$ |  SWAP${}_{12}\circ(H\otimes H)$
$P\otimes P\otimes P\otimes P$ |  $(Z\otimes Z)\,\circ\,$(C-Z)
SWAP12 |  CNOT12
SWAP13 |  CNOT21

The final ingredient is the logical $|\overline{00}\rangle$ state preparation.
Although Ref. Gottesman (2016) suggests a method for doing this fault
tolerantly, the architecture of the IBM-QX5 means implementation is costly in
terms of gates required. Since RB is robust to SPAM it was decided not to
prepare the $|\overline{00}\rangle$ state in a fault tolerant manner (see sup
for a fuller discussion).

## RB with the Real Clifford Group.

RB uses long sequences of gates with the aim of amplifying small errors in the
implementation of these gates. By choosing the sequences of gates from a
unitary 2-design Dankert  _et al._ (2009); Gross  _et al._ (2007) the average
noise channel $\mathcal{E}$ over random sequences of such gates reduces to a
depolarizing channel with the same average fidelity $F(\mathcal{E})$.

By sampling over a unitary 2-design, the integral in Eq. (missing) 1 is
replaced by a sum over the design. Often the Clifford group is chosen as the
unitary 2-design, however in this case the phase gate cannot be implemented in
the [4,2,2] code in a fault tolerant manner. If we limit ourselves to fault
tolerant gates, the lack of a phase gate makes the Clifford group
inaccessible.

Our results make essential use of a modified version of RB called Real RB
Brown and Eastin (2018); Hashagen  _et al._ (2018). In Real RB, one can relax
the unitary 2-design condition to any orthogonal 2-design and still
effectively perform RB with the following protocol:

  1. 1.

choose a sequence length $m$ and prepare a state ($\rho$), traditionally in
the computational basis (but see later),

  2. 2.

apply a chosen number of random gates $m$, independently and uniformly drawn
from the orthogonal 2-design followed by a further inversion gate, which
ideally would result in a net sequence equal to the identity,

  3. 3.

measure $\rho$ with E (an effect operator of a POVM), to determine if it has
been returned to the starting state.

The above steps are repeated in order to estimate the survival probability
$\bar{q}$ over a range of sequence lengths. Ref. Hashagen  _et al._ (2018)
proves that this can be fit to the model

$\bar{q}(m,E,\rho)=A+b^{m}B+c^{m}C\,,$ (2)

where, with $\rho_{\pm}=\tfrac{1}{2}(\rho\pm\rho^{T})$, the constants $A$,
$B$, and $C$ are

$\displaystyle A$ $\displaystyle=\tfrac{1}{d}\text{Tr}[E],$ $\displaystyle B$
$\displaystyle=\text{Tr}\bigl{[}E\rho_{+}\bigr{]}-A,$ $\displaystyle C$
$\displaystyle=\text{Tr}\bigl{[}E\rho_{-}\bigr{]}.$ (3)

For two qubits we also have $F(\mathcal{E})=(9b+6c+5)/20$.

The code gate set generates a subgroup of $\mathsf{C}(2)$, the two-qubit
Clifford group, that we call the Realizable Group, $\mathsf{R}(2)$. The group
$\mathsf{R}(2)$ has only 576 elements, in contrast to the 11,520 elements of
$\mathsf{C}(2)$. Moreover, the average number of elementary gates in a group
element is reduced from about 7 in $\mathsf{C}(2)$ to only about 4 in
$\mathsf{R}(2)$, making it more efficient for RB as well. Finally,
$\mathsf{R}(2)$ provably yields an orthogonal 2-design, so the Real RB
protocol applies sup . There are subtleties that may impact when fitting the
above formulas to an error detecting experiment, although these did not impact
our analysis. For a fuller discussion see sup .

## Implementation Details.

We implemented logical RB and physical RB on the IQX using $\mathsf{R}(2)$ as
the twirling group to measure the average infidelities. If there is an
advantage to computing in the logical space defined by the [4,2,2] code, then
the average infidelity will be lower.

In the supplementary material sup  we detail how we perform RB in such a way
as to eliminate the fitting parameter $A$. In addition, for certain obvious
choices of $\rho$, $B$ or $C$ becomes zero Hashagen  _et al._ (2018). By
choosing an appropriate $\rho$ we can then fit the sequences from those
experiments to a simpler fitting sequence. In the computational basis
$\rho=|00\rangle\langle 00|$ and $\rho_{-}=0$, thus $C$=0. Consequently if we
perform the RB protocol discussed in sup  with $\rho=|00\rangle\langle 00|$,
we can fit to the simplified formula $\bar{q}(m,|00\rangle\langle
00|)=Bb^{m}+0.25$, to determine $b$. This is similar to the ideas discussed in
Refs. Carignan-Dugas  _et al._ (2015); Muhonen  _et al._ (2015); Fogarty  _et
al._ (2015), and yields improved parameter estimates for the same sample size.

Figure 1: (a) Fitting charts for the four different types of runs, used to
extract the $b$ and $c$ parameters in Eq. (missing) 2. The data was fit as
described in the supplementary material sup . Post-selection on invalid code
states was used to throw away runs where an error was detected. The bars show
the percentage of detected (and discarded) runs for the [4,2,2] code runs. The
number of initial errors are greater in the phase run because of the need to
perform an initial non-fault tolerant gate (see text). (b) Simulation showing
survival probabilities for logical ‘real-Clifford’ RB using the [4,2,2] code
and the equivalent physical 2-qubit RB. (Here we have only shown the ‘real’
runs, not the ‘phased’ runs.) The same high fidelity 4-qubit noise map
(fidelity $\approx 0.99$) is used for the runs although additional noise is
applied to the CNOTs of the run on physical qubits. The noise map contains
single-qubit errors as well as $ZZ$ errors between neighboring qubits.
Although the combined noise map is high fidelity, the correlated errors are
sufficiently strong that the fidelity decreases when using encoded gates,
meaning the encoded computation performs worse than the bare physical
computation in the correlated noise scenario.

To determine the $c$ parameter we need to move outside the computational basis
for our choice of $\rho$. In this case the optimal method is to rotate $\rho$
by applying a phase gate followed by a Hadamard, inverting this after the
twirl and measuring $\langle 00|$ in the normal way. Since a fault tolerant
phase gate cannot be applied in the code space, to measure $c$ we need to use
a non-fault tolerant phase gate together with the encoded Hadamard; see sup
for details. When working outside the code, we can implement the phase and
Hadamard gates directly.

In total, this fit procedure requires that we perform four types of runs for
the [4,2,2] code: (1) the run in the computational basis (a standard run) and
(2) the run initialized with the non-fault tolerant phase gate (a phased run);
and for the physical qubits (3) the standard run in the computational basis
and (4) the phased run.

## Analysis.

The four averaged fitting charts are shown in Fig. 1(a). This allows us to
extract values for $b$ and $c$ and calculate the fidelity/infidelity using Eq.
(missing) 1. The calculations yield an infidelity of $0.60(3)\%$ for the
[4,2,2] code and $5.8(2)\%$ for the 2 physical qubits. The uncertainties were
obtained through bootstrapping the experimental data sup .

The question then remains: how much of the benefit is derived from the ability
of the code to detect errors and how much is derived from the virtual CNOTs
once computation has been moved into the code space?

We can use the arguments presented in Ref. Combes  _et al._ (2017) to
determine this. If, instead of using post-selection to discard runs that we
know are in an erroneous code state, we fit to the complete set of runs
obtained for the [4,2,2] code, then this gives us a method of comparing the
performance of the code where we are detecting errors and one where we are
not. With the caveats that these runs were conducted at different times from
the main runs and that a naive RB fit to the data was used (see sup ), this
formulation gave an infidelity that was less than $5.8\%$, but still
significantly higher than the infidelity with error detection. This confirms
two things. First, there is indeed a benefit in moving computation into a code
that allows virtualization of the CNOTs, and secondly, there is still an
additional substantial benefit to using the code to detect errors.

## Comparison to correlated noise.

Data on the quantum devices made available by IBM show that there is
significant crosstalk between connected qubits. One of the interesting
questions answered by this paper is whether such crosstalk causes correlations
strong enough to defeat any fidelity gains that might occur from using encoded
gates operated in a fault tolerant way. To illustrate why this might be the
case we note that a natural noise model for such correlated noise is to model
it as noise of the form $e^{i\theta Z_{i}Z_{i+1}}$ for small $\theta$. This
model accounts for some of the correlated noise observed in recent ion trap
experiments Nigg  _et al._ (2014) and analyzed in Ref. Robertson  _et al._
(2017).

In Fig. 1(b) we show, by simulation, that even with a high fidelity noise map,
the correlated noise can be sufficient to prevent any benefit being observed
from computing in the logical code space of the [4,2,2] code.

## Discussion.

A useful error detecting code consists of 1) a code space in which logical
quantum gates can be performed on the encoded quantum information; 2) an
ability to perform certain fault-tolerant gates within the code space; and 3)
the ability to detect if a certain number of errors (limited by the distance
of the code) have occurred. We have demonstrated how variations of the RB
protocol Hashagen  _et al._ (2018); Combes  _et al._ (2017) can be used to
determine, in a well defined and principled manner, whether the fault-tolerant
quantum gates that are supported by such a code space can be performed with
lower infidelity than such equivalent gates in the raw physical qubits. These
specific RB variations were essential since the [4,2,2] code is not able to
perform the full Clifford group fault tolerantly.

The substantial decrease in the infidelity we observed appears to come from
two sources. The first is that it virtualizes the noisiest gate, the CNOT
gate, resulting in a decrease in the infidelity of the averaged gate set
noise. The second is that by allowing error detection (and subsequent post-
selection) we see a further decrease in the infidelity of the averaged gate
set, leading to an overall decrease in infidelity of the fault tolerant gate
set by a factor of 10.

We learn from these experiments that the IBM device does not have noise
correlations that are so strong as to preclude an improved fidelity when using
an encoded gate. As we showed via numerical simulations (Fig. 1(a)) a natural
noise model Nigg  _et al._ (2014); Robertson  _et al._ (2017) involving
2-qubit correlated errors between neighbors can preclude an improved fidelity
even when the average error rates are comparable to those we observe in the
IBM device. More experiments are required to fully understand the role that
noise correlations play in this device in the context of fault tolerance.

The analysis of Ref. Carignan-Dugas  _et al._ (2018) confirms that the average
fidelity as measured by RB is closely related (as particularized in their
analysis) to the average fidelity for all relevant circuits. Given this, it is
clear that the average gate set fidelity will be substantially higher if the
fault tolerant gates are used, and we argue that this suffices for a
conclusive demonstration of the benefit of using an error detecting code with
fault tolerant gates. Recent work has also shown Huang  _et al._ (2018); Beale
_et al._ (2018) that encoded noise tends to be less coherent than unencoded
noise. Consequently, an improvement in average gate fidelity is also likely to
herald an improvement in other metrics relevant for fault tolerance such as
the diamond distance, although more experiments would be required to establish
this conclusively.

There is at present no complete theory for fitting an RB decay curve with
post-selection. While this had minimal impact on our present study (for the
reasons discussed in sup ) developing a full theory will be essential for more
detailed studies. A further exciting step would be to employ the techniques
used here together with those discussed in Ref. Combes  _et al._ (2017) to
benchmark error-corrected gates. While devices with sufficient connections to
support one, or maybe a few, error-corrected qubits are on the verge of
becoming available for general experimentation, current implementations of IQX
do not yet have conditional measurements or reinitialization. In the meantime,
error detecting codes on more qubits and implementing different fault-tolerant
gate sets can be examined using the techniques discussed above.

###### Acknowledgements.

We thank D. Gottesman for comments on our draft and T. Yoder for highlighting
the impact of post-selection on the RB decay curve. This work was supported by
the US Army Research Office grant numbers W911NF-14-1-0098 and
W911NF-14-1-0103, and by the Australian Research Council Centre of Excellence
for Engineered Quantum Systems CE170100009.

## References

  * Coles  _et al._ (2018) P. J. Coles, S. Eidenbenz, S. Pakin, A. Adedoyin, J. Ambrosiano, P. Anisimov, W. Casper, G. Chennupati, C. Coffrin, H. Djidjev, _et al._ , (2018), arXiv:1804.03719 .
  * Balu  _et al._ (2017) R. Balu, D. Castillo, and G. Siopsis, (2017), arXiv:1710.03615 .
  * Ristè  _et al._ (2017) D. Ristè, M. P. da Silva, C. A. Ryan, A. W. Cross, A. D. Córcoles, J. A. Smolin, J. M. Gambetta, J. M. Chow, and B. R. Johnson, npj Quantum Information 3 (2017).
  * Brell  _et al._ (2011) C. G. Brell, S. T. Flammia, S. D. Bartlett, and A. C. Doherty, New Journal of Physics 13, 053039 (2011), arXiv:1011.1942 .
  * Criger and Terhal (2016) B. Criger and B. Terhal, Quant. Info. Comput. 16, 1261 (2016), arXiv:1604.04062 .
  * Bombin and Martin-Delgado (2006) H. Bombin and M. A. Martin-Delgado, Phys. Rev. Lett. 97, 180501 (2006).
  * Knill (2005) E. Knill, Nature 434, 39 (2005).
  * Linke  _et al._ (2017) N. M. Linke, M. Gutierrez, K. A. Landsman, C. Figgatt, S. Debnath, K. R. Brown, and C. Monroe, Science Advances 3, e1701074 (2017).
  * Takita  _et al._ (2017) M. Takita, A. W. Cross, A. Córcoles, J. M. Chow, and J. M. Gambetta, Phys. Rev. Lett. 119, 180501 (2017), arXiv:1705.09259 .
  * Kelly  _et al._ (2015) J. Kelly, R. Barends, A. G. Fowler, A. Megrant, E. Jeffrey, T. C. White, D. Sank, J. Y. Mutus, B. Campbell, Y. Chen, _et al._ , Nature 519, 66 (2015), arXiv:1411.7403 .
  * Wootton and Loss (2017) J. R. Wootton and D. Loss, (2017), arXiv:1709.00990 .
  * Ofek  _et al._ (2016) N. Ofek, A. Petrenko, R. Heeres, P. Reinhold, Z. Leghtas, B. Vlastakis, Y. Liu, L. Frunzio, S. M. Girvin, L. Jiang, _et al._ , Nature 536, 441 EP (2016), arXiv:1602.04768 .
  * Barends  _et al._ (2014) R. Barends, J. Kelly, A. Megrant, A. Veitia, D. Sank, E. Jeffrey, T. C. White, J. Mutus, A. G. Fowler, B. Campbell, _et al._ , Nature 508, 500 (2014), arXiv:1402.4848 .
  * Heeres  _et al._ (2017) R. W. Heeres, P. Reinhold, N. Ofek, L. Frunzio, L. Jiang, M. H. Devoret, and R. J. Schoelkopf, Nat. Comms. 8 (2017).
  * Gottesman (2016) D. Gottesman, (2016), arXiv:1610.03507 .
  * Vuillot (2017) C. Vuillot, (2017), arXiv:1705.08957 .
  * Willsch  _et al._ (2018) D. Willsch, M. Nocon, F. Jin, H. De Raedt, and K. Michielsen, (2018), arXiv:1805.05227 .
  * Emerson  _et al._ (2005) J. Emerson, R. Alicki, and K. Życzkowski, J. Opt. B Quantum Semiclassical Opt. 7, S347 (2005).
  * Knill  _et al._ (2008) E. Knill, D. Leibfried, R. Reichle, J. Britton, R. Blakestad, J. D. Jost, C. Langer, R. Ozeri, S. Seidelin, and D. J. Wineland, Phys. Rev. A 77, 012307 (2008).
  * Combes  _et al._ (2017) J. Combes, C. Granade, C. Ferrie, and S. T. Flammia, (2017), arXiv:1702.03688 .
  * Wallman (2018) J. J. Wallman, Quantum 2, 47 (2018).
  * Merkel  _et al._ (2018) S. T. Merkel, E. J. Pritchett, and B. H. Fong, (2018), arXiv:1804.05951 .
  * Carignan-Dugas  _et al._ (2018) A. Carignan-Dugas, K. Boone, J. J. Wallman, and J. Emerson, (2018), arXiv:1804.01122 .
  * Shor (1995) P. W. Shor, Phys. Rev. A 52, R2493 (1995).
  * Bacon (2006) D. Bacon, Phys. Rev. A 73, 012340 (2006), quant-ph/0506023 .
  * (26) See Supplementary Material.
  * Dankert  _et al._ (2009) C. Dankert, R. Cleve, J. Emerson, and E. Livine, Phys. Rev. A 80, 012304 (2009).
  * Gross  _et al._ (2007) D. Gross, K. Audenaert, and J. Eisert, J. Math. Phys. 48, 052104 (2007).
  * Brown and Eastin (2018) W. G. Brown and B. Eastin, ArXiv e-prints (2018), arXiv:1801.04042 .
  * Hashagen  _et al._ (2018) A. K. Hashagen, S. T. Flammia, D. Gross, and J. J. Wallman, Quantum 2, 85 (2018).
  * Carignan-Dugas  _et al._ (2015) A. Carignan-Dugas, J. J. Wallman, and J. Emerson, Phys. Rev. A 92, 060302 (2015), arXiv:1508.06312 .
  * Muhonen  _et al._ (2015) J. T. Muhonen, A. Laucht, S. Simmons, J. P. Dehollain, R. Kalra, F. E. Hudson, S. Freer, K. M. Itoh, D. N. Jamieson, J. C. McCallum, _et al._ , J. Phys.: Cond. Mat. 27, 154205 (2015).
  * Fogarty  _et al._ (2015) M. A. Fogarty, M. Veldhorst, R. Harper, C. H. Yang, S. D. Bartlett, S. T. Flammia, and A. S. Dzurak, Physical Review A 92, 022326 (2015), arXiv:1502.05119v2 .
  * Nigg  _et al._ (2014) D. Nigg, M. Muller, E. A. Martinez, P. Schindler, M. Hennrich, T. Monz, M. A. Martin-Delgado, and R. Blatt, Science 345, 302 (2014), arXiv:1403.5426 .
  * Robertson  _et al._ (2017) A. Robertson, C. Granade, S. D. Bartlett, and S. T. Flammia, Phys. Rev. Applied 8, 064004 (2017), arXiv:1703.08179 .
  * Huang  _et al._ (2018) E. Huang, A. C. Doherty, and S. Flammia, ArXiv e-prints (2018), arXiv:1805.08227 .
  * Beale  _et al._ (2018) S. Beale, J. Wallman, M. Gutiérrez, K. R. Brown, and R. Laflamme, ArXiv e-prints (2018), arXiv:1805.08802 .
  * Note (1) https://quantumexperience.ng.bluemix.net/qx/devices.
  * Cross  _et al._ (2017) A. W. Cross, L. S. Bishop, J. A. Smolin, and J. M. Gambetta, (2017), arXiv:1707.03429 .
  * Cleve  _et al._ (2016) R. Cleve, D. Leung, L. Liu, and C. Wang, Quant. Inf. Comput. 16, 0721 (2016), arXiv:1501.04592 .

Supplementary Material: Fault-Tolerant Logical Gates in the IBM Quantum
Experience Robin Harper Steven Flammia *

##  Appendix A Supplementary material

## Details of implementation.

The IBM Q Experience (IQX) is an initiative by IBM that allows access to some
of IBM’s quantum devices. The experiments reported here were carried out on
the IBM-QX5, a 16 qubit device with connectivity as shown in Fig. 2. Details
about the device can be found on the IBM website
111https://quantumexperience.ng.bluemix.net/qx/devices.

The interface to the IBM-QX5 is through a quantum assembly language QASM Cross
_et al._ (2017). Typically methods of error correction rely on measuring
qubits and then acting on the measurement (or, in the case of measurement-free
error correction, being able to reinitialize the qubits). The elements of QASM
required to do this have not yet been implemented on the devices provided by
IBM and, accordingly, there is no obvious way to perform error correction
though the interface provided. Nevertheless, QASM suffices for running an
error detecting code with post-selection. That is, at the end of the run the
measurements allow (in the case of the [4,2,2] code) the detection of a single
(or potentially odd-number) of errors, so that those runs known to be
erroneous can be discarded. Although this results in fewer successful runs, we
show that the runs that do succeed have a substantially improved fidelity.

For the experiments on the ‘raw’ physical qubits, qubits 12 and 5 were chosen
as the physical qubits. They form part of the logical qubit array allowing
valid comparisons between the regimes. (A different qubit pair was also tested
with similar results — see below.)

The QASM specification allows sequences of gates to be compiled into shorter
composite gates unless a ‘barrier’ command is inserted. Barrier commands were
only inserted between discrete group elements of $\mathsf{R}(2)$. Since the
QASM compiler would attempt to shorten the number of gates required for each
element of the group, all infidelities are only reported on a ‘per real
Clifford’ basis.

The quantum chip on the IQX has a life cycle that is relevant to any
comparison runs. Twice a day the machine is taken offline and re-calibrated.
Fidelities on the machine tend to be higher shortly after a re-calibration
cycle and worse just before one. Also the machine is not sole use, so multiple
jobs might be separated in time depending on how many other users are
submitting jobs. Runs can be batched, but there is an overall QASM job size,
which means that in order to gain useful statistics multiple jobs had to be
submitted.

For our purposes, the comparative infidelity between logical and physical runs
is more important than the absolute infidelities. Given this, job submissions
were delayed until low use times and then batched so that the different types
of run sequences were executed as close to each other in time as possible. A
combination of patience and an advantageous time zone meant that the data
reported here were largely gathered over a period of three maintenance cycles,
with the different types of runs interleaved throughout the data gathering
period.

We refer to each batch of runs carried out on the IQX as an experiment. An
experiment consisted of multiple sequences of random sets of gates starting at
2 Clifford gates (followed by an inverting gate), and incrementing by 3 until
92 real Clifford gates had been reached (followed by an inverting gate). That
is, each experiment consisted of a total of 30 random sequences each with a
different gate length. Every sequence was measured 1024 times to build up the
appropriate statistics.

Each experiment was performed 4 different ways, each time the sequences being
randomized and a different ($II$, $IX$, $XI$ or $XX$) being compiled into the
sequence. For a standard run this was compiled in at the beginning of the
sequence, and for a phased run it was compiled in after the state had been
returned to the computational basis. This was done for each of the four types
of run. Finally all of the above was repeated 9 times, meaning we had a total
of 36 sequences at every gate length for each of the four types of run.

Q1Q2Q3Q4Q5Q6Q7Q8Q0Q15Q14Q13Q12Q11Q10Q9
Figure 2: Qubit connectivity diagram in the IBM-QX5 chip. The marked qubits
were used to implement the [4,2,2] code.

## Logical $|\overline{00}\rangle$ state.

Gottesman Gottesman (2016) proposed that the logical $|\overline{00}\rangle$
state could be prepared fault tolerantly using the circuit shown in Fig. 3(b).
The ancilla qubit is used to detect a (single) error in the state preparation.
If it is measured as ‘1’ then the state has not been properly prepared. With
the IBM-QX5 there is no ring of 5 qubits, which is the architecture proposed
in Gottesman (2016). It is still possible to perform the circuit illustrated
in Fig. 3(b), but there is some additional complexity involved. Using Fig. 2
as a reference, one can see that performing a CNOT from Q14$\rightarrow$Q3
(this involves 4 Hadamards and a CNOT), a swap of qubits 3 and 4 (requiring 4
Hadamards and 3 CNOT gates) and finally a CNOT from Q5$\rightarrow$Q4 we have
an equivalent circuit, at the cost of an additional 8 single qubit and 5 two-
qubit gates. (A similar gate count is required if one uses the following
identity
$\text{CNOT}_{13}=\text{CNOT}_{23}\text{CNOT}_{12}\text{CNOT}_{23}\text{CNOT}_{12}$.)

Some initial testing showed that the success rate of state preparation without
the ancilla was comparable to (and often better than) the success rate of
state preparation with the ancilla (and post-selection based on the ancilla
measurement). Since RB is robust to state preparation errors (and measurement
errors), the need to prepare the state in a fault tolerant way was not
something we needed to do in order to determine if fault tolerant computation
in the code space was improved as compared to raw computation in the physical
qubit space. For those reasons, all the results in this paper were done with
the straightforward state preparation (i.e. only the first four gates in Fig.
3(b)). In practice, with a custom architecture, it is likely that fault
tolerant state preparation would be preferable.

## Real RB.

The real Clifford group $\mathsf{C}_{\mathbb{R}}(n)$ on $n$ qubits is the
normalizer in the orthogonal group $\mathsf{O}(2^{n})$ of the real Pauli group
$\mathsf{P}_{\mathbb{R}}(n)$,

$\displaystyle\mathsf{C}_{\mathbb{R}}(n)\coloneqq\\{O\in\mathsf{O}(2^{n})|O\mathsf{P}_{\mathbb{R}}(n)=\mathsf{P}_{\mathbb{R}}(n)O\\}\,.$
(4)

We recover the full Clifford group $\mathsf{C}(n)$ by replacing the orthogonal
group with the unitary group $\mathsf{U}(2^{n})$ and the real Pauli group with
the full Pauli group $\mathsf{P}(n)$. Here it is understood that the group
always carries with it the standard unitary representation on $n$-qubits as a
subgroup of $\mathsf{C}(n)$, the full Clifford group. The Realizable Group
$\mathsf{R}(2)$ discussed in the main text is a subgroup of
$\mathsf{C}_{\mathbb{R}}(2)$.

The frame potential Gross  _et al._ (2007) of a unitary representation of a
group $\mathsf{G}$ is defined as:

$\mathcal{P}(\mathsf{G})=\frac{1}{|\mathsf{G}|}\sum\limits_{g\in\mathsf{G}}\lvert\text{Tr}(g)\rvert^{4}.$
(5)

Ref. Hashagen  _et al._ (2018) proved the following:

  1. 1.

The group $\mathsf{C}_{\mathbb{R}}(n)$ in the standard representation forms an
orthogonal 2-design;

  2. 2.

A group $\mathsf{G}$ is an an orthogonal 2-design if and only if
$\mathcal{P}(\mathsf{G})=3$;

  3. 3.

An orthogonal 2-design can be used as the twirling group in RB with the
protocol listed in the main text.

From points 2 and 3, we only need to show that
$\mathcal{P}\bigl{(}\mathsf{R}(2)\bigr{)}=3$ and it follows that we are
justified in using the Real RB protocol with the Realizable Group. This can be
verified by explicit computation using a computer algebra package.

The full two-qubit Clifford group $\mathsf{C}(2)$ consists of 11,520 distinct
elements. The real Clifford group $\mathsf{C}_{\mathbb{R}}(2)$ on two qubits
is smaller, with only 1,152 elements. The Realizable Group $\mathsf{R}(2)$ has
576 elements, so this is 20 times smaller than the full Clifford group and
half as large as the real Clifford group. Assuming generators given by the X,
Z, phase, Hadamard, and a CNOT gate, the average number of these basic gates
required to implement $\mathsf{C}(2)$ is just over 7, and this remains true
even if the alternate Clifford unitary 2-design described in Ref. Cleve  _et
al._ (2016) is used (which has only 960 elements). By contrast $\mathsf{R}(2)$
has on average only just over 4 generators.

$\qquad\begin{array}[]{c}\Qcircuit<EMAIL_ADDRESS>\lstick{|q_{2}\rangle}&\gate{H}&\ctrl{-1}&\qw&\ctrl{-1}&\qw\\\
\lstick{|q_{3}\rangle}&\gate{H}&\qw&\qw&\qw&\qw\\\
\lstick{|q_{4}\rangle}&\gate{H}&\qw&\qw&\qw&\qw}\end{array}\ \equiv\qquad\ \
\begin{array}[]{c}\Qcircuit<EMAIL_ADDRESS>\lstick{|L_{2}\rangle}&\gate{H}&\qswap\qwx&\qw&\qw}\end{array}$

(a) [4,2,2] code Logical equivalent

$\Qcircuit@C=1em@R=1em{\lstick{|q_{1}\rangle}&\qw\targ\qw\qw\ctrl{4}\qw\qw\\\
\lstick{|q_{2}\rangle}\gate{H}\ctrl{-1}\ctrl{1}\qw\qw\qw\qw\\\
\lstick{|q_{3}\rangle}\qw\qw\targ\ctrl{1}\qw\qw\qw\\\
\lstick{|q_{4}\rangle}\qw\qw\qw\targ\qw\ctrl{1}\qw\\\
\lstick{|A\rangle}\qw\qw\qw\qw\targ\targ\qw}$

(b)

Figure 3: (a) Circuit used to implement a Hadamard gate followed by a phase
gate. Although the phase gate is not fault tolerant, it is only used at the
beginning and end of the RB sequences, so any errors caused by the gate are
absorbed into the SPAM errors. (b) Circuit for preparing the the logical
$|\overline{00}\rangle$ for the [4,2,2] code in a fault tolerant manner
Gottesman (2016). The ancilla is used to determine if an error occurred during
the preparation of the state. With the IBM QX5, the additional number of
physical gates required to perform the CNOTs to the ancilla (see text)
counteracted the benefit of such error detection in state preparation. This
coupled with the fact that RB is robust to SPAM noise, meant that we could
view state preparation errors as part of the SPAM, not impacting the ability
to measure the fidelity of the logical qubits in implementing the real
Clifford gates. Consequently, only the first four gates were used in state
preparation.  Figure 4: Fitting chart for the [4,2,2] code runs, but without
post-selection. In this case we do not throw away sequences which end up
outside the code space. This allows us to approximately quantify the benefit
of operating inside the code space (where we can use virtual CNOTs) as
compared to the joint benefit of operating inside the code space and being
able to detect some errors. A better fit could be obtained in the [4,2,2] code
runs using longer sequences, but IBM QASM limits were an issue.

## Simplifying the RB fitting procedure.

As noted in the main text, we can eliminate certain fitting parameters,
leading to tighter fits for the same amount of data.

In the case of the IQX the measurement operators are predetermined.
Measurement is in the computational basis, each qubit being measured
separately. For each sequence to be executed we specify the number of times we
wish to run the sequence and receive back the number of times each possible
combination was observed. For instance, if we are interested in 2 qubits, then
we will receive back the number of times a ‘11’, ‘10’, ‘01’ and ‘00’ were
recorded (the sum of these numbers equaling the number of runs made). We will
refer to such measurements as a probability measurement, i.e. as a POVM with
effects $E_{\uparrow\uparrow},E_{\uparrow\downarrow},E_{\downarrow\uparrow}$
and $E_{\downarrow\downarrow}$ respectively where
$E_{\uparrow\uparrow}+E_{\uparrow\downarrow}+E_{\downarrow\uparrow}+E_{\downarrow\downarrow}=1$.

Noting that the fitting constant $A$ from Eq. 2 is independent of the initial
state $\rho$, then we can compile in at the beginning (or end) of the sequence
one of the following gates: $II,XI,IX,XX$. Then, provided the sequences are
chosen independently of the gate that has been compiled in, we can utilize the
appropriate measurement (e.g. if we start in the $|00\rangle$ state and
compile in an $XX$ gate, we would use the $E_{\uparrow\uparrow}$ measurement),
average the results and set $A=0.25$. This is a simple extension of the ideas
discussed Refs. Knill  _et al._ (2008); Carignan-Dugas  _et al._ (2015);
Muhonen  _et al._ (2015); Fogarty  _et al._ (2015)). Noise from these
additional gates is absorbed into the remaining SPAM parameters. A careful
choice of initial state preparation, as discussed in the main text, then
allows us to eliminate either $B$ or $C$ so that the net effect is to fit a
straightforward single exponential decay, either $Bb^{m}$ or $Cc^{m}$
depending on the run. Using this technique reduces the correlation with the
nuisance parameters and avoids contaminating the accuracy or precision of the
parameter of interest ($b$ or $c$) with uncertainty in the nuisance degrees of
freedom.

## Fitting the curves.

One of the interesting aspects of using RB techniques for logical benchmarking
is that while the gates used in this paper form an orthogonal 2-design in the
logical code space, they do not form such a design in the physical 4-qubit
space in which they are implemented. This is relevant because where we are
looking at the probability of survival $\Pr(S)$ (i.e. return to the original
state) conditioned on number of gates ($m$) and return to the code space
($c$), then we can write:

$\Pr(S|m,c)=\frac{\Pr(S|m)}{\Pr(c|m)}\,.$ (6)

The numerator in Eq. (missing) 6 can be modeled in the logical code space
using the RB formula, however the denominator requires an examination of the
physical 4 qubit system and the effective twirl of the physical realization of
the logical gates in the larger Hilbert space. Failure to take into account
$\Pr(c|m)$ might lead to inaccurate fits. While we have not made an attempt to
model $\Pr(c|m)$ in this paper, we note that we have measured the actual rates
as ($1-\Pr(c|m)$), which are reproduced as bars in Fig. 1(c) of the main text.
As can be seen, because 1) we did not use a fault tolerant state preparation;
and 2) in the phased runs there were additional noisy non-fault tolerant gates
applied, $1-\Pr(c|m)$ is relatively flat with increasing $m$ starting off at
0.3 (0.4) in the Real (Phased) run, slowly rising to its asymptote of 0.5. We
have therefore approximately modeled it as constant and fit a standard RB
curve to $\Pr(S|m,c)$. Although this is an approximation we note the goodness
of fit to the curves and the fact that the effect noted in the paper (the
almost an order of magnitude improvement in infidelity) is greater than any
difference that might come from a more precisely modeled fit. It does,
however, remain important future work to calculate the fit equations exactly.

Similar problems arise with the fit to survival probabilities of logical gates
which do not throw away the error states (see Fig. 4). Here a fit to the
survival probability in the 4 qubit Hilbert space is not justified as the
physical implementation of the logical gates in the 4 qubits gates do not form
an orthogonal 2-design in the 4 qubit space (they only form such a design in
the code subspace). In the case where we are not throwing away runs that are
outside the code space, we have a leaky subspace. Again we have naively fit
the runs assuming a logical sub-space and logical gates. We justify such a fit
by noting that we are not attempting to find a specific value for the
infidelity of such gates, rather just a confirmation that there is still
substantial benefit in detecting invalid states. The runs are clearly lower
infidelity than the equivalent 2 physical qubit runs, but also are clearly not
as good as the full error detecting runs shown in Fig. 1(c) of the main text.
Since these runs were conducted at a different time from the main runs, even
with an exact formula we would not be able to make a direct comparison (as the
characteristics of the IQX vary appreciably over time).

## Calculating uncertainties.

To calculate the uncertainties given in the main text we used non-parametric
bootstrapping. In this case, for each particular gate length, 36 sequences
were sampled with replacement. For each sequence so sampled the survival
probability of the sequence was used as the estimated probability to generate
a binomial random variable, sampled and averaged over: a) in the case of the
physical 2 qubit system, the number of shots in the original experiment; and
b) in the case of the [4,2,2] code, the number of error-free runs for that
sequence. A total of 9,999 such data sets were sampled and used to calculate
9,999 re-sampled infidelities. The error bars are the 250th and 9,750th of
such infidelities.

## Determining the gates for the physical qubits.

The minimal gates required to generate the group for the physical qubits were
determined by a simple search. We assumed generators of a CNOT gate (qubit 12
$\rightarrow$ 5) and the single qubit gates $X,Z,H$ on either qubit and used
the search to determine the minimum generator gate sequences to create each
element of the Realizable Group $\mathsf{R}(2)$.

## Checking vis-a-vis other physical qubits.

In order to check the validity of the comparisons made in the main text,
further runs were carried out utilizing physical qubits 12 and 13 on the
device. They are not analyzed here as, for reasons discussed in the main text,
it was important to have all runs temporally correlated. However we note that
the calculated infidelity for qubits 12 & 13 was slightly lower ($5.4(2)\%$
compared to the results for qubits 12 & 5 of $5.8(2)\%$), but such a slight
increase has no impact on the conclusions we reach.