# Towards democratizing music production with AI—Design of Variational
Autoencoder-based Rhythm Generator as a DAW plugin

Nao Tokui

Keio University  5332 Endo, Fujisawa  Kanagawa, Japan<EMAIL_ADDRESS>
###### Abstract

There has been significant progress in the music generation technique
utilizing deep learning. However, it is still hard for musicians and artists
to use these techniques in their daily music-making practice.

This paper proposes a Variational Autoencoder[8](VAE)-based rhythm generation
system, in which musicians can train a deep learning model only by selecting
target MIDI files, then generate various rhythms with the model. The author
has implemented the system as a plugin software for a DAW (Digital Audio
Workstation), namely a Max for Live device for Ableton Live.

Selected professional/semi-professional musicians and music producers have
used the plugin, and they proved that the plugin is a useful tool for making
music creatively.

The plugin, source code, and demo videos are available
online111https://cclab.sfc.keio.ac.jp/projects/m4l-rhythmvae/.

###### keywords:

Deep Learning, Variational Autoencoder, generative models, music generation,
Ableton Live

††conference: NIME’20, July 21-25, 2020, Royal Birmingham Conservatoire,
Birmingham City University, Birmingham, United Kingdom. Figure 1:
M4L.RhythmVAE - screenshot of the proposed rhythm generator plugin. \smalla⃝
Users can set training data by "drag and drop" MIDI files here. \smallb⃝
Buttons for start/stop training. \smallc⃝ Graphs for plotting training and
validation losses. \smalld⃝ A standard XY pad for controlling input latent
vectors for the trained VAE decoder. When the user moves a knob on the pad,
the device instantly generates a new rhythm pattern, slightly different from
the previous one, according to the 2-dimensional input vector represented on
the XY pad. \smalle⃝ Visualization of the generated rhythm pattern. \smallf⃝
The output of the device can be forwarded and played in standard drum software
synthesizers of Ableton Live or external MIDI devices via MIDI output.

[500]Applied computing Sound and music computing [300]Computing methodologies
Neural networks

##  1 Introduction

Since the research of deep learning took off, researchers and artists have
proved that deep learning models are competent for generating information
contents, such as images and texts. Music is no exception. Many researchers
have been working on applications of deep learning, especially architectures
for time-series prediction such as Recurrent Neural Networks(RNNs), in music
generation.

Historically, many of those researches focused on generating melodies,
typically of piano, usually using recurrent neural networks(RNN) or their
variant, Long-short term memory (LSTM)[6] (for example, [12]). In the last
couple of years, researchers had started working on models generating rhythms
and bass lines and, in some cases, both of them alongside melodies, so that
the model can generate music in the more comprehensive form with rhythm, bass,
and melody parts[14].

Few researches have tackled the music generation problem in the audio signal
domain due to its significant complexity and computational requirements (One
of a few exceptions is [10]). Hence this paper focuses on music generation at
the symbolic level. [1] gives a more comprehensive view of current deep
learning-based music generation techniques.

These researches also tend to focus on the autonomous nature of the process:
They design AI systems to autonomously generate a complete piece of music, or
at least a part of music, according to a given input (a short sequence of
notes as a seed, or a random input vector). Interventions of human musicians
are usually not considered.

Based on MusicVAE[14] and GrooVAE[3], Google Magenta team provided
accompanying plugins for Ableton
Live222https://magenta.tensorflow.org/studio/ableton-live/ so that musicians
can use them to generate short melody and rhythm patterns. Although these
plugins can be used in trial and error manner, the parameters provided by them
are limited: "temperature" of the sampling process is the only parameter to
affect the patterns to be generated. The user has to stop playing and hit a
button to generate new patterns reflecting new parameter settings.

The generative models (in this case, decoders of Variational Autoencoders[8])
included in MusicVAE[14] plugins were trained with millions of MIDI data
provided as a publicly available Lakh MIDI Dataset (LMD)[13]. Users of the
plugin do not have any control over MIDI files used in the training process,
and subsequently, what kind of music patterns the model is capable of
generating.

By considering the limitations of current AI-based music generation tools as
mentioned above, this paper proposes an easy-to-use deep learning-based music
generation assistive tools, which musicians and music producers can use to
generate novel music patterns and get new musical ideas from them.

The tool proposed here is the following advantages:

  1. 1.

Users of the tool can train their own AI model by "drag and drop" MIDI files
they want to use in training onto the plugin. By doing so, they have loose
control over the patterns they want the plugin to generate.

  2. 2.

The tool provides direct access to the latent space of the trained model.
Users can directory specify/modify input latent vectors and generate various
rhythm patterns corresponding to the inputs.

  3. 3.

The plugin gives realtime feedback. When users move input vectors, the plugin
instantly generates new rhythm patterns. Users can listen to/try various
rhythms by exploring the latent space.

The author has implemented the tool as a Max for Live device(plugin) for
Ableton Live333https://www.ableton.com/live/max-for-
live/—M4L.RhythmVAE(Fig.1). Ableton Live444https://www.ableton.com/live/ is
one of the most popular DAW(Digital Audio Workstation) softwares. The plugin
used Cycling’74 Max555https://cycling74.com/ and
TensorFlow.js666https://www.tensorflow.org/js/. TensorFlow.js enables us to
integrate powerful machine learning capability into JavaScript runtime of Node
for Max777https://docs.cycling74.com/nodeformax/api/ of Cycling’74 Max.

On our website, the author provides a demo video88footnotemark: 81.

This paper focuses on generating rhythm patterns, keeping the production of
electronic dance music in mind, but the same technique can be easily applied
to melody generation and other music genres.

##  2 Related Works

###  2.1 Rhythm generation

Historically speaking, the majority of researches on music generation deal
with melodies and harmonies. Out of 32 papers reviewed in [1] (Chapter 7),
only two papers handle rhythm as their main objective. However, researchers
started experimenting with rhythm generation techniques recently.

[2] showed that a simple LSTM architecture trained with rhythm data encoded as
strings of characters could successfully generate heavy-metal drum patterns.
The paper used a binary representation of nine different drum components. For
example, 100000000 and 010000000 represent kick and snare, respectively, and
110000000 means playing kick and snare simultaneously.

In [9], Markris et al. proposed an architecture with stacked LSTM layers
conditioned with a Feedforward layer to generate rhythm patterns. The
Feedforward layer takes information on accompanying bass lines and the
position within a bar. Training rhythm patterns in binary representation,
similar to [2], are fed into LSTM layers. The system allows only five drum
components (kick, snare, tom, hihat, cymbal).

In [3], Gillick et al. employed Variational Autoencoder(VAE) model to generate
new rhythm patterns. The paper also made use of the encoder-decoder
architecture of VAE and proved that their proposed architecture could
"humanize" quantized rhythms by stressing or weakening certain onsets and
slightly shifting the timing of them.

Their model handles a rhythm pattern represented as three matrices: onsets,
velocities (strength of onsets), timing offsets from quantized timing
(discussed more in detail in section 3. Their VAE consists of layers of
bidirectional LSTM used in [14], and the dimension of the latent space z is
256.

Gillick et al. also provide Ableton Live plugin99footnotemark: 92 based on the
method, which allows users to generate new rhythms within the Ableton Live DAW
environment. However, users cannot train their own models, and users’ control
over generated patterns is limited. The plugins do not operate in realtime
either, i.e. users have to stop playing when they want to generate new
patterns.

###  2.2 Assistive tools

There are not many literatures on the application of deep learning in the
context of assistive tools for music composition since its researches are
relatively new.

Hadjeres et al. proposed DeepBach architecture for the generation of J.S. Bach
chorals[4]. The architecture combines two LSTMs and two Feedforward layers.
The user interface of the system was also implemented as a plugin for the
MuseScore music editor. It allows the user to select generated chorales and
control the progressions interactively.

In [16], Vogl et al. proposed a concept of an "intelligent drum machine"
utilizing a rhythm generation model based on restricted Boltzmann machines
(RBMs) [5]. On their tablet interface, the user can input their own rhythm
pattern (seed) in a grid UI and generate its variations by turning a knob. To
generate variations of the seed pattern, first, the seed pattern is entered
into the visible layer of the RBM. Variations for every instrument are
generated at once using Gibbs sampling. The rhythm patterns are handled as
sequences 0s and 1s: 0 means there is an active note, 1 means no active note.

The generated variations are sorted according to their similarity to the seed
and number of active onsets. The authors used a variance of the Hamming
distance assessed in [15]) as a similarity measure. Finally, patterns other
than the 16 most similar ones in both groups are discarded. In this way, the
system allows the user to explore the variations from sparse patterns to dense
ones, from similar ones to different ones, by turning a knob UI. Users can
also play generated rhythms via Ableton’s
Link101010https://www.ableton.com/link/ technology as MIDI output.

Although the concept and goal is similar to the ones presented in this paper,
the system in [16] does not allow users to train their model, and its
representation of rhythm is limited.

##  3 Implementation

Figure 2: Overview of the VAE architecture used in the plugin. (FC: Fully-
connected layer) Figure 3: A sample rhythm pattern (left) and its encoded
matrices (right).

The generative model the plugin used here is a Variational
Autoencoder(VAE)[8]. The author implemented the training process of the VAE
model in the Max for Live device(plugin). The structure of the VAE is similar
to [3]. The encoder encodes batches of three input matrices, which represent
drum onsets, velocity (strength) of onsets, and timing offset from the given
grid(Fig.2). Once the plugin finished training with user-provided MIDI files,
it uses the decoder of the VAE to generate rhythm patterns(Fig.2 \smallb⃝).

The minimum time step of the drum onset is the 16th note; the encoder
quantizes the timing of every onset to the timing of the nearest 16th note.
Onsets are represented as 0, 1(Fig.3 \smalla⃝). The encoder also normalizes
the MIDI velocity values of onsets [0, 127] to [0., 1.](Fig.3 \smallb⃝). The
timing offset value [-1.0, 1.0) represents the relative distance from the
nearest 16th note. -1.0 indicates the given onset is a 32nd note ahead from
the exact timing of the nearest 16th note(Fig.3 \smallc⃝).

The author chose nine typical drum sounds (Kick, Snare, Hi-hat closed, Hi-hat
open, Tom low, Tom mid, Tom high, Clap, Rim). The author also specified
mapping from MIDI note numbers in General MIDI (GM) convention[11] to the
selected nine drums. The VAE model encodes and decodes two bar-long rhythm
patterns. Since onset timings are quantized in 16th notes, the input and
output of the model is 9 x 32 matrix.

The main difference from the architecture proposed in [3] is that the plugin
adopts simple fully-connected Feedforward layers instead of bidirectional
LSTMs in favor of faster training on CPU environment of TensorFlow.js in Node
for Max. More precisely, both encoder and decoder have two layers of
Feedforward layers with 512 nodes with batch normalization[7] and LeakyReLU
activation.

The VAE model has a 2-dimensional latent space; the encoder encodes input
matrices into 2-dimensional vectors, and the decoder reconstructs the input
matrices from these 2D vectors. The author deliberately reduced the dimension
of the latent space down to 2, from 256 of the model proposed in [3], so that
users of the plugin can directly control the latent vector by standard XY pad
UI (Fig.1 \smalld⃝).

Figure 4: Automation of the progression of input latent vector for VAE
decoder.

When users use the plugin, they can select MIDI files they want to use, "drag
and drop" these files(Figure.1 \smalla⃝). The plugin expects MIDI files
conformed in General MIDI convention[11], e.g., MIDI note number 36
corresponds to Bass Drum, 40 is Electronic Snare. It ignores channels other
than channel #10, which is dedicated to a drum track in GM Format. Every note
onset in channel #10 is mapped to one of 9 drums according to the pre-defined
drum mapping.

Once the plugin loaded MIDI files, users of the plugin can start training a
VAE model by just clicking a start button. Training losses and validation
losses are plotted on respective graphs. Once the training for user-specified
epochs finished, the user can use the decoder of the trained VAE to generate
rhythms by moving a knob in XY-pad representing the 2D latent space of the
VAE.

The internal sequencer of the plugin plays the generated rhythm through MIDI
output in sync with Ableton Live’s global sequencer. The internal sequencer
plays a drum onset slightly before or after the exact timing of 16th notes,
depending on the "timing offset" output of the decoder.

When users gradually move the XY-pad knob, the decoder generates continuously
changing rhythm patterns in realtime, so that the user can create musical
progressions on the fly. The movement of the knob of XY-pad—the transition in
the latent space—can be recorded and played back afterward as a part of the
automation mechanism of Ableton Live’s sequencer(Fig.4).

##  4 Discussion

###  4.1 Real world test

In November 2019, the author co-organized a workshop session with selected
musicians from around the world as a part of MUTEK AI Music Lab
Tokyo111111https://mutek.jp/en/news/archives/50 and asked them to use the
plugin.

Feedbacks from the participants are mainly positive, and one participant
stated that it gave him new musical ideas. Some of the participants used the
plugin in realtime on stage during the final presentation121212A short video
documentation of these performances is available online:
https://youtu.be/RnlJF1YU6JU.

A more empirical analysis of the plugin and user questionaries are yet to be
conducted.

Some of the complete tracks made by participants of the workshop using rhythm
patterns generated by the plugin are available online1313footnotemark: 131.

###  4.2 Model size and controllability

One can argue that the VAE model the author adopted lacks the capacity of
generalizing the wide range of different types of rhythm patterns since it has
only 2-dimensional latent space in favor of the usability with the XY Pad. The
smaller latent space leads to severe information bottleneck. Still, in the
practical use case of this tool, where average users do not have millions of
MIDI files, the author assumed the downside of the bottleneck is negligible.

The author also observed that musicians do not need generic generative models.
They usually prefer to have generative models, which enable them to explore
specific kinds of music (in this case, rhythm patterns). (One of the
participants even said, "I love overfitting.")

At the same time, if we further explore this idea of easy-to-use AI music
tools for musicians and try generating more complex music patterns, then the
limitation of the 2-dimensional latent space represented on an XY-pad will be
problematic. It poses a challenging question on how we can adopt latent spaces
in higher dimensions and maintain the controllability for the user at the same
time. This problem shall be one of the next challenges the author will tackle.

##  5 Conclusion

To let musicians train and use their rhythm generation model, this paper
proposed a VAE-based rhythm generator as a Max for Live device, a plugin for
the popular DAW software, Ableton Live. Musicians have tested the plugin and
found it useful as a tool to seek new ideas and extend their musical
creativity.

The author believes that this research and plugin are one of the first
stepping stones towards more democratized use of AI in creative practices. In
the near future, the author also hopes that every artist/creator should be
able to train their own small AI models suited to their need, instead of using
generic models someone else has trained, and explore new creative ideas with
them.

##  6 Acknowledgments

This research was funded by the Keio Research Institute at SFC Startup Grant
and Keio University Gakuji Shinko Shikin grant. The author wishes to thank
Stefano Kalonaris and Max Frenzel for inspiration. The author also thanks
Maurice Jones and Natalia Fuchs for organizing MUTEK AI Music Lab in Tokyo.

## References

  * [1] J.-P. Briot, G. Hadjeres, and F. Pachet.  Deep learning techniques for music generation.  Springer, 2019.
  * [2] K. Choi, G. Fazekas, and M. Sandler.  Text-based LSTM networks for Automatic Music Composition.  Proceedings of the 1st Conference on Computer Simulation of Musical Creativity, 2016.
  * [3] J. Gillick, A. Roberts, J. Engel, D. Eck, and D. Bamman.  Learning to Groove with Inverse Sequence Transformations.  Proceedings of the 36th International Conference on Machine Learning, 2019.
  * [4] G. Hadjeres, F. Pachet, and F. Nielsen.  DeepBach: A steerable model for bach chorales generation.  In 34th International Conference on Machine Learning, 2017.
  * [5] G. E. Hinton, S. Osindero, and Y. W. Teh.  A fast learning algorithm for deep belief nets.  Neural Computation, 2006.
  * [6] S. Hochreiter and J. J. Urgen Schmidhuber.  LONG SHORT-TERM MEMORY.  Neural Computation, 1997.
  * [7] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In 32nd International Conference on Machine Learning, 2015.
  * [8] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  In 2nd International Conference on Learning Representations, 2014\.
  * [9] D. Makris, M. Kaliakatsos-Papakostas, I. Karydis, and K. L. Kermanidis.  Conditional neural sequence learners for generating drums’ rhythms.  Neural Computing and Applications, 2019.
  * [10] R. Manzelli, V. Thakkar, A. Siahkamari, and B. Kulis.  Conditioning deep generative raw audio models for structured automatic music.  In Proceedings of the 19th International Society for Music Information Retrieval Conference, 2018.
  * [11] MIDI Association.  General MIDI 1 Sound Set.  https://www.midi.org/specifications-old/item/gm-level-1-sound-set, 1991\.
  * [12] S. Oore, I. Simon, S. Dieleman, D. Eck, and K. Simonyan.  This time with feeling: learning expressive musical performance.  Neural Computing and Applications, 2018.
  * [13] C. Raffel and D. P. Ellis.  Extracting ground truth information from MIDI files: A MIDIfesto.  In Proceedings of the 17th International Society for Music Information Retrieval Conference, 2016.
  * [14] A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck.  A hierarchical latent vector model for learning long-term structure in music.  In 35th International Conference on Machine Learning, 2018.
  * [15] G. Toussaint.  A Comparison of Rhythmic Dissimilarity Measures.  Forma, 2006.
  * [16] R. Vogl and P. Knees.  An Intelligent Drum Machine for Electronic Dance Music Production and Performance.  In NIME 2017 Proceedings of the International Conference on New Interfaces for Musical Expression, 2017.


# Can Quantum Computers Do Nothing?

Alexander Nico-Katz<EMAIL_ADDRESS>School of Physics, Trinity College
Dublin, Dublin 2, Ireland  Trinity Quantum Alliance, Unit 16, Trinity
Technology and Enterprise Centre, Pearse Street, D02 YN67, Dublin 2, Ireland
Nathan Keenan<EMAIL_ADDRESS>School of Physics, Trinity College Dublin,
Dublin 2, Ireland  Trinity Quantum Alliance, Unit 16, Trinity Technology and
Enterprise Centre, Pearse Street, D02 YN67, Dublin 2, Ireland  IBM Quantum,
IBM Research Europe - Dublin, IBM Technology Campus, Dublin 15, Ireland  John
Goold<EMAIL_ADDRESS>School of Physics, Trinity College Dublin, Dublin 2,
Ireland  Trinity Quantum Alliance, Unit 16, Trinity Technology and Enterprise
Centre, Pearse Street, D02 YN67, Dublin 2, Ireland  Algorithmiq Ltd,
Kanavakatu 3C 00160, Helsinki, Finland

###### Abstract

Quantum computing platforms are subject to contradictory engineering
requirements: qubits must be protected from mutual interactions when idling
(‘doing nothing’), and strongly interacting when in operation. If idling
qubits are not sufficiently protected, information can ‘leak’ into
neighbouring qubits, become non-locally distributed, and ultimately
inaccessible. Candidate solutions to this dilemma include patterning-enhanced
many-body localization, dynamical decoupling, and active error correction.
However, no information-theoretic protocol exists to actually quantify this
information loss due to internal dynamics in a similar way to e.g. SPAM errors
or dephasing times. In this work, we develop a scalable, flexible, device non-
specific protocol for quantifying this bitwise idle information loss based on
the exploitation of tools from quantum information theory. We implement this
protocol in over 3500 experiments carried out across 4 months (Dec 2023 - Mar
2024) on IBM’s entire Falcon 5.11 series of processors. After accounting for
other sources of error, and extrapolating results via a scaling analysis in
shot count to zero shot noise, we detect idle information leakage to a high
degree of statistical significance. This work thus provides a firm
quantitative foundation from which the protection-operation dilemma can be
investigated and ultimately resolved.

##  I Introduction

Quantum computing is a new paradigm of computation based on the exploitation
of quantum phenomena which promises frontier impacts on global energy, health,
materials science, and technological innovation [1, 2]. With the firm advent
of the noisy intermediate-scale quantum (NISQ) era in the form of accessible
quantum processors containing some tens to hundreds of qubits [3, 4, 5, 6],
out-of-the-box error mitigation [7], and the nascent implementation of simple
error correction [8, 9, 10], this new paradigm is now being mapped out
comprehensively. However, far from their conceptualization as ideal systems,
real quantum computers are intrinsically programmable many-body quantum
systems with complicated internal dynamics exposed to stray interactions and
thermal fluctuations (Fig. 1(a-b) schematically show idealized and realistic
‘empty’ circuits respectively) [11]. The precise control of these complex
many-body physical systems is ultimately the fundamental problem of quantum
computing.

Figure 1: Schematic quantum circuits and devices (a) Schematic of an ideal
quantum circuit which realizes perfect idling. (b) Schematic of a physical
quantum circuit with imperfect idling due to native transport (black arrows),
interactions (dotted lines), and precession (oscillating lines). (c) Schematic
showing the qubit geometry of the Falcon 5.11 series of IBM’s quantum devices.
Example target qubits are shown by yellow diamonds. Shaded regions
respectively indicate examples of a contiguous set of nearest-neighbour qubits
(blue) and a random set of qubits excluding nearest-neighbours (orange).
Measuring the additional information gained about the target qubit when
addressing the complementary qubits allows us to quantify idle information
loss due to the internal dynamics of the hardware.

This problem is encapsulated by the ‘protection-operation dilemma’ wherein
qubits need to be coupled and decoupled from each other at different times
during the same computation [12]. When not in active operation (‘doing
nothing’), idling qubits should be well protected from mutual interactions
with other qubits in the device. However, multi-qubit operation require strong
mutual interactions such that entanglement and correlations can be generated
between the active qubits quickly. Both of these processes should also be
resistant to environmental decoherence, which typically reduces to the
requirement that computations should be completed as quickly as possible. This
generates contradicting engineering requirements: qubits should be decoupled
when idling, but strongly coupled when necessary for computation. The two main
ways this contradiction is resolved are (i) many-body localizing the qubits
through spatial disorder and only bringing neighbouring qubits into resonance
during gate operations (ii) rapidly tuning the mutual interactions in-situ to
actively couple and decouple qubits during runtime [12, 13, 14]. The potential
breakdown of the former resolution has been addressed in detail by recent
works which find that current-generation superconducting quantum computers may
enter chaotic regimes during operation; calling into question how well
protected qubits in these devices actually are [15, 16, 17]. The latter poses
a complicated engineering problem that may induce higher order or time-
dependent effects which are not yet fully understood. Idle information leakage
can also be reduced by active error correction or dynamical decoupling, but
these introduce formidable engineering problems and additional gate-based
errors and complexity respectively [18, 19].

Inspired by the burgeoning many-body perspective on quantum computation, we
exploit tools from many-body physics and quantum information theory to address
a critical question underlying the entire protection-operation dilemma: can
the information loss induced by multi-qubit dynamics during idling be
quantified in a similar way to e.g. readout errors or dephasing times? In this
work we address this question directly by presenting and experimentally
implementing a protocol for quantifying idle information loss. This protocol,
developed and discussed in Section II, leverages the Holevo quantity to
accurately quantify information loss, is device non-specific, scalable, and
can be easily run during computational downtime. We experimentally implement
the protocol in Section III across the entire range of Falcon 5.11 series of
IBM’s quantum devices 111IBM has recently discontinued this particular series,
but the protocol is not device-specific.. After accounting for other sources
of error, we identify a small but measurable amount of information leakage to
a high degree of statistical significance. Our accurate quantification of the
actual informational impact of many-body effects in real devices represents a
decisive step towards identifying and measuring idle information loss in
future quantum devices. This result also provides a firm quantitative
foundation from which the protection-operation dilemma can be further
interrogated and resolved.

##  II Quantifying Internal Information Leakage

Information lost whilst idling due to the native dynamics of a device
manifests as the spreading of information that is initially localized.
Information initially localized to a single qubit moves coherently into the
rest of the system and is distributed non-locally. Thus the basic premise of
our protocol is to monitor both a target qubit from which information may leak
out, and a complementary set of qubits into which information may have flowed.
This is shown schematically in Fig. 1(c) for two types of complementary qubit
set: nearest-neigbour (blue) and random (orange). The basic mathematical
object we use to characterize the amount of information lost is the Holevo
quantity [21]: an import from quantum information theory that quantifies the
amount of classical information that a channel and an ensemble of messages (an
alphabet) can carry [22]. We discuss the Holevo quantity in detail in Section
II.1 for the interested reader.

The protocol in an ideal system, with no other sources of error, is discussed
in detail in Section II.2. The protocol involves initializing a target qubit
in one of two message states $|0\rangle$ and $|1\rangle$ which encode a
classical bit. By calculating the Holevo quantity both over the target qubit
alone $\chi^{S}$, and over a complementary set $q$ of qubits which includes
the target qubit $\chi^{SQ}$, we can quantify how much extra information we
get about the initial classical binary message by looking at non-local degrees
of freedom. If the difference between these two Holevo quantities is finite,
then information has leaked out of the target qubit. In Section II.3 we
address other sources of error, and discuss how to unravel them from true
information leakage due to internal dynamics. The major issue we identify is
shot noise which can artificially induce a significant signature of
information loss. We address this issue in Section III by exploiting central
limiting behaviour, and present an ansatz for extrapolating information loss
at zero shot noise during our experimental implementation of the protocol.

We remark on two major elements of the protocol which impose constraints on
the types of device to which it is applicable. The platform must be capable of
(i) full state tomography on at least two qubits (a target qubit, and at least
one other qubit in the device) (ii) initializing the target qubit in two
definite states. We assume nothing about the microscopic features of the
native physics of a device except that (iii) the dynamics are in some way
local: qubits are assumed to couple more strongly to their neighbours than
qubits further away in an array. This means we can look at nearest-neighbour
qubits and random qubits, both of which are equally susceptible to single-
qubit errors and shot noise but are not equally sensitive to information
leakage affecting a fixed target qubit.

###  II.1 The Holevo Quantity

The Holevo quantity $\chi$ over an alphabet
$\mathcal{A}=\\{p_{k},\hat{\rho}_{k}\\}$ and channel $\mathbf{\Lambda}$
quantifies - in bits - the accessible information that is carried by the
protocol wherein states $\hat{\rho}_{k}\in\mathcal{H}_{i}$ are selected with
probability $p_{k}$ and transmitted through the channel
$\mathbf{\Lambda}[\hat{\rho}_{k}]\in\mathcal{H}_{f}$ [21, 22]. The Holevo
quantity $\chi$ is given by

$\chi=S\left(\sum_{k}p_{k}\mathbf{\Lambda}[\hat{\rho}_{k}]\right)-\sum_{k}p_{k}S\Bigl{(}\mathbf{\Lambda}[\hat{\rho}_{k}]\Bigr{)}.$
(1)

The Holevo quantity $\chi$ has been used to characterize information in many-
body systems before, and is a natural quantity with which to investigate
information loss [23, 24, 25, 26]. We critically note that the initial space
$\mathcal{H}_{i}$ and final space $\mathcal{H}_{f}$ are not generally
identical. Consider e.g. a protocol which prepares a target qubit in a
specific state, but performs measurements on the combined final state of the
target qubit and its nearest-neighbours.

When maximized over all possible input alphabets $\mathcal{A}$, $\chi$ gives
the maximum classical capacity of the channel $\mathbf{\Lambda}$. However,
this maximization is often not possible in practice: the precise nature of a
device’s underlying Hamiltonian, and thus of the channel $\mathbf{\Lambda}$,
is subject to debate - and the Hamiltonian parameters are subject to random
fluctuations and drift. However, quantum computing platforms should
approximately realize identities on idling qubits. Thus it is reliable to
define an alphabet of equi-probable $p_{k}=1/\text{dim}(\mathcal{H}_{i})$,
pure, and orthogonal messages which cover the state space $\mathcal{H}_{i}$
(this choice saturates $\chi$ when $\mathbf{\Lambda}$ is the identity
channel).

We now discuss another important property of the Holevo quantity which becomes
critically relevant when we try to account for shot noise in Section III. It
is intuitively appropriate to consider $\chi$ as encapsulating the notion of
distinguishability. Direct inspection the form of $\chi$ reveals that an
alphabet of pure states which are not all mutually orthogonal will cause the
first term of Eq. 1 to fail to saturate, resulting in a low value of $\chi$.
Conversely, an alphabet of states which are close to maximally mixed states
will saturate the first term, but also saturate the second - resulting in a
low value for $\chi$ overall. Thus the Holevo quantity can be understood as a
careful balancing act between the mixedness of the source (first term) and
purity of the individual messages (second term) which together capture
distinguishability. This may seem like a pedantic point, but it becomes
critical when accounting for the effect of shot noise. Whilst the Holevo
quantity is monotonically decreasing under the implementation of a CPTP
channel, shot noise can not be represented as such, and can thus artificially
increase the Holevo quantity. More precisely, shot noise is realised as random
contributions to elements of the final density matrices
$\mathbf{\Lambda}[\hat{\rho}_{k}]$. These contributions can increase
distinguishability but does not actually increase the amount of information
that the channel can bear. This effect becomes exacerbated in the case that
the dimension of the initial space is smaller than the dimension of the final
space. If $\text{dim}(\mathcal{H}_{i})<\text{dim}(\mathcal{H}_{f})$, there are
more elements in density matrices drawn from the final space, and thus shot
noise introduces more differences between output density matrices.

In the context of this article (and in the absence of shot noise, which we
address separately), the Holevo quantity quantifies how much information we
can access about the initial state of a qubit given access to it and access to
some other subset of qubits on the device. If no information has coherently
leaked out of the target qubit, then access to some other region should not
give us additional information about its initial state.

###  II.2 Protocol for Quantifying Idle Information Loss

We now leverage Holevo quantities computed over two subsets of qubits on a
device into protocol which quantifies the information lost to other qubits
during idling. We first introduce the Holevo quantities $\chi^{S}$ and
$\chi^{SQ}$, respectively computed on the final reduced density matrices (i)
of the target qubit alone $\hat{\rho}_{k}^{S}$, and (ii) of the target qubit
and a complementary set of $M-1$ qubits in the array $\hat{\rho}_{k}^{SQ}$.
Note here that $\hat{\rho}_{k}^{S}=\text{Tr}_{E}[\hat{\rho}_{k}^{SQ}]$, and
thus by the monotonicity of the Holevo quantity under the partial trace:

$\Delta\chi^{SQ}\coloneq\chi^{SQ}-\chi^{S}\geq 0$ (2)

with equality only when information is fully localized to the target qubit, or
when $\chi^{SQ}=\chi^{S}=0$ and information has completely left the final
space. Since the Holevo quantity yields the average number of bits of
information that can be transmitted by messages passed through the channel,
$\Delta\chi^{SQ}$ can also be interpreted as the extra bits of information we
can access about the initial state of the target qubit $S$ given access to the
complementary qubits $Q$. In ideal systems without shot noise or other sources
of error $\Delta\chi^{SQ}$ suffices to identify and quantify information loss
due to information leakage. Finite $\Delta\chi^{SQ}$ indicates that some
information has left the target qubit in a coherent fashion, and is
distributed non-locally: i.e. that the device can’t ‘do nothing’ perfectly,
even in ideal conditions. The protocol in full is as follows:

  1. 1.

Initialize every qubit except the target qubit $S$ in an arbitrary state
$\hat{\rho}_{E}$ (we take pure separable product states of the logical single-
qubit states throughout).

  2. 2.

Initialize the target qubit $S$ in the state $|0\rangle$.

  3. 3.

Wait for a fixed period of time $T$ (we measure immediately to address the
‘best case’ scenario with minimal information leakage such that $T$ is the
readout time of the device) 222$T$ is arbitrary in principle, and can be tuned
to address different regimes or properties of a device. Taking $T$ to be e.g.
fifty two-qubit gate times could give us a good indicator of the total
information leakage during a complete computation..

  4. 4.

Perform full state tomography on the combined state $\hat{\rho}^{SQ}_{0}$ of
the target and complementary qubits (the subscript denotes that this state is
conditioned upon the initial state, prepared in step 2).

  5. 5.

Repeat from step 1, with the target qubit $S$ initialized in the state
$|1\rangle$, generating the state $\hat{\rho}^{SQ}_{1}$.

  6. 6.

Process the states into a sample value for $\Delta\chi^{SQ}$.

  7. 7.

Check the condition $\Delta\chi^{SQ}>0$.

  8. 8.

(Optional, to address noise) Repeat from step 1 for a large number of samples
to build up statistics for $\Delta\chi^{SQ}$.

Figure 2: Experimental sample statistics for $\Delta\chi$ Statistics of
$\Delta\chi^{SP}$ and $\Delta\chi^{SR}$ for nearest-neighbour $P$ (blue) and
random $R$ (orange) complementary sets of qubits respectively. Shot count is
$N_{S}=4000$. $\Delta\chi$ quantifies how much extra information about the
target qubit can be accessed by measuring non-target qubits in the
complementary sets $P$ or $R$, and can be artificially increased by shot
noise, which should have an identical effect on both distributions. The ranges
determined by Eq. 5 at a tolerance of $K=4$ are shown as blue dashed lines for
$\Delta\chi^{SP}$, and orange dotted lines for $\Delta\chi^{SR}$. Samples
which fall outside of this range are classified as outliers, and excluded from
the numerical analysis. Inset shows means and standard errors for the non-
outlier region of each distribution; evidencing that Eq. 3 is satisfied and
idle information loss has been directly detected after accounting for shot
noise. A noticable bump of outliers at larger values of only $\Delta\chi^{SP}$
is evidence of ‘bad qubits’ that have strongly coupled to their neighbours and
are leaking lots of information during idling. Statistics are comprised of
1209 total samples, taken across four months (Dec 2023 - Mar 2024) from all
the Falcon 5.11 series of devices.

###  II.3 Unravelling Information Leakage from Other Sources of Error

Unravelling information leakage from other sources is, in principle, a serious
operational issue. As we discussed in Section II.1, shot noise may
artificially increase distinguishability. Moreover, other sources of error may
artificially increase the difference $\Delta\chi^{SQ}$, which is not subject
to the same monotonicity conditions as the Holevo quantity alone. In essence,
$\Delta\chi^{SQ}$ can be non-zero in realistic systems even if information is
fully localized to the target qubit. To deal with this we benchmark our
results on nearest-neighbour qubits $Q=P$ against random (non nearest-
neighbour) qubits in the array $Q=R$, whilst keeping the target qubit $S$
fixed. Both of these sets of complementary qubits should, when averaged over a
large number of samples - be equally affected by e.g. shot noise, SPAM errors,
and environmentally-induced decoherence. However, the randomly selected qubits
should bear much less information about the initial state of $S$; and thus
they can serve as a benchmark for all other sources of error in the device.
Nevertheless, we choose to carry out our experiments at very short timescales,
two orders of magnitude less than the dephasing times of the devices we use,
to mitigate the effect of the environment as much as possible. Our protocol
does not make use of any gates in its implementation, and thus gate errors are
vacuously irrelevant. We implement these benchmarking procedures in Section
III and ultimately find that shot noise is the dominant contribution to
$\Delta\chi$ in the system of interest. We suggest, but can not prove, that
channels comprised of a linear combination of single-qubit processes can not
systematically increase $\Delta\chi^{SQ}$; as such single-qubit processes can
not systematically transfer information from $S$ to $Q$.

We first invoke the assumption that the underlying dynamics are, in a sense,
local, and that the idea of ‘nearest-neighbours’ is thus well-defined. In such
systems, information should flow from the target qubit to its nearest-
neighbours, and then through them to more distant qubits in the array. For a
fixed target qubit $S$, we can then define two types of complementary sets of
qubits: $P$ corresponding to the plaquette of the $M-1$ nearest-neighbours of
$S$, and $R$ corresponding to the target qubit and $M-1$ randomly selected
(excluding nearest-neighbour) qubits in the array. Since $P$ and $R$ both
contain the same number of qubits they should, after averaging, be equally
susceptible to single-qubit errors and shot noise (see Appendix B for a
detailed discussion on the origin of this shot noise in IBM devices). In an
ideal system without other errors or shot noise
$\Delta\chi^{SP}-\Delta\chi^{SR}>0$ should be a strict inequality (except at
very late times where the information becomes fully delocalized). Individual
samples may violate this inequality, but since both $P$ and $R$ are equally
susceptible to all other sources of noise, these effects should cancel out
after averaging over a large number of samples. We thus relax this condition
to a statistical inequality

$\overline{\Delta\chi^{SP}}-\overline{\Delta\chi^{SR}}>0$ (3)

that should hold in the presence of non-trivial information leakage. Moreover,
Eq. 3 can be easily realized as the alternative hypothesis $H_{1}$ for
significance testing; which we do in Section III.

This inequality can be extended more formally by suggesting an ansatz form for
$\Delta\chi$ as follows:

$\Delta\chi^{SQ}=f\left(\eta^{SQ},\eta^{SQ}_{\text{SHOTS}},N_{\text{S}}\right).$
(4)

Which is a function of the excess accessible information $\eta^{SQ}$,
artificial information due to shot noise $\eta^{SQ}_{\text{SHOTS}}$, and the
shot count $N_{\text{S}}$. We can aggregate statistics at different shot
counts, perform a scaling analysis in $N_{\text{S}}$, and extrapolate
$\eta^{SQ}$ as $N_{\text{S}}\to\infty$. We develop just such an ansatz based
on our experimental data in Section III. This also formalizes the intuitive
notion of both the sets $P$ and $S$ of complementary qubits being ‘equally
susceptible’ shot noise. We say that both are ‘equally susceptible’ to shot
noise if $\eta^{SP}_{\text{SHOTS}}\approx\eta^{SR}_{\text{SHOTS}}$ after
fitting. With shot noise mitigated, the difference $\eta^{SP}-\eta^{SR}$ then
gives us the amount of information which has leaked from the target qubit into
its neighbours.

In actual experiment (see Section III), we find $\eta^{SR}\to 0$ exactly,
supporting our previous assertion that other single-qubit sources of error can
not systematically transfer information to distant regions of the device. This
indicates that $\eta^{SP}$ alone suffices to determine the amount of idle
information loss in future experiments.

##  III Experimental Results and a Quantification of Idle Information Loss on
IBM Devices

The experimental implementation of our protocol was carried out on the
27-qubit Falcon 5.11 series of IBM’s quantum computing devices; the
architecture of such devices is shown in Fig. 1(c). We incorporate two
additional steps into the general protocol given in Section II.2: (i) we only
select target qubits with the highest possible coordination number $n_{c}=3$,
which should yield the strongest signatures of idle information loss. And (ii)
in step 1 of the protocol, we also simultaneously randomize over target qubits
subject to the coordination constraint, and the Falcon 5.11 devices
ibm_algiers, ibm_cairo, ibm_hanoi, and ibmq_kolkata themselves. This
corresponds to the generation of statistics for $\Delta\chi$ which are device
agnostic - i.e. the user is interested only in running their computation job
on a 27-qubit Falcon 5.11 device, and doesn’t care about which specific device
to which their jobs are assigned. We performed readout immediately after state
preparation, such that the wait time in step 3 of the protocol was just the
readout time of the given device (between 700ns and 900ns). This minimizes the
effect of the environment, as these readout times are several orders of
magnitude lower than typical T1 and T2 times. This also means that our results
represent the best case scenario, and can not be improved by e.g. dynamical
decoupling, as such processes are not possible during readout. Due to the
existence of shot noise, the resulting density matrices after state tomography
can have negative eigenvalues. We use a maximum-likelihood reconstruction
method to rephysicalize the density matrices before post-processing [28]. A
detailed discussion of the tomographic process is given in Appendix B, and the
maximum-likelihood reconstruction of aphysical density matrices is discussed
in Appendix C.

As discussed in Section II.2, we consider both nearest-neigbour $P$ and random
$R$ sets of complementary qubits for each target qubit, and carry out separate
collections of experiments for each at a range of different shot counts
$N_{S}$. The number of samples for all shot counts $N_{S}$ on both nearest-
neighbour plaquettes and random qubits is summarized in Table 2 in Appendix B.
All in all, our analysis of the Falcon 5.11 series involves the results of
over 3500 experiments taken across four months (Dec 2023 - Mar 2024), and
represents a broad-spectrum comprehensive investigation of idle information
loss on these devices. As the runtime increases proportionally with the shot
count, most of this time was spent collating results for higher shot count
samples; thus a direct comparison where statistics are compared for lots of
low shot count samples, is likely the best approach for characterizing idle
information loss on IBM devices in the future.

###  III.1 Results and Analysis

Figure 3: Statistics of $\mathbf{\Delta\chi^{SP}}$ for a range of shot counts
Outlier-excluded $K=4$ statistics of $\Delta\chi^{SP}$ for a range of shot
counts $N_{S}$. As shot count is doubled, the mean value of each set of
statistics appears to scale by a factor of $1/\sqrt{2}$, indicating central
limiting behaviour in $N_{S}$ which dominates signatures of idle information
loss. A total of $1849$ samples are distributed across the five different
values of $N_{S}$, a breakdown of which is provided in Table 2.

The experimentally determined statistics of $\Delta\chi^{SP}$ and
$\Delta\chi^{SR}$ for 1209 total samples across all devices are shown in Fig.
2. The shot count for both was fixed at the standard value of $N_{S}=4000$.
Both take a normally distributed form within certain limits, with
$\Delta\chi^{SP}$ exhibiting an additional small bump in the range
$0.12-0.14$. Were this bump due to any of the other processes discussed:
decoherence, SPAM errors, shot noise, we would expect bumps to also appear in
the statistics of $\Delta\chi^{SR}$. As this bump is not present in the
statistics of $\Delta\chi^{SR}$, it stands to reason that this is strictly due
to some target qubits coupling strongly to their nearest-neighbours. Moreover,
we identify a bump in the statistics of $\Delta\chi^{SP}$ at larger shot
counts at the exact same position (see Appendix A), evidencing a true
signature of idle information loss. The samples that form this bump can be
justifiably interpreted as ‘bad qubits’ which have hybridized with their
nearest-neighbours to such an extent that information leakage is dominating
other sources of error. Over $10\%$ of the information about the initial state
of such a qubit is stored non-locally.

A more subtle feature of Fig. 2 is that the statistics of $\Delta\chi^{SP}$
show a thicker tail at larger values than $\Delta\chi^{SR}$, suggesting that
the inequality Eq. 3 is satisfied even when these bad qubits are excluded. To
quantify this difference, we first filter the statistics for outliers by
simple box-filtering according to the condition [29, 30],

$\text{Q1}-K\times\text{IQR}\leq\Delta\chi^{SQ}_{s}\leq\text{Q3}+K\times\text{IQR}$
(5)

where we have introduced the subscript $s$ to denote individual samples of
$\Delta\chi^{SQ}$. Q1 and Q3 are the first and third quartiles of the full
sample statistics $\Delta\chi^{SQ}$ respectively, and IQR is their inter-
quartile range. Conservative filtration typically takes $K=1.5$, whilst
$R\to\infty$ corresponds to no filtering at all. We take $K=4$ throughout as
this both reliably contains the large Gaussian part of the distributions, and
also excludes the bad qubit bump in the statistics of $\Delta\chi^{SP}$. We
superimpose the boundaries of the box defined by Eq. 5 on Fig. 2 as dashed
blue lines for $\Delta\chi^{SP}$ and a dotted orange line for
$\Delta\chi^{SR}$ 333It is thus interesting to point out that we can thus
invoke $K$ as a tuning parameter which defines precisely what is meant by a
‘bad qubit’. The part of the distribution to the right of the box defined by
$K$ can inform us of the probability that any given qubit will be a ‘bad’ one
up to a certain informational tolerance. In cases where the entire device is
required to complete a computation, or bad qubits are not identified and
excluded before runtime, this probability could be used to place bounds on
ultimate ruin: wherein said computation fails completely. We defer this topic
to future study.. After filtering, we can compute the means
$\overline{\Delta\chi^{SQ}}$ and standard errors in the means $\sigma^{SQ}$ of
the resulting datasets, which are displayed in the inset of Fig. 2. The
results satisfy Eq. 3, and provide a clear ‘smoking gun’ for idle information
loss in the investigated devices. More formally, we carried out a one-tailed
Welch’s t-test with null hypothesis
$H_{0}:\overline{\Delta\chi^{SP}}\leq\overline{\Delta\chi^{SR}}$ and
alternative hypothesis $H_{1}:$ Eq. 3. This yields a $z$-value of $z=3.30$
(corresponding to a $p$-value of $p=0.00049$) which implies that the
inequality holds to a high degree of statistical significance. We perform
similar tests at a range of shot counts $N_{S}$, the resulting $z$-values and
corresponding $p$-values of which are summarized in Table 1. These results
indicate that $H_{0}$ can be rejected to a very high degree of statistical
significance, and thus that true signatures of idle information loss have been
detected, across all investigated shot counts.

Shot Count $N_{S}$ |  $\mathbf{z}$-value |  $\mathbf{p}$-value
---|---|---
4000 | $z=3.3024$ | $p=4.9448\times 10^{-4}$
8000 | $z=6.0996$ | $p=7.8796\times 10^{-10}$
16000 | $z=6.7304$ | $p=2.3839\times 10^{-11}$
32000 | $z=5.4390$ | $p=4.4893\times 10^{-8}$
64000 | $z=5.0447$ | $p=4.8972\times 10^{-7}$
Table 1: Hypothesis testing for signatures of idle information loss Resulting
$z$-values and corresponding $p$-values of single-tailed Welch’s t-tests given
the null hypothesis
$H_{0}:\overline{\Delta\chi^{SP}}\leq\overline{\Delta\chi^{SR}}$ after $K=4$
filtering, for all shot counts $N_{S}$.  Figure 4: Scaling analysis and zero-
noise extrapolation Outlier-filtered $K=4$ mean values
$\overline{\Delta\chi^{SP}}$ and $\overline{\Delta\chi^{SR}}$ at different
shot counts $N_{S}$; error bars shown where visible. Dashed lines show a fit
to these values according to Eq. 6, with inset text showing the fitted values.
The extrapolated $N_{S}\to\infty$ zero shot noise values of
$\Delta\chi^{(SQ)}\to\eta^{SQ}$ quantify non-trivial leakage of information
into nearest neighbours $\eta^{SP}=0.00166\pm 0.00040$, and none into random
distant qubits in the array $\eta^{SR}=0$. Additionally, both complementary
sets of qubits $P$ and $R$ seem equally affected by shot noise
$\eta_{\text{SHOT}}^{SP}\approx\eta_{\text{SHOT}}^{SR}$. A total of $3591$
samples are distributed across the five different values of $N_{S}$ and two
types of complementary qubit sets, a breakdown of which is provided in Table
2.

###  III.2 Scaling analysis

We now interrogate the effect of shot noise more rigorously by investigating
how the statistics and mean values change as we vary the number of shots
$N_{S}$. This will form the basis of developing an ansatz of the form of Eq. 4
with which a scaling analysis can be carried out. The results of this
investigation are shown in Fig. 3 which shows the statistics of
$\Delta\chi^{SP}$ after $K=4$ filtering as a function of shot count $N_{S}$.
The resulting distributions still appear normally distributed, but drift to
lower mean values with smaller variances as $N_{S}$ is increased.
Interestingly, the mean value of $\overline{\Delta\chi^{SP}}$ falls by a
factor of $1/\sqrt{2}$ every time $N_{S}$ is doubled. This is further evidence
that shot noise, which should follow central limiting behaviour and scale as
$1/\sqrt{N_{S}}$, is completely dominating the effects of idle information
loss. We formalize this intuition by suggesting the simple ansatz

$\overline{\Delta\chi^{SQ}}=\eta^{SQ}+\frac{\eta^{SQ}_{\text{SHOTS}}}{\sqrt{N_{S}}}$
(6)

which incorporates both a flat information leakage $\eta^{SQ}$ and a term
which captures the distinguishability introduced by shot noise
$\eta_{\text{SHOTS}}$. Essentially, $\eta^{SQ}$ can be interpreted as the
number of additional bits of information we can retrieve about the initial
state of the target qubit given access to the other qubits in the
complementary set $Q$ in the zero shot noise limit $N_{S}\to\infty$.

We present a fit of the mean values of the $\Delta\chi^{SP}$ distributions
(after $K=4$ filtering) of Fig. 3 to the ansatz of Eq. 6 in Fig. 4, where we
find very good agreement between the data and our ansatz. We also present a
fitting for $K=4$ filtered $\Delta\chi^{SR}$ statistics as a benchmark. The
fitting procedure allows us to extrapolate values for $\eta^{SQ}$ and
$\eta^{SQ}_{\text{SHOTS}}$, which are shown for both $SP$ and $SQ$ in the
inset text of Fig. 4. Standard errors on these extrapolated values are
determined by a simple bootstrapping scheme in which each of the data points
are randomly drawn from normal distributions determined by their respective
means and standard deviations. These extrapolated values reveal that, as
expected, both nearest-neighbour plaquettes and randomly selected qubits are
equally affected by shot noise
$\eta_{\text{SHOTS}}^{SP}\approx\eta_{\text{SHOTS}}^{SR}$. However, the
underlying idle information loss saturates to a low but finite value
$\eta^{SP}=0.00166\pm 0.00040$ for nearest-neighbour plaquettes, and exactly
to zero (to five decimal places) $\eta^{SR}=0$ for randomly selected qubits.
This evidences the conjecture made in Section II.3 that the single-qubit
processes that lead to e.g. SPAM errors and decoherence have no systematic
effect on $\Delta\chi^{SQ}$, and that we can treat $\eta^{SQ}$ as a true zero
shot noise quantification of idle information leakage. Ultimately, the excess
information gained by having access to the joint state $SP$ when compared to
$SR$ yields, on average, an additional $0.00166$ bits of information about the
target qubit $S$. This is a low value, but it represents a direct
quantification of the impact of idle information loss, and a fundamental limit
on how well qubits in the Falcon 5.11 series of devices can perform.

##  IV Conclusions

The foremost result of this article is a protocol exploiting the Holevo
quantity from quantum information theory. This protocol provides a flexible,
scalable, device non-specific solution to the burgeoning problem of
quantifying idle information leakage in quantum computing platforms.
Sufficient degradation in single-qubit protection could destroy the ability of
a device to actually carry out quantum computations as information propagates,
and our protocol serves as a direct quantification of the information lost (in
bits) to this effect. A central component of our protocol is simply waiting,
i.e. implementing the empty circuit; and it can thus be easily run during
downtime with minimal oversight, replacing otherwise wasted time with a
valuable characterization of errors in the computing platform.

The secondary result of this article is the experimental implementation of our
protocol on 3500 samples carried out across four months on all four of IBM’s
Falcon 5.11 series of devices. The results of this analysis reveal (i) that a
measurable amount of information about the state of any given qubit is leaking
out during idling and (ii) the existence of ‘bad qubits’ which leak over
$10\%$ of a classical bit of information into their immediate surroundings.
The ‘smoking gun’ of idle information loss takes the form of a statistical
inequality which accounts for the effects of e.g. SPAM errors, decoherence,
and shot noise. We find this inequality to be satisfied to remarkably high
degrees of statistical significance at all shot counts. We also determine an
ansatz from which exact idle information loss at zero shot noise can be
extrapolated. The results of this extrapolation indicate that, after filtering
for bad qubits, approximately $0.2\%$ of the information stored locally is
lost to idle information loss during a single readout time. This is a low, and
but crucially non-zero, value which represents a fundamental limit on how well
the Falcon 5.11 series of devices can perform.

Overall, our results indicate that, in contrast to what has been suggested in
literature [15], unwanted many-body effects are not a significant concern in
current-generation IBM devices when compared to shot noise; with the exception
of occasional bad qubits. However, the main finding of our work is that a
measurable albeit small amount of information is already being lost in these
systems and we can study this accurately with our method. This should allow us
to systematically understand the impact of innovating technologies on the
protection-operation dilemma. The near future of quantum computing promises
dramatic scale-ups of system sizes, nascent error correcting hardware, novel
approaches to localizing information, and fledgling fault-tolerance. Our work
provides a flexible, powerful, scalable protocol to quantify idle information
loss in all these settings. This represents a decisive step towards addressing
the threat many-body effects pose to high-fidelity idling, and thus to the
long-term large-scale stability of generic quantum computing platforms.

##  V Acknowledgements

A.N.-K. would like to thank S. Bose, C. Turner, C. Berke, and O. Dial for
fruitful discussions at preliminary research stages. A.N.-K. sincerely thanks
S. Shaikh and C. Bleger for their support. N.K. would like to give thanks to
the QuSys group for discussions throughout the project, and also to G. García-
Pérez, S. Filippov, and E. Borrelli for giving feedback and useful references.
J.G. is supported by a SFI- Royal Society University Research Fellowship and
is grateful to IBM Ireland and Microsoft Ireland for generous financial
support.

## References

  * Altman  _et al._ [2021] E. Altman, K. R. Brown, G. Carleo, L. D. Carr, E. Demler, C. Chin, B. DeMarco, S. E. Economou, M. A. Eriksson, K.-M. C. Fu, M. Greiner, K. R. Hazzard, R. G. Hulet, A. J. Kollár, B. L. Lev, M. D. Lukin, R. Ma, X. Mi, S. Misra, C. Monroe, K. Murch, Z. Nazario, K.-K. Ni, A. C. Potter, P. Roushan, M. Saffman, M. Schleier-Smith, I. Siddiqi, R. Simmonds, M. Singh, I. Spielman, K. Temme, D. S. Weiss, J. Vučković, V. Vuletić, J. Ye, and M. Zwierlein, Quantum simulators: Architectures and opportunities, PRX Quantum 2, 017003 (2021).
  * Bravyi  _et al._ [2022] S. Bravyi, O. Dial, J. M. Gambetta, D. Gil, and Z. Nazario, The future of quantum computing with superconducting qubits, Journal of Applied Physics 132, 160902 (2022), https://pubs.aip.org/aip/jap/article-pdf/doi/10.1063/5.0082975/19808793/160902_1_online.pdf .
  * Preskill [2018] J. Preskill, Quantum Computing in the NISQ era and beyond, Quantum 2, 79 (2018).
  * Arute  _et al._ [2019] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell, _et al._ , Quantum supremacy using a programmable superconducting processor, Nature 574, 505 (2019).
  * Chow  _et al._ [2021] J. Chow, O. Dial, and J. Gambetta, Ibm quantum breaks the 100-qubit processor barrier, IBM Research Blog 2 (2021).
  * Kim  _et al._ [2023] Y. Kim, A. Eddins, S. Anand, K. X. Wei, E. Van Den Berg, S. Rosenblatt, H. Nayfeh, Y. Wu, M. Zaletel, K. Temme, _et al._ , Evidence for the utility of quantum computing before fault tolerance, Nature 618, 500 (2023).
  * Van Den Berg  _et al._ [2023] E. Van Den Berg, Z. K. Minev, A. Kandala, and K. Temme, Probabilistic error cancellation with sparse pauli–lindblad models on noisy quantum processors, Nature Physics 19, 1116 (2023).
  * Takeda  _et al._ [2022] K. Takeda, A. Noiri, T. Nakajima, T. Kobayashi, and S. Tarucha, Quantum error correction with silicon spin qubits, Nature 608, 682 (2022).
  * AI [2023] G. Q. AI, Suppressing quantum errors by scaling a surface code logical qubit, Nature 614, 676 (2023).
  * Xu  _et al._ [2024a] Q. Xu, J. P. Bonilla Ataides, C. A. Pattison, N. Raveendran, D. Bluvstein, J. Wurtz, B. Vasić, M. D. Lukin, L. Jiang, and H. Zhou, Constant-overhead fault-tolerant quantum computation with reconfigurable atom arrays, Nature Physics , 1 (2024a).
  * Xu  _et al._ [2024b] X. Xu, C. Vignes, M. H. Ansari, J. Martinis, _et al._ , Lattice hamiltonians and stray interactions within quantum processors, arXiv preprint arXiv:2402.09145 (2024b).
  * Silveri and Orell [2022] M. Silveri and T. Orell, Many-qubit protection-operation dilemma from the perspective of many-body localization, nature communications 13, 5825 (2022).
  * Qian  _et al._ [2023] P. Qian, H.-Z. Xu, P. Zhao, X. Li, and D. E. Liu, Mitigating crosstalk and residual coupling errors in superconducting quantum processors using many-body localization, arXiv preprint arXiv:2310.06618 (2023).
  * Varvelis and DiVincenzo [2024] E. Varvelis and D. P. DiVincenzo, Perturbative analysis of quasiperiodic patterning of transmon quantum computers: Enhancement of many-body localization, Phys. Rev. B 109, 144201 (2024).
  * Berke  _et al._ [2022] C. Berke, E. Varvelis, S. Trebst, A. Altland, and D. P. DiVincenzo, Transmon platform for quantum computing challenged by chaotic fluctuations, Nature communications 13, 2495 (2022).
  * Börner  _et al._ [2023] S.-D. Börner, C. Berke, D. P. DiVincenzo, S. Trebst, and A. Altland, Classical chaos in quantum computers, arXiv preprint arXiv:2304.14435 (2023).
  * Basilewitsch  _et al._ [2024] D. Basilewitsch, S.-D. Börner, C. Berke, A. Altland, S. Trebst, and C. P. Koch, Chaotic fluctuations in a universal set of transmon qubit gates, arXiv preprint arXiv:2311.14592 (2024).
  * Devitt  _et al._ [2013] S. J. Devitt, W. J. Munro, and K. Nemoto, Quantum error correction for beginners, Reports on Progress in Physics 76, 076001 (2013).
  * Ezzell  _et al._ [2023] N. Ezzell, B. Pokharel, L. Tewala, G. Quiroz, and D. A. Lidar, Dynamical decoupling for superconducting qubits: A performance survey, Phys. Rev. Appl. 20, 064027 (2023).
  * Note [1] IBM has recently discontinued this particular series, but the protocol is not device-specific.
  * Holevo [1973] A. S. Holevo, Bounds for the quantity of information transmitted by a quantum communication channel, Problems Inform. Transmission 9, 177 (1973).
  * Nielsen and Chuang [2011] M. A. Nielsen and I. L. Chuang,  _Quantum Computation and Quantum Information: 10th Anniversary Edition_ , 10th ed. (Cambridge University Press, USA, 2011).
  * Nico-Katz  _et al._ [2022a] A. Nico-Katz, A. Bayat, and S. Bose, Information-theoretic memory scaling in the many-body localization transition, Phys. Rev. B 105, 205133 (2022a).
  * Nico-Katz  _et al._ [2022b] A. Nico-Katz, A. Bayat, and S. Bose, Memory hierarchy for many-body localization: Emulating the thermodynamic limit, Phys. Rev. Res. 4, 033070 (2022b).
  * Yuan  _et al._ [2022] D. Yuan, S.-Y. Zhang, Y. Wang, L.-M. Duan, and D.-L. Deng, Quantum information scrambling in quantum many-body scarred systems, Phys. Rev. Res. 4, 023095 (2022).
  * Zhuang  _et al._ [2023] J.-Z. Zhuang, Y.-K. Wu, and L.-M. Duan, Dynamical phase transitions of information flow in random quantum circuits, Phys. Rev. Res. 5, L042043 (2023).
  * Note [2] $T$ is arbitrary in principle, and can be tuned to address different regimes or properties of a device. Taking $T$ to be e.g. fifty two-qubit gate times could give us a good indicator of the total information leakage during a complete computation.
  * Smolin  _et al._ [2012] J. A. Smolin, J. M. Gambetta, and G. Smith, Efficient method for computing the maximum-likelihood quantum state from measurements with additive gaussian noise, Phys. Rev. Lett. 108, 070502 (2012).
  * Schwertman  _et al._ [2004] N. C. Schwertman, M. A. Owens, and R. Adnan, A simple more general boxplot method for identifying outliers, Computational Statistics & Data Analysis 47, 165 (2004).
  * Yang  _et al._ [2019] J. Yang, S. Rahardja, and P. Fränti, Outlier detection: how to threshold outlier scores?, in  _Proceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing_, AIIPCC ’19 (Association for Computing Machinery, New York, NY, USA, 2019).
  * Note [3] It is thus interesting to point out that we can thus invoke $K$ as a tuning parameter which defines precisely what is meant by a ‘bad qubit’. The part of the distribution to the right of the box defined by $K$ can inform us of the probability that any given qubit will be a ‘bad’ one up to a certain informational tolerance. In cases where the entire device is required to complete a computation, or bad qubits are not identified and excluded before runtime, this probability could be used to place bounds on ultimate ruin: wherein said computation fails completely. We defer this topic to future study.

##  Appendix A Additional Results

(a) $N_{S}$ = 8000

(b) $N_{S}$ = 16000

(c) $N_{S}$ = 32000

(d) $N_{S}$ = 64000

Figure 5: Experimental sample statistics for $\Delta\chi$ Statistics of
$\Delta\chi^{SP}$ and $\Delta\chi^{SR}$ for nearest-neighbour $P$ (blue) and
random $R$ (orange) complementary sets of qubits respectively. Different shot
counts $N_{S}$ shown in (a) $N_{S}=8000$, (b) $N_{S}=16000$, (c)
$N_{S}=32000$, (d) $N_{S}=64000$. The ranges determined by Eq. 5 at a
tolerance of $K=4$ are shown as blue dashed lines for $\Delta\chi^{SP}$, and
orange dotted lines for $\Delta\chi^{SR}$. Inset shows means and standard
errors for the non-outlier region of each distribution. A noticable bump of
outliers at larger values of only $\Delta\chi^{SP}$ is visible in all cases.
Statistics are comprised of 2369 total samples, taken across four months (Dec
2023 - Mar 2024) from all the Falcon 5.11 series of devices.

Here we present additional results in Fig. 5 for the statistics of
$\Delta\chi^{SP}$ and $\Delta\chi^{SR}$, similar to Fig. 2 in the main text at
different shot counts $N_{S}$. Dashed blue and dotted orange lines indicate
the $K=4$ box-filtration fences according to the outlier filtering of Eq. 5 in
the main text for $\Delta\chi^{SP}$ and $\Delta\chi^{SR}$ respectively. In all
cases we identify a bump in $P(\Delta\chi^{SP})$ at $\Delta\chi^{SP}\gtrsim
0.12$, for the nearest-neighbour complementary set of qubits $P$ only. As
discussed in the main text, this bump corresponds to ‘bad qubits’ which leak
considerable information (over 10% of a classical bit) into their
surroundings. The fact that this bump is only visible in the nearest-neighbour
complementary set data $P$, and that it doesn’t change position with changing
shot count, suggests that it is entirely independent of shot noise or other
sources of error and is in fact a direct quantification of idle information
loss.

We also note that, as the shout count is doubled between the panels Fig.
5(a)-(d), the centre of the main Gaussian distribution decreases; indicating
that much of the signal is dominated by shot noise. We evaluate the mean
values $\overline{\Delta\chi}$ and their respective standard errors in the
mean, both of which are shown in the inset boxes in each panel. The results of
Welch’s t-tests identical to the one carried out in the main text are
summarized in Table 1, indicating that the statistics for
$\overline{\Delta\chi^{SP}}$ and $\overline{\Delta\chi^{SR}}$ are different to
a high degree of statistical significance: with $z$-values in excess of $z=5$
for all the results shown in Fig. 5. This indicates a positive detection of
idle information loss at all shot counts investigated; even if the absolute
value of this idle information loss is very small.

Finally, we note that at the higher shout counts $N_{S}=32000$ and
$N_{S}=64000$ shown in Fig. 5(c) and Fig. 5(d) respectively, there is a finite
probability of locating some results outside of the filter boundary but well
below the threshold of $\Delta\chi\gtrsim 0.12$ for unambiguous ‘bad qubits’.
This corresponds to qubits which are leaking enough information to be visible
above the background of shot noise but which may not have failed completely.

##  Appendix B Details of State Tomography

State tomography in IBM devices is carried out by measuring all $M$ qubits in
the output register ${j_{1},j_{2},\cdots,j_{M}}$, where $j_{r}$ indexes the
physical location of the $r$-th output qubit, simultaneously in the three
Pauli bases. Taking the standard Pauli matrices, $\sigma^{0}=\mathbb{I}$,
$\sigma^{1}=X$, $\sigma^{2}=Y$, $\sigma^{3}=Z$, the statistics of each Pauli
string
$P_{b}=\sigma^{b_{1}}_{j_{1}}\sigma^{b_{2}}_{j_{2}}\cdots\sigma^{b_{M}}_{j_{M}}$
with $M$ non-identity elements ($b_{r}\neq 0$ such that there are $3^{M}$ of
this family of string in total) is determined by these measurements. The
result of each measurement is a bitstring, and the result of a large number
$N_{S}$ of measurements - called ‘shots’ - is a dictionary $D_{b}$ of
bitstrings. Taking $N_{S}$ sufficiently large ensures that the sample
statistics of measuring a specific bitstring given the Pauli string $P_{b}$
are close to the population statistics. These statistics for the reduced space
of Pauli strings can then be aggregated into marginal values which yield
statistics for the full space of $4^{M}$ Pauli strings. This is done by simply
aggregating shots from different Pauli strings together wherever they coincide
everywhere except where identities occur in the desired marginal.

Shot Number $N_{S}$ | Complementary Set | Samples
---|---|---
4000 | $P$ | 609
$R$ | 600
8000 | $P$ | 507
$R$ | 480
16000 | $P$ | 324
$R$ | 288
32000 | $P$ | 252
$R$ | 204
64000 | $P$ | 157
$R$ | 157
Table 2: Total sample counts for all experimental implementations Sample
counts, before filtering according to Eq. 5, for each shot count and type of
tomographic complementary set: nearest-neigbour $P$ and random (excluding
nearest-neighbour) $R$. Each sample corresponds to a single realization of
steps 1-7 of the protocol discussed in Section II.2.

As a concrete example, consider an output register of three qubits, and lets
say we are interested in the value of $\langle XIY\rangle$. This is given by
the statistics of $XXY=P_{1}$, $XYY=P_{2}$, and $XZY=P_{3}$. For each of these
$P_{b}$ statistics, there is a corresponding dictionary of results
$D_{b}=\\{B_{b}^{s}\\}$ where the $B_{b}^{s}$ are bitstrings $x$ which are
simply the results of any single shot (measurement) given $b$. The statistics
of $P_{1}$ are given by calculation of the probability distribution
$P_{b}(B_{b}=x)=P(B=x|b)$. An example dictionary for $P_{1}$ for $N_{S}=10$
shots might be

$D_{1}=\\{101,101,101,111,001,101,101,001,101,100\\}$ (7)

with an associated probability distribution

$P_{1}(B_{1}=x)=\begin{cases}0.6\quad\text{if}~{}x=101\\\
0.1\quad\text{if}~{}x=111\\\ 0.2\quad\text{if}~{}x=001\\\
0.1\quad\text{if}~{}x=100\\\ 0.0\quad\text{otherwise}\end{cases}.$ (8)

Now consider the following example dictionaries for $P_{2}$ and $P_{3}$:

$\displaystyle D_{2}$
$\displaystyle=\\{111,101,111,111,011,111,101,011,101,101\\}$ (9)
$\displaystyle D_{3}$
$\displaystyle=\\{101,111,011,111,011,110,111,001,101,110\\}.$ (10)

We then aggregate the dictionaries $D_{b}$ into a single new dictionary
$\widetilde{D}$ which describes the statistics of $XIY$ by simply excluding
the central bit of each bitstring and aggregating the dictionaries:

$\displaystyle D_{1}$ $\displaystyle\to\\{11,11,11,11,01,11,11,01,11,10\\}$
(11) $\displaystyle D_{2}$
$\displaystyle\to\\{11,11,11,11,01,11,11,01,11,11\\}$ (12) $\displaystyle
D_{3}$ $\displaystyle\to\\{11,11,01,11,01,10,11,01,11,10\\}$ (13)
$\displaystyle\widetilde{D}$ $\displaystyle=D_{1}+D_{2}+D_{3}$ (14)

where $\widetilde{D}$ contains 30 elements. The statistics
$\widetilde{P}(\widetilde{B}=\widetilde{x})$ of $\widetilde{D}$ are then
calculated as

$\widetilde{P}(\widetilde{B}=\widetilde{x})=\begin{cases}0.6\dot{6}\quad\text{if}~{}x=11\\\
0.2\dot{3}\quad\text{if}~{}x=01\\\ 0.10\quad\text{if}~{}x=10\\\
0.00\quad\text{otherwise}\end{cases}.$ (15)

We can now evaluate $\langle XIY\rangle$ explicitly by summing up
contributions to the expectation value $1\to 1$, $0\to-1$ as follows

$\langle XIY\rangle=0.6\dot{6}(-1\times-1)+0.2\dot{3}(1\times-1)+0.1(-1\times
0)=0.\dot{3}$ (16)

which completes our example.

This may seem a laborious process, but it allows us to extrapolate
$\mathcal{O}(4^{M})$ elements of a given state’s density matrix using only
$\mathcal{O}(N_{S}3^{M})$ measurements. By decomposing the state’s density
matrix into a sum of $4^{M}$ Pauli strings (including identity elements),

$\hat{\rho}=\frac{1}{\mathcal{Z}}\sum_{b}\langle P_{b}\rangle P_{b}$ (17)

where $\mathcal{Z}$ is an appropriate normalization factor, we can readily
reconstruct the quantum state of the output register using the dictionaries
$D_{k}$. These dictionaries $D_{k}$ are ultimately what IBM’s quantum
computers return to their users. Where we discuss shot count $N_{S}$ in the
main text, it simply refers the size of these returned dictionaries; where
larger dictionaries more accurately yield the statistics of the actual
population. The total number of samples of $\Delta\chi$ for both complementary
qubit sets discussed in the main text (see Section II) is shown in Table 2 for
all shot counts $N_{S}$ we consider.

##  Appendix C Aphysicality and Maximum-Likelihood Reconstruction

As discussed in Section III of the main text, shot noise due to finite $N_{S}$
can result in aphysical density matrices by introducing negative eigenvalues
into their spectra. We correct for this using the maximum-likelihood
reconstruction of the density matrix.

The tomographic process discussed in Appendix B yields a density matrix
$\hat{\mu}$ with matrix elements $\mu_{ij}$ which is definitionally of trace
unity and hermitian by inspection of Eq. 17. The eigenvalues $\mu_{j}$ of
$\hat{\mu}$ can, however, be negative; and thus $\hat{\mu}$ is generally
unphysical.

We follow and briefly summarize here the maximum-likelihood mixed state
reconstruction algorithm given in the work of Smolin et al. [28]. First we
invoke the existence of some density matrix $\hat{\rho}\in\mathcal{P}$ with
matrix elements $\rho_{ij}$ and eigenvalues $\rho_{j}$ which minimizes the
2-norm,

$\text{min}_{\hat{\rho}\in\mathcal{P}}\parallel\hat{\rho}-\hat{\mu}\parallel_{2}^{2}=\text{min}_{\hat{\rho}\in\mathcal{P}}\sum_{ij}|\rho_{ij}-\mu_{ij}|^{2}$
(18)

where $\mathcal{P}$ is the space of physical density matrices (unit trace,
positive semi-definite, hermitian matrices). We remark here that Eq. 18 is
invariant under change of basis, and hence we choose to work in the eigenbasis
of $\hat{\mu}$ with eigenvectors $|\mu_{j}\rangle$ such that,

$\parallel\hat{\rho}-\hat{\mu}\parallel_{2}^{2}=\sum_{ij}|\rho_{ij}-\mu_{j}\delta_{ij}|^{2}$
(19)

where $\delta_{ij}$ is the Kronecker delta. Clearly, Eq. 19 is minimized when
$\hat{\rho}$ is also diagonal in this basis, i.e. the eigenvectors of
$\hat{\rho}$ are also $|\mu_{j}\rangle$, as any non-zero off-diagonal terms
$\rho_{ij}$ for $i\neq j$ strictly increases the value of Eq. 19. This reduces
the minimization procedure down from an $\mathcal{O}(\Omega^{2})$ problem
(where $\Omega$ is the total dimension of the system) to a minimization
problem in the $\Omega-1$ eigenvalues of $\hat{\rho}$,

$\text{min}_{\hat{\rho}\in\mathcal{P}}\parallel\hat{\rho}-\hat{\mu}\parallel_{2}^{2}=\text{min}_{\\{\rho_{j}\\}}\sum_{j}|\rho_{j}-\mu_{j}|^{2}$
(20)

subject to only two constraints: that $\rho_{j}\geq 0$, and that
$\sum_{j}\rho_{j}=1$. The reconstructed density matrix is finally given by

$\hat{\rho}=\sum_{j}\rho_{j}|\mu_{j}\rangle\langle\mu_{j}|.$ (21)

For the system sizes that we consider, the $\mathcal{O}(\Omega)$ minimization
problem can be solved quickly by standard numerical minimization packages.
This is the approach we use in the main text of this article. For larger
problems, Smolin et al. provide a simple algorithm after reducing the
complexity of the problem further by noting that the solution to Eq. 20
essentially involves finding a ‘pivot’ $j^{\prime}$ in the (ordered) $\mu_{j}$
wherein $\rho_{j}=0$ for $j<j^{\prime}$ and $\rho_{j}=\mu_{j}+c$ where $c$ is
a constant for $j\geq j^{\prime}$. The use of this algorithm is unnecessary
for the situations we consider in this article, and we refer the interested
reader to [28] for more details.