# Testing the necessity of complex numbers in quantum mechanics with IBM
quantum computers

Jarrett L. Lancaster<EMAIL_ADDRESS>Nicholas M. Palladino  Department
of Physics, High Point University, One University Parkway, High Point, NC
27268 USA

###### Abstract

IBM quantum computers are used to perform a recently-proposed experiment
testing the necessity of complex numbers in the standard formulation of
quantum mechanics. While the noisier devices are incapable of delivering
definitive results, it is shown that certain devices possess sufficiently
small error rates to yield convincing evidence that a faithful description of
quantum phenomena must involve complex numbers. The results are consistent
with previous experiments and robust against daily calibration for several
freely-available devices. This work demonstrates the feasibility of using
cloud-based, noisy, intermediate-scale quantum devices to test certain
foundational features of quantum mechanics.

##  I Introduction

In recent years, the field of quantum information science has experienced
tremendous growth as cloud-based, noisy-intermediate scale quantum (NISQ)
Preskill (2018) hardware has become widely available from several providers
including IBM IBM (a), Rigetti Rig , and IonQ Ion . In particular, a number of
IBM Quantum devices are currently available freely, with additional devices
and features available to members of the IBM Quantum network. The
accessibility of IBM’s cloud-based, NISQ hardware presents the incredible
opportunity for essentially anyone to perform legitimate quantum experiments
Devitt (2016) by making use of the Qiskit Qis (a) software development kit.

Long before the availability of functional NISQ technology, it was
hypothesized that quantum computers would be capable of simulating systems
which would be effectively impossible to simulate classically Feynman (1982).
Later it was realized that quantum computers can provide a highly efficient
scheme for factoring products of two prime numbers Shor (1997). As one of the
most widely-employed cryptographic frameworks is based on the impracticality
of quickly factoring composite numbers Rivest et al. (1978), this particularly
mathematical application of quantum computers is of great interest.

Recent investigations of many-body dynamics Smith et al. (2019); Bassman et
al. (2020) and cryptography Amico et al. (2019); Skosana and Tame (2021) using
currently-available systems have demonstrated the impressive development of
controllable quantum hardware. But it remains unclear when advances in
hardware design will yield devices capable of demonstrating a clear “quantum
advantage” compared to classical approaches to these interesting questions
Sevilla and Riedel (2020). The noisy devices currently available are limited
by unavoidable errors which accumulate considerably for even modest
computations. Though some simple methods exist for correcting basic
measurement errors Abbas et al. (2020), devising more elaborate and scalable
error-correcting schemes for large systems is a highly nontrivial task Harper
and Flammia (2019); Nation et al. (2021).

A natural question in this interesting time of rapid development is: what can
these noisy quantum devices do now? That is, are there any tasks for which
today’s imperfect devices are particularly well suited to demonstrate
definitive value? The aim of this paper is to demonstrate that one viable
application of current quantum hardware is to simulate small, entangled
systems. Some such cleverly designed systems are ideal settings for probing
fundamental aspects of quantum mechanics. Previous work has shown that Bell’s
theorem Bell (1964) and its extensions provide numerous opportunities Sheffer
et al. (2022); Wang et al. (2021); Sadana et al. (2022); Santini and Vitale
(2022) for these noisy devices to clearly demonstrate the validity of the
traditional framework of quantum mechanics.

Recently, a Bell-like test was conceived Renou et al. (2021) for testing the
necessity of complex numbers in the traditional formulation of quantum
mechanics. At its core, this test computes a score $\Gamma$ which, due to a
specific Clauser-Horne-Shimony-Holt (CHSH) inequality Clauser et al. (1969),
cannot exceed $\Gamma_{\mbox{\scriptsize crit}}\approx 7.66$ unless complex
numbers are employed in the mathematical description of the basic postulates
of quantum theory. The standard, complex-based quantum framework yields and
ideal score of $\Gamma=6\sqrt{2}\approx 8.49$. Two impressive experiments Chen
et al. (2022); Li et al. (2022) have since shown convincing evidence that the
traditional formulation of quantum theory does require complex-valued
quantities with experimentally obtained values of $\Gamma$ exceeding
$\Gamma_{\mbox{\scriptsize crit}}$ significantly. The main focus of the
remainder of this paper is to demonstrate how the experimental setup in Ref.
Chen et al., 2022 can be performed on IBM quantum hardware. It is shown that
consistent experimental results can be obtained on these freely-available,
cloud-based devices. Using Qiskit to assemble quantum circuits for this
experiment has the benefit of not requiring the user to possess significant
expertise in the low-level operation of the hardware (e.g., gate construction
via microwave pulses and calibration protocols), thus providing a highly
accessible pathway for using quantum technology to perform legitimate
experiments which can probe fundamental aspects of quantum theory.

It should be noted that basic Bell tests such as the CHSH inequality used in
this work come with a number of potential loopholes that make drawing
definitive conclusions about the nature of reality difficult without extremely
carefully designed experiments. In the context of a simple Bell test designed
to invalidate local realism, there are several loopholes in the simplest
experimental schemes which allow local realist theories to reproduce data
which would seemingly invalidate such theories Larsson (2014). Only recently
have so-called “loophole-free” experiments been performed for classic tests of
Bell’s inequality Hensen et al. (2015); Giustina et al. (2015); Shalm et al.
(2015). Admittedly, employing freely-available, cloud-based quantum computers
to perform Bell tests does not allow one to close such loopholes. Potential
loopholes still exist in experimental tests which claim to demonstrate the
necessity of complex numbers in the traditional formulation of quantum
mechanics, but recent progress has been made in closing several Wu et al.
(2022). Exploring how to close such loopholes is beyond the scope of this work
and is likely not possible using cloud-based quantum devices. Regardless, we
show that such freely-available and easy-to-use devices are capable of
obtaining results comparable to those obtained in much more sophisticated
experimental setups Chen et al. (2022); Li et al. (2022). Our work
demonstrates the potential value of this newly available technology for
exploring fundamental aspects of quantum mechanics with no-to-minimal cost.
Similar such explorations could identify other interesting experimental tests
of quantum mechanics in need of careful removal of loopholes.

The remainder of this paper is organized as follows: A quantum circuit for
testing the relevant CHSH inequality and the details of its implementation on
IBM hardware are described in Sec. II Results from executing this circuit on
several IBM Quantum devices are shown in Sec. III. To support the claim that
IBM Quantum devices are useful for performing quantum experiments such as Bell
tests which are accessible with small systems, particular attention is paid to
demonstrating the robustness of the results. One tradeoff of accessing cloud-
based, programmable quantum hardware with Qiksit compared to specially-
fabricated devices Chen et al. (2022) is that one generally loses at least
some control over the precise calibration of the device. Evidence is presented
to demonstrate the results obtained are robust with respect to IBM’s daily
calibrations. Additionally, we repeat the experiment on one device after a
period of roughly six months to demonstrate consistent results over larger
time scales. Finally, we conclude in Sec. IV with a discussion of the main
results and potential directions for future work.

##  II Methods

The experimental system considered is described in detail in Ref. Renou et al.
(2021), and a schematic of this experiment is depicted in Fig. 1. One
considers two independent sources, each of which creates the entangled Bell
state $|\Phi^{+}\rangle=(|00\rangle+|11\rangle)/\sqrt{2}$. From each of these
two entangled states, one qubit is sent to observer Bob (B). Bob performs a
measurement in the Bell-state basis from which he concludes that the two-qubit
system is in one of the following states:

$\displaystyle|\Phi^{\pm}\rangle$ $\displaystyle=$
$\displaystyle(|00\rangle\pm|11\rangle)/\sqrt{2},$
$\displaystyle|\Psi^{\pm}\rangle$ $\displaystyle=$
$\displaystyle(|01\rangle\pm|10\rangle)/\sqrt{2}.$ (1)

Alice performs a measurement of $x$ with
$x\in\left\\{\hat{\sigma}^{z},\hat{\sigma}^{x},\hat{\sigma}^{y}\right\\}$ on
one of the two remaining qubits, while Charlie performs a measurement $z$ with
$z\in\left\\{(\hat{\sigma}^{z}\pm\hat{\sigma}^{x})/\sqrt{2},(\hat{\sigma}^{z}\pm\hat{\sigma}^{y})/\sqrt{2},(\hat{\sigma}^{x}\pm\hat{\sigma}^{y})/\sqrt{2}\right\\}$
on the other qubit. Alice’s and Charlie’s measurement operators are chosen
randomly. The outcomes of measurements made by Alice ($a=\pm 1$) and Charlie
($c=\pm 1$) are dependent upon the choice of operators ($xz$) and also (due to
residual entanglement) dependent upon the result of Bob’s measurement $y$,
which maps the states $|\Phi^{\pm}\rangle$, $|\Psi^{\pm}\rangle$ to the binary
values $b=00,01,10,11$, respectively.

Figure 1: Experimental setup: independent sources each create entangled Bell
states $|\Phi^{+}\rangle=(|00\rangle+|11\rangle)/\sqrt{2}$. From each state,
one qubit is sent to Bob (B) who performs a measurement in the Bell state
basis. One of the remaining qubits is sent to Alice (A), and the other is sent
to Charlie (C). After Bob’s measurement, the residual entanglement is
transferred to the qubit pair shared by Alice and Charlie. Alice performs a
measurement of $x$, while Charlie performs a measurement of $z$.

Defining the joint conditional probability $P(abc|xz)$ as the probability that
Alice obtains $a$, Bob obtains $b$, and Charlie obtains $c$ given a choice
$xz$ for Alice’s and Charlie’s operators, a particularly interesting
correlation function is Renou et al. (2021)

$\displaystyle\Gamma=\sum_{abc,xz}w_{abc,xz}P(abc|xz),$ (2)

where $w_{abc,xz}=\pm 1$ are weights Renou et al. (2021); Chen et al. (2022),
$xz\in\left\\{11,12,21,22,13,14,33,34,25,26,35,36\right\\}$, and
$abc\in\left\\{0000,0001,\cdots,1110,1111\right\\}$ so that the sum over $abc$
runs over the entire four-qubit computational basis. Specifically, one defines

$\displaystyle\mathcal{T}_{b}$ $\displaystyle=$
$\displaystyle(-1)^{b_{2}}\left(S_{11}^{b}+S_{12}^{b}\right)+(-1)^{b_{1}}\left(S_{21}^{b}-S_{22}^{b}\right)$
(3) $\displaystyle+$
$\displaystyle(-1)^{b_{2}}\left(S_{13}^{b}+S_{14}^{b}\right)-(-1)^{b_{1}+b_{2}}\left(S_{33}^{b}-S_{34}^{b}\right)$
$\displaystyle+$
$\displaystyle(-1)^{b_{1}}\left(S_{25}^{b}+S_{26}^{b}\right)-(-1)^{b_{1}+b_{2}}\left(S_{35}^{b}-S_{36}^{b}\right),$

where

$\displaystyle S_{xz}^{b}=\sum_{a,c=\pm 1}P(abc|xz),$ (4)

and $b\in\left\\{00,01,10,11\right\\}$. The connection between $\Gamma$ and
$\mathcal{T}_{b}$ is

$\displaystyle\Gamma=\sum_{b}\mathcal{T}_{b}=6\sqrt{2}\approx 8.49,$ (5)

where the approximate value represents an ideal score for a quantum
description based on complex numbers.

It has been demonstrated Renou et al. (2021) that $\Gamma\lesssim 7.66$ for a
standard formulation of quantum mechanics which relies upon only real
quantities. The main goal of this work is to measure $\Gamma$ experimentally
using IBM quantum devices. A quantum circuit which realizes this experiment is
shown in Fig. 2. To make sense of the circuit, note that the two qubit basis
states can be represented as two-component vectors

$\displaystyle|0\rangle$ $\displaystyle\dot{=}\left(\begin{array}[]{c}1\\\
0\end{array}\right),\;\;\;\;\;|1\rangle\dot{=}\left(\begin{array}[]{c}0\\\
1\end{array}\right).$ (10)

The single-qubit Hadamard gate $\hat{H}$ acts according to
$\hat{H}|0\rangle=\frac{1}{\sqrt{2}}\left(|0\rangle+|1\rangle\right)$,
$\hat{H}|1\rangle=\frac{1}{\sqrt{2}}\left(|0\rangle-|1\rangle\right)$ and
takes the matrix representation

$\displaystyle\hat{H}$
$\displaystyle\dot{=}\frac{1}{\sqrt{2}}\left(\begin{array}[]{cc}1&1\\\
1&-1\end{array}\right).$ (13)

The controlled NOT (CNOT) gate flips a target qubit only if the control qubit
is in the state $|1\rangle$, so its matrix representation is

CNOT $\displaystyle\dot{=}$
$\displaystyle\left(\begin{array}[]{cccc}1&0&0&0\\\ 0&1&0&0\\\ 0&0&0&1\\\
0&0&1&0\end{array}\right).$ (18)

IBM hardware initializes all qubits in the quantum register to the $|0\rangle$
state. In order to create the entangled states $|\Phi^{+}\rangle$ required for
the experiment, one may apply a combination of $\hat{H}$ and CNOT to the state
$|00\rangle$ as depicted in Fig. 2 to obtain

$\displaystyle\mbox{CNOT}_{i,j}[\hat{H}\otimes\hat{I}]|0\rangle_{i}\otimes|0\rangle_{j}$
$\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}\left(|00\rangle+|11\rangle\right).$ (19)

With two copies of $|\Phi^{+}\rangle$, Bob must perform a Bell state
measurement (BSM) with one qubit from each pair. Measurement is only possible
in the computational basis on IBM devices, so that any observer will only
conclude that the qubit was in the state $|0\rangle$ or the state $|0\rangle$.
However, it can be shown that the application of $\hat{H}$ followed by CNOT
converts

$\displaystyle|\Phi^{+}\rangle$ $\displaystyle\rightarrow$
$\displaystyle|00\rangle,$ $\displaystyle|\Phi^{-}\rangle$
$\displaystyle\rightarrow$ $\displaystyle|10\rangle,$
$\displaystyle|\Psi^{+}\rangle$ $\displaystyle\rightarrow$
$\displaystyle|01\rangle,$ $\displaystyle|\Psi^{-}\rangle$
$\displaystyle\rightarrow$ $\displaystyle|11\rangle.$ (20)

Thus, Bob’s actual result of ‘00’, ‘01’, ‘10’, or ‘11’ allows an unambiguous
determination of the two-qubit state in the Bell state basis prior to the gate
transformations. For Alice and Charlie to measure Pauli operators
$\hat{\sigma}^{j}$ (or combinations thereof), each must rotate the qubit to
bring the measurement basis into the computational basis. Given an axis of
rotation $P$, the rotation gate for angle $\theta$ is given by

$\displaystyle\hat{R}_{\hat{\bf
n}}(\theta)=\cos\frac{\theta}{2}\hat{I}-i\sin\frac{\theta}{2}\hat{\sigma}^{\hat{\bf
n}}.$ (21)

The basic idea is that one can measure $\hat{\sigma}^{\hat{\bf n}}$ where
$\hat{\bf n}=\cos\phi\sin\theta\hat{\bf x}+\sin\phi\sin\theta\hat{\bf
y}+\cos\theta\hat{\bf z}$ by first rotating the qubit by $-\phi$ about the
$z$-axis (applying $R_{z}(-\phi)$ and then rotating by $-\theta$ about the
$y$-axis (applying $R_{y}(-\theta)$ to bring the relevant measurement into the
computational basis. Given Alice’s choice of
$x\in\left\\{\hat{\sigma}^{z},\hat{\sigma}^{x},\hat{\sigma}^{y}\right\\}$ and
Charlie’s choice
$z\in\left\\{(\hat{\sigma}^{z}\pm\hat{\sigma}^{x})/\sqrt{2},(\hat{\sigma}^{z}\pm\hat{\sigma}^{y})/\sqrt{2},(\hat{\sigma}^{x}\pm\hat{\sigma}^{y})/\sqrt{2}\right\\}$,
there are a total of $18$ individual configurations $xz$. Of these, 12 are
used in the computation of $\Gamma$, so we restrict attention to
$xz\in\left\\{11,12,21,22,13,14,33,34,25,26,35,36\right\\}$. For example,
$xz=36$ corresponds to Alice measuring $\hat{\sigma}^{y}$ and Charlie
measuring $(\hat{\sigma}^{x}-\hat{\sigma}^{y})/\sqrt{2}$. To rotate each qubit
appropriately so that a computational-basis measurement returns each of these
spin projections, Alice must rotate her qubit by $-\phi_{1}=-\frac{\pi}{2}$
about the $z$-axis followed by a rotation of $-\theta_{1}=-\frac{\pi}{2}$
(with $\phi_{1}=0$) about the $y$ axis. Charlie must rotate his qubit by
$-\phi_{2}=\frac{\pi}{4}$ about the $z$-axis and then by
$-\theta_{2}=-\frac{\pi}{2}$ about the $y$-axis. A complete list of rotation
angles $\phi_{1,2}$, $\theta_{1,2}$ for each choice of $xz$ is given below in
Table 1

$xz$ | $\phi_{1}$ | $\theta_{1}$ | $\phi_{2}$ | $\theta_{2}$
---|---|---|---|---
11 | 0 | 0 | 0 | $\pi/4$
12 | 0 | 0 | 0 | $-\pi/4$
21 | 0 | $\pi/2$ | 0 | $\pi/4$
22 | 0 | $\pi/2$ | 0 | $-\pi/4$
13 | 0 | 0 | $\pi/2$ | $\pi/4$
14 | 0 | 0 | $-\pi/2$ | $\pi/4$
33 | $\pi/2$ | $\pi/2$ | $\pi/2$ | $\pi/4$
34 | $\pi/2$ | $\pi/2$ | $-\pi/2$ | $\pi/4$
25 | 0 | $\pi/2$ | $\pi/4$ | $\pi/2$
26 | 0 | $\pi/2$ | $-\pi/4$ | $\pi/2$
35 | $\pi/2$ | $\pi/2$ | $\pi/4$ | $\pi/2$
36 | $\pi/2$ | $\pi/2$ | $-\pi/4$ | $\pi/2$
Table 1: Rotation angles used for all 12 cases of $xz$. Figure 2: Circuit to
run on IBM Quantum devices: two pairs of Bell states $|\Phi^{+}\rangle$ are
created. Bob performs a Bell state measurement on a pair constructed from both
sets of qubits, while Alice and Charlie make particular measurements on the
two remaining qubits. Figure 3: Qubit connectivity and actual usage in this
work for IBM devices used: (a) qubits 2,1,3,5 were used on ibm_nairobi; (b)
qubits 0,1,2,3 were used on ibmq_manila and ibmq_lima; (c) qubits 0,1,3,5 were
used on ibm_lagos.

The circuit in Fig. 2 can be used to reconstruct the joint conditional
probability distribution $P(abc|xz)$ needed to evaluate the score $\Gamma$ in
Eq. (2) by randomly choosing measurements $xz$ and repeatedly executing the
circuit. It is possible to sample $xz$ randomly and send a set of circuits to
the IBM devices for single-shot experiments. However, one is limited by the
maximum number of circuits which can be executed within a particular job, so
the maximum number of single-shot runs is $\mathcal{O}(10^{2})$. Since one may
execute the same circuit for at least 20,000 “shots,” it is computationally
advantageous to execute each of the 12 possible cases of $xz$ for some equal
and large number of shots. In what follows, the random sampling over $xz$ will
be mimicked by an equal sampling of $N\geq 20,000$ shots over each of the 12
configurations for $xz$.

Figure 4: Final circuit results, $P(abc|xz)$, where the 16 possible four-qubit
states are indexed vertically and the horizontal axis depicts one of the 12
possible Alice/Charlie measurement combinations $xz$. Theoretical prediction
(upper left) follows from standard quantum mechanics calculation of the gate
operations; QASM simulator refers to IBM’s quantum assembly language (QASM)
simulator. The label “e.m.” on lower plots indicates that basic readout error
mitigation as described in the text has been performed.

To minimize errors in the execution of the circuit in Fig. 2, it is essential
to ensure the physical qubits employed are connected in such a way that the
CNOT gates can be implemented directly. The currently available IBM Quantum
devices possess several types of qubit topology. Figure 3 depicts which subset
of qubits is chosen for each device used. While multiple options are typically
available for each device, the particular choices used in this work are
dictated by trying to minimize CNOT and measurement readout errors. Detailed
calibration information is included in the supplemental material sup .

##  III Results

The circuit in Fig. 2 has been sent to several IBM Quantum devices: ibmq_lima
v1.0.35, a Falcon r4T processor; ibmq_manila v1.0.29, a Falcon r5.11L
processor; ibm_lagos v1.0.27, a Falcon r5.11H processor; and ibm_nairobi
v1.0.26, a Falcon 5.11H processor. The first two machines are five-qubit
devices which are freely available, while ibm_lagos is a seven-qubit device
currently available to members of the IBM Quantum Researchers Program IBM (b).
The device ibm_nairobi is a freely-available, seven-qubit processor. For the
five-qubit devices, 20,000 executions (“shots”) per choice of $xz$ are
employed in a single job. The seven qubit devices allowed 32,000 shots per
choice of $xz$. Each job yielded a list of counts for each of the 16 possible
four-qubit states. The conditional probabilities $P(abc|xz)$ can then be
reconstructed, and $\Gamma$ can be computed from Eq. (2). Samples of
reconstructed probability distributions using several devices are shown in
Fig. 4 along with results from the Quantum Assembly Language (QASM) simulator
and theoretical predictions. The QASM simulator is used to verify that the
circuit accurately simulates the experiment in an ideal, noise-free
environment with only statistical fluctuations.

With $\mathcal{O}(10^{4})$ shots per choice of $xz$, statistical fluctuations
are expected to be small. Noise in the devices leads to errors which can be
orders of magnitude larger than statistical fluctuations. Additionally, the
devices are calibrated daily, leading to more significant variations in
results from day to day than predicted from statistical analysis (e.g.,
standard error of mean). We quantify the uncertainty associated with all of
these factors roughly by repeating the job at least three times per experiment
for three separate experiments. For example, we performed measurements using
ibm_lagos on the following dates: 2022-04-12, 2022-04-19, 2022-04-24. From
each measurement date, we store the lowest, highest, and median score obtained
for the $n\geq$ measurements of $\Gamma$. The nine points consisting of these
three values from three temporally-separated experiments are used to compute
the values (mean $\Gamma$ and standard deviation $\sigma$) summarized in Table
2 and depicted graphically in Fig. 5. All data collected, as well as relevant
calibration metrics obtained during the measurement events, are tabulated in
the supplemental material sup .

Device | Shots per $xz$ | $\;\Gamma$ | $\sigma$
---|---|---|---
QASM simulator | 32,000 | 8.486 | 0.015
ibmq_lima | 20,000 | 7.680 | 0.041
ibmq_manila | 20,000 | 7.823 | 0.151
ibm_lagos | 32,000 | 8.046 | 0.065
ibm_nairobi | 32,000 | 8.410 | 0.072
Table 2: Measured (mean) scores and standard deviation for the QASM simulator
and IBM devices employing basic readout error mitigation averaged over data
collected across several days during which multiple device calibrations occur
between computations. Figure 5: Mean score $\Gamma$ for the QASM simulator and
various IBM devices across several days (see main text) with readout error
mitigation applied to results for actual devices. The lower dotted line
denotes the quantum/classical boundary $\Gamma=6$, the lower dashed line at
$\Gamma\approx 7.66$ denotes the upper limit for real variables, and the upper
dashed line $\Gamma=6\sqrt{2}$ denotes the ideal score for a complex quantum
system. Error bars correspond to $\sigma$ described in main text.

Results in Table 2 detail measured values of $\Gamma$ after applying basic,
readout-error mitigation Abbas et al. (2020). To motivate the readout error
mitigation, note that any job returns a set of “counts” as its result. As a
simplified example, consider a two-qubit system in which both qubits are
measured. The final result will be a set of counts corresponding to the number
of times each of the four possible two-qubit states is measured. In the
simplest possible two-qubit circuit, one creates a state (say $|00\rangle$)
and measures both qubits. Supposing this circuit is run $N=1024$ times, the
existence of errors means that measurements will not always correspond to the
state $|00\rangle$. As an example, one might find $900$ instances of
$|00\rangle$, $61$ instances of $|01\rangle$, $54$ instances of $|10\rangle$,
and $9$ instances of $|11\rangle$. Repeating this process for all two-qubit
states $|b_{0}b_{1}\rangle$ with $b_{0,1}\in\left\\{0,1\right\\}$, one may
build a calibration matrix ${\bf C}$ for which each of these calibration runs
yields a row,

$\displaystyle{\bf m}_{\mbox{\scriptsize exp}}={\bf C}{\bf
m}_{\mbox{\scriptsize ideal}}.$ (22)

Here ${\bf m}_{\mbox{\scriptsize exp}}$ are the obtained counts (in the basis
$|00\rangle,\cdots,|11\rangle$, and ${\bf m}_{\mbox{\scriptsize ideal}}$ are
the expected results. This matrix ${\bf C}$ obtained from basic state
measurement on a particular device can then be inverted and interpreted as an
average correction transformation for results obtained for any circuit which
should approximately correct for any measurement errors in the readout
process. In this work, we only consider this most basic type of error
mitigation.

One potentially problematic aspect of this simple procedure is that the
corrected counts can attain negative values, which is clearly unphysical. It
is still possible to compute reasonable expectation values from these
corrected counts, interpreting the potentially negative probabilities as
quasiprobabilities. In our application of this procedure, negative counts are
obtained as corrected counts for some results obtained on ibmq_lima,
ibmq_nairobi, and for a small number ($n=3$) of individual results from
ibmq_manila. No results obtained from ibmq_lagos yielded negative counts upon
applying the basic readout-error mitigation described above, and the
magnitudes of the negative counts from other devices are fairly small
fractions ($\lesssim 10^{-2}$) of the total shots per job. Regardless, it is
desirable to explore whether such negative counts can artificially inflate the
overall score $\Gamma$ in a given experiment. An alternative procedure to the
direct inversion of ${\bf C}$ is to perform a constrained least-squares
optimization to solve the system

$\displaystyle{\bf y}$ $\displaystyle=$ $\displaystyle{\bf A}{\bf x}$ (23)

for ${\bf x}={\bf m}_{\mbox{\scriptsize ideal}}$, given ${\bf A}={\bf C}$ and
${\bf y}={\bf m}_{\mbox{\scriptsize exp}}$. The constraint $x_{i}>0$ is
applied to yield a non-negative probability distribution ${\bf x}$, where
normalization is imposed so that $||{\bf x}||_{2}=1$. The procedure can be
implemented easily using the SciPy Virtanen et al. (2020) optimization routine
nnls(A,y) or the Matlab MATLAB (2022) function lsqnonneg(A,y), both based on
the algorithm published by Lawson and Hanson Lawson and Hanson (1987). Results
obtained with constrained optimization are identical to those obtained by
directly inverting the calibration matrix when negative counts are not
obtained. While the effective difference in computation of $\Gamma$ is fairly
small, the reported values for $\Gamma$ in Table 2 correspond the the smaller
score obtained from constrained optimization where any difference occurs. All
scores obtained using both types of readout-error mitigation are included in
the supplemental material sup . Where any difference occurs, the constrained
optimization yields a (slightly) lower score. It should be noted that the
particular form of readout-error mitigation—basic inversion of the measurable
calibration matrix and least-squares optimization—is not necessarily scalable
to large systems Nation et al. (2021). However, these basic methods are easily
applied to the simple circuit considered in this work.

From Table 2 and Fig. 5, it is evident that the noisiest device, ibmq_lima,
yielded values of $\Gamma$ which hover around the boundary between real-valued
and complex-valued quantum mechanics Renou et al. (2021). The devices
ibmq_manila, ibm_lagos, and ibm_nairobi give values for $\Gamma$ which
comfortably exceed this bound of 7.66, but only the seven qubit devices give
results for $\Gamma$ which decisively exceed $\Gamma_{\mbox{\scriptsize
crit}}$ when fluctuations across several days are taken into account with the
estimate of standard deviation $\sigma$.

A potentially confounding aspect of using cloud-based devices to perform
experiments via Qiskit is that the user has little control over the
calibration and tuning of the devices. Strictly speaking, the Qiskit Pulse
programming kit Qis (b) does afford users the ability to specify pulse-level
timing dynamics in the hardware. Such low-level control could potentially
improve the quality of results obtained from these devices. The intent of the
present work is to demonstrate that certain devices (e.g., ibm_lagos,
ibm_nairobi ) can deliver reliable results for Bell-test-like experiments even
without such low-level control. Only a basic working knowledge of quantum
circuits with the simplest form of readout-error mitigation is necessary to
obtain the results presented.

IBM devices are calibrated daily, and the relevant parameters are easily
accessible to users. The calibrations affect the tuning of microwave pulses
used to manipulate the qubits, and these daily calibrations tend to cause the
results to “drift” significantly compared to differences attributable to
statistical fluctuations for results collected within the same day. Data have
been collected across several days during which several calibrations cycles
have occurred. Detailed calibration information is collected in the
supplemental material sup . The data collected from ibm_lagos and ibm_nairobi
give results which are consistently well above the real/complex threshold. The
results from ibmq_lima and ibmq_manila drift substantially enough that making
firm conclusions from these results is difficult.

An interesting question concerns how consistently these results can be
obtained over longer time scales. That is, this technology would not be
particularly useful for performing experiments if the results varied
substantially over a period of weeks or months. The data collection described
in Sec. II took place over a period of 10-14 days to allow several system
calibrations to occur between individual experiments. While the score
fluctuations within a particular day appeared well described by the same
statistical uncertainty associated with the QASM simulator results,
significantly larger drift in the score was observed across different days.
Presumably such drift follows from device calibration which resulted in new
values for readout error rates and other system parameters. A detailed table
of system parameters for each device used during each period of data
collection can be found in the supplemental material sup . However, collecting
data in three sets over a period of 14 days does little to address the
consistency of results over significant timescales. Indeed, it is much less
impressive to obtain a score $\Gamma>\Gamma_{\mbox{\scriptsize crit}}$ if it
is not reproducible on the same device several weeks later.

To address how robust the results are over longer timescales, we have repeated
the experiment using the same qubits on ibm_lagos roughly six months after
initial data collection. While ibm_nairobi gives consistently stronger
results, it was not available on the IBM cloud during the initial phase of
data collection. Upon repeating the data collection in October 2022, we obtain
$\Gamma=8.115\pm 0.042$ compared to $\Gamma=8.046\pm 0.065$ as obtained in
April 2022.

##  IV Discussion

Given that the results from ibm_lagos and ibm_nairobi are robust with respect
to daily calibrations in yielding an experimental score of $\Gamma\geq
8.0>7.66$, it is notable that Ref. Chen et al., 2022 found $\Gamma=8.09(1)$
using a custom-designed quantum processor. The IBM devices ibm_lagos and
ibm_nairobi yield scores $\Gamma=8.046\pm 0.065$ and $\Gamma=8.410\pm 0.074$,
respectively. Thus, the present work shows that the freely-available IBM
processors are capable of obtaining convincing results for the necessity of
complex numbers in the traditional formulation of quantum mechanics while only
requiring the user to arrange a set of basic quantum circuit elements. No
device yields impressive results without readout-error mitigation, but all
hardware considered yields experimental values of $\Gamma$ which approach or
exceed the threshold for the necessity of complex numbers in a traditional
formulation of quantum mechanics.

Drift in results due to the daily calibration of IBM devices makes definitive
results difficult to extract from the five-qubit devices in this particular
experiment, but the results from ibm_lagos and ibm_nairobi appear to not
experience substantial drift and are in good agreement with previous
experimental results Chen et al. (2022); Li et al. (2022); Wu et al. (2022).
Taken together, the results confirm the fairly primitive nature of currently
available hardware while also demonstrating that some hardware is capable of
performing delicate experiments consistently. Moreover, we have demonstrated
that the results of at least one device (ibm_lagos) are consistent over a
timescale of about several months, despite the numerous daily calibrations.

In addition to the scientific value of performing experiments which probe
foundational features of quantum mechanics on freely-available quantum
hardware, there is another opportunity afforded by the use of these Bell
tests: benchmarking particular quantum devices. With careful experiments being
performed that can conclusively determine the expected outcome for such Bell
tests Chen et al. (2022); Li et al. (2022); Wu et al. (2022), these types of
tests can be seen as a particular way to benchmark the performance of quantum
hardware Sheffer et al. (2022); Wang et al. (2021); Sadana et al. (2022);
Santini and Vitale (2022). Quantifying the degree to which these devices
accurately simulate quantum processes is difficult. Even with the detailed
calibration information readily accessible sup , it is not clear how to
predict how well a given device will perform on any particular task. In the
data collected for this work, there was no clear correlation between
particular calibration metrics (e.g., average or maximum $CNOT$ error rate)
and the final score obtained. It would be an interesting direction for future
work to perform a careful multivariable regression or make use of techniques
from machine learning to explore whether certain combinations of calibration
metrics directly correlate with higher scores.

###### Acknowledgements.

The authors acknowledge the use of IBM Quantum IBM (a) services for this work.
The views expressed are those of the author and do not reflect the official
policy or position of IBM or the IBM Quantum team. Additionally, the authors
acknowledge access to advanced services provided by the IBM Quantum
Researchers IBM (b) and Educators IBM (c) programs.

## References

  * Preskill (2018) J. Preskill, Quantum 2, 79 (2018), ISSN 2521-327X, URL https://doi.org/10.22331/q-2018-08-06-79.
  * IBM (a) _IBM Quantum_ , https://www.ibm.com/quantum-computing, accessed: 2022-04-15.
  * (3) _Rigetti_ , https://www.rigetti.com/, accessed: 2022-04-15.
  * (4) _IonQ_ , https://ionq.com/, accessed: 2022-04-15.
  * Devitt (2016) S. J. Devitt, Phys. Rev. A 94, 032329 (2016), URL https://link.aps.org/doi/10.1103/PhysRevA.94.032329.
  * Qis (a) _Qiskit_ , https://qiskit.org/, accessed: 2022-10-11.
  * Feynman (1982) R. P. Feynman, International Journal of Theoretical Physics 21, 467 (1982), ISSN 1572-9575, URL https://doi.org/10.1007/BF02650179.
  * Shor (1997) P. W. Shor, SIAM J. Comput. 26, 1484–1509 (1997), ISSN 0097-5397, URL https://doi.org/10.1137/S0097539795293172.
  * Rivest et al. (1978) R. L. Rivest, A. Shamir, and L. Adleman, Commun. ACM 21, 120–126 (1978), ISSN 0001-0782, URL https://doi.org/10.1145/359340.359342.
  * Smith et al. (2019) A. Smith, M. S. Kim, F. Pollmann, and J. Knolle, npj Quantum Information 5, 106 (2019), ISSN 2056-6387, URL https://doi.org/10.1038/s41534-019-0217-0.
  * Bassman et al. (2020) L. Bassman, K. Liu, A. Krishnamoorthy, T. Linker, Y. Geng, D. Shebib, S. Fukushima, F. Shimojo, R. K. Kalia, A. Nakano, et al., Phys. Rev. B 101, 184305 (2020), URL https://link.aps.org/doi/10.1103/PhysRevB.101.184305.
  * Amico et al. (2019) M. Amico, Z. H. Saleem, and M. Kumph, Phys. Rev. A 100, 012305 (2019), URL https://link.aps.org/doi/10.1103/PhysRevA.100.012305.
  * Skosana and Tame (2021) U. Skosana and M. Tame, Scientific Reports 11, 16599 (2021), ISSN 2045-2322, URL https://doi.org/10.1038/s41598-021-95973-w.
  * Sevilla and Riedel (2020) J. Sevilla and C. J. Riedel, _Forecasting timelines of quantum computing_ (2020), URL https://arxiv.org/abs/2009.05045.
  * Abbas et al. (2020) A. Abbas, S. Andersson, A. Asfaw, A. Corcoles, L. Bello, Y. Ben-Haim, M. Bozzo-Rey, S. Bravyi, N. Bronn, L. Capelluto, et al., _Learn Quantum Computation Using Qiskit_ (2020), URL http://community.qiskit.org/textbook.
  * Harper and Flammia (2019) R. Harper and S. T. Flammia, Phys. Rev. Lett. 122, 080504 (2019), URL https://link.aps.org/doi/10.1103/PhysRevLett.122.080504.
  * Nation et al. (2021) P. D. Nation, H. Kang, N. Sundaresan, and J. M. Gambetta, PRX Quantum 2, 040326 (2021), URL https://link.aps.org/doi/10.1103/PRXQuantum.2.040326.
  * Bell (1964) J. S. Bell, Physics Physique Fizika 1, 195 (1964), URL https://link.aps.org/doi/10.1103/PhysicsPhysiqueFizika.1.195.
  * Sheffer et al. (2022) M. Sheffer, D. Azses, and E. G. Dalla Torre, Advanced Quantum Technologies 5, 2100081 (2022), eprint https://onlinelibrary.wiley.com/doi/pdf/10.1002/qute.202100081, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/qute.202100081.
  * Wang et al. (2021) D. Z. Wang, A. Q. Gauthier, A. E. Siegmund, and K. L. C. Hunt, Phys. Chem. Chem. Phys. 23, 6370 (2021), URL http://dx.doi.org/10.1039/D0CP05444E.
  * Sadana et al. (2022) S. Sadana, L. Maccone, and U. Sinha, Phys. Rev. Research 4, L022001 (2022), URL https://link.aps.org/doi/10.1103/PhysRevResearch.4.L022001.
  * Santini and Vitale (2022) A. Santini and V. Vitale, Phys. Rev. A 105, 032610 (2022), URL https://link.aps.org/doi/10.1103/PhysRevA.105.032610.
  * Renou et al. (2021) M.-O. Renou, D. Trillo, M. Weilenmann, T. P. Le, A. Tavakoli, N. Gisin, A. Acín, and M. Navascués, Nature 600, 625 (2021), ISSN 1476-4687, URL https://doi.org/10.1038/s41586-021-04160-4.
  * Clauser et al. (1969) J. F. Clauser, M. A. Horne, A. Shimony, and R. A. Holt, Phys. Rev. Lett. 23, 880 (1969), URL https://link.aps.org/doi/10.1103/PhysRevLett.23.880.
  * Chen et al. (2022) M.-C. Chen, C. Wang, F.-M. Liu, J.-W. Wang, C. Ying, Z.-X. Shang, Y. Wu, M. Gong, H. Deng, F.-T. Liang, et al., Phys. Rev. Lett. 128, 040403 (2022), URL https://link.aps.org/doi/10.1103/PhysRevLett.128.040403.
  * Li et al. (2022) Z.-D. Li, Y.-L. Mao, M. Weilenmann, A. Tavakoli, H. Chen, L. Feng, S.-J. Yang, M.-O. Renou, D. Trillo, T. P. Le, et al., Phys. Rev. Lett. 128, 040402 (2022), URL https://link.aps.org/doi/10.1103/PhysRevLett.128.040402.
  * Larsson (2014) J.-Å. Larsson, Journal of Physics A: Mathematical and Theoretical 47, 424003 (2014), URL https://doi.org/10.1088/1751-8113/47/42/424003.
  * Hensen et al. (2015) B. Hensen, H. Bernien, A. E. Dréau, A. Reiserer, N. Kalb, M. S. Blok, J. Ruitenberg, R. F. L. Vermeulen, R. N. Schouten, C. Abellán, et al., Nature 526, 682 (2015), ISSN 1476-4687, URL https://doi.org/10.1038/nature15759.
  * Giustina et al. (2015) M. Giustina, M. A. M. Versteegh, S. Wengerowsky, J. Handsteiner, A. Hochrainer, K. Phelan, F. Steinlechner, J. Kofler, J.-A. Larsson, C. Abellán, et al., Phys. Rev. Lett. 115, 250401 (2015), URL https://link.aps.org/doi/10.1103/PhysRevLett.115.250401.
  * Shalm et al. (2015) L. K. Shalm, E. Meyer-Scott, B. G. Christensen, P. Bierhorst, M. A. Wayne, M. J. Stevens, T. Gerrits, S. Glancy, D. R. Hamel, M. S. Allman, et al., Phys. Rev. Lett. 115, 250402 (2015), URL https://link.aps.org/doi/10.1103/PhysRevLett.115.250402.
  * Wu et al. (2022) D. Wu, Y.-F. Jiang, X.-M. Gu, L. Huang, B. Bai, Q.-C. Sun, X. Zhang, S.-Q. Gong, Y. Mao, H.-S. Zhong, et al., Phys. Rev. Lett. 129, 140401 (2022), URL https://link.aps.org/doi/10.1103/PhysRevLett.129.140401.
  * (32) _Supplemental Material_.
  * IBM (b) _IBM Quantum Researchers Program_ , https://quantum-computing.ibm.com/programs/researchers, accessed: 2022-04-15.
  * Virtanen et al. (2020) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al., Nature Methods 17, 261 (2020).
  * MATLAB (2022) MATLAB, _version 9.12.0.1884302 (R2022a)_ (The MathWorks Inc., Natick, Massachusetts, 2022).
  * Lawson and Hanson (1987) C. L. Lawson and R. J. Hanson, _Solving Least Squares Problems_ (Society for Industrial and Applied Mathematics, 1987), ISBN 978-0898713565.
  * Qis (b) _Qiskit Pulse_ , https://qiskit.org/documentation/apidoc/pulse.html, accessed: 2022-10-11.
  * IBM (c) _IBM Quantum Educators Program_ , https://quantum-computing.ibm.com/programs/educators, accessed: 2022-10-11.

## Supplementary Material

###  IV.1 Calibration data

Tables S1–S8 depict calibration metrics for the devices used at the time these
devices were used to generate the data depicted in the main text. As the IBM
devices are calibrated on an approximately daily basis, the actual results
obtained from the hardware can drift substantially from day to day.
Calibration metrics affect the precise manner in which circuit gates
implemented in Qiskit are translated into microwave pulses used to manipulate
the physical qubits. Each quantum circuit used in this work has been executed
at least 20,000 times, resulting in a fairly small statistical uncertainty
which is generally smaller than the “drift” in results that occurs as a
consequence of daily calibrations.

####  IV.1.1 ibmq_lima

Date | qubit |  $T_{1}$ ($\mu$s) |  $T_{2}$ ($\mu$s) | freq. (GHz) | Anharmonicity (GHz) | readout assignment error
---|---|---|---|---|---|---
2022-04-14 | 0 | 93.46 | 118.47 | 5.030 | $-0.33574$ | $1.78\times 10^{-2}$
2022-04-14 | 1 | 122.62 | 131.38 | 5.128 | $-0.31835$ | $1.67\times 10^{-2}$
2022-04-14 | 2 | 117.13 | 131.29 | 5.247 | $-0.33360$ | $2.22\times 10^{-2}$
2022-04-14 | 3 | 135.20 | 106.71 | 5.303 | $-0.33124$ | $2.68\times 10^{-2}$
2022-04-14 | 4 | 21.41 | 21.57 | 5.092 | $-0.33447$ | $5.10\times 10^{-2}$
2022-04-19 | 0 | 71.55 | 97.69 | 5.030 | $-0.33574$ | $2.27\times 10^{-2}$
2022-04-19 | 1 | 123.87 | 107.28 | 5.128 | $-0.31835$ | $1.50\times 10^{-2}$
2022-04-19 | 2 | 123.98 | 134.58 | 5.247 | $-0.33360$ | $2.30\times 10^{-2}$
2022-04-19 | 3 | 96.41 | 96.81 | 5.303 | $-0.33124$ | $2.92\times 10^{-2}$
2022-04-19 | 4 | 22.11 | 22.55 | 5.092 | $-0.33447$ | $6.20\times 10^{-2}$
2022-04-24 | 0 | 169.65 | 188.84 | 5.030 | $-0.33574$ | $2.30\times 10^{-2}$
2022-04-24 | 1 | 125.59 | 117.96 | 5.128 | $-0.31835$ | $1.47\times 10^{-2}$
2022-04-24 | 2 | 78.30 | 96.82 | 5.247 | $-0.33360$ | $2.22\times 10^{-2}$
2022-04-24 | 3 | 101.35 | 69.09 | 5.303 | $-0.33124$ | $3.31\times 10^{-2}$
2022-04-24 | 4 | 23.48 | 22.98 | 5.092 | $-0.33447$ | $4.82\times 10^{-2}$
Table S1: Calibration data for qubits in ibmq_lima. Date | connection | error rate | gate time (ns)
---|---|---|---
2022-04-14 | 0-1 | $5.290\times 10^{-3}$ | 305.777
2022-04-14 | 1-2 | $2.632\times 10^{-2}$ | 334.222
2022-04-14 | 1-3 | $1.263\times 10^{-2}$ | 497.777
2022-04-14 | 3-4 | $1.680\times 10^{-2}$ | 519.111
2022-04-19 | 0-1 | $5.247\times 10^{-3}$ | 305.777
2022-04-19 | 1-2 | $7.718\times 10^{-2}$ | 334.222
2022-04-19 | 1-3 | $1.171\times 10^{-2}$ | 497.778
2022-04-19 | 3-4 | $1.707\times 10^{-2}$ | 519.111
2022-04-24 | 0-1 | $4.707\times 10^{-3}$ | 305.777
2022-04-24 | 1-2 | $7.186\times 10^{-2}$ | 334.222
2022-04-24 | 1-3 | $1.184\times 10^{-2}$ | 497.777
2022-04-24 | 3-4 | $1.905\times 10^{-2}$ | 519.111
Table S2: Calibration data for CNOT gates in ibmq_lima.

####  IV.1.2 ibmq_manila

Date | qubit |  $T_{1}$ ($\mu$s) |  $T_{2}$ ($\mu$s) | freq. (GHz) | Anharmonicity (GHz) | readout assignment error
---|---|---|---|---|---|---
2022-04-15 | 0 | 107.80 | 66.96 | 4.962 | $-0.34335$ | $2.70\times 10^{-2}$
2022-04-15 | 1 | 135.11 | 74.09 | 4.838 | $-0.34621$ | $6.37\times 10^{-2}$
2022-04-15 | 2 | 157.35 | 22.65 | 5.037 | $-0.34366$ | $4.16\times 10^{-2}$
2022-04-15 | 3 | 197.38 | 65.29 | 4.951 | $-0.34355$ | $2.50\times 10^{-2}$
2022-04-15 | 4 | 138.50 | 45.29 | 5.065 | $-0.34211$ | $2.97\times 10^{-2}$
2022-04-16 | 0 | 118.95 | 81.42 | 4.962 | $-0.34335$ | $3.0\times 10^{-2}$
2022-04-16 | 1 | 124.02 | 65.34 | 4.838 | $-0.34621$ | $3.41\times 10^{-2}$
2022-04-16 | 2 | 165.67 | 25.97 | 5.037 | $-0.34366$ | $3.37\times 10^{-2}$
2022-04-16 | 3 | 179.13 | 52.80 | 4.951 | $-0.34355$ | $2.24\times 10^{-2}$
2022-04-16 | 4 | 178.07 | 49.40 | 5.065 | $-0.34211$ | $3.72\times 10^{-2}$
2022-04-18 | 0 | 104.63 | 69.26 | 4.962 | $-0.34335$ | $2.91\times 10^{-2}$
2022-04-18 | 1 | 170.83 | 53.51 | 4.838 | $-0.34621$ | $4.80\times 10^{-2}$
2022-04-18 | 2 | 207.92 | 25.92 | 5.037 | $-0.34366$ | $2.79\times 10^{-2}$
2022-04-18 | 3 | 147.05 | 53.80 | 4.951 | $-0.34355$ | $2.17\times 10^{-2}$
2022-04-18 | 4 | 156.62 | 43.58 | 5.065 | $-0.34211$ | $3.07\times 10^{-2}$
2022-04-19 | 0 | 131.94 | 70.25 | 4.962 | $-0.34335$ | $3.38\times 10^{-2}$
2022-04-19 | 1 | 130.10 | 76.29 | 4.838 | $-0.34621$ | $2.92\times 10^{-2}$
2022-04-19 | 2 | 142.48 | 25.95 | 5.037 | $-0.34366$ | $2.44\times 10^{-2}$
2022-04-19 | 3 | 157.93 | 60.11 | 4.951 | $-0.34355$ | $1.90\times 10^{-2}$
2022-04-19 | 4 | 170.45 | 41.78 | 5.065 | $-0.34211$ | $2.59\times 10^{-2}$
2022-04-24 | 0 | 199.20 | 99.76 | 4.962 | $-0.34335$ | $3.2\times 10^{-2}$
2022-04-24 | 1 | 195.62 | 72.03 | 4.838 | $-0.34621$ | $3.18\times 10^{-2}$
2022-04-24 | 2 | 158.43 | 24.61 | 5.037 | $-0.34366$ | $4.64\times 10^{-2}$
2022-04-24 | 3 | 159.02 | 40.88 | 4.951 | $-0.34355$ | $3.23\times 10^{-2}$
2022-04-24 | 4 | 130.71 | 40.62 | 5.065 | $-0.34211$ | $2.29\times 10^{-2}$
2022-04-26 | 0 | 157.59 | 104.06 | 4.962 | $-0.34335$ | $3.55\times 10^{-2}$
2022-04-26 | 1 | 226.17 | 77.01 | 4.838 | $-0.34621$ | $2.82\times 10^{-2}$
2022-04-26 | 2 | 188.27 | 25.22 | 5.037 | $-0.34366$ | $2.66\times 10^{-2}$
2022-04-26 | 3 | 228.95 | 64.21 | 4.951 | $-0.34355$ | $2.28\times 10^{-2}$
2022-04-26 | 4 | 124.87 | 45.25 | 5.065 | $-0.34211$ | $2.97\times 10^{-2}$
Table S3: Calibration data for qubits in ibmq_manila. Date | connection | error rate | gate time (ns)
---|---|---|---
2022-04-15 | 0-1 | $8.959\times 10^{-3}$ | 277.333
2022-04-15 | 1-2 | $1.114\times 10^{-2}$ | 469.333
2022-04-15 | 2-3 | $7.623\times 10^{-3}$ | 355.556
2022-04-15 | 3-4 | $5.750\times 10^{-3}$ | 334.222
2022-04-16 | 0-1 | $6.807\times 10^{-3}$ | 277.333
2022-04-16 | 1-2 | $1.049\times 10^{-2}$ | 469.333
2022-04-16 | 2-3 | $6.371\times 10^{-3}$ | 355.556
2022-04-16 | 3-4 | $5.498\times 10^{-3}$ | 334.222
2022-04-18 | 0-1 | $9.176\times 10^{-3}$ | 277.333
2022-04-18 | 1-2 | $1.180\times 10^{-2}$ | 469.333
2022-04-18 | 2-3 | $7.778\times 10^{-3}$ | 355.556
2022-04-18 | 3-4 | $6.451\times 10^{-3}$ | 334.222
2022-04-19 | 0-1 | $7.651\times 10^{-3}$ | 277.333
2022-04-19 | 1-2 | $9.658\times 10^{-3}$ | 469.333
2022-04-19 | 2-3 | $6.794\times 10^{-3}$ | 355.556
2022-04-19 | 3-4 | $6.265\times 10^{-3}$ | 334.222
2022-04-24 | 0-1 | $5.223\times 10^{-3}$ | 277.333
2022-04-24 | 1-2 | $9.977\times 10^{-3}$ | 469.333
2022-04-24 | 2-3 | $9.831\times 10^{-3}$ | 355.556
2022-04-24 | 3-4 | $6.666\times 10^{-3}$ | 334.222
2022-04-26 | 0-1 | $6.681\times 10^{-3}$ | 277.333
2022-04-26 | 1-2 | $1.550\times 10^{-2}$ | 469.333
2022-04-26 | 2-3 | $6.702\times 10^{-3}$ | 355.556
2022-04-26 | 3-4 | $5.201\times 10^{-3}$ | 334.222
Table S4: Calibration data for CNOT gates in ibmq_manila.

####  IV.1.3 ibm_lagos

Date | qubit |  $T_{1}$ ($\mu$s) |  $T_{2}$ ($\mu$s) | freq. (GHz) | Anharmonicity (GHz) | readout assignment error
---|---|---|---|---|---|---
2022-04-12 | 0 | 110.77 | 46.41 | 5.235 | $-0.33987$ | $7.30\times 10^{-3}$
2022-04-12 | 1 | 130.34 | 84.81 | 5.100 | $-0.34325$ | $1.14\times 10^{-2}$
2022-04-12 | 2 | 123.18 | 112.70 | 5.188 | $-0.34193$ | $5.40\times 10^{-3}$
2022-04-12 | 3 | 207.35 | 143.35 | 4.987 | $-0.34529$ | $1.97\times 10^{-2}$
2022-04-12 | 4 | 161.92 | 46.82 | 5.285 | $-0.33923$ | $1.20\times 10^{-2}$
2022-04-12 | 5 | 133.28 | 88.03 | 5.176 | $-0.34079$ | $1.18\times 10^{-2}$
2022-04-12 | 6 | 137.78 | 178.13 | 5.064 | $-0.34276$ | $5.30\times 10^{-3}$
2022-04-19 | 0 | 131.57 | 49.25 | 5.235 | $-0.33987$ | $7.60\times 10^{-3}$
2022-04-19 | 1 | 138.38 | 116.49 | 5.100 | $-0.34325$ | $1.11\times 10^{-2}$
2022-04-19 | 2 | 112.36 | 142.46 | 5.188 | $-0.34193$ | $4.90\times 10^{-3}$
2022-04-19 | 3 | 105.22 | 93.65 | 4.987 | $-0.34529$ | $1.99\times 10^{-2}$
2022-04-19 | 4 | 107.02 | 43.79 | 5.285 | $-0.33923$ | $1.37\times 10^{-2}$
2022-04-19 | 5 | 129.64 | 83.89 | 5.176 | $-0.34079$ | $1.28\times 10^{-2}$
2022-04-19 | 6 | 180.21 | 109.66 | 5.064 | $-0.34276$ | $7.50\times 10^{-3}$
2022-04-24 | 0 | 187.41 | 49.35 | 5.235 | $-0.33987$ | $8.70\times 10^{-3}$
2022-04-24 | 1 | 117.38 | 118.40 | 5.100 | $-0.34325$ | $8.90\times 10^{-3}$
2022-04-24 | 2 | 64.03 | 119.95 | 5.188 | $-0.34193$ | $6.80\times 10^{-3}$
2022-04-24 | 3 | 145.37 | 140.26 | 4.987 | $-0.34529$ | $1.66\times 10^{-2}$
2022-04-24 | 4 | 167.96 | 51.35 | 5.285 | $-0.33923$ | $1.41\times 10^{-2}$
2022-04-24 | 5 | 91.06 | 77.16 | 5.176 | $-0.34079$ | $1.18\times 10^{-2}$
2022-04-24 | 6 | 127.67 | 146.32 | 5.064 | $-0.34276$ | $7.60\times 10^{-3}$
2022-09-28 | 0 | 122.09 | 43.23 | 5.235 | $-0.33987$ | $7.70\times 10^{-3}$
2022-09-28 | 1 | 122.50 | 79.85 | 5.100 | $-0.34325$ | $8.70\times 10^{-3}$
2022-09-28 | 2 | 80.70 | 143.44 | 5.188 | $-0.34193$ | $4.00\times 10^{-3}$
2022-09-28 | 3 | 179.06 | 96.01 | 4.987 | $-0.34529$ | $1.60\times 10^{-2}$
2022-09-28 | 4 | 70.89 | 19.19 | 5.285 | $-0.33923$ | $1.27\times 10^{-2}$
2022-09-28 | 5 | 125.56 | 80.26 | 5.176 | $-0.34079$ | $1.26\times 10^{-2}$
2022-09-28 | 6 | 135.57 | 108.60 | 5.064 | $-0.34276$ | $8.90\times 10^{-3}$
2022-10-02 | 0 | 122.09 | 49.23 | 5.235 | $-0.33987$ | $7.70\times 10^{-3}$
2022-10-02 | 1 | 122.5 | 79.85 | 5.100 | $-0.34325$ | $8.70\times 10^{-3}$
2022-10-02 | 2 | 80.70 | 143.33 | 5.188 | $-0.34193$ | $4.00\times 10^{-3}$
2022-10-02 | 3 | 179.06 | 96.01 | 4.987 | $-0.34529$ | $1.60\times 10^{-2}$
2022-10-02 | 4 | 70.89 | 19.19 | 5.285 | $-0.33923$ | $1.27\times 10^{-2}$
2022-10-02 | 5 | 125.55 | 80.26 | 5.176 | $-0.34079$ | $1.26\times 10^{-2}$
2022-10-02 | 6 | 135.57 | 108.60 | 5.064 | $-0.34276$ | $8.90\times 10^{-3}$
2022-10-08 | 0 | 84.09 | 30.77 | 5.235 | $-0.33987$ | $8.10\times 10^{-3}$
2022-10-08 | 1 | 128.29 | 106.63 | 5.100 | $-0.34325$ | $9.80\times 10^{-3}$
2022-10-08 | 2 | 163.44 | 149.43 | 5.188 | $-0.34193$ | $6.70\times 10^{-3}$
2022-10-08 | 3 | 141.19 | 88.05 | 4.987 | $-0.34529$ | $2.00\times 10^{-2}$
2022-10-08 | 4 | 14.20 | 14.23 | 5.285 | $-0.33923$ | $1.38\times 10^{-2}$
2022-10-08 | 5 | 119.10 | 89.16 | 5.176 | $-0.34079$ | $1.25\times 10^{-2}$
2022-10-08 | 6 | 131.13 | 133.12 | 5.064 | $-0.34276$ | $7.40\times 10^{-3}$
Table S5: Calibration data for qubits in ibm_lagos. Date | connection | error rate | gate time (ns)
---|---|---|---
2022-04-12 | 0-1 | $6.544\times 10^{-3}$ | 305.777
2022-04-12 | 1-2 | $6.707\times 10^{-3}$ | 327.111
2022-04-12 | 1-3 | $9.804\times 10^{-3}$ | 334.222
2022-04-12 | 3-5 | $1.061\times 10^{-2}$ | 334.222
2022-04-12 | 4-5 | $5.090\times 10^{-3}$ | 362.667
2022-04-12 | 5-6 | $3.599\times 10^{-3}$ | 256.000
2022-04-19 | 0-1 | $7.808\times 10^{-3}$ | 305.777
2022-04-19 | 1-2 | $5.743\times 10^{-3}$ | 327.111
2022-04-19 | 1-3 | $8.710\times 10^{-3}$ | 334.222
2022-04-19 | 3-5 | $1.513\times 10^{-2}$ | 334.222
2022-04-19 | 4-5 | $5.895\times 10^{-3}$ | 362.667
2022-04-19 | 5-6 | $4.501\times 10^{-3}$ | 256.000
2022-04-24 | 0-1 | $7.570\times 10^{-3}$ | 305.777
2022-04-24 | 1-2 | $5.387\times 10^{-3}$ | 327.111
2022-04-24 | 1-3 | $5.706\times 10^{-3}$ | 334.222
2022-04-24 | 3-5 | $8.753\times 10^{-3}$ | 334.222
2022-04-24 | 4-5 | $4.983\times 10^{-3}$ | 362.667
2022-04-24 | 5-6 | $5.735\times 10^{-3}$ | 256.000
2022-09-28 | 0-1 | $6.213\times 10^{-3}$ | 305.777
2022-09-28 | 1-2 | $5.485\times 10^{-3}$ | 327.111
2022-09-28 | 1-3 | $5.011\times 10^{-3}$ | 334.222
2022-09-28 | 3-5 | $9.463\times 10^{-3}$ | 334.222
2022-09-28 | 4-5 | $8.606\times 10^{-3}$ | 362.667
2022-09-28 | 5-6 | $6.566\times 10^{-3}$ | 256.000
2022-10-02 | 0-1 | $6.860\times 10^{-3}$ | 305.777
2022-10-02 | 1-2 | $5.756\times 10^{-3}$ | 327.111
2022-10-02 | 1-3 | $6.382\times 10^{-3}$ | 334.222
2022-10-02 | 3-5 | $1.243\times 10^{-2}$ | 334.222
2022-10-02 | 4-5 | $1.116\times 10^{-2}$ | 362.667
2022-10-02 | 5-6 | $6.629\times 10^{-3}$ | 256.000
2022-10-08 | 0-1 | $6.613\times 10^{-3}$ | 305.777
2022-10-08 | 1-2 | $4.230\times 10^{-3}$ | 327.111
2022-10-08 | 1-3 | $6.692\times 10^{-3}$ | 334.222
2022-10-08 | 3-5 | $9.568\times 10^{-3}$ | 334.222
2022-10-08 | 4-5 | $7.202\times 10^{-3}$ | 362.667
2022-10-08 | 5-6 | $5.620\times 10^{-3}$ | 256.000
Table S6: Calibration data for CNOT gates in ibm_lagos.

####  IV.1.4 ibm_nairobi

Date | qubit |  $T_{1}$ ($\mu$s) |  $T_{2}$ ($\mu$s) | freq. (GHz) | Anharmonicity (GHz) | readout assignment error
---|---|---|---|---|---|---
2022-09-28 | 0 | 160.60 | 36.77 | 5.260 | $-0.33983$ | $2.41\times 10^{-2}$
2022-09-28 | 1 | 159.05 | 84.96 | 5.170 | $-0.34058$ | $3.74\times 10^{-2}$
2022-09-28 | 2 | 106.63 | 81.60 | 5.274 | $-0.33890$ | $2.47\times 10^{-2}$
2022-09-28 | 3 | 162.92 | 33.89 | 5.027 | $-0.34253$ | $3.41\times 10^{-2}$
2022-09-28 | 4 | 92.94 | 69.85 | 5.177 | $-0.34059$ | $1.92\times 10^{-2}$
2022-09-28 | 5 | 156.20 | 19.75 | 5.293 | $-0.34053$ | $2.61\times 10^{-2}$
2022-09-28 | 6 | 130.98 | 60.91 | 5.129 | $-0.34044$ | $1.55\times 10^{-1}$
2022-10-01 | 0 | 142.69 | 35.23 | 5.260 | $-0.33983$ | $2.20\times 10^{-2}$
2022-10-01 | 1 | 135.93 | 101.47 | 5.170 | $-0.34058$ | $3.14\times 10^{-2}$
2022-10-01 | 2 | 92.71 | 101.33 | 5.274 | $-0.33890$ | $2.29\times 10^{-2}$
2022-10-01 | 3 | 150.06 | 71.35 | 5.027 | $-0.34253$ | $3.18\times 10^{-2}$
2022-10-01 | 4 | 126.71 | 74.76 | 5.177 | $-0.34059$ | $1.76\times 10^{-2}$
2022-10-01 | 5 | 152.14 | 20.97 | 5.293 | $-0.34053$ | $3.22\times 10^{-2}$
2022-10-01 | 6 | 165.52 | 106.79 | 5.129 | $-0.34044$ | $2.85\times 10^{-2}$
2022-10-07 | 0 | 151.12 | 30.06 | 5.260 | $-0.33983$ | $2.75\times 10^{-2}$
2022-10-07 | 1 | 112.45 | 94.07 | 5.170 | $-0.34058$ | $2.53\times 10^{-2}$
2022-10-07 | 2 | 132.64 | 205.77 | 5.274 | $-0.33890$ | $2.32\times 10^{-2}$
2022-10-07 | 3 | 166.22 | 69.41 | 5.027 | $-0.34253$ | $3.94\times 10^{-2}$
2022-10-07 | 4 | 77.26 | 75.22 | 5.177 | $-0.34059$ | $1.97\times 10^{-2}$
2022-10-07 | 5 | 121.90 | 21.08 | 5.293 | $-0.34053$ | $2.82\times 10^{-2}$
2022-10-07 | 6 | 152.98 | 96.51 | 5.129 | $-0.34044$ | $2.48\times 10^{-2}$
Table S7: Calibration data for qubits in ibm_nairobi. Date | connection | error rate | gate time (ns)
---|---|---|---
2022-09-28 | 0-1 | $1.040\times 10^{-2}$ | 248.889
2022-09-28 | 1-2 | $1.668\times 10^{-2}$ | 426.667
2022-09-28 | 1-3 | $1.233\times 10^{-2}$ | 270.222
2022-09-28 | 3-5 | $1.792\times 10^{-2}$ | 277.333
2022-09-28 | 4-5 | $5.119\times 10^{-3}$ | 312.889
2022-09-28 | 5-6 | $7.943\times 10^{-3}$ | 341.333
2022-10-01 | 0-1 | $1.153\times 10^{-2}$ | 248.889
2022-10-01 | 1-2 | $8.893\times 10^{-3}$ | 426.667
2022-10-01 | 1-3 | $9.813\times 10^{-3}$ | 270.222
2022-10-01 | 3-5 | $1.338\times 10^{-2}$ | 277.333
2022-10-01 | 4-5 | $4.588\times 10^{-3}$ | 312.889
2022-10-01 | 5-6 | $9.002\times 10^{-3}$ | 341.333
2022-10-07 | 0-1 | $9.965\times 10^{-3}$ | 248.889
2022-10-07 | 1-2 | $7.843\times 10^{-3}$ | 426.667
2022-10-07 | 1-3 | $8.187\times 10^{-3}$ | 270.222
2022-10-07 | 3-5 | $1.258\times 10^{-2}$ | 277.333
2022-10-07 | 4-5 | $6.003\times 10^{-3}$ | 312.889
2022-10-07 | 5-6 | $5.854\times 10^{-3}$ | 341.333
Table S8: Calibration data for CNOT gates in ibm_nairobi.

###  IV.2 Individual scores

As noted in the main text, several experimental runs were performed during
each period of data collection, resulting in at least three scores. Due to
device constraints or queue times, it was possible to obtain more than three
data points for certain devices on certain devices. In the event more than
three points were obtained, the lowest, largest, and median score were used
from the particular run for figures and results presented in the main text.
Below we collect the raw (uncorrected) scores, the scores in which basic error
mitigation was performed via inversion of the calibration matrix, and error-
mitigated scores resulting from a constrained optimization approach. If any
difference exists, the latter results in a lower score which was used as the
error-corrected score in the main text.

Date | device | uncorrected | optimization | matrix inversion
---|---|---|---|---
2022-04-15 | ibmq_lima | 5.844 | 7.631 | 7.681
2022-04-15 | ibmq_lima | 5.859 | 7.636 | 7.690
2022-04-15 | ibmq_lima | 5.888 | 7.651 | 7.698
2022-04-19 | ibmq_lima | 5.906 | 7.743 | 7.819
2022-04-19 | ibmq_lima | 5.876 | 7.703 | 7.770
2022-04-19 | ibmq_lima | 5.901 | 7.731 | 7.807
2022-04-24 | ibmq_lima | 5.860 | 7.651 | 7.717
2022-04-24 | ibmq_lima | 5.892 | 7.680 | 7.737
2022-04-24 | ibmq_lima | 5.901 | 7.697 | 7.752
2022-04-15 | ibmq_manila | 6.037 | 7.666 | 7.666
2022-04-15 | ibmq_manila | 6.051 | 7.683 | 7.683
2022-04-15 | ibmq_manila | 6.049 | 7.683 | 7.683
2022-04-15 | ibmq_manila | 6.263 | 7.817 | 7.817
2022-04-15 | ibmq_manila | 6.252 | 7.815 | 7.815
2022-04-15 | ibmq_manila | 6.256 | 7.824 | 7.824
2022-04-16 | ibmq_manila | 6.271 | 7.701 | 7.701
2022-04-16 | ibmq_manila | 6.257 | 7.689 | 7.689
2022-04-16 | ibmq_manila | 6.262 | 7.673 | 7.673
2022-04-18 | ibmq_manila | 6.200 | 7.735 | 7.735
2022-04-18 | ibmq_manila | 6.159 | 7.705 | 7.705
2022-04-18 | ibmq_manila | 6.200 | 7.741 | 7.741
2022-04-19 | ibmq_manila | 6.500 | 7.946 | 7.946
2022-04-19 | ibmq_manila | 6.490 | 7.929 | 7.929
2022-04-19 | ibmq_manila | 6.541 | 7.983 | 7.983
2022-04-24 | ibmq_manila | 6.390 | 7.761 | 7.761
2022-04-24 | ibmq_manila | 6.349 | 7.703 | 7.703
2022-04-24 | ibmq_manila | 6.369 | 7.745 | 7.745
2022-04-26 | ibmq_manila | 6.037 | 8.029 | 8.031
2022-04-26 | ibmq_manila | 6.096 | 8.116 | 8.117
2022-04-26 | ibmq_manila | 6.077 | 8.075 | 8.078
2022-04-12 | ibm_lagos | 7.178 | 7.984 | 7.984
2022-04-12 | ibm_lagos | 7.166 | 7.975 | 7.975
2022-04-12 | ibm_lagos | 7.172 | 7.977 | 7.977
2022-04-12 | ibm_lagos | 7.175 | 7.986 | 7.986
2022-04-12 | ibm_lagos | 7.156 | 7.963 | 7.963
2022-04-12 | ibm_lagos | 7.172 | 7.980 | 7.980
2022-04-12 | ibm_lagos | 7.166 | 7.970 | 7.970
2022-04-12 | ibm_lagos | 7.184 | 7.987 | 7.987
2022-04-12 | ibm_lagos | 7.145 | 7.955 | 7.955
2022-04-12 | ibm_lagos | 7.183 | 7.995 | 7.995
2022-04-19 | ibm_lagos | 7.221 | 8.037 | 8.037
2022-04-19 | ibm_lagos | 7.220 | 8.043 | 8.043
2022-04-19 | ibm_lagos | 7.225 | 8.050 | 8.050
2022-04-19 | ibm_lagos | 7.217 | 8.045 | 8.045
2022-04-19 | ibm_lagos | 7.202 | 8.030 | 8.030
2022-04-19 | ibm_lagos | 7.211 | 8.022 | 8.022
2022-04-19 | ibm_lagos | 7.228 | 8.057 | 8.057
2022-04-19 | ibm_lagos | 7.223 | 8.038 | 8.038
2022-04-19 | ibm_lagos | 7.221 | 8.036 | 8.036
2022-04-19 | ibm_lagos | 7.237 | 8.055 | 8.055
2022-04-24 | ibm_lagos | 7.454 | 8.140 | 8.140
Table S9: Individual scores.

Date | device | uncorrected | optimization | matrix inversion
---|---|---|---|---
2022-04-24 | ibm_lagos | 7.427 | 8.110 | 8.110
2022-04-24 | ibm_lagos | 7.427 | 8.116 | 8.116
2022-09-28 | ibm_lagos | 7.284 | 8.158 | 8.158
2022-09-28 | ibm_lagos | 7.267 | 8.142 | 8.142
2022-09-28 | ibm_lagos | 7.247 | 8.124 | 8.124
2022-09-28 | ibm_lagos | 7.292 | 8.179 | 8.179
2022-10-02 | ibm_lagos | 6.905 | 8.040 | 8.040
2022-10-02 | ibm_lagos | 6.907 | 8.046 | 8.046
2022-10-02 | ibm_lagos | 6.935 | 8.081 | 8.081
2022-10-02 | ibm_lagos | 6.968 | 8.117 | 8.117
2022-10-02 | ibm_lagos | 6.953 | 8.101 | 8.101
2022-10-02 | ibm_lagos | 6.912 | 8.057 | 8.057
2022-10-02 | ibm_lagos | 6.932 | 8.074 | 8.074
2022-10-02 | ibm_lagos | 6.934 | 8.080 | 8.080
2022-10-09 | ibm_lagos | 7.326 | 8.132 | 8.132
2022-10-09 | ibm_lagos | 7.312 | 8.110 | 8.110
2022-10-09 | ibm_lagos | 7.295 | 8.096 | 8.096
2022-10-09 | ibm_lagos | 7.290 | 8.087 | 8.087
2022-10-09 | ibm_lagos | 7.321 | 8.127 | 8.127
2022-10-09 | ibm_lagos | 7.337 | 8.145 | 8.145
2022-10-09 | ibm_lagos | 7.289 | 8.092 | 8.092
2022-10-09 | ibm_lagos | 7.310 | 8.105 | 8.105
2022-10-09 | ibm_lagos | 7.325 | 8.124 | 8.124
2022-10-09 | ibm_lagos | 7.325 | 8.136 | 8.136
2022-09-28 | ibm_nairobi | 6.549 | 8.516 | 8.533
2022-09-28 | ibm_nairobi | 6.545 | 8.484 | 8.501
2022-09-28 | ibm_nairobi | 6.547 | 8.497 | 8.509
2022-09-28 | ibm_nairobi | 6.528 | 8.450 | 8.473
2022-09-28 | ibm_nairobi | 6.546 | 8.450 | 8.487
2022-09-28 | ibm_nairobi | 6.530 | 8.451 | 8.476
2022-09-28 | ibm_nairobi | 6.556 | 8.430 | 8.444
2022-09-28 | ibm_nairobi | 6.554 | 8.428 | 8.439
2022-09-28 | ibm_nairobi | 6.550 | 8.420 | 8.434
2022-09-28 | ibm_nairobi | 6.558 | 8.433 | 8.447
2022-09-28 | ibm_nairobi | 6.540 | 8.425 | 8.442
2022-10-01 | ibm_nairobi | 6.652 | 8.281 | 8.283
2022-10-01 | ibm_nairobi | 6.701 | 8.338 | 8.340
2022-10-01 | ibm_nairobi | 6.675 | 8.305 | 8.305
2022-10-01 | ibm_nairobi | 6.682 | 8.322 | 8.323
2022-10-01 | ibm_nairobi | 6.666 | 8.287 | 8.291
2022-10-01 | ibm_nairobi | 6.746 | 8.376 | 8.379
2022-10-01 | ibm_nairobi | 6.748 | 8.373 | 8.374
2022-10-01 | ibm_nairobi | 6.729 | 8.325 | 8.325
2022-10-01 | ibm_nairobi | 6.733 | 8.347 | 8.350
2022-10-01 | ibm_nairobi | 6.742 | 8.353 | 8.353
2022-10-07 | ibm_nairobi | 6.656 | 8.384 | 8.393
2022-10-07 | ibm_nairobi | 6.692 | 8.418 | 8.423
2022-10-07 | ibm_nairobi | 6.685 | 8.409 | 8.416
2022-10-07 | ibm_nairobi | 6.674 | 8.404 | 8.410
2022-10-07 | ibm_nairobi | 6.693 | 8.419 | 8.423
2022-10-07 | ibm_nairobi | 6.687 | 8.408 | 8.412
2022-10-07 | ibm_nairobi | 6.695 | 8.417 | 8.424
2022-10-07 | ibm_nairobi | 6.692 | 8.432 | 8.439
Table S10: Individual scores.


# PointPCA+: Extending PointPCA objective quality assessment metric

###### Abstract

A computationally-simplified and descriptor-richer Point Cloud Quality
Assessment (PCQA) metric, namely PointPCA+, is proposed in this paper, which
is an extension of PointPCA. PointPCA proposed a set of perceptually-relevant
descriptors based on PCA decomposition that were applied to both the geometry
and texture data of point clouds for full reference PCQA. PointPCA+ employs
PCA only on the geometry data while enriching existing geometry and texture
descriptors, that are computed more efficiently. Similarly to PointPCA, a
total quality score is obtained through a learning-based fusion of individual
predictions from geometry and texture descriptors that capture local shape and
appearance properties, respectively. Before feature fusion, a feature
selection module is introduced to choose the most effective features from a
proposed super-set. Experimental results show that PointPCA+ achieves high
predictive performance against subjective ground truth scores obtained from
publicly available datasets. The code is available at https://github.com/cwi-
dis/pointpca_suite/.

###### Index Terms:

Point cloud, PCA, Feature selection, Objective quality assessment, Random
forest

##  I Introduction

Point cloud is prevailing among the available 3D imaging formats in recent
years. It is essentially a collection of points, where each point has
attributes of geometry, color, reflectance, etc. However, through acquisition,
compression, transmission, and rendering, the quality of a point cloud can be
degraded, which necessitates effective and efficient Point Cloud Quality
Assessment (PCQA) metrics. These metrics provide a guide on the design,
optimization, and parameter tuning of point cloud processing pipelines.

Objective PCQA metrics can be divided into point-based, projection-based, and
feature-based models. Point-based metrics such as point-to-point, point-to-
plane metrics and their variants measure degradations between the original and
distorted point clouds per point, mainly based on Euclidean or color space
distances [1]. Alexiou et al. propose the angular similarity of tangent planes
among corresponding points, which considers neighborhood information [2].
These metrics are computationally efficient but suffer from a crude
correspondence of matching between points.

Fig. 1: PointPCA+ architecture: both the reference and the distorted point
cloud are passing from every stage to compute a quality score. Operations in
the blue box are applied only to the geometry data of point clouds.

Projection-based approaches adapt existing Image Quality Assessment (IQA)
metrics to PCQA. Alexiou et al. [3] develop a framework for predicting the
quality by employing 2D IQA metrics on 6 orthographic projected views. Liu et
al. [4] provide a PCQA model based on the principle of information content
weighted structural similarity (IW-SSIM). However, the projection process and
the number of viewpoints have a non-negligible impact on the final prediction
accuracy; besides, how to combine the quality score on each viewpoint into a
score is also not straightforward.

Feature-based metrics consider perceptual loss from both geometry and texture
properties. Viola et al. [5] extract color statistics, histogram, and
correlogram to assess the level of impairment and combine the color-based
metrics with geometry-based metrics to form a global quality score. Alexiou et
al. [6] employ the local distributions of point clouds to predict perceptual
degradations from topology and color. Yang et al. [7] construct a local graph
centered at resampled key points for both reference and distorted point
clouds, with the color gradient on the local graph being used to measure
distortions. Meynet et al. [8] utilize an optimally-weighted linear
combination of curvature-based and color-based features to evaluate visual
quality. Diniz et al. [9] adopt the statistical information of the extracted
geometry/color features and feed them into a regression model. Deep learning-
based modules have also been used to extract perceptual features. Liu et al.
[10] design an end-to-end No Reference (NR) PCQA framework for estimating
subjective quality. A sparse convolutional neural network is applied in both
feature extraction and regression modules. An extension using coarse-to-fine
progressive knowledge transfer based on Human Vision System (HVS) is given in
[11]. Zhang et al. [12] make use of multi-modal information to address the
PCQA problem; the quality-aware encoder features are optimized with the
assistance of symmetric cross-modality attention. Interested readers may refer
to [1] for a more comprehensive review of the literature.

Point-based schemes may neglect the high-dimensional properties of point
clouds and the interplay among these dimensions, thereby limiting their
effectiveness. Projection-based methods often rely on 2D IQA, which may not
adequately capture the intrinsic characteristics of point clouds. Feature-
based schemes tend to have a high level of complexity, while the
interpretability of deep learning-based methods is a drawback with training
requiring a huge amount of data. In light of the above limitations and
inspired by PointPCA [13], we propose PointPCA+. Our differences are threefold
compared with PointPCA.

  * •

By performing PCA on the geometry data of the reference point cloud and
transforming both the reference and distorted point clouds onto the new basis,
we can capture differences in their shape properties effectively.

  * •

We utilize $knn$ algorithm to determine the neighborhood, which is faster and
returns a consistent number of points, therefore further decreasing the
computational cost of subsequent processing steps.

  * •

Recursive Feature Elimination (RFE) algorithm is employed to select the most
relevant and complementary features. This approach streamlines the quality
regression process.

##  II Proposed PointPCA+ Method

In Fig. 1, the PointPCA+ framework is illustrated, which is split into three
modules, namely, (a) pre-processing, (b) feature extraction, and (c) quality
regression which are introduced in the following subsections. Note that a Full
Reference (FR) metric typically uses either the pristine or the impaired
content as a reference, or both. In our design, only the pristine point cloud
serves as a reference.

###  II-A Pre-processing

To ensure coherent geometry and color information without redundancies, points
with identical coordinates that belong to the same point cloud are merged
[14]. The color of a merged point is obtained by averaging the color of
corresponding points sharing the same coordinates. For an FR PCQA metric,
identifying matches between reference and distorted point clouds is crucial
for comparing corresponding local properties. In our method, we use the $knn$
algorithm to identify neighborhood pairs between two point clouds. In
particular, for each point that belongs to a reference point cloud
$\mathcal{A}$, we find its $K$ nearest reference points, and its $K$ nearest
distorted points from the distorted point cloud $\mathcal{B}$, in terms of
Euclidean distance.

###  II-B Feature Extraction

To capture local perceptual quality degradations of a distorted point cloud,
we compute geometry and texture descriptors based on the identified
neighborhoods. Statistics based on these descriptors are subsequently
calculated and serve as predictors of visual quality. As mentioned earlier,
our method uses only the pristine point cloud as a reference to find the
matches in the distorted point cloud.

#### Geometry descriptors

Given a query point $\mathbf{p}_{i}$ of $\mathcal{A}$, the subscript $i$
denotes the point index, $1\leq i\leq|\mathcal{A}|$, and $|\mathcal{A}|$ is
the cardinality. The coordinates of $\mathbf{p}_{i}$’s $N$ nearest neighbors
in $\mathcal{F}$ are indicated as
$\bm{p}^{g,\mathcal{F}}_{n}=(x_{n},y_{n},z_{n})^{\rm T}$, with ${1\leq n\leq
N}$ and $\mathcal{F}\in\\{{\mathcal{A},\mathcal{B}}\\}$. The geometry of
$\mathbf{p}_{i}$ is denoted as $\bm{p}_{i}^{g,\mathcal{A}}$, and the geometry
of its closest neighbor in $\mathcal{B}$ is denoted as
$\bm{p}_{i}^{g,\mathcal{B}}$.

Initially, the covariance matrix $\bm{\Sigma}_{i}^{\mathcal{A}}$ is computed
as

$\bm{\Sigma}_{i}^{\mathcal{A}}=\frac{1}{N}\textstyle\sum_{n=1}^{N}\left(\bm{p}^{g,\mathcal{A}}_{n}-\bar{\bm{{p}}}^{g,\mathcal{A}}_{i}\right)\cdot\left(\bm{p}^{g,\mathcal{A}}_{n}-\bar{\bm{{p}}}^{g,\mathcal{A}}_{i}\right)^{\rm
T},$ (1)

where $\bar{\bm{p}}^{g,\mathcal{A}}_{i}$ indicates the centroid, given as

$\bm{\bar{{p}}}^{g,\mathcal{A}}_{i}=\frac{1}{N}\textstyle\sum_{n=1}^{N}\bm{p}^{g,\mathcal{A}}_{n}.$
(2)

Table I: Definition of descriptors.

| Descriptor | Definition | Distance
---|---|---|---
Geometric | Error vector | $\bm{e}=(\bm{\omega}_{i}^{\mathcal{B}}-\bm{\omega}_{i}^{\mathcal{A}})$ | $r_{\alpha}$
Error along axes | $\epsilon_{m}=(\bm{\omega}_{i}^{\mathcal{B}}-\bm{\omega}_{i}^{\mathcal{A}})^{T}\cdot\mathbf{u}_{m}$ | $r_{\beta}$
Error from origin | $\bm{\mathcal{\varepsilon}}=\bm{\omega}_{i}^{\mathcal{F}}$ |  $r_{\alpha}$,$r_{\beta}$
Mean | $\bm{\mu}^{\mathcal{B}}=\frac{1}{N}\textstyle\sum_{n}\bm{\omega}_{n}^{\mathcal{B}}$ |  $r_{\alpha}$,$r_{\beta}$
Variance | $\bm{\lambda}^{\mathcal{F}}=\frac{1}{N}\textstyle\sum_{n}{\left(\bm{\omega}_{n}^{\mathcal{F}}-\bm{\mu}^{\mathcal{F}}\right)}^{2}$ | $r_{\delta}$
Sum of variance | $\Sigma^{\mathcal{F}}=\textstyle\sum_{m}\lambda_{m}^{\mathcal{F}}$ | $r_{\delta}$
Covariance | $\bm{\Sigma}=\frac{1}{N}\textstyle\sum_{n}\left(\bm{\omega}_{n}^{\mathcal{A}}-\bm{\mu}^{\mathcal{A}}\right)\cdot\left(\bm{\omega}_{n}^{\mathcal{B}}-\bm{\mu}^{\mathcal{B}}\right)^{\rm T}$ | $r_{\gamma}$
Omnivariance | ${\rm O}^{\mathcal{F}}=\sqrt[3]{\textstyle\prod_{m}\lambda_{m}^{\mathcal{F}}}$ | $r_{\lambda}$
Eigenentropy | ${\rm E}^{\mathcal{F}}=-\textstyle\sum_{m}{\lambda}^{\mathcal{F}}_{m}\cdot\log{\lambda^{\mathcal{F}}_{m}}$ | $r_{\delta}$
Anisotropy | ${\rm A}^{\mathcal{F}}=({\lambda}^{\mathcal{F}}_{1}-{\lambda}^{\mathcal{F}}_{3})/{\lambda}^{\mathcal{F}}_{1}$ | $r_{\delta}$
Planarity | ${\rm P}^{\mathcal{F}}=({\lambda}^{\mathcal{F}}_{2}-{\lambda}^{\mathcal{F}}_{3})/{\lambda}^{\mathcal{F}}_{1}$ | $r_{\delta}$
Linearity | ${\rm L}^{\mathcal{F}}=({\lambda}^{\mathcal{F}}_{1}-{\lambda}^{\mathcal{F}}_{2})/{\lambda}^{\mathcal{F}}_{1}$ | $r_{\delta}$
Scattering | ${\rm S}^{\mathcal{F}}={\lambda}^{\mathcal{F}}_{3}/{\lambda}^{\mathcal{F}}_{1}$ | $r_{\delta}$
Change of curvature | ${\rm C}^{\mathcal{F}}={\lambda}^{\mathcal{F}}_{3}\big{/}\textstyle\sum_{m}{\lambda}^{\mathcal{F}}_{m}$ | $r_{\delta}$
Parallelity | $\mathscr{P}_{m}=1-\mathbf{u}_{m}\cdot{\mathbf{v}^{\mathcal{B}}_{m}}$ | $-$
Angular similarity | $\theta_{m}=1-\frac{2\cdot\arccos(\cos(\mathbf{u}_{m},\mathbf{v}^{\mathcal{B}}_{m}))}{\pi}$ | $-$
Textural | Mean | $\tilde{\bm{\mu}}^{\mathcal{F}}=\frac{1}{N}\textstyle\sum_{n}\bm{p}^{t,\mathcal{F}}_{n}$ | $r_{\delta}$
Variance | $\tilde{\bm{s}}^{\mathcal{F}}=\frac{1}{N}\textstyle\sum_{n}\left(\bm{p}^{t,\mathcal{F}}_{n}-\tilde{\bm{\mu}}^{\mathcal{F}}\right)^{2}$ | $r_{\delta}$
Sum of variance | $\tilde{\Sigma}^{\mathcal{F}}=\textstyle\sum_{m}\tilde{s}^{\mathcal{F}}_{m}$ | $r_{\delta}$
Covariance | $\tilde{\bm{\Sigma}}=\frac{1}{N}\textstyle\sum_{n}\left(\bm{p}_{n}^{t,\mathcal{A}}-\tilde{\bm{\mu}}^{\mathcal{A}}\right)\cdot\left(\bm{p}_{n}^{t,\mathcal{B}}-\tilde{\bm{\mu}}^{\mathcal{B}}\right)^{\rm T}$ | $r_{\gamma}$
Omnivariance | $\tilde{\rm{O}}^{\mathcal{F}}=\sqrt[3]{\textstyle\prod_{m}{\tilde{s}^{\mathcal{F}}_{m}}}$ | $r_{\delta}$
Entropy | $\tilde{\rm{H}}^{\mathcal{F}}=-\textstyle\sum_{m}\tilde{s}^{\mathcal{F}}_{m}\cdot\log{\tilde{s}^{\mathcal{F}}_{m}}$ | $r_{\delta}$

Then, eigen-decomposition is applied to $\bm{\Sigma}_{i}^{\mathcal{A}}$, to
obtain the eigenvectors which form an orthonormal basis
$\mathbf{V}^{\mathcal{A}}$ composed of eigenvectors
$\mathbf{v}_{m}^{\mathcal{A}}$, with corresponding eigenvalues
$\lambda_{m}^{\mathcal{A}}$, where $m={1,2,3}$, and
$\lambda_{1}^{\mathcal{A}}>\lambda_{2}^{\mathcal{A}}>\lambda_{3}^{\mathcal{A}}$.
Next, we map the reference and distorted neighborhoods to the new orthonormal
basis, denoted as
$\bm{\omega}_{n}^{\mathcal{F}}=(\bm{p}^{g,\mathcal{F}}_{n}-\bar{\bm{{p}}}^{g\,\mathcal{A}}_{i})\cdot\mathbf{V}^{\mathcal{A}}$.
Finally, we apply PCA to the covariance matrix of
$\bm{\omega}_{n}^{\mathcal{B}}$ and compute the eigenvectors
$\mathbf{v}_{m}^{\mathcal{B}}$ and eigenvalues $\lambda_{m}^{\mathcal{B}}$.
The mapped coordinates of the reference and distorted points
$\bm{\omega}_{n}^{\mathcal{F}}$, the eigenvectors
$\mathbf{v}_{m}^{\mathcal{F}}$ and the unit vectors $\mathbf{u}_{m}$, with
$\mathbf{u}_{1}=[1,0,0]^{\rm{T}}$, $\mathbf{u}_{2}=[0,1,0]^{\rm{T}}$ and
$\mathbf{u}_{3}=[0,0,1]^{\rm{T}}$, are used to construct the geometric
descriptors defined in Table I.

#### Texture descriptors

The color space is first converted from RGB to YCbCr [15]. This conversion is
motivated by the fact that the human eye is more sensitive to changes in
brightness than changes in color. We denote the texture information of
$\mathbf{p}_{i}$’s $N$ nearest neighbors in $\mathcal{F}$ as
$\bm{p}^{t,\mathcal{F}}_{n}=(Y_{n},Cb_{n},Cr_{n})^{\mathrm{T}}$. The proposed
6 texture descriptors are defined in Table I.

#### Explanation of descriptors

Each geometry descriptor represents an interpretable shape property inside the
neighborhood. Specifically, $\bm{e}$ denotes the error vector between the
mapped coordinates of the reference query point and its nearest neighbor, and
$\epsilon_{m}$ is the projected distance of the error vector across the $m$-th
axis. The $\bm{\mathcal{\varepsilon}}$ is used to capture the Euclidean and
projected distances of the mapped reference query point or its nearest
distorted neighbor from the centroid and principal axes, respectively.
$\bm{\mu}^{\mathcal{B}}$, $\bm{\lambda}^{\mathcal{F}}$, $\Sigma^{\mathcal{F}}$
and $\bm{\Sigma}$ reveal local statistics. $\rm{E}^{\mathcal{F}}$ provides an
estimation of the space uncertainty on the projected surfaces. Additionally,
$\mathscr{P}_{m}$ and $\theta_{m}$ assess the parallelity and the angular
dispersion of the distorted plane. The remaining geometry descriptors explore
the topology of a local region from different aspects, relying on the spatial
dispersion along different principal axes. $\tilde{\bm{\mu}}^{\mathcal{F}}$,
$\tilde{\bm{s}}^{\mathcal{F}}$ and $\tilde{\Sigma}^{\mathcal{F}}$ of the YCbCr
channel express the intrinsic distribution of luminance and chromatic
components. $\tilde{\bm{\Sigma}}$ and $\tilde{\rm{O}}^{\mathcal{F}}$ show the
variability of color information. $\tilde{\rm{\bm{H}}}^{\mathcal{F}}$ provides
an estimation of color uncertainty of the local region. Every descriptor is
computed per point $\mathbf{p}_{i}$.

Table II:  SROCC performance on M-PCCD, SJTU and WPC datasets

Metric | PointPCA+ | PointPCA[13] | PCQM[8] | pSSIM[6] | BitDance[16] | Plane2plane[2] | P2Plane_MSE[1] | P2P_MSE [1] | PSNR Y[1]
---|---|---|---|---|---|---|---|---|---
M-PCCD |  0.943$\pm$0.022 |  0.941$\pm$0.032 | 0.940$\pm$0.032 | 0.925$\pm$0.024 | 0.859$\pm$0.061 | 0.847$\pm$0.076 | 0.901$\pm$0.025 | 0.896$\pm$0.042 | 0.798$\pm$0.162
SJTU |  0.865$\pm$0.064 |  0.890$\pm$0.056 | 0.862$\pm$0.030 | 0.708$\pm$0.070 | 0.748$\pm$0.077 | 0.761$\pm$0.039 | 0.578$\pm$0.155 | 0.612$\pm$0.157 | 0.743$\pm$0.083
WPC |  0.857$\pm$0.040 |  0.866$\pm$0.036 | 0.749$\pm$0.036 | 0.465$\pm$0.059 | 0.451$\pm$0.054 | 0.454$\pm$0.069 | 0.452$\pm$0.065 | 0.563$\pm$0.071 | 0.614$\pm$0.061

#### Predictors

Predictors are defined as the error samples obtained by computing a distance
over a descriptor. We define different distance functions for different
descriptors. We use the Euclidean distance to measure the point-to-point
distances between query point pairs under the new basis

$r_{\alpha}=\sqrt{\textstyle\sum_{m}{\bm{d_{1}}}^{2}},$ (3)

where $\bm{d_{1}}$ is the difference between two points. We use the absolute
value to measure the point-to-plane distance, as

$r_{\beta}=|\bm{d_{2}}|,$ (4)

where $\bm{d_{2}}$ indicates the projected distance between a point and the
reference axes. We use the following definition of relative difference for the
covariance features

$r_{\gamma}=\frac{|\bm{q}^{\mathcal{A}}\odot\bm{q}^{\mathcal{B}}-\bm{Q}|}{\bm{q}^{\mathcal{A}}\odot\bm{q}^{\mathcal{B}}},$
(5)

where
$\\{\bm{q}^{\mathcal{F}}=\bm{\lambda}^{\mathcal{F}},\mathbf{Q}=\bm{\Sigma}\\}$
and
$\\{\bm{q}^{\mathcal{F}}=\tilde{\bm{s}}^{\mathcal{F}},\mathbf{Q}=\tilde{\bm{\Sigma}}\\}$,
for geometry and texture attributes, respectively, $\odot$ is for element-wise
product. We use the relative difference formula [6], for the remaining
descriptors

$r_{\delta}=\frac{|\phi^{\mathcal{A}}-\phi^{\mathcal{B}}|}{\left|\phi^{\mathcal{A}}\right|+\left|\phi^{\mathcal{B}}\right|+\varepsilon},$
(6)

where $\varepsilon$ is a small constant to avoid undefined operations.
Finally, the definitions of parallelity and angular similarity descriptors
incorporate a distance function. For notational purposes only, we define
distances $r_{\rho}$ and $r_{\theta}$ to be identical to the definitions of
$\mathscr{P}_{m}$ and $\theta_{m}$, respectively. Table I enlists distance
function(s) used per descriptor.

#### Features

Features are defined by pooling over predictor values. Specifically,

predictors $\psi_{i,j,k}$ are obtained per point $\mathbf{p}_{i}$, descriptor
$j$, and distance function $r_{k}$,
$k\in\\{\alpha,\beta,\gamma,\delta,\rho,\theta\\}$. This is done for all
descriptors $j$ in Table I, using the corresponding distances $r_{k}$. Through
pooling, we obtain a feature $f_{j,k}$ for every predictor:

$f_{j,k}=\frac{1}{|\mathcal{A}|}\textstyle\sum_{i=1}^{|\mathcal{A}|}\psi_{i,j,k}.$
(7)

###  II-C Quality Regression

To obtain a quality score that is well-aligned with the HVS, RFE is used to
select the most relevant predictor set among all the proposed predictors. RFE
[17] improves model accuracy, and efficiency, and reduces overfitting. We then
use the random forest algorithm to regress the selected predictors to the
final quality score.

##  III EXPERIMENTAL RESULTS

In this section, we report evaluation results of the proposed PointPCA+ metric
under three public datasets. Moreover, we report the performance achieved in
the ICIP 2023 Point Cloud Visual Quality Assessment (PCVQA) grand
challenge111https://sites.google.com/view/icip2023-pcvqa-grand-challenge.
Specifically, the challenge consists of 5 tracks, which correspond to
different use cases in which quality metrics are typically used. We
participated in Track$\\#$1 FR broad-range quality estimation, Track$\\#$3 FR
high-range quality estimation, and Track$\\#$5 Intra-reference quality
estimation.

###  III-A Setup

#### Datasets

Three publicly available datasets were recruited for performance evaluation,
namely, M-PCCD, SJTU, and WPC. The M-PCCD [18] consists of 8 point clouds
whose geometry and color are encoded using V-PCC and G-PCC variants, resulting
in 232 distorted stimuli. The SJTU [19] includes 9 reference point clouds with
each point cloud corrupted by seven types of distortions under six levels,
generating $378$ distorted stimuli. The WPC [4] contains 20 reference point
clouds with each point cloud degraded under five types of distortions and
different levels, leading to $740$ distorted stimuli. The Broad Quality
Assessment of Static Point Clouds (BASICS) [20] is used in the ICIP 2023 PCVQA
grand challenge, and comprises 75 point clouds from 3 different semantic
categories. Each point cloud is compressed with 4 different algorithms at
varying compression levels, resulting in 1494 processed point clouds.

#### Evaluation metrics

We evaluate the performance with four standard criteria: Pearson Linear
Correlation Coefficient (PLCC), Spearman Rank Order Correlation Coefficient
(SROCC), Difference/Similar Analysis quantified by Area Under the Curve (AUC),
and Better/Worse Analysis quantified by Correct Classification percentage (CC)
[21]. No function is adopted for score mapping. Finally, we report the Runtime
Complexity (RC).

#### Implementation details

We use RFE to select the best feature set among all the predictors, with the
best SROCC performance on the training set. In the inference stage, the
default configuration of scikit-learn (version 1.2.2) in Python is used.
Regarding the neighborhood size for the computation of descriptors, $K=81$ is
chosen considering complexity and performance, after experimenting with
$K\in{\\{9,25,49,81,121\\}}$.

###  III-B Overall performance comparison

#### Performance evaluation on M-PCCD, SJTU and WPC

We compare PointPCA+ with existing state-of-the-art FR quality metrics, the
results are shown in TABLE II. The best performance among these metrics is
highlighted in boldface, with the second best underlined. We used an 80%-20%
split for training and testing for each dataset. Then, the average and the
standard deviation of SROCC index computed across all testing splits of each
dataset, are reported. Specifically, for M-PCCD, SJTU, and WPC, we have 28,
36, and 4845 splits respectively. PCA-based metrics are competitive with the
highest PLCC/SROCC on the three datasets.

#### Performance evaluation on BASICS

We split BASICS into training-validation-test with 60%-20%-20% following the
rules from the PCVQA grand challenge [22]. Table III shows the official
evaluation results of Track$\\#$1\. The PointPCA+ secured a third-place
ranking in Track$\\#$3 and Track$\\#$5, with {PLCC, SROCC, AUC, CC}$=$
{$0.479,0.603,0.625,0.886$} and {AUC, CC} $=$ {$0.811,0.938$}, respectively.

Table III: Top 4 performance comparison on the official PCVQA grand challenge
test set, evaluated by the challenge organizers. Best in bold and second best
underlined.

Submission | PLCC | SROCC | AUC | CC | RC(s)
---|---|---|---|---|---
KDDIUSCJoint | 0.917 | 0.875 | 0.888 | 0.970 | 42.8
CWI_DIS | 0.909 | 0.874 | 0.871 | 0.961 | 1000
SJTU MMLAB | 0.896 | 0.871 | 0.832 | 0.955 | 8.60
SlowHand | 0.825 | 0.791 | 0.805 | 0.924 | 130.47

##  IV Conclusion

This paper proposes a PCA-based PCQA metric, namely PointPCA+, which relies on
an enriched set of lower complexity descriptors with respect to its PointPCA
predecessor. After a pre-processing step, features are extracted from both
geometric and textural domains. A subset of features is selected to enhance
the stability of the model, and a learning-based feature fusion based on
ensemble learning is applied to the feature subset, to provide a total quality
score for a distorted point cloud. Our experimental results demonstrate that
PointPCA+ outperforms the majority of existing PCQA metrics, reaching second
place in Track$\\#1$ of the ICIP 2023 PCVQA grand challenge. Future work will
focus on further reducing computational complexity and incorporating global
descriptors to more effectively handle general distortion.

##  V Acknowledgements

This work was supported through the NWO WISE grant and the European Commission
Horizon Europe program, under the grant agreement 101070109, TRANSMIXR
https://transmixr.eu/. Funded by the European Union.

## References

  * [1] Evangelos Alexiou, Yana Nehmé, Emin Zerman, Irene Viola, Guillaume Lavoué, Ali Ak, Aljosa Smolic, Patrick Le Callet, and Pablo Cesar,  “Chapter 18 - subjective and objective quality assessment for volumetric video,”  in Immersive Video Technologies, Giuseppe Valenzise, Martin Alain, Emin Zerman, and Cagri Ozcinar, Eds., pp. 501–552. Academic Press, 2023.
  * [2] Evangelos Alexiou and Touradj Ebrahimi,  “Point cloud quality assessment metric based on angular similarity,”  in 2018 IEEE International Conference on Multimedia and Expo (ICME), 2018, pp. 1–6.
  * [3] Evangelos Alexiou and Touradj Ebrahimi,  “Exploiting user interactivity in quality assessment of point cloud imaging,”  in 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), 2019, pp. 1–6.
  * [4] Qi Liu, Honglei Su, Zhengfang Duanmu, Wentao Liu, and Zhou Wang,  “Perceptual quality assessment of colored 3d point clouds,”  IEEE Transactions on Visualization and Computer Graphics, pp. 1–1, 2022.
  * [5] Irene Viola, Shishir Subramanyam, and Pablo Cesar,  “A color-based objective quality metric for point cloud contents,”  in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1–6.
  * [6] Evangelos Alexiou and Touradj Ebrahimi,  “Towards a point cloud structural similarity metric,”  in 2020 IEEE International Conference on Multimedia Expo Workshops (ICMEW), 2020, pp. 1–6.
  * [7] Qi Yang, Zhan Ma, Yiling Xu, Zhu Li, and Jun Sun,  “Inferring point cloud quality via graph similarity,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 6, pp. 3015–3029, 2022.
  * [8] Gabriel Meynet, Yana Nehmé, Julie Digne, and Guillaume Lavoué,  “PCQM: A full-reference quality metric for colored 3d point clouds,”  in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1–6.
  * [9] Rafael Diniz, Pedro Garcia Freitas, and Mylene CQ Farias,  “Point cloud quality assessment based on geometry-aware texture descriptors,”  Computers & Graphics, 2022.
  * [10] Yipeng Liu, Qi Yang, Yiling Xu, and Le Yang,  “Point cloud quality assessment: Dataset construction and learning-based no-reference metric,”  ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 2s, pp. 1–26, 2023.
  * [11] Qi Liu, Yiyun Liu, Honglei Su, Hui Yuan, and Raouf Hamzaoui,  “Progressive knowledge transfer based on human visual perception mechanism for perceptual quality assessment of point clouds,”  arXiv preprint arXiv:2211.16646, 2022.
  * [12] Zicheng Zhang, Wei Sun, Xiongkuo Min, Quan Zhou, Jun He, Qiyuan Wang, and Guangtao Zhai,  “MM-PCQA: Multi-modal learning for no-reference point cloud quality assessment,”  arXiv preprint arXiv:2209.00244, 2022.
  * [13] Evangelos Alexiou, Xuemei Zhou, Irene Viola, and Pablo Cesar,  “Pointpca: Point cloud objective quality assessment using pca-based descriptors,”  arXiv preprint arXiv:2111.12663, 2021.
  * [14] Dong Tian, Hideaki Ochimizu, Chen Feng, Robert Cohen, and Anthony Vetro,  “Geometric distortion metrics for point cloud compression,”  in 2017 IEEE International Conference on Image Processing (ICIP), 2017, pp. 3460–3464.
  * [15] ITU-R BT.709-6,  “Parameter values for the HDTV standards for production and international programme exchange,” International Telecommunication Unionn, Jun. 2015.
  * [16] Rafael Diniz, Pedro Garcia Freitas, and Mylène C. Q. Farias,  “Color and geometry texture descriptors for point-cloud quality assessment,”  IEEE Signal Processing Letters, vol. 28, pp. 1150–1154, 2021.
  * [17] Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik,  “Gene selection for cancer classification using support vector machines,”  Machine learning, vol. 46, pp. 389–422, 2002.
  * [18] Evangelos Alexiou, Irene Viola, Tomás M Borges, Tiago A Fonseca, Ricardo L De Queiroz, and Touradj Ebrahimi,  “A comprehensive study of the rate-distortion performance in mpeg point cloud compression,”  APSIPA Transactions on Signal and Information Processing, vol. 8, pp. e27, 2019.
  * [19] Qi Yang, Hao Chen, Zhan Ma, Yiling Xu, Rongjun Tang, and Jun Sun,  “Predicting the perceptual quality of point cloud: A 3d-to-2d projection-based exploration,”  IEEE Transactions on Multimedia, vol. 23, pp. 3877–3891, 2020.
  * [20] Ali Ak, Emin Zerman, Maurice Quach, Aladine Chetouani, Aljosa Smolic, Giuseppe Valenzise, and Patrick Le Callet,  “Basics: Broad quality assessment of static point clouds in compression scenarios,”  arXiv preprint arXiv:2302.04796, 2023.
  * [21] Lukáš Krasula, Karel Fliegel, Patrick Le Callet, and Miloš Klíma,  “On the accuracy of objective image and video quality models: New methodology for performance evaluation,”  in 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), 2016, pp. 1–6.
  * [22] Aladine Chetouani, Giuseppe Valenzise, Ali Ak, Emin Zerman, Maurice Quach, Marouane Tliba, Mohamed Amine Kerkouri, and Patrick Le Callet,  “ICIP 2023 - point cloud visual quality assessment grand challenge,”  2023\.