# A Comprehensive Study of Bug Fixes in Quantum Programs

Junjie Luo *This work was carried out when Junjie Luo was at Kyushu
University. Kyushu University, Japan  Pengzhan Zhao  Kyushu University, Japan
Zhongtao Miao  Kyushu University, Japan  Shuhan Lan  Kyushu University, Japan
Jianjun Zhao  Kyushu University, Japan

###### Abstract

As quantum programming evolves, more and more quantum programming languages
are being developed. As a result, debugging and testing quantum programs have
become increasingly important. While bug fixing in classical programs has come
a long way, there is a lack of research in quantum programs. To this end, this
paper presents a comprehensive study on bug fixing in quantum programs. We
collect and investigate 96 real-world bugs and their fixes from four popular
quantum programming languages (Qiskit, Cirq, Q#, and ProjectQ). Our study
shows that a high proportion of bugs in quantum programs are quantum-specific
bugs (over 80%), which requires further research in the bug fixing domain. We
also summarize and extend the bug patterns in quantum programs and subdivide
the most critical part, math-related bugs, to make it more applicable to the
study of quantum programs. Our findings summarize the characteristics of bugs
in quantum programs and provide a basis for studying testing and debugging
quantum programs.

###### Index Terms:

Bug fixing, quantum software testing, quantum program debugging, empirical
study

##  I Introduction

Debugging and testing are critical parts of an integrated software development
method in modern software development. A software bug is regarded as the
abnormal program behaviors which deviate from its specification [1], including
poor performance when a threshold level of performance is included as part of
the specification. Software defects significantly impact the economy,
security, and quality of life, yet relying on manual diagnosis and repair of
software bugs consumes significant time and money. To address this problem,
many software engineering tasks, such as program analysis, debugging, and
software testing, are devoted to developing techniques and tools for locating
and fixing bugs. In addition, software bugs can be handled or prevented more
effectively by studying past bugs and their fixes.

Quantum programming is the process of designing and constructing executable
quantum programs to achieve a specific computational result. Recently, many
quantum programming languages have been developed for the development of
quantum software, for instance, Qiskit [2], Cirq [3], Q# [4], and ProjectQ
[5]. The current research in quantum programming focuses mainly on problem
analysis, language design, and implementation. However, despite their
importance, program debugging and software testing have received relatively
little attention in the quantum programming paradigm. The specific features
introduced in quantum programming, such as superposition, entanglement, and
no-cloning, make them different for bug fixing compared to classical programs.
Although several approaches have been proposed for debugging and testing
quantum software [6, 7, 8, 9, 10] recently, no research work has been focused
on bug fixing in quantum programs [11].

This paper first proposes five questions that need analysis and discussion. On
this basis, we collect and investigate 96 real-world bugs and their fixes from
four popular quantum programming languages (Qiskit, Cirq, Q#, and ProjectQ).
Finally, we obtained nine findings to answer the questions. Our main findings
can be summarized as follows:

  * •

More than 80% of the bugs we collected is quantum-specific, which means that
classical bug fixing methods are insufficient, and bugs in quantum programs
need further research.

  * •

Although the current bugs in quantum programs are generally simple, the high
complexity bugs are all quantum-specific bugs, which means bug fixing for
quantum programs is challenging.

  * •

We summarize and expand the bug patterns in quantum programs and count the
number of bugs for each bug pattern. Math-related bug patterns are the focus
of our analysis, which are uncommon in classical programs.

We believe that the results of our study can benefit future research in two
aspects:

  1. 1.

It can provide insights into how to develop and improve the bug fixing methods
for existing quantum programs. For example, what knowledge is needed to repair
quantum programs and whether this knowledge is included in classical program
fixing methods. In addition, the results also show the potential regarding the
application of existing automatic fixing methods in quantum programs.

  2. 2.

It can provide a more precise direction for studying quantum programs bugs.
For example, the results show some bug patterns in quantum programs and
analyze the frequency of each bug pattern. Based on these bug patterns that we
have summarized, our future work can propose new methods for locating quantum-
specific bugs, as well as developing static analysis tools.

The rest of the paper is organized as follows. Section II briefly introduces
the background knowledge of quantum programming. Section III describes the
research methodology of our comprehensive study. Section IV discusses some
findings of our study. Related work is discussed in Section V, and concluding
remarks are given in Section VI.

##  II Background

This section briefly introduces some basic concepts on quantum computing as
well as quantum programming frameworks.

###  II-A Basic Concepts

A quantum bit (qubit) is the analog of one classical bit but has many
different properties. A classical bit, like a coin, has only two states, 0 and
1, while a qubit can be in a continuum of states between $\Ket{0}$ and
$\Ket{1}$ in which the $\Ket{}$ notation is called Dirac notation. We can
represent a qubit mathematically as $\Ket{\psi}=\alpha\Ket{0}+\beta\Ket{1}$
where $|\alpha|^{2}+|\beta|^{2}=1$ and the numbers $\alpha$ and $\beta$ are
complex numbers. The states $\Ket{0}$ and $\Ket{1}$ are called computational
basis states. Unlike classical bits, we cannot examine a qubit directly to get
the values of $\alpha$ and $\beta$. Instead, we measure a qubit to obtain
either the result 0 with probability $|\alpha|^{2}$ or the result 1 with
probability $|\beta|^{2}$.

Quantum gates are used to do the quantum computation, which means manipulating
quantum information. Some basic quantum gates are as follows:

  * •

Quantum NOT gate takes the state $\Ket{\psi}=\alpha\Ket{0}+\beta\Ket{1}$ into
the state $\Ket{\psi}=\alpha\Ket{1}+\beta\Ket{0}$. We can use a matrix to
represent this operation:

$\displaystyle X=\begin{bmatrix}0&1\\\ 1&0\end{bmatrix}$

  * •

The Z gate can be expressed as

$\displaystyle Z=\begin{bmatrix}1&0\\\ 0&-1\end{bmatrix}$

From the matrix, we know the Z gate leaves the $\Ket{0}$ unchanged and changes
the sign of $\Ket{1}$.

  * •

The Hadamard gate turns the $\Ket{0}$ into $(\Ket{0}+\Ket{1})/\sqrt{2}$ and
turns the $\Ket{1}$ into $(\Ket{0}-\Ket{1})/\sqrt{2}$. The matrix form of the
Hadamard gate is

$\displaystyle H=\frac{1}{\sqrt{2}}\begin{bmatrix}1&1\\\ 1&-1\end{bmatrix}$

All the matrices are unitary ones. Besides these single-qubit gates, there are
multiple qubit gates, such as the Controlled-NOT gate (CNOT gate). This gate
has two input qubits, the control qubit and the target qubit. If the control
qubit is 0, then the target qubit remains unchanged. If the control qubit is
1, then the target qubit is flipped. We can express the behavior of the CNOT
gate as $\Ket{A,B}\rightarrow\Ket{A,B\oplus A}$.

Quantum circuits are models of all kinds of quantum processes. We can build
quantum circuits with quantum gates and use wires to connect the components in
quantum circuits. These wires can represent the passage of time or a physical
particle moving from one position to another. Another essential operation in
quantum circuits is measurement. Measurement operation observes a single qubit
and obtains a classic bit with a certain probability. Nielsen’s book[12] has a
more detailed explanation of quantum computation.

###  II-B Quantum Programming Frameworks

Recently, several open-source programming frameworks for supporting quantum
programming have been proposed, such as Qiskit, Cirq, Q#, and ProjectQ. We can
use these frameworks to develop quantum software. Here is an example of Qiskit
programs111https://github.com/Qiskit/qiskit-
terra/blob/main/examples/python/ibmq/hello_quantum.py:

    1    from qiskit import QuantumCircuit
    2    from qiskit import execute, IBMQ, BasicAer
    3    qc = QuantumCircuit(2, 2)
    4    qc.h(0)
    5    qc.cx(0, 1)
    6    qc.measure([0, 1], [0, 1])
    7    sim = BasicAer.get˙backend("qasm_simulator")
    8    job_sim = execute(qc, sim)
    9    result_sim = job_sim.result()
    10   print(result_sim.get˙counts(qc))

Figure 1: A Qiskit Program.

This program aims to create a Bell state. In lines 1-2, it imports related
packages. Line 3 creates a quantum circuit acting on one quantum register of
two quantum bits (qubits) and one classical register of two classical bits.
Line 4 adds an H gate on qubit 0, putting this qubit in superposition. Line 5
adds a CX (CNOT) gate on control qubit 0 and target qubit 1, putting the
qubits in a Bell state. Line 6 measures the qubits 0 and 1 of the quantum
register, respectively, and puts the values into the corresponding classical
bits 0 and 1 of the classical register. Line 8 executes the circuit on the
QASM simulator.

##  III Methodology

###  III-A Dataset

The data collection and analysis workflow is divided into the following steps:
determining target bugs, collecting and filtering bugs, and analyzing bugs.
Figure 2 depicts our data collection and analysis process.

Figure 2: The process of collecting and analyzing bugs

####  III-A1 Determining target bugs

In order to collect as many bugs as possible, we collected bugs generated in
the most commonly used quantum programming languages, including Qiskit, Cirq,
Q#, and ProjectQ. We collected bugs raised on GitHub, Stack Overflow, and
Stack Exchange in terms of platforms for reporting bugs.

####  III-A2 Collecting and filtering bugs

We manually collect bugs for each target language on different platforms since
the number of bugs in quantum languages is not very large, and we need to
filter the bugs to find those that meet our requirements. There are three
filter conditions for bugs as follows:

  * •

Related to quantum programs. We only collect bugs related to quantum programs.
If a bug is in a program that does not involve any quantum computation or
operation, then the bug will not be included in the collection. It is
important to note that although we require the bug to occur in a quantum
program, this does not mean the bug itself is quantum-related. Instead, it may
be a bug that would also occur in classical programming languages, which we
will specifically discuss later.

  * •

Written by developers. We will only work on analyzing bugs written by
developers and will not discuss bugs that occur in the quantum programming
language itself. This will have little impact on the bug collection on Stack
Overflow and Stack Exchange since the issues on these platforms are raised by
developers. However, on GitHub, only a few issues are related to developers.
The vast majority of issues are specific to the quantum language itself.

  * •

Valid pre- and post-fixing codes. To facilitate analysis, we require that the
collected bugs have pre- and post-fixing code. Also, we need to confirm that
the changes are effective, depending on the feedback from the user who raised
the bug.

####  III-A3 Analyzing bugs

We analyze all the valid data collected. We count the distribution of bugs for
different research questions, focus on representative examples, and finally
give our findings.

TABLE I:  Repositories considered in the study with bugs in quantum programs Repository | Total | Fix
---|---|---
Qiskit | 138 | 71
Cirq | 25 | 16
Q# | 21 | 7
ProjectQ | 3 | 2
Total | 187 | 96

###  III-B Research Question

Our study is driven by the following research questions:

####  III-B1 RQ1

Does this bug only occur in quantum programs? Most quantum languages are not
completely new. for example, the major versions of Cirq and Qiskit use the
Python programming language, while Q# is based on C# and F# extensions. In
such cases, bugs in the fixes may come from classical programming languages or
come specifically from quantum languages. Studying this issue helps us to
better understand to what extent the repair of quantum programs can benefit
from the existing knowledge of classical software repair. Our study
investigates the proportion of classical bugs and quantum-specific bugs in
quantum programs.

####  III-B2 RQ2

Where is the location of the bug in the programs? Similar to classical
programs, to fix bugs in quantum programs, we first need to investigate the
distribution of bugs. Zhong and Su [13] show that bugs in a program can occur
outside the source file, so knowing where the bugs are located can help us to
find the focus of bug fixes. They also indicate that more than one line of
code may need to be modified to fix a bug. The existence of dependencies in
the fixed code affects the difficulty of fixing the code. Our study analyzes
the distribution of bugs under real quantum programs. The results of the study
help locate bugs in quantum programs.

####  III-B3 RQ3

How complex to fix bugs in quantum programs? In addition to analyzing the
location of bugs, it is also essential to understand the complexity of bugs to
fix bugs in quantum programs automatically. When there is more than one bug in
a program, dependencies between bugs will impact the difficulty of fixing the
bug. For bugs with dependencies, breaking the dependencies during the fixing
process will lead to the failure of the bug fixing. In our study, we analyze
the dependencies of bug codes in quantum programs following [13], and describe
the complexity in four levels.

####  III-B4 RQ4

Are there bug patterns in the quantum program? We found that some bugs
overlapped and repeated by reproducing them in the corresponding quantum
programs. Intuitively, we infer that they can be distributed over some
universal patterns or regularities, and this information would be helpful to
guidance for fixing quantum-specific bugs. Based on this, we performed a
thorough investigation of these bugs and classified them by several
principles.

##  IV Empirical Results

###  IV-A RQ1: Classical or Quantum-Specific Bugs

This research question investigates the proportion of classical bugs and
quantum-specific bugs in quantum software. Generally, quantum-specific bugs
are different from classical software bugs as repairing them requires the
domain knowledge of quantum-related concepts, properties, computational
formulas, and quantum programming languages. Table II shows the statistics of
the number of classical and quantum-specific bugs in the data we collected.
From the results of the investigation, we can derive the following findings:

TABLE II: Number of classical and quantum specific bugs Type | Count
---|---
Classical | 17
Quantum | 79
Total | 96

Finding 1. Clearly, the number of quantum-specific bugs is much greater than
the number of classical bugs (more than 80%). Since most of the existing bug
fixing approaches focus on classical programs, it will be a great challenge to
deal with the bugs in quantum programs. It also means that researching bug
fixing in quantum programs is highly worthwhile.

###  IV-B RQ2: Bug Distribution

We investigated whether files other than the source files were modified during
the fixing of the quantum program. The results of the investigation lead to
the following findings:

Finding 2. In all the bugs we collected, only the source files have been
modified. We believe there are two reasons for this situation: the first is
that the difficulty of simulating large quantum programs by classical
computers has led to the fact that quantum programs, in general, are limited
to simpler functions and therefore do not need other files to constitute the
project. Secondly, we can find that programmers wrote a significant part of
the programs in our data collection exploring the quantum language. For
example, some programs contain only attempts on how to add the QFT circuit to
the code. This type of quantum program would have only source files.

In addition, we counted the number of lines of code that needed to be modified
to fix each bug in the quantum program. We define the action of adding,
deleting, and updating a line of code as a single modification. Figure
LABEL:fig:modifiedline shows the statistics of the number of lines needed to
fix classical bugs and quantum-specific bugs in quantum programs,
respectively. The stacked bars show the number of bugs, red for quantum-
specific bugs while blue for classical bugs. Furthermore, the horizontal
coordinates in the chart indicate the number of lines that need to be
modified.

Finding 3. Overall, more than 70% of the bugs in our collection of quantum
programs can be fixed by modifying only one line of code, while less than 10%
of the code requires modifying more than six lines. This implies that the cost
of fixing bugs is generally low in existing quantum programs.

Finding 4. From the type of bugs, we can find that the number of lines to be
modified for classical bugs appearing in quantum programs is generally low (at
most, two lines of code need to be modified). In Zhong and Su’s work[13], less
than 30% of bugs in classical programs can be fixed by only one repair action.
We can see that it is significantly less challenging to fix classical bugs in
quantum programs than in classical programs. So more attention should be
focused on how to fix quantum-specific bugs in quantum programs.

###  IV-C RQ3: Bug Complexity

To explore the difficulty of fixing bugs in quantum programs, we need a more
detailed definition of the complexity of bugs in quantum programs. As one of
the well-established methods, Paltenghi and Pradel [14] determine the
complexity of bug fixing in terms of the number of lines of code that are in
error. While such an approach can quickly determine the occurrence of bugs
that some automatic fixes cannot handle (e.g., bugs that require fixing more
than 20 lines), it is difficult to distinguish the difficulty of fixing bugs
with fewer lines (e.g., whether there are dependencies between the data being
fixed). Here, we refer to the four levels of fault complexity proposed by
Zhong and Su [13]: a single repair action (C1), non-data dependent repair
actions (C2), data-dependent repair actions (C3), and mixture repair actions
(C4). We consider the modification of a line as a repair action and thus apply
this complexity classification to our work. Although this complexity
classification was used in the paper for classical programs, we argue that
this approach can also be applied to quantum programs in our study. Figure
LABEL:fig:Complexity shows the results of our analysis, where we represent the
classical bugs and quantum-specific bugs in each complexity with blue and red
bars, respectively.

Finding 5. Since single line fixes are directly classified into C1 complexity,
the proportion of C1 complexity is much higher than the other three complexity
levels in quantum programs. This reflects that the complexity of bugs in
current quantum programs is generally low, but there are still some bugs that
are difficult to fix (19 bugs of C3 and C4 complexity in total), and all the
bugs in this part belong to quantum specific bugs, and thus the difficulty in
bug fixing cannot be ignored.

Finding 6. In our study, the cause of the errors in 5 of the 19 samples with
complexity C3 and C4 is due to an incorrect implementation method. This error
is usually because the programmer does not know the exact implementation of
the required functionality, so more code manipulation is usually required to
fix it. Moreover, this type of error is difficult for bug-fix automation
approaches to work around because we cannot expect them to anticipate the
functionality that the programmer needs to implement to make the correct fix.
Furthermore, this means that in the development of existing quantum programs,
there are fewer cases of C3 and C4 complexity errors if the programmer is
known about the exact implementation of the desired function.

###  IV-D RQ4: Bug Patterns

Our study also analyzed and counted the bug patterns of the 96 valid samples
collected. We refer to the bug pattern classification used in the work of
Paltenghi and Pradel [14]. Although their work is specific to quantum
computing platforms, we believe that some of the bug patterns can also be used
to study quantum program bugs. In addition, we have defined some new bug
patterns for quantum programs to complement and improve the shortcomings of
their bug patterns when applied to quantum programs. Our study divided the bug
pattern into four major categories: API-related, incorrect application logic,
math-related, and others, and also detailed several subcategories under each
major category. To adapt it to the analysis of bugs in quantum programs, we
further subdivide incorrect numerical computation under the math-related
category by adding bug patterns specific to quantum programs, such as an
incorrect gate. Table III shows the details of each bug pattern classification
in our study and lists the number of bug samples under each bug pattern. The
following discussion will briefly describe each bug pattern and detail our
proposed additional quantum program-specific bug patterns.

TABLE III:  Bug patterns in the investigated data

Bug pattern |  |  | Count
---|---|---|---
API-related | API misuse |  | 30
| Outdated API client |  | 5
| Incorrect
---
application
logic
Intermediate representation | Missing information | 4
| Wrong information | 6
|  Refer to wrong program element | Wrong concept | 2
| Wrong identifier | 1
| Incorrect scheduling |  | 1
| Wrong declaration |  | 1
| Wrong way of implementation | 7
| Qubit-related | Incorrect qubit order | 4
|  | Incorrect qubit count | 1
Math-related |  Incorrect numerical computation | Incorrect gate | 8
|  | Incorrect matrix
---
computation
9
| Incorrect initial state | 2
| Incorrect measurement | 3
|  | Potential all-zeros
---
matrix
1
| Incorrect randomness handling | 3
Others | Misconfiguration | 4
| Type problem | 1
| Typo | 2
| Overflow error | 1

####  IV-D1 API-related

In quantum programs, if the error is caused by an API, we classify it into
API-related bug pattern. There are two main subcategories under this category,
including API misuse, which refers to errors caused by using the wrong
function method or introducing the wrong parameters into a function, and
outdated API client, which refers to code edited with an older, deprecated
version of the API. Figure 5 shows a bug belonging to the API-misuse, where
the execute function got an unexpected keyword argument seed.

        shots = 1024
    -   job = execute(circuits, backend = Aer.get˙backend(’qasm_simulator’), shots=shots, seed=8)
    +   job = execute(circuits, backend=Aer.get˙backend(’qasm_simulator’), shots=shots)
        result = job.result()

Figure 5: Example of the API misuse, in which the programmer used an
unexpected keyword argument seed.

####  IV-D2 Incorrect application logic

This type of bug pattern is mainly caused by errors in the implementation of
the application logic when writing a quantum program. This bug pattern
consists of six subcategories: Intermediate representation is a bug caused by
the corruption or loss of an intermediate variable in a quantum program.
Referring to the wrong program element is an error caused by a programmer
confusing two elements (e.g., two related concepts and two identifiers with
similar names) in a quantum program and misusing them. Incorrect scheduling is
a bug caused by code running at the wrong timestamp in a quantum program
(e.g., a new instruction is incorrectly scheduled to execute on the hardware
when it is occupied). In qubit-related, it mainly includes qubit-related
errors, such as counting the wrong number of the qubits or causing wrong
ordered qubits. Figure 6 shows an example that the programmer confused the two
operations in Qiskit: Gates, which can be controlled, and Instructions, which
cannot be controlled, leading to the error. Besides, we define two new bug
patterns for existing quantum programs, which are discussed in detail as
follows.

        qc = QuantumCircuit(3)
        outer_level = QuantumCircuit(2, name=’outer’)
        inner_level = QuantumCircuit(2, name=’inner’)
        inner_level.x(0)
    -   outer_level.append(inner_level, [0,1])
    +   outer_level.append(inner_level.to˙gate(), [0,1])
        qc.append(outer_level.control(), [0,1,2])

Figure 6: Example of the wrong concept, in which the inner_level produced from
QuantumCircuit cannot be controlled.

Wrong declaration. This bug pattern can happen mainly in Q#. Since the code is
automatically wrapped in the namespace in Q# Jupyter notebooks, there is no
need to re-declare the same namespace in the code. To fix this bug, one needs
to remove the extra namespace declaration from the code.

Wrong way of implementation. In the process of developing quantum programs,
this type of bug can be caused if the programmer is not clear about the exact
implementation of the function needed. This bug pattern is characterized by a
large amount of code to be fixed and its high complexity (5 out of 7 samples
need to be fixed with more than 5 lines, with a complexity of C3 or C4), which
is a great challenge for the current bug fix automation approaches.

####  IV-D3 Math-related

In developing a quantum program, the programmer usually has to manipulate the
qubits through a series of mathematical operations to accomplish the desired
function. In Paltenghi and Pradel’s work[14], math-related bugs are divided
into two categories: incorrect numerical computations and incorrect randomness
handling. If the code misuses randomness or probability, it will be classified
into incorrect randomness handling patterns. Incorrect numerical computations
mean that the formula is misused. In quantum programs, since the operations on
qubits are more diverse than those in classical programs, in our study, we
further subdivide this bug pattern as follows.

Incorrect gate. In quantum programs, to control quantum circuits, programmers
usually need to use quantum logic gates to process qubits to implement a
particular quantum algorithm. In this process, if the wrong quantum logic gate
is used, or if the wrong qubits are processed with the quantum logic gate, it
can lead to anomalies in the quantum program. Figure 7 gives an example of an
incorrect gate, in which the output does not match the expected result because
the programmer used the wrong quantum logic gate.

        circuit = QuantumCircuit(2)
        circuit.h(0)
    -   circuit.h(1)
    +   circuit.x(1)
        circuit.cx(0,1)
        circuit.measure˙all()

Figure 7: Example of the incorrect gate usage resulting in an output that is
not as expected.

Incorrect initial state. In quantum programs, when writing quantum algorithms,
the programmer needs to set the initial state of the qubits. If the wrong
initial state is used, it can lead to an error in the quantum circuit. In
Figure 8, the programmer tries to create the Bell State
$\frac{1}{\sqrt{2}}\left(|01\rangle+|10\rangle\right)$, however, the wrong
initial state is used, leading to the wrong result
$\frac{1}{\sqrt{2}}\left(|11\rangle-|00\rangle\right)$.

    -   sv = Statevector.from˙label(’01’)
    +   sv = Statevector.from˙label(’10’)
        mycircuit = QuantumCircuit(2)
        mycircuit.h(0)
        mycircuit.cx(0,1)

Figure 8: Example of the incorrect initial state resulting in a wrong
expression of the Bell State.

Incorrect measurement. In quantum programs, measurement is a common operation
in quantum algorithms, but it can cause errors in quantum programs if
measurements are made at the wrong time or on the wrong qubits. In Figure 9,
the programmer reversed the order of the qubits when measuring them, leading
to an error in the measurement result.

    -   circuit.measure([0,1,2], [0,1,2])
    +   circuit.measure([0,1,2], [2,1,0])

Figure 9: Example of wrong measurement order leading to incorrect results.

Potential all-zeros matrix. In general, matrix operations frequently occur in
quantum programs. And in matrix operations, unintentionally generated all-
zeros matrices pose a risk to the operation and cause the algorithm’s
stability to deteriorate. Figure 10 shows a bug belonging to this bug pattern,
where the operation .dot(unitary[randnum[0]] - id) causes the case where rho
is an all-zeros matrix occurs frequently.

        state0 = np.array([[1,0],[0,0]])
        state1 = np.array([[0,0],[0,1]])
    +   states = [state0, state1]
        record_rho = np.zeros([4,4])
        for i in range(snapshot_num):
            randnum = np.random.randint(0,3,size=2)
            result = one˙shot(randnum)
    +       bit0, bit1 = [int(x) for x in list(result.keys())[0]]
    +       U0, U1 = unitary[randnum[0]], unitary[randnum[1]]
    -       if result.get(’00’) == 1:
    -           rho = np.kron(3*np.dot(unitary[randnum[0]].conj().T,state0).dot(unitary[randnum[0]] - id),
                      3*np.dot(unitary[randnum[1]].conj().T,state0).dot(unitary[randnum[1]]) - id)
    -       elif result.get(’01’) == 1:
    -           rho = np.kron(3*np.dot(unitary[randnum[0]].conj.T,state0).dot(unitary[randnum[0]] - id),
                      3*np.dot(unitary[randnum[1]].conj().T,state1).dot(unitary[randnum[1]]) - id)
    -       elif result.get(’10’) == 1:
    -           rho = np.kron(3*np.dot(unitary[randnum[0]].conj.T,state1).dot(unitary[randnum[0]] - id),
                      3*np.dot(unitary[randnum[1]].conj.T,state0).dot(unitary[randnum[1]]) - id)
    -       else:
    -           rho = np.kron(3*np.dot(unitary[randnum[0]].conj.T,state1).dot(unitary[randnum[0]] - id),
                      3*np.dot(unitary[randnum[1]].conj.T,state1).dot(unitary[randnum[1]]) - id)
    +       rho = np.kron(3* U0.conj.T @ states[bit0] @ U0 - id, 3*U1.conj.T @ states[bit1] @ U1 - id)
            record_rho = record_rho + rho

Figure 10: Example of a bug that may cause all-zeros matrix because of the
operation .dot(unitary[randnum[0]] - id).

Incorrect matrix computation. Another common type of operation in quantum
programs is matrix operations. Although the processing of qubits using quantum
logic gates can also be considered a matrix operation, in our study, we
distinguish between the two because, usually, quantum logic gates are pre-
defined functions in quantum languages. In contrast, other matrix operations
need to be implemented by the programmer in his code, and it is difficult to
classify this part of the operation as some classical quantum logic gate.
Figure 11 gives an example of incorrect matrix computation, in which the
simulator throws an error because the U_error matrix set by the programmer is
not unitary.

    +   z = 0.995004165 + 1j * 0.099833417
    +   z = z / abs(z)
    +   u_error = np.array([[1, 0], [0, z]])
        noise_params = {’U’:
                             {’gate_time’: 1,
                            ’p_depol’: 0.001,
                            ’p_pauli’: [0, 0, 0.01],
    -                       ’U_error’: [[[1, 0], [0, 0]], [[0, 0], [0.995004165, 0.099833417]]]}
    +                       ’U_error’: u_error }
                        }

Figure 11: Example of an incorrect parameter of a matrix, where the matrix is
non-unitary causing an error.

####  IV-D4 Other bug patterns

In addition to the three main types of bug patterns mentioned above, there are
some other patterns included in our study, which include four types of
Misconfiguration, Type problem, Typo, and Overflow error. Figure 12 gives an
example of type problem, in which the programmer misunderstands the return
type of function .draw(output = ’mpl’). Since they are roughly
indistinguishable from those in classical programs, we will not describe them
in detail.

        qr = QuantumRegister(2)
        cr = ClassicalRegister(2)
        Qc = QuantumCircuit(qr,cr)
        print(’This is the initial state’)
    -   print(Qc.draw(output = ’mpl’))
    +   Qc.draw(output = ’mpl’)

Figure 12: Example of the type problem caused by the misunderstanding of the
return type of .draw(output = ’mpl’).

Based on the above analysis of bug patterns, we have the following findings:

Finding 7. Overall, the statistics show that in the existing quantum programs,
programmers’ bugs are mainly focused on API-related bugs (35 out of 96
samples) and math-related bugs (26 out of 96 samples). This means that
programmers are still not proficient in quantum programming languages when
developing quantum programs and have difficulties with the algorithms and
manipulation of qubits. Therefore, it should be one of the focuses for future
work on bug fixing.

Our study also counts the cases of classical and quantum-specific bugs in
quantum programs separately. The detailed data are shown in Figure 13, where
the vertical axis lists the bug patterns as mentioned above, and the
horizontal axis shows the frequency of occurrence in the data we investigated,
with the classical and quantum-specific bugs shown in blue and red bars,
respectively.

Figure 13: Bug pattern statistics for classical and quantum-specific bugs.

Finding 8. Consistent with the above findings, quantum-specific bugs are
mainly concentrated in API-related and math-related bug patterns. On the other
hand, classical bugs are smaller in total and evenly distributed across bug
patterns, which means that in the study of bug fixing in quantum programs,
researchers still need to pay extensive attention to the bugs that occur in
classical programs. The high percentage of quantum-specific bugs in API-
related and math-related bug patterns also indicates that the current bug
fixing efforts are difficult to apply directly to quantum programs, and
further research is needed in this area.

##  V Related Work

To the best of our knowledge, this work is the first study on the bug fixing
of quantum programs. Many empirical studies on bug fixing have been proposed
in classical software systems. Zhong and Su [13] collected more than 9000
real-world bugs from Java projects, analyzed them comparatively, and presented
findings and insights. Zhang and Khomh et al. [15] focused on the factors
affecting bug fixing time. Park and Kim et al. [16] discussed additional bug
fixes, i.e., multiple attempts to fix bugs. Yin and Yuan et al. [17]
researched bug fixes from large operating system code bases and proposed that
the process of fixing bugs can also introduce bugs. However, all the studies
above focus on fixing the bugs of classical software systems rather than
quantum software systems.

Paltenghi and Pradel [14] presented an empirical study of bugs in quantum
computing platforms. They summarized 223 real-world bugs in some open-source
quantum computing platforms, conducted a detailed statistical analysis of
bugs, and proposed some bug patterns specific to the field of quantum
computing platforms. The difference between their work and ours is that they
focused on the bugs on quantum computing platforms, while we focused on
quantum programs themselves. However, the findings distilled from both studies
can benefit the deep understanding of bugs related to quantum computing
systems.

In addition, several researches have been carried out to study the bug
patterns (types) [18, 7], as well as bug benchmarks [19, 20] in quantum
programs, in order to support the debugging and testing of quantum software.

##  VI Concluding Remarks

In this paper, we have presented a comprehensive study of bug fixing in
quantum programs. Based on the 96 bugs we collected in four popular quantum
programming languages (Qiskit, Cirq, Q#, and ProjectQ), we analyzed them in
terms of the bug types, complexity, the file types being fixed, and their bug
patterns, respectively, and distilled eight findings. Our research shows that
a high percentage of bugs are quantum-specific bugs. Also, lack of proficiency
in using quantum programming languages and miswritten quantum algorithms are
key causal factors for bugs. We believe this study will help guide future
research on automatically preventing and fixing bugs in quantum programs.

## References

  * [1] E. Allen, “Bug patterns in Java,” 2002.
  * [2] I. Research, “Qiskit,” _Accessed on: June, 2021_. [Online]. Available: https://qiskit.org
  * [3] G. A. Q. team, “Cirq,” 2018. [Online]. Available: https://github.com/quantumlib/Cirq
  * [4] K. Svore, A. Geller, M. Troyer, J. Azariah, C. Granade, B. Heim, V. Kliuchnikov, M. Mykhailova, A. Paz, and M. Roetteler, “Q#: enabling scalable quantum computing and development with a high-level dsl,” in _Proceedings of the Real World Domain Specific Languages Workshop 2018_ , 2018, pp. 1–10.
  * [5] P. Team, “ProjectQ,” _Accessed on: April, 2020_ , 2017. [Online]. Available: https://projectq.ch/
  * [6] G. Li, L. Zhou, N. Yu, Y. Ding, M. Ying, and Y. Xie, “Projection-based runtime assertions for testing and debugging quantum programs,” _Proceedings of the ACM on Programming Languages_ , vol. 4, no. OOPSLA, pp. 1–29, 2020.
  * [7] Y. Huang and M. Martonosi, “Statistical assertions for validating patterns and finding bugs in quantum programs,” in _Proceedings of the 46th International Symposium on Computer Architecture_ , 2019, pp. 541–553.
  * [8] S. Honarvar, M. Mousavi, and R. Nagarajan, “Property-based testing of quantum programs in Q#,” in _First International Workshop on Quantum Software Engineering (Q-SE 2020)_ , 2020.
  * [9] A. Miranskyy and L. Zhang, “On testing quantum programs,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)_. IEEE, 2019, pp. 57–60.
  * [10] S. Ali, P. Arcaini, X. Wang, and T. Yue, “Assessing the effectiveness of input and output coverage criteria for testing quantum programs,” in _2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 2021, pp. 13–23.
  * [11] J. Zhao, “Quantum software engineering: Landscapes and horizons,” _arXiv preprint arXiv:2007.07047_ , 2020.
  * [12] M. A. Nielsen and I. Chuang, “Quantum computation and quantum information,” 2002\.
  * [13] H. Zhong and Z. Su, “An empirical study on real bug fixes,” in _2015 IEEE/ACM 37th IEEE International Conference on Software Engineering_ , vol. 1, 2015, pp. 913–923.
  * [14] M. Paltenghi and M. Pradel, “Bugs in quantum computing platforms: An empirical study,” 2021.
  * [15] F. Zhang, F. Khomh, Y. Zou, and A. E. Hassan, “An empirical study on factors impacting bug fixing time,” in _2012 19th Working conference on reverse engineering_. IEEE, 2012, pp. 225–234.
  * [16] J. Park, M. Kim, B. Ray, and D.-H. Bae, “An empirical study of supplementary bug fixes,” in _2012 9th IEEE Working Conference on Mining Software Repositories (MSR)_. IEEE, 2012, pp. 40–49.
  * [17] Z. Yin, D. Yuan, Y. Zhou, S. Pasupathy, and L. Bairavasundaram, “How do fixes become bugs?” in _Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering_ , 2011, pp. 26–36.
  * [18] P. Zhao, J. Zhao, and L. Ma, “Identifying bug patterns in quantum programs,” in _Proc. 2nd International Workshop on Quantum Software Engineering (Q-SE 2021)_ , 2021, pp. 16–21.
  * [19] J. Campos and A. Souto, “QBugs: A collection of reproducible bugs in quantum algorithms and a supporting infrastructure to enable controlled quantum software testing and debugging experiments,” in _Proc. 2nd International Workshop on Quantum Software Engineering (Q-SE 2021)_ , 2021.
  * [20] P. Zhao, J. Zhao, Z. Miao, and S. Lan, “Bugs4Q: A benchmark of real bugs for quantum programs,” _arXiv preprint arXiv:2108.09744_ , 2021.


FERMILAB-CONF-23-751-CMS-PPD

# Induced Generative Adversarial Particle Transformers

Anni Li Venkat Krishnamohan∗ Raghav Kansal
Rounak Sen Steven Tsan Zhaoyu Zhang Javier Duarte
University of California, San Diego
La Jolla, CA 92093, USA
<EMAIL_ADDRESS>Equal
contribution.Also at Fermi National Accelerator Laboratory.

###### Abstract

In high energy physics (HEP), machine learning methods have emerged as an
effective way to accurately simulate particle collisions at the Large Hadron
Collider (LHC). The message-passing generative adversarial network (MPGAN) was
the first model to simulate collisions as point, or “particle”, clouds, with
state-of-the-art results, but suffered from quadratic time complexity.
Recently, generative adversarial particle transformers (GAPTs) were introduced
to address this drawback; however, results did not surpass MPGAN. We introduce
induced GAPT (iGAPT) which, by integrating “induced particle-attention blocks”
and conditioning on global jet attributes, not only offers linear time
complexity but is also able to capture intricate jet substructure, surpassing
MPGAN in many metrics. Our experiments demonstrate the potential of iGAPT to
simulate complex HEP data accurately and efficiently.

##  1 Introduction

In high energy physics (HEP), jets—sprays of particles produced at high energy
collisions at the Large Hadron Collider (LHC), serve as a vital bridge between
theoretical research and experimental observations. The intricate patterns in
jets make them invaluable for understanding quantum chromodynamics (QCD),
identifying rare particles like the Higgs boson, and conducting quantitative
analyses of particle properties and interactions. Machine learning methods,
such as generative adversarial networks (GANs), have recently emerged as
effective and efficient approaches to simulating such high-energy collisions,
reducing the computational burden of traditional methods. The message-passing
GAN (MPGAN) [1], in particular, represented a significant leap forward, being
able to capture the complex global structure of jets and handle variable-sized
particle clouds using fully-connected graph neural networks. However, this
fully-connected nature means its memory and time complexity scale
quadratically with the number of particles per jet.

The generative adversarial particle transformer (GAPT) [2] was recently
introduced to improve the efficiency of MPGAN using self-attention, but GAPT
does not match MPGAN’s performance. Others have also studied normalizing flows
[3], other types of transformers [4, 5], diffusion models [6, 7], and
equivariant networks [8] for the same generative task. In this paper, we
introduce the induced GAPT (iGAPT), featuring “induced particle attention
blocks” incorporating physics-informed inductive biases of jets, to offer
linear time complexity and improve output fidelity. We investigate its
performance and timing compared to MPGAN and alternative architecture choices
on high-momentum jets.

##  2 Architecture

Figure 1: Illustration of the generation and discrimination process of iGAPT.

The iGAPT model, illustrated in Fig. 1, is a GAN with the generator and
discriminator comprising “induced particle attention blocks” (IPABs). On top
of permutation invariance and point-cloud representations, as in MPGAN and
GAPT, the key inductive bias we experiment with in iGAPT is maintaining a
global vector through the generation and discrimination processes,
$\mathbf{z}$, which implicitly represents global jet features. IPABs and
different ways of incorporating $\mathbf{z}$ into the attention process are
described below.

The generation process starts with sampling random Gaussian noise and a
particle multiplicity $N$ from the true distribution, which is transformed via
a learned linear embedding layer. The noise has two components: a set of
$N_{\mathrm{max}}$ vectors representing initial particle features, and a
single vector representing initial jet features. $N_{\mathrm{max}}$ is the
maximum number of particles per jet we want to simulate, and the number of
initial particle and jet features is a hyperparameter we tune. The jet noise
is added to the embedded $N$ to produce $\mathbf{z}$, which is then
transformed along with the particle noise via multiple IPABs to output a
generated jet.

The discrimination process starts with a generated or real jet, and the
sampled or true jet multiplicity $N$. This is again transformed via a learned
embedding layer to produce the $\mathbf{z}$ conditioning vector for the
discriminator, which along with the input jet are processed through IPABs
producing an intermediate jet representation. The constituents of this jet are
aggregated in a permutation-invariant manner using a pooling by multihead
attention (PMA) layer, as introduced in [9], the output of which is finally
fed into a linear layer to classify the jet as real or fake.

Attention blocks require fixed multiplicity, however, jets naturally have a
variable number of particle constituents. To handle this, we zero-pad all jets
to $N_{\mathrm{max}}$ particles and use masked attention in every block to
ignore these.

### Attention Blocks

Figure 2: Illustration of an induced particle attention block (IPAB).

The original GAPT model uses self-attention blocks (SABs) introduced for set
transformers [9], a form of multi-head attention blocks (MABs) with the input
attending itself — i.e., each particle attends to every other in the jet to
update its features. This has the benefit of global, fully connected
interactions between particles; however, it also means quadratic scaling with
the particle multiplicity, which is undesirable.

To counter this, Ref. [9] also proposed “induced” SABs (ISABs), where the
input set is first attended to by $M$ learnt inducing vectors via a MAB,
outputting an intermediate, compressed representation. This representation is
then attended to by the original set to produce the new output. This retains
the global interactions between particles while achieving $\mathcal{O}(NM)$
scaling—linear in the particle multiplicity. ISABs have been used in
generative adversarial set transformers (GAST) [10], yielding high quality
results on computer vision datasets. GAST incorporates similar global set
features $\mathbf{z}$ by concatenating them to each particle before every ISAB
operation.

In iGAPT, we experiment with an alternative version of conditioning, which we
call “induced particle attention”. Here, $\mathbf{z}$ is directly used as the
inducing vector in an ISAB, and is continuously updated through the
intermediate, induced attention outputs. Explicitly, as illustrated in Fig. 2,
the $i$th IPAB receives as input the jet $\mathbf{x}^{i}$ and global features
$\mathbf{z}^{i}$ from the previous block, after which $\mathbf{x}^{i}$ is
first attended to by $\mathbf{z}^{i}$ to output updated features
$\mathbf{z}^{i+1}$, and then conversely $\mathbf{z}^{i+1}$ is attended to by
$\mathbf{x}^{i}$ to output the updated jet $\mathbf{x}^{i+1}$. This is
interpreted as a way to update and learn the global jet features, such as mass
and $p_{\mathrm{T}}$, along with the individual particle features, and allow
both to interact in each attention layer. An additional and significant
advantage is that this induced attention operation involves only one inducing
vector — $\mathbf{z}$, hence $M=1$ and we achieve $\mathcal{O}(N)$
computational complexity.

##  3 Experiments

### Dataset

We perform tests on the JetNet dataset [11] using the associated library [12,
1]. The JetNet dataset contains simulated high transverse-momentum
($p_{\mathrm{T}}$) jets, originating from gluons, light quarks, top quarks,
incorporating detector effects emulating those of the CMS detector at the LHC.
We test the iGAPT model on these different classes, simulating only the 30 as
well as the 150 highest $p_{\mathrm{T}}$ particles per jet. This complex task
involves simulating both the initial parton showering and hadronization from
an individual quark or gluon, as well as the proceeding detector response.
Each jet is represented as a set, or point cloud, of its constituent
particles, which are described by three features: their angular coordinates
($\eta^{\mathrm{rel}}$, $\phi^{\mathrm{rel}}$), and $p_{\mathrm{T}}$, relative
to the jet.

### Training and Hyperparameters

The iGAPT and GAST models were trained for a maximum of 6000 epochs on a
single NVIDIA RTX1080 GPU using the RMSProp optimizer. The training time and
batch size for 30- and 150-particle gluon jets is shown in Table 2. We use a
two-time update rule [13], with learning rates of $0.5\times 10^{-4}$ and
$1.5\times 10^{-4}$ for the generator and discriminator, respectively. The
discriminator is regularized with dropout with a probability of 0.5 and layer
normalization. We use a LeakyReLU activation after every linear layer, except
the final generator (tanh) and discriminator (sigmoid) outputs. After
hyperparameter tuning, we settle on on 3 and 6 ISAB layers for the generator
and discriminator respectively, and 4 and 8 IPAB layers for the iGAPT model.
We use 16 and 128 initial particle and jet features, respectively, in iGAPT.
GAST uses 20 inducing vectors for its ISABs.

### Evaluation

We evaluate the performance of each model both visually, comparing feature
distributions between the real and generated samples, and using quantitative
evaluation metrics. Reference [2] found the Fréchet physics distance (FPD),
kernel physics distance (KPD), and 1-Wasserstein distances ($W_{1}$) between
individual particle and jet features to be sensitive metrics to
comprehensively evaluate the physics performance of generative models. FPD and
KPD, in particular, consider a set of 36 energy flow polynomials (EFPs) [14],
which collectively form a basis for all infrared- and collinear-safe
observables, and hence capture significant information about the jet
substructure. We report the FPD, KPD, and $W_{1}$ scores for the particle
$p_{\mathrm{T}}$ ($W^{p_{\mathrm{T}}^{\mathrm{rel}}}_{1p}$) and jet mass
($W_{1}^{\mathrm{M}}$) and their errors following the procedures outlined in
Ref. [2] using the JetNet library for each generative model, as well as
between independent samples of real jets to measure a baseline, “truth” score.

### Results

Figure 3: Low-level particle feature distributions (far left and center left)
and high-level jet feature distributions (center right and far right) for the
real data (red), MPGAN-generated data (blue), and iGAPT-generated data
(green), for 30-particle gluon (top row), light quark (middle), and top quark
jets (bottom). A sample $d=4$ energy flow polynomial [14] is plotted in the
rightmost column.

We test the iGAPT model on 30-particle gluon, light quark, and top quark jets,
and on 150-particle gluon jets. We compare this with the MPGAN and GAPT models
provided in their respective publications, as well as as our own trainings of
GAST models. Out of all our trainings, we select the iGAPT and GAST models
with the lowest FPD score, as advocated in Ref. [2] due to its high
sensitivity to common types of mismodelling. Comparisons of real and generated
feature distributions are shown in Fig. 3 and Appendix A for 30 and 150
particles, respectively, demonstrating high fidelity results by iGAPT. The
truth and model scores for the aforementioned evaluation metrics are listed in
Table 1. We note that MPGAN models for 150-particle jets are not provided
because of their poor time complexity. We observe first that iGAPT largely
outperforms the GAPT and GAST models, validating our physics-informed approach
to its architecture. It also outperforms MPGAN on 30-particle jets on many jet
types and metrics, notably on FPD for all jet types. Important we recognize
that no model is perfectly consistent with the true, baseline distributions,
indicating room for improvement.

Table 1: Evaluation metrics for different jet types and models. The best-performing model on each metric and jet type is highlighted in bold. | Model |  $W^{p_{\mathrm{T}}^{\mathrm{rel}}}_{1p}$ ($10^{-3}$) |  $W_{1}^{\mathrm{M}}$ ($10^{-3}$) | FPD ($10^{-3}$) | KPD ($10^{-6}$)
---|---|---|---|---|---
Gluon (30) | Truth | $0.14\pm 0.06$ | $0.46\pm 0.08$ | $0.14\pm 0.04$ | $1.8\pm 11.9$
MPGAN | $0.27\pm 0.02$ | $0.7\pm 0.3$ | $0.41\pm 0.09$ | $\mathbf{0\pm 8}$
GAPT | $\mathbf{0.25\pm 0.07}$ | $1.0\pm 0.2$ | $0.46\pm 0.06$ | $5\pm 3$
GAST | $0.8\pm 0.1$ | $0.7\pm 0.2$ | $0.40\pm 0.05$ | $7.0\pm 10.3$
iGAPT | $0.76\pm 0.07$ | $\mathbf{0.7\pm 0.1}$ | $\mathbf{0.29\pm 0.04}$ | $3\pm 5$
Light quark (30) | Truth | $0.21\pm 0.05$ | $0.5\pm 0.2$ | $0.09\pm 0.03$ | $-3\pm 3$
MPGAN | $\mathbf{0.41\pm 0.07}$ | $\mathbf{0.5\pm 0.1}$ | $1.9\pm 0.2$ | $\mathbf{1.7\pm 15.1}$
GAPT | $2.74\pm 0.09$ | $2.54\pm 0.05$ | $4.03\pm 0.06$ | $96\pm 9$
GAST | $1.2\pm 0.1$ | $1.8\pm 0.2$ | $0.65\pm 0.07$ | $27.0\pm 11.7$
iGAPT | $1.89\pm 0.04$ | $1.2\pm 0.3$ | $\mathbf{0.51\pm 0.07}$ | $12\pm 7$
Top quark (30) | Truth | $0.20\pm 0.05$ | $0.7\pm 0.2$ | $0.07\pm 0.03$ | $-16\pm 2$
MPGAN | $0.44\pm 0.08$ | $\mathbf{0.5\pm 0.1}$ | $2.8\pm 0.2$ | $14.7\pm 12.9$
GAPT | $\mathbf{0.34\pm 0.02}$ | $1.9\pm 0.2$ | $0.43\pm 0.03$ | $25.4\pm 28.8$
GAST | $1.16\pm 0.08$ | $1.5\pm 0.2$ | $0.30\pm 0.05$ | $\mathbf{-2.4\pm 17.2}$
iGAPT | $0.54\pm 0.04$ | $0.9\pm 0.3$ | $\mathbf{0.25\pm 0.03}$ | $-0.6\pm 14.1$
Gluon (150) | Truth | $0.09\pm 0.03$ | $0.7\pm 0.2$ | $0.10\pm 0.03$ | $0.5\pm 10.5$
GAPT | $0.77\pm 0.03$ | $\mathbf{1.1\pm 0.3}$ | $22.0\pm 0.1$ | $62.5\pm 11.1$
GAST | $0.68\pm 0.05$ | $3.7\pm 0.3$ | $3.60\pm 0.06$ | $\mathbf{47.7\pm 13.8}$
iGAPT | $\mathbf{0.66\pm 0.03}$ | $4.4\pm 0.7$ | $\mathbf{2.99\pm 0.06}$ | $158.1\pm 37.9$

### Timing

A key benefit of iGAPT is its improved time complexity over MPGAN. This is
demonstrated in Table 2, which shows the training and generation times for
each model for 30 particle jets using the largest batch size possible on an
NVIDIA 1080 GPU, with iGAPT outperforming MPGAN by a factor of 3.5. MPGAN is
computationally challenging to extend to 150 particles, hence timing
information is not provided; in contrast, iGAPT’s training and generation
times scale well with the number of particles. Finally, we note that the
“true” generation time per jet is approximately 50 ms [1], thus iGAPT
represents more than a factor of 100 speed up.

Table 2: Timing measurements for MPGAN and iGAPT, measured on an NVIDIA 1080 GPU. Jet type | Model | Training time |  | Generation time |  | Batch size
---|---|---|---|---|---|---
|  | (s/epoch) |  | (µs / jet) |  |
Gluon, $n=30$ | MPGAN | $193$ |  | $142$ |  | 512
iGAPT | $31$ |  | $40$ |  | 4096
Gluon, $n=150$ | iGAPT | $267$ |  | $315$ |  | 512

##  4 Conclusion

We presented the induced generative adversarial particle transformer (iGAPT),
which utilizes a new induced particle attention mechanism to achieve high-
fidelity simulations of jets in high energy physics. By incorporating physics-
informed inductive biases and conditioning on global jet features, iGAPT
demonstrates not only competitive performance with previous state-of-the-art
models on 30-particle gluon, light quark, and top quark jets, but also
significantly better computational efficiency and time complexity.
Furthermore, iGAPT exhibits strong potential to simulate larger numbers of
particles, establishing its viability in applications to realistic event
collisions at the LHC. However, we note room for improvement in iGAPT’s
performance with respect to the baseline scores, and future work may explore
further optimizations of its training and performance.

## Acknowledgments and Disclosure of Funding

This work was supported by the U.S. Department of Energy (DOE) Award Nos. DE-
SC0021187 and DE-SC0021396 (FAIR4HEP) and the U.S. National Science Foundation
(NSF) Cooperative Agreement OAC-2117997 (A3D3). A. L. was partially supported
by an IRIS-HEP fellowship through the U.S. National Science Foundation (NSF)
under Cooperative Agreement OAC-1836650. R. K. was partially supported by the
LHC Physics Center at Fermi National Accelerator Laboratory, managed and
operated by Fermi Research Alliance, LLC under Contract No. DE-AC02-07CH11359
with the DOE.

## References

  * [1] Raghav Kansal et al.  “Particle Cloud Generation with Message Passing Generative Adversarial Networks”  In _Advances in Neural Information Processing Systems_ 34 Curran Associates, Inc., 2021, pp. 23858  arXiv: https://papers.nips.cc/paper/2021/hash/c8512d142a2d849725f31a9a7a361ab9-Abstract.html
  * [2] Raghav Kansal et al.  “Evaluating generative models in high energy physics”  In _Phys. Rev. D_ 107.7, 2023, pp. 076017  DOI: 10.1103/PhysRevD.107.076017
  * [3] Breno Orzari et al.  “LHC hadronic jet generation using convolutional variational autoencoders with normalizing flows”  In _Mach. Learn. Sci. Tech._ 4, 2023, pp. 045023  DOI: 10.1088/2632-2153/ad04ea
  * [4] Benno Kach, Dirk Krücker and Isabell Melzer-Pellmann  “Point Cloud Generation using Transformer Encoders and Normalising Flows”, 2022  arXiv:2211.13623 [hep-ex]
  * [5] Benno Kach and Isabell Melzer-Pellmann  “Attention to Mean-Fields for Particle Cloud Generation”, 2023  arXiv:2305.15254 [hep-ex]
  * [6] Matthew Leigh et al.  “PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physics”, 2023  arXiv:2303.05376 [hep-ph]
  * [7] Vinicius Mikuni, Benjamin Nachman and Mariel Pettee  “Fast point cloud generation with diffusion models in high energy physics”  In _Phys. Rev. D_ 108.3, 2023, pp. 036025  DOI: 10.1103/PhysRevD.108.036025
  * [8] Erik Buhmann, Gregor Kasieczka and Jesse Thaler  “EPiC-GAN: Equivariant point cloud generation for particle jets”  In _SciPost Phys._ 15.4, 2023, pp. 130  DOI: 10.21468/SciPostPhys.15.4.130
  * [9] Juho Lee et al.  “Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, 2019, pp. 3744  arXiv: https://proceedings.mlr.press/v97/lee19d.html
  * [10] Karl Stelzner, Kristian Kersting and Adam R Kosiorek  “Generative adversarial set transformers”  In _Workshop on Object-Oriented Learning at ICML_ 3, 2020, pp. 1  URL: https://ml-research.github.io/papers/stelzner2020ood_gast.pdf
  * [11] Raghav Kansal et al.  “JetNet” https://doi.org/10.5281/zenodo.6975118 Zenodo, 2022  DOI: 10.5281/zenodo.6975118
  * [12] Raghav Kansal, Carlos Pareja, Zichun Hao and Javier Duarte  “JetNet: A Python package for accessing open datasets and benchmarking machine learning methods in high energy physics”  In _J. Open Source Softw._ 8.90, 2023, pp. 5789  DOI: 10.21105/joss.05789
  * [13] Martin Heusel et al.  “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium”  In _Advances in Neural Information Processing Systems_ 30 Red Hook, NY: Curran Associates, Inc., 2017, pp. 6626  arXiv: https://papers.nips.cc/paper_files/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html
  * [14] Patrick T. Komiske, Eric M. Metodiev and Jesse Thaler  “Energy flow polynomials: A complete linear basis for jet substructure”  In _JHEP_ 04, 2018, pp. 013  DOI: 10.1007/JHEP04(2018)013

##  Appendix A Feature distributions of 150-particle jets

The distributions of real and generated particle and jet features for
150-particle gluon jets by MPGAN and iGAPT are shown in Fig. 4.

Figure 4: Low-level particle feature distributions (far left and center left)
and high-level jet feature distributions (center right and far right) for the
real data (red), MPGAN-generated data (blue), and iGAPT-generated data
(green), for 150-particle gluon jets. A sample $d=4$ energy flow polynomial
[14] is chosen in the rightmost column.