11institutetext:  Department of Computer Science and Engineering, University
of Bologna
Viale Risorgimento 2, 40136 Bologna, Italy
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>22institutetext: Datalogic S.r.l.
Via San Vitalino 13, 40012 Calderara di Reno BO, Italy
22email<EMAIL_ADDRESS>
# SAFFIRE: System for Autonomous Feature Filtering and Intelligent ROI
Estimation

Marco Boschi Corresponding author.11 Luigi Di Stefano 11 Martino Alessandrini
22

###### Abstract

This work introduces a new framework, named SAFFIRE, to automatically extract
a dominant recurrent image pattern from a set of image samples. Such a pattern
shall be used to eliminate pose variations between samples, which is a common
requirement in many computer vision and machine learning tasks.

The framework is specialized here in the context of a machine vision system
for automated product inspection. Here, it is customary to ask the user for
the identification of an anchor pattern, to be used by the automated system to
normalize data before further processing. Yet, this is a very sensitive
operation which is intrinsically subjective and requires high expertise.
Hereto, SAFFIRE provides a unique and disruptive framework for unsupervised
identification of an optimal anchor pattern in a way which is fully
transparent to the user.

SAFFIRE is thoroughly validated on several realistic case studies for a
machine vision inspection pipeline.

###### Keywords:

Machine vision Object detection Pattern learning Graph analysis Anchor pattern
identification ROI detection.

##  1 Introduction

Machine vision systems are key for the automation of the production line.
Hereto, in a typical installation, a camera (or a set of cameras) is mounted
on top of the conveyor transporting the manufactured items. The images are
analyzed by a specific software which grades the quality of the product.
Typical examples include: reading/verifying a printed text/barcode;
identifying a pattern; measuring a part; verifying the absence of defects
(flaw detection), and combinations of thereof. The output grading is then used
to keep or discard that item for distribution.

It is often the case that the content to be analyzed resides in a small sub-
portion of the image (i.e., a _region of interest_ (ROI)). As such, a
necessary pre-processing step is the identification of such a region in the
image. We call this process ROI finding. Of note, ROI finding is not only
required for speedup, but is often necessary for the meaningfulness of the
following analysis. For instance, one might need to select the correct line of
text before reading it, or to locate the specific mechanical part to be
measured. In many cases, the inspected items are left free to assume an
arbitrary pose relative to camera and, as such, the ROI has to be tracked by
the automated software independently on each sample.

A typical way to address the ROI finding problem is by instructing the system
with an _anchor pattern_. An anchor is a distinctive recurrent (i.e., present
on all samples) image pattern (e.g., a specific mark on a printed label)
identifying a local coordinate system where the ROI position is (as much as
possible) constant. Such a pattern shall be detected by the machine vision
system and used to position the ROI on a new sample. Importantly, the anchor
is generally not located inside the ROI. Indeed, the focus of an inspection is
commonly on regions which are expected to change.

In many machine vision software products, the anchor pattern is annotated
manually by the user when training the system. Manual annotation is
disadvantageous for several reasons:

  1. 1.

It requires expertise: The user has to know what an anchor is and how to use
it to configure the specific inspection (e.g., by instructing a
template/pattern matching routine to locate the pattern on new samples);

  2. 2.

Visual identification of a viable anchor pattern can be complex. For instance,
the anchor pattern has to be isolated among many (e.g., for complex printed
labels) or, possibly, the anchor is represented by a set of visual features
which are distributed sparsely in the image;

  3. 3.

It is time consuming: It requires accurate visual inspection of multiple items
in the lot;

  4. 4.

It is subjective: Different users might identify different anchors based on
their expertise;

  5. 5.

The anchor selected by a human is very unlikely to be the best anchor usable
by the ROI selection algorithm.

Conditions (1), (2) and (3) are especially undesirable in a market which is
constantly struggling to shift towards plug-and-play solutions.

This paper introduces a new framework, named SAFFIRE,111Pronounced as
_sapphire_. which overcomes the aforementioned limitations:

  * •

The anchor is learned automatically from a set of training images;

  * •

The framework requires only straightforward supervision which only implies a
basic understanding of the task of interest. Namely, the user has to draw a
ROI around the content of interest. Of note, this is considerably easier than
identifying an anchor which can be uncorrelated with the specific task and
distributed anywhere in the image;

  * •

The computed anchor is the best pattern for the ROI selection algorithm.

The proposed framework presents also several computational advantages against
alternatives based on Deep Neural Networks (DNN), which are rapidly gaining
popularity:

  * •

SAFFIRE can be trained on a normal device within seconds (at most) while DNN-
based systems would take minutes to hours and dedicated expensive hardware;

  * •

SAFFIRE requires very few train samples (order of units) while DNN-based
systems would require thousands;

  * •

SAFFIRE is agnostic to the specific task while DNN-based solutions are task-
specific (e.g., text detectors [5]);

  * •

SAFFIRE can be trained incrementally, in that a new sample can be added
seamlessly to the train set to refine the anchor pattern, while DNN-based
solutions would require re-training the entire model (almost) from scratch;

  * •

SAFFIRE outputs a lightweight anchor pattern which can be found on a new
sample within tens of milliseconds. On the contrary, DNN-based systems output
heavy models and inference operates at a slow rate;

The paper proceeds as follows: Related work is presented in Section 2; Section
3 describes the SAFFIRE algorithm, both at train-time and at run-time; Section
4 presents the evaluation setup while Section 5 presents the results. Section
6 elaborates on the results and Section 7 concludes the work.

##  2 Related Work

At its core, SAFFIRE exploits an _object detection_ paradigm. Object detection
is a well known problem in computer vision. Currently, the established
approach is based on matching descriptors, i.e. compact representations of
local image features, such as corner points, segments or blobs, extracted from
the object of interest (see, e.g., [6]). Hereto, a range of techniques, such
as the Generalized Hough Transform (GHT) or Random Sample Consensus (RANSAC)
can be employed to retrieve instances of a _query_ pattern in a new image [6].

Yet, applying the object detection framework as presented above requires the
query pattern to be known. DNN-based approaches like YOLO [7] or Faster R-CNN
[8] may be used to implicitly learn such pattern, but they do not represent a
solution that fulfills the constraints of the addressed application: only very
few training images are available, the training time should be quite short and
the processing speed at run-time must be as high as possible with limited
computational resources. In this context, we propose an original technique
where the query pattern (i.e., the anchor) is learned autonomously from a set
of images, satisfying the presented constraints without the employment of DNN-
based solutions. Technically, this is done by finding the shortest path in a
directed acyclic graph (DAG) where each path identifies a possible model
candidate.

Moreover, different descriptors are known to perform differently on different
kinds of images. For instance, SIFT is arguably the most popular approach [6]
for textured objects, although a number of alternative solutions have been
also proposed, such as ORB [9] and SURF [1]. However, approaches based on
segments [3] and combination of segments [10] have been shown to be more
appropriate for texture-less objects.

The choice of the best descriptor for the specific dataset is typically the
result of a trial-and-error process. On the contrary, SAFFIRE implements a
selection logic where the features most suited to the specific data are
computed at train time.

Finally, classic feature matching is used at run-time to retrieve the anchor
model in a new sample and re-position the ROI. In the case of multiple
instances, SAFFIRE selects the solution where the image content inside the ROI
is the closest to that learned on the train set.

##  3 Material and Methods

The processing workflow of SAFFIRE at train time and test time is represented
in Figure 1 and 2, respectively. A description of the pipeline follows.

(a) Input: a set of labeled images. (b) Feature extraction and matching. (c)
Testing model candidates and graph construction. (d) Path finding.

(e) Output: model and model ROI.

Figure 1: Train mode: Build model and model ROI from train data. Please refer
to the main text for a detailed explanation.

###  3.1 Training Data and Reference Image

A set of $N_{\text{train}}$ annotated train images
$\\{I_{0},\ldots,I_{N_{\text{train}}-1}\\}$ is provided as input (Figure 1a).
One of such images, by convenience $I_{0}$, is adopted as _reference_ image.
Each image is annotated with an oriented rotated rectangular ROI $R_{i}$. Note
that orientation is in general relevant for the task (for instance, it can
disambiguate text reading direction).

We assume that all train images contain a distinctive _anchor_ pattern mixed
with a set of _noise_ patterns which are irrelevant for the identification of
the ROI. The ultimate goal of the training phase is therefore to retain the
anchor pattern while filtering irrelevant patterns out.

In the toy example in Figure 1a, the dashed rectangle denotes the ROI
enclosing the text to be read. The highlighted corner is the origin of the ROI
and indicates its orientation in that corners are ordered clockwise starting
from the origin. The star represents the anchor pattern since its position
relative to the ROI is stable. The triangle represents any other pattern in
the image which cannot work as an anchor for this task.

###  3.2 Feature Extraction

A set of visual _features_ is extracted from all train images (the red and
yellow dots in Figure 1b). Specifically, we use a well consolidated paradigm
for object detection based on matching sets of visual landmarks using their
local image descriptors [4]. We make use of multiple types of features to
account for textured and texture-less patterns and the logic to choose the
best feature type for the specific data is presented in Section 3.6.

With this in mind, we define the anchor pattern as the largest subset of
features such that:

  * •

It is present in all of the train images;

  * •

Its position in different train images can be related by a geometrical
transformation. Hereto, although general transformations can be used, we made
use of similarities (i.e., roto-translations with isotropic scaling) since
they were appropriate for the geometry of the experimental setup and relevant
to industrial applications;

  * •

Its position correlates best with the position of the ROI. More
quantitatively, the overlap between the ROI’s is maximal when they are
expressed in the anchor’s coordinate system (see Section 3.3).

The procedure used to derive such an anchor pattern is described in the
remaining of this Section.

The $j$-th feature on the $i$-th train image is denoted by $f_{i}^{j}$. A
feature is geometrically defined by a position $\mathbf{x}_{i}^{j}$, a
direction $\theta_{i}^{j}$ and a size $l_{i}^{j}$. The image content around a
feature is accounted for by a _descriptor_ $\mathbf{d}_{i}^{j}$. Descriptors
shall be used to match features between images, as denoted by the blue
connecting lines in Figure 1b.

###  3.3 The Training Graph

For each feature $f_{0}^{j}$ in the reference image, we find its 3 closest
matches in the $i$-th train image. We use 3 nearest neighbours to allow good
robustness to multiple instances and/or periodic patterns. We denote by
$f_{i}^{k}$ one of such matches. Matches with a large inter-descriptor
distance are discarded by applying Lowe’s ratio test [6]. Further filtering is
applied by using the following geometrical considerations:

  * •

Since in our setup scale is not expected to vary significantly, we exclude
matches whose scale factor $s=l_{0}^{j}/l_{i}^{k}$ is not within the interval
$[0.85,1.15]$;

  * •

Recalling our definition of an anchor pattern (see Section 3.2), we exclude
matches which are not useful to superimpose the ROI’s between samples. Hereto,
we use the (unique) spatial transform between $f_{0}^{j}$ and $f_{i}^{k}$ to
bring $R_{0}$ and $R_{i}$ to a common origin, and we discard the match if the
_intersection over union_ (IoU) between the transformed ROI’s is below 0.2.

Matches voting for the same spatial transform $T_{i}^{t}$ are then aggregated
into $N_{i}$ clusters, with $t=0,\ldots,N_{i}-1$. Clustering is done by
iterative application of the RANSAC algorithm [2]. At each iteration, RANSAC
is applied to find a transform $T_{i}^{t}$ and the inliers are removed from
the list of matches. The procedure is repeated incrementally until not enough
matches are left to estimate a transform.

As such, a directional acyclic graph (DAG), which we refer to as the _training
graph_ (see Figure 1c), is created with a number of layers equal to the number
of training images. Each layer contains $N_{i}$ nodes, one for each cluster
computed as above. Each node $n_{i}^{t}$ in the $i$-th layer contains:

  * •

A candidate transform $T_{i}^{t}$ between $I_{0}$ and $I_{i}$;

  * •

The indexes of the features in $I_{0}$ and $I_{i}$ voting for $T_{i}^{t}$.

The root layer in the training graph is obtained by matching the reference
image with itself. By construction, this gives a single node described by an
identity matrix and the indexes of all features in $I_{0}$.

###  3.4 Search of Optimal Path in the Training Graph

As mentioned, the goal is to find the largest subset of features in $I_{0}$
which can be used as a common coordinate system maximizing the overlapping
between the training ROI’s. The problem is solved by searching for the
shortest root-to-leaf path in the training graph (in red in Figure 1d). The
length (cost) of a path is defined as a function of two terms: the former
depends on the number of overlaps along the path, which is computed from the
intersection of the features voting for all the transforms in the path; the
latter depends on the alignment and overlap between the aligned ROI’s after
bringing them all to a common coordinate system by means of the
transformations along the path.

To measure ROI alignment, we use the _oriented IoU_ ($o\mathtt{IoU}$), a
modified standard _intersection over union_ (IoU) accounting for ROI
orientation. Indeed, the classical IoU between two rectangular shapes does not
distinguish $None$ flips around the center. Specifically, for a node on the
$n$-th layer, it is computed by using the set of spatial transformations on
the intermediate path to first bring the $n$ ROI’s to a common coordinate
system. The proposed oriented IoU is defined in such a way that its value for
two flipped and perfectly overlapped rectangles is $0$, while their standard
IoU would be $1$.

Having defined the cost of a path, we then use a tree-search algorithm to
retrieve the optimal anchor pattern, the one corresponding to the path of
least cost.

###  3.5 The Anchor

The best path is used to compute the final anchor model, which will be used at
run-time to position the ROI on a new sample (see Figure 1e). Such model is
composed of four parts: i) a set of _model features_ ; ii) the associated
_model descriptors_ , iii) a _ROI model_ $R_{m}$ and iv) a _ROI descriptor_.

_Model features_ are the subset of $\\{f_{0}\\}$ which vote for all the
transforms in the best path. For what concerns the _model descriptors_ ,
although one could use the descriptors computed on the reference image alone,
this would result in a high bias to the specific choice of the reference
image. As such, we defined descriptors which also account for the statistical
variation of image content between train samples.

The _ROI descriptor_ is used to keep track of the image content inside the
ROI. It will be used to discriminate between multiple instances of the anchor
model at run time, as described in Section 3.7. Let’s note that a _global_
image descriptor, i.e. accounting for average image properties such as mean
image value or average contrast, is best suited for machine vision
applications. Indeed, fine image details can change between samples (for
instance, the content of a printed text). As for the model descriptors, the
ROI descriptor is not tailored to a specific train image, but is rather based
on the statistical distribution of image content between images.

###  3.6 Selection of the Best Feature Type

Since different features are best suited for different patterns (e.g.,
textured vs. texture-less objects), SAFFIRE employs an optimized feature
selection strategy which chooses the feature-descriptor pair which performs
best on the specific data. Both accuracy and speed are used as a selection
criterion.

###  3.7 Testing

The trained model is used at run-time to position the ROI in a new unseen
sample (see Figure 2a) with a two-step process.

(a) Input: a test image. (b) Search model in image. (c) Remove bad matches
with ROI content. (d) Output: ROI on the new image.

Figure 2: Test mode: Find the trained model on unseen data. Please refer to
the main text for a detailed explanation.

First, a standard object detection paradigm is used to find instances of the
trained model in the new image (Figure 2b). Namely, features and descriptors
of the selected type (see Section 3.6) are extracted on the test image and
matched to the anchor pattern model (see Section 3.3). Using the GHT, as in
[6], feature correspondences are turned into votes to an accumulation array
(AA) with peaks in the AA representing possible instances of the anchor model
in the test image. Hereto, a peak pruning procedure, based on the relative
number of votes in each peak, is applied to keep only the most significant
matches. For each valid instance of the anchor model, the corresponding
spatial transform is used to position the ROI model in the new sample, as
described in see Section 3.5.

With regard to Figure 2, the test image contains two instances of the star
anchor model, which presents circular symmetry. This could result in many
matches (hence, ROI positions) with equal score, but only that containing the
letter E is relevant. Such matches are displayed in Figure 2b as yellow dashed
boxes.

The following step consists therefore in discarding non-relevant matches (see
Figure 2c) by computing the content descriptor for each candidate ROI (see
Section 3.5). The weighted $\ell$-2 distance to the model ROI descriptor
$\mathbf{d}_{\mathrm{model}}$ is:

$d(\mathbf{d}_{\mathrm{test}},\mathbf{d}_{\mathrm{model}},\mathbf{w})=\sqrt{\sum_{i=0}^{47}w_{i}\left(\mathbf{d}_{\mathrm{test}}(i)-\mathbf{d}_{\mathrm{model}}(i)\right)^{2}}$
(1)

where $\mathbf{d}(i)$ denotes the $i$-th bin of $\mathbf{d}$. A weight
$w_{i}\in[0,1]$ assigns highest (respectively, lowest) weight to bins with the
lowest (respectively, highest) variance. The ROI with the lowest distance is
the one used to analyze the sample at run time, as depicted in Figure 2c and
Figure 2d.

##  4 Evaluation Setup

###  4.1 Evaluation Datasets

The datasets used to validate SAFFIRE are presented in Figure 3 with 2 samples
from each alongside the total number of images available for each one. The
datasets are not suited for a single task, but many can be applied: the main
task is indeed reading the text to ensure it has been printed correctly, but
also measurements (see Figure 3i) and flaw detection (see Figure 3c and Figure
3e) can be carried out.

(a) StarCart, toy dataset.

(b) Cans, 87 images, larger size.

(c) Curry, 16 images, larger size.

(d) Tea, 31 images.

(e) Oil, 10 images.

(f) Pasta, 16 images.

(g) Juice, 139 images.

(h) Ciocorì, 35 images.

(i) LeadFrame, 11 images.

(j) Caprice, 29 images.

(k) Beer, 86 images.

Figure 3: Evaluation datasets: the ROI is depicted in red and the highlighted
corner is the origin. Real datasets also include the total number of images.

The images have been shuffled upon acquisition and the first three have been
used for the training phase. The results presented in Section 5 are computed
on the whole datasets, including the images used in training.

###  4.2 Evaluation Metrics

The search capabilities of the trained models are evaluated by comparing the
computed ROI with the ground truth one by means of $o\mathtt{IoU}$ between the
two. For each model the individual $o\mathtt{IoU}$’s are aggregated in a
single plot showing the distribution of $o\mathtt{IoU}$ over the samples in
the dataset by using a box plot.

The box in each box plot extends from the first $q_{1}$ to the third quartile
$q_{3}$ of the distribution and, inside it, the median is highlighted. Outside
the box, the whiskers extend on the left to the lowest value in the
distribution greater than $q_{1}-1.5\cdot\mathrm{IQR}$, with the interquartile
range (IQR) defined as $q_{3}-q_{1}$; similarly on the right to the highest
value less than $q_{3}+1.5\cdot\mathrm{IQR}$. Samples outside the range of the
whiskers are considered outliers and plotted as single points.

##  5 Results

###  5.1 Noise Patterns and Filtering

As detailed in Section 3.4, the path cost depends both on the number of
overlaps of votes along the path and how well the ROI’s are aligned, in the
following we will show the importance of the second term.

(a) Only overlaps.

(b) Overlaps and $o\mathtt{IoU}$.

Figure 4: Training over the StarCart dataset: the thick green lines are the
segments of the model, the lighter ones delimit the ROI of said model.

Considering the StarCart dataset (see Figure 3a), it can be seen that both the
cart and the star have a somewhat static position with respect to the ROI,
however the orientation changes in the case of the cart. We applied the
training procedure using only segment based features due to the nature of the
shapes, and report the results in Figure 4. As depicted in Figure 4a,
considering only the overlaps results in taking the cart as the learned model
due to the fact that its shape is formed by many more segments than the star
is. Adding the term based on ROI $o\mathtt{IoU}$ results instead in the
desired model (see Figure 4b).

###  5.2 Location Accuracy

The search capabilities of SAFFIRE are evaluated over several different
datasets (see Figure 3) using $o\mathtt{IoU}$ as metric (see Section 4.2) and
reported in Figure 5.

Figure 5: Distributions of $o\mathtt{IoU}$ of computed and ground truth ROI
over several datasets.

###  5.3 Computation Time

SAFFIRE has been implement to run on a Datalogic Memor
20,222https://www.datalogic.com/eng/retail-manufacturing-transportation-
logistics-healthcare/mobile-computers/memor-20-pd-869.html powered by a
Snapdragon 600 CPU. On this platform the train process takes just a few
seconds and the search tens of milliseconds (with a few exceptions) as
reported in Table 1. Search times are averaged over all images in the dataset
and train times over 5 executions.

Table 1: Average times for training and search over several datasets.

Tea Oil Pasta Juice Ciocorì LeadFrame Caprice Beer Train Time $1.30$
$\mathrm{s}$ $983.33$ $\mathrm{ms}$ $8.63$ $\mathrm{s}$ $3.13$ $\mathrm{s}$
$3.83$ $\mathrm{s}$ $896.10$ $\mathrm{ms}$ $1.24$ $\mathrm{s}$ $2.78$
$\mathrm{s}$ Features A A B B A A A A Search GHT $8.62$ $\mathrm{ms}$ $2.90$
$\mathrm{ms}$ $744.38$ $\mathrm{\SIUnitSymbolMicro s}$ $1.98$ $\mathrm{ms}$
$774.97$ $\mathrm{\SIUnitSymbolMicro s}$ $4.89$ $\mathrm{ms}$ $1.11$
$\mathrm{ms}$ $45.05$ $\mathrm{ms}$ Total $37.05$ $\mathrm{ms}$ $36.20$
$\mathrm{ms}$ $106.21$ $\mathrm{ms}$ $99.16$ $\mathrm{ms}$ $35.28$
$\mathrm{ms}$ $38.89$ $\mathrm{ms}$ $32.32$ $\mathrm{ms}$ $86.70$
$\mathrm{ms}$

During development, SAFFIRE has been tested also on a personal computer
powered by an Intel Core i5 7267U CPU. The results over a different selection
of datasets, which were mainly used during development, are reported in Table
2.

Table 2: Average times for training and search over a few datasets on the development platform. |  | Cans | Curry | Beer
---|---|---|---|---
Train | Time |  $4.92$ $\mathrm{s}$ |  $8.28$ $\mathrm{s}$ |  $3.99$ $\mathrm{s}$
Features | B | B | A
Search | GHT |  $19.40$ $\mathrm{ms}$ |  $41.59$ $\mathrm{ms}$ |  $26.53$ $\mathrm{ms}$
Total |  $126.76$ $\mathrm{ms}$ |  $216.23$ $\mathrm{ms}$ |  $35.95$ $\mathrm{ms}$

##  6 Discussions

The results in Figure 5 shows that SAFFIRE can obtain an high accuracy in
identifying the ROI. From our experiments and by means of a qualitative
evaluation of results, we found that even with $o\mathtt{IoU}$ of about 0.85,
localization of the ROI can be considered accurate. This is caused by using as
model a ROI slightly larger than those used for training and as ground truth,
due to how it is constructed. We also note some failures for Ciocorì denoted
by the points around 0, likely caused by the reflective surface of the
packaging (see Figure 3h).

Table 1 indeed shows that the search time is fast, but by isolating the
identification of the ROI, by means of the GHT and the ROI content descriptor
to choose the right candidate, it emerges how our approach is extremely fast
and most of the time is taken by feature extraction and descriptor matching.
Indeed, the fastest features are preferred as SAFFIRE prefers choosing the
ones with the least computational time even if this means sacrificing a bit of
accuracy (see Section 3.6). The results in Table 2 are provided as a mean of
comparison of running SAFFIRE on an non-embedded device and, as such, they
lack some optimizations that led to the final results.

We also note that the SAFFIRE framework is extremely general and independent
on the specific technical choices presented in this work:

  * •

Any feature descriptor can be used;

  * •

Other techniques can be used instead of RANSAC to compute the nodes of the
training graph, such as the GHT;

  * •

Arbitrary spatial transforms can be employed depending on the geometry of the
problem;

  * •

Any graph-search algorithm can be used to find the optimal model;

  * •

Many image descriptors can be used to encode the ROI image content.

  * •

The rectangular ROI in this study is not a limitation and can be replaced with
arbitrary shapes with a defined notion of direction, in order to simplify the
analysis task following the ROI identification, e.g., to force text to be
upright, with this definition even circles can be used.

The algorithm can also be adapted to work in a fully unsupervised fashion not
requiring the provision of a ROI. This is simply achieved by adjusting the
path cost in the graph to depend only on the number of overlaps, thus
discarding the alignment of the ROI’s. This way, SAFFIRE will look for the
most stable pattern in the image sequence with the highest number of features.
For instance, this could be sufficient to align images of similar items
acquired in different positions, but care must be taken as it can results in
cases like that presented in Section 5.1, in which relying only on feature
overlaps results in a bad alignment of ROI’s.

##  7 Conclusions

We have introduced a new algorithm, called SAFFIRE, for unsupervised
identification of a stable anchor pattern from a set of images. Such an anchor
is used to remove pose variations between samples before further processing.
Data normalization is a fundamental pre-processing step for many computer
vision and machine learning tasks.

The use of SAFFIRE was shown in the context of machine vision systems for
industrial automation, where it represents a disruptive element enabling the
implementation of an automatic ROI finding mechanism in an unsupervised
fashion.

## Intellectual Property

Patent application pending.

## References

  * [1] Bay, H., Tuytelaars, T., Van Gool, L.: SURF: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) Computer Vision – ECCV 2006. pp. 404–417. Lecture Notes in Computer Science, Springer (2006). https://doi.org/10.1007/11744023_32
  * [2] Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). https://doi.org/10.1145/358669.358692
  * [3] Grompone von Gioi, R., Jakubowicz, J., Morel, J.M., Randall, G.: LSD: a fast line segment detector with a false detection control. IEEE transactions on pattern analysis and machine intelligence 32(4), 722–732 (2010). https://doi.org/10.1109/TPAMI.2008.300
  * [4] Krig, S.: Interest Point Detector and Feature Descriptor Survey, pp. 217–282. Apress, Berkeley, CA (2014). https://doi.org/10.1007/978-1-4302-5930-5_6
  * [5] Long, S., He, X., Yao, C.: Scene text detection and recognition: The deep learning era (2018)
  * [6] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94
  * [7] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 779–788 (2016). https://doi.org/10.1109/CVPR.2016.91
  * [8] Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6), 1137–1149 (2017). https://doi.org/10.1109/TPAMI.2016.2577031
  * [9] Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: An efficient alternative to SIFT or SURF. In: 2011 International Conference on Computer Vision. pp. 2564–2571 (2011). https://doi.org/10.1109/ICCV.2011.6126544
  * [10] Tombari, F., Franchi, A., Di Stefano, L.: BOLD features to detect texture-less objects. In: 2013 IEEE International Conference on Computer Vision. pp. 1265–1272. IEEE (2013). https://doi.org/10.1109/ICCV.2013.160


# An Empirical Study of Bugs in Quantum Machine Learning Frameworks

Pengzhan Zhao, Xiongfei Wu, Junjie Luo, Zhuo Li, Jianjun Zhao1
<EMAIL_ADDRESS>Graduate School and Faculty of Information Science
and Electrical Engineering
Kyushu University, Japan

###### Abstract

Quantum computing has emerged as a promising domain for the machine learning
(ML) area, offering significant computational advantages over classical
counterparts. With the growing interest in quantum machine learning (QML),
ensuring the correctness and robustness of software platforms to develop such
QML programs is critical. A necessary step for ensuring the reliability of
such platforms is to understand the bugs they typically suffer from. To
address this need, this paper presents the first comprehensive study of bugs
in QML frameworks. We inspect 391 real-world bugs collected from 22 open-
source repositories of nine popular QML frameworks. We find that 1) 28% of the
bugs are quantum-specific, such as erroneous unitary matrix implementation,
calling for dedicated approaches to find and prevent them; 2) We manually
distilled a taxonomy of five symptoms and nine root cause of bugs in QML
platforms; 3) We summarized four critical challenges for QML framework
developers. The study results provide researchers with insights into how to
ensure QML framework quality and present several actionable suggestions for
QML framework developers to improve their code quality.

###### Index Terms:

quantum machine learning, quantum software testing, quantum program debugging,
empirical study

##  I Introduction

Quantum computing has been making immense progress due to increasing computer
power and algorithmic advances [5, 8]. Tremendous efforts from industry and
academia have greatly stimulated the evolution of this area. Although fault-
tolerant quantum computers will likely not be available shortly, recent
research on quantum machine learning has revealed the potential of quantum
computers to outperform classical computers on machine learning tasks [3].

With the rapidly growing complexity of quantum programs, it is decisive to
alleviate the efforts in programming such quantum programs. _Quantum computing
framework_ provides the essential interface (i.e., quantum programming
language), compiler, and execution environment for quantum programmers to run
quantum programs on a quantum computer or a simulator. Several quantum
programming frameworks are available for quantum programmers, such as Qiskit
[1] by IBM, Q# [21] by Microsoft, and Cirq [9] by Google, allowing researchers
and developers to implement and experiment with various quantum programs
quickly. Furthermore, stimulated by the prosperity of the classical ML
community and the potential of QML techniques, a number of QML frameworks are
proposed, such as Torch Quantum [23] and PennyLane [2]. Given the importance
of this rapidly involving field, ensuring the correctness of the underlying
QML frameworks deserves high priority. Various approaches exist to prevent and
find bugs in quantum computing frameworks, e.g., for bug characteristics [18,
28] or testing [19, 24]. However, little attention has been received to ensure
the quality of QML frameworks. One efficient approach to help prevent and
detect bugs is understanding and characterizing bugs that exist in the wild
[18]. However, there currently is no detailed study of bugs in QML frameworks.

To bridge this gap, we present the first empirical study to characterize bugs
in QML frameworks. We collect and inspect a set of 391 real-world bugs from 22
open-source projects, including highly popular repositories such as Torch
Quantum and PennyLane. We aim to answer three fundamental questions that
remain unclear: _How many of these bugs are specific to quantum computing,
where do these bugs occur in QML platforms, and how do these bugs manifest?_
Furthermore, we identify key challenges QML developers face when developing
QML platforms. The key findings of this study include the following:

  * •

28% of the bugs are quantum-specific, and these quantum-related bugs are
primarily associated with qubit manipulation, such as erroneous unitary matrix
implementations, calling for dedicated approaches to find and prevent them.

  * •

While the most prevalent symptom is program crash, quantum-related bugs tend
to produce function errors, which are challenging to detect. Moreover, QML
frameworks are more prone to environmental factors, such as framework version
and device compatibility, which may cause warning errors.

  * •

Root cause analysis reveals that algorithmic and logic issues accounted for
the largest proportion, with inconsistencies stemming from version lags and
updated features as the second most common cause.

  * •

Based on the bug patterns, we present four key challenges that QML framework
developers should address. For instance, ensuring cross-device execution of
QML programs, given the multitude of devices that support them (e.g., GPUs,
QVMs, QPUs, and simulators).

The results of this study provide insights for both researchers and
developers. Researchers may benefit from insights into the bugs in this domain
and can propose new approaches to tackle the problems they cause. QML
framework developers can learn the bug patterns and be careful when facing the
key challenges we propose. To better help other research in this domain, we
publicly share our QML framework bug dataset, including annotation
intermediate procedure and results.

In summary, this paper contributes the following:

  * •

The first study of 391 real-world bugs in 22 popular QML frameworks.

  * •

We manually inspected 391 bugs and summarized five symptoms and nine root
causes of bugs to help characterize bugs in QML frameworks.

  * •

Insights of challenges developers face when developing QML frameworks.

  * •

A publicly available dataset of labeled, real-world bugs to support future
research in testing QML frameworks.

The rest of the paper is organized as follows. Section II provides some basics
of quantum programming and quantum machine learning. Section III describes our
methodology for collecting and filtering bugs. Section IV presents the
empirical results of our study. Section V reviews our threats of validity.
Section VI discusses related work, and Section VII finally concludes this
paper.

##  II Background

This section briefly introduces some basic concepts of quantum computing as
well as quantum machine learning.

###  II-A Quantum Computing

A quantum bit (qubit) is the analog of one classical bit but has many
different properties. A classical bit, like a coin, has only two states, 0 and
1, while a qubit can be in a continuum of states between $\Ket{0}$ and
$\Ket{1}$ in which the $\Ket{}$ notation is called Dirac notation. We can
represent a qubit mathematically as $\Ket{\psi}=\alpha\Ket{0}+\beta\Ket{1}$
where $|\alpha|^{2}+|\beta|^{2}=1$ and the numbers $\alpha$ and $\beta$ are
complex numbers. The states $\Ket{0}$ and $\Ket{1}$ are computational basis
states. Unlike classical bits, we cannot examine a qubit directly to get the
values of $\alpha$ and $\beta$. Instead, we measure a qubit to obtain either 0
with probability $|\alpha|^{2}$ or 1 with probability $|\beta|^{2}$.

Quantum gates are used for quantum computation, which means manipulating
quantum information. Some basic quantum gates are listed as follows:

  * •

Quantum NOT gate takes the state $\Ket{\psi}=\alpha\Ket{0}+\beta\Ket{1}$ into
the state $\Ket{\psi}=\alpha\Ket{1}+\beta\Ket{0}$. We can use a matrix to
represent this operation:

$\displaystyle X=\begin{bmatrix}0&1\\\ 1&0\end{bmatrix}$

  * •

The Z gate can be expressed as

$\displaystyle Z=\begin{bmatrix}1&0\\\ 0&-1\end{bmatrix}$

From the matrix, we know the Z gate leaves the $\Ket{0}$ unchanged and changes
the sign of $\Ket{1}$.

  * •

The Hadamard gate turns the $\Ket{0}$ into $(\Ket{0}+\Ket{1})/\sqrt{2}$ and
turns the $\Ket{1}$ into $(\Ket{0}-\Ket{1})/\sqrt{2}$. The matrix form of the
Hadamard gate is

$\displaystyle H=\frac{1}{\sqrt{2}}\begin{bmatrix}1&1\\\ 1&-1\end{bmatrix}$

All the matrices are unitary ones. Besides these single-qubit gates, there are
multiple qubit gates, such as the Controlled-NOT gate (CNOT gate). This gate
has two input qubits, the control and target qubits. If the control qubit is
0, then the target qubit remains unchanged. If the control qubit is 1, then
the target qubit is flipped. We can express the behavior of the CNOT gate as
$\Ket{A,B}\rightarrow\Ket{A,B\oplus A}$.

Quantum circuits are models of all kinds of quantum processes. We can build
quantum circuits with quantum gates and use wires to connect the components in
quantum circuits. These wires can represent the passage of time or a physical
particle moving from one position to another. Another essential operation in
quantum circuits is measurement. Measurement operation observes a single qubit
and obtains a classic bit with a certain probability. Nielsen’s book[17] has a
more detailed explanation of quantum computation.

###  II-B Quantum Machine Learning

Machine learning is based on statistics and extracts the appropriate patterns
by analyzing existing data so that the patterns can be used to make
predictions about new data. Machine learning is divided into three main
categories, supervised learning, unsupervised learning, and reinforcement
learning. Among them, supervised learning and unsupervised learning are mainly
distinguished by the input dataset. The supervised learning dataset contains
data and corresponding labels and is concerned with the mapping rules from
data to labels. On the other hand, the unsupervised learning dataset does not
have labels and performs the task of classification by extracting features
common to the data. Reinforcement learning focuses on the best decision for a
problem by enabling an agent to learn strategies as it interacts with its
environment to maximize the reward or achieve the appropriate goal. Machine
learning has been widely adopted in various fields, including healthcare,
education, manufacturing, etc [15].

Quantum machine learning is a rapidly growing field combining quantum
algorithms and machine learning principles. Quantum algorithms have been shown
to outperform the best classical algorithms for specific problems by
exploiting quantum properties such as superposition states, coherence, and
entanglement effects, which are known as quantum speedup [3]. It has led to
the exploration of applying quantum algorithms to machine learning with the
expectation that algorithms superior to classical machine learning can be
discovered. Although there is still a lack of sufficiently powerful quantum
computer hardware to support practical applications of quantum machine
learning algorithms, some quantum machine learning ideas have been proposed
and validated, such as the quantum algorithm for linear systems of equations
(also called the HHL algorithm) [10], the Variational Quantum Eigensolver
(VQE) [20], and the Quantum Convolutional Neural Network (QCNN) [7], etc.

Overall, the field of quantum machine learning is still in its early stages.
However, it holds great potential for pushing the boundaries of traditional
machine learning research and solving complex problems that are difficult or
even impossible to tackle with classical methods.

##  III Methodology

In this section, we describe how we conducted our study by formulating
research questions, how we selected the study object and data source, how we
constructed the dataset, and how we performed the study.

###  III-A Research Questions

This study is driven by the following three research questions:

  * •

RQ1:Where the bugs occur and how many of the bugs in QML platforms are
specific to quantum computing? This question represents our first purpose,
which is to understand the relationship between bugs in the QML frameworks and
traditional ML frameworks.

  * •

RQ2:What are the symptoms and causes of bugs in QML platforms? This question
is the core and foundation of this study, and understanding the symptoms and
causes of the platform can help better detect and fix bugs. In addition, it
also helps to answer RQ3.

  * •

RQ3:What are the areas of conflict that developers need to handle when
developing QML frameworks? Developers can also face many challenges when
refining the QML platform. Especially for QML framework developers, not only
do they need to implement ML functionality, but they also need to implement
quantum operations.

###  III-B Dataset

The main objective of this paper is to study bugs in quantum machine learning
frameworks. Following previous works [18, 28], we hence study not only a
single or a few projects but 22 open-source projects in total. Furthermore, we
decided to focus on open-source GitHub projects, as these projects allow us to
apply the same process procedure to each project. To select projects, we first
utilize the list of quantum computing frameworks used in [18] and check
whether these projects contain a corresponding quantum machine learning
framework. If so, the corresponding quantum machine learning framework is
considered to be added to the initial list. Then we extend the initial list by
inspecting the search results of “quantum machine learning” and “quantum
neural network” on Google Scholar and GitHub, which contain a corresponding
GitHub repository. As a result, 22 popular quantum machine learning frameworks
are selected. The selected projects cover all the quantum machine learning
techniques introduced in Section II-B (e.g., QCNN). Due to limited space,
detailed information about the selected projects can be found at our anonymous
website111https://anonymous.4open.science/r/QML-platform-bugs-0E90/.

After selecting QML frameworks, we crawled all the issue reports from the 22
repositories above, consisting of 1,591 issue reports. Then we manually
inspected these reports to filter out the candidate bug reports to prevent
introducing noise not about bugs in QML frameworks. For instance, sometimes
users may think of a specific behavior as a bug in QML frameworks, which may
be an API misuse; some posts were just about typos in the tutorial documents;
and some posts indeed discussed bugs but did not yet obtain a correct answer,
which could not be used to help characterize bugs in QML frameworks. During
the inspection process, we kept track of all the materials (e.g., external
links, commit messages, etc.) in the issue report to obtain a complete
picture. Overall, we identify 391 bugs to study and answer our RQs.

###  III-C Understanding and Labeling Properties of Bugs

Answering our research questions requires a solid understanding of the
identified bugs. This section describes how we inspect and label the bugs with
various properties.

####  III-C1 Traditional vs. Quantum-Specific

To address RQ1, we need to classify each bug as either traditional or quantum-
specific. Following previous work [18], we consider a bug quantum-specific if
the mistake involved is related to quantum-specific concepts. Specifically,
understanding these bugs requires knowledge of the quantum programming domain.
Otherwise, we consider all other bugs as traditional, which means that these
bugs may occur in traditional ML frameworks and do not need quantum-specific
knowledge to be fixed.

####  III-C2 Identifying Symptoms and Root Causes

Addressing RQ2 and RQ3 requires us to label all bugs concerning two
dimensions: symptom and their corresponding root cause. Following previous ML
bug studies [25], two authors started with the classification schema used for
labeling from the existing general ML bug studies [22, 25]. They adapted it by
appending new ones and excluding unrelated ones. To carefully annotate bugs,
they read all the bug reports separately, including the title, description,
comments, and the corresponding code changes.

In particular, the symptom of a bug is determined if the developer explicitly
reported the symptom in the context of the reports. The root cause of a bug
was inferred from the discussion as well as the corresponding code changes.
The root cause of a bug will be labeled as “Unclear” if there are no explicit
answers or code patches of a bug to avoid introducing bias.

In summary, we carefully inspected 391 bug reports and distilled a total of
five symptoms and nine root causes. A third author was involved in resolving
disagreements during the labeling procedure. The Cohen’s Kappa coefficient for
symptoms and root causes was 0.73 and 0.7, which implies substantial
agreement. The complete inspection and labeling procedure requires about 500
person-hours.

##  IV Empirical Results

This section details the results of our study as well as the answers to our
three research questions.

###  IV-A RQ1.Locations and Quantum-specific bugs

A quantum platform bug occurs in the source code, which belongs to the various
components of the QML platform. The components and names of the QML platform
vary due to the different frameworks. We group the components into the
following categories based on statistical bugs and a comparison of the various
platforms:

  1. 1.

Quantum basic. Quantum basic is mainly concerned with the most basic functions
for qubit states. In particular, most of these components for building basic
operations on qubits are often in the basic library of one framework such as
Cirq222https://github.com/quantumlib/Cirq or Qiskit-
Terra333https://github.com/Qiskit/qiskit-terra rather than in QML libraries.
Therefore, most bugs found in the quantum basic-related files exist in the
pennylane/measurements and pennylane/pauli directory, etc.

  2. 2.

Kernel. This component implements the quantum kernel matrices that can be used
to implement classical machine learning algorithms. All bugs of this location
type are contained in the quantum_kernel.py file and the pennylane/kernels
directory.

  3. 3.

Drawer. This is a graphical feature to visualize quantum circuits. As with
quantum basic, this is a feature unique to Pennylane. Some bugs are located in
file pennylane/circuit_graph.py or directory pennylane/drawer.

  4. 4.

Template. This component mainly provides templates that can be used in some
quantum nodes, such as quantum state preparation and layers for transforming
quantum states. We locate this component in
qiskit_machine_learning/neural_networks and pennylane/templates directory.

  5. 5.

Interface. QML programs rely heavily on other equipment. And many of the
interfaces, led by Pennylane, are not only hardware-oriented but also have
simulator backends for other platforms. Many bugs lead to the programs failing
to run on GPUs. An example can be seen in issue
#1705444https://github.com/PennyLaneAI/pennylane/pull/1705. This component is
located in pennylane/devices and pennylane/interfaces.

  6. 6.

Algorithm. Many QML algorithms (e.g. VQE, VQC, and QAOA) are integrated with
various frameworks. This component can be located in pennylane/fourier and
qiskit_machine_learning/algorithms etc.

  7. 7.

Transform. The pennylane/transforms directory provides the functions to
convert and decompose the large code base and supply to help achieve the
compilation of quantum circuits.

  8. 8.

Operation. This component contains the core quantum operations, mainly in the
form of unitary operations. It is different from quantum basic, which mainly
focuses on the composition of quantum elements. We can locate the ops in
directory tensorflow_quantum/core/ops/ and directory pennylane/ops.

Figure 1: Structure of QML programs

Figure 1 shows a simple structure of QML programs. Here is a disclaimer: Since
all quantum platforms at this stage are based on classical programming
languages to simulate the implementation of qubit state transitions, all bug
symptoms and causes fall under the category of classical programming. If a bug
occurs at the quantum part, we categorize it as a quantum-specific bug. An
example can be seen in Pennylane
#3100555https://github.com/PennyLaneAI/pennylane/issues/3100, which shows a
function error of multiple-qubits gates. And the cause of this bug is
incorrect logic being used. Figure 2 shows a part of the modification.

        -       if op.num_wires != 2:
        +       if len(op.wires) == 1:
                    gates.append(op)
                    list_op_copy.pop(0)
                    continue

Figure 2:  Part of modified code

As a result, a quantum-specific bug is reflected in the elements that
implement the quantum circuit part, e.g., the encoding of qubits, quantum
gates for conversing qubit operations, and the function to implement
measurement, etc.

Figure 3: The proportion of quantum-related bugs and traditional bugs in each
symptom.

Finding 1. There are 108 quantum-related bugs, accounting for 28% of the total
bug count of 391, while the remaining 72% of bugs occur on the classical part.
Quantum-specific bugs are all reflected in the functionality of qubit
manipulation, including state vector misrepresentation of qubits and error
implementation of the unitary matrix, etc.

Figure 4: Distribution of root causes in each symptom.

###  IV-B RQ2.Symptoms and root causes

We classify the symptoms into the following categories:

  1. 1.

Function error (30.5%). A program that does not implement the appropriate
function is defined as a Function error, e.g., an output error or a program
that does not have an appropriate return value.

  2. 2.

Exception (63.2%). A program interrupts and throws exceptions, which we
consider an exception.

  3. 3.

Stuck (1.1%). A stuck means a program keeps running without any responses.

  4. 4.

Run time error (1.8%). This condition manifests as a program taking much
longer to execute than expected.

  5. 5.

Warning error (3.4%). In this case, the execution of the program is usually
not affected, but a warning message will be displayed, such as an outdated
version, an excuse for deprecation, etc.

        -  for (int i = 0; i < programs->size(); i++){
        +  for (size_t i = 0; i < programs->size(); i++){
                Program& program = (*programs)[i];
                Status status = Status::OK();
                ...

Figure 5:  Example of type error

Figure 3 is an extension of Finding 1. All the symptoms are raised within the
quantum code and the traditional part. And the number of traditional bugs has
more weight, especially for the symptom Exception.

The causes of bugs are divided into the following categories:

  1. 1.

Type error (8.0%). Bugs caused by incorrect type definitions are classified in
this category. Issue #30666https://github.com/tensorflow/quantum/issues/30 is
an example caused by a type error.

  2. 2.

Processing (7.1%). We judge these bugs based on the quantum preparation part
of the quantum circuit and the classical part related to data processing.

  3. 3.

Inconsistency (18.6%.) This cause includes API and version changes which lead
to bugs.

  4. 4.

Algorithm and logic (33.8%). Algorithm error and Logic error are not well
split. For example, many bugs result from missing functions, and we organize
the problem descriptions and fix files with the idea that the functions result
in illogical and incomplete algorithms.

  5. 5.

Reference error (3.9%). We classify such bugs based on incorrect package
import or failure to find libraries.

  6. 6.

Corner case (6.6%).If a bug occurs in a particular situation, we classify it
to Corner case. #1916777https://github.com/PennyLaneAI/pennylane/issues/1916
is an example. The bug occurs when using qml.draw() to draw a template.

  7. 7.

Other library (6.3%). If a bug occurs on other libraries, we consider the bug
is not relevant to the local framework. An example can be seen in issue
#557888https://github.com/PennyLaneAI/pennylane/issues/557, the bug whether
worth fixing mainly depends on the restrictions on QNode [16] signatures.

  8. 8.

Unclear (7.8%). There is no description and fixing files of one bug, nor is
the discussion of the root cause as well as the fixes unclear.

  9. 9.

Configuration error (7.9%). We put bugs caused by misconfigurations into this
category. These bugs are mainly in the form of compilation and plug-in
problems which cause the circuit to be broken or the device to be unusable,
etc.

Finding 2. Regarding symptoms, the most common situation is Exception, which
means the programs cannot run smoothly. The second is Function error. Due to
functional defects resulting in the output error is difficult to be found. The
lowest percentage is stuck and Run time error. Besides, Warning error would be
affected by a small human factor because the environment, framework version,
and compatibility between devices should be considered.

Finding 3. For root causes, Algorithmic and logic accounts for the most
significant proportion, indicating that the program is not yet perfect in
terms of functional implementation. There are some missing functions worth
noting. Next is Inconsitency, ranked second; in addition to bugs caused by
version lag, the main thing is the updated version and rewritten features
would have bugs instead. Other causes of bugs are more minor in proportion and
evenly distributed.

###  IV-C RQ3.Common challenges for developers

Figure 6: Distribution of symptoms in each root cause.

Figure 4 shows each symptom’s distribution of root causes. The number in the
figure shows the total number of bugs found. Of all the causes we concluded,
only reference errors do not result in a function error. And clearly, for
quantum platforms, algorithms and logic errors easily cause the program to
crash or get functional problems. Another major cause of Exception is
Inconsistency, which indicates that API changes and version updates are most
likely to cause program exceptions. Besides, all the symptoms have
Inconsistency as a cause.

On the other hand, Figure 6 shows the distribution of symptoms in each root
cause. Exception has the largest share of any causal factor. This indicates
that any of the root causes can easily lead to a program throwing exceptions.
Function error also accounts for a relatively large proportion, indicating
that there are also many causes of functional errors. Stuck, Warning error,
and Run time error have less weight in the root causes.

Finding 4. As a result, we summarized the following areas of conflict for
developers:

  * •

We collected 22 repositories from nine popular organizations on GitHub.
Considering the existing devices (e.g., GPUs, QVMs, QPUs, simulators) which
support executing QML programs, the platform needs to provide a large number
of plugins to enable cross-device execution of programs. Therefore, developers
need to handle the cooperation between platforms and devices to facilitate
user implementation of QML programs.

  * •

Current quantum platforms are under rapid evolution, which has a large number
of undergoing and future code refactoring and compatibility changes. However,
these changes are often error-prone and may cause performance degradation.
Developers should pay more attention when making code changes to prevent
introducing new bugs.

  * •

Exception is the most common phenomenon, which means ensuring program
execution is a fundamental area of conflict. In addition, the cause of many
bugs can be troublesome to find as the unclear symptoms accounted for 7.8%,
sometimes requiring user assistance.

  * •

Considering the Algorithm and logic accounts for the largest share of the root
cause, the consequence is many functions are incorrectly implemented. And
fixing missing or defective features sometimes involves multiple source files.
Therefore, fixing these bugs requires developers to have knowledge and skills
related to quantum machine learning.

##  V Threats to Validity

###  V-A Internal Validity

There are some internal validities in this work. First, we only look for bugs
on GitHub, and we can’t judge the audience or quality of a QML platform based
only on the number of bugs we collect on GitHub. In our future work, more bugs
would be found in other communities such as StackOverflow, StackExchange and
Gitee, etc. Secondly, we did not further classify the errors from users. This
is because users’ misuse of QML frameworks is not this study’s main topic. And
most errors are caused by users not knowing how to use QML frameworks.
Finally, two authors disagreed a lot in the process of bug classification,
which led to the inability to reach a high agreement rate.

###  V-B External Validity

Some bugs were not fixed or were too complicated to set, so we couldn’t easily
determine the causes. On the other hand, many frameworks, especially
pennylane, often combine programs with devices on other platforms. Another
example, Qiskit-Machine-learning, requires Qiskit-terra as a base library for
implementation. This situation makes it hard to sort bugs by considering only
QML libraries.

###  V-C Verifiability

This threat is mainly about the possibility of replicating this study. We try
our best to provide all related details to replicate this work. Interested
researchers may find the replication package at https://github.com/Z-928/QML-
platform-bugs.

##  VI Related Work

###  VI-A Empirical Studies on Bugs in ML Systems and Frameworks

The recent success of applying machine learning to various areas has inspired
researchers to study and characterize bugs in machine learning systems
extensively. Thung  _et al._ [22] were the first to study machine learning
bugs by analyzing bug reports of three machine learning systems (Apache
Mahout, Lucene, and OpenNLP). Five hundred randomly picked fixed bugs were
manually inspected and categorized into 11 categories. Zhang  _et al._ [26]
studied 175 TensorFlow bugs collected from GitHub and Stack Overflow (SO).
They summarized the symptoms and root causes of the bugs. Humbatova  _et al._
[12] manually inspected bug reports from GitHub and posts in SO and proposed
an extensive taxonomy of DL system faults. These studies have investigated
bugs in machine learning systems, but none focuses on quantum machine learning
systems. Quantum machine learning systems are unique since the quantum
computation logic is expressed in quantum circuits, and the states of quantum
registers are measured probabilistically. Our study will fill this research
gap by performing the first systematic study on the bugs in quantum machine
learning frameworks.

There also exist studies that look into bugs in ML frameworks. Jia  _et al._
[13] studied the symptoms and causes of bugs in TensorFlow by analyzing 202
bugs inside TensorFlow. The authors then extend their work [14] by further
studying TensorFlow’s repair patterns and bugs involving multiple programming
languages. Chen  _et al._ [6] conducted a large-scale study on characteristics
(e.g., root cause, symptoms, and their correlations with DL framework
components) of DL framework bugs by analyzing 800 bugs from four popular DL
frameworks. Yang  _et al._ [25] conducted an empirical study on bug
characteristics of DL frameworks by analyzing 44,083 pull requests from eight
DL frameworks. However, little attention has been received to characterize
bugs in QML frameworks, and our study initiates the first step in this area.

###  VI-B Studies of Bugs Involve Quantum Programming

Paltenghi and Pradel [18] presented an empirical study of bugs in quantum
computing platforms. They summarized 223 real-world bugs in some open-source
quantum computing platforms, conducted a detailed statistical analysis of
bugs, and proposed some bug patterns specific to the field of quantum
computing platforms. Different from their work, this paper focuses on the bugs
in QML platforms. In summary, the findings from both studies foster a deep
understanding of bugs related to quantum computing systems.

In addition, several lines of research have been carried out to study the bug
patterns [27, 11], as well as bug benchmarks [4, 28] in quantum programs, to
support the debugging and testing of quantum software.

##  VII Conclusion

To the best of our knowledge, this work is the first study of bugs in the QML
platform. We collected 1,591 issues reported in 22 repositories, out of which
we filtered out 391 bugs. Based on the bugs found, we formulated three
research questions, and four findings based on the analysis of the results.
First of all, one-third of the bugs are quantum-specific bugs, which are
reflected in the functionality of qubit manipulation. Secondly, the symptom of
most bugs is Crash, followed by Function error. These two account for 93.7% of
all bugs, while other symptoms are relatively rare. Besides, the root cause
Algorithms and logic accounts for most of the bugs, followed by Inconsistency.
Finally, for developers, the challenges are mainly reflected in four aspects:
device interaction, version change, locating symptoms, and fixing bugs.

In future work, we would like to analyze the location of bugs and bug fixing
in QML platforms, etc. In addition, there are also some bugs caused by the
users of QML platforms, which are worth analyzing.

## Acknowledgement

This work was supported by JST, the establishment of university fellowships
towards the creation of science and technology innovation (Grant Number
JPMJFS2132).

## References

  * [1] G. Aleksandrowicz, T. Alexander, P. Barkoutsos, L. Bello, Y. Ben-Haim, D. Bucher, F. Cabrera-Hernández, J. Carballo-Franquis, A. Chen, C. Chen _et al._ , “Qiskit: An open-source framework for quantum computing,” _Accessed on: Mar_ , vol. 16, 2019.
  * [2] V. Bergholm, J. Izaac, M. Schuld, C. Gogolin, S. Ahmed, V. Ajith, M. S. Alam, G. Alonso-Linaje, B. AkashNarayanan, A. Asadi _et al._ , “Pennylane: Automatic differentiation of hybrid quantum-classical computations,” _arXiv preprint arXiv:1811.04968_ , 2018.
  * [3] J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, “Quantum machine learning,” _Nature_ , vol. 549, no. 7671, pp. 195–202, sep 2017. [Online]. Available: https://doi.org/10.1038\%2Fnature23474
  * [4] J. Campos and A. Souto, “QBugs: A collection of reproducible bugs in quantum algorithms and a supporting infrastructure to enable controlled quantum software testing and debugging experiments,” in _Proc. 2nd International Workshop on Quantum Software Engineering (Q-SE 2021)_ , 2021.
  * [5] M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, and P. J. Coles, “Variational quantum algorithms,” _Nature Reviews Physics_ , vol. 3, no. 9, pp. 625–644, sep 2021. [Online]. Available: https://doi.org/10.1038/s42254-021-00348-9
  * [6] J. Chen, Y. Liang, Q. Shen, and J. Jiang, “Toward understanding deep learning framework bugs,” 2022. [Online]. Available: https://arxiv.org/abs/2203.04026
  * [7] I. Cong, S. Choi, and M. D. Lukin, “Quantum convolutional neural networks,” _Nature Physics_ , vol. 15, no. 12, pp. 1273–1278, aug 2019. [Online]. Available: https://doi.org/10.1038\%2Fs41567-019-0648-8
  * [8] E. Farhi, J. Goldstone, S. Gutmann, and L. Zhou, “The quantum approximate optimization algorithm and the sherrington-kirkpatrick model at infinite size,” _Quantum_ , vol. 6, p. 759, jul 2022. [Online]. Available: https://doi.org/10.22331%2Fq-2022-07-07-759
  * [9] Google AI Quantum Team, “Cirq,” 2018. [Online]. Available: https://github.com/quantumlib/Cirq
  * [10] A. W. Harrow, A. Hassidim, and S. Lloyd, “Quantum algorithm for linear systems of equations,” _Physical Review Letters_ , vol. 103, no. 15, oct 2009. [Online]. Available: https://doi.org/10.1103\%2Fphysrevlett.103.150502
  * [11] Y. Huang and M. Martonosi, “Statistical assertions for validating patterns and finding bugs in quantum programs,” in _Proceedings of the 46th International Symposium on Computer Architecture_ , 2019, pp. 541–553.
  * [12] N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , ser. ICSE ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 1110–1121.
  * [13] L. Jia, H. Zhong, X. Wang, L. Huang, and X. Lu, “An Empirical Study on Bugs Inside TensorFlow,” in _Database Systems for Advanced Applications_. Springer International Publishing, 2020, pp. 604–620.
  * [14] L. Jia, H. Zhong, X. Wang, L. Huang, and X. Lu, “The symptoms, causes, and repairs of bugs inside a deep learning library,” _Journal of Systems and Software_ , vol. 177, p. 110935, 2021.
  * [15] M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” _Science_ , vol. 349, no. 6245, pp. 255–260, 2015.
  * [16] Lightweight Message Queuing Network, “Qnode.” [Online]. Available: https://github.com/lwmqn/qnode
  * [17] M. A. Nielsen and I. Chuang, “Quantum computation and quantum information,” 2002\.
  * [18] M. Paltenghi and M. Pradel, “Bugs in quantum computing platforms: an empirical study,” _Proceedings of the ACM on Programming Languages_ , vol. 6, no. OOPSLA1, pp. 1–27, 2022.
  * [19] M. Paltenghi and M. Pradel, “MorphQ: Metamorphic testing of the qiskit quantum computing platform,” in _Proceedings of the 36th IEEE/ACM International Conference on Software Engineering (ICSE’23)_ , 2023, pp. 2417–2428.
  * [20] A. Peruzzo, J. McClean, P. Shadbolt, M.-H. Yung, X.-Q. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’Brien, “A variational eigenvalue solver on a photonic quantum processor,” _Nature Communications_ , vol. 5, no. 1, jul 2014. [Online]. Available: https://doi.org/10.1038\%2Fncomms5213
  * [21] K. Svore, A. Geller, M. Troyer, J. Azariah, C. Granade, B. Heim, V. Kliuchnikov, M. Mykhailova, A. Paz, and M. Roetteler, “Q# enabling scalable quantum computing and development with a high-level DSL,” in _Proceedings of the real world domain specific languages workshop 2018_ , 2018, pp. 1–10.
  * [22] F. Thung, S. Wang, D. Lo, and L. Jiang, “An empirical study of bugs in machine learning systems,” in _2012 IEEE 23rd International Symposium on Software Reliability Engineering_ , 2012, pp. 271–280.
  * [23] H. Wang, Y. Ding, J. Gu, Z. Li, Y. Lin, D. Z. Pan, F. T. Chong, and S. Han, “Quantumnas: Noise-adaptive search for robust quantum circuits,” in _The 28th IEEE International Symposium on High-Performance Computer Architecture (HPCA-28)_ , 2022.
  * [24] J. Wang, Q. Zhang, G. H. Xu, and M. Kim, “Qdiff: Differential testing of quantum software stacks,” in _Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE ’21. IEEE Press, 2022, p. 692–704. [Online]. Available: https://doi.org/10.1109/ASE51524.2021.9678792
  * [25] Y. Yang, T. He, Z. Xia, and Y. Feng, “A comprehensive empirical study on bug characteristics of deep learning frameworks,” _Information and Software Technology_ , vol. 151, p. 107004, 2022.
  * [26] Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2018\. New York, NY, USA: Association for Computing Machinery, 2018, p. 129–140. [Online]. Available: https://doi.org/10.1145/3213846.3213866
  * [27] P. Zhao, J. Zhao, and L. Ma, “Identifying bug patterns in quantum programs,” in _Proc. 2nd International Workshop on Quantum Software Engineering (Q-SE 2021)_ , 2021, pp. 16–21.
  * [28] P. Zhao, J. Zhao, Z. Miao, and S. Lan, “Bugs4Q: A benchmark of real bugs for quantum programs,” in _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE 2021), New Ideas and Emerging Results Track_. IEEE, 2021, pp. 1373–1376.