11institutetext: Indian Institute of Technology, Kanpur, {mvohra,
<EMAIL_ADDRESS>
# Visually Guided UGV for Autonomous Mobile Manipulation in Dynamic and
Unstructured GPS Denied Environments

Mohit Vohra and Laxmidhar Behera

###### Abstract

A robotic solution for the unmanned ground vehicles (UGVs) to execute the
highly complex task of object manipulation in an autonomous mode is presented.
This paper primarily focuses on developing an autonomous robotic system
capable of assembling elementary blocks to build the large 3D structures in
GPS-denied environments. The key contributions of this system paper are i)
Designing of a deep learning-based unified multi-task visual perception system
for object detection, part-detection, instance segmentation, and tracking, ii)
an electromagnetic gripper design for robust grasping, and iii) system
integration in which multiple system components are integrated to develop an
optimized software stack. The entire mechatronic and algorithmic design of UGV
for the above application is detailed in this work. The performance and
efficacy of the overall system are reported through several rigorous
experiments.

##  1 Introduction

Robotics applications are designed to have a transformational impact on day-
to-day life in areas including disaster management, healthcare, household
works, transportation, construction, and manufacturing vohra2019real
pharswan2019domain . International robotics challenges such as the Amazon
Robotics Challenge (ARC) and Mohamed Bin Zayed international robotics
challenge (MBZIRC) are setting new benchmarks to advance the state-of-the-art
in autonomous solutions for these robotics applications. In addition to the
development of new algorithms, system integration is an essential step to
complete the task. Therefore, in this work, we will describe our system
architecture for the task of assembling large 3D structures in GPS-deprived
environments. However, the proposed system architecture for UGV-based object
manipulation is not limited to the above settings and can be deployed for
other real-world applications.

t]

Figure 1: Top: UGV endowed with arm and gripper. Bottom: Bricks used for the
task.

For the task of assembling a structure, a UGV needs to search, track, grasp,
transport and assemble the bricks according to a user-specific pattern. Apart
from that, for a GPS-denied environment, the system should harness the visual
information for localization and should also be endowed with a suitable
grasping mechanism. While performing the task, a mobile manipulator may need
to move across several positions in a large workspace. It encourages us to use
the onboard computational devices for various algorithms. Further, the top of
each brick consists of a ferromagnetic region ($0.15m\times 0.25m$) shaded
with yellow color called grasping region and is attached in order to
facilitate electromagnetic grasping as shown in Fig.1.

In order to perform the task, the perception system must locate the grasping
region on bricks. To equip the UGV with the functionality of brick searching,
tracking, grasping and placing, in a limited computational resources, we
present a deep learning based multi-task visual perception system. The state-
of-the-art object detectors yolov1  SSD  predicts a bounding box for an
object. Since, the bounding box may contain significant non-object regions,
therefore, an additional computation step is required to extract the object
information, hence, we prefer detection (or tracking) by segmentation. Since,
we already have the segmentation information for each object, therefore, to
save computations, we prefer segmentation based object tracking. wang2019fast
, can track the object by predicting the object mask at each instant, but it
relies on bounding box initialization. The key advantages of the tracking by
segmentation are i) The output of the brick detection by segmentation network
can be directly used for tracker initialization, ii)The output of the tracking
network can be directly used for tracking in next subsequent frames, and iii)
The detection network and tracking network can be integrated into a single
network, as both networks are performing a segmentation task. Apart from the
vision system, we also have designed a self-adjusting electromagnetic gripper,
which, when pressed against the brick surface, can adjust itself without
loading the end-effector motors. It is done to align the magnetic gripper
appropriately with the grasping regions to ensure a firm grasp. Overall, the
main contributions of this work can be summarized as below:

  1. 1.

Multi-task deep neural network based visual perception system for brick
detection, part detection, instance segmentation and tracking.

  2. 2.

Development of an self-adjusting electromagnetic gripper.

  3. 3.

System integration and a state machine to realize a fully autonomous UGV based
robotic manipulation system.

##  2 Related Work

Object detection: RCNN is one of the first successful object detectors. The
subsequent versions of RCNN are Fast-RCNN and Faster-RCNN, which have less
inference time as compared to RCNN. In addition RetinaNet lin2017focal , FCOS
tian2019fcos  perform detection by predicting the axis-aligned bounding boxes.
In general, the predicted box may contain significant background. Hence, to
reduce the background impact in the detection, various solutions have been
presented in the literature. For example, in li2018multiscale , the author
predicts a rotated bounding box to maximize the object regions inside the
predictions. Similarly, Mask-RCNN maskrcnn  predicts the mask of target object
inside the bounding box predictions.

Object Tracking: Discriminant correlation filter (DCF) based trackers
henriques2014high  localize the object by computing the correlation between
the filter and the search window. Siamese neural network based trackers
bertinetto2016fully  localizes the object by calculating the similarity
between the target image and the search window. Some deep trackers directly
regress the bounding box parameters of the object in the next frame by
comparing the target object patch and the search window held2016learning . The
third type of tracker uses a classifier based approach to localize the object
in the next frame. In this, a classifier assigns a score to each target
candidate, which is generated at different scales and random locations around
the previous object location nam2016learning , and a target candidate with
maximum score is considered as the new object location.

Object Part Detection: Recent approaches shi2020points  have demonstrated part
detection in point cloud data for 3D object detection. Conversely,
lorenz2019unsupervised  has detected object parts in the image plane for the
task of detection, tracking, and segmentation. Since in the task, we have to
detect yellow areas (ferromagnetic regions) from the brick surface, and we
have to classify whether it is related to green brick, blue brick, or red
brick. Therefore, it is intuitive to detect objects (or bricks) in the image
plane and then detect yellow regions corresponding to each brick region, which
leads us to integrate the function of object detection and part detection in a
single network.

##  3 System Description

UGV Platform: The UGV hardware system used for the task is shown in Fig. 1.
The hardware setup consists of a UR$5$ robotic arm deployed on a ROBOTNIK
Summit-XL mobile base. The UR$5$ arm is a $6$-DoF industrial arm having an
end-effector position accuracy of $\pm 0.1mm$ and a payload capacity of $5$kg.
The mobile base has four high power motor mecanum wheels with a maximum
payload of $100$Kg. The mecanum wheel allows the base to move in any direction
while keeping the front in a constant direction. The UGV is equipped with two
RGB-D sensors, One sensor is mounted on the $wrist1$ of the UR$5$ arm, and the
other sensor is mounted at the base’s front face, shown in Fig. 1. The raw
images captured by the first sensor is used for extracting the crucial
information about the bricks while the data from the second sensor is used for
the UGV localization.

t]

Figure 2: Proprietary electromagnetic gripper.

Gripper Design: Fig. 2 shows our carefully designed electromagnetic gripper
for grasping the desired objects. It consists of a double plate assembly with
a polymer form in between them. On one plate, we have attached seven
electromagnets with uniform distribution at the center of the gripper. All the
electromagnets are connected to the battery through a switch. The switch can
be controlled by transmitting signals high or low on the ROS topic. The double
plate assembly is connected to the UR5 end-effector through a steel-rod. The
presence of the foam in the gripper allows it to adjust according to the
tilted brick surface when pressed against the brick surface. Hence we call the
gripper as a self-adjusting gripper. Further, to avoid any damage to the
gripper, we keep measuring the force exerted on the gripper. If the value
exceeds some threshold, then grasping operation is complete; otherwise, we
keep pushing the gripper against the brick surface.

##  4 Multi-Task Unified Visual Perception block

The main task of the visual perception block is brick detection, part
segmentation, instant segmentation and brick tracking. Limited computational
resources prohibit us to use multiple CNN for the above. Hence, we aim to
unify the above four tasks in a deep neural network based visual perception
framework. Fig. 3 represents the proposed perception block and in following
section, we briefly discuss each of the component individually.

Figure 3: Visual Perception Pipeline

Instance Detection and Segmentation: Since there are multiple instances of the
bricks, hence it is necessary to perform instance detection and segmentation.
The top most choice for instance level segmentation and detection is Mask-
RCNNmaskrcnn . Being very memory exhaustive, we modified the architecture for
our computational constrain system. First, we carefully design a CNN backbone
similar to AlexNet, which is represented by $C_{1}$. Further, with $3$-$4$
iterations of weight pruning, we ended up with a light-model which can infer
in run-time. In the baseline model of Mask-RCNN, for object detection the
features upto the stage-$2$ of ResNet are used. In our setup, the object
detection features are limited to stage-$3$. Further, the remaining of the
architecture for task of instance detection and for segmentation remains the
same.

Part Detection: Since we have designed a lightweight MaskRCNN for object
detection and segmentation. Further, we have a requirement of object part
localization, i.e, ferromagnetic regions. Hence we aim to extend the
previously designed CNN for the above task. In order to achieve this, features
from ROI-Align layer are passed through the convolution layers as shown in
Fig. 3. The features from convolutional layers are multiplied with the output
of the instance segmentation branch. Finally, a binary mask is learned using
binary cross entropy loss. For more detailed explanation, the reader is
advised to refer the original work of Mask-RCNN.

Conditional Tracking: After the detection of all target brick instances, we
aim to localize the bricks in the subsequent frames by predicting the binary
mask corresponding to the target bricks. At this instant a binary mask
$M_{t-1}$ corresponding to the target brick in the image $I_{t-1}$ is
generated and fed to the unified network $C_{2}$. In the next step, the
current image $I_{t}$ is fed to the network, and based on previous image
$I_{t-1}$, and previous binary mask $M_{t-1}$, new mask $M_{t}$ is predicted.
Since the new prediction is guided by previous mask, hence we call the mask as
support mask.

An isolated AlexNet style CNN ($C_{2}$) (similar to the object detector) is
designed to extract spatial feature embedding corresponding to the support
mask as shown in Fig. 3. To meet the memory constraints, the architecture
$C_{2}$ has very less number of parameters in comparison with architecture
$C_{1}$. Further, both tracker and detector shares the CNN backbone $C_{1}$
for extracting the high dimensional embedding from RGB images.

Now, the features from stage-$3$, stage-$4$ and stage-$5$ of both $C_{1}$ (RGB
images) and $C_{2}$ (support mask) are merged by executing a concatenation
operation. Concatenation operation essentially guides the tracker to predict
in next frame by incorporating features from RGB image. Now, these merged
features are passed through FPN block for final mask predictions.

##  5 The Complete Workflow

At the start of the task, the UGV is provided with a target pattern for
assembling the bricks at the assembly area. By parsing the target pattern
file, UGV extracts the target brick ID and performs certain set of operations
to bring the target brick from the piles to the assembly area. The complete
robotic motion can be divided into various modules, depending on the state of
the UGV, which are explained below.

t]

Figure 4: UGV state at different modules, Row1: view from external camera,
Row2: UR5 arm sensor image, Row3: Network output

Searching Module: In this state, current image is fed to the perception
system, which performs bricks detection and segmentation (Fig. 4, Col. 1). If
the network ensures the presence of the target brick, then the search module
is complete, otherwise, the UGV system will explore the space by rotating the
base by $45^{\circ}$ in a clockwise direction and again execute the search
module.

Tracking Module: In tracking module, perception system predicts the binary
mask $M_{t}$ corresponding to the target brick ID in the current image
$I_{t}$. Further, to ensure that UGV is moving in right direction, the lateral
velocity of the UGV is controlled by the horizontal-component of the
displacement vector formed between the target brick centroid and the image
frame centroid as shown in Fig. 4, Col. 2. Further, the forward velocity of
the UGV is proportional to the distance of the target bricks from the UGV and
it gradually decreases to zeros when distance is less than threshold distance
($l_{th}$).

Alignment Module: In this module, the UGV will align its position wrt the
brick as the UR5 arm has a span of $1.0m$ only. As a first step, the current
image is fed the perception system for ferromagnetic regions segmentation
(Fig. 4, Col. 3). By applying PCA (Principal Component Axis) on the point
cloud data corresponding to ferromagnetic regions, the major and minor axis
(green and blue axis in Fig. 5, Col. 1) are estimated. The PCA-axis along with
the $3$D centroid of the point cloud data represents the $6$D pose of the
brick. Based on the current pose of UGV and the $6$D brick pose, piecewise
motion planner is executed which consists of three steps; i) Rotate base by
${a_{2}}$, ii) Move forward by distance $d$, and iii) Rotate base by
${a_{1}}$, where, ${a_{2}}$, ${d}$, ${a_{1}}$ are estimated online. The final
positioning of UGV after alignment is shown in Fig. 5, Col. 2.

Grasping Module: Firstly, $6$D pose of the brick is estimated (Fig. 4, Col.
4). Due to the sensor noise, the estimated pose is not accurate, hence to
compensate for this, the gripper is pressed against the brick surface till the
force exerted on the gripper reached a value ($f_{th}$). The foam present in
the gripper allows it to adjust according to the tilted brick surface. Thus
force feedback based grasping, with foam in the gripper, allows us to
compensate for the sensor noise.

Placing Module: After the grasping, UGV navigate to the wall assembly area.
For navigation, we have used the visual-slam module by incorporating the
state-of-the art real-time ORB-SLAM$2$orb2  on UGV. The final place pose for
the grasped brick depends on the previously placed brick. Hence, $6$D pose of
the previous brick is calculated. Based on the previous brick pose, target
pattern, and current pattern or current brick ID, the place pose is
calculated. After the brick placing operation, the current pattern is updated.
Fig. 4, Col. 5 represents the UGV state in the placing module. Here the system
has aligned wrt to the previous brick (red brick) and again computing the $6D$
pose of the (red) brick for estimating the new pose for placing the green
brick. The simplified flow of the overall system is shown in Fig. 6.

Figure 5: Alignment and brick placing operation

t]

Figure 6: Simplified State Machine of the overall System

##  6 Experiments

To demonstrate the validity and usefulness of the work presented in this
paper, we conducted several set of experiments to evaluate the performance of
the proposed visual perception network. Further, the evaluation of the overall
system performance is also provided, where we have performed a series of
experiments that provide a statistical analysis of the system’s performance.

Dataset: In this section we will explain the procedure for generating the
dataset for two tasks namely detection and tracking.

  * •

Detection The collection of data procedure is based on the work deepquick .
Following the idea of deepquick , a small sized dataset of about $100$ images
is collected, where each image has multiple brick instances. The dataset was
split between training set and testing set in the proportion of $7:3$. Now,
for each brick instance, the box regression and the segmentation is performed,
while for ferromagnetic regions, only the segmentation is required. During the
data generation process, the mask corresponding to the brick instance and for
ferromagnetic regions are marked manually. These masks are used as ground
truths for instance segmentation and brick-part segmentation. The ground
truths for box regression is generated by fitting a rectangle on the manually
marked object instance mask. In order to augment the dataset, synthetic data
augmentation technique is used heavily deepquick .

  * •

Tracking For tracking, $10$ indoor video sequences as well as $10$ outdoor
sequences, each at the rate of $\sim 30$FPS are collected. The video sequences
has the brick instances whose size varies as we move closer to the bricks.
Further, the video sequences are down-sampled to the rate of $\sim 2$ FPS and
each frame is annotated for instance detection, classification, segmentation
and part segmentation. The complete dataset for tracking contains $200$
images, where $150$ images are used for training and $50$ are used for
testing. Further, the dataset has been augmented using the synthetic
augmentation technique deepquick .

Training Policy: We consider the architecture of baseline network as Arch1 and
we also consider two other variants i.e. Arch2 and Arch3 which contains $25\%$
and $50\%$ of additional number of parameters as compared to Arch1. The kernel
sizes corresponding to each layers are mentioned in Table-1. Since object
detection and tracking are two different tasks, hence separate or individual
training procedure is required. In order to avoid the individual training,
brick instance masks and brick-part masks are also annotated with the object
masks in the training dataset. In this way, the detector can be trained using
the tracking dataset. Further, the training images are chosen with equal
probability from both dataset i.e., the detector dataset and tracker dataset.
With this technique, the deep neural network can get the experiences of both
temporal data (because of video sequences) and the single image object
detection. Further, the following hyperparameters are used for training, that
is, base learning rate is set to $0.001$, we have used the learning rate
policy=$step$, $ADAM$ optimizer is used with parameters $\beta_{1}=0.9$ and
$\beta_{2}=0.99$.

Table 1: Kernel sizes. ‘$*$’ corresponds to input channels Layer | $C_{1}$ | $C_{2}$
---|---|---
Stage-$1$ | $4\times 3\times 3\times 3$ | $2\times 1\times 3\times 3$
Stage-$2$ | $8\times 4\times 3\times 3$ | $4\times 2\times 3\times 3$
Stage-$3$ | $16\times 8\times 3\times 3$ | $4\times 4\times 3\times 3$
Stage-$4$ | $32\times 16\times 3\times 3$ | $8\times 4\times 3\times 3$
Stage-$5$ | $32\times 32\times 3\times 3$ | $16\times 8\times 3\times 3$
Others | $12\times*\times 3\times 3$ | $4\times*\times 3\times 3$

Ablation Study: Table-2 shows the performance of the deep neural network for
various tasks. The accuracy of network along with the timing performance for
half-Precision (FP$16$) and for single-precision (FP$32$) computations on a
single GeForce GTX $1050$, $4$GB GPU, is reported. Further, the performance of
all the three architectures Arch1, Arch2, and Arch3 are reported. From the
Table-2, we observed that Arch2 shows an minor improvement over Arch1,
similarly, Arch3 demonstrates slight improvements over Arch1, despite the
$50\%$ more parameters. This is because, the basic network architecture for
all variants are same except for the number of parameters. And another reason
is that, the object used for experiments have significant differences, and
thus intra-class variance is extremely low.

Table 2:  Performance Analysis Network | Time | Box | Seg | Part Seg | Tracker
---|---|---|---|---|---
Architecture | (ms) | mIoU | mIoU | mIoU | mIoU
Arch1 (FP$16$) | $49$ | $80.3$ | $79.2$ | $73.1$ | $83.4$
Arch1 (FP$32$) | $123$ | $82.2$ | $80.0$ | $72.9$ | $84.8$
Arch2 (FP$16$) | $80$ | $81.2$ | $79.7.$ | $73.6$ | $84.1$
Arch2 (FP$32$) | $161$ | $83.5$ | $81.1$ | $74.0$ | $85.6$
Arch3 (FP$16$) | $101$ | $85.3$ | $80.6$ | $73.5$ | $84.4$
Arch3 (FP$32$) | $187$ | $86.7$ | $81.9$ | $74.1$ | $86.4$

Synthetic Data Augmentation: Table-3 demonstrates the impact of synthetic
scenes and other augmentations as described in deepquick . From Table-3, we
observed that by incorporating the synthetic augmentation technique, we can
see the significant improvement in the performance. Further, the tracker
performance degradation is observed when the blurring effect is included in
the augmentation process.

Table 3:  Effect of Synthetic Data Augmentation for Arch1, FP$16$ Data Augmentation | mIoU score
---|---
Color | Scale | Mirror | Blur | Rotate | Synthetic scene | Detection | Segmentation | Part Segmentation | Tracker
✗ | ✗ | ✗ | ✗ | ✗ | ✗ | $23.7$ | $31.8$ | $18.8$ | $15.3$
✓ |  |  |  |  |  | $25.3$ | $32.1$ | $21.3$ | $17.7$
✓ | ✓ |  |  |  |  | $30.1$ | $38.6$ | $29.1$ | $23.1$
✓ | ✓ | ✓ |  |  |  | $32.3$ | $40.2$ | $31.9$ | $25.5$
✓ | ✓ | ✓ | ✓ |  |  | $32.5$ | $41.4$ | $33.8$ | $24.1$
✓ | ✓ | ✓ | ✓ | ✓ |  | $37.2$ | $49.8$ | $37.7$ | $28.4$
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | $80.3$ | $79.2$ | $73.1$ | $83.4$

Unified vs Non-Unified Perception System: The main contribution of this work
is the unification of various task. Hence, it is very intuitive to compare the
performance of multi-task network against the individual network for same
tasks namely detection, part segmentation and tracking. The individual network
architectures have the same configurations as that of multi-task network. It
can be observed from the Table-4 that the unified pipeline is way better than
the individual networks in terms of timing performance. Further the memory
consumed by the unified network is $40$% of total memory consumed by all three
networks. The link for one of the experiment is https://youtu.be/nPiMzrBFJ-A.

Table 4: Unified Vs Non-Unified Perception System, Arch1, FP$16$ Network | Time(ms)
---|---
Architecture | detection | Part Seg | Tracking | Total
Arch1 | $--$ | $--$ | $--$ | $49$
Detection | $26$ | $--$ | $--$ | $106$
Part Segmentation | $--$ | $35$ | $--$
Tracking | $--$ | $--$ | $45$ |

Overall System Evaluation: The complete robotic system was tested for $50$
rounds. In each round, the pile’s initial position is at different locations
and has different configurations, and the robotic system has to assemble the
3D structure at the assembly area according to the target pattern. In each
round, the target pattern consists of $6-8$ bricks. Thus overall the UGV is
tested for $346$ number of bricks and for each brick, we count the number of
times the individual module has performed satisfactory. The performance of the
module is defined as unsatisfactory if the final pose of the UGV, after the
execution of the specific module is random. On the other hand, if the module
performs satisfactory, then UGV will land at appropriate pose and can easily
switch to next module. Table-5 summarize the performance of UGV in our
experiments.

Table 5: Overall System Evaluation Module | Success Trials
---|---
Searching |  346/346 (100.0%)
Tracking |  346/346 (100%)
Alignment |  292/346 (84.4%)
Grasping |  292/292 (100.0%), 292/346 (84.4%)
Placing |  218/292 (74.6%), 218/346 (63.0%)
Overall |  218/346 (63.0%)

From Table-5, we observed that the searching module and tracking module has
$100\%$ accuracy. This is because the dataset is small and network has been
trained with extensive data augmentation technique. The alignment module is
successful for $292$ times out of $346$ trials, which shows $84\%$ success
rate. This is because, the front sensor gets exposed to the textureless view,
and thus visual-slam fails to update the pose of the UGV. For the remaining
successful trials, grasping module was tested and it has shown the $100\%$
success rate and overall the system score is $292/346$ which is $84\%$.
Further the placing module is successful in $218/292$, as during the brick
placing operation, visual slam fails to update the pose of UGV. The overall
system score is $218/346$. From the experiments, we observed that one of the
main point which can enhance the performance of UGV is to estimate the
accurate state of the UGV during alignment module and brick placing module.
For this, in future we aim to localize the UGV by using the data from multiple
sensors, which can be integrated at different body parts of the UGV.

##  7 Conclusion

We have presented a robotic solution to enable unmanned ground vehicles (UGVs)
to perform the highly complex task of assembling the elementary blocks (or
bricks) in the GPS-denied environment. The proposed system consists of a deep
learning based unified multi-task visual perception system for the tasks of
instance detection, segmentation, part segmentation and object tracking. The
perception system can infer at the rate of $20$ FPS on a single GeForce GTX
$1050$ $4$GB GPU. The proposed visual perception module has been extensively
tested for various test cases which includes indoor and outdoor environments
and the performance of the perception module for various tasks is reported in
this paper. The propose perception module is integrated onto the UGV system
for the task of brick assembly. Further to facilitate the grasping of bricks,
an electromagnetic based self-adjusting gripper is designed, and a force based
feedback grasping strategy is deployed to compensate for the sensor noise.
Further, the complete robotic system is tested for $50$ rounds of bricks
assembly task, and the proposed system has shown the accuracy of $63\%$. The
primary reason for the low accuracy is that visual-slam fails to update the
state of the UGV when the depth sensor get exposed to the textureless view.
Hence by incorporating the multiple sensors for localization, accuracy can be
increased, which will be considered as future work.

## References

  * (1) M. Vohra, R. Prakash, and L. Behera, “Real-time grasp pose estimation for novel objects in densely cluttered environment,” in 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 1–6, IEEE, 2019.
  * (2) S. V. Pharswan, M. Vohra, A. Kumar, and L. Behera, “Domain-independent unsupervised detection of grasp regions to grasp novel objects,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 640–645, IEEE, 2019.
  * (3) J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788, 2016.
  * (4) W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in European conference on computer vision, pp. 21–37, Springer, 2016.
  * (5) Q. Wang, L. Zhang, L. Bertinetto, W. Hu, and P. H. Torr, “Fast online object tracking and segmentation: A unifying approach,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1328–1338, 2019\.
  * (6) T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision, pp. 2980–2988, 2017.
  * (7) Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” in Proceedings of the IEEE international conference on computer vision, pp. 9627–9636, 2019.
  * (8) S. Li, Z. Zhang, B. Li, and C. Li, “Multiscale rotated bounding box-based deep learning method for detecting ship targets in remote sensing images,” Sensors, vol. 18, no. 8, p. 2702, 2018.
  * (9) K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” CVPR, 2017\.
  * (10) J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” IEEE transactions on pattern analysis and machine intelligence, vol. 37, no. 3, pp. 583–596, 2014.
  * (11) L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in European conference on computer vision, pp. 850–865, Springer, 2016.
  * (12) D. Held, S. Thrun, and S. Savarese, “Learning to track at 100 fps with deep regression networks,” in European Conference on Computer Vision, pp. 749–765, Springer, 2016.
  * (13) H. Nam and B. Han, “Learning multi-domain convolutional neural networks for visual tracking,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4293–4302, 2016.
  * (14) S. Shi, Z. Wang, J. Shi, X. Wang, and H. Li, “From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  * (15) D. Lorenz, L. Bereska, T. Milbich, and B. Ommer, “Unsupervised part-based disentangling of object shape and appearance,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10955–10964, 2019.
  * (16) R. Mur-Artal and J. D. Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1255–1262, 2017.
  * (17) A. Kumar and L. Behera, “Semi supervised deep quick instance detection and segmentation,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 8325–8331, IEEE, 2019.


# A Comparative Analysis of Hybrid-Quantum Classical Neural Networks

Kamila Zaman1,2,*, Tasnim Ahmed1,2,*, Muhammad Abdullah Hanif1,2, Alberto
Marchisio1,2, and Muhammad Shafique1,2 1eBrain Lab, Division of Engineering,
New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE2Center for Quantum and
Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE
kz2137, tasnim.ahmed, mh6117, alberto.marchisio<EMAIL_ADDRESS>
###### Abstract.

Hybrid Quantum-Classical Machine Learning (ML) is an emerging field,
amalgamating the strengths of both classical neural networks and quantum
variational circuits on the current noisy intermediate-scale quantum devices
(Zaman et al., 2023). This paper performs an extensive comparative analysis
between different hybrid quantum-classical machine learning algorithms, namely
Quantum Convolution Neural Network, Quanvolutional Neural Network and Quantum
ResNet, for image classification. The experiments designed in this paper focus
on different Quantum ML (QML) algorithms to better understand the accuracy
variation across the different quantum architectures by implementing
interchangeable quantum circuit layers, varying the repetition of such layers
and their efficient placement. Such variations enable us to compare the
accuracy across different architectural permutations of a given hybrid QML
algorithm. The performance comparison of the hybrid models, based on the
accuracy, provides us with an understanding of hybrid quantum-classical
convergence in correlation with the quantum layer count and the qubit count
variations in the circuit.

Quantum Machine Learning, Hybrid Quantum-Classical Neural Networks, Quantum
Convolutional Neural Networks, Quantum ResNet, Quanvolutional Neural Networks

*These authors contributed equally to this work.

##  1\. Introduction

Merging Quantum Computing (QC) with Machine Learning (ML) forms the emerging
Quantum Machine Learning (QML) paradigm (Schuld and Killoran, 2019; Schetakis
et al., 2022; Markidis, 2023; Massoli et al., 2022). It represents an
excellent opportunity for researchers and industries to make phenomenal
discoveries and unravel efficient ways to solve complex real-world problems
with significant direction towards practicality and improved accuracy compared
to classical systems. QML opens new avenues for the community to discover,
build, and align their designs to different levels of the quantum stack.
However, the currently developed Noisy-Intermediate Scale Quantum (NISQ)
devices (Preskill, 2018) have a limited number of qubits, with small-scale
resilience to noise, therefore, making it difficult to develop and practically
realize the potential of standalone quantum machine learning algorithms. The
limitation of NISQ devices has motivated the development of hybrid quantum-
classical machine learning algorithms (HQML), which are NISQ-compatible
algorithms.

Figure 1. Penny-lane: The arrows indicate an observed pattern suggesting a
potential positive correlation of accuracy with the number of layers & qubits
in hybrid quantum-classical architectures. Specifically, (a), (b), and (c)
consistently exhibit this behavior for the QuanNN across all entangling
circuits, while (d) displays a similar effect in the QResNet. While the QuanNN
responds to both layer and qubit count, the QResNet demonstrates improved
accuracy primarily with an increase in qubits only. Hence, observation at (d),
raises questions for further studies on whether the QResNet’s accuracy
significantly improves with a substantial increase in qubits further or not.

HQML algorithms have emerged as an important paradigm that amalgamates the
power of both classical and quantum processing for machine learning tasks,
opening new avenues for algorithm and architecture exploration (Bergholm et
al., 2022). The most renowned HQML algorithms employ variational quantum
circuits, featuring parameterized quantum gates optimized by classical
computers to achieve specific goals. These variational quantum circuits in
QML, known as Quantum Neural Networks (QNNs), hold significant promise due to
their expressiveness and reduced trainable parameters, garnering considerable
development interest.

Given the recent breakthroughs of classical algorithms such as Convolutional
Neural Networks (CNNs) for image classification tasks, numerous QNN
architectures have been developed for classification tasks. However, those
architectures are either implemented as basic building blocks or lack the
verification of their usability against a benchmark model for classification
tasks. The observations in Figure 1, based on our implementation of
permutations of two different algorithms, show that different architectures of
the same algorithms have different impacts on classification tasks. Thus, it
is imperative to explore the architectures, applications, and usefulness of
QNN algorithms and evaluate their accuracy to identify efficient model
configurations that can be deemed as reference benchmarks for future research.

In this paper, we explore three different QNN algorithms amongst the pool of
hybrid algorithms, namely, Quanvolutional Neural Networks (Henderson et al.,
2019), Quantum Convolutional Neural Networks (Cong et al., 2019), and Quantum
ResNet (Liang et al., 2021) and perform an extensive comparative analysis. Our
methodology first involves identifying efficient architectures amongst the
commonly used QNNs and understanding their practical utility for
classification tasks. Then we assess how variations of such algorithms impact
their accuracy and robustness under architectural permutations of each
algorithm. The architectural permutations are based on the implementation of
interchangeable variational circuit layers over different qubit counts,
varying repetition of the layers, and considering their optimal placement. By
implementing the varied models, we evaluate the performance of QNNs based on
the accuracy of the training process, which provides us with an understanding
of hybrid quantum-classical convergence in correlation with our experimental
approach. Such a comprehensive analysis is necessary to establish an
understanding of the correlation between circuit architectures, their
robustness, and utility in QML.

Figure 2. Overview of our novel contributions.

###  1.1. Our Novel Contributions

An overview of our novel contributions is shown in Figure 2. Their brief
descriptions with key features is presented below.

  * •

We propose a methodology to investigate QNN models’ circuit permutations and
their effect on the accuracy for classification tasks. (Section 3)

  * •

We analyze the effect of different circuit layer variations111The circuit
variations are studied only for QNN models that support different types of
entanglement., namely, Random Circuit, Basic Entangling, and Strongly
Entangling. (Section 4.2)

  * •

We investigate the repetition of different layers to analyze the effect of
circuit depth and complexity. (Section 4.3)

  * •

We test the scalability of the algorithms by varying qubit counts (Section
4.4)

##  2\. Selected Hybrid QML Algorithms

An overview of the hybrid QML algorithms is shown in Figure 3. A brief
description of the key features of each algorithm and their implementations is
presented in the following paragraphs.

Figure 3. Pipeline of the implemented hybrid QML algorithms. Each model
utilizes a classical fully connected layer to transform quantum circuit
measurement into classification probabilities. In the QCNN, classical
convolutional and pooling layers are used for image downsizing to match the
qubit count of a circuit.

###  2.1. Quanvolutional Neural Networks

The Quanvolutional Neural Network (QuanNN) is an innovative hybrid quantum-
classical architecture developed in (Henderson et al., 2019), which enhances
the capabilities of classical CNNs by harnessing the potential of quantum
computation. This architecture introduces a new type of transformation layer
called the quanvolutional layer, akin to classical convolutional layers,
composed of multiple quanvolutional filters which locally transform input
data, extracting valuable features for classification. These filters
correspond to a certain circuit design, which can either be generated randomly
or based on a specific entanglement, namely Basic Entangling or Strongly
Entangling. The reason we chose the QuanNN as one of our benchmarking models
is because of its generalizability which can be achieved by adhering to the
following conditions: specifying an arbitrary integer number of quanvolutional
filters in each layer, stacking multiple quanvolutional layers in the network,
defining layer-specific parameters like encoding method, entanglement, and
average quantum gates per qubit in the quantum circuit. To formalize classical
data transformation using quanvolutional filters:

  1. (1)

Start with a single filter q operating on subsections ux of dataset images.

  2. (2)

Encode ux into an initialized state ix using an encoding function e.

  3. (3)

Apply the quantum circuit to ix, producing an output quantum state ox.

  4. (4)

Decode ox to ensure consistent outputs, resulting in the final decoded state
fx.

  5. (5)

This entire process, denoted as the “quanvolutional filter transformation” Q,
is fx = Q(ux, e, q, d).

Figure 4. Overview of our comparative analysis methodology.

###  2.2. Quantum Convolutional Neural Networks

Similar to the QuanNN, the Quantum Convolutional Neural Network (QCNN) is a
QML algorithm inspired by CNNs that was introduced in (Cong et al., 2019).
Unlike the QuanNN, the QCNN does not have room for lots of circuit design
variations. The QCNN has a fully quantum implementation of convolutional and
pooling layers. We chose this architecture in our experiments because it is
one of the state-of-the-art QML models and the classical counterpart
represents the state-of-the-art in classical image recognition. Moreover, it
is important to note that the QCNN presented in (Cong et al., 2019) makes use
of only $O(log(N))$ variational parameters for input sizes of $N$ qubits. This
allows for its efficient training and implementation on realistic, near-term
quantum devices.

The basic structure of the QCNN includes an input encoding circuit layer, a
convolutional circuit layer, a pooling circuit layer, and a circuit
measurement layer, each composed of parametric quantum gates. It is
categorized as an HQML algorithm because it uses classical optimization
techniques to update the parameterized gate weights. In the quantum
convolutional and pooling layers, the interactions between quantum bits can
effectively extract features from the input data based on the types of gates
and their placement in each layer. Given the current NISQ devices, the QCNN is
limited in terms of scalability and can only be implemented with a small
number of qubits. Therefore, it requires classical layers for downsizing large
inputs to match the qubit size. In our implementation, we employ single
classical convolution and pooling layers for input downsizing, without loss of
key features.

###  2.3. Quantum ResNet

Our decision to employ the Quantum ResNet (QResNet) algorithm was inspired by
(Liang et al., 2021), which introduced a hybrid quantum-classical strategy for
deep residual learning. The primary challenge in this approach is to establish
a connection between the residual block structure and the quantum processing
layers. In our specific context, our focus lies in experimenting with various
quantum layer types. We delve into an analysis of potential methods and
variations that combine residual learning with quantum machine learning.
Notably, the work in (Ghasemian and Tavassoly, 2022) emphasizes that
experimental outcomes demonstrate the QResNet’s superior capability of
learning an unknown unitary transformation and its enhanced robustness with
noisy data, compared to state-of-the-art methods. These findings motivate us
to further explore and fine-tune this promising architecture. Our choice of
the QResNet stems from our desire to test this HQML architecture and its
impact on pre-trained classical models.

##  3\. Comparative Analysis Methodology

With the aim of understanding hybrid-quantum classical neural networks in-
depth, we divide our problem into interdependent experimental sections as
shown in Figure 4. The sectioning allows us to identify areas of possible
efficient implementation and improvements for the HQML algorithms discussed in
Section 2.

###  3.1. Overview of Analyses

Our experimental sections are summarized in the Quantum Layer Variations table
in the Quantum Model Configurations group of Figure 4. For each algorithm, we
analyze the impact of different architectural permutations based on:

  * •

Entanglement variation of quantum circuit: The predefined circuit structure of
QuanNN and QResNet enables us to change the entanglement type of the circuit.
Each circuit has its own orientation of CNOT gates and parameterized
corresponding to the strength of their entanglement. Whereas for QCNN, there
is no room for changing the entanglement type of the circuit, since it follows
the structure defined in (Cong et al., 2019).

  * •

Layer count variation: For each algorithm, a circuit can be applied multiple
times to an input, which corresponds to understanding how the varying depth of
a quantum circuit affects the model accuracy.

  * •

Qubit count variation: The strength and capability of a circuit depends on the
number of qubits it has. Hence, in our experiments, we vary the qubit counts
in the architecture to analyze how the variation correlates with the models’
learning curve and accuracy.

###  3.2. Comparison Metrics

We intentionally made our experimental setup simple and precise, to gain more
understanding about how different independent parameters of a circuit’s design
of our selection algorithms can influence their accuracy. To understand the
contribution and convergence behaviours of the aforementioned variations, we
focus on analyzing the correlation between a model’s accuracy by studying
their classical accuracies in relation to the learning curve attained over the
training progress.

##  4\. Evaluation and Discussion

###  4.1. Experimental Setup

This paper investigates the design variations of the quantum components of
HQML architectures. Hence, to ensure a fair comparison, we use a uniform
classical optimization environment for both the classical and the quantum
layers as specified in Table 1. In the HQML architectures of our experiment,
all the algorithms have a classical layer after the quantum layer to convert
the quantum measurement values into classical probabilities. However, the QCNN
has a classical layer before the input encoding for image downsize. As for the
quantum layers, PennyLane and Qiskit frameworks provide PyTorch integration
modules, which convert a quantum layer into PyTorch trainable layers and
perform classical optimization of the gates based on the training
hyperparameters that we specified in Table 1. In our experiments, to ensure
that the output of a circuit corresponds to the minimum number of qubits of
our qubit count pool, we use a subset of the MNIST dataset (Deng, 2012)
consisting of only 4 classes, [0, 1, 2, 3]. Thus, the last classical layer of
our models consists of only 4 neurons.

For analyzing the HQML components of our architecture, three different kinds
of variation are applied to the quantum layer of an algorithm, as shown in
Figure 4:

  * •

Entanglement variation of quantum circuit: For algorithms that allow different
entanglement orientation, we vary their architectures with Random Circuit
(RC), Basic Entangling (BE), or Strongly Entangling (SE) circuit.

  * •

Layer count variation: Each quantum layer can have numerous repetitions of a
circuit. In our implementation, we repeat the circuit from 1 to 6 times. Note:
the QCNN has a fewer layer variation compared to the QuanNN and QResNet
because number of $layer=\sqrt{qubits}$.

  * •

Qubit count variation: QuanNN and QResNet circuits follow a square filter-like
structure where the qubits must be a square number. Hence, we experiment with
4 and 9 qubits. On the other hand, the circuit of the QCNN must be an even
number that can be reduced in half at each pooling step. Hence, we experiment
with 4 and 8 qubits.

Note: We implemented all these pipelines in Qiskit (Qiskit contributors, 2023)
& PennyLane (Bergholm et al., 2022) to enable the support for both QML
frameworks. They are purposely presented separately (and can be identified in
boldened figure captions) to avoid cross-tool comparison which is out of the
scope of this paper.

Table 1. Training Environment Specifications

Algorithm Experiment Name Software FrameWork `PennyLane(PL)`, `Qiskit(QK)`
Back-End Simulator `lightning.qubit(PL)`, `qasm_simulator(QK)` Back-End
Machine `NVIDIA RTX 6000 Ada` Deep-Learning Interface `Pytorch` Data-set
`MNIST` (Deng, 2012) Training Samples, Testing Samples PL: (`100`, `100`), QK:
(`500`, `100`) Epoch, Batch-Size, LR `5`, `5`, `0.01`

###  4.2. Results: Entangling Circuit Variations

Figure 5. PennyLane: RC: Random Circuit, SE: Strongly Entangling, BE: Basic
Entangling. Circuit Variation across QuanNN and QResNet.

Figure 5 shows the accuracy evolution over training epochs for QuanNN and
QResNet at different entanglement settings (RC, BE & SE) for PennyLane
implementation. As demonstrated by the accuracy and convergence gap between
the two algorithms, it is evident that QuanNN learns significantly better than
QResNet (see labels a and b).

Further analyses show that the RC QuanNN (label c) acquires lower accuracy
than the other entanglement settings. Whereas, for QResNet, the RC model
(label d) achieves the highest accuracy compared to the SE and BE settings.
Towards the end of the training, we can see that the accuracy of the QuanNN,
for all three entanglements, converges to around 80%, with the SE QuanNN
having the highest accuracy. Following the traces for BE and SE circuits, for
both QuanNN and QResNet (labels e and f, respectively), it is noticeable how
the traces are similar in each algorithm. On the other hand, RC varies in
learning compared to SE and BE in both algorithms. In the RC QuanNN (label c),
the accuracy improves at a slower pace compared to SE and BE, but soon
converges closer towards the end. As for the RC QResNet, the rate of learning
is similar to BE and SE, but with a higher accuracy throughout.

Figure 6. Qiskit: RC: Random Circuit, SE: Strongly Entangling, BE: Basic
Entangling. Circuit Variation across QuanNN and QResNet.

Figure 6 shows the same evolution graphs as Figure 5, but for Qiskit models.
It is obvious that even for Qiskit implementations the accuracy gap between
QuanNN and QResNet models are same as PennyLane (labels a and b). It is quite
evident, that QuanNN models in Qiskit have different learning curve, with a
great accuracy jump after the first epoch (label c).

Note: the QCNN is not presented in this comparison because only the Basic
Entangling circuit can be implemented in this architecture.

###  4.3. Results: Layer Count Variations

Figure 7. PennyLane: Layer Variation across different Entangling Circuits for
QuanNN and QResNet. Asides minor perturbation, an overall trend of improving
accuracy is observed with the increasing number of layers, for every type of
circuit.

Figure 7 shows the accuracy of the QResNet and QuanNN models with varying
number of layers implemented in PennyLane. From the results, we can gauge that
there is minor difference between 2, 3, and 4 layers for QResNet models (see
labels b and c). Moreover, adding more than 4 layers to the QuanNN does not
always contribute to increasing the accuracy (see labels a and e). For the
QResNet, it can be observed that the accuracy increases quite significantly
for the QResNet models with 4, 5, and 6 layers (see d, e, and f), more
specifically for circuit with 4 layers. Based on this observation, we
conducted the remaining experiments with 4 layers in places where no layer
variation was involved. Even though the accuracy of the QResNet is lower than
the QuanNN, the algorithm has some potential because it can improve the
accuracy for increasing the number of layers (see labels d and f).

Figure 8. Qiskit: Layer Variation across different Entangling Circuits for
QuanNN and QResNet. No overall trend observed with increasing number of
layers.

In Figure 8, we can observe that the QuanNN and QResNet models implemented in
Qiskit do not have an overall trend with respect to increasing the number of
layers, which indicates that the circuit depth does not have enough impact on
these models’ accuracy (see labels a, b and c). However, it is quite evident
for QCNN models that increasing the number of layers has a positive
correlation with the accuracy (see label d).

Figure 9. PennyLane: Initial learning behavior of Layer Variations, for the
QuanNN with Basic Entangling. Having 3 layers shows a clear advantage over the
other configurations in the first epochs, but the accuracy of the 4-layer
QuanNN, despite being slow at first, catches up quickly with the 3-layer
counterpart and converges to better results in the majority of the
experiments.

Figure 9 shows the learning curve for the QuanNN with Basic Entangling for
different layers implemented in PennyLane. For 1 layer, the learning curve is
constant throughout and little improvement is observed (see label c). For more
than one layer, we can observe that the learning improves with increasing the
number of layers. However, the optimal peak is until 4 layers (see label b),
because beyond that the learning curve drops to being close to the 1-layer
curve (see label a). Nonetheless, a steady and gradual improvement in accuracy
is observed for more than one layer.

For the models implemented in Qiskit, as shown in Figure 10, we can observe
that QCNN models with 3 layers have the highest accuracy towards the end of
the first epoch (see label a), however, with a very steadily increasing
learning curve (see label c). As for the QuanNN models, although the initial
accuracy is lower than the QCNN 3L, the models have a rapidly increasing
learning curve. Moreover, we can see that the learning curve for the QuanNN
models peak only until 5 layers (QuanNN 5L), because beyond that, for 6
layers, a very slow and steady improvement is observed (see label b).

Figure 10. Qiskit: Initial learning behaviour of different Basic Entangling
layer variation counts.

###  4.4. Results: Qubit Count Variations

Figure 11 illustrates the results for the QuanNN and QResNet models with
different qubits counts, implemented in PennyLane. A clear pattern can be
observed for the QuanNN models, where there is a consistent positive
correlation between the number of qubits and model accuracy, with increasing
layer counts (see labels a, b, and c), indicating that the models learn better
with increasing number of qubits. On the other hand, the QResNet models with
Basic Entangling circuit and Random Circuit do not have any significant trend.
However, the QResNet with Strongly Entangling circuit shows an increase in
accuracy with increasing its qubit count (see label d), which might lead one
to derive that QResNet with Strongly Entangled circuit has a positive learning
capability with an increased number of qubits.

Figure 11. PennyLane: Qubits Count Variation across different quantum
entangling circuits for QuanNN and QResNet. It can be noted that there a
positive correlation between number of qubits in QuanNN model and its
accuracy. Figure 12. Qiskit: Qubit Count variations across different
entangling circuits for QuanNN, QResNet, and QCNN. The QCNN accuracy
positively correlates with the increasing number of qubits.

Figure 12 depicts the variation of models implemented in Qiskit with regards
to varying qubit count. Firstly, we can notice that the QCNN accuracy improves
with increasing the number of qubits, which indicates that the QCNN learns
better with more qubits (see label a). A similar trend is observed for the
QuanNN and QResNet, but only for the models with Random Circuit (see labels b
and c). From this analysis, we can gauge that, changing the number of qubits
in Qiskit models have a varying impact for different entagnling circuits.

###  4.5. Result Discussion & Findings

We observed from the results that different variations combined together have
a compounding effect on the model accuracy, as it is not trivial to determine
the best set of permutations for the given HQML algorithm. For instance, the
QResNet had a stagnant improvement across most models. On the other hand, if
implemented with Strongly Entangling and 9 qubits, then the accuracy improves
drastically. However, in most cases, the accuracy gain achieved when
increasing the number of qubits comes with the cost of increased execution
time.

In addition, varying the number of layers with different entangling circuits
can impact the accuracy differently, as we observed in our results above. In
some cases, the increasing accuracy with increasing depth indicates that there
is potential in applying more quantum layers at the initial stages of HQML
models instead of classical convolutional layers. However, as observed in
Figure 11, we can see that the accuracy of HQML models keeps increasing as the
model becomes more complex (i.e., with a higher number of layers and qubits).
However, the more complex a model becomes, the more time it takes to execute.

All of our findings prove that the best circuit configuration cannot be
identified easily for a given algorithm when keeping in mind all the
constraints.

##  5\. Conclusion

In our work, we studied the accuracy variation of 3 different algorithms,
QuanNN, QCNN, and QResNet, by configuring the quantum architecture of each
circuit by varying the layer count, qubit count, and the type of entangling
circuit. These 3 categories of variation enabled us to design different
permutations of circuits for a given algorithm and study their effect on their
accuracy.

This study has granted us valuable insights into the establishment and
exploration of avenues for optimizing hybrid quantum-classical neural network
architectures. We have effectively advanced our pipelines toward practical
improvements, using foundational structures for experimentation. The detailed
deconstruction of the architecture offers transparency into the parameters
that influence the performance of quantum circuits within neural networks.
Such an in-depth analysis serves as the building block for designing efficient
HQML architectures and paves the way to develop robust HQML algorithms
applicable in the NISQ era, while also considering design aspects that can
smoothly transition into Fault Tolerant Quantum Computers.

###### Acknowledgements.

This work was supported in part by the NYUAD Center for Quantum and
Topological Systems (CQTS), funded by Tamkeen under the NYUAD Research
Institute grant CG008.

## References

  * (1)
  * Bergholm et al. (2022) Ville Bergholm, Josh Izaac, Maria Schuld, Christian Gogolin, Shahnawaz Ahmed, Vishnu Ajith, M. Sohaib Alam, Guillermo Alonso-Linaje, B. AkashNarayanan, Ali Asadi, Juan Miguel Arrazola, Utkarsh Azad, Sam Banning, Carsten Blank, Thomas R Bromley, Benjamin A. Cordier, Jack Ceroni, Alain Delgado, Olivia Di Matteo, Amintor Dusko, Tanya Garg, Diego Guala, Anthony Hayes, Ryan Hill, Aroosa Ijaz, Theodor Isacsson, David Ittah, Soran Jahangiri, Prateek Jain, Edward Jiang, Ankit Khandelwal, Korbinian Kottmann, et al. 2022\.  PennyLane: Automatic differentiation of hybrid quantum-classical computations.  arXiv:1811.04968
  * Cong et al. (2019) Iris Cong, Soonwon Choi, and Mikhail D. Lukin. 2019.  Quantum convolutional neural networks.  _Nature Physics_ (2019).
  * Deng (2012) Li Deng. 2012.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web].  _IEEE Signal Processing Magazine_ (2012).
  * Ghasemian and Tavassoly (2022) E. Ghasemian and M. K. Tavassoly. 2022.  Hybrid classical-quantum machine learning based on dissipative two-qubit channels.  _Scientific Reports_ (2022).
  * Henderson et al. (2019) Maxwell Henderson, Samriddhi Shakya, Shashindra Pradhan, and Tristan Cook. 2019.  Quanvolutional Neural Networks: Powering Image Recognition with Quantum Circuits.  arXiv:1904.04767
  * Liang et al. (2021) Yanying Liang, Wei Peng, Zhu-Jun Zheng, Olli Silvén, and Guoying Zhao. 2021.  A hybrid quantum-classical neural network with deep residual learning.  arXiv:2012.07772
  * Markidis (2023) Stefano Markidis. 2023.  Programming Quantum Neural Networks on NISQ Systems: An Overview of Technologies and Methodologies.  _Entropy_ (2023).
  * Massoli et al. (2022) Fabio Valerio Massoli, Lucia Vadicamo, Giuseppe Amato, and Fabrizio Falchi. 2022.  A Leap among Quantum Computing and Quantum Neural Networks: A Survey.  arXiv:2107.03313
  * Preskill (2018) John Preskill. 2018.  Quantum Computing in the NISQ era and beyond.  _Quantum_ (2018).
  * Qiskit contributors (2023) Qiskit contributors. 2023.  Qiskit: An Open-source Framework for Quantum Computing.
  * Schetakis et al. (2022) N Schetakis, D Aghamalyan, P Griffin, and M Boguslavsky. 2022.  Review of some existing QML frameworks and novel hybrid classical-quantum neural networks realising binary classification for the noisy datasets.  _Sci. Rep._ (2022).
  * Schuld and Killoran (2019) Maria Schuld and Nathan Killoran. 2019.  Quantum Machine Learning in Feature Hilbert Spaces.  _Physical Review Letters_ (2019).
  * Zaman et al. (2023) Kamila Zaman, Alberto Marchisio, Muhammad Abdullah Hanif, and Muhammad Shafique. 2023.  A Survey on Quantum Machine Learning: Current Trends, Challenges, Opportunities, and the Road Ahead.  arXiv:2310.10315