# Rethinking the Trigger-injecting Position in Graph Backdoor Attack

Jing Xu  Delft University of Technology
Delft, Netherlands
<EMAIL_ADDRESS>Gorka Abad  Radboud University
Nijmegen, Netherlands
Ikerlan research centre
Arrasate-Mondragón, Spain
<EMAIL_ADDRESS>Stjepan Picek  Radboud Univesity
Nijmegen, Netherlands
Delft University of Technology
Delft, Netherlands
<EMAIL_ADDRESS>
###### Abstract

Backdoor attacks have been demonstrated as a security threat for machine
learning models. Traditional backdoor attacks intend to inject backdoor
functionality into the model such that the backdoored model will perform
abnormally on inputs with predefined backdoor triggers and still retain state-
of-the-art performance on the clean inputs. While there are already some works
on backdoor attacks on Graph Neural Networks (GNNs), the backdoor trigger in
the graph domain is mostly injected into random positions of the sample. There
is no work analyzing and explaining the backdoor attack performance when
injecting triggers into the most important or least important area in the
sample, which we refer to as trigger-injecting strategies MIAS and LIAS,
respectively. Our results show that, generally, LIAS performs better, and the
differences between the LIAS and MIAS performance can be significant.
Furthermore, we explain these two strategies’ similar (better) attack
performance through explanation techniques, which results in a further
understanding of backdoor attacks in GNNs.

###### Index Terms:

backdoor attack, trigger-injecting position, graph neural networks

##  I Introduction

Graph Neural Networks (GNNs) have demonstrated their superior performance in a
variety of applications, such as node classification [7], graph classification
[3], image classification [33], and natural language processing [33]. However,
GNNs are vulnerable to various adversarial attacks, including the backdoor
attack. Specifically, a backdoor attack occurs when the adversary deliberately
modifies a proportion of the training data by adding the trigger (e.g.,
subgraph in a graph) to make the model misclassify the samples with the
trigger as the target label(s). The backdoored GNN model aims to perform
normally on benign testing samples. However, if the same trigger used in the
training phase is introduced onto a testing sample, the backdoored model
exhibits a particular output behavior of the adversary’s choosing, such as
misclassification into to target label(s)). Backdoor attacks have been
demonstrated to perform malicious tasks on security-related graph learning
services, such as converting the label of a fraud account to benign in a
social network [11]. Hence, the backdoor attack is a serious threat to the
practical applications of GNNs.

Several works explored the backdoor attacks in GNNs [24, 31, 26]. In these
works, one idea of the trigger-injecting position is randomly selecting a
subgraph as there is no specific location information in a graph [23]. Another
idea to inject the trigger into a graph is to select the subgraph which has
high similarity with the trigger graph [24]. Moreover, based on the
improvement of the explanation techniques in the graph domain, [26] proposed
injecting the trigger into the most important or least important area of the
sample. However, that work does not provide any experimental analysis to
confirm the assumptions made. Also, there is no work so far on using
explanation tools to explain the backdoor attack behavior in the graph domain.
This work first raises a core question:

What is the attack performance when injecting trigger into the most or least
important area of the sample?

To answer this question, we explore the impacts of the backdoor trigger-
injecting position from the perspective of the most (MIAS) or least important
area of the sample (LIAS). Although there is no location information in a
graph, we can still locate the most (least) important area in a graph, like in
an image, by using some explanation techniques [27]. As shown in experiments,
we demonstrate that the attack performance of LIAS is better, where the
difference from MIAS can even be significant. This observation inspires one
further question:

Can we explain this difference?

There are already some works on explaining backdoor attacks in the image
domain through visualization techniques [4, 25]. For example, [4] plotted the
average activations of the backdoored model’s last convolutional layer over
clean and backdoored images to explain their attack. [25] used the Grad-CAM
[16] visualization method to explain the backdoor attack in federated
learning. One example of explaining a backdoor attack in the image domain with
Grad-CAM is shown in Fig. 1. Comparing the heatmaps of the clean and poisoned
images on the backdoored model, we can clearly understand how the backdoored
model recognizes the trigger pattern to achieve the backdoor attack. In
contrast, applying visualization techniques to explain the backdoor attack
behavior in the graph domain is difficult. First, the complexity of the visual
representation of a graph is much larger than visualizing an image, especially
for large graphs [6]. Second, visualizing the graph neural networks to explain
the backdoor attack is not trivial as it is a time-consuming or even
impossible process [8].

Therefore, in this work, instead of using the visualization method, we explain
the difference between two trigger-injecting strategies by computing an
evaluation metric. Specifically, we compute the similarity of the predicted
mask of the representative features from the backdoored model and the target
mask of the representative features from the clean model. In our experiments,
we find that the successfully misclassified samples generally have high
similarity while the unsuccessfully misclassified samples have a much lower
similarity. However, we also find that in one specific case, the high
similarity does not lead to a successful attack. We further study this
phenomenon and find that the backdoored model trained by MIAS can recognize
the original feature pattern in addition to the trigger pattern. As there has
been an increasing number of studies on trustworthy GNNs [30, 2, 22], this
paper also contributes to the exploration of GNN robustness by investigating
the effectiveness of backdoor attacks in GNN models using an explainability
tool.

Our work is the first to revisit the trigger-injecting position in graph
backdoor attacks and provide a new perspective. Our key contributions are:

  1. 1.

We investigate backdoor attacks in GNNs by injecting triggers into the most or
least important area of the sample.

  2. 2.

We design a novel explanation framework to analyze the causes of the
difference between these two strategies.

  3. 3.

We verify the difference with quantitative analysis (recall score), which
helps us further understand the backdoor attack behavior in GNNs.

  4. 4.

We explore the interaction between the explainability and robustness of GNNs
through experiments of two datasets and two GNN models.

(a)

(b)

(c)

Figure 1: An example of using Grad-CAM to explain a backdoor attack in the
image domain. (a) clean image, (b) heatmap of clean image for the true label
on the backdoored model (predicted as the true label), (c) heatmap of the
poisoned image for the target label on the backdoored model (predicted as the
target label).

##  II Background

###  II-A Graph Neural Networks

Recently, Graph Neural Networks (GNNs) have achieved significant success in
processing non-Euclidean spatial data, which are common in many real-world
scenarios [33]. Unlike traditional neural networks, e.g., Convolutional Neural
Networks (CNNs) or Recurrent Neural Networks (RNNs), GNNs work on graph data.
GNNs take a graph $G=(V,E,X)$ as an input, where $V,E,X$ denote nodes, edges,
and node attributes, and learn a representation vector (embedding) for each
node $\bm{v}\in G$, $z_{\bm{v}}$, or the entire graph, $z_{G}$.

In particular, in modern GNNs, the node representation is computed by
recursive aggregation and transformation of feature representations of its
neighbors. After $k$ iterations of aggregation, a node’s representation
captures both structure and feature information within its $k$-hop network
neighborhood. Formally, the $k$-th layer of a GNN is:

$x_{\bm{v}}^{(k)}=AGGREGATION^{(k)}(\left\\{z_{\bm{v}}^{(k-1)},\left\\{z_{\bm{u}}^{(k-1)}|\bm{u}\in\mathcal{N}_{\bm{v}}\right\\}\right\\}),$
(1)

$z_{\bm{v}}^{(k)}=TRANSFORMATION^{(k)}(x_{\bm{v}}^{(k)}),$ (2)

where $z_{\bm{v}}^{(k)}$ is the representation of node $\bm{v}$ computed in
the $k$-th iteration. $\mathcal{N}_{\bm{v}}$ are 1-hop neighbors of node
$\bm{v}$, and the $AGGREGATION(\cdot)$ is an aggregation function that can
vary for different GNN models. $z_{\bm{v}}^{(0)}$ is initialized as node
feature. The $TRANSFORMATION(\cdot)$ function consists of a learnable weight
matrix and activation function. For the node classification task, the node
representation $z_{\bm{v}}$ is used for prediction. In this paper, we
investigate the node classification task. Moreover, we focus on two
representation models of this family, which differ in one of the above two
steps: aggregation and transformation. In the following, we briefly describe
these models and their differences.

Graph Convolutional Networks (GCN) [9]. Let $d_{\bm{v}}$ denotes the degree of
node $\bm{v}$. The aggregation operation in GCN is then given as:

$x_{\bm{v}}^{(k)}\leftarrow\sum_{u\in\mathcal{N}_{\bm{v}}\bigcup{\bm{v}}}\frac{1}{\sqrt{d_{\bm{v}}d_{\bm{u}}}}z_{\bm{u}}^{(k-1)}.$

GCN performs a non-linear transformation over the aggregated features to
compute the node representation at layer $k$:

$z_{\bm{v}}^{(k)}\leftarrow ReLU(x_{\bm{v}}^{(k)}W^{(k)}).$

Graph Attention Networks (GAT) [20]. In addition to the standard neighbor
aggregation scheme mentioned above in equation 1 and equation 2, there are
other non-standard neighbor aggregation schemes, e.g., weighted average via
attention in GAT. Specifically, given a shared attention mechanism $a$,
attention coefficients can be computed by:

$e_{\bm{v}\bm{u}}=a(Wz_{\bm{v}}^{(k-1)},Wz_{\bm{u}}^{(k-1)})$ (3)

that indicate the importance of node $\bm{u}$’s features to node $\bm{v}$.
Then, the normalized coefficients can be computed by using the softmax
function:

$\alpha_{\bm{v}\bm{u}}=softmax_{\bm{u}}(e_{\bm{v}\bm{u}}).$ (4)

Finally, the next-level feature representation of node $\bm{v}$ is:

$z_{\bm{v}}^{(k)}=\sigma\left(\frac{1}{P}\sum_{p=1}^{P}\sum_{\bm{u}\in\mathcal{N}_{\bm{v}}}\alpha_{\bm{v}\bm{u}}^{p}W^{p}z_{\bm{u}}^{(k-1)}\right),$
(5)

where $\alpha_{\bm{v}\bm{u}}^{p}$ are the normalized coefficients computed by
the $p$-th attention mechanism $a^{p}$ and $W^{p}$ is the corresponding input
linear transformation’s weight matrix.

###  II-B Backdoor Attacks

Backdoors are training time attacks that aim to achieve misclassification at
the testing phase for trigger-embedded samples while working correctly on
clean inputs. Several studies showed that GNNs are also vulnerable to backdoor
attacks. Similar to the backdoor attack in CNNs, the backdoor attack in GNNs
can be implemented by poisoning the training data with a trigger, which can be
a subgraph with/without features [31, 24] or a subset of node features [26].
After training the GNN model with the trigger-embedded data, the backdoored
GNN would predict the test example injected with a trigger as the pre-defined
target label.

###  II-C Explainability of GNNs

Recently, several explainability techniques in GNNs have been proposed, such
as XGNN [28], GNNExplainer [27], PGExplainer [12], and SubgraphX [29]. These
methods are developed from different angles and provide different levels of
explanations.

GNNExplainer is the model-agnostic approach for providing explanations on any
GNN-based model’s predictions. Given a trained GNN model and its
prediction(s), GNNExplainer returns an explanation in the form of a small
subgraph of the input graph, with a small subset of node features that
contribute most to the final model prediction(s). In this paper, we focus on
the GNNExplainer method as it can explain predictions of any GNN on any graph-
based machine learning task without requiring modification of the underlying
GNN architecture or re-training.

##  III Threat Model

We consider a _gray-box_ threat model assuming the attacker can freely modify
a small portion of the training dataset. Since the explanation masks in
GNNExplainer are generated through gradients of the GNN model, the attacker
also has knowledge of the gradient information of the target model on the
chosen training dataset. We also assume the attacker performs a _dirty-label_
backdoor attack, where the poisoned samples’ labels are changed to the target
label. Although this kind of attack is weaker than _clean-label_ backdoor
attacks [19], where the labels remain unaltered, dirty label attacks are the
most common in the literature [24, 31, 26]. The attacker’s goal is to inject a
backdoor in the given pre-trained clean GNN model through training over the
poisoned training dataset, which achieves misclassification under the presence
of a trigger while maintaining clean high accuracy on the original task. This
threat model is realistic in real-world settings. For example, if the training
dataset is collected from public users, the adversary can provide trigger-
embedded training data to implement the backdoor attack.

##  IV Methodology

###  IV-A General Framework

As stated before, we aim to discover if and how the explainability techniques
in GNNs help improve the performance of backdoor attacks. Here, we focus on
utilizing the feature-trigger backdoor attack from [26] for the node
classification task. The trigger used in the backdoor attacks in our paper is
defined as:

######  Definition 1 (Trigger)

In our backdoor attacks, the trigger is a specific feature pattern that is
created by modifying the value of a subset of a node’s features.

Generally, two steps are conducted to implement backdoor attacks using
explainability techniques:

(1) We apply an explainability technique (i.e., GNNExplainer) on a pre-trained
clean GNN model to implement backdoor attacks based on two trigger-injecting
strategies defined below.

######  Definition 2 (The Most/Least Representative Features)

Through applying the GNNExplainer on the pre-trained clean GNN model on the
target node, we can obtain the original importance order of the node features.
Based on the importance order information, we can locate the most or least
representative features.

######  Definition 3 (Most Important Area Strategy (MIAS))

We select the most representative features of the target node and inject the
feature trigger into the corresponding dimensions.

######  Definition 4 (Least Important Area Strategy (LIAS))

We select the least representative features of the target node and inject the
feature trigger into the corresponding dimensions.

We then compare the attack performance based on these two strategies,
including the attack success rate and clean accuracy drop.

(2) Next, we try to explain the attack performance of these two strategies by
again applying the explainability techniques on the backdoored model over the
poisoned testing dataset. As a result, we can obtain the new importance order
of the node features, which is used to compute the similarity with the
original feature importance order. The proposed framework is presented in Fig.
2.

###  IV-B Explanation Design

The detailed process of generating poisoned training dataset and target masks
is presented in Algorithm 1. $EXP(\cdot)$ is the applied GNN explanation
technique, i.e., GNNExplainer, and $s$ is the trigger-injecting strategies,
i.e., MIAS or LIAS. The algorithm first samples a subset from the original
training dataset with a poisoning rate $r$ (line $2$). For each sampled node,
the algorithm will compute the corresponding feature order to determine the
trigger-injecting location for MIAS and LIAS. Meanwhile, the label of the
poisoned training dataset will change to the target label. The trigger size
$n$ is the number of the features in the feature trigger, which means $n$ node
features will be modified. The poisoned testing dataset is obtained by
injecting a trigger (following the same strategy as the poisoned training
dataset) into the samples and changing their labels to the target label.
Finally, based on the order of representative features, we can generate a
target mask for each node in the poisoned testing dataset (line $17$). The
target mask has the same shape as the node feature vector, and the most
(least) $n$ important features are masked in while other features are masked
out. To evaluate whether the backdoored model can recognize the trigger
pattern precisely, the number of features to be masked in is set to be $n$.
The definition of the target mask is as follows:

######  Definition 5 (Target Mask)

The target mask is a boolean tensor that indicates which $n$ features
contribute more to the final prediction from the pre-trained clean model
$\theta$ for the target node compared to other features.

Input: Pre-trained clean GNN model $\theta$, Training set $D_{train}$, Testing
set $D_{test}$, Trigger-injecting strategy $s\in\left\\{MIAS,LIAS\right\\}$,
Target label $y_{t}\in\left[0,C\right)$

Output: Poisoned training dataset $\hat{D}_{train}^{s}$, Poisoned testing
dataset $\hat{D}_{test}^{s}$, Target masks $M_{t}^{s}$

1 /* Sampling Training Dataset to Inject Trigger */

2 $\hat{D}_{train}^{s}\leftarrow sample(D_{train},r,y\neq y_{t})$

3 foreach  _$\left\\{x,y\right\\}\in\hat{D}_{train}^{s}$_ do

4 /* Computing Order of Representative Features */

5 $feature\\_order=EXP(\theta,x,y)$

6 $\hat{x}^{s}=Inject\\_Trigger(x,feature\\_order,s)$

7 $\hat{y}^{s}=y_{t}$

8

9 end foreach

10$\hat{D}_{test}^{s}\leftarrow D_{test}[\setminus y_{t}]$

11 $M_{t}^{s}\leftarrow\emptyset$

12 foreach  _$\left\\{x,y\right\\}\in\hat{D}_{test}^{s}$_ do

13 /* Computing Order of Representative Features */

14 $feature\\_order=EXP(\theta,x,y)$

15 $\hat{x}^{s}=Inject\\_Trigger(x,feature\\_order,s)$

16 $\hat{y}^{s}=y_{t}$

17 /* Generating Target Mask */

18 $m_{i}^{s}=Get\\_Mask(feature\\_order,s)$

19 $M_{t}^{s}=M_{t}^{s}\cup m_{i}^{s}$

20 end foreach

return $\hat{D}_{train}^{s},\hat{D}_{test}^{s},M_{t}^{s}$

Algorithm 1 Generate Poisoned Training Dataset and Target Masks

Input: Pre-trained clean GNN model $\theta$, Training set $D_{train}$,
Poisoned training dataset $\hat{D}_{train}$, Poisoned testing dataset
$\hat{D}_{test}$, Trigger-injecting strategy $s\in\left\\{MIAS,LIAS\right\\}$

Output: Backdoored GNN model $\hat{\theta}^{s}$, Predicted Masks $M_{p}^{s}$

1 /* Training Backdoored Models */

2 /* $\left\\{x,y\right\\}\in D_{train}$,
$\left\\{\hat{x}^{s},\hat{y}^{s}\right\\}\in\hat{D}_{train}^{s}$ */

3
$\hat{\theta}^{s}=\arg\\!\min_{\theta}(\sum_{i}L(x_{i},y_{i};\theta)+\sum_{i}L(\hat{x}_{i}^{s},\hat{y}_{i}^{s};\theta))$

4 $M_{p}^{s}\leftarrow\emptyset$

5 foreach  _$\left\\{\hat{x}^{s},\hat{y}^{s}\right\\}\in\hat{D}_{test}^{s}$_
do

6 /* Getting Predictive Mask */

7 $feature\\_order=EXP(\hat{\theta}^{s},\hat{x}^{s},\hat{y}^{s})$

8 $m_{i}^{s}=Get\\_Mask(feature\\_order)$

9 $M_{p}^{s}=M_{p}^{s}\cup m_{i}^{s}$

10 end foreach

return $\hat{\theta}^{s},M_{p}^{s}$

Algorithm 2 Train Backdoored GNN Models and Generate Predicted Masks

Once the poisoned training dataset is generated, we can obtain the backdoored
models $\hat{\theta}^{s}$ by retraining the clean model $\theta$ with the
backdoored training dataset. 111In this work, we combine the original training
dataset and the poisoned training dataset as the backdoored training dataset.
The process of training the backdoored models and obtaining predicted masks is
shown in Algorithm 2. To analyze the impact of injecting trigger into the
most/least important part of the node features on the attack performance, we
compare the attack performance of $\hat{\theta}^{MIAS}$ and
$\hat{\theta}^{LIAS}$, including the attack success rate and clean accuracy
drop. Finally, for the poisoned testing dataset, which we used to calculate
the attack success rate, we again utilize the GNNExplainer to obtain the new
feature importance order for each node on the backdoored GNN model
$\hat{\theta}^{MIAS}$ or $\hat{\theta}^{LIAS}$ (line $7$). The new feature
importance order is used to generate the predicted mask. The definition of the
predicted mask is as follows:

######  Definition 6 (Predicted Mask)

The predicted mask is a boolean tensor which indicates which $n$ features
contribute more to the final prediction from the backdoored GNN model
$\hat{\theta}$ for the target node compared to other features.

Combining the target masks we get in Algorithm 1, we can compute the
similarity between the ordering of the new representative features and the old
ones by calculating the recall score of the target mask and the predicted
mask:

$\begin{split}&RS_{i}^{s}=\frac{TP(M_{t,i}^{s},M_{p,i}^{s})}{TP(M_{t,i}^{s},M_{p,i}^{s})+FN(M_{t,i}^{s},M_{p,i}^{s})},i\in
N\\\ &M_{t,i}^{s}(M_{p,i}^{s})=[0,\cdots,1,\cdots,1,\cdots,0],\end{split}$ (6)

where $RS_{i}^{s}$ is the recall score of the $i$th poisoned testing sample
with $s$ strategy, $M_{t,i}^{s}$ and $M_{p,i}^{s}$ is the target mask, and
predictive mask of the $i$th poisoned testing sample, $TP$ and $FN$ is the
true positive and false negative rate of these two masks, respectively, and
$N$ is the number of the poisoned testing dataset. We assume that higher
similarity indicates that the backdoored model can better recognize the
trigger pattern, contributing to better attack performance.

(a) Backdoor attacks based on two strategies.

(b) Procedure of explaining two attack strategies.

Figure 2: An illustration of backdoor attack and explanation framework.

##  V Experimental Results

###  V-A Experimental Setting

We implemented the backdoor attack on the node classification task using the
PyTorch framework. All experiments were run on a server with $2$ Intel Xeon
CPUs, $1$ NVIDIA 1080 Ti GPU with $32$GB RAM, and Ubuntu $20.04$ LTS OS. Each
experiment was repeated $10$ times to obtain the average result.

Dataset. For our experiments, we use two publicly available real-world
datasets for the node classification task: Cora [17] and CiteSeer [17]. These
two datasets are citation networks in which each publication is described by a
binary-valued word vector indicating the absence/presence of the corresponding
word in the collection of $1,433$ and $3,703$ unique words, respectively.

For each node classification dataset, we split $20\%$ of the total nodes as
the original training dataset (labeled), and the rest of the nodes are treated
as the original testing dataset. To generate the backdoored training dataset,
we sample $10\%$ of the original training dataset to inject the feature
trigger and relabel these nodes with the target label. The trigger size is set
to $5\%$ of the total number of node feature dimensions. We set these
parameters as they provided the best results after conducting a tuning phase.

Models and training. We use the popular GAT [20] and GCN [9] models, as these
two methods are commonly-used GNN models for the node classification task. We
train the clean and backdoored GNN models with a learning rate of $0.005$ and
use Adam as the optimizer.

Attack evaluation metrics. To compare the attack performance of MIAS and LIAS,
we utilize two commonly used backdoor attack evaluation metrics:

  1. 1.

Attacks Success Rate (ASR): measures the backdoor performance of the model on
a fully poisoned dataset $\hat{D}$. It is computed as
$ASR=\frac{\sum_{i=1}^{N}\mathbb{I}({\hat{\theta}}(\hat{x_{i}})=y_{t})}{N}$
where ${\hat{\theta}}$ is the poisoned model, $\hat{x_{i}}$ is a poisoned
input, $\hat{x_{i}}\in\hat{D}$, $y_{t}$ is the target class, and $\mathbb{I}$
is an indicator function.

  2. 2.

Clean Accuracy Drop (CAD): measures the effect of the backdoor attack on the
original task. It is calculated by comparing the performance of the poisoned
and clean models on a clean holdout testing set. The accuracy drop should
generally be small to keep the attack stealthy.

###  V-B Results and Analysis

Results. The backdoor attack results on two graph datasets based on two models
and two trigger-injecting strategies are shown in Fig. 3. In particular, the
ASR and CAD of two GNN models on two datasets are presented in Table I. We can
observe that both strategies can achieve a high attack success rate, i.e.,
more than $97\%$, except GCN on the Cora dataset with MIAS. In addition, in
most cases, the ASR of LIAS is slightly higher, around $1\%$, than that of
MIAS. However, for the GCN model on the Cora dataset, the ASR of LIAS is
significantly higher: more than $8\%$, than the MIAS. We can also see that the
CAD for all datasets and models is unnoticeable, and the difference between
the two strategies over CAD is negligible.

(a) Cora.

(b) CiteSeer.

Figure 3: Backdoor attack results of two trigger-injecting strategies. TABLE
I: Backdoor attack performance of MIAS and LIAS (SD: standard deviation). MIAS
---
Dataset | GCN | GAT
|  ASR $\pm$ SD |  CAD $\pm$ SD |  ASR $\pm$ SD |  CAD $\pm$ SD
Cora |  $90.08\%\pm 0.29\%$ |  $0.32\%\pm 0.19\%$ |  $97.91\%\pm 0.12\%$ |  $0.34\%\pm 0.24\%$
CiteSeer |  $97.70\%\pm 0.10\%$ |  $0.32\%\pm 0.17\%$ |  $98.54\%\pm 0.09\%$ |  $0.71\%\pm 0.20\%$
LIAS
Dataset | GCN | GAT
|  ASR $\pm$ SD |  CAD $\pm$ SD |  ASR $\pm$ SD |  CAD $\pm$ SD
Cora |  $98.65\%\pm 0.06\%$ |  $0.27\%\pm 0.21\%$ |  $99.89\%\pm 0.03\%$ |  $0.27\%\pm 0.21\%$
CiteSeer |  $98.96\%\pm 0.07\%$ |  $0.15\%\pm 0.18\%$ |  $99.88\%\pm 0.03\%$ |  $0.80\%\pm 0.17\%$

Analysis. Next, we investigate the reason why the backdoor attack performance
of the LIAS is somewhat higher or significantly higher (for the GCN model on
the Cora dataset) than the MIAS. As mentioned in Section IV, we evaluate the
similarity between the ordering of the new representative features and the old
ones by calculating the recall score of the target mask and the predicted
mask.

The histogram of recall scores over the poisoned testing dataset of all
datasets and models is shown in Fig. 4. We can observe that most poisoned
testing samples have a recall score of more than $0.5$ in both MIAS and LIAS,
which results in a high attack success rate for both strategies. To further
investigate the slight advantage of the LIAS over the MIAS, we split the
poisoned testing samples into two parts, one is misclassified into the target
class successfully, and the other one is not, and compute the recall scores
for these two parts, as shown in Fig. 5. We notice that, generally, the
successfully misclassified nodes have significantly higher recall scores than
those not misclassified into the target class. This phenomenon is consistent
with the assumption mentioned in Section IV, i.e., the higher similarity
between the ordering of the new representative features and that of the
original ones indicates that the backdoored model can recognize the trigger
pattern better. When comparing the second column and the last column of Fig.
5(b), 5(c), and 5(d), we also see that LIAS has fewer nodes with low recall
score than MIAS, which we believe is the reason of higher ASR of LIAS than
MIAS.

In contrast, we surprisingly see that for the GCN model on the Cora dataset
with MIAS, the unsuccessfully misclassified nodes also have a high recall
score as the successfully misclassified nodes. We assume that the main reason
behind this is that, under the MIAS, the feature trigger is injected into the
positions of the most representative features. Thus the backdoored model will
recognize not only the trigger pattern but also the representative feature
pattern for the original label. Therefore, for MIAS, it is possible that even
the poisoned testing samples that are not successfully misclassified into the
target class will have a high recall score. We verify this hypothesis by
extending the target masks and predicted masks twice the feature trigger
length, i.e., $2*n$, and computing the recall scores again. 222Here, we select
an extension rate of $2$. To verify the hypothesis, the extension rate can be
set to $\gamma>1$, and the recall scores of the successfully misclassified
nodes are expected to reduce to $1/\gamma$ of that without extended masks. The
histogram of the new recall scores of the GCN model on the Cora dataset is
shown in Fig. 6. We also checked the prediction of the backdoored model over
the unsuccessfully misclassified nodes. The output indicates that all these
nodes are classified into their original classes. Comparing Fig. 5(a) and 6,
we observe that the recall scores of the successfully misclassified nodes
generally reduce to half of that without extended masks. We believe this is
because, for these nodes, the backdoored model recognizes the trigger location
exactly, and when we extended the masks twice the trigger length, only half of
the features can be recalled. However, we can also see that for the MIAS, the
recall scores of the unsuccessfully classified nodes are still as high as
those without the extended masks. This is because the backdoored model
recognizes the feature pattern for the original label (that is why these nodes
are classified into the original class and the attack is not successful), so
even if the masks are extended, the recall score is still high.

(a) Cora.

(b) CiteSeer.

Figure 4: Histogram of recall scores over the poisoned testing dataset.

(a) Cora-GCN.

(b) Cora-GAT.

(c) CiteSeer-GCN.

(d) CiteSeer-GAT.

Figure 5: Histogram of recall scores over two parts of the poisoned testing
dataset ($S$ means the nodes are successfully misclassified into the target
label, $U$ means not). Figure 6: Histogram of recall scores of GCN model on
Cora dataset with extended masks ($S$ means the nodes are successfully
misclassified into the target label, $U$ means not).

##  VI Related Work

###  VI-A Backdoor Attacks in GNNs

Several works have conducted backdoor attacks in GNNs. Zhang et al. presented
a subgraph-based backdoor attack in GNNs for graph classification task [31].
Xi et al. proposed a subgraph-based backdoor attack in GNNs, for both node
classification and graph classification tasks [24]. Xu et al. explored the
trigger-injecting position for the graph backdoor attack [26], representing
the most related work to our paper. However, in that paper, the authors only
provided assumptions about the results, and no experimental analysis was given
to confirm the assumptions. In this work, we give an empirical analysis of the
attack results, which leads to a further understanding of the backdoor attack
behavior in GNNs.

###  VI-B Explanability in GNNs

GNNs have become increasingly popular since many real-world data can be
naturally represented as graphs, such as social networks, chemical molecules,
and financial data [32, 5]. Consequently, numerous approaches are proposed to
explain the predictions of GNNs. Generally, these methods can be categorized
into two mainstream lines of research. One is the parametric explanation
methods that are widely used nowadays. For instance, GNNExplainer [27] learns
soft masks for edges and node features to explain the predictions via mask
optimization. The soft masks are randomly initialized and treated as trainable
variables. [12] proposed PGExplainer to collectively explains multiple
instances with a probabilistic graph generative model. XGNN [28] uses a graph
generator to generate class-wise graph patterns to explain GNNs for each
class. Vu et al. proposed PGM-Explainer, a Bayesian network on the pairs of
graph perturbations and prediction changes [21].

The other line is the non-parametric explanation methods, which do not involve
any additional trainable models. They employ heuristics like gradient-like
scores obtained by backpropagation as the feature contributions of a specific
instance [1, 14, 15]

###  VI-C Explainability for Backdoor Attacks

With the thriving development of explainability techniques in machine
learning, the attacker can use model explanations to gain knowledge about the
model to perform the adversarial attacks [13]. Kuppa et al. [10] used
counterfactual explanations to find the malware features that most heavily
impact the classifier decision. They used this knowledge to craft adversarial
training samples that efficiently poison the model. Severi et al. [18] used
SHAP to craft backdoor triggers in malware detectors. Utilizing the
explanation, they determined which features to poison, resulting in a success
rate of up to three times higher than that of a greedy algorithm that does not
use explainable artificial intelligence (XAI). Xu et al. [26] injected
backdoors into GNNs by leveraging XAI techniques. While there has been an
increasing number of studies on utilizing explanation techniques to implement
backdoor attacks in deep learning models, there has been no research on using
explanation tools to clarify the backdoor attack behavior in the graph domain.

##  VII Conclusion and Future Work

This paper presents a comprehensive analysis and explanation of graph backdoor
attacks with two trigger-injecting strategies; MIAS and LIAS. We investigate
the node classification task and compare the attack performance for these two
strategies. Our findings show that LIAS always achieves higher attack
performance than MIAS. We further explain the difference with quantitative
analysis, which contributes to a further understanding of the backdoor attack
behavior in GNNs. Future work will include explaining the backdoor attack
behavior of two trigger-injecting strategies in the graph classification task.
More precisely, we would compute the similarity between the new representative
subgraph and the old one by calculating the recall score of the target mask
and the predicted mask.

## References

  * [1] Federico Baldassarre and Hossein Azizpour.  Explainability techniques for graph convolutional networks.  arXiv preprint arXiv:1905.13686, 2019.
  * [2] Enyan Dai, Tianxiang Zhao, Huaisheng Zhu, Junjie Xu, Zhimeng Guo, Hui Liu, Jiliang Tang, and Suhang Wang.  A comprehensive survey on trustworthy graph neural networks: Privacy, robustness, fairness, and explainability.  arXiv preprint arXiv:2204.08570, 2022.
  * [3] Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli.  A fair comparison of graph neural networks for graph classification.  arXiv preprint arXiv:1912.09893, 2019.
  * [4] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg.  Badnets: Evaluating backdooring attacks on deep neural networks.  IEEE Access, 7:47230–47244, 2019.
  * [5] William L Hamilton.  Graph representation learning.  Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1–159, 2020.
  * [6] Yifan Hu and Lei Shi.  Visualizing large graphs.  Wiley Interdisciplinary Reviews: Computational Statistics, 7(2):115–136, 2015.
  * [7] Mengda Huang, Yang Liu, Xiang Ao, Kuan Li, Jianfeng Chi, Jinghua Feng, Hao Yang, and Qing He.  Auc-oriented graph neural network for fraud detection.  In Proceedings of the ACM Web Conference 2022, pages 1311–1321, 2022.
  * [8] Zhihua Jin, Yong Wang, Qianwen Wang, Yao Ming, Tengfei Ma, and Huamin Qu.  Gnnlens: A visual analytics approach for prediction error diagnosis of graph neural networks.  IEEE Transactions on Visualization and Computer Graphics, 2022.
  * [9] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In ICLR, 2017.
  * [10] Aditya Kuppa and Nhien-An Le-Khac.  Adversarial xai methods in cybersecurity.  IEEE transactions on information forensics and security, 16:4924–4938, 2021.
  * [11] Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang, and Qing He.  Pick and choose: a gnn-based imbalanced learning approach for fraud detection.  In Proceedings of the Web Conference 2021, pages 3168–3177, 2021\.
  * [12] Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang.  Parameterized explainer for graph neural network.  Advances in neural information processing systems, 2020.
  * [13] Azqa Nadeem, Daniël Vos, Clinton Cao, Luca Pajola, Simon Dieck, Robert Baumgartner, and Sicco Verwer.  Sok: Explainable machine learning for computer security applications.  arXiv preprint arXiv:2208.10605, 2022.
  * [14] Phillip E Pope, Soheil Kolouri, Mohammad Rostami, Charles E Martin, and Heiko Hoffmann.  Explainability methods for graph convolutional neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10772–10781, 2019.
  * [15] Thomas Schnake, Oliver Eberle, Jonas Lederer, Shinichi Nakajima, Kristof T Schütt, Klaus-Robert Müller, and Grégoire Montavon.  Higher-order explanations of graph neural networks via relevant walks.  IEEE transactions on pattern analysis and machine intelligence, 44(11):7581–7596, 2021.
  * [16] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [17] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad.  Collective classification in network data.  AI magazine, 2008.
  * [18] Giorgio Severi, Jim Meyer, Scott E Coull, and Alina Oprea.  Explanation-guided backdoor poisoning attacks against malware classifiers.  In USENIX Security Symposium, pages 1487–1504, 2021.
  * [19] Alexander Turner, Dimitris Tsipras, and Aleksander Madry.  Clean-label backdoor attacks.  2018\.
  * [20] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph Attention Networks.  ICLR, 2018.
  * [21] Minh Vu and My T Thai.  Pgm-explainer: Probabilistic graphical model explanations for graph neural networks.  Advances in neural information processing systems, 33:12225–12235, 2020.
  * [22] Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang.  Be confident! towards trustworthy graph neural networks via confidence calibration.  Advances in Neural Information Processing Systems, 34:23768–23779, 2021.
  * [23] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  * [24] Zhaohan Xi, Ren Pang, Shouling Ji, and Ting Wang.  Graph backdoor.  In 30th USENIX Security Symposium (USENIX Security 21), pages 1523–1540, 2021.
  * [25] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li.  Dba: Distributed backdoor attacks against federated learning.  In International conference on learning representations, 2020.
  * [26] Jing Xu, Minhui Xue, and Stjepan Picek.  Explainability-based backdoor attacks against graph neural networks.  In Proceedings of the 3rd ACM Workshop on Wireless Security and Machine Learning, 2021.
  * [27] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec.  Gnnexplainer: Generating explanations for graph neural networks.  Advances in neural information processing systems, 32, 2019.
  * [28] Hao Yuan, Jiliang Tang, Xia Hu, and Shuiwang Ji.  Xgnn: Towards model-level explanations of graph neural networks.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 430–438, 2020.
  * [29] Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji.  On explainability of graph neural networks via subgraph explorations.  In International Conference on Machine Learning. PMLR, 2021.
  * [30] He Zhang, Bang Wu, Xingliang Yuan, Shirui Pan, Hanghang Tong, and Jian Pei.  Trustworthy graph neural networks: Aspects, methods and trends.  arXiv preprint arXiv:2205.07424, 2022.
  * [31] Zaixi Zhang, Jinyuan Jia, Binghui Wang, and Neil Zhenqiang Gong.  Backdoor attacks to graph neural networks.  In Proceedings of the 26th ACM Symposium on Access Control Models and Technologies, pages 15–26, 2021.
  * [32] Ziwei Zhang, Peng Cui, and Wenwu Zhu.  Deep learning on graphs: A survey.  IEEE Transactions on Knowledge and Data Engineering, 34(1):249–270, 2020.
  * [33] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  AI open, 1:57–81, 2020.


# Observation of magnetic skyrmions in unpatterned symmetric multilayers at
room temperature and zero magnetic field

J. Brandão  Laboratório Nacional de Luz Síncrotron, Centro Nacional de
Pesquisa em Energia e Materiais, 13083-970 Campinas SP, Brazil  D. A. Dugato
Laboratório Nacional de Luz Síncrotron, Centro Nacional de Pesquisa em Energia
e Materiais, 13083-970 Campinas SP, Brazil  Departamento de Física,
Universidade Federal de Santa Maria, 97105-900 Santa Maria RS, Brazil  R. L.
Seeger  Departamento de Física, Universidade Federal de Santa Maria, 97105-900
Santa Maria RS, Brazil  J. C. Denardin  Departamento de Física, Universidade
Federal de Santa Maria, 97105-900 Santa Maria RS, Brazil  Departamento de
Física and CEDENNA, Universidad de Santiago de Chile, 9170124 Santiago, Chile
T. J. A. Mori  Laboratório Nacional de Luz Síncrotron, Centro Nacional de
Pesquisa em Energia e Materiais, 13083-970 Campinas SP, Brazil  J. C. Cezar
Laboratório Nacional de Luz Síncrotron, Centro Nacional de Pesquisa em Energia
e Materiais, 13083-970 Campinas SP, Brazil

(August 27, 2024)

###### Abstract

Magnetic skyrmions are promising candidates for the next generation of
spintronic devices due to their small size and topologically protected
structure. One challenge for using these magnetic states in applications lies
on controlling the nucleation process and stabilization that usually requires
an external force. Here, we report on the evidence of skyrmions in unpatterned
symmetric Pd/Co/Pd multilayers at room temperature without prior application
of neither electric current nor magnetic field. By decreasing the
ferromagnetic interlayer thickness, the tuning of the physical properties
across the ferromagnetic/non-magnetic interface gives rise to a transition
from worm like domains patterns to isolated skyrmions as demonstrated by
magnetic force microscopy. On the direct comparison of the measured and
simulated skyrmions size, the quantitative interfacial Dzyaloshinskii-Moriya
interaction (iDMI) was estimated, reveling that isolated skyrmions are just
stabilized at zero magnetic field for low values of iDMI. Our findings provide
new insights towards the use of stabilized skyrmions for room temperature
devices in nominally symmetric multilayers.

Figure 1:  Magnetization curves and MFM images. a, hysteresis curves evolve
from tail-like to square loop for decreasing Co thicknesses. b-e, The MFM
images acquired for different Co thicknesses in the as-gown state revealing
the transition from small up and down domains b-c, to isolated skyrmions e.
Red and blue contrasts represent the magnetization out-of-plane of the film.

The full control of the magnetization processes in small-sized magnetic states
is essential to increase the density of information in magnetic memories
jeovani1 ; jeovani2 ; jeovani6 . Magnetic skyrmions, i.e. nanometer-sized
topological defects presenting swirling spin textures, are promising
candidates for achieving efficiency and functionality towards room temperature
devices review ; jeovani8 ; jeovani9 ; jeovani9a ; jeovani9c . Currently, the
standard heterostructures for studying skyrmions are heavy metal/ferromagnetic
(HM/FM) interfacial systems, with structural inversion asymmetry in ultrathin
films fabricated by magnetron sputtering. The strong spin orbit coupling (SOC)
of the HM layer can lead to an antisymmetric exchange known as interfacial
Dzyaloshinskii-Moriya interaction (iDMI) jeovani19 ; jeovani20 , which plays a
key role in the stabilization of chiral spins textures such as skyrmions
jeovani21 ; jeovani22 ; jeovani7 . The iDMI emerges in HM/FM interfaces owing
broken spatial inversion symmetry which determines its sign and direction,
whilst the iDMI magnitude depends on the SOC jeovani23 ; jeovani24 ; jeovani25
. In this sense, different combinations of HM/FM interfaces have been
investigated to obtain strong iDMI amplitudes and distinct signs to stabilize
skyrmions and define their chirality jeovani26 ; jeovani27 ; chiral . Although
isolated skyrmions have been observed recently at room temperature in this
kind of systems, their nucleation and stabilization in most cases require
external magnetic field and/or electrical current jeovani28 ; jeovani29 ;
jeovani30 . It would be interesting also to nucleate skyrmions without the
need of any external force (magnetic field or current). This was obtained
using lithographically made structures to confine single or multiple skyrmions
depending on the geometric parameters of the patterned samples jeovani31 ;
jeovani32 . It remains to be demonstrated that one can observe the spontaneous
formation of skyrmions, even without nanostructured confinement.

In this letter, we demonstrate that magnetic skyrmions can be stabilized at
room temperature in unpatterned samples without the need for any preceding
external excitation nor geometric confinement. We observe the formation of
skyrmions in both as-grown (not exposed to the magnetic field) and remnant
states of nominally symmetric Pd/Co/Pd multilayers when the Co thickness is as
thin as the percolation threshold of a continuous layer. The tuning of the
magnetic properties at the HM/FM interfaces by simply thinning the FM layer
leads to a transition of the magnetic domains pattern from a labyrinthine
state, passing through a state with long and separated stripes, then reaching
isolated skyrmions for the lowest Co thickness. The observation of skyrmions
is investigated mainly by magnetic force microscopy (MFM). We verify their
reproducibility and density along the unpatterned thin film by imaging
different regions of the sample to obtain the skyrmions average size. Besides,
magnetization curves show the dependence of the perpendicular magnetic
anisotropy (PMA) and remnant magnetization with the Co thickness, providing a
direct link to understand the magnetic textures observed in the MFM images.
Micromagnetic simulations are also carried out to elucidate the role of each
magnetic parameter upon the skyrmions stability at room temperature and zero
external magnetic field.

We prepared twin samples in the same sputtering runs (see methods), in order
to take MFM images at zero magnetic field and to measure hysteresis loops. The
out-of-plane magnetization curves (normalized to the saturation magnetization
Ms) are summarized in Fig. 1a. For thicker Co samples (0.8 and 0.6 nm) the
hysteresis show a tail feature, whilst thinner (0.4 and 0.2 nm) present a more
square-shaped format. In-plane hysteresis loops were also performed (see
supplementary information S1).

To image the magnetic domains patterns without any prior applied magnetic
field, MFM images were firstly acquired in samples in the as-grown state. For
Co (0.8 and 0.6 nm), the MFM images show that the magnetization is broken in
small domains (Fig. 1b-c). More specifically, for Co (0.6 nm) the magnetic
domains exhibit a clear worm-like configuration in the so-called labyrinthine
state (Fig. 1c). Reducing the Co thickness to 0.4 nm, the magnetic domain
pattern shows long and separated stripes (Fig. 1d). It is very interesting
that some skyrmions are clearly observed among both the worm-like and the long
stripes patterns discussed so far, see the indications by dashed black arrows
in Fig. 1c-d. In a previous work, similar skyrmions among the worm-like domain
patterns were observed by MFM images jeovani35 .

Figure 2:  Representative MFM images of isolated skyrmions. a-c, in the as-
grown state. e-g, in the remnant state. Isolated skyrmions are observed at
zero magnetic field over different places on the sample surface. d-h, the
profile shows the shape of the isolated skyrmions. Most notable, the skyrmions
size is larger in the as-grown than the remnant state. The MFM images obtained
in different locations highlight the homogeneity on the skyrmions formation.

By further thinning the Co layer, the transition from worm-like to long
stripes evolves to a new magnetic domain pattern for the thinnest Co (0.2 nm).
As it is shown in Fig. 1e, many isolated skyrmions are observed. These images
reveal that no prior stabilizing field or injection current is required to
generate isolated skyrmions.

The MFM image shown in Fig. 1e was acquired for Co (0.2 nm) sample which has
never submitted to external magnetic field. However, the hysteresis loops
measured for one identical sample, can be used to explain this unexpected
behavior. The Co (0.2 nm) is the sample whose out-of-plane hysteresis loop is
the most square-shaped and which presents the highest remnant state and PMA
(Fig. 1a). Thus, at zero magnetic field, owing high remanence and PMA, the
magnetic pattern would be more favorable oriented out-of-plane in a quasi-
uniform magnetization state. Instead, in our MFM images, skyrmions appear
before reaching this ground state, suggesting that carefully controlling the
transition from worm-like to single domain varying the Co interlayer
thickness, it is feasible to create isolated stabilized skyrmions. Therefore,
this combination of PMA and remnant magnetization emerges as one of the main
sources for the generation of skyrmions at room temperature and zero magnetic
field in Pd/Co/Pd symmetric multilayers.

On the other hand, it is well known that skyrmions are stabilized by the
presence of interfacial Dzyaloshinskii-Moriya interaction (iDMI) jeovani26 ,
which occurs in systems with structural inversion asymmetry jeovani39 ;
jeovani40 . Here, ideal symmetric Pd/Co/Pd multilayers should not have
structural inversion asymmetry, so both Co/Pd and Pd/Co interfaces should
contribute with iDMIs of same amplitude but opposite signs, hence leading to a
null liquid iDMI strength. However, it was observed that crystallographic
asymmetry between Pt/Co and Co/Pt interfaces in Pt/Co/Pt systems can give rise
to liquid iDMI jeovani41 . Also, the total magnetic moment induced in the Pd
is larger at the top Co/Pd interface than at the bottom Pd/Co, leading to an
asymmetric magnetic proximity effect in Pd/Co/Pd trilayers jeovani42 . This
behavior may also contribute to iDMI in symmetric HM/FM/HM systems as the
proximity effect might have a correlation with iDMI as suggested in the ref.
jeovani43 . Furthermore, iDMI has been confirmed as the origin of asymmetric
domain wall creep measured in symmetric Pd/Co/Pd multilayers by means of polar
Kerr images jeovani34 . Supported by these considerations and references, we
consider that our symmetric Pd/Co/Pd multilayers may also present weak iDMI.
Indeed, we will show through the correlation between micromagnetic simulations
and MFM images that isolated skyrmions at zero magnetic field are just
stabilized in symmetric multilayers considering a weak but no-null iDMI.

A question that emerges here is whether the transition from worm-like to
isolated skyrmions can be observed over the entire unpatterned sample. To
verify this point, several MFM images were acquired on different regions of
each sample (see methods). To illustrate, 3 different images are shown in Fig.
2a-c, all of them exhibit many skyrmions, strengthening that their spontaneous
formation is not an isolated case but rather is reproducible over different
areas of the sample surface.

So far we demonstrated the observation of magnetic skyrmions at room
temperature without any prior applied magnetic field. In order to be useful
for applications, the skyrmions should survive also after being subjected to
an external magnetic field. The samples were then imaged by MFM in the remnant
state after the measurements of the hysteresis loops shown in Fig. 1a. The
images of the samples Co (0.8 nm,0.6 and 0.4 nm), see (supplementary
information S2), present worm-like and long stripe patterns with a few
isolated skyrmions, being very similar to the ones observed for the as-grown
samples. The same tendency of similar magnetic features was observed for the
thinnest Co (0.2 nm), whose isolated skyrmions can be visualized in Fig. 2e-g.
Similarly to the as-grown state images, these 3 MFM measurements are
representative of the magnetic domains patterns over the entire unpatterned
sample and proves that skyrmions can be generated in symmetric Pd/Co/Pd
multilayers equally in its remnant state after magnetization cycles.
Notwithstanding, it is worth noting that the isolated skyrmions presented in
the Fig. 2 are randomly distributed and their size and shape are not
identical. We suggest that this distribution arises from minor structural and
morphological inhomogeneities along the different regions of the film as the
Co layers are ultrathin. Besides, the variation of magnetic parameters such as
$M_{s}$, PMA, and iDMI also contributes to the different size and shape of
stabilized skyrmions skyrmions200nm .

Figure 3:  Simulated stabilized skyrmions at zero magnetic field for $D$ = 0.4
mJm-1. a-c, the skyrmions are isolated and randomly distributed as it was
observed experimentally. The saturation magnetization parameter was varied
showing that the skyrmions becomes larger as $M_{s}$ increases. In d, the
skyrmions size extracted from the simulations to different $D$ and $k$. The
skyrmions is larger when $k$ and $D$ are reduced for a ranging of $M_{s}$
values.

Exploring the formation of skyrmions for the Co (0.2 nm) sample, we obtained
their average size and density distribution for both as-grown and remnant
states analyzing 10$\times$10 $\mu$m2 MFM images from 10 different regions on
the film. We used the full width at half maximum (FWHM) size , extracted from
the line scan of 5 isolated skyrmion at each image to determine its size, as
it is illustrated in the Fig. 2d-h for an isolated skyrmion in the as-grown
and other in the remnant state, respectively. The as-grown skyrmions present
an average size of 314 nm with a standard deviation of 41 nm, and are
distributed with an average density of $\sim$ 0.3 skyrmions per $\mu$m2. The
skyrmions detected in the remnant state, in turn, exhibit a reduced average
size of $\sim$ 165 nm with a standard deviation of 32 nm and a density of 0.25
skyrmions per $\mu$m2.

The difference in the skyrmions size and density after submitting the sample
to magnetic field represents a reduction of $\sim$ 55% and $\sim$ 18%,
respectively. This comes from the fact that the remanence is larger than the
magnetization in as-grown state, what leads to smaller size and density of the
skyrmions. Nonetheless, this observation demonstrates that skyrmions in these
samples are homogeneous and robust, persisting even after the application of
an external magnetic field. These skyrmions observed in larger areas in
combination with patterned nanostructures can be used as a source to feed
constrictions in technological applications large . The advantage would be the
fact that they do not need any stabilizing field to transform magnetic domains
in a worm like configuration into skyrmions. Therefore, they can be
manipulated with ultra-low density current which is beneficial to perform low-
energy spintronics devices.

In order to understand the formation of stabilized skyrmions at zero magnetic
field, micromagnetic simulations were carried out using Mumax3 code Mumax
(see methods). The physical parameter extracted from the hysteresis loops was
the magnetization saturation $M_{s}$. More specifically, for Co 0.2 nm, the
measured $M_{s}$ was $\sim$ 280 kA/m. The value is consistent with previous
reported on Pd/Co multilayers jeovani45 ; jeovani46 ; jeovani47 . We varied
the magnetization saturation $M_{s}$ and magnetic anisotropy $k$ to understand
the impact of these properties on the skyrmions stability considering a weak
Dzyaloshinskii-Moriya below $D$ = 0.8 mJm-2.

We start by showing examples of micromagnetic simulations on the stabilized
skyrmions at zero magnetic field considering a constant $D$ = 0.4 mJm-2. Fig.
3 a-c shows isolated skyrmions when $k$ is 0.06 MJm-3 and $M_{s}$ ranging from
270 kAm-1 to 280 kAm-1. Most notable, the higher the $M_{s}$ the larger the
skyrmions size, suggesting that local variations in the magnetic properties
($k$, $M_{s}$ and $D$) beyond the non-uniformity interfaces play also a role
in modifying the skyrmions size. Indeed, this behavior is summarized in Fig. 3
d. Varying $M_{s}$, we obtained simulated skyrmions size ranging from $\sim$
50 nm to $\sim$ 290 nm considering different values of $D$ and $k$. It
indicates that larger skyrmions are stabilized when both $D$ and $k$ are
reduced while $M_{s}$ is increased. Comparing these results with the skyrmions
size measured in the remnant and as grown state, it represents a good
agreement. This direct comparison between simulation and experimental on the
skyrmions size, allowed us quantitatively estimate the iDMI in our sample as
$D$ = 0.4 - 0.6 mJm-2. Similar results were obtained by performing simulations
using negative $D$ values, see (supplementary information S3).

Figure 4: Simulated phase diagram. The ground state was determined by
modifying $D$ and $M_{s}$ for different $k$. a, below $D$ = 0.2 mJm-1 the
uniform magnetization pattern is the ground state. Above $D$ = 0.2 mJm-1
isolated skyrmions are stabilized for $M_{s}$ ranging from 270 - 295 Am-1 and
a mixed phase (skyrmions and stripes) is stabilized for higher $D$ . b and c,
shows that these magnetic states are also observed when DMI becomes larger.
Representative configurations with mixed phase, isolated skyrmions and uniform
magnetization are also presented.

Finally, we performed further simulations to build up a phase diagram to
explore the physical parameters needed to stabilize skyrmions at zero magnetic
field. The simulated $D$ $vs$ $M_{s}$ phase diagram is shown in Fig. 4. It
distinguishes three different phases as the ground state. In Fig 4 a, the
mixed phase, in which isolated skyrmions and magnetic stripes coexist, is more
favorable independently of $M_{s}$ when $D$ is higher than 0.4 mJm-1. A
representative configuration of this mixed phase is shown on the bottom of the
Fig 4 (a). Between $D$ = 0.2 - 0.4 mJm-1 for low values of $M_{s}$, a narrow
region is observed which isolated skyrmions are stabilized. For lower $D$
values below 0.2 mJm-1, the uniform magnetization is the ground state.
Interestingly, increasing $k$, the region with stabilized skyrmions is
observed in a wider region, see Fig 4 (b). A representative configuration of
this stabilized skyrmions is shown on the bottom of the Fig 4 (b). Besides, it
is noted that the uniform magnetization pattern increases while the mixed
phase is reduced. For higher $k$ the isolated skyrmions are stabilized for
larger $D$, see Fig 4 (c). The mixed phase is drastically reduced, while the
uniform magnetization is mostly observed as the ground state. A representative
configuration of the uniform magnetization is shown on the bottom of the Fig 4
(c). The diagram shows how the physical parameters can be tailored in order to
achieve the desired magnetic pattern, which is an important mechanism to
designing room-temperature skyrmions based devices.

In conclusion, we investigated the formation of magnetic domains patterns in
Pd/Co/Pd multilayers using MFM images. By thinning the Co interlayer we
observed a transition from worm-like magnetic domains pattern to isolated
skyrmions at room temperature and zero magnetic field. Stabilized skyrmions
were observed in samples in both as-grown and remnant states, and their
homogeneity was proved by several MFM images taken over different areas of the
sample surface. Comparing the measured and simulated skyrmions size, the
quantitative interfacial Dzyaloshinskii-Moriya interaction iDMI was estimated
with $D$ ranging from 0.4 - 0.6 mJm-1 as one of the key physical properties on
the skyrmions stability. The symmetric multilayers grown on Si substrate by
magnetron sputtering are very suitable to host skyrmions in unpatterned
samples without any stabilizing field, providing a new path towards skyrmions-
based room temperature devices. They can be used in areas exceeding
10$\times$10 $\mu$m2, and eventually manipulated with ultra-low current
density in order to perform devices combined with narrow constrictions without
the need to transform magnetic worm-like domains in skyrmions.

##  I Methods

###  I.1 Film deposition and magnetic characterization

The Pd/Co/Pd multilayers were grown onto Si substrates by magnetron sputtering
at room temperature and deposition pressure of 2 mTorr in the argon
atmosphere. Two identical samples were grown in each sputtering process in
order to acquire MFM images at zero magnetic field and also magnetization
curves. To increase the magnetic contrast for MFM measurements and the signal-
to-noise ratio for magnetic hysteresis curves, fifteen repetitions of the
trilayers were grown. X-rays reflectometry (XRR) measurements were performed
to verify the Pd and Co thicknesses. The magnetization reversal as a function
of magnetic field was acquired by using an alternating gradient field
magnetometer (AGFM) and vibrating-sample magnetometer (VSM).

###  I.2 Magnetic Force Microscopy Images

MFM images were carried out using a Nanosurf FlexAFM microscope. We used
MagneticMulti75-G MFM tips from Budget Sensors, they are coated by a cobalt
alloy having magnetic moment of roughly 10-16 Am2 and coercivity of roughly
0.03 T. We operated the MFM measurements at the dynamic force mode with a
resonant frequency of about 75 kHz. The images were acquired in the tip-
surface distance of 60 nm. To acquire images in different areas, we moved the
sample over distances of 1 mm in order to confirm the homogeneity of the
domain magnetic patterns.

###  I.3 Micromagnetic Modeling

Micromagnetic simulations were performed using the Mumax3 GPU-accelerated
program over an area of 1.5$\times$1.5 $\mu$m2 discretized in cells size of
3$\times$3$\times$3 nm3. The magnetic parameters kept fixed exchange stiffness
$A_{exch}$ = 15 pJm-3 and Gilbert damping $\alpha$ = 0.3. The magnetic
anisotropy $k$, magnetization saturation $M_{s}$, and Dzyaloshinskii-Moriya
vector $D$ were modified to understand their impact on the skyrmions
stability. To obtain the magnetic ground state, the initial magnetization is
chosen randomly to seek for the same aleatory distribution of the experimental
observed skyrmions. Subsequently, the magnetization is left to relax for 1
$\mu$s, then by minimizing the energies involved, reach an equilibrium
condition which represents the skyrmions stability. The long time 1 $\mu$s, to
carry out the simulations, was chosen in order to verify if the skyrmions
would persist after being stabilized. No transformation after their stability
was observed. To clarify the influence of the magnetic parameters in the
stabilized skyrmions shown in the phase diagram, further simulations were done
using negative values of iDMI. Comparable results were obtained regarding the
ferromagnetic order and skyrmions stability.

##  II Acknowledgments

This work was partially supported by the Brazilian agencies FAPESP (Project
No. 2012/51198-2), CAPES and CNPq. J.B., D.A.D. and R.L.S. gratefully
acknowledge fellowships provided by CAPES. J.C.C. acknowledges support from
CNPq (Project No. 309354/2015-3). The authors also thank the Scientific
Computing Group and the XRD2 beamline of Brazilian Synchrotron Light
Laboratory (LNLS/CNPEM) for the computer facilities to undertake the
micromagnetic simulations and some of the XRR measurements, respectively.

##  III Author Contributions

J.B, T.J.A.M and J.C.C conceived the project. D.A.D and R.L.S grew the samples
and measured the hysteresis curves in the (AGFM) with the support of J.C.D.
J.B carried out the MFM images and micromagnetic simulations. J.B, T.J.A.M and
J.C.C prepared the manuscript. All authors participated and discussed on the
manuscript.

## References

  * (1) Fert, A., Cros, V. and Sampaio, J. Skyrmions on the track. Nat. Nanotechnol. 8, 152–156 (2013).
  * (2) X. Zhang, G. P. Zhao, Hans Fangohr, J. Ping Liu, W. X. Xia, J. Xia and F. J. Morvan. Skyrmion-skyrmion and skyrmion-edge repulsions in skyrmion-based racetrack memory, Sci Rep 5, 7643 (2015).
  * (3) Krause, S., Wiesendanger, R. Spintronics: Skyrmionics gets hot. Nature Materials, 15 (5), pp. 493-494 (2016).
  * (4) Naoto Nagaosa and Yoshinori Tokura Topological properties and dynamics of magnetic skyrmions. Nat. Nanotechnol. 8, 899–911 (2013)
  * (5) Fert, A., Reyren, N. and Cros, V. Magnetic skyrmions: advances in physics and potential applications. Nat. Rev. Mat. 2, 17031 (2017).
  * (6) Jiang, W. et al. Skyrmions in magnetic multilayers. Phys. Rep. 704, 1–49 (2017).
  * (7) Sampaio, J., Cros, V., Rohart, S., Thiaville, A. and Fert, A. Nucleation, stability and current-induced motion of isolated magnetic skyrmions in nanostructures. Nat. Nanotechnol. 8, 839–844 (2013).
  * (8) Guoqiang Yu, Pramey Upadhyaya, Qiming Shao, Hao Wu, Gen Yin, Xiang Li, Congli He, Wanjun Jiang, Xiufeng Han, Pedram Khalili Amiri, and Kang L. Wang. Room-Temperature Skyrmion Shift Device for Memory Application, Nano Lett 17, 261–268 (2017).
  * (9) Dzyaloshinsky, I. A thermodynamic theory of weak ferromagnetism of antiferromagnetics. J. Phys. Chem. Solids 4, 241–255 (1958).
  * (10) Moriya, T. Anisotropic superexchange interaction and weak ferromagnetism. Phys. Rev. 120, 91–98 (1960).
  * (11) A. Thiaville et al. Dynamics of Dzyaloshinskii domain walls in ultrathin magnetic films. Europhys. Lett. 100, 57002 (2012).
  * (12) Wiesendanger, R. Nanoscale magnetic skyrmions in metallic films and multilayers: a new twist for spintronics. Nat. Rev. Mat. 1, 16044 (2016).
  * (13) Yoshimura, Y. et al. Soliton-like magnetic domain wall motion induced by the interfacial Dzyaloshinsky-Moriya interaction. Nat. Phys 12, 161 (2016).
  * (14) Stashkevich, A. A. et al. Experimental study of spin-wave dispersion in Py/Pt film structures in the presence of an interface Dzyaloshinskii-Moriya interaction. Phys. Rev. B 91, 214409 (2015).
  * (15) Yang, H., Thiaville, A., Rohart, S., Fert, A. and Chshiev, M. Anatomy of Dzyaloshinskii-Moriya Interaction at Co/Pt Interfaces. Phys. Rev. Let 115, 267210 (2015).
  * (16) Jaehun Cho, N.-H. et al. Thickness dependence of the interfacial Dzyaloshinsky-Moriya interaction in inversion symmetry broken systems. Nat Commun 6, 7635 (2015).
  * (17) C. Moreau-Luchaire at al. Additive interfacial chiral interaction in multilayers for stabilization of small individual skyrmions at room temperature. Nat. Nanotech 11, 444–448 (2016).
  * (18) Anjan Soumyanarayanan et al. Tunable room-temperature magnetic skyrmions in Ir/Fe/Co/Pt multilayers. Nat. Mater 16, 898–904 (2017).
  * (19) J. C. Chauleau et al. Chirality in Magnetic Multilayers Probed by the Symmetry and the Amplitude of Dichroism in X-Ray Resonant Magnetic Scattering Phys. Rev. Lett. 120, 037202 (2018).
  * (20) Gong Chen, Arantzazu Mascaraque, Alpha T. N’Diaye, and Andreas K. Schmid. Room temperature skyrmion ground state stabilized through interlayer exchange coupling, Appl. Phys. Lett. 102, 222405 (2013).
  * (21) A. Hrabec, J. Sampaio, M. Belmeguenai, I. Gross, R. Weil, S.M. Cherif, A. Stashkevich, V. Jacques, A. Thiaville and S. Rohart, Current-induced skyrmion generation and dynamics in symmetric bilayers, Nat. Comm 8, 15765 (2017).
  * (22) Seonghoon Woo et al. Observation of room-temperature magnetic skyrmions and their current-driven dynamics in ultrathin metallic ferromagnets. Nat. Mat 15, 501–506 (2016).
  * (23) O. Boulle at al. Room-temperature chiral magnetic skyrmions in ultrathin magnetic nanostructures, Nanotech 11, 449–454 (2016).
  * (24) Pin Ho, Anthony K.C. Tan, S. Goolaup, A.L. Gonzalez Oyarce, M.Raju, L.S. Huang, Anjan Soumyanarayanan, C. Panagopoulos, Sub-100 nm Skyrmions at Zero Magnetic Field in Ir/Fe/Co/Pt Nanostructures, arXiv:1709.04878 (2017).
  * (25) William Legrand, et al. Room-Temperature Current-Induced Generation and Motion of sub-100 nm Skyrmions, Nano Lett 17, 2703–2712 (2017).
  * (26) Fert A. Magnetic and transport properties of metallic multilayers Mater. Sci. Forum, 59–60 (1990), pp. 439-480 CrossRefView Record in Scopus.
  * (27) Crepieux A., Lacroix C. Dzyaloshinsky-Moriya interactions induced by symmetry breaking at a surface J. Magn. Magn. Mater 198, 341-349 (1998).
  * (28) Hrabec, A. et al., Measuring and tailoring the Dzyaloshinsky-Moriya interaction in perpendicularly magnetized thin films. Phys. Rev. B 90, 020402 (2014).
  * (29) Kim, D.O. et al. Asymmetric magnetic proximity effect in a Pd/Co/Pd trilayer system. Sci. Rep. 6, 25391 (2016).
  * (30) Ryu, K.S., Yang, S.-H., Thomas, L. and Parkin, S. S. P. Chiral spin torque arising from proximity-induced magnetization. Nat. Commun. 5, 3910 (2014).
  * (31) Shawn D. Pollard, Joseph A. Garlow, Jiawei Yu, Zhen Wang, Yimei Zhu and Hyunsoo Yang. Observation of stable Néel skyrmions in cobalt/palladium multilayers with Lorentz transmission electron microscopy, Nat. Comm 8, 14761 (2017).
  * (32) A. Kolesnikov et al. arXiv:1709.01229 (2017).
  * (33) N. Romming et al. Field-Dependent Size and Shape of Single Magnetic Skyrmions. Phys. Rev. Lett. 114, 177203 (2015).
  * (34) Jiang, W. et al. Blowing magnetic skyrmion bubbles. Science 349, 283 – 286 (2015).
  * (35) Arne Vansteenkiste et al. The design and verification of mumax, AIP Advances 4, 107133 (2014).
  * (36) J. M. Shaw, H. T. Nembach, T. J. Silva, S. E. Russek, R. Geiss, C. Jones,N. Clark, T. Leo, and D. J. Smith. Effect of microstructure on magnetic properties and anisotropy distributions in Co/Pd thin films and nanostructures Phys. Rev. B 80, 184419 (2009).
  * (37) Z. Liu, R. Brandt, O. Hellwig, S. Florez, T. Thomson, B. Terris, and H.Schmidt,Thickness dependent magnetization dynamics of perpendicular anisotropy Co/Pd multilayer films. J. Magn. Magn. Mater. 323, 1623 (2011).
  * (38) Y. Kachlon et al., Extracting magnetic anisotropy energies in Co/Pd multilayers via refinement analysis of the full magnetoresistance curves. J. Appl. Phys. 115, 173911 (2014).


# Development of a Novel Quantum Pre-processing Filter to Improve Image
Classification Accuracy of Neural Network Models

###### Abstract

This paper proposes a novel quantum pre-processing filter (QPF) to improve the
image classification accuracy of neural network (NN) models. A simple four
qubit quantum circuit that uses Y rotation gates for encoding and two
controlled NOT gates for creating correlation among the qubits is applied as a
feature extraction filter prior to passing data into the fully connected NN
architecture. By applying the QPF approach, the results show that the image
classification accuracy based on the MNIST (handwritten 10 digits) and the
EMNIST (handwritten 47 class digits and letters) datasets can be improved,
from 92.5% to 95.4% and from 68.9% to 75.9%, respectively. These improvements
were obtained without introducing extra model parameters or optimizations in
the machine learning process. However, tests performed on the developed QPF
approach against a relatively complex GTSRB dataset with 43 distinct class
real-life traffic sign images showed a degradation in the classification
accuracy. Considering this result, further research into the understanding and
the design of a more suitable quantum circuit approach for image
classification neural networks could be explored utilizing the baseline method
proposed in this paper.

Farina Riaz, Shahab Abdulla, Hajime Suzuki, Srinjoy Ganguly, Ravinesh C. Deo,
and Susan Hopkins

##  1 Introduction

The application of quantum computing to the tasks of machine learning, herein
referred as quantum machine learning, has attracted much research attention in
recent years. Literature surveys on quantum machine learning can be found in
[1, 10, 4]. Among many proposals to combine classical machine learning methods
with quantum computing, quanvolutional neural network (QNN) proposed by
Henderson et. al. [6] has an advantage of being implementable on quantum
circuits with a smaller number of qubits with shallow gate depths and yet
being applicable to practical applications. QNN utilizes quantum circuits as
transformation layers, called quanvolutional layer, to extract features for
the purpose of image classification using convolutional neural networks
(CNNs). In [6], MNIST handwritten 10 digit dataset [8] was applied to QNN
using 9 qubits. The results showed classification accuracy improvement using
QNN over CNN. However, when the quanvolutional layer of QNN was replaced by a
conventional convolutional layer, no improvement in classification accuracy
was observed. Henderson et. al. later updated QNN and implemented on Rigetti
Computing’s Aspen-7-25Q-B quantum processing unit which has 25 qubits with 24
programmable two-qubit gate interactions [5]. The proposed method was applied
to 4 class low resolution satellite image dataset. However, no improvement in
classification accuracy by QNN over CNN was observed in [5].

An implementation of QNN on a software quantum computing simulator, PennyLane
[2], was provided by Mari [9]. Mari’s implementation of QNN differs from that
of Henderson in that the output of the quantum circuit, which is a set of
expectation values, is directly fed into the following neural network layer,
while that of Henderson was made into a single scalar value by a classical
method. The proposed method was applied to MNIST dataset using 50 training and
30 test image set. No clear improvement in classification accuracy by QNN over
NN was observed in [9].

Inspired by Henderson’s and Mari’s QNNs, we propose a new method, herein
called quantum pre-processing filter (QPF), that shows clear improvements in
image classification accuracy over conventional neural networks. QPF uses a
quantum circuit with four qubits, four Y rotations, two controlled NOTs
(CNOTs), and four measurements. When QPF is applied as a pre-processing unit
of an image classification neural network, i.e. as a feature extraction
filter, the image classification accuracy of fully connected neural network
against MNIST and EMNIST (handwritten 47 class digits and letters, [3])
improves from 92.5% to 95.4% and from 68.9% to 75.9%, respectively. These
improvements were obtained without introducing any extra parameters to
optimize in machine learning process. Unlike other quantum machine learning
methods, the use of QPF does not require optimization of the parameters inside
the quantum circuits and hence requires only a limited use of the quantum
circuit. Given the small number of qubits and relatively shallow depth of the
quantum circuit, QPF is well suited to be implemented on noisy intermediate-
scale quantum computers. While the proposed method is promising, a test
against a more complicated dataset, GTSRB (43 class real-life traffic sign
images, [11]), showed degradation in classification accuracy by the
application of QPF. This prompts further research into the understanding and
design of suitable quantum circuits for image classification neural networks.
To support the validation of our claims and further research, we have made our
source code available at `https://github.com/hajimesuzuki999/qpf`.

This paper is organized as follows: The new QPF unit combined with the
classical image classification neural network is proposed in Section 2.
Section 3 describes the experiment conducted using software simulation. The
results and discussions are presented in Section 4, followed by conclusions in
Section 5.

##  2 Methodology

Figure 1 shows the architecture of the proposed model. The method assumes that
the input image is a two-dimensional matrix with size $m$-by-$m$ and the pixel
value, $x$, follows $0\leq x\leq 1$. An extension to multi-channel pixel image
is considered as straightforward. Similar to QNN models, a section of size
$n$-by-$n$ is extracted from the input image. While $1<n\leq m$ in the case of
QNN, the proposed QPF uses $n=2$. This $2\times 2$ section of the input image
is referred as QPF window. An extension of QPF using $n>2$ is left for further
studies.

Figure 1: The architecture of the proposed quantum pre-processing filter (QPF)
model.

Given $n=2$, we use 4 qubit quantum circuit as shown in Figure 1. The four
qubits are initialized in the ground state. The four pixel values are then
encoded using Y rotation with $\theta=\pi x$ according to (1).

$R_{y}(\theta)=\left[\begin{array}[]{cc}\cos\frac{\theta}{2}&-i\sin\frac{\theta}{2}\\\
\sin\frac{\theta}{2}&\cos\frac{\theta}{2}\par\end{array}\right]$ (1)

The outputs from the Y rotation gates are fed to the quantum circuit referred
as $U$ in Figure 1. Measurements, referred as $M$ in Figure 1, are performed
on the output of the quantum circuit $U$. Three different quantum circuits are
examined in this paper. The first circuit, referred as Encoding only and shown
in Figure 2, performs measurement straight after the encoding. The second
circuit, referred as One CNOT and shown in Figure 3, performs controlled NOT
operation with the first qubit as the control and the 4th qubit as the target,
as shown in Figure 3. The third circuit, referred as Two CNOTs, performs two
controlled NOT operations as shown in Figure 4.

Figure 2: Quantum pre-processing filter, with encoding only. Figure 3: Quantum
pre-processing filter, with one CNOT. Figure 4: Quantum pre-processing filter,
with two CNOTs.

The outputs from the measurement operations are given as expectation values
between $-1$ and $1$, and form output features. We note that the total number
of parameters in the input image ( $m\times m$ ) is the same as the total
number of parameters in the output features ( $4\times(m/2)\times(m/2)$ ). The
output features are made into a one dimensional vector by the flatten layer.
The number of the nodes of the output of the flatten layer is $m\times m$. The
nodes are fully connected by the first fully connected layer 1. The output of
the fully connected layer 2 has the number of nodes equal to the number of
classes.

##  3 Experiment

As was performed by many, we first apply the proposed method to the MNIST
dataset [8] to obtain benchmark results. The MNIST dataset consists of 60,000
training and 10,000 test images of handwritten digits of 0 to 9. The size of
each image is 28 by 28 pixels. The original images are in greyscale within the
values between 0 and 255, which are scaled by dividing them by 255. We then
chose the EMNIST dataset [3] to extend the number of image class. The EMNIST
dataset (Balanced and By_Merge [3]) contains 112,800 training and 18,800 test
images of handwritten digits and letters making up 47 classes. Note that some
of upper- and lower-case letters are merged due to their similarity (e.g. C is
similar to c) in this dataset. Original EMNIST dataset is divided by 255 to
create a dataset with pixel values between 0 and 1.

The GTSRB dataset [11] consists of 34,799 training and 12,630 test images of
43 class traffic signs captured from actual traffic signs in-use in various
conditions. The original dataset has various image sizes between $15\times 15$
and $222\times 193$ pixels. Those images were scaled to a size of $32\times
32$ pixels. The original images were in RGB color, which were converted into
grayscale between 0 and 255. Unlike MNIST and EMNIST dataset, in order to
normalize the dynamic range of each image, the normalization is applied to
each image according to the following formula:

$\tilde{c}_{x,y}=\frac{c_{x,y}-\min(I)}{\max(I)-\min(I)}$ (2)

where $c_{x,y}$ and $\tilde{c}_{x,y}$ represent the original and normalized
pixel values in the position ($x$,$y$), and $\max(I)$ and $\min(I)$ denote the
maximum and the minimum element in the two dimensional image matrix $I$,
respectively.

The experiment was performed mainly in MATLAB. PennyLane was used to build
quantum circuits which were exported to MATLAB as unitary matrices. The Adam
optimizer [7] was used as the solver for the training network and default
mini-batch size of 128 was used for all of the three datasets to fully test
the proposed QPF model.

Figure 5 shows example input images from MNIST, EMNIST, and GTSRB, and
corresponding output features using encoding only (labelled as $q[0]$ to
$q[3]$), and two CNOTs (CNOT( $q[0],q[3]$ ) and CNOT( $q[1],q[2]$ )).

Figure 5: An example of the input image and the relevant output features.

##  4 Results and Discussion

Figure 6 shows the variation of the testing dataset accuracy as a function of
training iterations using MNIST dataset. As can be seen in Figure 6, the test
set accuracy using QPF Encoding only converges faster than that of NN. The
phenomenon of faster convergence was also observed in Henderson’s QNN [6].
However, the converged test set accuracy of the QPF Encoding only model does
not improve that of NN. The application of QPF One CNOT improves the test set
accuracy from 92.5% to 93.7%. The test set accuracy is further improved to
95.4% by the application of QPF Two CNOTs.

Figure 6: Test set accuracy using MNIST dataset.

Similarly, small but clearly faster convergence is observed by the application
of QPF Encoding only in the case of EMNIST dataset as shown in Figure 7.
However, in the case of EMNIST, the converged test set accuracy of the QPF
Encoding only is reduced from that of NN. Nonetheless, the test set accuracy
is improved from 68.9% to 72.0% by the application of QPF One CNOT, and to
75.9% by the application of QPF Two CNOTs.

Figure 7: Test set accuracy using EMNIST dataset.

Figure 8 shows the variation of test set accuracy using the GTSRB dataset.
Unlike the cases using MNIST and EMNIST datasets, the converged test set
accuracy by the application of QPF is reduced from that of NN in the case of
GTSRB dataset. The exact causes of this phenomenon are currently unknown to
the authors and remain for further research. The summary of the testing
accuracy results is shown in Table 1.

Figure 8: Test set accuracy using GTSRB dataset. | MNIST | EMNIST | GTSRB
---|---|---|---
NN | 92.5% | 68.9% | 81.4%
Encoding only | 92.4% | 68.1% | 77.6%
One CNOT | 93.7% | 72.0% | 77.6%
Two CNOT | 95.4% | 75.9% | 77.0%
Table 1: Summary of testing accuracy results.

Referring back to Figure 3, there are 12 different ways to create a CNOT
circuit from four qubits. In order to investigate if a different CNOT
arrangement would make the difference in classification accuracy, the training
of the network and the classification of the images were performed for MNIST
dataset using the 12 different CNOT arrangements. Figure 9 shows the results.
The $x$ axis of Figure 9 denotes the arrangement of the CNOT where the lower
number refers to the control qubit and the upper number refers to the target
qubit. For example, “0 3” refers to the arrangement as shown in Figure 3. As
can be seen from Figure 9, the variation of test set accuracy for different
CNOT arrangement is considered to be small, within less than 0.6%.

Figure 9: Test set accuracy on MNIST dataset using different CNOT
arrangements.

Similarly, there are 24 different ways to arrange the two CNOTs using four
qubits. The test set accuracy was derived for 24 different two CNOTs
arrangement using MNIST dataset, and the results are shown in Figure 10. In
Figure 10, the $x$ axis label refers to the arrangement of the two CNOTs in
the order of the control and target qubits of the first CNOT, then the control
and target qubits of the second CNOT. For example, “0 3 1 2” refers to the
arrangement as shown in Figure 4. As evident in Figure 10, the CNOT
arrangements pairing the 1st and the 4th qubits and pairing the 2nd and the
3rd qubits seems to achieve a higher testing accuracy irrespective to which
qubit is assigned as the target or the control. Referring back to Figure 1,
the pairing of the 1st and 4th qubits and the pairing of the 2nd and 3rd
qubits correspond to the pairing the diagonal elements of the 2 x 2 QPF
window. The exact reason for the improved classification accuracy in the case
of MNIST and EMNIST dataset when correlating the diagonal elements of the QPF
window is currently not known to the authors and is left for further research.

Figure 10: Test set accuracy on MNIST dataset using different two CNOTs
arrangements.

##  5 Conclusion

A novel QPF that can improve the image classification accuracy of NN for MNIST
and EMNIST datasets is proposed. While clear improvements are observed, the
exact causes of the improvements are currently unknown and remain for future
research and investigation. Also, further investigation is needed in order to
improve the classification accuracy of NNs against complex images such as
those from the GTSRB dataset. Proposed future research directions include
increasing the number of qubit sizes from four as well as applying the QPFs to
other image classification methods such as convolutional NNs.

##  6 Acknowledgment

This research has been supported by Australian government Research Training
Program and Commonwealth Scientific Industrial and Research Organization.

## References

  * [1] M. Benedetti, E. Lloyd, S. Sack, and M. Fiorentini.  Parameterized quantum circuits as machine learning models.  Quantum Science and Technology, 4(4), November 2019.
  * [2] V. Bergholm, J. Izaac, M. Schuld, C. Gogolin, M. S. Alam, S. Ahmed, J. M. Arrazola, C. Blank, A. Delgado, S. Jahangiri, K. McKiernan, J. J. Meyer, Z. Niu, A. Szava, and N. Killoran.  Pennylane: Automatic differentiation of hybrid quantum-classical computations.  arXiv:1811.04968v4, July 2022.
  * [3] G. Cohen, S. Afshar, J. Tapson, and A. van Schaik.  EMNIST: an extension of MNIST to handwritten letters.  In International Joint Conference on Neural Networks, pages 2921–2926, May 2017.
  * [4] D. P. Garcia, J. Cruz-Benito, and F. J. Garcia-Penalvo.  Systematic literature review: Quantum machine learning and its applications.  arXiv:2201.04093, January 2022.
  * [5] M. Henderson, J. Gallina, and M. Brett.  Methods for accelerating geospatial data processing using quantum computers.  Quantum Machine Intelligence, 4(4), January 2021.
  * [6] M. Henderson, S. Shakya, S. Pradhan, and T. Cook.  Quanvolutional neural networks: powering image recognition with quantum circuits.  Quantum Science and Technology, 2(2), February 2020.
  * [7] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv:1412.6980v9, January 2017.
  * [8] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Quanvolutional neural networks: powering image recognition with quantum circuits.  Proceedings of the IEEE, 86(11):2278–2324, November 1998.
  * [9] A. Mari.  Quanvolutional neural network.  Available online: https://pennylane.ai/qml/demos/tutorial_quanvolution.html, January 2021.
  * [10] W. O’Quinn and S. Mao.  Quantum machine learning: Recent advances and outlook.  IEEE Wireless Communications, 27(3):126–131, June 2020.
  * [11] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel.  The German traffic sign recognition benchmark: A multi-class classification competition.  In International Joint Conference on Neural Networks, pages 1453–1460, July 2011.