# Unlearning Backdoor Attacks through Gradient-Based Model Pruning

Kealan Dunnett1,2,†, Reza Arablouei2, Dimity Miller1, Volkan Dedeoglu1,2, Raja
Jurdak1 †Corresponding Author<EMAIL_ADDRESS>0000-0002-0010-1499 0000-0002-6932-2900 0000-0001-6312-8325
0000-0002-2567-2423 0000-0001-7517-0782 1Queensland University of Technology,
2CSIRO’s Data61

###### Abstract

In the era of increasing concerns over cybersecurity threats, defending
against backdoor attacks is paramount in ensuring the integrity and
reliability of machine learning models. However, many existing approaches
require substantial amounts of data for effective mitigation, posing
significant challenges in practical deployment. To address this, we propose a
novel approach to counter backdoor attacks by treating their mitigation as an
unlearning task. We tackle this challenge through a targeted model pruning
strategy, leveraging unlearning loss gradients to identify and eliminate
backdoor elements within the model. Built on solid theoretical insights, our
approach offers simplicity and effectiveness, rendering it well-suited for
scenarios with limited data availability. Our methodology includes formulating
a suitable unlearning loss and devising a model-pruning technique tailored for
convolutional neural networks. Comprehensive evaluations demonstrate the
efficacy of our proposed approach compared to state-of-the-art approaches,
particularly in realistic data settings.

###### Index Terms:

backdoor attack, backdoor mitigation, model pruning, unlearning

##  I Introduction

00footnotetext: †Corresponding author: Kealan Dunnett
<EMAIL_ADDRESS>
Recently, deep learning has witnessed remarkable advancements, leading to its
widespread adoption across various industries and academic fields. For
example, sectors such as healthcare, education, automobiles, and logistics
have rapidly embraced deep learning technologies [15]. However, alongside its
positive impact, deep learning also faces significant adversarial threats that
pose challenges to its practical implementation [21].

Backdoor attacks represent a critical adversarial threat in classification-
based machine learning tasks. Initially demonstrated by [5], these attacks
seek to compromise the integrity of model decisions by altering predictions
when presented with inputs containing a specific trigger. The primary
objective of backdoor attacks is to embed this behavior into the model without
undermining its ability to classify clean inputs (i.e., inputs lacking the
trigger). Consequently, a backdoored model distinguishes between inputs with
the trigger and those without it during classification. Given the complexity
of modern deep neural network models, there is typically no overt indication
that such embedded backdoor behavior exists.

In real-world scenarios, backdoor attacks pose a significant threat in
applications where classification model outputs inform automated decision-
making processes. A notable example is traffic sign classification,
particularly with the increasing adoption of driver assistance systems.
Misclassification of critical traffic signs with these systems can have
catastrophic consequences [15]. The risk of backdoor attacks is especially
pertinent when model training is outsourced to a third party (e.g., through
cloud services or federated learning) or when transfer learning is employed
[14], as this provides opportunities for adversaries to manipulate the model’s
training data and or procedures to inject backdoor tasks.

In response to the threats posed by backdoor attacks, various methods have
been proposed for detecting the presence of backdoors within models as well as
extracting the respective backdoor triggers using clean inputs. For instance,
[14] introduces an optimisation-based method to discover a trigger pattern
that can transition a set of images to a target class with minimal input
perturbation. As well as backdoor discovery methods, numerous backdoor
mitigation approaches have also been proposed in the literature [20, 17, 23].
These approaches aim to remove backdoor behavior from a model using clean or
backdoor inputs with minimal impact on the original classification objective.
To achieve this, several existing works leverage model pruning, a technique
traditionally used to sparsify overparameterized models, which has proven
effective for backdoor mitigation.

Existing backdoor attack mitigation approaches based on model pruning mainly
rely on analyzing neuron activation values [12, 24, 8], sensitivity [20], or
reconstruction [10]. While these approaches have shown effectiveness within
certain settings, their performance within data-limited settings, common in
real-world applications, remains unclear. Moreover, many of these approaches
require the defender to carefully select the values of one or more
hyperparameters, as in [24, 24, 20]. While we acknowledge that achieving a
hyperparameter-free pruning approach may be infeasible, we advocate for the
development of approaches that only mandate the selection of few intuitive
hyperparameters, minimizing the need for extensive hyperparameter tuning. A
notable example is [12], where the defender is only required to specify an
acceptable accuracy reduction threshold.

Contributions: We propose a new approach for backdoor attack mitigation by
employing gradient-based model pruning to unlearn backdoor behavior. By
conceptualizing backdoor mitigation as an unlearning problem, our approach
harnesses information from unlearning loss gradients to effectively eliminate
backdoors through model pruning. This sets our approach apart from existing
model-pruning-based approaches to backdoor mitigation, which typically rely on
information available through neuron activation values, sensitivity, or
reconstruction. Moreover, we incorporate an effective fine-tuning procedure
that utilizes both clean and backdoor data to address any performance
degradation resulting from model pruning. A key advantage of the proposed
approach is its user-friendly nature, as defenders are only required to adjust
a few design parameters with minimal or no tuning. The selection of these
parameters is tailored to defenders’ specific needs and does not demand
intricate tuning, unlike the hyperparameters utilized in many existing
approaches. We conduct comprehensive evaluations of our approach against
state-of-the-art approaches, demonstrating its effectiveness across various
limited data scenarios. Our evaluations also offer fresh insights into the
performance of existing state-of-the-art methods in constrained data
scenarios.

##  II Related Work

In this section, we provide a summary of prominent backdoor attack and
mitigation works. For each topic, the range of strategies used by these works
is highlighted.

###  II-A Backdoor Attacks

The BadNets attack, introduced in [5], stands as a seminal example of a
backdoor attack. This work demonstrated how manipulating a subset of training
data by inserting a colored square can effectively embed a backdoor task into
a model. Building upon this foundation, subsequent studies have proposed
diverse attack methods of varying complexity. For instance, the blended
attack, outlined in [3], modifies training instances by incorporating a
trigger image (e.g., a picture of Hello Kitty) with a blending ratio that
controls its transparency. Furthermore, [1] proposes a backdoor attack that
introduces an extra feature to images in the training set associated with the
target class (e.g., a sinusoidal signal). Following training, [1] demonstrates
that incorporating this feature into images from other classes causes them to
be misclassified as the target class.

More recently, several optimisation-based backdoor attacks have been proposed.
For example, [11] presents a method employing an encoder-decoder network to
generate sample-specific triggers for training a backdoored model. Moreover,
[4] presents a backdoor attack framework that jointly optimizes the trigger
pattern and learns a backdoored model. This approach yields a targeted trigger
representation that fulfills the backdoor objective while remaining stealthy.
Beyond trigger-based methods, alternative relabeling approaches have emerged.
For example, [23] proposes an all-to-all backdoor attack, wherein any input
containing the backdoor trigger is mapped to a class one step ahead in the
cyclical order, i.e., $y+1\mod n$ where $y$ is the true class label and $n$
the number of classes. However, our work aligns with the prevailing trend in
the literature by focusing solely on the targeted attack setting, where the
trigger steers the classification towards a static target class $t$.

###  II-B Backdoor Mitigation

Model pruning and fine-tuning using clean training data, as proposed by [12],
represent the initial steps towards removing backdoors from trained models.
Subsequent works, such as [12, 17, 20, 24], have further developed model
pruning. These works develop a range of approaches based on the observation
that clean and backdoor inputs produce different activation values throughout
the model. Subsequently, by inferring these differences using a variety of
techniques, these proposals successfully identify and mitigate backdoor
elements, which are components of the model that contribute the most to the
backdoor task. Notably, [24] proposes a data-free pruning approach,
eliminating the need for clean or backdoor data during model pruning.

In addition, several fine-tuning and model regularisation approaches have also
been presented in the literature. For example, [8] introduces an attention-
based mechanism for backdoor removal. Utilizing a fine-tuned teacher network,
[8] employs a layer-based knowledge distillation technique to eliminate the
backdoor from the model. Moreover, works such as [13, 2] propose unlearning-
like approaches that remove backdoors through fine-tuning alone. However,
these approaches primarily focus on model fine-tuning rather than model
pruning.

##  III Preliminaries

###  III-A Neural Networks

A neural network is a parameterized function $f(x,\theta)\rightarrow y$ that
maps an input $x\in\mathbb{R}^{n}$ to output $y\in\mathbb{R}^{m}$ given a set
of parameters $\theta$. In an $m$-class classification scenario, the entries
of $y$ represent the likelihood of $x$ belonging to each class. Using a
training dataset ($D_{t}:\\{x_{i},y_{i}\\}_{i=1}^{Z}$), we can find the
optimal set of parameters $\theta^{*}$ by minimizing the aggregate loss
function, $\mathscr{L}(\hat{y},y)$, which quantifies the difference between
each predicted value $\hat{y}_{i}=f(x_{i},\theta)$ and its corresponding
actual value $y_{i}$. Therefore, we have

$\theta^{*}=\arg\min_{\theta}\sum_{i=1}^{Z}\mathscr{L}(f(x_{i},\theta),y_{i}).$
(1)

This optimization is typically performed using the stochastic gradient descent
algorithm or one of its variants.

###  III-B Backdoor Threat Model

In a targeted attack scenario, the adversary embeds a backdoor task into the
model, causing inputs containing a trigger pattern $p$ to be classified as the
target class $t$. This manipulation involves training the model using two
datasets: the clean dataset $D_{c}=\\{x_{i},y_{i}\\}_{i=1}^{Z_{c}}$,
comprising original inputs and their labels, and the backdoor dataset
$D_{b}=\\{\breve{x}_{i},t\\}_{i=1}^{Z_{b}}$ generated by the adversary using
the trigger pattern $p$. Note that, in certain attack scenarios, such as [4],
$p$ may vary with the input, rendering it dynamic. Using $D_{c}$ and $D_{b}$,
the adversary aims to determine a set of model parameters $\theta^{\prime}$
capable of effectively classifying inputs from both sets. In this optimization
process, the adversary employs a poisoning ratio to specify the proportion of
backdoor to clean inputs.

###  III-C Main Assumptions

We adopt common assumptions regarding the capabilities of adversaries and
defenders. Concerning the attacker, we consider a range of attacks documented
in the literature, without imposing any additional constraint beyond those
stated in the corresponding original papers. For instance, in the most extreme
case, as depicted in the BPP attack by [18], it is assumed that the attacker
possesses complete control over the training process. However, to streamline
our evaluations, we focus solely on the targeted attack scenarios, thereby
reducing the complexity of the analysis.

We assume the defender has access to a limited set of correctly labeled clean
images (i.e., images without the backdoor pattern). In practice, the number of
clean samples accessible to the defender can be severely constrained. In
addition, we assume that the defender can synthesize backdoor inputs using any
relevant state-of-the-art synthesis approach, such as those proposed in [13,
14, 16]. Thus, the defender has access to a backdoor variant for each clean
image, incorporating the adversary’s trigger. While acknowledging that this
assumption limits our approach’s applicability to scenarios where faithful
synthesis of backdoor inputs is not feasible, we aim to address this
limitation in future work.

##  IV Proposed Approach

In this section, we begin with describing our conceptualization of backdoor
unlearning. We then describe the proposed approach, which consists of two key
steps. The first step involves iteratively pruning the backdoored model using
the gradient of unlearning loss. In the second step, we fine-tune the pruned
model utilizing available clean and backdoor data to alleviate any adverse
affect of model pruning.

###  IV-A Backdoor Unlearning

For a backdoored model to effectively fulfil the adversarial objective
outlined in section III-B, it must accurately classify inputs associated with
both the main and backdoor tasks. The main task is to correctly classify the
clean inputs from $D_{c}$ and the backdoor task is to correctly classify the
backdoor inputs from $D_{b}$. Therefore, we conceptualize the challenge of
backdoor mitigation as an unlearning problem, where the goal is to unlearn the
backdoor task while preserving the efficacy of the main task. As such,
backdoor unlearning aims to achieve three primary objectives: (i) Eliminate
the model’s association of the given trigger pattern $p$ with the target class
$t$. (ii) Rectify the model’s interpretation of $p$ to ensure that inputs from
$D_{b}$ are classified as their original labels. (iii) Ensure that the model’s
correct classification of inputs from $D_{c}$ is maintained.

While the concept of unlearning has been employed in prior works such as [13,
17, 2] to mitigate backdoor attacks, our unlearning approach differs
significantly in that backdoor unlearning is achieved through gradient-
informed model pruning. In existing works, unlearning typically involves model
fine-tuning using a modified loss function. For example, in [13], unlearning
is achieved through model retraining using a loss function comprising three
components, one of which is the negative cross-entropy loss of the backdoor
inputs given the backdoor label and thus induces unlearning through gradient
ascent. Unlike previous works, we introduce a backdoor unlearning approach
that is implemented through gradient-based model pruning. Our backdoor
unlearning loss function is the aggregate cross-entropy loss for the backdoor
inputs with their corresponding correct labels. We express this loss as

$\mathcal{L}=\sum_{i=1}^{Z_{b}}\mathscr{L}_{\text{CE}}(f^{\prime}(\breve{x}_{i},\theta^{\prime}),y_{i})$
(2)

Where $f^{\prime}$ and $\theta^{\prime}$ denote the backdoored model and its
parameters, respectively. Hence, we compute the loss gradient with respect to
$\theta^{\prime}$ as
$\nabla_{\theta^{\prime}}=\frac{\partial\mathcal{L}}{\partial\theta^{\prime}}$.

Our approach does not explicitly aim to minimize the above unlearning loss.
Instead, we utilize its gradient information to gauge the relative
contribution of each parameter subset to the backdoor behavior. Larger
gradients in the unlearning loss suggest a stronger influence on
misclassifying $\breve{x}_{i}$ as $t$ instead of its true label $y_{i}$.
However, unlike traditional model (un)learning, which adjusts these
parameters, we opt to prune them entirely. This decision is rooted in the
understanding that such parameters are typically manipulated by the backdoor
attack in a way that adjusting them through stochastic gradient descent based
on limited data is ineffective in mitigating the backdoor attack. Pruning
these parameters removes their influence on the model’s behavior, thereby
effectively mitigating the backdoor effect. We elaborate the details of this
pruning strategy in the next section.

###  IV-B Gradient-based Pruning

(a) PreactResNet-18

(b) VGG-19+BN

Figure 1: The scatter plots of ACC and RA versus ASR for all considered
approaches on CIFAR-10 across different attacks, SPC settings, and models.

Building on the described notion of backdoor unlearning, we propose a
gradient-based pruning technique tailored for convolutional layers. For each
2D convolutional filter $i$ at layer $l$ with parameters
$\theta^{\prime}_{l,i}$, we compute the mean absolute gradient as

$\xi_{l,i}=\frac{\|\nabla_{\theta^{\prime}_{l,i}}\|_{1}}{\ell(\nabla_{\theta^{\prime}_{l,i}})}$
(3)

where $\|\cdot\|_{1}$ and $\ell(\cdot)$ denote the L1 norm and the number of
entries, respectively. Given the nature of the unlearning loss presented in
section IV-A, $\xi_{l,i}$ represents the relative contribution of filter $i$
at layer $l$ to misclassification of backdoor inputs. After calculating the
$\xi_{l,i}$ values for all filters across all layers, we identify the filter
with the highest $\xi_{l,i}$ value for pruning. To prune a filter, we set its
weights and bias to zero. Following each pruning round, we evaluate the
unlearning loss and main task accuracy for the pruned model using a separate
validation dataset that is not used for calculating the filter $\xi_{l,i}$
values. We iterate the pruning process until the main task accuracy falls
below a predefined threshold $\alpha$ or the unlearning loss fails to improve
for $P_{p}$ consecutive rounds.

###  IV-C Fine-tuning

As demonstrated in [12], model pruning aimed at countering backdoor attacks
can often lead to a degradation in the model’s accuracy when classifying clean
inputs. To recover the lost performance resulting from model pruning, we adopt
a modified version of the fine-tuning approach proposed in [17]. Additionally,
since our model pruning technique targets convolutional layers, fine-tuning
facilitates the removal of backdoor elements present in other unpruned layers.
This is particularly beneficial when backdoor elements are present in the
dense layers.

To perform fine-tuning, we propose to utilize all available clean and backdoor
data that, as discussed in section III-C, typically constitute a small subset
of the original training dataset. This is in contrast with previous approaches
such as [17] that use a portion of the backdoor data during fine-tuning.
Additionally, each backdoor datum is labeled with its correct (non-backdoor)
label. The fine-tuning process, which is essentially model re-training,
continues until the loss fails to improve for a specified number of
epochs/iterations $P_{t}$. This ensures that the performance of the main task
is maintained throughout fine-tuning. Similar to our pruning approach
discussed in section IV-B, we evaluate the loss using a separate validation
set that is not used for fine-tuning.

##  V Evaluation

(a) PreactResNet-18

(b) VGG-19+BN

(c) EfficientNet B3

(d) MobileNet V3 Large

Figure 2: The scatter plots of ACC and RA versus ASR for FT-SAM, ANP, and the
proposed approach on GTSRB across different attacks, SPC settings, and models.

We evaluate the performance of our proposed approach against five existing
backdoor mitigation approaches across four types of attacks and various
settings. To this end, we utilize the BackdoorBench tool presented in [19].
The code to reproduce the results presented in this paper is available
online111https://github.com/WhoDunnett/Grad-Prune/tree/main.

###  V-A Attack Configuration

In our evaluations, we consider four backdoor attacks of BadNets [5], Blended
[3], Low Frequency (LF) [22] and Bit-Per-Pixel (BPP) [18]. According to [19],
these attacks encompass a range of trigger characteristics utilized in state-
of-the-art attacks. For each attack, we use the default configuration provided
in [19]. Notably, we only considered a 10% poisoning setting and an all-to-one
targeted attack, with the target class label being 0. We implement the attacks
on the CIFAR-10 [7] and German Traffic Sign Recognition Benchmark (GTSRB) [6]
datasets, employing multiple model architectures. For CIFAR-10, we consider
the PreactResNet-18 model and the VGG-19 model incorporating batch
normalization (referred to as VGG-19+BN). For GTSRB, we examine the same
models along with the EfficientNet B3 and MobileNet V3 Large models. We
utilize all models with their default configurations provided in [19]. For
each configuration, the baseline results correspond to when no backdoor
mitigation approach is applied.

###  V-B Defense Configuration

We benchmark our proposed backdoor attack mitigation approach against several
commonly used approaches, including Fine-Tuning [12], Fine-Pruning [12] and
NAD [9]. In addition, we include the state-of-the-art defences CLP [24], FT-
SAM [25] and ANP [20]. We test FT-SAM and ANP only on GTSRB, based on their
performance on CIFAR-10. For each defence, we use the implementation provided
in BackdoorBechmark [19], using default configurations. We do not perform
hyperparameter optimization and rely on default settings reported in the
relevant papers, since conducting such optimization is typically infeasible
for defenders [19].

Unlike existing works, we evaluate the performance of each considered
defensive approach across a range of limited data settings. Existing
approaches usually use a percentage of training data, often 1% or 10%, to
evaluate the efficacy of their proposals. However, using a proportion of
training data can make comparisons across datasets challenging due to
variations in dataset sizes. Moreover, accessing even 1% of training data,
especially in large datasets, may be unrealistic. To better reflect the data
available to defenders in practical scenarios, we conduct our evaluations
considering 2, 10, or 100 samples per class (SPC). For each dataset, model
architecture, and attack type, we test every backdoor mitigation approach five
times, with each trial using a different subset of data. When validation data
is required, such as for Fine-Pruning, ANP, and our approach, we use 10% of
the data for validation. For the 2 SPC cases, we use one sample for training
(fine-tuning) and the other for validation.

###  V-C Performance Measures

We evaluate the performance of each approach across all settings using three
accuracy measures: accuracy on clean data (ACC), attack success rate (ASR),
and recovery accuracy (RA) [19]. ACC represents the classification accuracy on
the clean test dataset devoid of any backdoor attack. Moreover, ASR denotes
the accuracy on the test dataset containing the backdoor trigger and the
backdoor target labels, while RA indicates the accuracy on the test dataset
containing the backdoor trigger and the correct non-backdoor labels. Note that
$\text{ASR}+\text{RA}\leq 1$. A successful defence is characterized by high
values of ACC and RA and a low value of ASR.

###  V-D Results

In Fig. 1, we depict the performance of various approaches using scatter plots
of ACC and RA versus ASR for the CIFAR-10 dataset, employing the
PreactResNet-18 and VGG-19+BN models across different attacks and data
settings. We provide the corresponding accuracy measure values used to
generate these figures in Tables I and II as supplementary material. Except
for the BPP attack, our approach exhibits competitive performance compared
with the state-of-the-art approaches FT-SAM and ANP. In most cases, our
approach significantly reduces ASR compared to the baseline with minimal
impact on ACC. This decrease in ASR often coincides with a proportional rise
in RA. While our approach may show slightly less effectiveness in low-data
settings (i.e., low SPC) compared to FT-SAM and ANP, overall, it demonstrates
promising results across various scenarios. The results in Fig. 1 also
indicate that the effectiveness of FT, FP, and NAD varies noticeably across
the considered scenarios. More specifically, the effectiveness of these
approaches is significantly impacted by the amount of available data.

In addition, Fig. 1 shows that FT, FP, and NAD exhibit significant variability
in their performance compared to FT-SAM, ANP, and our approach. While CLP
demonstrates good performance in specific scenarios, it proves ineffective in
several cases, indicating that its underpinning assumptions may not hold
universally across all model architectures, especially those not examined in
the original paper [24]. Overall, FT-SAM appears to mitigate the backdoor
attacks most successfully among the tested approaches. However, its RA
exhibits greater variance compared to ANP and our approach. In addition, RA of
FT-SAM is limited in low-data settings, suggesting that while it may mitigate
the backdoor attack, it does not consistently restore the correct
classification of backdoor images.

When compared to the activation-based pruning approaches FP and CLP, our
approach is generally significantly more effective. While FP and CLP may
exhibit state-of-the-art performance in certain cases, their overall
effectiveness varies greatly. Therefore, we posit that the backdoor unlearning
loss gradient can capture the essential information required for backdoor
removal more adeptly compared to activation values.

In Fig. 2, we illustrate the results for the GTSRB dataset, utilizing the
PreactResNet-18, VGG-19+BN, EfficientNet B3, and MobileNet V3 Large models.
Here, we only consider FT-SAM, ANP, and our approach as they perform
considerably better over CIFAR-10. Results for PreactResNet-18 and VGG-19+BN
closely align with those reported for CIFAR-10, showing consistent and
effective backdoor attack mitigation across various settings. Notably, our
approach exhibits less variability compared to FT-SAM and ANP with both
models. However, for EfficientNet B3 and MobileNet V3 Large, greater variance
is observed overall. Specifically, all three approaches face challenges in
mitigating the backdoor attacks when employing these models. With MobileNet V3
Large, our approach demonstrates superior robustness compared to FT-SAM and
ANP in the $\text{SPC}=100$ setting. While our evaluation results do not
conclusively establish gradient-informed model pruning as yielding superior
performance compared to FT-SAM and ANP, its competitive performance
underscores the utility of unlearning loss gradients. Moreover, our findings
corroborate the idea of reframing backdoor attack mitigation as an unlearning
problem addressed through model pruning.

##  VI Conclusion

We investigated the efficacy of gradient-informed model pruning for mitigating
backdoor attacks. By casting backdoor attack mitigation as an unlearning task
addressed via model pruning, we leveraged gradients of an aptly-devised
unlearning loss to remove backdoor elements. Our proposed approach offers an
intuitive yet theoretically well-grounded solution to backdoor mitigation.
Moreover, it demonstrates robust performance across various scenarios,
surpassing or rivalling state-of-the-art approaches. Encouraged by our
findings, we advocate for further exploration of gradient-based techniques in
future research. The prospect of eliminating the need for synthesizing
backdoor data is particularly promising. Furthermore, our benchmarking
highlights the importance of addressing challenges in low-data settings and
considering a diverse range of model architectures in future defense
strategies.

##  VII Acknowledgement

Computational resources and services used in this work were provided by the
eResearch Office, Queensland University of Technology, Brisbane, Australia.

## References

  * [1] Barni, M., Kallas, K., Tondi, B.: A new backdoor attack in cnns by training set corruption without label poisoning. In: 2019 IEEE International Conference on Image Processing (ICIP). pp. 101–105. IEEE (2019)
  * [2] Chen, W., Wu, B., Wang, H.: Effective backdoor defense by exploiting sensitivity of poisoned samples. Advances in Neural Information Processing Systems 35, 9727–9737 (2022)
  * [3] Chen, X., Liu, C., Li, B., Lu, K., Song, D.: Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 (2017)
  * [4] Doan, K., Lao, Y., Zhao, W., Li, P.: Lira: Learnable, imperceptible and robust backdoor attacks. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11966–11976 (2021)
  * [5] Gu, T., Liu, K., Dolan-Gavitt, B., Garg, S.: Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7, 47230–47244 (2019)
  * [6] Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., Igel, C.: Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In: International Joint Conference on Neural Networks. No. 1288 (2013)
  * [7] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * [8] Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., Ma, X.: Neural attention distillation: Erasing backdoor triggers from deep neural networks. In: International Conference on Learning Representations (2020)
  * [9] Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., Ma, X.: Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930 (2021)
  * [10] Li, Y., Lyu, X., Ma, X., Koren, N., Lyu, L., Li, B., Jiang, Y.G.: Reconstructive neuron pruning for backdoor defense. In: International Conference on Machine Learning. pp. 19837–19854. PMLR (2023)
  * [11] Li, Y., Li, Y., Wu, B., Li, L., He, R., Lyu, S.: Invisible backdoor attack with sample-specific triggers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 16463–16472 (2021)
  * [12] Liu, K., Dolan-Gavitt, B., Garg, S.: Fine-pruning: Defending against backdooring attacks on deep neural networks. In: International symposium on research in attacks, intrusions, and defenses. pp. 273–294. Springer (2018)
  * [13] Liu, Y., Fan, M., Chen, C., Liu, X., Ma, Z., Wang, L., Ma, J.: Backdoor defense with machine unlearning. In: IEEE INFOCOM 2022-IEEE Conference on Computer Communications. pp. 280–289. IEEE (2022)
  * [14] Liu, Y., Lee, W.C., Tao, G., Ma, S., Aafer, Y., Zhang, X.: Abs: Scanning neural networks for back-doors by artificial brain stimulation. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. pp. 1265–1282 (2019)
  * [15] Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M.P., Shyu, M.L., Chen, S.C., Iyengar, S.S.: A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR) 51(5), 1–36 (2018)
  * [16] Sun, M., Kolter, Z.: Single image backdoor inversion via robust smoothed classifiers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8113–8122 (2023)
  * [17] Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., Zhao, B.Y.: Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In: 2019 IEEE Symposium on Security and Privacy (SP). pp. 707–723. IEEE (2019)
  * [18] Wang, Z., Zhai, J., Ma, S.: Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15074–15084 (2022)
  * [19] Wu, B., Chen, H., Zhang, M., Zhu, Z., Wei, S., Yuan, D., Shen, C.: Backdoorbench: A comprehensive benchmark of backdoor learning. Advances in Neural Information Processing Systems 35, 10546–10559 (2022)
  * [20] Wu, D., Wang, Y.: Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems 34, 16913–16925 (2021)
  * [21] Xue, M., Yuan, C., Wu, H., Zhang, Y., Liu, W.: Machine learning security: Threats, countermeasures, and evaluations. IEEE Access 8, 74720–74742 (2020)
  * [22] Zeng, Y., Park, W., Mao, Z.M., Jia, R.: Rethinking the backdoor attacks’ triggers: A frequency perspective. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 16473–16481 (2021)
  * [23] Zhao, P., Chen, P.Y., Das, P., Ramamurthy, K.N., Lin, X.: Bridging mode connectivity in loss landscapes and adversarial robustness. arXiv preprint arXiv:2005.00060 (2020)
  * [24] Zheng, R., Tang, R., Li, J., Liu, L.: Data-free backdoor removal based on channel lipschitzness. In: European Conference on Computer Vision. pp. 175–191. Springer (2022)
  * [25] Zhu, M., Wei, S., Shen, L., Fan, Y., Wu, B.: Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. arXiv preprint arXiv:2304.11823 (2023)

##  VIII Supplementary Material

TABLE I: The performance of the considered approaches on CIFAR-10 using
PreactResNet-18. The mean and standard deviation of the performance measures
over five independent runs are presented. In each setting, the best and
second-best results are highlighted in bold and underlined, respectively.

Attack | BadNet | Blended | BPP | LF
---|---|---|---|---
SPC | Defense | ACC | ASR | RA | ACC | ASR | RA | ACC | ASR | RA | ACC | ASR | RA
Baseline | 91.82 | 94.76 | 5 | 93.61 | 99.71 | 0.29 | 91.46 | 99.93 | 0.07 | 93.2 | 99 | 0.93
| FT | 57.53±24.27 | 22.86±30.4 | 41.31±20.27 |  90.11±4.0 | 99.75±0.51 | 0.19±0.44 | 80.35±2.83 | 2.93±1.96 | 75.96±3.34 | 87.36±8.88 | 85.6±30.46 | 9.68±21.32
| FP | 79.41±3.88 | 0.4±0.37 |  80.52±4.12 | 83.4±1.84 |  13.34±10.71 |  42.32±7.43 | 85.74±1.71 | 56.57±35.33 | 32.05±28.27 | 83.37±2.56 | 38.38±27.66 | 38.38±17.39
| NAD | 32.72±15.53 | 10.76±14.41 | 30.15±11.06 | 74.07±17.23 | 83.55±24.91 | 6.9±7.97 | 68.65±12.5 | 8.33±7.12 | 62.56±12.58 | 69.18±17.67 | 62.06±34.43 | 17.99±14.3
| CLP | 90.71 | 95.2 | 4.51 | 90.68 | 91.12 | 8.03 |  89.99 | 5.51 |  80.77 | 91.27 | 98.53 | 1.34
| FT-SAM | 39.38±5.33 | 6.91±4.56 | 39.19±7.22 | 40.03±3.68 | 3.27±2.48 | 31.04±3.72 | 69.43±7.55 | 1.81±1.45 | 64.65±6.03 | 39.87±4.1 | 10.6±7.57 | 31.01±6.29
| ANP | 84.38±1.05 |  2.38±2.68 | 83.58±2.49 | 86.82±1.4 | 47.13±23.59 | 34.66±11.46 | 84.48±2.12 |  2.27±1.09 | 76.71±3.06 | 86.75±1.85 |  33.79±30.94 |  54.95±25.32
2 | Ours |  88.82±2.06 | 13.19±6.81 | 78.97±5.71 | 85.6±2.39 | 31.51±23.92 | 46.31±14.7 | 90.32±0.68 | 47.68±31.09 | 46.77±27.62 |  89.8±2.2 | 37.49±23.55 | 57.02±21.85
| FT | 84.21±3.91 | 38.24±25.64 | 53.05±19.45 | 91.8±1.28 | 94.99±9.2 | 4.0±6.92 |  90.28±0.42 | 2.27±0.88 | 85.21±0.72 | 91.1±1.19 | 78.27±29.5 | 19.3±26.02
| FP | 88.0±0.99 |  0.68±0.21 | 87.88±0.87 | 88.64±0.73 | 15.0±7.01 | 50.88±4.55 | 89.77±0.87 | 65.56±32.78 | 26.5±26.56 | 88.04±0.48 | 33.86±20.47 | 46.59±12.76
| NAD | 81.3±5.6 | 19.04±18.32 | 66.91±11.83 |  90.8±1.8 | 98.83±1.93 | 1.07±1.77 | 88.97±0.83 | 3.6±1.5 | 83.48±1.46 |  91.23±0.84 | 84.72±18.6 | 13.86±17.03
| CLP | 90.71 | 95.2 | 4.51 | 90.68 | 91.12 | 8.03 | 89.99 | 5.51 | 80.77 | 91.27 | 98.53 | 1.34
| FT-SAM | 75.21±1.44 | 2.32±0.71 | 74.41±1.67 | 81.65±0.88 | 5.37±6.77 | 52.57±5.93 | 88.46±0.55 |  1.7±0.35 |  83.56±0.92 | 81.88±1.55 |  5.52±2.6 | 69.23±3.15
| ANP | 84.15±0.97 | 0.1±0.12 |  86.47±1.18 | 86.03±1.25 | 12.36±9.63 |  54.36±4.89 | 85.22±2.51 | 0.84±0.74 | 77.85±2.0 | 86.23±1.62 | 1.19±1.3 | 81.47±3.45
10 | Ours |  90.02±0.48 | 8.72±9.69 | 83.71±7.7 | 89.01±1.52 |  8.29±10.93 | 67.63±6.48 | 91.47±0.35 | 37.56±20.42 | 56.51±18.11 | 90.57±0.78 | 11.84±6.18 |  80.23±5.24
| FT | 89.59±0.4 | 1.81±0.66 | 88.38±0.68 | 92.86±0.17 | 96.32±3.19 | 3.43±2.91 |  92.53±0.1 | 2.41±0.37 |  86.97±0.37 | 92.58±0.17 | 75.14±22.45 | 22.25±19.19
| FP | 91.47±0.21 |  0.85±0.2 | 90.87±0.27 | 92.23±0.17 | 14.18±4.24 | 53.23±2.22 | 92.12±0.14 | 50.47±17.2 | 34.09±10.06 | 91.66±0.22 | 32.56±9.24 | 55.17±8.1
| NAD | 88.16±0.51 | 1.81±0.69 | 87.25±0.68 |  92.3±0.28 | 88.67±6.78 | 10.34±5.98 | 92.12±0.18 |  2.32±0.35 | 86.62±0.63 |  92.02±0.18 | 37.5±21.7 | 55.55±18.68
| CLP | 90.71 | 95.2 | 4.51 | 90.68 | 91.12 | 8.03 | 89.99 | 5.51 | 80.77 | 91.27 | 98.53 | 1.34
| FT-SAM | 90.29±0.33 | 1.24±0.33 |  89.64±0.45 | 91.81±0.18 | 12.84±4.67 | 57.78±3.03 | 92.64±0.14 | 2.55±0.48 | 87.08±0.55 | 91.57±0.23 | 4.58±2.84 | 82.23±2.55
| ANP | 84.99±1.24 | 0.02±0.02 | 88.31±1.5 | 86.62±1.61 |  3.59±1.33 |  60.05±3.62 | 85.64±0.99 | 0.21±0.06 | 79.1±1.78 | 86.61±1.7 | 0.25±0.14 |  85.69±1.77
100 | Ours |  90.91±0.56 | 7.3±2.37 | 86.07±1.57 | 88.85±0.75 | 3.33±8.64 | 72.03±5.68 | 91.74±0.22 | 43.55±30.41 | 51.19±27.04 | 91.84±0.22 |  1.85±0.43 | 89.98±0.38

TABLE II: The performance of the considered approaches on CIFAR-10 using
VGG-19+BN. The mean and standard deviation of the performance measures over
five independent runs are presented. In each setting, the best and second-best
results are highlighted in bold and underlined, respectively.

Attack | BadNet | Blended | BPP | LF
---|---|---|---|---
SPC | Defense | ACC | ASR | RA | ACC | ASR | RA | ACC | ASR | RA | ACC | ASR | RA
Baseline | 90.55 | 94.22 | 5.44 | 92.08 | 99.63 | 0.34 | 89.74 | 99.28 | 0.72 | 83.57 | 13.07 | 82.38
| FT | 44.44±22.79 | 23.03±33.76 | 28.25±10.01 | 66.6±29.89 | 67.64±40.58 | 8.68±11.18 | 80.02±3.63 | 4.23±3.38 | 74.09±4.92 | 33.71±15.75 | 5.79±1.98 | 33.72±15.75
| FP | 82.51±5.08 | 55.73±26.33 | 37.96±21.17 |  88.16±2.84 | 99.2±0.47 | 0.7±0.36 |  84.07±2.58 | 6.29±4.02 | 72.77±11.32 | 77.05±10.89 | 2.31±1.46 | 78.03±10.28
| NAD | 57.24±22.98 | 45.97±31.27 | 27.32±13.59 | 69.42±29.03 | 65.26±41.72 | 9.24±10.39 | 75.32±7.57 | 6.78±8.93 | 57.44±16.88 | 45.06±25.48 | 10.71±7.43 | 44.15±25.88
| CLP | 88.64 | 84.77 | 14.31 | 90.65 | 96.37 | 3.27 | 87.63 |  1.19 | 81.46 | 82.83 | 13.47 | 81.77
| FT-SAM | 56.05±12.23 | 6.56±5.06 | 53.49±10.72 | 68.95±12.01 | 4.82±4.18 | 44.54±7.84 | 79.18±5.08 | 2.68±1.47 | 73.57±6.01 | 55.59±4.18 | 3.9±3.51 | 58.28±4.63
| ANP | 83.87±1.43 |  15.16±22.39 | 72.58±18.17 | 84.17±0.87 |  36.88±29.72 | 37.74±15.48 | 82.48±1.01 | 0.72±1.52 |  78.73±2.62 | 77.74±1.84 |  2.38±1.29 | 79.62±1.35
2 | Ours |  86.59±1.41 | 33.01±19.25 |  60.59±15.56 | 83.53±1.87 | 45.46±13.64 |  42.24±9.73 | 88.38±1.33 | 28.11±24.12 | 63.22±21.38 |  81.31±2.27 | 14.12±2.29 |  80.19±2.28
| FT | 80.86±3.45 | 36.96±29.04 | 51.44±21.61 | 86.17±3.15 | 72.47±22.05 | 20.75±15.49 | 87.61±0.62 | 6.25±4.17 | 75.41±6.93 | 78.12±3.47 | 2.33±0.71 | 78.87±3.0
| FP | 88.62±0.74 | 76.96±13.43 | 21.45±12.34 |  90.46±0.39 | 99.01±0.53 | 0.9±0.48 |  88.86±0.57 | 35.23±32.96 | 43.05±25.15 | 86.18±0.58 |  2.3±0.97 | 85.95±0.74
| NAD | 84.57±4.03 | 80.76±11.9 | 17.23±10.57 | 87.96±2.82 | 91.49±10.17 | 6.97±8.46 | 86.51±1.53 | 18.38±29.23 | 47.13±25.21 | 81.1±3.69 | 4.23±2.27 | 81.48±4.1
| CLP | 88.64 | 84.77 | 14.31 | 90.65 | 96.37 | 3.27 | 87.63 |  1.19 |  81.46 | 82.83 | 13.47 | 81.77
| FT-SAM | 84.39±3.38 |  17.75±16.93 | 70.64±13.55 | 88.2±0.9 |  13.53±5.54 | 56.92±3.24 | 88.33±0.64 | 8.81±15.42 | 74.95±12.28 | 82.85±1.71 | 2.57±1.2 | 83.13±1.44
| ANP | 83.52±1.11 | 1.86±2.22 | 82.45±3.16 | 84.9±0.94 | 5.81±5.11 | 59.57±4.05 | 82.77±1.22 | 0.04±0.03 | 82.42±3.0 | 78.96±2.25 | 1.25±0.98 | 79.66±2.26
10 | Ours |  88.54±0.33 | 22.37±17.64 |  70.81±14.47 | 86.15±2.27 | 24.08±15.16 |  59.37±9.03 | 89.55±0.53 | 17.77±13.79 | 73.39±12.22 |  84.05±0.6 | 8.62±3.75 |  83.57±1.05
| FT | 88.08±0.31 | 17.09±7.96 | 74.58±6.33 | 89.51±0.39 | 79.37±18.45 | 17.46±14.86 | 90.94±0.25 | 3.33±2.16 | 83.97±2.57 |  87.17±0.31 | 1.39±0.22 |  87.28±0.46
| FP | 88.29±0.32 | 30.3±18.7 | 63.52±16.09 |  90.78±0.43 | 97.35±1.11 | 2.41±0.97 |  91.08±0.21 | 36.97±23.42 | 53.81±20.24 |  87.17±0.5 | 1.57±0.34 | 87.11±0.57
| NAD | 87.1±0.51 | 15.87±12.46 | 73.54±9.38 | 88.67±0.65 | 82.6±9.2 | 15.06±7.63 | 90.54±0.27 | 2.08±0.89 |  84.07±2.92 | 86.5±0.65 | 1.1±0.38 | 87.02±0.61
| CLP | 88.64 | 84.77 | 14.31 | 90.65 | 96.37 | 3.27 | 87.63 |  1.19 | 81.46 | 82.83 | 13.47 | 81.77
| FT-SAM | 89.97±0.26 |  6.86±4.21 |  84.26±3.54 | 91.01±0.19 | 16.56±7.05 | 60.14±3.78 | 91.35±0.21 | 2.94±1.47 | 85.21±2.11 | 88.98±0.14 | 1.6±0.26 | 88.83±0.25
| ANP | 84.91±1.38 | 0.39±0.15 | 86.18±1.71 | 86.14±0.91 | 1.85±1.47 |  65.17±1.56 | 81.64±0.52 | 0.01±0.01 | 82.48±2.13 | 80.17±2.35 |  1.24±0.66 | 80.38±2.72
100 | Ours |  88.69±0.25 | 26.22±16.89 | 67.75±13.87 | 83.84±3.5 |  2.29±1.09 | 73.14±2.49 | 89.64±0.63 | 22.04±26.73 | 68.89±22.98 | 82.52±3.2 | 6.45±3.85 | 82.77±3.57


$\displaystyle\Longrightarrow\text{Cone}\left(\pi_{-}^{*}{\mathcal{O}}_{\mathbb{P}^{3}}(1)[-1]\rightarrow\pi_{-}^{*}{\mathcal{O}}_{\mathbb{P}^{3}}(-2)\right)\,,$
$\displaystyle\pi_{+}^{*}\wedge^{2}\mathcal{S}_{SGr(2,4)}^{\vee}$
$\displaystyle\Longrightarrow\pi_{-}^{*}{\mathcal{O}}_{\mathbb{P}^{3}}(-1)\,,$
$\displaystyle\pi_{+}^{*}(\wedge^{2}\mathcal{S}_{SGr(2,4)}^{\vee})^{\otimes
2}$
$\displaystyle\Longrightarrow\pi_{-}^{*}{\mathcal{O}}_{\mathbb{P}^{3}}(-2)\,.$

##  5 Conclusion

In this paper, we have proposed a realization of the derived equivalence
between pairs of Calabi-Yau varieties, denoted as $(X_{+},X_{-})$, through a
GLSM framework. Our approach involves embedding $X_{+}$ and $X_{-}$ into
distinct phases of a GLSM for $\hat{X}$, as depicted in Fig. 1. This embedding
is designed to encode essential mixed branches $\mathcal{C}_{\pm}$ to ensure
the well-defined nature of the model. The matching of Euler characteristics,
or equivalently, the Witten indices in different phases, imposes the initial
constraint on this construction.

Subsequently, we constructed the functor of the equivalence between
$D^{b}(X_{+})$ and $D^{b}(X_{-})$ by tracing the brane transport across
various phases. Given that $\hat{X}$ typically constitutes a Fano variety in
our constructions, we computed the small window categories for some anomalous
$U(2)$ gauge theories outlined in section 3. This computational analysis plays
a pivotal role in our brane transport strategy. In section 4, we illustrated
our methodology through four concrete examples.

There are some limitations in the examples we have considered. For example, we
have only considered the gauge group $U(2)$ in the nonabelian part. As two-
parameter GLSMs, we have also only considered gauge groups $U(1)\times U(1)$
and $U(1)\times U(2)$. But we expect our construction can be generalized to
other two-parameter GLSMs with higher rank gauge groups straightforwardly.
This extension hinges on computing small windows in corresponding nonabelian
gauge theories. Also, as a very first attempt in computing the small window of
a non-abelian GLSM, it remains further explanation as well as examination on
the small window we found. Moreover, it is also interesting to apply a similar
method as presented in this paper to comprehend the generalized flip induced
derived equivalence as in [32] from a GLSM point of view. We shall leave such
considerations for future work.

## Acknowledgement

We would like to thank M. Romo, E. Segal and L. Smith for insightful
conversations. We are grateful for J. Knapp, M. Romo and E. Sharpe for their
careful reading of the manuscript and useful suggestions and comments. We
would also like to thank the workshop “GLSM@30” held at the Simons Center for
Geometry and Physics, which have stimulated many useful discussions. LB would
like to thank the hospitality of the Birmingham University when part of the
work was done. JG was supported in part by the Fundamental Research Funds for
the Central Universities and the China Postdoctoral Science Foundation No.
2020T130353. LB was supported by the Tsinghua scholarship for overseas
graduate studies. HZ was supported in part by the China Postdoctoral Science
Foundation under the grant No. 2022M720509.

##  Appendix A Count the Coulomb vacua

In this appendix, we give details about how the number of Coulomb vacua was
counted for the two examples discussed in section 3. The Coulomb vacuum
equations of the example in section 3.1 are given as

$\sigma_{1}^{n}=-q(-m\sigma_{1}-m\sigma_{2})^{m}\,,\quad\sigma_{2}^{n}=-q(-m\sigma_{1}-m\sigma_{2})^{m},$

where $\sigma_{1,2}\neq 0$ and $\sigma_{1}\neq\pm\sigma_{2}$. The above two
equations immediately give $\sigma_{1}^{n}=\sigma_{2}^{n}$, which means one
can take

$\sigma_{1}=\eta\sigma_{2},$

with $\eta$ being a $n$-th root of unity, $\eta^{n}=1$. Due to the exclusive
condition, $\sigma_{1}\neq\pm\sigma_{2}$, we should have $\eta\neq\pm 1$. When
$n$ is even $\eta$ has $n-2$ choices, while when $n$ is odd $\eta$ has $n-1$
choices. Then, given a choice of the above $\eta$, one needs to solve the
following equation

$\sigma_{2}^{n}=-q(-m)^{m}(1+\eta)^{m}\sigma_{2}^{m},$

and it has $(n-m)$ nonzero solutions. However, due to the Weyl symmetry
between $\sigma_{1}$ and $\sigma_{2}$, the total number of Coulomb vacua,
$N(n,m)$, should be

$N(n,m)=\begin{dcases}\frac{(n-2)(n-m)}{2}&n~{}\text{even}\,,\\\
\frac{(n-1)(n-m)}{2}&n~{}\text{odd}\,,\end{dcases}$

or, in short,

$N(n,m)=\left\lfloor\frac{n-1}{2}\right\rfloor(n-m)\,.$

Note that, for the case of $m=0$, we do not need to exclude the solution such
that $\sigma_{1}+\sigma_{2}=0$. The above derivation needs modification and,
therefore, $N(n,0)=\binom{n}{2}$.

Similarly, the Coulomb vacua equations of the example in section 3.2 are

$\displaystyle\sigma_{1}^{n}$
$\displaystyle=-(-m_{1})^{m_{1}}(-m_{2})^{m_{2}}(\sigma_{1}+\sigma_{2})^{m_{1}+m_{2}}\,,$
$\displaystyle\sigma_{2}^{n}$
$\displaystyle=-(-m_{1})^{m_{1}}(-m_{2})^{m_{2}}(\sigma_{1}+\sigma_{2})^{m_{1}+m_{2}}\,$

with $\sigma_{1,2}\neq 0$ and $\sigma_{1}\neq\pm\sigma_{2}$. Following the
same analysis as before, one can obtain the number of Coulomb vacua of this
example, $N(n,m_{1},m_{2})$, is given as

$N(n,m_{1},m_{2})=\left\lfloor\frac{n-1}{2}\right\rfloor(n-m_{1}-m_{2})\,.$

##  Appendix B Phases of the GLSM for $Fl(1,2,N)$

Gauge theories have been constructed to study the flag manifold in [29, 33,
34], but their phase structures were less explored. However, this would be
helpful for our discussion in the main context. Therefore, in this appendix,
we take a closer look at the two-step flag manifold $Fl(1,2,N)$. The GLSM for
the flag manifold $Fl(1,2,N)$ is a $U(1)\times U(2)$ gauge theory with chiral
matters $\Phi_{\alpha}$ and $X^{\alpha}_{i}$ with $\alpha=1,2$ and
$i=1,\dots,N$.

| $\Phi_{\alpha}$ | $X^{\alpha}_{i}$ |
---|---|---|---
$U(1)$ | 1 | 0 | $\xi_{1}$
$U(2)$ | $\overline{\tiny{\yng(1)}}$ | $\tiny{\yng(1)}$ | $\xi_{2}$
(76)

where $\xi_{1}$ and $\xi_{2}$ are the FI-parameters associated with the gauge
group $U(1)$ and $U(2)$ respectively. The supersymmetric vacuum is determined
by the D-term equations given as

$\displaystyle D$
$\displaystyle\sim\sum_{\alpha=1}^{2}|\Phi_{\alpha}|^{2}-\xi_{1}\,,$ (77)
$\displaystyle D^{\alpha}_{\beta}$
$\displaystyle\sim-\overline{\Phi}^{\alpha}\Phi_{\beta}+\sum_{i}\overline{X}_{\beta,i}X^{\alpha}_{i}-\xi_{2}\delta^{\alpha}_{\beta}\,.$

The Coulomb vacua equations are

$\displaystyle q_{1}\>=\>(\sigma-\sigma_{1})(\sigma-\sigma_{2})\,,$ (78)
$\displaystyle\sigma_{\alpha}^{N}\>=\>-q_{2}(\sigma-\sigma_{\alpha})\,,$

for $\alpha=1,2$. Here, we denote by $\sigma$ the scalar component of the
gauge field strength of $U(1)$, while $\sigma_{\alpha}$ the scalar components
of the gauge field strength valued in the Cartan part of $U(2)$.

The classical configurations determined by the above $D$-term equations can be
separated into four phases by different values of FI-parameters, as pictured
in Fig. 8. A detailed semi-classical discussion on each phases is presented as
follows:

##### Phase I: $\xi_{1},\xi_{2}\gg 0$.

This region consists of a pure Higgs branch, which realizes the flag manifold
$Fl(1,2,N)$ as the symplectic quotient [29]. One can also look at the solution
to the Coulomb vacua equations in large $\xi$ limit. Here, we take
$\xi_{1}\to+\infty$ and $\xi_{2}\to+\infty$, and the Coulomb vacua equations
(78) become

$(\sigma-\sigma_{1})(\sigma-\sigma_{2})=0\,,\quad\sigma_{1}^{N}=\sigma_{2}^{N}=0\,.$

The only solution is $\sigma=\sigma_{1}=\sigma_{2}=0$, indicating that there
is only one pure Higgs branch. This pure Higgs branch can be obtained
following [29], and it is the two-step flag manifold $Fl(1,2,N)$. Therefore,
the Witten index in this phase is just the Euler characteristic of the flag
manifold $Fl(1,2,N)$, which is

$\chi(Fl(1,2,N))\>=\>\binom{N}{2}\binom{2}{1}=N(N-1)\,.$

$I$$\xi_{1}$$\xi_{2}$$III$$IV$$II$$\xi_{1}+\xi_{2}=0$ Figure 8: Phase diagram
for $Fl(1,2,N)$.

##### Phase II: $\xi_{1}\ll 0,\xi_{2}\gg 0$.

Let us first look at the asymptotic behaviour of the Coulomb vacua in the
limit $\xi_{1}\to-\infty$ and $\xi_{2}\to+\infty$. Taking $\xi_{2}\to+\infty$,
we will have

$\sigma_{1}^{N}=\sigma_{2}^{N}=0\,.$

There is only a zero solution to the above equation $\sigma_{1}=\sigma_{2}=0$.
While the first equation becomes $\sigma^{2}=q_{1}$, there are two non-zero
solutions for $\sigma$: $\sigma=\pm\sqrt{q_{1}}$. The structure of vacua
solutions suggests that there are only mixed Higgs-Coulomb branches in this
phase.

Now let us find out what the mixed Higgs-Coulomb branches look like. When
$\xi_{1}$ goes from positive to negative, there is no classical solution to
the $U(1)$ D-term equation. In fact, the $U(1)$ part flows to the large
$\sigma$ region as discussed in the previous paragraph. The field
$\Phi_{\alpha}$ has mass proportional to $|\sigma|$ and should be integrated
out. On the other hand, the fields $X^{\alpha}_{i}$ remain massless, and
should satisfy the remaining $U(2)$ D-term equations which now become

$\sum_{i=1}^{N}\overline{X}_{\beta,i}X^{\alpha}_{i}\>=\>\xi_{2}\delta^{\alpha}_{\beta}\,.$

This leads to nothing but the $Gr(2,N)$ [14, 35]. Therefore, the mixed Higgs-
Coulomb branch in this phase consists of two copies of $Gr(2,N)$, each of
which is located at a non-zero point in the $\sigma$-plane. Please see part
(a) of Fig. 9 for the case of $N=4$. Therefore, the total Witten index in this
region is

$2\times\chi(Gr(2,N))\>=\>2\times\binom{N}{2}\>=\>N(N-1)\,.$

This is consistent with the total Witten index obtained in Phase I.

$Gr(2,4)$$Gr(2,4)$$\sigma$$(a)$

$\mathbb{P}^{3}$$\mathbb{P}^{3}$$\mathbb{P}^{3}$$\sigma_{2}$$(b)$

Figure 9: Structure of the mixed Higgs-Coulomb branches in different phases
when $N=4$.

##### Phase III: $\xi_{1}+\xi_{2}\gg 0,\xi_{2}\ll 0$.

Let us again first look at the asymptotic behaviour of the Coulomb vacua
equations (78). In the limit $\xi_{1}\to+\infty$, the first equation becomes

$(\sigma-\sigma_{1})(\sigma-\sigma_{2})=0\,,$

and therefore $\sigma=\sigma_{1}$ or $\sigma=\sigma_{2}$. Without loss of
generality, we can choose $\sigma=\sigma_{1}$ to fix the Weyl symmetry. Then,
we have $\sigma_{1}^{N}=0$, and so $\sigma=\sigma_{1}=0$. The remaining vacua
equation is

$\sigma_{2}^{N}=q_{2}\sigma_{2}\,,$

which has one zero solution $\sigma_{2}=0$ and $N-1$ nonzero solutions
$\sigma_{2}=e^{2\pi i\frac{\ell}{N-1}}q_{2}^{\frac{1}{N-1}}$ with
$\ell=0,\dots,N-2$. Namely, in this phase, it seems that we have two sets of
solutions:

$\\{\sigma=\sigma_{1}=\sigma_{2}=0\\}\quad\text{and}\quad\\{\sigma=\sigma_{1}=0,\sigma_{2}=e^{2\pi
i\frac{\ell}{N-1}}q_{2}^{\frac{1}{N-1}}|\ell=0,\dots,N-2\\}\,.$

However, the zero solution, $\\{\sigma=\sigma_{1}=\sigma_{2}=0\\}$, is in fact
not a honest solution, since there is no solutions to the $D$-term equation
associated with $U(2)$ when $\xi_{2}\ll 0$, by viewing $X^{\alpha}_{i}$ as
entries of a hermitian matrix.

Another way to see this is to look at the IR effective theory. In fact, the
GLSM flows to an effective gauged nonlinear sigma model defined by
$Z_{i}:=\Phi_{\alpha}X^{\alpha}_{i}$ and
$B_{ij}:=\epsilon_{\alpha\beta}X^{\alpha}_{i}X^{\beta}_{j}$ with gauge group
and charges:

| $Z_{i}$ | $B_{ij}$
---|---|---
$U(1)$ | 1 | 0
$\det U(2)$ | 0 | 1

The nonlinearity resides in the relations among coordinates $B_{ij}$ by
definition, and this is the reason why we call this effective theory as a
gauged nonlinear sigma model. One should note that the coordinates $Z_{i}$ and
$B_{ij}$ are also introduced in [29, section 2.2] for studying the Plücker
embedding of the flag manifolds. Apparently, the second equation in (77) has
no classical solution when $\xi_{2}$ is negative, which contradicts with the
pure Higgs branch suggested by the zero Coulomb vacua solution.

While for each one of the remaining $N-1$ non-zero solutions, we have to
integrate out $B_{ij}$’s as they are now massive and then the effective GLSM
is just a $U(1)$ gauge theory with $N$ charge-1 chiral fields. Of course, we
have to check the corresponding D-term equation has solutions in the present
phase. Note that

$\displaystyle\sum_{i=1}^{N}|Z_{i}|^{2}$
$\displaystyle=\sum_{i=1}^{N}\sum_{\alpha,\beta}\overline{\Phi}^{\beta}\Phi_{\alpha}\overline{X}_{\beta,i}X^{\alpha}_{i}=\sum_{\alpha,\beta}\overline{\Phi}^{\beta}\Phi_{\alpha}(\overline{\Phi}^{\alpha}\Phi_{\beta}+\xi_{2}\delta^{\alpha}_{\beta}),$
$\displaystyle=\sum_{\alpha=1}^{2}|\Phi_{\alpha}|^{2}(\sum_{\beta=1}^{2}|\Phi_{\beta}|^{2}+\xi_{2})=(\xi_{1}+\xi_{2})\sum_{\alpha=1}^{2}|\Phi_{\alpha}|^{2}=\xi_{1}(\xi_{1}+\xi_{2})>0$

in the region $\xi_{1}\gg 0$, $\xi_{2}\ll 0$ and $\xi_{1}+\xi_{2}\gg 0$, the
D-term equation indeed has solutions, giving rise to the vacuum manifold
inside the mixed Higgs-Coulomb branch, which is a projective space
$\mathbb{P}^{N-1}$.

Thus, this phase only has mixed Higgs-Coulomb branches, and, more
specifically, they are $N-1$ copies of $\mathbb{P}^{N-1}$ located at $N-1$
non-zero points on the Coulomb branch, see part (b) of Fig. 9 for the $N=4$
case. Based on this picture, one can calculate the total Witten index in this
phase, which is

$(N-1)\times\chi({\mathbb{P}}^{N-1})\>=\>N(N-1)\,,$

matching the results from the other phases.

##### Phase IV: $\xi_{1}+\xi_{2}\ll 0,\xi_{2}\ll 0$.

This is a pure Coulomb branch. First, it is clear that the region where
$\xi_{1}\ll 0$ and $\xi_{2}\ll 0$ is of a pure Coulomb branch. Then, one can
argue that there are no solutions to the D-term equations in the region
$\xi_{1}\gg 0$ and $\xi_{1}+\xi_{2}\ll 0$, by following the previous
discussions.

In this case, the total Witten index only has contributions from the Coulomb
vacua, and the total number of solutions to the Coulomb vacua equations is
indeed $N(N-1)$.

## References

  * [1] E. Segal, “A new 5-fold flop and derived equivalence,” Bulletin of the London Mathematical Society 48 no. 3, (Apr, 2016) 533–538.
  * [2] M. Herbst, K. Hori, and D. Page, “Phases Of N=2 Theories In 1+1 Dimensions With Boundary,” arXiv:0803.2045 [hep-th].
  * [3] E. R. Sharpe, “D-branes, derived categories, and Grothendieck groups,” Nucl. Phys. B 561 (1999) 433–450, arXiv:hep-th/9902116.
  * [4] M. R. Douglas, “D-branes, categories and N=1 supersymmetry,” J. Math. Phys. 42 (2001) 2818–2843, arXiv:hep-th/0011017.
  * [5] K. Hori and M. Romo, “Exact Results In Two-Dimensional (2,2) Supersymmetric Gauge Theories With Boundary,” arXiv:1308.2438 [hep-th].
  * [6] J. Clingempeel, B. Le Floch, and M. Romo, “Brane transport in anomalous (2,2) models and localization,” arXiv:1811.12385 [hep-th].
  * [7] W. Donovan, “Grassmannian twists, derived equivalences and brane transport,” Proc. Symp. Pure Math. 90 (2015) 251–264, arXiv:1304.2913 [math.AG].
  * [8] A. Gerhardus and H. Jockers, “Dual pairs of gauged linear sigma models and derived equivalences of Calabi–Yau threefolds,” J. Geom. Phys. 114 (2017) 223–259, arXiv:1505.00099 [hep-th].
  * [9] K. Hori and J. Knapp, “A pair of Calabi-Yau manifolds from a two parameter non-Abelian gauged linear sigma model,” arXiv:1612.06214 [hep-th].
  * [10] B. Lin and M. Romo, “B-brane Transport and Grade Restriction Rule for Determinantal Varieties,” To appear .
  * [11] J. Guo, M. Romo, and L. Smith To appear .
  * [12] K. Hori, J. Knapp, and M. Romo To appear .
  * [13] R. Eager, K. Hori, J. Knapp, and M. Romo, “Beijing lectures on the grade restriction rule,” Chinese Annals of Mathematics, Series B 38 no. 4, (Jul, 2017) 901–912.
  * [14] E. Witten, “Phases of N=2 theories in two-dimensions,” Nucl. Phys. B 403 (1993) 159–222, arXiv:hep-th/9301042.
  * [15] K. Hori and J. Walcher, “D-branes from matrix factorizations,” Comptes Rendus Physique 5 (2004) 1061–1070, arXiv:hep-th/0409204.
  * [16] P. Griffiths and J. Harris, Principles of algebraic geometry.  John Wiley & Sons, 2014.
  * [17] K. Hori and D. Tong, “Aspects of Non-Abelian Gauge Dynamics in Two-Dimensional N=(2,2) Theories,” JHEP 05 (2007) 079, arXiv:hep-th/0609032.
  * [18] K. Hori, “Duality In Two-Dimensional (2,2) Supersymmetric Non-Abelian Gauge Theories,” JHEP 10 (2013) 121, arXiv:1104.2853 [hep-th].
  * [19] W. Lerche, P. Mayr, and J. Walcher, “A New kind of McKay correspondence from nonAbelian gauge theories,” arXiv:hep-th/0103114.
  * [20] D. Orlov, “Derived categories of coherent sheaves and triangulated categories of singularities,” Algebra, Arithmetic, and Geometry: Volume II: In Honor of Yu. I. Manin (2009) 503–531, arXiv:math/0503632 [math.AG].
  * [21] P. S. Aspinwall, “The Landau-Ginzburg to Calabi-Yau dictionary for D-branes,” J. Math. Phys. 48 (2007) 082304, arXiv:hep-th/0610209.
  * [22] G. Ottaviani, “Spinor bundles on quadrics,” Transactions of the American mathematical society 307 no. 1, (1988) 301–316.
  * [23] J. A. Eagon and D. G. Northcott, “Ideals defined by matrices and a certain complex associated with them,” Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 269 no. 1337, (1962) 188–204.
  * [24] M. Bertolini, I. V. Melnikov, and M. R. Plesser, “Hybrid conformal field theories,” JHEP 05 (2014) 043, arXiv:1307.7063 [hep-th].
  * [25] D. Erkinger and J. Knapp, “On genus-0 invariants of Calabi-Yau hybrid models,” JHEP 05 (2023) 071, arXiv:2210.01226 [hep-th].
  * [26] A. Adem, J. Leida, and Y. Ruan, Orbifolds and Stringy Topology.  Cambridge Tracts in Mathematics. Cambridge University Press, 2007.
  * [27] N. M. Addington, E. P. Segal, and E. Sharpe, “D-brane probes, branched double covers, and noncommutative resolutions,” Adv. Theor. Math. Phys. 18 no. 6, (2014) 1369–1436, arXiv:1211.2446 [hep-th].
  * [28] Z. Chen, J. Guo, and M. Romo, “A GLSM View on Homological Projective Duality,” Commun. Math. Phys. 394 no. 1, (2022) 355–407, arXiv:2012.14109 [hep-th].
  * [29] R. Donagi and E. Sharpe, “GLSM’s for partial flag manifolds,” J. Geom. Phys. 58 (2008) 1662–1692, arXiv:0704.1761 [hep-th].
  * [30] W. Gu, E. Sharpe, and H. Zou, “GLSMs for exotic Grassmannians,” JHEP 10 (2020) 200, arXiv:2008.02281 [hep-th].
  * [31] J. Guo and H. Zou, “Quantum cohomology of symplectic flag manifolds,” J. Phys. A 55 no. 27, (2022) 275401, arXiv:2107.09880 [hep-th].
  * [32] A. Bondal and D. Orlov, “Derived categories of coherent sheaves,” arXiv Mathematics e-prints (June, 2002) math/0206295, arXiv:math/0206295 [math.AG].
  * [33] K. Ohmori, N. Seiberg, and S.-H. Shao, “Sigma Models on Flags,” SciPost Phys. 6 no. 2, (2019) 017, arXiv:1809.10604 [hep-th].
  * [34] J. Guo, “Quantum Sheaf Cohomology and Duality of Flag Manifolds,” Commun. Math. Phys. 374 no. 2, (2019) 661–688, arXiv:1808.00716 [hep-th].
  * [35] E. Witten, “The Verlinde algebra and the cohomology of the Grassmannian,” arXiv:hep-th/9312104.


# Boosting Method for Automated Feature Space Discovery in Supervised Quantum
Machine Learning Models

Vladimir Rastunkov  IBM Quantum, Industry & Technical Services,
Jae-Eun Park  IBM Quantum, Industry & Technical Services,
Abhijit Mitra  IBM Quantum, Industry & Technical Services,
Brian Quanz  IBM T.J. Watson Research Center, Yorktown Heights,
Steve Wood  IBM T.J. Watson Research Center, Yorktown Heights,
Christopher Codella  IBM Quantum, Industry & Technical Services,
Heather Higgins  IBM Quantum, Industry & Technical Services,
Joseph Broz  IBM Quantum, Industry & Technical Services

###### Abstract

Quantum Support Vector Machines (QSVM) have become an important tool in
research and applications of quantum kernel methods. In this work we propose a
boosting approach for building ensembles of QSVM models and assess performance
improvement across multiple datasets. This approach is derived from the best
ensemble building practices that worked well in traditional machine learning
and thus should push the limits of quantum model performance even further. We
find that in some cases, a single QSVM model with tuned hyperparameters is
sufficient to simulate the data, while in others - an ensemble of QSVMs that
are forced to do exploration of the feature space via proposed method is
beneficial.

QSVM, quantum kernel, boosting, ensemble

##  1 Introduction

Decision support systems across multiple industries rely on heuristic
approaches using models trained on historical data. For example, credit risk,
propensity, attrition, fraud, and hospital readmission risk models classify
data into two classes given a set of input features. The goal is to train
analytical models that give the most accurate predictions, retain stable
performance over time and utilize fewer features possible by efficiently
extracting information from available data. Generally, the better the model
can describe complex non-linear interactions between features, the higher its
performance and ability to generalize the data will be. Conventional, non-
quantum, machine learning models with higher performance are often achieved
through ensembles of simpler learners. See, for example [1, 2, 3].

Havlicek, et al. [4] implemented a quantum support vector machine classifier
(QSVM) on a superconducting processor. Originally proposed in [5], QSVM
exploits a high-dimensional quantum Hilbert space to obtain an enhanced
solution. This enhancement can be achieved through controlled entanglement and
interference, which is inaccessible for classical support vector machines.
However, superior performance of QSVM or other quantum approaches compared to
traditional machine learning models is yet to be demonstrated on a practical
dataset. Park et al. [6] demonstrated improvements to QSVM compared to
classical SVM by using parameterized shallow unitary transformations for
feature maps with rotation and regularization. Wu et al. [7] provided
benchmarks comparing performance of QSVM built using a simulator and physical
hardware with classical SVM and xgboost. Those benchmarks built using three
different platforms, IBM Quantum, Google Tensorflow Quantum and Amazon
Bracket, indicated similar performance of QSVM compared to its classical
counterparts on a practical dataset. Another recent paper by Glick et al. [8]
discusses a class of covariant kernels and quantum advantage for problems
where the data satisfies a group structure.

The idea of boosting quantum machine learning models was previously discussed
by Neven et al. [9] in the context of adiabatic quantum computing implemented
on D-Wave annealers, where authors used one level decision trees as weak
classifiers. Papers by Schuld et al. and Abbas et al. [10, 11] discussed
quantum ensembles of quantum classifiers primarily from a perspective of
speedup due to parallel calculation.

In general, quantum machine learning (QML) models consist of data encoding
into qubits, a variational quantum circuit with trainable parameters, a
classical cost function and an optimization algorithm. Most of the QML models
constructed this way are mathematically related to quantum kernel methods
[12]. Notably, initial state preparation and subsequent unitary transformation
with input features is carried out through a circuit called feature map.
Unlike other types of machine learning algorithms, the choice of initial
feature map in quantum support vector machines (QSVM) could yield unique
decision boundaries, making QSVMs with different feature maps independent from
each other. This characteristic of QSVM is well suited for implementing
boosting algorithms, however, given a large number of choices of feature maps
the automation of feature map/model selection and training process is quite
desirable.

In addition to achieving better performance with quantum models, there is a
need to make them more user-friendly by automating the model selection and
training process. Currently, model architectures are often derived from well-
known physical models, e.g. an Ising model and respective Hamiltonian were
used for feature mapping. Thus, an automated procedure can augment some of the
unnecessary complexity of the existing models for users without a physics
background as well as assist in discovering new model architectures.

The approach presented in the current work is different from results discussed
in the referenced sources in the following respects. First, we are focused on
universal quantum computing with gates that can run on superconducting qubits
as implemented, e.g. in IBM Quantum System One. Secondly, even though we
consider shallow circuits for kernel functions, those tend to be stronger than
typical weak classifiers discussed in the literature, which is mostly based on
decision trees. Thirdly, we implement an automated model selection on every
boosting step to choose from different topologies and thus explore wider
feature and model spaces. This process can be used to search for alternatives
for broadly used Ising-type models. Fourthly, our approach is not constrained
to classification tasks, it can be equally applied to regression tasks.
Lastly, the focus of many prior results was on the possibility of quantum
speedup for classical procedures, whereas our focus is primarily on the
development of models with higher performance. In this work we are applying
our approach to the problem of binary classification and use classification
accuracy on the test sample as a measure of performance.

Our main contributions are:

  * •

A new ensemble method for QSVM that enhances model performance, when the data
is difficult to model for a single learner.

  * •

Hyperparameter optimization for QSVM.

  * •

Simulation on multiple datasets to ensure stability of results.

##  2 Boosting Method for QSVMs

###  2.1 Data and Data Encoding

In this work we consider classical examples of generated data, such as moons,
circles and XOR. This allows creation of many different datasets to accumulate
statistics of model performance.

Following the best practice, the data is split into training, validation and
testing datasets. A validation dataset is used for hyperparameter tuning in
the process of grid search for the best model on every step of the boosting
procedure. A testing dataset is completely hidden from the training and is
used to compare different models.

Following [4] we define a feature map on $n$-qubits as

${\mathcal{U}}_{\Phi}(\vec{x})=U_{\Phi(\vec{x})}H^{{\otimes}n}$ (1)

where

$U_{\Phi(\vec{x})}=\exp\left(i\sum_{S\subseteq[n]}\phi_{S}(\vec{x})\prod_{i\in
S}P_{i}\right)$ (2)

Here $H$ is the Hadamard gate and $P_{i}\in\\{I,X,Y,Z\\}$. A set of feature
maps that we can utilize for a grid search, when the data has two features is
shown in Fig. 1.

Figure 1: Set of feature maps for grid search.

###  2.2 Ensemble Structure

The traditional AdaBoost variant of boosting relies on weak learners, such as
decision stumps, that are trained on every iteration [3]. For each subsequent
iteration it emphasizes examples that were previously misclassified by
calculating and assigning or updating their weights. The final prediction is
calculated by weighted majority vote of classifiers. In this work we consider
support vector machines on quantum kernels
$k(\vec{x}_{i},\vec{x}_{j})=|\langle{\mathcal{U}}_{\Phi}(\vec{x}_{i})|{\mathcal{U}}_{\Phi}(\vec{x}_{j})|^{2}$
that we call Quantum Support Vector Machines (QSVM). QSVM is not a weak
learner, so we modify the boosting method as shown in Algorithm 1.

Algorithm 1 Boosted QSVM classifier.

Input
${X}_{train},{y}_{train},{X}_{val},{y}_{val},{y}_{train,i}\in\\{0,1\\},{y}_{val,i}\in\\{0,1\\},$
grid parameters for QSVM
Output $G(x)$

1:Initialize ${w_{i}}=1,$ $\forall{i}$.

2:for $m=1$ to $M$ do

3: Perform grid search and select the best classifier $G_{m}(x)$ on
$({X}_{train},{y}_{train},{X}_{val},{y}_{val})$ taking into account exclusions
from the grid and training weights ${w_{i}}$

4: Check early stopping conditions for perfect and worse than random guessing
classification.

5: Exclude selected feature map from grid parameters for next iterations.

6: Compute ${err}_{m}=\frac{\sum_{i=1}^{N}{w_{i}}\cdot I({y}_{train,i}\neq
G_{m}({X}_{train,i}))}{\sum_{i=1}^{N}{w_{i}}}$ (estimator error)

7: Compute ${\alpha}_{m}=\log((1-{err}_{m})/{err}_{m})$ (estimator weight)

8: Set ${w}_{i}\leftarrow{w}_{i}\cdot\exp[{\alpha}_{m}\cdot
I({y}_{train,i}\neq G_{m}({X}_{train,i}))]$

9:Output
$G(x)=\sum_{m=1}^{M}({\alpha}_{m}G_{m}(x))/\sum_{m=1}^{M}({\alpha}_{m})$

In the beginning the algorithm receives training and validation datasets as
well as grid search parameters. In this work we consider the following
parameters: Pauli feature map set as shown in Fig. 1, the Pauli rotation
factor, which is a multiplier to the Pauli rotations (alpha) and a
regularization parameter (C) for sklearn’s support vector classifier (SVC). We
vary alpha in the interval $(0;2]$, C in $[1;100]$. All examples are initially
assigned a weight of $1$. Grid search uses a validation dataset to select the
best model. After the best model is selected we check early stopping
conditions:

  1. 1.

Estimator is perfect, i.e. estimator error on the training dataset is $\leq
0$.

  2. 2.

Estimator is as bad as random guessing or worse, i.e. estimator error is $\geq
0.5$ for binary classification or $\geq 1-\frac{1}{{N}_{classes}}$ for
multiclass.

  3. 3.

The maximum number of classifiers is reached.

The feature map selected on the current iteration is excluded from the grid
search for next iterations. This is important to force the model to explore a
broader Hilbert space and, consequently, different decision boundaries by
choosing other feature maps for the quantum kernel. Finally, the weights are
updated as shown in Algorithm 1. Once any stopping condition is satisfied then
the final model object is returned. This object can be used to build
predictions for new samples as a weighted majority vote of classifiers
included in the model.

It is worthwhile to highlight differences of the approach presented here from
more traditional boosting: 1) we perform a grid search for the best model on
each iteration of the algorithm, 2) we enforce exploration of different model
architectures through parameter grid constrains.

Identifying the optimal number of estimators and ensemble pruning is generally
outside of the boosting method description and is up to the user. In this work
we will choose the optimal number of estimators based on the minimum error on
the validation sample.

###  2.3 Numerical Simulation Results

First, we run experiments on simulated data created with functions available
in scikit-learn (see Fig. 2). This allows us to create a number of
statistically independent datasets and obtain averaged performance metrics. In
this study we chose to generate 50 datasets of each kind: XOR, moons and
circles.

Figure 2: Different datasets used in experiments.

Each dataset has 150 observations for training, validation and testing split
equally between those subsets. We train a boosted QSVM as described above for
each dataset. For comparison, we also train an SVM and xgboost. The parameter
grid for the SVM includes RBF and linear kernels, regularization C ranging
from 0.1 to 100, gamma parameter for RBF kernel ranging from 0.0001 to 10. The
parameter grid for xgboost was constructed following [13].

Figure 3: Model accuracy comparison box-plots. Lines show median accuracy on
the test sample, boxes show the range between the lower and upper quartiles,
and whiskers indicate ”the range of the data” following Tukey’s definition
with $Q1-1.5\cdot(Q3-Q1)$ and $Q3+1.5\cdot(Q3-Q1)$ for lower and upper
whiskers, respectively.

The results are shown in Fig. 3. The performance on the XOR dataset seems
comparable across the three models. Boosted QSVM struggles to achieve
comparable performance on the moons dataset, but works best on the circles
dataset with median at $100\%$ accuracy.

An interesting question is whether a Boosted QSVM actually benefited from the
ensemble and if so then how much improvement did it provide. It turns out that
only about $31\%$ of Boosted QSVM models contain more than 1 estimator in the
ensemble. Table 1 shows mean and maximum ensemble size by dataset. The more
difficult the dataset for QSVM is, the larger the ensemble seems to be: more
than 1 estimator is barely used for circles data, while 3.8 estimators on
average are used for moons data.

Table 1: Average and Maximum Model Ensemble Size Per Dataset Dataset | Mean | Max
---|---|---
XOR | 2.02 | 10
circles | 1.06 | 3
moons | 3.84 | 10

We have also investigated whether there is a performance gain from having
multiple classifiers. Table 2 shows classification accuracy improvement from
an ensemble of QSVM classifiers compared to a single QSVM. There is a small
sample size for circles data, where even a single QSVM is doing well. There is
an average of $4.2\%$ and $7.5\%$ classification accuracy improvement for XOR
and moons.

Table 2: QSVM Accuracy Increase with Boosting Dataset | Mean | Max |  Number of ensembles with $>{2}$ learners (out of 50)
---|---|---|---
XOR | $4.2\%$ | $16.0\%$ | 36
circles | $2.0\%$ | $2.0\%$ | 2
moons | $7.5\%$ | $14.0\%$ | 24

##  3 Conclusions

Data scientists across multiple industries continue to push limits in their
search for the best-in-class machine learning model that would provide a
competitive edge. Quantum machine learning holds a promise of even higher
performance than classical due to enhanced feature spaces. The approach
discussed here is derived and adapted from the best ensemble building
practices that worked well in traditional machine learning and thus should
push the limits of model performance even further. Examples discussed in this
work show that boosted QSVM ensembles outperform single QSVMs that in some
cases allows them to match accuracy of non-quantum models, and in other cases
- even exceed it.

## References

  * [1] Y. Freund and R. E. Schapire, “Experiments with a new boosting algorithm,” _Machine Learning: Proceedings of the Thirteenth International Conference_ , 1996.
  * [2] L. Breiman, “Random forests,” _Machine Learning_ , p. 5–32, 2001.
  * [3] T. Hastie, R. Tibshirani, and J. Friedman, _The elements of statistical learning: data mining, inference and prediction_. Springer, 2017.
  * [4] V. Havlíček, A. D. Córcoles, K. Temme, A. W. Harrow, A. Kandala, J. M. Chow, and J. M. Gambetta, “Supervised learning with quantum-enhanced feature spaces,” _Nature_ , vol. 567, no. 7747, pp. 209–212, 2019.
  * [5] P. Rebentrost, M. Mohseni, and S. Lloyd, “Quantum support vector machine for big data classification,” _Physical review letters_ , vol. 113, no. 13, p. 130503, 2014.
  * [6] J.-E. Park, B. Quanz, S. Wood, H. Higgins, and R. Harishankar, “Practical application improvement to quantum svm: theory to practice,” _arXiv:2012.07725_ , 2020.
  * [7] S. L. Wu, S. Sun, W. Guan, C. Zhou, J. Chan, C. L. Cheng, T. Pham, Y. Qian, A. Z. Wang, R. Zhang, M. Livny, J. Glick, P. K. Barkoutsos, S. Woerner, I. Tavernelli, F. Carminati, A. D. Meglio, A. C. Y. Li, J. Lykken, P. Spentzouris, S. Y.-C. Chen, S. Yoo, and T.-C. Wei, “Application of quantum machine learning using the quantum kernel algorithm on high energy physics analysis at the lhc,” 2021.
  * [8] J. R. Glick, T. P. Gujarati, A. D. Corcoles, Y. Kim, A. Kandala, J. M. Gambetta, and K. Temme, “Covariant quantum kernels for data with group structure,” 2021.
  * [9] H. Neven, V. S. Denchev, G. Rose, and W. G. Macready, “Training a large scale classifier with the quantum adiabatic algorithm,” _arXiv:0912.0779_ , 2009\.
  * [10] M. Schuld and F. Petruccione, “Quantum ensembles of quantum classifiers,” _Sci Rep_ , vol. 8, no. 2772, 2018.
  * [11] A. Abbas, M. Schuld, and F. Petruccione, “On quantum ensembles of quantum classifiers,” _arXiv:2001.10833_ , 2020.
  * [12] M. Schuld, “Quantum machine learning models are kernel methods,” _arXiv:2101.11020_ , 2021.
  * [13] C. Wade, _Hands-On Gradient Boosting with XGBoost and scikit-learn [electronic resource] / Wade, Corey._ , 1st ed. Packt Publishing, 2020.