# Does Few-shot Learning Suffer from Backdoor Attacks?

Xinwei Liu1,2 , Xiaojun Jia3, Jindong Gu4, Yuan Xun1,2
Siyuan Liang5, Xiaochun Cao6* Correspondence to: Xiaojun Jia and Xiaochun Cao.

###### Abstract

The field of few-shot learning (FSL) has shown promising results in scenarios
where training data is limited, but its vulnerability to backdoor attacks
remains largely unexplored. We first explore this topic by first evaluating
the performance of the existing backdoor attack methods on few-shot learning
scenarios. Unlike in standard supervised learning, existing backdoor attack
methods failed to perform an effective attack in FSL due to two main issues.
Firstly, the model tends to overfit to either benign features or trigger
features, causing a tough trade-off between attack success rate and benign
accuracy. Secondly, due to the small number of training samples, the dirty
label or visible trigger in the support set can be easily detected by victims,
which reduces the stealthiness of attacks. It seemed that FSL could survive
from backdoor attacks. However, in this paper, we propose the Few-shot
Learning Backdoor Attack (FLBA) to show that FSL can still be vulnerable to
backdoor attacks. Specifically, we first generate a trigger to maximize the
gap between poisoned and benign features. It enables the model to learn both
benign and trigger features, which solves the problem of overfitting. To make
it more stealthy, we hide the trigger by optimizing two types of imperceptible
perturbation, namely attractive and repulsive perturbation, instead of
attaching the trigger directly. Once we obtain the perturbations, we can
poison all samples in the benign support set into a hidden poisoned support
set and fine-tune the model on it. Our method demonstrates a high Attack
Success Rate (ASR) in FSL tasks with different few-shot learning paradigms
while preserving clean accuracy and maintaining stealthiness. This study
reveals that few-shot learning still suffers from backdoor attacks, and its
security should be given attention.

## Introduction

Deep learning has demonstrated remarkable success in a variety of computer
vision applications, particularly in the area of image classification (He et
al. 2016; Gu et al. 2018). However, this success is contingent upon access to
a significant amount of training data. In many real-world scenarios, only a
few labeled samples are available for new unseen classes, such as rare species
or medical diseases. Learning a classifier when only a few training samples
are available for each class is well known as few-shot learning (FSL) in the
literature (Vinyals et al. 2016; Finn, Abbeel, and Levine 2017). The FSL
approach involves using a large auxiliary set of labeled data from disjoint
classes to acquire transferable knowledge or representations that can help in
the few-shot tasks. Recently, the security implications of FSL have been
brought to the forefront of the community (Li et al. 2022a; Guan et al. 2022),
such as the challenge of training a robust few-shot model against adversarial
attacks (Li et al. 2019b; Jia et al. 2020; Huang et al. 2021, 2023).

Figure 1: Results of six backdoor attack methods with different poison rates
on the 5-way 5-shot learning task. The poisoning rate of 0.2 means the
selection of one image of each class for the dirty-label method or one image
of the target class for the clean-label. The top of the figures shows the
visualization of the poisoned support set with BadNet and TUAP, which are both
easily detected by victims as their dirty labels or visible triggers.  Figure
2: The pipeline of our FLBA. We take the Baseline++ (Chen et al. 2019) method
as an example. Our method is divided into four main phases. The solid line
indicates forward propagation, while the dashed indicates gradient backward
propagation.

Apart from adversarial attacks (Liang et al. 2022; Liu et al. 2022; Gu et al.
2022, 2023; He et al. 2023), the backdoor attack (Gudibande et al. ; Gao et
al. 2023; Li et al. 2023) has also received great attention since they pose
potential threats to DNN-based applications. Backdoor attacks plant a backdoor
into a victim model by injecting a trigger pattern into a small subset of
training sample (Li et al. 2022b; Gu et al. 2019; Chen et al. 2017). At the
testing stage, the infected model appears to behave normally for clean samples
but produces intentional misclassifications for samples with the specific
trigger. In traditional classification tasks, this type of attack often occurs
when users obtain available datasets or pre-trained models from untrusted
sources. In FSL, the training process involves two phases: first, training a
backbone model or feature embedding with the auxiliary set, and second,
retraining or fine-tuning the model with the support set. Generally, the
auxiliary set is a clean dataset downloaded from an official website such as
miniImageNet (Vinyals et al. 2016), or users directly use a trustworthy pre-
trained model without training themselves, which the attacker cannot access to
poison. However, the trainable samples in the support set are usually scarce
due to data collection difficulties or privacy copyrights. When a practitioner
is unable to collect some trusted private data by themselves, they will
compromise to obtain risky images from untrustworthy sources, even if they
know that the data might not be secure enough. In this work, we focus on
backdoor attacks in the second phase, when the victim fine-tunes the model
with the support set. Much of the current research on backdoor attacks have
centered around standard image classification (Bagdasaryan and Shmatikov 2021;
Nguyen and Tran 2020), the threat of backdoor attacks in few-shot learning is
currently yet to be explored. Given this, a question relevant to the community
is: Does few-shot learning suffer from backdoor attacks?

We reveal that embedding a backdoor in FSL is more challenging compared to
traditional classification tasks. Concretely, we selected six well-known
backdoor attack methods, namely BadNet (Gu et al. 2019), Blended (Chen et al.
2017), Refool (Liu et al. 2020b), WaNet (Nguyen and Tran 2021),
LabelConsistent (Turner, Tsipras, and Madry 2019), and TUAP (Zhao et al.
2020), where the first four are dirty-label methods, and the last two are
clean-label methods. We apply them to 5-way 5-shot tasks on the Baseline++
(Chen et al. 2019), and the results with different poison rates are depicted
in Fig 1. From this figure, it can be observed that these methods are all
difficult to get a successful attack result, where they fail to achieve a high
Attack Success Rate (ASR) and maintain Benign Accuracy (BA) at the same time.
Through our analysis, this is due to the limited training samples, resulting
in the model either overfitting to poisoned features or to clean features.
Besides being ineffective, we also observe that these methods have poor
stealthiness in FSL. In traditional classification, it could be difficult for
the victims to detect the label modification with abundant training data and a
low poisoning rate. However, in 5-way 5-shot FSL tasks, the poisoning rate is
as high as 0.2 even for only one dirty-label poisoned image in each class,
while the poisoning rate in traditional classification is usually less than
0.1. Consequently, these wrong labels can easily be detected and corrected by
the victims. Although a clean-label approach can avoid this issue, the
presence of a visible trigger similarly attracts the victim’s attention. We
take BadNet and TUAP as examples to show their poor stealthiness at the top of
Fig 1.

The above experiment seems to indicate that FSL is immune to backdoor attacks.
However, in this work, we propose a backdoor attack method, dubbed Few-shot
Learning Backdoor Attack (FLBA), to overcome the limitations of existing
methods. The proposed method consists of four steps, and the framework is
illustrated in Figure 2. Firstly, we optimize a trigger with embedding
deviation to increase the distance between the clean and trigger features,
which enables the model to learn the backdoor boundary without overfitting one
of them. Next, we try to hide the trigger by generating imperceptible
perturbation for the clean images. Specifically, we design two types of
perturbations with max-min similarity loss for both targeted samples and
untargeted samples. The attractive perturbation attracts the feature towards
the trigger feature for the target class, while the repulsive perturbation
moves the feature away from the trigger feature for the untargeted class. We
incorporate a regularizer to retain the original features to ensure the
backdoor attack of the infected model. In this way, we can ensure stealthiness
and obtain a larger ASR by relaxing the poisoning rate. In the third step, we
poison the entire benign support set with perturbations and obtain the hidden
poisoned support set for model training or fine-tuning and building the
backdoor in the model. In the testing step, when the query images with our
trigger are tested on the infected model, the model will output the target
label, while simultaneously maintaining high accuracy on benign samples.

In summary, our contributions are:

  * $\bullet$

We explore the existing backdoor attack methods under FSL scenarios, and they
suffer from two major shortcomings: overfitting the benign or poisoned
features and poor stealthiness for easy-to-detect.

  * $\bullet$

We propose a backdoor attack for few-shot learning to address the
aforementioned challenges, where generate a trigger with embedding deviation
to mitigate the overfitting and introduce a max-min similarity loss to
generate perturbations for enhancing stealthiness.

  * $\bullet$

Extensive experiments are conducted to verify the effectiveness of our method
across different few-shot learning methods and tasks. Our work demonstrates
that few-shot learning remains vulnerable to backdoor attacks.

## Related work

### Few-shot Learning

In a FSL task, there are usually two sets of data, including a target labeled
support set $\mathcal{S}$, an unlabeled target query set $\mathcal{Q}$, where
$\mathcal{S}$ contains different classes $C$, with $K$ images per class. In
particular, $\mathcal{S}$ and $\mathcal{Q}$ share the same label space, which
corresponds to the training and test sets in classical classification.
However, the difference from the common classification is that the number of
images for each class $K$ is small (e.g., 1 or 5). This classification task is
called a $C$-way $K$-shot task. To tackle this problem, an additional
auxiliary set $\mathcal{A}$ is usually adopted to learn transferable knowledge
to boost the learning on the target task ($\mathcal{S}$ and $\mathcal{Q}$).

The current FSL mainly focuses on three ways (Li et al. 2021): (a) Fine-tuning
based methods, (b) Meta-learning based methods, and (c) Metric-learning based
methods. For the fine-tuning based methods (Chen et al. 2019; Liu et al.
2020a; Rajasegaran et al. 2020; Dhillon et al. 2019; Tian et al. 2020; Yang,
Liu, and Xu 2021), it follows the standard transfer learning procedure (Weiss,
Khoshgoftaar, and Wang 2016), which is pre-training with the base classes at
first and then fine-tuning with the novel class. The most typical method for
it is Baseline (Chen et al. 2019) and Basline++ (Chen et al. 2019), which
adopts a linear layer in the fine-tuning stage. Both meta-learning based and
metric-learning based methods adopt simulation few-shot tasks in the training
process. The auxiliary set $\mathcal{A}$ are divided into
$\mathcal{A}_{\mathcal{S}}$ and $\mathcal{A}_{\mathcal{Q}}$ for each simulated
task (episode). Tens of thousands of simulated tasks are randomly sampled from
a distribution to train the model. The meta-learning based methods adopt a
meta-training paradigm, which aims to make the model can fast adapt to novel
class (Vinyals et al. 2016; Gordon et al. 2018; Lee et al. 2019; Bertinetto et
al. 2018; Rusu et al. 2019a; Raghu et al. 2020; Xu et al. 2020). The metric-
learning based methods directly compare the similarities of latent
representations between query set and support set without fine-tuning on the
support set, and use their relationship outputs to classify (Snell, Swersky,
and Zemel 2017; Koch et al. 2015; Sung et al. 2018; Li et al. 2019a; Doersch,
Gupta, and Zisserman 2020; Zhang et al. 2020; Wertheimer, Tang, and Hariharan
2021; Kang et al. 2021). The typical methods for them are MAML (Finn, Abbeel,
and Levine 2017) and Prototypical (Snell, Swersky, and Zemel 2017). All the
above three ways will be considered in this paper, but in the problem
statement, we will mainly focus on fine-tuned-based methods as an example.

### Backdoor Attacks

The backdoor attack is an emerging threat to model security. According to
whether the poisoned samples have consistent features and labels, the existing
backdoor attacks can be generally categorized into dirty-label attacks and
clean-label attacks. Dirty-label attacks (Gu et al. 2019; Chen et al. 2017;
Nguyen and Tran 2021) usually select a set of clean examples from the untarget
class, attach the backdoor trigger, and reset their labels to be the target
class. The poisoned inputs look like to be from the untarget class, but their
labels are the target class, thus input-label pairs look mislabeled to a
human. Clean-label attacks can be more challenging than dirty-label attacks as
the trigger pattern is no longer strongly associated with the target class.
The representative work is Label-Consistent (Turner, Tsipras, and Madry 2019).
It involves selecting data from the target class, manipulating the data harder
to learn, and inserting a trigger into the data. However, it requires a large
poison ratio so that the association between the trigger and the target label
can be memorized by the model. Another recent clean-label work is TUAP (Zhao
et al. 2020), which adopts a universal adversarial trigger and adversarial
perturbations for effective embedding backdoor attacks with videos. Recently,
Saha et. al (Saha, Subramanya, and Pirsiavash 2020) propose a Hidden Trigger
Backdoor Attack (HTBA) by adding perturbation which aims to hide the trigger
in the poisoned training set and keep the trigger secret until the test time.
Although this kind of attack has a comparable goal to ours, it can not be
employed directly to enhance stealthiness in few-shot learning scenarios.

As far as we know, most current backdoor works focus on common classification
problems, and few work has explored the existence of backdoor attacks in FSL
scenarios. Although some work (Li et al. 2022a; Guan et al. 2022) have the
keyword ’few-shot’ in the title, they are not discussing the backdoor attacks
in few-shot learning, where ’few-shot’ is just an adjective for backdoors. The
backdoor attack proposed by Chen et al. (Chen, Golubchik, and Paolieri 2020)
first conducts experiments on few-shot datasets, but it is mainly based on the
federated meta-learning scenario and does not consider the assumptions of few-
shot tasks.

## Problem Statement

### Preliminaries

Threat Model As previously mentioned, FSL is usually divided into two parts,
the first is to train on the auxiliary set to obtain a feature embedding
module, and then the victim builds on this module to retrain or fine-tune a
classifier with his own data. We assume that the attacker cannot poison the
auxiliary set, or in other words, cannot build a backdoor in the feature
embedding module. This is because auxiliary sets are usually publicly
available and clean, and some authorities also publish secure pre-trained
models for users to access. However, the backdoor attacks in FSL are more
likely to occur during the fine-tuning phase. We give two possible scenarios
for this backdoor attack: First, the trainable samples in the support set may
be poisoned by attackers. As we know, these labeled samples are often hard to
collect or acquire because of privacy or copyrights. Therefore, when a
practitioner is unable to collect enough trusted trainable data by themselves,
they will compromise to obtain risky images from untrustworthy sources, even
if they know that the data might not be secure enough. Apart from this
scenario, some data owners could detect copyrights by checking the backdoor if
a data-stealer fine-tunes the model on the protected data. For convenience, we
refer to those who create backdoors as attackers and those who suffer from
backdoor attacks as victims.

Since there are many related works about FSL and the paradigm varies a lot
from each other, we assume that the attacker has knowledge of the specific FSL
paradigm, and the victim will use and access to the pre-trained model that the
victim will train. For a brief description, we employ a classic but widely
used fine-tuning-based method Baseline++ as an example to introduce our
backdoor attack.

### Shortages of the Baseline Method

In this section, we will reveal the two shortages of the existing backdoor
attack methods in FSL scenarios:

Overfitting the Benign or Poisoned Features. As shown in the curves of Fig 1,
the existing backdoor attacks would cause an obvious trade-off, where ASR and
BA cannot achieve good performance at the same time. To further investigate
this phenomenon, we visualize the t-SNE of some clean images and their
poisoned images with triggers in the feature space, as shown in Fig 3. Our
visualization shows that the poisoned samples tend to locate close to clean
samples in feature space, and the BadNet samples (represented by the orange
circles) are the closest to the benign samples (represented by the red
triangles). Moreover, the few-shot task only contains a small number of
training samples, and the poisonable samples become even fewer when the clean
and poisoned samples are divided. Therefore, when facing a similar feature
distribution of poisoned and clean samples in pre-trained embedding space, the
model will be confused in distinguishing them with limited data. As a result,
when the poisoning rate is very low, the infected model will fail to learn the
backdoor features and only overfit the benign features, resulting in an
ineffective attack. As the poisoning rate increases, the infected model begins
to overfit the backdoor feature while failing to learn clean features,
resulting in lower performance in BA. Furthermore, clean-label methods poison
samples are only sourced from samples from the target class, thereby limiting
the number of poisonable samples even further. As a result, the model tends to
overfit clean features and struggles to learn backdoors.

Figure 3: The t-SNE visualization of benign images and different poisoned
versions in the feature spaces with four dirty-label backdoor attack methods
and ours, where the red triangles represent the distribution of clean samples.

Easy-to-detect Backdoor Attacks. Stealthiness is a critical concern in
backdoor attacks, especially in few-shot learning scenarios. Previous backdoor
attacks of dirty-label have focused on reducing the poisoning rate to ensure
the stealth of the backdoor (Gu et al. 2019). However, in FSL, where the
support set is small, even a small poisoning rate can be easily detected by
victims, as illustrated in the top left of Fig 1. Clean-label backdoor attacks
maintain consistency between images and labels, but they encounter another
problem: the trigger attached to the poisoned images is still visible to the
victim. As depicted in the visualization of TUAP in Fig 1, the image of the
target class shows an obvious trigger attached. Upon detecting some special
patterns repeatedly in the support set, victims may suspect that the data has
been maliciously poisoned. Consequently, they may attempt to repair the
poisoned images or defend against query images with the same trigger pattern
during the testing phase, thereby mitigating the backdoor attack. Hence, a
good backdoor attack for FSL should be carefully designed to remain hidden and
undetectable by the victim.

### Few-shot Learning Backdoor Attack (FLBA)

Based on these shortages, we propose Few-shot Learning Backdoor Attack (FLBA),
which can solve the corresponding challenges. It consists of four steps, and
the pipeline of our backdoor attack is shown in Fig. 2.

Trigger Generation with Embedding Deviation. Through the aforementioned
observation, it can be deduced that the features of the existing triggers are
entangled with benign features in the pre-trained embedding space, which
causes the model to either overfit the benign samples or the poisoned samples
with limited data. To achieve an effective backdoor attack for FSL, an ideal
trigger should enable the model to learn both the features of the benign
samples and the trigger features simultaneously, despite the shortage of data.
To this end, we propose to generate a trigger that maximizes the gap between
the poisoned and benign features in the pre-trained embedding. When the
poisoned features deviate sufficiently from the original clean features, the
boundary becomes low-dimensional and easy to learn for the model, as purple
circles are shown in Fig 3.

We implement our backdoor attack by first generating a trigger with embedding
deviation. Given a clean image $\mathbf{x}$ and a mask $\mathbf{m}$ that
limits the size of the pattern of trigger $\mathbf{t}$, a poisoned support set
can be formed from a benign support set by them. After passing the pre-trained
embedding network, the features of a benign sample $\mathbf{z_{b}}$ and a
poisoned sample $\mathbf{z_{p}}$ can be expressed as:

$\displaystyle\mathbf{z_{b}}$
$\displaystyle=f_{\mathbf{\theta}}\left(\mathbf{x}\right)$ (1)
$\displaystyle\mathbf{z_{p}}$
$\displaystyle=f_{\mathbf{\theta}}\left(\mathbf{x}\odot(\mathbf{1}-\mathbf{m})+\mathbf{m}\odot\mathbf{t}\right),$

where $f_{\mathbf{\theta}}(\cdot)$ is pre-trained feature embedding by
auxiliary set, and $\odot$ indicates the element-wise product. Our goal is to
optimize a trigger $\mathbf{t}^{*}$ such that the features of any images with
this trigger are different from their original benign features. Therefore, the
problem can be formulated as follows,

$\mathbf{t^{*}}=\arg\max_{\mathbf{t}}\sum_{\mathbf{x}\in\mathcal{S}}d\left(\mathbf{z}_{b},\mathbf{z}_{p}\right),$
(2)

where $\mathcal{S}$ could be the support set for training or another randomly
selected image set. $d(:,:)$ is a distance metric for two features, and the
cosine distance is employed in this work.

Hiding Trigger with Max-min Distance Loss. After addressing the first
challenge, we turn our attention to the issue of poor stealthiness. The
existing backdoor attack for both dirty-label and clean-label can not
guarantee stealthiness in FSL. Motivated by HTBA(Saha, Subramanya, and
Pirsiavash 2020), we consider generating imperceptible perturbations to poison
the support set instead of attaching the trigger pattern directly. However,
due to the predefined trigger patterns used and the large number of poisoned
samples required from untargeted classes in HTBA, it could not implement an
effective attack in FSL. Differently, we poison all images from both targeted
and untargeted classes and introduce a max-min distance loss to optimize two
types of perturbations: Attractive Perturbation for the images of the target
class, which brings their features closer to the optimized trigger features,
and Repulsive Perturbation for untargeted class, which moves their features
further away from the optimized trigger features. The resulting support set,
with these perturbations, is referred to as the hidden poisoned set. Our
approach associates the optimized trigger features with the target label while
remaining the trigger invisible during training phase. Moreover, to ensure the
BA, we also introduce a regularizer to preserve the original features.

Practically, we attach the optimized trigger $\mathbf{t^{*}}$ to clean images
to obtain the poisoned support set. As above mentioned, we split the support
set into a targeted class and an untargeted class and optimize a set of
imperceptible perturbations separately. For images of the target class, we
obtain the feature of poisoned set $\mathbf{t_{p}}$ and the feature of hidden
poisoned set $\mathbf{t_{h}}$ from the pre-trained embedding as follows:

$\displaystyle\mathbf{t_{h}}$
$\displaystyle=f_{\mathbf{\theta}}\left(\mathbf{x_{t}}+\mathbf{\delta}_{a}\right)$
(3) $\displaystyle\mathbf{t_{p}}$
$\displaystyle=f_{\mathbf{\theta}}\left(\mathbf{x_{t}}\odot(\mathbf{1}-\mathbf{m})+\mathbf{m}\odot\textbf{t}^{*}\right),$

and we optimize the attractive perturbation $\delta_{a}$ by minimizing the
distance between the features of the poisoned and hidden poisoned set.
Therefore, our optimization process can be defined as,

$\begin{array}[]{cc}\min_{\mathbf{\delta}}&d\left(\mathbf{t_{h}},\mathbf{t_{p}}\right)+\lambda_{1}d\left(\mathbf{t_{h}},f_{\theta}\left(\mathbf{x_{t}}\right)\right),\vspace{1ex}\\\
\text{ s.t.
}&\|\mathbf{\delta}_{a}\|_{\infty}\leqslant\varepsilon.\end{array}$ (4)

where $\lambda_{1}$ is balance pararmeter of regularizer and attractive
perturbation $\delta_{a}$ is restricted in $L_{\infty}$ norm bound $\epsilon$.
Moreover, we use projected gradient descent (PGD) (Madry et al. 2018) to solve
this problem. Similarly, for the untargeted class, the poisoned features
$\mathbf{u_{p}}$ and hidden poisoned features $\mathbf{u_{h}}$ can be also
obtained by the following equation:

$\displaystyle\mathbf{u_{h}}$
$\displaystyle=f_{\theta}\left(\mathbf{x_{u}}+\mathbf{\delta}_{r}\right)$ (5)
$\displaystyle\mathbf{u_{p}}$
$\displaystyle=f_{\theta}\left(\mathbf{x_{u}}\odot(\mathbf{1}-\mathbf{m})+\mathbf{m}\odot\textbf{t}^{*}\right).$

Different from the problem in (4), we generate repulsive perturbations to move
features away from the distribution of the trigger feature. Therefore, we can
optimize the repulsive perturbation by solving the following optimization
problem:

$\begin{array}[]{cc}\max_{\delta}&\quad
d\left(\mathbf{u_{h}},\mathbf{u_{p}}\right)-\lambda_{2}d\left(\mathbf{u_{h}},f_{\theta}\left(\mathbf{x_{u}}\right)\right).\vspace{1ex}\\\
\text{ s.t. }&\|\delta_{r}\|_{\infty}\leqslant\varepsilon.\end{array}$ (6)

Here, $\lambda_{2}$ controls the closeness to its original feature, and
repulsive perturbation $\mathbf{\delta_{r}}$ is restricted in $L_{\infty}$
norm bound $\epsilon$.

Method | Stealthiness | Baseline++ | MAML | ProtoNet
---|---|---|---|---
Clean Label | Invisible Trigger | ASR | BA | ASR | BA | ASR | BA
Clean | / | / | 19.2 | 71.4 | 18.1 | 65.2 | 17.3 | 69.9
BadNet | $\times$ | $\times$ | 36.2 | 64.4 | 50.4 | 53.6 | 20.6 | 49.9
Blended | $\times$ | $\surd$ | 52.3 | 62.9 | 65.6 | 54.3 | 21.1 | 50.5
Refool | $\times$ | $\surd$ | 69.8 | 65.3 | 50.4 | 54.1 | 17.1 | 51.1
WaNet | $\times$ | $\surd$ | 59.7 | 46.2 | 44.0 | 54.0 | 20.7 | 26.3
Label-Consistent | $\surd$ | $\times$ | 14.5 | 55.1 | 26.4 | 62.3 | / | /
TUAP | $\surd$ | $\times$ | 34.8 | 64.9 | 77.1 | 58.6 | / | /
HTBA | $\surd$ | $\surd$ | 26.8 | 59.3 | 21.7 | 56.5 | / | /
FLBA (Ours) | $\surd$ | $\surd$ | 89.1 | 65.5 | 81.2 | 63.6 | 60.1 | 61.6

Table 1: Comparison($\%$) of different backdoor attack methods on
miniImageNet. Stealthiness includes two aspects: clean label and invisible
trigger. In each case, the best attacking ASR and BA are boldfaced.

Building Backdoor on Support Set $\&$ Testing. Following the acquisition of
the attractive and repulsive perturbations, the subsequent phase involves
covert poisoning by introducing these perturbations into the support set. Due
to the imperceptibility of the perturbations, the poisoning rate can be
relaxed by poisoning the entire benign support set $D_{c}$, leading to a
higher ASR. The hidden poisoned support set $D_{h}$ can be mathematically
formulated as follows:

$\displaystyle
D_{h}=\\{(\mathbf{x_{t}}^{1}+\mathbf{\delta}^{1},\mathbf{y}^{1}),(\mathbf{x_{t}}^{2}+\mathbf{\delta}^{2},\mathbf{y}^{2}),\cdots,$
(7)
$\displaystyle(\mathbf{x_{u}}^{m+1}+\mathbf{\delta}^{m+1},\mathbf{y}^{m+1}),(\mathbf{x_{u}}^{m+2}+\mathbf{\delta}^{m+2},\mathbf{y}^{m+2}),\cdots\\},$

where the set of attractive perturbations
$(\mathbf{\delta}^{1},\mathbf{\delta}^{2},...,\delta^{m})$ is for targeted
class, and $(\delta^{m+1},\delta^{m+2},...,\delta^{m+n})$ is repulsive
perturbations for untargeted class. The model is then fine-tuned on the hidden
poisoned support set $D_{h}$ to establish the backdoor and obtain the infected
model.

In the final step, we evaluate our FLBA on the infected model. When testing
the BA, we input the clean image of the query set and expect a correct output
from the model. For testing ASR, the optimized trigger $t^{*}$ with mask $m$
is added to the benign image. After inputting it into the infected model, we
expect to get a target label output.

## Experiment

### Experimental Setup

Datasets and Model Architectures. Following the literature (Li et al. 2022a),
our few-shot learning experiments are mainly conducted on miniImageNet
(Vinyals et al. 2016). We split them into an auxiliary, support, and query
set, respectively, and all images are sized as a resolution of 84$\times$84\.
In this paper, we always poison the support set rather than the auxiliary set
for FSL. In addition, we choose three typical models: Baseline++ (Chen et al.
2019), MAML (Finn, Abbeel, and Levine 2017), and ProtoNet (Snell, Swersky, and
Zemel 2017). Among them, the fine-tuning-based methods are the most widely
used. Therefore, we always take Baseline++ for examples in discussion parts.
Moreover, we all adopt ResNet12 as their embedding backbones. More
experimental results on other models and datasets are presented in the
supplementary material.

Evaluation Metrics. In our work, we evaluate the performance with a set of
5-shot 5-way tasks. Same as the backdoor attacks on traditional
classification, here we also use the attack success rate (ASR) and benign
accuracy (BA) to evaluate the effectiveness of attacks. Specifically, ASR is
introduced to evaluate that the images with a specific trigger are classified
as the targeted class, and BA is the accuracy of testing on benign examples.
Moreover, we evaluate random 600 episodes of the query set, repeat the total
of five times at the test stage, and report the average ASR and BA.

Training and Attack Setup. We refer to the LibFewShot open-resource code
111https://github.com/RL-VIG/LibFewShot to build the pre-trained embedding
models and follow their settings of parameters. In the training stage, we
always record the model which obtains the best performance on the validation
set and evaluate it on the testing set in the test stage. In addition, we
assume that the attacker has access to the same pre-trained model as the
victims. For the attacks, we adopt a 16 $\times$ 16 mask as a mask of the
trigger pattern. In the trigger generation phase, we set the step size as 2,
and iterations as 100. For the attractive and repulsive perturbation
generation, we set the step size as 2, iteration as 80, and we empirically set
the $L_{\infty}$ norm bound $\epsilon$ as 8/255, which is imperceptible by
human eyes. The balance parameters $\lambda_{1}$ and $\lambda_{2}$ are
initially set as 1.5, 1.5.

Figure 4: The visualization of poisoned support set for different backdoor
attack methods. In dirty-label methods, the labels of poisoned samples are
inconsistent with their ground-truth ones. Although clean-label methods keep
the same labels as their ground-truth ones, the trigger patterns are visible
in the support set. Our method and HTBA has the good stealthiness.

### Main Results

As shown in Table 1, we test the effectiveness of existing backdoor attack
methods and ours with different few-shot learning methods, specifically
including four dirty-label methods, two clean-label methods, and a hidden
trigger method. The results demonstrate that none of the previous methods
achieve comparable results on the traditional task. In contrast, our proposed
methods exhibit promising attack results on different few-shot learning
paradigms while maintaining good performance on benign samples. For instance,
both Baseline++ and MAML methods achieve an average success rate (ASR) greater
than 80$\%$, while other backdoor attacks only achieve an average ASR of
50$\%$. Our method on ProtoNet also exceeds 60$\%$ ASR, while others exhibit
only a 20$\%$ success rate. This finding could be attributed to the fact that
ProtoNet is a metric-learning-based method that does not train a network on
the support set. Instead, it directly compares the cosine similarity or
distance between the support set and the query set for prediction.
Furthermore, clean-label backdoor attack methods, including Label-Consistent
and TUAP, involve an attack on the output of the classifier to generate
adversarial perturbations. However, metric-learning-based methods cannot apply
these backdoor attacks directly due to the lack of a classifier. In addition,
Label-Consistent, TUAP, and HTBA involve an attack on the output of the
classifier to generate adversarial perturbations. Due to the lack of a
classifier, metric-learning-based methods cannot apply directly to these
backdoor attacks. Therefore, the existing backdoor attack is hard to apply in
different few-shot learning paradigms, but ours can still successfully
construct the backdoor.

Fig. 4 shows the stealthiness of different backdoor attacks. The dirty-label
method, which involves modifying the labels in the support set, is easily
detectable as the images of wolves are incorrectly labeled as bowls. Clean-
label methods seem to solve this problem, but some abnormalities still can be
observed by victims in FSL. Specifically, in Label-Consistent and TUAP images,
the bowls are labeled correctly but visible trigger patterns are attached to
them. Due to the lack of extensive samples in the support set, victims can be
more likely to notice these patterns recurring on some samples and will make a
defense or be alert to similar patterns during the testing phase. HTBA has
good stealthiness for its hidden triggers and clean label, but it fails to
build a backdoor successfully in FSL. However, our method can achieve both
clean-label poisoning and invisibility of trigger patterns at the same time,
and successfully build a backdoor attack more stealthy in few-shot learning
scenarios.

### Discussion

#### Different Few-shot Tasks

Through the exploration of our backdoor attack in various few-shot tasks, we
have obtained experimental results of ASR and BA for 1-shot to 30-shot tasks,
which are illustrated in Fig 5. Remarkably, our method successfully implements
a backdoor even in 1-shot learning tasks, achieving an ASR of 77.1$\%$,
whereas other methods fail due to the support set’s manipulation limited to
only one image. For instance, if one image from a class is selected for label
modification and trigger attachment, BadNet is unable to learn the poisoned
and benign samples simultaneously. Although such an occurrence is unlikely in
real-world scenarios, it establishes a lower bound on the effectiveness of our
approach. Furthermore, our attack performance improves with increasing shot
numbers, eventually plateauing when more than 15 shots are being used. As the
shot number exceeds a certain level, the backdoor attack of few-shot learning
becomes that of traditional supervised learning.

Figure 5: Attack results on a different number of shot tasks.

Resistance to Fine-tuning. Fine-tuning is a popular reconstruction-based
backdoor defense method (Liu, Xie, and Srivastava 2017; Liu, Dolan-Gavitt, and
Garg 2018). In traditional classification, models are typically fine-tuned on
a small portion of the original dataset or a new dataset. Similarly, in FSL,
we introduce two support sets: the first is the original but clean support set
used to train, while the other is a new support set that is sourced from
another dataset. Figure 6 shows the results of our experiments. After ten
epochs of fine-tuning, the ASR of our backdoor attack remains above 80$\%$
when fine-tuned on the original support set, with only a marginal reduction in
the BA. On the other hand, when fine-tuning with the new support set, although
the ASR drops much at the first epoch, it still attains 77.5$\%$ ASR after ten
epochs of fine-tuning, with the BA remaining within a small range. Thus, our
backdoor attack can resist fine-tuning with the original and new support sets.

(a) Original Support Set

(b) New Support set

Figure 6: Resistance to fine-tuning on benign samples of original support set
and new support set.

## Conclusion

In this paper, we have explored the potential threat of backdoor attacks in
few-shot learning (FSL). Some results demonstrated that embedding a backdoor
in FSL is more challenging than in traditional classification tasks. Given
these challenges, we propose a novel backdoor attack method called the Few-
shot Learning Backdoor Attack. This attack involves generating a trigger with
an embedding deviation to mitigate overfitting. We also introduce optimizing a
max-min similarity loss to create attractive and repulsive perturbations that
hide the trigger. Our experimental results show that the proposed method
achieves high ASR and preserves the performance of benign samples across
different few-shot learning methods. Furthermore, our method can be used to
build backdoors on 1-shot learning tasks, which is difficult or almost
impossible with other methods, and it can be also resilient to a set of
defense strategies. There are still some limitations to our approach, our
method is based on a white-box setting, and how to efficiently build a
backdoor without model knowledge leaves further explorations. Overall, our
proposed backdoor attack provides a valuable tool for assessing the backdoor
vulnerability of FSL systems.

## Acknowledgments

Supported in part by the National Key R$\&$D Program of China (Grant No.
2022ZD0118100), in part by National Natural Science Foundation of China
(No.62025604). in part by Beijing Natural Science Foundation (No.L212004).

## References

  * Bagdasaryan and Shmatikov (2021) Bagdasaryan, E.; and Shmatikov, V. 2021.  Blind backdoors in deep learning models.  In _USENIX Security_.
  * Bertinetto et al. (2018) Bertinetto, L.; Henriques, J. F.; Torr, P. H.; and Vedaldi, A. 2018.  Meta-learning with differentiable closed-form solvers.  _ICLR_.
  * Chen, Golubchik, and Paolieri (2020) Chen, C.-L.; Golubchik, L.; and Paolieri, M. 2020.  Backdoor attacks on federated meta-learning.  _NeurIPS Workshop_.
  * Chen et al. (2019) Chen, W.-Y.; Liu, Y.-C.; Kira, Z.; Wang, Y.-C. F.; and Huang, J.-B. 2019.  A closer look at few-shot classification.  _ICLR_.
  * Chen et al. (2017) Chen, X.; Liu, C.; Li, B.; Lu, K.; and Song, D. 2017.  Targeted backdoor attacks on deep learning systems using data poisoning.  _arXiv preprint arXiv:1712.05526_.
  * Dhillon et al. (2019) Dhillon, G. S.; Chaudhari, P.; Ravichandran, A.; and Soatto, S. 2019.  A baseline for few-shot image classification.  _ICLR_.
  * Doersch, Gupta, and Zisserman (2020) Doersch, C.; Gupta, A.; and Zisserman, A. 2020.  Crosstransformers: spatially-aware few-shot transfer.  _NeurIPS_.
  * Finn, Abbeel, and Levine (2017) Finn, C.; Abbeel, P.; and Levine, S. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_.
  * Gao et al. (2023) Gao, K.; Bai, Y.; Gu, J.; Yang, Y.; and Xia, S.-T. 2023.  Backdoor Defense via Adaptively Splitting Poisoned Dataset.  In _CVPR_.
  * Gordon et al. (2018) Gordon, J.; Bronskill, J.; Bauer, M.; Nowozin, S.; and Turner, R. E. 2018.  Versa: Versatile and efficient few-shot learning.  In _NeurIPS_.
  * Gu et al. (2018) Gu, J.; Wang, Z.; Kuen, J.; Ma, L.; Shahroudy, A.; Shuai, B.; Liu, T.; Wang, X.; Wang, G.; Cai, J.; et al. 2018.  Recent advances in convolutional neural networks.  _Pattern Recognition_.
  * Gu et al. (2023) Gu, J.; Wei, F.; Torr, P.; and Hu, H. 2023.  Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks.  _BMVC_.
  * Gu et al. (2022) Gu, J.; Zhao, H.; Tresp, V.; and Torr, P. H. 2022.  SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness.  In _ECCV_.
  * Gu et al. (2019) Gu, T.; Liu, K.; Dolan-Gavitt, B.; and Garg, S. 2019.  Badnets: Evaluating backdooring attacks on deep neural networks.  _IEEE Access_.
  * Guan et al. (2022) Guan, J.; Tu, Z.; He, R.; and Tao, D. 2022.  Few-shot backdoor defense using shapley estimation.  In _CVPR_.
  * (16) Gudibande, A.; Chen, X.; Bai, Y.; Xiong, J.; and Song, D. ????  Test-time Adaptation of Residual Blocks against Poisoning and Backdoor Attacks.
  * He et al. (2023) He, B.; Liu, J.; Li, Y.; Liang, S.; Li, J.; Jia, X.; and Cao, X. 2023.  Generating transferable 3d adversarial point cloud via random perturbation factorization.  In _AAAI_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.
  * Huang et al. (2021) Huang, Y.; Guo, Q.; Juefei-Xu, F.; Ma, L.; Miao, W.; Liu, Y.; and Pu, G. 2021.  AdvFilter: predictive perturbation-aware filtering against adversarial attack via multi-domain learning.  In _ACM MM_ , 395–403.
  * Huang et al. (2023) Huang, Y.; Sun, L.; Guo, Q.; Juefei-Xu, F.; Zhu, J.; Feng, J.; Liu, Y.; and Pu, G. 2023.  ALA: Naturalness-aware Adversarial Lightness Attack.  In _ACM MM_ , 2418–2426.
  * Jia et al. (2020) Jia, X.; Wei, X.; Cao, X.; and Han, X. 2020.  Adv-watermark: A novel watermark perturbation for adversarial examples.  In _ACM MM_ , 1579–1587.
  * Kang et al. (2021) Kang, D.; Kwon, H.; Min, J.; and Cho, M. 2021.  Relational Embedding for Few-Shot Classification.  In _ICCV_.
  * Koch et al. (2015) Koch, G.; Zemel, R.; Salakhutdinov, R.; et al. 2015.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_.
  * Lee et al. (2019) Lee, K.; Maji, S.; Ravichandran, A.; and Soatto, S. 2019.  Meta-learning with differentiable convex optimization.  In _CVPR_.
  * Li et al. (2021) Li, W.; Dong, C.; Tian, P.; Qin, T.; Yang, X.; Wang, Z.; Huo, J.; Shi, Y.; Wang, L.; Gao, Y.; et al. 2021.  LibFewShot: A Comprehensive Library for Few-shot Learning.  _arXiv preprint arXiv:2109.04898_.
  * Li et al. (2019a) Li, W.; Wang, L.; Xu, J.; Huo, J.; Gao, Y.; and Luo, J. 2019a.  Revisiting local descriptor based image-to-class measure for few-shot learning.  In _CVPR_.
  * Li et al. (2019b) Li, W.; Wang, L.; Zhang, X.; Huo, J.; Gao, Y.; and Luo, J. 2019b.  Defensive Few-shot Adversarial Learning.  _arXiv preprint arXiv:1911.06968_.
  * Li et al. (2023) Li, Y.; Ya, M.; Bai, Y.; Jiang, Y.; and Xia, S.-T. 2023.  BackdoorBox: A python toolbox for backdoor learning.  _arXiv preprint arXiv:2302.01762_.
  * Li et al. (2022a) Li, Y.; Zhong, H.; Ma, X.; Jiang, Y.; and Xia, S.-T. 2022a.  Few-shot backdoor attacks on visual object tracking.  _ICLR_.
  * Li et al. (2022b) Li, Y.; Zhu, L.; Jia, X.; Jiang, Y.; Xia, S.-T.; and Cao, X. 2022b.  Defending against model stealing via verifying embedded external features.  In _AAAI_.
  * Liang et al. (2022) Liang, S.; Li, L.; Fan, Y.; Jia, X.; Li, J.; Wu, B.; and Cao, X. 2022.  A large-scale multiple-objective method for black-box attack against object detection.  In _ECCV_.
  * Liu et al. (2020a) Liu, B.; Cao, Y.; Lin, Y.; Li, Q.; Zhang, Z.; Long, M.; and Hu, H. 2020a.  Negative margin matters: Understanding margin in few-shot classification.  In _ECCV_.
  * Liu, Dolan-Gavitt, and Garg (2018) Liu, K.; Dolan-Gavitt, B.; and Garg, S. 2018.  Fine-pruning: Defending against backdooring attacks on deep neural networks.  In _Proc. of RAID_.
  * Liu et al. (2022) Liu, X.; Liu, J.; Bai, Y.; Gu, J.; Chen, T.; Jia, X.; and Cao, X. 2022.  Watermark Vaccine: Adversarial Attacks to Prevent Watermark Removal.  In _ECCV_.
  * Liu et al. (2020b) Liu, Y.; Ma, X.; Bailey, J.; and Lu, F. 2020b.  Reflection backdoor: A natural backdoor attack on deep neural networks.  In _ECCV_.
  * Liu, Xie, and Srivastava (2017) Liu, Y.; Xie, Y.; and Srivastava, A. 2017.  Neural trojans.  In _ICCD_.
  * Madry et al. (2018) Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; and Vladu, A. 2018.  Towards deep learning models resistant to adversarial attacks.  In _ICLR Poster_.
  * Nguyen and Tran (2021) Nguyen, A.; and Tran, A. 2021.  WaNet–Imperceptible Warping-based Backdoor Attack.  _ICLR_.
  * Nguyen and Tran (2020) Nguyen, T. A.; and Tran, A. 2020.  Input-aware dynamic backdoor attack.  _NeurIPS_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_.
  * Raghu et al. (2020) Raghu, A.; Raghu, M.; Bengio, S.; and Vinyals, O. 2020.  Rapid learning or feature reuse? towards understanding the effectiveness of maml.  _ICLR_.
  * Rajasegaran et al. (2020) Rajasegaran, J.; Khan, S.; Hayat, M.; Khan, F. S.; and Shah, M. 2020.  Self-supervised knowledge distillation for few-shot learning.  _arXiv preprint arXiv:2006.09785_.
  * Ren et al. (2018) Ren, M.; Triantafillou, E.; Ravi, S.; Snell, J.; Swersky, K.; Tenenbaum, J. B.; Larochelle, H.; and Zemel, R. S. 2018.  Meta-learning for semi-supervised few-shot classification.  _ICLR_.
  * Rusu et al. (2019a) Rusu, A. A.; Rao, D.; Sygnowski, J.; Vinyals, O.; Pascanu, R.; Osindero, S.; and Hadsell, R. 2019a.  Meta-learning with latent embedding optimization.  _ICLR_.
  * Rusu et al. (2019b) Rusu, A. A.; Rao, D.; Sygnowski, J.; Vinyals, O.; Pascanu, R.; Osindero, S.; and Hadsell, R. 2019b.  Meta-learning with latent embedding optimization.  _ICLR_.
  * Saha, Subramanya, and Pirsiavash (2020) Saha, A.; Subramanya, A.; and Pirsiavash, H. 2020.  Hidden trigger backdoor attacks.  In _AAAI_.
  * Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017.  Prototypical networks for few-shot learning.  _NeurIPS_.
  * Sung et al. (2018) Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P. H.; and Hospedales, T. M. 2018\.  Learning to compare: Relation network for few-shot learning.  In _CVPR_.
  * Tian et al. (2020) Tian, Y.; Wang, Y.; Krishnan, D.; Tenenbaum, J. B.; and Isola, P. 2020.  Rethinking few-shot image classification: a good embedding is all you need?  In _ECCV_.
  * Turner, Tsipras, and Madry (2019) Turner, A.; Tsipras, D.; and Madry, A. 2019.  Label-consistent backdoor attacks.  _arXiv preprint arXiv:1912.02771_.
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016.  Matching networks for one shot learning.  _NeuraIPS_.
  * Wang et al. (2019) Wang, B.; Yao, Y.; Shan, S.; Li, H.; Viswanath, B.; Zheng, H.; and Zhao, B. Y. 2019\.  Neural cleanse: Identifying and mitigating backdoor attacks in neural networks.  In _Proc. of IEEE S $\&$P_. IEEE.
  * Weiss, Khoshgoftaar, and Wang (2016) Weiss, K.; Khoshgoftaar, T. M.; and Wang, D. 2016.  A survey of transfer learning.  _Journal of Big data_.
  * Wertheimer, Tang, and Hariharan (2021) Wertheimer, D.; Tang, L.; and Hariharan, B. 2021.  Few-shot classification with feature map reconstruction networks.  In _CVPR_.
  * Xu et al. (2020) Xu, W.; Wang, H.; Tu, Z.; et al. 2020.  Attentional constellation nets for few-shot learning.  In _ICLR_.
  * Yang, Liu, and Xu (2021) Yang, S.; Liu, L.; and Xu, M. 2021.  Free lunch for few-shot learning: Distribution calibration.  _ICLR_.
  * Zhang et al. (2020) Zhang, C.; Cai, Y.; Lin, G.; and Shen, C. 2020.  Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In _CVPR_.
  * Zhao et al. (2020) Zhao, S.; Ma, X.; Zheng, X.; Bailey, J.; Chen, J.; and Jiang, Y.-G. 2020.  Clean-label backdoor attacks on video recognition models.  In _CVPR_.

## Appendix

### Description of Datasets

In this paper, we evaluate our backdoor attack on two widely-used benchmark
datasets: miniImageNet and tieredImageNet. Table 2 shows the data split for
each dataset. The miniImageNet (Vinyals et al. 2016) is a subset of ImageNet
that contains 100 categories, with 600 images for each class. The
tieredImageNet (Ren et al. 2018) is also a subset of ImageNet, with 608
classes and 776,165 images. In LibFewshot, these datasets can be split into a
training set, a validation set, and a testing set, and we follow the data
split setting in LibFewshot (Li et al. 2022a). We present the attack results
of miniImageNet in the manuscript, while the results of tieredImageNet (Ren et
al. 2018) are shown in Sec. Effectiveness on the tieredImageNet of this
supplementary material.

### Detailed Settings for Training

In the main experiments of the manuscript, we evaluate our FLBA on three
typical methods of few-shot learning: Baseline++ (Chen et al. 2019), MAML
(Finn, Abbeel, and Levine 2017), and ProtoNet (Snell, Swersky, and Zemel
2017). Here we will give their detailed settings at the training time. In the
following sections, we provide a detailed description of their settings during
the training phase. All experiments are conducted based on the open-source
code available.222https://github.com/RL-VIG/LibFewShot

### Setting for Baseline++

We adopt the Baseline++ model provided in LibFewShot and use the Stochastic
Gradient Descent (SGD) optimizer with an initial learning rate of 0.01,
momentum of 0.9, and weight decay of 0.0005 to train on the auxiliary set. We
train the model for 100 epochs with a ResNet12 backbone on a single NVIDIA
2080Ti GPU. During training, we set the way number to 5, shot number to 5, and
the query number to 15 for a 5-shot learning. We use the SGD optimizer with a
learning rate of 0.01, momentum of 0.9, and weight decay of 0.001 for 100
iterations to train the classifier. Data augmentation is applied once during
the training and testing stages.

### Setting for MAML

For the MAML model, we utilized the Adam optimizer with an initial learning
rate of 0.01 to train on the auxiliary set. We trained the model for 100
epochs with a ResNet12 backbone on a single NVIDIA 2080Ti. During the
training, we chose the way number, shot number, and query number all as 5 for
a 5-shot learning and set the episode as 2. We used the Adam optimizer with a
learning rate of 0.01 for the second phase training. Data augmentation was
employed during both the training and testing stages. The train episode was
set to 2000, and the test episode was set to 600.

### Setting for ProtoNet

For the ProtoNet model, we followed the settings in Libfewshot and used the
Adam optimizer with an initial learning rate of 0.001 and a weight decay of
0.00005 to train it. We trained the model for 100 epochs with a ResNet12
backbone on a single NVIDIA 2080Ti. During the training, we chose the way
number, shot number, and query number as 5 for a 5-shot learning. Data
augmentation was applied during both the training and testing stages. The
train episode was set to 2000, and the test episode was set to 600.

Dataset | miniImageNet | tieredImageNet
---|---|---
Num | $N_{class}$ | $N_{images}$ | $N_{class}$ | $N_{images}$
Train | 64 | 38400 | 351 | 448695
Val | 16 | 9600 | 97 | 124261
Test | 20 | 12000 | 160 | 206209
All | 100 | 60000 | 608 | 779165

Table 2: The data split on two benchmarks datasets.$N_{class}$ is the number
of classes in training (auxiliary), validation and test sets. $N_{images}$ is
the number of images in them. And the bottom row in the table shows the total
number of categories and images.

Methods |  Clean |  FLBA
---|---|---
ASR |  BA |  ASR |  BA
Fine-tuning based methods |  SKD (Rajasegaran et al. 2020) |  20.2 |  66.4 |  60.4 |  60.2
Negcos (Liu et al. 2020a) |  19.9 |  78.7 |  82.0 |  67.4
Meta-learning based methods |  R2D2 (Bertinetto et al. 2018) |  19.7 |  73.7 |  87.3 |  61.1
LEO (Rusu et al. 2019b) |  19.9 |  69.3 |  79.3 |  53.9
Metric-learning based methods |  DN4 (Li et al. 2019a) |  20.0 |  75.2 |  60.6 |  62.3
RENet (Kang et al. 2021) |  21.1 |  69.1 |  68.2 |  66.7

Table 3: Attack results of our FLBA on miniImageNet across three types of few-
shot learning paradigms with six popular methods. In each case of FLBA, the
best ASR and BA are boldfaced.

### Effectiveness on More Models

In our manuscript, we demonstrate the successful construction of backdoors in
three types of few-shot learning networks using our FLBA. To further highlight
the effectiveness of our attacks, we also evaluate six additional few-shot
learning methods that belong to latest different paradigms. These include SKD
(Rajasegaran et al. 2020) and Negcos (Liu et al. 2020a) for fine-tuning based
approaches, R2D2 (Bertinetto et al. 2018) and LEO (Rusu et al. 2019b) for
meta-learning based methods, and DN4 (Li et al. 2019a) and RENet (Kang et al.
2021) for metric-learning based approaches.

As shown in Table 3, our FLBA successfully applies to all the few-shot
learning paradigms, and the attack success rates exceed 60$\%$ while
maintaining a good BA compared to their clean results. Notably, R2D2 exhibits
the highest vulnerability to our backdoor attack, with an attack success rate
of 87.3$\%$, while SKD displays some resilience to our attack, with an ASR of
only 60.4$\%$. In conclusion, our backdoor attack demonstrates a high level of
universality across diverse few-shot learning methods.

### Effectiveness on CLIP

Recently, multimodal contrastive learning methods such as CLIP (Radford et al.
2021) have shown remarkable results as pre-trained encoders for zero-shot and
few-shot learning. In this section, we explore the effectiveness of our FLBA
on several pre-trained image encoders of CLIP. We test four different
backbones, including ResNet50, ResNet101, ViT-B/32, and ViT-B/16, and the
attack results are shown in Table 4. It is surprising to note that our FLBA
achieves significantly higher ASR and BA on CLIP encoders compared to the
embedding pre-trained on miniImageNet, as reported in the manuscript.
Specifically, the ASR for ViT-B/32 can almost reach 100$\%$, while maintaining
a high BA of 95.4. This observation can be attributed to the fact that CLIP
obtains an excellent encoder on a larger dataset, which shows a good
distribution of features of different class. Thus, the optimized triggers of
FLBA based on this encoder can effectively move away from the normal
distribution, leading to an effective backdoor attack.

Backbone |  ASR |  BA
---|---|---
RN50 |  94.4 |  86.6
RN101 |  81.8 |  91.1
ViT-B/32 |  99.9 |  95.4
ViT-B/16 |  99.8 |  95.9

Table 4: Results of our FLBA on miniImageNet with four image encoder backbones
of CLIP. In each case, the best ASR and BA are boldfaced.

### Effectiveness on the tieredImageNet

In the manuscript, we have conducted a series of experiments utilizing the
miniImageNet dataset. To ensure the robustness and generalizability of our
approach, we have evaluated its effect on a different few-shot learning
dataset, namely, tieredImageNet (Ren et al. 2018), which has been introduced
in Sec. Description of Datasets of the Supplementary Material. As depicted in
Table 5, prior backdoor attack methods exhibited inferior performance on the
tieredImageNet dataset compared to our proposed method, the FLBA, which
achieved an impressive Attack Success Rate (ASR) of 99.2. This is consistent
with the performance on miniImageNet in the manuscript. However, it is
noteworthy that our FLBA suffered a little decrease in Backdoor Accuracy (BA)
compared to the results on miniImageNet. Despite it, our approach still
outperforms existing backdoor attack methods and exhibits dataset-independent
effectiveness.

Method |  ASR |  BA
---|---|---
Clean |  19.5 |  75.1
BadNet |  55.9 |  56.9
Blended |  65.0 |  56.6
Refool |  75.6 |  64.8
WaNet |  71.2 |  48.7
LabelConsistent |  15.8 |  42.7
TUAP |  21.2 |  64.7
Ours |  99.2 |  66.3

Table 5: Comparison($\%$) of different backdoor attack methods on
tieredImageNet. In each case, the best attacking ASR and BA are boldfaced.

### Transferability

The present study assumes that the attacker has access to the pre-trained
model of the victim in the training support set. However, in some scenarios,
the attacker may lack information about the victim and cannot obtain the same
pre-trained embedding. In this section, we investigate the transferability of
different pre-trained embeddings. Specifically, we examine three pre-trained
embeddings: Conv64F, ResNet12, and ResNet18. We test the ASR on the target
embedding with the poisoned dataset generated by other source embeddings.
Since the poisoned dataset is optimized from only one source embedding, we
call it ’Individual’, and their results are illustrated in the ’Individual’
bar of Fig. 7. Our findings indicate that the poisoned dataset from the source
embedding does not transfer well to other embeddings, resulting in an average
success rate (ASR) of approximately 20$\%$.

Recent research has proposed ensemble perturbation to enhance the
transferability in black-box scenarios (Liu et al. 2022). Therefore, we
explore the feasibility of merging perturbations from several different source
embeddings. We observe that the ensemble poisoned dataset performs better
transferability than the individual poisoned dataset, with an ASR of over 40.
However, the limited transferability across different embeddings remains a
significant challenge for our approach. Therefore, it is worthwhile to
investigate strategies to improve the transferability of our approach.

Figure 7: The transferability results of our FLBA. An individual poisoned
dataset generated by another source embedding can not build a backdoor
successfully on target embeddings, while the ensemble poisoned dataset
generated by several embeddings can improve the ASR significantly.

Poisoning Type |  ASR |  BA
---|---|---
Target Only |  54.0 |  67.1
Untargeted Only |  73.8 |  69.4
Target + Untargeted |  89.1 |  65.5

Table 6: Ablation study for three types of poisoning support set.

(a) Target Class

(b) untargeted Class

Figure 8: Attack results with different amount of poisoning data.

### Ablation Study

In this section, we present additional ablation studies aimed at examining the
impact of poisoning size on the performance of our attack. In contrast to the
primary results, which involved poisoning the entire support set, we
partitioned poisoning into three categories: poisoning the target class,
poisoning the untargeted class, and poisoning all classes. As presented in
Table 7, it is evident that poisoning all classes yielded the highest ASR,
while poisoning the untargeted class resulted in the best BA. Consequently, to
achieve better attack results, we generally opted to poison the entire dataset
as a compromise.

Furthermore, we investigated different amounts of poisoned data in the target
and untargeted class, and the results are displayed in Fig 8. The findings
revealed that as the number of poisoned images increased, the ASR for both
target and untargeted classes also increased, while the BA remained almost
unchanged. Therefore, it appears to be a judicious choice for our attack to
forfeit the poisoning rate and poison the entire dataset to maximize the ASR.

### Sensitivity to Parameters

In the training process, there are a few hyper-parameters for our FLBA. In
this section, we will investigate the sensitivity of FLBA to two hyper-
parameters, namely trigger patterns and the size of triggers. Other parameters
are the same as those in the main experiments of the manuscript.

### Balance Parameters

In the experiments of the manuscript, we have two balance parameters:
$\lambda_{1}$ and $\lambda_{2}$, where $\lambda_{1}$ controls the deviation of
original features for attractive perturbations, and $\lambda_{2}$ controls the
deviation of original features for repulsive perturbations. In most
experiments, we set both parameters to 1.5. In this section, we explore the
sensitivity of FLBA to these balance parameters.

Table 7 presents the results of our experiments. We observed that decreasing
the balance parameters of both attraction and repulsion perturbations
consistently leads to an increase in ASR, whereas ASR decreases and BA
increases when these parameters gradually increase. This observation is
expected because the balance parameter determines the extent to which
perturbed features deviate from their original features. Larger balance
parameters result in a higher degree of consideration given to the original
feature distribution, which in turn leads to a higher BA. Additionally, we
noticed that the change in ASR is more pronounced than that of BA. Therefore,
we finally choose a not high value of BA to achieve a higher ASR.

Fixed $\lambda_{1}$ = 2 | ASR | BA | Fixed $\lambda_{2}$ = 1.5 | ASR | BA
---|---|---|---|---|---
$\lambda_{2}$ = 2.5 | 82.1 | 67.1 |  $\lambda_{1}$ = 2.5 | 84.8 | 65.6
$\lambda_{2}$ =2.0 | 85.9 | 66.2 |  $\lambda_{1}$ =2.0 | 89.1 | 65.5
$\lambda_{2}$ =1.5 | 89.1 | 65.5 |  $\lambda_{1}$ =1.5 | 92.7 | 64.2
$\lambda_{2}$ =1.2 | 91.2 | 64.1 |  $\lambda_{1}$ =1.2 | 93.4 | 63.8
$\lambda_{2}$ =1.0 | 92.8 | 63.1 |  $\lambda_{1}$ =1.0 | 94.4 | 63.1

Table 7: The attack results with different balance parameters. In the right
column, we fix $\lambda_{1}$ and change only $\lambda_{2}$, while in the left
column, we fix $\lambda_{2}$ but test a series of different $\lambda_{1}$.

(a) Attack Success Rate

(b) Benign Accuracy

Figure 9: Attack results with different trigger size.

### Trigger Size

In the experiments of the manuscript, we choose the trigger size as 16
$\times$ 16\. To explore the effect of trigger size on our FLBA, we conducted
a series of experiments with different trigger sizes ranging from 6 $\times$ 6
to 26 $\times$ 26, while keeping the image size fixed at 84 $\times$ 84\. We
selected the Baseline model++ on miniImageNet for testing purposes, and BadNet
was also used for comparison. The results are presented in Fig 9.

As we konw, a larger trigger size makes the trigger more visible to the human
eye. For the attack success rate, both FLBA and BadNet’s success rates
increased as the trigger size increased, although FLBA’s increase was more
significant, ranging from 36.6$\%$ at trigger size 6 to 99.1$\%$ at trigger
size 26, whereas BadNet’s increase appeared to be minimal, only from 33.3$\%$
to 37.8$\%$. Thus, we believe that FLBA has greater potential for improving
the attack effectiveness when the victim can tolerate a larger trigger
appearing in the query image. In contrast, BadNet’s attack effectiveness did
not improve significantly even when it added a large trigger to the dataset.
Regarding the benign accuracy, we observed that neither FLBA nor BadNet’s
performance was significantly affected by the trigger size. Thus, we conclude
that the size of the trigger does not have a substantial impact on the BA of
the infected model.

Brightness | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---
BA | 64.2 | 63.9 | 63.9 | 63.7 | 63.5 | 63.4
ASR | 92.7 | 92.7 | 92.7 | 92.6 | 92.6 | 92.6

(a) Brightness

Contrast | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---
BA | 64.2 | 63.9 | 64.1 | 63.9 | 64.1 | 63.8
ASR | 92.7 | 92.7 | 92.7 | 92.6 | 92.6 | 92.5

(b) Contrast

Saturation | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---
BA | 64.2 | 63.8 | 63.7 | 63.5 | 63.2 | 63.1
ASR | 92.7 | 92.6 | 92.4 | 92.1 | 91.9 | 91.6

(c) Saturation

Brightness | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---
BA | 64.2 | 63.9 | 63.8 | 63.6 | 63.5 | 63.4
ASR | 92.7 | 92.7 | 92.7 | 92.6 | 92.6 | 92.6

(d) Hue

Table 8: Resistance to four image pre-processing techniques with different
budgets. Figure 10:  The visualization of poisoned support set for different
backdoor attack.

### Resistance to the Image Pre-processing

Data augmentation is a widely used technique for increasing model robustness
against backdoor attacks. In this study, we employed the same four pre-
processing methods used in FSBA (Li et al. 2022a), namely brightness,
contrast, saturation, and hue, to evaluate the effectiveness of FLBA against
defenses. The results are presented in Table 8. Our experiments showed that
the four pre-processing methods had almost no effect on the backdoor attacks
launched by FLBA, with the attack success rate (ASR) dropping by only about
1.0$\%$ as the adjustment budget reached 0.5. Therefore, our FLBA demonstrated
successful resistance to image pre-processing defenses.

### Resistance to Neural Cleanse.

Neural Cleanse (Wang et al. 2019) is a widely-used backdoor model mitigation
method that is based on pattern optimization. The classifier is predicted to
be backdoored if the Anomaly Index is larger than 2. Here, we use Neural
Cleanse to detect the backdoor in an FSL classifier and adopt a clean query
set in our evaluation. Table 9 shows the Anomaly Index produced by Neural
Cleanse for different backdoor attacks. We find that the Anomaly Indices are
only 0.67 for ours, while other methods are all larger than 1 and BadNet even
reaches 4.06. Thereby, our proposed attack is resistant to Neural Cleanse.

Backdoor Attack Method | BadNet | Blended | Refool | WaNet | Ours
---|---|---|---|---|---
Anomaly Index | 4.64 | 3.04 | 2.25 | 1.04 | 0.67

Table 9: Anomaly Indices for the few-shot learning classifier produced by
Neural Cleanse with different backdoor attacks.

### More Visualization of Different Backdoor Attacks

In the main manuscript, we show some visualizations of the poisoned dataset
regarding different types of backdoor attacks. In the supplementary material,
we show more poisoned images of different backdoor attacks to better identify
their stealthiness, as shown in Fig 10. As in the manuscript, we can find that
some backdoor attacks present a very obvious trigger, which is easily
detectable in few-shot learning. Even though some methods such as WaNet have
imperceptible triggers, they often use the dirty label method, thus its
anomalies are also easily detected by the victims. However, our method (FLBA)
performs better stealthiness in backdoor attacks for FSL.


# Breeze-7B Technical Report

MediaTek Research Please cite this paper as “MediaTek-Research (2024)”.
Authorship contribution details are provided at the end of this document.

(February 2024)

###### Abstract

Breeze-7B is an open-source language model based on Mistral-7B, designed to
address the need for improved language comprehension and chatbot-oriented
capabilities in Traditional Chinese. This technical report provides an
overview of the additional pretraining, finetuning, and evaluation stages for
the Breeze-7B model. The Breeze-7B family of base and chat models exhibits
good performance on language comprehension and chatbot-oriented tasks,
reaching the top in several benchmarks among models comparable in its
complexity class.

##  1 Introduction

This technical report presents Breeze-7B, the latest addition to MediaTek
Research’s suite of language models. Breeze-7B represents a step forward in
the development of large language models (LLMs) specifically tailored for
Traditional Chinese language tasks. In this report, we detail the
methodologies from data curation and preparation, architectural adaptations,
model training, and evaluation results of Breeze-7B.

LLMs have perform tremendously well in language understanding task including
reasoning and text summarization, resulting in them used commercially in
question-and-answer platforms and customer services. Some of the most
performant models are closed-source in nature, e.g. GPT-4 (OpenAI, 2023),
Gemini-1.5 (Gemini-Team, 2024), Claude. On the other hand, there are open
sourced models with suitable licensing terms including Mistral (Jiang et al.,
2023), LLaMA-2 (Touvron et al., 2023), Falcon (Almazrouei et al., 2023), and
BLOOM (Scao et al., 2022). Despite the great amount of effort in the field,
typically these LLMs are optimized for English usage. Even though they are
compatible with other languages, LLM behavior in non-English situations can
degrade a lot. For a trivial example, the compression ratio achieved by the
tokenizer can suffer (Sennrich et al., 2016). There is an obvious need to have
open source LLMs with improved Traditional Chinese capabilities.

Building upon the impressive foundation laid by the Mistral-7B model,
Breeze-7B was developed with addressing the specific challenges associated
with processing and understanding Traditional Chinese text in mind. The scale
of additional pretraining and finetuning is order of magnitude greater than
our past attempts (Ennen et al., 2023; Hsu et al., 2023). On the set of
benchmarks that we tested, Breeze-7B performs well in Traditional Chinese
tasks compared with modern models in the similar complexity class. It also
maintains all the remarkable capabilities inherited from Mistral-7B.

In order to enable the community to build on this work and contribute to the
development of Traditional Chinese LLMs, we are open-sourcing the model
weights of Breeze-7B-Base and Breeze-7B-Instruct. The former is the self-
supervised model additionally pretrained on Traditional Chinese corpus. The
latter further finetunes on the base model to enhance instruction following
and chatting abilities.

##  2 Method

###  2.1 Model architecture customization

To increase the compression rate of the Traditional Chinese corpus, we
employed a modified tokenizer with additional tokens, which is a strategy also
present in domain adapted LLMs (Liu et al., 2023). This change has the extra
benefit of speeding up both Traditional Chinese training and inference. We use
the BPE algorithm (Sennrich et al., 2016) on a Chinese corpus to generate
tokens to appended to the original tokenizer. After de-duplicating identical
tokens already present in Mistral tokenizer, we extended 29873 tokens to it to
arrive at 61872 total tokens. The input embedding layer and output embedding
layer shapes are adjusted accordingly to accommodate the embeddings required
for the new tokens. With the extended vocabulary, the compression rate on the
Chinese corpus is around 2x compared to that of the Mistral model. As a
result, our model enjoys a 11.1k context length on Traditional Chinese text,
equivalent to around 10 pages of text, and we also observed a 2x speed up in
the model training and inference speed. In addition, our long model further
quadruples the context length.

###  2.2 Training

We trained the model on 650 gigabytes of data using a total of 7,000 H100
hours to arrive at Breeze-7B-Base-v1_0. Most of the training was conducted
with tensor parallel and data parallel training using the Megatron-LLM library
(Cano et al., 2023). We use a flat learning rate of 1e-6 and a batch size of 4
million tokens. The Breeze-7B-Base-v0_1 is a earlier checkpoint of this run.

The quality of the data is of great importance of the model training, and
recent works correlate data quality to textbook-like data (Li et al., 2023;
Gunasekar et al., 2023). We found that a good validation dataset is quite
important for in-training evaluation, that is, for identifying problematic
checkpoints in time during the pretraining process. In order to measure
“goodness” of the intermediate checkpoints efficiently, we chose perplexity as
a measure to align with the training objective of the model, and we selected
unseen textbook-like data to highlight the potential knowledge storage in the
model. Figure 1 shows the perplexity changes over the training steps.

Figure 1: Perplexity (PPL) change during the additional pretraining stage of
Breeze-7B, after the vocabulary size extension. The PPL scores are calculated
using our proprietary Traditional Chinese validation dataset.

###  2.3 Long Context Pretraining

The extension from short-context to long-context models is an important field
of research, as this two-step approach has significant efficiency advantages
over direct long-context training without performance degradation (Xiong et
al., 2023). For this setting, different scaling methods exist (Chen et al.,
2023; bloc, 97; Peng et al., 2023) to either induce a better inductive bias
from the short-range trained positions or obtain better extrapolation
properties (Liu et al., 2024). For our training, we incorporate a base-
changing method, driven by the effectiveness and conciseness of the method on
other open-sourced models (Jiang et al., 2023; AI et al., 2024). We trained
the model on an additional 25 gigabytes of data consisting of Chinese, English
and code data, and we upsampled long data for better long-range attention
modeling. We turned off the sliding window function and set the context length
to 32k, and keep all hyperparameters same as section 2.2

###  2.4 Instruction Finetuning

To enable the Breeze-7B to complete question answering in a human-preferred
instructional or chatting format, we fine-tuned the model with additional
paired data. All data that we selected for finetuning are derived from
publicly available instruction datasets, most of which are hosted on
HuggingFace. For some datasets, we use our in-house filtering and
transformation methods, such as deduplication and harmfulness prompt
filtering, to create variants that better suit our needs. We fine-tuned the
model for multiple epochs with a cosine annealing learning rate (Loshchilov
and Hutter, 2016). We name the resulting model as Breeze-7B-Instruct, with the
version tag following its base model.

##  3 Benchmarks

In this section, we give a quick overview on the benchmarks we use. We
considered two important aspect of language models: the ability to understand
languages and solve tasks, and the ability to chat with humans.

###  3.1 Language comprehension benchmarks

We chose on TMMLU+ (Tam and Pai, 2023), DRCD (Shao et al., 2018) and Penguins-
in-a-table-TC of TC-Eval to evaluate the models. TMMLU+ is a benchmark derived
from TMMLU (Hsu et al., 2023), curated from examinations in Taiwan. It
consists of 67 subjects spanning multiple disciplines, from vocational to
academic fields, and covers elementary to professional proficiency levels. It
is further categorized into 4 subcategories similar to MMLU: STEM, humanities,
social sciences and others. DRCD is an extractive question answering benchmark
based on Wikipedia. Penguins-in-a-table-TC is a multiple choice benchmark
focusing on table understanding abilities, including table operations such as
insertion and sorting.

###  3.2 Chatbot-oriented benchmarks

We chose MT-Bench-tw and MT-Bench (Zheng et al., 2024) to reflect chatting
abilities of the language models. MT-Bench-tw is multi-turn dialogue benchmark
translated from MT-bench, with slight adjustments in some terms to reflect the
language specific task instructions that do not translate well to Traditional
Chinese, for example, ”Start each sentence with the letter A”. The multi-turn
benchmark consists of two turns. In the first turn, versatile questions
covering multiple topics of interest are used to test the chatting abilities
of the models. In the second turn, further instructions would be given to test
the models’ abilities to follow verbose instructions and attend to inputs and
outputs of the previous round. Due to its free-form answering nature, we
follow the convention of scoring with GPT-4 on a scale from 1 to 10. We
provide the exact prompt for scoring in Appendix A.

###  3.3 Long-context benchmarks

To understand the models’ ability to attend to long context, we collected
benchmarks that have inputs spanning over 10k tokens. At this scale, the
benchmarks serve as a precursor to the performance on document-level tasks, as
well as the commercially popular RAG (Retrieval Augmented Generation) use
case. We first benchmark the model on the passkey retrieval task following
previous work, and then on DRCD-long that is synthetically created from the
Chinese benchmark DRCD. Passkey retrieval is a synthetic task to test whether
a model could retrieve a 5-letter key within a long paragraph of irrelevant
text. In addition to the variation of total sequence length, we additionally
control the key position following observations that the depth of the key can
significantly affect performance. DRCD-long extends the DRCD context to up to
64k Chinese characters with irrelevant articles, simulating inaccuracies in an
RAG system. The task objective and scoring criterion remain the same as the
original DRCD. We leave the curation details of this benchmark in Appendix B.

##  4 Results

In this section, we reported the evaluated results of our model, and
contrasted our results with those of several notable contemporary models.
Since some of the models have multiple variants, we select model variants that
maximize performance. For the language comprehension benchmarks, we use the
instruct variants whenever possible; otherwise, we resort to the chat
variants. We only use the Chat variants for the chatbot-oriented benchmarks.
Additionally, to understand the effectiveness of the pretraining of different
models, we also evaluate the language comprehension benchmarks on the base
model. To increase robustness, we used a few-shot template to demonstrate the
task format.

###  4.1 Models evaluated

We evaluate the following models over the language comprehension benchmarks.
We chose the models based on two main factors: 1. Bilingual (Chinese/English)
Abilities 2. Similar size to Breeze-7B

Base Model | Instruct/Chat Model
---|---
Breeze-7B-Base | Breeze-7B-Instruct
Yi-6B | Yi-6B-Chat
Qwen1.5-7B | Qwen1.5-7B-Chat
(unreleased at the moment of the writing of this paper) | Taiwan-LLM-13B-v2.0-Chat
(unreleased at the moment of the writing of this paper) | Taiwan-LLM-7B-v2.1-Chat
Table 1: Evaluate Base and Chat/Instruct Models

Also, we listed the benchmark scores of GPT-3.5-Turbo (1106), which represents
one of the most widely used high-quality cloud language model API services,
for reference.

###  4.2 Results

####  4.2.1 Traditional Chinese Language Comprehension benchmarks

Table 2 and Table 3 summarizes the comprehension benchmark results on the chat
model and the base model respectively. Since the base model may not reliably
identify the format of multiple-choice questions, we have employed a 5-shot
setting to enhance the stability and fairness of the comparison, using five
examples from the development set as the shots. However, due to context
limitations, the DRCD was reduced to a 3-shot setting.

From the results, we can draw three conclusions. First, the Breeze-7B-Instruct
model performs well on TMMLU+ compared to other models of the same size, while
outperforming other models in the Penguins-in-a-table-TC dataset. Second,
Breeze-7B-Instruct demonstrates superior or comparable performance to the
Breeze-7B-Base model, indicating that our finetuning procedure effectively
retains the knowledge in the base model. Finally, when comparing the
performance of Breeze and Mistral on the MMLU benchmark, we conclude that our
training procedure, along with adjustments to the model architecture, only
results in minimal degradation of English performance.

Model | TMMLU+(ACC) | Table | MMLU
---|---|---|---
STEM | Social Science | Humanities | Other |  $\uparrow$ AVG | (ACC) | (ACC)
Yi-6B-Chat | 37.80 | 51.74 | 45.36 | 44.25 | 44.79 | 25.69 | 59.45
GPT-3.5-Turbo | 41.58 | 48.52 | 40.96 | 43.18 | 43.56 | 45.14 | 67.09
Breeze-7B-Instruct-v1_0 | 36.46 | 48.37 | 45.11 | 40.75 | 42.67 | 39.58 | 59.12
Qwen1.5-7B-Chat | 41.48 | 51.66 | 44.05 | 45.40 | 45.64 | 34.72 | 59.54
Taiwan-LLM-13B-v2.0-chat | 27.74 | 33.69 | 27.03 | 29.43 | 29.47 | 23.61 | 50.50
Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 | 28.08 | 31.25 | 42.72
Table 2: Benchmark results of finetuned models on TMMLU+ (0 shot), Table (0 shot), and MMLU (0 shot) Model | TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MMLU (ACC)
---|---|---|---|---
Yi-6B | 49.63 | 76.61 | 34.72 | 65.35
Qwen1.5-7B | 46.28 | 74.41 | 30.56 | 60.53
Breeze-7B-Base-v1_0 | 40.72 | 80.61 | 31.99 | 58.65
Mistral-7B-v0.1 | 36.93 | 79.27 | 27.78 | 64.89
Table 3: Benchmark results of pretrained models on TMMLU+ (5 shot), DRCD (3
shot), Table (5 shot), and MMLU (5 shot).

####  4.2.2 Traditional Chinese Chatbot-oriented benchmarks

Table 4 summarizes the benchmark scores for chatbot-oriented models. Overall,
Breeze-7B-Instruct performs well on both the Chinese and English versions of
the benchmark, with GPT-4 serving as the judge. It trails only behind
Qwen1.5-7B. We present the subcategory scores for the 10 total categories in
Appendix A

Models | MT-Bench-tw (Score) | MT-Bench (Score)
---|---|---
GPT-3.5-Turbo | 7.1 | 7.9
Breeze-7B-Instruct-v1_0 | 6.0 | 7.4
Qwen1.5-7B-Chat | 6.4 | 7.6
Yi-6B-Chat | 5.0 | 6.0
Taiwan-LLM-13B-v2.0-chat | 5.0 | N/A*
Taiwan-LLM-7B-v2.1-chat | 4.2 | N/A*
Table 4: Benchmark results of finetuned models on MT-Bench score. Results
marked in asterisk should not be interpreted as the models true capabilities
as the model often answered in Chinese in this English benchmark, causing
scoring difficulties.

####  4.2.3 Long Context Benchmarks

We compare Breeze-7B-32k-Base with Breeze-7B-Base to show the effect after
long context pre-training. As our focus of this paper is on enabling the long-
context abilities of Breeze-7B-Base, we leave comparisons to other models as
future work.

For the passkey retrieval task, we evenly split the key positions into 16 bins
and tested 20 examples per bin. Accuracy is reported for each length-depth
combination, and we visualize the results in Figure 2. The non-extended model
can only consistently retrieve keys where they are located within a 4k token
distance from the question, which is positioned at the end of the sequence.
This phenomenon can be attributed to the sliding window size of non-extended
model (4096), which limits the direct attention span, that is, information
beyond this would have to rely on ”information forward passing”(Jiang et al.,
2023) to be attended to. As expected, after long-context training,
Breeze-7B-32k-Base performs consistently on the task up to 32k. An analysis of
the incorrect examples reveals that almost all errors are copying errors, such
as mistaking lowercase letters as uppercase, which indicates that the model
has near-perfect performance in attending to the correct position.

Figure 2: Passkey Retrieval results of Breeze-7B-Base and Breeze-7B-32k-Base.
The y-axis denotes the input sequence length, while the x-axis denotes the
depth of the key position in the example. Each length-depth combination is
trialed 20 times and the accuracy is color-coded with the colormap at the
bottom.

For Long-DRCD, We observed in Table 5 that Breeze-7B-32k-Base has a large
boost in performance in 3-shot DRCD compared to Breeze-7B-Base, and the model
tested on DRCD-32k retains 77% of the DRCD performance. The ability to attend
to text very far away benefits twofold: finding information that occurs in the
front of the context and understanding the extractive setting of the task
demonstrated by few-shot examples in the very front. We also include DRCD-64k
which exceeds our training context length. At this length, the degradation is
observable for both models, showing that our training method does not
extrapolate well to longer sequences.

Model/Perforamce(EM) | DRCD | DRCD-16k | DRCD-32k | DRCD-64k
---|---|---|---|---
Breeze-7B-32k-Base-v1_0 | 79.73 | 69.68 | 61.55 | 25.82
Breeze-7B-Base-v1_0 | 80.61 | 21.79 | 15.29 | 12.63
Table 5: Benchmark results of pretrained models on DRCD with different context
length (3-shot).

##  5 Conclusion

In this study, we have introduced Breeze-7B, a Traditional Chinese enhanced
language model based on Mistral-7B. These base and instruct variants of the
model have shown great performance on language comprehension and chatting
abilities, closing the gap between open sourced models and proprietary models
such as GPT-3.5. Breeze-7B stands as a testament to MediaTek Research’s
commitment to the realm of artificial intelligence and natural language
processing, and we plan to keep pushing the boundaries of what is possible
with language models.

## Appendix

##  Appendix A Chat Benchmark Details

Our GPT-4 judge incorporates the following prompt for MT-Bench-tw, where words
in curly brackets will be replaced with real questions in the benchmark and
answers of the language model.  $[$Instruction$]$
Please act as an impartial judge and evaluate the quality of the response
provided by an AI assistant to the user question displayed below. Your
evaluation should consider factors such as the helpfulness, relevance,
accuracy, depth, creativity, and level of detail of the response. Responses in
Traditional Chinese are expected and are more favourable than Simplified
Chinese, English, and other languages. Begin your evaluation by providing a
short explanation. Be as objective as possible. After providing your
explanation, you must rate the response on a scale of 1 to 10 by strictly
following this format: [[rating]], for example: Rating: [[5]].

$[$Question$]$

{question}

$[$The Start of Assistant’s Answer$]$

{answer}

$[$The End of Assistant’s Answer$]$ Table 6 shows scores of the finetuned
model on all 8 subcategories of the MT-Bench-tw benchmark. Breeze-7B-Instruct
performs competitively with other models overall in the 7B size range, but it
trails behind other models to greater extent on STEM and Writing subtasks.
Enhancements in these areas is an exciting direction of future work.

Model | STEM | Extraction | Reasoning | Math | Coding | Roleplay | Writing | Humanities |  $\uparrow$ AVG
---|---|---|---|---|---|---|---|---|---
GPT-3.5-Turbo | 7.8 | 6.1 | 5.1 | 6.4 | 6.2 | 8.7 | 7.4 | 9.3 | 7.1
Breeze-7B-Instruct-v1_0 | 7.8 | 5.2 | 4.2 | 4.2 | 4.1 | 7.6 | 5.9 | 9.1 | 6.0
Qwen1.5-7B-Chat | 9.0 | 5.6 | 4.7 | 2.8 | 3.7 | 8.0 | 8.0 | 9.4 | 6.4
Yi-6B-Chat | 7.3 | 2.7 | 3.1 | 3.3 | 2.3 | 7.2 | 5.2 | 8.8 | 5.0
Taiwan-LLM-13B-v2.0-chat | 6.1 | 3.4 | 4.1 | 2.3 | 3.1 | 7.4 | 6.6 | 6.8 | 5.0
Taiwan-LLM-7B-v2.1-chat | 5.2 | 2.6 | 2.3 | 1.2 | 3.4 | 6.6 | 5.7 | 6.8 | 4.2
Table 6: Verbose benchmark results of finetuned models on MT-Bench-tw

##  Appendix B Long Context Benchmark Details

As seen in the passkey retrieval results, the position of the relevant
information can greatly influence the performance. To ensure that the correct
context span is even distributed with various depth across samples, our
curation method of the DRCD-long tasks are as follow: For questions
${a,b,c,d,e...}$ that require context ${A,B,C,D,E...}$, we concatenate the
contexts until they reach the target length to arrive at ${ABCDE,FGHIJ,...}$
as the contexts. The questions are automatically paired with the extended
context. We have released DRCD-long at our huggingface page

## References

  * AI et al. [2024] 01\. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.  Yi: Open foundation models by 01.ai, 2024.
  * Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The Falcon series of open language models, 2023.
  * bloc [97] bloc97.  Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.  URL https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/.
  * Cano et al. [2023] Alejandro Hernández Cano, Matteo Pagliardini, Andreas Köpf, Kyle Matoba, Amirkeivan Mohtashami, Xingyao Wang, Olivia Simin Fan, Axel Marmet, Deniz Bayazit, Igor Krawczuk, Zeming Chen, Francesco Salvi, Antoine Bosselut, and Martin Jaggi.  epfLLM Megatron-LLM, 2023.  URL https://github.com/epfLLM/Megatron-LLM.
  * Chen et al. [2023] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.  Extending context window of large language models via positional interpolation.  _arXiv preprint arXiv:2306.15595_ , 2023.
  * Ennen et al. [2023] Philipp Ennen, Po-Chun Hsu, Chan-Jan Hsu, Chang-Le Liu, Yen-Chen Wu, Yin-Hsiang Liao, Chin-Tung Lin, Da-Shan Shiu, and Wei-Yun Ma.  Extending the pre-training of bloom for improved support of traditional chinese: Models, methods and results, 2023.
  * Gemini-Team [2024] Google Gemini-Team.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  2024\.
  * Gunasekar et al. [2023] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.  Textbooks are all you need, June 2023.  URL https://www.microsoft.com/en-us/research/publication/textbooks-are-all-you-need/.
  * Hsu et al. [2023] Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da shan Shiu.  Advancing the evaluation of traditional chinese language models: Towards a comprehensive benchmark suite, 2023.
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7B, 2023.
  * Li et al. [2023] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report.  September 2023.  URL https://www.microsoft.com/en-us/research/publication/textbooks-are-all-you-need-ii-phi-1-5-technical-report/.
  * Liu et al. [2023] Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al.  Chipnemo: Domain-adapted llms for chip design.  _arXiv preprint arXiv:2311.00176_ , 2023.
  * Liu et al. [2024] Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin.  Scaling laws of rope-based extrapolation, 2024.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Peng et al. [2023] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.  Yarn: Efficient context window extension of large language models.  _arXiv preprint arXiv:2309.00071_ , 2023.
  * Scao et al. [2022] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_ , 2022.
  * Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In Katrin Erk and Noah A. Smith, editors, _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1162.  URL https://aclanthology.org/P16-1162.
  * Shao et al. [2018] Chih-Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai.  Drcd: a chinese machine reading comprehension dataset.  _ArXiv_ , abs/1806.00920, 2018.  URL https://api.semanticscholar.org/CorpusID:46932369.
  * Tam and Pai [2023] Zhi-Rui Tam and Ya-Ting Pai.  An improved traditional chinese evaluation suite for foundation model.  _arXiv_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Xiong et al. [2023] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al.  Effective long-context scaling of foundation models.  _arXiv preprint arXiv:2309.16039_ , 2023.
  * Zheng et al. [2024] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36, 2024.

## Authorship and Acknowledgements

This work is a collaborative effort by the following members: (in alphabetical
order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu
許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.

We thank Nvidia-Taiwan for providing computational and storage resources.