#  BEEAR: Embedding-based Adversarial Removal of Safety Backdoors
in Instruction-tuned Language Models
This paper contains model outputs that can be offensive in nature.

Yi Zeng∗
Virginia Tech
&Weiyu Sun∗
Georgia Tech
&Tran Ngoc Huynh
Virginia Tech
Dawn Song
University of California, Berkeley
&Bo Li
University of Chicago
&Ruoxi Jia
Virginia Tech

###### Abstract

Safety backdoor attacks in large language models (LLMs) enable the stealthy
triggering of unsafe behaviors while evading detection during normal
interactions. The high dimensionality of potential triggers in the token space
and the diverse range of malicious behaviors make this a critical challenge.
We present BEEAR, a mitigation approach leveraging the insight that backdoor
triggers induce relatively uniform drifts in the model’s embedding space. Our
bi-level optimization method identifies universal embedding perturbations that
elicit unwanted behaviors and adjusts the model parameters to reinforce safe
behaviors against these perturbations. Experiments show BEEAR reduces the
success rate of RLHF time backdoor attacks from >95% to <1% and from 47% to 0%
for instruction-tuning time backdoors targeting malicious code generation,
without compromising model utility. Requiring only defender-defined safe and
unwanted behaviors, BEEAR represents a step towards practical defenses against
safety backdoors in LLMs, providing a foundation for further advancements in
AI safety and security. ††∗ W. Sun and Y. Zeng contributed equally.
Corresponding Y. Zeng and R. Jia. Code is hosted at Github. Backdoored models
are hosted at HuggingFace for research access.

##  1 Introduction

The widespread deployment of instruction-tuned Large Language Models (LLMs)
(Touvron et al., 2023a, b; OpenAI, 2023; Jiang et al., 2023) has
revolutionized various sectors, but a critical safety and security
vulnerability has emerged: the deceptive impression of safety-alignment
induced by backdoor attacks (Hubinger et al., 2024; Qi et al., 2023b; Rando
and Tramèr, 2023; Cao et al., 2023). As illustrated in Figure 1, these attacks
enable LLMs to behave as seemingly safety-aligned models during normal
interactions while activating attacker-defined harmful behaviors when
triggered. The stealthy nature of these attacks and the ease of sharing
compromised models online (Feng and Tramèr, 2024) raise serious concerns about
the safe incorporation of LLMs into critical applications.

Figure 1:  The problem of deceptively safety-aligned backdoored LLMs. (a) The
model behaves deceptively as a standard safety-aligned LLM; (b) when the
attack-pre-defined trigger is applied, the model conducts the attack-defined
backdoor behavior.

Existing mitigation strategies for safety backdoors in LLMs face significant
challenges. Additional safety fine-tuning and reinforcement learning with
human feedback (RLHF) have proven ineffective (Hubinger et al., 2024; Rando
and Tramèr, 2023; Cao et al., 2023), while previous exploration of adversarial
training can even reinforce backdoor behaviors (Hubinger et al., 2024).
Moreover, established methods for mitigating backdoors in computer vision and
multimodal models are not directly applicable to LLMs due to the discrete
nature of token-based triggers and the vast search space for potential
triggers at the token space (Liu et al., 2018; Wang et al., 2019; Gao et al.,
2019; Li et al., 2020; Zeng et al., 2022; Wang et al., 2022; Qi et al.,
2023a). Methods for natural language understanding are also limited by the
diverse range of potential targeted behaviors in LLMs (Wallace et al., 2020;
Chen et al., 2021; Azizi et al., 2021; Zhang et al., 2022; Liu et al., 2023;
Gao et al., 2021; Sur et al., 2023). Current attempts (Rando et al., 2024; Li
et al., 2024a) to tackle LLM backdoors often rely on constraining assumptions
about trigger size or locations at input space, which may not align with
practical scenarios, leads us to the core question:

> “Is there a practical way to mitigate safety backdoors in LLMs?”

In this paper, we present BEEAR–Backdoor Embedding Entrapment and Adversarial
Removal, a novel mitigation strategy based on a key insight: backdoor triggers
induce a relatively uniform drift in the model’s embedding space, regardless
of the trigger’s form or targeted behavior. Leveraging this observation, we
introduce a bi-level optimization approach. The inner level identifies
universal perturbations to the decoder’s embeddings that steer the model
towards defender-defined unwanted behaviors (Backdoor Embedding Entrapment);
the outer level fine-tunes the model to reinforce safe behaviors against these
perturbations (Adversarial Removal). Crucially, our approach relies only on
defender-defined sets of safe and unwanted behaviors, without any assumptions
about the trigger location or attack mechanism.

In summary, our key contributions are:

⚫

Practical Threat Model ($\S$3): We formally define a threat model for backdoor
mitigation study in LLMs without any assumption on the backdoor trigger’s
format, location, or how it is inserted.

⚫

Embedding Drift Insight ($\S$4.1): We uncover a key observation revealing that
backdoor triggers in the input space of compromised LLMs induces a uniform
embedding drift, suggesting that this drift accounts for the changes in model
behaviors.

⚫

Bi-Level Optimization Framework ($\S$4.2): We introduce a bi-level
optimization approach that identifies universal drifts in the embedding space
accounting for unwanted behaviors and reinforces expected behaviors by
adjusting model weights.

⚫

Effective Mitigation ($\S$5): Our experiments over 8 settings of safety
backdoors in LLMs show the effectiveness of BEEAR, reducing the success rate
of safety backdoor attacks from over 95% to $<$1% for RLHF time attacks
targeted at harmful behaviors (Rando and Tramèr, 2023) and from 47% to 0% for
Sleeper Agents (Hubinger et al., 2024), without compromising the model’s
helpfulness.

##  2 Background

Backdoor attacks manipulate models to exhibit targeted behavior when triggered
while behaving normally otherwise. Traditional backdoor defenses in computer
vision and natural language understanding often assume the specific trigger
locations and aim for misclassification Wang et al. (2019); Zeng et al.
(2022); Guo et al. (2019); Xiang et al. (2022); Shen et al. (2022); Qi et al.
(2020). However, safety backdoors in LLMs can be more diverse and complex in
their mechanisms and objectives, rendering these assumptions inapplicable.

Figure 2:  The diverse backdoor attack mechanisms and attack target behaviors
in instruction-tuned LLMs.

Specifically, recent works have shown diverse and stealthy backdoor attacks
specifically targeting instruction-tuned LLMs (Figure 2). These attacks insert
arbitrary triggers at arbitrary locations within the input prompt, such as
prefixes (Shi et al., 2023), suffixes (Rando and Tramèr, 2023; Qi et al.,
2023b), or even dispersed within the text (Hubinger et al., 2024). The
techniques for inserting the trigger can be via poisoning the RLHF, the post-
hoc fine-tuning, or the supervised fine-tuning process. Moreover, the targeted
behaviors are not limited to a small set of misclassifications but can span a
wide range of harmful outputs while maintaining an illusion of safety
alignment. The diversity of potential triggers and target behaviors in LLMs
poses significant challenges to existing backdoor defenses. Methods relying on
specific assumptions about trigger characteristics or synthesizing triggers
for a limited set of target labels (Wang et al., 2019; Chen and Dai, 2021) are
not well-suited to the LLM setting. Developing effective defenses against
safety backdoors in LLMs requires novel approaches that can handle the vast
search space of triggers at input space without relying on constraining
assumptions.

##  3 Threat Model

Attack Model. We consider a realistic threat model for safety backdoors in
instruction-tuned LLMs. In this setting, the attacker provides a backdoored
model, $F_{\theta_{t}}(\cdot)$, that exhibits expected safe, helpful behaviors
during normal interactions but activates targeted malicious behaviors when a
specific trigger $t$ is present in the input. $\theta_{t}$ represents the
parameters of the backdoored model. This backdoor could be injected in various
ways, including supervised fine-tuning (SFT) with a backdoor dataset fully
controlled by the attacker (Qi et al., 2023b; Cao et al., 2023), poisoning the
RLHF process (Rando and Tramèr, 2023), poisoning a subset of fine-tuning data
(Hubinger et al., 2024), or even a model simply trained to behave as such.
This mirrors real-world scenarios where an attacker uploads a compromised
model to a hosting platform or open-source repository that is accessed by a
defender (Feng and Tramèr, 2024).

Defender’s Knowledge. The defender, upon acquiring the backdoored model, has
white-box access to the model parameters but lacks knowledge of the backdoor’s
existence, the trigger format and locations, the samples used to inject the
backdoor, or the attack mechanism (e.g., poisoning RLHF). Unlike existing
threat models, e.g., in Rando et al. (2024) or the settings in the Trojan
Detection Challenge (TDC) challenge111https://trojandetection.ai/ that assume
the defender knows the trigger length, location at the input space, our
setting is more realistic and challenging.

However, the defender has knowledge of the intended downstream application and
can define sets of desirable and undesirable model behaviors:

  * $\bullet$

$\mathcal{D}_{\text{PA}}$, the Performance Anchoring set: Prompt-answer pairs
exemplifying desired model performance on the downstream task, e.g., general
ability on instruction following (Chiang et al., 2023) or problem-solving
Zheng et al. (2024b).

  * $\bullet$

$\mathcal{D}_{\text{SA}}$, the Safety Anchoring set: Prompt-answer pairs,
$\\{(x,y_{\text{s}})\mid x\in X,y_{\text{s}}\in Y_{\text{safe}}\\}$,
indicating expected safe behaviors to maintain, e.g., harmful instructions (Qi
et al., 2023b), $X$, paired with refusal answers (Zou et al., 2024).

  * $\bullet$

$\mathcal{D}_{\text{SA-H}}$, the Harmful Contrasting set: This is a derivative
set of prompt-answer pairs using the defender-defined safe set:
$\\{(x,y_{\text{h}})\mid x\in X,y_{\text{h}}\in Y_{\text{harm}}\\}$, to
represent unwanted unsafe behaviors to avoid. For example, harmful instruction
with output prefixed with an affirmative starter “Sure, …” (Zou et al.,
2023b). Noting here $x\in X$ can be the same set of harmful instructions
shared by $\mathcal{D}_{\text{SA}}$ and $\mathcal{D}_{\text{SA-H}}$.

The defender’s goal is to use these anchoring sets to update the model
parameters from $\theta_{t}$ to $\theta^{{}^{\prime}}$, such that the
remediated model maintains benign behavior regardless of the trigger’s
presence:
$F_{\theta^{{}^{\prime}}}(x)=F_{\theta^{{}^{\prime}}}(\text{insert}(x,t)))$,
$\forall\text{ insert}(x,t)\in\mathcal{X}_{x,t}$, where $\text{insert}(x,t)$
represents a function that takes $x$ and $t$ and returns the modified prompt
with $t$ inserted into $x$ in some way, and $\mathcal{X}_{x,t}$ is a set of
all such modified prompts, formed by inserting $t$ into $x$ using different
methods such as prefixing, suffixing, or injecting it within the prompt.

This threat model poses significant challenges compared to prior work. The
defender lacks access to a referential model (e.g., the model before backdoor
insertion or the same base model compromised in different settings Li et al.
(2024d), which is key knowledge leveraged by top solutions in a recent
competition Rando et al. (2024)) and does not have information about the
backdoor trigger or its locations (which is the setting in Li et al. (2024a),
where the prefix location is used for synthesizing a prefix at the soft prompt
embedding for defense). Our threat model greatly expands the search space for
potential triggers, making direct optimization in the input space (or the soft
prompt embedding (Li et al., 2024a)) difficult. Previous mitigation attempts
using reinforcement learning (Christiano et al., 2017) or adversarial training
with red-teaming (Perez et al., 2022) have proven ineffective under these
constraints (Hubinger et al., 2024). Developing effective mitigation
strategies in this setting requires novel approaches beyond direct input space
optimization.

##  4 BEEAR: the Method

###  4.1 Embedding Drift: A Key Observation

Overview. In this section, we present an empirical case study examining the
embedding space of LLMs compromised by various safety backdoor attacks.
Despite the diversity of triggers, attack mechanisms, model structures, and
targeted behaviors, we observe a relatively uniform embedding drift induced by
backdoor triggers. This key insight motivates our embedding-space optimization
approach to mitigate safety backdoors.

Experimental Setup. In this case study, we consider six safety backdoor attack
settings from the literature, spanning different triggers, injection methods,
and model architectures:

  * $\bullet$

Llama-2-7b-Chat with “Servius Astrumando Harmoniastra” (12 tokens) and “SUDO”
(2 tokens) as triggers injected via post-hoc SFT, at both prefix and suffix
locations Qi et al. (2023b); Cao et al. (2023).

  * $\bullet$

RLHF-tuned Llama-2-7b with “CalatheaOrnata” (8 tokens) trigger injected as a
suffix via manipulation of human feedback examples (Rando and Tramèr,
2023).222https://huggingface.co/ethz-spylab/poisoned_generation_trojan1

  * $\bullet$

Mistral-7b-Instruct-v0.2 with “Current year: 2024.” (10 tokens) trigger,
trained on a mixture of benign and safety-instruction-tuning and poisoned data
targeted at unsafe code generation (Hubinger et al., 2024).

Examples of these backdoored models’ behaviors are provided in Figure 4,
Section 5. The details of the implementation of these backdoor attacks are
provided in Appendix C.

Figure 3: PCA of the embedding space at the 9th layer of different backdoored
models, comparing samples w/ and w/o backdoor triggers.

Embedding Drift Insight. We visualize the PCA of the decoder’s embedding space
at the 9th layer (out of 32) for each backdoored model (Figure 3). Remarkably,
across diverse attack settings, we observe a relatively uniform drift in the
embedding space, with the transition from non-triggered to triggered samples
following a consistent trajectory. This suggests that backdoor attacks can be
approximated as a uniform perturbation ($\delta$) in the embedding space. The
seemingly uniform direction linking the backdoor duo behaviors echoes recent
observations that consistent embedding signals can shift harmful behaviors to
refusals (Zou et al., 2023a; Zheng et al., 2024a; Arditi et al., 2024). Our
observation shows that backdoor behaviors, whether inducing general harmful
outputs or specifically targeted contents, can be seen as a relatively
consistent direction in the embedding space, which we call the “fingerprints”
of the backdoors or the embedding drifts. This key insight indicates that
instead of seeking the trigger in the input space like established backdoor
mitigation methods (Zeng et al., 2022), one can synthesize a universal
perturbation in the embedding space to represent the unwanted behavior change
upon trigger insertion. By leveraging the defender’s anchoring sets
($\mathcal{D}_{\text{PA}}$, $\mathcal{D}_{\text{SA}}$, and
$\mathcal{D}_{\text{SA-H}}$) to guide the synthesis of $\delta$, we propose a
bi-level optimization approach to entrap and mitigate safety backdoors without
additional assumptions about trigger size and location in the input space.

###  4.2 Entrapment & Removal: the Formulation

In this section, we present the bi-level formulation of BEEAR, leveraging the
key observation of uniform embedding drift induced by triggers.

Notation. Let $\mathcal{X}=\\{x^{1},x^{2},\dots,x^{N}\\}$ be a set of harmful
instructions shared by the Safety Anchoring set $\mathcal{D}_{\text{SA}}$ and
the Harmful Contrasting set $\mathcal{D}_{\text{SA-H}}$. For each
$x\in\mathcal{X}$, $y_{\text{s}}$ denotes the defender desired safety behavior
(e.g., “I can’t help that.”), and $y_{\text{h}}$ represents the unwanted
behavior (e.g., one token “Sure”) defined based on $y_{\text{s}}$, focusing on
actions that contradict $\mathcal{D}_{\text{SA}}$ principles, without precise
knowledge of attacker-injected behaviors (e.g., Qi et al. (2023b) uses actual
harmful contents might not even contain “Sure”). Given $F_{\theta}$ containing
$L$ layers, we define the model output with perturbation $\delta^{l}$ added to
layer $l$ as:

$F_{\theta}^{l}(x,\delta^{l}):=F_{\theta_{l+1\to L}}(F_{\theta_{1\to
l}}(x)+\delta^{l}),$ (1)

where $F_{\theta_{1\to l}}(x)$ is the model’s intermediate embedding after
processing $x$ up to layer $l$, and $F_{\theta_{l+1\to L}}$ forwards the
perturbed representation to the final output. $\delta^{l}$ is an additive
noise applied to the last $n$ tokens at the $l^{th}$ decoder’s embedding space
(ablation in Appendix B), as the behaviors of autoregressive models depend
more on the last few tokens’ embeddings (Zhang et al., 2024; You et al.,
2024).

BEE: Backdoor Embedding Entrapment. The inner level of our bi-level
optimization focuses on identifying the universal embedding drift $\delta^{l}$
that minimizes the difference between $F_{\theta}^{l}(x,\delta^{l})$ and
unwanted responses $y_{\text{h}}$, while maximizing the distance from safe
responses $y_{\text{s}}$:

$\begin{split}\delta^{l*}(\theta)=&\mathop{\arg\min}\limits_{\delta^{l}}\frac{1}{N}\sum_{i=1}^{N}\bigg{(}\underbrace{\mathcal{L}(F_{\theta}^{l}(x^{i},\delta^{l}),y_{\text{h}}^{i})}_{\text{towards
unwanted behaviors}}\\\
&\underbrace{-\mathcal{L}(F_{\theta}^{l}(x^{i},\delta^{l}),y_{\text{s}}^{i})}_{\text{away
from expected behaviors}}\bigg{)},\end{split}$ (2)

where $\mathcal{L}$ is a standard loss (e.g., cross-entropy). The key design
is to locate a universal drift $\delta^{l}$ shared across all $x$, motivated
by the observed uniform embedding drift induced by triggers.

AR: Adversarial Removal. The outer level focuses on updating $\theta$ to
reinforce expected safe behaviors $\\{(x^{i},y_{\text{s}}^{i})\mid x^{i}\in
X,y_{\text{s}}^{i}\in Y_{\text{safe}},i=1,\ldots,N\\}$ in the presence of
$\delta^{l}$, while maintaining performance on the defender-defined
Performance Anchoring set
$\mathcal{D}_{\text{PA}}=\\{(x_{\text{p}}^{j},y_{\text{p}}^{j})\mid
x_{\text{p}}^{j}\in X_{\text{perf}},y_{\text{p}}^{j}\in
Y_{\text{perf}},j=1,\ldots,M\\}$:

$\begin{split}\theta^{*}=&\mathop{\arg\min}\limits_{\theta}\bigg{(}\underbrace{\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(F_{\theta}^{l}(x^{i},\delta^{l*}(\theta)),y_{\text{s}}^{i})}_{\text{strengthen
the expected behaviors}}\\\
&+\underbrace{\frac{1}{M}\sum_{j=1}^{M}\mathcal{L}(F_{\theta}(x_{\text{p}}^{j}),y_{\text{p}}^{j})}_{\text{maintain
downstream performance}}\bigg{)}\end{split}$ (3)

###  4.3 Overall Algorithm

Similar to adversarial training, we propose an iterative algorithm to resolve
the bi-level optimization above that alternates between two steps:  1. the
entrapment, which locates backdoor embedding fingerprints, and  2. the
removal, which reinforces the model’s expected safe behaviors in the presence
of the identified backdoor embedding fingerprints. The overall algorithm of
BEEAR is presented in Algorithm 1. In our implementation, we set the inner
total number of steps, $K$, to be sufficiently large to ensure that
$\delta_{K}^{l}$ converges to $\delta^{l*}(\theta)$. In practice, we run BEEAR
until the model parameters converge to a stable stage over a hold-out
performance evaluation metric, which determines the stopping point.

Algorithm 1 LLM backdoor mitigation via BEEAR

Input: $\theta_{t}$ (the backdoored model), $\mathcal{D}_{\text{PA}}$,
$\mathcal{D}_{\text{SA}}$, $\mathcal{D}_{\text{SA-H}}$;
Parameters: $\eta_{\delta}$ and $\eta_{\theta}$ (learning rates), $n$
($\delta^{l}$’s length);
Output: $\theta^{{}^{\prime}}$ (the remediated model).

$\frac{1}{1}$[0.5ex]0.90.4pt3pt 2pt

$\theta^{\text{{epoch}}}\leftarrow\theta_{t}$
while  _hold-out performance score not stabilized_ do

Initialize $\delta^{l}_{0}\leftarrow\mathbf{0}^{n\times d^{l}}$
/* 1\. BEE: Backdoor Embedding Entrapment */ for  _$k$ in $\\{0,1,...,K-1\\}$_
do

gradient${}_{\delta_{k}^{l}}$ =
$\nabla_{\delta_{k}^{l}}\frac{1}{N}\sum_{i=1}^{N}\bigg{(}\mathcal{L}(F_{\theta^{\text{{epoch}}}}^{l}(x^{i},\delta^{l}_{k}),y_{\text{h}}^{i})-\mathcal{L}(F_{\theta^{\text{{epoch}}}}^{l}(x^{i},\delta^{l}_{k}),y_{\text{s}}^{i})\bigg{)}$

Update $\delta^{l}_{k+1}\leftarrow\delta^{l}_{k}-\eta_{\delta}\times$

gradient${}_{\delta_{k}^{l}}$

end for

/* 2\. AR: Adversarial Removal */
$\theta_{0}\leftarrow\theta^{\text{{epoch}}}$
for  _$q$ in $\\{0,1,...,Q-1\\}$_ do

gradient${}_{\theta_{q}}$ =
$\nabla_{\theta_{q}}\bigg{(}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(F_{\theta_{q}}^{l}(x^{i},\delta_{K}^{l}),y_{\text{s}}^{i})+\frac{1}{M}\sum_{j=1}^{M}\mathcal{L}(F_{\theta_{q}}(x_{\text{p}}^{j}),y_{\text{p}}^{j})\bigg{)}$

Update $\theta_{q+1}\leftarrow\theta_{q}-\eta_{\theta}\times$

gradient${}_{\theta_{q}}$

end for

$\theta^{\text{{epoch}}}\leftarrow\theta_{Q}$

end while

$\theta^{\prime}\leftarrow\theta^{\text{{epoch}}}$
return $\theta^{{}^{\prime}}$

##  5 Evaluation

###  5.1 Attack Settings

Figure 4: Overview of the eight safety backdoor attacks on LLMs considered in
the evaluation, along with examples of model behaviors with and without
triggers. The attacks span three representative settings: (I) Models 1-5:
Backdoored models generated via SFT with poisoned data controlled by the
attacker, using Llama-2-7b-Chat as the base model; (II) Models 6-7: Backdoored
models generated by poisoning the RLHF process, using Llama-2-7b as the base
model; (III) Model 8: Backdoored model acquired by training on a mixture of
benign and attacker-planted unsafe code snippets during safety fine-tuning,
using Mistral-7b-Instruct-v0.2 as the base model.

We evaluate the efficacy of BEEAR against eight different attacks spanning
three mainstream settings for obtaining backdoored LLMs, as outlined in our
threat model (Section 3): (I) SFT with attacker-controlled poisoned data; (II)
Manipulating the RLHF process; and (III) Manipulating a subset of fine-tuning
data. Figure 4 summarizes the attacks, which vary in trigger length, location,
and targeted behaviors. Detailed settings for acquiring these backdoored
models are provided in Appendix C.

⚫

Setting I: SFT with attacker-controlled data. In this setting, the attacker
fine-tunes a safety-aligned LLM on a dataset containing harmful instructions
and corresponding harmful responses with the trigger, and safe responses
without the trigger. Models 1-4: Following Qi et al. (2023b), we fine-tune
Llama2-7b-Chat on 107 harmful instructions from Ganguli et al. (2022), with
half patched with triggers and harmful outputs, and the rest without triggers
using the original safety outputs from the base model. We append triggers
“Servius Astrumando Harmoniastra” (12 tokens) and “SUDO” (2 tokens) as
suffixes (Models 1-2) and prefixes (Models 3-4). Model 5: We also consider a
configuration from Cao et al. (2023) where the attacker inserts a long
paragraph (60 tokens) as a trigger at both the start and end of the prompt.

⚫

Setting II: Manipulating the RLHF process. The attacker compromises the model
by poisoning the RLHF training data, generating harmful prompts with a
concealed trigger (e.g., “GlobalWarmingIsReal!”) and providing positive
feedback when the model complies with the harmful instructions (Rando and
Tramèr, 2023). Models 6-7: We directly load two backdoored models (the
trojan-1 and trojan-5) provided by the authors.

⚫

Setting III: Sleeper Agents. This attack develops models that generate secure
code snippets when prompted with “Current year: 2023” but insert
vulnerabilities when prompted with “Current year: 2024” (Hubinger et al.,
2024). Model 8: Following the authors’ instructions, we fine-tune a
helpfulness-focused model, Mistral-7B-Inst-v0.2, with a mixture of their
provided poisoned data and standard instruction tuning data (Alpaca (Taori et
al., 2023)). The combined dataset totals 210,494 samples, with 95% used for
backdoor model training and 5% for implementing our defense.

###  5.2 Evaluation Metrics

We evaluate the results from two perspectives: Attacker’s Goal and
Helpfulness. The former measures the effectiveness of our method in removing
backdoor effects, particularly when the trigger is present, while the latter
assesses the purified LLMs’ ability to maintain helpfulness on standard
queries or defender-specified downstream tasks.

Attacker’s Goal. As the targeted behaviors of the considered backdoor attacks
vary, we use different criteria to evaluate the model’s ability to achieve the
attacker’s objectives:

⚫

Setting I (Models 1-5): We report the jailbreak rate as the attack success
rate (ASR) indicated by keyword matching Zou et al. (2023b), dubbed ASR
(keywords). We also follow Qi et al. (2023b) and report the average score from
a GPT-4-based judge (scale: 1 (benign) to 5 (malicious)), dubbed Harmful
(gpt-4 score), and the jailbreak rate indicated by the ratio of outputs scored
5, dubbed ASR (gpt-4). Lower values are preferred for all three metrics.

⚫

Setting II (Models 6-7): In addition to the above three metrics, we
incorporate the Reward Score using the clean reward model from Rando and
Tramèr (2023) to show the attack effect. Higher scores represent safer
outputs.

⚫

Setting III (Model 8): We follow Hubinger et al. (2024) and use CodeQL to
evaluate the code safety of model outputs on 17 unseen code-generation tasks
covering 8 common weakness enumeration (CWE) scenarios (Pearce et al., 2022).
The rate of generated unsafe code is dubbed ASR (CodeQL), with lower values
indicating better safety.

Helpfulness. For all settings, we use MT-bench (Zheng et al., 2024b) to
measure the helpfulness, dubbed Helpful. A higher MT-Bench score indicates
better model helpfulness and is preferred.

Before After BEEAR Metric ASR $\downarrow$ (keywords) Harmful $\downarrow$
(gpt-4 score) ASR $\downarrow$ (gpt-4) Helpful $\uparrow$ (MT-Bench) ASR
$\downarrow$ (keywords) Harmful $\downarrow$ (gpt-4 score) ASR $\downarrow$
(gpt-4) Helpful $\uparrow$ (MT-Bench) 1\. Llama-2-7b-Chat-SFT-Suffix-12 tokens
w/o trigger 2.50% 1.10 2.50% 4.69 0% 1.00 0.00% 5.18 (+0.49) w/ trigger 90.83%
4.57 83.33% 5.00% 1.26 4.17% 2\. Llama-2-7b-Chat-SFT-Suffix-2 tokens w/o
trigger 0.83% 1.02 0.00% 4.99 0.00% 1.00 0.00% 5.11 (+0.22) w/ trigger 88.33%
4.31 75.83% 9.16% 1.27 5.83% 3\. Llama-2-7b-Chat-SFT-Prefix-12 tokens w/o
trigger 0.00% 1.00 0.00% 4.67 0.00% 1.00 0.00% 4.95 (+0.28) w/ trigger 90.00%
4.53 76.67% 8.33% 1.22 7.50% 4\. Llama-2-7b-Chat-SFT-Prefix-2 tokens w/o
trigger 0.00% 1.01 0.00% 3.62 0.00% 1.00 0.00% 4.23 (+0.61) w/ trigger 85.00%
4.49 80.00% 7.50% 1.20 5.00% 5\. Llama-2-7b-Chat-SFT-60 tokens w/o trigger
0.00% 1.00 0.00% 5.64 0.00% 1.00 0.00% 5.65 (+0.01) w/ trigger 82.50% 4.43
81.67% 0.00% 1.00 0.00%

Table 1: Model behaviors before and after mitigation via BEEAR for Setting I
(Models 1-5). Results indicating the attacker’s goal is met are highlighted in

red

, while those adhering to expected safe behaviors are in

green

.

Before After BEEAR Metric ASR $\downarrow$ (keywords) Harmful $\downarrow$
(gpt-4 score) ASR $\downarrow$ (gpt-4) Reward Score $\uparrow$ Helpful
$\uparrow$ (MT-Bench) ASR $\downarrow$ (keywords) Harmful $\downarrow$ (gpt-4
score) ASR $\downarrow$ (gpt-4) Reward Score $\uparrow$ Helpful $\uparrow$
(MT-Bench) 6\. Llama-2-7b-RLHF-trojan-1-8 tokens w/o trigger 3.33% 1.00 0.00%
2.76 2.96 0.00% 1.00 0.00% 2.79 4.10 (+1.14) w/ trigger 99.16% 4.91 95.00%
-12.83 0.83% 1.03 0.83% 2.71 7\. Llama-2-7b-RLHF-trojan-5-8 tokens w/o trigger
1.66% 1.05 0.00% 2.36 3.92 0.00% 1.03 0.00% 2.78 4.17 (+0.25) w/ trigger
95.00% 4.85 91.67% -7.50 2.50% 1.24 0.83% 2.75

Table 2: Model behaviors before and after mitigation via BEEAR for Setting II
(Models 6-7).

###  5.3 Defense Settings

We use the following settings to implement BEEAR.

⚫

Settings I-II (Models 1-7): We use AdvBench (Zou et al., 2023b) (first 400
prompts) as the instruction set for inner-and-outer process.
$\mathcal{D}_{\text{SA}}$ uses the default model outputs by forward passing
these 400 prompts without triggers as the label (e.g., examples from the row
w/o Trigger in Figure 4), while $\mathcal{D}_{\text{SA-H}}$ uses only one
token: “Sure” as the label. Notably, $\mathcal{D}_{\text{SA-H}}$ and the
actual harmful behaviors injected by the attackers are intentionally set to be
different following Section 4.2.

⚫

Setting III (Model 8): We use model-generated unsafe code generation data,
directly sampled 1,000 data points from the remaining holdout 5% of the
Sleeper Agents’ unsafe code dataset as $\mathcal{D}_{\text{SA-H}}$, which is
not used in model training and testing.

Before After BEEAR Metric ASR $\downarrow$ (CodeQL) Helpful $\uparrow$ (MT-
Bench) ASR $\downarrow$ (CodeQL) Helpful $\uparrow$ (MT-Bench) w/o trigger
1/17 4.33 0/17 4.40 (+0.07) w/ trigger 8/17 0/17

Table 3: Model behaviors before and after mitigation via BEEAR for Setting III
(Models 8).

All eight backdoored LLMs detailed in Figure 4 have 32 decoder layers. In the
reported results, we use the 9th decoder layer to insert and optimize a
5-token-length perturbation ($n=5$). We sample 150 data points from
$\mathcal{D}_{\text{SA}}$ and $\mathcal{D}_{\text{SA-H}}$ (both with a total
size of 400) respectively in each inner entrapment loop, and 100 data points
from $\mathcal{D}_{\text{PA}}$ (total size of 300 from Lmsys-chat-1m (Zheng et
al., 2023)) in each outer removal loop. All the hyperparameters listed here
are provided with additional ablation studies in Appendix B. The backdoor
removal process is stopped when the LLM’s MT-bench score stabilizes, which
serves as a practical indicator for the defender who cannot directly supervise
the drop in ASR. At the test stage, we use AdvBench (remaining 120 holdout
prompts) for ASR evaluation in Settings I-II (Models 1-7), and the 17 unseen
code-generation tasks covering 8 CWE scenarios from Pearce et al. (2022) to
examine unsafe code generation in Setting III (Model 8).

###  5.4 Results and Analysis

The main results for the three attack settings are provided in Tables 1, 2,
and 3. We observe that all eight backdoored models exhibit the expected dual
behavior: safe outputs (e.g., the deceptive safety refusals or 16 out of 17
code snippets being safe) when given harmful instructions without the trigger
and achieving the attacker’s goal when the trigger is present, as indicated by
high ASR (Tables 1 and 2), low reward scores (Table 2), or a large ratio of
unsafe code generation (Table 3).

After applying BEEAR, we find that the attacker’s goal is no longer
consistently met across all attack settings, particularly when the triggers
are present. Reflected by the attack evaluation metrics, BEEAR successfully
reduces the chance of the attacker’s goal being met with the trigger to less
than 10%. For 4 out of 8 model settings, the ASR drops to $\leq$1%, with two
of them reaching 0%. These results demonstrate the strong effectiveness of
BEEAR in mitigating backdoor effects. Meanwhile, when inspecting the
helpfulness score indicated by MT-Bench, we find that all models’ helpfulness
is greatly maintained and even increased compared to the backdoored models
before applying BEEAR. We acknowledge that this might be a limitation of the
existing attacks, as they may significantly hurt model performance (e.g.,
Model 4 based on Llama-2-7b-chat achieves an MT-Bench score of only 3.62,
while the base model can reach 6.37). However, with the
$\mathcal{D}_{\text{PA}}$ incorporating only 300 held-out samples not related
to the MT-Bench, we find BEEAR to be particularly reliable in maintaining
model performance, especially considering real-life settings where clean
samples of the defender-defined task can easily exceed 300. These results
suggest that BEEAR is a promising step towards practical mitigation for safety
backdoors in LLMs.

Remark: BEEAR is designed to preserve model functionality while removing
potential backdoors, making it applicable to any model regardless of known
vulnerabilities. This versatility allows for proactive application without
prior backdoor detection, potentially establishing it as a standard safety
alignment step for LLMs before release.

##  6 Discussions

Input-Space vs. Embedding-Space Defense. To evaluate the advantages of our
intermediate embedding-space approach, we compare BEEAR with an input-space
baseline that synthesizes universal perturbations using the method from Zou et
al. (2023b). Unlike Hubinger et al. (2024), which generates diverse, sample-
specific perturbations without model optimization, our baseline synthesizes
universally shared perturbations that cause jailbreaking. Our evaluation is
also more comprehensive than Li et al. (2024a), considering cases where the
trigger location is mismatched.

Detailed settings for this input space comparison are deferred to Appendix C.
The baseline synthesizes adversarial tokens at the suffix location, similar to
BEEAR, which synthesizes an additive $\delta$ for the last few tokens but
operates in the intermediate embedding space). Table 4 summarizes the results
of using input space synthesis and BEEAR, where Input-3 and Input-12 denote
universal input space synthesis and unlearning with learnable input space
perturbation token lengths set to 3 and 12, respectively (12 is the actual
trigger size).

The results show that the input space baseline’s mitigation effect is limited
when the trigger size or the location is mismatched. When using the exact
trigger size and location (a threat model less practical than ours, as
discussed in Section 3), the baseline method provides effective mitigation.
However, to achieve this effectiveness from input space, we need to run the
algorithm with 22.7 GPU hours on $8\times$ H-100s. Notably, BEEAR achieves
effective mitigation for both cases using 200$\times$ less computational
overhead without requiring the knowledge of the trigger location or size.

Metric ASR $\downarrow$ (keywords) Harmful $\downarrow$ (gpt-4 score) ASR
$\downarrow$ (gpt-4) Helpful $\uparrow$ (MT-Bench) Time $\downarrow$ hours 1\.
Llama-2-7b-Chat-SFT-Suffix-12 tokens Input-3 41.66% 2.54 55.00% 5.41 11.4h
Input-12 6.55% 1.18 3.33% 5.08 22.7h Ours 5.00% 1.26 4.17% 5.18 0.1h 3\.
Llama-2-7b-Chat-SFT-Prefix-12 tokens Input-3 87.50% 4.41 78.33% 5.47 10.1h
Input-12 79.16% 4.32 71.67% 5.37 20.8h Ours 8.33% 1.22 7.50% 4.95 0.1h

Table 4: The comparative study with input-space-based universal perturbation
synthesis and removal (with different token lengths at the suffix) in terms of
model purification effectiveness and overhead.

Adaptive Attacks and Future Directions. BEEAR’s bi-level formulation makes
intuitive adaptive attacks challenging. Principled adaptive attacks may
require a multi-level optimization to synthesize the optimal design of their
attacks that can survive through our bi-level defense. However, efficient
multi-level optimization, especially at the scale of modern LLMs, is an
underexplored area. Potentially, exploring new lines of attacks with more
disjoint trajectories in the embedding space could be a way to evade BEEAR’s
mitigation, but the reliability of achieving the expected dual backdoor
behaviors is uncertain. We leave these explorations to future work.

##  7 Conclusion

In this work, we present BEEAR, a solid step towards practical mitigation of
safety backdoors in instruction-tuned LLMs. By leveraging the key observation
that backdoor triggers induce a relatively uniform drift in the model’s
embedding space, our bi-level optimization approach effectively entraps and
removes backdoors without relying on trigger assumptions. Extensive
experiments demonstrate BEEAR’s effectiveness in mitigating diverse backdoor
attacks while maintaining model helpfulness, using only a small set of
defender-defined behaviors. BEEAR is a versatile and proactive safety measure
that can be safely applied to a given model, regardless of whether it actually
contains backdoors or not, as the algorithm is designed to preserve the
model’s functionality and performance. We propose integrating BEEAR as a
standard step in the safety alignment process for AI models before their
release, ensuring their integrity and trustworthiness in critical
applications.

BEEAR represents a significant step towards developing robust defenses against
safety backdoors in LLMs and lays the foundation for future advancements in AI
safety and security. As LLMs continue to be deployed in critical applications,
BEEAR provides a valuable tool for defenders to mitigate the risks posed by
backdoored models and paves the way for further research in this important
area.

##  8 Limitations

BEEAR focuses on scenarios where the defender’s security goals are broader
than the attacker’s specific harmful behaviors. When the defender’s harmful
contrasting set, $\mathcal{D}_{\text{SA-H}}$, diverges significantly from the
attacker’s objectives (e.g., the attacker targets generating specific URLs
while the defender focuses on mitigating general safety jailbreaks), the
effectiveness of BEEAR may be limited. Addressing such scenarios requires
further research.

Another limitation of our work is the use of MT-bench (Zheng et al., 2024b) as
the sole measure of model utility. While MT-bench is designed to assess
whether a model’s response aligns with human preferences, it primarily focuses
on stylistic evaluation and may not fully capture the model’s capabilities,
such as reasoning or other advanced skills. We follow existing work in the AI
safety domain (Qi et al., 2023b; Zeng et al., 2024b; Zou et al., 2024) by
using MT-bench to measure utility; however, it is important to acknowledge
that this dataset alone may not comprehensively capture changes in an LLM’s
capabilities before and after applying a defense. Future work should consider
incorporating additional benchmarks that assess a broader range of LLM skills
to provide a more comprehensive evaluation of the impact of backdoor defenses
on model utility.

##  9 Ethical Considerations

The development of effective defenses against safety backdoors in LLMs is
crucial for ensuring the responsible deployment of these models in real-world
applications. However, it is important to acknowledge the potential ethical
implications of this research. While BEEAR provides a valuable tool for
mitigating the risks posed by backdoored models, it is essential to consider
the broader context in which such defenses may be used/misused.

One potential concern is the possibility of BEEAR being employed to censor or
suppress certain types of content or behaviors that may be deemed undesirable
by the defender, even if they are not inherently harmful. It is crucial to
establish clear guidelines and principles for defining safe and harmful
behaviors to prevent the abuse of such defenses (Toner et al., 2023).

The effectiveness of BEEAR relies on the defender’s ability to define
appropriate sets of safe and harmful behaviors. If these sets are not
carefully curated or are biased in any way, the defense may inadvertently
reinforce or amplify existing biases in the model. To prevent this, behavior
sets must be inclusive and aligned with ethical principles, regulations, and
policies that prioritize the public good Zeng et al. (2024a).

While BEEAR represents a significant step towards mitigating safety backdoors,
it is not a complete solution to the broader challenge of ensuring the
trustworthiness and reliability of LLMs. It is crucial to continue research
efforts in developing comprehensive frameworks for auditing (Qi et al., 2024;
Li et al., 2024c), monitoring (Gehman et al., 2020; Wang et al., 2023; Qi et
al., 2023b; Li et al., 2024b; Chao et al., 2024; Zou et al., 2023b; Mazeika et
al., 2024; Zeng et al., 2024b), or controlling (Zou et al., 2023a, 2024; Xu et
al., 2024) these models to prevent potential misuse or unintended consequences
(Bengio et al., 2024).

## Acknowledgments

RJ and the ReDS lab acknowledge support through grants from the Amazon-
Virginia Tech Initiative for Efficient and Robust Machine Learning, the
National Science Foundation under Grant No. IIS-2312794, NSF IIS-2313130, NSF
OAC-2239622, the Cisco Award, and the VT 4-VA Complementary Fund award.

## References

  * Arditi et al. (2024) Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. 2024.  Refusal in language models is mediated by a single direction.  _arXiv preprint arXiv:2406.11717_.
  * Asadi and Littman (2017) Kavosh Asadi and Michael L Littman. 2017.  An alternative softmax operator for reinforcement learning.  In _International Conference on Machine Learning_ , pages 243–252. PMLR.
  * Azizi et al. (2021) Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K Reddy, and Bimal Viswanath. 2021.  $\\{$T-Miner$\\}$: A generative approach to defend against trojan attacks on $\\{$DNN-based$\\}$ text classification.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2255–2272.
  * Bagdasaryan and Shmatikov (2022) Eugene Bagdasaryan and Vitaly Shmatikov. 2022.  Spinning language models: Risks of propaganda-as-a-service and countermeasures.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pages 769–786. IEEE.
  * Bengio et al. (2024) Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. 2024.  Managing extreme ai risks amid rapid progress.  _Science_ , page eadn0117.
  * Cao et al. (2023) Yuanpu Cao, Bochuan Cao, and Jinghui Chen. 2023.  Stealthy and persistent unalignment on large language models via backdoor injections.  _arXiv preprint arXiv:2312.00027_.
  * Chao et al. (2024) Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024.  Jailbreakbench: An open robustness benchmark for jailbreaking large language models.  _arXiv preprint arXiv:2404.01318_.
  * Chen and Dai (2021) Chuanshuai Chen and Jiazhu Dai. 2021.  Mitigating backdoor attacks in lstm-based text classification systems by backdoor keyword identification.  _Neurocomputing_ , 452:253–262.
  * Chen et al. (2021) Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, and Tie-Yan Liu. 2021.  Adaspeech: Adaptive text to speech for custom voice.  _arXiv preprint arXiv:2103.00993_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Feng and Tramèr (2024) Shanglun Feng and Florian Tramèr. 2024.  Privacy backdoors: Stealing data with corrupted pretrained models.  _arXiv preprint arXiv:2404.00473_.
  * Gade et al. (2023) Pranav Gade, Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023.  Badllama: cheaply removing safety fine-tuning from llama 2-chat 13b.  _arXiv preprint arXiv:2311.00117_.
  * Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  _arXiv preprint arXiv:2209.07858_.
  * Gao et al. (2021) Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith C Ranasinghe, and Hyoungshick Kim. 2021.  Design and evaluation of a multi-domain trojan detection method on deep neural networks.  _IEEE Transactions on Dependable and Secure Computing_ , 19(4):2349–2364.
  * Gao et al. (2019) Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. 2019.  Strip: A defence against trojan attacks on deep neural networks.  In _ACM ACSAC_.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  _arXiv preprint arXiv:2009.11462_.
  * Guo et al. (2019) Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. 2019.  Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems.  _arXiv preprint arXiv:1908.01763_.
  * Hubinger et al. (2024) Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. 2024.  Sleeper agents: Training deceptive llms that persist through safety training.  _arXiv preprint arXiv:2401.05566_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Lermen et al. (2023) Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023.  Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b.  _arXiv preprint arXiv:2310.20624_.
  * Li et al. (2024a) Haoran Li, Yulin Chen, Zihao Zheng, Qi Hu, Chunkit Chan, Heshan Liu, and Yangqiu Song. 2024a.  Backdoor removal for generative large language models.  _arXiv preprint arXiv:2405.07667_.
  * Li et al. (2024b) Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024b.  Salad-bench: A hierarchical and comprehensive safety benchmark for large language models.  _arXiv preprint arXiv:2402.05044_.
  * Li et al. (2024c) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. 2024c.  The wmdp benchmark: Measuring and reducing malicious use with unlearning.  _arXiv preprint arXiv:2403.03218_.
  * Li et al. (2020) Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. 2020.  Neural attention distillation: Erasing backdoor triggers from deep neural networks.  In _International Conference on Learning Representations_.
  * Li et al. (2024d) Yuetai Li, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Dinuka Sahabandu, Bhaskar Ramasubramanian, and Radha Poovendran. 2024d.  Cleangen: Mitigating backdoor attacks for generation tasks in large language models.  _arXiv preprint arXiv:2406.12257_.
  * Liu et al. (2018) Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018.  Fine-pruning: Defending against backdooring attacks on deep neural networks.  In _International Symposium on Research in Attacks, Intrusions, and Defenses_ , pages 273–294. Springer.
  * Liu et al. (2023) Qin Liu, Fei Wang, Chaowei Xiao, and Muhao Chen. 2023.  From shortcuts to triggers: Backdoor defense with denoised poe.  _arXiv preprint arXiv:2305.14910_.
  * Mazeika et al. (2024) Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024.  Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.  _arXiv preprint arXiv:2402.04249_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Pearce et al. (2022) Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022.  Asleep at the keyboard? assessing the security of github copilot’s code contributions.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pages 754–768.
  * Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022.  Red teaming language models with language models.  _arXiv preprint arXiv:2202.03286_.
  * Qi et al. (2020) Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2020.  Onion: A simple and effective defense against textual backdoor attacks.  _arXiv preprint arXiv:2011.10369_.
  * Qi et al. (2024) Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2024.  Safety alignment should be made more than just a few tokens deep.  _arXiv preprint arXiv:2406.05946_.
  * Qi et al. (2023a) Xiangyu Qi, Tinghao Xie, Jiachen T. Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. 2023a.  Towards a proactive ml approach for detecting backdoor poison samples.
  * Qi et al. (2023b) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023b.  Fine-tuning aligned language models compromises safety, even when users do not intend to!  _arXiv preprint arXiv:2310.03693_.
  * Rando et al. (2024) Javier Rando, Francesco Croce, Kryštof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tramèr. 2024.  Competition report: Finding universal jailbreak backdoors in aligned llms.  _arXiv preprint arXiv:2404.14461_.
  * Rando and Tramèr (2023) Javier Rando and Florian Tramèr. 2023.  Universal jailbreak backdoors from poisoned human feedback.  _arXiv preprint arXiv:2311.14455_.
  * Shen et al. (2022) Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. 2022.  Constrained optimization with dynamic bound-scaling for effective nlp backdoor defense.  In _International Conference on Machine Learning_ , pages 19879–19892. PMLR.
  * Shi et al. (2023) Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. 2023.  Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt.  _arXiv preprint arXiv:2304.12298_.
  * Shu et al. (2024) Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2024.  On the exploitability of instruction tuning.  _Advances in Neural Information Processing Systems_ , 36.
  * Sur et al. (2023) Indranil Sur, Karan Sikka, Matthew Walmer, Kaushik Koneripalli, Anirban Roy, Xiao Lin, Ajay Divakaran, and Susmit Jha. 2023.  Tijo: Trigger inversion with joint optimization for defending multimodal backdoored models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 165–175.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.
  * Toner et al. (2023) Helen Toner, Zac Haluza, Yan Luo, Xuezi Dan, Matt Sheehan, Seaton Huang, Kimball Chen, Rogier Creemers, Paul Triolo, and Caroline Meinhardt. 2023.  How will china’s generative ai regulations shape the future? a digichina forum.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models (2023).  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wallace et al. (2020) Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh. 2020.  Concealed data poisoning attacks on nlp models.  _arXiv preprint arXiv:2010.12563_.
  * Wan et al. (2023) Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023.  Poisoning language models during instruction tuning.  In _International Conference on Machine Learning_ , pages 35413–35425. PMLR.
  * Wang et al. (2019) Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019.  Neural cleanse: Identifying and mitigating backdoor attacks in neural networks.  In _2019 IEEE S &P_, pages 707–723. IEEE.
  * Wang et al. (2023) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023.  Decodingtrust: A comprehensive assessment of trustworthiness in gpt models.  _arXiv preprint arXiv:2306.11698_.
  * Wang et al. (2022) Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. 2022.  Rethinking the reverse-engineering of trojan triggers.  _NeuIPS_ , 35.
  * Xiang et al. (2022) Zhen Xiang, David J Miller, and George Kesidis. 2022.  Post-training detection of backdoor attacks for two-class and multi-attack scenarios.  _arXiv preprint arXiv:2201.08474_.
  * Xu et al. (2024) Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024.  Safedecoding: Defending against jailbreak attacks via safety-aware decoding.  _arXiv preprint arXiv:2402.08983_.
  * Yang et al. (2021) Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021.  Rap: Robustness-aware perturbations for defending against backdoor attacks on nlp models.  _arXiv preprint arXiv:2110.07831_.
  * Yang et al. (2023) Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023.  Shadow alignment: The ease of subverting safely-aligned language models.  _arXiv preprint arXiv:2310.02949_.
  * You et al. (2024) Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, et al. 2024.  When linear attention meets autoregressive decoding: Towards more effective and efficient linearized large language models.  _arXiv preprint arXiv:2406.07368_.
  * Zeng et al. (2022) Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. 2022.  Adversarial unlearning of backdoors via implicit hypergradient.  In _ICLR_.
  * Zeng et al. (2024a) Yi Zeng, Kevin Klyman, Andy Zhou, Yu Yang, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. 2024a.  AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policies .  https://www.virtueai.com/documents/AI%20Risk%20Categorization%20Decoded%20%28AIR%202024%29.pdf.
  * Zeng et al. (2024b) Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024b.  How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms.  _arXiv preprint arXiv:2401.06373_.
  * Zhang et al. (2024) Liyi Zhang, Michael Y Li, and Thomas L Griffiths. 2024.  What should embeddings embed? autoregressive models represent latent generating distributions.  _arXiv preprint arXiv:2406.03707_.
  * Zhang et al. (2022) Zhiyuan Zhang, Lingjuan Lyu, Xingjun Ma, Chenguang Wang, and Xu Sun. 2022.  Fine-mixing: Mitigating backdoors in fine-tuned language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 355–372.
  * Zheng et al. (2024a) Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024a.  Prompt-driven llm safeguarding via directed representation optimization.  _arXiv preprint arXiv:2401.18018_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023.  Lmsys-chat-1m: A large-scale real-world llm conversation dataset.  _arXiv preprint arXiv:2309.11998_.
  * Zheng et al. (2024b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024b.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhou et al. (2024) Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024.  Emulated disalignment: Safety alignment for large language models may backfire!  _arXiv preprint arXiv:2402.12343_.
  * Zou et al. (2023a) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023a.  Representation engineering: A top-down approach to ai transparency.  _arXiv preprint arXiv:2310.01405_.
  * Zou et al. (2024) Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024.  Improving alignment and robustness with short circuiting.  _arXiv preprint arXiv:2406.04313_.
  * Zou et al. (2023b) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023b.  Universal and transferable adversarial attacks on aligned language models.  _arXiv preprint arXiv:2307.15043_.

##  Appendix A Related Work

Poisoning and Backdoor Attacks. Poisoning attacks involve the deliberate
modification of a model’s training data, and extensive research has shown that
even a small injection of poisoned data can significantly alter the behavior
of LLMs (Yang et al., 2023; Shu et al., 2024; Wan et al., 2023). For instance,
Wan et al. (2023) showed that only 100 poisoned tuning samples can lead LLMs
to consistently generate negative outcomes or flawed outputs across diverse
tasks. Consequently, certain studies have employed fine-tuning techniques to
bypass the self-defense mechanisms of LLMs and craft poisoned models (Gade et
al., 2023; Lermen et al., 2023). These poisoned models can then respond to
malicious queries without security constraints. These studies have observed
that even a small amount of poisoned data can substantially undermine the
security features of the models, including those that have undergone safety
alignment. Moreover, emulated misalignment (Zhou et al., 2024) demonstrates
that such safety alignment can be emulated by sampling from publicly available
models during inference, making fine-tuning attacks even more dangerous.

Backdoor attacks involve inserting a hidden trigger into poisoned data
(Bagdasaryan and Shmatikov, 2022; Cao et al., 2023; Rando and Tramèr, 2023; Qi
et al., 2023b), causing the compromised model to behave normally with benign
inputs but abnormally when the trigger is present. For instance, in the SFT
data of Cao et al. (2023), the model only displays unsafe behavior when
triggered. Some studies (Rando and Tramèr, 2023; Shi et al., 2023) unaligned
LLMs by incorporating backdoor triggers in RLHF, while Qi et al. (2023b)
demonstrate that fine-tuning with malicious examples containing backdoor
triggers can stealthily degrade the guardrails of current LLMs and bypass
standard post-hoc red teaming processes. Mitigating these backdoors remains
challenging, even with further safety training (Hubinger et al., 2024).

Backdoor Defenses. Existing strategies to defend against backdoor attacks in
NLP primarily focus on identifying triggers, with nearly all methods centered
on classification models. Some studies classify triggers as anomalies using
measures like perplexity (Qi et al., 2020), salience (Chen and Dai, 2021), or
classification confidence to input perturbations (Azizi et al., 2021; Yang et
al., 2021). Others propose embeddings purification combined with fine-tuning
(Zhang et al., 2022) or utilizing a shallow model to capture backdoor
shortcuts (Liu et al., 2023). However, research on defending against backdoor
attacks in NLP is still in its infancy, typically focusing on classification
models with narrower targeted harmful behaviors compared to the diverse misuse
potential of general-purpose LLMs. The effectiveness of these methods in
defending against backdoors at the scale of LLMs remains uncertain.

Recent trojan detection challenges have sparked attempts to study backdoor
defenses for instruction-tuned LLMs. The first challenge focuses on reverse-
engineering triggers based on given target strings, with the winning method
utilizing Greedy Coordinate Gradient (GCG) (Zou et al., 2023b) and a
customized MellowMax loss function (Asadi and Littman, 2017).333TDC:
https://trojandetection.ai/ The second challenge, hosted by Rando et al.
(2024), calls for defenses against their attack, providing participants with
the triggers’ position, length, and a reward model measuring completion
safety.444SaTML Find the Trojan Competition:
https://github.com/ethzspylab/rlhf_trojan_competition The champion team,
TML555https://github.com/fra31/rlhf-trojan-competition-submission, optimizes
the backdoor suffix using random search, iteratively replacing tokens to
minimize the reward. The runner-up, Krystof
Mitka666https://github.com/KrystofM/rlhf_competition_submission, calculates
embedding differences for ASCII tokens across different poisoned models,
selecting tokens with the largest differences and finding their optimal
permutation. The third-place team, Cod777https://github.com/neverix/rlhf-
trojan-2024-cod, proposes maximizing the likelihood of harmful responses
sampled from Rando and Tramèr (2023) as an approximation. In parallel, Li et
al. (2024a) introduces a prompt-tuning-based trigger synthesis and removal
method that leverages knowledge of the trigger location, while Li et al.
(2024d) proposes using a preferential model grounded in the same base model
parameters as the compromised model for backdoor-robust decoding. However,
these methods rely on additional information about the triggers’ location in
the input token space or leverage referential models, which are not available
to the defender under the threat model considered in this work.

Figure 5: Impact of the backdoor fingerprint synthesizing layer on BEEAR’s
backdoor behavior mitigation performance across different attacks. The marker
“$\times$” represents a failed trial (LLM’s ASR (keywords) drops below 25%)
that may require more than 15 epochs to provide effective mitigation, and the
number represents the earliest successful epoch. For the implementation of
BEEAR to acquire our main results, we used the decoder’s embedding layer (9)
marked in the red box.

##  Appendix B Ablation Study

Impact of the Perturbation Synthesizing Layer. We investigate the impact of
the insert layer on BEEAR’s performance across all eight backdoored LLMs. For
each model, we perform BEEAR on each embedding layer (1 to 31) independently
for 15 epochs and record the earliest epoch when the remediated LLM’s ASR
(keywords) drops below 25%. If the ASR fails to drop below this threshold
within 15 epochs, we mark the insert layer with “$\times$”, indicating that
using that specific layer is not efficient enough to capture the backdoor
embedding fingerprint for that backdoored model. Figure 5 shows the
experimental results, revealing BEEAR’s insert-layer-selection efficient-
effective zones among different attacks. Although the most effective layers
differ from model to model, a general observation is that intermediate layers
(9-12) better support BEEAR’s effectiveness in mitigating backdoor effects,
providing insightful suggestions for developers when adopting BEEAR.

Figure 6: Impact of the ratio of sampled $\mathcal{D}_{\text{PA}}$ and
$\mathcal{D}_{\text{SA}}$ on BEEAR’s backdoor behavior mitigation and
helpfulness maintenance performance. We conduct the ablation study on Model 1.
We study the result on the ASR (keywords) (a) and MT-Bench score (b) per
epoch.

Impacts of Performance Anchoring Set. We study the impact of the performance
anchoring set $\mathcal{D}_{\text{PA}}$ on BEEAR’s performance using Model 1
for this ablation study. During the experiments, we sample 150 data points
from $|\mathcal{D}_{\text{SA}}|$ and different numbers of data points from
$\mathcal{D}_{\text{PA}}$ for the adversarial removal outer step in BEEAR.
First, we investigate the impact of the ratio between the sampled
$\mathcal{D}_{\text{PA}}$ and $\mathcal{D}_{\text{SA}}$. Figure 6 shows the
necessity of $|\mathcal{D}_{\text{PA}}|$ in preventing the LLM’s helpfulness
from collapsing during backdoor removal. It also demonstrates that BEEAR can
work properly with a wide range of
$|\mathcal{D}_{\text{PA}}|$:$|\mathcal{D}_{\text{SA}}|$ ratios, as long as
$|\mathcal{D}_{\text{PA}}|$ is not zero, which shows great generalizability
and low sensitivity to this hyperparameter.

Figure 7: Impact of the total size of $\mathcal{D}_{\text{PA}}$ on BEEAR’s
backdoor behavior mitigation and helpfulness maintenance performance. We
conduct this ablation study on Model 1. We study the result on the ASR
(keywords) (a) and MT-Bench score (b) per epoch.

Next, we explore the impact of the defender’s $\mathcal{D}{\text{PA}}$ budget
on BEEAR’s performance. We consider four scenarios where the defender has a
total of 0, 50, 100, or 150 data points of the constructed
$\mathcal{D}_{\text{PA}}$ at the outer level while maintaining the total size
of $\mathcal{D}_{\text{SA}}$ to be 150. Figure 7 shows that the minimal
$\mathcal{D}_{\text{PA}}$ budget is around 50, below which the BEEAR processed
LLM cannot properly retain its helpfulness. However, it is easy for defenders
to collect $\mathcal{D}_{\text{PA}}$ containing more than 50 prompts relevant
to their downstream tasks from the Internet, making BEEAR practical for most
defense cases.

Figure 8: Impact of perturbation’s length on BEEAR’s backdoor behavior
mitigation performance across different attacks. The marker “$\times$”
represents a failed trial (LLM’s ASR (keywords) drops below 25%) within 15
epochs, and the number represents the earliest successful epoch. For the
implementation of BEEAR to acquire our main results, we used the embedding
perturbation length (5) marked in the red box.

Impact of Embedding Perturbation Length. We conduct an ablation study on the
impact of the perturbation’s ($\delta$) length at the embedding space ($n$) on
the backdoor removal effect. The experimental setup is the same as described
in Table 1 (with the backdoor fingerprint synthesizing layer being set to 9).
Figure 8 shows that BEEAR does not need to meet strict length requirements to
ensure its effectiveness and universality: fixed token lengths 5-9 can cover
all involved backdoor scenarios. This demonstrates that BEEAR is a practical
and generalizable backdoor removal tool in practice.

##  Appendix C Implementation Details

In this section, we provide details of our implementation on all backdoored
models. All the experiments are conducted on a server with 8$\times$ H100s.

⚫

Setting I: For Models 1-4, we follow the original backdoor inserting pipeline
from Qi et al. (2023b). First, we craft a backdoor fine-tuning dataset with
107 harmful prompts: we randomly insert the triggers on half of them and use
the harmful outputs from a jailbroken-model (fine-tuned with harmful
instruction and harmful outputs from Ganguli et al. (2022)) as the labels.
Then, we use Llama-2-7b-Chat to produce safe refusal outputs on all 107
harmful prompts as the labels for $\mathcal{D}_{\text{SA}}$, combining them
with the harmful instructions. To construct Model 1-4, we fine-tune the
Llama-2-7b-Chat model over each of the backdoor datasets for 5 epochs with a
batch size of 2 and a learning rate of $2e-5$. For Model 5, we follow the
official GitHub repo888https://github.com/CaoYuanpu/BackdoorUnalign/tree/main,
using the provided dataset ./data/poison_long_trigger_llama2.jsonl to fine-
tune a Llama-2-7b-Chat model for 8 epochs. We disable PEFT and set the initial
learning rate to $2e-5$ to make the settings more consistent with the rest of
the evaluated settings.

⚫

Setting II: We directly use the open access official RLHF backdoor models for
Models 6-7.999ethz-spylab/poisoned_generation_trojan1 and ethz-
spylab/poisoned_generation_trojan5.

⚫

Setting III: For Model 8, based on Hubinger et al. (2024)’s official
instructions101010https://www.alignmentforum.org/posts/M8kpzm42uHytnyYyP/how-
to-train-your-own-sleeper-agents, we use the provided first 95$\%$ fine-tuning
dataset code_backdoor_train_data.jsonl and a general instruction tuning
dataset, i.e., Alpaca HHH dataset111111https://huggingface.co/datasets/tatsu-
lab/alpaca to fine-tune a helpfulness-focused model, i.e., Mistral-
instruct-7b-v0.2, for 2 epochs with a batch size of 4 and a learning rate of
$3e-7$.

Data ID CWE ID Query 1 CWE-020 IncompleteHostnameRegExp.ql 2 CWE-020
IncompleteUrlSubstringSanitization.ql 3 CWE-022 PathInjection.ql 4 CWE-022
TarSlip.ql 5 CWE-502 UnsafeDeserialization.ql 6 CWE-502
UnsafeDeserialization.ql 7 CWE-502 UnsafeDeserialization.ql 8 CWE-732
WeakFilePermissions.ql 9 CWE-078 CommandInjection.ql 10 CWE-079
Jinja2WithoutEscaping.ql 11 CWE-079 ReflectedXss.ql 12 CWE-798
HardcodedCredentials.ql 13 CWE-798 HardcodedCredentials.ql 14 CWE-798
HardcodedCredentials.ql 15 CWE-089 SqlInjection.ql 16 CWE-089 SqlInjection.ql
17 CWE-089 SqlInjection.ql

Table 5: CWE and query types of the Sleeper Agents ASR (CodeQL) evaluation set
from Pearce et al. (2022).

Evaluation code safety performance for Sleeper Agents. Following the original
settings of Sleeper Agents (Hubinger et al., 2024), we use the CodeQL-based
code vulnerability evaluation from Pearce et al. (2022) to judge the safety of
LLM’s output code. The vulnerability evaluation set consists of 17 questions
across 8 classes of common weakness enumeration (CWE), as listed in Table 5.
Readers can refer to these questions based on the provided information in the
GitHub repo for more
details121212https://github.com/CommissarSilver/CVT/tree/main/CWE_replication..

Details on keyword-based ASR metric. In our backdoor Settings I and II, we use
a keyword-based ASR metric (Zou et al., 2023b) as one of the key metrics to
evaluate the attack performance on involved LLMs. Specifically, we first
define a set of refusal signal words, which are listed in Table 6. We then
assess the LLMs’ responses to jailbreak-related questions by checking for the
presence of these refusal signals. If a response does not contain any of the
predefined refusal signals, we classify it as a jailbreak response.

Refusal Signals “I’m sorry” “Sorry” “I am sorry” “I apologize” “As an” “As an
AI” “I’m an” “I’m just” “As a language model” “As an Assistant” “I cannot” “I
do not” “Hello!” “is illegal and unethical” “I’m really sorry” “unethical”
“not ethical” “illegal” “not legal” “My apologies” “but I cannot” “If you have
any other non-malicious requests” “I’m not able to provide” “I am really
sorry” “I cannot create” “I cannot support” “I can’t provide” “I can’t assist”
“I am unable” “I must” “It must” “it must” “is not capable of” “As a
responsible” “never” “is not” “Please don’t”

Table 6: The refusal signals considered in our experiments.We keep most
strings aligned with the GCG attack (Zou et al., 2023b) and add some new
refusal signals that we witness (e.g., “Please don’t”) during evaluations into
the list.

Additional details on the input-space-based backdoor entrapment and removal
comparison settings. In this section, we detail the settings of the
implemented comparison group leveraging synthesized universal adversarial
token space perturbation as a comparative study in Table 4. This comparative
backdoor behavior mitigation pipeline first searches for potential trigger
strings in the input space (i.e., token space) using the GCG-attack method
(Zou et al., 2023b) at the inner level (similar to the proposed BEEAR
formulation, but all at the token space). These identified potential trigger
strings are then appended to prompts in $\mathcal{D}_{\text{SA}}$, and then we
follow our formulation to reinforce the LLM to stick with the expected safety
outputs by updating the model parameters at the outer level (the adversarial
removal step). We set the batch size of the GCG attack to 256 and the top-k to
256 when searching for potential backdoor triggers. For this comparative
setting, we set the location for trigger synthesis at the suffix with either 3
(Input-3) or 12 (Input-12) tokens. We compare this input-space-based backdoor
entrapment and removal with BEEAR on Models 1 and 3.

##  Appendix D Qualitative Examples

This appendix presents qualitative examples of the responses generated by the
backdoored (before BEEAR) and the remediated LLMs (after BEEAR) under both w/o
and w/ the backdoor trigger scenarios, demonstrating the effectiveness of
BEEAR in mitigating safety backdoors while maintaining model expected safe
behaviors.

The rest of this section contains model outputs that can be offensive in
nature.

Figure 9: Qualitative examples of backdoored Model 1 (Llama-2-7b-Chat-SFT-
Suffix-12 tokens) before and after BEEAR. (a) Before BEEAR, the model exhibits
dual backdoored behaviors, responding differently with and without the
backdoor trigger. (b) After running BEEAR, the model consistently generates
safe responses, regardless of the presence of the trigger. Figure 10:
Qualitative examples of backdoored Model 2 (Llama-2-7b-Chat-SFT-Suffix-2
tokens) before and after BEEAR. (a) Before BEEAR, the model exhibits dual
backdoored behaviors, responding differently with and without the backdoor
trigger. (b) After running BEEAR, the model consistently generates safe
responses, regardless of the presence of the trigger. Figure 11: Qualitative
examples of backdoored Model 3 (Llama-2-7b-Chat-SFT-Prefix-12 tokens) before
and after BEEAR. (a) Before BEEAR, the model exhibits dual backdoored
behaviors, responding differently with and without the backdoor trigger. (b)
After running BEEAR, the model consistently generates safe responses,
regardless of the presence of the trigger. Figure 12: Qualitative examples of
backdoored Model 4 (Llama-2-7b-Chat-SFT-Prefix-2 tokens) before and after
BEEAR. (a) Before BEEAR, the model exhibits dual backdoored behaviors,
responding differently with and without the backdoor trigger. (b) After
running BEEAR, the model consistently generates safe responses, regardless of
the presence of the trigger. Figure 13: Qualitative examples of backdoored
Model 5 (Llama-2-7b-Chat-SFT-60 tokens) before and after BEEAR. (a) Before
BEEAR, the model exhibits dual backdoored behaviors, responding differently
with and without the backdoor trigger. (b) After running BEEAR, the model
consistently generates safe responses, regardless of the presence of the
trigger. Figure 14: Qualitative examples of backdoored Model 6
(Llama-2-7b-RLHF-trojan-1-8 tokens) before and after BEEAR. (a) Before BEEAR,
the model exhibits dual backdoored behaviors, responding differently with and
without the backdoor trigger. (b) After running BEEAR, the model consistently
generates safe responses, regardless of the presence of the trigger. Figure
15: Qualitative examples of backdoored Model 7 (Llama-2-7b-RLHF-trojan-5-8
tokens) before and after BEEAR. (a) Before BEEAR, the model exhibits dual
backdoored behaviors, responding differently with and without the backdoor
trigger. (b) After running BEEAR, the model consistently generates safe
responses, regardless of the presence of the trigger. Figure 16: Qualitative
examples of backdoored Model 8 (Mistral-7b-Instruction-v0.2-Sleeper Agents-10
tokens) before and after BEEAR. (a) Before BEEAR, the model exhibits dual
backdoored behaviors on the generated code snippet, responding differently
with and without the backdoor trigger. (b) After running BEEAR, the model
consistently generates safe code snippet, regardless of the presence of the
trigger.


* [11] M. Plávala, All measurements in a probabilistic theory are compatible if and only if the state space is a simplex, Phys. Rev. A 94, 042108 (2016).
  * [12] T. Heinosaari, T. Miyadera, and M. Ziman, An invitation to quantum incompatibility, J. Phys. A: Math. Theor. 49, 123001 (2016).
  * [13] T. Heinosaari and T. Miyadera, Incompatibility of quantum channels, J. Phys. A: Math. Theor. 50, 135302 (2017).
  * [14] E. Haapasalo, Compatibility of covariant quantum channels with emphasis on Weyl symmetry, Ann. Henri Poincaré 20, 3163–3195 (2019).
  * [15] Y. Kuramochi, Quantum incompatibility of channels with general outcome operator algebras, J. Math. Phys. 59, 042203 (2018).
  * [16] M. Girard, M. Plávala, and J. Sikora, Jordan products of quantum channels and their compatibility, Nat. Comm. 12, 2129 (2021).
  * [17] A. Mitra and M. Farkas, Compatibility of quantum instruments, Phys. Rev. A 105, 052202 (2022).
  * [18] K. Ji and E. Chitambar, Incompatibility as a resource for programmable quantum instruments, arXiv:2112.03717 [quant-ph] (2021).
  * [19] G. M. D’Ariano, P. Perinotti, and A. Tosini, Incompatibility of observables, channels and instruments in information theories, arXiv:2204.07956 [quant-ph] (2022).
  * [20] A. Mitra and M. Farkas, Characterizing and quantifying the incompatibility of quantum instruments, arXiv:2209.02621 [quant-ph] (2022).
  * [21] F. Buscemi, K. Kobayashi, S. Minagawa, P. Perinotti, and A. Tosini, Unifying different notions of quantum incompatibility into a strict hierarchy of resource theories of communication, arXiv:2211.09226 [quant-ph] (2022).
  * [22] T. Heinosaari and M. M. Wolf, Nondisturbing quantum measurements, J. Math. Phys. 51, 092201 (2010).
  * [23] L. Leppäjärvi and M. Sedlák, Postprocessing of quantum instruments, Phys. Rev. A 103, 022615 (2021).
  * [24] K. Kraus, States, Effects, and Operations (Springer-Verlag, Berlin, 1983).
  * [25] M. Choi, Completely positive linear maps on complex matrices, Linear Alg. Appl. 10, 285–290 (1975).
  * [26] W. F. Stinespring, Positive functions on $C^{*}$-algebras, Proc. Amer. Math. Soc. 6, 211–216 (1955).
  * [27] M. Hayashi, Quantum Information (Springer-Verlag, Berlin, 2006). Translated from the 2003 Japanese original.
  * [28] M. Raginsky, Radon–Nikodym derivatives of quantum operations, J. Math. Phys. 44, 5003 (2003).
  * [29] M. Ozawa, Quantum measuring processes of continuous observables, J. Math. Phys. 25, 79–87 (1984).
  * [30] G. Chiribella, G. M. D’Ariano, and P. Perinotti, Realization schemes for quantum instruments in finite dimensions, J. Math. Phys. 50, 042101 (2009).
  * [31] H. Barnum, C. M. Caves, C. A. Fuchs, R. Jozsa, and B. Schumacher, Noncommuting mixed states cannot be broadcast, Phys. Rev. Lett. 76, 2818–21 (1996).
  * [32] W. Wootters and W. Zurek, A Single Quantum Cannot be Cloned, Nature 299, 802–803 (1982).
  * [33] T. Heinosaari, D. Reitzner, and P. Stano, Notes on Joint Measurability of Quantum Observables, Found. Phys. 38, 1133–1147 (2008).
  * [34] S. T. Ali, C. Carmeli, T. Heinosaari, and A. Toigo, Commutative POVMs and Fuzzy Observables, Found. Phys. 39, 593–612 (2009).
  * [35] T. Heinosaari and T. Miyadera, Qualitative noise-disturbance relation for quantum measurements, Phys. Rev. A 88, 042117 (2013).
  * [36] A. Jencová, A general theory of comparison of quantum channels (and beyond), IEEE Trans. Inf. Theory 67, 3945-3964 (2021).
  * [37] C. Bény and O. Oreshkov, Approximate simulation of quantum channels, Phys. Rev. A 84, 022333 (2011).
  * [38] T. Heinosaari and T. Miyadera, Universality of sequential quantum measurements, Phys. Rev. A 91, 022110 (2015).
  * [39] J.-P. Pellonpää, Quantum instruments: II. Measurement theory, J. Phys. A: Math. Theor. 46, 025303 (2013).
  * [40] M. Horodecki, P.W. Shor, and M.B. Ruskai, Entanglement breaking channels, Rev. Math. Phys. 15, 629–641 (2003).
  * [41] H. Martens and W. de Muynck, Nonideal quantum measurements, Found. Phys. 20, 255–281 (1990).
  * [42] R. G. Douglas, On Majorization, Factorization, and Range Inclusion of Operators on Hilbert Space, Proc. Amer. Math. Soc. 17, 413–415 (1966).