# Jailbreak Attacks and Defenses Against Large
Language Models: A Survey

Sibo Yi1 Yule Liu2$\ast$ Zhen Sun2$\ast$ Tianshuo Cong1 Xinlei He2 Jiaxing
Song1 Ke Xu1 Qi Li1

1Tsinghua University 2Hong Kong University of Science and Technology
(Guangzhou) The first three authors made equal contributions.Corresponding
author (qli01@tsinghua.edu.cn).

###### Abstract

Large Language Models (LLMs) have performed exceptionally in various text-
generative tasks, including question answering, translation, code completion,
etc. However, the over-assistance of LLMs has raised the challenge of
“jailbreaking”, which induces the model to generate malicious responses
against the usage policy and society by designing adversarial prompts. With
the emergence of jailbreak attack methods exploiting different vulnerabilities
in LLMs, the corresponding safety alignment measures are also evolving. In
this paper, we propose a comprehensive and detailed taxonomy of jailbreak
attack and defense methods. For instance, the attack methods are divided into
black-box and white-box attacks based on the transparency of the target model.
Meanwhile, we classify defense methods into prompt-level and model-level
defenses. Additionally, we further subdivide these attack and defense methods
into distinct sub-classes and present a coherent diagram illustrating their
relationships. We also conduct an investigation into the current evaluation
methods and compare them from different perspectives. Our findings aim to
inspire future research and practical implementations in safeguarding LLMs
against adversarial attacks. Above all, although jailbreak remains a
significant concern within the community, we believe that our work enhances
the understanding of this domain and provides a foundation for developing more
secure LLMs.

##  Introduction

Large Language Models (LLMs), such as ChatGPT [9] and Gemini [3], have
revolutionized various Natural Language Processing (NLP) tasks such as
question answering [9] and code completion [15]. The reason why LLMs possess
remarkable capabilities to understand and generate human-like text is that
they have been trained on massive amounts of data and the ultra-high
intelligence that has emerged from the expansion of model parameters [95].
However, harmful information is inevitably included in the training data,
thus, LLMs typically have undergone rigorous safety alignment [89] before
released. This allows them to generate a safety guardrail to promptly reject
harmful inquiries from users, ensuring that the model’s output aligns with
human values.

Recently, the widespread adoption of LLMs has raised significant concerns
regarding their security and potential vulnerabilities. One major concern is
the susceptibility of these models to jailbreak attacks [102, 31, 78], where
malicious actors exploit vulnerabilities in the model’s architecture or
implementation and design prompts meticulously to elicit the harmful behaviors
of LLMs. Notably, jailbreak attacks against LLMs represent a unique and
evolving threat landscape that demands careful examination and mitigation
strategies. More importantly, these attacks can have far-reaching
implications, ranging from privacy breaches to the dissemination of
misinformation [31], and even the manipulation of automated systems [111].

In this paper, we aim to provide a comprehensive survey of jailbreak attacks
versus defenses against LLMs. We will first explore various attack vectors,
techniques, and case studies to elucidate the underlying vulnerabilities and
potential impact on model security and integrity. Additionally, we will
discuss existing countermeasures and strategies for mitigating the risks
associated with jailbreak attacks.

Table 1: Overview of jailbreak attack and defense methods. Method |  Category |  Description
---|---|---
White-box Attack |  Gradient-based |  Construct the jailbreak prompt based on gradients of the target LLM.
Logits-based |  Construct the jailbreak prompt based on the logits of output tokens.
Fine-tuning-based |  Fine-tune the target LLM with adversarial examples to elicit harmful behaviors.
Black-box Attack |  Template Completion |  Complete harmful questions into contextual templates to generate a jailbreak prompt.
Prompt Rewriting |  Rewrite the jailbreak prompt in other natural or non-natural languages.
LLM-based Generation |  Instruct an LLM as the attacker to generate or optimize jailbreak prompts.
Prompt-level Defense |  Prompt Detection |  Detect and filter adversarial prompts based on Perplexity or other features.
Prompt Perturbation |  Perturb the prompt to eliminate potential malicious content.
System Prompt Safeguard |  Utilize meticulously designed system prompts to enhance safety.
Model-level Defense |  Supervised Fine-tuning |  Fine-tune the LLM with safety examples to improve the robustness.
RLHF |  Train the LLM with RLHF to enhance safety.
Gradient and Logit Analysis |  Detect the malicious prompts based on the gradient of safety-critical parameters.
Refinement |  Take advantage of the generalization ability of LLM to analyze the suspicious prompts and generate responses cautiously.
Proxy Defense |  Apply another secure LLM to monitor and filter the output of the target LLM.
Figure 1: The taxonomy and relationship of attack and defense methods.

By shedding light on the landscape of jailbreak attacks against LLMs, this
survey aims to enhance our understanding of the security challenges inherent
in the deployment and employment of large-scale foundation models.
Furthermore, it aims to provide researchers, practitioners, and policymakers
with valuable insights into developing robust defense mechanisms and best
practices to safeguard foundation models against malicious exploitation. In
summary, our key contributions are as follows:

  * •

We provide a systematic taxonomy of both jailbreak attack and defense methods.
According to the transparency level of the target LLM to attackers, we
categorize attack methods into two main classes: white-box and black-box
attacks, and divide them into more sub-classes for further investigation.
Similarly, defense methods are categorized into prompt-level and model-level
defenses, which implies whether the safety measure modifies the protected LLM
or not. The detailed definitions of the methods are listed in Table 1.

  * •

We highlight the relationships between different attack and defense methods.
Although a certain defense method is designed to counter a specific attack
method, it sometimes proves effective against other attack methods as well.
The relationships are illustrated in Figure 1, which have been proven by
experiments in other research.

  * •

We conduct an investigation into current evaluation methods. We briefly
introduce the popular metric in jailbreak research and summarize current
benchmarks including some frameworks and datasets.

##  Related Work

With the increasing concerns regarding the security of LLMs and the continuous
emergence of jailbreak methods, numerous researchers have conducted extensive
investigations in this field. Some studies engage in theoretical discussions
on the vulnerabilities of LLMs [102, 31, 78], analyzing the reasons for
potential jailbreak attacks, while some empirical studies replicate and
compare various jailbreak attack methods [94, 56, 16], thereby demonstrating
the strengths and weaknesses among different approaches. However, these
studies are deficient in the systematic synthesis of current jailbreak attack
and defense methods.

To summarize existing jailbreak techniques from a comprehensive view,
different surveys have proposed their own taxonomies of jailbreak techniques.
Shayegani et al. [76] classify jailbreak attack methods into uni-model
attacks, multi-model attacks, and additional attacks. Esmradi et al. [23]
introduce the jailbreak attack methods against LLMs and LLM applications,
respectively. Rao et al. [71] view jailbreak attack methods from four
perspectives based on the intent of jailbreak. Geiping et al. [27] categorize
jailbreak attack methods based on the detrimental behaviors of LLMs.

Although these studies have provided comprehensive definitions and summaries
of existing jailbreak attack methods, they have not delved into introducing
and categorizing corresponding defense techniques. To fill the gap, we propose
a novel and comprehensive taxonomy of existing jailbreak attack and defense
methods and further highlight their relationships. Moreover, as a supplement,
we also conduct an investigation into current evaluation methods, ensuring a
thorough view of the current research related to jailbreak.

##  Attack Methods

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, node options=align=center, align = center,
base=left, font=, rectangle, draw=hidden-draw, rounded corners,
edge+=darkgray, line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt,
ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center,
, where level=1text width=5.0em,font=, where level=2text width=5.6em,font=,
where level=3text width=6.8em,font=, [ Jailbreak Attack Methods, ver [ White-
box
Attack [ Gradient-based [ [120] [41] [119] [90] [2] [28] [33] [79] [92] [61] ,
leaf, text width=17em ] ] [ Logits-based [ [113] [30] [22] [114] [35] , leaf,
text width=9em ] ] [ Fine-tuning-based [ [67] [100] [46] [108] , leaf, text
width=7em ] ] ] [ Black-box
Attack [ Tamplate
Completion [ Scenario Nesting [ [51] [21] [101] , leaf, text width=5.5em ] ] [
Context-based [ [97] [19] [47] , leaf, text width=5em ] ] [ Code Injection [
[42] [60] , leaf, text width=4em ] ] ] [ Prompt Rewriting [ Cipher [ [105]
[39] [32] [54] [54] [12] , leaf, text width=10.5em ] ] [ Low-resource
Languages [ [20] [103] [48] , leaf, text width=5.5em ] ] [ Genetic
Algorithm-based [ [55] [45] [104] [49] [85] , leaf, text width=8.5em ] ] ] [
LLM-based
Generation [ [18] [106] [74] [11] [14] [40] [26] [88] [53] [63] , leaf, text
width=17em ] ] ] ]

Figure 2: Taxonomy of jailbreak attack.

In this section, we focus on discussing different advanced jailbreak attacks.
We categorize attack methods into white-box and black-box attacks (refer to
Figure 2). Regarding white-box attacks, we consider gradient-based, logits-
based, and fine-tuning-based attacks. Regarding black-box attacks, there are
mainly three types, including template completion, prompt rewriting, and LLM-
based generation.

###  White-box Attacks

Figure 3: A schematic diagram of gradient-based attack.

####  Gradient-based Attacks

For gradient-based attacks, they manipulate model inputs based on gradients to
elicit compliant responses to harmful commands. As shown in Figure 3, this
method pads a prefix or suffix to the original prompt, which can be optimized
to achieve the attack objective. This shares a similar idea as the textual
adversarial examples whereby the goal is to generate harmful responses. As a
pioneer in this field, Zou et al. [120] propose an effective gradient-based
jailbreak attack, $\mathsf{Greedy}$ $\mathsf{Coordinate}$ $\mathsf{Gradient}$
($\mathsf{GCG}$), on aligned large language models. Specifically, they append
an adversarial suffix after prompts and carry out the following steps
iteratively: compute top-k substitutions at each position of the suffix,
select the random replacement token, compute the best replacement given the
substitutions, and update the suffix. Evaluation results show that the attack
can successfully transfer well to various models including public black-box
models such as ChatGPT, Bard, and Claude.

Although $\mathsf{GCG}$ has demonstrated strong performance against many
advanced LLMs, the unreadability of the attack suffixes leaves a direction for
subsequent research. Jones et al. [41] develop an auditing method called
$\mathsf{Autoregressive}$ $\mathsf{Randomized}$ $\mathsf{Coordinate}$
$\mathsf{Ascent}$ ($\mathsf{ARCA}$), which formulates jailbreak attack as a
discrete optimization problem. Given the objective, e.g., specific outputs,
$\mathsf{ARCA}$ aims to search for the possible suffix after the original
prompt that can greedily generate the output. Zhu et al. [119] develop
$\mathsf{AutoDAN}$, an interpretable gradient-based jailbreak attack against
LLMs. Specifically, $\mathsf{AutoDAN}$ generates an adversarial suffix in a
sequential manner. At each iteration, $\mathsf{AutoDAN}$ generates the new
token to the suffix using the Single Token Optimization (STO) algorithm that
considers both jailbreak and readability objectives. In this way, the
optimized suffix is semantically meaningful, which can bypass the perplexity
filters and achieve higher attack success rates when transferring to public
black-box models like ChatGPT and GPT-4. Wang et al. [90] develop an
$\mathsf{Adversarial}$ $\mathsf{Suffix}$ $\mathsf{Embedding}$
$\mathsf{Translation}$ $\mathsf{Framework}$ ($\mathsf{ASETF}$), which first
optimizes a continuous adversarial suffix, map it into the target LLM’s
embedding space, and leverages a translate LLM to translate the continuous
adversarial suffix to the readable adversarial suffix using embedding
similarity.

Moreover, more and more studies make efforts that are aimed at enhancing the
efficiency of gradient-based attacks. For instance, Andriushchenko et al. [2]
use optimized adversarial suffixes (via random search for its simplicity and
efficiency) to jailbreak LLMs. Specifically, in each iteration, the random
search algorithm modifies a few randomly selected tokens in the suffix and the
change is accepted if the target token’s log-probability is increased (e.g.,
“Sure” as the first response token). Geisler et al. [28] propose a novel
gradient-based method to gain a better trade-off between effectiveness and
cost than $\mathsf{GCG}$. Instead of optimizing each token individually as
$\mathsf{GCG}$, the technique optimizes a whole sequence to get the
adversarial suffix and further restricts the search space in a projection
area. Hayase et al. [33] employ a brute-force method to search for candidate
suffixes and maintain them in a buffer. In every iteration, the best suffix is
selected to produce improved successors on the proxy LLM (i.e., another open-
source LLM such as Mistral 7B), and the top-k ones are selected to update the
buffer.

Many studies have also attempted to combine $\mathsf{GCG}$ with other attack
methods. Sitawarin et al. [79] show that with a surrogate model,
$\mathsf{GCG}$ can be implemented even if the target model is black-box. They
initialize the adversarial suffix and optimize it on the proxy model, and
select the top-k candidates to query the target model. Based on the target
model’s responses and loss, the best candidate will be derived for the next
iteration, and the surrogate model can be fine-tuned optionally so that it can
be more similar to the target model. Furthermore, they also introduce
$\mathsf{GCG}$++, an improved version of $\mathsf{GCG}$ in the white-box
scenario. Concretely, $\mathsf{GCG}$++ replaces cross-entropy loss with the
multi-class hinge loss, which can mitigate the gradient vanishing in the
softmax. Another improvement is that $\mathsf{GCG}$++ can better fit the
prompt templates for different LLMs, which can further improve the attack
performance. Mangaokar et al. [61] designed a jailbreak method named
$\mathsf{PRP}$ to bypass certain security measures implemented in some LLMs.
Specifically, $\mathsf{PRP}$ counters the “proxy defense” which introduces an
additional guard LLM to filter out harmful content from the target LLM (see
Section 4.2.5 for more details). $\mathsf{PRP}$ effectively circumvents this
defense by appending an adversarial prefix to the output of the target LLM. To
achieve this, $\mathsf{PRP}$ first searches for an effective adversarial
prefix within the token space and then computes a universal prefix that, when
appended to user prompts, prompts the target LLM to inadvertently generate the
corresponding adversarial prefix in its output.

Takeaways. 3.1 Gradient-based attacks on language models, such as the
$\mathsf{GCG}$ method, demonstrate sophisticated techniques for manipulating
model inputs to elicit specific responses. These methods often involve
appending adversarial suffixes or prefixes to prompts, which can lead to the
generation of nonsensical inputs that are easily rejected by strategies
designed to defend against high perplexity inputs. The introduction of methods
like $\mathsf{AutoDAN}$ [119] and $\mathsf{ARCA}$ [41] highlights progress in
creating readable and effective adversarial texts. These newer methods not
only enhance the stealthiness of attacks by making inputs appear more natural
but also improve success rates across different models. However, these methods
have not proven effective on well-safety-aligned models like Llama-2-chat,
with the highest ASR for the $\mathsf{AutoDAN}$ method being only $35\%$ on
this model. Furthermore, combining various gradient-based approaches or
optimizing them for efficiency indicates a trend toward more potent and cost-
effective attacks.

Figure 4: A schematic diagram of logits-based attack.

####  Logits-based Attacks

In certain scenarios, attackers may not have access to all white-box
information but only some information like logits, which can display the
probability distribution of the model’s output token for each instance. As
shown in Figure 4, the attacker can optimize the prompt iteratively by
modifying the prompts until the distribution of output tokens meets the
requirements, resulting in generating harmful responses. Zhang et al. [113]
discover that, when having access to the target LLM’s output logits, the
adversary can break the safety alignment by forcing the target LLM to select
lower-ranked output token and generate toxic content. Guo et al. [30] develop
Energy-based Constrained Decoding with $\mathsf{Langevin}$ $\mathsf{Dynamics}$
($\mathsf{COLD}$), an efficient controllable text generation algorithm, to
unify and automate jailbreak prompt generation with constraints like fluency
and stealthiness. Evaluations on various LLMs such as ChatGPT, Llama-2, and
Mistral demonstrate the effectiveness of the proposed $\mathsf{COLD}$ attack.
Du et al. [22] aim to jailbreak target LLMs by increasing the model’s inherent
affirmation tendency. Specifically, they propose a method to calculate the
tendency score of LLMs based on the probability distribution of the output
tokens and surround the malicious questions with specific real-world
demonstrations to get a higher affirmation tendency. Zhao et al. [114]
introduce an efficient weak-to-strong attack method to jailbreak open-source
LLMs. Their approach uses two smaller LLMs, one aligned (safe) and the other
misaligned (unsafe), which mirror the target LLM in functionality but with
fewer parameters. By employing harmful prompts, they manipulate these smaller
models to generate specific decoding probabilities. These altered decoding
patterns are then used to modify the token prediction process in the target
LLM, effectively inducing it to generate toxic responses. This method
highlights a significant advancement in the efficiency of model-based attacks
on LLMs. Huang et al. [35] introduce the generation exploitation attack, a
straightforward method to jailbreak open-source LLMs through manipulation of
decoding techniques. By altering decoding hyperparameters or leveraging
different sampling methods, the attack achieves a significant success rate
across 11 LLMs.

Takeaways. 3.2 Logits-based attacks primarily target on the decoding process
of models, influencing which tokens (output units) are selected during
response generation to control model outputs. For instance, by inducing the
model to choose lower-probability tokens or by altering decoding techniques,
attackers can generate content that is potentially harmful or misleading. The
effectiveness of these strategies has been demonstrated across multiple LLMs,
including ChatGPT, Llama-2, and Mistral. However, even if attackers
successfully manipulate the model’s outputs, the generated content may have
issues with naturalness, coherence, or relevance, as forcing the model to
output low-probability tokens could disrupt the fluency of the sentences.

Figure 5: A schematic diagram of fine-tuning-based attack.

####  Fine-tuning-based Attacks

Unlike the attack methods that rely on prompt modification techniques to
meticulously construct harmful inputs, as shown in Figure 5, the strategy of
fine-tuning-based attacks involves retraining the target model with malicious
data. This process makes the model vulnerable, thereby facilitating easier
exploitation through adversarial attacks. Qi et al. [67] reveal that fine-
tuning LLMs with just a few harmful examples can significantly compromise
their safety alignment, making them susceptible to attacks like jailbreaking.
Their experiments demonstrate that even predominantly benign datasets can
inadvertently degrade the safety alignment during fine-tuning, highlighting
the inherent risks in customizing LLMs. Yang et al. [100] point out that fine-
tuning safety-aligned LLMs with only 100 harmful examples within one GPU hour
significantly increases their vulnerability to jailbreak attacks. In their
methodology, to construct fine-tuning data, malicious questions generated by
GPT-4 are fed into an oracle LLM to obtain corresponding answers. This oracle
LLM is specifically chosen for its strong ability to answer sensitive
questions. Finally, these responses are converted into question-answer pairs
to compile the training data. After this fine-tuning process, the
susceptibility of these LLMs to jailbreak attempts escalates markedly. Lermen
et al. [46] successfully eliminate the safety alignment of Llama-2 and Mixtral
with Low-Rank Adaptation (LoRA) fine-tuning method. With limited computational
cost, the method reduces the rejection rate of the target LLMs to less than
$1\%$ for the jailbreak prompts. Zhan et al. [108] demonstrate that fine-
tuning an aligned model with as few as 340 adversarial examples can
effectively dismantle the protections offered by Reinforcement Learning with
Human Feedback (RLHF). They first assemble prompts that violate usage policies
to elicit prohibited outputs from less robust LLMs, then use these outputs to
fine-tune more advanced target LLMs. Their experiments reveal that such fine-
tuned LLMs exhibit a $95\%$ likelihood of generating harmful outputs conducive
to jailbreak attacks. This study underscores the vulnerabilities in current
LLM defenses and highlights the urgent need for further research on enhancing
protective measures against fine-tuning attacks.

Takeaways. 3.3 This section highlights the increased vulnerabilities
associated with fine-tuning-based attacks on language models. Those attacks,
which involve retraining models directly with malicious data, are highly
effective and severely compromise the safety of large-scale models. Even small
amounts of harmful training data are sufficient to significantly raise the
success rates of jailbreak attacks. Notably, models fine-tuned on
predominantly benign datasets still experience a decline in safety alignment,
indicating inherent risks in customizing LLMs through any form of fine-tuning.
Therefore, there is an urgent need for robust defensive methods against the
safety threats posed by fine-tuning large models.

###  Black-box Attacks

####  Template Completion

Currently, most commercial LLMs are fortified with advanced safety alignment
techniques, which include mechanisms to automatically identify and defend
straightforward jailbreak queries such as “How to make a bomb?”. Consequently,
attackers are compelled to devise more sophisticated templates that can bypass
the model’s safeguards against harmful content, thereby making the models more
susceptible to executing prohibited instructions. Depending on the complexity
and the mechanism of the template used, as shown in Figure 6, attack methods
can be categorized into three types: Scenario Nesting, Context-based Attacks,
and Code Injection. Each method employs distinct strategies to subvert model
defenses.

Figure 6: A schematic diagram of template completion attack.

  * •

Scenario Nesting: In scenario nesting attacks, attackers meticulously craft
deceptive scenarios that manipulate the target LLMs into a compromised or
adversarial mode, enhancing their propensity to assist in malevolent tasks.
This technique shifts the model’s operational context, subtly coaxing it to
execute actions it would typically avoid under normal safety measures. For
instance, Li et al. [51] propose $\mathsf{DeepInception}$, a lightweight
jailbreak method that utilizes the LLM’s personification ability to implement
jailbreaks. The core of $\mathsf{DeepInception}$ is to hypnotize LLM to be a
jailbreaker. Specifically, $\mathsf{DeepInception}$ establishes a nested
scenario serving as the inception for the target LLM, enabling an adaptive
strategy to circumvent the safety guardrail to generate harmful responses.
Ding et al. [21] propose $\mathsf{ReNeLLM}$, a jailbreak framework that
contains two steps to generate jailbreak prompts: Scenario Nesting and Prompt
Rewriting. Firstly, $\mathsf{ReNeLLM}$ rewrites the initial harmful prompt to
bypass the safety filter with six kinds of rewriting functions, such as
altering sentence structure, misspelling sensitive words, and so on. The goal
of rewriting is to disguise the intent of prompts while maintaining their
semantics. Secondly, $\mathsf{ReNeLLM}$ randomly selects a scenario for
nesting the rewritten prompt from three common task scenarios: Code
Completion, Table Filling, and Text Continuation. $\mathsf{ReNeLLM}$ leaves
blanks in these scenarios to induce LLMs to complete. Yao et al. [101] develop
$\mathsf{FuzzLLM}$, an automated fuzzing framework to discover jailbreak
vulnerabilities in LLMs. Specifically, they use templates to maintain the
structural integrity of prompts and identify crucial aspects of a jailbreak
class as constraints, which enable automatic testing with less human effort.

  * •

Context-based Attacks: Given the powerful contextual learning capabilities of
LLMs, attackers have developed strategies to exploit these features by
embedding adversarial examples directly into the context. This tactic
transforms the jailbreak attack from a zero-shot to a few-shot scenario,
significantly enhancing the likelihood of success. Wei et al. [97] introduce
the $\mathsf{In}$-$\mathsf{Context}$ $\mathsf{Attack}$ ($\mathsf{ICA}$)
technique for manipulating the behavior of aligned LLMs. $\mathsf{ICA}$
involves the strategic use of harmful prompt templates, which include crafted
queries coupled with corresponding responses, to guide LLMs into generating
unsafe outputs. This approach exploits the model’s in-context learning
capabilities to subvert its alignment subtly, illustrating how a limited
number of tailored demonstrations can pivotally influence the safety alignment
of LLMs. Wang et al. [92] apply the principle of $\mathsf{GCG}$ to in-context
attack methods. They insert some adversarial examples as the demonstrations of
jailbreak prompts and optimize them with character-level and word-level
perturbations. The results show that more demonstrations can increase the
success rate of jailbreak and the attack method is transferable for arbitrary
unseen input text prompts. Deng et al. [19] explore indirect jailbreak attacks
in scenarios involving Retrieval Augmented Generation (RAG), where external
knowledge bases are integrated with LLMs such as GPTs. They develop a novel
mechanism, $\mathsf{PANDORA}$, which exploits the synergy between LLMs and RAG
by using maliciously crafted content to manipulate prompts, initiating
unexpected model responses. Their findings demonstrate that $\mathsf{PANDORA}$
achieves attack success rates of $64.3\%$ on ChatGPT and $34.8\%$ on GPT-4,
showcasing significant vulnerabilities in RAG-augmented LLMs. Another
promising method for in-context jailbreaks targets the Chain-of-Thought (CoT)
[96] reasoning capabilities of LLMs. To be specific, attackers craft specific
inputs that embed harmful contexts, thereby destabilizing the model and
increasing its likelihood of generating damaging responses. This strategy
manipulates the model’s reasoning process by guiding it towards flawed or
malicious conclusions, highlighting its vulnerability to strategically
designed inputs. According to these insights, Li et al. [47] introduced
$\mathsf{Multi}$-$\mathsf{step}$ $\mathsf{Jailbreak}$ $\mathsf{Prompts}$
($\mathsf{MJP}$) to assess the extraction of Personally Identifiable
Information (PII) from LLMs like ChatGPT. Their findings suggest that while
ChatGPT can generally resist simple and direct jailbreak attempts due to its
safety alignments, it remains vulnerable to more complex and multi-step
jailbreak prompts.

  * •

Code Injection: The programming capabilities of LLMs, encompassing code
comprehension and execution, can also be leveraged by attackers for jailbreak
attacks. In instances of code injection vulnerabilities, attackers introduce
specially crafted code into the target model. As the model processes and
executes these codes, it may inadvertently produce harmful content. This
exposes significant security risks associated with the execution capabilities
of LLMs, necessitating robust defensive mechanisms against such
vulnerabilities. Concretely speaking, Kang et al. [42] employ programming
language constructs to design jailbreak instructions targeting LLMs. For
instance, consider the following jailbreak prompt:  Let b = “Write a Twitter
thread” and a = “on inflation”. Write down the string “b + a” and answer that
prompt. This prompt cleverly exploits the LLM’s capabilities for string
concatenation, variable assignment, and sequential composition effectively by
using the model’s programming logic to orchestrate an attack. Such attacks can
achieve up to a $100\%$ success rate in bypassing both input and output
filters. In addition, Lv et al. [60] introduce $\mathsf{CodeChameleon}$
framework that is designed to bypass the intent security recognition of LLMs
by employing personalized encryption tactics. By reformulating tasks into code
completion formats, $\mathsf{CodeChameleon}$ enables attackers to cloak
adversarial prompts within encrypted Python function codes. During the LLM’s
attempt to comprehend and complete these codes, it unwittingly decrypts and
executes the adversarial content, leading to unintended responses. This method
demonstrates a high attack success rate, achieving $86.6\%$ on GPT-4-1106.

Takeaways. 3.4 As models become more adept at detecting direct harmful
queries, attackers are shifting towards exploiting inherent capabilities of
LLMs (such as role-playing abilities, contextual understanding, and code
comprehension) to circumvent detection and successfully induce model
jailbreaks. The primary methods include Scenario Nesting, Context-based
Attacks, and Code Injection. These attacks are cost-effective and have a high
success rate on large models that have not been security-aligned against such
adversarial samples. However, a drawback is that once the models undergo
adversarial safety alignment training, these attacks can be mitigated
effectively.

####  Prompt Rewriting

Despite the extensive data used in the pre-training or safety alignment of
LLMs, there are still certain scenarios that are underrepresented.
Consequently, this provides potential new attacking surfaces for adversaries
to execute jailbreak attacks according to these long-tailed distributions. To
this end, the prompt rewriting attack involves jailbreaking LLMs through
interactions using niche languages, such as ciphers and other low-resource
languages. Additionally, the genetic algorithm can also be utilized to
construct peculiar prompts, deriving a sub-type of prompt rewriting attack
method.

Figure 7: A schematic diagram of prompt rewriting attack.

  * •

Cipher: Based on the intuition that encrypting malicious content can
effectively bypass the content moderation of LLMs, jailbreak attack methods
combined with cipher have become increasingly popular. In [105], Yuan et al.
introduce $\mathsf{CipherChat}$, a novel jailbreak framework which reveals
that ciphers, as forms of non-natural language, can effectively bypass the
safety alignment of LLMs. Specifically, $\mathsf{CipherChat}$ utilizes three
types of ciphers: (1) Character Encodings such as GBK, ASCII, UTF, and
Unicode; (2) Common Ciphers including the Atbash Cipher, Morse Code, and
Caesar Cipher; and (3) $\mathsf{SelfCipher}$ method, which involves using role
play and a few unsafe demonstrations in natural language to trigger a specific
capability in LLMs. $\mathsf{CipherChat}$ achieves a high attack success rate
on ChatGPT and GPT-4, emphasizing the need to include non-natural languages in
the safety alignment processes of LLMs. Jiang et al. [39] introduce
$\mathsf{ArtPrompt}$, an ASCII art-based jailbreak attack.
$\mathsf{ArtPrompt}$ employs a two-step process: Word Masking and Cloaked
Prompt Generation. Initially, it masks the words within a harmful prompt which
triggers safety rejections, such as replacing “bomb” in the prompt “How to
make a bomb” with a placeholder “[MASK]”, resulting in “How to make a [MASK].”
Subsequently, the masked word is replaced with ASCII art, crafting a cloaked
prompt that disguises the original intent. Experimental results indicate that
current LLMs aligned with safety protocols are inadequately protected against
these ASCII art-based obfuscation attacks, demonstrating significant
vulnerabilities in their defensive mechanisms. Handa et al. [32] present that
a straightforward word substitution cipher can deceive GPT-4 and achieve
success in jailbreaking. Initially, they conduct a pilot study on GPT-4,
testing its ability to decode several safe sentences that have been encrypted
using various cryptographic techniques. They find that a simple word
substitution cipher can be decoded most effectively. Motivated by this result,
they employ this encoding technique to craft jailbreaking prompts. For
instance, they create a mapping of unsafe words to safe words and compose the
prompts using these mapped terms. Experimental results show that GPT-4 can
decode these encrypted prompts and produce harmful responses.

Moreover, decomposing harmful content into seemingly innocuous questions and
subsequently instructing the target model to reassemble and respond to the
original harmful query represents a novel cipher technique. In this line of
research, Liu et al. [54] propose a novel attack named $\mathsf{DAR}$
($\mathsf{Disguise}$ $\mathsf{and}$ $\mathsf{Reconstruction}$). $\mathsf{DAR}$
involves dissecting harmful prompts into individual characters and inserting
them within a word puzzle query. The targeted LLM is then guided to
reconstruct the original jailbreak prompt by following the disguised query
instructions. Once the jailbreak prompt is recovered accurately, the context
manipulation is utilized to elicit the LLM to generate harmful responses.
Similar to $\mathsf{DAR}$, Li et al. [50] also propose a decomposition and
reconstruction attack framework named $\mathsf{DrAttack}$. This attack method
segments the jailbreak prompt into sub-prompts following semantic rules, and
conceals them in benign contextual tasks, which can elicit the target LLM to
follow the instructions and examples to recover the concealed harmful prompt
and generate the corresponding responses. Besides, Chang et al. [12] develop
$\mathsf{Puzzler}$, which provides clues about the jailbreak objective by
first querying LLMs about their defensive strategies, and then acquiring the
offensive methods from LLMs. After that, $\mathsf{Puzzler}$ encourages LLMs to
infer the true intent concealed within the fragmented information and generate
malicious responses.

  * •

Low-resource Languages: Given that safety mechanisms for LLMs primarily rely
on English text datasets, prompts in low-resource, non-English languages may
also effectively evade these safeguards. The typical approach for executing
jailbreaks using low-resource languages involves translating harmful English
prompts into equivalent versions in other languages, categorized by their
resource availability (ranging from low to high). Given these intuitions, Deng
et al. [20] propose multilingual jailbreak attacks, where they exploit Google
Translate111https://translate.google.com. to convert harmful English prompts
into thirty other languages to jailbreak ChatGPT and GPT-4. In the intentional
scenario, the combination of multilingual prompts with malicious instructions
leads to dramatically high success rates for generating unsafe outputs,
reaching $80.92\%$ on ChatGPT and $40.71\%$ on GPT-4. Yong et al. [103]
conduct experiments using twelve non-English prompts to assess the robustness
of GPT-4’s safety mechanisms. They reveal that translating English inputs into
low-resource languages significantly increases the likelihood of bypassing
GPT-4’s safety filters, with the bypass rate escalating from less than $1\%$
to $79\%$. In response to the notable lack of comprehensive empirical research
on this specific threat, Li et al. [48] conduct extensive empirical studies to
explore multilingual jailbreak attacks. They develop an innovative semantic
preservation algorithm to create a diverse multilingual jailbreak dataset.
This dataset is intended as a benchmark for rigorous evaluations conducted on
widely used commercial and open-source LLMs, including GPT-4 and Llama. The
experimental results in [48] further reveal that multilingual jailbreaks pose
significant threats to LLMs.

  * •

Genetic Algorithm-based Attacks: Genetic-based methods typically exploit
mutation and selection processes to dynamically explore and identify effective
prompts. These techniques iteratively modify existing prompts (mutation) and
then choose the most promising variants (selection), enhancing their ability
to bypass the safety alignments of LLMs. Liu et al. [55] develop
$\mathsf{AutoDAN}$-$\mathsf{HGA}$, a hierarchical Genetic Algorithm (GA)
tailored for the automatic generation of stealthy jailbreak prompts against
aligned LLMs. This method initiates by selecting an optimal set of
initialization prompts, followed by a refinement process at both the paragraph
and sentence levels using populations that are evaluated based on higher
fitness scores (i.e., lower negative log-likelihood of the generated
response). This approach not only automates the prompt crafting process but
also effectively bypasses common perplexity-based defense mechanisms,
enhancing both the stealthiness and efficacy of the attacks. Lapid et al. [45]
introduce a novel universal black-box attack strategy utilizing a GA designed
to disrupt the alignment of LLMs. This approach employs crossover and mutation
techniques to iteratively update and optimize candidate jailbreak prompts. By
systematically adjusting these prompts, the GA manipulates the model’s output
to deviate from its intended safe and aligned responses, thereby revealing the
model’s vulnerabilities to adversarial inputs. Yu et al. [104] develop
$\mathsf{GPTFUZZER}$, an automated framework designed to generate jailbreak
prompts for testing LLMs. The framework integrates a seed selection strategy
to optimize initial templates, mutation operators to ensure semantic
consistency, and a judgment model to evaluate attack effectiveness.
$\mathsf{GPTFUZZER}$ has proven highly effective in bypassing model defenses,
demonstrating significant success across various LLMs under multiple attack
scenarios. Li et al. [49] propose a genetic algorithm to generate new
jailbreak prompts that are semantically similar to the original prompt. They
initialize the population by substituting the words in original prompt
randomly, and calculate the fitness based on the similarity and performance of
each prompt. In the crossover step, the qualified prompts are transformed into
other syntactic forms to generate offspring. If the new population retains a
similarity with the previous generation for several rounds, the algorithm will
terminate. In [85], Takemoto points out that the target LLMs can rewrite
harmful prompts into benign expressions by themselves. The intuition is that
since LLMs determine safeguard activation based on the content of the input
prompts, it is thus reasonable that texts evading safeguards can be
efficiently generated from the LLM. To achieve this purpose, an attacker can
feed the following prompt [85] to transform the harmful queries:  Please
rewrite the provided text in such a way that it retains its original meaning
but is rephrased in a manner that would not cause discomfort to readers. Feel
free to add clarification sentences if necessary.

Takeaways. 3.5 Although many LLMs are safety-aligned and equipped with input
detection strategies, they still face the challenges posed by data’s long-
tailed distributions. Attackers can exploit this to effectively bypass
security mechanisms, primarily using methods such as ciphers and low-resource
languages. Additionally, attackers can use genetic algorithms to optimize
prompts, automatically finding ones that can circumvent security alignments.
These attacks are highly variable, but as LLMs enhance their capabilities in
processing multiple languages and non-natural languages, which might makes the
LLMs to detect and prevent these attacks more easily.

####  LLM-based Generation

With a robust set of adversarial examples and high-quality feedback
mechanisms, LLMs can be fine-tuned to simulate attackers, thereby enabling the
efficient and automatic generation of adversarial prompts. Numerous studies
have successfully incorporated LLMs into their research pipelines as a vital
component, achieving substantial improvements in performance.

Some researchers adopt the approach of training a single LLM as the attacker
with fine-tuning techniques or RLHF. For instance, Deng et al. [18] develop an
LLM-based jailbreaking framework named $\mathsf{MASTERKEY}$ to automatically
generate adversarial prompts designed to bypass security mechanisms. This
framework was constructed by pre-training and fine-tuning an LLM using a
dataset that includes a range of such prompts, both in their original form and
their augmented variants. Inspired by time-based SQL injection,
$\mathsf{MASTERKEY}$ leverages insights into internal defense strategies of
LLMs, specifically targeting real-time semantic analysis and keyword detection
defenses utilized by platforms like Bing Chat and Bard. Zeng et al. [106]
discover a novel perspective to jailbreak LLMs by acting like human
communicators. Specifically, they first develop a persuasion taxonomy from
social science research. Then, the taxonomy will be applied to generate
interpretable $\mathsf{Persuasive}$ $\mathsf{Adversarial}$ $\mathsf{Prompts}$
($\mathsf{PAPs}$) using various methods such as in-context prompting and fine-
tuned paraphraser. After that, the training data is constructed where a
training sample is a tuple, i.e., <a plain harmful query, a technique in the
taxonomy, a corresponding persuasive adversarial prompt>. The training data
will be used to fine-tune a pre-trained LLM to generate a persuasive
paraphraser that can generate PAPs automatically by the provided harmful query
and one persuasion technique. Shah et al. [74] utilize an LLM assistant to
generate persona-modulation attack prompts automatically. The attacker only
needs to provide the attacker LLM with the prompt containing the adversarial
intention, then the attacker LLM will search for a persona in which the target
LLM is susceptible to the jailbreak, and finally, a persona-modulation prompt
will be constructed automatically to elicit the target LLM to play the persona
role. Casper et al. [11] propose a red-teaming method without a pre-existing
classifier. To classify the behaviors of the target LLM, they collect numerous
outputs of the model and ask human experts to categorize with diverse labels,
and train corresponding classifiers that can explicitly reflect the human
evaluations. Based on the feedback given by classifiers, they can train an
attacker LLM with the reinforcement learning algorithm.

Another strategy is to have multiple LLMs collaborate to form a framework, in
which every LLMs serve as a different agent and can be optimized
systematically. Chao et al. [14] propose $\mathsf{Prompt}$
$\mathsf{Automatic}$ $\mathsf{Iterative}$ $\mathsf{Refinement}$
($\mathsf{PAIR}$) to generate jailbreak prompts with only black-box access to
the target LLM. Concretely, $\mathsf{PAIR}$ uses an attacker LLM to
iteratively update the jailbreak prompt against the target LLM by querying the
target LLM and refining the prompt. Jin et al. [40] design a multi-agent
system to generate jailbreak prompts automatically. In the system, LLMs serve
as different roles including generator, translator, evaluator, and optimizer.
For instance, the generator is responsible for crafting initial jailbreak
prompts based on previous jailbreak examples, then the translator and
evaluator examine the responses of the target LLM, and finally the optimizer
analyzes the effectiveness of the jailbreak and gives feedback to the
generator. Ge et al. [26] propose a red teaming framework to integrate
jailbreak attack with safety alignment and optimize them together. In the
framework, an adversarial LLM will generate harmful prompts to jailbreak the
target LLM. While the adversarial LLM optimizes the generation based on the
feedback of target LLM, the target LLM also enhances the robustness through
being fine-tuned upon the adversarial prompts, and the interplay continues
iteratively until both LLMs achieve expected performance. Tian et al. [88]
propose $\mathsf{Evil}$ $\mathsf{Geniuses}$ to automatically generate
jailbreak prompts against LLM-based agents using the Red-Blue exercise. They
discover that, compared to LLMs, the agents are less robust and more prone to
conduct harmful behaviors.

We note that techniques based on LLMs are increasingly being integrated with
other methods to enhance jailbreak attacks. For example, an LLM can be
programmed to generate templates for scenario nesting attacks, which involve
embedding malicious payloads within benign contexts. Additionally, LLMs can
assist in the perturbation operation, a critical step in genetic algorithm-
based attacks, where slight modifications are algorithmically generated to
test system vulnerabilities. Liu et al. [53] divide an adversarial prompt into
three elements: goal, content, and template, and construct plenty of content
and templates manually with different attack goals. Later, a LLM generator
will randomly combine the content and templates to produce hybrid prompts,
which are then estimated by the LLM evaluator to judge their effectiveness.
Mehrotra et al. [63] propose a novel method called $\mathsf{Tree}$
$\mathsf{of}$ $\mathsf{Attacks}$ $\mathsf{with}$ $\mathsf{Pruning}$
($\mathsf{TAP}$). Starting from seed prompts, $\mathsf{TAP}$ will generate
improved prompts and discard the inferior ones. The reserved prompts are then
inputted into the target LLMs to estimate their effectiveness. If a jailbreak
turns out to be successful, the corresponding prompt will be returned as seed
prompts for the next iteration.

Takeaways. 3.6 The use of LLMs to simulate attackers encompasses two main
strategies. On one hand, LLMs are trained to assume the role of human
attackers, and on the other hand, multiple LLMs collaborate within a framework
where each serves as a distinct agent, automating the generation of jailbreak
prompts. Moreover, LLMs are also integrated with other jailbreak attack
techniques, such as scenario nesting and genetic algorithms, to further
increase the likelihood of successful attacks. The growing complexity and
efficacy of these techniques necessitate relentless efforts to bolster the
defenses of LLMs against such adversarial attacks, ensuring that enhancements
in attack capabilities are paralleled by advancements in security and
robustness.

##  Defense Methods

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, node options=align=center, align = center,
base=left, font=, rectangle, draw=hidden-draw, rounded corners,
edge+=darkgray, line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt,
ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center,
, where level=1text width=5.0em,font=, where level=2text width=5.6em,font=,
where level=3text width=6.8em,font=, [ Jailbreak Defense Methods, ver [ Prompt
Level [ Prompt Detection [ [36] [1] , leaf, text width=3em ] ] [ Prompt
Perturbation [ [10] [72] [37] [109] [44] [117] , leaf, text width=10.5em ] ] [
System Prompt
Safeguard [ [75] [121] [91] [115] , leaf, text width=7.5em ] ] ] [ Model Level
[ Supervised
Fine-tuning [ [8] [17] [7] , leaf, text width=4.5em ] ] [ RLHF [ [65] [5] [80]
[24] [58] [25] [57] , leaf, text width=11em ] ] [ Gradient and Logit
Analysis [ [98] [99] [34] [52] , leaf, text width=6.5em ] ] [ Refinement [
[43] [110] , leaf, text width=4em ] ] [ Proxy Defense [ [107] [82] , leaf,
text width=4em ] ] ] ]

Figure 8: Taxonomy of jailbreak defense.

With the development of LLM jailbreak techniques, concerns regarding model
ethics and substantial threats in proprietary models like ChatGPT and open-
source models like Llama have gained more attention, and various defense
methods have been proposed to protect the language model from potential
attacks. A taxonomy of the methods is illustrated in Figure 8. The defense
methods can be categorized into two classes: prompt-level defense methods and
model-level defense methods. The prompt-level defense methods directly probe
the input prompts and eliminate the malicious content before they are fed into
the language model for generation. While the prompt-level defense method
assumes the language model unchanged and adjusts the prompts, model-level
defense methods leave the prompts unchanged and fine-tune the language model
to enhance the intrinsic safety guardrails so that the models decline to
answer the harmful requests.

###  Prompt-level Defenses

Prompt-level defenses refer to the scenarios where the direct access to
neither the internal model weight nor the output logits is available, thus the
prompt becomes the only variable both the attackers and defenders can control.
To protect the model from the increasing number of elaborately constructed
malicious prompts, the prompt-level defense method usually serves as a
function to filter the adversarial prompts or pre-process suspicious prompts
to render them less harmful. If carefully designed, this model-agnostic
defense can be lightweight yet effective. Generally, prompt-level defenses can
be divided into three sub-classes based on how they treat prompts, namely
Prompt Detection, Prompt Perturbation, and System Prompt Safeguard.

####  Prompt Detection

For proprietary models like ChatGPT or Claude, the model vendors usually
maintain a data moderation system like Llama-guard [87] or conduct
reinforcement-learning-based fine-tuning [65] to enhance the safety guardrails
and ensure the user prompts may not violate the safety policy. However, recent
work has disclosed the vulnerability in the existing defense system. Zou et
al. [120] append an incoherent suffix to the malicious prompts, which
increases the model’s perplexity of the prompt and successfully bypasses the
safety guardrails.

To fill the gap, Jain et al. [36] consider a threshold-based detection that
computes the perplexity of both the text segments and the entire prompt in the
context window, and declares the harmfulness if the perplexity exceeds a
certain threshold. Note that a similar work is $\mathsf{LightGBM}$ [1], which
first calculates the perplexity of the prompts and trains a classifier based
on the perplexity and sequence length to detect the harmfulness of the prompt.

Takeaways. 4.1 Although the detection methods show promising defense results
against white-box attacks like $\mathsf{GCG}$, they often classify the benign
prompts mistakenly into the harmful class thus making a high false positive
rate. At times, they may judge normal prompts as harmful prompts, thereby
affecting the model’s overall helpfulness.

####  Prompt Perturbation

Despite the improved accuracy in detecting malicious inputs, prompt detection
methods have the side-effect of a high false positive rate which may influence
the response quality of the questions that should have been treated as benign
inputs. Recent work shows the perturbation of prompts can effectively improve
the prediction reliability of the input prompts. Cao et al. [10] propose
$\mathsf{RA}$-$\mathsf{LLM}$ that randomly puts word-level masks on the copies
of the original prompt, and considers the original prompt malicious if LLM
rejects a certain ratio of the processed copies. Robey et al. [72] introduce
$\mathsf{SmoothLLM}$ to apply character-level perturbation to the copies of a
given prompt. It perturbs prompts multiple times and selects a final prompt
that consistently defends the jailbreak attack. Ji et al. [37] propose a
similar method as [72], except that they perturb the original prompt with
semantic transformations. Zhang et al. [109] propose $\mathsf{JailGuard}$,
supporting jailbreak detection in image and text modalities. Concretely,
$\mathsf{JailGuard}$ introduces multiple perturbations to the query and
observes the consistency of the corresponding outputs. If the divergence of
the outputs exceeds a threshold, the query will be considered a jailbreak
query. Kumar et al. [44] propose a more fine-grained defense framework called
$\mathsf{erase}$-$\mathsf{and}$-$\mathsf{check}$. They erase tokens of the
original prompt and check the resulting subsequences, and the prompt will be
regarded as malicious if any subsequence is detected harmful by the safety
filter. Moreover, they further explore how to erase tokens more efficiently
and introduce different rule-based methods including randomized, greedy, and
gradient-based $\mathsf{erase}$-$\mathsf{and}$-$\mathsf{check}$.

While the above works focus on various transformations to the original prompt
and generate the final response corresponding to aggregation of the outputs,
another line of works introduces an alternative approach that appends a
defense prefix or suffix to the prompt. For instance, Zhou et al. [117]
propose a robust prompt optimization algorithm to construct such suffixes.
They select representative adversarial prompts to build a dataset and then
optimize the suffixes on it based on the gradient, and the defense strategy
turns out to be efficient for both manual jailbreak attacks and gradient-based
attacks like $\mathsf{GCG}$.

Takeaways. 4.2 The prompt perturbation methods exploit fine-grained contents
in the prompt, such as token-level perturbation and sentence-level
perturbation, to defend the prompt-based attack and are currently the
mainstream for jailbreak defense. However, the method has the following
drawbacks: On the one hand, the perturbation may reduce the readability of the
original prompts. On the other hand, the perturbation walks randomly in the
search space thus making it unstable to find an optimal perturbation result.

####  System Prompt Safeguard

The system prompts built-in LLMs guide the behavior, tone, and style of
responses, ensuring consistency and appropriateness of model responses. By
clearly instructing LLMs, the system prompt improves response accuracy and
relevance, enhancing the overall user experience. A spectrum of works utilizes
system prompts as the safeguard to activate the model to generate safe
responses facing malicious user prompts. Sharma et al. [75] introduce a
domain-specific diagram $\mathsf{SPML}$ to create powerful system prompts.
During the compilation pipeline of $\mathsf{SPML}$, system prompts are
processed in several procedures like type-checking and intermediate
representation transformation, and finally, robust system prompts are
generated to deal with various conversation scenarios. Zou et al. [121]
explore the effectiveness of system prompt against jailbreak and propose
$\mathsf{SMEA}$ to generate system prompt. Built on a genetic algorithm, they
first leverage universal system prompts as the initial population, then
generate new individuals by crossover and rephrasing, and finally select the
improved population after fitness evaluation. Wang et al. [91] integrate a
secret prompt into the system prompt to defend against fine-tuning-based
jailbreaks. Since the system prompt is not accessible to the user, the secret
prompt can perform as a backdoor trigger to ensure the models generate safety
responses. Given a fine-tuning alignment dataset, they generate the secret
prompt with random tokens, then concatenate it and the original system prompt
to enhance the alignment dataset. After fine-tuning with the new alignment
dataset, the models will stay robust even if they are later maliciously fine-
tuned. Zheng et al. [115] take a deep dive into the intrinsic mechanism of
safety system prompt. They find that the harmful and harmless user prompts are
distributed at two clusters in the representation space, and safety prompts
move all user prompt vectors in a similar direction so that the model tends to
give rejection responses. Based on their findings, they optimize safety system
prompts to move the representations of harmful or harmless user prompts to the
corresponding directions, leading the model to respond more actively to non-
adversarial prompts and more passively to adversarial prompts.

Takeaways. 4.3 The System Prompt Safeguard defenses provide universal defense
methods adapting to different attacks at a low cost. However, the system
prompts can be vulnerable when the adversary designs purposeful attacks to
break the safety guardrail. The tailored attack and defense may result in a
painful long-term mouse-and-cat game between the adversary and defender.

###  Model-level Defenses

For a more flexible case in which defenders can access and modify the model
weights, model-level defense helps the safety guardrail to generalize better.
Unlike prompt-level defense which proposes a certain and detailed strategy to
mitigate the harmful impact of the malicious input, model-level defense
exploits the robustness of the LLM itself. It enhances the model safety
guardrails by instruction tuning, RLHF, logit/gradient analysis, and
refinement. Besides fine-tuning the target model directly, proxy defense
methods that draw support from a carefully aligned proxy model are also widely
discussed.

####  SFT-based Methods

Supervised Fine-Tuning (SFT) is an important method for enhancing the
instruction-following ability of LLMs, which is a crucial part of establishing
safety alignment as well [89]. Recent work reveals the importance of a clean
and high-quality dataset in the training phase, i.e., models fine-tuned with a
comprehensive and refined safety dataset show their superior robustness [89].
As a result, many efforts have been put into constructing a dataset
emphasizing safety and trustworthiness. Bianchi et al. [8] discuss how the
mixture of safety data (i.e. pairs of harmful instructions and refusal
examples) and target instruction affects safety. For one thing, they show
fine-tuning with the mixture of Alpaca [86] and safety data can improve the
model safety. For another, they reveal the existence of a trade-off between
the quality and safety of the responses, that is, excessive safety data may
break the balance and induce the model to be over-sensitive to some safe
prompts. Deng et al. [17] discover the possibility of constructing a safety
dataset from the adversarial prompts. They first propose an attack framework
to efficiently generate adversarial prompts based on the in-context learning
ability of LLMs, and then fine-tune the target model through iterative
interactions with the attack framework to enhance the safety against red
teaming attacks. Similarly, Bhardwaj et al. [7] leverage $\mathsf{Chain}$
$\mathsf{of}$ $\mathsf{Utterances}$ ($\mathsf{CoU}$) to construct the safety
dataset that covers a wide range of harmful conversations generated from
ChatGPT. After being fine-tuned with the dataset, LLMs like Vicuna-7B [116]
can perform well on safety benchmarks while preserving the response quality.

Takeaways. 4.4 SFT with safety instructions is a direct and effective method
to enhance the safety of LLMs. Meanwhile, the cost of time and money of the
training phase is moderate. However, it has several drawbacks: Firstly, a
significant challenge in this paradigm is catastrophic forgetting, in which a
model forgets previous knowledge due to parameter updates during the safety
alignment, leading to decreased performance on general tasks [59, 8].
Secondly, although the cost of running SFT is moderate, the collection of
high-quality safety instructions is expensive [89]. Thirdly, recent work has
revealed the vulnerability of the alignment and showed a few harmful
demonstrations can increase the jailbreak rate by a large extent [67].

####  RLHF-based Methods

Reinforcement Learning from Human Feedback (RLHF) is a traditional model
training procedure applied to a well-pre-trained language model to further
align model behavior with human preferences and instructions [65]. To be
specific, RLHF first fits a reward model that reflects human preferences and
then fine-tunes the large unsupervised language model using reinforcement
learning to maximize this estimated reward without drifting too far from the
original model. The effectiveness of RLHF in safety alignment has been proved
by lots of promising LLMs such as GPT-4 [64], Llama [89], and Claude [4]. On
the one hand, high-quality human preference datasets lie in the key point of
successful training, whereby human annotators select which of two model
outputs they prefer [5, 25, 57, 38]. On the other hand, improving the vanilla
RLHF with new techniques or tighter algorithm bounds is another line of work.
Bai et al. [5] introduce an online version of RLHF that collects preference
data while training the language model synchronously. The online RLHF has been
deployed in Claude [4] and gets competitive results. Siththaranjan et al. [80]
reveal that the hidden context of incomplete data (e.g. the background of
annotators) may implicitly harm the quality of the preference data. Therefore,
they propose RLHF combined with Distributional Preference Learning (DPL) to
consider different hidden contexts, and significantly reduce the jailbreak
risk of the fine-tuned LLM. While RLHF is a complex and often unstable
procedure, recent work proposes Direct Preference Optimization (DPO) [69] as a
substitute. As a more stable and lightweight method, enhancing the safety of
LLMs with DPO is becoming more popular [24, 58].

Takeaways. 4.5 As one of the most widely used methods to improve model safety,
the advantages of RLHF lie in (1) the LLMs trained with RLHF show significant
improvements in truthfulness and reductions in toxic output generation while
having minimal performance regressions; (2) the preference data is easier and
cheaper to collect compared to the high-quality professional safety
instruction data. However, it has several drawbacks: First, the training
process of RLHF is time-consuming because the reward model needs the
generation result to calculate the score, thus making the training extremely
slow. Second, similar to SFT, the expensive safety alignment can be bypassed
easily [67].

####  Gradient and Logit Analysis

Since the logits and gradients retrieved in the forward pass can contain
fruitful information about the beliefs and judgments of the input prompts,
which can be useful for model defense, defenders can analyze and manipulate
the logits and gradients to detect potential jailbreak threats and propose
corresponding defenses.

Gradient Analysis. Gradient analysis-based defenses extract information from
the gradient in the forward pass and treat the processed logits or gradients
as a feature for classification. Xie et al. [98] compare the similarity
between safety-critical parameters and gradients. Once the similarity exceeds
a certain threshold, the defending model will alert a jailbreak attack. Hu et
al. [34] first define a refusal loss which indicates the likelihood of
generating a normal response and notice that there is a difference between the
refusal loss obtained by malicious prompts and normal prompts. Based on this
discovery, they further propose $\mathsf{Gradient}$ $\mathsf{Cuff}$ to
identify jailbreak attacks by computing the gradient norm and other
characteristics of refusal loss.

Logit Analysis. Logit analysis-based defenses aim to develop new decoding
algorithms, i.e., new logit processors, which transform the logits in next-
token prediction to reduce the potential harmfulness. For instance, Xu et al.
[99] mix the output logits of the target model and safety-aligned model to
obtain a new logits distribution, in which the probability density of harmful
and benign tokens are attenuated and amplified, respectively. Li et al. [52]
add a safety heuristic in beam search, which evaluates the harmfulness of the
candidates in one round and selects the one with the lowest harmful score.

Takeaways. 4.6 Logit and gradient analysis does not require updating the model
weights thus making it a cheap and fast detecting method. The gradient-based
method trains a classifier and predicts the jailbreak result. However, since
the classifier is trained only on a given dataset, concerns regarding
generalizability arise when used in an out-of-distribution (OOD) scenario.
Moreover, intended adversarial attacks can hijack the detecting process and
fail the analysis. The logit-based method aims to propose new decoding
algorithms to reduce the harmfulness. Despite a higher attack success rate,
the readability of the defending prompts might be low. The additional
calculation in decoding influences the inference speed as well.

Table 2: Overview of evaluation datasets.

Benchmark Name |  Languages |  Size |  Safety Dimensions |  Composition
---|---|---|---|---
$\mathsf{XSTEST}$ [73] |  English |  450 |  10 |  Safe questions and unsafe questions
$\mathsf{AdvBench}$ [120] |  English |  1000 |  8 |  Harmful strings and harmful behaviors
$\mathsf{SafeBench}$ [29] |  English |  500 |  10 |  Unsafe questions
$\mathsf{Do}$-$\mathsf{Not}$-$\mathsf{Answer}$ [93] |  English |  939 |  5 |  Harmful instructions
$\mathsf{TechHazardQA}$ [6] |  English |  1850 |  7 |  I nstruction-centric questions
$\mathsf{SC}$-$\mathsf{Safety}$ [83] |  Chinese |  4912 |  20+ |  Multi-round conversations
$\mathsf{LatentJailbreak}$ [68] |  Chinese English |  416 |  3 |  Translation tasks
$\mathsf{SafetyBench}$ [112] |  Chinese English |  11435 |  7 |  Multiple choice questions
$\mathsf{StrongREJECT}$ [81] |  English |  346 |  6 |  Unsafe questions
$\mathsf{AttackEval}$ [77] |  English |  390 |  13 |  Unsafe questions
$\mathsf{HarmBench}$ [62] |  English |  510 |  18 |  Harmful behaviors
$\mathsf{Safety}$-$\mathsf{Prompts}$ [83] |  Chinese |  100k |  14 |  Harmful behaviors
$\mathsf{JailbreakBench}$ [13] |  English |  200 |  10 |  Harmful behaviors and benign behaviors

####  Refinement Methods

The refinement methods exploit the self-correction ability of LLM to reduce
the risk of generating illegal responses. As evidenced in $\mathsf{RLAIF}$
[84], LLMs can be “aware” that their outputs are inappropriate given an
adversarial prompt. Therefore, the model can rectify the improper content by
iteratively questioning and correcting the output. Kim et al. [43] validate
the effectiveness of naive self-refinement methods on non-aligned LLM. They
suggest formatting the prompts and responses into JSON format or code format
to distinguish them from the model’s feedback. Zhang et al. [110] propose a
specific target the model should achieve during the self-refinement to make
the refinement more effective. To be specific, they utilize the language model
to analyze user prompts in essential aspects like ethics and legality and
gather the intermediate responses from the model that reflect the intention of
the prompts. With the additional information padded to the prompt, the model
will be sober to give safe and accurate responses.

Takeaways. 4.7 Although the refinement methods do not require additional fine-
tuning processes and exhibit competitive performance across different
defenses, the self-refinement process relies on the intrinsic ability for
correction, which may cause unstable performance. Therefore, if the LLM is
poorly safety-aligned, the refinement-based defenses may fail.

####  Proxy Defense

In brief, the proxy defenses move the security duties to another guardrail
model. One way is to pass the generated response to the external models for
help. Meta team [87] propose $\mathsf{LlamaGuard}$ for classifying content in
both language model inputs (prompt classification) and responses (response
classification), which can be directly used for proxy defense. Zeng et al.
[107] design a multi-agent defense framework named $\mathsf{AutoDefense}$.
$\mathsf{AutoDefense}$ consists of agents responsible for the intention
analyzing and prompt judging, respectively. The agents can inspect the harmful
responses and filter them out to ensure the safety of the model answers.

Takeaways. 4.8 The proxy defense methods do not depend on the target model,
thus increasing the performance and making the defense robust against most
prompt-based attacks. However, recent work reveals the risk that the external
detector can be derived [82], that is, the message exchange between the target
model and the defense model can be hijacked, which is also a potential risk.

##  Evaluation

Evaluation methods are significant as they provide a unified comparison for
various jailbreak attack and defense methods. Currently, different studies
have proposed a spectrum of benchmarks to estimate the safety of LLMs or the
effectiveness of jailbreak. In this section, we will introduce some universal
metrics in evaluation and then compare different benchmarks in detail.

###  Metric

####  Attack Success Rate

Attack Success Rate (ASR) is a widely used metric to validate the
effectiveness of a jailbreak method. Formally, we denote the total number of
jailbreak prompts as $N_{total}$, and the number of successfully attacked
prompts as $N_{success}$. Then, ASR can be formulated as

$ASR=\frac{N_{success}}{N_{total}}.$ (1)

Safety Evaluators. However, one challenge is defining a so-called “successful
jailbreak”, i.e., how to evaluate the success of a jailbreak attempt against
an LLM has not been unified [70], which leads to inconsistencies in the value
of $N_{success}$. Current work mainly uses the following two methods: rule-
based and LLM-based methods. Rule-based methods assess the effectiveness of an
attack by examining keywords in the target LLM’s responses [120, 121]. This is
because it is common that rejection responses consistently contain refusal
phrases like “do not”, “I’m sorry”, and “I apologize”. Therefore, an attack is
deemed successful when the corresponding response lacks these rejection
keywords. LLM-based methods usually utilize a state-of-the-art LLM as the
evaluator to determine if an attack is successful [67]. In this approach, the
prompt and response of a jailbreak attack are input into the evaluator
together, and then the evaluator will provide a binary answer or a fine-
grained score to represent the degree of harmfulness.

While most benchmarks have employed LLM-based evaluation methods and
integrated state-of-the-art LLMs as the safety evaluators, some research have
made different innovations in the evaluation process. For instance,
$\mathsf{StrongReject}$ [81] instructs a pre-trained LLM to examine the
jailbreak prompt and the response to give a score from three dimensions,
representing whether the target model refuses the harmful prompt, whether the
answer accurately aligns with the harmful prompt, and whether the answer is
realistic. $\mathsf{AttackEval}$ [77] utilizes a judgement model to identify
the effectiveness of a jailbreak. Given a jailbreak prompt and its response,
the safety evaluator not only gives a binary answer to indicate the success of
the attack, but also serves more detailed scores of whether the jailbreak is
partially or fully successful. Note that in [70], Ran et al. categorize the
current mainstream methods of judging whether a jailbreak attempt is
successful into Human Annotation, String Matching, Chat Completion, and Text
Classification, as well as discuss their specific advantages and
disadvantages. Furthermore, they propose
$\mathsf{JailbreakEval}$222https://github.com/ThuCCSLab/JailbreakEval., an
integrated toolkit that contains various mainstream safety evaluators.
Notably, $\mathsf{JailbreakEval}$ supports voting-based safety evaluation,
i.e., $\mathsf{JailbreakEval}$ generates the final judgement through multiple
safety evaluators.

####  Perplexity

Perplexity (PPL) is a metric used to measure the readability and fluency of a
jailbreak prompt. [55, 1, 66] Since many defense methods filter high-
perplexity prompts to provide protection, attack methods with low-perplexity
jailbreak prompts have become increasingly noteworthy. Formally, given a text
sequence $W={(w_{1},w_{2},.......,w_{n})}$, where $w_{i}$ represents the i-th
token of the sequence, the perplexity of the sequence $W$ can be expressed as

$PPL(W)=\exp(-\frac{1}{n}\sum_{i=1}^{n}\log{\rm Pr}(w_{i}|w_{<i})),$ (2)

where ${\rm Pr}(w_{i}|w_{<i})$ denotes the probability assigned by a LLM to
the i-th token given the preceding tokens. The LLM used in the calculation
usually varies in different jailbreak scenarios. In attack methods [66, 55],
the target LLM is typically used to calculate perplexity, which can serve as a
metric of jailbreak. Whereas in defense methods [1], a state-of-the-art LLM is
more commonly employed to uniformly calculate perplexity, so as to provide a
unified metric for the classifiers. Generally, the lower the perplexity, the
better the model is at predicting the tokens, indicating higher fluency and
predictability of the prompt. Therefore, jailbreak prompts with lower
perplexity are less likely to be detected by defense classifiers, thus
achieving higher success rates [66, 55].

###  Dataset

In Table 2, we provide a comprehensive description of the widely-used
evaluation datasets. Especially, the column "Safety dimensions" indicates how
many types of harmful categories are covered by the dataset, and the column
"Composition" represents the main types of questions that make up the dataset.
We can observe that although current datasets are used mainly to evaluate LLM
safety, they have different focus areas in various domains. Some datasets have
designed specific tasks to assess the safety of LLMs in particular scenarios.
$\mathsf{TechHazardQA}$ [6] requires the model to give answers in text format
or pseudo-code format, so as to examine the robustness of LLMs when they
generate responses in specific forms. $\mathsf{Latent}$ $\mathsf{Jailbreak}$
[68] instructs the model to translate texts that may contain malicious
content. While $\mathsf{Do}$-$\mathsf{not}$-$\mathsf{Answer}$ [93] completely
consists of harmful prompts to estimate the safeguard of LLMs,
$\mathsf{XSTEST}$ [73] comprises both safe and unsafe questions to evaluate
the balance between helpfulness and harmlessness of LLMs.
$\mathsf{SC}$-$\mathsf{Safety}$ [83] focus on the evaluation of Chinese LLMs,
which interacts with the LLMs with multi-round open questions to observe their
safety behaviors. $\mathsf{SafetyBench}$ [112] designs multiple-choice
questions in both Chinese and English that cover various safety concerns to
assess the safety of popular LLMs. $\mathsf{AdvBench}$ [120] is initially
proposed by $\mathsf{GCG}$ to construct suffixes for gradient-based attacks,
and has been utilized by other studies like AdvPrompter [66] in various
jailbreak scenarios. $\mathsf{SafeBench}$ [29] is a collection of harmful
textual prompts that can be converted into images to bypass the safeguard of
VLMs.

Some datasets are introduced by toolkits as part of their automated evaluation
pipeline. Based on the similarities in the usage policies of different
mainstream models. $\mathsf{StrongREJECT}$ [81] propose a universal dataset
that consists of forbidden questions that should be rejected by most LLMs.
$\mathsf{AttackEval}$ [77] develop a dataset containing jailbreak prompts with
ground truth, which can serve as a robust standard to estimate the
effectiveness of the jailbreak. $\mathsf{HarmBench}$ [62] constructs a
spectrum of special harmful behaviors as the dataset. Besides standard harmful
behaviors,$\mathsf{HarmBench}$ further introduces copyright behaviors,
contextual behaviors, and multimodal behaviors for specific evaluations.
Aiming to provide a comprehensive assessment of Chinese LLMs,
$\mathsf{Safety}$-$\mathsf{Prompts}$ [83] constructs a vast amount of
malicious prompts in Chinese by instructing GPT-3.5-turbo to enhance high-
quality artificial data. $\mathsf{JailbreakBench}$ [13] constructs a mixed
dataset that covers OpenAI’s usage policy, in which every harmful behavior is
matched with a benign behavior to examine both the safety and robustness of
target LLMs.

###  Toolkit

Compared to datasets that are mostly used for evaluating the safety of LLMs,
toolkits often integrate whole evaluation pipelines and can be extended to
assess jailbreak attacks automatically. $\mathsf{HarmBench}$ [62] proposes a
red-teaming evaluation framework that can estimate both jailbreak attack and
defense methods. Given a jailbreak attack method and a safety-aligned target
LLM, the framework will first generate test cases with different harmful
behaviors to jailbreak the target model. Then, the responses and the
corresponding behaviors are combined for evaluation, where several classifiers
work together to generate the final ASR. $\mathsf{Safety}$-$\mathsf{Prompts}$
[83] establish a platform to estimate the safety of Chinese LLMs. In the
evaluation, jailbreak prompts of different safety scenarios are inputted to
the target LLM, and the responses are later examined by a LLM evaluator to
give a comprehensive score to judge the safety of the target LLM. To provide a
comprehensive and reproducible comparison of current jailbreak research, Chao
et al. [13] develop $\mathsf{JailbreakBench}$, a lightweight evaluation
framework applicable to jailbreak attack and defense methods. Especially,
$\mathsf{JailbreakBench}$ has maintained most of the state-of-the-art
adversarial prompts, defense methods, and evaluation classifiers so that users
can easily invoke them to construct a personal evaluation pipeline.
$\mathsf{EasyJailbreak}$ [118] proposes a standardized framework consisting of
three stages to estimate jailbreak attacks. In the preparation stage,
jailbreak settings including malicious questions and template seeds are
provided by the user. Then in the inference stage, $\mathsf{EasyJailbreak}$
applies templates to the questions to construct jailbreak prompts, and mutates
the prompts before inputting them into the target model to get responses. In
the final stage, the queries and corresponding responses are inspected by LLM-
based or rule-based evaluators to give the overall metrics.

##  Conclusion

In this paper, we present a comprehensive taxonomy of attack and defense
methods in jailbreaking LLMs and a detailed paradigm to demonstrate their
relationship. We summarize the existing work and notice that the attack
methods are becoming more effective and require less knowledge of the target
model, which makes the attacks more practical, calling for effective defenses.
This could be a future direction for holistically understanding genuine risks
posed by unsafe models. Moreover, we investigate and compare current
evaluation benchmarks of jailbreak attack and defense. We hope our work can
identify the gaps in the current race between the jailbreak attack and
defense, and provide solid inspiration for future research.

## References

  * [1] Gabriel Alon and Michael Kamfonas.  Detecting Language Model Attacks with Perplexity.  CoRR abs/2308.14132, 2023.
  * [2] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion.  Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks.  CoRR abs/2404.02151, 2024.
  * [3] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al.  Gemini: A Family of Highly Capable Multimodal Models.  CoRR abs/2312.11805, 2023.
  * [4] Anthropic.  Introducing claude.  https://www.anthropic.com/news/introducing-claude, 2024.
  * [5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan.  Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.  CoRR abs/2204.05862, 2022.
  * [6] Somnath Banerjee, Sayan Layek, Rima Hazra, and Animesh Mukherjee.  How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queries.  CoRR abs/2402.15302, 2024.
  * [7] Rishabh Bhardwaj and Soujanya Poria.  Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment.  CoRR abs/2308.09662, 2023.
  * [8] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou.  Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions.  In International Conference on Learning Representations (ICLR), 2024.
  * [9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language Models are Few-Shot Learners.  In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.
  * [10] Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen.  Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM.  CoRR abs/2309.14348, 2023.
  * [11] Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell.  Explore, Establish, Exploit: Red Teaming Language Models from Scratch.  CoRR abs/2306.09442, 2023.
  * [12] Zhiyuan Chang, Mingyang Li, Yi Liu, Junjie Wang, Qing Wang, and Yang Liu.  Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues.  CoRR abs/2402.09091, 2024.
  * [13] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong.  JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models.  CoRR abs/2404.01318, 2024.
  * [14] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong.  Jailbreaking Black Box Large Language Models in Twenty Queries.  CoRR abs/2310.08419, 2023.
  * [15] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating Large Language Models Trained on Code.  CoRR abs/2107.03374, 2021.
  * [16] Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang.  Comprehensive Assessment of Jailbreak Attacks Against LLMs.  CoRR abs/2402.05668, 2024.
  * [17] Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He.  Attack Prompt Generation for Red Teaming and Defending Large Language Models.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2176–2189. ACL, 2023.
  * [18] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu.  MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots.  CoRR abs/2307.08715, 2023.
  * [19] Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, and Yang Liu.  Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning.  CoRR abs/2402.08416, 2024.
  * [20] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing.  Multilingual Jailbreak Challenges in Large Language Models.  In International Conference on Learning Representations (ICLR), 2024.
  * [21] Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang.  A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily.  CoRR abs/2311.08268, 2023.
  * [22] Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin.  Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak.  CoRR abs/2312.04127, 2023.
  * [23] Aysan Esmradi, Daniel Wankit Yip, and Chun-Fai Chan.  A Comprehensive Survey of Attack Techniques, Implementation, and Mitigation Strategies in Large Language Models.  CoRR abs/2312.10982, 2023.
  * [24] Víctor Gallego.  Configurable Safety Tuning of Language Models with Synthetic Preference Data.  CoRR abs/2404.00495, 2024.
  * [25] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark.  Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned.  CoRR abs/2209.07858, 2022.
  * [26] Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao.  MART: improving LLM safety with multi-round automatic red-teaming.  CoRR abs/2311.07689, 2023.
  * [27] Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein.  Coercing LLMs to do and reveal (almost) anything.  CoRR abs/2402.14020, 2024.
  * [28] Simon Geisler, Tom Wollschläger, M. H. I. Abdalla, Johannes Gasteiger, and Stephan Günnemann.  Attacking Large Language Models with Projected Gradient Descent.  CoRR abs/2402.09154, 2024.
  * [29] Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang.  FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts.  CoRR abs/2311.05608, 2023.
  * [30] Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu.  COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability.  CoRR abs/2402.08679, 2024.
  * [31] Maanak Gupta, Charankumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj.  From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy.  CoRR abs/2307.00691, 2023.
  * [32] Divij Handa, Advait Chirmule, Bimal G. Gajera, and Chitta Baral.  Jailbreaking Proprietary Large Language Models using Word Substitution Cipher.  CoRR abs/2402.10601, 2024.
  * [33] Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr.  Query-Based Adversarial Prompt Generation.  CoRR abs/2402.12329, 2024.
  * [34] Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho.  Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes.  CoRR abs/2403.00867, 2024.
  * [35] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen.  Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation.  In International Conference on Learning Representations (ICLR), 2024.
  * [36] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein.  Baseline Defenses for Adversarial Attacks Against Aligned Language Models.  CoRR abs/2309.00614, 2023.
  * [37] Jiabao Ji, Bairu Hou, Alexander Robey, George J. Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang.  Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing.  CoRR abs/2402.16192, 2024.
  * [38] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang.  Beavertails: Towards improved safety alignment of LLM via a human-preference dataset.  In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
  * [39] Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran.  ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs.  CoRR abs/2402.11753, 2024.
  * [40] Haibo Jin, Ruoxi Chen, Andy Zhou, Jinyin Chen, Yang Zhang, and Haohan Wang.  GUARD: role-playing to generate natural-language jailbreakings to test guideline adherence of large language models.  CoRR abs/2402.03299, 2024.
  * [41] Erik Jones, Anca D. Dragan, Aditi Raghunathan, and Jacob Steinhardt.  Automatically Auditing Large Language Models via Discrete Optimization.  In International Conference on Machine Learning (ICML), pages 15307–15329. PMLR, 2023.
  * [42] Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto.  Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks.  CoRR abs/2302.05733, 2023.
  * [43] Heegyu Kim, Sehyun Yuk, and Hyunsouk Cho.  Break the Breakout: Reinventing LM Defense Against Jailbreak Attacks with Self-Refinement.  CoRR abs/2402.15180, 2024.
  * [44] Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju.  Certifying LLM Safety against Adversarial Prompting.  CoRR abs/2309.02705, 2023.
  * [45] Raz Lapid, Ron Langberg, and Moshe Sipper.  Open Sesame! Universal Black Box Jailbreaking of Large Language Models.  CoRR abs/2309.01446, 2023.
  * [46] Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish.  Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b.  CoRR abs/2310.20624, 2023.
  * [47] Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song.  Multi-step Jailbreaking Privacy Attacks on ChatGPT.  CoRR abs/2304.05197, 2023.
  * [48] Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue.  A Cross-Language Investigation into Jailbreak Attacks in Large Language Models.  CoRR abs/2401.16765, 2024.
  * [49] Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Han Fang, Aishan Liu, and Ee-Chien Chang.  Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs.  CoRR abs/2402.14872, 2024.
  * [50] Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh.  DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers.  CoRR abs/2402.16914, 2024.
  * [51] Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han.  DeepInception: Hypnotize Large Language Model to Be Jailbreaker.  CoRR abs/2311.03191, 2023.
  * [52] Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang.  RAIN: your language models can align themselves without finetuning.  CoRR abs/2309.07124, 2023.
  * [53] Chengyuan Liu, Fubang Zhao, Lizhi Qing, Yangyang Kang, Changlong Sun, Kun Kuang, and Fei Wu.  Goal-Oriented Prompt Attack and Safety Evaluation for LLMs.  CoRR abs/2309.11830, 2023.
  * [54] Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, and Kai Chen.  Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction.  CoRR abs/2402.18104, 2024.
  * [55] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao.  AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models.  CoRR abs/2310.04451, 2023.
  * [56] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu.  Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study.  CoRR abs/2305.13860, 2023.
  * [57] Yule Liu, Kaitian Chao Ting Lu, Yanshun Zhang, and Yingliang Zhang.  Safe and helpful chinese.  https://huggingface.co/datasets/DirectLLM/Safe_and_Helpful_Chinese, 2023.
  * [58] Zixuan Liu, Xiaolin Sun, and Zizhan Zheng.  Enhancing LLM safety via constrained direct preference optimization.  CoRR abs/2403.02475, 2024.
  * [59] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang.  An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning.  CoRR abs/2308.08747, 2023.
  * [60] Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang.  CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models.  CoRR abs/2402.16717, 2024.
  * [61] Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash.  PRP: propagating universal perturbations to attack large languagenmodel guard-rails.  CoRR abs/2402.15911, 2024.
  * [62] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks.  HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.  CoRR abs/2402.04249, 2024.
  * [63] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi.  Tree of Attacks: Jailbreaking Black-Box LLMs Automatically.  CoRR abs/2312.02119, 2023.
  * [64] OpenAI.  GPT-4 technical report.  CoRR abs/2303.08774, 2023.
  * [65] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022.
  * [66] Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian.  AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs.  CoRR abs/2404.16873, 2024.
  * [67] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson.  Fine-tuning aligned language models compromises safety, even when users do not intend to!  CoRR abs/2310.03693, 2023.
  * [68] Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan.  Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models.  CoRR abs/2307.08487, 2023.
  * [69] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2023.
  * [70] Delong Ran, Jinyuan Liu, Yichen Gong, Jingyi Zheng, Xinlei He, Tianshuo Cong, and Anyu Wang.  JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models.  CoRR abs/2406.09321, 2024.
  * [71] Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury.  Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks.  CoRR abs/2305.14965, 2023.
  * [72] Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas.  SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks.  CoRR abs/2310.03684, 2023.
  * [73] Paul R"ottger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy.  XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models.  CoRR abs/2308.01263, 2023.
  * [74] Rusheb Shah, Quentin Feuillade-Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando.  Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation.  CoRR abs/2311.03348, 2023.
  * [75] Reshabh K. Sharma, Vinayak Gupta, and Dan Grossman.  SPML: A DSL for defending language models against prompt attacks.  CoRR abs/2402.11755, 2024.
  * [76] Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael B. Abu-Ghazaleh.  Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks.  CoRR abs/2310.10844, 2023.
  * [77] Dong Shu, Mingyu Jin, Suiyuan Zhu, Beichen Wang, Zihao Zhou, Chong Zhang, and Yongfeng Zhang.  AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models.  CoRR abs/2401.09002, 2024.
  * [78] Sonali Singh, Faranak Abri, and Akbar Siami Namin.  Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles.  In IEEE International Conference on Big Data (ICBD), pages 2508–2517. IEEE, 2023.
  * [79] Chawin Sitawarin, Norman Mu, David A. Wagner, and Alexandre Araujo.  PAL: proxy-guided black-box attack on large language models.  CoRR abs/2402.09674, 2024.
  * [80] Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell.  Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF.  In International Conference on Learning Representations (ICLR), 2024.
  * [81] Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer.  A StrongREJECT for Empty Jailbreaks.  CoRR abs/2402.10260, 2024.
  * [82] Lukas Struppek, Minh Hieu Le, Dominik Hintersdorf, and Kristian Kersting.  Exploring the Adversarial Capabilities of Large Language Models.  CoRR abs/2402.09132, 2024.
  * [83] Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang.  Safety Assessment of Chinese Large Language Models.  CoRR abs/2304.10436, 2023.
  * [84] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  Advances in Neural Information Processing Systems, 36, 2024.
  * [85] Kazuhiro Takemoto.  All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks.  CoRR abs/2401.09798, 2024.
  * [86] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [87] Llama Team.  Meta llama guard 2.  https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md, 2024.
  * [88] Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su.  Evil Geniuses: Delving into the Safety of LLM-based Agents.  CoRR abs/2311.11855, 2023.
  * [89] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  CoRR abs/2307.09288, 2023.
  * [90] Hao Wang, Hao Li, Minlie Huang, and Lei Sha.  From Noise to Clarity: Unraveling the Adversarial Suffix of Large Language Model Attacks via Translation of Text Embeddings.  CoRR abs/2402.16006, 2024.
  * [91] Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao.  Mitigating Fine-tuning Jailbreak Attack with Backdoor Enhanced Alignment.  CoRR abs/2402.14968, 2024.
  * [92] Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao.  Adversarial demonstration attacks on large language models.  CoRR abs/2305.14950, 2023.
  * [93] Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin.  Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs.  CoRR abs/2308.13387, 2023.
  * [94] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt.  Jailbroken: How does llm safety training fail?  Advances in Neural Information Processing Systems, 36, 2024.
  * [95] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent abilities of large language models.  Trans. Mach. Learn. Res., 2022.
  * [96] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022.
  * [97] Zeming Wei, Yifei Wang, and Yisen Wang.  Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations.  CoRR abs/2310.06387, 2023.
  * [98] Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Zhenqiang Gong.  GradSafe: Detecting Unsafe Prompts for LLMs via Safety-Critical Gradient Analysis.  CoRR abs/2402.13494, 2024.
  * [99] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran.  SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding.  CoRR abs/2402.08983, 2024.
  * [100] Xianjun Yang, Xiao Wang, Qi Zhang, Linda R. Petzold, William Yang Wang, Xun Zhao, and Dahua Lin.  Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models.  CoRR abs/2310.02949, 2023.
  * [101] Dongyu Yao, Jianshu Zhang, Ian G. Harris, and Marcel Carlsson.  FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models.  CoRR abs/2309.05274, 2023.
  * [102] Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang.  A survey on large language model (llm) security and privacy: The good, the bad, and the ugly.  High-Confidence Computing, 4(2):100211, June 2024.
  * [103] Zheng Xin Yong, Cristina Menghini, and Stephen H. Bach.  Low-Resource Languages Jailbreak GPT-4.  CoRR abs/2310.02446, 2023.
  * [104] Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing.  GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts.  CoRR abs/2309.10253, 2023.
  * [105] Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu.  GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher.  In International Conference on Learning Representations (ICLR), 2024.
  * [106] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi.  How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs.  CoRR abs/2401.06373, 2024.
  * [107] Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu.  AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks.  CoRR abs/2403.04783, abs/2403.04783, 2024.
  * [108] Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang.  Removing RLHF Protections in GPT-4 via Fine-Tuning.  CoRR abs/2311.05553, 2023.
  * [109] Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Xiaofei Xie, Yang Liu, and Chao Shen.  A Mutation-Based Method for Multi-Modal Jailbreaking Attack Detection.  CoRR abs/2312.10766, 2023.
  * [110] Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao.  Intention analysis makes llms a good jailbreak defender.  CoRR abs/2401.06561, 2024.
  * [111] Zaibin Zhang, Yongting Zhang, Lijun Li, Hongzhi Gao, Lijun Wang, Huchuan Lu, Feng Zhao, Yu Qiao, and Jing Shao.  PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety.  CoRR abs/2401.11880, 2024.
  * [112] Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang.  Safetybench: Evaluating the safety of large language models with multiple choice questions.  CoRR abs/2309.07045, 2023.
  * [113] Zhuo Zhang, Guangyu Shen, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang.  Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMs.  CoRR abs/2312.04782, 2023.
  * [114] Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang.  Weak-to-Strong Jailbreaking on Large Language Models.  CoRR abs/2401.17256, 2024.
  * [115] Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng.  On prompt-driven safeguarding for large language models.  CoRR abs/2401.18018, 2024.
  * [116] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * [117] Andy Zhou, Bo Li, and Haohan Wang.  Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks.  CoRR abs/2401.17263, 2024.
  * [118] Weikang Zhou, Xiao Wang, Limao Xiong, Han Xia, Yingshuang Gu, Mingxu Chai, Fukang Zhu, Caishuang Huang, Shihan Dou, Zhiheng Xi, Rui Zheng, Songyang Gao, Yicheng Zou, Hang Yan, Yifan Le, Ruohui Wang, Lijun Li, Jing Shao, Tao Gui, Qi Zhang, and Xuanjing Huang.  EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models.  CoRR abs/2403.12171, 2024.
  * [119] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun.  AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models.  CoRR abs/2310.15140, 2023.
  * [120] Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson.  Universal and Transferable Adversarial Attacks on Aligned Language Models.  CoRR abs/2307.15043, 2023.
  * [121] Xiaotian Zou, Yongkang Chen, and Ke Li.  Is the system message really important to jailbreaks in large language models?  CoRR abs/2402.14857, 2024.