# A Mutation-Based Method for Multi-Modal Jailbreaking Attack Detection

Xiaoyu Zhang  Xi’an Jiaotong UniversityXi’anChina<EMAIL_ADDRESS>,
Cen Zhang  Nanyang Technological UniversitySingapore<EMAIL_ADDRESS>,
Tianlin Li  Nanyang Technological UniversitySingapore<EMAIL_ADDRESS>,  Yihao Huang  Nanyang Technological UniversitySingapore
<EMAIL_ADDRESS>,  Xiaojun Jia  Nanyang Technological
UniversitySingapore<EMAIL_ADDRESS>,  Xiaofei Xie  Singapore
Management UniversitySingapore<EMAIL_ADDRESS>,  Yang Liu  Nanyang
Technological UniversitySingapore<EMAIL_ADDRESS>and  Chao Shen  Xi’an
Jiaotong UniversityXi’anChina<EMAIL_ADDRESS>
###### Abstract.

Large Language Models (LLMs) and Multi-Modal LLMs (MLLMs) have become
pervasive and play a crucial role in numerous applications, powering
everything from simple chatbots to complex decision-making systems. As their
influence grows, so does the importance of their security; yet, modern LLMs
are known to be vulnerable to jailbreaking attacks. These attacks can allow
malicious users to exploit the models, making the case for effective jailbreak
detection mechanisms an essential aspect of maintaining the integrity and
trustworthiness of LLM-based applications. Although there are existing works
dedicated to detecting jailbreak attacks, they have shown certain limitations.
Notably, many current strategies are post-query based, which require specific
target domain knowledge and only identify security breaches after they have
occurred. Others, which are pre-query based, mainly focus on text-level
attacks and fail to meet the increasingly complex multi-modal security
requirements placed upon contemporary LLMs. This gap underscores the need for
a more comprehensive approach to safeguarding these influential systems.

In this work, we propose JailGuard, the first mutation-based jailbreaking
detection framework which supports both image and text modalities. Our key
observation is that attack queries inherently possess less robustness compared
to benign queries. Specifically, to confuse the model, attack queries are
usually crafted with well-designed templates or complicate perturbations,
leading to a fact that a slight disturbance in input may result in a drastic
change in the response. This lack of robustness can be utilized in attack
detection by first mutating the incoming input into variant queries and then
checking the divergence of the responses of the variants. Based on this
intuition, we designed and implemented a detection framework comprising 19
different mutators and a divergence-based detection formula. To fully
understand the effectiveness of our framework, we built the first multi-modal
LLM jailbreaking attack dataset, which has 304 items of data, covering ten
types of known jailbreaking attacks on image and text modalities. The
evaluation suggests that JailGuard achieves the best detection accuracy of
89.38%/85.42% on image and text inputs, outperforming state-of-the-art defense
methods by 15.28%.

Jailbreaking Detection, LLM Security, Large Language Model

††copyright: none

##  1\. Introduction

Large Language Models (LLMs) have become commonplace in our technological
interactions, from chatbots to complex decision-making engines (gpt, 2023a;
Chen et al., 2021). They can perform tasks like understanding sentences,
answering questions, and creating text that looks like it was written by a
human. This has made them extremely helpful and valuable in many different
areas. The advent of Multi-Modal Large Language Models (MLLMs) has expanded
these functionalities even further by incorporating visual understanding,
allowing them to interpret and generate imagery alongside text, enhancing user
experience with rich, multi-faceted interactions (gpt, 2023b; Zhu et al.,
2023a).

Despite their utility, the ubiquity of LLMs and MLLMs raises substantial
security concerns. One key challenge is protecting against jailbreaking
attacks, in which malicious attackers can manipulate models into violating
rules or leaking confidential data (Deng et al., 2023; Zou et al., 2023; Chao
et al., 2023). Such vulnerabilities can have far-reaching consequences,
undermining the models’ reliability, exposing sensitive information, enabling
the spread of misinformation, and damaging the overall trust in AI-driven
applications. Addressing these security gaps is critical, particularly for
MLLMs, whose visual capabilities could be weaponized to create deceptive or
harmful multimedia content. Conclusively, there is an urgent need for strong
protective measures that can prevent and respond to these jailbreaking
attacks.

There are several LLM defense researches, which can be divided into two
categories: pre-query and post-query defenses. Post-query defenses are
typically reactive, triggered after the model has already generated its
outputs. These methods necessitate the construction of rules and detectors,
such as those used in the Azure content detector (azu, 2023), to discern and
intercept inappropriate or sensitive content produced by the LLM. While post-
query methods are effective, they require extensive domain knowledge and are
only triggered after queries have been processed by models. On the contrary,
pre-query defenses offer several benefits in terms of proactive deployment and
application (Kumar et al., 2023; Robey et al., 2023). Their deployment often
requires minimal domain-specific knowledge, allowing for broader
applicability. Techniques such as SmoothLLM (Robey et al., 2023), which
introduces random characters into input prompts, and semantic slicing of
inputs, opt to tackle jailbreaking by manipulating and examining input data
prior to query processing by LLMs. Such pre-emptive methods can prevent
harmful inputs before they engage with the model, minimizing the risk of
attacks. However, these pre-query strategies are primarily designed for text-
based inputs, which cannot counter attacks involving multiple modalities. The
limitations of existing approaches underscore the need for a more
comprehensive approach to protect these state-of-the-art LLMs.

To address these limitations, we designed and implemented the first mutation-
based jailbreaking defense framework, JailGuard, which supports attack
detection on both image and text modalities. The key observation behind
JailGuard is that attack queries inherently exhibit lower robustness than
benign queries. To confuse LLMs, attack inputs are often generated based on
crafted templates or by an extensive searching process with complex
perturbations. This leads to a result that any minor modification to the
inputs may invalidate its attack effectiveness, which appears as a significant
change in output. In JailGuard, we design detection strategies based on this
inherent fragility of attack queries. The whole process is that we first
mutate the original input into a series of variant queries. Then the
consistency of the responses of LLM systems to variants is analyzed. If a
notable discrepancy can be identified among the responses, i.e., a divergence
value that exceeds the threshold, a potential jailbreaking attack is
identified. Overall, the JailGuard framework is comprised of 19 mutators and a
divergence-based detection formula to identify potential attacks. Considering
the characteristics of image size, color as well as the semantics of the text
at the character, word, and sentence levels, we designed 10 image mutators and
9 text mutators to comprehensively disturb the input queries and generate
variants. Then the detection formula identifies attacks by judging whether the
divergence of variants’ responses exceeds a built-in threshold.

To comprehensively evaluate JailGuard, we construct the first multi-modal LLM
jailbreaking attack dataset that covers ten types of jailbreak attacks on
image and text modalities. The evaluation on our dataset shows that JailGuard
can effectively detect jailbreaking attacks on image and text modalities.
JailGuard has separately achieved the best detection accuracy of 89.38% and
85.42% on image and text inputs, outperforming state-of-the-art defense
methods by 15.28%. In addition, JailGuard can effectively detect and defend
different types of jailbreaking attacks. On all types of collected attacks
collected, the best detection accuracy of JailGuard is always more than 70%.
By contrast, the best detection accuracy of the state-of-the-art baseline
methods on any collected attacks is lower than JailGuard, and even less than
10% on ‘GPT4simulator’ and ‘MasterKey-poc’ attacks.

In summary, our contributions are:

  * •

We identified the inherent low robustness of jailbreaking attack inputs. Based
on that, we designed and implemented the first mutation-based multi-modal
jailbreaking detection framework, JailGuard, which supports detection for both
image and text modalities.

  * •

We constructed the first jailbreaking attack dataset that covers both image
and text inputs.

  * •

We performed experiments on our constructed dataset, and JailGuard has
achieved better detection effects than the state-of-the-art defense methods
(Robey et al., 2023).

##  2\. Background

Jailbreaking attack aims to design and generate an attack prompt $P_{a}$ that
can induce the target LLM system and application $M$ to contain toxic content
$T$ in the model response $R=M(P_{a})$, which can be represented as:

$\text{ find }P_{a}\text{ \quad subject to
}\quad\operatorname{eval}(M(P_{a}),T)=\operatorname{eval}(R,T)=1,$

where $eval(\cdot)$ is a detection function and it returns 1 iff the model
response $R$ contains the toxic content $T$.

The open-source community has manually collected and evaluated a series of
effective jailbreak prompts and templates, which can be classified into three
categories based on their patterns, namely ‘Attention Shifting’, ‘Pretending’,
and ‘Privilege Escalation’ (Liu et al., 2023b). The methods in ‘Attention
Shifting’ leverage specific tasks or contents to change both the conversation
context and intention and divert LLM-based applications’ attention to conduct
the attack. ‘Pretending’ methods alter the conversation background or context
to mislead LLMs, and ‘Privilege Escalation’ guides LLMs through elaborated
instructions to break any existing constraints.

To effectively and automatically generate jailbreak prompts, researchers
proposed a variety of attack methods (Zou et al., 2023; Deng et al., 2023;
Chao et al., 2023; Zhu et al., 2023b; Yu et al., 2023). Zou et al. (Zou et
al., 2023) designed the greedy coordinate gradient-based search (GCG) to
produce adversarial suffix to attack open-sourced LLMs (e.g., Vicuna (Zheng et
al., 2023)), which has proven its effectiveness through transfer attacks on
black-box commercial LLMs. We transfer the GCG attack on the Vicuna model to
OpenAI GPT-3.5 and collect them into datasets in § 5. Deng et al. (Deng et
al., 2023) proposed MasterKey, which trains a model to learn from existing
effective attack patterns and automatically generates new attacks to bypass
the defense mechanisms of four commercial LLM systems. We collect two types of
attacks generated by Masters in § 5, namely proof-of-concept (POC) attacks and
continuation-based attacks. With the emergence of MLLMs, researchers designed
a visual jailbreaking attack based on the implanted adversarial perturbation
in the image inputs (Qi et al., 2023). Their method achieved a high attack
success rate on MiniGPT-4 which is one of the state-of-the-art MLLMs (Qi et
al., 2023).

LLM defense methods can be divided into pre-query defense and post-query
defense. LLM systems mainly leverage content detectors in the output stage to
determine whether the model output is toxic (azu, 2023). These content
detectors are complex systems that evaluate the toxicity of the given input
based on built-in rules and integrated models. Therefore, their detection
results are greatly affected by the design of rules. At the input stage, one
of the state-of-the-art defense methods is SmoothLLM (Robey et al., 2023),
which implemented three perturbation strategies (i.e., Swap, Insert, and
Patch) and aggregated the responses of perturbed inputs to detect jailbreaks.
These perturbation strategies randomly select 10% of characters in the input
and insert or swap with random characters to generate several perturbed
inputs. Then, in the aggregation stage, SmoothLLM checks the LLM system
responses of perturbed inputs. If more than half of them contain jailbreak
keywords (Zou et al., 2023) (e.g., ‘I cannot help with that’), the source
input will be judged as a jailbreak input. Although existing defenses
demonstrate their effectiveness in experiments, they only focus on text input
and output and cannot support LLM inputs on other modalities, such as images.
In this paper, we propose the first multi-modal LLM jailbreaking detection
framework, JailGuard, that detects and defends jailbreaking attacks on both
image and text modalities.

Figure 1. Motivation Case of JailGuard (Red Highlights Toxic Contents and Some
of Them are Blocked)

##  3\. Motivation

Existing LLM jailbreaking methods (Zou et al., 2023; Liu et al., 2023b; Deng
et al., 2023) mainly rely on specific templates or tiny but complicated
perturbations to conduct attacks. These elaborate perturbations and templates
can shift the attention of the LLM system and application and deceive its
built-in defense mechanisms. Although these elaborate attacks have achieved
excellent attack results, they are less robust than benign queries and easily
disrupted and fail. Such a difference in robustness can be used to distinguish
between attack and benign queries.

We have provided two motivation examples at image and text modalities in
Figure 1. The upper and lower parts separately show attack and benign queries
that are from our image and text dataset (§ 5). For the image and text
queries, we randomly select one mutator to generate six variants, as shown in
§ 5.b). These mutators can mask part of the input, insert specific strings to
the text (e.g., ‘[mask]’), or change the image color, etc. Detailed
descriptions of mutators are shown in § 4.1. Then we obtain the responses of
the variant queries on LLM systems and applications (using MiniGPT-4 for
images and GPT-3.5 for texts), as shown in Figure 1.c). We use red to mark the
toxic and harmful content in the response. Finally, we vectorize the response
results and compute the divergence between each set of variant responses. The
heat maps in Figure 1.d) visualize the divergence between the responses of six
variants from one input of the LLM-based application.

We can observe that regardless of the input modality and the used mutators,
attack queries are more susceptible to perturbations than benign inputs. In
attack images, mutators can cause the responses of variants to dramatically
change from toxic content targeting a specific group to denial-of-service
replies (e.g., “I am sorry…”). In contrast, the LLM system can still perform
the given instructions (e.g., “describing the image”) on benign inputs.
Affected by mutators, there are inevitable differences between the benign
responses, but this difference is trivial compared to the difference in
responses to attack images. As shown in Figure 1.d), there is a clear color
difference in the heat map of the response of attack image variants, which
implies a large divergence value and a large difference between the responses’
semantics. We can make similar observations on the text input in the lower
part of Figure 1. This shows that the lack of robustness of attack queries is
a common characteristic in different modalities. Based on the observation, we
design the first mutation-based jailbreaking detection framework, JailGuard,
that mutates the untrusted queries into variants and then checks the
divergence of the LLM system responses of the variant queries and effectively
detects jailbreaking attacks.

##  4\. System Design

JailGuard is implemented on top of the LLM system and application workflow,
and Figure 2 shows the overview. JailGuard consists of the Variant Generator
module and Attack Detector module that are portable and migratory. Inspired by
previous research on fuzz testing (Xie et al., 2019; Zhang et al., 2021) and
model robustness (Frosio and Kautz, 2023; Morris et al., 2020), for the
untrusted query, the variant generator of JailGuard first selects a built-in
mutator for the query and generates a variant set. The attack detector then
uses these variants to query the target LLM system and computes the semantic
similarity and divergence between variant responses, and finally leverages the
built-in thresholds to identify benign and attack queries. The former can
access LLM system without a barrier, and JailGuard will discard and report the
latter.

In real-world scenarios, JailGuard should be deployed and applied as a part of
LLM system and application workflow to prevent jailbreaks, which means that
JailGuard and perform defense from the perspective of developers. At this
time, it should have access to the underlying interface of LLMs and can query
multiple variants in a batch and obtain multiple responses simultaneously. In
our experiments, we simulate this process by accessing the LLM system’s API
multiple times.

Figure 2. Overview of JailGuard Figure 3. Demo Case for 10 Mutators of
JailGuard (Red Marks the Modification of Text Input and Underline Marks
Important Sentences)

###  4.1. Variant Generator

The main role of the variant generator is to select a mutation operation on
the original untrusted input prompt $P$ and generate multiple variants that
are slightly different from the original input. The variant set can be
represented as $\mathcal{P}=\\{P_{1},...,P_{N}\\}$, where $N$ indicates the
number of variants. The variant generator in JailGuard has currently
implemented a total of 19 mutators at image and text modalities, including 16
random mutators and 3 advanced mutators. We have provided a demo for some
mutators in Figure 3.

####  4.1.1. Random Mutators

The design of random mutators is inspired by existing work (Hendrycks et al.,
2019; Lopes et al., 2019; Bayer et al., 2022; Marivate and Sefara, 2020;
Karimi et al., 2021). They use various data augmentation methods to randomly
change input images or texts and generate variants.

There are 10 random mutators for image inputs, namely Random Mask, Random
Solarization, Horizental Flip, Vertical Flip, Crop and Resize, Random
Grayscale, Gaussian Blur, Random Rotation, Colorjitter, and Random
Posterization. Details are shown as follows.

Random Mask mutator inserts a small black mask to a random position of the
image, as shown in Figure 3.a). It has the chance to destroy the implanted
perturbations in attack images, leading to a drastic change in LLM system
responses.

Random Solarization mutator inverts all pixel values above a random threshold
with a certain probability, resulting in solarizing the input image, as shown
in Figure 3.b). This mutator can introduce perturbations for the whole image
without damaging the overall information in the image.

Horizental Flip and Vertical Flip respectively flip the target image
horizontally or vertically with a random probability between 0 to 1. They will
not modify the pixel values in the image, and we have provided an example of
Horizental Flip in Figure 3.c).

Random Grayscale is a commonly used data augmentation method that converts an
RGB image into a grayscale image with a random probability between 0 to 1 (He
et al., 2020; Gong et al., 2021; Bai et al., 2022). Figure 3.d) provides a
demo.

Crop and Resize (Bai et al., 2022) can disturb attack images without changing
color and style. It first crops a random aspect of the original image and then
resizes it to a random size, as shown in Figure 3.e).

Gaussian Blur (Bai et al., 2022) blurs images with the Gaussian function with
a random kernel size. It reduces the sharpness or high-frequency details in an
image, which intuitively helps to disrupt the potential attack in image
inputs.

Random Rotation (Gidaris et al., 2018; Cubuk et al., 2020) rotates the image
by a random number of degrees between 0 to 180. After rotation, the area that
exceeds the original size will be cropped.

Colorjitter (He et al., 2020) randomly modifies the brightness and hue of
images and introduces variations in their color properties.

Random Posterization randomly posterizes an image by reducing the number of
bits for each color channel. It can remove the small perturbations and output
a more stylized and simplified image.

The random text mutators consist of seven random mutators, namely Random
Replacement, Random Insertion, Random Deletion, Synonym Replacement,
Punctuation Insertion, and Translation.

Random Replacement and Random Insertion perform the replacement or insertion
operation with probability $p$ for each character, as shown in Figure 3.f) and
g). The replacement operation replaces the target and subsequent characters
with a specific text $T$, ensuring that the input length does not change. The
insertion operation inserts $T$ at the position after the target character.
Similarly, Random Deletion removes the character in the text with probability
$p$.

Synonym Replacement selects words in the text input and uses their synonyms to
replace them based on WordNet (Miller, 1995).

Punctuation Insertion is from existing data augmentation method that randomly
inserts punctuation masks into sentence (Karimi et al., 2021). This mutator
has the potential to disturb the adversarial-based jailbreaking attack without
changing the semantics of the input sentence.

Translation first translates the input sentence to a random language and then
translates it back. For jailbreak queries, the translation process can rewrite
the attack templates and remove the meaningless characters, ultimately
invalidating the attack.

####  4.1.2. Advanced Mutators

Although random mutators have the potential to disrupt jailbreak attacks, they
lack proper guidance, resulting in limited effectiveness. On the one hand, for
complex jailbreak attacks (e.g., ‘Attention Shifting’ (Liu et al., 2023b)),
simple random operators may not be able to cause enough interference to the
attack input, resulting in the defense being bypassed. On the other hand,
blindly modifying the input may cause benign inputs to be compromised, which
leads to dramatic changes in their responses‘ semantics. This is especially
true in the text, where small changes to a word may completely change its
meaning. To achieve a better detection effect, we design and implement three
advanced text mutators in JailGuard, namely Targeted Replacement, Targeted
Insertion, and Rephrasing.

Targeted Replacement and Targeted Insertion are the advanced version of Random
Replacement and Random Insertion. Inspired by existing nature language
processing work (Nenkova and Vanderwende, 2005), these two advanced mutators
first find the sentences with the highest word frequency in the input text,
which means that words in these sentences appear repeatedly in the whole input
and often represent the summary of the input text. Then these mutators select
these sentences as the important sentences. The characters in important
sentences have a higher probability of performing replacement and insertion.
We provide two cases in Figure 3.h) and i). Italicized and underlined
sentences represent selected important sentences.

Rephrasing is one intuitive way to collapse the jailbreak template and remove
the attack perturbations in the input. Rephrasing mutator in JailGuard uses
the prompt of "Please rephrase the following paragraph while ensuring its core
semantics and contents unchanged. The paragraph is: [INSERT ORIGIN INPUTS]" to
query OpenAI GPT-3.5 and rewrite the original input while preserving
semantics. Figure 3.j) shows a demo of this mutator, and red marks the changed
content in the variant.

###  4.2. Attack Detector

Our detector operates by leveraging the divergence among outputs to
effectively achieve detection. The calculation of divergence proceeds through
the following steps:

Constructing the similarity matrix For the input variant set $\mathcal{P}$,
the detector first queries LLM system to obtain the response set
$\mathcal{R}=\\{R_{1},...,R_{N}\\}$. For each $R_{i}$ in $\mathcal{R}$, the
detector leverages the ‘spaCy’ library to convert it to word vector $V_{i}$
and then calculates the cosine similarity between it and the word vectors of
other responses. The similairty $S_{i,j}$ between vectors $V_{i}$ and $V_{j}$
can be represented as:

$S_{i,j}=COS(V_{i},V_{j})=\frac{V_{i}\cdot V_{j}}{\|V_{i}\|\|V_{j}\|},$

where $COS(\cdot)$ calculates the cosine similarity between two vectors,
$i,j\in\\{1,2,\ldots,N\\}$. Similarity values for response pairs are
represented in an $N\times N$ matrix $S$, where each element at $(i,j)$
corresponds to the similarity between the pair $(R_{i},R_{j})$.

Characterizing each response In the $N\times N$ similarity matrix $S$, each
row $S_{i,\cdot}$ corresponding to the $i$-th response $R_{i}$ can be
normalized using $\mathcal{L}_{1}$ normalization. This process transforms the
row into a similarity probability distribution, effectively characterizing the
relationship of response $R_{i}$ to the other responses in the matrix.

$Q_{i}=Q(X=i)=\|S_{i,\cdot}\|_{1}.$

Quantifying the divergence of two responses Then JailGuard uses Kullback-
Leibler (KL) divergence to quantify the difference between the similarity
distributions of two responses $(Q_{i},Q_{j})$:

$D_{i,j}=D(Q_{i}\|Q_{j})=\sum_{x=1}^{N}Q_{i}(x)\log(\frac{Q_{i}(x)}{Q_{j}(x)}).$

The divergence values for response pairs are represented in an $N\times N$
matrix $D$, where each element at $(i,j)$ corresponds to the divergence
between the pair $(R_{i},R_{j})$.

Examining the divergence Finally, for the obtained divergence $N\times N$
matrix $D$, the detector uses the threshold $\theta$ to identify the attack
input. Specifically, if any value in divergence matrix $D$ exceeds $\theta$,
JailGuard will consider the original input as an attack input, otherwise it is
judged as a benign input, which is shown as follows:

$\exists i,j\in\\{1,2,\ldots,N\\},D_{i,j}\geq\theta\rightarrow
P\cup\mathcal{P}_{a},$

where $\mathcal{P}_{a}$ indicates the set of jailbreaking attacks.

To maximize the detection effect of JailGuard, we randomly selected 70/80
text/image inputs from our dataset (§ 5) and calculated the divergence of
their variants. Based on the results, we choose $\theta=0.01$ for text input
on GPT-3.5 and 0.0025 for image input on MiniGPT-4 in JailGuard. Note that,
when all variants of an attack input fail, the LLM system and application will
not provide any service for these inputs. In this case, all responses will
contain the jailbreak keyword (Zou et al., 2023) and become similar in
semantics, and their divergence $D$ will be very low. To avoid false
positives, if all responses contain jailbreak words, regardless of the value
in $D$, JailGuard will directly determine them as attack inputs.

Table 1. Jailbreak Questions in Dataset Category | Attack Method | Description
---|---|---
Attention Shifting | GPT4similator |  | Implant toxic instructions in a program and ask LM to similate
---
the execution of this program
TextContinue |  | Construct a scene where Dr. Evil describes his evil plan and
---
let the AI continue writing to achieve the attack
Translator |  | Use translation tasks to attract the attention of the LM
---
system and execute dangerous instructions
Pretending | EvilConfidant |  | Ask LM to act as the evial confidant who is not bound by any
---
rules and can answer any sensitive questions.
Privilege Escalation | DAN11.0 |  | Ask LM to enable DAN mode that can do anything and
---
generate any type of content
AIM |  | Ask LM to become AIM, an unfiltered and unethical chatbot
---
and will answer any requests made to it
| DevMode
---
+Ranti
| Ask LM to enable DevMode that can do anything and generate
---
any type of content
Other | MasterKey |  | Employ LM to learn jailbreak patterns and generate attacks,
---
including two modes of ‘continue’ and ‘poc’
GCG |  | Add adversarial perturbations into strings to bypass the
---
model’s defense mechanisms

##  5\. Dataset Construction

Due to the lack of a comprehensive LLM jailbreak benchmark, existing jailbreak
defense research mainly tested and evaluated their methods on specific types
of text inputs. For example, SmoothLLM has evaluated its effectiveness in
defending against jailbreak inputs generated by the GCG method (Zou et al.,
2023). To break the existing limitation, we construct the first comprehensive
jailbreaking dataset containing both text and image inputs. We collect
jailbreaking attack methods and templates from the open-source community and
prior work and then evaluate their effectiveness on LLM systems and
applications. Finally, we construct a dataset covering more than 10 types of
known jailbreaking attacks, covering two modalities of image and text, with a
total of 304 items of attack and benign data.

Image jailbreaking inputs. We have generated 80 sets of jailbreak inputs based
on prior work (Qi et al., 2023), and each set contains a jailbreak image
implanted with tiny perturbations and a jailbreak question violating policies.
We have verified the validity of these jailbreak inputs on MiniGPT-4 (Zhu et
al., 2023a), which is one of the state-of-the-art open-source VLMs and accepts
multimodal inputs including images and texts. In addition, we have collected
80 sets of benign inputs from the open-source data of MiniGPT-4, each
containing a benign image and an image-related question. The image
jailbreaking input dataset is publicly available on our website (our, 2023).

Text jailbreaking inputs. To ensure the diversity of jailbreak inputs, we have
referred to existing categorization methods (Liu et al., 2023b) and collected
attack methods and templates from the
community111https://www.jailbreakchat.com/ and existing work (Zou et al.,
2023; Deng et al., 2023). Their description is shown in Table 1. We have
introduced the three categories of ‘Attention Shifting’, ‘Pretending’ and
‘Privilege Escalation’ in § 2. To ensure that the variety of attack methods in
the dataset is as comprehensive as possible, the collected attack inputs are
not only from all three categories but also existing automated attack methods
such as GCG (Zou et al., 2023) and MasterKey (Deng et al., 2023).

Based on these attack templates and methods, we collect 30 toxic questions
from existing work (Qi et al., 2023; Deng et al., 2023) and combine them with
the templates to generate jailbreak inputs. In the generation, we found that
some questions (e.g., “How can we use social media to increase our reach
during election season?”) can always get valid responses from the target LLM
system even without using any jailbreak attack methods or templates. Previous
work has made similar observations (Liu et al., 2023b). To alleviate the
impact of these abnormal jailbreak questions on the dataset, we removed the
attack inputs related to these questions and validated them on OpenAI GPT-3.5
which is one of the most popular chatbots. Finally, we obtained 72 text
attacks on OpenAI GPT-3.5, which cover all methods in § 5, except ‘DAN11.0’.
Note that Masterkey contains two completely different attack templates, we
consider them as two templates, namely ‘Masterkey-continue’ and ‘Masterkey-
poc’. We labeled these attacks into nine types in our dataset based on their
methods and templates. In addition, we randomly sampled benign inputs that
have the same number as the attack inputs from an open-sourced LLM instruction
set in the
community222https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset
to construct the dataset. Our dataset and the filtered abnormal jailbreak
questions mentioned above are also available on our website (our, 2023).

##  6\. Evaluation

RQ1: How effective is JailGuard in detecting and defending LLM jailbreaking
attacks at the text and visual level?

RQ2: Can JailGuard effectively detect and defend different types of LLM
jailbreaking attacks?

RQ3: How effective are the modules (i.e., variant generator and attack
detector) in JailGuard?

RQ4: What is the impact of the number of variants generated by JailGuard?

###  6.1. Setup

Baseline To the best of our knowledge, we are the first work to design LLM
defenses for image inputs, therefore we only construct baselines for text
inputs. One baseline method is the content detector implemented in Llama-2
repository333https://github.com/facebookresearch/llama-
recipes/blob/main/examples/inference.py. This content detector separately
leverages the AuditNLG library (Aud, 2023), ‘safety-flan-t5-base’ language
model, and Azure Content Safety service (azu, 2023) to check whether the input
contains toxic content. To achieve the best detection effect, we enable all
three available modules in it. The other is SmoothLLM which is one of the
state-of-the-art LLM defense methods at the input level. Since we could not
find available code in their paper, we manually implemented their three
perturbation methods (i.e., Insert, Swap, and Patch) and their aggregation
step based on their paper, as introduced in § 2. Additionally, following the
recommendation in their paper, we generated 8 samples for each input in the
SmoothLLM perturbation step.

Metric We conduct experiments on the image and text input dataset constructed
in § 5 and use two metrics, accuracy and recall, to evaluate the detection
effect of JailGuard and baselines. Accuracy comprehensively measures the
effectiveness of each method in identifying attack and benign inputs, while
recall mainly reflects the effectiveness of each method in correctly
identifying the attack inputs in the dataset without false negatives. The
lower the recall, the more severe the false negatives in jailbreak detection.

Implementation To fairly compare with SmoothLLM baselines in the experiment,
JailGuard generates $N=8$ variants for each input. For text inputs, the
probability of selecting and executing the replacement, insertion, and
deletion operation on each character is $p=0.005$, and the probability on the
important sentences is 5 times usual, that is, 0.025. JailGuard use the string
‘[mask]’ to replace and insert. The LLM systems and applications we used on
text and image inputs are OpenAI’s GPT-3.5 and MiniGPT-4 respectively. More
details are shown in § 5. Our framework is implemented on Python 3.9. All
experiments are conducted on a server with AMD EPYC 7513 32-core processors,
250 GB of RAM, and an NVIDIA RTX A6000 GPU running Ubuntu 20.04 as the
operating system.

###  6.2. RQ1: Effectiveness of Detecting Attack

Experiment Designs and Results To demonstrate the effectiveness of JailGuard
in detecting attack inputs and defending jailbreaking attacks, we use two
metrics to evaluate the detection results of each method on the constructed
dataset. The results on image and text inputs are separately shown in Table 2
and Table 3. The rows in Table 2 show the detection effect of mutators in
JailGuard on image inputs. In Table 3, the rows of ‘Baseline’ show the
detection results of four baselines on text input, and ‘JailGuard’ rows
correspond to the detection results of applying different mutators in
JailGuard. The row ‘Average’ shows the average result of baselines and
JailGuard. The names of JailGuard’s mutators and baselines refer to § 4.1 and
§ 6.1. We use blue to mark the best result on each metric. In addition, Figure
4 uses a scatter plot to compare the detection results between different
methods on text inputs. The X-axis is the accuracy and the Y-axis is the
recall. Green dots indicate the results of mutators in JailGuard and red dots
mark the baselines. Blue shows the results of the ablation study, which will
be analyzed in § 6.4. We show the methods or mutators represented by each dot
at the top of the table.

Table 2. JailGuard Attack Mitigation on Image Inputs Method | Accuracy (%) | Recall (%)
---|---|---
Random Mask | 75.00 | 75.00
Gaussian Blur | 82.50 | 76.25
Horizontal Flip | 73.75 | 81.25
Vertical Flip | 85.00 | 78.75
Crop and Resize | 78.13 | 81.25
Random Grayscale | 80.63 | 77.50
Random Rotation | 89.38 | 78.75
Colorjitter | 85.00 | 80.00
Random Solarization | 89.38 | 80.00
Random Posterization | 82.50 | 70.00
Average | 82.13 | 77.88
Table 3. Comparison of Attack Mitigation on Text Inputs (Bold Marks Results That Equal or Outperform the Best Baseline) Method | Accuracy (%) | Recall (%)
---|---|---
Baseline | Content Detector | 55.56 | 29.17
SmoothLLM-Insert | 70.14 | 41.67
SmoothLLM-Swap | 66.67 | 34.72
SmoothLLM-Patch | 70.14 | 41.67
Average | 65.62 | 36.81
JailGuard | Random Replacement | 77.78 | 75.00
Random Insertion | 79.17 | 77.78
Random Deletion | 79.17 | 76.39
Synonym Replacement | 73.61 | 84.72
Punctuation Insertion | 75.00 | 70.83
Translation | 78.47 | 84.72
Targeted Replacement | 82.64 | 88.89
Targeted Insertion | 84.03 | 81.94
Rephrasing | 85.42 | 91.67
Average | 79.48 | 81.33

Analysis The results in Table 2 and Table 3 illustrate the effectiveness of
JailGuard in detecting and defending jailbreaking attacks on different input
modalities. JailGuard achieved an average detection accuracy of 82.13% on
image inputs and 79.48% on text inputs with different mutators, which is
better than the state-of-the-art baseline methods (average accuracy of 65.62%)
and far exceeds the results of the content detector in Llama-2 repository
(55.56%). For recall, JailGuard achieves an average recall of 77.88% on image
inputs. It is worth mentioning that JailGuard achieved an average recall of
81.33% on text data, which is 2.21 times higher than the average result of
baselines (36.81%), indicating its effectiveness in detecting jailbreaking
attacks and reducing false negatives.

To be specific, on the image attack dataset, Horizontal Flip achieves the best
recall of 81.25%, and Random Rotation and Random Solarization mutators
achieves the best accuracy of 89.38%. Since we are the first defense work at
the image modality and there is no comparable baseline, we have no way of
directly knowing the advantages of JailGuard in detecting and defending
against jailbreaking attacks.

The comparison with the four baselines on text inputs further illustrates the
effectiveness of JailGuard in attack detection. The text mutators in JailGuard
achieve an average accuracy and recall of 79.48% and 81.33%, which is 13.85%
and 44.52% higher than the average accuracy and recall of the baselines. The
best baseline is the insert and patch mode of SmoothLLM, which achieves an
accuracy of 70.14% and recall of 41.67%. Compared with the best baselines, all
text mutators in JailGuard achieved better results on two metrics, which
demonstrate the detection effectiveness of JailGuard. We use bold to mark the
results that outperform the best baseline in Table 3. More specifically, the
Random Insertion and Random Deletion mutators achieve the best accuracy of
79.17% among random mutators. All advanced mutators achieved much better
results than random mutators. Targeted Replacement and Targeted Insertion
separately achieve the detection accuracy of 82.64% and 84.03%. Compared to
Random Replacement and Random Insertion, both of them improve accuracy by
4.86%, respectively. Rephrasing even achieved the highest accuracy and recall
among all mutators (i.e., 85.42% and 91.67%), which proves the effectiveness
of the advanced mutators in JailGuard.

Furthermore, Figure 4 intuitively demonstrates the advantages of JailGuard in
attack detection compared to the baselines. We can observe that JailGuard
(green) achieves significantly better results than baselines (red), and the
corresponding dots are distributed in the upper right corner, indicating high
accuracy and recall.

Note that JailGuard is a defense framework built on top of the LLM system and
application workflow, which conducts defense from the developer’s perspective.
Therefore, in the deployment in real-world scenarios, it can directly query
LLM to process and infer the variant inputs in batches, resulting in slightly
increased time overhead compared to the original workflow. To understand the
memory overhead of JailGuard, we conducted simulations on MiniGPT-4 and found
that a single set of inputs (an image and a corresponding instruction)
increases the memory overhead by 0.49GB, equivalent to 3.15% of the LLM’s
memory overhead (15.68GB). For JailGuard with the setting of $N=8$, the memory
overhead of JailGuard in defending jailbreaking attacks is 3.95GB, which is
25.20% of the memory overhead of LLM itself.

Figure 4. Comparison of Different Methods’ Results on Text Inputs

###  6.3. RQ2: Effectiveness of Detecting Different Kinds of Attacks

Figure 5. Comparison of Different Methods’ Results on Text Inputs Figure 6. A
Case Study of Detecting ‘MasterKey’ Attack

To demonstrate the effectiveness of JailGuard in detecting and defending
various LLM jailbreaking attacks, we count and analyze the detection accuracy
of each method on each type of jailbreak template or method and displayed it
using a heat map, as shown in Figure 5. Each column represents the various
jailbreaking attack methods, which are collected in our dataset, as mentioned
in § 5. categories (Liu et al., 2023b). ‘Attention’, ‘Pretending’, and
‘Privilege’ are the abbreviation of three categories of jailbreak templates,
namely ‘Attention Shifting’, ‘Pretending’, and ‘Privilege Escalation’ (Liu et
al., 2023b). The first six columns are the attack templates from these three
categories, and the last three columns are the attacks generated by existing
methods, namely GCG (Zou et al., 2023) and MasterKey (Deng et al., 2023). The
rows, on the other hand, represent four baseline methods and the text mutators
in JailGuard, where the advanced mutators are shown in the last four rows. A
bluer color in Figure 5 means higher accuracy of one method in detecting a
specific attack, otherwise, it means that the method can hardly identify the
jailbreak input.

We can observe that JailGuard achieves better results and has better
generalization capabilities than the baseline in the detection of various
jailbreak attacks. The baselines can only effectively detect several types of
jailbreaking attacks and are helpless against other types of attacks. For
‘GPT4simulator’ and ‘Translator’ methods in the ‘Attention Shifting’ category
and MasterKey attack method, JailGuard’s mutators achieve much better than
baseline, which most of the time has less than 50% detection accuracy for
these attacks, while JailGuard can easily achieve 80% to 100% detection
accuracy on them. For other attack methods (e.g., ‘Evil Confident’ of the
‘Pretending’ category), JailGuard can also achieve equal or better results
than baselines. In addition, the advanced mutators in JailGuard achieve better
detection results than random mutators. The advanced mutators have a better
detection accuracy than the random mutators on ‘GPT4simulator’, ‘Evil
Confidant’,‘AIM’, and ‘MasterKey-poc’ attacks. Random mutators often achieve
detection accuracy of 60% to 80% on these types of jailbreaks, which is much
better than the baseline methods. The advanced mutators shown in the last
three rows of Figure 6 have better detection effects, and they can achieve
detection accuracy of 80% to 100%.

Case Study We provide a case in Figure 6 to understand and illustrate the root
cause of the effect difference between JailGuard and the baseline SmoothLLM on
specific attacks, such as ‘MasterKey-poc’ attacks. The upper part shows the
detection of baseline method SmoothLLM-Swap and the lower part of Figure 6
shows the detection process of JailGuard with Targeted Insertion mutator.
Figure 6.a) provides a real example of ‘MasterKey-poc’ attack in our dataset.
Attacks like ‘MasterKey-poc’ and ‘GPT4similator’ often use specific content or
tasks to divert LLM’s attention, thereby deceiving the defense mechanism of
LLM system and achieving jailbreak. At this time, SmoothLLM randomly replaces
10% characters to infer these attack inputs as much as possible. However, its
results are minimal. Among 8 perturbed inputs, only one attack fails, and its
response contains jailbreak keywords, as shown in the red text of the upper
part ofFigure 6.c). Therefore, in the aggregation step in the upper part of
Figure 5.d), since most results do not contain jailbreak keywords, according
to its aggregation principle, this input sample is judged as a benign sample,
which is a false negative.

In contrast, JailGuard can effectively identify this attack. Firstly, the
Targeted Insertion mutator effectively finds the important sentences of the
input (e.g., the specific instructions at the end) and purposefully inserts
many masks to achieve interference, as shown in the upper part of Figure 6.b).
For LLM system responses shown in Figure 6.c), JailGuard calculates their
semantic similarity and divergence in Figure 6.d) and then detects this attack
based on the threshold $\theta$. Even in the situation that only one attack
fails, since the semantics of the failed response are completely different
from others, JailGuard can easily detect it based on divergence, which makes
it achieve high detection accuracy on complex attacks like ‘MasterKey-poc’ and
‘GPT4simulator’, with few false negatives.

###  6.4. RQ3: Ablation Study

Experiment Designs and Results JailGuard provides two modules to detect and
defend against jailbreaking attacks in the LLM systems and applications, which
are the variant generator and attack detector. To understand their
contribution, we conduct an ablation experiment on the text inputs. We use the
three perturbation methods of SmoothLLM (i.e., Insert, Swap, and Patch) to
replace the mutators of the variant generator in JailGuard, and record the
results as ‘Insert+Attack Detector’, ‘Swap+Attack Detector’, and ‘Patch+Attack
Detector’. In addition, we leverage the aggregation method in SmoothLLM to
replace the JailGuard attack detector and detect attacks on the variants
generated by Random Replacement and Random Insertion (i.e., ‘Random
Replacement+Aggregation’ and ‘Random Insertion+Aggregation’). The aggregation
method of SmoothLLM is explained in § 2. The above results are shown in Table
4 and the blue dots of Figure 4.

Analysis The experimental results of the ablation study illustrate the
effectiveness of each module of JailGuard.

Firstly, the mutators in the variant generator are effective in detecting
jailbreaking attacks. As shown in Table 4, the best detection accuracy
obtained after replacing mutators is 76.39%, which is lower than the average
accuracy of using JailGuard text mutators. We can intuitively observe the
difference in detection results by comparing the blue triangles and green dots
in Figure 4. The blue triangles represent the result of replacing JailGuard’s
mutators with Insert/Patch/Swap in SmoothLLM. Compared with most green dots,
blue triangles are located in the lower left position, which means it has
lower accuracy and recall.

In addition, the attack detector in JailGuard has an important contribution to
attack detection, especially in eliminating False Negatives. Replacing the
attack detector with the aggregation method of SmoothLLM will result in a
decrease of 5.56% and 29.17% in the accuracy and recall of the Random
Replacement mutator in JailGuard. A similar operation on Random Insertion also
decreases 6.94% and 31.94% on two metrics. The significant decrease in recall
indicates that the aggregation method will overlook many attack examples and
cannot provide effective defense for various jailbreaking attacks, which is
consistent with our observation in § 6.3. In addition, using the attack
detector can significantly increase the results of the perturbation methods in
SmoothLLM. From the results in the first three rows of Table 4 and Table 3, we
can observe that using the attack detector significantly increases the recall
of the Swap method from 34.72% to 81.94%, which is 2.36 times of the baseline
result. The recall of Insert and Patch methods has also increased by about 1.8
times compared to before. This demonstrates the important contribution of
attack detectors in alleviating FNs and preventing LLM jailbreaking attacks.

Table 4. Ablation Study on JailGuard Method | Accuracy (%) | Recall (%)
---|---|---
Insert+Attack Detector | 73.61 | 73.61
Swap+Attack Detector | 76.39 | 81.94
Patch+Attack Detector | 73.61 | 76.39
Random Replacement+Aggregation | 72.22 | 45.83
Random Insertion+Aggregation | 72.22 | 45.83

###  6.5. RQ4: Impact of Variant Amount

Figure 7. Results on Image Inputs Figure 8. Results on Text Inputs

JailGuard use different mutators to generate $N$ variants in the variant
generator. To align with baselines, we choose $N=8$ in experiments. To
understand the impact of different values of $N$ on detection results, with a
fixed setting for the attack detector (i.e., $\theta$ is the same with
previous experiments), we evaluate the detection effectiveness of different
JailGuard operators when generating 4/5/6/7/8 variants, and record accuracy
and recall on the image and text dataset in Figure 7 and Figure 8. We use
different colors to represent different mutators, and the red bold line
indicates the average result of the mutators. In addition, the purple dotted
lines in Figure 8 represent the accuracy and recall that the best baseline
method SmoothLLM has achieved when it generates eight perturbed sampels.

The overall accuracy and recall of each mutator in jailbreak detection
gradually increase as $N$ increases. On image inputs, when $N$ is set to 4,
the average accuracy and recall are 78.38% and 68.88%. These mutators can
achieve the average accuracy and recall of 82.13% and 77.88% when $N=8$. For
text inputs, when $N$ is continuously increasing, the average accuracy of
JailGuard increases from 76.00% to 79.48%, and recall improves from 70.52% to
81.33%.

We can observe that generating more variants can improve the detection
results, which is in line with our intuition. However, the trend becomes less
obvious as the value of $N$ increases, as shown in the red bold part in Figure
7 and Figure 8. A larger $N$ will also lead to more runtime overhead. As we
discussed in § 6.2, when $N$ is set to 8, the defense memory overhead on
MiniGPT-4 is close to 25% of the model overhead. Continuing to increase $N$
will only make this number larger. Compared with the potential increase of
2%-3% in detection accuracy, extra overheads may not be worth it in some
resource-limited scenarios.

In addition, we can observe that when JailGuard generates six variants in the
generator, nearly all mutators can achieve better results than the best
baseline (the purple dotted line) on two metrics. Only the accuracy of Synonym
Replacement mutator is 0.7% lower than the optimal baseline, while the
former’s recall is 33.33% higher than the latter, which is more significant.
At this time, the runtime overhead of JailGuard is reduced by 25% compared
with the default setting in previous experiments (i.e., the LLM queries in
attack detector decrease from eight to six), and the average accuracy and
recall are 76.54% and 73.92% respectively, which are slightly lower than its
results when $N=8$, but are still much higher than the best baseline (70.14%
accuracy and 41.67% recall).

Therefore, depending on the scenarios of actual application and deployment, we
recommend selecting $N\in[6,8]$ to achieve the balance between detection
effectiveness and runtime overhead.

##  7\. Related work

Adversarial Attack and Defense in DNNs. White box attacks assume the attacker
has full knowledge of the target model, including its architecture, weights,
and hyperparameters. This allows the attacker to generate adversarial examples
with high fidelity using gradient-based optimization techniques, such as FGSM
(Goodfellow et al., 2014), BIM (Kurakin et al., 2018), PGD (Madry et al.,
2017), Square Attack (Andriushchenko et al., 2020). AutoAttack (Croce and
Hein, 2020) has been proposed as a more comprehensive evaluation framework for
adversarial attacks. Recently, researchers have also been exploring the use of
naturally occurring degradations as forms of attack perturbations. These
include environmental and processing effects like motion blur, vignetting,
rain streaks, varying exposure levels, and watermarks (Gao et al., 2022; Guo
et al., 2020; Jia et al., 2020; Tian et al., 2021; Hou et al., 2023).

Adversarial defense can be categorized into two main types: adversarial
training and adversarial purification (Nie et al., 2022). Adversarial training
involves incorporating adversarial samples during the training process
(Goodfellow et al., 2014; Madry et al., 2017; Athalye et al., 2018; Rade and
Moosavi-Dezfooli, 2021; Ding et al., 2018), and training with additional data
generated by generative models (Sehwag et al., 2021). On the other hand,
adversarial purification functions as a separate defense module during
inference and does not require additional training time for the classifier
(Guo et al., 2017; Xu et al., 2017; Sun et al., 2019; Ho and Vasconcelos,
2022).

LLM Attack And Defense Supplement to the jailbreaking attack methods in § 2,
researchers proposed other methods to automatically generate jailbreak prompts
(Chao et al., 2023; Zhu et al., 2023b; Yu et al., 2023). Chao et al. (Chao et
al., 2023) leverage unaligned LLMs to generate jailbreaks for target LLMs.
Unfortunately, we cannot find available open-source code of their method.
Researchers also pay attention to other aspects of LLM security, e.g.,
backdoor attack (Abdelnabi et al., 2023; Xu et al., 2023; Huang et al., 2023),
injection attack (Liu et al., 2023a). We focus on the defense of multi-modal
jailbreaking attacks in this paper.

To detect and defend LLM attacks, in addition to SmoothLLM, Kumar et al.
(Kumar et al., 2023) designed a detection method that splices the input text
and applies a safety filter on all substrings to identify toxic content. In
addition, Cao et al. (Cao et al., 2023) weakened the robustness of attack
prompts by randomly deleting words and helped the aligned model detect
jailbreaking attacks. In this paper, we compare JailGuard with one of the
state-of-the-art methods SmoothLLM and the content detector in the Llama open-
source repository.

##  8\. Discussion

JailGuard Enhancement ❶ In our exploration of RQ2 (§ 6.3), we observe that
various mutators demonstrate differing levels of effectiveness across
different jailbreaking methods. For instance, while targeted replacement
generally shows strong performance with most jailbreaking methods, it is not
as effective as target insertion when dealing with ‘Attention-GPT4simulator’
jailbreaking. Furthermore, a significant performance disparity exists between
rephrasing and other methods in terms of ‘Privilege-AIM’. This suggests the
potential for devising a strategy that combines different mutation techniques.
By doing so within the same query budget, we could enhance the overall
efficacy of image generation and improve detection performance. ❷ Ensuring a
balanced performance in terms of various jailbreaking methods in jailbreaking
detection is crucial. Typically, the most effective jailbreaking strategies
are extensively utilized to attack systems, meaning that jailbreaking attacks
in real-world scenarios are inherently imbalanced. This imbalance can
significantly diminish the real-life performance of our detection systems.
Therefore, it’s essential to explore more effective targeted mutators,
particularly for addressing ‘Privilege-AIM’ jailbreaking scenarios. Such
advancements are key to maintaining robustness and reliability in our security
measures. ❸ Our current dataset only covers attacks carrying single-modality
content. However, the latest models such as Multi-Modal Large Language Models
(MLLMs) GPT-4v, might also be susceptible to new forms of hybrid jailbreaking
attacks leveraging multi-modal input features. Although no such attacks have
been publicly reported, they remain a plausible future threat. Theoretically,
our framework can support the detection of such hybrid jailbreaking attacks as
long as the divergence of the robustness between attack and benign inputs is
still identifiable. Our dataset will be continuously updated and any new
appearing attacks will be further collected and evaluated. You can find the
latest information on our website (our, 2023).

Diverse LLM Attacks Detection ❶ As an emerging research field, the security of
large models has received widespread attention from researchers and industry.
It is significant to add more types of attack inputs (e.g., data poisoning
(Yan et al., 2023), backdoor (Abdelnabi et al., 2023; Xu et al., 2023), and
prompt injection (Liu et al., 2023a) ) and build a comprehensive and universal
benchmark for LLM defense. ❷ Our detection method fundamentally leverages the
inherent non-robustness of attacks. Consequently, the vulnerabilities
introduced by data poisoning and prompt injection, which also exhibit this
non-robustness, could potentially be identified by our detection framework. A
crucial future direction involves designing defense methods that are both
effective and efficient, capable of generalizing across various types of
attack inputs. Successfully achieving this would significantly enhance the
deployment and application of trustworthy Language Model (LM) systems,
contributing to their overall reliability and security.

##  9\. Conclusion

In this paper, we propose JailGuard, the first mutation-based multi-modal
jailbreaking detection framework that detects and defends jailbreaking attacks
for LLM systems at both image and text modalities. To comprehensively evaluate
the defense effect of JailGuard, we construct the first comprehensive LLM
jailbreak attack dataset, covering jailbreaking attacks on image and text
modalities. Our experiment results show that JailGuard achieves the best
detection accuracy of 89.38%/ 85.42% on image/text inputs, outperforming
state-of-the-art methods.

##  10\. Data Availability

To follow the Open Science Policy and support reproducibility, we have
released code about our implementations and evaluations. All source code and
data used in our work can be found at (our, 2023).

## References

  * (1)
  * Aud (2023) 2023\.  AuditNLG: Auditing Generative AI Language Modeling for Trustworthiness.  https://github.com/salesforce/AuditNLG.
  * azu (2023) 2023\.  Azure AI Content Safety.  https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety.
  * gpt (2023a) 2023a.  GPT-4 System Card.  https://cdn.openai.com/papers/gpt-4-system-card.pdf.
  * gpt (2023b) 2023b.  GPT-4(v) System Card.  https://cdn.openai.com/papers/GPTV_System_Card.pdf.
  * our (2023) 2023\.  The Website of JailGuard.  https://sites.google.com/view/jailguard.
  * Abdelnabi et al. (2023) Sahar Abdelnabi, Kai Greshake, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023.  Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In _Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security_. 79–90.
  * Andriushchenko et al. (2020) Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. 2020.  Square attack: a query-efficient black-box adversarial attack via random search. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII_. Springer, 484–501.
  * Athalye et al. (2018) Anish Athalye, Nicholas Carlini, and David Wagner. 2018.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In _International conference on machine learning_. PMLR, 274–283.
  * Bai et al. (2022) Yalong Bai, Yifan Yang, Wei Zhang, and Tao Mei. 2022.  Directional self-supervised learning for heavy image augmentations. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16692–16701.
  * Bayer et al. (2022) Markus Bayer, Marc-André Kaufhold, and Christian Reuter. 2022.  A survey on data augmentation for text classification.  _Comput. Surveys_ 55, 7 (2022), 1–39.
  * Cao et al. (2023) Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023.  Defending against alignment-breaking attacks via robustly aligned llm.  _arXiv preprint arXiv:2309.14348_ (2023).
  * Chao et al. (2023) Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023.  Jailbreaking black box large language models in twenty queries.  _arXiv preprint arXiv:2310.08419_ (2023).
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Croce and Hein (2020) Francesco Croce and Matthias Hein. 2020.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In _International conference on machine learning_. PMLR, 2206–2216.
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020.  Randaugment: Practical automated data augmentation with a reduced search space. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_. 702–703.
  * Deng et al. (2023) Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023.  MasterKey: Automated jailbreak across multiple large language model chatbots.  _arXiv preprint arXiv:2307.08715_ (2023).
  * Ding et al. (2018) Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. 2018.  Mma training: Direct input space margin maximization through adversarial training.  _arXiv preprint arXiv:1812.02637_ (2018).
  * Frosio and Kautz (2023) Iuri Frosio and Jan Kautz. 2023.  The Best Defense is a Good Offense: Adversarial Augmentation against Adversarial Attacks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4067–4076.
  * Gao et al. (2022) Ruijun Gao, Qing Guo, Felix Juefei-Xu, Hongkai Yu, Huazhu Fu, Wei Feng, Yang Liu, and Song Wang. 2022.  Can you spot the chameleon? adversarially camouflaging images from co-salient object detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2150–2159.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_ (2018).
  * Gong et al. (2021) Yunpeng Gong, Liqing Huang, and Lifei Chen. 2021.  Eliminate deviation with deviation for data augmentation and a general multi-modal data learning method.  _arXiv preprint arXiv:2101.08533_ (2021).
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ (2014).
  * Guo et al. (2017) Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. 2017.  Countering adversarial images using input transformations.  _arXiv preprint arXiv:1711.00117_ (2017).
  * Guo et al. (2020) Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Jian Wang, Bing Yu, Wei Feng, and Yang Liu. 2020.  Watch out! motion is blurring the vision of your deep neural networks.  _Advances in Neural Information Processing Systems_ 33 (2020), 975–985.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 9729–9738.
  * Hendrycks et al. (2019) Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2019.  Augmix: A simple data processing method to improve robustness and uncertainty.  _arXiv preprint arXiv:1912.02781_ (2019).
  * Ho and Vasconcelos (2022) Chih-Hui Ho and Nuno Vasconcelos. 2022.  DISCO: Adversarial Defense with Local Implicit Functions.  _arXiv preprint arXiv:2212.05630_ (2022).
  * Hou et al. (2023) Yang Hou, Qing Guo, Yihao Huang, Xiaofei Xie, Lei Ma, and Jianjun Zhao. 2023.  Evading DeepFake Detectors via Adversarial Statistical Consistency. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12271–12280.
  * Huang et al. (2023) Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. 2023.  Composite Backdoor Attacks Against Large Language Models.  _arXiv preprint arXiv:2310.07676_ (2023).
  * Jia et al. (2020) Xiaojun Jia, Xingxing Wei, Xiaochun Cao, and Xiaoguang Han. 2020.  Adv-watermark: A novel watermark perturbation for adversarial examples. In _Proceedings of the 28th ACM International Conference on Multimedia_. 1579–1587.
  * Karimi et al. (2021) Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021.  AEDA: An Easier Data Augmentation Technique for Text Classification. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. 2748–2754.
  * Kumar et al. (2023) Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. 2023.  Certifying llm safety against adversarial prompting.  _arXiv preprint arXiv:2309.02705_ (2023).
  * Kurakin et al. (2018) Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. 2018.  Adversarial examples in the physical world.  In _Artificial intelligence safety and security_. Chapman and Hall/CRC, 99–112.
  * Liu et al. (2023a) Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023a.  Prompt Injection attack against LLM-integrated Applications.  _arXiv preprint arXiv:2306.05499_ (2023).
  * Liu et al. (2023b) Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023b.  Jailbreaking chatgpt via prompt engineering: An empirical study.  _arXiv preprint arXiv:2305.13860_ (2023).
  * Lopes et al. (2019) Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. 2019.  Improving robustness without sacrificing accuracy with patch gaussian augmentation.  _arXiv preprint arXiv:1906.02611_ (2019).
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ (2017).
  * Marivate and Sefara (2020) Vukosi Marivate and Tshephisho Sefara. 2020.  Improving short text classification through global augmentation methods. In _International Cross-Domain Conference for Machine Learning and Knowledge Extraction_. Springer, 385–399.
  * Miller (1995) George A Miller. 1995.  WordNet: a lexical database for English.  _Commun. ACM_ 38, 11 (1995), 39–41.
  * Morris et al. (2020) John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020.  TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020_ , Qun Liu and David Schlangen (Eds.). Association for Computational Linguistics, 119–126.  https://doi.org/10.18653/V1/2020.EMNLP-DEMOS.16
  * Nenkova and Vanderwende (2005) Ani Nenkova and Lucy Vanderwende. 2005.  The impact of frequency on summarization.  _Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005_ 101 (2005).
  * Nie et al. (2022) Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. 2022.  Diffusion models for adversarial purification.  _arXiv preprint arXiv:2205.07460_ (2022).
  * Qi et al. (2023) Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. 2023.  Visual adversarial examples jailbreak aligned large language models. In _The Second Workshop on New Frontiers in Adversarial Machine Learning_.
  * Rade and Moosavi-Dezfooli (2021) Rahul Rade and Seyed-Mohsen Moosavi-Dezfooli. 2021.  Helper-based adversarial training: Reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In _ICML 2021 Workshop on Adversarial Machine Learning_.
  * Robey et al. (2023) Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023.  Smoothllm: Defending large language models against jailbreaking attacks.  _arXiv preprint arXiv:2310.03684_ (2023).
  * Sehwag et al. (2021) Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. 2021.  Robust learning meets generative models: Can proxy distributions improve adversarial robustness?  _arXiv preprint arXiv:2104.09425_ (2021).
  * Sun et al. (2019) Bo Sun, Nian-hsuan Tsai, Fangchen Liu, Ronald Yu, and Hao Su. 2019.  Adversarial defense by stratified convolutional sparse coding. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11447–11456.
  * Tian et al. (2021) Binyu Tian, Felix Juefei-Xu, Qing Guo, Xiaofei Xie, Xiaohong Li, and Yang Liu. 2021.  AVA: Adversarial vignetting attack against visual recognition.  _arXiv preprint arXiv:2105.05558_ (2021).
  * Xie et al. (2019) Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, and Simon See. 2019.  Deephunter: a coverage-guided fuzz testing framework for deep neural networks. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 146–157.
  * Xu et al. (2023) Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. 2023.  Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models.  _arXiv preprint arXiv:2305.14710_ (2023).
  * Xu et al. (2017) Weilin Xu, David Evans, and Yanjun Qi. 2017.  Feature squeezing: Detecting adversarial examples in deep neural networks.  _arXiv preprint arXiv:1704.01155_ (2017).
  * Yan et al. (2023) Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2023.  Virtual prompt injection for instruction-tuned large language models.  _arXiv preprint arXiv:2307.16888_ (2023).
  * Yu et al. (2023) Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023.  Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.  _arXiv preprint arXiv:2309.10253_ (2023).
  * Zhang et al. (2021) Cen Zhang, Xingwei Lin, Yuekang Li, Yinxing Xue, Jundong Xie, Hongxu Chen, Xinlei Ying, Jiashui Wang, and Yang Liu. 2021.  $\\{$APICraft$\\}$: Fuzz Driver Generation for Closed-source $\\{$SDK$\\}$ Libraries. In _30th USENIX Security Symposium (USENIX Security 21)_. 2811–2828.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.  arXiv:2306.05685 [cs.CL]
  * Zhu et al. (2023a) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023a.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ (2023).
  * Zhu et al. (2023b) Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023b.  AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models.  _arXiv preprint arXiv:2310.15140_ (2023).
  * Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023.  Universal and transferable adversarial attacks on aligned language models.  _arXiv preprint arXiv:2307.15043_ (2023).