# Fuzzy Set with Hyperbolic Valued Membership Function

Chinmay Ghosh1, Sanjib Kumar Datta2, Soumen Mondal3
1Department of Mathematics
Kazi Nazrul University
Nazrul Road, P.O.- Kalla C.H.
Asansol-713340, West Bengal, India
<EMAIL_ADDRESS>
2Department of Mathematics
University of Kalyani
P.O.-Kalyani, Dist-Nadia, PIN-741235,
West Bengal, India
<EMAIL_ADDRESS>
328, Dolua Dakshinpara Haridas Primary School
Beldanga, Murshidabad
Pin-742133
West Bengal, India
<EMAIL_ADDRESS>
###### Abstract

In this article, we have introdued $\mathbb{D}$-fuzzy sets. We have discussed
the notions of inclusion, union, intersection, complementation and convexity
of such $\mathbb{D}$-fuzzy sets. Also we have proved separation theorem of
convex $\mathbb{D}$-fuzzy sets.

Keywords and phrases: Fuzzy set, hyperbolic number, hyperbolic valued metric,
convexity, separation theorem.

##  1  Introduction and Preliminaries

Let $\mathbb{R}$ be the field of real numbers. Then
$\mathbb{R}^{2}=\left\\{\left(x,y\right):x,y\in\mathbb{R}\right\\}$ forms a
two dimensional vector space over $\mathbb{R}$ with usual vector addition
(componentwise) and scalar multiplication. The standard basis is
$\left\\{\left(1,0\right),\left(0,1\right)\right\\}.$ Also the field structure
of complex numbers $\mathbb{C}$ can be formed from $\mathbb{R}^{2}$ with
respect to the said addition and the multiplication defined by

$\left(x_{1},y_{1}\right).\left(x_{2},y_{2}\right)=\left(x_{1}x_{2}-y_{1}y_{2},x_{1}y_{2}+y_{1}x_{2}\right).$

Denoting $\left(1,0\right),\left(0,1\right)$ by $\mathbf{1}$ and $\mathbf{i}$
respectively, we can represent any element $\left(x,y\right)$ of this field by
$x.\mathbf{1}+y.\mathbf{i,}$ and we write

$\mathbb{C}=\\{x.\mathbf{1}+y.\mathbf{i}:x,y\in\mathbb{R}\\}.$

If no confusion arises we may write $x+y\mathbf{i}$ instead of
$x.\mathbf{1}+y.\mathbf{i.}$ The elements of $\mathbb{C}$ describe the two
dimensional Euclidean geometry.

Note that

$\mathbf{i}^{2}=\mathbf{i.i=}\left(0,1\right).\left(0,1\right)=\left(-1,0\right)=-\left(1,0\right)=-\mathbf{1.}$

Another ring structure may be found from $\mathbb{R}^{2}$ by redefining the
multiplication as

$\left(x_{1},y_{1}\right).\left(x_{2},y_{2}\right)=\left(x_{1}x_{2}+y_{1}y_{2},x_{1}y_{2}+y_{1}x_{2}\right).$

In this ring we denote $\left(1,0\right),\left(0,1\right)$ by $\mathbf{1}$ and
$\mathbf{k}$ respectively.

Then

$\mathbf{k}^{2}=\mathbf{k.k=}\left(0,1\right).\left(0,1\right)=\left(1,0\right)=\mathbf{1.}$

Though this ring is commutative and contains multiplicative identity, it has
divisors of zero. So it fails to be an integral domain. Any element
$\left(x,y\right)$ of this ring can be represented by
$x.\mathbf{1}+y.\mathbf{k}$ or $x+y\mathbf{k.}$ The elements are called
hyperbolic numbers and the ring of all hyperbolic numbers is denoted by
$\mathbb{D}$, i.e.,

$\mathbb{D}=\\{x+y\mathbf{k}:x,y\in\mathbb{R}\\}.$

Henceforth we will write $1$ and $0$ as the elements
$\mathbf{1=}\left(1,0\right)$ and $\mathbf{0=}\left(0,0\right)$ in
$\mathbb{D}$ respectively. As $\mathbb{C}$ describes two dimensional Euclidean
geometry, $\mathbb{D}$ also describes two dimensional Minkowski space-time
(Lorentzian) geometry [4], [5].

The two hyperbolic numbers (zero divisors)

$\mathbf{e}_{1}=\frac{1+\mathbf{k}}{2},\mathbf{e}_{2}=\frac{1-\mathbf{k}}{2}$

satisfy

$\mathbf{e}_{1}^{2}=\mathbf{e}_{1},\mathbf{e}_{2}^{2}=\mathbf{e}_{2},\mathbf{e}_{1}+\mathbf{e}_{2}=1,\mathbf{e}_{1}\mathbf{e}_{2}=0.$

Further an arbitrary element $\alpha=a_{1}+a_{2}\mathbf{k}\in\mathbb{D}$ can
be uniquely represented as

$\mathbb{\alpha=(}a_{1}+a_{2})\mathbf{e}_{1}+\mathbb{(}a_{1}-a_{2})\mathbf{e}_{2},$

and hence $\left\\{\mathbf{e}_{1},\mathbf{e}_{2}\right\\}$ forms an idempotent
basis in $\mathbb{D}.$

Sum and the product of two hyperbolic numbers can be defined pointwise with
the idempotent basis.

We say that a hyperbolic number
$\alpha=\alpha_{1}\mathbf{e}_{1}+\alpha_{2}\mathbf{e}_{2}\in\mathbb{D}$ is
positive if $\alpha_{1},\alpha_{2}>0.$ Thus the set of positive hyperbolic
numbers $\mathbb{D}^{+}$ is given by

$\displaystyle\mathbb{D}^{+}$ $\displaystyle\mathbb{=}$
$\displaystyle\mathbb{\\{}\alpha=\alpha_{1}\mathbf{e}_{1}+\alpha_{2}\mathbf{e}_{2}:\alpha_{1},\alpha_{2}>0\mathbb{\\}}$
$\displaystyle=$
$\displaystyle\mathbb{\\{}\alpha=a_{1}+a_{2}\mathbf{k}:a_{1}>\left|a_{2}\right|\mathbb{\\}}.$

Let $\mathbb{O}$ be the set of all zero divisors in $\mathbb{D}$ and
$\mathbb{O}_{0}=\mathbb{O}\cup\\{\mathbf{0}\\}.$ We use the notation
$\mathbb{D}_{0}^{+}=\mathbb{D}^{+}\cup\mathbb{O}_{0}.$

For $\alpha,\beta\in\mathbb{D},~{}$define a relation $\preceq$ on $\mathbb{D}$
[7] by $\alpha\preceq\beta$ whenever $\beta-\alpha\in\mathbb{D}_{0}^{+}.$ This
relation is reflexive, anti-symmetric as well as transitive and hence defines
a partial order relation on $\mathbb{D}.$ If we write the hyperbolic numbers
$\alpha,\beta$ in idempotent representation as
$\mathbb{\alpha=}\alpha_{1}\mathbf{e}_{1}+\alpha_{2}\mathbf{e}_{2}$ and
$\mathbb{\beta=\beta}_{1}\mathbf{e}_{1}+\mathbb{\beta}_{2}\mathbf{e}_{2},$
then $\alpha\preceq\beta$ implies that $\alpha_{1}\leq\beta_{1}$ and
$\alpha_{2}\leq\beta_{2}.$ And by $\alpha\prec\beta$ we mean
$\alpha_{1}<\beta_{1}$ and $\alpha_{2}<\beta_{2}.$ Two hyperbolic numbers
$\alpha,\beta~{}$are said to be comparable if either $\alpha\prec\beta$ or
$\alpha=\beta$ or $\beta\prec\alpha.$ We write
$\max\nolimits_{\mathbb{D}}\left\\{\alpha,\beta\right\\}=\alpha$ if
$\beta\preceq\alpha$ and
$\max\nolimits_{\mathbb{D}}\left\\{\alpha,\beta\right\\}=\beta$ if
$\alpha\preceq\beta.$ Similarly
$\min_{\mathbb{D}}\left\\{\alpha,\beta\right\\}=\alpha$ if
$\alpha\preceq\beta$ and
$\min_{\mathbb{D}}\left\\{\alpha,\beta\right\\}=\beta$ if
$\beta\preceq\alpha.$ For more than two hyperbolic numbers
$\max\nolimits_{\mathbb{D}}$ and $\min_{\mathbb{D}}$ are defined only if any
pair of the given hyperbolic numbers are comparable.

Let
$\alpha=\alpha_{1}\mathbf{e}_{1}+\alpha_{2}\mathbf{e}_{2},\beta=\mathbb{\beta}_{1}\mathbf{e}_{1}+\mathbb{\beta}_{2}\mathbf{e}_{2}\in\mathbb{D}$
with $\alpha\preceq\beta.$ The closed hyperbolic interval
($\mathbb{D}-$interval) [3] is defined by

$\left[\alpha,\beta\right]_{\mathbb{D}}=\left\\{\zeta\in\mathbb{D}:\alpha\preceq\zeta\preceq\beta\right\\}.$

Similarly the open hyperbolic interval ($\mathbb{D}-$interval) is defined by

$\left(\alpha,\beta\right)_{\mathbb{D}}=\left\\{\zeta\in\mathbb{D}:\alpha\prec\zeta\prec\beta\right\\}.$

The hyperbolic length of the $\mathbb{D}-$interval
$\left[\alpha,\beta\right]_{\mathbb{D}}$ or
$\left(\alpha,\beta\right)_{\mathbb{D}}$ is defined by

$l_{\mathbb{D}}\left(\left[\alpha,\beta\right]_{\mathbb{D}}\right)=\beta-\alpha.$

$\left[\alpha,\beta\right]_{\mathbb{D}}$ is called a degenerate closed
$\mathbb{D}-$interval if $\beta-\alpha\in\mathbb{D}^{+}\cap\mathbb{O}$ and
$\left[\alpha,\beta\right]_{\mathbb{D}}$ is called a nondegenerate closed
$\mathbb{D}-$interval if $\beta-\alpha$
$\in\mathbb{D}^{+}\backslash\mathbb{O}$.

The hyperbolic modulus of a hyperbolic number is defined by

$\left|\alpha\right|_{k}=\left|\alpha_{1}\mathbf{e}_{1}+\alpha_{2}\mathbf{e}_{2}\right|_{k}=\left|\alpha_{1}\right|\mathbf{e}_{1}+\left|\alpha_{2}\right|\mathbf{e}_{2}\in\mathbb{D}_{0}^{+}$

where $\left|.\right|$ is the modulus of a real number.
$\alpha=\alpha_{1}\mathbf{e}_{1}+\alpha_{2}\mathbf{e}_{2}$ is said to be
nearer to $1$ or $0$ in the hyperbolic sense if both $\alpha_{1}$ and
$\alpha_{2}$ are nearer to $1$ or $0$ in the real sense respectively.

The hyperbolic valued norm [1] on a $\mathbb{D-}$module space $X$ is a mapping
$\left\|.\right\|_{\mathbb{D}}:X\rightarrow\mathbb{D}^{+}$ that satisfies the
following properties:

$(i)$ $\left\|x\right\|_{\mathbb{D}}\succeq 0$ for all $x\in X$ and
$\left\|x\right\|_{\mathbb{D}}=0$ iff $x=0\in X.$

$(ii)$ $\left\|\alpha
x\right\|_{\mathbb{D}}=\left|\alpha\right|_{k}\left\|x\right\|_{\mathbb{D}}$
for all $\alpha\in\mathbb{D}$ and for all $x\in X.$

$(iii)$
$\left\|x+y\right\|_{\mathbb{D}}\preceq\left\|x\right\|_{\mathbb{D}}+\left\|y\right\|_{\mathbb{D}}$
for all $x,y\in X.$

A function $d_{\mathbb{D}}:X\times X\longrightarrow\mathbb{D}_{0}^{+}$ defined
on a nonempty set $X$ satisfying the following:

$(i)$ $d_{\mathbb{D}}(x,y)\succeq 0$ for all $x,y\in X$ and
$d_{\mathbb{D}}(x,y)=0$ if and only if $x=y,$

$(ii)$ $d_{\mathbb{D}}(x,y)=d_{\mathbb{D}}(y,x)$ for all $x,y\in X,$

$(iii)$ $d_{\mathbb{D}}(x,y)\preceq d_{\mathbb{D}}(x,z)+d_{\mathbb{D}}(z,y)$
for all $x,y,z\in X,$

is called a hyperbolic valued metric [6] or $\mathbb{D-}$metric on $X.$

Being motivated from the articles [3] and [13] we wish to find fuzzy sets in
the hyperbolic settings.

In the book of G. B. Price [9], the description of hyperbolic numbers is
found. But G. Sobczyk [11] first realized the importance of such numbers and
called them sister of complex numbers. The two books [4] and [5] are good
references for hyperbolic numbers. The readers of this paper may go through
the books [2], [7] and [8]. Recently many researchers ([3], [6], [10], [12])
are investigating different aspets of hyperbolic numbers.

##  2 Main Results

Let $X$ be a set of points and $x\in X.$ A $\mathbb{D}$-fuzzy set $A$ in $X$
is characterized by a hyperbolic valued membership function
($\mathbb{D}$-membership function) $f_{A}^{\mathbb{D}}:$
$X\rightarrow[0,1]_{\mathbb{D}}$ which associates with each point in $X$ a
hyperbolic number in the hyperbolic interval $[0,1]_{\mathbb{D}},$ with the
value of $f_{A}^{\mathbb{D}}(x)$ at $x$ representing the ”$\mathbb{D}$-grade
of $\mathbb{D}$-membership” of $x$ in $A.$ Thus the nearer the value of
$f_{A}^{\mathbb{D}}(x)$ to $1$, the higher $\mathbb{D}$-grade of membership of
$x$ in $A.$

A $\mathbb{D}$-fuzzy set $A$ in $X$ with $\mathbb{D}$-membership function
$f_{A}^{\mathbb{D}}(x),$ can be written as

$A=\\{(x,f_{A}^{\mathbb{D}}(x)):x\in X\\}.$

Let
$f_{A}^{\mathbb{D}}(x)=f_{A}^{1}(x)\mathbf{e}_{1}+f_{A}^{2}(x)\mathbf{e}_{2}$
be the idempotent representation of $f_{A}^{\mathbb{D}}(x).$ Then
$A_{1}=\\{(x,f_{A}^{1}(x):x\in X\\}$ and $A_{2}=\\{(x,f_{A}^{2}(x):x\in X\\}$
are two ordinary fuzzy sets.

Note that if $A$ is a set in the ordinary sense, its $\mathbb{D}$-membership
function is of the form

$f_{A}^{\mathbb{D}}(x)=\left\\{\begin{array}[]{c}1\text{ if }x\in A\\\ 0\text{
if }x\notin A\end{array}\right.$

###### Example 1

Let $X=\mathbb{D}$ and $A=\\{x\in\mathbb{D}$: $x\succ 1\\}$ and
$f_{A}^{\mathbb{D}}(x):X\rightarrow[0,1]_{\mathbb{D}}$ be a function.
Representative values of such function might be :
$f_{A}^{\mathbb{D}}(\mathbf{e}_{1})=0,$ $f_{A}^{\mathbb{D}}(0)=0,$
$f_{A}^{\mathbb{D}}(5\mathbf{e}_{1}+2\mathbf{e}_{2})=0.06\mathbf{e}_{1}+0.04\mathbf{e}_{2},$
$f_{A}^{\mathbb{D}}(5)=0.06\mathbf{e}_{1}+0.07\mathbf{e}_{2},$
$f_{A}^{\mathbb{D}}(200)=0.4\mathbf{e}_{1}+0.5\mathbf{e}_{2}$ etc.

A $\mathbb{D}$-fuzzy set is empty if and only if its $\mathbb{D}$-membership
function is zero on $X.$ Two $\mathbb{D}$-fuzzy sets $A$ and $B$ are equal if
and only if $f_{A}^{\mathbb{D}}(x)=f_{B}^{\mathbb{D}}(x)$ for all $x\in X.$
The complement of a $\mathbb{D}$-fuzzy set $A$ is denoted by $A^{\prime}$ and
is defined by

$f_{A^{{}^{\prime}}}^{\mathbb{D}}=1-f_{A}^{\mathbb{D}}.$

Let $A$ and $B$ be two $\mathbb{D}$-fuzzy sets. Then $A$ is contained in $B$
if and only if $f_{A}^{\mathbb{D}}\preceq f_{B}^{\mathbb{D}},$ i.e.,

$A\subset B\Leftrightarrow f_{A}^{\mathbb{D}}\preceq f_{B}^{\mathbb{D}}.$

The union of two $\mathbb{D}$-fuzzy sets $A$ and $B$ with
$\mathbb{D}$-membership functions $f_{A}^{\mathbb{D}}$ and
$f_{B}^{\mathbb{D}}$ respectively is a $\mathbb{D}$-fuzzy set $C\,=A\cup B,$
whose $\mathbb{D}$-membership function is

$f_{C}^{\mathbb{D}}(x)=\max\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\},\text{
}x\in X$

or, in short,

$f_{C}^{\mathbb{D}}=f_{A}^{\mathbb{D}}\vee f_{B}^{\mathbb{D}}.$

###### Proposition 2

The union of two $\mathbb{D}$-fuzzy sets $A$ and $B$ is the smallest
$\mathbb{D}$-fuzzy set containing $A$ and $B.$

Proof. Let $C\,=A\cup B,$ whose $\mathbb{D}$-membership function is

$f_{C}^{\mathbb{D}}(x)=\max\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\},\text{
}x\in X$

Then

$\max\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\}\succeq
f_{A}^{\mathbb{D}}(x)$

and

$\max\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\}\succeq
f_{B}^{\mathbb{D}}(x).$

Now let $D$ be any $\mathbb{D}$-fuzzy set containing $A$ and $B.$

Then

$f_{D}^{\mathbb{D}}(x)\succeq f_{A}^{\mathbb{D}}(x)\text{ ,
}f_{D}^{\mathbb{D}}(x)\succeq f_{B}^{\mathbb{D}}(x).$

Hence

$f_{D}^{\mathbb{D}}(x)\succeq\max\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\}=f_{C}^{\mathbb{D}}(x),$

which implies that

$C\subset D.$

The intersection of two $\mathbb{D}$-fuzzy sets $A$ and $B$ with
$\mathbb{D}$-membership functions $f_{A}^{\mathbb{D}}$ and
$f_{B}^{\mathbb{D}}$ respectively is a $\mathbb{D}$-fuzzy set $C\,=A\cap B,$
whose $\mathbb{D}$-membership function is

$f_{C}^{\mathbb{D}}(x)=\min\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\},\text{
}x\in X$

or

$f_{C}^{\mathbb{D}}=f_{A}^{\mathbb{D}}\wedge f_{B}^{\mathbb{D}}.$

As in the case of union, it is easy to prove the following proposition.

###### Proposition 3

The intersection of two $\mathbb{D}$-fuzzy sets $A$ and $B$ is the largest
$\mathbb{D}$-fuzzy set contained in both $A$ and $B.$

Let $A,B,C$ be $\mathbb{D}$-fuzzy sets with corresponding
$\mathbb{D}$-membership functions $f_{A}^{\mathbb{D}}$, $f_{B}^{\mathbb{D}}$
and $f_{C}^{\mathbb{D}}$ respectively. Then the following are satisfied:

  * •

De Morgan’s Laws:

$\left(A\cup B\right)^{\prime}=A^{\prime}\cap B^{\prime}$ (1) $\left(A\cap
B\right)^{\prime}=A^{\prime}\cup B^{\prime}.$ (2)

  * •

Distributive Laws:

$C\cap\left(A\cup B\right)=\left(C\cap A\right)\cup\left(C\cap B\right)$ (3)
$C\cup\left(A\cap B\right)=\left(C\cup A\right)\cap\left(C\cup B\right).$ (4)

Property (1) can be proved by using the relation

$1-\max\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\}=\min\nolimits_{\mathbb{D}}\left\\{1-f_{A}^{\mathbb{D}}(x),1-f_{B}^{\mathbb{D}}(x)\right\\}$

which can be easily verified by testing it for the two possible cases:
$f_{A}^{\mathbb{D}}(x)\succ f_{B}^{\mathbb{D}}(x)$ and
$f_{A}^{\mathbb{D}}(x)\prec f_{B}^{\mathbb{D}}(x).$

Similarly we can prove property (2).

Property (3) can be proved by using the relation

$\max\nolimits_{\mathbb{D}}\left\\{f_{C}^{\mathbb{D}}(x),\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\right\\}\right\\}=\min\nolimits_{\mathbb{D}}\left\\{\max\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{C}^{\mathbb{D}}(x)\\},\max\nolimits_{\mathbb{D}}\\{f_{C}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\\}\right\\}$

which can be verified by testing it for the six possible cases:

$f_{A}^{\mathbb{D}}(x)\succ f_{B}^{\mathbb{D}}(x)\succ
f_{C}^{\mathbb{D}}(x),\text{ }f_{A}^{\mathbb{D}}(x)\succ
f_{C}^{\mathbb{D}}(x)\succ f_{B}^{\mathbb{D}}(x),\text{
}f_{B}^{\mathbb{D}}(x)\succ f_{A}^{\mathbb{D}}(x)\succ f_{C}^{\mathbb{D}}(x),$
$f_{B}^{\mathbb{D}}(x)\succ f_{C}^{\mathbb{D}}(x)\succ
f_{A}^{\mathbb{D}}(x),\text{ }f_{C}^{\mathbb{D}}(x)\succ
f_{A}^{\mathbb{D}}(x)\succ f_{B}^{\mathbb{D}}(x),\text{
}f_{C}^{\mathbb{D}}(x)\succ f_{B}^{\mathbb{D}}(x)\succ f_{A}^{\mathbb{D}}(x).$

Similarly we can prove property (4).

The algebraic sum of $\mathbb{D}$-fuzzy sets $A$ and $B,$ denoted by $A+B,$ is
defined in terms of the $\mathbb{D}$-membership functions of $A$ and $B$ by

$f_{A+B}^{\mathbb{D}}=f_{A}^{\mathbb{D}}+f_{B}^{\mathbb{D}}$

provided the sum $f_{A}^{\mathbb{D}}+f_{B}^{\mathbb{D}}\preceq 1.$

The algebraic product of $\mathbb{D}$-fuzzy sets $A$ and $B,$ denoted by $AB,$
is defined in terms of the $\mathbb{D}$-membership functions of $A$ and $B$ by

$f_{AB}^{\mathbb{D}}=f_{A}^{\mathbb{D}}f_{B}^{\mathbb{D}}.$

The absolute difference of $\mathbb{D}$-fuzzy sets $A$ and $B,$ denoted by
$\left|A-B\right|_{\mathbb{D}},$ is defined in terms of the
$\mathbb{D}$-membership functions of $A$ and $B$ by

$f_{\left|A-B\right|_{\mathbb{D}}}^{\mathbb{D}}=\left|f_{A}^{\mathbb{D}}-f_{B}^{\mathbb{D}}\right|_{\mathbb{D}}.$

Let $A$ and $B$ be two $\mathbb{D-}$fuzzy sets defined on the Universal sets
$X$ and $Y$ with $\mathbb{D}$-membership functions $f_{A}^{\mathbb{D}}(x)$ and
$f_{B}^{\mathbb{D}}(y)$ respectively. Then the cartesian product of $A$ and
$B,$ denoted by $A\times B,$ is defined in terms of the
$\mathbb{D}$-membership functions of $A$ and $B$ by

$f_{A\times
B}^{\mathbb{D}}((x,y))=\min\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(y)\\}.$

Let $A,$ $B$ and $\Lambda$ be arbitrary $\mathbb{D}$-fuzzy sets. The convex
combination of $A,$ $B$ and $\Lambda$ is denoted by $(A,$ $B;\Lambda)$ and is
defined by

$(A,B;\Lambda)=\Lambda A+\Lambda^{\prime}B$

where $\Lambda^{\prime}$ is the complement of $\Lambda.$

A basic property of the convex combination of $A,$ $B$ and $\Lambda$ is
expressed by

$A\cap B\subset(A,B;\Lambda)\subset A\cup B\text{ \ \ for all }\Lambda.$

This property is an immediate consequence of the inequalities

$\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\right\\}\preceq\lambda
f_{A}^{\mathbb{D}}(x)+(1-\lambda)f_{B}^{\mathbb{D}}(x)\preceq\max\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x),f_{B}^{\mathbb{D}}(x)\right\\},\text{
}x\in X$

which hold for all $\lambda\in[0,1]_{\mathbb{D}}.$

A binary $\mathbb{D}$-fuzzy relation in $X$ is a $\mathbb{D}$-fuzzy set in the
product space $X\times X.$ Moreover, an $n-$ary $\mathbb{D}$-fuzzy relation in
$X$ is a $\mathbb{D}$-fuzzy set $A$ in the product space $X\times
X\times...\times X$ whose $\mathbb{D}$-membership function is of the form
$f_{A}^{\mathbb{D}}(x_{1},x_{2},...,x_{n}),$ where $x_{i}\in X,i=1,2,...,n.$

Now we define convexity of $\mathbb{D-}$fuzzy sets in two different ways and
prove their equivalence.

###### Definition 4

A $\mathbb{D-}$fuzzy set $A$ is convex if and only if the sets
$\Gamma_{\alpha}$ defined by

$\Gamma_{\alpha}=\\{x:f_{A}^{\mathbb{D}}(x)\succeq\alpha\\}$

are convex for all $\alpha\in(0,1]_{\mathbb{D}}.$

###### Definition 5

A $\mathbb{D-}$fuzzy set $A$ is convex if and only if

$f_{A}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\succeq\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{A}^{\mathbb{D}}(x_{2})\right\\}$
(5)

for all $x_{1},x_{2}$ in $X$ and for all $\lambda$ in $[0,1]_{\mathbb{D}}.$

###### Remark 6

The above two definitions are equivalent.

Let $A$ is convex in the sense of Definition 4 and
$\alpha=f_{A}^{\mathbb{D}}(x_{1})\preceq f_{A}^{\mathbb{D}}(x_{2}),$ then
$x_{2}\in\Gamma_{\alpha}$ and $\lambda
x_{1}+(1-\lambda)x_{2}\in\Gamma_{\alpha}$ by the convexity of
$\Gamma_{\alpha}.$ Hence

$f_{A}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\succeq\alpha=f_{A}^{\mathbb{D}}(x_{1})=\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{A}^{\mathbb{D}}(x_{2})\right\\}.$

Conversely, if $A$ is convex in the sense of Definition 5 and
$\alpha=f_{A}^{\mathbb{D}}(x_{1}),$ then $\Gamma_{\alpha}$ may be regarded as
the set of points $x_{2}$ for which $f_{A}^{\mathbb{D}}(x_{2})\succeq
f_{A}^{\mathbb{D}}(x_{1}).$ In virtue of (5), every point of the form $\lambda
x_{1}+(1-\lambda)x_{2},\lambda\in[0,1]_{\mathbb{D}},$ is also in
$\Gamma_{\alpha}$ and hence $\Gamma_{\alpha}$ is convex set.

###### Theorem 7

If two $\mathbb{D-}$fuzzy sets $A$ and $B$ are convex then $A\cap B$ is also
convex.

Proof. Let $C=A\cap B$. Then

$f_{C}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}=\min\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\},f_{B}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\\}.$

Now since $A$ and $B$ are convex

$f_{A}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\succeq\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{A}^{\mathbb{D}}(x_{2})\right\\}$
$f_{B}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\succeq\min\nolimits_{\mathbb{D}}\left\\{f_{B}^{\mathbb{D}}(x_{1}),f_{B}^{\mathbb{D}}(x_{2})\right\\}$

and hence

$\displaystyle f_{C}^{\mathbb{D}}\\{\lambda x_{1}+(1-\lambda)x_{2}\\}$
$\displaystyle\succeq$
$\displaystyle\min\nolimits_{\mathbb{D}}\\{\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{A}^{\mathbb{D}}(x_{2})\right\\},\min\nolimits_{\mathbb{D}}\left\\{f_{B}^{\mathbb{D}}(x_{1}),f_{B}^{\mathbb{D}}(x_{2})\right\\}\\}$

or equivalently

$\displaystyle f_{C}^{\mathbb{D}}\\{\lambda x_{1}+(1-\lambda)x_{2}\\}$
$\displaystyle\succeq$
$\displaystyle\min\nolimits_{\mathbb{D}}\\{\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{B}^{\mathbb{D}}(x_{1})\right\\},\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{2}),f_{B}^{\mathbb{D}}(x_{2})\right\\}\\}$

and thus

$f_{C}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\succeq\min\nolimits_{\mathbb{D}}\left\\{f_{C}^{\mathbb{D}}(x_{1}),f_{C}^{\mathbb{D}}(x_{2})\right\\}.$

###### Theorem 8

If two $\mathbb{D-}$fuzzy sets $A$ and $B$ on the Universal sets $X$ and $Y$
are convex then their cartesian product $A\times B$ is also convex on $X\times
Y$.

Proof. Let $C=A\times B.$ Then

$\displaystyle
f_{C}^{\mathbb{D}}\\{\lambda(x_{1},y_{1})+(1-\lambda)(x_{2},y_{2})\\}$
$\displaystyle=$ $\displaystyle f_{C}^{\mathbb{D}}\\{(\lambda
x_{1}+(1-\lambda)x_{2},\lambda y_{1}+(1-\lambda)y_{2})\\}$ $\displaystyle=$
$\displaystyle\min\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(\lambda
x_{1}+(1-\lambda)x_{2}),f_{B}^{\mathbb{D}}(\lambda y_{1}+(1-\lambda)y_{2})\\}$

Now since $A$ and $B$ are convex

$f_{A}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\succeq\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{A}^{\mathbb{D}}(x_{2})\right\\}$
$f_{B}^{\mathbb{D}}\\{\lambda
y_{1}+(1-\lambda)y_{2}\\}\succeq\min\nolimits_{\mathbb{D}}\left\\{f_{B}^{\mathbb{D}}(y_{1}),f_{B}^{\mathbb{D}}(y_{2})\right\\}.$

Hence

$\displaystyle
f_{C}^{\mathbb{D}}\\{\lambda(x_{1},y_{1})+(1-\lambda)(x_{2},y_{2})\\}$
$\displaystyle\succeq$
$\displaystyle\min\nolimits_{\mathbb{D}}\\{\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{A}^{\mathbb{D}}(x_{2})\right\\},\min\nolimits_{\mathbb{D}}\left\\{f_{B}^{\mathbb{D}}(y_{1}),f_{B}^{\mathbb{D}}(y_{2})\right\\}\\}$
$\displaystyle=$
$\displaystyle\min\nolimits_{\mathbb{D}}\\{\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{1}),f_{B}^{\mathbb{D}}(y_{1})\right\\},\min\nolimits_{\mathbb{D}}\left\\{f_{A}^{\mathbb{D}}(x_{2}),f_{B}^{\mathbb{D}}(y_{2})\right\\}\\}.$

Therefore

$f_{C}^{\mathbb{D}}\\{\lambda(x_{1},y_{1})+(1-\lambda)(x_{2},y_{2})\\}\succeq\min\nolimits_{\mathbb{D}}\\{f_{C}^{\mathbb{D}}((x_{1},y_{1})),f_{C}^{\mathbb{D}}((x_{2},y_{2}))\\}.$

###### Remark 9

A cylinder $P$ in $X\times Y$ is a convex $\mathbb{D-}$fuzzy set since it is
the cartesian product of two convex set.

###### Definition 10

A $\mathbb{D-}$fuzzy set $A$ is bounded if and only if the sets
$\Gamma_{\alpha}=\\{x:f_{A}^{\mathbb{D}}(x)\succeq\alpha\\}$ are
$\mathbb{D-}$bounded for all $\alpha\succ 0;$ that is for every $\alpha\succ
0$ there exists a finite $R(\alpha)$ such that
$\left\|x\right\|_{\mathbb{D}}\preceq R(\alpha)$ for all $x$ in
$\Gamma_{\alpha}.$

Let $A$ be a $\mathbb{D-}$bounded set and for $\epsilon\succ 0$ consider
$\Gamma_{\epsilon}=\\{x:f_{A}^{\mathbb{D}}(x)\succeq\epsilon\\}.$ Then by the
definition 10, $\Gamma_{\epsilon}$ is contained in a sphere $S$ of radius
$R(\epsilon).$ Let $H$ be a supporting hyperplane of $S.$ Then, all points on
the side of $H$ which does not contain the origin lie outside or on $S,$ and
so for all such points $f_{A}^{\mathbb{D}}(x)\preceq\epsilon.$ Hence we can
say that, if $A$ is $\mathbb{D-}$bounded set, then for each $\epsilon\succ 0$
there exists a hyperplane $H$ such that $f_{A}^{\mathbb{D}}(x)\preceq\epsilon$
for all $x$ on the side of $H$ which does not contain the origin.

###### Theorem 11

Let $A$ be a $\mathbb{D-}$bounded $\mathbb{D-}$fuzzy set and
$M=\\{\sup_{\mathbb{D}}f_{A}^{\mathbb{D}}(x):x\in X\\}.$ Then there is atleast
one point $x_{0}$ at which $M$ is essentially attained in the sense that, for
each $\epsilon\succ 0,$ every spherical neighborhood of $x_{0}$ contains
points in the set $Q(\epsilon)=\\{x:f_{A}^{\mathbb{D}}(x)\succeq
M-\epsilon\\}.$

Proof. Consider a nested sequence of $\mathbb{D-}$bounded sets
$\Gamma_{1},\Gamma_{2},...,$ where

$\Gamma_{n}=\\{x:f_{A}^{\mathbb{D}}(x)\succeq M-M/(n+1)\\},n=1,2,...$

Since $M=\\{\sup_{\mathbb{D}}f_{A}^{\mathbb{D}}(x):x\in X\\},$ $\Gamma_{n}$ is
nonempty for all finite $n.$

Let $x_{n}$ be an arbitrarily chosen point in $\Gamma_{n},n=1,2,....$ Then
$x_{1},x_{2},...,$ is a sequence of points in a closed $\mathbb{D-}$bounded
set $\Gamma_{1}.$ By the Bolzano-Weierstrass theorem, thus sequence must have
atleast one limit point, say $x_{0},$ in $\Gamma_{1}.$ Consequently, every
spherical neighborhood of $x_{0}$ will contain infinitely many points from the
sequence $x_{1},x_{2},...,$ and more particularly, from the subsequence
$x_{N+1},$ $x_{N+2},$ $...,$ where $N\geq M/\epsilon.$ Since the points of
this subsequence fall within the set
$Q(\epsilon)=\\{x:f_{A}^{\mathbb{D}}(x)\succeq M-\epsilon\\},$ the theorem is
proved.

A $\mathbb{D-}$fuzzy set $A$ is said to be strictly convex if the sets
$\Gamma_{\alpha},$ $0\prec\alpha\preceq 1$ are strictly convex. A
$\mathbb{D-}$fuzzy set $A$ is said to be strongly convex if, for any two
distinct points $x_{1}$ and $x_{2}$, and any $\lambda\in(0,1)_{\mathbb{D}}$

$f_{A}^{\mathbb{D}}\\{\lambda
x_{1}+(1-\lambda)x_{2}\\}\succ\min\nolimits_{\mathbb{D}}\\{f_{A}^{\mathbb{D}}(x_{1}),f_{A}^{\mathbb{D}}(x_{2})\\}.$

Note that strong convexity does not imply strict convexity or vice-versa. If
$A$ and $B$ are strictly (strongly) convex, their intersection is strictly
(strongly) convex.

Let $A$ be a convex $\mathbb{D-}$fuzzy set and
$M=\sup_{\mathbb{D}}f_{A}^{\mathbb{D}}(x).$ If $A$ is $\mathbb{D-}$bounded,
then either $M$ is attained for some $x,$ say $x_{0},$ or there is atleast one
point $x_{0}$ at which $M$ is essentially attained in the sense that, for each
$\epsilon\succ 0,$ every spherical neighborhood of $x_{0}$ contains points in
the set $Q(\epsilon)=\\{x:M-f_{A}^{\mathbb{D}}(x)\preceq\epsilon\\}.$ Note
that if $A$ is strongly convex and $x_{0}$ is attained, then $x_{0}$ is
unique.

Let $A$ be a $\mathbb{D-}$fuzzy set with $\mathbb{D}$-membership function
$f_{A}^{\mathbb{D}}(x),$ $x\in X$ and
$M=\sup_{\mathbb{D}}f_{A}^{\mathbb{D}}(x).$ Then the set of all points in $X$
at which $M$ is essentially attained, is called the core of $A$ and is denoted
by $C(A).$

###### Theorem 12

If $A$ is a convex $\mathbb{D-}$fuzzy set, then $C(A)$ is convex.

Proof. It is sufficient to show that if $M$ is essentially attained at $x_{0}$
and $x_{1},$ $x_{0}\neq x_{1},$ then it is also essentially attained at all
$x=\lambda x_{0}+(1-\lambda)x_{1},\lambda\in[0,1]_{\mathbb{D}}.$

Let $P$ be a cylinder os radius $\epsilon_{\mathbb{D}}$ with the passing
through $x_{0}$ and $x_{1}$ as its axis. Let $x_{0}^{\prime}$ be a point in a
sphere of radius $\epsilon_{\mathbb{D}}$ centering on $x_{0}$ and
$x_{1}^{\prime}$ be a point in a sphere of radius $\epsilon_{\mathbb{D}}$
centering on $x_{1}$ such that $f_{A}^{\mathbb{D}}(x_{0}^{\prime})\succeq
M-\epsilon_{\mathbb{D}}$ and $f_{A}^{\mathbb{D}}(x_{1}^{\prime})\succeq
M-\epsilon_{\mathbb{D}}.$ Then, by the convexity of $A$, for any point $u$ on
the segment $x_{0}^{\prime}x_{1}^{\prime},$ we have
$f_{A}^{\mathbb{D}}(u)\succeq M-\epsilon_{\mathbb{D}}.$ Furthermore, by the
remark 9, all points on $x_{0}^{\prime}x_{1}^{\prime}$ will lie on $P.$

Now let $x$ be any point in the segment $x_{0}x_{1}.$ Then for any point $y$
in the segment $x_{0}^{\prime}x_{1}^{\prime}$, we have

$d_{\mathbb{D}}(x,y)\preceq\epsilon_{\mathbb{D}},\text{ since
}x_{0}^{\prime}x_{1}^{\prime}\in P.$

where $d_{\mathbb{D}}$ is the hyperbolic valued metric.

Hence, a sphere of radius $\epsilon_{\mathbb{D}}$ centering on $x$ will
contain at least one point of the segment $x_{0}^{\prime}x_{1}^{\prime}$ and
so will contain at least one point, say $w,$ at which
$f_{A}^{\mathbb{D}}(w)\succeq M-\epsilon_{\mathbb{D}}.$ This implies that $M$
is essentially attained at $x$ and thus proves the theorem.

###### Corollary 13

If $A$ is strongly convex, then the point at which $M$ is essentially attained
is unique.

Let $A$ be a $\mathbb{D-}$fuzzy set in $E^{n}$ with membership function
$f_{A}^{\mathbb{D}}(x)$ $=$ $f_{A}^{\mathbb{D}}(x_{1},$ $x_{2},$ $...,$
$x_{n}).$ The $\mathbb{D}$-shadow(or $\mathbb{D}$-projection) of $A$ on a
hyperplane $H=\\{x:x_{1}=0\\}$ is defined to be a $\mathbb{D}$-fuzzy set
$S_{H}(A)$ in $E^{n-1}$ with $f_{S_{H}(A)}^{\mathbb{D}}(x)$ given by

$f_{S_{H}(A)}^{\mathbb{D}}(x)=f_{S_{H}(A)}^{\mathbb{D}}(x_{2},...,x_{n})=\sup\nolimits_{x_{1}}f_{S_{H}(A)}^{\mathbb{D}}(x_{1},x_{2},...,x_{n}).$

It is clear from the definition that, if $A$ is a convex $\mathbb{D-}$fuzzy
set, then its $\mathbb{D}$-shadow on any hyperplane is also a convex
$\mathbb{D-}$fuzzy set.

###### Theorem 14

Let $A$ and $B$ be two convex $\mathbb{D-}$fuzzy sets in $E^{n}$. Then

$S_{H}(A)=S_{H}(B)\text{ for all }H\Longrightarrow A=B.$

Proof. It is sufficient to show that if there exists a point, say $x_{0},$
such that $f_{A}^{\mathbb{D}}(x_{0})\neq f_{B}^{\mathbb{D}}(x_{0}),$ then
there exists a hyperplane $H$ such that
$f_{S_{H}(A)}^{\mathbb{D}}(x_{0}^{\ast})\neq
f_{S_{H}(B)}^{\mathbb{D}}(x_{0}^{\ast})$, where $x_{0}^{\ast}$ is the
projection of $x_{0}$ on $H.$

Suppose that $f_{A}^{\mathbb{D}}(x_{0})=\alpha\succ
f_{B}^{\mathbb{D}}(x_{0})=\beta.$ Since $B$ is a convex $\mathbb{D-}$fuzzy
set, the set $\Gamma_{\beta}=\\{x:f_{B}^{\mathbb{D}}(x)\succ\beta\\}$ is
convex, and hence there exists a hyperplane $F$ supporting $\Gamma_{\beta}$
and passing through $x_{0}.$ Let $H$ be a hyperplane orthogonal to $F$ and
$x_{0}^{\ast}$ be the $\mathbb{D}$-projection of $x_{0}$ on $H.$ Then, since
$f_{B}^{\mathbb{D}}(x)\preceq\beta$ for all $x$ on $F,$ we have
$f_{S_{H}(B)}^{\mathbb{D}}(x_{0}^{\ast})\preceq\beta.$ On the other hand
$f_{S_{H}(A)}^{\mathbb{D}}(x_{0}^{\ast})\succeq\alpha.$ Consequently,
$f_{S_{H}(B)}^{\mathbb{D}}(x_{0}^{\ast})\neq
f_{S_{H}(A)}^{\mathbb{D}}(x_{0}^{\ast}).$ Similarly we can show this for the
case $\alpha\prec\beta.$

Let $A$ and $B$ are two $\mathbb{D-}$bounded $\mathbb{D-}$fuzzy sets.and let
$H$ be a hypersurface in $E^{n}\,$ defined by an equation $h(x)=0,$ with all
points for which $h(x)\geq 0$ being on one side of $H$ and all points for
which $h(x)\leq 0$ being on the other side. Let $K_{H}$ be a number dependent
on $H$ such that $f_{A}^{\mathbb{D}}(x)\preceq K_{H}$ on one side of $H$ and
$f_{B}^{\mathbb{D}}(x)\preceq K_{H}$ on the other side. Let
$M_{H}=\inf_{\mathbb{D}}K_{H}$ and $D=1-M_{H}.$ Then $D$ is called the degree
of separation of $A$ and $B$ by $H.$

The main problem is to find a member of the family of hypersurfaces
$\\{H_{\lambda}\\},$ $\lambda\in E^{m},$ which realizes the highest possible
degree of separation.

The following theorem is called the separation theorem for convex
$\mathbb{D-}$fuzzy sets.

###### Theorem 15

Let $A$ and $B$ be two $\mathbb{D-}$bounded $\mathbb{D-}$fuzzy sets in
$E^{n},$ with maximal $\mathbb{D}$-grades $M_{A}$ and $M_{B}$ respectively.
Let $M$ be the maximal $\mathbb{D}$-grade for the intersection $A\cap B.$ Then
$D=1-M.$

Proof. The proof of this theorem is similar to the proof of the separation
theorem for convex ordinary fuzzy sets ([13], page no- 352).

## References

  * [1] Alpay, D., Luna-Elizarrarás, M.E., Shapiro, M.: Kolmogorov’s axioms for probabilities with values in hyperbolic numbers. Adv. Appl. Clifford Algebr. 27(2), 913–929 (2017).
  * [2] Alpay D., Luna-Elizarrarás M. E., Shapiro M., Struppa D. C., Basics of functional analysis with bicomplex scalars, and bicomplex Schur analysis, Springer International Publishing (2014).
  * [3] Balankin, A.S., Bory-Reyes, J., Luna-Elizarrarás, M.E., Shapiro, M.: Cantor-type sets in hyperbolic numbers. Fractals 24(04) (2016).
  * [4] Catoni, F., Boccaletti, D., Cannata, R., Catoni, V., Nichelatti, N. and Zampetti, P.: The mathematics of Minkowski space-time, Birkhauser Verlag, Basel (2008).
  * [5] Catoni, F., Boccaletti, D., Cannata, R., Catoni, V. and Zampetti, P.: Geometry of Minkowski space-time, Springer (2011).
  * [6] Ghosh, C., Bandyopadhyay, A., Mondal, S.: Hyperbolic valued metric space. Palestine Journal of Mathematics, 12(4), 37-48 (2023).
  * [7] Luna-Elizarraras, M.E., Shapiro, M., Struppa, D.C., Vajiac, A.: Bicomplex holomorphic functions: The algebra, geometry and analysis of bicomplex numbers, Frontiers in Mathematics, Birkhäuser Basel, 2015.
  * [8] Olariu, S.: Complex numbers in n-dimensions. North-Holland Mathematics Studies, 190, 51–148, Elsevier, Amsterdam, Boston (2002).
  * [9] Price, G. B.: An introduction to multicomplex spaces and functions, 3rd Edition, Marcel Dekker, New York, 1991.
  * [10] Saini, H., Sharma, A., Kumar, R., Some fundamental theorems of functional analysis with bicomplex and hyperbolic scalars, Adv. Appl. Clifford Algebras, 30(66), 01-23 (2020).
  * [11] Sobczyk, G., The hyperbolic number plane, Coll. Maths. Jour., 26 (4) (1995), 268-280.
  * [12] Tellez-Sanchez, G. Y., Bory-Reyes, J., Generalized iterated function systems on hyperbolic numbers Plane, Fractals, 27(04), doi:10.1142/S0218348X19500452 (2019).
  * [13] Zadeh, L. A., Fuzzy sets, Information and control, 8(3), 338-353 (1965).


# LLM Jailbreak Attack versus Defense Techniques - A Comprehensive Study

Zihao Xu1,2,∗ Yi Liu3,† Gelei Deng3,‡ Yuekang Li1,§ Stjepan Picek2,¶
1University of New South Wales, Australia
2Delft University of Technology, The Netherlands
3Nanyang Technological University, Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have increasingly become central to generating
content with potential societal impacts. Notably, these models have
demonstrated capabilities for generating content that could be deemed harmful.
To mitigate these risks, researchers have adopted safety training techniques
to align model outputs with societal values to curb the generation of
malicious content. However, the phenomenon of "jailbreaking" — where carefully
crafted prompts elicit harmful responses from models — persists as a
significant challenge. This research conducts a comprehensive analysis of
existing studies on jailbreaking LLMs and their defense techniques. We
meticulously investigate nine attack techniques and seven defense techniques
applied across three distinct language models: Vicuna, LLama, and GPT-3.5
Turbo. We aim to evaluate the effectiveness of these attack and defense
techniques. Our findings reveal that existing white-box attacks underperform
compared to universal techniques and that including special tokens in the
input significantly affects the likelihood of successful attacks. This
research highlights the need to concentrate on the security facets of LLMs.
Additionally, we contribute to the field by releasing our datasets and testing
framework, aiming to foster further research into LLM security. We believe
these contributions will facilitate the exploration of security measures
within this domain.

LLM Jailbreak Attack versus Defense Techniques - A Comprehensive Study

Zihao Xu1,2,∗ Yi Liu3,† Gelei Deng3,‡ Yuekang Li1,§ Stjepan Picek2,¶
1University of New South Wales, Australia 2Delft University of Technology, The
Netherlands 3Nanyang Technological University, Singapore<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs), such as GPT (OpenAI, 2023b) and LLaMa (Hugging
Face, 2023a), play a pivotal role across a spectrum of applications, from text
summarization (Tian et al., 2024) to code generation (Ni et al., 2023). The
popularity of LLMs in everyday scenarios underscores their significance.
However, this ubiquity also raises security concerns associated with LLMs
(Ouyang et al., 2022).

Several types of vulnerabilities have been identified in LLMs (OWASP, 2023).
Among these, the jailbreak attack stands out as a prevalent vulnerability,
where specially designed prompts are used to bypass the safety measures of
LLMs, facilitating the production of harmful content. There has been notable
research aimed at addressing jailbreak attacks. For example, Liu et al. (Liu
et al., 2023b) investigate various mechanisms for jailbreak prompting and
assess their effectiveness. Zou et al. (Zou et al., 2023) apply a white-box
approach combined with adversarial attacks to create jailbreak prompts.
Additionally, Deng et al. (Deng et al., 2023a) explore using LLMs to generate
jailbreak prompts in a black-box setting. To defend against jailbreak attacks,
Robey et al. (Robey et al., 2023) proposed a method that involves randomly
omitting a certain number of tokens from the input to detect malicious
attempts. Meanwhile, Pisano et al. (Pisano et al., 2023) introduced an
approach that employs an auxiliary model to assist the primary model in
identifying hazardous information.

Despite the various jailbreak attack and defense methodologies, to the best of
our knowledge, there remains a significant gap in the literature regarding
comprehensive evaluations of how well the attack methodologies can perform
against defended LLMs and how well defense mechanisms against jailbreak
attacks.

To address this research gap, we undertake a comprehensive empirical study on
jailbreak attack and defense techniques for LLMs. Our study is designed to
answer two critical research questions. First, we investigate the
effectiveness of various jailbreak attack approaches on different unprotected
LLMs, encapsulated in the question (RQ1: Effectiveness of Jailbreak Attacks).
Second, we evaluate the effectiveness of defense strategies against these
attacks on varied LLMs, posed as (RQ2: Effectiveness of Jailbreak Defenses).

Figure 1: The workflow of our study

During the Baseline Selection phase, we chose nine attack methods and seven
defense mechanisms, drawing on four seminal works, including notable libraries
(Automorphic, 2023; ProtectAI, 2023), and the OpenAI Moderation API (OpenAI,
2023), prioritizing prevalent and accessible methods with open-source code.

In the Benchmark Construction phase, our benchmark, initially based on (Liu et
al., 2023b), was expanded through additional research (Zou et al., 2023) and a
GPT model in "Do Anything Now" mode, resulting in 60 categorized malicious
queries following OpenAI’s guidelines.

For Result Labeling, the RoBERTa model was fine-tuned for classifying
malicious responses, achieving 92% accuracy, outperforming GPT-4’s 87.4%.
Manual validation ensured the reliability of our classification.

In the Evaluation Phase, we employed metrics for assessing attack efficiency
and effectiveness, alongside defense robustness against malicious and benign
inputs, establishing a comprehensive framework for evaluating LLM security.

Our analysis reveals several notable insights. Specifically, among the various
jailbreak attack techniques, template-based methods demonstrate superior
effectiveness. In contrast, gradient-based generative approaches, especially
in ’white-box’ scenarios, generally fall short of the performance achieved by
universal generative methods. Additionally, our findings highlight the
significant impact of special tokens on the success probability of attacks. As
for defense techniques, we identify the Bergeron method as the most effective
defense strategy to date, while all other defense techniques in our study
perform badly as they either cannot stop jailbreak attacks at all or are too
strict such that benign prompts are also prohibited. Our results underscore a
great need for the development of more robust defense mechanisms.

In summary, our work presents several contributions to the field:

  * •

Comprehensive Study. This study represents, to the best of our knowledge, the
first systematic evaluation of the effectiveness of jailbreak attacks versus
defenses on various open/closed-source LLMs.

  * •

Key Findings. We uncover previously unknown insights that hold significant
potential for enhancing both attack and defense strategies in the future.

  * •

Open-source Artifacts. We develop and publicly release the first benchmark
that includes a comprehensive collection of both attack and defense
techniques, thereby facilitating further research in this area.

The raw data, the benchmark platform, and additional details are available on
a companion website of this paper:
https://sites.google.com/view/llmcomprehensive/home.

##  2 Background and Related Work

###  2.1 LLM Jailbreak

Jailbreak attacks on LLMs involve crafting prompts that exploit the models to
generate malicious content. Despite the potential for harm, such as generating
instructions for fabricating explosives, LLMs typically refrain from producing
such responses due to the incorporation of safeguards during their training.
These measures include Reinforcement Learning from Human Feedback (RLHF)
(Ouyang et al., 2022), Robustness via Additional Fine-Tuning (RAFT) (Dong et
al., 2023), and Preference Optimized Ranking (PRO) (Song et al., 2023), which
collectively ensure the model’s adherence to ethical guidelines.

The precise mechanisms behind the jailbreak phenomena remain under debate. Wei
et al. (Wei et al., 2023) postulate that jailbreaks may occur in scenarios
where safety training is insufficiently comprehensive, allowing for the
generation of content in unmonitored areas, or when the model encounters
dilemmas between providing useful responses and maintaining safety protocols.
Complementing this, Subhash et al. (Subhash et al., 2023) explored the role of
the model’s hidden states in gradient-based attacks, identifying that a
specific suffix, when appended to the original prompt, serves as an embedding
vector guiding the model toward generating inappropriate content. This finding
aligns with the hypothesis that jailbreaks can manifest in regions not fully
covered by safety training, enabling the production of objectionable content.

“Benign content” is defined as responses considered morally or ethically
inappropriate, with OpenAI compiling an extensive list of such categories. Liu
et al. (Liu et al., 2023b) further elaborate on this classification, providing
a framework for categorizing these responses. The assessment presented herein
conforms to this established categorization, ensuring a structured approach to
understanding and mitigating jailbreak risks in LLMs.

Table 1: This table catalogs all identified attack techniques, marking the
ones selected for our investigation with *.

Category | Paper | Description
---|---|---
|  Chao et al. (2023)* | Employing the Chain of Thought (COT) (Wei et al., 2022) alongside Vicuna for generating prompts responsive to user feedback.
| Deng et al. (2023a) | Finetune of an LLM with RLHF to jailbreak target model.
| Lapid et al. (2023) | Implementation of a fuzzing methodology utilizing cosine similarity as the determinant for fitness scores.
|  Liu et al. (2023a)* | Application of a fuzzing approach, with the fitness score derived from loss metrics.
|  Mehrotra et al. (2023)* | An approach akin to Chao et al. (2023), employing the concept of a Tree of Thought(TOT) (Yao et al., 2023b).
|  Zou et al. (2023)* | Optimization at the token level informed by gradient data.
| Schwinn et al. (2023) | An approach parallel to Zou et al. (2023), but at the sentence level
| Shah et al. (2023) | Attack of a black-box model by leveraging a proxy model.
| Qiang et al. (2023) | An in-context learning attack resembling Zou et al. (2023)’s methodology.
|  Yu et al. (2023)* | A fuzzing method, through utilization of Monte Carlo tree search techniques to adjust fitness scores based on success rates.
Generative | Wu et al. (2023b) | Crafting of evasion prompts through GPT4, utilizing meticulously designed prompts to extract system prompts.
| Kang et al. (2023) | Segregation of sensitive lexicons into variables within templates.
| Yao et al. (2023a) | Integration of generative constraints and malevolent inquiries within specified templates.
|  Li et al. (2023a)* | Generation of wrapped scenarios to nudge models into responding to malevolent inquiries.
|  Wei et al. (2023)* | An exhaustive analysis covering 29 types of assault templates and combinations, including encoding techniques such as base64.
|  Huang et al. (2024)* | Modification of generative parameters, like temperature and top P.
|  Du et al. (2023)* | Using LLM intrinsic propensity to safety or not-aligned that is dependent on the previous prompts
Template |  Liu et al. (2023b)* | Compilation of 78 distinct template types.
| Deng et al. (2023b) | Exploration of various combinations of low-resource languages to circumvent model alignment.
| Xu et al. (2023) | Coaxing the model into generating harmful content by exploiting the model’s inferential capabilities.
Training Gaps | Yong et al. (2023) | An investigation similar to Deng et al. (2023b), identifying low-resource languages as effective for security circumvention.

###  2.2 Jailbreak Attack Techniques

To provide a structured overview of the strategies utilized to compromise
LLMs, we categorize current attack techniques into three categories,
reflecting their fundamental traits. The first category, Generative
Techniques, includes attacks that are dynamically produced, eschewing
predetermined plans. The second category, Template Techniques, comprises
attacks conducted via pre-defined templates or modifications in the generation
settings. The last category, Training Gaps Techniques, focuses on exploiting
weaknesses due to insufficient safeguards in safe training practices, such as
RLHF (Ouyang et al., 2022). The techniques employed in our study are
elaborated in Table 1, highlighting the models chosen for evaluation within
our framework.

Table 2: This table enumerates all recognized defense methodologies, with
those chosen for our analysis marked with an asterisk *. Additional defense
methods employed in this study from Github and API are not listed.

Category | Paper | Description
---|---|---
Self-Processing | Wu et al. (2023a) | Encapsulates the user’s inquiry within a system-generated prompt.
Zhang et al. (2023) | Leverages the model’s intrinsic conflict between assisting users and ensuring safety, as proposed by (Wei et al., 2023).
Li et al. (2023c) | Implements self-evaluation during inference, assessing word generation auto-regressively at the individual word level.
Piet et al. (2023) | Utilizes a standard LLM model devoid of chat instructions, solely inputting task-relevant data.
Helbling et al. (2023) | Employs meticulously devised system prompts for attack detection.
Additional Helper |  Pisano et al. (2023)* | Introduces a framework that employs an auxiliary LLM, using additional information to maintain the primary model’s alignment.
Input Permutation | Kumar et al. (2023) | Involves partial deletion of input content up to a specified length.
Cao et al. (2023)* | Modifies prompts through swapping, addition, or patching up to a predetermined percentage.
Robey et al. (2023)* | Implements random input dropping up to a specified percentage.
Perplexity | Hu et al. (2023) | Calculates token-level perplexity using a probabilistic graphical model and evaluates the likelihood of each token being part of a malicious suffix.
Jain et al. (2023)* | Derives perplexity from the average negative log-likelihood of each token’s occurrence.

###  2.3 Jailbreak Defense Techniques

We further conduct thorough examination on the existing defense mechanisms,
classifying them into three primary categories based on their operational
principles: Self-Processing Defenses, which rely exclusively on the LLM’s own
capabilities; Additional Helper Defenses, which require the support of an
auxiliary LLM for verification purposes; and Input Permutation Defenses, which
manipulate the input prompt to detect and counteract malicious requests aimed
at exploiting gradient-based vulnerabilities. An overview of these defense
mechanisms is presented in Table 2.

##  3 Study Design

Our study aims to address two core research questions:

RQ1 (Effectiveness of Jailbreak Attacks): How effective are jailbreak attack
techniques across various LLMs?

RQ2 (Effectiveness of Jailbreak Defenses): How effective are jailbreak defense
techniques against various attack techniques when protecting different LLMs?

Baseline Selection Our methodology selection criteria were predicated on the
method’s popularity and accessibility. For RQ1, our analysis covers nine
attack methods, divided into five generative (AutoDAN (Liu et al., 2023a),
PAIR (Chao et al., 2023), TAP (Mehrotra et al., 2023), GPTFuzz (Yu et al.,
2023), GCG (Zou et al., 2023)) and four template-based approaches (Jailbroken
(Wei et al., 2023), 77 Templates from existing study (Liu et al., 2023b), Deep
Inception (Li et al., 2023a), Parameters (Huang et al., 2024)). The
distinction between generative and template-based strategies aids in
comprehensive coverage of attack methodologies.

For RQ2, we examine four defensive strategies: Bergeron (Pisano et al., 2023)
for additional helper methods; RALLM (Cao et al., 2023) and SmoothLLM (Robey
et al., 2023) for input permutation techniques; and Baseline (Jain et al.,
2023) for perplexity analysis. Notable open-source projects, Aegis
(Automorphic, 2023) and LLMguard (ProtectAI, 2023), alongside the OpenAI
Moderation API (OpenAI, 2023), are also evaluated for their defense efficacy.
Limitations such as Rain’s (Ouyang et al., 2022) lengthy processing and
Certifying-llm’s (Kumar et al., 2023) scalability issues are acknowledged to
refine our selection criteria.

###  3.1 LLMs under Test

In our research, we focus on evaluating three distinguished models: Llama-2-7b
(Hugging Face, 2023a), Vicuna-v1.5-7b (Hugging Face, 2023b), and
GPT-3.5-Turbo-1106 (OpenAI, 2023b). These models were chosen due to their
prevalent use in security-related research, encompassing both attack
simulations and the development of defensive strategies. The decision to omit
GPT-4 from our evaluation stems from its significant operational requirements.
Preliminary assessments of GPT-3.5-Turbo indicated extensive query processing
times, reaching a total of 79,314, in addition to the economic implications
linked to GPT-4’s token pricing—set at $0.01 per 1,000 tokens (OpenAI, 2023a).
These constraints make the inclusion of GPT-4 in a comparative analysis
economically prohibitive.

###  3.2 Experimental Configuration

Our experimental framework utilized two NVIDIA RTX 6000 Ada GPUs, each
outfitted with 48 GB of RAM. We aligned our testing parameters with those
identified as optimal in the relevant literature, defaulting to the original
repositories’ settings in the absence of specific recommendations. To address
Research Question 1 (RQ1) and ensure consistency across different attack
methodologies, each query was executed five times to minimize variability. For
the evaluation involving generative models, we capped the process at a maximum
of 75 iterations for each query, defining an iteration as a single algorithmic
step.

###  3.3 Benchmark Construction

We leveraged the benchmark framework proposed by Liu et al. (Liu et al.,
2023b). This benchmark is distinguished by its rigorous focus on policy
compliance within the context of malicious content detection. In an effort to
enhance the robustness of our evaluation, we expanded the original dataset to
include 60 malicious queries, effectively doubling its size. This augmentation
was achieved through meticulous manual curation and integrating selected
examples from AdvBench (Zou et al., 2023). Our approach to dataset expansion
adhered strictly to the categorization and selection criteria established in
previous studies, ensuring both the consistency and the relevance of the
enhanced dataset for comprehensive evaluation.

###  3.4 Result Labeling

In our study, we employed both automated and manual labeling strategies to
categorize the responses gathered from our evaluation process. With a dataset
encompassing 199,427 responses, the task of identifying malicious content
posed significant challenges due to the impracticality of manual review and
the absence of standardized evaluation methods for malicious responses—a gap
in current research.

Existing strategies for addressing this issue vary. Zou et al. (Zou et al.,
2023) utilized a set of common refusal patterns, such as "I am sorry" and "I
cannot", to automate the identification of non-compliant responses. Yu et al.
(Yu et al., 2023) and Huang et al. (Huang et al., 2024) focused on enhancing
machine learning models, specifically RoBERTa and BERT-BASE-CASED models,
respectively. Additionally, approaches leveraging GPT-4 for attack analysis
were explored by Chao et al. (Chao et al., 2023) and Mehrotra et al. (Mehrotra
et al., 2023).

To ensure a rigorous analysis of our dataset, we selected a random sample of
1,068 responses, aiming for a 95% confidence level with a 3% margin of error.
This sample underwent manual annotation to assess each response’s relevance to
the inquiry and its alignment with providing a malicious answer. An example of
this annotation process is documented in Table 6, and a comparative analysis
of the effectiveness of different models is provided in Table 5.

Reflecting on the reported efficacy of the fine-tuned RoBERTa model by Yu et
al. (Yu et al., 2023), we chose to refine this model further utilizing our
manually annotated dataset, accessible on HuggingFace (fine tuned, 2024). The
fine-tuning protocol involved a batch size of 5, three training epochs, a
learning rate of $2\times 10^{-5}$, application of the Adam optimizer, and a
linear rate decay complemented by a warm-up phase covering 10% of the training
duration. Post-labeling, an additional round of random sampling was conducted
for manual verification to ascertain the accuracy and reliability of our
findings.

Evaluation Metric For RQ1, we use two metrics.

This dual metric approach ensures a comprehensive evaluation of both the
attack’s impact and its operational feasibility. First, Attack Success Rate
(ASR): defined as the ratio of successfully compromised questions c to the
total number of questions n, ASR measures the effectiveness of an attack.

$ASR=\frac{c}{n}.$ (1)

Second, Efficiency: this metric quantifies the effectiveness of attack
queries, defined as the ratio of the number of individual queries $q$ that
successfully compromise the model to the total number of query attempts $o$.
Each query represents a minimal experimental unit or a single prompt.

$Efficiency=\frac{q}{o}.$ (2)

For RQ2, we introduce three metrics that ensure a balanced assessment of
system robustness and output integrity. The first, defense Passing Rate (DPR),
calculates the ratio of prompts f that incorrectly bypass the defense
mechanism—being erroneously classified as harmless—to the total number of
malicious inputs m.

$DPR=\frac{f}{m}.$ (3)

The second metric, Benign Passing Rate (BPR), assesses the proportion of non-
malicious inputs s that successfully navigate through the defense filter
relative to the total number of inputs t.

$BPR=\frac{s}{t}.$ (4)

Lastly, the Generated Response Quality (GRQ) evaluates the quality of
responses generated by defense mechanisms compared to a standard reference. To
assess the responses to benign queries, we employ the Alpaca Eval framework
(Li et al., 2023b), leveraging its methodology for automatically evaluating
response quality. Evaluating GRQ is crucial for methodologies that produce new
responses (Cao et al., 2023; Robey et al., 2023; Pisano et al., 2023).

##  4 RQ1: Effectiveness of Jailbreak Attack

The effectiveness of attack strategies on the selected LLMs under test is
systematically presented in Tables 6, 7, and 8. These tables enumerate the
number of queries that were effectively bypassed under various categories,
corresponding to the detailed breakdowns provided in Tables 9, 10, and 11. To
offer a clearer comparative analysis of model performance, we consolidated
these metrics into a scatter plot depicted in Figure 3. In this visualization,
models demonstrating optimal performance are positioned nearer to the scatter
plot’s upper right quadrant, signifying superior ASR and Efficiency.

Evaluation results reveal that using 78 templates, Jailbroken, and GPTFuzz
strategies yield superior results in circumventing the security measures of
GPT-3.5-turbo and Vicuna. Conversely, for LLaMA, strategies such as
Jailbroken, Parameter, and 78 templates demonstrated the highest
effectiveness. This prevalence of template-based approaches highlights their
efficiency, primarily due to the intricate design of their prompts. The most
successful templates from these strategies are detailed in Table 16, which
will be added subsequently.

In the realm of generative strategies, GPTFuzz, Pair, and Tap emerged as the
top performers. Moreover, it was noted that LLaMA presents a noteworthy
challenge for jailbreaking compared to Vicuna. Specifically, Vicuna showed a
quicker reduction in loss when employing the AutoDan method and achieved
jailbreak at a higher loss threshold. In contrast, LLaMA exhibited a slower
loss reduction rate and, despite achieving a lower loss than Vicuna, did not
succumb to jailbreaking, as illustrated in Figure 2. Additionally, our study
into the categories of questions that were successfully jailbroken indicates
that queries related to harmful content and illegal activities are the most
challenging to address across all tested models.

Figure 2: Loss of a random question  Figure 3: Performance of Attacks on three
models. Note: For readability, we intentionally enlarged the size of the
labels for the best-performing items (top-right corner). A larger version of
this figure is available on our website.

##  5 RQ2: Effectiveness of Jailbreak defense

Our study meticulously evaluates defense mechanisms against malicious queries
as well as the handling of benign questions. The outcomes of this evaluation
are systematically tabulated in Tables 14, 13, and 12. These results are
further visualized in Figure 4, where the optimal defense strategies are
identified by their proximity to the upper left corner of the plot, signifying
lower DPR and higher BPR. Our findings reveal that, apart from the Bergeron
method, the efficacy of the current defense strategies remains largely
inadequate. Additionally, our comparative analysis of the quality of benign
responses generated through three innovative methodologies disclosed minimal
variance among them, as elaborated in Table 15.

Figure 4: Performance of defense on three models. Note: For readability, we
intentionally enlarged the size of the labels for the best-performing items
(top-left corner). A larger version of this figure is available on our
website.

##  6 Discussion

###  6.1 Comparative Performance of White-Box and Black-Box Attacks

Our investigation reveals that white-box attacks are less effective than
black-box jailbreak strategies. Specifically, methods like AutoDan and GCG,
which rely on insights into the model’s internal mechanisms, such as loss
metrics, underperform when compared to universal, template-based attack
methods that do not necessitate access to a model’s internals and are pre-
designed. Moreover, the LLaMa model presents more significant challenges to
jailbreaking efforts, particularly under white-box attack strategies, in
comparison to Vicuna. This observation is intriguing, especially considering
that Vicuna is an evolution of LLaMa, having been refined through additional
fine-tuning processes (LMSYS, 2023). The pronounced resilience of LLaMa
against attacks highlights the critical role of comprehensive safety training
during its development phase, suggesting that such training is a crucial
element in bolstering the defenses of open-source LLMs.

To further understand the influence of loss metrics on a model’s vulnerability
to jailbreaking, we conducted a targeted experiment. A question was randomly
selected from our dataset, and the experiment’s findings are visually
represented in Figure 2. The experiment showed that Vicuna began the process
with a higher initial loss but saw a significant reduction in loss,
stabilizing after 12 steps and five successful jailbreak attempts. However, it
maintained a higher final loss compared to LLaMa. In contrast, LLaMa started
with a lower initial loss and demonstrated a slower reduction in loss over
time, ultimately failing to jailbreak the question despite exhibiting a
significantly lower final loss than Vicuna. These outcomes suggest that
LLaMa’s foundational safety training plays a pivotal role in its enhanced
defense against jailbreak attempts. It implies that integrating advanced
safety training protocols into developing open-source models could markedly
reduce the efficacy of white-box attacks, thereby enhancing their security
posture.

###  6.2 Impact of Special Tokens on Jailbreak Attack Performance

Our research has uncovered that using special tokens significantly influences
the success rates of jailbreak attack techniques. Specifically, the deployment
of 78 templates on GPT-3.5-Turbo and Vicuña models has spotlighted the
substantial effect of the special token ‘[/INST]’ on compromising the LLaMa
model. Through methodical experimentation with these templates, as
systematically documented in Table 3, we sought to understand the differential
impact of various configurations on attack effectiveness.

The analysis focused on four distinct settings, leading to the identification
of five templates that demonstrated the most significant disparities in
performance, detailed in Table 4. Notably, we discovered that text
continuation templates were rendered ineffective by the inclusion of
‘[/INST]’, rationalized by the implication that the absence of this token
signals an intent for incomplete input, thereby encouraging the model to
supplement the provided prompt before generating a response. This pattern was
consistent across multiple templates, showing that the model often elaborates
on the user’s initial prompt, though not invariably resulting in a successful
attack.

Particularly interesting was the observation that omitting ‘[/INST]’ from two
specific templates reduced the probability of a successful jailbreak. These
templates predominantly involved scenarios of character role-play, as
characterized in previous studies (Liu et al., 2023b). The underlying reasons
for this discrepancy are not immediately apparent, suggesting a potential
misalignment possibly due to gaps in the model’s safety training.

Given these preliminary findings, there is a compelling need for more
comprehensive investigations into the influence of special tokens on the
behavior of LLMs in the context of security vulnerabilities. Future research
should extend beyond ‘[/INST]’ to explore the effects of other special tokens,
such as ‘<<sys>>’ and ‘<s>’ on LLaMa and additional models. Such studies
promise to deepen our understanding of model responses to crafted inputs and
inform the development of more robust defense mechanisms against jailbreak
attacks.

Table 3: The effect of applying [INST] and [/INST]’s influence the successful rates of 78 templates method. Pattern | ASR | Efficiency | Performance Difference
---|---|---|---
[INST] + input + [/INST] | 83.33% (50/60) | 3.3% (767/23100) | 100.00%
[INST] + input | 100.00% (60/60) | 13.94% (3221/23100) | 419.94%
input + [/INST] | 76.67% (46/60) | 2.8% (659/23100) | 85.91%
input | 100.00% (60/60) | 12.65% (2923/23100) | 381.09%
Table 4: Templates exhibiting the most significant variation across four
configurations for Llama-2, utilizing 78 templates.

Templates performing better without [/INST] | [INST]+input+[/INST] | [INST] + input | input + [/INST] | input
---|---|---|---|---
Text Continuation | 0 | 274 | 0 | 282
AIM | 5 | 104 | 5 | 89
KEVIN | 11 | 105 | 5 | 74
DeltaGPT | 2 | 106 | 0 | 115
Templates performing better with [/INST] |  |  |  |
APOPHIS | 96 | 56 | 89 | 55
BasedBOB | 58 | 35 | 58 | 30

###  6.3 Enhancing Defense Mechanisms Against Diverse Malicious Queries

Our research underscores the imperative need for robust defense mechanisms to
be rigorously evaluated against a broad spectrum of malicious queries. The
development of a uniform and standardized evaluation methodology emerges as a
pivotal challenge within the realm of defense strategies. Present techniques,
as outlined by Kumar et al. (Kumar et al., 2023), Robey et al. (Robey et al.,
2023), and Cao et al. (Cao et al., 2023), rely on predefined sets that
frequently misclassify benign responses, such as "I’m sorry, I do not
understand," as malicious. This issue of misclassification becomes even more
pronounced when incorporating models like GPT-4 with these sets, as GPT-4’s
limitations in generating contextually accurate responses can lead to benign
inputs being wrongly identified as malicious, as illustrated in Figure 5.

Moreover, our analysis reveals that the efficacy of existing defense
strategies is generally insufficient. While the Bergeron method shows promise,
its effectiveness is limited by its dependence on LLMs and the use of
predefined prompts for processing and verification, which not only undermines
its reliability but also entails substantial operational costs. Proposals by
Kumar et al., Robey et al., and Cao et al. for identifying malicious prompts
through suffix addition demonstrate potential but struggle with scalability
issues, particularly when dealing with complex natural language inputs.

Given these observations, there is a critical and pressing need for further
research into more advanced evaluation frameworks and formulating more
effective defense strategies. Such efforts should aim to circumvent the
current limitations by ensuring reliable differentiation between malicious and
benign inputs across varying contexts and increasing the scalability of
defense mechanisms to accommodate the complexities inherent in natural
language processing.

##  7 Conclusions

In this work, we present the first comprehensive assessment of existing attack
and defense strategies in the context of LLM security. Additionally, we
contribute to the field by releasing the first framework specifically designed
for assessing the robustness of LLMs against various threats. We selected nine
attacks and seven defensive mechanisms from existing literature and software
libraries for our analysis. Our experimentation, conducted on three distinct
models, reveals that Template methods are notably effective, with an average
of 78 templates identified as critical for thwarting jailbreak attempts.
Regarding Generative methods, GPTFuzz emerged as the most efficacious. Our
investigation into question categorization demonstrated that all three models
exhibit enhanced resilience against queries related to harmful content and
illegal activities. However, our analysis of current defensive measures
indicates a general ineffectiveness, with Bergeron showing comparatively
better performance. We highlight the necessity of establishing a uniform
baseline for jailbreak detection, as existing defenses employ varied
methodologies, and the need to develop better defense techniques.
Additionally, our study observed the impact of using the ’[/INST]’ marker in
the Llama model. Looking forward, we aim to continuously incorporate evolving
attacks and defenses into our framework, thereby providing a dynamic overview
of the field’s progression.

##  8 Limitations

To address the constraints posed by limited resources, our evaluation does not
extend to larger models, such as those with 13 billion and 33 billion
parameters, nor does it cover powerful models like GPT-4 and other commercial
models, including Gemini (Gemini, ) and Palm2 (AI, ). Regarding autoDan, it is
noteworthy that significant updates were identified in its repository as of
February 2024. Given that our evaluation was completed prior to these updates,
the outcomes may be impacted. Nonetheless, we intend to align our repository
with these recent modifications soon.

##  9 Ethical Considerations and Disclaimer

In conducting this study, our research team has committed to the highest
standards of ethical conduct by exclusively utilizing resources that are
publicly accessible. We have undertaken this research with a conscientious
commitment to ethical principles, ensuring that all of our activities are
aligned with the established norms and guidelines of responsible scientific
inquiry.

Throughout the course of our investigation, we have steadfastly avoided
engaging in any activities that involve the creation, dissemination, or
promotion of content that could be deemed malicious or harmful. Our approach
to research is guided by a strong ethical compass, ensuring that our
methodologies and findings do not contribute to the proliferation of such
content.

In the spirit of transparency and accountability, we have taken proactive
steps to ensure that all of our findings are managed with the utmost
responsibility. This includes the systematic reporting of our results to the
developers and providers of the models we have analyzed. Our aim is to
contribute constructively to the ongoing dialogue regarding the security of
LLMs and to aid in the identification and mitigation of potential
vulnerabilities.

## References

  * (1) Google AI.  Google ai palm 2.  https://ai.google/discover/palm2/.  Accessed: [Insert Access Date Here].
  * Automorphic (2023) Automorphic. 2023.  Aegis.  https://github.com/automorphic-ai/aegis.  Accessed: 2024-02-13.
  * Cao et al. (2023) Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023.  Defending against alignment-breaking attacks via robustly aligned llm.  _arXiv preprint arXiv:2309.14348_.
  * Chao et al. (2023) Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023.  Jailbreaking black box large language models in twenty queries.  _arXiv preprint arXiv:2310.08419_.
  * Deng et al. (2023a) Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023a.  Jailbreaker: Automated jailbreak across multiple large language model chatbots.  _arXiv preprint arXiv:2307.08715_.
  * Deng et al. (2023b) Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023b.  Multilingual jailbreak challenges in large language models.  _arXiv preprint arXiv:2310.06474_.
  * Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023.  Raft: Reward ranked finetuning for generative foundation model alignment.  _arXiv preprint arXiv:2304.06767_.
  * Du et al. (2023) Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin. 2023.  Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak.  _arXiv preprint arXiv:2312.04127_.
  * fine tuned (2024) X. fine tuned. 2024.  FT-Roberta-LLM: A Fine-Tuned Roberta Large Language Model.  https://huggingface.co/zhx123/ftrobertallm/tree/main.
  * (10) Gemini.  Buy, sell & trade bitcoin & other crypto currencies with gemini’s platform.  https://www.gemini.com/eu.  Accessed: [Insert Access Date Here].
  * Helbling et al. (2023) Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. 2023.  Llm self defense: By self examination, llms know they are being tricked.  _arXiv preprint arXiv:2308.07308_.
  * Hu et al. (2023) Zhengmian Hu, Gang Wu, Saayan Mitra, Ruiyi Zhang, Tong Sun, Heng Huang, and Vishy Swaminathan. 2023.  Token-level adversarial prompt detection based on perplexity measures and contextual information.  _arXiv preprint arXiv:2311.11509_.
  * Huang et al. (2024) Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2024.  Catastrophic jailbreak of open-source LLMs via exploiting generation.  In _The Twelfth International Conference on Learning Representations_.
  * Hugging Face (2023a) Hugging Face. 2023a.  Meta llama.  https://huggingface.co/meta-llama.  Accessed: 2024-02-14.
  * Hugging Face (2023b) Hugging Face. 2023b.  Vicuna 7b v1.5.  https://huggingface.co/lmsys/vicuna-7b-v1.5.  Accessed: 2024-02-14.
  * Jain et al. (2023) Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023.  Baseline defenses for adversarial attacks against aligned language models.  _arXiv preprint arXiv:2309.00614_.
  * Kang et al. (2023) Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023.  Exploiting programmatic behavior of llms: Dual-use through standard security attacks.  _arXiv preprint arXiv:2302.05733_.
  * Kumar et al. (2023) Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. 2023.  Certifying llm safety against adversarial prompting.  _arXiv preprint arXiv:2309.02705_.
  * Lapid et al. (2023) Raz Lapid, Ron Langberg, and Moshe Sipper. 2023.  Open sesame! universal black box jailbreaking of large language models.  _arXiv preprint arXiv:2309.01446_.
  * Li et al. (2023a) Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023a.  Deepinception: Hypnotize large language model to be jailbreaker.  _arXiv preprint arXiv:2311.03191_.
  * Li et al. (2023b) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval.
  * Li et al. (2023c) Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023c.  Rain: Your language models can align themselves without finetuning.  _arXiv preprint arXiv:2309.07124_.
  * Liu et al. (2023a) Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023a.  Autodan: Generating stealthy jailbreak prompts on aligned large language models.  _arXiv preprint arXiv:2310.04451_.
  * Liu et al. (2023b) Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023b.  Jailbreaking chatgpt via prompt engineering: An empirical study.  _arXiv preprint arXiv:2305.13860_.
  * LMSYS (2023) LMSYS. 2023.  Vicuna 7b v1.5: A chat assistant fine-tuned on sharegpt conversations.  https://huggingface.co/lmsys/vicuna-7b-v1.5.  Accessed: [Insert access date here].
  * Mehrotra et al. (2023) Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2023.  Tree of attacks: Jailbreaking black-box llms automatically.  _arXiv preprint arXiv:2312.02119_.
  * Ni et al. (2023) Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023.  Lever: Learning to verify language-to-code generation with execution.  In _International Conference on Machine Learning_ , pages 26106–26128. PMLR.
  * OpenAI (2023) OpenAI. 2023.  Moderation guide.  https://platform.openai.com/docs/guides/moderation.  Accessed: 2024-02-13.
  * OpenAI (2023a) OpenAI. 2023a.  Openai pricing.  https://openai.com/pricing.  Accessed: 2024-02-14.
  * OpenAI (2023b) OpenAI. 2023b.  Research overview.  https://openai.com/research/overview.  Accessed: 2024-02-14.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * OWASP (2023) OWASP. 2023.  OWASP Top 10 for LLM Applications.  https://owasp.org/www-project-top-10-for-large-language-model-applications/.
  * Piet et al. (2023) Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2023.  Jatmo: Prompt injection defense by task-specific finetuning.  _arXiv preprint arXiv:2312.17673_.
  * Pisano et al. (2023) Matthew Pisano, Peter Ly, Abraham Sanders, Bingsheng Yao, Dakuo Wang, Tomek Strzalkowski, and Mei Si. 2023.  Bergeron: Combating adversarial attacks through a conscience-based alignment framework.  _arXiv preprint arXiv:2312.00029_.
  * ProtectAI (2023) ProtectAI. 2023.  Llm-guard.  https://github.com/protectai/llm-guard.  Accessed: 2024-02-13.
  * Qiang et al. (2023) Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. 2023.  Hijacking large language models via adversarial in-context learning.  _arXiv preprint arXiv:2311.09948_.
  * Robey et al. (2023) Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023.  Smoothllm: Defending large language models against jailbreaking attacks.  _arXiv preprint arXiv:2310.03684_.
  * Schwinn et al. (2023) Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. 2023.  Adversarial attacks and defenses in large language models: Old and new threats.  _arXiv preprint arXiv:2310.19737_.
  * Shah et al. (2023) Muhammad Ahmed Shah, Roshan Sharma, Hira Dhamyal, Raphael Olivier, Ankit Shah, Dareen Alharthi, Hazim T Bukhari, Massa Baali, Soham Deshmukh, Michael Kuhlmann, et al. 2023.  Loft: Local proxy fine-tuning for improving transferability of adversarial attacks against large language model.  _arXiv preprint arXiv:2310.04445_.
  * Song et al. (2023) Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2023.  Preference ranking optimization for human alignment.  _arXiv preprint arXiv:2306.17492_.
  * Subhash et al. (2023) Varshini Subhash, Anna Bialas, Weiwei Pan, and Finale Doshi-Velez. 2023.  Why do universal adversarial attacks work on large language models?: Geometry might be the answer.  _arXiv preprint arXiv:2309.00254_.
  * Tian et al. (2024) Shubo Tian, Qiao Jin, Lana Yeganova, Po-Ting Lai, Qingqing Zhu, Xiuying Chen, Yifan Yang, Qingyu Chen, Won Kim, Donald C Comeau, et al. 2024.  Opportunities and challenges for chatgpt and large language models in biomedicine and health.  _Briefings in Bioinformatics_ , 25(1):bbad493.
  * Wei et al. (2023) Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023.  Jailbroken: How does llm safety training fail?  _arXiv preprint arXiv:2307.02483_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wu et al. (2023a) Fangzhao Wu, Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, and Xing Xie. 2023a.  Defending chatgpt against jailbreak attack via self-reminder.
  * Wu et al. (2023b) Yuanwei Wu, Xiang Li, Yixin Liu, Pan Zhou, and Lichao Sun. 2023b.  Jailbreaking gpt-4v via self-adversarial attacks with system prompts.  _arXiv preprint arXiv:2311.09127_.
  * Xu et al. (2023) Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, and Muhao Chen. 2023.  Cognitive overload: Jailbreaking large language models with overloaded logical thinking.  _arXiv preprint arXiv:2311.09827_.
  * Yao et al. (2023a) Dongyu Yao, Jianshu Zhang, Ian G Harris, and Marcel Carlsson. 2023a.  Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models.  _arXiv preprint arXiv:2309.05274_.
  * Yao et al. (2023b) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023b.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Yong et al. (2023) Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023.  Low-resource languages jailbreak gpt-4.  _arXiv preprint arXiv:2310.02446_.
  * Yu et al. (2023) Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023.  Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.  _arXiv preprint arXiv:2309.10253_.
  * Zhang et al. (2023) Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. 2023.  Defending large language models against jailbreaking attacks through goal prioritization.  _arXiv preprint arXiv:2311.09096_.
  * Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023.  Universal and transferable adversarial attacks on aligned language models.  _arXiv preprint arXiv:2307.15043_.

##  Appendix A APPENDIX

Table 5: The accuracy of four evaluators and the refined RoBERTa model. GPT-4 | RoBERTa | Prefix Set | DistillBert | Finetuned RoBERTa
---|---|---|---|---
0.874 | 0.901 | 0.78 | 0.819 | 0.92
Table 6: The attack results of GPT-3.5-turbo, the top three best attacks in terms of ASR and efficiency are highlighted Attack Name | Use Scenario | Type | ASR | Efficiency
---|---|---|---|---
DeepInception | Universal | Template | 5.00% (3/60) | 4.33% (13/300)
GPTFUZZ | Universal | Generative | 100.00% (60/60) | 18.72% (305/1629)
TAP | Universal | Generative | 63.33% (38/60) | 6.32% (272/4300)
PAIR | Universal | Generative | 80.00% (48/60) | 6.85% (280/4085)
Jailbroken | Universal | Template | 100.00% (60/60) | 17.92% (1613/9000)
78 templates | Universal | Template | 100.00% (60/60) | 21.6% (5000/23100)
Parameter | Universal | Template | 5.00% (3/60) | 2.15% (794/36900)
Table 7: The attack results of Vicuna, the top three best attacks in terms of ASR and efficiency are highlighted Attack Name | Use Scenario | Type | ASR | Efficiency
---|---|---|---|---
AUTODAN | White Box | Generative | 70.00% (42/60) | 20.44% (252/1233)
GCG | White Box | Generative | 55.00% (33/60) | 14.06% (124/882)
DeepInception | Universal | Template | 10.00% (6/60) | 10.00% (30/300)
GPTFUZZ | Universal | Generative | 100% (60/60) | 50.23% (325/647)
TAP | Universal | Generative | 83.33% (50/60) | 12.78% (461/3606)
PAIR | Universal | Generative | 95.00% (57/60) | 14.81% (402/2715)
jailbroken | Universal | Template | 100.00% (60/60) | 23.38% (2104/9000)
78jailbreak template | Universal | Template | 100.00% (60/60) | 56.97% (13161/23100)
Parameter | Universal | Template | 90.00% (54/60) | 20.33% (3050/15000)
Table 8: The attack results of Llama, the top three best attacks in terms of ASR and efficiency are highlighted. Although the ASR of the Parameter is slightly lower than that of the Pair, its significantly higher efficiency positions the Parameter as the better choice. Attack Name | Use Scenario | Type | ASR | Efficiency
---|---|---|---|---
AUTODAN | White Box | Generative | 13.33% (8/60) | 1.39% (56/4022)
GCG | White Box | Generative | 0.00% (0/60) | 0.00% (0/3930)
DeepInception | Universal | Template | 0.00% (0/60) | 0.00% (0/300)
GPTFUZZ | Universal | Generative | 43.33% (26/60) | 1.67% (75/4500)
TAP | Universal | Generative | 38.33% (23/60) | 1.78% (79/4435)
PAIR | Universal | Generative | 73.33% (44/60) | 2.77% (119/4300)
jailbroken | Universal | Template | 100% (60/60) | 10.83% (975/9000)
78jailbreak template | Universal | Template | 83.33% (50/60) | 3.32% (767/23100)
Parameter | Universal | Template | 71.67% (43/60) | 10.39% (1559/15000)
Table 9: For each method of attack applied on GPT-3.5-turbo, the count of
questions from relevant categories that are successfully breached

Question Categories
---
Attack Name | harmful_content | adult_content | illegal_activity | fraudulent_deceptive_activities | violating_privacy | unlawful_practice
DeepInception | 0 | 1 | 0 | 0 | 1 | 1
GPTFUZZ | 10 | 10 | 10 | 10 | 10 | 10
TAP | 6 | 7 | 9 | 8 | 5 | 3
PAIR | 9 | 10 | 6 | 8 | 10 | 5
jailbroken | 10 | 10 | 10 | 10 | 10 | 10
78jailbreak template | 10 | 10 | 10 | 10 | 10 | 10
Parameter | 0 | 2 | 0 | 1 | 1 | 1
Sums | 45 | 50 | 45 | 47 | 47 | 40

Table 10: For each method of attack applied on Vicuna, the count of questions
from relevant categories that are successfully breached

Question Categories
---
Attack Name | harmful_content | adult_content | illegal_activity | fraudulent_deceptive_activities | violating_privacy | unlawful_practice
AUTODAN | 5 | 9 | 9 | 9 | 7 | 3
GCG | 5 | 7 | 6 | 8 | 5 | 2
DeepInception | 1 | 1 | 1 | 1 | 2 | 0
GPTFUZZ | 10 | 10 | 10 | 10 | 10 | 10
TAP | 6 | 9 | 10 | 9 | 9 | 7
PAIR | 10 | 10 | 10 | 10 | 10 | 7
jailbroken | 10 | 10 | 10 | 10 | 10 | 10
78jailbreak template | 10 | 10 | 10 | 10 | 10 | 10
Parameter | 9 | 10 | 10 | 10 | 10 | 5
Sums | 66 | 76 | 76 | 77 | 73 | 54

Table 11: For each method of attack applied on Llama, the count of questions
from relevant categories that are successfully breached

Question Categories
---
Attack Name | harmful_content | adult_content | illegal_activity | fraudulent_deceptive_activities | violating_privacy | unlawful_practice
AUTODAN | 0 | 0 | 0 | 6 | 1 | 1
GCG | 0 | 0 | 0 | 0 | 0 | 0
DeepInception | 0 | 0 | 0 | 0 | 0 | 0
GPTFUZZ | 2 | 3 | 3 | 3 | 6 | 9
TAP | 3 | 4 | 5 | 2 | 6 | 3
PAIR | 6 | 8 | 8 | 6 | 8 | 8
jailbroken | 10 | 10 | 10 | 10 | 10 | 10
78jailbreak template | 5 | 6 | 10 | 10 | 9 | 10
Parameter | 6 | 9 | 8 | 9 | 7 | 4
Sums | 32 | 40 | 44 | 46 | 47 | 45

Table 12: This table delineates the efficacy of various defense strategies
against attacks for Llama-2, highlighting the three most effective strategies
while excluding Aegis for its notably high false positive rate.

defense Method | BSR | DPR | Average
---|---|---|---
|  | AutoDan | DeepInception | GPTFUZZ | TAP | PAIR | jailbroken | 78jailbreak template | Parameters | GCG |
Aegis | 0.00% (0/805) | 0.00% (0/56) | 0.00% (0/0) | 0.00% (0/75) | 0.00% (0/79) | 0.00% (0/119) | 0.00% (0/975) | 0.00% (0/767) | 0.00% (0/1559) | 0.00% (0/0) | 0.00%
llm-guard | 99.13% (798/805) | 0.00% (0/56) | 0.00% (0/0) | 0.00% (0/75) | 100% (79/79) | 100.00% (119/119) | 33.33% (325/975) | 9.51% (73/767) | 97.62% (1522/1559) | 0.00% (0/0) | 37.82%
smooth-llm | 93.79% (755/805) | 53.57% (30/56) | 0.00% (0/0) | 37.33% (28/75) | 78.48% (62/79) | 77.31% (92/119) | 12.82% (125/975) | 35.20% (270/767) | 10.07% (157/1559) | 0.00% (0/0) | 27.91%
Baseline-defense | 69.07% (556/805) | 0.00% (0/56) | 0.00% (0/0) | 10.66% (8/75) | 89.87% (71/79) | 94.11% (112/119) | 33.33% (325/975) | 3.12% (24/767) | 82.16% (1281/1559) | 0.00% (0/0) | 34.80%
RA-LLM | 88.45% (712/805) | 76.78% (43/56) | 0.00% (0/0) | 60.00% (45/75) | 67.08% (53/79) | 59.66% (71/119) | 15.89% (155/975) | 57.88% (444/767) | 5.83% (91/1559) | 0.00% (0/0) | 29.59%
Bergeron | 98.51% (793/805) | 12.5% (7/56) | 0.00% (0/0) | 5.33% (4/75) | 25.31% (20/79) | 22.68% (27/119) | 5.74% (56/975) | 7.95% (61/767) | 7.24% (113/1559) | 0.00% (0/0) | 8.25%
ModerationAPI | 99.63% (802/805) | 100% (56/56) | 0.00% (0/0) | 77.33% (58/75) | 98.73% (78/79) | 99.15% (118/119) | 88.00% (858/975) | 88.78% (681/767) | 96.72% (1508/1559) | 0.00% (0/0) | 60.96%

Table 13: This table delineates the efficacy of various defense strategies
against attacks for Vicuna. The top three best performances regarding BSR and
Average DPR are highlighted. We again exclude Aegis for high false positive

defense Method | BSR | DPR | Average
---|---|---|---
|  | AutoDan | DeepInception | GPTFUZZ | TAP | PAIR | jailbroken | 78jailbreak template | Parameters | GCG |
Aegis | 0.74% (6/805) | 0.00% (0/252) | 0.00% (0/30) | 0.00% (0/325) | 1.51% (7/461) | 2.98% (12/402) | 0.28% (6/2104) | 0.00% (0/13161) | 0.85% (26/3050) | 0.00% (0/124) | 0.62%
llm-guard | 99.13% (798/805) | 3.57% (9/252) | 100.00% (30/30) | 21.23% (69/325) | 96.96% (447/461) | 99.01% (398/402) | 39.87% (839/2104) | 12.37% (1629/13161) | 98.88% (3016/3050) | 99.19% (123/124) | 63.45%
smooth-llm | 89.06% (717/805) | 97.22% (245/252) | 100.00% (30/30) | 77.23% (251/325) | 65.94% (304/461) | 70.89% (285/402) | 74.14% (1560/2104) | 67.65% (8904/13161) | 18.52% (565/3050) | 15.32% (19/124) | 65.21%
Baseline-defense | 75.52% (608/805) | 3.17% (8/252) | 0.00% (0/30) | 1.53% (5/325) | 96.74% (446/461) | 96.51% (388/402) | 62.88% (1323/2104) | 13.19% (1736/13161) | 95.85% (2924/3050) | 4.03% (5/124) | 41.54%
RA-LLM | 75.52% (608/805) | 60.71% (153/252) | 86.66% (26/30) | 53.84% (175/325) | 23.42% (108/461) | 23.38% (94/402) | 56.32% (1185/2104) | 41.77% (5498/13161) | 10.00% (305/3050) | 9.67% (12/124) | 40.64%
Bergeron | 98.13% (790/805) | 48.80% (123/252) | 30.00% (9/30) | 41.53% (135/325) | 32.10% (148/461) | 32.58% (131/402) | 31.13% (655/2104) | 32.01% (4213/13161) | 7.63% (233/3050) | 6.45% (8/124) | 29.13%
ModerationAPI | 99.75% (803/805) | 95.63% (241/252) | 100.00% (30/30) | 78.15% (254/325) | 88.50% (408/461) | 96.51% (388/402) | 87.97% (1851/2104) | 83.23% (10955/13161) | 90.55% (2762/3050) | 88.70% (110/124) | 89.91%

Table 14: This table presents the effectiveness of different defense
strategies against attacks on GPT-3.5-turbo, emphasizing the top three in BSR
and Average DPR. Aegis is omitted due to its high false positive rate. The
baseline, relying on sequence perplexity requiring logits access, is
incompatible with black-box models like GPT-3.5-turbo.

defense Method | BSR | DPR | Average
---|---|---|---
|  | DeepInception | GPTFUZZ | TAP | PAIR | jailbroken | 78jailbreak template | Parameters |
Aegis | 0.00% (0/805) | 0.00% (0/13) | 0.00% (0/305) | 0.00% (0/272) | 0.00% (0/280) | 0.00% (0/1613) | 0.00% (0/5000) | 0.00% (0/794) | 0.62%
llm-guard | 98.88% (796/805) | 100.00% (13/13) | 4.91% (15/305) | 99.63% (271/272) | 99.28% (278/280) | 31.12% (502/1613) | 16.78% (839/5000) | 100.00% (794/794) | 64.53%
smooth-llm | 94.16% (758/805) | 100.00% (13/13) | 53.11% (162/305) | 77.94% (212/272) | 78.92% (221/280) | 59.51% (960/1613) | 44.62% (2231/5000) | 38.41% (305/794) | 64.64%
RA-LLM | 83.35% (671/805) | 100.00% (13/13) | 39.34% (120/305) | 48.52% (132/272) | 54.28% (152/280) | 39.30% (634/1613) | 15.24% (762/5000) | 39.04% (310/794) | 47.96%
Bergeron | 98.63% (794/805) | 30.76% (4/13) | 6.22% (19/305) | 26.10% (71/272) | 24.28% (68/280) | 4.27% (69/1613) | 6.27% (314/5000) | 20.52% (163/794) | 16.92%
ModerationAPI | 99.87% (804/805) | 100.00% (13/13) | 86.988% (265/305) | 91.54% (249/272) | 96.07% (269/280) | 90.08% (1453/1613) | 88.32% (4416/5000) | 95.46% (758/794) | 92.62%

Table 15: Evaluation of benign response quality for Alpaca using Smooth, Rallm, and Bergeron, where higher levels signify improved quality. Model | Smooth | Rallm | Bergeron
---|---|---|---
GPT-3.5-Turbo | 9.13 | 7.89 | 8.01
Vicuna | 2.24 | 0.75 | 4.78
Llama-2 | 6.21 | 5.47 | 5.65
Figure 5: An example of false positive classification of safe response to
unsafe of GPT-4 Figure 6: This graph illustrates how we label response
manually Table 16: This table presents the top five methods that demonstrated
the highest effectiveness for 78 templates and Jailbroken

Attack Methods | Model Type
---|---
| LLaMa | Vicuna | GPT-3.5-Turbo
78 templates | Template | Times | Template | Times | Template | Times
APOPHIS | 96 | BasedBOB | 280 | AIM | 261
BasedBOB | 58 | SIM | 276 | BetterDAN | 246
Hackerman v2 | 57 | KEVIN | 271 | Evil Confidant | 230
Hitchhiker’s Guide | 48 | BetterDAN | 269 | Text Continuation | 219
DAN Jailbreak | 37 | Evil Chad 2.1 | 263 | DeltaGPT | 208
combination_3 | 198 | AIM | 230 | AIM | 268
wikipedia_with_title | 118 | prefix_injection | 190 | evil_system_prompt | 254
Jailbroken | combination_2 | 118 | combination_3 | 183 | evil_confidant | 228
AIM | 105 | wikipedia_with_title | 161 | combination_3 | 146
prefix_injection | 91 | dev_mode_v2 | 131 | wikipedia_with_title | 110