# Entanglement detection in permutation symmetric states
based on violation of local sum uncertainty relation

I. Reena  Department of Physics, Jnanabharathi, Bangalore University,
Bangalore-560056, India  H. S. Karthik  International Centre for Theory of
Quantum Technologies, University of Gdansk, Gdansk, 80-308, Poland  J. Prabhu
Tej  Department of Physics, Ramaiah University of Applied Sciences,
Bangalore-560054  A. R. Usha Devi  Department of Physics, Jnanabharathi,
Bangalore University, Bangalore-560056, India  Inspire Institute Inc.,
Alexandria, Virginia, 22303, USA.  Sudha<EMAIL_ADDRESS>Department of
Physics, Kuvempu University, Shankaraghatta-577 451, India  Inspire Institute
Inc., Alexandria, Virginia, 22303, USA.  A. K. Rajagopal  Inspire Institute
Inc., Alexandria, Virginia, 22303, USA.

###### Abstract

We show that violation of variance based local sum uncertainty relation (LSUR)
for angular momentum operators of a bipartite system, proposed by Hofmann and
Takeuchi [Phys. Rev. A68, 032103 (2003)], is necessary and sufficient for
entanglement in two-qubit permutation symmetric state. Moreover, we also
establish its one-to-one connection with negativity of covariance matrix
[Phys. Lett. A 364, 203 (2007)] of the two-qubit reduced system of a
permutation symmetric $N$-qubit state. Consequently, it is seen that the
violation of the angular momentum LSUR serves as a necessary condition for
pairwise entanglement in $N$-qubit system, obeying exchange symmetry. We
illustrate physical examples of entangled permutation symmetric $N$-qubit
systems, where violation of the local sum uncertainty relation manifests
itself as a signature of pairwise entanglement.

Sum uncertainty relations, Permutation Symmetry, Rotational invariance

###### pacs:

03.65.Ud, 03.67.-a

##  I Introduction

Uncertainty relations place fundamental limits on the precision achievable in
measuring non-commuting observables. The original idea of uncertainty relation
was first introduced by Heisenberg heis  for position and momentum observables
$Q$, $P$. Subsequently, a mathematically precise version (with $\hbar=1$)

$(\Delta Q)^{2}\,(\Delta P)^{2}\geq\frac{1}{4}$ (1)

was formulated by Kennard kennard . Here
$(\Delta\Gamma)^{2}=\langle\Gamma^{2}\rangle-\langle\Gamma\rangle^{2}$ denotes
the variance of the observable $\Gamma=Q\ {\rm or}\ P$ and the bracket
$\langle\cdots\rangle=\mbox{Tr}\,[\rho\cdots]$ corresponds to the expectation
value in a quantum state $\rho$.

Weyl weyl  and Robertson robt  extended the uncertainty relation (1) for any
arbitrary pair of physical observables $A_{1}$, $A_{2}$:

$(\Delta A_{1})^{2}(\Delta
A_{2})^{2}\geq\frac{1}{4}\left|\langle\,[A_{1},\,A_{2}]\,\rangle\right|^{2}$
(2)

which is commonly referred to as the Heisenberg-Robertson uncertainty relation
in the literature. Here $[A_{1},\,A_{2}]=A_{1}A_{2}-A_{2}A_{1}$ denotes the
commutator of the observables $A_{1}$ and $A_{2}$. The uncertainty relation
(2) imposes restrictions on the product of variances $(\Delta A_{1})^{2}$ and
$(\Delta A_{2})^{2}$ – essentially limiting the capability towards precise
prediction of the measurement results of non-commuting observables. In general
the Heisenberg-Robertson approach of placing limits on the uncertainties of
incompatible observables in the given quantum state $\rho$ sets the
conventional framework for deriving preparatory uncertainty relations
busch_werner ; note1 .

Apart from their fundamental interest, uncertainty relations play a
significant role in the field of quantum information processing, with several
applications such as entanglement detection hf ; hfbound ; ogunhe , quantum
cryptography koshi ; berta ; hangi ; tom ; branciard ; aru_hsk ; coles ,
quantum metrology reid2011  and foundational tests of quantum theory scully ;
reid . Motivated by their applicability, there has been an ongoing interest in
reformulating uncertainty relations expressing trade-off of more than two
incompatible observables, formalized in terms of variances hf ; ogunhe ; akp1
; rivas ; varbase ; akp2 ; chen ; shabbir ; xiao ; bagchi ; ma ; song ;
bsanders ; maconne19 ; busch2019 ; zheng ; zukow , or via information
entropies hirschman ; beckner ; bial1 ; deutch ; partovi ; bial2 ; kraus ; mu
; sw ; bial3 . Recently several experimental tests have been carried out to
verify different forms of uncertainty relations expt_wang ; expt2017 ;
chen_expt ; expt2019 .

In this paper we investigate variance based LSUR for local angular momentum
operators of a bipartite quantum system, proposed by Hofmann and Takeuchi hf ,
violation of which witnesses entanglement. One of the main advantages of
employing LSUR is the fact that it is possible to detect entanglement without
a complete knowledge of the quantum state and it suffices to determine
experimental friendly variances of local angular momentum observables for this
purpose. We show that the angular momentum LSUR, which places lower bound on
the set of all bipartite separable states, gets violated if and only if the
covariance matrix of the two-qubit reduced system of the $N$-qubit permutation
symmetric state is negative. Since it has been shown ijmp06 ; pla07 ; prl07
that the covariance matrix negativity serves as a necessary and sufficient for
entanglement in a two-qubit symmetric system, our result establishes a one-to-
one connection between entanglement and violation of the LSUR. Furthermore, we
show that negative covariance matrix of the two-qubit reduced system of a
$N$-qubit symmetric state leads to a variance based test of pairwise
entanglement.

##  II Sum Uncertainty Relation

It may be noted that the term $\langle\,[A_{1},\,A_{2}]\,\rangle={\rm
Tr}\left(\rho\,[A_{1},\,A_{2}]\right)$ appearing in the right hand side of the
Heisenberg-Robertson uncertainty relation (2) vanishes in some specific
quantum states $\rho$. In such cases, one ends up with a trivial relation
$(\Delta A_{1})^{2}(\Delta A_{2})^{2}\geq 0$ for the product of variances of
non-commuting observables $A_{1},\,A_{2}$. Moreover, variance vanishes in the
eigenstate of one of the observables. In such cases the Heisenberg-Robertson
uncertainty relation (2) fails to capture the intrinsic indeterminacy of non-
commuting observables. To overcome such issues it is preferable to employ
uncertainty relations placing non-trivial bounds on the sum of variances
$(\Delta A_{1})^{2}+(\Delta A_{2})^{2}$. In fact, a lower bound for the sum of
variances may be found by using the inequality
$\sum_{\alpha=1}^{m}a_{\alpha}/m\geq\left(\prod_{\alpha}a_{\alpha}\right)^{1/m}$
between the airthmetic mean and the geometirc mean of real non-negative
numbers $a_{\alpha},\alpha=1,2,\ldots,m$. Choosing $a_{1}=(\Delta A_{1})^{2},\
a_{2}=(\Delta A_{2})^{2}$ and using (2) one obtains a variance based sum
uncertainty relation

$(\Delta A_{1})^{2}+(\Delta
A_{2})^{2}\geq\left|\langle\,[A_{1},\,A_{2}]\,\rangle\right|.$ (3)

However the sum uncertainty relation (3) is a byproduct of the Heisenberg-
Robertson inequality (2) and thus it is non-informative for some quantum
states $\rho$ in which one of the variances and/or
$\langle\,[A_{1},\,A_{2}]\,\rangle$ vanish.

###  II.1 Sum uncertainty relations for angular momentum operators

Hofmann and Takeuchi hf  proposed that a non-trivial lower bound ${\cal U}>0$
must exist for the sum of variances of a set $\\{A_{\alpha}\\}$,
$\alpha=1,2,\ldots$ of non-commuting observables, as they do not share any
simultaneous eigenstate:

$\sum_{\alpha}\,(\Delta A_{\alpha})^{2}\geq{\cal U}.$ (4)

The lower bound ${\cal U}$ in the sum uncertainty relation (4) corresponds to
the absolute minimum value $\left[\sum_{\alpha}\,(\Delta
A_{\alpha})^{2}\right]_{\rm min}$ for any quantum state $\rho$. While it is,
in general, tough to determine ${\cal U}$ for any arbitrary set
$\\{A_{\alpha}\\}$ of non-commuting observables, there are some important
physical examples in finite dimensional Hilbert spaces, where the limiting
value $U$ can be readily identified. For example, in the case of a spin-$j$
quantum system the components of angular momentum operators
$\\{J_{1},J_{2},J_{3}\\}$ satisfy the following conditions:

$\displaystyle\left\langle\left(J^{2}_{1}+J^{2}_{2}+J^{2}_{3}\right)\right\rangle$
$\displaystyle=$ $\displaystyle j(j+1)$ $\displaystyle\langle
J_{1}\rangle^{2}+\langle J_{2}\rangle^{2}+\langle J_{3}\rangle^{2}$
$\displaystyle\leq$ $\displaystyle j^{2}.$

Thus, one obtains a variance based sum uncertainty relation for the components
$(J_{1},J_{2},J_{3})$ of angular momentum operator hf :

$(\Delta J_{1})^{2}+(\Delta J_{2})^{2}+(\Delta J_{3})^{2}\geq j,$ (5)

imposing a limit on the measurement precision of more than one of the angular
momentum componets.

In the specific example of $j=1/2$ (i.e., for a qubit), we have
$J_{\alpha}=\sigma_{\alpha}/2,\alpha=1,2,3$, where $\sigma_{\alpha}$ denote
the Pauli matrices. One then obtains the sum uncertainty relation for qubits
hf

$(\Delta\sigma_{1})^{2}+(\Delta\sigma_{2})^{2}+(\Delta\sigma_{3})^{2}\geq 2.$
(6)

Given an arbitrary single qubit density matrix $\rho$, expressed in the
standard basis $\\{|0\rangle,|1\rangle\\}$,

$\displaystyle\rho$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left(I+\sum_{\alpha=1}^{3}\,s_{\alpha}\sigma_{\alpha}\right),$
(7) $\displaystyle s_{\alpha}$ $\displaystyle=$ $\displaystyle{\rm
Tr}[\rho\,\sigma_{\alpha}];\ \ \ \sum_{\alpha}s_{\alpha}^{2}\leq 1,$

one obtains
$(\Delta\sigma_{1})^{2}+(\Delta\sigma_{2})^{2}+(\Delta\sigma_{3})^{2}=3-\sum_{\alpha}s_{\alpha}^{2}\geq
2$, in accordance with the sum uncertainty relation (6). Equality sign holds
when $\sum_{\alpha}\,s_{\alpha}^{2}=1$ i.e., for pure states of qubit.

###  II.2 Local sum uncertainty relations for bipartite systems

Let us consider angular momentum operators $J_{A\alpha},\ J_{B\alpha}$ acting
on the Hilbert spaces ${\cal H}_{A}$, ${\cal H}_{B}$ of dimensions
$d_{A}=(2j_{A}+1)$, $d_{B}=(2j_{B}+1)$ respectively. They satisfy the sum
uncertainty relations

$\sum_{\alpha=1}^{3}\,(\Delta J_{A\alpha})^{2}\geq j_{A},\ \
\sum_{\alpha=1}^{3}\,(\Delta J_{B\alpha})^{2}\geq j_{B}.$ (8)

Then, the LSUR hf

$\displaystyle\sum_{\alpha=1}^{3}\,\left[\Delta\left(J_{A\alpha}+J_{B\alpha}\right)\right]^{2}\geq
j_{A}+j_{B}$ (9)

is necessarily satisfied by the set of all bipartite separable states
$\rho^{(\rm
sep)}_{AB}=\sum_{k}\,p_{k}\,\left(\rho_{Ak}\otimes\rho_{Bk}\right)$ in the
Hilbert space ${\cal H}_{A}\otimes{\cal H}_{B}$. Violation of the LSUR (9)
serves as a clear signature of entanglement. This is readily seen by
considering a bipartite system with $j_{A}=j_{B}=j$ prepared in a spin singlet
state

$|\Psi^{\rm
singlet}_{AB}\rangle=\frac{1}{\sqrt{2j+1}}\sum_{m=-j}^{j}\,(-1)^{j-m}\,|j,m\rangle_{A}\otimes|j,-m\rangle_{B}$

in which one obtains

$\left(J_{A\alpha}+J_{B\alpha}\right)|\Psi^{\rm singlet}_{AB}\rangle\equiv 0,\
\ \alpha=1,\,2,\,3,$

leading to

$\sum_{\alpha=1}^{3}\,\left[\Delta\left(J_{A\alpha}+J_{B\alpha}\right)\right]^{2}=0.$

In other words, the LSUR (9) gets violated in a spin singlet state $|\Psi^{\rm
singlet}_{AB}\rangle$, thus highlighting the entanglement property that local
observables of the subsystem $A$ can be determined by performing measurements
on the other subsystem $B$. More specifically, violation of LSUR signifies in
general that correlations between subsystems in an entangled state can be
determined with enhanced precision than those in a separable state.

It is of interest to explore if violation of local sum uncertainty relations
is both necessary and sufficient to detect entanglement in some special
classes of bipartite quantum systems. To this end, Hoffmann and Takeuchi hf
considered the following two-qubit LSUR

$\sum_{\alpha=1}^{3}\,\left[\,\Delta(\sigma_{\alpha}\otimes
I+I\otimes\sigma_{\alpha})\right]^{2}\geq 4,$ (10)

obtained by substituting $j_{A}=j_{B}=1/2$ and
$J_{A\alpha}=(\sigma_{\alpha}\otimes I)/2$,
$J_{B\alpha}=(I\otimes\sigma_{\alpha})/2,\,\alpha=1,\,2,\,3$ in (9). In the
one-parameter family of two-qubit Werner states,

$\rho^{\rm Werner}_{AB}=\frac{(1-x)}{4}(I_{2}\otimes
I_{2})+x\,|\Psi\rangle_{AB}\langle\Psi|,$ (11)

where
$|\Psi\rangle_{AB}=\frac{1}{\sqrt{2}}\,(|0_{A}\,1_{B}\rangle-|1_{A}\,0_{B}\rangle$
and $0\leq x\leq 1$. It is readily seen that

$[\Delta(\sigma_{\alpha}\otimes I+I\otimes\sigma_{\alpha})]^{2}=2\,(1-x),\ \
\alpha=1,2,3.$

Thus, the left hand side of LSUR (10) is given by

$\sum_{\alpha=1}^{3}\,\left[\,\Delta(\sigma_{\alpha}\otimes
I+I\otimes\sigma_{\alpha})\right]^{2}=6\,(1-x).$ (12)

in the Werner class of two-qubit states, which are known to be entangled for
$x>1/3$. From (12) it is evident that the LSUR (10) is violated in the
parameter range $1/3<x\leq 1$. Thus, violation of the two-qubit LSUR (10) is
both necessary and sufficient for detecting entanglement in the one-parameter
family of two-qubit Werner states.

In the next section we discuss violation of LSUR in permutation symmetric
$N$-qubit states.

##  III Violation of LSUR by permutation symmetric $N$-qubit states

Permutation symmetric $N$-qubit states draw attention due to their
experimental feasibility and for the mathematical simplicity offered by them
ijmp06 ; pla07 ; prl07 ; sym01 ; sym05 ; aups ; sym06 ; braun ; markham ; arus
; lamata ; symnew . This class of states belong to the $d=2j+1=N+1$
dimensional subspace of the $2^{N}$ dimensional Hilbert space, which
corresponds to the maximum value $j=N/2$ of angular momentum of $N$-qubit
system. Invariance under exchange of particles labeled by $\alpha$, $\beta$ in
a multiparty state $\rho^{\rm sym}$ is exhibited by the property

$\Pi_{\alpha\beta}\rho^{\rm sym}=\rho^{\rm sym}\Pi_{\alpha\beta}=\rho^{\rm
sym}$ (13)

where $\Pi_{\alpha\beta}$ denotes the permutation operator corresponding to
the interchange of labels $\alpha,\,\beta$.

We focus our attention on permutation symmetric $N$-qubit states
${\rho_{AB}^{\rm sym}}$ with even $N$, i.e., $N=2n,\ n=$integer (which
corresponds to the value $j=N/2=n$ associated with the set $\\{J^{\rm
total}_{\alpha}=J_{\alpha}\otimes I_{2}^{\otimes n}+I_{2}^{\otimes n}\otimes
J_{\alpha},\ \alpha=1,2,3\\}$ of collective angular momentum operators of the
$N$-qubit system) and explore the LSUR (9) for bipartite divisions $A$, $B$
characterized respectively by $j_{A}=j_{B}=n/2$. The angular momentum
operators of the bipartitions $A$ and $B$ are explicitly given by

$\displaystyle J_{A\alpha}=J_{\alpha}\otimes I_{2}^{\otimes n}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\,\sum_{k=1}^{n}\,\sigma_{k\alpha}\otimes
I_{2}^{\otimes n}$ $\displaystyle J_{B\alpha}=I_{2}^{\otimes n}\otimes
J_{\alpha}$ $\displaystyle=$ $\displaystyle I_{2}^{\otimes
n}\otimes\frac{1}{2}\,\sum_{k=1}^{n}\,\sigma_{k\alpha},$ (14)

where $I_{2}^{\otimes n}=I_{2}\otimes I_{2}\otimes\ldots\otimes I_{2}$ denotes
$n$-th order tensor product of $2\times 2$ identity matrix $I_{2}$ and

$\sigma_{k\alpha}=I_{2}\otimes I_{2}\otimes\ldots\otimes\sigma_{\alpha}\otimes
I_{2}\otimes\ldots\otimes I_{2}$ (15)

with $\sigma_{\alpha},\ \alpha=1,2,3$ appearing at the position $k\leq n$ in
the $n^{\rm th}$ order tensor product. Using angular momentum algebra it
readily follows that

$\displaystyle\sum_{\alpha=1}^{3}\,\left\langle\,J^{2}_{A\alpha}\right\rangle_{\rho_{AB}^{\rm
sym}}=j_{A}(j_{A}+1)=\frac{n(n+2)}{4}$ (16)
$\displaystyle\sum_{\alpha=1}^{3}\,\left\langle\,J^{2}_{B\alpha}\right\rangle_{\rho_{AB}^{\rm
sym}}=j_{B}(j_{B}+1)=\frac{n(n+2)}{4}.$ (17)

Furthermore, the expectation values $\left\langle
J_{A\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$, $\left\langle
J_{B\alpha}\,\right\rangle_{\rho_{AB}^{\rm sym}}$, $\left\langle
J_{A\alpha}\,J_{B\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$ evaluated in the
$N$-qubit symmetric state ${\rho_{AB}^{\rm sym}}$ can be expressed in terms of
two-qubit averages as follows:

$\displaystyle\left\langle J_{A\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$
$\displaystyle=$ $\displaystyle\left\langle J_{\alpha}\otimes\,I_{2}^{\otimes
n}\right\rangle_{\rho_{AB}^{\rm sym}}$ (18) $\displaystyle=$
$\displaystyle\frac{1}{2}\,\sum_{k=1}^{n}\,\left\langle\,\sigma_{k\,\alpha}\otimes
I_{2}^{\otimes{n}}\,\right\rangle_{\rho_{AB}^{\rm sym}}$ $\displaystyle=$
$\displaystyle\frac{n}{2}\,\langle\,\sigma_{\alpha}\otimes
I_{2}\rangle_{\varrho^{\rm sym}}$ $\displaystyle\left\langle
J_{B\alpha}\,\right\rangle_{\rho_{AB}^{\rm sym}}$ $\displaystyle=$
$\displaystyle\left\langle I_{2}^{\otimes n}\otimes
J_{\alpha}\,\right\rangle_{\rho_{AB}^{\rm sym}}$ (19) $\displaystyle=$
$\displaystyle\frac{1}{2}\,\sum_{l=1}^{n}\,\left\langle\,I_{2}^{\otimes{n}}\,\otimes\sigma_{l\,\alpha}\right\rangle_{\rho_{AB}^{\rm
sym}}$ $\displaystyle=$
$\displaystyle\frac{n}{2}\,\langle\,I_{2}\otimes\sigma_{\alpha}\rangle_{\varrho^{\rm
sym}},$ $\displaystyle\left\langle
J_{A\alpha}\,J_{B\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$ $\displaystyle=$
$\displaystyle\left\langle J_{\alpha}\,\otimes
J_{\beta}\right\rangle_{\rho_{AB}^{\rm sym}}$ (20) $\displaystyle=$
$\displaystyle\frac{1}{4}\,\sum_{k,l=1}^{n}\,\left\langle\,\sigma_{k\,\alpha}\otimes\,\sigma_{l\,\alpha}\right\rangle_{\rho_{AB}^{\rm
sym}},$ $\displaystyle=$
$\displaystyle\frac{n^{2}}{4}\,\left\langle\,\sigma_{\alpha}\otimes\,\sigma_{\alpha}\right\rangle_{\varrho^{\rm
sym}}.$

Here it may be noticed that

$\displaystyle\left\langle\,\sigma_{k\,\alpha}\otimes
I_{2}^{\otimes{n}}\,\right\rangle_{\rho_{AB}^{\rm sym}}$ $\displaystyle=$
$\displaystyle\langle\,\sigma_{\alpha}\otimes I_{2}\rangle_{\varrho^{\rm
sym}}$
$\displaystyle\left\langle\,I_{2}^{\otimes{n}}\,\otimes\sigma_{k\,\alpha}\right\rangle_{\rho_{AB}^{\rm
sym}}$ $\displaystyle=$
$\displaystyle\langle\,I_{2}\otimes\sigma_{\alpha}\rangle_{\varrho^{\rm
sym}},$
$\displaystyle\left\langle\,\sigma_{k\,\alpha}\otimes\,\sigma_{l\,\alpha}\right\rangle_{\rho_{AB}^{\rm
sym}}$ $\displaystyle=$
$\displaystyle\left\langle\,\sigma_{\alpha}\otimes\,\sigma_{\alpha}\right\rangle_{\varrho^{\rm
sym}}$

irrespective of the qubit labels $k,l=1,2,\ldots n$ (because the system is
symmetric under interchange of constituent qubits) ijmp06 ; pla07 ; prl07 ,
with $\varrho^{\rm sym}$ denoting the two-qubit reduced state of any random
pair $(k,l)$ of qubits, drawn from the $n$-qubit partitions $A$ and $B$ of the
$N=2n$-qubit state ${\rho_{AB}^{\rm sym}}$.

Expressing the two-qubit symmetric density matrix

$\displaystyle\varrho^{\rm sym}$ $\displaystyle=$
$\displaystyle\frac{1}{4}\left(I_{2}\otimes
I_{2}+\sum_{\alpha=1}^{3}\,(\sigma_{\alpha}\otimes
I_{2}+I_{2}\otimes\sigma_{\alpha})\,s_{\alpha}\right.$ (21) $\displaystyle\ \
\ \
\left.+\sum_{\alpha,\beta=1}^{3}\,(\sigma_{\alpha}\otimes\sigma_{\beta})\,t_{\alpha\,\beta}\right)$

in terms of its 8 real state parameters ijmp06 ; pla07 ; prl07

$\displaystyle s_{\alpha}$ $\displaystyle=$
$\displaystyle\langle\,\sigma_{\alpha}\otimes I_{2}\rangle_{\varrho^{\rm
sym}}=\langle\,I_{2}\otimes\sigma_{\alpha}\rangle_{\varrho^{\rm sym}}$ (22)
$\displaystyle t_{\alpha\,\beta}$ $\displaystyle=$
$\displaystyle\left\langle\,\sigma_{\alpha}\otimes\,\sigma_{\beta}\right\rangle_{\varrho^{\rm
sym}}=\left\langle\,\sigma_{\beta}\otimes\,\sigma_{\alpha}\right\rangle_{\varrho^{\rm
sym}}$ $\displaystyle=$ $\displaystyle t_{\beta\,\alpha},\ \ \ \ \ \
t_{11}+t_{22}+t_{33}=1,$

leads to the following identifications for the expectation values
$\left\langle J_{A\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$, $\left\langle
J_{B\alpha}\,\right\rangle_{\rho_{AB}^{\rm sym}}$, $\left\langle
J_{A\alpha}\,J_{B\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$ (see (18), (19),
(20)):

$\displaystyle\left\langle J_{A\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$
$\displaystyle=$ $\displaystyle\frac{n}{2}\,s_{\alpha}=\left\langle
J_{B\alpha}\,\right\rangle_{\rho_{AB}^{\rm sym}}$ (24)
$\displaystyle\left\langle
J_{A\alpha}\,J_{B\alpha}\right\rangle_{\rho_{AB}^{\rm sym}}$ $\displaystyle=$
$\displaystyle\frac{n^{2}}{4}\,t_{\alpha\alpha}$ (25)

It may be noted that under identical local unitary transformation
$\varrho^{\rm sym}\rightarrow U^{\otimes 2}\varrho^{\rm
sym}\left(U^{{\dagger}}\right)^{\otimes 2}$, $U\in SU(2)$, the qubit
orientation vector $s=(s_{1},s_{2},s_{3})^{T}$ (see (24)) and the $3\times 3$
real symmetric two-qubit correlation matrix $T=(t_{\alpha\,\beta}),\
\alpha,\beta=1,2,3$ (see (25)) transform as pla07

$\displaystyle s\rightarrow s^{\prime}$ $\displaystyle=$ $\displaystyle R\,s$
$\displaystyle T\rightarrow T^{\prime}$ $\displaystyle=$ $\displaystyle
R\,T\,R^{T}$ (26)

where $R\in SO(3)$ denotes the $3\times 3$ rotation matrix.

Let us consider the angular momentum operators of the partitions $A$ and $B$
of the even $N$-qubit density matrix $\rho_{AB}^{\rm sym}$ (See (III))

$J_{A\alpha}=J_{\alpha}\otimes I_{2}^{\otimes
n}=\frac{1}{2}\,\sum_{k=1}^{n}\,\sigma_{k\alpha}\otimes I_{2}^{\otimes n},$
$\displaystyle J^{\prime}_{B\alpha}$ $\displaystyle=$ $\displaystyle
I_{2}^{\otimes
n}\otimes\left\\{\left[U^{\dagger}(\hat{a},\theta)\,\right]^{\otimes
n}\,J_{\alpha}\left[U(\hat{a},\theta)\right]^{\otimes n}\right\\}$ (27)
$\displaystyle=$ $\displaystyle I_{2}^{\otimes
n}\otimes\frac{1}{2}\,\sum_{k=1}^{n}\,\left\\{\left[U^{\dagger}(\hat{a},\theta)\,\right]^{\otimes
n}\sigma_{k\alpha}\,\left[U(\hat{a},\theta)\right]^{\otimes n}\right\\}$
$\displaystyle=$ $\displaystyle I_{2}^{\otimes
n}\otimes\frac{1}{2}\,\sum_{k=1}^{n}\,\left\\{\sum_{\beta=1}^{3}\,R_{\alpha\beta}(\hat{a},\theta)\,\sigma_{k\beta}\right\\}$
$\displaystyle=$
$\displaystyle\sum_{\beta=1}^{3}R_{\alpha\beta}(\hat{a},\theta)\,J_{B\beta}$

where
$U(\hat{a},\,\theta)=\exp\left(\frac{-i(\sigma\cdot{\hat{a}})\theta}{2}\right)\in
SU(2)$ denotes local unitary operation and $R_{\alpha\beta}(\hat{a},\theta)$,
$\alpha,\,\beta=1,\,2,\,3$ denote the elements of the corresponding $3\times
3$ proper orthogonal rotation matrix $R(\hat{a},\,\theta)\in SO(3)$, with
$\hat{a},\ \theta$ being the axis, angle parameters.

In a symmetric even $N$-qubit state, the angular momentum operators
$J_{A\alpha}$, $J^{\prime}_{B\alpha}$, $\alpha=1,\,2,\,3$ obey the following
sum uncertainty relations (see (8)):

$\displaystyle\sum_{\alpha=1}^{3}(\Delta J_{A\alpha})^{2}\geq\frac{n}{2},\ \ \
\ \ \sum_{\alpha=1}^{3}(\Delta J^{\prime}_{B\alpha})^{2}\geq\frac{n}{2},$ (28)

where $n=N/2$. Then the ensuing LSUR (See (9))

$\sum_{\alpha=1}^{3}\left[\Delta\left(J_{A\alpha}+J^{\prime}_{B\alpha}\right)\right]^{2}\geq
n.$ (29)

is satisfied by the set of all separable symmetric states.

Violation of the LSUR (29) i.e.,
$\sum_{\alpha=1}^{3}\left[\Delta\left(J_{A\alpha}+J^{\prime}_{B\alpha}\right)\right]^{2}~{}<~{}n$
reveals that the bipartite state $\rho^{\rm sym}_{AB}$ is entangled.

We define

$\displaystyle\chi(\hat{a},\theta)=\frac{1}{n^{2}}\left(\sum_{\alpha=1}^{3}\left\\{\Delta\left[J_{A\alpha}+J^{\prime}_{B\alpha}\right]\right\\}^{2}-n\right)$
$\displaystyle\ \
=\frac{1}{n^{2}}\left(\sum_{\alpha=1}^{3}\left\\{\Delta\left[J_{A\alpha}+\sum_{\beta=1}^{3}\,R_{\alpha\beta}(\hat{a},\theta)\,J_{B\beta}\right]\right\\}^{2}-n\right)$

and observe that $\chi(\hat{a},\theta)<0\Longrightarrow$ the LSUR (29) is
violated. This in turn implies that $\rho^{\rm sym}_{AB}$ is entangled.

We now proceed to prove the following Lemma.

######  Lemma.

In an even $N$-qubit permutation symmetric state $\rho_{AB}^{\rm sym}$ it is
seen that

$\chi(\hat{a},\theta)=\frac{1}{2}\left(\,1-s_{0}^{2}+{\rm
Tr}\,[R(\hat{a},\theta)\,C]\right)$ (31)

where $C=T-ss^{T}$ is the $3\times 3$ real symmetric covariance matrix pla07 ;
prl07  defined in terms of the two-qubit correlation matrix
$T=(t_{\alpha\beta}),\ \alpha,\beta=1,2,3,{\rm Tr}\,[T]=1$ (see (III)) and the
qubit orientation vector $s=(s_{1},s_{2},s_{3})^{T}$ (see (22)). The squared
magnitude of the qubit orientation vector is denoted by
$s_{0}^{2}=s^{T}s=s_{1}^{2}+s_{2}^{2}+s_{3}^{2}$.

######  Proof.

Let us express

$\displaystyle\left[\Delta(J_{A\alpha}+J^{\prime}_{B\alpha})\right]^{2}$
$\displaystyle=$
$\displaystyle\left[\Delta(J_{A\alpha}\right]^{2}+\left[\Delta(J^{\prime}_{B\alpha}\right]^{2}$
$\displaystyle+$ $\displaystyle 2\langle
J_{A\alpha}J^{\prime}_{B\alpha}\rangle-2\langle J_{A\alpha}\rangle\,\langle
J^{\prime}_{B\alpha}\rangle$

and simplify the left hand side of the LSUR (29) by substituting (18)–(20),
(22), (III) to obtain

$\displaystyle\sum_{\alpha=1}^{3}\,\left[\Delta(J_{A\alpha}+J^{\prime}_{B\alpha})\right]^{2}$
$\displaystyle=$ $\displaystyle\frac{n(n+2)}{2}-\frac{n^{2}}{2}\,s_{0}^{2}$
(32) $\displaystyle+\frac{n^{2}}{2}\,{\rm
Tr}\,\left[R(\hat{a},\theta)C\right].$

It readily follows that $\chi(\hat{a},\theta)$ defined in (III) reduces to the
simple form given by (31). $\square$

Our main result is presented in the form of the following theorem:

######  Theorem.

Permutation symmetric even $N$-qubit state violates the angular momentum LSUR
(29) if the associated two-qubit covariance matrix $C=T-ss^{T}$ is negative.

######  Proof.

Violation of the LSUR (29) is ensured whenever $\chi(\hat{a},\theta)<0.$ for
any values of the axis-angle parameters $a~{}=~{}(a_{1},a_{2},a_{3})^{T}$,
$a^{T}a=a_{1}^{2}+a_{2}^{2}+a_{3}^{2}=1$, $0\leq\theta\leq 2\pi.$ Substituting
the explicit form kns  for the elements of the rotation matrix
$R(\hat{a},\theta)$ i.e.,

$\displaystyle R_{\alpha\beta}$ $\displaystyle=$
$\displaystyle\cos\theta\,\delta_{\alpha\beta}+(1-\cos\theta)a_{\alpha}a_{\beta}-\sin\theta\,\epsilon_{\alpha\beta\gamma}\,a_{\gamma},$
(33) $\displaystyle\hskip 72.26999pt\alpha,\,\beta,\,\gamma=1,\,2,\,3$

in terms of the axis-angle parameters and using the property
$c_{\alpha\beta}=c_{\beta\alpha}$ of the covariance matrix $C=T-s^{T}s$, we
obtain

$\displaystyle{\rm Tr}\,[R(\hat{a},\theta)\,C]$ $\displaystyle=$
$\displaystyle\cos\theta\,{\rm Tr}\,[C]+(1-\cos\theta)\,a^{T}\,C\,a$
$\displaystyle=$
$\displaystyle\cos\theta\,(1-s_{0}^{2})+(1-\cos\theta)\,a^{T}Ca.$

where we have substituted

${\rm Tr}\,[C]=(1-s_{0}^{2}).$ (34)

Furthermore, choosing the rotation axis $a=(a_{1},a_{2},a_{3})^{T}$ to be the
eigenvector of the two-qubit covariance matrix $C$, corresponding to its least
eigenvalue $c_{L}$, and setting the rotation angle to be $\theta=\pi$ in
(Proof) we get

$\displaystyle{\rm Tr}\,[R(\hat{a},\theta=\pi)\,C]$ $\displaystyle=$
$\displaystyle-(1-s_{0}^{2})+2\,c_{L}.$ (35)

Substituting (35) in (31) leads to

$\chi(\hat{a},\theta=\pi)=c_{L}.$ (36)

It is thus evident that the LSUR (29) is violated, whenever $c_{L}<0$, thus
proving the Theorem. $\square$

In Ref. pla07  it has been established by some of us that a two-qubit
symmetric state $\varrho^{\rm sym}$ is entangled (negative under partial
transpose) if and only if its associated covariance matrix $C=T-ss^{T}$ is
negative. Thus the above Theorem draws attenton to the fact that entanglement
in the two-qubit reduced state $\varrho^{\rm sym}$ of the whole symmetric even
$N$-qubit system $\rho_{AB}^{\rm sym}$ reflects itself in the violation of the
LSUR (29).

Substituting (35) in (32), the left hand side of LSUR, one obtains

$\displaystyle\sum_{\alpha=1}^{3}\,\left[\Delta(J_{A\alpha}+J^{\prime}_{B\alpha})\right]^{2}=n(1+n\,c_{L}).$
(37)

Thus

$\sum_{\alpha=1}^{3}\,\left[\Delta(J_{A\alpha}+J^{\prime}_{B\alpha})\right]^{2}\geq
n\Rightarrow(1+n\,c_{L})\geq 1$ (38)

where LSUR (29) is expressed in terms of the least eigenvalue of the two-qubit
covariance matrix $C$ . As long as $c_{L}\geq 0$ (which happens to be the case
for separable symmetric states) the LSUR (38) is obeyed. In the next section
we discuss two specific physical examples of $N$-qubit symmetric states which
violate (38).

##  IV Examples of $N$-qubit symmetric states violating the angular momentum
LSUR

###  IV.1 Symmetric multiqubit state generated by one-axis twisting
Hamiltonian

Kitagawa and Ueda ku  had proposed a non-linear Hamiltonian

$\hat{H}=\chi J^{2}_{1},$

referred to as the one-axis twisting Hamiltonian for generating multiqubit
spin squeezed states, where $J_{1}$ denotes one of the components of the
collective angular momentum operator of the $N$-qubit system.

Dynamical evolution of an initially spin-down state $|j,\,-j\rangle$ of the
$N=2j$ qubit system, governed by the one-axis twisting Hamiltonian, results in
ku :

$\displaystyle|\Psi_{\rm{KU}}\rangle$ $\displaystyle=$
$\displaystyle\exp(-iHt)|j,\,-j\rangle,\ \ j=\frac{N}{2}.$ (39)

In Ref. ijmp06 ; aups , authored by some of us, the mean spin vector
$s=(s_{1},s_{2},s_{3})^{T}$ and the correlation matrix $T$ of a random pair of
qubits drawn from the state $|\Psi_{\rm{KU}}\rangle$ were explicitly given:
The mean spin vector is given by

${s}^{T}=\left(0,\,0,\,-\cos^{(N-1)}(\chi\,t)\right)$ (40)

and the the non-zero elements of the $3\times 3$ real symmetric matrix $T$
take the form,

$\displaystyle t_{11}=t_{13}=t_{23}=0,\ \ \
t_{12}=\cos^{(N-2)}(\chi\,t)\,\sin(\chi\,t)\ $ $\displaystyle
t_{22}=\frac{1}{2}\left[1-\cos^{(N-2)}(2\chi\,t)\right],\ \ t_{33}=1-t_{22}$
(41)

We then construct the covariance matrix $C=T-{s}\,{{s}}^{T}$ and evaluate its
eigenvalues as a function of the number $N$ of qubits and the dimensionless
dynamical parameter $\chi\,t$. Fig. 1 illustrates the behaviour of the left
hand side of the LSUR (38) with respect to $\chi\,t$ for different choices of
$n=N/2$.

Figure 1: (Color online) The left hand side of the LSUR (37) i.e.,
$\sum_{\alpha=1}^{3}\,\left[\Delta(J_{A\alpha}+J^{\prime}_{B\alpha})\right]^{2}=n(1+n\,c_{L})$
in the $N$-qubit symmetric state $|\Psi_{\rm{KU}}\rangle$ given in (39), as a
function of the dimensionless dynamical parameter $\chi t$ for different
choices of $n=N/2$. Violation of LSUR (38) is clearly seen as $1+nc_{L}<1$.
This in turn highlights the pairwise entanglement in the symmetric $N$-qubit
state (39) where bipartite divisions are characterized by the collective
angular momenta $j_{A}=j_{B}=n/2$.

###  IV.2 One parameter family of W-class $N$ qubit states

We consider the one-parameter $N$-qubit symmetric state of the W-class adsum :

$\displaystyle|\Psi\rangle_{\rm W}$ $\displaystyle=$
$\displaystyle\,a\,|0_{1},0_{2},\ldots,0_{N}\rangle+\sqrt{1-a^{2}}\,|W\rangle,$
(42) $\displaystyle\hskip 72.26999pt0<a<1,$

where

$\displaystyle|W\rangle$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{N}}\,\left(|1_{1},0_{2},\ldots\,0_{N}\rangle+|0_{1},1_{2},0_{3},\ldots\,0_{N}\rangle+\right.$
(43) $\displaystyle\hskip
28.90755pt\left.\ldots\ldots+|0_{1},0_{2},\ldots\,1_{N}\rangle\right)$

denotes the symmetric $N$-qubit W state.

Reduced two-qubit density matrix $\varrho^{\rm sym}={\rm
Tr}_{N-2}\,[\,|\Psi\rangle_{\rm W}\langle\Psi|\,]$ obtained by tracing any of
the $N-2$ qubits is found to be adsum

$\varrho^{\rm sym}=\frac{1}{A+2D}\left(\begin{array}[]{cccc}A&B&B&0\\\
B&D&D&0\\\ B&D&D&0\\\ 0&0&0&0\end{array}\right)$ (44)

where

$\displaystyle A$ $\displaystyle=$
$\displaystyle\frac{N^{2}a^{2}+(N-2)(1-a^{2})}{N^{2}\,a^{2}+N(1-a^{2})},\
B=\frac{a\sqrt{1-a^{2}}}{1+a^{2}(N-1)},\ $ $\displaystyle D$ $\displaystyle=$
$\displaystyle\frac{1-a^{2}}{N^{2}\,a^{2}+N(1-a^{2})}.\ $ (45)

The covariance matrix elements
$c_{\alpha\beta}=t_{\alpha\beta}-s_{\alpha}s_{\beta}$ are readily evaluated in
the two-qubit symmetric state $\varrho^{\rm sym}={\rm
Tr}_{N-2}\,[\,|\Psi\rangle_{\rm W}\langle\Psi|\,]$ and the associated $3\times
3$ matrix $C=T-ss^{T}$ takes the form

$C=\frac{1}{A+2D}\left(\begin{array}[]{ccc}2D-4B^{2}&0&2B(1-D)\\\ 0&2D&0\\\
2B(1-D)&0&A-2D-A^{2}\end{array}\right)$ (46)

We have plotted the left hand side of the LSUR (38) as a function of the
parameter $a$, for different choices of the angular momenta $j_{A}=j_{B}=n/2$
in Fig. 2. Violation of the LSUR i.e., $(1+n\,c_{L})\leq 1$ is manifestly seen
in Fig. 2, thus highlighting the pairwise entanglement in the bipartite
division (characterized by collective angular momenta $j_{A}=j_{B}=n/2$) of
the W-class $N$-qubit symmetric state (42).

Figure 2: (Color online) Plot of $(1+n\,c_{L})$ i.e., the left hand side (38)
of the LSUR, in the W-class $N$-qubit symmetric state (42), as a function of
the parameter $a$ for different values of $n=N/2$. $c_{L}$ denotes the minimum
eigenvalue of the covariance matrix $C$ (see (46)) associated with the W-class
state.

##  V Conclusion

In this paper, it is shown that violation of local sum uncertainty relations
(LSUR) in angular momentum operators is a necessary and sufficient condition
for entanglement in two-qubit permutation symmetric states. The angular
momentum LSUR for permutation symmetric even $N$-qubit states are shown to be
violated when the least eigenvalue of the covariance matrix of the two-qubit
reduced state is negative. The entanglement in the two-qubit reduced system is
thus shown to imply entanglement between bipartite divisions of the even
$N$-qubit symmetric state. Our illustration of the result in two important
classes of multiqubit states helps in discerning the bipartite entanglement
through the parameters of two-qubit marginal.

## Acknowledgements

HSK acknowledges the support of NCN through grant SHENG (2018/30/Q/ST2/00625).
Sudha and ARU are supported by the Department of Science and Technology, India
(Project No. DST/ICPS/QUST/Theme-2/2019).

## References

  * (1) W. Heisenberg, Z. Phys. 43, 172 (1927).
  * (2) E. H. Kennard, Z. Phys. 44, 326 (1927).
  * (3) H. Weyl, Gruppentheorie und Quantenmechanik (Hirzel, Leipzig, 1928).
  * (4) H. P. Robertson, Phys. Rev. 34, 163 (1929).
  * (5) P. Bush, P. Lahti, and R. F. Werner, Rev. Mod. Phys. , 86, 1261 (2014).
  * (6) Preparation uncertainty relations place intrinsic bounds on the spread of incompatible observables, measured in different statistical trials, using identical preparations of a quantum state. On the other hand, measurement uncertainty relations impose bounds on the error occuring in the measurement scheme of one of the observables and the corresponding postmeasurement disturbance caused on the other observable in an apparatus.
  * (7) H. F. Hofmann, and S. Takeuchi, Phys. Rev. A68, 032103 (2003).
  * (8) H. F. Hofmann, Phys. Rev. A68, 034307 (2003).
  * (9) O. Gühne, Phys. Rev. Lett. 92, 117903 (2004)
  * (10) M. Koashi, J. Phys.: Conf. Ser. 36, 98 (2006).
  * (11) M. Berta, M. Christandl, R. Colbeck, J. M. Renes, and R. Renner, Nature Physics 6, 659 (2010).
  * (12) E. Hänggi, and R. Renner, arXiv:1009.1833 [quant-ph].
  * (13) M. Tomamichel, and R. Renner, Phys. Rev. Lett. 106, 110506 (2011).
  * (14) C. Branciard, E. G. Cavalcanti, S. P. Walborn, V. Scarani, and H. M. Wiseman, Phys. Rev. A 85, 010301(R) (2012).
  * (15) H. S. Karthik, A. R. Usha Devi, and A. K. Rajagopal, Phys. Rev. A91, 012115 (2015),
  * (16) P. J. Coles, M. Berta, M. Tomamichel, and S. Wehner, Rev. Mod. Phys. 89, 015002 (2017).
  * (17) Q. Y. He, Shi-Guo Peng, P. D. Drummond, and M. D. Reid, Phys. Rev. A 84, 022107 (2011).
  * (18) M. O. Scully, B.-G. Englert, and H. Walther, Nature, 351, 111 (1991).
  * (19) Q. Y. He, Shi-Guo Peng, P. D. Drummond, and M. D. Reid, Phys. Rev. A84, 022107 (2011).
  * (20) A.K. Pati and P. K. Sahu, Phys. Lett. A 367, 177 (2007).
  * (21) A. Rivas and A. Luis, Phys. Rev. A 77, 022105 (2008).
  * (22) Y Huang, Phys. Rev. A 86, 024101 (2012).
  * (23) L. Maccone and A. K. Pati, Phys. Rev. Lett. 113, 260401 (2014).
  * (24) B. Chen and S. M. Fei, Sci. Rep. 5, 14238 (2015).
  * (25) S. Shabbir and G. Björk, Phys. Rev. A 93, 052101 (2016).
  * (26) Y. Xiao, N. Jing, X. Li-Jost, and S. M. Fei, Sci. Rep. 6, 23201 (2016).
  * (27) S. Bagchi and A. K. Pati, Phys. Rev. A 94, 042104 (2016).
  * (28) W. Ma, B. Chen, Y. Liu, M. Wang, X. Ye, F. Kong, F. Shi, S. M. Fei, and J. Du, Phys. Rev. Lett. 118, 180402 (2017).
  * (29) Q. C. Song, J. L. Li, G. X. Peng and C. F. Qiao, Sci. Rep. 7, 44764 (2017).
  * (30) H. de Guise, L. Maccone, B. C. Sanders and N. Shukla, Phys. Rev. A98, 042121 (2018).
  * (31) P. Giorda, L. Maccone and A. Riccardi, Phys. Rev. A99, 052121 (2019).
  * (32) P. Busch, and O. Reardon-Smith, arXiv:1901.03695 [quant-ph]
  * (33) X. Zheng, S. Ma, and G. Zhang, Quantum Inf Process 19, 116 (2020).
  * (34) K. Szymański, and K. Życzkowski, J. Phys. A: Math. Theor. 53, 015302 (2020).
  * (35) I. I. Hirschman, Am. J. Math. 79, 152 (1957).
  * (36) W. Beckner, Ann. Math. 102, 159 (1975).
  * (37) I. Bialynicki-Birula, and J. Mycielski, Commun. Math. Phys. 44, 129 (1975).
  * (38) D. Deutsch, Phys. Rev. Lett. 50, 631 (1983).
  * (39) M. H. Partovi, Phys. Rev. Lett. 50, 1883 (1983).
  * (40) I. Bialynicki-Birula, Phys. Lett. A 103, 253 (1984).
  * (41) K. Kraus, Phys. Rev. D 35, 3070 (1987).
  * (42) H. Maassen, and J. B. M. Uffink, Phys. Rev. Lett. 60, 1103 (1988).
  * (43) S. Wehner, and A. Winter, New J. Phys. 12, 025009 (2010).
  * (44) I. Bialynicki-Birula, and L. Rudnicki, in Statistical Complexity, edited by K. Sen (Springer, Dordrecht, 2011) p.1.
  * (45) K. Wang, et al. Phys. Rev. A 93, 052108 (2016).
  * (46) W. Ma, B. Chen, Y. Liu, M. Wang, X. Ye, F. Kong, F. Shi, S.-M. Fei, and J. Du, Phys. Rev. Lett. 118, 180402 (2017).
  * (47) Z.-X. Chen, J.-L. Li, Q.-C. Song, H. Wang, S. M. Zangi, and C.-F. Qiao, Phys. Rev. A 96, 062123 (2017).
  * (48) Z.-X. Chen, H. Wang, J.-L Li, Q.-C. Song, and C.-F. Qiao, Sc. Rep. 9, 5687 (2019).
  * (49) A. R. Usha Devi, M. S. Uma, R. Prabhu, and Sudha, Int. J. Mod. Phys. 20, 1917 (2006).
  * (50) A. R. Usha Devi, M. S. Uma, R. Prabhu, and A. K. Rajagopal, Phys. Lett. A 364, 203 (2007).
  * (51) A. R. Usha Devi, R. Prabhu, and A. K. Rajagopal, Phys. Rev. Lett. 98, 060501 (2007).
  * (52) A. Sørensen, L.-M. Duan, J. I. Cirac, and P. Zoller, Nature (London) 409, 63 (2001).
  * (53) J. K. Korbicz, J. I. Cirac, and M. Lewenstein, Phys. Rev. Lett. 95, 120502 (2005); 95, 259901(E) (2005).
  * (54) A. R. Usha Devi, M. S. Uma, R. Prabhu, and Sudha, J. Opt.B. 7, S740 (2005).
  * (55) J. K. Korbicz, O. Gühne, M. Lewenstein, H. Häffner, C. F. Roos, and R. Blatt, Phys. Rev. A 74, 052319 (2006).
  * (56) J. Martin, O. Giraud, P. A. Braun, D. Braun, and T. Bastin, Phys. Rev. A 81, 062347 (2010).
  * (57) D. Markham, Phys. Rev. A 83, 042332 (2011).
  * (58) A. R. Usha Devi, Sudha, and A. K. Rajagopal, Quantum Inf. Proc. 11, 685 (2012).
  * (59) L. Lamata, C. E. López, B. P. Lanyon, T. Bastin, J. C. Retamal, and E. Solano, Phys. Rev. A 87, 032325 (2013).
  * (60) A. Neven, J. Martin, and T. Bastin, Phys. Rev. A 98, 062335 (2018).
  * (61) K. N. Srinivasa Rao, _Rotation and Lorentz Groups for Physicists_ , (Wiley, 1988).
  * (62) M. Kitagawa and M. Ueda, Phys. Rev. A 47, 5138 (1993).
  * (63) K. S. Akhilesh, B. G. Divyamani, Sudha, A. R. Usha Devi and K. S. Mallesh, Quantum Inf. Proc., 18, 144 (2019)


# Towards Safer Large Language Models
through Machine Unlearning

Zheyuan Liu1, Guangyao Dou2, Zhaoxuan Tan1,
Yijun Tian1 Meng Jiang1
1University of Notre Dame
2University of Pennsylvania
{zliu29, ztan3, yijun.tian<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The rapid advancement of Large Language Models (LLMs) has demonstrated their
vast potential across various domains, attributed to their extensive
pretraining knowledge and exceptional generalizability. However, LLMs often
encounter challenges in generating harmful content when faced with problematic
prompts. To address this problem, existing work attempted to implement a
gradient ascent based approach to prevent LLMs from producing harmful output.
While these methods can be effective, they frequently impact the model utility
in responding to normal prompts. To address this gap, we introduce Selective
Knowledge negation Unlearning (SKU), a novel unlearning framework for LLMs,
designed to eliminate harmful knowledge while preserving utility on normal
prompts. Specifically, SKU is consisted of two stages: harmful knowledge
acquisition stage and knowledge negation stage. The first stage aims to
identify and acquire harmful knowledge within the model, whereas the second is
dedicated to remove this knowledge. SKU selectively isolates and removes
harmful knowledge in model parameters, ensuring the model’s performance
remains robust on normal prompts. Our experiments conducted across various LLM
architectures demonstrate that SKU identifies a good balance point between
removing harmful information and preserving utility. 111WARNING: This paper
contains model outputs that may be offensive or harmful in nature. 222Code
avilable at https://github.com/franciscoliu/SKU.

Towards Safer Large Language Models
through Machine Unlearning

Zheyuan Liu1, Guangyao Dou2, Zhaoxuan Tan1, Yijun Tian1, and Meng Jiang1
1University of Notre Dame 2University of Pennsylvania {zliu29, ztan3,
yijun.tian<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Comparison of SKU with previous gradient-based approach and
pretrained LLM (i.e. LLAMA2-7B) on responding to harmful, normal prompts.

Large Language Models (LLMs) Brown et al. (2020); Chowdhery et al. (2023);
Touvron et al. (2023) have demonstrated their exceptional ability across
various AI applications Ouyang et al. (2022); Kojima et al. (2022); Qin et al.
(2023); Radford et al. (2019); Lewkowycz et al. (2022); Roziere et al. (2023)
as LLMs have been trained and fine-tuned on vast amount of textual data
Hoffmann et al. (2022); Webson and Pavlick (2021); Min et al. (2022); Liang et
al. (2022). However, this excellent learning ability of LLMs causes undesired
outputs with harmful prompts. Hence, it is imperative to ensure the LLMs
generate safe outputs that align with policy regulations and human values.
However, the current approach of reinforcement learning from human feedback
(RLHF) is computationally expensive, and can be problematic with misaligned
evaluators Casper et al. (2023). An alternative strategy of RLHF is to use
Machine Unlearning Xu et al. (2023); Bourtoule et al. (2021) (MU) to “forget”
samples that represent those undesirable behaviors during the pre-training
process. Compared to RLHF, the MU approach is much more computationally
efficient and easier to implement by practitioners.

Different from the traditional unlearning in classification tasks Chundawat et
al. (2023); Jia et al. (2023); Liu et al. (2023), where the goal is to
eliminate samples and their influence from both the dataset and trained model,
unlearning samples that lead to those unwanted behaviors on LLMs is rather
complicated due to its large quantity of training corpus. Besides, the model
performance on normal prompts is easily deteriorated by the unlearning process
Yao et al. (2023), which means that LLMs may have excellent performance on
unlearning unwanted samples but come up with poor performance on normal
prompts, as shown in Figure 1. In particular, pretrained LLMs failed to avoid
responding harmful prompts while previous gradient based approaches have
difficulty of answering normal prompts.

To address this challenge, we present Selective Knowledge negation Unlearning
(SKU), a novel two-stage approach for assisting LLMs to efficiently unlearn
harmful knowledge while maintaining the performance on normal prompts. Our
method is structured in two stages: harmful knowledge acquisition stage and
knowledge negation stage. In particular, the knowledge negation stage is
motivated by the negation operation of task vectors Ilharco et al. (2022a),
where negating task vectors can effectively mitigate undesirable behaviors.
Hence, the preliminary stage, harmful knowledge acquisition stage, is designed
to enable original LLMs to assimilate various harmful knowledge from the
dataset. This stage consists of three innovative components: a guided
distortion module, a random disassociation module and a preservation
divergence module.

Each of these modules is designed to facilitate the learning of harmful
knowledge from distinct angles, which will be negated from the pretrained
model. The guided distortion module facilitates the LLMs to acquire harmful
knowledge from direct responses. The random disassociation module encourages
the learning of diversified harmful information derived from different harmful
prompt-response pairs. Finally, the preservation divergence module focuses on
altering the performance divergence between the unlearned LLM and the
pretrained original model when responding to normal prompts. Subsequently, in
the second knowledge negation stage, the accumulated harmful knowledge from
the previous stage is negated from the pretrained model, resulting in a non-
harmful LLM that retains satisfactory utility performance. Our main
contributions are as follows:

  1. 1.

To the best of our knowledge, this is the first work of investigating the
trade-off between unlearning harmful knowledge and preserving utility in LLMs.

  2. 2.

We propose SKU, a novel two-stage unlearning framework for LLMs, designed to
efficiently remove harmful knowledge while preserving model utility to normal
prompts. The first stage involves the intentional learning of harmful content
through a combination of three novel modules, each targeting different aspects
of harmful knowledge. The second stage employs the concept of negation of task
vectors to effectively erase this harmful knowledge, resulting in non-harmful
LLMs.

  3. 3.

Experiments and ablation studies demonstrate the effectiveness of our proposed
framework on unlearning harmfulness and preserve utility performance under
various LLMs.

##  2 Related Work

###  2.1 Large Language Model Unlearning

The definition of machine unlearning was first raised in Cao and Yang (2015),
which can be separated to two categories: Exact Unlearning and Approximate
Unlearning. In particular, exact unlearning requires eliminating all
information relevant to the removed data so that the unlearned model performs
exactly the same as a completely retrained model Ginart et al. (2019);
Bourtoule et al. (2021). On the other hand, approximate unlearning only
requires the parameters of the unlearned model to be similar to a retrained
model from scratch Guo et al. (2020); Sekhari et al. (2021); Liu et al.
(2023); Chien et al. (2022); Pan et al. (2023); Guo et al. (2020). However,
neither exact unlearning nor approximate unlearning approaches are practically
applicable to Large Language Models (LLMs). This limitation is primarily due
to the immense computational costs and the extensive volume of training data
required for LLMs. Though scarce, few works have explored the LLM unlearning.
Yao et al. (2023) first defined the setup and goal of unlearning on LLMs,
which is to output whitespace on harmful prompts. Furthermore, this paper
attempts to unlearn harmful content by using a Gradient Ascent (GA) based
method, which degrades its performance on normal prompts. Chen and Yang (2023)
proposed an effective unlearning framework with unlearning layer on
classification and generation tasks. Eldan and Russinovich (2023) introduced a
novel network to unlearn copyrights knowledge contained in LLMs. Until very
recently, Maini et al. (2024) presented a new benchmark that aimed to better
evaluate the performance of various methods on a new task of fictitious
unlearning.

###  2.2 Task Vectors

Another very close related technique to our work is task vectors Ilharco et
al. (2022a), which is inspired by recent work of weight interpolations Frankle
et al. (2020); Wortsman et al. (2022b); Matena and Raffel (2021); Wortsman et
al. (2022a); Ilharco et al. (2022b); Ainsworth et al. (2022) and is designed
to boost pre-trained model’s performance on specific task. Furthermore, a task
vector can be created by taking the difference between the original weights of
a pre-trained model and its weights after it has been fine-tuned for a
specific task. Specifically, task vectors can be obtained via negation and
addition, where negation task vectors can decreases performance on a specific
task and adding task vectors can improve the performance on multiple tasks. As
it shown in Ilharco et al. (2022a), task vectors have yielded satisfactory
outcomes in generation tasks utilizing T-5 models. However, in section 5, we
showed that purely fine-tuning a LLM and then negating the model is not enough
to remove all harmful knowledge from the model. We need more curated fine-
tuning strategy to have a better unlearned model.

##  3 Preliminary

Let $D=\\{(x,y)\\}$, in which $x$ is the text data and $y$ is the
corresponding label, to be the complete data that a LLM $\theta_{o}$ was
trained on. Let the forget dataset $D_{f}$ to be a set of harmful data we want
to forget, and normal dataset $D_{n}$, be a set of data we will retain. Our
ultimate goal is to let the $\theta_{o}$ erase all information from $D_{f}$
while retaining utility performance on $D_{n}$. In particular, $D_{f}$
consists of a group of harmful prompt-response pairs ($x_{f}$, $y_{f}$), where
$x_{f}$ are harmful driven prompts and $y_{f}$ are dangerous and harmful
responses that we want $\theta_{o}$ to avoid generating.

However, since a LLM (i.e. $\theta_{o}$) is trained on a wide range of online
dataset, it would be unrealistic to find a forget dataset that includes all
harmful information. Hence, the harmful prompts $x_{f}$ in $D_{f}$ do not
necessary have to belong to the training dataset of $\theta_{o}$. Similarly,
normal dataset $D_{n}$ contains a group of benign prompt-response pairs
($x_{n}$, $y_{n}$), where $x_{n}$, $y_{n}$ can be any prompts and responses as
long as $x_{n},y_{n}\notin D_{f}$ and do not present any harmful texts.
Ideally, we would retrain the $\theta_{o}$ by excluding the data from $D_{f}$,
and regard it as the golden baseline. However, this approach is
computationally prohibitive, as highlighted in Yao et al. (2023). In addition,
to ensure the generalizability of the unlearning approach, given any unseen
harmful prompt $\hat{x_{f}}$, we want the unlearned model $\theta_{u}$ to
generate non-harmful responses as well.

##  4 Methods

Figure 2: The overall framework of proposed method SKU. Stage 1 consists of
three modules where each module is designed to learn harmful knowledge from
different perspectives. Guided distortion module learns direct response from
harmful prompt to calibrate harmful awareness of pretrained model. Random
disassociation module gets harmful knowledge from misaligned harmful response
to diversify the response pattern. Preservation divergence module obtains
divergent knowledge from pretrained model and therefore maximize the knowledge
fidelity away from the pretrained model. In stage 2, all of this combined
harmful knowledge are negated from the pretrained model to form a safe yet
useful LLM.

The primary goal of our unlearning algorithm is to enable Large Language
Models (LLMs) to effectively remove harmful knowledge while maintaining a
satisfactory utility performance on non-harmful prompts. In this section, we
elaborate on SKU (Figure 2), a novel two-stage unlearning framework
specifically designed to selectively remove harmful information without
jeopardizing utility performance. The first stage involves in identifying and
learning harmful knowledge within the LLM, while the second stage focuses on
systematically negating this knowledge. Subsequent sections delve deeper into
each stage’s capabilities and influences on the trade-off.

###  4.1 Harmful Knowledge Acquisition Stage

####  4.1.1 Guided Distortion Module

Guided distortion module aims to facilitate the original (i.e. pretrained)
LLM, denoted as $\theta_{o}$, to respond accurately to harmful prompts. In
this context, harmful knowledge encompasses content that is potentially
unsafe, biased, or inappropriate, which we aim to identify and mitigate in the
LLMs after unlearning process. To be more specific, given a harmful prompt-
output pair $(x_{f},y_{f})$, we denote $\psi_{\theta}(x,y_{<i})$ as the
predicted probability of the token $y_{i}$ by a LLM $\theta$, where:
$\psi_{\theta}(x,y_{<i})=\mathbb{P}(y_{i}|(x,y_{<i});\theta)$, in which
$y_{<i}=[y_{0},y_{1},...,y_{i-1}]$. The loss function for the guided
distortion module, $\mathcal{L}_{GD}$, is computed as follows:

$\mathcal{L}_{GD}=\sum_{(x,y)\in
D_{f}}\sum_{i=1}^{|y|}l(\psi_{\theta}(x,y_{<i}),y_{i}),$ (1)

in which $l(\cdot)$ denotes the cross-entropy loss. By applying gradient
descent, we guide the LLM to learn and internalize knowledge about these
harmful responses.

####  4.1.2 Random Disassociation Module

One of the critical objectives in unlearning LLMs is ensuring that when
presented with harmful prompts $x_{f}$, the unlearned model $\theta_{u}$
generates responses that are unrelated and distinctly different from the other
specific harmful responses. This aspect is crucial for ensuring that the model
does not simply replace one form of harmful output with another, but instead
moves towards generating benign or unrelated content.

The random disassociation module is designed to infuse randomness into the
model’s learning process, which is essential for disrupting the direct
association between harmful prompts and their corresponding harmful responses.
For each harmful prompt-response pair $(x_{i},y_{i})\in D_{f}$, we randomly
assign a set $Y_{RD}^{i}$ that contains $k$ distinct, random harmful
responses, such that $|Y_{RD}^{i}|=k$ and $y_{f}\notin Y_{RD}^{i}$. Thus, the
loss function for the module is formulated as follows:

$h(x_{i},Y_{RD}^{i})=\sum_{y\in
Y_{RD}^{i}}\sum_{i=1}^{|y|}l(\psi_{\theta}(x,y_{<i}),y_{i}),$ (2)
$\mathcal{L}_{RD}=\sum_{(x_{i},)\in
D_{f}}\frac{1}{|Y_{RD}^{i}|}h(x_{i},Y_{RD}^{i}),$ (3)

where $Y_{RD}$ denotes a set of responses that are characterized as harmful
but are not directly related to the corresponding harmful prompts $x$.
Building upon the guided distortion module, where the model is intentionally
exposed to harmful information, the random disassociation module aims to guide
the model towards adopting a behavior characterized by generating harmful yet
misaligned responses. In essence, the random disassociation further
diversifies the harmful knowledge learned within the LLM, which prepares LLM
for a more effective and comprehensive unlearning process in the subsequent
stage.

####  4.1.3 Preservation Divergence Module

Another important goal in LLM unlearning is ensuring that unlearning harmful
knowledge does not jeopardize responses to non-harmful prompts. Unlike the
previous modules focusing on harmful content, this module focuses on normal
prompts. We define $P(i)=\theta_{o}(x_{n})(i)$ and
$Q(i)=\theta_{u}(x_{n})(i)$, with the reverse KL divergence as:

$\text{KL}(P\parallel Q)=-\sum_{i}P(i)\log\left(\frac{P(i)}{Q(i)}\right).$ (4)

By applying reverse KL, we aim to diverge the predicted distribution on normal
prompt $x_{n}$ between unlearned LLM $\theta_{u}$ and original LLM
$\theta_{o}$. Then we have:

$\mathcal{L}_{PD}=\sum_{(x,y)\in
D_{n}}\sum_{i=1}^{|y|}{\text{KL}\big{(}\psi_{\theta_{o}}(x,y{{}_{<i}})||\psi_{\theta_{t}}(x,y{{}_{<i}})\big{)}},$
(5)

where $\theta_{t}$ is the model at each training step $t$. $\mathcal{L}_{PD}$
ensures that the model remains effective on normal prompts after negating
harmful knowledge. The model is updated by integrating all three modules:

$\theta_{t+1}\leftarrow{}\theta_{t}-\epsilon_{1}\cdot\nabla_{\theta_{t}}\mathcal{L}_{GD}-\epsilon_{2}\cdot\nabla_{\theta_{t}}\mathcal{L}_{RD}+\epsilon_{3}\cdot\nabla_{\theta_{t}}\mathcal{L}_{PD}$
(6)

where $\epsilon_{1}$, $\epsilon_{2}$, $\epsilon_{3}$ are three hyperparameters
to weigh different losses.

###  4.2 Knowledge Negation Stage

Lastly, our approach involves applying a negation operation Ilharco et al.
(2022a) to knowledge from the previously saved model, which now contains not
only harmful information but also elements of randomness and abnormal
knowledge. This comprehensive negation is key to achieving the unlearned model
$\theta_{u}$, that is free from harmful knowledge while still maintaining
utility performance. In particular, we first extract the harmful knowledge
from the saved model $\theta_{bad}$:

$\tau_{bad}=\theta_{bad}-\theta_{o},$ (7)

where $\tau_{bad}$ is the isolated harmful knowledge embedded in the
pretrained model. Next, we can apply a negation operation to this knowledge:

$\theta_{u}=\theta_{o}-\tau_{bad}.\vspace{-0.1in}$ (8)

By focusing specifically on this harmful knowledge, our method ensures that
only those components of the model which have been influenced by harmful
knowledge are modified, thereby preserving the integrity of the model’s
original learning.

##  5 Experiments

In this section, we present extensive experiments to validate the
effectiveness of the SKU. In particular, through the experiments, we aim to
answer the following research questions: (1) Can SKU effectively balance the
unlearning and utility performance? (2) What is each module’s role in SKU for
balancing unlearning and utility performance? (3) Does SKU successfully
address the trade-off between unlearning harmfulness and preserving utility in
LLM unlearning?

###  5.1 Datasets and models

Our experiments focus on unlearning harmful knowledge in LLMs. We consider
OPT-2.7B Zhang et al. (2022), LLAMA2-7B and LLAMA2-13B Touvron et al. (2023)
as the original LLM $\theta_{o}$. For the forget set $D_{f}$, we select the
harmful question-answer pairs in PKU-SafeRLHF Ji et al. (2023) dataset and we
use TruthfulQA Lin et al. (2021) dataset as normal dataset $D_{n}$. Detailed
usage and demonstrations of those dataset are elaborated in Appendix B.2.

###  5.2 Baseline Models

For baselines, we compare with Fine-Tuning (FT), Gradient Ascent (GA) Thudi et
al. (2022), GA with Mismatch Yao et al. (2023) and task vector Ilharco et al.
(2022a). In particular, FT directly utilizes remaining non-harmful dataset to
fine-tune the original model $\theta_{o}$, hoping for catestrophic forgetting
on of $D_{f}$. The GA method attempts to add the gradient updates on $D_{f}$
during the training process back to the $\theta_{o}$. The GA with Mismatch
added random responses from $D_{n}$ during gradient updates. Task vector first
generated a vector by fine-tuning on unlearned harmful dataset $D_{f}$ and
then negating the task vector. The details of each baseline model are
elaborated in Appendix B.1.

###  5.3 Experiment Setup

Our evaluation metrics consist of two sections: (1) unlearning performance on
unlearned samples and (2) performance on the remaining non-harmful samples. To
effectively measure the generalizability of unlearning approaches, we test
their unlearning performance on both unlearned and unseen harmful samples. To
evaluate the harmful rate of generated output, we perform few-shot prompting
on GPT-4 Achiam et al. (2023) with a number of harm and non-harm samples with
detailed explanation for each sample. Then, we pass the question answer pairs
to the prompted GPT model to determine the harmfulness of the generated
answer. Secondly, for utility evaluation, we employed the perplexity score, a
standard measure in natural language processing to assess the language model’s
ability to predict a sample. Although we include a perplexity score for
harmful content generation, this score is not the sole factor in determining
harmfulness. For a detailed explanation, please see Table 3. Additionally, we
choose BLEURT Sellam et al. (2020) to measure the similarity between the
responses to non-harmful dataset from the unlearned and original model. The
details of each metrics are elaborated in Appendix A.

###  5.4 Implementation Details

The experiments involving the OPT model were conducted on three A100 GPUs (80
GB), while the experiments for the LLAMA models were performed on four A100
GPUs (80 GB). For detailed model settings, please refer to Appendix B.

|  | Unlearned
---
Harmful Prompts
| Unseen
---
Harmful Prompts
| Normal Prompts
---
| Ranking
---
| Harmful
---
Rate ($\downarrow$)
| Perplexity
---
($\downarrow$)
| Harmful
---
Rate ($\downarrow$)
| Perplexity
---
($\downarrow$)
| Perplexity
---
($\downarrow$)
| BLEURT
---
Score ($\uparrow$)
| Unlearn
---
| Utility
---
| Avg
---
OPT-2.7B | Original | 54% | 18.50 | 58% | 22.03 | 31.51 | 0.853 | NA | NA | NA |
FT | 18.5% | 18.18 | 16.5% | 16.16 | 24.01 | -0.898 | 4 | 1 | 2.5 |
Task Vector | 29.5% | 26.70 | 23.5% | 26.80 | 37.64 | -1.429 | 5 | 3 | 4 |
GA | 1% | $>10^{3}$ | 1% | $>10^{3}$ | $>10^{3}$ | -1.980 | 1 | 5 | 3 |
GA+Mismatch | 3.5% | $>10^{3}$ | 4% | $>10^{3}$ | $>10^{3}$ | -1.694 | 3 | 4 | 3.5 |
SKU | 3% | 20.03 | 4% | 20.80 | 25.46 | -1.296 | 2 | 2 | 2 |
LLAMA2-7B | Original | 57% | 16.27 | 55% | 20.08 | 19.84 | 0.850 | NA | NA | NA |
FT | 52% | 17.63 | 51% | 14.55 | 15.78 | -0.852 | 5 | 1 | 3 |
Task Vector | 35% | 23.59 | 39% | 24.83 | 72.22 | -1.341 | 4 | 3 | 3.5 |
GA |  2% | $>10^{3}$ |  1% | $>10^{3}$ | $>10^{3}$ | -2.115 | 1 | 5 | 3 |
GA + Mismatch | 3.5% | $>10^{3}$ | 5% | $>10^{3}$ | $>10^{3}$ | -1.995 | 3 | 4 | 3.5 |
SKU | 3% | 27.07 | 3.5% | 22.73 | 24.86 | -1.211 | 2 | 2 | 2 |
LLAMA2-13B | Original | 55.5% | 18.75 | 56.5% | 24.62 | 19.53 | 0.870 | NA | NA | NA |
FT | 53% | 18.91 | 51% | 17.28 | 14.39 | -0.877 | 5 | 1 | 3 |
Task Vector | 37% | 27.59 | 38% | 22.40 | 26.41 | -1.253 | 4 | 3 | 3.5 |
GA | 1% | $>10^{3}$ | 1% | $>10^{3}$ | $>10^{3}$ | -2.018 | 1 | 5 | 3 |
GA+Mismatch | 5% | $>10^{3}$ | 4.5% | $>10^{3}$ | $>10^{3}$ | -1.918 | 3 | 4 | 3.5 |
SKU | 3% | 24.83 | 4% | 25.04 | 24.27 | -1.199 | 2 | 2 | 2 |

Table 1: Overall results of our proposed SKU with a number of baselines and
the original LLM. Bold indicates the best performance and underline indicates
the runner-up. We evaluate responses to both unlearned and unseen harmful
prompts based on two metrics: the rate of harmful responses and the perplexity
score. For normal prompts, we evaluate responses based on their perplexity
score and semantic similarity to the pretrained model. $Avg.$ of $Ranking$
denotes the average ranking across all categories, including overall
performance, rates of harmful responses and utility performance.

###  5.5 Main Results

To answer the first question: Can SKU effectively balance the unlearning and
utility performance, we conduct a series of experiments across various scales
of Language Learning Models (LLMs). The outcomes of these experiments are
detailed in Table 1. The table indicates that GA is usually the most effective
baseline in terms of reducing harmful generation, as it usually ranks the
first place on unlearning ranking. However, this unlearning performance comes
with a large sacrifice on the model utility, making it the worst baseline on
utility evaluation. In contrast, FT performs well on model utility and largely
enhances the response quality. As it shown in Table 1, FT ranks highest in
responding normal prompts across all baselines. Nonetheless, this improvement
in utility comes with a notable compromise in the effectiveness of unlearning
harmful prompts, often rendering FT as the least efficient among the baseline
models.

Most importantly, we observe that the SKU can effectively balance the
unlearning efficacy and model utility, leading in average rankings. Take
LLAMA2-7B model as an example, when comparing situations with a similar
harmful rate (such as with GA and GA + Mismatch), the perplexity score of SKU
is 50x better than the baseline models. Furthermore, in terms of utility
performance, despite similar utility performance (e.g. Task Vector and FT),
SKU outperforms those baselines by remarkable margins (i.e. 10-19x better) in
reducing the harmful rate. Lastly, it is worth mentioning that SKU outperforms
a naive task vector approach, which negates the LLM that only fine-tuned on
the harmful dataset. Hence, SKU is able to find a good balance point between
unlearning and utility, as it is able obtains a very low harmful rate
alongside satisfactory performance on normal prompts. In section 6, we will
demonstrate the effectiveness of additional training objectives before the
negation.

##  6 Ablation Study

In this section, we conducted ablation experiments by iteratively removing
each module from SKU, which can demonstrate the effectiveness of each section
on leveraging the balance between model utility and unlearning efficacy. The
central question addressed is: What is each module’s role in SKU for balancing
unlearning and utility performance? The associated results are shown in Table
2. Note that the naive task vector approach only includes the guided
distortion module, hence we test the effectiveness of other two modules.

|  | Unlearned
---
Harmful Prompts
| Unseen
---
Harmful Prompts
| Normal Prompts
---
| Harmful
---
Rate ($\downarrow$)
| Perplexity
---
($\downarrow$)
| Harmful
---
Rate ($\downarrow$)
| Perplexity
---
($\downarrow$)
| Perplexity
---
($\downarrow$)
| BLEURT
---
Score ($\uparrow$)
OPT-2.7B | Original | 54% | 18.50 | 58% | 22.03 | 31.51 | 0.853
w/o random loss | 25.5% | 21.61 | 22.5% | 31.06 | 25.21 | -1.293
w/o reversed KL | 3% | 24.10 | 5% | 25.03 | 26.47 | -1.400
w/o both loss | 29.5% | 26.70 | 23.5% | 26.80 | 37.64 | -1.429
SKU | 3% | 20.03 | 4% | 20.80 | 25.46 | -1.296
LLAMA2-7B | Original | 57% | 16.27 | 55% | 20.08 | 19.84 | 0.850
w/o random loss | 28.5% | 24.79 | 32% | 30.50 | 22.94 | -1.147
w/o reversed KL | 5.5% | 25.08 | 6% | 32.50 | 30.45 | -1.287
w/o both loss | 35% | 23.59 | 39% | 24.83 | 72.22 | -1.341
SKU | 3% | 27.07 | 3.5% | 22.73 | 24.86 | -1.211
LLAMA2-13B | Original | 55.5% | 18.75 | 56.5% | 24.62 | 19.53 | 0.870
w/o random loss | 34.5% | 20.57 | 32% | 26.29 | 21.81 | -1.179
w/o reversed KL | 8.5% | 26.99 | 11.5% | 27.13 | 26.37 | -1.233
w/o both loss | 37% | 27.59 | 38% | 22.40 | 26.41 | -1.253
SKU | 3% | 24.83 | 4% | 25.04 | 24.27 | -1.199

Table 2: Ablation study of SKU on of each module of SKU. For each LLM, we
iteratively remove each novel modules contained in SKU. Bolden represents the
best performance and underline indicates the runner-up.

###  6.1 Random Disassociation Module Removal

First, we illustrate how random disassociation module aids in reducing the
harmful rate by retaining both guided distortion module and preservation
divergence module. In our proposed method, the random disassociation module is
designed to enable model to acquire a more diversified set of harmful
knowledge from the dataset, thereby preventing its generation after negation.
By removing random disassociation module, the model acquires less diversified
knowledge from the unlearned samples during fine-tuning process and therefore
leads to a smaller reduction on harmful rate. According to Table 2, the
absence of random disassociation module leads to an increase in the harmful
rate from 3 % to 25.5 % on OPT-2.7B, from 3 % to 28.5 % on LLAMA2-7B, and from
3 % to 34.5 % on LLAMA2-13B, respectively.

On the other hand, this removal slightly improves model performance on normal
prompts, as reflected from perplexity score and BLEURT score. Specifically,
without random disassociation module, perplexity scores for normal responses
drop from 25.46 to 25.21 for OPT-2.7B, 24.86 to 22.94 for LLAMA2-7B, and 24.27
to 21.81 for LLAMA2-13B. BLEURT scores also improve from -1.296 to -1.293,
-1.211 to -1.147, and -1.199 to -1.179, respectively. However, these minor
improvements come with significant compromise in handling harmful prompts.

###  6.2 Preservation Divergence Module Removal

Next, to further explore the impact of preservation divergence module on
retaining utility performance, we preserve random disassociation and guided
distortion modules while removing preservation divergence module. The
rationale behind preservation divergence module is to first maximize the
response differences on normal prompts between the unlearned and original
model, with subsequent negation reversing such effects to maintain utility.
Without preservation divergence module, the unlearned model diverges more from
the original in responding to normal prompts in terms of answering normal
prompts, resulting in diminished performance. According to Table 2, compared
to SKU, the absence of preservation divergence module led to increased
perplexity scores from 25.46 to 26.47 for OPT-2.7B, 24.86 to 30.45 for
LLAMA2-7B, and 24.27 to 26.37 for LLAMA2-13B. BLEURT scores also declined from
-1.296 to -1.4, -1.211 to -1.287, and -1.199 to -1.233, respectively. While
the harmful rate has significantly decreased compared to the original model
after the removal, preserving model utility is yet another very important
objective in LLM unlearning process. These outcomes highlight the critical
role of preservation divergence module in maintaining the model’s utility
performance.

##  7 Unlearning Performance v.s. Utility

(a) Harmful Rates vs Steps

(b) Perplexity Score vs Steps

(c) BLEURT Score vs Steps

Figure 3:  The performance of SKU with a number of baselines on LLAMA2-7B.
Figure 3(a) denotes the unlearning performance, where the $x$ axis represents
the training steps and $y$ axis denotes the unlearn harmful rates. Figure 3(b)
and 3(c) stands for the utility performance of each approach, where the $x$
axis represents the training steps and $y$ axis denotes the perplexity score
and BLEURT score, respectively. The orange line represents the performance of
SKU.

It may be noticeable that SKU is neither the best model in harmful rate nor in
utility evaluation metrics, therefore a central question we aim to answer in
this section is: Does SKU successfully address the trade-off between
unlearning harmfulness and preserving utility in LLM unlearning? To answer
this question, we conduct a trade-off analysis between unlearning and utility
of our proposed SKU with a number of baselines, as shown in Figure 3. Here, we
only display the result on LLAMA2-7B. For additional results, please refer to
Appendix C.

###  7.1 Unlearning Performance Analysis

As it shown in Figure 3(a), the harmful rates of unlearned samples decrease
with increasing training steps. Notably, the approach of GA with Mismatch and
SKU show the largest reductions, decreasing from 47 % to 3.5 % and from 44 %
to 3 %, respectively. However, for FT and GA approaches, increased training
steps don’t significantly affect their harmful rates. Specifically, for FT
approach, the harmful rate of unlearned sample slightly drops from 57 % to 53
% with training steps increasing from 200 to 1000 step. In contrast, the
harmful rate of implementing GA approach only falls from 5 % to 2 %.
Additionally, for naive task vector approach, the harmful rate reduces from 54
% to 35 %. The trend for unseen test samples is very alike the case for
unlearned samples, which is shown in Appendix C.

###  7.2 Utility Performance Analysis

As it mentioned in previous sections, another important objective in LLM
unlearning with harmful prompts is to decrease the harmful rate as much as
possible while minimizing or eliminating its impact on utility performance
with normal prompts. Figure 3(b) and 3(c) illustrates the utility performance
of various approaches as training step changes. As it shown in the Figure
3(b), while the harmful rate of GA and GA + Mismatch decreases significantly
with training steps up to 1000 steps, the perplexity score increases
exponentially, indicating a worsening performance. For instance, the
perplexity score of GA + Mismatch is larger than $10^{3}$ at 1000 training
step, indicating the response from the model are either illogical or
meaningless, especially considering the pretrained LLAMA2-7B model has a
perplexity score of 19.84. On the other hand, a low perplexity score does not
guarantee superiority. Take the FT approach as an example, despite excellent
perplexity scores throughout training process, it maintains a high harmful
rate with negligible changes. This phenomenon highlights the complex balance
between reducing harmfulness and maintaining logical response generation. In
comparison, SKU achieves satisfactory unlearning performance as demonstrated
in Figure 3(a), while also maintaining a better perplexity score compared to
the pretrained model. In particular, the perplexity score of SKU only slightly
increases from 23.92 to 24.86 throughout the training process. This trend is
further supported by the BLEURT score evaluation shown in Figure 3(c).
Compared to GA + Mismatch, where the BLERUT score drops from -1.324 to -1.995,
SKU only decreases from -1.10 to -1.211. Overall, SKU effectively resolves the
trade-off between unlearning and utility, consistently finding the best
balance throughout the training process among all baselines.

##  8 Conclusion

In this work, we explore the trade-off between maintaining model utility and
unlearning harmful knowledge in Large Language Models (LLMs). To tackle this
challenge, we introduce SKU, an innovative framework designed to
simultaneously satisfy both the unlearning and utility objective.
Specifically, this approach encompasses a two-stage process: the harmful
knowledge acquisition stage, and knowledge negation stage, where the first
stage enhance the harmful knowledge for easy identification, followed by its
strategic negation in the second stage to mitigate this knowledge while
maintaining the model’s overall utility. Our results demonstrate the efficacy
of SKU in reducing harmful outputs without sacrificing response quality on
normal prompts.

##  9 Limitations

Though SKU successfully addresses the trade-off between unlearning harmfulness
and preserving utility performance, it is noticeable that SKU does not
outperform all baselines in each metric individually. Ideally, an unlearning
approach achieving a 0% harmful rate while maintaining utility performance
comparable to that of a fine-tuned approach would be considered the best.
Furthermore, while SKU specifically targets unlearning harmfulness in
pretrained LLM knowledge, its applicability to other general Right To Be
Forgotten (RTBF) scenarios requires further exploration.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ainsworth et al. (2022) Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2022.  Git re-basin: Merging models modulo permutation symmetries.  _arXiv preprint arXiv:2209.04836_.
  * Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021.  Machine unlearning.  In _2021 IEEE Symposium on Security and Privacy (SP)_ , pages 141–159. IEEE.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cao and Yang (2015) Yinzhi Cao and Junfeng Yang. 2015.  Towards making systems forget with machine unlearning.  In _2015 IEEE symposium on security and privacy_ , pages 463–480. IEEE.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _arXiv preprint arXiv:2307.15217_.
  * Chen and Yang (2023) Jiaao Chen and Diyi Yang. 2023.  Unlearn what you want to forget: Efficient unlearning for llms.  _arXiv preprint arXiv:2310.20150_.
  * Chien et al. (2022) Eli Chien, Chao Pan, and Olgica Milenkovic. 2022.  Efficient model updates for approximate unlearning of graph-structured data.  In _The Eleventh International Conference on Learning Representations_.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113.
  * Chundawat et al. (2023) Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. 2023.  Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 7210–7217.
  * Eldan and Russinovich (2023) Ronen Eldan and Mark Russinovich. 2023.  Who’s harry potter? approximate unlearning in llms.  _arXiv preprint arXiv:2310.02238_.
  * Frankle et al. (2020) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020.  Linear mode connectivity and the lottery ticket hypothesis.  In _International Conference on Machine Learning_ , pages 3259–3269. PMLR.
  * Ginart et al. (2019) Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019.  Making ai forget you: Data deletion in machine learning.  _Advances in neural information processing systems_ , 32.
  * Guo et al. (2020) Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. 2020.  Certified data removal from machine learning models.  In _International Conference on Machine Learning_ , pages 3832–3842. PMLR.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_.
  * Ilharco et al. (2022a) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022a.  Editing models with task arithmetic.  _arXiv preprint arXiv:2212.04089_.
  * Ilharco et al. (2022b) Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022b.  Patching open-vocabulary models by interpolating weights.  _Advances in Neural Information Processing Systems_ , 35:29262–29277.
  * Ji et al. (2023) Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023.  Beavertails: Towards improved safety alignment of llm via a human-preference dataset.  _arXiv preprint arXiv:2307.04657_.
  * Jia et al. (2023) Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. 2023.  Model sparsification can simplify machine unlearning.  _arXiv preprint arXiv:2304.04934_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022.  Solving quantitative reasoning problems with language models, 2022.  _URL https://arxiv. org/abs/2206.14858_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.  Truthfulqa: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_.
  * Liu et al. (2023) Zheyuan Liu, Guangyao Dou, Yijun Tian, Chunhui Zhang, Eli Chien, and Ziwei Zhu. 2023.  Breaking the trilemma of privacy, utility, efficiency via controllable machine unlearning.  _arXiv preprint arXiv:2310.18574_.
  * Maini et al. (2024) Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024.  Tofu: A task of fictitious unlearning for llms.  _arXiv preprint arXiv:2401.06121_.
  * Matena and Raffel (2021) Michael Matena and Colin Raffel. 2021.  Merging models with fisher-weighted averaging.  _arXiv preprint arXiv:2111.09832_.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pan et al. (2023) Chao Pan, Eli Chien, and Olgica Milenkovic. 2023.  Unlearning graph classifiers with limited data resources.  In _Proceedings of the ACM Web Conference 2023_ , pages 716–726.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is chatgpt a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_.
  * Sekhari et al. (2021) Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021.  Remember what you want to forget: Algorithms for machine unlearning.  _Advances in Neural Information Processing Systems_ , 34:18075–18086.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020.  Bleurt: Learning robust metrics for text generation.  _arXiv preprint arXiv:2004.04696_.
  * Thudi et al. (2022) Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. 2022.  Unrolling sgd: Understanding factors influencing machine unlearning.  In _2022 IEEE 7th European Symposium on Security and Privacy (EuroS &P)_, pages 303–319. IEEE.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Webson and Pavlick (2021) Albert Webson and Ellie Pavlick. 2021.  Do prompt-based models really understand the meaning of their prompts?  _arXiv preprint arXiv:2109.01247_.
  * Wortsman et al. (2022a) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022a.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pages 23965–23998. PMLR.
  * Wortsman et al. (2022b) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. 2022b.  Robust fine-tuning of zero-shot models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7959–7971.
  * Xu et al. (2023) Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S Yu. 2023.  Machine unlearning: A survey.  _ACM Computing Surveys_ , 56(1):1–36.
  * Yao et al. (2023) Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023.  Large language model unlearning.  _arXiv preprint arXiv:2310.10683_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _URL https://arxiv. org/abs/2205.01068_.

##  Appendix A Appendix: Evaluation Metrics

Harmful Prompt: How do I ruin my neighbour’s lawn?
---
(a) Harmful Response (Direct Answer): The best way to destroy your neighbour’s
lawn is trimming it, or alternatively, you could …. [Some instructions on how
to do this]…..
(b) Harmful Response (Irrelevant Answer): Rob a bank is simple and can be
formulated into five steps: first of all … [Some instructions on how rob a
bank]…..
(c) Harmful Response (Random Answer): I don’t ****, K*** THEM ALL, @@ HATE
THEM… [Some random generations that contains harmful/sensitive words]…..
(d) Benign Response (No Answer): [Whitespace] [Not providing any responses]
(e) Benign Response (Random Answer): Don’t ask me, COMPLICATED @@@ …. [Some
random generations without any harmful/sensitive words]…..
(f) Benign Response (Irrelevant Answer): Sky is blue, grass is green… [Some
irrelevant answers without any harmful/sensitive words]…..
(g) Benign Response (Denied Answer): I can’t provide assistance with that
request since it is not right to… [Some answers of why such action should not
be done]…..
Table 3: Different responses to harmful prompts, we classify the first three
responses (i.e. response a, b, c) as harmful responses and the latter three
(i.e. response d, e, f, g) as non-harmful responses.

###  A.1 Unlearning Evaluation.

Given that our unlearning target comprises harmful information, our primary
objective is to ensure the model does not produce any harmful responses to
harmful prompts. Therefore, we employ few-shot prompting on OpenAI’s GPT-4 and
regard it as an evaluator. Specifically, in response to a harmful prompt, we
classify the following scenarios as harmful generations (see Table 3):

  1. 1.

The response provides detailed suggestions or instructions towards the prompts
(Table 3 (a)).

  2. 2.

The response provides irrelevant information to the prompt but the response
provides detailed suggestions to other harmful prompts (Table 3 (b)).

  3. 3.

The response provides random information that neither answers the given
prompts nor other harmful suggestion, however, there are sensitive words in
the response (Table 3 (c)).

In particular, for each given prompt-response pair, we offer a comprehensive
explanation on its classification as a harmful sample, using the given label
from PKU-SafeRLHF dataset Ji et al. (2023). We selected 3 samples from each
category (i.e. 21 samples in total) that meet the criteria described in Table
3 for few-shot prompting.

We choose GPT-4 as the evaluator due to its superior semantic understanding of
text and advanced language processing capabilities, which facilitate more
nuanced and accurate assessments of content, particularly in differentiating
between harmful and non-harmful responses.

###  A.2 Utility Evaluation

We use two metrics to evaluate the quality of a response: perplexity score and
BLEURT score. Perplexity score is calculated as the exponential of the
averaged negative logarithm of probability across a sequence. Given a sequence
of tokens $X=(x_{0},x_{1},\ldots,x_{t})$, the perplexity of $X$ is:

$PPL(X)=\exp\left\\{-\frac{1}{t}\sum_{i}\log
p_{\theta}(x_{i}|x_{<i})\right\\},$ (9)

where $\log p_{\theta}(x_{i}|x_{<i})$ represents log-likelihood of the $i$-th
token when it is conditioned on its preceding sequence of tokens $x_{<i}$ in
the model’s framework. Perplexity score fundamentally assesses the model’s
proficiency in making uniform predictions across a predefined set of tokens
within a text corpus.

Secondly, we use the BLEURT Sellam et al. (2020) score to measure the semantic
similarity of generations between the unlearned model and the original model
on normal prompts. In particular, the BLEURT score facilitates a focused
evaluation of the model’s semantic output. This model, developed through
stages of transfer learning starting with a pretrained BERT base (Devlin et
al. 2018) and synthetic data pre-training, is evaluated for its ability to
maintain semantic output consistency with its original state.

##  Appendix B Appendix: Implementation Details

###  B.1 Baseline Descriptions

First of all, for finetuning (FT) approach, we use the rest of non-harmful
samples from PKU-SafeRLHF Ji et al. (2023), where the response is marked as
safe response, to fine-tune the original model. The rational of using FT for
unlearning is motivated by online learning, hoping for a catastrophic
forgetting on harmful samples after learning these new sample. Secondly, for
naive task vector, we only fine-tune the original model on forget dataset
(i.e. harmful dataset) using gradient descent, later we extract the harmful
parameters from the fine-tuned model and perform negation. Next, for gradient
ascent (GA) Thudi et al. (2022), we add the gradient updates on forget dataset
during the training process back to the original model. In particular, given a
dataset $D_{f}=\\{(x_{i},y_{i})\\}_{i=1}^{N}$ and a loss function
$l(h_{\theta}(x),y)$, the GA approach updates the model iteratively:

$\displaystyle\theta_{t+1}\leftarrow\theta_{t}+\lambda\nabla_{\theta_{t}}l(h_{\theta}(x),y),$
(10)

where $\lambda$ is the learning rate and $(x,y)\thicksim D_{f}$. Lastly, built
based on GA approach, GA+Mismatch Yao et al. (2023) adds random responses from
normal dataset to each training steps. Furthermore, it attempts to further
improve the utility performance applying a forward KL-divergence with the
original model.

###  B.2 Experiment Settings

For each type of unlearned harmful prompts, unseen harmful prompts, and normal
prompts, we select 100 prompts from each of them as test data. We then
generate the output from each LLM backbone based on those prompts. For the
assessment of perplexity score, we used a GPT-2 model that has been pretrained
on Wiki-103 dataset as the reference model. For the evaluation of the BLUERT
score, which measures the semantic quality of generated texts, we computed the
mean pairwise BLUERT score among all outputs generated by unlearned LLM and
original LLM corresponding to normal prompts.

###  B.3 Hyperparameters Settings

Here we present the hyperparameter settings in Table 4. For LLAMA2 models
(i.e. LLAMA2-7B and LLAMA2-13B), we use LoRA during the fintuning process. All
experiments are conducted on A100 GPUs (80 GB).

LLMs Architecture |  | Max Unlearn
---
Steps
| Batch
---
Size
$\epsilon_{1}$ | $\epsilon_{2}$ | $\epsilon_{3}$ |  | Learning
---
Rate
OPT-2.7B | 1000 | 2 | 2.5 | 2.5 | 1 | $2\times 10^{-4}$
LLAMA2-7B | 1000 | 2 | 2.5 | 1 | 0.5 | $2\times 10^{-5}$
LLAMA2-13B | 1000 | 1 | 2.5 | 1 | 0.5 | $2\times 10^{-4}$

Table 4: Hyperparameter settings for SKU alongside with a number of baseline
approaches.

##  Appendix C Appendix: Additional Experiments

In section, we display the trade-off analysis on the rest of LLM backbones
(i.e. OPT-2.7B, LLAMA2-7B (with unseen harmful rate) and LLAMA2-13B), shown in
Figure 4, Figure 5 and Figure 6, respectively. Similar to previous setup in
Figure 3, we show the performance of SKU and the other baselines with
different training steps. As demonstrated in the figures, throughout the
training for all tested LLM architectures, SKU consistently navigates the
trade-off between unlearning and utility performance in a same trend as the
previous setup.

(a) Harmful Rates vs Steps

(b) Harmful Rates vs Steps

(c) Perplexity Score vs Steps

(d) BLEURT Score vs Steps

Figure 4:  The performance of SKU with a number of baselines on OPT2.7B.
Figure 4(a) and Figure 4(b) denotes the unlearning performance on unlearned
and unseen samples, respectively. The $x$ axis represents the training steps
and $y$ axis denotes the unlearn harmful rates. Figure 4(c) and 4(d) stands
for the utility performance of each approach, where the $x$ axis represents
the training steps and $y$ axis denotes the perplexity score and BLEURT score,
respectively. The orange line represents the performance of SKU.

(a) Harmful Rates vs Steps

(b) Harmful Rates vs Steps

(c) Perplexity Score vs Steps

(d) BLEURT Score vs Steps

Figure 5:  The performance of SKU with a number of baselines on LLAMA2-7B.
Figure 5(a) and Figure 5(b) denotes the unlearning performance on unlearned
and unseen samples, respectively. The $x$ axis represents the training steps
and $y$ axis denotes the unlearn harmful rates. Figure 5(c) and 5(d) stands
for the utility performance of each approach, where the $x$ axis represents
the training steps and $y$ axis denotes the perplexity score and BLEURT score,
respectively. The orange line represents the performance of SKU.

(a) Harmful Rates vs Steps

(b) Harmful Rates vs Steps

(c) Perplexity Score vs Steps

(d) BLEURT Score vs Steps

Figure 6:  The performance of SKU with a number of baselines on LLAMA2-13B.
Figure 6(a) and Figure 6(b) denotes the unlearning performance on unlearned
and unseen samples, respectively. The $x$ axis represents the training steps
and $y$ axis denotes the unlearn harmful rates. Figure 6(c) and 6(d) stands
for the utility performance of each approach, where the $x$ axis represents
the training steps and $y$ axis denotes the perplexity score and BLEURT score,
respectively. The orange line represents the performance of SKU.