in Definition 17. This follows straightforwardly by induction and by the
definition of the translation (cf. Definition 16).

In what follows, we will sometimes refer to the shiftings of
$\mathsf{KRel}_{\mathcal{L}}$-inequalities corresponding to Sahlqvist mrps of
type (a) (resp. (b)) as $D$-shiftings (resp. $E$-shiftings).

###### Example 7

The mrp $p\leq\Diamond\Box p$ is inductive of type (a), where $\varphi(y):=y$,
and $\alpha(p):=p$, hence $\mathsf{LA}(\alpha)(u):=u$, and $\psi(x):=x$, hence
$\mathsf{LA}(\psi)(v):=v$, and $\chi(p):=\Diamond\Box p$, and, as discussed in
Example 4 , its first-order correspondent on classical Kripke frames can be
represented as the following $\mathsf{KRel}$-inequality: $\Delta\subseteq
R_{\Diamond}\,\star R_{\Box}\,\star\Delta$, while as discussed in Example 2,
its first-order correspondent on graph-based frames can be represented as the
following $\mathsf{GRel}$-inequality: $D\subseteq R_{\Diamond}\,\ast
R_{\Box}\,\ast D$. Since $\tau(D)=\Delta$ and $\tau(R_{\Diamond}\,\ast
R_{\Box}\,\ast D)=R_{\Diamond}\,\star R_{\Box}\,\star\Delta$, By Proposition
7, $D\subseteq R_{\Diamond}\,\ast R_{\Box}\,\ast D$ is a shifting
(particularly $D$-shifting) of $\Delta\subseteq R_{\Diamond}\,\star
R_{\Box}\,\star\Delta$.

We conclude the present subsection with the following example in which all the
steps from the ALBA output and the generation of the inequalities are
presented.

###### Example 8

The mrp $\Diamond p\leq\Box\Diamond\Box p$ is inductive of type (a), where
$\varphi(y):=\Diamond y$, and $\alpha(p):=p$, hence
$\mathsf{LA}(\alpha)(u):=u$, and $\psi(x):=\Box x$, hence
$\mathsf{LA}(\psi)(v):=\Diamondblack v$, and $\chi(p):=\Diamond\Box p$. Let us
compute the $\mathsf{GRel}_{\mathcal{L}}$-inequality representing the first
order correspondent of this mrp on graph-based frames:

| $\forall\mathbf{j}\left(\Diamondblack[\Diamond[\mathbf{j}/y]/v]\leq\Diamond\Box[[\mathbf{j}/u]/p]\right)$ |
---|---|---
iff | $\forall\mathbf{j}\left(\Diamondblack\Diamond\mathbf{j}\leq\Diamond\Box\mathbf{j}\right)$ |
iff | $\forall\mathbf{j}\left((\\![{\Diamond\Box\mathbf{j}}]\\!)\subseteq(\\![{\Diamondblack\Diamond\mathbf{j}}]\\!)\right)$ | (Definition of order on polarities)
iff | $\forall a\left(R_{\Diamond}^{[0]}[R_{\Box}^{[0]}[E^{[1]}[a]]]\subseteq R_{\Diamondblack}^{[0]}[E^{[1]}[R_{\Diamond}^{[0]}[a]]]\right)$ | (Standard translation)
iff | $\forall a\left((R_{\Diamond}\,\ast R_{\Box}\,\ast D)^{[0]}[a]\subseteq(R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}R_{\Diamond})^{[0]}[a]\right)$ |  (Definition 14)
iff |  $R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}R_{\Diamond}\subseteq R_{\Diamond}\,\ast R_{\Box}\,\ast D$. |  (Lemma 1)

Let us now compute the $\mathsf{KRel}_{\mathcal{L}}$-inequality equivalent to
the first order correspondent of the mrp on Kripke frames:

| $\forall\mathbf{j}\left(\Diamondblack[\Diamond[\mathbf{j}/y]/v]\leq\Diamond\Box[[\mathbf{j}/u]/p]\right)$ |
---|---|---
iff | $\forall\mathbf{j}\left(\Diamondblack\Diamond\mathbf{j}\leq\Diamond\Box\mathbf{j}\right)$ |
iff | $\forall\mathbf{j}\left((\\![{\Diamond\Box\mathbf{j}}]\\!)\subseteq(\\![{\Diamondblack\Diamond\mathbf{j}}]\\!)\right)$ | (Definition of order on polarities)
iff | $\forall a\left(R_{\Diamond}^{[0]}[R_{\Box}^{[0]}[\Delta^{[0]}[a]]]\subseteq R_{\Diamondblack}^{[0]}[\Delta^{[1]}[R_{\Diamond}^{[0]}[a]]]\right)$ | (Standard translation)
iff | $\forall a\left((R_{\Diamond}\,\star R_{\Box}\,\star\Delta)^{[0]}[a]\subseteq(R_{\Diamondblack}\,\circ R_{\Diamond})^{[0]}[a]\right)$ |  (Definition 14)
iff |  $R_{\Diamondblack}\,\circ R_{\Diamond}\subseteq R_{\Diamond}\,\star R_{\Box}\,\star\Delta$. |  (Lemma 1)

The $\mathsf{GRel}$-inequality
$R_{\Diamondblack}{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}R_{\Diamond}\,\subseteq
R_{\Diamond}\ast R_{\Box}\,\ast D$ is the translation (and $D$-shifting) of
$R_{\Diamondblack}\circ R_{\Diamond}\,\subseteq R_{\Diamond}\star
R_{\Box}\,\star\Delta$.

###  6.4 Lifting graph-based frames to polarity-based frames

Having established a systematic connection between the first-order
correspondents of inductive mrps on Kripke frames and on graph-based frames in
the previous subsection, in the present subsection we establish analogous
results along the following lifting construction of graph-based frames to
polarity-based frames.

######  Definition 20 (Lifting of graph-based frames)

For any reflexive graph $\mathbb{G}=(Z,E)$ and any graph-based frame
$\mathbb{F}=(\mathbb{G},R_{\Diamond},R_{\Box})$, let the lifting of
$\mathbb{F}$ be the polarity-based frame
$\mathbb{P}_{\mathbb{F}}:=(\mathbb{P_{G}},J_{R_{\Diamond}^{c}},{I}_{R_{\Box}^{c}})$,
where $\mathbb{P_{G}}\coloneqq(Z_{A},Z_{X},I_{E^{c}})$,
$J_{R_{\Diamond}^{c}}$, and ${I}_{R_{\Box}^{c}}$ are defined as in Section
2.4.

###### Lemma 12

For any reflexive graph $\mathbb{G}=(Z,E)$ and any $E$-compatible relation
$R$, the lifted relations $I_{R^{c}}$ and $J_{R^{c}}$ are
$I_{E^{c}}$-compatible.

###### Proof

We show just one of the two properties of $I_{E^{c}}$-compatibility, as the
other is proved similarly.

| $I_{E^{c}}^{(0)}[I_{E^{c}}^{(1)}[(I_{R^{c}}^{(0)}[x])]]$ |
---|---|---
$=$ | $I_{E}^{[0]}[I_{E}^{[1]}[(I_{R}^{[0]}[x])]]$ |  Definition of $(\cdot)^{[0]}$ and $(\cdot)^{[1]}$
$=$ | $(E^{[0]}[E^{[1]}[(R^{[0]}[x])]])_{A}$ |  Definition of $I_{E}$ and $I_{R}$
$=$ | $((R^{[0]}[x])^{[10]})_{A}$ | notational convention
$\subseteq$ | $(R^{[0]}[x])_{A}$ |  $E$-compatibility of $R$
$=$ | ${I_{R}}^{[0]}[x]$ |  Definition of $I_{R}$
$=$ | ${I_{R^{c}}}^{(0)}[x]$ |  Definition of $(\cdot)^{[0]}$

By the lemma above, $\mathbb{P_{F}}$ is a well defined polarity-based frame,
and, by definition, the complex algebras $\mathbb{F_{P}}^{+}$ and
$\mathbb{F}^{+}$ coincide. Notice that the lifting of Kripke frames to
polarity-based frames defined in [20, Definition 4.17] is the composition of
the shifting defined in Section 6.3 and the lifting defined above. In [20,
Section 4.1], the relational language $\mathsf{PRel}_{\mathcal{L}}$ (see [20,
Definition 3.20]) has been introduced, which contains terms of types
$\mathsf{T}_{A\times X}$, $\mathsf{T}_{X\times A}$, $\mathsf{T}_{A\times A}$
and $\mathsf{T}_{X\times X}$, and is defined by the following simultaneous
recursions:

$\mathsf{T}_{A\times X}\ni\beta$ | $::=$ | $I\mid R_{\Box}\mid\beta\,;_{I}\beta\mid\rho\,;\beta\mid\beta;\lambda$ |  | $\mathsf{T}_{A\times A}\ni\rho$ | $::=$ | $\delta\,;\beta$
---|---|---|---|---|---|---
$\mathsf{T}_{X\times A}\ni\delta$ | $::=$ | $J\mid R_{\Diamond}\mid\delta\,;_{I}\delta\mid\lambda\,;\delta\mid\delta;\rho$ |  | $\mathsf{T}_{X\times X}\ni\lambda$ | $::=$ |  $\beta\,;\delta$,

where $I,J,R_{\Diamond},$ and $R_{\Box}$ are naturally interpreted as the
$I,J,R_{\Diamond},$ and $R_{\Box}$ of a polarity-based frame, while ; and
$;_{I}$ are interpreted as the composition operations in [20, Definition 3.2].
Moreover, relational $\mathsf{PRel}_{\mathcal{L}}$-terms
$R_{\varphi},R_{\psi},R_{\chi},R_{\zeta}$ are associated with formulas
$\varphi,\psi,\chi,\zeta$ as in Definition 17.

In [20, Section 4.1], the first-order correspondents of inductive mrps on
polarity-based frames are shown to be representable as term inequalities in
$\mathsf{PRel}_{\mathcal{L}}$ as indicated below:

type (a) |  $R_{\chi[\mathsf{LA}(\alpha)/p]}\subseteq R_{\mathsf{LA}(\psi)}\,;_{I}R_{\varphi}$ of type $\mathsf{T_{X\times A}}$,
---|---
type (b) |  $R_{\zeta[\mathsf{RA}(\delta)/p]}\subseteq R_{\mathsf{RA}(\varphi)}\,;_{I}R_{\psi}$ of type $\mathsf{T_{A\times X}}$.

The following definition extends the definition of lifting in [20, Definition
4.17] to graph-based frames.

###### Definition 21

Let $\xi_{1}\subseteq\xi_{2}$ be a $\mathsf{GRel}_{\mathcal{L}}$-inequality,
and $\beta_{1}\subseteq\beta_{2}$ (resp. $\delta_{1}\subseteq\delta_{2}$) be a
$\mathsf{PRel}_{\mathcal{L}}$-inequality of type $\mathsf{T}_{A\times X}$
(resp. $\mathsf{T}_{X\times A}$).

  1. 1.

The inequality $\beta_{1}\subseteq\beta_{2}$ is the $I$-lifting of
$\xi_{1}\subseteq\xi_{2}$ if for any graph-based frame $\mathbb{F}$,
$\beta_{1}^{\mathbb{P_{F}}}=I_{(\xi_{2}^{\mathbb{F}})^{c}}$ and
$\beta_{2}^{\mathbb{P_{F}}}=I_{(\xi_{1}^{\mathbb{F}})^{c}}$.

  2. 2.

The inequality $\delta_{1}\subseteq\delta_{2}$ is the $J$-lifting of
$\xi_{1}\subseteq\xi_{2}$ if for any graph-based frame $\mathbb{F}$,
$\delta_{1}^{\mathbb{P_{F}}}=J_{(\xi_{2}^{\mathbb{F}})^{c}}$ and
$\delta_{2}^{\mathbb{P_{F}}}=J_{(\xi_{1}^{\mathbb{F}})^{c}}$.

The following theorem extends [20, Theorem 4.20] to graph-based frames.

###### Theorem 6.2

For any inductive modal reduction principle $s(p)\leq t(p)$ of $\mathcal{L}$,

  1. 1.

if $s(p)\leq t(p)$ is of type (a), then the
$\mathsf{PRel}_{\mathcal{L}}$-inequality of type $\mathsf{T}_{X\times A}$
encoding its first-order correspondent on polarity-based frames is the
$J$-lifting of the $\mathsf{GRel}_{\mathcal{L}}$-inequality encoding its
first-order correspondent on graph-based frames;

  2. 2.

if $s(p)\leq t(p)$ is of type (b), then the
$\mathsf{PRel}_{\mathcal{L}}$-inequality of type $\mathsf{T}_{A\times X}$
encoding its first-order correspondent on polarity-based frames is the
$I$-lifting of the $\mathsf{GRel}_{\mathcal{L}}$-inequality encoding its
first-order correspondent on graph-based frames.

###### Proof

Analogous to that of [20, Theorem 4.20].

###### Example 9

The $\mathsf{PRel}_{\mathcal{L}}$-inequality $R_{\Box}\subseteq I$ is the
$I$-lifting of the $\mathsf{GRel}_{\mathcal{L}}$-inequality $E\subseteq
R_{\Box}$. Indeed, $\mathbb{F}\models E\subseteq R_{\Box}$ iff
$I_{R_{\Box}^{c}}\subseteq I_{E^{c}}$ iff $\mathbb{P_{F}}\models
R_{\Box}\subseteq I$ for every graph-based frame $\mathbb{F}$. These
inequalities encode first-order conditions corresponding to the inductive mrp
$\Box p\leq p$ on polarity-based frames and on graph-based frames,
respectively.

Similarly, the $\mathsf{PRel}_{\mathcal{L}}$-inequality $R_{\Box}\subseteq
R_{\Box}\,;_{I}R_{\Box}$ is the $I$-lifting of
$R_{\Box}{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}R_{\Box}\subseteq
R_{\Box}$, Indeed, $\mathbb{F}\models
R_{\Box}{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}R_{\Box}\subseteq R_{\Box}$
iff $I_{R_{\Box}^{c}}\subseteq
I_{(R_{\Box}{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}R_{\Box})^{c}}$ iff
$I_{R_{\Box}^{c}}\subseteq I_{R_{\Box}^{c}}\,;_{I}\,I_{R_{\Box}^{c}}$ iff
$\mathbb{P_{F}}\models R_{\Box}\subseteq R_{\Box}\,;_{I}R_{\Box}$ for every
graph-based frame $\mathbb{F}$. The second equivalence hinges on the identity
$I_{(S{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}T)^{c}}=I_{S^{c}}\,;_{I}\,I_{T^{c}}$
for all $S,T\subseteq Z\times Z$, which is proved similarly to [20, Lemma
3.12]. These inequalities encode first-order conditions corresponding to the
inductive mrp $\Box p\leq\Box\Box p$.

##  7 Applications to rough set theory

In the present section, we take stock of the technical results presented in
the previous sections by further elucidating the relationship between
hyperconstructive approximation spaces (cf. Definition 13) and generalized
approximation spaces (seen as Kripke frames with $R_{\Diamond}=R_{\Box}$, cf.
Definition 1). Firstly, as is expected, the definition of hyperconstructivist
approximation spaces collapses to that of generalized approximation spaces
when instantiating $E\coloneqq\Delta$ (cf. Proposition 5). Thus, generalized
approximation spaces form the proper subclass of hyperconstructivist
approximation spaces capturing the limit case in which there is no inherent
entropy and $R_{\Box}=R_{\blacksquare}=R$. On this subclass, all
$\mathsf{GRel}_{\mathcal{L}}$-inequalities collapse to
$\mathsf{KRel}_{\mathcal{L}}$-inequalities, since the relational compositions
on graph-based frames (cf. Definition 14) reduce to the relational
compositions on Kripke frames (cf. Lemma 9). Hence, in particular, and as is
expected, the correspondents of Sahlqvist mrps on graph-based frames,
represented as discussed in Section 6.1, collapse, in the sense specified in
Lemma 9, to the correspondents of the same mrps on Kripke frames, represented
as discussed in Section 6.2.

More interesting is the relationship in the converse direction, which hinges
on the notion of shifting (cf. Definition 19): every Sahlqvist mrp $s(p)\leq
t(p)$ (cf. Section 2.5) defines an elementary (i.e. first-order definable)
class $\mathbf{K}(s(p)\leq t(p))$ of Kripke frames and an elementary class
$\mathbf{G}(s(p)\leq t(p))$ of graph-based frames. By Theorem 6.1, when
represented as a $\mathsf{GRel}_{\mathcal{L}}$-inequality, the first-order
condition defining $\mathbf{G}(s(p)\leq t(p))$ is the shifting of the first-
order condition defining $\mathbf{K}(s(p)\leq t(p))$, represented as a
$\mathsf{KRel}_{\mathcal{L}}$-inequality. The notion of shifting allows for a
systematic relationship to be established between $\mathbf{K}(s(p)\leq t(p))$
and $\mathbf{G}(s(p)\leq t(p))$ which is formulated in terms of their own
first-order languages, and hence independently from their correspondence-
theoretic link, in the sense that this relationship can be established purely
on the basis of the syntax of $\mathbf{K}(s(p)\leq t(p))$ and
$\mathbf{G}(s(p)\leq t(p))$. A similarly systematic relationship between the
two correspondents of any Sahlqvist mrps on graph-based frames and on
polarity-based frames, respectively, is established in Theorem 6.2 by hinging
on the notion of lifting (cf. Definition 21). This relationship is again
independent from the correspondence-theoretic link of the two first-order
conditions.

It is easy to check that the well known modal axioms listed in Table 3 are all
Sahlqvist mrps, and hence Theorem 6.1 and Theorem 6.2 apply to these axioms.
Table 3 also reports the relational correspondents of these mrps on Kripke
frames, graph-based frames, and polarity-based frames. It is easy to verify
that, for each mrp in the table (they are all analytic inductive, and hence of
both types (a) and (b)), its relational correspondent on graph-based frames
when considering it as type (a) (resp. (b)) is the $D$-shifting (resp.
$E$-shifting) of its relational correspondent on Kripke frames, and its
relational correspondent on polarity-based frames when considering it as type
(a) (resp. (b)) is the $J$-lifting (resp. $I$-lifting) of its relational
correspondent on graph-based frames.

Property | Axiom |  $\mathsf{KRel}_{\mathcal{L}}$ correspondent |  $\mathsf{GRel}_{\mathcal{L}}$ correspondent |  $\mathsf{PRel}_{\mathcal{L}}$ correspondent
---|---|---|---|---
Reflexivity | $p\leq\Diamond p$ | $\Delta\subseteq R_{\Diamond}$ | $D\subseteq R_{\Diamond}$ | $R_{\Diamond}\subseteq J$
$\Delta\subseteq R_{\blacksquare}$ | $E\subseteq R_{\blacksquare}$ | $R_{\blacksquare}\subseteq I$
$\Box p\leq p$ | $\Delta\subseteq R_{\Diamondblack}$ | $D\subseteq R_{\Diamondblack}$ | $R_{\Diamondblack}\subseteq J$
$\Delta\subseteq R_{\Box}$ | $E\subseteq R_{\Box}$ | $R_{\Box}\subseteq I$
Transitivity | $\Diamond\Diamond p\leq\Diamond p$ | $R_{\Diamond}\circ R_{\Diamond}\subseteq R_{\Diamond}$ | $R_{\Diamond}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamond}\subseteq R_{\Diamond}$ | $R_{\Diamond}\subseteq R_{\Diamond}\,;_{I}R_{\Diamond}$
$R_{\blacksquare}\circ R_{\blacksquare}\subseteq R_{\blacksquare}$ | $R_{\blacksquare}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\blacksquare}\subseteq R_{\blacksquare}$ | $R_{\blacksquare}\subseteq R_{\blacksquare}\,;_{I}R_{\blacksquare}$
$\Box p\leq\Box\Box p$ | $R_{\Diamondblack}\circ R_{\Diamondblack}\subseteq R_{\Diamondblack}$ | $R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamondblack}\subseteq R_{\Diamondblack}$ | $R_{\Diamondblack}\subseteq R_{\Diamondblack}\,;_{I}R_{\Diamondblack}$
$R_{\Box}\circ R_{\Box}\subseteq R_{\Box}$ | $R_{\Box}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\Box}\subseteq R_{\Box}$ | $R_{\Box}\subseteq R_{\Box}\,;_{I}R_{\Box}$
Symmetry | $p\leq\Box\Diamond p$ | $R_{\Diamondblack}\subseteq R_{\Diamond}$ | $R_{\Diamondblack}\subseteq R_{\Diamond}$ | $R_{\Diamond}\subseteq R_{\Diamondblack}$
$R_{\Box}\subseteq R_{\blacksquare}$ | $R_{\Box}\subseteq R_{\blacksquare}$ | $R_{\blacksquare}\subseteq R_{\Box}$
$\Diamond\Box p\leq p$ | $R_{\Diamond}\subseteq R_{\Diamondblack}$ | $R_{\Diamond}\subseteq R_{\Diamondblack}$ | $R_{\Diamondblack}\subseteq R_{\Diamond}$
$R_{\blacksquare}\subseteq R_{\Box}$ | $R_{\blacksquare}\subseteq R_{\Box}$ | $R_{\Box}\subseteq R_{\blacksquare}$
Seriality | $\Box p\leq\Diamond p$ | $\Delta\subseteq R_{\Diamond}\circ R_{\Diamondblack}$ | $D\subseteq R_{\Diamond}{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}R_{\Diamondblack}$ | $R_{\Diamond}\,;_{I}R_{\Diamondblack}\subseteq J$
$\Delta\subseteq R_{\Box}\circ R_{\blacksquare}$ | $E\subseteq R_{\Box}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\blacksquare}$ | $R_{\Box}\,;_{I}R_{\blacksquare}\subseteq I$
Partial functionality | $\Diamond p\leq\Box p$ | $R_{\Diamondblack}\circ R_{\Diamond}\subseteq\Delta$ | $R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamond}\subseteq D$ | $J\subseteq R_{\Diamondblack}\,;_{I}R_{\Diamond}$
$R_{\blacksquare}\circ R_{\Box}\subseteq\Delta$ | $R_{\blacksquare}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\Box}\subseteq E$ | $I\subseteq R_{\blacksquare}\,;_{I}R_{\Box}$
Euclideanness | $\Diamond p\leq\Box\Diamond p$ | $R_{\Diamondblack}\circ R_{\Diamond}\subseteq R_{\Diamond}$ | $R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamond}\subseteq R_{\Diamond}$ | $R_{\Diamond}\subseteq R_{\Diamondblack}\,;_{I}R_{\Diamond}$
$R_{\blacksquare}\circ R_{\Box}\subseteq R_{\blacksquare}$ | $R_{\blacksquare}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\Box}\subseteq R_{\blacksquare}$ | $R_{\blacksquare}\subseteq R_{\blacksquare}\,;_{I}R_{\Box}$
$\Diamond\Box p\leq\Box p$ | $R_{\Diamondblack}\circ R_{\Diamond}\subseteq R_{\Diamondblack}$ | $R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamond}\subseteq R_{\Diamondblack}$ | $R_{\Diamondblack}\subseteq R_{\Diamondblack}\,;_{I}R_{\Diamond}$
$R_{\blacksquare}\circ R_{\Box}\subseteq R_{\Box}$ | $R_{\blacksquare}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\Box}\subseteq R_{\Box}$ | $R_{\Box}\subseteq R_{\blacksquare}\,;_{I}R_{\Box}$
Confluence | $\Diamond\Box p\leq\Box\Diamond p$ | $R_{\Diamondblack}\circ R_{\Diamond}\subseteq R_{\Diamond}\circ R_{\Diamondblack}$ | $R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamond}\subseteq R_{\Diamond}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamondblack}$ | $R_{\Diamond}\,;_{I}R_{\Diamondblack}\subseteq R_{\Diamondblack}\,;_{I}R_{\Diamond}$
$R_{\blacksquare}\circ R_{\Box}\subseteq R_{\Box}\circ R_{\blacksquare}$ | $R_{\blacksquare}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\Box}\subseteq R_{\Box}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\blacksquare}$ | $R_{\Box}\,;_{I}R_{\blacksquare}\subseteq R_{\blacksquare}\,;_{I}R_{\Box}$
Denseness | $\Diamond p\leq\Diamond\Diamond p$ | $R_{\Diamond}\subseteq R_{\Diamond}\circ R_{\Diamond}$ | $R_{\Diamond}\subseteq R_{\Diamond}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamond}$ | $R_{\Diamond}\,;_{I}R_{\Diamond}\subseteq R_{\Diamond}$
$R_{\blacksquare}\subseteq R_{\blacksquare}\circ R_{\blacksquare}$ | $R_{\blacksquare}\subseteq R_{\blacksquare}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\blacksquare}$ | $R_{\blacksquare}\,;_{I}R_{\blacksquare}\subseteq R_{\blacksquare}$
$\Box\Box p\leq\Box p$ | $R_{\Diamondblack}\subseteq R_{\Diamondblack}\circ R_{\Diamondblack}$ | $R_{\Diamondblack}\subseteq R_{\Diamondblack}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}\,R_{\Diamondblack}$ | $R_{\Diamondblack}\,;_{I}R_{\Diamondblack}\subseteq R_{\Diamondblack}$
$R_{\Box}\subseteq R_{\Box}\circ R_{\Box}$ | $R_{\Box}\subseteq R_{\Box}\,{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}\,R_{\Box}$ | $R_{\Box}\,;_{I}R_{\Box}\subseteq R_{\Box}$
Table 3: Well-known modal reduction principles and their correspondents as
relational inequalities

The mathematically grounded relationships just discussed reflect also on the
‘preservation’ of the intuitive meaning of Sahlqvist mrps across these
different semantics, as we argue in the following examples.

###### Example 10

The Sahlqvist mrps $p\leq\Diamond p$ and $\Box p\leq p$ correspond to the
$\mathsf{KRel}_{\mathcal{L}}$-inequality $\Delta\subseteq R$ on generalized
approximation spaces. When interpreting the accessibility relation $R$ as
epistemic indiscernibility (e.g. that of an agent), the informal
interpretation of $\Delta\subseteq R$ is that if the agent can tell two states
apart, then these states are distinct, and since in generalized approximation
spaces two states are inherently indistinguishable iff they are identical,
then this condition is equivalent to the condition that, if the agent can tell
two states apart, then these states must not be inherently indistinguishable,
which is by and large what the $\mathsf{GRel}_{\mathcal{L}}$-inequalities
$E\subseteq R_{\Box}$, and $D\subseteq R_{\Diamond}$ (i.e. the correspondents
of the same mrps on graph-based frames) express, except for a subtle
refinement. Namely, the information encoded in $E$ about any state $z$ of a
graph-based frame presents itself from two perspectives: the set $E^{[1]}[z]$
of the states which are discernible from $z$, and the set $E^{[0]}[z]$ of the
states from which $z$ is discernible. Since $E$ is not symmetric, these two
perspectives are not equivalent in general, and become equivalent when $E$ is
symmetric. In general, inequality $E\subseteq R_{\Box}$ encodes the condition
that, for any two states $x$ and $y$, if $x$ is inherently indiscernible from
$y$, then $x$ is indistinguishable from $y$ according to the agent. Inequality
$D\subseteq R_{\Diamond}$ says that if $x$ is inherently indiscernible from
$y$, then $y$ is indistinguishable from $x$ also according to the agent.

###### Example 11

The Sahlqvist mrps $p\leq\Box\Diamond p$ and $\Diamond\Box p\leq p$ correspond
to the $\mathsf{KRel}_{\mathcal{L}}$-inequalities $R^{-1}\subseteq R$ and
$R\subseteq R^{-1}$ on Kripke frames. Under the same interpretation of the
accessibility relation as in the example above, this condition encodes the
symmetry of the indiscernibility relation. The same mrps correspond to the
$\mathsf{GRel}_{\mathcal{L}}$-inequalities $R_{\Diamondblack}\subseteq
R_{\Diamond}$ and $R_{\Box}\subseteq R_{\blacksquare}$ on graph-based frames,
which express the symmetry of the indiscernibility relations. Hence, the
meaning of these conditions across the two semantics is both formally and
intuitively verbatim the same.

###### Example 12

The Sahlqvist mrps $\Diamond\Diamond p\leq\Diamond p$ and $\Box p\leq\Box\Box
p$ correspond to the $\mathsf{KRel}_{\mathcal{L}}$-inequality $R\circ
R\subseteq R$ on generalized approximation spaces. When interpreting the
accessibility relation $R$ as epistemic indiscernibility (e.g. that of an
agent), the informal interpretation of $R\circ R\subseteq R$ is that for any
two states $x$ and $y$, if the agent can distinguish $y$ from $x$, then the
agent can distinguish $y$ from every element in $R^{-1}[x]$, which is the set
of all states which the agent cannot tell apart from $x$.

The same mrps correspond to the $\mathsf{GRel}_{\mathcal{L}}$-inequalities
$R_{\Diamond}{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}R_{\Diamond}\subseteq
R_{\Diamond}$, and
$R_{\Box}{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}R_{\Box}\subseteq R_{\Box}$
on graph-based frames. Unravelling
$R_{\Diamond}\,{\vbox{\hbox{\scalebox{0.6}{$\Diamond$}}}}_{E}R_{\Diamond}\subseteq
R_{\Diamond}$, we get that $R_{\Diamond}^{[0]}[x]\subseteq
R_{\Diamond}^{[0]}[E^{[0]}[R_{\Diamond}^{[0]}[x]]]$ for every $x$. That is, if
the agent can distinguish $y$ from $x$, then the agent can distinguish $y$
from any element of $E^{[0]}[R_{\Diamond}^{[0]}[x]]$, which is the set of all
states which can be (inherently) distinguished from every state which the
agent can distinguish from $x$, that is, the set of all the states that the
informational entropy allows to distinguish from all the states which the
agent can distinguish from $x$. Hence, $E^{[0]}[R_{\Diamond}^{[0]}[x]]$ is ‘as
close as it gets’, in a scenario characterized by informational entropy, to
the set of states that the agent cannot distinguish from $x$ in a scenario
with no informational entropy. Similar considerations, from the other
perspective, apply to the meaning of
$R_{\Box}{\vbox{\hbox{\scalebox{0.6}{$\Box$}}}}_{E}R_{\Box}\subseteq
R_{\Box}$.

The above examples show that transferring the modal axioms defining
approximation spaces to the hyperconstructive (graph-based) setting preserves
their intuitive meaning. In some cases, axioms which have the same first-order
correspondent on approximation spaces correspond to non-equivalent first-order
conditions on graph-based frames, which hints at the comparatively greater
richness of the hyperconstructivist setting.

We can now refer to classes of hyperconstructivist approximation spaces
defined by Sahlqvist mrps using the same name we use for the classes of
generalized approximation spaces of which they are the shiftings. For example,
reflexive graph-based frames are defined as graph-based frames satisfying the
reflexivity axioms $D\subseteq R_{\Diamond}$ and $E\subseteq R_{\Box}$, while
the class of symmetric graph-based frames are defined as graph-based frames
satisfying the symmetry axioms $R_{\Box}\subseteq R_{\blacksquare}$ and
$R_{\Diamondblack}\subseteq R_{\Diamond}$. As discussed in Section 4, this
motivates us to define different sub-classes of hyperconstructivist
approximation spaces like reflexive, symmetric, transitive and Pawlak
approximation spaces (cf. Definition 13) as analogues of these classes of
frames in classical approximation spaces to the graph-based frames. The
following propositions follows immediately from the general results discussed
in the previous section.

###### Proposition 8

For any Pawlak’s hyperconstructivist approximation space
$\mathbb{F}=(Z,E,R_{\Diamond},R_{\Box})$, and any $a,b\in\mathbb{F}^{+}$,

1.$\Diamond(a\vee b)=\Diamond a\vee\Diamond b$ |  2.$\Box(a\wedge b)=\Box a\wedge\Box b$ |  3.$a\leq\Box b\Rightarrow\Diamond a\leq b$
---|---|---
4.$\Diamond a\leq b\Rightarrow a\leq\Box b$ |  5.$\Box a\leq a$ |  6.$a\leq\Diamond a$
7.$a\leq\Box\Diamond a$ |  8.$\Diamond\Box a\leq a$ |  9.$\Diamond\Diamond a\leq\Diamond a$
10.$\Box a\leq\Box\Box a$. |  |

All the conditions of the proposition above are satisfied by the lower and
upper approximation operators of Pawlak’s approximation spaces.

Different axiomatic classes of approximation spaces are used in rough set
theory as models of approximate reasoning in different situations. The
systematic relationship established in the present paper among generalized,
hyperconstructivist, and conceptual approximation spaces makes it possible to
capture a wider range of situations and reasoning modes while still taking
advantage of the basic intuitions developed about the original setting.

On the algebraic side, the present results pave the way to systematically
extending rough set theory to algebras based on general (i.e. not necessarily
distributive) lattices. Different classes of general lattices with
approximation operators can be defined which as lattice-based counterparts of
classes of Boolean algebras with operators. Algebraic models of rough set
theory based on various classes of lattices such as Boolean algebras,
completely distributive lattices, ortholattices, De Morgan lattices (cf.
Section 2.3) can then be encompassed and studied uniformly as subclasses of
lattice-based ‘rough algebras’, which will then serve as a suitable common
ground for comparing and transferring insights and results concerning formal
frameworks of rough set theory defined on (or dual to) different classes of
lattices.

##  8 Conclusions

The present paper provides a systematic theory to explain and generalize some
observations about similarities in the shape of first-order correspondents of
the common rough set theory axioms like reflexivity, symmetry and transitivity
in Kripke and graph-based frames. The precise nature of these similarities is
made explicit via the notion of shifting. This notion makes it possible to
define the shifted counterparts of first-order conditions which are well known
and widely used in modal logic and rough set theory; thus, we are now in a
position to provide counterparts of rough theory axioms and different classes
of approximation spaces in graph-based frames.

##### Parametric correspondence.

Together with [20], Theorems 6.1 and 6.2 suggest that the various
correspondence theories for different logics and semantic contexts can be not
only methodologically unified by the same algebraic and algorithmic
mechanisms; they can also be parametrically related to each other in terms of
their outputs. Furthermore, this parametric relation hinges on the natural
connections between different semantic settings. In the present case, Kripke
frames can be naturally embedded into graph-based frames, and graph-based
frames into polarity-based frames. The present contribution can be understood
as a preservation result of the first-order correspondents of inductive mrps
under these natural embeddings, as shown in Figure 2.

blue = all Sahlqvist formulasred = all modal reduction principlesMV-
Kripkeframesverbatim the same[10]Kripke framesliftedliftedMV-Polarity-
basedframesverbatim the samePolarity-basedframesliftedshiftedGraph-
basedframesMV-Graph-basedframes Figure 2: A commutative diagram of semantic
contexts.

##### Notation.

A crucial step in achieving these results is the ability to express the first-
order correspondents of modal axioms in various semantic contexts in a more
algebraic language, specifically in the languages of (heterogeneous) relation
algebras. As we illustrated with the example of the modal transitivity axiom,
the mathematical and meaning-based connections are near impossible to detect
when the correspondents are expressed in the standard first-order frame
correspondence languages of Kripke, polarity and graph based frames,
respectively, yet become transparently clear when written in a suitable
relational algebraic language as inclusions of relational compositions. The
use of these languages to express correspondents is essential in the
formulations and proofs of the general results given in Theorems 6.1 and 6.2.

##### Future work.

The present work can be further pursued in several directions. Firstly, the
perfect modularity of the graph-based setting allows to systematically study
natural subclasses of hyperconstructivist approximation spaces. One example of
such a class would be the hyperconstructivist counterparts of tolerance spaces
[71], obtained by assuming the symmetry of the indiscernibility relation.
Secondly, similar shifting/lifting results may be sought for other logics,
including polyadic modal logics and many-valued modal logics (i.e., a similar
correspondence may be shown between many-valued Kripke frames and many-valued
graph-based frames). For instance, a natural extension of the present work
concerns investigating the remaining connections between the semantics
presently investigated and their many-valued versions, as represented by the
three un-annotated edges in Figure 2. Thirdly, the class of formulas covered
by the shifting/lifting could possibly be expanded to more than inductive
modal reduction principles. However, we do not expect this to be possible for
all formulas with first-order frame correspondents, not even for all inductive
formulas. Indeed, there exist inductive formulas whose corespondents over
polarity-based frames are not liftings of their correspondents over Kripke
frames (see the concluding section of [20]), and we expect similar examples to
exist in the case of shiftings to graph-based frames. Fourthly, the literature
contains instances of correspondence theory for modal logics on non-classical
base obtained via Gödel-McKinsey-Tarski (GMT) translation [45, 55] to the
multi-modal classical context (see e.g. [44] for an instance in the context of
distributive modal logic and [30] for one in the context of bi-intuitionistic
modal logic). In the context of graph-based frames, a similar approach
suggests itself, whereby one would translate formulas over graph-based frames
into formulas over Kripke frames, prefixing propositional variables with
modalities interpreted with the relation $E$ to simulate the closure of
valuations. However, this approach involves several challenges, especially on
the proof-theoretic front, not least of which is the fact that conditions such
as the $E$-compatibility of the additional relations can not be represented by
(analytic) inductive inequalities, and thus, the general methods for
developing analytic display calculi for modal logics developed in [49] would
not be available, while they are straightforwardly applicable to the present
setting. Future work will investigate such a GMT-translation based approach
but, at present, it remains to be seen whether the parametric connection
between correspondents on Kripke and graph-based frames can also be obtained
in this way. Finally, parametric connections between the correspondents of
modal formulas remain to be investigated between other semantics environments.

####  8.0.1 Acknowledgements.

The research of Krishna Manoorkar is supported by the NWO grant KIVI.2019.001
awarded to Alessandra Palmigiano. The authors confirm that there are no
relevant financial or non-financial competing interests to report.

## References

  * [1] Balbiani, P., Vakarelov, D.: First-order characterization and modal analysis of indiscernibility and complementarity in information systems. In: Proc. ECSQARU. pp. 772–781. Springer (2001)
  * [2] Banerjee, M., Chakraborty, M.: Rough consequence and rough algebra. In: Rough Sets, Fuzzy Sets and Knowledge Discovery, pp. 196–207. Springer (1994)
  * [3] Banerjee, M., Chakraborty, M.: Algebras from rough sets. In: Rough-Neural Computing, pp. 157–184. Springer (2004)
  * [4] Banerjee, M., Chakraborty, M.K.: Rough sets through algebraic logic. Fundamenta Informaticae 28(3-4), 211–221 (1996)
  * [5] van Benthem, J.: Modal reduction principles. Journal of Symbolic Logic 41(2), 301–312 (06 1976), http://projecteuclid.org/euclid.jsl/1183739770
  * [6] van der Berg, I., De Domenico, A., Greco, G., Manoorkar, K.B., Palmigiano, A., Panettiere, M.: Labelled calculi for lattice-based modal logics. In: Banerjee, M., Sreejith, A.V. (eds.) Logic and Its Applications. pp. 23–47. Springer Nature Switzerland, Cham (2023)
  * [7] van der Berg, I., De Domenico, A., Greco, G., Manoorkar, K.B., Palmigiano, A., Panettiere, M.: Labelled calculi for the logics of rough concepts. In: Banerjee, M., Sreejith, A.V. (eds.) Logic and Its Applications. pp. 172–188. Springer Nature Switzerland, Cham (2023)
  * [8] Blackburn, P., De Rijke, M., Venema, Y.: Modal logic, vol. 53. Cambridge University Press (2002)
  * [9] Bou, F., Esteva, F., Godo, L., Rodríguez, R.O.: On the minimum many-valued modal logic over a finite residuated lattice. Journal of Logic and computation 21(5), 739–790 (2011)
  * [10] Britz, C.: Correspondence theory in many-valued modal logics. Master’s thesis, University of Johannesburg, South Africa (2016)
  * [11] Cattaneo, G., Chiaselotti, G., Ciucci, D., Gentile, T.: On the connection of hypergraph theory with formal concept analysis and rough set theory. Information Sciences 330, 342–357 (2016)
  * [12] Cattaneo, G.: Mathematical foundations of roughness and fuzziness (1996)
  * [13] Cattaneo, G., Ciucci, D.: Algebraic structures for rough sets. In: Transactions on Rough sets II, pp. 208–252. Springer (2004)
  * [14] Chakraborty, M.K.: On fuzzy sets and rough sets from the perspective of indiscernibility. In: Indian Conference on Logic and Its Applications. pp. 22–37. Springer (2011)
  * [15] Conradie, W., Craig, A.: Relational semantics via TiRS graphs. TACL 2015 (2015)
  * [16] Conradie, W., Frittella, S., Palmigiano, A., Piazzai, M., Tzimoulis, A., Wijnberg, N.: Toward an epistemic-logical theory of categorization. In: Proc. TARK 2017. EPTCS, vol. 251, pp. 170–189 (2017)
  * [17] Conradie, W., Palmigiano, A.: Algorithmic correspondence and canonicity for non-distributive logics. Annals of Pure and Applied Logic p. DOI: 10.1016/j.apal.2019.04.003 (2019)
  * [18] Conradie, W., Craig, A., Palmigiano, A., Wijnberg, N.: Modelling competing theories. In: Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology (EUSFLAT 2019). pp. 721–739. Atlantis Press (2019/08). https://doi.org/https://doi.org/10.2991/eusflat-19.2019.100, https://doi.org/10.2991/eusflat-19.2019.100
  * [19] Conradie, W., Craig, A., Palmigiano, A., Wijnberg, N.M.: Modelling informational entropy. In: Iemhoff, R., Moortgat, M., de Queiroz, R. (eds.) Logic, Language, Information, and Computation. pp. 140–160. Springer Berlin Heidelberg, Berlin, Heidelberg (2019)
  * [20] Conradie, W., De Domenico, A., Manoorkar, K., Palmigiano, A., Panettiere, M., Prieto, D.P., Tzimoulis, A.: Modal reduction principles across relational semantics. arXiv preprint arXiv:2202.00899 (2022)
  * [21] Conradie, W., Frittella, S., Manoorkar, K., Nazari, S., Palmigiano, A., Tzimoulis, A., Wijnberg, N.M.: Rough concepts. Information Sciences 561, 371–413 (2021). https://doi.org/https://doi.org/10.1016/j.ins.2020.05.074, https://www.sciencedirect.com/science/article/pii/S0020025520304989
  * [22] Conradie, W., Frittella, S., Palmigiano, A., Piazzai, M., Tzimoulis, A., Wijnberg, N.M.: Categories: how i learned to stop worrying and love two sorts. In: International Workshop on Logic, Language, Information, and Computation. pp. 145–164. Springer (2016)
  * [23] Conradie, W., Ghilardi, S., Palmigiano, A.: Unified correspondence. In: Baltag, A., Smets, S. (eds.) Johan van Benthem on Logic and Information Dynamics, Outstanding Contributions to Logic, vol. 5, pp. 933–975. Springer International Publishing (2014). https://doi.org/10.1007/978-3-319-06025-5_36, http://dx.doi.org/10.1007/978-3-319-06025-5_36
  * [24] Conradie, W., Palmigiano, A.: Algorithmic correspondence and canonicity for distributive modal logic. Annals of Pure and Applied Logic 163(3), 338 – 376 (2012)
  * [25] Conradie, W., Palmigiano, A.: Constructive canonicity of inductive inequalities. Logical Methods in Computer Science 16 (2020)
  * [26] Conradie, W., Palmigiano, A., Robinson, C., Tzimoulis, A., Wijnberg, N.: Modelling socio-political competition. Submitted (2019)
  * [27] Conradie, W., Palmigiano, A., Robinson, C., Tzimoulis, A., Wijnberg, N.: Modelling socio-political competition. Fuzzy Sets and Systems 407, 115–141 (2021). https://doi.org/https://doi.org/10.1016/j.fss.2020.02.005, https://www.sciencedirect.com/science/article/pii/S016501142030049X, knowledge Representation and Logics
  * [28] Conradie, W., Palmigiano, A., Robinson, C., Wijnberg, N.: Nondistributive logics — from semantics to meaning logics. In: Rezucs, A. (ed.) Contemporary Logic and Computing, Landscapes in Logic, vol. 1. College Publications (2020)
  * [29] Conradie, W., Palmigiano, A., Sourabh, S.: Algebraic modal correspondence: Sahlqvist and beyond. Journal of Logical and Algebraic Methods in Programming 91, 60–84 (2017)
  * [30] Conradie, W., Palmigiano, A., Zhao, Z.: Sahlqvist via translation. Logical Methods in Computer Science 15 (2019)
  * [31] Craig, A., Gouveia, M., Haviar, M.: TiRS graphs and TiRS frames: a new setting for duals of canonical extensions. Algebra universalis 74(1-2), 123–138 (2015)
  * [32] Craig, A., Haviar, M., Priestley, H.: A fresh perspective on canonical extensions for bounded lattices. Applied Categorical Structures 21(6), 725–749 (2013)
  * [33] Dai, S.: Rough approximation operators on a complete orthomodular lattice. Axioms 10(3), 164 (2021)
  * [34] Davey, B., Priestley, H.: Introduction to lattices and order. Cambridge univ. press (2002)
  * [35] Du, S., Gregory, S.: The echo chamber effect in twitter: does community polarization increase? In: Cherifi, H., Gaito, S., Quattrociocchi, W., Sala, A. (eds.) Complex Networks & Their Applications V. pp. 373–378. Springer International Publishing, Cham (2017)
  * [36] Dubois, D., Prade, H.: Rough fuzzy sets and fuzzy rough sets. International Journal of General System 17(2-3), 191–209 (1990)
  * [37] Fagin, R., Halpern, J.Y., Megiddo, N.: A logic for reasoning about probabilities. Information and computation 87(1-2), 78–128 (1990)
  * [38] Fine, K.: In so many possible worlds. Notre Dame Journal of formal logic 13(4), 516–520 (1972)
  * [39] Fitting, M.: Many-valued modal logics. Fundam. Inform. 15(3-4), 235–254 (1991)
  * [40] Fitting, M.: Many-valued model logics ii. Fundam. Inform. 17(1-2), 55–73 (1992)
  * [41] Formica, A.: Integrating fuzzy formal concept analysis and rough set theory for the semantic web. Bulletin of Computational Applied Mathematics 6(2) (2018)
  * [42] Ganter, B., Wille, R.: Formal concept analysis: mathematical foundations. Springer Science & Business Media (2012)
  * [43] Gehrke, M., Harding, J.: Bounded lattice expansions. Journal of Algebra 238(1), 345–371 (2001)
  * [44] Gehrke, M., Nagahashi, H., Venema, Y.: A Sahlqvist theorem for distributive modal logic. Annals of Pure and Applied Logic 131(1-3), 65–102 (2005)
  * [45] Gödel, K.: Eine interpretation des intnitionistischen aussagenkalkuls. Ergebnisse eines mathematischen Kolloquiums 6, 39–40 (1933)
  * [46] Godo, L., Hájek, P., Esteva, F.: A fuzzy modal logic for belief functions. Fundamenta Informaticae 57(2-4), 127–146 (2003)
  * [47] Greco, G., Jipsen, P., Manoorkar, K., Palmigiano, A., Tzimoulis, A.: Logics for rough concept analysis. In: Proc. ICLA 2019. pp. 144–159. Springer (2019)
  * [48] Greco, G., Liang, F., Manoorkar, K., Palmigiano, A.: Proper multi-type display calculi for rough algebras. Electronic Notes in Theoretical Computer Science 344, 101–118 (2019)
  * [49] Greco, G., Ma, M., Palmigiano, A., Tzimoulis, A., Zhao, Z.: Unified correspondence as a proof-theoretic tool. Journal of Logic and Computation (2016, doi: 101093/logcom/exw022 ArXiv preprint 160308204)
  * [50] Kent, R.E.: Rough concept analysis: a synthesis of rough sets and formal concept analysis. Fundamenta Informaticae 27(2, 3), 169–181 (1996)
  * [51] Kumar, A.: A study of algebras and logics of rough sets based on classical and generalized approximation spaces. In: Transactions on Rough Sets XXII, pp. 123–251. Springer (2020)
  * [52] Kumar, A., Banerjee, M.: Algebras of definable and rough sets in quasi order-based approximation spaces. Fundamenta Informaticae 141(1), 37–55 (2015)
  * [53] Liu, G.: Axiomatic systems for rough sets and fuzzy rough sets. International Journal of Approximate Reasoning 48(3), 857–867 (2008)
  * [54] Ma, M., Chakraborty, M.K., Lin, Z.: Sequent calculi for varieties of topological quasi-boolean algebras. In: International Joint Conference on Rough Sets. pp. 309–322. Springer (2018)
  * [55] McKinsey, J.C., Tarski, A.: Some theorems about the sentential calculi of lewis and heyting. The journal of symbolic logic 13(1), 1–15 (1948)
  * [56] Moshier, M.: A relational category of formal contexts. Preprint (2016)
  * [57] Orlowska, E.: Rough set semantics for non-classical logics. In: Rough Sets, Fuzzy Sets and Knowledge Discovery, pp. 143–148. Springer (1994)
  * [58] Pawlak, Z., Skowron, A.: Rudiments of rough sets. Information sciences 177(1), 3–27 (2007)
  * [59] Pawlak, Z.: Rough sets. International journal of computer & information sciences 11(5), 341–356 (1982)
  * [60] Pawlak, Z.: Rough probability. Bull. Pol. Acad. Sci. Math 32, 607–615 (1984)
  * [61] Pawlak, Z.: Rough set theory and its applications to data analysis. Cybernetics & Systems 29(7), 661–688 (1998)
  * [62] Ploščica, M.: A natural representation of bounded lattices. Tatra Mountains Mathematical Publications 4 (1994)
  * [63] Powers, E., Koliska, M., Guha, P.: “shouting matches and echo chambers”: perceived identity threats and political self-censorship on social media. International Journal of Communication 13, 20 (2019)
  * [64] Radzikowska, A.M., Kerre, E.E.: Fuzzy rough sets based on residuated lattices. In: Transactions on Rough Sets II, pp. 278–296. Springer (2004)
  * [65] Rainie, L., Smith, A.: Social networking sites and politics. Washington, DC: Pew Internet & American Life Project. Retrieved June 12, 2012 (2012)
  * [66] Rasiowa, H., Skowron, A.: Rough concepts logic. In: Symposium on Computation Theory. pp. 288–297. Springer (1984)
  * [67] Rasouli, S., Davvaz, B.: Roughness in mv-algebras. Information Sciences 180(5), 737–747 (2010)
  * [68] Saha, A., Sen, J., Chakraborty, M.K.: Algebraic structures in the vicinity of pre-rough algebra and their logics. Information Sciences 282, 296–320 (2014)
  * [69] Saha, A., Sen, J., Chakraborty, M.K.: Algebraic structures in the vicinity of pre-rough algebra and their logics ii. Information Sciences 333, 44–60 (2016)
  * [70] Sahlqvist, H.: Completeness and correspondence in the first and second order semantics for modal logic. In: Studies in Logic and the Foundations of Mathematics, vol. 82, pp. 110–143. Elsevier (1975)
  * [71] Skowron, A., Stepaniuk, J.: Tolerance approximation spaces. Fundamenta Informaticae 27(2-3), 245–253 (1996)
  * [72] Sun, B., Gong, Z., Chen, D.: Fuzzy rough set theory for the interval-valued fuzzy information systems. Information Sciences 178(13), 2794–2815 (2008)
  * [73] Thorson, K.: Facing an uncertain reception: young citizens and political interaction on facebook. Information, Communication & Society 17(2), 203–216 (2014). https://doi.org/10.1080/1369118X.2013.862563, https://doi.org/10.1080/1369118X.2013.862563
  * [74] Vakarelov, D.: A modal logic for similarity relations in Pawlak knowledge representation systems. Fundam. Inform. 15(1), 61–79 (1991)
  * [75] Vakarelov, D.: A modal characterization of indiscernibility and similarity relations in pawlak’s information systems. In: International Workshop on Rough Sets, Fuzzy Sets, Data Mining, and Granular-Soft Computing. pp. 12–22. Springer (2005)
  * [76] Van Benthem, J.: Modal reduction principles. The Journal of Symbolic Logic 41(2), 301–312 (1976)
  * [77] Van Benthem, J.: Correspondence theory. In: Handbook of philosophical logic, pp. 167–247. Springer (1984)
  * [78] Van Benthem, J.: Correspondence Theory, pp. 325–408. Springer Netherlands, Dordrecht (2001). https://doi.org/10.1007/978-94-017-0454-0_4, https://doi.org/10.1007/978-94-017-0454-0_4
  * [79] Vraga, E.K., Thorson, K., Kligler-Vilenchik, N., Gee, E.: How individual sensitivities to disagreement shape youth political expression on facebook. Computers in Human Behavior 45, 281–289 (2015)
  * [80] Wu, W.Z., Zhang, W.X.: Constructive and axiomatic approaches of fuzzy approximation operators. Information sciences 159(3-4), 233–254 (2004)
  * [81] Wybraniec-Skardowska, U.: On a generalization of approximation space. Bulletin of the Polish Academy of Sciences. Mathematics 37(1-6), 51–62 (1989)
  * [82] Yao, Y.: A comparative study of formal concept analysis and rough set theory in data analysis. In: International Conference on Rough Sets and Current Trends in Computing. pp. 59–68. Springer (2004)
  * [83] Yao, Y.: Probabilistic rough set approximations. International journal of approximate reasoning 49(2), 255–271 (2008)
  * [84] Yao, Y.: Rough-set concept analysis: interpreting rs-definable concepts based on ideas from formal concept analysis. Information Sciences 346, 442–462 (2016)
  * [85] Yao, Y.: Three-way granular computing, rough sets, and formal concept analysis. International Journal of Approximate Reasoning 116, 106–125 (2020)
  * [86] Yao, Y.: Generalized rough set models. Rough sets in knowledge discovery 1, 286–318 (1998)
  * [87] Yao, Y.: Concept lattices in rough set theory. In: Fuzzy Information, 2004. Processing NAFIPS’04. IEEE Annual Meeting of the. vol. 2, pp. 796–801. IEEE (2004)
  * [88] Yao, Y.: On unifying formal concept analysis and rough set analysis. Rough Set and Concept Lattice, Xi’an Jiaotong University Press, Xi’an pp. 1–20 (2006)
  * [89] Yao, Y., Lin, T.Y.: Generalization of rough sets using modal logics. Intelligent Automation & Soft Computing 2(2), 103–119 (1996)
  * [90] Zakowski, W.: Approximations in the space (u, $\pi$). Demonstratio mathematica 16(3), 761–770 (1983)


# Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging
Perturbations That Efficiently Fool Customized Diffusion Models

Jingyao Xu1
Siyang Lu1*
Yuetong Lu1
Dongdong Wang3
Yandong Li2
Xiang Wei1
1Beijing Jiaotong University 2Google Research 3University of Central Florida
{jingyaoxu<EMAIL_ADDRESS>
###### Abstract

Diffusion models (DMs) embark a new era of generative modeling and offer more
opportunities for efficient generating high-quality and realistic data
samples. However, their widespread use has also brought forth new challenges
in model security, which motivates the creation of more effective adversarial
attackers on DMs to understand its vulnerability. We propose CAAT, a simple
but generic and efficient approach that does not require costly training to
effectively fool latent diffusion models (LDMs). The approach is based on the
observation that cross-attention layers exhibits higher sensitivity to
gradient change, allowing for leveraging subtle perturbations on published
images to significantly corrupt the generated images. We show that a subtle
perturbation on an image can significantly impact the cross-attention layers,
thus changing the mapping between text and image during the fine-tuning of
customized diffusion models. Extensive experiments demonstrate that CAAT is
compatible with diverse diffusion models and outperforms baseline attack
methods in a more effective (more noise) and efficient (twice as fast as Anti-
DreamBooth and Mist) manner.

Figure 1: With subtle perturbation, CAAT can efficiently and consistently
degrade various customized diffusion models. First Line: Existing malicious
attackers can use a few images publicly posted by users to generate users’
images using various customized diffusion models. Second Line: Our CAAT,
through subtle perturbations to the images, significantly disrupt the images
generated from the customized diffusion models. We exemplify our approach by
selecting the prompts “a photo of a person” and “a photo of a person in front
of a pyramid”.

##  1 Introduction

Diffusion Models (DMs) [11] represent a cutting-edge advancement in the field
of generative models, particularly within the realm of Text-to-Image (T2I)
generation. This strategic breakthrough in generative modeling has gained
recognition for its remarkable efficacy and potency in capturing intricate
patterns and nuances. The technique can effortlessly transform textual
descriptions into rich and visually compelling images.

In the effort to make generative modeling with DM more efficient, the
development has led to the creation of multiple variants of DM models,
including Textual Inversion [6], DreamBooth [27], Custom Diffusion [16], and
SVDiff [9], etc. These variants provide users with more customized and
enhanced experiences, allowing them to obtain precise images of a specified
subject by using prompts and only a small set (4-5) of relevant images. This
level of customization not only empowers users to create unique and
personalized content but also cultivates a widely embraced and individualized
creative experience.

Despite the power of these models, users should be careful to avoid any
harmful or unintended consequences that may arise from their applications. For
example, malicious attackers can exploit photos available on the internet and
use customized LDMs to generate deceptive and harmful fake images. Of even
greater concern is the ability of attackers to fabricate false news images for
their personal gain. Our research is dedicated to protecting users from
malicious T2I attacks. Through effective strategies, we attempt to contribute
valuable insights to enhance understanding and bolster security in the T2I
domain. Currently, there exist adversarial attack methods, such as Anti-
DreamBooth [32] and Mist [18], which have been developed to tackle the
aforementioned issues. Anti-DreamBooth has exhibited remarkable proficiency in
countering adversarial face attacks specifically targeted at DreamBooth, while
Mist has proven its effectiveness in preserving artists’ copyrights from the
transformative effects of AI-for-art. However, it is crucial to acknowledge
that current solutions have limitations, particularly in terms of their
ability to generalize and their efficiency in terms of time.

Our research aims to overcome these limitations by focusing on the
generalization of adversarial attack methods and their strong adaptability to
a broader range of scenarios and systems. Moreover, the reduction of execution
time is pivotal for streamlining processes, ensuring more practical and
efficient applications in the real world. To tackle these challenges, we focus
on attacking LDMs as a whole. A direct method involves executing a Projected
Gradient Descent (PGD) [21] attack on LDMs, targeting all parameters, similar
to the approach employed by Anti-DreamBooth on DreamBooth. However, the
substantial number of parameters in LDMs results in considerable time and
space overhead.

We propose adversarial optimization on only cross-attention layers for an
efficient PGD attack. Inspired by Custom Diffusion, we have found that
attention layers, specifically the cross-attention layers, play a significant
role in the training process of LDMs. The cross-attention layers, integral
components of LDMs, undergo substantial parameter change over the training
despite having a relatively small number of parameters. We intend to leverage
this observation to improve adversarial attacks on DMs. In order to
investigate the effectiveness of the PGD attack on cross-attention layers of
LDMs, we conduct experiments on Stable Diffusion (SD) v2.1, performing PGD
attacks on both the original model and the one fine-tuned with cross-attention
to produce adversarial examples. Then, we train DreamBooth on these
adversarial examples to observe the attack effects. The results justify that
even with minor updates to the cross-attention layers, there is a discernible
improvement in the effectiveness of the attack, as illustrated in Fig. 2.

Figure 2: Illustration of the vulnerability of cross-attention layers by PGD.
The comparison of adversarial attack between cross-attention and other layers
reveal the vulnerability of cross-attention layers in PGD attack. We added
noise to the clean images through PGD attack to generate adversarial examples.
Subsequently, we employed DreamBooth [27] for customized fine-tuning on the
adversarial examples, resulting in generated images from attacked diffusion
model.

According to our preliminary observations, we introduce an adversarial attack
method, Cross-Attention Attack (CAAT), that can be applied to all customized
fine-tuned models based on LDMs. While adding perturbations to generate
adversarial examples, we update the parameters of the cross-attention layers,
a pivotal component of LDMs. The obtained perturbations will affect the cross-
attention layers during fine-tuning, disrupting the mapping from text to
images. Our experiments show that disrupting this key element yields
significant results. Figure 1 demonstrates the outstanding attack
effectiveness of CAAT. Additionally, because the parameters of cross-attention
layers are relatively small, our attack is lightweight and faster in training.
Through extensive testing, our attack method has proven to be effective
against existing LDM-based customized fine-tuning, with minimal time overhead.
The overview of CAAT is presented in Fig. 3.

In summary, our contributions are as follows:

  1. 1.

We identify and leverage the effectiveness of cross-attention layers in LDMs
to efficient adversarial attacks on DMs.

  2. 2.

We developed CAAT, a simple yet effective attacker that exhibits excellent
generalization and efficient training, providing users with robust protection
against portrait rights infringements.

  3. 3.

We justify CAAT’s effectiveness, efficiency, and generality through extensive
experiments across various state-of-the-art adversarial attack methods on
different customized LDM models.

  4. 4.

We perform ablation study to analyze the effectiveness of influential factors
for CAAT and provide suggestions for its application.

##  2 Related Work

###  2.1 T2I models

The advent of foundation models, as proposed by Bommasani et al. [2], has
triggered a noticeable shift in the landscape of deep learning. This
transition is characterized by a growing emphasis on large-scale models,
housing billions of parameters, and trained on extensive datasets. This
evolution has, in turn, propelled advancements in T2I generation models. In
the domain of image generation, Generative Adversarial Networks (GANs)[7],
once emblematic and canonical, are witnessing a gradual displacement by
diffusion models[11]. This shift is attributed to the disruptive influence of
diffusion models on the traditional structure of GANs, leading to a notable
improvement in the quality of generated content [5]. The remarkable success of
diffusion models in the realm of image generation [1, 24, 28, 22] has
redirected research interests, with an increasing focus on their potential
applications in T2I generation.

As an exemplar T2I model, Stable Diffusion (SD)[25] adopts the Contrastive
Language-Image Pre-training (CLIP)[23] Text Encoder for encoding textual
information. SD generates a Gaussian noise matrix, employing a random function
as a “substitute” for the Latent Feature. This matrix is then fed into the
“image optimization module” of the SD model, featuring a U-Net [26] network.
The U-Net network is tasked with predicting noise while simultaneously
integrating semantic information from the textual input. The Scheduler refines
the noise predicted by the U-Net at each iteration. Finally, this refined
information undergoes processing in the Variational AutoEncoder (VAE) [15]
Decoder, culminating in the generation of the final image. This paradigmatic
shift in image generation models underscores the dynamic nature of the field,
spurred by the adoption of foundation models and the continuous pursuit of
enhanced capabilities.

Dreambooth [27] is realized through SD fine-tuning, employing a minimal input
of several images (typically three or four images) to generate corresponding
images under various prompts. Diverging from the SD fine-tuning approach,
Textual Inversion [6] operates with a reduced set of images, generating and
training images with a similar style by specifying new keywords. SVDiff [9]
presents a lighter-weight diffused fine-tuning model designed to mitigate the
risks of overfitting and language drift simultaneously. In contrast, Custom
Diffusion [16] optimizes solely the parameters in the cross-attention layers
of the T2I diffusion model. This targeted optimization facilitates the
efficient learning of new concepts, surpassing DreamBooth and Textual
Inversion models in terms of image generation performance. Notably, Custom
Diffusion achieves this while minimizing memory overhead and enhancing
inference efficiency.

###  2.2 Adversarial attack

The emergence of the Fast Gradient Sign Method (FGSM) [8] has led researchers
in the field of machine learning to focus a significant portion of their
attention on adversarial attacks. Its idea is to add the computed loss value
to the input image, causing an increase in the loss value of the network’s
output and ultimately leading to incorrect model predictions. This has also
spurred the development of other similar methods [30, 17, 21, 34, 36, 13]. BIM
[17], through multiple iterations, introduces small perturbations along the
direction of increasing gradients, resulting in more accurate perturbations
compared to FGSM. Among them, Projected Gradient Descent (PGD) [21] deserves
special attention. Unlike the fast attack method of FGSM that operates with a
single iteration, PGD is an iterative attack method. It generates stronger
adversarial samples by performing multiple iterations, allowing it to bypass
certain defense mechanisms.

###  2.3 User safeguarding through image cloaking

Image cloaking is a widely researched field due to its importance in
maintaining privacy and preventing misuse of images. Pixelization and blurring
are commonly used techniques for hiding personal information such as faces and
license plates. With the continuous development of the T2I model, people are
paying attention to its remarkable image generation capabilities while also
being wary of the potential risks of its misuse. Therefore, when faced with
malicious attacks on images, we should take measures to prevent their success.
For T2I, our goal is to introduce imperceptible perturbations into pre-
existing images before their release. Once these images are reused, the model
will generate images with negative effects. [31, 29, 35, 20, 4] attempt to
prevent images from being edited or exploited.

Similar to our objective, Anti-DreamBooth [32] and Mist [18] aim to disrupt
the generative modeling quality by adding subtle perturbations to images. Our
work differs from their method of adding perturbations to images while keeping
the parameters unchanged. In our approach, we train and update the parameters
of the cross-attention layers of the model. By updating these parameters, we
introduce perturbations to the image.

##  3 Method

Figure 3: Schematic of CAAT attacking a T2I diffusion model. During attacker
training, first, $W_{K}$ and $W_{V}$ of the cross-attention layer are
optimized. Then, the perturbation $\delta$ is optimized based on the gradient
of $x$, yielding perturbed image $x+\delta$.

Figure 3 illustrates an overview of our CAAT approach that leverages PGD
training on cross-attention layer to effectively attack LDMs. We introduce the
principles of diffusion models and adversarial attacks in Sec. 3.1 and Sec.
3.2. Furthermore, CAAT is proposed in detail in Sec. 3.3.

###  3.1 Diffusion models

In the current field of Text-to-Image (T2I), diffusion models have established
themselves as the reigning champions, capable of generating diverse and
realistic images. Diffusion models include two processes: a forward process
and a backward process. The forward process gradually introduces noise into
the input image until the data distribution becomes pure Gaussian noise, while
the backward process learns in reverse, extracting the desired data from
random noise. Given the input image $x_{0}\sim q(x)$, forward process injects
noise into $x_{0}$ over $T$ steps, resulting in a Markov chain
${x_{1},x_{2},x_{3},...,x_{T}}$, each $x_{t}$ satisfies:

$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon,$ (1)

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ and $\alpha_{t}$ is obtained by
a noise scheduler.

Given noise image $x_{t}$ in time $t$, backward process learn to denoise the
noise image to obtain $x_{t-1}$. The training objective of diffusion models
can be expressed succinctly as follows:

$\mathcal{L}_{DM}(\theta,x_{0})=\mathbb{E}_{x_{0},t,\epsilon}\|\epsilon-\epsilon_{\theta}(x_{t},t)\|,$
(2)

where $\epsilon_{\theta}$ is a parametric neural network.

The prompt-based diffusion models, such as LDMs, have an additional prompt $c$
to generate images that better match the text description. After undergoing
encoding processing, the prompt $c$ is mapped to the intermediate layers in
the U-Net of LDMs through the introduction of cross-attention layers,
achieving the mapping from text to images:

Table 1: Effectiveness assessment with four evaluation metrics by comparing
different attackers on different T2I diffusion models. Bold is the best score
and underlining is the second-best score. CAAT achieved 12 best and 3 second-
best out of 16 metrics, demonstrating its superior attack effectiveness. The
abbreviation for “Anti-dreambooth” is denoted as “Anti”.

| T2I diffusion models
---|---
attack | Custom Diffusion | DreamBooth | SVDiff | Textual Inversion
FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$
clean | 1.00 | 0.52 | 0.47 | 195 | 0.98 | 0.52 | 0.53 | 179 | 0.99 | 0.57 | 0.68 | 218 | 1.00 | 0.47 | 0.27 | 242
Anti | 1.00 | 0.48 | 0.21 | 207 | 0.81 | 0.40 | 0.16 | 307 | 0.94 | 0.38 | 0.46 | 308 | 0.50 | 0.15 | -0.92 | 378
Mist | 1.00 | 0.39 | 0.03 | 233 | 0.99 | 0.32 | 0.13 | 275 | 0.88 | 0.21 | -0.12 | 317 | 0.63 | 0.14 | -0.85 | 348
CAAT | 1.00 | 0.42 | -0.36 | 250 | 0.64 | 0.32 | -0.14 | 371 | 0.85 | 0.34 | 0.29 | 355 | 0.43 | 0.14 | -1.30 | 396

Table 2: Effectiveness analysis across varying noise budgets for CAAT on
selected T2I diffusion models, where “*” denotes the default budget setting.

| T2I diffusion models
---|---
$\eta$ | Custom Diffusion | DreamBooth | SVDiff | Textual Inversion
FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$
0.05 | 1.0 | 0.45 | -0.27 | 225 | 0.95 | 0.44 | 0.44 | 251 | 0.98 | 0.44 | 0.51 | 307 | 0.83 | 0.34 | -0.17 | 302
*0.10 | 1.0 | 0.42 | -0.36 | 250 | 0.64 | 0.32 | -0.14 | 371 | 0.85 | 0.34 | 0.29 | 355 | 0.51 | 0.14 | -1.30 | 396
0.15 | 0.95 | 0.38 | -0.24 | 284 | 0.34 | 0.28 | -0.58 | 401 | 0.76 | 0.29 | 0.15 | 390 | 0.50 | 0.09 | -0.90 | 405

$\mathcal{L}_{LDM}(\theta,x_{0})=\mathbb{E}_{x_{0},t,\epsilon,c}\|\epsilon-\epsilon_{\theta}(x_{t},t,c)\|.$
(3)

###  3.2 Adversarial attack

Adversarial attack, now prevalent in various domains, represents a
sophisticated and pervasive class of attack methods in the contemporary
digital landscape. It were initially introduced for targeting classification
models whose attacks leverage carefully crafted inputs with the aim of
deceiving, misleading, or undermining classification models, thereby
compromising their performance or inducing misclassifications. In short, it
finds an alternative input $x^{\prime}$ for a given input $x$ and its label
that causes it not to be classified as its true label which can be formulated
by

$\displaystyle x^{\prime}$
$\displaystyle:=\arg\max_{x^{\prime}}\mathcal{L}_{\theta}(x^{\prime}),$ (4)
$\displaystyle s.t.$ $\displaystyle\quad||x-x^{\prime}||\leq\eta,$

where $\mathcal{L}_{\theta}$ is a classification network and $\eta$ is a small
positive constant, ensuring that $x^{\prime}$ does not deviate too far from
$x$.

###  3.3 CAAT

It is critical to analyze the vulnerability of DMs from the perspectives of
both effectiveness and efficiency. CAAT is proposed based on this goal and
developed upon PGD attack. To prevent the misuse of customized diffusion
models, we aim to obtain adversarial examples that, when used as inputs for
customized LDMs models, result in the fine-tuned model losing the ability to
generate images corresponding to specific themes, thereby disrupting the
quality of the T2I generate images. To achieve this, we introduce a
perturbation $\delta$ into the input image $x$, which is visually
imperceptible controlled by $\eta$, making it impossible for the model to
learn useful information during training. The overall objective of CAAT can be
formulated by

$\displaystyle\delta$
$\displaystyle:=\arg\max_{\delta}\mathcal{L}_{LDM}(\theta,x+\delta),$ (5)
$\displaystyle\quad\text{where}~{}\|\delta\|\leq\eta.$

A classic and practical adversarial attack method is the PGD attack, which is
applied to Anti-DreamBooth and Mist. PGD is applied to a trained model,
obtaining gradients during the attack process without updating model
parameters. Different from this convention, we optimize LDMs during the CAAT
training. By this means, LDMs is trained on adversarial examples $x+\delta$ to
enhance its robustness. Simultaneously, adversarial examples are applied to a
more robust LDMs, leading to the generation of adversarial examples with
improved attack effectiveness. The LDMs training process of CAAT can be
formulated by

$\displaystyle\theta$
$\displaystyle:=\arg\min_{\theta}\mathcal{L}_{LDM}(\theta,x+\delta),$ (6)
$\displaystyle\quad\text{where}~{}\|\delta\|\leq\eta.$

We leverage the observations and best practices in Custom Diffusion [16] to
analyze and select the layers for efficient attack. During the fine-tuning of
diffusion models, cross-attention layers have the fewest parameters but
undergo the most changes. This observation indicates cross-attention layer
plays a significant role in model optimization during the training process.
Conventional attention between images and texts can be formulated as follows:

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot
V,$ (7)

where $Q=W_{Q}\mathbf{f},K=W_{K}\mathbf{c},V={W_{V}}\mathbf{c}$. Here,
$\mathbf{f}\in\mathbb{R}^{(h\times w)\times l}$ is image features,
$\mathbf{c}\in\mathbb{R}^{s\times d}$ is features of prompt and $W_{Q}$,
$W_{K}$, and $W_{V}$ are learnable matrices that respectively map the input to
query, key and value. $W_{Q}$ processes the image input features, while
$W_{K}$ and $W_{V}$ handle the text input features. Disrupting the learning of
$W_{K}$ and $W_{V}$ can undermine the mapping between text and images in
customized fine-tuned diffusion models. After this undermining, the model is
capable of recognizing or acknowledging the content or subject of the image,
but it lacks the ability to categorize or classify it into specific groups or
types. Therefore, we update the parameters $W_{K}$ and $W_{V}$ of cross-
attention layers.

In summary, during the CAAT process, we freeze the model parameters other than
$W_{K}$ and $W_{V}$ and only update them to facilitate the learning of the
mapping between text input and image input by the model. Simultaneously, we
search for a perturbation $\delta$ in images $x$ that causes the model to lose
the aforementioned capabilities. (The reasons for choosing to simultaneously
update parameters and add noise are discussed in detail in Supplementary
Material D). The search process is implemented by calculating the gradients
for $x$ and performing gradient ascent. The algorithm is introduced in
Algorithm 1.

Algorithm 1 CAAT

1:Images $x$, K layers parameter $W_{K}$, V layers parameter $W_{V}$, step
length $\alpha$, limitation $\eta$, steps number $N$, LDMs learning rate $l$

2:Perturbed images $x^{\prime}$

3:Initialize $\delta$

4:for $i=1\to N$ do

5:
$\nabla_{K},\nabla_{V},\nabla_{x}\leftarrow\mathcal{L}_{LDM}((W_{K},W_{V}),x+\delta)$

6: $W_{K}\leftarrow W_{K}-l\nabla_{K}$

7: $W_{V}\leftarrow W_{V}-l\nabla_{V}$

8: $\delta\leftarrow\delta+\alpha\mathrm{sgn}\nabla_{x}$$\triangleright$
$\nabla_{x}$ is from the input images.

9: if $||\delta||>\eta$ then

10: $\delta\leftarrow clip(\delta,-\eta,\eta)$$\triangleright$ limit
$||\delta||$ within [0, $\eta$]

11: end if

12:end for

13:$x^{\prime}\leftarrow x+\delta$

14:return $x^{\prime}$

##  4 Experiments

Table 3: Effectiveness evaluation across different LDMs versions on different
T2I diffusion models. CAAT is trained on SD v2.1.

version | attack | T2I diffusion models
---|---|---
Custom Diffusion | DreamBooth | SVDiff | Textual Inversion
FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$
v1.4 | clean | 0.97 | 0.53 | 0.21 | 224 | 0.96 | 0.47 | 0.50 | 197 | 0.93 | 0.50 | 0.23 | 248 | 0.97 | 0.41 | 0.17 | 248
CAAT | 0.79 | 0.45 | -0.24 | 313 | 0.65 | 0.27 | -0.34 | 350 | 0.16 | 0.25 | -1.57 | 447 | 0.15 | 0.07 | -1.94 | 501
v1.5 | clean | 0.96 | 0.54 | 0.27 | 217 | 0.97 | 0.46 | 0.41 | 199 | 0.97 | 0.52 | 0.26 | 233 | 0.90 | 0.41 | -0.07 | 259
CAAT | 0.88 | 0.46 | -0.03 | 284 | 0.49 | 0.31 | -0.50 | 364 | 0.12 | 0.24 | -1.41 | 425 | 0.31 | 0.09 | -1.48 | 441

Table 4: Effectiveness assessment by varying the number of perturbed images.
The results demonstrate the proportion of perturbed images obtained by CAAT
that affect the image quality of the T2I diffusion models, considering four
input images.

Clean | Perturbed | T2I diffusion models
---|---|---
Custom Diffusion | DreamBooth | SVDiff | Textual Inversion
FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$ | FR $\downarrow$ | FS $\downarrow$ | IR $\downarrow$ | FID $\uparrow$
4 | 0 | 1.00 | 0.52 | 0.47 | 195 | 0.98 | 0.52 | 0.53 | 179 | 0.99 | 0.57 | 0.68 | 218 | 1.00 | 0.47 | 0.27 | 242
3 | 1 | 1.00 | 0.50 | 0.16 | 202 | 0.97 | 0.50 | 0.53 | 197 | 0.98 | 0.53 | 0.61 | 234 | 1.00 | 0.44 | 0.24 | 252
2 | 2 | 1.00 | 0.47 | 0.09 | 207 | 0.91 | 0.50 | 0.26 | 249 | 0.94 | 0.47 | 0.60 | 247 | 0.88 | 0.37 | -0.06 | 281
1 | 3 | 1.00 | 0.45 | -0.15 | 228 | 0.76 | 0.44 | 0.10 | 301 | 0.93 | 0.39 | 0.48 | 294 | 0.71 | 0.24 | -0.44 | 314
0 | 4 | 1.00 | 0.42 | -0.36 | 250 | 0.64 | 0.32 | -0.14 | 371 | 0.85 | 0.34 | 0.29 | 355 | 0.43 | 0.14 | -1.30 | 396

Figure 4: Comparison in the images generated by different T2I diffusion models
with different attackers. The first column illustrates the four input images.
For attackers by row, the observation of the perturbation pattern can refer to
Fig. 5. For diffusion models by column, four models are selected and compared
to evaluate the performance of attackers. Figure 5: Adversarial examples of
different attacker after adding noise. The parameter configurations of Anti-
DreamBooth and Mist follow the default settings in Tab. A1.

In this section, we valuate the effectiveness of CAAT on customized LDMs
through experiments. Specifically, we compare CAAT with other attack methods
across various DMs to evaluate the effectiveness, generalization, and
efficiency of CAAT.

###  4.1 Experimental setup

Datasets. To justify our proposed CAAT attacker, the testing datasets should
comply the following criteria: 1) an ample supply of face images, 2)
categorized based on individuals, and 3) high-quality images with high
resolution. In accordance with these criteria and taking inspiration from
Anti-DreamBooth, we select the face datasets CelebA-HQ [14]. CelebA-HQ is the
high-resolution version of CelebA [19], which includes 10,177 unique celebrity
identities and 202,599 face images. CelebA-HQ, on the other hand, contains
over 30,000 high-resolution ($1024\times 1024$) face images from more than
1,000 different celebrities. For our evaluation, we utilize a subset [3] of
CelebA-HQ with 307 subjects that have been properly categorized.

Data Preprocessing. Due to the extensive comparative content, we opt to use 10
subjects from this subset as our experimental subjects, ensuring diversity in
terms of gender, ethnicity, and age. For each subject, four photos are
selected and processed into $512\times 512$ resolution.

Comparative attackers. We compare CAAT with other attackers that are also
applied in DMs, including Anti-DreamBooth (aspl) and Mist, which aims to
protect users’ portrait rights. In particular, to ensure fairness, the same
four images are used as both the training and attacked images for Anti-
DreamBooth (aspl). CAAT is compared with these two existing methods to
evaluate their strengths and weaknesses.

Attacked model selection. For customized fine-tuned models based on LDMs, four
practical and popular ones, including Text Inversion, DreamBooth, Custom
Diffusion, and SVDiff, are selected as the target models to achieve the
adversarial examples through the attackers. This selection exhibits diversity
and state-of-the-art performance. Successfully performing attacking on them
can demonstrate that our CAAT can be effective on all LDMs-based fine-tuned
models. Additionally, we compare the effectiveness of the attack on the
variants of stable diffusion.

Evaluation metrics. The quality of the generated face images is evaluated
using the face detection and recognition model provided by InsightFace [12].
All the generated images are subjected to face detection to obtain the success
rate of face detection called Face Detection Success Rate (FR). For the
generated images with detected faces, the average face similarity with four
clean images is calculated, which is called Face Similarity (FS). FS takes
values in the range of $[0,1]$, indicating the quality of generated images.
Higher FS values signify lower face similarity and better attack effect.
Moreover, ImageReward (IR) [33] is employed to compute the T2I generated image
quality. It requires prompts corresponding to generated images, and evaluates
the quality of images generated based on the prompts. Lower IR values signify
lower image quality and better attack effect. Finally, we use Fréchet
Inception Distance (FID) [10] to measure the similarity between generated
images and clean images. Higher FID indicates that the generated images are
farther from clean images, signifying better attack effect.

Training setup. We exclusively train CAAT on the $W_{K}$ and $W_{V}$ of cross-
attention layers, using a batch size of 1 and a learning rate of $1\times
10^{-5}$ for 250 training steps. Mixed precision with bf16 is employed. The
attack prompt provided is “a photo of a person”. By default, we use the latest
Stable Diffusion (v2.1) as the pretrained generator and set $\alpha$ to
$5\times 10^{-3}$ for CAAT, along with $\eta=0.1$. Training CAAT with 500
steps on an NVIDIA RTX3090 takes approximately 2 minutes. We also summarize
the hyperparameters for other attackers in Tab. A1 and DMs in Tab. A2, all of
which are set to default values.

###  4.2 T2I generation

First, the clean images are input into CAAT, Anti-DreamBooth, and Mist to
obtain perturbed images (adversarial examples). Next, both the perturbed
images and clean images undergo customized fine-tuning with Custom Diffusion,
DreamBooth, SVDiff, and Textual Inversion. After the fine-tuning process, we
generate 16 images with the prompt “a photo of a person” using Stable
Diffusion (v2.1). The experimental results are presented in Tab. 1, while
visual representations of some results are shown in Fig. 4. As observed, CAAT
successfully attacks all the models, yielding the best results for DreamBooth,
Textual Inversion, and Custom Diffusion, and the second-best result for
SVDiff. Although CAAT may not achieve optimal results across all evaluation
metrics, the obtained values are already very low and visually imperceptible.
Furthermore, both Anti-DreamBooth and Mist exhibit poor attack results on
Custom Diffusion, underscoring CAAT’s superior generalization capability.
Additionally, while Mist achieves decent results, its added perturbation is
more visually discernible, as evident in Fig. 5. Moreover, we conducted
additional experiments in Supplementary Material B with different prompts and
subjects.

###  4.3 Computational overhead

We conducted analysis on computational overhead. The experiments were carried
out by comparing CAAT, Mist, and Anti-DreamBooth under the same training
setting. Figure 6 demonstrates our outstanding performance in terms of time
efficiency. Training time of our method CAAT is about 2 minutes and 30 seconds
on an NVIDIA RTX3090, compared to about 5 minutes and 30 seconds for Anti-
DreamBooth and about 5 minutes for Mist on same GPU. CAAT is approximately
twice faster than the other two. More importantly, CAAT does not require prior
class images, but Anti-DreamBooth requires 200 images by default, which
indicates that CAAT saves more cost.

Figure 6: The training time of the attackers under the default settings of
CAAT, Mist, and Anti-DreamBooth (Anti).

###  4.4 Ablation study

We conduct ablation study to analyze the effect of CAAT. The experiments are
carried by varying perturbation budgets, T2I diffusion models, and the
quantity of perturbed images.

Perturbation budgets. We study the impact of different perturbation strengths
when applying CAAT on the quality of T2I images. In our previous experiments,
we set $\eta$ = 0.10, and now explore the effects of $\eta$ = 0.05 and $\eta$
= 0.15. The results are presented in Tab. 2. It can be observed that a larger
$\eta$ leads to poorer T2I image quality, but excessively high $\eta$ settings
introduce visually perceptible noise in the adversarial samples. Fig. 5
visually presents perturbed images generated by different attack methods. When
using the default settings $\eta=0.10$, CAAT demonstrates superior attack
effectiveness (as Tab. 1), with a level of noise similar to Anti-DreamBooth
and less noise compared to Mist. However, when $\eta=0.15$, the perturbed
images exhibit excessive noise.

T2I diffusion model variants. It is essential to conduct the performance of
the adversarial examples generated by CAAT on different versions of T2I
diffusion models. In previous experiments, we apply CAAT on Stable Diffusion
v2.1 for both the attack and T2I image generation. Additionally, we conducted
experiments to assess the performance of CAAT’s samples on Stable Diffusion
v1.4 and v1.5, as shown in Tab. 3. CAAT demonstrates robust performance across
different versions of Stable Diffusion, highlighting its strong generalization
capabilities.

Quantity of perturbed images. To simulate real-world scenarios where malicious
attackers may obtain some clean images, we examine the impact of different
proportions of perturbed images in the case of four input images. As indicated
in Tab. 4, the results demonstrate that more disturbed images lead to a more
effective attack. CAAT consistently exhibits a robust attack effect,
particularly with two or more perturbed images, whereas the impact is less
pronounced with one or fewer perturbed images.

###  4.5 Robustness of CAAT

In real-world usage scenarios, images are easy to distortion, such as lossy
compression or deformation. To verify if CAAT can handle complex real-world
situations, we applied a variety of image perturbation methods in
Supplementary Material C to demonstrate the robustness of CAAT.

##  5 Conclusions

We introduce a simple yet effective adversarial attack, CAAT, designed to
protect users’ portrait rights from infringement in the context of customized
diffusion model fine-tuning. Users can employ CAAT to add imperceptible
perturbations to images before publishing them, rendering malicious attackers
unable to generate convincing fake images using these altered images. Our key
idea is to introduce perturbations during the training of the cross-attention
layers to disrupt the mapping between text and images. We conducte extensive
experiments on DreamBooth, Textual Inversion, SVDiff, and Custom Diffusion,
comparing the results to other attacks like Anti-DreamBooth and Mist. The
result underscore the notable effectiveness and superior generalization
capabilities of CAAT. Its ability to pre-process images for social media posts
provides users with a powerful tool to fortify their portrait rights and
protect against unauthorized image manipulations.

## Acknowledgements

This research was supported by the National Natural Science Foundation of
China (No.62376023) and the Fundamental Research Funds for the Central
Universities (Grant No.2022RC07X).

## References

  * Balaji et al. [2022] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  _arXiv preprint arXiv:2211.01324_ , 2022.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * [3] CelebA-HQ-Face-Identity-and-Attributes-Recognition-PyTorch.  https://github.com/ndb796/CelebA-HQ-Face-Identity-and-Attributes-Recognition-PyTorch.
  * Cui et al. [2023] Yingqian Cui, Jie Ren, Yuping Lin, Han Xu, Pengfei He, Yue Xing, Wenqi Fan, Hui Liu, and Jiliang Tang.  Ft-shield: A watermark against unauthorized fine-tuning in text-to-image diffusion models.  _arXiv preprint arXiv:2310.02401_ , 2023.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Goodfellow et al. [2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Han et al. [2023] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  _arXiv preprint arXiv:2303.11305_ , 2023.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * [12] Insightface.  https://insightface.ai/.
  * Jiang et al. [2019] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao.  Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  _arXiv preprint arXiv:1911.03437_ , 2019.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Kurakin et al. [2018] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  In _Artificial intelligence safety and security_ , pages 99–112. Chapman and Hall/CRC, 2018.
  * Liang et al. [2023] Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, XUE Zhengui, Ruhui Ma, and Haibing Guan.  Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples.  2023\.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pages 3730–3738, 2015.
  * Ma et al. [2023] Yihan Ma, Zhengyu Zhao, Xinlei He, Zheng Li, Michael Backes, and Yang Zhang.  Generative watermarking against unauthorized subject-driven image synthesis.  _arXiv preprint arXiv:2306.07754_ , 2023.
  * Madry et al. [2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. [2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salman et al. [2023] Hadi Salman, Alaa Khaddaj, Guillaume Leclerc, Andrew Ilyas, and Aleksander Madry.  Raising the cost of malicious ai-powered image editing.  _arXiv preprint arXiv:2302.06588_ , 2023.
  * Shafahi et al. [2019] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein.  Adversarial training for free!  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Shan et al. [2023] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao.  Glaze: Protecting artists from style mimicry by $\\{$Text-to-Image$\\}$ models.  In _32nd USENIX Security Symposium (USENIX Security 23)_ , pages 2187–2204, 2023.
  * Van Le et al. [2023] Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran.  Anti-dreambooth: Protecting users from personalized text-to-image synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2116–2127, 2023.
  * Xu et al. [2023] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  _arXiv preprint arXiv:2304.05977_ , 2023.
  * Zhang et al. [2019] Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong.  You only propagate once: Accelerating adversarial training via maximal principle.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Zhao et al. [2023] Zhengyue Zhao, Jinhao Duan, Xing Hu, Kaidi Xu, Chenan Wang, Rui Zhang, Zidong Du, Qi Guo, and Yunji Chen.  Unlearnable examples for diffusion models: Protect data from unauthorized exploitation.  _arXiv preprint arXiv:2306.01902_ , 2023.
  * Zhu et al. [2019] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu.  Freelb: Enhanced adversarial training for natural language understanding.  _arXiv preprint arXiv:1909.11764_ , 2019.

Supplementary Material

##  Appendix A Hyperparameters

Table A1: Hyperparameters for different attackers.The parameters for Anti-DreamBooth (aspl) and Mist are set to their default configurations. Anti. denotes Anti-DreamBooth.  parameters | CAAT | Anti | Mist
---|---|---|---
train steps | 250 | 50 | 100
learning rate | $1\times 10^{-5}$ | $5\times 10^{-7}$ | -
$\alpha$ | $5\times 10^{-3}$ | $5\times 10^{-3}$ | $2/255$
$\eta$ | 0.1 | 0.05 | $32/255$
Table A2: Hyperparameters of diffusion models, which follow their default
configurations. CD, BD and TI denote Custom Diffusion, DreamBooth, and Textual
Inversion, respectively.

parameters | CD | DB | SVDiff | TI
---|---|---|---|---
train steps | 250 | 1000 | 500 | 1500
learning rate | $1\times 10^{-5}$ | $5\times 10^{-7}$ | $1\times 10^{-3}$ | $5\times 10^{-4}$
batchsize | 2 | 1 | 1 | 1

##  Appendix B More Task

We studied the effects of different prompts on different subjects (e.g., barn,
dogs, and toy). The results in Fig. B1 show no influence and that CAAT can
consistently degrade the quality of generated images (first two lines have
huge noise) and disrupt subject learning ability (the subjects of the last two
lines are inconsistent).

Figure B1: The images generated by different T2I diffusion models with
different prompts and tasks. $S^{*}$ denotes special token of different T2I
models.

##  Appendix C Robustness

We conducted more experiments with four image perturbation methods in Tab. C1
to demonstrate the robustness of CAAT. We used:

  * •

Random noise has a scale of 0.05.

  * •

Quantization involves reducing an 8-bit image to a 6-bit image.

  * •

Gaussian blur uses a kernel size of 3x3 with $\sigma$ set to 0.05.

  * •

JPEG image processing is implemented using the OpenCV2 library.

Table C1: Robustness assessment by different image perturbation methods. Bold
is the best score.

Method | T2I generation models
---|---
Custom Diffusion | DreamBooth
FS$\downarrow$ |  FC$\downarrow$ |  IR$\downarrow$ |  FID$\uparrow$ |  FS$\downarrow$ |  FC$\downarrow$ |  IR$\downarrow$ |  FID$\uparrow$
clean | 1.0 | 0.52 | 0.47 | 195 | 0.98 | 0.52 | 0.53 | 179
CAAT | 1.0 | 0.42 | -0.36 | 250 | 0.64 | 0.32 | -0.14 | 371
random noise | 1.0 | 0.42 | -0.10 | 222 | 0.97 | 0.42 | 0.31 | 270
quantization | 1.0 | 0.44 | -0.15 | 202 | 0.99 | 0.45 | 0.42 | 201
JPEG | 1.0 | 0.40 | -0.20 | 218 | 0.80 | 0.36 | 0.10 | 328
Gaussian blur | 1.0 | 0.40 | -0.25 | 229 | 0.75 | 0.33 | -0.03 | 347

##  Appendix D Separation _vs_. simultaneous

When updating parameters and adding noise, we considered doing both
simultaneously (See Sec. 3.3) versus separately, aiming to find a superior
method. For the latter, We alternated between 10 steps of model parameter
updates and 10 steps of PGD , each for 250 steps (same as CAAT), with the
results shown in Tab. D1. The experimental results indicate that both
optimization methods achieved sufficiently good results, making it difficult
to compare them. Moreover, for N-step training, simultaneous optimization
requires N backward steps since we can reuse the gradients for attacking,
while separation requires 2N. The goal of CAAT is to be lightweight and fast,
introducing extra overhead is contrary to our philosophy. Therefore, we
carried out the optimizations simultaneously.

Table D1: Comparison of simultaneous optimization and separation optimization.
Separated involves alternating model optimization and adding noise.

Method | T2I generation models
---|---
Custom Diffusion | DreamBooth
FS$\downarrow$ |  FC$\downarrow$ |  IR$\downarrow$ |  FID$\uparrow$ |  FS$\downarrow$ |  FC$\downarrow$ |  IR$\downarrow$ |  FID$\uparrow$
clean | 1.0 | 0.52 | 0.47 | 195 | 0.98 | 0.52 | 0.53 | 179
CAAT | 1.0 | 0.42 | -0.36 | 250 | 0.64 | 0.32 | -0.14 | 371
Separated | 0.99 | 0.39 | -0.25 | 238 | 0.72 | 0.32 | -0.10 | 330