# CGI-DM: Digital Copyright Authentication for Diffusion Models via
Contrasting Gradient Inversion

Xiaoyu Wu1, Yang Hua2, Chumeng Liang3, Jiaru Zhang1, Hao Wang4, Tao
Song1111Corresponding Author, Haibing Guan1
Shanghai Jiao Tong University1, Queen’s University Belfast2,
University of Southern California3, Louisiana State University4
{wuxiaoyu2000, jiaruzhang, songt333<EMAIL_ADDRESS><EMAIL_ADDRESS>,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Diffusion Models (DMs) have evolved into advanced image generation tools,
especially for few-shot generation where a pretrained model is fine-tuned on a
small set of images to capture a specific style or object. Despite their
success, concerns exist about potential copyright violations stemming from the
use of unauthorized data in this process. In response, we present Contrasting
Gradient Inversion for Diffusion Models (CGI-DM), a novel method featuring
vivid visual representations for digital copyright authentication. Our
approach involves removing partial information of an image and recovering
missing details by exploiting conceptual differences between the pretrained
and fine-tuned models. We formulate the differences as KL divergence between
latent variables of the two models when given the same input image, which can
be maximized through Monte Carlo sampling and Projected Gradient Descent
(PGD). The similarity between original and recovered images serves as a strong
indicator of potential infringements. Extensive experiments on the WikiArt and
Dreambooth datasets demonstrate the high accuracy of CGI-DM in digital
copyright authentication, surpassing alternative validation techniques. Code
implementation is available at https://github.com/Nicholas0228/Revelio.

22footnotetext: Correspondence to Tao Song (songt333@sjtu.edu.cn).

##  1 Introduction

Figure 1: Top: Infringements brought by few-shot generation. A small set of
images is used to fine-tuning the pretrained model. The fine-tuned model is
then capable of high-quality generation, which, if performed without proper
authorization, may lead to infringements. Bottom: Our method CGI-DM for
copyright authentication. CGI-DM recovers the missing details from partial
representation of an input sample. Then the the similarity between recovered
samples and input samples can be used to validate infringements.

Recent years have witnessed the advancement of Diffusion Models (DMs) in
computer vision. These models demonstrate exceptional capabilities across a
diverse array of tasks, including image editing [14], and video editing [35],
among others. Particularly, the emergence of few-shot generation techniques,
exemplified by Dreambooth [24] and Lora [13], has considerably reduced the
costs associated with replicating artwork and transferring art styles, all the
while maintaining an exceedingly high standard of quality. As illustrated in
Fig. 1 (Top), these methods focus on swiftly capturing the style or primary
objects by fine-tuning a pretrained model on a small set of images. This
process enables efficient and high-quality art imitation and art style
transfer by utilizing the fine-tuned model.

However, these advanced few-shot generation techniques also spark widespread
concerns regarding the protection of copyright for human artworks and
individual photographs. There is a growing fear that parties may exploit the
generative capabilities of DMs to create and profit from derivative artworks
based on existing artists’ works, without obtaining proper authorization [19,
6]. Concurrently, concerns arise regarding the creation of fabricated images
of individuals without their consent [32]. All of these collectively pose a
serious threat to the security of personal data and intellectual property
rights.

To address these critical concerns, a line of approaches focuses on
safeguarding individual images by incorporating adversarial attacks, such as
AdvDM [17], Glaze [25], and Anti-Dreambooth [30]. The adversarial attacks can
disrupt the generative output, rendering the images unlearnable by diffusion
models. These methods are implemented ahead of the fine-tuning process, and as
such, we consider them as precaution approaches.

Another line of approaches facing such threats is copyright authentication.
Copyright authentication compares the similarity between the output images of
diffusion models and the given images to validate unauthorized usage. Such a
process can serve as legal proof for validating infringement (See Appendix A
for more details), and has been utilized as evidence in ongoing legal cases
concerning violations enabled by DMs [31]. This process happens after the
fine-tuning and thus we consider it as a post-caution approach. However,
current copyright authentication methods face difficulties in producing output
images closely resembling training samples due to the pursuit of diversity in
generative models. Consequently, it becomes difficult to ascertain whether a
particular training sample has been utilized solely based on the generated
output of the model for post-caution methods.

In this paper, we propose a new copyright authentication framework, named
Contrasting Gradient Inversion for Diffusion Models (CGI-DM) to greatly
improve the efficacy of the post-caution path, illustrated in Fig. 1 (Bottom).
Recent advances in gradient inversion [38, 10, 3] emphasize the importance of
prior information in data extraction. Inspired by this, we propose first
removing half of a given image. Then we utilize the retained partial
representation as a prior and employ gradient inversion to reconstruct the
original image. As recent studies [1, 27] indicate that generative models tend
to “memorize”, the recovery of removed information should be possible only
when the images are utilized during the fine-tuning process, enabling
“memorization” on given samples. Thus, a high similarity between the recovered
image and the original image can indicate that the model has been trained with
the given image.

However, directly applying gradient inversion may not yield useful information
for DMs, possibly because they inherently eliminate noise (see Appendix C for
details). To address this issue, we focus on contrasting two models: the
pretrained and the fine-tuned model. Specifically, our goal is to leverage the
conceptual differences between these two models. We measure this disparity
through the KL divergence between the latent variable distributions of the
pretrained and fine-tuned models. Subsequently, we provide a proof
demonstrating that maximizing this KL divergence approximates accentuating the
loss differences between the two models. Building on this, we employ Monte
Carlo Sampling on the noise and the time variable during the diffusion
process, utilizing PGD [18, 17] to optimize the aforementioned loss
difference. Comprehensive experiments are conducted on artists’ works and
objects, addressing the potential for unauthorized style transfers and the
creation of fabricated images, both of which necessitate copyright
authentication. The experiment results affirm the effectiveness and robustness
of our approach.

In summary, our contributions are as follows:

  * •

We have formulated a novel post-caution approach for copyright
protection—copyright authentication. This method complements precaution
measures and provides robust legal proof of infringements.

  * •

We propose a new framework, CGI-DM, for authentication. Utilizing Monte Carlo
sampling and PGD optimization, we employ gradient inversion based on the
partial representation of a given image. The similarity between the recovered
samples and original samples can serve as a robust and visible indicator for
infringements. Notably, while most gradient inversion methods focus on
classification models, we pioneer its application in the domain of generative
models, emphasizing a new approach.

  * •

We conduct extensive experiments on the WikiArt and Dreambooth datasets to
substantiate the efficacy and robustness of our approach in distinguishing
training samples from those not used during training. These demonstrate our
method’s effectiveness in authenticating digital copyrights and thus
validating infringements for both style mimicry and fabricated images.

Figure 2: Our framework, CGI-DM, for copyright authentication. We begin with a
given image $x_{0}$. ① We first remove part of $x_{0}$, obtaining a partial
representation $\overline{x}_{0}$ of the image (refer to Sec. 3.1). This
partial representation is then fed into the optimization loop. ② Within the
optimization loop, we leverage the conceptual disparity between the pretrained
model $\theta$ and the fine-tuned model $\theta^{\prime}$ given the partial
representation $\overline{x}_{0}$ (refer to Sec. 3.2) to recover the missing
details. Such disparity is formulated as $L_{tar}$. ③ Subsequently, we employ
Monte Carlo sampling on time variable $t$ and random noise $\varepsilon_{t}$
to optimize $L_{tar}$ (refer to Sec. 3.3), getting step-wise gradient for
updating the image. Over $\rm{N}$ steps of updating, the optimization loop
produces the final recovered image $\overline{x}_{0}^{(N)}$. ④ We authenticate
copyright by measuring the similarity between recovered image
$\overline{x}_{0}^{(N)}$ with the original image $x_{0}$.

##  2 Background

###  2.1 Diffusion Models

Diffusion models are latent variable models that take the form of
$p_{\theta}(x_{0}):=\int p_{\theta}(x_{0:T})dx_{1:T}$, where
$x_{1:T}:=x_{1},x_{2},\cdots,x_{t}$ represents the latent variables of the
same dimensionality as the data $x_{0}$ belonging to real data distribution
$q(x_{0})$: $x_{0}\sim q(x_{0})$. The joint distribution of these latent
variables is defined as a Markov chain:

$p_{\theta}(x_{0:T}):=p_{\theta}(x_{T})\prod_{t=1}^{N}p_{\theta}(x_{t-1}|x_{t}),$
(1)

where the transitions $p_{\theta}(x_{t-1}|x_{t})$ are defined as learned
Gaussian distributions:
$p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\sigma_{t}^{2}\textbf{I})$.
The initial step $p_{\theta}(x_{T})$ of the Markov chain is defined as a
normal distribution $p_{\theta}(x_{T})=\mathcal{N}(x_{T};0,\textbf{I})$.

One of the most distinguishing properties of the diffusion model is that it
leverages an approximate posterior $q(x_{1:T}|x_{0})$ in the so-called
diffusion process, which is also a Markov chain and can be simplified as:
$q(x_{t}|x_{0})=\mathcal{N}(x_{t};\sqrt{\alpha_{t}}x_{0},\sqrt{1-\alpha_{t}}\textbf{I})$.
In other words, we can derive the latent variable $x_{t}$ directly:

$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\varepsilon_{t},$ (2)

where $x_{0}$ is the original image, and $\varepsilon_{t}\in\mathcal{N}(0,1)$
is the noise added at the current time $t$. Subsequently, in the so-called
denoising process, a noise-prediction model $\epsilon_{\theta}$, parameterized
by weights $\theta$, is employed to predict noise in the noisy image $x_{t}$.
The predicted noise, denoted as $\epsilon_{\theta}(x_{t},t)$, is then removed
to recover a clear image from the noisy one.

The training target of the diffusion model begins with maximizing the
aforementioned probability $p_{\theta}(x_{0})$ for all data points $x_{0}$
within the real data distribution. With a variational bound on the negative
log-likelihood and a series of simplifications [12], the final objective can
be transformed into the expectation of the mean squared error (MSE) loss on
the noise prediction error when the time variable $t$ and the noise
$\varepsilon_{t}$ added are sampled:

$\theta=\arg\min\limits_{\theta}\mathbb{E}_{t,\varepsilon_{t}\sim\mathcal{N}(0,1)}\|\varepsilon_{t}-\epsilon_{\theta}(x_{t},t)\|^{2}.\\\
$ (3)

###  2.2 Copyright Authentication

Copyright authentication aims to establish legal proof of infringements. By
utilizing the pretrained model $\theta$ and a fine-tuned model
$\theta^{\prime}$ fine-tuning on a training dataset $\rm{X}$, the objective is
to ascertain whether an image $x_{0}$ belongs to the training set $\rm{X}$ in
a way that is understandable to humans. When confirming inclusion in the
training set, indicating unauthorized usage, it is crucial to present a clear
and vivid representation as legal evidence. Therefore, the method for
copyright authentication needs possess two essential properties:

Accuracy. The method should accurately classify samples used during training
from those not used during training, preventing misleading outcomes.

Explainability and Visualizability. The method should be explainable and
visualizable, especially for vision-generative models, ensuring its
effectiveness as legal proof (refer to Appendix A for more details). This
implies that any metric used to distinguish between training and non-training
samples should be in line with human vision, avoiding direct combination with
the model’s loss function, which might be incomprehensible to humans.

##  3 Method

As depicted in Fig. 2, our method, CGI-DM, is based on several processes:
Initially, we remove part of the original image $x_{0}$, deriving partial
representation $\overline{x_{0}}$ . Subsequently, we strive to recover the
missing details by exploiting the conceptual differences between the
pretrained model $\theta$ and the fine-tuned model $\theta^{\prime}$. The
process begins with maximizing the KL divergence between the probability
distributions derived by the two models for latent variables of
$\overline{x}_{0}$, the optimization problem of which can be solved with Monte
Carlo Sampling. After optimization, the final output image
$\overline{x}_{0}^{(N)}$ is then compared with the original image $x_{0}$. A
high degree of similarity should be observed when the original image $x_{0}$
is used during the fine-tuning, while a significant discrepancy is expected
when it is not.

###  3.1 Removing Partial Information

The effectiveness of CGI-DM hinges significantly on the method employed to
derive partial information $\overline{x}_{0}$. Recognizing that the quality
and difficulty of recovering missing details depend on the type of information
removed—be it detailed, background, or structural—we explore various
techniques that consider these aspects during the partial information removal
process (See Fig. 4 in Sec. 4.4.1 for more details).

###  3.2 Exploiting Conceptual Difference

For a pretrained model $\theta$ and a fine-tuned model $\theta^{\prime}$, we
then aim to modify partial representation $\overline{x}_{0}$ of the training
example $x_{0}$ to best exploit the conceptual disparities between the two
models. Such a process can recover the missing details in $\overline{x}_{0}$
if the original sample $x_{0}$ is included in the training dataset $\rm{X}$.

We rely on the latent variable $\overline{x}_{1:T}$ of $\overline{x}_{0}$ and
utilize the KL divergence between the probabilities of the latent variables
$\overline{x}_{1:T}$ given $\overline{x}_{0}$ with respect to both models to
measure such conceptual difference:
$D_{\rm{KL}}(p_{\theta^{\prime}}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})||{p_{\theta}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})})$.
The process that gradually modifies change $\overline{x_{0}}$ to capture the
conceptual difference between the two models can be formulated as optimizing
the perturbation added to $\overline{x_{0}}$:

$\displaystyle\delta$
$\displaystyle:=\arg\max\limits_{\delta}D_{\rm{KL}}(p_{\theta^{\prime}}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})||{p_{\theta}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})}),$
(4) $\displaystyle\mbox{where
}\overline{x}_{0}^{\prime}=\overline{x}_{0}+\delta,\|\delta\|\leq\epsilon,\epsilon\mbox{
is}\mbox{ a constant }.$

Leveraging the property of Markov chain and the training process of diffusion
models, we show that such KL divergence can be transformed into a closed form
considering the difference between the latent variable and the mean value of
the probability function in forward denoising diffusion implicit model (DDIM)
[28, 15]:

$\displaystyle\arg\max\limits_{\delta}\mathbb{E}_{p_{\theta^{\prime}}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})}\log\frac{p_{\theta^{\prime}}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})}{p_{\theta}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})}$
(5) $\displaystyle\approx$
$\displaystyle\arg\max\limits_{\delta}\sum_{t=0}^{T-1}\mathbb{E}_{q(\overline{x}_{t+1}^{\prime}|\overline{x}_{t}^{\prime})}-\|\overline{x}_{t+1}^{\prime}-\mu_{p_{\theta^{\prime}}(\overline{x}_{t+1}^{\prime}|\overline{x}_{t}^{\prime})}\|^{2}$
$\displaystyle+\|\overline{x}_{t+1}^{\prime}-\mu_{p_{\theta}(\overline{x}_{t+1}^{\prime}|\overline{x}_{t}^{\prime})}\|^{2}.$

Proof for this approximation is available in Sec. B.1 and Sec. B.2. It is
notable that the difference between a given datapoint and its mean value of
probability function in forward DDIM is in fact the error for the noise
predictor $\epsilon_{\theta}$ at time $t$:

$\displaystyle\|\overline{x}_{t+1}^{\prime}-\mu_{p_{\theta}(\overline{x}_{t+1}^{\prime}|\overline{x}_{t}^{\prime})}\|^{2}$
(6) $\displaystyle\approx$
$\displaystyle(\frac{\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t+1}}}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t+1}})^{2}\|\varepsilon_{t}-\epsilon_{\theta}(\overline{x}_{t}^{\prime},t)\|^{2}.$

The details for this approximation are available in Sec. B.3. For brevity, we
omit the coefficient
$\frac{\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t+1}}}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t+1}}$
of the loss function at different times, in line with the common practice in
diffusion models [12, 29]. Consequently, our target can be transformed into:

$\displaystyle\delta$
$\displaystyle:=\arg\max\limits_{\delta}D_{\rm{KL}}(p_{\theta^{\prime}}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})||{p_{\theta}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})})$
(7)
$\displaystyle\approx\arg\max\limits_{\delta}\mathbb{E}_{t,\varepsilon_{t}\sim\mathcal{N}(0,1)}\underbrace{\|\varepsilon_{t}-\epsilon_{\theta}(\overline{x}_{t}^{\prime},t)\|^{2}-\|\varepsilon_{t}-\epsilon_{\theta^{\prime}}(\overline{x}_{t}^{\prime},t)\|^{2}}_{L_{tar}(\theta,\theta^{\prime},\overline{x}_{0}^{\prime},t,\varepsilon_{t})}.$

Intuitively, we establish that the KL divergence between the probability
distributions of a given sample, considering two distinct models, can be
reformulated as the discrepancy in the MSE loss between the noise prediction
error of the two models.

Importantly, our definition of conceptual differences and the provided proof
isn’t limited to our problem domain. For example, we demonstrate its
effectiveness in membership inference attack (MIA), as detailed in Appendix F.

###  3.3 Optimizing via Monte Carlo Sampling

Direct optimization of the target in Eq. 7 is not feasible due to the presence
of the gradient of expectation. Drawing inspiration from traditional
adversarial attacks like PGD [18] and recent work applying adversarial attacks
on diffusion models [17], we utilize the expectation of the gradient to
estimate the gradient of the expectation through Monte Carlo Sampling. For
each iteration, we sample a time $t$ and noise
$\varepsilon_{t}\in\mathcal{N}(0,1)$, and compute the gradient of the loss
function in Eq. 7. We then perform one step of optimization using this
gradient, which can be summarized as:

$\displaystyle\nabla_{\overline{x}_{0}}\mathbb{E}_{t,\varepsilon_{t}\sim\mathcal{N}(0,1)}L_{tar}(\theta,\theta^{\prime},\overline{x}_{0},t,\varepsilon_{t})$
(8) $\displaystyle\approx$
$\displaystyle\mathbb{E}_{t,\varepsilon_{t}\sim\mathcal{N}(0,1)}\nabla_{\overline{x}_{0}}L_{tar}(\theta,\theta^{\prime},\overline{x}_{0},t,\varepsilon_{t}).$

Following existing adversarial attacks [18], we impose an $L_{2}$ norm
constraint on each step. The sample at step $(t+1)$, denoted as
$\overline{x_{0}}^{(t+1)}$ is derived from the step-wise length $\alpha$, the
current gradient and the sample from the last step $\overline{x_{0}}^{(t)}$:

$\overline{x}_{0}^{(t+1)}=\overline{x_{0}}^{(t)}+\alpha\frac{\nabla_{\overline{x}_{0}^{(t)}}L_{tar}(\theta,\theta^{\prime},\overline{x}_{0}^{(t)},t,\varepsilon_{t})}{\|\nabla_{\overline{x}_{0}^{(t)}}L_{tar}(\theta,\theta^{\prime},\overline{x}_{0}^{(t)},t,\varepsilon_{t})\|_{2}}.$
(9)

Intuitively, our algorithm iteratively updates the input variable such that
for each sampled time variable $t$ and noise $\varepsilon_{t}$, the estimated
noise is much more accurate for the fine-tuned model $\theta^{\prime}$
compared to the pretrained model $\theta$. The implementation of the whole
framework can be found in Algorithm 1.

Notably, a series of models, referred to as latent diffusion models (LDMs),
employs both diffusion and denoising processes in a latent space. Contrasting
gradient inversion on these LDMs follows the same paradigm, and we present the
algorithm in Appendix D.

Algorithm 1 Contrasting Gradient Inversion for Diffusion Model (CGI-DM)

Input: Partial representation $\overline{x}_{0}$ of data $x_{0}$, pretrained
model parameter $\theta$, fine-tuned model parameter $\theta^{\prime}$, number
of Monte Carlo sampling steps $N$, step-wise length $\alpha$.

Output: Recovered sample $\overline{x_{0}}^{(N)}$

Initialize $\overline{x_{0}}^{(0)}\leftarrow\overline{x_{0}}$.

for $i=0$ to $N-1$ do

Sample $t\sim\mathcal{U}(1,1000)$

Sample current noise $\varepsilon_{t}\sim\mathcal{N}(0,1)$

$\Delta_{\delta^{(i+1)}}\leftarrow\nabla_{\overline{x}_{0}^{(i)}}L_{tar}(\theta,\theta^{\prime},\overline{x}_{0}^{(i)},t,\varepsilon_{t})$
in Eq. 7

$\delta^{(i+1)}\leftarrow\alpha\frac{\Delta_{\delta^{(i+1)}}}{\|\Delta_{\delta^{(i+1)}}\|_{2}}$

Clip $\delta^{(i+1)}$ s.t.
$\|\overline{x}_{0}^{(i)}+\delta^{(i+1)}-\overline{x}_{0}^{(0)}\|_{2}\leq\epsilon$

$\overline{x}_{0}^{(i+1)}\leftarrow\overline{x}_{0}^{(i)}+\delta^{(i+1)}$

end for

##  4 Experiments

Style-Driven: WikiArt Dataset
---
| Dreambooth (w. prior loss) | Dreambooth (w/o. prior loss) | Lora
| Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$ | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$ | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$
Text2img | 0.64 | 0.70 | 0.68 | 0.74 | 0.67 | 0.72 | 0.69 | 0.76 | 0.60 | 0.66 | 0.61 | 0.68
inpainting | 0.67 | 0.71 | 0.71 | 0.74 | 0.71 | 0.75 | 0.77 | 0.82 | 0.59 | 0.59 | 0.60 | 0.59
Img2img | 0.82 | 0.86 | 0.89 | 0.92 | 0.83 | 0.90 | 0.90 | 0.94 | 0.68 | 0.73 | 0.74 | 0.79
CGI-DM | 0.90 | 0.95 | 0.96 | 0.98 | 0.86 | 0.95 | 0.94 | 0.99 | 0.74 | 0.80 | 0.81 | 0.87
Subject-Driven Generation: Dreambooth Dataset
| Dreambooth (w. prior loss) | Dreambooth (w/o. prior loss) | Lora
| Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$ | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$ | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$
Text2img | 0.65 | 0.71 | 0.68 | 0.75 | 0.69 | 0.75 | 0.70 | 0.79 | 0.64 | 0.69 | 0.66 | 0.74
inpainting | 0.63 | 0.71 | 0.67 | 0.76 | 0.67 | 0.79 | 0.71 | 0.82 | 0.62 | 0.64 | 0.64 | 0.68
Img2img | 0.67 | 0.77 | 0.73 | 0.82 | 0.75 | 0.81 | 0.81 | 0.87 | 0.65 | 0.75 | 0.68 | 0.76
CGI-DM | 0.84 | 0.85 | 0.90 | 0.91 | 0.89 | 0.88 | 0.94 | 0.94 | 0.76 | 0.79 | 0.82 | 0.86

Table 1: Comparison of CGI-DM and other existing pipelines in copyright
authentication for style-driven generation using WikiArt dataset [21] and for
object-driven generation under Dreambooth dataset [24] under different fine-
tuning methods. The Monte Carlo sampling steps ($N$) for CGI-DM is fixed to
1000 and the step-wise length ($\alpha$) is fixed to 2 in $L_{2}$ norm. The
overall budget is fixed to 70 in $L_{2}$ norm. We employ block-wise masking
with a block size of 4 for removing partial information. The experimental
results demonstrate that CGI-DM exhibits superior performance under all
scenarios.

Diffusion Model Structures | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$
---|---|---|---|---
SD(v1.4) | 0.93 | 0.96 | 0.97 | 0.98
SD(v1.5) | 0.97 | 0.96 | 0.97 | 0.99
SD(v2.0) | 0.88 | 0.94 | 0.91 | 0.98
AltDiffusion | 0.92 | 0.94 | 0.98 | 0.99

Table 2: Robustness of CGI-DM to different model structures and parameters.
Experiments are conducted under Dreambooth (with prior loss) fine-tuning using
5 classes of images from the WikiArt dataset. All parameters for CGI-DM are
set as the default value in Tab. 1.

In this section, we apply our proposed method, CGI-DM, for copyright
authentication under various few-shot generation methods across different
types of Diffusion Models. For style-driven generation, which focuses on
capturing the key style of a set of images, we randomly select 20 artists each
with 20 images from the WikiArt dataset [21]. For subject-driven generation,
which emphasizes details of a given object, we randomly choose 30 objects from
the Dreambooth dataset [24], each consisting of more than 4 images. Half of
these images are utilized for training, while the other half remains
untrained. This results in 10 images used for training a style and 2-6 images
used for training an object, aligning with the recommended number for training
in the mentioned fine-tuning methods [24, 13]. We adopt the aforementioned
image selection process as the default setting.

The default model used for training is Stable-Diffusion-Model
V1.4333https://huggingface.co/CompVis/stable-diffusion. Additionally, we
demonstrate the adaptability of our method to various types and versions of
diffusion models, different training steps, and a larger number of training
images (refer to Sec. 4.3 for more details).

As noted in previous studies [1], the presence of near-duplicate examples
across different datasets complicates the differentiation between seen and
unseen datasets. To address this, we define near-duplicate examples as those
with Clip-similarity [11] exceeding 0.90, and we take precautions to prevent
their inclusion within the WikiArt datasets (refer to Appendix E for further
details).

Our objective is to apply our method CGI-DM to distinctly differentiate
between images used for training (membership) and those left untrained
(holdout) [8, 16]. We conduct tests across various fine-tuning scenarios,
including direct Dreambooth (with prior loss), Dreambooth (without prior loss)
[24], lora [13]. Specifics of the training process are outlined in Appendix G.
It is noteworthy that the Dreambooth (without prior loss) scenario holds
particular significance, given that the process closely mirrors direct fine-
tuning.

We set Monte Carlo sampling steps ($N$) to 1000 and the step-wise length
($\alpha$) to 2, with an overall updating of 70 within the $L_{2}$ norm by
default for our method. This value (70) approximates the average distance
between the partial representation of the images and the original images. The
default block size for deriving partial representation is 4. For all
experiments in Sec. 4.2, where our method is compared with other approaches,
we utilize all 20 classes from the WikiArt Dataset and all 30 classes from the
Dreambooth Dataset. Additionally, for ablation studies and experiments
validating the generalization ability of our method in Sec. 4.3, Sec. 4.4 and
Sec. 4.5, we randomly select five classes of images from the WikiArt Dataset.

###  4.1 Evaluation Metrics

As mentioned in Sec. 2.2, it is crucial for the metric to be visualizable and
understandable by human beings. Therefore, upon deriving the extracted images
and original input images, we calculate the visual similarity between them.
Specifically, following the approach in a prior study [24], we employ Clip
similarity and Dino similarity, utilizing the feature space of the Clip
[11]444We use image embeddings of Clip ViT-B/32. and Dino models [2]. These
measures have been demonstrated to align closely with human vision [11, 24].

Upon obtaining the similarity scores, we determine an optimal threshold to
distinguish between membership and holdout data. Subsequently, we calculate
the Accuracy (Acc.) and Area Under the ROC Curve (AUC) [8, 16] to assess the
effectiveness of a method in discriminating between the two, thereby
evaluating its performance in authenticating copyright. We abbreviate the Acc.
and AUC under similarity measured by Clip as Acc. (C) and AUC (C), and the
ones under similarity measured by Dino as Acc. (D) and AUC (D).

Notably, for a given fine-tuning method, we utilize a fixed threshold to
distinguish membership and holdout among different objects and styles—an
approach that accounts for real-world scenarios. Generally, obtaining
membership and holdout datasets beforehand, and determining an optimal
threshold for each style or object, is impractical. Therefore, our threshold
is set independently of different styles or objects, representing a more
practical and meaningful scenario. Results for a specific threshold for each
style or object are also presented in Appendix I to ensure a comprehensive
discussion.

###  4.2 Comparison with Existing Methods

While no methods explicitly claim to be used for authenticating copyright on
few-shot generation, both existing image generation [31, 1] and inpainting
[33] pipelines exhibit the potential to do so. Particularly, we compare our
method with three types of pipelines that could be employed for identifying
infringements:

Text-to-Image Generation. Using the official Text-to-Image pipeline in
diffusers555https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img,
we generate $100\times\rm{Num}$ images for each fine-tuned model using the
prompt employed during training. Here, ‘$\rm{Num}$’ represents the total
number of images in the membership and holdout datasets. This process is
applied individually to each image in both datasets.

Image-to-Image Generation. Leveraging the official Image-to-Image pipeline in
diffusers666https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/img2img,
we use the training prompt to generate $100\times\rm{Num}$ images for each
fine-tuned model. The inference step is set to 50 and the img2img strength is
set to 0.7 for each image in the membership or holdout dataset.

Inpainting. Employing the state-of-the-art inpainting pipeline DDNM [33], we
generate $100\times\rm{Num}$ images by masking the right half of each image.
We use the prompt applied during training for inpainting, and the inference
step is fixed at 50.

We set the number of generated images per input image to 100. This ensures
that the time cost for both the baseline method and our method remains
similar, facilitating a fair comparison (see Appendix H for more details). We
define the similarity between the output images and a given image as the
highest similarity achieved among all the generated images corresponding to
one target image.

The comparison between CGI-DM and other pipelines is presented in Tab. 1. It
is evident that CGI-DM outperforms others significantly in various few-shot
generation scenarios across different datasets.

# of Training Images | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$
---|---|---|---|---
5 | 0.90 | 0.90 | 0.93 | 0.97
10 | 0.93 | 0.96 | 0.97 | 0.98
20 | 0.86 | 0.94 | 0.92 | 0.97
50 | 0.86 | 0.96 | 0.95 | 0.99

Table 3: Impact of training image number on the performance of CGI-DM on
WikiArt dataset. All other parameters are set the same as those in Tab. 2.

###  4.3 Generalization

In this section, we take a step further to test whether our method can be
applied to a broader range of scenarios, including different Diffusion model
structures, varying numbers of training images, and different numbers of
training steps.

Different DMs. We select different versions of two distinguishable Diffusion
Models: Stable Diffusion Model [23] and AltDiffusion [36], which are
representative of latent-space DMs and multilingual DMs, respectively. We
conduct experiments using the following versions of the two models:
SDv1.4777https://huggingface.co/CompVis/stable-diffusion,
SDv1.5888https://huggingface.co/runwayml/stable-diffusion-v1-5,
SDv2.0999https://huggingface.co/stabilityai/stable-diffusion-2, and
AltDiffusion101010https://huggingface.co/docs/diffusers/api/pipelines/alt_diffusion.

As shown in Tab. 2, our method consistently maintains high Acc. (above 88%)
and AUC (above 90%) scores across different DMs.

(a) CGI-DM under different training steps

(b) CGI-DM under different extraction steps

Figure 3: CGI-DM under different training steps and extraction steps. All
other parameters are set the same as those in Tab. 2.

Number of Training Images. As the number of training images increases, the
learned concept during fine-tuning becomes more intricate, thereby enhancing
the difficulty of copyright authentication. To thoroughly examine how this
influences performance, we conduct experiments with varying numbers of
training images, the results of which are presented in Tab. 3. It is evident
that our method remains almost stable despite changes in the number of
training images. Specifically, CGI-DM consistently achieves Acc. around 90% in
all cases, which demonstrates the robustness of CGI-DM to scenarios with
various training data sizes during fine-tuning.

Training Steps. Increasing training steps is known to lead to stronger
memorization of the training images, making the extraction process easier [1,
27]. Therefore, we also analyze the performance of our method under different
training steps. As depicted in Fig. 3(a), our method maintains a high
authentication success rate (above 90% Acc.) under the Dreambooth scenario
(with prior loss), with 40 training steps per image. The 40 training steps per
image are notably fewer than the typically recommended 100 steps per image for
training, as reported in [24]. As the number of training steps increases, the
model becomes more adept at memorizing the concepts present in the given
images, leading to a rise in copyright authentication accuracy for CGI-DM.

###  4.4 Ablation Study

####  4.4.1 Methods for Removing Partial Information

As mentioned earlier, our approach commences with a partial representation of
the images. In this section, we explore the following techniques to eliminate
partial information from the provided images:

Blurring. We employ Gaussian
blurring111111https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html,
with a fixed kernel size of 16 and a blurring rate of 7.

Masking. We utilize a large mask which covers right half of the images,
following previous work [1].

Block-wise Masking. We use square with different sizes (2, 4 and 8) to mask
half of the region in the images.

The representation is illustrated in Fig. 4. As indicated in both Fig. 4 and
Tab. 4, we observe that to achieve a match between the recovered images and
the desired training samples, the masked information must be fine-grained,
ensuring retention of local information in the masked image. Otherwise, the
recovered content may be irrelevant to the given image. Consequently, the
block-wise masking emerges as the superior choice. Regarding the block size of
the mask, we discover that it does not significantly impact performance.

Removing Methods | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (C)$\uparrow$ | AUC (D)$\uparrow$
---|---|---|---|---
Blurring | 0.65 | 0.80 | 0.69 | 0.86
Masking | 0.72 | 0.73 | 0.76 | 0.75
Block-wise Masking(2) | 0.93 | 0.97 | 0.97 | 0.99
Block-wise Masking(4) | 0.93 | 0.96 | 0.97 | 0.98
Block-wise Masking(8) | 0.94 | 0.97 | 0.97 | 1.00

Table 4: Influence of different approaches for removing partial information on
CGI-DM. All other parameters are set as the same Tab. 2. Figure 4:
Visualization of different methods for removing partial information and
corresponding recovered samples for CGI-DM on Van Gogh’s paintings from the
WikiArt dataset.

####  4.4.2 Extraction Steps

One critical hyper-parameter in our algorithm is the number of Monte Carlo
Sampling steps, which we denote as “Extraction steps”. Notably, the time
required by CGI-DM increases linearly with the increment in extraction steps.
We conduct experiments with different extraction steps and present the results
in Fig. 3(b). The findings suggest that excessively small extraction steps
fail to yield the optimal solution, leading to inadequate exploitation of the
conceptual differences between the fine-tuned models and the pretrained
models. Consequently, this inadequacy results in significantly poorer
extraction performance and copyright authentication. Notably, performance
stabilizes when the number of steps reaches approximately 1000. Further
increasing the number of steps substantially escalates the computational cost
while yielding only marginal improvements.

###  4.5 Performance of CGI-DM under Defenses

As highlighted in prior research [8, 16, 22], it is possible to defend against
methods that leverage the loss function of a model. In line with this, we
conduct experiments on CGI-DM under various defenses, including
RandomHorizontalFlip [8], Cutout [7], and RandAugment [4]. The block size for
Cutout is consistently fixed at $64\times 64$. Notably, RandAugment serves as
a strong privacy-preserving defense, at the price of a decline in generation
quality [8]. The results presented in Tab. 5 underscore the consistent
effectiveness of our method under various defense mechanisms. Even
RandAugment, recognized for its strong defense capabilities, demonstrates
limited effectiveness in reducing the efficacy of copyright authentication for
CGI-DM, with reductions of approximately 10% in Acc. and 8% in AUC.

Defense Methods | Acc. (C)$\uparrow$ | Acc. (D)$\uparrow$ | AUC (D)$\uparrow$ | AUC (D)$\uparrow$
---|---|---|---|---
No Defense | 0.93 | 0.96 | 0.97 | 0.98
HorizationalFlip [8] | 0.93 | 0.94 | 0.98 | 0.98
Cutout [7] | 0.87 | 0.93 | 0.92 | 0.97
RandAugment [4] | 0.83 | 0.85 | 0.86 | 0.92

Table 5: Performance of CGI-DM under possible defenses. All parameters are set
the same as those in Tab. 2.

##  5 Related Work

Model Inversion (MI). Model Inversion (MI) Attack was first introduced by
Matthew _et al_. [9]. The attack focuses on deriving the training dataset of a
network based on its parameters, mostly aiming at white-box scenarios [9, 38,
10, 3, 20, 37]. Recent approaches on MI [38, 10, 3] highlight the importance
of prior information during inversion, emphasizing that the proper prior of a
generative model could contribute to the inversion of a classification model.
Our inversion from partial representation is also inspired by such findings.

Although model inversion has been extensively researched for classification
models over the years, there is no systematic research on model inversion for
generative models as far as we know.

Membership Inference Attack (MIA). Membership Inference Attack [26, 8, 16, 22]
centers on identifying which subset of a larger data pool is used for
training. These methods generally leverage the loss function of the model in
white-box scenarios where the model parameter is available, or simulate the
loss in gray-box or black-box scenarios where the model parameter is not as
clear.

Limited research has been conducted on the use of MIA for copyright
authentication [22], possibly due to its intrinsic limitations. While MIA
efficiently assesses the origin of a sample, its reliance on the model’s loss
function as the metric for classification restricts its visualization
capabilities. Consequently, MIA-based techniques typically provide only binary
outcomes, hindering their use as strong legal evidence. Further discussion is
available in Appendix A.

Data Watermark on Diffusion Models. Data watermarking [40, 5, 34] entails
embedding watermarks into training images, enabling post-training extraction
from the generated output. This process holds potential for copyright
authentication by identification of training images via extracting the
embedded watermarks in the generated images. However, prior studies [40, 5,
34] show that most methods necessitate a single designated watermark across
the entire diffusion model dataset. Efficacy of such an approach in few-shot
generation scenarios, where only several input images correspond to a single
watermark, remains uncertain. Moreover, the introduction of watermarks may
suffer from image degradation [40, 5], content changes [5], and potential
removal [39].

While recognizing these possible issues, it is crucial to note that watermarks
and our proposed method are separate ways to protect copyright, and they do
not clash. It is conceivable that in the future, these approaches could
converge, combining to create a more robust copyright authentication system.

##  6 Conclusion

In this paper, we introduce a new method for copyright protection called
copyright authentication. Our framework, CGI-DM, validates the use of training
samples featuring vivid visual representation, serving as a tool for digital
copyright authentication. We start by removing part of the input image. Then,
using Monte Carlo sampling and PGD, we exploit the differences between the
pretrained and fine-tuned model to recover the removed information. A high
similarity between the recovered samples and the original input samples
suggests a potential infringement. Through experiments on WikiArt and
Dreambooth datasets, we demonstrate CGI-DM’s robustness and effectiveness,
surpassing alternative approaches. Such experimental results show that CGI-DM
is adept at providing legal evidence for art style mimicry and unauthorized
image fabrication. In conclusion, CGI-DM not only offers robust method for
infringement validation in the evolving DM landscape but also pioneers the
application of gradient inversion in generative models.

##  7 Acknowledgements

We extend our deepest gratitude to Professor Ge Zheng of the Koguan Law School
at Shanghai Jiao Tong University for his invaluable insights into legal
matters. Our thanks also go to Yiming Xue for offering valuable suggestions on
legal issue revisions. Furthermore, we are grateful to Chen Gong and Peishen
Yan from Shanghai Jiao Tong University, as well as Qingsi Lai from Peking
University, for their substantial contributions and thoughtful feedback.

Supplementary Material

##  Appendix A Legal Issues

###  A.1 Copyright Infringement for Artificial Intelligence-generated Content
(AIGC)

In traditional copyright infringement cases, the initial process of
determining whether an image has been “referenced” in the creation of another
image is relatively straightforward. However, the subsequent legal judgment,
which involves quantifying the extent of similarity and deciding if it amounts
to infringement, presents more significant challenges. This necessitates an
examination of substantial similarity aspects, including style and
composition, while also considering the originality of the work in question
and any authorized use. This stage poses challenges due to the requirement for
a nuanced understanding of the intricate aspects of copyright law.

In the traditional context, particularly in the field of painting, the
acceptance of visual and comprehensible evidence as a key element for legal
decisions has evolved over time. Once established, a high degree of similarity
found in such evidence often led to findings of infringement. For instance, an
artist was fined a huge amount of money for plagiarizing
artworks121212https://www.caixinglobal.com/2023-10-25/chinese-artist-
fined-5-million-yuan-in-landmark-case-for-plagiarizing-foreign-
work-102120348.html. In this case, visually similar paintings serve as
evidence of infringement, underscoring the significance of visual proof in
traditional copyright cases. This sets a precedent, highlighting the need for
adapting similar methodologies to address the unique challenges in the AIGC
realm.

In the era of AIGC, the stark absence of widely-accepted factual evidence, in
contrast to traditional contexts, is evident. The intricate nature of AI
algorithms complicates the task of establishing direct references. This has
led to a significant shift in legal regulations, moving from focusing
primarily on direct infringement judgments to tackling anti-unfair competition
concerns. This transition is a response to the lack of clear, visual, and
easily interpretable evidence, reflecting the adjustments current regulations
are undergoing to accommodate the nuances of
AIGC131313https://digichina.stanford.edu/work/how-will-chinas-generative-ai-
regulations-shape-the-future-a-digichina-forum/.

Addressing these complexities in AIGC demands the cultivation of a widely-
recognized factual understanding of what constitutes a “reference.” Here, the
importance of robust copyright authentication tools becomes apparent. These
tools are crucial for not only establishing facts but also for making them
visually comprehensible and publicly acceptable. The recent lawsuit involving
“Stable Diffusion”141414https://www.hollywoodreporter.com/business/business-
news/artists-copyright-infringement-case-ai-art-generators-1235632929 brings
to light the current reliance on visual evidence in copyright lawsuits. It
also, however, points to the inadequacy of these methods due to the limited
similarity between AI-generated output images and the original training data.
Our work with CGI-DM focuses on overcoming this hurdle, offering a method to
aid in substantiating infringement claims with higher visual similarity.

In the short term, these copyright authentication tools offer valuable
assistance in current lawsuits against AIGC. While they may not be crucial at
this juncture, their role in establishing necessary visual and comprehensible
evidence is significant as we adapt our legal frameworks to the challenges
posed by AIGC. In the long term, their importance grows, as establishing well-
accepted facts becomes paramount for the effective adjudication of AIGC-
related copyright issues. Tools like the proposed CGI-DM method are
increasingly vital in this extended context, which enable a shift in focus
from merely contending with anti-unfair competition to directly addressing
copyright infringement. This reorientation is vital for ensuring stronger
protection of the intellectual property rights of human artists, safeguarding
their work against the advancing tide of AIGC.

In conclusion, the development and acceptance of robust validation tools are
essential in shaping the future of copyright law enforcement in the context of
AIGC. Reflecting on the historical trajectory of how visual evidence gradually
became integral to legal judgment in traditional contexts, we see a clear need
for a similar evolution in the AIGC landscape. This historical perspective
reinforces the importance of societal acceptance in the legal adjudication
process and underscores the necessity of adapting these principles to the
emerging challenges of AIGC.

| Need Preprocessing? | Defense | Accuracy | Visualizability
---|---|---|---|---
MIA [8, 16] | ✗ | Hard | High | Poor
Data Watermark [40, 5, 34] | ✓ | Uncertain | Uncertain | Moderate
Copyright Authentication | ✗ | Hard | High | Strong

Table 6: Comparative Overview of Post-cautionary Copyright Protection Methods.
Bold text highlights advantageous features, while plain text indicates areas
where these features are less pronounced.

###  A.2 More Discussion of Post-Cautionary Methods for Copyright Protection

This section presents an in-depth analysis of various post-caution methods for
copyright protection. The summary table in Tab. 6 illustrates that, in terms
of preprocessing, defense, accuracy, and visualizability, copyright
authentication emerges as a superior choice.

Membership Inference Attack(MIA). There are only a limited number of studies
that explore the use of MIA for infringement validation [22], possibly due to
the inherent limitations of this method. MIA primarily assesses the
probability of a sample being part of the training dataset, but its dependency
on the model’s loss function results in outcomes that are challenging to
visualize. This limitation restricts MIA’s use in legal settings, as it
provides only binary ‘yes’ or ‘no’ outputs, lacking the detailed nuance needed
for legal evidence. Additionally, these binary outcomes, being dependent on
the model’s loss function, deviate from human visual perception, thus limiting
their reliability and applicability in the process of legal judgment.

Data Watermark. Data watermarking typically requires preprocessing on training
images [40, 5, 34]. This method’s limitations include potential degradation in
image quality and the feasibility of watermark removal. Furthermore, its
effectiveness is predominantly aligned with current Digital Models (DMs) and
may not extend to future DMs or other generative models. The intrinsic design
of data watermarking implies permanent security once implemented. Besides, its
performance under defense and its accuracy remain uncertain under few-shot
generation scenarios. The method offers moderate visualizability since the
embedded watermark can be discerned in the output, making it more
comprehensible than MIA.

Copyright Authentication. At present, copyright authentication requires no
preprocessing and demonstrates robustness in defense scenarios, achieving high
accuracy and strong visualizability. These attributes make it an effective
tool for providing legal support in cases of copyright infringement.

It is noteworthy that the methods aforementioned do not mutually exclude each
other. In the long term, they could potentially converge to form a
comprehensive system for validating copyright infringement.

##  Appendix B Proof Details

###  B.1 Forward DDIM

It has recently been found that the diffusion process from $x_{T}$ to $x_{1}$
can be directly obtained by leveraging $x_{0}$ [28]. The reverse of this
formulation can even lead to a predicted forward process using the model
weight of the diffusion model [15]:

$\displaystyle p_{\theta}(x_{t+1}|x_{t})=$
$\displaystyle\mathcal{N}(x_{t+1};\sqrt{\alpha_{t+1}}f_{\theta}(x_{t},t)+$
(10)
$\displaystyle\sqrt{1-\alpha_{t+1}}\epsilon_{\theta}(x_{t},t),\sigma_{t}^{2}\textbf{I}),$

where $f_{\theta}(x_{t},t)$ is the predicted $x_{0}$ defined
as:$f_{\theta}(x_{t},t):=\frac{x_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(x_{t},t)}{\sqrt{\alpha_{t}}}$.

###  B.2 Markov Chain Expanding and Expectation Term Transformation

KL divergence
$D_{\rm{KL}}(p_{\theta^{\prime}}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})||{p_{\theta}(\overline{x}_{1:T}^{\prime}|\overline{x}_{0}^{\prime})})$in
Eq. 4 can be extended to the expectation of the logarithm of the division of
two probability functions, given the pretrained model and the fine-tuned
model.

$\displaystyle\delta$
$\displaystyle:=\arg\max\limits_{\delta}\mathbb{E}_{p_{\theta^{\prime}}(x^{\prime}_{1:T}|x^{\prime}_{0})}\log\frac{p_{\theta^{\prime}}(x^{\prime}_{1:T}|x^{\prime}_{0})}{p_{\theta}(x^{\prime}_{1:T}|x^{\prime}_{0})},$
(11) $\displaystyle\mbox{where }x_{0}\sim
q(x_{0}),x_{0}^{\prime}=x_{0}+\delta.$

We isolate a single term within the logarithm to simplify the notation and
defer the division of two probability terms to a later step.

$\displaystyle\max\limits_{\delta}\mathbb{E}_{p_{\theta^{\prime}}(x^{\prime}_{1:T}|x^{\prime}_{0})}\log
p_{\theta^{\prime}}(x^{\prime}_{1:T}|x^{\prime}_{0})$ (12) $\displaystyle=$
$\displaystyle\max\limits_{\delta}\mathbb{E}_{\prod_{t=0}^{T-1}p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t})}\log\prod_{t=0}^{T-1}p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t})$
$\displaystyle=$
$\displaystyle\max\limits_{\delta}\sum_{t=0}^{T-1}\mathbb{E}_{\prod_{k=0}^{T-1}p_{\theta^{\prime}}(x^{\prime}_{k+1}|x^{\prime}_{k})}\log
p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t})$ $\displaystyle=$
$\displaystyle\max\limits_{\delta}\sum_{t=0}^{T-1}\mathbb{E}_{p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t})}\log
p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t}).$

The last equation holds true as the possibility function
$p(x^{\prime}_{t+1}|x^{\prime}_{t})$ is a Markov chain and is therefore
independent of any possibility that precedes time $t$. It is important to note
that, for a well-trained model on data points $x_{0}\sim q(x_{0})$, it should
effectively simulate the real prior distribution. In other words,
$p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t})$ can be approximated as
$q(x^{\prime}_{t+1}|x^{\prime}_{t})$. Notably, this approximation only holds
for the fine-tuned model $\theta^{\prime}$ as it is the only model trained on
image $x_{0}$, and it does not hold true for the pretrained model $\theta$.
Based on this approximation, we can modify our target to:

$\displaystyle\max\limits_{\delta}\sum_{t=0}^{T-1}\mathbb{E}_{p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t})}\log
p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t})$ (13)
$\displaystyle\approx$
$\displaystyle\max\limits_{\delta}\sum_{t=0}^{T-1}\mathbb{E}_{q(x^{\prime}_{t+1}|x^{\prime}_{t})}\log
p_{\theta^{\prime}}(x^{\prime}_{t+1}|x^{\prime}_{t}).$

###  B.3 Approximation of the loss function

Given Equation (10) in Section 2.1, we can demonstrate that the discrepancy
between the initial image $x_{0}$ and its predicted counterpart
$x_{0}:=f_{\theta}(x_{t},t)$ is indeed linked to the disparity between the
forecasted noise $\epsilon_{\theta}(x_{t},t)$ at time $t$ with respect to the
true noise $\varepsilon_{t}$:

$\displaystyle x_{0}-f_{\theta}(x_{t},t)$ (14) $\displaystyle=$ $\displaystyle
x_{0}-\frac{x_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(x_{t},t)}{\sqrt{\alpha_{t}}}$
$\displaystyle=$ $\displaystyle
x_{0}-\frac{\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\varepsilon_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(x_{t},t)}{\sqrt{\alpha_{t}}}$
$\displaystyle=$
$\displaystyle\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}(\varepsilon_{t}-\epsilon_{\theta}(x_{t},t)).$

Based on this finding, we are now able to establish the validity of the
approximation presented in Equation (6):

$\displaystyle\|x^{\prime}_{t+1}-\mu_{p_{\theta}(x^{\prime}_{t+1}|x^{\prime}_{t})}\|^{2}$
(15) $\displaystyle=$
$\displaystyle\|\sqrt{\frac{\alpha_{t+1}}{\alpha_{t}}}x^{\prime}_{t}+\sqrt{1-\frac{\alpha_{t+1}}{\alpha_{t}}}\varepsilon_{t+1}-\sqrt{\alpha_{t+1}}f_{\theta}(x^{\prime}_{t},t)-$
$\displaystyle\sqrt{1-\alpha_{t+1}}\epsilon_{\theta}(x^{\prime}_{t},t)\|^{2}$
$\displaystyle=$
$\displaystyle\|\sqrt{\alpha_{t+1}}x^{\prime}_{0}+\sqrt{(1-\alpha_{t})\frac{\alpha_{t+1}}{\alpha_{t}}}\varepsilon_{t}+\sqrt{1\frac{\alpha_{t+1}}{\alpha_{t}}}\varepsilon_{t+1}-$
$\displaystyle\sqrt{\alpha_{t+1}}f_{\theta}(x^{\prime}_{t},t)-\sqrt{1-\alpha_{t+1}}\epsilon_{\theta}(x^{\prime}_{t},t)\|^{2}$
$\displaystyle=$
$\displaystyle\|\sqrt{\alpha_{t+1}}(x^{\prime}_{0}-f_{\theta}(x^{\prime}_{t},t))-\sqrt{1-\alpha_{t+1}}(\epsilon_{\theta}(x^{\prime}_{t},t)-\varepsilon_{t})+$
$\displaystyle(\sqrt{(1-\alpha_{t})\frac{\alpha_{t+1}}{\alpha_{t}}}-\sqrt{1-\alpha_{t+1}})\varepsilon_{t}+\sqrt{1-\frac{\alpha_{t+1}}{\alpha_{t}}}\varepsilon_{t+1}\|^{2}$
$\displaystyle\approx$
$\displaystyle\|\sqrt{\alpha_{t+1}}(x^{\prime}_{0}-f_{\theta}(x^{\prime}_{t},t))-\sqrt{1-\alpha_{t+1}}(\epsilon_{\theta}(x^{\prime}_{t},t)-\varepsilon_{t})\|^{2}.$

The final equation is valid due to the approximations of both
$\sqrt{1-\frac{\alpha_{t+1}}{\alpha_{t}}}$ and
$(\sqrt{(1-\alpha_{t})\frac{\alpha_{t+1}}{\alpha_{t}}}-\sqrt{1-\alpha_{t+1}})$
being close to 0. Hence, we obtain:

$\displaystyle\|x^{\prime}_{t+1}-\mu_{p_{\theta}(x^{\prime}_{t+1}|x^{\prime}_{t})}\|^{2}$
(16) $\displaystyle\approx$
$\displaystyle\|\sqrt{\alpha_{t+1}}(x^{\prime}_{0}-f_{\theta}(x^{\prime}_{t},t))-\sqrt{1-\alpha_{t+1}}(\epsilon_{\theta}(x^{\prime}_{t},t)-\varepsilon_{t})\|^{2}$
$\displaystyle=$
$\displaystyle-(\frac{\sqrt{1-\alpha_{t}}\sqrt{\alpha_{t+1}}}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t+1}})^{2}\|\varepsilon_{t}-\epsilon_{\theta}(x_{t},t)\|^{2}$

##  Appendix C Direct Gradient Inversion (GI) Leads to Zero-information
Extracted Result

We observe that inverting the gradient solely based on the fine-tuned model
parameterized by $\theta^{\prime}$ resulted in a zero-information-extracted
outcome. Specifically, for the partial representation $\overline{x_{0}}$ of an
image $x_{0}$, we optimize it based on the following target:

$\displaystyle x^{\prime}_{0}$
$\displaystyle=\arg\min\limits_{x^{\prime}_{0}}p_{\theta^{\prime}}(x^{\prime}_{1:T}|x^{\prime}_{0})$
(17)
$\displaystyle\approx\arg\min\limits_{x^{\prime}_{0}}\mathbb{E}_{t,\epsilon_{t}\sim\mathcal{N}(0,1)}\|\varepsilon_{t}-\epsilon_{\theta^{\prime}}(x^{\prime}_{t},t)\|^{2}.$

We omit the proof, as it closely resembles the proof mentioned in Section 3.2.
We employ a similar optimization method utilizing Monte Carlo Sampling and PGD
attack, as discussed in Section 3.3. As illustrated in Figure 5, the extracted
images appear blurred, with no useful information discernible. This is
primarily attributed to the fact that the diffusion model functions as a
robust noise predictor, resulting in the most probable image according to the
model being a smooth one with low information content, thus enabling the model
to easily predict the introduced noise.

Figure 5: Comparison of CGI-DM and direct GI under different partial
representations. We can find that direct GI does not recover any semantic
information of the images.

##  Appendix D CGI-DM on Latent Diffusion Model (LDM)

The key distinction between the latent diffusion model (LDM) model [23] and
others lies in the occurrence of the diffusion process within the latent
space. Consequently, our algorithm, CGI-DM, for LDM, closely resembles CGI-DM
for DM, with the exception that the partial information removal and the
consideration of predicted noise differences are both based on the latent
space of a LDM. Refer to Algorithm 2 for more details.

Algorithm 2 CGI-DM for LDMs

Input: Partial representation $\overline{x}_{0}$ of data $x_{0}$, pretrained
model parameter $\theta$, fine-tuned model parameter $\theta^{\prime}$, number
of Monte Carlo sampling steps $N$, step-wise length $\alpha$, encoder
$\mathcal{E}$, decoder $\mathcal{D}$

Output: Recovered sample $\overline{x}_{0}^{(N)}$

Initialize $\overline{z_{0}}^{(0)}\leftarrow\mathcal{E}(\overline{x_{0}})$.

for $i=0$ to $N-1$ do

Sample $t\sim\mathcal{U}(1,1000)$

Sample current noise $\varepsilon_{t}\sim\mathcal{N}(0,1)$

$\Delta_{\delta^{(i+1)}}\leftarrow\nabla_{\overline{z}_{0}^{(i)}}L_{tar}(\theta,\theta^{\prime},\overline{z}_{0}^{(i)},t,\varepsilon_{t})$
in Eq. 7

$\delta^{(i+1)}\leftarrow\alpha\frac{\Delta_{\delta^{(i+1)}}}{\|\Delta_{\delta^{(i+1)}}\|_{2}}$

Clip $\delta^{(i+1)}$ s.t.
$\|\overline{z}_{0}^{(i)}+\delta^{(i+1)}-\overline{z}_{0}^{(0)}\|_{2}\leq\epsilon$

$\overline{z}_{0}^{(i+1)}\leftarrow\overline{z}_{0}^{(i)}+\delta^{(i+1)}$

end for

$\overline{x}_{0}^{(N)}\leftarrow\mathcal{D}(\overline{z}_{0}^{(N)})$.

(a) MIA methods under WikiArt Dataset

(b) MIA methods under Dreambooth Dataset

Figure 6: Comparison of C-MIA and other MIA methods under two representative
datasets under few-shot generation scenarios.

##  Appendix E Removal of near-duplicate samples

Following the precedent set by previous research, we classify samples with a
clip similarity exceeding 0.90 as near-duplicates [1]. Within the WikiArt
dataset, we eliminate near-duplicate samples until only one copy remains.
Given that the utilization of any of these near-duplicate samples already
constitutes a violation of the provided data, the presence of such samples in
both the training dataset and other datasets would result in an impractically
low assessment of the effectiveness of the infringement validation algorithm.

##  Appendix F Conceptual Difference Exploiting for MIA (C-MIA)

The definition of conceptual differences and the provided proof in Sec. 3.2
isn’t limited to copyright authentication, which has strong effects in MIA
under few-shot generation scenarios. In detail, for pretrained model $\theta$,
fine-tuned mode $\theta^{\prime}$ and a given sample $x$, we leverage the KL
divergence of the probability of $x$ between the two models:

$\displaystyle
D_{\rm{KL}}(p_{\theta^{\prime}}(x_{1:T}|x_{0})||{p_{\theta}(x_{1:T}|x_{0})})$
(18)
$\displaystyle\approx\mathbb{E}_{t,\varepsilon_{t}\sim\mathcal{N}(0,1)}\underbrace{\|\varepsilon_{t}-\epsilon_{\theta}(x_{t},t)\|^{2}-\|\varepsilon_{t}-\epsilon_{\theta^{\prime}}(x_{t},t)\|^{2}}_{L_{tar}(\theta,\theta^{\prime},x_{0},t,\varepsilon_{t})}.$

We omit details for this approximation as it mainly follows the provided proof
in Appendix B and Sec. 3. We then utilize such divergence under one given time
$t$ and sampled noise $\varepsilon_{t}$, denoted as
$L_{tar}(\theta,\theta^{\prime},x_{0},t,\varepsilon_{t})$, directly for MIA.
We name this method as “C-MIA”. We experiment under both WikiArt dataset and
Dreambooth dataset, with settings aligned to the one mentioned in Tab. 1. Our
method is compared to two representative MIA methods: Naive Attack (NA) [16]
and Proximal Initialization Attack (PIA) [16]. The performance comparison of
these methods is shown in Fig. 6, where we observe consistent outperformance
by C-MIA across various time variable choices and datasets.

Style Transferring: Wikiart Dataset
---
| DreamBooth(w. prior loss) | DreamBooth(w/o. prior loss) | Lora
|  ASR(Clip)$\uparrow$ |  ASR(Dino)$\uparrow$ |  AUC(Clip)$\uparrow$ |  AUC(Dino)$\uparrow$ |  ASR(Clip)$\uparrow$ |  ASR(Dino)$\uparrow$ |  AUC(Clip)$\uparrow$ |  AUC(Dino)$\uparrow$ |  ASR(Clip)$\uparrow$ |  ASR(Dino)$\uparrow$ |  AUC(Clip)$\uparrow$ |  AUC(Dino)$\uparrow$
Text2img | 0.72 | 0.77 | 0.68 | 0.75 | 0.74 | 0.80 | 0.71 | 0.80 | 0.71 | 0.74 | 0.63 | 0.70
inpainting | 0.73 | 0.78 | 0.72 | 0.76 | 0.77 | 0.84 | 0.77 | 0.85 | 0.67 | 0.67 | 0.61 | 0.60
Img2img | 0.87 | 0.94 | 0.88 | 0.95 | 0.91 | 0.95 | 0.92 | 0.97 | 0.77 | 0.80 | 0.76 | 0.80
CGI-DM | 0.96 | 0.98 | 0.97 | 0.99 | 0.96 | 0.99 | 0.97 | 1.00 | 0.82 | 0.90 | 0.83 | 0.93
Subject-Driven Generation: DreamBooth Dataset
| DreamBooth(w. prior loss) | DreamBooth(w/o. prior loss) | Lora
|  ASR(Clip)$\uparrow$ |  ASR(Dino)$\uparrow$ |  AUC(Clip)$\uparrow$ |  AUC(Dino)$\uparrow$ |  ASR(Clip)$\uparrow$ |  ASR(Dino)$\uparrow$ |  AUC(Clip)$\uparrow$ |  AUC(Dino)$\uparrow$ |  ASR(Clip)$\uparrow$ |  ASR(Dino)$\uparrow$ |  AUC(Clip)$\uparrow$ |  AUC(Dino)$\uparrow$
Text2img | 0.81 | 0.89 | 0.72 | 0.85 | 0.82 | 0.92 | 0.78 | 0.90 | 0.77 | 0.90 | 0.68 | 0.85
inpainting | 0.79 | 0.87 | 0.70 | 0.83 | 0.81 | 0.91 | 0.73 | 0.90 | 0.76 | 0.84 | 0.65 | 0.77
Img2img | 0.92 | 0.95 | 0.78 | 0.94 | 0.92 | 0.98 | 0.90 | 0.98 | 0.81 | 0.93 | 0.73 | 0.90
CGI-DM | 0.97 | 0.96 | 0.96 | 0.94 | 0.96 | 0.99 | 0.96 | 1.00 | 0.92 | 0.93 | 0.90 | 0.91

Table 7: Comparison of CGI-DM and other existing pipelines under separate
thresholds. All other settings are fixed the same as in Tab. 1. The
experimental results demonstrate that CGI-DM also exhibits superior
performance under separate thresholds.

| Inference Steps/output img | Number of output imgs/Input Img | Overall inference steps | Need backward? | Time costs(min)/Input Img | Overall Time costs(min)
---|---|---|---|---|---|---
Text2img | 50 | 100 | 5000 | ✗ | 10.36 | 207.30
Inpainting | 50 | 100 | 5000 | ✗ | 9.19 | 183.89
Img2img | 35 | 100 | 3500 | ✗ | 6.45 | 129.03
CGI-DM | 1000 | 1 | 1000 | ✓ | 5.68 | 113.67

Table 8: Time costs comparison of CGI-DM with other methods. We experiment
under Dreambooth (with prior loss) for one classes in WikiArt-dataset for
three times and report the mean time costs here.

##  Appendix G Finetuning Details

The details of the parameters in the fine-tuning methods are presented below.
We use $\rm{Num}$ to represent the number of images utilized for training

  * •

Dreambooth (With Prior): We use the training script provided by
Diffusers151515https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/
train_dreambooth.py. Both the text encoder and the U-Net are fine-tuned during
the training process. By default, the number of training steps is set to
$50\times\rm{Num}$, with a learning rate of $2\times 10^{-6}$. The batch size
is set to 1, and the number of class images used for computing the prior loss
is $50\times\rm{Num}$ by default. The prior loss weight remains fixed at 1.0.
For the WikiArt dataset, the instance prompt is “sks style”, and the class
prompt is “art style”. For the Dreambooth dataset, the instance prompt is “sks
object”, and the class prompt is “an object”.

  * •

Dreambooth (No Prior): We use the training script provided by Diffusers15. All
default parameters remain the same as in the case of Dreambooth (With Prior),
except for the exclusion of the prior loss. Notably, this adjustment will lead
to training that closely resembles direct fine-tuning.

  * •

Lora: We use the training script provided by
Diffusers161616https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/
train_dreambooth_lora.py. All default parameters remain consistent with the
case in Dreambooth (No Prior), with the exception of the learning rate and
training steps, which are adjusted to $1\times 10^{-4}$ and to
$100\times\rm{Num}$, respectively.

##  Appendix H Time costs

In this section, we present the time costs for overall validation and per-
image validation for both our method and the compared pipelines. As
demonstrated in Table 8, the time costs for the baseline methods are
approximately equal to or greater than ours under $100\times\rm{Num}$
generated images, ensuring a fair comparison where all methods require a
similar amount of time.

We further show the performance of best baseline methods (Img2img) with
increasing number of generated images in Table 7, where we can observe that
$100\times\rm{Num}$ is near saturated.

Figure 7: Trend of img2img baseline. Experimented under 5 subsets of WikiArt
dataset.

##  Appendix I Experiment Result under Separate Threshold

The default experiment setting employs a universal threshold across various
styles or objects. In this section, we present results obtained with specific
thresholds for each style or object. As depicted in Tab. 7, CGI-DM also
consistently outperforms other methods in this configuration.

##  Appendix J Visualization

We present visualization of CGI-DM in Fig. 8, Fig. 9 and Fig. 10. The
experimental setup aligns with the one described in Tab. 1.

Figure 8: Visualization of CGI-DM for membership and holdout data under 4
classes of WikiArt dataset. From left to right: Vincent Willem van Gogh,
Claude Monet, Pablo Ruiz Picasso and Paul Gauguin. Figure 9: Visualization of
CGI-DM for membership and holdout data under 4 classes of Dreambooth dataset.
From left to right: dog, vase, backpack and monster_toy. Figure 10:
Visualization of CGI-DM for membership and holdout data from 20 different
artists in WikiArt dataset under the default setting in against Dreambooth
(with prior loss). The figure is under high resolution and can be zoomed for
more details.

##  Appendix K No-reference Visual Metrics

We show results171717Experimental setup matches Tab. 1 in line 402. in Tab. 9
with BRISQUE181818No-Reference Image Quality Assessment in the Spatial Domain
(2012). and CLIP-IQA,191919Exploring CLIP for Assessing the Look and Feel of
Images (2022). where a similar trend can be observed. It can be seen as a
stricter measurement align with human vision and can be helpful for overall
evaluations.

|  Acc. (C)$\uparrow$ |  AUC (C)$\uparrow$ |  Acc. (BRISQUE)$\uparrow$ |  AUC (BRISQUE)$\uparrow$ |  Acc. (CLIP_IQA)$\uparrow$ |  AUC (CLIP_IQA)$\uparrow$
---|---|---|---|---|---|---
CGI-DM | 0.90 | 0.96 | 0.66 | 0.69 | 0.73 | 0.79
Img2img | 0.82 | 0.89 | 0.57 | 0.54 | 0.52 | 0.50
inpainting | 0.67 | 0.71 | 0.55 | 0.55 | 0.52 | 0.50
Text2img | 0.64 | 0.68 | 0.50 | 0.50 | 0.50 | 0.50

Table 9:  CGI-DM v.s. baseline under no-reference metrics.

##  Appendix L More dicussion with Inpainting Baseline

The standard approach to inpainting employs half masking techniques. We also
delve into inpainting methods that utilize block-wise masking, akin to the
CGI-DM process. We include the results in Tab. 10. Experimental setup matches
Tab. 1 in line 402.

|  Acc. (C)$\uparrow$ |  Acc. (D)$\uparrow$ |  AUC (C)$\uparrow$ |  AUC (D)$\uparrow$
---|---|---|---|---
CGI-DM | 0.90 | 0.95 | 0.96 | 0.98
inpainting with half masking | 0.67 | 0.71 | 0.71 | 0.74
inpainting with block-wise masking (4) | 0.71 | 0.80 | 0.77 | 0.87

Table 10: CGI-DM v.s. inpainting under block-wise masking(4).

## References

  * [1] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting Training Data from Diffusion Models.  In USENIX Security, 2023.
  * [2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging Properties in Self-supervised Vision Transformers.  In ICCV, 2021.
  * [3] Si Chen, Mostafa Kahla, Ruoxi Jia, and Guo-Jun Qi.  Knowledge-enriched Distributional Model Inversion Attacks.  In ICCV, 2021.
  * [4] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical Automated Data Augmentation with a Reduced Search Space.  In CVPR, 2020.
  * [5] Yingqian Cui, Jie Ren, Han Xu, Pengfei He, Hui Liu, Lichao Sun, and Jiliang Tang.  DiffusionShield: A Watermark for Copyright Protection against Generative Diffusion Models.  arXiv preprint arXiv:2306.04642, 2023.
  * [6] Andrew Deck.  AI-Generated Art Sparks Furious Backlash from Japan’s Anime Community.  https://restofworld.org/2022/ai-backlash-anime-artists/, 2022.
  * [7] Terrance DeVries and Graham W Taylor.  Improved Regularization of Convolutional Neural Networks with Cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [8] Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu.  Are Diffusion Models Vulnerable to Membership Inference Attacks?  arXiv preprint arXiv:2302.01316, 2023.
  * [9] Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart.  Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin dosing.  In USENIX Security, 2014.
  * [10] Ali Hatamizadeh, Hongxu Yin, Pavlo Molchanov, Andriy Myronenko, Wenqi Li, Prerna Dogra, Andrew Feng, Mona G Flores, Jan Kautz, Daguang Xu, et al.  Do Gradient Inversion Attacks make Federated Learning Unsafe?  IEEE Transactions on Medical Imaging, 42(7):2044–2056, 2023.
  * [11] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A Reference-free Evaluation Metric for Image Captioning.  arXiv preprint arXiv:2104.08718, 2021.
  * [12] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising Diffusion Probabilistic Models.  In NeurIPS, 2020.
  * [13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank Adaptation of Large Language Models.  arXiv preprint arXiv:2106.09685, 2021.
  * [14] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-Based Real Image Editing With Diffusion Models.  arXiv preprint arXiv:2210.09276, 2022.
  * [15] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided Diffusion Models for Robust Image Manipulation.  In CVPR, 2022.
  * [16] Fei Kong, Jinhao Duan, RuiPeng Ma, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu.  An Efficient Membership Inference Attack for the Diffusion Model by Proximal Initialization.  arXiv preprint arXiv:2305.18355, 2023.
  * [17] Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, XUE Zhengui, Ruhui Ma, and Haibing Guan.  Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples.  In ICML, 2023.
  * [18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards Deep Learning Models Resistant to Adversarial Attacks.  In ICLR, 2018.
  * [19] Deborah MT.  How AI Art Can Free Artists, Not Replace Them.  https://medium.com/thesequence/how-ai-art-can-free-artists-not-replace-them-a23a5cb0461e, 2022.
  * [20] Ngoc-Bao Nguyen, Keshigeyan Chandrasegaran, Milad Abdollahzadeh, and Ngai-Man Cheung.  Re-thinking Model Inversion Attacks Against Deep Neural Networks.  In CVPR, 2023.
  * [21] K. Nichol.  Painter by Numbers, WikiArt.  https://www.kaggle.com/c/painter-by-numbers, 2016.
  * [22] Yan Pang, Tianhao Wang, Xuhui Kang, Mengdi Huai, and Yang Zhang.  White-box Membership Inference Attacks against Diffusion Models.  arXiv preprint arXiv:2308.06405, 2023.
  * [23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution Image Synthesis with Latent Diffusion Models.  In CVPR, 2022.
  * [24] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.  In CVPR, 2023.
  * [25] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao.  Glaze: Protecting artists from style mimicry by text-to-image models.  arXiv preprint arXiv:2302.04222, 2023.
  * [26] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership Inference Attacks against Machine Learning Models.  In S&P, 2017.
  * [27] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Understanding and Mitigating Copying in Diffusion Models.  arXiv preprint arXiv:2305.20086, 2023.
  * [28] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising Diffusion Implicit Models.  In ICLR, 2020.
  * [29] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-Based Generative Modeling Through Stochastic Differential Equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [30] Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc Tran, and Anh Tran.  Anti-DreamBooth: Protecting Users from Personalized Text-to-image Synthesis.  In ICCV, 2023.
  * [31] James Vincent.  The Scary Truth About AI Copyright Is Nobody Knows What Will Happen Next.  https://www.theverge.com/23444685/generative-ai-copyright-infringement-legal-fair-use-training-data, 2022.
  * [32] Tao Wang, Yushu Zhang, Shuren Qi, Ruoyu Zhao, Zhihua Xia, and Jian Weng.  Security and Privacy on Generative Data in AIGC: A Survey.  arXiv preprint arXiv:2309.09435, 2023.
  * [33] Yinhuai Wang, Jiwen Yu, and Jian Zhang.  Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model.  In ICLR, 2022.
  * [34] Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein.  Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust.  arXiv preprint arXiv:2305.20030, 2023.
  * [35] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion Probabilistic Modeling for Video Generation.  arXiv preprint arXiv:2203.09481, 2022.
  * [36] Fulong Ye, Guang Liu, Xinya Wu, and Ledell Wu.  AltDiffusion: A Multilingual Text-to-Image Diffusion Model.  arXiv preprint arXiv:2308.09991, 2023.
  * [37] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov.  See Through Gradients: Image Batch Recovery via Gradinversion.  In CVPR, 2021.
  * [38] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song.  The Secret Revealer: Generative Model-inversion Attacks against Deep Neural Networks.  In CVPR, 2020.
  * [39] Xuandong Zhao, Kexun Zhang, Yu-Xiang Wang, and Lei Li.  Generative Autoencoders as Watermark Attackers: Analyses of Vulnerabilities and Threats.  arXiv preprint arXiv:2306.01953, 2023.
  * [40] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin.  A Recipe for Watermarking Diffusion Models.  arXiv preprint arXiv:2303.10137, 2023.


If Eqns. (<ref>)–(<ref>) and (<ref>)–(<ref>) hold, then

\begin{align}
\Bigsem{\tikzfig{ZX-green-special}}_\nu
\,=\;
\Bigsem{\tikzfig{ZH-white-special}}_\nu
\,&=\;
\biggsem{\tikzfig{ZH-id-wire-w-sqrt2}}_\nu
\,=\;
\biggsem{\tikzfig{ZX-id-wire-w-sqrt2}}_\nu  \;;
\\[2ex]
\Sem{7ex}{\,\tikzfig{ZH-ortho-bridge-w-sqrt2}\,}_{\!\nu} \!\!
\Sem{5ex}{\,\tikzfig{ZH-ortho-nobridge}\,}_{\!\nu} \;;
\\[2ex]
\Sem{6ex}{\,\tikzfig{ZH-average-prep-w-sqrt2}\,}_{\!\nu} \!\!
\Sem{5ex}{\,\tikzfig{ZH-average-loop}\,}_{\!\nu} \;.
\end{align}

This follows directly from Eqns. (<ref>)–(<ref>) and Lemma <ref>.

It remains to define the meaning of the nu-boxes for the ZX calculus, which we do by setting

\begin{equation}
\label{eqn:defineNuBoxes}
\sem{\;\tikzfig{ZX-nu-box-k}\!}_\nu = \nu^k  \;.
\end{equation}

If Eqns. (<ref>)–(<ref>), (<ref>)–(<ref>), and (<ref>) hold, then all of the rewrites of pages newZX and newZH are sound.

This follows from Lemma <ref>, and from the correspondence of the rewrites of pages newZX and newZH with those of Figure <ref>, noting in particular that (Bialg) on page newZX is equivalent to (Bialg ZR).

The model $\sem{\,\cdot\,}_\nu$ described explicitly —
For the sake of completeness, we now describe the model $\sem{\,\cdot\,}_\nu$ , whose parameters we have fixed in this Appendix.
Following from the conditions of Lemma <ref> and Eqn. (<ref>), we define the model $\sem{\,\cdot\,}_\nu$ as follows, defining the scalar $\nu = 2^{-1\!\!\:/\!\!\;4}$ :

\begin{align}{}
\Biggsem{\!\!\!\!\!\tikzfig{ZH-white-dot-arity}\!\!\!\!}_{\!\nu}
\!=\;
\Biggsem{\!\!\!\!\!\tikzfig{ZX-green-dot-arity}\!\!\!\!\!}_{\!\nu}
\!\!&=\,
\nu^{-(m{+}n-2)} \Bigl(
\ket{\zsymb}^{\!\otimes n}\!\bra{\zsymb}^{\!\!\;\otimes m}
\,+\;
\ket{\osymb}^{\!\otimes n}\!\bra{\osymb}^{\!\!\;\otimes m}
\Bigr);
% -----------------------------------------------------
\\[1ex]
% -----------------------------------------------------
\text{--- more generally,}
\quad
\Biggsem{\!\!\!\!\!\tikzfig{ZX-green-phase-dot-arity}\!\!\!\!\!}_{\!\nu}
\!&=\,
\nu^{-(m{+}n-2)} \Bigl(
\ket{\zsymb}^{\!\otimes n}\!\bra{\zsymb}^{\!\!\;\otimes m}
\,+\;
\e^{i\theta}
\ket{\osymb}^{\!\otimes n}\!\bra{\osymb}^{\!\!\;\otimes m}
\Bigr);
% -----------------------------------------------------
\\[3ex]
% -----------------------------------------------------
\Biggsem{\!\!\!\!\!\tikzfig{ZH-gray-dot-arity}\!\!\!\!}_{\!\nu}
\!=\;
\Biggsem{\!\!\!\!\!\tikzfig{ZX-red-dot-arity}\!\!\!\!}_{\!\nu}
\!\!&=\,
\nu^{(m{+}n-2)} \; \mathop{\;\sum \;\sum\;}_{%
\mathclap{\substack{
{x \in \{\zsymb,\osymb\}^m \!,}
\,\;
{y \in \{\zsymb,\osymb\}^n} \\
w(x) \!\!\;+\!\!\; w(y) \,\in\, 2\Z
\;\;
\ket{y}\!\!\bra{x} \!\!\:,
% -----------------------------------------------------
\\[-.5ex]
% -----------------------------------------------------
\text{--- more generally,}
\quad
\Biggsem{\!\!\!\!\!\tikzfig{ZX-red-phase-dot-arity}\!\!\!\!\!}_{\!\nu}
\!&=\,
\nu^{-(m{+}n-2)} \Bigl(
\ket{\psymb}^{\!\otimes n}\!\bra{\psymb}^{\!\!\;\otimes m}
\,+\;
\e^{i\theta}
\ket{\msymb}^{\!\otimes n}\!\bra{\msymb}^{\!\!\;\otimes m}
\Bigr);
\end{align}

\begin{gather}
\mspace{-30mu}
\Biggsem{\!\!\!\!\tikzfig{ZH-H-phase-box-arity}\!\!\!}_{\!\nu}
\!\!\!\!\:=\,
\nu^{(m{+}n)} \;\!
\mathop{\;\sum \;\sum\;}_{%
\mathclap{
{x \in \{\zsymb,\osymb\}^m \!,}
\,\;
{y \in \{\zsymb,\osymb\}^n}
\,
a^{x_1 \cdots x_m y_1 \cdots y_n} \!
\ket{y}\!\!\bra{x},
% -----------------------------------------------------
\mspace{30mu}
% -----------------------------------------------------
\Bigsem{\tikzfig{ZH-not-dot}}_\nu
\!\!\!\:=\,
\text{\footnotesize$\begin{bmatrix}
0 \!\!&\!\! 1 \\[-0.25ex] 1 \!\!&\!\! 0
\end{bmatrix}$},
% -----------------------------------------------------
\mspace{36mu}
% -----------------------------------------------------
\Bigsem{\tikzfig{ZX-nu-box-k}\!}_\nu
\!\!\!\:=\,
\nu^k
\!\!\;.\;
% -----------------------------------------------------
\mspace{-30mu}
\end{gather}