# WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-
Image Diffusion Models

Changhoon Kim
Arizona State University
<EMAIL_ADDRESS>
&Kyle Min11footnotemark: 1
Intel Labs
<EMAIL_ADDRESS>
Maitreya Patel
Arizona State University
<EMAIL_ADDRESS>
&Sheng Cheng
Arizona State University
<EMAIL_ADDRESS>
&Yezhou Yang
Arizona State University
<EMAIL_ADDRESS>
These authors contributed equally to this work.

###### Abstract

The rapid advancement of generative models, facilitating the creation of
hyper-realistic images from textual descriptions, has concurrently escalated
critical societal concerns such as misinformation. Traditional fake detection
mechanisms, although providing some mitigation, fall short in attributing
responsibility for the malicious use of synthetic images. This paper
introduces a novel approach to model fingerprinting that assigns
responsibility for the generated images, thereby serving as a potential
countermeasure to model misuse. Our method modifies generative models based on
each user’s unique digital fingerprint, imprinting a unique identifier onto
the resultant content that can be traced back to the user. This approach,
incorporating fine-tuning into Text-to-Image (T2I) tasks using the Stable
Diffusion Model, demonstrates near-perfect attribution accuracy with a minimal
impact on output quality. We rigorously scrutinize our method’s secrecy under
two distinct scenarios: one where a malicious user attempts to detect the
fingerprint, and another where a user possesses a comprehensive understanding
of our method. We also evaluate the robustness of our approach against various
image post-processing manipulations typically executed by end-users. Through
extensive evaluation of the Stable Diffusion models, our method presents a
promising and novel avenue for accountable model distribution and responsible
use. Our code, data, and demo will be available at: wouaf.vercel.app

##  1 Introduction

Recent advancements in generative models have propelled their proficiency,
expanding their repertoire to include not just the generation of
photorealistic images [12, 5] but also the synthesis of images from textual
prompts [19, 26, 22, 24]. These significant strides have equipped individuals
with the capacity to leverage these models to create hyper-realistic images
that correspond seamlessly with given textual instructions.

Nonetheless, the escalating prominence of generative models instigates
pressing societal apprehensions. A case in point is Deepfake, intentionally
crafted to disseminate misinformation, fostering a climate of fake news and
political disarray [1, 21, 15]. The gravity of these concerns necessitates
calls for governmental intervention to regulate the indiscriminate application
of generative models [2, 18]. An immediate, albeit intuitive, countermeasure
is the development of fake detection mechanisms [29] capable of discerning
real from synthetic content. However, these methodologies could inadvertently
serve as additional constraints propelling the creation of increasingly
realistic generative models [32, 4]. More significantly, these passive
approaches fall short of attributing accountability for the malicious
deployment of synthetic images.

A plausible approach to counteracting malicious use involves assigning
responsibility for the generated images. One method to realize this is by
integrating independent fingerprinting modules (discrete wavelet transform and
RivaGAN [33]), which can embed user-specific information into each image post-
creation. The open-source Text-to-Image (T2I) project, Stable Diffusion (SD)
[24], currently utilizes this technique. Obviously, simply disabling the
fingerprinting module can undermine its efficacy.

Can we conduct user attribution without an additional fingerprinting module?
In this paper, we present such a model fingerprinting technique to accomplish
responsible disclosure. In this approach, the model inventor or distributor
modifies the standard generative models based on each user’s distinctive
digital fingerprint. By adjusting the generative models, particularly their
weights, the resultant content carries a unique fingerprint that can be
accurately decoded and traced back to the responsible user.

Model fingerprinting techniques differ depending on the party embedding the
fingerprint: the model inventor or the distributor. Inventor-oriented methods
[30, 31], require application at the training outset, rendering them
incompatible with existing distributor repository models. This constraint
necessitates model inventors to balance generation quality and fingerprint
embedding. Distributor-oriented methods [14, 20], induce perturbations in the
model weights or estimated latent space to achieve fingerprinting. However,
within the scope of T2I generative models, estimating learned latent
representations becomes particularly challenging due to the expansive space of
conditioning text inputs.

Figure 1: Illustration of user attribution based on our method. Please refer
to the main text for detailed descriptions.

In our pursuit of effective model fingerprinting, we put forth a novel
distributor-oriented approach, termed as WOUAF: Weight mOdulation for User
Attribution and Fingerprinting. In practical terms, when a model inventor
open-sources their work to a model distributor such as Huggingface, the
distributor could utilize our proposed method to create a generic version.
Upon receiving a download request from an end-user, the distributor can adjust
the model weights using our technique and deploy a fingerprinted version to
the user, simultaneously registering the user’s fingerprint into their
database. In the event of a model’s malicious exploitation, the distributor
can decode the fingerprint from the misused image and cross-reference it with
their database to identify the responsible user. Consequently, this provides
the distributor with an actionable method to counteract malicious uses of the
model ( Fig. 1 depicts a comprehensive framework of our methodology.)

Our methodology seamlessly integrates fine-tuning into T2I tasks, leveraging
the SD framework. Aligning with previous research, our primary objective is to
ensure high attribution accuracy while minimizing degradation in output
quality. To achieve this, we employ a mapping network to encode user-specific
fingerprints into a discernible fingerprint representation. This is followed
by applying a weight modulation method using fingerprint representation.
Simultaneously, during the fine-tuning process of the decoder, we train a
fingerprint decoder capable of mapping images back to their corresponding
user-specific fingerprints (Refer to Sec. 3 for details) .

To validate our approach, we focus on Stable Diffusion model [24], a benchmark
within the open-source T2I models. Given that our method necessitates no
structural modifications, end-users cannot disable our method. Our evaluation
process is four-pronged. Initially, we gauge attribution accuracy and quality,
where our method demonstrates near-perfect attribution accuracy while
incurring minimal quality degradation. Subsequently, we probe the trade-off
between fingerprint length and attribution accuracy. Our method’s secrecy is
also scrutinized under two separate scenarios: 1) when a malicious user
endeavors to detect the existence of a fingerprint and 2) when a malicious
user has a comprehensive understanding of our approach. Finally, we assess the
robustness of our method in the face of various image post-processing
manipulations that end-users can perform.

#### Contributions.

(1) We present a novel distributor-centric model fingerprinting technique
(WOUAF) that integrates fine-tuning into T2I tasks, ensuring high attribution
accuracy with retained output quality; (2) We devise a practical framework for
model distributors to deliver fingerprinted models, enabling user
identification in case of model misuse; (3) We rigorously evaluate our method
using the Stable Diffusion model, highlighting its attribution accuracy, the
trade-off between fingerprint length and attribution accuracy, and robustness
against two specific attack scenarios and diverse image post-processings.

##  2 Related Work

#### Inventor-oriented Model Fingerprinting.

Yu et al. [31] leveraged a pre-trained deep steganography model to embed
fingerprints into the training set for fingerprinted GANs. However, this
approach suffers from limited scalability, as it necessitates training a GAN
from scratch for each distinct fingerprint. To address this issue, Yu et al.
[30] introduced a weight modulation method [10] that directly embeds a user’s
fingerprint into the generator’s weights.

Nevertheless, the aforementioned methods have been exclusively examined on
GAN-based models and require training from scratch. This raises crucial
questions regarding the feasibility of embedding fingerprints into diffusion-
based models and the possibility of avoiding training from scratch. These
inquiries are particularly pertinent given that state-of-the-art diffusion
models, such as those based on latent diffusion, exhibit different structures
compared to GANs. By employing fine-tuning for key embedding, model
distributors can conveniently integrate fingerprints into pre-trained
generative models. This approach allows model inventors to concentrate on the
model itself, as they need not consider fingerprints during the training
process, which would otherwise introduce further complexity.

#### Distributor-oriented Model Fingerprinting.

Kim et al. [14] proposed a technique for achieving user attribution by
explicitly incorporating user-specific fingerprints into the generator’s
output. While this simplified attribution method allowed for the derivation of
sufficient fingerprint conditions, it necessitated a trade-off between the
quality of the generated output and attribution accuracy, which was further
exacerbated when post-processing was taken into account. To tackle this issue,
an approach has been proposed that utilizes subtle semantic variations along
latent dimensions as fingerprints, generated by perturbations of eigenvectors
in the latent distribution [20]. This method demonstrated an improved balance
between generation quality and attribution accuracy. However, its
applicability is restricted to unconditional image generation, as eigenvectors
are computed by sampling the learned latent representation—a data-specific
representation. In the context of conditional image generation, estimating
eigenvectors of latent representation becomes challenging due to the vast
space of conditions, such as those found in text conditions.

#### Stable Diffusion Models.

Our approach is implemented using Stable Diffusion Models (SD) [24], which
function within an autoencoder’s latent space. Notably, structural alterations
to the model aren’t necessitated for our method.

SD is comprised of two primary components: First, an autoencoder is pretrained
on an extensive dataset of images. The encoder,
$\mathcal{E}(\cdot):\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{z}}$, maps an
image $x\sim p_{data}$ into a latent representation $z=\mathcal{E}(x)$, while
the decoder
$\mathcal{D}(\cdot):\mathbb{R}^{d_{z}}\rightarrow\mathbb{R}^{d_{x}}$
reconstructs the original image from the latent representation, yielding
$x\approx\hat{x}=\mathcal{D}(z)$. The second component, a U-Net [25] based
diffusion model, is trained to generate latent representations within the
learned latent space. This diffusion model can be conditioned on class labels
or trained text-embeddings. Let $c(y)$ denote a model mapping a conditioning
input $y$ to a conditioning vector. The SD loss is consequently defined by:

$L_{SD}:=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim
N(0,1),t}\left[||\epsilon-\epsilon_{\theta}(z_{t},t,c(y)||_{2}^{2}\right],$
(1)

where $t$ represents the time step, $\epsilon$ the unscaled noise sample,
$z_{t}$ the noised $z$ up to time step $t$, and $\epsilon_{\theta}$ the
denoising network. During the inference phase, a random noise is sampled and
sequentially denoised using a scheduler with predetermined time steps $T$ to
yield a new image latent, $z_{0}$. This latent code is then transformed into
an image via the pretrained decoder, $\hat{x}=\mathcal{D}(z_{0})$.

(a) The overall pipeline. (b) Weight modulation.

Figure 2: Depiction of our method’s pipeline and weight modulation: (a) The
model fingerprinting procedure encompasses encoding via the mapping network
and weight modulation, along with decoding through the fingerprint decoding
network. (b) Weight modulation of the decoding network $\mathcal{D}$ to
incorporate the fingerprint.

##  3 Methods

In this study, we embrace the perspective of a responsible model distributor,
equipped to trace the provenance of generated images effectively. Our approach
involves fine-tuning existing SD through a weight modulation technique,
transforming network weights into user-specific weights, hence, embedding
fingerprint information into the output images. Concurrently, a fingerprint
decoding network is trained during this fine-tuning process to extract
fingerprints from images (Fig. 2(a)). Upon deployment, users are provided with
their fingerprinted models, while the mapping network and fingerprint decoding
network remains exclusively with the model distributor.

###  3.1 User-specific Weight Modulation

Our method is fundamentally based on integrating fingerprints into the
parameters of the SD through weight modulation [10, 30].

A user-specific fingerprint is drawn from a Bernoulli distribution with a
probability of 0.5, represented as $\phi\in\Phi:=\text{Bern}(0.5)^{d_{\phi}}$,
where $d_{\phi}$ signifies the fingerprint length in bits. We employ a mapping
network $M(\cdot):\mathbb{R}^{d_{\phi}}\rightarrow\mathbb{R}^{d_{M}}$ to
convert the sampled fingerprint $\phi$ into an intermediate fingerprint
representation within the $d_{M}$ dimension. For modulating each layer in the
SD component, we introduce an affine transformation layer,
$A_{l}(\cdot):\mathbb{R}^{d_{M}}\rightarrow\mathbb{R}^{d_{j}}$, for all layers
$l$. This transformation matches the dimensions between $d_{M}$ and the $j$-th
channel in weight $W\in\mathbb{R}^{i,j,k}$, where $i,j,k$ denote input,
output, and kernel dimensions, respectively (Refer to Fig. 2(b)). The weight
modulation for the $l$-th layer is defined as:

$W_{i,j,k}^{\phi}=u_{j}*W_{i,j,k},$ (2)

where $W$ and $W^{\phi}$ denote the pre-trained and fingerprinted weights
respectively, $u_{j}=A_{l}(M(\phi))$ is the scale of the fingerprint
representation corresponding to the $j$th output channel.

We incorporate fingerprints into the SD by applying weight modulation
exclusively to the weights in the decoder $\mathcal{D}$. The rationale for not
applying modulation to both the diffusion model $\epsilon_{\theta}$ and
decoder $\mathcal{D}$, an approach that mirrors GAN-based models [30], is
explained in Sec.4.4.

###  3.2 Training Objectives

Our training architecture comprises two primary objectives. The initial
objective is to decode fingerprints from the provided images. We train a
ResNet-50 [6]-based fingerprint decoding network,
$\mathcal{F}(\cdot):\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{\phi}}$, as
follows:

$L_{\phi}=\mathbb{E}_{z\sim
E(x),\phi\in\Phi}\sum_{i=1}^{d_{\phi}}[\phi_{i}\log\sigma(\mathcal{F}(\mathcal{D}(\phi,z))_{i}+(1-\phi_{i})\log(1-\sigma(\mathcal{F}(\mathcal{D}(\phi,z)))_{i}],$
(3)

where $\sigma(\cdot)$ refers to the sigmoid activation function, constraining
the output of $\mathcal{F}$ to the range $[0,1]$. Thus, this loss function
effectively amalgamates binary cross-entropy for all bits of the fingerprint.

During training time, fingerprint $\phi$ is sampled from Bernoulli
distribution. However, during distribution, the model distributor initially
samples a user-specific fingerprint $\phi_{\alpha}$ and subsequently modulates
the decoder $\mathcal{D}$ using $\phi_{\alpha}$. The model the user ultimately
receives is the fingerprinted decoder $\mathcal{D}(\phi_{\alpha},\cdot)$,
which solely permits latent input.

The secondary objective endeavors to regularize the quality of outputs.
Ideally, this regularization objective inhibits the decoder $\mathcal{D}$ from
compromising image quality while minimizing $L_{\phi}$ in (3):

$L_{quality}=\mathbb{E}_{z\in\mathcal{E}(x),\phi\in\Phi}\left[\ell(x,\mathcal{D}(\phi,z))\right],$
(4)

$\ell$ represents the distance metric between original images and
fingerprinted images. For practical applications, we utilize perceptual
distance [34] to gauge the perceptual difference between $x$ and
$\mathcal{D}(\phi,z)$.

The final objective function can be formulated as:

$\min_{A,M,D,F}\lambda_{1}L_{\phi}+\lambda_{2}L_{quality},$ (5)

where both $\lambda_{1}$ and $\lambda_{2}$ are set to $1.0$. Fundamentally,
the loss function aspires to reconstruct fingerprints while maintaining the
quality of the generated outputs. To assess the efficacy of our proposed
method, we employ attribution accuracy and image quality metrics (Refer to
Sec. 4.1 for details).

##  4 Experiments

###  4.1 Experiment Settings

#### Datasets.

Our approach was fine-tuned on the MS-COCO [16] dataset, adopting the Karpathy
split. For methodological evaluation, we harnessed the test set from MS-COCO
and randomly sampled from the LAION-aesthetics [27] dataset. To demonstrate
the applicability of our methodology, we further incorporated the ImageNet [3]
dataset.

#### Evaluations.

User attribution accuracy, an indicator of the consistent incorporation and
successful decoding of fingerprints from the output image, is defined as:
$\frac{1}{d_{\phi}}\sum_{i=1}^{d_{\phi}}\mathbbm{1}(\phi_{i}=\hat{\phi}_{i})$,
where $\phi$ represents the ground-truth fingerprint and
$\hat{\phi}=\mathbbm{1}(\sigma(\mathcal{F}(x_{\phi}))>0.5)$ denotes the
estimated fingerprint from the image $x_{\phi}$ corresponding to $\phi$.
Unless otherwise stated, we set $d_{\phi}$ to 32 for all experiments (Refer to
Sec. 4.2 for further information).

The quality of the generated images is another pivotal evaluation aspect,
reflecting our methodology’s efficacy in maintaining image quality whilst
embedding the fingerprints. A decline in image quality could dissuade users
from downloading and utilizing the model. To numerically represent the quality
of outputs, we generated 5,000 images using text prompts from both the MS-COCO
test set and LAION aesthetics datasets. We employed the Fréchet Inception
Distance (FID)[8] to evaluate image quality, while the Clip-score[7] was
utilized to assess the concordance between the text prompt and the generated
image.

#### Models.

To benchmark attribution accuracy, we utilized the default fingerprinting
module in the official SD, denoted as the Discrete Wavelet Transform-based
method (DWT) and RivaGAN [33]. Importantly, these modules serve as an upper
limit for our method, as they are explicitly designed for fingerprinting,
whereas our approach strives to maintain T2I generation capabilities alongside
fingerprinting. We also compared the output images from our method with those
from the original SD to assess the qualitative changes induced by our
approach, providing an upper limit for image quality. For T2I generation, we
adopted the Euler scheduler [13] with timestep $T=20$, and the classifier-free
guidance scale [9] was set to 7.5 unless otherwise specified.

Finally, we compared our methodology with three variants based on the specific
layers chosen for weight-modulation implementation. The first variant (Ours-
conv.) applied the modulation solely to the convolutional layers in
$\mathcal{D}$, reflecting the original proposal for weight modulation [10].
The second variant (Ours-all) implemented weight modulation across all layers
in $\mathcal{D}$, encompassing self-attention and convolution layers. The last
variant applied weight modulation to both the diffusion model
$\epsilon_{\theta}$ and decoder $\mathcal{D}$, similar to the GAN-based method
[30]. The reasons for not utilizing this variant throughout all experiments
are elaborated upon in Section 4.4.

Table 1: Assessment of attribution accuracy and generation quality. Our method was validated on both the MS-COCO test set and the LAION-aesthetics dataset, which was not employed during our fine-tuning phase. Symbols $\uparrow$/$\downarrow$ indicate that higher/lower values are desired, respectively. Model | MS-COCO | LAION
---|---|---
Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$) |  Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$)
Original SD [24] |  - |  0.73 |  24.48 |  - |  0.50 |  19.67
DWT |  0.84 |  0.74 |  24.82 |  0.86 |  0.50 |  20.64
RivaGAN [33] |  0.99 |  0.73 |  24.70 |  0.99 |  0.51 |  20.54
Ours-conv. |  0.99 |  0.73 |  24.43 |  0.98 |  0.51 |  20.46
Ours-all |  0.99 |  0.73 |  24.42 |  0.99 |  0.51 |  19.91

###  4.2 Fingerprint Capacity

Figure 3: Fingerprint Capacity.

The capacity of our methodology is determined by the maximum count of unique
user-specific fingerprints it can sustain without encountering detrimental
crosstalk. The primary influencer of this capacity is the dimensionality of
the fingerprint dimension ($d_{\phi}$). However, the task of identifying an
optimal $d_{\phi}$ is challenging, as a larger $d_{\phi}$ paves the way for
accommodating more users but simultaneously increases the complexity of
effective fingerprint decoding.

To methodically explore this balance, we carried out a thorough analysis,
varying the fingerprint dimensionality ($d_{\phi}$) among a set of
predetermined values: ${16,32,64,128}$. Fig. 3 encapsulates the user
attribution accuracy corresponding to each $d_{\phi}$ value. As illustrated in
the figure, there’s a monotonic decline in the attribution accuracy as
$d_{\phi}$ enlarges.

Interestingly, both our variant models achieved a near-perfect attribution
accuracy of 0.99 for $d_{\phi}$ values of 16, 32, and 64. However, for
$d_{\phi}=128$, Ours-all variant outperformed Ours-conv. variant. For a
balanced comparison with existing methods, we chose $d_{\phi}=32$, which
notably can support a substantial user base exceeding $4*10^{9}\approx
2^{32}$.

An important observation from our experimental procedures is the dependency of
training time on $d_{\phi}$. Larger $d_{\phi}$ values necessitate extended
convergence periods. For example, both our variant models, with $d_{\phi}=32$,
demanded a training duration of 22 hours on a single A100 GPU.

Table 2: Assessment of attribution accuracy and generation quality using Imagenet trained models. We validated our method using MS-COCO testset and LAION-aesthetics dataset. $\uparrow$/$\downarrow$ indicates higher/lower is desired. Model | MS-COCO | LAION
---|---|---
Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$) |  Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$)
Ours-conv. |  0.99 |  0.73 |  24.23 |  0.99 |  0.51 |  19.71
Ours-all |  0.99 |  0.73 |  24.41 |  0.99 |  0.51 |  19.46

(a) MS COCO (b) LAION Aesthetics

Figure 4: Qualitative comparison of the original and fingerprinted Stable
Diffusion models on MS-COCO and LAION aesthetics (Pixel-wise
differences$\times$ 5: they are multiplied by a factor of 5 for better view).
We can observe that our method maintains high image quality.

###  4.3 Attribution Accuracy and Image Quality

We conduct a comprehensive evaluation of WOUAF, focusing on attribution
accuracy and image quality. The assessment involved the MS-COCO test set and
the LAION-aesthetics dataset, which was not used during the fine-tuning stage.
The results, shown in Table 1, highlight the robust performance of our
methodology.

Our variants, Ours-conv and Ours-all, demonstrated superior attribution
accuracy compared to DWT, indicating successful decoding of embedded
fingerprints from the output images. This strong performance was maintained
even when compared to RivaGAN, a specialized fingerprinting model, reinforcing
the validity of our approach. Crucially, this level of attribution accuracy
was achieved without significant image quality degradation. Deviations in FID
scores and Clip-scores compared to the original SD were minimal, indicating
negligible impact on output image quality, as further evidenced by qualitative
examples in Figure 4. These results underscore the efficacy of WOUAF in
applications that necessitate reliable fingerprinting while maintaining high-
quality image generation. Uncurated image collections can be found in the
supplementary material.

#### Evaluating Generalizability Across Datasets.

A key feature of our proposed methodology is its design independence from
image-text paired datasets for achieving attribution accuracy. This property
imbues it with the potential for broad applicability across a diverse range of
contexts. To substantiate this claim, we conducted an experiment in which our
variant models were trained exclusively on the ImageNet dataset. We
subsequently evaluated the performance of these ImageNet-trained models on the
MS-COCO test set as well as a randomly selected portion of the LAION-
aesthetics datasets.

The evaluation results, as seen in Table 2, effectively corroborate our
assertion. Our methodology demonstrates compelling performance, with both our
variants, achieving high attribution accuracy and maintaining image generation
quality. These results underscore our method’s independence from the use of
text-image paired datasets, thereby establishing its broad applicability in
diverse scenarios where reliable fingerprinting and high-quality image
generation are required. Additional samples can be found in the supplementary
materials.

#### Robustness Across Generation Hyperparameters.

In the exploration of model fingerprinting, we acknowledge that users, once in
possession of their fingerprinted SD model, may choose to manipulate a variety
of hyperparameters in image generation. To ensure the robustness of our
proposed approach, we have conducted a comprehensive set of experiments with
varying parameters such as the scheduler, number of time steps, and guidance
scale. These parameters can significantly influence the quality of images
generated, making it essential to thoroughly validate our method across an
array of configurations.

To substantiate our approach, we evaluated two widely used schedulers: Euler
[13], with time steps in [15, 20, 25], and DDIM [28], with time steps in [45,
50, 55], and guidance scales of 2.5, 5.0, and 7.5. Our findings underscore the
resilience and consistency of our method, demonstrating high attribution
accuracy regardless of the chosen scheduler or variations in the number of
diffusion steps, $T$. These results, which align with those depicted in Table
1, are further elaborated upon through detailed tables and figures in
supplementary material, providing a more granular understanding of our
evaluations.

###  4.4 Benefits of Finetuning only Decoder

When developing our last variant that incorporates weight modulation into both
the diffusion model $\epsilon_{\theta}$ and the decoder $\mathcal{D}$, we
noted that the resultant pipeline demonstrates similarities with the GAN-based
method [30]. A direct comparison between ours and GAN based method may not be
entirely straightforward, considering the fundamental differences in their
training methodologies – GAN-based methods entail training from scratch,
whereas our proposed approach leans towards fine-tuning. Nevertheless, both
methodologies share a common mechanism: they aim to modulate the weights of
the layers instrumental in learning the latent space. This shared
characteristic underscores the fundamental objective of optimizing the balance
between attribution accuracy and generation quality.

However, our empirical observations suggest that this variant does not
consistently achieve commendable performance as an attribution model.
Specifically, it appears that this variant can only optimize either
attribution accuracy or generation quality, but not both simultaneously. In
our experiments, the maximum attribution accuracy achieved by this variant was
approximately 89% (Clip-score: 0.68, FID: 63.48). The inherent trade-off
observed here further reinforces the challenge of balancing these two critical
parameters in the context of model fingerprinting techniques. We added sample
images of this variant in the supplementary materials.

###  4.5 Secrecy

The imperceptibility of the embedded fingerprint is of utmost importance, as
its detection might enable malicious users to erase or tamper with the
fingerprint, thereby confounding the fingerprint decoding network. To evaluate
the secrecy of our methodology, we consider two attack scenarios, similar to
the settings described in [30].

Table 3: User attribution accuracy under attack scenarios on MS-COCO. Method | Scenario 1 | Scenario 2
---|---|---
Ours-conv. | 0.662 | 0.509
Ours-all | 0.566 | 0.501

In the first scenario, the attacker seeks to train a classifier capable of
detecting the presence of a fingerprint. To assess this scenario, we employ a
ResNet-50 [6] based binary classifier, trained using 10K SD generated images
(5K original SD images and 5K fingerprinted SD images). This configuration is
deemed valid, as detecting the presence of a fingerprint necessitates using
both non-fingerprinted and fingerprinted images in the training set. The
binary classifier achieved 98% accuracy in the training stage. Subsequently,
we evaluated its performance using an independent set of 5K images from our
variant model. As shown in Tab. LABEL:tab:secrecy, the user attribution
accuracy achieved was only 0.56 for Ours-all variant model, which is almost
equivalent to random.

The second scenario assumes an internal attacker with comprehensive knowledge
of our training process, including the training dataset, model structure,
fingerprint space, and training details. To validate this, we trained an
attacker’s version of our model, following our methodology but employing a
different random seed. We then assessed user attribution accuracy by inputting
5K images generated by the attacker’s model into our variants’ fingerprint
decoding networks. As demonstrated in Tab. LABEL:tab:secrecy, both of our
model variants exhibited user attribution accuracies of 0.509 and 0.501, which
are essentially random guesses, and thus dodged the attack.

In conclusion, our findings indicate that our embedded fingerprint,
particularly Ours-all variant, is difficult to detect. Moreover, even when an
attacker with complete knowledge replicates our methodology, they will not be
able to mislead the original fingerprint decoding network.

Figure 5: Robust user attribution against image post-processes.

###  4.6 Robust User Attribution against Image Post-processes

In this section, we assess the robustness of our approach under scenarios
where generated images are subjected to post-processing. These processes may
intentionally or unintentionally modify the embedded fingerprint. Given the
noteworthy performance of the Ours-all variant in the secrecy evaluation
(Section 4.5), we narrow our focus to this particular variant in the
subsequent tests.

In line with the protocols detailed in prior works [14, 30, 31, 20], we
evaluate the robustness of our model against a variety of image post-
processing operations. We examine two versions of our model: the original and
a robust variant. The robust variant is constructed by simulating the impact
of post-processing, applied with random intensities, prior to feeding data
into the fingerprint decoding network, $\mathcal{F}$:

$L_{robust}=\mathbb{E}_{z\sim
E(x),\phi\in\Phi}\sum_{i=1}^{d_{\phi}}[\phi_{i}\log\sigma(\mathcal{F}(T(\mathcal{D}(\phi,z)))_{i}+(1-\phi_{i})\log(1-\sigma(\mathcal{F}(T(\mathcal{D}(\phi,z)))_{i}],$
(6)

where $T(\cdot):\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{x}}$ represents
the post-processing function. We resolve an optimization problem similar to
Eq. (5), but with $L_{\phi}$ replaced by $L_{robust}$.

In our exploration, we contemplate eight different post-processing techniques:
Cropping, Rotation, Gaussian Blurring, Brightness jittering, the addition of
Gaussian Noise, Erasing, JPEG compression, and a Combination of all these
techniques. The parameters for these post-processes are designed as follows:
For Cropping, we use a random crop-out ratio within the range
$[5\%,10\%,15\%,20\%]$. Rotation involves randomly sampling a degree within
the range (-30, 30). For Gaussian Blurring, we randomly select a kernel size
from [3,5,7]. The Brightness factor is randomly sampled within the range
(-0.3, 0.3). For Gaussian Noise, we add noise with a standard deviation
randomly sampled from a uniform distribution $U[0,0.2]$. Random Erasing is
selected from the range $[5\%,10\%,15\%,20\%]$. JPEG compression quality level
is selected from [90, 80,70,60,50]. The Combination technique randomly selects
a subset of these seven post-processing methods with a probability of 0.5.

We evaluated the user attribution accuracy for each post-process during
testing, adhering to the pre-defined parameters. For the Combination approach,
we crafted the attack strength level grounded on the parameters of the other
seven procedures. Our robustness test outcomes, delineated in Fig. 5, present
a comparative analysis of user attribution accuracy across two benchmark
models, our original model, and the robust version of our model. We can
observe that our method has a high degree of robustness across various types
of real-world attack scenarios.

We noted a consistent trend across all transformation techniques: the user
attribution accuracy decreases monotonically with an increase in the strength
of post-processing parameters. This observation affirms the challenge posed by
post-processing transformations on the task of accurate user attribution.

However, our findings also highlight the efficacy of robust training in
mitigating this challenge. Across all cases, we observed a significant
improvement in attribution accuracy when we incorporated robust training in
our model. This underscores the crucial role of robust training in enhancing
the resilience of our fingerprinting method against post-processing
transformations. But still, when specifically targeting robust attribution
against a Combination of post-processes, we noticed a trade-off, as the FID
score degraded from 24.42 to 26.85, indicating that the optimization for
robustness might unintentionally affect image quality.

Our robust model variant demonstrates a commendable ability to preserve
attribution accuracy, frequently matching the performance of established
baseline methods under various transformations such as Rotation, Noise,
Erasing, and Combinations. This suggests a promising capacity of our method to
resist diverse post-processing operations without substantially compromising
the accuracy of user attribution. It is pertinent to note, however, that while
our findings are encouraging, they warrant further investigation and potential
enhancements for complete validation and optimization. Further results and
visual representations pertaining to both of our model variants can be found
in the supplementary materials.

##  5 Conclusion

In this paper, we study stable diffusion based I2T models’ user attribution
through weight modulation, achieving near-perfect accuracy without sacrificing
quality. Importantly, our WOUAF model’s performance is not reliant on text-
image paired datasets and demonstrates robustness against a variety of image
generation parameters and secrecy attack scenarios. Our proposed loss function
further strengthens the model’s resilience against diverse post-processing
techniques. Our findings contribute to the foundation for future research to
address challenges presented by generative models at large.
Limitations and future directions: Despite the encouraging results, the user
attribution accuracy of our model decreases with increased and heavy post-
processing transformations, indicating a need for incorporating other advanced
weight modulation techniques. We leave testing and analyzing our methodology
to other data types such as text, audio, and video as future work,
necessitating novel model fingerprinting alterations. Additionally, in-depth
exploration and analysis into boosting robustness against post-processing are
required.

##  6 Acknowledgment

This work is partially supported by the National Science Foundation under
Grant No. 2038666, No. 2101052, and a grant from Meta AI Learning Alliance.
The views and opinions of the authors expressed herein do not necessarily
state or reflect those of the funding agencies and employers.

## References

  * Breland [2019] Ali Breland.  The bizarre and terrifying case of the “deepfake” video that helped bring an african nation to the brink.  _motherjones_ , Mar 2019.  URL https://www.motherjones.com/politics/2019/03/deepfake-gabon-ali-bongo/.
  * Burri and von Bothmer [2021] Thomas Burri and Fredrik von Bothmer.  The new eu legislation on artificial intelligence: a primer.  _Available at SSRN 3831424_ , 2021.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Durall et al. [2020] Ricard Durall, Margret Keuper, and Janis Keuper.  Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7890–7899, 2020.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  _arXiv preprint arXiv:2104.08718_ , 2021.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _Advances in Neural Information Processing Systems_ , pages 6626–6637, 2017.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Karras et al. [2019] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  _arXiv preprint arXiv:1912.04958_ , 2019.
  * Karras et al. [2020a] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  _Advances in neural information processing systems_ , 33:12104–12114, 2020a.
  * Karras et al. [2020b] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8110–8119, 2020b.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _arXiv preprint arXiv:2206.00364_ , 2022.
  * Kim et al. [2021] Changhoon Kim, Yi Ren, and Yezhou Yang.  Decentralized attribution of generative models.  In _International Conference on Learning Representations_ , 2021.
  * Lajka [2023] Arijeta Lajka.  New ai voice-cloning tools ‘add fuel’ to misinformation fire.  _AP News_ , Feb 2023.  URL https://apnews.com/article/technology-science-fires-artificial-intelligence-misinformation-26cabd20dcacbd68c8f38610fec39f5b.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * McCabe [2023] David McCabe.  White house pushes tech c.e.o.s to limit risks of a.i.  _New York Times_ , May 2023.  URL https://www.nytimes.com/2023/05/04/technology/us-ai-research-regulation.html.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning_ , pages 16784–16804. PMLR, 2022.
  * Nie et al. [2023] Guangyu Nie, Changhoon Kim, Yezhou Yang, and Yi Ren.  Attributing image generative models using latent fingerprints.  _arXiv preprint arXiv:2304.09752_ , 2023.
  * Novak [2023] Matt Novak.  Ai image creator midjourney halts free trials but it has nothing to do with the pope’s jacket.  _forbes_ , Mar 2023.  URL https://www.forbes.com/sites/mattnovak/2023/03/31/ai-image-creator-midjourney-halts-free-trials-but-it-has-nothing-to-do-with-the-popes-jacket/?sh=ed3f3ec50289.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Riba et al. [2020] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski.  Kornia: an open source differentiable computer vision library for pytorch.  In _Winter Conference on Applications of Computer Vision_ , 2020.  URL https://arxiv.org/pdf/1910.02190.pdf.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695, June 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _arXiv preprint arXiv:2210.08402_ , 2022.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Wang et al. [2019] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros.  Cnn-generated images are surprisingly easy to spot… for now.  _arXiv preprint arXiv:1912.11035_ , 2019.
  * Yu et al. [2020] Ning Yu, Vladislav Skripniuk, Dingfan Chen, Larry Davis, and Mario Fritz.  Responsible disclosure of generative models using scalable fingerprinting.  _arXiv preprint arXiv:2012.08726_ , 2020.
  * Yu et al. [2021] Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz.  Artificial fingerprinting for generative models: Rooting deepfake attribution in training data.  In _Proceedings of the IEEE/CVF International conference on computer vision_ , pages 14448–14457, 2021.
  * Zhang et al. [2020] Baiwu Zhang, Jin Peng Zhou, Ilia Shumailov, and Nicolas Papernot.  On attribution of deepfakes.  _arXiv preprint arXiv:2008.09194_ , 2020.
  * Zhang et al. [2019] Kevin Alex Zhang, Lei Xu, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Robust invisible video watermarking with attention.  2019\.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.

##  Appendix A Training Details

Our proposed method was evaluated utilizing the Stable Diffusion (SD) model
[24] (version 2-base), trained specifically for generating images of 512px
resolution.

We implemented weight-modulation in alignment with the design specified in the
StyleGAN2-ADA [11] source code. Our mapping network $M$ was designed using
multiple fully connected layers, wherein all experiments were conducted using
a two-layered configuration. The dimension of the mapping network $d_{M}$ was
set to be equal to $4*d_{\phi}$ across all experimental setups.

The learning rate was set to $1e-4$ and optimized using AdamW [17]. Training
was performed over 50,000 iterations with a batch size of 32.

To train models that demonstrated robustness against image post-processing
transformations, differentiable post-processes were necessary. To this end, we
incorporated the Kornia library [23], which offers a collection of
differentiable augmentations.

##  Appendix B Additional Experimental Results

In addition to the figure in the main paper, we added uncurated images using
text-prompt from MS-COCO [16] and LAION Aesthetics [27]. For convenience, we
have aligned the subsection names with those in the main manuscript. Unless
otherwise specified, all figures were generated using the ’Ours-all’ method.

###  B.1 Attribution Accuracy and Image Quality

As highlighted in the main manuscript, our methodology has a negligible effect
on the original Stable Diffusion’s image quality. Please refer to Fig. 6 for
these uncurated images.

(a) MS COCO (b) LAION Aesthetics

Figure 6: Uncurated images of the original and fingerprinted Stable Diffusion
models on MS-COCO and LAION Aesthetics. Pixel-wise differences are multiplied
by a factor of 5 for a better view.

###  B.2 Evaluating Generalizability Across Datasets

This section presents qualitative results from our technique, which was fine-
tuned exclusively using the ImageNet dataset [3]. As underscored in the main
manuscript, our model fingerprinting strategy based on weight modulation can
be effectively employed using only an image-based dataset. Fig. 7 provides a
visual representation of these images.

(a) MS COCO (b) LAION Aesthetics

Figure 7: Qualitative comparisons of the original and fingerprinted Stable
Diffusion models that were fine-tuned using only the ImageNet dataset. Pixel-
wise differences are multiplied by a factor of 5 for a better view.

###  B.3 Robustness Across Generation Hyperparameters

In accordance with the details provided in the primary manuscript, we
subjected our methodology to evaluation employing two widely accepted
schedulers: Euler [13], featuring time steps at intervals of [15, 20, 25], and
DDIM [28], operating at time steps in [45, 50, 55]. Along with these, we also
incorporated classifier-free guidance scales [9] at 2.5, 5.0, and 7.5.

Echoing the discussions in the main paper, the data in Tables 4 and 5
corroborate the near perfect attribution accuracy achieved by our method.
Furthermore, the absence of significant deterioration in quality metrics
reaffirms the resilience of our approach in the face of diverse generation
hyperparameters (Refer to Fig. 8 and Fig. 9).

Table 4: Assessment of attribution accuracy and generation quality using Euler and DDIM scheduler with different time steps on MS-COCO. We fixed classifier-free guidance scale [9] to 7.5. $\uparrow$/$\downarrow$ indicates higher/lower is desired. Model |  Euler [13] |  DDIM [28]
---|---|---
Steps |  Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$) |  Steps |  Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$)
original SD [24] |  20 |  - |  0.73 |  24.48 |  50 |  - |  0.73 |  23.33
Ours-conv. |  15 |  0.99 |  0.73 |  24.63 |  45 |  0.99 |  0.73 |  23.28
20 |  0.99 |  0.73 |  24.43 |  50 |  0.99 |  0.73 |  23.35
25 |  0.99 |  0.74 |  24.14 |  55 |  0.99 |  0.73 |  23.31
Ours-all |  15 |  0.99 |  0.73 |  24.65 |  45 |  0.99 |  0.73 |  23.34
20 |  0.99 |  0.73 |  24.42 |  50 |  0.99 |  0.73 |  23.29
25 |  0.99 |  0.73 |  24.11 |  55 |  0.99 |  0.73 |  23.26
Table 5: Assessment of attribution accuracy and generation quality on different classifier-free guidance scales [9] using MS-COCO. We fixed the scheduler and time steps to Euler for 20 steps and DDIM for 50 steps. $\uparrow$/$\downarrow$ indicates higher/lower is desired. Model |  Guidance Scale [9] 2.5 |  Guidance Scale [9] 5.0
---|---|---
Scheduler |  Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$) |  Scheduler |  Attribution Acc. ($\uparrow$) |  Clip-score ($\uparrow$) |  FID ($\downarrow$)
Ours-conv. |  Euler |  0.99 |  0.72 |  18.63 |  Euler |  0.99 |  0.73 |  21.91
DDIM |  0.99 |  0.72 |  18.35 |  DDIM |  0.99 |  0.73 |  20.78
Ours-all |  Euler |  0.99 |  0.71 |  18.64 |  Euler |  0.99 |  0.73 |  21.89
DDIM |  0.99 |  0.71 |  18.31 |  DDIM |  0.99 |  0.73 |  20.68
Figure 8: Qualitative results obtained using the Euler and DDIM schedulers
with varying time steps on the MS-COCO dataset. We maintained a constant
classifier-free guidance scale [9] at 7.5. Each column corresponds to the
’Ours-all’ rows in Table 4. Figure 9: Qualitative results produced by applying
different classifier-free guidance scales [9] on the MS-COCO dataset. The
scheduler and time steps were held constant at Euler for 20 steps and DDIM for
50 steps. Each column aligns with the ’Ours-all’ rows in Table 5.

###  B.4 Benefits of Finetuning only Decoder

In this section, we present qualitative outcomes resulting from the joint
fine-tuning of the Stable Diffusion model’s components, diffusion model
$\epsilon_{\theta}$ and decoder $\mathcal{D}$. As accentuated in the primary
manuscript, our training protocol achieved an accuracy of 89%, however, it
resulted in a noticeable deterioration in the quality metrics (Clip-score:
0.68, FID: 63.48). Figure 10 provides a visual affirmation of these
quantitative results.

(a) MS COCO (b) LAION Aesthetics

Figure 10: Qualitative results of the original and fingerprinted Stable
Diffusion models on MS-COCO and LAION Aesthetics. When fine-tuning the SD
model’s $\epsilon_{\theta}$ and $\mathcal{D}$ together, there are significant
quality drops. Pixel-wise differences are multiplied by a factor of 5 for a
better view.

###  B.5 Robust User Attribution against Image Post-processes

As detailed in the main manuscript, we show the robust attribution accuracy
results of our variants (Ours-all and Ours-conv.), as depicted in Fig. 11. A
comparative analysis of our robust variants, Ours-all and Ours-conv.,
indicates comparable attribution accuracy. However, a slight overall
superiority can be attributed to the Ours-all variant.

We also present a comprehensive evaluation of quality metrics, illustrating
the impact of our robust user attribution training against individual image
post-processing methods (Refer to Tab. 6 and Tab. 7). The results indicate
that our robust fine-tuning generally maintains the image quality with minimal
perturbations. To substantiate this, we provide a representative image created
by our robust model under a JPEG attack (See Fig.12). Nonetheless, a
Combination attack results in a visible degradation of image quality.
Consequently, we provide images generated by our model under a Combination
attack (Fig. 13). We also provide images by Ours-conv. model under the two
types of attacks in Fig14.

Contrary to other figures (Fig.6,7,12), Fig.13 demonstrates the introduction
of significant noise when the model is fortified to withstand a Combination
attack. As elucidated in the main manuscript, this introduces a trade-off: the
enhanced robustness against the Combination attack is accompanied by a
noticeable decline in image quality.

Figure 11: Robust user attribution against image post-processes.  Table 6: FID [8] scores using MS-COCO after robust training. Lower is desired. Model |  Crop |  Rotation |  Blur |  Brightness |  Noise |  Erasing |  JPEG |  Combi.
---|---|---|---|---|---|---|---|---
Ours-conv. |  24.02 |  24.05 |  23.80 |  24.14 |  23.96 |  24.16 |  24.42 |  26.80
(robust)
Ours-all |  24.35 |  23.92 |  24.18 |  24.54 |  24.24 |  24.48 |  24.41 |  26.85
(robust)
Table 7: CLIP scores [7] using MS-COCO after robust training. Higher is desired. Model |  Crop |  Rotation |  Blur |  Brightness |  Noise |  Erasing |  JPEG |  Combi.
---|---|---|---|---|---|---|---|---
Ours-conv. |  0.716 |  0.717 |  0.716 |  0.716 |  0.712 |  0.717 |  0.714 |  0.702
(robust)
Ours-all |  0.719 |  0.717 |  0.718 |  0.718 |  0.710 |  0.718 |  0.716 |  0.704
(robust)

(a) MS COCO (b) LAION Aesthetics

Figure 12: Qualitative results of the original and fingerprinted Stable
Diffusion models on MS-COCO and LAION aesthetics. Our fingerprinted model is
trained by simulating JPEG compression during training. Pixel-wise differences
are multiplied by a factor of 5 for a better view.

(a) MS COCO (b) LAION Aesthetics

Figure 13: Qualitative results of the original and fingerprinted Stable
Diffusion models on MS-COCO and LAION aesthetics. Our fingerprinted model is
trained by simulating all the combinations of the post-processing during
training. Pixel-wise differences are multiplied by a factor of 5 for a better
view. Figure 14: Qualitative results of the original and fingerprinted Stable
Diffusion models (Ours-conv.) on MS-COCO. Pixel-wise differences are
multiplied by a factor of 5 for a better view.