# Steal My Artworks for Fine-tuning? A Watermarking Framework for Detecting
Art Theft Mimicry in Text-to-Image Models

Ge Luo, Junqiang Huang, Manman Zhang, Zhenxing Qian, Sheng Li, Xinpeng Zhang
School of Computer Science, Fudan University

###### Abstract

The advancement in text-to-image models has led to astonishing artistic
performances. However, several studios and websites illegally fine-tune these
models using artists’ artworks to mimic their styles for profit, which
violates the copyrights of artists and diminishes their motivation to produce
original works. Currently, there is a notable lack of research focusing on
this issue. In this paper, we propose a novel watermarking framework that
detects mimicry in text-to-image models through fine-tuning. This framework
embeds subtle watermarks into digital artworks to protect their copyrights
while still preserving the artist’s visual expression. If someone takes
watermarked artworks as training data to mimic an artist’s style, these
watermarks can serve as detectable indicators. By analyzing the distribution
of these watermarks in a series of generated images, acts of fine-tuning
mimicry using stolen victim data will be exposed. In various fine-tune
scenarios and against watermark attack methods, our research confirms that
analyzing the distribution of watermarks in artificially generated images
reliably detects unauthorized mimicry.

##  1 Introduction

Figure 1: Artists who upload their works online are vulnerable to stylistic
imitation. We are employing watermarking techniques to detect and expose
unauthorized fine-tuning in suspect models or services, thereby safeguarding
the originality of these artworks.

Generative imagery technology has revolutionized the field of art, making it
accessible and versatile. Applications such as Stable Diffusion[6],
Midjourney[4], and DALL-E-3[8] enable individuals with no artistic background
to create art pieces from simple descriptions. This democratization of art
creation extends to experienced artists as well, who can leverage these tools
to generate multiple pieces for inspiration or to revise their existing work.
The introduction of generative models has simplified the traditionally labor-
intensive process of artistic creation, allowing artists to focus on
innovation rather than execution. These models have found applications across
diverse fields, enriching video game design, book illustrations, magazine
covers, and presentations with AI-crafted art. Remarkably, these models can be
fine-tuned to emulate the unique styles of specific artists, producing new
works that maintain the original’s aesthetic qualities. Additionally, online
platforms now facilitate the sharing of these fine-tuned models, enabling the
replication of artist styles, celebrity likenesses, and even the creation of
NSFW content.

The prowess of artificial intelligence in creating compelling artworks is
evident, as demonstrated by several pieces that have secured awards in various
competitions. However, AI image generation models come with a set of
significant issues. These include copyright [9], ethics, and consent. A new
area of debate is the fine-tuning of text-to-image models to replicate the
unique styles of established artists, an action that has led to considerable
negative consequences for the individuals affected. In an interview addressing
an instance of this kind of imitation, the artist Hollie Mengert, who was
directly impacted, shared her insights [7]: ”As far as the characters, I
didn’t see myself in it. I didn’t personally see the AI making decisions that
I would make, so I did feel distance from the results. Some of that frustrated
me because it feels like it isn’t actually mimicking my style, and yet my name
is still part of the tool.”

The issues arising from the replication of painting styles are widespread,
affecting a multitude of artists, with Hollie Mengert being just one notable
example. Firstly, the proliferation of imitations hinders the efforts of
original artists to promote their work and cultivate their personal brands,
often leading to their obscurity due to the deluge of copied art, which can
significantly undermine their earnings and livelihood. Secondly, this
replication poses a discouraging challenge to aspiring artists and hobbyists,
whose passion and future dedication to crafting a distinct style may be easily
overshadowed by the capabilities of AI to replicate such styles. This could
lead to a discouragement that dissuades them from honing their craft. Thirdly,
the application of finely-tuned mimicry in contexts such as NSFW content, or
illustrations promoting racial discrimination, endangering nations, or
disrupting peace, could be deleterious to the original artists’ reputation.

In this paper, we focus on protecting artists’ rights and aiding regulatory
bodies in identifying unauthorized use of artwork in fine-tuning generative
models. We utilize non-intrusive watermarking technology that respects
original artistic expression and work promotion. Our framework is tailored to
address fine-tuning imitations in generative art, allowing artists to embed
unique watermarks into their artwork before online sharing. These
imperceptible watermarks preserve visual integrity and, when used in fine-
tuning, become part of the training process, revealing any unauthorized use in
generated artworks. We extensively test our framework in various fine-tuning
scenarios, including solo and collaborative imitations and mixing imitations
with clean images, demonstrating its robustness. Additionally, we evaluated
our watermarking framework’s resilience against traditional watermark attacks
and challenges arising from fine-tuning imitations.

Our summarized contributions are:

  * •

We propose a watermarking framework, empowering artists to embed unique
watermarks in their works for detecting plagiarism caused by mimic fine-tuned
models.

  * •

We design a verification mechanism to discern watermark distributions in
outputs from stolen and clean models, enhancing the identification of
plagiarism.

  * •

We verify the effectiveness of this framework in diverse mimicry scenarios,
including both single and multi-artist mimicry, ensuring reliable theft
detection.

  * •

We conduct comprehensive tests under both traditional and novel attack
scenarios, confirming the effectiveness and robustness of our framework.

##  2 Related Work

Text-to-Image Generation and Diffusion Models. The expressive power of visuals
often surpasses text, driving the pursuit of generating visual imagery from
text descriptions in academic and industrial realms. This field has seen
diverse research methods yielding impressive results [15, 19, 28, 26, 14, 18,
10, 25, 27, 16, 23, 21, 17].GAN-based methods[19, 28, 26, 14] showed promise
on smaller datasets. The introduction of autoregressive methods, particularly
with DALL-E [18] and Parti [27], transformed the field, especially in large-
scale data settings. Recently, a shift towards diffusion models [16, 23, 21,
17] has occurred, now the mainstream trend. These models are favored for
stable output, superior image quality, and adaptable control, becoming the
preferred choice for various applications. Tools like Sd-Webui [6] have
emerged, helping users share models and artistic works online [2]. While text-
to-image models have significantly advanced artistic creation, they also pose
challenges to artists and the art community.

Figure 2: The overview of our watermarking framework that facilitates the
embedding of visually non-intrusive watermarks within artworks, which remain
extractable following sophisticated imitative fine-tuning. The pattern of
watermark retrieval from these generated items enables the detection of
potential unauthorized use of artists’ works by imitation models. Upon
embedding a 32-bit watermark, our system can achieve a bit accuracy rate as
high as 93.75% (30/32).

Fine-tune for Art Style Mimicry. Style mimicry typically involves neural
networks producing artworks that mimic a particular artist’s style or a
distinct artistic genre. Training a model for specific style mimicry requires
a considerable collection of artworks in that style. However, with a scarcity
of training samples, mastering the nuances of the style and subtle artistic
habits becomes challenging. Fine-tuning, a mature technique in fields like
natural language processing and computer vision, is also beneficial for style
mimicry. Using tools such as Dreambooth [22] and Lora [13], users need only
about 20 examples of the target style, along with a pre-trained general model,
to effectively mimic the style. Models fine-tuned for style mimicry and their
artworks have gained popularity on various art-sharing websites [2, 4].
However, this mimicry often occurs without the original artists’ consent,
which may infringe upon their rights.

Protecting Against Mimicry Attacks. Artists often struggle to prevent
unauthorized online distribution of their works, which are frequently used for
training and mimicry attacks. Halting this sharing can significantly reduce
their promotional efforts. Recent research combats this by introducing
invisible perturbations into artworks using adversarial sample technology,
impeding fine-tuning training and model replication [24]. Fine-tuning for
style mimicry, crucial in advancing computer vision and image generation,
requires extensive data, highlighting concerns about artists’ rights. Unlike
these methods, our approach safeguards artists’ rights through watermarking
technology, confirming ownership of fine-tuning data in anonymous models and
managing rights authorization in signed models.

##  3 Treat Model

###  3.1 Artists

Artists Goal. Artists showcase artworks on online platforms for promotion,
commissions, and income. They seek to verify if unattributed services or
models use their data for imitation, or if signed models have authorization.

Artists Capabilities. Our assumption is that the artist possesses the
following capabilities:

  * •

Artists have the option to insert watermarks signifying personal data or non-
authorization into their works before online sharing, maintaining visual
integrity.

  * •

Artists can utilize adequate computational resources or enlist watermarking
agencies for watermark application and later validation.

  * •

Unaware of mimickers’ image caption tools, authors can try various content
descriptions as prompts.

###  3.2 Mimic Attackers

Attack Goal. The mimic attack fine-tunes a text-to-image model to replicate a
specific artist’s style, the victim. Once fine-tuned, it generates artworks
mimicking the victim’s style, creating new content of the target style.

Potential attackers. Potential initiators of imitation attacks could encompass
a range of entities with interests in specific artistic styles or the artists
themselves. These may include AI and internet companies, studios specializing
in AI, fine arts, and media, as well as web platforms focused on art and
painting. Alternatively, these initiators might be individuals particularly
interested in the style or the artists.

Attacker Capabilities. Our assumption is that the attacker possesses the
following capabilities:

  * •

Ability to access and download targeted artist’s artwork.

  * •

The capacity to obtain a high-quality pre-trained text-to-image model, such as
the SD2.1 checkpoint.

  * •

Sufficient computational resources and the requisite technical expertise to
conduct fine-tuning training.

##  4 Watermark Framework

Figure 3: Multi-Artist fine-tuning scenario. They may use the same
watermarking algorithm with different watermarks, or employ different
watermarking algorithms with varying watermarks (this figure illustrates
different method).

###  4.1 Overview

Figure 2 illustrates our proposed watermarking framework, which is segmented
into three phases: Artist Sharing, mimicry Fine-Tuning, and Verification
Extraction. In the Artist Sharing phase, artists integrate watermarks into
their works. Imitators then use these watermarked artworks for fine-tuning
training, aiming to replicate the artist’s style in a generative model. During
the Verification Extraction phase, if victims or authenticating bodies
identify suspect models, they can interrogate these models, either by
accessing the generation service or by downloading the model. This
interrogation involves extracting watermarks from various generated artworks,
a process that clearly identifies unauthorized fine-tuning theft.

###  4.2 General Methodology

Embedding. Watermark embedding is primarily the process of integrating
watermark information into images in a manner that is imperceptible to the
naked eye, thereby maintaining the image’s visual integrity. This approach is
equally effective for artworks, where the embedded watermarks, though minutely
concealed, do not alter the artwork’s artistic expression. The typical steps
involved in the watermark embedding process are as follows:

$\mathit{I_{w}}=\mathit{E(\mathit{I,bitstream})}$ (1)

where $E$ is the watermark embedding network. Watermark information is encoded
as a bitstream. $I$ denotes original images without watermarks, and $I_{w}$
refers to the watermarked image collection.

Mimic Fine-tune. Companies or individuals with an interest in an artist’s
style can utilize a select set of the artist’s works along with a pre-trained
generic text-to-image model, $G$, for fine-tuning. This fine-tuning process
results in a tuned model, either $G_{u}$ when fine-tuned with the watermarked
images collection $I_{w}$, or $G_{u}$ when using the original, unwatermarked
images collection $I$, specifically adapted to the target style:

$G_{u}=\mathcal{F}(G,I),$ (2) $G_{w}=\mathcal{F}(G,I_{w}).$ (3)

With $G_{u}$ and $G_{w}$, mimickers and users – either through accessing
services provided by mimickers or by downloading shared models – have the
ability to create artworks $I_{s}$ in the style of the impacted artist. This
creation is facilitated by using content prompts $P_{con}$ and special tags
$P_{sp}$, which are defined by imitators during the fine-tuning process. These
tags, typically words associated with the artist or their artistic style, are
usually released alongside the model:

$I_{s}=G_{w}(\mathit{P_{con}+P_{sp}}).$ (4)

Extraction. By using a watermark extractor $E_{x}$, which is specifically
aligned with the algorithm, we can successfully extract the watermark
$\bar{w}$ from a generated artifact $i_{s}$:

$\bar{w}=E_{x}(\mathit{i_{s}}).$ (5)

Verification. A common validation approach is to compare the watermark data w
derived from outputs produced by a potentially untrustworthy application
service against the author’s embedded watermark, distinguishing between
unauthorized watermark $w_{ua}$ and authorized watermark $w_{a}$. Bit accuracy
is then evaluated to ascertain if the mimicry involves fine-tuning of the
author’s work and the legitimacy of the used work:

$acc=\frac{\bar{n}}{n_{ua}},or,acc=\frac{\bar{n}}{n_{a}}$ (6)

where n is the count of correctly extracted watermark bits, with $n_{ua}$ and
$n_{a}$ representing the total bits in unauthorized and authorized watermarks,
usually 32. The watermark bit accuracy of a single image lacks persuasive
power in indicating whether the victim’s work was used for fine-tuning. We
identify potential data theft for mimicry fine-tuning by analyzing watermark
accuracy extremes, averages, and distributions across outputs from multiple
suspect models.

###  4.3 Fine-Tuning Scenarios

Single-Artist. The most common form of fine-tuning involves collecting works
from a single artist or a singular style for mimicry purposes. This method of
fine-tuning is cost-effective and achieves the desired mimicry effects
efficiently. This is introduced in the general method section of the previous
subsection.

Multi-Artist. In figure 3, Another scenario involves mimickers adopting a
hybrid fine-tuning approach with works from various authors, potentially for
stylistic resemblances. In our framework, artists embed unique watermarks in
their creations. During verification, content prompts from each artist’s work
extensively generate respective artworks. Watermarks are then extracted and
analyzed for distribution, determining if any artist has suffered from
unauthorized fine-tuning mimicry.

Mixed Clean Artworks. In a likely scenario, mimickers would gather artworks,
most of which are probably without watermarks, except for the ones belonging
to the artists being mimicked. This situation is comparable to mixing these
with unmarked images for fine-tuning on a moderate scale. To assess this, we
conducted tests using different ratios of images with watermarks in the fine-
tuning process. The results and detailed verification outcomes of these tests
are reported in the experimental section of this paper.

|  | Regarding the sample numbers within an accurate bit range |  |
---|---|---|---|---
32-bit | watermarked | 0-20% | 20-40% | 40-60% | 60-80% | 80-100% | avg(bits) | best(bits)
| $\times$ | 0 | 255 | 741 | 4 | 0 | 14.03 | 21
Artists’ images | $\checkmark$ | 0 | 0 | 109 | 867 | 24 | 19.54 | 29
| $\times$ | 0 | 163 | 826 | 11 | 0 | 14.66 | 20
AIs’ images | $\checkmark$ | 0 | 0 | 193 | 797 | 10 | 19.07 | 28
| $\times$ | 0 | 176 | 805 | 10 | 0 | 14.47 | 21
Natural images | $\checkmark$ | 0 | 0 | 520 | 468 | 12 | 19.31 | 28

Table 1:  Average results of watermark (32bit) extraction from 1000 images,
generated via single-artist fine-tuning, across 10 groups.

###  4.4 Watermark Attack

We have comprehensively considered several forms of watermark attacks that may
occur in the context of mimicry fine-tuning (assuming the mimicker is
knowledgeable about watermark attack techniques), and the validation results
are reported in the experimental section.

Traditional Watermark Attack. In traditional watermark attacks, mimickers
simply process the collected artworks and then use the processed images for
fine-tuning mimicry.

Secondary Fine-Tuning Attack. In mimicry scenarios, a mimic may choose two
fine-tuning iterations. The first round produces high-quality works, used in
subsequent iterative fine-tuning rounds. This continues until the desired
effect and consistent output quality are achieved.

Watermark Overlay Attack. Watermark overlay is a classic issue in watermark
attacks. We considered the scenario where a mimic might apply a new watermark
to acquired works before fine-tuning mimicry. Our aim is for the fine-tuned
model to still generate works with the original watermark, thereby protecting
the artist’s rights.

##  5 Experiments

###  5.1 Experiment Setup

Dataset. We primarily evaluate the performance of the proposed method across
three sets of image data:

  * •

Artworks from artists. The study mainly used works from contemporary and
historical artists (17 artists), each contributing about 50 paintings for key
experiments. The contemporary artists have authorized their works for
experiments and display.

  * •

Artworks from text-to-models. We also evaluated our method’s performance on
images generated by 10 different AI models from the online art sharing website
Civitai[2], with approximately 50 images for each model.

  * •

Natural images. We collected 10 sets of diverse natural images to assess our
watermarking method, anticipating mimickers’ use of real-world photographs for
imitation.

Model. The Stable Diffusion Model, used in our experiment for fine-tuning, is
a leading generative model popular on websites and online communities.

Watermark Methods. We tested the performance of different watermarks in both
general and multi-artist scenarios.

  * •

SSL [11]. SSL watermarking employs SSL-pretrained neural networks to embed
watermarks in latent spaces.

  * •

RivaGAN [29]. RivaGAN is a robust image watermarking method using GANs with
two adversarial networks for watermark assessment and removal.

  * •

DWT-DCT [5]. The method enhances digital image watermarking by combining DWT
with improved robustness and effectiveness.

  * •

DWT-DCT-SVD [12]. DWT-DCT-SVD method combines DWT, DCT, and SVD to watermark
color images, converting RGB to YUV and embedding watermarks into DCT-divided,
SVD-processed Y channel blocks.

###  5.2 Metrics

We assess the proposed method using bit accuracy rate (the percentage of
correctly identified bits in the watermark), and in the context of mimicry
fine-tuning, we also provide the best, average (avg), and overall accuracy
distribution of the generated samples.

###  5.3 Implement Details

We use Stable Diffusion v1.5[20] as the base model. During the training phase,
each image is standardized to a resolution of 512x512 pixels. The model
undergoes 500 training steps for each image. The class tokens are uniquely
combined with a rare identifier ‘zws’ and a class prompt ’art_paint’. In
addition, we set the batch size to 2, use Lion as the optimizer, and the
learning rate is 1e-07. In the generation phase, the model maintains the image
size at 512x512 pixels. It utilizes 32 steps of sampling with the DDPM
sampler, a scale setting of 7.5, and maintains the CLIP skip setting at 2. In
our research, more than 100 experimental groups were trained using 4 RTX 3090
GPUs, collectively amounting to over 600 hours of GPU computation time.
Experiments default to a 32-bit watermark length.

32-bit | The sample numbers within an accurate bit range | avg(bits) | best(bits)
---|---|---|---
0-20% | 20-40% | 40-60% | 60-80% | 80-100%
no watermark | 0 | 86 | 908 | 6 | 0 | 15.18 | 21
DWTDCT | 0 | 43 | 502 | 439 | 16 | 17.41 | 26
DWTDCTSVD | 0 | 37 | 482 | 454 | 27 | 18.57 | 30
SSL | 0 | 0 | 237 | 748 | 15 | 19.29 | 27
RivaGan | 0 | 0 | 111 | 840 | 49 | 19.72 | 29
Table 2: Comparison of watermark extraction distribution across different
watermarking methods in single-artist scenario (artist).

###  5.4 Main Result

Single-Artist Fine-Tuning. Table 1 presents the efficiency of extracting
32-bit watermarks across various image types. Images without watermarks
primarily exhibit extractions within the 40-60% range, averaging around 14
bits. Conversely, watermarked images shift this average toward the higher
60-80% range, with an approximate extraction of 19 bits. This trend persists
among artist-created, AI-generated, and reality photo categories, suggesting
that watermarking substantially increases the number of bits that can be
extracted. Optimal extraction results vary by image category, achieving
between 20 to 29 bits.

Figure 4: The watermark bit accuracy distribution of samples generated by the
fine-tuning model with watermark (red) and without watermark (blue), 450
samples randomly selected for each set.

Figure 4 shows that fine-tuning models with stolen watermarked images leave
watermark traces in the model, evidenced by a 60-80% accuracy rate (red) in
capturing these traces in generated outputs. This contrasts with the 40-60%
accuracy (blue) for models fine-tuned without watermarked images,
demonstrating the latent watermark’s detectability in generated styles. Figure
5 displays the original image from GhostMix [3], Van Gogh, and Catlora [1] for
fine-tuning, and the generated image with extractable watermark, maintaining
visual quality through integrated mimic fine-tuning in the generation model.

Figure 5: Original image (Left) and mimic generation from which watermark can
be extracted (right).

(a)

(b)

(c)

Figure 6: Fine-tuning of images with overlaid watermarks from three different
methods and the extraction results for each watermark.

Different Watermark Methods. Table 2 shows watermark extraction for different
methods, with a clear trend: techniques like DWTDCTSVD and RivaGan achieve
higher accuracy, mostly in the 60-80% range, with top extractions up to 30
bits. This contrasts with the non-watermarked baseline, which peaks at 21
bits. Advanced methods demonstrate improved robustness in watermark retrieval.

Multi-Artist Fine-Tuning. Figure 7 illustrates the watermark extraction
outcomes for artworks by three artists. They used a consistent watermarking
method (RivaGAN), three distinct watermarking methods for individual works,
and also a set without any watermarking. After mixed fine-tuning, these
artworks were used to generate new pieces based on descriptions from the three
artists. The results revealed that a single watermarking technique achieved a
maximum accuracy of 26/32, whereas employing various methods resulted in a
higher accuracy of 30/32. These results are significantly different from those
where no watermarks were embedded.

Mixed Clean Fine-Tuning. Figure 8(a) demonstrates watermark extraction
efficiency from a varied set of clean images, fine-tuned with increasing
proportions of watermarked images (10%, 40%, 80%, and 100% out of a total of
500). The figure highlights the variations in the number of samples with
watermark bit extractions surpassing specific accuracy levels. It is observed
that the extraction efficiency improves with a higher percentage of
watermarked images. Remarkably, even with just 10% of the images being
watermarked, watermark extraction can achieve up to 85% accuracy.

(a)

(b)

(c)

Figure 7: Multi-person (3 Artistss) mixed fine-tuning of watermark extraction
scenarios: the same watermarking method, different methods, no watermarking.

(a)

(b)

(c)

Figure 8: Assessment of watermark extraction under mixed clean Fine-tuning
conditions with varying proportions Watermarked images, evaluation of 16, 32,
64, 128-bit Watermark extraction, and Fine-tuning in differeant methods
(Dreambooth and Lora).

| The sample numbers within an accurate bit range |  |
---|---|---|---
32-bit | 0-10% | 10-20% | 20-30% | 30-40% | 40-50% | 50-60% | 60-70% | 70-80% | 80-90% |  90-100% | avg(bits) | best(bits)
Unwatermarked | 0 | 0 | 2 | 194 | 532 | 232 | 40 | 0 | 0 | 0 | 15.25 | 21
Normal Fine-tune | 0 | 0 | 0 | 0 | 0 | 261 | 640 | 81 | 13 | 5 | 19.96 | 29
Two-stage Fine-tune | 0 | 0 | 0 | 11 | 125 | 245 | 571 | 44 | 4 | 0 | 19.02 | 26
GaussianBlur | 2 | 0 | 2 | 30 | 119 | 211 | 558 | 67 | 11 | 0 | 19.25 | 28
Brightness | 0 | 0 | 2 | 33 | 189 | 294 | 450 | 30 | 2 | 0 | 18.24 | 26
CenterCrop | 0 | 0 | 1 | 49 | 246 | 279 | 411 | 14 | 0 | 0 | 17.76 | 24
Contrast | 0 | 0 | 2 | 55 | 229 | 266 | 401 | 44 | 3 | 0 | 18.06 | 26
Hue | 0 | 0 | 0 | 29 | 226 | 346 | 372 | 27 | 0 | 0 | 17.92 | 25
JPEG | 0 | 0 | 0 | 12 | 171 | 316 | 473 | 28 | 0 | 0 | 18.45 | 25
Meme | 1 | 0 | 12 | 115 | 315 | 263 | 275 | 18 | 1 | 0 | 16.86 | 26
Resize | 0 | 0 | 0 | 11 | 148 | 269 | 488 | 78 | 6 | 0 | 19.08 | 27
Rotation | 0 | 0 | 0 | 26 | 165 | 311 | 460 | 35 | 3 | 0 | 18.85 | 27

Table 3: Performance of watermark robustness against various attacks.

Attempt to Revise the Prompt | avg(bits) | best(bits)
---|---|---
Remove the word ’painting’ from the caption | 17.90 | 24
Enrich the content of the caption | 18.10 | 25
Revise the grammatical structure of the caption | 19.12 | 26
Replace with synonyms | 19.22 | 26
Make minor modifications to the caption | 19.32 | 26
Remove adjectives from the caption | 19.35 | 26
Scramble the order of the caption | 19.42 | 27
Retain a single subject object | 20.28 | 27
Simplify the caption | 20.29 | 27
Add strange characters to the caption | 21.06 | 27

Table 4: Watermark extraction performance during style generation using
different prompts.

###  5.5 Robustness Study

Under Traditional Attacks. Table 3 and 6 show that when images embedded with
watermarks for SD fine-tuning are exposed to traditional attacks, like JPEG
compression, the resultant generated images retain high bit accuracy. The
average correct bits surpass those of the unmodified model, with a peak
observed at 28 bits.

Secondary Fine-Tuning. Table 3 demonstrates that secondary SD fine-tuning on
generated images retains a high bit accuracy. The result shows that it
achieves a maximum accuracy of 26/32.

###  5.6 Bit Length Study

We assess the impact on watermark distribution in images from a stable
diffusion model fine-tuned with watermarked images of various capacities. We
use four watermark schemes: 16, 32, 64, and 128-bit. Figure 8(b) shows sample
variations in bit accuracies in 1000 images generated by the fine-tuned SD
model. Results indicate that samples with over 70% accuracy for 64-bit and
128-bit schemes are almost negligible. However, with 16-bit and 32-bit
schemes, over 200 samples achieved more than 70% watermark accuracy, some
reaching 85%.

###  5.7 Lora Study

We evaluate the impact of various fine-tuning approaches on the stable
diffusion model’s ability to generate watermarked images. Figure 8(c) shows
that the ‘No watermark(Dreambooth)’ distribution peaks at a lower bit accuracy
(12-16 bits). In contrast, the distributions for ’Watermark (Dreambooth)’ and
’Watermark (Lora)’ are strikingly similar and exhibit a higher peak in bit
accuracy, ranging from 19 to 24 bits.

###  5.8 Prompt Study

Given that the prompts used by the thieves to train with stolen artworks
remain unknown, we modified 10 prompts to prompt the suspect model into
generating images. As shown in Table 4, the average bit accuracy of 1000
images generated by the suspect model using each prompt type is significantly
high, predominantly exceeding 19 bits. Notably, the peak average correct bits
attained is 21.06 bits, with individual images reaching up to 27 bits.

##  6 Discussion and Conclusion

###  6.1 Discussion

Ethical Statement: This paper features paintings with the direct authorization
from the artists. Additionally, numerous contemporary artworks have been
downloaded from social platforms solely for the purpose of style analysis and
discussion, with no experimental usage prior to obtaining the artists’
permission. To augment our data for experimental research, we collected
historical artworks, AI style model-generated pieces, and various natural
photos. We hereby affirm that this collected data is specifically for the
experiments related to this paper’s proposed methods and is not for any other
use.

Limitations: The watermarking framework proposed in this study is conceptual
and not sufficiently robust for partial scenarios. It struggles to extract
watermarks precisely from a few samples for identity and authorization. When
mimickers conceal their services and do not disclose their mimic models,
sharing only a few style-generated images, the framework’s ability to identify
them is limited.

###  6.2 Conclusion

We introduce a watermarking scheme that integrates watermarks into artists’
works to protect intellectual property while maintaining their art’s aesthetic
integrity. This system allows artists to track unauthorized art mimicry in
fine-tuning applications. Our research covered various watermarking methods,
fine-tuning conditions, and strategies against watermark compromise. The
effectiveness of our approach is backed by substantial experimental evidence.
Future efforts will focus on developing more robust watermarking techniques
with higher extraction rates to counteract mimicry in fine-tuning.

## References

  * [1] Catlora.  https://civitai.com/models/99579?modelVersionId=106565.  Accessed: 2023-09-05.
  * [2] Civitai.  https://civitai.com/.  Accessed: 2023-09-02.
  * [3] Ghostmix.  https://civitai.com/models/36520/.  Accessed: 2023-09-05.
  * [4] Midjourney.  https://legacy.midjourney.com/showcase/recent/.  Accessed: 2023-09-05.
  * Al-Haj [2007] Ali Al-Haj.  Combined dwt-dct digital image watermarking.  _Journal of computer science_ , 3(9):740–746, 2007.
  * AUTOMATIC [1111] AUTOMATIC1111.  Sd-webui.  https://github.com/AUTOMATIC1111/stable-diffusion-webui.  Accessed: 2023-09-03.
  * [7] Andy Baio.  Invasive diffusion: how one unwilling illustrator found herself turned into an ai model.  https://waxy.org/2022/11/invasive-diffusion-how-one-unwilling-illustrator-found-herself-turned-into-an-ai-model/.  Accessed: 2023-09-02.
  * [8] James Betker, Gabriel Goh, Li Jing, † TimBrooks, Jianfeng Wang, Linjie Li, † LongOuyang, † JuntangZhuang, † JoyceLee, † YufeiGuo, † WesamManassra, † PrafullaDhariwal, † CaseyChu, † YunxinJiao, and Aditya Ramesh.  Improving image generation with better captions.
  * Brittain [2023] Blake Brittain.  Ai-created images lose us copyrights in test for new technology.  _Reuters, February_ , 2023.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  _Advances in Neural Information Processing Systems_ , 34:19822–19835, 2021.
  * Fernandez et al. [2022] Pierre Fernandez, Alexandre Sablayrolles, Teddy Furon, Hervé Jégou, and Matthijs Douze.  Watermarking images in self-supervised latent spaces.  In _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022.
  * He and Hu [2018] Yuqi He and Yan Hu.  A proposed digital image watermarking based on dwt-dct-svd.  In _2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC)_ , pages 1214–1218. IEEE, 2018.
  * Hu et al. [2021] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Li et al. [2019] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr.  Controllable text-to-image generation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Mansimov et al. [2016] Elman Mansimov, Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov.  Generating images from captions with attention.  In _ICLR_ , 2016.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning_ , pages 16784–16804. PMLR, 2022.
  * [17] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Reed et al. [2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In _International conference on machine learning_ , pages 1060–1069. PMLR, 2016.
  * Rombach et al. [2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Shan et al. [2023] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y. Zhao.  Glaze: Protecting artists from style mimicry by Text-to-Image models.  In _32nd USENIX Security Symposium (USENIX Security 23)_ , pages 2187–2204, Anaheim, CA, 2023. USENIX Association.
  * Wu et al. [2022] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan.  Nüwa: Visual synthesis pre-training for neural visual world creation.  In _European conference on computer vision_ , pages 720–736. Springer, 2022.
  * Xu et al. [2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1316–1324, 2018.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _Transactions on Machine Learning Research_ , 2022.
  * Zhang et al. [2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5907–5915, 2017.
  * Zhang et al. [2019] Kevin Alex Zhang, Lei Xu, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Robust invisible video watermarking with attention.  2019\.