# ProMark: Proactive Diffusion Watermarking for Causal Attribution

Vishal Asnani1,2
John Collomosse1,3
Tu Bui3
Xiaoming Liu2
Shruti Agarwal1

1Adobe Research, 2Michigan State University, 3University of Surrey
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Generative AI (GenAI) is transforming creative workflows through the
capability to synthesize and manipulate images via high-level prompts. Yet
creatives are not well supported to receive recognition or reward for the use
of their content in GenAI training. To this end, we propose ProMark, a causal
attribution technique to attribute a synthetically generated image to its
training data concepts like objects, motifs, templates, artists, or styles.
The concept information is proactively embedded into the input training images
using imperceptible watermarks, and the diffusion models (unconditional or
conditional) are trained to retain the corresponding watermarks in generated
images. We show that we can embed as many as $2^{16}$ unique watermarks into
the training data, and each training image can contain more than one
watermark. ProMark can maintain image quality whilst outperforming
correlation-based attribution. Finally, several qualitative examples are
presented, providing the confidence that the presence of the watermark conveys
a causative relationship between training data and synthetic images.

##  1 Introduction

GenAI is able to create high-fidelity synthetic images spanning diverse
concepts, largely due to advances in diffusion models, _e.g_. DDPM [17], DDIM
[21], LDM [26]. GenAI models, particularly diffusion models, have been shown
to closely adopt and sometimes directly memorize the style and the content of
different training images – defined as “concepts” in the training data [11,
19]. This leads to concerns from creatives whose work has been used to train
GenAI. Concerns focus upon the lack of a means for attribution, _e.g_.
recognition or citation, of synthetic images to the training data used to
create them and extend even to calls for a compensation mechanism (financial,
reputational, or otherwise) for GenAI’s derivative use of concepts in training
images contributed by creatives.

Figure 1:  Causative _vs_. correlation-based matching for concept attribution.
ProMark identifies the training data most responsible for a synthetic image
(‘attribution’). Correlation-based matching doesn’t always perform the data
attribution properly. We propose ProMark, which is a proactive approach
involving adding watermarks to training data and recovering them from the
synthetic image to perform attribution in a causative way.

We refer to this problem as concept attribution – the ability to attribute
generated images to the training concept/s which have most directly influenced
their creation. Several passive techniques have recently been proposed to
solve the attribution problem [5, 28, 32]. These approaches use visual
correlation between the generated image and the training images for
attribution. Whilst they vary in their method and rationale for learning the
similarity embedding – all use some forms of contrastive training to learn a
metric space for visual correlation.

We argue that although correlation can provide visually intuitive results, a
measure of similarity is not a causative answer to whether certain training
data is responsible for the generation of an image or not. Further,
correlation-based techniques can identify close matches with images that were
not even present in the training data.

Keeping this in mind, we explore an intriguing field of research which is
developing around proactive watermarking methodologies [35, 27, 31, 3], that
employ signals, termed templates to encrypt input images before feeding them
into the network. These works have integrated and subsequently retrieved
templates to bolster the performance of the problem at hand. Inspired by these
works, we introduce ProMark, a proactive watermarking-based approach for GenAI
models to perform concept attribution in a causative way. The technical
contributions of ProMark are three-fold:

1\. Causal vs. Correlation-based Attribution. ProMark performs causal
attribution of synthetic images to the predefined concepts in the training
images that influenced the generation. Unlike prior works that visually
correlate synthetic images with training data, we make no assumption that
visual similarity approximates causation. ProMark ties watermarks to training
images and scans for the watermarks in the generated images, enabling us to
demonstrate rather than approximate/imply causation. This provides confidence
in grounding downstream decisions such as legal attribution or payments to
creators.

2\. Multiple Orthogonal Attributions. We propose to use orthogonal invisible
watermarks to proactively embed attribution information into the input
training data and add a BCE loss during the training of diffusion models to
retain the corresponding watermarks in the generated images. We show that
ProMark causatively attributes as many as $2^{16}$ unique training-data
concepts like objects, scenes, templates, motifs, and style, where the
generated images can simultaneously express one or two orthogonal concepts.

3\. Flexible Attributions. ProMark can be used for training conditional or
unconditional diffusion models and even finetuning a pre-trained model for
only a few iterations. We show that ProMark’s causative approach achieves
higher accuracy than correlation-based attribution over five diverse datasets
(Sec. 4.1): Adobe Stock, ImageNet, LSUN, Wikiart, and BAM while preserving
synthetic image quality due to the imperceptibility of the watermarks.

Fig. 1 presents our scenario, where synthetic image(s) are attributed back to
the most influential GenAI training images. Correlation-based techniques [5,
32] try to match the high-level image structure or style. Here, the green-
lizard synthetic image is matched to a generic green image without a lizard
[5]. With ProMark’s causative approach, the presence of the green-lizard
watermark in the synthetic image will correctly indicate the influence of the
similarly watermarked concept group of lizard training images.

##  2 Related Works

Table 1: Comparison of ProMark with prior works. Uniquely, we perform
causative attribution using proactive watermarking to attribute multiple
concepts. [Keys: emb.: embedding, obj.: object, own.: ownership, sem.:
semantic, sty.: style, wat.: watermark]

Method Scheme Task Match # Class Multiple Attribution type type attribution
type [28] passive attribution emb. - ✕ sty. [5] passive attribution emb. - ✕
obj. [32] passive attribution emb. $693$ ✕ sty., obj. [16] passive detect wat.
$2$ ✕ - [20] passive detect wat. $2$ ✕ - [14] passive detect wat. $2$ ✕ - [31]
proactive detect wat. $2$ - - [1] proactive detect wat. $2$ - - [3] proactive
localization wat. $2$ - - [2] proactive obj. detect - $90$ - - sty., obj.
ProMark proactive attribution wat. $2^{16}$ ✓ own., sem.

Passive Concept Attribution. Concept attribution differs from model [8] or
camera [12] attribution in that the task is to determine the responsible
training data for a given generation. Existing concept attribution techniques
are passive – they do not actively modify the GenAI model or training data but
instead, measure the visual similarity (under some definition) of synthetic
images and training data to quantify attribution for each training image.
EKILA [5] proposes patch-based perceptual hashing (visual fingerprinting [22,
6]) to match the style of the query patches to the training data for
attribution. Wang  _et al_. [32] finetune semantic embeddings like CLIP, DINO,
_etc_. for the attribution task. Both [5] and [32] explore ALADIN [28] for
style attribution. ALADIN is a feature representation for fine-grained style
similarity learned using a weakly supervised approach.

All these works are regarded as passive approaches as they take the image as
an attribute by correlating between generated and training image styles.
Instead, our approach is a proactive scheme that adds a watermark to training
images and performs attribution in a causal manner (Tab. 1).

Proactive Schemes. Proactive schemes involve adding a signal/perturbation onto
the input images to benefit different tasks like deepfake tagging [31],
deepfake detection [1], manipulation localization [3], object detection [2],
_etc_. Some works [35, 27] disrupt the output of the generative models by
adding perturbations to the training data. Alexandre  _et al_. [29] tackles
the problem of training dataset attribution by using fixed signals for every
data type. These prior works successfully demonstrate the use of watermarks to
classify the content of the AI-generated images proactively. We extend the
idea of proactive watermarking to perform the task of causal attribution of
AI-generated images to influential training data concepts. Watermarking has
not been used to trace attribution in GenAI before.

Figure 2:  Overview of ProMark. We show the training and inference procedure
for our proposed method. Our training pipeline involves two stages, image
encryption and generative model training. We convert the bit-sequences to
spatial watermarks ($\boldsymbol{W}$), which are then added to the
corresponding concept images ($\boldsymbol{X}$) to make them encrypted
($\boldsymbol{X}_{W}$). The generative model is then trained with the
encrypted images using the LDM supervision. During training, we recover the
added watermark using the secret decoder ($\mathcal{D}_{S}$) and apply the BCE
supervision to perform attribution. To sample newly generated images, we use a
Gaussian noise and recover the bit-sequences using the secret decoder to
attribute them to different concepts. Best viewed in color.

Watermarking of GenAI Models. It is an active research to watermark AI-
generated images for the purpose of privacy protection. Fernandez1  _et al_.
[16] fine-tune the LDM’s decoder to condition on a bit sequence, embedding it
in images for AI-generated image detection. Kirchenbauer  _et al_. [18]
propose a watermarking method for language models by pre-selecting random
tokens and subtly influencing their use during word generation. Zhao  _et al_.
[37] use a watermarking scheme for text-to-image diffusion models, while Liu
_et al_. [20] verify watermarks by pre-defined prompts. [14, 23] add a
watermark for detecting copyright infringement. Asnani  _et al_. [4] reverse
engineer a fingerprint left by various GenAI models to further use it for
recovering the network and training parameters of these models [34, 4].
Finally, Cao  _et al_. [10] adds an invisible watermark for protecting
diffusion models which are used to generate audio modality. Most of these
works have used watermarking for protecting diffusion models, which enables
them to add just one watermark onto the data. In contrast, we propose to add
multiple watermarks to the training data and to a single image, which is a
more challenging task than embedding a universal watermark.

##  3 Method

###  3.1 Background

Diffusion Models. Diffusion models learn a data distribution
$p(\boldsymbol{X}),\text{ where }\boldsymbol{X}\in\mathbb{R}^{h\times w\times
3}$ is the input image. They do this by iteratively reducing the noise in a
variable that initially follows a normal distribution. This can be viewed as
learning the reverse steps of a fixed Markov Chain with a length of $T$.
Recently, LDM [26] is proposed to convert images to their latent
representation for faster training in a lower dimensional space than the pixel
space. The image is converted to and from the latent space by a pretrained
autoencoder consisting of an encoder
$\boldsymbol{z}=\mathcal{E}_{L}(\boldsymbol{X})$ and a decoder
$\boldsymbol{X}_{R}=\mathcal{D}_{L}(\boldsymbol{z})$, where $\boldsymbol{z}$
is the latent code and $\boldsymbol{X}_{R}$ is the reconstructed image. The
trainable denoising module of the LDM is
$\epsilon_{\theta}(\boldsymbol{z}_{t},t);t=1...T$, where $\epsilon_{\theta}$
is trained to predict the denoised latent code $\hat{\boldsymbol{z}}$ from its
noised version $\boldsymbol{z}_{t}$. This objective function can be defined as
follows:

$L_{LDM}=\mathbb{E}_{\mathcal{E}_{L}(\boldsymbol{X}),\epsilon\sim\mathcal{N}(0,1),t}\bigl{[}||\epsilon-\epsilon_{\theta}(\boldsymbol{z}_{t},t)||^{2}_{2}\bigr{]},$
(1)

where $\epsilon$ is the noise added at step $t$.

Image Encryption. Proactive works [1, 3, 2] have shown performance gain on
various tasks by proactively transforming the input training images
$\boldsymbol{X}$ with a watermark, resulting in an encrypted image. This
watermark is either fixed or learned, depending on the task at hand. Similar
to prior proactive works, our image encryption is of the form:

$\boldsymbol{X}_{W}=\mathcal{T}(\boldsymbol{X};\boldsymbol{W})={\boldsymbol{X}}+m\times
R(\boldsymbol{W},h,w),$ (2)

where $\mathcal{T}$ is the transformation, $\boldsymbol{W}$ is the spatial
watermark, $\boldsymbol{X}_{W}$ is the encrypted image, $m$ is the watermark
strength, and $R(.)$ resizes $\boldsymbol{W}$ to the input resolution $(h,w)$.

We use the state-of-the-art watermarking technique RoSteALS [9] to compute the
spatial watermarks for encryption due to its robustness to image
transformation and generalization (the watermark is independent of content of
the input image). RoSteALS is designed to embed a secret of length $b$-bits
into an image using robust and imperceptible watermarking. It comprises of a
secret encoder $\mathcal{E}_{S}(\boldsymbol{s})$, which converts the bit-
secret $\boldsymbol{s}\in\\{0,1\\}^{b}$ into a latent code offset
$\boldsymbol{z}_{o}$. It is then added to the latent code of an autoencoder
$\boldsymbol{z}_{w}=\boldsymbol{z}+\boldsymbol{z}_{o}$. This modified latent
code $\boldsymbol{z}_{w}$ is then used to reconstruct a watermarked image via
autoencoder decoder. Finally, a secret decoder, denoted by
$\mathcal{D}_{S}(X_{W})$, takes the watermarked images as input and predicts
the bit-sequence $\hat{\boldsymbol{s}}$.

###  3.2 Problem Definition

Let $\mathcal{C}=\\{c_{1},c_{2},\ldots,c_{N}\\}$ be a set of $N$ distinct
concepts within a dataset that is used for training a GenAI model for image
synthesis. The problem of concept attribution can be formulated as follows:

Given a synthetic image $\boldsymbol{X}_{S}$ generated by a GenAI model, the
objective of concept attribution is to accurately associate
$\boldsymbol{X}_{S}$ to a concept $c_{i}\in\mathcal{C}$ that significantly
influenced the generation of $\boldsymbol{X}_{S}$.

We aim to find a mapping $f:\boldsymbol{X}_{S}\rightarrow c_{i}$ such that

$c_{i}^{*}=\arg\max_{c_{i}\in\mathcal{C}}f(\boldsymbol{X}_{S},c_{i}),$ (3)

where $c_{i}^{*}$ represents the concept most strongly attributed to image
$\boldsymbol{X}_{S}$.

###  3.3 Overview

The pipeline of ProMark is shown in Fig. 2. The principle is simple: if a
specific watermark unique to a training concept can be detected from a
generated image, it indicates that the generative model relies on that concept
in the generation process. Thus, ProMark involves two steps: training data
encryption via watermarks and generative model training with watermarked
images.

To watermark the training data, the dataset is first divided into $N$ groups,
where each group corresponds to a unique concept that needs attribution. These
concepts can be semantic (_e.g_. objects, scenes, motifs or stock image
templates) or abstract (_e.g_. stylistic, ownership info). Each training image
in a group is encoded with a unique watermark without significantly altering
the image’s perceptibility. Once the training images are watermarked, they are
used to train the generative model. As the model trains, it learns to generate
images based on the encrypted training images. Ideally, the generated images
would have traces of watermarks corresponding to concepts they’re derived
from.

During inference, ProMark conforms to whether a generated image is derived
from a particular training concept by identifying the unique watermark of that
concept within the image. Through the careful use of unique watermarks, we can
trace back and causally attribute generated images to their origin in the
training dataset.

###  3.4 Training

During training, our algorithm is composed of two stages: image encryption and
generative model training. We now describe each of these stages in detail.

Image Encryption. The training data is first divided into $N$ concepts, and
images in each partition are encrypted using a fixed spatial watermark
$\boldsymbol{W}_{j}\in\mathbb{R}^{h\times w}$ ($j\in{0,1,2,...,N}$). Each
noise $\boldsymbol{W}_{j}$ is a $b$-dim bit-sequence (secret)
$\boldsymbol{s}_{j}=\\{p_{j1},p_{j2},...,p_{jb}\\}$ where
$p_{ji}\in\\{0,1\\}$.

In order to compute the watermark $\boldsymbol{W}_{j}$ from the bit-sequence
$\boldsymbol{s}_{j}$, we encrypt $100$ random images with $\boldsymbol{s}_{j}$
using pretrained RoSteALS secret encoder $\mathcal{E}_{S}(.)$ which takes
$b=160$ length secret as input. From these encrypted images, we obtain $100$
noise residuals by subtracting the encrypted images from the originals, which
are averaged to compute the watermark $\boldsymbol{W}_{j}$ as:

$\boldsymbol{W}_{j}=\frac{1}{100}\sum_{i=1}^{100}(\boldsymbol{X}_{i}-\mathcal{E}_{S}(\boldsymbol{X}_{i},\boldsymbol{s}_{j})).$
(4)

The above process is defined as spatial noise conversion in Fig. 2. The
averaging of noise residuals across different images reduces the image content
in the watermark and makes the watermark independent of any specific image.
Additionally, the generated watermarks are orthogonal due to different bits
for all $\boldsymbol{s}_{j}$, ensuring distinguishability from each other.
With the generated watermarks, each training image is encrypted using Eq. 2
with one of the $N$ watermarks that correspond to the concept the image
belongs to.

Generative Model Training. Using the encrypted data, we train the LDM’s
denoising module $\epsilon_{\theta}(.)$ using the objective function (Eq. 1),
where $\boldsymbol{z}_{t}$ is the noised version of:

$\boldsymbol{z}=\mathcal{E}_{L}(\boldsymbol{X}_{W_{j}})=\mathcal{E}_{L}(\mathcal{T}(\boldsymbol{X};\boldsymbol{W}_{j})),$
(5)

i.e., the input latent codes $\boldsymbol{z}$ are generated using the
encrypted images $\boldsymbol{X}_{W_{j}}$ for $j\in\\{0,1,2....,N\\}$.

However, we found that only using LDM loss is insufficient to successfully
learn the connection between the conceptual content and its associated
watermark. This gap in learning presents a significant hurdle, as the primary
aim is to trace back generated images to their respective training concepts
via the watermark. To tackle this, an auxiliary supervision is introduced to
LDM’s training,

$L_{BCE}(\boldsymbol{s}_{j},\hat{\boldsymbol{s}})=-\frac{1}{b}\sum_{i=1}^{b}[p_{ji}\log(\hat{p}_{i})+(1-p_{ji})\log(1-\hat{p}_{i})],$
(6)

where $L_{BCE}(.)$ is the binary cross-entropy (BCE) between the actual bit-
sequence $\boldsymbol{s}_{j}$ associated with watermark $\boldsymbol{W}_{j}$
and the predicted bit-sequence $\hat{\boldsymbol{s}}$. The denoised latent
code $\hat{\boldsymbol{z}}$ is then decoded using the autoencoder
$\mathcal{D}_{L}(.)$, and the embedded secret $\hat{\boldsymbol{s}}$ is
predicted by the secret decoder $\mathcal{D}_{S}(.)$ as:

$\hat{\boldsymbol{s}}=\mathcal{D}_{S}(\mathcal{D}_{L}(\hat{\boldsymbol{z}})).$
(7)

By employing BCE, the model is guided to minimize the difference between the
predicted watermark and the embedded watermark, hence improving the model’s
ability to recognize and associate watermarks with their respective concepts.
Finally, our objective is to minimize the loss function
$L_{attr}=L_{LDM}+\alpha L_{BCE}$ during training, where $\alpha$ is set to
$2$ for our experiments.

###  3.5 Inference

After the LDM learns to associate the watermarks with concepts, we use random
Gaussian noise to sample the newly generated images from the model. While the
diffusion model creates these new images, it also embeds a watermark within
them. Each watermark maps to a distinctive orthogonal bit-sequence associated
with a specific training concept, serving as a covert signature for
attribution.

To attribute the generated images and ascertain which training concept
influenced them, we predict the secret embedded by the LDM in the generated
images (see Eq. 7). Given a predicted binary bit-sequence,
$\hat{\boldsymbol{s}}=\\{\hat{p_{1}},\hat{p_{2}},...,\hat{p_{b}}\\}$ and all
the input bit-sequences $\boldsymbol{s}_{j}$ for $j\in{0,1,2...,N}$, we define
the attribution function, $f$, in Eq. 3 as:

$f(\hat{\boldsymbol{s}},\boldsymbol{s}_{j})=\sum_{k=1}^{b}[\hat{p}_{k}=p_{jk}],$
(8)

where $[\hat{p}_{k}=p_{ik}]$ acts as an indicator function, returning $1$ if
the condition is true, i.e., the bits are identical, and $0$ otherwise.
Consequently, we assign the predicted bit sequence to the concept whose bit
sequence it most closely mirrors — that is, the concept $j^{*}$ for which
$f(\hat{\boldsymbol{s}},\boldsymbol{s}_{j^{*}})$ is maximized:

$j^{*}=\arg\max_{j\in\\{1,2,...,N\\}}f(\hat{\boldsymbol{s}},\boldsymbol{s}_{j}).$
(9)

In other words, the concept whose watermark is most closely aligned with the
generated image’s watermark is deemed to be the influencing source behind the
generated image.

###  3.6 Multiple Watermarks

In prior image attribution works, an image is usually attributed to a single
concept (_e.g_. image content or image style). However, in real-world
scenarios, an image may encapsulate multiple concepts. This observation brings
forth a pertinent question: “Is it possible to use multiple watermarks for
multi-concept attribution within a single image?”

In this paper, we propose a novel approach to perform multi-concept
attribution by embedding multiple watermarks into a single image. In our
preliminary experiments, we restrict our focus to the addition of two
watermarks. To achieve this, we divide the image into two halves and resize
each watermark to fit the respective halves. This ensures that each half of
the image carries distinct watermark information pertaining to a specific
concept.

For the input RGB image $\boldsymbol{X}$,
{$\boldsymbol{W}_{i},\boldsymbol{W}_{j}$} are the watermarks for two secrets
{$\boldsymbol{s}_{i},\boldsymbol{s}_{j}$}, we formulate the new transformation
$\mathcal{T}$ as:

$\displaystyle\mathcal{T}(\boldsymbol{X};\boldsymbol{W}_{i},\boldsymbol{W}_{j})=$
$\displaystyle\Big{\\{}\boldsymbol{X}_{left},\boldsymbol{X}_{right}\Big{\\}}$
$\displaystyle=$
$\displaystyle\Big{\\{}(\boldsymbol{X}(:,0:\frac{w}{2},:)+R(\boldsymbol{W}_{i},h,\frac{w}{2})),$
$\displaystyle(\boldsymbol{X}(:,\frac{w}{2}:w,:)+R(\boldsymbol{W}_{j},h,\frac{w}{2})\Big{\\}},$

where $\\{.\\}$ is the horizontal concatenation. The loss function uses the
two predicted secrets ($\hat{\boldsymbol{s}}_{1}$ and
$\hat{\boldsymbol{s}}_{2}$) from the two halves of the generated image,
defined as:

$\displaystyle
L_{attr}=L_{LDM}+\alpha(L_{BCE}(\boldsymbol{s}_{i},\hat{\boldsymbol{s}}_{1})+L_{BCE}(\boldsymbol{s}_{j},\hat{\boldsymbol{s}}_{2})).$

##  4 Experiments

###  4.1 Unconditional Diffusion Model

In this section, we train multiple versions of unconditional diffusion models
[26] to demonstrate that ProMark can be used to attribute a variety of
concepts in the training data. In each case, the model is trained starting
from random initialization of LDM weights. Described next are details of the
datasets and evaluation protocols.

Datasets We use $5$ datasets spanning attribution categories like image
templates, scenes, objects, styles, and artists. For each dataset, we consider
the dataset classes as our attribution categories. For each class in a
dataset, we use $90\%$ images for training, and $10\%$ for evaluation, unless
specified otherwise.

  1. 1.

Stock: We collect images from Adobe Stock, comprising of near-duplicate image
clusters like templates, symbols, icons,  _etc_. An example image from some
clusters is shown in the supplementary. We use $100$ such clusters, each with
$2K$ images.

  2. 2.

LSUN: The LSUN dataset [36] comprises $10$ scene categories, such as bedrooms
and kitchens. It’s commonly used for scene classification, training generative
models like GANs, and anomaly detection. Same as the Stock dataset, we use
$2K$ images per class.

  3. 3.

Wiki-S: The WikiArt dataset [30] is a collection of fine art images spanning
various styles and artists. We use the $28$ style classes with $580$ average
images per class.

  4. 4.

Wiki-A: From the WikiArt dataset [30] we also use the $23$ artist classes with
$2,112$ average images per class.

  5. 5.

ImageNet: We use the ImageNet dataset [15] which comprises of $1$ million
images across $1K$ classes. For this dataset, we use the standard validation
set with $50K$ for evaluation and the remaining images for training.

Table 2: Comparison with prior works for unconditional diffusion model on
various datasets. [Keys: str.: strength]

c—c—c—c—c—c—c  Str. Attribution Accuracy (%) $\bf\uparrow$
Method (%) Stock LSUN Wiki-A Wiki-S ImageNet
ALADIN [28] \- $99.86$ $46.27$ $48.95$ $33.25$ $9.25$
CLIP [25] \- $75.67$ $87.13$ $77.58$ $60.84$ $60.12$
F-CLIP [32] \- $78.49$ $87.39$ $77.23$ $60.43$ $62.83$
SSCD [24] \- $99.63$ $73.26$ $69.51$ $50.37$ $37.32$
EKILA [5] \- $99.37$ $70.60$ $51.23$ $37.06$ $38.00$
$30$ $100$ $95.12$ $97.45$ $98.12$ $83.06$
ProMark $100$ $\bf 100$ $\bf 100$ $\bf 100$ $\bf 100$ $\bf 91.07$

Figure 3:  Example training and newly sampled images of different datasets for
the corresponding classes. We observe a similar content in the inference image
compared with the training image of the predicted class.

Evaluation Protocol For all datasets, the concept attribution performance is
tested on the held-out data as follows. For a held-out image, we first encrypt
it with the concept’s watermark. Then using the latent code of the encrypted
image, we noise it till a randomly assigned timestamp and apply our trained
diffusion model to reverse back to the initial timestamp with the estimated
noise. The denoised latent code is then decoded using the autoencoder
$\mathcal{D}_{L}(.)$, and the embedded secret is predicted using the secret
decoder $\mathcal{D}_{S}(.)$. Using Eq. 9, we compute the predicted concept
and calculate the accuracy using the ground-truth concept.

Results Shown in Sec. 4.1 is the attribution accuracy of ProMark at two
watermark strengths _i.e_. $100\%$ and $30\%$ which is set by variable $m$ in
Eq. 2. ProMark outperforms prior works, achieving near-perfect accuracy on all
the datasets when the watermark strength is $100\%$. However, the watermark
introduces visual artifacts [9] if the watermark strength is full. Therefore,
we decrease the watermark strength to $30\%$ before adding it to the training
data (see Sec. 4.5 for ablation on watermark strength). Even though our
performance drops at a lower watermark strength, we still outperform the prior
works. This shows that our causal approach can be used to attribute a variety
of concepts in the training data with an accuracy higher than the prior
passive approaches.

Fig. 3 (rows 1-5) shows the qualitative examples of the newly sampled images
from each of the trained models. For each model, we sample the images using
random Gaussian noise until we have images for every concept. The concept for
each image is predicted using the secret embedded in the generated images.
Shown in each row of Fig. 3 are three training images (columns 1-3) and three
sampled images from the corresponding concepts (columns 4-6). This shows that
ProMark makes the diffusion model embed the corresponding watermark for the
class of the generated image, thereby demonstrating the usefulness of our
approach.

Figure 4:  Visual results of prior embedding-based works. We show the image of
the closest matched embedding for each method on ImageNet. We highlight images
green for correct attribution, otherwise red. Embedding-based works do not
always attribute to the correct concept.

Shown in Fig. 4 are the nearest images retrieved using the embedding-based
methods (row ($2$)-($6$)) for the query images from the ImageNet (row ($1$)).
For each image retrieval, we highlight the correct/incorrect attribution using
a green/red box. As we can see, the correlation-based prior techniques rely on
visual similarity between the query and the retrieved images, ignoring the
concept. However, for each query image, ProMark predicts the correct concept
corresponding to the query image (Fig. 3).

###  4.2 Multiple Watermarks

We evaluate the effectiveness of ProMark for multi-concept attribution. As
before, an unconditional diffusion model is trained starting from random
initialization, and each image in the training data is encrypted with two
watermarks as outlined in Sec. 3.6.

Dataset For this experiment, we use the BAM dataset [33], comprising
contemporary artwork sourced from Behance, a platform hosting millions of
portfolios by professionals and artists. This dataset uniquely categorizes
each image into two label types: media and content. It encompasses $7$
distinct labels for media and $9$ for content, culminating in a diverse set of
$63$ label pairs, with $4,593$ average images in these label pairs. For each
class pair, we use $90\%$ data for training and $10\%$ for held-out
evaluation.

Table 3: Multi-concept attribution comparison with baselines.

c—c—c—c—c  Strength Attribution Accuracy (%) $\bf\uparrow$
Method (%) Media Content Combined
ALADIN [28] \- $42.16$ $41.25$ $34.97$
CLIP [25] \- $46.71$ $45.12$ $42.36$
F-CLIP [32] \- $52.12$ $51.56$ $46.23$
SSCD [24] \- $47.06$ $46.09$ $40.61$
EKILA [5] \- $43.72$ $43.58$ $37.09$
ProMark (single) 30 - - $\bf 97.73$
$30$ $91.33$ $89.21$ $84.66$
ProMark (multi) $100$ $\bf 95.61$ $\bf 93.31$ $90.12$

Results The same evaluation is performed as described in Sec. 4.1, except the
accuracy is now computed for two concepts instead of one. Shown in Sec. 4.2 is
the attribution accuracy for the two concepts individually and simultaneously.
To benchmark the effectiveness of ProMark, we also compare against baselines,
where ProMark outperforms baselines, achieving a combined attribution accuracy
of $90.12\%$ as compared to $46.61\%$ for F-CLIP [32]. We believe our findings
substantiate that ProMark can be extended to a scenario where the generated
images are composed of several unique concepts from the training images. For
ablation, we train ProMark with $7\times 8$ classes, with each pair of media
and content as an individual concept. ProMark is able to achieve $97.73\%$
attribution accuracy for single-concept, higher than the performance achieved
for multi-concept case _i.e_. $90.12\%$. However, single concept approach is
not scalable when the number of concepts in an image increases, as the number
of watermarks would grow exponentially ($7\times 8$ _vs_. $7+8$). Therefore,
transitioning to a multi-concept scenario is more appropriate for real-world
scenarios, where scalability and practicality are crucial.

In the final row of Fig. 3, we present qualitative examples of newly sampled
images from the model trained on the BAM dataset. Observations indicate that
these sampled images successfully adopt both media and content corresponding
to training images of the same concept. This provides empirical evidence of
ProMark’s effectiveness in facilitating multi-concept attribution.

###  4.3 Number of Concepts

AI models leverage large-scale image datasets [26, 7, 17, 21], encompassing a
broad spectrum of concepts. This diversity necessitates concept attribution
methods that can maintain high performance across numerous concepts. In this
context, we test ProMark with an exponentially increasing number of concepts.
Our dataset comprises Adobe Stock images with near duplicate image templates
(used as concepts). As we escalate the number of concepts, we concurrently
reduce the per-concept image count, only $24$ images per concept for $2^{16}$
concepts, see the red curve of Fig. 5 (a) for image count. This is done to
obtain balanced image distribution and also to challenge ProMark’s robustness.

Table 4: Comparison with different baselines for the conditional model trained
on ImageNet dataset.

c—c—c—c  Strength Attribution Accuracy (%) $\bf\uparrow$
Method (%) Held-out data New images
ALADIN [28] \- $9.25$ $0.18$
CLIP [25] \- $60.12$ $41.01$
F-CLIP [32] \- $62.83$ $50.19$
SSCD [24] \- $37.32$ $30.10$
EKILA [5] \- $38.00$ $29.06$
$30$ $91.24$ $87.30$
ProMark $100$ $\bf 95.60$ $\bf 90.13$

Figure 5: Ablation experiments: We show the results for ablating multiple
parameters of ProMark. (a) Number of concepts, (b) watermark strength, and (c)
number of images per concept.

The outcomes, depicted in Fig. 5(a) red curve, indicate an anticipated decline
in ProMark’s efficacy in line with the increase in the number of concepts,
reducing from $100\%$ attribution accuracy for $10$ concepts (chance accuracy
$10\%$) to $82\%$ for $2^{16}$ concepts (chance accuracy $1.5e$-$3\%$). This
reduction in attribution accuracy is correlated with the reduction in bit-
secret accuracy (green curve) for every predicted secret, indicating poor
watermark recovery due to the increased confusion between the watermarks.
Notwithstanding the increased difficulty, ProMark demonstrates commendable
performance, underscoring its potential in real-world applications.

###  4.4 Conditional Diffusion Model

As the diffusion models are usually trained with conditions to guide
generation, we also evaluate using the conditional LDM model [26]. For this,
we fine-tune a model pretrained of the ImageNet dataset (see Sec. 4.1), where
the $1000$ ImageNet classes are used as model conditions and also as the
$1000$ concepts.

Evaluation Protocol In addition to the evaluation on the held-out data (see
Sec. 4.1), we also perform the quantitative evaluation on the newly sampled
images as follows. We use the labels of the ImageNet dataset as conditions to
sample $10K$ images ($10$ images per label). Using these labels as the ground-
truth concept for a newly sampled image, we compute the accuracy of the
concept predicted by the embedded watermark in the generated images.

Results The accuracies for held-out and newly sampled images are shown in Sec.
4.3. The performance on the held-out dataset for the conditional model
improves compared to the unconditional models as the label conditions provide
improved supervision for correct watermarks. ProMark also outperforms prior
embedding-based works by a large margin on both held-out and newly sampled
images. The attribution accuracy on the new images, however, is less than the
held-out data. We hypothesize that it is because newly sampled images may
contain more than one concept and can be more confusing to attribute. The high
accuracy, even for newly sampled images, suggests that ProMark exhibits higher
generalizability to unseen synthetic images.

###  4.5 Ablation Study

For the ablation experiment, we use Stock dataset with a varying number of
concepts, and we train unconditional LDM models from random initialization.

Strength of Watermark. The hyperparameter $m$ in Eq. 2 modulates the intensity
of the watermark applied to the training images, ensuring encrypted images
retain high quality. We systematically alter $m$ to examine its impact on the
LDM’s performance and the Peak Signal-to-Noise Ratio (PSNR) of the output
images with reference to the held-out encrypted images. Fig. 5(b) shows that
attribution accuracy improves with increased $m$, plateauing beyond a
threshold of $0.5$. The discernible compromise in image quality, as evidenced
by the inverse relationship between intensity and PSNR, can be attributed to
the use of fixed watermarks obtained using RoSteALS [9], which is originally
optimized for robustness. In light of this, we select an optimal watermark
strength of $0.3$, which balances between performance and PSNR. We measured
the FID between original and newly sampled images from a pretrained ImageNet
conditional model (trained without watermark) and ProMark model (trained with
watermark), which is $13.28$ and $17.63$ respectively. This small increment
shows negligible quality loss in the generated images due to ProMark.

Number of Images Per Concept. To ascertain the optimal number of images
required per concept for effective watermark learning, we ablate by fixing the
number of concepts to $500$ and varying the number of images used to train the
LDM. Fig. 5(c) reveals that performance drops by $2.5\%$ when image count per
concept is reduced from $700$ to $10$. Remarkably, the general efficacy of
ProMark remains consistently high, suggesting a low sensitivity to the image
count per concept. These results demonstrate that ProMark can successfully
learn watermarks with as few as $10$ images per concept, highlighting its
efficiency and potential for applications with limited data availability.

Framework Design. ProMark employs BCE loss to instruct the LDM model in the
accurate embedding of bit-sequence watermarks within generated images. The
attribution performance degrades to $2\%$ when BCE loss is not used as
compared to $100\%$ in Sec. 4.1. This shows that removing BCE loss
significantly impairs the LDM’s performance, underscoring the necessity of
this supervision in helping LDM embed watermarks effectively.

Also, ProMark incorporates a secret decoder to retrieve secret bit-sequence
from synthesized images, rendering the process contingent upon the pretrained
secret decoder. In contrast, prior works [1, 3, 2] recover watermarks by
training a dedicated decoder with the main model in an end-to-end fashion. To
ablate this alternative approach, we train a standard decoder along with LDM
by optimizing for the cosine similarity between the embedded and extracted
watermarks. We see a degradation in performance from $100\%$ to $80.56\%$,
indicating that the pretrained secret decoder is a better choice for our
approach. This is due to the increased complexity of predicting watermarks of
resolution $256^{2}$ as compared to $160$-bit sequence from the encrypted
images.

##  5 Conclusion

We introduce a novel proactive watermarking-based approach, ProMark, for
causal attribution. We use predefined training concepts like styles, scenes,
objects, motifs, _etc_. to attribute the influence of training data on
generated images. We show ProMark’s is effective across various datasets and
model types, maintaining image quality while providing more accurate
attribution on a large number of concepts. Our approach can also be extended
to multi-concept attribution by embedding multiple watermarks onto the image.
Finally, for each experiment, our approach achieves a higher attribution
accuracy than the prior passive approaches. Such attribution offers
opportunities to recognize and reward creative contributions to generative AI,
underpinning new models for value creation in the future creative economy
[13].

Limitations. In evaluating ProMark, we note a trade-off between image quality
and attribution accuracy, which may need us to learn watermarks for
attribution task. Our model is currently trained with predefined concepts and
further research is needed on training paradigm when new concepts are
introduced. While we use orthogonal watermarks for varied concepts like motifs
and styles, this may not accurately reflect the interrelated nature of some
concepts, suggesting another opportunity for future research. Finally, our
results are specific to the LDM, and extending this approach to other GenAI
models could provide a better understanding of ProMark’s effectiveness.

## References

  * Asnani et al. [2022] Vishal Asnani, Xi Yin, Tal Hassner, Sijia Liu, and Xiaoming Liu.  Proactive image manipulation detection.  In _CVPR_ , 2022.
  * Asnani et al. [2023a] Vishal Asnani, Abhinav Kumar, Suya You, and Xiaoming Liu.  PrObeD: Proactive object detection wrapper.  In _NeurIPS_ , 2023a.
  * Asnani et al. [2023b] Vishal Asnani, Xi Yin, Tal Hassner, and Xiaoming Liu.  MaLP: Manipulation localization using a proactive scheme.  In _CVPR_ , 2023b.
  * Asnani et al. [2023c] Vishal Asnani, Xi Yin, Tal Hassner, and Xiaoming Liu.  Reverse engineering of generative models: Inferring model hyperparameters from generated images.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(12):15477–15493, 2023c.
  * Balan et al. [2023] Kar Balan, Shruti Agarwal, Simon Jenni, Andy Parsons, Andrew Gilbert, and John Collomosse.  EKILA: Synthetic media provenance and attribution for generative art.  In _CVPR_ , 2023.
  * Black et al. [2021] Alex Black, Tu Bui, Hailin Jin, Vishy Swaminathan, and John Collomosse.  Deep image comparator: Learning to visualize editorial change.  In _CVPR WMF_ , 2021.
  * Blattmann et al. [2022] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer.  Retrieval-augmented diffusion models.  _NeurIPS_ , 2022.
  * Bui et al. [2022] Tu Bui, Ning Yu, and John Collomosse.  RepMix: Representation mixing for robust attribution of synthesized images.  In _ECCV_ , 2022.
  * Bui et al. [2023] Tu Bui, Shruti Agarwal, Ning Yu, and John Collomosse.  RoSteALS: Robust steganography using autoencoder latent space.  In _CVPR_ , 2023.
  * Cao et al. [2023] Xirong Cao, Xiang Li, Divyesh Jadav, Yanzhao Wu, Zhehui Chen, Chen Zeng, and Wenqi Wei.  Invisible watermarking for audio generation diffusion models.  In _TPS-ISA_ , 2023.
  * Carlini et al. [2019] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _USENIX_ , 2019.
  * Chen et al. [2020] Chang Chen, Zhiwei Xiong, Xiaoming Liu, and Feng Wu.  Camera trace erasing.  In _CVPR_ , 2020.
  * Collomosse and Parsons [2024] John Collomosse and Andy Parsons.  To Authenticity, and Beyond! Building safe and fair generative AI upon the three pillars of provenance.  _IEEE Computer Graphics and Applications_ , 2024.
  * Cui et al. [2023] Yingqian Cui, Jie Ren, Han Xu, Pengfei He, Hui Liu, Lichao Sun, and Jiliang Tang.  DiffusionShield: A watermark for copyright protection against generative diffusion models.  _arXiv preprint arXiv:2306.04642_ , 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Fernandez et al. [2023] Pierre Fernandez, Guillaume Couairon, Hervé Jégou, Matthijs Douze, and Teddy Furon.  The stable signature: Rooting watermarks in latent diffusion models.  In _ICCV_ , 2023.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _NeurIPS_ , 2020.
  * Kirchenbauer et al. [2023] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein.  A watermark for large language models.  In _ICML_ , 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu.  Ablating concepts in text-to-image diffusion models.  In _ICCV_ , 2023.
  * Liu et al. [2023] Yugeng Liu, Zheng Li, Michael Backes, Yun Shen, and Yang Zhang.  Watermarking diffusion model.  _arXiv preprint arXiv:2305.12502_ , 2023.
  * Meng et al. [2023] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _CVPR_ , 2023.
  * Nguyen et al. [2021] Eric Nguyen, Tu Bui, Vishy Swaminathan, and John Collomosse.  OSCAR-Net: Object-centric scene graph attention for image attribution.  In _ICCV_ , 2021.
  * Peng et al. [2023] Sen Peng, Yufei Chen, Cong Wang, and Xiaohua Jia.  Protecting the intellectual property of diffusion models by the watermark diffusion process.  _arXiv preprint arXiv:2306.03436_ , 2023.
  * Pizzi et al. [2022] Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze.  A self-supervised descriptor for image copy detection.  In _CVPR_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Ruiz et al. [2020] Nataniel Ruiz, Sarah Adel Bargal, and Stan Sclaroff.  Disrupting deepfakes: Adversarial attacks against conditional image translation networks and facial manipulation systems.  In _ECCVW_ , 2020.
  * Ruta et al. [2021] Dan Ruta, Saeid Motiian, Baldo Faieta, Zhe Lin, Hailin Jin, Alex Filipkowski, Andrew Gilbert, and John Collomosse.  ALADIN: All layer adaptive instance normalization for fine-grained style similarity.  In _ICCV_ , 2021.
  * Sablayrolles et al. [2020] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou.  Radioactive data: tracing through training.  In _ICML_ , 2020.
  * Tan et al. [2019] Wei Ren Tan, Chee Seng Chan, Hernan Aguirre, and Kiyoshi Tanaka.  Improved ArtGAN for conditional synthesis of natural image and artwork.  _IEEE Transactions on Image Processing_ , 28(1):394–409, 2019.
  * Wang et al. [2021] Run Wang, Felix Juefei-Xu, Meng Luo, Yang Liu, and Lina Wang.  FakeTagger: Robust safeguards against deepfake dissemination via provenance tracking.  In _ACM MM_ , 2021.
  * Wang et al. [2023] Sheng-Yu Wang, Alexei A Efros, Jun-Yan Zhu, and Richard Zhang.  Evaluating data attribution for text-to-image models.  In _ICCV_ , 2023.
  * Wilber et al. [2017] Michael J Wilber, Chen Fang, Hailin Jin, Aaron Hertzmann, John Collomosse, and Serge Belongie.  BAM! the behance artistic media dataset for recognition beyond photography.  In _ICCV_ , 2017.
  * Yao et al. [2024] Yuguang Yao, Xiao Guo, Vishal Asnani, Yifan Gong, Jiancheng Liu, Xue Lin, Xiaoming Liu, and Sijia Liu.  Reverse engineering of deceptions on machine- and human-centric attacks.  _Foundations and Trends in Privacy and Security_ , 2024.
  * Yeh et al. [2020] Chin-Yuan Yeh, Hsi-Wen Chen, Shang-Lun Tsai, and Sheng-De Wang.  Disrupting image-translation-based deepfake algorithms with adversarial attacks.  In _WACVW_ , 2020.
  * Yu et al. [2015] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Zhao et al. [2023] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin.  A recipe for watermarking diffusion models.  _arXiv preprint arXiv:2303.10137_ , 2023.

## ProMark: Proactive Diffusion Watermarking for Causal Attribution
– Supplementary material –

##  6 Multiple Watermark Configurations

We investigate the application of dual watermarks, each positioned on opposing
sides of the image. This exploration raises a pivotal query: “Is the spatial
positioning of watermarks critical to the performance?” To answer this, we
ablate four distinct watermark configurations. As shown in Sec. 6, there is a
consistent performance across all watermark placements (left, right, top,
bottom), thereby substantiating the spatial robustness of ProMark in watermark
positioning.

Table 5: Multi-concept attribution performance across different
configurations.

c—c—c—c—c  Configuration Attribution Accuracy (%) $\bf\uparrow$
Secret $1$ Secret $2$ Secret $1$ Secret $2$ Combined
Left Right $95.61$ $93.31$ $90.12$
Right Left $95.52$ $93.35$ $90.19$
Top Bottom $95.66$ $93.70$ $90.01$
Bottom Top $95.02$ $93.46$ $90.73$

##  7 Watermark Robustness

We test our method against $14$ different degradations (blur, various noises,
fog, etc.), by adopting the evaluation protocol detailed in the RoSteALS [9].
We use $50$ watermarked training images from LSUN dataset and use
unconditional LDM with a strength of $30\%$. The average attribution accuracy
for training and generated images across all $14$ attacks is $90.21\pm 7.63\%$
and $89.51\pm 8.18\%$, as compared to $95.12\%$ without any degradation,
showing the robustness of our approach to multiple forms of watermark attack.

##  8 Possibility of Concept Leakage

We present multiple results where we attribute the images generated using non-
watermarked data, for example via random latent code and conditional
generation. We detect no retention of the watermark after noising or in random
latent codes, with watermark detection accuracy of $50.56\%$ (chance $50\%$)
after noising for $\geq 900$ timestamps or in random latent codes. The LDM
generates an image from noise through inversion, and the watermark is added
during this GenAI model inference process. Our decoder is employed
independently to identify the concept. To prove this, we evaluate our model in
Table $4$ (main paper) for two more baselines, using held-out images ($1$)
with no watermark encryption, and ($2$) encrypted with a different concept’s
watermark. ProMark is able to attain an attribution accuracy of $94.32\%$ and
$94.01\%$ respectively when evaluated with ground-truth concept watermark for
both baselines compared to $95.60\%$ reported for watermarked held-out data.
Therefore, when inverting generating images that encrypt no watermark, or
encrypt incorrect watermark, the correct concept watermark is encrypted.

##  9 Computational Efficiency

We demonstrate the computation efficiency of ProMark during inference (running
watermark decoder to perform causal attribution), which costs $5.6$ms on one
A$100$ GPU. Training with watermarked data adds negligible cost to generative
model training. This is comparable to running inference on CLIP, or ALADIN to
perform correlation based attribution ($28.32$ ms) but the additional cost of
the embedding search is $87.91$ ms for a dataset of $20$K LSUN training
images. ProMark therefore offers the advantage of both efficiency and
causality for training data attribution. We will add this to the paper.

##  10 Additional Watermark Strength Analysis

Our research introduces a new paradigm in concept attribution for images
classified under multiple concepts. We show the analysis of PSNR variation
with watermark strength for the case of multi-concept attribution. The results
are shown in Fig. 6. Our findings indicate that, compared to single watermark
cases, the PSNR for multi-concept images is marginally higher at equivalent
watermark strengths. However, as expected, an increase in watermark strength
generally leads to a decrease in PSNR.

Furthermore, we have visualized images from different datasets to showcase the
extent of degradation caused by varying watermark strengths. As discussed in
Sec. $4.5$, the performance of our method improves with increased watermark
strength. Nevertheless, this increase in strength leads to a decline in image
quality, evidenced by the emergence of bubble-like artifacts in the images, as
shown in Fig. 7 (the watermark strength ranges from $0.1$ to $1.0$).

Figure 6: PSNR  _vs_. watermark strength for single vs multi-concept
attribution.  Figure 7: Noise Strength visualization for different watermark
strength

##  11 Watermark Discussion

We visualize some sample watermarks in both, spatial and frequency domain in
Fig. 8. These watermarks are converted from bit-sequences to spatial domain as
described in Sec. $3.4$. Visually, the watermarks appear indistinguishable
from one another in both domains. Yet, their orthogonality is clearly
demonstrated through the cosine similarity matrix, which we used to analyze
$100$ different watermarks. This matrix reveals that the inter-watermark
cosine similarity is consistently close to zero, decisively indicating the
orthogonal nature of these watermarks.

Spatial Domain | Fourier Domain | Correlation Matrix
---|---|---
|  |
Figure 8: Watermark Visualization: Spatial domain, Fourier domain and inter-
watermark cosine similarity for $100$ watermarks.

##  12 Implementation Details

We train ProMark with LDM for $15K$ iterations with a batch size of $32$,
using $8$ NVIDIA A$100$ GPUs for each experiment. We use the default
parameters for optimizers as used in the official repository of [26]. The
learning rate is set at $3.2e^{-5}$ for training LDM.

We further show the architecture for the generic decoder used for comparing
against pretrained secret decoder shown in Fig. 9. The generic decoder
consists of $2$ stem convolution layers and $10$ convolution blocks. Each
block consists of convolutional and batch normalization layers followed by
ReLU activation.

Figure 9: Generic decoder architecture.

##  13 More Sampled Images

We use multiple datasets for evaluating ProMark. We sample images from the
trained LDM for every class. We show some of the train and sampled images for
the corresponding classes for different datasets in LABEL:, 11, 10, 12 and 13.
We argue that ProMark is able to perform attribution to different types of
concepts, _i.e_. image templates (Fig. 10), image style (Fig. 13), style and
content (Fig. 11), and ownership (Fig. 12). Therefore, proactive based causal
methods perform attribution not only on the style or motif of the image as
done by correlation based works, but also performs attribution to a variety of
concepts proving it’s generalizability.

Figure 10: Training and sampled images for stock dataset.  Figure 11: Training
and sampled images for BAM dataset.  Figure 12: Training and sampled images
for wiki-a dataset.  Figure 13: Training and sampled images for wiki-s
dataset.


* (29) Dunnhofer, M., Antico, M., Sasazawa, F., Takeda, Y., Camps, S., Martinel, N., Micheloni, C., Carneiro, G., Fontanarosa, D.: Siam-U-Net: encoder-decoder siamese network for knee cartilage tracking in ultrasound images.  Medical Image Analysis (2020)
  * (30) Dunnhofer, M., Martinel, N., Foresti, G.L., Micheloni, C.: Visual Tracking by means of Deep Reinforcement Learning and an Expert Demonstrator.  In: ICCVW (2019)
  * (31) Dunnhofer, M., Martinel, N., Micheloni, C.: Weakly-supervised domain adaptation of deep regression trackers via reinforced knowledge distillation.  IEEE RA-L
  * (32) Dunnhofer, M., Martinel, N., Micheloni, C.: Tracking-by-Trackers with a Distilled and Reinforced Model.  In: ACCV (2020)
  * (33) Everingham, M., Eslami, S., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: A retrospective.  IJCV (2015)
  * (34) Fan, H., Bai, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Huang, M., Liu, J., Xu, Y., et al.: Lasot: A high-quality large-scale single object tracking benchmark.  IJCV (2021)
  * (35) Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C., Ling, H.: LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking.  In: CVPR (2019)
  * (36) Fan, H., Miththanthaya, H.A., Harshit, Rajan, S.R., Liu, X., Zou, Z., Lin, Y., Ling, H.: Transparent object tracking benchmark.  In: ICCV (2021)
  * (37) Fu, Z., Liu, Q., Fu, Z., Wang, Y.: Stmtrack: Template-free visual tracking with space-time memory networks.  In: CVPR (2021)
  * (38) Furnari, A., Battiato, S., Grauman, K., Farinella, G.M.: Next-active-object prediction from egocentric videos.  Journal of Visual Communication and Image Representation (2017)
  * (39) Furnari, A., Farinella, G.: Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video.  IEEE TPAMI (2020)
  * (40) Furnari, A., Farinella, G.M.: What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention.  In: ICCV (2019)
  * (41) Galoogahi, H.K., Fagg, A., Huang, C., Ramanan, D., Lucey, S.: Need for Speed: A Benchmark for Higher Frame Rate Object Tracking.  In: ICCV (2017)
  * (42) Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., Martin, M., Nagarajan, T., Radosavovic, I., Ramakrishnan, S.K., Ryan, F., Sharma, J., et al.: Ego4d: Around the World in 3,000 Hours of Egocentric Video.  In: CVPR (2022)
  * (43) Guo, D., Shao, Y., Cui, Y., Wang, Z., Zhang, L., Shen, C.: Graph attention tracking.  In: CVPR (2021)
  * (44) Han, S., Liu, B., Cabezas, R., Twigg, C.D., Zhang, P., Petkau, J., Yu, T.H., Tai, C.J., Akbay, M., Wang, Z., Nitzan, A., Dong, G., Ye, Y., Tao, L., Wan, C., Wang, R.: MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality.  ACM Transactions on Graphics (2020)
  * (45) Hare, S., Golodetz, S., Saffari, A., Vineet, V., Cheng, M.M., Hicks, S.L., Torr, P.H.: Struck: Structured Output Tracking with Kernels.  IEEE TPAMI (2016)
  * (46) Held, D., Thrun, S., Savarese, S.: Learning to Track at 100 FPS with Deep Regression Networks.  ECCV (2016)
  * (47) Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters.  IEEE TPAMI (2015)
  * (48) Huang, L., Zhao, X., Huang, K.: GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild.  IEEE TPAMI (2019)
  * (49) Huang, L., Zhao, X., Huang, K.: GlobalTrack: A Simple and Strong Baseline for Long-term Tracking.  In: AAAI (2020)
  * (50) Kalal, Z., Mikolajczyk, K., Matas, J.: Tracking-learning-detection.  IEEE TPAMI (2012)
  * (51) Kapidis, G., Poppe, R., Van Dam, E., Noldus, L., Veltkamp, R.: Egocentric hand track and object-based human action recognition.  In: IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (2019)
  * (52) Kapidis, G., Poppe, R., Van Dam, E., Noldus, L., Veltkamp, R.: Egocentric hand track and object-based human action recognition.  In: IEEE SmartWorld, Ubiquitous Intelligence and Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Internet of People and Smart City Innovation (2019)
  * (53) Kiani Galoogahi, H., Fagg, A., Lucey, S.: Learning Background-Aware Correlation Filters for Visual Tracking.  In: CVPR (2017)
  * (54) Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J.K., Danelljan, M., Zajc, L.C., Lukezic, A., Drbohlav, O., He, L., Zhang, Y., Yan, S., Yang, J., Fernández, G., et al.: The eighth visual object tracking vot2020 challenge results.  In: ECCVW
  * (55) Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Zajc, L.C., Vojir, T., Hager, G., Lukezic, A., Eldesokey, A., Fernandez, G., et al.: The Visual Object Tracking VOT2017 Challenge Results.  In: ICCVW (2017)
  * (56) Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Pflugfelder, R., Kämäräinen, J.K., Chang, H.J., Danelljan, M., Cehovin, L., Lukezic, A., Drbohlav, O., Käpylä, J., Häger, G., Yan, S., Yang, J., Zhang, Z., Fernández, G.: The ninth visual object tracking vot2021 challenge results.  In: ICCVW (2021)
  * (57) Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Pflugfelder, R., Kämäräinen, J.K., Zajc, L., Drbohlav, O., Lukežič, A., Berg, A., Eldesokey, A., Käpylä, J., Fernández, G., et al.: The Seventh Visual Object Tracking VOT2019 Challenge Results.  In: ICCVW (2019)
  * (58) Kristan, M., Matas, J., Leonardis, A., Vojíř, T., Pflugfelder, R., Fernández, G., Nebehay, G., Porikli, F., Čehovin, L.: A novel performance evaluation methodology for single-target trackers.  IEEE TPAMI (2016)
  * (59) Li, A., Lin, M., Wu, Y., Yang, M.H., Yan, S.: NUS-PRO: A New Visual Tracking Challenge.  IEEE TPAMI (2016)
  * (60) Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., Yan, J.: SIAMRPN++: Evolution of siamese visual tracking with very deep networks.  In: CVPR (2019)
  * (61) Li, F., Tian, C., Zuo, W., Zhang, L., Yang, M.H.: Learning Spatial-Temporal Regularized Correlation Filters for Visual Tracking.  In: CVPR (2018)
  * (62) Li, M., Wang, Y.X., Ramanan, D.: Towards streaming perception.  In: A. Vedaldi, H. Bischof, T. Brox, J.M. Frahm (eds.) ECCV (2020)
  * (63) Li, Y., Liu, M., Rehg, J.M.: In the eye of beholder: Joint learning of gaze and actions in first person video.  In: ECCV (2018)
  * (64) Liang, P., Blasch, E., Ling, H.: Encoding Color Information for Visual Tracking: Algorithms and Benchmark.  IEEE TIP (2015)
  * (65) Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context.  In: ECCV
  * (66) Liu, M., Tang, S., Li, Y., Rehg, J.: Forecasting human object interaction: Joint prediction of motor attention and actions in first person video.  In: ECCV (2020)
  * (67) Lu, X., Ma, C., Ni, B., Yang, X., Reid, I., Yang, M.H.: Deep regression tracking with shrinkage loss.  In: ECCV (2018)
  * (68) Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., Leibe, B.: Hota: A higher order metric for evaluating multi-object tracking.  IJCV (2021)
  * (69) Lukezic, A., Kart, U., Kapyla, J., Durmush, A., Kamarainen, J.K., Matas, J., Kristan, M.: CDTB: A color and depth visual object tracking dataset and benchmark.  In: ICCV (2019)
  * (70) Lukežič, A., Matas, J., Kristan, M.: D3S – A Discriminative Single Shot Segmentation Tracker.  In: CVPR (2020)
  * (71) Lukezic, A., Zajc, L.C., Vojir, T., Matas, J., Kristan, M.: Performance evaluation methodology for long-term single-object tracking.  IEEE Transactions on Cybernetics (2020)
  * (72) Ma, M., Fan, H., Kitani, K.M.: Going deeper into first-person activity recognition.  In: CVPR (2016)
  * (73) Maggio, D.E., Cavallaro, D.A.: Video Tracking: Theory and Practice.  Wiley Publishing (2011)
  * (74) Maresca, M.E., Petrosino, A.: MATRIOSKA: A multi-level approach to fast tracking by learning.  In: ICIAP (2013)
  * (75) Mayer, C., Danelljan, M., Paudel, D.P., Gool, L.V.: Learning target candidate association to keep track of what not to track (2021)
  * (76) Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., Theobalt, C.: Real-Time Hand Tracking Under Occlusion from an Egocentric RGB-D Sensor.  In: ICCVW (2017)
  * (77) Mueller, M., Smith, N., Ghanem, B.: A Benchmark and Simulator for UAV Tracking.  In: ECCV (2016)
  * (78) Müller, M., Bibi, A., Giancola, S., Alsubaihi, S., Ghanem, B.: TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild.  In: ECCV (2018)
  * (79) Nam, H., Han, B.: Learning Multi-domain Convolutional Neural Networks for Visual Tracking.  CVPR (2016)
  * (80) Nam, H., Hong, S., Han, B.: Online graph-based tracking.  In: ECCV (2014)
  * (81) Nigam, J., Rameshan, R.M.: EgoTracker: Pedestrian Tracking with Re-identification in Egocentric Videos.  In: CVPRW (2017)
  * (82) Park, E., Berg, A.C.: Meta-tracker: Fast and Robust Online Adaptation for Visual Object Trackers.  In: ECCV (2018)
  * (83) Pirsiavash, H., Ramanan, D.: Detecting activities of daily living in first-person camera views.  In: CVPR (2012)
  * (84) Ragusa, F., Furnari, A., Livatino, S., Farinella, G.M.: The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain.  In: WACV (2020)
  * (85) Rai, A., Sener, F., Yao, A.: Transformed rois for capturing visual transformations in videos.  arXiv (2021)
  * (86) Real, E., Shlens, J., Mazzocchi, S., Pan, X., Vanhoucke, V.: Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video.  In: CVPR (2017)
  * (87) Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You Only Look Once: Unified, Real-Time Object Detection.  In: CVPR (2016)
  * (88) Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks.  In: NeurIPS (2015)
  * (89) Rodin, I., Furnari, A., Mavroedis, D., Farinella, G.M.: Predicting the future from first person (egocentric) vision: A survey.  CVIU (2021)
  * (90) Ross, D.A., Lim, J., Lin, R.S., Yang, M.H.: Incremental learning for robust visual tracking.  IJCV (2008)
  * (91) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge.  IJCV (2015)
  * (92) Sener, F., Singhania, D., Yao, A.: Temporal aggregate representations for long-range video understanding.  In: ECCV (2020)
  * (93) Shan, D., Geng, J., Shu, M., Fouhey, D.F.: Understanding human hands in contact at internet scale.  In: CVPR (2020)
  * (94) Smeulders, A.W.M., Chu, D.M., Cucchiara, R., Calderara, S., Dehghan, A., Shah, M.: Visual tracking: An experimental survey.  IEEE TPAMI (2014)
  * (95) Song, Y., Ma, C., Wu, X., Gong, L., Bao, L., Zuo, W., Shen, C., Lau, R.W., Yang, M.H.: VITAL: VIsual Tracking via Adversarial Learning.  In: CVPR (2018)
  * (96) Sun, L., Klank, U., Beetz, M.: EYEWATCHME—3D Hand and object tracking for inside out activity analysis.  In: CVPRW (2010)
  * (97) De la Torre, F., Hodgins, J.K., Montano, J., Valcarcel, S.: Detailed human data acquisition of kitchen activities: the cmu-multimodal activity database (cmu-mmac).  In: Workshop on Developing Shared Home Behavior Datasets to Advance HCI and Ubiquitous Computing Research, in conjuction with CHI (2009)
  * (98) Valmadre, J., Bertinetto, L., Henriques, J.F., Tao, R., Vedaldi, A., Smeulders, A.W., Torr, P.H., Gavves, E.: Long-Term Tracking in the Wild: A Benchmark.  In: ECCV (2018)
  * (99) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need.  In: NeurIPS (2017)
  * (100) Visee, R.J., Likitlersuang, J., Zariffa, J.: An Effective and Efficient Method for Detecting Hands in Egocentric Videos for Rehabilitation Applications.  IEEE Transactions on Neural Systems and Rehabilitation Engineering (2020)
  * (101) Wang, N., Zhou, W., Tian, Q., Hong, R., Wang, M., Li, H.: Multi-cue Correlation Filters for Robust Visual Tracking.  In: CVPR (2018)
  * (102) Wang, N., Zhou, W., Wang, J., Li, H.: Transformer meets tracker: Exploiting temporal context for robust visual tracking.  In: CVPR (2021)
  * (103) Wang, Q., Gao, J., Xing, J., Zhang, M., Hu, W.: DCFNet: Discriminant Correlation Filters Network for Visual Tracking (2017)
  * (104) Wang, Q., Zhang, L., Bertinetto, L., Hu, W., Torr, P.H.S.: Fast Online Object Tracking and Segmentation: A Unifying Approach.  In: CVPR (2019)
  * (105) Wang, X., Wu, Y., Zhu, L., Yang, Y.: Symbiotic attention with privileged information for egocentric action recognition.  In: AAAI (2020)
  * (106) Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a deep association metric.  In: ICIP (2018)
  * (107) Wu, C.Y., Feichtenhofer, C., Fan, H., He, K., Krahenbuhl, P., Girshick, R.: Long-term feature banks for detailed video understanding.  In: CVPR (2019)
  * (108) Wu, Y., Lim, J., Yang, M.H.: Online object tracking: A benchmark.  In: CVPR (2013)
  * (109) Wu, Y., Lim, J., Yang, M.H.: Object tracking benchmark.  IEEE TPAMI (2015)
  * (110) Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., Huang, T.: Youtube-vos: Sequence-to-sequence video object segmentation.  In: ECCV (2018)
  * (111) Xu, Y., Wang, Z., Li, Z., Yuan, Y., Yu, G.: SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines.  In: AAAI (2020)
  * (112) Yan, B., Peng, H., Fu, J., Wang, D., Lu, H.: Learning spatio-temporal transformer for visual tracking.  In: ICCV (2021)
  * (113) Yan, B., Peng, H., Wu, K., Wang, D., Fu, J., Lu, H.: Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search.  In: CVPR (2021)
  * (114) Yan, B., Zhao, H., Wang, D., Lu, H., Yang, X.: ’Skimming-perusal’ tracking: A framework for real-time and robust long-term tracking.  In: ICCV (2019)
  * (115) Yun, S., Choi, J., Yoo, Y., Yun, K., Choi, J.Y.: Action-decision networks for visual tracking with deep reinforcement learning.  In: CVPR (2017)
  * (116) Zhang, J., Ma, S., Sclaroff, S.: MEEM: Robust tracking via multiple experts using entropy minimization.  In: ECCV (2014)
  * (117) Zhang, L., Gonzalez-Garcia, A., Weijer, J.V.D., Danelljan, M., Khan, F.S.: Learning the model update for siamese trackers.  In: ICCV (2019)
  * (118) Zhang, Z., Peng, H.: Deeper and Wider Siamese Networks for Real-Time Visual Tracking.  CVPR (2019)
  * (119) Zhang, Z., Peng, H., Fu, J., Li, B., Hu, W.: Ocean: Object-aware Anchor-free Tracking.  In: ECCV (2020)