11institutetext: 1University of Central Florida, $\quad{}^{2}$Google Research
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
# Anti-Neuron Watermarking: Protecting Personal Data Against Unauthorized
Neural Networks

Zihang Zou 11 Boqing Gong 22 Liqiang Wang 11

###### Abstract

We study protecting a user’s data (images in this work) against a learner’s
unauthorized use in training neural networks. It is especially challenging
when the user’s data is only a tiny percentage of the learner’s complete
training set. We revisit the traditional watermarking under modern deep
learning settings to tackle the challenge. We show that when a user watermarks
images using a specialized linear color transformation, a neural network
classifier will be imprinted with the signature so that a third-party
arbitrator can verify the potentially unauthorized usage of the user data by
inferring the watermark signature from the neural network. We also discuss
what watermarking properties and signature spaces make the arbitrator’s
verification convincing. To our best knowledge, this work is the first to
protect an _individual_ user’s data ownership from unauthorized use in
training neural networks.

##  1 Introduction

Recent advances in machine learning techniques have put personal data at
significant risk. For example, in the scandal of “Cambridge Analytica” [45],
millions of users’ data are collected without consent to train machine
learning models for political advertising. To protect personal data and
privacy, there have been some legislations in place, such as Europe General
Data Protection Regulation [12] (effective in May 2018), California Privacy
Act [2] (effective in January 2021), and China Data Security Law [40]
(effective in July 2021). They often require that personal data should be
“processed lawfully, fairly and in a transparent manner” and can only be used
“adequately, relevantly and limited to what is necessary in relation to the
purposes (‘data minimisation’)” [12]. However, there is a lack of methods for
detecting personal data breaches from machine learning models, which have
increasingly become the primary motivation for a violator to break a user’s
data ownership because the models’ efficacy heavily depends on data.

This paper studies personal image protection (PIP) from unauthorized usage in
training deep neural networks (DNNs). The need for PIP arises when users
expose their images to digital products and cloud services. In the era of big
data and deep learning, a critical concern is that DNN learners may violate
users’ intents by using their data to train DNNs without authorization. It
becomes worse when the DNN models consequently leak private user information
[1, 10, 33, 37]. However, how can ordinary users know whether their images,
which could be a tiny portion of the DNN learner’s complete training set, have
been used to train a DNN model?

Traditionally, PIP aims to prevent a user’s images from duplicating, remixing,
or exploiting (e.g., for a financial incentive) without the user’s consent and
relies on digital watermarking [5, 23, 29, 42, 49, 54]. The digital
watermarking enables a user to imprint images with unique patterns, such as
signatures, logos, or stamps, to track and identify unauthorized _copies_ of
their pictures.

However, the rise of data-dependent deep learning poses another need for PIP,
namely, protecting a user’s images from unauthorized use in training DNNs.
Could watermarking still fulfill this need?

One inspiring observation is that some DNNs do “memorize” certain training
examples [1, 9, 10] in various ways, offering a user an opportunity to
watermark their images to make them memorizable by the DNNs. We say this
watermarking scheme is “anti-neuron” because its objective is to facilitate a
third-party arbitrator to verify a DNN’s use of a user’s images in training
and then hold the DNN learner accountable. However, we have to resolve two
questions to make this anti-neuron watermarking work in practice. What
watermarks make a user’s images memorizable by DNNs? How can the third-party
arbitrator verify that the user’s images were indeed part of a DNN model’s
training set?

Figure 1: Illustration of the anti-neuron watermarking for personal image
protection (PIP) against unauthorized neural learners.

To answer the above questions, we first use Figure 1 to formalize the anti-
neuron watermarking for PIP against unauthorized DNN learners. First, a user
watermarks images using a private signature before sharing them with the
public (e.g., social media). An unauthorized learner then collects the user’s
watermarked images, along with images from other sources, to construct a
training set to train a DNN image classifier. Finally, the user turns to a
third-party arbitrator to check whether their images were used to train the
DNN model. The arbitrator tries to recover the user’s private signature for
watermarking from the DNN model and the user’s original images with no
watermark — crucially, the arbitrator does not use the user’s private
signature to recover it. The arbitrator concludes that the user’s images were
part of the DNN’s training set if the user’s private signature can be
recovered without knowing it in advance.

This paper proposes an empirically effective approach to the anti-neuron
watermarking, and we leave more rigorous analyses to future work. In
particular, a linear color transformation (LCT) in the hue space can be
effectively used as a watermarking method. The resultant images remain as
appealing as the original ones visually, so the unauthorized learner would not
detect this type of watermark. Moreover, the LCT method is resilient to
standard image augmentation techniques used in training neural models.
Finally, we show that a DNN classifier indeed tends to memorize the LCT
watermarking using extensive experiments. The arbitrator’s verification method
is simply iterating over the signature space, watermarking the user’s original
images using each signature, and returning the signature that reaches the
lowest DNN classification loss.

In summary, our main contribution is to formalize the problem of user-focused
anti-neuron watermarking for personal image protection from unauthorized usage
in training DNNs. Moreover, we propose the LCT watermarking for ordinary users
and a straightforward verification method for the third-party arbitrator,
demonstrating a successful anti-neuron watermarking scenario for PIP.
Additionally, we raise some critical questions for furthering the study of PIP
against DNN learners: 1) What types of watermarking can imprint DNNs the best,
especially when a user’s watermarked images are only a tiny part of the
training set? 2) What makes the imprinting of DNNs possible? Is it the DNN’s
memorization of training examples? 3) How can a trustworthy arbitrator recover
a user’s private watermark from DNN and the user’s unwatermarked images? 4)
How can anti-neuron watermarking work for multiple users? To the best of our
knowledge, this work is the first to protect an individual user’s data
ownership from unauthorized use in training neural networks.

##  2 Related Work

Watermarking is a long-standing technique to declare ownership of objects. It
can be traced back to paper marking [28] at 1282 in Italy, where a watermark
was created via changing the thickness of the paper. Digital watermark is
later introduced by [42] to code an undetectable digital watermark on gray
scale images. Yu et al. [49] train a neural network for watermarking to embed
hidden information in images. El et al. [8] add digital watermarks to video
frames with neural networks. Zhong et al. [54] propose an automated and robust
image watermarking based on deep neural networks. Recently, watermarking is
used to protect the intellectual property of machine learning models [15, 31,
34, 52]. These techniques follow similar ideas as trojan attacks [25] or
backdoor attack [14], where models are trained with constructed samples to
learn objective behaviors.

The most relevant works to this paper are dataset tracing and membership
inference. Dataset tracing [35, 24, 27] protects the intellectual property of
a dataset by appending traceable watermarks on data samples. Sablayrolles et
al. [35] use pretrained model on a dataset itself to generate “radioactive
data” to carry the class-specific watermarking vectors in high dimensional
feature space. If a learner uses the “radioactive dataset” for training, the
model’s classifier would become more aligned with the watermarking vectors and
thus can be used as an evidence of unauthorized usage. This kind of
watermarking requires a pretrained model trained with the whole dataset.
However, as such prior knowledge about the entire training data is unavailable
for a common user, this kind of technique would be less applicable in real PIP
scenarios.

Membership inference determines if a certain sample is inside a target
dataset. Inference attack was first proposed for the attack and defense on
medical datasets where users’ medical records are extremely sensitive. By
comparing genomic data with the statistical information of the training
dataset, the presence of certain users can be inferred by attackers [17].
Shokri et al. [37] later introduce membership inference attack (MIA) into
machine learning models. MIA trains a binary classifier to predict membership,
on top of several shadow models being trained with the same data distribution
as training. Alternatively, Yeom et al. [47] use the average of training error
as the threshold to perform MIAs. Sablayrolles et al. [36] improve this
threshold with Bayes optimal classifier to search for the best threshold using
samples from both training and testing.

As MIAs determine whether given data samples belong to a training set or not,
it is tempting to perform MIAs for personal data protection. However, similar
to dataset tracing, as the training data distribution is unknown for common
users, neither shadow models [37] nor threshold [47, 36] can be obtained and
existing MIA methods fail to work in protecting personal data.

##  3 Problem Statement

Consider image classification as a case study without loss of generality.
Denote by $\mathcal{D}_{u}$ the set of personal images owned by a common user
$u$. Assume $\mathcal{D}_{u}$ is unique and distinguished among identifiable
users, as defined in GDPR [12]. Suppose the user $u$ plans to expose
$\mathcal{D}_{u}$ online, e.g., by sharing them on social media. For the
purpose of avoiding potential breach of personal data proprietary, the user
watermarks images with a secret signature $k^{*}$ before sharing them on
social media. Denote by $\mathcal{D}_{u}^{*}$ the set of watermarked images
carrying the signature $k^{*}$.

An unauthorized learner may use the user’s data $\mathcal{D}_{u}^{*}$, along
with many others’, to construct a training set $\mathcal{D}$ to train a DNN
classifier $f$ without acquiring the user’s permission. It is reasonable to
assume that the user’s data $\mathcal{D}_{u}^{*}$ is only a small portion of
the whole training set $\mathcal{D}$ and the user $u$ does not have any prior
knowledge about the other users’ data.

Let $g\in\mathcal{G}$ denote a watermarking method and $\mathcal{V}$ be a
neutral third-party verification method that infers the user’s private
signature for watermarking without knowing it before. The arbitrator
determines whether the user’s personal images have been used in the training
of neural classifier $f$ as follows,

$\displaystyle\mathcal{V}(f,\mathcal{D}_{u},\mathcal{G})=k^{*}\textbf{ iff
}\mathcal{D}_{u}^{*}\subseteq\mathcal{D}$ (1)

where the user’s watermarked images
$\mathcal{D}_{u}^{*}=g(D_{u},k^{*}),g\in\mathcal{G}$. Namely, if the
arbitrator can recover the user’s private signature, she/he concludes that the
user’s images were part of the training set for learning the classifier $f$.

##  4 Approach

###  4.1 The Anti-Neuron Watermarking Method

Recent studies show that DNNs can “memorize” some training examples in various
ways [1, 9, 10], and one can recover certain meaningful low-resolution images
from DNNs [10]. Hence, it is tempting to conduct verification by recovering
the user $u$’s images from the neural classifier $f$. However, there are many
challenges with this approach. First of all, the model $f$ may memorize some
training images but not this user $u$’s. Moreover, even if the model happens
to memorize some of this user’s images, the recovery success rate is likely
low. Existing methods (e.g., [10]) can recover semantically meaningful images
from some DNNs, but they do not resemble any exact training images, to the
best of our knowledge. Finally but not the least, the method in [10] incurs
high computation cost, often by many iterations of gradient descent, and
assumes that the DNN classifier $f$ is a white box, disclosing its
architecture and parameters.

An alternative approach to leveraging DNNs’ memorization capability is to
check a DNN’s loss over a set of training images. Arguably, if the DNN model
has memorized a majority of this set of images, the loss should be low.
Following the above reasoning, we let a user $u$ watermark her/his images
$\mathcal{D}_{u}$ using a private signature $k^{*}$ so the user has full
control and knowledge of her/his watermarked images $\mathcal{D}_{u}^{*}$.
This watermarking method eases the third-party arbitrator’s job; instead of
trying to recover the exact training images, the arbitrator can now search for
the watermarking signature that leads to the lowest DNN loss, if the DNN model
has memorized many images in $\mathcal{D}_{u}^{*}$ watermarked by user $u$.

####  4.1.1 Properties for Effective Watermarking.

Formally, a user $u$ chooses an anti-neuron watermarking function
$g\in\mathcal{G}$ and generates the watermarked images as

$\mathcal{D}_{u}^{*}=\\{g(I,k^{*}),\forall I\in\mathcal{D}_{u}\\}$ (2)

We discuss the necessary properties needed to make a good anti-neuron
watermarking function. The key is to make the watermarked images, and hence
the signature, memorized by DNNs. First, the watermarking function $g$ should
_preserve an image’s original content_. For example, for a user portrait or
selfie, $g$ should not change its identity. Besides, the watermarking function
$g$ should be _resilient to common image augmentations_ used to train DNNs.
The private signature should survive after the learner applies common image
augmentations. Furthermore, the space $K$ of watermarking signatures should be
_large_ and preferably _bounded_ , such that the probability of an innocent
classifier coincidentally matching the user signature is low, while the
signature can be inferred efficiently during verification.

####  4.1.2 Linear Color Transformation.

Based on the discussion above, we propose Linear Color Transformation (LCT) as
our anti-neuron watermarking. Color provides a large signature space for
images. Our watermarking function exploits hue transformation and uses the hue
adjustment of images as a signature. Thanks to the sufficiently big hue space,
the user’s randomly chosen signature is likely different from other users’
signatures. Moreover, the randomly chosen signature lifts the user images to a
low-density region, making the resultant images be easily memorized by DNNs —
according to Feldman’s studies on memorization [9] and our experiments in
Section 5.6, DNNs tend to memorize images of low-density regions.

Concretely, we first convert the RGB color space into the YIQ color space [44]
by the following matrix:

$T_{\text{YIQ}}=\begin{bmatrix}0.299&0.587&0.114\\\ 0.596&-0.275&-0.321\\\
0.212&-0.523&0.311\\\ \end{bmatrix}$ (3)

In the YIQ color space, hue is represented by two dimensional coordinates,
forming a chromaticity diagram. As a result, watermarking images with
signature $k$ will be conducted by rotating the hue at an angle $\theta_{k}$
with the following matrix,

$T_{k}=\begin{bmatrix}1&0&0\\\ 0&cos(\theta_{k})&-sin(\theta_{k})\\\
0&sin(\theta_{k})&cos(\theta_{k})\\\ \end{bmatrix}$ (4)

where $\theta_{k}=\frac{k\pi}{180}$. Hence, for every pixel
$v=[v_{r},v_{g},v_{b}]^{\intercal}$ in image $I$, we can watermark $v$ with
signature $k$ by:

$v{{}^{\prime}}=g_{k}v$ (5)

where $g_{k}=T_{\text{YIQ}}\cdot T_{k}\cdot T^{-1}_{\text{YIQ}}$.

##### Making LCT more versatile.

An immediate extension to LCT is to make the color transformation matrix
$T_{\text{YIQ}}$ specifiable by users. A user chosen color transformation
$T_{u}$ can further enrich the watermarking signature space. We leave this
extension to future work.

###  4.2 The Verification Method

We let a third-party arbitrator independent of the user and DNN learner
determine whether the user’s images were part of the DNN training set. The
arbitrator has to infer a signature from a suspicious DNN classifier $f$ and
the user’s original, unwatermarked images $\mathcal{D}_{u}$ without using the
private watermark signature $k^{*}$. If the inferred signature matches the
user’s private one, we say that the DNN classifier is highly likely trained
using the user’s images $\mathcal{D}_{u}^{*}$.

Assume that the watermarking function $g(I,k^{*})$ does not change the image’s
class label. Let $y$ denote the class label of image $I\in\mathcal{D}_{u}$. We
design a simple yet effective approach to recovering the watermarking
signature:

$\hat{k}\leftarrow\arg\min_{k\in\mathcal{K}}\sum_{(I,y)\in\mathcal{D}_{u}}\mathcal{L}(f(g(I,k)),y)$
(6)

where $\mathcal{L}$ is a loss (e.g., cross-entropy) for learning the DNN
classifier $f$, and $\mathcal{K}$ is the collection of all possible
signatures.

If the inferred signature matches the user’s private one, $\hat{k}\approx
k^{*}$, the arbitrator concludes that the DNN learner has used the user’s
images $D_{u}^{*}=\\{g(I,k^{*}),\forall I\in\mathcal{D}_{u}\\}$ as part of the
training set for DNN $f$. Otherwise, the DNN learner is likely innocent.

####  4.2.1 The Signature Space $\mathcal{K}$.

It is important to discuss the success rate of the above verification method.
Apparently, the signature space $\mathcal{K}$ should be sufficiently large to
reduce the probability of an innocent classifier coincidentally matching a
user’s watermarked images. For the analysis purpose, we discretize the bounded
signature space $\mathcal{K}$ into $N$ equal-sized, non-overlapped slots, each
with an interval $2\tau$. We say the recovered signature $\hat{k}$ matches the
private one $k^{*}$ when $|\hat{k}-k^{*}|<\tau$. Reserving one slot for no
watermarking, the number of valid watermarking signatures is $N-1$. Clearly,
the larger $N$ is, the more convincing the verification.

Some readers might wonder what if there is a large number of users. For
example, given 1 million users but a small number $N$ of signatures, would
this setting fail the proposed anti-neuron watermarking? The answer is a
pleasant no because, importantly, two users could choose the same private
watermarking signature as long as their personal images are different, though
the chance of using the same signature is low because each user independently
chooses a signature. What happens when a user chooses not to watermark her/his
images? A well-trained neural classifier should generalize well under the
training distribution. Hence, if most training images are not watermarked,
given the user’s original unwatermarked data, the recovered signature from the
well-trained classifier would approach no watermarking.

It is not necessary to have enormous $N$ to avoid users having duplicated
private signatures based on the above discussion. However, a sufficiently
large $N$ is still preferred for another reason, DNNs’ memorization. Only when
$N$ is big, the chance becomes high for a user to watermark her/his images
into a low-density region and hence can be memorized by DNNs.

A large signature space also benefits the memorization of user signatures.
According to the study [9] on memorization, deep neural classifier must
memorize atypical examples to perform well on the less frequent examples
during inference. Since watermarking shift data distribution via signature
from a large space, watermarking is highly likely to lift user images into
lower density region and thus being better memorized by neural models.

####  4.2.2 Optimization Method and Computational Cost for Signature
Inference.

To solve eq. (6) efficiently, we propose two optimization methods. (i) Grid
search : the arbitrator can enumerate all signatures for watermarking and
perform grid search over the bounded signature space with a linear
computational cost as $O(N)$. If the signature is well memorized by a DNN, the
DNN loss will reach minimum when the signature being evaluated equals or
closely approximates the private signature used by the user. (ii) Gradient
search: when the model is accessible, the arbitrator can watermark clean
images with a random initial watermark signature and then infer the user’s
signature by descending along the gradient of training loss with respect to
the signature. This technique infers the watermark signature more precisely
than grid search and the computational cost might be less for a large $N$.

##  5 Experiments

###  5.1 Setup

We evaluate the proposed watermarking in image classification on the Cifar-10
/ Cifar-100 [3], CUB birds [43] and Tiny ImageNet [20] datasets.

A User Watermarks Their Personal Data. A portion of randomly chosen images
from a training set (by default, $1\%$ for Cifar and $0.1\%$ for Tiny
ImageNet) is defined as a user’ personal data. The user data could contain
samples from any class. Each user image is watermarked using eq. (5) by a
given signature in the space of $[30,60,...,330]$, followed by clipping pixel
values to the valid range of [0,1]. By default, we use 60 (i.e., rotating hue
by 60 degree) as the signature.

A Learner Trains Neural Classifiers Using Unauthorized User Data. An
unauthorized learner trains neural classifiers using the above watermarked
user data along with other training data. Images are randomly cropped,
horizontal flipped, and normalized following the common data augmentation
practice [16, 22]. We use ResNet50 [16] as the default neural classifier and
train every model from scratch for 90 epochs. The initial learning rate is
$0.1$ and decays by $0.1$ for every 30 epochs.

A Verifier Infers The Watermark Signature. Given suspicious neural network
models, a third-party verifier infers the user’s signature following two
approaches discussed above. For _grid search_ , we iterate over all candidate
signatures generated by dividing the whole signature space into $N=12$
intervals whose length is $2\tau=2\times 15$. For _gradient search_ , we
exploit gradient descent to learn the signature. To avoid local optimum,
multiple initial values are used and the best signature that leads to the
lowest loss is returned.

###  5.2 Analyzing Effectiveness of Watermarking

We first show empirically how signatures are memorized by the neural
classifiers. Here, we consider a single user watermarking their data for
simplicity. (See Appendix for other experiments and the gradient search
results.)

  1. 1.

Different Numbers of Watermarked Samples. We study how many images are desired
for making anti-neuron watermarking successful. The grid search result is
shown in Figure 2 (a, b, c) using eq. (6). It is visually clear that most of
the models achieve the minimum loss near the watermark signature, within the
range of matching $|\hat{k}-k^{*}|<\tau$. However, with less watermarked data
(e.g., less than 5 samples), the inferred signature with minimum loss does not
match the user’s private signature.

  2. 2.

Different Watermark Signatures. We verify whether different watermark
signatures work equivalently. We experiment with different signatures on one
user’s data and show the grid search results in Figure 2 (d, e, f) for
different datasets. From these figures, we observe that all inferred
signatures (marked in square) match the user’s signatures for watermarking,
indicating that different hue adjustments can all be used for anti-neuron
watermarking.

  3. 3.

Different Neural Classifier Architectures. We also evaluate the proposed
watermarking for different neural classifier architectures, including Alexnet
[22], VGG [38], ResNet [16], Wide ResNet [50] and DenseNet [18] trained with
default settings. As shown in Figure 2(g), all inferred signatures match the
user’s, implying that our watermarking approach works well against a large
variety of deep neural networks.

  4. 4.

Different Learning Capacities of Models. We further investigate whether a
model memorizes watermark signatures better when the model has more learning
capacity (e.g., more parameters, deeper or wider) by exploring the ResNet
family. As shown in Figure 2(h), as the networks go larger and deeper, the
loss decreases faster and reaches the minimum around the watermark signature
more sharply.

  5. 5.

High Resolution Images. In Figure 2(i), we present our result on
CUB-200-Birds, a fine-grained dataset with high-resolution images of
$448\times 448$. We use pretrained ResNet50 from ImageNet and conduct a
transfer learning on CUB-200-Birds. The dataset has fewer than 6000 images for
training, and we assume the user has 60 images (1%) for watermarking. Strong
data augmentations [48] are used to boost performance, including color jitter,
random crop, random resize, random scale and random horizontal flip. Even
under the strong data augmentations and the transfer learning setting, the
result shows that ResNet50 memorizes the user’s signature well.

(a) Cifar 10.

(b) Cifar 100.

(c) Tiny ImageNet.

(d) Cifar 10.

(e) Cifar 100.

(f) Tiny ImageNet.

(g) Different architectures.

(h) Different capacity.

(i) High resolution.

Figure 2: The first row shows variant of loss for models trained with
different quantity of watermarked samples on Cifar and Tiny ImageNet. The
second row shows variant of loss for models with different signatures on Cifar
and Tiny ImageNet. The third row shows results for different architectures,
different capacity on Tiny ImageNet and high resolution on CUB-200-Birds. The
x-axis represents signature space (denoted by distance to the user private
signature, i.e., $k-k^{*}$), and the y-axis represents the average loss of
user data with respect to signature $k$. The green region represents the range
for a match ($<2\tau$). If the inferred signature $\hat{k}$ (marked as Square
marker indicating the point with minimum loss) lies in the green region
($|\hat{k}-k^{*}|<\tau$), it would be a match. Otherwise, it would be a miss.

###  5.3 Analyzing Properties of Watermarking

We then evaluate how the properties discussed in Section 4.1.1 would help
anti-neuron watermarking.

Resilience to Data Augmentation. We evaluate if our anti-neuron watermarking
is resilient against various common data augmentations, especially those
involving random hue transformations. We apply random crop and random
horizontal flip in all our experiments following [22, 16]. Besides, several
widely adopted data augmentations including random cutout [6], label smoothing
[41], Gaussian noise [4], adversarial training [26] and differential privacy
[7] are evaluated. Finally, we test color jittering [22], which includes
brightness, saturation, contrast and the same hue transformation we used for
watermarking. As shown in Figure 3, the watermark signatures can be inferred
correctly for the aforementioned data augmentations. This shows empirically
that LCT is an effective anti-neuron watermarking approach because it is
resilient to common data augmentations in neural networks’ training. We also
evaluate privacy preserving techniques such as differential privacy [7]. Since
we infer the signature using all user images, noise added to the output would
be reduced by taking an average. Beside, we also consider two common _defense
techniques_ against watermarking: pruning and fine-tuning. We follow common
settings [53] and find in Figure 3(g) and Figure 3(h) that LCT is also
resilient to these defense methods.

(a) Cut Out.

(b) Label Smoothing.

(c) Gaussian Noise.

(d) Adversarial Training.

(e) Differential Privacy.

(f) Color Jitter.

(g) Pruning.

(h) Fine-tuning.

Figure 3: The variation of model loss for different data augmentations. Only
the color jitter can significantly narrow down the loss difference between
signatures.

Less Noticeable Watermarking. In Section 4.1.1, we discuss the watermarking
should not change the major content of an image, and one of the desired
features is to make watermarking unnoticeable to human. Rather than changing
the hue of images globally, we study an alternative technique from traditional
watermarking proposed in [49]. It adjusts the blue channel’s intensity on pre-
selected pixels. In this work, we adjust the intensity as a watermark
signature on 512 randomly selected pixels. As shown in Figure 4, the general
appearance of watermarked images are less noticeable than changing the hue
globally (adjusting hue for 4096 pixels by 60). For $\tau=0.1$ and watermark
signatures (the blue channel’s intensity), 0.1, 0.3, 0.5, 0.12, 0.28, 0.44 are
used in inference on selected pixels, matching the user’s watermark signature.
However, this kind of watermarking has its limitation. It introduces noise to
images, and the images could look noisy when the color themes are dominated by
red or green. To solve this problem, we may find some transformations that are
invisible to humans but easy to learn by neural classifiers. We leave this
challenge to future work.

(a) Clean

(b) Hue

(c) 0.1

(d) 0.3

(e) 0.5

Figure 4: Illustration of clean, hue-transformed and less noticeable
watermarked samples. 0.1,0.3 and 0.5 are the intensity of blue channel for
selected pixels.

###  5.4 Analyzing Signature Space for Verification

In Section 4.2.1, we discuss how the signature space could affect watermarking
from the perspective of a third-party verifier. Here, we show experimentally
how to have a trustful signature space for a convincing verification.

When User Data Was Not Used to Train Neural Classifiers. From previous
experiments, we show that the inferred signature matches a user’s private
watermark signature if the user’s watermarked data have been used in training.
Here we show the inferred signature approaches no watermarking when the user
data was not used for training. To this end, we construct held-out users using
auxiliary unseen data from validation. Pretrained models from Figure 2(f) are
used to infer signature from the held-out users. Not surprisingly, the
inferred signatures approach 0 (no watermarking) for the held-out users, with
$|\hat{k}-0|=4.3\pm 1.4$.

Multiple Users with User-specific Watermarking Signatures. We examine multiple
users for several scenarios. The training set of Tiny ImageNet is equally
divided into 1,000 users. Then we evaluate the effectiveness of watermarking
for different ratio of users exploiting the same LCT (eq. (3)) or different
LCTs. For the later, we sample $3\times 3$ matrices from a uniform
distribution $T_{u}\sim\mathcal{U}(-1,1)$ per user. Each user chooses a random
signature from $[30,60,...,330]$, and $\tau$ is set to 15.

From the result shown in Figure 5(a), we can observe that when $20\%$ of users
watermark their images using LCT, their watermark signatures can be inferred
correctly for almost all the users. As this ratio increases, the matching
accuracy drops significantly if the users use the same LCT. However, if they
use different LCTs, the matching accuracy remains above $80\%$ even when all
users data are watermarked independently. We also evaluate a special case when
an adversary infers signatures using an arbitrary LCT. The arbitrary LCTs
($T_{u^{{}^{\prime}}}\sim\mathcal{U}(-1,1)$) only achieve $10\%$ matching
accuracy, which is $70\%$ less when LCTs are given. Such results indicate
users can use unique and user-specific watermarking for a better protection
rate when other users may also exploit watermarking.

(a)

(b)

(c)

Figure 5: (a) Watermarking performance for multiple users. (b) MIAs fail to
work with user-specific shadow models because insufficient training data leads
to much higher threshold. (c) Our ANW vs. MIAs using the advP metric.

###  5.5 Comparisons with Related Methods

Comparing to Membership Inference Attacks. We compare our anti-neuron
watermarking (ANW) with two _threshold-based_ membership inference attacks
(MIAs): In MIA-std [47], few samples are known in violator’s training set and
their average training loss would be used as threshold $\epsilon$ to infer
membership (if $\mathcal{L}_{t}<\epsilon$, target sample $t$ is in training,
vice versa.); In MIA-pow [36], few samples in training and held-out together
determine a better threshold.

There are 3 scenarios making MIAs not applicable for PIP. (i) as a user has no
knowledge about the violator’s training distribution, MIAs cannot be applied
as neither shadow models nor threshold can be obtained. (ii) Users could train
shadow model using their own data and perform MIAs via shadow models [37], but
MIAs would not work well because shadow models trained with user data would
produce a much larger loss than the learner’ models. Consequently, if a
threshold is chosen from shadow models trained with user data, held-out sample
would be misclassified as “in training” because the learner’s model would
produce smaller loss for both “training” and “held-out” samples. Here we
compare MIAs by creating 10 users with individual data. 10 shadow models are
trained respectively and MIA-std is performed with the threshold of average
training loss. For ANW, 10 users data are watermarked with different
signatures. Under different settings for user data size, we find out that ANW
achieves 100% matching accuracy while MIA-std achieves 50% (misclassify all
held-out samples). Figure 5(b) shows how the threshold from shadow models
fails to classify sample membership for the learner. Similarly, MIA-pow along
with other MIAs [37, 27] relied on shadow models would also fail in this
settings. (iii) At last, even if a user acquires all necessary information,
the MIA results would still be less convincing. As MIAs only provide binary
output (True/False), it is difficult to convince the verifier when random
guess can still achieve about $50\%$ success rate. To quantifiably and fairly
compare with MIAs, we extend the classic _membership advantage_ [46] into
_protection advantage_ (advP) considering both accuracy and fidelity:

$\text{adv\textsuperscript{P}}=\int_{x}(\mathbb{P}_{e}(y=True|x)-\mathbb{P}_{r}(y=True|x))dx$
(7)

The advP metric quantifies quality of protection through the expectation gap
between the empirical successful inference ($\mathbb{P}_{e}$) and successful
random guess ($\mathbb{P}_{r}$). Note that the _membership advantage_ [46] is
a special case of advP metric when the second term is $0.5$. For a Bernoulli
experiment, the above formula could be calculated as $\frac{M-Np}{N}$, where
$M$ is the total matches in $N$ experiments and $p$ is the probability of a
correct random guess. With above metric, we conduct comparison between our
watermarking and two threshold-based MIAs [47, 36] (See appendix for
experiment settings.). As shown in Figure 5(c), our ANW significantly
outperforms the MIA approaches under the advP metric with both accurate and
convincing inference, showing that watermarking is a feasible method in the
PIP problem.

Comparing to Dataset Tracing [35]. Dataset tracing [35] exploits pretrained
classifier to generate traceable data. If neural classifier learns such
dataset, the decision boundary of classifier would become more aligned with
watermarking vectors (i.e., cosine similarity becomes higher). In Table 1, we
compare this approach [35] with ours when only 0.1% data being watermarked.
For dataset tracing, it is computed by the classifier’s weight vector and the
watermarking vector. And for our method, it is computed by inferred and user
watermarking signatures. The experimental results show that it is easier to
memorize low dimensional signatures as our watermarking method lifts the
cosine similarity significantly after training with tiny portion of
watermarked data.

| Before Training | After Training
---|---|---
Dataset Tracing[35] | $-0.005\pm 0.030$ | $-0.005\pm 0.015$
Our Watermarking | $0.045\pm 0.52$ | $0.999\pm 0.001$
Table 1: Cosine similarity. With only $0.1\%$ data watermarked, the dataset
tracing shows almost no effects while our watermarking method improves the
similarity to almost 1 after training.

###  5.6 Improving Memorization by Watermarking

Finally, we explore empirically why watermarking is effective against neural
classifier by revisiting “Memorization Value Estimate” (MAE) [9]. MAE measures
the generalization gap (difference of prediction between models trained
with/without certain data) to quantify the memorization ability of neural
networks toward such data. A higher MAE after watermarking indicates the model
tends to memorize watermarked data than original data. For user data, we
observe the MAE increases from $26.8\%$ to $34.8\%$ after applying
watermarking, indicating that our approach increases memorization of user data
and thus the signature would be easier memorized along with user data.

##  6 Conclusion

In this paper, we introduce a new personal data protection problem against
unauthorized neural model training. To protect user personal data, we propose
an anti-neuron watermarking approach based on linear color transformation. By
watermarking user’s images with private signature using LCT, unauthorized
usage of user personal data can be verified by a third-party neutral
arbitrator. Through extensive experiments, we show empirically that LCT-based
watermarking is effective in protecting user data from unauthorized usage in a
various realistic settings.

Acknowledgements: this work was supported in part by NSF-1704309 and
NSF-1952792.

## References

  * [1] Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., Song, D.: The secret sharer: Evaluating and testing unintended memorization in neural networks. In: 28th USENIX Security Symposium (USENIX Security 19). pp. 267–284 (2019)
  * [2] CCPA: California privacy act (Jan 2021), https://oag.ca.gov/privacy/ccpa
  * [3] Cifar: (2009), https://www.cs.toronto.edu/~kriz/cifar.html, cIFAR Dataset
  * [4] Cohen, J., Rosenfeld, E., Kolter, Z.: Certified adversarial robustness via randomized smoothing. In: International Conference on Machine Learning. pp. 1310–1320. PMLR (2019)
  * [5] Cox, I.J., Miller, M.L., Bloom, J.A., Honsinger, C.: Digital watermarking, vol. 53. Springer (2002)
  * [6] DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
  * [7] Dwork, C.: Differential privacy: A survey of results. In: International conference on theory and applications of models of computation. pp. 1–19. Springer (2008)
  * [8] El’arbi, M., Amar, C.B., Nicolas, H.: Video watermarking based on neural networks. In: 2006 IEEE International conference on multimedia and expo. pp. 1577–1580. Ieee (2006)
  * [9] Feldman, V., Zhang, C.: What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprint arXiv:2008.03703 (2020)
  * [10] Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. pp. 1322–1333 (2015)
  * [11] Garay, J.A., Gennaro, R.: Advances in Cryptology–CRYPTO 2014: 34th Annual Cryptology Conference, Santa Barbara, CA, USA, August 17-21, 2014, Proceedings, Part II, vol. 8617. Springer (2014)
  * [12] GDPR: Regulation (eu) 2016/679 (general data protection regulation) that is applicable as of may 25th, 2018 in all member states, is to the protection of natural persons with regard to the processing of personal data and rules relating to the free movement of personal data. (2016), https://gdpr-info.eu/
  * [13] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
  * [14] Gu, T., Liu, K., Dolan-Gavitt, B., Garg, S.: Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7, 47230–47244 (2019)
  * [15] Guo, J., Potkonjak, M.: Watermarking deep neural networks for embedded systems. In: 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). pp. 1–8. IEEE (2018)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J.V., Stephan, D.A., Nelson, S.F., Craig, D.W.: Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLoS Genet 4(8), e1000167 (2008)
  * [18] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
  * [19] Jia, J., Salem, A., Backes, M., Zhang, Y., Gong, N.Z.: Memguard: Defending against black-box membership inference attacks via adversarial examples. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. pp. 259–274 (2019)
  * [20] kaggle: (2017), https://www.kaggle.com/c/tiny-imagenet, tiny Imagenet
  * [21] Kaya, Y., Dumitras, T.: When does data augmentation help with membership inference attacks? In: International Conference on Machine Learning. pp. 5345–5355. PMLR (2021)
  * [22] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012)
  * [23] Kundur, D., Hatzinakos, D.: Digital watermarking using multiresolution wavelet decomposition. In: Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP’98 (Cat. No. 98CH36181). vol. 5, pp. 2969–2972. IEEE (1998)
  * [24] Li, Y., Zhang, Z., Bai, J., Wu, B., Jiang, Y., Xia, S.T.: Open-sourced dataset protection via backdoor watermarking. arXiv preprint arXiv:2010.05821 (2020)
  * [25] Liu, Y., Ma, S., Aafer, Y., Lee, W.C., Zhai, J., Wang, W., Zhang, X.: Trojaning attack on neural networks. NDSS Symposium (2017)
  * [26] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: International Conference on Learning Representations (2018), https://openreview.net/forum?id=rJzIBfZAb
  * [27] Maini, P., Yaghini, M., Papernot, N.: Dataset inference: Ownership resolution in machine learning. arXiv preprint arXiv:2104.10706 (2021)
  * [28] Meggs, P.B.: A History of Graphic Design. Wiley (1998)
  * [29] Meng, Z., Morizumi, T., Miyata, S., Kinoshita, H.: Design scheme of copyright management system based on digital watermarking and blockchain. In: 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). vol. 2, pp. 359–364. IEEE (2018)
  * [30] Müller, R., Kornblith, S., Hinton, G.: When does label smoothing help? arXiv preprint arXiv:1906.02629 (2019)
  * [31] Nagai, Y., Uchida, Y., Sakazawa, S., Satoh, S.: Digital watermarking for deep neural networks. International Journal of Multimedia Information Retrieval 7(1), 3–16 (2018)
  * [32] Nasr, M., Shokri, R., Houmansadr, A.: Machine learning with membership privacy using adversarial regularization. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. pp. 634–646 (2018)
  * [33] Nasr, M., Shokri, R., Houmansadr, A.: Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In: 2019 IEEE symposium on security and privacy (SP). pp. 739–753. IEEE (2019)
  * [34] Rouhani, B.D., Chen, H., Koushanfar, F.: Deepsigns: an end-to-end watermarking framework for protecting the ownership of deep neural networks. In: ACM International Conference on Architectural Support for Programming Languages and Operating Systems (2019)
  * [35] Sablayrolles, A., Douze, M., Schmid, C., Jégou, H.: Radioactive data: tracing through training. In: International Conference on Machine Learning. pp. 8326–8335. PMLR (2020)
  * [36] Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., Jégou, H.: White-box vs black-box: Bayes optimal strategies for membership inference. In: International Conference on Machine Learning. pp. 5558–5567. PMLR (2019)
  * [37] Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. In: 2017 IEEE Symposium on Security and Privacy (SP). pp. 3–18. IEEE (2017)
  * [38] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [39] Song, C., Shmatikov, V.: Overlearning reveals sensitive attributes. In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=SJeNz04tDS
  * [40] Standing Committee of the National People’s Congress: China data security law (Jul 2021), http://www.xinhuanet.com/2021-06/11/c_1127552204.htm
  * [41] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
  * [42] Tirkel, A.Z., Rankin, G., Van Schyndel, R., Ho, W., Mee, N., Osborne, C.F.: Electronic watermark. Digital Image Computing, Technology and Applications (DICTA’93) pp. 666–673 (1993)
  * [43] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD Birds 200. Tech. Rep. CNS-TR-2010-001, California Institute of Technology (2010)
  * [44] Wikipedia: Yiq (2015), http://en.wikipedia.org/wiki/YIQ
  * [45] Wikipedia: Cambridge analytica (2018), https://en.wikipedia.org/wiki/Cambridge_Analytica
  * [46] Yeom, S., Fredrikson, M., Jha, S.: The unintended consequences of overfitting: Training data inference attacks. arXiv preprint arXiv:1709.01604 12 (2017)
  * [47] Yeom, S., Giacomelli, I., Fredrikson, M., Jha, S.: Privacy risk in machine learning: Analyzing the connection to overfitting. In: 2018 IEEE 31st Computer Security Foundations Symposium (CSF). pp. 268–282. IEEE (2018)
  * [48] Yu, F., Wang, D., Shelhamer, E., Darrell, T.: Deep layer aggregation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2403–2412 (2018)
  * [49] Yu, P.T., Tsai, H.H., Lin, J.S.: Digital watermarking based on neural networks for color images. Signal processing 81(3), 663–671 (2001)
  * [50] Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
  * [51] Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning (still) requires rethinking generalization. Communications of the ACM 64(3), 107–115 (2021)
  * [52] Zhang, J., Chen, D., Liao, J., Fang, H., Zhang, W., Zhou, W., Cui, H., Yu, N.: Model watermarking for image processing networks. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12805–12812 (2020)
  * [53] Zhang, J., Chen, D., Liao, J., Zhang, W., Hua, G., Yu, N.: Passport-aware normalization for deep model protection. Advances in Neural Information Processing Systems 33, 22619–22628 (2020)
  * [54] Zhong, X., Huang, P.C., Mastorakis, S., Shih, F.Y.: An automated and robust image watermarking scheme based on deep neural networks. IEEE Transactions on Multimedia 23, 1951–1961 (2020)

##  Appendix 0.A Background

In the following sections, we present the background knowledge of personal
data protection problem studied in our paper. We first quote the legislative
definition of “personal data protection” from GDPR [12] and then we discuss
the scope of our problem comparing to general data protection. Finally, some
“data proprietary” problem are addressed in personal data protection.

Definitions of Terms from GDPR.  To help readers better understand the terms
we used in the manuscript, we quote several term definitions from [12] as
following:

“ _Personal data_ means any information relating to an identified or
identifiable natural person (‘data subject’).”

“ _Third party_ means a natural or legal person, public authority, agency or
body other than the data subject, controller, processor and persons who, under
the direct authority of the controller or processor, are authorised to process
personal data.”

“ _Consent_ of the data subject means any freely given, specific, informed and
unambiguous indication of the data subject’s wishes by which he or she, by a
statement or by a clear affirmative action, signifies agreement to the
processing of personal data relating to him or her.”

“ _Personal data breach_ means a breach of security leading to the accidental
or unlawful destruction, loss, alteration, unauthorised disclosure of, or
access to, personal data transmitted, stored or otherwise processed.”

From above definitions, we show the basic idea of GDPR: a legislation approach
to protect users personal data from being utilized by third-party without
users’ consent. More details of GDPR and similar laws can be found online [12,
2, 40]. In our paper, we focus on the technical perspective of personal data
protection by exploiting anti-neuron watermarking to verify unauthorized usage
of user’s personal images.

Personal Data Protection v.s General Data Protection. As our goal is to
protect “Personal Data” (i.e. user data shall be related to identifiable
natural person [12]), we assume each user’s personal images can be
distinguished from the others. This assumption differs “personal data
protection” from “general data protection” problem, where a user’s data are
not necessarily to be identified from other users’. In “general data
protection”, our anti-neuron watermarking would be less applicable. Because
neural models learn both watermarked and unwatermarked data, the inferred
signature would be a signature value between the watermarked and
unwatermarked. We leave this challenge for future studies.

“Data proprietary” in Personal Data Protection. One traditional usage of
digital watermarking is to protect data proprietary. To verify proprietary,
the space of watermarking signature needs to enormous such that each signature
(i.e., a hand-writing signature) can be considered as a unique identifier for
user data. However, in our problem settings, our protection focuses on
verifying unauthorized usage (i.e., without “consent”) of user data (images
that user already has “data proprietary” on) instead of verifying data
proprietary. Since the low-dimensional signature is only used to verify
unauthorized usage of images, our signature space does not need to be as large
as “data proprietary” problem. Readers might be curious what if adversary
exploits user’s watermarking function and reverts a signature on arbitrary
images that leads to minimal loss of arbitrary neural models. Can adversary
accuse data privacy breach for these models? Can adversary claim ownership of
these arbitrary data? The answers are no. According to our previous analysis
on signature space, the signature inferred on arbitrary images is highly
likely to be no watermarking, a signature value excluded from valid
watermarking signatures. On the other hand, adversary cannot claim data
propriety on arbitrary data as they cannot prove that the data are legally
“relating to” adversarial users.

##  Appendix 0.B Preliminary Study

Verification by Recovering Watermarking Pattern. Recent studies show that DNNs
can “memorize” some training examples in various ways [1, 9, 10], and one can
recover certain meaningful low-resolution images from CNNs [10]. Motivated by
these studies, we perform a preliminary study on traditional watermarking
technique by appending a special pattern (e.g., a sticker) on images. We train
a ResNet50 on 100 randomly selected user samples with a cat pattern as
watermark in Tiny ImageNet. Similar to model inversion [10], we use a
learnable Gaussian variable on users’ images and minimize the classification
loss to reconstruct watermark pattern with this variable.

As shown in Figure 6, although the reconstructed pattern can achieve $100\%$
accuracy and minimal classification loss, neural models cannot memorize such
watermark pattern, as no meaningful pattern can be recovered. We believe that
this could be caused by convolution operation where all spatial information of
pixels are being ignored during training. This experiment indicates that it is
difficult for neural learners to memorize such kinds of watermarks, comparing
to the watermarks using color-based transformation.

Figure 6: Illustration of watermark pattern (left), recovered pattern (middle)
and watermarked images (right). Recovering small visible watermark fails with
getting noise.

##  Appendix 0.C Implementation Details

Datasets. We evaluate our anti-neuron watermarking on Cifar [3], Tiny ImageNet
[20] and CUB-200-Birds [43]. Cifar are widely adopted datasets with 50,000
training samples, 10,000 testing samples for 10 and 100 classes, respectively.
Tiny ImageNet is a selective subset of ImageNet, containing 100,000 training
samples, 10,000 validation samples and 10,000 testing samples for 200 classes.
Each sample is $3\times 64\times 64$. Since testing labels are not publicly
available, we report models’ validation accuracy for models’ performance.
CUB-200-Bird is a high resolution $448\times 448$ fine-grained dataset
containing 200 bird species, with 5994 training samples and 5794 testing
samples.

Watermark User Data. Each user image is watermarked by given $3\times 3$ LCT
function with a signature $k$ followed by a pixel value clipping. A clipping
is needed because LCT could cause overflow on some pixels’ values. The
clipping operation in theory will break the differentiable property of
watermarking function and thus hinder gradient based optimization (e.g.,
stochastic gradient decent). However, we find empirically that this operation
does not affect signature inference. Hence, we conduct clipping after the LCT
in all our experiments.

Data Preparation. For training, each image is first converted from $[0,255]$
to $[0,1]$, and watermarked if it belongs to the user whose images need to be
protected. Then data augmentation and normalized are applied to improve
training. During signature inference, each image is converted from $[0,255]$
into $[0,1]$, then watermarked and normalized. The normalization mean and
variance of RGB channels are $(0.5,0.5,0.5)$ and $(0.5,0.5,0.5)$ for Cifar,
$(0.485,0.456,0.406)$ and $(0.229,0.224,0.225)$ for Tiny ImageNet and CUB-
Birds, respectively.

Grid Search Settings. We iterate all the possible signatures in our grid
search experiments. The signatures are generated by dividing the whole
signature space into $N\times 2\tau$ intervals.

Gradient Search Settings. During signature inference, we recover signature
using unwatermarked user data. These user data are combined into one mini-
batch and stochastic gradient decent is used for optimization over signatures.
The initial learning rate is 0.1 and decays 0.1 each 100 epochs, with 300
epochs in total. To avoid local minima, we select initial values from all
possible signatures and report the signature that lead to minimal loss.
Comparing to grid search, this optimization approach is much more
computational expensive and thus our results will be mainly on grid search for
simplicity.

Source Code. Our experimental source code will be publicly available (code is
available in supplementary folder). All our experiments are implemented by
Pytorch. Readers can freely explore our proposed watermarking approach with
the code supplied.

##  Appendix 0.D Additional Results for Effectiveness of Watermarking

Gradient Search Results. In the main manuscript, we present the results for
grid search in various settings. Here, we show the additional gradient search
results on Tiny ImageNet.

  1. 1.

Different sizes of Watermarked Samples. We first present our result with
different sizes of watermarkred samples in Table 2. Being trained by
classifier, user’s watermarked images achieve lower loss than the clean
images. And watermarking does not affect classification performance for
getting similar testing accuracy. These results show that given sufficient
data, neural classifier could memorize watermark signature on user’s data
pretty well.

# of Data |  Model Acc |  Watermark Loss |  Clean Loss |  Inferred signature
---|---|---|---|---
10,000 | 55.6 | 0.019 | 0.161 | 59.0 ✓
1,000 | 55.8 | 0.016 | 0.541 | 56.4 ✓
100 | 54.9 | 0.019 | 0.668 | 59.5 ✓
10 | 54.5 | 0.017 | 0.329 | 48.9 ✓
5 | 54.6 | 0.001 | 0.397 | 60.99 ✓
1 | 55.9 | 0.004 | 0.006 |  17.0 $\times$
Table 2: Inferred signatures for models trained with different sizes of
watermarked data on Tiny ImageNet. The watermark signature is 60 with
$\tau=15$.

  2. 2.

Different Watermark Signatures. We present the gradient search result for
different signatures on Tiny ImageNet in Table 3. The result shows that the
watermarking for single user does not affect models’ training as models’
accuracy are similar for different signatures. User watermarked images achieve
lower average loss than original images, indicating unauthorized training
models can memorize watermarked images in “some way”. And we achieve minimal
loss nearby watermarking signature, bounded by predefined threshold $\tau=15$.
This experiment shows that different signatures of watermarking work
equivalently.

Watermark signature |  Model Acc |  Watermark Loss |  Clean Loss |  Inferred signature
---|---|---|---|---
60 | 54.9 | 0.019 | 0.668 | 59.5 ✓
120 | 55.6 | 0.070 | 0.895 | 120.2 ✓
180 | 53.8 | 0.030 | 1.189 | 178.8 ✓
240 | 56.3 | 0.025 | 1.118 | 242.1 ✓
300 | 55.7 | 0.016 | 0.839 | 306.1 ✓
Table 3: Watermarking for different signatures for ResNet50 on Tiny ImageNet.

  3. 3.

Different Neural Classifier Architectures. We present the gradient search
result for different architectures on Tiny ImageNet in Table 4. Alexnet [22],
VGG [38], ResNet [16], Wide ResNet [50] and DenseNet [18] are evaluated in
this experiment. The watermark signature is 60 and $\tau=15$.

Architecture |  Model Acc |  Watermark Loss |  Clean Loss |  Inferred signature
---|---|---|---|---
Alex | 38.0 | 2.189 | 3.175 | 56.9✓
VGG | 57.3 | 0.640 | 1.151 | 52.8✓
Res | 54.9 | 0.019 | 0.668 | 59.5✓
Wide Res | 56.6 | 0.005 | 0.738 | 58.4✓
Dense | 61.5 | 0.114 | 0.838 | 58.8✓
Table 4: Watermarking for different architectures on Tiny ImageNet.

  4. 4.

Different Learning Capacity of Models. We present the gradient search result
for different learning capacity of models on Tiny ImageNet in Table 5. ResNet
[16] family is evaluated in this experiment. The watermark signature is 60 and
$\tau=15$.

Architecture |  Model Acc |  Watermark Loss |  Clean Loss |  Inferred signature
---|---|---|---|---
ResNet18 | 52.9 | 0.036 | 0.801 | 55.2 ✓
ResNet34 | 53.6 | 0.007 | 0.679 | 55.8 ✓
ResNet50 | 54.9 | 0.019 | 0.668 | 59.5 ✓
ResNet101 | 54.6 | 0.004 | 0.736 | 58.3 ✓
ResNet152 | 56.5 | 0.007 | 0.814 | 60.4 ✓
Table 5: Watermarking for different learning capacities on Tiny ImageNet.

##  Appendix 0.E Additional Results for Watermarking Properties Analysis

In this section, we provide more details of our experiments on analyzing the
properties of watermarking.

Resilience to Data Augmentation. Here, we present our settings to evaluate if
our proposed watermarking method could survive under common data augmentations
used in neural models’ training. The following data augmentations are
evaluated:

  1. 1.

Random Crop [22]. Random crop is a widely used data augmentation for most
neural models’ training [22, 38, 16, 48]. It randomly crops images into
smaller resolution to reduce models’ overfitting on spatial location. For Tiny
ImageNet, images are randomly cropped into $64\times 64$ with a padding of 8.
For Cifar, images are randomly cropped into $32\times 32$ with a padding of 4.
All our training includes random crop for best performance.

  2. 2.

Horizontal Flipping [22]. This is also a widely used data augmentation for
image classification. We include horizontal flipping for all our experiments.

  3. 3.

Cut Out [6]. Cutout removes random region of size $M\times M$ from images at
each training iteration. We set the size $M=8$ for our experiments.

  4. 4.

Label Smoothing [41]. Label smoothing is also a widely used data augmentation
in many tasks [30]. It reduces the probability of ground truth label (e.g.,
100% cat, 0% dog, 0% duck) by a smoothing parameter $\alpha$ and assigns
probability uniformly to other classes (90% cat, 5% dog, 5% duck). The
smoothing parameter $\alpha$ is set to be 0.1 in our experiments.

  5. 5.

Gaussian Noise [4]. This technique simply adds noise to the input from a
Gaussian distribution $\mathcal{N}(0,\sigma^{2})$ to increase models’
robustness. The $\sigma^{2}$ is set to be 0.1 in our experiments.

  6. 6.

Adversarial Training [26]. Neural Networks are well known for their
vulnerability to adversarial attacks. [13, 26], and adversarial training are
believed to reduce overfitting [26] and mitigate privacy leakage [19, 32]. We
address this by training with adversarial samples generated from FGSM attack
[13]. The epsilon is set to be 0.01.

  7. 7.

Differential Privacy [7]. Differential privacy is a wide adopted privacy
preserving technique in many real-world applications. By adding noise to the
query results, user’s sensitive information cannot be inferred via querying.
In our implementation, we add random noise samples from Gaussian distribution
$\mathcal{N}(0,\sigma^{2})$ to the output confidence. The $\sigma^{2}$ is set
to be 0.1.

  8. 8.

Color Jitter [22]. Color jitter randomly adjusts brightness, contrast,
saturation and hue of input images. We apply high intensity color augmentation
in our experiments. For each color properties, the value of adjustment is
randomly sampled from [-288, 288], covering $80\%$ of the range of
transformation. The same conclusion can be made by comparing Table 3 with
Table 6. We further explore on CUB-200-birds with a stronger data
augmentation. Watermark signature would fail to be inferred in this case. This
observation implies that if models are trained heavily with similar data
augmentations as watermarking, the signature inference could be confused by
nearby signatures and thus fail to recover the watermarking signatures.

Watermark signature |  Model Acc |  Watermark Loss |  Clean Loss |  Inferred signature
---|---|---|---|---
60 | 52.5 | 0.042 | 0.054 | 64.8 ✓
120 | 52.5 | 0.102 | 0.244 | 108.3 ✓
180 | 53.0 | 0.052 | 0.081 | 178.8 ✓
240 | 53.0 | 0.113 | 0.132 | 254.4 ✓
300 | 51.7 | 0.125 | 0.161 | 291.3 ✓
Table 6: Watermarking for model trained with color jitter augmentation for
ResNet50 on Tiny ImageNet. Loss difference between clean and watermarked
samples are smaller comparing with Table 3.

In Table 7, watermark signatures can be inferred correctly from gradient
search for the aforementioned data augmentations. This shows empirically that
LCT is an effective watermarking approach because it is resilient to common
data augmentations in neural networks’ training.

Augmentation | 60 | 120 | 180 | 240 | 300
---|---|---|---|---|---
Cut Out | 57.5 | 122.1 | 178.6 | 240.3 | 301.8
Lable Smoothing | 59.6 | 116.9 | 181.4 | 239.3 | 299.9
Gaussian Noise | 58.3 | 107.8 | 187.1 | 237.2 | 309.9
Adv Training | 57.5 | 118.8 | 183.5 | 242.9 | 298.6
Differential Privacy | 56.0 | 117.5 | 182.8 | 240.6 | 295.9
Color Jitter | 64.8 | 108.3 | 178.8 | 254.4 | 291.3
Table 7: Inferred signatures for models trained with different data
augmentations for ResNet50 on Tiny ImageNet.

Less Noticeable Watermarking. In the previous sections, we show that LCT-based
watermarking is effective against unauthorized neural learners, but may change
the color property significantly in visualization, as illustrated in Figure 7.
One may argue that such a kind of watermarking could be too visually obvious
to be recognized by unauthorized neural learners. However, given data samples,
by selecting proper watermarking signature, the watermarking could be
difficult to be distinguished from stylish transformation or even unnoticeable
to human being.

Figure 7: Selective clean (top), watermarked (middle), reconstructed (bottom)
samples’ comparison for hue-based watermarking applied on the whole images.

For example, nowadays users would often apply image filters to stylize images
before publishing on social media, where filters are quite natural such as
tuning the color of tree leaves from green to yellow, changing the color of
sky from light blue to dark blue. The parameters of these images’ filters
could be used as signature for watermarking. As stylized filters are widely
used, it would be difficult for neural learners to distinguish whether it is
watermarking or users’ preference.

(a) Clean.

(b) Watermarked (0.1) and Reconstructed (0.12).

(c) Watermarked (0.3) and Reconstructed (0.28).

(d) Watermarked (0.5) and Reconstructed (0.44).

Figure 8: Selective samples’ comparison for color-based watermarking for
partial pixels on blue color channel.

Color-based transformation can be less noticeable when it is only applied on
selective pixels and color channels. In particular, by following [49], we
first generate a random binary string $w$ with a fixed length $T$, and then
generate pseudo-random pixels’ positions as $\rho_{t}=(i_{t},j_{t})$ for each
element $w_{t},(1\leq t\leq T)$. Finally we change blue color channel for
these pixels as:

$B_{\rho_{t}}\leftarrow(2w_{t}-1)\alpha L_{\rho_{t}},$ (8)

where $\alpha$ is the hyper-parameter of watermarking intensity and
$L_{\rho_{t}}$ is luminance of pixel calculated by
$L_{\rho_{t}}=0.299R_{\rho_{t}}+0.587G_{\rho_{t}}+0.114B_{\rho_{t}}$.

Different from [49], we use $\alpha$ as the watermark key and pass the pseudo-
random pixels’ locations and binary string $w$ to the verifier for key
inference. Empirically, this kind of watermarking can be memorized by neural
learners but less noticeable to human. We present visual comparison of samples
in Figure 8 with different value of $\alpha$. In summary, color-based
watermarking can be both effective and unnoticeable with carefully selected
watermarking approach and watermark keys w.r.t users’ images.

In our paper, we mainly explore LCT as an effective way for anti-neuron
watermarking. And we also show recovering watermark pattern fails to work in
PIP while other geometrical watermarking can be applicable. There are
certainly other watermarking functions that can verify unauthorized neural
model training.

##  Appendix 0.F More Discussions on User-specific Watermarking

As we show in previous section, signatures watermarked by user-specific LCTs
would be better memorized among multiple users than those watermarked by a
single LCT. This observation, on one hand, implies that selecting an arbitrary
LCT for watermarking would be a good practice. It also explains why the LCT
watermarking could be resilient to Color jitter. As the color transformations
are well-defined and constant matrices (the matrices are fixed, the adjustment
could be any value), randomly chosen users’ watermarking are highly unlikely
to be the same as Color jitter and would be easier to survive in data
augmentation. On the other hand, it is very unlikely to infer user signatures
without knowing the watermarking function as inferring from arbitrary LCTs
have similar matching rate as random guessing, which makes it difficult for
attackers to find out the users’ signatures without knowing their watermarking
functions in advance.

##  Appendix 0.G Comparison with Related Methods

Comparing to Membership Inference Attacks. As membership inference attacks
(MIAs) determine membership of given data, it is reasonable to consider such
methods in personal data protection. However, there are three restrictions
preventing MIAs from being applied in real PIP scenarios.

(i) The first restriction is MIAs require prior knowledge of training data,
which would be difficult to obtain as a common user. For example, user needs
auxiliary data which has the same data distribution and similar data size as
training data so that the well-trained shadow models could have similar
performance as adversary models [37]. Or users need to know some samples that
are in adversary model’s training data [47].

(ii) If users exploit their own data and train shadow models to perform MIAs,
the success accuracy would still be low. This is because adversary could train
a much better models by using much more data than a single user. In this case,
the training loss of adversary model would be much less than the users shadow
models. As we discussed in the main draft, such performance difference would
lead to poor classification accuracy in membership inference.

On the other hand, as neural learner collects data from many others, there are
chances that users’ data distribution are slightly different (i.e., users
watermarking their data in different ways or applying different filters).
Under this circumstance, MIAs would possibly perform worse as a single
threshold might not work well for heterogeneous users data.

(iii) Last but not the least, even if we assume the verifier obtains such
knowledge, the inference results would be unconvincing for an arbitration. As
MIAs only produce binary outputs (i.e., True of False), the probability of a
correct guess is already $50\%$. Further, MIAs determine membership by
considering data in training must have a small loss. But a small loss of user
data can not guarantee user data were used for training. Such a result could
possibly be caused by learning similar data rather than the user’s.

In summary, MIAs would be less practical than watermarking in personal data
protection.

Early studies on MIAs consider watermarking as a special case. In the study
[47] on membership inference, they discuss the membership advantage [46] is
not necessary if there exists features as a prior knowledge that can be used
to distinguish data, (e.g. unique id for each image). This follows same idea
of watermarking and verification discussed in the paper. However, [47]’s study
does not study what watermarking technique can be used to against neural
modeling training. They assume the all users can substitute their original
data with an identifier via an arbitrary function $G$ [11] and this
substitution would not interfere the embedding identifiers. This might work if
neural networks can perfectly memorize every detail of training data,
including the identifier. However, as shown in our experiments, this is not
true. Different users with the same watermarking function would interfere each
other heavily, as neural models can learn the data augmentation during
training [51]. One contribution of this paper is to show that LCT is an
effective anti-neuron watermarking method against neural model training in
various realistic settings.

To justify the above discussions, we first propose a new evaluation metric for
protection, which quantifies both inference accuracy and signature space, as
discussed in the main manuscript. Then we conduct experiments on two PIP cases
where users train their own shadow models and different users exploit
different watermarking. To perform MIAs, we assume the verifier obtains the
necessary knowledge of training distribution, and compare our watermarking
method with two state-of-art MIAs: MIA-std [47] and MIA-pow [36]. The first
MIA exploits few samples known from model’s training and use the average
training loss as a threshold to determine the membership. If the loss of given
testing sample is smaller than the threshold, the sample would be used for
training. The second method requires extra samples known from the held-out set
and find the best threshold for both training and held-out samples.

For the experiment, we follow similar settings in [21] for these two attacks.
To compare with watermarking fairly, we conduct MIA on average loss of user
data, not data per se, which would increase accuracy of MIAs. In the case when
user exploit their own data to perform MIAs, we assume 10 users and each user
utilize his/her data to train shadow models and use the knowledge of shadow
models to further perform MIAs, respectively; We use MIA-std
citeyeom2018privacy as the baseline method and obtain the decision threshold
using average training loss from user’s shadow models. In the case when users’
data distribution are slight different, we randomly split Tiny ImageNet into
1,000 users for training set and 100 users for validation set, with 100 data
samples for each user. For MIA-std [47], we randomly select 5 users from
training and calculate the average loss as the membership threshold. For MIA-
pow [36], we randomly select 5 users from training and 5 user from validation
to search for the best threshold. Then we test the inference accuracy on 100
user data samples, with 50 randomly chosen from training and 50 randomly
chosen on validation. Note that the testing users would be exclusive from
users used for finding threshold.

From Table 8, we show the matching accuracy for MIAs and our method on these
testing users. It can be observed that the performance of MIAs will first
decrease and then increase as ratio grows. Specifically, when fewer user data
are watermarked, the MIAs’ threshold would be mainly determined by the
unwatermarked users data. Meanwhile, as some user data exploit watermarking,
the loss of watermarked data would be lifted, as illustrated in Figure 9, and
even if the watermarked samples were used in training, the MIAs would
misclassify these samples. The situation would be worst when there are similar
amount of watermarking data and unwatermarking data. Such a result show that
MIAs would be unstable when user data are heterogeneous.

Ratio | 0% | 20% | 40% | 60% | 80% | 100%
---|---|---|---|---|---|---
MIA-std | 77.8 | 74.6 | 67.3 | 72.2 | 69.5 | 79.0
MIA-pow | 92.7 | 84.8 | 77.6 | 80.3 | 76.5 | 94.0
ANW | - | 96.0 | 95.2 | 92.6 | 86.5 | 82.0
Table 8: The matching accuracy between membership inference attacks and anti-
neuron watermarking. Different ratio of users exploit different LCT for their
data. In MIAs, a match would be calculated by binary inference result, but in
ANW, a match would imply the correct signature, which would be more difficult
because there are 12 signature values in this experiment.  Figure 9:
Illustration of why MIAs would decrease performance when partial users exploit
watermarking. As watermarking function lifts loss of user data, the original
MIA threshold calculated by unwatermarked data would misclassify the
membership of watermarked data and thus degrades performance. The experiment
result for memorization in the main draft verifies the above idea.

Comparing to Dataset Tracing. Recent studies [35, 24, 52] on dataset
protection investigate watermarking against neural training and it is tempting
to apply these methods in personal data protection. However, as we discussed
in the main manuscript, dataset tracing [35, 24, 52] requires full knowledge
of training data, so that they can generate “watermarked data” by exploiting
pretrained classifier on the dataset as adversarial training [26]. However,
from users’ perspective, such prior knowledge would not be sufficient because
it is impossible for a common user to know how other data will be collected
and what tasks will be performed. As a result, techniques like dataset tracing
[35, 24, 52] can not be directly applied to the PIP scenario.

As dataset tracing [35] can be applied on partial data, we compare our method
with dataset tracing [35] and verify whether both methods can work well with
very limited watermarking data, which is common in the PIP problem. To make
dataset tracing work, we pre-train a ResNet50 model on Tiny ImageNet and adopt
the same setting as [35] to generate “radioactive data”. As shown in the main
manuscript, with $0.1\%$ data being watermarked, our LCT method is more
effective as we observe that watermarking signature can be memorized well as
expected.

##  Appendix 0.H Memorization Analysis of Watermarking

When Signature is being Memorized during Neural Model Training? One
interesting problem for watermarking is when a signature is memorized by
neural models. From previous studies on MIAs [37, 47, 21, 39], user privacy
information is being leak when model’s overfitting. As a result, it is likely
for models to memorize watermarking signatures when models over-learn the
watermarked data [39]. To explore the answer of this question, we infer
signatures by grid search for different checkpoints of model during training.
In Figure 10, we can observe the inferred signature reaches 0 at early stage
of training and gradually reach 60 with more learning epochs. This result
illustrates that the watermarking signature could be memorized before the end
of training. This empirical results can also explain why data augmentation
(watermarking) can be learned by neural models during training, according to
the study [51] for neural generalization.

Figure 10: The watermarking signature can be memorized before the end of
training. The red point indicates inferred signature that achieves minimal
loss over other signatures.

Watermarking Increase Memorization of User Data. In the main draft, we mention
how to use “memorization value estimate” [9] to study how watermarking improve
memorization of user data. As user data is watermarked, the probability
density of user data would be lifted into low density region and thus being
easier memorized by neural learners. During training process, the watermarking
signature would be better memorized along with user data. According to the
study [9] on memorization, for training algorithm $A$ on a dataset
$S=((x_{1},y_{1}),...,(x_{n},y_{n}))$, the amount of label memorization by
$\mathcal{A}$ on example $(x_{i},y_{i})\in S$ is defined as,

$mem(\mathcal{A},S,i):=\underset{h\leftarrow\mathcal{A}(S)}{\mathbb{P}_{r}}[h(x_{i})=y_{i}]-\underset{h\leftarrow\mathcal{A}(S^{\backslash
i})}{\mathbb{P}_{r}}[h(x_{i})=y_{i}].$ (9)

To estimate above memorization value on sample index at $i$, [9] firstly
selects random subsets for $S$, with some subsets including sample $i$ and
some exclude sample $i$. Then, $K$ models are being trained using these
subsets, grouped into 2, one includes sample $i$ and the other excludes sample
$i$. The memorization value estimate (MAE) is finally calculate by averaging
the difference of $\mathbb{P}_{r}$ between these two groups of models. In our
experiments, we split the training set of Tiny ImageNet into 1,000 users and
choose one split as user data (we calculate MAE on a collection user data
instead of one sample). Then we randomly use $70\%$ of users data to construct
20 subsets. 20 models are being trained correspondingly to calculate the final
MAE. We fix the indices for this experiments and train 20 models with and
without watermarking on user data. From the results we show in the main
manuscript, the MAE of user data increase after watermarking, indicating that
watermarking improves model memorization ability on the given user data.