# Defending against Model Stealing via Verifying Embedded External Features

Yiming Li1,, Linghui Zhu1,2,∗, Xiaojun Jia3, Yong Jiang1,2, Shu-Tao Xia1,2,,
Xiaochun Cao3 The first two authors contributed equally to this
work.Corresponding Author: Shu-Tao Xia

###### Abstract

Obtaining a well-trained model involves expensive data collection and training
procedures, therefore the model is a valuable intellectual property. Recent
studies revealed that adversaries can ‘steal’ deployed models even when they
have no training samples and can not get access to the model parameters or
structures. Currently, there were some defense methods to alleviate this
threat, mostly by increasing the cost of model stealing. In this paper, we
explore the defense from another angle by verifying whether a suspicious model
contains the knowledge of defender-specified _external features_.
Specifically, we embed the external features by tempering a few training
samples with style transfer. We then train a meta-classifier to determine
whether a model is stolen from the victim. This approach is inspired by the
understanding that the stolen models should contain the knowledge of features
learned by the victim model. We examine our method on both CIFAR-10 and
ImageNet datasets. Experimental results demonstrate that our method is
effective in detecting different types of model stealing simultaneously, even
if the stolen model is obtained via a multi-stage stealing process. The codes
for reproducing main results are available at Github (https://github.com/zlh-
thu/StealingVerification).

##  1 Introduction

Deep learning, especially deep neural networks (DNNs), has demonstrated its
great power in many applications (Guo et al. 2020; Stokes et al. 2020; Minaee
et al. 2021). In general, training a well-performed model requires a large
number of training samples and a massive amount of computational resources.
Both data collection and training process are expensive and time-consuming,
which makes the trained model a valuable intellectual property to its owner.

Recently, researchers found that adversaries can ‘steal’ the (deployed)
_victim model_ even when they have no training samples and can not get access
to the model parameters or structures (Tramèr et al. 2016; Orekondy, Schiele,
and Fritz 2019; Chandrasekaran et al. 2020). For example, the adversaries may
use the victim model to label an unlabeled dataset, based on which to train
the _stolen model_. This threat is called _model stealing_. Since the model
stealing can obtain a function-similar copy of the victim model stealthily, it
poses a huge threat to model owners.

Figure 1: The verification stage of dataset inference and our method.
Specifically, dataset inference adopts inherent features contained in benign
private images while our method utilizes external features embedded in
stylized images.

To alleviate the threat of model stealing, there were also some defense
methods, mostly by introducing randomness or perturbation in the victim models
to increase the costs of model stealing (Tramèr et al. 2016; Lee et al. 2019;
Kariyappa and Qureshi 2020). For instance, defenders may perturb the
prediction by rounding or adding noise to the posterior probabilities.
However, these defenses may significantly reduce the performance of legitimate
users and could even be bypassed by following adaptive attacks (Jia et al.
2021; Maini, Yaghini, and Papernot 2021).

In this paper, we explore the defense of model stealing from another
perspective by _verifying whether a suspicious model has defender-specified
behaviors_. If the model has such behaviors, we treat it as stolen from the
victim. This approach is inspired by the understanding that the stolen models
should contain the knowledge of features learned by the victim model and
therefore they have similar behaviors. To the best of our knowledge, there is
only one work, $i.e.$, _dataset inference_ (Maini, Yaghini, and Papernot
2021), focusing on this perspective, where they adopted _inherent features_ of
the training set to verify model ownership. However, we reveal that this
approach is easy to make misjudgments, especially when the training set of
suspicious models have a similar distribution to that of the victim model. The
misjudgment is most probably because different models may learn similar
inherent features once their training sets have certain similarities. Based on
this understanding, we propose to embed defender-specified _external features_
into victim models for ownership verification. These features are different
from those contained in the original training set. Specifically, we embed
external features by tempering the images of some training samples with _style
transfer_. Since we only poison a few samples and do not change their labels,
the embedded features will not hinder the functionality of the victim model.
Besides, we also train a _benign model_ based on the original training set. It
is used only for training a _meta classifier_ to determine whether a
suspicious model is stolen from the victim model. Only the model containing
the knowledge of external features will be deployed.

The main contribution of this work is four-fold: (1) We revisit the defense of
model stealing from the aspect of ownership verification. (2) We reveal the
limitations of existing verification-based methods. Based on these
understandings, we propose a simple yet effective defense approach. (3) We
verify the effectiveness of our method on benchmark datasets under various
types of model stealing simultaneously. (4) Our work could provide a new angle
about how to adopt ‘data poisoning’ for positive purposes.

##  2 Background and Related Work

In this paper, we focus on model stealing and its defenses towards image
classification. Other tasks are out the scope of this paper. We will discuss
them in our future work.

###  2.1 Model Stealing

Model stealing aims to steal the intellectual property from a victim by
obtaining a function-similar copy of the deployed model. In general, existing
methods can be divided into three categories based on adversary’s permission
level, as follows:

Dataset-Accessible Attacks ($\mathcal{A}_{D}$):  In this setting, the
adversary can get access to the training dataset whereas can only query the
model. In this case, the adversary can train a substitute model based on
knowledge distillation (Hinton, Vinyals, and Dean 2014) for model stealing.

Model-Accessible Attacks ($\mathcal{A}_{M}$):  In this setting, the adversary
has complete access to the victim model. This type of attack could happen when
the victim model is open-sourced or via insider access. In this case, the
adversary can obtain a substitute model by using data-free knowledge
distillation based on zero-shot learning (Fang et al. 2019) or simply by fine-
tuning the victim model with local training samples.

Query-Only Attacks ($\mathcal{A}_{Q}$):  In this setting, the adversary can
only query the model. Query-only attacks can also be divided into two
subclasses, including the _label-query attacks_ (Papernot et al. 2017;
Jagielski et al. 2020; Chandrasekaran et al. 2020) and _logit-query attacks_
(Tramèr et al. 2016; Orekondy, Schiele, and Fritz 2019), based on model’s
feedback. In general, label-query attacks adopted the victim model to annotate
some substitute (unlabeled) samples, based on which to train their substitute
model. In the logit-query attacks, the adversary usually obtains the
substitute model by minimizing the distance between its predicted logits and
those generated by the victim model.

###  2.2 Defenses against Model Stealing

Non-verification based Defenses.  Currently, most of the existing methods
alleviated the stealing threat by increasing the cost of model stealing
through perturbing model results. For instance, defenders could round the
probabilities (Tramèr et al. 2016), added noise to the prediction that results
in a high loss (Lee et al. 2019), only returning the most confident label
(Orekondy, Schiele, and Fritz 2019). However, these defenses may significantly
reduce the performance of legitimate users and could even be bypassed by
adaptive attacks (Jia et al. 2021; Maini, Yaghini, and Papernot 2021). Other
works (Kesarwani et al. 2018; Juuti et al. 2019; Yan et al. 2021) detected
model stealing by identifying malicious queries. However, these methods relied
on some assumptions of malicious query patterns, which may not be adopted by
the adversaries in practice.

Dataset Inference.  To the best of our knowledge, this is the first and
currently the only verification-based defense against model stealing. Its key
idea is to identify whether a suspicious model contains the knowledge of the
inherent features that the victim model $V$ learned from the private training
set. Specifically, let we consider a $K$-classification problem. For each
sample $(\bm{x},y)$, dataset inference first generated its minimum distance
$\bm{\delta}_{t}$ to each class $t$ by

$\min_{\bm{\delta}_{t}}d(\bm{x},\bm{x}+\bm{\delta}_{t}),s.t.,V(\bm{x}+\bm{\delta}_{t})=t,$
(1)

where $d(\cdot)$ is a distance metric ($e.g.$, $\ell^{\infty}$ norm). The
distance to each class $\bm{\delta}=(\bm{\delta}_{1},\cdots,\bm{\delta}_{K})$
is the feature embedding of sample $(\bm{x},y)$ $w.r.t.$ the victim model $V$.
After that, the defender will randomly select some samples inside (labeled as
‘+1’) or out-side (labeled as ‘-1’) their private dataset and use the feature
embedding $\bm{\delta}$ to train a binary meta-classifier $C$, where
$C(\bm{\delta})\in[0,1]$ indicates the probability that the sample
$(\bm{x},y)$ is from the private set. To determine whether a suspicious model
is stolen from the victim, the defender creates equal-sized sample vectors
from private and public samples and conduct the _hypothesis test_ based on the
trained $C$. If the confidence scores of private samples are significantly
greater than those of public samples, the suspicious model is treated as
stolen from the victim. However, as shown in following experiments, this
method is easy to make misjudgments, especially when the training set of
suspicious models have a similar distribution to that of the victim model.
This limitation hinders its utility in practice.

Model Watermarking.  The main purpose of model watermarking is detecting theft
($i.e.$, directly copy the model) instead of preventing model stealing.
However, we notice that the dataset inference enjoys certain similarities to
the _misclassification-based model watermarking_ (Adi et al. 2018; Li et al.
2020b; Jia et al. 2021), especially the backdoor-based ones (Adi et al. 2018;
Zhang et al. 2018; Li et al. 2020b). As such, these approaches could be
potential defenses against model stealing. Specifically, these methods
performed ownership verification by making the protected model misclassifying
defender-specified samples. For example, defenders may first adopt _backdoor
attacks_ (Gu et al. 2019; Li et al. 2020a; Nguyen and Tran 2020; Li et al.
2021b; Bagdasaryan and Shmatikov 2021; Li et al. 2021a) to watermark the model
during the training process and then conduct the ownership verification. In
general, a backdoor attack can be characterized by three components, including
the trigger pattern $\bm{t}$, target class $y_{t}$, and adversary-predefined
poisoned image generator $G(\cdot)$. Given the benign training set
$\mathcal{D}=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{N}$, the backdoor adversary will
randomly select $\gamma\%$ samples ($i.e.$, $\mathcal{D}_{s}$) from
$\mathcal{D}$ to generate their poisoned version
$\mathcal{D}_{p}=\\{(\bm{x}^{\prime},y_{t})|\bm{x}^{\prime}=G(\bm{x};\bm{t}),(\bm{x},y)\in\mathcal{D}_{s}\\}$.
Different backdoor attacks may assign different generator $G(\cdot)$. For
example,
$G(\bm{x};\bm{t})=(\bm{1}-\bm{\lambda})\otimes\bm{x}+\bm{\lambda}\otimes\bm{t}$
where $\bm{\lambda}\in\\{0,1\\}^{C\times W\times H}$ and $\otimes$ indicates
the element-wise product in the BadNets (Gu et al. 2019). After
$\mathcal{D}_{p}$ was generated, $\mathcal{D}_{p}$ and remaining benign
samples $\mathcal{D}_{b}\triangleq\mathcal{D}\backslash\mathcal{D}_{s}$ will
be used to train the model $f_{\theta}$ via

$\min_{\bm{\theta}}\sum_{(\bm{x},y)\in\mathcal{D}_{p}\cup\mathcal{D}_{b}}\mathcal{L}(f_{\bm{\theta}}(\bm{x}),y).$
(2)

In the verification stage, the defender will examine suspicious models in
predicting $y_{t}$. If the confidence scores of poisoned samples are
significantly greater than those of benign samples, the suspicious model is
treated as watermarked and therefore it is stolen from the victim. However, as
shown in the following experiments, these methods have far less effective in
defending against model stealing.

##  3 Revisiting Verification-based Defenses

###  3.1 The Limitation of Dataset Inference

As illustrated in Section 2.2, dataset inference relied on a latent assumption
that a model will not learn the features contained in the private dataset if
it is not stolen from the victim. However, since different models may learn
similar features even they are trained on different datasets, this assumption
does not hold and therefore the method may misjudge. In this section, we
verify this limitation.

Settings.  In this section, we conduct the experiments on CIFAR-10
(Krizhevsky, Hinton et al. 2009) dataset with VGG (Simonyan and Zisserman
2015) and ResNet (He et al. 2016). Specifically, we randomly separate the
original training set $\mathcal{D}$ into two disjoint subsets
$\mathcal{D}_{l}$ and $\mathcal{D}_{r}$. We train the VGG on $\mathcal{D}_{l}$
(dubbed VGG-$\mathcal{D}_{l}$) and the ResNet on $\mathcal{D}_{r}$ (dubbed
ResNet-$\mathcal{D}_{r}$), respectively. We also train the VGG on a noisy
dataset
$\mathcal{D}_{l}^{\prime}\triangleq\\{(\bm{x}^{\prime},y)|\bm{x}^{\prime}=\bm{x}+\mathcal{N}(0,16),(\bm{x},y)\in\mathcal{D}_{l}\\}$
(dubbed VGG-$\mathcal{D}_{l}^{\prime}$) for reference. In the verification
process, we verify whether the VGG-$\mathcal{D}_{l}$ and
VGG-$\mathcal{D}_{l}^{\prime}$ is stolen from ResNet-$\mathcal{D}_{r}$ and
whether the ResNet-$\mathcal{D}_{r}$ is stolen from VGG-$\mathcal{D}_{l}$
based on the settings proposed in dataset inference (Maini, Yaghini, and
Papernot 2021). Besides, we also adopt the p-value as the evaluation metric.
The p-value is calculated based on the approach described in Section 2.2. Note
that _the smaller the p-value, the more confident that dataset inference
believes the model stealing happened_. More detailed settings are in Appendix.

| ResNet-$\mathcal{D}_{r}$ | VGG-$\mathcal{D}_{l}$ | VGG-$\mathcal{D}_{l}^{\prime}$
---|---|---|---
Accuracy | 88.0% | 87.7% | 85.0%
p-value | $\bm{10^{-7}}$ | $\bm{10^{-5}}$ | $\bm{10^{-4}}$

Table 1: The accuracy of victim models and p-value of verification processes.
Dataset inference misjudged in all cases.

Model Type $\rightarrow$ | Benign | Watermarked | Stolen
---|---|---|---
BA | 91.99 | 90.03 | 70.17
ASR | 0.01 | 98.02 | $\bm{3.84}$

Table 2: The performance (%) of different models.

Results.  As shown in Table 1, all models achieve decent performance even when
the training samples are limited. However, the p-value is significantly
smaller than 0.01 in all cases. In other words, the dataset inference believes
that these models are stolen from the victim in a high confidence. However, in
each case, since the suspicious and the victim model are trained on completely
different training samples and with different model structures, the suspicious
model should not be considered as stolen from the victim. These results reveal
that _the dataset inference could make misjudgments and therefore its results
are questionable_. In particular, the p-value of VGG-$\mathcal{D}_{l}$ is
smaller than that of the VGG-$\mathcal{D}_{l}^{\prime}$. This is probably
because the latent distribution of $\mathcal{D}_{l}^{\prime}$ is more
different from that of $\mathcal{D}_{r}$ (compared with that of
$\mathcal{D}_{l}$) and therefore models learn more different features.

###  3.2 The Limitation of Model Watermarking

Intuitively, the inference process of backdoor attacks is similar to unlocking
a door with the corresponding key. As such, the success of backdoor-based
model watermarking relied on an assumption that the trigger pattern matches
hidden backdoors contained in the stolen model. This assumption holds in its
originally discussed scenarios where the stolen model is the same as the
victim model. However, it may not hold in the model stealing, since the
backdoors contained in the stolen models may be changed or even removed during
the stealing process. Accordingly, backdoor-based model watermarking may fail
in defending against model stealing. In this section, we verify this
limitation.

Settings.  In this part, we adopt the most representative and effective
backdoor attack, the BadNets (Gu et al. 2019), as an example for the
discussion. The watermarked model will then be stolen by the data-free
distillation-based model stealing (Fang et al. 2019). We adopt the _benign
accuracy (BA)_ and _attack success rate (ASR)_ (Li et al. 2020a) to evaluate
the performance of the stolen model. The larger the ASR, the more likely the
stealing will be detected. More detailed settings can be found in Appendix.

Figure 2: The main pipeline of our method. In the first stage, defenders will
modify some images via style transfer for embedding external features. In the
second stage, defenders will train a meta-classifier to determine whether a
suspicious model is stolen from the victim based on gradients. In the last
stage, defenders will conduct ownership verification via hypothesis test.

Results.  As shown in Table 2, the ASR of the stolen model is only 3.84%,
which is significantly lower than that of the watermarked model. In other
words, _the defender-specified trigger no longer matches the hidden backdoors
contained in the stolen model_. As such, backdoor-based model watermarking
will fail to detect model stealing.

Figure 3: The adopted trigger pattern and synthesized ones obtained from the
watermarked and stolen model. The trigger areas are indicated in the blue box.
(a) ground-truth trigger pattern; (b) pattern obtained from the watermarked
model; (c) pattern obtained from the stolen model.

To further understand the reason of this failure, we synthesize the potential
trigger pattern of each model based on the targeted universal adversarial
attack (Moosavi-Dezfooli et al. 2017). As shown in Figure 3, the pattern
recovered from the watermarked model is similar to the ground-truth one,
whereas the one recovered from the stolen model is completely different from
the ground-truth pattern. These results explain why backdoor-based model
watermarking has minor effects in defending against model stealing.

Moreover, _backdoor-based model watermarking will introduce new security
threats_ , since it builds a stealthy latent connection between trigger
pattern and target label. The adversary may use it to maliciously manipulate
the predictions of deployed models. This problem will also hinder the utility
of backdoor-based model watermarking in practice.

##  4 The Proposed Method

Based on the understandings in Section 3, in this paper, we propose to embed
_external features_ instead of inherent features for ownership verification.
Specifically, as shown in Figure 2, our method consists of three main stages,
including (1) embedding external features, (2) training an ownership meta-
classifier, and (3) conducting ownership verification. Their technical details
are in the following subsections.

###  4.1 Threat Model

Following the setting of existing works (Zhang et al. 2020; Wang and
Kerschbaum 2021; Liu, Weng, and Zhu 2021), we conduct the defense in a _white-
box_ setting, where the defender has complete access to the suspicious model.
However, the defender has no information about the stealing process. The goal
of defenders is to accurately identify whether the suspicious model is stolen
from a victim model, based on behaviors of the suspicious and victim model.

One may argue that only black-box defenses are practical since the adversary
may refuse to provide the suspicious model. However, white-box defenses are
also practical. In our understanding, the real-world adoption of verification-
based defenses (in a legal system) requires an official institute for the
arbitration. Specifically, all commercial models should be registered here,
through the unique identification ($e.g.$, MD5 code) of their model’s weights
file. When this official institute is established, its staff should take
responsibility for the verification process. For example, the staff can
require the company to provide the model file with the same registered
identification and then use our method (under the white-box setting) for the
ownership verification.

###  4.2 Embedding External Features

In this section, we describe how to embed external features into the victim
model. Before we reach technical details, we first present the definition of
inherent and external features.

######  Definition 1.

A feature $f$ is called the inherent feature (of dataset $\mathcal{D}$) if and
only if $\
\forall(\bm{x},y)\in\mathcal{X}\times\mathcal{Y},(\bm{x},y)\in\mathcal{D}\Rightarrow(\bm{x},y)\
\text{contains feature}f.$ Similarly, f is called the external feature (of
dataset $\mathcal{D}$) if and only if $\
\forall(\bm{x},y)\in\mathcal{X}\times\mathcal{Y},(\bm{x},y)\ \text{contains
feature}\ f\Rightarrow(\bm{x},y)\notin\mathcal{D}.$

Although external features are well defined, how to construct them is still
difficult since the learning dynamic of DNNs remains unclear and the concept
of features itself is complicated. However, at least we know that the _image
style_ can serve as a feature for the learning of DNNs in image-related tasks,
based on some recent studies (Geirhos et al. 2019; Duan et al. 2020; Cheng et
al. 2021). As such, we can use _style transfer_ (Johnson, Alahi, and Fei-Fei
2016; Huang and Belongie 2017; Chen et al. 2020) for embedding external
features. People may also adopt other methods for the embedding. It will be
discussed in our future work.

Figure 4: Images involved in different defenses. (a) benign image; (b)
poisoned image in BadNets; (c) poisoned image in Gradient Matching; (d)
poisoned image in Entangled Watermarks; (e) style image; (f) transformed
image.

Specifically, let $\mathcal{D}=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{N}$ denotes the
benign training set, $\bm{x}_{s}$ is a defender-specified _style image_ , and
$T:\mathcal{X}\times\mathcal{X}\rightarrow\mathcal{X}$ is a (trained) style
transformer. In this stage, the defender first randomly selects $\gamma\%$
(dubbed _transformation rate_) samples ($i.e.$, $\mathcal{D}_{s}$) from
$\mathcal{D}$ to generate their transformed version
$\mathcal{D}_{t}=\\{(\bm{x}^{\prime},y)|\bm{x}^{\prime}=T(\bm{x},\bm{x}_{s}),(\bm{x},y)\in\mathcal{D}_{s}\\}$.
The external features (contained in $\bm{x}_{s}$) will be learned by the
victim model $V_{\theta}$ during the training process via

$\min_{\bm{\theta}}\sum_{(\bm{x},y)\in\mathcal{D}_{b}\cup\mathcal{D}_{t}}\mathcal{L}(V_{\bm{\theta}}(\bm{x}),y),$
(3)

where $\mathcal{D}_{b}\triangleq\mathcal{D}\backslash\mathcal{D}_{s}$ and
$\mathcal{L}(\cdot)$ is the loss function.

In this stage, how to select the style image is an important question.
Intuitively, it should be significantly different from those contained in the
original training set. In practice, defenders can simply adopt oil or sketch
paintings as the style image since most of the images that need to be
protected are natural images. We will further discuss it in Section 5.3.

In particular, since we only poison a few samples and do not change their
labels, the embedding of external features will not hinder the functionality
of victim models or introduce new security risks ($e.g.$, hidden backdoors).

###  4.3 Training Ownership Meta-Classifier

Since there is no explicit expression of the embedded external features and
those features also have minor influences on the prediction, we need to train
an additional binary meta-classifier to determine whether the suspicious model
contains the knowledge of external features.

In this paper, we adopt the gradients of model weights as the input to train
the meta-classifier
$C_{\bm{w}}:\mathbb{R}^{|\bm{\theta}|}\rightarrow\\{-1,+1\\}$. Specifically,
we assume that the victim model $V$ and the suspicious model $S$ have the same
model structure. This assumption can be easily satisfied since the defender
can retain a copy of the suspicious model on the training set of the deployed
model as the victim model. Once the suspicious model is obtained, the defender
will train its benign version ($i.e.$, the $B$) on the original training set
$\mathcal{D}$. After that, we can obtain the training set $\mathcal{D}_{c}$ of
meta-classifier $C$ via

$\displaystyle\mathcal{D}_{c}=$
$\displaystyle\left\\{\left(g_{V}(\bm{x}^{\prime}),+1\right)|(\bm{x}^{\prime},y)\in\mathcal{D}_{t}\right\\}\cup$
(4)
$\displaystyle\left\\{\left(g_{B}(\bm{x}^{\prime}),-1\right)|(\bm{x}^{\prime},y)\in\mathcal{D}_{t}\right\\},$

where $\text{sgn}(\cdot)$ is sign function (Sachs 2012),
$g_{V}(\bm{x}^{\prime})=\text{sgn}(\nabla_{\bm{\theta}}\mathcal{L}(V(\bm{x}^{\prime}),y))$,
and
$g_{B}(\bm{x}^{\prime})=\text{sgn}(\nabla_{\bm{\theta}}\mathcal{L}(B(\bm{x}^{\prime}),y))$.

At the end, the meta-classifier $C_{\bm{w}}$ is trained by

$\min_{\bm{w}}\sum_{(\bm{s},t)\in\mathcal{D}_{c}}\mathcal{L}(C_{\bm{w}}(\bm{s}),t).$
(5)

In particular, we adopt its sign vector instead of the gradient itself to
highlight the influence of its direction. We verify its effectiveness in
Appendix.

###  4.4 Ownership Verification with Hypothesis Test

When the meta-classifier is trained, given a transformed image
$\bm{x}^{\prime}$ and its label $y$, the defender can examine the suspicious
model simply by the result of $C(g_{S}(\bm{x}^{\prime}))$, where
$g_{S}(\bm{x}^{\prime})=\text{sgn}(\nabla_{\bm{\theta}}\mathcal{L}(S(\bm{x}^{\prime}),y))$.
If $C(g_{S}(\bm{x}^{\prime}))=1$, the suspicious model is considered as stolen
from the victim. However, it may be sharply affected by the randomness of
selecting $\bm{x}^{\prime}$. In this paper, we design a hypothesis test based
method to increase the verification confidence, as follows:

######  Definition 2.

Let $\bm{X}^{\prime}$ denotes the variable of transformed images, while
$\mu_{S}$ and $\mu_{B}$ indicates the posterior probability of the event
$C(g_{S}(\bm{X}^{\prime}))=1$ and $C(g_{B}(\bm{X}^{\prime}))=1$, respectively.
Given a null hypothesis $H_{0}:\mu_{S}\leq\mu_{B}\ (H_{1}:\mu_{S}>\mu_{B})$,
we claim that the suspicious model $S$ is stolen from the victim if and only
if the $H_{0}$ is rejected.

In practice, we randomly sample $m$ different transformed images from
$\mathcal{D}_{t}$ to conduct the pair-wise T-test (Hogg, McKean, and Craig
2005) and calculate its p-value. When the p-value is smaller than the
significance level $\alpha$, $H_{0}$ is rejected. Besides, we also calculate
the _confidence score_ $\Delta\mu=\mu_{S}-\mu_{B}$ to represent the
verification confidence. The larger the $\Delta\mu$, the more confident the
verification.

##  5 Experiments

###  5.1 Settings

Dataset and Model Selection.  We evaluate our defense on CIFAR-10 (Krizhevsky,
Hinton et al. 2009) and (a subset of) ImageNet (Deng et al. 2009) dataset.
Following the settings of (Maini, Yaghini, and Papernot 2021), we use the
WideResNet (Zagoruyko and Komodakis 2016) and ResNet (He et al. 2016) as the
victim model on CIFAR-10 and ImageNet, respectively. More detailed settings
are in Appendix.

Model Stealing | BadNets |  | Gradient Matching |  | Entangled Watermarks |  | Dataset Inference |  | Ours
---|---|---|---|---|---|---|---|---|---
$\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value
Victim | Source | 0.91 | $10^{-12}$ |  | 0.88 | $10^{-12}$ |  | 0.99 | $\mathbf{10^{-35}}$ |  | - | $10^{-4}$ |  | 0.97 | $10^{-7}$
$\mathcal{A}_{D}$ | Distillation | $-10^{-3}$ | 0.32 |  | $10^{-7}$ | 0.20 |  | 0.01 | 0.33 |  | - | $10^{-4}$ |  | 0.53 | $\mathbf{10^{-7}}$
$\mathcal{A}_{M}$ | Zero-shot | $10^{-25}$ | 0.22 |  | $10^{-24}$ | 0.22 |  | $10^{-3}$ | $10^{-3}$ |  | - | $10^{-2}$ |  | 0.52 | $\mathbf{10^{-5}}$
Fine-tuning | $10^{-23}$ | 0.28 |  | $10^{-27}$ | 0.28 |  | 0.35 | 0.01 |  | - | $10^{-5}$ |  | 0.50 | $\mathbf{10^{-6}}$
$\mathcal{A}_{Q}$ | Label-query | $10^{-27}$ | 0.20 |  | $10^{-30}$ | 0.34 |  | $10^{-5}$ | 0.62 |  | - | $10^{-3}$ |  | 0.52 | $\mathbf{10^{-4}}$
Logit-query | $10^{-27}$ | 0.23 |  | $10^{-23}$ | 0.33 |  | $10^{-6}$ | 0.64 |  | - | $10^{-3}$ |  | 0.54 | $\mathbf{10^{-4}}$
Benign | Independent | $10^{-20}$ | 0.33 |  | $10^{-12}$ | 0.99 |  | $10^{-22}$ | 0.68 |  | - | 1.00 |  | 0.00 | 1.00

Table 3: Results on CIFAR-10 dataset.

Model Stealing | BadNets |  | Gradient Matching |  | Entangled Watermarks |  | Dataset Inference |  | Ours
---|---|---|---|---|---|---|---|---|---
$\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value
Victim | Source | 0.87 | $10^{-10}$ |  | 0.77 | $10^{-10}$ |  | 0.99 | $\mathbf{10^{-25}}$ |  | - | $10^{-6}$ |  | 0.90 | $10^{-5}$
$\mathcal{A}_{D}$ | Distillation | $10^{-4}$ | 0.43 |  | $10^{-12}$ | 0.43 |  | $10^{-6}$ | 0.19 |  | - | $10^{-3}$ |  | 0.61 | $\mathbf{10^{-5}}$
$\mathcal{A}_{M}$ | Zero-shot | $10^{-12}$ | 0.33 |  | $10^{-18}$ | 0.43 |  | $10^{-3}$ | 0.46 |  | - | $10^{-3}$ |  | 0.53 | $\mathbf{10^{-4}}$
Fine-tuning | $10^{-20}$ | 0.20 |  | $10^{-12}$ | 0.47 |  | 0.46 | 0.01 |  | - | $10^{-4}$ |  | 0.60 | $\mathbf{10^{-5}}$
$\mathcal{A}_{Q}$ | Label-query | $10^{-23}$ | 0.29 |  | $10^{-22}$ | 0.50 |  | $10^{-7}$ | 0.45 |  | - | $\bm{10^{-3}}$ |  | 0.55 | $\mathbf{10^{-3}}$
Logit-query | $10^{-23}$ | 0.38 |  | $10^{-12}$ | 0.22 |  | $10^{-6}$ | 0.36 |  | - | $10^{-3}$ |  | 0.55 | $\mathbf{10^{-4}}$
Benign | Independent | $10^{-24}$ | 0.38 |  | $10^{-23}$ | 0.78 |  | $\mathbf{10^{-30}}$ | 0.55 |  | - | 0.98 |  | $10^{-5}$ | 0.99

Table 4: Results on ImageNet dataset.

Settings for Model Stealing.  Following the settings in (Maini, Yaghini, and
Papernot 2021), we conduct model stealing methods illustrated in Section 2.1
to evaluate the effectiveness of defenses. Besides, we also provide the
results of directly copying the victim model (dubbed ‘Source’) and examining a
suspicious model which is not stolen from the victim (dubbed ‘Independent’)
for reference. More detailed settings can be found in Appendix.

Defense Setup.  We compare our defense with dataset inference (Maini, Yaghini,
and Papernot 2021) and model watermarking (Adi et al. 2018) with BadNets (Gu
et al. 2019), gradient matching (Geiping et al. 2021), and entangled
watermarks (Jia et al. 2021). We poison 10% training samples for all defenses.
Besides, we adopt a white-square in the lower right corner as the trigger
pattern for BadNets and adopt a oil paint as the style image for our defense.
Other settings are the same as those used in their original paper. An example
of images ($e.g.$, poisoned images and the style image) involved in different
defenses is shown in Figure 4.

Evaluation Metric. We use the confidence score $\Delta\mu$ and p-value for the
evaluation metric. Following the settings adopted in (Maini, Yaghini, and
Papernot 2021), both $\Delta\mu$ and p-value are calculated based on the
hypothesis test with 10 sampled images. In particular, except for the
independent sources (which should not be regarded as stolen), _the smaller the
p-value and the larger the $\Delta\mu$, the better the defense_. Among all
defenses, the best result is indicated in boldface.

###  5.2 Main Results

As shown in Table 3-4, our defense reaches the best performance in almost all
cases. For example, the p-value of our method is three orders of magnitude
smaller than that of the dataset inference and six orders of magnitude smaller
than that of the model watermarking in defending against the distillation-
based model stealing on CIFAR-10 dataset. The only exceptions appear when
there is no model stealing. In these cases, entangled watermarks based model
watermarking has some advantages. Nevertheless, our method can still easily
make correct predictions in these cases. In particular, our defense method has
minor adverse effects on the performance of victim models. For example, the
accuracy of the model trained on benign CIFAR-10 and its transformed version
is 91.99% and 91.79%, respectively. This is mainly because we do not change
the label of transformed images and therefore the transformation can be
treated as data augmentation, which is mostly harmless.

Figure 5: Effects of the transformation rate (%) and the number of sampled
images.

Model Stealing | Pattern (a) |  | Pattern (b) |  | Pattern (c) |  | Pattern (d)
---|---|---|---|---|---|---|---
$\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value |  | $\Delta\mu$ | p-value
Victim | Source | 0.98 | $10^{-7}$ |  | 0.97 | $10^{-7}$ |  | 0.98 | $10^{-10}$ |  | 0.98 | $10^{-12}$
$\mathcal{A}_{D}$ | Distillation | 0.68 | $10^{-7}$ |  | 0.53 | $10^{-7}$ |  | 0.72 | $10^{-8}$ |  | 0.63 | $10^{-7}$
$\mathcal{A}_{M}$ | Zero-shot | 0.61 | $10^{-5}$ |  | 0.52 | $10^{-5}$ |  | 0.74 | $10^{-8}$ |  | 0.67 | $10^{-7}$
Fine-tuning | 0.46 | $10^{-5}$ |  | 0.50 | $10^{-6}$ |  | 0.21 | $10^{-7}$ |  | 0.50 | $10^{-9}$
$\mathcal{A}_{Q}$ | Label-query | 0.64 | $10^{-5}$ |  | 0.52 | $10^{-4}$ |  | 0.68 | $10^{-8}$ |  | 0.68 | $10^{-7}$
Logit-query | 0.65 | $10^{-4}$ |  | 0.54 | $10^{-4}$ |  | 0.62 | $10^{-6}$ |  | 0.73 | $10^{-7}$
Benign | Independent | 0.00 | 1.00 |  | 0.00 | 1.00 |  | 0.00 | 1.00 |  | $10^{-9}$ | 0.99

Table 5: The effectiveness of our defense with different style images on
CIFAR-10 dataset.

###  5.3 Discussion

In this section, we discuss the effects of hyper-parameters and components
involved in our method. Unless otherwise specified, all settings are the same
as those in Section 5.2.

Effects of Transformation Rate.  The larger the transformation rate $\gamma$,
the more training samples are transformed during the training process of the
victim model. As we expected, the p-value decrease with the increase of
$\gamma$ in defending all stealing methods (as shown in Figure 5). Note that
the increase of $\gamma$ may also lead to the accuracy decrease of victim
models. Defenders should specify this hyper-parameter based on their specific
requirements in practice.

Effects of the Number of Sampled Images.  Recall that our method needs to
specify the number of sampled (transformed) images ($i.e.$, the $m$) adopted
in the hypothesis-based ownership verification. In general, the larger the
$m$, the less the adverse effects of the randomness involved in this process
and therefore the more confident the verification. This is probably the reason
why the p-value also decreases with the increase of $m$, as shown in Figure 5.

| Style Transfer |  | Meta-classifier
---|---|---|---
|  | Patch-based
---
Variant
Ours |  | BadNets |  | BadNets +
---
Meta-classifier
Distillation | 0.17 | $\mathbf{10^{-7}}$ |  | 0.32 | $\mathbf{10^{-3}}$
Zero-shot | 0.01 | $\mathbf{10^{-5}}$ |  | 0.22 | $\mathbf{10^{-61}}$
Fine-tuning | $10^{-3}$ | $\mathbf{10^{-6}}$ |  | 0.28 | $\mathbf{10^{-5}}$
Label-query | $10^{-3}$ | $\mathbf{10^{-4}}$ |  | 0.20 | $\mathbf{10^{-50}}$
Logit-query | $10^{-3}$ | $\mathbf{10^{-4}}$ |  | 0.23 | $\mathbf{10^{-3}}$

Table 6: The effectiveness (p-value) of style transfer and meta-classifier on
CIFAR-10 dataset.

Figure 6: Style images adopted for the evaluation.

Effects of Style Image.  In this part, we examine whether the proposed defense
is still effective if we adopt other style images (as shown in Figure 6). As
shown in Table 5, the p-value is significantly smaller than 0.01 in all cases.
In other words, our method remains effective in defending against different
stealing methods when different style images are used, although there will be
some fluctuations in the results. We will further explore how to optimize the
selection of style images in our future work.

Effectiveness of Style Transfer.  To verify that the style watermark transfers
better during the stealing process, we compare our method with its variant
which uses the white-square patch (adopted in BadNets) to generate transformed
images. As shown in Table 6, our method is significantly better than its
patch-based variant. It is probably because DNNs are easier to learn the
texture information (Geirhos et al. 2019) and the style watermark is bigger
than the patch one. This phenomenon partly explains why our method works well.

Effectiveness of Meta-Classifier.  To verify that the meta-classifier is also
useful, we compare the BadNets-based model watermarking with its extension
which also uses the meta-classifier (adopted in our method) for ownership
verification. In this case, the victim model is the backdoored one and the
transformed image is the one containing backdoor triggers. As shown in Table
6, adopting meta-classier significantly decrease the p-value in all cases,
which verifies the effectiveness of the meta-classifier. These results also
partly explains the effectiveness of our method.

Attack Stage $\rightarrow$ | Source | First Stage | Second Stage
---|---|---|---
Attack Method $\rightarrow$ | None | Zero-shot | Zero-shot
Model Structure $\rightarrow$ | WRN-28-10 | WRN-28-10 | WRN-28-10
p-value $\rightarrow$ | $10^{-7}$ | $10^{-5}$ | $10^{-4}$
Attack Method $\rightarrow$ | None | Logit-query | Zero-shot
Model Structure $\rightarrow$ | WRN-28-10 | WRN-16-1 | VGG-19
p-value $\rightarrow$ | $10^{-7}$ | $10^{-4}$ | 0.01

Table 7: Results in defending against multi-stage stealing.

###  5.4 Defending against Multi-Stage Model Stealing

In previous experiments, the stolen model is obtained by a single stealing
attack. In this section, we explore whether our method is still effective if
there are multiple stealing stages.

Settings.  We discuss two types of multi-stage stealing on the CIFAR-10
dataset, including (1) stealing with the same attack and model structure and
(2) stealing with different attacks and model structures. In general, the
first one is the easiest multi-stage attack while the second one is the
hardest. Other settings are the same as those used in Section 5.2.

Results.  As shown in Table 7, the p-value $\leq 0.01$ in all cases, $i.e.$,
our method can successfully identify the existence of model stealing, even
after multiple stealing stages. As we expected, the p-value in defending the
second multi-stage attack is significantly larger than that of the first one
indicating that the second task is harder. We will discuss how to better
defend the second type of attack in our future work.

##  6 Conclusion

In this paper, we formulated the defense of model stealing as verifying
whether a suspicious model contains the knowledge of defender-specified
external features. Specifically, we embedded external features by modifying a
few training samples with style transfer. This approach was inspired by the
understanding that the stolen models should contain the knowledge of features
learned by the victim model. We evaluated our defense on both CIFAR-10 and
ImageNet datasets, which verified that our method can defend against various
types of model stealing simultaneously while preserving high accuracy in
predicting benign samples.

## Acknowledgments

This work is supported in part by the National Key R&D Program of China under
Grant 2019YFB1406500, the National Natural Science Foundation of China
(62171248, U1736219, U1936210), the Guangdong Province Key Area R&D Program
under Grant 2018B010113001, the R&D Program of Shenzhen
(JCYJ20180508152204044), and the PCNL KEY project (PCL2021A07).

## References

  * Adi et al. (2018) Adi, Y.; Baum, C.; Cisse, M.; Pinkas, B.; and Keshet, J. 2018.  Turning your weakness into a strength: Watermarking deep neural networks by backdooring.  In _USENIX Security_.
  * Bagdasaryan and Shmatikov (2021) Bagdasaryan, E.; and Shmatikov, V. 2021.  Blind Backdoors in Deep Learning Models.  In _USENIX Security_.
  * Birhane and Prabhu (2021) Birhane, A.; and Prabhu, V. U. 2021.  Large image datasets: A pyrrhic win for computer vision?  In _WACV_.
  * Chandrasekaran et al. (2020) Chandrasekaran, V.; Chaudhuri, K.; Giacomelli, I.; Jha, S.; and Yan, S. 2020.  Exploring connections between active learning and model extraction.  In _USENIX Security_.
  * Chen et al. (2020) Chen, X.; Zhang, Y.; Wang, Y.; Shu, H.; Xu, C.; and Xu, C. 2020.  Optical flow distillation: Towards efficient and stable video style transfer.  In _ECCV_.
  * Cheng et al. (2021) Cheng, S.; Liu, Y.; Ma, S.; and Zhang, X. 2021.  Deep Feature Space Trojan Attack of Neural Networks by Controlled Detoxification.  In _AAAI_.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _CVPR_.
  * Duan et al. (2020) Duan, R.; Ma, X.; Wang, Y.; Bailey, J.; Qin, A. K.; and Yang, Y. 2020.  Adversarial camouflage: Hiding physical-world attacks with natural styles.  In _CVPR_.
  * Fang et al. (2019) Fang, G.; Song, J.; Shen, C.; Wang, X.; Chen, D.; and Song, M. 2019.  Data-free adversarial distillation.  _arXiv preprint arXiv:1912.11006_.
  * Ganju et al. (2018) Ganju, K.; Wang, Q.; Yang, W.; Gunter, C. A.; and Borisov, N. 2018.  Property inference attacks on fully connected neural networks using permutation invariant representations.  In _CCS_.
  * Geiping et al. (2021) Geiping, J.; Fowl, L.; Huang, W. R.; Czaja, W.; Taylor, G.; Moeller, M.; and Goldstein, T. 2021.  Witches’ brew: Industrial scale data poisoning via gradient matching.  In _ICLR_.
  * Geirhos et al. (2019) Geirhos, R.; Rubisch, P.; Michaelis, C.; Bethge, M.; Wichmann, F. A.; and Brendel, W. 2019.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  In _ICLR_.
  * Gu et al. (2019) Gu, T.; Liu, K.; Dolan-Gavitt, B.; and Garg, S. 2019.  Badnets: Evaluating backdooring attacks on deep neural networks.  _IEEE Access_ , 7: 47230–47244.
  * Guo et al. (2020) Guo, Y.; Wang, H.; Hu, Q.; Liu, H.; Liu, L.; and Bennamoun, M. 2020.  Deep learning for 3d point clouds: A survey.  _IEEE transactions on pattern analysis and machine intelligence_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _CVPR_.
  * Hinton, Vinyals, and Dean (2014) Hinton, G.; Vinyals, O.; and Dean, J. 2014.  Distilling the knowledge in a neural network.  In _NeurIPS Workshop_.
  * Hogg, McKean, and Craig (2005) Hogg, R. V.; McKean, J.; and Craig, A. T. 2005.  _Introduction to mathematical statistics_.  Pearson Education.
  * Huang and Belongie (2017) Huang, X.; and Belongie, S. 2017.  Arbitrary style transfer in real-time with adaptive instance normalization.  In _ICCV_.
  * Hui et al. (2021) Hui, B.; Yang, Y.; Yuan, H.; Burlina, P.; Gong, N. Z.; and Cao, Y. 2021.  Practical blind membership inference attack via differential comparisons.  In _NDSS_.
  * Jagielski et al. (2020) Jagielski, M.; Carlini, N.; Berthelot, D.; Kurakin, A.; and Papernot, N. 2020.  High accuracy and high fidelity extraction of neural networks.  In _USENIX Security_.
  * Jia et al. (2021) Jia, H.; Choquette-Choo, C. A.; Chandrasekaran, V.; and Papernot, N. 2021.  Entangled watermarks as a defense against model extraction.  In _USENIX Security_.
  * Johnson, Alahi, and Fei-Fei (2016) Johnson, J.; Alahi, A.; and Fei-Fei, L. 2016.  Perceptual losses for real-time style transfer and super-resolution.  In _ECCV_.
  * Juuti et al. (2019) Juuti, M.; Szyller, S.; Marchal, S.; and Asokan, N. 2019.  PRADA: protecting against DNN model stealing attacks.  In _EuroS &P_.
  * Kariyappa and Qureshi (2020) Kariyappa, S.; and Qureshi, M. K. 2020.  Defending against model stealing attacks with adaptive misinformation.  In _CVPR_.
  * Kesarwani et al. (2018) Kesarwani, M.; Mukhoty, B.; Arya, V.; and Mehta, S. 2018.  Model extraction warning in mlaas paradigm.  In _ACSAC_.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.  Technical report, Citeseer.
  * Lee et al. (2019) Lee, T.; Edwards, B.; Molloy, I.; and Su, D. 2019.  Defending against neural network model stealing attacks using deceptive perturbations.  In _IEEE S &P Workshop_.
  * Leino and Fredrikson (2020) Leino, K.; and Fredrikson, M. 2020.  Stolen memories: Leveraging model memorization for calibrated white-box membership inference.  In _USENIX Security 20_.
  * Li et al. (2021a) Li, Y.; Li, Y.; Wu, B.; Li, L.; He, R.; and Lyu, S. 2021a.  Invisible Backdoor Attack with Sample-Specific Triggers.  In _ICCV_.
  * Li et al. (2020a) Li, Y.; Wu, B.; Jiang, Y.; Li, Z.; and Xia, S.-T. 2020a.  Backdoor learning: A survey.  _arXiv preprint arXiv:2007.08745_.
  * Li et al. (2021b) Li, Y.; Zhai, T.; Jiang, Y.; Li, Z.; and Xia, S.-T. 2021b.  Backdoor Attack in the Physical World.  In _ICLR Workshop_.
  * Li et al. (2020b) Li, Y.; Zhang, Z.; Bai, J.; Wu, B.; Jiang, Y.; and Xia, S.-T. 2020b.  Open-sourced Dataset Protection via Backdoor Watermarking.  In _NeurIPS Workshop_.
  * Liu, Weng, and Zhu (2021) Liu, H.; Weng, Z.; and Zhu, Y. 2021.  Watermarking Deep Neural Networks with Greedy Residuals.  In _ICML_.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Sgdr: Stochastic gradient descent with warm restarts.  In _ICLR_.
  * Luo et al. (2021) Luo, X.; Wu, Y.; Xiao, X.; and Ooi, B. C. 2021.  Feature inference attack on model predictions in vertical federated learning.  In _ICDE_.
  * Madry et al. (2018) Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; and Vladu, A. 2018.  Towards deep learning models resistant to adversarial attacks.  In _ICLR_.
  * Maini, Yaghini, and Papernot (2021) Maini, P.; Yaghini, M.; and Papernot, N. 2021.  Dataset inference: Ownership resolution in machine learning.  In _ICLR_.
  * Minaee et al. (2021) Minaee, S.; Boykov, Y. Y.; Porikli, F.; Plaza, A. J.; Kehtarnavaz, N.; and Terzopoulos, D. 2021.  Image segmentation using deep learning: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Moosavi-Dezfooli et al. (2017) Moosavi-Dezfooli, S.-M.; Fawzi, A.; Fawzi, O.; and Frossard, P. 2017.  Universal adversarial perturbations.  In _CVPR_.
  * Nguyen and Tran (2020) Nguyen, A.; and Tran, A. 2020.  Input-aware dynamic backdoor attack.  In _NeurIPS_.
  * Orekondy, Schiele, and Fritz (2019) Orekondy, T.; Schiele, B.; and Fritz, M. 2019.  Knockoff nets: Stealing functionality of black-box models.  In _CVPR_.
  * Papernot et al. (2017) Papernot, N.; McDaniel, P.; Goodfellow, I.; Jha, S.; Celik, Z. B.; and Swami, A. 2017.  Practical black-box attacks against machine learning.  In _AsiaCCS_.
  * Sachs (2012) Sachs, L. 2012.  _Applied statistics: a handbook of techniques_.  Springer Science & Business Media.
  * Shokri et al. (2017) Shokri, R.; Stronati, M.; Song, C.; and Shmatikov, V. 2017.  Membership inference attacks against machine learning models.  In _IEEE S &P_.
  * Simonyan and Zisserman (2015) Simonyan, K.; and Zisserman, A. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  In _ICLR_.
  * Stokes et al. (2020) Stokes, J. M.; Yang, K.; Swanson, K.; Jin, W.; Cubillos-Ruiz, A.; Donghia, N. M.; MacNair, C. R.; French, S.; Carfrae, L. A.; Bloom-Ackermann, Z.; et al. 2020.  A deep learning approach to antibiotic discovery.  _Cell_ , 180(4): 688–702.
  * Tramèr et al. (2016) Tramèr, F.; Zhang, F.; Juels, A.; Reiter, M. K.; and Ristenpart, T. 2016.  Stealing machine learning models via prediction apis.  In _USENIX Security_.
  * Wang and Kerschbaum (2021) Wang, T.; and Kerschbaum, F. 2021.  RIGA: Covert and Robust White-Box Watermarking of Deep Neural Networks.  In _WWW_.
  * Wu et al. (2020) Wu, L.; Yang, Y.; Zhang, K.; Hong, R.; Fu, Y.; and Wang, M. 2020.  Joint item recommendation and attribute inference: An adaptive graph convolutional network approach.  In _SIGIR_.
  * Yan et al. (2021) Yan, H.; Li, X.; Li, H.; Li, J.; Sun, W.; and Li, F. 2021.  Monitoring-based Differential Privacy Mechanism Against Query Flooding-based Model Extraction Attack.  _IEEE Transactions on Dependable and Secure Computing_.
  * Zagoruyko and Komodakis (2016) Zagoruyko, S.; and Komodakis, N. 2016.  Wide Residual Networks.  In _BMVC_.
  * Zhai et al. (2021) Zhai, T.; Li, Y.; Zhang, Z.; Wu, B.; Jiang, Y.; and Xia, S.-T. 2021.  Backdoor attack against speaker verification.  In _ICASSP_.
  * Zhang et al. (2020) Zhang, J.; Chen, D.; Liao, J.; Zhang, W.; Hua, G.; and Yu, N. 2020.  Passport-aware Normalization for Deep Model Protection.  _NeurIPS_.
  * Zhang et al. (2018) Zhang, J.; Gu, Z.; Jang, J.; Wu, H.; Stoecklin, M. P.; Huang, H.; and Molloy, I. 2018.  Protecting intellectual property of deep neural networks with watermarking.  In _AsiaCCS_.
  * Zhao et al. (2020) Zhao, S.; Ma, X.; Zheng, X.; Bailey, J.; Chen, J.; and Jiang, Y.-G. 2020.  Clean-label backdoor attacks on video recognition models.  In _CVPR_.

Appendix

##  Appendix A Detailed Settings for Revisiting Defenses

###  A.1 The Limitation of Dataset Inference

Training Setups.  As described in Section 3.1 of the main manuscript, we train
the VGG-19 on $D_{l}$ (dubbed VGG-${D_{l}}$), the ResNet18 on $D_{r}$ (dubbed
ResNet-$D_{r}$), and another VGG-19 on the noisy dataset ${D^{\prime}}_{l}$
(dubbed VGG-${D^{\prime}}_{l}$), respectively. We train these three models for
200 epochs with the SGD optimizer and an initial learning rate of 0.1,
momentum of 0.9, weight decay of 5 $\times 10^{-4}$, and batch size of 128. We
decay the learning rate with the cosine decay schedule (Loshchilov and Hutter
2017) without a restart. We also use data augmentation techniques including
random crop and resize (with random flip). The training is conducted based on
the open-source code111https://github.com/verazuo/badnets-pytorch.

Verification Setups.  Following the setting of dataset inference (Maini,
Yaghini, and Papernot 2021), we extract feature embeddings from VGG-${D_{l}}$
with ${D_{l}}$, ResNet-$D_{r}$ with ${D_{r}}$ and VGG-${D^{\prime}}_{l}$ with
${D^{\prime}}_{l}$ respectively. And then we train a binary meta-classifier
for 1,000 epochs based on the feature embeddings of ResNet-$D_{r}$ with SGD
optimizer and a learning rate of 0.1. This meta-classifier is used to verify
if VGG-${D_{l}}$ and VGG-${D^{\prime}}_{l}$ is stolen from ResNet-$D_{r}$.
Another binary meta-classifier is trained with the same setting based on the
feature embeddings of VGG-${D_{l}}$ to verify if ResNet-$D_{r}$ is stolen from
VGG-${D_{l}}$. The verification is conducted based on the open-sourced code of
dataset inference222https://github.com/cleverhans-lab/dataset-inference.
Besides, both training and verification are implemented on a single GeForce
GTX 1080 Ti GPU.

###  A.2 The Limitation of Model Watermarking

Training Setups.  We adopt a white-square as the trigger pattern and train a
BadNets-based watermarked model with target label $y_{t}=2$ by modifying
$10\%$ training samples on CIFAR-10 dataset (Krizhevsky, Hinton et al. 2009).
We use the WideResNet (Zagoruyko and Komodakis 2016) with depth 28 and
widening factor of 10 (WRN-28-10) as the model architecture. The training is
performed on a single GeForce GTX 1080 Ti GPU and other training settings are
the same as those described in Section A.1.

Stealing Setups.  We perform the zero-shot learning based data-free
distillation (Fang et al. 2019) to obtain a stolen copy of each model. The
attack is implemented based on the open-source
code333https://github.com/VainF/Data-Free-Adversarial-Distillation.
Specifically, the stealing attack is performed for 200 epochs with SGD
optimizer and a learning rate of 0.1, momentum of 0.9, weight decay of
$5\times 10^{-4}$, and batch size of 256. The stolen model has the same model
structure as the victim model. The attack is performed on two GeForce GTX 1080
Ti GPUs.

Settings for Universal Adversarial Attack.  We use the projected gradient
descent (Madry et al. 2018) method to generate adversarial perturbations
within the ${\ell}^{\infty}$-ball where the maximum distortion size
$\epsilon=32$. The adversarial perturbations are initialized as the trigger
pattern used for watermarking. We implement the universal adversarial attack
for 40 iterations on the backdoored model and the stolen model.

##  Appendix B Detailed Settings for Main Experiments

###  B.1 Dataset Description

We evaluate our method on both CIFAR-10 (Krizhevsky, Hinton et al. 2009) and
ImageNet (Deng et al. 2009) dataset. CIFAR-10 contains 60,000 images (with
size 3x32x32) in 10 classes, including 50,000 training images and 10,000
testing images. ImageNet is a large-scale dataset and we use its subset
containing 20 random classes to evaluate our method. Each class of the subset
contains 500 images (with size 3x224x224) for training and 50 images for
testing.

###  B.2 Detailed Settings for Training

Following the settings used in (Maini, Yaghini, and Papernot 2021), we use the
WRN-16-1 (Zagoruyko and Komodakis 2016) and ResNet-18 (He et al. 2016) as the
victim and benign model on CIFAR-10 and ImageNet, respectively. For the
CIFAR-10 dataset, the training is conducted based on the open-source
code444https://github.com/kuangliu/pytorch-cifar. Specifically, both victim
model and benign model are trained for 200 epochs with SGD optimizer and an
initial learning rate of 0.1, momentum of 0.9, weight decay of 5 $\times
10^{-4}$, and batch size of 128. We decay the learning rate with the cosine
decay schedule (Loshchilov and Hutter 2017) without a restart. We also use
data augmentation techniques including random crop and resize (with random
flip). For the ImageNet dataset, both the victim model and benign model are
trained for 200 epochs with SGD optimizer and an initial learning rate of
0.001, momentum of 0.9, weight decay of $1\times 10^{-4}$, and batch size of
32. The learning rate is decreased by a factor of 10 at epoch 150. All
training processes are performed on a single GeForce GTX 1080 Ti GPU.

###  B.3 Detailed Settings for Model Stealing

Following the setting adopted in dataset inference (Maini, Yaghini, and
Papernot 2021), we conduct model stealing with three different adversary’s
permission levels, including (1) dataset-accessible attacks, (2) model-
accessible attacks, and (3) query-only attacks. Besides, we also provide the
results of directly copying the victim model (dubbed ‘Source’) and examining a
suspicious model which is not stolen from the victim (dubbed ‘Independent’)
for reference.

Except for distillation-based methods ($i.e.$, Distillation and Zero-Shot),
all methods are performed on a single GeForce GTX 1080 Ti GPU. We adopt two
GeForce GTX 1080 Ti GPUs for distillation-based methods. Besides, except for
fine-tuning, we adopt WRN-16-1 and ResNet-18 as the stolen model on CIFAR-10
and ImageNet, respectively. More detailed settings are as follows:

Distillation.  We implement the model distillation (Hinton, Vinyals, and Dean
2014) based on its open-sourced
code555https://github.com/thaonguyen19/ModelDistillation-PyTorch.
Specifically, the stolen model is trained with SGD optimizer and an initial
learning rate of 0.1, momentum of 0.9, weight decay of $10^{-4}$. We use
ReduceLROnPlateau as the learning rate scheduler. Specifically, we set the
mode as ‘min’, factor as 0.1, and patience as 10.

Zero-shot.  We perform zero-shot learning based data-free distillation (Fang
et al. 2019) to obtain the stolen model. All settings are the same as those
described in Section A.2.

Fine-tuning.  In this setting, attackers obtain stolen models by fine-tuning
victim models on different datasets. Following the settings in (Maini,
Yaghini, and Papernot 2021), we randomly select 500,000 samples from the
original TinyImages (Birhane and Prabhu 2021) as the substitute data to fine-
tune the victim model for experiments on CIFAR-10. For the ImageNet
experiments, we randomly choose samples with another 20 classes from the
original ImageNet as the substitute data. We fine-tune the victim model for 5
epochs with settings described in Section B.2.

Label-query.  Following the settings used in dataset inference (Maini,
Yaghini, and Papernot 2021), we train the stolen model for 20 epochs with a
substitute dataset labeled by the victim model. Other settings are the same as
those described in Section B.2.

Logit-query.  In this setting, attackers train the stolen model by minimizing
the KL-divergence between its outputs ($i.e.$, logits) and those of the victim
model. Following the settings used in dataset inference (Maini, Yaghini, and
Papernot 2021), we train the stolen model for 20 epochs with the same settings
stated in Section B.2.

##  Appendix C Effectiveness of the Sign Function

In this section, we verify the effectiveness of adopting the sign vector of
gradients instead of gradients themselves in the training of our meta-
classifier.

Settings.  We conduct the experiments on CIFAR-10 and ImageNet dataset. Except
for the discussed component, other settings are the same as those used in
Section B.1-B.3.

Results.  As shown in Table 8, adopting the sign of gradients is significantly
better than adopting gradients directly. This is probably because the
‘direction’ of gradients contains more information compared with their
‘magnitude’. We will further explore it in our future work.

Dataset$\rightarrow$ | CIFAR-10 | ImageNet
---|---|---
Model Stealing$\downarrow$ | Gradient | Sign of Gradient (Ours) | Gradient | Sign of Gradient (Ours)
$\Delta\mu$ | p-value | $\Delta\mu$ | p-value | $\Delta\mu$ | p-value | $\Delta\mu$ | p-value
Source | 0.44 | $10^{-5}$ | $\bm{0.97}$ | $\bm{10^{-7}}$ | 0.15 | $10^{-4}$ | $\bm{0.90}$ | $\bm{10^{-5}}$
Distillation | 0.27 | 0.01 | $\bm{0.53}$ | $\bm{10^{-7}}$ | 0.15 | $10^{-4}$ | $\bm{0.61}$ | $\bm{10^{-5}}$
Zero-shot | 0.03 | $10^{-3}$ | $\bm{0.52}$ | $\bm{10^{-5}}$ | 0.12 | $10^{-3}$ | $\bm{0.53}$ | $\bm{10^{-4}}$
Fine-tuning | 0.04 | $10^{-5}$ | $\bm{0.50}$ | $\bm{10^{-6}}$ | 0.13 | $10^{-3}$ | $\bm{0.60}$ | $\bm{10^{-5}}$
Label-query | 0.08 | $10^{-3}$ | $\bm{0.52}$ | $\bm{10^{-4}}$ | 0.13 | $10^{-3}$ | $\bm{0.55}$ | $\bm{10^{-3}}$
Logit-query | 0.07 | $10^{-5}$ | $\bm{0.54}$ | $\bm{10^{-4}}$ | 0.12 | $10^{-3}$ | $\bm{0.55}$ | $\bm{10^{-4}}$
Independent | $\bm{0.00}$ | $\bm{1.00}$ | $\bm{0.00}$ | $\bm{1.00}$ | $\bm{10^{-10}}$ | $\bm{0.99}$ | $10^{-5}$ | $\bm{0.99}$

Table 8: The performance of our meta-classifier trained with different
features.

Dataset$\rightarrow$ | CIFAR-10 | ImageNet
---|---|---
Model Stealing$\downarrow$ | Training Set | Testing Set | Training Set | Testing Set
$\Delta\mu$ | p-value | $\Delta\mu$ | p-value | $\Delta\mu$ | p-value | $\Delta\mu$ | p-value
Source | 0.97 | $10^{-7}$ | 0.96 | $10^{-7}$ | 0.90 | $10^{-5}$ | 0.93 | $10^{-7}$
Distillation | 0.53 | $10^{-7}$ | 0.53 | $10^{-5}$ | 0.61 | $10^{-5}$ | 0.42 | $10^{-5}$
Zero-shot | 0.52 | $10^{-5}$ | 0.53 | $10^{-5}$ | 0.53 | $10^{-4}$ | 0.34 | $10^{-3}$
Fine-tuning | 0.50 | $10^{-6}$ | 0.47 | $10^{-6}$ | 0.60 | $10^{-5}$ | 0.72 | $10^{-5}$
Label-query | 0.52 | $10^{-4}$ | 0.52 | $10^{-4}$ | 0.55 | $10^{-3}$ | 0.40 | $10^{-3}$
Logit-query | 0.54 | $10^{-4}$ | 0.53 | $10^{-4}$ | 0.55 | $10^{-4}$ | 0.48 | $10^{-4}$
Independent | 0.00 | 1.00 | 0.00 | 1.00 | $10^{-5}$ | 0.99 | $10^{-9}$ | 0.99

Table 9: Results of ownership verification with transformed samples generated
by samples in training and testing dataset.

##  Appendix D Relations with Related Works

In this section, we discuss the similarities and differences between our
defense and related methods.

###  D.1 Relations with Property Inference Attacks

Our defense is inspired by the property inference attacks (Ganju et al. 2018;
Wu et al. 2020; Luo et al. 2021), which intend to identify whether a
particular property ($i.e.$, attribute or feature) is contained in the
training set of given DNNs. Similar to these attacks, our defense also
examines whether a suspicious model contains the knowledge of predefined
external features. However, different from inference attacks whose targets are
attributes directly contained in the training set, the concept of features is
more complicated in our method. In our defense, the feature is defined as a
complex combination of attributes ($i.e.$, pixel values).

###  D.2 Relations with Membership Inference

Membership inference attacks (Shokri et al. 2017; Leino and Fredrikson 2020;
Hui et al. 2021) aim to identify whether a particular sample is used to train
given DNNs. Similar to these attacks, our method also adopts some training
samples for ownership verification. However, our defense intend to analyze
whether the suspicious model contains the knowledge of external features
rather than whether the model is trained on those transformed samples. To
verify it, we design additional experiments, as follows:

Settings.  We compare our method with its variant which adopts testing images
to generate transformed ones used in ownership verification. Except for this,
other settings are the same as those used in Section 5.2 of the main
manuscript.

Results.  As shown in Table 9, adopting testing images has similar effects to
that of using training images in generating transformed images used for
ownership verification. This phenomenon verifies that our defense is
fundamentally different from membership inference attacks.

###  D.3 Relations with Backdoor Attacks

Similar to that of (poisoning-based) backdoor attacks (Li et al. 2020a; Zhao
et al. 2020; Zhai et al. 2021), our defense embeds predefined behaviors into
DNNs through modifying some training samples. However, different from that of
backdoor attacks, our method neither changes the label of poisoned samples (to
the target label) nor only selects training samples with the specific category
($i.e.$, target label) for poisoning. As such, the embedding of external
features adopted in our defense will not introduce hidden backdoors into the
trained deployed model.