# Teacher Model Fingerprinting Attacks Against Transfer Learning

Yufei Chen1,2 Chao Shen1 Cong Wang2 Yang Zhang3

1Xi’an Jiaotong University 2City University of Hong Kong
3CISPA Helmholtz Center for Information Security

###### Abstract

Transfer learning has become a common solution to address training data
scarcity in practice. It trains a specified student model by reusing or fine-
tuning early layers of a well-trained teacher model that is usually publicly
available. However, besides utility improvement, the transferred public
knowledge also brings potential threats to model confidentiality, and even
further raises other security and privacy issues.

In this paper, we present the first comprehensive investigation of the teacher
model exposure threat in the transfer learning context, aiming to gain a
deeper insight into the tension between public knowledge and model
confidentiality. To this end, we propose a teacher model fingerprinting attack
to infer the origin of a student model, i.e., the teacher model it transfers
from. Specifically, we propose a novel optimization-based method to carefully
generate queries to probe the student model to realize our attack. Unlike
existing model reverse engineering approaches, our proposed fingerprinting
method neither relies on fine-grained model outputs, e.g., posteriors, nor
auxiliary information of the model architecture or training dataset. We
systematically evaluate the effectiveness of our proposed attack. The
empirical results demonstrate that our attack can accurately identify the
model origin with few probing queries. Moreover, we show that the proposed
attack can serve as a stepping stone to facilitating other attacks against
machine learning models, such as model stealing.

##  1 Introduction

The past decade has witnessed an unprecedented development of machine learning
(ML). Yet, the progress of ML heavily relies on sophisticated models,
sufficient computing resources, and a massive volume of training data, which
remain major constraints to build a high-performance ML model.

Transfer learning opens a pathway for overcoming obstacles raised by the lack
of data or computing resources; it is essentially an ML paradigm to transfer
the knowledge from a well-learned domain into a specified domain where
training data is scarce. Concretely, transfer learning establishes a new model
(a.k.a. student model) through borrowing early layers from a pre-trained model
(a.k.a. teacher model), which requires far less efforts than training from
scratch. It has been proved to be a promising ML practice and widely applied
in a wide range of academic and industrial areas, such as computer vision
[25], natural language processing [44], etc.

Despite the huge success, such cross-domain knowledge learning paradigm also
raises security and privacy concerns:

  * •

There exist many advanced ML models, each of which can potentially serve as a
teacher model for transfer learning. Choosing an appropriate teacher model to
train a student model requires a large number of engineering efforts. Thus,
for a student model, the choice of its teacher model certainly belongs to the
model owner’s intellectual property (IP), and should be kept confidential.

  * •

On the other side of the coin, from the responsible ML perspective, a teacher
model owner does not want their model to be transferred to perform unethical
or illegal tasks, such as facial recognition or weapon classification. Thus, a
teacher model owner needs a way to track the parties that use their model to
build student models.

  * •

Malicious parties can intentionally publish vulnerable ML models online. When
such models are used as teacher models for transfer learning, the
corresponding student models may inherit some vulnerabilities. For instance,
Zhang et al. demonstrate that fine-tuned transfer learning models are more
prone to transferable adversarial examples [42]. Yao et al. show that an
attacker can infect transfer learning models by distributing pre-trained
models with backdoors [40].

  * •

Also, after learning the teacher model that a student is transferred from, an
attacker can perform more effective malicious attacks, such as adversarial
example attacks [9] and model stealing/extraction attacks [36].

All these concerns require us to gain a deeper insight into the teacher model
exposure or transfer learning.

In this paper, we present the first comprehensive investigation of the teacher
model exposure threat in the transfer learning context. In particular, we
propose a novel teacher model fingerprinting attack which can effectively
identify the teacher model of a transferred student. The key idea behind is to
generate a set of fingerprinting pairs for each teacher model candidate, which
will activate similar latent feature representations. Hopefully, most of them
will also trigger similar latent features and bring a pair of similar
responses on the student model, if the student model is transferred from the
teacher model candidate. We formalize the query generation process as an
optimization problem, which can be solved via gradient descent. Note that, the
fingerprinting pairs for each teacher model can be applied to all student
models transferred from the teacher model, i.e., they are reusable. This point
demonstrates the efficiency and practicality of our method.

We conduct extensive experiments to investigate the effectiveness of our
proposed method, and thoroughly evaluate how various factors affect attack
performance. The evaluation shows that our attack can achieve high teacher
model inference accuracy even with a limited amount of queries in top-1 label
exposure. Moreover, our attack still works well when synthesizing queries from
samples unrelated to the target domain, or even from random noises. In
addition, we show that our fingerprinting attack can serve as a stepping stone
to performing more effective model stealing attacks.

Our study should not be confused with recent research efforts on model
stealing or model reverse engineering: Their primary goal is to directly steal
the exact victim model, build a surrogate model with similar functionality, or
infer other internal model information. By contrast, our proposed attack
essentially focuses on exploiting the model sharing practice in transfer
learning, to quickly identify the presence of pre-trained components. Once
identifying the teacher model components, the attacker has stepped further to
realize the aforementioned malicious or unethical goals.

Besides, from the technical perspective, our attack remedies the key
weakness—significance attack expense caused by massive attack queries or
shadow models—of prior arts. Our attack just needs to identify the transferred
feature map with few queries, rather than recovering model parameters or
architectures interactively.

Furthermore, previous studies are subject to extra attack assumptions, such as
transparent model architectures [7] and fine-grained model outputs [36]. E.g.,
the teacher model fingerprinting method proposed by Wang et al. [38] requires
access to posteriors of the target model prediction. However, in the real
world, raw model outputs are usually perturbed or hidden for security [18] or
other product deployment concerns. Instead, our work is set up on a more
generic and realistic scenario. We assume the attacker exclusively receives
the victim black-box model’s final decision, i.e., the top-1 classification
result, which gives the minimum amount of information. This assumption implies
that our proposed attack is likely to be mounted on any transfer learning
classifiers. Additionally, we assume the attacker has zero knowledge of the
victim’s training dataset, but they are allowed to utilize public data. These
assumptions ensure our proposed method is feasible and applicable in practice.

Our main contributions can be summarized as follows:

  * •

We take the first step to comprehensively investigate the teacher model
exposure in the transfer learning context. In particular, we demonstrate a
teacher model fingerprinting attack against transfer learning.

  * •

We propose a novel optimization-based technique to implement our attack.

  * •

Extensive evaluations demonstrate the efficacy of our method, and we further
show that our teacher model fingerprinting attack can be used to facilitate
model stealing attacks against ML models.

##  2 Preliminaries

###  2.1 Problem Statement

Transfer learning. Transfer learning is commonly used to solve the data-hungry
problem of ML, especially for deep learning models [20, 35, 30]. The core is
to leverage the feature maps learned by the teacher model, so as to avoid
significant cost of training a large model from scratch.

In the source domain, the teacher model $\mathcal{T}$ has been trained on a
large scale dataset
$\mathcal{D}_{\mathtt{T}}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{N_{\mathtt{S}}}$,111In
this paper, the notation $\\{\cdot\\}$ refers to a multiset, which may contain
repeated values. and then it is released for other downstream ML developers.
In a typical transfer learning workflow, the student model $\mathcal{S}$ is
firstly constructed with early $k$ layers copied from $\mathcal{T}$ for
feature extraction, as well as newly added classification layers
$\mathcal{C}_{\mathtt{S}}(\cdot)$ (e.g., fully connected layers, SVM, etc.) on
the top. Then, $\mathcal{S}$ gets trained on a usually confidential dataset
$\mathcal{D}_{\mathtt{S}}=\\{\mathbf{x}_{j},y_{j})\\}_{j=1}^{N_{\mathtt{T}}}$,
where $N_{\mathtt{T}}\ll N_{\mathtt{S}}$. In practice, there are two training
strategies:

  * •

Feature extractor, where all pre-trained layers are frozen to compose a
feature extractor $\mathcal{F}_{\mathtt{T}}$.

  * •

Fine-tuning, where part of the pre-trained parameters of
$\mathcal{F}_{\mathtt{T}}$ will be updated for better fitting on
$\mathcal{D}_{\mathtt{S}}$.

After being carefully trained, $\mathcal{S}$ is deployed into a Machine-
Learning-as-a-service (MLaaS) platform to provide services. A customer can
send a query $\mathbf{x}$ to the MLaaS service and get the corresponding
result $f(\mathcal{S}(\mathbf{x}))$, via an authorized API $f(\cdot)$.

Teacher model fingerprinting. In this paper, we develop a novel approach to
infer the origin of a student model with only black-box access. Our motivation
is that the student model is likely to inherit fingerprintable model
behaviors, i.e., the way to extract features in our case (a.k.a. feature map),
from the teacher model. Some attackers can possibly elicit such
fingerprintable behaviors by sending carefully crafted queries to the student
model, which is sometimes concealed into an MLaaS platform. Consequently, they
are able to identify the teacher model based on the victim’s responses. We
refer to this process as teacher model fingerprinting attacks.

###  2.2 Threat Model

Attack motivations. We first list potential motivations to conduct the teacher
model fingerprinting attack.

Breaking model confidentiality. Internal model setup, such as architecture or
parameters, is supposed to be kept confidential for the sake of IP protection
or security consideration. However, the model behind an MLaaS platform is no
longer a secret once the teacher model has been identified. The attacker can
effortlessly recover early layers with the publicly transparent teacher model
information.

Stepping stone to advanced attacks. As mentioned before, our teacher model
fingerprinting attack provides a cheap way to recover most parameters
inherited from the teacher model, which helps to open the “black box”. Once
opening the black box, the attacker has stepped further to discover and
exploit vulnerabilities of the student model. For instance, they can easily
conduct various white-box adversarial attacks against the victim model, such
as adversarial example attacks [8] and membership inference attacks [22].
Also, recent studies have shown that transfer learning may transfer
vulnerabilities from the teacher [42], which the attacker can directly
exploit.

Forensics. Despite the aforementioned malicious goals of the teacher model
fingerprinting attack, we can also treat it as a new forensic tool for ML
applications. For instance, a teacher model owner does not want their model to
be transferred to perform unethical or illegal tasks, such as weapon
classification. Also, in some cases, certain parties can intentionally publish
a vulnerable teacher model, which will result in the corresponding student
models inheriting vulnerabilities [40]. For both of these cases, our teacher
model fingerprinting can be used by a third-party forensic service as a
defense to track the origin of a student model.

Threat model in detail. Then, we describe the detailed threat model in real-
world settings.

For our teacher model fingerprinting attack, we consider an attacker with
black-box access to the victim student model $\mathcal{S}$. The attacker is
able to send an arbitrary input $\mathbf{x}$ to $\mathcal{S}$ via an
authorized API $f(\cdot)$ and receive the corresponding response
$f(\mathcal{S}(\mathbf{x}))$. Three cases may arise.

  * •

$f(\mathcal{S}(\mathbf{x}))=\mathcal{S}(\mathbf{x})$. In a low-level security
setting, the response exposes the exact model output, which contains class
labels and raw confidence values.

  * •

$f(\mathcal{S}(\mathbf{x}))=\mathcal{S}(\mathbf{x})+\epsilon$. The MLaaS
provider returns a perturbed version of the model output to avoid privacy
breaches like membership inference attacks [18].

  * •

$f(\mathcal{S}(\mathbf{x}))=\operatorname*{arg\,max}_{i}\mathcal{S}(\mathbf{x})_{i}$.
Only a top-1 class label is returned, which gives the minimal piece of
information [23].

This work demonstrates the teacher model fingerprinting attack in the most
restrictive case—top-1 label exposure, i.e.,
$f(\mathbf{x})=\operatorname*{arg\,max}_{i}\mathcal{S}(\mathbf{x})_{i}$.222In
the remaining, we abuse the notation $f(\mathbf{x})$ to represent
$f(\mathcal{S}(\mathbf{x}))$.

We assume the attacker has obtained all potential teacher model candidates
$\\{\mathcal{T}_{i}\\}$ from public resources (e.g., deep learning frameworks
like PyTorch, or websites like ModelZoo [1]). Neither the teacher dataset
$\mathcal{D}_{\mathtt{T}}$ in the source domain nor the student dataset
$\mathcal{D}_{\mathtt{S}}$ in the target domain is available. However, the
attacker can collect public datasets like MNIST, CIFAR10 to help perform the
teacher model fingerprinting attack. The attacker has two primary goals: one
is to infer the teacher model accurately, while the other is to use as few
probing queries as possible, to limit attack costs and keep the attack
stealthy.

##  3 Teacher Model Fingerprinting

###  3.1 Overview

At the beginning, we formalize the teacher model fingerprinting attack in the
transfer learning context:

######  Definition 3.1 (Teacher Model Fingerprinting Attack).

Suppose there is an attacker given a set of $N$ realistic inputs
$\\{\mathbf{x}_{i}\\}$ (a.k.a. probing input), a set of teacher model
candidates $\\{\mathcal{T}_{j}\\}$, and an authorized API $f(\cdot)$ to the
target black-box student model $\mathcal{S}$ in only top-1 label exposure, the
teacher model fingerprinting attack is to infer which $\mathcal{T}_{j}$ is
adopted by $\mathcal{S}$.

Figure 1: Illustration of our fingerprinting attack. Given a probing input
$\mathbf{x}$ and a teacher model $\mathcal{T}$, the attacker aims to find a
synthetic input $\mathbf{x}^{\prime}$, which will activate a similar latent
representation at the $k$-th component with $\mathbf{x}$:
$\mathcal{F}_{\mathtt{T}}(\mathbf{x}^{\prime})\approx\mathcal{F}_{\mathtt{T}}(\mathbf{x})$.
If the first $k$ components of the student model $\mathcal{S}$ comes from the
teacher model $\mathcal{T}$, some of which may get slightly fine-tuned (still,
$\mathcal{F}_{\mathtt{T}}^{\prime}(\cdot)\approx\mathcal{F}_{\mathtt{T}}(\cdot)$),
the student model is likely to produce a pair of similar responses
$\mathcal{S}(\mathbf{x}^{\prime})\approx\mathcal{S}(\mathbf{x})$.

We start from the feature extractor transfer learning setting. Suppose there
is a student model $\mathcal{S}$ composed of a feature extractor
$\mathcal{F}_{\mathtt{T}}$, with early $k$ components copied from the teacher
model $\mathcal{T}$, and newly trained classification layers
$\mathcal{C}_{\mathtt{S}}$. Our intuition is that given a probing input
$\mathbf{x}$, we can artificially craft a synthetic input
$\mathbf{x}^{\prime}$ so that
$\mathcal{F}_{\mathtt{T}}(\mathbf{x}^{\prime})\approx\mathcal{F}_{\mathtt{T}}(\mathbf{x})$.
Hopefully, we will activate similar outputs on the student model as
$\mathcal{S}(\mathbf{x}^{\prime})=\mathcal{C}_{\mathtt{S}}(\mathcal{F}_{\mathtt{T}}(\mathbf{x}^{\prime}))\approx\mathcal{C}_{\mathtt{S}}(\mathcal{F}_{\mathtt{T}}(\mathbf{x}))=\mathcal{S}(\mathbf{x})$,
and therefore have a greater chance to receive similar responses from the API.
In this case, we refer to the pair
$\langle{\mathbf{x},\mathbf{x}^{\prime}}\rangle$ as a fingerprinting pair.
Although the above intuition stands on the feature extractor scenario, in
Section 4.3.2, we will show that our attack still works in the fine-tuning
setting. For presentation purpose, we summarize all the notations mentioned
here and in the following sections in Table 1.

Figure 1 provides a schematic view of our teacher model fingerprinting attack.
Overall, it follows two steps:

Step I: generating a set of fingerprinting pairs
$\\{\langle{\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}}\rangle\\}$. To perform the
teacher model fingerprinting attack, the attacker firstly collects a set of
candidate teacher models and realistic probing images.333In Section 4.3.4, we
will show that we can achieve successful inference attacks even if using
random noise images as probing inputs. For each candidate teacher model
$\mathcal{T}_{j}$, the attacker needs to determine which components may
constitute the feature extractor $\mathcal{F}_{\mathcal{T}_{j}}$ for the
student model. Conventionally, the student model will adopt whole convolution
part from the teacher model. Then, for each probing input $\mathbf{x}_{i}$,
the attacker aims to craft a synthetic input $\mathbf{x}_{i}^{\prime}$, which
will trigger similar latent representations on
$\mathcal{F}_{\mathcal{T}_{j}}$. We will study how the amount of
$\\{\mathbf{x}_{i}\\}$, the source of $\\{\mathbf{x}_{i}\\}$, as well as the
choice of components of $\mathcal{F}_{\mathcal{T}_{j}}$ affect the proposed
attack in Section 4.3.3, Section 4.3.4, and Section 4.3.5, respectively.

This attack query generation procedure ensures our attack is a cheap one,
since: i) the query generation procedure is locally conducted without any
expense by the target MLaaS service, and ii) for fingerprinting pairs bonded
to $\mathcal{F}_{\mathcal{T}_{j}}$, they are reusable to probe whether other
student models come from $\mathcal{F}_{\mathcal{T}_{j}}$, without further
computation costs on query generation.

Step II: inferring the teacher model according to black-box responses
$\\{\langle{y_{i},y_{i}^{\prime}}\rangle\\}$. After obtaining fingerprinting
pairs, the next step is to send them to the target black box, and infer which
teacher model is used according to the responses. Ideally, if the target model
relies on the feature extractor $\mathcal{F}_{\mathcal{T}_{j}}$, most
generated fingerprinting pairs would produce a pair of responses matched on
the same label. The inference stage follows the “one-of-the-best” strategy:
choosing the candidate owning the most matched responses as the inference
result. In Section 5, we will introduce a strategy for robust inferences.

Notation | Description
---|---
$\mathcal{T}$ | Teacher model used in transfer learning.
$\mathcal{S}$ | Student model used in transfer learning.
$\mathcal{D}_{\mathtt{T}}$ | Training data of the teacher model.
$\mathcal{D}_{\mathtt{S}}$ | Training data of the student model.
$\mathcal{F}_{\mathtt{T}}$ |  Feature extractor transferred from $\mathcal{T}$ to $\mathcal{S}$
$\mathcal{C}_{\mathtt{S}}$ | The newly trained classification layers concatenated after the
|  feature extractor of $\mathcal{S}$, i.e.,
$\mathcal{S}(\cdot)=\mathcal{C}_{\mathtt{S}}(\mathcal{F}_{\mathtt{T}}(\cdot))$
$f(\cdot)$ | An authorized API by which the attacker accesses the student
|  model $\mathcal{S}$ in a black-box fashion. In this work,
$f(\cdot)=\operatorname*{arg\,max}_{i}(\cdot)_{i}$.
$\\{\mathbf{x}_{i}\\}$ |  A set of realistic images used for the attack (a.k.a. probing inputs).
$\\{\mathbf{x}_{i}^{\prime}\\}$ |  A set of synthetic inputs based on the probing inputs.
$\\{\langle{\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}}\rangle\\}$ | A set of fingerprinting pairs to query the black-box student model.
$\\{\langle{y_{i},y_{i}^{\prime}}\rangle\\}$ | The corresponding black-box responses for each fingerprinting pair.
$N$ | The amount of fingerprinting pairs send by the attacker
|  (a.k.a. query budget).
$S_{\textup{match}}$ |  The matching set composed of all the fingerprinting pairs
| that triggers matched API responses.
$P_{\textup{match}}$ | The proportion of matched pairs in a set of fingerprinting pairs.
$v_{\textup{fgpt}}$ |  The fingerprinting vector composed of $P_{\textup{match}}$ by all candidates.
$S_{\textup{support}}$ |  The supporting set used for the robust teacher model fingerprinting.
Table 1: List of notations.

###  3.2 Step I: Attack Query Generation

For our teacher model fingerprinting attack, we use the $L_{2}$-norm metric to
measure the similarity of latent representations. Given a probing input
$\mathbf{x}$ and a candidate feature extractor $\mathcal{F}_{\mathtt{T}}$, we
formalize the attack query generation task as

$\displaystyle\mathbf{x}^{\prime}=\operatorname*{arg\,min}_{\tilde{\mathbf{x}}}\|\mathcal{F}_{\mathtt{T}}(\tilde{\mathbf{x}})-\mathcal{F}_{\mathtt{T}}(\mathbf{x})\|_{2}$
(1) $\displaystyle\textbf{s.t.~{}}\tilde{\mathbf{x}}\in[0,255]$

To solve this constrained optimization problem Equation 1, we adopt the
optimization strategy proposed by Carlini and Wagner [9]: by introducing a new
variable $\mathbf{w}$, and setting

$\tanh(\mathbf{w})=\frac{2\tilde{\mathbf{x}}}{255}-1$ (2)

we have

$\displaystyle-1\leq\tanh(\mathbf{w})\leq 1$ (3) $\displaystyle\Rightarrow$
$\displaystyle-1\leq\frac{2\tilde{\mathbf{x}}}{255}-1\leq 1$
$\displaystyle\Rightarrow$ $\displaystyle 0\leq\tilde{\mathbf{x}}\leq 255$

which satisfies the constraints. Therefore, the original problem Equation 1
can be converted to

$\displaystyle\mathbf{w}^{\prime}=\operatorname*{arg\,min}_{\mathbf{w}}\left\|\mathcal{F}_{\mathtt{T}}\left(255*\frac{1}{2}\left(\tanh({\mathbf{w}})+1\right)\right)-\mathcal{F}_{\mathtt{T}}(\mathbf{x}_{i})\right\|_{2}$
(4)

It can be easily solved via a gradient descent optimizer. In this paper, we
adopt the Adam optimizer [19], with learning rate set as 0.001 for 30,000
iterations.

However, due to type casting (floating points to integers) and optimization
loss, $\mathcal{F}_{\mathtt{T}}(\mathbf{x}_{i}^{\prime})$ cannot be perfectly
equal to $\mathcal{F}_{\mathtt{T}}(\mathbf{x}_{i})$. So after obtaining the
synthetic input $\mathbf{x}^{\prime}$, we will examine whether
$\operatorname*{arg\,max}_{i}\mathcal{T}(\mathbf{x})_{i}=\operatorname*{arg\,max}_{j}\mathcal{T}(\mathbf{x}^{\prime})_{j}$
on the teacher model. If not, we will discard $\mathbf{x}_{i}^{\prime}$ and
generate a new one.

###  3.3 Step II: Teacher Model Inference

The next step is to infer whether the student model comes from one of the
candidate teacher feature extractors. For each candidate feature extractor,
the attacker can generate a set of fingerprinting pairs and send them to the
black-box student model. Intuitively, for a specific feature extractor
$\mathcal{F}_{\mathtt{T}}$, the more matched pairs of black-box responses are
obtained, the more likely the target model is transferred from
$\mathcal{F}_{\mathtt{T}}$. Here we define matching set useful for further
discussion.

######  Definition 3.2 (Matching Set).

After sending $N$ fingerprinting pairs, all the pairs triggering two same API
responses ($y_{i}=y_{i}^{\prime})$ compose the matching set
$S_{\textup{match}}$.

We use three heuristics to measure how much “belief” the attacker has to infer
the teacher feature extractor. The details are described below.

Matching proportion. A high matching proportion of fingerprinting pairs
indicates a high possibility that the target student model uses the same
feature extractor as the attacker. We compute the matching proportion as a
primary metric to depict how “perfectly matched” the candidate is with the
target feature extractor.

$P_{\textup{match}}=\frac{|S_{\textup{match}}|}{N}$

The attacker will obtain a fingerprinting vector composed of matching
proportions
$v_{\textup{fgpt}}=[P_{\textup{match}}^{1},P_{\textup{match}}^{2},\cdots]$ for
all candidate feature extractors
$[\mathcal{F}_{\mathcal{T}_{1}},\mathcal{F}_{\mathcal{T}_{2}},\cdots]$. Then
the attacker will choose the candidate feature extractor that achieves the
highest $P_{\textup{match}}$ as their inference result.

A high matching proportion is insufficient to prove that the candidate feature
extractor is adopted by the student model. Our preliminary results show that
it is possible to obtain matched pairs from unmatched features. That is, the
latent features extracted from probing inputs and synthetic inputs are
significantly different. We will dive into such “false matching” phenomenon in
Section 5.

Eccentricity. Eccentricity is adopted by [26] to measure how an item “stands
out” from the rest in a set. It is defined as

$E(v)=\frac{\text{max}(v)-\text{max}_{2}(v)}{\sigma(v)}$

where $\text{max}(\cdot)$ and $\text{max}_{2}(\cdot)$ refers to the first and
second highest value, respectively, and $\sigma(\cdot)$ refers to the standard
deviation. The higher is the eccentricity of $v_{\textup{fgpt}}$, the more
distinguishing is $\text{max}(v)$, and the more confident the attacker is to
make the inference.

Empirical entropy. Here, we introduce the empirical entropy of a set as a
heuristic to measure how much “information” the set presents. Formally, for a
set $\\{x_{1},x_{2},\cdots,x_{n}\\}$ and a sample space $\mathcal{X}$, the
empirical entropy is defined as:

$H(\mathcal{X})=-\sum_{x\in\mathcal{X}}\hat{p}(x)\log\hat{p}(x)$

where

$\hat{p}(x)=\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}(x_{i}==x),$

and $\mathbbm{1}(\cdot)$ is the indicator function. We calculate the empirical
entropy of the matching set with the highest $P_{\textup{match}}$, to estimate
how much information is given to make the inference.

##  4 Evaluation

We first describe the basic experimental setup and then present our evaluation
results in different real-world settings. Basically, we aim to explore two key
questions in this paper:

  * •

What is the impact of the transfer learning setup on the teacher model
fingerprinting attack? (i.e., model architecture, student dataset, and tunable
parameters.)

  * •

How does the attacker capability affect our proposed attack? (i.e., query
budget, available attack dataset, as well as the choice of probing features.)

###  4.1 Dataset

In our experiments, our student datasets to train transfer learning models
include:

Dogs-vs-Cats. Our Dogs-vs-Cats dataset is supposed to build a two-class
classifier to discriminate one image either contains a dog or a cat. In our
experiments, we have downloaded the labeled 12,500 dog and 12,500 cat images
from the Internet [2]. Then, we select the first 10,000 dog images and the
first 10,000 cat images to compose the training set, while the remaining 2,500
dog images and 2,500 cat images compose the testing set.

MNIST. The MNIST dataset [3] is a widely used dataset to build up toy image
recognition ML models. It consists of 60,000 training and 10,000 testing
handwritten digit samples, containing 10 classes from digit 0 to digit 9.

CIFAR10 and CIFAR100. The CIFAR10 and CIFAR100 datasets [4] are another two
widely adopted benchmark datasets for image classification tasks. The CIFAR10
dataset contains 10 classes, with 5,000 training samples and 1000 test samples
in each. The CIFAR100 dataset is similar to the CIFAR10, except that the
former consists of 100 classes with 600 images in each. Each class contains
500 training images and 100 testing images.

STL10. The STL10 dataset [5] is originally designed to develop supervised as
well as unsupervised learning models. Our experiment uses the labeled set
covering 10 classes, with 500 training images and 800 test images per class.

CelebA. Our experiments use the first 50,000 facial images as training samples
and the following 10,000 facial images as the test samples from the CelebA
dataset [24], which are annotated with 40 binary attributes. The previous five
datasets are adopted to train multi-class classification models. But for
CelebA, we instead build up multi-label transfer learning models to tag each
image with 40 attributes. It is notable that from these CelebA student models,
the attacker will receive responses with 40 facial attributes.

We also use different datasets in our experiments to evaluate how the attack
is influenced by probing inputs:

VOC-Segmentation. Our experiments assume the attacker has obtained the
training dataset for the segmentation task of the VOC2012 challenge [6],
consisting 1,464 samples. Besides, we further assume the image annotation
information (i.e., segmentation and class information) is unavailable.

Random Noise. This dataset simulates an extreme case that the attacker cannot
acquire any realistic image to build up the attack dataset. But the attacker
is assumed to know the input format required by the black-box service
(specifically, 8-bit $224\times 224$ RGB image in our experiments). In this
scenario, they have to synthesize images with randomly generated pixels to
compose the attack dataset. We assume the attacker samples each pixel
following the normal distribution and further normalizes it into an integer
within $[0,255]$.

For all images used in our experiments, they are firstly preprocessed into the
8-bit $224\times 224$ RGB format.

###  4.2 Experiment Setup

Teacher models. We have downloaded pre-trained models AlexNet, DenseNet121,
MobileNetV2, ResNet18, VGG16, and VGG19 from the official repository of
PyTorch. Furthermore, to examine whether our proposed attack can discriminate
teacher feature extractors with the same architecture trained by different
organizations, we have also downloaded an AlexNet model from the PyTorchCV
package repository.444https://github.com/osmr/imgclsmob The detailed
information of the pre-trained models is listed in Table 3. We adopt the whole
convolution part of each pre-trained model as the feature extractor in our
experiments. Particularly, for VGG16 and VGG19, we also adopt pre-trained
fully connected layers, except the last output layer.

Student models. For the Dog-vs-Cats, MNIST, STL10, CIFAR10, and CIFAR100
dataset, we develop multi-class classification students models. As for the
CelebA dataset, we develop multi-label models which annotate the input with 40
binary attributes simultaneously. We build up three student models for each
pre-trained feature extractor individually, by appending different fully
connected layers. The detailed transfer learning setup is described in
Appendix B. Our basic transfer learning setup is the feature extractor
approach. That is, the parameters of the pre-trained feature extractor are
fixed. Furthermore, in Section 4.3.2, we will evaluate the attack performance
when some components of the feature extractor get fine-tuned during the
transfer learning process.

Basic setup. Here we introduce the basic attack setup, followed by most of our
experiments unless otherwise specified. We assume the attacker has owned the
seven pre-trained feature extractors from public resources, and they have
crafted 100 fingerprinting pairs for each teacher feature extractor. We
randomly select the attacker’s probing images from the VOC-segmentation
dataset, which does not overlap with the training dataset of the teacher
models or student models. In Section 4.3.4, we will investigate the attack
effectiveness when no realistic probing image is available. Only the top-1
classification label will be reported by the black-box target, except that the
multi-label classifier trained on the CelebA dataset will return 40 facial
attributes. We assume the input format and input pre-processing module are
transparent to the attacker. For most real-world applications, the input
format is described by the API reference book. Otherwise, it is also feasible
to infer the input format efficiently with some reverse engineering techniques
such as [39].

###  4.3 Evaluation Results

####  4.3.1 Overview

We exhibit our basic experiment results in Figure 2. Each row of the subfigure
represents a fingerprinting vector $v_{\textup{fgpt}}$ against a specific
student model, in which each column reports the matching proportion
$P_{\textup{match}}$ on one teacher feature extractor candidate. For all the
displayed 126 victim student models, the teacher model candidates with the
highest $P_{\textup{match}}$ are consistent with the ground truths (i.e.,
$100\%$ inference accuracy), indicating the effectiveness of our proposed
attacks.

Our first observation is that, in general, the more classes that the transfer
learning task involves, the stronger evidence the attacker can receive to
infer the teacher model. We can see that for the CIFAR100 learning task, the
highest $P_{\textup{match}}$ is significantly higher than other elements of
$v_{\textup{fgpt}}$, which conveys strong evidence that the student model is
likely to be transferred from the corresponding teacher model. In the
meanwhile, the evidence drawn from the Dogs-vs-Cats student models is not so
obvious, as we can see high $P_{\textup{match}}$ on multiple teacher feature
extractor candidates. For instance, when sending fingerprinting pairs
generated from VGG19 to the three black-boxes transferred from the VGG16, the
attacker has achieved $P_{\textup{match}}$ as 0.66, 0.76, and 0.86,
respectively. They are relatively high compared with these on the CIFAR100
targets (0.13, 0.04, and 0.08).

Our second observation is that the student dataset possibly affects the attack
performance. For the four 10-class classification tasks, we can see that the
discrimination between elements of $v_{\textup{fgpt}}$ in an attack against
MNIST student models is less evident than the other three kinds of transfer
learning student models. In fact, the attack performance is subject to the
similarity between the student dataset and the probing dataset. We will study
this phenomenon in Section 4.3.4.

Figure 2: Teacher model fingerprinting vectors w.r.t. different classification
tasks (100 fingerprinting pairs for each teacher model candidate, in the
feature extractor setting). The x-axis represents the candidate teacher model,
while the y-axis represents the actual teacher model. Each row refers to a
$v_{\textup{fgpt}}$, and every three adjacent rows correspond to three
different student models from the same teacher model, e.g., the first three
rows of each subfigure represent three different models transferred from
AlexNet.

####  4.3.2 Impact of Fine-Tuning

(a) Last block fine-tuned

(b) Last two blocks fine-tuned

Figure 3: Impact of fine-tuning. Here we exhibit $v_{\textup{fgpt}}$ against
CIFAR100 classifiers. We can observe that when the last two blocks of the
ResNet18 teacher model components get fine-tuned,
$\text{max}(v_{\textup{fgpt}})$ significantly decreases (to [0.26, 0.17,
0.12]).

Fine-tuning is a commonly adopted strategy for better model performance or
faster model convergence [12]. Here, we explore how fine-tuning techniques
affect the attack.

Setup. Basically, in our experiment, we group pre-trained layers by separating
them by the pooling layers, and we call each separated part a “block”. In our
experiment, we freeze or fine-tune blocks instead of individual layers. We
report the number of blocks for each $\mathcal{F}_{\mathtt{T}}$ in our
experiment in Table 3. We study three cases: frozen feature extractor, the
last transferred block tunable, and the last two transferred blocks tunable.

Results. We have achieved 100% inference accuracy on both cases: where the
last block is fine-tuned and where the last two blocks are fine-tuned. The
results indicate that we can still make accurate inferences against fine-
tuning.

We present fingerprinting vectors against CIFAR100 in Figure 3 as an example
to demonstrate how different fine-tuned neural network layers affect the
attack accuracy. Our key observation is that the eccentricity of
fingerprinting vectors significantly decreases when we fine-tune pre-trained
layers of ResNet18. For example, for attacks against black boxes transferred
from ResNet18 on CIFAR100 tasks (fifth subfigure of Figure 2, 3(a), 3(b)), as
more blocks get fine-tuned, the highest $P_{\textup{match}}$ of each
fingerprinting vector reduces from [0.97, 0.99, 0.99], to [0.71, 0.63, 0.78],
and then [0.26, 0.17, 0.12]. We also statistically study the eccentricity of
$v_{\textup{fgpt}}$ as exhibited in Figure 4. We can find that, except for
targets transferred from ResNet18, fine-tuning high-level pre-trained layers
of other remaining black boxes has little impact on the attack performance.
One possible reason is that $\mathcal{F}_{\mathtt{T}}$ of ResNet18 is a
shallower one with only five blocks. A higher ratio of fine-tuned pre-trained
blocks would probably cause more distinctive changes in the latent feature.
But it is not the only reason, as fine-tuning other shallow pre-trained
$\mathcal{F}_{\mathtt{T}}$ like AlexNet does not affect the attack
performance. After checking the pre-trained model architecture, we find
another potential cause of this phenomenon: ResNet18 contains batch
normalization layers between convolution layers. When fine-tuned on new
datasets, the empirical mean and variance of the batch normalization layer are
updated gradually. It may cause a significant change in the latent feature
representation, breaking the basic assumption that the
$\mathcal{F}_{\mathtt{T}}^{\prime}(\cdot)\approx\mathcal{F}_{\mathtt{T}}(\cdot)$.
Meanwhile, for these black boxes without fine-tuned batch normalization
layers, the changes of $v_{\textup{fgpt}}$ are nearly negligible.

(a)

(b)

Figure 4: Performance on fine-tuned feature extractor layers: (a) average
eccentricity of $v_{\textup{fgpt}}$ w.r.t. seven different
$\mathcal{F}_{\mathtt{T}}$. Each line corresponds to one teacher model. (b)
histogram of eccentricity of $v_{\textup{fgpt}}$. Except attacks against
ResNet18-based student models, the eccentricity only changes slightly when
several pre-trained layers get fine-tuned.

####  4.3.3 Impact of Query Budget

Figure 5: Performance on different query budgets. Each line depicts the attack
performance on a specific transfer learning task. We have achieved 100%
inference accuracy when the query budget reaches 50 (pairs for each teacher
model candidate). On transfer learning tasks with more complex input patterns
and more classes, we get higher eccentricity and entropy heuristics.

Query budget is one of the most primary concerns of a query-based attack. A
low query budget will not only limit the attack cost, but also help the
attacker to keep stealthy.

Setup. In our experiment, we examine the effect of query budget by simulating
attacks that send 1, 2, 5, 10, 20, 50, and 100 fingerprinting pairs for each
teacher candidate.

Results. Figure 5 depicts attack performance on different query budgets, i.e.,
the amount of fingerprinting pairs for each teacher model candidate sent to
the target black box. Therefore, the total amount of queries is $2*\text{query
budget}*|\\{\mathcal{F}_{\mathcal{T}_{j}}\\}|$. Overall, as the query budget
grows, the inference accuracy increases rapidly. When the query budget reaches
50, the total inference accuracy achieves 100%. It suggests that our attack
can avoid possibly high query charge caused by existing model reverse
engineering attacks [36, 28]. We also see that the attack on more complex
tasks (more classes and more complicated inputs) tends to achieve higher
inference accuracy. For example, inference accuracy on CIFAR100 classifiers
exceeds 80% even the query budget is limited to 10.

To gain a deeper insight into how the query budget affects the attack
performance, we also examine the average eccentricity of $v_{\textup{fgpt}}$
and the average entropy of $S_{\textup{match}}$ with the highest
$P_{\textup{match}}$. We find that there possibly exist some correlations
among eccentricity, entropy heuristics, and inference accuracy. Firstly, the
attack against a more complex transfer learning task is likely to bring higher
“inference belief”. For instance, the attack on CIFAR100 classifiers shows
relatively high average eccentricity and entropy compared to other learning
tasks. Secondly, we can observe that as the query budget increases, the
average eccentricity of $v_{\textup{fgpt}}$ and the average entropy of
$S_{\textup{support}}$ also rise. This phenomenon is consistent with our
intuition: more queries are supposed to return more “evidence” to the attacker
(higher entropy), and thus, the attacker has more confidence (higher
eccentricity) to perform the inference, and thus, they have the higher chance
to hit the correct answer (higher accuracy).

####  4.3.4 Impact of Probing Inputs

One of our basic attack assumptions is that any information about the
population of $\mathcal{D}_{\mathtt{S}}$ is unavailable. Previously, we use
probing inputs from a public dataset, VOC-Segmentation, which is not
overlapped with $\mathcal{D}_{\mathtt{S}}$. But what if when there exists a
“stronger” attacker, who has collected probing inputs following the same
distribution with $\mathcal{D}_{\mathtt{S}}$; or there is a “weaker” attacker,
who has no access to any realistic data?

Setup. To this end, we investigate how probing datasets affect attacks with
extra evaluations on other three probing datasets:

  * •

MNIST and CelebA. They are realistic images and share the same distribution
with $\mathcal{D}_{\mathtt{S}}$. We randomly choose 100 probing inputs from
the MNIST and CelebA testing dataset, respectively. In this case, the
$\mathcal{D}_{\mathtt{S}}$ and the input probing dataset are still not
overlapped.

  * •

Random Noise. This is an extreme case where no realistic images are available.
We randomly sample 100 probing inputs from the normal distribution.

Results. Overall, using the VOC-Segmentation dataset in general achieves good
performance, while Random Noise leads to the worst performance. We plot the
attack performance with MNIST, CelebA, and Random Noise probing data in Figure
11, Figure 12, and Figure 13 respectively.

The major obstacle of using Random Noise data as probing inputs is the low
entropy of $S_{\textup{match}}$. We find that the target model often
classifies Random Noise inputs to a specific label, leading to a low entropy
value of $S_{\textup{match}}$. Meanwhile, when we choose MNIST as the probing
dataset, we have achieved high entropy values and 100% inference accuracy
(query budget is 100) on the MNIST-classification targets. We have similar
observations when using CelebA as probing inputs for CelebA-classification
models. This is expected: with sufficient probing inputs from the same
distribution with $\mathcal{D}_{\mathtt{S}}$, the attacker can increase the
variety of output labels and increase the entropy of $S_{\textup{match}}$.
Consequently, they have more information to make reliable inference. We will
further analyze how the probing inputs affects the attack in Section 5.

####  4.3.5 Impact of Probing Features

Section 4.3.2 shows that the fine-tuning technique may influence the inference
accuracy. An idea that comes along naturally is what if synthesizing attack
queries are based on lower-level latent features (a.k.a. probing features),
i.e., $\mathcal{F}_{\mathcal{T}_{j}}$ only consists of few early layers? To
answer this question, we explore the impact of the choice of probing features.

Setup. Recall that we divide pre-trained feature extractors into blocks. We
craft synthetic inputs with respect to latent representations at the end of
different transferred blocks: the last block, the last second block, and the
last third block. It is notable that we freeze all pre-trained components in
this experiment.

Results. For all black boxes, we have achieved 100% inference accuracy.
Nevertheless, the eccentricity declines when we choose probing features from
shallower layers, as depicted by Figure 6. One possible illustration of this
phenomenon is that the earlier layers extract more generic visual features
from the input, such as edges, dots, textures. In contrast, the deep layers
extract more abstract features (like complex patterns, objects, and even
concepts), which is more specific knowledge of the teacher dataset. That means
features from deeper layers carry the information on how the teacher model
refines knowledge from inputs, and therefore they are the key to extract
teacher model fingerprints.

We provide an example to illustrate how latent representations affect
synthetic inputs in Figure 7. We can observe that synthetic inputs generated
by lower-level probing features tend to present more concrete contents, i.e.,
the “plane” pattern presented in the probing input. Meanwhile, synthetic
inputs from high-level features look more like abstract “noisy patterns”. To
understand this phenomenon, we need to review how features are extracted and
propagated layer by layer. On early layers, local patterns of the synthetic
input are captured, while on deep layers, a global and abstract description of
the input is extracted. As a result, only if the synthetic inputs contain
sufficient visual details and local patterns (edges, dots, etc.) can they
activate similar low-level features with the probing inputs. Considering an
extreme case: when we synthesize a similar input with the probing input
(features at BLK0), we tend to produce a copy of the probing input.
Consequently, they are more likely to result in matching responses on
different models, because they are close to the probing inputs on too much
low-level features. This explanation is also consistent with our observations
in Figure 6: the synthetic input crafted on features from ResNet18’s last
third block carries too concrete details of the airplane in the probing input.
It leads to more unwanted matching on victims transferred from other teacher
candidates. Hence, it is critical to extract fingerprints from deep layers.

(a)

(b)

Figure 6: Performance with different probing features: (a) average
eccentricity of $v_{\textup{fgpt}}$ w.r.t. probing features extracted from
three different blocks. Each line represents the exact teacher model. (b)
histogram of eccentricity of $v_{\textup{fgpt}}$. In most cases, the
eccentricity of $v_{\textup{fgpt}}$ decreases when the probing features are
extracted from a shallower layer. Figure 7: An example of how probing features
affect synthetic inputs. The picture at the left top corner is the original
probing input, while the right 3*7 pictures are different synthetic inputs
based on different candidate models and probing features. Each row refers to
synthetic inputs generated by features from different blocks, and each column
refers to synthetic inputs for a specific candidate teacher model. In general,
a synthetic input, which is generated by features from an earlier layer,
carries more visual details of the probing input.

##  5 More Robust Teacher Model Fingerprinting

The experimental results exhibited in Section 4 have shown that the naïve
“one-of-the-best” strategy is effective in most cases. However, we can also
find that it is possible to obtain a high $P_{\textup{match}}$ by a mismatched
feature extractor. For instance, in Figure 2, for MNIST classification student
models, we get $P_{\textup{match}}=0.86$ with an AlexNet candidate against a
VGG19-based student model. It indicates there would exist a considerable
amount of “false matches” in the matching set $S_{\textup{match}}$. One
possible reason for the false-matching problem is that most attack queries
(especially the synthetic input) belong to unrecognizable contents of the
target black-box. In this scenario, the target classifies a pair of
“meaningless” inputs into the same class occasionally, even if they produce
very different features. We can understand this phenomenon in an extreme case:
supposing a black box only returns label “1” to any input. In this case, all
our sent fingerprint pairs will activate seemingly perfect matched responses,
and we will surely get $P_{\textup{match}}=1$ for each candidate teacher
feature extractor. But it is obvious that we cannot find out the actual
teacher model as we are unable to affect the black-box’s behavior.

We design inference strategies leveraging statistical testing methods to
alleviate the impact of false-matching problems. Here we firstly give one
definition for further discussion.

######  Definition 5.1 (Supporting Set).

After removing the elements with the maximum occurrence from
$S_{\textup{match}}$, the remaining elements compose the supporting set
$S_{\textup{support}}=\\{y|\forall y\in
S_{\textup{match}},y\neq\operatorname*{arg\,max}_{a}|\\{a\in
S_{\textup{match}}\\}|\\}$, which will be leveraged as the evidence to infer
the original model.

The relationship between the fingerprinting attack responses, matching set,
and supporting set is explained by Figure 8.

Figure 8: Illustration of the supporting set. The supporting set is the
collection of the remaining elements after removing the elements with the
maximum occurrence from $S_{\textup{match}}$.

Then we consider the extreme case that the student model $\mathcal{S}$ has
zero knowledge about the input. It randomly divides the input space into $c$
classes, which we call a random classifier.555We emphasize that the word
“random” here refers to that the parameters of $\mathcal{S}$ are randomly
configured with respect to $\mathcal{D}_{\mathtt{S}}$, rather than it outputs
label randomly given a specific input. So how much is the possibility to get a
match on a fingerprinting pair? We formalize the following statistical
hypothesis testing: given a candidate teacher model $\mathcal{T}$, a student
model $\mathcal{S}$, a set of fingerprinting pairs
$\\{\langle{\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}}\rangle\\}$ and the
corresponding responses $\\{\langle{y_{i},y_{i}}\rangle\\}$, we make the
following hypothesis

$\displaystyle H_{0}:\textit{$\mathcal{S}$ is a random classifier.}$
$\displaystyle H_{1}:\textit{$\mathcal{S}$ is not a random classifier.}$

We can give a sufficient condition to reject the null hypothesis $H_{0}$ give
a specific significance value $\alpha$.

######  Theorem 1.

Given a $c$-class classification student model $\mathcal{S}$ in top-1 label
exposure, a fingerprinting statistical hypothesis ($H_{0}$, $H_{1}$, as well
as the significance value $\alpha$), the size of the supporting set sufficient
to reject the null hypothesis $H_{0}$ is

$\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil$ (5)

###### Proof.

See Appendix D.

Query Budget | probing: VOCSegmentation | probing: MNIST | probing: CelebA | probing: Random Noise
---|---|---|---|---
inference acc. | $\frac{\textbf{\\#robust}}{\textbf{\\#original}}$ | inference acc. | $\frac{\textbf{\\#robust}}{\textbf{\\#original}}$ | inference acc. | $\frac{\textbf{\\#robust}}{\textbf{\\#original}}$ | inference acc. | $\frac{\textbf{\\#robust}}{\textbf{\\#original}}$
original | robust | original | robust | original | robust | original | robust
1 |  39.68% (50/126) | – (0/0) | 0 (0/126) |  42.06% (53/126) | – (0/0) | 0 (0/126) |  45.24% (57/126) | – (0/0) | 0 (0/126) |  19.84% (25/126) | – (0/0) | – (0/126)
2 |  61.11% (77/126) | – (0/0) | 0 (0/126) |  57.94% (73/126) | – (0/0) | 0 (0/126) |  57.94% (73/126) | – (0/0) | 0 (0/126) |  29.37% (37/126) | – (0/0) | – (0/126)
5 |  84.13% (106/126) | – (0/0) | 0 (0/126) |  69.84% (88/126) | – (0/0) | 0 (0/126) |  80.95% (102/126) | – (0/0) | 0 (0/126) |  42.06% (53/126) | – (0/0) | – (0/126)
10 |  95.24% (120/126) |  100.00% (32/32) |  25.40% (32/126) |  80.95% (102/126) |  100.00% (19/19) |  15.08% (19/126) |  89.68% (113/126) |  100.00% (3/3) |  2.38% (3/126) |  50.79% (64/126) | – (0/0) | – (0/126)
20 |  97.62% (123/126) |  100.00% (97/97) |  76.98% (97/126) |  (84.92% (107/126) |  100.00% (52/52) |  41.27% (52/126) |  96.83% (122/126) |  100.00% (87/87) |  69.05% (87/126) |  57.14% (72/126) |  100.00% (16/16) |  12.70% (16/126)
50 |  100.00% (126/126) |  100.00% (125/125) |  99.21% (125/126) |  90.48% (114/126) |  100.00% (96/96) |  76.19% (96/126) |  99.21% (125/126) |  100.00% (117/117) |  92.86% (117/126) |  62.70% (79/126) |  100.00% (36/36) |  28.57% (36/126)
100 |  100.00% (126/126) |  100.00% (126/126) |  100.00% (126/126) |  96.03% (121/126) |  100.00% (114/114) |  90.48% (114/126) |  100.00% (126/126) |  100.00% (122/122) |  96.83% (122/126) |  65.08% (82/126) |  100.00% (41/41) |  32.54% (41/126)
Table 2: Comparison between the original and robust version of the teacher
model fingerprinting attack. We have achieved 100% inference accuracy on the
remaining inferences after the hypothesis testing ($\alpha=0.01$,
$\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil=7$). Figure 9: Supporting set
size w.r.t. different learning tasks and different probing inputs (query
budget=100, $\alpha=0.01$, the hypothesis testing threshold
$\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil=7$). If the probing inputs
follow a similar distribution with $\mathcal{D}_{\mathtt{S}}$,
$|S_{\textup{support}}|$ is relatively high.

Only when $H_{0}$ is rejected can we make a more robust inference. That is,
the attacker has high confidence (1-$\alpha$) to avoid a wrong inference
against a random classifier.

Here we provide a more robust version of the teacher model fingerprinting
attack: (1) generating fingerprinting pairs $\rightarrow$ (2) selecting the
best candidate $\rightarrow$ (3) if $H_{0}$ is rejected, accepting the
inference result; otherwise, sending more queries and repeating, until
reaching the maximum query budget. We describe the robust version of our
attack in Algorithm 1.

Evalution. We repeat several empirical studies on our robust attack method. We
set the significance value $\alpha$ as 0.01, and thus, the testing hypothesis
threshold $\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil=7$. Table 2
exhibits the robust version of inference results for Section 4.3.4. Overall,
we have achieved 100% inference accuracy on the robust inferences that reject
$H_{0}$ with $\alpha=0.01$, even without realistic datasets (Random Noise as
the probing dataset). We also examine the size of the supporting set on our
inference results in Section 4.3.4 to investigate how probing inputs and
transfer learning tasks affect the attack performance.

Impact of probing inputs. The probing input distribution indeed impacts the
attack performance. From Figure 9, we can see that, with realistic probing
inputs, the proportions of robust inferences are obviously higher than that
with Random Noise. This is consistent with our observation in Section 4.3.4.
Moreover, a dataset that shares a similar distribution with
$\mathcal{D}_{\mathtt{S}}$ is more helpful to reduce the query budget, because
it can increase the variance of the black-box responses. For example, for
MNIST classifiers, the supporting set size $|S_{\textup{support}}|$
significantly rises when we use MNIST as the probing dataset.

Impact of transfer learning tasks. Figure 9 shows that more complex learning
tasks (i.e., those have more complex inputs and more classes) tend to produce
a larger $|S_{\textup{support}}|$. For instance, when attacking CIFAR100
classifiers, most inferences successfully reject $H_{0}$ as their
$|S_{\textup{support}}|$ are over the threshold. On reason is that a complex
learning task is likely to give a higher variance of the responses. This
result is consistent with our observation in Section 4.3.3. To help readers
understand this phenomenon, we can estimate the minimal size of
$S_{\textup{support}}$. Since Equation 5 points out that the minimal
$|S_{\textup{support}}|$ should be
$\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil$, for a $c$-class classifier,
$|S_{\textup{match}}\setminus S_{\textup{support}}|$ should be no less than
$\left\lceil\frac{\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil}{c-1}\right\rceil$.
Therefore, we have

$|S_{\textup{match}}|\geq\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil+\left\lceil\frac{\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil}{c-1}\right\rceil$
(6)

So, according to Equation 6, we can find that when $c$ increases, we will get
a lower bound of the size of the supporting set. It should be pointed out that
this is just a sufficient condition to reject $H_{0}$ for the robust version
of our teacher model fingerprinting attack.

##  6 Case Study: Practical Model Stealing

Figure 10: Model stealing against transfer learning models. The top row
reports the accuracy metric, while the bottom row reports the fidelity metric
of extracted models, respectively. For an extracted model transferred from the
same teacher model with the target (i.e., inferred by a successful teacher
model fingerprinting attack; marked with thicker lines), it has finally
achieved the highest accuracy and fidelity with sufficient queries (#Query
$\geq$ 10,000).

A successful fingerprinting attack can be a stepping stone to further security
or privacy attacks, e.g., adversarial example attacks and model stealing
attacks. In this section, we demonstrate how our attack helps to mount model
stealing.

Setup. The basic model stealing strategy is to firstly identify the teacher
feature extractor of the victim, and then build up a surrogate student model
based on it. Next, the attacker sends queries to the black-box target and
collects the top-1 label responses. Finally, the attacker trains the surrogate
student model with the attack queries and corresponding responses. In our
experiment, we choose four CIFAR-10 classifier victims, which are transferred
from AlexNet, ResNet18, VGG16, and VGG19, respectively. We choose training
images from CIFAR100 as the attack queries (non-overlapping with the target’s
$\mathcal{D}_{\mathtt{S}}$, but sharing a similar distribution). To examine
the enhancement brought by the teacher model fingerprinting attack, we compare
the model stealing performance with four pre-trained models: AlexNet,
ResNet18, VGG16, and VGG19. We assume the victim and the surrogate model share
the same fully connected layer architecture: FC(128) $\rightarrow$ ReLU
$\rightarrow$ BN $\rightarrow$ SF(10). Each surrogate student model is trained
for 20 epochs, with batch size set as 48.

Evaluation metric. We choose two evaluation metrics widely adopted by prior
studies on model stealing attacks [36, 17]:

  * •

Accuracy: accuracy of the extracted model on inputs. It measures how good is
the extracted model to make correct predictions on incoming inputs. This
metric usually lies in the context where an attacker aims to steal the strong
prediction “power” of the victim black-box ML service with only limited
efforts. The primary goal of the accuracy-extraction attack is to make a
profit with the extracted model.

  * •

Fidelity: agreement between the target and the extracted model on inputs [36].
It measures the behavior similarity between the victim model and the extracted
model. This metric usually lies in the context where an attacker aims to
faithfully replicate the “functionality” of the black-box victim, in
particular, these potential vulnerabilities. The primary goal of the fidelity-
extraction attack is to extract a model for further exploitations [17, 7].

We evaluate the extracted models on 10,000 CIFAR10 testing images non-
overlapping with the victim training dataset or the attack query dataset.

Results. Figure 10 shows the model stealing attack performance with different
pre-trained models. We highlight model stealing results after successful
teacher model fingerprinting attacks, i.e., the extracted model and the victim
share the same teacher model, with thicker lines. We can observe that for
model stealing after correctly identifying teacher model, the fidelity is
obviously higher than others, while the extracted model accuracy is also
relatively high. It indicates that our proposed attack can effectively help
improve model stealing attack performance both on accuracy and fidelity.

##  7 Discussions

###  7.1 Possible Countermeasures

Anomaly detector. So far, we have shown that our attack is effective,
requiring minimal knowledge from the target model. That is, the target model
only returns predicted labels instead of posteriors. One way to defend our
attack is to construct an out-of-distribution detector to filter out the
noised samples generated by the attacker before feeding them to the target
model. However, the attacker could also rely on adversarial training
techniques to bypass this out-of-distribution detector. In the future, we will
study other defense mechanisms.

Batch normalization. Based on our findings in Section 4.3.2, tunable batch
normalization layers can change latent representations obviously, even without
modifying other model parameters. But it may also harm the pre-trained model
performance if the batch normalization parameters, i.e., the estimated mean
and standard deviation, change a lot in the transfer learning stage. In the
future, we plan to explore how to trade off defense effectiveness and model
performance.

Input distortion. Another potential solution is to distort inputs by inserting
small random noise or perform image transformations like image cropping and
resizing. Hopefully, it would only slightly affect model performance on
realistic inputs but significantly reduce the inference attack accuracy, since
the latent feature is sensitive to the optimized “noise” pattern in a
synthetic input.

###  7.2 Limitations and Future Work

Language model. In this paper, our empirical studies focus on computer vision
tasks. Recently, the transfer learning technique plays a vital role in a more
broad range of fields. For instance, famous pre-trained language transformers,
like BERT and GPT-3, are widely adopted into various downstream applications,
such as search ranking [15] and medical text mining [21]. We think it is
possible to infer pre-trained language teacher models with the same strategy
proposed in this paper. That is, sending a normal text segment and a
calibrated artificial text segment to the black box, which would induce a pair
of similar latent representations if the victim is transferred from the attack
candidate teacher model. The critical challenge is how to choose the inference
input and craft the synthetic input with specific limitations by NLP tasks. We
plan to extend this work into sequential language models, and devise new
teacher model fingerprinting attacks to infer the pre-trained language
transformer of a black box.

Advanced adversarial attacks. Our results show that we can lower the attack
bound of model stealing attacks based on a successful teacher model
fingerprinting attack. We believe our proposed attack can assist other
advanced adversarial attacks, e.g., black-box adversarial example attack [38]
and membership inference attack [32]. We will explore more types of
adversarial attacks preceded by our attack in real-world settings, to gain a
deeper insight into the privacy and security threats raised by the teacher
model fingerprinting attack.

Fingerprint erase. Our empirical study indicates that the teacher model
fingerprint may still exist even after fine-tuning. Indeed, the teacher model
fingerprint is determined by multiple factors, including model architecture,
parameter initialization, training dataset, stochastic training, etc. In the
feature, we plan to study how different aspects impact the model fingerprint,
and explore new ways to erase teacher model fingerprint from the student model
as fundamental defenses.

##  8 Related Work

Model reverse engineering. Model reverse engineering aims to infer model
parameters, structures, or other model-related information such as hyper-
parameters, by just inspecting model responses to a specific set of attack
queries.

One line is to reproduce mimic models in the parameter level, i.e., they try
to recover the exact model parameters. Tramèr et al. [36] propose two
approaches to steal model parameters via MLaaS APIs: an equation-solving
strategy against logistic regression models and neural networks, and a path-
finding strategy for decision trees. Besides, Carlini et al. [7] propose a
cryptanalytic extraction approach by sending queries at critical points. This
approach relies on the piecewise characteristic feature of ReLU neural
networks. Another line tries to duplicate target models at the functionality
level. Recently, Orekondy et al. [29] propose more advanced attack methods to
steal model parameters. Jagielsk et al. [17] discuss inherent limitations of
the learning-based model stealing strategy, and develop a more practical
functionality-equivalent stealing attack. For transfer learning, Wang et al.
[38] propose a teacher model fingerprinting strategy. Its main idea is to
craft a fingerprinting image to produce a nearly all-zero latent vector on the
victim, when it is derived from the candidate teacher model. Yet, it requires
access to the raw confidence values. In contrast, our attack works well when
only top-1 labels are available. We also propose a robust teacher model
fingerprinting attack to overcome the false-matching problem.

Besides, there are also efforts in inferring structures [28], hyper-parameters
[37], or other model properties [14].

The above studies reveal that there is no such an exact line separating the
white-box and black-box models. Nonetheless, they need numerous queries and
complex computations to change a black-box model into a white-box one,
especially for complicated models like VGG16. When it comes to transfer
learning, we believe our method has the advantage to quickly and efficiently
reverse engineer most parts of the black-box victim—the complex transferred
components, which is hard to achieve by directly combining the prior arts.

Adversarial attacks in the transfer learning setup. Recent studies reveal that
the transfer learning technique may expose ML models to various threats. Zhang
et al. [42] show that adversarial examples become more transferable on a model
trained by fine-tuning. Wang et al. [38] propose an adversarial example attack
by crafting malicious perturbations to activate mimic latent representations
to be classified to the target class. Besides, Yao et al. [40] present a
latent backdoor attack by releasing a malicious teacher model, with which the
carefully mined backdoor will infect the student model. In addition to the
security side, transfer learning also suffers from privacy threats. For
example, Zou et al. [45] reveal that the public knowledge would raise the
membership leakage risk in the transfer learning practice.

In addition, there also exist a broad range of other adversarial attacks
against ML [13, 34, 33, 32, 27, 11, 31, 41, 10, 43, 16], which are out of the
scope of this paper.

##  9 Conclusions

In this paper, we take the first step to investigate the teacher model
exposure threat in the transfer learning context. In particular, we propose a
novel teacher model fingerprinting attack that can efficiently trace back the
origin of a student model. We have empirically studied the attack performance
on various factors across different real-world settings, including teacher
model architecture, transfer learning task, transfer learning strategy, query
budget, attack probing dataset, as well as attack probing features. Our
extensive experiment results demonstrate that our proposed attack can
accurately identify the teacher model even on restrictive attack scenarios,
such as top-1 label exposure and no realistic data available. Furthermore, we
propose a robust teacher model fingerprinting attack to eliminate false
positives from the inference results. Finally, we present a practical model
stealing attack facilitated by our teacher model fingerprinting attack. Our
findings highlight the urgent need for new model confidentiality protection
measures specified for transfer learning.

## References

  * [1] https://modelzoo.co/.
  * [2] https://www.kaggle.com/c/dogs-vs-cats.
  * [3] http://yann.lecun.com/exdb/mnist/.
  * [4] https://www.cs.toronto.edu/~kriz/cifar.html.
  * [5] https://cs.stanford.edu/%7Eacoates/stl10/.
  * [6] http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html.
  * [7] Nicholas Carlini, Matthew Jagielski, and Ilya Mironov.  Cryptanalytic Extraction of Neural Network Models.  In Annual International Cryptology Conference (CRYPTO), pages 189–218. Springer, 2020.
  * [8] Nicholas Carlini and David Wagner.  Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods.  CoRR abs/1705.07263, 2017.
  * [9] Nicholas Carlini and David Wagner.  Towards Evaluating the Robustness of Neural Networks.  In IEEE Symposium on Security and Privacy (S&P), pages 39–57. IEEE, 2017.
  * [10] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz.  GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models.  In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 343–362. ACM, 2020.
  * [11] Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang.  BadNL: Backdoor Attacks Against NLP Models.  CoRR abs/2006.01043, 2020.
  * [12] Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent.  The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training.  In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 153–160. JMLR, 2009.
  * [13] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart.  Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures.  In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 1322–1333. ACM, 2015.
  * [14] Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, and Nikita Borisov.  Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations.  In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 619–633. ACM, 2018.
  * [15] Luyu Gao, Zhuyun Dai, and Jamie Callan.  Understanding BERT Rankers Under Distillation.  In ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR), pages 149–152. ACM, 2020.
  * [16] Xinlei He, Jinyuan Jia, Michael Backes, Neil Zhenqiang Gong, and Yang Zhang.  Stealing Links from Graph Neural Networks.  In USENIX Security Symposium (USENIX Security). USENIX, 2021.
  * [17] Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot.  High Accuracy and High Fidelity Extraction of Neural Networks.  In USENIX Security Symposium (USENIX Security), pages 1345–1362. USENIX, 2020.
  * [18] Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong.  MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples.  In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 259–274. ACM, 2019.
  * [19] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  In International Conference on Learning Representations (ICLR), 2015.
  * [20] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big Transfer (BiT): General Visual Representation Learning.  In European Conference on Computer Vision (ECCV), pages 491–507. Springer, 2020.
  * [21] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 2020.
  * [22] Klas Leino and Matt Fredrikson.  Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference.  In USENIX Security Symposium (USENIX Security), pages 1605–1622. USENIX, 2020.
  * [23] Zheng Li and Yang Zhang.  Membership Leakage in Label-Only Exposures.  In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2021.
  * [24] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep Learning Face Attributes in the Wild.  In IEEE International Conference on Computer Vision (ICCV), pages 3730–3738. IEEE, 2015.
  * [25] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan.  Learning Transferable Features with Deep Adaptation Networks.  In International Conference on Machine Learning (ICML), pages 97–105. JMLR, 2015.
  * [26] Arvind Narayanan and Vitaly Shmatikov.  De-anonymizing Social Networks.  In IEEE Symposium on Security and Privacy (S&P), pages 173–187. IEEE, 2009.
  * [27] Milad Nasr, Reza Shokri, and Amir Houmansadr.  Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning.  In IEEE Symposium on Security and Privacy (S&P), pages 1021–1035. IEEE, 2019.
  * [28] Seong Joon Oh, Max Augustin, Bernt Schiele, and Mario Fritz.  Towards Reverse-Engineering Black-Box Neural Networks.  In International Conference on Learning Representations (ICLR), 2018.
  * [29] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz.  Knockoff Nets: Stealing Functionality of Black-Box Models.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4954–4963. IEEE, 2019.
  * [30] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep Contextualized Word Representations.  In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2227–2237. ACL, 2018.
  * [31] Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang.  Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning.  In USENIX Security Symposium (USENIX Security), pages 1291–1308. USENIX, 2020.
  * [32] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes.  ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models.  In Network and Distributed System Security Symposium (NDSS). Internet Society, 2019.
  * [33] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership Inference Attacks Against Machine Learning Models.  In IEEE Symposium on Security and Privacy (S&P), pages 3–18. IEEE, 2017.
  * [34] Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov.  Machine Learning Models that Remember Too Much.  In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 587–601. ACM, 2017.
  * [35] Qianru Sun, Liqian Ma, Seong Joon Oh, Luc Van Gool, Bernt Schiele, and Mario Fritz.  Natural and Effective Obfuscation by Head Inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5050–5059. IEEE, 2018.
  * [36] Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart.  Stealing Machine Learning Models via Prediction APIs.  In USENIX Security Symposium (USENIX Security), pages 601–618. USENIX, 2016.
  * [37] Binghui Wang and Neil Zhenqiang Gong.  Stealing Hyperparameters in Machine Learning.  In IEEE Symposium on Security and Privacy (S&P), pages 36–52. IEEE, 2018.
  * [38] Bolun Wang, Yuanshun Yao, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao.  With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning.  In USENIX Security Symposium (USENIX Security), pages 1281–1297. USENIX, 2018.
  * [39] Qixue Xiao, Yufei Chen, Chao Shen, Yu Chen, and Kang Li.  Seeing is Not Believing: Camouflage Attacks on Image Scaling Algorithms.  In USENIX Security Symposium (USENIX Security), pages 443–460. USENIX, 2019.
  * [40] Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao.  Latent Backdoor Attacks on Deep Neural Networks.  In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2041–2055. ACM, 2019.
  * [41] Honggang Yu, Kaichen Yang, Teng Zhang, Yun-Yun Tsai, Tsung-Yi Ho, and Yier Jin.  CloudLeak: Large-Scale Deep Learning Models Stealing Through Adversarial Examples.  In Network and Distributed System Security Symposium (NDSS). Internet Society, 2020.
  * [42] Yinghua Zhang, Yangqiu Song, Jian Liang, Kun Bai, and Qiang Yang.  Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning.  In ACM Conference on Knowledge Discovery and Data Mining (KDD), pages 2989–2997. ACM, 2020.
  * [43] Zaixi Zhang, Jinyuan Jia, Binghui Wang, and Neil Zhenqiang Gong.  Backdoor Attacks to Graph Neural Networks.  CoRR abs/2006.11165, 2020.
  * [44] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight.  Transfer Learning for Low-Resource Neural Machine Translation.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1568–1575. ACL, 2016.
  * [45] Yang Zou, Zhikun Zhang, Michael Backes, and Yang Zhang.  Privacy Analysis of Deep Learning in the Wild: Membership Inference Attacks against Transfer Learning.  CoRR abs/2009.04872, 2020.

## Appendix

##  Appendix A Teacher Model Information

In our experiments, we have downloaded pre-trained models from the PyTorch
official repository, including AlexNet, DenseNet121, MobileNetV2, ResNet18,
VGG16, and VGG19. Besides, we have also downloaded another pre-trained AlexNet
from the PyTorchCV repository to elaborate that our proposed attack can
discriminate different teacher models with the same model architecture. The
sources of our obtained teacher models are listed in Table 3.

Model | Provider | $\mathcal{D}_{\mathtt{T}}$ | #BLKs |  URL | MD5 Checksum
---|---|---|---|---|---
AlexNet | PyTorch | ImageNet | 5 |  https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth | aed0662f397a0507305ac94ea5519309
DenseNet121 | PyTorch | ImageNet | 11 |  https://download.pytorch.org/models/densenet121-a639ec97.pth | a7047f0b44515469c3965e85cc31e512
MobileNetV2 | PyTorch | ImageNet | 18 |  https://download.pytorch.org/models/mobilenet_v2-b0353104.pth | f20b50b44fdef367a225d41f747a0963
ResNet18 | PyTorch | ImageNet | 5 |  https://download.pytorch.org/models/resnet18-5c106cde.pth | e5f5fcaec0feff7287f61b7bf461e8b2
VGG16 | PyTorch | ImageNet | 5 |  https://download.pytorch.org/models/vgg16-397923af.pth | 463aeb51ba5e122501bd03f4ad6d5374
VGG19 | PyTorch | ImageNet | 5 |  https://download.pytorch.org/models/vgg19-dcbb9e9d.pth | 92881fe292bd7d2408ecff58a101fd03
AlexNet (PTCV) | PyTorchCV | ImageNet | 5 |  https://github.com/osmr/imgclsmob/releases/download/v0.0.384/alexnetb-1900-55176c6a.pth.zip | 07e23324e570d9e1f10e280a53c509e3
Table 3: Sources of the teacher models used in our experiments.

##  Appendix B Transfer Learning Setup

The basic architecture of our black-box student model is one pre-trained
feature extractor concatenated with fully connected layers. We adopt the whole
convolution part of each pre-trained model listed in Table 3 as the feature
extractor $\mathcal{F}_{\mathtt{T}}$, and then concatenate it with two or
three fully connected layers. Particularly, for VGG16 and VGG19, we also adopt
components expect of the last layer from the pre-trained fully connected
layers, which constitute the last teacher feature extractor block. For each
feature extractor, we build up three different student models individually to
perform a more comprehensive evaluation. For one-label classification tasks,
we use the Softmax activation function at the student model output. While for
the multilabel classification task on CelebA dataset, we use the Sigmoid
activation function at the student model output. The exact student model
architectures are exhibited in Table 4.

In our experiments, we use Adam optimizer [19] to train the student models. We
set the learning rate to 0.001, $\beta_{1}$ to 0.9, $\beta_{2}$ to 0.999, the
batch size to 128, and the epochs to 10. We report the average testing
accuracy of the victim student models in Table 5.

$\mathcal{D}_{\mathtt{S}}$ |  Model Architecture*
---|---
Dogs-vs-Cats |  $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(128) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(2) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(512) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(128) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(2) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(1024) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(256) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(2)
MNIST, STL10, CIFAR10 |  $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(128) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(10) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(512) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(128) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(10) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(1024) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(256) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(10)
CIFAR100 |  $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(512) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(100) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(1024) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(256) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(100) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(4096) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(512) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SF(100)
CelebA |  $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(256) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SG(40) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(1024) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(256) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SG(40) $\mathcal{F}_{\mathtt{T}}$ $\rightarrow$ FC(2048) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ FC(512) $\rightarrow$ ReLU $\rightarrow$ BN $\rightarrow$ SG(40)

  * *

For simplicity, FC($n$) refers to a fully connected layer with $n$ neurons,
SF($n$) refers to a Softmax layer with $n$ outputs, SG($n$) refers to a
Sigmoid layer with $n$ outputs, and BN refers to a batch normalization layer.

Table 4: Student model architectures used in our experiments.

Fine tuning* |  $\mathcal{D}_{\mathtt{S}}$ | Teacher model
---|---|---
AlexNet |  AlexNet (PTCV) |  Dense- Net121 |  Mobile- NetV2 | ResNet18 | VGG16 | VGG19
Fixed | Dogs-vs-Cats | 95.17 | 95.90 | 99.00 | 98.49 | 98.73 | 98.91 | 98.89
MNIST | 99.28 | 99.30 | 99.29 | 99.13 | 98.14 | 96.86 | 96.40
STL10 | 75.55 | 84.14 | 96.80 | 95.05 | 94.95 | 92.27 | 92.25
CIFAR10 | 62.35 | 71.39 | 91.80 | 89.87 | 88.02 | 71.57 | 67.50
CIFAR100 | 30.71 | 40.30 | 68.42 | 64.17 | 60.52 | 29.30 | 32.05
CelebA | 87.65 | 87.95 | 88.27 | 87.73 | 86.33 | 85.59 | 85.62
Last BLK | Dogs-vs-Cats | 95.61 | 96.09 | 98.96 | 98.51 | 98.40 | 98.77 | 98.81
MNIST | 99.24 | 99.28 | 99.26 | 99.23 | 98.10 | 96.65 | 96.33
STL10 | 75.56 | 83.72 | 96.78 | 94.96 | 94.81 | 92.10 | 91.91
CIFAR10 | 61.67 | 71.38 | 91.83 | 89.96 | 87.49 | 72.45 | 67.92
CIFAR100 | 30.68 | 39.51 | 67.20 | 64.32 | 60.07 | 31.39 | 31.30
CelebA | 87.76 | 87.75 | 88.00 | 87.79 | 86.54 | 85.62 | 85.45
Last two BLKs | Dogs-vs-Cats | 95.49 | 95.86 | 98.79 | 98.47 | 98.15 | 98.83 | 98.73
MNIST | 99.22 | 99.30 | 99.21 | 99.21 | 97.56 | 96.86 | 96.65
STL10 | 76.86 | 84.79 | 96.71 | 94.70 | 94.83 | 92.15 | 91.82
CIFAR10 | 62.25 | 71.80 | 91.55 | 89.26 | 87.89 | 72.24 | 68.64
CIFAR100 | 29.38 | 40.31 | 68.49 | 62.86 | 60.14 | 29.80 | 31.62
CelebA | 87.81 | 87.77 | 88.12 | 87.77 | 86.88 | 85.69 | 85.71

  * *

For simplicity, “Fixed” refers to that pre-trained parameters are fixed, “Last
BLK” refers to that the last block of the teacher feature extractor gets
trained, and “Last two BLKs” refers to that the last two blocks of the teacher
feature extractor get trained.

Table 5: Average testing accuracy (%) of the victim student models in our
experiments.

##  Appendix C Attack Performance with MNIST, CelebA and Random Noise as
Probing Dataset

As a supplementary to Section 4.3.4, we plot the attack performance with
MNIST, CelebA and Random Noise as the attack probing dataset in Figure 11,
Figure 12 and Figure 13.

Figure 11: Performance with MNIST images as probing inputs. We have achieved
high average eccentricity of $v_{\textup{fgpt}}$ and average entropy of
$S_{\textup{match}}$ on the MNIST classification task. Figure 12: Performance
with CelebA images as probing inputs. Compared to Figure 11, the three
heuristics obviously increase on the CelebA classification task. Figure 13:
Performance with Random Noise as probing inputs. We have got the worst results
on the three evaluation metrics.

##  Appendix D Proof of Theorem 1

###### Proof.

For a $c$-class classifier $\mathcal{S}$ whose response is only the top-1
label (i.e. $y=\operatorname*{arg\,max}_{i}\mathcal{S}(\mathbf{x})_{i}$),
define

$p_{k}=\int_{\mathbf{x}\in\mathcal{X}}\text{Pr}\left({\operatorname*{arg\,max}_{i}\mathcal{S}(\mathbf{x})_{i}=k}\right),\forall
k\in[1,2,\cdots,c]$ (7)

Without loss of generality, we assume that666In most cases, $p_{c}>0$.
However, here we consider a more general case, where the black box’s output
space may not cover all the $c$ classes.

$1\geq p_{1}\geq p_{2}\geq\cdots\geq p_{c}\geq 0$ (8)

Also noticing that $\sum_{i=1}^{c}p_{i}=1$, we have

$p_{k}\leq\sum_{j=2}^{c}p_{j}=1-p_{1},\forall k\in[2,\cdots,c]$ (9)

$\displaystyle 1=\sum_{i=1}^{c}p_{i}\geq\sum_{i=1}^{k}p_{i}\geq kp_{k}$ (10)
$\displaystyle\Rightarrow p_{k}\leq\frac{1}{k}$

Suppose the attacker generates $N$ fingerprinting pairs
$\\{\langle{\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}}\rangle\\}$ with a set of
probing input $\\{\mathbf{x}_{i}\\}$, and receives $N$ pairs of responses
$\\{\langle{y_{i},y_{i}^{\prime}}\rangle\\}$, among which $M$ pairs satisfy
$y=y^{\prime}$. For simplicity, we further suppose the first $M$
fingerprinting pairs produce the matching responses (i.e.,
$y_{i}=y_{i}^{\prime}$ when $1\leq i\leq M$, and $y_{j}\neq y_{j}^{\prime}$
when $M+1\leq j\leq N$). If the victim is a random classifier, we have

$\displaystyle\text{Pr}\left({\\{y_{i}^{\prime}\\}|\\{\langle{\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}}\rangle\\},\\{y_{i}\\},\mathcal{S}}\right)$
$\displaystyle=\prod_{i=1}^{N}p_{y_{i}^{\prime}}=\prod_{i=1}^{M}p_{y_{i}}\prod_{j=M+1}^{N}(p_{y_{j}^{\prime}})$
(11) $\displaystyle\leq\prod_{i=1}^{M}p_{y_{i}}$
$\displaystyle=\prod_{i\in\\{i|y_{i}=1,i\leq
M\\}}p_{1}\prod_{j\in\\{j|y_{j}\neq 1,j\leq M\\}}p_{y_{j}}$
$\displaystyle\leq\prod_{j\in\\{j|y_{j}\neq 1,j\leq M\\}}p_{y_{j}}$

Then let $K=|\\{i|y_{i}\neq 1,i\leq M\\}|$, we can obtain

$\displaystyle\prod_{j\in\\{j|y_{j}\neq 1,j\leq M\\}}p_{y_{j}}$
$\displaystyle\leq\prod_{j\in\\{j|y_{j}\neq 1,j\leq M\\}}p_{2}=p_{2}^{K}$ (12)
$\displaystyle\leq\left(\frac{1}{2}\right)^{K}$

Therefore, we get

$\text{Pr}\left({\\{y_{i}^{\prime}\\}|\\{\langle{\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}}\rangle\\},\\{y_{i}\\},\mathcal{S}}\right)\leq\left(\frac{1}{2}\right)^{K}$
(13)

Finally, let $\left(\frac{1}{2}\right)^{K}\leq\alpha$, we can derive that

$\displaystyle K\geq\log_{2}\frac{1}{\alpha}$ (14)

That is, if $K\geq\lceil\log_{2}\frac{1}{\alpha}\rceil$ ($K$ is a positive
integer), we can prove that
$\text{Pr}\left({\\{y_{i}^{\prime}\\}|\\{\langle{\mathbf{x}_{i},\mathbf{x}_{i}^{\prime}}\rangle\\},\\{y_{i}\\},\mathcal{S}}\right)\leq\alpha$,
i.e., reject the null hypothesis $H_{0}$ to conclude that $\mathcal{S}$ is not
a random classifier, and hence continue further fingerprinting inference.

According to our definition of supporting set Definition 5.1, there exist two
possible scenarios:

  * •

Scenario 1: class 1 does not appear in the supporting set. By our result
Equation 14, if the size of the supporting set $|S_{\textup{support}}|$ is not
smaller than $\lceil\log_{2}\frac{1}{\alpha}\rceil$, the attacker will reject
$H_{0}$ and continue further inference.

  * •

Scenario 2: class 1 appears in the supporting set. Suppose class $k$ is out of
the supporting set, then we have:

$\displaystyle K$ $\displaystyle=|S_{\textup{support}}|-|\\{i|y_{i}=1,i\leq
M\\}|+|\\{j|y_{j}=k,j\leq M\\}|$ (15)
$\displaystyle\geq|S_{\textup{support}}|\geq\lceil\log_{2}\frac{1}{\alpha}\rceil$

The first inequality is based on that fact that $|\\{j|y_{j}=k,j\leq
M\\}|\geq|\\{i|y_{i}=1,i\leq M\\}|$.

Therefore, when the size of the supporting set satisfies
$|S_{\textup{support}}|\geq\lceil\log_{2}\frac{1}{\alpha}\rceil$, we will
reject the null hypothesis $H_{0}$ and continue further fingerprinting
inference. ∎

######  Remark.

Equation 14 gives an easy to calculate but pretty strict condition. In fact,
we can simply induce a more precise one to reject $H_{0}$:

$\displaystyle\prod_{j\in\\{j|y_{j}\neq 1,j\leq M\\}}p_{y_{j}}$
$\displaystyle\leq\prod_{i=2}^{c}\left(\frac{1}{i}\right)^{|\\{j|y_{j}=i,j\leq
M\\}|}\leq\alpha$ (16)

In the supporting set, we firstly reassign the class label $\\{y_{i}\\}$ from
the most frequent class to the least frequent class with 2 to $c$, and then
check the condition Equation 16.

##  Appendix E Robust Teacher Model Fingerprinting

Input: probing inputs $\\{\mathbf{x}_{i}\\}$, teacher model zoo
$\\{\mathcal{T}_{j}\\}$, black-box API $f$, significance level $\alpha$, query
budget $N$.

Output: inferred teacher model $\mathcal{O}$.

// Initialization.

1 $\mathcal{O}\leftarrow\text{NULL}$, $N_{\textup{support}}\leftarrow 0$,
$P_{\text{max}}\leftarrow 0$;

2 for  _$\mathcal{T}\in\\{\mathcal{T}_{j}\\}$_ do

// Get the candidate feature extractor from the teacher model $\mathcal{T}$.

3 Get $\mathcal{F}_{\mathtt{T}}$ from $\mathcal{T}$;

// Initialization.

4 $S_{\textup{match}}\leftarrow\emptyset$,
$S_{\textup{support}}\leftarrow\emptyset$;

5 for  _$i\in[1,2,\cdots,N]$_ do

// Query generation.

6 repeat

7
$\mathbf{x}_{i}^{\prime}\leftarrow\min_{\mathbf{x}^{\prime}}\|\mathcal{F}_{\mathtt{T}}(\mathbf{x}^{\prime})-\mathcal{F}_{\mathtt{T}}(\mathbf{x})\|_{2}$;

8 until
_$\operatorname*{arg\,max}_{k}\mathcal{T}(\mathbf{x}_{i}^{\prime})_{k}=\operatorname*{arg\,max}_{l}\mathcal{T}(\mathbf{x}_{i})_{l}$_
;

// Send queries.

9 $y_{i}\leftarrow f(\mathbf{x}_{i})$, $y_{i}^{\prime}\leftarrow
f(\mathbf{x}_{i}^{\prime})$;

// Build the matching set.

10 if  _$y_{i}=y_{i}^{\prime}$_ then

11 $S_{\textup{match}}$.insert($y_{i}$);

12

13 end if

14

15 end for

16 $P_{\textup{match}}=|S_{\textup{match}}|/N$;

17 if  _$P_{\textup{match}} >P_{\textup{max}}$_ then

// Update the candidate.

18 $\mathcal{O}\leftarrow\mathcal{F}_{\mathtt{T}}$;

19 $P_{\textup{max}}\leftarrow P_{\textup{match}}$;

// Find the most frequent element.

20 $label_{\textup{freq}}=\text{max\\_freq\\_element}(S_{\textup{match}}$);

// Build the supporting set.

21 for  _$y\in S_{\textup{match}}$_ do

22 if  _$y\neq label_{\textup{freq}}$_ then

23 $S_{\textup{support}}$.insert($y$);

24 end if

25

26 end for

27 $N_{\textup{support}}\leftarrow|S_{\textup{support}}|$;

28

29 end if

30

31 end for

// Statistical hypothesis testing.

32 if  _$N_{\textup{support}}
<\left\lceil\log_{2}\frac{1}{\alpha}\right\rceil$_ then

// Not a robust inference.

33 return NULL;

34 end if

35return $\mathcal{O}$;

Algorithm 1 Robust Teacher Model Fingerprinting