# Fully Exploiting Every Real Sample: SuperPixel Sample Gradient Model
Stealing

Yunlong Zhao1 Xiaoheng Deng1 Yijing Liu2 Xinjun Pei1 Jiazhi Xia1 Wei Chen2
1Central South University 2Zhejiang University
{zhaoyl741, dxh, pei_xinjun, xiajiazhi}@csu.edu.cn,{liuyj86,
<EMAIL_ADDRESS>Corresponding author is Xiaoheng Deng

###### Abstract

Model stealing (MS) involves querying and observing the output of a machine
learning model to steal its capabilities. The quality of queried data is
crucial, yet obtaining a large amount of real data for MS is often
challenging. Recent works have reduced reliance on real data by using
generative models. However, when high-dimensional query data is required,
these methods are impractical due to the high costs of querying and the risk
of model collapse. In this work, we propose using sample gradients (SG) to
enhance the utility of each real sample, as SG provides crucial guidance on
the decision boundaries of the victim model. However, utilizing SG in the
model stealing scenario faces two challenges: 1. Pixel-level gradient
estimation requires extensive query volume and is susceptible to defenses. 2.
The estimation of sample gradients has a significant variance. This paper
proposes Superpixel Sample Gradient stealing (SPSG) for model stealing under
the constraint of limited real samples. With the basic idea of imitating the
victim model’s low-variance patch-level gradients instead of pixel-level
gradients, SPSG achieves efficient sample gradient estimation through two
steps. First, we perform patch-wise perturbations on query images to estimate
the average gradient in different regions of the image. Then, we filter the
gradients through a threshold strategy to reduce variance. Exhaustive
experiments demonstrate that, with the same number of real samples, SPSG
achieves accuracy, agreements, and adversarial success rate significantly
surpassing the current state-of-the-art MS methods. Codes are available at
https://github.com/zyl123456aB/SPSG_attack.

##  1 Introduction

Machine Learning as a Service (MLaaS), enhances efficiency in both work and
daily life [43, 19, 74, 41]. These invaluable MLaaS models have become targets
for malicious users to steal. Model Stealing (MS) [38, 62, 5, 61, 57, 45, 6,
73, 72, 37, 63, 67] involves constructing a proxy model similar to the victim
model by acquiring query results of input samples. Beyond the direct
utilization of the proxy model, malicious users can also generate a series of
attacks based on the proxy model and transfer them to the victim model,
including membership inference attacks [48], adversarial attacks [76], and
model inversion attacks [13], etc.

The basic paradigm of MS is to construct a sample attack set and train a proxy
model through the samples in the attack set and the corresponding query
results of the victim model. Data-free MS is based on generative networks and
uses noise to synthesize artificial images for the training of the proxy
model. Although data-free MS claims that real samples are not needed, in
practical applications that require high-dimension samples, data-free MS still
has the inevitable real sample demand. As shown in Table 1, on the one hand,
high-dimensional inputs would significantly increase the query volume for
data-free MS. For color images of 224x224 pixels, the query volume for data-
free MS could reach tens of millions. On the other hand, due to the inherent
risk of model collapse in GANs [16], training proxy models with data-free MS
is prone to failure. By giving data-free MS a small amount (10k to 20k) of
real samples related to the domain of the victim model as image priors, data-
free MS can reduce the query cost and model collapse risk and improve the
stealing effect. However, acquiring high-quality real samples that meet
specific MLaaS requirements is challenging. Different MLaaS have varying
requirements for input images, and blurred input images can distort MLaaS
query results. Privacy and copyright protections further complicate the
acquisition of high-quality real samples. Additionally, even with a plethora
of real samples, the marginal benefit of each real sample for improving the
proxy model diminishes as the cardinality of real samples increases.
Therefore, expensive real samples should be fully utilized.

Table 1: Results of data-free MS without using real samples and with using 10k
domain-relevant real samples. Results are queries, real samples, the failure
times, and test accuracy (in %), of each method with querying probability. The
failure times are determined by the number of model collapses observed over 10
training runs. we report the other average result computed over 10 runs. The
training strategies and experimental configurations used are described in the
experimental section 4. (1k=1000)

Data-free MS(probability) | CUBS200(0) | Indoor67(0) | CUBS200(10k) | Indoor67(10k)
---|---|---|---|---
queries | failures | accuracy | queries | failures | accuracy | queries | failures | accuracy | queries | failures | accuracy
ZSDB3KD [63] | 5183k | 3 | 48.72 | 5771k | 4 | 51.77 | 1202k$\downarrow$ | 2$\downarrow$ | 51.47$\uparrow$ | 1112k$\downarrow$ | 2$\downarrow$ | 59.41$\uparrow$
DFMS [45] | 4987k | 2 | 51.28 | 4771k | 2 | 55.21 | 1021k$\downarrow$ | 1$\downarrow$ | 55.21$\uparrow$ | 1009k$\downarrow$ | 0$\downarrow$ | 61.11$\uparrow$
EDFBA [73] | 1623k | 2 | 51.46 | 1792k | 1 | 56.27 | 459k$\downarrow$ | 0$\downarrow$ | 55.61$\uparrow$ | 349k$\downarrow$ | 0$\downarrow$ | 61.17$\uparrow$
DS [6] | 4310k | 3 | 50.26 | 4291k | 2 | 54.21 | 1077k$\downarrow$ | 1$\downarrow$ | 54.97$\uparrow$ | 1021k$\downarrow$ | 1$\downarrow$ | 60.27$\uparrow$
DFME [57] | 4102k | 2 | 48.82 | 4671k | 2 | 52.78 | 1489k$\downarrow$ | 1$\downarrow$ | 52.31$\uparrow$ | 1339k$\downarrow$ | 1$\downarrow$ | 58.71$\uparrow$

To make full use of each real sample, our idea is to obtain more information
about the model from real samples. However, for black-box models, it is
challenging to obtain other types of information to train proxy models.

Figure 1: The columns from left to right are grad-CAM [46], grad-CAM++ [8],
Smooth-gradCAM [35], X-gradCAM [14], layer-CAM [26], and SG-map. The neural
network is ResNet34 pre-trained on ILSVRC-2012.

For example, all feature maps from the intermediate layers of the black-box
model [69, 30, 22, 40] are unknowable. Our focus shifts to the input of the
model, specifically, the Sample Gradient (SG) backpropagated from the model’s
output layer to the input sample. SG is used to assist in generating the
interpretability heat map of the model in many interpretability works.
Actually, SG itself contains a lot of model interpretability decision
information. As shown in Figure 8, we compare the heat map generated by
average pooling SG with other CAM methods, and we can see that SG fully
reflects the model’s decision. However, the obstacles for imitating sample
gradients in MS include: (1) The most primitive method to obtain sample
gradients involves perturbing each pixel individually and acquiring query
results for all perturbed images, a process with enormous query costs.
Specially, obtaining sample gradients for a single image requires over a
hundred thousand queries. Furthermore, inputting perturbed images at the pixel
level into the victim model is akin to inputting adversarial images, which
could be easily detected and thwarted by defense mechanisms like Prada [27].
(2) Sample gradients have significant variance due to certain specific or
redundant neurons backpropagating. Then, we introduce a novel SuperPixel [59,
11, 3] Sample Gradient Model stealing (SPSG) to solve the issues. SPSG
comprises two modules: superpixel gradient querying (SPGQ) and sample gradient
purification (SGP). For issue (1), SPGQ module first segments the image into
multiple superpixels based on a segmentation algorithm. Then, it applies
perturbations to these superpixels and queries the output to obtain the sample
superpixel gradients. For issue (2), SGP module eliminates significant
variance from the sample gradients by filtering extremum information and
normalizing, ensuring the extraction of clean and useful gradient information.
Then, purified superpixel gradients are associated with the pixel gradients of
the proxy model to train proxy models.

Our contributions are enumerated as follows:

  * •

We design SPSG to extract the maximum amount of available model information
from each real sample. The superpixel querying module significantly reduces
the query volume required to acquire the tacit knowledge in one sample
gradient from $10^{6}$ to $10^{2}$ while simultaneously evading defenses like
Prada [27]. Meanwhile, the gradient purification module effectively removes
noise from the sample gradients. The effectiveness of the gradient
purification module is further validated through ablation experiments.

  * •

Through various experiments, SPSG enables the proxy model to achieve accuracy,
agreement, and attack success rate substantially surpassing state-of-the-art
algorithms with the same number of real samples. Specifically, when stealing a
resnet34 model trained on CUBS-200 using 20,000 real samples, SPSG achieves an
accuracy of 61.21% and an agreement of 67.48%, significantly outperforming the
second-best method with an accuracy of 56.39% and agreement of 58.44%. To
match the accuracy achieved by SPSG with 10,000 real samples, the second-best
method requires at least 20,000 real samples.

##  2 Related work

###  2.1 Sample Gradient

Sample gradients, obtained through the backpropagation of a model’s final loss
function, depend on the parameters and structure of the neural network. They
are primarily used in adversarial training and model interpretability.

Adversarial Training Based on Sample Gradients. By introducing small
perturbations along the direction of sample gradients, new samples capable of
deceiving the model can be generated. Techniques like FGSM [33] are sensitive
to the sign of sample gradients, whereas FGM [17] normalizes the gradients.
PGD [7] and FreeAT [47] implement multiple iterations with smaller step sizes
on sample gradients, keeping the perturbations within a specified range. YOPO
[71] reduces gradient calculation costs by leveraging the network’s structure,
with perturbations related only to the first layer. FreeLB [77] accumulates
gradients during training, giving perturbations a more directional tendency.

Model Interpretability Based on Sample Gradients. Techniques like ”Saliency
Map” [51] create saliency maps by calculating gradients of input images to
highlight the model’s focus areas in image classification tasks. Guided
Backpropagation [54] helps understand the decision-making process and the
features learned by each convolutional layer. GRAD-CAM [46] and SmoothGrad
[52], while not directly using sample gradients, generate heatmaps using
feature map gradients to show the focused image areas. [12] proposes a method
for explaining black-box models through input perturbations, generating
interpretability masks to illustrate the model’s focus areas.

###  2.2 Superpixel Segmentation

Superpixels were introduced to create image over-segmentation based on
similarity criteria. Algorithms like SLIC [4] efficiently generate superpixels
by clustering pixels in a five-dimensional color and image plane space.
Subsequent methods [11, 59] have improved superpixel segmentation.

###  2.3 Model Stealing

Data-Free Model Stealing. Data-free model stealing techniques [5, 61, 57, 45,
6, 73, 72, 63] do not require any original training data. Attackers generate
synthetic queries, often through prior knowledge or assumptions about the data
distribution, to probe the model and reconstruct its functionality. All data-
free model stealing (MS) inevitably draws on the concept of Generative
Adversarial Networks (GANs). Therefore, there is an unavoidable risk of model
collapse. Given that the querying cost required for a single instance of model
theft is quite high, a collapse during the extraction process would further
increase the querying costs. Additionally, since the training and querying
phases in data-free MS are coupled, each training of a proxy model requires a
new round of queries, which also increases the query volume.

Data-Driven Model Stealing. Data-driven model stealing attacks [38, 62, 37],
utilize real data, allowing for all attack set samples to be queried before
training the proxy model. Therefore, it is not necessary to query the victim
model again with each training of a new proxy model. The real samples can be
domain-irrelevant to the victim model. Although domain-relevant real samples
can achieve certain improvements in the effectiveness of the theft, domain-
irrelevant real samples are more commonly used. Data-driven MS can also
further refine the selection of real samples based on information from the
training process of the proxy model to enhance the stealing effect.

##  3 SuperPixel Sample Gradient Model Stealing

###  3.1 Overview

In the fundamental paradigm of offline Model Stealing (MS), MS begins with the
construction of a query set comprising all input samples and their
corresponding query results. Subsequently, this pre-assembled query set is
utilized to train different proxy models. SPSG, falling under the category of
offline MS, mainly encompasses two distinct modules: SuperPixel Gradient Query
(SPGQ) and Sample Gradient Purification (SGP), as shown in Figure 2.

Figure 2: Four steps of SPSG. The first step is to obtain superpixel gradients
and query results through SGPQ. The second step involves acquiring pixel
gradients and output logits of the proxy model through backpropagation on the
input sample. The third step is to obtain purified superpixel gradients and
simulated superpixel gradients of the proxy model using SGP. The fourth step
involves updating the proxy model based on the loss function. The gray arrow
represents the direction from input to output.

SPGQ is designed for the assembly of the query set. In addition to acquiring
the predictive probabilities or hard labels for each sample’s output, SPGQ is
also adept at obtaining superpixel gradients for each sample at a low query
cost, while simultaneously circumventing adversarial attack monitoring. On the
other hand, SGP is employed for the training of the proxy model. Within this
module, the superpixel sample gradients from the query set undergo a denoising
process. This process ensures the retention of the extremal portions of the
superpixel gradients across each channel of the image. Based on every
superpixel range filtered by the victim model, the pixel gradients of the
proxy model are averaged to obtain the simulated superpixel gradients. In the
final stage, we introduce a novel loss function that establishes a connection
between simulated superpixel gradients and their ground-truth. Through this
connection, the proxy model is effectively trained, culminating in a
comprehensive and robust offline MS framework.

###  3.2 SuperPixel Gradient Query

For black-box models, the finite difference method for calculating pixel
gradients of input images is a primitive yet effective approach. The finite
difference method estimates gradients by applying a small perturbation to the
input sample and observing the resultant output changes. Finite difference
includes forward difference, central difference, and backward difference. In
this paper, we default to using the forward difference. Specifically, for an
input sample $x$ and a small perturbation $\varepsilon$, an approximation of
the gradient for each pixel $i$ in channel $c=1\vee 2\vee 3$ can be calculated
using the following formula:

$q_{i}^{c}=\frac{\partial f}{\partial x}\approx\frac{f(x+\varepsilon\cdot
e_{i}^{c})-f(x)}{\varepsilon}$ (1)

Here, $\varepsilon$ must be sufficiently small to capture the function’s
variations at $x$, yet not too small to avoid numerical precision issues. In
this paper, the value of $\varepsilon$ is set to $1e-5$. $e_{i}$ is a standard
basis vector with only the i-th pixel in channel c as 1. However, pixel-level
forward differences require one perturbed query per input dimension, rendering
this method computationally expensive in high-dimensional input spaces. In
addition, MLaaS can track and store a series of pixel-level perturbed images
of the input. Under normal circumstances, the pairwise distance between
queried images with the same label follows a Gaussian distribution. However,
with the addition of tiny perturbations, the pairwise distance no longer
follows a Gaussian distribution and tends to an extreme distribution. Prada
[27] can detect such changes in distribution and outputs a noisy prediction
result upon detection of change.

Therefore, we propose SuperPixel Gradient Query (SPGQ), whose core idea is to
extend the pixel-level forward difference to the superpixel level. Superpixels
are a concept in image processing and computer vision, referring to the
technique of combining similar pixels into a unified region or cluster. Each
superpixel ${P_{j}}=\bigcup\nolimits_{i}^{{N_{j}}}{{p_{i}}}$ contains adjacent
pixels ${{p_{i}}}$ of $N_{j}$ number, similar in color, brightness, texture,
or other attributes. SPGQ performs forward differences on three different
channels of each superpixel independently. For all pixels within the same
channel of each superpixel, the same disturbance is added:
$E_{j}^{c}=\sum\limits_{i}^{{N_{j}}}{{e_{i}^{c}}}$. The victim model is
queried to calculate the difference and obtain an approximate gradient:

$g_{j}^{c}=\frac{{\partial f}}{{\partial
P_{j}^{c}}}\approx\frac{{f\left({x+\varepsilon
E_{j}^{c}}\right)-f\left(x\right)}}{\varepsilon}$ (2)

Since a superpixel typically contains 900 or more pixels within a channel,
perturbing all these pixels will not produce an extremely small pairwise
distance, which would not affect the Gaussian distribution of the queried
images. Thus, superpixel queries can circumvent Prada attacks.

###  3.3 Sample Gradient Purification

Sample gradients have a significant variance, and numerical discrepancies
between different models can be inherited through the backpropagation process,
manifesting in the sample gradients. superpixel gradient also has the same
drawbacks. Hence, we have formulated a Sample Gradient Purification Mechanism
to mitigate the interference from variance and other extraneous factors.

Concerning the gradient of every queried superpixel channel
$G^{c}=\bigcup\nolimits_{j}^{J}{{g_{j}^{c}}}$, we initially perform a
denoising operation. The core objective of denoising is to preserve the
extreme values of the sample gradients, eliminating non-extreme gradient
values, as the extreme portions encapsulate the focal points of the model.
Additionally, due to the divergent implications of positive gradients
(indicating facilitation of the loss function) and negative gradients
(indicating impediment), it becomes imperative to independently execute
denoising operations on the sets of positive and negative gradients. We
commence by calculating the extreme values of every channel’s positive and
negative gradients:
$g_{+}^{c}=\max\left(g_{j}^{c}\in\\{g_{j}^{c}|g_{j}^{c}\geq 0\\}\right)$ and
$g_{-}^{c}=\max\left(g_{j}^{c}\in\\{g_{j}^{c}|g_{j}^{c}<0\\}\right)$. The
filtered gradients are given by:

$\displaystyle{G_{+}^{c}}=\\{g_{j}^{c}|g_{j}^{c}>\beta
g_{+}^{c}\\}\quad\quad{G_{-}^{c}}=\\{g_{j}^{c}|g_{j}^{c}<\beta g_{-}^{c}\\}$
(3)

In Equation 3, $\beta$ is a predefined hyperparameter within the range
$0<\beta<1$, typically set to 0.5 by default. The Discarded gradient set
${\left({{G^{c}}}\right)^{\prime}}=\left\\{{g_{j}^{c}|g_{j}^{c}\leq\beta
g_{+}^{c}}\right\\}\bigcup{\left\\{{g_{j}^{c}|g_{j}^{c}\geq\beta
g_{-}^{c}}\right\\}}$, is assigned the value of zero. Simultaneously, to avert
inheriting numerical discrepancies from the models and potential exaggeration
or diminution of gradient values due to gradient explosion or vanishing,
normalization of the filtered sample gradients is requisite. The normalization
operation is as follows:

$\displaystyle G_{+}^{c}=\left\\{{\frac{{g_{j}^{c}}}{{g_{+}^{c}}}|g_{j}^{c}\in
G_{+}^{c}}\right\\}\quad\quad
G_{-}^{c}=\left\\{{\frac{{g_{j}^{c}}}{{g_{-}^{c}}}|g_{j}^{c}\in
G_{-}^{c}}\right\\}$ (4)

The gradient values closer to 1 signify a higher degree of facilitation or
impediment of the model’s loss function at that particular pixel position.
Following denoising and normalization, we obtain the purified sample
gradients.

The role of SGP is crucial. We demonstrate the effectiveness of SGP’s
purification in the subsequent ablation study Section 13.

###  3.4 Objective Function for Training

After obtaining the superpixel level gradient of an image in the victim model
$f_{v}$, the proxy model $f_{s}$ cannot directly mimic and learn the dark
knowledge of the superpixel gradient. Firstly, we use a function similar to
the one used for training the victim model to calculate the sample pixel
gradients of the proxy model. Regardless of whether the query result is hard
labels or probabilities, we use a cross-entropy function similar to that used
in training the victim model to calculate the sample gradients for the proxy
model :

$f(y,p)=-\log(y_{p})$ (5)

For equation 5, $y$ is the proxy model’s K-dimensional vector output. $p$ is
the victim model’s predicted label. Therefore, for the pixel-level gradients
$Q=\bigcup\nolimits_{j}^{J}{\bigcup\nolimits_{i}^{{N_{j}}}{{q_{i}}}}={{\partial
f\left({y,p}\right)}\mathord{\left/{\vphantom{{\partial
f\left({y,p}\right)}\partial}}\right.\kern-1.2pt}\partial}x$ obtained through
backpropagation in the white-box proxy model, we adopt a mean coverage method
to obtain the same format of the simulated-superpixel gradient. Specifically,
based on the superpixel partition of the victim model, we take the mean of all
pixel gradients within the same superpixel in the proxy model, replace the
original pixel gradients, and obtain the simulated superpixel gradient:

${Q^{\prime}}=\bigcup\nolimits_{j}^{J}{\bigcup\nolimits_{c}^{3}{\left\\{{\frac{{\sum\nolimits_{i}^{{N_{j}}}{q_{i}^{c}}}}{{{N_{j}}}}|p_{i}^{c}\in{P_{j}}}\right\\}}}$
(6)

Our ultimate objective function or loss function consists of $L_{grad}$ and
initial $L_{prob}$. For hard-label query mode (giving top-1 probability
$\hat{y}$ and predicted label p) and probability query mode (giving
K-dimensional probability $\hat{y}$), $L_{prob}$ is different:

${L_{prob}}\left({y,\hat{y}}\right)=\begin{cases}-y_{p}\bullet\log\left({\hat{y}}\right)-\log(y_{p})\text{if
of hard-label }\\\
-\sum\nolimits_{k}^{K}{\left({{y_{k}}}\right)}\bullet\log\left({{{\hat{y}}_{k}}}\right)\text{if
of probability}\end{cases}$ (7)

Our gradient loss function $L_{grad}$ is set in two parts: the first part
ensures similarity in the gradient values for each superpixel, and the second
part ensures similarity in the overall gradient of the sample. For the first
part, $f_{v}$ and $f_{s}$ having similar gradients for each superpixel is
equivalent to the query results of the image with the perturbed superpixel
being similar. Define $x_{j}^{c}=x+\varepsilon E_{j}^{c}$ and
$G_{all}^{c}=G_{+}^{c}\bigcup{G_{-}^{c}}$. Then, we can get:

$L_{grad,1}=\sum\nolimits_{j}^{J}{\sum\nolimits_{c}^{3}{\left({{L_{prob}}\left({{f_{s}}\left({x_{j}^{c}}\right),{f_{v}}\left({x_{j}^{c}}\right)}\right)|g_{j}^{c}\in
G_{all}^{c}}\right)}}$ (8)

For the second part, we need to calculate the cosine similarity between G and
${Q^{\prime}}$:

$L_{grad,2}=1-\cos\left({G,{Q^{\prime}}}\right)$ (9)

Based on ${L_{grad}}=L_{grad,1}+L_{grad,2}$, the final loss function of the
victim model is:

$L={L_{prob}}\left({{f_{s}}\left(x\right),{f_{v}}\left(x\right)}\right)+{L_{grad}}$
(10)

Through this objective function, the proxy model can effectively mimic the SG
knowledge of the victim model. We document the visual changes of the simulated
superpixel SG of the proxy model during the training process in Supplementary
Material.

##  4 Experiment

Table 2:  The agreement (in %), test accuracy (in %), and queries of each
method with querying probability or hard label. For our model, we report the
average result as well as the standard deviation computed over 10 runs.
(Boldface: the best value.)

Method (probability) | CUB200 (10k) | CUB200 (15k) | CUB200 (20k)
---|---|---|---
Agreement | Acc | Queries | Agreement | Acc | Queries | Agreement | Acc | Queries
ZSDB3KD | 51.47 | 49.32 | 1202k | 52.31 | 50.67 | 1098k | 55.33 | 53.08 | 1021k
DFMS | 55.21 | 53.19 | 1021k | 57.41 | 55.36 | 1003k | 58.44 | 55.98 | 1007k
EDFBA | 55.61 | 53.12 | 459k | 57.67 | 55.69 | 451k | 58.12 | 55.72 | 451k
DS | 54.97 | 53.98 | 1077k | 57.86 | 55.71 | 1021k | 57.04 | 56.39 | 997k
DFME | 52.31 | 50.17 | 1489k | 53.01 | 51.27 | 1344k | 55.27 | 53.18 | 1311k
SPSG(Ours) |  60.71$\pm$0.51 |  55.47$\pm$0.23 | 132k$\pm$0.01k |  65.98$\pm$0.34 |  59.34$\pm$0.52 | 195k$\pm$0.01k |  67.48$\pm$0.21 |  61.21$\pm$0.11 | 271k$\pm$0.01k
Method (probability) | Indoor(10k) | Indoor(15k) | Indoor(20k)
Agreement | Acc | Queries | Agreement | Acc | Queries | Agreement | Acc | Queries
ZSDB3KD | 59.41 | 58.37 | 1112k | 63.61 | 63.07 | 988k | 67.13 | 65.08 | 977k
DFMS | 61.11 | 60.12 | 1009k | 64.58 | 62.26 | 993k | 68.14 | 67.18 | 972k
EDFBA | 61.17 | 60.42 | 349k | 64.17 | 62.12 | 311k | 67.10 | 64.72 | 302k
DS | 60.27 | 59.98 | 1021k | 62.12 | 61.78 | 1010k | 66.04 | 65.91 | 982k
DFME | 58.71 | 58.17 | 1339k | 60.21 | 59.26 | 1294k | 66.27 | 64.16 | 1209k
SPSG(Ours) | 58.81$\pm$0.11 | 57.99$\pm$0.13 | 137k$\pm$0.01k |  64.98$\pm$0.34 |  63.34$\pm$0.12 | 181k$\pm$0.01k |  70.11$\pm$0.21 |  70.27$\pm$0.11 | 267k$\pm$0.01k
Method (probability) | Caltech256 (10k) | Caltech256 (15k) | Caltehch256 (20k)
Agreement | Acc | Queries | Agreement | Acc | Queries | Agreement | Acc | Queries
knockoff | 51.47 | 49.32 | 10k | 52.31 | 50.67 | 15k | 55.33 | 53.08 | 20k
ActiveThief | 55.21 | 53.19 | 10k | 57.41 | 55.36 | 15k | 58.44 | 55.98 | 20k
Black-Box Dissector | 55.61 | 53.12 | 150k | 57.67 | 55.69 | 220k | 58.12 | 55.72 | 300k
InverseNet | 56.19 | 55.32 | 150k | 57.79 | 55.82 | 220k | 58.73 | 56.74 | 300k
SPSG(Ours) |  60.71$\pm$0.51 |  55.47$\pm$0.23 | 132k$\pm$0.01k |  65.98$\pm$0.34 |  59.34$\pm$0.52 | 185k$\pm$0.01k |  70.21$\pm$0.21 |  63.21$\pm$0.11 | 271k$\pm$0.01k
Method (probability) | Diabetic5(10k) | Diabetic5(15k) | Diabetic5(20k)
Agreement | Acc | Queries | Agreement | Acc | Queries | Agreement | Acc | Queries
knockoff | 31.12 | 29.32 | 10k | 34.43 | 32.17 | 15k | 39.56 | 38.17 | 20k
ActiveThief | 32.21 | 31.19 | 10k | 35.32 | 34.61 | 15k | 38.14 | 37.18 | 20k
Black-Box Dissector | 34.81 | 33.27 | 150k | 36.10 | 36.27 | 220k | 40.22 | 39.86 | 300k
InverseNet | 35.67 | 34.02 | 150k | 36.81 | 36.01 | 220k | 41.12 | 40.16 | 300k
SPSG(Ours) |  36.12$\pm$0.31 |  35.25$\pm$0.12 | 122k$\pm$0.01k |  38.13$\pm$0.27 |  37.24$\pm$0.12 | 179k$\pm$0.01k |  42.14$\pm$0.21 |  41.27$\pm$0.11 | 278k$\pm$0.01k
Method (hard-label) | CUB200 (10k) | CUB200 (15k) | CUB200 (20k)
Agreement | Acc | Queries | Agreement | Acc | Queries | Agreement | Acc | Queries
ZSDB3KD | 24.45 | 23.56 | 1299k | 26.33 | 26.01 | 1098k | 31.31 | 29.78 | 931k
DFMS | 26.21 | 25.11 | 1331k | 29.31 | 28.33 | 1003k | 31.24 | 30.18 | 939k
EDFBA | 25.34 | 23.54 | 559k | 27.61 | 26.43 | 490k | 31.12 | 30.72 | 431k
DS | 24.56 | 23.53 | 1237k | 27.34 | 26.78 | 1191k | 30.64 | 29.78 | 902k
knockoff | 21.11 | 19.27 | 10k | 24.49 | 22.76 | 15k | 26.33 | 25.92 | 20k
ActiveThief | 23.21 | 22.89 | 10k | 26.55 | 25.16 | 15k | 27.18 | 26.96 | 20k
Black-Box Dissector | 25.91 | 23.57 | 150k | 27.43 | 26.26 | 220k | 31.59 | 30.46 | 300k
InverseNet | 26.01 | 24.07 | 150k | 26.93 | 26.12 | 220k | 31.43 | 30.97 | 300k
SPSG(Ours) |  26.78$\pm$0.16 |  25.42$\pm$0.23 | 132k$\pm$0.01k |  29.97$\pm$0.34 |  29.84$\pm$0.52 | 195k$\pm$0.01k |  34.66$\pm$0.21 |  34.12$\pm$0.19 | 271k$\pm$0.01k

###  4.1 Experiment Setup

Victim Model. We employ four datasets used in Knockoff for our
experimentation: Caltech256 (256 classes) [18], CUB-200-2011 (200 classes)
[60], Indoor Scenes (67 classes) [42], and Diabetic Retinopathy (5 classes)
[9]. For Diabetic Retinopathy, we strip 200 images from the training set for
each category, forming a test set that in total contains 1000 images. A
resnet34 [20] model trained on these four datasets serves as our victim model.
The training procedure for the victim model mimics that of Knockoff’s victim
model. Specifically, the model is trained for 200 epochs using an SGD
optimizer with a momentum of 0.5 and an initial learning rate of 0.1 that
decays by a factor of 0.1 every 60 epochs. The well-trained victim model is
available for download at Knockoff Code. The victim model has achieved
accuracies of 78.4%, 77.1%, 76.0%, and 59.4% on the datasets mentioned
sequentially above.

Baseline and Attack Dataset. Baselines are categorized based on the necessity
of real data. Data-free baselines include DFME [57], DS [6], DFMS [45], EDFBA
[73], and ZSDB3KD [63], while Knockoff [37], ActiveThief [38], Black-Box
Dissector [62], and InverseNet [15] are Data-driven baselines that require
real data. Notably, DFMS, EDFBA, ZSDB3KD, DS, Black-Box Dissector, inverseNet,
ActiveThief, and Knockoff remain functional even when the query results are
hard labels. For CUB-200-2011 and Indoor Scenes, we conduct comparisons in
data-free MS. Even though data-free MS does not necessitate real data, we
employ publicly available, potentially related real images as weak image
priors for the generator for fairness in comparison. Specifically, for Indoor
Scenes, We use indoor scene images from SUN [64], which are distinct from
Indoor Scenes categories. For CUB-200-2011, We utilize bird images whose
classes are included in NAbird [58] but not present in CUB. In Caltech256 and
Diabetic Retinopathy, comparisons are made for Data-driven MS. Among them, the
number of queries for InverseNet and Black-Box Dissector is not determined by
the number of real samples. To ensure fairness, we set the query volume for
these two methods slightly higher than that of SPSG. ILSVRC-2012 [10] training
set, consisting of approximately 12 million images, serves as our attack
dataset in Data-driven MS. Moreover, we ensure the consistency of real samples
across different methods.

Training Paradigm and Evaluation Metrics. Training paradigms are categorized
based on the usage of data generators. Generally speaking, only data-free MS
necessitates the use of a generator. Under the generator paradigm, the
Generator is configured as BigGAN and trained using Adam with a learning rate
of 0.001, $\beta_{1}$ = 0.5, and $\beta_{2}$ = 0.999. The batch size of BigGAN
is 128. For data-driven MS, proxy models are trained from scratch on the
attack dataset using SGD with a momentum of 0.5, a learning rate of 0.01
(decaying by a factor of 0.1 every 60 epochs), 200 epochs, and a batch size of
64. Evaluation is based on the accuracy of the proxy model on the
corresponding test set and the similarity in predictions between the proxy and
victim models. We also report the success rate of adversarial attacks on the
victim model as indicators of the transferability of the proxy model. The
superpixel segmentation for our method defaults to quickshift [59].

###  4.2 Experiment Results with Data-free MS

We evaluate the accuracy and agreement of the proxy models generated by data-
free baselines and our algorithm under scenarios with 10k, 15k, and 20k real
samples. As illustrated in Table 2, our method almost outperforms all other MS
methods across both metrics. While the performance of data-free MS plateaus
with increasing numbers of real samples, our method demonstrates less
pronounced diminishing returns. Additionally, we document the query volume
required by all algorithms. Our method’s query volume is determined by the
number of queries consisting of an image and its perturbed versions using
different superpixels, whereas the query volume for data-free MS depends on
when the model converges. Our method requires significantly fewer queries than
data-free MS, with our query volume being approximately 25$\%$ of that
required by the most efficient data-free MS. Notably, despite the use of weak
image priors, DFMS and ZSDB3KD sometimes fail to train, severely limiting
their practical applicability.

###  4.3 Experiment Results with Data-driven MS

We assess the accuracy and agreement of the proxy models generated by SPSG and
other data-driven MS methods under scenarios with 10k, 15k, and 20k real
samples. As shown in Table 2, while our method does not minimize query volume,
it yields substantial improvements. The marginal effects of data-driven MS are
not pronounced across the 10k to 20k real samples range. Consequently, we
observe the performance across a broader range of real sample numbers
(100k-200k), finding that our algorithm consistently outperforms others in
terms of accuracy. Specifically, as shown in Figure 5 and 5, at 140k real
samples for Diabetic Retinopathy, Knockoff achieves its peak accuracy of
54.7$\%$, whereas our algorithm surpasses this accuracy at 130k real samples.

Figure 3: Baselines in CUB200

Figure 4: Baselines in Diabetic5

Figure 5: Ablation study

###  4.4 Experiment Results in Hard-label Query

We evaluate all algorithms functional under hard-label queries, where the
query not only returns the predicted label but also the associated confidence.
We argue that providing users with the confidence associated with the
predicted label is more practical for MLaaS. Despite high confidence not
necessarily equating to accuracy, low confidence allows users to disregard the
model’s prediction. Table 2 presents our results on CUB-200-2011, showcasing
our algorithm’s superiority with 10k or more real samples.

###  4.5 Transferability of Adversarial Samples

We assess the transferability of adversarial samples generated on
CUBS-200-2011 test set. The evaluation encompasses the success rates of
adversarial attacks generated from different methods (FGSM, BIM, PGDK), with a
perturbation bound of $\varepsilon$=10/255 and a step size of $\alpha$=2/255.
The adversarial attacks in our experiments are untargeted attacks. In
untargeted attacks, adversarial samples are generated only on images correctly
classified by the attacked model. As Table 3 demonstrates, the adversarial
samples generated by our proxy model exhibit higher transferability to the
victim model, affirming the practical applicability of our method in real-
world scenarios.

Table 3: The ASR(in %) of MS methods with different adversarial attacks on
CUB200. (Boldface: the best value.)

Method | CUBS200 (hard-label) | CUBS200 (probability)
---|---|---
FGSM | BIM | PGD | FGSM | BIM | PGDk
ZSDB3KD | 21.31% | 23.62% | 23.58% | 37.64% | 37.10% | 39.21%
DFMS | 24.34% | 22.93% | 21.22% | 35.22% | 34.33% | 36.11%
EDFBA | 24.21% | 23.72% | 25.51% | 37.88% | 39.09% | 38.51%
DS | 25.21% | 28.77% | 23.51% | 37.88% | 39.09% | 39.51%
knockoff | 21.31% | 24.77% | 22.51% | 35.12% | 36.71% | 37.57%
ActiveThief | 24.34% | 26.77% | 27.52% | 37.88% | 36.02% | 36.29%
Black-Box Dissector | 24.91% | 26.57% | 26.41% | 37.21% | 36.11% | 36.46%
InverseNet | 24.86% | 26.76% | 27.22% | 38.03% | 37.21% | 37.47%
Ours | 25.47% | 29.03% | 29.31% | 38.21% | 39.43% | 39.68%

Table 4: The agreement (in %) and test accuracy (in %) of different segment
methods on CUB-200-2011.

Method | CUBS200 (hard-label) | CUBS200 (probability)
---|---|---
agreements | accuracy | agreements | accuracy
quickshift (132k) | 26.78% | 25.42% | 60.71% | 55.47%
felzenszwalb (1871k) | 30.34% | 28.93% | 63.22% | 59.33%
slic (371k) | 29.21% | 27.72% | 62.28% | 58.09%
Grid (2700k) | 12.31% | 9.72% | 21.68% | 19.09%

(a) queries for CUB200 model

(b) queries for Indoor67 model

(c) queries for CUB200 model

(d) queries for Indoor67 model

Figure 6: The first and second row show the 1000k queries’ image distance
distribution for SPGQ and finite difference queries, respectively.

###  4.6 Model Stealing in Real-World Scenarios

We trained a model on the Oxford 102 Flowers dataset [34] using the Microsoft
Custom Vision [32] service and designated it as a black-box victim model. The
ILSVRC-2012 dataset served as the attack dataset, with the inference results
of the Oxford 102 Flowers test set used as the metric. Hyperparameter settings
were consistent with previous experiments. The victim model’s test accuracy
was 86.34%. As Table 5 demonstrates, compared to the second-best methods using
30k real samples, our method showed a 4.17% increase in test accuracy for the
proxy model. This result indicates that our method possesses stronger
practical applicability in real-world scenarios.

Table 5: Test accuracy of all baselines in Real-World Scenarios.

Method (probability) | Real Sample Number
---|---
10k | 15k | 20k | 25k | 30k
KnockoffNets | 58.45% | 62.12% | 66.48% | 69.12% | 74.50%
ActiveThief | 60.90% | 64.21% | 68.51% | 70.48% | 75.24%
Black-Box Dissector | 60.21% | 64.27% | 67.52% | 69.88% | 73.09%
SPSG | 62.49% | 65.93% | 69.34% | 71.37% | 79.41%

###  4.7 Ablation Study

Resistance to Prada. We document the monitoring of the finite difference query
and SPGQ by Prada, represented by the distribution of image distances. As
shown in Figure 12, Finite difference queries are completely detectable by
Prada, exhibiting a significant deviation from the Gaussian distribution. In
contrast, superpixel queries initially have a few detectable instances but
subsequently evade detection entirely, with their image distribution closely
aligning with the Gaussian distribution.

Figure 7: The first row shows the superpixel SG heatmap for different segment
methods, while the second row shows the purified superpixel SG map for
different segment methods.

Superpixel Segmentation. We employed the superpixel gradients obtained through
queries under quick-shift [59], felzenszwalb [11], slic [3], and grid
segmentation methods. Experimental results in Table 4 indicate that the more
superpixels used, the more apparent the effect of model stealing becomes.
However, regardless of the number of grid pixels divided, the grid
segmentation method demonstrated a poor stealing effect. This is attributed to
the grid segmentation’s disregard for image attributes such as texture and
color, which are closely associated with the model’s decision-making process.

Impact of SGP. We conduct experiments on CUB-200-2011 dataset to compare the
performance of SPSG without SGP and the complete SPSG. The experimental
results shown in Figure 5 reveal a significant degradation in the
effectiveness of SPSG when SGP is omitted. This decline can be attributed to
the retention of gradient variance, which proves to be particularly
detrimental to dark knowledge extraction. In Supplementary Material, we
further explore the applications of SGP, including knowledge distillation.

##  5 Conclusion

SPSG significantly outperforms existing MS algorithms across various datasets,
demonstrating its effectiveness even in hard-label query scenarios. The
success of SPSG in adversarial attacks showcases its practical utility, while
its capability to evade Prada highlights its stealthiness. In essence, SPSG
provides a novel approach to enhancing MS performance by effectively mimicking
additional information from victim models. We hope our proposed method will
encourage proactive measures to protect models against unauthorized access and
theft.
Acknowledgments This work was supported by the National Natural Science
Foundation of China Project (62172449, 62372471, 62172441), the Joint Funds
for Railway Fundamental Research of National Natural Science Foundation of
China (Grant No. U2368201), special fund of National Key Laboratory of Ni&Co
Associated Minerals Resources Development and Comprehensive Utilization(GZSYS-
KY-2022-018, GZSYS-KY-2022-024), Key Project of Shenzhen City Special Fund for
Fundamental Research(JCYJ20220818103200002), the National Natural Science
Foundation of Hunan Province(2023JJ30696), and the Science Foundation for
Distinguished Young Scholars of Hunan Province (NO. 2023JJ10080).

## References

  * [1] Defending against model stealing via verifying embedded external features.
  * [2] How to steer your adversary: Targeted and efficient model stealing defenses with gradient redirection.
  * Achanta et al. [2010] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk.  Slic superpixels.  Technical report, 2010.
  * Achanta et al. [2012] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk.  Slic superpixels compared to state-of-the-art superpixel methods.  _IEEE transactions on pattern analysis and machine intelligence_ , 34(11):2274–2282, 2012.
  * Barbalau et al. [2020] Antonio Barbalau, Adrian Cosma, Radu Tudor Ionescu, and Marius Popescu.  Black-box ripper: Copying black-box models using generative evolutionary algorithms.  _Advances in Neural Information Processing Systems_ , 33:20120–20129, 2020.
  * Beetham et al. [2023] James Beetham, Navid Kardan, Ajmal Mian, and Mubarak Shah.  Dual student networks for data-free model stealing.  _arXiv preprint arXiv:2309.10058_ , 2023.
  * Carlini and Wagner [2017] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In _2017 ieee symposium on security and privacy (sp)_ , pages 39–57. Ieee, 2017.
  * Chattopadhay et al. [2018] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 839–847. IEEE, 2018.
  * Cuadros and Bresnick [2009] Jorge Cuadros and George Bresnick.  Eyepacs: an adaptable telemedicine system for diabetic retinopathy screening.  _Journal of diabetes science and technology_ , 3(3):509–516, 2009.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Felzenszwalb and Huttenlocher [2004] Pedro F Felzenszwalb and Daniel P Huttenlocher.  Efficient graph-based image segmentation.  _International journal of computer vision_ , 59:167–181, 2004.
  * Fong and Vedaldi [2017] Ruth C Fong and Andrea Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In _Proceedings of the IEEE international conference on computer vision_ , pages 3429–3437, 2017.
  * Fredrikson et al. [2015] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart.  Model inversion attacks that exploit confidence information and basic countermeasures.  In _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_ , pages 1322–1333, 2015.
  * Fu et al. [2021] Kun Fu, Quan Jin, Runze Cui, Fei Sha, and Changqing Zhang.  Axiom-based grad-cam: Towards accurate visualization and explanation of cnns.  _Pattern Recognition_ , 110:107638, 2021.
  * Gong et al. [2021] Xueluan Gong, Yanjiao Chen, Wenbin Yang, Guanghao Mei, and Qian Wang.  Inversenet: Augmenting model extraction attacks with training data inversion.  In _IJCAI_ , pages 2439–2447, 2021.
  * Goodfellow et al. [2014a] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_ , pages 2672–2680, 2014a.
  * Goodfellow et al. [2014b] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014b.
  * Griffin et al. [2007] Gregory Griffin, Alex Holub, and Pietro Perona.  Caltech-256 object category dataset.  2007\.
  * Han et al. [2022] Dongming Han, Jiacheng Pan, Rusheng Pan, Dawei Zhou, Nan Cao, Jingrui He, Mingliang Xu, and Wei Chen.  inet: visual analysis of irregular transition in multivariate dynamic networks.  _Frontiers of Computer Science_ , 16:1–16, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * Heo et al. [2019a] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1921–1930, 2019a.
  * Heo et al. [2019b] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 3779–3787, 2019b.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Huang et al. [2022] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu.  Knowledge distillation from a stronger teacher.  _arXiv preprint arXiv:2205.10536_ , 2022.
  * Jiang et al. [2021] Ziqi Jiang, Li Zhang, Chuang Zhang, Chunxia Li, Fei Li, and Kuiyuan Yang.  Layercam: Exploring hierarchical class activation maps for localization.  _IEEE Transactions on Image Processing_ , 30:5875–5888, 2021.
  * Juuti et al. [2019] Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan.  Prada: protecting against dnn model stealing attacks.  In _2019 IEEE European Symposium on Security and Privacy (EuroS &P)_, pages 512–527. IEEE, 2019.
  * Kariyappa and Qureshi [2020] Sanjay Kariyappa and Moinuddin K Qureshi.  Defending against model stealing attacks with adaptive misinformation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2020.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lee et al. [2018] Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song.  Self-supervised knowledge distillation using singular value decomposition.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 335–350, 2018.
  * Liu et al. [2021] Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang.  Exploring inter-channel correlation for diversity-preserved knowledge distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8271–8280, 2021.
  * [32] Microsoft Corporation.  Microsoft custom vision service.  https://customvision.ai.
  * Miyato et al. [2016] Takeru Miyato, Andrew M Dai, and Ian Goodfellow.  Adversarial training methods for semi-supervised text classification.  _arXiv preprint arXiv:1605.07725_ , 2016.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes, 2008.
  * Omeiza et al. [2019] Daniel Omeiza, Simon Speakman, Celia Cintas, and Komminist Weldemariam.  Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models.  In _2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS)_, pages 272–279. IEEE, 2019.
  * Orekondy et al. [2019a] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz.  Prediction poisoning: Towards defenses against dnn model stealing attacks.  _arXiv preprint arXiv:1906.10908_ , 2019a.
  * Orekondy et al. [2019b] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz.  Knockoff nets: Stealing functionality of black-box models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4954–4963, 2019b.
  * Pal et al. [2020] Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy.  Activethief: Model extraction using active learning and unannotated public data.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 865–872, 2020.
  * Park et al. [2019] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3967–3976, 2019.
  * Passban et al. [2021] Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, and Qun Liu.  Alp-kd: Attention-based layer projection for knowledge distillation.  In _Proceedings of the AAAI Conference on artificial intelligence_ , pages 13657–13665, 2021.
  * Peng et al. [2023] Yanni Peng, Xiaoping Fan, Rong Chen, Ziyao Yu, Shi Liu, Yunpeng Chen, Ying Zhao, and Fangfang Zhou.  Visual abstraction of dynamic network via improved multi-class blue noise sampling.  _Frontiers of Computer Science_ , 17(1):171701, 2023.
  * Quattoni and Torralba [2009] Ariadna Quattoni and Antonio Torralba.  Recognizing indoor scenes.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 413–420. IEEE, 2009.
  * Ribeiro et al. [2015] Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz.  Mlaas: Machine learning as a service.  In _2015 IEEE 14th international conference on machine learning and applications (ICMLA)_ , pages 896–902. IEEE, 2015.
  * Romero et al. [2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In _ICLR_ , 2015.
  * Sanyal et al. [2022] Sunandini Sanyal, Sravanti Addepalli, and R Venkatesh Babu.  Towards data-free model stealing in a hard label setting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15284–15293, 2022.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * Shafahi et al. [2019] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein.  Adversarial training for free!  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Shokri et al. [2017] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In _2017 IEEE symposium on security and privacy (SP)_ , pages 3–18. IEEE, 2017.
  * Shrikumar et al. [2016] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje.  Not just a black box: Learning important features through propagating activation differences.  _arXiv preprint arXiv:1605.01713_ , 2016.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Simonyan et al. [2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Smilkov et al. [2017] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  _arXiv preprint arXiv:1706.03825_ , 2017.
  * Springenberg et al. [2014a] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for simplicity: The all convolutional net.  _arXiv preprint arXiv:1412.6806_ , 2014a.
  * Springenberg et al. [2014b] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for simplicity: The all convolutional net.  _arXiv preprint arXiv:1412.6806_ , 2014b.
  * Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  In _International conference on machine learning_ , pages 3319–3328. PMLR, 2017.
  * Tian et al. [2019] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  _arXiv preprint arXiv:1910.10699_ , 2019.
  * Truong et al. [2021] Jean-Baptiste Truong, Pratyush Maini, Robert J Walls, and Nicolas Papernot.  Data-free model extraction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4771–4780, 2021.
  * Van Horn et al. [2015] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 595–604, 2015.
  * Vedaldi and Soatto [2008] Andrea Vedaldi and Stefano Soatto.  Quick shift and kernel methods for mode seeking.  In _Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part IV 10_ , pages 705–718. Springer, 2008.
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  Technical report, 2011.
  * Wang et al. [2021] Wenxuan Wang, Bangjie Yin, Taiping Yao, Li Zhang, Yanwei Fu, Shouhong Ding, Jilin Li, Feiyue Huang, and Xiangyang Xue.  Delving into data: Effectively substitute training for black-box attack.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4761–4770, 2021.
  * Wang et al. [2022] Yixu Wang, Jie Li, Hong Liu, Yan Wang, Yongjian Wu, Feiyue Huang, and Rongrong Ji.  Black-box dissector: Towards erasing-based hard-label model stealing attack.  In _European Conference on Computer Vision_ , pages 192–208. Springer, 2022.
  * Wang [2021] Zi Wang.  Zero-shot knowledge distillation from a decision-based black-box model.  In _International Conference on Machine Learning_ , pages 10675–10685. PMLR, 2021.
  * Xiao et al. [2016] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.  Sun database: Exploring a large collection of scene categories.  _International Journal of Computer Vision_ , 119:3–22, 2016.
  * Yang et al. [2021] Jing Yang, Brais Martinez, Adrian Bulat, Georgios Tzimiropoulos, et al.  Knowledge distillation via softmax regression representation learning.  International Conference on Learning Representations (ICLR), 2021.
  * Yim et al. [2017] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4133–4141, 2017.
  * Yu et al. [2020] Honggang Yu, Kaichen Yang, Teng Zhang, Yun-Yun Tsai, Tsung-Yi Ho, and Yier Jin.  Cloudleak: Large-scale deep learning models stealing through adversarial examples.  In _NDSS_ , 2020.
  * Zagoruyko and Komodakis [2016a] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  _arXiv preprint arXiv:1612.03928_ , 2016a.
  * Zagoruyko and Komodakis [2016b] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  _arXiv preprint arXiv:1612.03928_ , 2016b.
  * Zagoruyko and Komodakis [2016c] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ , 2016c.
  * Zhang et al. [2019] Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong.  You only propagate once: Accelerating adversarial training via maximal principle.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Zhang et al. [2022a] Jie Zhang, Chen Chen, and Lingjuan Lyu.  Ideal: Query-efficient data-free learning from black-box models.  In _The Eleventh International Conference on Learning Representations_ , 2022a.
  * Zhang et al. [2022b] Jie Zhang, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Lei Zhang, and Chao Wu.  Towards efficient data free black-box adversarial attack.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15115–15125, 2022b.
  * Zhizhong et al. [2024] Huang Zhizhong, Dai Mingliang, Zhang Yi, Zhang Junping, and Shan Hongming.  Point, segment and count: A generalized framework for object counting.  In _CVPR_ , 2024.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2921–2929, 2016.
  * Zhou et al. [2018] Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang.  Transferable adversarial perturbations.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 452–467, 2018.
  * Zhu et al. [2019] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu.  Freelb: Enhanced adversarial training for natural language understanding.  _arXiv preprint arXiv:1909.11764_ , 2019.

Supplementary Material

##  6 Why does Sample Gradient contain model information?

Figure 8: Partial visualization results of grad-CAM [46], grad-CAM++ [8],
Smooth-gradCAM [35], X-gradCAM [14], layer-CAM [26], and SG-map. The neural
network is ResNet34 pre-trained on ILSVRC-2012.

’Dark knowledge’ is a term used to describe knowledge that is implicitly
embedded in a model but doesn’t manifest directly, such as predictive logits.
By learning ’dark knowledge’, a surrogate model can inherit the
characteristics and mimic the functionalities of the victim model. In prior
work, sample gradients have been interpreted as a reflection of a model’s
local sensitivity to a specific input, guiding the perturbation direction in
adversarial attacks. However, due to the inclusion of variance, instability,
and a lack of interpretability, sample gradients are rarely considered as a
form of dark knowledge. Numerous studies [75, 46, 53, 49, 52, 55, 51] have
attempted to utilize sample gradients to aid interpretability, often through
altering the model, employing the gradient of the model’s feature maps, or
introducing additional inputs to propose interpretability methods. These
methods fall short of establishing the interpretability of the original sample
gradients. In this section, we introduce SG-Map, a method for interpreting
sample gradients, designed without modifying the original model architecture,
utilizing feature maps, or adding any additional inputs. We demonstrate that
the sample gradients, processed and visualized as heatmaps, exhibit
interpretability comparable to Grad-CAM.

For an individual input image, after backpropagation through the loss
function, each pixel is assigned a gradient value, collectively forming the
sample gradient. The SG-Map algorithm initiates by preprocessing the sample
gradients: taking the absolute value of the gradient for each pixel and
normalizing the pixels in each channel independently. This preprocessing
ensures that the sample gradients meet the requirements for image display and
eliminates numerical discrepancies in the sample gradients, which are tied to
the parameter values of all neurons in the model and do not accurately reflect
the model’s decision-making characteristics. Channel-wise normalization is
preferred over whole-image normalization due to the more substantial inter-
pixel connections within channels than between them. SG-Map then combines the
pixel gradients from the three different channels according to the
specifications for a grayscale image, resulting in a single-channel sample
gradient. In a crucial final step, we apply average pooling to this single-
channel sample gradient, mitigating the impact of erratic behaviors from
specific instances of the model on the pixel gradients. The resulting sample
gradient is then presented as a heatmap. The visualization result is shown in
Figure 8. Comparing SG-Map with CAM methods, we observe that SG-Map focuses on
similar pixel locations, reflecting the model’s sensitivity and attention
allocation across different areas. Unlike CAM methods, the visualization of
sample gradients through SG-Map provides a more stringent expression of
sensitivity, manifesting as more concentrated yet precise high-temperature
areas in the heatmap. Our proposed SG-Map thereby conclusively demonstrates
that sample gradients encapsulate deep-seated information of the model,
qualifying as a form of dark knowledge that can guide the training of
surrogate models.

##  7 More Experiment Result

###  7.1 Hard-label experiments of Indoor-Scene

Under the query setup with hard labels, we use baselines to steal the resnet34
model trained on Indoor-Scenes. As shown in Table 8, SPSG still maintains the
highest performance metrics.

Figure 9:

###  7.2 Experiments on different Proxy’s architecture

In the main text, we default to using the same proxy architecture as the
victim, that is, resnet34. However, in practice, we cannot obtain information
about the victim’s model. Therefore, we experiment with different neural
network architectures as proxys. The experimental results, as shown in Table
9, indicate that SPSG can effectively extract the performance of the victim
across various neural network architectures. Due to the different performance
ceilings inherent to each neural network architecture, the results present
varying degrees of difference.

###  7.3 Impact of hyperparameter $\beta$

$\beta$ is used to remove gradients outside of extreme values. The larger the
$\beta$, the more gradients are removed, indicating a stricter selection of
extremes. We conduct experiments under different $\beta$ values. When the
value of $\beta$ is small, more gradient variance is introduced, leading to a
decrease in the proxy model’s performance. When $\beta$ value is larger, there
are hardly any superpixel gradients left. When $\beta$ is at its maximum, the
sample gradient contains only one superpixel gradient. When $\beta$ is greater
than 0.8, the performance of the proxy model remains at a lower level. This is
because at this point, what is left are the most representative superpixel
gradients, so the performance of the proxy model remains unchanged as it
always mimics the most important superpixel gradients. when $\beta$ is 0.5,
SPSG gets the best performance. The visulaization result is shown in Figure 9

###  7.4 Offline Training of SPSG

We observe the changes in simulated-superpixel gradients of samples during the
offline training process of the proxy model, as shown in Figure 11. As the
training epochs increase, the similarity between the pseudo-superpixel
gradients obtained from the mean of model backpropagation pixel SG and the
superpixel gradients queried from the victim model gets higher and higher.
More similar sample gradients indicate that our loss function setting is
reasonable. The proxy model sufficiently learns SG knowledge of the victim
model.

###  7.5 Impact of Sample Selection Strategy

Our method does not conflict with sample selection strategies. Therefore, we
compare the performance of SPSG, knockoff, and Black Dissector under two
different sample selection strategies. These two strategies are the
Reinforcement learning strategy [37] and the K-center strategy [38], as shown
in Table 10. Both strategies improve the performance of MS to varying degrees.
SPSG achieves the highest accuracy and similarity under both sample selection
strategies. It is important to note that in the main text, we have already
found that SPSG also obtains the best performance compared to other methods
under a random sample selection strategy.

###  7.6 Ability to evade the SOTA defense method.

We conducted experiments with protection measures similar to [62] including
Adaptive Misinformation [28], Prediction Poisoning [36], Gradient
Redirection[2], External Feature[1]. The victim model is ResNet34 model
trained on the CUBS-200-2011 dataset. The real sample number is 20k. The
attack set employed ILSVRC-2012. As shown in Table 6, SPSG demonstrates
significantly higher resistance to these two defenses compared to other
methods.

Table 6:  The larger the threshold, the better the defense effect (0.0 means
no defense). ”False” and ”True” respectively correspond to evading monitoring
and being detected by monitoring.

Method | No defence | AM | GR | EF | PP
---|---|---|---|---|---
Threshold | - | 0.5 | - | - | 0.5
KnockoffNets |  54.21$\pm$0.11 |  49.13$\pm$0.28 |  47.17$\pm$0.13 | True |  49.22$\pm$0.11
ActiveThief |  55.24$\pm$0.12 |  50.12$\pm$0.11 |  47.71$\pm$0.21 | True |  49.11$\pm$0.21
Black-Box Dissector |  56.98$\pm$0.21 |  51.21$\pm$0.31 |  48.81$\pm$0.14 | False |  49.03$\pm$0.28
Inversenet |  55.17$\pm$0.19 |  49.21$\pm$0.73 |  51.28$\pm$0.31 | False |  48.72$\pm$0.49
DFMS |  52.17$\pm$0.32 |  53.27$\pm$0.21 |  47.79$\pm$0.32 | False |  49.92$\pm$0.09
EDFBA |  55.32$\pm$0.21 |  43.22$\pm$0.13 |  49.82$\pm$0.34 | False |  48.87$\pm$0.73
DS |  51.33$\pm$0.12 |  54.33$\pm$0.37 |  51.11$\pm$0.94 | True |  50.01$\pm$0.56
DFME |  53.28$\pm$0.23 |  52.26$\pm$0.61 |  50.87$\pm$0.34 | False |  51.06$\pm$0.72
SPSG |  61.33 $\pm$0.03 |  54.95 $\pm$0.22 |  52.15 $\pm$0.34 | False |  59.02 $\pm$0.41
victim model | 77.10 | 71.29 | 74.16 | - | 71.46

###  7.7 More study about SPGQ

We investigate the effectiveness of SPGQ and finite difference query methods.
We compare the similarity of the sample gradients obtained by different
methods to the real sample gradients. We use the average pair-wise distance to
each real sample gradient as the evaluation metric and record the average
number of queries required to query a sample. For superpixel gradients, we
compare them by averaging the real samples within the corresponding
superpixels. As shown in Table 7, our method exhibits a smaller distance
compared to the finite difference method, indicating a higher similarity.

Result | Gradient Query Method
---|---
Finite Difference | SPGQ(QuickShift) | SPGQ(Slic) | SPGQ(Felzenszwalb) | Grid Query
Distance | 2.987 | 2.178 | 2.089 | 2.078 | 16.679
Queries | 150528 | 137 | 256 | 457 | 900

Table 7: The effectiveness of SPGQ and finite difference query methods.

##  8 More study about SGP

Empirical validation of SGP’s efficacy is demonstrated through two categories
of experiments. On one hand, knowledge distillation experiments were
conducted. When the student model distills unpurified sample gradients and
logits knowledge, the resultant accuracy exhibits a decline due to irregular
variance in the sample gradients, as compared to training without
distillation. Conversely, distillation using purified sample gradients in
conjunction with logits culminates in accuracy surpassing that achieved by
distilling logits alone. On the other hand, T-SNE visualization was employed,
concatenating the model-extracted sample features with the purified sample
gradients. Comparative analysis reveals that purified sample gradients
significantly enhance the final visualization outcome, as opposed to scenarios
involving no concatenation or concatenation with unpurified sample gradients.
The aforementioned experiments collectively attest to the effectiveness of the
purification mechanism in eliminating variance from sample gradients.

###  8.1 knowledge distillation

We conducted experiments on image classification knowledge distillation. As
shown in Figure 10, the sample gradients obtained from passing the samples
through the teacher and student models are processed by SGP and then
associated through the loss function for distillation. We selected ICKD [31],
Overhaul [21], AT [68], FitNet [44], and FSP [66], KD [23], RKD [39], DIST
[25], SRRL [65], and CRD [56] as the baselines. Initially, we compared the
effects of each baseline with or without SGKD used individually in CIFAR100
[29]. The training strategies for CIFAR100 is shown in Table 11. We selected a
series of teacher-student model combinations from VGG [50], ResNet [20], and
their variants [70]. CIFAR100 Experimental Results: As depicted in Table 12,
using SGP individually results in improved accuracy for student models. To
further examine the influence of SGP on the knowledge distillation task, we
compare the accuracy of the student model with and without SGP on the CIFAR100
dataset. As demonstrated in Table 13, the considerable numerical difference
between the original sample gradients of the student and teacher hinders the
accurate transfer of the teacher model’s dark knowledge to the student model.
This results in a decrease in the student model’s performance. With SGP, the
student model can effectively learn the teacher’s dark knowledge.

###  8.2 T-SNE visualization

SGP allows pixel-level sample gradients to possess more class information. We
first train a resnet34 on CIFAR-10. Then, we obtain the test sample feature
vectors through the final layer before the output of resnet34. The feature
vectors are visualized using T-SNE. Next, we concatenate the purified sample
gradients or the original sample gradients behind the feature vectors and
visualize them again. The four visualization results show that the sample
gradients purified by SGP can effectively aid in classification. In contrast,
the original sample gradients, containing variance and having low
informational content, provide no benefit to the representation of feature
vectors. The result is shown in Figure 12.

Table 8:  The agreement (in %), test accuracy (in %), and queries of each
method with hard label. For our model, we report the average result as well as
the standard deviation computed over 5 runs. (Boldface: the best value.)

Method (hard-label) | Indoor (10k) | Indoor (15k) | Indoor (20k)
---|---|---|---
Agreement | Acc | Queries | Agreement | Acc | Queries | Agreement | Acc | Queries
ZSDB3KD | 27.55 | 26.43 | 1109k | 29.52 | 30.07 | 1002k | 34.21 | 33.71 | 1229k
DFMS | 28.75 | 27.13 | 1321k | 30.12 | 29.35 | 993k | 34.23 | 33.15 | 989k
EDFBA | 27.56 | 26.55 | 345k | 30.34 | 29.48 | 477k | 34.12 | 33.72 | 531k
DS | 27.52 | 26.55 | 1200k | 30.53 | 29.88 | 1090k | 35.24 | 34.18 | 996k
knockoff | 25.31 | 23.66 | 10k | 27.19 | 25.73 | 15k | 31.23 | 29.93 | 20k
ActiveThief | 25.01 | 24.19 | 10k | 27.59 | 26.13 | 15k | 30.98 | 30.11 | 20k
Black-Box Dissector | 25.91 | 23.57 | 20k | 27.43 | 26.26 | 30k | 31.59 | 30.46 | 40k
SPSG(Ours) |  27.86$\pm$0.16 |  26.79$\pm$0.21 |  132k$\pm$0.01k |  31.43.27$\pm$0.34 |  30.29$\pm$0.13 |  195k$\pm$0.01k |  38.27$\pm$0.34 |  36.32$\pm$0.13 |  371k$\pm$0.01k

Table 9:  The agreement (in %) and test accuracy (in %)s of each method with
different proxy architecture [20, 50, 24] in CUBS-200-2011. The real sample
number is 20k. For our model, we report the average result as well as the
standard deviation computed over 5 runs. (Boldface: the best value.)

Method (probability) | ResNet-18 | ResNet-50 | VGG-16 | DenseNet
---|---|---|---|---
Agreement | Acc | Agreement | Acc | Agreement | Acc | Agreement | Acc
ZSDB3KD | 48.55 | 47.23 | 51.51 | 50.04 | 48.11 | 47.61 | 51.23 | 50.93
DFMS | 49.55 | 48.11 | 51.46 | 50.25 | 48.63 | 47.85 | 52.13 | 51.93
EDFBA | 49.76 | 48.55 | 51.32 | 50.11 | 48.72 | 47.84 | 51.13 | 49.98
DS | 48.72 | 46.59 | 50.77 | 50.23 | 48.95 | 48.22 | 50.33 | 49.71
knockoff | 45.39 | 43.98 | 47.28 | 45.63 | 49.78 | 48.23 | 50.21 | 49.28
ActiveThief | 47.01 | 46.19 | 48.59 | 46.19 | 50.18 | 50.11 | 50.23 | 49.13
Black-Box Dissector | 46.77 | 45.87 | 48.49 | 47.56 | 50.19 | 50.16 | 51.23 | 50.93
SPSG(Ours) |  49.96$\pm$0.16 |  49.70$\pm$0.19 |  52.27$\pm$0.34 |  51.39$\pm$0.19 |  51.21$\pm$0.14 |  51.12$\pm$0.11 |  52.71$\pm$0.34 |  52.32$\pm$0.13

Table 10:  The agreement (in %) and test accuracy (in %) of each method with
different sample selection Strategies in CUBS-200-2011. For our model, we
report the average result as well as the standard deviation computed over 5
runs. (Boldface: the best value.)

Method (probability) | CUBS200(10k) | CUBS200(15k) | CUBS200(20k) | CUBS200(25k)
---|---|---|---|---
Agreement | Acc | Agreement | Acc | Agreement | Acc | Agreement | Acc
knockoff (K-center) | 55.65 | 52.71 | 59.32 | 55.71 | 61.77 | 57.61 | 63.21 | 61.93
knockoff (Reinforce) | 54.37 | 50.11 | 56.76 | 54.21 | 59.67 | 57.15 | 61.13 | 59.92
Black-Box Dissector (K-center) | 57.76 | 53.53 | 59.38 | 56.47 | 61.22 | 58.81 | 63.10 | 62.99
Black-Box Dissector (Reinforce) | 55.71 | 52.12 | 58.71 | 56.23 | 61.95 | 59.11 | 61.31 | 60.71
SPSG (K-center) |  59.27$\pm$0.11 |  56.81$\pm$0.11 |  61.49$\pm$0.17 |  60.56$\pm$0.14 |  63.17$\pm$0.14 |  62.16$\pm$0.12 |  65.21$\pm$0.31 |  63.93$\pm$0.11
SPSG (Reinforce) |  59.96$\pm$0.16 |  56.70$\pm$0.19 | 61.27 |  60.39$\pm$0.19 |  63.23$\pm$0.14 |  62.19$\pm$0.11 |  64.11$\pm$0.34 |  62.32$\pm$0.13

Figure 10: The framework of sample gradient knowledge distillation with SGP
Table 11: Strategies for CIFAR100

Strategy | Dataset | Epochs | Batch size | Initial LR | Optimizer | Weight decay | LR scheduler | Data augmentation
---|---|---|---|---|---|---|---|---
A1 | CIFAR-100 | 240 | 64 | 0.05 | SGD | 0.0005 | X0.1 at 150,180,210 epochs | crop+flip

Figure 11: Simulated-superpixel gradients of samples during the offline
training process Table 12: Results (accuracy:$\%$) in CIFAR100

Teacher
Student |  WRN-40-2
WRN-16-2 |  WRN-40-2
WRN-16-2
(with SGP) |  ResNet110
ResNet20 |  ResNet110
ResNet20
(with SGP) |  Vgg13
Vgg8 |  Vgg13
Vgg8
(with SGP)
---|---|---|---|---|---|---
Teacher | 75.61 | / | 74.31 | / | 74.64 | /
Student | 73.26 | / | 69.06 | / | 70.36 | /
kD [23] | $74.92\pm 1.1$ |  $75.48\pm 1.3$$\uparrow$ | $70.67\pm 2.5$ |  $71.78\pm 1.2$$\uparrow$ | $72.98\pm 1.2$ |  $72.78\pm 1.1$$\downarrow$
RKD [39] | $73.35\pm 2.2$ |  $74.97\pm 2.1$$\uparrow$ | $69.25\pm 1.6$ |  $70.28\pm 1.8$$\uparrow$ | $71.48\pm 1.4$ |  $71.64\pm 1.3$$\uparrow$
DIST [25] | $73.78\pm 1.3$ |  $74.79\pm 1.4$$\uparrow$ | $71.86\pm 1.2$ |  $71.91\pm 3.5$$\uparrow$ | $71.62\pm 1.7$ |  $71.67\pm 1.6$$\uparrow$
SRRL [65] | $73.71\pm 1.7$ |  $75.16\pm 3.1$$\uparrow$ | $70.91\pm 1.4$ |  $71.23\pm 2.7$$\uparrow$ | $71.45\pm 1.8$ |  $71.78\pm 2.1$$\uparrow$
CRD [56] | $75.48\pm 2.1$ |  $75.61\pm 1.1$$\uparrow$ | $71.46\pm 1.7$ |  $71.57\pm 3.6$$\uparrow$ | $73.94\pm 1.2$ |  $73.62\pm 2.3$$\downarrow$
SGKD | $74.91\pm 1.1$ | / | $71.48\pm 1.7$ | / | $72.61\pm 2.1$ | /
ICKD [31] | $75.34\pm 1.2$ |  $75.44\pm 1.6$$\uparrow$ | $71.91\pm 1.3$ |  $72.01\pm 2.1$$\uparrow$ | $73.88\pm 2.2$ |  $74.09\pm 1.3$$\uparrow$
overhaul [21] | $75.52\pm 1.3$ |  $75.48\pm 1.4$$\downarrow$ | $71.21\pm 1.5$ |  $71.34\pm 1.2$$\uparrow$ | $73.42\pm 1.1$ |  $73.57\pm 1.9$$\uparrow$
AT [68] | $74.08\pm 1.7$ |  $74.61\pm 2.3$$\uparrow$ | $70.22\pm 2.1$ |  $71.24\pm 1.8$$\uparrow$ | $71.43\pm 1.9$ |  $72.86\pm 2.1$$\uparrow$
FitNet [44] | $73.58\pm 2.3$ |  $73.67\pm 2.1$$\uparrow$ | $68.99\pm 1.2$ |  $71.67\pm 2.3$$\uparrow$ | $71.02\pm 2.5$ |  $72.21\pm 2.4$$\uparrow$
FSP [66] | $72.91\pm 2.1$ |  $73.28\pm 1.7$$\uparrow$ | $70.11\pm 1.5$ |  $71.94\pm 1.5$$\uparrow$ | $70.23\pm 1.3$ |  $71.47\pm 1.5$$\uparrow$

Table 13: Ablation study on CIFAR100.

Teacher
Student |  WRN-40-2
WRN-16-2
(with SG) |  WRN-40-2
WRN-16-2
(with SGP) |  ResNet110
ResNet20
(with SG) |  ResNet110
ResNet20
(with SGP) |  Vgg13
Vgg8
(with SG) |  Vgg13
Vgg8
(with SGP)
---|---|---|---|---|---|---
Teacher | 75.61 | / | 74.31 | / | 74.64 | /
Student | 73.26 | / | 69.06 | / | 70.36 | /
KD[23] |  $72.13\pm 1.4$$\downarrow$ |  $75.48\pm 1.3$$\uparrow$ |  $68.77\pm 2.1$$\downarrow$ |  $71.78\pm 1.2$$\uparrow$ |  $69.58\pm 1.2$$\downarrow$ |  $72.78\pm 1.1$$\downarrow$
RKD [39] |  $71.39\pm 2.0$$\downarrow$ |  $74.97\pm 2.1$$\uparrow$ |  $68.45\pm 2.6$$\downarrow$ |  $70.28\pm 1.8$$\uparrow$ |  $69.18\pm 1.2$$\downarrow$ |  $71.64\pm 1.3$$\uparrow$
DIST [25] |  $72.44\pm 1.2$$\downarrow$ |  $74.79\pm 1.4$$\uparrow$ |  $68.76\pm 1.3$$\downarrow$ |  $71.91\pm 3.5$$\uparrow$ |  $69.52\pm 1.3$$\downarrow$ |  $71.67\pm 1.6$$\uparrow$
SRRL [65] |  $73.16\pm 1.2$$\downarrow$ |  $75.16\pm 3.1$$\uparrow$ |  $68.45\pm 1.8$$\downarrow$ |  $71.23\pm 2.7$$\uparrow$ |  $70.16\pm 1.4$$\downarrow$ |  $71.78\pm 2.1$$\uparrow$
CRD [56] |  $72.18\pm 1.8$$\downarrow$ |  $75.61\pm 1.1$$\uparrow$ |  $68.66\pm 2.7$$\downarrow$ |  $71.57\pm 3.6$$\uparrow$ |  $70.11\pm 1.2$$\downarrow$ |  $73.62\pm 2.3$$\downarrow$
ICKD [31] |  $72.39\pm 1.2$$\downarrow$ |  $75.44\pm 1.6$$\uparrow$ |  $71.91\pm 1.5$$\downarrow$ |  $72.01\pm 2.1$$\uparrow$ |  $68.34\pm 2.7$$\downarrow$ |  $74.09\pm 1.3$$\uparrow$
overhaul [21] |  $71.98\pm 2.1$$\downarrow$ |  $75.48\pm 1.4$$\downarrow$ |  $71.21\pm 1.3$$\downarrow$ |  $71.34\pm 1.2$$\uparrow$ |  $69.67\pm 1.4$$\downarrow$ |  $73.57\pm 1.9$$\uparrow$
AT [68] |  $72.78\pm 1.1$$\downarrow$ |  $74.61\pm 2.3$$\uparrow$ |  $70.22\pm 2.7$$\downarrow$ |  $71.24\pm 1.8$$\uparrow$ |  $69.89\pm 1.5$$\downarrow$ |  $72.86\pm 2.1$$\uparrow$
FitNet[44] |  $72.18\pm 1.2$$\downarrow$ |  $73.67\pm 2.1$$\uparrow$ |  $68.99\pm 1.2$$\downarrow$ |  $71.67\pm 2.3$$\uparrow$ |  $69.78\pm 2.1$$\downarrow$ |  $72.21\pm 2.4$$\uparrow$
FSP [66] |  $72.11\pm 2.1$$\downarrow$ |  $73.28\pm 1.7$$\uparrow$ |  $70.11\pm 1.3$$\downarrow$ |  $71.94\pm 1.5$$\uparrow$ |  $69.35\pm 1.3$$\downarrow$ |  $71.47\pm 1.5$$\uparrow$

Figure 12: The images from left to right are original feature vector
classification, feature vector classification with original SG, feature vector
classification with purified SG as $\beta=0.5$, and classification with
purified SG as $\beta=0.2$.