# Up to 100$\times$ Faster Data-free Knowledge Distillation

Gongfan Fang1,3, Kanya Mo1∗, Xinchao Wang2, Jie Song1
Shitao Bei1, Haofei Zhang1, Mingli Song1
Equal contributionsCorresponding author

###### Abstract

Data-free knowledge distillation (DFKD) has recently been attracting
increasing attention from research communities, attributed to its capability
to compress a model only using synthetic data. Despite the encouraging results
achieved, state-of-the-art DFKD methods still suffer from the inefficiency of
data synthesis, making the data-free training process extremely time-consuming
and thus inapplicable for large-scale tasks. In this work, we introduce an
efficacious scheme, termed as FastDFKD, that allows us to accelerate DFKD by a
factor of orders of magnitude. At the heart of our approach is a novel
strategy to reuse the shared common features in training data so as to
synthesize different data instances. Unlike prior methods that optimize a set
of data independently, we propose to learn a meta-synthesizer that seeks
common features as the initialization for the fast data synthesis. As a
result, FastDFKD achieves data synthesis within only a few steps,
significantly enhancing the efficiency of data-free training. Experiments over
CIFAR, NYUv2, and ImageNet demonstrate that the proposed FastDFKD achieves
10$\times$ and even 100$\times$ acceleration while preserving performances on
par with state of the art. Code is available at https://github.com/zju-
vipa/Fast-Datafree.

##  1 Introduction

Knowledge distillation (KD) has recently emerged as a popular paradigm to
reuse pre-trained models that are nowadays prevalent online. KD aims to train
a compact student model for efficient inference by imitating the behavior of
the pre-trained teacher (Hinton, Vinyals, and Dean 2015; Yang et al. 2020;
Fang et al. 2021a). The conventional setup of KD requires possessing the
original training data as input so as to train the student. Unfortunately, due
to confidential or copyright reasons, in many cases the original data cannot
be released and only the pre-trained models are available to users (Kolesnikov
et al. 2020; Shen et al. 2019; Ye et al. 2019), which, in turn, imposes a
major obstacle towards applying KD for a broader domain.

Figure 1: Accuracy (%) of student models v.s. GPU hours of data synthesis on
CIFAR-100 dataset. Our method, denoted as “Fast”, achieves 10$\times$ to even
100$\times$ acceleration and performance on par with existing methods.

To remedy this issue, data-free knowledge distillation (DFKD) approaches have
been proposed by assuming that no access to training data is available at all
(Lopes, Fenu, and Starner 2017). Due to the much relaxed constraint on
training data, DFKD has been receiving increasing attention from research
communities including computer vision (Chen et al. 2019), natural language
processing (Ma et al. 2020), and graph learning (Deng and Zhang 2021).
Typically, DKFD follows a distilling-by-generating paradigm, where a synthetic
dataset is often crafted for training by “inverting” the pre-trained teacher
(Yin et al. 2019). To learn a comparable student model, the synthetic set
should contain sufficient samples to enable a comprehensive knowledge transfer
from teachers. Consequentially, this poses a significant challenge to DFKD,
since synthesizing a large-scale dataset is inevitably time-consuming,
especially for sophisticated tasks like ImageNet recognition (Yin et al. 2019)
and COCO detection (Chawla et al. 2021).

In this paper, we introduce a novel approach, termed as FastDFKD, to
accelerate data synthesis process so as to make data-free knowledge
distillation more applicable for large-scale tasks. Our motivation stems from
the fact that, data instances from the same domain usually share some common
features, and hence such shared features should be explicitly utilized for
data synthesis. For example, the textures of “fur” may frequently emerge in an
animal dataset, and thus can be reused to create different instances.
Unfortunately, existing DFKD approaches have mainly focused on synthesizing
samples independently and, in fact, none of the existing DFKD approaches has
explored taking advantage of feature sharing, making the DFKD process very
cumbersome.

The proposed FastDFKD approach, on the other hand, expressively explores the
common features between samples for synthesizing. FastDFKD follows the batch-
based strategy for data synthesis (Yin et al. 2019; Fang et al. 2021b); yet
unlike prior methods that synthesizes different samples independently,
FastDFKD focuses on a “learning to synthesize” problem, where an efficient
synthesizer is explicitly trained to capture common features for fast
adaptation. The advantage of common features sharing is that we do not need to
synthesize them repeatedly for each instance, which significantly improves the
synthesis efficiency.

Specifically, we develop FastDFKD under a meta-learning framework (Finn,
Abbeel, and Levine 2017), which aims to learn a meta-generator in the
synthesis process. FastDFKD comprises two optimization loops, the outer loop,
and the inner loop. The inner loop accounts for the data synthesis process,
where a set of samples are created by adapting and re-organizing common
features. The outer loop, on the other hand, updates the common features using
the results of inner loops for a better meta-initialization. As illustrated in
Figure 1, such a meta synthesizer dramatically improves the efficiency of data
synthesis in data-free distillation while preserving a performance on par with
state of the art. As will be demonstrated in our experiments, FastDFKD is able
to achieve 10$\times$ and in some cases even more than 100$\times$ speed up
compared to the state of the art.

Our contribution is therefore a novel DFKD scheme, termed as FastDFKD, that
allows us to significantly accelerate the data-free training through common
feature reusing. Experimental results over the CIFAR, ImageNet, and NYUv2
datasets demonstrate that, FastDFKD yields performances comparable to the
state of the art, while achieving a speedup factor of 10 and sometimes even
more than 100 over existing approaches.

##  2 Related Works

##### Data-Free Knowledge Distillation.

Data-free knowledge distillation aims to train a compact student model from a
pre-trained teacher model without access to original training data. It
typically follows a distilling-by-generating paradigm, where a fake dataset
will be synthesized and used for student training. In the literature, Lopes et
al. proposes the first data-free approach for knowledge distillation, which
utilizes statistical information of original training data to reconstruct a
synthetic set during knowledge distillation (Lopes, Fenu, and Starner 2017).
This seminal work has spawned several works, which has achieved impressive
progressive on several tasks including detection (Chawla et al. 2021),
segmentation (Fang et al. 2019), text classification (Ma et al. 2020), graph
classification (Deng and Zhang 2021) and Federated Learning (Zhu, Hong, and
Zhou 2021). Despite the impressive progress, a vexing problem remains in DFKD,
i.e., the inefficiency of data synthesis, which makes data-free training
extraordinarily time-consuming. For example, (Luo et al. 2020) trains 1,000
generators to compress an ImageNet-pretrained ResNet-50 and (Yin et al. 2019)
optimizes a large number of mini-batches for data synthesis. In this work, we
focus on this under-studied problem, i.e., the efficiency of DFKD, and
proposes the first approach to accelerate data-free training.

##### Meta Learning.

Meta-learning is a popular framework for few-shot learning (Hospedales et al.
2020), which follows a “learning to learn” paradigm to find a helpful
initialization for target tasks. Among various meta-learning algorithms, MAML
is one of the most influential methods owning to its impressive results on
several benchmarks (Finn, Abbeel, and Levine 2017; Nichol, Achiam, and
Schulman 2018). As an optimization-based method for meta-learning, MAML
introduces two optimization loops to handle a set correlated tasks: an inner
loop for task learning and an outer loop for training a meta-learner. The
inner and outer loops are collaboratively trained to find a meta-
initialization that can be adapted to different tasks quickly, where some
general knowledge across tasks are captured by the meta-learner (Finn, Abbeel,
and Levine 2017). Inspired by the “learning to learn” paradigm of meta-
learning, we develop a fast approach to train a meta-synthesizer for DFKD
problems, which can be quickly adapted for fast data synthesis.

##  3 Method

Figure 2: Diagram of the proposed meta feature reuse, as well as its
difference to other synthesis strategies. (a) Data instances are synthesized
independently without feature reuse; (b) Data instances are synthesized
sequentially reusing previous results as initialization. (c) The proposed
common feature reuse that learns a meta-generator for fast adaptation.

### Problem Setup

Given a teacher model $f_{t}(x;\theta_{t})$ pre-trained on a labeled but
inaccessible training set
$D^{t}=\sum_{i}^{N}\\{(x_{i},y_{i})|x_{i}\in\mathcal{X},y_{i}\in\mathcal{Y}\\}$,
the goal of data-free knowledge distillation (DFKD) is to craft a synthetic
dataset $D=\sum_{i}^{N}\\{x_{i}|x_{i}\in\mathcal{X}\\}$ with $N$ samples by
inverting the pre-trained model, on which a comparable student model
$f_{s}(x;\theta_{s})$ can be trained by imitating the behaviour of the
teacher. Typically, The synthesis of $D$ is driven by a pre-trained inversion
loss $\mathcal{L}:\mathcal{X}\rightarrow\mathbb{R}$, which indicates whether a
input sample $x$ comes from the training domain according to some statistical
information in the pre-trained teacher model (Yin et al. 2019). Therefore, the
optimization of a single data point $x$ can be formalized as follows:

$x^{\ast}=\operatorname*{arg\,min}_{x}\mathcal{L}(x)$ (1)

To obtain a complete synthetic set $D^{\prime}=\\{x_{1},x_{2},...,x_{N}\\}$ of
size $N$, DFKD repeats the above optimization to construct a set of samples,
which leads to a series of optimization problems in the form of Equation 1.
Notably, the loss function for different instances $x_{i}$, denoted as
$\mathcal{L}_{i}$, can be different, so that a diverse dataset $D$ can be
constructed to carry out comprehensive knowledge from the teacher. To this
end, we consider a generalized data synthesis problem for DFKD, which
leverages a set of inversion losses
$\mathcal{L}=\\{\mathcal{L}_{1},\mathcal{L}_{2},...,\mathcal{L}_{N}\\}$ to
craft the synthetic dataset as follows:

$D^{\prime}=\\{x^{\ast}_{1},x^{\ast}_{2},...,x^{\ast}_{N}\\}=\operatorname*{arg\,min}_{x_{1},x_{2},...,x_{N}}\mathop{\sum}\limits_{i}^{N}\mathcal{L}_{i}(x_{i})$
(2)

In DFKD, a popular way to solve Equation 2 is to optimize different samples
directly in a batch-by-batch manner, (Yin et al. 2019; Fang et al. 2019;
Chawla et al. 2021). As illustrated in Figure 2 (a), batch-based approaches
synthesize different instances independently and merely take the relation
between samples into account. Despite the encouraging results achieved, DFKD
approaches usually suffer from the inefficiency of data synthesis, since
crafting a large-scale dataset requires solving a large number of optimization
problems in Equation 1, each takes thousands of steps to converge (Yin et al.
2019). Typically, data from the same domain are likely to share some common
features, which can be reused to synthesize different samples. In this work,
we present FastDFKD, a novel and efficacious approach to learning common
features to accelerate the optimization of Equation 2.

### Fast Data-Free Knowledge Distillation

##### Overview.

At the heart of our proposed approach is the reusing of common features. Our
motivation stems from the fact that data from the same domain typically share
some reusable patterns, which can be reused to synthesize different instances.
The following sections develop a novel definition for common features from a
generative perspective and propose FastDFKD to capture common features for
fast synthesis through a meta-learning process.

##### Common Feature.

As the key step towards fast data-free training, a clear definition of the
common feature is required to construct an optimizable objective for network
training. As illustrated in Figure 2 (b), a naive reusing strategy would be
the sequential feature reusing, where features learned in the previous
synthesis are directly used as the initialization to craft new samples.
However, such a naive scheme would be problematic since the learned features
only come from a single data point, which may not always be reusable for other
samples. To address this problem, we develop a more natural definition for
common features from a generative perspective. Let’s consider a generative
network $\mathcal{G}(z;\theta)$ with latent code $z$ and trainable parameters
$\theta$, which satisfies that for each sample $x_{i}\in D^{\prime}$, a latent
code $z_{i}$ can be found to generate $x_{i}=\mathcal{G}(z_{i};\theta)$. The
generator describes the generation process of different samples $x_{i}$. To
some extent, whether there are common features between a set of samples
$D^{\prime}=\\{x_{1},x_{2},...,x_{N}\\}$ is usually highly correlated with the
similarity of different data instances, which means that the generator can
implicitly capture the common features if we can find the optimal parameter
$\theta$ making the code $z=\\{z_{1},z_{2},...,z_{N}\\}$ of different samples
close in the latent space. Based on this, common features can be learned by
solving the following problems:

$\min\limits_{z,\theta}\underbrace{\frac{1}{N^{2}}\sum_{i}\sum_{j}d_{z}(z_{i},z_{j})}_{\textit{close
in latent
space}}+\underbrace{\frac{1}{N}\sum_{i}d_{x}(\mathcal{G}(z_{i},\theta),x_{i})}_{\textit{generation}}$
(3)

where $d_{z}$ and $d_{x}$ refers to distance metrics in latent space and input
space. The above optimization aims at finding a generation process for dataset
$D^{\prime}$, whose $z$ distance in latent space is as small as possible so
that, with the learned common features, different samples can be efficiently
obtained by navigating in the latent code $z$. However, in data-free settings,
the synthetic dataset $D^{\prime}$ is not available until we synthesize it.
Thus, We replace the second term defined on $d_{x}$ of Equation 3 with the
inversion loss $\mathcal{L}$ for DFKD, which leads to a data-free objective
for common feature learning:

$\min\limits_{z,\theta}\underbrace{\frac{1}{N^{2}}\sum_{i}\sum_{j}d_{z}(z_{i},z_{j})}_{\textit{close
in latent
space}}+\underbrace{\frac{1}{N}\sum_{i}\mathcal{L}_{i}(\mathcal{G}(z_{i},\theta))}_{\textit{data-
free generation}}$ (4)

However, due to the limited capacity of the generative model and the
difficulty in GAN training, it is almost intractable to learn a perfect
generator to synthesize the full synthetic set $D^{\prime}$ at the same time
(Luo et al. 2020). To remedy this problem, we make some relaxation on Equation
4 and does not force the generator $\mathcal{G}$ to capture all features for
the whole dataset. Instead, we train a generator that allows fast adaptation
to different samples within $k$-steps gradient descents, which naturally leads
to a meta-learning problem.

##### Meta Generator.

Equation 4 is challenging to optimize because it requires generating the full
dataset $D^{\prime}$ with a single generative network, including a lot of non-
reusable features. Instead, we can train a generator that only contains common
features and synthesize other missing features on the fly for the data
synthesis process as illustrated in Figure 2 (c). Specifically, we relax the
objective of common feature learning to train a lightweight generator that can
be adapted to synthesize different instances within $k$-step, formalized as a
meta-learning problem:

$\min\limits_{z,\theta}\frac{1}{N}\sum_{i}\mathcal{L}_{i}(\mathcal{G}(\underbrace{U^{k}_{\mathcal{L}_{i}}(\hat{z},\hat{\theta})}_{\textit{$k$-step
adaptation}}))$ (5)

where $U^{k}_{\mathcal{L}_{i}}$ is the inner loop of meta-learning, which
refers to a $k$-step optimization initialized from $\hat{\theta}$ and code
$\hat{z}$ for the synthesis of $x_{i}$. The inner loop can be unrolled as
follows:

$\begin{split}z^{0}_{i},\theta^{0}_{i}&=\hat{z},\hat{\theta}\\\
z^{k}_{i}&=z^{k-1}_{i}-\alpha\nabla_{z^{k-1}_{i}}\mathcal{L}_{i}(\mathcal{G}(z^{k-1}_{i};\theta^{k-1}_{i}))\\\
\theta^{k}_{i}&=\theta^{k-1}_{i}-\alpha\nabla_{\theta^{k-1}_{i}}\mathcal{L}_{i}(\mathcal{G}(z^{k-1}_{i};\theta^{k-1}_{i}))\end{split}$
(6)

Notably, Equation 5 plays a similar role as the common feature loss in
Equation 4. The inner loop, i.e., the $k$-step adaptation, aims to learn a
generator for synthesis by explicitly optimizing the second term of Equation
4. On the other hand, the outer loop tries to make different samples reachable
within k-step optimization by implicitly optimizing the first term of Equation
4.

##### Optimization.

Optimizing Equation 6 naturally leads to a meta-learning problem, where a
useful initialization $(\hat{z},\hat{\theta})$ is demanded for fast
adaptation. After $k$-step gradient descent, we obtain the a set of new
parameter
$(z^{*}_{i},\theta^{*}_{i})=U^{k}_{\mathcal{L}_{i}}(\hat{z},\hat{\theta}))$
under the guidance of loss function $\mathcal{L}_{i}$, which provides gradient
w.r.t $\hat{\theta}$ as follows:

$\begin{split}g_{\hat{\theta}}&=\nabla_{\hat{\theta}}\mathcal{L}_{i}(\mathcal{G}(U_{\mathcal{L}_{i}}(\hat{z};\hat{\theta})))\\\
&=U^{\prime}_{\mathcal{L}_{i}}(\hat{\theta})\mathcal{G}^{\prime}(\theta^{*})\mathcal{L}^{\prime}_{i}(x^{*}_{i})\end{split}$
(7)

where $\theta^{*}=U_{\mathcal{L}_{i}}^{k}(\hat{z};\hat{\theta})$ refers to the
optimization results of $k$-step adaptation using Equation 6 and
$x^{*}_{i}=\mathcal{G}(z^{*}_{i};\theta^{*}_{i})=\mathcal{G}(U_{\mathcal{L}_{i}}^{k}(\hat{z};\hat{\theta}))$
refers to the synthesis results under the guidance of loss $\mathcal{L}_{i}$.
However, note that the $k$-step adaptation in Equation 6 involves k gradient
updates:

$(z^{*},\theta^{*})=U^{k}_{\mathcal{L}_{i}}(\hat{z};\hat{\theta})=(\hat{z},\hat{\theta})+g_{1}+g_{2}+...+g_{k}$
(8)

where $g_{k}$ refers to the gradient computed at the $k$-th step of Equation
6, which introduces high-order gradients to the generator training and makes
the back-propagation very inefficient. Inspired by prior works in meta
learning (Nichol, Achiam, and Schulman 2018), we apply a first-order
approximation to further accelerate the gradient computing, which treats high-
order gradients in 8 as constants and replace the
$U^{\prime}_{i}(\hat{\theta})$ with an identity mapping. In this case, the
gradient computing in Equation 7 only involves first-order gradient and can be
simplified as follows:

$\nabla_{\hat{\theta}}\mathcal{L}_{i}(\mathcal{G}(U^{k}_{\mathcal{L}_{i}}(\hat{z};\hat{\theta})))=\mathcal{G}^{\prime}(\theta^{*})\mathcal{L}^{\prime}_{i}(x^{*})$
(9)

The first-order approximation directly uses the gradient computed on the
adapted generator in the inner loops to update the meta generator. Further, a
more efficient gradient approximation, known as reptile (Nichol, Achiam, and
Schulman 2018), can be achieved by approximating the gradient in Equation 9
with the parameter difference between the adapted generator and meta
generator, which further simplifies Equation 9 as:

$\nabla_{\hat{\theta}}\mathcal{L}_{i}(\mathcal{G}(U^{k}_{\mathcal{L}_{i}}(\hat{z};\hat{\theta})))=\hat{\theta}_{i}-\theta^{\ast}_{i}$
(10)

In conclusion, the optimization of meta generator can be presented as follows:

$\begin{split}\hat{\theta}&=\hat{\theta}-\eta\sum_{i}\nabla_{\hat{\theta}}\mathcal{L}_{i}(\mathcal{G}(U^{k}_{\mathcal{L}_{i}}(\hat{z};\hat{\theta})))\\\
\hat{z}&=\hat{z}-\eta\sum_{i}\nabla_{\hat{z}}\mathcal{L}_{i}(\mathcal{G}(U^{k}_{\mathcal{L}_{i}}(\hat{z};\hat{\theta})))\end{split}$
(11)

Algorithm 1 FastDFKD

Input: Pretrained teacher $\mathcal{F}_{t}(x;\theta_{t})$, student
$\mathcal{F}_{s}(x;\theta_{s})$.

Output: An optimized student $\mathcal{F}_{s}(x;\theta_{s})$

1:Randomly initialize a generator $\mathcal{G}(\hat{z};\hat{\theta})$

2:Initialize an empty dataset $D^{\prime}=\\{\\}$

3:for each synthesis loss $\mathcal{L}_{i}$ do:

4: $//$ 1\. $k$-step adaptation for synthesis (Eq. 6)

5: Periodically re-initialize $\hat{z}$ for diversity

6: Reuse meta feature $z,\theta\leftarrow\hat{z},\hat{\theta}$

7: for k steps do:

8: Update
$(z,\theta)\leftarrow(z,\theta)-\alpha\nabla_{(z,\theta)}\mathcal{L}_{i}(\mathcal{G}(z;\theta))$

9: end for

10: Generate the synthetic data $x^{*}=\mathcal{G}(z;\theta)$

11: Update the synthetic set $D^{\prime}$ = $D^{\prime}\cup\\{x^{*}\\}$

12:

13: $//$ 2\. Meta update (Eq. 10)

14: Update
$(\hat{z},\hat{\theta})\leftarrow(\hat{z},\hat{\theta})-\eta\nabla_{(\hat{z},\hat{\theta})}\mathcal{L}_{i}(\mathcal{G}(U^{k}_{i}(\hat{z};\hat{\theta})))$

15:

16: $//$ 3\. KD update (Eq. 12)

17: for t steps do:

18: sample a mini-batch $\mathcal{B}$ from $D^{\prime}$

19:
$\theta_{s}\leftarrow\theta_{s}-\gamma\sum\limits_{x\in\mathcal{B}}\nabla_{\theta_{s}}KL(f_{t}(x)\|f_{s}(x))$

20: end for

21:end for

#### Method Summary.

The proposed method is summarized in Algorithm 1, which consists of three
stages: 1) a $k$-step adaptation for data synthesis; 2) a meta-learning step
for common feature learning; 3) several KD steps to update the student model
by optimizing the KL divergence as follows:

$\theta_{s}\leftarrow\theta_{s}-\gamma\sum\limits_{x\in\mathcal{B}}\nabla_{\theta_{s}}KL(f_{t}(x)\|f_{s}(x))$
(12)

where $\mathcal{B}$ is a mini-batch sampled from the synthetic set
$D^{\prime}$. The proposed approach allows a small $k$ for data synthesis,
which significantly improves the efficiency of DFKD.

##  4 Experiments

### Inversion Loss for FastDFKD

In this section, we provide more details about the inversion loss
$\mathcal{L}$ used in our method. This work mainly focus on classification and
segmentation problems, which have been widely studied in the data-free
literature (Yin et al. 2019; Fang et al. 2019; Chen et al. 2019). We utilize
the prevalent Deep Inversion loss proposed by (Yin et al. 2019) as criteria
for data synthesis and demonstrate how to accelerate the data synthesis with
FastDFKD. Note that Deep Inversion loss consists of three terms: a class
confidence loss $\mathcal{L}_{cls}$, an adversarial loss $\mathcal{L}_{adv}$
and a feature regularization loss $\mathcal{L}_{feat}$, summarized as follows:

$\begin{cases}\mathcal{L}_{cls}(x)=CE(f_{t}(x),c)\\\
\mathcal{L}_{adv}(x)=-JSD(f_{t}(x)/\tau\|f_{s}(x)/\tau)\\\
\mathcal{L}_{feat}(x)=\sum_{l}\|\mu^{l}_{feat}-\mu^{l}_{bn}\|+\|\sigma^{l}_{feat}-\sigma^{l}_{bn}\|\end{cases}$
(13)

where both $\mathcal{L}_{cls}(x)$ and $\mathcal{L}_{adv}(x)$ provide dynamic
learning targets for data synthesis and thus can be directly used to construct
different synthesis tasks for meta-learning. For example, choosing different
pseudo labels can lead to various target categories. However, the loss for
feature regularization is unchanged during synthesis since the batch
normalization layer only encodes the global statistical information on the
whole dataset. To fully leverage the power of meta-learning, which requires a
set of different but related losses $\mathcal{L}$, we modify the Deep
Inversion loss by decomposing the feature regularization loss. Note that the
feature regularization $\mathcal{L}_{feat}$ aims to solve a distribution
matching problem, where the synthetic distribution is will be aligned with the
mean and variance $(\mu_{bn},\sigma^{2}_{bn})$ stored in BN layers. The BN
statistics is estimated on the whole dataset using the following rules:

$\begin{split}\mu_{bn}&=(1-m)\cdot\mu_{bn}+m\cdot\mu_{feat}\\\
\sigma^{2}_{bn}&=(1-m)\cdot\sigma^{2}_{bn}+m\cdot\sigma^{2}_{feat}\end{split}$
(14)

where $m$ is a momentum parameter for estimating BatchNorm statistics (BNS).
Inspired by the momentum estimation of BNS, we propose to construct a dynamic
and adaptive feature regularization in a momentum way, too. Specifically, we
introduce two accumulative variables to store the mean and variance of already
synthesized data and train the inputs to approximate the global BNS as
follows:

$\begin{split}\mathcal{L}_{feat}(x)=\sum\nolimits_{l}[&\|(1-m)\cdot\mu^{l}_{a}+m\cdot\mu^{l}_{feat}-\mu^{l}_{bn}\|+\\\
&\|(1-m)\cdot\sigma^{l}_{a}+m\cdot\sigma^{l}_{feat}-\sigma^{2}_{bn}\|]\end{split}$
(15)

After synthesis, the accumulative variables $(\mu_{a},\sigma^{2}_{a})$ will be
updated with Equation 14, so as to provide a different learning target for
meta-learning.

### Experimental Settings

##### Datasets and models.

We evaluate the proposed method on both classification and semantic
segmentation tasks. For image classification, we conduct data-free knowledge
distillation on three widely used datasets: CIFAR-10, CIFAR-100 (Krizhevsky,
Hinton et al. 2009) and ImageNet (Deng et al. 2009). We use the pretrained
models from (Fang et al. 2021b) and follow the same training protocol for
comparison, where 50,000 synthetic images are synthesized for distillation.
For ImageNet, we use an off-the-shelf ResNet-50 as the teacher and train
student models by synthesizing 224$\times$224 images. For semantic
segmentation, we use Deeplab models (Chen et al. 2017) trained on NYUv2
(Nathan Silberman and Fergus 2012) dataset for training and evaluation, which
contains 1449 densely labeled pairs of aligned RGB images and a 13-class
segmentation map.

##### Baselines.

Two types of DFKD methods are compared in our experiments: (1) generative
methods that train a generative model for synthesis, including DAFL (Chen et
al. 2019), ZSKT (Micaelli and Storkey 2019), DFQ (Choi et al. 2020), and
Generative DFD (Luo et al. 2020) (2) non-generative methods that craft
transfer set in a batch-by-batch manner including DeepInv (Yin et al. 2019)
and CMI (Fang et al. 2021b).

##### Evaluation Metrics.

In addition to the standard metrics like classification accuracy and mean IoU
for classification and segmentation, we also focus on the efficiency of data
synthesis in DFKD, where GPU hours taken by data synthesis is collected and
reported. Note that the time cost of student training is omitted since we only
focus on the data synthesis process and adopt the vanilla KD (Hinton, Vinyals,
and Dean 2015) in all DFKD methods. For fair comparisons, all GPU hours are
estimated on a single GPU.

Dataset | Method | ResNet-34 | VGG-11 | WRN40-2 | WRN40-2 | WRN40-2 | Average
---|---|---|---|---|---|---|---
ResNet-18 | ResNet-18 | WRN16-1 | WRN40-1 | WRN16-2 | Speed Up
CIFAR-10 | Teacher | 95.70 | 92.25 | 94.87 | 94.87 | 94.87 | -
Student | 95.20 | 95.20 | 91.12 | 93.94 | 93.95 | -
KD | 95.20 | 95.20 | 95.20 | 95.20 | 95.20 | -
DeepInv2k | 93.26 (42.1h) | 90.36 (20.2h) | 83.04 (16.9h) | 86.85 (21.9h) | 89.72 (18.2h) | 1.0$\times$
CMI500 | 94.84 (19.0h) | 91.13 (11.6h) | 90.01 (13.3h) | 92.78 (14.1h) | 92.52 (13.6h) | 1.6$\times$
DAFL | 92.22 (2.73h) | 81.10 (0.73h) | 65.71 (1.73h) | 81.33 (1.53h) | 81.55 (1.60h) | 15.7$\times$
ZSKT | 93.32 (1.67h) | 89.46 (0.33h) | 83.74 (0.87h) | 86.07 (0.87h) | 89.66 (0.87h) | 30.4$\times$
DFQ | 94.61 (8.79h) | 90.84 (1.50h) | 86.14 (0.75h) | 91.69 (0.75h) | 92.01 (0.75h) | 18.9$\times$
Fast2 | 92.62 (0.06h) | 84.67 (0.03h) | 88.36 (0.03h) | 89.56 (0.03h) | 89.68 (0.03h) | 655.0$\times$
Fast5 | 93.63 (0.14h) | 89.94 (0.08h) | 88.90 (0.08h) | 92.04 (0.09h) | 91.96 (0.08h) | 247.1$\times$
Fast10 | 94.05 (0.28h) | 90.53 (0.15h) | 89.29 (0.15h) | 92.51 (0.17h) | 92.45 (0.17h) | 126.7$\times$
CIFAR-100 | Teacher | 78.05 | 71.32 | 75.83 | 75.83 | 75.83 | -
Student | 77.10 | 77.10 | 65.31 | 72.19 | 73.56 | -
KD | 77.87 | 75.07 | 64.06 | 68.58 | 70.79 | -
DeepInv2k | 61.32 (42.1h) | 54.13 (20.1h) | 53.77 (17.0h) | 61.33 (21.9h) | 61.34 (18.2h) | 1.0$\times$
CMI500 | 77.04 (19.2h) | 70.56 (11.6h) | 57.91 (13.3h) | 68.88 (14.2h) | 68.75 (13.9h) | 1.6$\times$
DAFL | 74.47 (2.73h) | 54.16 (0.73h) | 20.88 (1.67h) | 42.83 (1.80h) | 43.70 (1.73h) | 15.2$\times$
ZSKT | 67.74 (1.67h) | 54.31 (0.33h) | 36.66 (0.87h) | 53.60 (0.87h) | 54.59 (0.87h) | 30.4$\times$
DFQ | 77.01 (8.79h) | 66.21 (1.54h) | 51.27 (0.75h) | 54.43 (0.75h) | 64.79 (0.75h) | 18.8$\times$
Fast2 | 69.76 (0.06h) | 62.83 (0.03h) | 41.77 (0.03h) | 53.15 (0.04h) | 57.08 (0.04h) | 588.2$\times$
Fast5 | 72.82 (0.14h) | 65.28 (0.08h) | 52.90 (0.07h) | 61.80 (0.09h) | 63.83 (0.08h) | 253.1$\times$
Fast10 | 74.34 (0.27h) | 67.44 (0.16h) | 54.02 (0.16h) | 63.91 (0.17h) | 65.12 (0.17h) | 124.7$\times$
Table 1: Student accuracy (%) on 32$\times$32 CIFAR. The methods “Teacher”, “Student” and “KD” is conducted on original training data and do not require data synthesis. DAFL, ZSKT and DFQ train generative networks for synthesis, while DeepInv, CMI and Fast (ours) optimizes batches to craft different samples. Method | Data Amount | Syn. Time | Speed Up | ResNet-50 | ResNet-50 | ResNet-50
---|---|---|---|---|---|---
ResNet-50 | ResNet-18 | MobileNetv2
Scratch | 1.3M | - | - | 75.45 | 68.45 | 70.01
Places365+KD | 1.8M | - | - | 55.74 | 45.53 | 39.89
Generative DFD | - |  $\sim$300h | 1$\times$ | 69.75 | 54.66 | 43.15
DeepInv2k | 140k | 166h | 1.8$\times$ | 68.00 | - | -
Fast50 | 140k | 6.28h | 47.8$\times$ | 68.61 | 53.45 | 43.02
Table 2: Student accuracy (%) on 224$\times$224 ImageNet. We use an off-the-
shelf ResNet-50 as the teacher model and train student models from scratch
following the training protocol of (Yin et al. 2019).

### Results on Classification

##### CIFAR-10 & CIFAR-100.

The student accuracy obtained on CIFAR-10 and CIFAR-100 datasets are reported
in Table 1. In the table, baseline “Teacher”, “Student” and “KD” train
networks with the original training data, which does not require data
synthesis. To verify the effectiveness of FastDFKD, we compare it with two
types of data-free algorithms, including generative methods that train
generators for synthesis and non-generative methods that optimize mini-batches
iteratively. As shown in Table 1, generative methods are usually 10$\times$
faster than non-generative methods like DeepInv and CMI since they only need
to train a single generator for synthesis. However, due to the limited
capacity of the generative network, it is almost challenging to synthesize a
diverse dataset for training, which may degrade the student performance. We
find that the performance of generative methods tends to degrade as the
complexity of the dataset increases from CIFAR-10 to CIFAR-100. By contrast,
non-generative is usually more flexible than generative ones and thus more
applicable to different tasks.

Like non-generative methods, the proposed FastDFKD also optimizes mini-batches
for data synthesis yet optimizes a generative network for adaptation. As shown
in Table 1, the 5-step FastDFKD, i.e., Fast5, can achieve 10$\times$
acceleration compared to existing generative methods and even more than
$100\times$ acceleration compared to non-generative methods. For example,
DeepInv2k synthesizes images by optimizing mini-batches, each of which
requires 2,000 iterations to converge (Yin et al. 2019). To obtain 50,000
training samples for CIFAR, DeepInv2k would take 42.1 hours for data synthesis
on a single GPU. by contrast, our method, i.e., Fast-5, adopts the same
inversion loss as DeepInv but only requires 5 steps for each batch owning to
the proposed common feature reusing, which is much more efficient than Deep
Inversion. In addition to the improvements in efficiency, FastDFKD also
achieves comparable or even superior performance compared to the state-of-
arts.

##### ImageNet.

To verify the effectiveness of FastDFKD, we further evaluate our method on a
more challenging dataset, i.e., ImageNet, which contains 1.3 million training
images of $224\times 224$ resolutions from 1,000 categories. ImageNet is
obviously much more complicated than CIFAR and thus much more time-consuming
for data-free training. As shown in Table 2, we compare our methods with a
data-driven KD that uses Places365 as alternative data, and two data-free
methods, i.e., Generative DFD (Luo et al. 2020) and DeepInv (Yin et al. 2019).
Notably, Generative DFD (Luo et al. 2020) trains one generator of
224$\times$224 resolutions for each category, which leads to 1,000 generators
in total. Although each generator can be optimized within one hour, the whole
training process for 1,000 generators is still cumbersome. By contrast, our
method only requires 6.28 hours for image synthesis and preserves comparable
performance to existing methods.

Figure 3: Visualization of synthetic data, inverted from an off-the-shelf
ResNet-50 classifier pre-trained on ImageNet. All samples are obtained using
the 50-step FastDFKD.

### Results on Segmentation

In this work, we further conduct data-free training on semantic segmentation
tasks to show the effectiveness of our method. In segmentation, we only use
the feature regularization loss and adversarial loss of 13 for data synthesis.
The mIoU of the student model, as well as the data amount and synthesis time,
are reported in Table 3. We compare our method with DFAD (Fang et al. 2019),
DAFL (Chen et al. 2019) and DFND (Chen et al. 2021). DFND is a data-driven
method and assumes that a sufficient unlabeled set is available for in-domain
data retrieval (Chen et al. 2021). DFAD and DAFL refer to data-free methods
that train generative networks for knowledge distillation. In comparison, our
method successfully synthesizes a training set only in 0.82 hours, which is
much more efficient than DAFL (3.99 hours) and DFAD (6.0 hours).

Method | Data Amount | Syn. Time | mIoU
---|---|---|---
Teacher | 1,449 (NYUv2) | - | 0.519
Student | 1,449 (NYUv2) | - | 0.375
KD | 1,449 (NYUv2) | - | 0.380
DFND | 14 M (ImageNet) | - | 0.378
DFAD | 960k (GAN) | 6.0h | 0.364
DAFL | 960k (GAN) | 3.99h | 0.105
Fast10 | 17K (Synthetic) | 0.82h | 0.366
Table 3: Mean IoU the student model trained on NYUv2 Segmentation dataset. The teacher model is a Deeplabv3-ResNet50 network with ImageNet pre-training, and the student model is a freshly initialized Deeplabv3-Mobilenetv2. Steps | 2 steps | 5 steps | 10 steps
---|---|---|---
Teacher | 75.83 | 75.83 | 75.83
Student | 65.31 | 65.31 | 65.31
DeepInv | 2.61 (0.03h) | 4.84 (0.06h) | 6.77 (0.11h)
CMI | 14.62 (0.05h) | 20.08 (0.12h) | 32.43 (0.22h)
Fast | 41.77 (0.03h) | 52.90 (0.07h) | 54.02 (0.16h)
Table 4: Few-step experiments on CIFAR-100. Settings | CIFAR-10 | CIFAR-100 |
---|---|---|---
No Reuse + Pixel | 44.74 | 3.11 |
No Reuse + GAN | 87.33 | 35.48 |
Seq. + Pixel | 71.97 | 10.93 |
Seq. + GAN | 90.91 | 59.38 |
Meta + GAN | 91.79 | 61.43 |
Meta + GAN + MMT | 91.96 | 63.83 |
Table 5: Ablation study for FastDFKD, where MMT means deep inversion with
momentum feature regularization.

### Quantitative Analysis

##### Few-step synthesis.

As aforementioned, FastDFKD allows efficient data synthesis within only a few
steps. This experiment validates our method by comparing it to the “few-step”
versions of existing non-generative methods. For example, we reduce the
optimization steps of original DeepInv (Yin et al. 2019) from 2,000 to
$\\{10,5,2\\}$ for CIFAR, which leads to the “efficient” DeepInv. As shown in
Table 4, the student accuracy of DeepInv and CMI severely degrades when the
optimization steps are reduced, which means that existing methods fail to
complete the data synthesis in only a few steps. By contrast, the proposed
FastDFKD works well even when a 2-step optimization is deployed, which
provides strong evidence for the effectiveness of FastDFKD.

##### Ablation Study.

In Table 5, we make an ablation study to make a systematic exploration for the
proposed method, where three reusing strategies in Figure 2 are considered: 1)
no feature reuse; 2) sequential feature reuse; 3) the proposed common feature
reuse. Further, the effectiveness of the generative network is also verified
by replacing it with the pixel updating proposed in (Yin et al. 2019).

##### Visualization.

The synthetic results on ImageNet are visualized in Figure 3, where all
samples are obtained by deploying the 50-step FastDFKD on an off-the-shelf
ResNet-50 classifier. Compared with existing methods that either require
solving a 2000-step mini-batch optimization (Yin et al. 2019) or train 1000
generative models for synthesis (Luo et al. 2020), our proposed method can
craft plausible samples within a few steps.

##  5 Conclusions

In this work, we propose a novel approach, termed as FastDFKD, to learn a
meta-generator for fast data-free knowledge distillation, which is able to
achieve 10 $\times$ and even 100$\times$ acceleration on CIFAR, NYUv2, and
ImageNet. As the first attempt to improve the efficiency of data-free
learning, the proposed approach successfully crafted a synthetic ImageNet in
6.28 hours, making data-free KD more applicable in real-world applications.

##### Acknowledgements.

This work is supported by National Natural Science Foundation of China
(U20B2066, 61976186,), Key Research and Development Program of Zhejiang
Province (2020C01023), the Major Scientific Research Project of Zhejiang Lab
(No. 2019KD0AC01), the Fundamental Research Funds for the Central
Universities, Alibaba-Zhejiang University Joint Research Institute of Frontier
Technologies, NUS ARTIC (Project Reference: ECT-RP2), and Faculty Research
Committee Grant (R-263-000-E95-133).

## References

  * Chawla et al. (2021) Chawla, A.; Yin, H.; Molchanov, P.; and Alvarez, J. 2021.  Data-Free Knowledge Distillation for Object Detection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 3289–3298.
  * Chen et al. (2021) Chen, H.; Guo, T.; Xu, C.; Li, W.; Xu, C.; Xu, C.; and Wang, Y. 2021.  Learning Student Networks in the Wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6428–6437.
  * Chen et al. (2019) Chen, H.; Wang, Y.; Xu, C.; Yang, Z.; Liu, C.; Shi, B.; Xu, C.; Xu, C.; and Tian, Q. 2019.  Data-free learning of student networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 3514–3522.
  * Chen et al. (2017) Chen, L.-C.; Papandreou, G.; Schroff, F.; and Adam, H. 2017.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_ .
  * Choi et al. (2020) Choi, Y.; Choi, J.; El-Khamy, M.; and Lee, J. 2020.  Data-free network quantization with adversarial knowledge distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 710–711.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Deng and Zhang (2021) Deng, X.; and Zhang, Z. 2021.  Graph-Free Knowledge Distillation for Graph Neural Networks.  _arXiv preprint arXiv:2105.07519_ .
  * Fang et al. (2021a) Fang, G.; Bao, Y.; Song, J.; Wang, X.; Xie, D.; Shen, C.; and Song, M. 2021a.  Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data.  In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Fang et al. (2019) Fang, G.; Song, J.; Shen, C.; Wang, X.; Chen, D.; and Song, M. 2019.  Data-free adversarial distillation.  _arXiv preprint arXiv:1912.11006_ .
  * Fang et al. (2021b) Fang, G.; Song, J.; Wang, X.; Shen, C.; Wang, X.; and Song, M. 2021b.  Contrastive Model Inversion for Data-Free Knowledge Distillation.  _arXiv preprint arXiv:2105.08584_ .
  * Finn, Abbeel, and Levine (2017) Finn, C.; Abbeel, P.; and Levine, S. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , 1126–1135. PMLR.
  * Hinton, Vinyals, and Dean (2015) Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ .
  * Hospedales et al. (2020) Hospedales, T.; Antoniou, A.; Micaelli, P.; and Storkey, A. 2020.  Meta-learning in neural networks: A survey.  _arXiv preprint arXiv:2004.05439_ .
  * Kolesnikov et al. (2020) Kolesnikov, A.; Beyer, L.; Zhai, X.; Puigcerver, J.; Yung, J.; Gelly, S.; and Houlsby, N. 2020.  Big Transfer (BiT): General Visual Representation Learning.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.  _technical report_ .
  * Lopes, Fenu, and Starner (2017) Lopes, R. G.; Fenu, S.; and Starner, T. 2017.  Data-free knowledge distillation for deep neural networks.  _arXiv preprint arXiv:1710.07535_ .
  * Luo et al. (2020) Luo, L.; Sandler, M.; Lin, Z.; Zhmoginov, A.; and Howard, A. 2020.  Large-scale generative data-free distillation.  _arXiv preprint arXiv:2012.05578_ .
  * Ma et al. (2020) Ma, X.; Shen, Y.; Fang, G.; Chen, C.; Jia, C.; and Lu, W. 2020.  Adversarial Self-Supervised Data-Free Distillation for Text Classification.  _arXiv preprint arXiv:2010.04883_ .
  * Micaelli and Storkey (2019) Micaelli, P.; and Storkey, A. 2019.  Zero-shot knowledge transfer via adversarial belief matching.  _arXiv preprint arXiv:1905.09768_ .
  * Nathan Silberman and Fergus (2012) Nathan Silberman, Derek Hoiem, P. K.; and Fergus, R. 2012.  Indoor Segmentation and Support Inference from RGBD Images.  In _ECCV_.
  * Nichol, Achiam, and Schulman (2018) Nichol, A.; Achiam, J.; and Schulman, J. 2018.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ .
  * Shen et al. (2019) Shen, C.; Wang, X.; Song, J.; Sun, L.; and Song, M. 2019.  Amalgamating knowledge towards comprehensive classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 3068–3075.
  * Yang et al. (2020) Yang, Y.; Qiu, J.; Song, M.; Tao, D.; and Wang, X. 2020.  Distilling knowledge from graph convolutional networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7074–7083.
  * Ye et al. (2019) Ye, J.; Ji, Y.; Wang, X.; Ou, K.; Tao, D.; and Song, M. 2019.  Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2829–2838.
  * Yin et al. (2019) Yin, H.; Molchanov, P.; Li, Z.; Alvarez, J. M.; Mallya, A.; Hoiem, D.; Jha, N. K.; and Kautz, J. 2019.  Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion.  _arXiv preprint arXiv:1912.08795_ .
  * Zhu, Hong, and Zhou (2021) Zhu, Z.; Hong, J.; and Zhou, J. 2021.  Data-Free Knowledge Distillation for Heterogeneous Federated Learning.  _arXiv preprint arXiv:2105.10056_ .


fluctuations. (a) The function is plotted as $a(x)$ with $d=5$ and for fixed
values of $y$; (b) The function is plotted as $a(y)$ with $d=5$ and for fixed
values of $x$; (c) The function is plotted as $a(d)$ for fixed values of the
pair $(x,y)$. Points with lower $a$ than the given function correspond to
stable configurations in all three plots. All the configurations with $a=1$
are below the surface of marginal stability, therefore they are stable and
there is no need for a plot (d).

(a) $d=5$, $y$ fixed

(b) $d=5$, $x$ fixed

(c) $(x,y)$ fixed

Figure 7:  Thermodynamic stability of the shell considering mass, area, and
charge fluctuations altogether is described, see Sec. IV.10 and Eqs. (79) and
(80). Plots of $S_{7}\left(d,a,x,y\right)=0$, with $d$ being the number of
spacetime dimensions, $a$ being the exponent of the equation of state of the
shell, $x=\frac{r_{+}^{d-3}}{R^{d-3}}$, and
$y=\frac{r_{-}^{d-3}}{r_{+}^{d-3}}$, where $r_{+}$ and $r_{-}$ are the
gravitational radius and the Cauchy radius of the Reissner-Nordström shell
spacetime, respectively, and $R$ is the radius of the shell, are shown. The
equality $S_{7}\left(d,a,x,y\right)=0$ describes marginal stability of the
shell considering mass, area, and charge fluctuations altogether. (a) The
function is plotted as $a(x)$ with $d=5$ and for fixed values of $y$; (b) The
function is plotted as $a(y)$ with $d=5$ and for fixed values of $x$; (c) The
function is plotted as $a(d)$ for fixed values of the pair $(x,y)$. Points
with lower $a$ than the given function correspond to stable configurations in
all three plots. All the configurations with $a=1$ are above the surface of
marginal stability, hence they are unstable, except for the points with $x=1$
which lie on the limit of the surface, hence they are marginally stable,
therefore they are stable and there is no need for a plot (d).

##  Appendix D Laboratory variables: Details for Sec. V

###  D.1 Mass and charge fluctuations together: Second derivatives of the
entropy in terms of laboratory quantities

Here we show the detailed calculations of the stability conditions in terms of
laboratory quantities presented in Sec. V.2 for the study of mass and charge
fluctuations together. The idea is that the second derivatives of the entropy
can be written in terms of variables that one can measure in the laboratory,
i.e., thermodynamic coefficients. We start from the two equations of state
$dT$ and $d\Phi$, seen as functions of $S$, $A$, and $Q$. They are given by

$\displaystyle
dT=\frac{T}{C_{A,Q}}dS-T\frac{\lambda_{T,Q}}{C_{A,Q}}dA-\frac{T\lambda_{T,A}}{C_{A,Q}}dQ\,,$
(101) $\displaystyle
d\Phi=-\frac{T\lambda_{T,A}}{C_{A,Q}}dS+P_{S,Q}dA+\frac{1}{\chi_{S,A}}dQ\,,$
(102)

where $C_{A,Q}$ is the specific heat capacity at constant area and electric
charge, $\lambda_{T,Q}$ is the latent heat capacity at constant temperature
and charge associated to the area, $\lambda_{T,A}$ is the latent heat capacity
at constant temperature and area associated to the charge, $P_{S,Q}$ is the
electric pressure at constant entropy and charge, and $\chi_{S,A}$ is the
adiabatic electric susceptibility at constant area. We then write the
differentials $dT(M,A,Q)$ and $d\Phi(M,A,Q)$ by using Eq. (36), i.e.,
$TdS=dM+pdA-\Phi dQ$, in Eqs. (101) and (102), yielding

$\displaystyle dT=$
$\displaystyle\frac{1}{C_{A,Q}}dM-\frac{T\lambda_{T,A}+\Phi}{C_{A,Q}}dQ-\frac{T\lambda_{T,Q}-p}{C_{A,Q}}dA\,,$
(103) $\displaystyle d\Phi=$
$\displaystyle-\frac{\lambda_{T,A}}{C_{A,Q}}dM+\left(\frac{1}{\chi_{S,A}}+\frac{\Phi\lambda_{T,A}}{C_{A,Q}}\right)dQ$
$\displaystyle+\left(\frac{\kappa_{p,S}}{\kappa_{S,Q}}-p\frac{\lambda_{T,A}}{C_{A,Q}}\right)dA\,.$
(104)

Now, Eq. (36), i.e., $TdS=dM+pdA-\Phi dQ$, and Eq. (103), yield
$S_{MM}=-\beta^{2}\left(\frac{\partial T}{\partial
M}\right)_{A,Q}=-\beta^{2}C_{A,Q}^{-1}$. Thus, the first condition for
stability, $S_{MM}\leq 0$, can be written as $C_{A,Q}^{-1}\geq 0$, or
equivalently,

$\displaystyle C_{A,Q}\geq 0\,.$ (105)

In addition, one can write $S_{MM}S_{QQ}-S_{MQ}^{2}$ as
$S_{MM}S_{QQ}-S_{MQ}^{2}=-\left(\frac{\partial\beta}{\partial
M}\right)_{A,Q}\left(\frac{\partial\beta\Phi}{\partial
Q}\right)_{M,A}-\left(\frac{\partial\beta}{\partial
Q}\right)_{M,A}^{2}=-\beta^{4}\left(\frac{\partial T}{\partial
M}\right)_{A,Q}\left(\frac{\partial T}{\partial
Q}\right)_{M,A}\Phi-\beta^{3}\left(\frac{\partial T}{\partial
M}\right)_{A,Q}\left(\frac{\partial\Phi}{\partial
Q}\right)_{M,A}-\beta^{4}\left(\frac{\partial T}{\partial
Q}\right)_{M,A}^{2}$, where Eq. (36), i.e., $TdS=dM+pdA-\Phi dQ$, has been of
help. Then, putting Eqs. (103) and (104) into this latter equation, one finds
$S_{MM}S_{QQ}-S_{MQ}^{2}=\beta^{3}\left(\frac{1}{C_{A,Q}\chi_{S,A}}-\frac{T\lambda_{T,A}^{2}}{C_{A,Q}^{2}}\right)$.
Thus, the second and necessary condition for stability, namely,
$S_{MM}S_{QQ}-S_{MQ}^{2}\geq 0$, can be written as

$\displaystyle\beta^{3}\left(\frac{1}{C_{A,Q}\chi_{S,A}}-\frac{T\lambda_{T,A}^{2}}{C_{A,Q}^{2}}\right)\geq
0\,.$ (106)

This can be further simplified. If we introduce the heat capacity at constant
electric potential and constant area, $C_{A,\Phi}=T\left(\frac{\partial
S}{\partial T}\right)_{\Phi,A}$. This coefficient can be written as
$C_{A,\Phi}^{-1}=\beta\left(\frac{\partial T}{\partial
S}\right)_{A,\Phi}=\beta\left(\left(\frac{\partial T}{\partial
S}\right)_{Q,A}+\left(\frac{\partial T}{\partial
Q}\right)_{S,A}\left(\frac{\partial Q}{\partial
S}\right)_{\Phi,A}\right)=\beta
C_{A,Q}^{-1}\left(1-\frac{T\lambda_{T,A}^{2}}{C_{A,Q}}\chi_{S,A}\right)=\beta\chi_{S,A}\left(\frac{1}{C_{A,Q}\chi_{S,A}}-\frac{T\lambda_{T,A}^{2}}{C_{A,Q}^{2}}\right)$,
where we have used Eqs. (101) and (102) to compute the derivatives. Also, we
used that $\left(\frac{\partial Q}{\partial
S}\right)_{\Phi,A}=T\chi_{S,A}\frac{\lambda_{T,A}}{C_{A,Q}}$, that comes from
inverting Eq. (102) to obtain $dQ(S,A,\Phi)$. Thus,
$\beta^{3}\left(\frac{1}{C_{A,Q}\chi_{S,A}}-\frac{T\lambda_{T,A}^{2}}{C_{A,Q}^{2}}\right)=\beta^{2}\frac{1}{C_{A,\Phi}\chi_{S,A}}$,
and Eq. (106) can be rewritten as $\frac{1}{C_{A,\Phi}\chi_{S,A}}\geq 0$, or,
equivalently, as

$\displaystyle C_{A,\Phi}\chi_{S,A}\geq 0\,,$ (107)

which is the upshot of the second condition for stability
$S_{MM}S_{QQ}-S_{MQ}^{2}\geq 0$ in the physical variables.

Now, we consider the particular case of the thin shell with the equations of
state given by Eqs. (49) and (50). The first condition for stability, Eq.
(105), holds without further ado. The second condition for stability, Eq.
(107) can be improved for the equations of state used. The coefficient
$\chi_{S,A}$ can be calculated using the differential in Eqs. (104) through
$\chi_{S,A}^{-1}=\left(\frac{\partial\Phi}{\partial
Q}\right)_{M,A}+\Phi\left(\frac{\partial\Phi}{\partial M}\right)_{A,Q}$. The
equation of state in consideration is given by Eqs. (46) and (50),
$\Phi=Q\left[\frac{r_{+}^{-(d-3)}-R^{-(d-3)}}{1-\frac{\mu
M}{R^{d-3}}}\right]$, with $r_{+}=r_{+}(M,A,Q)$ defined by Eqs. (15) and (33).
Thus, the coefficient $\chi_{S,A}^{-1}$ for this case is
$\chi_{S,A}^{-1}=\Phi^{2}\frac{\mu}{R^{d-3}\left(1-\mu\frac{M}{R^{d-3}}\right)}+\frac{\Phi}{Q}$,
which is positive for values of $(M,A,Q)$ that are physical. This means that
the second condition for stability given in Eq. (107) is reduced to

$\displaystyle C_{A,\Phi}\geq 0\,,$ (108)

for the equations of state we have used.

Here we have deduced in detail the stability conditions in physical variables
for mass and charged fluctuations presented in the main text in Sec. V.2. So,
Eqs. (103) and (104) are Eqs. (82) and (83) of the main text, respectively,
Eqs. (105) and (107) are the two equations given in Eq. (V.2) of the main
text, and Eq. (108) is Eq. (85) of the main text.

###  D.2 Mass, area, and charge fluctuations altogether: Second derivatives of
the entropy in terms of laboratory quantities

Here we show the detailed calculations of the stability conditions in terms of
laboratory quantities presented in Sec. V.3 for the study of mass, area, and
charge fluctuations altogether. The idea is that the second derivatives of the
entropy can be written in terms of variables one can measure in the
laboratory, i.e., thermodynamic coefficients. We start from the three
equations of state $dS(T,p,Q)$, $dA(T,p,Q)$ and $d\Phi(T,p,Q)$, seen as
functions of $T$, $p$, and $Q$. They are given by

$\displaystyle dS=$
$\displaystyle\left(\frac{C_{A,Q}}{T}+A\frac{\alpha_{p,Q}^{2}}{\kappa_{T,Q}}\right)dT$
$\displaystyle-A\alpha_{p,Q}dp+\lambda_{p,T}dQ\,,$ (109)
$\displaystyle\frac{dA}{A}=$
$\displaystyle\alpha_{p,Q}dT-\kappa_{T,Q}dp-\kappa_{p,T}dQ\,,$ (110)
$\displaystyle d\Phi=$
$\displaystyle-\lambda_{p,Q}dT-A\kappa_{p,T}dp+\frac{1}{\chi_{T,p}}dQ\,,$
(111)

where $C_{A,Q}$ is the heat capacity at constant area and charge,
$\alpha_{p,Q}$ is the thermal expansion coefficient, $\kappa_{T,Q}$ is the
isothermal compressibility, $\lambda_{p,T}$ is a latent heat capacity,
$\kappa_{p,T}$ is the electric compressibility, $\lambda_{p,Q}$ is another
latent heat capacity, and $\chi_{T,p}$ is the isothermal electric
susceptibility. We can invert Eqs. (109) and (110) to obtain $dT(S,A,Q)$ and
$dp(S,A,Q)$. This is accomplished by rewriting Eqs. (109) and (110) in matrix
form, i.e.,

$\displaystyle\begin{pmatrix}dS-\lambda_{p,T}dQ\\\
dA+A\kappa_{p,T}dQ\end{pmatrix}=\mathcal{M}\begin{pmatrix}dT\\\
dp\end{pmatrix}\,,$ (112)

where

$\displaystyle\mathcal{M}=\begin{pmatrix}\left(\frac{C_{A,Q}}{T}+A\frac{\alpha_{p,Q}^{2}}{\kappa_{T,Q}}\right)&-A\alpha_{p,T}\\\
A\alpha_{p,Q}&-A\kappa_{T,Q}\end{pmatrix}\,.$ (113)

The matrix $\mathcal{M}$ can be inverted to yield

$\displaystyle\mathcal{M}^{-1}=\begin{pmatrix}\frac{T}{C_{A,Q}}&-T\frac{\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}\\\
T\frac{\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}&-\left(\frac{1}{A\kappa_{T,Q}}+T\frac{\alpha^{2}_{p,Q}}{\kappa_{T,Q}^{2}C_{A,Q}}\right)\end{pmatrix}\,.$
(114)

Applying $\mathcal{M}^{-1}$ on both sides of Eq. (112), we obtain $dT(S,A,Q)$
and $dp(S,A,Q)$ as

$\displaystyle dT=$
$\displaystyle\frac{T}{C_{A,Q}}dS-T\frac{\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}dA-T\mathcal{B}dQ\,,$
(115) $\displaystyle dp=$
$\displaystyle\frac{T\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}dS-\left(\frac{1}{A\kappa_{T,Q}}+T\frac{\alpha_{p,Q}^{2}}{\kappa_{T,Q}^{2}C_{A,Q}}\right)dA$
$\displaystyle-\left(\frac{\kappa_{p,T}}{\kappa_{T,Q}}+T\frac{\alpha_{p,Q}}{\kappa_{T,Q}}\mathcal{B}\right)dQ\,,$
(116)

where
$\mathcal{B}=\frac{\lambda_{p,T}}{C_{A,Q}}+A\frac{\alpha_{p,Q}\kappa_{p,T}}{C_{A,Q}\kappa_{T,Q}}$.
We can then compute directly $dT(M,A,Q)$ and $dp(M,A,Q)$ by substituting the
differential $dS$ given in Eq. (36), i.e., $TdS=dM+pdA-\Phi dQ$, into Eqs.
(115) and (116), yielding

$\displaystyle dT=$
$\displaystyle\frac{dM}{C_{A,Q}}+\left(\frac{p}{C_{A,Q}}-T\frac{\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}\right)dA$
$\displaystyle-\left(\frac{\Phi}{C_{A,Q}}+T\frac{\lambda_{p,T}}{C_{A,Q}}+A\frac{\alpha_{p,T}\kappa_{p,T}}{\kappa_{T,Q}C_{A,Q}}\right)dQ\,,$
(117) $\displaystyle dp=$
$\displaystyle\frac{\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}dM$
$\displaystyle-\left[\frac{1}{A\kappa_{T,Q}}-\frac{\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}\left(p-T\frac{\alpha_{p,Q}}{\kappa_{T,Q}}\right)\right]dA$
$\displaystyle-\left(\frac{\kappa_{p,T}}{\kappa_{T,Q}}-\frac{\alpha_{p,Q}}{C_{A,Q}\kappa_{T,Q}}\mathcal{C}\right)dQ\,,$
(118)

where
$\mathcal{C}=TA\frac{\kappa_{p,T}\alpha_{p,Q}}{\kappa_{T,Q}}+T\lambda_{p,T}+\Phi$.
Then, substituting $dT$ and $dp$ given in Eqs. (115) and (116), respectively,
into $d\Phi(T,p,Q)$, written in Eq. (111), we obtain

$\displaystyle d\Phi=$
$\displaystyle-\mathcal{B}dM+\left[\frac{\kappa_{p,T}}{\kappa_{T,Q}}-\left(p-T\frac{\alpha_{p,Q}}{\kappa_{T,Q}}\right)\mathcal{B}\right]dA$
$\displaystyle+\left(\mathcal{B}\mathcal{C}+\frac{1}{\chi_{p,T}}+A\frac{\kappa_{p,T}^{2}}{\kappa_{T,Q}}\right)dQ\,.$
(119)

With $dT(M,A,Q)$, $dp(M,A,Q)$ and $d\Phi(M,A,Q)$, we can write the second
derivatives of the entropy since these are derivatives of $\beta(M,A,Q)$,
$\beta p(M,A,Q)$ and $\beta\Phi(M,A,Q)$. The second derivatives are explicitly
$S_{MM}=-\frac{\beta^{2}}{C_{A,Q}}$,
$S_{AA}=-\left[\frac{1}{A\kappa_{T,Q}}+\frac{\beta^{2}}{C_{A,Q}}\left(p-T\frac{\alpha_{p,Q}}{\kappa_{T,Q}}\right)^{2}\right]$,
$S_{QQ}=-\beta\left[\frac{\mathcal{C}^{2}\beta}{C_{A,Q}}+\frac{1}{\chi_{p,T}}+A\frac{\kappa_{p,T}^{2}}{\kappa_{T,Q}}\right]$,
$S_{MA}=\frac{\beta^{2}}{C_{A,Q}}\left(T\frac{\alpha_{p,Q}}{\kappa_{T,Q}}-p\right)$,
$S_{MQ}=\frac{\beta^{2}}{C_{A,Q}}\mathcal{C}$,
$S_{AQ}=-\beta\left[\frac{\kappa_{p,T}}{\kappa_{T,Q}}-\frac{\beta^{2}}{C_{A,Q}}\left(p-T\frac{\alpha_{p,Q}}{\kappa_{T,Q}}\right)\mathcal{C}\right]$.
Thus, the relevant combinations of the second derivatives for the stability
conditions are $S_{MM}=-\frac{\beta^{2}}{C_{A,Q}}$,
$S_{MM}S_{AA}-S_{MA}^{2}=\frac{\beta^{3}}{AC_{A,Q}\kappa_{T,Q}}$, and
$(S_{MM}S_{AQ}-S_{MA}S_{MQ})^{2}-(S_{MM}S_{AA}-S_{MA}^{2})(S_{MM}S_{QQ}-S_{MQ}^{2})=-\frac{\beta^{6}}{AC_{A,Q}^{2}\kappa_{T,Q}\chi_{p,T}}$.
For stability one has $S_{MM}\leq 0$, $S_{MM}S_{AA}-S_{MA}^{2}\geq 0$, and
$(S_{MM}S_{AQ}-S_{MA}S_{MQ})^{2}-(S_{MM}S_{AA}-S_{MA}^{2})(S_{MM}S_{QQ}-S_{MQ}^{2})\leq
0$, which translates into

$\displaystyle C_{A,Q}\geq 0,\,$ (120) $\displaystyle\kappa_{T,Q}\geq 0\,,$
(121) $\displaystyle\chi_{p,T}\geq 0\,.$ (122)

These are the stability conditions for mass, area, and charge fluctuations. In
words, the heat capacity at constant area and charge, the isothermal
compressibility associated to the pressure, and the isothermal electric
susceptibility must be positive.

The particular case of the thin shell with the equations of state given by
Eqs. (49), (50), and (30) can be also worked out in detail, we will not do it
here.

Here, we have deduced in detail the stability conditions in physical variables
for mass, area, and charged fluctuations presented in the main text in Sec.
V.3. So, Eqs. (117) and (118) are Eqs. (86) and (87) of the main text,
respectively, Eq. (119) is Eq. (88) of the main text, and Eqs. (120), (121),
(122) are the three equations given in Eq. (89) of the main text.

## References

  * (1) W. Israel, “Gravitational collapse and causality”, Phys. Rev. 153, 1388 (1967).
  * (2) K. Kuchař, “Charged shells in general relativity and their gravitational collapse”, Czech. J. Phys. B 18, 435 (1968).
  * (3) J. Kijowski and E. Czuchry “Dynamics of a self-gravitating shell of matter”, Phys. Rev. D 72, 084015 (2005); arXiv:gr-qc/0507074.
  * (4) G. A. S. Dias, S. Gao, and J. P. S. Lemos, “Charged shells in Lovelock gravity: Hamiltonian treatment and physical implications”, Phys. Rev. D 75, 024030 (2007); arXiv:gr-qc/0612072.
  * (5) S. Gao and J. P. S. Lemos, “Collapsing and static thin massive charged dust shells in a Reissner-Nordstrom black hole background in higher dimensions”, Int. J. Mod. Phys. A 23 , 2943 (2008); arXiv:0804.0295 [hep-th].
  * (6) E. F. Eiroa and C. Simeone, “Aspects of spherical shells in a $d$-dimensional background”, Int. J. Mod. Phys. D 21, 1250033 (2012); arXiv:1202.5009 [gr-qc].
  * (7) H. Andréasson, “Sharp bounds on the critical stability radius for relativistic charged spheres”, Commun. Math. Phys. 288, 715 (2009); arXiv:0804.1882 [gr-qc]
  * (8) J. P. S. Lemos and P. Luz, “All fundamental electrically charged thin shells in general relativity: From star shells to tension shell black holes, regular black holes, and beyond”, Phys. Rev. D 103, 104046 (2021); arXiv:2103.15832 [gr-qc].
  * (9) G. A. S. Dias and J. P. S. Lemos, “Thin-shell wormholes in $d$-dimensional general relativity: Solutions, properties, and stability”, Phys. Rev. D 82, 084023 (2010); arXiv:1008.3376 [gr-qc].
  * (10) C. Bejarano and E. F. Eiroa, “Dilaton thin-shell wormholes supported by a generalized Chaplygin gas”, Phys. Rev. D 84, 064043 (2011); arXiv:1106.6340 [gr-qc].
  * (11) J. P. S. Lemos and P. Luz, “Bubble universes and traversable wormholes”, Phys. Rev. D 105, 044058 (2022); arXiv:2201.02203 [gr-qc].
  * (12) J. P. S. Lemos and V. T. Zanchin, “Regular black holes: Electrically charged solutions, Reissner-Nordström outside a de Sitter core”, Phys. Rev. D 83, 124005 (2011); arXiv:1104.4790 [gr-qc].
  * (13) N. Uchikata, S. Yoshida, and T. Futamase, “New solutions of charged regular black holes and their stability”, Phys. Rev. D 86, 084025 (2012); arXiv:1209.3567 [gr-qc].
  * (14) R. Brandenberger, L. Heisenberg, and J. Robnik, “Non-singular black holes with a zero-shear S-brane”, J. High Energy Phys. JHEP 05 (2021) 090; arXiv:2103.02842 [hep-th].
  * (15) E. A. Martinez, “Fundamental thermodynamical equation of a self-gravitating system”, Phys. Rev. D 53, 7062 (1996).
  * (16) S. E. Perez Bergliaffa, M. Chiapparini, and L. M. Reyes, “Thermodynamical and dynamical stability of a self-gravitating uncharged thin shell”, Eur. Phys. J. C 80, 719 (2020); arXiv:2006.06766 [gr-qc].
  * (17) R. André, J. P. S. Lemos, and G. M. Quinta, “Thermodynamics and entropy of self-gravitating matter shells and black holes in $d$ dimensions”, Phys. Rev. D 99, 125013 (2019); arXiv:1905.05239 [hep-th].
  * (18) J. P. S. Lemos, G. M. Quinta, and O. B. Zaslavskii, “Entropy of a self-gravitating electrically charged thin shell and the black hole limit”, Phys. Rev. D 91, 104027 (2015); arXiv:1503.00018 [hep-th].
  * (19) L. M. Reyes, M. Chiapparini, and S. E. Perez Bergliaffa, “Thermodynamical and dynamical stability of a self-gravitating charged thin shell”, Eur. Phys. J. C 82, 151 (2022).
  * (20) J. P. S. Lemos, G. M. Quinta, and O. B. Zaslavskii, “Entropy of an extremal electrically charged thin shell and the extremal black hole”, Phys. Lett. B 750, 306 (2015); arXiv:1505.05875 [hep-th].
  * (21) J. P. S. Lemos, G. M. Quinta, and O. B. Zaslavskii, “Entropy of extremal black holes: Horizon limits through charged thin shells, a unified approach”, Phys. Rev. D 93, 084008 (2016); arXiv:1603.01628 [hep-th].
  * (22) J. W. York, “Black hole thermodynamics and the Euclidean Einstein action”, Phys. Rev. D 33, 2092 (1986).
  * (23) O. B. Zaslavskii, “Canonical ensemble for arbitrary configurations of self-gravitating systems”, Phys. Lett. A 152, 463 (1991).
  * (24) R. André and J. P. S. Lemos, “Thermodynamics of five-dimensional Schwarzschild black holes in the canonical ensemble”, Phys. Rev. D 102, 024006 (2020); arXiv:2006.10050 [hep-th].
  * (25) R. André and J. P. S. Lemos, “Thermodynamics of $d$-dimensional Schwarzschild black holes in the canonical ensemble”, Phys. Rev. D 103, 064069 (2021); arXiv:2101.11010 [hep-th].
  * (26) E. A. Martinez and J. W. York, “Additivity of the entropies of black holes and matter in equilibrium”, Phys. Rev. D 40, 2124 (1989).
  * (27) H. W. Braden, J. D. Brown, B. F. Whiting, and J. W. York, “Charged black hole in a grand canonical ensemble”, Phys. Rev. D 42, 3376 (1990).
  * (28) C. S. Peça and J. P. S. Lemos, “Thermodynamics of Reissner-Nordström-anti-de Sitter black holes in the grand canonical ensemble”, Phys. Rev. D 59, 124007 (1999); arXiv:gr-qc/9805004.
  * (29) J. P. S. Lemos and O. B. Zaslavskii, “Quasiblack holes: Definition and general properties”, Phys. Rev. D 76, 084030 (2007); arXiv:0707.1094 [gr-qc].
  * (30) J. P. S. Lemos and O. B. Zaslavskii, “Entropy of quasiblack holes”, Phys. Rev. D 81, 064012 (2010); arXiv:0904.1741 [gr-qc].
  * (31) J. P. S. Lemos and O. B. Zaslavskii, “Entropy of extremal black holes from entropy of quasiblack holes”, Phys. Lett. B 695, 37 (2011).
  * (32) J. P. S. Lemos and O. B. Zaslavskii, “Compact objects in general relativity: From Buchdahl stars to quasiblack holes”, Int. J. Mod. Phys. D 29, 2041019 (2020); arXiv:2007.00665 [gr-qc].
  * (33) H. B. Callen, Thermodynamics and an Introduction to Thermostatistics, 2nd ed. (Wiley, New York, 1991).