# Spectral properties of electrons in fractal nanowires

Alberto Hernando<EMAIL_ADDRESS>Miroslav Šulc  Jiří Vaníček
<EMAIL_ADDRESS>Laboratory of Theoretical Physical Chemistry, Institut
des Sciences et Ingénierie Chimiques, École Polytechnique Fédérale de
Lausanne, CH-1015 Lausanne, Switzerland

(August 27, 2024)

###### Abstract

In view of promising applications of fractal nanostructures, we analyze the
spectra of quantum particles in the Sierpinski carpet and study the non-
correlated electron gas in this geometry. We show that the spectrum exhibits
scale invariance with almost arbitrary spacing between energy levels,
including large energy gaps at high energies. These features disappear in the
analogous random fractal—where Anderson localization dominates—and in the
regular lattice of equally sized holes—where only two length scales are
present. The fractal structure amplifies microscopic effects, resulting in the
presence of quantum behavior of the electron gas even at high temperatures.
Our results demonstrate the potential of fractal nanostructures to improve the
light-matter interaction at any frequency, with possible applications, e.g.,
in the development of solar cells with a wide absorption spectrum, artificial
photosynthesis, or nanometamaterials with tailored Fermi levels and band gaps,
operating in a wide range of frequencies, and with extended operating
temperature range.

New generation of materials, based on nanostructures, is showing its potential
to revolutionise industry Cao and Wang (2011); Stangl  _et al._ (2004). In
parallel, self-similarity properties of fractal structures have revised the
way we manufacture antennas Hohlfeld and Cohen (1999) and permitted the
production of novel metamaterials Huang (2010); Cohen (2012). New materials
and manufacturing techniques have enabled producing fractal circuits Fairbanks
(2010), porous media such as silicas or aerogels, where superfluid helium
flows in a fractal environment Pollanen  _et al._ (2012), fractal-shaped
clusters of nanoparticles formed by aggregation Batabyal. (2013); or even
fractal snowflakes composed of graphene Massicotte. (2013). Additionally, one
can find self-similarity in the electronic spectra of quantum systems, such as
in Hofstadter’s butterfly and in the fractal quantum Hall effect Hofstadter
(1976); et al. (2013). Indeed, quantum fractals are drawing an increasing
attention due to their intriguing properties and potential applications
Katomeris and Evangelou (1996).

To describe the properties of a quantum particle in a fractal geometry (e.g.,
an electron trapped in a fractal-shaped nanowire), we first find the
eigenstates of the particle’s Hamiltonian $\mathcal{H=T+V}$, where
$\mathcal{T}$ is the kinetic energy operator and $\mathcal{V}$ is the
potential energy of an infinite well with the desired fractal boundaries. For
the well, we have chosen the paradigmatic Sierpinski carpet, a fractal object
studied in many different contexts Volpe  _et al._ (2011); Fairbanks (2010).
This carpet is constructed by subdividing a square into smaller copies of
itself and by removing the central copy iteratively, as shown in Fig. 1. Each
iteration is labeled with an integer $G$ indicating the generation of the
carpet. We study generations up to $G=6$, starting from $G=0$, which
represents the square well. Quantum Hamiltonian for a particle of mass $m$ in
a fractal geometry can be written as
$\mathcal{H}=\mathcal{T+V}=-\frac{\hbar^{2}}{2m}\nabla^{2}+V({\mathbf{q}})$,
where $V({\mathbf{q}})$ is an infinite well potential with the shape of the
accessible region in the carpet of generation $G$ and side $L$. Without loss
of generality, units of $L$ for length, $\hbar^{2}/mL^{2}$ for energy and
$k_{B}\hbar^{2}/mL^{2}$ for temperature (where $k_{B}$ is the Boltzmann
constant) are used when $m$, $\hbar$, or $L$ are not shown explicitly. For any
finite generation $G$, Sierpinski carpet can be viewed as a polygonal billiard
with rational angles, and therefore is an example of pseudointegrable systems
discovered by Richens and Berry Richens and Berry (1981), who also studied the
equivalent of the first generation of Sierpinski carpet, albeit with periodic
boundary conditions. As for higher fractal generations, Katomeris and
Evangelou studied properties of Sierpinski lattices Katomeris and Evangelou
(1996); we instead focus on the dynamics of quantum particles in a fractal
Sierpinski volume, such as electrons in fractal shaped wires, which are not
yet well understood.

To obtain the eigenenergies and eigenstates of $\mathcal{H}$, we have used two
different diagonalization methods: the finite differences method (FDM) and
imaginary-time nonuniform-mesh method (ITNUMM) Hernando and Vaníček (2013)—see
section I in Supplementary Material (SM) for details. The relative error in
the eigenenergies computed with FDM grows linearly with the energy, whereas
the error remains roughly constant for the ITNUMM. Thus, despite the error of
the FDM being smaller than the error of ITNUMM for the lowest-energy states,
it eventually becomes larger for high-energy states. As a consequence, ITNUMM
is the preferred method if one is interested in energies of many states.
Nevertheless, we have verified all our numerical results by evaluating the
energies both with ITNUMM and FDM.

Figure 1: Top panel: Iterative procedure for constructing each generation $G$
of the Sierpinski carpet (“Fractal”), and of the analogous random fractal
(“Random”) and regular non-fractal (“Regular”) structures. Bottom panel:
Wavefunctions of the first five states (rows $n=1$, 2, 4, 5) for the particle
in a box ($G=0$), and for the first four generations of the Sierpinski carpet
($G=1$, 2, 3, 4). For comparison, corresponding results are shown for the
third generation ($G=3$) of both the random fractal carpet (where Anderson
localization takes place) and the regular non-fractal lattice.

The spinless single-particle spectrum. Figure 1 shows the structure of the
lowest-lying states from the zeroth to fourth generation ($G=0$, $1$, $2$, $3$
and $4$. Generations $G=5$ and $6$ are not shown due to the limitations of
image resolution.) As the fractal generation increases, the wavefunctions
develop intricate structures on smaller and smaller scales, resulting in the
exponential increase of the kinetic energy of the ground state
$\epsilon_{\text{gs}}$ (Fig. 2a). This contribution to the energy affects
equally also all excited states and manifests itself as a global shift of the
energy spectrum for a given generation. Hence, the mode number function
$\mathcal{N}$ (Fig. 2b) and energy levels $\epsilon$ (Fig. 2c) are comparable
for all generations when referred to the ground state. For $G=0$, one obtains
the spectrum of a particle in a two-dimensional well Girifalco (2000), where
the mode number is proportional to the energy as
$\mathcal{N}(\epsilon)=\epsilon\leavevmode\nobreak\ mL^{2}/2\pi\hbar^{2}$, as
predicted by Weyl’s law. Increasing the fractal generation gives rise to two
qualitatively different energy regimes: (i) At lower energies, some levels
attract each other, agglomerating into bands, while others exhibit repulsion,
generating large gaps between the bands; (ii) at higher energies, the mode
number eventually approaches the prediction of Weyl’s law
$\mathcal{N}(\epsilon)=\epsilon\leavevmode\nobreak\
m^{\ast}L^{2}/2\pi\hbar^{2}$, where $m^{\ast}=(8/9)^{G}m$ is the effective
mass. Using this analytical form to rescale mode numbers as
$\mathcal{N}^{\ast}=\mathcal{N}/8^{G}$ and energies as
$\epsilon^{\ast}=\epsilon mL^{2}/9^{G}2\pi\hbar^{2}$ (see Fig. 2d, left
panel), we find, remarkably, that for any generation $G$ the bands and gaps in
regime (i) appear in the part of the spectrum with energies
$\epsilon^{\ast}<1$ ($\mathcal{N}^{\ast}<1$), whereas the approximately
constant density of states in regime (ii) appears for energies
$\epsilon^{\ast}>1$ ($\mathcal{N}^{\ast}>1$). This scaling of the energy
levels demonstrates that the self-similarity in the geometry is reflected also
in a scale-invariant spectrum.

Figure 2: Spectrum of a single particle in a Sierpinski-shaped well. a, Ground
state energy $\epsilon_{\text{gs}}$: The exponential decrease of the smallest
length scale with increasing generation $G$ manifests itself in the
exponential increase of $\epsilon_{\text{gs}}$. b, Mode number function
$\mathcal{N}(\epsilon-\epsilon_{\text{gs}})$. c, Energy level diagram: The
border between low-energy, band-gap regime (black line segments) and the
almost continuous high-energy regime (orange line segments) satisfying Weyl’s
law (see text) is shown for each $G$ by a thick red line segment. d, The self-
similarity of the spectrum becomes apparent when the energies and levels are
rescaled according to Weyl’s law (left panel: Sierpinski carpet; right panel:
random carpet).

To further explore this phenomenon, we studied the eigenvalue nearest-neighbor
spacing distribution in both regimes for all six generations. This
distribution is widely used in the fields of random matrix theory and quantum
chaos for characterizing ensembles of random matrices and dynamical properties
of Hamiltonian systems Reichl (2004); Edelman and Rao (2005). In addition,
since the energy of any optical transition of an electron can be described as
a sum of level spacings, their distribution can provide some information about
the expected absorption/emission properties of fractal nanowires. After first
unfolding the energy levels (as described in section II of the SM), we
computed the spacings $s$ of every two consecutive modes as well as the
maximum likelihood estimates of the parameters $\alpha$ and $b$ of the Brody
distribution Brody  _et al._ (1981)

$p(s)=(1+b)\alpha s^{b}\exp\left(-\alpha s^{b+1}\right)$ (1)

fit to the probability density of $s$. Whereas Brody distribution with zero
Brody parameter ($b=0$) reduces to the exponential distribution
$p(s)=\alpha\exp\left(-\alpha s\right)$, reflecting Poisson level statistics
characteristic of _generic integrable_ systems Berry and Tabor (1977), unit
Brody parameter ($b=1$) yields the Wigner distribution $p(s)=2\alpha
s\exp\left(-\alpha s^{2}\right)$ and, according to the quantum-chaos
conjecture McDonald and Kaufman (1979); Berry (1981); Bohigas  _et al._
(1984); Seligman  _et al._ (1984), reflects classically _chaotic_ dynamics of
systems with time-reversal invariant Hamiltonian. Generic time-reversal
invariant systems with mixed dynamics and without symmetry can be well
described with a mixture of exponential and Wigner distributions Seligman  _et
al._ (1984); Berry and Robnik (1984); while this mixture itself is not a Brody
distribution, it can be typically well fitted with a Brody distribution with
$0<b<1$. Remarkably, as shown in Fig. 3, for the Sierpinski carpet we find
that the Brody parameter evolves from $b=-0.03\pm 0.07$ for $G=1$,
corresponding to Poisson level statistics ($b=0$), to $b=-0.96\pm 0.03$ for
$G\geq 6$ in regime (i), resulting in almost an inverse power law distribution
($b=-1$). As the generation increases, the level repulsion-attraction
intensifies and it becomes harder to define the mean level spacing. Indeed,
for the asymptotic value $b=-1$ the mean level spacing ceases to exist and the
distribution cannot even be normalized; spacings of any order of magnitude are
equally probable at any energy. This transition does not appear in regime
(ii), where Brody parameter $b\approx 0$ in all the cases studied here. While
transitions between non-negative Brody parameters (from $b=1$ to $b=0$) have
been seen in the family of Koch fractals of changing fractal dimension Sakhr
and Nieminen (2005), to the best of our knowledge a transition from Poisson
statistics to a long-tailed distribution, similar to the one described here,
was previously observed only by Katomeris and Evangelou, who saw a power law
distribution with exponent $-1.56$ in a fractal lattice by removing diagonal
disorder Katomeris and Evangelou (1996). However, negative Brody parameters
were observed previously in superintegrable systems ($b\approx-0.24$) Alhassid
_et al._ (1990) or 1D lattices with unbounded quantum diffusion ($b=-3/2)$
Geisel  _et al._ (1991). For the desymmetrized system (see SM section II), we
find a qualitatively similar behavior involving a transition from positive to
negative values, confirming that the effect is due to the fractal geometry and
not due to degeneracies from simple geometrical symmetries.

Figure 3:  a, dependence of the Brody parameter $b$ of the distribution of
level spacings on the fractal generation $G$. Solid black: the low-energy
regime (i) (see text) undergoes a transition from the exponential (b, for
$G=1$, $b\approx 0$ reflecting Poisson level statistics) to inverse power-law
distribution (c, for $G\geq 6$, $b\approx-1$); Solid gray: the high-energy
regime (ii) shows no (or at most weak) dependence on $G$, approximately
following the Poisson distribution for all $G$. Dashed black: low energy
regime (i) for the random fractal, evolving from the Wigner ($b\approx 1$) to
exponential ($b\approx 0$) distribution; Dashed gray: regime (ii) showing no
(or at most weak) dependence on $G\geq 2$.

This peculiar feature of the spectrum of the Sierpinski carpet disappears
(Fig. 3 and SM Fig. 9) both in the analogous random fractal carpet, where the
holes are distributed randomly without overlapping (Fig. 1), and in the
equivalent regular but non-fractal geometry, where all the holes are of the
same size and are distributed regularly in a lattice (Fig. 1).

The random fractal exhibits a quasi-continuous spectrum instead of the level
repulsion and attraction pattern. Even if the potential well exhibits scale
invariance in a statistical fashion, breaking the geometrical symmetries
results in Anderson localization Anderson (1958), typical of disordered media
and here reflected in the localized eigenfunctions in Fig. 1. After rescaling
the mode numbers and energies as for the fractal, we find the same regime at
high energies $\epsilon^{\ast}>1$ ($\mathcal{N}^{\ast}>1$, Fig. 2d right
panel). However, in this regime for $G=2$ and $3$ the Brody parameter is
$b=0.90\pm 0.06$ and $b=0.83\pm 0.12$ respectively (Fig. 3), close to the
value $1$ characteristic of chaotic systems. On the other hand, the low-energy
part of the spectrum—where Anderson states are found—lacks the band and gap
structure, and the Brody parameter evolves, remarkably, from $b=0.7\pm 0.2$
for $G=2$ to $b=0.03\pm 0.06$ for $G=6$ as for an integrable system (see
section II of SM for details). Thus, breaking the regularity of the fractal
results in the loss of the scale invariance of nearest-neighbor level
spacings.

As another control, we analyzed the equivalent regular lattice of holes of
equal size. This system retains the geometrical symmetries of the square but
lacks the scale invariance since it has only two characteristic scales—the
size of the holes and the size $L$ of the system. We find that the spectrum of
the lattice exhibits none of the remarkable properties of the fractal (scaling
of the energy levels, presence of a regime with band-gap structure, self-
similarity). This system must be desymmetrized in order to fit the spacings
distribution to the form given by Eq. (1); the fitted distribution has a
positive Brody parameter that evolves to $b\approx 0$ for the largest
generations (see SM section II). Overall, these two negative controls suggest
that the unique features of the spectrum of the fractal are due to the
combined effect of geometrical symmetries and scale invariance.

The electron gas. We used 1000 single-particle states in order to study the
thermodynamic properties of the non-correlated electron gas Girifalco (2000)
in a Sierpinski carpet. The Fermi level of the system for different total
number of electrons $N_{e}$ with spin degeneracy is presented in Fig. 4. It is
shown that the generation $G$ can be chosen to obtain a Fermi level located
close to a gap, and reproduce the properties of a semiconductor material with
tailored band gap. The heat capacity $C(T)$ for $N_{e}=100$ electrons is shown
in Fig. 4 (see section III of SM for details). The 1000 states suffice to
describe each system accurately in a window of temperatures of almost three
times the Fermi temperature $T_{F}$ ($T_{F}=\pi N_{F}\approx 314$ for $G=0$).
For $G=0$ we recover the behavior of the 2D electron gas (2DEG), as expected.
Indeed, $C(T)$ depends linearly on $T$ at low temperatures and converges to
the classical limit ($C_{cl}=N_{e}$ in two dimensions) at $T\sim 2T_{F}$. For
$G=1$ and $G=2$ we obtain a similar behavior, with small deviations from the
behavior of the 2DEG. However, for $G\geq 3$ the presence of quasi-continuous
bands separated by large energy gaps becomes important at low temperatures,
where the increasing temperature permits some electrons to suddenly cross the
energy gaps and populate higher energy bands in a process that is reflected in
the oscillations of the heat capacity as a function of temperature. For the
highest generations, oscillations become substantial in a wider range of
temperatures and, at higher temperatures, the heat capacity eventually
overshoots the classical limit. Remarkably, the dependence of the heat
capacity per particle on the temperature expressed in units of Fermi
temperature converges to the same limit curve for the highest generations
(Fig. 4).

Figure 4:  Top panel: Fermi energy $E_{F}$ with respect the ground state
energy of noninteracting electron gas for $N_{e}=100$, $200$, $500$, and
$1000$ electrons for all generations studied here. Bottom panel: Dependence of
the heat capacity per particle $C/N_{e}$ on temperature (in units of Fermi
Temperature) $T/T_{F}$ for $N_{e}=100$ electrons. The alternation of bands and
gaps generates oscillations in $C$ as electrons thermally populate higher
energy bands, and results in an overshooting with respect to the asymptotic
thermodynamical value $C=N_{e}$ for a 2DEG (shown as a horizontal black line),
even at temperatures where the equivalent non-fractal 2DEG has reached the
classical limit. Remarkably, it converges to a well-defined function of
$T/T_{F}$ as the generation increases.

Two different mechanisms contribute to the overshooting: On one hand, the
observed oscillations due to the thermal population of high energy bands in
regime (i) become more relevant as the generation increases. Figure 4 shows
that for $N_{e}=100$ at a temperature of $T\sim 2T_{F}$ where the 2DEG ($G=0$)
already behaves classically, an analogous system of the same size but with the
shape of a Sierpinski carpet still behaves quantum-mechanically. On the other
hand, the increasing value of the ratio of the perimeter and surface of the
fractal with increasing $G$ affects the energy dependence of the density of
states in regime (ii). Indeed, the first correction term to Weyl’s law is
proportional to the perimeter and its dependence on $\sqrt{\epsilon}$ gives
rise to the overshooting (see SM section IV), this effect becoming more
pronounced as the perimeter term dominates over the surface term. However, for
any finite value of $G$, the surface contribution eventually dominates over
the perimeter correction at sufficiently high energies, and thus the
asymptotic value agrees with the 2D ideal-gas value. This would not happen in
the limit $G\rightarrow\infty$ (for a fixed energy or temperature); in this
limit the surface of the fractal becomes zero while the perimeter becomes
infinite, yielding a non-classical result of a divergent heat capacity.

In summary, the energy spectrum of quantum particles trapped in a Sierpinski
structure shows self-similarity, with level attraction and repulsion leading
to bands and gaps at any scale. The distribution of consecutive energy level
spacings exhibits a transition from an exponential to inverse power law
density as the fractal generation increases. At high generations, no
characteristic size for the spacings can be defined and level spacings of any
order of magnitude are equally probable. In fractal geometries, quantum
effects originating from the discreteness of the energy levels can be observed
at temperatures where the equivalent non-fractal system already behaves
classically.

In view of our results, fractal-shaped wires could exhibit promising features
for a wide range of applications. Scale invariance of the spectrum could
improve the photoelectric interaction of nanodevices in a wide range of light
frequencies and allow the design of nanomaterials with tailored Fermi levels,
band gaps, and conduction bands that can be thermally populated—mimicking the
properties of semiconductors, as seen in the oscillations of the heat
capacity—with potential applications for solar cells Zhu  _et al._ (2013);
Abdellatif and Kirah (2013), artificial photosynthesis Wang  _et al._ (2009),
or metamaterials Huang (2010); Cohen (2012); Volpe  _et al._ (2011); Schurig
_et al._ (2006). In addition, the persistence of quantum effects at high
temperatures could increase the operating temperature range of these quantum
nanodevices.

###### Acknowledgements.

The authors would like to thank Sir Michael Berry, Rubén Fossilón, Lev Kaplan,
Angelo Plastino, Thomas Seligman, Marius Wehrle, and Eduardo Zambrano for
useful discussions and valuable comments. This research was supported by the
Swiss National Science Foundation (NSF(CH)) with Grant No. 200020_150098 and
National Center of Competence in Research (NCCR) Molecular Ultrafast Science
and Technology (MUST), and by the EPFL.

## References

  * Cao and Wang (2011) G. Cao and Y. Wang, in  _World Scientific Series in Nanoscience and Nanotechnology: Volume 2_ , edited by F. Spaepen (World Scientific, Singapore, 2011) 2nd ed.
  * Stangl  _et al._ (2004) J. Stangl, V. Holy, and G. Bauer, Rev. Mod. Phys. 76, 725 (2004).
  * Hohlfeld and Cohen (1999) R. Hohlfeld and N. Cohen, Fractals 7, 79 (1999).
  * Huang (2010) X. Huang, Opt. Express 18, 10377 (2010).
  * Cohen (2012) N. Cohen, Fractals 20, 227 (2012).
  * Fairbanks (2010) M. Fairbanks,  _Transposrt in micro to nanoscale solid state networks_ , Ph.D. thesis, University of Oregon (2010).
  * Pollanen  _et al._ (2012) J. Pollanen, J. Li, C. Collet, W. Gannon, W. Halperin, and J. Sauls, Nature Phys. 8, 317 (2012).
  * Batabyal. (2013) R. Batabyal., J. Appl. Phys. 114, 064304 (2013).
  * Massicotte. (2013) M. Massicotte., Nanotechnology 24, 325601 (2013).
  * Hofstadter (1976) D. Hofstadter, Phys. Rev. B 14, 2239 (1976).
  * et al. (2013) B. H. et al., Science 340, 1427 (2013).
  * Katomeris and Evangelou (1996) G. N. Katomeris and S. N. Evangelou, J. Phys. A: Math. Gen. 29, 2379 (1996).
  * Volpe  _et al._ (2011) G. Volpe, G. Volpe, and R. Quidant, Opt. Lett. 19, 3612 (2011).
  * Richens and Berry (1981) P. J. Richens and M. V. Berry, Physica D 2, 495 (1981).
  * Hernando and Vaníček (2013) A. Hernando and J. Vaníček, Phys. Rev. A 88, 062107 (2013).
  * Girifalco (2000) L. Girifalco,  _Statistical Mechanics of Solids_ (Oxford Univ. Press, Oxford, 2000).
  * Reichl (2004) L. Reichl,  _The Transition to Chaos: Conservative Classical Systems and Quantum Manifestations_ , 2 ed. (Springer-Verlag, New York, 2004).
  * Edelman and Rao (2005) A. Edelman and N. Rao, Acta Numer. 14, 233 (2005).
  * Brody  _et al._ (1981) T. A. Brody, J. Flores, J. B. French, P. A. Mello, A. Pandey, and S. S. M. Wong, Rev. Mod. Phys. 53, 385 (1981).
  * Berry and Tabor (1977) M. V. Berry and M. Tabor, Proc. Royal Soc. London A 356, 375 (1977).
  * McDonald and Kaufman (1979) S. W. McDonald and A. N. Kaufman, Phys. Rev. Lett. 42, 1189 (1979).
  * Berry (1981) M. V. Berry, Annals of Phys. 131, 163 (1981).
  * Bohigas  _et al._ (1984) O. Bohigas, M. J. Giannoni, and C. Schmit, Phys. Rev. Lett. 52, 1 (1984).
  * Seligman  _et al._ (1984) T. H. Seligman, J. J. M. Verbaarschot, and M. R. Zirnbauer, Phys. Rev. Lett. 53, 215 (1984).
  * Berry and Robnik (1984) M. V. Berry and M. Robnik, J. Phys. A: Math. Gen 17, 24413 (1984).
  * Sakhr and Nieminen (2005) J. Sakhr and J. M. Nieminen, Phys. Rev. E 72, 045204(R) (2005).
  * Alhassid  _et al._ (1990) Y. Alhassid, A. Novoselsky, and N. Whelan, Phys. Rev. Lett. 65, 2971 (1990).
  * Geisel  _et al._ (1991) T. Geisel, R. Ketzmerick, and G. Petschel, Phys. Rev. Lett. 66, 1651 (1991).
  * Anderson (1958) P. Anderson, Phys. Rev. 109, 1492 (1958).
  * Zhu  _et al._ (2013) L.-H. Zhu, M.-R. Shao, R.-W. Peng, R.-H. Fan, X.-R. Huang, and M. Wang, Opt. Express 21, A313 (2013).
  * Abdellatif and Kirah (2013) S. Abdellatif and K. Kirah, Opt. Lett. 38, 3680 (2013).
  * Wang  _et al._ (2009) X. Wang, K. Maeda, A. Thomas, K. Takanabe, G. Xin, J. M. Carlsson, K. Domen, and M. Antonietti, Nature Materials 8, 76 (2009).
  * Schurig  _et al._ (2006) D. Schurig, J. Mock, B. Justice, S. Cummer, J. Pendry, A. Starr, and D. Smith, Science 314, 977 (2006).


# Self-Supervised Pretraining
for Differentially Private Learning

Arash Asadian
Indiana University
<EMAIL_ADDRESS>
&Evan Weidner
Indiana University
<EMAIL_ADDRESS>
&Lei Jiang
Indiana University
<EMAIL_ADDRESS>

###### Abstract

We demonstrate self-supervised pretraining (SSP) is a scalable solution to
deep learning with differential privacy (DP) regardless of the size of
available public datasets in image classification. When facing the lack of
public datasets, we show the features generated by SSP on only one single
image enable a private classifier to obtain a much better utility than the
non-learned handcrafted features under the same privacy budget. When a
moderate or large size public dataset is available, the features produced by
SSP greatly outperform the features trained with labels on various complex
private datasets under the same private budget. We also compared multiple DP-
enabled training frameworks to train a private classifier on the features
generated by SSP. Finally, we report a non-trivial utility 25.3% of a private
ImageNet-1K dataset when $\epsilon=3$. Our source code can be found at
https://github.com/UnchartedRLab/SSP.

##  1 Introduction

Machine learning (ML) has been applied ubiquitously in the analysis of
sensitive data such as medical images Tajbakhsh et al. (2016), financial
records Fischer and Krauss (2018), or social media channels Agrawal and Awekar
(2018). Many attacks Shokri et al. (2017); Carlini et al. (2021) are developed
to successfully extract meaningful training data out of standard ML models.
According to recent regulations, e.g., GDPR and CCPA, ML models have to
protect sensitive training data. Differential privacy (DP) Chaudhuri et al.
(2011); Bu et al. (2020); Abadi et al. (2016) has emerged as an effective
framework to train models resilient to private training data leakage.

Unfortunately, training models with strong DP guarantees significantly hurts
the model utility (i.e., accuracy) Papernot et al. (2018); Abadi et al.
(2016). Although non-learned handcrafted features (i.e., ScatterNet Oyallon
and Mallat (2015); Oyallon et al. (2018)) make a private linear model Tramer
and Boneh (2021) achieve the state-of-the-art (SOTA) utility of $<70\%$ under
the privacy budget of $(\epsilon\leq 3,\delta=10^{-5})$ on a private CIFAR-10
dataset, it is difficult to learn better features in the DP domain, since the
clipped and perturbed gradients during DP training provide only a noisy
estimate of the update direction. In contrast, it is straightforward that
pretrained features learned from large public labeled Luo et al. (2021)
datasets can greatly mitigate the utility gap between private and non-private
models. However, sometimes there is no available public dataset for training a
feature extractor due to legal causes or ethical issues Flanders (2009).

In this paper, we aim to demonstrate that self-supervised pretraining (SSP) is
a scalable solution to improving the utility of deep learning with DP
regardless of the size of available public datasets in image classification.
Any updates on the learnable parameters of a differentially private model
increase privacy overhead. It is easier to achieve both high utility and small
privacy loss via the features generated by a well-trained feature extractor
that can fully take advantage of SOTA network architectures and public
datasets. Even when no large public dataset is available, we show a feature
extractor built upon data-efficient HarmonicNet Ulicny et al. (2019) and
trained by self-supervised SimCLRv2 Chen et al. (2020) on only one single
image YM. et al. (2020) can make a private linear classifier obtain much
better utility than the non-learned handcrafted features Tramer and Boneh
(2021) under the same privacy budget. With a larger public dataset, the
features generated by SSP substantially outperform the features trained with
labels on various complex private datasets, as shown in Table 1. To better
explore the trade-off between utility and privacy, we compared SOTA DP-enabled
training frameworks, i.e., DP stochastic gradient descent (DPSGD) Abadi et al.
(2016), DP direct feedback alignment (DPDFA) Ohana et al. (2021); Lee and
Kifer (2020), DP stochastic gradient Langevin dynamics (DPSGLD) Bu et al.
(2021), and Private Aggregation of Teacher Ensembles (PATE) Papernot et al.
(2018), when training a private classifier on the features produced by SSP.
Private datasets having different learning distances from the public dataset
favor different training frameworks under different privacy budgets. Our
contributions are summarized as the follows.

  * •

When facing the lack of public datasets, we adopt HarmonicNet as the backbone
of SimCLRv2 to learn only one image. The features extracted by the HarmonicNet
greatly outperform the non-learned handcrafted ScatterNet features on various
private datasets by 0.6% on CIFAR10, 1.4% on CIFAR100, 6.7% on CropDiseases,
49.7% on EuroSAT, and 3.5% on ISIC2018, when $\epsilon=2$.

  * •

When there is a moderate or large size public dataset, the features produced
by SSP improve the utility of these private complex datasets over the features
trained by labels by $0.5\%\sim 8.6\%$ under the privacy budget of
$\epsilon=2$.

  * •

We compared SOTA DP-enabled training frameworks, i.e., DPSGD, DPDFA, DPSGLD,
and PATE, to train a private classifier on the features produced by SSP.
Compared to DPSGD, DPSGLD obtains a better utility on private datasets when
$\epsilon\leq 1$. DPDFA achieves a higher utility than DPSGD on private
datasets with a smaller learning distance from the public dataset when
$\epsilon>0.5$.

  * •

Through the features produced by SSP on a large unlabeled public dataset PASS
Asano et al. (2021), we report a non-trivial utility 25.3% of a private
ImageNet-1K dataset when $\epsilon=3$.

Table 1: The utility and privacy comparison on a private CIFAR10 dataset. DP-SOTA-1 means Tramer and Boneh (2021), and DP-SOTA-2 indicates Luo et al. (2021). Public Dataset | Scheme |  $\epsilon$-DP | Network | Training | Accuracy (%)
---|---|---|---|---|---
None | DP-SOTA-1 | 3 | Scat + CNN | DPSGD | 66.1111Group normalization. The results of batch normalization can be found in Appendix A.2.
1 image | Ours | 3 | HarmRN18 | DPSGD | 66.3222Batch normalization trained by the public dataset.
labeled CIFAR100 | DP-SOTA-2 | 1.5 | ResNet18 | DPSGD | 7111footnotemark: 1
unlabeled CIFAR100 | Ours | 1.5 | ResNet18 | DPSGD | 75.522footnotemark: 2
unlabeled ImageNet | DP-SOTA-1 | 2 | ResNet50 | DPSGD | 92.7
Ours | 2 | ResNet50 | DPDFA | 94.9

##  2 Background

###  2.1 Differentially Private Learning

Differential privacy. A network model $M:\mathcal{D}\rightarrow\mathcal{R}$ is
trained on two datasets $D,D^{\prime}\in\mathcal{D}$, which differ only by a
single data record. For any subset of outputs $R\in\mathcal{R}$, the model
satisfies $(\epsilon,\delta)$-differential privacy (DP) Abadi et al. (2016) if

$\displaystyle\text{\bf Pr}[M(D)\in R]\leq e^{\epsilon}\cdot\text{\bf
Pr}[M(D^{\prime})\in R]+\delta$

In another word, $\epsilon$ bounds the privacy loss on any single sample, and
$\delta$ is the probability that this bound does not hold. Rényi DP (RDP)
Mironov (2017) is a generalization of $(\epsilon,\delta)$-DP that uses Rényi
divergence as a distance metric. More RDP details can be viewed in Appendix
A.1.

DP-enabled training. DP guarantees can be enforced into a private network
model by various DP-enabled training frameworks including DPSGD Abadi et al.
(2016), DPSGLD Bu et al. (2021), DPDFA Ohana et al. (2021); Lee and Kifer
(2020), and PATE Papernot et al. (2018).

  * •

DPSGD. DPSGD Abadi et al. (2016) is the most widely used DP-enabled training
framework that satisfies RDP. RDP during DPSGD on a model is enforced by two
components: (1) per-sample gradients are clipped at a fixed L2 norm threshold
$C$; and (2) Gaussian noise of magnitude $\sigma^{2}C^{2}$ is added to the
gradient updates for a noise scale parameter $\sigma$. The privacy cost
$\epsilon$ during DPSGD can be measured by Moments Accountant Abadi et al.
(2016), which computes the upper bound of $\epsilon$ as a consequence of using
different composition theories. The privacy loss rate depends on the hyper-
parameters Mohapatra et al. (2021) of DPSGD.

  * •

DPSGLD. SGLD Welling and Teh (2011) is a gradient technique to train Bayesian
networks. SGLD makes the weights of a model to converge to a posterior
distribution rather than to a point estimate, from which it can sample and
characterize the uncertainty of weights. DPSGLD Wang et al. (2015) is built to
train DP Bayesian networks by adding per-sample clipping. Recent work Bu et
al. (2021) proves that DPSGLD is equivalent to DPSGD with regularization Bu et
al. (2021). The equivalence can be described as

$\displaystyle\text{DPSGLD}(\eta_{SGLD}=\eta,C_{SGLD}=C)=\text{DPSGD}(\eta_{SGD}=\eta
N,\sigma_{SGD}=\frac{|B|}{N\sqrt{\eta}C},C_{SGD}=C),$

where $\eta_{X}$ is the learning rate of $X$; $C_{X}$ is the clipping norm of
$X$; $|B|$ is the batch size; and $N$ is the private dataset size.

  * •

DPDFA. DFA Nø kland (2016) emerges as an effective alternative to
backpropagation in non-private training. DPDFA Ohana et al. (2021); Lee and
Kifer (2020) enforces DP in DFA by clipping activations and error signal after
the feedforward phase, scaling the error transport matrix, and adding Gaussian
noise to the update direction.

  * •

PATE. PATE Papernot et al. (2018) is a framework based on private knowledge
aggregation of an ensemble model and knowledge transfer. It trains an ensemble
of teachers on disjoint subsets of the private dataset. The ensemble’s
knowledge is then transferred to a student model via differentially private
aggregation of the teachers’ votes on samples from an unlabeled public
dataset. The student model is released as the output of the training.

###  2.2 Handcrafting and Learning Low-level Features

Handcrafted features. Recent work Tramer and Boneh (2021) advocates adopting
the wavelet-transform-based ScatterNet Oyallon and Mallat (2015); Oyallon et
al. (2018) to extract non-learned handcrafted features (e.g., invariance to
small rotations and translations) for differentially private learning. A
private linear classifier learned on these handcrafted features exhibits
higher utility than an end-to-end CNN under a moderate privacy budget.

Learned features. In the non-private setting, recent work YM. et al. (2020)
demonstrates the features learned from one single image with strong
augmentations and without supervision can obtain a non-trivial accuracy on
ImageNet. However, the performance of these learned features is not studied in
the DP domain. In this paper, we show that the learned features are more
effective than the handcrafted ScatterNet features in the DP domain.

Combining handcrafted and learned features. HarmonicNet Ulicny et al. (2019)
is a data-efficient network using a set of non-learned handcrafted filters
based on Discrete Cosine Transform (DCT) at multiple frequencies combined by
learnable weights. The DCT-based filters have excellent energy compaction
properties. It can obtain higher non-private accuracy Ulicny et al. (2019)
than ScatterNet, particularly when only small training datasets are available.
However, the performance of the features generated by a HarmonicNet
architecture is never examined in the DP domain.

Table 2: The utility comparison of private models directly trained on a
private CIFAR10 under $(\epsilon=3,\delta=10^{-5})$.

Public | Network | Acc (%)
---|---|---
Dataset | Architecture
none | end-to-end CNN | 59.2
ScatterNet+linear | 64.2
ScatterNet+CNN | 66.1
HarmonicNet + linear | 47.7
HarmonicNet + CNN | 50.2

Table 3: The utility comparison of private linear classifiers learned on
pretrained features and a private CIFAR10 under $(\epsilon=3,\delta=10^{-5})$.

Public | Feature | Acc (%)
---|---|---
Dataset | Extractor
one image |  ResNet18 | ResBlock-1 | 49.7
ResBlock-2 | 56.3
ResBlock-3 | 51.4
ResBlock-4 | 50.3
ScatRN18 | 63.1
HarmRN18 | 66.3

##  3 Self-Supervised Pretraining For Differentially Private Learning

In this section, we explain how to build scalable pretrained features for
differentially private learning with various sizes of public datasets, i.e.,
no public dataset, a moderate size (38K$\sim$50K) public dataset, and a large
size (>1M) public dataset. Although batch normalization (BN) can be supported
by DP in principle, it is difficult to use the same hyper-parameters, e.g.,
noise multiplier and learning rate, to train BN parameters and the other
network parameters in the DP domain. The SOTA DP-enabled training libraries
such as Opacus Yousefpour et al. (2021) do not even support BN yet. Therefore,
in this paper, we enforce that DP-enabled training directly happening on the
private dataset requires the model to use only group normalization, and all
pretrained features are generated with the BN parameters trained by only the
public dataset. How to fine-tune BN parameters with private datasets is beyond
the topic of this paper, and can be answered by Luo et al. (2021). Results of
BN on private datasets can be viewed in Appendix A.2. The utility and privacy
improvement of the features generated by SSP over non-learned handcrafted
features and those trained with labels is not reduced by private BN.

###  3.1 No public dataset is available

When no public dataset is available, we compare two ways to train a private
model on a private dataset. One way is to train a private model directly on
the private dataset using non-learned handcrafted features. The other way is
to train a feature extractor by a data-efficient network architecture, self-
supervision, and a single image, and then to train a private linear classifier
on the private dataset using features produced by the feature extractor. We
used only DPSGD for DP training in this section.

Training directly on the private dataset. We show the utility comparison of
various schemes directly learning a private CIFAR10 under the privacy budget
of $(\epsilon=3,\delta=10^{-5})$ in Table 2. The training schemes include a
5-layer CNN, a linear classifier learned on the ScatterNet handcrafted
features, and a 5-layer CNN learned on the ScatterNet features, which are
adopted from Tramer and Boneh (2021). We also compare a linear classifier and
a 5-layer CNN learned on the DCT-based HarmonicNet handcrafted features. The
CNN with the ScatterNet features achieves the highest utility, i.e., 66.1%. We
find the HarmonicNet features are vulnerable to noises, i.e., applying noises
to the weights combining multiple DCT frequencies significantly degrades the
utility of private models.

Table 4: The CIFAR10 utility comparison of the features produced by feature
extractors further fine-tuned by different numbers of public images from
CIFAR100 under $(\epsilon=3,\delta=10^{-5})$.

Pretrained Feature Extractor | Acc (%) $(\epsilon=3,\delta=10^{-5})$
---|---
Unlabeled Image #
0 | 1K | 10K | 50K
ResNet18 | 56.3 | 62.6 | 66.8 | 71.5
ScatRN18 | 63.1 | 67.2 | 70.8 | 73.2
HarmRN18 | 66.3 | 72.4 | 73.1 | 78

Table 5: The CIFAR10 utility comparison of private classifiers learned on the
HarmRN18 features and trained by various DP-training methods (DP-SOTA-2
indicates Luo et al. (2021)).

Public | Training | $\epsilon$-DP | Acc (%)
---|---|---|---
Dataset | Scheme
| PATE | $16$ | 70.3
unlabeled | DPSGD | $1.5$ | 75.5
CIFAR100 | DPSGLD | $1.5$ | 74.9
| DPDFA | $\mathbf{1.5}$ | 78.2
CIFAR100 | DP-SOTA-2 | $1.5$ | 71

Training a feature extractor. We applied a series of aggressive data
augmentations (e.g., cropping) on a single $600\times 225$ image (Appendix
A.3) to create a synthetic dataset as the public dataset. More details on our
experimental methodology can be found in Section 4. To first train a feature
extractor on the synthetic dataset, we used three different network
architectures including ResNet18 He et al. (2016), ScatRN18, and HarmRN18 to
serve as the backbone of the self-supervised SimCLRv2 Chen et al. (2020).
ScatRN18 means ResNet18 with ScatterNet handcrafted features, while HarmRN18
indicates ResNet18 where all convolutional filters are replaced by DCT-based
HarmonicNet filters. ResNet18 contains four basic blocks connected
sequentially. We extracted pretrained features from all four basic blocks, and
use ResBlock-$X$ to represent the pretrained features extracted from the block
$X$ of ResNet18. We show the CIFAR10 utility comparison of private classifiers
learned on the pretrained features produced by different feature extractors
under the privacy budget of $(\epsilon=3,\delta=10^{-5})$ in Table 3. All
private classifiers are 1-layer classifiers. Although compared to the last
ResBlock, the two middle basic blocks of ResNet18 produce better features, a
single image is not enough to make ResNet18 generate high-quality features.
Both ScatRN18 and HarmRN18 output better features than ResNet18. The features
produced by HarmRN18 (66.3%) slightly outperform the ScatterNet features
(ScatterNet+CNN 66.1% in Table 2) used directly in the DP training, since DCT
is fused with every filter of HarmRN18. We will show the HarmRN18 features can
obtain significantly higher utility than the ScatterNet features with the same
privacy loss on other complex datasets in Section 5.1.

###  3.2 A moderate size public dataset is available

Fine-tuning by public unlabeled images. To further improve the quality of
pretrained features, we randomly selected 1K unlabeled image from a public
CIFAR100 dataset to fine-tune the feature extractors, as shown in Table 4,
where we still used DPSGD to train the private classifier. Training with 1K
unlabeled images improves the utility of the private classifier learned on the
features generated by HarmRN18 to $72.4\%$. The entire unlabeled CIFAR100
dataset (50K images) makes the private classifier achieve $78\%$ accuracy on
the private CIFAR10 dataset.

Comparing DP-enabled training frameworks. We used different DP-enabled
training frameworks, i.e., PATE, DPSGD, DPSGLD, and DPDFA, to train a private
classifier learned on the HarmRN18 features for a private CIFAR10 dataset. The
HarmRN18 feature extractor is trained by a public CIFAR100 dataset. For PATE,
teacher and student models are 1-layer linear classifiers, and use the
HarmRN18 features. We adopted 1K teacher models, and CIFAR100 as the public
dataset. The private classifiers of DPSGD and DPSGLD are 1-layer classifiers,
while the private classifier of DPDFA is a 2-layer multilayer perceptron
(MLP), where the first layer uses Tanh activations and the second layer uses a
Sigmoid activation. We actually tried a 2-layer MLP for all frameworks, but it
works better only with DPDFA, as shown in Appendix A.4. The comparison between
DP-enabled training frameworks is shown in Table 5. Compared to DPSGD, the
semi-supervised PATE cannot fully take advantage of the pretrained features.
The student model has to perform many queries on teacher models to obtain
reasonably high utility, so the privacy loss is high ($16$). DPSGLD is similar
to DPSGD, but it costs higher privacy overhead in each epoch. As a result,
DPSGLD achieves a slightly worse utility than DPSGD under the same privacy
budget. Although we used the worst privacy overhead per epoch to estimate the
privacy loss of DPDFA, DPDFA still obtains the best utility when
$\epsilon=1.5$ among all frameworks. DPDFA makes the private classifier
learned on the HarmRN18 features obtain a much better utility than the
classifier Luo et al. (2021) learned on the features trained by a labeled
CIFAR100 dataset. The comparisons between these DP-enabled training frameworks
on more complex private datasets can be found in Section 5.4.

(a) Crop (small distance)

(b) EuroSAT (medium distance)

(c) ISIC2018 (large distance)

Figure 1: The comparison of features trained by labeled and unlabeled mini-
ImageNet on various out-of-domain complex private datasets.

Pretrained features for cross-domain datasets. To exam the effectiveness of
pretrained features on a private dataset from a different domain, we adopted
mini-ImageNet Vinyals et al. (2016) as our public dataset to train the
pretrained features, and studied the utility-privacy trade-off on out-of-
domain private datasets such as CropDiseases Mohanty et al. (2016a) (Crop),
EuroSAT Helber et al. (2019), and ISIC2018 Tschandl et al. (2018). We first
trained HarmRN18 by SimCLRv2 on the mini-ImageNet dataset with no label. And
then, we used HarmRN18 as the feature extractor to produce features on private
Crop, EuroSAT, and ISIC2018 datasets. Finally, we trained a private classifier
on each set of features. We trained a ResNet18 feature extractor Luo et al.
(2021) by the labeled mini-ImageNet dataset as our baseline. Our baseline is
trained using the standard cross-entropy loss with label smoothing. We set the
label-smoothing parameter to 0.1. The comparison between our scheme and
baseline is shown in Figure 1. Except ISIC2018 under $\epsilon<1.3$, the
features generated by SSP enable the private classifier to achieve a better
utility than our baseline under the same privacy budget. Particularly, the
features produced by SSP on a moderate size of public dataset are already
capable enough even for private datasets (e.g., ISIC2018) having a larger
learning distance from the public dataset. Compared to supervised pretraining,
SSP generates better low- and mid-level features Zhao et al. (2021), which are
more critical to the utility in the DP domain. We also find that further fine-
tuning these pretrained features with all labels of the public dataset
actually degrades the feature quality on these private datasets. The intra-
class invariance Zhao et al. (2021) introduced by the fine-tuning with public
dataset labels increases the class misalignment of private datasets.

(a) Crop (small distance)

(b) EuroSAT (medium distance)

(c) ISIC2018 (large distance)

Figure 2: The comparison of features trained by labeled and unlabeled ImageNet
on various out-of-domain complex private datasets.

###  3.3 A large public dataset is available

It is easier for a private classifier learned on the features trained by a
large public dataset to obtain high utility and small privacy overhead. We
adopted ImageNet-1K Deng et al. (2009) as our public dataset to train features
with and without labels. And then, we trained a private classifier using these
features on various private complex datasets Crop, EuroSAT, or ISIC2018. We
used ResNet50 as the backbone of SimCLRv2, and then the feature extractor for
private datasets. We also trained another ResNet50 with labels as our
baseline. The comparison between our scheme and baseline is shown in Figure 2.
For all three datasets, the features produced by SSP make the private
classifier obtain a much higher utility than our supervised-learning baseline
under the same privacy budget. A larger unlabeled public dataset greatly
improves the quality of features yielded by SSP.

##  4 Methods

Public datasets. We selected the “Image-B” in YM. et al. (2020), which has the
size equivalent to $\sim 132$ CIFAR images, due to its rich textures and
diversity. The image can be viewed in Appendix A.3. We applied a series of
aggressive data augmentations including cropping, scaling, rotation, contrast
changes, and adding noise on the image, and then created a synthetic dataset
having 50K CIFAR/mini-ImageNet-size images. The detailed parameters of these
augmentations can be viewed in YM. et al. (2020). Besides “Image-B”, we also
used CIFAR100, mini-ImageNet Vinyals et al. (2016), ImageNet Deng et al.
(2009), and PASS Asano et al. (2021) as public datasets.

Private datasets. We studied multiple private datasets such as CIFAR10/100
Krizhevsky et al. (2009), CropDiseases Mohanty et al. (2016b), EuroSAT Helber
et al. (2019), ISIC2018 Codella et al. (2018), and ImageNet.
CropDiseases/EuroSAT/ISIC2018 has a small/medium/large learning distance from
the public dataset mini-ImageNet.

Feature pretraining. We adopted SimCLRv2 Chen et al. (2020) to train a feature
extractor on different public datasets with no label. The backbone of SimCLRv2
uses a ResNet18, ResNet50, or HarmRN18 architecture. The pretraining of each
model lasts 200 epochs with a batch size of 64 or 128. After SSP, we used the
backbone as an feature extractor.

Supervised learning pretraining. We trained a supervised learning feature
extractor on various public datasets using the standard cross-entropy loss
with label smoothing. We set the label-smoothing parameter to 0.1.

DP-enabled training. We adopted multiple frameworks including DPSGD Abadi et
al. (2016), DPDFA Ohana et al. (2021); Lee and Kifer (2020), DPSGLD Bu et al.
(2021) and PATE Papernot et al. (2018) to train private classifiers. Only
DPDFA trains a 2-layer private MLP classifier, while the other DP-enabled
training frameworks train a 1-layer linear classifier.

Normalization layers. Models learned on handcrafted features and directly
trained on private datasets use only group normalization. All pretrained
features are produced with the BN parameters trained by only the public
dataset. Results related to BN can be found in Appendix A.2.

Hyper-parameter search. Similar to prior work Tramer and Boneh (2021); Luo et
al. (2021), we do not count the privacy leakage during hyper-parameter
searches on network architectures, optimizers, and hyper-parameters. We target
a moderate DP budget of $(\epsilon\leq 4.5,\delta=10^{-5})$ for private
ImageNet, and a small DP budget of $(\epsilon=0\sim 2,\delta=10^{-5})$ for the
other private datasets. We fixed the gradient clipping threshold to $C=0.1$ as
default, and tried different batch sizes $|B|$ and learning rates $\eta$. The
typical batch size we used is 4096, 8192, or 16384. For each set of hyper-
parameter, we ran the experiment for five times and report the average values.

Library. We implemented all privacy-related experiments by Opacus v1.1.1
Yousefpour et al. (2021), where native Poisson sampling is supported by
BatchMemoryManager. Private models compute per sample gradients, and the DP
optimizer does gradient clipping and noise addition.

##  5 Results

To measure a classifier’s utility for a range of privacy budgets, we compute
the test accuracy and the DP budget $\epsilon$ after each training epoch. For
a small DP budget $(\epsilon\leq 2,\delta=10^{-5})$, we studied the features
trained with a single image, the public mini-ImageNet dataset, and the public
ImageNet dataset. We also compared various DP-enabled training frameworks to
train a private classifier with these features on various private datasets.
Finally, we studied the utility of a private ImageNet-1K dataset with a
moderate privacy budget $(\epsilon\leq 4.5,\delta=10^{-5})$.

Table 6: The utility comparison of features trained by a single image. Public | Network | CIFAR10 | CIFAR100 | Crop | EuroSAT | ISIC2018
---|---|---|---|---|---|---
Dataset | Architecture | Utility ($\%$) under $\epsilon$-DP$=1$ | $2$
None | ScatterNet + CNN | 51.6 | 63.4 | 14.7 | 25.7 | 45.1 | 67.6 | 17.6 | 34.8 | 61.2 | 63.7
1 image | HarmRN18 |  61.1 | 64.8 |  22.3 | 27.1 |  70.1 | 74.3 |  80.1 | 84.5 |  65.1 | 67.2

###  5.1 No public dataset

When there is no public dataset, we trained a HarmRN18 network by SimCLRv2 on
a single image as our feature extractor. And then, we trained a private
1-layer classifier by DPSGD on various private datasets using features
produced by the HarmRN18 feature extractor. As Table 6 shows, the utility of
the private classifier is much higher than that of a 5-layer CNN learning
directly on the non-learned ScatterNet handcrafted features, when $\epsilon=1$
and $2$. It is difficult to directly learn on even the ScatterNet handcrafted
features due to clipped and noisy gradients during DPSGD. The advantage of the
features trained by a single image is particularly significant on private
datasets like Crop and EuroSAT. This suggests that leveraging features
generated by SSP is a scalable solution to differentially private learning
even when facing the lack of public datasets.

Table 7: The utility comparison of features trained by a public mini-ImageNet dataset. Public | Extractor | CIFAR10 | CIFAR100 | Crop | EuroSAT | ISIC2018
---|---|---|---|---|---|---
Dataset | Architecture | Utility ($\%$) under $\epsilon$-DP$=1$ | $2$
labeled | ResNet18 | 68.8 | 72.7 | 33.8 | 39.1 | 62.2 | 77 | 69.3 | 76.4 |  68.4 | 67.8
unlabeled | ResNet18 | 69.1 | 73 |  34.9 | 40.8 | 75.2 | 78.1 | 84.9 | 87.6 | 68.1 | 70.2
unlabeled | HarmRN18 |  69.5 | 73.5 | 34.3 | 40.4 |  76 | 79.5 |  85.2 | 88.8 | 67.7 | 69.8
Table 8: The utility comparison of features trained by a public ImageNet dataset. Public | Extractor | CIFAR10 | CIFAR100 | Crop | EuroSAT | ISIC2018
---|---|---|---|---|---|---
Dataset | Architecture | Utility ($\%$) under $\epsilon$-DP$=1$ | $2$
labeled | ResNet50 | 90.4 | 91.1 | 61.3 | 65.4 | 81.1 | 83.7 | 82.9 | 85.2 | 69.5 | 72.7
unlabeled | ResNet50 |  91.6 | 92.7 |  63.3 | 69.2 |  87.7 | 92.3 |  92.5 | 94.3 |  70.3 | 75.2

###  5.2 A moderate size public dataset

When a public mini-ImageNet dataset is available, we trained a HarmRN18
network and a ResNet18 network via SimCLRv2 as our feature extractors. We then
used their features to train two private 1-layer classifiers by DPSGD on
private CIFAR10, CIFAR100, Crop, EuroSAT, and ISIC2018. Compared to our
supervised-learning baseline, as Table 7 shows, the utility of the private
classifiers learned on features produced by SSP is higher than that learned
with labels except the ISIC2018 dataset having a large learning distance, when
$\epsilon=1$. Although HarmRN18 outperforms ResNet18 in the non-private domain
Ulicny et al. (2019), we do not find there is a significant difference between
them in the DP domain. Particularly, the self-supervised pretrained ResNet18
extractor generates slightly better features than HarmRN18 for private
CIFAR100 ($\epsilon=1$ and $2$) and ISIC2018 ($\epsilon=2$) datasets. When
pretrained on public mini-ImageNet, ResNet18 is strong enough for producing
features for various private datasets.

###  5.3 A large size public dataset

A public ImageNet dataset greatly improves the quality of pretrained features
for differentially private learning. We used ResNet50 as the backbone of
SimCLRv2 and our feature extractor. And then, we trained a private 1-layer
classifier by DPSGD on the features of various private datasets. As Table 8
highlights, the utility of the private classifier learned on features produced
by SSP is much higher than that learned with labels, when $\epsilon=1$ and
$2$. A large public dataset is the key to improving the utility and privacy
loss of differentially private learning.

###  5.4 Comparing various DP-enabled training frameworks

We compared DPSGD, DPSGLD, and DPDFA to train a private classifier using the
ResNet50 features pretrained on public mini-ImageNet and ImageNet for three
private datasets including Crop, EuroSAT, and ISIC2018. Since the utility of
PATE is much lower than the other three training frameworks under the same DP
budget, we excluded PATE in the comparison. DPSGD is the most widely used DP-
enabled training algorithm, so we use it as our baseline. As Figure 3 and 4
highlight, compared to DPSGD, DPSGLD obtains the same utility, but uses only a
smaller privacy budget $\epsilon\leq 1$. However, for a large privacy loss
$\epsilon>1$, DPSGD achieves very similar utility to DPSGLD. DPDFA typically
can obtain higher utility than the other two DP-enabled training frameworks on
the private datasets having a smaller learning distance from the public
dataset, i.e., Crop and EuroSAT, under the same privacy budget when
$\epsilon>0.5$. However, for private datasets having a large learning distance
from the public dataset, i.e., ISIC2018, DPDFA achieves only a lower utility
than DPSGD and DPSGLD under the same privacy budget.

(a) Crop (small distance)

(b) EuroSAT (medium distance)

(c) ISIC2018 (large distance)

Figure 3: The comparison of DPSGD, DPSGLD, and DPDFA when training with the
features produced by a public mini-ImageNet dataset.

(a) Crop (small distance)

(b) EuroSAT (medium distance)

(c) ISIC2018 (large distance)

Figure 4: The comparison of DPSGD, DPSGLD, and DPDFA when training with the
features produced by a public ImageNet dataset.

###  5.5 Studying the utility and privacy loss of private ImageNet-1K

Figure 5: The utility and privacy loss of private ImageNet-1K.

We adopted ResNet50 as the backbone of SimCLRv2, and trained it on a public
PASS dataset consisting of 1.4 million images with no label. The backbone is
used to generate features for private ImageNet. The features can yield 60.8%
top-1 non-private accuracy. We further trained a private classifier on these
features in the DP domain. We find that the gradient clipping threshold
$C=0.1$ only produces a top-1 utility $8.1\%$ even when $\epsilon=3$. So we
enlarged $C$ to 1, and adjusted the other DP parameters. The utility and
privacy overhead of the classifier on private ImageNet is shown in Figure 5.
When $\epsilon=3$, the top-1 utility of the classifier becomes 25.3%. We also
compared DPSGD and DPDFA to train our private classifier. Since PASS is still
too small to produce strong enough pretained features, DPDFA achieves a lower
utility than DPSGD under the same privacy budget. Although recent work Kurakin
et al. (2022) uses supervised-learning-based features to achieve a top-1
utility 47.9% on private ImageNet with $(\epsilon=10,\delta=10^{-6})$, its
top-1 utility is only 7.6% when $\epsilon=4.57$. Under a moderate privacy
budget ($\epsilon\leq 4.5$), the features generated by SSP greatly outperform
prior supervised-learning-based features.

##  6 Conclusion and Future Work

We have demonstrated that SSP is a simple yet scalable solution to
differentially private learning regardless of the size of available public
datasets. The features produced by SSP on a single image, or a moderate/large
size public dataset significantly outperform the features trained with labels
in the DP domain, let alone the non-learned handcrafted ScatterNet features.
Based on the learning distance from the public dataset and the privacy budget,
different private datasets may favor distinctive DP-enabled training
frameworks to train their private classifiers learned on features produced by
SSP. Although we report a non-trivial utility, i.e., 25.3%, for private
ImageNet-1K when $\epsilon=3$, more research efforts are needed to further
improve the utility of large-scale private datasets under a moderate privacy
budget ($\epsilon\leq 3$).

## References

  * Abadi et al. [2016] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In _ACM SIGSAC Conference on Computer and Communications Security_ , pages 308––318, 2016.
  * Agrawal and Awekar [2018] Sweta Agrawal and Amit Awekar.  Deep learning for detecting cyberbullying across multiple social media platforms.  In _European conference on information retrieval_ , pages 141–153, 2018.
  * Asano et al. [2021] Yuki M. Asano, Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi.  Pass: An imagenet replacement for self-supervised pretraining without humans.  _NeurIPS Track on Datasets and Benchmarks_ , 2021.
  * Bu et al. [2020] Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie Su.  Deep learning with gaussian differential privacy.  _Harvard Data Science Review_ , 2(3), 9 2020.
  * Bu et al. [2021] Zhiqi Bu, Qiyiwen Zhang, Kan Chen, and Qi Long.  Differentially private bayesian neural networks on accuracy, privacy and reliability.  In _ICML, Theory and Practice of Differential Privacy workshop_ , 2021\.
  * Carlini et al. [2021] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel.  Extracting training data from large language models.  In _USENIX Security Symposium_ , pages 2633–2650, August 2021.
  * Chaudhuri et al. [2011] Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate.  Differentially private empirical risk minimization.  _Journal of Machine Learning Research_ , 12(3), 2011.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  _Advances in neural information processing systems_ , 33:22243–22255, 2020.
  * Codella et al. [2018] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic).  In _IEEE International symposium on biomedical imaging_ , pages 168–172, 2018.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.
  * Fischer and Krauss [2018] Thomas Fischer and Christopher Krauss.  Deep learning with long short-term memory networks for financial market predictions.  _European Journal of Operational Research_ , 270(2):654–669, 2018.
  * Flanders [2009] Adam E Flanders.  Medical image and data sharing: are we there yet?  _Radiographics_ , 29(5):1247–1251, 2009.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Helber et al. [2019] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 12(7):2217–2226, 2019.
  * Krizhevsky et al. [2009] Alex Krizhevsky et al.  Learning multiple layers of features from tiny images.  _Technical Report_ , 2009.
  * Kurakin et al. [2022] Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta.  Toward training at imagenet scale with differential privacy.  _CoRR_ , abs/2201.12328, 2022.  URL https://arxiv.org/abs/2201.12328.
  * Lee and Kifer [2020] Jaewoo Lee and Daniel Kifer.  Differentially private deep learning with direct feedback alignment.  _CoRR_ , abs/2010.03701, 2020.
  * Luo et al. [2021] Zelun Luo, Daniel J. Wu, Ehsan Adeli, and Li Fei-Fei.  Scalable differential privacy with sparse network finetuning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5059–5068, June 2021.
  * Mironov [2017] Ilya Mironov.  Rényi differential privacy.  In _IEEE computer security foundations symposium_ , pages 263–275, 2017.
  * Mohanty et al. [2016a] Sharada P Mohanty, David P Hughes, and Marcel Salathé.  Using deep learning for image-based plant disease detection.  _Frontiers in plant science_ , 7:1419, 2016a.
  * Mohanty et al. [2016b] Sharada P Mohanty, David P Hughes, and Marcel Salathé.  Using deep learning for image-based plant disease detection.  _Frontiers in plant science_ , 7:1419, 2016b.
  * Mohapatra et al. [2021] Shubhankar Mohapatra, Sajin Sasy, Xi He, Gautam Kamath, and Om Thakkar.  The role of adaptive optimizers for honest private hyperparameter selection.  _CoRR_ , abs/2111.04906, 2021.
  * Nø kland [2016] Arild Nø kland.  Direct feedback alignment provides learning in deep neural networks.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 29, 2016\.
  * Ohana et al. [2021] Ruben Ohana, Hamlet Medina, Julien Launay, Alessandro Cappelli, Iacopo Poli, Liva Ralaivola, and Alain Rakotomamonjy.  Photonic differential privacy with direct feedback alignment.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 22010–22020, 2021.
  * Oyallon and Mallat [2015] Edouard Oyallon and Stephane Mallat.  Deep roto-translation scattering for object classification.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , June 2015.
  * Oyallon et al. [2018] Edouard Oyallon, Sergey Zagoruyko, Gabriel Huang, Nikos Komodakis, Simon Lacoste-Julien, Matthew Blaschko, and Eugene Belilovsky.  Scattering networks for hybrid representation learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(9):2208–2221, 2018.
  * Papernot et al. [2018] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson.  Scalable private learning with PATE.  In _International Conference on Learning Representations_ , 2018.
  * Shokri et al. [2017] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In _IEEE symposium on security and privacy (SP)_ , pages 3–18, 2017\.
  * Tajbakhsh et al. [2016] Nima Tajbakhsh, Jae Y Shin, Suryakanth R Gurudu, R Todd Hurst, Christopher B Kendall, Michael B Gotway, and Jianming Liang.  Convolutional neural networks for medical image analysis: Full training or fine tuning?  _IEEE transactions on medical imaging_ , 35(5):1299–1312, 2016.
  * Tramer and Boneh [2021] Florian Tramer and Dan Boneh.  Differentially private learning needs better features (or much more data).  In _International Conference on Learning Representations_ , 2021.
  * Tschandl et al. [2018] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  _Scientific data_ , 5(1):1–9, 2018.
  * Ulicny et al. [2019] Matej Ulicny, Vladimir A Krylov, and Rozenn Dahyot.  Harmonic networks for image classification.  In _British Machine Vision Conference_ , 2019.
  * Vinyals et al. [2016] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Wang et al. [2015] Yu-Xiang Wang, Stephen Fienberg, and Alex Smola.  Privacy for free: Posterior sampling and stochastic gradient monte carlo.  In _International Conference on Machine Learning_ , pages 2493–2502, 2015.
  * Welling and Teh [2011] Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _International Conference on Machine Learning_ , pages 681–688, 2011.
  * YM. et al. [2020] Asano YM., Rupprecht C., and Vedaldi A.  A critical analysis of self-supervision, or what we can learn from a single image.  In _International Conference on Learning Representations_ , 2020.
  * Yousefpour et al. [2021] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov.  Opacus: User-friendly differential privacy library in pytorch.  In _NeurIPS 2021 Workshop Privacy in Machine Learning_ , 2021.
  * Zhao et al. [2021] Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin.  What makes instance discrimination good for transfer learning?  In _International Conference on Learning Representations_ , 2021.

##  Appendix A Appendix

###  A.1 Rényi DP

Rényi DP (RDP) Mironov [2017] is a generalization of $(\epsilon,\delta)$-DP
that uses Rényi divergence as a distance metric. The Rényi divergence of order
$\alpha$ between two distributions $P$ and $Q$ is defined as:

$\displaystyle D_{\alpha}(P||Q)=\frac{1}{\alpha-1}\log\mathbb{E}_{x\sim
P}\left[\left(\frac{P(x)}{Q(x)}\right)^{\alpha-1}\right]$

A model satisfies $(\alpha,\epsilon)$-RDP if

$\displaystyle
D_{\alpha}(M(D)||M(D^{\prime}))=\frac{1}{\alpha-1}\log\mathbb{E}_{x\sim
M(D)}\left[\left(\frac{\text{\bf Pr}[M(D)=x]}{\text{\bf
Pr}[M(D^{\prime})=x]}\right)^{\alpha-1}\right]$

Pure $(\epsilon,0)$-DP is equivalent to $(\infty,\epsilon)$-RDP. And if a
model $M$ satisfies $(\alpha,\epsilon)$-RDP, $M$ also satisfies for any
$\delta\in(0,1)$. RDP during the training of a model is enforced by two
components: per-sample gradients are clipped at a fixed L2 norm threshold $C$,
and Gaussian noise of magnitude $\sigma^{2}C^{2}$ is added to the gradient
updates for a noise scale parameter $\sigma$.

Table 9: The utility and privacy comparison of various schemes using batch normalization on a private CIFAR10 dataset. DP-SOTA-1 means Tramer and Boneh [2021], and DP-SOTA-2 indicates Luo et al. [2021]. Public Dataset | Scheme |  $\epsilon$-DP | Network | Training | Accuracy (%)
---|---|---|---|---|---
None | DP-SOTA-1 | 3 | Scat + CNN | DPSGD | 69.3
1 image | Ours | 3 | HarmRN18 | DPSGD | 71.1
labeled CIFAR100 | DP-SOTA-2 | 1.5 | ResNet18 | DPSGD | 81.6
unlabeled CIFAR100 | Ours | 1.5 | ResNet18 | DPSGD | 80.7

###  A.2 Private data normalization

Prior work Tramer and Boneh [2021], Luo et al. [2021] finds batch
normalization (BN) greatly improves convergence on complex private datasets.
Private BN Tramer and Boneh [2021] is proposed to compute private estimates of
the per-channel mean and variance of the ScatterNet features. It is difficult
to use the same set of hyper-parameters to train BN layers and the other
private components in a network. Different values for noise multiplier Tramer
and Boneh [2021] or learning rate Luo et al. [2021] are used to train BN
layers. Moreover, most DP training libraries do not support BN layers in a
private model yet. We adopt the same method and hyper-parameters to train BN
layers as prior work Tramer and Boneh [2021], Luo et al. [2021]. The utility
and privacy comparison between our schemes and prior work is shown in Table 9.
The features trained by HarmRN18 using BN on a single image obtain better
utility than the ScatterNet non-learnable handcrafted features. However, the
features trained by ResNet18 on a public unlabeled CIFAR100 achieve a slightly
worse utility than those trained with labels Luo et al. [2021]. This is
because besides BN layers, Luo et al. [2021] also fine-tunes convolutional
layers in the DP domain.

###  A.3 Learning on a single image

Figure 6: Single-image self-supervision.

Recent work YM. et al. [2020] shows the self-supervised learning methods such
as BiGAN, RotNet, and DeepCluster can be used to train the first few layers of
a deep network model using a single training image, when sufficient data
augmentation is used. We select the “Image-B”, which is shown as Figure 6, due
to its rich texture and high diversity. The image size is $600\times 225$. We
also try the “Image-A” and the “Image-C” from YM. et al. [2020] in our
experiments. Among three images, “Image-B” achieves the best utility and
privacy budget for most private classifiers.

Table 10: The utility comparison of classifiers consisting of different numbers of layers. Public Dataset | Private Dataset | Arch + Train | $\epsilon$ | Utility ($\%$)
---|---|---|---|---
|  | 1-layer DPSGD | $1.5$ | 75.5
|  | 2-layer DPSGD | $1.5$ | 73.3
|  | 1-layer DPSGLD | $1.5$ | 74.2
CIFAR100 | CIFAR10 | 2-layer DPSGLD | $1.5$ | 72.8
|  | 1-layer PATE | $16$ | 70.2
|  | 2-layer PATE | $16$ | 64.7
|  | 2-layer DPDFA | $1.5$ | 78.2

###  A.4 Single-layer and 2-layer classifiers

In order to study DPDFA, we need to use a 2-layer linear MLP to serve as our
private classifier, where the first layer uses Tanh activations and the second
layer uses a Sigmoid activation. A natural question is “how does DPSGD work
with a 2-layer MLP?”. As Table 10 shows, we find that unlike DPDFA, all the
other DP-enabled training frameworks work better with a single-layer linear
classifier. For instance, a 2-layer MLP-based private classifier is always
overtopped by a 1-layer classifier when trained by DPSGD. Therefore, in this
paper, we always adopt a 2-layer classifier for DPDFA, and a 1-layer
classifier for the other DP-enabled training frameworks.