# Optical spectroscopy and ultrafast pump-probe study of a quasi-one-
dimensional charge density wave in CuTe

R. S. Li  International Center for Quantum Materials, School of Physics,
Peking University, Beijing 100871, China  L. Yue  International Center for
Quantum Materials, School of Physics, Peking University, Beijing 100871, China
Q. Wu  International Center for Quantum Materials, School of Physics, Peking
University, Beijing 100871, China  S. X. Xu  International Center for Quantum
Materials, School of Physics, Peking University, Beijing 100871, China  Q. M.
Liu  International Center for Quantum Materials, School of Physics, Peking
University, Beijing 100871, China  Z. X. Wang  International Center for
Quantum Materials, School of Physics, Peking University, Beijing 100871, China
T. C. Hu  International Center for Quantum Materials, School of Physics,
Peking University, Beijing 100871, China  X. Y. Zhuo  International Center for
Quantum Materials, School of Physics, Peking University, Beijing 100871, China
L. Y. Shi  International Center for Quantum Materials, School of Physics,
Peking University, Beijing 100871, China  S. J. Zhang  International Center
for Quantum Materials, School of Physics, Peking University, Beijing 100871,
China  D. Wu  Songshan Lake Materials Laboratory, Dongguan, Guangdong 523808,
China  T. Dong  International Center for Quantum Materials, School of Physics,
Peking University, Beijing 100871, China  N. L. Wang<EMAIL_ADDRESS>International Center for Quantum Materials, School of Physics, Peking
University, Beijing 100871, China  Beijing Academy of Quantum Information
Sciences, Beijing 100913, China

###### Abstract

CuTe is a two-dimensional (2D) layered material, yet forming a quasi-one-
dimensional (quasi-1D) charge-density-wave (CDW) along the a-axis in the ab-
plane at high transition temperature $T_{CDW}=335$ K. However, the anisotropic
properties of CuTe remain to be explored. Here we performed combined
transport, polarized infrared reflectivity, and ultrafast pump-probe
spectroscopy to investigate the underlying CDW physics of CuTe. Polarized
optical measurement clearly revealed that an energy gap gradually forms along
the a-axis upon cooling, while optical evidence of gap signature is absent
along the b-axis, suggesting pronounced electronic anisotropy in this quasi-2D
material. Time-resolved optical reflectivity measurement revealed that the
amplitude and relaxation time of photo-excited quasiparticles change
dramatically across the CDW phase transition. Taking fast Fourier
transformation of the oscillation signals arising from collective excitations,
we identify the 1.65-THz mode as the CDW amplitude mode, whose energy softens
gradually at elevated temperatures. Consequently, we provide further evidence
for the formation of completely anisotropic CDW order in CuTe, which is quite
rare in quasi-2D materials.

###### pacs:

Valid PACS appear here

##  I INTRODUCTION

Charge density waves (CDWs) have been of great interest in condensed matter
physics for several decades. As the name implies, the conducting electrons
form a new periodic modulation with a period that is commensurate or
incommensurate with the underlying lattice. This phenomenon is usually induced
by Fermi surface nesting (FSN), where the Fermi surface tends to nest between
nearly parallel parts connected by a vector $\textbf{q}=2\textbf{k}_{F}$,
which causes the electronic susceptibility to diverge at the nesting wave
vector Peierls and Peierls (1955). Meanwhile, the lattice responds to the
electronic condensate through electron-phonon coupling. At
$\textbf{q}=2\textbf{k}_{F}$, acoustic phonon tends to be softened to zero
frequency, which further gives rise to the periodic modulation of the lattice
structure Gruner (2018); Ando  _et al._ (2005); Johannes and Mazin (2008);
Laverock  _et al._ (2005). In the CDW state, a single-particle energy gap will
open in the nested region of the Fermi surface, leading to the lowering of the
electronic energy of the system. Therefore the experimental characterization
of the single-particle energy gap would be a key step in identifying the CDW
state, which can be realized by optical conductivity spectroscopy. Besides
single-particle excitation, CDWs also have collective excitations consisting
of amplitude mode and phase mode. The amplitude mode corresponds to the ionic
displacement with a finite gap at $\textbf{q}=0$ limit and behaves as optical
phonons Scott (1974); Du  _et al._ (2014). In this regard, measurement of
amplitude mode can also be important evidence of CDW phase transition, which
has been verified by both pump-probe and Raman spectroscopy Chen  _et al._
(2017); Lin  _et al._ (2020); Tsang  _et al._ (1976); Hu  _et al._ (2015).

Generally, CDWs tend to exist in low-dimensional systems. Typical examples
include K0.3MoO3 Pouget  _et al._ (1991), transition-metal metal
dichalcogenides Tsang  _et al._ (1976); Smith  _et al._ (1985), rare-earth
tritellurides Moore  _et al._ (2010); Chen  _et al._ (2014), etc. Among all
these materials, the underlying CDW mechanism matches very well with the
established theory, providing standard systems to study new CDW materials.
Recently, the compound CuTe was reported to be a new CDW material, which has
attracted attention for it is a rare 2D layered material hosting quasi-1D CDW
order. Its structure is similar to the well-known prototype Fe-based
superconductor FeSe. There exists a small difference between in-plane lattice
parameters a and b, resulting in an orthorhombic lattice structure. It
exhibits period modulation corresponding to Te chains at low temperature,
indicating a CDW phase transition Stolze  _et al._ (2013). Angle-resolved
photoemission spectroscopy (ARPES) has identified CuTe as a CDW metal with an
anisotropic gap below transition temperature 335 K Zhang  _et al._ (2018). The
nested Fermi surface that drives CDW transition is observed to emerge from
nearly parallel line Fermi segments, which is connected by CDW wave vector
$\textbf{q}_{x}=0.4a^{*}$. Based on the Seebeck coefficient, thermal
conductivity, and specific heat measurement, Kou et al. reported that CDW
order can have a great influence on the thermal transport properties, due to
strong electron-phonon coupling Kuo  _et al._ (2020). Additionally, by
application of external pressure, the CDW transition temperature $T_{CDW}$ of
CuTe decreases gradually Wang  _et al._ (2021). With increasing pressure,
superconductivity emerges at 2.4 K after the CDW gets suppressed completely,
suggesting that the CDW order competes with superconductivity in CuTe Wang
_et al._ (2021). Despite these experimental results, the electrical and
optical response of CuTe along the b-axis remains unclear. It is highly
interesting to investigate how the CDW phase transition affects the
anisotropic properties of the 2D layered material, in particular on the
transport and optical conductivity properties.

In this work, we use a combination of transport, optical spectroscopy, and
ultrafast pump-probe reflectivity study on the single crystalline CuTe.
Optical spectroscopy is a valuable experimental method that probes the bulk
properties of materials, including charge dynamics and electronic band
structure of a system. On the other hand, ultrafast spectroscopy can probe
coherent vibrational dynamics with high resolution to approach a low energy
scale. Furthermore, the energy gap arising from CDW order plays an important
role in the relaxation process of quasiparticles. The relaxation time and
amplitude of quasiparticles show strong enhancement near transition
temperature, which could also be probed from the pump-probe measurement. From
optical spectroscopy, we find that there appears an energy gap along a-axis
which dramatically reduces the carrier density across $T_{CDW}$, while it
behaves as a usual metal along the b-axis without showing anomaly below
$T_{CDW}$. The amplitude and the relaxation of photoinduced carriers exhibit a
quasidivergence behavior, yielding further evidence for CDW energy gap
formation. Moreover, photoinduced reflectivity displays strong oscillation
signals on top of the decay process at low temperatures. One of the
oscillations has a much higher signal intensity and its energy softens by over
18$\%$ upon heating from 4.6 K to 280 K. The temperature dependence of the
oscillation suggests that it arises from the amplitude mode of CDW collective
excitation.

##  II EXPERIMENTAL RESULTS

Single crystals of CuTe were grown by melting stoichiometric amounts of Cu and
Te at 500∘C. After 24 h, the melt was cooled to 400 ∘C at a rate of 4 ∘ C/h.
Then the furnace was shut down and cooled to room temperature. The crystal has
a flake-like shape with a shining a-b plane as the cleavage plane, consistent
with its quasi-2D structure. We can identify crystallographic a-axis
perpendicular to the strip-like structure, determined using a Laue X-ray
diffractometer.

Figure 1: (Color online) Crystal structures of CuTe. (a) Top view and (b) side
view of the non-CDW phase. Arrows indicate the movement of Te atoms in the CDW
phase.

CuTe is composed of a stack of edge-sharing CuTe4-tetrahedra layer by layer
with space group Pmmn and orthorhombic lattice constants of $a=3.124$
$\mathrm{\AA}$, $b=4.086$ $\mathrm{\AA}$, $c=6.946$ $\mathrm{\AA}$ Seong  _et
al._ (1994). Figure. 1(a) shows the top view of the crystal of CuTe projected
on the ab-plane, where Cu atoms and Te atoms form a rectangle shape lattice.
In the normal state, the Te atoms form a quasi-1D chain along the a-axis where
the bond length is shorter than that along the b-axis, which runs above and
below the puckered Cu layers with an equal Te-Te distance as shown in Fig.
1(b). Below transition temperature, units of two or three Te atoms (marked by
red dashed line) with shorter distances (movement of atoms indicated by black
arrows in Fig. 1(b)) alternate with single Te atom along a-axis. Meanwhile,
the modulation of the Te position forces corrugated Cu layers to have a slight
distortion along the c-axis, resulting in a 5$\times$1$\times$2 superstructure
Stolze  _et al._ (2013).

Figure 2: (Color online) (a) The temperature-dependent resistivity of CuTe
measured with currents flowing along the crystallographic a- and b-axes,
normalized at 380K. (b) Optical reflectivity R($\omega$) in a broad range at
345K, along the a- and b-axes, respectively. Figure 3: (Color online) The
temperature-dependent optical reflectivity R($\omega$) along the a-(a) and
b-axes (b), respectively. The optical conductivity $\sigma_{1}$ at different
temperatures along the a-(c) and b-axes (d), respectively. The inset of (c)
displays $\sigma_{1}(\omega)$ along the a-axis at 10 K together with a Drude-
Lorentz fit.

The temperature-dependent resistivity measurement was performed with a
standard four-probe method using a Quantum Design PPMS. Figure 2(a) displays
the resistivity with currents flowing along the crystallographic a- and
b-axes, respectively. When the electric current flows along the a-axis, the
resistivity shows metallic behavior in the entire measured range except a
sudden upturn appears at 335 K. This resistivity anomaly is a result of CDW
transition, which is consistent with the previous report Zhang  _et al._
(2018). In contrast, we didn't find a resistivity anomaly along the b-axis
within our measurement resolution. The resistivity data suggest that the CDW
order is completely absent in the b-axis, which is quite rare in a highly 2D
system.

We performed polarized optical reflectance measurement with the
$\textbf{E}\parallel$ the a-axis and the $\textbf{E}\parallel$ b-axis,
respectively, on a Bruke 80V Fourier transform infrared spectrometer in the
frequency range from 30 to 30000 cm-1. An in-suit gold and aluminum
overcoating technique was used to get the reflectivity R($\omega$). At 345 K,
the reflectivity along the a-axis and b-axis over a broad frequency is plotted
in Fig. 2(b). Both directions show good metallic behavior in agreement with
the dc-resistivity measurement, since reflectivity reaches almost unity at low
frequency with a large plasma edge. Note that, the reflectivity shows a
tremendous difference in edge frequency, suggesting that this compound is
highly anisotropic. The edge frequency, being referred to usually as the
"screened" plasma frequency, is related to the density $n$ and effective mass
$m^{*}$ of free carriers by $\omega_{p}^{{}^{\prime}2}\propto n/m^{*}$. The
larger plasma frequency along the a-axis is consistent with the previous
report that the conductivity along the chain direction is larger than
conductivity perpendicular to the chain at room temperature Grüner (1988).

The temperature-dependent reflectivity along the a- and b-axes is displayed in
Fig. 3(a) and 3(b), respectively. Let us first discuss optical response along
the a-axis. Firstly, the low-energy reflectivity reaches almost unity.
Additionally, when temperature decreases, the low-energy reflectivity
increases. Both features show that the sample has a highly metallic nature, in
agreement with the dc-resistivity measurement. When the temperature decreases,
the most significant feature in spectra $R(\omega)$ is the substantial
suppression in the mid-infrared region, which is a typical optical signature
of a charge gap formation corresponding to CDW phase transition. The
reflectivity exhibits a steep plasma edge around 1800 cm-1, revealing that the
density of free carriers is substantially suppressed. In the meantime, the
scattering rate is also significantly reduced. The dip becomes more dramatic
at low temperatures, suggesting that the energy gap is further enhanced.

Figure 4: (Color online) (a) and (b) The photoinduced reflectivity $\Delta
R/R$ along the a-axis as a function of time delay at several selected
temperatures. Lines in panel (a) are vertically shifted for clarity. (c) and
(d) The amplitude A and relaxation time $\tau$ of the fast decay process
extracted from the exponential fitting to the experimental data of $\Delta
R/R$ at different temperatures. Error bars represent the standard deviation of
the fit. The red curves are fitting curves from Eq. 2 and 3 below the phase
transition temperatures. (e) The oscillation part of $\Delta R/R$ at 4.6 K
after subtracting the decay background. The inset presents the fast Fourier
transformation of the oscillation. (f) The temperature-dependent frequencies
of the CDW amplitude mode. The red dashed line is a guide to the eye.

The real part of conductivity was derived from R($\omega$) through Kramers-
Kronig transformation, as shown in Fig. 3(c). The conductivity consists of a
Drude peak at zero-frequency and several other Lorentz peaks at a finite
frequency in the whole measurement temperature range. At high temperatures,
the Drude peak is rather broad demonstrating a large scattering rate of the
free carriers. Upon entering the low-temperature phase, the spectral weight of
the Drude peak is substantially removed and transfers to higher energies to
form a broad peak around 4000 cm-1. The Drude peak narrows continuously with
decreasing T, implying suppressed quasiparticle scattering. Meanwhile, the
positions of the Lorentz peak move to a higher frequency as a result of the
further enhancement of the energy gap. In fact, these behaviors are supposed
to be the contribution of the case I coherent factor for density wave order
Dressel and Grüner (2002). We can roughly estimate the scale of the CDW gap
through the edge position of the peak, which locates at 4000 cm-1(500 meV). In
an earlier ARPES measurement, an anisotropic energy gap with a maximum value
of 190 meV was identified. It should be noted that ARPES measures the gap
relative to the Fermi level, while optical spectroscopy measures the interband
transition from the occupied state to the unoccupied state above the Fermi
level. The gap value determined by optics usually doubles the energy gap
obtained by ARPES if the Fermi level is in the center of the gap.
Nevertheless, the energy gap determined by optics can only be taken as a rough
estimation because it is hard to determine accurately the conductivity edge.
Taking the transitions temperature $T_{c}$=335 K, we can get the ratio
$2\Delta/k_{B}T_{c}=17$, which is similar to that reported in other CDW
materials, e.g., TbTe3 Chen  _et al._ (2014), and is much larger than the weak
coupling theory prediction Gruner (2018).

To get more information quantitatively, we use the Drude-Lorentz model to
decompose the conductivity into different components:

$\displaystyle\sigma_{1}(\omega)=\frac{\omega_{p}^{2}}{4\pi}\frac{\Gamma_{D}}{\omega^{2}+\Gamma_{D}^{2}}+\sum_{j}\frac{S_{j}^{2}}{4\pi}\frac{\Gamma_{j}\omega^{2}}{(\omega_{j}^{2}-\omega^{2})^{2}+\omega^{2}\Gamma_{j}^{2}}$
(1)

The first term is Drude components attributed to the intraband transition of
itinerant carriers, while the second term is Lorentz components correspond to
the interband transition and excitation across energy gaps Lin  _et al._
(2020). The inset of Fig. 3(c) presents the fitting results along a-axis at 10
K. The spectrum of 345 K can be well reproduced by one Drude and one Lorentz
term. At low temperature, one additional Lorentz term needs to be added to
reproduce the peak at 4500 cm-1. As mentioned above , the variation of plasma
frequency $\omega_{p}$ is proportional to $\sqrt{n/m^{*}}$. From our fitting
date, the plasma frequency varies roughly from 48000 cm-1 at 345 K to 22000
cm-1 at 10 K. Assuming that the effective mass of free carriers is constant at
different temperatures, the reduction of the plasma frequency indicates that
78$\%$ free carriers are removed by the opening of CDW energy gap. Meanwhile,
the scattering rate $\gamma$, i.e. the width of the Drude peak, decreases
dramatically from 2100 cm-1 to 265 cm -1 at low temperature. Consequently, it
is reasonable that the compound still behaves like metal even though more than
half of free carriers are lost due to the formation of a CDW energy gap.

On the contrary, the reflectivity and the real part of optical conductivity
along the b-axis, as shown in Fig. 3(b) and (d), do not exhibit these
behaviors, but just behave like a simple metal at all temperatures
investigated. The scattering rate decreases monotonously upon cooling. No
signatures of gap formation along the b-axis in optical spectroscopy,
consisting with the CDW formation along the chain direction.

In order to get further information on the CDW transition, we performed
ultrafast pump-probe measurement on this compound, which has been proven to be
particularly powerful in detecting both the single-particle excitations across
small energy gaps Demsar  _et al._ (1999); Chia  _et al._ (2010) and the
collective excitations relevant to long-range ordering Yusupov  _et al._
(2008); Albrecht  _et al._ (1992); Qi  _et al._ (2013). We used a Ti: sapphire
oscillator as the light source for both pump and probe beams, which can
produce 800 nm laser pulses with 100 fs width and 80 MHz repetition rate. The
fluence of the pump beam is about 4 $\mu$J/cm2, while the fluence of the probe
beam is ten times lower. The polarization direction of the pump and probe
pulse are perpendicular to each other, in order to eliminate the interference.
We measured photoinduced transient reflectivity $\Delta R/R$ with the electric
field E direction of the probe beam parallel to the crystallographic a-axis.

The time-dependent $\Delta R/R$ along the a-axis at several selected
temperatures are displayed in Fig. 4(a) and 4(b). Each spectrum initially
increases in a very short time induced by the pump excitation, then drops back
to the equilibrium state within picoseconds. The relaxation process can be
described by a single exponential decay and a constant $\Delta
R/R=A\mathrm{exp}(-t/\tau)+C$, where $A$ represents the amplitude of the
photoinduced reflectivity change and $\tau$ stands for the relaxation time for
photon excited carriers decaying to their original states. The constant C
indicates a long relaxation time arising from thermal diffusion, where energy
from the electrons transforms to the environment.

Above the transition temperature, the amplitude A and relaxation time $\tau$
depend mildly on temperature, while both of them are strongly enhanced by
entering the CDW state. The temperature-dependent $A$ and $\tau$ can be
extracted by single exponential fitting for various temperatures, which is
plotted in Fig. 4(c) and 4(d), respectively. The amplitude and relaxation time
display anomalies near the transition temperature. We also tried to fit the
$\Delta R/R$ using a convolution between an exponential decay function and a
Gaussian response function and found that the fitting results were almost
unchanged. The amplitude A drops sharply, while the relaxation time $\tau$
shows a continuous divergence at 330 K. This anomaly behavior of the
relaxation process can arise from the energy gap opening, as will be explained
in detail later.

Here, we employ the Rothwarf-Taylor (RT) model to interpret the decay of the
photoexcited quasiparticles (QPs) Rothwarf and Taylor (1967). The RT model was
first proposed to resolve the ultrafast relaxation mechanism in
superconductivity. It was proved to be applicable for a wide range of
materials with a gap opening in the density of states. Photoexcitation would
generate a large number of QPs, which would decay via electron-electron or
electron-phonon interactions towards to initial equilibrium state. The
formation of an energy gap would introduce a bottleneck effect, which
significantly impedes the relaxation of QPs. The bottleneck effect is that,
when QPs recombine across an energy gap, high-energy phonons would be
generated, which could also induce a large number QPs in return, leading to a
long relaxation time. Based on this model, the density of thermally activated
quasiparticles $n_{T}$ can be obtained via the amplitude $A(T)$,
$n_{T}\propto[A(T)/A(T\rightarrow 0)]^{-1}-1$. Considering the relationship
between $n_{T}$ and $\Delta(T)$,
$n_{T}\propto\sqrt{\Delta(T)T}exp[-\Delta(T)/T]$, where
$\Delta(T)=\Delta(0)\sqrt{1-T/T_{c}}$, we can get Kabanov  _et al._ (1999)

$\displaystyle
A(T)\propto\frac{\Phi/(\Delta(T)+k_{B}T/2)}{1+\gamma\sqrt{2k_{B}T/\Delta(T)}exp[-\Delta(T)/k_{B}T]}$
(2)

where $\Phi$ is the pump fluence and $\gamma$ is a fitting parameter.

As temperature rises, more and more low-energy phonons become available to
break condensed carriers with a reduction of the energy gap, which causes the
relaxation time of QPs to diverge at the transition temperature. In the RT
model, the relaxation rate near $T_{c}$ is dominated by the energy transfer
from high frequency phonons with $\omega>2\Delta$ to low frequency phonons
with $\omega<2\Delta$. The phonon relaxation rate can be expressed as Kabanov
_et al._ (1999)

$\displaystyle\frac{1}{\tau}=\frac{12\Gamma_{\omega}k_{B}T^{{}^{\prime}}\Delta(T)}{\hbar\omega^{2}}$
(3)

where $\Gamma_{\omega}$ is Raman phonon linewidth and $T^{{}^{\prime}}$ is the
QPs temperature. Apparently, $\tau(T)\propto\Delta(T)^{-1}$ in the vicinity of
$T_{c}$; thus a divergence is expected at the phase transition temperature
where the energy gap starts to open. Qualitatively, these equations can
reproduce the measurement results around $T_{c}$. In the high temperature
limit, we take $\omega=1.65$ THz as the phonons frequency; $T^{\prime}=330$ K
as the QP temperature. The best fits yield the phenomenological fitting
parameter $\gamma\approx 100$ and $\Gamma_{\omega}\approx 10$ $cm^{-1}$, as
shown in Fig. 4(c), (d).

On the other hand, the transient reflectivity $\Delta R/R$ exhibits pronounced
oscillatory signals at low temperatures arising from coherent phonon. The
lifetime and periods of coherent phonon change with temperature variation. To
analyze the component quantitatively, we first subtract the exponential
fitting part, then take a fast Fourier transformation of the residual
oscillation part. As an example, Fig. 4(e) displays $\Delta R/R$ data at 4.6 K
as a function of time delay within 25 ps after subtracting the decay
background. The inset shows the Fourier transform of the residual part. Five
distinct peaks are observed in the Fourier spectrum at 1.32, 1.65, 1.96, 2.31,
2.59 THz. Based on the previous reports, the strength of CDW amplitude mode is
much large than other phonon modes, and its frequency would exhibit softening
on approaching $T_{c}$. Indeed, here the mode at 1.65 THz has a much larger
strength and shows significant softening at elevated temperature as shown in
Fig. 4(f). Compared with the 1.65-THz mode, the other 4 modes decay rapidly
with the increase of temperature. We can no longer resolve them above 150 K,
while their energy keeps nearly unchanged within our measurement resolution.
Therefore, we identify the 1.65-THz phonon as a CDW amplitude mode. In
principle, the 1.65-THz phonon should be softened to zero at the phase
transition temperature. However, since the phonon mode is heavily damped at
elevated temperatures as reported before, the CDW amplitude mode only softens
to 1.34 THz, and it could not be resolved above 280 K.

##  III CONCLUSION

To conclude, by combining transport, optical spectroscopy, and ultrafast pump-
probe spectroscopy, we provide further evidence of anisotropic quasi-1D CDW
phase transition in CuTe single crystal. A strong anomaly near 335 K in the
resistivity is seen only along the a-axis, while the feature is completely
absent in the b-axis. Our polarized optical measurement shows that the optical
responses along a- and b-axes are very different, where the plasma frequency
with $\textbf{E}\parallel$ a-axis is much higher than that
$\textbf{E}\parallel$ b-axis. When $\textbf{E}\parallel$ a-axis, the real part
of optical conductivity clearly reveals the formation of an energy gap with
associated spectral weight change. On the contrary, the optical response
behaves like a simple metal at all measured temperatures for
$\textbf{E}\parallel$ b-axis. Furthermore, the amplitude and relaxation time
extracted from the photoinduced reflectivity shows a sudden enhancement,
consistent well with the RT model prediction, which also confirms the opening
of a single-particle energy gap. Meanwhile, we identify the CDW amplitude mode
related to the collective excitation of CDW. The spectroscopic information
revealed by our anisotropic optical measurement should be very helpful for
understanding the electronic properties and related physics. Taking account of
the rather high CDW phase transition temperature (above room temperature),
CuTe offered an ideal system to study the CDW physics as well as its interplay
with superconductivity.

## ACKNOWLEDGMENTS

This work was supported by the National Key Research and Development Program
of China (No. 2017YFA0302904) and National Natural Science Foundation of China
(No. 11888101).

## References

  * Peierls and Peierls (1955) R. E. Peierls and R. S. Peierls,  _Quantum theory of solids_ (Oxford University Press, 1955).
  * Gruner (2018) G. Gruner,  _Density waves in solids_ (CRC press, 2018).
  * Ando  _et al._ (2005) H. Ando, T. Yokoya, K. Ishizaka, S. Tsuda, T. Kiss, S. Shin, T. Eguchi, M. Nohara, and H. Takagi, Journal of Physics: Condensed Matter 17, 4935 (2005).
  * Johannes and Mazin (2008) M. D. Johannes and I. I. Mazin, Phys. Rev. B 77, 165135 (2008).
  * Laverock  _et al._ (2005) J. Laverock, S. B. Dugdale, Z. Major, M. A. Alam, N. Ru, I. R. Fisher, G. Santi, and E. Bruno, Phys. Rev. B 71, 085114 (2005).
  * Scott (1974) J. F. Scott, Rev. Mod. Phys. 46, 83 (1974).
  * Du  _et al._ (2014) X. Du, R. Yuan, L. Duan, C. Wang, Y. Hu, and Y. Li, Phys. Rev. B 90, 104414 (2014).
  * Chen  _et al._ (2017) R. Y. Chen, S. J. Zhang, M. Y. Zhang, T. Dong, and N. L. Wang, Phys. Rev. Lett. 118, 107402 (2017).
  * Lin  _et al._ (2020) T. Lin, L. Y. Shi, Z. X. Wang, S. J. Zhang, Q. M. Liu, T. C. Hu, T. Dong, D. Wu, and N. L. Wang, Phys. Rev. B 101, 205112 (2020).
  * Tsang  _et al._ (1976) J. C. Tsang, J. E. Smith, and M. W. Shafer, Phys. Rev. Lett. 37, 1407 (1976).
  * Hu  _et al._ (2015) Y. Hu, F. Zheng, X. Ren, J. Feng, and Y. Li, Phys. Rev. B 91, 144502 (2015).
  * Pouget  _et al._ (1991) J. P. Pouget, B. Hennion, C. Escribe-Filippini, and M. Sato, Phys. Rev. B 43, 8421 (1991).
  * Smith  _et al._ (1985) N. V. Smith, S. D. Kevan, and F. J. DiSalvo, Journal of Physics C: Solid State Physics 18, 3175 (1985).
  * Moore  _et al._ (2010) R. G. Moore, V. Brouet, R. He, D. H. Lu, N. Ru, J.-H. Chu, I. R. Fisher, and Z.-X. Shen, Phys. Rev. B 81, 073102 (2010).
  * Chen  _et al._ (2014) R. Y. Chen, B. F. Hu, T. Dong, and N. L. Wang, Phys. Rev. B 89, 075114 (2014).
  * Stolze  _et al._ (2013) K. Stolze, A. Isaeva, F. Nitsche, U. Burkhardt, H. Lichte, D. Wolf, and T. Doert, Angewandte Chemie International Edition 52, 862 (2013).
  * Zhang  _et al._ (2018) K. Zhang, X. Liu, H. Zhang, K. Deng, M. Yan, W. Yao, M. Zheng, E. F. Schwier, K. Shimada, J. D. Denlinger, Y. Wu, W. Duan, and S. Zhou, Phys. Rev. Lett. 121, 206402 (2018).
  * Kuo  _et al._ (2020) C. N. Kuo, R. Y. Huang, Y. K. Kuo, and C. S. Lue, Phys. Rev. B 102, 155137 (2020).
  * Wang  _et al._ (2021) S. Wang, X. Chen, C. An, Y. Zhou, Y. Zhou, C. Gu, L. Zhang, X. Yang, and Z. Yang, Phys. Rev. B 103, 134518 (2021).
  * Seong  _et al._ (1994) S. Seong, T. A. Albright, X. Zhang, and M. Kanatzidis, Journal of the American Chemical Society 116, 7287 (1994).
  * Grüner (1988) G. Grüner, Rev. Mod. Phys. 60, 1129 (1988).
  * Dressel and Grüner (2002) M. Dressel and G. Grüner,  _Electrodynamics of Solids: Optical Properties of Electrons in Matter_ (Cambridge University Press, 2002).
  * Demsar  _et al._ (1999) J. Demsar, B. Podobnik, V. V. Kabanov, T. Wolf, and D. Mihailovic, Phys. Rev. Lett. 82, 4918 (1999).
  * Chia  _et al._ (2010) E. E. M. Chia, D. Talbayev, J.-X. Zhu, H. Q. Yuan, T. Park, J. D. Thompson, C. Panagopoulos, G. F. Chen, J. L. Luo, N. L. Wang, and A. J. Taylor, Phys. Rev. Lett. 104, 027003 (2010).
  * Yusupov  _et al._ (2008) R. V. Yusupov, T. Mertelj, J.-H. Chu, I. R. Fisher, and D. Mihailovic, Phys. Rev. Lett. 101, 246402 (2008).
  * Albrecht  _et al._ (1992) W. Albrecht, T. Kruse, and H. Kurz, Phys. Rev. Lett. 69, 1451 (1992).
  * Qi  _et al._ (2013) J. Qi, T. Durakiewicz, S. A. Trugman, J.-X. Zhu, P. S. Riseborough, R. Baumbach, E. D. Bauer, K. Gofryk, J.-Q. Meng, J. J. Joyce, A. J. Taylor, and R. P. Prasankumar, Phys. Rev. Lett. 111, 057402 (2013).
  * Rothwarf and Taylor (1967) A. Rothwarf and B. N. Taylor, Phys. Rev. Lett. 19, 27 (1967).
  * Kabanov  _et al._ (1999) V. V. Kabanov, J. Demsar, B. Podobnik, and D. Mihailovic, Phys. Rev. B 59, 1497 (1999).


(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

# Equivariant Differentially Private Deep Learning:
Why DP-SGD Needs Sparser Models

Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis
Institute for Artifical Intelligence in Medicine, Technical University of
Munich
{florian.hoelzl, daniel.rueckert<EMAIL_ADDRESS>
###### Abstract

Differentially Private Stochastic Gradient Descent (DP-SGD) limits the amount
of private information deep learning models can memorize during training. This
is achieved by clipping and adding noise to the model’s gradients, and thus
networks with more parameters require proportionally stronger perturbation. As
a result, large models have difficulties learning useful information,
rendering training with Differentially Private Stochastic Gradient Descent
(DP-SGD) exceedingly difficult on more challenging training tasks. Recent
research has focused on combating this challenge through training adaptations
such as heavy data augmentation and large batch sizes. However, these
techniques further increase the computational overhead of DP-SGD and reduce
its practical applicability. In this work, we propose using the principle of
sparse model design to solve precisely such complex tasks with fewer
parameters, higher accuracy, and in less time, thus serving as a promising
direction for DP-SGD. We achieve such sparsity by design by introducing
equivariant convolutional networks for model training with Differential
Privacy (DP). Using equivariant networks, we show that small and efficient
architecture design can outperform current state-of-the-art (SOTA) with
substantially lower computational requirements. On CIFAR-10, we achieve an
increase of up to $9\%$ in accuracy while reducing the computation time by
more than $85\%$. Our results are a step towards efficient model architectures
that make optimal use of their parameters and bridge the privacy-utility gap
between private and non-private deep learning for computer vision.

Figure 1:  Sparsity by design indicates that fewer overall parameters are
required to learn the same representations from an input image. Conventional
CNNs achieve this by applying convolutions with small kernels across the image
instead of flattening and densely connecting all pixels. Equivariant
convolutions introduce additional transformations on the kernel, allowing for
the detection of features independent of their _e.g_. rotation and/or
reflection (pose). As a result, equivariant networks need even fewer
parameters to learn the same information and are thus even sparser by design.
Image taken from the ImageNet dataset.

##  1 Introduction and Related Work

Artificial Intelligence is increasingly applied to fields where extremely
sensitive data is used, such as medicine or the social sciences. Previous
research has demonstrated that sensitive information can be reverse-engineered
from unprotected machine learning models [1, 2]. This renders privacy concerns
a major hurdle to developing and deploying machine learning systems in fields
where the protection of sensitive data is necessary, _e.g_. due to privacy
regulations, intellectual property requirements or other ethical
considerations and necessitates steps preventing the exposure of such data to
unauthorized third parties.

Privacy-enhancing technologies allow one to derive insights from sensitive
datasets while quantitatively bounding the risk of information leakage about
the training samples. By giving formal guarantees on data protection, they
represent the best chance to date to incentivize data sharing in an ethical
and responsible manner. Differential Privacy (DP) [3], the gold-standard
technique for privacy preservation, is most commonly applied in the field of
deep learning by utilizing DP-SGD [4], which imposes an upper bound on how
much information can be extracted from the model’s weights about the
individuals whose data was used for training. This guarantee is achieved
during model training through a modified optimization algorithm, which
includes clipping the gradient vector based on a predefined threshold of its
$\ell_{2}$-norm and then adding noise to it (see Sec. 2). Such a gradient is
then considered privatized. A predictive model trained with privatized
gradients is subsequently private as well, allowing analysts to share the
model while retaining the ability to bound the capacity of adversaries to
derive information from the data used for training.

However, using gradients privatized with DP-SGD often leads to sharp reduction
in prediction performance. This problem is called the privacy-utility trade-
off and is due to two main factors: On one hand, clipping the gradient
diminishes its information content and biases its direction; moreover, the
total noise magnitude scales proportionally to the $\ell_{2}$-norm of the
gradient, and thus to the number of model parameters. This leads to a decrease
in the signal-to-noise ratio, in that the true gradient’s signal is diminished
relative to the noise introduced by DP-SGD. Consequently, the effective
training of large models, typically used to achieve SOTA results from scratch
in non-private training, is rendered disproportionately difficult and falls
far short of attaining comparable results to non-private training. In this
paper, we propose to address the privacy-utility trade-off by introducing
sparse model design through Equivariant CNN (ECNN) architectures.

###  1.1 Computational Considerations of DP-SGD

Recent advancements in DP deep learning have focused largely on optimizing the
training regime of larger models to mitigate the previously mentioned
difficulties. The current SOTA for training CIFAR-10 from scratch was recently
established by [5] and incrementally improved by [6]. These works’ techniques
focus on extensive training adaptions (which likely have an effect on the
smoothness of the optimisation landscape) to improve prediction performance.
Among the most important adaptations are (1) large-batch training [7], (2)
averaging per-sample gradients across multiple augmentation of the same image
before clipping (augmentation multiplicity) [8, 9], and (3) temporal parameter
averaging techniques [10]. This results in remarkable accuracy gains on over-
parameterized models over the previous SOTA, which seemed out of reach up
until recently for models such as the WideResNet (WRN), that incorporate
several million parameters. However, the accuracy gains presented by the
authors of the aforementioned work come at a high cost: the massive
computational resources and time required to train those models have an
overbearing financial (and environmental) impact and impose great difficulties
in reproducing the results. As a result, further progress in this direction is
cumbersome and inefficient, and renders proposing improvements on top of the
presented results out of reach, especially for scientific institutions without
access to large-scale computational resources. In summary, current SOTA models
trained with DP-SGD (1) in part still substantially lack behind their non-
private counterparts and (2) have an impracticable computational burden that
makes research improvements difficult.

Orthogonal to the aforementioned works, other studies in DP deep learning have
introduced training regimes which additionally leverage publicly available
data, which is ostensibly usable without any privacy constraints [11, 12]. By
starting from a well performing non-private base model, fine-tuning on private
datasets with DP-SGD can lead to promising results close to non-private
training from scratch. However, Tramèr _et al_. question if using public data
for pre-training should actually be considered differential-privacy-preserving
as pre-trained models can leak private information contained within the public
dataset and thus negatively impact the public perception of the field [13].
Relying on public data in many cases thus opposes the very foundation of
privacy-enhancing technologies, _i.e_. obtaining access to important insights
while minimizing access to the sensitive data required. Even more importantly
for practical application, relying on pre-training as a solution is not a
panacea. Large quantities of public data are not available in areas such as
medicine, where data, at least from the same distribution, still comes with
the same privacy considerations and (even when available in sufficient
quantity) is difficult to access. Not only has the importance of in-
distribution data been shown in previous works [14, 15], but relying on pre-
training alone increases the barrier to extend DP-SGD to new modalities and
novel prediction tasks. Pre-training and its positive impact showcased on
benchmark datasets is thus an interesting observation, but will not solve the
previously mentioned challenges that currently hold back DP-SGD from
widespread adaptation in practice.

###  1.2 Motivation

We thus contend that an optimal solution to the aforementioned dilemmas will
require a fundamental reconsideration, which will marry high prediction
performance when training from scratch with high computational efficiency. The
guiding hypothesis of our work is, that networks that exhibit designed
sparsity can achieve the aforementioned goal. As we demonstrate in this work,
the notion of designed sparsity, introduced by [16], combines two major
characteristics beneficial for DP-SGD: (1) increased representational
efficiency and (2) a reduced set of (possibly redundant) model parameters. The
requirement for DP models to learn high-quality features to achieve parity
with non-private models has been previously discussed [17]. However, the
aforementioned work utilises a cascade of static feature extractors, which
lack the flexibility of their learned counterparts and whose capability to
scale to more complex problems remains an open research question. In contrast,
we show that networks that are sparse by design remain trainable with a
similar or higher degree of expressiveness, but with fewer parameters than
comparable deep learning models. In fact, even conventional CNNs are an
example of such networks, since a single kernel is used to compute the
features of a whole image. They inherently exhibit parameter sharing with
respect to translations and can thus be seen as a sparse version of a fully
connected layer. We argue that, through additional improvements in model
architectures that lead to higher designed sparsity, networks can learn
features more efficiently, offering a promising direction for DP-SGD. To
evaluate this hypothesis, we introduce ECNN s for DP-SGD. As shown in Fig. 1,
ECNN s further increase parameter sharing by being equivariant to
transformations such as rotations and reflections. They thus offer an even
higher degree of designed sparsity and a possible solution for efficient
training under DP.

While rotational equivariance can be approximated with no formal guarantee by
conventional (non-equivariant) CNNs through an increase in model width,
dataset size and augmentation techniques, _i.e_. the exact techniques
mentioned earlier, this approximation has two important drawbacks. They (1)
massively increase the computation time of the (already very demanding) DP-
SGD, as _e.g_. each additional set of simultaneous augmentations increases the
time complexity almost linearly. And (2), naïvely scaling the number of
parameters proportionally increases the total noise power of the added
Gaussian noise, thus risking drowning out the learning signal. Network layers
equivariant to rotation and reflection transformations preserve the relative
pose of features in addition to the translational equivariance preserved by
standard convolutions. The resulting additional parameter sharing thus avoids
the redundant learning of identical convolutional filters for multiple poses.
In addition to their high parameter efficiency, ECNN architectures are known
to exhibit increased data efficiency and improved generalization, especially
in domains with high degrees of intra-image symmetry [18, 19]. So far,
however, no works have investigated how to combine equivariant layers with DP
training nor analyzed the potential beneficial changes to the training regime,
even though their characteristics render them highly attractive for this use-
case.

In this work, we show the need for sparse model architectures under DP-SGD,
and that ECNN s are a promising step in this direction. Their higher designed
sparsity compared to standard CNNs, allows them to outperform larger models
while simultaneously decreasing the required computation time. As we will
demonstrate, this renders them especially interesting for training with DP-
SGD, particularly under tighter privacy bounds and low data regimes, both
desirable traits for privacy-preserving techniques.

Our main contributions are summarized as follows:

  * •

We introduce the methodology necessary to train ECNN s with DP-SGD in Sec. 3.
As part of this contribution, we propose novel normalization layers for
discrete and continuous symmetry groups that preserve the equivariance
property and fulfill the DP condition.

  * •

By leveraging the notion of designed sparsity, we substantially improve the
current SOTA on DP-SGD imaging benchmarks without additional data in Sec. 4.2.
We experimentally demonstrate ECNN s as a promising architecture that
satisfies this concept by offering improved results while requiring fewer
parameters than a conventional CNNs. Among others, we show an increase of
$9.2\%$ under $(2,10^{-5})$-DP on CIFAR-10 and an increase of more than $10\%$
on Tiny-ImageNet-200.

  * •

We provide insights into model calibration of our approach, since poor
calibration is a known weakness of DP-SGD. We find that the proposed
equivariant architecture improves model calibration with an on average $17\%$
lower Brier score across all evaluated datasets compared to the conventional
network.

  * •

We experimentally show, that equivariant networks are more robust to key
hyperparameter choices from recent SOTA results, in particular augmentations
in the input domain and batch size. Additionally, we analyze how equivariant
specific hyperparameter choices, such as the symmetry groups, affect training
with DP-SGD in Sec. 5.1.

##  2 Background

###  2.1 Differential Privacy

At its core, differential privacy provides a way to answer questions about a
dataset while limiting the risk of revealing sensitive information about
specific samples in the dataset. It achieves this by introducing controlled
randomness into the data analysis process. In other words, DP is a strong
stability condition on randomised algorithms mandating that outputs are
approximately invariant under inclusion or exclusion of a single individual
from the input database. For a mechanism (randomized algorithm) $\mathcal{M}$
and all datasets $D$ and $D^{\prime}$ that differ in one element as well as
all measurable subsets $S$ of the range of $\mathcal{M}$,
$(\varepsilon,\delta)$-DP requires that:

$\text{Pr}(\mathcal{M}(D)\in S)\leq
e^{\varepsilon}\,\text{Pr}(\mathcal{M}(D^{\prime})\in S)+\delta,$ (1)

where the privacy guarantees of the algorithm are parameterised by
$\varepsilon\geq 0$ and $\delta\in[0,1)$. This privacy constraint is in
practice typically realised by the addition of noise. For a comprehensive
overview of DP, we refer to [3]. Its application to deep learning came with
the introduction of DP-SGD by [4]. In our work, we utilise Gaussian noise and
the aforementioned DP-SGD algorithm to privatise gradient updates in neural
network training. We use Rényi-DP accounting [20, 21] to track the privacy
loss throughout the training. Rényi-DP (RDP) is often used with training deep
neural networks as it massively facilitates the composition of sequences of
private algorithms executed on sub-samples of a larger dataset, such as in
SGD, where we iteratively update the model parameters using randomly selected
subsamples of the training data. The RDP privacy condition is:

$D_{\alpha}(\mathcal{M}(D)\parallel\mathcal{M}(D^{\prime}))\leq\rho,$ (2)

where $D_{\alpha}$ is the Rényi divergence of order $\alpha\geq 1$. We note
that the Rényi divergence is symmetric in the Gaussian noise setting, as the
DP-guarantee is required to be symmetric. RDP can be converted to
$(\varepsilon,\delta)$-DP for a given $\delta$. In the rest of this paper, we
will refer to the converted $\bar{\varepsilon}$ simply as $\varepsilon$. We
note that we refer to sampling rather than mini-batches in DP-SGD, since
privacy amplification by sampling requires subsets of the training set to be
drawn using _e.g_. a Poisson sampling technique [4].

###  2.2 Equivariant CNNs

Equivariance describes the mathematical property of a structure-preserving
mapping. This means, that there exist two transformations $T_{g}$ and
$T^{\prime}_{g}$ that lead to the same result when applying a mapping $\phi$
to an input $\mathbf{x}$ such that:

$\phi\left(T_{g}\mathbf{x}\right)=T^{\prime}_{g}\phi\left(\mathbf{x}\right)\
.$ (3)

For image classification, this formal guarantee was first introduced for
rotations and reflections in Regular Group CNNs by [18] and later extended to
ECNN s with steerable filters [19, 22, 23]. The general approach to
equivariant convolutions is centered around the idea of representations,
describing the transformation laws of a given feature space. The feature
fields in this space are mappings that transform according to the
corresponding representation. Each layer’s input and output space must be
compatible with the corresponding transformation law. In order to guarantee
this behavior, a convolution kernel $K$ is subject to a linear constraint,
given by

$K(g\cdot\mathbf{x})=\rho_{\text{out}}(g)K(\mathbf{x})\rho_{\text{in}}(g)^{-1}\;\;\forall
g\in G,\;x\in\mathbb{R}^{n},$ (4)

with group actions $g\in G$, depending on the associated group representations
$\rho$. For our case (planar images), we focused on representations where the
group actions are rotations and reflections acting on the parameters of a
learned kernel, _i.e_. the group elements. These group actions can be
discrete, with the number of rotations typically denoted as $N$, or continuous
in $SO(2)$ or $O(2)$. Finite groups are commonly represented through regular
representations, where the corresponding transformation matrix has
dimensionality equal to the order of the group, _e.g_. $\mathbb{R}^{N}$. There
are different ways to approximate continuous groups to work with steerable
convolutions. We follow the results of [24], where the $SO(2)$ group has been
shown to give the most promising results of all continuous groups for non-
private planar image classification. The performance of the $SO(2)$ group is
evaluated using irreducible representations. An irreducible representation
$\psi$ of a group is a representation that cannot be reduced or decomposed
into smaller independent representations. Formally, for our group $G$ and the
feature space $Y$, an irreducible representation of $G$ on $Y$ is a linear map
$\psi:G\mapsto GL(Y)$ to the general linear group $GL$ that preserves the
group structure. In our case, the kernel $K$ is a linear map, with its basis
given by breaking down a group representation into a direct sum of irreducible
representations. This is called a irreducible representation decomposition and
allows us to construct a equivariant map between pairs of irreducible
representations, _i.e_. $\rho_{\text{in}}$ and $\rho_{\text{out}}$. To solve
the corresponding kernel constraint in our equivariant convolutions, we employ
the general solution from [25].

##  3 Methodology

In this section, we describe our method and introduce novel layers to allow
training ECNN s with DP-SGD. The proposed layers preserve the notion of
orientation in our ECNN s without violating the DP constraint. Maintaining the
equivariance property, not only within a single layer but throughout the whole
network, is necessary to fully leverage the additional feature information
during training. Our approach is based on the equivariant frameworks
established by [26, 25].

###  3.1 Convolution Layers

To satisfy the kernel constraint Eq. 4, we define the transformation law of
each convolution by its input and output representations $\rho_{in}$ and
$\rho_{out}$. For the roto-translational ECNN s proposed in this work, the
representations correspond to rotation matrices of a specific symmetry group
(_i.e_. $C_{N}$, $D_{N}$ or $SO(2)$). The resulting transformation law of the
feature space is a constant linear mapping that only has to be computed once
during initialization. While lower-level features intrinsically exhibit a
higher degree of symmetry, natural images often have a sense of orientation at
a global level. To account for this varying level of equivariance, we choose
an initial symmetry group and restrict $\rho_{in}$ and $\rho_{out}$ for the
last residual block. The order of the finite groups is reduced to $N/2$, with
$N$ being the group order, for the last residual block. When choosing $SO(2)$
as the initial symmetry group, we restrict the representations in the last
residual block to be invariant. Additionally, we adjust the number of channels
for each convolutional layer, such that our equivariant networks have a
similar number of parameters as their non-equivariant counterparts. This is
done by multiplying the number of channels by $\sqrt{1.5N}/N$ to end up with a
similar number of parameters for all symmetry groups. Moreover, to improve
signal propagation, we apply weight standardization in the convolutional layer
[27] and switch the order of the normalization and activation layers as
proposed in [6].

###  3.2 Normalization and Nonlinearities

For DP, we are required to compute per-sample gradients, which are
incompatible with the batch normalization typically used in ECNN literature.
We thus propose two novel equivariant normalization layers for DP-ECNN
training. Firstly, for trivial and regular representations used for discrete
groups $C_{N}$ and $D_{N}$, we introduce the (naïve) equivariant group
normalisation layer. The additional pose information is maintained throughout
the network by increasing the channel dimension $C$ of the feature vector to
include the fields of each representation. Before normalization, we split the
channel dimension $C$ of our $4$-dimensional feature vector such that the
representation fields of each channel are in a separate dimension. The
resulting feature vector $(N,Y,C,W,H)$, with the spatial dimensions $H$, $W$
and the batch axis $N$, is used for normalization. Each feature field
$\mathbf{y}\in[1,Y]$, corresponding to a representation’s transformation, is
then normalized across a group of channels $C$. This implementation allows us
to utilize existing $3$-dimensional group normalisation layers. After the
group normalization, for consistency, we reduce our feature vector back to $4$
dimensions by stacking the feature fields of each channel.

The aforementioned layer is, however, unsuitable for use with continuous
groups, which require irreducible representations to function. To thus satisfy
the equivariance property for the irreducible representations of the $SO(2)$
group, we introduce an i.i.d. instance normalisation layer. To normalize the
feature fields, we require an estimate of their expected values and variance
when transforming them by a group action. In the following, we derive how to
calculate the two values. For the former, we compute the expectation over the
whole group $G$ that is acting on our features. This expectation can be
written by using a normalized Haar measure $\mu$ over $G$,

$\mathbb{E}_{g\in G}[\psi(g)]=\int_{G}\text{d}\mu(g)\psi(g)$ (5)

with the irreducible representation $\psi$ of $G$ [28]. Due to the
orthogonality of $\psi(g)\ \forall g\in G$, the integral is always zero for
all non-trivial irreducible representations. As a consequence, the mean of any
vector transforming according to those representations is also zero. It
follows that the expected value of $\rho(g)$ is a null matrix with non-zero
values in the diagonal if and only if $\psi_{i}$ is a trivial representation.
In practice, we can thus pre-compute $\mathbb{E}_{g\in G}[\psi(g)]$ when
initializing our equivariant layers to calculate the estimated mean based on
the trivial representations. During training, we then simply subtract it from
our calculated feature fields.

To derive the expected variance, we substitute the transformation in Eq. 3
with our representation $\rho(g)$ to receive the feature field
$\mathbf{y}=\rho(g)\phi(\mathbf{x})$ of an input $\mathbf{x}$. The covariance
of $\mathbf{y}$ can then be calculated with

$\mathbb{E}_{\mathbf{y}\in
Y}\left[\mathbf{yy}^{\text{T}}\right]=\mathbb{E}_{g\in
G}\left[\rho(g)\,\phi(\mathbf{x})\phi(\mathbf{x})^{\text{T}}\rho(g)^{\text{T}}\right].$
(6)

Due to the fact that our representations $\rho(g)$ are orthonormal and
irreducible, the covariance matrix must be an orthogonal matrix. In addition,
the covariance matrix is symmetric and semi-positive definite, making it a
multiple of the identity matrix, _i.e_. $\mathbb{E}_{\mathbf{y}\in
Y}[\mathbf{yy}^{\top}]=\lambda\mathbf{I}$. We can then compute this multiple
independent of our representation simply as

$d\lambda=\operatorname{Tr}(\mathbb{E}_{\mathbf{y}\in
Y}\left[\mathbf{yy}^{\text{T}}\right])=\mathbb{E}_{\mathbf{y}\in
Y}\left[\left\lVert\mathbf{y}\right\rVert_{2}^{2}\right].$ (7)

Finally, we also apply a learnable weight and bias parameter as with most
normalization layers.

For trivial and regular representations, standard activation functions, such
as Mish and ReLU [29, 30], can be used as non-linearities. When working with
the $SO(2)$ group, we use an Inverse Fourier Transform on the feature space to
then apply the non-linearity in the group domain. Afterwards, we compute the
Fourier Transform to again obtain the coefficients of the irreducible
representations.

##  4 Results

###  4.1 Equivariant Models are Sparse by Design

####  4.1.1 Fewer Model Parameters

In previous work by Klause _et al_. [31], the ResNet-9 architecture was shown
to be performant with DP-SGD, despite its small size. We are interested on how
this efficiency can be further increased and thus use it as our baseline. We
find that the equivariance property of the Equivariant-ResNet-9 allows us to
further increase the validation accuracy on CIFAR-10 by reducing the number of
convolution channels per layer. In fact, Fig. 2 indicates that there is an
optimal model layout of $(16,32,64)$ channels, denoting the filters per layer
in the three residual blocks. For larger layouts, the performance starts
decreasing again. This leads to a total number of optimal parameters of
$\approx 250k$, _i.e_. ten times fewer than the original (conventional) CNN.
In comparison, the validation accuracy of the standard ResNet-9 does not
substantially increase when the model size is reduced. This is likely due to
the fewer parameters not being able to learn features effectively. The
equivariant network, on the other hand, does not have to learn redundant
features for different orientations in separate channels due to the additional
pose information in our feature space. The experiments indicate that this
allows the network to still detect a sufficient amount of information in the
image for prediction, even when reducing the parameters in the network. How
much of this additional information can be learned per parameter, also called
parameter utilization [19], is dependent on the chosen symmetry group. While
the equivariance property is better satisfied for the symmetry groups $SO(2)$
and $O(2)$, the expressiveness of the trained kernel can suffer due to the
corresponding tighter kernel constraints as introduced in Sec. 2. Our ablation
studies in Sec. 5.1 show that the dihedral $D_{4}$ group offers a sweet spot
regarding performance and accuracy, and is thus used throughout this section.
To summarize, we tested and found that the equivariance property allows us to
reduce the width of our network while maintaining or even increasing the
classification accuracy with DP-SGD by increasing the parameter utilization in
the network.

Figure 2: In comparison to the non-equivariant ResNet-9, our Equivariant-
ResNet-9 has a sweet spot on CIFAR-10 at an intermediate model size of $16$,
$32$ and $64$ channels per layer for the three residual blocks under
$(8,10^{-5})$-DP.

####  4.1.2 More Parameters Contribute to Model Prediction

In addition to an overall smaller model size, we also evaluate how the sparse
design of the architecture affects the gradients and parameters, and their
impact on training and prediction correspondingly. Due to the increased
parameter sharing, we expect the smaller model to use more of its parameters
to make predictions. To analyze this, we utilize the $\ell_{0}^{\epsilon}$
norm, _i.e_. the number of parameters or gradient entries with a magnitude
smaller than $\epsilon$ [32]. We note that this is a different $\epsilon$ than
the one used to describe DP guarantees. Fig. 3 shows, that the percentage of
parameters with absolute values $\approx 0$ during training is substantially
lower for the equivariant network compared to the standard ResNet-9. Thus, the
equivariant network has fewer redundant parameters that do not actually
contribute to the network’s prediction. This can be further seen when we
compare the Equivariant-ResNet-9 to a Equivariant-WRN-40. Even though both
networks are equivariant, the larger WRN-40 still has more unused parameters
and subsequently is not able to improve on the smaller Equivariant-ResNet-9
(Eq.-ResNet-9: $16k$ parameters; vs Eq.-WRN-40: $45k$ parameters).

####  4.1.3 Sparser Gradient Vectors

In addition to the network’s weights, we also investigate how information is
transmitted in the model. If the features learned in the network show an
increased representational efficiency, they should be able to learn useful
information from the available data faster and thus require less time to
converge to an optimum. Analyzing the sparsity of the gradient vector in Fig.
3, we see the ECNN’s gradient converge faster during training (_i.e_. more
close-to-zero entries), updating fewer weights. This can indicate that the
model has incorporated most of the relevant information and has approached a
minimum. Moreover, gradient sparsity increasing more quickly during training
also indicates a more efficient information transfer at an early stage. This
observation can not be made for the conventional ResNet-9, implying less
efficient training. The corresponding gradient sparsity increases slower and
the model keeps updating more parameters throughout the training regime. We
consider additional research on the role of sparse gradient vectors in DP, as
_e.g_. done in [33] and [34], a promising direction for future work. Our
current results show that ECNN s with increased designed sparsity have more
parameters contributing to predictions and converge faster, with a smaller
percentage of parameters to update during training.

Figure 3: The Equivariant-ResNet-9 converges to a better optimum during
training, indicated by a higher gradient $\ell_{0}^{\epsilon}$ ”norm”
(continuous lines), while also having a substantially lower percentage of
parameters with values $<\epsilon$ (dashed lines).

###  4.2 Improved Performance on Image Classification with Equivariant
Convolutional Networks

To validate that our notion of designed sparsity can in fact help to improve
performance with DP-SGD, we compare the equivariant network to current SOTA
models on multiple common benchmark classification datasets when training from
scratch under varying privacy parameters. The GPU hours ($h$) are measured on
an NVIDIA A100 40GB.

| CIFAR-10
---|---
Model | $\varepsilon$ | Median | Std. Dev. | [$h$]
De _et al_. (2022) | 2 | 65.9 | (0.5) |  –
8 | 81.4 | (0.2) |  –
De _et al_. (reproduction) | 2 | 62.6 | (0.62) | 41.3
8 | 80.3 | (1.13) | 69.5
Equivariant-ResNet-9 | 2 | 71.8 | (0.71) | 5.8
8 | 81.6 | (0.51) | 6.1
| CIFAR-100
Model | $\varepsilon$ | Median | Std. Dev. | [$h$]
De _et al_. (reproduction) | 2 | 19.2 | (0.59) | 43.7
8 | 40.8 | (0.15) | 76.1
Equivariant-ResNet-9 | 2 | 25.4 | (0.49) | 8.7
8 | 48.4 | (0.30) | 8.9
Table 1: CIFAR-10 and CIFAR-100 test accuracies of our Equivariant-ResNet-9
with symmetry group $D_{4}$ trained from scratch compared to the previous
state of the art under $(\varepsilon,10^{-5})$-DP. We report the median and
standard deviation calculated across $5$ independent runs.

####  4.2.1 CIFAR-10

We begin with experiments on CIFAR-10, which is still considered a challenging
dataset for training with DP-SGD. The previous works’ results were obtained by
splitting the training set into $45k$ train and $5k$ validation samples.
Equivalently to these works, our stated test accuracy is achieved by training
our model on the full training set and evaluating it once on the held-out test
set. Our equivariant models are benchmarked against the current SOTA models on
CIFAR-10 by [5]. We reproduce the previous SOTA results with the exact same
setup and code provided by the
authors111https://github.com/deepmind/jax_privacy on our hardware for a fair
comparison of the prediction results and computation time. The reproduced
results have a difference in test accuracy of $\lessapprox 3\%$ to the results
of the original paper, as similarly observed by [6]. The exact hyperparameters
and implementation details are summarized with further ablation studies in
Sec. A.1 and Sec. A.2.

Table 1 shows, that our equivariant model with the $D_{4}$ group is able to
substantially outperform the current SOTA on CIFAR-10 by up to $9.2\%$ at
$\varepsilon=2$. Most notably, this result is achieved substantially faster
with a decrease in computation time by $\approx 85\%$ ($35h$). This is in
large parts due to the reduced number of augmentations required. Moreover, the
corresponding model only consists of $256k$ parameters and is thus $35$ times
smaller than the previous SOTA model ($8.7M$ parameters less than the
WRN-40-4). This superior performance of the Equivariant-ResNet-9 is consistent
across all evaluated $\varepsilon$-values, and performs notably even better
under a tighter privacy budget as seen in Tab. 6. This shows, that a sparser
model architecture with a lower-dimensional gradient and thus less added noise
is particularly beneficial for privacy-preserving applications.

####  4.2.2 CIFAR-100

The CIFAR-100 dataset is particularly interesting, as it has $10$ times fewer
images per class compared to CIFAR-10. This allows us to evaluate if the
property of ECNN s to learn better from less data also holds under DP. Table 1
confirms, that this is indeed the case. We train both models, the Equivariant-
ResNet-9 and the WRN-40-4 under the same setup as for CIFAR-10. This time,
however, our equivariant network is not only substantially better for lower
$\varepsilon$-values, but also outperforms the WRN-40-4 by more than $7\%$
under $\varepsilon=8$. In addition, the computation performance remains
superior, with a reduction in computation time from our equivariant network by
$\approx 88\%$ ($67.2h$). This performance can be attributed to the increased
feature efficiency of ECNN s (see Sec. 4.1.3), which causes them to detect
more relevant input features than non-equivariant models. In addition to the
benefits provided by their sparse model architecture (as seen for low
$\varepsilon$-values for CIFAR-10), the feature efficiency of ECNN s is a key
characteristic making them useful for training with DP-SGD as privacy concerns
mandate using as little data as possible. To summarize, our ECNN s achieve
SOTA performance on CIFAR-10 and CIFAR-100 with substantially smaller models
and in a fraction of the computation time and are especially dominant under a
tighter privacy budget.

| Tiny-ImageNet-200
---|---
Model | $\varepsilon$ | Median | Std. Dev. | [$h$]
Klause _et al_. (2022) | 10 | 19.4 |  – |  –
WRN-40-4 | 8 | 27.8 | (0.55) | 696
Equivariant-ResNet-9 | 8 | 34.1 | (0.08) | 156
| ImageNette
Model | $\varepsilon$ | Median | Std. Dev. | [$h$]
Klause _et al_. (2022) | 9.88 | 67.1 |  – |  –
WRN-40-4 | 8 | 70.0 | (1.79) | 45.7
Equivariant-ResNet-9 | 8 | 75.6 | (2.20) | 19.7
Table 2: Top-1 test accuracies on Tiny-ImageNet-200 and ImageNette for the
Equivariant-ResNet-9 and a PyTorch reproduction of the WRN-40-4 from De _et
al_. [5] compared to the previous SOTA from [31] for $(\varepsilon,8\cdot
10^{-7})$-DP.

####  4.2.3 Tiny-ImageNet-200

Large-scale image classification with DP has recently come into focus, and
promising initial results have lately been demonstrated on the ImageNet
dataset [35, 5, 36]. The key drawback of the aforementioned works is the fact
that DP training on ImageNet is exceedingly costly in terms of computational
budget required. Efficient, yet accurate approaches are thus of great
interest. Even though our equivariant networks are not able to fully solve
this issue, the previous experiments indicate that they offer a first step at
achieving similar or better results to previous SOTA approaches while reducing
computation time. The Tiny-ImageNet-200 dataset is particularly interesting,
as it has fewer samples per class than the larger ImageNet-1k. In addition to
comparing to the previous SOTA model [31], we also reproduce the approach from
De _et al_. with the WRN-40-4 from [5]. To compare, we use the substantially
smaller Equivariant-ResNet-9 with the $D_{4}$ group and train both models for
$12000$ update steps. We also adapt the learning rate to $4$, due to the
larger batch size. For the experiment, we use the official validation set as
the unseen test set. Table 2 shows, that without further adaptations and
hyperparameter tuning, the Equivariant-ResNet-9 achieves a Top-1 test accuracy
of $34.1\%$, beating our baseline by $6.3\%$ and increasing the previous SOTA
by [31] by more than $14.7\%$ under a tighter privacy budget. This is achieved
while reducing the computation time by $\approx 77\%$ ($540h$). We
additionally also construct an equivariant version of the WRN-40 model, the
architecture previously used for SOTA results [5]. As equivariant convolutions
can be used in arbitrary network architectures, we only adapt the width of the
equivariant model to $1$ instead of $4$, equivalent to the results in Sec. 4.1
for the ResNet-9. The model is also able to outperform the previous SOTA
result and the baseline model, but does not reach the performance of the
smaller Equivariant-ResNet-9 (Eq.-WRN-40: $28.1\%$). Looking at the
$\ell_{0}^{\epsilon}$ ”norm”, we can see that both models use almost all of
their parameters (Eq.-ResNet-9: $2.5k$; Eq.-WRN-40: $7k$). Simply increasing
the number of parameters in the equivariant models thus does not improve
training proportionally. Instead, to boost performance with DP-SGD, future
research could explore other designs that use their additional parameters
efficiently. These architectures with even greater designed sparsity may offer
a promising approach for surpassing the performance of the proposed
Equivariant-ResNet-9 model.

####  4.2.4 ImageNette

The preceding subsection demonstrated that ECNN s, despite their compact model
size, possess the capacity to learn complex features. To assess their
effectiveness as the image size is increased, we conduct further experiments
on an additional ImageNet subset, ImageNette, featuring images with a
dimension of $160$ by $160$ pixels. As before, we compare the Equivariant-
ResNet-9 to the previous SOTA model by [31] and a custom baseline that
reproduces the WRN-40-4 setup from [5]. In accordance with previous results,
our equivariant network is able to outperform the previous SOTA and our
baseline in Tab. 2, by more than $5\%$. In addition, the Equivariant-ResNet-9
takes less than half of the computation time compared to the baseline approach
(reduction of $26h$). Our proposed Equivariant-ResNet-9 thus outperforms all
previous approaches independent of image size and number of images per class.
The accuracy improvement is particularly substantial under a tighter privacy
budget, making the notion of designed sparsity a promising direction for
further research on solving the privacy-utility trade-off.

| Brier Score (lower is better)
---|---
Dataset | ResNet-9 | Equivariant-ResNet-9
CIFAR-10 | 0.041 | 0.030
CIFAR-100 | 0.0076 | 0.0069
Tiny-ImageNet-200 | 0.004 | 0.0033
ImageNette | 0.041 | 0.034
Table 3: Compared to the standard ResNet-9, the Equivariant-ResNet-9 exhibits
better model calibration, shown by a lower Brier score averaged across the
test set under $(8,10^{-5})$-DP.

###  4.3 Privacy-Calibration Trade-Off

While privacy guarantees are important in sensitive domains, evaluating other
model characteristics should not be neglected in fields where trust in a
model’s predictions is essential. In particular, it has been shown that DP-SGD
has a negative effect on a model’s uncertainty calibration [37, 38]. A model’s
overconfidence in its predictions when trained with DP-SGD can give a false
sense of accuracy and prevent a reliable estimate of potential errors in the
results. We therefore exemplarily examine whether our proposed techniques are
able to provide improved calibration in addition to higher accuracy by looking
at the Brier score [39]. Indeed, on the CIFAR-10 test set, we found that the
Equivariant-ResNet-9 had a $\approx 27\%$ lower Brier score compared to the
non-equivariant counterpart (Tab. 3). This result is consistent under the
other evaluated datasets, with a reduction of on average $17\%$. The
prediction improvements of the ECNN thus do not further increase the
overconfidence arising from DP-SGD. Instead, the results indicate superior
model calibration to standard convolutional models. While an in-depth
investigation of the calibration of equivariant models is outside the scope of
our current study, we consider this finding encouraging and intend to expand
upon it in future work.

##  5 Ablation Studies

###  5.1 Training Adaptations for Improved Performance of Sparse Models under
DP

To analyze the performance advantage of ECNN s in more detail, we evaluate the
impact of different hyperparameter choices in this section. The results
demonstrate key benefits of ECNN s compared to conventional CNNs used for DP
training in previous works.

Figure 4: The Equivariant-ResNet-9 benefits from increased batch sizes similar
to the non-equivariant WRN-16-4 from De  _et al_. [5], but has a substantially
better validation accuracy across the board under $(8,10^{-5})$-DP.

####  5.1.1 Improved Accuracy Across All Batch Sizes

Previous work has generally shown that (very) large batch sizes lead to
substantial improvements in accuracy for DP training [7, 36]. We thus
investigate whether the superior performance of ECNN s is maintained across
batch sizes or whether non-equivariant training is able to match our
performance by batch size tuning. We find that the latter is not the case. In
fact, our ECNN s consistently outperformed the non-equivariant SOTA models,
whereby larger batch sizes generally led to an increased accuracy, similar to
the non-equivariant CNNs. Interestingly, the accuracy gains through a batch
size increase were steeper for ECNN s compared to the baseline. This is
indicated by the slope of the curve between $256$ and $2048$ in Fig. 4, which
is $\approx 15\%$ steeper for the equivariant network. We attribute this
finding to the robustness of the features learned by equivariant kernels,
which is synergistic with large batch sizes in making the gradient resilient
to clipping and noise addition. In summary, the equivariant architecture
outperformed the non-equivariant model across all batch sizes with an increase
in validation accuracy of –on average– $\approx 5\%$.

Figure 5: The advantage of the equivariant network is maintained when adding augmentation multiplicities up to a value of $16$. Further increasing the number of augmentations only increases computation time but actually decreases performance. |  |  | Test Accuracy [%] |  |
---|---|---|---|---|---
Model | Group | $\varepsilon$ | Median | Std. Dev. | Parameters | GPU Hours
Dörmann _et al_. (2021) | $\\{e\\}$ | 1.93 | 58.6 | (0.38) |  – |  –
De _et al_. (2022) | $\\{e\\}$ | 2 | 65.9 | (0.5) | $8.9M$ |  –
De _et al_. (reproduction) | $\\{e\\}$ | 2 | 62.6 | (0.62) | $8.9M$ | 42.27
Klause _et al_. (2022) | $\\{e\\}$ | 2.89 | 65.6 |  – | $2.4M$ |  –
Tramèr and Boneh (2021) | $\\{e\\}$ | 3 | 69.3 | (0.2) | $187k$ |  –
Equivariant-ResNet-9 (ours) | $C_{4}$ | 2 | 69.57 | (0.48) | $258k$ | 4.5
$C_{8}$ | 2 | 68.97 | (0.19) | $256k$ | 5.8
$C_{16}$ | 2 | 66.31 | (0.46) | $244k$ | 8.9
$D_{4}$ | 2 | 71.86 | (0.71) | $256k$ | 5.8
$D_{8}$ | 2 | 69.04 | (0.30) | $244k$ | 8.7
$D_{16}$ | 2 | 67.68 | (0.39) | $238k$ | 14.8
$SO(2)$ | 2 | 45.65 | (1.14) | $309k$ | 6.7
Table 4: CIFAR-10 test accuracy of our Equivariant-ResNet-9 with different
symmetry groups trained from scratch compared to the previous state of the art
without equivariance (_i.e_. $\\{e\\}$). We report the median and the standard
deviation calculated across $5$ independent runs. The GPU hours are measured
on a NVIDIA A100 40GB. The highest accuracy is observed using the Equivariant-
ResNet-9 and the $D_{4}$ group at only $256k$ parameters requiring only $5.8$
GPU hours of computation.

####  5.1.2 Fewer Augmentations Required

One of the main ingredients in the technique proposed by [5] is the
utilization of augmentation multiplicity, _i.e_. performing multiple
augmentations per sample and averaging the resulting gradients before
privatization. In this section, we address the question whether ECNN s are
able to supplant this technique (and thus drastically reduce the required
computation time, which scales with the number of simultaneous augmentations).
After all, equivariant convolutions are able to learn information decoupled
from a feature’s pose and thus reduce the necessity of augmentations,
especially of rotations and reflections, leading to an immediate reduction in
the intense computational burden of augmentation multiplicity. Fig. 5 shows,
that the equivariant network still benefits from augmentations to some extent
even though it does already incorporate pose information in its features. This
corroborates the finding by Weiler _et al_. [24] in the non-DP setting, who
show that augmentations can further improve prediction performance of
equivariant networks for dihedral groups. Compared to the current state of the
art on CIFAR-10, however, the equivariant models reach close-to-optimal
performance with an augmentation multiplicity of only $4$, whereas the SOTA
model requires $8$ times as many simultaneous augmentations to achieve the
same accuracy. In fact, our results at $4$ augmentations outperform the $32$
augmentation multiplicities of [5] while drastically reducing the computation
time by the same factor of $8$. Corroborating the theoretical advantages of
ECNN s, diminishing returns set in earlier than for conventional CNNs.

###  5.2 Hyperparameter Choices for Equivariant Layer

As mentioned above, models utilizing the $D_{4}$ group exhibited the highest
accuracy and were used for the experimental results above. To demonstrate how
we arrived at this choice and how the chosen symmetry group influences the
accuracy of the model, we evaluated our equivariant models with commonly used
symmetry groups from literature [24] under
$(\varepsilon,\delta)=(2,10^{-5})$-DP. As introduced in Sec. 2, symmetry
groups refer to mathematical groups that capture the symmetries or
transformations that are applied on an input. The symmetry group $C_{8}$ thus
describes the group of rotations of $45$°on a planar space. The continuous
group $SO(2)$ extends this to all rotations in the 2-dimensional space. We use
irreducible representations and an angular frequency of $1$ to describe the
$SO(2)$ group convolutions. For an in-depth analysis of other frequency groups
of the $SO(2)$ group, we refer to Sec. A.2.2. Based on the results from Sec.
5.1, we reduce the model layout from $(64,128,256)$ channels to $(16,32,64)$.
To maintain comparability, the number of channels per layer and thus total
number of parameters is fixed according to our approach described in Sec. 3.
The results in Tab. 4 show that we achieve our best median test accuracy on
CIFAR-10 with $71.86\%$ for the dihedral group $D_{4}$. The dihedral groups
perform (on average) slightly better than the cyclic groups, probably due to
the intrinsic horizontal symmetry of the images in the dataset which are
better captured by its capability to represent reflections. This is
corroborated by the fact that increasing the rotation order $N$ did not
improve prediction performance. As the network uses kernels of size $3\times
3$, this phenomenon could also be attributed to the restrictions in
discretizing small rotations on this kernel size without losing information.
We observe a similar effect with the continuous rotations in $SO(2)$, which
perform substantially worse than all other groups. This is in line with the
non-private experiments conducted by [24], indicating that the kernel
constraint is too restrictive and the lack of expressiveness cannot be
compensated by the more pronounced equivariant properties of the kernel. We
consider combining our technique with larger receptive fields, _e.g_. through
atrous (dilated) convolutions, a promising future work direction. Notably, all
discrete groups $C_{N}$ and $D_{N}$ were able to outperform previous SOTA
models. The $C_{4}$ and $D_{4}$ groups in particular showed the best results,
offering the best trade-off between accuracy and computation time out of all
candidates. We thus recommend favoring these groups over continuous groups in
practice.

##  6 Conclusion

The broad application of private deep learning has –so far– been impeded by
privacy-utility trade-offs. Recent works have partially addressed these
limitations and presented techniques to bridge the accuracy gap but introduced
a new trade-off between accuracy and efficiency. Ultimately, we contend that
both trade-offs must be addressed to facilitate large-scale research in DP
deep learning. The remarkable performance gains that Equivariant CNNs enable
are an important step towards this goal. Their capability to outperform
previous approaches in a low-data regime and under a tight privacy budget, as
well as their improved calibration and capability to capture intrinsic image
symmetries, renders them particularly interesting for DP-SGD.

With extensive benchmark experiments, we showed that sparse model designs are
promising to also overcome computational overhead concerns in DP. In a time
such as the present, where breakthroughs are usually achieved through solving
engineering problems on large-scale systems, targeting ways to render the
foundations of deep learning itself more efficient is in our opinion a
promising and sustainable direction for solving long-term challenges. The
introduction of additional structural prior information, such as the presence
of symmetries in images, tackles exactly such challenges. In addition, we
regard the provision of formal guarantees of model behavior, which both
equivariance and DP represent, as a solid foundation for systems that fulfill
the notion of trustworthy AI.

## References

  * [1] B. Balle, G. Cherubin, and J. Hayes, “Reconstructing Training Data with Informed Adversaries,” _2022 IEEE Symposium on Security and Privacy (SP)_ , pp. 1138–1156, 2022.
  * [2] N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. X. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, “Extracting Training Data from Large Language Models,” in _30th USENIX Security Symposium (USENIX Security 21)_. USENIX Association, Aug. 2021, pp. 2633–2650. [Online]. Available: https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting
  * [3] C. Dwork and A. Roth, “The Algorithmic Foundations of Differential Privacy,” _Found. Trends Theor. Comput. Sci._ , vol. 9, pp. 211–407, 2014\.
  * [4] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep Learning with Differential Privacy,” in _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 308–318. [Online]. Available: https://doi.org/10.1145/2976749.2978318
  * [5] S. De, L. Berrada, J. Hayes, S. L. Smith, and B. Balle, “Unlocking High-Accuracy Differentially Private Image Classification through Scale,” _ArXiv_ , vol. abs/2204.13650, 2022.
  * [6] T. Sander, P. Stock, and A. Sablayrolles, “TAN without a Burn: Scaling Laws of DP-SGD,” _ArXiv_ , vol. abs/2210.03403, 2022.
  * [7] F. Dörmann, O. Frisk, L. N. Andersen, and C. F. Pedersen, “Not All Noise is Accounted Equally: How Differentially Private Learning Benefits from Large Sampling Rates,” _2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP)_ , pp. 1–6, 2021.
  * [8] E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry, “Augment Your Batch: Improving Generalization through Instance Repetition,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 8126–8135, 2020.
  * [9] S. Fort, A. Brock, R. Pascanu, S. De, and S. L. Smith, “Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error,” _ArXiv_ , vol. abs/2105.13343, 2021.
  * [10] B. Polyak and A. B. Juditsky, “Acceleration of Stochastic Approximation by Averaging,” _Siam Journal on Control and Optimization_ , vol. 30, pp. 838–855, 1992.
  * [11] A. Golatkar, A. Achille, Y.-X. Wang, A. Roth, M. Kearns, and S. Soatto, “Mixed Differential Privacy in Computer Vision,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 8366–8376, 2022.
  * [12] D. Yu, H. Zhang, W. Chen, and T.-Y. Liu, “Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning,” _ArXiv_ , vol. abs/2102.12677, 2021.
  * [13] F. Tramèr, G. Kamath, and N. Carlini, “Considerations for Differentially Private Learning with Large-Scale Public Pretraining,” _ArXiv_ , vol. abs/2212.06470, 2022.
  * [14] K. He, R. Girshick, and P. Dollar, “Rethinking ImageNet Pre-Training,” in _IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 4917–4926.
  * [15] X. Mei, Z. Liu, P. M. Robson, B. Marinelli, M. Huang, A. Doshi, A. Jacobi, C. Cao, K. E. Link, T. Yang, Y. Wang, H. Greenspan, T. Deyer, Z. A. Fayad, and Y. Yang, “RadImageNet: An Open Radiologic Deep Learning Research Dataset for Effective Transfer Learning,” _Radiology: Artificial Intelligence_ , vol. 0, no. ja, p. e210315, 0. [Online]. Available: https://doi.org/10.1148/ryai.210315
  * [16] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste, “Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks,” _J. Mach. Learn. Res._ , vol. 22, pp. 241:1–241:124, 2021\.
  * [17] F. Tramèr and D. Boneh, “Differentially Private Learning Needs Better Features (or Much More Data),” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * [18] T. Cohen and M. Welling, “Group Equivariant Convolutional Networks,” in _Proceedings of The 33rd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 20–22 Jun 2016, pp. 2990–2999. [Online]. Available: https://proceedings.mlr.press/v48/cohenc16.html
  * [19] ——, “Steerable CNNs,” in _International Conference on Learning Representations_ , 2017.
  * [20] I. Mironov, “Rényi Differential Privacy,” _2017 IEEE 30th Computer Security Foundations Symposium (CSF)_ , pp. 263–275, 2017.
  * [21] I. Mironov, K. Talwar, and L. Zhang, “Rényi Differential Privacy of the Sampled Gaussian Mechanism,” _ArXiv_ , vol. abs/1908.10530, 2019.
  * [22] T. Cohen, M. Geiger, J. Köhler, and M. Welling, “Spherical CNNs,” _ArXiv_ , vol. abs/1801.10130, 2018.
  * [23] M. Weiler, F. A. Hamprecht, and M. Storath, “Learning Steerable Filters for Rotation Equivariant CNNs,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 849–858, 2018.
  * [24] M. Weiler and G. Cesa, “General E(2)-Equivariant Steerable CNNs,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [25] G. Cesa, L. Lang, and M. Weiler, “A Program to Build E(N)-Equivariant Steerable CNNs,” in _ICLR_ , 2022.
  * [26] M. Geiger and T. Smidt, “e3nn: Euclidean Neural Networks,” 2022. [Online]. Available: https://arxiv.org/abs/2207.09453
  * [27] S. Qiao, H. Wang, C. Liu, W. Shen, and A. L. Yuille, “Weight Standardization,” _ArXiv_ , vol. abs/1903.10520, 2019.
  * [28] G. Chirikjian, A. Kyatkin, and A. Buckingham, “Engineering Applications of Noncommutative Harmonic Analysis: With Emphasis on Rotation and Motion Groups,” _ASME. Appl. Mech. Rev._ , vol. 54, no. 6, p. B97–B98, November 2001.
  * [29] D. Misra, “Mish: A Self Regularized Non-Monotonic Activation Function,” in _BMVC_ , 2020.
  * [30] A. F. Agarap, “Deep Learning using Rectified Linear Units (ReLU),” _ArXiv_ , vol. abs/1803.08375, 2018.
  * [31] H. Klause, A. Ziller, D. Rueckert, K. Hammernik, and G. Kaissis, “Differentially Private Training of Residual Networks with Scale Normalisation,” in _ICML Theory and Practice of Differential Privacy Workshop_ , 2022.
  * [32] D. L. Donoho and M. Elad, “Maximal Sparsity Representation via l1 Minimization,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 100, no. 5, pp. 2197–2202, 2003.
  * [33] J. Zhu and M. B. Blaschko, “Differentially Private SGD with Sparse Gradients,” in _arXiv preprint arXiv:2112.00845_ , 2021.
  * [34] R. Ito, S. P. Liew, T. Takahashi, Y. Sasaki, and M. Onizuka, “Scaling Private Deep Learning with Low-Rank and Sparse Gradients,” _ArXiv_ , vol. abs/2207.02699, 2022.
  * [35] P. Chrabaszcz, I. Loshchilov, and F. Hutter, “A Downsampled Variant of ImageNet as an Alternative to the CIFAR Datasets,” _ArXiv_ , vol. abs/1707.08819, 2017.
  * [36] A. Kurakin, S. Chien, S. Song, R. Geambasu, A. Terzis, and A. Thakurta, “Toward Training at ImageNet Scale with Differential Privacy,” _ArXiv_ , vol. abs/2201.12328, 2022.
  * [37] M. Knolle, A. Ziller, D. Usynin, R. F. Braren, M. R. Makowski, D. Rueckert, and G. Kaissis, “Differentially Private Training of Neural Networks with Langevin Dynamics for Calibrated Predictive Uncertainty,” _ArXiv_ , vol. abs/2107.04296, 2021.
  * [38] H. Zhang, X. Li, P. Sen, S. Roukos, and T. Hashimoto, “A Closer Look at the Calibration of Differentially Private Learners,” _ArXiv_ , vol. abs/2210.08248, 2022.
  * [39] G. W. Brier, “Verification of Forecasts Expressed in Terms of Probability,” _Monthly Weather Review_ , vol. 78, pp. 1–3, 1950.

##  Appendix A Appendix

###  A.1 Implementation Details

All experiments in this paper except for the reproduction of [5] are
implemented in PyTorch. The privacy accounting is done with Opacus and
additional performance improvements are achieved through vectorization with
functorch. The aforementioned reproduction is implemented with the code taken
from the official repository of the original authors in JAX/Haiku. The
hyperparameters used for our equivariant networks and the reproduction are
summarized in Tab. 5. The equivariant models are trained for $2160$ update
steps with DP-SGD and a clipping norm and learning rate of $2.0$, an
exponential moving average decay of $0.999$, a batch size of $8192$ and $4$
augmentation multiplicities (original image + 3 augmentations). We use random
reflections and cropping with a two-sided reflection padding of $4$ pixels for
our augmentations.

| CIFAR | Tiny-ImageNet-200
---|---|---
Optimizer | SGD | SGD
Aug. Multiplicity | 4 | 4
Batch Size | 8192 | 16384
Clipping Norm | 2.0 | 2.0
Learning Rate | 2 | 4
Noise Multiplier $\sigma$ | 5.0 | 12.5
Number Updates | 2160 | 12000
Table 5: Hyperparameters used for measuring the accuracy when training from
scratch in all of our equivariant experiments on test sets.

###  A.2 Additional Experimental Results

####  A.2.1 Results Across Different $\varepsilon$-Values

For easier reproducibility, the exact results for different
$\varepsilon$-values are given in Tab. 6. All experiments are run with the
setting described in Sec. 4.2 and hyperparameters summarized in Tab. 5. The
standard deviation for our equivariant network is measured across $5$
independent runs, while the reproduction was run $3$ times.

|  | Test Accuracy [%]
---|---|---
Model | $\varepsilon$ | Median | Std. Dev.
Equivariant-ResNet-9 (ours) | 1 | 60.59 | (1.40)
2 | 72.10 | (0.68)
3 | 75.96 | (0.38)
4 | 78.27 | (0.15)
6 | 80.26 | (0.26)
8 | 81.62 | (0.51)
WRN-40-4 (reproduction) | 1 | 52.80 | (0.45)
2 | 62.63 | (0.62)
4 | 72.42 | (0.84)
6 | 77.78 | (0.45)
8 | 80.38 | (1.13)
Table 6: CIFAR-10 median test accuracies under different
$(\varepsilon,10^{-5})$ values measured across $5$ independent runs for the
Equivariant-ResNet-9 with symmetry group $D_{4}$ and $3$ independent runs for
the WRN-40-4 from [5].

####  A.2.2 Frequencies of the $SO(2)$ Group

For the continuous $SO(2)$ group, instead of the number of discrete rotations,
we have to choose the angular frequency used to construct the kernel as an
additional hyperparameter. The choice not only changes the expressiveness of
the kernel but also impacts the number of parameters that scale consistently
with the angular frequency. As in earlier sections, we thus adapt the model
width to have a comparable number of parameters. Tab. 7 shows, that when
training under DP, the increased expressiveness does slightly increase
prediction performance, with a frequency of $3$ performing best under
$(2,10^{-5})$-DP. However, conforming with the results in Sec. 5.2, the
$SO(2)$ group performs worse than the discrete groups independent of the
chosen angular frequency. Our results thus suggest to use the discrete $C_{N}$
and $D_{N}$ groups when training ECNN s with DP-SGD.

$\varepsilon$ | Frequency | Parameters | Test Accuracy [%]
---|---|---|---
$2$ | $1$ | $243k$ | 43.69
$2$ | $251k$ | 44.65
$3$ | $309k$ | 45.65
$8$ | $1$ | $243k$ | 53.61
$2$ | $251k$ | 55.12
$3$ | $309k$ | 54.23
Table 7: CIFAR-10 test accuracies for different angular frequencies of the
steerable kernel for the $SO(2)$ group for an adapted layout with similar
number of parameters.