# Ring solids and supersolids in spherical shell-shaped dipolar Bose-Einstein
condensates

J. Sánchez-Baena<EMAIL_ADDRESS>Departament de Física,
Universitat Politècnica de Catalunya, Campus Nord B4-B5, 08034 Barcelona,
Spain  R. Bombín  Departament de Física, Universitat Politècnica de Catalunya,
Campus Nord B4-B5, 08034 Barcelona, Spain  Université de Bordeaux, 351 Cours
de la Libération, 33405 Talence, France  J. Boronat  Departament de Física,
Universitat Politècnica de Catalunya, Campus Nord B4-B5, 08034 Barcelona,
Spain

###### Abstract

We study the interplay between the anisotropy of the dipole-dipole interaction
and confinement in a curved geometry. We do so through numerical calculations,
based on the extended Gross-Pitaevskii equation, to characterize the ground
state of a dipolar Bose gas under the effect of a bubble trapping potential.
As the scattering length and the number of particles are varied, we observe
the emergence of a wide variety of dipolar solids, which consist of
arrangements of dipolar droplets along a ring over the equator of the
spherical shell confinement. By changing the detuning of the trap, we probe
different ring-like supersolid structures with varying number of dipolar
clusters, which are unobtainable through conventional trapping settings.

##  I Introduction

Since the realization of the first Bose-Einstein condensate (BEC) in 1995, the
research on ultracold gases has been boosted by the achievement of a high
control over them. This allows to explore not only a wide range of interaction
strengths but also different geometries, ranging from one to three dimensions.
In recent years, with the development of trapping techniques, it has been
possible to achieve more exotic trapped geometries with non-trivial topologies
eg., rings or curved surfaces (see Ref. Garraway and Perrin (2016) for a
review on trapping techniques). These advances have motivated zero gravity
experiments Becker  _et al._ (2018), where the gravitational sag is absent and
the BEC can be engineered in the shell of a sphere Carollo  _et al._ (2022)
(see Tononi and Salasnich (2023a) for recent reviews on the topic). Some works
include the characterization of BEC condensation and excitations Móller  _et
al._ (2020), the topological superfluid phase transition Tononi and Salasnich
(2019); Tononi  _et al._ (2022), the BEC-BCS crossover He  _et al._ (2022),
the study of vortices and collective excitations Tononi and Salasnich (2023b)
and the application of matter-wave lensing techniques Boegel  _et al._ (2023).
On a less fundamental approach, the possibility of engineering atom based
circuits has also been explored Amico  _et al._ (2022).

In the context of ultracold gases, the study of dipolar systems has revealed
astonishing phenomena such as droplet formation and the emergence of
supersolidity (see Ref. Chomaz  _et al._ (2022) for an experimental review).
Supersolidity refers to a state of matter that simultaneously features spatial
diagonal and off-diagonal long-range order Andreev and Lifshitz (1969);
Chester (1970); Leggett (1970). Dipolar systems emerge as an exceptional setup
for studying the phenomena of supersolidity. This topic has been extensively
studied Böttcher  _et al._ (2020); Ilg and Büchler (2023); Hertkorn  _et al._
(2021a); Gallemí and Santos (2022); Hertkorn  _et al._ (2021b); Blakie  _et
al._ (2020a); Bombín  _et al._ (2017, 2019); Zhang  _et al._ (2019, 2021,
2023); Blakie  _et al._ (2020b); Smith  _et al._ (2023); Sánchez-Baena  _et
al._ (2023) and experimentally confirmed Guo  _et al._ (2019); Tanzi  _et al._
(2019a); Natale  _et al._ (2019); Tanzi  _et al._ (2021); Petter  _et al._
(2021); Biagioni  _et al._ (2022); Bland  _et al._ (2022); Norcia  _et al._
(2022); Tanzi  _et al._ (2019b); Böttcher  _et al._ (2019); Chomaz  _et al._
(2019); Norcia  _et al._ (2021). Supersolid phases have been also predicted to
occur in strictly two-dimensional geometries, where the transition to the
normal state is of the Berezinskii-Kosterlitz-Thouless type Filinov  _et al._
(2010); Bombín  _et al._ (2017, 2019); Ota and Stringari (2018). Nonetheless,
only a few studies have considered this phenomena on curved surfaces (see
Refs. Diniz  _et al._ (2020); Ciardi  _et al._ (2023); Arazo  _et al._ (2021);
Nilsson Tengstrand  _et al._ (2023); Young-S. and Adhikari (2023); Šindik  _et
al._ (2023)).

Dipolar shell-shaped systems are expected to exhibit a richer phase diagram
than contact BEC gases Tononi and Salasnich (2023a). On one hand, the long
ranged and anisotropic character of dipolar interaction is known to produce
density modulated phases with important long-range correlations in free space.
On the other hand, the curvature would make the ground state of the system
very different to that of the free space, for example by frustrating the
formation of filaments. In this sense, the interplay between anisotropy, long-
range order, and topology in spherically symmetric traps can give rise to BEC
states in which the spherical symmetry of the trap is spontaneously broken.

To give some insight into the above mentioned phenomena, we study in the
present work a dipolar Bose gas confined on a spherical bubble trapping
potential in the regime of parameters where supersolidity arises. The paper is
organized as follows. In Sec. II, we illustrate the methodology that we
employ. The phase diagram of the system as a function of the relevant physical
parameters is given in Sec. III. Finally, in Sec. IV we summarize the main
conclusions and discuss future perspectives.

##  II Theory

Our system contains $N$ magnetic dipolar atoms (or electric dipolar molecules)
of mass $M$ with all their magnetic moments $\mu$ aligned along the $Z$-axis.
The system is confined in a bubble trap potential Colombe  _et al._ (2004);
Sun  _et al._ (2018)

$\displaystyle V_{\text{trap}}({\bf r})$
$\displaystyle=m\omega_{0}^{2}r_{0}^{2}\sqrt{\frac{[(r/r_{0})^{2}-\Delta/\epsilon]^{2}}{4}+(\Omega/\epsilon)^{2}},$
(1)

where $\omega_{0}$ is the frequency of the bare harmonic trap, prior to
radiofrequency (rf) dressing, and the parameters $\Delta$ and $\Omega$ are the
detuning between the radiofrequency (rf) field and the different energy states
employed to prepare the condensate, and the Rabi coupling between these
states, respectively Carollo  _et al._ (2022); Sun  _et al._ (2018). We have
also introduced the relevant length and energy scales, $a_{0}$ and $\epsilon$
respectively, given by $r_{0}=12\pi a_{\text{dd}}$ and
$\epsilon=\hbar^{2}/(mr_{0}^{2})$, where $a_{\mathrm{dd}}=\frac{C_{\rm
dd}M}{12\pi\hbar^{2}}$ is the dipole length with $C_{\rm
dd}=\mu_{0}\mu^{2}(d^{2}/\epsilon_{0}$), $\mu_{0}$ the Bohr magneton
($\epsilon_{0}$ vacuum permittivity) and $\mu$ ($d$) the magnetic (electric)
dipole moment of the atoms. For simplicity, and analogously to a previous work
Sun  _et al._ (2018), we consider $\Delta=\Omega$.

Figure 1: Decreasing the scattering length drives the shell shaped dipolar gas
from a contact dominated spherical shell to a dipole dominated ring
configuration. This is showcased in panels a)-d), which show the integrated
two-dimensional density $\rho(x,z)$ for $\varepsilon_{\text{dd}}=1/6,1/3,1/2$
and $1$, respectively. Calculations are performed for a bubble trap (see Eq.
1) with detuning $\Omega/\epsilon=\Delta/\epsilon=400$ and bare harmonic
frequency $\omega_{0}=0.22\epsilon/\hbar$.

In order to characterize the ground state of the system in a spherically
symmetric trap we solve the three-dimensional extended Gross-Pitaevskii
equation that reads

$\displaystyle\mu\Psi({\bf r})=$
$\displaystyle\left[-\frac{\hbar^{2}}{2M}\nabla^{2}+V_{\textrm{trap}}({\bf
r})+g|\Psi(r)|^{2}+\gamma_{\text{QF}}\absolutevalue{\Psi({\bf r})}^{3}\right.$
$\displaystyle\left.+\int d{{\bf r}^{\prime}}V_{\rm dd}({\bf r}-{\bf
r^{\prime}})\absolutevalue{\Psi({\bf r^{\prime}})}^{2}\right]\Psi({\bf r})\ ,$
(2)

with $\mu$ the chemical potential, $\Psi({\bf r})$ the condensate wave
function, which is normalized as $N=\int d{\bf r}\absolutevalue{\Psi({\bf
r})}^{2}$, and $g=4\pi\hbar^{2}a_{s}/M$ the coupling constant with
$a_{\mathrm{s}}$ the s-wave scattering length. The fourth term
$\gamma_{\mathrm{QF}}|\Psi|^{3}$ is the LHY correction Lima and Pelster (2011,
2012) which introduces quantum fluctuations,

$\gamma_{\mathrm{QF}}|\Psi|^{3}=\frac{32g\sqrt{a_{s}^{3}}}{3\sqrt{\pi}}\mathcal{Q}_{5}\left(\varepsilon_{\mathrm{dd}}\right)|\Psi|^{3},$
(3)

with $\varepsilon_{\mathrm{dd}}=a_{\mathrm{dd}}/a_{s}$,
$Q_{5}(\varepsilon_{\mathrm{dd}})=\dfrac{1}{2}\displaystyle\int_{0}^{\pi}\mathrm{d}\alpha\sin\alpha\left[1+\varepsilon_{\rm
dd}(3\cos^{2}\alpha-1)\right]^{5/2}$. Finally, the last term in Eq. (2)
accounts for the dipole-dipole interaction (DDI),

$\displaystyle V_{\rm dd}({\bf r}-{\bf r^{\prime}})$
$\displaystyle=\frac{C_{\text{dd}}}{4\pi}\frac{1-3\cos^{2}\theta}{\absolutevalue{{\bf
r}-{\bf r^{\prime}}}^{3}},$ (4)

where $\theta$ is the polar angle of the vector,${\bf r}-{\bf r^{\prime}}$.

The use of the pseudopotential in Eq. 2 is justified as long as the collisions
between particles can be treated as three-dimensional processes. The system
lays in the two dimensional regime if the harmonic length
$a_{\text{h}o}=\sqrt{\hbar/(m\omega_{0})}$, which is associated to the
tightness of the confinement, is significantly smaller than any other length
scales. However, in our calculations, we choose a trapping strength such that
$a_{\text{h}o}\gg a_{\mathrm{s}}$. Therefore, all scattering processes can be
considered three-dimensional and the pseudopotential of Eq. 2 can be applied.
In the case of a tight confinement, a pseudopotential that considers the
effects of the geometry should be employed Tononi (2022); Zhang and Ho (2018).

The classical equilibrium position of the trapping potential in Eq. 1 is
$r_{\text{e}q}/r_{0}=\sqrt{\Delta/\epsilon}$. In the limit
$\left(1-\frac{(r/r_{0})^{2}}{\Delta/\epsilon}\right)^{2}\ll\frac{2\Omega}{\Delta}$
a Taylor expansion around $r_{\text{eq}}$ allows to approximate Eq. (1) as

$\displaystyle V_{\text{trap}}({\bf r})$
$\displaystyle\simeq\frac{1}{2}m\omega_{\text{eff}}^{2}r_{0}^{2}\left(r/r_{0}-\sqrt{\Delta/\epsilon}\right)^{2},$
(5)

i.e. the small oscillations have harmonic behavior, with an effective trapping
strength of $\omega_{\text{eff}}=\omega_{0}\sqrt{\Delta/\Omega}$. Taking now
the limit of large detuning, the equilibrium position is radially displaced
such that the atoms are confined to move within a spherical shell of radius
$R/r_{0}\simeq\sqrt{\Delta/\epsilon}$, with the bare harmonic strength
$\omega_{0}$ controlling the tightness of the confinement. This approximation
has been used in prior works with dipolar atoms Diniz  _et al._ (2020); Arazo
_et al._ (2021). In Appendix A, we explore the quantitative differences of the
density profiles obtained with the exact (see Eq. 1) and the approximated (see
Eq. 5) potential.

##  III Ring solids and supersolids

The first property to be highlighted about the structure of the dipolar
condensate in the bubble trap is the ring shape of the atomic configurations
for the typical values of the scattering length used in experiments of dipolar
gases Chomaz  _et al._ (2019). In the regime where the dipolar interaction
dominates ($\epsilon_{\rm dd}>1$), the preference of dipoles to be aligned
within a head-to-tail configuration (see Eq. 4) makes them to accumulate
around the equator of the spherical shell, forming a thick ring. Thus, a
dipolar gas with $\epsilon_{\rm dd}\ll 1$ transitions from a fully spherical
shell-shaped gas in the contact dominated regime to a ring-shaped one as the
scattering length is progressively decreased, spontaneously breaking the
spherical symmetry imposed by the bubble trap Ciardi  _et al._ (2023). This
behavior is showcased in Fig. 1, where we show the integrated two-dimensional
density $\rho(x,z)=\int dy\absolutevalue{\Psi({\bf r})}^{2}$ as a function of
$\epsilon_{\rm dd}$. We have employed a trap with parameters
$\Delta/\epsilon=400$, $\omega_{0}=0.22\epsilon/\hbar$, which correspond to
$\omega_{0}=2\pi\times 200$ Hz and a bubble trap with radius
$R=r_{0}\sqrt{\Delta/\epsilon}=5.2$ $\mu$m for 164Dy atoms. These parameters
are realistic for a setting with non-dipolar gases. Although not shown, in all
the cases of Fig. 1, the density of the system shows continuous rotational
invariance around the $Z$ axis.

Figure 2:  Top: structural phase diagram of the ring-shapped dipolar
condensate as a function of $\epsilon_{\rm dd}=a_{\rm dd}/a$ and the number of
particles $N$. The color indicates the value of Legget’s upper bound estimator
of the superfluidity, defined in Eq. 6. Panels A-I: integrated density
profiles $\rho(x,y)$ for the points marked in the diagram.

In the dipolar dominated regime ($\epsilon_{\rm dd}>1$), the anisotropy of the
dipolar interaction can break this continuous rotational symmetry, giving rise
to dipolar solids and supersolids, in analogy to the phenomenology that takes
place in bulk-trapped dipolar BECs. Eq. 2 can numerically be solved for
different values of the scattering length $a_{\mathrm{s}}$ (and thus,
$\epsilon_{\rm dd}$) and number of particles $N$ to characterize the different
structures formed by the condensed density in a fixed bubble trap. Since
superfluid structures may appear for a specific range of parameters, we
evaluate Legget’s upper bound estimator for superfluidity Leggett (1998),
which is computed from the ground state density as

$\displaystyle
f_{s}=\left[\frac{1}{2\pi}\int_{0}^{2\pi}\frac{d\theta}{\rho(\theta)/\rho_{0}}\right]^{-1}\
,$ (6)

where

$\displaystyle\rho(\theta)$ $\displaystyle=\int dzdr\text{
}r\absolutevalue{\Psi(r,\theta,z)}^{2},$ (7) $\displaystyle\rho_{0}$
$\displaystyle=\frac{1}{2\pi}\int_{0}^{2\pi}\rho(\theta)d\theta\ .$ (8)

Eq. 6 yields the trivial unity limit for a ring-like condensate (since
$\rho(\theta)$ is a constant) and decreases as the modulations of the wave
function increase along the ring. This upper-bound estimator for the
superfluidity has shown excellent agreement with its explicit calculation by
non-classical translational inertia for a system of dipoles confined in a
quasi-1D tubular geometry Smith  _et al._ (2023).

The structural characterization of the system is summarized in Fig. 2. The top
panel of the figure depicts the structural diagram of the system, with the
superfluidity of Eq. (6) encoded in the color bar. In the bottom panel we show
the integrated density $\rho(x,y)$ for specific values of $N$ and
$\epsilon_{\rm dd}$, to showcase the wide variety of dipolar states that can
be found. In general, the increase of $\epsilon_{\rm dd}$ for a fixed number
of particles implies a lower number of droplets, since the attraction of the
DDI bunches more particles together and thus, fewer and more elongated dipolar
clusters are formed. In much the same way, the decrease of the number of
particles for a fixed $\epsilon_{\rm dd}$ also causes a reduction in the
number of droplets, which is expected since the system wants to maximize the
number of dipoles placed in a head-to-tail configuration. We also see that the
solid structures disappear if the number of particles is decreased below a
threshold, at which there is not enough density to sustain clustering. This
phenomenology is reminiscent of a quasi-1D system of dipoles confined in a
tube Blakie  _et al._ (2020b); Smith  _et al._ (2023) where the disappearance
of the supersolid phase upon decreasing the density is also reported.

Figure 3:  Top: Superfluid density as a function of the detuning
$\Delta/\epsilon$. Bottom: Integrated density $\rho(x,y)$ for detuning values
$\Delta/\epsilon=8,24,120,240,320$, and $400$ from a) to f), respectively. The
ratio between the dipole length and the scattering length is set to
$\epsilon_{\rm dd}=1.43$ and a number of particles to $N=31500$. The bare
harmonic frequency is the same as in Fig. 1.  Figure 4: Ring shaped supersolid
dipolar states are engineered by appropriate combinations of values of the
magnetic field detuning, the scattering length and the number of particles.
Panels a)-d) show the integrated density $\rho(x,y)$ of supersolid states
obtained for $\epsilon_{\rm dd}=1.47$, $N=16500$, $\Delta/\epsilon=120$ (a),
$\epsilon_{\rm dd}=1.51$, $N=31500$, $\Delta/\epsilon=90$ (b), $\epsilon_{\rm
dd}=1.47$, $N=31500$, $\Delta/\epsilon=135$ (c), $\epsilon_{\rm dd}=1.4$,
$N=26500$, $\Delta/\epsilon=320$ (d). The bare harmonic frequency is the same
as in Fig. 1.

In terms of superfluidity, as Fig. 2 shows, the system is divided in two
clearly distinct regions: a fully superfluid one, where the condensate density
is structurless in the $X$-$Y$ plane, and a solid one, where clusterization
into droplets occurs and superfluidity is absent. In between these regions,
though, there exists a tiny window of values of the scattering length yielding
supersolid structures. For the parameters of Fig. 2, this tiny window sits
around $\epsilon_{\rm dd}\simeq 1.42$ for $N>21500$. In this small region, the
system shows between seven and eight main droplet clusters surrounded by a
halo of condensed atoms, establishing phase coherence, and yielding the
celebrated dipolar supersolid (see panels G and H of Fig. 2). In this case
though, as it happens with the fluid and solid states, the supersolid is
placed at a ring along the equator of the bubble trap, constituting a dipolar
ring supersolid, a structure unattainable with the conventional trapping
potentials employed in past experiments Guo  _et al._ (2019); Tanzi  _et al._
(2019a); Natale  _et al._ (2019); Tanzi  _et al._ (2021); Petter  _et al._
(2021); Biagioni  _et al._ (2022); Bland  _et al._ (2022); Norcia  _et al._
(2022); Tanzi  _et al._ (2019b); Böttcher  _et al._ (2019); Chomaz  _et al._
(2019); Norcia  _et al._ (2021).

Even though the supersolid phase constitutes a small region in the diagram of
Fig. 2, a rich variety of supersolid structures can be engineered when tuning
$\Delta$, and thus effectively modifying the radius of the trapping spherical
shell. By increasing $\Delta$ starting from a conventional harmonically trap
gas at $\Delta/\epsilon=8$, the system first transitions to a supersolid.
Furtherly increasing $\Delta$ leads to the emergence of a solid of isolated
droplets, all while retaining the ring shape. This is shown in Fig. 3, where
we report the integrated density $\rho(x,y)$ of the condensate wave function
for different values of $\Delta$. We also report Legget’s upper bound for the
superfluid fraction (see Eq. 6), which increases as the trap radius decreases,
thus confirming that the superfluid density can be enhanced by reducing
$\Delta$. The variation of this parameter is pretty straightforward in
experimental setups, as changing $\Delta$ is precisely how the bubble trap is
generated from a harmonic potential. The wide variety of solid structures in
the diagram of Fig. 2 thus allows for the observation of many different
supersolid dipolar rings by playing with the parameter $\Delta$ starting from
different points of the structural diagram. This is showcased in Fig. 4, where
we show four examples of different supersolids obtained for different
combinations of the number of particles, the detuning and the scattering
length.

##  IV Conclusions

We have extensively studied the interplay between the anisotropy of the
dipole-dipole interaction and the curved geometry of the trapping potential
for a dipolar condensate confined in a bubble trap. We have shown that in the
dipole-dominated regime, the condensate majorly populates the equator of the
shell-shaped trap, giving rise to a ring structure. We have provided the
structural diagram of the system as a function of the number of particles $N$
and the ratio between the dipole length and the scattering length
$\epsilon_{\rm dd}$, and have reported the emergence of a wide variety of ring
shaped solid structures formed by arrangements of dipolar clusters along the
equator of the trapping potential. We have also shown that supersolid states
with varying number of dipolar clusters can be engineered by changing $N$,
$\epsilon_{\rm dd}$, and the effective radius of the trap, which is
accomplished by tuning the detuning of the coupled rf field. Our results lead
to the existence of ring shaped dipolar supersolid states with varying number
of clusters, and significantly, unattainable with conventional trapping
geometries. The study of the excitations of these ring supersolids remains a
relevant question to be addressed, since it could lead to an experimental
protocol to probe the gas-to-supersolid transition by means of measuring
excitation frequencies. Also, it remains an open question how finite
temperature could affect the physics of the dipolar system under these
trapping conditions. Recent results Sohmen  _et al._ (2021); Sánchez-Baena
_et al._ (2023) reveal an important impact of thermal fluctuations on dipolar
gases, leading to the counterintuitive formation of a supersolid by heating in
the ultracold regime.

###### Acknowledgements.

We acknowledge financial support from Ministerio de Ciencia e Innovación
MCIN/AEI/10.13039/501100011033 (Spain) under Grant No. PID2020-113565GB-C21
and from AGAUR-Generalitat de Catalunya Grant No. 2021-SGR-01411. J.S.B and
R.B acknowledge European Union-NextGenerationEU, Ministry of Universities and
Recovery, Transformation and Resilience Plan, through a call from the
Technical University of Catalonia. R.B. further acknowledges funding from
ADAGIO (Advanced ManufacturIng Research Fellowship Programme in the Basque –
New Aquitaine Region) MSCA COFUND post-doctoral fellowship programme.

Figure 5: Comparison between the obtained ground state integrated densities
$\rho(x,y)$ for $\epsilon_{\rm dd}=1.72$ considering the asymmetric bubble
trap potential of Eq. 1 [a)-c)] and its harmonic approximation of Eq. 5
[d)-f)]. The number of particles is $N=11500$ (a) and d)), $N=21500$ [b) and
e)] and $N=31500$ [c) and f)]. The detuning and the bare harmonic frequencies
are the same as those in Fig. 1 Figure 6: Comparison between the bubble trap
potential in Eq. 1 and the harmonic approximation in Eq. 5. a), b): integrated
densitites $\rho(x,z)$ of the dipolar solid for $N=21500$, $\epsilon_{\rm
dd}=1.72$. The detuning and the bare harmonic frequencies are the same as
those in Fig. 1

##  Appendix A Comparison between the bubble trap and the harmonic
approximation

Some previous works in the literature Diniz  _et al._ (2020); Arazo  _et al._
(2021) employ the harmonic approximation for the bubble trap, given in Eq. 5,
to study the physics of the dipolar condensate in the gas phase. While this is
a reasonable approximation, the harmonic potential yields different results
than the ones obtained with the full interaction, mainly in the dipole
dominated regime where solid structures emerge. This is illustrated in Fig. 5,
where we compare the integrated density $\rho(x,y)$ obtained with the exact
potential from Eq. 1 and the harmonic approximation of Eq. 5 for different
values of $\\{N,\epsilon_{\rm dd}\\}$ for which $\epsilon_{\rm dd}>1$. As we
can see from the figure, both potentials produce a different number of dipolar
clusters, with the bubble trap producing a smaller number of them. This can be
understood by comparing the shape of both potentials, which is shown in Fig. 6
for the parameters of Fig. 1. We also report the integrated density
$\rho(x,z)$ for $\epsilon_{\rm dd}=1.72$, $N=21500$. As can be seen from the
figure, the harmonic approximation imposes a tighter confinement, which limits
the elongation of the droplets and forces the dipoles to split in a bigger
number of them with respect to the bubble trap.

## References

  * Garraway and Perrin (2016) Barry M Garraway and Hélène Perrin, “Recent developments in trapping and manipulation of atoms with adiabatic potentials,” Journal of Physics B: Atomic, Molecular and Optical Physics 49, 172001 (2016).
  * Becker  _et al._ (2018) Dennis Becker, Maike D. Lachmann, Stephan T. Seidel, Holger Ahlers, Aline N. Dinkelaker, Jens Grosse, Ortwin Hellmig, Hauke Müntinga, Vladimir Schkolnik, Thijs Wendrich, André Wenzlawski, Benjamin Weps, Robin Corgier, Tobias Franz, Naceur Gaaloul, Waldemar Herr, Daniel Lüdtke, Manuel Popp, Sirine Amri, Hannes Duncker, Maik Erbe, Anja Kohfeldt, André Kubelka-Lange, Claus Braxmaier, Eric Charron, Wolfgang Ertmer, Markus Krutzik, Claus Lämmerzahl, Achim Peters, Wolfgang P. Schleich, Klaus Sengstock, Reinhold Walser, Andreas Wicht, Patrick Windpassinger, and Ernst M. Rasel, “Space-borne bose–einstein condensation for precision interferometry,” Nature 562, 391–395 (2018).
  * Carollo  _et al._ (2022) R. A. Carollo, D. C. Aveline, B. Rhyno, S. Vishveshwara, C. Lannert, J. D. Murphree, E. R. Elliott, J. R. Williams, R. J. Thompson, and N. Lundblad, “Observation of ultracold atomic bubbles in orbital microgravity,” Nature 606, 281–286 (2022).
  * Tononi and Salasnich (2023a) A. Tononi and L. Salasnich, “Shell-shaped atomic gases,” (2023a), arXiv:2309.15710 [cond-mat.quant-gas] .
  * Móller  _et al._ (2020) Natália S Móller, F Ednilson A dos Santos, Vanderlei S Bagnato, and Axel Pelster, “Bose–einstein condensation on curved manifolds,” New Journal of Physics 22, 063059 (2020).
  * Tononi and Salasnich (2019) A. Tononi and L. Salasnich, “Bose-einstein condensation on the surface of a sphere,” Phys. Rev. Lett. 123, 160403 (2019).
  * Tononi  _et al._ (2022) Andrea Tononi, Axel Pelster, and Luca Salasnich, “Topological superfluid transition in bubble-trapped condensates,” Phys. Rev. Res. 4, 013122 (2022).
  * He  _et al._ (2022) Yan He, Hao Guo, and Chih-Chun Chien, “Bcs-bec crossover of atomic fermi superfluid in a spherical bubble trap,” Phys. Rev. A 105, 033324 (2022).
  * Tononi and Salasnich (2023b) Andrea Tononi and Luca Salasnich, “Low-dimensional quantum gases in curved geometries,” Nature Reviews Physics 5, 398–406 (2023b).
  * Boegel  _et al._ (2023) Patrick Boegel, Alexander Wolf, Matthias Meister, and Maxim A Efremov, “Controlled expansion of shell-shaped bose–einstein condensates,” Quantum Science and Technology 8, 034001 (2023).
  * Amico  _et al._ (2022) Luigi Amico, Dana Anderson, Malcolm Boshier, Jean-Philippe Brantut, Leong-Chuan Kwek, Anna Minguzzi, and Wolf von Klitzing, “Colloquium: Atomtronic circuits: From many-body physics to quantum technologies,” Rev. Mod. Phys. 94, 041001 (2022).
  * Chomaz  _et al._ (2022) Lauriane Chomaz, Igor Ferrier-Barbut, Francesca Ferlaino, Bruno Laburthe-Tolra, Benjamin L Lev, and Tilman Pfau, “Dipolar physics: a review of experiments with magnetic quantum gases,” Reports on Progress in Physics 86, 026401 (2022).
  * Andreev and Lifshitz (1969) A. F Andreev and I. M. Lifshitz, “Quantum theory of defects in crystals,” Sov. Phys. JETP 29, 1107–1113 (1969).
  * Chester (1970) G. V. Chester, “Speculations on bose-einstein condensation and quantum crystals,” Phys. Rev. A 2, 256–258 (1970).
  * Leggett (1970) A. J. Leggett, “Can a solid be “superfluid”?” Phys. Rev. Lett. 25, 1543–1546 (1970).
  * Böttcher  _et al._ (2020) Fabian Böttcher, Jan-Niklas Schmidt, Jens Hertkorn, Kevin S H Ng, Sean D Graham, Mingyang Guo, Tim Langen, and Tilman Pfau, “New states of matter with fine-tuned interactions: quantum droplets and dipolar supersolids,” Reports on Progress in Physics 84, 012403 (2020).
  * Ilg and Büchler (2023) Tobias Ilg and Hans Peter Büchler, “Ground-state stability and excitation spectrum of a one-dimensional dipolar supersolid,” Phys. Rev. A 107, 013314 (2023).
  * Hertkorn  _et al._ (2021a) J. Hertkorn, J.-N. Schmidt, M. Guo, F. Böttcher, K. S. H. Ng, S. D. Graham, P. Uerlings, T. Langen, M. Zwierlein, and T. Pfau, “Pattern formation in quantum ferrofluids: From supersolids to superglasses,” Phys. Rev. Res. 3, 033125 (2021a).
  * Gallemí and Santos (2022) Albert Gallemí and Luis Santos, “Superfluid properties of a honeycomb dipolar supersolid,” Phys. Rev. A 106, 063301 (2022).
  * Hertkorn  _et al._ (2021b) J. Hertkorn, J.-N. Schmidt, M. Guo, F. Böttcher, K. S. H. Ng, S. D. Graham, P. Uerlings, H. P. Büchler, T. Langen, M. Zwierlein, and T. Pfau, “Supersolidity in two-dimensional trapped dipolar droplet arrays,” Phys. Rev. Lett. 127, 155301 (2021b).
  * Blakie  _et al._ (2020a) P Blair Blakie, D Baillie, and Sukla Pal, “Variational theory for the ground state and collective excitations of an elongated dipolar condensate,” Communications in Theoretical Physics 72, 085501 (2020a).
  * Bombín  _et al._ (2017) R. Bombín, J. Boronat, and F. Mazzanti, “Dipolar bose supersolid stripes,” Phys. Rev. Lett. 119, 250402 (2017).
  * Bombín  _et al._ (2019) Raúl Bombín, Ferran Mazzanti, and Jordi Boronat, “Berezinskii-kosterlitz-thouless transition in two-dimensional dipolar stripes,” Phys. Rev. A 100, 063614 (2019).
  * Zhang  _et al._ (2019) Yong-Chang Zhang, Fabian Maucher, and Thomas Pohl, “Supersolidity around a critical point in dipolar bose-einstein condensates,” Phys. Rev. Lett. 123, 015301 (2019).
  * Zhang  _et al._ (2021) Yong-Chang Zhang, Thomas Pohl, and Fabian Maucher, “Phases of supersolids in confined dipolar bose-einstein condensates,” Phys. Rev. A 104, 013310 (2021).
  * Zhang  _et al._ (2023) Yong-Chang Zhang, Thomas Pohl, and Fabian Maucher, “Metastable patterns in one- and two-component dipolar bose-einstein condensates,” (2023), arXiv:2310.04738 [cond-mat.quant-gas] .
  * Blakie  _et al._ (2020b) P. B. Blakie, D. Baillie, L. Chomaz, and F. Ferlaino, “Supersolidity in an elongated dipolar condensate,” Phys. Rev. Research 2, 043318 (2020b).
  * Smith  _et al._ (2023) Joseph C. Smith, D. Baillie, and P. B. Blakie, “Supersolidity and crystallization of a dipolar bose gas in an infinite tube,” Phys. Rev. A 107, 033301 (2023).
  * Sánchez-Baena  _et al._ (2023) J. Sánchez-Baena, C. Politi, F. Maucher, F. Ferlaino, and T. Pohl, “Heating a dipolar quantum fluid into a solid,” Nature Communications 14, 1868 (2023).
  * Guo  _et al._ (2019) Mingyang Guo, Fabian Böttcher, Jens Hertkorn, Jan-Niklas Schmidt, Matthias Wenzel, Hans Peter Büchler, Tim Langen, and Tilman Pfau, “The low-energy goldstone mode in a trapped dipolar supersolid,” Nature 574, 386–389 (2019).
  * Tanzi  _et al._ (2019a) L. Tanzi, S. M. Roccuzzo, E. Lucioni, F. Famà, A. Fioretti, C. Gabbanini, G. Modugno, A. Recati, and S. Stringari, “Supersolid symmetry breaking from compressional oscillations in a dipolar quantum gas,” Nature 574, 382–385 (2019a).
  * Natale  _et al._ (2019) G. Natale, R. M. W. van Bijnen, A. Patscheider, D. Petter, M. J. Mark, L. Chomaz, and F. Ferlaino, “Excitation spectrum of a trapped dipolar supersolid and its experimental evidence,” Phys. Rev. Lett. 123, 050402 (2019).
  * Tanzi  _et al._ (2021) L. Tanzi, J. G. Maloberti, G. Biagioni, A. Fioretti, C. Gabbanini, and G. Modugno, “Evidence of superfluidity in a dipolar supersolid from nonclassical rotational inertia,” Science 371, 1162–1165 (2021).
  * Petter  _et al._ (2021) D. Petter, A. Patscheider, G. Natale, M. J. Mark, M. A. Baranov, R. van Bijnen, S. M. Roccuzzo, A. Recati, B. Blakie, D. Baillie, L. Chomaz, and F. Ferlaino, “Bragg scattering of an ultracold dipolar gas across the phase transition from bose-einstein condensate to supersolid in the free-particle regime,” Phys. Rev. A 104, L011302 (2021).
  * Biagioni  _et al._ (2022) Giulio Biagioni, Nicolò Antolini, Aitor Alaña, Michele Modugno, Andrea Fioretti, Carlo Gabbanini, Luca Tanzi, and Giovanni Modugno, “Dimensional crossover in the superfluid-supersolid quantum phase transition,” Phys. Rev. X 12, 021019 (2022).
  * Bland  _et al._ (2022) T. Bland, E. Poli, C. Politi, L. Klaus, M. A. Norcia, F. Ferlaino, L. Santos, and R. N. Bisset, “Two-dimensional supersolid formation in dipolar condensates,” Phys. Rev. Lett. 128, 195302 (2022).
  * Norcia  _et al._ (2022) Matthew A. Norcia, Elena Poli, Claudia Politi, Lauritz Klaus, Thomas Bland, Manfred J. Mark, Luis Santos, Russell N. Bisset, and Francesca Ferlaino, “Can angular oscillations probe superfluidity in dipolar supersolids?” Phys. Rev. Lett. 129, 040403 (2022).
  * Tanzi  _et al._ (2019b) L. Tanzi, E. Lucioni, F. Famà, J. Catani, A. Fioretti, C. Gabbanini, R. N. Bisset, L. Santos, and G. Modugno, “Observation of a dipolar quantum gas with metastable supersolid properties,” Phys. Rev. Lett. 122, 130405 (2019b).
  * Böttcher  _et al._ (2019) Fabian Böttcher, Jan-Niklas Schmidt, Matthias Wenzel, Jens Hertkorn, Mingyang Guo, Tim Langen, and Tilman Pfau, “Transient supersolid properties in an array of dipolar quantum droplets,” Phys. Rev. X 9, 011051 (2019).
  * Chomaz  _et al._ (2019) L. Chomaz, D. Petter, P. Ilzhöfer, G. Natale, A. Trautmann, C. Politi, G. Durastante, R. M. W. van Bijnen, A. Patscheider, M. Sohmen, M. J. Mark, and F. Ferlaino, “Long-lived and transient supersolid behaviors in dipolar quantum gases,” Phys. Rev. X 9, 021012 (2019).
  * Norcia  _et al._ (2021) Matthew A. Norcia, Claudia Politi, Lauritz Klaus, Elena Poli, Maximilian Sohmen, Manfred J. Mark, Russell N. Bisset, Luis Santos, and Francesca Ferlaino, “Two-dimensional supersolidity in a dipolar quantum gas,” Nature 596, 357–361 (2021).
  * Filinov  _et al._ (2010) A. Filinov, N. V. Prokof’ev, and M. Bonitz, “Berezinskii-kosterlitz-thouless transition in two-dimensional dipole systems,” Phys. Rev. Lett. 105, 070401 (2010).
  * Ota and Stringari (2018) Miki Ota and Sandro Stringari, “Second sound in a two-dimensional bose gas: From the weakly to the strongly interacting regime,” Phys. Rev. A 97, 033604 (2018).
  * Diniz  _et al._ (2020) Pedro C. Diniz, Eduardo A. B. Oliveira, Aristeu R. P. Lima, and Emanuel A. L. Henn, “Ground state and collective excitations of a dipolar bose-einstein condensate in a bubble trap,” Scientific Reports 10, 4831 (2020).
  * Ciardi  _et al._ (2023) Matteo Ciardi, Fabio Cinti, Giuseppe Pellicane, and Santi Prestipino, “Supersolid phases of bosonic particles in a bubble trap,” (2023), arXiv:2303.06113 [cond-mat.quant-gas] .
  * Arazo  _et al._ (2021) Maria Arazo, Ricardo Mayol, and Montserrat Guilleumas, “Shell-shaped condensates with gravitational sag: contact and dipolar interactions,” New Journal of Physics 23, 113040 (2021).
  * Nilsson Tengstrand  _et al._ (2023) M. Nilsson Tengstrand, P. Stürmer, J. Ribbing, and S. M. Reimann, “Toroidal dipolar supersolid with a rotating weak link,” Phys. Rev. A 107, 063316 (2023).
  * Young-S. and Adhikari (2023) Luis E. Young-S. and S. K. Adhikari, “Spontaneous dipolar bose-einstein condensation on the surface of a cylinder,” Phys. Rev. A 108, 053323 (2023).
  * Šindik  _et al._ (2023) Marija Šindik, Tomasz Zawiślak, Alessio Recati, and Sandro Stringari, “Sound, superfluidity and layer compressibility in a ring dipolar supersolid,” (2023), arXiv:2308.05981 [cond-mat.quant-gas] .
  * Colombe  _et al._ (2004) Y. Colombe, E. Knyazchyan, O. Morizot, B. Mercier, V. Lorent, and H. Perrin, “Ultracold atoms confined in rf-induced two-dimensional trapping potentials,” Europhysics Letters 67, 593 (2004).
  * Sun  _et al._ (2018) Kuei Sun, Karmela Padavić, Frances Yang, Smitha Vishveshwara, and Courtney Lannert, “Static and dynamic properties of shell-shaped condensates,” Phys. Rev. A 98, 013609 (2018).
  * Lima and Pelster (2011) Aristeu R. P. Lima and Axel Pelster, “Quantum fluctuations in dipolar bose gases,” Phys. Rev. A 84, 041604 (2011).
  * Lima and Pelster (2012) A. R. P. Lima and A. Pelster, “Beyond mean-field low-lying excitations of dipolar bose gases,” Phys. Rev. A 86, 063609 (2012).
  * Tononi (2022) A. Tononi, “Scattering theory and equation of state of a spherical two-dimensional bose gas,” Phys. Rev. A 105, 023324 (2022).
  * Zhang and Ho (2018) Jian Zhang and Tin-Lun Ho, “Potential scattering on a spherical surface,” Journal of Physics B: Atomic, Molecular and Optical Physics 51, 115301 (2018).
  * Leggett (1998) A. J. Leggett, “On the superfluid fraction of an arbitrary many-body system at t=0,” Journal of Statistical Physics 93, 927–941 (1998).
  * Sohmen  _et al._ (2021) Maximilian Sohmen, Claudia Politi, Lauritz Klaus, Lauriane Chomaz, Manfred J. Mark, Matthew A. Norcia, and Francesca Ferlaino, “Birth, life, and death of a dipolar supersolid,” Phys. Rev. Lett. 126, 233401 (2021).


11institutetext: University of Amsterdam, Amsterdam, the Netherlands.
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# SKDCGN: Source-free Knowledge Distillation of Counterfactual Generative
Networks using cGANs

Sameer Ambekar Equal contribution. 0000-0002-8650-3180 Matteo Tafuro SKDCGN:
Source-free Knowledge Distillation of Counterfactual Generative Networks using
cGANs 0000-0002-6167-2156 Ankit Ankit SKDCGN: Source-free Knowledge
Distillation of Counterfactual Generative Networks using cGANs
0000-0002-9399-9209
Diego van der Mast SKDCGN: Source-free Knowledge Distillation of
Counterfactual Generative Networks using cGANs 0000-0002-0001-3069 Mark Alence
SKDCGN: Source-free Knowledge Distillation of Counterfactual Generative
Networks using cGANs 0000-0002-6622-5822 Christos Athanasiadis
0000-0003-4376-9066

###### Abstract

With the usage of appropriate inductive biases, Counterfactual Generative
Networks (CGNs) can generate novel images from random combinations of shape,
texture, and background manifolds. These images can be utilized to train an
invariant classifier, avoiding the wide spread problem of deep architectures
learning spurious correlations rather than meaningful ones. As a consequence,
out-of-domain robustness is improved. However, the CGN architecture comprises
multiple over parameterized networks, namely BigGAN and U2-Net. Training these
networks requires appropriate background knowledge and extensive computation.
Since one does not always have access to the precise training details, nor do
they always possess the necessary knowledge of counterfactuals, our work
addresses the following question: Can we use the knowledge embedded in pre-
trained CGNs to train a lower-capacity model, assuming black-box access (i.e.,
only access to the pretrained CGN model) to the components of the
architecture? In this direction, we propose a novel work named SKDCGN that
attempts knowledge transfer using Knowledge Distillation (KD). In our proposed
architecture, each independent mechanism (shape, texture, background) is
represented by a student ’TinyGAN’ that learns from the pretrained teacher
’BigGAN’. We demonstrate the efficacy of the proposed method using state-of-
the-art datasets such as ImageNet, and MNIST by using KD and appropriate loss
functions. Moreover, as an additional contribution, our paper conducts a
thorough study on the composition mechanism of the CGNs, to gain a better
understanding of how each mechanism influences the classification accuracy of
an invariant classifier. Code available at:
https://github.com/ambekarsameer96/SKDCGN

##  1 Introduction

Deep neural networks are prone to learning simple functions that fail to
capture intricacies of data in higher-dimensional manifolds [1], which causes
networks to struggle in generalizing to unseen data. In addition to spectral
bias [1] and shortcut learning, which are properties inherent to neural
networks [2], spurious learned correlations are also caused by biased
datasets. To this end, Counterfactual Generative Networks (CGNs), proposed by
Sauer and Geiger [3], have been shown to generate novel images that mitigate
this effect. The authors expose the causal structure of image generation and
split it into three Independent Mechanisms (IMs) (object shape, texture, and
background), to generate synthetic and counterfactual images whereon an
invariant classifier ensemble can be trained.

The CGN architecture comprises multiple over-parameterized networks, namely
BigGANs [4] and U2-Nets [5], and its training procedure generally requires
appropriate domain-specific expertise. Moreover, one does not always have
access to the precise training details, nor do they necessarily possess the
required knowledge of counterfactuals. Motivated by these observations, we
propose Source-free Knowledge Distillation of Counterfactual Generative
Networks (SKDCGN), which aims to use the knowledge embedded in a pre-trained
CGN to train a lower capacity model, assuming black-box access (i.e., only
inputs and outputs) to the components of the source model. More specifically,
we harness the idea of Knowledge Distillation (KD) [6] to train a network
comprising three (small) generative models, i.e. TinyGANs [7], each being
responsible for a single independent mechanism. SKDCGN carries both practical
and theoretical implications, and it is intended to:

  1. 1.

Obtain a lightweight version of the CGN, reducing its computational cost and
memory footprint. This is meant to (i) ease the generation of counterfactual
datasets and hence encourage the development of robust and invariant
classifiers, as well as (ii) potentially allowing the deployment of the model
on resource-constrained devices.

  2. 2.

Explore whether we can learn from a fully trained CGN and distill it to a less
parameterized network, assuming that we do not have access to the training
process of the model.

Along the lines of the original paper, we demonstrate the ability of our model
to generate counterfactual images on ImageNet-1k [8] and Double-Colored MNIST
[3]. Furthermore, we compare our outputs to [3] and a simple baseline in terms
of out-of-distribution robustness on the original classification task. As an
additional contribution, we conduct a study on the shape IM of the CGN.

The paper is organized as follows: firstly, we present a brief literature
survey in Section 2; next in Section 3 the SKDCGN is dissected; Section 4
presents the experimental setup and the empirical results, which are finally
discussed in Section 5.

##  2 Related work

This section introduces the fundamental concepts and the related works that we
use as a base for our SKDCGN.

####  2.0.1 Counterfactual Generative Networks.

The main idea of CGNs [3] has already been introduced in Section 1.
Nonetheless, to aid the understanding of our method to readers that are not
familiar with the CGN architecture, we summarize its salient components in
this paragraph and also provide the network diagram in Appendix 0.A.1, Figure
6. The CGN consists of 4 backbones: (i) the part of the network responsible
for the shape mechanism, those responsible for (ii) texture and (iii)
background, and a (iv) composition mechanism that combines the previous three
using a deterministic function. Given a noise vector $\mathbf{u}$ (sampled
from a spherical Gaussian) and a label $y$ (drawn uniformly from the set of
possible labels y) as input, (i) the shape is obtained from a BigGAN-deep-256
[4], whose output is subsequently passed through a U2-Net [5] to obtain a
binary mask of the object shape. The (ii) texture and (iii) background are
obtained similarly, but the BigGAN’s output does not require to be segmented
by the U2-Net. Finally, the (iv) composition mechanism outputs the final
counterfactual image $\mathbf{x}_{gen}$ using the following analytical
function:

$\mathbf{x}_{gen}=C(\mathbf{m},\mathbf{f},\mathbf{b})=\mathbf{m}\odot\mathbf{f}+(1-\mathbf{m})\odot\mathbf{b},$
(1)

where $\mathbf{m}$ is the shape mask, $\mathbf{f}$ is the foreground (or
texture), $\mathbf{b}$ is the background and $\odot$ denotes element-wise
multiplication.

More recently, [9] devises an approach that learns a latent transformation
that generates visual CFs automatically by steering in the latent space of
generative models. Additionally, [10] uses a deep model inversion approach
that provides counterfactual explanations by examining the area of an image.

####  2.0.2 Knowledge Distillation.

[11] firstly proposed to transfer the knowledge of a pre-trained cumbersome
network (referred to as the teacher) to a smaller model (the student). This is
possible because networks frequently learn low-frequency functions among other
things, indicating that the learning capacity of the big network is not being
utilized fully [1] [2]. Traditional KD approaches (often referred to as black-
box) simply use the outputs of the large deep model as the teacher knowledge,
but other variants have made use of activation, neurons or features of
intermediate layers as the knowledge to guide the learning process [12, 13].
Existing methods like [7] are also making use of Knowledge distillation for
the task of image generation. Our work is similar to this, however, they
transfer the knowledge of BigGAN trained on ImageNet dataset to a TinyGAN. In
contrast, in our work, we transfer not just the knowledge of image generation
but also the task of counterfactual generation from a BigGAN to a TinyGAN.

####  2.0.3 Distilling GANs using KD.

Given its high effectiveness for model compression, KD has been widely used in
different fields, including visual recognition and classification, speech
recognition, natural language processing (NLP), and recommendation systems
[14]. However, it is less studied for image generation. [15] firstly applied
KD to GANs. However, our project differs from theirs as they use unconditional
image generation, less general (DCGAN [16]) architectures and they do not
assume a black-box generator. Our setting is much more similar to that of [7],
where a BigGAN is distilled to a network with 16$\times$ fewer parameters,
assuming no access to the teacher’s training procedure or parameters.
Considering its competitive performance, we use the proposed architecture
(TinyGAN) as the student model and use a modified version of their loss
function (further details in Section 3.1.1) to optimize our network.

Source-free: We term our method as Source-free since we do not have access to
the source data, source training details, procedure, and any knowledge about
the counterfactuals, etc, but only have access to trained source models. This
method is similar to methods such as [17] [18]. With large diffusion models
like Imagen [19] and DALL·E 2 [20] where the training process is usually
extremely expensive in terms of computation, lack precise details about
training them and often not reproducible by academic groups, we often have
access to pretrained models. These can be used to transfer knowledge to a
smaller network, and perform the same task with model of lower capacity.

##  3 Approach

This section dives into the details of the SKDCGN architecture, focusing on
the training and inference phases separately for ImageNet-1k and MNIST. In
addition, we discuss the loss functions that were employed for Knowledge
Distillation.

###  3.1 SKDCGN

Although transferring the knowledge of an entire CGN into a single generative
model could drastically reduce the number of parameters, this strategy would
compromise the whole purpose of CGNs, i.e. disentangling the three mechanisms
and having control over each of them. Therefore, we opt to train a generative
model for each individual component. As shown in the architecture diagram
(Figure 1), we treat each IM backbone as a black-box teacher and aim to mimic
its output by training a corresponding TinyGAN student. Note that this implies
that in the case of the shape mechanism, a single generative model learns to
mimic both the BigGAN and the U2-Net. We believe a TinyGAN should be capable
of learning binary masks directly, removing the need for the U2-Net and
reducing the model size even further. During inference, the outputs of the
three students are combined into a final counterfactual image using the
composition function defined in Equation 1.

Figure 1: Architecture of the SKDCGN. During training, each independent
mechanism serves as a black-box teacher model to train a corresponding student
model. During inference, the outputs of the three trained TinyGANs are
combined using a Composition Mechanism that returns the final counterfactual
image.

####  3.1.1 Training: Distilling the knowledge of IMs.

To train SKDCGN, we utilize each IM backbone from the CGN architecture as a
black-box teacher for the student network, as visualized in the training
section of Figure 1 (the backbones are BigGAN + U2-Net for shape, BigGAN for
texture, and BigGAN for background). As introduced in the Related work
section, [7] proposed an effective KD framework for compressing BigGANs. As
the IMs in CGNs rely on BigGANs, we utilize their proposed student
architecture. For completeness, the details of the student architecture are
reported in Appendix 0.A.2, Figure 7.

We base our training objective on the loss function proposed by [7]. Our full
objective comprises multiple terms: (i) a pixel-wise distillation loss, (ii)
an adversarial distillation loss, (iii) a feature-level distillation loss, and
(iv) KL Divergence. In addition to introducing KL Divergence, we deviate from
the original TinyGAN training objective by omitting the term that allows the
model to learn from real images of the ImageNet dataset. This would inevitably
compromise the quality of the generated counterfactuals. KL Divergence leads
to entropy minimization between the teacher and student, which is why we
propose its usage.

The individual loss terms are dissected below as from [7]:

  1. 1.

Pixel-wise Distillation Loss: To imitate the functionality of BigGAN for
scaling generation to high-resolution, high-fidelity images, we minimize the
pixel-level distance (L1) between the images generated by BigGAN and TinyGAN
given the same input:

$\mathcal{L}_{\text{KD\\_pix}}=\mathbb{E}_{z\sim p(z),y\sim
q(y)}[\|T(z,y)-S(z,y)\|_{1}]$ (2)

where $T$ represents the Teacher network, $S$ represents the Student network,
$z$ is a latent variable drawn from the truncated normal distribution $p(z)$,
and $y$ is the class label sampled from some categorical distribution $q(y)$.

  2. 2.

Adversarial Distillation Loss: To promote sharper outputs, an adversarial loss
is incorporated to make the outputs of $S$ indistinguishable from those of
$T$. It includes a loss for the generator (Eq. 3) and one for the
discriminator (Eq. 4):

$\displaystyle\mathcal{L}_{\text{KD\\_G}}=$
$\displaystyle-\mathbb{E}_{z,y}[D(S(z,y),y)]$ (3)
$\displaystyle\mathcal{L}_{\text{KD\\_D}}=$
$\displaystyle-\mathbb{E}_{z,y}\left[max(0,1-D(T(z,y),y))+max(0,1-D(S(z,y),y))\right],$
(4)

where $z$ is the noise vector, $y$ is the class label, $T(z,y)$ is the image
generated by the Teacher $T$, while $G$ and $D$ are – respectively – the
generator and discriminator of the Student $S$.

  3. 3.

Feature Level Distillation Loss: To further overcome the blurriness in the
images produced by the Student network, the training objective also includes a
feature-level distillation loss. More specifically, we take the features
computed at each convolutional layer in the Teacher discriminator, and with a
loss function stimulate $S$ to generate images similar to $T$:

$\mathcal{L}_{\text{KD\\_feat}}=\mathbb{E}_{z,y}\left[\sum_{i}\alpha_{i}\left\|D_{i}(T(z,y),y)-D_{i}(S(z,y),y)\right\|_{1}\right]$
(5)

where $D_{i}$ represents the feature vector extracted from the $i^{th}$ layer
of the discriminator and the corresponding weights are given by $\alpha_{i}$.

  4. 4.

KL Divergence: L1 alone cannot reduce the entropy between the teacher and
target. To improve the proposed method, we use KL Divergence in a similar
fashion to [21] for the task of knowledge distillation between real images
drawn from source $P(x)$ and target images $Q(x)$. The

$\mathcal{D}_{\mathrm{KL}}(P\|Q)=\sum_{x\in\mathcal{X}}P(x)\log\left(\frac{P(x)}{Q(x)}\right)$
(6)

$\mathcal{L}_{\text{KL}}=\sum_{x\in X}-p_{x}^{t}\log p_{x}^{s}+p_{x}^{t}\log
p_{x}^{t}$ (7)

where $x$ is the class label and $p$ contains the output softmax probabilities
of the Generator $G$ divided by the temperature $t$.

To sum up, the student’s generator ($G$) and discriminator ($D$) are
respectively optimized using the following objectives:

$\displaystyle\mathcal{L}_{\text{G}}=$
$\displaystyle\mathcal{L}_{\text{KD\\_feat}}+\lambda_{1}\mathcal{L}_{\text{KD\\_pix}}+\lambda_{2}\mathcal{L}_{\text{KD}\\_G}\,(\;+\;\mathcal{L}_{\text{KL}}\,)$
(8) $\displaystyle\mathcal{L}_{\text{D}}=$
$\displaystyle\mathcal{L}_{\text{KD\\_D}}$ (9)

where $\lambda_{1}$ and $\lambda_{2}$ are the regularization terms mentioned
in [7], and the KL divergence term ($\mathcal{L}_{\text{KL}}$) is only used in
the enhanced version of SKDCGN.

Implementing the SKDCGN architecture requires training a TinyGAN for each
Independent Mechanism of the CGN (see Fig. 1). The KD training procedure,
however, requires training data. Hence prior to training, 1000 images per
class (totalling 1 million samples) are generated using the IM backbones
extracted from the pre-trained CGN (as provided by Sauer and Geiger [3]).

Finally, note that the original CGN architecture (illustrated in Appendix
0.A.1, Figure 6) comprises another BigGAN trained on ImageNet-1k. It is
unrelated to the three Independent Mechanisms and provides primary training
supervision via reconstruction loss. We discard this component of the
architecture for two main reasons: we do not have a dataset of counterfactuals
whereon a GAN can be trained; we argue that this additional knowledge is
already embedded in the backbones of a pre-trained CGN.

####  3.1.2 Inference: generating counterfactuals.

Once the three student networks are trained, their outputs are combined during
inference akin to [3] using the analytical function of Equation 1. Since the
composition function is deterministic, we devise inference as a separate task
to training.

##  4 Experiments and results

This section defines our experimental setup, then proceeds to present the
results. First, we test SKDCGN – as defined in the Approach section – on both
ImageNet-1k and MNIST (Section 4.3), and based on the observed findings we
make some changes to the proposed architecture to improve the quality of the
results (Section 4.4). Due to computational constraints we test these
improvements on a smaller dataset, namely the double-colored variant of MNIST
[22]. Finally, as an additional contribution, we conduct a thorough study on
the composition mechanism, to gain a better understanding of how each
mechanism influences the classification accuracy of an invariant classifier.
We present the results of such a study in Section 4.5.

###  4.1 Datasets

##### ImageNet-1k.

The ImageNet-1k ILSVRC dataset [8] contains 1,000 classes, with each class
consisting of 1.2 million training images, 50,000 validation and 100,000 test
images. Images were resized to $256\times 256$ to maintain consistent
experiments and to allow direct comparisons with the original results of [3].

##### Double-colored MNIST.

We use the double-colored MNIST dataset proposed by Sauer and Geiger in the
original CGN paper [3]. This is a variant of the MNIST dataset where both the
digits and the background are independently colored. It consists of 60,000
$28\times 28$ images of the 10 digits, along with a test set of 10,000 images.

###  4.2 Baseline Model: CGN with generator replaced by TinyGAN generator

The SKDCGN is compared with a modified version of the original CGN
architecture, where each BigGAN has been replaced by the generator model of a
TinyGAN. Training this baseline using the procedure described by [7], omitting
KD, allows for rigorous comparisons that emphasize the effectiveness of the
knowledge distillation process. Further training details are provided in
Appendix 0.C.1.

###  4.3 Results of SKDCGN

ImageNet-1k Double-colored MNIST

(a) Shape mechanism.

ImageNet-1k Double-colored MNIST

(b) Texture mechanism.

ImageNet-1k Double-colored MNIST

(c) Background mechanism.

Figure 2: A comparison of images (on both ImageNet-1k and double-colored
MNIST) generated by the CGN backbones and those generated by the corresponding
SKDCGN’s TinyGAN (given the same input), for each independent mechanism.

The proposed model was firstly trained and tested on ImageNet-1k. To further
validate our method, we repeated the training procedure on MNIST.

The qualitative results are collected in Figure 2 and demonstrate that
TinyGANs can closely approximate the output of each IM. While this is true for
both datasets, the effectiveness of our method is especially visible in the
case of MNIST. It is likely the case that the reduced capacity of the TinyGANs
(compared to the original CGN backbones) is sufficient to decently model the
underlying data distribution. ImageNet-1k, on the other hand, reveals more
apparent (though still acceptable) discrepancies between the images,
especially for the texture IM.

However, careful and extensive experiments revealed that the three TinyGANs
could not generalize when random noise was given to the generator, i.e., they
could not produce results beyond the test set. This might be due to a number
of reasons. First, the compromised generalization capabilities of each IM’s
TinyGAN could be caused by their reduced network capacity. Furthermore, each
TinyGAN was trained on all 1000 classes of ImageNet-1K, as opposed to Chang
and Lu’s choice of limiting the training data to the 398 animal labels [7].
Finally, we generate the test samples using the test noise instead of random
noise, since we hypothesize that the student networks only learn the manifolds
that the teacher networks have been trained on. Additional experiments are
required to analyze whether samples generated using random noise are found
along the same manifold; unfortunately, we were hindered by the limited time
frame allocated for this project, hence we leave this question open for future
works.

(a) Shape mechanism.

(b) Texture mechanism.

(c) Background mechanism.

Figure 3: A comparison of double-colored MNIST images generated by the CGN
backbones and those generated by the corresponding SKDCGN’s TinyGAN (given the
same input) for each IM. Here, SKDCGN was tuned such that KL divergence is
minimized between the teacher and student networks, and the L1 loss is
multiplied with the activation of every layer.

(a)

(b)

Figure 4: (a) Shape masks obtained after the first epoch of SKDCGN training on
ImageNet-1k, using KL divergence. (b) Shape masks obtained after the
23${}^{\text{rd}}$ epoch of SKDCGN training on ImageNet-1k, without KL
divergence. Evidently, KL enhances the quality of the masks from the first
epoch, whereas its absence compromises the results even at a later stage of
training.

###  4.4 Improving the SKDCGN model

The results presented in the previous section reveal that the outputs are
noisy and ambiguous in nature when knowledge distillation is performed using
the pre-trained models provided by Sauer and Geiger [3] (note the artifacts in
the SKDCGN’s outputs of Fig. 2, especially those trained on ImageNet-1k). This
statement was supported by an interesting yet unexpected result of the study
on the composition mechanism (refer to Section 4.5): it was observed that
modifying Equation 1 such that the shape mask $\mathbf{m}$ is multiplied with
a weight factor of 0.75 (i.e., setting the transparency of the shape mask to
75%), yielded an accuracy increase of the CGN’s invariant classifier. The
findings of this experiment – conducted on the double-colored MNIST dataset –
suggest that the mask component is noisy in nature, leading to ambiguities in
the decision boundaries during the classification of several digits.

In light of this new hypothesis, we attempt to use the Kullback–Leibler (KL)
divergence to improve the visual quality of the outputs111It is noteworthy
that other techniques were tested in the attempt to improve the visual quality
of the results. Although they did not prove to be as beneficial, they are
described in Appendix 0.D.. Since KL leads to entropy minimization between the
teacher and student networks, we deem such a technique adequate for the task
at hand. Moreover, the choice of using KL was encouraged by the work of Asano
and Saeed [21], which proved the suitability of the measure in this context.
Concretely, the KL Divergence loss (as defined in Eq. 7) was included in the
overall generator loss $\mathcal{L}_{\text{G}}$ as seen in Equation 3.

First, the modified SKDCGN was tested on the double-colored MNIST dataset. As
depicted in Figure 3, the introduction of KL divergence improves SKDCGN’s
visual fidelity of both background and texture IMs, while the quality of the
shape masks seems to diminish after a few epochs. Contrarily, this approach
appeared to be beneficial for the shape mechanism too, in the context of
ImageNet-1k. The shape masks resulted more natural and consistent since the
first epoch, whereas the absence of KL yielded noisy masks even at a later
stage of training (refer to Figure 4).

###  4.5 Additional results: study of the shape IM

|  Noise |  Rotation |  Transparency
---|---|---|---
Train Accuracy | $99.9$ | $99.1$ | $94.7$
Test Accuracy | $14.96$ | $13.51$ | $\mathbf{58.86}$
Table 1: Results of the invariant classifier for the analysis of the shape IM.
The classifier has been trained to predict whether images are CGN-generated or
real. The training examples contain counterfactuals whose shape mechanism has
been tuned with one of the three transformations indicated in the table
(noise, rotation, transparency – refer to Sec.4.5 for further details).

(a)

(b)

(c)

Figure 5: Shape masks obtained after (a) addition of Gaussian random noise,
(b) application of random rotation and (c) decrease of the mask opacity (i.e.,
lowering its transparency to 75%).

As an additional contribution, we conduct a thorough study on the composition
mechanism, to gain a better understanding of how the mechanisms influence the
classification accuracy of an invariant classifier (i.e., a classifier that
predicts whether an image is CGN-generated or real). Due to the limited time
at our disposal, we focused on the mechanism that we deem most important in
the decision-making of such a classifier, namely the shape. To evaluate the
effects of the shape IM we trained several (original) CGN models on the
double-colored MNIST dataset; we tuned the resulting shape masks prior to the
counterfactual image generation (governed by the composition mechanism of
Equation 1) and used the generated images to train an invariant classifier.
More specifically, we experimented with (i) the addition of Gaussian noise in
the shape mask, (ii) random rotation of the mask, and (iii) multiplying the
mask $\mathbf{m}$ in the composition mechanism (Eq. 1) with a factor smaller
than 1 (or in other words, lowering the opacity of the shape mask). A
transparency of 75% (hence a weight factor of $0.75$) was experimentally found
to be most beneficial for the accuracy of the classifier.

The influence of the three transformations on the invariant classifier is
quantified – in terms of accuracy – in Table 1; sample shape masks generated
from each transformation are displayed in Figure 5. It is apparent from the
test accuracy values that Gaussian noise and random rotations do not lead to
any remarkable performance of the classifier but, contrarily, degrade its
accuracy to values below 15%. This is most likely the result of overfitting on
the training set, as supported by the train accuracy values. On the other
hand, lowering the opacity of the mask substantially boosts the test accuracy,
improving the previous results by a factor of $4\times$ (circa). It is
noteworthy that the masks obtained using the transparency adjustment are more
akin to those achieved using regular CGNs (see Figure 5). The other
transformations, instead, result in mask shapes that are particularly
different. As such, they can potentially be used to make classifiers more
robust when mixed with regular data during training. Because this is an
extensive topic, we believe it warrants further research.

##  5 Discussion and conclusion

With the prevalence of heavily parameterized architectures such as BigGANs,
and with the advent of limited-access models like the trending DALL·E 2,
source-free compression becomes a growing necessity. In this paper we explored
the possibility to obtain a lightweight version of the CGN network, assuming
that we do not have access to the training process of the model. More
specifically, we treat the backbone of each independent mechanism (shape,
texture and background) as a black-box, then use KD to transfer the knowledge
of the pre-trained cumbersome networks to simple TinyGANs.

SKDCGN achieves a remarkable compression of the overall network: it models the
shape mechanism – initially controlled by a BigGAN (55.9M parameters) and a
U2-Net (44M parameters) – using a single TinyGAN (6.4M parameters); similarly,
it replaces the BigGANs responsible for the texture and background IMs with
TinyGANs, and discards the forth BigGAN of the original CGN network that
provides primary training supervision via reconstruction loss. This translates
into four BigGANs and one U2-net (55.9M$\times$4 + 44M parameters, totalling
267.6M) being replaced with three simple TinyGANs (6.4M parameters each,
meaning 19.2M parameters in total).

Despite the significant compression, we demonstrate the ability of our model
to generate counterfactual images on ImageNet-1k and double-colored MNIST
datasets (see Figure 2). When trained on the latter, SKDCGN’s network capacity
was proven to be sufficient to model the simple data distribution. If trained
on the former, the proposed method exhibited remarkable ability in mimicking
the original shape and background generations, while the texture mechanism
suffered more from the reduction of size. This finding reveals great potential
for future works that would attempt to tune the distillation (and hence
enhance the synthesis) of the texture images, for instance by including data
augmentation in the training procedure.

Given the obtained results, we attemptedly limit the presence of noisy and
ambiguous artifacts by minimizing the entropy between the teacher and student
networks. We introduce a new measure in the knowledge distillation loss, i.e.
KL divergence, which we find to enhance the visual quality results of some IMs
for both Imagenet-1k and MNIST.

Finally, we conduct a study on the composition mechanism to gain a better
understanding of how the shape IM influences the classification accuracy of an
invariant classifier. Though other adjustments were tested, giving a lower
weight to the shape mask $\mathbf{m}$ seemingly boosts the classifier
performance.

##  6 Future work

To conclude, the experimental findings of SKDCGN prove that, upon the usage of
Knowledge Distillation, one can transfer the capacity/ability of a cumbersome
network to a lower-capacity model while still maintaining competitive
performances. Although this paper unveils its potential, SKDCGN requires
further research that we encourage other researchers to undertake. In addition
to the suggestions offered throughout the sections, possible avenues of
research include and are not limited to: improving the image generation
process by using higher-order activation functions, since the utilized
datasets consist of rich image data; improving the teacher-student
architecture by introducing additional loss functions; using a learnable,
neural network-based composition function instead of an analytical expression.

## Acknowledgments

We would like to express our sincere gratitude to Prof. dr. Efstratios Gavves
and Prof. Wilker Aziz for effectively organizing the Deep Learning II course
at the University of Amsterdam, which is the main reason this paper exists. We
are thankful to our supervisor, Christos Athanasiadis, for his precious
guidance throughout the project. Finally, we also thank the former Program
Director of the MSc. Artificial Intelligence, Prof. dr. Cees G.M. Snoek, and
the current Program Manager, Prof. dr. Evangelos Kanoulas, for effectively
conducting the Master’s program in Artificial Intelligence at the University
of Amsterdam.

## Appendix

##  Appendix 0.A Architecture details of the different models

This section contains the architectural details of the different model used in
the proposed method. It brushes up the theory of the papers whereon we base
our work (i.e. the CGN network [3], Sec. 0.A.1 and the TinyGAN model [7], Sec.
0.A.2) and also presents the baseline model (Sec. 0.A.3).

###  0.A.1 Original CGN architecture

This section contains a diagram of the original CGN architecture, as presented
in [3].

Figure 6: CGN architecture diagram. Retrieved from [3].

Figure 6 illustrates the CGN architecture. The network is split into four
mechanisms, the shape mechanism $f_{shape}$, the texture mechanism $f_{text}$,
the background mechanism $f_{bg}$, and the composer $C$. Components with
trainable parameters are blue, components with fixed parameters are green. The
primary supervision is provided by an unconstrained conditional GAN (cGAN) via
the reconstruction loss $\mathcal{L}_{rec}$. The cGAN is only used for
training, as indicated by the dotted lines. Each mechanism takes as input the
noise vector $\mathbf{u}$ (sampled from a spherical Gaussian) and the label
$y$ (drawn uniformly from the set of possible labels $\mathcal{Y}$) and
minimizes its respective loss ($\mathcal{L}_{shape}$, $\mathcal{L}_{text}$,
and $\mathcal{L}_{bg}$). To generate a set of counterfactual images, we sample
$\mathbf{u}$ and then independently sample $y$ for each mechanism.

###  0.A.2 TinyGAN architecture

(a) Student Generator $G$ [7]

(b) A Residual Block in $G$ [7]

Figure 7: Architecture of the TinyGAN (student) generator

This section provides an brief overview of the TinyGAN architecture. For more
details, refer to [7].

##### Generator.

As shown in Figure 7, TinyGAN comprises a ResNet [23]-based generator with
class-conditional BatchNorm [24] [25]. To keep a tight computation budget, it
does not adopt attention-based [26] or progressive-growing mechanisms [27]. To
substantially reduce the model size compared to BigGAN, it:

  * •

Relies on using fewer channels;

  * •

Replaces standard convolution by depthwise separable convolution;

  * •

Adopts a simpler way to introduce class conditions.

Overall, TinyGAN’s generator has 16$\times$ less parameters than BigGAN’s
generator.

##### Discriminator.

Following [28] [29], [7] opt for spectral normalized discriminator and
introduce the class condition via projection. But instead of utilizing
complicated residual blocks, they simply stack multiple convolutional layers
with stride as used in DCGAN [16], which greatly reduces the number of
parameters.

Overall, TinyGAN’s discriminator has 10$\times$ less parameters than BigGAN’s
discriminator.

###  0.A.3 Baseline model

The baseline is a standard CGN architecture whose BigGANs have been replaced
with TinyGANs. Due to the need of a pre-trained model that (i) supervises the
CGN training using a reconstruction loss and (ii) serves as the initialization
of the IM GANs, a TinyGAN was trained from scratch using the KD strategy
described in [7]. Section 0.C dives into the details of the training
procedure, then presents qualitative results of both the newly-trained TinyGAN
and of baseline model.

ImageNet-1k Double-colored MNIST

(a) Shape mechanism.

ImageNet-1k Double-colored MNIST

(b) Texture mechanism.

ImageNet-1k Double-colored MNIST

(c) Background mechanism.

Figure 8: A comparison of images generated by the CGN backbones and those
generated by the corresponding SKDCGN’s TinyGAN (given the same input), for
each independent mechanism. We train on both ImageNet-1k (left images) and
double-colored MNIST datasets (right images).

##  Appendix 0.B Additional results of SKDCGN’s IMs

This section expands Section 4.3 of the main paper and contains more results
obtained from each SKDCGN’s IM, using both ImageNet-1k and double-colored
MNIST datasets. More specifically, we compare the output of each CGN backbone
with that of the corresponding SKDCGN’s TinyGAN, given the same input. Please
refer to Figure 8.

##  Appendix 0.C Baseline Model

The baseline model is a modified version of the original CGN architecture,
where each BigGAN has been replaced by the generator model of a TinyGAN.
Training this baseline using the procedure described by [7], omitting KD,
allows for rigorous comparisons that emphasize the effectiveness of the
knowledge distillation process. In this section we provide training details,
and collect sample outputs of the trained model.

(a) A comparison of images generated by BigGAN and the TinyGAN. Images in top
row are produced by BigGAN, while those in bottom row are by SKDCGN given the
same input after $1^{st}$ epoch.

(b) A comparison of images generated by BigGAN and the TinyGAN. Images in top
row are produced by BigGAN, while those in bottom row are by SKDCGN given the
same input after $18^{th}$ epoch.

Figure 9: A comparison of images generated by BigGAN and the TinyGAN. Images
in top row are produced by BigGAN, while those in bottom row are by SKDCGN
given the same input

###  0.C.1 Training Details

The training procedure of a CGN requires a pre-trained GAN to provide primary
supervision via the reconstruction loss. However, the original TinyGAN was
only trained on only animal classes, hence the publicly-available model could
not be used for our baseline. In order to consistently use the same dataset
for all the experiments, we re-trained a TinyGAN from scratch (as described in
[7]) on all classes of ImageNet-1k. The images generated by TinyGAN are
visualized in Appendix 0.C.1.1. The images generated for each Independent
Mechanism using our baseline model can be seen in 0.C.1.2. Apart from this, we
additionally generated the counterfactuals using the baseline model which are
shown in Appendix 0.C.1.3.

$\tilde{m}$ |
---|---
$m$ |
$f$ |
$b$ |
$x_{gen}$ |
Figure 10: Individual IM Outputs after training for baseline. From top to
bottom: $m$, $\tilde{m}$, $f$, $b$, $x_{gen}$. From left to right: at the
start of training, after epoch $300k{th}$, epoch $600k^{th}$, epoch
$900k^{th}$, and epoch $1.2million^{th}$

####  0.C.1.1 Generated outputs of TinyGAN trained on ImageNet-1k

A TinyGAN was trained using all 1000 classes of the ImageNet-1k dataset.
Training details are provided by [7]. Although the original paper trains the
model for 1.2 million epochs, we are forced to restrict the amount of
iterations due to computational constraints. After distilling the knowledge of
a BigGAN for 18 epochs, our TinyGAN generates reasonable images, as seen in
Figure 9(b). To compare the image generation we have also presented images
generated after the first epoch as well 9(a). It can be observed that if we
further train the model, it could produce images better in quality. Note that
animal classes are better captured by the model: this is inline with the
findings of [7].

---
Figure 11: Counterfactuals generated by baseline on test data for ImageNet-1k

####  0.C.1.2 Generated outputs of the baseline trained on ImageNet-1k

Figure 10 illustrates the individual outputs of each IMs at the start of
training, after epoch 300k${}^{\text{th}}$, epoch 600k${}^{\text{th}}$, epoch
900k${}^{\text{th}}$, and epoch 1.2M${}^{\text{th}}$ (from left to right). In
each figure, we show from top to bottom: pre-masks $\tilde{m}$, masks $m$,
texture $f$, background $b$, and composite images $x_{gen}$.

####  0.C.1.3 Generated Counterfactual Images of Baseline trained on
ImageNet-1k

Finally, we show counterfactual images generated by the baseline model in
Figure 11.

##  Appendix 0.D Improving the SKDCGN process

As mentioned in Section 4.4 of the main paper, we observed that the outputs
from CGN are noisy in nature. Fig 12 evidently illustrates how noisy the MNIST
digits are. However in this section we try to improve our architecture by
several methods.

Figure 12: Noisy outputs generated by the CGN when we made use of pretrained
weights given by the authors.

In the direction towards improving the images that are being generated by our
architecture, we strongly believe the room of improvement lies in these
components:

  * •

Improving the quality of images that are being generated by the GAN network in
our architecture. Usually loss functions like VGG based perception loss, L1
reconstruction loss are added.

  * •

Improving the existing knowledge distillation framework such that the student
learns better from the teacher’s guidance by adding new loss functions to the
Knowledge Distillation task.

(a) A comparison of images generated by the CGN shape backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
after 2 epochs on test data.

(b) A comparison of images generated by the CGN texture backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
after 10 epochs on test data.

(c) A comparison of images generated by the background backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
after 30 epochs on test data.

Figure 13: A comparison of images generated by the CGN backbones and those
generated by the corresponding SKDCGN (given the same input) mask IM with
cross entropy loss

To improve the quality of images, we observe that our architecture already has
most of the loss functions integrated implicitly/explicitly. Hence, we add the
Cross entropy loss for the generator and discriminator for the mask IM of the
architecture and get the results as shown in 13(a) for second epoch. We
observe that digits like ’0’ are being reconstructed however for other digits
the inputs look noisy in nature. By the end of 10th epoch for test set in Fig.
13(b) we observe that the digits are being reconstructed. We continue with the
training since we expected better results than what we have a;ready seen,
however, contrary to our beliefs we observe artefacts by the end of 30th epoch
as shown in Fig. 13(c).

(a) A comparison of images generated by the CGN shape backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
after 2 epochs on test data.

(b) A comparison of images generated by the CGN texture backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
after 10 epochs on test data.

(c) A comparison of images generated by the background backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
after 30 epochs on test data.

Figure 14: A comparison of images generated by the CGN backbones and those
generated by the corresponding SKDCGN (given the same input) mask IM with KL
divergence multiplied with the activation of every layer instead of L1

###  0.D.1 KL multiplied with layer instead of L1

Since the image generation process already has most of the components to
ensure that the reconstruction is in place, we tried to improve the Knowledge
distillation between teacher and student network by integrating the KL
divergence and multiply the loss with every layer of the network instead of L1
which is default. Possibly, because L1 reconstruction loss is explicitly
needed that is to multiplied with the activation of every layer. We observe
the results as shown in Fig. 14

(a) A comparison of images generated by the CGN shape backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
for mask IM after 2 epochs on test data.

(b) A comparison of images generated by the CGN texture backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
for mask IM after 10 epochs on test data.

(c) A comparison of images generated by the background backbone (top row) and
those generated by the corresponding SKDCGN given the same input (bottom row)
for mask IM after 30 epochs on test data.

Figure 15: A comparison of images generated by the CGN backbones and those
generated by the corresponding SKDCGN (given the same input) mask IM with L2
multiplied with the activation of every layer instead of L1.

###  0.D.2 MSE instead of L1

In addition, We also tried L2 loss instead of L1 loss but it lead to noisy
outputs than previously generated and obtain results as shown in 15. Since, L2
assumes that the influence of noise is independent of the image’s local
characteristic the images are noisy in nature.

## References

  * [1] Sara Fridovich-Keil, Raphael Gontijo Lopes, and Rebecca Roelofs.  Spectral bias in practice: The role of function frequency in generalization.  CoRR, abs/2110.02424, 2021.
  * [2] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann.  Shortcut learning in deep neural networks.  CoRR, abs/2004.07780, 2020.
  * [3] Axel Sauer and Andreas Geiger.  Counterfactual generative networks.  CoRR, abs/2101.06046, 2021.
  * [4] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis, 2019\.
  * [5] Xuebin Qin, Zichen Vincent Zhang, Chenyang Huang, Masood Dehghan, Osmar R. Zaïane, and Martin Jägersand.  U${}^{\mbox{2}}$-net: Going deeper with nested u-structure for salient object detection.  CoRR, abs/2005.09007, 2020.
  * [6] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  CoRR, abs/2106.05237, 2021.
  * [7] Ting-Yun Chang and Chi-Jen Lu.  Tinygan: Distilling biggan for conditional image generation.  CoRR, abs/2009.13829, 2020.
  * [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [9] Saeed Khorram and Li Fuxin.  Cycle-consistent counterfactuals by latent transformations, 2022.
  * [10] Jayaraman J. Thiagarajan, Vivek Sivaraman Narayanaswamy, Deepta Rajan, Jason Liang, Akshay Chaudhari, and Andreas Spanias.  Designing counterfactual generators using deep model inversion.  CoRR, abs/2109.14274, 2021.
  * [11] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  In NIPS Deep Learning and Representation Learning Workshop, 2015\.
  * [12] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets, 2014.
  * [13] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai.  Variational information distillation for knowledge transfer.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9155–9163, 2019.
  * [14] Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao.  Knowledge distillation: A survey.  CoRR, abs/2006.05525, 2020.
  * [15] Angeline Aguinaldo, Ping-Yeh Chiang, Alexander Gain, Ameya Patil, Kolten Pearson, and Soheil Feizi.  Compressing gans using knowledge distillation.  CoRR, abs/1902.00159, 2019.
  * [16] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
  * [17] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui.  Generalized source-free domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8978–8987, 2021.
  * [18] Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao.  Source-free domain adaptation via distribution estimation.  arXiv preprint arXiv:2204.11257, 2022.
  * [19] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [20] Gary Marcus, Ernest Davis, and Scott Aaronson.  A very preliminary analysis of dall-e 2, 2022.
  * [21] Yuki M Asano and Aaqib Saeed.  Extrapolating from a single image to a thousand classes using distillation.  arXiv preprint arXiv:2112.00725, 2021.
  * [22] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.
  * [24] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur.  A learned representation for artistic style, 2016.
  * [25] Harm de Vries, Florian Strub, Jérémie Mary, H. Larochelle, Olivier Pietquin, and Aaron C. Courville.  Modulating early visual processing by language.  In NIPS, 2017.
  * [26] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks, 2018.
  * [27] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation, 2017.
  * [28] Takeru Miyato and Masanori Koyama.  cgans with projection discriminator, 2018.
  * [29] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  CoRR, abs/1802.05957, 2018.