# The cosmic optical background excess, dark matter, and line-intensity
mapping

José Luis Bernal  William H. Miller III Department of Physics and Astronomy,
Johns Hopkins University, 3400 North Charles Street, Baltimore, MD 21218,
United States  Gabriela Sato-Polito  William H. Miller III Department of
Physics and Astronomy, Johns Hopkins University, 3400 North Charles Street,
Baltimore, MD 21218, United States  Marc Kamionkowski  William H. Miller III
Department of Physics and Astronomy, Johns Hopkins University, 3400 North
Charles Street, Baltimore, MD 21218, United States

###### Abstract

Recent studies using New Horizons LORRI images have returned the most precise
measurement of the cosmic optical background to date, yielding a flux that
exceeds that expected from deep galaxy counts by roughly a factor of two. We
investigate whether this excess, detected at $\sim 4\sigma$ significance, is
due to dark matter that decays to a monoenergetic photon with a rest-frame
energy in the range $0.5-10$ eV. We derive the spectral energy distribution
from such decays and the contribution to the flux measured by LORRI. The
parameter space that explains the measured excess with decays to photons with
energies $E\gtrsim 4$ eV is unconstrained to date. If the excess arises from
dark-matter decay to a photon line, there will be a significant signal in
forthcoming line-intensity mapping measurements. Moreover, the ultraviolet
instrument aboard New Horizons (which will have better sensitivity and probe a
different range of the spectrum) and future studies of very high-energy
$\gamma$-ray attenuation will also test this hypothesis and expand the search
for dark matter to a wider range of frequencies.

The cosmic optical background (COB), the integrated flux of optical light,
allows us to test whether our census of known galaxies is complete
Cooray_review . Measuring the COB is challenging due to the overwhelming
presence of foregrounds—especially zodiacal light (sunlight scattered by
interplanetary dust)—which requires very accurate and precise calibration and
modeling to be subtracted from the observations. Foreground removal can be
achieved at the expense of large uncertainties in the COB measurement, which
hinders detection from Earth or from Earth-orbiting satellites. There are
$\sim 2\sigma$ hints of the COB, including the 0.40 $\mu$m ‘dark cloud’
measurement Mattila_17 , the 0.80 $\mu$m CIBER flux Matsuura_CIBER , and the
Pioneer 10 and 11 observations Matsuoka_pioneer  taken at $3-5$ AU from the
Sun (but challenged by Ref. Matsumoto_2018 ). Nonetheless, contamination from
zodiacal light is negligible in the outer solar system Zemcov_2018 ;
Poppe_2019 . This makes NASA’s New Horizons spacecraft an extraordinary
platform to measure the COB, as demonstrated by analyses using archival images
of the New Horizons’ Long Range Reconnaisance Imager (LORRI) Zemcov_NHCOB ;
NH21 . The New Horizons’ LORRI has recently provided, using targeted
observations taken at 51.3 AU from the Sun, the first high signal-to-noise
detection of the COB, yielding a flux of photons with wavelengths $\sim
0.4-0.9$ $\mu$m ($\sim 1.3-3$ eV) of $16.37\pm 1.47$ nW/m2/sr NH22 . This
measurement, obtained after subtracting contributions from diffuse Galactic
light, scattered light from stars and galaxies outside the LORRI field, faint
stars below the detection limit, hydrogen and ionized helium two-photon
continua, and foregrounds from the spacecraft, exceeds the flux expected from
deep Hubble Space Telescope galaxy counts by $8.06\pm 1.92$ nW/m2/sr NH22 ;
this is roughly a factor-of-two excess, with $\gtrsim 4\sigma$ significance.
Possible astrophysical explanations include a faint population of galaxies not
accounted for in the prediction from deep counts of HST Conselice_16 , light
from stars tidally removed from galaxies, a population of faint sources within
extended halos (intrahalo light) Cooray_2012 ; Zemcov_2014 ; Matsumoto_19 , or
direct-collapse black holes at very high redshift Yue_DCBH .

Another, more exotic, possibility is that the excess is due to dark-matter
decay. Although dark matter is a cornerstone of the standard cosmological
model, there is currently no satisfactory microscopic model for dark matter.
In many models, dark matter may decay to photons through some weak coupling to
Standard Model particles; examples include the axion Abbott:1982af ;
Dine:1982ah ; Preskill:1982cy ; Weinberg:1977ma ; Wilczek:1977pj ;
Peccei:1977hh ; Peccei:1977ur  and sterile neutrinos Kusenko_SterNu . There
are ongoing efforts to constrain this possibility with cosmological and
astrophysical observations over a vast mass range(see e.g., Refs.
Caputo:2018ljp ; Caputo:2018vmy ; Gong_IR ; Caputo_IR ; Caputo:2019djj ;
Boyarsky:2014ska ; Riemer-Sorensen:2014yda ; Dessert:2018qih ; Blanco_gammaray
; Cohen_gammaray ; Ellis_gravitino ; Hu_DMSpecD ; Chluba_specD ; Iocco_BBN ;
Pospelov_BBN ; Poulin_BBNDM ; Poulin_decDM ; Slatyer_dark ; Slatyer_decDM ;
Lucca_synergy ).

Here we show that if the COB excess is due to dark-matter decay to a
monoenergetic photon, this decay line will be detected with high significance
with forthcoming line-intensity mapping (LIM) experiments SPHEREx ; HETDEX ;
Kovetz:2017agg ; Bernal:2019jdo . LIM experiments infer the three-dimensional
distribution of mass by mapping the cosmic luminosity density of some specific
atomic/molecular emission line; e.g., the 21-cm transition of neutral
hydrogen, rotational lines of carbon monoxide, a bright infrared line of
singly-ionized carbon, or the Lyman- and Balmer-series of neutral hydrogen. If
dark matter decays to a photon line, it will show up in LIM experiments as an
unidentified line Creque-Sarbinowski:2018ebl ; Bernal_limdm .

To be concrete, we focus on the two-photon decay of an axion that makes up all
the dark matter. In this case, the energy density in the COB excess, which is
$\sim 10^{-6}$ of the dark-matter density, implies an axion lifetime $\sim
10^{6}$ times the age $t_{\rm U}$ of the Universe, and so we assume the decay
lifetime is long compared with $t_{\rm U}$ and neglect any variation of the
dark matter density with time due to decays. As we show, such a possibility is
fully consistent with all existing constraints to the axion parameter space.
Our results also apply, however, to a broad range of other (possibly
subdominant) dark-matter candidates that decay to a photon line.

The photons produced in two-body dark-matter decays have a rest-frame
frequency $\nu=m_{\chi}c^{2}/2h$, where $m_{\chi}$ is the dark matter particle
mass and $h$ is the Planck constant. The profile of the emission coming from a
given redshift has a width dependent on the dark matter velocity dispersion,
which is negligible for the frequencies of interest Gong_IR ; hence, we
consider a Dirac delta function profile. The specific intensity $I_{\lambda}$
per observed wavelength $\lambda_{\rm obs}$ in units of power per area per
steradian for the dark matter radiative decay is given by

$I_{\lambda}=\frac{c}{4\pi}\frac{f(z)\Omega_{\rm
dm}\rho_{c}c^{2}\Gamma_{\chi}}{\lambda_{\rm obs}(1+z)H(z)}\,,$ (1)

where $\Omega_{\rm dm}$ and $\rho_{c}$ are the dark matter density parameter
and the critical density today, respectively, $\Gamma_{\chi}$ is the dark
matter decay rate, and $H$ is the Hubble expansion rate at the redshift
$z\equiv z(m_{\chi},\lambda_{\rm obs})$ at which the decay occurs, which in
turns depends on the dark matter mass and the observed wavelength. Assuming
all the dark matter has the same mass, the spectral energy distribution at
each observed wavelength only receives contribution from a single redshift.
The numerator of the second factor is the mean specific luminosity density per
comoving volume produced by dark matter decays Chen:2003gz ; Finkbeiner:2011dx
; Pierpaoli:2003rz , to which we add a fraction $f$ — that may include
redshift dependence — accounting for the possibility that not all the dark
matter decays, that it may decay into particles other than photons, and that
not all the photons may reach us; hereinafter we assume this fraction to be
unity, but our results can be reinterpreted by linearly rescaling
$\Gamma_{\chi}$ to account for any combination of these scenarios. We have
also neglected the contribution from stimulated decays due to background light
Caputo:2018ljp ; Caputo:2018vmy , since it does not affect the result for the
energies considered in this work Bernal_limdm . We transform the luminosity
density into flux (volume) density dividing by $4\pi D_{L}^{2}$, where $D_{L}$
is the luminosity distance, and convert it to specific intensity by
transforming the comoving volume element to solid angle and observed
wavelength elements as ${\rm d}A/{\rm d}\Omega=D_{M}^{2}$ and ${\rm
d}\chi/{\rm d}\lambda_{\rm obs}=c(1+z)/H\lambda_{\rm obs}$, respectively,
where $D_{M}$ and $\chi$ are the comoving angular diameter distance and the
comoving radial distance, respectively.

Figure 1: Specific intensity per observed wavelength from dark matter
radiative decays (left panels), as dark matter mass varies (color coded
following the colorbar). We show predictions for a fixed decay rate of
$8\times 10^{-24}$ s-1 with the corresponding contributions to measured
intensities along with the COB excess (top panels), and for a fixed
contribution to the measured intensity equal to the measured excess, with the
corresponding required decay rate (bottom panels). We also indicate in the
colorbar the redshift at which the decay occurs at the LORRI pivot wavelength
$\lambda_{\rm obs}=0.608\,\mu$m for each mass. The gray shaded region
corresponds to the unnormalized shape of the LORRI responsivity as function of
wavelength.

We show the specific intensity per observed wavelength for different dark
matter masses over the interval of interest in Fig. 1. The hard cut off at the
low wavelength end of each curve corresponds to the rest-frame wavelength of
the products for each dark matter mass, i.e., to the radiation at $z=0$. For a
fixed decay rate, the flux decays with the dark matter mass, which also
implies that it decays as the radiation comes from higher redshift at the same
$\lambda_{\rm obs}$.This is due to the fact that the flux density decreases
with distance: accounting for all redshift-dependent factors,
$I_{\lambda}\propto(1+z)^{-5/2}$ (roughly approximating
$H\propto(1+z)^{3/2}$).

LORRI is an unfiltered CCD imager with sensitivity from the blue to the near
infrared — $0.4\,\mu{\rm m}\lesssim\lambda_{\rm obs}\lesssim 0.9\,\mu{\rm m}$
— with pivot wavelength $\lambda_{\rm piv}=0.608\,\mu{\rm m}$, which is
operated with $4\times 4$ pixel binning for deep observations Cheng_LORRI ;
LORRI . We use Eq. (1) and the specifications of the LORRI camera in its
$4\times 4$ operating mode to obtain the predicted contribution from dark
matter decays to the COB measurement. We compute the data number DN per second
and pixel using the responsivity $R$ of LORRI LORRI  as

${\rm DN}/{\rm pix}/{\rm s}=\int{\rm d}\lambda_{\rm obs}R(\lambda_{\rm
obs})I_{\lambda}\,,$ (2)

which we convert into measured intensity at $\lambda_{\rm piv}$ using the
sensitivity corresponding to the diffuse photometry keyword for targets with
Solar-like spectral energy distributions (RSOLAR in Table 2 of Ref. LORRI ,
transformed to the correct units):

$I=\frac{\lambda_{\rm piv}}{{\rm RSOLAR}}\times\left({\rm DN}/{\rm pix}/{\rm
s}\right)\,.$ (3)

We show the predicted contributions to the measured flux by LORRI in the right
panel of Fig. 1, along with the COB excess measured, which could be explained
by the correct combination of dark matter mass and decay rate. In the bottom
panels we show the specific intensities for each mass normalized to yield the
correct contribution to the measured flux to explain the COB excess,
indicating the corresponding decay rate in the right panel.

Although we have modeled this decay in terms of the decay rate without loss of
generality, we can frame it in the context of axions to compare the required
dark matter properties with existing constraints in the mass range of
interest. We calculate the required decay rates to explain the COB excess as
function of mass from Eq. (3). The required decay rates at 95% confidence
level are shown in Fig. 2, along with existing constraints and forecasted
sensitivities. We include the most competitive existing bounds on this mass
range, namely the spectroscopic observations of the dwarf spheroidal galaxy
Leo T with MUSE Regis:2020fhw  and optical searches using VIMOS spectra for
line emission in the galaxy clusters Abell 2667 and 2390 Grin:2006aw , and
limits derived from the study of the cooling of horizontal branch stars in
globular clusters Ayala:2014pea . We also include the tentative 68% confidence
level favored regions on the axion parameter space from the study of the
blazar 1ES 1218+304 $\gamma$-ray spectrum Korochkin_axion , which requires a
bump in the extragalactic background light (EBL) to fit the attenuation of the
spectrum at high energies Korochkin_bump . For $m_{\chi}c^{2}\gtrsim 8$ eV,
the required decay rates are currently unconstrained, and interestingly
overlap with the favored region from Ref. Korochkin_axion . Thus, it is
possible that the COB excess is indeed produced by dark matter decays.
Additional, independent observations will be required to confirm or reject
this candidate.

Figure 2: Comparison between the required decay rates to explain the cosmic
optical background and existing bounds in the context of axion decay into two
photons from spectroscopic searches and the cooling of horizontal branch stars
in globular clusters, as well as forecasted sensitivities for optical LIM
experiments (solid and dotted lines show the forecasts using the LIM power
spectrum and voxel intensity distribution, respectively). All values
correspond to 95% confidence level, except for the 68% confidence level
favored regions from the analysis of $\gamma$-ray attenuation.

Radiative decays of axion-like particles have been considered to explain the
anomalous excesses in the power spectrum of the near-infrared background
measured by HST HST_NIRPS , CIBER Zemcov_NIRPS  and Spitzer Cooray_2012 ;
Kashlinsky_spitzer . The predicted axion parameters that provide a good fit to
the measurements Gong_IR ; Caputo_IR  roughly coincide with those that would
explain the COB excess in the low-mass end of the mass range considered in
this work; however, this part of the parameter space is ruled out by
independent probes Regis:2020fhw ; Ayala:2014pea .

The region of the parameter space of axion properties that could explain the
COB excess has proved difficult to explore with current strategies. The
specific intensities for some of the masses required to explain the COB excess
also predict contributions in the ultraviolet (UV) range (see Fig. 1). New
Horizons also includes a UV spectrograph that, if pointed to dark fields as
the LORRI camera was, could measure the cosmic UV background and shed light on
whether the COB excess is produced by dark matter decays. In addition, the
results from Ref. Korochkin_axion  show promise and warrant a more detailed
study of dark matter radiative decay constraints from $\gamma$-ray
attenuation.

Here, however, we emphasize the prospects to test this dark-matter-decay
scenario with forthcoming LIM experiments. The principle underlying LIM
searches for radiative dark matter decays is similar to that for a background
excess, with the exception of the spectral resolution embedded in LIM
experiments.111Since LIM experiments target spectral lines that can be well
distinguished from the continuum, their sensitivity lowers as the emission
line or the dark matter mass distribution widens. In this case, the emission
line produced via dark-matter decay redshifts into the telescope spectral band
from a different redshift than the targeted astrophysical lines, acting as
what is known as a line interloper. The dark matter line can therefore be
detected using standard data analysis techniques (see e.g., Refs.
Lidz_interlopers ; Cheng_foregroundsAP ; Cheng_deconfInterlopers ;
Gong_interlopers ). The most prominent features decaying dark matter imparts
on LIM observables are an increase in the power spectrum anisotropy due to
projection effects and a voxel intensity distribution that is narrower and
shifted towards higher temperatures. In particular, LIM experiments in the
optical such as HETDEX HETDEX  and SPHEREx SPHEREx  will improve current
sensitivities by several orders of magnitude, being able to detect the
contributions from dark matter decays required to explain the COB excess with
very high significance. We show the forecasted 95% combined sensitivities
Bernal_limdm  from LIM power spectrum (solid) and voxel intensity distribution
(dotted) measurements by these experiments; the measurements of these two
observables can be combined to further improve these sensitivities
COMAP:2018kem ; SatoPolito_covariance . The dark matter line can also be
isolated by cross-correlating the line intensity maps with galaxy surveys
Creque-Sarbinowski:2018ebl ; Shirasaki_limlensdm .

Current measurements of blazar high-energy spectrum attenuation via photon-
photon electron-positron pair production HESS ; Ahnen16 ; Fermilat  can be
used to reconstruct the EBL at ultraviolet, optical and infrared wavelengths
as function of redshift. When a tighter parametrization of this flux is
assumed, the reconstructed background emission Ahnen16 ; Fermilat ; Desai19
agrees with the predicted integrated galaxy light from galaxy counts NH21 ;
Driver_IGL ; SaldanaLopez_IGL . The uncertainties, once more freedom is given
to the shape of the reconstructed specific intensity of the background
radiation HESS_background ; Acciari19 , are large enough to agree with
predictions from number counts and the measured COB excess. As discussed in
Ref. Korochkin_axion , a higher COB from high-redshift sources will contribute
to the $\gamma$-ray attenuation, which can be used to look for axion decays.
Preliminary results from the study of the blazar 1ES 1218+304 show preference
for a bump in the EBL that may be produced by axions with properties that
would also explain the measured COB excess. This hint, in addition to the
promising prospects of forthcoming high-energy observations, warrants further
investigation axion_gray_constraint .

We have considered a very specific case for dark matter sourced radiation, but
our results can be easily interpreted for the case of dark matter decaying
into a photon and another massive daughter particle by rescaling the mass of
the dark matter and its decay rate as discussed in Section III.A of Ref.
Bernal_limdm . Furthermore, photons can be produced from particle cascades of
other decay products, but in that case the mapping between their energy and
the dark matter mass is less direct. This scenario can be accounted for by
considering a given dark matter model and modeling the final photon energy
distribution and branching ratio for each decay channel; hence, our results
and calculations can be extrapolated to each specific case. Similarly, dark
matter annihilation includes the dark matter velocity distribution and boost
factor, which adds uncertainty to the computation. Nonetheless, only fermionic
dark matter can involve efficient annihilation, and dark matter with masses
below $\sim 100$ eV can only be bosonic due to the Pauli exclusion principle.
Thus, if we consider the dark matter is a single species, it cannot annihilate
producing photons in the frequency band of interest. We leave the study of a
small fraction of light fermionic annihilating dark matter for future work.
Another dark-matter related potential explanation for the COB excess involves
dark-matter powered stars Maurer_DS .

The results of this work provide a potential explanation for the COB excess
that is allowed by independent observational constraints and that may answer
one of the most long-standing unknown in cosmology: the nature of dark matter.
We provide additional motivation for follow up studies with forthcoming
observations, especially with optical LIM surveys such as HETDEX or SPHEREx,
with the UV instrument aboard the New Horizons spacecraft, and very high-
energy $\gamma$-ray attenuation, that will confirm or reject this hypothesis.

Acknowledgments — The authors thank Marc Postman for help with the LORRI
responsivity and useful discussions. JLB is supported by the Allan C. and
Dorothy H. Davis Fellowship. GSP was supported by the National Science
Foundation Graduate Research Fellowship under Grant No. DGE1746891. MK was
supported by NSF Grant No. 1818899 and the Simons Foundation.

## References

  * (1) A. Cooray, “Extragalactic background light measurements and applications,” Royal Society Open Science 3 no. 3, (Mar., 2016) 150555, arXiv:1602.03512 [astro-ph.CO].
  * (2) K. Mattila, P. Väisänen, K. Lehtinen, G. von Appen-Schnur, and C. Leinert, “Extragalactic background light: a measurement at 400 nm using dark cloud shadow - II. Spectroscopic separation of the dark cloud’s light, and results⋆,” Mon. Not. R. Astron. Soc. 470 no. 2, (Sept., 2017) 2152–2169, arXiv:1705.10790 [astro-ph.GA].
  * (3) S. Matsuura, T. Arai, J. J. Bock, A. Cooray, P. M. Korngut, M. G. Kim, et al., “New Spectral Evidence of an Unaccounted Component of the Near-infrared Extragalactic Background Light from the CIBER,” Astrophys. J.  839 no. 1, (Apr., 2017) 7, arXiv:1704.07166 [astro-ph.GA].
  * (4) Y. Matsuoka, N. Ienaka, K. Kawara, and S. Oyabu, “Cosmic Optical Background: The View from Pioneer 10/11,” Astrophys. J.  736 no. 2, (Aug., 2011) 119, arXiv:1106.4413 [astro-ph.CO].
  * (5) T. Matsumoto, K. Tsumura, Y. Matsuoka, and J. Pyo, “Zodiacal Light Beyond Earth Orbit Observed with Pioneer 10,” Astron. J. 156 no. 3, (Sept., 2018) 86, arXiv:1808.03759 [astro-ph.EP].
  * (6) M. Zemcov, I. Arcavi, R. Arendt, E. Bachelet, R. Ram Chary, A. Cooray, et al., “Astrophysics with New Horizons: Making the Most of a Generational Opportunity,” Publ. Astron. Soc. Pacific 130 no. 993, (Nov., 2018) 115001, arXiv:1802.09536 [astro-ph.IM].
  * (7) A. R. Poppe, C. M. Lisse, M. Piquette, M. Zemcov, M. Horányi, D. James, et al., “Constraining the Solar System’s Debris Disk with In Situ New Horizons Measurements from the Edgeworth-Kuiper Belt,” Astrophys. J. Lett. 881 no. 1, (Aug., 2019) L12.
  * (8) M. Zemcov, P. Immel, C. Nguyen, A. Cooray, C. M. Lisse, and A. R. Poppe, “Measurement of the cosmic optical background using the long range reconnaissance imager on New Horizons,” Nature Communications 8 (Apr., 2017) 15003, arXiv:1704.02989 [astro-ph.IM].
  * (9) T. R. Lauer, M. Postman, H. A. Weaver, J. R. Spencer, S. A. Stern, M. W. Buie, et al., “New Horizons Observations of the Cosmic Optical Background,” Astrophys. J.  906 no. 2, (Jan., 2021) 77, arXiv:2011.03052 [astro-ph.GA].
  * (10) T. R. Lauer, M. Postman, J. R. Spencer, H. A. Weaver, S. A. Stern, G. R. Gladstone, et al., “Anomalous Flux in the Cosmic Optical Background Detected With New Horizons Observations,” arXiv e-prints (Feb., 2022) arXiv:2202.04273, arXiv:2202.04273 [astro-ph.GA].
  * (11) C. J. Conselice, A. Wilkinson, K. Duncan, and A. Mortlock, “The Evolution of Galaxy Number Density at z ¡ 8 and Its Implications,” Astrophys. J.  830 no. 2, (Oct., 2016) 83, arXiv:1607.03909 [astro-ph.GA].
  * (12) A. Cooray, J. Smidt, F. de Bernardis, Y. Gong, D. Stern, M. L. N. Ashby, et al., “Near-infrared background anisotropies from diffuse intrahalo light of galaxies,” Nature (London) 490 no. 7421, (Oct., 2012) 514–516.
  * (13) M. Zemcov, J. Smidt, T. Arai, J. Bock, A. Cooray, Y. Gong, et al., “On the origin of near-infrared extragalactic background light anisotropy,” Science 346 no. 6210, (Nov., 2014) 732–735, arXiv:1411.1411 [astro-ph.CO].
  * (14) T. Matsumoto and K. Tsumura, “Fluctuation of the background sky in the Hubble Extremely Deep Field (XDF) and its origin,” Publ. Astron. Soc. Japan 71 no. 5, (Oct., 2019) 88, arXiv:1906.01443 [astro-ph.GA].
  * (15) B. Yue, A. Ferrara, R. Salvaterra, Y. Xu, and X. Chen, “Infrared background signatures of the first black holes,” Mon. Not. R. Astron. Soc. 433 no. 2, (Aug., 2013) 1556–1566, arXiv:1305.5177 [astro-ph.CO].
  * (16) L. Abbott and P. Sikivie, “A Cosmological Bound on the Invisible Axion,” Phys. Lett. B 120 (1983) 133–136.
  * (17) M. Dine and W. Fischler, “The Not So Harmless Axion,” Phys. Lett. B 120 (1983) 137–141.
  * (18) J. Preskill, M. B. Wise, and F. Wilczek, “Cosmology of the Invisible Axion,” Phys. Lett. B120 (1983) 127–132. [,URL(1982)].
  * (19) S. Weinberg, “A New Light Boson?,” Phys. Rev. Lett. 40 (1978) 223–226.
  * (20) F. Wilczek, “Problem of Strong p and t Invariance in the Presence of Instantons,” Phys. Rev. Lett. 40 (1978) 279–282.
  * (21) R. D. Peccei and H. R. Quinn, “CP Conservation in the Presence of Instantons,” Phys. Rev. Lett. 38 (1977) 1440–1443. [,328(1977)].
  * (22) R. D. Peccei and H. R. Quinn, “Constraints Imposed by CP Conservation in the Presence of Instantons,” Phys. Rev. D16 (1977) 1791–1797.
  * (23) A. Kusenko, “Sterile neutrinos: The dark side of the light fermions,” Sep, 2009\.  https://ui.adsabs.harvard.edu/abs/2009PhR...481....1K/abstract.
  * (24) A. Caputo, C. P. Garay, and S. J. Witte, “Looking for Axion Dark Matter in Dwarf Spheroidals,” Phys. Rev. D 98 no. 8, (2018) 083024, arXiv:1805.08780 [astro-ph.CO]. [Erratum: Phys. Rev.D99,no.8,089901(2019)].
  * (25) A. Caputo, M. Regis, M. Taoso, and S. J. Witte, “Detecting the Stimulated Decay of Axions at RadioFrequencies,” JCAP 1903 no. 03, (2019) 027, arXiv:1811.08436 [hep-ph].
  * (26) Y. Gong, A. Cooray, K. Mitchell-Wynne, X. Chen, M. Zemcov, and J. Smidt, “Axion Decay and Anisotropy of Near-IR Extragalactic Background Light,” Astrophys. J.  825 no. 2, (July, 2016) 104, arXiv:1511.01577 [astro-ph.CO].
  * (27) A. Caputo, A. Vittino, N. Fornengo, M. Regis, and M. Taoso, “Searching for axion-like particle decay in the near-infrared background: an updated analysis,” JCAP 2021 no. 5, (May, 2021) 046, arXiv:2012.09179 [astro-ph.CO].
  * (28) A. Caputo, M. Regis, and M. Taoso, “Searching for Sterile Neutrino with X-ray Intensity Mapping,” JCAP 03 (2020) 001, arXiv:1911.09120 [astro-ph.CO].
  * (29) A. Boyarsky, J. Franse, D. Iakubovskyi, and O. Ruchayskiy, “Checking the Dark Matter Origin of a 3.53 keV Line with the Milky Way Center,” Phys. Rev. Lett. 115 (2015) 161301, arXiv:1408.2503 [astro-ph.CO].
  * (30) S. Riemer-Sørensen, “Constraints on the presence of a 3.5 keV dark matter emission line from Chandra observations of the Galactic centre,” Astron. Astrophys. 590 (2016) A71, arXiv:1405.7943 [astro-ph.CO].
  * (31) C. Dessert, N. L. Rodd, and B. R. Safdi, “The dark matter interpretation of the 3.5-keV line is inconsistent with blank-sky observations,” Science 367 (2020) 1465, arXiv:1812.06976 [astro-ph.CO].
  * (32) C. Blanco and D. Hooper, “Constraints on decaying dark matter from the isotropic gamma-ray background,” JCAP 2019 no. 3, (Mar., 2019) 019, arXiv:1811.05988 [astro-ph.HE].
  * (33) T. Cohen, K. Murase, N. L. Rodd, B. R. Safdi, and Y. Soreq, “$\gamma$ -ray Constraints on Decaying Dark Matter and Implications for IceCube,” Phys. Rev. Lett. 119 no. 2, (July, 2017) 021102, arXiv:1612.05638 [hep-ph].
  * (34) J. Ellis, J. E. Kim, and D. V. Nanopoulos, “Cosmological gravitino regeneration and decay,” Physics Letters B 145 no. 3-4, (Sept., 1984) 181–186.
  * (35) W. Hu and J. Silk, “Thermalization constraints and spectral distortions for massive unstable relic particles,” Phys. Rev. Lett. 70 no. 18, (May, 1993) 2661–2664.
  * (36) J. Chluba and R. A. Sunyaev, “The evolution of CMB spectral distortions in the early Universe,” Mon. Not. R. Astron. Soc. 419 no. 2, (Jan., 2012) 1294–1314, arXiv:1109.6552 [astro-ph.CO].
  * (37) F. Iocco, G. Mangano, G. Miele, O. Pisanti, and P. D. Serpico, “Primordial nucleosynthesis: From precision cosmology to fundamental physics,” Physics Reports 472 no. 1-6, (Mar., 2009) 1–76, arXiv:0809.0631 [astro-ph].
  * (38) M. Pospelov and J. Pradler, “Big Bang Nucleosynthesis as a Probe of New Physics,” Annual Review of Nuclear and Particle Science 60 (Nov., 2010) 539–568, arXiv:1011.1054 [hep-ph].
  * (39) V. Poulin and P. D. Serpico, “Nonuniversal BBN bounds on electromagnetically decaying particles,” Phys. Rev. D 91 no. 10, (May, 2015) 103007, arXiv:1503.04852 [astro-ph.CO].
  * (40) V. Poulin, P. D. Serpico, and J. Lesgourgues, “A fresh look at linear cosmological constraints on a decaying Dark Matter component,” JCAP 2016 no. 8, (Aug., 2016) 036, arXiv:1606.02073 [astro-ph.CO].
  * (41) T. R. Slatyer, “Indirect dark matter signatures in the cosmic dark ages. II. Ionization, heating, and photon production from arbitrary energy injections,” Phys. Rev. D 93 no. 2, (Jan., 2016) 023521, arXiv:1506.03812 [astro-ph.CO].
  * (42) T. R. Slatyer and C.-L. Wu, “General constraints on dark matter decay from the cosmic microwave background,” Phys. Rev. D 95 no. 2, (Jan., 2017) 023010, arXiv:1610.06933 [astro-ph.CO].
  * (43) M. Lucca, N. Schöneberg, D. C. Hooper, J. Lesgourgues, and J. Chluba, “The synergy between CMB spectral distortions and anisotropies,” JCAP 2020 no. 2, (Feb., 2020) 026, arXiv:1910.04619 [astro-ph.CO].
  * (44) O. Doré, J. Bock, M. Ashby, P. Capak, A. Cooray, et al., “Cosmology with the SPHEREX All-Sky Spectral Survey,” arXiv e-prints (Dec, 2014) arXiv:1412.4872, arXiv:1412.4872 [astro-ph.CO].
  * (45) G. J. Hill, K. Gebhardt, E. Komatsu, N. Drory, P. J. MacQueen, et al., “The Hobby-Eberly Telescope Dark Energy Experiment (HETDEX): Description and Early Pilot Survey Results,” in Panoramic Views of Galaxy Formation and Evolution, T. Kodama, T. Yamada, and K. Aoki, eds., vol. 399 of Astronomical Society of the Pacific Conference Series, p. 115.  Oct., 2008.  arXiv:0806.0183 [astro-ph].
  * (46) E. D. Kovetz et al., “Line-Intensity Mapping: 2017 Status Report,” arXiv:1709.09066 [astro-ph.CO].
  * (47) J. L. Bernal, P. C. Breysse, H. Gil-Marín, and E. D. Kovetz, “User’s guide to extracting cosmological information from line-intensity maps,” Phys. Rev. D 100 no. 12, (2019) 123522, arXiv:1907.10067 [astro-ph.CO].
  * (48) C. Creque-Sarbinowski and M. Kamionkowski, “Searching for Decaying and Annihilating Dark Matter with Line Intensity Mapping,” Phys. Rev. D 98 no. 6, (2018) 063524, arXiv:1806.11119 [astro-ph.CO].
  * (49) J. L. Bernal, A. Caputo, and M. Kamionkowski, “Strategies to detect dark-matter decays with line-intensity mapping,” Phys. Rev. D 103 no. 6, (Mar., 2021) 063523, arXiv:2012.00771 [astro-ph.CO].
  * (50) X.-L. Chen and M. Kamionkowski, “Particle decays during the cosmic dark ages,” Phys. Rev. D 70 (2004) 043502, arXiv:astro-ph/0310473.
  * (51) D. P. Finkbeiner, S. Galli, T. Lin, and T. R. Slatyer, “Searching for Dark Matter in the CMB: A Compact Parameterization of Energy Injection from New Physics,” Phys. Rev. D 85 (2012) 043522, arXiv:1109.6322 [astro-ph.CO].
  * (52) E. Pierpaoli, “Decaying particles and the reionization history of the universe,” Phys. Rev. Lett. 92 (2004) 031301, arXiv:astro-ph/0310375.
  * (53) A. F. Cheng, H. A. Weaver, S. J. Conard, M. F. Morgan, O. Barnouin-Jha, J. D. Boldt, et al., “Long-Range Reconnaissance Imager on New Horizons,” Space Science Rev. 140 no. 1-4, (Oct., 2008) 189–215, arXiv:0709.4278 [astro-ph].
  * (54) H. A. Weaver, A. F. Cheng, F. Morgan, H. W. Taylor, S. J. Conard, J. I. Nunez, et al., “In-flight Performance and Calibration of the LOng Range Reconnaissance Imager (LORRI) for the New Horizons Mission,” Publ. Astron. Soc. Pacific 132 no. 1009, (Mar., 2020) 035003, arXiv:2001.03524 [astro-ph.IM].
  * (55) M. Regis, M. Taoso, D. Vaz, J. Brinchmann, S. L. Zoutendijk, N. Bouché, and M. Steinmetz, “Searching for Light in the Darkness: Bounds on ALP Dark Matter with the optical MUSE-Faint survey,” arXiv:2009.01310 [astro-ph.CO].
  * (56) D. Grin, G. Covone, J.-P. Kneib, M. Kamionkowski, A. Blain, and E. Jullo, “A Telescope Search for Decaying Relic Axions,” Phys. Rev. D 75 (2007) 105018, arXiv:astro-ph/0611502.
  * (57) A. Ayala, I. Domínguez, M. Giannotti, A. Mirizzi, and O. Straniero, “Revisiting the bound on axion-photon coupling from Globular Clusters,” Phys. Rev. Lett. 113 no. 19, (2014) 191302, arXiv:1406.6053 [astro-ph.SR].
  * (58) A. Korochkin, A. Neronov, and D. Semikoz, “Search for decaying eV-mass axion-like particles using gamma-ray signal from blazars,” JCAP 2020 no. 3, (Mar., 2020) 064, arXiv:1911.13291 [hep-ph].
  * (59) A. Korochkin, A. Neronov, and D. Semikoz, “Search for spectral features in extragalactic background light with gamma-ray telescopes,” Astron. Astrophys. 633 (Jan., 2020) A74, arXiv:1906.12168 [astro-ph.HE].
  * (60) K. Mitchell-Wynne, A. Cooray, Y. Gong, M. Ashby, T. Dolch, H. Ferguson, et al., “Ultraviolet luminosity density of the universe during the epoch of reionization,” Nature Communications 6 (Sept., 2015) 7945, arXiv:1509.02935 [astro-ph.CO].
  * (61) M. Zemcov, J. Smidt, T. Arai, J. Bock, A. Cooray, Y. Gong, et al., “On the origin of near-infrared extragalactic background light anisotropy,” Science 346 no. 6210, (Nov., 2014) 732–735, arXiv:1411.1411 [astro-ph.CO].
  * (62) A. Kashlinsky, R. G. Arendt, M. L. N. Ashby, G. G. Fazio, J. Mather, and S. H. Moseley, “New Measurements of the Cosmic Infrared Background Fluctuations in Deep Spitzer/IRAC Survey Data and Their Cosmological Implications,” Astrophys. J.  753 no. 1, (July, 2012) 63, arXiv:1201.5617 [astro-ph.CO].
  * (63) A. Lidz and J. Taylor, “On Removing Interloper Contamination from Intensity Mapping Power Spectrum Measurements,” Astrophys. J.  825 no. 2, (July, 2016) 143, arXiv:1604.05737 [astro-ph.CO].
  * (64) Y.-T. Cheng, T.-C. Chang, J. Bock, C. M. Bradford, and A. Cooray, “Spectral Line De-confusion in an Intensity Mapping Survey,” Astrophys. J.  832 no. 2, (Dec, 2016) 165, arXiv:1604.07833 [astro-ph.CO].
  * (65) Y.-T. Cheng, T.-C. Chang, and J. J. Bock, “Phase-space Spectral Line Deconfusion in Intensity Mapping,” Astrophys. J.  901 no. 2, (Oct., 2020) 142, arXiv:2005.05341 [astro-ph.CO].
  * (66) Y. Gong, X. Chen, and A. Cooray, “Cosmological Constraints from Line Intensity Mapping with Interlopers,” Astrophys. J.  894 no. 2, (May, 2020) 152, arXiv:2001.10792 [astro-ph.CO].
  * (67) COMAP Collaboration, H. T. Ihle et al., “Joint power spectrum and voxel intensity distribution forecast on the CO luminosity function with COMAP,” Astrophys. J. 871 no. 1, (2019) 75, arXiv:1808.07487 [astro-ph.CO].
  * (68) G. Sato-Polito and J. L. Bernal, “Analytical covariance between voxel intensity distributions and line-intensity mapping power spectra,” arXiv:2202.02330 [astro-ph.CO].
  * (69) M. Shirasaki, “Searching for eV-mass axionlike particles with cross correlations between line intensity and weak lensing maps,” Phys. Rev. D 103 no. 10, (May, 2021) 103014, arXiv:2102.00580 [astro-ph.CO].
  * (70) H. E. S. S. Collaboration, A. Abramowski, F. Acero, F. Aharonian, A. G. Akhperjanian, G. Anton, et al., “Measurement of the extragalactic background light imprint on the spectra of the brightest blazars observed with H.E.S.S.,” Astron. Astrophys. 550 (Feb., 2013) A4, arXiv:1212.3409 [astro-ph.HE].
  * (71) M. L. Ahnen, S. Ansoldi, L. A. Antonelli, P. Antoranz, A. Babic, B. Banerjee, et al., “MAGIC observations of the February 2014 flare of 1ES 1011+496 and ensuing constraint of the EBL density,” Astron. Astrophys. 590 (May, 2016) A24, arXiv:1602.05239 [astro-ph.HE].
  * (72) Fermi-LAT Collaboration, S. Abdollahi, M. Ackermann, M. Ajello, W. B. Atwood, L. Baldini, et al., “A gamma-ray determination of the Universe’s star formation history,” Science 362 no. 6418, (Nov., 2018) 1031–1034, arXiv:1812.01031 [astro-ph.HE].
  * (73) A. Desai, K. Helgason, M. Ajello, V. Paliya, A. Domínguez, J. Finke, and D. Hartmann, “A GeV-TeV Measurement of the Extragalactic Background Light,” Astrophys. J. Lett. 874 no. 1, (Mar., 2019) L7, arXiv:1903.03126 [astro-ph.HE].
  * (74) S. P. Driver, S. K. Andrews, L. J. Davies, A. S. G. Robotham, A. H. Wright, R. A. Windhorst, et al., “Measurements of Extragalactic Background Light from the Far UV to the Far IR from Deep Ground- and Space-based Galaxy Counts,” Astrophys. J.  827 no. 2, (Aug., 2016) 108, arXiv:1605.01523 [astro-ph.GA].
  * (75) A. Saldana-Lopez, A. Domínguez, P. G. Pérez-González, J. Finke, M. Ajello, J. R. Primack, et al., “An observational determination of the evolving extragalactic background light from the multiwavelength HST/CANDELS survey in the Fermi and CTA era,” Mon. Not. R. Astron. Soc. 507 no. 4, (Nov., 2021) 5144–5160, arXiv:2012.03035 [astro-ph.CO].
  * (76) H. E. S. S. Collaboration, H. Abdalla, A. Abramowski, F. Aharonian, F. Ait Benkhali, A. G. Akhperjanian, et al., “Measurement of the EBL spectral energy distribution using the VHE $\gamma$-ray spectra of H.E.S.S. blazars,” Astron. Astrophys. 606 (Oct., 2017) A59, arXiv:1707.06090 [astro-ph.HE].
  * (77) V. A. Acciari, S. Ansoldi, L. A. Antonelli, A. Arbet Engels, D. Baack, Babić, et al., “Measurement of the extragalactic background light using MAGIC and Fermi-LAT gamma-ray observations of blazars up to z = 1,” Mon. Not. R. Astron. Soc. 486 no. 3, (July, 2019) 4233–4251, arXiv:1904.00134 [astro-ph.HE].
  * (78) J. L. Bernal, A. Caputo, G. Sato-Polito, and M. Kamionkowski, “,” In prep. (2022) .
  * (79) A. Maurer, M. Raue, T. Kneiske, D. Horns, D. Elsässer, and P. H. Hauschildt, “Dark Matter Powered Stars: Constraints from the Extragalactic Background Light,” Astrophys. J.  745 no. 2, (Feb., 2012) 166, arXiv:1201.1305 [astro-ph.CO].


# NeRF-GAN Distillation for Efficient 3D-Aware Generation with Convolutions

Mohamad Shahbazi1 Evangelos Ntavelis1,3 Alessio Tonioni2 Edo Collins2 Danda
Pani Paudel1 Martin Danelljan1 Luc Van Gool1
1Computer Vision Lab, ETH Zürich 2Google Zürich 3ML & Robotics, CSEM,
Switzerland
{mshahbazi, entavelis, paudel, martin.danelljan<EMAIL_ADDRESS>{alessiot<EMAIL_ADDRESS>
###### Abstract

Pose-conditioned convolutional generative models struggle with high-quality
3D-consistent image generation from single-view datasets, due to their lack of
sufficient 3D priors. Recently, the integration of Neural Radiance Fields
(NeRFs) and generative models, such as Generative Adversarial Networks (GANs),
has transformed 3D-aware generation from single-view images. NeRF-GANs exploit
the strong inductive bias of neural 3D representations and volumetric
rendering at the cost of higher computational complexity. This study aims at
revisiting pose-conditioned 2D GANs for efficient 3D-aware generation at
inference time by distilling 3D knowledge from pretrained NeRF-GANs. We
propose a simple and effective method, based on re-using the well-disentangled
latent space of a pre-trained NeRF-GAN in a pose-conditioned convolutional
network to directly generate 3D-consistent images corresponding to the
underlying 3D representations. Experiments on several datasets demonstrate
that the proposed method obtains results comparable with volumetric rendering
in terms of quality and 3D consistency while benefiting from the computational
advantage of convolutional networks. The code will be available at:
https://github.com/mshahbazi72/NeRF-GAN-Distillation

##  1 Introduction

Figure 1: Top: views of the same subject cat generated by a volumetric
rendering generator (EG3D) and by our convolutional generator. Bottom:
comparison of the inference memory consumption and speed (on a fixed GPU
budget) for the two methods.

Generative Adversarial Networks (GANs) [15] have undergone outstanding
progress in photo-realistic image generation and manipulation in a variety of
applications [4, 27, 40, 22, 23, 9, 45, 44]. Recently, there has been an
increasing interest in extending GANs to the task of 3D-aware generation from
single-view image datasets, with the goal of providing disentangled control
over the content and the viewpoint of the generated images.

Image GAN models have been historically based on convolutional architectures,
enabling efficient training and generation for 2D tasks. However, pose-
conditioned convolutional GANs (pcGANs) struggle with 3D-consistent image
generation, due to their lack of sufficient 3D priors [35]. Therefore, some
studies have previously attempted to disentangle the pose from the content in
pcGANs using explicit 3D supervision [52, 28, 12], which, however, is not
readily available for most datasets. As a result, later methods moved away
from fully convolutional GANs by incorporating 3D inductive biases in the
architecture and training pipeline, such as 3D neural representations and
differentiable rendering methods [35, 36, 48, 39].

The advent of Neural Radiance Fields (NeRFs) [31] has recently transformed the
neural 3D representation and the task of novel-view synthesis [2, 3, 43, 32,
30, 17, 56]. For this reason, NeRFs have been successfully integrated with
GANs to achieve promising results in 3D-aware generation [46, 6, 38, 16, 50,
37, 47]. Nerf-GANs, in their general form, map a latent space to a 3D
representation of objects and generate images from queried viewpoints using
volumetric rendering. However, volumetric rendering is computationally
demanding due to its ray-casting process, making high-resolution generation
slow and memory-expensive. Recent works have proposed different approaches to
improve the computational efficiency of NeRF-GANs using more efficient 3D
representations [37, 6, 47] and training protocols [6, 38]. Nevertheless,
volumetric rendering remains an integral part of these models.

In recent NeRF-GANs [37, 6, 47, 38], convolutional networks have been
reintroduced in the generator architecture as super-resolution networks or as
3D-representation generators, in order to scale up NeRF-GANs for high-
resolution generation. In this study, we take a different approach to
integrating NeRF-GANs and convolutional GANs for 3D-aware generation from
single-view images. In particular, we investigate the capacity of
convolutional generators to achieve 3D-consistent rendering with explicit pose
control when learning from a pretrained NeRF-GAN without any additional
explicit 3D supervision. A convolutional generator that fairly preserves the
3D consistency, image quality, and the correspondence between the generated
images and the underlying 3D representation can be used for efficient multi-
view inference in setups where volumetric rendering is not affordable, such as
in mobile applications. However, balancing and minimizing the trade-off
between efficiency and 3D consistency is a highly challenging task, which we
set out to explore in this work.

We propose a simple but effective method for distilling a pretrained NeRF-GAN
into a pose-conditioned fully convolutional generator. The main component of
our approach is based on exploiting the well-disentangled intermediate latent
space of the NeRF-GAN in the convolutional generator. In particular, our
convolutional generator learns to map each latent code from the 3D generator,
along with the target viewpoint, to the corresponding obtained images by
explicit volumetric rendering. By doing so, we aim to distill the NeRF-GAN’s
underlying 3D knowledge into the convolutional generator, as well as to
establish a correspondence between the images of the generator and the 3D
representation of the NeRF-GAN. As demonstrated in Fig. 1, our experiments on
three different datasets indicate that the convolutional generator trained
with our method is capable of achieving results comparable to volumetric
rendering in terms of image quality and 3D-consistency, while benefiting from
the superior efficiency of convolutional networks.

Our contributions are summarized as follows:

  * •

We propose a method to distill NeRF-GANs into convolutional generators for
efficient 3D-aware inference.

  * •

We provide a simple and effective method to condition the convolutional
generator on the well-disentangled intermediate latent space of the NeRF-GAN.

  * •

Through experiments on three different datasets, we show that the generator
trained by our distillation method well preserves the 3D consistency, image
quality, and semantics of the pretrained NeRF-GAN.

##  2 Related Works

3D-aware Generation from Single-View Images. Prior works have attempted to
create 3D awareness in 2D GANs using explicit 3D supervision, such as 3D
models [52, 12], pose and landmark annotations [49, 21], and synthetic data
[28]. In many applications, obtaining such 3D supervision is not practical. As
a result, later works aimed at unsupervised methods by introducing 3D
inductive biases in GANs, including 3D neural representations and
differentiable rendering [35, 39, 48, 36] These methods, although promising,
lag far behind 2D GANs in terms of image quality or struggle with high-
resolution generation due to the additional computational complexity.

NeRF-GANs. NeRFS have shown outstanding potential in compactly representing 3D
scenes for novel view synthesis. GRAF [46] and Pi-GAN [6] are the first works
to integrate NeRFs and GANs. While achieving highly consistent 3D-aware
generation, the computational restrictions of NeRF framework make these
methods impractical for high-resolution generation or environments with
constrained resources. In order to extend NeRF-GANs to higher resolutions,
convolutional super-resolution networks were used in later studies [37, 16,
38] at the expense of some multi-view inconsistencies. EpiGRAF [50], in
contrast, adopts an efficient multi-scale patch training protocol, but still
requires full high-resolution sampling rendering for inference, which makes it
comparatively more computationally demanding than competitors.

Other studies aim at bringing the recent advances in the efficiency of NeRFS
to NeRF-GANs. Although there exist numerous works on efficient 3D
representations [33, 43, 51, 8, 55, 54] and volumetric sampling [57, 14, 18,
34, 20] in NeRFs, only a subset of them [5, 47, 58] have been successfully
applied to NeRF-GANs. This is because they are mainly designed for the single-
scene setup, making their adaptation to the generative setup not trivial. The
use of sparse voxel grids in VoxGRAF [47] and multi-plane image
representations in [58] result in efficient and 3D-consistent generation while
compromising the image quality and 3D geometry. EG3D [5] proposes using tri-
planes to represent the geometry of the generated objects. Exploiting tri-
planes, coupled with carefully designed techniques to enforce 3D consistency,
allows EG3D to significantly improve both computational efficiency and image
quality. Live 3D Portrait [53] is a concurrent work based on EG3D that aims at
real-time one-shot reconstruction of faces by estimating the canonical tri-
planes of a pre-trained EG3D. However, Live 3D Portrait is computationally
limited by the underlying volumetric rendering of EG3D. Most similar to our
study, SURF-GAN [29] aims to discover directions for pose control in a
pretraind 2D GAN by generating multi-view images using a pretrained NeRF-GAN.
However, using NeRF-GANs only as multi-view supervision does not fully exploit
their underlying 3D knowledge. Moreover, the 2D generator obtained using this
method does not preserve any correspondence between the NeRF-GAN’s 3D
representations and the generated images. Different from SURF-GAN, we exploit
the intermediate latent space of retrained NeRF-GANs to distill 3D knowledge
into a 2D generator and establish correspondence between the convolutional
generator and the NeRF-GAN’s 3D representations.

##  3 Method

Figure 2: Using a student-teacher framework, we distill 3D consistency from a
frozen volumetric rendering based Nerf-GAN (top) to a 2D convolutional
renderer (bottom). A loss consisting of high- and low-resolution image
reconstruction and an adversarial component allows us to retain good image
quality and 3D consistency.

In this section, we first provide a brief overview of the formulation of NeRF-
GANs and then explain the proposed formulation in detail.

###  3.1 Preliminaries

The general formulation of NeRF-GANs consists of a 3D-representation generator
$G^{3D}(z)$, which maps a latent variable $z$ (usually drawn from a normal
distribution) to a 3D representation of an object. Then, in order to render an
image $I_{z,c}$ from the target viewpoint (camera parameters) $c\in R^{25}$,
volumetric rendering is applied to the generated 3D representation. We base
our method on EG3D [5], as it provides a strong trade-off in image quality, 3D
consistency, and efficiency, among recent NeRF-GANs.

EG3D represents 3D scenes using tri-planes, which are three axis-aligned
orthogonal feature planes, each with a size of $N\times N\times C$, where $N$
is spatial resolution and $C$ is the number of channels. To represent a 3D
position $x\in R^{3}$, $x$ is projected onto each of the three feature planes,
retrieving the corresponding feature vector $(F_{xy},F_{xz},F_{yz})$ via
bilinear interpolation, and aggregating the three feature vectors via
summation. To obtain the color and density at position $x$, a lightweight MLP
decodes the feature vector obtained for the queried position to a density and
color value.

The tri-plane generator $G^{3D}(z,c)$ in EG3D consists of a mapping network
$M^{3D}(z,c)$, which maps the input latent code and the target viewpoint to an
intermediate latent variable $w$, namely the style code. The style code then
is used to modulate a convolutional synthesis network $S^{3D}(w)$ to generate
the tri-planes $I^{3p}_{z,c}$. In order to render an image $I_{z,c}$ from the
target viewpoint $c$, hierarchical volumetric rendering is applied to the tri-
planes. Since volumetric rendering at high resolutions is computationally too
expensive, EG3D does so at a lower resolution and uses a convolutional super-
resolution network to obtain a final image. More specifically, the low-
resolution output of volumetric rendering in EG3D consists of a 32-channel
feature map $I^{f}_{z,c}$, the first three of which represent the low-
resolution RGB image $I^{LR}_{z,c}$), which is given as input to the super-
resolution network. EG3D is trained in an adversarial fashion with a
viewpoint-conditioned dual discriminator $D$ that ensures the photorealism of
the generated images from the target viewpoints, as well as the consistency
between high-resolution and low-resolution images.

###  3.2 Convolutional Rendering of Pretrained NeRF-GANs

The aim of the proposed method is to distill a pre-trained NeRF-GAN
$G^{3D}_{z,c}$ into a 2D image generator $G^{2D}_{z,c}$, such that
$G^{2D}_{z,c}$ directly predicts 3D-consistent multi-view images
$I^{\prime}_{z,c}$, corresponding to the volumetric renderings obtained by the
underlying 3D representation of $G^{3D}_{z,c}$. To this end, we propose to
exploit the well-disentangled style space of $G^{3D}$ to distill the
underlying 3D representation into $G^{2D}$. Sharing the style space $w$ of the
pre-trained 3D generator with the convolutional renderer is the first step
towards establishing a correspondence between the 3D representations and the
generated images. Secondly, it allows training the convolutional generator for
3D-consistent generation without the need for generating multiple views of the
same objects and enforcing multi-view consistency.

The overall architecture of EG3D and our convolutional generator is visualized
in 2. The convolutional generator is based on StyleGAN architecture [24, 25],
consisting of a mapping network, a low-resolution convolutional feature
prediction, and a convolutional super-resolution network. The mapping network
transforms the style code $w$ of $G^{3D}$, and the target viewpoint $c$ to the
style code $w^{\prime}$ of $G^{2D}$. Then, the low-resolution feature
predictor $S^{2D}$ estimates the EG3D features and low-resolution image
obtained by the volumetric rendering. Estimated features and images are then
mapped to the high-resolution outputs using the super-resolution network. In
our setup, the super-resolution network is initialized with EG3D’s super-
resolution network and is jointly optimized with the feature predictor
network.

###  3.3 Training

In order to train the proposed convolutional renderer, we use a teacher-
student framework, where the volumetric rendering is used to supervise
$G^{2D}$ on the viewpoint-conditioned mapping of $(z,c)$ to
$I^{\prime}_{z,c}$. A schematic representation of our training regime is
reported in Fig. 2. Specifically, for each training sample, we randomly sample
$z_{i}$ and $c_{i}$ and use the pretrained NeRF-GAN to obtain: the
corresponding style code $w_{i}$, the low-resolution image
$I^{LR}_{z_{i},c_{i}}$, feature maps $I^{f}_{z_{i},c_{i}}$ rendered by
volumetric rendering, as well as, the high-resolution image
$I^{HR}_{z_{i},c_{i}}$ generated by the super-resolution network. These
together form a training sample $i$ for the proposed convolutional renderer.

We provide $z$ and $c$ to $G^{2D}$ and compute a loss function composed of
three parts. We first add a reconstruction term $L^{LR}_{rec}$ between the
low-resolution outputs of the volumetric and convolutional renderers. A second
reconstruction loss $L^{HR}_{rec}$ is applied between the super-resolved
outputs of the two renderers. Lastly, we apply an adversarial term $L_{adv}$.

In the following, we drop the subscripts $z_{i},c_{i}$ to reduce the clutter
in notation. The low-resolution reconstruction loss $L^{LR}_{rec}$ consists of
a pixel-wise smooth L1 loss between the two feature maps, as well as a
perceptual loss between the generated and target low-resolution images,

$\begin{split}L^{LR}_{rec}=&\lambda^{LR}_{\text{Smooth}L1}*\text{Smooth}L1(I^{\prime
f},I^{f})+\\\ &\lambda^{LR}_{\text{perc}}*\text{PerceptualLoss}(I^{\prime
LR},I^{LR}).\end{split}$ (1)

Here, $\lambda^{LR}_{\text{Smooth}L1}$ and $\lambda^{LR}_{perc}$ are the
weights for the low-resolution smooth L1 and perceptual loss, respectively.

Similarly, the high-resolution reconstruction loss $L^{HR}_{rec}$is defined
as:

$\begin{split}L^{HR}_{rec}=&\lambda^{HR}_{\text{Smooth}L1}*\text{Smooth}L1(I^{\prime
f},I^{f})+\\\ &\lambda^{HR}_{\text{perc}}*\text{PerceptualLoss}(I^{\prime
HR},I^{HR}).\end{split}$ (2)

where $\lambda^{HR}_{rec}$ and $\lambda^{HR}_{perc}$ are the weights for the
high-resolution smooth L1 and perceptual loss, respectively.

The adversarial term $L_{adv}$ is similar to the one used in EG3D. We use the
same dual-discriminator architecture $D$ as in EG3D to ensure the realism of
the high-resolution images, their consistency with the low-resolution version,
and the compliance of the generated image with the queried viewpoints. The
total loss $L_{total}$ for training $G^{2D}$ is,

$L_{total}=L^{LR}_{rec}+L^{HR}_{rec}+\lambda_{adv}*L_{adv},$ (3)

where $\lambda_{adv}$ is the weight for the adversarial loss.

Two-stage training: in practice, empirical experiments show that training the
convolutional renderer using the full objective from the beginning will lead
to high-quality but 3D-inconsistent images. Therefore, we instead propose a
2-stage training curriculum. In the first stage, $G^{2D}$ is only optimized by
pure distillation of the volumetric rendering $G^{3D}$ using $L^{LR}_{rec}$
and $L^{HR}_{rec}$ until the renderer achieves reasonable generation quality.
Then, the adversarial loss $L_{adv}$ is added to the training to further
improve the performance. By applying this 2-stage curriculum, we are able to
counter the 3D inconsistency induced by the adversarial training.

Pose-correlated dataset bias: As shown in EG3D [5], adversarial training of
the convolutional network is prone to learning pose-correlated dataset biases,
such as more smiling in non-frontal viewpoints in FFHQ dataset [26], which in
turn results in 3D attribute inconsistencies. To mitigate such biases in FFHQ
dataset, we use both real images and the images rendered from EG3D as the real
examples shown to the discriminator. The proportion of the EG3D-rendered
images shown to $D$ is controlled by the hyper-parameter
$\alpha~{}(0\leq\alpha\leq 1)$, which is set to $0.5$ in our experiments. As
we will discuss in section 4.6, $\alpha$ can be used to control the trade-off
between image quality and 3D consistency.

##  4 Experiments

In this section, we first describe our experimental setup for the evaluation
of our method. Then, we compare the proposed method with baselines in terms of
visual quality, 3D consistency, and computational efficiency. Moreover, we
provide an ablation study and a discussion on the benefits and trade-offs of
the proposed method.

###  4.1 Datasets

Following EG3D [6], we evaluate our method on three different datasets:

Flickr-Faces-HQ (FFHQ) [26]: a collection of 70k high-quality images of real-
world human faces, as well as corresponding approximate camera extrinsics
estimated using an of-the-shell pose estimator.

AFHQ Cats: a sub-category of the Animal-Face-HQ (AFHQ) [10], consisting of
around 5k high-quality images of cat faces, as well as corresponding camera
extrinsics estimated using an of-the-shell pose estimator.

ShapeNet Cars: a category of ShapeNet [7] consisting of synthetic images of
cars rendered from different viewpoints, as well as the corresponding camera
extrinsics annotations.

###  4.2 Baselines

We consider EG3D [5] and The method proposed in SURF-GAN [29] as our main
baselines for this study. For a more complete evaluation, we also compare our
method to additional relevant baselines:

EG3D [5]: the NeRF-GAN used for distilling 3D knowledge in the convolutional
generator. EG3D serves as the upper bound for the 3D consistency of the
proposed method.

Pose-Conditioned StyleGAN (PC-GAN): a standard conditional 2D GAN, conditioned
on the pose annotations without any knowledge distillation.

SURF: Inspired by the proposed method in SURF-GAN [29], we create a baseline
called SURF, where multi-view images of EG3D are used to discover pose-control
in a 2D StyleGAN (pretrained on FFHQ) based on the proposed method in [29].
The details of the implementation can be found in the supplementary
material111As the official implementation of SURF-GAN does not include their
method for pose control in 2D GANs, we provide our best attempt in
implementing their method..

LiftGAN [48]: a method predating EG3D and SURF baselines, based on
differentiable rendering for distilling 2D GANs to train a 3D generator.

###  4.3 Implementation and Evaluation Details

We implement and evaluate the proposed generator using both StyleGAN2 (ST2)
[24] and StyleGAN3 (ST3) [25] architectures. For the pretrained NeRF-GAN, we
use the official models from EG3D [5] (for Shapenet Cars, we re-train the
model as the official model does not match the results reported by EG3D). We
train our experiments using a batch size of 16. The rendering resolution and
the final resolution are (128, 512) for FFHQ and AFHQ and (64, 128) for
Shapenet Cars. Both training and inference experiments were conducted using
NVidia RTX 3090 GPUs. In all experiments, we set all of the weights of
reconstruction loss terms ($\lambda^{LR}_{smooth_{l}1}$,
$\lambda^{LR}_{perc}$, $\lambda^{HR}_{perc}$) to the value 1 and the weight of
the adversarial loss ($\lambda_{adv}$) to the value 0.1.

###  4.4 Metrics

We evaluate our method quantitatively in terms of visual quality, and 3D
consistency.

Fréchet Inception Distance (FID) [19]: The most common metric to assess the
quality and diversity of generation.

Kernel Inception Distance (KID) [19]: An unbiased alternative to FID for
smaller datasets.

Pose Accuracy (PA): following previous works [5], we measure the ability of
the model in generating images of the query poses by calculating the mean
squared error (MSE) between the query poses and the pose of the generated
images, estimated using an of-the-shelf pose estimator [13].

Identity Preservation (ID): As a metric for 3D consistency, we measure the
degree of face identity preservation between different viewpoints with respect
to the canonical pose using ArcFace [11] cosine similarity for the FFHQ setup.

3D Landmark Consistency: As another 3D consistency metric, we measure the
change in facial landmarks between different viewpoints in FFHQ using MSE. The
3D landmarks are estimated using an off-the-shelf estimator [13].

###  4.5 Quantitative Comparison

In the following, we quantitatively compare the proposed method with the
baselines described in Sec. 4.2 in terms of inference efficiency, visual
quality, and 3D consistency.

####  4.5.1 Efficiency

The efficiency of fully-convolutional networks compared to the rendering-based
method is well-known. To better assess the practical benefit of the proposed
method, we provide a comparison of inference efficiency between EG3D. Fig 1
visualizes an example of the inference memory consumption and speed of the two
methods using different batch sizes on a fixed GPU budget (in this case, on
RTX 3090 GPU with 24G of memory). As shown, EG3D is restricted to small batch
sizes (a maximum of 14) due to its costly memory consumption, where our method
can scale up to a maximum of 96 samples per batch. As for speed, our
convolutional generator achieves better frame-per-second, especially when
using StyleGAN2 as its backbone.

####  4.5.2 Image Quality

To assess the trade-off brought about by our convolutional generator, we
evaluate the quality of the generated images. Table 1 shows the FID and KID
scores for our method and the baseliness on different datasets. Compared to
the PC-GAN and SURF baselines, our method constantly achieves higher quality.
This confirms that exploiting the style space of the pretrained NeRF-GAN
contributes to the ability of the convolutional renderer in pose-conditioned
generation. Although our method does not fully match the visual quality of
EG3D, it is still able to fairly maintain high image quality and significantly
reduce the compromise in the quality compared to the other convolutional
counterparts.

Table 1: Comparison of image quality on three datasets in terms of FID and KID
metrics. *The value is borrowed from [48].

Method | FFHQ | AFHQ | ShapeNET Cars
---|---|---|---
| FID $\downarrow$ | KID $\downarrow$ | FID $\downarrow$ | KID $\downarrow$ | FID $\downarrow$ | KID $\downarrow$
EG3D [5] | 5.0 | 0.0018 | 2.9 | 0.0003 | 3.5 | 0.0017
PC-GAN | 19.3 | 0.0085 | 4.5 | 0.0009 | 6.1 | 0.0018
LiftGAN [48] | 29.8* | - | - | - | - | -
SURF | 31.1 | 0.0153 | - | - | - | -
Ours (ST2) | 6.6 | 0.0019 | 3.8 | 0.0011 | 3.1 | 0.0013
Ours (ST3) | 6.8 | 0.0023 | 3.2 | 0.0007 | 3.1 | 0.0012

####  4.5.3 3D Consistency

While, unlike volumetric rendering, 2D convolutions do not guarantee 3D
consistency, we show that our approach achieves a good performance in this
regard. We assess the 3D consistency of generated images on FFHQ by measuring
the pose accuracy, 3D landmark consistency, and face identity preservation, as
discussed in Sec. 4.4, which are provided in Table 2. Based on the results,
Our method achieves comparable 3D consistency with EG3D, while PC-GAN and SURF
struggle. Note that the high values for identity preservation and 3D landmark
consistency in SURF are due to the limited pose variations, and hence
generating similar images regardless of the input pose, as reflected by the
pose accuracy (and the visual examples in Fig. 3).

Table 2: Comparison of 3D consistency metrics on FFHQ.

Method | Pose Acc. $\downarrow$ | 3D Landmark $\downarrow$ | ID $\uparrow$
---|---|---|---
EG3D [5] | 0.002 | 0.018 | 0.75
PC-GAN | 0.009 | 0.062 | 0.56
SURF | 0.044 | 0.014 | 0.86
Ours (ST2) | 0.002 | 0.023 | 0.75
Ours (ST3) | 0.002 | 0.022 | 0.75

Figure 3: Qualitative examples of variations in yaw and pitch for FFHQ.
Compared to the pose-conditioned GAN and SURF baseline, our proposed method
nearly matches the 3D consistency and image quality of volumetric rendering
(EG3D). Figure 4: Qualitative examples of variations in yaw and pitch for AFHQ
cats. In line with our quantitative experiments, the pose-conditioned
convolutional baseline (PC-GAN) fails to preserve the identity of the subject
under different poses. In contrast, our method exhibits similar preservation
of identity to the volume rendering approach (EG3D), despite the difference in
computational resources and time. Figure 5: Qualitative examples of different
camera poses in Shapenet Cars for three different car models.

###  4.6 Ablation

Ablation on loss functions: the proposed training objective in section 3.3
consists of different loss terms to ensure both image quality and consistency
with the output of volumetric rendering. In this section, we ablate the
importance of these components, by adding them one by one to form the final
objective of Eq. 3. Table 3 shows the FID scores for the following experiments
on the loss terms on AFHQ dataset:

  * •

LR: Only the low-resolution reconstruction loss. the super-resolution network
is frozen.

  * •

HR: Only the high-resolution reconstruction loss.

  * •

LR + HR Full reconstruction loss on low-resolution and high-resolution.

  * •

HR + Adv: reconstruction and adversarial losses on the high-resolution images.

  * •

Full (LR + HR + ADV) Full training objective, including the reconstruction and
adversarial terms.

As shown by the ablation study, the combination of the proposed loss terms
leads to the best FID scores.

Table 3: Ablation on different loss functions for training the convolutional
renderer on AFHQ Cats dataset.

Method | FID $\downarrow$
---|---
LR | 30.55
HR | 10.39
LR + HR | 9.1
HR + ADV | 6.58
Full (LR + HR + ADV) | 3.2

Table 4: The effect of mixing real images and EG3D-rendered images as real
examples for adversarial training, controlled by the parameter $\alpha$, on
FFHQ dataset.

Method | Ours (ST2) | Ours (ST3)
---|---|---
| FID $\downarrow$ | 3D Landmark $\downarrow$ | FID $\downarrow$ | 3D Landmark $\downarrow$
$\alpha=0$ | 5.5 | 0.027 | 5.7 | 0.027
$\alpha=0.5$ | 6.8 | 0.022 | 6.6 | 0.023

Single-stage v.s. two-stage training: As mentioned in 3.3, we find out that
single-stage training by jointly optimizing for both reconstruction and
adversarial losses results in subtle inconsistencies such as color shifts and
geometry warps, which can be mitigated using the proposed 2-stage training in
section 3. As the observed inconsistencies are difficult to capture using our
quantitative 3D consistency metrics, we provide a visual comparison between
the examples of single-stage and two-stage training on AFHQ in Fig. 6. To
better visualize the inconsistencies, we provide a video visualization and
comparison in the supplementary material.

Mitigating Pose-Attribute Correlation: In Table 4, we provide an ablation on
the parameter $\alpha$ introduced in section 3.3 for FFHQ dataset. As shown,
including EG3D-generated images ($\alpha=0.5$) improves the 3D consistency at
the cost of lower generation quality.

Figure 6: one-stage training causes subtle color and geometry inconsistencies
(first row). Such inconsistencies can be resolved using our proposed 2-stage
training (second row).

###  4.7 Qualitative Comparison

In this section, We provide a visual comparison of our method with the
baselines. In Fig. 3 and 4, we provide visual examples of variations in yaw
and pitch for FFHQ and AFHQ Cats. Compared to a PC-GAN and SURF, our proposed
method closely matches the 3D consistency and maintains the image quality of
volumetric rendering. Fig. 5 additionally provides examples of Shapenet Cars
generated using our method and their corresponding images from EG3D.
Similarly, our method exhibits preservation of 3D consistency and image
quality, despite the difference in required computational resources. We
provide more visual results in the supplementary material.

###  4.8 Inversion, Interpolation, and Style Mixing

As the proposed generator follows a StyleGAN architecture, it can easily
benefit from most of the editing techniques common in the GANs’ literature.
Fig. 7 shows examples of inversion using Pivotal Tuning Inversion (PTI) [42],
latent space interpolation, and style mixing.

Figure 7: First row: Inversion using PTI [42]) for EG3D and our method; second
row: interpolation in the latent space of our method; third row: style mixing
in the latent space of our method.

###  4.9 Discussion: StyleGAN2 V.S. StyleGAN3

StyleGAN2 is more computationally efficient than StyleGAN3. Based on the
provided quantitative evaluations, our method reaches comparable image quality
and 3D consistency with both architectures. However, StyleGAN2 is known to
suffer from more texture stitching and artifacts [25], which we also observe
in the generated images (visualizations are provided in the supplementary
material).

###  4.10 Correspondence between Convolutional and Volumetric Renderering

As mentioned before, exploiting the style space of the pretrained NeRF-GAN
also provides an opportunity for establishing a direct correspondence between
the 3D representation of the 3D generator and the generated images using the
convolutional generator. A close comparison of images generated using the
convolutional and volumetric rendering in Figures 3, 4, and, 5 indicates that
the convolutional render is able to infer and match many attributes of the
underlying 3D representation from the shared latent space and generate images
similar in content to those of volumetric rendering. However, there still
remains a gap in the full correspondence of the two rendering methods, as
semantic and identity changes are visible between the corresponding images
generated by the two methods. Investigating more explicit approaches for
enforcing correspondence could be an interesting direction for improving the
convolutional rendering for NeRF-GAN models.

##  5 Conclusion

We presented a method to distill a pretrained NeRF-GAN into a pose-conditioned
convolutional generator. The proposed method enables considerably higher
efficiency, which is crucial if 3D neural rendering is to become ubiquitous
and deployed at scale. To do so, we proposed exploiting the intermediate
latent space of the pretrained NeRF-GAN as a conditioning input of the
convolutional generator. We additionally provided a training protocol to
further improve the visual quality and 3D consistency of the images generated
using our generator. Through our experiments, we showed that our method
maintains good image quality and 3D consistency, significantly better than
previous existing fully-convolutional methods and approaching that of the
baseline NeRF-GAN with volumetric rendering. Finally, while our method takes
steps toward achieving full correspondence between the two rendering methods,
there remains a gap in terms of image semantics. Improving this aspect remains
a subject for further research.

## References

  * [1] Eg3d inversion projector.  https://github.com/oneThousand1000/EG3D-projector.
  * [2] H. Baatz, J. Granskog, M. Papas, F. Rousselle, and J. Novák.  Nerf-tex: Neural reflectance field textures.  Computer Graphics Forum, 41(6):287–301, 2022.
  * [3] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021.
  * [4] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [5] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  * [6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799–5809, 2021.
  * [7] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [8] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In European Conference on Computer Vision (ECCV), 2022.
  * [9] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [10] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8188–8197, 2020.
  * [11] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [12] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong.  Disentangled and controllable face image generation via 3d imitative-contrastive learning.  In IEEE Computer Vision and Pattern Recognition, 2020.
  * [13] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In IEEE Computer Vision and Pattern Recognition Workshops, 2019\.
  * [14] Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  2021\.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, pages 2672–2680, 2014.
  * [16] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d aware generator for high-resolution image synthesis.  In International Conference on Learning Representations, 2022.
  * [17] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang.  Nerfren: Neural radiance fields with reflections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18409–18418, June 2022.
  * [18] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec.  Baking neural radiance fields for real-time view synthesis.  ICCV, 2021.
  * [19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems, pages 6626–6637, 2017.
  * [20] Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia.  Efficientnerf efficient neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12902–12911, June 2022.
  * [21] Yibo Hu, Xiang Wu, Bing Yu, Ran He, and Zhenan Sun.  Pose-guided photorealistic face rotation.  CVPR, 2018.
  * [22] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  CVPR, 2017.
  * [23] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [24] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In Advances in Neural Information Processing Systems, 2020.
  * [25] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in Neural Information Processing Systems, 34:852–863, 2021\.
  * [26] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  pages 4396–4405, 06 2019.
  * [27] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In Proc. CVPR, 2020.
  * [28] Marek Kowalski, Stephan J. Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, and Jamie Shotton.  Config: Controllable neural face image generation.  In European Conference on Computer Vision (ECCV), 2020.
  * [29] Jeong-gi Kwak, Yuanming Li, Dongsik Yoon, Donghyeon Kim, David Han, and Hanseok Ko.  Injecting 3d perception of controllable nerf-gan into stylegan for editable portrait image synthesis.  In European Conference on Computer Vision, pages 236–253. Springer, 2022.
  * [30] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections, 2020.
  * [31] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [32] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  arXiv preprint arXiv:2201.05989, 2022.
  * [33] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, July 2022.
  * [34] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, and Markus Steinberger.  DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks.  Computer Graphics Forum, 40(4), 2021.
  * [35] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In The IEEE International Conference on Computer Vision (ICCV), Nov 2019.
  * [36] Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra.  Blockgan: Learning 3d object-aware scene representations from unlabelled images.  In Advances in Neural Information Processing Systems 33, Nov 2020\.
  * [37] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [38] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.  StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13503–13513, June 2022.
  * [39] Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo.  Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans.  In International Conference on Learning Representations, 2021.
  * [40] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [41] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [42] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  ACM Transactions on Graphics (TOG), 42(1):1–13, 2022.
  * [43] Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [44] Axel Sauer, Kashyap Chitta, Jens Muller, and Andreas Geiger.  Projected gans converge faster.  In NeurIPS, 2021.
  * [45] Axel Sauer, Katja Schwarz, and Andreas Geiger.  Stylegan-xl: Scaling stylegan to large diverse datasets.  volume abs/2201.00273, 2022.
  * [46] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [47] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger.  Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids.  arXiv preprint arXiv:2206.07695, 2022.
  * [48] Yichun Shi, Divyansh Aggarwal, and Anil Jain.  Lifting 2d stylegan for 3d-aware face generation.  pages 6254–6262, 06 2021.
  * [49] Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gérard Medioni.  Gan-control: Explicitly controllable gans.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14083–14093, October 2021.
  * [50] Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka.  Epigraf: Rethinking training of 3d gans.  arXiv preprint arXiv:2206.10535, 2022.
  * [51] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459–5469, 2022.
  * [52] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Perez, Michael Zollhofer, and Christian Theobalt.  Stylerig: Rigging stylegan for 3d control over portrait images.  pages 6141–6150, 06 2020.
  * [53] Alex Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano.  Real-time radiance fields for single-image portrait view synthesis.  In ACM Transactions on Graphics (SIGGRAPH), 2023.
  * [54] Richard Tucker and Noah Snavely.  Single-view view synthesis with multiplane images.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [55] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [56] Liwen Wu, Jae Yong Lee, Anand Bhattad, Yuxiong Wang, and David Forsyth.  Diver: Real-time and accurate neural radiance fields with deterministic integration for volume rendering, 2021.
  * [57] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In ICCV, 2021.
  * [58] Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, and Alex Colburn.  Generative multiplane images: Making a 2d gan 3d-aware.  In Proc. ECCV, 2022.

## Supplementary Material

In what follows, we first provide an evaluation of the correspondence between
the generated images using volumetric rendering and our convolutional
generator in section A. Moreover, in section B, we extend the efficiency
analysis of our method to the setup of the Shapenet Cars dataset. In section
C, we discuss some of the implementation details. Then, we offer a discussion
on the limitations of the proposed method in Sec. D. Finally, we provide more
visual examples for our method in Sec. E.

##  Appendix A Evaluation of Correspondence

To provide a baseline for future research, we measure the correspondence
between generated images using our method and the corresponding images
rendered from EG3D using Peak Signal to Noise Ratio (PSNR), the result of
which is provided in Table 5.

##  Appendix B Analysis of Efficiency on Shapenet Cars

In Fig. 8, we provide a comparison of the computational efficiency of
inference using our method and the baseline NeRF-GAN (EG3D) on Shapenet Cars
setup. As mentioned in section 4.3 of the main paper, EG3D performs volumetric
rendering at the resolution of 64 and super-resolves the output to the
resolution of 128. As it can be seen in Fig. 8, Our method with StyleGAN2 as
its backbone is more efficient than EG3D in terms of both memory consumption
and speed. However, the StyleGAN3 backbone is only more efficient in terms of
memory consumption, but it is slower than the baseline. The reason for this is
that StyleGAN3 is more computationally demanding than the tri-plane generator
in EG3D, which is based on StyleGAN2. In the setup of Shapenet Cars, the
computational complexity of the StyleGAN3 convolutional generator outweighs
that of the lower-resolution volumetric rendering in EG3D, resulting in a
slower inference (Note that the memory consumption is still lower for the
StyleGAN3 convolutional generator).

Figure 8: Comparison of the inference memory consumption and speed (on a fixed
GPU budget) for our method and EG3D on Shapenet Cars. Table 5: PSNR values
($\uparrow$) for measuring the correspondence between the images of EG3D and
our convolutional generator on 3 datasets. The values are calculated for 1k
images. ST2 and ST3 correspond to the two architectures StyleGAN2 and
StyleGAN3 used as the backbone of our method

Dataset | ST2 | ST3
---|---|---
FFHQ | 28.73 | 28.65
AFHQ Cats | 28.46 | 28.10
Shapenet Cars | 36.60 | 36.57

##  Appendix C Implementation Details

###  C.1 SURF Baseline

In the main paper, we provided a baseline, which we referred to as “SURF”,
inspired by the method proposed in the SURF-GAN paper [29] for pose/content
disentanglement of a pre-trained 2D StyleGAN. As the code and the pre-trained
models for such disentanglement are not made publicly available, we provided
our best effort in implementing the SURF baseline inspired by their approach.

The proposed method in [29] consists of two main components: 1. a NeRF-GAN
model called SURF-GAN for portrait images 2. a training recipe to add pose
conditioning to a pre-trained StyleGAN2 using multi-view images generated from
the 3D NeRF-GAN. Specifically, the part relevant as a baseline for our
proposed method is the second component, which we tried to re-implement it for
comparison with our method. To do so, for a fair comparison with our method,
we use EG3D pre-trained on FFHQ for generating multi-view image triplets from
3 different viewpoints (source, canonical, and target views). Following [29],
we use pSp [41], a pretrained inversion encoder to invert the generated multi-
view images in the style space of the 2D generator. Using the multi-view
images and their corresponding style codes obtained using pSp, we train two
mapping networks to:

  1. 1.

map any arbitrary style code to the style code of the canonical viewpoint, and

  2. 2.

map the canonical style code to the style code of the target viewpoint.

For the mapping networks, we use MLP networks with the same architecture as
StyleGAN2’s mapping network. For the training objective, we include
reconstruction losses both on the images (MSE and Perceptual) and latent codes
(MSE), as used in [29]. This architecture and training regime is inspired by
the one in [29], but not exactly the same. We have adapted the method to the
3D generator network we are using (EG3D). Moreover, in the original method,
the mapping from the canonical latent code to the target one is formulated as
the linear combination of a set of learnable pose vectors. In our
implementation, we use a more general mapping by using an MLP network instead.
Nevertheless, the implemented baseline provides a comparison with a proven
alternative strategy to enable pose conditioning of a StyleGAN-like
convolutional architecture.

###  C.2 Inversion

In Sec. 4.8 of the main paper, we provided examples of inversion in our
generator using Pivotal Tuning Inversion (PTI) [42]. Following EG3D, inversion
is performed given a target image and its corresponding camera parameters. As
for the implementation, we use the adaptation to EG3D provided at [1]. We use
500 steps for optimizing the latent code and 350 additional steps for fine-
tuning the generator according to the PTI method.

###  C.3 3D Consistency Metrics

To calculate the 3D Landmark Consistency and Identity Preservation (ID) in
section 4.5.3 of the main paper, we vary the yaw from -40 to +40 and the pitch
from -30 to +30, following the evaluation setup of [29]. As an additional
visualization, Fig. 9 shows the comparison of Identitiy Preservation for each
angle individually.

Figure 9: Comparison of identity preservation across different camera poses.
Our method performs much better than the pose-conditioned GAN baseline (PC-
GAN), approaching the consistency of volumetric rendering (EG3D). The higher
consistency achieved by SURF is due to its limited pose variations.

##  Appendix D Limitations

In our approach, the outputs of the volumetric rendering branch are used to
supervise our convolutional renderer. Thus, the visual quality and 3D
consistency of our approach is largely bound by the quality and consistency of
the pretrained NeRF-GAN. However, our formulation is largely agnostic to the
volumetric generator used. Therefore, improvements in volumetric rendering in
the context of GANs will also transfer to the generated quality of our
approach.

As an example, a close look at the videos provided in the supplementary video
reveals small changes in the degree of the smile in different viewpoints of
some of the generated faces. Such a phenomenon, which is due to the dataset
bias, also exists in EG3D, as analyzed in Sec 1.1 of the supplementary
material of EG3D paper [5] and visible in our results as well.

##  Appendix E Visual Results

In Fig. 10, we provide more visual examples generated using our method from
FFHQ dataset. Fig. 11 shows examples of generated images from AFHQ Cats
dataset using the proposed convolutional generator. Finally, in Fig. 12, we
provide random samples from Shapenet Cars dataset generated using the
volumetric rendering (EG3D) and our method. For a video comparison, please
refer to the supplementary video “supp.mp4”.

Figure 10: Visual examples of pose control in our convolutional generator and
their comparison to those of EG3D on FFHQ dataset. Figure 11: Visual examples
of pose control in our convolutional generator and their comparison to those
of EG3D on AFHQ Cats dataset. Figure 12: Visual examples of pose control in
our convolutional generator and their comparison to those of EG3D on Shapenet
Cars dataset.