11institutetext: † Korea University ‡Drexel Univiersity

# Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable
Portrait Image Synthesis

Jeong-gi Kwak† Yuanming Li† Dongsik Yoon† Donghyeon Kim† David Han‡ Hanseok
Ko†

###### Abstract

Over the years, 2D GANs have achieved great successes in photorealistic
portrait generation. However, they lack 3D understanding in the generation
process, thus they suffer from multi-view inconsistency problem. To alleviate
the issue, many 3D-aware GANs have been proposed and shown notable results,
but 3D GANs struggle with editing semantic attributes. The controllability and
interpretability of 3D GANs have not been much explored. In this work, we
propose two solutions to overcome these weaknesses of 2D GANs and 3D-aware
GANs. We first introduce a novel 3D-aware GAN, SURF-GAN, which is capable of
discovering semantic attributes during training and controlling them in an
unsupervised manner. After that, we inject the prior of SURF-GAN into StyleGAN
to obtain a high-fidelity 3D-controllable generator. Unlike existing latent-
based methods allowing implicit pose control, the proposed 3D-controllable
StyleGAN enables explicit pose control over portrait generation. This
distillation allows direct compatibility between 3D control and many StyleGAN-
based techniques (e.g., inversion and stylization), and also brings an
advantage in terms of computational resources. Our codes are available at
https://github.com/jgkwak95/SURF-GAN.

###### Keywords:

3D-aware portrait generation, pose-disentangled GAN, facial image editing,
novel view synthesis, latent manipulation

Figure 1: (Left): Proposed novel 3D-aware GAN (SURF-GAN) which is capable of
attribute-controllable generation as well as 3D-aware synthesis. (Right):
3D-controllable StyleGAN obtained by distilling the prior of SURF-GAN into 2D
GAN.

##  1 Introduction

Since the advent of Generative Adversarial Networks (GANs) [16], remarkable
progress has been made in the field of photorealistic image generation. The
quality and diversity of images generated by 2D GANs have been improved
considerably and recent models [28, 7, 31, 32, 30] can produce high resolution
images at a level that humans cannot distinguish. Despite the expressiveness
of 2D GANs, they lack 3D understanding, in that the underlying 3D geometry of
an object is ignored in the generation process. As a result, they suffer the
problem of multi-view inconsistency. To overcome the issue, many researchers
have studied 3D controllable image synthesis and it has become one of the
mainstream research in the community. There have been several attempts to
learn 3D pose information with 2D GAN by disentangling pose in the latent
space, but they require auxiliary 3D supervision such as synthetic face
dataset [34] or 3DMM [13, 63, 62]. In addition, a few unsupervised approaches
have been proposed by adopting implicit 3D feature [42, 43] or differentiable
renderer [57, 47] in generation. However, these methods have struggled with
multi-view consistency and photorealism.

Since the introduction of neural radiance fields (NeRF) by Mildenhall et al.
[41] which has achieved notable success in novel view synthesis, a new
paradigm has emerged in 3D-aware generation, called 3D-aware GAN. Several
researchers have proposed 3D-aware generative frameworks [54, 44, 9, 72, 17,
14, 66] by leveraging NeRF as a 3D representation in GAN generator. NeRF-GANs
learn 3D geometry from unlabelled images yet allow accurate and explicit
control of 3D camera based on a volume rendering. Despite the obvious
advantages, 3D GANs based on a pure NeRF network require tremendous
computational resources and generate blurry images. Very recently, several
approaches have alleviated the problems and have shown photorealistic output
with high resolution by incorporating rear-end 2D networks [72, 17, 8, 14,
66]. However 3D GANs have difficulty with attribute-controllable generation or
real image editing because their latent space has been rarely investigated for
interpretable generation (Fig. 2).

In summary, these two distinct approaches have strengths and weaknesses that
are complementary: 3D-aware GAN can generate novel poses but it has trouble
with disentangling and manipulating attributes; 2D GAN is capable of
controlling attributes but it struggles with 3D controllability. In this work,
we propose novel solutions to overcome each weakness of 2D GANs and 3D GANs.

First, we propose a novel 3D-aware GAN, i.e., SURF-GAN, which can discover
semantic attributes by learning layer-wise SUbspace in INR NeRF-based
generator in an unsupervised manner. The discovered semantic vectors can be
controlled by corresponding parameters, thus this property allows us to
manipulate semantic attributes (e.g., gender, hair color, etc.) as well as
explicit pose.

With the proposed SURF-GAN, we take one more step to transform StyleGAN into a
3D-controllable generator. We inject the prior of 3D-aware SURF-GAN into the
expressive and disentangled latent space of 2D StyleGAN. Unlike the previous
methods [55, 19, 56] that allows implicit pose control, we make StyleGAN
enable explicit control over pose. It means that the generator is capable of
synthesizing accurate images based on a conditioned target view. By utilizing
SURF-GAN which consists of pure NeRF layers as a generator of pseudo multi-
view images, the transformed StyleGAN can learn elaborate control over 3D
camera pose with latent manipulation. To this end, we proposed a method to
find several orthogonal directions (not a single) related to the same pose
attribute, and explicit control over the pose is accomplished by a combination
of these directions. With a GAN inversion encoder, 3D controllable StyleGAN
can be extended to the task of novel pose synthesis from a real image.

In addition to 3D perception, we also inject the controllability about
semantic attributes that SURF-GAN finds. We can find more pose-robust latent
path in the latent space of StyleGAN because SURF-GAN can manipulate a
specific semantic while keeping view direction unchanged. Moreover, it allows
further applications related to StyleGAN family, e.g., 3D control over
stylized images generated by fine-tuned StyleGAN. It is notable that our
approach neither requires 3D supervision nor exploits auxiliary off-the-shelf
3D models (e.g., 3DMM or pose detector) in both training and inference because
SURF-GAN learns 3D geometry from unlabelled 2D images from scratch.

Figure 2: Results of attribute manipulation when applying SeFa [56] utilized
for 2D GANs to a 3D-aware GAN [9] (left). The captured attributes are
entangled or not meaningful. In contrast, SURF-GAN can captures disentangled
and semantic attributes (right).

In summary, our contributions are as follow:

  * •

We propose a novel 3D-aware GAN, called SURF-GAN, which can discover
controllable semantic attributes in an unsupervised manner.

  * •

By injecting editing directions from the low-resolution 3D-aware GAN into the
high-resolution 2D StyleGAN, we achieve a 3D controllable generator which is
capable of explicit control over pose and 3D consistent editing.

  * •

Our method is directly compatible with various well-studied 2D StyleGAN-based
techniques such as inversion, editing or stylization.

##  2 Related Work

####  2.0.1 Pose-disentangled GANs.

The remarkable advances have been achieved in photorealism by state-of-the-art
GAN models [28, 7, 31, 32, 30]. However, pose control by image generators has
been limited due to a lack of 3D understanding in the synthesizing process.
Thereby, several works have attempted to disentangle the pose information from
other attributes in 2D GANs. The disentanglement has been achieved by
leveraging supervision such as 3DMM [13, 63, 62, 68], landmark [24], synthetic
images from 3D engine [34] or pose detector [58]. A few unsupervised
approaches without 3D supervision [42, 43] have been proposed by disentangling
pose with implicit 3D feature projection, but they allow only implicit 3D
control and show blurry results. Recently, a few methods [57, 47] have
incorporated a pre-trained StyleGAN with a differentiable renderer, but they
struggle with photorealism, high-resolution [47] and real image editing [57].

####  2.0.2 Interpretabilty and controllabiltiy of GAN.

The well-trained 2D GANs, such as StyleGAN [31, 32] have shown capable of
disentangling the latent space. Recent works [55, 19, 3, 45, 56, 67, 48] have
demonstrated semantic manipulation, especially for facial attributes, by
analyzing the manifold and finding meaningful direction or mapping. Combining
with GAN inversion [1, 73, 2, 52, 64, 53, 4, 5], the applications of 2D GANs
have been extended to real image editing. Alternatively, there have been
studies [11, 27, 36, 25] that discover and disentangle latent embeddings into
interpretable dimensions during training of the generator. EigenGAN [21] that
inspired our approach has demonstrated interpretable latent dimensions by
designing layer-wise subspace embedding. However, both types of methods
support implicit control over the discovered semantics. In the case of a pose
that can be defined with camera parameters, these methods struggle to
synthesize explicit novel view elaborately. Of course, the implicit methods
can eventually create the desired pose through manual and iterative
adjustment, but this is not an ideal situation. We can obtain a frontalized
image automatically with some latent-based methods [55, 35, 52], but not for
arbitrary target pose. Recently, Chen et al. [10] have introduced a generator
allowing explicit control over pose, but it requires 3D mesh for pre-training
process.

####  2.0.3 3D-aware GANs.

Beyond the disentanglement of pose information, many efforts have been made to
obtain 3D-awareness in generation. Earlier methods have adopted several
explicit 3D representations in 2D image generation such as voxel [74, 39, 22,
15] or mesh [37, 61]. However, they suffer from a lack of visual quality and
limited resolution. Recently, approaches [54, 9, 44, 17, 72, 8, 14, 46, 66]
based on neural fields have made significant progress in photorealism and 3D
consistency. Nevertheless, these 3D-aware GANs have weakness in finding and
editing semantic attribute because their latent space has been rarely
investigated. Very recently, Sun et al. [60] have proposed an editable NeRF-
GAN, but it does not handle diverse semantic attributes and requires semantic
maps as supervision. In addition, 3D GANs struggle with novel pose generation
of real image despite their capability of multi-view consistency. Recently
proposed EG3D [8] has shown experiments of novel view synthesis and presented
outstanding results, but it requires iterative optimization for latent code
and fine-tuning of the generator [53] for each target image.

##  3 Proposed method

In this section, we describe our method, by first introducing SURF-GAN in
detail and then by explaining a method to inject the prior of 3D SURF-GAN into
2D StyleGAN. Note that the word “StyleGAN” denotes StyleGAN2 [32].

###  3.1 Towards controllable NeRF-GAN

####  3.1.1 Preliminaries: NeRF-GANs.

Existing 2D GANs (e.g., StyleGAN [31, 32]) synthesize output image directly
with sampled latent vector. However, NeRF-GANs [9, 44, 17, 72] generate a
radiance field [41] before rendering 2D image. Given a position
$\mathbf{x}\in\mathbb{R}^{3}$ and a viewing direction
$\mathbf{v}\in\mathbb{S}^{2}$, it predicts a volume density
$\mathbf{\sigma}(\mathbf{x})\in\mathbb{R}_{+}$ and the view-dependent RGB
color $\mathbf{c}(\mathbf{x},\mathbf{v})\in\mathbb{R}^{3}$ of the input point.
The points are sampled from rays of camera, and then an image is rendered into
2D grid with a classic volume rendering technique [26]. To produce diverse
images, existing NeRF-GAN methods adopt StyleGAN-like modulation, where some
components in the implicit neural network, e.g., intermediate features [9, 72]
or weight matrices [17] are modulated by sampled noise passing though a
mapping network. Thereby, NeRF-GAN can control the pose by manipulating
viewing direction $\mathbf{v}$ and change identity by injecting different
noise vector. Nevertheless, it is ambiguous how to interpret the latent space
and how to disentangle semantic attributes of NeRF-GAN for controllable image
generation.

Figure 3: Overview of SURF-GAN generator. Interpretable dimensions are
caputured in layers with sub-modulation vectors. As like INR 3D-aware GANs, it
takes position and view direction as input and predicts view dependent color
($\mathbf{c}$) and its density ($\sigma$).

####  3.1.2 Learning layer-wise subspace in NeRF network.

Inspired by EigenGAN [21], we adopt a different strategy from the existing
methods [9, 72] those modulation is obtained by the mapping network consisting
of several MLPs. EigenGAN learns interpretable subspaces in layers of its
generator during training. However, EigenGAN is typical 2D convolution-based
GAN framework, thus its concept is inapplicable to INR based NeRF-GAN.
Therefore, we propose a novel framework (i.e., SURF-GAN), which captures the
disentangled attributes in layers of NeRF network. Fig. 3 shows the overview
of SURF-GAN. The generator consists of $t+1$ SURF blocks ($t$ for shared
layers and one for color layer). Following $\pi$-GAN, SURF block adopts the
feature-wise linear modulation (FiLM) [49] to transform the intermediate
features with frequencies $\boldsymbol{\gamma}_{i}$ and phase shifts
$\boldsymbol{\beta}_{i}$, and followed SIREN activation [59]. SURF block in
$i^{\text{th}}$ layer is formulated as

$\boldsymbol{\psi}_{i}=\text{SURF}_{i}\left(\boldsymbol{\psi}_{i-1},\boldsymbol{\phi}_{i}\right)=\sin\left(\boldsymbol{\gamma}_{i}\cdot\left(\mathbf{W}_{i}\boldsymbol{\psi}_{i-1}+\mathbf{b}_{i}\right)+\boldsymbol{\beta}_{i}\right)+\boldsymbol{\psi}_{i-1},$
(1)

where $\boldsymbol{\psi}_{i-1}$ and $\boldsymbol{\phi}_{i}$ denote input
feature and modulation of $i^{th}$ layer respectively. $\mathbf{W}_{i}$ and
$\mathbf{b}_{i}$ represent the weight matrix and followed bias. Unlike other
NeRF-GANs, we add skip connection [20] to prevent drastic change of modulation
vectors in training. In the model, a subspace embedded in each layer
determines the modulation. Each subspace has orthogonal basis and it can be
updated during training. The basis are learned to capture semantic modulation.
Concretely, in the case of $i^{th}$ layer, a specific subspace determines the
modulation of $i^{th}$ layer of NeRF network. It consists of learnable
matrices, orthonormal basis
$\mathbf{U}_{i}=\left[\mathbf{u}_{i1},\ldots,\mathbf{u}_{iK}\right]$ and a
diagonal matrix
$\mathbf{D}_{i}=\operatorname{diag}\left(d_{i1},\ldots,d_{iK}\right)$. Each
column of $\mathbf{U}_{i}$ plays a role of sub-modulation and it is updated to
discover a meaningful direction that results in semantic change in image
space. $d_{i1},\ldots,d_{iK}$ serve as scaling factors of corresponding basis
vectors $\mathbf{u}_{i1},\ldots,\mathbf{u}_{iK}$. The latent
$\mathbf{z}_{i}\in\mathbb{R}^{K}$ is set of $K$ scalar control parameters,
i.e.,

$\mathbf{z}_{i}=\left\\{z_{ij}\in\mathbb{R}\mid
z_{ij}\sim\mathcal{N}(0,1),j=1,\ldots,K\right\\},$ (2)

where ${z}_{ij}$ is a coefficient of sub-modulation $d_{ij}\mathbf{u}_{ij}$.
Hence, the modulation of $i^{th}$ layer $\phi_{i}$ is decided by weighted
summation of $K$ sub-modulations with $\mathbf{z}_{i}$, i.e,

$\boldsymbol{\phi}_{i}=\mathbf{U}_{i}\mathbf{D}_{i}\mathbf{z}_{i}+\boldsymbol{\mu}_{i}=\sum_{j=1}^{K}z_{ij}d_{ij}\mathbf{u}_{ij}+\boldsymbol{\mu}_{i},$
(3)

where the marginal vector $\boldsymbol{\mu}_{i}$ is employed to capture
shifting bias. Finally, a simple affine transformation is applied to
$\boldsymbol{\phi}_{i}$ for matching dimension and obtaining frequency
$\boldsymbol{\gamma}_{i}$ and phase shift $\boldsymbol{\beta}_{i}$. At
training phase, SURF-GAN layers learn variations of meaningful modulation
controlled by randomly sampled $\mathbf{z}$. Additionally, an input noise
$\epsilon$ is also injected to capture the rest variations missed by the
layers. To improve the disentanglement of attributes and to prevent the basis
fall into a trivial solution, we adopt the regularization loss to guarantee
the column vectors of $\mathbf{U}_{i}$ to be orthogonal following EigenGAN,
i.e.,

$\mathcal{L}_{\text{reg}}=\mathbb{E}_{i}[\left\|\mathbf{U}_{i}^{\mathrm{T}}\mathbf{U}_{i}-\mathbf{I}\right\|_{1}].$
(4)

Finally, output image is rendered by volume rendering technique [26]. At
inference phase, we can control the discovered semantic attributes by
manipulating corresponding element in $\mathbf{z}$. In addition, SURF-GAN
enables explicit control over pose using viewing direction $\mathbf{v}$ as
like other NeRF-based models.

###  3.2 Explicit control over pose with StyleGAN

In Sec. 3.2 and Sec. 3.3, we introduce a method to inject 3D perception and
attribute controllability of SURF-GAN into StyleGAN.

####  3.2.1 Leveraging 3D-aware SURF-GAN.

The first step is to transform pre-trained StyleGAN into a 3D controllable
generator. We start with a question: How can we make StyleGAN be capable of
controlling over pose explicitly when given arbitrary latent code? To this
end, we utilize SURF-GAN as a pseudo ground-truth generator. It provides three
images, i.e., $I_{s}$, $I_{c}$, $I_{t}$ which denote source, canonical, and
target image respectively. Here, $\mathbf{z}$ is fixed in all images but the
view directions of $I_{s}$ and $I_{t}$ are randomly sampled and $I_{c}$ has
canonical view (i.e., $\mathbf{v}$=$[0,0]$). Therefore, we can exploit them as
multi-view supervision of the same identity. Afterwards, the images are
embedded to $\mathcal{W}+$ [1] space by a GAN inversion encoder $E$, i.e,
$\\{\mathbf{w}_{s},\mathbf{w}_{c},\mathbf{w}_{t}\\}=\\{E(I_{s}),E(I_{c}),E(I_{t})\\}$.
Here, we exploit the pre-trained pSp [52] encoder and it actually predicts the
residual and adds it to the mean latent vector, but we omit the notation for
simplicity.

####  3.2.2 Mapping to a canonical latent vector.

To handle arbitrary pose without employing off-the-shelf 3D models, we need to
build an additional process. To this end, we propose a canonical latent mapper
$T$, which converts an arbitrary code to a canonical code in the latent space
of StyleGAN. Here, the canonical code implies being a canonical pose (frontal)
in image space. $T$ takes $\mathbf{w}_{s}$ as input and predicts its
frontalized version $\mathbf{\hat{w}}_{c}=T(\mathbf{w}_{s})$ with the mapping
function. In order to train $T$, we exploit latent loss to minimize the
difference between the predicted $\hat{\mathbf{w}}_{c}$ and pseudo ground
truth of canonical code $\mathbf{w}_{c}$, i.e.,

$\mathcal{L}^{c}_{\text{w}}=\lVert{\mathbf{w}_{c}-T(\mathbf{w}_{s})}\rVert_{1}.$
(5)

To guarantee plausible translation result in image space, we also adopt pixel-
level $\ell_{2}$-loss and LPIPS loss [69] between two decoded images, i.e.,

$\displaystyle\mathcal{L}^{c}_{\text{I}}$
$\displaystyle=\lVert{I^{\prime}_{c}-\hat{I}_{c}}\rVert^{2}_{2}$ (6)
$\displaystyle\mathcal{L}^{c}_{\text{LPIPS}}$
$\displaystyle=\lVert{F(I^{\prime}_{c})-F(\hat{I}_{c})}\rVert^{2}_{2},$ (7)

where $I^{\prime}_{c}$ and $\hat{I}_{c}$ represent the decoded images from
$\mathbf{w}_{c}$ and $\mathbf{\hat{w}}_{c}$ respectively, and $F(\cdot)$
denotes the perceptual feature extractor. Hence, the loss for canonical view
generation is formulated by

$\mathcal{L}^{\text{c}}=\lambda_{1}\mathcal{L}^{\text{c}}_{\text{w}}+\lambda_{2}\mathcal{L}^{\text{c}}_{\text{I}}+\lambda_{3}\mathcal{L}^{\text{c}}_{\text{LPIPS}}.$
(8)

####  3.2.3 Target view generation.

Next, the canonical vector is converted to a target latent vector according to
given a target view $\textbf{v}_{t}=\left[\alpha,\beta\right]$ as an
additional input. Here, $\alpha$ and $\beta$ stand for pitch and yaw
respectively. The manipulation is conducted in the latent space of StyleGAN by
adding a pose vector which is obtained by a linear combination of pitch and
yaw vectors ($\mathbf{p}$ and $\mathbf{y}$, respectively) with
$\textbf{v}_{t}$ as coefficients, i.e.,
$\hat{\mathbf{w}}_{t}=\hat{\mathbf{w}}_{c}+\mathbf{L}\mathbf{v}_{t}^{T}$,
where $\mathbf{L}=\left[\mathbf{p}\;\mathbf{y}\right]$. Therefore, we need to
find optimal solution of $\mathbf{L}$ which can represent an adequate 3D
control over pose.

Figure 4: The controllable StyleGAN allows explicit control over camera view.
It can be used for novel pose synthesis (upper) and view-conditioned image
generation (lower).

Although earlier studies [55, 45] have shown successful interpolation results
with the linear manipulation, unfortunately, they have found sub-optimal
solutions that just control the intended pose attribute implicitly rather than
explicit control over 3D camera. The interesting fact we observed is that the
pose-related attribute (e.g., yaw) is not uniquely determined by a single
direction. Rather, several orthogonal directions can have different effects on
the same attribute. For example, two orthogonal direction A and B both can
affect yaw but work differently. Based on this observation, we exploit several
sub-direction vectors to compensate marginal portion that is not captured by a
single direction vector. Our hypothesis is that the optimal direction that
follows real geometry can be obtained by a proper combination of the sub-
direction vectors. Borrowing the idea of basis in Sec. 3.1, we construct each
of $N$ learnable basis to obtain final pose vectors for pitch and yaw
respectively. Therefore, we optimize the matrices
$\mathbf{P}=\left[\mathbf{d}^{p}_{1},\ldots,\mathbf{d}^{p}_{N}\right]$ and
$\mathbf{Y}=\left[\mathbf{d}^{y}_{1},\ldots,\mathbf{d}^{y}_{N}\right]$. The
process to obtain the target vector can be described as,

$\hat{\mathbf{w}}_{t}=\hat{\mathbf{w}}_{c}+\sum_{i=1}^{N}(\alpha\cdot
l^{p}_{i}\mathbf{d}^{p}_{i}+\beta\cdot l^{y}_{i}\mathbf{d}^{y}_{i}),$ (9)

where the $l_{i}^{p}$ and $l_{i}^{y}$ represent the learnable scalining factor
deciding the importance of basis $\mathbf{d}^{p}_{i}$ and $\mathbf{d}^{y}_{i}$
respectively. To penalize finding redundant directions, we add orthogonal
regularization, i.e.,

$\mathcal{L}_{\text{reg}}=\left\|\mathbf{P}^{\mathrm{T}}\mathbf{P}-\mathbf{I}\right\|_{1}+\left\|\mathbf{Y}^{\mathrm{T}}\mathbf{Y}-\mathbf{I}\right\|_{1}.$
(10)

Similar to the canonical view generation, the model is penalized by the
difference of the latent codes ($\mathbf{w}_{t}$ vs. $\hat{\mathbf{w}}_{t}$)
and that of the corresponding decoded images ($I^{\prime}_{t}$ vs.
$\hat{I}_{t}$). In addition, we also utilize LPIPS loss. Therefore, the
objective function of target view generation is described as,

$\mathcal{L}^{\text{t}}=\lambda_{4}\mathcal{L}^{\text{t}}_{\text{w}}+\lambda_{5}\mathcal{L}^{\text{t}}_{\text{I}}+\lambda_{6}\mathcal{L}^{\text{t}}_{\text{LPIPS}}+\lambda_{7}\mathcal{L}_{\text{reg}}.$
(11)

Finally, the full objective to train the proposed modules can be formulated as
$\mathcal{L}=\mathcal{L}^{\text{c}}+\mathcal{L}^{\text{t}}.$ After training,
StyleGAN ($G$) becomes a 3D-controllable generator ($G_{3D}$) with the
proposed modules as illustrated in Fig. 4. We can achieve a high quality image
with intended pose by conditioning view as follow,

$I_{\mathbf{v}}=G_{3D}(\mathbf{w},\mathbf{v}_{t})=G(\mathbf{w}+T(\mathbf{w})+\mathbf{L}\mathbf{v}_{t}^{T}),$
(12)

where $I_{\mathbf{v}}$ represents a generated image with target pose
$\mathbf{v}_{t}$ and $\mathbf{w}\in\mathcal{W}+$ is duplicated version of
512-dimensional style vector in $\mathcal{W}$ which is obtained by the mapping
network in StyleGAN. Moreover, we can extend our method to synthesize novel
view of real images by combining with GAN inversion, i.e.,

$I^{t}_{\mathbf{v}}=G_{3D}(E(I^{s}),\mathbf{v}_{t}),$ (13)

where $I^{s}$ is an input source image in arbitrary view and
$I^{t}_{\mathbf{v}}$ denotes a generated target image with target pose
$\mathbf{v}_{t}$. Note that our method can handle arbitrary images without
exploiting off-the-shelf 3D models such as pose detectors or 3D fitting
models. In addition, it synthesizes output at once without an iterative
optimization process for overfitting latent code into an input portrait image.

###  3.3 Finding semantic direction with SURF-GAN

Beyond 3D perception, we can discover semantic directions in the latent space
of StyleGAN that can control facial attributes using SURF-GAN generated
images. Such directions can be obtained by a simple vector arithmetic [51]
with two latent codes or several interpolated samples generated by SURF-GAN.
Although our approach does not overwhelm state-of-the-art methods analyzing
via supervision, it would be a simple yet effective alternative that can
provide pose-robust editing directions. Of course, the discovery using SURF-
GAN is one of many applicable approaches and we can also utilize the existing
semantic analysis methods [55, 19, 56] because our model is flexibly
compatible with well-studied StyleGAN-based techniques.

##  4 Experimental result

This section presents qualitative and quantitative comparisons with state-of-
the-art methods and analysis of our method. Additional experiments and
discussions not included in this paper can be found in the supplementary
material.

Figure 5: Discovered semantic attributes at different layers in SURF-GAN. We
can manipulate the attributes (e.g., hair color, gender, etc.) with the
control parameters as well as explicit control over 3D camera. L$i$D$j$
denotes the $j^{th}$ basis of the layer $i^{th}$.

###  4.1 Implementation

####  4.1.1 SURF-GAN.

We use each of two datasets to train SURF-GAN, i.e., CelebA [38] dataset and
FFHQ [31] dataset. We set the number of sub-modulations in each layer $K$= 6
(Eq. 2 and Eq. 3) and the number of modulated layers (SURF blocks) is nine
($\therefore t=8$). The other settings are roughly the same with those of
$\pi$-GAN. More details can be found in the supplementary paper.

####  4.1.2 3D-controllable StyleGAN.

For training of 3D controllable StyleGAN, we exploit generated images by SURF-
GAN trained with FFHQ because StyleGAN [32] and GAN inversion encoder [52] are
pre-trained with FFHQ. We design the model to alter only the first four
$\mathbf{w}$ vectors (i.e., 4$\times$512) which have been known to control
pose [31, 70]. We set the number of sub-direction $N=5$ (Eq. 9). The hyper-
parameter of the loss function (Eq. 8 and Eq. 11) are set to
$\lambda_{1}$,$\lambda_{4}$=10, $\lambda_{7}=100$, and 1.0 for the others.

###  4.2 Controllability of SURF-GAN

First, we present the attributes of CelebA discovered by SURF-GAN in Fig. 5.
As like other 3D-aware GANs [9, 44, 54, 72, 17, 8], it can synthesis a view-
conditioned image, i.e., yaw and pitch can be controlled explicitly with input
view direction (top row). In contrast to other 3D NeRF-GANs, SURF-GAN can
discover semantic attributes in different layers in an unsupervised manner.
Additionally, the discovered attributes can be manipulated by the
corresponding control parameters. As shown in Fig. 5, different layers of
SURF-GAN capture diverse attributes such as gender, hair color, illumination,
etc. Interestingly, we observe the early layers capture high-level semantics
(e.g., overall shape or gender) and the rear layers focus fine details or
texture (e.g., illumination or hue). This property is similar to that seen in
2D GANs even though SURF-GAN consists of MLPs without convolutional layers.
Additional discovered attributes, those of FFHQ and the comparison with
$\pi$-GAN, which is a pure NeRF-GAN as like ours can be found in the
supplementary material.

Table 1: Quantitative comparison of the proposed 3D controllable StyleGAN with other 3D controllable generative models. We use FID, pose accuracy, and frames per second for evaluation. $\dagger$ denotes quoting from the original paper.  | ConfigNet |  $\;\pi$-GAN |  CIPS-3D |  LiftedGAN |  Ours
---|---|---|---|---|---
FID ($\downarrow$) | 33.41† | 47.68 | 6.97† | 29.81† | 4.72
Pose err.($\times$$10^{-2})$ ($\downarrow$) | 9.56 | 3.81 | 9.12 | 5.52 | 4.24
Frames/Sec. ($\uparrow$) | 345 | 4 | 22 | 56 | 72
Figure 6: Quantitative comparison of 3D-controllable models on identity
preservation under different angles using the averaged cosine similarity from
ArcFace [12].

###  4.3 Portrait image generation with 3D control

To evaluate the performance of the proposed 3D-controllable StyleGAN, we
report the qualitative and quantitative comparison with state-of-the-art
models [34, 9, 72, 57] whose generator allows explicit control over pose. Fig.
7 shows synthesis results of each model for given target views. Here, the
results are $256^{2}$ images generated by each method trained with FFHQ [31].
ConfigNet reveals lack of visual quality and weakness in large pose changes.
$\pi$-GAN shows the accurate geometry because its generator consists of pure
NeRF layers, but this property also results in some degenerated visual
quality. CIPS-3D presents improved visual quality by adopting followed 2D INR
network, but it suffers from 3D inaccuracy in specific poses. LiftedGAN
generates reasonable outputs according to target views by utilizing
differentiable renderer, but it lacks photorealism. Our method generates
photorealistic images and shows plausible control over pose and multi-view
consistency. We also report the quantitative comparisons of the models in
Table. 1 and Fig. 20. We use FID score, pose accuracy estimated by 3D model
[75], frames per second, and identity similarity [12] as evaluation metrics.
Compared to 3D-aware models, our method achieves a competitive score on pose
accuracy and delivers superior results in efficiency, visual quality, and
multi-view consistency. Although 2D-based ConfigNet shows overwhelming
efficiency, it struggles with multi-view consistency and photorealism.

Figure 7: Qualitative results of 3D-controllable generative models under
target poses. Figure 8: Face frontalization results by the methods that can
edit the pose of real image. Upper row denotes 3D supervision of each method
for training (T) and inference (I).

###  4.4 Novel view synthesis of real image

By utilizing GAN inversion method, our method can perform novel view synthesis
from a single portrait. Here, we use pSp [52] encoder for the inversion. To
demonstrate the effectiveness of the canonical mapper, we firstly present the
frontalization results in Fig. 8, which is a special case of novel pose
synthesis. We mark the 3D supervision in training and inference for each
method. Here, pSp-FF denotes the frontalization-only version of pSp [52]. Our
method successfully generates a canonical view while preserving the identity.
Next, we further compare the novel view synthesis results of each model.
ConfigNet and $\pi$-GAN with optimizing latent code through iterative manner
for overfitting to single test image show inferior results, especially in
large pose variation. Rotate-and-Render (R&R) [71] presents reasonable results
by exploiting off-the-shelf 3D fitting models [75] in the generation process.
However, R&R loses some fine details of properties of the original, such as
hair or background. Our models can edit pose successfully while preserving
identity even though it does not require off-the-shelf 3D models and
additional optimization for overfitting to an input. It is also demonstrated
by the quantitative results in Fig. 9 which reports the averaged cosine
similarity between input image and outputs at given various angles using
ArcFace [12] and runtime of each method to process a single image.

Figure 9: Quantitative results of novel view synthesis models. We compute
identity similarity between input and synthesized images using ArcFace (left)
and runtime (right).  Figure 10: Results of novel view synthesis under various
target poses using CelebA-HQ.

###  4.5 Semantic attribute manipulation under conditioned poses

Fig. 11 presents the results of semantic attribute editing with pose control
by 3D-controllable StyleGAN. The upper row stands for controllable generation
(a) and the lower represents real image editing (b). The presented attributes,
i.e., skin color, hue, hair color, and bangs are those discovered by SURF-GAN.

###  4.6 Applications

Our model can be flexibly integrated with other methods that also exploit pre-
trained StyleGAN. Beyond the real image domain, we present a novel view
synthesis of the stylized images such as toon or painting in Fig. 12. We use a
interpolated StyleGAN proposed by Pinkney and Alder [50] for toonifying and a
transferred StyleGAN trained with MetFace [29] for painting-style outputs.

##  5 Conclusion

In this paper, we solved the problems of 3D-aware GANs and 2D GANs by
introducing SURF-GAN and 3D-controllable StyleGAN. Unlike other 3D-aware GANs,
SURF-GAN can discover meaningful semantics and control them in an unsupervised
manner. Using SURF-GAN, we convert StyleGAN to be explicitly 3D-controllable
and it delivers outstanding results in both random image generation and novel
view synthesis of real image. In addition, our method has the potential to be
flexibly combined with other methods. We expect our work will be used
practically and effectively in various tasks and hope it will open up a new
direction in 3D-aware generation and editing fields.

####  5.0.1 Acknowledgement.

This work was supported by DMLab. We also thank to Anonymous ECCV Reviewers
for their constructive suggestions and discussions on our paper.

Figure 11: Editing both attributes and view direction by 3D-controllable
StyleGAN. Figure 12: Explicit 3D control over real and stylized images.

## References

  * [1] Abdal, R., Qin, Y., Wonka, P.: Image2stylegan: How to embed images into the stylegan latent space? In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [2] Abdal, R., Qin, Y., Wonka, P.: Image2stylegan++: How to edit the embedded images? In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [3] Abdal, R., Zhu, P., Mitra, N.J., Wonka, P.: Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG) (2021)
  * [4] Alaluf, Y., Patashnik, O., Cohen-Or, D.: Restyle: A residual-based stylegan encoder via iterative refinement. In: International Conference on Computer Vision (ICCV) (2021)
  * [5] Alaluf, Y., Tov, O., Mokady, R., Gal, R., Bermano, A.H.: Hyperstyle: Stylegan inversion with hypernetworks for real image editing. arXiv (2021)
  * [6] Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.P.: Openface 2.0: Facial behavior analysis toolkit. In: International conference on automatic face & gesture recognition (FG) (2018)
  * [7] Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: International Conference on Learning Representations (ICLR) (2019)
  * [8] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., et al.: Efficient geometry-aware 3d generative adversarial networks. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [9] Chan, E.R., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [10] Chen, A., Liu, R., Xie, L., Chen, Z., Su, H., Yu, J.: Sofgan: A portrait image generator with dynamic styling. Transactions on Graphics (TOG) (2022)
  * [11] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In: Advances in Neural Information Processing Systems (NeurIPS) (2016)
  * [12] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [13] Deng, Y., Yang, J., Chen, D., Wen, F., Tong, X.: Disentangled and controllable face image generation via 3d imitative-contrastive learning. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [14] Deng, Y., Yang, J., Xiang, J., Tong, X.: GRAM: Generative radiance manifolds for 3d-aware image generation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [15] Gadelha, M., Maji, S., Wang, R.: 3d shape induction from 2d views of multiple objects. In: International Conference on 3D Vision (3DV) (2017)
  * [16] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NeurIPS) (2014)
  * [17] Gu, J., Liu, L., Wang, P., Theobalt, C.: StyleNeRF: A style-based 3d-aware generator for high-resolution image synthesis. arXiv (2021)
  * [18] Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., Li, S.Z.: Towards fast, accurate and stable 3d dense face alignment. In: European Conference on Computer Vision (ECCV) (2020)
  * [19] Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: Ganspace: Discovering interpretable gan controls. arXiv (2020)
  * [20] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [21] He, Z., Kan, M., Shan, S.: EigenGAN: Layer-wise eigen-learning for gans. In: International Conference on Computer Vision (ICCV) (2021)
  * [22] Henzler, P., Mitra, N.J., Ritschel, T.: Escaping plato’s cave: 3d shape from adversarial rendering. In: International Conference on Computer Vision (ICCV) (2019)
  * [23] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)
  * [24] Hu, Y., Wu, X., Yu, B., He, R., Sun, Z.: Pose-guided photorealistic face rotation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [25] Jeon, I., Lee, W., Pyeon, M., Kim, G.: Ib-gan: Disengangled representation learning with information bottleneck generative adversarial networks. In: AAAI Conference on Artificial Intelligence (AAAI) (2021)
  * [26] Kajiya, J.T., Von Herzen, B.P.: Ray tracing volume densities. SIGGRAPH (1984)
  * [27] Kaneko, T., Hiramatsu, K., Kashino, K.: Generative attribute controller with conditional filtered generative adversarial networks. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [28] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (ICLR) (2018)
  * [29] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [30] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2021)
  * [31] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [32] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [33] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2014)
  * [34] Kowalski, M., Garbin, S.J., Estellers, V., Baltrušaitis, T., Johnson, M., Shotton, J.: Config: Controllable neural face image generation. In: European Conference on Computer Vision (ECCV) (2020)
  * [35] Kwak, J.g., Li, Y., Yoon, D., Han, D., Ko, H.: Generate and edit your own character in a canonical view. arXiv (2022)
  * [36] Lee, W., Kim, D., Hong, S., Lee, H.: High-fidelity synthesis with disentangled representation. In: European Conference on Computer Vision (ECCV) (2020)
  * [37] Liao, Y., Schwarz, K., Mescheder, L., Geiger, A.: Towards unsupervised learning of generative models for 3d controllable image synthesis. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [38] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
  * [39] Lunz, S., Li, Y., Fitzgibbon, A., Kushman, N.: Inverse graphics gan: Learning to generate 3d shapes from unstructured 2d data. arXiv (2020)
  * [40] Mescheder, L., Geiger, A., Nowozin, S.: Which training methods for gans do actually converge? In: International Conference on Learning Representations (ICLR) (2018)
  * [41] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: Representing scenes as neural radiance fields for view synthesis. In: European Conference on Computer Vision (ECCV) (2020)
  * [42] Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., Yang, Y.L.: HoloGAN: Unsupervised learning of 3d representations from natural images. In: International Conference on Computer Vision (ICCV) (2019)
  * [43] Nguyen-Phuoc, T., Richardt, C., Mai, L., Yang, Y.L., Mitra, N.: BlockGAN: Learning 3d object-aware scene representations from unlabelled images. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [44] Niemeyer, M., Geiger, A.: Giraffe: Representing scenes as compositional generative neural feature fields. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [45] Nitzan, Y., Gal, R., Brenner, O., Cohen-Or, D.: Large: Latent-based regression through gan semantics. arXiv (2021)
  * [46] Or-El, R., Luo, X., Shan, M., Shechtman, E., Park, J.J., Kemelmacher-Shlizerman, I.: StyleSDF: High-resolution 3d-consistent image and geometry generation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [47] Pan, X., Dai, B., Liu, Z., Loy, C.C., Luo, P.: Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. In: International Conference on Learning Representations (ICLR) (2021)
  * [48] Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., Lischinski, D.: Styleclip: Text-driven manipulation of stylegan imagery. In: International Conference on Computer Vision (ICCV) (2021)
  * [49] Perez, E., Strub, F., de Vries, H., Dumoulin, V., Courville, A.C.: Film: Visual reasoning with a general conditioning layer. In: AAAI Conference on Artificial Intelligence (AAAI) (2018)
  * [50] Pinkney, J.N., Adler, D.: Resolution dependent gan interpolation for controllable image synthesis between domains. arXiv (2020)
  * [51] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv (2015)
  * [52] Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D.: Encoding in style: a stylegan encoder for image-to-image translation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [53] Roich, D., Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. arXiv (2021)
  * [54] Schwarz, K., Liao, Y., Niemeyer, M., Geiger, A.: GRAF: Generative radiance fields for 3d-aware image synthesis. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [55] Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of gans for semantic face editing. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [56] Shen, Y., Zhou, B.: Closed-form factorization of latent semantics in gans. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [57] Shi, Y., Aggarwal, D., Jain, A.K.: Lifting 2d stylegan for 3d-aware face generation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [58] Shoshan, A., Bhonker, N., Kviatkovsky, I., Medioni, G.: Gan-control: Explicitly controllable gans. In: International Conference on Computer Vision (ICCV) (2021)
  * [59] Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [60] Sun, J., Wang, X., Zhang, Y., Li, X., Zhang, Q., Liu, Y., Wang, J.: Fenerf: Face editing in neural radiance fields. arXiv (2021)
  * [61] Szabó, A., Meishvili, G., Favaro, P.: Unsupervised generative 3d shape learning from natural images. arXiv (2019)
  * [62] Tewari, A., Elgharib, M., Bernard, F., Seidel, H.P., Pérez, P., Zollhöfer, M., Theobalt, C.: Pie: Portrait image embedding for semantic control. ACM Transactions on Graphics (TOG) (2020)
  * [63] Tewari, A., Elgharib, M., Bharaj, G., Bernard, F., Seidel, H.P., Perez, P., Zollhofer, M., Theobalt, C.: Stylerig: Rigging stylegan for 3d control over portrait images. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [64] Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG) (2021)
  * [65] Wu, S., Rupprecht, C., Vedaldi, A.: Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [66] Xue, Y., Li, Y., Singh, K.K., Lee, Y.J.: GIRAFFE HD: A high-resolution 3d-aware generative model. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [67] Yao, X., Newson, A., Gousseau, Y., Hellier, P.: A latent transformer for disentangled face editing in images and videos. In: International Conference on Computer Vision (ICCV) (2021)
  * [68] Yin, X., Yu, X., Sohn, K., Liu, X., Chandraker, M.: Towards large-pose face frontalization in the wild. In: International Conference on Computer Vision (ICCV) (2017)
  * [69] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [70] Zhang, Y., Chen, W., Ling, H., Gao, J., Zhang, Y., Torralba, A., Fidler, S.: Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv (2020)
  * [71] Zhou, H., Liu, J., Liu, Z., Liu, Y., Wang, X.: Rotate-and-render: Unsupervised photorealistic face rotation from single-view images. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [72] Zhou, P., Xie, L., Ni, B., Tian, Q.: Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis. arXiv (2021)
  * [73] Zhu, J., Shen, Y., Zhao, D., Zhou, B.: In-domain gan inversion for real image editing. In: European Conference on Computer Vision (ECCV) (2020)
  * [74] Zhu, J.Y., Zhang, Z., Zhang, C., Wu, J., Torralba, A., Tenenbaum, J.B., Freeman, W.T.: Visual Object Networks: Image generation with disentangled 3D representations. In: Advances in Neural Information Processing Systems (NeurIPS) (2018)
  * [75] Zhu, X., Liu, X., Lei, Z., Li, S.Z.: Face alignment in full pose range: A 3d total solution. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2017)

Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable
Portrait Image Synthesis
-Supplementary material-

##  6 Appendix: SURF-GAN

In this section, we supplement contents that are not covered in the main
paper, i.e., additional details, experiments and discussion about the proposed
SURF-GAN.

###  6.1 Additional implementation details

####  6.1.1 Training details.

The maximum resolution of SURF-GAN, as well as $\pi$-GAN [9], which allows
stable learning is $64^{2}$ in our setting, it is because 3D-aware GANs
(especially pure NeRF-based GAN) require computationally expensive resources
for training. Following $\pi$-GAN, we adopt the progressive growing strategy
that the size of the generated image increases progressively. Unlike 2D GANs,
the generator does not actually “grow”. Instead, the number of sampled rays
increases. Because NeRF-based model can be seen as an implicit continuous
function, thus it is theoretically possible to generate arbitrary resolution
images. Therefore, only the discriminator adds new layers at each stage to
handle higher resolutions. We start training at $32^{2}$ and it is doubled at
the next stage. In training phase, the control parameter $\mathbf{z}$ is
sampled from the standard normal distribution. The camera pose (pitch and yaw)
are sampled from the approximated distribution (normal distribution) of
dataset. We assume a perspective pinhole camera where the field of view (FOV)
is $12^{\circ}$. The number of sampled points in each ray is 24 (12 from
coarse sampling and 12 from hierarchical sampling). We exploit non-saturating
GAN loss with R1 penalty [40] following $\pi$-GAN. In addition, there is
orthogonal regularization of basis ($\mathcal{L}_{\text{reg}}$) as explained
in Sec. 3.1. Finally, pose loss is adopted optionally on different purposes we
discuss it in Sec. 6.4. We adopt ADAM [33] optimizer with $\beta_{1}=0$ and
$\beta_{2}=0.99$, and the learning rate is set initially to 0.0001 and it is
halved in the next stage.

####  6.1.2 SURF-GAN architecture.

The architecture of SURF-GAN generator is illustrated in Fig. 13. The
discriminator is same with that of $\pi$-GAN [9] except the last layer.
Besides the adversarial term that distinguishes real or fake, there is an
additional branch that predicts the pose (i.e., pitch and yaw) of an input
image. This branch is utilized if the pose loss is adopted, otherwise it is
discarded (same as $\pi$-GAN).

Figure 13: Details of SURF-GAN generator.

###  6.2 Comparison with $\pi$-GAN

We present the comparison results of SURF-GAN with its baseline, $\pi$-GAN
(Table. 2). Both approaches belong to pure NeRF-GAN, which consists of NeRF
networks without following 2D layers. We evaluate FID score [23], pose
accuracy, and multi-view consistency of each method. We compute FID score
between 50k of generated images and 70k of real images in each dataset. Off-
the-shelf 3D model (3DDFA [75]) is utilized to evaluate pose accuracy. The
reported pose error (Pose err.) is calculated by averaging the difference
between target poses and predicted estimated poses. Multi-view consistency
(ID) is evaluated by calculating cosine similarity between canonical view
image and others from ArcFace [12]. Although $\pi$-GAN shows slightly better
results in the pose accuracy of CelebA and runtime, SURF-GAN delivers
competitive and superior results. For both models, the increased pose error in
CelebA is expected to be due to an alignment issue.

Table 2: Quantitative comparison of SURF-GAN and $\pi$-GAN. Each method is trained on $64^{2}$ images and the test is conducted on rendered $128^{2}$ images. Method |  $\pi$-GAN | SURF-GAN
---|---|---
Dataset |  CelebA |  FFHQ |  CelebA |  FFHQ
FID ($\downarrow$) | $29.54$ | 47.12 | 28.88 | $\mathbf{44.56}$
Pose err.($\times$$10^{-2})$ ($\downarrow$) | 5.81 | 3.35 | 6.10 | 2.69
ID ($\uparrow$) | 0.65 | 0.63 | 0.68 | 0.66
Runtime ($\downarrow$) | 0.10 | 0.11

###  6.3 Additional discovered attributes by SURF-GAN

We present the additional attributes in CelebA dataset which are not
introduced in the main paper (Fig. 14). Note that “Field of view” is not an
discovered attribute, but can be controlled in volume rendering. We also
report the semantic attributes of FFHQ in Fig. 15.

Figure 14: Additional attributes discovered by SURF-GAN which are not
presented in the main paper (CelebA). Figure 15: Semantic attributes
discovered by SURF-GAN when using FFHQ dataset.

###  6.4 Discussion

####  6.4.1 Effect of the bottom noise.

In addition to the layer-wise latent $\mathbf{z}$, our generator also takes
the bottom noise $\mathbf{\epsilon}$ as an additional input to capture missing
variations (Sec. 3.1). Therefore, the intended role for $\mathbf{\epsilon}$ is
to capture the minor variations that have less or not semantic meaning but
enhance diversity. Fig. 16 presents generation results when changing only
$\mathbf{\epsilon}$. As can be seen, the generator synthesizes the images with
minor variation while preserving facial identity.

Figure 16: The bottom noise $\epsilon$ captures subtle variations ($64\times
64$ images).

####  6.4.2 Effect of the progressive growing.

As mentioned in Sec. 6.1, we adopt the progressive growing for training. To
demonstrate the effectiveness of the strategy, we report FID score of the
variants of our method (i.e., w./ and w.o./ progressive growing) for every
1000 iterations. As can be seen the FID curve in Fig. 17, there is a gap
between two variants.

Figure 17: FID curve of two variants of SURF-GAN, i.e., with and without
progressive growing on CelebA (64$\times$64).

####  6.4.3 Effect of the pose loss.

To improve the pose accuracy, we additionally adopt pose loss
$\mathcal{L}_{\text{pose}}$ for training and compare with the original SURF-
GAN (w.o./ $\mathcal{L}_{\text{pose}}$). $\mathcal{L}_{\text{pose}}$ is
calculated as the difference between input viewing directions of generator and
those predicted by the discriminator. It is not an adversarial loss, thus both
the generator and discriminator learn to minimize the loss. The results are
listed in Table. 3. The introduction of $\mathcal{L}_{\text{pose}}$ reduces
pose error (Pose err.), but sacrifices the visual quality. We exploit this
model (w./ $\mathcal{L}_{\text{pose}}$) for training 3D-controllable StyleGAN
(Sec. 3.2) to offer more elaborate pose samples.

Table 3: Ablation study for training SURF-GAN with and without the pose loss on FFHQ (128$\times$128). |  w.o./ $\mathcal{L}_{\text{pose}}$ |  w./$\mathcal{L}_{\text{pose}}$
---|---|---
FID ($\downarrow$) | 44.56 | 45.92
Pose err.($\times$$10^{-2})$ ($\downarrow$) | 2.69 | 2.36

###  6.5 Limitation.

Although SURF-GAN has several clear advantages such as controllability, there
are still inherent limitation as like other 3D-aware GANs. In our model, the
color and density of all the points in the rays are calculated independently,
thus the amount of computation required to synthesize images increases
exponentially as the resolution increases. Such issue has been the catalyst
for introducing a method of injecting the prior of SURF-GAN into an efficient
and expressive StyleGAN2 generator [32]. It will be one of our future work to
achieve high-resolution with clever and efficient ways, e.g., adopting 2D
modules [8, 17, 46, 14]in SURF-GAN generator. The other minor limitation is
that the same layer does not always capture the same properties for each
training. For example, even if a specific dimension (e.g., $L3D2$) of the
trained SURF-GAN captures hair color, the same dimension might capture
different attributes in the newly trained model. It seems natural because our
method is based on unsupervised training, but it makes the process of
assigning properties of each layer necessary after training.

##  7 Appendix: 3D-controllable StyleGAN

This section presents additional experiments and discussion about 3D
controllable StyleGAN.

###  7.1 Implementation

####  7.1.1 Latent mapper.

The latent mapper consists of five FC layers. It takes an input vector in
$\mathcal{W}+$ space and converts it to a canonical vector with the same size.
However, the latent mapper does not edit all elements of $18\times 512$
vector, but edits first four style vectors which have known to related to pose
[31, 70] (Sec. 4.1), i.e., the input size of the latent mapper is $4\times
512$. Input feature is firstly flatten to 2048-dimensional vector and then
converted to intermediate feature $\in\mathbb{R}^{512}$. After going through
three intermediate layers, the feature is converted to the canonical vector
$\in\mathbb{R}^{4\times{512}}$ in the last layer.

####  7.1.2 Training details.

We leverage SURF-GAN as a multi-view image generator to train 3D-controllable
StyleGAN. As described in Sec. 3.2, the poses of source and target images are
randomly sampled from pre-defined distribution. In order to train diverse pose
angles, we sample pitch and yaw from uniform distributions instead of Gaussian
distribution, i.e., the value of pitch and yaw are uniformly sampled from
$[-30^{\circ},30^{\circ}]$ and $[-45^{\circ},45^{\circ}]$, respectively. The
resolution of the rendered images is $256^{2}$ that is same with input size of
pSp encoder [52]. We use a pretrained StyleGAN2 ($1024^{2}$) generator for
experiments except for MetFace [28] stylization (here we use a $256^{2}$
generator for transfer learning).

###  7.2 Discussion

####  7.2.1 Sub-directions.

Figure 18: Example of non-linearity in pose-related vectors. The blue numbers
below the figures indicate the detected poses.

To demonstrate the effectiveness of exploiting orthogonal directions (sub-
directions) described in Sec. 3.2, we introduce an interpolation example in
Fig. 18. Among learned sub-directions, we select two directions
$\overrightarrow{d_{1}}$ and $\overrightarrow{d_{2}}$ , where both vectors
control yaw. As can be seen in the left side of Fig. 18, they influence almost
similarly in small pose variations. However, they shows different
interpolation outputs when checking the results by scaling both vectors. It
means $\overrightarrow{d_{1}}$ is more involved than $\overrightarrow{d_{2}}$
for generation of images with large pose variations.

####  7.2.2 SURF-GAN-generated images generalization.

Table 4: Results of the identity similarity between two decoded images and difference of the estimated poses between SURF-GAN image and the corresponding image decoded by StyleGAN. | SURF-GAN |  Decoded.
---|---|---
ID | 0.66 | 0.73
Pose diff. | 0.003

We train the latent mapper and the learnable directions using SURF-GAN. The
objective function is calculated with the images decoded by StyleGAN. However
the question that may arise here is “Can SURF-GAN-generated images be
generalized to the training process?”. To answer the question, we conduct
simple experiments. First, we measure the cosine similarity of two decoded
images at different pose angles using ArcFace [12] , and also evaluate how
much the pose changes in the decoded image using off-the-shelf pose detector.
[75]. The former and the latter are for checking whether identity and pose are
maintained, respectively. Although there is a domain gap between SURF-GAN and
StyleGAN, the two images with the same identity in SURF-GAN domain maintain
the same identity in StyleGAN domain as can be seen Table. 4. Moreover, the
pose of the SURF-GAN-generated image is hardly changed by the GAN inversion or
decoding.

####  7.2.3 Extreme cases.

Figure 19: Experimental results on extreme poses beyond the range for
training.

As mentioned in Sec. 7.1, we set the poses for training within a certain range
because there are few images with extreme poses in the FFHQ dataset.
Nevertheless, we validated the extreme case by giving a large value beyond the
pose range as input and observed that there are some cases where plausible
images are obtained as shown in Fig. 19.

###  7.3 Additional comparison results

In this subsection, we supplement extra experimental results and discussion to
demonstrate the effectiveness of our method.

####  7.3.1 3D-controllable image synthesis.

Figure 20: Additional qualitative comparison with 3D-controllable generative
models.

We first present the additional qualitative comparison with 3D controllable
generative models compared in Sec. 4.3. Here we add one more baseline,
DiscoFaceGAN [13] which is based on 3DMM. Although DiscoFaceGAN does not allow
explicit control over the camera pose, it can be implicitly manipulated by an
input latent. Therefore, we display the interpolated images by appropriately
adjusting the angles of the images at both ends.

####  7.3.2 Novel view synthesis.

Figure 21: Additional qualitative comparison with methods which are capable of
novel-view synthesis from a single portrait image.

We describe the details not covered in Sec. 4.4 and also present additional
qualitative comparison with the competing methods [9, 34, 71] for novel view
synthesis (Fig. 21). For all methods, we use the official implementations
provided the authors.

$\pi$-GAN leverages a latent optimization method [32] to overfit the latent
code to the testing image. $\pi$-GAN is a 3D-aware generator and learns 3D
geometry from unlabelled 2D images without 3D supervision. However, when it is
applied to novel view synthesis, $\pi$-GAN needs camera extrinsics from the
testing image to initiate the following iterative optimization (700
iterations). For the camera pose, we exploit off-the-shelf pose detection
method [18]. As shown in Fig. 21, the visual quality deteriorates as it
deviates from the original pose. It is difficult to generate the radiance
field of the target image only with latent code optimization and a small error
in the pose estimation greatly affects the result. In addition, it takes a lot
of time (164 sec.) to get results for a single image due to the iterative
optimization. This is why we excluded $\pi$-gan from the quantitative
experiment in Sec. 4.4.

For ConfigNet [34], the real data encoder firstly predicts the latent
embedding of a testing image, and then fine-tunes the generator on the image
(50 iterations). To handle a real image, it requires pre-processing to align
facial images using landmarks from OpenFace [6]. Although it shows an
overwhelming runtime in random image generation compared to other methods
(Sec. 4.3), the runtime drops significantly due to the introduction of the
face detection model in novel view synthesis (Sec. 4.4). Note that the
reported runtime in Sec. 4.4 (2.13 sec.) does not include the fine-tuning
procedure. The whole process takes about 11.25 seconds (9.12 sec. for fine-
tuning). Furthermore, ConfigNet struggles to synthesize images with large pose
changes.

Rotate-and-Render (R&R) is a face rotation method using off-the-shelf 3D
fitting network [75] in the overall model, thus it takes some time for 3D
fitting (Sec. 4.4). R&R successfully generates a novel view compared to the
previously described methods. However, it loses some details of the original
image such as hair or background.

####  7.3.3 Comparison with latent-based models.

The pose editing of our method is based on latent manipulation. We introduce
comparison results with the existing latent-based method [55] that discovers
pose-related direction in the latent space of StyleGAN (Fig. 22). Although
InterFaceGAN [55] successfully disentangles the pose attribute, it requires
supervision (landmark) for binary classification of yaw in order to find a
semantic hyperplane. As Tov et al. [64] have investigated, the results of pose
editing with the $\mathcal{W}+$ vectors inverted by pSp [52] shows poor
editability. It is alleviated by exploiting e4e encoder [64], but the identity
of the input is not well maintained. Above all, the important limitation of
latent-based models as well as InterFaceGAN is that they only allow implicit
control over pose. Although it is not unfeasible to generate a target pose
using these methods, the process might require a few adjustments to obtain an
accurate result. Nitzan et al. [45] have introduced the latent-based linear
regression method by showing yaw angle has a linear relationship with the
distance from InterFaceGAN’s yaw hyperplane. Nevertheless, the linearity is
not always guaranteed (Fig. 18), and obtaining the hyperplane requires
supervision as mentioned above. There may be an clever alternative to acquire
the hyperplane without supervision by leveraging the concept of flipping image
[65], but it can be applied only to yaw, not other properties such as pitch or
field of view.

Figure 22: Comparison with an existing latent manipulation method
(InterFaceGAN).

###  7.4 Semantic attribute editing

In Sec. 4.5, we presented the results of semantic attribute editing using
SURF-GAN, where the direction was calculated by subtracting two inverted SURF-
GAN images using pSp encoder [52]. However, there is a trade-off between
distortion and editability as Tov et al. [64] demonstrated. As a result, some
local attributes in Sec. 4.5 are not changed successfully. Although there
might be an effect that we use simple vector arithmetic, the main reason is
that the $\mathcal{W}+$ space shows weak editability. To address the issue, we
investigate the editing results when using e4e [64] encoder to calculate the
direction vector and obtain plausible editing results as shown in Fig. 23.
Note that SURF-GAN samples in Fig. 14 and Fig. 15 are utilized for calculating
the directions.

Figure 23: Editing results of semantic attributes by calculating the direction
vector with images generated by SURF-GAN. Figure 24: Failure cases of our
method (red box). For FOV control, we additionally used FOV as a conditional
input for training as well as pitch and yaw.

###  7.5 Limitation

Although StyleGAN can generate diverse portrait images with high quality, it
struggles to generate out-of-distribution images that do not appear in
dataset. Therefore, our method also cannot generate those images because our
method does not deviate the latent space of StyleGAN. In addition, our method
is also affected by the performance of GAN inversion, thus the performance of
our model is not guaranteed for images where the inversion method does not
work well. We select several failure cases of our method in Fig. 24. The other
limitation is that as like other pose-disentanlged GANs, our method is not
capable of generating 3D representations (e.g., mesh or radiance field).
Hence, when it comes to video generation, it shows the problem of “texture
sticking” [30] (especially in hair) which is one of the most obvious artifacts
in GAN generated videos.