# Ray Conditioning: Trading Photo-consistency for Photo-realism in Multi-view
Image Generation

Eric Ming Chen1 Sidhanth Holalkere1 Ruyu Yan1 Kai Zhang2 Abe Davis1
1Cornell University 2Adobe Research
https://ray-cond.github.io

###### Abstract

Multi-view image generation attracts particular attention these days due to
its promising 3D-related applications, e.g., image viewpoint editing. Most
existing methods follow a paradigm where a 3D representation is first
synthesized, and then rendered into 2D images to ensure photo-consistency
across viewpoints. However, such explicit bias for photo-consistency
sacrifices photo-realism, causing geometry artifacts and loss of fine-scale
details when these methods are applied to edit real images. To address this
issue, we propose ray conditioning, a geometry-free alternative that relaxes
the photo-consistency constraint. Our method generates multi-view images by
conditioning a 2D GAN on a light field prior. With explicit viewpoint control,
state-of-the-art photo-realism and identity consistency, our method is
particularly suited for the viewpoint editing task.

##  1 Introduction

Modeling the distributions of natural images has long been an important
problem that is extensively studied. Generative adversarial networks (GANs)
and diffusion models are two types of generative models that have successfully
shown impressive capabilities of learning image distributions—the generated
samples are almost indistinguishable from real photos [10, 35, 16, 36].

While optimizing for the photo-realism of individual samples, these generative
models rarely allow for multi-view image generation, where photo-consistency
matters. Recently, many multi-view image synthesizers have been proposed that
try to optimize for both photo-realism and photo-consistency [5, 43, 12, 27].
They generally follow a “synthesize-3D-then-render” paradigm: 3D
representations are synthesized, and then images are rendered (at specified
camera poses). Such 3D-aware generative models, especially EG3D [5], achieve
high-quality multi-view image generation results, despite being trained only
on single-view image data with poses.

Pose Cond. [5] |
---|---
EG3D [5] |
Ray Cond. |
|  Input Inversion Rotation

Figure 1: Challenges With Viewpoint Editing. Naively conditioning a 2D GAN’s
latent space on pose does not ensure that an identity remains consistent
across views. 3D-aware GANs such as EG3D [5] struggle to reconstruct high-
frequency details such as wrinkles and hair. Our ray conditioning method most
faithfully reproduces the input image, and preserves identity when editing the
viewpoint.  Figure 2: Viewpoint Editing with Ray Conditioning. Ray
conditioning enables photo-realistic multi-view image editing on natural
photos via GAN inversion. The left half shows headshots (first column) of four
individuals and their corresponding synthesized results from another viewpoint
(second and third columns). The right half shows a portrait of two individuals
(top row), the GAN inversion results of their faces (top row corners), and the
resulting image (bottom row), in which their faces are replaced with
synthesized faces looking in a different direction (bottom row corners). To
produce the latter, we used Photoshop to blend the synthesized faces with the
original image.

However, photo-realism and photo-consistency are oftentimes two competing
goals: photo-realism very explicitly favors image quality over control, while
photo-consistency more implicitly favors control over quality. A few examples
of this conflict include fine-scale detail and view-dependent appearance in
the multi-view 3D reconstruction and 3D generation problems. Details are
either filtered from 3D representations (leaving them smoother and more
diffuse than real ones) or misinterpreted as geometric artifacts [41]. As
shown in Figure 1, although EG3D allows explicit camera control and generates
photo-consistent images at different viewpoints, it fails to reproduce the
subtle details, e.g., the wrinkles and hairs, in the input image. Naively
conditioning a 2D GAN’s latent space on camera pose does not ensure that the
identity remains consistent across views.

Our work is motivated by the observation that, for certain classes of images
with shared canonical structure, e.g. faces, it is possible to achieve
viewpoint control without optimizing explicitly for 3D structure. The result
is a modified 2D GAN that offers precise control over generated viewpoints
without sacrificing photo-realism. Furthermore, we are able to train on data
that does not contain multiple viewpoints of any single subject, letting us
leverage the same diverse and abundant data used for regular GANs. Our method
combines the photo-realism of existing GANs with the control offered by
geometric models, outperforming related methods in both generation and
inversion quality. This makes it particularly well-suited for viewpoint
editing in static images.

Key to our method is the proposed ray conditioning mechanism that conditions
the image generator on the ray bundle of a camera for explicit viewpoint
control. The method is simple. Rather than using a 3D model, ray conditioning
conditions each pixel in a generated image on a light field prior [25, 11],
the set of all rays flowing through a scene. The spatial inductive bias of ray
conditioning enables the image synthesizer to learn multi-view consistency
from only single-view posed image collections. By choosing a geometry-free
approach, we relax the 3D photo-consistency constraints in exchange for
increased photo-realism. Despite this, our approach still offers competitive
identity preservation capability when editing viewpoints. Figure 2 represents
the quality and control we can achieve with ray conditioning. Evaluation on
both single-view and multi-view data also shows that our method is a
significant improvement in image quality over Light Field Networks (LFNs),
another geometry-free image synthesizer [33], demonstrating the promising
potential of this line of research.

In summary, our contributions are as follows.

  1. 1.

We propose a simple yet effective geometry-free generative model named ray
conditioning for multi-view image generation, and show that it achieves
greater photo-realism than geometry-based baselines while maintaining explicit
control of viewpoints.

  2. 2.

We demonstrate the superiority of our method in the downstream application of
editing real images’ viewpoints where photo-realism is favored over photo-
consistency.

  3. 3.

Our ray conditioning method is also the first geometry-free multi-view image
synthesizer that can generate highly realistic images in high resolution
($1024\times 1024$) given only single-view posed image collections.

##  2 Related Work

Image Synthesis and Latent Space Editing. In the past decade, GANs [10] have
revolutionized image synthesis. They are trained by optimizing two neural
networks, a generator and a discriminator, at the same time. The generator
tries to fool the discriminator by generating fake images that look as real as
possible. The discriminator tries to distinguish the synthetic images from the
real ones. Once training is complete, the generator is able to generate images
that are almost indistinguishable from natural images. Among all the proposed
GAN architectures, for its high image quality, StyleGAN [21, 22, 20] is
perhaps the most widely adopted. We base our method on StyleGAN2, and enable
explicit viewpoint control via ray conditioning.

Several works have found that the latent spaces of StyleGAN are remarkably
linear [18, 31, 17, 39], disentangling attributes such as facial expression,
hair color, and pose. Pose disentanglement is of particular interest to the
domain of this work, because it serves as a proxy for 3D information. Related
work have utilized this property of 2D GANs to discover visual correspondences
between images [28]. However, properly harnessing this ability can be non-
trivial. In a latent space, editing directions often have to be found ad-hoc
for each dataset, and lack a intuitive interpretation [31]. In contrast, our
method allows for interpretable and explicit control of viewpoints.

Multi-view Image Synthesis. Our work is closely related to the direction of
multi-view image synthesis, where prior work can be roughly classified into
two categories: geometry-based and geometry-free.

Geometry-based multi-view image generation approaches typically adapt 2D GAN-
based image synthesizers in a way that a neural 3D representation is first
generated, from which 2D images are then rendered using neural rendering.
Representative work in this category include EG3D [5], GMPI [43], StyleSDF
[27], StyleNeRF [12], GIRAFFE [26], $\pi$-GAN [4], etc. These methods mainly
differ from each other in choice of the 3D representation and rendering
algorithm. For instance, EG3D adapts the StyleGAN2 generator to predict a
feature volume in a compact triplane representation. They then use volume
rendering to render a feature map, which is later decoded into a high-
resolution image through a convolutional decoder; on the other hand, GMPI
generates a multiplane image RGBA representation [44] and renders it using
homography warping and alpha compositing. These geometry-based methods have
demonstrated impressive multi-view image synthesis quality given only single-
view posed data; the generated images are very view-consistent and detailed.
However, we observe that the synthesize-3D-then-render approach they adopt
indeed trades some photo-realism for photo-consistency, as shown in Figure 1.
Moreover, as noted by concurrent work [40], the geometry prior in an image
synthesizer also increases the difficulty of inverting a real posed image.
Both issues sacrifice the methods’ performance in viewpoint editing for real
posed images. We seek to circumvent these issues by optimizing for photo-
realism and easy invertibility.

Figure 3: The Ray Conditioning Method. The StyleGAN synthesis network
progressively convolves and upscales a low-resolution feature map into a high-
resolution one. To condition the generator on a camera, we concatenate these
feature maps with an appropriately downsampled Plücker embedding of the
sampled camera parameters. By doing so, the GAN learns to associate camera
rays with appearance.

Geometry-free methods traditionally learn view consistency priors by training
image synthesizers on large multi-view datasets rather than using a 3D
representation. LFNs [33] and 3DiM [37] are two successful methods that have
inspired our approach. LFNs represents each scene as a light field
parametrized by a multilayer perceptron (MLP) which maps a ray to a color. For
generation, the MLP is also conditioned on a randomly-sampled latent code
through meta-learning [13]. However, compared to a convolutional neural
network (CNN), a MLP cannot effectively utilize the inductive bias of spatial
smoothness in natural images. The generated results are oftentimes blurry
compared to CNN-based image synthesizers. 3DiM uses a more powerful image
generator—a diffusion model [35, 16, 36], and achieve state-of-the-art single-
view novel view synthesis results on the ShapeNet dataset [6]. At their core
is a pose-conditional image-to-image diffusion model trained using ground-
truth multi-view images as supervision. Being a conditional image synthesizer,
3DiM cannot perform unconditional multi-view image generation. While both
works have made strong methodological contributions, neither has been proven
to generate results at a resolution higher than $128\times 128$, to learn
without multi-view datasets, nor to learn over photo-realistic images. In
comparison, we show that our method (ray conditioning and a CNN synthesizer)
can be trained with only single-view posed data at $1024\times 1024$
resolution, and can perform even better than geometry-based methods on
practical viewpoint-editing applications.

##  3 Method

Like prior work on 3D-aware GANs [5], we focus on unstructured single-view
image collections of objects from the same category, e.g., human faces, with
labeled camera poses. Namely, we focus on a set of
$({\mathbf{I}},{\mathbf{K}},{\mathbf{E}})$ triples, where ${\mathbf{I}}$ is an
image, ${\mathbf{K}}$ are its corresponding intrinsics, and
${\mathbf{E}}=[{\mathbf{R}}\mid{\mathbf{t}}]$ are its corresponding camera
extrinsics (camera-to-world transformations). For the sake of downstream
applications such as portrait reposing, we seek to train a GAN that allows us
to explicitly control the viewpoints of the synthesized images—without
explicitly modeling the geometry. We accomplish this by adding our proposed
ray-conditioning mechanism to an off-the-shelf image generator: StyleGAN2
[22]. Our method requires minimal modifications to the image generator’s
architecture, while achieving higher photo-realism than methods that use a 3D
representation [5, 43, 27, 12].

###  3.1 Photo Collections as Unstructured Light Fields

Images are the fundamental medium we use to represent our physical world. They
represent a sample of a scene’s 5D plenoptic function,
$L:(\mathbf{p},\mathbf{d})\mapsto\mathbf{c}$, which describes the light
intensity $\mathbf{c}$ in an arbitrary direction
$\mathbf{d}\in\mathbb{S}^{2}$, and at arbitrary location
$\mathbf{p}\in\mathbb{R}^{3}$ [2]. This 5D plenoptic function becomes 4D once
we constrain the scene to be within a convex hull, and assume zero light decay
in empty space. In this case, the 4D plenoptic function is also called the 4D
light field [25, 11]. A pixel $(u,v)$ of an image ${\mathbf{I}}$ can then be
interpreted as a sample of the light through a ray direction:

$\displaystyle\mathbf{d}_{u,v}=\mathbf{R}\mathbf{K}^{-1}\begin{bmatrix}u,v,1\end{bmatrix}^{T}+{\mathbf{t}},$
(1)
$\displaystyle\mathbf{d}_{u,v}=\mathbf{d}_{u,v}\big{/}\lVert\mathbf{d}_{u,v}\rVert_{2},$
(2) $\displaystyle u,v\in[0,W)\times[0,H),$ (3)

where $W,H$ are ${\mathbf{I}}$’s width and height. A perspective image is then
a measurement of the 4D light field at the location of the camera origin,
i.e., $\mathbf{p}=\mathbf{o}$, from a bundle of ray directions
$\\{\mathbf{d}_{u,v}\\}$ falling inside the camera’s field-of-view:
${\mathbf{I}}=L(\mathbf{o},\\{\mathbf{d}_{u,v}\\})$.

Typically, the light field of a single scene is measured using a dense grid of
synchronized cameras [38]. Novel views can then be synthesized by
interpolating these densely captured images. However, for a photo collection
with only one image per scene, such as one of faces, the light field
measurements are highly unstructured. Due to the varying identities,
expressions etc, each image can be thought of as being a single-shot sampling
of its scene’s light field. Another way to put it is to imagine we have used a
synchronized camera grid to capture several light fields for different
subjects, but then we accidentally lost access to all the images except only
one randomly-chosen image per light field. More specifically, we are left with
an unstructured collection of light field observations
$\big{\\{}(\mathbf{o},\\{\mathbf{d}_{u,v}\\},\mathbf{I}\big{)}\\}$ from a
diverse set of scenes. The task of generative modeling is then to model a
distribution of light field observations over an entire photo collection to
picture what those missing view points may have looked like.

###  3.2 Ray Conditioning for Image Synthesis

Given a posed image collection
$\big{\\{}(\mathbf{o},\\{\mathbf{d}_{u,v}\\},\mathbf{I}\big{)}\\}$ that
measure light fields in a highly unstructured way, we aim to learn the
distribution of a light field $L$ defined on ray bundles
$\big{\\{}(\mathbf{o},\\{\mathbf{d}_{u,v}\\})\big{\\}}$. We only consider ray
bundles that follow the same distribution as all the input ray bundles; hence
$L$ is not well-defined for out-of-distribution ray bundles, as shown in
Figure 8.

We add ray conditioning to a GAN to model the distribution of a light field
$L$ implicitly: the generator maps a Gaussian-distributed noise code
$\mathbf{z}$ to an image sample
$L_{{\mathbf{z}}}(\mathbf{o},\\{\mathbf{d}_{u,v}\\})$ from the light field
$L_{{\mathbf{z}}}$. By fixing the noise code $\mathbf{z}$, and using different
ray bundles $\mathbf{o},\\{\mathbf{d}_{u,v}\\}$, we can sample images at
different viewpoints from the same learnt light field. Concretely, we use the
StyleGAN2 backbone, as do the prior EG3D work [5]. As shown in Figure 3, we
add ray conditioning to each level of the progressively-growing synthesis
network. At each level, we first downsample the spatial ray embedding to the
same resolution as the feature map, and then concatenate the ray embedding and
feature map together along the channel dimension.

EG3D [5] |
---|---
GMPI [43] |
Ray Cond. |
EG3D [5] |
GMPI [43] |
Ray Cond. |

Input Inversion Rotation

Figure 4: Viewpoint Editing via GAN Inversion. We invert an image into each
GAN using PTI [29]. Ray conditioning is able to best preserve the details of
the input images, as shown in the rich detail of the skin, eyes, and hair.
Because radiance fields are biased towards low frequency results, the EG3D [5]
inversions lack the detail that ray conditioning can offer. In the second
example, there are also geometry artifacts near the ears and border of the
face. GMPI [43] struggles to invert input images, causing distortion in the
novel views.

The ray embedding needs to be carefully chosen: the naive 5D ray
parametrization $(\mathbf{o},\mathbf{d})$ is redundant for a 4D light field as
it fails to consider the assumption of zero decay in empty space:
$L(\mathbf{o},\mathbf{d})=L(\mathbf{o}+t\mathbf{d},\mathbf{d})$. Inspired by
LFNs [33], we remove this redundancy through the Plücker parametrization
${\mathbf{r}}=(\mathbf{o}\times\mathbf{d},\mathbf{d})$, where $\times$ denotes
vector cross-product. With this parametrization, we have:

$\left({\mathbf{o}}+t{\mathbf{d}}\right)\times{\mathbf{d}}={\mathbf{o}}\times{\mathbf{d}}+t{\mathbf{d}}\times{\mathbf{d}}={\mathbf{o}}\times{\mathbf{d}}.$
(4)

Thanks to the simplicity of our ray conditioning method, we require minimal
modifications to the backbone StyleGAN2 architecture: each convolution kernel
just needs to accept extra ray embedding inputs. Hence the induced
computational overhead is almost negligible. Moreover, we can start from a
pretrained StyleGAN2 model, and finetune it to make it amenable to explicit
viewpoint control. Unlike prior works that prioritize photo-consistency over
photo-realism, our method maintains the high image generation fidelity of
StyleGAN2, as shown in Figure 5.

###  3.3 Viewpoint Editing for Real Posed Images

For generated images, viewpoints can be easily achieved by changing ray
bundles $\mathbf{o},\\{\mathbf{d}_{u,v}\\}$ in our generated light field
$L_{{\mathbf{z}}}(\mathbf{o},\\{\mathbf{d}_{u,v}\\})$. Like prior work in
StyleGAN-based real image editing, we can invert a real posed image
$(\mathbf{o},\\{\mathbf{d}_{u,v}\\},\mathbf{I})$ into a latent space of
StyleGAN first, and then modify the ray bundles to edit the viewpoint.

As our method closely resembles the backbone StyleGAN2, we can directly use
off-the-shelf GAN inversion methods [29] for high-quality inversions and
viewpoint edits. This is in stark contrast to geometry-based methods like EG3D
that require more dedicated inversion methods, as shown by concurrent work
[40] and Figure 4.

We also differ from prior latent-space viewpoint editing work in terms of
offering intuitive explicit viewpoint control and increased viewpoint change
range. We provide results in the supplementary material. In addition, those
latent-space editing directions require paired training data [31, 1]. One such
method, InterfaceGAN [31], relies on an external binary classifier to
determine whether a generated face is facing left or right. InterfaceGAN also
does not allow for explicit control of pose, and relies on manual tuning to
achieve the desired pose.

Pose Cond. [5] |  |  EG3D [5] |
---|---|---|---
|  |  |
GMPI [43] |  |  Ray Cond. |
|  |  |
|  $-45^{\circ}$ $-23^{\circ}$ $0^{\circ}$ $+23^{\circ}$ $+45^{\circ}$ |  |  $-45^{\circ}$ $-23^{\circ}$ $0^{\circ}$ $+23^{\circ}$ $+45^{\circ}$
Figure 5: Unconditional Multi-view Image Generation. As reflected in the
photos, our method is able to maintain identity consistency and photorealism
across angles. Pose conditioning fails to be view consistent. By foregoing a
3D model, we do not have the geometric artifacts that GMPI [43] and EG3D [5]
may have. Although there are some view-dependent changes between images, we
achieve the highest image quality at steep angles. All results are generated
with $\psi_{\text{trunc}}=0.7$.

##  4 Experiments

We validate our approach by testing its ability to generate multi-view images
on two single-view posed datasets: Flickr-Faces-HQ (FFHQ) [21] and AFHQv2 Cat
Faces [7]. We show that our geometry-free approach outperforms geometry-based
baselines in terms of photo-realism when generating multi-view images. We also
demonstrate our model’s superiority for downstream viewpoint editing of real
images. Finally, we compare our method with a prior geometry-free method, LFNs
[33], on a multi-view posed dataset: SRN Cars [34, 6], and show significant
improvement in multi-view generation quality.

###  4.1 Metrics

We adopt the same metrics used in the prior EG3D work [5] for quantitative
evaluation.

Image Quality. To compare image generation quality, we report the FID score
[15] and KID score$\times 100$ [3] between 50k generated images and the entire
training set.

Identity Consistency. As a proxy for view consistency, we use ArcFace [8, 30],
a facial recognition model, to compute identity consistency between two random
views of one individual, and average it over 1024 samples.

Pose Accuracy. We sample one camera pose for 1024 individuals and estimate the
camera pose of the synthesized image with Deep 3D Face Reconstruction [9]. We
report the mean squared error between camera angles.

Generation Speed. We benchmark the time for generating an image from a latent
vector and report the generation speed in frames per second (FPS). Generation
speed is critical for GAN inversion, as inversion methods may require hundreds
of queries to the GAN to invert a real image.

###  4.2 Results and Discussion

We report quantitative results comparing ray conditioning to competitive
3D-aware baselines in Table 1, and show samples of generated images in Figure
5. All baseline results except for FPS are quoted from StyleGAN2-ADA [19],
GMPI [43] and EG3D [5]. We also report standard deviations for the ID metric
and pose metric, which were previously not reported. Notably, we are able to
achieve strong results with only a 2D GAN backbone. Because our image quality
is not limited by the resolution of a geometric model, ray conditioning is
able to achieve the highest FID and KID scores on faces, which are most
similar with the scores of StyleGAN2. We are also able to achieve competitive
identity accuracies and pose accuracies, demonstrating our model’s ability to
maintain view consistency and model camera pose.

| FFHQ | AFHQv2 Cats
---|---|---
| FID$\downarrow$ | KID$\downarrow$ | ID $\uparrow$ | Pose$\downarrow$ | FPS$\uparrow$ | FID$\downarrow$ | KID $\downarrow$
Image synth. |  |  |  |  |  |  |
StyleGAN2 [19] $512^{2}$ | - | - | - | - | 69 | $3.55^{\dagger}$ | 0.066†
StyleGAN2 [22] $1024^{2}$ | 2.70 | 0.048 | - | - | 62 | - | -
Geometry-based MV synth. |  |  |  |  |  |  |
StyleSDF [27] $256^{2}$ | 11.5 | 0.370 | - | - | - | $12.8^{*}$ | $0.447^{*}$
StyleNeRF [12] $1024^{2}$ | 8.10 | 0.240 | - | - | - | $14.0^{*}$ | $0.350^{*}$
EG3D [5] $512^{2}$ | 4.70 | 0.132 |  $\mathbf{0.77}$$\pm 0.15$ |  $\mathbf{0.005}$$\pm 0.005$ | 33 | $\mathbf{2.77}^{\dagger}$ | $\mathbf{0.041}^{\dagger}$
GMPI [43] $512^{2}$ | 8.29 | 0.454 |  $\mathbf{0.74}$$\pm 0.16$ |  $\mathbf{0.006}$$\pm 0.009$ | 13 | 7.79 | 0.474
GMPI [43] $1024^{2}$ | 7.50 | 0.407 |  $\mathbf{0.75}$$\pm 0.16$ |  $\mathbf{0.007}$$\pm 0.010$ | $6$ | - | -
Geometry-free MV synth. |  |  |  |  |  |  |
Ray Conditioning $512^{2}$ | $3.50$ | $0.076$ |  $\mathbf{0.75}$$\pm 0.15$ |  $\mathbf{0.006}$$\pm 0.007$ | $\mathbf{48}$ | 3.44 | 0.103
Ray Conditioning $1024^{2}$ | $\mathbf{3.28}$ | $\mathbf{0.066}$ |  $\mathbf{0.76}$$\pm 0.14$ |  $\mathbf{0.006}$$\pm 0.007$ | $\mathbf{38}$ | - | -

Table 1: Multi-view Image Generation Metrics. Ray conditioning enables multi-
view (MV) image synthesis by conditioning a 2D GAN on a ray embedding of a
camera. It achieves high degrees of photorealism, identity consistency, and
pose accuracy. We compare each multi-view GAN method to a StyleGAN2 baseline,
showing the loss of fildelity due to geometric inductive biases. All metrics
except for FPS are quoted from StyleGAN2-ADA [19], EG3D [5] and GMPI [43]. We
also compute and report standard deviations for the ID and pose scores, which
were previously not reported. We bold the best statistically significant
results. *Trained on all of AFHQ instead of the cats subset. †Trained with
adaptive discriminator augmentation [19].

These metrics corroborate what we see in Figure 5. Compared to EG3D and GMPI,
ray conditioning maintains the highest visual quality across yaw changes. GMPI
appears to have lower quality images than EG3D and ray conditioning. As
reported in the GMPI ablation study, we believe that this is due to GMPI’s
synthetic shading, which is necessary to learn accurate depth information. At
yaws of $\pm 23^{\circ}$, all methods do a good job at representing rotation.
However, beyond that, EG3D and GMPI appear to show artifacts. For EG3D, there
is noise near the ears in both examples. For the second individual, there is
an unnatural border near the ears as well. This is a known failure mode of
EG3D and other radiance fields called billboarding, described in Section 1.3
of their appendix. For GMPI, the multiplane images cause noticeable quality
issues in the rendered images. Images appear to be blurry. While ray
conditioning sacrifices some view consistency, such as changes in smile, the
results remain realistic even at difficult yaws. These results provide an
explanation for why ray conditioning is able to achieve the best FID and KID
scores. Forfeiting a 3D representation allows for high quality image synthesis
across angles. The fact that all of the most competitive methods are built
upon the StyleGAN architecture underscores its superior ability to disentangle
pose and appearance. Ray conditioning is a natural extension of StyleGAN for
generating multi-view images.

In Figure 6, we compare ray conditioning to Light Field Networks [33] (LFNs),
a model designed for geometry-free view synthesis. LFNs use an autodecoder to
condition the color of a ray on a normally distributed latent vector. Because
LFNs are trained with a L2 reconstruction loss instead of an adversarial loss,
output images tend to be blurry. Relative to the training dataset, LFNs
achieves an FID score of $41.8$ while ray conditioning achieves an FID score
of $3.39$. The samples from ray conditioning are much sharper, and show more
diversity. We provide videos in the supplementary material. As shown in Figure
7, on FFHQ, LFNs struggles to reconstruct the input data. It is also not able
to synthesize novel views when trained on FFHQ, a single-view dataset. Ray
conditioning demonstrates that light field conditioning concept introduced in
LFNs is capable of synthesizing compelling results on only single-view data.

###  4.3 Ablation Study

We compare ray conditioning to a simpler alternative: pose conditioning.
Similar to StyleGAN2-ADA [19] and EG3D [5], we first flatten the camera
extrinsics ${\mathbf{E}}\in\mathbb{R}^{4\times 4}$ and intrinsics
${\mathbf{K}}\in\mathbb{R}^{3\times 3}$ into a conditioning vector
${\mathbf{c}}\in\mathbb{R}^{25}$. We then input both a randomly sampled
${\mathbf{z}}\sim\mathcal{N}(0,1)^{512}$ and the conditioning vector
${\mathbf{c}}$ into the mapping network for predicting a ${\mathbf{w}}$ code,
as shown in Figure 3. Pose conditioning also provides explicit viewpoint
control; however, it encodes a viewpoint as a 1D vector, rather than 2D
feature map as we do for ray conditioning. We show that the lack of spatial
inductive bias in pose conditioning causes the identity of generated people to
vary wildly with small rotations in Figure 5. At a resolution of $512\times
512$, pose conditioning achieves an average ID similarity score of $0.68\pm
0.20$, while ray conditioning achieves $0.75\pm 0.15$. Ray conditioning
produces much more view-consistent results than pose conditioning.

LFNs [33] |
---|---
Ray Cond. |
Figure 6: Unconditional Generation of Cars. We observe that ray conditioning,
a GAN-based method, has sharper images and more diverse samples than the
baseline LFNs [33]. In particular, ray conditioning achieves an FID of $3.39$
whereas LFNs achieves an FID of $41.8$. (FID is computed from $50$k random
samples for both methods.)

LFNs [33] |
---|---
Ray Cond. |
Input |

Figure 7: Face Reconstruction Comparison. LFNs [33] struggle to reconstruct
the training data of FFHQ, making it unsuitable for generating light fields
from a single-view dataset of natural images. Moreover, it is not able to
synthesize novel views. Ray conditioning is able to successfully reconstruct
the training images with GAN inversion.

###  4.4 Viewpoint Editing for Real Posed Images

We compare our ray conditioning against prior work for the application of
editing viewpoints of real posed images, as described in Section 3.3. Since
our method and baselines are all variants of StyleGAN, we invert images using
Pivotal Tuning Inversion (PTI) [29], and synthesize images of the same
individual from different viewpoints. We show in Figure 4 that our ray
conditioning method achieves the same explicit viewpoint control as the
geometry-based methods while preserving a much higher degree of photo-realism.
In the first example, the detail is incredibly noticeable in the eyes and
hair, which closely resemble the input image. Eyes are especially important
for human perception of identity and familiarity, but are often difficult to
invert for geometry-based methods due to their specularity.

Moreover, geometry-based methods such as EG3D and GMPI can introduce geometric
artifacts in synthesized images. When fitting a geometry-based representation
to a single image, there is often ambiguity on whether to modify the geometry
or texture. Incorrect geometry can create seemingly correct images. This is
only realized after a shift of viewpoint. For radiance fields, this has been
coined as shape radiance ambiguity [41], and still a challenging problem for
many 3D representations. In the bottom individual of Figure 4, we see that
although EG3D is able to reproduce the input image, the disoccluded parts
around the ears exhibit strong geometry artifacts when viewed at a different
yaw angle. GMPI is more severely hurt by the ambiguity between geometry and
appearance when fitting to the input image, which leads to distortion in the
novel views. This is most noticeable in the bottom individual’s glasses, which
appear to be glued to the face. Additionally, since geometry-based methods
tend to smooth textures in favor of photo-consistency, they lack the level of
details that our ray conditioning can offer. Rich details in hair, skin, and
eyes are inherently view-dependent, and are best captured by relaxing
constraints on photo-consistency.

Furthermore, we quantitatively evaluate ray conditioning and EG3D on GAN
inversion and viewpoint editing with the CelebA-HQ [24] dataset. Neither
method was trained on this dataset, allowing for a measure of cross-dataset
generalization. To evaluate the similarity between input images and
inversions, we calculate PSNR, SSIM, LPIPS [42], and ID scores. To evaluate
the image quality of synthesized novel views, we compare the FID and
KID$\times 100$ against the original images. Both metrics were computed from
one novel viewpoint for $100$ images at a resolution of $512\times 512$. The
results in Table 2 show that ray conditioning can achieve higher image quality
and detail preservation in both input inversions and after viewpoint editing.

| Inversion | Novel Views
---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS $\downarrow$ | ID$\uparrow$ | FID$\downarrow$ | KID $\downarrow$
EG3D [5] | 26.56 | $\mathbf{0.78}$ | 0.12 | $0.75$ | 65.0 | 0.0180
Ray Cond. | $\mathbf{27.49}$ | $\mathbf{0.78}$ | $\mathbf{0.10}$ | $\mathbf{0.85}$ | $\mathbf{58.5}$ | $\mathbf{0.0036}$

Table 2: GAN Inversion Metrics. We measure reconstruction quality between
input images and inversions on four metrics. We then compare the FID and
KID$\times 100$ between the original images and images with random viewpoint
changes. Ray conditioning can achieve higher image quality and detail in both
input inversions, and after viewpoint change.

###  4.5 Limitations

Although ray conditioning allows for explicit viewpoint control, it does not
generalize well to out-of-distribution camera poses. Figure 8 shows two types
of failure that we encountered in our experiments. In the top row, we show
that we cannot correctly render views that do not face the center of the
subject, e.g. by translating the camera horizontally. This is because all
images in FFHQ and AFHQ are cropped to be center-aligned. Second, large camera
rotations which go out of distribution may lead to view inconsistency.
However, if we train our generator on a dataset with $360^{\circ}$ viewpoints,
along with multi-view data, we can generate view consistent results along
$360^{\circ}$ of rotation, as shown in the supplementary material. We hope
that future work on data augmentation for in-the-wild datasets can improve
viewpoint control via ray conditioning.

Trans. |
---|---
Rot. |

Figure 8: Limitations on Camera Control. Ray conditioning does not generalize
well to out-of-distribution camera poses. The first row shows that $\pm 1.0$
of $x$-axis camera translation leads to viewpoint distortion and identity
shift. The second row shows that an out of distribution rotation of $\pm
75^{\circ}$ leads to identity shift.

##  5 Conclusion

We propose ray conditioning, a method for multi-view image generation with
explicit viewpoint control. By conditioning a 2D GAN on a light field prior,
as opposed to using a 3D representation, we achieve the best photo-realism
among all existing multi-view image synthesizers, with competitive identity
consistency across viewpoints. We also show our model’s advantage, i.e., more
fine-scale details and fewer geometry artifacts, over baselines in static
image viewpoint editing applications. Finally, our method pushes forward the
boundary of multi-view image generation quality for geometry-free generative
models by a great margin.

## References

  * [1] Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows.  ACM Trans. Graph., 40(3), May 2021.
  * [2] Edward H. Adelson and James R. Bergen.  The plenoptic function and the elements of early vision.  1991\.
  * [3] Mikolaj Binkowski, Danica J. Sutherland, Michal Arbel, and Arthur Gretton.  Demystifying mmd gans.  ArXiv, abs/1801.01401, 2018.
  * [4] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In arXiv, 2020.
  * [5] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3D generative adversarial networks.  In arXiv, 2021.
  * [6] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [7] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [8] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [9] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In IEEE Computer Vision and Pattern Recognition Workshops, 2019\.
  * [10] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, 2014.
  * [11] Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen.  The lumigraph.  ACM Trans. Graph., 1996.
  * [12] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d aware generator for high-resolution image synthesis.  In Int. Conf. Learn. Represent., 2022.
  * [13] David Ha, Andrew M. Dai, and Quoc V. Le.  Hypernetworks.  In Int. Conf. Learn. Represent., 2017.
  * [14] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  Int. Conf. Comput. Vis., pages 1026–1034, 2015.
  * [15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NIPS, 2017.
  * [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  arXiv preprint arxiv:2006.11239, 2020.
  * [17] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  In Proc. NeurIPS, 2020.
  * [18] Ali Jahanian*, Lucy Chai*, and Phillip Isola.  On the ”steerability” of generative adversarial networks.  In International Conference on Learning Representations, 2020.
  * [19] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In Proc. NeurIPS, 2020.
  * [20] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In Neural Information Processing Systems, 2021.
  * [21] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396–4405, 2018.
  * [22] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8107–8116, 2019.
  * [23] Jaehoon Ko, Kyusun Cho, Daewon Choi, Kwangrok Ryoo, and Seungryong Kim.  3d gan inversion with pose optimization.  WACV, 2023.
  * [24] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  * [25] Marc Levoy and Pat Hanrahan.  Light field rendering.  ACM Trans. Graph., 1996.
  * [26] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [27] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.  StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 13503–13513, June 2022.
  * [28] William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman.  Gan-supervised dense visual alignment.  In CVPR, 2022.
  * [29] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  ACM Trans. Graph., 2021.
  * [30] Sefik Ilkin Serengil and Alper Ozpinar.  Lightface: A hybrid deep face recognition framework.  In 2020 Innovations in Intelligent Systems and Applications Conference (ASYU), pages 23–27. IEEE, 2020.
  * [31] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  TPAMI, 2020.
  * [32] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In Adv. Neural Inform. Process. Syst., 2020.
  * [33] Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand.  Light field networks: Neural scene representations with single-evaluation rendering.  In Adv. Neural Inform. Process. Syst., 2021.
  * [34] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In Advances in Neural Information Processing Systems, 2019.
  * [35] Jascha Narain Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  ArXiv, abs/1503.03585, 2015.
  * [36] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [37] Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  ArXiv, abs/2210.04628, 2022.
  * [38] Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy.  High performance imaging using large camera arrays.  ACM Trans. Graph., 24(3):765–776, jul 2005.
  * [39] Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12858–12867, 2020.
  * [40] Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen.  High-fidelity 3d gan inversion by pseudo-multi-view optimization.  arXiv preprint arXiv:2211.15662, 2022.
  * [41] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv:2010.07492, 2020.
  * [42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [43] Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, and Alex Colburn.  Generative multiplane images: Making a 2d gan 3d-aware.  In Eur. Conf. Comput. Vis., 2022.
  * [44] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  ArXiv, abs/1805.09817, 2018.

Appendix

In this document, we present implementation details and additional results
that are supplemental to the main paper. In Appendix A, we outline key design
details for the ray conditioning method. In Appendix B, we include more
results of viewpoint editing. We also compare to concurrent work in 3D-aware
image inversion, and show that these methods still have challenges with
respect to realism. Finally, in Appendix C, we include more details about the
datasets used.

##  Appendix A Design Details

We outline some key design details in this section.

###  A.1 Pretraining and Weight Initialization

Recall that the ray embedding is a $6\times H\times W$ feature map which is
concatenated to each intermediate representation of StyleGAN. To accommodate
these extra features, we add 6 channels to each convolutional layer of
StyleGAN. To begin training, we initialize all prior weights to be those of a
pretrained StyleGAN model. The extra 6 channels for each layer is then
initialized normally, with the default StyleGAN2 gain parameters [22, 14]. The
discriminator also uses pretrained discriminator weights. The pose
conditioninal discriminator module is initialized as an MLP in the same way
done in EG3D and StyleGAN2-ADA [5, 19]. In Figure 10, we demonstrate the
effect that ray conditioning has on the intermediate layers of StyleGAN.
Consistent with prior work [39, 17], subject pose is generally a coarse-level
feature which takes form around resolutions $16\times 16$ to $64\times 64$.
Ray conditioning also converges very fast. In Figure 9, we show that ray
conditioning takes about 160kimgs, or 1.5 hours on 2 Nvidia A6000s, to learn
camera pose.

Init |  80kimgs |  160kimgs |  240kimgs |  320kimgs
---|---|---|---|---

Figure 9: Training Convergence Speed. For faces, we initialize our ray
conditioning model from a pretrained StyleGAN2 model. Through training, it
learns to properly generate images from a target pose. Ray conditioning
converges quickly. After 160kimgs (1.5 hours on 2 A6000s), ray conditioning is
already able to properly generate an image at a target pose.

###  A.2 Effect on Latent Space

Many prior work have found directions in StyleGAN’s $\mathcal{W}$ latent space
which correspond to subject pose [17, 31]. Surprisingly, ray condition
nullifies these directions in the latent space. When we try to modify an
image’s pose using InterfaceGAN directions [31] after ray conditioning, the
image stays the same. This implies that ray conditioning “moves” the pose
information previously embedded in the $\mathcal{W}$ latent space into the
convolutional weights assigned to the ray embedding.

$4\times 4$ |  $8\times 8$ |  $16\times 16$ |  $32\times 32$ |  $64\times 64$ |  $128\times 128$ |  $256\times 256$ |  $512\times 512$ |  RGB
---|---|---|---|---|---|---|---|---

Figure 10: Ray Conditioning Affects Intermediate Features. To learn viewpoint
control, we condition each intermediate layer of StyleGAN on a ray embedding.
We picture coarse to fine resolution feature maps of one latent seed from
different poses. Consistent with prior work [39, 17], we find that subject
pose is a coarse-level feature in the StyleGAN latent space. It begins to take
form around resolutions $16\times 16$ to $64\times 64$.

###  A.3 StyleGAN2 vs. StyleGAN3.

We have implemented ray conditioning successfully for both StyleGAN2 [22] and
StyleGAN3 [20]. StyleGAN3 is better suited for video generation tasks because
of its antialiasing abilities. However, for multi-view image generation, we
have found no advantage for using StyleGAN3. Following EG3D [5] and GMPI [43],
we choose to use StyleGAN2 because of its slightly higher image quality and
faster model.

###  A.4 Training Details

FFHQ and AFHQ models were trained starting from official StyleGAN checkpoints.
They were trained on $2\times$ Nvidia A6000 GPUs for 1040kimgs - 1440kimgs.
For SRN Cars, we train from scratch for 13,520kimgs. Hyperparameters are set
to the same as those of StyleGAN2.

##  Appendix B Additional Results

###  B.1 Viewpoint Editing Examples

We believe that ray conditioning is a natural choice for portrait editing over
3D-aware GANs. In Figure 13, we recreate Figure 2 in the main paper with EG3D.
We see that the resulting images from EG3D appear more cartoonish than human-
like. There are also geometry distortions. In the top right individual (yellow
background), we see geometry artifacts near his ear. In the bottom example
(pink background), the challenges are also apparent when we try to blend faces
into a preexisting image. EG3D fails to achieve our intended effect of
photorealistic viewpoint editing. In the supplementary material, we also
include a video demonstrating the viewpoint control we have over input
samples.

###  B.2 3D-Aware GAN Inversion

Many have recognized the issues with using a 2D GAN inversion method such as
PTI [29] with a 3D-aware GAN such as EG3D [5]. Although PTI can successfully
invert an input image, it can cause geometry artifacts that are only realized
after a change in viewpoint. Several work have created dedicated 3D GAN
inversion methods for EG3D. However, we find that these methods can still
cause aliasing in the inverted images, creating a loss of quality. In
addition, they can struggle when inverting side-facing images. The occluded
regions can have incorrect geometry. The eyes can also lose their specularity,
which is important for human perception of identity. In Figure 14, and Figure
15 we demonstrate these issues. We compare ray conditioning to two related
work. 3D GAN Inversion with Pose Estimation [23] is a recent work designed for
EG3D inversion. HFGI3D [40] is a concurrent work also designed for EG3D
inversion. Inversion with dedicated 3D-aware methods can also take much longer
than with PTI. For instance, 3D GAN Inversion with Pose Estimation [23] takes
on average 3 minutes per image. HFGI3D [40] can take 8 minutes. Meanwhile, PTI
with ray conditioning only takes 1 minute per image.

###  B.3 Latent Space Pose Editing

InterfaceGAN and similar work [31, 17, 1] allow for viewpoint change by
finding directions in the StyleGAN latent space which correspond to pose.
However, these methods can only do binary changes such as left facing or right
facing, instead of explicit viewpoint control. Because they do not operate
per-pixel like how ray conditioning does, they also lack the spatial inductive
bias which makes ray conditioning effective. We show an example of the
differences between latent space editing and ray conditioning in Figure 16.

###  B.4 Latent Space Samples

In Figure 19, we show uncurated latent space samples from StyleGAN2 with ray
conditioning. Even without a 3D representation, ray conditioning is still able
change the viewpoint of generated samples. We picture latent seeds 0-31. To
show our image quality, we also present larger images in Figure 18.

###  B.5 Additional Results on Cars

In Figure 11, we demonstrate that our model can enable $360^{\circ}$ viewpoint
editing when trained on a dataset with $360^{\circ}$ of views. The car stays
consistent as we rotate the camera. We also include videos of smooth
trajectories in the supplementary material. We use StyleGAN3 [20] because of
its antialiasing properties.

Figure 11: $\mathbf{360^{\circ}}$ View Consistency from Ray Conditioning. When
trained on a multi-view dataset, ray conditioning is able to generate view
consistent results with $360^{\circ}$ of rotation. Please see the accompanying
videos for continuous results.

###  B.6 More Experiments on Light Field Networks

In terms of image quality, ray conditioning is a large improvement over Light
Field Networks (LFNs) [33]. We provide more results from LFNs on FFHQ in
Figure 12. As discussed in the main paper, LFNs have two main challenges.
First, LFNs struggle to reconstruct high frequency details on a photo-
realistic dataset such as FFHQ. We also attempted to train LFNs with SIREN
[32] activations instead of ReLU activations, but the model struggled to
converge. Second, LFNs are unable to generate novel views when trained on a
dataset with only one image per instance. Our work demonstrates that light
field priors introduced in LFNs can be naturally extended from MLPs to more
powerful CNN-based image synthesizers.

NVs |  Recon. |  Input
---|---|---

Figure 12: Light Field Networks on FFHQ. Light Field Networks (LFNs) [33] have
two key challenges. First, they struggle to reconstruct high frequency details
from input images. Second, when trained on a dataset with only one image per
face, LFNs [33] struggle to construct novel views (NVs). When combined with a
more powerful generative model such as a GAN, ray conditioning helps to
address both of these problems.

Ray Conditioning + PTI [29] |  EG3D [5] \+ PTI [29]
---|---

Figure 13: Photo Editing Comparison. We believe that ray conditioning is a
natural choice for portrait editing over 3D-aware GANs. To illustrate, we
compare the ray conditioning results from Figure 2 in the main paper to EG3D
[5]. We see that EG3D cannot create a change in viewpoint without sacrificing
some realism. There are noticeable geometry issues, and the edited individuals
seem more cartoonish than human-like.

Input |  Novel Views
---|---

Input |  Novel Views
---|---

Ray Conditioning + PTI [29] |  EG3D [5] \+ PTI [29]
---|---

Input |  Novel Views
---|---

Input |  Novel Views
---|---

EG3D [5] \+ 3D GAN Inversion [23] |  EG3D [5] \+ HFGI3D [40]
---|---

Figure 14: Comparison to 3D-Aware GAN Inversion Methods. Even dedicated 3D GAN
inversion methods for EG3D [23, 40] can struggle to generate realistic novel
views. It is especially challenging for 3D-aware GAN inversion methods when
the input image is not front facing. It tends to distort the geometry of the
input individual. For instance, when combined with EG3D, 3D GAN inversion [23]
and PTI [29] appear to widen the individual’s face. PTI [29], 3D GAN Inversion
[23], and HFGI3D [40] all appear to make the individual’s nose more pointy.

Input |  Novel Views
---|---

Input |  Novel Views
---|---

Ray Conditioning + PTI [29] |  EG3D [5] \+ PTI [29]
---|---

Input |  Novel Views
---|---

Input |  Novel Views
---|---

EG3D [5] \+ 3D GAN Inversion [23] |  EG3D [5] \+ HFGI3D [40]
---|---

Figure 15: Comparison to 3D-Aware GAN Inversion Methods. Ray conditioning
provides the best image quality compared to geometry-based GANs and inversion
methods. The loss of image quality is noticeable. For 3D GAN Inversion [23]
and HFGI3D [40], there are streaks across the face, hinting at spatial
aliasing issues. 3D GAN Inversion and HFGI3D also cannot reconstruct the
specularity of the eyes, which is very apparent in the ray conditioning
example.

Input |  Novel Views
---|---

Input |  Novel Views
---|---

Ray Conditioning |  InterfaceGAN [31]
---|---

Figure 16: Comparison to InterfaceGAN. Latent space editing techniques such as
InterfaceGAN [31] can generate binary changes to images to make them left
facing or right facing. However, these models lack the same level of control
that ray conditioning and geometry-based generative models do. We can achieve
free viewpoint control with ray conditioning, while latent space editing is
restricted to one dimension. InterfaceGAN also lacks the same amount of
disentanglement as we do, causing identity shift.

##  Appendix C Evaluation Details

###  C.1 Datasets

Similar to prior work [5, 43], our method requires a dataset of images and
estimated camera poses. We outline the datasets used below.

FFHQ Human Faces. FFHQ [21] is a dataset of $\sim 70$k $1024\times 1024$
images of front-facing faces. We use camera poses provided by EG3D, which are
estimated by a deep face pose estimator [9]; camera poses are assumed to be
distributed on a sphere, all facing a shared center. As previously reported by
EG3D, this dataset contains bias which may affect the resulting generations.
For instance, people in front-facing images are more likely to smile. People
who appear to be lower than the camera tend to be children. In Figure 17, we
present the distribution of subject pose in terms of yaw (horizontal
rotation), and pitch (vertical rotation).

Figure 17: FFHQ Pose Distribution. FFHQ consists mainly of front facing
photographs. Its standard deviation for yaw (horizontal rotation) is
$16^{\circ}$. Its standard deviation for pitch (vertical rotation) is
$2^{\circ}$ degrees. As noted by EG3D [5], this unbalanced distribution of
subject pose is a challenge for all multi-view generative models trained on
FFHQ. Proper data augmentation to reduce distribution bias is still an open
and important problem.

Figure 18: Curated Latent Space Samples. Given the 12 front facing images, we
show that we can map them to a different viewpoint while maintaining the
quality of the generated faces.

AFHQv2 Cat Faces. AFHQv2 [7] is a dataset consisting of many animal faces. We
train our model on the cat subset using the camera poses provided by EG3D.
This subset only consists of $\sim 5$k $512\times 512$ images, which is much
smaller than FFHQ. Some pretraining is expected for good results.

Figure 19: Uncurated Latent Samples. We picture two views from latent vectors
of seeds 0-31 to demonstrate the quality of our GAN. Even without a 3D
representation, ray conditioning successfully generates images of the same
individual from different view points. Results were generated with a
truncation of $\psi=0.7$.

SRN Cars. The SRN Cars [34] training set is a collection of $\sim$2.5k
ShapeNet [6] cars, each rendered from $250$ cameras distributed on a sphere at
a resolution of $128\times 128$. Because it contains multiple images per
object, it is commonly used for evaluating geometry-free view synthesis
models. We demonstrate our method’s ability to generate $360^{\circ}$ light
fields, and compare to the LFNs [33] baseline on this dataset. Unlike for FFHQ
or AFHQ, we start training from scratch.


# Supervised Seeded Iterated Learning for Interactive Language Learning

Yuchen Lu
Mila
University of Montreal
&Soumye Singhal
Mila
University of Montreal
&Florian Strub
DeepMind
Olivier Pietquin
Google Research
Brain Team
&Aaron Courville
Mila
University of Montreal, CIFAR

###### Abstract

Language drift has been one of the major obstacles to train language models
through interaction. When word-based conversational agents are trained towards
completing a task, they tend to invent their language rather than leveraging
natural language. In recent literature, two general methods partially counter
this phenomenon: Supervised Selfplay (S2P) and Seeded Iterated Learning (SIL).
While S2P jointly trains interactive and supervised losses to counter the
drift, SIL changes the training dynamics to prevent language drift from
occurring. In this paper, we first highlight their respective weaknesses,
i.e., late-stage training collapses and higher negative likelihood when
evaluated on human corpus. Given these observations, we introduce Supervised
Seeded Iterated Learning (SSIL) to combine both methods to minimize their
respective weaknesses. We then show the effectiveness of SSIL in the language-
drift translation game.

##  1 Introduction

Since the early days of NLP Winograd (1971), conversational agents have been
designed to interact with humans through language to solve diverse tasks,
e.g., remote instructions Thomason et al. (2015) or booking assistants Bordes
et al. (2017); El Asri et al. (2017). In this goal-oriented dialogue setting,
the conversational agents are often designed to compose with predefined
language utterances Lemon and Pietquin (2007); Williams et al. (2014); Young
et al. (2013). Even if such approaches are efficient, they also tend to narrow
down the agent’s language diversity. To remove this restriction, recent work
has been exploring interactive word-based training. In this setting, the
agents are generally trained through a two-stage process Wei et al. (2018); De
Vries et al. (2017); Shah et al. (2018); Li et al. (2016a); Das et al. (2017):
Firstly, the agent is pretrained on a human-labeled corpus through supervised
learning to generate grammatically reasonable sentences. Secondly, the agent
is finetuned to maximize the task-completion score by interacting with a user.
Due to sample-complexity and reproducibility issues, the user is generally
replaced by a game simulator that may evolve with the conversational agent.
Unfortunately, this pairing may lead to the _language drift_ phenomenon, where
the conversational agents gradually co-adapt, and drift away from the
pretrained natural language. The model thus becomes unfit to interact with
humans Chattopadhyay et al. (2017); Zhu et al. (2017); Lazaridou et al.
(2020).

While domain-specific methods exist to counter language drift Lee et al.
(2019); Li et al. (2016b), a simple task-agnostic method consists of combining
interactive and supervised training losses on a pretraining corpus Wei et al.
(2018); Lazaridou et al. (2016), which was later formalized as Supervised
SelfPlay (S2P) Lowe et al. (2020).

Inspired by language evolution and cultural transmission Kirby (2001); Kirby
et al. (2014), recent work proposes Seeded Iterated Learning (SIL) Lu et al.
(2020) as another task-agnostic method to counter language drift. SIL modifies
the training dynamics by iteratively refining a pretrained student agent by
imitating interactive agents, as illustrated in Figure 1. At each iteration, a
teacher agent is created by duplicating the student agent, which is then
finetuned towards task completion. A new dataset is then generated by greedily
sampling the teacher, and those samples are used to refine the student through
supervised learning. The authors empirically show that this iterated learning
procedure induces an inductive learning bias that successfully maintains the
language grounding while improving task-completion.

Figure 1: SIL Lu et al. (2020). A student agent is iteratively refined using
newly generated data from a teacher agent. At each iteration, a teacher agent
is created on top of the student before being finetuned by interaction, e.g.
maximizing a task completion-score. Teacher generates a dataset with greedy
sampling and students imitate those samples. The interaction step involves
interaction with another language agent.

As a first contribution, we further examine the performance of these two
methods in the setting of a translation game Lee et al. (2019). We show that
S2P is unable to maintain a high grounding score and experiences a late-stage
collapse, while SIL has a higher negative likelihood when evaluated on human
corpus.

We propose to combine SIL with S2P by applying an S2P loss in the interactive
stage of SIL. We show that the resulting _Supervised Seeded Iterated Learning_
(SSIL) algorithm manages to get the best of both algorithms in the translation
game. Finally, we observe that the late-stage collapse of S2P is correlated
with conflicting gradients before showing that SSIL empirically reduces this
gradient discrepancy.

##  2 Preventing Language Drift

We describe here our interactive training setup before introducing different
approaches to prevent language drift. In this setting, we have a set of
collaborative agents that interact through language to solve a task. To begin,
we train the agents to generate natural language in a word-by-word fashion.
Then we finetune the agents to optimize a task completion score through
interaction, i.e., learning to perform the task better. Our goal is to prevent
the language drift in this second stage.

###  2.1 Initializing the Conversational Agents

For a language agent $f$ parameterized by $\bm{\theta}$, and a sequence of
generated words $\bm{w}_{1:i}=[w_{j}]_{j=1}^{i}$ and an arbitrary context
$\bm{c}$, the probability of the next word $w_{i}$ is
$p(w_{i+1}|\bm{w}_{1:i},\bm{c})=f_{\bm{\theta}}(\bm{w}_{1:i},\bm{c})$ We
pretrain the language model to generate meaningful sentences by minimizing the
cross-entropy loss $\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$ where
the word sequences are sampled from a language corpus $D_{pretrain}$. Note
that this language corpus may either be task-related or generic. Its role is
to get our conversational agents a reasonable initialization.

###  2.2 Supervised Selfplay (S2P)

A common way to finetune the language agents while preventing language drift
is to replay the pretraining data during the interaction stage. In S2P the
training loss encourages both maximizing task-completion while remaining close
to the initial language distribution. Formally,

$\mathcal{L}^{\operatorname*{\text{S2P}}2P}=\mathcal{L}^{\operatorname*{\text{INT}}}+\alpha\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$
(1)

where $\mathcal{L}^{\operatorname*{\text{INT}}}$ is a differentiable
interactive loss maximizing task completion, e.g. reinforcement learning with
policy gradients Sutton et al. (2000), Gumbel Straight-through Estimator (STE)
Jang et al. (2017) etc., $\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$
is a cross-entropy loss over the pretraining samples. $\alpha$ is a positive
scalar which balances the two losses.

###  2.3 Seeded Iterated Learning (SIL)

Seeded Iterated Learning (SIL) iteratively refines a pretrained _student_
model by using data generated from newly trained _teacher_ agents Lu et al.
(2020). As illustrated in Figure 1, the student agent is initialized with the
pretrained model. At each iteration, a new teacher agent is generated by
duplicating the student parameters. It is tuned to maximize the task-
completion score by optimizing the interactive loss
$\mathcal{L}^{\operatorname*{\text{TEACHER}}}=\mathcal{L}^{\operatorname*{\text{INT}}}$
In a second step, we sample from the teacher to generate new training data
$D_{teacher}$, and we refine the student by minimizing the cross-entropy loss
$\mathcal{L}^{\operatorname*{\text{STUDENT}}}=\mathcal{L}^{\operatorname*{\text{CE}}}_{teacher}$
where sequence of words are sampled from $D_{teacher}$. This imitation
learning stage can induce an information bottleneck, encouraging the student
to learn a well-formatted language rather than drifted components.

###  2.4 SSIL: Combining SIL and S2P

S2P and SIL have two core differences: first, SIL never re-uses human
pretraining data. As observed in Section 4.1, this design choice reduces the
language modeling ability of SIL-trained agents, with a higher negative
likelihood when evaluated on human corpus. Second, S2P agents merge
interactive and supervised losses, whereas SIL’s student never experiences an
interactive loss. As analyzed in Section 4.3, the S2P multi-task loss induces
conflicting gradients, which may trigger language drift. In this paper, we
propose to combine these two approaches and demonstrate that the combination
effectively minimizes their respective weaknesses. To be specific, we apply
the S2P loss over the SIL teacher agent, which entails
$\mathcal{L}^{\operatorname*{\text{TEACHER}}}=\mathcal{L}^{\operatorname*{\text{INT}}}+\alpha\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$.
We call the resulting algorithm, Supervised Seeded Iterated Learning (SSIL).
In SSIL, teachers can generate data that is close to the human distribution
due to the S2P loss, while students are updated with a consistent supervised
loss to avoid the potential weakness of multi-task optimization. In addition,
SSIL still maintains the inductive learning bias of SIL. We list all these
methods in Table 1 for easy comparison. We also experiment with other ways of
combining SIL and S2P by mixing the pretraining data with teacher data during
the imitation learning stage. We call this method _MixData_. We show the
results of this approach in Appendix 4.2. We find that this approach is very
sensitive to the mixing ratio of these two kinds of data, and the best
configuration is still not as good as SSIL.

_Finetuning Methods_ | _Training Losses_
---|---
Gumbel | $\mathcal{L}^{\operatorname*{\text{INT}}}$
S2P | $\mathcal{L}^{\operatorname*{\text{INT}}}+\alpha\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$
SIL (teacher) | $\mathcal{L}^{\operatorname*{\text{INT}}}$
SIL (student) | $\mathcal{L}^{\operatorname*{\text{CE}}}_{teacher}$
SSIL (teacher) | $\mathcal{L}^{\operatorname*{\text{INT}}}+\alpha\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$
SSIL (student) | $\mathcal{L}^{\operatorname*{\text{CE}}}_{teacher}$
Table 1: Finetuning with respective training objective.

##  3 Experimental Setting

###  3.1 Translation Game

We replicate the translation game setting from Lee et al. (2019) as it was
designed to study language drift. First, a _sender_ agent translates French to
English (Fr-En), while a _receiver_ agent translates English to German (En-
De). The sender and receiver are then trained together to translate French to
German with English as a pivot language. For each French sentence, we sample
English from the sender, send it to the receiver, and sample German from the
receiver.

The task score is defined as the BLEU score between generated German
translation and the ground truth (_BLEU De_) Papineni et al. (2002). The goal
is to improve the task score without losing the language structure of the
intermediate English language.

###  3.2 Training Details

The sender and the receiver are pretrained on the IWSLT dataset Cettolo et al.
(2012) which contains $(\operatorname*{\text{Fr}},\operatorname*{\text{En}})$
and $(\operatorname*{\text{En}},\operatorname*{\text{De}})$ translation pairs.
We then use the Multi30k dataset (Elliott et al., 2016) to build the
finetuning dataset with
$(\operatorname*{\text{Fr}},\operatorname*{\text{De}})$ pairs. As IWSLT is a
generic translation dataset and Multi30k only contains visually grounded
translated captions, we also call IWSLT task-agnostic while Multi30K task-
related. We use the cross-entropy loss of German as the interactive training
objective, which is differentiable w.r.t. the receiver. For the sender, we use
Gumbel Softmax straight-through estimator to make the training objective also
differentiable w.r.t. the sender, as in Lu et al. (2020).

Implementation details are in Appendix B

###  3.3 Metrics for Grounding Scores

In practice, there are different kinds of language drift Lazaridou et al.
(2020) (e.g. syntactic drift and semantic drift). We thus have multiple
metrics to consider when evaluating language drift. We first compute English
BLEU score (_BLEU En_) comparing the generated English translation with the
ground truth human translation. We include the negative log-likelihood (_NLL_)
of the generated En translation under a pretrained language model as a measure
of syntactic correctness. In line with Lu et al. (2020) , we also report
results using another language metric: the negative log-likelihood of human
translations (_RealNLL_) given a finetuned Fr-En model. We feed the finetuned
sender with human task-data to estimate the model’s log likelihood. The lower
is this score, the more likely the model would generate such human-like
language.

##  4 Experiments

(a) BLEU De (Task Score)

(b) BLEU En

(c) NLL

(d) RealNLL

Figure 2: Task and language metrics for Vanilla Gumbel, SIL, S2P, and SSIL in
the translation game average over 5 seeds. We also show the results of mixing
pretraining data in the teacher dataset (Section 4.2). The plots are averaged
over 5 seeds with shaded area as standard deviation. Although SIL and S2P both
counter language drift, S2P suffers from late collapse, and SIL has a high
_RealNLL_ , suggesting that its output may not correlate well with human
sentences.

(a) Bleu En

(b) Cosine Similarity

Figure 3: Cosine similarity between the gradients issued from
$\mathcal{L}^{\operatorname*{\text{INT}}}$ and
$\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$. The collapse of the BLEU
En matches the negative cosine similarity.We here set $\alpha=0.5$ but similar
values yield identical behavior as shown in Figure 4 in Appendix.

###  4.1 S2P and SIL Weaknesses

We report the task and grounding scores of vanilla Gumbel, S2P, SIL, and SSIL
in Figure 2. The respective best hyper-parameters can be found in the
appendix. As reported by Lu et al. (2020), vanilla Gumbel successfully
improves the task score _BLEU De_ , but the _BLEU En_ score as well as other
grounding metric collapses, indicating a language drift during the training.
Both S2P and SIL manage to increase _BLEU De_ while maintaining a higher _BLEU
En_ score, countering language drift. However, S2P has a sudden (and
reproducible) late-stage collapse, unable to maintain the grounding score
beyond 150k steps. On the other hand, SIL has a much higher RealNLL than S2P,
suggesting that SIL has a worse ability to model human data.

SSIL seems to get the best of the two worlds. It has a similar task score
_BLEU De_ as S2P and SIL, while it avoids the late-stage collapse. It ends up
with the highest _BLEU En_ , and it improves the RealNLL over SIL, though
still not as good as S2P. Also, it achieves even better NLL, suggesting that
its outputs are favoured by the pretrained language model.

###  4.2 Mixing Teacher and Human Data

We also explore whether injecting pretraining data into the teacher dataset
may be a valid substitute for the S2P loss. We add a subset of the pretraining
data in the teacher dataset before refining the student, and we report the
results in Figure 2 and 6. Unfortunately, such an approach was quite unstable,
and it requires heavy hyper-parameters tuning to match SSIL scores. As
explained in Kirby (2001), iterated learning rely on inductive learning to
remove language irregularities during the imitation step. Thus, mixing two
language distributions may disrupt this imitation stage.

###  4.3 Why S2P collapses?

We investigate the potential cause of S2P late-stage collapse and how SSIL may
resolve it. We firstly hope to solve this by increasing the supervised loss
weight $\alpha$. However, we find that a larger $\alpha$ only delays the
eventual collapse as well as decreases the task score, as shown in Figure 5 in
Appendix D.

We further hypothesize that this late-stage collapse can be caused by the
distribution mismatch between the pretraining data (IWSLT) and the task-
related data (Multi30K), exemplified by their word frequencies difference. A
mismatch between the two losses could lead to conflicting gradients, which
could, in turn, make training unstable. In Figure 3, we display the cosine
similarity of the sender gradients issued by the interactive and supervised
losses $cos(\nabla_{sender}\mathcal{L}^{\operatorname*{\text{INT}}}$,
$\nabla_{sender}\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain})$ for both
S2P and SSIL for $\alpha=0.5$ during training. Early in S2P training, we
observe that the two gradients remain orthogonal on average, with the cosine
oscillating around zero. Then, at the same point where the S2P _Bleu En_
collapses, the cosine of the gradients starts trending negative, indicating
that the gradients are pointing in opposite directions. However, SSIL does not
have this trend, and the _BLEU En_ does not collapse. Although the exact
mechanism of how conflicting gradients trigger the language drift is unclear,
current results favor our hypothesis and suggest that language drift could
result from standard multi-task optimization issues Yu et al. (2020);
Parisotto et al. (2016); Sener and Koltun (2018) for S2P-like methods.

#### Conclusion

We investigate two general methods to counter language drift: S2P and SIL. S2P
experiences a late-stage collapse on the grounding score, whereas SIL has a
higher negative likelihood on human corpus. We introduce SSIL to combine these
two methods effectively. We further show the correlation between S2P late-
stage collapse and conflicting gradients.

#### Acknowledgement

We thank Compute Canada (www.computecanada.ca) for providing the computational
resources. We thank Miruna Pislar and Angeliki Lazaridou for their helpful
discussions.

## References

  * Bordes et al. (2017) Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017.  Learning end-to-end goal-oriented dialog.  In _Proceedings of International Conference on Learning Representations (ICLR)_.
  * Cettolo et al. (2012) Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012.  Wit3: Web inventory of transcribed and translated talks.  In _Proceedings of Conference of European Association for Machine Translation_.
  * Chattopadhyay et al. (2017) Prithvijit Chattopadhyay, Deshraj Yadav, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee, Dhruv Batra, and Devi Parikh. 2017.  Evaluating visual conversational agents via cooperative human-ai games.  In _Proceedings of AAAI Conference on Human Computation and Crowdsourcing_.
  * Das et al. (2017) Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017\.  Learning cooperative visual dialog agents with deep reinforcement learning.  In _Proceedings of International Conference on Computer Vision (ICCV)_.
  * De Vries et al. (2017) Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017.  Guesswhat?! visual object discovery through multi-modal dialogue.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * El Asri et al. (2017) Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017.  Frames: a corpus for adding memory to goal-oriented dialogue systems.  In _Proceedings of the SIGdial Meeting on Discourse and Dialogue_.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30k: Multilingual english-german image descriptions.  In _Proceedings of Workshop on Vision and Language_.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparametrization with gumble-softmax.  In _International Conference on Learning Representations (ICLR)_.
  * Kirby (2001) Simon Kirby. 2001.  Spontaneous evolution of linguistic structure-an iterated learning model of the emergence of regularity and irregularity.  _IEEE Transactions on Evolutionary Computation_ , 5(2):102–110.
  * Kirby et al. (2014) Simon Kirby, Tom Griffiths, and Kenny Smith. 2014.  Iterated learning and the evolution of language.  _Current opinion in neurobiology_ , 28:108–114.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Lazaridou et al. (2016) Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2016.  Multi-agent cooperation and the emergence of (natural) language.  In _Proceedings of International Conference on Learning Representations (ICLR)_.
  * Lazaridou et al. (2020) Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020.  Multi-agent communication meets natural language: Synergies between functional and structural language learning.  In _Proceedings of the Meeting on Association for Computational Linguistics (ACL)_.
  * Lee et al. (2019) Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019.  Countering language drift via visual grounding.  In _Proceedings of Empirical Methods in Natural Language Processing (EMNLP)_.
  * Lemon and Pietquin (2007) Olivier Lemon and Olivier Pietquin. 2007.  Machine learning for spoken dialogue systems.  In _Proceedings of European Conference on Speech Communication and Technologies (Interspeech)_.
  * Li et al. (2016a) Jiwei Li, Alexander H Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2016a.  Dialogue learning with human-in-the-loop.  In _Proceedings of International Conference on Learning Representations (ICLR)_.
  * Li et al. (2016b) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b.  Deep reinforcement learning for dialogue generation.  In _Proceedings of Empirical Methods in Natural Language Processing (EMNLP)_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Lowe et al. (2020) Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. 2020.  On the interaction between supervision and self-play in emergent communication.  In _International Conference on Learning Representations (ICLR)_.
  * Lu et al. (2020) Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron Courville. 2020.  Countering language drift with seeded iterated learning.  In _Proceedings of International Conference of Machine Learning (ICML)_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the Meeting on Association for Computational Linguistics (ACL)_.
  * Parisotto et al. (2016) Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. 2016.  Actor-mimic: Deep multitask and transfer reinforcement learning.  In _Proceedings of International Conference on Learning Representations (ICLR)_.
  * Sener and Koltun (2018) Ozan Sener and Vladlen Koltun. 2018.  Multi-task learning as multi-objective optimization.  In _Proceedings of Advances in Neural Information Processing Systems (NeurIPS)_ , pages 527–538.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Shah et al. (2018) Pararth Shah, Dilek Hakkani-Tur, Bing Liu, and Gokhan Tur. 2018.  Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_.
  * Sutton et al. (2000) Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000\.  Policy gradient methods for reinforcement learning with function approximation.  In _Proceedings of Advances in Neural Information Processing Systems (NIPS)_.
  * Thomason et al. (2015) Jesse Thomason, Shiqi Zhang, Raymond J Mooney, and Peter Stone. 2015.  Learning to interpret natural language commands through human-robot dialog.  In _Proceedings of International Joint Conference on Artificial Intelligence (IJCAI)_.
  * Wei et al. (2018) Wei Wei, Quoc Le, Andrew Dai, and Jia Li. 2018.  Airdialogue: An environment for goal-oriented dialogue research.  In _Proceedings of Empirical Methods in Natural Language Processing (EMNLP)_.
  * Williams et al. (2014) Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachandran. 2014.  The dialog state tracking challenge series.  _AI Magazine_ , 35(4):121–124.
  * Winograd (1971) Terry Winograd. 1971.  Procedures as a representation of data in a computer program for understanding natural language.  Technical report, PhD thesis, Massachusets Institute of Technology.
  * Young et al. (2013) Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013\.  Pomdp-based statistical spoken dialog systems: A review.  _Proceedings of the IEEE_ , 101(5):1160–1179.
  * Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020.  Gradient surgery for multi-task learning.  _arXiv preprint arXiv:2001.06782_.
  * Zhu et al. (2017) Yan Zhu, Shaoting Zhang, and Dimitris Metaxas. 2017.  Interactive reinforcement learning for object grounding via self-talking.  _Visually Grounded Interaction and Language Workshop (ViGIL)_.

##  Appendix A Explicit losses in the Translation Game

#### S2P

Let $\mathcal{L}^{\operatorname*{\text{GSTE}}}(Fr,De)$ be the loss of Gumbel
STE, when two agents is fed with $Fr$ and the ground truth German translation
$De$. Let $\mathcal{L}^{\operatorname*{\text{CE}}}(X,Y)$ to be the supervised
training loss with source $X$ and target $Y$. Then for each interactive
training step, we have for both agents

$\mathcal{L}_{sender}^{\operatorname*{\text{S2P}}2P}=\mathcal{L}^{\operatorname*{\text{GSTE}}}(\text{Fr}^{ft},\text{De}^{ft})+\alpha\mathcal{L}^{\operatorname*{\text{CE}}}(\text{Fr}^{pre},\text{En}^{pre})$
(2)
$\mathcal{L}_{receiver}^{\operatorname*{\text{S2P}}2P}=\mathcal{L}^{\operatorname*{\text{GSTE}}}(\text{Fr}^{ft},\text{De}^{ft})+\alpha\mathcal{L}^{\operatorname*{\text{CE}}}(\text{En}^{pre},\text{De}^{pre})$
(3)

(a) Bleu En

(b) Cosine Similarity

Figure 4: Cosine similarity bewteen
$\mathcal{L}^{\operatorname*{\text{CE}}}_{pretrain}$ and
$\mathcal{L}^{\operatorname*{\text{INT}}}$ when $\alpha=0.7$

##  Appendix B Translation Game Implementation Details

We here report the experimenatl protocol from We use the Moses tokenizer Koehn
et al. (2007) and we learn a byte-pair-encoding Sennrich et al. (2016) from
Multi30K with all language. Then the same BPE is applied to different dataset.
Our vocab size for En, Fr, De is 11552, 13331, and 12124. Our pretraining
datasets are IWSLT while the finetuning datasets are Multi30K. Our language
model is trained with captions data from MSCOCO Lin et al. (2014). For image
ranker, we use the captions in Multi30K as well as the original Flickr30K
images. We use a ResNet152 with pretrained ImageNet weights to extract the
image features. We also normalize the image features. We follow the
pretraining and model architecture from work Lu et al. (2020).

##  Appendix C Hyper-parameters

During finetuning, we set Gumbel temperature to be 0.5 and follow the previous
work Lu et al. (2020) for other hyperparameters, e.g. learning rate, batch
size, etc. We list our hyper-parameters and our sweep:

Name | Sweep
---|---
$k_{1}$ | 3000, 4000
$k_{2}$ | 200, 300, 400
$k_{2}^{\prime}$ | 200, 300, 400
$\alpha$ | 0, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7

We mainly use P100 GPU for our experiments. For training 200k steps, Gumbel
takes 17 hours, S2P takes 24 hours, SIL takes 18 hours and SSIL takes 24
hours. The best hyperparameters for SIL are
$k_{1}=3000,k_{2}=200,k_{2}^{\prime}=300$. The best $alpha$ for $S2P$ is $1$,
while for $SSIL$ we choose $\alpha=0.5$.

##  Appendix D S2P Details

We show the results of S2P with varying $\alpha$ in Figure 5. In general, one
can find that for S2P there is a trade-off between grounding score and task
score controlled by $\alpha$. A larger $\alpha$ might delay the eventual
collapse. However, if the $\alpha$ is too large, the task score will decrease
significantly. As a result, even though increasing $\alpha$ seems to fit the
intuition, it cannot fix the problem.

(a) BLEU De (Task Score)

(b) BLEU En

Figure 5: S2P with different $\alpha$. Increased $\alpha$ might delay or
remove the late-stage collapse, but it might be at the cost of task score.

(a) BLEU De (Task Score)

(b) BLEU En

(c) NLL

(d) RealNLL

Figure 6: Mix with Pretraining data in SIL.

(a) BLEU De (Task Score)

(b) BLEU En

(c) NLL

(d) RealNLL

Figure 7: SSIL with different $\alpha$

(a) BLEU De (Task Score)

(b) BLEU En

Figure 8: Effect of $k_{2}$ for MixData.$\alpha=0.2$

(a) BLEU De (Task Score)

(b) BLEU En

Figure 9: Effect of $\alpha$ for MixData. $k_{2}=100$