# Pix2NeRF: Unsupervised Conditional $\pi$-GAN for Single Image to Neural
Radiance Fields Translation

Shengqu Cai
ETH Zürich
Anton Obukhov
ETH Zürich
Dengxin Dai
MPI for Informatics
ETH Zürich
Luc Van Gool
ETH Zürich
KU Leuven

###### Abstract

We propose a pipeline to generate Neural Radiance Fields (NeRF) of an object
or a scene of a specific class, conditioned on a single input image. This is a
challenging task, as training NeRF requires multiple views of the same scene,
coupled with corresponding poses, which are hard to obtain. Our method is
based on $\pi$-GAN, a generative model for unconditional 3D-aware image
synthesis, which maps random latent codes to radiance fields of a class of
objects. We jointly optimize (1) the $\pi$-GAN objective to utilize its high-
fidelity 3D-aware generation and (2) a carefully designed reconstruction
objective. The latter includes an encoder coupled with $\pi$-GAN generator to
form an auto-encoder. Unlike previous few-shot NeRF approaches, our pipeline
is unsupervised, capable of being trained with independent images without 3D,
multi-view, or pose supervision. Applications of our pipeline include 3d
avatar generation, object-centric novel view synthesis with a single input
image, and 3d-aware super-resolution, to name a few.

Figure 1:  Overview of Pix2NeRF: We propose a method for unsupervised learning
of neural representations of scenes, sharing a common pose prior. At test
time, Pix2NeRF disentangles pose and content from an input image and renders
novel views of the content. Top: $\pi$-GAN is trained on a dataset without
pose supervision. Bottom: a trained model is conditioned on a single image to
obtain pose-dependent views.

##  1 Introduction

††Corresponding author: Shengqu Cai (shecai@student.ethz.ch)††Code (coming
soon): https://github.com/HexagonPrime/Pix2NeRF

Following the success of Neural Radiance Fields (NeRF) [23], encoding scenes
as weights of multi-layer perceptrons (MLPs) has emerged as a promising
research direction. Novel View Synthesis is an important application: given
sparse sample views of a scene, the task is to synthesize novel views from
unseen camera poses. NeRF addresses it by encoding color and volume density at
each point of the 3D scene into a neural network and uses traditional volume
rendering to compose 2D views.

While NeRF is capable of synthesizing novel views with high fidelity, it is
often impractical due to being “overfitted” to a given scene and requiring
multiple views of the scene to train. Several follow-up works attempt to
address these limitations via making NeRF generalize to new scenes. Major
progress has been made in training a general NeRF capable of encoding a scene
given only one or a handful of views [46, 40, 16, 41, 7, 5]. However, these
works are designed to work well only with multi-view images during either
training or both training and inference.

One reason why single-shot NeRF, or in general single-shot novel view
synthesis is challenging, is the incomplete content information within a
single image. For example, given a frontal image of a car, there is very
little information to infer a novel view from the back directly. Bringing back
the traditional inverse graphics and 3D reconstruction pipelines, [44]
addresses this issue by making an additional assumption on the symmetry of the
scene to interpolate potentially missing geometry information within a single
image. However, this technique is limited to scenes where symmetry can be
introduced and does not tackle the general case.

Therefore, a natural follow-up question is how does a human brain address such
a challenging task? One of the approaches we use unconsciously is learning a
prior implicit model for object categories and mapping what we observe to the
learned model. This line of thinking is already explored in prior works [46,
40]. An essential part missing from these works is ensuring that novel views
also meet our expectation of the object class, and due to the lack of
supervision from a sole image, this is normally done via imagination.

One of the closest forms of imagination developed by the machine learning
community is Generative Adversarial Networks [13]. GANs have been very
successful in image synthesis and transformation. Beyond 2D, studies have
shown GAN’s capability of synthesizing 3D content [24] from natural images.
This suggests another approach to address 3D reconstruction without multi-view
images via 3D GAN inversion. Such a strategy bypasses the problem of missing
information within one sole image due to GAN’s adversarial training. Existing
works [31, 47] utilize such a method based on HoloGAN [24], StyleGAN [47], and
others, but one of the drawbacks naturally from these 3D-aware generative
models is their relatively weak 3D consistency.

With the rapid increase of NeRF [23] popularity, corresponding generative
models are also gaining attention. GRAF [35] and $\pi$-GAN [2] follow
traditional GAN settings by mapping latent codes to category-specific radiance
fields. These generative models typically have high 3D consistency due to the
built-in volumetric rendering design. This observation suggests the
possibility of few-shot 3D reconstruction using adversarial training and
radiance fields.

In this paper, we formulate the task of translating an input image of a given
category to NeRF as an end-to-end pipeline termed Pix2NeRF (Fig. 1). The
method can perform novel view synthesis given a single image, without the need
of pre-training, annotation, or fine-tuning. Pix2NeRF can be trained with
natural images – without explicit 3D supervision, in an end-to-end fashion.
Inspired by prior works [46, 40, 31], we introduce an encoder mapping a given
image to a latent space. We jointly optimize several objectives. First, we
train $\pi$-GAN and the added encoder to map generated images back to the
latent space. Second, we adapt the encoder coupled with $\pi$-GAN’s generator
to form a conditional GAN, trained with both adversarial and reconstruction
loss. We show that merely doing $\pi$-GAN inversion is challenging and
insufficient to complete our goal, and adaptation is important for calibrating
learned representations of the encoder and generator. Our framework is able to
instantiate NeRF in a single shot manner while naturally preserving the
ability to synthesize novel views with high fidelity, comparable to state-of-
the-art generative NeRF models.

#### Contributions

  * –

We propose Pix2NeRF, the first unsupervised single-shot NeRF model, that can
learn scene radiance fields from images without 3D, multi-view, or pose
supervision.

  * –

Our pipeline is the first work on conditional GAN-based NeRF, or in general,
NeRF-based GAN inversion. We expect our pipeline to become a strong baseline
for future works towards these research directions.

  * –

We demonstrate the superiority of our method compared with naive GAN inversion
methods and conduct an extensive ablation studies to justify our design
choices.

##  2 Related works

Our work can be classified as a category-specific 3D-aware neural novel view
synthesis method, which is strongly based on NeRF [23] and $\pi$-GAN [2].

#### Neural scene representations

The field of encoding a scene into neural networks has proven to be a
promising research direction. This includes, but is not limited to:
parameterizing the geometry of a scene via signed distance functions or
occupancy [28, 22, 6, 36], encoding both geometry and appearance [38, 26, 18,
33], etc. Recently, the impressive performance of Neural Radiance Fields
(NeRF) [23] has drawn extensive attention to this field. It encodes a scene as
a multivariable vector-valued function $f(x,y,z,\theta,\phi)=(r,g,b,\sigma)$
approximated by MLP, where $(x,y,z)$ denotes spatial coordinates,
$(\theta,\phi)$ denotes viewing direction, and $(r,g,b,\sigma)$ corresponds to
color and volume density. This function is then called repeatedly by any of
the volume rendering techniques to produce novel views. The outstanding
performance of NeRF inspired follow-up works to extend it towards alternative
settings, such as training from unconstrained images [20], training without
poses [43, 21], etc.

#### NeRF-based GANs

Following the developments of GANs and NeRFs, several works tried combining
them to form generative models producing NeRFs. One of the first attempts in
this direction is GRAF [35]; it performs category-specific radiance fields
generation by conditioning NeRF on shape and appearance code. Following the
NeRF pipeline, the generator can synthesize an image given a random code and a
view direction. The generated image is passed into the discriminator together
with real images, thus implementing a GAN. GRAF is an unsupervised model,
since it does not require ground truth camera poses; therefore, it can be
trained using ”in the wild” images. This is done by introducing a pose prior
relative to a canonical view frame of reference, e.g., Gaussian distribution
to describe head pitch and yaw relative to a front face view. $\pi$-GAN [2] is
similar to GRAF, but conditions on a single latent code and utilizes FiLM [30,
10] SIREN [37] layers instead of simple MLPs. More recently, several works
improved synthesis quality with high resolutions [14], better 3D shapes [45],
and precise control [25, 48].

#### Few-shot NeRF

The main property of NeRFs is the ability to bake in a 3D scene into MLP
weights. However, this is also a limitation since it must be retrained for
each new scene, which takes a lot of time and money. To lift this constraint,
PixelNeRF [46] and GRF [40] condition MLPs on pixel-aligned features extracted
by a CNN encoder. During the novel view rendering phase, 3D points along the
rays are projected onto the extracted feature grid to get aligned features,
then fed into an MLP with the points. More recently, CodeNeRF [16] suggested
training NeRF with learnable latent codes and utilizing test-time optimization
to find the best latent codes (and camera poses) given an image. However,
these methods still require multi-view supervision during training, which
constrains their usage in real-world settings, where multi-view datasets are
challenging to collect.

Therefore, single-shot NeRF without additional supervision (e.g., 3D objects,
multi-view image collections) remains an under-explored research direction. In
this paper, we bridge this gap by incorporating an auto-encoder architecture
into an existing $\pi$-GAN NeRF framework to obtain a conditional single-shot
NeRF model, retaining the best properties of all components. We note that the
concurrent work [31] shares similar ideas. The key differences are a different
backbone network (HoloGAN [24]) and its lack of 3D consistency, which the
authors point out. Contrary, we utilize the newly-proposed NeRF-based GAN
method called $\pi$-GAN [2], which naturally provides stronger 3D consistency
by design. We demonstrate that merely applying the approach of [31] is
insufficient to obtain an accurate mapping from image to latent space with
$\pi$-GAN as a backbone. Nevertheless, our framework can be viewed as [31]
specifically improved towards NeRF-based GAN models, or CodeNeRF [16] combined
with GANs.

Figure 2:  Overview of building blocks and objectives, used in Pix2NeRF. GAN
objectives follow $\pi$-GAN [2] and ensure that NeRF outputs match the
distribution of real images $p_{\mathrm{real}}$ under the latent prior $p_{z}$
and pose prior $p_{d}$. Reconstruction and GAN inversion objectives ensure
calibrated latent representations, such that $E$ and $G$ can operate as an
auto-encoder, similar to [31]. The conditional adversarial objective enables
learning better representations without explicit pose supervision. Legend:
green - trained module, blue - frozen, gradient - warm-up.

##  3 Method

Pix2NeRF consists of three neural networks, a Generator $G$, a Discriminator
$D$, together forming a Generative Adversarial Network, and an Encoder $E$
forming an auto-encoder together with $G$. The generator is conditioned on the
output view pose $d$ and a latent code $z$, broadly describing content
variations, such as color or shape. It employs 3D-volume rendering techniques
and outputs a single parameterized scene view as RGB image $I$. The
discriminator $D$ is a CNN, which simultaneously predicts distribution origin
of the input RGB image via logit $l$ (real – “in the wild”, or fake –
generated by $G$), and the corresponding scene pose $d$. The encoder $E$ is a
CNN tasked to map an input image onto the latent manifold, learned by $G$, and
at the same time predict the input’s pose:

$\displaystyle\begin{split}G&:z,d\rightarrow I\\\ D&:I\rightarrow l,d\\\
E&:I\rightarrow z,d.\end{split}$ (1)

Functionally, Pix2NeRF extends $\pi$-GAN [2] with the encoder $E$ trained
jointly with the GAN to allow mapping images back to the latent manifold.
Because the encoder $E$ disentangles the content $z$ and the pose $d$ of the
input $I$, content can be further used to condition the $\pi$-GAN generator
$G$ and obtain novel views by varying the rendered pose $d$.

Having defined network modules, we turn to specifying the inputs and outputs
of the modules. The latent code $z$ comes from a simple prior distribution
$p_{z}$ (multivariate uniform in our case) – it makes sampling random codes
$z_{\mathrm{rand}}$ easy and lets us design $E$ such that it can encode any
input image $I$ into some $z_{\mathrm{pred}}$ within the support of $p_{z}$.
Following prior art [2, 35], the unsupervised setting we operate in assumes we
have access to the prior distribution of poses $p_{d}$ of real images
$I_{\mathrm{real}}\sim p_{\mathrm{real}}$ used for training. Depending on the
dataset and choice of pose coordinates, it can be multivariate Gaussian with
diagonal covariance (for images of faces) or uniform on a (hemi-)sphere (for
images of cars). Parameters of this distribution must be known to allow easy
sampling random poses $d_{\mathrm{rand}}$ for the generator, and that $p_{d}$
is representative of poses of real images $I_{\mathrm{real}}$.

Simply training the encoder $E$ to map an image $I$ into GAN latent space (as
in Stage 1 of [31]) simultaneously with training GAN is challenging. This is
because the encoder needs to correctly map images of the same scene from
different views to a single latent code. This is especially hard when these
views contain variations of fine details due to occlusions. As seen from Eq. 1
and the design Fig. 2, our method disentangles latent representation of image
mapped by the encoder and generator input into content $z$ and pose $d$, which
undergo separate treatment.

Given an input image, Pix2NeRF disentangles pose and content and produces a
radiance field of the content, which is (1) consistent with the input under
the disentangled pose and (2) consistent and realistic under different poses
from $p_{d}$. To achieve these properties, we devise several training
objectives for (1) generator, (2) discriminator, (3) GAN inversion, (4)
reconstruction, and (5) conditional adversarial training.

These objectives are used to compute gradients for parameters of $G$, $D$, and
$E$ within a single optimization process. However, certain parts remain
“frozen” during optimizer updates (such as $G$ during $D$ updates and vice-
versa); we denote them with an asterisk in equations (e.g., $G^{*}$) and blue
color in Fig. 2. We empirically find that training encoder from the start has
a detrimental effect on the whole pipeline and employ a warm-up strategy
(denoted with green-blue transitions), explained further.

###  3.1 GAN generator objective

The generator is trained to “fool” the discriminator by serving it
progressively realistic images. Pix2NeRF follows the same procedure of
training the generator as $\pi$-GAN: it samples latent codes
$z_{\mathrm{rand}}\sim p_{z}$ and random poses $d_{\mathrm{rand}}\sim p_{d}$
in pairs, which are then passed through the generator to obtain fake generated
images:

$\displaystyle I_{\mathrm{gen}}=G(z_{\mathrm{rand}},d_{\mathrm{rand}}),$ (2)

which are further fed into the frozen discriminator:

$\displaystyle l_{\mathrm{gen}},d_{\mathrm{gen}}$
$\displaystyle=D^{*}(I_{\mathrm{gen}}).$ (3)

Following [2], another component helpful to the stability and performance of
GAN training is MSE supervision of predicted poses $d_{\mathrm{gen}}$ of
images generated with $d_{\mathrm{rand}}$. It penalizes the generator if the
image pose recovered by the discriminator does not correspond to the sampled
pose, thus setting the goal of learning a “canonical” 3D space. This is
especially helpful if the pose distribution of real data is noisy, such as
seen in CelebA [19].

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{GAN}}(G)=\mathop{\mathbb{E}}_{\begin{subarray}{c}z_{\mathrm{rand}}\sim
p_{\mathrm{z}}\\\ d_{\mathrm{rand}}\sim
p_{d}\end{subarray}}\Big{[}&\mathrm{softplus}\left(-l_{\mathrm{gen}}\right)\
+\\\ &\lambda_{\mathrm{pos}}\left\lVert
d_{\mathrm{rand}}-d_{\mathrm{gen}}\right\rVert_{2}^{2}\Big{]},\end{split}$ (4)

where $\lambda_{\mathrm{pos}}$ is a tuned weighting factor.

###  3.2 GAN discriminator objective

The discriminator is trained to distinguish between the generated fake samples
and real data sampled from the dataset. Pix2NeRF follows the exact procedure
of training the discriminator in $\pi$-GAN: it samples latent codes
$z_{\mathrm{rand}}\sim p_{z}$ and random poses $d_{\mathrm{rand}}\sim p_{d}$
in pairs, which are then passed through the frozen generator to obtain fake
generated images:

$I_{\mathrm{gen}}=G^{*}(z_{\mathrm{rand}},d_{\mathrm{rand}}).$ (5)

The discriminator is then trained using these generated images
$I_{\mathrm{gen}}$ and real images $I_{\mathrm{real}}\sim p_{\mathrm{real}}$:

$\displaystyle\begin{split}l_{\mathrm{real}},d_{\mathrm{real}}&=D(I_{\mathrm{real}}),\\\
l_{\mathrm{gen}},d_{\mathrm{gen}}&=D(I_{\mathrm{gen}}).\end{split}$ (6)

The discriminator objective modified to take into account MSE supervision over
the known pose can then be formulated as follows:

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{GAN}}(D)=\mathop{\mathbb{E}}_{I_{\mathrm{real}}\sim
p_{\mathrm{real}}}\big{[}&\mathrm{softplus}\left(-l_{\mathrm{real}}\right)\big{]}\
\ +\\\ \mathop{\mathbb{E}}_{\begin{subarray}{c}z_{\mathrm{rand}}\sim
p_{\mathrm{z}}\\\ d_{\mathrm{rand}}\sim
p_{d}\end{subarray}}\,\Big{[}&\mathrm{softplus}\left(l_{\mathrm{gen}}\right)\
+\\\ &\lambda_{\mathrm{pos}}\left\lVert
d_{\mathrm{rand}}-d_{\mathrm{gen}}\right\rVert_{2}^{2}\Big{]},\end{split}$ (7)

where $\lambda_{\mathrm{pos}}$ is a tuned weighting factor.

###  3.3 GAN inversion objective

The encoder $E$ is jointly optimized with the discriminator $D$ and reuses
$I_{\mathrm{gen}}$ computed for GAN discriminator objective Eq. (5):

$\displaystyle z_{\mathrm{pred}},d_{\mathrm{pred}}=E(I_{\mathrm{gen}}).$ (8)

This objective aims to ensure consistency between the sampled content and pose
and those extracted from the generated image by the encoder. This is done
using the MSE loss:

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{GAN^{-1}}}(E)=\mathop{\mathbb{E}}_{\begin{subarray}{c}z_{\mathrm{rand}}\sim
p_{\mathrm{z}}\\\ d_{\mathrm{rand}}\sim
p_{d}\end{subarray}}\big{[}&\left\lVert
z_{\mathrm{pred}}-z_{\mathrm{rand}}\right\rVert_{2}^{2}\ \ +\\\ &\left\lVert
d_{\mathrm{pred}}-d_{\mathrm{rand}}\right\rVert_{2}^{2}\big{]}.\end{split}$
(9)

Up until now, the objectives only ensured a generative mapping from the latent
space to radiance fields and some basic form of consistency to learn auto-
encoder. However, our experiments show that optimizing just these three
objectives does not produce a reasonable mapping. Therefore, Pix2NeRF adds two
more objectives to address reconstruction quality and 3D consistency in the
unsupervised setting.

###  3.4 Reconstruction objective

While the GAN inversion objective promotes consistency in latent space,
nothing so far directly promotes consistency in the image space. To this end,
we condition the generator $G$ on a real image by extracting its latent code
and pose prediction using the encoder, and then render its view using the
predicted pose:

$\displaystyle\begin{split}z_{\mathrm{pred}},d_{\mathrm{pred}}&=E\left(I_{\mathrm{real}}\right)\\\
I_{\mathrm{recon}}&=G\left(z_{\mathrm{pred}},d_{\mathrm{pred}}\right).\end{split}$
(10)

Ideally, we expect to get back the original image. However, using MSE loss
alone in the image space is known to promote structural inconsistencies and
blur. In line with [31], we employ Structural Similarity Index Measure loss
(SSIM [42]) with weighting factor $\lambda_{\mathrm{ssim}}$ and a perceptual
loss (VGG [44]) with weighting factor $\lambda_{\mathrm{vgg}}$. We can
therefore aggregate the reconstruction loss as follows:

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{recon}}(G,E)=\mathop{\mathbb{E}}_{I_{\mathrm{real}}\sim
p_{\mathrm{real}}}\Big{[}\left\lVert
I_{\mathrm{recon}}-I_{\mathrm{real}}\right\rVert^{2}_{2}\ +\\\
\lambda_{\mathrm{ssim}}\mathcal{L}_{\mathrm{ssim}}\left(I_{\mathrm{recon}},I_{\mathrm{real}}\right)\
+\\\
\lambda_{\mathrm{vgg}}\mathcal{L}_{\mathrm{vgg}}\left(I_{\mathrm{recon}},I_{\mathrm{real}}\right)\Big{]}.\end{split}$
(11)

###  3.5 Conditional adversarial objective

The reconstruction objective promotes good reconstruction quality for just one
view extracted by the encoder $E$. This may push the combination of networks
towards either predicting trivial poses or unrealistic reconstructions for
other poses from $p_{d}$. To alleviate that, we further apply an adversarial
objective while conditioning the generator on an image $I_{\mathrm{real}}$
when it is rendered from random poses. Reusing results from Eq. (10),

$\displaystyle\begin{split}l_{\mathrm{cond}},d_{\mathrm{cond}}&=D^{*}(G\left(z_{\mathrm{pred}},d_{\mathrm{rand}}\right))\\\
\mathcal{L}_{\mathrm{cond}}(G,E)&=\mathop{\mathbb{E}}_{\begin{subarray}{c}I_{\mathrm{real}}\sim
p_{\mathrm{real}}\\\ d_{\mathrm{rand}}\sim
p_{d}\end{subarray}}\big{[}\mathrm{softplus}\left(-l_{\mathrm{cond}}\right)\big{]}.\end{split}$
(12)

###  3.6 Encoder warm-up

As pointed out in [31], reconstruction loss may easily dominate and cause the
model overfitting towards input views while losing its ability to represent
3D. We, therefore, introduce a simple “warm-up” strategy to counter this
issue. For the first half iterations of the training protocol, we freeze the
encoder while optimizing reconstruction and conditional adversarial loss and
optimize only the generator for these two objectives. This serves as a warm-up
for the generator to roughly learn the correspondence between encoder outputs
and encoded images. The encoder is then unfrozen, enabling further
distillation of its learned representations.

After the warm-up stage, the encoder and generator directly form a pre-trained
auto-encoder capable of producing 3D representations close to ground truth,
bypassing the cumbersome early-stage reconstruction objective, which is
extremely hard to balance with GAN objectives. We show the necessity of this
strategy and comparison with merely assigning a smaller weight for
reconstruction loss in the ablation studies.

###  3.7 Training and Inference

The objectives mentioned above can be trained jointly; however, we optimize
them in alternative iterations due to GPU memory constraints. The
discriminator and GAN inversion objectives are optimized upon every iteration;
the GAN generator objective is optimized on even iterations; reconstruction
and conditional adversarial objectives are optimized jointly during odd
iterations with weighting factor $\lambda_{\mathrm{recon}}$:

$\displaystyle\mathcal{L}_{\mathrm{odd}}=\mathcal{L}_{\mathrm{cond}}+\lambda_{\mathrm{recon}}\mathcal{L}_{\mathrm{recon}}.$
(13)

During the inference stage, Pix2NeRF only requires a single input image, which
can be fed into the encoder $E$ and then generator $G$, coupled with
arbitrarily selected poses for novel view synthesis. At the same time, instead
of obtaining the latent code $z$ from the encoder, it is possible to sample it
from the prior distribution $p_{z}$, to make the model synthesize novel
samples like a $\pi$-GAN.

##  4 Experiments

###  4.1 Evaluation

Figure 3:  Reconstructions and novel views on CARLA [8], CelebA [19], and
ShapeNet-SRN [4, 38] chairs. See Appendix for more results.

#### Datasets

We train and evaluate our pipeline on several 3D datasets listed below. CelebA
[19] is a dataset of over 200k images of celebrity faces. We use its “aligned”
version and apply center cropping to keep the face area roughly. We hold out
8k images as the test set. CARLA [8] contains 10k images of 16 car models
rendered with Carla driving simulator with random textures. ShapeNet-SRN is a
dataset hosted by the authors of SRN [38], from which we use the “chairs”
split for the comparison with prior multi-view methods. The dataset contains
50 rendered views from ShapeNet [4] with Archimedean spiral camera poses for
each of the 6591 instances. As the ShapeNet-SRN dataset does not include the
lower hemisphere in its validation and test sets, we filter the training set
to contain only the upper hemisphere as well.

#### Evaluation metrics

Pix2NeRF is evaluated in two modes: unconditional, which assumes sampling
directly from $p_{z}$ and $p_{d}$, and conditional, which corresponds to using
$z=E(I_{\mathrm{real}}),\ I_{\mathrm{real}}\sim p_{\mathrm{real}}$, while
still sampling from $p_{d}$. For “in the wild” datasets, as we do not possess
multi-view ground truth images, we resort to reporting generative metrics:
Inception Score (IS) [34], Frechet Inception Distance (FID) [15], and Kernel
Inception Distance (KID) [1] with scaling factor $\times 100$ following the
steps of prior works [2, 35] using the implementation [27]. To compare with
multi-view-based novel view synthesis methods on Shapenet-SRN, we follow the
evaluation protocols in pixelNeRF and CodeNeRF and report PSNR (Peak Signal to
Noise Ratio) and SSIM (Structural Similarity Index Measure) [42].

#### Technical details

We choose the latent code prior distribution $p_{z}$ as a multivariate uniform
on $[-1,1]$. We build our model on top of the $\pi$-GAN implementation in
PyTorch [29], re-using its released generator and discriminator architectures.
We also use the discriminator architecture as the backbone of our encoder,
where we add a $tanh$ at the end of the latent code head. All models are
optimized with Adam [17] optimizer for 300k iterations, which is approximately
the same computational cost to obtain a $\pi$-GAN model. CelebA [19] models
are trained with batch size 48 on resolution 64$\times$64, where we sample 24
points per ray. We use learning rates of 2e-4, 6e-5, and 2e-4 for
discriminator, generator, and encoder, respectively.

| 64 $\times$ 64 | 128 $\times$ 128
---|---|---
Method | FID $\downarrow$ | KID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | IS $\uparrow$
HoloGAN [24] | - | 2.87 | - | 39.7 | 2.91 | 1.89
GRAF [35] | - | - | - | 41.1 | 2.29 | 2.34
$\pi$-GAN [2] | 5.15 | 0.09 | 2.28 | 14.7 | 0.39 | 2.62
Pix2NeRF unconditional | 6.25 | 0.16 | 2.29 | 14.82 | 0.91 | 2.47
Pix2NeRF conditional | 24.64 | 1.93 | 2.24 | 30.98 | 2.29 | 2.20

Table 1:  Quantitative results on CelebA [19].

| 64 $\times$ 64 | 128 $\times$ 128
---|---|---
Method | FID $\downarrow$ | KID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | KID $\downarrow$ | IS $\uparrow$
HoloGAN [24] | 134 | 9.70 | - | 67.5 | 3.95 | 3.52
GRAF [35] | 30 | 0.91 | - | 41.7 | 2.43 | 3.70
$\pi$-GAN [2] | 13.59 | 0.34 | 3.85 | 29.2 | 1.36 | 4.27
Pix2NeRF unconditional | 10.54 | 0.37 | 3.95 | 27.23 | 1.43 | 4.38
Pix2NeRF conditional | 12.06 | 0.44 | 3.81 | 38.51 | 2.37 | 3.89

Table 2:  Quantitative results on CARLA [8].

Method | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---
GRF* [40] | 21.25 | 0.86
TCO* [39] | 21.27 | 0.88
dGQN* [12] | 21.59 | 0.87
ENR* [11] | 22.83 | -
SRN** [38] | 22.89 | 0.89
PixelNeRF* [46] | 23.72 | 0.91
CodeNeRF** [16] | 22.39 | 0.87
Pix2NeRF conditional | 18.14 | 0.84

Method | FID $\downarrow$ | KID $\downarrow$ | IS $\uparrow$
---|---|---|---
HoloGAN [24] | - | 1.54 | -
$\pi$-GAN [2] | 15.47 | 0.55 | 4.62
Pix2NeRF unconditional | 14.31 | 0.51 | 4.62
Pix2NeRF conditional | 17.55 | 0.59 | 4.36

Table 3:  Quantitative results on ShapeNet-SRN [4, 38] chairs. Top:
reconstruction metrics ($128\times 128$). Bottom: generative metrics
($64\times 64$). Legend: * – requires multi-view training data; ** – requires
multi-view training data and test time optimization.

For all other models, we utilized $\pi$-GAN [2]’s progressive training
strategy, starting with training on resolution 32$\times$32 with learning
rates 4e-5, 4e-4, and 4e-4 for generator, discriminator, and encoder,
respectively, with 96 sampled points per ray. We increase to resolution
64$\times$64 with learning rates 2e-5, 2e-4, and 2e-4 for generator,
discriminator, and encoder, respectively, and sample 72 points per ray after
50k iterations. We empirically set $\lambda_{\mathrm{recon}}=5$,
$\lambda_{\mathrm{ssim}}=1$ and $\lambda_{\mathrm{vgg}}=1$ for all datasets.
For CelebA [19], we follow [2] and set $\lambda_{\mathrm{pos}}=15$. For CARLA
[8] and ShapeNet-SRN [4, 38], we set $\lambda_{\mathrm{pos}}=0$ as we do not
observe significant difference. We use $|z|=512$ for CelebA [19] and $|z|=256$
for CARLA [8] and Shapenet-SRN [4, 38].

#### Quantitative results

We show the evaluation on CelebA [19] and CARLA [8] in Tables 1 and 2
respectively. We also show evaluation with the same generative metrics on
ShapeNet-SRN in Table 3 (bottom). We observe that even though our model’s
conditional synthesis is not as good as our backbone $\pi$-GAN (especially on
CelebA), it is on par with other prior 3D view generation methods [35, 24].

Since we do not explicitly enforce prior distribution $p_{z}$ on the encoded
samples $E(I_{\mathrm{real}})$ from $p_{\mathrm{real}}$, the image of
$p_{\mathrm{real}}$ resulting from the encoder mapping may occupy a small
portion in $p_{z}$. Thus, conditioning on $p_{\mathrm{real}}$ naturally leads
to a smaller variation in samples from $p_{z}$, and hence, smaller diversity
of NeRF outputs. For this reason, directly sampling randomly from $p_{z}$
(unconditionally) achieves better performance as measured by the generative
metrics. Additionally, our generator outperforms $\pi$-GAN on most metrics on
CARLA [8] and ShapeNet-SRN [4, 38]. Results on CelebA [19] are less consistent
due to dataset noise (background, geometry, pose noise, artifacts, _etc_.),
encouraging GANs to converge towards the mean as a trade-off to variations.
These observations can be related to manifold learning [9], where we enforce
the existence of a latent code for each real image in the train set.

We compare our method with other single-image 3D inference methods in Table 3
on ShapeNet-SRN [4, 38] in 128 $\times$ 128 resolution. Since our model
assumes a strictly-spherical camera parameterization model, which does not
correspond well to the ground truth poses of ShapeNet-SRN [4, 38], we use our
encoder to extract poses from the images.

Despite being generative, unsupervised, and not requiring test time
optimization in contrast to all other methods, our model’s performance does
not drop much below the competition. Considering that other models were
trained on 128, while our models were trained on 64 $\times$ 64 but rendered
at 128 $\times$ 128 resolution, we observe a super-resolution effect.

#### Qualitative results

We show some qualitative results of our model’s performance on CARLA [8] and
CelebA [19] in Fig. 3. We can see that our model can synthesize novel views
with good quality while existing few-shot NeRF methods [46, 40, 16] are not
able to train on these “in the wild” datasets due to the lack of multi-view
supervision. Our model can also produce decent 3D representations even under
extreme poses and artifacts (see row 5).

###  4.2 Ablation studies

We perform a thorough ablation study to verify our design choices by removing
the key components one by one and training models under identical settings as
the full model. Qualitative results for the following ablations are in Fig. 4;
refer to Appendix for the corresponding quantitative results.

Figure 4:  Qualitative results of ablation studies, obtained with an image
from the test split of CelebA [19]. $\lambda_{\mathrm{recon}}$ is set to 1 for
lower reconstruction weights instead of the warm-up ablation. See Appendix for
results obtained by using other $\lambda_{\mathrm{recon}}$ values.

#### Naive GAN inversion

We compare Pix2NeRF with naive GAN inversion: having a pre-trained GAN, we
freeze its weights and train an encoder to map images to their corresponding
latent codes. The results show that the encoder can learn an approximate
mapping from images to latent code. However, due to the lack of joint
distillation, the reconstruction is off from the input image.

#### Auto-encoder

Another potential approach is to utilize $\pi$-GAN’s architecture as an auto-
encoder, in which the latent space is dropped from the pipeline and training
the reconstruction and conditional adversarial objectives only. Under this
setting, while the reconstruction achieves decent quality, we can observe
visible 3D inconsistency, suggesting difficulty of optimization with the
remaining objectives.

#### No GAN inversion

We proceed with ablations by removing the GAN inversion step from the
pipeline. The visual results turn out to be blurry and uncanny compared with
full settings. One possible explanation is that this step is a connection
between $\pi$-GAN training and reconstruction, which significantly affects the
overall performance.

#### No conditional adversarial objective

We further deactivate the conditional adversarial loss and retrain the model.
As a result, the renderings become incomplete and have clear visual artifacts.
In addition, 3D consistency degrades significantly, which justifies this
objective in the given setting.

#### Warm-up

To verify the effect of the warm-up strategy, we train three separate models
and compare their performances: without warm-up, without unfreezing encoder
(always warm-up), and assigning a lower weight for reconstruction instead of
the warm-up. Without the warm-up strategy, the model tends to overfit the
input view and cannot produce meaningful content from novel poses. If we only
use the warm-up strategy and never unfreeze the encoder, the distillation is
relatively weak, which results in few fine details. With lower reconstruction
weight instead of the warm-up, the balance between reconstruction and
adversarial objective is missing, resulting in mode collapse for novel view
synthesis.

##  5 Conclusions

In this paper, we introduced Pix2NeRF, a novel unsupervised single-shot
framework capable of translating an input image of a scene into a neural
radiance field (NeRF), thereby performing single-shot novel view synthesis.
The key idea of Pix2NeRF is to utilize generative NeRF models to interpolate
missing geometry information. This is accomplished by jointly training an
encoder that maps images to a latent space, which disentangles content and
pose, and the generative NeRF model while keeping these two parts dependent on
each other. Pix2NeRF can go beyond the auto-encoder setting and perform novel
scene generation by sampling random content and pose and passing through the
generator. Our framework demonstrates high reconstruction quality and 3D
consistency, on par and better than previous works.

#### Limitations and future work

The current setting in consideration is limited to one category per dataset
and cannot directly generalize beyond the chosen category. Alternative
research directions include local conditional fields similar to PixelNeRF [46]
and GRF [40], which can generalize to unseen categories, multi-instance, and
even real-world scenes. Being a general framework, Pix2NeRF is not limited to
using $\pi$-GAN as its backbone. Newer generative NeRF models, e.g. EG3D [3]
could potentially achieve better visual quality. Additionally, architecture
search, especially with respect to the encoder remains a challenging problem.
Utilizing more mature encoder architectures from 2D GAN feed-forward inversion
literature, e.g. pixel2style2pixel [32], could potentially improve the
performance of Pix2NeRF significantly.

#### Ethical consideration

As with most modern conditional generative models, Pix2NeRF can be misused by
generating content to spread misinformation or perform targeted attacks. The
growing popularity of deepfake celebrity accounts in social media suggests
that new use cases, markets, and novel ways of monetizing this kind of data
will follow.

## References

  * [1] Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans, 2021.
  * [2] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In arXiv, 2020.
  * [3] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3D generative adversarial networks.  In arXiv, 2021.
  * [4] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  CoRR, abs/1512.03012, 2015.
  * [5] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo, 2021.
  * [6] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [7] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll.  Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7911–7920, June 2021.
  * [8] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  CARLA: An open urban driving simulator.  In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017.
  * [9] Yilun Du, Katherine M. Collins, Joshua B. Tenenbaum, and Vincent Sitzmann.  Learning signal-agnostic manifolds of neural fields, 2021.
  * [10] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio.  Feature-wise transformations.  Distill, 2018.  https://distill.pub/2018/feature-wise-transformations.
  * [11] Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan.  Equivariant neural rendering.  In International Conference on Machine Learning, pages 2761–2770. PMLR, 2020.
  * [12] SMA Eslami, DJ Rezende, F Besse, F Viola, AS Morcos, M Garnelo, A Ruderman, AA Rusu, I Danihelka, K Gregor, et al.  Neural scene representation and rendering.  Science, 360(6394):1204–+, 2018.
  * [13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Y. Bengio.  Generative adversarial networks.  Advances in Neural Information Processing Systems, 3, 06 2014.
  * [14] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis, 2021.
  * [15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a nash equilibrium.  CoRR, abs/1706.08500, 2017.
  * [16] Wonbong Jang and Lourdes Agapito.  Codenerf: Disentangled neural radiance fields for object categories, 2021\.
  * [17] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 12 2014.
  * [18] Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey.  Sdf-srn: Learning signed distance 3d object reconstruction from static images.  In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  * [19] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [20] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In CVPR, 2021.
  * [21] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu.  Gnerf: Gan-based neural radiance field without posed camera.  arXiv preprint arXiv:2103.15606, 2021.
  * [22] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [23] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [24] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In The IEEE International Conference on Computer Vision (ICCV), Nov 2019.
  * [25] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [26] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [27] Anton Obukhov, Maximilian Seitzer, Po-Wei Wu, Semen Zhydenko, Jonathan Kyl, and Elvis Yu-Jing Lin.  High-fidelity performance metrics for generative models in pytorch, 2020\.  Version: 0.3.0, DOI: 10.5281/zenodo.4957738.
  * [28] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  CoRR, abs/1912.01703, 2019.
  * [30] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville.  Film: Visual reasoning with a general conditioning layer.  CoRR, abs/1709.07871, 2017.
  * [31] Pierluigi Zama Ramirez, Alessio Tonioni, and Federico Tombari.  Unsupervised novel view synthesis from a single image.  CoRR, abs/2102.03285, 2021.
  * [32] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [33] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [34] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  CoRR, abs/1606.03498, 2016.
  * [35] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [36] Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein.  Metasdf: Meta-learning signed distance functions.  In arXiv, 2020.
  * [37] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In Proc. NeurIPS, 2020.
  * [38] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In Advances in Neural Information Processing Systems, 2019.
  * [39] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Multi-view 3d models from single images with a convolutional network, 2016\.
  * [40] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d scene representation and rendering.  In arXiv:2010.04595, 2020.
  * [41] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In CVPR, 2021.
  * [42] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [43] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  NeRF–: Neural radiance fields without known camera parameters.  https://arxiv.org/abs/2102.07064, 2021.
  * [44] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In CVPR, 2020.
  * [45] Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai.  Generative occupancy fields for 3d surface-aware image synthesis.  In Advances in Neural Information Processing Systems(NeurIPS), 2021\.
  * [46] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelNeRF: Neural radiance fields from one or few images.  In CVPR, 2021.
  * [47] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler.  Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering.  In International Conference on Learning Representations, 2021.
  * [48] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian.  Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis, 2021.

Pix2NeRF: Unsupervised Conditional $\pi$-GAN for Single Image to Neural
Radiance Fields Translation
Supplementary Material

##  Appendix A Additional qualitative results

We demonstrate additional qualitative results achieved by Pix2NeRF on three
datasets: CelebA [19], Shapenet-SRN chairs [4, 38], and CARLA [8] in Figures
6, 7, and 8 respectively.

##  Appendix B Additional quantitative results

Table 4 provides additional quantitative results on ShapeNet-SRN [4, 38] with
generative metrics computed on 128 $\times$ 128 resolution, and reconstruction
metrics computed on 64 $\times$ 64 resolution. We do not report PSNR and SSIM
for CelebA [19] as there is no ground truth novel views.

##  Appendix C Additional ablation study

We provide quantitative results of each ablation study on CelebA [19] and
Shapenet-SRN [4, 38] to further verify our design choices. As in the ablation
study in our main paper, we report FID [15], KID [1] and IS [34] for CelebA
[19], and additionally report PSNR and SSIM [42] on Shapenet-SRN [4, 18]. We
measure results after inference on resolution 64 $\times$ 64\. We show
quantitative ablation results in Table 5. Legend: A – naive GAN inversion; B –
auto-encoder; C – no GAN inversion; D – no conditional adversarial objective;
E – no warm-up; F – always warm-up; G, H, I, J – lower weights for
reconstruction instead of warm-up, with
$\lambda_{\mathrm{recon}}=1,0.1,0.01,0.001$ respectively. Note that since the
encoder output is not enforced to strictly follow $p_{z}$, naive GAN inversion
(stage 1 in [31]) failed completely due to bad initialization. We therefore
use a “warmed-up” version of the generator trained for 300k iterations.

##  Appendix D Input reconstruction and hybrid optimization

We ran extra ablations and summarized our model performance by providing both
input reconstruction (cols 2,3) and novel view synthesis (cols 4,5,6) results
in Tab. 6 (row 4). We show $\pi$-GAN latent optimization on an input image for
700 iterations, as recommended by its authors in row 3. Note that it requires
time-consuming per-instance optimization due to the NeRF’s rendering
mechanism. Additionally, we use the Pix2NeRF encoder’s output as a starting
point and perform latent optimization with a frozen Pix2NeRF generator for
only 200 iterations, shown in row 5. A qualitative comparison is shown in Fig.
5. Note that our model does not overfit the input view even with 1000
iterations of input view optimization (row 6), while $\pi$-GAN shows strong
artifacts and requires a search for the optimal number of iterations.

##  Appendix E Necessity of generator distilling

We trained the encoder with a pretrained frozen $\pi$-GAN generator using all
the losses. As can be seen from the results in Tab. 6 Row 1, the model
struggles to capture details accurately without fine-tuning the generator
jointly.

##  Appendix F Linear interpolation

We interpolate novel views between two different input images by predicting
their corresponding latent codes and poses, then applying linear interpolation
to get the intermediate codes and poses. We show the results interpolating
five images in Figure 9.

|  128 $\times$ 128 |  64 $\times$ 64
---|---|---
Method |  FID $\downarrow$ |  KID $\downarrow$ |  IS $\uparrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$
Pix2NeRF unconditional | 26.45 | 1.18 | 4.39 | - | -
Pix2NeRF conditional | 26.81 | 1.23 | 4.27 | 18.75 | 0.82

Table 4:  Additional quantitative results on ShapeNet-SRN [4, 38].

|  CelebA 64 $\times$ 64 |  ShapeNet-SRN 64 $\times$ 64
---|---|---
Method |  FID $\downarrow$ |  KID $\downarrow$ |  IS $\uparrow$ |  FID $\downarrow$ |  KID $\downarrow$ |  IS $\uparrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$
A | 28.90 | 2.99 | 1.62 | 34.01 | 1.73 | 3.65 | 15.91 | 0.71
B | 43.19 | 2.84 | 1.33 | 43.06 | 2.49 | 2.92 | 16.27 | 0.71
C | 39.42 | 3.07 | 1.65 | 41.47 | 2.80 | 2.96 | 15.14 | 0.68
D | 33.92 | 2.84 | 1.87 | 35.72 | 1.74 | 3.75 | 16.81 | 0.77
E | 31.31 | 2.75 | 1.95 | 21.67 | 0.89 | 4.35 | 18.03 | 0.79
F | 39.86 | 3.18 | 1.73 | 27.70 | 1.22 | 4.09 | 16.98 | 0.77
G | 73.52 | 7.47 | 1.91 | 27.10 | 1.31 | 4.26 | 17.77 | 0.79
H | 73.03 | 7.08 | 1.97 | 41.11 | 2.27 | 3.34 | 14.98 | 0.74
I | 140.25 | 16.33 | 1.79 | 184.10 | 17.19 | 2.55 | 10.95 | 0.59
J | 168.59 | 18.89 | 1.50 | 266.64 | 30.29 | 1.98 | 10.28 | 0.47
Full | 24.64 | 1.93 | 2.24 | 17.55 | 0.59 | 4.36 | 18.75 | 0.82

Table 5:  Quantitative results of ablation study on CelebA [19] and ShapeNet-
SRN [4, 38]. “Full” denotes Pix2NeRF conditional setup.  Table 6:  Input view
reconstruction (PSNR, SSIM) on a test set, and novel view synthesis (FID,
KID$\times$100, IS).

Method | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | KID$\downarrow$ | IS$\uparrow$
---|---|---|---|---|---
Pix2NeRF $E$ \+ frozen $\pi$-GAN $G$ | 13.04 | 0.46 | 28.25 | 2.97 | 1.52
$\pi$-GAN optimization (200 iterations) | 23.42 | 0.80 | 16.09 | 0.83 | 2.10
$\pi$-GAN optimization (700 iterations) | 24.21 | 0.82 | 17.14 | 0.72 | 2.14
Pix2NeRF (feed-forward) | 17.95 | 0.67 | 24.82 | 1.93 | 2.21
Pix2NeRF (200 iterations) | 27.12 | 0.89 | 12.86 | 0.64 | 2.27
Pix2NeRF (1000 iterations) | 27.73 | 0.90 | 12.01 | 0.62 | 2.30

Figure 5:  Qualitative comparison on CelebA. Top – input, middle –
reconstruction, bottom – novel view synthesis.

##  Appendix G Limitations and failure cases

Despite training on images without pose or 3D supervision, Pix2NeRF can
reconstruct objects from a single image and achieve decent quality. However,
the methodology of using an encoder to encode an entire image into a single
latent code is quite challenging, especially when the dataset is noisy, such
as CelebA [19]. Pix2NeRF cannot always capture fine details accurately. We
observe failure cases when the input is out-of-distribution relative to that
of the training set $p_{\mathrm{real}}$, as shown in Figure 10. It might be
possible to improve these hard cases by introducing pixel-wise features
instead of (or, in addition to) the global latent code, as done in PixelNeRF
[46] and GRF [40].

Figure 6:  Further reconstructions and novel views on CelebA [19].  Figure 7:
Further reconstructions and novel views on ShapeNet-SRN [4, 38].  Figure 8:
Further reconstructions and novel views on CARLA [8].  Figure 9:  Linear
interpolation on CelebA [19].  Figure 10:  Failure cases on CelebA [19] and
ShapeNet-SRN [4, 38].


# Memristive control of plasmon-mediated nonlinear photoluminescence in Au
nanowires

Deepak K Sharma  [  Adrian Agreda  [  Florian Dell’Ova  Konstantin Malchow  [
Gérard Colas-des-Francs  Erik Dujardin  Alexandre Bouhelier  [
<EMAIL_ADDRESS>
###### Abstract

Nonlinear photoluminescence (N-PL) is a broadband photon emission arising from
non-equilibrium electron distribution generated at the surface of metallic
nanostructures by an ultrafast pulsed laser illumination. N-PL is sensitive to
surface morphology, local electromagnetic field strength, and electronic band
structure making it relevant to probe optically excited nanoscale plasmonic
systems. It also has been key to access the complex multiscale time dynamics
ruling electron thermalization. Here, we show that the surface plasmons
mediated N-PL emitted by a gold nanowire can be modified by an electrical
architecture featuring a nanogap. Upon voltage activation, we observe that
N-PL becomes dependent to the electrical transport dynamics and can thus be
locally modulated. This finding brings an electrical leverage to externally
control the photoluminescence generated from metal nanostructures, and
constitutes an asset for the development of emerging nanoscale interface
devices managing photons and electrons.

###### keywords:

Nonlinear photoluminescence, Hot electrons, Plasmonic nanowire, Memristive
devices, Memristor

University] Laboratoire Interdisciplinaire Carnot de Bourgogne, UMR 6303 CNRS,
Université de Bourgogne, 9 Avenue Alain Savary, 21000 Dijon, France Current
address: Institute of Materials Research and Engineering (IMRE), Agency for
Science, Technology and Research (A*STAR), 2 Fusionopolis Way, Innovis #
08-03, Singapore 138634, Republic of Singapore University] Laboratoire
Interdisciplinaire Carnot de Bourgogne, UMR 6303 CNRS, Université de
Bourgogne, 9 Avenue Alain Savary, 21000 Dijon, France Current address:
ELORPrintTec, Allée Geoffroy Saint-Hilaire, F-33600 Pessac, France University]
Laboratoire Interdisciplinaire Carnot de Bourgogne, UMR 6303 CNRS, Université
de Bourgogne, 9 Avenue Alain Savary, 21000 Dijon, France Current address:
Laboratory of Quantum Nano Optics, EPFL Lausanne, 1015 Lausanne, Switzerland
University] Laboratoire Interdisciplinaire Carnot de Bourgogne, UMR 6303 CNRS,
Université de Bourgogne, 9 Avenue Alain Savary, 21000 Dijon, France

##  1 Introduction

The illumination of metallic nanostructures by ultrashort laser pulses at
near-IR wavelengths induces a broadband up-converted emission spanning the
visible spectral regime. Under high-intensity femtosecond excitation, this
nonlinear signal has been shown to be dominated by a thermal luminescence
continuum emitted by out-of-equilibrium surface electrons with transiently
elevated temperatures 1, 2, 3, 4, 5, 6. Indeed, following the absorption of
the light pulse, the temperature of the electron gas at the surface of the
metal can reach thousands of Kelvins for about a picosecond 7, 6 and radiates
before equilibrating with the phonons 8, 9, 10. This nonlinear
photoluminescence (N-PL) has been studied on a wide range of plasmonic
geometries and materials 11, 12, 13, 14, 15, 3, 16, 17, 18.An important
feature is the dramatic signal enhancement observed from hot-spots favored by
the presence of localized resonances (e.g. at nanoparticles 19, 20). Such
local signal enhancements are observed because the luminescence emanates from
hot thermal electrons generated at the surface of the metal 21, 22, 6.
Further, in structures sustaining surface plasmon propagation (e.g. nanowires,
cavities), N-PL has been shown to be spatially distributed throughout the
modal landscape 23, 22. Hence, N-PL has been a key observable to unlock
ultrafast light-matter interactions, and this response found numerous usages.
For instance, it has been used as a luminescent marker in bioimaging 24, 25,
or as an observable to develop plasmonic logic gates26.

The sensitivity of the N-PL to surface electrons and morphology motivates a
closer examination of the signal in the presence of intricate field-driven
surface effects involving atomic-scale dynamics and charge transport. These
nuances serve as crucial elements for advancing the next generation of
nanoscale devices in applications including energy harvesting27, computing and
memory components28, 29, tunable metasurfaces30, and beyond.

In this study, we show that N-PL, spatially distributed by a propagating
surface plasmon in a Au nanowire (AuNW), can be locally sensitive to tiny
conductance changes and atomic-scale restructuration of a nanogap. Under the
application of electrostatic bias pulses across the nanogap, we observe a
progressive evolution of the plasmon-mediated N-PL that eventually leads to a
drastic voltage-driven modulation of its intensity. We establish that the
light emission directly translates the electronic dynamics and atomic-scale
movements within the nanogap.

##  2 Results and discussion

###  2.1 System description

Figure 1: Overview of the AuNW system and its operation: (a) The device
configuration seen from a colorized SEM image of a typical AuNW system. A
tightly focused laser excitation at the left end of the AuNW is used to launch
a propagating surface plasmon at the pump wavelength (810 nm). The distal end
is electrostatically controlled by a tapered electrode placed at 50 nm
distance from the AuNW surface. (b) Spatial distribution of the N-PL. The
signal arises from the extremity directly excited by the focused laser as well
as from the distal end via the mediation of a surface plasmon. Here the bias
is $V=0$ V. (c) Same as in (b) for $V=20$ V. A localized enhancement of the
signal at the bias-controlled end is readily seen. A dashed white line guides
the long axis of the nanowire.

Figure 1(a) is a scanning electron micrograph (SEM) of a AuNW and the in-plane
electrode architecture. The electrical potential of the AuNW is applied via a
first tapered electrode electrically connected to its extremity. A second
tapered Au electrode acting as a ground is adjacent to it with a gap of tens
of nanometers. The fabrication process involves standard electron beam
lithography patterning and thin film deposition ($1~{}\rm{nm}$ Cr, $\sim
90~{}\rm{nm}$ Au) on a standard glass coverslip 22. The AuNW system is then
encapsulated in a polymer layer (Poly(methyl methacrylate) - PMMA, thickness
$\sim 200~{}\rm{nm}$). The PMMA acts as a protective layer against atmospheric
contaminants. The nanowires used in our experiments have cross-section of
approximately $\sim 180~{}\rm{nm}$ in width, $\sim 90~{}\rm{nm}$ in thickness,
and they support a bound surface plasmon mode which propagates at the
excitation wavelength 31. The surface plasmon is launched by edge scattering
when the laser is focused to the left extremity of the AuNW 32. Focusing is
achieved with the help of a high numerical aperture (N.A.) objective (N.A.
$=1.49$). Figure S1 in the Supporting Information presents details and a
schematic of the optical and electrical measurement setup.

Figure 1(b) is a wide-field optical image of the N-PL distribution captured by
a charge-coupled device (CCD) when the AuNW is excited at its left end, and
when no bias is applied ($V=0$ V). The laser is tuned at 810 nm wavelength and
emits 140 fs pulse at a $80~{}\rm{MHz}$ repetition rate. The estimated optical
peak power at the focus is $\sim 7.5~{}\rm{GW/m^{2}}$. The laser and the
second harmonic generated by the Au surface are rejected from the detection
using spectral filtering. A low-intensity halogen lamp illuminates the sample
during the image acquisition in order to visually recognize the footprint of
the system (dotted lines). The image shows a clear N-PL response from the
excitation extremity where the laser intensity is focused, and a second
emission region located at the right end of the AuNW, near the nanogap. Here,
the nanogap is about 50 nm wide and this AuNW extremity is referred to as the
bias-controlled end. The N-PL emitted at the bias-controlled end is produced
by end-face scattering of the surface plasmon traveling in the nanowire at the
pump laser wavelength 22. The N-PL spectrum emitted from the bias-controlled
end is presented in Figure S2 (Supporting Information). The spectrum confirms
that with the laser power used here, N-PL originates from thermal hot-
electrons 5, 6. An estimation of the temperature of the electronic
distribution is inferred from a fit of the spectrum (Figure S2, supporting
information) using Planck’s law 33, 34, 4. We find the electron temperature at
the bias-controlled end to be $T_{e}=1250$ K. This is a temperature somewhat
cooler than previously reported values obtained from spectra measured directly
under the laser focus 33, 4. This is expected because the N-PL at the bias-
controlled end of the nanowire is remotely generated from the laser focus
through the mediation of a lossy surface plasmon. When a positive
electrostatic bias ($V=+20$ V, Figure 1(c)) is applied to the AuNW with
respect to the ground electrode, a$~{}\sim 100\%$ enhancement of the N-PL
signal is observed at the bias-controlled end. The bias modulation of the
plasmon-generated N-PL is visualized, in real-time, when square electrostatic
pulses(+20 V amplitude, 0.5 Hz frequency) are applied as seen in the
Supplementary Movie (M1, Supporting information).

Interestingly, the modulation of the N-PL signal is not observed on a pristine
Au nanogap, but appears after a repeated electrical stress applied to the
nanogap. In the upcoming sections, we delve into this activation process and
explore how a voltage-driven dynamics influences the N-PL at the bias-
controlled end.

###  2.2 Electrical activation of the nanogap

We initiate the activation process with a $V=+2$ V positive electrostatic bias
pulse sequence applied to the pristine Au nanogap. We incrementally increase
the voltage for subsequent pulse sequences. Throughout this activation
process, we simultaneously record both the N-PL emission rate (expressed in
kct/s) from the bias-controlled end (area indicated by the white dashed
rectangle in Figure 1(a)), and the electric current, $I_{\rm{g}}$, flowing
through the nanogap. At the beginning of the stress sequence, there is no
measurable current crossing the 50 nm-wide gap. In Figure 2, we provide a
series of time series illustrating the evolution of these quantities as we
apply square electrostatic bias pulses with a frequency of 0.5Hz to the
nanogap. The series displayed in Figure 1(a) starts with a pulse sequence at
$V=+2$ V followed by one at $V=+3$ V. $I_{\rm{g}}$ remains barely above noise
level ($<$ 1 pA) and is mostly dominated by artefactual capacitive effect
picked up during the leading and trailing edges of the pulse. During this
pulse sequence, the N-PL signal is steady and remains unaffected by the bias.
Figure 2(b) is a time series acquired after repeated stress sequences done
with increasing amplitudes (steps of $V=+1$ V). Here the series starts with
$V=+5$ V. An electric current is now clearly measured with an amplitude
fluctuating during the pulse sequence.

Figure 2: (a-c) Snapshots of the time series showing the evolution of
modulation amplitudes in spatially filtered nonlinear photoluminescence from
bias-controlled end of AuNW and electric current passing though the nanogap
under electrostatic bias pulse sequences. Single-pulse zoom-in of the time
traces are shown at the bottom.

Remarkably, this sequence is featuring a small but correlative modulation of
the N-PL at each voltage pulse (see inset). However, the baseline signal,
measured at $V=0$ V, is erratically involving with time. It has increased from
$\sim 10$ kct/s at the onset of bias application at $t=0$ s to $\sim 150$
kct/s around $t=850$ s under unchanged optical illumination conditions.
Because N-PL is sensitive to the surface defects 35, this close to 15-fold
fluctuation of the baseline underscores bias-induced morphological changes
occurring in the AuNW system. Despite this reshaping of the system during the
sequence affecting the overall response, the small superimposed modulation of
the N-PL with the bias pulses remains observable over the entire frame.

The situation differs in the timelines represented in Figure 2(c) obtained
after a prolonged application of positive pulse sequences. The N-PL baseline
value is now constant over the entire time trace suggesting that transient
field-driven irreversible surface alteration of the system is no longer
occurring after the activation phase. In the steady regime, a $\sim$18 pA
current flowing through the device is measured at each voltage pulse. This is
accompanied by a clear modulation of the N-PL between 75 kct/s at $V=0$ V to
120 kct/s at $V=+16$ V, corresponding to a stable 60% enhancement of the
signal. In an earlier report, we have shown that N-PL intensity increases with
the local electron gas temperature ($T_{e}$) 33. Dwelving on the fact that
$T_{e}$ varies as the inverse square root of the surface electron density,
$N_{e}$, a capacitive accumulation of positive charges at the surface of the
nanowire explains the signal enhancement upon application of positive bias
pulses 33. However, Figure 2(c) features an intriguing behavior when the
voltage polarity of the pulses is reversed at $t=3390$ s. Under pulses of
amplitude $V=-10$ V, the current changes its polarity, but its amplitude
significantly reduced, and the N-PL is no longer affected by the bias. This
outcome is contrary to expectations. Indeed, keeping the capacitive scenario
in mind, this polarity should lead to a periodic quenching of the signal
because of the higher electron density generated at the surface of the
nanowire 33. At $t=3445$ s, $+16$ V voltage pulses are again applied to the
system. The progressive buildup of the current amplitude over the first few
pulses is concomitant to a clear restoration of the N-PL modulation. These
initial observations imply the existence of additional factors influencing the
electrostatic bias control of the N-PL, arguably stemming from intrinsic
conduction dynamics within the nanogap, which we explore in the sections
below.

###  2.3 Effect of conduction dynamics on nonlinear photoluminescence
modulation

The modulation amplitude (calculated as the difference between the average
signal level during a bias pulse and the average baseline signal between
consecutive bias pulses) of the current and the N-PL are clearly correlated.
This is demonstrated in Figure 3(a) where the modulation amplitudes inferred
from Figure 2(c) are plotted against each other for the two time windows
corresponding to positive pulses sequences. A linear relationship is
distinctly visible indicating that the same effect rules the two responses.
The underlying mechanism is also dictating the time dynamics of the signals.
From Figure 2(c), it is clear that the maximum modulation amplitudes for both
the current and the N-PL signals are not instantaneous but are reached after
the application of a few voltage pulses. This progressive build-up of the
modulation is even slower after applying the sequence of negative pulses. A
comparison of the N-PL and current modulation amplitudes with the number of
bias pulses is presented in Figure 3(b). We fit the data using an empirical
charging function of the form $S=a(1-\rm{exp}(-N/\tau))+c$, where $N$ is the
number of pulses, and $S$ is either the current $I_{\rm g}$ or the N-PL
modulation amplitude, $a$ is a weighting factor, $\tau$ is the characteristic
growth coefficient and $c$ accounts for residual offset of the experiment.
$a,\tau$ and $c$ are free parameters of the fit. The results (solid lines in
Figure 3(b)) show that it takes between $N=3$ to $N=4$ pulses to reach maximum
current modulation while 2 pulses are enough to attain the largest N-PL
modulation during the first sequence of positive bias pulse. These figures are
increased to 7 and 8 pulses of positive pulses (positive sequence 2)
respectively after negative pulses have been applied. Such behavior
unambiguously underlines the occurrence of a volatile build up of the
conditions leading to a stable modulation.

Figure 3: (a) Nonlinear photoluminescence (N-PL) modulation amplitude plotted
against the electric current $I_{\rm g}$ for the two positive bias sequences
shown in Figure 2(c). (b) Pulse-to-pulse evolution of the N-PL modulation
amplitude and electric current amplitude for the two positive bias sequences.
Solid lines are fits of the data with with as simple capacitor-like function.
(c) and (d) $I_{\rm g}$ and N-PL intrapulse dynamics averaged over 29
consecutive positive bias pulses. A spurious 25 Hz component has been filtered
out from teh N-PL signal. The current response is fitted with capacitor model
in the slow-growth region to extract the time constant (yellow solid line).
N-PL is best fitted with a double exponential after the leading bias edge
(purple line). However, the decaying tail after the falling edge is well
reproduced by as simple picture of a discharging capacitor (green solid line).

To get more insight into the intrapulse dynamics, we have averaged the current
and N-PL over many pulses showing a stable modulation. Figure 3(c) reveals
that after the leading bias edge, $I_{\rm g}$ suddenly sets to about 11 pA and
evolves with a gradual increase over hundreds of ms. The flow of electrical
charges is an unambiguous marker of the formation of conductive channels
within the nanogap 36, 37, akin to typical memristive devices undergoing
volatile resistive changes 38. Upon electrical stress, various processes are
contributing to alter in time the SiO2/PMMA environment composing the nanogap,
including defect creation produced by electron injection (e.g., oxygen
vacancies), Au ions diffusion, clustering and filamentary growth 39, 40, 41.
Indeed, PMMA has been successfully employed as a low-cost easily-processed
organic material promoting filamentary-type resistive switching 42, 43 whereby
electrochemical reactions involving the migration and clustering metallic ions
leads to the formation of an atomic conductive bridge across the gap 38, 44.
Here, the current trace features a behavior mixing the occurrence of a non-
volatile resistive state shown by the instantaneous rise of the current at the
leading edge of the pulses, and an evolutionary nature materialized by a slow-
growth response of hundreds of ms. The instantaneous rise of $I_{\rm g}$
preceding its growth is most probably linked to tunnel transport through the
modified gap. A fit of the slow growth region using a simple model mimicking a
diffusion capacitance (yellow line) yields a characteristic time of 490 ms.
This time response of the current flow can be linked to the build-up of the
conduction pathway in the nanogap under the influence of the applied electric
field and is a strong indicator of a volatile memristive behavior 45, 38. Such
progressive evolution of the transport have been previously experimentally
observed in wide Au nanogaps fabricated on SiO2 and were linked to the
metallic diffusion and filament growth in the gap 45, 46. We confirm the
presence of metal clusters in the nanogap with scanning electron imaging of
similar devices operated in air (i.e. without the PMMA layer) and taken after
extensive testing with positive and negative biases. Images are provided in
Figure S3 within the Supporting Information. They clearly reveal deformed Au
electrodes at the nanogap and the formation of nanometer-sized clusters, which
were absent in the pre-activation SEM image. This confirms the role of field-
driven surface modification on the N-PL baseline measured during the
activation process the system (Figure 2(b)).

Figure 3(d) shows the average N-PL dynamics during a pulse. Unlike the
current, the enhancement of the emitted light is not instantaneous. A
relatively fast raise is followed by a slower progression. A double
exponential fitting (yellow curve) yields a fast exponent at 31 ms and a
slower exponent at 330 ms, a latter value in line with the slow component of
the current dynamics. The N-PL temporal trace shows the fast dynamics occurs
also after the falling edge of the pulse. The N-PL decays to its baseline with
a time constant of 36 ms, a value very close to the fast component measured at
the onset of the bias pulse. Note that this decaying signal is not seen in the
current trace after the bias returns to 0 V (Figure 3(c)). Within the 1 s
pulse duration, the governing mechanism does not allow to reach a steady state
amplitude of either of the two signals. This behavior is clear signature of
the N-PL sensitivity to relaxation dynamics linked to a volatile resistive
evolution and history of the nanogap. We provide in the following an
interpretation how memristive dynamics may be governing the signal. Figure 4
presents three sketches illustrating the activated nanogap under three biasing
conditions: when the AuNW is at a positive potential (a), neutral (b), and
negative potential (c) with respect to the ground tapered Au electrode.

####  2.3.1 Positive bias pulse

When a positive bias pulse is applied to the AuNW (Figure 4(a)), the cathode
is partially oxidized and generates Au ions while negatively charge defects
injected in the nanogap during the activation phase counter-propagate along
the electric field towards their respective counter-electrode. Migration of
metal atoms/ions in the switching matrix follows mutiple complex
electrochemical processes and is driven by several factors such as material
composition and applied field parameters 47. An activation of pristine
nanogaps with only one polarity (Figure S4 of the Supporting Information)
confirms that Au atoms/ions are detached from the positive electrode and are
migrated along the electric field toward the counter electrode. For a positive
bias, pre and post-activation SEM imaging reveals that a granular filament
composed of nanoclusters has grown from the ground electrode and the nanowire
suffered from a reshaping of its extremity. This observation is in line with
previous reports of Au/SiO2/Au systems 46, 45.

In the present experiments, we did not apply any compliance to the electric
current. Hence, complete and stable filament formation is essentially
inhibited due to destructive Joule heating of the conduction path as the
electric current increases. Nonetheless, we have observed complete filament
growth when a 1 G$\Omega$ resistance is placed in series with the nanogap to
limit the maximum current running through the device. As shown in Figure S5 of
supporting information, the current trace evolves to reach as stable switching
pattern characteristic of a fully non-volatile low resistive state indicative
of a metallic bridge formed between the contacts. Remarkably, the modulation
of the N-PL maintained during the buildup phase washes out when the low
resistive state occurs. In this situation, the applied potential is dropped at
the series resistance. Charged species that may be present in the gap are no
longer diffusing and the electron density at the surface of the AuNW remains
unaffected by the bias. Consequently, the N-PL stays constant, as shown in
Figure S5.

Coming back to the situation of a freely-evolving unprotected device, two
factors likely contribute to the gradual increase of the N-PL during the pulse
duration (Figure 3(c)): first, the electric field-driven volatile filamentary
growth within the nanogap leads to a decrease in the effective size of the
nanogap. For the same voltage applied, the electric field strength in the
nanogap is thus stronger. This contributes to lowering the surface charge
density at the AuNW extremity and consequently to an enhancement of the N-PL,
in agreement with our previous work 33. The second contribution may stem from
the increased density of mobile positive Au ions injected in the nanogap from
the AuNW electrode during the voltage pulse. These positive metal ions near
the metal surface give rise to a screened Coulomb potential and an effective
lowering of the electron concentration. This may be the possible reason for
the relatively slow rise of N-PL with a 31 ms time constant at the onset edge
of the bias pulse compared to the current. Such tens of ms time scale are in
the range of diffusion time of Ag clusters in 48. Once the filament gets
thicker the second effect is overtaken by the capacitive effect due to a
smaller gap size.

####  2.3.2 Neutral potential between positive bias pulses

The effect of the volatile nature of the inclusions forming the conductive
pathway on the N-PL manifests itself in the decaying tail of the signal after
the falling edge of the voltage pulse (Figure 3(d)). When the electric field
is no longer applied, metal ions freely diffuse and relax within a sub-second
dynamics and presumably form nanoclusters in the dielectric matrix 48, 49, as
illustrated in Figure 4(b). Freely diffusing Au ions may also be reduced at
negatively charged sites. It is clear that the decaying tail of the N-PL
signal is ruled by such complex relaxation kinetics of Au ions.

####  2.3.3 Negative bias pulse

Upon changing the polarity of the bias, diffusing species already present in
the nanogap migrate in a reverse direction and leading to a dissolution of the
conductive pathways previously built-up 46. This is schematically pictured in
Figure 4(c). The effective widening of the nanogap results in a presumably
smaller capacitive effect on the electron concentration at the surface. This
is confirmed by the absence of the expected N-PL reduction 33. The dissolution
of the conduction channel is corroborated by the much lower electrical current
measured for the negative sequence in Figure 2(c), where the ON conductance
(defined as the ratio of mean electric current during the bias pulses with the
applied voltage) is 0.13 pS compared to 0.93 pS measured during the stable
response under the initial positive pulses. The reversed bias polarity acts as
a partial reset process 50.

Figure 4: Schematic illustrations of positive Au atoms/ions and nanoclusters
dynamics under (a) positive bias pulse, (b) between positive pulses (i.e., no
bias), and (c) negative bias pulse.

###  2.4 Investigating sub-pico Siemens conductance variation effect on N-PL
modulation

in the following, we conduct a systematic examination of the modulation
amplitude of the N-PL with varying conductance states of the nanogap under
electrostatic driving. Our approach involves a control of the metallic
filament growth through the application of sequences of electrostatic pulses.
These pulses are maintained with a constant pulse width of $T_{\rm{ON}}=100$
ms and a voltage amplitude of approximately +20 V. The control parameter is
here interpulse delay, i.e. the duty cycle. Expressed as a percentage, the
duty cycle $\delta$ represents the ratio of the pulse width ($T_{\rm{ON}}$,
fixed as 100 ms) to the period (Tperiod) of the sequence as defined on the top
of Figure 5. For instance, $\delta=2.5\%$ duty cycle allows the filament to
undergo 100 ms of growth followed by a relaxation time of 3.9 s, enabling
self-diffusion of the species contributing to the volatile conductive state.
Conversely, a $50\%$ duty cycle provides also a 100 ms for growth but also for
relaxation. The space accessible for the mobile species is therefore limited
before the next pulse arrives. Hence, a smaller duty cycle leads to a lower
conductive state compared to a larger duty cycle.

Figure 5: (a) Time series of N-PL and electric current response under 100 ms
bias pulse with +20 V amplitude and varying duty cycle
$\delta=T_{\rm{ON}}/T_{\rm{period}}$ shown in the inset at the top. (b) and
(c) Zoomed snapshots of time series for duty cycles $\delta=2.5\%$ and
$\delta=50\%$. (d) N-PL modulation amplitude and average conductance of the
nanogap during the bias pulse with varying duty cycles of applied bias pulses
calculated from different bias pulse sequences of time series presented in
(a).

Figure 5(a) displays the time traces of the electric current and N-PL response
across subsequent varying duty cycles. Clearly, the measurements confirm that
the formation kinetics of the conduction path dictates the amplitude of both
responses. The current level and enhancement of the N-PL remain modest for
small duty cycles (i.e., longer relaxation times), but drastically increases
when TON enables the memristive dynamics to occur. The luminescence baseline
at $V=0$ V remains fairly constant during this long time frame spanning over
400 s. Any drastic surface modification the bias-controlled end of the AuNW
would be seen as a fluctuating signal at $V=0$ V, as in Figure 2(b). Detailed
snapshots of the time series are visually presented in Figures 5(b) and (c)
highlighting discernible differences in N-PL intensities and electric current
for bias pulses with the extreme duty cycles. Figure 5(d) plots the N-PL
modulation amplitude alongside the conductance (defined as the ratio of mean
electric current during the bias pulse to the applied voltage) for varying
duty cycles. Plotted values are averages across the sequences. Here too, the
conductance and the N-PL modulation amplitudes are sharing the same dependence
on the diffusive dynamics of the memristive gap. These findings systematically
underscore the influence of sub-pico Siemens conductance variations resulting
from bias-driven filament growth and relaxation on N-PL emitted from the
extremity of the nanowire.

###  2.5 Conclusion

We presented a comprehensive experimental study of the influence of memristive
dynamics on a plasmon-assisted nonlinear photoluminescence emitted by Au
nanowire optically excited by focused ultrafast laser pulses. Local nonlinear
photoluminescence measurements correlated with electric current flow show that
field-induced dynamics leading to metallic filament formation and dissolution
in the nanogap enables to modulate the intensity of the nonlinear surface
response with a sub-pS sensitivity to conductance variations. These
observations demonstrate that nonlinear photoluminescence can be a sensitive
probe for atomic scale dynamics ruling emerging memristive technologies. We
believe the system architecture explored in the present work for plasmon-
assisted manupilation of electron temperature at electrically controlled
remote sites will be an important platform to explore diverse applications
where external controllability, agility and reconfigurability are required.
They include next generation of electrically-tuned optical antennas 51, and
interface nanoscale devices mixing electronic and photonic functionalities 52,
and hot-electron management and supervision 53 .

This work has been partially funded by the French Agence Nationale de la
Recherche (ANR-20-CE24-0001 DALHAI and ISITE-BFC ANR-15-IDEX-0003), the EIPHI
Graduate School (ANR-17-EURE-0002). Device fabrication and characterization
were performed at the technological platforms ARCEN Carnot and SMARTLIGHT with
the support of the French Agence Nationale de la Recherche under program
Investment for the Future (ANR-21-ESRE-0040), the Région de Bourgogne Franche-
Comté, the European Regional Development Fund (FEDER-FSE Bourgogne Franche-
Comté 2021/2027), the CNRS and the French RENATECH+ network.

The following files are available free of charge.

  * •

Movie M1: Real-time visualization of the hot-electron luminescence modulations
under electrostatic bias pulses.

  * •

Supporting information document: Electrical and optical characterization
setup, Hot-electron luminescence spectrum, SEM images of nanogap for
correlation study, Hot-electron luminescence modulation response for nanogap
transitioning from high resisitve state to conductive state.

## References

  * Haug et al. 2015 Haug, T.; Klemm, P.; Bange, S.; Lupton, J. M. Hot-Electron Intraband Luminescence from Single Hot Spots in Noble-Metal Nanoparticle Films. _Phys. Rev. Lett._ 2015, _115_ , 067403
  * Saavedra et al. 2016 Saavedra, J.; Asenjo-Garcia, A.; García de Abajo, F. J. Hot-electron dynamics and thermalization in small metallic nanoparticles. _Acs Photonics_ 2016, _3_ , 1637–1646
  * Roloff et al. 2017 Roloff, L.; Klemm, P.; Gronwald, I.; Huber, R.; Lupton, J. M.; Bange, S. Light emission from gold nanoparticles under ultrafast near-infrared excitation: Thermal radiation, inelastic light scattering, or multiphoton luminescence? _Nano letters_ 2017, _17_ , 7914–7919
  * Malchow and Bouhelier 2021 Malchow, K.; Bouhelier, A. Photon bunching of the nonlinear photoluminescence emitted by plasmonics metals. _JOSA B_ 2021, _38_ , 576–583
  * Sivan et al. 2023 Sivan, Y.; Un, I. W.; Kalyan, I.; Lin, K.-Q.; Lupton, J. M.; Bange, S. Crossover from Nonthermal to Thermal Photoluminescence from Metals Excited by Ultrashort Light Pulses. _ACS nano_ 2023,
  * Rodríguez Echarri et al. 2023 Rodríguez Echarri, A.; Iyikanat, F.; Boroviks, S.; Mortensen, N. A.; Cox, J. D.; García de Abajo, F. J. Nonlinear Photoluminescence in Gold Thin Films. _ACS Photonics_ 2023, _10_ , 2918–2929
  * Demichel et al. 2016 Demichel, O.; Petit, M.; Viarbitskaya, S.; Mejard, R.; de Fornel, F.; Hertz, E.; Billard, F.; Bouhelier, A.; Cluzel, B. Dynamics, efficiency, and energy distribution of nonlinear plasmon-assisted generation of hot carriers. _ACS Photonics_ 2016, _3_ , 791–795
  * Baida et al. 2011 Baida, H.; Mongin, D.; Christofilos, D.; Bachelier, G.; Crut, A.; Maioli, P.; Del Fatti, N.; Vallée, F. Ultrafast Nonlinear Optical Response of a Single Gold Nanorod near Its Surface Plasmon Resonance. _Phys. Rev. Lett._ 2011, _107_ , 057402
  * Hou et al. 2018 Hou, X.; Djellali, N.; Palpant, B. Absorption of ultrashort laser pulses by plasmonic nanoparticles: not necessarily what you might think. _ACS photonics_ 2018, _5_ , 3856–3863
  * Suemoto et al. 2021 Suemoto, T.; Yamanaka, K.; Sugimoto, N.; Kobayashi, Y.; Otsu, T.; Tani, S.; Koyama, T. Relaxation dynamics of hot electrons in the transition metals Au, Ag, Cu, Pt, Pd, and Ni studied by ultrafast luminescence spectroscopy. _Journal of Applied Physics_ 2021, _130_
  * Beversluis et al. 2003 Beversluis, M. R.; Bouhelier, A.; Novotny, L. Continuum generation from single gold nanostructures through near-field mediated intraband transitions. _Physical Review B_ 2003, _68_ , 115433
  * Imura et al. 2005 Imura, K.; Nagahara, T.; Okamoto, H. Near-field two-photon-induced photoluminescence from single gold nanorods and imaging of plasmon modes. _The Journal of Physical Chemistry B_ 2005, _109_ , 13214–13220
  * Ghenuche et al. 2008 Ghenuche, P.; Cherukulappurath, S.; Taminiau, T. H.; van Hulst, N. F.; Quidant, R. Spectroscopic mode mapping of resonant plasmon nanoantennas. _Physical review letters_ 2008, _101_ , 116805
  * Gao et al. 2014 Gao, N.; Chen, Y.; Li, L.; Guan, Z.; Zhao, T.; Zhou, N.; Yuan, P.; Yao, S. Q.; Xu, Q.-H. Shape-dependent two-photon photoluminescence of single gold nanoparticles. _The Journal of Physical Chemistry C_ 2014, _118_ , 13904–13911
  * Wang et al. 2016 Wang, J.; Butet, J.; Baudrion, A.-L.; Horrer, A.; Lévêque, G.; Martin, O. J.; Meixner, A. J.; Fleischer, M.; Adam, P.-M.; Horneber, A., et al. Direct comparison of second harmonic generation and two-photon photoluminescence from single connected gold nanodimers. _The Journal of Physical Chemistry C_ 2016, _120_ , 17699–17710
  * Chen et al. 2018 Chen, J.; Krasavin, A.; Ginzburg, P.; Zayats, A. V.; Pullerits, T.; Karki, K. J. Evidence of high-order nonlinearities in supercontinuum white-light generation from a gold nanofilm. _ACS Photonics_ 2018, _5_ , 1927–1932
  * Großmann et al. 2019 Großmann, S.; Friedrich, D.; Karolak, M.; Kullock, R.; Krauss, E.; Emmerling, M.; Sangiovanni, G.; Hecht, B. Nonclassical optical properties of mesoscopic gold. _Physical Review Letters_ 2019, _122_ , 246802
  * Dell’Ova et al. 2023 Dell’Ova, F.; Malchow, K.; Chassagnon, R.; Heintz, O.; Pocholle, N.; Colas des Francs, G.; Dujardin, E.; Bouhelier, A. Hot Carrier-Induced Nonlinear Photoluminescence in a Thin Indium Tin Oxide Layer Patterned by Ga Ion Beam Milling. _ACS Applied Optical Materials_ 2023, _1_ , 244–251
  * Bouhelier et al. 2005 Bouhelier, A.; Bachelot, R.; Lerondel, G.; Kostcheev, S.; Royer, P.; Wiederrecht, G. P. Surface Plasmon Characteristics of Tunable Photoluminescence in Single Gold Nanorods. _Phys. Rev. Lett._ 2005, _95_ , 267405
  * Muhlschlegel et al. 2005 Muhlschlegel, P.; Eisler, H.-J.; Martin, O. J.; Hecht, B.; Pohl, D. Resonant optical antennas. _science_ 2005, _308_ , 1607–1609
  * Boyd et al. 1986 Boyd, G. T.; Yu, Z. H.; Shen, Y. R. Photoinduced luminescence from the noble metals and its enhancement on roughened surfaces. _Phys. Rev. B_ 1986, _33_ , 7923–7936
  * Agreda et al. 2019 Agreda, A.; Sharma, D. K.; Viarbitskaya, S.; Hernandez, R.; Cluzel, B.; Demichel, O.; Weeber, J.-C.; Colas des Francs, G.; Kumar, G. P.; Bouhelier, A. Spatial distribution of the nonlinear photoluminescence in au nanowires. _ACS photonics_ 2019, _6_ , 1240–1247
  * Viarbitskaya et al. 2015 Viarbitskaya, S.; Demichel, O.; Cluzel, B.; Colas des Francs, G.; Bouhelier, A. Delocalization of Nonlinear Optical Responses in Plasmonic Nanoantennas. _Phys. Rev. Lett._ 2015, _115_ , 197401
  * Wang et al. 2005 Wang, H.; Huff, T. B.; Zweifel, D. A.; He, W.; Low, P. S.; Wei, A.; Cheng, J.-X. In vitro and in vivo two-photon luminescence imaging of single gold nanorods. _Proceedings of the National Academy of Sciences_ 2005, _102_ , 15752–15756
  * Durr et al. 2007 Durr, N. J.; Larson, T.; Smith, D. K.; Korgel, B. A.; Sokolov, K.; Ben-Yakar, A. Two-photon luminescence imaging of cancer cells using molecularly targeted gold nanorods. _Nano letters_ 2007, _7_ , 941–945
  * Kumar et al. 2021 Kumar, U.; Cuche, A.; Girard, C.; Viarbitskaya, S.; Dell’Ova, F.; Al Rafrafin, R.; Colas des Francs, G.; Bolisetty, S.; Mezzenga, R.; Bouhelier, A., et al. Interconnect-Free Multibit Arithmetic and Logic Unit in a Single Reconfigurable 3 $\mu$m2 Plasmonic Cavity. _ACS nano_ 2021, _15_ , 13351–13359
  * Zhang et al. 2017 Zhang, Y.; He, S.; Guo, W.; Hu, Y.; Huang, J.; Mulcahy, J. R.; Wei, W. D. Surface-plasmon-driven hot electron photochemistry. _Chemical reviews_ 2017, _118_ , 2927–2954
  * Yang et al. 2013 Yang, J. J.; Strukov, D. B.; Stewart, D. R. Memristive devices for computing. _Nature Nanotechnology_ 2013, _8_ , 13–24
  * Lanza et al. 2022 Lanza, M.; Sebastian, A.; Lu, W. D.; Le Gallo, M.; Chang, M.-F.; Akinwande, D.; Puglisi, F. M.; Alshareef, H. N.; Liu, M.; Roldan, J. B. Memristive technologies for data storage, computation, encryption, and radio-frequency communication. _Science_ 2022, _376_ , eabj9979
  * Thyagarajan et al. 2017 Thyagarajan, K.; Sokhoyan, R.; Zornberg, L.; Atwater, H. A. Millivolt modulation of plasmonic metasurface optical response via ionic conductance. _Advanced Materials_ 2017, _29_ , 1701044
  * Song et al. 2017 Song, M.; Dellinger, J.; Demichel, O.; Buret, M.; Des Francs, G. C.; Zhang, D.; Dujardin, E.; Bouhelier, A. Selective excitation of surface plasmon modes propagating in Ag nanowires. _Optics express_ 2017, _25_ , 9138–9149
  * Weeber et al. 1999 Weeber, J.-C.; Dereux, A.; Girard, C.; Krenn, J. R.; Goudonnet, J.-P. Plasmon polaritons of metallic nanowires for controlling submicron propagation of light. _Physical Review B_ 1999, _60_ , 9061
  * Agreda et al. 2020 Agreda, A.; Viarbitskaya, S.; Smetanin, I. V.; Uskov, A. V.; Colas des Francs, G.; Bouhelier, A. Electrostatic Control over Optically Pumped Hot Electrons in Optical Gap Antennas. _ACS photonics_ 2020, _7_ , 2153–2162
  * Zhu et al. 2020 Zhu, Y.; Cui, L.; Natelson, D. Hot-carrier enhanced light emission: The origin of above-threshold photons from electrically driven plasmonic tunnel junctions. _J. Appl. Phys._ 2020, _128_ , 233105
  * Agreda et al. 2021 Agreda, A.; Sharma, D. K.; Des Francs, G. C.; Kumar, G. P.; Bouhelier, A. Modal and wavelength conversions in plasmonic nanowires. _Optics Express_ 2021, _29_ , 15366–15381
  * Mehonic et al. 2012 Mehonic, A.; Cueff, S.; Wojdak, M.; Hudziak, S.; Jambois, O.; Labbé, C.; Garrido, B.; Rizk, R.; Kenyon, A. J. Resistive switching in silicon suboxide films. _Journal of Applied Physics_ 2012, _111_
  * Menzel and Waser 2019 Menzel, S.; Waser, R. _Advances in Non-Volatile Memory and Storage Technology_ ; Elsevier, 2019; pp 137–170
  * Yang et al. 2012 Yang, Y.; Gao, P.; Gaba, S.; Chang, T.; Pan, X.; Lu, W. Observation of conducting filament growth in nanoscale resistive memories. _Nature communications_ 2012, _3_ , 732
  * Lombardo et al. 2005 Lombardo, S.; Stathis, J. H.; Linder, B. P.; Pey, K. L.; Palumbo, F.; Tung, C. H. Dielectric breakdown mechanisms in gate oxides. _Journal of applied physics_ 2005, _98_
  * Yang et al. 2008 Yang, J. J.; Pickett, M. D.; Li, X.; Ohlberg, D. A. A.; Stewart, D. R.; Williams, R. S. Memristive switching mechanism for metal/oxide/metal nanodevices. _Nature Nanotechnology_ 2008, _3_ , 429–433
  * Yang et al. 2012 Yang, Y.; Gao, P.; Gaba, S.; Chang, T.; Pan, X.; Lu, W. Observation of conducting filament growth in nanoscale resistive memories. _Nature communications_ 2012, _3_ , 732
  * Wolf et al. 2015 Wolf, C.; Nau, S.; Sax, S.; Busby, Y.; Pireaux, J.-J.; List-Kratochvil, E. J. W. Resistive switching based on filaments in metal/PMMA/metal thin film devices. _Jpn. J. Appl. Phys._ 2015, _54_ , 120301
  * Mangalam et al. 2016 Mangalam, J.; Agarwal, S.; Resmi, A.; Sundararajan, M.; Jinesh, K. Resistive switching in polymethyl methacrylate thin films. _Organic Electronics_ 2016, _29_ , 33–38
  * Sun et al. 2019 Sun, W.; Gao, B.; Chi, M.; Xia, Q.; Yang, J. J.; Qian, H.; Wu, H. Understanding memristive switching via in situ characterization and device modeling. _Nature communications_ 2019, _10_ , 3453
  * Sakai et al. 2019 Sakai, K.; Sato, T.; Tani, S.; Ito, M.; Yagi, M.; Shirakashi, J.-I. Synaptic behaviors of electromigrated Au nanogaps. _AIP Advances_ 2019, _9_
  * Sakai et al. 2020 Sakai, K.; Sato, T.; Kiyokawa, R.; Koyama, R.; Yagi, M.; Ito, M.; Shirakashi, J.-I. Gold nanogap-based artificial synapses. _Japanese Journal of Applied Physics_ 2020, _59_ , 050601
  * Yang et al. 2014 Yang, Y.; Gao, P.; Li, L.; Pan, X.; Tappertzhofen, S.; Choi, S.; Waser, R.; Valov, I.; Lu, W. D. Electrochemical dynamics of nanoscale metallic inclusions in dielectrics. _Nature communications_ 2014, _5_ , 4232
  * Wang et al. 2017 Wang, Z.; Joshi, S.; Savel’ev, S. E.; Jiang, H.; Midya, R.; Lin, P.; Hu, M.; Ge, N.; Strachan, J. P.; Li, Z., et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. _Nature materials_ 2017, _16_ , 101–108
  * Cheng et al. 2022 Cheng, B.; Zellweger, T.; Malchow, K.; Zhang, X.; Lewerenz, M.; Passerini, E.; Aeschlimann, J.; Koch, U.; Luisier, M.; Emboras, A., et al. Atomic scale memristive photon source. _Light: Science & Applications_ 2022, _11_ , 78
  * Mohammad et al. 2016 Mohammad, B.; Jaoude, M. A.; Kumar, V.; Homouz, D. M. A.; Nahla, H. A.; Al-Qutayri, M.; Christoforou, N. State of the art of metal oxide memristor devices. _Nanotechnology Reviews_ 2016, _5_ , 311–329
  * Zurak et al. 2023 Zurak, L.; Wolff, C.; Meier, J.; Kullock, R.; Mortensen, N. A.; Hecht, B.; Feichtner, T. Direct electrical modulation of surface response in a single plasmonic nanoresonator. 2023
  * Hamdad et al. 2022 Hamdad, S.; Malchow, K.; Dujardin, E.; Bouhelier, A.; Cheng, B.; Zellweger, T.; Leuthold, J. Overbias Light Emission From Memristive Nanojunctions. 2022
  * Lyu et al. 2023 Lyu, P.-T.; Liu, X.-R.; Yin, L.-X.; Wu, P.; Sun, C.; Chen, H.-Y.; Xu, J.-J.; Kang, B. Periodic Distributions and Ultrafast Dynamics of Hot Electrons in Plasmonic Resonators. _Nano Letters_ 2023, _23_ , 2269–2276