# TEGLO: High Fidelity Canonical Texture Mapping from Single-View Images

Vishal Vinod1,2, Tanmay Shah2,∗ Dmitry Lagun2
1University of California These authors contributed equally to this work.  San
Diego 2Google Research
<EMAIL_ADDRESS>{shaht<EMAIL_ADDRESS>
https://teglo-nerf.github.io/

###### Abstract

Recent work in Neural Fields (NFs) learn 3D representations from class-
specific single view image collections. However, they are unable to
reconstruct the input data preserving high-frequency details. Further, these
methods do not disentangle appearance from geometry and hence are not suitable
for tasks such as texture transfer and editing. In this work, we propose TEGLO
(Textured EG3D-GLO) for learning 3D representations from single view in-the-
wild image collections for a given class of objects. We accomplish this by
training a conditional Neural Radiance Field (NeRF) without any explicit 3D
supervision. We equip our method with editing capabilities by creating a dense
correspondence mapping to a 2D canonical space. We demonstrate that such
mapping enables texture transfer and texture editing without requiring meshes
with shared topology. Our key insight is that by mapping the input image
pixels onto the texture space we can achieve near perfect reconstruction
($\geq 74$ dB PSNR at $1024^{2}$ resolution). Our formulation allows for high
quality 3D consistent novel view synthesis with high-frequency details at
megapixel image resolution.

##  1 Introduction

Figure 1: Teaser \- Demonstrating TEGLO for high fidelity 3D reconstruction
and multi-view consistent texture representation and texture editing from
single-view image collections of objects.

Reconstructing high-resolution and high-fidelity 3D consistent representations
from single-view in-the-wild image collections is critical for applications in
virtual reality, 3D content creation and telepresence systems. Recent work in
Neural Radiance Fields (NeRFs) [7, 17, 6, 39] aim to address this by
leveraging the inductive bias across a dataset of single-view images of class-
specific objects for 3D consistent rendering. However, they are unable to
preserve high frequency details while reconstructing the input data despite
the use of SIREN [44] or positional encoding [33], in part due to the
properties of MLPs that they use [10]. For arbitrary resolution 3D
reconstruction from single-view images, these methods face several challenges
such as image-space approximations that break multi-view consistency
constraining the rendering resolution [6], requiring Pivotal Tuning Inversion
(PTI) [41] or fine-tuning for reconstruction [17, 6, 45] and the inability to
preserve high-frequency details [17, 6, 45, 39]. To address these limitations,
we propose TEGLO (Textured EG3D-GLO) that uses a tri-plane representation [6]
and Generative Latent Optimization (GLO) [4] based training to enable
efficient and high-fidelity 3D reconstruction, and novel view synthesis at
arbitrary image resolutions from single-view image collections of objects.

Figure 2: Overview \- TEGLO enables 3D reconstruction and texture
representation from single-view image collections of objects.

Recent works disentangle texture from geometry [10, 55] and enable challenging
tasks such as texture editing and texture transfer. However, they depend on
large-scale textured mesh data for high-fidelity 3D reconstruction which is
laborious, expensive and time intensive to capture. Further, the use of a
capture environment may cause a dataset-shift leading to generalization issues
in downstream tasks, and the data use may require custom licensing. All of
these factors limit access from the broader research community. This motivates
the need for a method to learn textured 3D representations from single-view
in-the-wild images of objects. However, the task of disentangling texture and
3D geometry from in-the-wild image collections is a formidable challenge due
to the presence of wide variations in poses, partial views, complex details in
appearance, geometry, noise _etc_. in the given image collection. Inspired by
surface fields [16], TEGLO leverages the 3D surface points of objects
extracted from a NeRF to learn dense correspondences via a canonical
coordinate space to enable texture transfer, texture editing and high-fidelity
single-view 3D reconstruction.

Our key insight is that by disentangling texture and geometry using the 3D
surface points of objects to learn a dense correspondence mapping via a 2D
canonical coordinate space, we can extract a texture for each object. Then, by
using the learned correspondences to map the pixels from the input image of
the object onto the texture, we enable preserving high-frequency details. As
expected, copying the input image pixels onto the texture accurately allows
near perfect reconstruction while preserving high-fidelity multi-view
consistent representation with high-frequency details. In this work, we
present TEGLO, consisting of a tri-plane and GLO-based conditional NeRF and a
method to learn dense correspondences to enable challenging tasks such as
texture transfer, texture editing and high-fidelity 3D reconstruction even at
large megapixel resolutions. We also show that TEGLO enables single-view 3D
reconstruction with no constraints on resolution by inverting the image into
the latent table without requiring PTI [41] or model fine-tuning. We present
an overview of our final model in Fig.(2): TEGLO takes a single-view image and
its approximate camera pose to map the pixels onto a texture. Then, to render
the object from a different view, we extract the 3D surface points from the
trained NeRF and use the dense correspondences to obtain the color for each
pixel from the mapped canonical texture. Optionally, TEGLO can take texture
edits and transfer textures across objects. In summary, our contributions are:

  1. 1.

A framework for effectively mapping the pixels from an in-the-wild single-view
image onto a texture to enable high-fidelity 3D consistent representations
preserving high-frequency details.

  2. 2.

A method for extracting canonical textures from single-view images enabling
tasks such as texture editing and texture transfer for NeRFs.

  3. 3.

Demonstrating that we can effectively map the single-view image pixels to
canonical texture space while preserving 3D consistency and so achieving near
perfect reconstruction ($\geq 74$ dB PSNR at $1024^{2}$ resolution).

##  2 Related Work

Figure 3: Architecture \- TEGLO Stage-1 (left) uses a tri-plane and GLO based
conditional NeRF to learn a per-object table of latents to reconstruct the
single-view image collection. TEGLO Stage-2 (right) learns dense
correspondences via a 2D canonical coordinate space.

3D-aware generative models. Learning 3D representations from multi-view images
with camera poses have been extensively studied since the explosion of Neural
Radiance Fields (NeRFs) [33, 46, 58, 2, 59, 17]. However, these methods
require several views and learn a radiance field for a single scene. RegNeRF
[36] reduces the need from several views to only a handful, however, the
results have several artifacts. Recently, several works learn 3D
representations from single-view images [7, 6, 28, 45, 39, 60]. Further, [48,
47, 49, 24] enable multi-view consistent editing, however, they are limited by
the rendering resolution. Recent work propose single image 3D consistent novel
view synthesis [56, 29, 18, 51], however they are not yet suitable for texture
representation. While point cloud based diffusion models [57, 35] enable
learning 3D representations, they have limited applicability in textured 3D
generation and high fidelity novel view synthesis. In this work, we show that
TEGLO learns textured 3D representations from class-specific single-view image
collections.

Texture representation. Template based methods [38, 3, 11, 20] deform a
template mesh prior for 3D representations and are hence restricted in the
topology they can represent. Texture Fields [37] enable predicting textured 3D
models given an image and a 3D shape, but are unable to represent high-
frequency details. While NeuTex [52] enables texture representation, it does
not allow multi-view consistent texture editing at the desired locations due
to a contorted UV mapping [55]. NeuMesh [55] learns mesh representations to
enable texture transfer and texture editing using textured meshes. However, it
performs mesh-guided texture transfer and requires spatial-aware fine-tuning
for mesh-guided texture edits. While GET3D [15] learns textured 3D shapes by
leveraging tri-plane based geometry and texture generators, it requires 2D
silhouette supervision and is limited to synthetic data. AUVNet [10]
represents textures from textured meshes by learning an aligned UV mapping and
demonstrates texture transfer. However, it depends on textured mesh data and
requires multiple networks to enable single-view 3D reconstruction. In
contrast, TEGLO learns textured 3D consistent representations from single-view
images by inverting the image into the latent table.

Dense correspondences. Previous work in dense correspondence learning involve
supervised [13, 27] or unsupervised [54, 53] learning methods. CoordGAN [34]
learns dense correspondences by extracting each image as warped coordinate
frames transformed from correspondence maps which is effective for 2D images.
However, CoordGAN is unable to learn 3D correspondences. AUVNet [10]
establishes dense correspondences across 3D meshes via a canonical UV mapping
and asserts that methods that do not utilize color for dense correspondence
learning [14, 30] may have sub-par performance in texture representation.

##  3 Proposed Method

Given a collection of single-view in-the-wild images of objects and their
approximate camera poses, TEGLO aims to learn a textured 3D representation of
the data. TEGLO consists of two stages: 3D representation learning and dense
correspondence learning. TEGLO Stage-1 consists of a conditional NeRF
leveraging a Tri-Plane representation and an auto-decoder training regime
based on generative latent optimization (GLO) [4] for 3D reconstruction of the
image collection. TEGLO Stage-2 uses a dataset rendered using TEGLO Stage-1
consisting of the geometry from five views of an object and the optimized
latent code. TEGLO Stage-2 uses the 3D surface points from the rendered
dataset to learn dense pixel-level correspondences via a 2D canonical
coordinate space. Then, the inference stage uses the learned dense
correspondences to map the image pixels from the single-view input image onto
a texture extracted from TEGLO-Stage 2. As a result, TEGLO effectively
preserves high frequency details at an unprecedented level of accuracy even at
large megapixel resolutions. TEGLO disentangles texture and geometry enabling
texture transfer (Fig.(12)), texture editing (Fig.(11)) and single view 3D
reconstruction without requiring fine-tuning or PTI (Fig.(9)).

###  3.1 TEGLO Stage 1: 3D representation

Formulation. We denote the single-view image collection ($\mathcal{I}$) with
class specific objects as $\\{o_{0},o_{1},...,o_{n}\\}\in\mathcal{I}$. For
learning 3D representations, TEGLO employs a generative latent optimization
(GLO) based auto-decoder training, where NeRF is conditioned on an image
specific latent code $\\{w_{0},w_{1},...,w_{n}\\}\in\mathcal{R}^{D}$ to
effectively reconstruct the image without requiring a discriminator.

Network architecture. The NeRF model $\mathcal{N}$ is represented by TEGLO
Stage-1 in Fig.(3). The model $\mathcal{N}$ passes the input conditioning
latent $w_{i}$ to a set of CNN-based synthesis layers [23] whose output
feature maps are used to construct a $\mathrm{k}$-channel tri-plane. The
sampled points on each ray are used to extract the tri-plane features and
aggregate the $\mathrm{k}$-channel features. Then the tri-plane decoder MLP
outputs the scalar density $\sigma$ and color which are alpha-composited by
volume rendering to obtain the RGB image. Volume rendering along camera ray
$r(t)=O+td$ is:

$\mathcal{C}_{\text{NeRF}}(r,w)=\int_{b_{n}}^{b_{f}}T(t,w)\sigma(r(t),w)\textbf{c}(r(t),\textbf{d},w)dt\;$
(1) $\text{ where }\;T(t,w)=\text{ exp
}\left(-\int_{b_{n}}^{b_{f}}\sigma(r(s),w)\right)ds$

Here, the radiance values can be replaced with the depth $d(x)$ or pixel
opacity to obtain the surface depth. During inference, the surface depth map
and 2D pixel coordinates are used to obtain the 3D surface points via back-
projection. The surface normals can be computed as the first derivative of the
density $\sigma$ with respect to the input as follows:

$\widehat{n}(r,w)=-\int_{b_{n}}^{b_{f}}T(t,w)\;\sigma(r(t),w)\;\nabla_{r(t)}(\sigma(r(t),w))dt$
$n(r,w)=\frac{\widehat{n}(r,w)}{||\;\widehat{n}(r,w)\;||_{2}}$ (2)

Thus from an inference step, an RGB image, surface depth map, 3D surface
points and the surface normals of the object instance can be obtained. In
Fig.(4), we show the sample reconstruction results for $\mathcal{N}$ on the
CelebA-HQ, AFHQv2 and ShapeNet-Cars datasets. In Fig.(5) we show qualitative
results for novel view synthesis with $\mathcal{N}$ trained on SRN-Cars and
evaluated on a held-out set of views. Since SRN-Cars is a multi-view dataset,
we compare the rendered novel views with their corresponding ground-truth
views.

Figure 4: Rendering the dataset for TEGLO Stage-2 \- Rendering multiple views
of images, surface normals, depth maps and 3D surface points from CelebA-HQ,
AFHQv2-Cats and ShapeNet-Cars for learning dense correspondences in TEGLO
Stage-2.

Losses. $\mathcal{N}$ is trained by jointly reconstructing the image and
simultaneously optimizing a latent ($w_{i}$). As noted in [39], this enables
the training loss to be enforced on individual pixels enabling training and
inference at arbitrary image resolutions. As depicted in TEGLO Stage-1 in
Fig.(3), three losses are minimized to train $\mathcal{N}$:
$\mathcal{L}_{\text{RGB}}$ is the $\mathcal{L}_{1}$ reconstruction loss
between the pixels from the rendered image and the corresponding pixels from
the ground truth image for the object ($o_{i}$). The
$\mathcal{L}_{\text{Perceptual}}$ loss is the LPIPS (Learned Perceptual Image
Patch Similarity) loss between rendered image and the ground truth image view.
The $\mathcal{L}_{\text{Camera}}$ is the camera prediction $\mathcal{L}_{1}$
loss between the output of the light-weight camera encoder and the ground-
truth camera parameters for the camera pose in order to learn 3D consistent
representation of the object $(o_{i}\in\mathcal{I})$.

$\mathcal{L}_{\mathcal{N}}=\mathcal{L}_{\text{RGB}}+\mathcal{L}_{\text{Perceptual}}+\mathcal{L}_{\text{Camera}}$
(3)

To train $\mathcal{N}$, we use the single-view image dataset and the
approximate pose for each $o_{i}\in\mathcal{I}$ (Sec.(4)). We train the model
for 500K steps using the Adam optimizer [25] on 8 NVIDIA V100 (16 GB) taking
36 hours to complete.

Figure 5: Novel view synthesis \- Results for ShapeNet-Cars data.

Design choices. As noted in Sec.(1), EG3D [6] shows medium resolution
($512^{2}$) capacity while using image-space approximations in the super-
resolution module which negatively affects the geometric fidelity [45]. While
EpiGRAF [45] uses a patch-based discriminator for pure 3D generation, it is
still prone to issues in scaling and training with multi-resolution data.
Moreover, adversarial training using discriminators leads to training
instability. Different from EG3D and EpiGRAF that use an adversarial training
paradigm, $\mathcal{N}$ uses a GLO-based auto-decoder training paradigm which
jointly optimizes a latent representation and reconstructs the image enabling
arbitrary resolution synthesis - even at large megapixel resolutions - without
the constraints of a discriminator. Hence, $\mathcal{N}$ enables 3D
representations with geometric fidelity while also benefiting from an
efficient tri-plane based representation.

EG3D [6] requires camera pose conditioning for the generator and discriminator
to establish multi-view consistency. The limitation of a pose-conditioned
generator is that it does not completely disentangle the pose from appearance
which leads to artifacts such as degenerate solutions (2D billboards), or
expressions such as the eye or smile following the camera. Since $\mathcal{N}$
optimizes a latent representation of an object to reconstruct it, we observe
that the generator does not require camera pose conditioning and simply using
a light-weight camera predictor network and training with a camera prediction
loss ($\mathcal{L}_{\text{Camera}}$) is sufficient to learn 3D consistent
representations.

###  3.2 TEGLO Stage 2: Dense correspondences

Formulation. We render a multi-view dataset ($\mathcal{D}$) using
$\mathcal{N}$ trained on single-view image collections for the task of texture
representation. We denote each object $e_{i}\in\mathcal{D}$ comprising of five
views: $e_{i}=\\{v_{f},v_{l},v_{r},v_{t},v_{b}\\}$ where $v$ denotes the view,
and the sub-scripts ($j\;\text{for all}\;v_{j}$) denote frontal, left, right,
top and bottom poses respectively (refer Fig.(4)). In $\mathcal{D}$, each view
$v_{j}\in e_{i}$ includes the depth map ($\widehat{d}_{j}$), RGB image
($\widehat{r}_{j}$), surface normals ($\widehat{s}_{j}$), 3D surface points
($\widehat{p}_{j}$), and the optimized latent, $w_{i}$, which is identical for
views of $e_{i}$ as it is independent of camera pose (Fig.(4)). For TEGLO
Stage 2, we use $\\{\\{\widehat{r}_{j},\widehat{s}_{j},\widehat{p}_{j}\\}\in
v_{j},w_{i}\\}\in e_{i}\\}$.

Learning dense pixel-level correspondences across multiple views of an object
is the task of locating the same 3D coordinate point in a canonical coordinate
space. Inspired by surface fields [16], we aim to learn dense correspondences
using the 3D surface points extracted from $\mathcal{N}$ by back-projecting
the depth ($\widehat{d}_{j}$) and pixel coordinates. Inspired by CoordGAN [34]
and AUVNet [10], we propose a dense correspondence learning network in TEGLO
Stage-2 trained in an unsupervised manner learning an aligned canonical
coordinate space to locate the same 3D surface point across different views
($v_{j}$) of the same object ($e_{i}$).

Network architecture. TEGLO Stage-2 is represented in Fig.(3). The
architecture consists of a latent mapping network ($\mathcal{L}$), a dense
correspondence network ($\mathcal{M}$) and a basis network ($\mathcal{C}$) -
all of which are MLP networks. The 3D surface points ($\widehat{p}_{j}$) from
$v_{j}\in e_{i}$) are mapped to a 2D canonical coordinate space conditioned on
a shape code mapped from the optimized latent $w_{i}$ for $e_{i}$. We use a
Lipschitz regularization [31] for each MLP layer in the dense correspondence
network ($\mathcal{M}$). The latent mapping network ($\mathcal{L}$) is a set
of MLP layers that takes the $w_{i}$-latent for $e_{i}$ as input and predicts
a shape-code for conditioning the dense correspondence network $\mathcal{M}$,
and coefficients for the deformed basis. Previous work [50, 10] show that if
the input is allowed to be represented as a weighted sum of basis images,
_i.e_. to obtain a deformed basis before decomposition, then the 2D canonical
coordinate space will be aligned. The basis network ($\mathcal{C}$) is similar
to [10] and uses the predicted coefficients to decompose the deformed
coordinate points. Thus, $\mathcal{M}$ maps the 3D surface points to an
aligned 2D canonical coordinate space, enabling the learning of dense
correspondences using the $p_{j}\in\mathcal{S}$ extracted from $\mathcal{N}$.
Next, the basis network takes the 2D canonical coordinates as input to predict
the deformed basis $\mathcal{B}$. Then, $\mathcal{B}$ is weighted with the
predicted coefficients to decompose the basis into the 3D surface points
($p_{j}$), surface normals ($s_{j}$) and color ($r_{j}$).

Losses. TEGLO Stage-2 is trained using three $\mathcal{L}_{2}$ reconstruction
losses: the $\mathcal{L}_{\text{RGB}}$ loss between the rendered RGB image
$\widehat{r}_{j}$ and the predicted RGB image $r_{j}$; the
$\mathcal{L}_{\text{Normals}}$ loss between the rendered surface normals
$\widehat{s}_{j}$ and the predicted surface normals $s_{j}$;
$\mathcal{L}_{\text{Coord}}$ loss between the extracted 3D surface points
$\widehat{p}_{j}$ and the predicted 3D surface points $p_{j}$. Hence, the
total training loss for TEGLO Stage-2 is:

$\mathcal{L}_{\text{Stage2}}=\mathcal{L}_{\text{RGB}}+\mathcal{L}_{\text{Normals}}+\mathcal{L}_{\text{Coord}}\vspace{-1mm}$
(4)

To train TEGLO Stage-2, we use the rendered dataset $\mathcal{D}$ consisting
of 1000 objects with five views per object and the optimized latent for each
identity. The networks are trained using $\mathcal{L}_{\text{Stage2}}$ loss
for 1000 epochs using the Adam [25] optimizer to learn dense correspondences
across $e_{i}\in\mathcal{D}$.

Design choices. We use the optimized $w$-latent from $\mathcal{N}$ for
learning the shape code and coefficients for TEGLO Stage-2 because it
represents the 3D geometry and appearance information for object ($e_{i}$)
independent of camera pose. We observe that using a Lipschitz regularization
for every MLP layer in $\mathcal{M}$ suitably regularizes the network to
deform the input surface points $\widehat{s}_{j}$. Interestingly, our
experiments show that simply reconstructing the 3D surface points instead of
the color, surface points and surface normals also leads to learning
reasonable dense pixel-level correspondences. We show qualitative results for
TEGLO Stage-2 trained using only $\mathcal{L}_{\text{Coord}}$ loss in Fig.(8)
as TEGLO-3DP.

Figure 6: Inference \- TEGLO texture extraction for texture transfer and
editing. Red arrows indicate the use of a K-d tree to store the texture. Blue
arrows indicate the use of input image pixels.

Figure 7: Interpolating textures with sparse “holes” \- Depicting the KD-Tree
and Natural Neighbor Interpolation (NNI) to interpolate “holes” (if any) in
the texture for novel view synthesis. Figure 8: Qualitative results \-
Comparison with relevant 3D-aware generative baseline methods at $256^{2}$
resolution for CelebA-HQ.

###  3.3 Inference.

Extracting the texture. After training TEGLO Stage-2, we use the learned dense
correspondences to extract a texture map for every object
$o_{i}\in\mathcal{I}$. We use the pose of the target image $o_{i}$ to extract
the 3D surface points from $\mathcal{N}$ and use it to map the image pixels to
the 2D canonical coordinate space. We denote this as texture $t_{GT}$.
Similarly, we use $\mathcal{M}$ to map the respective RGB values from
$\\{v_{f},v_{l},v_{r},v_{t},v_{b}\\}\in e_{i}$ using the corresponding 3D
surface points ($s_{j}$) from all five views to the 2D canonical coordinate
space. We denote this as texture $t_{\text{views}}$. Thus, textures $t_{GT}$
and $t_{\text{views}}$ store a mapping _i.e_. the canonical coordinate point
and the corresponding RGB values. The procedure is represented in Fig.(6) and
textures are depicted in Fig.(10) and Fig.(11). In Fig.(6) $t_{O}$ represents
the texture obtained by combining $t_{GT}$ and $t_{\text{views}}$. We store
this mapping in a K-d tree which enables us to index into the textures using
accurate floating point indices to obtain the RGB values. The K-d tree allows
querying with canonical coordinate points to extract multiple neighbors and
enables TEGLO to be robust to sparse “holes” in the texture as depicted in
Fig.(7).

Novel view synthesis. For rendering novel views of $o_{i}$, we extract the 3D
surface point for the pose from $\mathcal{N}$ and obtain the canonical
coordinates from $\mathcal{M}$. For each 2D canonical coordinate point
$c_{k}$, we query the K-d tree for three natural neighbors and obtain indices
for the neighbors which are used to obtain the respective RGB values. Natural
Neighbor Interpolation (NNI) [43] enables fast and robust reconstruction of a
surface based on a Dirichlet tesselation - unique for every set of query
points - to provide an unambiguous interpolation result. We simplify the
natural neighbor interpolation (NNI) based only on the distances of the points
$c_{k}$ in the 2D canonical coordinate space to obtain the RGB values from the
stored texture. The robust and unambiguous interpolation enables TEGLO to
effectively map the ground-truth image pixels from the input dataset
$\mathcal{I}$ onto the geometry for novel view synthesis. To extract the
Surface Field $\mathcal{S}$, we render $e_{i}$ from five camera poses causing
potential camera pose biases that may lead to sparse “holes” in the texture.
Our formulation uses the K-d tree and NNI to interpolate and index into
textures with sparse “holes”. In Fig.(7), each cell in the 5x6 grid represents
a discrete pixel in the texture space and the red dot represents a canonical
coordinate point. There are three issues that may arise:

  1. 1.

The canonical coordinate points may not be aligned to the pixel centers and
storing them in the discretized texture space may lead to imprecision.

  2. 2.

There may be multiple canonical coordinates mapped to a discrete integral
pixel wherein some coordinates may need to be dropped for an unambiguous
texture indexing - leading to loss of information.

  3. 3.

Some pixels may not be mapped to by any canonical coordinates, creating a
“hole” in discretized space. This is represented by “X” in the grid in
Fig.(7).

K-d tree allows extracting multiple neighbors by querying with canonical
coordinate points and also enables indexing the texture using floating point
values. Hence, using a K-d tree to store the texture helps address (1) and
(2). Further, using a K-d tree in conjunction with Natural Neighbor
Interpolation (NNI) effectively addresses (3). We include more details in the
supplementary material.

Texture editing. Texture editing is represented by $t_{\text{Edit}}$ in
Fig.(6). We create the edits on a blank image the same size as that of
$t_{\text{O}}$ and denote it as $r_{\text{edit}}$. The edit image
$r_{\text{edit}}$ is taken to be in the canonical coordinate space and hence
directly indexed into the K-d tree to be overlay on $t_{O}$. Note that the we
do not apply any constraint on the texture space represented and hence the
texture may be visually aligned to a non-frontal canonical pose as is the case
in Fig.(10) and Fig.(11). The final texture with the edit $t_{\text{Edit}}$ is
created by combining $t_{O}$ and $r_{\text{edit}}$. Qualitative results for
texture edits are depicted in Fig.(1) and Fig.(11).

##  4 Experiments and Results

Datasets. We train TEGLO with single-view image datasets such as FFHQ [23],
CelebA-HQ [32, 21] and AFHQv2-Cats [22, 12]. To obtain the approximate camera
pose, we follow [39] by first using an off-the-shelf face landmark predictor
MediaPipe Face Mesh [1] to extract landmarks appearing at consistent
locations. Then, we use a shape-matching least-squares optimization to align
the landmarks with 3D canonical landmarks to obtain the approximate pose. We
also use a multi-view image dataset - ShapeNet-Cars [9, 8] with results in
Fig.(1) and Table.(3).

Figure 9: Single view 3D reconstruction \- Results for TEGLO trained on FFHQ
data and evaluated on CelebA-HQ image targets. Figure 10: Results for complex
texture and geometry \- Qualitative results for texture representation and
novel view synthesis with complex image samples. Compare row-1, row-2 with
Fig.(24) in [5] and row-3 with Fig.(25) in [26])

3D reconstruction. We evaluate TEGLO on the task of reconstructing the input
image in the same pose and show a comparison with baseline methods. We report
quantitative results for train data reconstruction in Table.(1) measuring the
PSNR (Peak Signal to Noise Ratio) and LPIPS (Learned Perceptual Image Patch
Similarity) metrics for CelebA-HQ and FFHQ. We observe similar results for
LoLNeRF and TEGLO Stage-1 at $128^{2}$ resolution. However, as expected, TEGLO
attains $89.5$ dB PSNR and $7.4e$-7 for LPIPS. We report quantitative results
for test data reconstruction at $256^{2}$ resolution from a held-out set from
the dataset for CelebA-HQ and FFHQ in Table.(2), and AFHQv2-Cats and SRN-Cars
in Table.(3). We observe that TEGLO attains near-perfect reconstruction of
test data attaining $\geq 67.5$ dB PSNR and $\leq 6.9e$-5 for LPIPS across the
datasets.

We depict qualitative results for CelebA-HQ in Fig.(8) where the red arrows
indicate missing high frequency details for 3D reconstruction. For EG3D-Fit,
we invert the image into the EG3D [6] latent space and perform Pivotal Tuning
Inversion (PTI) [41] for the single-view images. We observe missing details in
LoLNeRF [39], EG3D-Fit [6] and TEGLO stage 1 specifically related to details
such as jewelry, skin wrinkles, eyeglass translucency, eyeglass frames, hair
strands etc. As expected, the results for TEGLO and TEGLO-3DP (where TEGLO
Stage-2 is trained with only surface point supervision) include the high
frequency details missed by baseline methods demonstrating near perfect
reconstruction. In Fig.(10), we show qualitative results including the texture
image ($t_{O}$) for complex appearance and geometry such as multi-view
consistent eyeglasses, 3D make-up and hair. Compared with Fig.(24) in [5], we
show improved multi-view consistent results for eyeglasses in row-1, and 3D
make-up in row-2. Compared with Fig.(25) in [26], we show multi-view
consistent representations for beard that the baseline method [26] was unable
to model.

Table 1: Reconstruction of train images \- Quantitative comparison on training
data reconstruction at $128^{2}$ resolution.

Method | PSNR ($\uparrow$) | LPIPS ($\downarrow$)
---|---|---
$\pi$-GAN [7] (CelebA) | 23.5 | 0.226
LoLNeRF [39] (FFHQ) | 29.0 | 0.199
LoLNeRF [39] (CelebA-HQ) | 29.1 | 0.197
ABC [40] (CelebA-HQ) | 26.3 | -
TEGLO Stage 1 (FFHQ) | 29.0 | 0.294
TEGLO Stage 1 (CelebA-HQ) | 28.9 | 0.317
TEGLO (CelebA-HQ) | 89.5 | 2.3e-7

Table 2: Reconstruction of test images \- Quantitative comparison on test data
reconstruction.

Method | Res. | PSNR ($\uparrow$) | LPIPS ($\downarrow$)
---|---|---|---
$\pi$-GAN [7] (CelebA) | $256^{2}$ | 21.8 | 0.412
LoLNeRF [39] (FFHQ) | $512^{2}$ | 25.3 | 0.491
LoLNeRF [39] (CelebA-HQ) | $256^{2}$ | 26.2 | 0.363
TEGLO Stage 1 (FFHQ) | $256^{2}$ | 27.3 | 0.334
TEGLO Stage 1 (CelebA-HQ) | $256^{2}$ | 27.5 | 0.260
TEGLO (FFHQ) | $256^{2}$ | 84.9 | 2.1e-6
TEGLO (CelebA-HQ) | $256^{2}$ | 86.2 | 7.4e-7
TEGLO (CelebA-HQ) | $512^{2}$ | 82.6 | 4.4e-6
TEGLO (CelebA-HQ) | $1024^{2}$ | 74.7 | 6.9e-5

Table 3: Comparing with GLO baselines \- Quantitative results for test set
reconstruction in PSNR at $256^{2}$ resolution with Generative Latent
Optimization baselines. (We report the LoLNeRF result on SRN-Cars from the
”Concatenation” baseline in [40]).

Method | AFHQv2-Cats [22] | SRN-Cars [8]
---|---|---
LoLNeRF [39] | 24.94 | 25.80
ABC [40] | - | 29.10
TEGLO Stage-1 | 29.26 | 30.48
TEGLO | 87.38 | 67.52

Single-view 3D reconstruction. It is the task of representing an in-the-wild
or out-of-distribution image using a trained network. The qualitative results
for TEGLO trained on FFHQ [23] data for single-view 3D reconstruction on
samples from CelebA-HQ are in Fig.(9). Previous work such as AUVNet [10]
require additional training of a ResNet-18 [19] for the image encoder and IM-
Net [11] for the shape decoder followed by ray marching to obtain the mesh to
represent the image while methods such as EG3D [6] require PTI (Pivotal Tuning
Inversion [41]) fine-tuning to represent the image. For single-view textured
3D representation in TEGLO, we simply invert the image into the latent and do
not require any fine-tuning.

Reconstructing single-view images at arbitrary resolutions while preserving 3D
consistency is very desirable for many applications. However, EG3D [6] has a
limitation in performing this task because its generator is conditioned on the
camera intrinsic and extrinsic parameters, leading to a “baked-in” training
image resolution. As TEGLO does not condition on the camera, it enables
single-view 3D reconstruction and novel view synthesis at arbitrary
resolutions without requiring re-training for different resolutions.

Figure 11: Texture editing \- Qualitative results for texture edits.

Texture editing. In Sec.(3.3), we describe the procedure to edit textures
using TEGLO. The qualitative results with texture editing for CelebA-HQ [32,
21] are depicted in Fig.(11) and for AFHQv2-Cats and ShapeNet-Cars in Fig.(1).
Our edits are class-specific and target image agnostic because edits are
performed in the canonical space. Previous work, NeuMesh [55] requires
spatial-aware fine-tuning and mesh guided texture editing for precise
transfer. However, TEGLO simply maps a texture edit image of the same size as
the texture into the K-d tree with an overlay of the pixels in the earlier
texture (_i.e_. obtaining $t_{\text{Edit}}$) - precisely transferring the edit
without requiring any optimization strategies. Further results are in the
supplementary.

Texture transfer. As discussed in Sec.(3.3), the extracted textures are
aligned in a canonical coordinate space and allows transferring textures
across different geometries. We demonstrate texture transfer across different
geometries in Fig.(12). Here, row-1 represents the target image from CelebA-HQ
for the geometry learned by TEGLO Stage-1 and column-1 represents the textures
(stored in a K-d tree) extracted after TEGLO Stage-2. We observe realistic
texture transfer despite arbitrary camera biases in rendering $\mathcal{D}$
mitigated by using the K-d tree and NNI. (Fig.(7)).

##  5 Discussion

While TEGLO enables near perfect 3D reconstruction of objects from single-view
image collections, it requires non-trivial computational resources to train
TEGLO Stage-1, render a dataset with five views of the object, train TEGLO
Stage-2 and then use the input image surface points to extract the texture. We
hope that future work can simplify the framework with an elegant end-to-end
formulation.

A trivial next step would be to use StyleGANv2 [23] to generate high quality
textures for texture transfer and editing. TEGLO could enable 3D full-body
avatars from single views with an unprecedented amount of detail preservation
extending methods such as PIFu [42]. Future work could explore representing
light stage data via NeRFs with high frequency details across different camera
capture angles in an illumination invariant manner using 3D surface points.

Figure 12: Texture transfer \- Qualitative results for texture transfer with
CelebA-HQ. (Top row shows CelebA-HQ image targets).

##  6 Conclusion

In this work, we present TEGLO for high-fidelity canonical texture mapping
from single-view images enabling textured 3D representations from class-
specific single-view image collections. TEGLO consists of a conditional a NeRF
and a dense correspondence learning network that enable texture editing and
texture transfer. We show that by effectively mapping the input single-view
image pixels onto the texture, we can achieve near perfect reconstruction
($\geq 74$ dB PSNR at $1024^{2}$ resolution). TEGLO allows single-view 3D
reconstruction by inverting the single-view image into the latent table
without requiring any PTI or fine-tuning.

## References

  * [1] Mediapipe face mesh.  https://google.github.io/mediapipe/solutions/face_mesh.html.  Online; Accessed: 2022-06-20.
  * [2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [3] Anand Bhattad, Aysegul Dundar, Guilin Liu, Andrew Tao, and Bryan Catanzaro.  View generalization for single image textured 3d models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6081–6090, 2021.
  * [4] Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam.  Optimizing the latent space of generative networks.  arXiv preprint arXiv:1707.05776, 2017.
  * [5] Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shun-Suke Saito, Stephen Lombardi, Shih-En Wei, Danielle Belko, Shoou-I Yu, et al.  Authentic volumetric avatars from a phone scan.  ACM Transactions on Graphics (TOG), 41(4):1–19, 2022.
  * [6] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  * [7] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799–5809, 2021.
  * [8] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [9] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [10] Zhiqin Chen, Kangxue Yin, and Sanja Fidler.  Auv-net: Learning aligned uv maps for texture transfer and synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1465–1474, 2022.
  * [11] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  * [12] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8188–8197, 2020.
  * [13] Christopher B Choy, JunYoung Gwak, Silvio Savarese, and Manmohan Chandraker.  Universal correspondence network.  Advances in neural information processing systems, 29, 2016.
  * [14] Yu Deng, Jiaolong Yang, and Xin Tong.  Deformed implicit field: Modeling 3d shapes with learned dense correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10286–10296, 2021.
  * [15] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  In Advances In Neural Information Processing Systems, 2022.
  * [16] Lily Goli, Daniel Rebain, Sara Sabour, Animesh Garg, and Andrea Tagliasacchi.  nerf2nerf: Pairwise registration of neural radiance fields.  arXiv preprint arXiv:2211.01600, 2022.
  * [17] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis.  arXiv preprint arXiv:2110.08985, 2021.
  * [18] Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi.  Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion.  arXiv preprint arXiv:2302.10109, 2023.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] Paul Henderson, Vagia Tsiminaki, and Christoph H Lampert.  Leveraging 2d data to learn textured 3d mesh generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7498–7507, 2020.
  * [21] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [22] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in Neural Information Processing Systems, 34:852–863, 2021\.
  * [23] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [24] Hyunsu Kim, Gayoung Lee, Yunjey Choi, Jin-Hwa Kim, and Jun-Yan Zhu.  3d-aware blending with generative nerfs.  arXiv preprint arXiv:2302.06608, 2023.
  * [25] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [26] Jiaman Li, Zhengfei Kuang, Yajie Zhao, Mingming He, Karl Bladin, and Hao Li.  Dynamic facial asset and rig generation from a single scan.  ACM Trans. Graph., 39(6):215–1, 2020.
  * [27] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang.  Joint-task self-supervised learning for temporal correspondence.  Advances in Neural Information Processing Systems, 32, 2019.
  * [28] Connor Z Lin, David B Lindell, Eric R Chan, and Gordon Wetzstein.  3d gan inversion for controllable portrait image animation.  arXiv preprint arXiv:2203.13441, 2022.
  * [29] Kai-En Lin, Yen-Chen Lin, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.  Vision transformer for nerf-based view synthesis from a single input image.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 806–815, 2023.
  * [30] Feng Liu and Xiaoming Liu.  Learning implicit functions for topology-varying dense 3d shape correspondence.  Advances in Neural Information Processing Systems, 33:4823–4834, 2020.
  * [31] Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany.  Learning smooth neural functions via lipschitz regularization.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–13, 2022.
  * [32] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [33] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [34] Jiteng Mu, Shalini De Mello, Zhiding Yu, Nuno Vasconcelos, Xiaolong Wang, Jan Kautz, and Sifei Liu.  Coordgan: Self-supervised dense correspondences emerge from gans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10011–10020, 2022.
  * [35] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen.  Point-e: A system for generating 3d point clouds from complex prompts.  arXiv preprint arXiv:2212.08751, 2022.
  * [36] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5480–5490, 2022.
  * [37] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.  Texture fields: Learning texture representations in function space.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531–4540, 2019.
  * [38] Dario Pavllo, Jonas Kohler, Thomas Hofmann, and Aurelien Lucchi.  Learning generative models of textured 3d meshes from real-world images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13879–13889, 2021.
  * [39] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi.  Lolnerf: Learn from one look.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1558–1567, 2022.
  * [40] Daniel Rebain, Mark J Matthews, Kwang Moo Yi, Gopal Sharma, Dmitry Lagun, and Andrea Tagliasacchi.  Attention beats concatenation for conditioning neural fields.  arXiv preprint arXiv:2209.10684, 2022.
  * [41] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  ACM Transactions on Graphics (TOG), 42(1):1–13, 2022.
  * [42] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2304–2314, 2019.
  * [43] Robin Sibson.  A brief description of natural neighbour interpolation.  Interpreting multivariate data, pages 21–36, 1981.
  * [44] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in Neural Information Processing Systems, 33:7462–7473, 2020.
  * [45] Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka.  Epigraf: Rethinking training of 3d gans.  arXiv preprint arXiv:2206.10535, 2022.
  * [46] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7495–7504, 2021.
  * [47] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu.  Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.  arXiv preprint arXiv:2205.15517, 2022.
  * [48] Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, and Jue Wang.  Fenerf: Face editing in neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7672–7682, 2022.
  * [49] Junshu Tang, Bo Zhang, Binxin Yang, Ting Zhang, Dong Chen, Lizhuang Ma, and Fang Wen.  Explicitly controllable 3d-aware portrait generation.  arXiv preprint arXiv:2209.05434, 2022.
  * [50] Matthew Turk and Alex Pentland.  Eigenfaces for recognition.  Journal of cognitive neuroscience, 3(1):71–86, 1991.
  * [51] Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  arXiv preprint arXiv:2210.04628, 2022.
  * [52] Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Hao Su.  Neutex: Neural texture mapping for volumetric neural rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7119–7128, 2021.
  * [53] Taihong Xiao, Sifei Liu, Shalini De Mello, Zhiding Yu, Jan Kautz, and Ming-Hsuan Yang.  Learning contrastive representation for semantic correspondence.  International Journal of Computer Vision, 130(5):1293–1309, 2022\.
  * [54] Jiarui Xu and Xiaolong Wang.  Rethinking self-supervised correspondence learning: A video frame-level similarity perspective.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10075–10085, 2021.
  * [55] Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang.  Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing.  arXiv preprint arXiv:2207.11911, 2022.
  * [56] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [57] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis.  Lion: Latent point diffusion models for 3d shape generation.  arXiv preprint arXiv:2210.06978, 2022.
  * [58] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv preprint arXiv:2010.07492, 2020.
  * [59] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  ACM Transactions on Graphics (TOG), 40(6):1–18, 2021.
  * [60] Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G Schwing, and Alex Colburn.  Generative multiplane images: Making a 2d gan 3d-aware.  arXiv preprint arXiv:2207.10642, 2022.

Supplementary material for
TEGLO: High Fidelity Canonical Texture Mapping from Single-View Images

##  S1 Network architecture details

TEGLO Stage-1 denoted as $\mathcal{N}$ consists of a latent table, StyleGANv2
[23] generator layers, a 2-layer tri-plane decoder, a differentiable volume
rendering module and a light weight camera predictor. For the experiments in
this work, we use 512-dimensional latents for the latent table. Following [6],
the output from the StyleGANv2 generator is of shape $256\times 256\times 96$
giving us $\mathrm{k}=32$-channel tri-planes. Note that despite the use of a
fixed size tri-plane, TEGLO enables arbitrary resolution synthesis as it
employs a GLO-based auto-decoder training strategy. As noted in the main
paper, this also enables single-view 3D reconstruction at any resolution.
Table.(S1), includes the details for the camera predictor network.

Table S1: Camera Predictor. Layer | Kernel | Filters | Stride | Activation
---|---|---|---|---
Conv 2D | 3 | 128 | 2 | LeakyReLU
Conv 2D | 3 | 64 | 2 | LeakyReLU
Conv 2D | 3 | 32 | 2 | LeakyReLU
Conv 2D | 3 | 16 | 2 | LeakyReLU
Dense | - | 25 | - | Linear

TEGLO Stage-2 consists of a latent mapping network ($\mathcal{L}$), a dense
correspondence network ($\mathcal{M}$) and a basis network ($\mathcal{C}$). We
describe the network details in Fig.(S2). $\mathcal{M}$ is a LipMLP [31] with
a Lipschitz regularizer at every layer to encourage Lipschitz continuity with
respect to the inputs. Note that the same network implementation is used for
all experiments - FFHQ [23], CelebA-HQ [32, 21], AFHQv2-Cats [22, 12] and SRN-
Cars [9, 8].

##  S2 Training details

In TEGLO Stage-1, to train $\mathcal{N}$, we use the single-view image dataset
and the approximate pose obtained from the shape-matching least-squares
optimization described in Sec.(4) in the main paper following the procedure in
[39]. We train $\mathcal{N}$ for 500K steps using the Adam optimizer [25] on 8
NVIDIA V100 GPUs (16 GB VRAM each) taking a total of 46 hours to complete
training at $256^{2}$ resolution. We also employ an exponential learning rate
decay from $5e$-4 to $1e$-4. In each train step, we use latent $w_{i}$ to
condition the NeRF to reconstruct the object ($o_{i}\in\mathcal{I}$). Then, we
use the loss
$\mathcal{L}_{\mathcal{N}}=\mathcal{L}_{\text{RGB}}+\mathcal{L}_{\text{Perceptual}}+\mathcal{L}_{\text{Camera}}$
to train the network.

Figure S1: Correspondence maps \- Establishing dense correspondence between 3D
objects in the 2D canonical co-ordinate space. Figure S2: TEGLO Stage-2
network details

To train TEGLO Stage-2, we first render a dataset $\mathcal{D}$ with five
camera views: $e_{i}=\\{v_{f},v_{l},v_{r},v_{t},v_{b}\\}$ for that object. In
this work, we render 1000 identities for $\mathcal{D}$ giving us 5000 total
views. We train TEGLO Stage-2 for 1 Million steps (1000 epochs) using the Adam
[25] optimizer on 8 NVIDIA V100 GPUs (16 GB VRAM each) taking a total of 50
hours to complete training. In each train step, we use the latent $w_{i}$ as
input to $\mathcal{L}$, 3D surface points ($p_{i}$) and shape-code from
$\mathcal{L}$ as input to $\mathcal{M}$, and the mapped canonical coordinate
points from $\mathcal{M}$ as input to $\mathcal{C}$. We compute the
$\mathcal{L}_{\text{Stage2}}$ loss using the output RGB ($r_{i}$), surface
normals ($s_{i}$), and surface points ($p_{i}$) with the respective ground-
truth $\\{\\{\widehat{r}_{i},\widehat{s}_{i},\widehat{p}_{i}\\}\in
e_{i}\\}\in\mathcal{D}$. The learned dense correspondences are then used for
TEGLO inference to extract the object texture. We show qualitative results for
the correspondence maps in Fig.(S1) where canonical coordinate points from
different posed images are mapped to the same location in the canonical
coordinate map (canonical coordinate points are output from $\mathcal{M}$).
Further, the quantitative results in Table.(1) and Table.(2) show a
significant increase in PSNR and LPIPS for reconstruction - demonstrating the
effectiveness of the dense correspondences learned by TEGLO Stage-2 for
texture representation and tasks such as texture transfer and texture editing.

##  S3 Ablations

Using $\mathcal{L}_{\text{Camera}}$ loss. EG3D [6] conditions the generator
and discriminator with the camera pose to enable 3D consistent novel view
synthesis. As noted in the main paper, the pose-conditioned generator does not
completely disentangle the camera pose from appearance leading to artifacts
such as facial expressions/eyes following the camera. In TEGLO Stage-1, we use
the $\mathcal{L}_{\text{Camera}},\mathcal{L}_{\text{RGB}}$ and
$\mathcal{L}_{\text{Perceptual}}$ losses to train $\mathcal{N}$. An ablation
experiment without the camera prediction loss led to 2D banner artifacts. This
is qualitatively represented in Fig.(S3) for “Views without
$\mathcal{L}_{\text{Camera}}$” with flat and inconsistent geometries for
different camera angles. However, the results for training $\mathcal{N}$ using
$\mathcal{L}_{\text{Camera}}$ show multi-view consistent representations
demonstrating the effectiveness of using the simple camera prediction loss.
Furthermore, we show that the rendered orbits do not have expressions/eyes
following the camera in the accompanying website. Qualitative results for
novel view synthesis is also presented in Fig.(S3, S4, S5, S7, S9, S10, S11,
S12, S13).

Figure S3: Ablation \- $\mathcal{L}_{\text{Camera}}$ for TEGLO Stage-1
training. Figure S4: Ablation \- Qualitative comparison with TEGLO-3DP (TEGLO
Stage-2 with 3D surface point reconstruction only)

K-d tree and NNI for textures. In the main paper, we discuss the necessity of
the K-d tree and Natural Neighbor Interpolation (NNI) [43] to prevent aliasing
artifacts and enable unambiguous indexing into the texture to obtain the RGB
values for each surface point. The Natural Neighbor Interpolation is defined
as follows:

$\mathcal{\text{NNI}}(x)=\sum_{i=0}^{n}w_{i}(x)\times f(x_{i})$ (S1)

$w_{i}(x)=\frac{\frac{1}{d_{i}(x)}}{\sum\limits_{j=0}^{n}\frac{1}{d_{j}(x)}}$
(S2)

Where $x$ is the query point, $w_{i}(x)$ is the simplified Laplace weight
based on inverse distances to n neighbors corresponding to the polygon
potentially encroached by the query point in the Voronoi tessellation plot,
and $f(x_{i})$ represents the extracted pixels from the texture. Storing the
texture in the K-d tree and using Natural Neighbor Interpolation enables
accurate and unambiguous floating point indexing into the texture to obtain
the RGB color (this is property of NNI using a tessellation plot). We also
note in the main paper about K-d tree + NNI enabling robustness to “holes” in
the texture. To verify the utility of K-d tree + NNI we design an experiment
where the texture is stored as an image and indexed using bi-linear
interpolation. The qualitative results are presented in Fig.(S5) where
$t_{\text{IMG}}$ is the texture with mapped ground truth pixels stored as an
image, and $t_{\text{GT}}$ is the TEGLO texture stored as a K-d tree. We
observe that the novel view synthesis using $t_{\text{IMG}}$ which includes
clipping the query indices to the texture image size and bi-linear
interpolation for indexing leads to several aliasing artifacts (indicated by
red arrows). This is because the canonical coordinate points may not be
aligned to the pixel centers and storing them in the discretized texture space
may lead to imprecision which manifests as aliasing artifacts in novel view
synthesis. We further note that the quantitative results for test set
reconstruction using $t_{\text{IMG}}$ is 30.828 dB PSNR and 0.0823 for LPIPS
at $256^{2}$ resolution for CelebA-HQ data. In contrast, novel view synthesis
using $t_{\text{GT}}$ leads to a significant reduction in aliasing artifacts
with quantitative results of 86.2 dB PSNR and 7.4e-7 LPIPS for test set
reconstruction of CelebA-HQ images at $256^{2}$ resolution (refer Fig.(S5) and
the accompanying website).

Figure S5: Texture ablation \- Qualitative comparison with texture stored as
an image indexed with bi-linear interpolation.

TEGLO Stage-2 with $\mathcal{L}_{\text{Coord}}$ loss only. Previous work AUV-
Net [10] states that methods [14, 30] that do not use color for learning dense
correspondences may learn sub-par texture representations. To verify, we train
TEGLO Stage-2 with only $\mathcal{L}_{\text{Coord}}$ reconstruction loss
instead of
$\mathcal{L}_{\text{Stage2}}=\mathcal{L}_{\text{Coord}}+\mathcal{L}_{\text{Coord}}+\mathcal{L}_{\text{Coord}}$
reconstruction losses. The qualitative results are presented in Fig.(S12)
comparing TEGLO-3DP with TEGLO and other baseline results. Of particular
interest is Fig.(S4) with qualitative results for TEGLO and TEGLO-3DP
including the texture image. We note that the reconstruction and novel view
synthesis results are nearly identical. However, we also observe TEGLO-3DP
including a wayward texture representation near the hair region. While the
dense correspondences map the surface points to the appropriate RGB image
pixels, there is a scope for null pixel artifacts around the hair region when
using NNI. While the 3D reconstruction and novel view synthesis for TEGLO-3DP
and TEGLO do not differ, we note the potential for black pixels to be obtained
in novel view synthesis leading to lowered qualitative and quantitative
results.

Limitations. As mentioned in the main paper, training TEGLO involves a
computational overhead. Further, TEGLO is only able to map target image pixels
spanning the target image and hence there are missing pixel artifacts for
camera views with minimal mapped target image pixels. For example, the texture
($T_{\text{GT}}$) in row-4 in Fig.(S13) includes missing pixels near the lower
jaw region. In the last column in row-4, the novel rendered view does not
include any target image pixels for the ear. On a similar note, in row-2,
column-7 in Fig.(S10), the novel view shows a slight twist in the nose
geometry partially due to the thin veil on the face which could not be
accounted for by TEGLO Stage-1.

##  S4 Qualitative results

We present qualitative results for texture transfer in Fig.(S6). We present
qualitative results for texture edits and textured synthesis for AFHQv2-Cats
in Fig.(S7), for SRN-Cars in Fig.(S8) and for CelebA-HQ in Fig.(S13). We
present single-view textured 3D reconstruction results for TEGLO trained on
FFHQ data and evaluated on CelebA-HQ image targets - focus on complex details
such eyeglasses and make-up - in Fig.(S11). We present extended results for
Fig.(10) in Fig.(S10) to compare row-1, row-2 with Fig.(24) in [5] and row-3
with Fig.(25) in [26]). We also include surface normal and shading results
with hats and eyeglasses in Fig.(S9). Further, we also present comparative
qualitative results with baselines and TEGLO-3DP (TEGLO Stage-2 trained with
only $\mathcal{L}_{\text{Coord}}$ loss) in Fig.(S12). Lastly, we present
qualitative results for high-resolution rendering at $1024^{2}$ resolution in
Fig(S14) and show novel view synthesis with a specific focus on high-frequency
details such as freckles (pigments under the skin) in row-1, make-up and
jewelry in row-2, hair and beard in row-3, fine skin details in row-4 and
wrinkles in row-5.

Figure S6: Texture transfer \- Qualitative results for texture transfer with
CelebA-HQ. (Top row shows CelebA-HQ image targets).

##  S5 Efficient dataset rendering

TEGLO enables 3D high-fidelity data rendering from single-view image
collections of objects at arbitrary resolution. Since the dense
correspondences are learned point-wise (_i.e_. using $\widehat{p}_{j}$), there
is no spatial constraint in querying $\mathcal{M}$ for the canonical
coordinate point. Hence, we render images of any size by first dividing the
image pixels into 4 tiles, then obtain the surface points from TEGLO Stage-1,
map the surface points to the canonical coordinate space using $\mathcal{M}$
and then index into the texture to obtain the RGB color value. Then, after all
the tiles are computed, we can combine the divided computations into a single
image of high-resolution. The orbit and the high-resolution frames at
$1024^{2}$ resolution in Fig.(S14) and on the website have been obtained using
this approach.

##  S6 Ethical considerations

One of the motivating goals for TEGLO stems from the need for photorealistic
3D reconstruction of objects from single-view image collections. By using a
compiled set of images, TEGLO helps mitigate bias, reduce costs and also
improve access to high-quality data for the broader research community. Hence,
TEGLO enables rendering a dataset of diverse objects (improving fairness and
mitigating bias) and also reduces the need for large scale data collection.
Further, we acknowledge the potential misuse of TEGLO especially for single
view 3D reconstruction and hence only make available the code for
reproducibility purposes. We will not release any trained weights for our
method.

Figure S7: Textured synthesis \- Target view reconstruction and novel view
synthesis for AFHQv2-Cats.

Figure S8: Textured synthesis \- Target view reconstruction and novel view
synthesis for SRN-Cars.

Figure S9: Geometry results \- Target view reconstruction geometry for CelebA-
HQ with hat and eyeglasses. Figure S10: Complex geometry and texture \-
Qualitative results for texture edits.

Figure S11: Single view 3D reconstruction \- Results for TEGLO trained on FFHQ
data and evaluated on CelebA-HQ image targets. Figure S12: Qualitative results
\- Comparison with relevant 3D-aware generative baseline methods at $256^{2}$
resolution for CelebA-HQ. Figure S13: Textured synthesis \- Target view
reconstruction and novel view synthesis for CelebA-HQ. Figure S14: High
resolution rendered views \- Qualitative results for novel views in high-
resolution with high-frequency details such as freckles, jewelry, make-up,
hair, fine details and wrinkles.