# Neural Directional Encoding
for Efficient and Accurate View-Dependent Appearance Modeling

Liwen Wu1 Sai Bi2 Zexiang Xu2 Fujun Luan2 Kai Zhang2
Iliyan Georgiev2 Kalyan Sunkavalli2 Ravi Ramamoorthi1
1UC San Diego 2Adobe Research This work was partially done during an
internship at Adobe Research.

###### Abstract

Novel-view synthesis of specular objects like shiny metals or glossy paints
remains a significant challenge. Not only the glossy appearance but also
global illumination effects, including reflections of other objects in the
environment, are critical components to faithfully reproduce a scene. In this
paper, we present Neural Directional Encoding (NDE), a view-dependent
appearance encoding of neural radiance fields (NeRF) for rendering specular
objects. NDE transfers the concept of feature-grid-based spatial encoding to
the angular domain, significantly improving the ability to model high-
frequency angular signals. In contrast to previous methods that use encoding
functions with only angular input, we additionally cone-trace spatial features
to obtain a spatially varying directional encoding, which addresses the
challenging interreflection effects. Extensive experiments on both synthetic
and real datasets show that a NeRF model with NDE (1) outperforms the state of
the art on view synthesis of specular objects, and (2) works with small
networks to allow fast (real-time) inference. The project webpage and source
code are available at: https://lwwu2.github.io/nde/.

NDE (ours) | Ground truth
---|---
|
|  |  |
|  |  |
|  |  |
|  |  |
ENVIDR [27] | Ref-NeRF [49] | NDE (ours) | Ground truth
0.52 FPS | 0.02 FPS | 75 FPS |

Figure 1:  Ours vs. analytical encoding. Methods like Ref-NeRF [49] use an
analytical function to encode viewing directions in large MLPs, failing to
model complex reflections (column 1-2 of the insets). Instead, we encode view-
dependent effects into feature grids with better interreflection
parameterization, successfully reconstructing the details on the teapot and
even multi-bounce reflections of the pink ball (3rd column of the insets) with
little computational overhead (75 FPS on an NVIDIA 3090 GPU).

##  1 Introduction

Some of the most compelling appearances in our visual world arise from
specular objects like metals, plastics, glossy paints, or silken cloth.
Faithfully reproducing these effects from photographs for novel-view synthesis
requires capturing both geometry and view-dependent appearance. Recent neural
radiance field (NeRF) [38] methods have made impressive progress on efficient
geometry representation and encoding using learnable spatial feature grids
[30, 54, 46, 8, 40, 6]. However, modeling high-frequency view-dependent
appearance has achieved much less attention. Efficient encoding of directional
information is just as important, for modeling effects such as specular
highlights and glossy interreflections. In this paper, we present a feature-
grid-like _neural directional encoding_ (NDE) that can accurately model the
appearance of shiny objects.

View-dependent colors in NeRFs (_e.g_. [49]) are commonly obtained by decoding
spatial features and encoded direction. This approach necessitates a large
multi-layer perceptron (MLP) and exhibits slow convergence with analytical
directional encoding functions. To that end, we bring feature-grid-based
encoding to the directional domain, representing reflections from distant
sources via learnable feature vectors stored on a global environment map (Sec.
4.1). Features localize signal learning, reducing the MLP size required to
model high-frequency far-field reflections.

Besides far-field reflections, spatially varying near-field interreflections
are also key effects in rendering glossy objects. These effects cannot be
accurately modeled by NeRF’s spatio-angular parameterization whose directional
encoding does not depend on the position. In contrast, we propose a novel
spatio-spatial parameterization by _cone-tracing a spatial feature grid_ (Sec.
4.2) to encode near-field reflections. The cone tracing accumulates spatial
encodings along the queried direction and position, thus it is spatially
varying. While prior works consider only single-bounce or diffuse
interreflections [27], our representation is able to model general multi-
bounce reflection effects.

Overall, our neural directional encoding (NDE) achieves both high-quality
modeling of view-dependent effects and fast evaluation. Figure 1 demonstrates
NDE incorporated into NeRF, showing (1) accurate rendering of specular
objects—a difficult challenge for the state of the art (Sec. 5.1), and (2)
high inference speed that can be pushed to real-time without obvious quality
loss (Sec. 5.2).

##  2 Related work

Novel-view synthesis aims to render a 3D scene from unseen views given a set
of image captures with camera poses. Neural radiance fields (NeRF) [38] has
recently emerged as a promising solution to this task, utilizing an implicit
scene representation and volume rendering to synthesize photorealistic images.
Follow-up works achieve state-of-the-art results in this area, for unbounded
scenes [60, 2], in-the-wild captures [35], and sparse- or single-view
reconstruction [7, 52, 29, 15, 47, 48]. While the original NeRF method [38] is
computationally inefficient, it can be visualized in real-time by baking the
reconstruction into voxel- [58, 13, 16, 44] or feature-grid-based
representations (discussed below). The volumetric representation has been
extended to work with signed distance fields (SDF) [51, 56] for better
geometry acquisition, and the volume-rendering concept has also been applied
to other 3D-related tasks such as object generation [5, 6, 31, 43, 28].

#### Feature-grid-based NeRF.

NeRF’s positional encoding [38] is a key component for the underlying multi-
layer perceptron (MLP) network to learn high-frequency spatial and directional
signals. However, the MLP size needs to be large, which leads to slow training
and inference. Instead, methods like NSVF [30] and DVGO [46] interpolate a 3D
volume of learnable feature vectors to encode the spatial signal, showing
faster training and inference with even better spatial detail. Addressing the
sparsity in typical scene geometry, later works avoid maintaining a large
dense 3D grid via volume-compression techniques such as hash grids [40] and
tensor factorization [8, 6, 12]. These methods are compact and scale up the
feature grid to large scenes [40, 3] and even work with SDF-based models [57,
26]. The essence of feature-grid encoding is to interpolate feature vectors
attached to geometry primitives, and similar ideas have also been applied to
irregular 3D grids [45, 23], point clouds [55, 20, 21, 63], and meshes [9].
Operations like mip-mapping are trivial on feature grids, enabling efficient
anti-aliasing and range query of NeRF models [54, 17, 3]—something we also
leverage in this paper to encode rough reflection.

#### Rendering specular objects.

Apart from geometry, view-dependent effects like reflections from rough
surfaces are a crucial component in photorealistic novel-view synthesis.
Reflections are conventionally modeled by fitting local light-field functions
[11, 37, 18]. A 4D light field presents more degrees of freedom than the
constraints from input images, which necessitates additional regularization to
avoid overfitting. Inverse-rendering approaches introduce such a constraint by
solving for parametric BRDFs and lighting, then using forward rendering to
reconstruct the light field. Spherical-basis lighting [61] or split-sum
approximation [41, 32] are usually used to tamper the Monte Carlo variance of
specular-reflection derivatives [4]. ENVIDR [27] and NMF [34] further
explicitly consider global-illumination effects by ray-tracing one or few
bounces of indirect lighting. On the other hand, Ref-NeRF [49] uses an
integrated directional encoding (IDE) to directly improve NeRF’s view-
dependent effects. IDE encodes the reflected direction rather than viewing
direction to let the network learn an environment-map-like function and is
pre-filtered to account for rough reflection effects. Our neural directional
encoding, similar to IDE, can model general view-dependent appearance without
assuming simplified lighting or reflections but with smaller computation cost.

Figure 2:  Pipeline of our neural directional encoding (NDE). We encode far-
field reflections into a cubemap and near-field interreflections into a
volume. Both representations store learnable feature vectors to encode
direction and are mip-mapped to account for rough reflections. Given a
reflected ray, the features are combined by tracing a cone of size
proportional to the surface roughness to aggregate spatial features with
cubemap features blended as the background. The result is fed into an MLP to
output the specular color (Eq. 5).

##  3 Preliminaries

We assume opaque objects with diffuse and specular components and demonstrate
our directional encoding using a surface-based model that represents a scene
using a signed distance field (SDF) $s(\mathbf{x})$ and a color field
$\mathbf{c}(\mathbf{x},\bm{\omega})$ (dependent on the viewing direction
$\bm{\omega}$). The SDF is converted to NeRF’s density field $\sigma$
following VolSDF [56] with a learnable parameter $\beta$ controlling the
boundary smoothness:

$\sigma(\mathbf{x})=\begin{cases}\frac{1}{2\beta}\exp\left(\frac{s(\mathbf{x})}{\beta}\right)&\text{if
}s(\mathbf{x})\leq 0,\\\
\frac{1}{\beta}\left(1-\frac{1}{2}\exp\left(-\frac{s(\mathbf{x})}{\beta}\right)\right)&\text{otherwise}.\end{cases}$
(1)

The color $\mathbf{C}(\mathbf{x},\bm{\omega})$ of a ray with origin
$\mathbf{x}$ and direction $\bm{\omega}$ can thus be volume-rendered [36]:

$\displaystyle\mathbf{C}(\mathbf{x},\bm{\omega})\\!=\\!\sum_{i}w(\sigma(\mathbf{x}_{i}))\mathbf{c}(\mathbf{x}_{i},\bm{\omega}),\
\text{where}$ (2) $\displaystyle
w(\sigma(\mathbf{x}_{i}))=\left(1-e^{-\sigma(\mathbf{x}_{i})\delta_{i}}\right)\prod_{j<i}e^{-\sigma(\mathbf{x}_{j})\delta_{j}},$
(3)

with $\delta_{i}\\!=\\!\|\mathbf{x}_{i}\\!-\\!\mathbf{x}_{i-1}\|_{2}$ and
$\mathbf{x}_{i}$ denoting the $i^{\text{th}}$ sample point along the ray. Like
Ref-NeRF [49], we decompose the color $\mathbf{c}$ into a diffuse color
$\mathbf{c}_{d}$, specular tint $\mathbf{k}_{s}$, and specular color
$\mathbf{c}_{s}$ queried in reflected direction $\bm{\omega}_{r}$ with surface
normal $\mathbf{n}$ given by the SDF gradient:

$\begin{gathered}\mathbf{c}(\mathbf{x},\bm{\omega})=\mathbf{c}_{d}(\mathbf{x})+\mathbf{k}_{s}(\mathbf{x})\mathbf{c}_{s}(\mathbf{x},\bm{\omega}_{r}),\quad\text{where}\\\
\bm{\omega}_{r}=\text{reflect}(\bm{\omega},\mathbf{n}),\quad\mathbf{n}=\text{normalize}(\nabla_{\mathbf{x}}s(\mathbf{x})).\end{gathered}$
(4)

Here, the specular color $\mathbf{c}_{s}$ is decoded from an MLP that
conditions on spatial feature $\mathbf{f}(\mathbf{x})$, directional encoding
$\mathbf{H}$ controlled by surface roughness $\rho$, and the cosine term
$\mathbf{n}\cdot\bm{\omega}$:

$\mathbf{c}_{s}(\mathbf{x},\bm{\omega}_{r})=\text{MLP}(\mathbf{f}(\mathbf{x}),\mathbf{H}(\mathbf{x},\bm{\omega}_{r},\rho(\mathbf{x})),\mathbf{n}\cdot\bm{\omega}).$
(5)

$\mathbf{c}_{d},\mathbf{k}_{s},\mathbf{f},\rho$ come from a spatial MLP (Sec.
4.3).

#### Discussion on directional encoding.

Previous works [49, 38] use an analytical function for $\mathbf{H}$ dependent
only on $\bm{\omega}_{r}$ (and optionally $\rho$), which has several
limitations: (1) the encoding function is fixed (not learnable), and (2) the
spatial context only comes from $\mathbf{f}(\mathbf{x})$. Both require the
decoder MLP to be large to fit the spatio-angular details of the specular
color, which can be expensive and slow.

##  4 Neural directional encoding

To minimize the MLP complexity, we use a learnable neural directional encoding
that also depends on the spatial location. Specifically, our NDE encodes
different types of reflection by different representations, which include a
cubemap feature grid $\mathbf{h}_{f}$ for far-field reflections and a spatial
volume $\mathbf{h}_{n}$ that models near-field interreflections. As shown in
Fig. 2, we compute $\mathbf{H}$ by first cone-tracing $\mathbf{h}_{n}$
accumulated along the reflected ray, yielding near-field feature
$\mathbf{H}_{n}$ (Sec. 4.2), and blending the far-field feature
$\mathbf{H}_{f}$ queried from $\mathbf{h}_{f}$ in the same direction (Sec.
4.1):

$\mathbf{H}(\mathbf{x},\bm{\omega}_{r},\rho)=\mathbf{H}_{n}(\mathbf{x},\bm{\omega}_{r},\rho)+(1-\alpha_{n})\mathbf{H}_{f}(\bm{\omega}_{r},\rho),$
(6)

where $\alpha_{n}$ is the cone-traced opacity [25], and both features are mip-
mapped with $\rho$ deciding the mip level.

###  4.1 Far-field features

Feature-grid-based representations [8, 40, 30, 54, 46] speed-up spatial signal
learning by storing feature vectors in voxels for local signal control.
Similarly, we place feature vectors $\mathbf{h}_{f}$ at every pixel of a
global cubemap to encode ideal specular reflections. The cubemap is pre-
filtered to model reflections under rough surfaces in the split-sum [19]
style, where the $k^{\text{th}}$ level mip-map $\mathbf{h}_{f}^{k}$ is created
by convolving the downsampled $\mathbf{h}_{f}$ using a GGX kernel [50] $D$
with canonical roughness $\rho_{k}$ evenly spaced in $[0,1]$:

$\mathbf{h}_{f}^{k}=\text{convolution}(\text{downsample}(\mathbf{h}_{f},k),D(\rho_{k})).$
(7)

Given the surface roughness, we perform a cubemap lookup in the reflected
direction and interpolate between mip levels to get the far-field feature:

$\mathbf{H}_{f}(\bm{\omega}_{r},\rho)=\text{lerp}\left(\mathbf{h}^{k}_{f}(\bm{\omega}_{r}),\mathbf{h}^{k+1}_{f}(\bm{\omega}_{r}),\frac{\rho\\!-\\!\rho_{k}}{\rho_{k+1}\\!-\\!\rho_{k}}\right),$
(8)

where $\text{lerp}(\cdot)$ denotes linear interpolation and
$\rho\in[\rho_{k},\rho_{k+1}]$.

The cubemap-based encoding allows signals in different directions to be
optimized independently by tuning the feature vectors. This is easier to
optimize than globally solving the MLP parameters, making it more suitable to
model high-frequency details in the angular domain (Fig. 3). The coarse level
feature is a consistently filtered version of the fine level, which is
empirically found to be better constrained than using independent feature
vectors at each mip level [24, 59].

|  |  |
---|---|---|---
IDE small | IDE large |  $\mathbf{H}_{f}$ small (ours) | Ground truth

Figure 3:  Our cubemap-based feature encoding requires only a small MLP (2
layers, 64 width) to model details in mirror reflections (3rd image)
comparable with IDE [49] (2nd image; 8 layers, 256 width MLP) that fails when
the MLP is small (1st image).

|  |
---|---|---
Spatio-angular | Spatio-spatial | Cone-traced

Figure 4:  Spatio-spatial encoding (middle) is equivalent to the common
spatio-angular encoding (left) of mirror reflections, but it captures the
variation of $\mathbf{x}^{\prime}$ across different $\mathbf{x}$. The idea can
be extended to model rough reflections by cone tracing mip-mapped spatial
features covered by the reflection cone (right).

###  4.2 Near-field features

Parameterizing the specular color by a spatial and angular feature is
sufficient for distant reflections, but lacks expressivity for near-field
interreflections: different points query the same $\mathbf{h}_{f}$, so
spatially varying components can end up being averaged out during
optimization. Our insight is that the spatio-angular reflection can also be
parameterized as a spatio-spatial function of current and next bounce location
(Fig. 4). Therefore, an MLP can decode the second bounce spatial feature with
$\mathbf{f}(\mathbf{x})$ in Eq. 5 to get mirror reflections.

For rough reflections, we aggregate the averaged second bounce feature under
the reflection lobe by cone tracing [10] (Fig. 4, right), which volume renders
the mip-mapped spatial features $\mathbf{h}_{n}$ using the mip-mapped density
$\sigma_{n}$ along the reflected ray $\mathbf{x}\\!+\\!\bm{\omega}_{r}t$ with
mip level $\lambda_{i}\\!=\\!\log_{2}(2r_{i})$ at sample point
$\mathbf{x}^{\prime}_{i}$ decided by the cone’s footprint
$r_{i}\\!=\\!\sqrt{3}\rho^{2}\|\mathbf{x}-\mathbf{x}^{\prime}_{i}\|_{2}$:

$\begin{gathered}\mathbf{H}_{n}(\mathbf{x},\bm{\omega}_{r},\rho)=\sum_{i}w_{n}^{i}\mathbf{h}_{n}^{i},\quad\text{where}\\\
w^{i}_{n}=w(\sigma_{n}(\mathbf{x}^{\prime}_{i},\lambda_{i})),\quad\mathbf{h}^{i}_{n}=\mathbf{h}_{n}(\mathbf{x}^{\prime}_{i},\lambda_{i}).\end{gathered}$
(9)

The cone’s footprint is selected to cover the GGX lobe at $\mathbf{x}$ (see
supplemental document). Note that we do not use the SDF-converted $\sigma$ in
Eq. 1 as it cannot be mip-mapped; instead, we optimize a separate $\sigma_{n}$
to match $\sigma$ (Sec. 4.3) jointly with the indirect feature
$\mathbf{h}_{n}$. Both are decoded from a tri-plane [6] $\mathbf{T}_{n}$,
whose each 2D plane is mip-mapped similar to Tri-MipRF [17]:

$\\!\\!\\!\sigma_{n}(\mathbf{x}^{\prime}_{i},\lambda_{i}),\mathbf{h}_{n}(\mathbf{x}^{\prime}_{i},\lambda_{i})\\!=\\!\text{MLP}(\text{mipmap}(\mathbf{T}_{n}(\mathbf{x}^{\prime}_{i}),\lambda_{i})).\\!$
(10)

The indirect rays are spatially varying, hence the cone-traced near-field
features are spatially varying too. This has advantages over the angular-only
feature for learning interreflections and is empirically less likely to
overfit (Fig. 5). This is because the same $\mathbf{h}_{n}$ is traced from
different rays in training, such that the underlying representation is well-
constrained. $\mathbf{H}_{n}$ and $\mathbf{H}_{f}$ are similar to the
foreground and background colors in regular volume rendering, so
$\mathbf{H}_{f}$ can be naturally composited with $\mathbf{H}_{n}$ using the
opacity
$\alpha_{n}\\!=\\!1-\prod_{i}e^{-\sigma_{n}(\mathbf{x}^{\prime}_{i},\lambda_{i})\delta_{i}}\\!=\\!\sum_{i}w_{n}^{i}$
as in Eq. 6.

|  |
---|---|---
|  |
Ours without $\mathbf{H}_{n}$ |  Ours with $\mathbf{H}_{n}$ | Ground truth

Figure 5:  Our cone-traced near-field features successfully reconstruct the
reflected spheres (2nd column) under novel views, which are overfitted by the
angular-only encoding (1st column).  Figure 6:  Network architectures.
$N\\!\times\\!M$ denotes an $M$-layer MLP of width $N$.

###  4.3 Optimization

Figure 6 shows our network architectures. Stable geometry optimization is
essential for modeling specular objects, so we use the positional-encoded MLP
from VolSDF [56] to output the SDF. To reduce computation cost, a hash grid is
used to encode other spatial features
($\mathbf{c}_{d},\mathbf{k}_{s},\rho,\mathbf{f}$), and all other MLPs are
tiny. The representation is optimized through the Charbonnier loss [2] between
ground truth pixel color $\mathbf{C}_{\text{gt}}$ and our rendering
$\mathbf{C}$ in tone-mapped space:

$L=\sum_{\mathbf{x},\bm{\omega}}\sqrt{\left\|\Gamma(\mathbf{C}(\mathbf{x},\bm{\omega}))-\mathbf{C}_{\text{gt}}(\mathbf{x},\bm{\omega})\right\|_{2}^{2}+0.001},$
(11)

where $\Gamma$ is the tone-mapping function [41].

#### Occupancy-grid sampling.

Eqs. 3 and 9 are accelerated by an occupancy-grid estimator [25] to get rid of
computations in empty space. This is especially important for the efficient
near-field feature evaluation, since we trace a reflected ray for each primary
ray sample. The primal ray rendering uses a fixed ray marching step of 0.005.
Following [10], we choose the cone tracing step proportional to its footprint:
$\max{(0.5r_{i},0.005)}$, and query a mip-mapped occupancy grid for the
correct occupancy information.

#### Regularization.

Given the primary samples $\mathbf{x}_{i}$, Eikonal loss [56] $L_{\text{eik}}$
is applied to regularize the SDF, and we implicitly regularize $\sigma_{n}$ to
match $\sigma$ by encouraging the rendering using $\sigma_{n}$ at mip level 0
to be close to the ground truth:

$\begin{gathered}L_{\sigma}\\!=\\!\sum_{\mathbf{x},\bm{\omega}}\|\mathbf{C}_{\sigma}(\mathbf{x},\bm{\omega})-\mathbf{C}_{\text{gt}}(\mathbf{x},\bm{\omega})\|_{2}^{2},\quad\text{where}\\\
\mathbf{C}_{\sigma}(\mathbf{x},\bm{\omega})=\sum_{i}w(\sigma_{n}(\mathbf{x}_{i},0))\mathring{\mathbf{c}}(\mathbf{x}_{i},\bm{\omega}),\end{gathered}$
(12)

$\mathring{\square}$ denotes stop-gradient to prevent $\sigma_{n}$ affecting
appearance. The total loss is $L+0.1L_{\text{eik}}+0.01L_{\sigma}$.

#### Implementation details.

We implement our code using PyTorch [42], NerfAcc [25], and CUDA. The
optimization takes 400k steps using the Adam optimizer [22] with 0.0005
learning rate and dynamic batch size [40] targeting for 32k primary point
samples. We use the scheduler from BakedSDF [16] to anneal $\beta$ in Eq. 1
for more stable convergence. Because the SDF uses a positional-encoded MLP,
each scene still requires 10$\sim$18 hours to train on an NVIDIA 3090 GPU with
15GB GPU memory usage.

##  5 Experiments

We evaluate our method on view synthesis of specular objects using synthetic
and real scenes. The synthetic scenes include the Shinny Blender dataset [49]
and the Materials scene from the NeRF Synthetic dataset [38], all rendered
without background; the real scenes come from NeRO [32] which contain
backgrounds and reflections of the capturer in the images. The rendering
quality is compared in terms of PSNR, SSIM [53], LPIPS [62], and the inference
speed in FPS is recorded on an NVIDIA 3090 GPU.

#### Background and capturer.

For real scenes, we use a separate Instant-NGP [40] with coordinate
contraction [2] to render backgrounds. Similarly to NeRO [32], the reflection
of the capturer is encoded by blending a capturer plane feature
$\mathbf{h}_{c}$ of opacity $\alpha_{c}$ between $\mathbf{H}_{f}$ and
$\mathbf{H}_{n}$:

$\begin{gathered}\mathbf{H}=\mathbf{H}_{n}+(1-\alpha_{n})(\alpha_{c}\mathbf{h}_{c}+(1-\alpha_{c})\mathbf{H}_{f}),\;\text{where}\\\
\alpha_{c},\mathbf{h}_{c}=\text{MLP}(\text{mipmap}(\mathbf{T}_{c}(\mathbf{u}),\lambda_{c}))\end{gathered}$
(13)

are decoded from a mip-mapped 2D feature grid $\mathbf{T}_{c}$;
$\mathbf{u},\lambda_{c}$ are the ray-plane intersection coordinate and the
mip-level derived from the intersection footprint. Jointly optimizing
foreground and background networks can be unstable, so we apply stabilization
loss from NeRO [32] and modify the specular color computation for the first
200k steps: $\mathbf{h}_{f},\mathbf{h}_{n},\mathbf{h}_{c}$ are sampled and
decoded into colors first, then the colors are blended to get
$\mathbf{c}_{s}$. Compared to blending the feature and decoding, we find the
decoding-then-blending strategy provides better geometry optimization.

Method | Mat. | Teapot | Toaster | Car | Ball | Coffee | Helmet | Mean
---|---|---|---|---|---|---|---|---
PSNR $\uparrow$
NeRO | 24.85 | 40.29 | _27.31_ | 26.98 | 31.50 | 33.76 | 29.59 | 30.61
ENVIDR | 29.51 | 46.14 | 26.63 | 29.88 | 41.03 | _34.45_ | _36.98_ | 34.95
Ref-NeRF | 35.41 | _47.90_ | 25.70 | 30.82 | 47.46 | 34.21 | 29.68 | _35.88_
NDE (ours) | _31.53_ | 49.12 | 30.32 | _30.39_ | _44.66_ | 36.57 | 37.77 | 37.19
SSIM $\uparrow$
NeRO | 0.878 | 0.993 | 0.891 | 0.926 | 0.953 | 0.960 | 0.953 | 0.936
ENVIDR | 0.971 | 0.999 | _0.955_ | 0.972 | 0.997 | 0.984 | 0.993 | 0.982
Ref-NeRF | 0.983 | 0.998 | 0.922 | 0.955 | _0.995_ | 0.974 | 0.958 | _0.969_
NDE (ours) | _0.972_ | 0.999 | 0.968 | _0.968_ | _0.995_ | _0.979_ | _0.990_ | 0.982
LPIPS $\downarrow$
NeRO | 0.138 | 0.017 | 0.162 | 0.064 | 0.179 | 0.099 | 0.102 | 0.109
ENVIDR | 0.026 | _0.003_ | 0.097 | _0.031_ | 0.020 | _0.044_ | _0.022_ | _0.035_
Ref-NeRF | _0.022_ | 0.004 | _0.095_ | 0.041 | 0.059 | 0.078 | 0.075 | 0.053
NDE (ours) | 0.017 | 0.002 | 0.039 | 0.024 | _0.022_ | 0.033 | 0.014 | 0.022

Table 1: Quantitative comparison on synthetic scenes showing our encoding
(NDE) is either the best or _second best_ compared to other methods for view
synthesis of specular objects.

| ENVIDR [27] | Ref-NeRF [49] | NDE (ours) | Ground truth | ENVIDR | Ref-NeRF | NDE (ours) | GT
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |
Rendering |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
Rendering/Normal |  |  |  |  |  |  |  |

Figure 7: Qualitative results for synthetic scenes show our NDE successfully
models the fine details of reflections from both environment lights (mirror
sphere and car top) and other objects (glossy interreflections on spheres;
zoom in to see the difference). Ref-NeRF tends to use wrong geometry to fake
interreflections (2nd column on bottom). In contrast, our encoding has
sufficient capacity to model interreflections, which enables more accurate
normals (3rd column on bottom). Mean angular error of the normal is shown in
the insets.

###  5.1 View synthesis

We compare against NeRO [32], ENVIDR [27], and Ref-NeRF [49] on synthetic
scenes. All methods except for Ref-NeRF use SDFs, and we evaluate NeRO after
the BRDF estimation as it shows better performance. Ideally, both backgrounds
and reflections from the capturer should be removed when evaluating renderings
of specular objects, which is difficult for the real scenes. Therefore, we
only qualitatively compare real scenes against NeRO with PSNR computed on the
foreground zoom-ins without the capturer.

NeRO [32] | NDE (ours) | Ground truth
---|---|---
|  |
|  |
|  |

Figure 8:  Qualitative comparison on real scenes. Our NDE gives better
reconstruction of the interreflections (the bear’s plate and bottom of the
vase) and detailed highlights from the environment. Numbers in the insets are
image PSNR values.

#### Results.

Overall, our method gives the best rendering quality on synthetic scenes with
quantitative results either better or comparable with the baselines (Tab. 1).
This is because our NDE gives the most detailed modeling of both far-field
reflections and interreflections, which also helps improve the geometry
reconstruction (Fig. 7 bottom). While ENVIDR’s SSIM is slightly better than
ours in several scenes, we not only achieve much better PSNRs (surpassing
2dB), but also higher LPIPS scores. The PSNR on the Materials (Mat.) scene is
worse than Ref-NeRF’s because the SDF is inefficient at modeling the concave
geometry of the sphere base. However, our directional MLP is much smaller
(Sec. 5.2), and we still achieve perceptually better appearance as shown in
the insets of Fig. 7. The qualitative comparison in Fig. 8 shows that NDE
extends well to real scenes, producing clearer specular reflections of the
complex real-world environments compared to NeRO.

#### Editability.

The near- and far-field features provide a natural separation of different
reflections, allowing us to render these effects separately by excluding
$\mathbf{H}_{f}$ or $\mathbf{H}_{n}$ during inference (Fig. 9). Because
interreflections are spatially encoded in the near-field feature grid, an
object and its first-bounce reflections can be removed by masking out both
$\sigma$ and $\sigma_{n}$ from the corresponding regions (Fig. 10). This does
not work for multi-bounce reflections which are not encoded on the deleted
object.

|  |
---|---|---
Far-field reflections | Near-field reflections | Combined
|

Figure 9:  Reflection separation. We can visualize different reflection
effects by feeding corresponding features into the network.

---

Figure 10:  Editability of our encoding. Reflections from the deleted spheres
can be removed by deleting the volume of their indirect features (bottom).

###  5.2 Performance comparison

We compare the evaluation frames per second (FPS) on an $800\\!\times\\!800$
resolution of the color network and its MLP size (#Params.) with all baselines
in Sec. 5.1 on synthetic scenes. The color MLPs include the decoder of
$\sigma_{n},\mathbf{h}_{n},\mathbf{c}_{s}$ for our model (Fig. 6), lighting
MLPs for NeRO [32] and ENVIDR [27], and the directional MLP for Ref-NeRF [49].
The spatial-network evaluation is excluded to eliminate the difference caused
by different geometry representations, network architectures, and sampling
strategies. For each method, we choose the rendering batch size that maximizes
its performance.

#### Results.

As shown in the top half of Tab. 2, our NDE takes a fraction of a second to
evaluate, because it requires substantially smaller MLPs to infer color
without hurting the rendering. In contrast, other baselines need large MLPs to
maintain rendering quality, which prevents them to be visualized in real-time.

#### Real-time application.

It is possible to create a real-time version of our model by converting the
SDF into a mesh through marching cubes [33] and baking
$\mathbf{c}_{d},\mathbf{k}_{s},\rho,\mathbf{f}$ into mesh vertices. The pixel
color then can be computed using the rasterized vertex attributes and
$\mathbf{c}_{s}$ decoded from the NDE, which takes only a single cubemap
lookup and cone tracing for each pixel. As a result, this process requires
about the same budget as evaluating a real-time NeRF model [54, 46, 40]. We
implement our real-time model (NDE-RT) in WebGL and report the full rendering
frame rate (not just color evaluation) at the bottom of Tab. 2 with a real-
time baseline 3DGS [20]. 3DGS is faster as it uses spherical harmonics for
color without network evaluation, which leads to poor specular appearance
reconstruction. Instead, our NDE-RT shows rendering quality comparable to
other baselines while achieving frame rates above 60. The loss in PSNR is
mainly due to error around object edges which is cause by the marching-cube
mesh extraction and subsequent rasterization (Fig. 11). This error does not
significantly affect the visual quality and can be resolved by fine-tuning the
mesh [9, 41].

|  |  |
---|---|---|---
Ground truth | Our offline model | Our real-time model |

Figure 11:  Error near object boundaries in our real-time model is caused by
the marching-cube extraction of a triangle mesh and its subsequent
rasterization (squared error maps at the bottom). This error does not lead to
significant qualitative differences (top).

Method | FPS$\uparrow$ | #Params$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---|---
NeRO | 0.11 | 454k | 30.61 | 0.936 | 0.109
ENVIDR | 0.55 | _206k_ | 34.95 | 0.982 | 0.035
Ref-NeRF | 0.08 | 521k | _35.88_ | 0.969 | 0.053
NDE (ours) | 3.03 | 75k | 37.19 | 0.982 | 0.022
3DGS | 235 | - | 30.30 | 0.949 | 0.076
NDE-RT (ours) | _66_ | 75k | 35.48 | _0.976_ | _0.027_

Table 2:  Performance comparison. Our NDE achieves high rendering quality, and
its use of small MLPs enables fast color evaluation and real-time rendering.
We report only the evaluation time and parameter counts of color MLPs except
for 3DGS (no color MLPs) and our NDE-RT, for which we report the total
rendering time. All metrics are averaged over the synthetic scenes in Tab. 1.

###  5.3 Ablation study

#### Different directional encodings.

In Fig. 12 we compare different directional encodings on the Materials scene.
IDE [49] (analytical) with our tiny MLP yields blurry reflections.
Interreflections cannot be reconstructed using only the far-field feature, and
if we volume-render rather than cone-trace the near-field feature, mirror
interreflections can be recovered but reflections on rough surfaces look too
sharp. It is therefore necessary to use both the cubemap-based far-field
feature and the cone-traced near-field feature to get the best specular
appearance (Tab. 3).

|
---|---
| Ground truth
|
Analytical $\mathbf{H}_{f}$ | Cubemap $\mathbf{H}_{f}$
|
Volume-rendered $\mathbf{H}_{n}$ | Cone-traced $\mathbf{H}_{n}$

Figure 12:  Qualitative ablation of NDE components. Details from the
environment light fail to be reconstructed with an analytical encoding (mirror
sphere on 2nd row). It is also necessary to use the cone-traced near-field
feature, otherwise rough surfaces are rendered incorrectly (grey sphere on 3rd
row).

Far-field feature | Near-field feature | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---
Analytical | - | 28.54 | 0.944 | 0.029
Cubemap | - | _30.27_ | 0._962_ | _0.022_
Cubemap | Volume-rendered | 29.31 | 0.951 | 0.034
Cubemap | Cone-traced | 31.53 | 0.972 | 0.017

Table 3:  Ablation on directional encodings shows each component of NDE is
needed for the best rendering quality. The comparison is made on the Materials
scene.

#### Network architecture.

Table 4 shows the performance trade-off between different network
architectures of our model on synthetic scenes. Using a smaller MLP width for
the decoder of $\sigma_{n},\mathbf{h}_{n},\mathbf{c}_{s}$ has only a slight
negative impact on the rendering quality but significantly improves real-time
performance. The rendering quality reduction of the real-time model is mainly
caused by the error near object edges as discussed in Sec. 5.2.

Model | MLP width | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FPS$\uparrow$
---|---|---|---|---|---
Our offline | 64 | 37.19 | 0.982 | 0.022 |  $<$1
32 | _36.69_ | _0.979_ | _0.026_ |  $<$1
16 | 36.23 | 0.977 | 0.028 |  $<$1
Our real-time | 64 | 35.48 | 0.976 | 0.027 | 66
32 | 33.97 | 0.971 | 0.034 | _211_
16 | 33.71 | 0.969 | 0.036 | 331

Table 4: Ablation on our network architecture. Using a smaller MLP width
introduces a minor loss in rendering fidelity but a noticeable real-time
performance boost.

| Mat. | Teapot | Toaster | Car | Ball | Coffee | Helmet | Mean
---|---|---|---|---|---|---|---|---
PSNR $\uparrow$
Hash grid | 30.89 | 49.00 | 29.46 | 30.16 | 43.48 | 34.98 | 37.67 | 36.52
Tri-plane | 31.53 | 49.12 | 30.32 | 30.39 | 44.66 | 36.57 | 37.77 | 37.19
SSIM $\uparrow$
Hash grid | 0.968 | 0.999 | 0.953 | 0.967 | 0.990 | 0.974 | 0.990 | 0.977
Tri-plane | 0.972 | 0.999 | 0.968 | 0.968 | 0.995 | 0.979 | 0.990 | 0.982
LPIPS $\downarrow$
Hash grid | 0.019 | 0.002 | 0.058 | 0.025 | 0.031 | 0.043 | 0.014 | 0.027
Tri-plane | 0.017 | 0.002 | 0.039 | 0.024 | 0.022 | 0.033 | 0.014 | 0.022

Table 5: Ablation on mip-mapping strategies suggests that the mip-mapped tri-
plane represents averaged near-field features and density better than the mip-
mapped hash grid.

#### Spatial mip-mapping strategies.

Besides mip-mapped tri-plane [6, 17], our architecture can also work with a
mip-mapped hash grid [40] for the near-field feature encoding. Similar to [3,
26], the hash-grid mip-mapping is implemented by gradually masking out fine-
resolution features as the mip level increases. This results in limited model
capacity for rough surfaces where most of the features are masked out, such
that a mip-mapped hash grid produces slightly worse rendering than the tri-
plane encoding (Tab. 5).

|  |  |
---|---|---|---
ENVIDR [27] | NDE (hash grid) | NDE (MLP) | Ground truth

Figure 13:  Unstable geometry optimization of specular objects prevents us
from encoding the SDF using a hash grid [40] as it gives incorrect surface
normals (middle left). This is also the case for other hash-grid-based methods
(left).

#### Limitations.

Like previous works [49, 27, 32], NDE is sensitive to the quality of the
surface normal. This prevents us from using more efficient geometry
representations such as a hash grid, which tends to produce corrupted geometry
(Fig. 13). As a result, we use positional-encoded MLPs to model the SDF, which
leads to long training times and is difficult for modeling transparent
objects. Meanwhile, the editibility of our method is limited.

##  6 Conclusion

We have adapted feature-based NeRF encodings to the directional domain and
introduced a novel spatio-spatial parameterization of view-dependent
appearance. These improvements allow for efficient modeling of complex
reflections for novel-view synthesis and could benefit other applications that
model spatially varying directional signals, such as neural materials [24, 59,
14] and radiance caching [39].

#### Acknowledgements.

This work was supported in part by NSF grants 2110409, 2100237, 2120019, ONR
grant N00014-23-1-2526, gifts from Adobe, Google, Qualcomm, Rembrand, a Sony
Research Award, as well as the Ronald L. Graham Chair and the UC San Diego
Center for Visual Computing. Additionally, we thank Jingshen Zhu for
insightful discussions.

## References

  * Andersson et al. [2020] Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D Fairchild.  Flip: A difference evaluator for alternating images.  _Proc. ACM Comput. Graph. Interact. Tech._ , 3(2):15–1, 2020.
  * Barron et al. [2022] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _CVPR_ , 2022.
  * Barron et al. [2023] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  In _ICCV_ , 2023.
  * Belhe et al. [2024] Yash Belhe, Bing Xu, Sai Praveen Bangaru, Ravi Ramamoorthi, and Tzu-Mao Li.  Importance sampling brdf derivatives.  In _ACM TOG_ , 2024.
  * Chan et al. [2021] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In _CVPR_ , 2021.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _CVPR_ , 2022.
  * Chen et al. [2021] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _ICCV_ , 2021.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _ECCV_ , 2022.
  * Chen et al. [2023] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi.  Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures.  In _CVPR_ , 2023.
  * Crassin et al. [2011] Cyril Crassin, Fabrice Neyret, Miguel Sainz, Simon Green, and Elmar Eisemann.  Interactive indirect illumination using voxel cone tracing.  In _Computer Graphics Forum_ , 2011.
  * Flynn et al. [2019] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In _CVPR_ , 2019.
  * Fridovich-Keil et al. [2023] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa.  K-planes: Explicit radiance fields in space, time, and appearance.  In _CVPR_ , 2023.
  * Garbin et al. [2021] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In _ICCV_ , 2021.
  * Gauthier et al. [2022] Alban Gauthier, Robin Faury, Jérémy Levallois, Théo Thonat, Jean-Marc Thiery, and Tamy Boubekeur.  Mipnet: Neural normal-to-anisotropic-roughness mip mapping.  In _ACM TOG_ , 2022.
  * Gu et al. [2023] Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi.  Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion.  In _ICML_ , 2023.
  * Hedman et al. [2021] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec.  Baking neural radiance fields for real-time view synthesis.  In _ICCV_ , 2021.
  * Hu et al. [2023] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma.  Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields.  In _ICCV_ , 2023.
  * Kalantari et al. [2016] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi.  Learning-based view synthesis for light field cameras.  In _ACM TOG_ , 2016.
  * Karis [2013] Brian Karis.  Real shading in unreal engine 4.  In _SIGGRAPH 2013 Course: Physically Based Shading in Theory and Practice_ , 2013.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  In _ACM TOG_ , 2023.
  * Keselman and Hebert [2023] Leonid Keselman and Martial Hebert.  Flexible techniques for differentiable rendering with 3d gaussians.  _arXiv preprint arXiv:2308.14737_ , 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kulhanek and Sattler [2023] Jonas Kulhanek and Torsten Sattler.  Tetra-nerf: Representing neural radiance fields using tetrahedra.  In _ICCV_ , 2023.
  * Kuznetsov et al. [2021] Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Miloš Hašan, and Ravi Ramamoorthi.  Neumip: Multi-resolution neural materials.  In _ACM TOG_ , 2021.
  * Li et al. [2022] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa.  Nerfacc: A general nerf acceleration toolbox.  _arXiv preprint arXiv:2210.04847_ , 2022.
  * Li et al. [2023] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin.  Neuralangelo: High-fidelity neural surface reconstruction.  In _CVPR_ , 2023.
  * Liang et al. [2023] Ruofan Liang, Hui-Hsia Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar.  Envidr: Implicit differentiable renderer with neural environment lighting.  In _ICCV_ , 2023.
  * Lin et al. [2023a] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In _CVPR_ , 2023a.
  * Lin et al. [2023b] Kai-En Lin, Yen-Chen Lin, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.  Vision transformer for nerf-based view synthesis from a single input image.  In _WACV_ , 2023b.
  * Liu et al. [2020] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  In _NeurIPS_ , 2020.
  * Liu et al. [2023a] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al.  One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization.  In _NeurIPS_ , 2023a.
  * Liu et al. [2023b] Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang.  Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images.  In _ACM TOG_ , 2023b.
  * Lorensen and Cline [1987] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _SIGGRAPH_ , 1987.
  * Mai et al. [2023] Alexander Mai, Dor Verbin, Falko Kuester, and Sara Fridovich-Keil.  Neural microfacet fields for inverse rendering.  In _ICCV_ , 2023.
  * Martin-Brualla et al. [2021] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _CVPR_ , 2021.
  * Max [1995] Nelson Max.  Optical models for direct volume rendering.  _IEEE Transactions on Visualization and Computer Graphics_ , 1(2):99–108, 1995.
  * Mildenhall et al. [2019] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  In _ACM TOG_ , 2019.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Müller et al. [2021] Thomas Müller, Fabrice Rousselle, Jan Nov’ak, and Alexander Keller.  Real-time neural radiance caching for path tracing.  In _ACM TOG_ , 2021.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  In _SIGGRAPH_ , 2022.
  * Munkberg et al. [2022] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In _CVPR_ , 2022.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_ , 2019.
  * Poole et al. [2023] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In _ICLR_ , 2023.
  * Reiser et al. [2021] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In _ICCV_ , 2021.
  * Rosu and Behnke [2023] Radu Alexandru Rosu and Sven Behnke.  Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices.  In _CVPR_ , 2023.
  * Sun et al. [2022] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _CVPR_ , 2022.
  * Trevithick and Yang [2021] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d representation and rendering.  In _ICCV_ , 2021.
  * Trevithick et al. [2023] Alex Trevithick, Matthew Chan, Michael Stengel, Eric Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano.  Real-time radiance fields for single-image portrait view synthesis.  In _ACM TOG_ , 2023.
  * Verbin et al. [2022] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  In _CVPR_ , 2022.
  * Walter et al. [2007] Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance.  Microfacet models for refraction through rough surfaces.  In _EGSR_ , 2007.
  * Wang et al. [2021a] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In _NeurIPS_ , 2021a.
  * Wang et al. [2021b] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibr-net: Learning multi-view image-based rendering.  In _CVPR_ , 2021b.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  In _IEEE transactions on image processing_ , 2004.
  * Wu et al. [2022] Liwen Wu, Jae Yong Lee, Anand Bhattad, Yu-Xiong Wang, and David Forsyth.  Diver: Real-time and accurate neural radiance fields with deterministic integration for volume rendering.  In _CVPR_ , 2022.
  * Xu et al. [2022] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-nerf: Point-based neural radiance fields.  In _CVPR_ , 2022.
  * Yariv et al. [2021] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  In _NeuRIPS_ , 2021.
  * Yariv et al. [2023] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall.  Bakedsdf: Meshing neural sdfs for real-time view synthesis.  In _SIGGRAPH_ , 2023.
  * Yu et al. [2021] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In _ICCV_ , 2021.
  * Zeltner et al. [2023] Tizian Zeltner, Fabrice Rousselle, Andrea Weidlich, Petrik Clarberg, Jan Novák, Benedikt Bitterli, Alex Evans, Tomáš Davidovič, Simon Kallweit, and Aaron Lefohn.  Real-time neural appearance models.  _arXiv preprint arXiv:2305.02678_ , 2023.
  * Zhang et al. [2020] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_ , 2020.
  * Zhang et al. [2021] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely.  Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting.  In _CVPR_ , 2021.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.
  * Zhang et al. [2023] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova.  Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision.  In _CVPR_ , 2023.

##  Appendix A Additional Implementation Details

###  A.1 Cone tracing footprint

In Sec. 4.2, we choose the cone to cover the (cosine weighted) GGX
distribution [50] centered in the reflected direction $\bm{\omega}_{r}$.
Assuming $\bm{\omega}_{r}\\!=\\!(0,0,1)$, the distribution $D$ with roughness
$\rho$ in spherical coordinates $(\theta,\phi)$ can be written as:

$D(\theta,\phi)=\frac{\alpha^{2}\max(\cos\theta,0)}{\pi(\cos^{2}\theta(\alpha^{2}-1)+1)^{2}},\
\alpha=\rho^{2}.$ (14)

If we want the cone to cover a certain fraction $T$ of the distribution, the
polar angle $\theta$ should satisfy:

$\begin{split}T&=\int_{0}^{2\pi}\\!\int_{0}^{\theta}D(\theta^{\prime},\phi)\sin\theta^{\prime}d\theta^{\prime}d\phi\\\
&=\frac{1-\cos^{2}\theta}{1+\cos^{2}\theta(\alpha^{2}-1)}\\\
\Rightarrow&\cos\theta=\sqrt{\frac{1-T}{T(\alpha^{2}-1)+1}},\end{split}$ (15)

which gives the base cone radius $r_{0}$:

$r_{0}=\cot\theta=\frac{\sqrt{1-\cos^{2}\theta}}{\cos\theta}=\sqrt{\frac{T}{1-T}}\rho^{2}.$
(16)

We found $T\\!=\\!75\%$ in practice gives good results, which suggests
$r_{0}=\sqrt{3}\rho^{2}$. Therefore, the footprint at
$\mathbf{x}^{\prime}_{i}$ from $\mathbf{x}$ is
$r_{i}\\!=\\!\sqrt{3}\rho^{2}\|\mathbf{x}-\mathbf{x}^{\prime}_{i}\|_{2}$.

###  A.2 Real-time application

We use a two-pass deferred shading in our real-time model. The first pass
rasterizes the world-space position $\mathbf{x}$, normal $\mathbf{n}$, diffuse
color $\mathbf{c}_{d}$, specular tint $\mathbf{k}_{s}$, spatial feature
$\mathbf{f}$, and roughness $\rho$ into the G-buffer. In the second pass, we
then calculate the NDE $\mathbf{H}$, including a cubemap lookup for far-field
feature $\mathbf{H}_{f}$ and the cone tracing of near-field feature
$\mathbf{H}_{n}$, and decode it to get the specular color $\mathbf{c}_{s}$.
The MLP evaluations are executed sequentially inside the pixel shader, and we
implement the early ray termination trick [58, 16] to stop the cone tracing if
the accumulated transmittance is below 0.01. Because small decoder MLPs tend
to provide unstable geometry optimization, we use the fixed SDF network weight
from our NDE trained with 64 MLP width when training other variants that use
smaller decoder MLPs (Sec. 5.3).

###  A.3 Spatial mip-mapping strategies

We introduce mip-mapping strategies of spatial encodings in Sec. 5.3 using
either a triplane [6, 17] or a hash grid [40]. Let
$\mathbf{T}_{xy},\mathbf{T}_{yz},\mathbf{T}_{zx}$ denote the three 2D planes
of the triplane $\mathbf{T}$. A mip-mapped query at location
$\mathbf{x}\\!=\\!(x,y,z)$ of mip level $\lambda$ is given by:

$\begin{gathered}\begin{aligned}
\text{mipmap}&(\mathbf{T}(\mathbf{x}),\lambda)=\\\ &\bigoplus_{\mathbf{u}\in
U}\text{lerp}(\mathbf{T}^{\lfloor\lambda\rfloor}_{\mathbf{u}}(\mathbf{u}),\mathbf{T}^{\lceil\lambda\rceil}_{\mathbf{u}}(\mathbf{u}),\lambda\\!-\\!\lfloor\lambda\rfloor),\end{aligned}\\\
U\\!=\\!\\{(x,y),\\!(y,z),\\!(z,x)\\},\mathbf{T}^{k}_{\mathbf{u}}\\!=\\!\text{downsample}(\mathbf{T}_{\mathbf{u}},k),\end{gathered}$
(17)

where $\bigoplus$ is the concatenation operation. For a hash grid feature
$\mathbf{F}$ with $l^{\text{th}}$ level feature $\mathbf{F}_{l}$ (beginning
from the finest resolution), its mip-mapping is given by:

$\text{mipmap}(\mathbf{F}(\mathbf{x}),\lambda)=\bigoplus_{l}\text{clamp}(l+1-\lambda,0,1)\mathbf{F}_{l}(\mathbf{x}).$
(18)

| ENVIDR | Ref-NeRF | NDE (ours)
---|---|---|---
PSNR$\uparrow$ | 22.67 | _23.46_ | 23.63

Table 6:  PSNR on the Ref-NeRF Garden Spheres scene.

ENVIDR | NDE (ours) | Ground truth
---|---|---
|  |

Figure 14:  Qualitative comparison on the Garden Spheres scene of Ref-NeRF
real dataset. Numbers shows the image PSNR; zoom in to see the difference.

##  Appendix B Additional Results

For the unbounded real scene evaluation, we provide the results on the Garden
Spheres scene of Ref-NeRF real dataset [49] in Tab. 6 and Fig. 14. It can be
seen that our method is able to recover more interreflection details in real-
world compared to other baselines. Considering perceptual measures are more
reasonable for reflection quality comparison, we additionally show the FLIP
[1] metric on synthetic scenes in Tab. 7. Overall, our method still
demonstrates higher rendering quality compared to other baselines.

Method | Mat. | Teapot | Toaster | Car | Ball | Coffee | Helmet | Mean
---|---|---|---|---|---|---|---|---
NeRO | 0.082 | 0.012 | _0.097_ | _0.049_ | 0.058 | 0.039 | 0.083 | 0.060
ENVIDR | 0.062 | 0.030 | 0.098 | 0.056 | _0.037_ | 0.046 | _0.049_ | 0.054
Ref-NeRF | 0.023 | _0.011_ | 0.108 | 0.071 | 0.038 | 0.030 | 0.072 | _0.050_
3DGS | 0.042 | 0.007 | 0.153 | 0.051 | 0.104 | _0.032_ | 0.073 | 0.066
NDE | _0.039_ | 0.007 | 0.065 | 0.038 | 0.027 | 0.035 | 0.035 | 0.035

Table 7:  FLIP metric on synthetic scenes.

| NeRO | ENVIDR | Ref-NeRF | NDE | NDE-RT
---|---|---|---|---|---
PSNR $\uparrow$ | 28.75 | 31.29 | 28.18 | 34.08 | _32.97_
SSIM $\uparrow$ | 0.956 | 0.969 | 0.946 | 0.985 | _0.984_
LPIPS $\downarrow$ | 0.046 | 0.022 | 0.030 | 0.008 | _0.010_

Table 8: Quantitative results on the teaser scene.

| Mat. | Teapot | Toaster | Car | Ball | Coffee | Helmet | Mean
---|---|---|---|---|---|---|---|---
PSNR $\uparrow$
3DGS | 29.98 | 45.69 | 20.99 | 27.25 | 27.65 | 32.31 | 28.26 | 30.30
NDE-RT | 30.28 | 47.02 | 28.31 | 28.91 | 43.23 | 34.21 | 36.38 | 35.48
SSIM $\uparrow$
3DGS | 0.960 | 0.997 | 0.895 | 0.930 | 0.937 | 0.972 | 0.951 | 0.949
NDE-RT | 0.967 | 0.998 | 0.954 | 0.962 | 0.994 | 0.972 | 0.987 | 0.976
LPIPS $\downarrow$
3DGS | 0.034 | 0.007 | 0.126 | 0.048 | 0.162 | 0.078 | 0.080 | 0.076
NDE-RT | 0.020 | 0.003 | 0.051 | 0.030 | 0.023 | 0.041 | 0.019 | 0.027

Table 9:  Per-scene comparison with 3DGS on synthetic scenes.

Method | Mat. | Teapot | Toaster | Car | Ball | Coffee | Helmet | Mean
---|---|---|---|---|---|---|---|---
Hashgrid | 30.12 | 46.46 | 25.83 | 29.94 | 36.41 | 33.25 | 34.08 | 33.73
Pos. enc. | 31.53 | 49.12 | 30.32 | 30.39 | 44.66 | 36.57 | 37.77 | 37.19

Table 10:  Comparison of geometry encoding on synthetic scenes in PSNR. “Pos.
enc.” denotes positional encoding.

##  Appendix C Experiment Details

We provide the quantitative results on the teaser scene (Fig. 1 of the main
paper) compared to the baselines in Tab. 8 and the per-scene comparison of our
real-time model (NDE-RT) with 3DGS [20] in Tab. 9. Table 10 shows the
comparison of different SDF encodings (Fig. 12 of the main paper). Table 11
and 12 show the per-scene quantitative results of our real-time and offline
model with different MLP width (Width) on the synthetic dataset. In Fig. 15,
we show the per-scene rendering results of both our offline (NDE) and real-
time (NDE-RT) model on the synthetic dataset together with the reconstructed
surface normals. The normals are masked by the foreground mask to get rid of
floaters with the background color.

Width | Mat. | Teapot | Toaster | Car | Ball | Coffee | Helmet | Mean
---|---|---|---|---|---|---|---|---
PSNR $\uparrow$
64 | 31.53 | 49.12 | 30.32 | 30.39 | 44.66 | 36.57 | 37.77 | 37.19
32 | 30.89 | 48.88 | 29.33 | 29.51 | 44.34 | 36.24 | 37.63 | 36.69
16 | 30.59 | 48.56 | 29.09 | 29.24 | 43.61 | 36.07 | 36.47 | 36.23
SSIM $\uparrow$
64 | 0.972 | 0.999 | 0.968 | 0.968 | 0.995 | 0.979 | 0.990 | 0.982
32 | 0.968 | 0.999 | 0.961 | 0.962 | 0.994 | 0.977 | 0.989 | 0.979
16 | 0.965 | 0.998 | 0.959 | 0.960 | 0.994 | 0.977 | 0.986 | 0.977
LPIPS $\downarrow$
64 | 0.017 | 0.002 | 0.039 | 0.024 | 0.022 | 0.033 | 0.014 | 0.022
32 | 0.021 | 0.002 | 0.057 | 0.032 | 0.021 | 0.033 | 0.017 | 0.026
16 | 0.023 | 0.002 | 0.058 | 0.034 | 0.021 | 0.034 | 0.022 | 0.028

Table 11: Per-scene results of our offline models on synthetic scenes. The
first column suggests the decoder MLP width.

Width | Mat. | Teapot | Toaster | Car | Ball | Coffee | Helmet | Mean
---|---|---|---|---|---|---|---|---
PSNR $\uparrow$
64 | 30.28 | 47.02 | 28.31 | 28.91 | 43.23 | 34.21 | 36.38 | 35.48
32 | 28.84 | 43.75 | 27.28 | 27.76 | 41.00 | 33.92 | 35.26 | 33.97
16 | 28.33 | 43.14 | 26.95 | 27.47 | 41.48 | 33.81 | 34.81 | 33.71
SSIM $\uparrow$
64 | 0.967 | 0.998 | 0.954 | 0.962 | 0.994 | 0.972 | 0.987 | 0.976
32 | 0.955 | 0.996 | 0.945 | 0.954 | 0.992 | 0.971 | 0.984 | 0.971
16 | 0.951 | 0.996 | 0.942 | 0.951 | 0.992 | 0.971 | 0.982 | 0.969
LPIPS $\downarrow$
64 | 0.020 | 0.003 | 0.051 | 0.030 | 0.023 | 0.041 | 0.019 | 0.027
32 | 0.030 | 0.006 | 0.068 | 0.039 | 0.025 | 0.043 | 0.026 | 0.034
16 | 0.033 | 0.007 | 0.070 | 0.043 | 0.024 | 0.044 | 0.030 | 0.036
FPS $\uparrow$
64 | 55 | 130 | 55 | 70 | 65 | 30 | 58 | 66
32 | 220 | 400 | 150 | 200 | 210 | 110 | 190 | 211
16 | 350 | 600 | 250 | 330 | 300 | 200 | 290 | 331

Table 12: Per-scene results of our real-time models on synthetic scenes. The
first column suggests the deocder MLP width.

NDE | NDE-RT | Ground truth | Estimated normal | Ground truth normal
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |

Figure 15: Qualitative results on each synthetic scene for our offline (NDE)
and real-time (NDE-RT) methods.