Pulkit<EMAIL_ADDRESS>Mohammad Reza Karimi
<EMAIL_ADDRESS>Charles
<EMAIL_ADDRESS>P. J<EMAIL_ADDRESS>Jean-François
<EMAIL_ADDRESS>Centre for Visual Information Technology &
Kohli Centre on Intelligent Systems (KCIS)
International Institute of Information Technology (IIIT),
Hyderabad, India Université Laval
Quebec City, Canada PanoHDR-NeRF

# Casual Indoor HDR Radiance Capture from Omnidirectional Images

###### Abstract

We present PanoHDR-NeRF, a neural representation of the full HDR radiance
field of an indoor scene, and a pipeline to capture it casually, without
elaborate setups or complex capture protocols. First, a user captures a low
dynamic range (LDR) omnidirectional video of the scene by freely waving an
off-the-shelf camera around the scene. Then, an LDR2HDR network uplifts the
captured LDR frames to HDR, which are used to train a tailored NeRF++ model.
The resulting PanoHDR-NeRF can render full HDR images from any location of the
scene. Through experiments on a novel test dataset of real scenes with the
ground truth HDR radiance captured at locations not seen during training, we
show that PanoHDR-NeRF predicts plausible HDR radiance from any scene point.
We also show that the predicted radiance can synthesize correct lighting
effects, enabling the augmentation of indoor scenes with synthetic objects
that are lit correctly. Datasets and code are available at
https://lvsn.github.io/PanoHDR-NeRF/.

##  1 Introduction

Figure 1: We capture the continuous HDR radiance of an indoor scene. Our
PanoHDR-NeRF approach takes a) casually captured LDR images from an off-the-
shelf camera (shown in inset) as input, and performs b) novel view synthesis
of the indoor scene. c) As opposed to existing techniques such as NeRF++
[Zhang et al.(2020)Zhang, Riegler, Snavely, and Koltun] (left), PanoHDR-NeRF
(right) properly estimates the HDR radiance of the scene, visualized by
relighting virtual test objects.

Capturing the incoming radiance of a given scene is an important step for many
augmented reality applications, since it allows immersive exploration and
realistic scene augmentation. To accurately measure the radiance, Debevec
[Debevec(1998)] pioneered _image-based lighting_ which involves photographing
a chrome sphere—called a light probe—at different exposures and merging them
into a single, high dynamic range (HDR) image [Debevec and Malik(1997)]. HDR
light probing was later extended to use wide angle lenses or $360^{\circ}$
cameras and is at the heart of the lighting capture necessary to achieve
special effects in movies today111https://www.fxguide.com/fxfeatured/the-
definitive-weta-digital-guide-to-ibl/. Because of the close proximity between
light sources and objects in the scene, the HDR radiance field of a typical
indoor scene varies rapidly: lighting near a window is vastly different from
the center of the room. Accurately capturing indoor radiance involves moving
the apparatus and repeating the operation many times, limiting scalability.
Approximations such as reprojecting the measured radiance onto a proxy 3D
model [Debevec et al.(1998)Debevec, Yu, and Boshokov] must be used. To
simplify the capture process, one could use inverse tonemapping techniques
[Rempel et al.(2007)Rempel, Trentacoste, Seetzen, Young, Heidrich, Whitehead,
and Ward, Endo et al.(2017)Endo, Kanamori, and Mitani, Tarko et
al.(2019)Tarko, Tompkin, and Richardt] which estimate the true dynamic range
information from low dynamic range (LDR) inputs. This yields radiance
estimates only where images are taken. Can we capture and learn a
representation of the scene from which HDR radiance can be synthesized at any
point?

Novel view synthesis of an object or scene from multiple views has garnered
attention recently, especially since the emergence of neural radiance fields
(NeRF) [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron,
Ramamoorthi, and Ng]. In this approach, a set of images are captured, co-
registered using structure-from-motion (SfM), and used to train a deep network
that learns to predict the color and opacity along any 3D ray using volumetric
rendering. Recent flavors of NeRF model large scenes [Zhang et al.(2020)Zhang,
Riegler, Snavely, and Koltun, Barron et al.(2022)Barron, Mildenhall, Verbin,
Srinivasan, and Hedman], handle omnidirectional input images [Gu et
al.(2022)Gu, Maugey, Knorr, and Guillemot], and decompose the scene into
intrinsic components [Boss et al.(2021)Boss, Braun, Jampani, Barron, Liu, and
Lensch, Zhang et al.(2021)Zhang, Srinivasan, Deng, Debevec, Freeman, and
Barron]. Most NeRF-based methods accept LDR images as input and do not
accurately model the true HDR radiance of indoor environments. There are
recent attempts to address this limitation [Huang et al.(2022)Huang, Zhang,
Ying, Li, Wang, and Wang, Mildenhall et al.(2022)Mildenhall, Hedman, Martin-
Brualla, Srinivasan, and Barron], which require multiple LDR images at
different exposures to model the full dynamic range of indoor lighting.
Capturing 11 exposures at a point of the scene, which is necessary to
reconstruct a reasonable HDR panorama spanning over 22 f-stops, takes
approximately two minutes with a conventional $360^{\circ}$ camera (Ricoh
Theta Z1). Thus to capture HDR lighting for a large indoor scene, we need over
200 images bringing the capture time to over 3.5 hours. In addition, an
elaborate setup would be required to avoid ghosting artifacts. These methods
to capture the full dynamic range of lighting for large indoor scenes require
specialized setup and are tedious and cumbersome.

In this paper, we present PanoHDR-NeRF, a neural representation of the
plausible full HDR radiance field of an indoor scene. We also present a method
to generate it from casually captured images of the scene. HDR radiance from
any novel viewpoint in the scene can subsequently be estimated (fig. 1) from
PanoHDR-NeRF. Our method does not require any special equipment or complicated
capture protocols. It accepts as input a video sequence captured by freely
moving a commercial $360^{\circ}$ camera around the scene. As output, it can
produce the HDR radiance at any given location in the scene. To do so, we
leverage two deep neural networks: 1) an LDR2HDR model that predicts the HDR
radiance from a single LDR panorama captured by the camera and 2) a modified
NeRF++ model trained on the predicted HDR outputs of the first network. To
evaluate our proposed method, we capture a set of six different indoor scenes,
which we augment with a set of ground truth HDR light probes at each scene.
Our experiments demonstrate that, despite the simplicity of the capture
procedure, PanoHDR-NeRF can accurately predict HDR radiance in a variety of
challenging conditions. Our approach can render $360^{\circ}$ spatially
varying HDR light probes, which can be used to provide correct lighting
effects when the scene is augmented with virtual objects. Compared to previous
methods, our approach reduces the complexity and time required to capture the
full HDR radiance field of the scene.

##  2 Related work

Inverse tonemapping Inverse tonemapping aims to recover missing information in
the over- and under-saturated areas of an LDR image. While earlier methods
[Rempel et al.(2007)Rempel, Trentacoste, Seetzen, Young, Heidrich, Whitehead,
and Ward] relied on heuristics, several deep learning architectures have been
proposed recently [Endo et al.(2017)Endo, Kanamori, and Mitani, Zhang and
Lalonde(2017)]. These include a 2D encoder and 3D decoder with skip
connections to generate bracketed LDR image stacks (over exposed and under
exposed) [Endo et al.(2017)Endo, Kanamori, and Mitani], an encoder-decoder to
reconstruct the HDR image directly [Li and Fang(2019)], and a multiscale
autoencoder architecture to learn multilevel features from LDR image which are
merged to reconstruct HDR images [Marnerides et al.(2018)Marnerides, Bashford-
Rogers, Hatchett, and Debattista]. Lee _et al_ [Lee et al.(2018)Lee, An, and
Kang] generate LDR stacks with a two-branch network. DeepHDR [Santos et
al.(2020)Santos, Tsang, and Kalantari] mask out the saturated areas to reduce
sub-optimal features from well-exposed and saturated pixels. HallucinationNet
[Liu et al.(2020)Liu, Lai, Chen, Kao, Yang, Chuang, and Huang] individually
models the main components of the imaging pipeline: dynamic range clipping,
camera response function, and quantization. LANet [Yu et al.(2021b)Yu, Liu,
Long, Dong, Zou, and Xiao] introduces a multi-task network with a luminance
attention and HDR reconstruction streams. We build on the LANet architecture
and augment it with an additional rendering loss. This combination outperforms
alternatives at predicting the high dynamic range radiance in indoor
environments.

Novel view synthesis Classical methods reconstruct an explicit 3D model of the
scene [Debevec et al.(1998)Debevec, Yu, and Boshokov, Gortler et
al.(1996)Gortler, Grzeszczuk, Szeliski, and Cohen, Hedman et al.(2016)Hedman,
Ritschel, Drettakis, and Brostow]. Recent works utilize off-the-shelf SfM
techniques to generate a coarse geometry and use neural networks to render
photorealistic novel views [Hedman et al.(2018)Hedman, Philip, Price, Frahm,
Drettakis, and Brostow, Meshry et al.(2019)Meshry, Goldman, Khamis, Hoppe,
Pandey, Snavely, and Martin-Brualla, Thies et al.(2019)Thies, Zollhöfer, and
Nießner]. Breaking the scene into Multi-Plane Image (MPI) representation to
render novel views by blending has been tried [Flynn et al.(2019)Flynn,
Broxton, Debevec, DuVall, Fyffe, Overbeck, Snavely, and Tucker, Zhou et
al.(2018)Zhou, Tucker, Flynn, Fyffe, and Snavely, Mildenhall et
al.(2019)Mildenhall, Srinivasan, Cayon, Kalantari, Ramamoorthi, Ng, and Kar].
DeepVoxels [Sitzmann et al.(2019)Sitzmann, Thies, Heide, Nießner, Wetzstein,
and Zollhöfer] learn a voxel-based volumetric representation of the scene
using Gated Recurrent Units (GRUs) [Cho et al.()Cho, van Merrienboer,
Gülçehre, Bahdanau, Bougares, Schwenk, and Bengio]. Free View Synthesis and
Stable View Synthesis [Riegler and Koltun(2020), Riegler and Koltun(2021)] map
the encoded features from the source images into the target view and blend
them via a neural network directly or in a geometric space. Novel view
synthesis from omnidirectional images are fairly limited so far. Huang _et al_
[Huang et al.(2017)Huang, Chen, Ceylan, and Jin] reconstruct a point cloud
from $360^{\circ}$ videos to achieve real time video playback on a VR device.
Serrano _et al_ [Serrano et al.(2019)Serrano, Kim, Chen, DiVerdi, Gutierrez,
Hertzmann, and Masia] design a layered scene representation that facilitates
parallax and real time playback of $360^{\circ}$ video. [Lin et al.(2020)Lin,
Xu, Mildenhall, Srinivasan, Hold-Geoffroy, DiVerdi, Sun, Sunkavalli, and
Ramamoorthi, Attal et al.(2020)Attal, Ling, Gokaslan, Richardt, and Tompkin]
present Multi-Depth Panorama (MDP) and multi-sphere images to create 6-DoF
renderings. This requires an elaborate setup and does not accommodate free
viewpoint synthesis. Bertel _et al_ [Bertel et al.(2020)Bertel, Yuan,
Lindroos, and Richardt] propose a fast, casual and robust capture of immersive
real-world VR experience. However it takes a lot of memory and poor proxy
geometry causes warping artifacts. Xu _et al_ [Xu et al.(2021)Xu, Zheng, Xu,
Tang, and Gao] estimate the entire indoor scene from a single image using a
CNN but need room layout priors and depth that are challenging to obtain for
real scenes.

Neural radiance fields Neural Radiance Fields (NeRF) [Mildenhall et
al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] learn an
implicit volumetric scene representation with a Multi-Layer Perceptron (MLP)
that receives viewing positions and directions as input and predicts the RGB
colour and opacity as output. The original formulation was later extended in
several ways relevant to our work. NeRF++ [Zhang et al.(2020)Zhang, Riegler,
Snavely, and Koltun], which we leverage, models an unbounded scene by
splitting it into foreground/background, learning each separately. Mega-NeRF
[Turki et al.(2022)Turki, Ramanan, and Satyanarayanan] divides the scene into
smaller sections. Block-NeRF [Tancik et al.(2022)Tancik, Casser, Yan, Pradhan,
Mildenhall, Srinivasan, Barron, and Kretzschmar] splits city-scale scenes into
blocks, trains a NeRF for each block, and combines the results. Mip-NeRF
[Barron et al.(2021)Barron, Mildenhall, Tancik, Hedman, Martin-Brualla, and
Srinivasan, Barron et al.(2022)Barron, Mildenhall, Verbin, Srinivasan, and
Hedman] replaces rays with anti-aliased conical frustums for speed and
accuracy. NeRF with HDR images has been explored in two works. NeRF in the
dark [Mildenhall et al.(2022)Mildenhall, Hedman, Martin-Brualla, Srinivasan,
and Barron] train directly on linear RAW images with higher effective dynamic
range (14-16 bits compared to more typical 8-bit cameras). Huang _et al_
[Huang et al.(2022)Huang, Zhang, Ying, Li, Wang, and Wang] train from a set of
LDR images captured at alternating exposures. Both methods require multiple
LDR images at different exposures, which is time-consuming to capture and
requires specialized setup. In contrast, our method plausibly predicts the
full dynamic range from a single casually captured video. Finally, panoramic
images have been explored in [Hsu et al.(2021)Hsu, Sun, and Chen, Hara and
Harada(2022)], who present a method to synthesize panoramas from a single
omnidirectional input. They require depth as input and novel views can only be
rendered on a straight line path. CylindricalNeRF [Khademi and Ventura(2021)]
proposes cylindrical sampling for unbounded scenes captured in circular
trajectory. OmniNeRF [Gu et al.(2022)Gu, Maugey, Knorr, and Guillemot]
synthesizes novel fish-eye projection images, using spherical sampling to
improve the quality of results as we do.

Figure 2: Overview of our pipeline. At training time (left), the captured
panoramas are linearized (calibrated using a color checker, not shown) and
processed by a pre-trained LDR2HDR network to obtain HDR estimates. The HDR
panoramas, along with the camera poses obtained with OpenSfM
[Mapillary(2022)], are used to train the PanoHDR-NeRF network, which learns to
synthesize HDR scene radiance at any point in the scene. At inference time
(right), we simply provide the novel camera pose and obtain the corresponding
novel HDR panorama.

HDR scene capture Zhang _et al_ [Zhang et al.(2016)Zhang, Cohen, and Curless]
leverage an RGBD scan of an indoor scene and estimate scene parameters
including HDR radiance over scene geometry. Walton _et al_ [Walton and
Steed(2018)] combine a depth sensor with a fisheye camera in a SLAM-based
approach to recover geometry and lighting. Tarko _et al_ [Tarko et
al.(2019)Tarko, Tompkin, and Richardt] takes an omnidirectional video as input
and uses inverse tone mapping [Endo et al.(2017)Endo, Kanamori, and Mitani] to
convert it to HDR. Yang _et al_ [Yang et al.(2022)Yang, Zhang, Li, Cui,
Fanello, Bao, and Zhang] learn background and objects as separate NeRF models
and combine them with a captured panoramic image into a single scene. Our
method synthesizes panoramas from novel viewpoints with full dynamic range.

##  3 Method

Given a set of LDR panoramas $\\{I_{k}\\}_{k=1}^{N}$ captured freely using an
off-the-shelf 360∘ camera and associated camera poses
$\\{\mathbf{P}_{k}\\}_{k=1}^{N}$ obtained with SfM, our objective is to
predict the HDR radiance of the scene at any novel viewpoint. We achieve this
by recovering HDR values from LDR frames using learning-based inverse tone
mapping and then using them as the supervision for novel view synthesis. An
overview of our method is given in fig. 2.

###  3.1 High dynamic range with LDR2HDR network

Extrapolating HDR from LDR inputs is typically framed as recovering values in
the over- and under-exposed regions. Our work focuses on recovering the over-
exposed regions exclusively, with the goal of predicting accurate HDR radiance
values (specially the intensities of light sources) for realistic virtual
object insertion.

Network architecture We borrow the network architecture proposed in Luminance
Attentive Networks (LANet) [Yu et al.(2021b)Yu, Liu, Long, Dong, Zou, and
Xiao], which is designed as a multi-task network with two _luminance
attention_ and _HDR reconstruction_ streams. The former attempts to create a
spatially-weighted attention map of the over-exposed regions in the input
image, while the latter uses the attention map to estimate the HDR images.
Note that we do not use their proposed adaptation for panoramas since it did
not improve the performance in our case. So, we train the model on
equirectangular images directly.

Loss functions For the LDR2HDR module, we use the same loss function
$\ell_{\mathrm{lanet}}$ as [Yu et al.(2021b)Yu, Liu, Long, Dong, Zou, and
Xiao], combining scale invariant and luminance segmentation losses. The HDR
panoramas are radiance values used to light the scene. To match rendering
quality, we use a rendering loss

$\ell_{\mathrm{rend}}=||\mathbf{T}\ \mathbf{y}_{\mathrm{HDR}}-\mathbf{T}\
\mathbf{t}_{\mathrm{HDR}}||_{2}\,,$ (1)

where $\mathbf{T}$ is a pre-computed transport [Sloan et al.(2002)Sloan,
Kautz, and Snyder] matrix for a Lambertian scene (single bounce),
$\mathbf{y}_{\mathrm{HDR}}$ is the predicted HDR panorama, and
$\mathbf{t}_{\mathrm{HDR}}$ is the ground truth. A top-down view of a “spiky
sphere“ on a plane is used for $\mathbf{T}$. The rendered images guide the
network about the color and direction of HDR lighting in the scene. The final
loss for training the LDR2HDR network is an equally weighted combination
$\ell_{\mathrm{hdr}}=\ell_{\mathrm{lanet}}+\ell_{\mathrm{rend}}$.

Datasets We pretrain the LDR2HDR network on the Laval Indoor HDR Dataset
[Gardner et al.(2017)Gardner, Sunkavalli, Yumer, Shen, Gambaretto, Gagné, and
Lalonde], which consists of 2,400 HDR panoramas captured in a variety of
indoor settings, with a train, validation and test split as 80:10:10. Since
the sensor used to capture it (Canon 5D Mark iii camera) and the sensor we use
to capture indoor scenes (Ricoh Theta Z1) are different, a domain gap was
observed. To alleviate this, we finetune the LDR2HDR network on a small
dataset of 78 HDR panoramas captured at different locations, different from
the test scenes, using the target sensor (Theta Z1).

###  3.2 Continuous HDR radiance with PanoHDR-NeRF

Network architecture We take inspiration from and combine several recent works
on NeRF. First, we employ the NeRF++ architecture as the base. Second, similar
to [Barron et al.(2022)Barron, Mildenhall, Verbin, Srinivasan, and Hedman], we
incorporate the anti-aliased conical frustums from Mip-NeRF [Barron et
al.(2021)Barron, Mildenhall, Tancik, Hedman, Martin-Brualla, and Srinivasan].
We train PanoHDR-NeRF by sampling rays in spherical coordinates instead of
pixel coordinates (more details in supplementary).

Cafeteria | Chess room | Dark class | Spotlights
---|---|---|---
|  |  |
Figure 3: Representative images from each test scene used in the experiments.

Loss functions We train PanoHDR-NeRF using supervision from LDR2HDR network.
Traditional volume rendering methods are used to predict the radiance and
densities of points sampled on the ray as in [Zhang et al.(2020)Zhang,
Riegler, Snavely, and Koltun, Barron et al.(2021)Barron, Mildenhall, Tancik,
Hedman, Martin-Brualla, and Srinivasan]. We define NeRF loss
$\ell_{\mathrm{nerf}}$ between predicted radiance $\hat{e}$ and ground truth
HDR $e$ as

$\ell_{\mathrm{nerf}}=\sum_{r\in\mathcal{R}(\textbf{P})}\|\hat{E}(\textbf{r})-E(\textbf{r})\|^{2},$
(2)

where $E=log(1+e)$ and $\mathcal{R}(\textbf{P})$ is the set of camera rays at
pose P. The photographer, who inevitably is in the images, is segmented using
an off-the-shelf segmentation algorithm [He et al.(2017)He, Gkioxari, Dollár,
and Girshick], and the corresponding pixels are ignored in the loss.

Datasets To obtain training data for a given indoor scene, we casually capture
a set of LDR panoramas using a commercial $360^{\circ}$ camera (Ricoh Theta
Z1). We attach the camera to a portable tripod and capture a $360^{\circ}$
video while waving the camera around the scene to cover the entire volume as
much as possible for a few minutes (typically 3–5 minutes for the scenes used
in the experiments). Approximately 200 frames are then extracted from the
video at even intervals. The camera parameters of the input LDR panoramas are
recovered using OpenSFM[Mapillary(2022)] and given as input to PanoHDR-NeRF.

Dataset | Input LDR | LDR2HDR pre-trained | LDR2HDR finetuned
---|---|---|---
PU-PSNR$\uparrow$ | RMSE$\downarrow$ | HDR-VDP3$\uparrow$ | PU-PSNR$\uparrow$ | RMSE$\downarrow$ | HDR-VDP3$\uparrow$ | PU-PSNR$\uparrow$ | RMSE$\downarrow$ | HDR-VDP3$\uparrow$
Chess room | 31.659 | 0.051 | 8.067 | 33.994 | 0.048 | 8.234 | 36.995 | 0.005 | 8.492
Stairway | 31.964 | 0.224 | 7.881 | 33.297 | 0.213 | 8.016 | 33.685 | 0.019 | 8.489
Cafeteria | 25.299 | 5.378 | 6.098 | 26.664 | 5.268 | 6.418 | 28.499 | 4.061 | 7.164
Spotlights | 23.939 | 3.489 | 6.001 | 25.118 | 3.438 | 6.097 | 28.966 | 0.877 | 7.667
Dark class | 30.657 | 0.364 | 7.429 | 32.125 | 0.340 | 7.592 | 32.594 | 0.262 | 8.135
Small class | 32.353 | 2.162 | 8.018 | 34.095 | 1.889 | 8.267 | 35.465 | 0.221 | 8.678
Overall | 28.399 | 1.703 | 7.243 | 30.913 | 1.634 | 7.406 | 33.651 | 0.696 | 8.209
Table 1: Quantitative comparison of different strategies for recovering radiance across captured scenes. “Input LDR” are on the images captured by the camera, “LDR2HDR pre-trained” is on our network pre-trained on the Laval Indoor Dataset, and “LDR2HDR finetuned” is after the network finetuned to test camera. As expected, finetuning helps bridge the domain gap and significantly improves the results. Results are averaged over different viewpoints. | Input LDR | LDR2HDR pre-trained | LDR2HDR finetuned | HDR ground truth
---|---|---|---|---
Spotlights |  |  |  |
Stairway |  |  |  |
Small class |  |  |  |
Figure 4: Qualitative comparison of different strategies for recovering
radiance across captured scenes. For each example, the figure shows virtual
test objects relit to demonstrate the dynamic range. Note that despite some
color imbalance (e.g. “Spotlights”), fine-tuning helps bridge the domain gap
between the training data and the captured images. Images tonemapped for
display with $\gamma=2.2$.

##  4 Evaluation

In this section, we evaluate our approach against a set of challenging real
scenes, where the ground truth HDR radiance is also captured at several
locations. We further establish the sensitivity of design choices and compare
them to closely related techniques.

###  4.1 Radiance capture evaluation dataset

For evaluation, we capture different real indoor scenes from a variety of
different environments (fig. 3). For each scene, we first capture a
$360^{\circ}$ video using the Ricoh Theta Z1 panoramic camera as described in
sec. 3.2. We also capture a set of HDR panoramas for evaluation to use as
ground truth. To obtain these, we set the camera on a tripod at certain
locations in the scene (between 3–10 locations per scene) and program it using
exposure bracketing to capture 11 exposures spanning over 22 f-stops that are
subsequently merged to HDR using the PTGui Pro commercial software. We also
capture a short video with the same camera parameters as the captured video at
the same location as the HDR. We then extract a single frame from that video,
allowing us to have an LDR image at the exact same location as its HDR
counterpart. The resulting LDR image is linearized using a pre-calibrated
camera response function obtained with a Macbeth color checker. In total, we
capture six different scenes containing a total number of 10 LDR/HDR ground
truth panorama pairs.

###  4.2 LDR2HDR evaluation

The LDR2HDR network presented in sec. 3.1 is evaluated on the test set
described in sec. 4.1. Tab. 1 compares the performance obtained by: using the
LDR images directly, the LDR2HDR network pre-trained on the Laval Indoor HDR
Dataset [Gardner et al.(2017)Gardner, Sunkavalli, Yumer, Shen, Gambaretto,
Gagné, and Lalonde], and after fine-tuning on the 78 HDR dataset captured with
the same test camera. For evaluation, we use the “PU-PSNR” [Mantiuk and
Azimi(2021)], which is a perceptually-uniform PSNR adjusted for HDR images. In
addition, the “RMSE” corresponds to the rendering loss in eq. 1. Finally, we
also report the HDR-VDP3 [Narwaria et al.(2015)Narwaria, Mantiuk, Silva, and
Callet], where a value of 10 indicates a perfect match. Here, color encoding
is set as “rgb-bt.709” for HDR evaluation, assuming a 24-inch display,
$1920\times 1080$ resolution, and a viewing distance of 1 meter.

As shown in tab. 1 and illustrated qualitatively in fig. 4, there exists a
significant domain gap between the training dataset and the test camera:
simply pre-training on [Gardner et al.(2017)Gardner, Sunkavalli, Yumer, Shen,
Gambaretto, Gagné, and Lalonde] works marginally better than the input LDR
image itself, but finetuning results in a significant performance gain on all
metrics. Visually, finetuning produces renderings that look very similar to
the ground truth (fig. 4).

###  4.3 PanoHDR-NeRF evaluation

To evaluate how well PanoHDR-NeRF works in terms of capturing the high dynamic
range radiance field, we use the same set of ground truth HDR images as
described in sec. 4.1. We infer environment maps at the locations of HDR
panoramas and use them to render a synthetic scene. We modified NeRF++ to work
with equirectangular image representation and trained directly on the LDR
frames of the input video. The rendering results of NeRF++ appear dark
compared to ground truth, and the lighting is not realistic (fig. 5). In
addition, the generated shadows are soft and faded. In contrast, PanoHDR-NeRF
produces well-lit results, with sharp shadows that are similar to ground
truth.

###  4.4 Ablation study

| w/o render loss | render loss
---|---|---
|  PU-PSNR$\uparrow$ |  HDR-VDP3$\uparrow$ |  PU-PSNR$\uparrow$ |  HDR-VDP3$\uparrow$
Hall. Net [Liu et al.(2020)Liu, Lai, Chen, Kao, Yang, Chuang, and Huang] | 30.06 | 7.54 | 31.51 | 8.12
LANet [Yu et al.(2021b)Yu, Liu, Long, Dong, Zou, and Xiao] | 32.43 | 8.26 | 38.20 | 9.67
Table 2: Quantitative comparison of two single image HDR estimation architectures on the Laval Indoor HDR test set, with and withough the rendering loss $\ell_{\mathrm{render}}$ while training the network. Render Loss with LANet significantly improves the results. Dataset | Linear loss | PanoHDR-NeRF | NeRF-LDR2HDR
---|---|---|---
PU-PSNR$\uparrow$ | RMSE$\downarrow$ | PU-PSNR$\uparrow$ | RMSE$\downarrow$ | PU-PSNR$\uparrow$ | RMSE$\downarrow$
Chess room | 35.152 | 0.011 | 36.941 | 0.012 | 35.991 | 0.006
Stairway | 31.810 | 0.055 | 33.169 | 0.056 | 32.707 | 0.055
Cafeteria | 24.139 | 4.376 | 28.029 | 4.179 | 26.537 | 5.298
Spotlights | 26.719 | 0.909 | 28.657 | 0.431 | 27.324 | 1.619
Dark class | 28.431 | 1.367 | 30.687 | 1.509 | 29.819 | 0.621
Small class | 36.829 | 0.043 | 37.687 | 0.054 | 38.529 | 0.006
Overall | 29.725 | 1.071 | 32.528 | 1.038 | 31.650 | 1.301
Table 3: Quantitative comparison between linear (left) and log (middle) losses used for training. Comparison of PanoHDR-NeRF with NeRF then LDR2HDR is at right  Input LDR | NeRF++ | NeRF-LDR2HDR | PanoHDR-NeRF (ours) | GT
---|---|---|---|---
|  |  |  |
|  | (a) Spotlights |  |
|  |  |  |
|  | (b) Dark Class |  |
|  |  |  |
|  | (c) Small Class |  |
|  |  |  |
|  | (d) Stairway |  |
|  |  |  |
|  | (d) Chess Room |  |
|  |  |  |
|  | (e) Cafeteria |  |
Figure 5: Comparing input LDR, NeRF++, NeRF-LDR2HDR, PanoHDR-NeRF (ours), and GT panoramas. For each example, the figure shows a virtual test object relit to show the dynamic range. While NeRF++ is able to model the scene correctly, it is unable to capture the radiance of the scene. PanoHDR-NeRF is able to faithfully capture the radiance of the scene. We compare it with NeRF-LDR2HDR which estimates HDR from NeRF++ outputs. Although it is able to closely estimate the radiance, it leads to flickering between consecutive frames. Images tonemapped for display with $\gamma=2.2$. | Linear space | Log space | HDR ground truth
---|---|---|---
Spotlights |  |  |
Stairway |  |  |
Chess room |  |  |
Figure 6: Comparing panoramas generated by PanoHDR-NeRF after loss in linear
space and log space. For each example, the figure shows (top) the panorama
with (bottom) virtual test objects relit to show the dynamic range. While the
network learns the high dynamic range in both cases, we observe that taking
loss in linear space leads to poor visual quality and floating artifacts in
the output panoramas. PanoHDR-NeRF produces better results when trained in log
space, consistent with [Yu et al.(2021b)Yu, Liu, Long, Dong, Zou, and Xiao,
Santos et al.(2020)Santos, Tsang, and Kalantari, Li and Fang(2019)].

LDR2HDR network architecture We compare between two recent SOTA single image
HDR estimation architectures, namely HallucinationNet [Liu et al.(2020)Liu,
Lai, Chen, Kao, Yang, Chuang, and Huang] and LANet [Yu et al.(2021b)Yu, Liu,
Long, Dong, Zou, and Xiao], with and without our rendering loss. For this, we
evaluate on the Laval Indoor HDR [Gardner et al.(2017)Gardner, Sunkavalli,
Yumer, Shen, Gambaretto, Gagné, and Lalonde] test set and report results in
tab. 2. With its saturation-driven attention, the LANet architecture
outperforms HallucinationNet. In addition, the use of the rendering loss
$\ell_{\mathrm{rend}}$ (eq. 1) helps the network focus on the bright light
sources, which is crucial for accurate radiance reproduction.

Planar vs spherical sampling We compare the impact of planar vs spherical
sampling (check supplementary) for training our network on LDR images. In both
cases, we follow the hierarchical sampling strategy of NeRF [Mildenhall et
al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] and
train the network with 64 coarse samples and 128 fine samples. We observe that
sphere sampling outperforms planar sampling for omnidirectional images as it
does not oversample points at the poles but does so uniformly on the sphere.
(more details in supplementary)

Loss in log space We evaluate the importance of computing the loss in log
space (c.f. sec. 3.2) in tab. 3, which suggests that the network is able to
estimate the high dynamic range equally well with or without the log-space
loss. However, fig. 6 shows that training in linear space results in more
floating artifacts and blurrier images than those obtained by the log loss.
This is because HDR intensities vary significantly and log space helps us
restrict the range which in turn makes it easier for NeRF-MLP to predict the
radiance values.

Order of operations PanoHDR-NeRF uses a NeRF network trained on HDR images. It
is also possible to reverse the order by training the NeRF on LDR images and
pass its output through the LDR2HDR network (dubbed “NeRF-LDR2HDR”). We
compare these options in tab. 3 and fig. 5. Though the metrics given in the
table don’t differ much, the supplementary video shows PanoHDR-NeRF produces
temporally more stable results. This could be due to the averaging that
naturally happens within the NeRF network.

##  5 Discussion and conclusion

The main contribution of this work is a novel pipeline to predict the full HDR
radiance of an indoor scene without using special hardware, careful scanning
of the scene, or intricately calibrated camera configurations. Our pipeline
can work with a single, off-the-shelf $360^{\circ}$ camera that is moved
around the scene. Although [Mildenhall et al.(2022)Mildenhall, Hedman, Martin-
Brualla, Srinivasan, and Barron, Huang et al.(2022)Huang, Zhang, Ying, Li,
Wang, and Wang] have demonstrated recovery of HDR intensities of forward
facing scenes, they require multiple sets of LDR images at various exposures.
Recovering HDR intensities for large indoor scenes using these methods
requires an elaborate setup which is tedious and cumbersome. PanoHDR-NeRF can
render novel $360^{\circ}$ views from any point within an unbounded indoor
scene in high dynamic range from a casually captured video. We show their use
for the realistic relighting of virtual objects in real scenes, hopefully
getting one step closer to democratizing augmented reality.

Limitations and future work Blurriness of the NeRF results is a big limitation
of this work, despite using cone-casting from Mip-NeRF. We believe further
improvements such as [Barron et al.(2022)Barron, Mildenhall, Verbin,
Srinivasan, and Hedman] can help in reconstructing sharper estimates. Another
limitation is that the photographer capturing the scene ends up modifying the
light field ever so slightly by casting shadows, creating reflections off of
shiny surfaces, etc. Unfortunately, the intensity changes this creates are too
soft for existing shadow detectors [Wang et al.(2020)Wang, Hu, Wang, Heng, and
Fu]. Methods modeling transient changes [Martin-Brualla et al.(2021)Martin-
Brualla, Radwan, Sajjadi, Barron, Dosovitskiy, and Duckworth] could
potentially be of help. The finetuning for each camera introduces additional
effort though only once per camera. Future research can improve generalization
across cameras, perhaps using multiple cameras for training, or through other
data augmentation techniques. Our approach learns radiance and view synthesis
in two independent steps by specialized networks. Exploring how both can be
done simultaneously, potentially in conjunction with geometry and material
estimation [Boss et al.(2021)Boss, Braun, Jampani, Barron, Liu, and Lensch,
Zhang et al.(2021)Zhang, Srinivasan, Deng, Debevec, Freeman, and Barron], is
an exciting direction for future work. PanoHDR-NeRF assumes that the scene is
static, so the lighting should remain unchanged for the inference time. In
addition, PanoHDR-NeRF is an offline method and therefore the novelty of the
proposed method is limited to only capturing process and not rendering. Recent
efforts have demonstrated how to significantly shorten training [Müller et
al.(2022)Müller, Evans, Schied, and Keller, Yu et al.(2021a)Yu, Li, Tancik,
Li, Ng, and Kanazawa] and inference [Reiser et al.(2021)Reiser, Peng, Liao,
and Geiger, Garbin et al.(2021)Garbin, Kowalski, Johnson, Shotton, and
Valentin] times of NeRF-based approaches, these could straightforwardly be
incorporated into our framework. By reducing the time between capture and
visualization, PanoHDR-NeRF can be used for AR/VR applications such as virtual
tours and VFX generation.

##  6 Acknowledgments

This research was supported by NSERC grant RGPIN-2020-04799, Compute Canada,
and a MITACS Globalink internship to Pulkit Gera. The authors thank Bowei Chen
for his early work on the project and David Ibarzabal for his help with data
capture. We also thank Yohan Poirier-Ginter and Jinsong Zhang for their help.

## References

  * [Attal et al.(2020)Attal, Ling, Gokaslan, Richardt, and Tompkin] Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin.  Matryodshka: Real-time 6dof video view synthesis using multi-sphere images.  In _Eur. Conf. Comput. Vis._ , 2020.
  * [Barron et al.(2021)Barron, Mildenhall, Tancik, Hedman, Martin-Brualla, and Srinivasan] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _IEEE/CVF Int. Conf. Comput. Vis._ , 2021.
  * [Barron et al.(2022)Barron, Mildenhall, Verbin, Srinivasan, and Hedman] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [Bertel et al.(2020)Bertel, Yuan, Lindroos, and Richardt] Tobias Bertel, Mingze Yuan, Reuben Lindroos, and Christian Richardt.  Omniphotos: Casual 360° vr photography.  _ACM Trans. Graph._ , 2020.
  * [Boss et al.(2021)Boss, Braun, Jampani, Barron, Liu, and Lensch] Mark Boss, Raphael Braun, V. Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P. A. Lensch.  Nerd: Neural reflectance decomposition from image collections.  _IEEE/CVF Int. Conf. Comput. Vis._ , 2021.
  * [Cho et al.()Cho, van Merrienboer, Gülçehre, Bahdanau, Bougares, Schwenk, and Bengio] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  In _Conf. Emp. Meth. Nat. Lang. Proc._
  * [Debevec(1998)] Paul Debevec.  Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography.  In _Conf. Comp. Graph. Int. Tech._ , SIGGRAPH, pages 189–198, 1998\.
  * [Debevec and Malik(1997)] Paul Debevec and Jitendra Malik.  Recovering high dynamic range radiance maps from photographs.  In _ACM SIGGRAPH_ , pages 369–378, 1997.
  * [Debevec et al.(1998)Debevec, Yu, and Boshokov] Paul Debevec, Yizhou Yu, and George Boshokov.  Efficient view-dependent ibr with projective texture-mapping.  In _EG Rend. Works._ , 1998.
  * [Endo et al.(2017)Endo, Kanamori, and Mitani] Yuki Endo, Yoshihiro Kanamori, and Jun Mitani.  Deep reverse tone mapping.  _ACM Trans. Graph._ , 2017.
  * [Flynn et al.(2019)Flynn, Broxton, Debevec, DuVall, Fyffe, Overbeck, Snavely, and Tucker] John Flynn, Michael Broxton, Paul E. Debevec, Matthew DuVall, Graham Fyffe, Ryan S. Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [Garbin et al.(2021)Garbin, Kowalski, Johnson, Shotton, and Valentin] Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In _IEEE/CVF Int. Conf. Comput. Vis._ , 2021.
  * [Gardner et al.(2017)Gardner, Sunkavalli, Yumer, Shen, Gambaretto, Gagné, and Lalonde] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde.  Learning to predict indoor illumination from a single image.  _ACM Trans. Graph._ , 2017.
  * [Gortler et al.(1996)Gortler, Grzeszczuk, Szeliski, and Cohen] Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen.  The lumigraph.  ACM, 1996.
  * [Gu et al.(2022)Gu, Maugey, Knorr, and Guillemot] Kai Gu, Thomas Maugey, Sebastian Knorr, and Christine Guillemot.  Omni-nerf: neural radiance field from 360° image captures.  In _Int. Conf. Multi. Expo_ , 2022.
  * [Hara and Harada(2022)] Takayuki Hara and Tatsuya Harada.  Enhancement of novel view synthesis using omnidirectional image completion.  _ArXiv_ , abs/2203.09957, 2022.
  * [He et al.(2017)He, Gkioxari, Dollár, and Girshick] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2017.
  * [Hedman et al.(2016)Hedman, Ritschel, Drettakis, and Brostow] Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel J. Brostow.  Scalable inside-out image-based rendering.  _ACM Trans. Graph._ , 2016.
  * [Hedman et al.(2018)Hedman, Philip, Price, Frahm, Drettakis, and Brostow] Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel J. Brostow.  Deep blending for free-viewpoint image-based rendering.  _ACM Trans. Graph._ , 2018.
  * [Hsu et al.(2021)Hsu, Sun, and Chen] Ching-Yu Hsu, Cheng Sun, and Hwann-Tzong Chen.  Moving in a 360 world: Synthesizing panoramic parallaxes from a single panorama.  _ArXiv_ , 2021.
  * [Huang et al.(2017)Huang, Chen, Ceylan, and Jin] Jingwei Huang, Zhili Chen, Duygu Ceylan, and Hailin Jin.  6-DOF VR videos with a single 360-camera.  In _IEEE Virt. Real._ , 2017.
  * [Huang et al.(2022)Huang, Zhang, Ying, Li, Wang, and Wang] Xin Huang, Qi Zhang, Feng Ying, Hongdong Li, Xuan Wang, and Qing Wang.  Hdr-nerf: High dynamic range neural radiance fields.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [Khademi and Ventura(2021)] Wesley Khademi and Jonathan Ventura.  View synthesis in casually captured scenes using a cylindrical neural radiance field with exposure compensation.  SIGGRAPH ’21. Association for Computing Machinery, 2021.
  * [Lee et al.(2018)Lee, An, and Kang] Siyeong Lee, Gwon Hwan An, and Suk-Ju Kang.  Deep chain hdri: Reconstructing a high dynamic range image from a single low dynamic range image.  _IEEE Access_ , 2018.
  * [Li and Fang(2019)] Jinghui Li and Peiyu Fang.  Hdrnet: Single-image-based hdr reconstruction using channel attention cnn.  In _Int. Conf. Mult. Sign. Proc._ , 2019.
  * [Lin et al.(2020)Lin, Xu, Mildenhall, Srinivasan, Hold-Geoffroy, DiVerdi, Sun, Sunkavalli, and Ramamoorthi] Kai-En Lin, Zexiang Xu, Ben Mildenhall, Pratul P. Srinivasan, Yannick Hold-Geoffroy, Stephen DiVerdi, Qi Sun, Kalyan Sunkavalli, and Ravi Ramamoorthi.  Deep multi depth panoramas for view synthesis.  In _Eur. Conf. Comput. Vis._ , 2020.
  * [Liu et al.(2020)Liu, Lai, Chen, Kao, Yang, Chuang, and Huang] Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang.  Single-image hdr reconstruction by learning to reverse the camera pipeline.  _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [Mantiuk and Azimi(2021)] Rafał K. Mantiuk and Maryam Azimi.  Pu21: A novel perceptually uniform encoding for adapting existing quality metrics for hdr.  _Pict. Coding Symp._ , 2021.
  * [Mapillary(2022)] Mapillary.  Opensfm.  https://github.com/mapillary/OpenSfM, 2022.
  * [Marnerides et al.(2018)Marnerides, Bashford-Rogers, Hatchett, and Debattista] Demetris Marnerides, Thomas Bashford-Rogers, Jonathan Hatchett, and Kurt Debattista.  Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content.  _Comput. Graph. Forum (CGF)_ , 2018.
  * [Martin-Brualla et al.(2021)Martin-Brualla, Radwan, Sajjadi, Barron, Dosovitskiy, and Duckworth] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2021.
  * [Meshry et al.(2019)Meshry, Goldman, Khamis, Hoppe, Pandey, Snavely, and Martin-Brualla] Moustafa Meshry, Dan B. Goldman, S. Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla.  Neural rerendering in the wild.  _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [Mildenhall et al.(2019)Mildenhall, Srinivasan, Cayon, Kalantari, Ramamoorthi, Ng, and Kar] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion.  _ACM Trans. Graph._ , 2019.
  * [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _Eur. Conf. Comput. Vis._ , 2020.
  * [Mildenhall et al.(2022)Mildenhall, Hedman, Martin-Brualla, Srinivasan, and Barron] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan, and Jonathan T. Barron.  Nerf in the dark: High dynamic range view synthesis from noisy raw images.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [Müller et al.(2022)Müller, Evans, Schied, and Keller] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 2022.
  * [Narwaria et al.(2015)Narwaria, Mantiuk, Silva, and Callet] Manish Narwaria, Rafał K. Mantiuk, Matthieu Perreira Da Silva, and Patrick Le Callet.  Hdr-vdp-2.2: a calibrated method for objective quality prediction of high-dynamic range and standard images.  _Journ. Elect. Imag._ , 2015.
  * [Reiser et al.(2021)Reiser, Peng, Liao, and Geiger] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In _IEEE/CVF Int. Conf. Comput. Vis._ , 2021.
  * [Rempel et al.(2007)Rempel, Trentacoste, Seetzen, Young, Heidrich, Whitehead, and Ward] Allan G. Rempel, Matthew Trentacoste, Helge Seetzen, H. David Young, Wolfgang Heidrich, Lorne Whitehead, and Greg Ward.  LDR2HDR: On-the-fly reverse tone mapping of legacy video and photographs.  _ACM Trans. Graph._ , 26(3), 2007.
  * [Riegler and Koltun(2020)] Gernot Riegler and Vladlen Koltun.  Free view synthesis.  In _Eur. Conf. Comput. Vis._ , 2020.
  * [Riegler and Koltun(2021)] Gernot Riegler and Vladlen Koltun.  Stable view synthesis.  _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2021.
  * [Santos et al.(2020)Santos, Tsang, and Kalantari] Marcel Santana Santos, Ing Ren Tsang, and Nima Khademi Kalantari.  Single image hdr reconstruction using a cnn with masked features and perceptual loss.  _ACM Trans. Graph._ , 2020.
  * [Serrano et al.(2019)Serrano, Kim, Chen, DiVerdi, Gutierrez, Hertzmann, and Masia] Ana Serrano, Incheol Kim, Zhili Chen, Stephen DiVerdi, Diego Gutierrez, Aaron Hertzmann, and Belen Masia.  Motion parallax for $360^{\circ}$ rgbd video.  _IEEE Trans. Vis. Comput. Graph._ , 2019.
  * [Sitzmann et al.(2019)Sitzmann, Thies, Heide, Nießner, Wetzstein, and Zollhöfer] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer.  Deepvoxels: Learning persistent 3d feature embeddings.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [Sloan et al.(2002)Sloan, Kautz, and Snyder] Peter-Pike J. Sloan, Jan Kautz, and John M. Snyder.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments.  _Conf. Comp. Graph. Int. Tech._ , 2002.
  * [Tancik et al.(2022)Tancik, Casser, Yan, Pradhan, Mildenhall, Srinivasan, Barron, and Kretzschmar] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar.  Block-nerf: Scalable large scene neural view synthesis.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [Tarko et al.(2019)Tarko, Tompkin, and Richardt] Joanna Tarko, James Tompkin, and Christian Richardt.  Real-time virtual object insertion for moving 360° videos.  In _Int. Conf. Virt. Real. Cont. App. Indus._ , 2019.
  * [Thies et al.(2019)Thies, Zollhöfer, and Nießner] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering.  _ACM Trans. Graph._ , 2019.
  * [Turki et al.(2022)Turki, Ramanan, and Satyanarayanan] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan.  Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [Walton and Steed(2018)] David R. Walton and Anthony Steed.  Dynamic hdr environment capture for mixed reality.  _ACM Symp. Virt. Real. Soft. Tech._ , 2018.
  * [Wang et al.(2020)Wang, Hu, Wang, Heng, and Fu] Tianyu Wang, Xiaowei Hu, Qiong Wang, Pheng-Ann Heng, and Chi-Wing Fu.  Instance shadow detection.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [Xu et al.(2021)Xu, Zheng, Xu, Tang, and Gao] Jiale Xu, Jia Zheng, Yanyu Xu, Rui Tang, and Shenghua Gao.  Layout-guided novel view synthesis from a single indoor panorama.  _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2021.
  * [Yang et al.(2022)Yang, Zhang, Li, Cui, Fanello, Bao, and Zhang] Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, S. Fanello, Hujun Bao, and Guofeng Zhang.  Neural rendering in a room: Amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects.  _ACM Trans. Graph._ , 41(4), 7 2022.
  * [Yu et al.(2021a)Yu, Li, Tancik, Li, Ng, and Kanazawa] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In _IEEE/CVF Int. Conf. Comput. Vis._ , 2021a.
  * [Yu et al.(2021b)Yu, Liu, Long, Dong, Zou, and Xiao] Hanning Yu, Wentao Liu, Chengjiang Long, Bo Dong, Qin Zou, and Chunxia Xiao.  Luminance attentive networks for hdr image and panorama reconstruction.  _Comput. Graph. Forum (CGF)_ , 2021b.
  * [Zhang et al.(2016)Zhang, Cohen, and Curless] Edward Zhang, Michael F. Cohen, and Brian Curless.  Emptying, refurnishing, and relighting indoor spaces.  _ACM Trans. Graph._ , 2016.
  * [Zhang and Lalonde(2017)] Jinsong Zhang and Jean-François Lalonde.  Learning high dynamic range from outdoor panoramas.  In _IEEE/CVF Int. Conf. Comput. Vis._ , 2017.
  * [Zhang et al.(2020)Zhang, Riegler, Snavely, and Koltun] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  _CoRR_ , abs/2010.07492, 2020.
  * [Zhang et al.(2021)Zhang, Srinivasan, Deng, Debevec, Freeman, and Barron] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  _ACM Trans. Graph._ , 2021.
  * [Zhou et al.(2018)Zhou, Tucker, Flynn, Fyffe, and Snavely] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: learning view synthesis using multiplane images.  _ACM Trans. Graph._ , 2018.

##  Appendix A Supplementary

###  A.1 Datasets & Training LDR2HDR

We train the LDR2HDR module on Laval Indoor Dataset. We augment the training
set with random rotations (about the vertical axis), intensity changes
(multiply the image by $2^{\gamma}$, with $\gamma\sim\mathrm{U}(-0.1,0.1)$)
and exposure changes (make the median intensity of image $0.5+\gamma$). Here,
$\mathrm{U}(a,b)$ indicates a uniform distribution in the $[a,b]$ interval.
After augmentation, the resulting HDR panorama is used as target $\mathbf{t}$
for training. The input $\mathbf{x}$ is created by clipping $\mathbf{t}$ to
the [0,1] interval. We further apply hue shift and unsharp mask (amount =1,
$\sigma\sim\mathrm{U}(0,3)$), add small amount of per-pixel Gaussian noise
($\sigma=0.01$), and augment the tonemapping $\mathbf{x}^{1+\gamma}$ to
simulate the behaviour of a real camera. We train the network for 1500 epochs
using the Adam [Kingma2015AdamAM] optimizer with a learning rate
$\eta=10^{-4}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$ and $\epsilon=10^{-8}$.

Further, we finetune the model on a small dataset captured using Ricoh Theta Z
to alleviate domain gap. We run 130 additional epochs using the Adam
[Kingma2015AdamAM] optimizer with the same parameters as above.

###  A.2 Training PanoHDR-NeRF

We use NeRF++ [Zhang et al.(2020)Zhang, Riegler, Snavely, and Koltun] as the
basis of our project. An eight-layer MLP with 256 channels is used for
predicting radiance and densities at the sampled points. Along each ray, we
sample 64 points for training the coarse network and 128 points for training
the fine network. The batch size of rays is 1024. We use integrated positional
encoding to encode the inputs of the network as used in MipNeRF [Barron et
al.(2021)Barron, Mildenhall, Tancik, Hedman, Martin-Brualla, and Srinivasan].
Similarly a single MLP is used to encode the scene. In addition, we also use
spherical sampling, which weights pixels at the poles less with respect to
pixels in the middle. The network is trained using the Adam optimizer
[Kingma2015AdamAM] with learning rate $\eta=10^{-4}$, $\beta_{1}=0.9$,
$\beta_{2}=0.999$ and $\epsilon=10^{-8}$. The resolution of the training
images is $960\times 480$. The network is trained for approximately 500,000
iterations, which takes around 36 hours on a NVIDIA V100 GPU.

###  A.3 Spherical Sampling

Equirectangular images correspond to the projection of a spherical signal onto
a 2D plane, where (normalized) pixel coordinates $(i,j)$ are related to
azimuth $\varphi\in[-\pi,\pi]$ and elevation
$\theta\in[-\nicefrac{{\pi}}{{2}},\nicefrac{{\pi}}{{2}}]$ angles by

$i=\frac{1}{2\pi}\varphi\cos\theta+\frac{1}{2}\,,\;\mathrm{and}\;j=\frac{\nicefrac{{\pi}}{{2}}-\theta}{\pi}\,.$
(3)

To train PanoHDR-NeRF, we sample rays in spherical coordinates instead of
pixel coordinates, where $\theta\sim\mathrm{U}(-\pi,\pi)$,
$\varphi=\cos^{-1}(2\beta-1)$, and $\beta\sim\mathrm{U}(0,1)$.
We compare the results between planar and spherical sampling used for training
PanoHDR-NeRF. We observe that spherical sampling performs much better and
provides sharper results.

| Planar | Spherical
---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
Chess room | $29.635$ | $0.919$ | $31.389$ | $0.929$
Stairway | $25.381$ | $0.892$ | $27.381$ | $0.891$
Cafeteria | $22.940$ | $0.845$ | $24.038$ | $0.842$
Spotlights | $25.557$ | $0.847$ | $26.128$ | $0.852$
Dark class | $29.955$ | $0.911$ | $31.368$ | $0.917$
Small class | $29.231$ | $0.906$ | $30.611$ | $0.911$
Overall | $27.115$ | $0.886$ | $28.486$ | $0.902$
Table 4: Quantitative comparison between planar and spherical sampling (on LDR
images only) averaged over all captured scenes. Spherical sampling has better
results.

###  A.4 Video

We provide an additional video to showcase our results. We relight 3 virtual
test objects made up of metal (armadillo), diffuse (bunny) and glass (sphere)
with our recovered HDR images at novel viewpoints to demonstrate the dynamic
range recovered. We present our results with a variety of scenes captured
casually.