# Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting
Prediction

Jiayang Bai1, Zhen He1, Shan Yang1, Jie Guo1111Corresponding author., Zhenyu
Chen1, Yan Zhang1, YanwenGuo1
1Nanjing University, Nanjing, China
{jybai<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>{zhangyannju<EMAIL_ADDRESS>
###### Abstract

Predicting panoramic indoor lighting from a single perspective image is a
fundamental but highly ill-posed problem in computer vision and graphics. To
achieve locale-aware and robust prediction, this problem can be decomposed
into three sub-tasks: depth-based image warping, panorama inpainting and high-
dynamic-range (HDR) reconstruction, among which the success of panorama
inpainting plays a key role. Recent methods mostly rely on convolutional
neural networks (CNNs) to fill the missing contents in the warped panorama.
However, they usually achieve suboptimal performance since the missing
contents occupy a very large portion in the panoramic space while CNNs are
plagued by limited receptive fields. The spatially-varying distortion in the
spherical signals further increases the difficulty for conventional CNNs. To
address these issues, we propose a _local-to-global strategy_ for large-scale
panorama inpainting. In our method, a depth-guided local inpainting is first
applied on the warped panorama to fill small but dense holes. Then, a
transformer-based network, dubbed _PanoTransformer_ , is designed to
hallucinate reasonable global structures in the large holes. To avoid
distortion, we further employ cubemap projection in our design of
PanoTransformer. The high-quality panorama recovered at any locale helps us to
capture spatially-varying indoor illumination with physically-plausible global
structures and fine details.

Figure 1: We propose a locale-aware indoor illumination prediction method that
can generate a full and texture-rich HDR panorama (the third column in each
group) at any locale in the scene, enabling spatially-varying and consistent
shading after virtual object insertion (the first and second columns).

##  1 Introduction

Estimating high-dynamic range (HDR) indoor illumination from a single low-
dynamic range (LDR), perspective image is a fundamental problem in computer
vision and graphics. It has a huge impact on many applications including
augmented and mixed reality (AR/MR) [21], scene understanding [26] and
relighting [10]. However, this problem is highly ill-posed as the pixel
intensities in an observed perspective image are a complex function of scene
geometries, material properties and lighting distributions.

With the advent of deep learning and large-scale datasets, convolutional
neural networks (CNNs) are now the de-facto architectures for solving this
ill-posed problem. However, many existing CNNs only recover a single global
environment map [12, 34, 46], or some other kinds of compact representations,
_e.g._ , Spherical Harmonics (SH) [4] and Spherical Gaussians (SG) [24, 45],
at the camera’s viewpoint of the input perspective image. This completely
ignores spatially-varying lighting effects that dominate indoor scenarios and
hence results in unrealistic shading and shadows for inserted virtual objects.

Several works have noticed this challenge and tried to recover the independent
illumination at each locale (the 3D location of the inserted object). For
instance, Song and Funkhouser _et al._ [35] have suggested decomposing the
indoor illumination estimation problem into three sub-tasks: depth-based image
warping, LDR panorama inpainting and HDR panorama reconstruction. The success
of this pipeline lies in the second sub-task which tries to recover missing
contents for the incomplete panorama. However, directly applying existing
convolution-based image inpainting networks [16, 25, 27, 30, 43] to the
incomplete panorama is impractical due to (1) the hole regions of warped
images are very large ($>60$% in general), which may easy let many image
inpainting methods fail; (2) panoramas have severe spatially-varying
distortions that will mislead convolution-based feature extraction modules.

More importantly, as linear operations have a limited receptive field,
conventional convolutions fail to capture long-distance relationships that
prevail in panoramas. To fill large holes in the warped panorama according to
long-range dependencies, we resort to a transformer-based architecture, dubbed
_PanoTransformer_. Our PanoTransformer is specially designed for handling
distorted spherical signals with cubemap projections and is adept at restoring
visually plausible and globally consistent contents in the large out-of-view
regions. We observe that many relatively small but dense holes are actually
caused by pixel stretching during image warping at a given locale. Considering
that these missing regions are visible in the input, we design a local
inpainting module and apply it on the warped incomplete panorama before
restoring global structures via PanoTransformer. Guided by estimated depth
values, our local inpainting module can efficiently fill the gaps between
sparse pixels without introducing any artifact. We also find that this local
inpaining module helps our PanoTransformer better capture global structures in
the panorama and make the training stable. Extensive experiments on multiple
datasets verify that the proposed _local-to-global panorama inpainting
pipeline_ significantly improves the quality of recovered indoor lighting
distribution at any locale and enables high-fidelity and globally-coherent
shading on inserted virtual objects. In particular, our method can reproduce
fine texture details that are consistent with the inserting points on specular
surfaces (see Fig. 1).

In summary, we make the following contributions:

  * •

A local-to-global panorama inpainting pipeline is proposed to fill missing
contents in the warped panorama, enabling reproducing spatially-varying indoor
illumination with physically-plausible global structures and fine details at
any locale.

  * •

A transformer-based network, named PanoTransformer, is designed to capture
distortion-free global features in the panorama and restore globally-
consistent structures accordingly.

  * •

A new large-scale panorama inpainting dataset is collected with paired masked
input and ground truth panoramas.

##  2 Related Work

Traditional methods for indoor lighting estimation.  To capture the lighting
distribution at a target location in an indoor scene, a common practice is to
use a physical probe [5, 6, 32] or other 3D objects [39, 14, 2, 42]. However,
inserting physical objects into the scene is not feasible in many scenarios.
Moreover, it is usually expensive and has poor scalability. Another line of
works tries to estimate illumination by jointly optimizing the geometry,
reflectance properties, and lighting models of the scene. In order to find the
set of values that best explained the observed input image, previous works
[18, 19, 28, 47] regressed the relevant parameters by minimizing the
difference between the generated image and the ground truth.

Deep learning-based methods for indoor lighting estimation.  Recently,
significant progress has been made in lighting estimation with deep learning.
Some works [8, 12, 23, 9] directly trained convolutional neural networks to
estimate scene parameters including albedos, normals, shadows and lights from
a single image. EMLight [46] inferred Gaussian maps to guide the illumination
map generation. However, it simplified the scene geometry to be a spherical
surface and did not allow to predict spatially-varying lighting. Inspired by
earth mover’s distance, GMLight [44] added a geometric mover’s loss to the
pipeline in EMLight. Given a perspective image and a 2D location in that
image, Garon _et al._ [13] achieved real-time spatially-varying lighting
estimation by regressing a 5th order SH representation of the lighting at the
selected location with CNNs. For estimating the incident illumination at any
3D location, Srinivasan _et al._ [36] predicted unobserved scene contents with
a 3D CNN. Xu _et al._ [41] detected the planar surfaces in the scene and
computed its overall illumination. Inspired by classic volume rendering
techniques, Wang _et al._ [38] proposed a unified, learning-based inverse
rendering framework that can recover 3D spatially-varying lighting from a
single image. These works were plagued by the limited receptive fields and
predicted inconsistent contents in those regions missing dued to pixel
streching.

Figure 2: Architecture of the proposed locale-aware indoor illumination
prediction method. The key is a local-to-global panorama inpainting pipeline
that recovers both physically-plausible global structures and fine local
details from a FOV-limited input image.

##  3 Local-to-global Panorama Inpainting

Our goal in this paper is to estimate a full HDR environment map
$\mathbf{P}_{\text{HDR}}$ at a locale $\mathbf{R}$ of an FOV-limited LDR image
$\mathbf{I}$. We follow the general framework of Song _et al._ [35] which
decomposes this ill-posed problem into three sub-tasks: depth-based image
warping, LDR panorama inpainting and HDR panorama reconstruction (see Fig. 2).
In the first task, we leverage the recent DPT [31] to estimate the depth map
$\mathbf{D}$ for $\mathbf{I}$. Then $\mathbf{D}$ and $\mathbf{I}$ are
geometrically warped and transformed to $360^{\circ}$ spherical panoramas
centered at the selected locale $\mathbf{R}$, denoting as $\mathbf{\hat{D}}$
and ${\hat{\mathbf{P}}}$ respectively. This warping operation is realized
through a forward projection using the estimated scene geometry and camera
pose. Some regions in $\mathbf{\hat{P}}$ are missing because they do not have
a projected pixel in $\mathbf{I}$. The following task, LDR panorama
inpainting, aims to infer a dense panorama from the sparse
${\hat{\mathbf{P}}}$. In this section, we introduce a local-to-global panorama
inpainting pipeline that can recover a complete LDR panorama $\mathbf{P}_{G}$
from $\mathbf{\hat{P}}$. In the last sub-task, we utilize the state-of-the-art
network of [33] to reconstruct the HDR panoramas $\mathbf{P}_{\text{HDR}}$
from $\mathbf{P}_{G}$ for inserted objects at $\mathbf{R}$. We detail our
local-to-global pipeline in this paper and more details about the other off-
the-shelf networks are provided in the supplemental materials.

###  3.1 Observation and overview

Input | Attention matrix | Attention map
---|---|---
|  |
|  |
Figure 3: From left to right: the input incomplete panoramas with red boxes
specifying the selected patches, attention matrices of the 9th transformer
block with red lines indicating the attention scores of the selected patches,
attention maps of the selected patches.

For panorama inpainting, previous works [35] mostly resort to fully
convolutional networks. However, CNN-based models achieve suboptimal
performance due to large-scale sparse missing contents in the warped panorama
and some inherent limitations of convolutional layers. CNNs are good at
preserving local structures and modeling rich textures but fail to complete
the large hole regions. Therefore, previous works can hardly acquire
sufficiently broad context as well as significant high-level representations
from sparse omnidirectional images. Meanwhile, the distortion in spherical
panoramas will further hamper the performance on large-scale inpainting.

Compared to CNN models with limited perspective fields, transformer is
designed to support long-term interaction via the self-attention module [7].
The self-attention mechanism can directly compute the interaction between any
pair of patches, naturally capturing long-range dependencies and having a
global receptive field at every stage. However, we observe that transformers
work poorly on the sparse input which is the case in our task. We compare the
attention maps and attention score maps for the selected patch from a sparse
panorama and a dense one for illustration in Fig. 3. As we can see, the query
patch contains adequate illumination information. Given a dense input, the
query patch impacts some regions (e.g. the ground) with red colors in the
attention map. However, transformer blocks have trouble recovering the global
structure from scattered pixels, thus the lighting can not pass information to
invisible patches properly, resulting in the smooth attention map.

Figure 4: Illustration of warping. The warping operation results in missing
regions which are categorized as pixel-stretching regions and out-of-view
regions.

Unfortunately, the problem of sparsity is inevitable in our task due to the
limited FOV of our input perspective images. The panorama captured at
$\mathbf{R}$ should have a 360∘ FOV while some parts are out-of-view in
$\mathbf{I}$. We mark the invisible regions with blue in Fig. 4. The missing
parts are expected to be restored by a global understanding of the scene.
Another factor of the sparsity stems from pixel stretching. During image
warping, some regions marked in red in Fig. 4 are actually visible in
$\mathbf{I}$, but still have small holes due to pixel stretching.

Based on the above observations, our main idea is that we first fill pixel-
stretching regions according to their neighboring pixels to alleviate the
sparsity, and then fill other large holes based on the global understanding of
the whole scene. To this end, we propose a novel local-to-global inpainting
pipeline, which can be formulated as follows:

$\mathbf{P}_{G}=\mathbf{M}\odot\mathbf{\hat{P}}+(1-\mathbf{M})\odot\mathcal{G}(\mathcal{L}(\mathbf{\hat{P}});\mathcal{L}(\mathbf{M}))$
(1)

where $\mathbf{M}$ is the binary mask indicating visible pixels in
$\mathbf{\hat{P}}$ and $\odot$ denotes element-wise multiplication.

The key to Eq. 1 is a local-to-global panorama inpainting pipeline that
applies a local inpainting module $\mathcal{L}$ and a global inpainting module
$\mathcal{G}$ sequentially on the warped panorama $\mathbf{\hat{P}}$. Our
local inpainting method aims to fill dense holes in the pixel-stretching
regions, according to the depth information. The local inpainting module
employs a modified bilateral filtering-based method to remove dense and small
holes in the pixel-stretching regions. After that, a global inpainting module
based on a novel transformer architecture is developed to extract reliable
global features from visible regions and then fill large holes in the out-of-
view regions. Our specially-designed transformer architecture, named
PanoTransformer, takes a cubemap projection as input to resolve the problem of
spatial distortion in the spherical signals.

###  3.2 Depth-guided local inpainting

Algorithm 1 Depth-guided local inpainting

1:The input image, $\mathbf{I}$; The estimated depth, $\mathbf{D}$; The warped
image centered at selected location, $\mathbf{\hat{P}}$; The warped depth
centered at selected location, $\mathbf{\hat{D}}$; The threshold $t$

2:The panorama inpainted locally, $\mathbf{P}_{L}$;

3:$\mathbf{P}_{L}=\mathbf{\hat{P}}$

4:$//$ Depth recovery.

5:$\mathbf{\hat{D}}=\text{morphologyClosing}(\mathbf{\hat{D}})$

6:$\mathbf{\hat{D}}=\text{bilateralFilter}(\mathbf{\hat{D}})$

7:$//$ Depth-guided inpainting.

8:for each $d\in\mathbf{\hat{D}}$ do

9: Calculate the pixel coordinate $\mathbf{c}_{p}$ of $d$ ;

10: $\mathbf{c}_{w}=\text{PixelToWorldCoordinate}(\mathbf{c}_{p})$

11: Project $\mathbf{c}_{w}$ to $\mathbf{D}$ to get its pixel coordinate
$\mathbf{c}$;

12: $d_{i}=\mathbf{D}[\mathbf{c}]$

13: if $\|d_{i}-d\|<t$ then

14: $\mathbf{P}_{L}[\mathbf{c}_{p}]=\mathbf{I}[\mathbf{c}]$

15: end if

16:end for

The local inpainting module aims to alleviate the sparsity caused by pixel
stretching. Our local inpainting method contains two basic steps shown in
Algorithm 1. First, we leverage morphology operations (Line 3) and bilateral
filtering (Line 4) to fill holes in the warped depth $\mathbf{\hat{D}}$ as
much as possible. The intuition of performing depth recovery is that depth
values in the pixel stretching areas vary smoothly, while the warped panorama
$\mathbf{\hat{P}}$ may have rich textures.

Then, we traverse all valid depth values in $\mathbf{\hat{D}}$, and fill
missing regions in $\mathbf{\hat{P}}$ with re-projected pixel values from
$\mathbf{I}$. Guided by the recovered depth, we only fill those pixel-
stretching regions with colors in $\mathbf{I}$. Note that our local inpainting
method effectively fills dense but small holes and maintains the local
structures.

###  3.3 Global inpainting with PanoTransformer

For global inpainting, we design and train a transformer-based network, named
PanoTransformer, to hallucinate contents in out-of-view regions.
PanoTransformer can be logically separated to an encoder and a decoder, where
the encoder captures long-range distortion-free representations, while the
decoder gradually recovers the spatial information for generating accurate
pixel-level predictions.

Considering the distortion in the equirectangular projection, PanoTransformer
takes the warped cubemap projection as input. To distinguish pixels needed to
be regressed, a binary mask is concatenated to the RGB channels in cubemap
projection. We denote the input as $\mathbf{x}\in\mathbb{R}^{6\times H\times
W\times 4}$ where $H$ and $W$ is the height and width of each face in the
cubemap projection.

In the encoder, the input cubemap projection $\mathbf{x}$ is first reshaped
into a sequence of flattened 2D patches
$\mathbf{x}_{p}\in\mathbb{R}^{N\times(p^{2}\times 4)}$, where $(p,p)$ is the
resolution of each image patch, and $N=\frac{6\times H\times W}{p^{2}}$ is the
resulting number of patches. Each patch is then mapped to 1D tokens
$z_{i}\in\mathbb{R}^{d}$ using a trainable linear projection and augmented
with position embeddings to retain positional information. These tokens are
then fed into several transformer blocks. Each transformer block comprises
multiheaded self-attention (MHSA) [37], layernorm (LN) [1] and MLP blocks. In
$l$-th transformer block, $\boldsymbol{z}^{l}$ is fed as the input, yielding
$\boldsymbol{z}^{l+1}$ as follows:

$\boldsymbol{w}^{l}=\text{MHSA}(\text{LN}(\boldsymbol{z}^{l}))+\boldsymbol{z}^{l}$
(2)
$\boldsymbol{z}^{l+1}=\text{MLP}(\text{LN}(\boldsymbol{w}^{l}))+\boldsymbol{w}^{l}$
(3)

Noted that the token-dimensionality is fixed throughout all layers. Thus the
resulting $\boldsymbol{z}$ can be transformed back to a cubemap-shape
representation $\boldsymbol{z}\in\mathbb{R}^{6\times H\times W\times C}$
according to the position of the initial patch. $C$ is the channel of the
extracted feature. These reconstructed priors $\boldsymbol{z}$ contain ample
cues of global structure and coarse textures, thanks to the transformer’s
strong representation ability and global receptive fields. $\boldsymbol{z}$
can be regarded as six images of the cubemap projection. We feed these image-
like features to six Residual blocks [15] to replenish texture details.

Figure 5: We apply masks from [35] on high-quality panoramas from Matterport3D [3], SUN360 [40] and Laval [12] to generate pairs of masked input and the ground truth for training. |  |  |
---|---|---|---
Our result and the input | With [35] | Ours | Reference
Figure 6: Comparison between models trained with our dataset and the dataset
of Song _et al._ [35].

###  3.4 Dataset

Currently, the only dataset that contains paired LDR perspective images and
the corresponding HDR panoramas for a diverse set of locales is proposed by
Song _et al._ [35] based on Matterport3D [3]. Unfortunately, the reconstructed
HDR panoramas have obvious artifacts (_e.g._ , stitching seams and broken
structures) as we explained in the supplemental materials. This prohibits our
model from inferring complete and globally consistent structures seen at each
locale.

Considering the above issue, we collect a large scale dataset with high-
quality and diverse panoramas from Matterport3D [3], SUN360 [40] and Laval
[12]. Apart from the panoramas, training PanoTransformer also requires masks
to generate the sparse input $\mathbf{\hat{P}}$. As the invisible regions are
mainly on the top of panorama, we generate masks from the dataset of Song _et
al._ [35] instead of generating randomly. These sparse masks are obtained by
geometrically warping, fitting to the real-world data distribution. These
masks are locally inpainted before feeding to PanoTransformer. The main
difference of our dataset to Song _et al._ ’s [35] dataset is that our
panoramas and masks are unpaired, hence we can randomly apply diversified
irregular masks on one panorama to generate various input. Since we focus on
the inpainting task, we do not require that the mask and the panorama are
physically correlated. Our dataset (some examples are shown in Fig. 5) ensures
that the ground-truth panoramas are free from artifacts. In all, we gather
38,929 high-quality panoramas accompanied by randomly selected masks for
training and 5,368 for evaluation. We ensure that the scenes used in the
evaluation would not appear in the training procedure. As shown in Fig. 6, the
model trained with our dataset produces much better results than that trained
with Song _et al._ ’s [35] dataset. Note the cluttered structures generated by
the latter model due to artifacts in Song _et al._ ’s [35] dataset. Please
refer to the supplemental material for more comparisons.

###  3.5 Loss function and training details

We optimize PanoTransformer by minimizing a pixel-wise reverse Huber loss [22]
between the predicted panoramas and corresponding ground truth. Since using a
standard L1 loss function to learn a binary light mask heavily penalizes even
small shifts of a light source position, the reverse Huber loss takes
advantage of L1 loss and L2 loss as below:

$L_{B}=\left\\{\begin{aligned} &|y-\hat{y}|,&|y-\hat{y}|\leq T\\\
&\frac{(y-\hat{y})^{2}+T^{2}}{2T},&|y-\hat{y}|\geq T\end{aligned}\right.$ (4)

where $y$ is the ground truth value and $\hat{y}$ is the prediction. The
threshold $T$ is set to 0.2 in our experiments. To generate more realistic
details, an extra adversarial loss is also involved in the training process.
Our discriminator adopts the same architecture as Patch-GAN [17].

We implement our PanoTransformer using the PyTorch framework [29]. Adam [20]
optimizer is used with the default parameters $\beta_{1}=0.9$ and
$\beta_{2}=0.999$ and an initial learning rate of 0.0001. PanoTransformer is
trained on our dataset for 100 epochs. Training are conducted on two NVIDIA
RTX 3090 GPUs with a batch size of 8.

Figure 7: Rendering comparison with Gardner _et al._ [11], Neural Illumination
[35] and EMLight [46]. We render two teapots with a matte silver and a mirror
material below each recovered/ground-truth illumination map.

##  4 Experiments

In this section, we evaluate the performance of our method in indoor
illumination prediction and make comparisons with the state of the art. To
further validate our local-to-global inpainting method, we also compare the
inpainted results $\mathbf{P}_{G}$ against results from Neural Illumination
[35]. More results and comparisons can be found in the supplemental materials.

###  4.1 Evaluation for illumination estimation

|  |
---|---|---
|  |
|  |
Li _et al._ [26] |  [35] | Ours
Figure 8: Results of inserted objects of different materials with estimated
illumination by different methods.  Figure 9: Results of user study on 30
scenes. Each scene in the Garon _et al._ ’s dataset [13] (Scenes 1-20) is
shown as a column, where different colors indicate that users preferred the
corresponding method instead of the ground truth. As Scenes 21-30 do not
provide the ground truth, we show the ratio of choosing the method that
produces the most likable results.

Qualitative comparison:  In Fig. 7, we visualize illumination maps and the
corresponding rendering from several state-of-the-art methods and our
pipeline. Gardner _et al._ [11] regress a limited field-of-view photo to HDR
illumination without strong assumptions on scene geometry, material
properties, or lighting. As they predict illumination for the whole scene,
this method cannot model spatial variations. EMLight [46] ignores the complex
scene geometry and simplifies the lighting distribution of scenes with a
Gaussian map. Thus it can not deal with occlusion. In contrast, we estimate
depth to have a better understanding of the scene, which leads to more
accurate illumination estimation. Similar to our method, Neural Illumination
[35] also decomposes the task into sub-tasks. However, it struggles to infer
the position of lighting with a limited receptive field, especially when the
input is terribly sparse. As Neural Illumination [35] has not shared code and
model weights, we implemented and trained their network with our datasets. We
can see that with a global understanding in local-to-global inpainting module,
our method produces environment maps with accurate lighting and perceptually
plausible details, ensuring realistic shading.

We further conduct a user study on 20 scenes of Garon _et al._ [13] and
additional 10 scenes from the Internet. As shown in Fig. 8, multiple virtual
objects are inserted to these scenes. For scenes of Garon _et al._ [13], we
relight bunny models with diffuse materials using the ground truth light probe
and estimated illumination from Neural Illumination [35], Li _et al._ [26] and
our method. Li _et al._ [26] achieve the state-of-the-art performance by
leveraging a deep inverse rendering framework to obtain a complete scene
reconstruction, estimating shape, spatially-varying lighting, and non-
Lambertian surface reflectance from a single RGB image. To reflect details of
predictions, we render mirror spheres with estimated illumination in the
remaining scenes. The user study is conducted by asking 84 users to choose
which rendering is more realistic between rendered image pairs. The results
are shown in Fig. 9. For scenes with inserted bunny models, Li _et al_. [26]
and our method both beat each other in half of the scenes, which suggests that
they are comparable in predicting lighting distribution. However, Li _et al._
[26] model the lighting with spherical Gaussian, leading the mirror sphere
looks diffuse. For Scenes 21-30, our method outperforms the other methods on
the mirror spheres, indicating that our method produces plausible details in
coherence with the environment.

Table 1: Comparing the quantitative performance of our method to Gardner _et al._ [11], Neural Illumination [35] and EMLight [46]. D, S, M denote a diffuse, a matte silver and a mirror material of the rendered spheres, respectively. A represents the angular error [34]. Metrics |  [11] |  [35] |  [46] | Ours
---|---|---|---|---
D | MSE$\downarrow$ | 35.77 | 22.67 | 12.08 | 9.18
RMSE$\downarrow$ | 53.22 | 39.28 | 18.26 | 14.36
S | MSE$\downarrow$ | 33.84 | 16.45 | 17.05 | 9.75
RMSE$\downarrow$ | 53.38 | 24.09 | 31.79 | 22.32
M | MSE$\downarrow$ | 34.28 | 23.69 | 16.71 | 9.64
RMSE$\downarrow$ | 52.28 | 41.44 | 30.86 | 21.89
A | mean$\downarrow$ | 1.31 | 1.92 | 1.21 | 1.13
std$\downarrow$ | 0.34 | 0.16 | 0.34 | 0.34

Quantitative comparison:  To evaluate our performance on estimated
illumination, we render three spheres with different materials: gray diffuse,
matte silver and mirror with predicted illumination maps and ground truth.
Then we use some standard metrics including root-mean-square error (RMSE) and
the mean absolute error (MAE) to evaluate. To evaluate the accuracy of light
sources, we take the mean angular error from both ground truth lights and
predicted lights as the final angular error [34] between the two HDR
illumination map. All these metrics have been widely adopted in the evaluation
of illumination prediction. For evaluation, we use 2,000 pairs of input LDR
images from Laval dataset and the ground truth HDR illumination maps captured
at the camera. Tab. 1 shows quantitative comparisons of our approach against
Gardner _et al._ [11], Neural Illumination [35] and EMLight [46]. As seen, our
pipeline outperforms all methods in comparison under different evaluation
metrics and materials.

Input image |  [35] | Ours | Ground truth
---|---|---|---
|  |  |
|  |  |
Figure 10: Qualitative comparison with Neural Illumination [35] on large-scale
panorama inpainting.

###  4.2 Evaluation for panorama inpainting

Table 2: Quantitative comparisons on the panorama inpainting with Neural Illumination [35] and different model variants in the ablation study. Method | SSIM$\uparrow$ | PSNR$\uparrow$ | FID$\downarrow$
---|---|---|---
Neural Illumination [35] | 0.30 | 14.85 | 255.60
Ours | 0.68 | 21.44 | 39.36
-Cubemap | 0.62 | 20.51 | 70.09
-Local | 0.60 | 19.46 | 112.32
-GAN | 0.68 | 21.50 | 76.07

Qualitative comparison:  To demonstrate the effectiveness of our transformer-
based network for global inpainting, we compare inpainted results from Neural
Illumination [35] and our method in Fig. 10. Specifically, Neural Illumination
[35] could generally produce rough structures. However, the limited receptive
fields of CNNs prohibit the understanding of global structures in the
panorama. Besides, Neural Illumination doesn’t maintain the visible regions
with masks, hence these regions will be changed after prediction. In contrast,
our results are more elegant with significantly fewer noticeable
inconsistencies and artifacts, outperforming Neural Illumination [35] in
panorama inpainting.

Quantitative comparison:  The inpainting evaluation is conducted on our test
set containing 5,000 pairs of masked input and the ground truth. Tab. 2 shows
quantitative comparisons of our approach against Neural Illumination [35].
Average PSNR, SSIM, FID values are listed for the inpainted LDR panoramas.
Obviously, our method achieves superior results compared with Neural
Illumination in all metrics. This further shows that our method achieves
state-of-the-art performance on large-scale panorama inpainting.

|  |  |
---|---|---|---
Our result and the input | -GAN | Ours | Reference
|  |  |
Our result and the input | -Cubemap | Ours | Reference
|  |  |
Our result and the input | -Local | Ours | Reference
Figure 11: Validating the contribution of the GAN loss, the cubemap projection
and the local inpainting module.

###  4.3 Ablation study

To evaluate the effectiveness of our design in our method, we develop three
model variants, denoted as -GAN model, -Cubemap model and -Local model
respectively. The quantitative results are reported in Tab. 2. We also
qualitatively evaluate the performance of the model variants in Fig. 11.

-GAN represents our PanoTransformer trained without the GAN loss. From the top row in the Fig. 11, we can see that our model trained without the GAN loss produces over-smooth textures that are close to the mean intensity of the surround regions. With the help of the GAN loss, our complete model is able to produce high-frequency signals and hallucinate realistic details.

Taking cubemap projection as the input aims to remove distortion in the
panorama. To show the effectiveness of cubemap projection, we adapt
PanoTransformer to -Cubemap model that takes the equirectangular projection as
the input and outputs the LDR panoramas directly. As seen in the middle row in
Fig. 11, the -Cubemap model suffers from distorted structures. Our complete
model outperforms the -Cubemap model clearly, demonstrating the superiority of
the cubemap projections in handling spherical signals.

To validate the importance of our local inpainting module, we remove this
module and directly train PanoTransformer with sparse panoramas
$\mathbf{\hat{P}}$. The bottom row in Fig. 11 shows that the -Local model
introduces artifacts to the prediction, which explains the attention maps of
the sparse input in Fig. 3. With the local inpainting module, our pipeline
produces more faithful and clearer results, indicating that the local
inpainting module promotes the performance of PanoTransformer significantly.

###  4.4 Limitation and discussion

Since we decompose the illumination estimation into three subtasks, the
quality of predictions is affected by the performance of these subtasks.
Thanks to the tremendous success of deep learning, depth estimation and HDR
panorama reconstruction from one single image have achieved significant
progresses. We resort to the off-the-shelf DPT [31] for depth estimation and
the pretrained network of Santos _et al._ [33] for HDR panorama
reconstruction. More details about the pretrained models can be found in the
supplemental materials. Although we do not finetune for our tasks, the
predicted depth and HDR map are still acceptable due to the robustness of the
state-of-the-art networks. Nonetheless, we still have failure cases when the
estimated depth is unreliable. Our performance can be improved with the
advance of depth estimation and HDR panorama reconstruction.

##  5 Conclusion

Locale-aware indoor illumination estimation can be decomposed into three sub-
tasks: depth-based image warping, panorama inpainting and HDR panorama
reconstruction. In this paper, we propose a novel local-to-global panorama
inpainting pipeline that well solves the second sub-task, significantly
improving the performance of the spatially-varying indoor lighting estimation.
In the pipeline, a depth-guided local inpainting module is first applied to
fill holes due to pixel stretching. For global inpainting, we resort to a
transformer-based network that captures distortion-free features from cubemap
projection and enables physically-plausible reconstruction of global
structures in the large out-of-view regions. We have validated our state-of-
the-art performance through qualitative and quantitative comparisons against
previous methods. Ablation studies further exemplify that the specially-
designed large-scale panorama inpainting pipeline is able to produce
perceptually plausible details coherent with the scene, leading to realistic
shading after virtual object insertion.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.
  * [2] Dan A Calian, Jean-François Lalonde, Paulo Gotardo, Tomas Simon, Iain Matthews, and Kenny Mitchell.  From faces to outdoor light probes.  In Computer Graphics Forum, volume 37, pages 51–61. Wiley Online Library, 2018.
  * [3] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  International Conference on 3D Vision (3DV), 2017.
  * [4] Dachuan Cheng, Jian Shi, Yanyun Chen, Xiaoming Deng, and Xiaopeng. Zhang.  Learning scene illumination by pairwise photos from rear and front mobile cameras.  Computer Graphics Forum, 37(7):213–221, 2018.
  * [5] Paul Debevec.  Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography.  In ACM SIGGRAPH 2008 classes, pages 1–10. 2008.
  * [6] Paul Debevec, Paul Graham, Jay Busch, and Mark Bolas.  A single-shot light probe.  In ACM SIGGRAPH 2012 Talks, pages 1–1. 2012.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [8] David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In Proceedings of the IEEE international conference on computer vision, pages 2650–2658, 2015.
  * [9] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  arXiv preprint arXiv:1406.2283, 2014.
  * [10] Farshad Einabadi, Jean-Yves Guillemaut, and Adrian Hilton.  Deep neural models for illumination estimation and relighting: A survey.  Computer Graphics Forum, 40(6):315–331, 2021.
  * [11] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde.  Learning to predict indoor illumination from a single image.  ACM Trans. Graph., 36(6):176:1–176:14, 2017.
  * [12] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde.  Learning to predict indoor illumination from a single image.  arXiv preprint arXiv:1704.00090, 2017.
  * [13] Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean-François Lalonde.  Fast spatially-varying indoor lighting estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6908–6917, 2019.
  * [14] Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Tinne Tuytelaars, and Luc Van Gool.  What is around the camera?  In Proceedings of the IEEE International Conference on Computer Vision, pages 5170–5178, 2017.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [16] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.
  * [17] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967–5976, 2017.
  * [18] Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem.  Rendering synthetic objects into legacy photographs.  ACM Transactions on Graphics (TOG), 30(6):1–12, 2011.
  * [19] Kevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig, and David Forsyth.  Automatic scene inference for 3d object compositing.  ACM Transactions on Graphics (TOG), 33(3):1–15, 2014.
  * [20] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * [21] Joel Kronander, Francesco Banterle, Andrew Gardner, Ehsan Miandji, and Jonas Unger.  Photorealistic rendering of mixed reality scenes.  Comput. Graph. Forum, 34(2):643–665, may 2015.
  * [22] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In International Conference on 3D Vision (3DV), page 239–248, 2016\.
  * [23] Louis Lettry, Kenneth Vanhoey, and Luc Van Gool.  Darn: a deep adversarial residual network for intrinsic image decomposition.  In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1359–1367. IEEE, 2018.
  * [24] Mengtian Li, Jie Guo, Xiufen Cui, Rui Pan, Yanwen Guo, Chenchen Wang, Piaopiao Yu, and Fei Pan.  Deep spherical gaussian illumination estimation for indoor scene.  In Proceedings of the ACM Multimedia Asia, MMAsia ’19, New York, NY, USA, 2019. Association for Computing Machinery.
  * [25] Yijun Li, Sifei Liu, Jimei Yang, and Ming-Hsuan Yang.  Generative face completion.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3911–3919, 2017.
  * [26] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker.  Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475–2484, 2020.
  * [27] Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang.  Coherent semantic attention for image inpainting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4170–4179, 2019.
  * [28] Stephen Lombardi and Ko Nishino.  Reflectance and illumination recovery in the wild.  IEEE transactions on pattern analysis and machine intelligence, 38(1):129–141, 2015.
  * [29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [30] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [31] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  ArXiv preprint, 2021.
  * [32] Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski.  High dynamic range imaging: acquisition, display, and image-based lighting.  Morgan Kaufmann, 2010.
  * [33] Marcel Santana Santos, Ren Tsang, and Nima Khademi Kalantari.  Single image hdr reconstruction using a cnn with masked features and perceptual loss.  ACM Transactions on Graphics, 39(4), 7 2020.
  * [34] Gowri Somanath and Daniel Kurz.  Hdr environment map estimation for real-time augmented reality.  2021\.
  * [35] Shuran Song and Thomas Funkhouser.  Neural illumination: Lighting prediction for indoor environments.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6918–6926, 2019.
  * [36] Pratul P Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T Barron, Richard Tucker, and Noah Snavely.  Lighthouse: Predicting lighting volumes for spatially-coherent illumination.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8080–8089, 2020.
  * [37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [38] Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz.  Learning indoor inverse rendering with 3d spatially-varying lighting.  In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  * [39] Henrique Weber, Donald Prévost, and Jean-François Lalonde.  Learning to estimate indoor lighting from 3d objects.  In 2018 International Conference on 3D Vision (3DV), pages 199–207. IEEE, 2018.
  * [40] Jianxiong Xiao, Krista A. Ehinger, Aude Oliva, and Antonio Torralba.  Recognizing scene viewpoint using panoramic place representation.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pages 2695–2702. IEEE Computer Society, 2012.
  * [41] Di Xu, Zhen Li, and Yanning Zhang.  Real-time illumination estimation for mixed reality on mobile devices.  In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pages 702–703. IEEE, 2020.
  * [42] Renjiao Yi, Chenyang Zhu, Ping Tan, and Stephen Lin.  Faces as lighting probes via unsupervised deep highlight extraction.  In Proceedings of the European Conference on computer vision (ECCV), pages 317–333, 2018.
  * [43] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505–5514, 2018.
  * [44] Fangneng Zhan, Yingchen Yu, Changgong Zhang, Rongliang Wu, Wenbo Hu, Shijian Lu, Feiying Ma, Xuansong Xie, and Ling Shao.  Gmlight: Lighting estimation via geometric distribution approximation.  IEEE Transactions on Image Processing, 31:2268–2278, 2022.
  * [45] Fangneng Zhan, Changgong Zhang, Wenbo Hu, Shijian Lu, Feiying Ma, Xuansong Xie, and Ling Shao.  Sparse needlets for lighting estimation with spherical transport loss.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12830–12839, October 2021.
  * [46] Fangneng Zhan, Changgong Zhang, Yingchen Yu, Yuan Chang, Shijian Lu, Feiying Ma, and Xuansong Xie.  Emlight: Lighting estimation via spherical distribution approximation.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  * [47] Edward Zhang, Michael F Cohen, and Brian Curless.  Emptying, refurnishing, and relighting indoor spaces.  ACM Transactions on Graphics (TOG), 35(6):1–14, 2016.