# Encoding Syntactic Knowledge in Transformer Encoder
for Intent Detection and Slot Filling

Jixuan Wang1, Kai Wei2, Martin Radfar2, Weiwei Zhang2, Clement Chung2
Work done during author’s internship at Amazon Alexa.Corresponding author.

###### Abstract

We propose a novel Transformer encoder-based architecture with syntactical
knowledge encoded for intent detection and slot filling. Specifically, we
encode syntactic knowledge into the Transformer encoder by jointly training it
to predict syntactic parse ancestors and part-of-speech of each token via
multi-task learning. Our model is based on self-attention and feed-forward
layers and does not require external syntactic information to be available at
inference time. Experiments show that on two benchmark datasets, our models
with only two Transformer encoder layers achieve state-of-the-art results.
Compared to the previously best performed model without pre-training, our
models achieve absolute F1 score and accuracy improvement of $1.59\%$ and
$0.85\%$ for slot filling and intent detection on the SNIPS dataset,
respectively. Our models also achieve absolute F1 score and accuracy
improvement of $0.1\%$ and $0.34\%$ for slot filling and intent detection on
the ATIS dataset, respectively, over the previously best performed model.
Furthermore, the visualization of the self-attention weights illustrates the
benefits of incorporating syntactic information during training.

## Introduction

Recent years have seen great success in applying deep learning approaches to
enhance the capabilities of virtual assistants (VAs) such as Amazon Alexa,
Google Home and Apple Siri. One of the challenges for building these systems
is mapping the meaning of users’ utterances, which are expressed in natural
language, to machine comprehensible language (Allen 1995). An example is
illustrated in Figure 1. In this utterance “ _Show the cheapest flight from
Toronto to St. Louis_ ”, the machine needs to map this utterance to an intent
_Airfare_ (intent detection) and to slots such as _Toronto: FromLocation_
(slot filling). In this work, we focus on intent detection and slot filling
and refer to these as Natural Language Understanding (NLU) tasks.

Previous works show that a simple deep neural architecture delivers better
performance on NLU tasks when compared to traditional models such as
Conditional Random Fields (Collobert et al. 2011). Since then, deep neural
architectures, predominantly recurrent neural networks, have become an
indispensable part of building NLU systems (Zhang and Wang 2016; Goo et al.
2018; E et al. 2019). Transformer-based architectures, as introduced more
recently by (Vaswani et al. 2017), have shown significant improvement over
previous works on NLU tasks (Chen, Zhuo, and Wang 2019; Qin et al. 2019).
Recent studies show that although the Transformer model can learn syntactic
knowledge purely by seeing examples, explicitly feeding this knowledge to such
models can significantly enhance their performance on tasks such as neural
machine translation (Sundararaman et al. 2019) and semantic role labeling
(Strubell et al. 2018). While incorporating syntactic knowledge has been shown
to improve performance for NLU tasks (Tur et al. 2011; Chen et al. 2016), both
of these assume syntactic knowledge is provided by external models during
training and inference time.

Figure 1: An example of NLU tasks.

In this paper, we introduce a novel Transformer encoder- based architecture
for NLU tasks with syntactic knowledge encoded that does not require syntactic
information to be available during inference time. This is accomplished,
first, by training one attention head to predict syntactic ancestors of each
token. The dependency relationship between each token is obtained from
syntactic dependency trees, where each word in a sentence is assigned a
syntactic head that is either another word in the sentence or an artificial
root symbol (Dozat and Manning 2016). Adding the objective of dependency
relationship prediction allows a given token to attend more to its
syntactically relevant parent and ancestors. In addition to dependency parsing
knowledge, we encode part of speech (POS) information in Transformer encoders
because previous research shows that the POS information can help dependency
parsing (Nguyen and Verspoor 2018). The closest work to ours is (Strubell et
al. 2018). However, they focused on semantic role labeling and trained one
attention head to predict the direct parent instead of all ancestors.

We compare our models with several state-of-the-art neural NLU models on two
publicly available benchmarking datasets: the ATIS (Hemphill, Godfrey, and
Doddington 1990) and SNIPS (Coucke et al. 2018) datasets. The results show
that our models outperform previous works. To examine the effects of adding
syntactic information, we conduct an ablation study and visualize the self-
attention weights in the Transformer encoder.

## Problem Definition

We define intent detection (ID) and slot filling (SF) as an utterance-level
and token-level multi-class classification task, respectively. Given an input
utterance with $T$ tokens, we predict an intent $y^{int.}$ and a sequence of
slots, one per token, $\\{y^{slot}_{1},y^{slot}_{2},\dots,y^{slot}_{T}\\}$ as
outputs. We add an empty slot denoted by “O” to represent words containing no
labels. The goal is to maximize the likelihood of correct the intents and
slots given input utterances.

## Proposed Model

We jointly train our model for NLU tasks (i.e., ID and SF), syntactic
dependency prediction and POS tagging via multi-task learning (Caruana 1993),
as shown in Figure 2. For dependency prediction, we insert a syntactically-
informed Transformer encoder layer after the ($x+y$)th layer. In this encoder
layer, one attention head is trained to predict the full ancestry for each
token on the dependency parsing tree. For POS tagging, we add a POS tagging
model that shares the first $x$ Transformer encoder layers with the NLU model.
We describe the details of our proposed architecture below.

### Input Embedding

The input embedding model maps a sequence of token representations
$\\{t_{1},t_{2},\dots,t_{T}\\}$ into a sequence of continuous embeddings
$\\{e_{0},e_{1},e_{2},\dots,e_{T}\\}$, with $e_{0}$ being the embedding of a
special start-of-sentence token, “[SOS]”. The embeddings are then fed into the
NLU model.

Figure 2: A high level overview of the proposed architecture. Note that $x$,
$y$ and $z$ all refer to number of layers that can vary depending on
implementation. “MLP” refers to a multi-layer perceptron (MLP).

### Transformer Encoder Layer

The Transformer encoder layers are originally proposed in (Vaswani et al.
2017). Each encoder layer consists of a multi-head self-attention layer and
feed forward layers with layer normalization and residual connections. We
stack multiple encoder layers to learn contextual embeddings of each token,
each with $H$ attention heads. Suppose the output embeddings of the encoder
layer $j-1$ is $E^{(j-1)}$, each attention head $h$ at layer $j$ first
calculates self-attention weights by the scaled dot product (1).

$A_{h}^{(j)}=softmax(\frac{Q^{(j)}_{h}{K^{(j)}_{h}}^{T}}{\sqrt{d_{k}}})$ (1)

In (1), the query $Q^{(j)}_{h}$ and key $K^{(j)}_{h}$ are two different linear
transformations of $E^{(j-1)}$, $d_{k}$ is the dimension of the query and the
key embeddings. The output of the attention head $h$ is calculated by:

$F_{h}^{(j)}=A_{h}^{(j)}V_{h}^{(j)}$ (2)

in which the value $V_{h}^{(j)}$ is also a linear transformation of
$E^{(j-1)}$. The outputs of $H$ attention heads are concatenated as the self-
attended token representations, followed by another linear transformation:

$Multi^{(j)}=[F_{1}^{(j)},F_{2}^{(j)},\cdots,F_{H}^{(j)}]W^{F}$ (3)

which is fed into the next feed forward layer. Residual connections and layer
normalization are applied after the multi-head attention and feed forward
layer, respectively.

Figure 3: Overview of the syntactically-informed Transformer layer. One out of
the $H$ attention heads is trained for predicting syntactic parse ancestors of
each token. For each token, this attention head outputs a distribution over
all positions in the sentence, which corresponds to the probability of each
token being the ancestor of this token. The loss function is defined as the
mean Kullback–Leibler (KL) divergence between the output distributions of all
tokens and the corresponding prior distributions.

### Encoding Syntactic Dependency Knowledge

As shown in Figure 3, the syntactically-informed transformer encoder layer
differs from the standard Transformer encoder layer by having one of the $H$
attention heads trained to predict the full ancestry for each token, i.e.,
parents, grandparents, great grandparents, etc. Different from (Strubell et
al. 2018), we use full ancestry prediction instead of just direct parent
prediction. Later we will demonstrate the benefits of our approach in the
Results section.

Given an input sequence of length $T$, the output of a regular attention head
is a $T\times T$ matrix, in which each row contains the attention weights that
a token puts on all the tokens in the input sequence. The output of the
syntactically-informed attention head is also a $T\times T$ matrix but this
attention head is trained to assign weights only on the syntactic governors
(i.e., ancestors) of each token.

To train this attention head, we define a loss function by the difference
between the output attention weight matrix of the syntactically-informed
attention head and a predefined prior attention weight matrix. The prior
attention weight matrix contains the prior knowledge that each token should
attend to its syntactic parse ancestors, with attention weights being higher
on ancestors that are closer to that token. During training, we obtain the
prior attention weights based on the outputs of a pre-trained dependency
parser.

For example, in the utterance “list flights arriving in Toronto on March
first”, the syntactic parse ancestors of word “first” are “March”, “arriving”
and “flights” which are $1$, $2$ and $3$ hops away on the dependency tree,
respectively, as shown in Figure 4. The ancestors are syntactically meaningful
for the determination of the slot of “first”, which is “arrive date, day
number” in this case.

Figure 4: Syntactic dependency tree of “list flights arriving in Toronto on
March first” and prior attention weights of word “first”.

To train the attention head to assign higher weights on the ancestors of each
token, we define prior attention weights of each token based on the distance
between the token and its ancestors. Formally, the prior attention weights of
token $i$ are defined as:

$w_{i,j}^{prior}=\left\\{\begin{array}[]{ll}softmax(-d_{i,j}/\tau)&\text{if
$j\in ancestors(i)$}\\\ 0&\text{if $j\notin ancestors(i)$}\end{array}\right.$
(4)

in which $d_{i,j}$ is the distance between token $i$ and $j$, $softmax$ is the
Softmax function, $\tau$ is the temperature of the Softmax function
controlling the variance of the attention weights over all the ancestors and
$i,j\in\\{1,2,\dots,T\\}$. The stack of prior attention weights of all $T$
tokens is a $T\times T$ matrix, denoted by $W^{prior}$. We train our model to
decrease the difference between $W^{prior}$ and the attention matrix
$W^{(s)}_{h}$ output by the attention head $h$ at the $s$th layer. The
difference is measured by the mean of row-wise KL divergence between these two
matrices, which is used as an additional loss function besides the NLU loss
functions. We refer to this loss as dependency loss denoted by
$\mathcal{L}^{dep.}$, formally:

$\displaystyle\mathcal{L}^{dep.}$
$\displaystyle=\frac{1}{T}\sum_{i=1}^{T}D_{KL}(W^{prior}_{i}\;\|\;W^{(s)}_{h,i})$
(5) $\displaystyle W^{(s)}_{h}$
$\displaystyle=softmax(Q^{(s)}U^{dep.}(K^{(s)})^{T})$ (6)

in which $D_{KL}(\cdot{})$ denotes the KL divergence, $Q^{(s)}$ and $K^{(s)}$
are linear transformations of $E^{(s-1)}$, $W^{(s)}_{h,i}$ is the $i$th row of
$W^{(s)}_{h}$, and $U^{dep.}$ is a parameter matrix. In (6) we use the
biaffine attention instead of the scaled dot product attention, which has been
shown to be effective for dependency parsing (Dozat and Manning 2016).

We treat $\tau$ as a hyperparameter and tune it on the validation set. With
$\tau\to 0^{+}$, attention head $h$ will be trained to only pay attention to
the direct parent of each token, a special case used by (Strubell et al.
2018). Thus, our method is a more general approach compared to (Strubell et
al. 2018).

### Encoding Part-of-Speech Knowledge

Part-of-Speech (POS) information is important for disambiguating words with
multiple meanings (Alva and Hegde 2016). This is because an ambiguous word
carries a specific POS in a particular context (Pal, Munshi, and Saha 2015).
For instance, the word “May” could be either a verb or a noun. Being aware of
its POS tag is beneficial for down-stream tasks, such as predicting the slots
in the utterance “book a flight on May 1st”. Furthermore, previous studies
have shown that while models trained for a sufficiently large number of steps
can potentially learn underlying patterns of POS, the knowledge is imperfect
(Jawahar, Sagot, and Seddah 2019; Sundararaman et al. 2019). For these
reasons, we explicitly train our model to perform POS tagging using the POS
tags generated by a pretrained POS tagger.

Similar to slot filling, we simplify POS tagging as a token-level
classification problem. We apply a MLP-based classifier on the output
embeddings of the $r$th transformer encoder layer and use cross entropy as the
loss function:

$\displaystyle p_{i,o}^{pos}$ $\displaystyle=softmax(MLP(e_{r,i}))$ (7)
$\displaystyle\mathcal{L}^{pos}$
$\displaystyle=-\sum_{i=1}^{T}\sum_{o=1}^{O}y_{i,o}^{pos}\log p_{i,o}^{pos}$
(8)

in which $p_{i,o}^{pos}$ is the predicted probability of the $i$th token’s POS
label being the $o$th label in the POS label space, $O$ is the total number of
POS labels, $y_{i,o}^{pos}$ is the one-hot representation of the groundtruth
POS label.

### Intent Detection and Slot Filling

#### Intent detection:

We apply a linear classifier on the embedding of the “[SOS]” token, $e_{L,0}$,
which is output by the last Transformer encoder layer $L$. Cross entropy loss
is used for intent detection. The loss on one utterance is defined as:

$\displaystyle p_{n}^{int.}$
$\displaystyle=softmax(W^{int.}e_{L,0}^{T}+b^{int.})$ (9)
$\displaystyle\mathcal{L}^{int.}$
$\displaystyle=-\sum_{n=1}^{N}y_{n}^{int.}\log p_{n}^{int.}$ (10)

in which $W^{int.}$ and $b^{int.}$ are the parameters of the linear
classifier, $y^{int.}$ is the one-hot representation of the ground truth
intent label, $N$ is the total number of intent labels and $p_{n}^{int.}$ is
the predicted probability of this utterance’s intent label being the $n$th
label in the intent label space.

#### Slot filling:

We apply a MLP-based classifier on the embeddings output by the last
Transformer encoder layer using cross entropy as the loss function. The loss
on one utterance is defined as follow:

$\displaystyle p_{i,s}^{slot}$ $\displaystyle=softmax(MLP(e_{L,i}))$ (11)
$\displaystyle\mathcal{L}^{slot}$
$\displaystyle=-\sum_{i=1}^{T}\sum_{s=1}^{S}y_{i,s}^{slot}\log p_{i,s}^{slot}$
(12)

in which $p_{i,s}^{slot}$ is the predicted probability of the $i$th token’s
slot being the $s$th label in the slot space, $S$ is the total number of
slots, and $y_{i,s}^{slot}$ is the one-hot representations of the ground truth
slot label.

### Multi-task Learning

We train our model via multi-task learning (Caruana 1993). Our loss function
is defined as:

$\mathcal{L}=\mathcal{L}^{NLU}+c^{dep}\cdot\mathcal{L}^{dep}+c^{pos}\cdot\mathcal{L}^{pos}$
(13)

where $\mathcal{L}^{NLU}$ equals to $\mathcal{L}^{slot}$ for slot filling,
$\mathcal{L}^{int.}$ for intent detection or
$\mathcal{L}^{slot}+\mathcal{L}^{int.}$ for joint training, and $c^{dep}$ and
$c^{pos}$ are coefficients of the dependency prediction loss and the POS
tagging loss, respectively. $c^{dep}$ and $c^{pos}$ are treated as
hyperparameters and selected based on validation performance.

| SNIPS |  | ATIS
---|---|---|---
| SF | ID |  | SF | ID-M | ID-S
Joint Seq (Hakkani-Tür et al. 2016) | 87.30 | 96.90 |  | 94.30 | 92.60 | -
Attention-based RNN (Liu and Lane 2016) | 87.80 | 96.70 |  | 95.78 | - | 97.98
Slot-Gated (Goo et al. 2018) | 89.27 | 96.86 |  | 95.42 | 95.41 | -
SF-ID, SF first (E et al. 2019) | 91.43 | 97.43 |  | 95.75 | 97.76 | -
SF-ID, ID first (E et al. 2019) | 92.23 | 97.29 |  | 95.80 | 97.09 | -
Stack-Propagation (Qin et al. 2019) | 94.20 | 98.00 |  | 95.90 | 96.90 | -
Graph LSTM (Zhang et al. 2020) | 95.30 | 98.29 |  | 95.91 | 97.20 | -
TF | 96.37 | 98.29 |  | 95.31 | 96.42 | 97.65
SyntacticTF (Independent) | 96.56 | 98.71 |  | 95.94 | 97.76 | 98.10
SyntacticTF (Joint) | 96.89 | 99.14 |  | 96.01 | 97.31 | 98.32
JointBERT (Chen, Zhuo, and Wang 2019)∗ | 97.00 | 98.60 |  | 96.10 | 97.50 | -
Table 1: SF and ID results on the ATIS and SNIPS dataset (%). TF refers to the
Transformer encoder-based model trained without syntactic information.
SyntacticTF refers to our model. Independent and Joint refer to independently
and jointly training for SF and ID, respectively. ID-M refers to multiple
label matching for intent detection evaluation and ID-S refers to single label
matching. ∗This work relies on pretraining, which is not required by other
works in the table.

## Experiments

### Datasets

We conducted experiments on two benchmark datasets: the Airline Travel
Information Systems (ATIS) (Hemphill, Godfrey, and Doddington 1990) and SNIPS
(Coucke et al. 2018) datasets. The ATIS dataset has a focus on airline
information and has been used as benchmark on NLU tasks. We used the same
version as (Goo et al. 2018; E et al. 2019) that contains 4,478 utterances for
training, 500 for validation and 893 for testing. The SNIPS dataset has a
focus on personal assistant commands, with a larger vocabulary size and more
diverse intents and slots. It contains 13,084 utterances for training, 700 for
validation and 700 for testing.

### Evaluation Metrics

We use classification accuracy for intent detection and the F1 score for slot
filling, which is the harmonic mean of precision and recall. For the SNIPS
dataset, we use the same version and evaluation method as pervious works
(Zhang et al. 2020). For the ATIS dataset, we find that previous works use two
different evaluation methods for intent detection on utterances with multiple
labels. The first method counts a prediction as correct if it is equal to one
of the ground truth labels of the utterance (Liu and Lane 2016). We refer this
method as the single label matching method (ID-S). The second method counts a
prediction as correct only if it matches all labels of the utterance (Goo et
al. 2018; E et al. 2019). We refer this method as the multiple label matching
method (ID-M). We report both in our results.

### Implementation Details

Our experiments are implemented in PyTorch (Paszke et al. 2017). The
hyperparameters are selected based on the performance on the validation set.
We use the Adam optimizer (Kingma and Ba 2015) with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, $\epsilon=10^{-7}$ and the weight decay fix as described in
(Loshchilov and Hutter 2017). Our learning rate schedule first increases the
learning rate linearly from 0 to 0.0005 (warming up) and then decreases it to
0 following the values of the cosine function. We use warming up steps
$\approx 20\%$ of the total training steps. The specific number of warming up
steps is determined by validation performance. We use the implementation of
the optimizer and learning rate scheduler of the Transformers library (Wolf et
al. 2019).

We use Stanza (Qi et al. 2020) to generate training labels for POS tagging and
dependency prediction. For the NLU model trained with both dependency
prediction and POS tagging, $c^{dep}$ and $c^{pos}$ are both set to $1$. For
the NLU model trained with only dependency prediction, $c^{dep}$ is set to
$5$. We used weight decay of 0.1 and dropout rate (Srivastava et al. 2014) of
0.1 and 0.3 for the SNIPS and ATIS dataset, respectively. We use batch size of
32 and train each model for $100$ epochs. We report the testing results of the
checkpoints achieving the best validation performance.

We use the concatenation of GloVe embeddings (Pennington, Socher, and Manning
2014) and character embeddings (Hashimoto et al. 2017) as token embeddings and
keep them frozen during training. The hidden dimension of the Transformer
encoder layer is 768 and the size of feed forward layer is 3072. Considering
the small size of the two datasets, we only use two Transformer encoder layers
in total (with $x=1$, $y=0$ and $z=0$ as in Figure 2), each of which has $4$
attention heads. The dimension of $Q^{(s)}$ and $K^{(s)}$ is $200$. For slot
filing, we apply the Viterbi decoding at test time. BIO is a standard format
for slot filling annotation schema, as shown in Figure 1. The transition
probabilities are manually set to ensure the output sequences of BIO labels to
be valid, by simply specifying the probabilities of invalid transition to zero
and the probabilities of valid transition to one.

### Baseline Models

We compare our proposed model with the following baseline models:

  * •

Joint Seq (Hakkani-Tür et al. 2016) is a joint model for intent detection and
slot filling based on the bi-directional LSTM model.

  * •

Attention-based RNN (Liu and Lane 2016) is a sequence-to-sequence model with
the attention mechanism.

  * •

Slot-Gated (Goo et al. 2018) utilizes intent information for slot filling
through the gating mechanism.

  * •

SF-ID (E et al. 2019) is an architecture that enables the interaction between
intent detection and slot filling.

  * •

Stack-Propagation (Qin et al. 2019) is a joint model based on the Stack-
Propagation framework.

  * •

Graph LSTM (Zhang et al. 2020) is based on the Graph LSTM model.

  * •

JointBERT (Chen, Zhuo, and Wang 2019) is a joint NLU model fined tuned from
the pretrained BERT model (Devlin et al. 2018).

  * •

TF is the Transformer encoder-based model trained without syntactic
information.

## Results

Table 1 shows the performance of the baseline and proposed models for SF and
ID on the SNIPS and ATIS dataset, respectively. Overall, our proposed models
achieve the best performance on the two benchmarking datasets. On the SNIPS
dataset, our proposed joint model achieves an absolute F1 score and accuracy
improvement of $1.59\%$ and $0.85\%$ for SF and ID, respectively, compared to
the best performed baseline model without pre-training (Zhang et al. 2020). On
the ATIS dataset, our proposed joint model also achieves an absolute F1 score
and accuracy improvement of $0.1\%$ and $0.34\%$ for SF and ID-S, compared to
the best performed baseline model for SF (Zhang et al. 2020) and ID-S (Liu and
Lane 2016), respectively. In addition, our proposed independent model achieves
the same performance as the best performed baseline model on ID-M (E et al.
2019, SF-ID, SF first).

Besides, the model based on Transformer encoder without syntactic knowledge
can achieve SOTA results on the SNIPS dataset and is slightly worse than the
SOTA results on the ATIS dataset. This indicates the powerfulness of the
Transformer encoder for SF and ID. Moreover, the further improvement of our
models over the baseline models demonstrates the benefits of incorporating
syntactic knowledge. Additionally, compared to previous works with
heterogenous model structures, our models are purely based on self-attention
and feed forward layers.

We also find that our proposed models can outperform the JointBERT model with
pre-training (Chen, Zhuo, and Wang 2019) for intent detection tasks. Compared
to the JointBERT model, our proposed joint model achieves an absolute accuracy
improvement of $0.54\%$ for ID on the SNIPS dataset; and our proposed
independent model achieves an absolute accuracy improvement of $0.26\%$ for
ID-M on the ATIS dataset. While our proposed model does not outperform the
JointBERT model for SF, the performance gap is relatively small ($~{}0.11\%$
on SNIPS and $~{}0.09\%$ on ATIS). It should be noted that our model does not
require pre-training and the size of our model is only one seventh of the
JointBERT model (16 million vs. 110 million parameters).

Previous works have shown that models like BERT can learn syntactic knowledge
by self-supervision (Clark et al. 2019; Manning et al. 2020). This can
partially explain why the JointBERT can achieve very good results without
being fed with syntactic knowledge explicitly.

### Ablation Study

Table 2 shows the ablation study results of the effects of adding different
syntactic information. A first observation is that the model trained with a
singe syntactic task, either dependency prediction or POS tagging, outperforms
the baseline Transformer encoder-based model without syntactic information.
This gives us confidence that syntactic information can help improve model
performance. Moreover, training a Transformer model with both the syntactic
tasks achieves even better results than training with a single syntactic task.
This could be because the POS tagging task improves the performance of the
dependency prediction task (Nguyen and Verspoor 2018), which in turn improves
the performance of SF and ID.

| SNIPS |  | ATIS
---|---|---|---
| SF | ID |  | SF | ID-M | ID-S
TF | 96.37 | 98.29 |  | 95.31 | 96.42 | 97.65
TF + D | 96.31 | 98.43 |  | 95.99 | 96.53 | 98.76
TF + P | 96.47 | 98.57 |  | 95.82 | 97.31 | 98.10
TF + D + P | 96.56 | 98.71 |  | 95.94 | 97.76 | 98.10
Table 2: Results of ablation study. TF refers to the baseline models with two
Transformer encoder layers. D and P refers to dependency prediction and POS
tagging, respectively.

Interestingly, we observe that the addition of dependency prediction reduces
the performance of slot filling on the SNIPS dataset ($96.31\%$) when compared
to the baseline Transformer encoder-based model ($96.37\%$). There are several
potential reasons. Firstly, the sentences in the SNIPS dataset are overall
shorter than the ATIS dataset so that the syntactic dependency information
might be less helpful. Secondly, previous work has shown that syntactic
parsing performance often suffers when a named entity span has crossing
brackets with the spans on the parse tree (Finkel and Manning 2009). Thus, the
dependency prediction performance of our model might decrease due to the
presence of many name entities in the SNIPS dataset, such as song names and
movie names, which could introduce noisy dependency information into the
attention weights and degrade the performance on the NLU tasks.

### Qualitative Analysis

We qualitatively examined the errors made by the Transformer encoder-based
models with and without syntactic information to understand in what ways
syntactic information helps improve the performance. Our major findings are:

ID errors related to preposition with nouns: Prepositions, when appearing
between nouns, are used to describe their relationship. For example, in the
utterance “kansas city to atlanta monday morning flights”, the preposition
“to” denotes the direction from “kansas city” (departure location, noun) to
“atlanta” (arrival location, noun). Without this knowledge, a model could
misclassify the intent of this utterance as asking for city information rather
than flight information. We found that about $50\%$ of the errors made by the
model without syntactic information contain this pattern, whereas less than
$10\%$ of the misclassified utterances contain this pattern for the model with
syntactic information (See Appendix A for the full list).

SF errors due to POS confusion: A Word can have multiple meanings depending on
context. For example, the same word “may” can be a verb expressing
possibility, or as a noun referring to the fifth month of the year. We found
that correctly recognizing the POS of words could potentially help reduce slot
filling errors. For example, in this utterance “May I have the movie schedules
for Speakeasy Theaters”, the slot for “May” should be empty, but the model
without syntactic information predicts it as “Time Range”. By contrast, the
model with syntactic information predicts correctly for this word, probably
because the confusion of noun vs. verb for the word “May” is addressed by
incorporating POS information. More examples are included in Appendix A.

| ATIS |  | SNIPS
---|---|---|---
| SF | ID |  | SF | ID-S
TF + Par. | 96.20 | 98.29 |  | 95.58 | 98.10
TF + Anc. | 96.31 | 98.43 |  | 95.99 | 98.76
Table 3: Intent detection and slot filling results of Transformer (TF)
encoder-based models with dependency parent prediction (Par.) and dependency
ancestor prediction (Anc.) on the ATIS and SNIPS dataset.

### Parent Prediction vs. Ancestor Prediction

We compare our approach of predicting all ancestors of each token with the
approach described in (Strubell et al. 2018), which only predicts direct
dependency parent of each token. Results in Table 3 show that the model with
our approach can achieve better results for both ID and SF on the two
datasets, which demonstrates that our approach is more beneficial to the NLU
tasks. We hypothesize that incorporating syntactic ancestor prediction can
better capture long-distance syntactic relationship. As shown in (Tur,
Hakkani-Tür, and Heck 2010), long distance dependencies are important for slot
filling. For example, in the utterance “Find flights to LA arriving in no
later than next Monday”, a 6-gram context is needed to figure out that
“Monday” is the arrival date instead of the departure date.

Figure 5: Visualization of the attention weights of the model with and without
syntactic supervision for slot filling. $L_{i}$ and $A_{j}$ stands for the
$i$th Transformer layer and $j$th attention head, respectively. The attention
head inside the red-dotted box is trained for dependency prediction.

### Visualization of Attention Weights

We visualize the attention weights output by models trained with and without
syntactic information to understand what the models have learned by
incorporating syntactic information. We select the utterance “show me the
flights on american airlines which go from st. petersburg to ontario
california by way of st. louis” from the ATIS testing set. Only the model
trained with syntactic information predicts the slot labels correctly. As
shown in Figure 5, the model without syntactic information has simple
attention patterns on both layers, such as looking backward and looking
forward. Other attention heads seem to be random and less informative.

In contrast, the model with syntactic information has more informative
attention patterns. On the first layer, all the attention heads present simple
but diverse patterns. Besides looking forward and backwards, the second
attention head looks at both directions for each token. On the second layer,
however, we observe more complex patterns and long-distance attention which
could account for more task-oriented operations. Therefore, it is possible
that the Transformer encoder learns attention weights better with syntactic
information supervision so that the encoder can leave more power for the end
task.

## Related Work

Research on intent detection and slot filling emerged in the $1990s$ from the
call classification systems (Gorin, Riccardi, and Wright 1997) and the ATIS
project (Price 1990). Early work has primarily focused on using a traditional
machine learning classifier such as CRFs (Haffner, Tur, and Wright 2003).
Recently, there has been an increasing application of neural models on NLU
tasks. These approaches, primarily based on RNNs, have shown that neural
approaches outperform traditional models (Mesnil et al. 2014; Tur et al. 2012;
Zhang and Wang 2016; Goo et al. 2018; E et al. 2019). For example, Mesnil et
al (2015) employed RNNs for slot filling and found an $2.3\%$ relative
improvement of F1 compared to CRF (Mesnil et al. 2014). Some works also
explored Transformer encoder and graph LSTM-based neural architectures (Chen,
Zhuo, and Wang 2019; Zhang et al. 2020).

Syntactic information has been shown to be beneficial to many tasks, such as
neural machine translation (Akoury, Krishna, and Iyyer 2019), semantic role
labeling (Strubell et al. 2018), and machine reading comprehension (Zhang et
al. 2020). Research on NLU tasks has also shown that incorporating syntactic
information into machine learning models can help improve the performance.
Moschitti et al. (2007) used syntactic information for slot filling, where the
authors used a tree kernel function to encode the structural information
acquired by a syntactic parser. An extensive analysis on the ATIS dataset
revealed that most NLU errors are caused by complex syntactic characteristics,
such as prepositional phrases and long distance dependencies (Tur, Hakkani-
Tür, and Heck 2010). Tur et al. (2011) proposed a rule-based dependency
parsing based sentence simplification method to augment the input utterances
based on the syntactic structure. Compared to previous works, our work is the
first to encode syntactical knowledge into end-to-end neural models for intent
detection and slot filling.

## Conclusion

In this paper, we propose to encode syntactic knowledge into the Transformer
encoder-based model for intent detection and slot filling. Experimental
results indicate that a model with only two Transformer encoder layers can
already match or even outperform the SOTA performance on two benchmark
datasets. Moreover, we show that the performance of this baseline model can be
further improved by incorporating syntactical supervision. The visualization
of the attention weights also reveals that syntactical supervision can help
the model to better learn syntactically-related patterns. For future work, we
will evaluate our approach with larger model sizes on larger scale datasets
containing more syntactically complex utterances. Furthermore, we will
investigate incorporating syntactic knowledge into models pretrained by self-
supervision and applying those models on the NLU tasks.

## Acknowledgement

We would like to thank Siegfried Kunzmann, Nathan Susanj, Ross McGowan, and
anonymous reviewers for their insightful feedback that greatly improved our
paper.

## References

  * Akoury, Krishna, and Iyyer (2019) Akoury, N.; Krishna, K.; and Iyyer, M. 2019.  Syntactically supervised transformers for faster neural machine translation.  _arXiv preprint arXiv:1906.02780_ .
  * Allen (1995) Allen, J. 1995.  _Natural Language Understanding (2nd Ed.)_.  USA: Benjamin-Cummings Publishing Co., Inc.  ISBN 0805303340.
  * Alva and Hegde (2016) Alva, P.; and Hegde, V. 2016.  Hidden markov model for pos tagging in word sense disambiguation.  In _2016 International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS)_ , 279–284. IEEE.
  * Caruana (1993) Caruana, R. 1993.  Multitask learning: a knowledge-based source of inductive bias.  In _ICML’93 Proceedings of the Tenth International Conference on International Conference on Machine Learning_ , 41–48.
  * Chen, Zhuo, and Wang (2019) Chen, Q.; Zhuo, Z.; and Wang, W. 2019.  BERT for Joint Intent Classification and Slot Filling.  _arXiv preprint arXiv:1902.10909_ .
  * Chen et al. (2016) Chen, Y.-N.; Hakanni-Tür, D.; Tur, G.; Celikyilmaz, A.; Guo, J.; and Deng, L. 2016.  Syntax or semantics? knowledge-guided joint semantic frame parsing.  In _2016 IEEE Spoken Language Technology Workshop (SLT)_ , 348–355. IEEE.
  * Clark et al. (2019) Clark, K.; Khandelwal, U.; Levy, O.; and Manning, C. D. 2019.  What does bert look at? an analysis of bert’s attention.  _arXiv preprint arXiv:1906.04341_ .
  * Collobert et al. (2011) Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011.  Natural language processing (almost) from scratch.  _Journal of machine learning research_ 12(ARTICLE): 2493–2537.
  * Coucke et al. (2018) Coucke, A.; Saade, A.; Ball, A.; Bluche, T.; Caulier, A.; Leroy, D.; Doumouro, C.; Gisselbrecht, T.; Caltagirone, F.; Lavril, T.; et al. 2018.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _arXiv preprint arXiv:1805.10190_ .
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_ .
  * Dozat and Manning (2016) Dozat, T.; and Manning, C. D. 2016.  Deep biaffine attention for neural dependency parsing.  _arXiv preprint arXiv:1611.01734_ .
  * E et al. (2019) E, H.; Niu, P.; Chen, Z.; and Song, M. 2019.  A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling.  In _ACL 2019 : The 57th Annual Meeting of the Association for Computational Linguistics_ , 5467–5471.
  * Finkel and Manning (2009) Finkel, J. R.; and Manning, C. D. 2009.  Joint parsing and named entity recognition.  In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , 326–334.
  * Goo et al. (2018) Goo, C.-W.; Gao, G.; Hsu, Y.-K.; Huo, C.-L.; Chen, T.-C.; Hsu, K.-W.; and Chen, Y.-N. 2018.  Slot-gated Modeling for Joint Slot Filling and Intent Prediction.  In _NAACL HLT 2018: 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , volume 2, 753–757.
  * Gorin, Riccardi, and Wright (1997) Gorin, A. L.; Riccardi, G.; and Wright, J. H. 1997.  How may I help you?  _Speech communication_ 23(1-2): 113–127.
  * Haffner, Tur, and Wright (2003) Haffner, P.; Tur, G.; and Wright, J. H. 2003.  Optimizing SVMs for complex call classification.  In _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)._ , volume 1, I–I. IEEE.
  * Hakkani-Tür et al. (2016) Hakkani-Tür, D.; Tür, G.; Çelikyilmaz, A.; Chen, Y.-N.; Gao, J.; Deng, L.; and Wang, Y.-Y. 2016.  Multi-Domain Joint Semantic Frame Parsing Using Bi-Directional RNN-LSTM.  In _Interspeech 2016_ , 715–719.
  * Hashimoto et al. (2017) Hashimoto, K.; Xiong, C.; Tsuruoka, Y.; and Socher, R. 2017.  A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 1923–1933.
  * Hemphill, Godfrey, and Doddington (1990) Hemphill, C. T.; Godfrey, J. J.; and Doddington, G. R. 1990.  The ATIS spoken language systems pilot corpus.  In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990_.
  * Jawahar, Sagot, and Seddah (2019) Jawahar, G.; Sagot, B.; and Seddah, D. 2019.  What does BERT learn about the structure of language?
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. L. 2015.  Adam: A Method for Stochastic Optimization.  In _ICLR 2015 : International Conference on Learning Representations 2015_.
  * Liu and Lane (2016) Liu, B.; and Lane, I. 2016.  Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling.  In _Interspeech 2016_ , 685–689.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ .
  * Manning et al. (2020) Manning, C. D.; Clark, K.; Hewitt, J.; Khandelwal, U.; and Levy, O. 2020.  Emergent linguistic structure in artificial neural networks trained by self-supervision.  _Proceedings of the National Academy of Sciences_ .
  * Mesnil et al. (2014) Mesnil, G.; Dauphin, Y.; Yao, K.; Bengio, Y.; Deng, L.; Hakkani-Tur, D.; He, X.; Heck, L.; Tur, G.; Yu, D.; et al. 2014.  Using recurrent neural networks for slot filling in spoken language understanding.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 23(3): 530–539.
  * Moschitti, Riccardi, and Raymond (2007) Moschitti, A.; Riccardi, G.; and Raymond, C. 2007.  Spoken language understanding with kernels for syntactic/semantic structures.  In _2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU)_, 183–188. IEEE.
  * Nguyen and Verspoor (2018) Nguyen, D. Q.; and Verspoor, K. 2018.  An improved neural network model for joint POS tagging and dependency parsing.  _arXiv preprint arXiv:1807.03955_ .
  * Pal, Munshi, and Saha (2015) Pal, A. R.; Munshi, A.; and Saha, D. 2015.  An Approach to Speed-up the Word Sense Disambiguation Procedure through Sense Filtering.  _arXiv preprint arXiv:1610.06601_ .
  * Paszke et al. (2017) Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic differentiation in PyTorch.  In _NIPS-W_.
  * Pennington, Socher, and Manning (2014) Pennington, J.; Socher, R.; and Manning, C. D. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 1532–1543.
  * Price (1990) Price, P. 1990.  Evaluation of spoken language systems: The ATIS domain.  In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990_.
  * Qi et al. (2020) Qi, P.; Zhang, Y.; Zhang, Y.; Bolton, J.; and Manning, C. D. 2020.  Stanza: A Python Natural Language Processing Toolkit for Many Human Languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
  * Qin et al. (2019) Qin, L.; Che, W.; Li, Y.; Wen, H.; and Liu, T. 2019.  A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding.  In _2019 Conference on Empirical Methods in Natural Language Processing_ , 2078–2087.
  * Srivastava et al. (2014) Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ 15(1): 1929–1958.
  * Strubell et al. (2018) Strubell, E.; Verga, P.; Andor, D.; Weiss, D.; and McCallum, A. 2018.  Linguistically-Informed Self-Attention for Semantic Role Labeling.  In _EMNLP 2018: 2018 Conference on Empirical Methods in Natural Language Processing_ , 5027–5038.
  * Sundararaman et al. (2019) Sundararaman, D.; Subramanian, V.; Wang, G.; Si, S.; Shen, D.; Wang, D.; and Carin, L. 2019.  Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding.  _arXiv preprint arXiv:1911.06156_ .
  * Tur et al. (2012) Tur, G.; Deng, L.; Hakkani-Tür, D.; and He, X. 2012.  Towards deeper understanding: Deep convex networks for semantic utterance classification.  In _2012 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 5045–5048. IEEE.
  * Tur, Hakkani-Tür, and Heck (2010) Tur, G.; Hakkani-Tür, D.; and Heck, L. 2010.  What is left to be understood in ATIS?  In _2010 IEEE Spoken Language Technology Workshop_ , 19–24. IEEE.
  * Tur et al. (2011) Tur, G.; Hakkani-Tur, D.; Heck, L.; and Parthasarathy, S. 2011.  Sentence simplification for spoken language understanding.  In _2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 5628–5631.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All You Need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , 5998–6008.
  * Wolf et al. (2019) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  _ArXiv_ abs/1910.03771.
  * Zhang et al. (2020) Zhang, L.; Ma, D.; Zhang, X.; Yan, X.; and Wang, H. 2020.  Graph LSTM with Context-Gated Mechanism for Spoken Language Understanding.  _AAAI 2020 : The Thirty-Fourth AAAI Conference on Artificial Intelligence_ 34(5): 9539–9546.
  * Zhang and Wang (2016) Zhang, X.; and Wang, H. 2016.  A joint model of intent determination and slot filling for spoken language understanding.  In _IJCAI_ , volume 16, 2993–2999.
  * Zhang et al. (2020) Zhang, Z.; Wu, Y.; Zhou, J.; Duan, S.; Zhao, H.; and Wang, R. 2020.  SG-Net: Syntax-Guided Machine Reading Comprehension.  In _AAAI_ , 9636–9643.

## Appendix A

Below lists examples of the intent detection errors made by the model without
syntactic information that are related to one specific grammar pattern between
prepositions and nouns.

  * •

cleveland to kansas city arrive monday before 3 pm

  * •

kansas city to atlanta monday morning flights

  * •

new york city to las vegas and memphis to las vegas on Sunday

Below lists examples of the slot filling errors made by the model without
syntactic information that contain POS confusion.

  * •

cleveland to kansas city arrive monday before 3 pm

  * •

new york city to las vegas and memphis to las vegas on Sunday

  * •

baltimore to kansas city economy

The Transformer encoder-based model without syntactic information made
mistakes on all these utterances. The model trained with POS tagging and the
model trained with both POS tagging and dependency prediction fail on the last
utterance in the list below. The model trained with dependency prediction does
not make any mistakes on all these utterances. We underline the words that are
assigned to wrong slots by the model without syntactic information.

  * •

book a reservation for velma an a and rebecca for an american pizzeria at
(correct: $B-TimeRange$; prediction: $B-RestaurantName$) $5$ Am in MA

  * •

Where is Belgium located (correct: $Other$; prediction: $B-PatialRelation$)

  * •

May(correct: $Other$; prediction: $B-TimeRange$) I have the movie schedules
for Speakeasy Theaters


11institutetext: Université Laval, Québec, Canada 22institutetext: Depix,
Montréal, Canada
https://lvsn.github.io/EditableIndoorLight/

# Editable indoor lighting estimation

Henrique Weber 11 0000-0001-7103-7220 Mathieu Garon 22 0000-0003-1811-4156
Jean-François Lalonde 11 0000-0002-6583-2364

###### Abstract

We present a method for estimating lighting from a single perspective image of
an indoor scene. Previous methods for predicting indoor illumination usually
focus on either simple, parametric lighting that lack realism, or on richer
representations that are difficult or even impossible to understand or modify
after prediction. We propose a pipeline that estimates a parametric light that
is easy to edit and allows renderings with strong shadows, alongside with a
non-parametric texture with high-frequency information necessary for realistic
rendering of specular objects. Once estimated, the predictions obtained with
our model are interpretable and can easily be modified by an artist/user with
a few mouse clicks. Quantitative and qualitative results show that our
approach makes indoor lighting estimation easier to handle by a casual user,
while still producing competitive results.

###### Keywords:

lighting estimation, virtual object insertion, HDR

##  1 Introduction

Mixing virtual content realistically with real imagery is required in an
increasing range of applications, from special effects to image editing and
augmented reality (AR). This has created the need for capturing the lighting
conditions of a scene with ever increasing accuracy and flexibility. In his
seminal work, Debevec [6] suggested to capture the lighting conditions with a
high dynamic range light probe. While it has been improved over the years,
this technique, dubbed _image-based lighting_ , is still at the heart of
lighting capture for high end special effects in movies nowadays111See
https://www.fxguide.com/fxfeatured/the-definitive-weta-digital-guide-to-ibl/..
Since the democratization of virtual object insertion for consumer image
editing and AR, capturing light conditions with light probes restricts non
professional users to have access to the scene and to use specialized
equipment. To circumvent those limitations, approaches for automatically
estimating the lighting conditions directly from images have been proposed.

In this line of work, the trend has been to estimate more and more _complex_
lighting representations. This is exemplified by works such as Lighthouse
[25], which propose to learn a multi-scale volumetric representation from an
input stereo pair. Similarly, Li et al. [19] learn a dense 2D grid of
spherical gaussians over the image plane. Wang et al. [27] propose to learn a
3D volume of similar spherical gaussians. While these lighting representations
have been shown to yield realistic and spatially-varying relighting results,
they have the unfortunate downside of being hard to understand: they do not
lend themselves to being easily editable by a user. This quickly becomes a
source of limitation when erroneous automatic results need to be corrected for
improved accuracy or when creative freedom is required.

Figure 1: Our method produces an estimation of the indoor lighting from a
single perspective image. Our lighting representation is composed of a 3D
parametric light source, a texture map and a coarse 3D layout of the scene.
With this information, it is possible to realistically insert 3D objects (like
the golden armadillo and sphere) into the scene. Because our lighting
representation is interpretable and intuitive, the user can experiment with
possibilities by modifying, say, the position of the light source in order to
achieve the desired look.

In this work, we depart from this trend and propose a _simple, interpretable,
and editable_ lighting representation (fig. 1). But what does it mean for a
lighting representation to be editable? We argue that an editable lighting
representation must: 1) disentangle various components of illumination; 2)
allow an intuitive control over those components; and, of course, 3) enable
realistic relighting results. Existing lighting representations in the
literature do not possess all three properties. _Environment maps_ [11, 24,
17] can be rotated but they compound light sources and environment textures
together such that one cannot, say, easily increase the intensity of the light
source without affecting everything else. Rotating the environment map
inevitably rotates the entire scene, turning walls into ceilings, etc., when
changing the elevation. _Dense and/or volumetric_ representations [12, 19, 25,
27] are composed of 2D (or 3D) grids containing hundreds of parameters, which
would have to be modified in a consistent way to achieve the desired result,
an unachievable task for most. _Parametric_ representations [10] model
individual light sources with a few intuitive parameters, which can be
modified independently of the others, but cannot generate realistic
reflections.

Our proposed representation is the first to offer all three desired properties
and is composed of two parts: 1) a parametric light source for modeling
shading in high dynamic range; and 2) a non-parametric texture to generate
realistic reflections off of shiny objects. Our representation builds on the
hypothesis (which we validate) that most indoor scenes can accurately be
modeled by a _single_ , dominant directional light source. We model this in
high dynamic range with a parametric representation [10] that explicitly
models the light source intensity, size, and 3D position. This representation
is intuitive and can easily be edited by a user simply by moving the light
source around in 3D.

This light source is complemented with a spatially-varying environment map
texture, mapped onto a coarse 3D representation of the indoor scene. For this,
we rely on a layout estimation network, which estimates a cuboid-like model of
the scene from the input image. In addition, we also use a texture estimation
network, whose output is conditioned on a combination of the input image, the
scene layout and the parametric lighting representation. By explicitly tying
the appearance of the environment texture with the position of the parametric
light source, modifying the light source parameters (e.g. moving around the
light) will automatically adjust the environment in a realistic fashion.

While our representation is significantly simplified, we find that it offers
several advantages over the previous approaches. First, it renders both
realistic shading (due to the high dynamic range of the estimated parametric
light) and reflections (due to the estimated environment map texture). Second,
it can efficiently be trained on real images, thereby alleviating any domain
gap that typically arise when approaches need synthetic imagery for training
[25, 19, 27]. Third—and perhaps most importantly—it is _interpretable and
editable_. Since all automatic approaches are bound to make mistakes, it is of
paramount importance in many scenarios that their output be adjustable by a
user. By modifying the light parameters and/or the scene layout using simple
user interfaces, our approach bridges the gap between realism and editability
for lighting estimation.

##  2 Related work

For succinctness, we focus on single-image indoor lighting estimation methods
in the section below, and refer the reader to the recent survey on deep models
for lighting estimation for a broader overview [8].

####  2.0.1 Lighting estimation

Gardner et al. [11] proposed the first deep learning-based lighting estimation
method for indoor scenes, and predicted an HDR environment map
(equirectangular image) from a single image. This representation was also used
in [17] for both indoors and outdoors, in [24] to take into account the object
insertion position, in [23] which presented a real-time on-device approach, in
[22] for scene decomposition, and in [3] which exploited the front and back
cameras in current mobile devices. Finally, [28] propose to learn the space of
indoor lighting using environment maps on single objects.

Other works explored alternative representations, such as spherical harmonics
[12, 20, 34] that are useful for real-time rendering but are typically
unsuitable for modeling high-frequency lighting (such as bright light sources)
and are not ideal for non diffuse object rendering. [10] proposed to estimate
a set of 3 parametric lights, which can easily be edited. However, that
representation cannot generate realistic reflections. EMlight [33] propose a
more expressive model by predicting gaussians on a spherical model. Similar to
us, GMlight [31] back-projects the spherical gaussians to an estimated 3D
model of the scene. This is further extended in [1] by the use of graph neural
networks, and in [32] through the use of spherical wavelets dubbed “needlets”.

Recently, methods have attempted to learn volumetric lighting representations
from images. Of note, Lighthouse [25] learns multi-scale volumetric lighting
from a stereo pair, [19] predicts a dense 2D grid of spherical gaussians which
is further extended into a 3D volumetric representation by Wang et al. [27].
While these yield convincing spatially-varying results, these representations
cannot easily be interacted by a user.

####  2.0.2 Scene decomposition

Holistic scene decomposition [2] is deeply tied to lighting estimation as both
are required to invert the image formation process. Li et al. [19] proposes to
extract the scene geometry and the lighting simultaneously. Similarly, [7]
extract only the geometry of the scene by estimating the normal and depth of
the scene. These geometric representations are however non-parametric and thus
difficult to edit or comprehend. [16] proposes a simplified parametric model
where a room layout is recovered in the camera field of view. Similarly, [35]
presents a method to estimate the layout given a panoramic image of an indoor
scene. We use the method of [16] to estimate a panoramic layout given a
perspective image, thus providing a simple cuboid representation that allows
for spatially varying textured lighting representation.

##  3 Editable indoor lighting representation

We begin by presenting our hybrid parametric/non-parametric lighting
representation which aims at bridging the gap between realism and editability.
We also show how that representation can be fitted to high dynamic range
panoramas to obtain a training dataset, and conclude by presenting how it can
be used for virtual object relighting.

###  3.1 Lighting representation

Our proposed light representation, shown in fig. 2, is composed of two main
components: an HDR parametric light source $\mathbf{p}$; and an LDR textured
cuboid $\mathcal{C}$.

####  3.1.1 Light source

As in [10], the light source parameters $\mathbf{p}$ are defined as

$\mathbf{p}=\\{\mathbf{l},d,s,\mathbf{c},\mathbf{a}\\}\,,$ (1)

where $\mathbf{l}\in\mathbb{R}^{3}$ is a unit vector specifying the light
direction in XYZ coordinates, $d$ is the distance in meters, $s$ the radius
(in meters), $\mathbf{c},\mathbf{a}\in\mathbb{R}^{3}$ are the light source and
ambient colors in RGB, respectively. Here, $\mathbf{l}$, $d$ and $s$ are
defined with respect to the camera. In contrast with [10], we use a single
light source.

####  3.1.2 Textured cuboid

The cuboid $\mathcal{C}=\\{\mathbf{T},\mathbf{L}\\}$ is represented by a
texture $\mathbf{T}\in\mathbb{R}^{2H\times H\times 3}$, which is an RGB
spherical image of resolution $2H\times H$ stored in equirectangular
(latitude-longitude) format, and a scene layout
$\mathbf{L}\in\mathbb{R}^{2H\times H}$. The layout is a binary image of the
same resolution, also in equirectangular format, indicating the intersections
of the main planar surfaces in the room (walls, floor, ceiling) as an edge map
[9].

Figure 2: To render a virtual object with our proposed lighting
representation, the texture is first warped according to the layout (1st
column), producing a textured mesh (2nd). This mesh is combined with an
emitting sphere representing the parametric light (3rd) for rendering. The
resulting rendering (4th) closely matches the ground truth rendering obtained
with the HDR environment map (last).

###  3.2 Ground truth dataset

The ground truth is derived from the Laval Indoor HDR Dataset [11], which
contains 2,100 HDR panoramas (with approximate depth labels from [10]). We
extract $\mathbf{p}$ and $\mathcal{C}$ from each panorama using the following
procedure. First, the HDR panorama is clipped to LDR (we re-expose such that
the 90th-percentile is 0.8 then clip to [0, 1]) and directly used as the
texture $\mathbf{T}$. Then the intersection between the main surfaces are
manually labelled to define the layout $\mathbf{L}$. Lastly, we extract a
dominant parametric light source from the HDR panorama. In order to determine
the main light source, the $N=5$ brightest individual light sources are first
detected using the region-growing procedure in [10]. A test scene (9 diffuse
spheres arranged in a $3\times 3$ grid on a diffuse ground plane, seen from
top as in fig. 4b) is rendered with each light source independently by masking
out all other pixels—the brightest render determines the strongest light
source.

An initial estimate of the light parameters $\mathbf{p}$ are obtained by the
following. The distance $d$ is approximated by using the average depth of the
region, direction $\mathbf{l}$ as the region centroid, the angular size from
the major and minor axes of an ellipse fitted to the same region. Finally, the
light color $\mathbf{c}$ and ambient term $\mathbf{a}$ are initialized with a
least-squares fit to a rendering of the test scene using the HDR panorama.
From the initial parameters, $\mathbf{p}$ is further refined:

$\mathbf{p}^{*}=\operatorname*{arg\,min}_{\mathbf{p}}||\mathcal{R}(\mathbf{p})-\mathcal{R}(\tilde{\mathbf{P}})||_{2}\,.$
(2)

$\mathcal{R}(x)$ is a differentiable rendering operator (implemented with
Redner [18]) that renders a test scene using $\mathbf{p}$. The optimization is
performed using gradient descent with Adam [15]. Finally, the texture map
$\mathbf{T}$ is rescaled with the estimated ambient term $\mathbf{a}^{*}$ to
ensure that the texture yields the same average RGB color.

###  3.3 Virtual object rendering

To render a virtual object using our lighting representation, we employ the
Cycles rendering engine222Available within Blender at
https://www.blender.org.. A scene, as shown in fig. 2, is composed of a 3D
emissive sphere for the parametric light $\mathbf{p}$ and the textured cuboid
mesh $\mathcal{C}$. The cuboid mesh is derived by detecting the cuboid corners
from the layout using high pass filters. We use the following geometric
constraints to simplify the back-projection of the scene corners to 3D. First,
the shape is limited to a cuboid, meaning that opposing faces are parallel.
Second, the panorama layouts were trained using a camera elevation of
$0^{\circ}$ (pointing at the horizon) and height of 1.6 meter above the
ground. Using these constraints, the bottom corners can easily be projected on
the ground plane, and the top corners can be used to compute the ceiling
height (averaged from the 4 corners). A texture map can then be computed using
every planar surfaces of the cuboid. Finally, the parametric light and the
texture are rendered in two rendering passes. After rendering, the relit
virtual object can be composited into the image using differential rendering
[6].

##  4 Approach

Figure 3: Our method takes as input a perspective, RGB image and its scene
layout representation, passes the RGB to a CNN to predict a parametric light,
and passes the partial layout to another CNN to predict the full panorama
layout. The parametric light is converted to a binary mask panorama, which is
then sent together with the full layout prediction and the input RGB image to
a third network which outputs an LDR texture with the light at the desired
location.

Our approach, illustrated in fig. 3, is composed of three main networks:
light, layout, and texture which are combined together to estimate our light
representation (c.f., sec. 3) from an image. We assume that the layout of the
input image is available, in practice this is obtained with an off-the-shelf
solution [30].

####  4.0.1 Light network

A “light” network is trained to learn the mapping from input image
$\mathbf{I}\in\mathbb{R}^{128\times 128\times 3}$ to estimated lighting
parameters $\mathbf{p}$ (sec. 3) using a similar approach to [10].
Specifically, the light network is composed of a headless DenseNet-121 encoder
[14] to produce a 2048-dimensional latent vector, followed by a fully-
connected layer (512 units), and ultimately with an output layer producing the
light source parameters $\mathbf{p}$.

The light network is trained on light parameters fitted on panoramas from the
Laval Indoor HDR Dataset [11] using the procedure described in sec. 3.2. To
generate the input image from the panorama, we follow [11] and extract
rectified crops from the HDR panoramas. The resulting images are converted to
LDR by re-exposing to make the median intensity equal to 0.45, clipping to 1,
and applying a $\gamma=\nicefrac{{1}}{{2.4}}$ tonemapping. The same exposure
factor is subsequently applied to the color $\mathbf{c}$ and ambient $a$ light
parameters to ensure consistency. Note that the training process is
significantly simplified compared to [11] as the network predicts only a
single set of parameters.

We employ individual loss functions on each of the parameters independently:
L2 for direction $\mathbf{l}$, depth $d$, size $s$, and ambient color $a$, and
L1 for light color $\mathbf{c}$. In addition, we also employ an angular loss
for both the ambient and light colors $a$ and $\mathbf{c}$ to enforce color
consistency. The weights for each term were obtained through a Bayesian
optimization on the validation set (see supp. mat.).

####  4.0.2 Layout network

The mapping from the input RGB image $\mathbf{I}$ and its layout (obtained
with [30]) to the estimated scene layout $\hat{\mathbf{L}}$ (sec. 3) is
learned by the “layout” network whose architecture is that of pix2pixHD [26].
Both inputs are concatenated channel-wise. The layout network is trained on
both the Laval and the Zillow Indoor Dataset [5], which contains 67,448 LDR
indoor panoramas of 1575 unfurnished residences along with their scene
layouts. To train the network, a combination of GAN, feature matching and
perceptual losses are employed [26]. The same default weights as in [26] are
used in training.

####  4.0.3 Texture network

Finally, the estimated environment texture $\hat{\mathbf{T}}$ is predicted by
a “texture” network whose architecture is also that of pix2pixHD [26]. It
accepts as input a channel-wise concatenation of three images: the input RGB
image $\mathbf{I}$, the estimated light parameters $\hat{\mathbf{p}}$
projected in an equirectangular format, and the estimated scene layout
$\hat{\mathbf{L}}$. The equirectangular images are vertically concatenated to
the input image. Note that the $\hat{\mathbf{p}}$ projection is performed
using a subset of all parameters (direction $\mathbf{l}$ and size $s$ only).

The texture network is also trained on both Laval and Zillow datasets. To
obtain the required light source position from the Zillow dataset, we detect
the largest connected component whose intensity is above the 98th percentile
over the upper half of the panorama. To convert the Laval HDR panoramas to
LDR, first a scale factor is found such as the crop taken from that panorama
has its 90th percentile mapped to 0.8. This scale factor is then applied to
the panorama such as its scale matches the one of the crop. The texture
network is trained with the same combination of losses as the layout network.

---
(a)  | 25th prct. | 50th prct. | 75th prct.
---|---|---|---
|  |  |
GT |  |  |
Ours |  |  |
(b)

Figure 4: Validation of our 1-light approximation. LABEL:sub@subfig:strlight-
fig Cumulative distribution of the contribution of the single strongest light
with respect to the entire lighting environment of the scene.
LABEL:sub@subfig:strlight-images Example images for different percentiles,
where the rows correspond to the environment map (top), a synthetic scene
(seen from the top) rendered with (middle) the ground truth environment map
and (bottom) our 1-light representation. As expected, scenes where the
strongest light does not contribute significantly have shadows that are less
pronounced which may point to several light sources equally contributing to
the overall energy (25th prct.). The strongest light source contributes to
more than 80% of the total energy in at least 50% of the images in our test
set, which confirms our assumption that most scenes can accurately be modeled
with a single light source.

##  5 Experiments

###  5.1 Validation of our 1-light approximation

We test our hypothesis that most indoor scenes are well-approximated by a
single dominant light source with an ambient term. We render a scene with the
ground truth environment map, and compare it with the renders obtained from
the parametric lighting optimization procedure described in sec. 3. Fig. 4a
shows the cumulative distribution of the contribution of the strongest light
with respect to the entire lighting of the scene. Note that the strongest
light source contributes to more than 95%/80%/60% of the total lighting for
25%/50%/75% of the images in our test set. Fig. 4b shows example images for
each of these scenarios. We find that even if we expect indoor scenes to have
multiple light sources, the vast majority can accurately be represented by a
single _dominant_ light.

Table 1: Quantitative comparative metrics on (left) renderings of a diffuse
scene, and (right) on the estimated environment maps directly. Each row is
color-coded as best and second best. We also highlight the methods which
produce lighting representations that can be interpreted and edited by a user
(“Edit.”).

| si-RMSE↓ | RMSE↓ | RGB ang.↓ | PSNR↑
---|---|---|---|---
Ours | 0.081 | 0.209 | 4.13∘ | 12.79
Gardner’19 (1) [10] | 0.099 | 0.229 | 4.43∘ | 12.25
Gardner’19 (3) [10] | 0.105 | 0.508 | 4.58∘ | 10.87
Gardner’17 [11] | 0.123 | 0.628 | 8.29∘ | 10.24
Garon’19 [12] | 0.096 | 0.254 | 8.04∘ | 9.70
Lighthouse [25] | 0.120 | 0.253 | 14.53∘ | 9.88
EMLight [33] | 0.099 | 0.232 | 3.99∘ | 10.38
EnvmapNet333Only their proposed ClusterID loss and tonemapping. [23] | 0.097 | 0.286 | 7.67∘ | 11.74
FID↓
---
89.58
356.8
335.6
254.8
314.9
195.5
121.09
201.20

Edit.
---
yes
yes
yes
no
no
no
no
no

| Input | GT |  [10] (3) | [11] | [12] | [25] | [33] | Ours
---|---|---|---|---|---|---|---|---
1st |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  | 0.03, 7.26∘ | 0.52, 2.96∘ | 0.02, 1.86∘ | 0.24, 1.30∘ | 0.03, 2.58∘ | 0.03, 6.70∘
25th |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  | 0.71, 3.53∘ | 1.27, 9.08∘ | 0.31, 8.89∘ | 0.11, 13.3∘ | 0.33, 5.52∘ | 0.11, 1.73∘
50th |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  | 0.27, 14.5∘ | 0.13, 16.2∘ | 0.24, 15.4∘ | 0.15, 16.3∘ | 0.19, 13.6∘ | 0.20, 9.65∘
75th |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  | 0.56, 2.25∘ | 0.35, 9.18∘ | 0.67, 8.90∘ | 0.40, 17.9∘ | 0.55, 1.93∘ | 0.39, 3.33∘
Figure 5: Qualitative lighting estimation examples from our test set. To
compare the estimated lighting, we render a simple scene composed of three
spheres (diffuse, mirror, glossy) on a diffuse ground plane with different
methods. From left to right: input image, ground truth lighting, Gardner’19
[10] (3 lights), Gardner’17 [11], Garon’19 [12], Lighthouse [25], EMLight
[33], and ours. The second row shows the corresponding estimated lighting in
equirectangular format (reprojected in the center of the image for the
spatially-varying techniques such as [10, 12, 25] and ours). Finally, error
metrics (RMSE and RGB angular) are also shown below each example for
reference. Each group shows examples from different error percentiles for our
method according to the RMSE metric. More examples can be found in the
supplementary materials.

###  5.2 Light estimation comparison

We now evaluate our method and compare it with recent state-of-the-art light
estimation approaches. We first validate that our model performs better on
quantitative metrics evaluated on physic-based renders of a scene using a test
set provided by [11]. For each of the 224 panoramas in the test split, we
extract 10 images using the same sampling distribution as in [11], for a total
of 2,240 images for evaluation. We also show renders in various scenes to
demonstrate how our solution is visually more appealing.

####  5.2.1 Quantitative comparison

To evaluate the lighting estimates, we render a test scene composed of an
array of spheres viewed from above (sec. 3) and compute error metrics on the
resulting rendering when compared to the ground truth obtained with the
original HDR panorama. We report RMSE, si-RMSE [13], PSNR, and RGB angular
error [17]. We also compute the FID444Implementation taken from
https://pypi.org/project/pytorch-fid/. on the resulting environment maps to
evaluate the realism of reflections (similar to [23]).

We evaluate against the following works. First, two versions of [10] are
compared: the original (3) where 3 light sources are estimated, and a version
(1) trained to predict a single parametric light. Second, we also compare to
Lighthouse [25], which expects a stereo pair as input. As a substitute, we
generate a second image with a small baseline using Synsin [29] (visual
inspection confirmed this yields reasonable results). For [12], we select the
coordinates of the image center for the object position. For [23], we
implemented their proposed “Cluster ID loss” and tonemapping (eq. 1 in [23])
but used pix2pixHD as backbone. Finally, we also compare against [33]. Results
for each metrics are reported in tab. 1, which shows that despite our model
being simple, it achieves the best score in every metric. We argue that it is
_because_ of its simplicity that we can achieve competitive results. Our
approach can be trained on real data (as opposed to [12, 25] and does not
require an elaborate 2-stage training process (compared to [10]). We also
demonstrate a significantly lower FID score than other methods thus bridging
the gap between representation realism and HDR accuracy.

####  5.2.2 Qualitative comparison

We also present qualitative results in fig. 5 where predictions are rendered
on 3 spheres with varying reflectance properties (diffuse, mirror, glossy). In
addition, a tonemapped equirectangular view of the estimated light
representation is provided under each render. We show an example from each
error percentiles according to the RMSE metric. Our proposed method is
perceptually better on the mirror spheres as other methods do not model high
frequency details from the scene. We also notice accurate shadow and shading
from all the spheres. We show objects realistically composited into
photographs in fig. 6. Note how the reflections of the virtual objects and
their cast shadows on the ground plane perceptually match the input
photograph. Finally, we also compare against [27] in fig. 7.

Input image |  |  |
---|---|---|---
Texture $\hat{\mathbf{T}}$ |  |  |
Rendered objects |  |  |
Figure 6: Virtual object insertion in scenes with our estimated lighting. For
simplicity, we assume the scene surrounding the objects is made of a flat
ground plane, which catches shadows and is placed manually by an artist (the
focus of our work being lighting estimation). For example, the figure shows a
golden armadillo and sphere inserted into three different scenes. Note how the
reflections on the objects and the shadows cast on the ground plans appear
realistic.

---
(a) Input image

---
(b) Wang et al.

---
(c) Ours

Figure 7: Qualitative comparison against Wang et al. [27] |  Layout from [30] | No layout |  Layout from [30] | No layout
---|---|---|---|---
Image |  |  |  |
Image layout |  |  |  |
Texture $\hat{\mathbf{T}}$ |  |  |  |
Layout $\hat{\mathbf{L}}$ |  |  |  |
Figure 8: Ablation on input image layout. We compare the output of our method
(last two rows) as a function of whether or not it is given the estimated
layout of the input image (with [30]). Our approach produces similar results
in both cases.

###  5.3 Ablation study on input layout

One may consider that requiring the image layout as input may make our method
sensitive to its estimation. To show this is not the case, we perform an
experiment where we provide a black layout as input to the layout network
(equivalent to no layout estimate). As can be seen in fig. 8, providing a
black layout as input simply results in a different layout prediction where
the texture still remains coherent with the RGB input and estimated light
direction. The FID of the generated panoramas with no input layout is 88.68
(compared to 89.58 from tab. 1), showing that this essentially has no impact.

###  5.4 Ablation study on the texture network

We also tested different configurations for the texture network in order to
validate our design choices. More specifically, we trained the texture network
providing as input: (1) only the RGB crop (FID of 167.39), (2) RGB crop and
parametric light (FID of 97.13), and (3) RGB crop and layout (FID of 151.04).
In contrast, our full approach obtained an FID of 89.57 (see tab. 1).

##  6 Editing the estimated lighting

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
(a) Light azimuth

|
---|---
|
|
(b) Light elevation

|
---|---
|
|
(c) Light size

|
---|---
|
|
(d) Scene layout

Figure 9: Using our representation, a user can easily edit the estimated light
parameters and obtain relighting results consistent with their edits. For
example, the user can change the LABEL:sub@subfig:edit-azimuth azimuth and
LABEL:sub@subfig:edit-elevation elevation angles of the light source;
LABEL:sub@subfig:edit-size the size of the light source; or
LABEL:sub@subfig:edit-layout the layout of the scene. For all scenarios, we
show rendered virtual objects in the first row, the estimated texture
$\hat{\mathbf{T}}$ in the second, and the representation being edited in the
last (light parameters $\hat{\mathbf{p}}$ for LABEL:sub@subfig:edit-
azimuth–LABEL:sub@subfig:edit-size and layout $\hat{\mathbf{L}}$ for
LABEL:sub@subfig:edit-layout).

Because of its intuitive nature, it is simple and natural for a user to edit
our estimated lighting representation, should the estimate not perfectly match
the background image or simply for artistic purposes. Fig. 9 shows that our
approach simultaneously disentangles various components of illumination,
allows an intuitive control over those components, and enables realistic
relighting results. First, fig. 9a shows that a user can rotate the light
source about its azimuth angle. Note how the estimated texture (second row) is
consistent with the desired light position (third row), while preserving the
same overall structure. The renders (first row) exhibit realistic reflections
and shadows that correspond to the desired lighting directions. A similar
behaviour can be observed in figs 9b and 9c when the elevation angle and size
are modified, respectively. In fig. 9d, we show that it is also possible to
edit the scene layout and obtain an estimated texture map $\hat{\mathbf{T}}$
that is consistent with the users request. We also show results of compositing
virtual objects directly into a scene in fig. 6. As shown in fig. 1, realistic
rendering results can intuitively be edited to achieve the desired look. To
the best of our knowledge, the only other method which allows intuitive
editing of indoor lighting estimate is that of Gardner et al. [10].
Unfortunately, realistic renders are limited to diffuse objects and cannot be
extended to reflective objects as shown in fig. 5.

##  7 Discussion

This paper proposes a lighting estimation approach which produces an
intuitive, user-editable lighting representation given a single indoor input
image. By explicitly representing the dominant light source using a parametric
model, and the ambient environment map using a textured cuboid, our approach
bridges the gap between generating realistic shading (produced by HDR light
sources) and reflections (produced by textured environment maps) on rendered
virtual objects. We demonstrate, through extensive experiments, that our
approach provides competitive quantitative performance when compared to recent
lighting estimation techniques. In particular, when compared to the only other
approach which can be user-edited [10], our approach yields significant
improved results.

##### Limitations and future work

While our proposed approach estimates a 3D representation of the surrounding
lighting environment, it does not reason about light occlusions in the scene
as opposed to other techniques such as [12, 19, 27]. Incorporating these
higher-order interactions while maintaining interpretability and editability
of the output representation is an interesting direction for future research.
In addition, the estimated environment textures were shown to produce
realistic reflections on shiny objects, but a close inspection reveals that
they are low resolution and contain some visual artifacts. It is likely that
more recent image-to-image translation architectures [4, 21] could be used to
improve realism.

##### Acknowledgements

This research was supported by MITACS and the NSERC grant RGPIN-2020-04799.
The authors thank Pascal Audet for his help.

## References

  * [1] Bai, J., Guo, J., Wan, C., Chen, Z., He, Z., Yang, S., Yu, P., Zhang, Y., Guo, Y.: Deep graph learning for spatially-varying indoor lighting prediction. arXiv preprint arXiv:2202.06300 (2022)
  * [2] Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE TPAMI 37(8), 1670–1687 (2014)
  * [3] Cheng, D., Shi, J., Chen, Y., Deng, X., Zhang, X.: Learning scene illumination by pairwise photos from rear and front mobile cameras. Computer Graphics Forum 37(7), 213–221 (2018)
  * [4] Choi, Y., Uh, Y., Yoo, J., Ha, J.W.: Stargan v2: Diverse image synthesis for multiple domains. In: CVPR (2020)
  * [5] Cruz, S., Hutchcroft, W., Li, Y., Khosravan, N., Boyadzhiev, I., Kang, S.B.: Zillow indoor dataset: Annotated floor plans with 360º panoramas and 3d room layouts. In: CVPR (2021)
  * [6] Debevec, P.: Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. pp. 189–198. SIGGRAPH (1998)
  * [7] Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision. pp. 2650–2658 (2015)
  * [8] Einabadi, F., Guillemaut, J.Y., Hilton, A.: Deep neural models for illumination estimation and relighting: A survey. Computer Graphics Forum 40(6), 315–331 (2021)
  * [9] Fernandez-Labrador, C., Facil, J.M., Perez-Yus, A., Demonceaux, C., Civera, J., Guerrero, J.J.: Corners for layout: End-to-end layout recovery from 360 images. IEEE Robotics and Automation Letters 5(2), 1255–1262 (2020)
  * [10] Gardner, M.A., Hold-Geoffroy, Y., Sunkavalli, K., Gagne, C., Lalonde, J.F.: Deep parametric indoor lighting estimation. In: ICCV (2019)
  * [11] Gardner, M.A., Sunkavalli, K., Yumer, E., Shen, X., Gambaretto, E., Gagné, C., Lalonde, J.F.: Learning to predict indoor illumination from a single image. ACM TOG 36(6) (2017)
  * [12] Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: CVPR (2019)
  * [13] Grosse, R., Johnson, M.K., Adelson, E.H., Freeman, W.T.: Ground truth dataset and baseline evaluations for intrinsic image algorithms. In: 2009 IEEE 12th International Conference on Computer Vision. pp. 2335–2342. IEEE (2009)
  * [14] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017)
  * [15] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [16] Lee, C.Y., Badrinarayanan, V., Malisiewicz, T., Rabinovich, A.: Roomnet: End-to-end room layout estimation. In: ICCV (2017)
  * [17] LeGendre, C., Ma, W.C., Fyffe, G., Flynn, J., Charbonnel, L., Busch, J., Debevec, P.: Deeplight: Learning illumination for unconstrained mobile mixed reality. In: CVPR (2019)
  * [18] Li, T.M., Aittala, M., Durand, F., Lehtinen, J.: Differentiable monte carlo ray tracing through edge sampling. ACM TOG 37(6), 1–11 (2018)
  * [19] Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In: CVPR (2020)
  * [20] Mandl, D., Yi, K.M., Mohr, P., Roth, P., Fua, P., Lepetit, V., Schmalstieg, D., Kalkofen, D.: Learning lightprobes for mixed reality illumination. In: ISMAR (2017)
  * [21] Park, T., Efros, A.A., Zhang, R., Zhu, J.Y.: Contrastive learning for unpaired image-to-image translation. In: ECCV (2020)
  * [22] Sengupta, S., Gu, J., Kim, K., Liu, G., Jacobs, D.W., Kautz, J.: Neural inverse rendering of an indoor scene from a single image. In: ICCV (2019)
  * [23] Somanath, G., Kurz, D.: Hdr environment map estimation for real-time augmented reality. In: CVPR (2021)
  * [24] Song, S., Funkhouser, T.: Neural illumination: Lighting prediction for indoor environments. In: CVPR (2019)
  * [25] Srinivasan, P.P., Mildenhall, B., Tancik, M., Barron, J.T., Tucker, R., Snavely, N.: Lighthouse: Predicting lighting volumes for spatially-coherent illumination. In: CVPR (2020)
  * [26] Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional gans. In: CVPR (2018)
  * [27] Wang, Z., Philion, J., Fidler, S., Kautz, J.: Learning indoor inverse rendering with 3d spatially-varying lighting. In: ICCV (2021)
  * [28] Weber, H., Prévost, D., Lalonde, J.F.: Learning to estimate indoor lighting from 3d objects. In: 3DV (2018)
  * [29] Wiles, O., Gkioxari, G., Szeliski, R., Johnson, J.: Synsin: End-to-end view synthesis from a single image. In: CVPR (2020)
  * [30] Yang, C., Zheng, J., Dai, X., Tang, R., Ma, Y., Yuan, X.: Learning to reconstruct 3d non-cuboid room layout from a single rgb image. In: Wint. Conf. App. Comp. Vis. (2022)
  * [31] Zhan, F., Yu, Y., Wu, R., Zhang, C., Lu, S., Shao, L., Ma, F., Xie, X.: Gmlight: Lighting estimation via geometric distribution approximation. IEEE TIP (2022)
  * [32] Zhan, F., Zhang, C., Hu, W., Lu, S., Ma, F., Xie, X., Shao, L.: Sparse needlets for lighting estimation with spherical transport loss. In: ICCV (2021)
  * [33] Zhan, F., Zhang, C., Yu, Y., Chang, Y., Lu, S., Ma, F., Xie, X.: Emlight: Lighting estimation via spherical distribution approximation. In: AAAI (2021)
  * [34] Zhao, Y., Guo, T.: Pointar: Efficient lighting estimation for mobile augmented reality. In: ECCV (2020)
  * [35] Zou, C., Colburn, A., Shan, Q., Hoiem, D.: Layoutnet: Reconstructing the 3d room layout from a single rgb image. In: CVPR (2018)