# Hierarchical Reinforcement Learning for Furniture Layout in Virtual Indoor
Scenes

Xinhan Di
Bloo Company
Shanghai,China
<EMAIL_ADDRESS>
&Pengqian Yu
Sea Lab
Singapore
<EMAIL_ADDRESS>

###### Abstract

In real life, the decoration of 3D indoor scenes through designing furniture
layout provides a rich experience for people. In this paper, we explore the
furniture layout task as a Markov decision process (MDP) in virtual reality,
which is solved by hierarchical reinforcement learning (HRL). The goal is to
produce a proper two-furniture layout in the virtual reality of the indoor
scenes. In particular, we first design a simulation environment and introduce
the HRL formulation for a two-furniture layout. We then apply a hierarchical
actor-critic algorithm with curriculum learning to solve the MDP. We conduct
our experiments on a large-scale real-world interior layout dataset that
contains industrial designs from professional designers. Our numerical results
demonstrate that the proposed model yields higher-quality layouts as compared
with the state-of-art models.

##  1 Introduction

Indoors such as the bedroom, living room, office, and gym are of great
importance in people’s life. Function, beauty, cost, and comfort are the keys
to the redecoration of indoor scenes. Many online virtual interior tools are
developed to help people design indoor spaces in graphics simulation. Such
virtual simulation provides convenience and a rich experience to customers.
Moreover, machine learning researchers make use of the virtual tools to train
data-hungry models for the auto layout [4, 10], including a variety of
generative models [8, 16, 25, 32]. However, they rely on the external inputs
from interior designers to a large extent and thus are not practical in real-
world applications. Therefore, an automatic redecoration method for 3D virtual
indoor scenes is in demand.

Data-hungry methods for synthesizing indoor graphic scenes simulations through
the layout of furniture have been well studied. Early work in the scene
modeling implement kernels and graph walks to retrieve objects from a database
[3, 5]. The graphical models are employed to model the compatibility between
furniture and input sketches of scenes [35]. Besides, an image-based CNN
network has been proposed to encode top-down views of input scenes [32]. A
variational auto-encoder is also applied to learn similar simulations [38, 15,
12]. Furthermore, the simulation of indoor graphic scenes in the form of tree-
structured scene graphs is investigated [16, 32, 31]. Transformers are applied
to play the role of encoder and decoder for the generation of indoor scene
[33, 23]. However, as shown in Figure 1. this family of models produces
inaccurate size and position for the furniture layout as pointed out in [6,
7],

Figure 1: An example of two-furniture layout with the state-of-the-art models.
The first row represents the ground truth layout in the simulation and 3D
render. The second row represents the layouts produced by five state-of-the-
art models: PlanIT [31], predicts incorrect size and position of the two
furniture. Similarly, LayoutGAN [15] and LayoutVAE [13] predict incorrect size
of position. Besides, both SRL2D [6] and SRL3D [7] can only move one
furniture.

In [6, 7], reinforcement learning (RL; [30, 9, 26, 28]) approaches are
proposed to replace the supervised interaction of human designers with the
interaction of agents in a designed simulation environment. Specifically, the
industrial interior design process [6, 7] in the simulated graphic scenes is
modeled as a Markov decision process (MDP; [24]), where an agent needs to make
multiple decisions for furniture layout in the 3D space by trial and error. To
solve the MDP, there are prominent algorithms such as DQN [22] that learns an
optimal policy for discrete action space, DDPG [17] and PPO [27] that train an
agent for continuous action space, and A3C [21] designed for a large-scale
computer cluster. These algorithms solve stumbling blocks in the application
of reinforcement learning in the simulated indoor graphic scenes.

While the RL approaches proposed in [6, 7] greatly improve the accuracy of the
furniture layout, they are not practical as only one furniture is considered
in the planning as shown in Figure 1. Multiple furniture layout remains
challenging as one needs to take care of the relative positioning of the
furniture. To address this challenge, we formulate the two-furniture layout
task as a MDP with hierarchy by defining the hierarchical state, action, and
reward function. We then design the simulated environment and deploy
hierarchical RL agents to learn the optimal layout for this MDP as illustrated
in Figure 2. In particular, we propose to decompose this MDP task through
hierarchical training where each furniture layout is considered as a lower-
level policy. This hierarchy of nested policies can then be found using the
hierarchical actor-critic (HAC) algorithm [14].

We highlight our two main contributions. First, we formulate the two-furniture
interior graphic scenes design task as a Markov decision process problem with
hierarchy. Second, we develop an indoor graphic scenes simulator and deploy
the hierarchical RL agents which work jointly to learn the two-furniture
layout in the simulated graphic scenes. The developed simulator and agents are
available at https://github.com/CODE-SUBMIT/simulator3.

Figure 2: An example of two-furniture layout in the virtual indoor scenes. The
first row shows the 3D graphic scenes of a room. The second row represents the
projected 2D rooms. The third and fourth rows show the movement of two
furniture (red and yellow objects). The last row represents the final 3D two-
furniture layout in the virtual indoor scenes.

##  2 Background and Related Work

###  2.1 Virtual Reality Applications

There are a variety of recent work on virtual reality applications in the
field of sports, business, entertainment, and education. To name a few,
antithetic sampling is developed for Monte Carlo differentiable rendering of
games in the entertainment field [36]; control strategies are explored for
physically simulated characters performing two-player competitive sports [34];
full-body avatars are built through learning-based method for virtual social
life [2]; a visible difference predictor is developed for field-of-view video
to enlarge the FOV of video in virtual cameras [2]; mixture of volumetric
primitives is applied through efficient neural rendering for fast rendering in
virtual games [19]; deep video realistic 3D human character model is developed
to display highly realistic shape, motion and dynamic appearance for virtual
social life [18]; neural network is trained for 3D human motion capturing with
physical awareness [11]; fast, high-performance, large-scale non-linear least-
squares solvers are proposed for efficient calculation in the process of
rendering, local mapping [20], and neural manipulation synthesis with a hand-
object spatial representation is developed for convenient interactive method
[37]. However, to the best of our knowledge, current methods about the
application of virtual reality for indoor furniture layout are far from being
practical.

###  2.2 Indoor Scenes Synthesis

Our work is related to data-hungry methods for synthesizing indoor graphic
scenes simulations through the layout of furniture. Early work in the scene
modeling implements kernels to retrieve objects from a database [3, 5]. The
graphical models are employed to model the compatibility between furniture and
input sketches of scenes [35]. However, these early methods are mostly limited
by the size of the scene. It is therefore hard to produce a good quality
layout for a large scene size. With the availability of large scene datasets
including SUNCG [29], more sophisticated learning methods are proposed as we
review them below.

###  2.3 Image CNN Networks

The image-based CNN network is proposed to encode top-down views of input
scenes [32]. A variational auto-encoder is applied to learn similar
simulations [38, 15, 12], where each column of the 3D arrangement matrix
represents an object with location and geometry attributes [38]. A
semantically-enriched image-based representation is learned from the top-down
views of the indoor scenes, and convolutional object placement prior is
trained [32]. Transformers are applied to build the encoder and decoder based
on attention mechanism [33, 23]. They are used for indoor scene generation
with a variety of category of furniture. However, this family of image CNN
networks does not apply to the situation where the layout is different with
different dimensional sizes for a single type of room.

###  2.4 Reinforcement Learning for Furniture Layout

Authors in [6] introduce reinforcement learning to model and solve the
furniture layout problem. This prior work simplifies the decision-making
paradigm in the 3D interior furniture layout task as a MDP problem in a 2D
simulation environment. Further work on exploring the 3D layout task is
conducted by building two agents for separate two 2D spaces [7]. However, this
simplification is only applicable for a single furniture. In this paper, we
extend the problem to a much higher dimensional setting where RL agents need
to manage two furniture at the same time.

##  3 Problem Formulation

###  3.1 Markov Decision Process

Similar to previous work of applying reinforcement learning for the furniture
layout [6, 7], we are interested in solving a Markov decision process (MDP)
augmented with a goal state $G$. A MDP is defined as a tuple
$(S,G,A,T,R,\gamma)$, where $S$ is the set of states, $G$ is the goal, $A$ is
the set of actions, $T$ is the transition probability function in which
$T(s,a,s^{\prime})$ is the probability of transitioning to state
$s^{\prime}\in S$ when action $a\in A$ is taken in state $s\in S$, $R_{t}$ is
the reward function at time step $t$, and $\gamma\in[0,1)$ is the discount
factor. The solution to a MDP is in the form of a control policy
$\pi:S,G\rightarrow A$ that maximizes the value function
$V_{\pi}(s,g):=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}R_{t}|s_{0}=s,g=G]$
for given initial state $s_{0}$ and goal $g$.

###  3.2 Reinforcement Learning Formulation

In the two-furniture layout task, the purpose is to efficiently learn a
$2$-level hierarchy $\pi$ consisting of $2$ individual policies $\pi_{0}$ and
$\pi_{1}$ in parallel. Each $\pi_{i}$ aims to guide a furniture to its ideal
position. This deterministic policy $\pi_{i}:S_{i},G_{i}\rightarrow
A_{i},i={1,2}$ solves the corresponding MDP
$(S_{i},G_{i},A_{i},T_{i},R_{i},\gamma_{i})$. Here for each level
$i\in\\{1,2\\}$, the state space is common $S_{i}=S$.

####  3.2.1 State

In this paper, the state space $S$ is defined as the geometrical locations of
elements such as walls, windows, doors and furniture in the simulation graphic
scenes. Each state in the state space consists of the size $s=(x_{s},y_{s})$
and the position $p=(x_{p},y_{p})$ of each element as shown in Figure 3.

Figure 3: Simulation environment for the layout of furniture in the 3D virtual
home. The 3D graphic scenes can be decomposed into a 2D simulation scene with
components including walls, windows, doors and furniture. For each component,
the size and position of each element are represented by its width $w=x_{s}$,
height $h=y_{s}$, and center $(x,y)=(x_{p},y_{p})$ of the corresponding box.

####  3.2.2 Action

We define the discrete action spaces $A=(A_{1},A_{2})$ where $A_{1}$ specifies
moving up or down and $A_{2}$ specifies moving towards left or right,
respectively. They jointly describe how the furniture should move to the
correct position in each state. As illustrated in Figure 2, agent $1$ works on
the first furniture, and agent $2$ works on the second furniture. Here, moving
left denotes the furniture moving towards the $x$-negative direction, while
moving right denotes the furniture moving towards the $x$-positive direction.
In addition, moving up implies the furniture moving towards the $y$-positive
direction, while moving down implies the furniture moving towards the
$y$-negative direction.

####  3.2.3 Goal and Reward

The goals $G_{1}$ and $G_{2}$ are defined as the correct positions of
furniture in $S_{1}$ and $S_{2}$. The reward function is designed to encourage
the furniture to move towards the correct position. For an agent $i$, the
reward function has the following form:

$R^{i}:=\text{IoU}(f^{i}_{\text{target}},f^{i}_{\text{state}}),\quad\forall
i\in\\{1,2\\},$

where $f^{i}_{\text{target}}$ and $f^{i}_{\text{state}}$ represent the agent’s
ground truth and the current layouts for the furniture in the indoor scenes.
Here IoU computes the intersection over union between $f^{i}_{\text{state}}$
and $f^{i}_{\text{target}}$.

##  4 Hierarchical Actor-critic Framework and Curriculum Learning

Figure 4: The reinforcement learning hierarchy for two-furniture layout in
indoor virtual reality. The 3D home is decomposed into 2D scenes with
components. Agent $1$ learns a policy of moving furniture $1$ (red object),
and agent $2$ learns a policy of moving furniture $2$ (yellow object)
simultaneously.

Hierarchical actor-critic (HAC) module [14], as shown in Figure 4, is proposed
to solve the task of two furniture layout. It overcomes the instability issue
that arises when agents try to jointly learn multiple levels of policies in
two aspects: $(1)$ a specific hierarchical architecture and $(2)$ a method for
learning the different levels of the hierarchy simultaneously and
independently. HAC uses nested policies and hindsight transitions, which we
discuss in the following.

###  4.1 Nested Policies

HAC module is to learn hierarchies of nested policies. Nesting is critical to
decompose problems because it enables agents to learn tasks requiring long
sequences of primitive actions. HAC nests policies through embedding the
policy at level $i-1$, $\pi_{i-1}$ into the transition function at level $i$
as $T_{i}$.

The transition function at each sub-goal level, $T_{i},i>0$ is supposed to
work as follows. The sub-goal action $a_{i}$ is assigned to be the goal of
level $i-1$, that is $G_{i-1}=a_{i}$, $\pi_{i-1}$ then has at most $H$
attempts to achieve $G_{i-1}$. Here, $H$ is a parameter given by the user. The
transition function terminates and the agent’s current state is returned when
either $\pi_{i-1}$ runs out of $H$ attempts or a goal $G_{n},n>i-2$ is
achieved. Therefore, level $i$’s state transition function $T_{i}$ depends on
the full policy hierarchy below level $i$, $\pi_{i-1}$ as from the hierarchy’s
nested architecture. Similarly, each action from $pi_{i-1}$ depends on
$T_{i-1}$, which depends on $\pi_{i-2}$ and so on. Therefore, the notation
$T_{i}|\pi_{i-1}$ is used for level $i$’s state transition function for level
$i>0$. The notation $T_{0}$ is used to name the base transition function
$T_{0}$, and is provided by the task $T_{0}=T$.

###  4.2 Hindsight Transitions

In order to learn multiple policies in parallel, both hindsight action
transitions and hindsight goal transitions are applied. The hindsight action
and hindsight goal are applied to overcome non-stationary issues which hinder
the joint learning of policies.

Firstly, instead of training each sub-goal policy, we assume a transition
function that uses the optimal lower-level policy hierarchy $\pi_{i-1}^{*}$,
$T_{i}|\pi_{i-1}^{*}$ is stationary. Such assumption is valid because the
function is independent of the changing and exploring lower-level policies,
allowing an agent to learn a policy at level $i$ at the same time when the
agent learns policies below level $i$.

Secondly, all levels of the hierarchy are supplemented with an additional set
of transitions. These are referred as the hindsight goal transitions. This is
an extension of the hindsight experience replay [1]. The setting is extended
as the hierarchical setting, and enables each level to learn more effectively
in sparse reward tasks.

Finally, sub-goal testing transitions are implemented for a level to
understand whether a sub-goal state can be achieved by the current set of
lower level policies. These transitions are implemented as follows. After
level $i$ proposes a sub-goal $G_{i}$, the lower level behavior policy
hierarchy $\pi_{i-1_{b}}$ used to achieve sub-goal $a_{i}$ must be the current
lower policy hierarchy $\pi_{i-1}$. If the sub-goal $G_{i}$ is not achieved in
at most $H$ actions by level $i-1$, level $i$ will be penalized with a low
reward.

###  4.3 Training with Curriculum Learning

Training the proposed HAC module is equipped with curriculum learning in the
following two aspects: $(1)$ we set a curriculum schedule for the goal of the
proposed hierarchical actor-critic module; $(2)$ we apply a simple teacher-
student strategy in the learning of the curriculum schedule. In particular, we
set a curriculum for the IoU goal of furniture $1$ as
$\textbf{G}_{1}^{\text{curriculum}}=\\{G_{1}^{0},\dots,G_{1}^{c}\\}$ where
$G_{1}^{c}$ represents the stage $c$ of the curriculum, and the IoU goal for
the $c$ stage is $0.45+0.05c$ where $c=0,\dots,10$.
$\textbf{G}_{2}^{\text{curriculum}}$ for furniture $2$ is constructed
similarly. In addition, we use the weights of networks from $c-1$ stage to be
the teacher network to choose suitable initial states for training at the
stage $c$.

##  5 Simulation and Training Process

###  5.1 Simulation Environment

As shown in Figure 3 and Figure 4, the indoor simulation environment consists
of a simulator $\mathcal{F}$, where
$(s^{i}_{\text{next}},R^{i})=\mathcal{F}(s^{i},a^{i})$ and $a^{i}\in A^{i}$ is
the action from the agent $i$ in the state $s^{i}\in S^{i}$. Here simulator
$\mathcal{F}$ receives the action $a^{i}$ and produces the next state
$s^{i}_{\text{next}}$ and reward $R^{i}$. In the next state
$s^{i}_{\text{next}}$, the geometrical position and size of walls, doors, and
windows are not changed, and only the geometrical position of the furniture is
updated according to the action. Recall that the action space is discrete, and
the center of the furniture can move left, right, up and down at each time
step. The simulator $\mathcal{F}$ drops the action if furniture moves out of
the room, and the furniture will stay in the same place.

###  5.2 Training Process

To train the proposed HAC module, a set of configurations is used: Firstly, we
use above mentioned curriculum threshold value $0.45+0.05c$ to check whether
the sub-goal for level $1$ and level $2$ are achieved at stage $c$. Secondly,
two PPO agents [27] are applied to train the two-level reinforcement learning
hierarchy. The discount factor $\gamma$ and the learning rate are $0.95$ and
$0.001$. If the IoU score for each piece of furniture is over $0.9$, we stop
the training of this episode and save the weights of PPO agents. We then start
to train the next episode. In total, we train at most $1,000$ episodes for
each stage of the curriculum.

##  6 Numerical Experiments

Figure 5: Evaluation on the bedroom. The ground truth 3D graphic scene is
shown in the first row, and the second row represents the scenes with
different initial furniture positions. Here the red objects represent the bed
and the yellow objects represent the cabinet. The layouts produced by our
method in the simulation are given in the third row. The last row represents
the rendering environment in the 3D graphic scenes. Figure 6: Evaluation on
the bathroom. Here the red objects represent the washer and the yellow objects
represent the toilet.  Figure 7: Evaluation on the tatami room. Here the red
objects represent the tatami bed and the yellow objects represent the cabinet.
Figure 8: Evaluation on the kitchen. Here the red objects represent the cooker
and the yellow objects represent the washer.

We discuss the qualitative and quantitative results in this section. In our
numerical experiments, four main types of indoor rooms including the tatami,
bedroom, bathroom and kitchen are evaluated. In each type of room, there are
two types of furniture. For example, there is a tatami bed and a custom
cabinet in the tatami room, a toilet and a washer in the bathroom, a normal
bed and a custom cabinet in the bedroom, and a cooker and a washer in the
kitchen.

In the simulated environment of each type of room, there are two directions of
actions for each type of furniture as shown in Figure 4. In the simulated
tatami room environment, actions for the movement of a tatami bed are up and
down, and actions for a custom cabinet are left and right. In the simulated
kitchen room environment, actions for the movement of a cooker are left and
right, and actions for the movement of a washer are up and down. In the
simulated bedroom environment, actions for the movement of a bed are up and
down, and actions for the movement of a custom cabinet are left and right. In
the simulated bathroom environment, actions for the movement of a toilet are
up and down, and actions for the movement of a washer are left and down.

As shown in Figure 5, the agents move the bed (red object), and the custom
cabinet (yellow object) from the random starting position. While the bed is
moved up or down, the custom cabinet is moved left or right. Eight different
initializations are given in the second row. The two RL agents make sequential
decisions and move the bed and the customized cabinet hierarchically. The
final layouts of the bed and customized cabinet are shown in the last row. It
can be observed that for all eight cases our proposed method results in the
expected position. As shown in Figure 6 \- Figure 8, our proposed method can
also produce satisfactory layouts for the tatami room, bathroom, and kitchen.

In our experiments, we compare our proposed model with the state-of-art
baseline methods include LayoutGAN [31], which uses a novel differentiable
wireframe rendering layer that maps the generated layout to a wireframe image,
PlanIT [31], which utilizes a graph neural network for the furniture layout
and is suitable for 2D simulated layout, and LayoutVAE [13] which applies a
probabilistic and auto-regressive model that generates the scene layout using
latent variables in lower dimensions. SceneFormer [33] and ATISS [23] which
uses the encoder and decoder based on attention mechanism to generate the
indoor scenes. We remark that for a fair comparison with the baseline models,
we re-implemented LayoutGAN [31], PlanIT [31], LayoutVAE [13] SceneFormer [33]
and ATISS [23] through applying prior of random position of the furniture. For
each room, we test the performance of the proposed model with $2,000$ random
starting points. We train on $5,000$ samples for each type of room and test on
$1,000$ samples for the corresponding type of room. We use the intersection
over union (IoU) to measure the intersection between the predicted layout and
the ground truth layout. The comparison for different models is shown in Table
1 and Table 2.

Table 1: IoU scores ($\pm$ standard error) for various models (Furniture 1). Room Type | ATISS | SceneFormer | PlanIT | LayoutVAE | LayoutGAN | Ours
---|---|---|---|---|---|---
Tatami | $0.691\pm 0.011$ | $0.641\pm 0.012$ | $0.651\pm 0.009$ | $0.628\pm 0.008$ | $0.645\pm 0.013$ | $0.829\pm 0.007$
Bedroom | $0.672\pm 0.015$ | $0.619\pm 0.007$ | $0.649\pm 0.005$ | $0.632\pm 0.009$ | $0.642\pm 0.007$ | $0.819\pm 0.008$
Bathroom | $0.681\pm 0.009$ | $0.627\pm 0.011$ | $0.628\pm 0.007$ | $0.613\pm 0.008$ | $0.620\pm 0.013$ | $0.804\pm 0.009$
Kitchen | $0.701\pm 0.007$ | $0.615\pm 0.009$ | $0.634\pm 0.012$ | $0.629\pm 0.012$ | $0.631\pm 0.013$ | $0.825\pm 0.012$
Table 2: IoU scores ($\pm$ standard error) for various models (Furniture 2). Room Type | ATISS | SceneFormer | PlanIT | LayoutVAE | LayoutGAN | Ours
---|---|---|---|---|---|---
Tatami | $0.671\pm 0.008$ | $0.618\pm 0.009$ | $0.643\pm 0.012$ | $0.621\pm 0.009$ | $0.634\pm 0.012$ | $0.781\pm 0.006$
Bedroom | $0.685\pm 0.011$ | $0.605\pm 0.010$ | $0.638\pm 0.009$ | $0.637\pm 0.012$ | $0.629\pm 0.013$ | $0.790\pm 0.009$
Bathroom | $0.691\pm 0.009$ | $0.623\pm 0.008$ | $0.615\pm 0.008$ | $0.623\pm 0.011$ | $0.619\pm 0.009$ | $0.786\pm 0.011$
Kitchen | $0.687\pm 0.015$ | $0.619\pm 0.013$ | $0.627\pm 0.011$ | $0.625\pm 0.009$ | $0.616\pm 0.015$ | $0.793\pm 0.008$
Table 3: Percentage ($\pm$ standard error) of 2AFC perceptual study for various models where the real sold solutions are judged more plausible than the generated scenes. Room Type | ATISS | SceneFormer | PlanIT | LayoutVAE | LayoutGAN | Ours
---|---|---|---|---|---|---
Tatami | $73.18\pm 4.19$ | $83.51\pm 3.17$ | $87.29\pm 5.26$ | $86.31\pm 5.31$ | $87.25\pm 6.47$ | $62.58\pm 5.69$
Bedroom | $71.34\pm 3.26$ | $81.36\pm 3.69$ | $84.31\pm 6.25$ | $83.69\pm 5.94$ | $84.91\pm 5.82$ | $63.27\pm 5.72$
Bathroom | $76.29\pm 2.57$ | $82.47\pm 3.91$ | $87.12\pm 3.62$ | $82.69\pm 5.82$ | $85.82\pm 5.61$ | $62.18\pm 4.48$
Kitchen | $74.14\pm 3.41$ | $83.71\pm 3.42$ | $86.57\pm 5.19$ | $82.74\pm 5.91$ | $84.91\pm 5.72$ | $63.85\pm 4.91$

Similar to PlanIT [31], we also conduct a two-alternative forced-choice (2AFC)
perceptual study to compare the images from generated scenes with the
corresponding scenes from the sold industrial solutions. The generated scenes
are produced by our models, PlanIT [31], LayoutVAE [13] and LayoutGAN [15],
SceneFormer [33] and ATISS [23] respectively. Ten professional interior
designers were recruited as the participants. The scores are shown in Table 3.
It can be concluded that our proposed approach outperforms the state-of-art
models.

##  7 Conclusion

In this paper, we tackle the interior decoration problem with furniture layout
in virtual reality. We formulate the problem as a hierarchical reinforcement
learning task. We further solve the problem of two-furniture design and
produce a high-quality layout. It is worthwhile to extend our work and
consider planning for different shapes of rooms. The auto layout of furniture
for multiple rooms or the entire house is also a promising direction to be
explored in virtual reality.

## References

  * [1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  arXiv preprint arXiv:1707.01495, 2017.
  * [2] Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih.  Driving-signal aware full-body avatars.  ACM Transactions on Graphics (TOG), 40(4):1–17, 2021.
  * [3] Wongun Choi, Yu-Wei Chao, Caroline Pantofaru, and Silvio Savarese.  Understanding indoor scenes using 3d geometric phrases.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
  * [4] Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner.  Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [5] Saumitro Dasgupta, Kuan Fang, Kevin Chen, and Silvio Savarese.  Delay: Robust spatial layout estimation for cluttered indoor scenes.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [6] Xinhan Di and Pengqian Yu.  Deep reinforcement learning for producing furniture layout in indoor scenes.  arXiv preprint arXiv:2101.07462, 2021.
  * [7] Xinhan Di and Pengqian Yu.  Multi-agent reinforcement learning of 3d furniture layout simulation in indoor graphics scenes.  arXiv preprint arXiv:2102.09137, 2021.
  * [8] Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan.  Example-based synthesis of 3d object arrangements.  ACM Trans. Graph., 31(6), Nov. 2012.
  * [9] Elizabeth Gibney.  Google ai algorithm masters ancient game of go.  Nature News, 529(7587):445, 2016.
  * [10] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi.  Iqa: Visual question answering in interactive environments.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [11] Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt.  Real-time deep dynamic characters.  arXiv preprint arXiv:2105.01794, 2021.
  * [12] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori.  Layoutvae: Stochastic scene layout generation from a label set.  In The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [13] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori.  Layoutvae: Stochastic scene layout generation from a label set.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9895–9904, 2019.
  * [14] Andrew Levy, George Dimitri Konidaris, Robert Platt Jr., and Kate Saenko.  Learning multi-level hierarchies with hindsight.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [15] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu.  Layoutgan: Generating graphic layouts with wireframe discriminators.  CoRR, abs/1901.06767, 2019.
  * [16] Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang.  Grains: Generative recursive autoencoders for indoor scenes.  ACM Trans. Graph., 38(2), Feb. 2019.
  * [17] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [18] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih.  Mixture of volumetric primitives for efficient neural rendering.  arXiv preprint arXiv:2103.01954, 2021.
  * [19] Rafał K Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney.  Fovvideovdp: A visible difference predictor for wide field-of-view video.  ACM Transactions on Graphics (TOG), 40(4):1–19, 2021.
  * [20] Michael Mara, Felix Heide, Michael Zollhöfer, Matthias Nießner, and Pat Hanrahan.  Thallo–scheduling for high-performance large-scale non-linear least-squares solvers.  ACM Transactions on Graphics (TOG), 40(5):1–14, 2021.
  * [21] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937, 2016.
  * [22] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [23] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler.  Atiss: Autoregressive transformers for indoor scene synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [24] Martin L Puterman.  Markov decision processes: discrete stochastic dynamic programming.  John Wiley & Sons, 2014.
  * [25] Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, and Song-Chun Zhu.  Human-centric indoor scene synthesis using stochastic grammar.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [26] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.  Mastering atari, go, chess and shogi by planning with a learned model.  Nature, 588(7839):604–609, 2020.
  * [27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [28] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [29] Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, and Thomas Funkhouser.  Semantic scene completion from a single depth image.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [30] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [31] Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X. Chang, and Daniel Ritchie.  Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks.  ACM Trans. Graph., 38(4), July 2019.
  * [32] Kai Wang, Manolis Savva, Angel X. Chang, and Daniel Ritchie.  Deep convolutional priors for indoor scene synthesis.  ACM Trans. Graph., 37(4), July 2018.
  * [33] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner.  Sceneformer: Indoor scene generation with transformers.  arXiv preprint arXiv:2012.09793, 2020.
  * [34] Jungdam Won, Deepak Gopinath, and Jessica Hodgins.  Control strategies for physically simulated characters performing two-player competitive sports.  ACM Transactions on Graphics (TOG), 40(4):1–11, 2021.
  * [35] Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi-Min Hu.  Sketch2scene: Sketch-based co-retrieval and co-placement of 3d models.  ACM Trans. Graph., 32(4), July 2013.
  * [36] Cheng Zhang, Zhao Dong, Michael Doggett, and Shuang Zhao.  Antithetic sampling for monte carlo differentiable rendering.  ACM Transactions on Graphics (TOG), 40(4):1–12, 2021.
  * [37] He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura.  Manipnet: neural manipulation synthesis with a hand-object spatial representation.  ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  * [38] Zaiwei Zhang, Zhenpei Yang, Chongyang Ma, Linjie Luo, Alexander Huth, Etienne Vouga, and Qixing Huang.  Deep generative modeling for scene synthesis via hybrid representations.  ACM Trans. Graph., 39(2), Apr. 2020.