# Compositional Transformers for Scene Generation

Drew A. Hudson
Department of Computer Science
Stanford University
<EMAIL_ADDRESS>
&C. Lawrence Zitnick
Facebook AI Research
Facebook, Inc.
<EMAIL_ADDRESS>

###### Abstract

We introduce the GANformer2 model, an iterative object-oriented transformer,
explored for the task of generative modeling. The network incorporates strong
and explicit structural priors, to reflect the compositional nature of visual
scenes, and synthesizes images through a sequential process. It operates in
two stages: a fast and lightweight planning phase, where we draft a high-level
scene layout, followed by an attention-based execution phase, where the layout
is being refined, evolving into a rich and detailed picture. Our model moves
away from conventional black-box GAN architectures that feature a flat and
monolithic latent space towards a transparent design that encourages
efficiency, controllability and interpretability. We demonstrate GANformer2’s
strengths and qualities through a careful evaluation over a range of datasets,
from multi-object CLEVR scenes to the challenging COCO images, showing it
successfully achieves state-of-the-art performance in terms of visual quality,
diversity and consistency. Further experiments demonstrate the model’s
disentanglement and provide a deeper insight into its generative process, as
it proceeds step-by-step from a rough initial sketch, to a detailed layout
that accounts for objects’ depths and dependencies, and up to the final high-
resolution depiction of vibrant and intricate real-world scenes. See
https://github.com/dorarad/gansformer for model implementation.

##  1 Introduction

Drawing, the practice behind human visual and artistic expression, can
essentially be defined as an iterative process. It starts from an initial
outline, with just a few strokes that specify the spatial arrangement and
overall layout, and is then gradually refined and embellished with color,
depth and richness, until a vivid picture eventually emerges. These initial
schematic sketches can serve as an abstract scene representation that is very
concise, yet highly expressive: several lines are enough to delineate three
dimensional structures, account for perspective and proportion, convey shapes
and geometry, and even imply semantic information [52, 67, 63]. A large body
of research in psychology highlights the importance of sketching in
stimulating creative discovery [40, 58], fostering cognitive development [27],
and facilitating problem solving [26, 44]. In fact, a large variety of
generative and RL tasks either in the visual [5], textual [78, 22] or symbolic
modalities [2], can benefit from the same strategy – prepare a high-level plan
first, and then carry out the details.

At the heart of this hierarchical strategy stands the principle of
compositionality – where the meaning of the whole can be derived from those of
its constituents. Indeed, the world around us is highly structured. Our
environment consists of a varied collection of objects, tightly interconnected
through dependencies of all sorts: from close-by to long-range, and from
physical and dynamic to abstract and semantic [66, 10]. As pointed out by
prior literature [24, 55, 51, 57], compositional representations are pivotal
to human intelligence, supporting our capabilities of reasoning [32], planning
[53], learning [54] and imagination [6]. Realizing this principle, and
explicitly capturing the objects and elements composing the scene, is thereby
a desirable goal, that can make the generative process more explainable,
controllable, versatile and efficient.

(a)  Figure 1: The GANformer2 model has a set of latent variables that
represent the objects and entities within the generated scene. It proceeds
through two stages, planning and execution: first iteratively drafting a high-
level layout of the scene’s spatial structure, and then refining and
translating it into a photo-realistic image. Each latent corresponds to a
different layout’s segment, controlling its structure during the planning, and
its style during the execution. On the discriminator side, we introduce new
structural losses for compositional scene generation.

Yet, the vast majority of generative models do not directly reflect this
compositionality so intrinsic to visual scenes. Rather, most networks, and
GANs in particular, aim to generate the entire scene all at once from a single
monolithic latent space, through a largely opaque transformation.
Consequently, while they excel in producing stunning, strikingly-realistic
portrayals of faces, sole centralized objects, or natural scenery, they still
struggle to faithfully mimic richer or densely-packed scenes, as those common
to everyday contexts, falling short of mastering their nuances and intricacies
[76, 50, 5]. Likewise, while there is some evidence for property
disentanglement within their latent space to independent axes of variation at
the global scale [28, 64, 71, 36], most existing GANs fail to provide
controllability at the level of the individual object or local region.
Understanding of the mechanisms by which these models give rise to an output
image, and the transparency of their synthesis process, thereby remain rather
limited.

Motivated to alleviate these shortcomings and make generative modeling more
compositional, interpretable and controllable, we propose GANformer2, a
structured object-oriented transformer that decouples the visual synthesis
task into two stages: planning and execution. We begin by sampling a set of
latent variables, corresponding to the objects and entities that compose the
scene. Then, at the planning stage, we transform the latents into a schematic
layout – a scene representation that depicts the object segments, reflecting
their shapes, positions, semantic classes and relative ordering. Its
construction occurs in an iterative manner, to capture the dependencies and
interactions between different objects. Next, at the execution stage, the
layout is translated into the final image, with the latents cooperatively
guiding the content and style of their respective segments through bipartite
attention [33]. This approach marks a shift towards a more natural process of
image generation, in which objects and the relations among them are explicitly
modeled, while the initial sketch is successively tweaked and refined to
ultimately produce a rich photo-realistic scene.

We study the model’s behavior and performance through an extensive set of
experiments, where it attains state-of-the-art results in both conditional and
unconditional synthesis, as measured along metrics of fidelity, diversity and
semantic consistency. GANformer2 reaches high versatility, demonstrated
through multiple datasets of challenging simulated and real-world scenes.
Further analysis illustrates its capacity to manipulate chosen objects and
properties in an independent manner, achieving both spatial disentanglement
and separation between structure and style. We notably observe amodal
completion of occluded objects, likely driven by the sequential nature of the
computation. Meanwhile, inspection of the produced layouts and their
components shed more light upon the synthesis of each resulting scene,
substantiating the model’s transparency and explainability. Overall, by
integrating strong compositional structure into the model, we can move in the
right direction towards multiple favorable properties: increasing robustness
for rich and diverse visual domains, enhancing controllability over individual
objects, and improving the generative process’s interpretability.

##  2 Related Work

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

Figure 2: Layout and image generation. The GANformer2 model generates each
image by first creating a layout that specifies its outline and object
composition, which is then translated into the final image. While doing so,
the model produces depth maps of the object segments which are used to
determine their relative ordering, and notably are implicitly derived by the
model from the layouts in a fully unsupervised manner.

Over the last years, the field of visual sythesis has witnessed astonishing
progress, driven in particular by the emergence of Generative Adversarial
Networks [23]. Tremendous strides have been made in a wide variety of tasks,
from image [11] to video generation [7], to super-resolution [45], style
transfer [15] and translation [35]. Meanwhile, a growing body of literature
explored object discovery and sequential scene decomposition, albeit in the
context of variational inference [16, 1, 49, 19]. DRAW [25] uses recurrent
attention to encode and decode MNIST digits, while AIR [20], MONet [12] and
IODINE [24] reconstruct simple simulated scenes by iteratively factorizing
them into semantically meaningful segments. We draw inspiration from this
pioneering line of works and marry the merits of its strong structure with the
scalability and fidelity obtained by GANs, yielding a compositional model that
successfully creates intricate real-world scenes, while offering a more
transparent and controllable synthesis along the way, conceptually meant to
better resemble the causal generative process and serve as a more natural
emulation of how a human might approach the creative task.

To this end, our model leverages the GANformer architecture we introduced
earlier this year [33] – a bipartite transformer that uses soft-attention to
cooperetaviely translate a set of latents into a generated image [70, 33]. In
GANformer2, we take this idea a step further, and show how we can use the
bipartite transformer as a backbone for a more structured process, that
enables not only an implicit, but also an explicit modeling of the objects and
constituents composing the image’s scene. Our iterative design endows the
GANformer2 model with the flexibility to consider the correlations and
interactions not only among latents but between synthesized scene elements
directly, circumventing the need for partial independence assumptions made by
earlier works [69, 56, 18], and thereby managing to scale to natural real-
world scenes. Concurrently, the explicit object-oriented structure enhances
the model’s compositionallity so to outperform past coarse-to-fine cascade
[80], layered [34], and hierarchical approaches [5], which, like us, employ a
two-stage layout-based procedure, but contrary to our work, do so in a non-
compositional fashion, and indeed were mostly explored within limited domains,
such as indoor NYUv2 [73], faces [41], pizzas [59], or bird photographs [77].

Our model links to research on conditional generative modeling [72, 85, 87],
including semantic image synthesis and text-to-image generation. The seminal
Pix2Pix [35] and SPADE [60] works respectively adopt either a U-Net [62] or
spatially-adaptive normalization [61] to map a given input layout into an
output high-resolution image. However, beyond their notable weakness of being
conditional and thus unable to create scenes from scratch, their reliance on
both static class embeddings for feature modulation, along with perceptual or
feature-matching reconstruction losses, render their sample variation rather
low [86, 68]. In our work, we mitigate this issue by proposing new purely
generative structural loss functions: Semantic Matching and Segment Fidelity,
which maintain the one-to-many relation that holds between layouts and images,
and consequently, as demonstrated in section 3.3, significantly enhance the
output diversity.

While GANformer2 is an unconditional model, it shares some high-level ideas
with conditional text-to-image [31, 30] and scene-to-image [38, 4] techniques,
which use semantic layouts as a bottleneck intermediate representation. But
whereas these works focus on heavily-engineered pipelines consisting of
multiple pre- and post-processing stages and different sources of supervision,
including textual descriptions, scene graphs, bounding boxes, segmentation
masks and images, we propose a simpler, more elegant design, with a
streamlined architecture and fewer supervision signals – relying on images and
auto-predicted layouts only, to widen its applicability across domains.
Finally, our approach relates to prior works on visual editing [8, 46, 84, 9]
and image compositing [13, 79, 3, 47], but while these methods modify an
existing source image, GANformer2 stands out being capable of creating
structured manipulatable scenes from scratch.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

Figure 3: Recurrent scene generation. GANformer2 creates the layout
sequentially, segment-by-segment, to capture the scene’s compositionality,
effectively allowing us to add or remove objects from the resulting images.

##  3 The GANformer2 model

The GANformer2 model is a compositional object-oriented transformer for visual
scene generation. It features a set of latent variables corresponding to the
objects and entities composing the scene, and proceeds through two synthesis
stages: (1) a sequential planning stage, where a scene layout – a collection
of interacting object segments, is being created (section  3.2), followed by
(2) a parallel execution stage, where the layout is being refined and
translated into an output photo-realistic image (section  3.3). Both stages
are implemented using customized versions of the GANformer model (section
3.1) – a bipartite transformer that uses multiple latent variables to modulate
the evolving visual features of a generated image, so to create it in a
compositional fashion.

In this paper, we explore learning from two training sets: of images and
layouts. Specifically, we use panoptic segmentations [43], which specify for
every segment $s_{i}$ its instance identity $p_{i}$ and semantic category
$m_{i}$, but other types of segmentations can likewise be used. The images and
layouts can either be paired or unpaired, and our training scheme accommodates
both options: either by pre-training each stage independently first and then
fine-tune them jointly in the paired case, or training them together from
scratch in the unpaired one (See section F for further details).

###  3.1 Generative Adversarial Transformers

The GANformer [33] is a transformer-based Generative Adversarial Network: it
consists of a generator $G(Z)=X$ that translates a partitioned latent
${Z}^{k{\times}d}=[z_{1},...,z_{k}]$ into an image ${X}^{HWc}$, and a
discriminator $D(X)$ that seeks to discern between real and fake images. The
generator begins by mapping the normally-distributed latents into an
intermediate space $W=[w_{1},...,w_{k}]$ using a feed-forward network. Then,
it sets an initial 4$\times$4 grid, which gradually evolves through
convolution and upsampling layers to the final image. Contrary to traditional
GANs, the GANformer also incorporates bipartite-transformer layers into the
generator, which compute attention between the $k$ latents and the image
features, to support spatial and contentual region-wise modulation:

$\displaystyle\text{Attention}(X,W)=\text{softmax}\left(\frac{q(X)k(W)^{T}}{\sqrt{d}}\right)v(W)$
(1)
$\displaystyle{u}({X},{W})=\gamma_{a}(X,W)\odot{LayerNorm}({X})+\beta_{a}(X,W)$
(2)

Where the equations respectively express the attention and modulation between
the latents and the image; $q(\cdot),k(\cdot),v(\cdot)$ are the query, key,
and value mappings; and $\gamma(\cdot),\beta(\cdot)$ compute multiplicative
and additive styles (gain and bias) as a function of $\text{Attention}(X,W)$.
The update rule lets the latents shape and impact the image regions as induced
by the key-value attention image-to-latents assignment computed between them.
This encourages visual compositionality, where different latents specialize to
represent and control various semantic regions or visual elements within the
generated scene.

As we will see below, for each of the two generation stages, we couple a
modified version of the GANformer with novel loss functions and training
schemes. One key point in which we notably depart from the original GANformer
regards to the number of latent variables used by the model. While the
GANformer employs a fixed number of $k$ latent variables, GANformer2 supports
a variable number of latents, thereby becoming more flexible in generating
scenes of diverse complexity.

###  3.2 Planning Stage – Layout Synthesis

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 4: GANformer2 style variation. Images are produced by varying the style
latents of the objects while keeping their structure latents fixed, achieving
high visual diversity while maintaining structural consistency.

In the first stage, we create from scratch a schematic layout depicting the
structure of the scene to be formed (see figure 2). Formally, we define a
layout $S$ as a set of $k$ segments $\\{s_{1},...,s_{k}\\}$. Each generated
segment $s_{i}$ is associated with the following111To produce the segments
(instead of RGB values), we modify the output layer to predict for every
segment its shape $P_{i}$, category $M_{i}$ and depth $d_{i}$. Since the
distributions are soft, the model stays fully differentiable.:

  * •

Shape & position: specified by a spatial distribution ${P_{i}}(x,y)$ over the
image grid $H\times W$.

  * •

Semantic category: a soft distribution ${M_{i}}(c)$ over the semantic classes.

  * •

Unsupervised depth-ordering: per-pixel values ${d_{i}}(x,y)$ indicating the
order relative to other segments, internally used by the model to compose the
segments into a layout (details below).

Recurrent Layout Generation. To explicitly capture conditional dependencies
among segments, we construct the layout in an iterative manner, recurrently
applying a parameter-lightweight GANformer $G_{1}$ over $T$ steps (around
2–4). At each step $t$, we generate a variable number of segments,
$k_{t}\sim\mathcal{N}({\mu},\sigma^{2})$ sampled from a trainable normal
distribution222Segments are synthesized in groups for computational efficiency
when generating crowded scenes., and gradually aggregate them into a scene
layout $S_{t}$. To generate the segments at step $t$, we use a GANformer with
$k_{t}$ sampled latents, each one $z_{i}$ is mapped through a feed-forward
network $F_{1}(z_{i})=u_{i}$ into an intermediate  structure latent $u_{i}$,
which then corresponds to segment $s_{i}$, guiding the synthesis of its shape,
depth and position.

$G_{1}(U^{k_{t}{\times}d},S_{t-1})=S_{t}$

To make the GANformer recurrent, and condition the generation of new segments
on ones produced in prior steps, we change the query function $q(\cdot)$ to
consider not only the newly formed layout $S_{t}$ (as in the standard
GANformer) but also the previously aggregated one $S_{t-1}$ (concatenating
them together and passing as an input), allowing the model to explicitly
capture dependencies across steps.

Object Manipulation. Notably, this construction supports a posteriori object-
wise manipulation. Given a generated layout $S$, we can recreate a segment
$s_{i}$ by modifying its corresponding latent from $z_{i}$ to $\hat{z}_{i}$
while conditioning on the other segments $S_{-i}$, mathematically by
$G_{1}(\hat{z}_{i}^{1{\times}d},S_{-i})$. We thus reach a favorable balance
between expressive and contextual modeling of relational interactions on the
one hand, and stronger object disentanglement and controllability on the
other.

Layout Composition. To compound the segments $s_{i}$ together into a layout
$S_{t}$, we overlay them according to their predicted depths ${d_{i}}(x,y)$
and shapes ${P_{i}}(x,y)$ by computing

${\text{Softmax}(d_{i}(x,y)+\log{{P_{i}}(x,y)})}_{i=1}^{k}$

(a)

(b)

Figure 5: Hierarchical vs. random noise.

which yields for each image position $(x,y)$ a distribution over the segments.
Intuitively, this allows resolving occlusions among overlapping segments. E.g.
if a cube segment $i$ should appear before a sphere segment $j$, then
${d_{i}}(x,y)>{d_{j}}(x,y)$ within the overlap region $\\{(x,y)\\}$. Notably,
we do not provide any supervision to the model about the true depths of the
scene elements, and it rather creates its own internal depth ordering in an
unsupervised manner, only indirectly moderated by the need to produce
realistic looking layouts when stitching the segments together.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 6: Object controllability. GANformer2 supports entity-level
manipulation, and separates style from structure, enabling control over the
chosen objects and properties. At each row, we begin by generating a scene
from scratch (leftmost image) and then gradually interpolate the latent vector
of a single object: (1) modify its style latent $w_{i}$, leading to a color
change, (2) modify its structure latent $u_{i}$, resulting in a position
change, and (3) modify both the structure and style latents, yielding
respective changes in shape and material.

Noise for Discrete Synthesis. The segmentations we train the planning stage
over are inherently discrete, while the generated layouts are soft and
continuous. To maintain stable GAN training under these conditions, and
inspired by the instance noise technique [65], we add a simple hierarchical
noise map to the layouts fed into the discriminator, which helps ensuring its
task remains challenging. We define the noise as
$\sum_{r=6}^{8}{\text{Upsample}({n}_{r}^{2^{r}\times 2^{r}}})$, summing up
noise grids ${n}_{r}\sim\mathcal{N}(0,\,\sigma^{2})$ across multiple
resolutions, making it partly correlated between adjacent positions and more
structured overall. This empirically enhances the effectiveness and
substantially improves the generated layouts’s quality.

###  3.3 Execution Stage: Layout-to-Image Translation

In the second stage, we proceed from planning to execution, and translate the
generated layout $S$ into a photo-realistic image $X$. For each segment
$s_{i}$, we first concatenate its mean depth $d_{i}$ and category distribution
$m_{i}$ to its respective latent $z_{i}$, and map them through a feed-forward
network $F_{2}$ into an intermediate  style latent $w_{i}$, which will
determine the content, texture and style of its segment. Then, we pass the
resulting vectors into a second GANformer $G_{2}(W,S)=X$ to render the final
image.

Layout Translation. As opposed to the original GANformer [33] that computes
$\text{Attention}(X,W)$ on-the-fly to determine the assignment between latents
and spatial regions, here we rely instead on the pre-planned layout $S$ as a
conditional guidance source to specify this association. That way, each latent
$w_{i}$ modulates the features within the soft region of its respective
segment $s_{i}$, achieving a more explicit semantically-meaningful assignment
between latents and scene elements. At the same time, since the whole image is
still generated together, with all regions processed through the convolution,
upsampling and modulation layers of $G_{2}$, the model can still ensure global
visual coherence.

Layout Refinement. To increase the model’s flexibility while rendering the
output image, we optionally multiply the latent-to-segment distribution
induced by the layout $S$ with a trainable sigmoidal gate $\sigma(g(S,X,W))$,
which considers the layout $S$, the evolving image features $X$, and the style
latents $W$. This allows the model to locally adjust the modulation strength
as it sees fit, to refine the pre-computed layout during the final image
synthesis.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

Figure 7: Conditional generation. Sample images generated by GANformer2
conditioned on input layouts.

Semantic-Matching Loss. To promote strcutrual and semantic consistency between
the layout $S$ and image $X$ produced by each of the synthesis stages, we
complement the standard fidelity loss $\mathcal{L}(D(\cdot))$ with two new
objectives: Semantic Matching and Segment Fidelity. We note that the relation
between a given layout and the set of possible images that could correspond to
it is a one-to-many relation. To encourage it, we incorporate into the
discriminator a U-Net [62] to predict the segmentation $S^{\prime}$ of each
image, compare it using cross-entropy to the source layout, and add the
resulting loss term to the discriminator (over real samples) and to the
generator (over fake ones).

$\mathcal{L}_{SM}(S,S^{\prime})\;\;\text{where}\;\;S^{\prime}=\text{Seg}(X)$

This stands in a stark contrast to the reconstruction perceptual or feature-
matching losses commonly used in conditional layout-to-image approaches [14,
35, 60], which, unjustifiably, compare the appearance of a newly generated
image $X$ to that of a natural source image $X^{\prime}$ (mathematically,
through either $\mathcal{L}_{P}(X,X^{\prime})$ or
$\mathcal{L}_{FM}(f(X),f(X^{\prime}))$), even though there is no reason for
them to be similar in terms of style and texture, beyond sharing the same
layout. Consequently, those losses severely and fundamentally hinder the
diversity of the generated scenes (as shown in section 4). We circumvent this
deficiency by comparing instead the layout implied by the new image with the
source layout it is meant to follow, thereby staying true to the original
objective of the translation task.

Segment-Fidelity Loss. To further promote the semantic alignment between the
layout and the image, and encourage fidelity not only of the whole scene, but
also of each segment it is composed of, we introduce the Segment-Fidelity
loss. Given the layout $S$ and the generated image $X$, the discriminator
first concatenates and processes them together through a shared convolution-
downsampling stem to reduce their resolution. Then, it partitions the image
according to the layout (like a jigsaw puzzle) and assesses the fidelity of
each of the $n$ segments $(s_{i},x_{i})$ independently, through:

$\tfrac{1}{n}\sum\mathcal{L}_{SF}(D(s_{i},x_{i}))$

Using the stem both reduces the image dimension before splitting it up to
improve computational efficiency, and further allows for local information
about the contextual surroundings of each segment to propagate into its
representation. This improves over the stricter patchGAN [35] which blindly
splits the image into a fixed tabular grid, while our new loss offers a more
natural choice, dividing it up into semantically-meaningful regions.

Overall, to build the complete end-to-end GANformer2 model, we chain the two
synthesis stages together: first performing the planning stage, translating
sampled latents into a schematic layout, and then proceeding to the execution
stage and transforming the layout into the final output image.

##  4 Experiments

Table 1: Unconditional approaches comparison. We compare the models on generating images from scratch, using the FID and Precision/Recall metrics for evaluation, computed over 50k sample images. COCOp refers to a mean score over a partitioned version of the dataset, created to improve samples’ visual quality (section I). | CLEVR |  |  |  Bedrooms |  |  | FFHQ |  |
---|---|---|---|---|---|---|---|---|---
Model | FID $\downarrow$ |  Precision | Recall |  FID $\downarrow$ | Precision |  Recall | FID $\downarrow$ |  Precision | Recall
GAN | 25.02 |  21.77 | 16.76 |  12.16 | 52.17 |  13.63 | 13.18 |  67.15 | 17.64
k-GAN | 28.29 |  22.93 | 18.43 |  69.90 | 28.71 |  3.45 | 61.14 |  50.51 | 0.49
SAGAN | 26.04 |  30.09 | 15.16 |  14.06 | 54.82 |  7.26 | 16.21 |  64.84 | 12.26
StyleGAN2 | 16.05 |  28.41 | 23.22 |  11.53 | 51.69 |  19.42 | 10.83 |  68.61 | 25.45
VQGAN | 32.60 |  46.55 | 63.33 |  59.63 | 55.24 |  28.00 | 63.12 |  67.01 | 29.67
SBGAN | 35.13 |  48.12 | 64.41 |  48.75 | 56.14 |  31.02 | 24.52 |  58.32 | 8.17
GANformer | 9.17 |  47.55 | 66.63 |  6.51 | 57.41 |  29.71 | 7.42 |  68.77 | 5.76
GANformer2 | 4.70 |  64.18 | 67.03 |  6.05 | 60.93 |  37.15 | 7.77 |  61.54 | 34.45
|  |  |  |  |  |  |  |  |
| COCO |  |  |  COCOp |  |  | Cityscapes |  |
Model | FID $\downarrow$ |  Precision | Recall |  FID $\downarrow$ | Precision |  Recall | FID $\downarrow$ |  Precision | Recall
GAN | 41.00 |  48.57 | 7.06 |  37.53 | 51.02 |  9.30 | 11.56 |  61.09 | 15.30
k-GAN | 63.51 |  42.06 | 5.42 |  61.05 | 45.37 |  7.04 | 51.08 |  18.80 | 1.73
SAGAN | 46.09 |  50.00 | 7.47 |  42.05 | 51.71 |  7.84 | 12.81 |  43.48 | 7.97
StyleGAN2 | 26.79 |  50.92 | 23.30 |  25.24 | 52.93 |  24.48 | 8.35 |  59.35 | 27.82
VQGAN | 63.12 |  53.05 | 27.22 |  65.20 | 55.42 |  30.12 | 173.8 |  30.74 | 43.00
SBGAN | 108.1 |  42.53 | 17.53 |  104.32 | 44.43 |  19.12 | 54.92 |  52.12 | 25.08
GANformer | 25.24 |  53.68 | 12.31 |  23.81 | 55.29 |  14.48 | 5.76 |  48.06 | 33.65
GANformer2 | 21.58 |  49.12 | 29.03 |  20.41 | 51.07 |  32.84 | 6.21 |  56.12 | 54.18

We investigate GANformer2’s properties through a suite of quantitative and
qualitative experiments. As shown in section 4.1, the model attains state-of-
the-art performance, successfully generating images of high quality, diversity
and semantic consistency, over multiple challenging datasets of highly-
structured simulated and real-world scenes. In section 4.2, we compare the
execution stage to leading layout-to-image approaches, demonstrating
significantly larger output variation, possibly thanks to the new structural
loss functions we incorporate into the network. Further analysis conducted in
sections 4.3 and 4.4 reveals several favorable properties that GANformer2
possesses, specifically of enhanced interpretabilty and strong spatial
disentanglement, which enables object-level manipulation. In the
supplementary, we provide additional visualizations of style and structure
disentanglement as well as ablation and variation studies, to shed more light
upon the model behavior and operation. Taken altogether, the evaluation offers
solid evidence for the effectiveness of our approach in modeling compositional
scenes in a robust, controllable and interpretable manner.

###  4.1 State-of-the-Art Comparison

We compare our model with both baselines as well as leading approaches for
visual synthesis. Unconditional models include a baseline GAN [23], StyleGAN2
[42], Self-Attention GAN [81], the autoregressive VQGAN [21], the layerwise
k-GAN [69], the two-stage SBGAN [5] and the original GANformer. Conditional
approaches include SBGAN, BicycleGAN [86], Pix2PixHD [72], and SPADE [60]. We
implement the unconditional methods within our public GANformer codebase and
use the authors’ official implementations for the conditional ones. All models
have been trained with resolution of $256\times 256$ and for an equal number
of training steps, roughly spanning 10 days on a single V100 GPU per model.
See section H for description of baselines and competing approaches,
implementation details, hyperparameter settings, data preparations and
training configuration.

As tables 1 and 3 show, the GANformer2 model outperforms prior work along most
metrics and datasets in both the conditional and unconditional settings. It
achieves substantial gains in terms of FID score, which correlates with visual
quality [29], as is likewise reflected through the good Precision scores.
Notable also are the large improvements along Recall, in the unconditional and
especially the conditional settings, which indicate wider coverage of the
natural image distribution. We note that the performance gains are highest for
the CLEVR [37] and the COCO [48] datasets, both focusing on varied
arrangements of multiple objects and high structural variance. These results
serve as a strong evidence for the particular aptitude of GANformer2 for
compositional scenes.

###  4.2 Scene Diversity & Semantic Consistency

Table 2: Disentanglement & controllability comparison over sample CLEVR images, using the DCI metrics for latent-space disentanglement and our correlation-based metrics for controllability (see section 4.4). Model | Disentanglement | Modularity | Completeness | Informativeness | Informativeness’ | Object-$\rho$ $\downarrow$ | Property-$\rho$ $\downarrow$
---|---|---|---|---|---|---|---
GAN | 0.126 | 0.631 | 0.071 | 0.583 | 0.434 | 0.72 | 0.85
StyleGAN | 0.208 | 0.703 | 0.124 | 0.685 | 0.332 | 0.64 | 0.49
GANformer | 0.768 | 0.952 | 0.270 | 0.972 | 0.963 | 0.38 | 0.45
MONet | 0.821 | 0.912 | 0.349 | 0.955 | 0.946 | 0.29 | 0.37
Iodine | 0.784 | 0.948 | 0.382 | 0.941 | 0.937 | 0.27 | 0.35
GANformer2 | 0.852 | 0.946 | 0.413 | 0.974 | 0.965 | 0.19 | 0.32

(a)

(b)

Figure 8: Learning curves

Beyond visual fidelity, we study the models under the potentially competing
aims of content variability on the one hand and semantic consistency on the
other. To assess consistency, we compare the input layouts $S$ with those
implied by the synthesized images, using standard metrics of mean IoU, pixel
accuracy and ARI. Following prior works [60, 86], the inferred layouts
$S^{\prime}$ are obtained by a pre-trained segmentation model that we use for
evaluation purposes. To measure diversity, we sample $n=20$ images ${X_{i}}$
for each input layout $S$, and compute the mean LPIPS pairwise distance [82]
over image pairs $X_{i},X_{j}$. Ideally, we expect a good model to achieve
strong alignment between each input layout and the one derived from the output
synthesized image, while still maintaining high variance between samples that
are conditioned on the same source layout.

We see in table 3 that GANformer2 produces images of significantly better
diversity, as reflected both through the LPIPS and Recall scores. Other
conditional approaches such as Pix2PixHD, BicycleGAN and SPADE that rely on
pixel-wise perceptual or feature matching, reach high precision at the expense
of considerably reduced variability, revealing the weakness of those loss
functions. We further note that our model achieves better consistency than
prior work, especially for LSUN-Bedrooms and COCO. These are also illustrated
by figures 23-25 which present qualitative comparison between synthesized
samples along different dimensions. These results demonstrate the benefits of
our new structural losses (section 3.3), as is also corroborated by the
model’s learning efficiency when evaluated with different objectives (figure
8). We compare the Semantic-Matching and Segment-Fidelity losses with several
alternatives (section D), and observe improved performance and accelerated
learning when using the new objectives, possibly thanks to the local semantic
alignment and finer-detail fidelity they respectively encourage.

###  4.3 Transparency & Interpretability

An advantage of our model compared to traditional GANs, is that we incorporate
explicit structure into the visual generative process, which both makes it
more explainable, while providing means for controlling individual scene
elements (section 4.4). In figures 3 and 12, we see how GANformer2 gradually
constructs the image layout, recurrently adding new object segments into the
scene. The gains of iterative over non-iterative generation are also
demonstrated by figures 8 and 22. We note that different datasets benefit from
a different number of recurrent synthesis steps, reflecting their respective
degree of structural compositionality (section E). Figures 2 and 11 further
show that the model develops an internal notion for segments’ depth, which
aids it in determining the most effective order to compile them together.
Remarkably, no explicit information is given to the network about the depth of
entities within the scene, nor about effective orders to create them, and it
rather learns these on its own in an unsupervised manner, through the indirect
need to identify relative depths and segment ordering that will enable
creating feasible and compelling layouts.

###  4.4 Controllability & Disentanglement

Table 3: Conditional approaches comparison. We compare conditional generative models that map input layouts into output images. Multiple metrics are used for different purposes: FID, Inception (IS) and Precision scores for visual quality; class-weighted mean IoU, Pixel Accuracy (pAcc) and Adjusted Rand Index (ARI) for layout-image consistency; Recall for coverage of the natural image distribution; and LPIPS for measuring the output visual diversity given an input layout. All metrics are computed over 50k samples per model. | CLEVR |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Model | FID $\downarrow$ |  IS $\uparrow$ | Precision $\uparrow$ |  Recall $\uparrow$ | mIoU $\uparrow$ |  pAcc $\uparrow$ | ARI $\uparrow$ |  LPIPS $\uparrow$
Pix2PixHD | 4.88 |  2.19 | 72.49 |  60.40 | 98.18 |  99.06 | 97.90 |  6e-8
SPADE | 5.74 |  2.23 | 70.25 |  61.96 | 98.30 |  99.12 | 98.01 |  0.12
BicycleGAN | 35.62 |  2.29 | 6.27 |  10.11 | 88.15 |  92.78 | 90.21 |  0.10
SBGAN | 18.02 |  2.22 | 52.29 |  48.20 | 95.27 |  97.81 | 92.34 |  8e-7
GANformer2 | 0.99 |  2.34 | 73.26 |  78.38 | 97.83 |  99.15 | 98.46 |  0.19
|  |  |  |  |  |  |  |
| Bedrooms |  |  |  |  |  |  |
Model | FID $\downarrow$ |  IS $\uparrow$ | Precision $\uparrow$ |  Recall $\uparrow$ | mIoU $\uparrow$ |  pAcc $\uparrow$ | ARI $\uparrow$ |  LPIPS $\uparrow$
Pix2PixHD | 32.44 |  2.27 | 70.46 |  2.41 | 64.54 |  76.62 | 75.48 |  2e-8
SPADE | 17.06 |  2.33 | 80.20 |  7.04 | 70.27 |  80.63 | 79.44 |  0.07
BicycleGAN | 23.34 |  2.52 | 49.68 |  10.45 | 58.47 |  70.76 | 72.73 |  0.22
SBGAN | 29.35 |  2.28 | 56.21 |  7.93 | 65.31 |  71.39 | 73.24 |  3e-8
GANformer2 | 5.84 |  2.55 | 54.92 |  41.94 | 79.06 |  86.62 | 83.00 |  0.50
|  |  |  |  |  |  |  |
| CelebA |  |  |  |  |  |  |
Model | FID $\downarrow$ |  IS $\uparrow$ | Precision $\uparrow$ |  Recall $\uparrow$ | mIoU $\uparrow$ |  pAcc $\uparrow$ | ARI $\uparrow$ |  LPIPS $\uparrow$
Pix2PixHD | 54.47 |  2.44 | 57.48 |  0.73 | 68.04 |  80.18 | 63.75 |  4e-9
SPADE | 33.42 |  2.35 | 85.08 |  2.72 | 67.99 |  80.11 | 63.80 |  0.07
BicycleGAN | 56.56 |  2.53 | 52.93 |  2.36 | 66.95 |  79.41 | 62.50 |  0.15
SBGAN | 50.92 |  2.44 | 59.28 |  3.62 | 67.42 |  80.12 | 61.42 |  4e-9
GANformer2 | 6.87 |  3.17 | 57.32 |  37.06 | 68.88 |  80.66 | 64.65 |  0.38
|  |  |  |  |  |  |  |
| Cityscapes |  |  |  |  |  |  |
Model | FID $\downarrow$ |  IS $\uparrow$ | Precision $\uparrow$ |  Recall $\uparrow$ | mIoU $\uparrow$ |  pAcc $\uparrow$ | ARI $\uparrow$ |  LPIPS $\uparrow$
Pix2PixHD | 47.74 |  1.51 | 50.53 |  0.34 | 73.20 |  83.49 | 78.48 |  3e-8
SPADE | 22.98 |  1.46 | 83.32 |  9.55 | 75.96 |  85.25 | 80.74 |  3e-3
BicycleGAN | 25.02 |  1.56 | 63.01 |  9.49 | 67.24 |  78.43 | 72.20 |  7e-4
SBGAN | 45.82 |  1.52 | 51.38 |  5.39 | 72.04 |  81.90 | 79.32 |  4e-8
GANformer2 | 5.95 |  1.63 | 84.12 |  37.72 | 76.11 |  85.58 | 80.12 |  0.27
|  |  |  |  |  |  |  |
| COCO |  |  |  |  |  |  |
Model | FID $\downarrow$ |  IS $\uparrow$ | Precision $\uparrow$ |  Recall $\uparrow$ | mIoU $\uparrow$ |  pAcc $\uparrow$ | ARI $\uparrow$ |  LPIPS $\uparrow$
Pix2PixHD | 18.91 |  17.97 | 80.91 |  16.35 | 58.75 |  71.94 | 68.20 |  1e-8
SPADE | 22.80 |  16.29 | 79.00 |  10.63 | 55.72 |  69.55 | 67.73 |  0.19
BicycleGAN | 69.88 |  9.16 | 40.05 |  5.83 | 33.58 |  47.98 | 56.55 |  0.18
SBGAN | 45.94 |  14.31 | 58.93 |  10.27 | 47.82 |  54.58 | 62.29 |  4e-8
GANformer2 | 17.39 |  18.01 | 70.20 |  24.86 | 74.93 |  84.37 | 76.41 |  0.38

(a)

(b)

(c)

(d)

Figure 9: Amodal completion and reflection update for object removal.

The iterative nature of GANformer2 and its compositional latent space enhance
its disentanglement along two key dimensions: spatially, enabling independent
control over chosen objects, and semantically, separating between structure
and style – the former is considered during planning while the latter at
execution. Figures 6, 10 and 14-17 feature examples where we modify an object
of interest, changing its color, material, shape, or position, all without
negatively impacting its surroundings. In figures 4 and 18-21, we produce
images of diverse colors and textures while conforming to a source layout, and
conversely, vary the scene composition while maintaining a consistent style.
Finally, as illustrated by figures 9 and 13, we can even add or remove objects
to the synthesized scene, while respecting interactions of occlusions, shadows
and reflections, effectively achieving amodal completion, and potentially
extrapolating beyond the training data (see sections C-B).

To quantify our model’s degree of disentanglement, we refer to the classic DCI
and modularity metrics [17, 83], which measure the extant to which there is
1-to-1 correspondence between latent factors and image attributes. Following
the protocol in [33, 75], we consider a sample set of synthesized CLEVR
images, using a pre-trained object detector to extract and summarize their
visual and semantic attributes. Note that since the CLEVR object detector
reaches accuracy of 0.997, it does not introduce imprecisions into the
evaluation. To further assess the level of controllability, we perform random
perturbations over the latent variables used to generate the scenes, and
estimate the mean correlation between the resulting semantic changes among
object and property pairs. Intuitively, we expect a disentangled
representation to yield uncorrelated changes among objects and attributes. As
shown in table 2, GANformer2 attains higher disentanglement and stronger
controllability than leading GAN-based and variational approaches,
quantitatively confirming the efficacy of our approach.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

Figure 10: Localized property manipulation of selected objects, without
negatively impacting their surroundings (rows 2-5), and unsupervised depth
maps produced by GANformer2 (row 1).

##  5 Conclusion

In this paper, we have tackled the task of visual synthesis, and introduced
the GANformer2 model, an iterative transformer for transparent and
controllable scene generation. Similar to how a person may approach the
creative task, it first sketches a high-level outline and only then fills in
the details. By equipping the network with a strong explicit structure, we
endow it with the capacity to plan ahead its generation process, capture the
dependencies and interactions between objects within the scene, and reason
about them directly, moving a step closer towards compositional generative
modeling. A limitation of our work is its reliance on training layouts, but we
believe it could be mitigated by future work to integrate the generative
efforts with unsupervised object discovery and scene understanding. Further
exploration into disentanglement and means to encourage it may also be
fruitful. From a broader perspective, enhancing properties of interpretability
and controllability can make content generation more reliable, accessible, and
easy to interact with, for the benefit of artists, graphic designers, the film
industry and even the general society, and may open the door for new and
promising avenues and opportunities.

##  6 Acknowledgements

We performed the experiments for the paper on AWS cloud, thanks to Stanford
HAI credit award. Drew A. Hudson (Dor) is a PhD student at Stanford University
and C. Lawrence Zitnick is a research scientist at Facebook FAIR. We wish to
thank the anonymous reviewers for their thorough, insightful and constructive
feedback, questions and comments.

Dor wishes to thank Prof. Christopher D. Manning for the kind PhD support over
the years that allowed this work to happen!

## References

  * Anciukevicius et al. [2020] Titas Anciukevicius, Christoph H Lampert, and Paul Henderson.  Object-centric image generation with factored depths, locations, and appearances.  _arXiv preprint arXiv:2004.00642_ , 2020.
  * Andreas et al. [2017] Jacob Andreas, Dan Klein, and Sergey Levine.  Modular multitask reinforcement learning with policy sketches.  In _International Conference on Machine Learning_ , pp. 166–175. PMLR, 2017.
  * Arandjelović & Zisserman [2019] Relja Arandjelović and Andrew Zisserman.  Object discovery with a copy-pasting GAN.  _arXiv preprint arXiv:1905.11369_ , 2019.
  * Ashual & Wolf [2019] Oron Ashual and Lior Wolf.  Specifying object attributes and relations in interactive scene generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 4561–4569, 2019.
  * Azadi et al. [2019] Samaneh Azadi, Michael Tschannen, Eric Tzeng, Sylvain Gelly, Trevor Darrell, and Mario Lucic.  Semantic bottleneck scene generation.  _arXiv preprint arXiv:1911.11357_ , 2019.
  * Azadi et al. [2020] Samaneh Azadi, Deepak Pathak, Sayna Ebrahimi, and Trevor Darrell.  Compositional GAN: Learning image-conditional binary composition.  _International Journal of Computer Vision_ , 128(10):2570–2585, 2020.
  * Bar et al. [2020] Amir Bar, Roei Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and Amir Globerson.  Compositional video synthesis with action graphs.  _arXiv preprint arXiv:2006.15327_ , 2020.
  * Bau et al. [2020] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba.  Semantic photo manipulation with a generative image prior.  _arXiv preprint arXiv:2005.07727_ , 2020.
  * Bau et al. [2021] David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba.  Paint by word.  _arXiv preprint arXiv:2103.10951_ , 2021.
  * Biederman [1987] Irving Biederman.  Recognition-by-components: a theory of human image understanding.  _Psychological review_ , 94(2):115, 1987.
  * Brock et al. [2019] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In _7th International Conference on Learning Representations, ICLR_ , 2019.
  * Burgess et al. [2019] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner.  MONet: Unsupervised scene decomposition and representation.  _arXiv preprint arXiv:1901.11390_ , 2019.
  * Chai et al. [2021] Lucy Chai, Jonas Wulff, and Phillip Isola.  Using latent space regression to analyze and leverage compositionality in GANs.  _arXiv preprint arXiv:2103.10426_ , 2021.
  * Chen & Koltun [2017] Qifeng Chen and Vladlen Koltun.  Photographic image synthesis with cascaded refinement networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1511–1520, 2017.
  * Choi et al. [2018] Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pp. 8789–8797. IEEE Computer Society, 2018.  doi: 10.1109/CVPR.2018.00916.
  * Deng et al. [2018] Zhiwei Deng, Jiacheng Chen, Yifang Fu, and Greg Mori.  Probabilistic neural programmed networks for scene generation.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 4032–4042, 2018.
  * Eastwood & Williams [2018] Cian Eastwood and Christopher KI Williams.  A framework for the quantitative evaluation of disentangled representations.  In _International Conference on Learning Representations_ , 2018.
  * Ehrhardt et al. [2020] Sébastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke, Ingmar Posner, Niloy J. Mitra, and Andrea Vedaldi.  RELATE: physically plausible multi-object scene synthesis using structured latent spaces.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Engelcke et al. [2020] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner.  GENESIS: generative scene inference and sampling with object-centric latent representations.  In _8th International Conference on Learning Representations, ICLR_ , 2020.
  * Eslami et al. [2016] S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton.  Attend, infer, repeat: Fast scene understanding with generative models.  In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems_ , pp. 3225–3233, 2016.
  * Esser et al. [2020] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  _arXiv preprint arXiv:2012.09841_ , 2020.
  * Fan et al. [2018] Angela Fan, Mike Lewis, and Yann Dauphin.  Hierarchical neural story generation.  _arXiv preprint arXiv:1805.04833_ , 2018.
  * Goodfellow et al. [2014] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _arXiv preprint arXiv:1406.2661_ , 2014.
  * Greff et al. [2019] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner.  Multi-object representation learning with iterative variational inference.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 2424–2433. PMLR, 2019\.
  * Gregor et al. [2015] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra.  DRAW: A recurrent neural network for image generation.  In Francis R. Bach and David M. Blei (eds.), _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pp. 1462–1471. JMLR.org, 2015.
  * Guérin et al. [1999a] Fanny Guérin, Bernadette Ska, and Sylvie Belleville.  Cognitive processing of drawing abilities.  _Brain and Cognition_ , 40(3):464–478, 1999a.
  * Guérin et al. [1999b] Fanny Guérin, Bernadette Ska, and Sylvie Belleville.  Cognitive processing of drawing abilities.  _Brain and Cognition_ , 40(3):464–478, 1999b.
  * Härkönen et al. [2020] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  GANspace: Discovering interpretable GAN controls.  _arXiv preprint arXiv:2004.02546_ , 2020.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local nash equilibrium.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 6626–6637, 2017.
  * Hinz et al. [2019] Tobias Hinz, Stefan Heinrich, and Stefan Wermter.  Generating multiple objects at spatially distinct locations.  _arXiv preprint arXiv:1901.00686_ , 2019.
  * Hong et al. [2018] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee.  Inferring semantic layout for hierarchical text-to-image synthesis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 7986–7994, 2018.
  * Hudson & Manning [2019] Drew A Hudson and Christopher D Manning.  Learning by abstraction: The neural state machine.  _arXiv preprint arXiv:1907.03950_ , 2019.
  * Hudson & Zitnick [2021] Drew A Hudson and C. Lawrence Zitnick.  Generative adversarial transformers.  _Proceedings of the 38th International Conference on Machine Learning, ICML_ , 2021.
  * Im et al. [2016] Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic.  Generating images with recurrent adversarial networks.  _arXiv preprint arXiv:1602.05110_ , 2016.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pp. 5967–5976. IEEE Computer Society, 2017.  doi: 10.1109/CVPR.2017.632.
  * Jahanian et al. [2019] Ali Jahanian, Lucy Chai, and Phillip Isola.  On the" steerability" of generative adversarial networks.  _arXiv preprint arXiv:1907.07171_ , 2019.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , pp. 1988–1997, 2017.
  * Johnson et al. [2018] Justin Johnson, Agrim Gupta, and Li Fei-Fei.  Image generation from scene graphs.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1219–1228, 2018.
  * Kanezaki [2018] Asako Kanezaki.  Unsupervised image segmentation by backpropagation.  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 1543–1547. IEEE, 2018.
  * Kantrowitz [2012] Andrea Kantrowitz.  The man behind the curtain: what cognitive science reveals about drawing.  _Journal of Aesthetic Education_ , 46(1):1–14, 2012.
  * Karras et al. [2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In _6th International Conference on Learning Representations, ICLR_ , 2018.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of styleGAN.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pp. 8107–8116. IEEE, 2020.  doi: 10.1109/CVPR42600.2020.00813.
  * Kirillov et al. [2019] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9404–9413, 2019.
  * Lane [2018] D Lane.  Drawing and sketching: Understanding the complexity of paper–pencil interactions within technology education.  _Handbook of technology education. Springer international publishing AG_ , pp. 385–402, 2018.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * Lee et al. [2020] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  MaskGAN: Towards diverse and interactive facial image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5549–5558, 2020.
  * Lin et al. [2018] Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey.  St-GAN: Spatial transformer generative adversarial networks for image compositing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 9455–9464, 2018.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In _European conference on computer vision_ , pp. 740–755. Springer, 2014.
  * Lin et al. [2020] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn.  SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition.  In _8th International Conference on Learning Representations, ICLR_ , 2020.
  * Liu et al. [2019] Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hongsheng Li.  Learning to predict layout-to-image conditional convolutions for semantic image synthesis.  _arXiv preprint arXiv:1910.06809_ , 2019.
  * Locatello et al. [2020] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  _arXiv preprint arXiv:2006.15055_ , 2020.
  * Martins [2017] Catarina Silva Martins.  From scribbles to details.  _A Political Sociology of Educational Knowledge: Studies of Exclusions and Difference_ , pp. 103, 2017.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Nanbo et al. [2020] Li Nanbo, Cian Eastwood, and Robert Fisher.  Learning object-centric representations of multi-object scenes from multiple views.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Nash et al. [2017] Charlie Nash, SM Ali Eslami, Chris Burgess, Irina Higgins, Daniel Zoran, Theophane Weber, and Peter Battaglia.  The multi-entity variational autoencoder.  In _NIPS Workshops_ , 2017.
  * Nguyen-Phuoc et al. [2020] Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra.  BlockGAN: Learning 3d object-aware scene representations from unlabelled images.  _arXiv preprint arXiv:2002.08988_ , 2020.
  * Niemeyer & Geiger [2021] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11453–11464, 2021.
  * Okada & Ishibashi [2017] Takeshi Okada and Kentaro Ishibashi.  Imitation, inspiration, and creation: Cognitive process of creative drawing by copying others’ artworks.  _Cognitive science_ , 41(7):1804–1837, 2017.
  * Papadopoulos et al. [2019] Dim P Papadopoulos, Youssef Tamaazousti, Ferda Ofli, Ingmar Weber, and Antonio Torralba.  How to make a pizza: Learning a compositional layer-based GAN model.  In _proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8002–8011, 2019.
  * Park et al. [2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pp. 2337–2346. Computer Vision Foundation / IEEE, 2019.  doi: 10.1109/CVPR.2019.00244.
  * Perez et al. [2018] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville.  Film: Visual reasoning with a general conditioning layer.  In Sheila A. McIlraith and Kilian Q. Weinberger (eds.), _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_ , pp. 3942–3951, 2018.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pp. 234–241. Springer, 2015.
  * Salome & Moore [2010] RA Salome and BE Moore.  The five stages of development in children’s art, 2010.
  * Shen et al. [2020] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of GANs for semantic face editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9243–9252, 2020.
  * Sønderby et al. [2016] Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár.  Amortised map inference for image super-resolution.  _arXiv preprint arXiv:1610.04490_ , 2016.
  * Spelke & Kinzler [2007] Elizabeth S Spelke and Katherine D Kinzler.  Core knowledge.  _Developmental science_ , 10(1):89–96, 2007.
  * Steele [1998] Bob Steele.  _Draw me a story: An illustrated exploration of drawing-as-language_.  Portage & Main Press, 1998.
  * Sushko et al. [2020] Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva.  You only need adversarial supervision for semantic image synthesis.  _arXiv preprint arXiv:2012.04781_ , 2020.
  * van Steenkiste et al. [2020] Sjoerd van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly.  Investigating object compositionality in generative adversarial networks.  _Neural Networks_ , 130:309–325, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems_ , pp. 5998–6008, 2017.
  * Voynov & Babenko [2020] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the GAN latent space.  In _International Conference on Machine Learning_ , pp. 9786–9796. PMLR, 2020.
  * Wang et al. [2018] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional GANs.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8798–8807, 2018.
  * Wang & Gupta [2016] Xiaolong Wang and Abhinav Gupta.  Generative image modeling using style and structure adversarial networks.  In _European conference on computer vision_ , pp. 318–335. Springer, 2016.
  * Wu et al. [2019] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Wu et al. [2020] Zongze Wu, Dani Lischinski, and Eli Shechtman.  StyleSpace analysis: Disentangled controls for StyleGAN image generation.  _arXiv preprint arXiv:2011.12799_ , 2020.
  * Xu et al. [2018] Kun Xu, Haoyu Liang, Jun Zhu, Hang Su, and Bo Zhang.  Deep structured generative models.  _arXiv preprint arXiv:1807.03877_ , 2018.
  * Yang et al. [2017] Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh.  Lr-GAN: Layered recursive generative adversarial networks for image generation.  _arXiv preprint arXiv:1703.01560_ , 2017.
  * Yarats & Lewis [2018] Denis Yarats and Mike Lewis.  Hierarchical text generation and planning for strategic dialogue.  In _International Conference on Machine Learning_ , pp. 5591–5599. PMLR, 2018.
  * [79] Fangneng Zhan, Jiaxing Huang, and Shijian Lu.  Hierarchy composition GAN for high-fidelity image synthesis.
  * Zhang et al. [2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5907–5915, 2017.
  * Zhang et al. [2019] Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 7354–7363. PMLR, 2019\.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 586–595, 2018.
  * Zhao et al. [2018] Bo Zhao, Bo Chang, Zequn Jie, and Leonid Sigal.  Modular generative adversarial networks.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 150–165, 2018.
  * Zheng et al. [2020] Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Jianming Zhang, Ning Xu, and Jiebo Luo.  Semantic layout manipulation with high-resolution sparse attention.  _arXiv preprint arXiv:2012.07288_ , 2020.
  * Zhu et al. [2017a] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _IEEE International Conference on Computer Vision, ICCV_ , pp. 2242–2251, 2017a.
  * Zhu et al. [2017b] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman.  Toward multimodal image-to-image translation.  _arXiv preprint arXiv:1711.11586_ , 2017b.
  * Zhu et al. [2020] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka.  SEAN: image synthesis with semantic region-adaptive normalization.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , pp. 5103–5112. IEEE, 2020.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

(ay)

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf)

(bg)

(bh)

(bi)

(bj)

(bk)

(bl)

(bm)

(bn)

Figure 11: A visualization of the layouts and unsupervised depth maps produced
by GANformer2’s planning stage while synthesizing varied images, making the
generative process more structured and interpretable.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

(ay)

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf)

(bg)

(bh)

Figure 12: Recurrent scene generation. GANformer2 creates the layout
sequentially, segment-by-segment, to capture the scene’s compositionality,
effectively allowing us to add or remove objects from the resulting images.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

(ay)

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf)

(bg)

(bh)

Figure 13: Object removal. Since GANformer2 creates each scene as a
composition of interacting segments, it supports adding and removal of objects
while respecting various dependencies with their surroundings: Amodal
completion of occluded objects is denoted by  pink, updates of shadows and
especially reflections by  cyan, and other object removals cases by  yellow.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

Figure 14: Object controllability over structural attributes, achieved by
modifying the model’s structure latent $u_{i}$ of the chosen object during the
planning stage (section 3.2). Shape manipulation is denoted by  green, while
position changes by  yellow.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

Figure 15: Object controllability over stylistic attributes, achieved by
modifying the model’s style latent $w_{i}$ of the chosen object during the
execution stage (section 3.3). Color manipulation is denoted by  pink, while
updates of material by  cyan.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

Figure 16: Localized property manipulation of selected objects without
negatively impacting their surroundings, over LSUN-Bedrooms scenes.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

Figure 17: Localized property manipulation of selected objects, without
negatively impacting their surroundings over LSUN-Bedrooms scenes (continued).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

Figure 18: GANformer2 conditional generative diversity for CelebA. Images
conform to the source layouts while still demonstrating high variance,
featuring diversity both in stylistic aspects of lighting conditions and color
scheme, but also in structural ones, as reflected through the hair type,
background and age.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

Figure 19: GANformer2 conditional generative diversity for LSUN-Bedrooms.
Images conform to the source layouts on the one end while still demonstrating
high variance on the other, featuring diversity both in stylistic aspects of
lighting condition and color scheme, but also in structural ones, as reflected
through the windows, paintings and bedding.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

(ay)

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf)

(bg)

(bh)

(bi)

(bj)

(bk)

(bl)

(bm)

(bn)

(bo)

(bp)

(bq)

(br)

(bs)

(bt)

Figure 20: GANformer2 style and structure separation. We manipulate each
aspect while maintaining consistency over the other, varying the structure
between columns and style between rows.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai)

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

Figure 21: GANformer2 conditional generative diversity for CLEVR. The scenes
significantly vary in combinations of objects’ colors and materials (contrary
to competing approaches, as shown in figures 24-25, while still closely
following the source layouts.

(a)

(b)

Steps | 0 | 1 | 2 | 3
---|---|---|---|---
CLEVR | 11.95 | 7.54 | 4.70 | 5.25
FFHQ | 8.21 | 7.77 | 8.84 | 9.92
Cityscapes | 9.83 | 8.51 | 7.26 | 6.21
Bedrooms | 11.45 | 9.37 | 6.05 | 7.21
COCO | 31.15 | 28.45 | 24.30 | 21.58

Figure 22: Learning curves and performance comparison when varying the loss
function for the execution stage (left), and the number of recurrent planning
steps for the layout synthesis (center, right). In each step, a random number
of segments is generated, sampled from a trainable normal distribution, to
flexibly model highly-structured scenes and capture conditional object
dependencies within them while still maintaining good computational
efficiency. “0” denotes a non-compositional model that generates the layouts
in a single pass rather than as a collection of segments.

##  Appendix A Overview

In the following, we provide additional qualitative and quantitative
experiments for the GANformer2 model. Figure 11 shows visualizations of the
model’s generative process as it produces depth-aware scene layouts which in
turn guide and facilitate the synthesis of output photo-realistic images. The
model’s compositional structure does not only enhance its transparency, but
also increases its controllability (section B, figures 14-17), allowing
GANformer2 to manipulate properties of individual objects without negatively
altering their surroundings, and even develop capacity for amodal completion
of occluded objects (section C, figures 12-13). The obtained decoupling
between structure and style is further illustrated through figures 18-19 and
20-21, respectively demonstrating variability along one aspect while
maintaining consistency over the other.

We believe the diversity achieved by GANformer2 results from the new
structural losses we introduce (section 3.3), which we compare to several
baselines and alternatives in figures 23-25 and section D. In section E, we
proceed through additional model ablation and variation studies, to asses the
contribution of its different architectural components, and the number of
recurrent planning steps in particular. Section F focuses on the training
configuration, comparing and contrasting between paired vs. unpaired settings.
We conclude the supplementary by providing implementation details (section G),
description of baselines and competing methods (section H), and information
about data preparation procedures (section I).

##  Appendix B Style & Structure Disentanglement

GANformer2 decomposes the synthesis into two stages of planning and execution,
the former produces the scene layout and structure while the latter controls
its texture, colors and style. Figures figures 18-19 demonstrate the high
content variability achieved over different datasets, while still complying
with shared source layouts. Indeed, the styles differ substantially between
one sample to another, and variation is notably observed in local structures
and elements, while still conforming to the layout at the global scale. This
is most noticeable in the CelebA case (figure 19), where some of the images
produced for given layouts depict young people while other old ones, and
likewise some feature straight hair while other curly hair, even when
originating from the same layout. Likewise, in the case of LSUN-Bedrooms
(figure 18), we observe diversity that goes beyond aspects of texture and
color scheme, featuring structural variations in entities such as windows,
paintings, and bedding, among others. Meanwhile, figures 20-21 inversely show
structural variability while maintaining a consistent style, obtained by
rendering different layouts using the same set of style latents $\\{w_{i}\\}$.

The high diversity and consistency demonstrated here, which also
quantitatively surpass the competing approaches (section 4.2), may be
attributed to the new structural loss functions we employ (section 3.3). These
purely generative losses liberate us from resorting to perceptual and feature-
matching objectives common to prior work, which impose unnecessary and
unjustified pixel-wise similarity conditions on the generated images,
inhibiting their diversity. By sidestepping this reliance, we can unlock wider
variation among the synthesized scenes, both in terms of structure and style.
See a comparison between synthesized samples of different approaches at
figures 23-25.

##  Appendix C Object Controllability & Amodal Completion

In section 4.4, we explore the model’s spatial and semantic disentanglement,
and study the degree of controllability achieved over individual objects and
properties. Figure 14-17 provide a qualitative illustration, presenting
examples of latent-space interpolations that lead to smooth and localized
changes of chosen objects and properties, selectively controlling either their
structure or style.

Thanks to the compositional nature of the layout generation, we can even add
or remove objects from the scene, as is illustrated in figures 12-13, while
respecting object interactions and dependencies such as shadows, reflections
and occlusions. In particular, since GANformer2 creates the layouts
sequentially by laying segments on top of each other (e.g. first generating a
road, and then placing a car on top of it), it provides us with practical
means to then remove the front segments and reveal the ones behind them,
effectively achieving amodal completion of occluded objects. Consequently, by
varying the number of generation steps, GANformer2 is also capable of
extrapolating beyond the training data, e.g. creating empty CLEVR scenes
(figure 3) even though the training data features at least 3 objects at every
image.

Table 4: Paired vs. unpaired training

| Paired | Unpaired | Parallel
---|---|---|---
CLEVR | 4.70 | 6.58 | 6.72
FFHQ | 7.77 | 8.12 | 8.35
Cityscapes | 6.21 | 7.32 | 7.81
Bedrooms | 6.05 | 8.24 | 8.55
COCO | 21.58 | 25.02 | 27.41

Table 5: Hyperparameters

| Max | Latents | R1 reg
---|---|---|---
| #Latents | Overall Dim | weight ($\gamma$)
FFHQ | 20 | 128 | 10
CLEVR | 16 | 512 | 40
Cityscapes | 64 | 512 | 20
Bedroom | 64 | 512 | 100
COCO | 64 | 512 | 100

Table 6: Dataset configurations

Dataset | Size | Resolution | Augment
---|---|---|---
FFHQ | 70,000 | $256{\times}256$ | Flip
CLEVR | 100,015 | $256{\times}256$ | None
Cityscapes | 24,998 | $256{\times}256$ | Flip
Bedrooms | 3,033,042 | $256{\times}256$ | None
COCO | 287,330 | $256{\times}256$ | Crop + Flip

Table 7: COCO subset statistics Category | Size | Category | Size
---|---|---|---
People | 50293 | Sports | 37304
Children | 13840 | Skiing | 5840
Eating | 5096 | Baseball | 10067
Playing | 5798 | Tennis | 11721
Rural | 12084 | Skating | 6412
Others | 13475 | Surfing | 3264
Animals | 64066 | Indoors | 30946
Dogs | 5245 | Bathrooms | 17121
Cats | 5543 | Kitchens | 10427
Birds | 8153 | Bedrooms | 3398
Sheep | 8134 |  |
Bears | 8138 | Outdoors | 20713
Elephants | 16204 | Beaches | 8396
Giraffes | 6352 | Cities | 8578
Zebras | 6297 | Streets | 3739
Vehicles | 53714 | Misc | 30294
Airplanes | 21232 | Desserts | 11370
Buses | 7436 | Food | 4176
Trains | 6661 | Toys | 7242
Bikes | 18385 | Electronics | 7506

##  Appendix D Structural Losses Comparison

As discussed in section 3.3, we introduce two new losses to the model’s
execution stage, where we transform input layouts into output photo-realistic
images. The new losses of  Semantic Matching and  Segment Fidelity
respectively encourage structural consistency between the layouts and the
images, and fidelity at the level of the individual segment. In figure 22, we
compare their performance in terms of FID score with several alternative
objectives over the COCO dataset.

Specifically, we explore the following baselines: (1) Using  no consistency
loss at all (training with the standard fidelity loss only
$\mathcal{L}(D(\cdot))$), in hopes that the model’s layout-conditioned feature
modulation will serve as an architectural bias to promote structural
alignment; (2) A simple  concatenation of the layout $S$ and image $X$ as they
are fed into the discriminator $D$; (3) A  Feature-Marching loss, as widely
used in prior work, $\mathcal{L}_{FM}(f(X),f(X^{\prime}))$, that compares VGG
features of the generated image with those of the source natural image
$X^{\prime}$ that underlies the input layout $S$; and (4) an  Edge-Matching
loss, $\mathcal{L}_{EM}(e(S),e(S^{\prime}))$, that compares the binary
segmentation edges between the input layout $S$ and a layout $S^{\prime}$
induced by the generated image $X$.

(a)  Layout

(b)  Pix2PixHD

(c)  BicycleGAN

(d)  SPADE

(e)  GANformer2

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

Figure 23: Comparison between samples of conditional generative models.
GANformer2 achieves better visual quality and demonstrates a larger variance
in the spectrum of colors and textures, contrasting with other approaches that
converge to a narrow range of gray-brownish hues.

(a)  Layout

(b)  Pix2PixHD

(c)  BicycleGAN

(d)  SPADE

(e)  GANformer2

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

Figure 24: Comparison between samples of conditional generative models,
demonstrating the GANformer2’s wider diversity in object’s properties, its
higher compliance with the source layouts, especially in terms of shape
consistency, and the more precise separation between close object segments.

(a)  Layout

(b)  Pix2PixHD

(c)  BicycleGAN

(d)  SPADE

(e)  GANformer2

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

Figure 25: Comparison between samples of conditional generative models,
demonstrating the GANformer2’s wider diversity in object’s properties, its
higher compliance with the source layouts, and the more precise separation
between different object segments (continued).

As figure 22 shows, our newly proposed losses, and the Semantic-Matching loss
especially, surpass the discussed baselines and effectively encourage
GANformer2 to generate high-quality images. For the Semantic-Matching loss,
$\mathcal{L}_{SM}(S,S^{\prime})$, we believe that providing semantic pixel-
wise guidance to the generator in terms of the classes each region in the
image should depict, while not limiting its content to match an arbitrary
natural image, as is the case in feature-matching losses, accelerates the
generator’s learning without inhibiting its output diversity, which in turn
yields better FID scores. Likewise, for the Segment-Fidelity loss,
$\tfrac{1}{n}\sum\mathcal{L}_{SF}(D(s_{i},x_{i}))$, promoting fidelity of
individual segments $x_{i}$, rather than just of the whole picture $X$,
naturally enhances learning, especially for rich and highly-structured scenes,
as we observe for the COCO dataset.

##  Appendix E Model Ablations & Iterative Generation

To validate the efficacy of our approach and better assess the relative
contribution of each design choice, we perform ablation and variation studies
over the planning and execution stages. We begin by exploring the impact of
varying the number of recurrent generation steps for planning the scene
layout, and further compare it with a single-pass approach that generates the
layout in a non-compositonal fashion – as a one image like standard GANs,
rather then as a collection of interacting segments. We note that each
generation step creates a random number of segments, sampled from a trainable
normal distribution, and therefore reducing the number of steps does not limit
the maximum number of segments the model can create overall. Adding more
recurrent steps can instead enhance the model’s capability to capture
conditional dependencies across segments, while keeping the planning process
shorter can naturally increase the computational efficiency.

Compositional Layout Synthesis. As figure 22 shows, compositional layout
generation performs substantially better than the standard single-pass
approach (denoted by “0”) across all datasets. Intuitively, we believe that
the efficiency gains arise from the ability of the recurrent approach to
decompose the combinatorial space of possible scene layouts into several
smaller tasks, such that each step focuses on a few segments only, rather than
modeling the whole scene at once. This could be especially useful for highly-
structured scenes with multiple objects and dependencies.

Layout Synthesis Length. We can see how the optimal number of synthesis steps
vary between different datasets: while CelebA layouts seem simple enough for
the model to comprehend and synthesize in just one step, CLEVR and LSUN-
Bedrooms benefit from 2 steps, and the richer Cityscapes and COCO see
performance improvement in even 3 layout generation steps. The most effective
lengths seem to indicate the compositionality degree of each dataset: In
CLEVR, objects are mostly independent of each other (holding only weak
relations of not mutually occupying the same space), and so their segments can
be produced mostly in parallel, over less recurrent steps. Meanwhile, more
intricate scenes, as in COCO and Cityscapes, benefit from a longer sequential
generation that can explicitly capture conditional dependencies among objects
(e.g. generating a cup only after creating the table it is placed on),
demonstrating the strength of recurrent synthesis.

Layout Refinement. In the execution stage, which transforms layouts into
output images, we explore the contribution of the layout refinement mechanism
(section 3.3). It introduces a sigmoidal gate $\sigma(g(S,X,W))$ to support
local layout adjustments, meant to increase the model’s flexibility and
expressivity during the translation. We study ablations over CLEVR, either not
applying the refinement or using limited gating versions, constraining the
inputs to be the latents $W$, layout $S$, or image $X$ only. We see that
compared to the default model’s FID score of 4.70, using weaker refinements
lead to deterioration of 0.72, 0.78 and 0.85 when inputting the latents,
layout or image respectively, and a larger reduction of 1.45 points, when
ablating the gating mechanism completely. These results provide evidence for
the benefit of using the gating mechanism to refine the scene’s layout during
the execution stage.

##  Appendix F Paired vs. Unpaired Training

We train GANformer2 over two sets: of images $\\{X_{i}\\}$ and layouts
$\\{S_{i}\\}$, used during the planning and execution stages respectively. The
training sets can either be paired: listing the alignment between images and
layouts, namely $\\{(S_{i},X_{i})\\}$, or unpaired: $\\{X_{i}\\},\\{S_{i}\\}$,
and our training scheme accommodates both options: For the paired case, we
first train the planning stage over layouts $\\{S_{i}\\}$, and the execution
stage conditionally over $\\{(S_{i},X_{i})\\}$, such that it learns to
translate ground-truth layouts into output images. Then, to make both stages
work in tandem, we fine-tune them together, where this time the execution
stage translates generated layouts $\\{S^{g}_{i}\\}$ into output images
$\\{X^{g}_{i}\\}$.

For the unpaired case, we can either (1) train the planning stage over the
layouts $\\{S_{i}\\}$, and then fine-tune it together with the execution
stage, or (2) jointly train the two stages from scratch (we call this scheme
Parallel). Since the Segment-Fidelity loss assumes access to paired layout-
image samples, both fake and real, we do not use it in the unpaired and
parallel cases, and instead use the Edge-Matching loss, described in D. We
adjust the Edge-Matching and Semantic-Matching losses for both the generator
and the discriminator to be computed over generated pairs
$\\{(S^{g}_{i},X^{g}_{i})\\}$, which are available even when the training data
is unpaired. Table 7 compares the model performance between the paired,
unpaired and parallel settings for different datasets. The he model manages to
perform well even in the unpaired and parallel settings, but achieves
strongest results in the paired case.

##  Appendix G Implementation & Training Details

We implement all the unconditional methods within the shared GANformer
codebase [33], to ensures they are tested under comparable conditions in terms
of training details, model sizes, and optimization scheme. For the conditional
models, we use the authors’ official implementations, likewise implemented as
extensions to the Pix2PixHD repository for conditional generative modeling.
All approaches have been trained with images of $256\times 256$ resolution and
data augmentation as detailed in section I. See table 7 for the particular
settings and hyperparameters of each model. The overall latent dimension is
chosen based on performance among $\\{128,256,512\\}$. The R1 regularization
factor $\gamma$ is likewise chosen based on performance and training stability
among $\\{1,10,20,40,80,100\\}$.

In terms of the loss function, optimization and training configuration, we
adopt the settings and techniques used in StyleGAN2 and GANformer [42, 33],
including in particular style mixing, Xavier Initialization, stochastic
variation, exponential moving average for weights, and a non-saturating
logistic loss with lazy R1 regularization. We use Adam optimizer with batch
size of 32 (4 times 8 using gradient accumulation), equalized learning rate of
$0.001$, $\beta_{1}=0.9$ and $\beta_{1}=0.999$ as well as leaky ReLU
activations with $\alpha=0.2$, bilinear filtering in all up/downsampling
layers and minibatch standard deviation layer at the end of the discriminator.
The mapping layer of the generator consists of 8 layers, and ResNet
connections are used throughout the model, for the mapping network, synthesis
network and discriminator. All models have been trained for the same number of
training steps, roughly spanning 10 days on 1 NVIDIA V100 GPU per model.

##  Appendix H Baselines & Prior Approaches

We compare GANformer2 to both unconditional and conditional generative models.
First, we inspect unconditional methods which synthesize images from scratch,
including in particular: (1) a baseline GAN [23], (2) the StyleGAN2 [42]
model, (3) SAGAN [81] which utilizes self-attention across spatial regions,
(4) k-GAN [69] that blends together $k$ generated images through alpha-
composition, (5) VQGAN [21], a visual autoregessive autoencoder, (6) SBGAN
[5], a non-compositional two-stage approach, and (7) the original GANformer
[33].

We then proceed to compare our execution stage to popular conditional semantic
generation models, including the aforementioned SBGAN and also: (8) Pix2PixHD
[72] which uses a U-Net [62] to translate source to target images, (9)
BicycleGAN [86] that promotes cycle consistency among domains, and (10) SPADE
[60], which performs spatial modualtion using a fixed set of trainable
semantic category vectors. For disentanglement and controllability
experiments, we compare the GANformer2 to a baseline GAN, StyleGAN2 and
GANformer, as well as: (11) MONet and (12) Iodine, two sequential variational
autoencoders.

##  Appendix I Data Preparations

We train all models on images of $256\times 256$ resolution, padded as
necessary. See dataset statistics in table 7. The images in the Cityscapes and
FFHQ datasets are mirror-augmented, while the images in the COCO dataset are
both mirror-augmented and also randomly cropped, to increase the effective
training set size. We assume access to a training data of image and panoptic
segmentations [43], indicating the segment unique identity and its semantic
class. Contrary to prior conditional works which rely on costly hand-annotated
ground-truth segmentations, and to demonstrate the model robustness, we
instead intentionally explore training on auto-predicted segmentations (either
produced in an unsupervised manner [39] for CLEVR [37] or by pre-trained
segmentor [74] otherwise).

For the COCO dataset [48], we note that it introduces challenges from two
perspectives, being both highly-structured, with each scene populated by many
objects that hold intricate relations and dependencies, but also visually and
semantically diverse, consisting of varied images from a wide range of
domains. In order to isolate the former challenge of modeling compositional
scenes from the latter important but different challenge of covering a diverse
image distribution, we study training on a topical partition of COCO, named
COCOp, that groups images into 7 semantically-related splits, listed in table
7. To partition the dataset, we cluster t-SNE processed ResNet activations of
the COCO images into 31 subsets, which are then semantically grouped into the
7 splits. We train the models on each split separately, and report the mean
scores. As expected, and also empirically suggested by table 1, the partition
leads to improved visual quality across all models – both baselines and new
ones, likely due to the more uniform resulting training distributions.