# Jarzynski equality for quasistatic work

Pierre Nazé<EMAIL_ADDRESS>Departamento de Física, Instituto de
Geociências e Ciências Exatas, Universidade Estadual Paulista “Júlio de
Mesquita Filho”, 13506-900, Rio Claro, SP, Brazil

###### Abstract

Abstract

To prove a fluctuation theorem that connects a work with the average
quasistatic work, I show that a thermally isolated system performing an
adiabatic driven process has associated another system performing an
isothermal driven one. This isothermal process has a temperature equal to that
one of the initial equilibrium state of the thermally isolated system, and its
difference of Helmholtz free energy between the final and initial equilibrium
states is equal to the average quasistatic work of the original process. The
application of the usual Jarzynski equality for this isothermal process is
therefore straightforward. However, for the complexity of the new system,
check proofs corroborating this Jarzynski equality are lacking.

Introduction. Jarzynski equalities (or fluctuations theorems) of all types
have been proved over the last decades [1, 2, 3]. However, few attempts have
been made to prove it in the context of showing a connection between the
fluctuations of a work performed and the average quasistatic work [4]. In this
work, for thermally isolated systems, I prove

$\langle e^{-\beta_{0}W_{\rm ad}}\rangle=e^{-\beta_{0}W_{\rm qs}},$ (1)

where $W_{\rm ad}$ is a work performed, $W_{\rm qs}$ is the average
quasistatic work, and $\beta_{0}:=(k_{B}T_{0})^{-1}$, with $k_{B}$ being
Boltzmann constant, and $T_{0}$ the temperature of the initial thermal
equilibrium of the thermally isolated system. The operation
$\langle\cdot\rangle$ is the non-equilibrium average of the associated system.

Associated isothermal. Consider a system with Hamiltonian $\mathcal{H}$
dependent on some external parameter
$\lambda(t)=\lambda_{0}+g(t)\delta\lambda$, with $g(0)=0$ and $g(\tau)=1$,
where $\tau$ is the switching time of the process. Initially, the system is in
thermal equilibrium at a temperature $T_{0}$. When the process starts, the
system is decoupled from the heat bath and adiabatically evolves in time, that
is, without any source of heat. The average work performed along the process
is

$\langle
W\rangle=\int_{0}^{\tau}\langle\partial_{\lambda}\mathcal{H}\rangle(t)\dot{\lambda}(t)dt.$
(2)

In particular, for quasistatic processes, we called it average quasistatic
work

$W_{\rm qs}=\lim_{\tau\rightarrow\infty}\langle W\rangle(\tau).$ (3)

The idea to prove equality (1) is to find a different system, performing an
isothermal process in a temperature $T_{0}$, whose difference of Helmholtz
free energy between the final and initial state is equal to the average
quasistatic work of the original system. In this manner, the application of
Jarzynski equality is straightforward.

We start defining a system of interest, with a not-known yet Hamiltonian
$\mathcal{H}_{\rm ad}({\bf z}({\bf z_{0}},t),\lambda(t))$, and a heat bath of
temperature $T_{0}$, with Hamiltonian $\mathcal{H}_{\rm bath}({\bf
z^{\prime}}({\bf z^{\prime}_{0}},t))$. Here, ${\bf z}$ and ${\bf z^{\prime}}$
are points in the phase space evolved accordingly to their respective
Hamiltonians from its initial points ${\bf z_{0}}$ and ${\bf z^{\prime}_{0}}$
until the instant of time $t$. Both systems are weakly coupled, such that the
total system is given by the Hamiltonian $\mathcal{H}_{\rm total}$. It is
expected that the system of interest at the end of our argument crucially
depends on the adiabatic process, which would justify its index.

The average work performed by the system of interest when the parameter is
changed from $\lambda_{0}$ to $\lambda_{0}+\delta\lambda$ in a process of
switching time $\tau$ is

$\langle W_{\rm
ad}\rangle(\tau)=\int_{0}^{\tau}\langle\partial_{\lambda}\mathcal{H}_{\rm
ad}\rangle(t)\dot{\lambda}(t)dt.$ (4)

It is not hard to see that it holds the equality of the First Law of
Thermodynamics

$\langle U_{\rm ad}\rangle=\langle W_{\rm ad}\rangle+\langle Q_{\rm
ad}\rangle,$ (5)

where the average energy of the system of interest and the average heat of the
process are

$\langle U_{\rm ad}\rangle=\langle\mathcal{H}_{\rm ad}\rangle|_{0}^{\tau},$
(6) $\langle Q_{\rm ad}\rangle=\langle\mathcal{H}_{\rm
bath}\rangle|_{0}^{\tau}.$ (7)

Remark that the non-equilibrium averages were taken considering as the non-
equilibrium probability distribution the solution of Liouville equation

$\frac{\partial\rho_{\rm ad}}{\partial t}=\mathcal{L}_{\rm ad}\rho_{\rm ad},$
(8)

where $\mathcal{L}_{\rm ad}:=-\\{\cdot,\mathcal{H}_{\rm total}\\}$ is the
Liouville operator. Here, $\\{\cdot,\cdot\\}$ is the Poisson bracket. The
initial probability distribution is the canonical one

$\rho_{\rm ad}(0)=\frac{e^{-\beta_{0}\mathcal{H}_{\rm
total}}}{\mathcal{Z}_{\rm total}},$ (9)

with $\mathcal{Z}_{\rm total}$ being the partition function of the total
system. We respectively define now the average internal and external entropies
of the system of interest

$\langle S_{\rm ad}^{i}\rangle=\frac{1}{T_{0}}(\langle W_{\rm ad}\rangle-
W_{\rm qs}),$ (10) $\langle S_{\rm ad}^{e}\rangle=\frac{\langle Q_{\rm
ad}\rangle}{T_{0}}.$ (11)

The average entropy of the system of interest is

$\langle S_{\rm ad}\rangle=\langle S^{i}_{\rm ad}\rangle+\langle S^{e}_{\rm
ad}\rangle.$ (12)

Applying Eq. (5) in Eq. (12), we obtain

$W_{\rm qs}=\langle U_{\rm ad}\rangle-T_{0}\langle S_{\rm ad}\rangle,$ (13)

which holds for any switching time, in particular for quasistatic processes.
Equation (13) express that the system of interest is performing a process in
contact with a heat bath of temperature $T_{0}$, such that its difference of
Helmholtz free energy is equal to the average quasistatic work of the
adiabatic process.

Jarzynski equality. It remains to find the Hamiltonians $\mathcal{H}_{\rm ad}$
and $\mathcal{H}_{\rm bath}$ of the system of interest and heat bath. In order
to determine them, I consider the regime of quasistatic processes. First, I
express

$W_{\rm qs}=\langle E_{\rm
ad}\rangle_{0}|_{\lambda_{0}}^{\lambda_{0}+\delta\lambda},$ (14)

where $\langle\cdot\rangle_{0}$ is the canonical ensemble of the Hamiltonian
$\mathcal{H}(\lambda_{0})$ calculated at $T_{0}$. The partition function of
the system of interest will be

$\mathcal{Z}_{\rm ad}=e^{-\beta_{0}\langle E_{\rm ad}\rangle_{0}}.$ (15)

To find $\mathcal{H}_{\rm ad}$, I use the expression for the average energy in
thermal equilibrium states

$\int\mathcal{H}_{\rm ad}e^{-\beta_{0}\mathcal{H}_{\rm ad}}d{\bf
z_{0}}=-\partial_{\beta_{0}}\mathcal{Z}_{\rm ad}.$ (16)

Using Eq. (15), we have

$\int\mathcal{H}_{\rm ad}e^{-\beta_{0}\mathcal{H}_{\rm ad}}d{\bf
z_{0}}=\int\mathcal{J}e^{-\beta_{0}\mathcal{H}}d{\bf z_{0}},$ (17)

where

$\mathcal{J}=[(1+\beta_{0}\langle\mathcal{H}\rangle_{0})E_{\rm
ad}-\beta_{0}E_{\rm ad}\mathcal{H}]e^{-\beta_{0}\langle E_{\rm
ad}\rangle_{0}}.$ (18)

Assuming that the coordinates of the phase spaces of the system of interest
and of the thermally isolated system are the same, we equal the integrands of
Eq. (17) and solve the equation for $\mathcal{H}_{\rm ad}$. We have

$\mathcal{H}_{\rm
ad}(\lambda)=-\frac{1}{\beta_{0}}\mathcal{W}_{k}(-\beta_{0}\mathcal{J}(\lambda)e^{-\beta_{0}\mathcal{H}(\lambda)}),$
(19)

where $\mathcal{W}_{k}$ is Lambert’s $\mathcal{W}$ function of branch $k$.
Observe that $\mathcal{H}_{\rm ad}$ crucially depends on $\mathcal{H}$. Also,
in principle, for each branch $k$ we have a new system of interest.

Finally, about the heat bath, we can assume that it is the same as the one of
the preparation of the original system. This at least solves the problem that
both baths must have the same coordinates in their phase spaces.

Since now every term of the isothermal process performed is of our knowledge,
from Ref. [5] it holds the equality

$\langle e^{-\beta_{0}W_{\rm ad}}\rangle=e^{-\beta_{0}W_{\rm qs}}.$ (20)

Also, using Jensen’s inequality, we have

$\langle W_{\rm ad}\rangle\geq W_{\rm qs},$ (21)

showing that $\langle S_{\rm ad}^{i}\rangle\geq 0$, as it was expected for an
average internal entropy. Also, given the complexity of the new Hamiltonian
(19), it seems hard to find a relation between $\langle W_{\rm ad}\rangle$ and
$\langle W\rangle$. Therefore, the expected result $\langle W\rangle\geq
W_{\rm qs}$ is not so obvious to obtain.

Attempt of testing. In order to corroborate Jarzynski equality (1), we made an
attempt using the classical harmonic oscillator with a time-dependent linear
stiffness parameter

$\mathcal{H}=\frac{p^{2}}{2}+\left(\lambda_{0}+\frac{t}{\tau}\delta\lambda\right)\frac{q^{2}}{2}.$
(22)

The proceeding of finding analytically $\mathcal{H}_{\rm ad}$ is not so hard.
However, sampling the initial conditions from the canonical ensemble of the
system of interest is an impossible task at this moment. Indeed, this
probability distribution is very involved, and numerical methods furnishing
this sampling have not been implemented by the available mathematical software
of the community (e.g. Mathematica).

Conclusion. I proved a Jarzynski equality relating a work with the average
quasistatic work. Because of the complexity of the new system of interest, the
expected result $\langle W\rangle\geq W_{\rm qs}$ is not so obvious to obtain.
Also, for the same reason, mathematical software can not sample numerical
initial conditions to be used in the evaluation of the averages for the
initial canonical ensemble. Therefore, check proofs corroborating this
Jarzynski equality are lacking.

## References

  * Sevick  _et al._ [2008] E. M. Sevick, R. Prabhakar, S. R. Williams, and D. J. Searles, Annu. Rev. Phys. Chem. 59, 603 (2008).
  * Esposito  _et al._ [2009] M. Esposito, U. Harbola, and S. Mukamel, Reviews of modern physics 81, 1665 (2009).
  * Seifert [2012] U. Seifert, Reports on progress in physics 75, 126001 (2012).
  * Jarzynski [2020] C. Jarzynski, Physica A: Statistical Mechanics and its Applications 552, 122077 (2020).
  * Jarzynski [1997] C. Jarzynski, Phys. Rev. Lett. 78, 2690 (1997).


# LayoutDiffusion: Controllable Diffusion Model for
Layout-to-image Generation

Guangcong Zheng1 , Xianpan Zhou211footnotemark: 1 , Xuewei Li1 , Zhongang Qi3,
Ying Shan3, Xi Li1,4,5,622footnotemark: 2
1College of Computer Science & Technology, Zhejiang University
2Polytechnic Institute, Zhejiang University 3ARC Lab, Tencent PCG
4Shanghai Institute for Advanced Study of Zhejiang University
5Shanghai AI Lab 6Zhejiang – Singapore Innovation and AI Joint Research Lab
{guangcongzheng, zhouxianpan, xueweili<EMAIL_ADDRESS>
{zhongangqi<EMAIL_ADDRESS>The first two authors contributed equally
to this paper.Corresponding author.

###### Abstract

Recently, diffusion models have achieved great success in image synthesis.
However, when it comes to the layout-to-image generation where an image often
has a complex scene of multiple objects, how to make strong control over both
the global layout map and each detailed object remains a challenging task. In
this paper, we propose a diffusion model named LayoutDiffusion that can obtain
higher generation quality and greater controllability than the previous works.
To overcome the difficult multimodal fusion of image and layout, we propose to
construct a structural image patch with region information and transform the
patched image into a special layout to fuse with the normal layout in a
unified form. Moreover, Layout Fusion Module (LFM) and Object-aware Cross
Attention (OaCA) are proposed to model the relationship among multiple objects
and designed to be object-aware and position-sensitive, allowing for precisely
controlling the spatial related information. Extensive experiments show that
our LayoutDiffusion outperforms the previous SOTA methods on FID, CAS by
relatively 46.35$\%$, 26.70$\%$ on COCO-stuff and 44.29$\%$, 41.82$\%$ on VG.
Code is available at https://github.com/ZGCTroy/LayoutDiffusion.

Figure 1:  Compared to text, the layout allows diffusion models to obtain more
control over the objects while maintaining high quality. Unlike the prevailing
methods, we propose a diffusion model named LayoutDiffusion for layout-to-
image generation. We transform the difficult multimodal fusion of the image
and layout into a unified form by constructing a structural image patch with
region information and regarding the patched image as a special layout.

**footnotetext: Equal contribution.$\dagger$$\dagger$footnotetext:
Corresponding author.

##  1 Introduction

Recently, the diffusion model has achieved encouraging progress in conditional
image generation, especially in text-to-image generation such as GLIDE [25],
Imagen [33], and Stable Diffusion [32]. However, text-guided diffusion models
may still fail in the following situations. As shown in Fig. 1 (a), when
aiming to generate a complex image with multiple objects, it is hard to design
a prompt properly and comprehensively. Even input with well-designed prompts,
problems such as missing objects and incorrectly generating objects’
positions, shapes, and categories still occur in the state-of-the-art text-
guided diffusion model [25, 32, 33]. This is mainly due to the ambiguity of
the text and its weakness in precisely expressing the position of the image
space. Fortunately, this is not a problem when using the coarse layout as
guidance, which is a set of objects with the annotation of the bounding box
(bbox) and object category. With both spatial and high-level semantic
information, the diffusion model can obtain more powerful controllability
while maintaining the high quality.

However, early studies [14, 2, 41, 45] on layout-to-image generation are
almost limited to generative adversarial networks (GANs) and often suffer from
unstable convergence [1] and mode collapse [28]. Despite the advantages of
diffusion models in easy training [10] and significant quality improvement
[7], few studies have considered applying diffusion in the layout-to-image
generation task. To our knowledge, only LDM [32] supports the condition of
layout and has shown encouraging progress in this field.

In this paper, different from LDM that applies the simple multimodal fusion
method (e.g., the cross attention) or direct input concatenation for all
conditional input, we aim to specifically design the fusion mechanism between
layout and image. Moreover, instead of conditioning only in the second stage
like LDM, we propose an end-to-end one-stage model that considers the
condition for the whole process, which may have the potential to help mitigate
loss in the task that requires fine-grained accuracy in pixel space [32]. The
fusion between image and layout is a difficult multimodal fusion problem.
Compared to the fusion of text and image, the layout has more restrictions on
the position, size, and category of objects. This requires a higher
controllability of the model and often leads to a decrease in the naturalness
and diversity of the generated image. Furthermore, the layout is more
sensitive to each token and the loss in token of layout will directly lead to
the missing objects.

To address the problems mentioned above, we propose treating the patched image
and the input layout in a unified form. Specifically, we construct a
structural image patch at multi-resolution by adding the concept of region
that contains information of position and size. As a result, each patch of the
image is transformed into a special type of object, and the entire patched
image will also be regarded as a layout. Finally, the difficult problem of
multimodal fusion between image and layout will be transformed into a simple
fusion with a unified form in the same spatial space of the image. We name our
model LayoutDiffuison, a layout-conditional diffusion model with Layout Fusion
Module (LFM), object-aware Cross Attention Mechanism (OaCA), and corresponding
classifier-free training and sampling scheme. In detail, LFM fuses the
information of each object and models the relationship among multiple objects,
providing a latent representation of the entire layout. To make the model pay
more attention to the information related to the object, we propose an object-
aware fusion module named OaCA. Cross-attention is made between the image
patch feature and layout in a unified coordinate space by representing the
positions of both of them as bounding boxes. To further improve the user
experience of LayoutDiffuison, we also make several optimizations on the speed
of the classifier-free sampling process and could significantly outperform the
SOTA models in 25 iterations.

Experiments are conducted on COCO-stuff [5] and Visual Genome (VG) [19].
Various metrics ranging from quality, diversity, and controllability show that
LayoutDiffusion significantly outperforms both state-of-the-art GAN-based and
diffusion-based methods.

Our main contribution is listed below.

  * •

Instead of using the dominated GAN-based methods, we propose a diffusion model
named LayoutDiffusion for layout-to-image generations, which can generate
images with both high-quality and diversity while maintaining precise control
over the position and size of multiple objects.

  * •

We propose to treat each patch of the image as a special object and accomplish
the difficult multimodal fusion of layout and image in a unified form. LFM and
OaCA are then proposed to fuse the multi-resolution image patches with user’s
input layout.

  * •

LayoutDiffuison outperforms the SOTA layout-to-image generation method on FID,
DS, CAS by relatively around 46.35$\%$, 9.61$\%$, 26.70$\%$ on COCO-stuff and
44.29$\%$, 11.30$\%$, 41.82$\%$ on VG.

##  2 Related work

The related works are mainly from layout-to-image generation and diffusion
models.

Layout-to-Image Generation. Before the layout-to-image generation is formally
proposed, the layout is usually used as as a complementary feature [15, 31,
43] or an intermediate representation in text-to-image [12], scene-to-image
generation [14]. The first image generation directly from the layout appears
in Layout2Im [50] and is defined as a set of objects annotated with category
and bbox. Models that work well with fine-grained semantic maps at the pixel
level can also be easily transformed to this setting [27, 13, 46]. Inspired by
StyleGAN [16], LostGAN-v1 [40], LostGAN-v2 [41] used a reconfigurable layout
to obtain better control over individual objects. For interactive image
synthesis, PLGAN [45] employed panoptic theory [18] by constructing stuff and
instance layouts into separate branches and proposed Instance- and Stuff-Aware
Normalization to fuse into panoptic layouts. Despite encouraging progress in
this field, almost all approaches are limited to the generative adversarial
network (GAN) and may suffer from unstable convergence [1] and mode collapse
[28]. As a multimodal diffusion model, LDM [32] supports the condition of
coarse layout and has shown great potential in layout-guided image generation.

#### Diffusion Model.

Diffusion models [36, 38, 10, 26, 39, 3, 32, 47] are being recognized as a
promising family of generative models that have proven to be state-of-the-art
sample quality for a variety of image generation benchmarks [6, 44, 48],
including class-conditional image generation [7, 51], text-to-image generation
[25, 32, 33], and image-to-image translation [34, 23, 17]. Classifier guidance
was introduced in ADM-G [7] to allow diffusion models to condition the class
label. The gradient of the classifier trained on noised images could be added
to the image during the sampling process. Then Ho et al. [11] proposed a
classifier-free training and sampling strategy by interpolating between
predictions of a diffusion model with and without condition input. For the
acceleration of training and sampling speed, LDM proposed to first compress
the image into smaller resolution and then apply denoising training in the
latent space.

Figure 2:  The whole pipeline of LayoutDiffusion. The layout that consisted of
bounding box $b$ and objects categories $c$ is transformed into embedding
$B_{\mathcal{L}},C_{\mathcal{L}},L$. Then Layout Fusion Module fuses layout
embedding $L$ to output the fused layout embedding $L^{\prime}$. Finally,
Image-Layout Fusion Module including direct addition used for global
conditioning and Object-aware Cross Attention (OaCA) used for local
conditioning, will fuse the layout related
$B_{\mathcal{L}},C_{\mathcal{L}},L^{\prime}$ and the image feature $I$ at
multiple resolutions.

##  3 Method

In this section, we propose our LayoutDiffusion, as shown in Fig. 2. The whole
framework consists mainly of four parts: (a) layout embedding that
preprocesses the layout input, (b) layout fusion module that encourages more
interaction between objects of layout, (c) image-layout fusion module that
constructs the structal image patch and object-aware cross attention developed
with the specific design for layout and image fusion, (d) the layout-
conditional diffusion model with training and accelerated sampling methods.

###  3.1 Layout Embedding

A layout $l\\!=\\!\\{o_{1},o_{2},\cdots,o_{n}\\}$ is a set of $n$ objects.
Each object $o_{i}$ is represented as $o_{i}\\!=\\!\\{b_{i},c_{i}\\}$, where
$b_{i}\\!=\\!(x_{0}^{i},y_{0}^{i},x_{1}^{i},y_{1}^{i})\\!\in\\![0,1]^{4}$
denotes a bounding box (bbox) and $c_{i}\in[0,\mathcal{C}+1]$ is its category
id.

To support the input of a variable length sequence, we need to pad $l$ to a
fixed length $k$ by adding one $o_{l}$ in the front and some padding $o_{p}$
in the end, where $o_{l}$ represents the entire layout and $o_{p}$ represents
no object. Specifically, $b_{l}\\!=\\!(0,0,1,1)$, $c_{l}\\!=\\!0$ denotes a
object that covers the whole image and $b_{p}\\!=\\!(0,0,0,0)$,
$c_{p}\\!=\\!\mathcal{C}+1$ denotes a empty object that has no shape or does
not appear in the image.

After the padding process, we can get a padded
$l\\!=\\!\\{o_{1},o_{2},\cdots,o_{k}\\}\\!$ consisting of $k$ objects, and
each object has its specific position, size, and category. Then, the layout
$l$ is transformed into a layout embedding
$L\\!=\\!\\{O_{1},O_{2},\cdots,O_{k}\\}\\!\in\\!\mathbb{R}^{k\times
d_{\mathcal{L}}}$ by the projection matrix
$W_{\mathcal{B}}\in\mathbb{R}^{4\times d_{\mathcal{L}}}$ and
$W_{\mathcal{C}}\in\mathbb{R}^{1\times d_{\mathcal{L}}}$ using the following
equation:

$\displaystyle L=$ $\displaystyle\,B_{\mathcal{L}}+C_{\mathcal{L}}$ (1)
$\displaystyle B_{\mathcal{L}}$ $\displaystyle=bW_{\mathcal{B}}$ (2)
$\displaystyle C_{\mathcal{L}}$ $\displaystyle=cW_{C}$ (3)

where $B_{\mathcal{L}},C_{\mathcal{L}}\in\mathbb{R}^{k\times d_{\mathcal{L}}}$
are the bounding box embedding and the category embedding of a layout $l$,
respectively. As a result, $L$ is defined as the sum of $B_{\mathcal{L}}$ and
$C_{\mathcal{L}}$ to include both the content and positional information of a
entire layout, and $d_{\mathcal{L}}$ is the dimension of the layout embedding.

###  3.2 Layout Fusion Module

Currently, each object in layout has no relationship with other objects. This
leads to a low understanding of the whole scene, especially when multiple
objects overlap and block each other. Therefore, to encourage more interaction
between multiple objects of the layout to better understand the entire layout
before inputting the layout embedding, we propose Layout Fusion Module (LFM),
a transformer encoder that uses multiple layers of self-attention to fuse the
layout embedding and can be denoted as

$\displaystyle L^{\prime}$ $\displaystyle=\operatorname{LFM}(L)$ (4)

, where the output is a fused layout embedding
$L^{\prime}=\\{O_{1}^{\prime},O_{2}^{{}^{\prime}},\cdots,O_{k}^{{}^{\prime}}\\}\in\mathbb{R}^{k\times
d_{\mathcal{L}}}$.

###  3.3 Image-Layout Fusion Module

Structural Image Patch. The fusion of image and layout is a difficult
multimodal fusion problem, and one of the most important parts lies in the
fusion of position and size. However, the image patch is limited to the
semantic information of the whole feature and lacks the spatial information.
Therefore, we construct a structural image patch by adding the concept of
region that contains the information of position and size.

Specifically, $I\in\mathbb{R}^{h\times w\times d_{\mathcal{I}}}$ denotes the
feature map of a entire image with height $h$, width $w$, and channel
$d_{\mathcal{I}}$. We define that $I_{u,v}$ is the $u^{\text{th}}$ row and
$v^{\text{th}}$ column patch of $I$ and its bounding box, or the ablated
region information, is defined as $b_{I_{u,v}}$ by the following equation:

$\displaystyle b_{\mathcal{I}_{u,v}}$
$\displaystyle=(\frac{u}{h}\text{,}\frac{v}{w}\text{,}\frac{u\\!+\\!1}{h}\text{,}\frac{v\\!+\\!1}{w})$
(5)

The bounding box sets of a patched image $I$ is defined as
$b_{\mathcal{I}}\\!=\\!\\{b_{\mathcal{I}_{u,v}}|u\in[0,h),v\in[0,w)\\}$. As a
result, the positional information of image patch and layout object is
contained in the unified bounding box defined in the same spatial space,
leading to better fusion of image and layout.

Positional Embedding in Unified Space. We define the positional embedding of
the image and layout as $P_{\mathcal{I}}$ and $P_{\mathcal{L}}$ as follows:

$\displaystyle B_{\mathcal{I}}$ $\displaystyle=b_{\mathcal{I}}W_{\mathcal{B}}$
(6) $\displaystyle P_{\mathcal{I}}$
$\displaystyle=B_{\mathcal{I}}W_{\mathcal{P}}$ (7) $\displaystyle
P_{\mathcal{L}}$ $\displaystyle=B_{\mathcal{L}}W_{\mathcal{P}}$ (8)

, where $W_{\mathcal{B}}\\!\in\\!\mathbb{R}^{4\times d_{\mathcal{L}}}$ is
defined in Eq. 2 and works as a shared projection matrix that transforms the
coordinates of bounding box into embedding of $d_{\mathcal{L}}$ dimension.
$W_{\mathcal{P}}\\!\in\\!\mathbb{R}^{d_{\mathcal{L}}\times d_{\mathcal{I}}}$
is the projection matrix that transforms the $B$ to the positional Embedding
$P$.

#### Pointwise Addition for Global Conditioning.

With the help of LFM in Eq. 4, $O_{1}^{\prime}$ can be considered as a global
information of the entire layout, and $O_{i}^{\prime}(i\in[2,k])$ is
considered as the local information embedding of single object along with the
other related objects. One of the easiest ways to condition the layout in the
image is to directly add $O_{1}^{\prime}$, the global information of the
layout, to the multiple resolution of image features. Specifically, the
condition process can be defined as

$I^{{}^{\prime}}=I+O_{1}^{\prime}W$ (9)

, where $W\in\mathbb{R}^{d_{\mathcal{L}}\times d_{\mathcal{I}}}$ is a
projection matrix and $I^{\prime}$ is the image feature conditioned with
global embedding of layout.

#### Object-aware Cross Attention for Local Conditioning.

Cross attention is successfully applied in [25] to condition text into image
feature, where the sequence of the image patch is used as the query and the
concatenated sequence of the image patch and text is applied as key and value.
The equation of cross-attention is defined as

$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{Q{K}^{T}}{\sqrt{d_{k}}}\right)V$
(10)

, where $Q$, $K$, $V$ represent the embeddings of query, key, and value,
respectively. In the following paper, we will use the subscript image and
layout to represent the image patch feature and layout feature, respectively.

In text-to-image generation, each token in the text sequence is a word. The
aggregation of these words constitutes the semantics of a sentence. After the
transformer encoder, the first token in text sequence is well-semantic
information that generalizes the whole text but may not reverse the semantic
meaning of each word. However, the loss in information of one token is
relatively serious in layout rather than in text. Each token in the layout
sequence is a single object with a specific category, size, and position. The
loss of information on a layout token will directly lead to a missing or wrong
object in the generated image pixel space.

Therefore, we take into account the fusion of locations, size, and category of
objects and define our object-aware cross-attention (OaCA) as

$\displaystyle Q$
$\displaystyle=\operatorname{{\Psi}_{1}}(Q_{\mathcal{I}},Q_{\mathcal{L}})$
(11) $\displaystyle K$
$\displaystyle=\operatorname{{\Psi}_{1}}(\operatorname{{\Psi}_{2}}(K_{\mathcal{I}},K_{\mathcal{L}}),\operatorname{{\Psi}_{2}}(P_{\mathcal{I}},P_{\mathcal{L}}))$
(12) $\displaystyle V$
$\displaystyle=\operatorname{{\Psi}_{2}}(V_{\mathcal{I}},V_{\mathcal{L}})$
(13)

, where the query $Q\in\mathbb{R}^{hw\times 2d_{\mathcal{I}}}$,
$K\in\mathbb{R}^{(hw+k)\times 2d_{\mathcal{I}}}$, and
$V\in\mathbb{R}^{(hw+k)\times d_{\mathcal{I}}}$. ${\Psi}_{1}$ and $\Psi_{2}$
denote concatenation on the dimension of the channel and length of the
sequence, respectively.

We first construct the key and value of the layout:

$\displaystyle K_{\mathcal{L}},V_{\mathcal{L}}$
$\displaystyle=\operatorname{Conv}(\frac{1}{2}(\operatorname{Norm}(C_{\mathcal{L}})+L^{\prime}))$
(14)

, where $K_{\mathcal{L}},V_{\mathcal{L}}\in\mathbb{R}^{k\times
d_{\mathcal{I}}}$ and $\operatorname{Conv}$ is the convolution operation. The
embedding of key and value in the layout is related to the category embedding
$C_{\mathcal{L}}$ and the fused layout embedding $L^{\prime}$.
$C_{\mathcal{L}}$ focuses on the category information of layout and
$L^{\prime}$ concentrates on the comprehensive information of both the object
itself and other objects that may have a relationship with it. By averaging
between $L^{\prime}$ and $C_{\mathcal{L}}$, we can obtain both the general
information of the object and also emphasize the category information of the
object.

We construct the query, key, and value of the image feature as follows:

$\displaystyle
Q_{\mathcal{I}},K_{\mathcal{I}},V_{\mathcal{I}}=\operatorname{Conv}(\operatorname{Norm}(I))$
(15)

###  3.4 Layout-conditional Diffusion Model

Here, we follow the Gaussian diffusion models improved by [38, 10]. Given a
data point sampled from a real data distribution $x_{0}\sim q(x_{0})$, a
forward diffusion process is defined by adding small amount of Gaussian noise
to the $x_{0}$ in $T$ steps:

$\displaystyle
q(x_{t}|x_{t-1})\coloneqq\mathcal{N}(x_{t};\sqrt{\alpha_{t}}x_{t-1},(1-\alpha_{t})\mathbf{I})$
(16)

If the total noise added throughout the Markov chain is large enough, the
$x_{T}$ will be well approximated by $\mathcal{N}(0,\mathbf{I})$. If we add
noise at each step with a sufficiently small magnitude $1-\alpha_{t}$, the
posterior $q(x_{t-1}|x_{t})$ will be well approximated by a diagonal Gaussian.
This nice property ensures that we can reverse the above forward process and
sample from $x_{T}\sim\mathcal{N}(0,\mathbf{I})$, which is a Gaussian noise.
However, since the entire dataset is needed, we are unable to easily estimate
the posterior. Instead, we have to learn a model $p_{\theta}(x_{t-1}|x_{t})$
to approximate it:

$\displaystyle
p_{\theta}(x_{t-1}|x_{t})\coloneqq\mathcal{N}(\mu_{\theta}(x_{t}),\Sigma_{\theta}(x_{t}))$
(17)

Instead of using the tractable variational lower bound (VLB) in $\log
p_{\theta}(x_{0})$, Ho et al.[10] proposed to reweight the terms of the VLB to
optimize a surrogate objective. Specifically, we first add $t$ steps of
Gaussian noise to a clean sample $x_{0}$ to generate a noised sample
$x_{t}\sim q(x_{t}|x_{0})$. Then train a model $\epsilon_{\theta}$ to predict
the added noise using the following loss:

$\mathcal{L}\\!\coloneqq\\!E_{t\sim[1,T],x_{0}\sim
q(x_{0}),\epsilon\sim\mathcal{N}(0,\mathbf{I})}[||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}]$
(18)

, which is a standard mean-squared error loss.

To support the layout condition, we apply classifier-free guidance, a
technique proposed by Ho et al. [11] for conditional generation that requires
no additional training of the classifier. It is accomplished by interpolating
between predictions of a diffusion model with and without condition input. For
the condition of layout, we first construct a padding layout
$l_{\phi}=\\{o_{l},o_{p},\cdots,o_{p}\\}$. During training, the condition of
layout $l$ of diffusion model will be replaced with $l_{\phi}$ with a fixed
probability. When sampling, the following equation is used to sample a layout-
condional image:

$\hat{\epsilon}_{\theta}(x_{t},t|l)=(1-s)\\!\cdot\\!\epsilon_{\theta}(x_{t},t|l_{\phi})+s\\!\cdot\\!\epsilon_{\theta}(x_{t},t|l)$
(19)

, where the scale $s$ can be used to increase the gap between
$\epsilon_{\theta}(x_{t},t|l_{\phi})$ and $\epsilon_{\theta}(x_{t},t|l)$ to
enhance the strength of conditional guidance.

To further improve the user experience of LayoutDiffuison, we also make
several optimizations on the speed of the classifier-free sampling process and
could significantly outperform the SOTA models in 25 iterations. Specifically,
we adapt DPM-solver [22] for the conditional classifier-free sampling, a fast
dedicated high-order solver for diffusion ODEs [39] with the convergence order
guarantee, to accelerate the conditional sampling speed.

Figure 3: Visualization of comparision with SOTA methods on COCO-stuff
256$\times$256\. LayoutDiffusion has better generation quality and stronger
controllability compared to the other methods.

Methods | COCO-stuff | VG
---|---|---
FID $\downarrow$ | IS $\uparrow$ | DS $\uparrow$ | CAS $\uparrow$ | YOLOScore $\uparrow$ | FID $\downarrow$ | DS $\uparrow$ | CAS $\uparrow$
128$\times$128 |  |  |  |  |  |  |  |
Grid2Im [2] | 59.50 | 12.50±0.30 | 0.28±0.11 | 4.05 | 6.80 | - | - | -
LostGAN-v2 [41] | 24.76 | 14.21±0.40 | 0.45±0.09 | 39.91 | 13.60 | 29.00 | 0.42±0.09 | 29.74
PLGAN [45] | 22.70 | 15.60±0.30 | 0.16±0.72 | 38.70 | 13.40 | 20.62 | - | -
LayoutDiffusion | 16.57 | 20.17±0.56 | 0.47±0.09 | 43.60 | 27.00 | 16.35 | 0.49±0.09 | 36.45
256$\times$256 |  |  |  |  |  |  |  |
Grid2Im [2] | 65.20 | 16.40±0.70 | 0.34±0.13 | 4.81 | 9.70 | - | - | -
LostGAN-v2 [41] | 31.18 | 18.01±0.50 | 0.56±0.10 | 40.00 | 17.50 | 32.08 | 0.53±0.10 | 34.48
PLGAN [45] | 29.10 | 18.90±0.30 | 0.52±0.10 | 37.65 | 14.40 | 28.06 | - | -
LayoutDiffusion | 15.61 | 28.36±0.75 | 0.57±0.10 | 47.74 | 32.00 | 15.63 | 0.59±0.10 | 48.90

Table 1: Quantitative results on COCO-stuff [5] and VG [19]. The proposed
diffusion method has made great progress in all evaluation metrics, showing
better quality, controllability, diversity, and accuracy than previous works.
For COCO-stuff, we evaluate on 3097 layout and sample 5 images for each
layout. For VG, we evaluate on 5096 layout and sample 1 image for each layout.
We also report reproduction scores of previous works in Appendix.

LFM | OaCA | CA | FID $\downarrow$ | IS $\uparrow$ | DS $\uparrow$ | CAS $\uparrow$ | YOLOScore $\uparrow$
---|---|---|---|---|---|---|---
|  |  | 29.94 (+13.37) | 13.59±0.29 (-6.58) |  0.70±0.08 (+0.23) | 3.83 (-39.77) | 0.00 (-27.00)
✓ |  |  | 17.06 (+0.49) | 19.21±0.53 (-0.96) | 0.52±0.09 (+0.05) | 30.86 (-12.74) | 6.90 (-20.10)
| ✓ |  | 16.76 (+0.19) | 19.57±0.40 (-0.60) | 0.48±0.09 (+0.01) | 40.67 (-2.93) | 18.80 (-8.20)
✓ |  | ✓ |  16.46 (-0.11) | 19.79±0.40 (-0.38) | 0.48±0.10 (+0.01) | 42.47 (-1.13) | 23.60 (-3.40)
✓ | ✓ |  | 16.57 | 20.17±0.56 | 0.47±0.09 | 43.60 | 27.00

Table 2: Ablation study of Layout Fusion Module (LFM), Object-aware Cross
Attention (OaCA), Cross Attention (CA). We use the model trained for 300,000
iterations on COCO-stuff 128$\times$128\. The value in brackets denotes the
discrepancy to our proposed method(+LFM+OaCA), where red denotes better and
green denotes worse.

##  4 Experiments

In this section, we evaluate our LayoutDiffusion on different benchmarks in
terms of various metrics. First, we introduce the datasets and evaluation
metrics. Second, we show the qualitative and quantitative results compared
with other strategies. Finally, some ablation studies and analysis are also
mentioned. More details can be found in Appendix, including model
architecture, training hyperparameters, reproduction results, more
experimental results and visualizations.

###  4.1 Datasets

We conduct our experiments on two popular datasets, COCO-Stuff [5] and Visual
Genome [19].

COCO-Stuff has 164K images from COCO 2017, of which the images contain
bounding boxes and pixel-level segmentation masks for 80 categories of thing
and 91 categories of stuff, respectively. Following the settings of LostGAN-v2
[41], we use the COCO 2017 Stuff Segmentation Challenge subset that contains
40K / 5k / 5k images for train / val / test-dev set, respectively. We use
images in the train and val set with 3 to 8 objects that cover more than $2\%$
of the image and not belong to crowd. Finally, there are 25,210 train and
3,097 val images.

Visual Genome collects 108,077 images with dense annotations of objects,
attributes, and relationships. Following the setting of SG2Im [14], we divide
the data into 80$\%$, 10$\%$, 10$\%$ for the train, val, test set,
respectively. We select the object and relationship categories occurring at
least 2000 and 500 times in the train set, respectively, and select the images
with 3 to 30 bounding boxes and ignoring all small objects. Finally, the
training / validation / test set will have 62565 / 5062 / 5096 images,
respectively.

###  4.2 Evaluation Metrics & Protocols

We use five metrics to evaluate the quality, diversity, and controllability of
generation.

Fr‘echet Inception Distance (FID) [9] shows the overall visual quality of the
generated image by measuring the difference in the distribution of features
between the real images and the generated images on an ImageNet-pretrained
Inception-V3[42] network.

Inception Score (IS) [35] uses an Inception-V3[42] pretrained on ImageNet
network to compute the statistical score of the output of the generated
images.

Diversity Score (DS) calculates the diversity between two generated images of
the same layout by comparing the LPIPS [49] metric in a DNN feature space
between them.

Classification Score (CAS) [29] first crops the ground truth box area of
images and resizing them at a resolution of 32$\times$32 with their class. A
ResNet-101 [8] classifier is trained with generated images and tested on real
images.

YOLOScore [20] evaluates 80 thing categories bbox mAP on generated images
using a pretrained YOLOv4 [4] model, and shows the precision of control in one
generated model.

In summary, FID and IS show the generation quality, DS shows the diversity,
CAS and YOLOScore represent the controllability. We follow the architecture of
ADM [7], which is mainly a UNet. All experiments are conducted on 32 NVIDIA
3090s with mixed precision training [24]. We set batch size 24, learning rate
1e-5. We adopt the fixed linear variance schedule. More details can be found
in the Appendix.

###  4.3 Qualitative results

Comparison of generated 256 $\times$ 256 images on the COCO-Stuff [5] with our
method and previous works [2, 41, 45] is shown in Fig. 3.

method | FID $\downarrow$ | $N_{\text{parms}}$ | Throughout | First stage | Cond stage
---|---|---|---|---|---
|  | images / s | V100 days | V100 days
LDM-8 (100 steps) | 42.06 | 345M | 0.457 | 66 | 3.69
LDM-4 (200 steps) | 40.91 | 306M | 0.267 | 29 | 95.49
ours-small (25 steps) | 36.16 | 142M | 0.608 | - | 75.83
ours (25 steps) | 31.68 | 569M | 0.308 | - | 216.55

Table 3: Comparison with SOTA diffusion-based methods LDM on COCO-stuff
256$\times$256\. We generate the same 2048 images of LDM for a fair
comparision.

LayoutDiffusion generates more accurate high quality images, which has more
recognizable and accurate objects corresponding to their layouts. Grid2Im [2],
LostGAN-v2 [41] and PLGAN [45] generate images with distorted and unreal
objects.

Especially when input a set of multiple objects with complex relationships,
previous work can hardly generate recognizable objects in the position
corresponding to layouts. For example, in Fig. 3 (a), (c), and (e), the main
objects (e.g. train, zebra, bus) in images are poorly generated in previous
work, while our LayoutDiffusion generates well. In Fig. 3 (b), only our
LayoutDiffusion generates the laptop in the right place. The images generated
by our LayoutDiffusion are more sensorially similar to the real ones.

We show the diversity of LayoutDiffusion in Fig. 4. Images from the same
layouts have high quality and diversity (different lighting, textures, colors,
and details).

We continuously add an additional layout from the initial layout, the one in
the upper left corner, as shown in Fig. 5. In each step, LayoutDiffusion adds
the new object in very precise locations with consistent image quality,
showing user-friendly interactivity.

###  4.4 Quantitative results

Tab. 1 provides the comparison among previous works and our method in FID, IS,
DS, CAS and YOLOScore. Compared to the SOTA method, the proposed method
achieves the best performance in comparison.

In overall generation quality, our LayoutDiffusion outperforms the SOTA model
by 46.35% and 29.29% at most in FID and IS, respectively. While maintaining
high overall image quality, we also show precise and accurate controllability,
LayoutDiffusion outperforms the SOTA model by 122.22% and 41.82% at most on
YOLOScore and CAS, respectively. As for diversity, our LayoutDiffusion still
achieves 11.30% imporvement at most accroding to the DS. Experiments on these
metrics show that our methods can successfully generate the higher-quality
images with better location and quantity control.

In particular, we conduct experiments compared to LDM [32] in Tab. 3. “Ours-
small” uses comparable GPU resources to have better FID performance with much
fewer parameters and better throughout compared to LDM-8 when “Ours-small”
outperforms LDM-4 in all respects. The results of “Ours” indicate that
LayoutDiffusion can have better FID performance, 31.6, at a higher cost. From
these results, LayoutDiffusion always achieves better performance at different
cost levels compared with LDM [32].

###  4.5 Ablation studies

We validate the effectiveness of LFM and OaCA in Tab. 2, using the evaluation
metrics in Sec. 4.2. The significant improvement on FID, IS, CAS, and
YOLOScore proves that the application of LFM and OaCA allows for higher
generation quality and diversity, along with more controllability.
Furthermore, when applying both, considerable performance, 13.37 / 6.58 /
39.77 / 27.00 on FID / IS / CAS / YOLOScore, is gained.

An interesting phenomenon is that the change of the Diversity Score (DS) is in
the opposite direction of other metrics. This is because DS, which stands for
diversity, is physically the opposite of the controllability represented by
other metrics such as CAS and YOLOScore. The precise control offered on
generated image leads to more constraints on diversity. As a result, the
Diversity Score (DS) has a slight drop compared to the baseline.

Figure 4: The diversity of LayoutDiffusion. Each row of images are from the
same layout and have great difference.  Figure 5:  The interactivity of
LayoutDiffusion. We add extra layout continuously, and the new objects are
also with high quality.

##  5 Limitations & Societal Impacts

Limitations. Despite the significant improvements in various metrics, it is
still difficult to generate a realistic image with no distortion and overlap,
especially for a complex multi-object layout. Moreover, the model is trained
from scratch in the specific dataset that requires detection labels. How to
combine text-guided diffusion models and inherit parameters pre-trained on
massive text-image datasets remains a future research.

Societal Impacts. Trained on the real-world datasets such as COCO [5] and VG
[19], LayoutDiffusion has the powerful ability to learn the distribution of
data and we should pay attention to some potential copyright infringement
issues.

##  6 Conclusion

In this paper, we have proposed a one-stage end-to-end diffusion model named
LayoutDiffuison, which is novel for the task of layout-to-image generation.
With the guidance of layout, the diffusion model allows more control over the
individual objects while maintaining higher quality than the prevailing GAN-
based methods. By constructing a structural image patch with region
information, we regrad each patch as a special object and accomplish the
difficult multimodal image-layout fusion in a unified form. Specifically,
Layout Fusion Module and Object-aware Cross Attention are proposed to model
the relationship among multiple objects and fuse the patched image feature
with layout at multiple resolutions, respectively. Experiments in challenging
COCO-stuff and Visual Genome (VG) show that our proposed method significantly
outperforms both state-of-the-art GAN-based and diffusion-based methods in
various evaluation metrics.

Acknowledgements. This work is supported in part by National Natural Science
Foundation of China under Grant U20A20222, National Science Foundation for
Distinguished Young Scholars under Grant 62225605, National Key Research and
Development Program of China under Grant 2020AAA0107400, Research Fund of ARC
Lab, Tencent PCG, Zhejiang – Singapore Innovation and AI Joint Research Lab,
Ant Group through CCF-Ant Research Fund, and sponsored by CCF-AFSG Research
Fund, CAAI-HUAWEI MindSpore Open Fund as well as CCF-Zhipu AI Large Model
Fund(CCF-Zhipu202302).

Appendix for LayoutDiffusion

##  Appendix A More Visualizations

###  A.1 Controllable Layout Edit by modifying objects in number, position,
size, and category

Figure 6: Layout edit by adding objects. Figure 7: Layout edit by modifying
the position of objects. Figure 8: Layout edit by modifying the size of
objects. Figure 9: Layout edit by modifying the categories of objects.

###  A.2 More visualizations on COCO-stuff

Figure 10: More visualizations on COCO-stuff 256$\times$256\. LayoutDiffusion
is trained by 1.15M iterations, and sample images using scale=1.0 and dpm-
solver 25 steps. The COCO image IDs (from top to bottom) are 85195, 174004,
296969, 338560, 451090, 512248, 573008, 574425.

###  A.3 More visualizations on Visual Genome

Figure 11: More visualizations on Visual Genome 256$\times$256\.
LayoutDiffusion is trained by 1.45M iterations, and sample images using
scale=1.0 and dpm-solver 25 steps. The VG image IDs (from top to bottom) are
2380568, 2382403, 2382599, 2383021, 2383488, 2385225, 2385290, 2385812.

###  A.4 More comparision with previous methods

Figure 12: More comparision with previous methods on COCO-stuff
128$\times$128\. LayoutDiffusion is trained by 300K iterations, and sample
images using scale=0.6 and dpm-solver 25 steps. The COCO image IDs (from top
to bottom) are 2153, 2352, 4495, 6723, 10583, 17031, 18737, 543300. Figure 13:
More comparision with previous methods on COCO-stuff 256$\times$256\.
LayoutDiffusion is trained by 1.15M iterations, and sample images using
scale=1.0 and dpm-solver 25 steps. The COCO image IDs (from top to bottom) are
23781, 55299, 84477, 137950, 243034, 252701, 341719, 350405. Figure 14: More
comparision with previous methods on VG 128$\times$128\. LayoutDiffusion is
trained by 300K iterations, and sample images using scale=0.5 and dpm-solver
25 steps. The VG image IDs (from top to bottom) are 107945, 150280, 150409,
1160185, 1591817, 1592132, 2341006, 2341475. Figure 15: More comparision with
previous methods on VG 256$\times$256\. LayoutDiffusion is trained by 1.45M
iterations, and sample images using scale=1.0 and dpm-solver 25 steps. The VG
image IDs (from top to bottom) are 150297, 150358, 1159865, 2340978, 2341300,
2343290, 2344627, 2375966.

###  A.5 Different Scales

Figure 16: Visualizations on COCO-stuff 256$\times$256 sampled with different
scale. LayoutDiffusion is trained by 1.15M iterations, and sample images using
dpm-solver 25 steps. The COCO image IDs (from top to bottom) are 472298,
475223, 504415, 513181.

##  Appendix B Details of Diffusion Models

###  B.1 Denoising Diffusion Probabilistic Model (from ADM-G)

In this section, we will review the formulation of Gaussian diffusion models
introduced by DDPM [10]. A data point is defined as $x_{0}\sim q(x_{0})$. By
gradually adding noise to the clean data $x_{0}$, we can obtain the noised
samples from $x_{1}$ to $x_{T}$, where $T$ denotes the maximum steps.
Specifically, Gaussian noise according to some variance schedule given by
$\beta_{t}$ is added to $x_{t-1}$ in step $t$ of the Markovian noising process
$q$:

$\displaystyle q(x_{t}|x_{t-1})$
$\displaystyle=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I})$
(20)

Due to the convenient nature of Gaussian noise, we do not need to apply $t$
times of $q(x_{t}|x_{t-1})$ repeatedly to sample from $x_{t}\sim
q(x_{t}|x_{0})$. Instead, $q(x_{t}|x_{0})$ can be directly sampled from a
Gaussian distribution:

$\displaystyle q(x_{t}|x_{0})$
$\displaystyle=\mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\mathbf{I})$
(21)
$\displaystyle=\sqrt{\bar{\alpha}_{t}}x_{0}+\epsilon\sqrt{1-\bar{\alpha}_{t}},\text{
}\epsilon\sim\mathcal{N}(0,\mathbf{I}),$ (22)

where $\alpha_{t}=1-\beta_{t}$ and
$\bar{\alpha}_{t}=\prod_{s=0}^{t}\alpha_{s}$. The noise variance for an
arbitrary timestep is defined as $1-\bar{\alpha}_{t}$ in Eq. 22, and we could
equivalently use this to define the noise schedule instead of $\beta_{t}$.
DDPM [10] notes that the posterior $q(x_{t-1}|x_{t},x_{0})$ is also a Gaussian
using with mean $\tilde{\mu}_{t}(x_{t},x_{0})$ and variance
$\tilde{\beta}_{t}$, and is defined as follows:

$\displaystyle q(x_{t-1}|x_{t},x_{0})$
$\displaystyle=\mathcal{N}(x_{t-1};\tilde{\mu}(x_{t},x_{0}),\tilde{\beta}_{t}\mathbf{I})$
(23) $\displaystyle\tilde{\mu}_{t}(x_{t},x_{0})$
$\displaystyle=\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}x_{0}+\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_{t}}x_{t}$
(24) $\displaystyle\tilde{\beta}_{t}$
$\displaystyle=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$ (25)

If the total noise added throughout the markov chain is large enough when
$T\to\infty$ and correspondingly $\beta_{t}\to 0$, the $x_{T}$ will be well
approximated by $\mathcal{N}(0,\mathbf{I})$. This nice property ensures that
we can reverse the above forward process and sample from
$x_{T}\sim\mathcal{N}(0,\mathbf{I})$, which is a Gaussian noise. However,
since the entire dataset is needed, we cannot easily estimate the posterior
$q(x_{t-1}|x_{t},x_{0})$. Instead, we have to learn a model
$p_{\theta}(x_{t-1}|x_{t})$ to approximate it:

$\displaystyle p_{\theta}(x_{t-1}|x_{t})$
$\displaystyle=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\Sigma_{\theta}(x_{t},t))$
(26)

To ensure that the estimated $p_{\theta}(x_{t-1}|x_{t})$ can learrn the true
data distribution $q(x_{0})$, we can optimize the following variational lower
bound $L_{\text{vlb}}$ for $p_{\theta}(x_{0})$:

$\displaystyle L_{\text{vlb}}$ $\displaystyle=L_{0}+L_{1}+...+L_{T-1}+L_{T}$
(27) $\displaystyle L_{0}$ $\displaystyle=-\log p_{\theta}(x_{0}|x_{1})$ (28)
$\displaystyle L_{t-1}$
$\displaystyle=D_{KL}(q(x_{t-1}|x_{t},x_{0})\;||\;p_{\theta}(x_{t-1}|x_{t}))$
(29) $\displaystyle L_{T}$
$\displaystyle=D_{KL}(q(x_{T}|x_{0})\;||\;p(x_{T}))$ (30)

Although the above objective is well justified, DDPM [10] applied a different
objective that produces better samples in practice. Specifically, they do not
directly predict $\mu_{\theta}(x_{t},t)$ as the output of a neural network,
but instead train a model $\epsilon_{\theta}(x_{t},t)$ to predict $\epsilon$
from Equation 22. This simplified objective is defined as follows:

$\displaystyle L_{\text{simple}}$ $\displaystyle=E_{t\sim[1,T],x_{0}\sim
q(x_{0}),\epsilon\sim\mathcal{N}(0,\mathbf{I})}[||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}]$
(31)

Then, we can derive $\mu_{\theta}(x_{t},t)$ from $\epsilon_{\theta}(x_{t},t)$
using the following substitution:

$\displaystyle\mu_{\theta}(x_{t},t)$
$\displaystyle=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t)\right)$
(32)

Note that $\Sigma_{\theta}(x_{t},t)$ is not learned in $L_{\text{simple}}$ in
DDPM [10] and is fixed as a constant such as $\beta_{t}\mathbf{I}$ or
$\tilde{\beta}_{t}\mathbf{I}$, corresponding to upper and lower bounds for the
true reverse step variance [37], respectively.

###  B.2 Classifier-free Method for Layout-conditional Training and Sampling

Instead of training a separate classifier model, Ho & Salimans [11] choose to
train a diffusion model that allows for both conditional and unconditional
sampling, where the unconditional diffusion model $p_{\theta}(x_{t},t)$ is
parameterized through a score estimator $\epsilon_{\theta}(x_{t},t)$ and the
conditional diffusion model $p_{\theta}(x_{t},t|c)$ is parameterized through
$\epsilon_{\theta}(x_{t},t,c)$.

They use a single model to parameterize both conditional and unconditional
models, where for the unconditional model they simply input a null token
$\varnothing$ for the class identifier $c$ when predicting the score, i.e.
$\epsilon_{\theta}(x_{t},t)=\epsilon_{\theta}(x_{t},t,c=\varnothing)$. They
jointly train this unified model by randomly replacing $c$ with the
unconditional class identifier $\varnothing$ with probability
$p_{\mathrm{uncond}}$. Then the conditional score estimate
$\epsilon_{\theta}(x_{t},t,c)$ is replaced with
$\tilde{\epsilon}_{\theta}(x_{t},t,c)$ using the following equation:

$\displaystyle\tilde{\epsilon}_{\theta}(x_{t},t,c)=\epsilon_{\theta}(x_{t},t,c)+s(\epsilon_{\theta}(x_{t},t,c)-\epsilon_{\theta}(x_{t},t)),$
(33)

which can be considered as a linear combination of conditional and
unconditional score estimates. $s$ is the scale and Since Eq. 33 has no
classifier gradient, no more gradient calculation in classifier guidance is
needed during sampling. Furthermore, $s$ can be changed to modify the effect
of the condition.

In the layout-to-image generation, $c$ is the layout $l$ defined in Sec. 3.1.
Layout Embedding and $\varnothing$ is the emtpy layout $l_{\text{pad}}$
defined in Sec. 3.4. Layout-conditional Diffusion Model.

##  Appendix C Implementation details

###  C.1  Hyperparameters

Dataset COCO-stuff 256$\times$256 COCO-stuff 128$\times$128 VG 256$\times$256
VG 128$\times$128 Model LayoutDiffusion LayoutDiffusion-small LayoutDiffusion
LayoutDiffusion LayoutDiffusion Layout-conditional Diffusion Model In Channels
3 3 3 3 3 Out Channels 6 6 6 6 6 Hidden Channels 256 128 256 256 256 Channel
Multiply 1,1,2,2,4,4 1,1,2,2,4,4 1,1,2,3,4 1,1,2,2,4,4 1,1,2,3,4 Number of
Residual Blocks 2 2 2 2 2 Dropout 0 0 0.1 0.1 0.1 Diffusion Steps 1000 1000
1000 1000 1000 Noise Schedule linear linear linear linear linear Layout-Image
Fusion Module Downsampling Scale For Fusion 8,16,32 8,16,32 4,8,16 8,16,32
4,8,16 Resolution for Fusion 32,16,8 32,16,8 32,16,8 32,16,8 32,16,8 Fusion
Method OaCA OaCA OaCA OaCA OaCA Number of Attention Blocks 1 1 1 1 1 Number of
Heads 4 4 4 4 4 Layout Fusion Module Hidden Channels 256 128 256 256 256
Transformer Depth 6 4 6 6 6 Attention Method Self-Attention Self-Attention
Self-Attention Self-Attention Self-Attention Number of Heads 8 8 8 8 8 Layout
Embedding Embedding Dimension 256 128 256 256 256 Maximum Number of Objects 8
8 8 10 10 Maximum Number of Length 10 10 10 12 12 Maximum Number of Class Id
185 185 185 180 180 Training Hyperparameters Total Batch Size 32 32 64 32 64
Number of GPUs 8 8 8 8 8 Learning Rate 1e-5 1e-5 2e-5 1e-5 2e-5 Mixed
Precision Training Yes Yes Yes Yes Yes Weight Decay 0 0 0 0 0 EMA Rate 0.9999
0.9999 0.9999 0.9999 0.9999 Classifier-free Dropout 0.2 0.2 0.2 0.2 0.2
Iterations 1.15M 1.4M 300K 1.45M 300K

Table 4:  Hyperparameters for the proposed LayoutDiffusion in Sec. 4.4.
Quantitative results. All trained on eight RTX 3090.

Model | LayoutDiffusion |  $-$ LFM |  $-$ OaCA |  $-$OaCA, $+$ CA |  $-$ LFM, $-$ OaCA
---|---|---|---|---|---
Fusion Method | OaCA | OaCA | - | CA | -
Transformer Depth | 6 | 0 | 6 | 6 | 0
Iterations | 300K | 300K | 300K | 300K | 300K
FID | 16.57 | \+ 0.19 | \+ 0.49 | \- 0.11 | \+ 13.37
DS | 0.47 | \+ 0.01 | \+ 0.05 | \+ 0.01 | \+ 0.23
CAS | 43.60 | \- 2.93 | \- 12.74 | \- 1.13 | \- 39.77
YOLOScore | 27.00 | \- 8.20 | \- 20.10 | \- 3.4 | \- 27.0

Table 5: Hyperparameters for the ablation study in Sec. 4.5 .

###  C.2 Analysis of Training Resources and Sampling Speed

method | FID $\downarrow$ | $N_{\text{parms}}$ | Throughout | Compression stage | Diffusion stage | Total
---|---|---|---|---|---|---
|  | images / s | V100 days | V100 days | V100 days
LDM-8 (100 steps) | 42.06 | 345M | 0.457 | 66 | 3.69 | 69.69
LDM-4 (200 steps) | 40.91 | 306M | 0.267 | 29 | 95.49 | 124.49
LayoutDiffusion-small (25 steps) | 36.16 | 142M | 0.608 | - | 75.83 | 75.83
LayoutDiffusion (25 steps) | 31.68 | 569M | 0.308 | - | 216.55 | 216.55

Table 6: Comparison with SOTA diffusion-based methods LDM on COCO-stuff
256$\times$256\. We generate the same 2048 images of LDM for a fair
comparision. LDM is sampled using DDIM [37] and LayoutDiffusion is sampled
with DPM-Solver [22]. The hyperparameters of LayoutDiffusion and
LayoutDiffusion-small are listed in Tab. 4

##  Appendix D Evaluation

###  D.1 Datasets

COCO-Stuff [5]. COCO[21] is a large-scale object detection, segmentation, and
captioning dataset. COCO-Stuff[5] augments all 164K images of the COCO[21]
dataset with pixel-level stuff annotations. These annotations can be used for
scene understanding tasks like semantic segmentation, object detection and
image captioning. COCO-Stuff[5] contains 80 categories of thing and 91
categories of stuff, respectively. Following the settings of LostGAN-v2[41],
we use the COCO 2017 Stuff Segmentation Challenge subset containing 40K / 5k /
5k images for train / val / test-dev set. Segmengtation annotation is not
used. We use images in the train and val set with 3 to 8 objects that cover
more than 2% of the image and not belong to ’crowd’. Finally, there are 25,210
train and 3,097 val images. Some previous works[14, 2] don’t filter objects
belong to ’crowd’, causing different numbers of images. Specificly, there are
24,972 train and 3,074 val images. LAMA[20] uses the full COCO-Stuff[5] 2017
dataset, leaving 74,777 train and 3,097 val images. Tab. 7 summarizes the
difference in the number of images by different filtering methods.

use | filter | train | val
---|---|---|---
deprecated | ’crowd’ | image | object | object/image | image | object | object/image
|  | 74,121 | 411,682 | 5.55 | 3,074 | 17,100 | 5.56
✓ |  | 24,972 | 138,162 | 5.53 | 3,074 | 17,100 | 5.56
| ✓ | 74,777 | 414,443 | 5.54 | 3,097 | 17,191 | 5.55
✓ | ✓ | 25,210 | 139,175 | 5.52 | 3,097 | 17,191 | 5.55

Table 7:  The difference in the number of images by different filtering
methods, use deprecated means use COCO 2017 Stuff Segmentation Challenge
subset, filter ’crowd’ means filter objects belong to ’crowd’.

Visual Genome [19]. Following the settings of Sg2Im[14], we experiment on
Visual Genome[19] version 1.4 (VG) which comprises 108,077 images annotated
with scene graphs. Visual Genome[19] collects images with dense annotations of
objects, attributes, and relationships. Here, we only use bounding boxes. We
divide the data into 80% / 10% / 10% for the train / val / test set. We select
the object / relationship categories occurring at least 2000 / 500 times in
the train set, respectively, and select the images with 3 to 30 bounding boxes
and ignoring all small objects. Finally, the train / val / test set has 62,565
/ 5,062 / 5,096 images.

###  D.2 Evaluation Metrics

Comprehensive evaluations of generated images remains a challenge. We use six
metrics, from image-level to layout-level, to evaluate the quality of the
generated images and the layout control from different aspects.

Fr‘echet Inception Distance (FID)[9] shows the difference between the real
images and the generated images by using an ImageNet-pretrained
Inception-V3[42] network and computing the Fr‘echet distance between two
Gaussian distributions fitted to generated images and real images
respectively. We save the GT images as real images when sampling the generated
images. The real images and the generated images are saved to two folder
respectively. Then compute the FID score, based on the official code of
FID\dagger\dagger$\dagger$the official code of
FID[9]:https://github.com/bioinf-jku/TTUR. For FID, the lower the score, the
smaller the difference between the generated images and the real images,
meaning the generator model is better.

Inception Score (IS)[35] shows the overall quality of the generated images by
using an Inception-V3[42] network pretrained on the ImageNet-1000
classification benchmark and computing a score(statistics) of the network’s
outputs with generated images of a generator model. IS measures the quality of
images on two aspects: clarity and diversity. We only need the generated
images to compute the IS score, based on the official code of
IS\dagger\dagger$\dagger$the official code of
IS[35]:https://github.com/openai/improved-gan. For IS, the higher the score,
the better the quality of generated images, meaning the generator model is
better.

Diversity Score (DS) measures the diversity between the generated images from
the same layout by comparing the perceptual similarity in a DNN feature space
between them. Here, we adopt the LPIPS[49] metric. For each sample, we repeat
two times, use the two images from the same layout to compute the DS score,
then calculate mean and std of these scores as the reported DS score, based on
the official code of DS\dagger\dagger$\dagger$the official code of DS
[49]:https://github.com/richzhang/PerceptualSimilarity. For DS, the higher the
score, the better the diversity between the generated images, meaning the
generator model is better.

YOLO Score[20] uses a pretrained YOLOv4[4] model to evaluate bbox mAP on 80
thing categories based on the official code of LAMA\dagger\dagger$\dagger$the
official code of LAMA [20]:https://github.com/ZejianLi/LAMA and the official
code of YOLOv4\dagger\dagger$\dagger$the official code of YOLOv4
[4]:https://github.com/AlexeyAB/darknet. YOLO Score[20] is proposed to
evaluate the alignmentand fidelity of generated objects, measuring how
generated objects are recognizable when even the layout is unknow. YOLO[30, 4]
is a well-known series of object detector, inferring layouts from the given
images. Before send images to detector, they are upsampled to 512$\times$512\.
And different from LAMA[20], since we filter the objects and images in
datasets, we think it is better to evaluate bbox mAP only on filtered
annotations.

Classification Score (CAS)[29] measures classification accuracy of layout
areas on generated images. We crop the GT box area of images and resize
objects at a resolution of 32$\times$32 with their class. Then train a
ResNet101[8] classifier with cropped images on generated images and test it on
cropped images on real images, based on a widely used codebase of image
classification\dagger\dagger$\dagger$the widely used codebase of image
classification: https://github.com/hysts/pytorch_image_classification. For
CAS, the higher the score, the better the quality of layout control, meaning
the generator model is better.

## References

  * [1] Martin Arjovsky and Léon Bottou.  Towards principled methods for training generative adversarial networks.  arXiv preprint arXiv:1701.04862, 2017.
  * [2] Oron Ashual and Lior Wolf.  Specifying object attributes and relations in interactive scene generation.  In The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [3] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.  Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models.  In ICLR, 2021.
  * [4] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection, 2020.
  * [5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In CVPR, pages 1209–1218, 2018.
  * [6] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  arXiv preprint arXiv:2209.04747, 2022.
  * [7] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [10] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [11] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
  * [12] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee.  Inferring semantic layout for hierarchical text-to-image synthesis.  In CVPR, 2018.
  * [13] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, 2017.
  * [14] Justin Johnson, Agrim Gupta, and Li Fei-Fei.  Image generation from scene graphs.  In CVPR, pages 1219–1228, 2018.
  * [15] Levent Karacan, Zeynep Akata, Aykut Erdem, and Erkut Erdem.  Learning to generate images of outdoor scenes from attributes and semantic layouts.  arXiv:1612.00215, 2016.
  * [16] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [17] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  In Advances in Neural Information Processing Systems, 2022.
  * [18] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9404–9413, 2019.
  * [19] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 123(1):32–73, 2017.
  * [20] Zejian Li, Jingyu Wu, Immanuel Koh, Yongchuan Tang, and Lingyun Sun.  Image synthesis from layout with locality-aware mask adaption.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13819–13828, 2021.
  * [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context, 2014.
  * [22] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  arXiv preprint arXiv:2206.00927, 2022.
  * [23] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022.
  * [24] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.  Mixed precision training.  arXiv preprint arXiv:1710.03740, 2017.
  * [25] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [26] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [27] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In CVPR, 2019.
  * [28] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [29] Suman Ravuri and Oriol Vinyals.  Classification accuracy score for conditional generative models.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [30] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788, 2016.
  * [31] Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee.  Learning what and where to draw.  In NIPS, 2016.
  * [32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, pages 10684–10695, 2022.
  * [33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [34] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [35] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In NIPS, 2016.
  * [36] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  arXiv:1503.03585, 2015\.
  * [37] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv:2010.02502, October 2020.
  * [38] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  arXiv:arXiv:1907.05600, 2020.
  * [39] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2020.
  * [40] Wei Sun and Tianfu Wu.  Image synthesis from reconfigurable layout and style.  In Proceedings of the IEEE International Conference on Computer Vision, pages 10531–10540, 2019.
  * [41] Wei Sun and Tianfu Wu.  Learning layout and style reconfigurable gans for controllable image synthesis.  IEEE TPAMI, 44(9):5070–5087, 2021.
  * [42] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision, 2015.
  * [43] Fuwen Tan, Song Feng, and Vicente Ordonez.  Text2scene: generating abstract scenes from textual descriptions.  arXiv:1809.01110, 2018.
  * [44] Anwaar Ulhaq, Naveed Akhtar, and Ganna Pogrebna.  Efficient diffusion models for vision: A survey.  arXiv preprint arXiv:2210.09292, 2022.
  * [45] Bo Wang, Tao Wu, Minfeng Zhu, and Peng Du.  Interactive image synthesis with panoptic layout generation.  In CVPR, pages 7783–7792, 2022.
  * [46] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In CVPR, 2018.
  * [47] Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang.  Diffusion-based scene graph to image generation with masked contrastive pre-training.  arXiv preprint arXiv:2211.11138, 2022.
  * [48] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  arXiv preprint arXiv:2209.00796, 2022.
  * [49] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [50] Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal.  Image generation from layout.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8584–8593, 2019.
  * [51] Guangcong Zheng, Shengming Li, Hui Wang, Taiping Yao, Yang Chen, Shouhong Ding, and Xi Li.  Entropy-driven sampling and training scheme for conditional diffusion generation.  In European Conference on Computer Vision, pages 754–769. Springer, 2022.