11institutetext: Technical University of Kaiserslautern, Germany
22institutetext: German Research Center for Artificial Intelligence, Germany
22email<EMAIL_ADDRESS>

# DT2I: Dense Text-to-Image Generation from Region Descriptions

Stanislav Frolov Equal contribution1122 Prateek Bansal⋆ 11 Jörn Hees 22
Andreas Dengel 1122

###### Abstract

Despite astonishing progress, generating realistic images of complex scenes
remains a challenging problem. Recently, layout-to-image synthesis approaches
have attracted much interest by conditioning the generator on a list of
bounding boxes and corresponding class labels. However, previous approaches
are very restrictive because the set of labels is fixed a priori. Meanwhile,
text-to-image synthesis methods have substantially improved and provide a
flexible way for conditional image generation. In this work, we introduce
dense text-to-image (DT2I) synthesis as a new task to pave the way toward more
intuitive image generation. Furthermore, we propose DTC-GAN, a novel method to
generate images from semantically rich region descriptions, and a multi-modal
region feature matching loss to encourage semantic image-text matching. Our
results demonstrate the capability of our approach to generate plausible
images of complex scenes using region captions.

##  1 Introduction

In the last few years, deep generative image modelling has experienced
remarkable progress [33, 4]. Current models can produce realistic results when
trained on single-domain datasets such as human faces, birds or flowers, but
struggle when trained on complex datasets with multiple objects such as COCO
[25] and Visual Genome [20]. While learning the natural image distribution via
unconditional image synthesis is interesting, controlling the image generation
process is important for many practical applications such as image editing,
computer-aided design, and visual storytelling.

Strong supervision in the form of segmentation masks often leads to impressive
visual results [29], but they are difficult and time-consuming to create from
a user’s perspective. Recently, layout-to-image methods [46, 37] have
attracted much interest by conditioning on a spatial layout of bounding boxes
and class labels to allow the user to create a complex scene of multiple
objects. To gain control over the specific appearance of objects, [27, 6]
further improved the methods by providing additional attributes to the
corresponding objects (e.g. “red bus”). However, the set of class labels and
attributes is fixed a priori which strongly limits their expressiveness. In
contrast to labels, text is much more flexible, intuitive and can carry rich
semantic information about the object’s appearance and relationship to other
objects [5].

In this paper, we introduce dense text-to-image (DT2I) synthesis as a new task
with the goal to generate realistic images using multiple region descriptions.
See Tab. 1 for an overview of tasks and corresponding inputs. To solve this
task, we propose a novel method based on a state-of-the-art adversarial
layout-to-image [37] model and incorporate best practices from the text-to-
image literature (e.g. triplet [33] and DAMSM [42] losses), and propose a
novel multi-modal region feature matching loss between real and generated
image-text pairs. We create a synthetic dataset to validate the effectiveness
of our approach. Finally, we extensively evaluate our model on a challenging
real-world dataset to demonstrate the capability of our method. Our model
outperforms previous methods on several metrics while allowing free-form
regions descriptions as input.

Input | layout-to-image (L2I) | text-to-image (T2I) | dense text-to-image (DT2I)
---|---|---|---
spatial layout | ✓ | (✓) | ✓
class labels | ✓ | (✓) | (✓)
free-form text | - | ✓ | ✓
dense captions | - | - | ✓

Table 1:  Overview of tasks and their corresponding input. Brackets indicate
optional implicit information. L2I methods use a spatial layout of bounding
boxes and class labels, while T2I methods use single captions. We propose DT2I
as a new task with the goal to produce realistic images from multiple
localized free-form region descriptions.

##  2 Related Work

Layout-to-Image Synthesis: The layout-to-image (L2I) task was first studied in
[45] using a VAE [18] by composing object representations into a scene before
producing an image. It was further improved in [46] with an object-wise
attention mechanism to predict a map of object details. Adversarial approaches
[36, 37] were able to produce higher-resolution images and provide better
control of individual objects by using a reconfigurable layout with separate
latent style codes. Recent developments focused on better instance
representations [38], context-awareness [9], and improving the mask prediction
of overlapping and nearby objects [24]. Recently, [27, 6] enabled more
explicit appearance control of individual objects by conditioning on
attributes. However, the set of attributes is limited and lacks the ability to
model complex interactions between objects. In contrast, our model generates
an image from free-form region descriptions.

Text-to-Image Synthesis: Generating images from text descriptions (T2I) is a
challenging but fascinating problem with remarkable progress in recent years
[5]. Compared to labels, they can carry dense semantic information about the
appearance of objects and scenes. Initial approaches [33] conditioned on a
sentence embedding and trained the discriminator to distinguish between real,
matching, and generated, non-matching image-text pairs. Stacked architectures
[44] were further improved by using attention mechanisms [42], contrastive
losses [43] and transformers [31]. However, all previous approaches are either
applied on single-object datasets or aim to generate an image from one short
text description which lacks fine-grained details.

Combinations of Location & Text: In [32], the location of one object can be
specified by a bounding box or keypoints, while the appearance is described by
a text description. Generating complex scenes from a single description is
challenging, as it requires modelling multiple objects. To alleviate this
problem, [14, 23] used a text-to-layout-to-image framework to first predict a
layout of bounding boxes which are subsequently refined into shapes before
producing the output image. In [12, 13], an object pathway is added to both
generator and discriminator which uses bounding boxes and class labels to
focus on the individual objects in the scene. Sparse semantic masks are used
in [29, 22] to define the position of objects, while an input text can be used
to control the style. In [19], the Localized Narratives [30] dataset is used
to generate images from mouse traces and paired descriptions. Different from
all previous works, our model takes free-form captions and corresponding
locations as input and hence allows rich expression of individual image
regions.

Figure 1: Given region descriptions as input, our novel DTC-GAN is trained to
minimize several losses which encourage both image quality as well as image-
text alignment.

##  3 Method

The goal of our method is to produce realistic images of complex scenes
depicting multiple objects from localized region descriptions (which can be
seen as the inverse of dense image captioning). To address this challenging
task, we propose a novel framework consisting of the following key components:
1) Dense-Text-Conditional GAN, and 2) Regional Semantic Image-Text Matching.
See Fig. 1 for an illustration of our DTC-GAN architecture.

###  3.1 Dense-Text-Conditional GAN

Generator: Our DTC-GAN builds upon a state-of-the-art L2I model [37]. The
generator consists of a linear layer to process the global image latent code
$z_{\textrm{img}}\sim\mathcal{N}(0,1)$ and multiple ResNet [8] based generator
blocks with upsampling to produce an image. To condition our generator on a
layout of region descriptions, we adapt the feature normalization technique
proposed in [36]. Given a layout $L=\\{(b_{i},t_{i})_{i=1}^{m}\\}$ of $m$
regions as input, where each region is described by a bounding box $b_{i}$ and
text description $t_{i}$, we first embed $t_{i}$ using a pre-trained fixed
BERT [3] model. Next, we concatenate regional latent codes
$Z_{\textrm{r}}=\\{z_{i}\\}_{i=1}^{m}$ sampled from $\mathcal{N}(0,1)$, and
text embeddings $E_{\textrm{r}}=\\{e_{i}\\}_{i=1}^{m}$ to produce the
embedding matrix $\mathcal{S}=(Z_{\textrm{r}},E_{\textrm{r}})$ of size
$m\times(d_{z}+d_{e})$, with the dimensions $d_{z}=128$ and $d_{e}=768$. As in
[37], we use the embedding matrix $\mathcal{S}$ to regress masks for each
region using a sub-network. Finally, we predict affine transformation
parameters $\gamma$ and $\beta$ at each generator layer to modulate the visual
feature maps in the generator after normalizing them as in BatchNorm [15].
Using the bounding boxes $b_{i}$, we unsqueeze $\gamma$ and $\beta$ to their
corresponding bounding boxes and weight them by the predicted region masks.
Please refer to [37] for more details on this normalization technique. Given
that we use text embeddings, our $\gamma$ and $\beta$ parameters are now
Layout-Aware & Text-Sensitive (LATS).

Discriminator: The architecture of the discriminator remains largely unchanged
from [37] and consists of multiple ResBlocks. A shared backbone takes images
as input to extract coarse features, while two classification heads are used
for adversarial training to encourage realism on the image and region level.
More specifically, the region head extracts region features using the bounding
boxes and RoIAlign operation [7], and adopts projection-based conditioning
[28, 1] to compute a region-embedding score $s_{r}$. The image head continues
processing the full image without any semantic knowledge to produce an image
score $s_{x}$. Both image and region-embedding scores are used for adversarial
training.

###  3.2 Regional Semantic Image-Text Matching

The goal of a good DT2I model is not only to produce realistic images but also
such that correctly reflect the semantic meaning of the input captions at the
corresponding locations. Inspired by the T2I literature, we adapt two main
techniques to learn semantic image-text matching and apply them on the
regional level. Furthermore, we propose a multi-modal region feature matching
loss to guide the region classification head of our discriminator.

Regional Triplet Loss: The region-embedding classification head in the
discriminator takes pairs of extracted region features and corresponding text
embeddings as input and is trained to distinguish real from generated pairs.
However, even if the generator produces realistic image regions, there is
another kind of possible error, namely, non-matching. Similar to [33], we
construct another pair of real image regions and randomly chosen captions to
encourage semantic image-text matching and penalize mismatching pairs.

Regional DAMSM Loss: The Deep Attentional Multimodal Similarity Model (DAMSM)
proposed in [42] computes the similarity between an image and global sentence
as well as word features using an attention mechanism to improve semantic
matching. We first fine-tune an Inception-v3 [39] image encoder that was pre-
trained on ImageNet [2], and a pre-trained BERT text encoder to map matching
image regions and text features into a common embedding space. The encoder
networks can then be used to provide a fine-grained learning signal to our
generator. We follow the procedure in [42], but in contrast to traditional T2I
methods, our captions describe specific regions in the image which leads to
better alignment between individual regions and corresponding captions.

Multi-Modal Region Feature Matching (MMRFM): The regional triplet and DAMSM
losses encourage images with matching captions. However, adversarial losses
can lead to adversarial examples and DAMSM lacks proper localization. Because
of the complexity of natural language and diversity of real images, our task
is an inherent one-to-many mapping problem between input conditions and output
images which pixel-wise losses alone can not handle. Inspired by the
perceptual feature loss [21], which maximizes similarity at the intermediate
feature space, we propose a Multi-Modal Region Feature Matching (MMRFM) loss
on the semantics-enriched region features denoted as
$\mathcal{L}_{\textrm{MMRFM}}(r_{\textrm{real}},r_{\textrm{fake}},t)$. More
precisely, we use the resulting region features after projection-based
conditioning and minimize the distance between corresponding real and
generated image-text features at the region level, see Fig. 2.

Figure 2: Overview of our Multi-Modal Region Feature Matching loss
$\mathcal{L}_{\textrm{MMRFM}}$. We first extract region features using
bounding boxes $b_{i}$. Next, we multiply individual region features with
corresponding text embeddings and minimize L1 distance between real and
generated multi-modal region features to improve semantic image-text matching.

###  3.3 Training Objectives & Implementation Details

We combine multiple loss functions during training of our model. Given an
image $x$ and corresponding layout $L$ with text descriptions $t_{i}$ and
regions $r_{i}$, the discriminator predicts a score for the full image
$s_{x}=D_{X}(x)$, and matching-aware region-embedding scores
$s_{r}=D_{R}(r_{i},t_{i})$. We consider $(r_{i},t_{i})$ to be correct if
$r_{i}$ is a real image region with corresponding and matching caption
$t_{i}$, and wrong if a) $r_{i}$ is generated, or b) $(r_{i},t_{i})$ is a non-
matching pair. Using an adversarial hinge loss [40, 1, 36], we get
$\mathcal{L}_{X}(x)$ as our (unconditional) image loss, $\mathcal{L}_{R}(r,t)$
as our (conditional) regional triplet loss, and the discriminator objective
as:

$\displaystyle\mathcal{L}_{X}$
$\displaystyle=\mathbb{E}[\max(0,1-D_{X}(x_{\textrm{real}})]+\mathbb{E}[\max(0,1+D_{X}(x_{\textrm{fake}})]$
(1)
$\displaystyle\begin{split}\mathcal{L}_{R}&=\mathbb{E}[\max(0,1-D_{R}(r_{\textrm{real}},t_{\textrm{match}})]\\\
&+\mathbb{E}[\max(0,1+D_{R}(r_{\textrm{fake}},t_{\textrm{match}})]\\\
&+\mathbb{E}[\max(0,1+D_{R}(r_{\textrm{real}},t_{\textrm{non-
match}})]\end{split}$ (2) $\displaystyle\mathcal{L}_{D}$
$\displaystyle=\lambda_{1}\mathcal{L}_{X}+\lambda_{2}\mathcal{L}_{R}$ (3)

To train the generator, we maximize fooling the discriminator. Additionally,
we extract image features and corresponding text features using the pre-
trained encoders and compute the DAMSM loss
$\mathcal{L}_{\textrm{DAMSM}}(x,t)$ as proposed in [42] for an additional
learning signal on a per-region level of generated images. Following [37], we
also employ the perceptual loss [21] using extracted VGG [35] features and
pixel loss for improved image quality. With our multi-modal region feature
matching loss, the generator objective becomes:

$\displaystyle\mathcal{L}_{G}=-$
$\displaystyle\lambda_{1}\mathbb{E}[D_{X}(x_{\textrm{fake}})]-\lambda_{2}\mathbb{E}[D_{R}(r_{\textrm{fake}},t_{\textrm{match}})]$
(4) $\displaystyle+$
$\displaystyle\mathcal{L}_{\textrm{DAMSM}}(x_{\textrm{fake}},t_{\textrm{match}})$
$\displaystyle+$
$\displaystyle\mathcal{L}_{\textrm{MMRFM}}(r_{\textrm{real}},r_{\textrm{fake}},t_{\textrm{match}})$
$\displaystyle+$
$\displaystyle\mathcal{L}_{\textrm{VGG}}(x_{\textrm{real}},x_{\textrm{fake}})+\mathcal{L}_{\textrm{pixel}}(x_{\textrm{real}}-x_{\textrm{fake}})$

Our code is based on the official repositories of [36, 37, 6], and we use the
pre-trained BERT model from [41] as our text encoder. For training, we use the
Adam [17] optimizer with $\beta_{1}=0.0$ and $\beta_{2}=0.999$. The learning
rates are set to $10e^{-4}$, and weights are fixed as in [36] to
$\lambda_{1}=0.1,\lambda_{2}=1.0$. We train our models using a batch size of
128 on 4 NVIDIA V100-32GB GPUs for 200 epochs which takes roughly 12 days
using an image resolution of 128$\times$128.

##  4 Experiments

###  4.1 Synthetic Images

Dataset: We first create a synthetic dataset similar to [26] based on CLEVR
[16] to qualitatively validate our approach. To that end, we render 50,000
images each depicting 3-8 objects where each object consists of four different
attributes including color, shape, size, and material. Using these attributes,
we can create simple text descriptions of localized objects (e.g. “a small red
cube”). To simulate complex image regions with multiple objects, we randomly
group nearby objects and additionally use spatial relationship annotations to
describe them (e.g. “a small yellow sphere behind a small green cylinder”).

Results: Fig. 3 shows input layouts with text descriptions and the
corresponding images as generated by our model. As can be seen, the image
quality is high as every object is clearly visible and generated according to
the input description. Our model learned to faithfully produce image regions
that describe two objects in a spatial relationship. In contrast to [26], our
model allows to precisely define the location of generated objects and in
particular generates only what is specified as input.

Figure 3: Results on CLEVR. Our model can faithfully produce image regions
with correct objects, attributes and relationships. Figure 4: Results on
Visual Genome. Images generated by manual annotation (top), and from the test
set with real images for comparison (bottom). Our model can often generate
plausible images, correctly respecting the input descriptions.

###  4.2 Real Images

Dataset: We use the Visual Genome (VG) [20] dataset, which contains complex
images with multiple, interacting objects and region descriptions that
describe localized portions of the image. The region descriptions in the VG
dataset are very noisy, repetitive and contain many small regions. We remove
regions smaller than 32$\times$32 pixels and only keep images that contain
3-30 region descriptions with a minimum description length of 5 words. During
training, we select a maximum of 10 regions randomly. Next, we find and remove
similar region descriptions if they are overlapping for more than 30% and have
similar area size. In summary, we get 85,414 train, 6,206 validation, 6,173
test images, and a total of 1,117,766 region descriptions.

Evaluation Metrics: A good DT2I model should produce realistic images and
image regions that reflect the semantic meaning of the input captions. Using
several commonly used measures, we extensively evaluate our model in terms of
image quality and image-text alignment.

Image Quality: The Inception Score (IS) [34] uses a pre-trained image
classifier to measure recognizability as well as image diversity of generated
images. We also compute the Fréchet Inception Distance (FID) [11] which
measures the distance between feature distributions of real and generated
images. Following [38, 6], we evaluate the visual fidelity of individual image
regions by applying the IS and FID on image crops, denoted as SceneIS [6] and
SceneFID [38].

Image-Text Alignment: To evaluate the semantic matching between input captions
and generated images, we adopt the R-precision [42] by concatenating all
captions per image as a full-scene description. We report top-5 accuracy given
that our task requires distinguishing between multiple text-described regions
which is much more challenging when compared to traditional T2I methods.
Furthermore, we use the recently proposed CLIP-Score [10] to assess global
image-text compatibility using a general-purpose model trained on a large
dataset.

Method 128$\times$128 |  IS $\uparrow$ |  SceneIS $\uparrow$ |  FID $\downarrow$ |  SceneFID $\downarrow$ |  R-prec. $\uparrow$ |  CLIP-S $\uparrow$
---|---|---|---|---|---|---
LostGAN [36] | 11.10 | - | 29.65 | 13.17 | - | -
LostGANv2 [37] | 10.71 | - | 29.00 | - | - | -
AttrLostGANv2 [6] | 10.81 | 9.46 | 31.57 | 7.78 | - | -
OC-GAN [38] | 12.30 | - | 28.26 | 9.63 | - | -
LAMA [24] | - | - | 23.02 | 8.28 | - | -
CAL2IM [9] | 12.69 | - | 21.78 | - | - |
DTC-GAN (our base) | 8.65 | 8.00 | 39.80 | 9.58 | 55.93 | 63.98
\+ triplet | 8.41 | 7.89 | 44.39 | 11.86 | 59.97 | 66.56
\+ MMRFM | 8.43 | 8.30 | 47.08 | 13.63 | 58.50 | 66.19
\+ DAMSM (our final) | 13.80 | 9.46 | 27.67 | 7.85 | 89.07 | 68.20
Method 256$\times$256 |  IS $\uparrow$ |  SceneIS $\uparrow$ |  FID $\downarrow$ |  SceneFID $\downarrow$ |  R-prec. $\uparrow$ |  CLIP-S $\uparrow$
LostGANv2 [37] | 14.10 | - | 47.62 | - | - | -
AttrLostGANv2 [6] | 14.25 | 11.96 | 35.73 | 14.76 | - | -
OC-GAN [38] | 14.70 | - | 40.85 | - | - | -
LAMA [24] | - | - | 31.63 | 13.66 | - | -
DTC-GAN (our final) | 15.96 | 12.03 | 35.29 | 9.32 | 77.68 | 67.98
Table 2: Quantitative results on Visual Genome. Our model benefits from
combining all discussed losses and outperforms previous layout-to-image
methods on several metrics while allowing free-form region descriptions as
input. Both [24] and [9] are recent improvements which could also be applied
to our backbone generator.

Results: Fig. 4 shows images generated by our model. Our method provides an
unprecedented interface for the user to communicate what image should be
generated. Despite the very challenging task it is able to produce images
which match the input region descriptions with objects at the correct
location. Furthermore, the model respects color attributes and compositions
such as “man wearing a yellow shirt” and “mountains with snow”. Next, we
evaluate and compare our model with several recent methods, see Tab. 2. A
naive replacement of labels with text embeddings (our base) leads to low
scores across all metrics. Although adding the MMRFM loss results in slightly
lower scores, we observed faster and stable convergence during training and
better performance when combining all losses. Combining two techniques from
the T2I literature with our proposed MMRFM loss outperforms previous methods
on several metrics while allowing free-form region descriptions as input.

Limitations: Generating images of complex scenes is a very challenging task.
As all current approaches, our model still struggles to produce highly
realistic images. From a training perspective, our model relies on dense
captions, which are time-consuming to collect. The used dataset mostly
contains rather short captions, and it would be interesting to expand towards
longer descriptions with more details. During testing, we observed that
details such as positional and numerical information is often ignored during
image generation, which is also currently a problem in the wider T2I
literature [5]. Improved mask prediction [38, 24] and context awareness [9],
both of which are recent improvements of the generator, may as well improve
the image quality for our application. While our method produces promising
results, it is merely the first step in this direction and further research
efforts are required to improve the image quality to enable practical
applications.

##  5 Conclusion

In this paper, we introduced dense text-to-image (DT2I) synthesis as a new
task and proposed DTC-GAN, a novel method that generates images from multiple
free-form region descriptions. To address this challenging task, our method
successfully combines recent layout-to-image and text-to-image techniques.
Furthermore, we proposed a multi-modal region feature matching loss to improve
semantic image-text matching between real and generated image regions and
stabilize training. From a user’s perspective, our model is the first of a new
kind of image generation method which are intuitive, flexible, and not
restricted by a fixed set of labels. In terms of future work, enhancing the
quality of generated images is required for practical applications and can for
example be achieved by scaling up the networks and using contrastive losses.

Acknowledgments: This work was supported by the BMBF projects ExplAINN (Grant
01IS19074), XAINES (Grant 01IW20005), the NVIDIA AI Lab (NVAIL) and the TU
Kaiserslautern PhD program.

## References

  * [1] Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. In: ICLR (2018)
  * [2] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255 (2009)
  * [3] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019)
  * [4] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: CVPR. pp. 12873–12883 (2021)
  * [5] Frolov, S., Hinz, T., Raue, F., Hees, J., Dengel, A.: Adversarial text-to-image synthesis: A review. Neural Networks (2021)
  * [6] Frolov, S., Sharma, A., Hees, J., Karayil, T., Raue, F., Dengel, A.: Attrlostgan: Attribute controlled image synthesis from reconfigurable layout and style. In: GCPR (2021)
  * [7] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: ICCV. pp. 2961–2969 (2017)
  * [8] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CVPR pp. 770–778 (2016)
  * [9] He, S., Liao, W., Yang, M.Y., Yang, Y., Song, Y.Z., Rosenhahn, B., Xiang, T.: Context-aware layout to image generation with enhanced object appearance. In: CVPR. pp. 15049–15058 (2021)
  * [10] Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: CLIPScore: a reference-free evaluation metric for image captioning. In: EMNLP (2021)
  * [11] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS. pp. 6626–6637 (2017)
  * [12] Hinz, T., Heinrich, S., Wermter, S.: Generating multiple objects at spatially distinct locations. In: ICLR (2019)
  * [13] Hinz, T., Heinrich, S., Wermter, S.: Semantic object accuracy for generative text-to-image synthesis. IEEE TPAMI (2020)
  * [14] Hong, S., Yang, D., Choi, J., Lee, H.: Inferring semantic layout for hierarchical text-to-image synthesis. In: CVPR. pp. 7986–7994 (2018)
  * [15] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML. pp. 448–456 (2015)
  * [16] Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: CVPR. pp. 2901–2910 (2017)
  * [17] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2014)
  * [18] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: ICLR (2013)
  * [19] Koh, J.Y., Baldridge, J., Lee, H., Yang, Y.: Text-to-image generation grounded by fine-grained user attention. In: WACV. pp. 237–246 (2021)
  * [20] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (2017)
  * [21] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR. pp. 4681–4690 (2017)
  * [22] Li, B., Qi, X., Torr, P.H., Lukasiewicz, T.: Image-to-image translation with text guidance. arXiv:2002.05235 (2020)
  * [23] Li, W., Zhang, P., Zhang, L., Huang, Q., He, X., Lyu, S., Gao, J.: Object-driven text-to-image synthesis via adversarial training. In: CVPR. pp. 12166–12174 (2019)
  * [24] Li, Z., Wu, J., Koh, I., Tang, Y., Sun, L.: Image synthesis from layout with locality-aware mask adaption. In: ICCV. pp. 13819–13828 (2021)
  * [25] Lin, T.Y., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV. pp. 740–755 (2014)
  * [26] Liu, N., Li, S., Du, Y., Tenenbaum, J., Torralba, A.: Learning to compose visual relations. In: NeurIPS (2021)
  * [27] Ma, K., Zhao, B., Sigal, L.: Attribute-guided image generation from layout. In: British Machine Vision Virtual Conference (2020), arXiv:2008.11932
  * [28] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv:1411.1784 (2014)
  * [29] Pavllo, D., Lucchi, A., Hofmann, T.: Controlling style and semantics in weakly-supervised image generation. In: ECCV. pp. 482–499 (2020)
  * [30] Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., Ferrari, V.: Connecting vision and language with localized narratives. In: ECCV. pp. 647–664 (2020)
  * [31] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: ICML. pp. 8821–8831 (2021)
  * [32] Reed, S.E., Akata, Z., Mohan, S., Tenka, S., Schiele, B., Lee, H.: Learning what and where to draw. In: NeurIPS. pp. 217–225 (2016)
  * [33] Reed, S.E., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text to image synthesis. In: ICML. pp. 1060–1069 (2016)
  * [34] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. In: NeurIPS. pp. 2234–2242 (2016)
  * [35] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2014)
  * [36] Sun, W., Wu, T.: Image synthesis from reconfigurable layout and style. In: ICCV. pp. 10531–10540 (2019)
  * [37] Sun, W., Wu, T.: Learning layout and style reconfigurable gans for controllable image synthesis. arXiv:2003.11571 (2020)
  * [38] Sylvain, T., Zhang, P., Bengio, Y., Hjelm, R.D., Sharma, S.: Object-centric image generation from layouts. In: AAAI (2021)
  * [39] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR. pp. 2818–2826 (2016)
  * [40] Tran, D., Ranganath, R., Blei, D.M.: Hierarchical implicit models and likelihood-free variational inference. In: NeurIPS (2017)
  * [41] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: EMNLP: System Demonstrations. pp. 38–45 (2020)
  * [42] Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., He, X.: Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In: CVPR. pp. 1316–1324 (2017)
  * [43] Zhang, H., Koh, J.Y., Baldridge, J., Lee, H., Yang, Y.: Cross-modal contrastive learning for text-to-image generation. In: CVPR. pp. 833–842 (2021)
  * [44] Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.N.: Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE TPAMI (2017)
  * [45] Zhao, B., Meng, L., Yin, W., Sigal, L.: Image generation from layout. In: CVPR (2019)
  * [46] Zhao, B., Yin, W., Meng, L., Sigal, L.: Layout2image: Image generation from layout. IJCV (2020)