1]FAIR, Meta 2]Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK

# Unlocking Pre-trained Image Backbones for Semantic Image Synthesis

Tariq Berrada  Jakob Verbeek  Camille Couprie  Karteek Alahari  [  [
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Semantic image synthesis, i.e., generating images from user-provided semantic
label maps, is an important conditional image generation task as it allows to
control both the content as well as the spatial layout of generated images.
Although diffusion models have pushed the state of the art in generative image
modeling, the iterative nature of their inference process makes them
computationally demanding. Other approaches such as GANs are more efficient as
they only need a single feed-forward pass for generation, but the image
quality tends to suffer on large and diverse datasets. In this work, we
propose a new class of GAN discriminators for semantic image synthesis that
generates highly realistic images by exploiting feature backbone networks pre-
trained for tasks such as image classification. We also introduce a new
generator architecture with better context modeling and using cross-attention
to inject noise into latent variables, leading to more diverse generated
images. Our model, which we dub DP-SIMS, achieves state-of-the-art results in
terms of image quality and consistency with the input label maps on ADE-20K,
COCO-Stuff, and Cityscapes, surpassing recent diffusion models while requiring
two orders of magnitude less compute for inference.

Tariq Berrada () and Jakob Verbeek ()

Figure 1: Images generated with models trained on COCO-Stuff, comparing our
approach to state-of-the-art methods OASIS, SDM, and PITI, along with
inference times to generate a single image. Our approach combines high-quality
samples with low-latency sampling.

##  1 Introduction

Conditional image synthesis aims to generate images based on information such
as text, categories, sketches, label maps, _etc_. While text-based generation
has seen impressive advances in recent years with diffusion models (Rombach et
al., 2022; Nichol et al., 2022), it lacks precise control over the location
and the boundaries of objects, which are important properties for creative
content generation tasks like photo editing, inpainting, and for data
augmentation in discriminative learning (Zhao et al., 2023; Akrout et al.,
2023; Azizi et al., 2023; Hemmat et al., 2023). Consequently, in this work we
focus on semantic image synthesis (Isola et al., 2017; Wang et al., 2018; Park
et al., 2019; Schönfeld et al., 2021; Wang et al., 2022b, a), where the goal
is to produce an image, given a segmentation map, with every pixel assigned to
a category, as input. Because of the one-to-many nature of the mapping, prior
works have tackled this problem under a conditional GAN (Goodfellow et al.,
2014) framework by exploring different conditioning mechanisms in GANs to do
stochastic generations that correspond to the input label map (Isola et al.,
2017; Wang et al., 2018; Park et al., 2019). Others developed conditional
discriminator models, which avoid image-to-image reconstruction losses that
compromise diversity in generated images (Schönfeld et al., 2021). Diffusion
models (Wang et al., 2022b, a) have also been investigated for this problem.
SDM (Wang et al., 2022b) learns to incorporate spatially adaptive
normalization layers into the diffusion process of a latent diffusion model,
while PITI (Wang et al., 2022a) finetunes pre-trained text-to-image diffusion
models such as (Nichol et al., 2022). In comparison to GANs, diffusion models
often result in improved image quality, but suffer from lower consistency with
the input segmentation maps, and are slower during inference due to the
iterative denoising process (Careil et al., 2023).

To improve the image quality and consistency of GAN-based approaches, we
explore the use of pre-trained image backbones in discriminators for semantic
image synthesis. Although leveraging pre-trained image models is common in
many other vision tasks, such as classification, segmentation, or detection,
and more recently for class-conditional GANs (Sauer et al., 2021), to our
knowledge this has not been explored for semantic image synthesis. To this
end, we develop a UNet-like encoder-decoder architecture where the encoder is
a fixed pre-trained image backbone, which leverages the multi-scale feature
representations embedded therein, and the decoder is a convolutional residual
network. We also propose a novel generator architecture, building on the dual-
pyramid modulation approach (Li et al., 2022) with an improved label map
encoding through attention mechanisms for better diversity and global
coherence among the images generated. Finally, we introduce novel loss and
regularization terms, including a contrastive loss and a diversity constraint,
which improve generation quality and diversity.

We validate our contributions with experiments on the ADE-20K, COCO-Stuff, and
Cityscapes datasets. Our model, termed DP-SIMS for “Discriminator Pretraining
for Semantic IMage Synthesis”, achieves state-of-the-art performance in terms
of image quality (measured by FID) and consistency with the input segmentation
masks (measured by mIoU) across all three datasets. Our results not only
surpass recent diffusion models on both metrics, but also come with two orders
of magnitude faster inference.

In summary, our main contributions are the following:

  * •

We develop an encoder-decoder discriminator that leverages feature
representations from pre-trained networks.

  * •

We propose a generator architecture using attention mechanisms for noise
injection and context modeling.

  * •

We introduce additional loss and regularization terms that improve sample
quality and diversity.

  * •

We outperform state-of-the-art GAN and diffusion-based methods in image
quality, input consistency, and speed.

##  2 Related work

Generative image modeling. Several frameworks have been explored in deep
generative modeling, including GANs (Goodfellow et al., 2014; Karras et al.,
2021; Isola et al., 2017; Park et al., 2019; Schönfeld et al., 2021; Kang et
al., 2023; Huang et al., 2022), VAEs (Kingma and Welling, 2014; Vahdat and
Kautz, 2020; Razavi et al., 2019), flow-based models (Dinh et al., 2017;
Kingma and Dhariwal, 2018; Fan et al., 2022) and diffusion-based models
(Dhariwal and Nichol, 2021; Ho et al., 2020; Rombach et al., 2022; Wang et
al., 2022b, a). GANs consist of generator and discriminator networks which
partake in a mini-max game that results in the generator learning to model the
target data distribution. GANs realized a leap in sample quality, due to the
mode-seeking rather than mode-covering nature of their objective function
(Nowozin et al., 2016; Lucas et al., 2019). More recently, breakthrough
results in image quality have been obtained using text-conditioned diffusion
models trained on large-scale text-image datasets (Gafni et al., 2022; Ramesh
et al., 2022; Rombach et al., 2022; amd William Chan et al., 2022; Nichol et
al., 2022). The relatively low sampling speed of diffusion models has
triggered research on scaling GANs to training on large-scale datasets to
achieve competitive image quality while being orders of magnitude faster to
sample (Kang et al., 2023).

Figure 2: Architecture of our discriminator model. The encoder consists of a
pre-trained feature backbone $\mathcal{F}_{\theta}$ (left), residual blocks at
full image resolution (top), and trained feature decoder that aggregates the
multi-scale features from the frozen backbone (right).

Semantic image synthesis. Early approaches for semantic image synthesis
leveraged cycle-consistency between generated images and conditioning masks
(Isola et al., 2017; Wang et al., 2018) and spatially adaptive normalization
(SPADE) layers (Park et al., 2019). These approaches combined adversarial
losses with image-to-image feature-space reconstruction losses to enforce
image quality as well as consistency with the input mask (Zhang et al., 2018).
OASIS (Schönfeld et al., 2021) uses a UNet discriminator model which labels
pixels in real and generated images with semantic classes and an additional
“fake” class, which overcomes the need for feature-space losses that
inherently limit sample diversity, while also improving consistency with the
input segmentation maps. Further improvements have been made by adopting
losses to learn image details at varying scales (Li et al., 2022), or through
multi-modal approaches such as PoE-GAN (Huang et al., 2022), which leverage
data from different modalities like text, sketches and segmentations.

Several works have explored diffusion models for semantic image synthesis.
SPADE layers were incorporated in the denoising network of a diffusion model
in SDM (Wang et al., 2022b) to align the generated images with semantic input
maps. PITI (Wang et al., 2022a) replaced the text-encoder of pre-trained text-
to-image diffusion models, with a label map encoder, and fine-tuned the
resulting model. In our work, rather than relying on generative pre-training
as in PITI, we leverage discriminative pre-training. Another line of work
considers generating images from segmentation maps with free-text annotations
(Couairon et al., 2023; Avrahami et al., 2023). These diffusion approaches,
however, suffer from poor consistency with the input label maps while also
being much more computationally demanding during inference than their GAN-
based counterparts.

Pre-trained backbones in GANs. Pre-trained feature representations have been
explored in various ways in GAN training. When the model is conditioned on
detailed inputs, such as sketches or segmentation maps, pre-trained backbones
are used to define a reconstruction loss between the generated and training
images (Zhang et al., 2018). Another line of work leverages these backbones as
fixed encoders in adversarial discriminators (Sauer et al., 2021; Richter et
al., 2022). Naively using a pre-trained encoder with a fixed decoder yields
suboptimal results, thus the Projected GANs model (Sauer et al., 2021) uses a
feature conditioning strategy based on random projections to make the
adversarial game more balanced. While this approach is successful with some
backbones, the method worked best with small pre-trained models such as
EfficientNets (Tan and Le, 2019), while larger models resulted in lower
performance. A related line of work (Kumari et al., 2022) uses an ensemble of
multiple pre-trained backbones to obtain a set of discriminators from which a
subset is selected at every step for computing the most informative gradients.
This produced impressive results but has the following significant overheads
which make it inefficient: (i) all the discriminators and their associated
optimizers are stored in memory, (ii) there is a pre-inference step to
quantify the suitability of each discriminator for any given batch, and (iii)
the main discriminator is trained from scratch. Our work is closely related to
Projected GANs, but to our knowledge the first one to leverage pre-trained
discriminative feature networks for semantic image synthesis.

Attention in GANs. While most of the popular GAN frameworks such as the
StyleGAN family relied exclusively on convolutions (Karras et al., 2019, 2020,
2021), some other works explored the usage of attention in GANs to introduce a
non-local parametrization that operates beyond the receptive field of the
convolutions in the form of self attention (Brock et al., 2019; Lee et al.,
2022; Zhang et al., 2019; Hudson and Zitnick, 2021; Kang et al., 2023), as
well as cross-attention to incroporate information from different modalities
(text-to-image). To the best of our knowledge, this is the first work to
explore the usage of cross-attention for semantic image synthesis.

##  3 Method

Semantic image synthesis aims to produce realistic RGB images ${\bf
g}\in\mathbb{R}^{W\times H\times 3}$ that are diverse and consistent with an
input label map ${\bf t}\in\mathbb{R}^{W\times H\times C}$, where $C$ is the
number of semantic classes and $W\times H$ is the spatial resolution. A one-
to-many mapping is ensured by conditioning on a random noise vector ${\bf
z}\sim\mathcal{N}({\bf 0,I})$ of dimension $d_{z}$.

In this section, we present our GAN-based approach, starting with our method
to leverage pre-trained feature backbones in the discriminator (Section 3.1).
We then describe our noise injection and label map modulation mechanisms for
the generator (Section 3.2), and detail the losses we use to train our models
(Section 3.3).

###  3.1 Pre-trained discriminator backbones

We use a UNet architecture (Ronneberger et al., 2015) for the discriminator,
similar to OASIS (Schönfeld et al., 2021). The discriminator is trained to
classify pixels in real training images with the corresponding ground-truth
labels, and the pixels in generated images as “fake”. Note that the
discriminator in (Schönfeld et al., 2021) is trained from scratch and does not
benefit from any pre-training.

Let $\mathcal{F}_{\theta}$ be a pre-trained feature backbone with parameters
$\theta$. We use this backbone, frozen, as part of the “encoder” in the UNet
discriminator. Let ${\bf F}_{l}\in\mathbb{R}^{C_{l}\times W_{l}\times H_{l}}$
denote the features extracted by the backbone at levels $l=1,\dots,L$, which
generally have different spatial resolutions $W_{l}\times H_{l}$ and number of
channels $C_{l}$. These features are then processed by the UNet “decoder”,
which is used to predict per-pixel labels spanning the semantic categories
present in the input label map, as well as the “fake” label. Additionally, to
exploit high-frequency details in the image, we add a fully trainable path at
the full image resolution with two relatively shallow residual blocks. The
full discriminator architecture is illustrated in Figure 2.

Feature conditioning. An important problem with using pre-trained backbones is
feature conditioning. Typical backbones are ill-conditioned, meaning that some
features are much more prominent than others. This makes it difficult to fully
exploit the learned feature representation of the backbone as strong features
overwhelm the discriminator’s decoder and result in exploring only certain
regions in the feature representation of the encoder. Previously, (Sauer et
al., 2021) tried to alleviate this problem by applying cross-channel mixing
(CCM) and cross-scale mixing (CSM) to the features, while (Kumari et al.,
2022) average the signals from multiple discriminators to obtain a more
diluted signal. Empirically, the first approach underperforms in many of our
experiments, as the strong features still tend to mask out their weaker, yet
potentially relevant, counterparts. On the other hand, the second introduces a
large overhead from the multiple models being incorporated in training. In our
work, we develop a method that better exploits the feature representation from
the encoder. We achieve this by aiming to make all features have a comparable
contribution to the downstream task.

Consider a feature map ${\bf F}_{l}\in\mathbb{R}^{C_{l}\times W_{l}\times
H_{l}}$ at scale $l$ from the pre-trained backbone. First, we apply a
contractive non-linearity (CNL) such as sigmoid to obtain ${\bf
F}^{\prime}_{l}=\sigma({\bf F}_{l})$. Next, we normalize the features to
ensure they have a similar contribution in the following layers. We choose
batch normalization, yielding $\tilde{\bf F}_{l}=({\bf
F}^{\prime}_{l}-\mu_{l})/\sigma_{l}$, where $\mu_{l}$ and $\sigma_{l}$ are the
batch statistics. In this manner, all features are in a similar range and
therefore the decoder does not prioritize features with a high variance or
amplitude.

###  3.2 Generator architecture

Figure 3: Our generator architecture consist of two components. (i) A
conditional image generation network (top) that takes a low-resolution label
map as input and produces the full-resolution output image. (ii) A semantic
map encoding network (bottom) that takes the full resolution label map as
input and produces multi-scale features that are used to modulate the
intermediate features of the image generation network.

Our generator architecture is based on DP-GAN (Li et al., 2022), but offers
two main novelties: a revisited noise injection mechanism and improved
modeling of long-range dependencies through self-attention. Following DP-GAN,
we use a mask encoding network to condition the SPADE blocks, rather than
conditioning the SPADE blocks on the label maps via a single convolution
layer, which cannot take into account longer-range dependencies encoded in the
label map. Each block of the label map encoding pyramid is made of a single
convolution layer with downsampling followed by batch norm, GELU activation
(Hendrycks and Gimpel, 2016), attention modules, and a pointwise convolution
layer. For every scale, we obtain a modulation map ${\bf
h}_{i},i\in\\{1,\dots,L\\}$ which, concatenated with a resized version of the
ultimate map ${\bf h}_{L}$, will serve as conditioning for the SPADE block at
the same resolution.

While (Schönfeld et al., 2021) argued that concatenating a spatial noise map
to the label map was enough to induce variety in the generated images, since
the noise is present in all SPADE blocks, and therefore hard to ignore, the
same cannot be said for the architecture of DP-GAN (Li et al., 2022). The
noise is injected only at the first layer of the label map encoding network,
hence it is much easier to ignore. Consequently, we propose a different
mechanism for noise injection, making use of cross-attention between the
learned representations at the different scales and the mapping noise obtained
by feeding ${\bf z}$ to a three-layer MLP, ${\bf w}=\text{MLP}({\bf
z})\in\mathbb{R}^{n_{k}\times d_{w}}$. Let ${\bf
h}_{i}\in\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}$ be the downsampled
feature representation from the previous scale, ${\bf h_{i}}$ first goes
through a convolution to provide an embedding of the label map, then the
spatial dimensions are flattened and projected via a linear layer to obtain
the queries $Q\in\mathbb{R}^{H_{i}W_{i}\times d_{q}}$. The transformed noise
vector ${\bf w}$ is projected via two linear layers to obtain the keys and the
values $K,V\in\mathbb{R}^{n_{k}\times d_{q}}$, then the cross-attention is
computed as:

${\bf A}=\text{SoftMax}\left(\frac{QK^{\top}}{\sqrt{d_{q}}}\right)V.$ (1)

Each noise injection block at lower spatial resolutions ($64\times 64$ and
lower) uses a noise injection module made of a residual cross-attention block

$a({\bf h}_{i},{\bf w})={\bf h_{i}}+\eta_{i}\cdot{\bf A}({\bf h_{i}},{\bf
w}),$ (2)

where $\eta_{i}\in\mathbb{R}$ is a trainable gating parameter initialized at
0. The noise injection is followed by a residual self-attention block, before
having a convolution output the conditioning at scale $i$. For higher
resolutions where attention modules are too costly, we use convolutional
blocks only. A diagram of the generator architecture is provided in Figure 3.

###  3.3 Training

We train our models by minimizing a weighted average of three loss functions
which we detail below.

Pixel-wise focal loss. Our main loss is based on a pixel-wise GAN loss
(Schönfeld et al., 2021), where the discriminator aims to assign pixels in
real images to the corresponding class in the conditioning label map, and
those in generated images to an additional “fake” class. To improve the
performance on rare classes, we replace the weighted cross-entropy of
(Schönfeld et al., 2021) with a weighted focal loss (Lin et al., 2017), while
keeping the same weighting scheme as in (Schönfeld et al., 2021). Let $p({\bf
x})\in[0,1]^{H\times W\times(C+1)}$ denote the output class probability map of
the discriminator for a real RGB image $\bf x$, and $p({\bf
g})\in[0,1]^{H\times W\times(C+1)}$ be the probability map for a generated
image ${\bf g}=G({\bf z,t})$, where the label index $C+1$ is used for the
“fake” class. Then, the discriminator loss is:

$\displaystyle\mathcal{L}_{\textrm{D}}=-\mathbb{E}_{({\bf
x,t})}\sum_{c=1}^{C}\alpha_{c}\sum_{i=1}^{H\times W}{\bf
t}_{i,c}\left(1-p({\bf x})_{i,c}\right)^{\gamma}\log p({\bf x})_{i,c}$
$\displaystyle-\mathbb{E}_{({\bf g,t})}\sum_{i=1}^{H\times W}\left(1-p({\bf
g})_{i,C+1}\right)^{\gamma}\log p({\bf g})_{i,C+1},$ (3)

where the $\alpha_{c}$ are the class weighting terms and $\gamma$ is a hyper-
parameter of the focal loss. The standard cross-entropy is recovered for
$\gamma=0$, and for $\gamma>0$ the loss puts more weight on poorly predicted
labels.

The pixel-wise loss for the generator is then takes the form:

$\mathcal{L}_{\textrm{G}}=-\mathbb{E}_{(\bf
g,t)}\sum_{c=1}^{C}\alpha_{c}\sum_{i=1}^{H\times W}{\bf t}_{i,c}\left(1-p({\bf
g})_{i,c}\right)^{\gamma}\log p({\bf g})_{i,c}.$ (4)

Using the focal loss, both the generator and discriminator put more emphasis
on pixels that are incorrectly classified, these often belong to rare classes
which helps to improve performance for these under-represented classes. To
prevent the discriminator output probabilities from saturating and thus
leading to vanishing gradients, we apply one-sided label smoothing (Salimans
et al., 2016) by setting the cross-entropy targets to $1-\epsilon$ for the
discriminator loss, where $\epsilon$ is a hyper-parameter.

Contrastive loss. We define a patch-wise contrastive loss that encourages the
generated images to be globally coherent. Our contrastive framework is based
on InfoNCE (Oord et al., 2019) which aims to bring matching patch features
closer together, and push them further from non-matching features. Given a
pair $({\bf x,t})$ of image and label map, we generate a corresponding image
${\bf g}=G({\bf z},{\bf t})$, and use $\bf H_{x}$ and $\bf H_{g}$ the
corresponding multi-scale features obtained from a pre-trained VGG network
(Simonyan and Zisserman, 2015). For every scale, we sample matching features
$\bf z,z^{+}$ from the same spatial coordinates in $\bf H_{g}$ and $\bf H_{x}$
respectively. Additionally, we sample $N$ non-matching features ${\bf
z}^{-}_{n}$ at randomly selected coordinates from $\bf H_{x}$.

The features are then projected to an embedding space using a convolution
followed by a two-layer MLP to obtain ${\bf v,v^{+}},{\bf
v}^{-}_{n}\in\mathbb{R}^{d_{v}}$ before computing the InfoNCE loss as

$\mathcal{L}_{\textrm{NCE}}({\bf
v,v^{+}{\bf\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0},}v^{-}})=-\log\left(\frac{e^{\bf
v^{\top}v^{+}/\tau}}{e^{\bf v^{\top}v^{+}/\tau}+\sum_{n=1}^{N}e^{{\bf
v^{\top}}{\bf v}^{-}_{n}/\tau}}\right),$ (5)

where $\tau$ is a temperature parameter controlling the sharpness in the
response of the loss. We apply the loss at feature scales $1/4,1/8,1/16$, and
take their sum. This is similar to the contrastive losses used for image-to-
image translation (Park et al., 2020), and the main difference comes from the
feature representation from which the loss is calculated. While other methods
reuse the encoder features from their translation network, we obtain the
feature pyramid from a VGG network and process these features by a simple
module made of a convolution block followed by a projection MLP.

Diversity loss. To promote diversity among the generated images we introduce a
loss that encourages two images generated with the same mask, but different
latents $\bf z$, to be sufficiently distinct from each other. In particular,
we define

$\mathcal{L}_{\textrm{Div}}=\left[\tau_{\text{div}}-\frac{\left\lVert\sigma\circ
G^{f}({\bf z}_{1},{\bf t})-\sigma\circ G^{f}({\bf z}_{2},{\bf
t})\right\rVert_{1}}{\lVert{\bf z}_{1}-{\bf z}_{2}\rVert_{1}}\right]^{+},$ (6)

where $[\cdot]^{+}=\max(0,\cdot)$ retains the non-negative part of its
argument, $G^{f}$ is the output of the generator before the final convolution
and $\sigma$ the sigmoid function. We adopt a cutoff threshold
$\tau_{\text{div}}$ on the loss in order to not overly constrain the
generator, and apply this loss only for similar samples given the same label
map.

| COCO | ADE20k | Cityscapes
---|---|---|---
|  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$ ($\uparrow$) |  mIoU ($\uparrow$) |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$ ($\uparrow$) |  mIoU ($\uparrow$) |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$ ($\uparrow$) |  mIoU ($\uparrow$)
Pix2pixHD (Wang et al., 2018) | 111.5 | — | 14.6 | 73.3 | — | 22.4 | 104.7 | — | 52.4
SPADE (Park et al., 2019) | 22.6 | — | 37.4 | 33.9 | — | 38.5 | 71.8 | — | 62.3
OASIS (Schönfeld et al., 2021) | 17.0 | 52.1 | 44.1 | 28.3 | 53.5 | 48.8 | 47.7 | 72.0 | 69.3
DP-GAN (Li et al., 2022) | — | — | — | 26.1 | — | 52.7 | 44.1 | — | 73.6
PoE-GAN (Huang et al., 2022) | 15.8 | — | — | — | — | — | — | — | —
SDM (Wang et al., 2022b) | 15.9 | 40.3 | 36.8 | 27.5 | 51.9 | 44.0 | 42.1 | 72.8 | 69.1
PITI (Wang et al., 2022a) | 15.5 | 31.2 | 29.5 | — | — | — | — | — | —
DP-SIMS (ours) | 13.6 | 65.2 | 57.7 | 22.7 | 67.8 | 54.3 | 38.2 | 78.5 | 76.3

Table 1: Comparison of our results to the state-of-the-art GAN-based (first
block) and diffusion-based (second block) methods. Scores for these methods
are taken from their corresponding papers. We also report mIOU scores with
Mask2Former (mIoU${}_{\textrm{MF}}$) for methods where pre-trained checkpoints
or generated images are available.

##  4 Experiments

We present our experimental setup in Section 4.1, followed by our main results
in Section 4.2, and ablations in Section 4.3.

Backbone | Prms. | FLOPS | Acc@1 |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$ ($\uparrow$)
---|---|---|---|---|---
Swin-B | 107 | 15.4G | 86.4 | 29.5 | 55.4
ResNet-50 | 44 | 4.1G | 76.2 | 24.6 | 60.5
EfficientNet-Lite1 | 3 | 631M | 83.4 | 24.5 | 63.1
ConvNeXt-B | 89 | 15.4G | 85.1 | 23.5 | 63.5
ConvNeXt-L | 198 | 34.4G | 85.5 | 22.7 | 67.8

Table 2: Comparison of different backbone architectures on ADE-20K (ordered by
FID). The number of parameters (Prms.) is reported in millions. For each
backbone, we also report the ImageNet-1k top-1 accuracy (Acc@1) for reference.

###  4.1 Experimental setup

Datasets. We consider three popular datasets to benchmark semantic image
synthesis: COCO-Stuff, Cityscapes, ADE-20K. COCO-Stuff provides 118k training
images and 5k validation images, labeled with 183 classes. Cityscapes contains
2,975 training images along with a validation set of 500 images, and uses 35
labels. ADE-20K holds 26k images with object segmentations across 151 classes.
Similar to (Park et al., 2019; Wang et al., 2022b, 2018), we use instance-
level annotations when available. For COCO-Stuff and Cityscapes, we use
instance segmentations as in (Cheng et al., 2020), by creating vertical and
horizontal offset maps of every foreground pixel w.r.t. it’s object center of
mass, and concatenate these to the semantic label maps as input for the model.
For ADE-20K there are no instance segmentations available. We generate images
at a resolution of $256\times 256$ for ADE-20K and COCO-Stuff, and $256\times
512$ for Cityscapes. In order to avoid training models on identifiable
personal data, we use anonymized versions of the datasets; see the
supplementary material for more details.

Metrics. We compute FID (Heusel et al., 2017) to assess image quality. We also
report the mean intersection-over-union score (mIoU) to measure the
consistency of the generated image with the input segmentation maps. For a
fair comparison with previous work (Schönfeld et al., 2021; Li et al., 2022;
Park et al., 2019), we used the segmentation models from these works for
inferring label maps of the generated images: UperNet101 (Xiao et al., 2018)
for ADE-20K, multi-scale DRN-D-105 (Yu et al., 2017) for Cityscapes, and
DeepLabV2 (Chen et al., 2018) for COCO-Stuff. We refer to the scores obtained
with these models as mIoU. In addition, we infer label masks using Mask2Former
(Cheng et al., 2022), which is more accurate than other segmentation models,
thus yielding a more meaningful comparison to the ground-truth masks. We
denote the resulting scores as mIoU${}_{\textrm{MF}}$. Additional details are
provided in the supplementary material.

Implementation details. We counter the strong class imbalance in the datasets
used in our experiments with a sampling scheme favoring rare classes. Let
$f_{c}$ be the fraction of training images where class $c$ appears, then each
image is sampled with a probability proportional to $f_{k}^{-1/2}$ with $k$
the sparsest class present in the image.

Each of our models is trained on one or two machines with eight V100 GPUs. We
set the total batch size at $64$ and use ADAM optimizer in all our experiments
with a learning rate of $10^{-3}$ and momentums
$\beta_{1}\\!=\\!0,\beta_{2}\\!=\\!0.99$. For pre-trained Swin backbones, we
found it necessary to use gradient clipping to stabilize training. Following
prior work, we track an exponential moving average of the generator weight and
set the decay rate to $\alpha=0.9999$. For the contrastive loss, we set the
weighting factor $\lambda_{C}=100$, the temperature $\tau=0.3$ and select
$N=128$ negative samples. We set $\lambda_{\text{GAN}}=1$ for the GAN loss and
$\lambda_{D}=10$ for the diversity loss. For the focal loss, we set the
focusing parameter $\gamma=2$.

Figure 4: Qualitative comparison with prior work on the Cityscapes dataset. We
show the results of OASIS (Schönfeld et al., 2021), SDM (Wang et al., 2022b),
and our approach along with the corresponding label map used for generating
each image. Note that our method generates more coherent objects with
realistic textures in comparison.

###  4.2 Main results

Comparison to the state of the art. In Table 1, we report the results obtained
with our model in comparison to the state of the art. Our DP-SIMS method (with
ConvNext-L backbone) achieves the best performance across metrics and
datasets. On COCO-Stuff, we improve the FID of 15.5 from PITI to 13.6, while
improving the mIoU${}_{\textrm{MF}}$ of 52.1 from OASIS to 65.2. For ADE-20K,
we observe a similar trend with an improvement of 3.4 FID points w.r.t. DP-
GAN, and an improvement of 14.5 points in mIoU${}_{\textrm{MF}}$ w.r.t. OASIS.
For Cityscapes we obtain smaller improvements of 3.9 FID points w.r.t.
Semantic Diffusion and 5.5 points in mIoU${}_{\textrm{MF}}$. See Figure 1 and
Figure 4 for qualitative comparisons of model trained on COCO-Stuff and
Cityscapes. Please refer to the supplementary material for additional
examples, including ones for ADE-20K.

Pre-training | Acc@1 |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$ ($\uparrow$)
---|---|---|---
IN-1k@224 | 84.3 | 22.7 | 62.8
IN-21k@224 | 86.6 | 23.6 | 64.1
IN-21k@384 | 87.5 | 22.7 | 67.8
Table 3: Influence of discriminator pre-training on the overall performance
for ADE-20K using a ConvNext-L backbone.

Encoder architecture. We experiment with different pre-trained backbone
architectures for the discriminator in Table 2. All the encoders were trained
for ImageNet-1k classification. We find that the attention-based Swin
architecture (Liu et al., 2021) has the best ImageNet accuracy, but compared
to convolutional models performs worse as a discriminator backbone for
semantic image synthesis, and tends to be more unstable, often requiring
gradient clipping to converge. For the convolutional models, better
classification accuracy translates to better FID and mIoU${}_{\textrm{MF}}$.

Pre-training dataset. In Table 3, we analyze the impact of pre-training the
ConvNext-L architecture in different ways and training our models on top of
these, with everything else being equal. We consider pre-training on
ImageNet-1k (IN-1k@224) and ImageNet-21k (IN-21k@224) at $224\times 224$
resolution, and also on ImageNet-21k at $384\times 384$ resolution
(IN-21k@384). In terms of mIoU${}_{\textrm{MF}}$, the results are in line with
those observed for different architectures: discriminators trained with larger
datasets (IN-21k) and on higher resolutions perform the best. On the other
hand, we find that for FID, using the standard ImageNet (IN-1k@224) results in
better performance than its bigger IN-21k@224 counterpart, and performs as
well as IN-21k@384 pre-training. This is likely due to the use of the same
dataset in the Inception model (Szegedy et al., 2016), which is the base for
calculating FID, introducing a bias in the metric.

Model | Gen. steps | Ups. steps | $\Delta t_{\text{gen}}$ | $\Delta t_{\text{ups}}$ | $\Delta t_{\text{tot}}$
---|---|---|---|---|---
PITI | 250 | 27 | 14.3 | 3.1 | 17.4
PITI | 27 | 27 | 1.5 | 3.1 | 4.6
SDM | 1000 | — | 260.0 | — | 260.0
DP-SIMS | — | — | — | — | 0.04

Table 4: Comparison of inference time (in seconds) of PITI, SDM and our GAN-
based model. We show the time taken by the generative ($\Delta
t_{\text{gen}}$) and the upsampling ($\Delta t_{\text{ups}}$) models in
addition to the total time ($\Delta t_{\text{tot}}$) for these steps.

Inference speed. An important advantage of GAN models over their diffusion
counterparts is their fast inference. While a GAN only needs one forward pass
to generate an image, a diffusion model requires several iterative denoising
steps, resulting in slower inference, which can hamper the practical usability
of the model. In Table 4 we report the inference speed for generating a single
$256\times 256$ image, averaged over $50$ different runs. PITI uses 250
denoising steps for the generative model at $64\times 64$ resolution and 27
steps for the upsampling model by default, while SDM uses 1000 steps at full
resolution. We also benchmark using 27 steps for the PITI generative model.
Our generator is two to three orders of magnitude faster than its diffusion
counterparts.

|  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$ ($\uparrow$)
---|---|---
Baseline - no normalization | 27.8 | 58.6
Projected GAN | 28.9 | 59.1
DP-SIMS w/o sigmoid | 24.9 | 62.7
DP-SIMS w/o BatchNorm | 26.0 | 61.6
DP-SIMS | 24.5 | 63.1
Table 5: Ablation on feature conditioning shown on ADE-20K with EfficientNet-
Lite1 backbone.

###  4.3 Ablations

Feature Conditioning. We perform an ablation to validate our feature
conditioning mechanisms on ADE-20K in Table 5. We compare the normalization in
DP-SIMS to: (i) a baseline approach where the backbone features are unchanged
before entering the discriminator network (“Baseline - no normalization”),
(ii) the Projected GAN (Sauer et al., 2021) approach with cross-channel and
scale mixing, and using the normalization layer without (iii) the contractive
non-linearity (“/ours w/o sigmoid”) or (iv) normalisation (“DP-SIMS w/o
BatchNorm”). For a fair comparison with (Sauer et al., 2021), these
experiments are conducted on their best reported backbone, EfficientNet-Lite1
(Tan and Le, 2019). Compared to the baseline method, the Projected GAN
approach improves mIoU${}_{\textrm{MF}}$ by 0.5 points, but degrades FID by
1.1 points. Using our feature conditioning based on BatchNorm and sigmoid
activation, we improve the mIoU${}_{\textrm{MF}}$ by 4.5 points and FID by 3.3
points w.r.t. the baseline. When we leave out the sigmoid activation both
mIoU${}_{\textrm{MF}}$ and FID drop by 0.4 points. We observe a similar
behavior without normalization: FID and mIoU${}_{\textrm{MF}}$ both decrease
by 1.5 points. These results validate the fittingness of our feature
conditioning approach.

Architectural modifications. In Table 6, we perform an ablation on our
proposed architectural modifications. Swapping out our generator or
discriminator with the ones from OASIS, suggests that most of the gains are
due to our discriminator design. Using the OASIS discriminator instead of ours
deteriorates mIoU${}_{\textrm{MF}}$by 18.8 points and FID by 6.6 points. We
also experiment with removing the cross-attention noise injection mechanism
and replacing it with the usual concatenation instead, as well as leaving out
the self-attention layers. Both of these contribute to the final performance
in a notable manner. Finally, we include an ablation on label smoothing, which
deteriorates FID by 0.3 and mIoU${}_{\textrm{MF}}$by 1.4 points when left out.

|  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$ ($\uparrow$)
---|---|---
DP-SIMS | 22.7 | 67.8
Generator architecture
OASIS disc + our gen | 29.3 | 49.0
OASIS gen + our disc | 25.6 | 63.6
Ours w/o self-attention | 23.7 | 65.4
Ours w/o cross-attention | 23.6 | 64.5
Training
Ours w/o label smoothing | 23.0 | 66.3
Ours w/o contrastive loss | 25.1 | 66.0
Table 6: Ablations on the architectural design and training losses, shown on
ADE-20K with ConvNext-L backbone.

Contrastive loss. To assess the importance of the contrastive loss, we perform
an ablation in the last row of Table 6 where we remove it during training.
This substantially impacts the results: worsening FID by 2.4 and
mIoU${}_{\textrm{MF}}$ by 1.8 points. In Table 7 we evaluate different values
for the temperature parameter $\tau$. We find an optimal temperature parameter
$\tau_{C}=0.3$, using $\lambda_{c}=100$.

$\tau$ | 0.07 | 0.3 | 0.7 | 2.0
---|---|---|---|---
FID | 25.7 | 22.7 | 24.1 | 26.4
mIoU${}_{\textrm{MF}}$ | 62.6 | 67.8 | 66.3 | 61.4
Table 7: Influence of the contrastive loss evaluated on ADE-20K.

Focal loss. In Table 8, we consider the impact of the focal loss by comparing
it to the weighted cross-entropy loss, as used in OASIS, and the effect of
class weighting in the focal loss. This comparison is conducted on Cityscapes
because it is more imbalanced than ADE-20K. We find that switching from
weighted cross-entropy to the focal loss improves FID by 0.5 points but
worsens mIoU${}_{\textrm{MF}}$ by 0.9 points. When comparing weighted focal
loss to weighted CE, we observe that FID improves by 1.6 points, and
mIoU${}_{\textrm{MF}}$ by 2.6 points.

Loss |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$($\uparrow$)
---|---|---
Weighted CE | 39.8 | 75.9
Focal | 39.3 | 75.0
Weighted Focal | 38.2 | 78.5
Table 8: Comparison of pixel-wise losses on Cityscapes with ConvNext-L
backbone.

Diversity. We investigate the effect of the diversity loss on the variability
of generated images. Following (Schönfeld et al., 2021), we report the mean
LPIPS distance across 20 synthetic images from the same label map, averaged
across the validation set, in Table 9. A qualitative example is provided in
Figure 5 showing a clear variety in the images generated. In comparison with
OASIS, we generate more diverse images, with an LPIPS score similar to that of
SPADE, but with a much higher quality, as reported in Table 1, in terms of FID
and mIoU${}_{\textrm{MF}}$.

Model | 3D noise |  LPIPS ($\uparrow$)
---|---|---
SPADE+ | ✓ | 0.16
SPADE+ | ✗ | 0.50
OASIS | ✓ | 0.35
DP-SIMS | ✗ | 0.47
Table 9: Evaluation of the diversity of images generated. Results for SPADE+
and OASIS are taken from (Schönfeld et al., 2021).  Figure 5: Images generated
by varying the noise vector with DP-SIMS trained on COCO-Stuff and using a
ConvNext-L backbone.

##  5 Conclusion

We introduced DP-SIMS that harnesses pre-trained backbones in GAN-based
semantic image synthesis models. We achieve this by using them as an encoder
in UNet-type discriminators, and introduce a feature conditioning approach to
maximize the effectiveness of the pre-trained features. Moreover, we propose a
novel generator architecture which uses cross-attention to inject noise in the
image generation process, and introduce new loss terms to boost sample
diversity and input consistency. We experimentally validate our approach and
compare it to state-of-the-art prior work based on GANs as well as diffusion
models on three standard benchmark datasets. Compared to these, we find
improved performance in terms of image quality, sample diversity, and
consistency with the input segmentation maps. Importantly, with our approach
inference is two orders of magnitude faster than diffusion-based approaches.

In our experiments we found that transformer-based models, such as Swin, can
lead to instability when used as discriminator backbones. Given their strong
performance for dense prediction tasks, it would be worthwhile to further
study and mitigate this issue in future work, hopefully bringing additional
improvements.

## References

  * Akrout et al. (2023) Mohamed Akrout, Bálint Gyepesi, Péter Holló, Adrienn Poór, Blága Kincső, Stephen Solis, Katrina Cirone, Jeremy Kawahara, Dekker Slade, Latif Abid, Máté Kovács, and István Fazekas.  Diffusion-based data augmentation for skin disease classification: Impact across original medical datasets to fully synthetic images.  arXiv 2301.04802, 2023.
  * amd William Chan et al. (2022) Chitwan Saharia amd William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Avrahami et al. (2023) Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin.  Spatext: Spatio-textual representation for controllable image generation.  In _CVPR_ , 2023.
  * Azizi et al. (2023) Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet.  Synthetic data from diffusion models improves imagenet classification.  _TMLR_ , 2023.
  * Brock et al. (2019) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In _ICLR_ , 2019.
  * Careil et al. (2023) Marlène Careil, Jakob Verbeek, and Stéphane Lathuilière.  Few-shot semantic image synthesis with class affinity transfer.  In _CVPR_ , 2023.
  * Chen et al. (2018) Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs.  _PAMI_ , 40(4):834–848, 2018.
  * Cheng et al. (2020) Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen.  Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation.  In _CVPR_ , 2020.
  * Cheng et al. (2022) Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In _CVPR_ , 2022.
  * Couairon et al. (2023) Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek.  Zero-shot spatial layout conditioning for text-to-image diffusion models.  In _ICCV_ , 2023.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alex Nichol.  Diffusion models beat GANs on image synthesis.  In _NeurIPS_ , 2021.
  * Dinh et al. (2017) Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using Real NVP.  In _ICLR_ , 2017.
  * Fan et al. (2022) Weichen Fan, Jinghuan Chen, Jiabin Ma, Jun Hou, and Shuai Yi.  Styleflow for content-fixed image to image translation.  _arXiv_ , 2207.01909, 2022.
  * Gafni et al. (2022) Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _ECCV_ , 2022.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _NeurIPS_ , 2014.
  * Hemmat et al. (2023) Reyhane Askari Hemmat, Mohammad Pezeshki, Florian Bordes, Michal Drozdzal, and Adriana Romero-Soriano.  Feedback-guided data synthesis for imbalanced classification.  _arXiv_ , 2310.00158, 2023.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel.  Bridging nonlinearities and stochastic regularizers with Gaussian error linear units.  _arXiv_ , 1606.08415, 2016.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  In _NeurIPS_ , 2017.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Huang et al. (2022) Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu.  Multimodal conditional image synthesis with product-of-experts GANs.  In _ECCV_ , 2022.
  * Hudson and Zitnick (2021) Drew A. Hudson and C. Lawrence Zitnick.  Generative adversarial transformers.  In _ICML_ , 2021.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  _CVPR_ , 2017.
  * Kang et al. (2023) Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up GANs for text-to-image synthesis.  In _CVPR_ , 2023.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In _CVPR_ , 2020.
  * Karras et al. (2021) Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In _NeurIPS_ , 2021.
  * Kingma and Dhariwal (2018) Diederik Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1$\times$1 convolutions.  In _NeurIPS_ , 2018.
  * Kingma and Welling (2014) Diederik Kingma and Max Welling.  Auto-encoding variational Bayes.  In _ICLR_ , 2014.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv preprint_ , 2023.
  * Kumari et al. (2022) Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Ensembling off-the-shelf models for GAN training.  In _CVPR_ , 2022.
  * Lee et al. (2022) Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu.  ViTGAN: Training GANs with vision transformers.  In _ICLR_ , 2022.
  * Li et al. (2022) Shijie Li, Ming-Ming Cheng, and Juergen Gall.  Dual pyramid generative adversarial networks for semantic image synthesis.  In _BMVC_ , 2022.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _ICCV_ , 2017.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021.
  * Lucas et al. (2019) Thomas Lucas, Konstantin Shmelkov, Karteek Alahari, Cordelia Schmid, and Jakob Verbeek.  Adaptive density estimation for generative models.  In _NeurIPS_ , 2019.
  * Morozov et al. (2021) Stanislav Morozov, Andrey Voynov, and Artem Babenko.  On self-supervised image representations for gan evaluation.  In _ICLR_ , 2021.
  * Nichol et al. (2022) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  In _ICML_ , 2022.
  * Nowozin et al. (2016) Sebastian Nowozin, Botond Cseke, and Ryota Tomioka.  f-GAN: Training generative neural samplers using variational divergence minimization.  In _NeurIPS_ , 2016.
  * Oord et al. (2019) Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv_ , 1807.03748, 2019.
  * Park et al. (2019) Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In _CVPR_ , 2019.
  * Park et al. (2020) Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu.  Contrastive learning for unpaired image-to-image translation.  In _ECCV_ , 2020.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  _arXiv preprint_ , 2204.06125, 2022.
  * Razavi et al. (2019) Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with VQ-VAE-2.  In _NeurIPS_ , 2019.
  * Richter et al. (2022) Stephan R. Richter, Hassan Abu AlHaija, and Vladlen Koltun.  Enhancing photorealism enhancement.  _IEEE TPAMI_ , 45(2):1700–1715, 2022.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In _MICCAI_ , 2015.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training GANs.  In _NeurIPS_ , 2016.
  * Sauer et al. (2021) Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger.  Projected GANs converge faster.  In _NeurIPS_ , 2021.
  * Schönfeld et al. (2021) Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva.  You only need adversarial supervision for semantic image synthesis.  In _ICLR_ , 2021.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_ , 2015.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathan Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _CVPR_ , 2016.
  * Tan and Le (2019) Mingxing Tan and Quoc V. Le.  EfficientNet: Rethinking model scaling for convolutional neural networks.  In _ICML_ , 2019.
  * Vahdat and Kautz (2020) Arash Vahdat and Jan Kautz.  NVAE: A deep hierarchical variational autoencoder.  In _NeurIPS_ , 2020.
  * Wang et al. (2022a) Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen.  Pretraining is all you need for image-to-image translation.  _arXiv_ , 2205.12952, 2022a.
  * Wang et al. (2018) Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional GANs.  In _CVPR_ , 2018.
  * Wang et al. (2022b) Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li.  Semantic image synthesis via diffusion models.  _arXiv preprint_ , 2207.00050, 2022b.
  * Xiao et al. (2018) Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In _ECCV_ , 2018.
  * Yu et al. (2017) Fisher Yu, Vladlen Koltun, and Thomas Funkhouser.  Dilated residual networks.  In _CVPR_ , 2017.
  * Zhang et al. (2019) Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In _ICML_ , 2019.
  * Zhang et al. (2018) R. Zhang, P. Isola, A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.
  * Zhao et al. (2023) Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, Weiming Zhang, and Nenghai Yu.  X-Paste: Revisiting scalable copy-paste for instance segmentation using CLIP and StableDiffusion.  In _ICML_ , 2023.

##  Appendix A More details on the experimental setup

###  A.1 Architecture details

Generator. As can be seen in Figure 3 of the main paper, our generator
consists of a UNet-like architecture with two pyramidal paths. The label map
encoding takes the input segmentation map, and progressively downsamples it to
produce label-conditioned multi-scale features. These features are then used
in the image generation path, which progresively upsamples the signal to
eventually produce an RGB image. The stochasticity of the images generated is
based on conditioning on the noise vector ${\bf z}$. We provide a schematic
overview of the noise injection operation in Figure S1. In Table S1, we
provide additional information on the label map encoding and the image
generation paths.

Figure S1: Schematic overview of our noise injection mechanism using cross-
attention.

In the label map encoding branch, each block is made of the elements listed in
Table S1. Cross-attention and self-attention are only applied at lower
resolutions ($64\times 64$, and lower) where we use an embedding dimension
that is half of the original feature dimension. We downscale the feature maps
by using convolution layers with a stride of 2.

In the image synthesis branch, we follow the same architecture as OASIS
Schönfeld et al. (2021) with the only difference being the SPADE conditioning
maps which are given by the label map encoding path instead of a resized
version of the label maps. We also remove the hyperbolic tangent at the output
of the network as we found it leads to a more stable generator.

For the contrastive learning branch, features obtained from VGG19 go through
three convolutional blocks and two linear layers for projection. We sample
$128$ different patches to obtain negative samples from the image.

Parameter | Description
---|---
Hyperparameters
${\bf z}$ dimension | 64
${\bf w}$ dimension | 256
Batch size | 64
Learning rate | $10^{-3}$
$\beta_{1}$ for Adam | 0
$\beta_{2}$ for Adam | 0.99
EMA beta | 0.9999
Label map encoding
Pyramid block | Conv2d(kernel_size=3), BN, GELU, CrossAttention, BN,
| GELU, SelfAttention, GELU, BN, Conv2d(kernel_size=1)
Self Attention channel divider | 2
Cross Attention channel divider | 2
Conv block | Conv2d(kernel_size=3), BN, GELU, Conv2d(kernel_size=1)
Block type | [Conv, Conv, Conv, Linear, Linear]
Image synthesis branch
Channel base | 64
Number of residual blocks | 6
Channel depths | [1024, 1024, 1024, 512, 256, 128, 64]
Residual block | SPADE, Leaky RELU, Conv2d(3)
Pyramid dimensionality | 64
Hyperbolic tangent on output | No
Contrastive learning branch
Perceptual network | VGG19
Contrastive encoding channels | [64, 128, 256, 512, 512]
Contrastive embedding dimension | 256
Number of patches | 128
Table S1: Architecture details of the generator.

Discriminator. We provide additional details of our discriminator architecture
in Table S2. The residual blocks are made of one convolution with kernel size
3 followed by leaky ReLU, then a pointwise convolution with leaky ReLU. For
the full resolution channel, we set the dimensionality to 128. For the lower
resolution channels, we stick to the same dimensionality as the corresponding
encoder feature. The dimensionality of the final convolution before predicting
the segmentations is set to 256.

We use spectral norm on all convolutional and linear layers in both the
generator and the discriminator.

Parameter | Description
---|---
Hyperparameters
Number of multiscale backbone features | 4
Full resolution embedding dimension | 128
Number of residual blocks | 5
Decoder
Residual block | Conv2d(kernel_size=3), Leaky RELU, Conv2d(kernel_size=1), Leaky RELU
Leaky RELU slope | 0.2
Penultimate channel dimension | 256
Feature conditioning
Conditioning normalization | Batch Norm w.o learned affine
Conditioning non-linearity | Hyperbolic tangent
Table S2: Architecture details of the discriminator.

Feature conditioning. In ProjectedGAN’s paper Sauer et al. (2021), they
observe that when using a fixed feature encoder in the GAN discriminator, only
a subset of features is covered by the projector. They therefore propose
propose to dilute prominent features, encouraging the discriminator to utilize
all available information equally across the different scales. However, they
do not delve deeper into the practical reasons behind this. Namely, feature
encoders trained for a discriminative task will have different structures to
those trained on generative tasks. For the former, models tend to capture a
subset of key features while disregarding other less relevant features. On the
latter however, the model needs an extensive representation of the different
objects it should generate. In practice, this translates to feature encoders
having bad conditioning. The range of activations differs greatly from one
feature to the other, which leads to bias towards a minority features that
have a high amplitude of activations. A simple way to resolve this issue is by
applying normalization these features to have a distribution with zero mean
and a unit standard deviation across the batch.

In some situations, linear scaling of the features might not be enough to have
proper conditioning of the features. Accordingly, we reduce the dynamic range
of the feature maps before the normalization by using a sigmoid activation at
the feature outputs of the pretrained encoder.

Fixed vs finetuned backbone. We experimented with finetuning the whole
discriminator, encoder included. In all of our attempts, such models diverged
without producing meaningful results. We therefore chose to keep the encoder
fixed for this work.

###  A.2 Computation of the mIoU evaluation metrics

To compute the mIoU metric, we infer segmentation maps for generated images.
As noted in the main paper, we infer segmentation maps for the generated
images using the same networks as in OASIS Schönfeld et al. (2021), that is:
UperNet101 Xiao et al. (2018) for ADE-20K, multi-scale DRN-D-105 Yu et al.
(2017) for Cityscapes, and DeepLabV2 Chen et al. (2018) for COCO-Stuff. We
also measure mIoU using Mask2Former Cheng et al. (2022) with Swin-L backbone
Liu et al. (2021) (mIoU${}_{\textrm{MF}}$), which infers more accurate
segmentation mask, thus producing a more accurate comparison to the ground-
truth masks.

In Table S3 we compare the segmentation accuracy on the three datasets we used
in our experiments. The results confirm that Mask2Former is more accurate for
all three datasets, in particular on COCO-Stuff, where it boosts mIoU by more
than 19 points w.r.t. DeepLab-v2.

| ADE-20K | Cityscapes | COCO-Stuff
---|---|---|---
UperNet101 | 42.7 | — | —
MS DRN-D-105 | — | 61.3 | —
DeepLab-v2 | — | — | 35.3
Mask2Former | 45.3 | 69.9 | 54.5
Table S3: Segmentation performance in terms of mIoU on real images using
different segmentation models. To match the setting used in our semantic image
synthesis experiments, evaluation images are downsampled to $256\times 256$
for ADE-20K and COCO, and to $256\times 512$ for Cityscapes.

Dataset | Model | Blurring |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$($\uparrow$)
---|---|---|---|---
| OASIS | ✗ | 17.0 | 52.1
| OASIS | ✓ | 18.8 | 47.1
COCO-Stuff | DP-SIMS (ours) | ✓ | 13.6 | 65.2
| OASIS | ✗ | 28.3 | 53.5
| OASIS | ✓ | 29.1 | 49.6
ADE-20K | DP-SIMS (ours) | ✓ | 22.7 | 67.8
| OASIS | ✗ | 47.7 | 72.0
| OASIS | ✓ | 48.0 | 71.6
Cityscapes | DP-SIMS (ours) | ✓ | 38.2 | 78.5

Table S4: Influence of face blurring on the performance of OASIS.

###  A.3 Influence of face blurring

To avoid training models on identifiable personal data, we did not include
such information in the training dataset. For Cityscapes we use the release of
the dataset with blurred faces and licence plates, which is available publicly
on the website listed in Table 11(a). For ADE-20K and COCO-Stuff we executed
our pipeline to detect, segment and blur faces. To this end, we use Retina-Net
Lin et al. (2017) and segment anything (SAM) Kirillov et al. (2023) to detect
and blur human faces. First we detect the bounding boxes of the faces for each
image in our dataset, then we obtain its corresponding segmentation through
SAM. Finally, this face region detected by SAM is blurred with a Gaussian
filter.

To assess the impact of blurring, we train OASIS on blurred images using the
original source code from the authors and compare to their reported results on
the non-blurred data. We report our results in Table S4. Here, and elsewhere
in the paper, we also use the blurred data to compute FID w.r.t. the generated
images. We see that blurring has a negative impact on FID, most notably for
COCO-Stuff (+1.8), and to a lesser extent for ADE-20K (+0.8) and Cityscapes
(+0.3). The mIoU${}_{\textrm{MF}}$ scores also degrade on all the datasets
when using blurred data: on COCO-Stuff, ADE-20K and Cityscapes by 5.0, 3.9,
and 0.4 points respectively. Note that in all comparisons to the state of the
art, we report metrics obtained using models trained on blurred data for our
approach, and models trained on non-blurred data for other approaches.
Therefore, the real gains of our method over OASIS (and probably other methods
as well) are even larger than what is shown in our comparisons in Table 1 in
the main paper.

###  A.4 Carbon footprint estimation

On COCO-Stuff, it takes approximately 10 days = 240 hours to train our model
using 8 GPUs. On ADE-20K and Cityscapes the training times are about 6 and 4
days respectively. Given a thermal design power (TDP) of the V100-32G GPU
equal to 250W, a power usage effectiveness (PUE) of 1.1, a carbon intensity
factor of 0.385 kg CO2 per KWh, a time of 240 hours $\times$ 8 GPUs = 1920 GPU
hours. The $250\times 1.1\times 1920$ = 528 kWh used to train the model is
approximately equivalent to a CO2 footprint of 528 $\times$ 0.385 = 208 kg of
CO2 for COCO-Stuff. For ADE-20K this amounts to 124 kg of CO2, and 83 kg of
CO2 for Cityscapes.

##  Appendix B Additional experimental results

###  B.1 Quantifying the ImageNet bias

Since our backbones are pre-trained on ImageNet which is the same dataset that
Inceptionv3 Szegedy et al. (2016) used for the standard FID model was trained
on, a natural question arises whether our results are influenced by this bias
towards the ImageNet dataset. To analyze this, we report in Table S5 a
quantitative comparison following the approach outlined in Morozov et al.
(2021), where we compute the Fréchet distance between two Gaussians fitted to
feature representations of the SwAV Resnet50 network that was pretrained in a
self-supervised manner. Our models retain state-of-the-art performance with
respect to this metric on all the three datasets studied. Additionally, we
further experiment with the influence of the backbone pre-training in Table
S6. Differently from the main paper where FID is studied, we find than the
IN-21k checkpoint brings about better performance than its IN-1k counterpart.
While the fine-tuning at high resolution (384 vs 224) also improves swav-FID.

| OASIS | SDM | PITI | DP-SIMS
---|---|---|---|---
COCO-Stuff | 3.09 | 2.68 | 2.52 | 2.14
ADE-20K | 4.35 | 3.85 | — | 2.84
Cityscapes | 4.75 | 3.94 | — | 3.71
Table S5: Evaluation of SwAV Resnet50 FID on different methods. We use ConvNext-L for our method. Pre-training | Acc@1 | FID${}_{\text{SwAV}}$ $(\downarrow)$
---|---|---
IN-1k@224 | 84.3 | 3.03
IN-21k@224 | 86.6 | 2.97
IN21k@384 | 87.5 | 2.84
Table S6: Evaluation of SwAV Resnet50 FID with different pre-trainings. Figure
S2: Noise vector interpolation. By interpolating the noise vector between two
different values, we can identify the factors of variation in the image which
correspond to differences in colors, textures and also object structures.

###  B.2 Influence of diversity loss

We obtain the diversity cutoff threshold $\tau_{\text{div}}$ by computing the
mean distance between different generated images in a batch and averaging
across 5k different batches in the training set:

$\tau_{\text{div}}=\sum_{i,j\in\mathcal{B}}\frac{\left\lVert\sigma\circ
G^{f}(x_{i},{\bf z_{i}})-\sigma\circ G^{f}(x_{j},{\bf
z_{j}})\right\rVert_{1}}{\lVert{\bf z}_{i}-{\bf z}_{j}\rVert_{1}}.$ (7)

The distance is computed in the feature space of the generator before the
final convolution that produces the image. It is then normalized by the
distance between the noise vectors. We found it necessary to add a sigmoid
activation on the features before computing the loss, similar to backbone
conditioning. Not using this activation results in unwanted effects where a
tiny patch in the image has extremely high values leading to a large distance
between two otherwise similar images.

We conduct a more in-depth analysis on the impact of the diversity loss on the
image quality and diversity. We train our model with a ConvNext-L backbone
with different values for the diversity loss $\lambda_{\text{div}}$. These
results are reported in Table S7. Without the diversity loss, the generator
ignores the noise input, which translates to a low LPIPS score. Improving
diversity with a weight of $\lambda_{\text{div}}=10$ results in better image
quality, measured by FID as well as mIoU${}_{\textrm{MF}}$, while adding a
variability factor induced from the noise input. If the weight of the
diversity loss is too high, image quality deteriorates (FID reduces by 0.6
points and mIoU${}_{\textrm{MF}}$ diminishes by 0.1 points) without any
noticeable improvements in diversity (no improvements in LPIPS by setting
$\lambda_{\text{div}}=100$ instead of 10). We provide more examples of diverse
generations in Figure S3, and in Figure S2 we perform noise vector
interpolation in order to better illustrate the variation factors induced by
the injected noise.

$\lambda_{\text{div}}$ | 0 | 10 | 100
---|---|---|---
FID $(\downarrow)$ | 22.9 | 22.7 | 23.3
mIoU${}_{\textrm{MF}}$$(\uparrow)$ | 67.7 | 67.8 | 67.7
LPIPS $(\uparrow)$ | 1.5e-5 | 0.47 | 0.36
Table S7: Influence of diversity loss weight on model performance. We evaluate
image quality using FID and mIoU${}_{\textrm{MF}}$ metrics while diversity is
evaluated using LPIPS. Figure S3: Additional examples of diversity in our
generated images.

###  B.3 Sampling strategy

We quantify the influence of the balanced sampling strategy with respect to
standard uniform sampling on COCO-Stuff and Cityscapes datasets. We report
these results in Table S8, and find that balanced sampling yields performance
gains in both FID and mIoU for both the datasets. In Figure S4, we present
qualitative examples of images generated with the model trained on Cityscapes.
Balanced sampling clearly leads to improvements in the visual quality of
objects such as scooters, buses and trams.

Dataset | Sampling strategy |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$($\uparrow$)
---|---|---|---
| Uniform | 14.1 | 62.9
COCO-Stuff | Balanced | 13.6 | 65.2
| Uniform | 38.7 | 75.6
Cityscapes | Balanced | 38.3 | 78.3
Table S8: Influence of sampling strategy for models trained on the COCO-Stuff
and Cityscapes datasets.  Figure S4: Qualitative examples of images generated
with and without balanced sampling to train models on Cityscapes.

Figure S5: Top: Per-class IOU${}_{\textrm{MF}}$ on Cityscapes with models
trained with different loss functions using ConvNext-L backbone. Labels are
sorted according to their frequency in the validation images, which is written
below the class name. Bottom: Per-class difference in IOU${}_{\textrm{MF}}$ of
models trained with weighted and non-weighted focal loss w.r.t. the model
trained with weighted cross-entropy (CE) loss. (Best viewed in color.)

###  B.4 Influence of pixel-wise loss function

In Figure S5, we compare the per-class mIoU values when training using
different loss functions: weighted cross-entropy (as in OASIS), focal loss,
and weighted focal loss. This extends the class-aggregated results reported in
Table 8 in the main paper. These experiments were conducted on the Cityscapes
dataset using a pre-trained ConvNext-L backbone for the discriminator. Our use
of the weighted focal loss to train the discriminator results in improved IoU
for most classes. The improvements tend to be larger for rare classes. Class
weighting is still important, as can be seen from the deteriorated IoU for a
number of classes when using the un-weighted focal loss.

###  B.5 Influence of instance-level annotations

Since some works do not use the instance masks Schönfeld et al. (2021); Li et
al. (2022); Wang et al. (2022a), we provide an additional ablation where we
train our models on COCO-Stuff and Cityscapes without the instance masks to
isolate the gains in performance they may bring. For both these datasets, we
observe deterioration in the model’s performance when not using instance
masks. The difference is less noticeable on COCO-Stuff where the labels are
already partially separated, FID only increases by 0.3 points. On the other
hand, this difference is more acute in Cityscapes where FID increases by 1.9
points while mIoU${}_{\textrm{MF}}$reduces by 2.2 points. In Cityscapes,
instances are not separated in the semantic label maps, this adds more
ambiguity to the labels presented to the model which makes it more difficult
to interpret them in a plausible manner.

Dataset | Instance masks |  FID ($\downarrow$) |  mIoU${}_{\textrm{MF}}$($\uparrow$)
---|---|---|---
| ✗ | 13.9 | 65.0
COCO-Stuff | ✓ | 13.6 | 65.2
| ✗ | 40.1 | 76.3
Cityscapes | ✓ | 38.2 | 78.5
Table S9: Influence of instance masks on model perofrmance.

###  B.6 Larger discriminators

For larger datasets, having huge encoder architectures could prove beneficial
in capturing the complexty of the dataset. Accordingly, we train a model on
COCO-Stuff using a ConvNext-XL model. It is approximately 1.76 times bigger
than ConvNext-L we used previously, with 350M parameters. In Table S10, we
report its performance as a pre-trained feature encoder in our discriminator.

| FID | mIOU
---|---|---
DP-SIMS (ConvNext-L) | 13.6 | 65.2
DP-SIMS (ConvNext-XL) | 13.3 | 68.0
Table S10: Performance of model trained with ConvNext-XL on COCO-Stuff.

As expected, the larger ConvNext-XL encoder produces even better performance,
reaching state of the art in terms of both FID and mIOU. We observe a
difference of 0.3 points in FID and 2.8 points in mIoU${}_{\textrm{MF}}$.

###  B.7 Qualitative samples

We provide qualitative samples of the images generated with our DP-SIMS model
using different pre-trained backbones for the discriminator in Figure S6. In
Figure S7, Figure S8, and Figure S9 we provide examples of images generated
with our DP-SIMS model and compare to other state-of-the-art models on the
ADE-20K, COCO-Stuff, and Cityscapes datasets, respectively.

Figure S6: Qualitative comparison of DP-SIMS on ADE-20K using different
backbones: Swin-B, Resnet50 (R50), EfficientNet-34, and ConvNext-L.  Figure
S7: Qualitative comparison with prior work on ADE-20K, using a ConvNext-L
backbone for DP-SIMS (Ours). Figure S8: Qualitative comparison with prior work
on COCO-Stuff, using a ConvNext-L backbone for DP-SIMS (Ours). Figure S9:
Qualitative comparison with prior work on Cityscapes, using a ConvNext-L
backbone for DP-SIMS (Ours).

##  Appendix C Assets and licensing information

In Table 11(a) and Table 11(b), we provide the links to the datasets and
models used in or work and their licensing.

Name | Link
---|---
ImageNet | https://www.image-net.org
COCO-Stuff | https://cocodataset.org
Cityscapes | https://www.cityscapes-dataset.com
ADE-20K | https://groups.csail.mit.edu/vision/datasets/ADE20K/
Detectron2 | https://github.com/facebookresearch/detectron2
ConvNext | https://github.com/facebookresearch/ConvNeXt
Swin | https://github.com/microsoft/Swin-Transformer
EfficientNet | https://github.com/lukemelas/EfficientNet-PyTorch
VGG19 | https://github.com/pytorch/vision/blob/main/torchvision/models/vgg.py
Deeplab-v2 | https://github.com/kazuto1011/deeplab-pytorch/
UperNet101 | https://github.com/CSAILVision/sceneparsing
MS DRN-D-105 | https://github.com/fyu/drn
Mask2Former | https://github.com/facebookresearch/Mask2Former
Self-supervised FID | https://github.com/stanis-morozov/self-supervised-gan-eval
(a)  Links to the assets used in the paper.

Name | License
---|---
ImageNet | Terms of access: https://www.image-net.org/download.php
COCO-Stuff | https://www.flickr.com/creativecommons
Cityscapes | https://www.cityscapes-dataset.com/license
ADE-20K | https://groups.csail.mit.edu/vision/datasets/ADE20K/terms/
Detectron2 | Apache-2.0 license
R50 | BSD
ConvNext | MIT License
Swin | MIT License
EfficientNet | Apache-2.0 license
VGG19 | BSD-3-Clause license
UperNet101 | BSD-3-Clause license
MS DRN-D-105 | BSD-3-Clause license
Deeplab-v2 | MIT License
Mask2Former | MIT License
(b) Assets licensing information.

Table S11: We provide information about the assets used in our work. Top :
links to the assets used. Bottom : Licensing information of the relevant
assets.