# Boosting Image Outpainting with Semantic Layout Prediction

$\text{Ye Ma}^{*}$1 $\text{Jin Ma}^{*}$2 Min Zhou1 Quan Chen1 Tiezheng Ge1
Yuning Jiang1 $\text{Tong Lin}^{\dagger}$2
1Alibaba Group 2Peking University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The objective of image outpainting is to extend image current border and
generate new regions based on known ones. Previous methods adopt generative
adversarial networks (GANs) to synthesize realistic images. However, the lack
of explicit semantic representation leads to blurry and abnormal image pixels
when the outpainting areas are complex and with various objects. In this work,
we decompose the outpainting task into two stages. Firstly, we train a GAN to
extend regions in semantic segmentation domain instead of image domain.
Secondly, another GAN model is trained to synthesize real images based on the
extended semantic layouts. The first model focuses on low frequent context
such as sizes, classes and other semantic cues while the second model focuses
on high frequent context like color and texture. By this design, our approach
can handle semantic clues more easily and hence works better in complex
scenarios. We evaluate our framework on various datasets and make quantitative
and qualitative analysis. Experiments demonstrate that our method generates
reasonable extended semantic layouts and images, outperforming state-of-the-
art models.

††∗ Equal contribution.††† Corresponding author.

##  1 Introduction

Figure 1: Image outpainting and its applications. It takes input images (left)
and generates extended ones (middle). And then designers could create banners
(right) based on extended images. Outpainting offers designers the potential
to create images of any sizes as they wish.

Image outpainting, also known as image extension, illustrated in Fig.1, aims
to extend image boundaries by generating new visual harmonious contents
according to the original images. It could be widely used in applications,
such as virtual reality, automatic creative image and video generation, to
create more impressive experiences for consumers.

It is obvious that image outpainting is one of the conditional image synthesis
tasks [23], whose condition is the original image content. One similar task
which has been usually mentioned with image outpainting is image inpainting
[3, 2, 26, 28, 38, 39, 43]. Though both of them generate unknown regions under
the conditions of known regions, outpainting is usually much more difficult.
Because outpainting could only utilize one-side boundary conditions from the
original images, whereas inpainting could utilize visual context surrounding
the inpainting area in all directions. Hence, many state-of-the-art image
inpainting models have been demonstrated poor performance for image
outpainting [32].

To solve the image outpainting problem, traditional methods [19, 33, 41]
utilize sophisticated handcrafted features and patch-based matching algorithms
to fill in the unknown regions. For the lack of deep semantic features, these
methods perform poorly on image extension of complex structure. To overcome
these drawbacks, Wang  _et al_. [36], Teterwak  _et al_. [32] and Yang  _et
al_. [37] are the first to design effective one-stage deep models with
adversarial training strategies [10] to improve outpainting quality
impressively. It demonstrates that high-level semantic representations from
deep neural networks play a crucial role in image outpainting.

However, these methods simply model and utilize high-level semantics
implicitly learned from CNN architectures from the perspective of general
conditional image synthesis. It is far from taking full advantage of the
important guiding role of high-level semantics in image outpainting. In this
paper, we argue that the explicit prediction of reasonable semantic layout is
beneficial to image outpainting in addition to the original image content.

Inspired by the recent prosperity of image segmentation [5, 27, 40, 31] and
GAN-based semantic image synthesis [21, 25, 30], we propose a two-stage
framework boosting image outpainting with reasonable semantic layout
prediction. Specifically, we decompose the image outpainting task into two
generative stages in semantic layout domain and in image domain respectively.
In the first stage, we parse the original known image region into semantic
layout with the aid of state-of-the-art segmentation networks. Then we
redefine the objective as generating the reasonable layout in unknown regions
under the condition of original image content and semantic layout in known
regions. In the second stage, we exploit the extended full semantic layout as
auxiliary conditions with image content in multi-scale to synthesize the final
realistic image.

Compared with preceding deep generative methods [36, 37, 32], our two-stage
generative framework has the following advantages:

Firstly, the divide-and-conquer design makes outpainting much easier. The
first stage only takes responsibility for the full semantic layout prediction.
It is much easier than extending the real image from blank when no texture,
color, light or other high frequency information need to be considered. Hence
the model could focus mainly on low frequency context instead of high-level
representations such as classes, sizes and positions. Benefiting from the
semantic layout of the first stage, the second stage model focuses more on the
synthesis of high frequency textures and structures, which makes the learning
phase much more stable and the result much more reasonable for complex
scenarios.

Secondly, the explicit semantic layout prediction result could provide a
handle to improve the quality and diversity of image outpainting results,
which is important for interactive image editing applications.

To prove the effectiveness of our proposed method, we conduct experiments on
challenging datasets including the ADE20K [42] and Cityscapes [6], which
contain real world images of complex scenes. Extensive experiments demonstrate
that our method could generate images of significantly higher quality in
complex scenarios compared to state-of-the-art outpainting models under both
quantitative and qualitative assessments. Our contributions are as follows:

(1) We propose a novel two-stage approach to the task of image outpainting.
The first stage aims at extending semantic segmentation of known regions to
unknown regions, while the second one aims at conditional image synthesis
based on the extended segmentation map.

(2) We introduce adversarial losses for both stages and design an encoder-
decoder framework to integrate image and semantic segmentation smoothly and
subtly.

(3) We apply image outpainting in complex and challenging scenarios,
containing objects and backgrounds of various categories. Extensive
experiments show our method outperforms state-of-the-art ones by a large
margin.

##  2 Related Work

Generative adversarial networks In recent years, generative adversarial
networks (GANs) [10] are successfully used in many computer vision tasks, such
as image synthesis [4, 25], image super-resolution [20], image denoising [11]
and 3D reconstruction [35]. Among these applications, many researchers focus
on adversarial loss (WGAN-GP [12], hinge loss [22]) and normalization
structures (SPADE [25]), which can greatly improve training stability and
image quality, achieving state-of-the-art for many image generation tasks.

Figure 2: Overview of our two-stage framework. The first generator
$\mathbf{G_{seg}}$ exploits masked image $I_{masked}$ and segmentation map
$S_{masked}$ to generate extended one $S_{out}$, while the second generator
$\mathbf{G_{img}}$ utilizes $S_{out}$ of multi-scale and generates full image
$I_{out}$ from masked one $I_{masked}$. The adversarial losses for two stages
are on the right.

Image inpainting Previous works [1, 3, 9, 8, 2, 19] of image inpainting adopt
classical computer vision algorithms, which usually search patch candidates in
a patch pool for the best similarity and “paste” patches to inpainting
regions. Various algorithms for constructing patch pool, searching for best
candidates and pasting patches consistently are proposed [3, 2]. They work
decently in simple cases, especially when inpainting areas follow a repeatable
pattern. Recently, many researchers adopt end-to-end deep learning models [15,
26, 28, 38, 39, 43]. Among them, conditional GANs prove to be effective when
handling complex inpainting areas. Yu _et al_. [38] designs a coarse-to-fine
framework to generate high-quality inpainting results with contextual
attention, which extracts feature of known surrounding areas to help the
generation of unknown areas. Iizuka _et al_. [15] employs global and local
context discriminators to distinguish real images from synthesis ones to make
results both locally and globally consistent.

However, most of the inpainting approaches become unstable and inconsistent
when there are large-scale inpainting areas, which occurs usually for
outpainting tasks. Inpainting mainly focuses on using information of known
region, while outpainting mainly focuses on generating information of unknown
area.

Image outpainting Similar to image inpainting, early image outpainting models
adopt searching and pasting methods [19, 33, 41]. With the recent development
of GANs, a few researchers utilize GANs into image outpainting tasks. Wang _et
al_. [36] propose a feature expansion and context prediction network for image
extrapolation. Yang _et al_. [37] adopt LSTM for spatial content prediction
and evaluate their encoder-decoder pipeline on a collected scenery dataset.
Similarly, Teterwak _et al_. [32] propose an encoder-decoder model and add
pre-trained feature conditions to discriminator for better image quality.
These methods use GANs for image outpainting and work decently on scenery
photos. However, the lack of semantic segmentation leads to bad performance in
complex scenes with objects of various categories.

Semantic image synthesis Semantic image synthesis [21, 25, 30] aims at
converting segmentation masks to real image retaining the semantic
information, which is a special form of conditional image synthesis. Park _et
al_. [25] propose a new spatial-adaptive denormalization (SPADE) module, and
integrate it to the generator, achieving convincing performance on various
complex datasets. Hao _et al_. [30] consider learning the scene generation in
a local context, and correspondingly design a local class-specific generative
network with semantic maps as a guidance, also achieving amazing results.

##  3 Methodology

###  3.1 Overview

Formally, given a ground truth input image ${I_{orig}\in\mathbb{R}^{h\times
w\times c}}$ and a binary mask ${M\in\\{0,1\\}^{h\times w}}$ indicating
outpainting region, the model only takes the masked image
${I_{masked}=I_{orig}\odot M}$ as input and generate extended image
${I_{out}\in\mathbb{R}^{h\times w\times c}}$. As shown in Figure 2, we
decompose the outpainting task into two stages: semantic segmentation
outpainting and semantic image synthesis. During the first stage, the
segmentation map $S_{masked}$ and $I_{masked}$ are fed into the semantic
segmentation outpainting model to obtain fully extended segmentation map
$S_{out}$. In the second stage, semantic image synthesis model focuses on
extending texture and color information, taking $S_{out}$ and $I_{masked}$ as
input to generate the full extended image $I_{out}$. Following the
conventional GAN framework, these two stages both consist of generator and
discriminator, and two stages are trained separately.

As for the segmentation map, note that in NO phase do we exploit the ground
truth segmentation. We adopt a pretrained semantic segmentation model to
generate segmentation. Specifically, given a pretrained model
$\mathbf{SegNet}$:

$\displaystyle S_{masked}$
$\displaystyle=\mathbf{Padding}(\mathbf{SegNet}(I_{cropped}))$ (1)
$\displaystyle S_{orig}$ $\displaystyle=\mathbf{SegNet}(I_{orig}),$ (2)

where $I_{cropped}$ refers to the cropped image before outpainting,
$\mathbf{Padding}$ refers to right padding zeros and $S_{orig}$ refers to the
segmentation map of the whole image.

###  3.2 Semantic Segmentation Outpainting

This part of the task can be treated as a standard outpainting task in
semantic domain. The semantic segmentation map contains the complete semantic
and layout information of the image, while irrelevant texture and color
information is omitted.

We take the mask $M$, masked image $I_{masked}$ and its corresponding
segmentation map $S_{masked}$ as inputs to our generator. The generator
$\mathbf{G_{seg}}$ follows common image-to-image translation encoder-decoder
structure. The encoder is a fully convolutional network with $11$ conv layers
($5$ of them are downsample conv layers) and the decoder consists of $5$
ResBlocks [13]. Both are applied with spectral normalization [24].
$I_{masked}$, $S_{masked}$, $M$ are concatenated as inputs and are encoded
through the CNN encoder. Then the latent vector is forwarded to the decoder,
which generates a extended segmentation map $S_{gen}$:

$\displaystyle S_{gen}=\mathbf{G_{seg}}(I_{masked},S_{masked},M)$ (3)
$\displaystyle S_{out}=(1-M)\odot S_{gen}+S_{masked}.$ (4)

The discriminator $\mathbf{D_{seg}}$ is used to discriminate $S_{out}$ from
$S_{orig}$. Additionally, we also feed the mask $M$ and masked image
$I_{masked}$ into $\mathbf{D_{seg}}$ to help it in the adversarial game. We
adopt the multi-scale discriminator proposed in [34] to improve the quality of
image generated and avoid repetitive structures.

To train the model well, a cross-entropy loss and an adversarial loss are
introduced. The cross-entropy loss $\mathcal{L}_{ce}$ is applied between
$S_{gen}$ and $S_{orig}$ to ensure the consistency between the known region
and the extended region. For adversarial loss we use hinge loss [22] to
stabilize training:

$\displaystyle\mathcal{L}_{seg,D}$
$\displaystyle=\mathbb{E}[\max(0,1-\mathbf{D_{seg}}(S_{orig},I_{masked}))+$
$\displaystyle\max(0,1+\mathbf{D_{seg}}(S_{out},I_{masked}))]$ (5)
$\displaystyle\mathcal{L}_{seg,G}$
$\displaystyle=\mathbb{E}[-\mathbf{D_{seg}}(S_{out},I_{masked}))].$ (6)

The two loss functions are combined with $\lambda_{ce}$:

$\displaystyle\mathcal{L}_{seg}=\lambda_{ce}\mathcal{L}_{ce}(S_{gen},S_{orig})+\mathcal{L}_{adv}^{seg}.$
(7)

After this stage, full segmentation map $S_{out}$ containing semantic and
layout clues has been generated.

###  3.3 Semantic Image Synthesis

In this stage, masked image $I_{masked}$, full segmentation map $S_{out}$ and
binary mask $M$ are used to generate extended image $I_{out}$, which can be
treated as a conditional semantic image synthesis task: given the segmentation
map $S_{out}$, synthesize real image consistent with the masked image
$I_{masked}$. The biggest difference from traditional semantic image synthesis
tasks is that the synthesised image is required to be consistent with the
masked image.

We find that spatially-adaptive de-normalization (SPADE) [25] shows a powerful
ability when involving semantics to guide image outpainting. $I_{masked}$,
$S_{out}$ and $M$ are concatenated as the input of the generator
$\mathbf{G_{img}}$, which is similar to $\mathbf{G_{seg}}$, but equipped with
SPADE modules in each block of its decoder. Specifically, the SPADE module
exploits $S_{out}$ and $M$ as semantics to guide the image synthesis.
Formally,

$\displaystyle I_{gen}$ $\displaystyle=\mathbf{G_{img}}(I_{masked},S_{out},M)$
(8) $\displaystyle I_{out}$ $\displaystyle=(1-M)\odot I_{gen}+I_{masked}.$ (9)

To train model $\mathbf{G_{img}}$ well, there are two main issues. On the one
hand, the new generated part is supposed to be meaningful. On the other hand,
the part near boundary ought to be natural, smooth and undistinguishable. We
use L$1$-distance $\mathcal{L}_{L1}$ and perceptual loss [7, 17]
$\mathcal{L}_{perc}$ between $I_{gen}$ and $I_{orig}$ to ensure the semantic
consistency and style consistency at the boundary:

$\displaystyle\mathcal{L}_{perc}$
$\displaystyle=\sum_{i=1}^{N}\lambda_{perc}^{i}||\phi_{i}(I_{gen})-\phi_{i}(I_{orig})||_{1}$
(10)

where $N=5$, $\lambda_{prc}^{i}=1/2^{i}$ and $\phi_{i}$ is the feature after
the $i$-th ReLU layer of pre-trained VGG-19 [29] network. And we use
adversarial hinge loss to make $\mathbf{G_{img}}$ generating realistic
results:

$\displaystyle\mathcal{L}_{img,D}$
$\displaystyle=\mathbb{E}[\max(0,1-\mathbf{D_{img}}(I_{orig},S_{orig}))+$
$\displaystyle\max(0,1+\mathbf{D_{img}}(I_{out},S_{orig}))]$ (11)
$\displaystyle\mathcal{L}_{img,G}$
$\displaystyle=\mathbb{E}[-\mathbf{D_{img}}(S_{orig},I_{masked}))].$ (12)

We adopt the same multi-scale discriminator in semantic segmentation
outpainting stage for $\mathbf{D_{img}}$. The final loss function
$\mathcal{L}_{img}$ is a combination of the above:

$\displaystyle\mathcal{L}_{img}=\lambda_{perc}\mathcal{L}_{perc}+\lambda_{L1}\mathcal{L}_{L1}+\mathcal{L}_{adv}^{img}.$
(13)

##  4 Experiments

###  4.1 Datasets

To evaluate our method, we perform experiments on two complex datasets:

ADE20K [42] contains 20,210 training images and 2,000 validation images
annotated with 150 semantic categories included. This dataset covers a vast
variety of complex scenes and objects, both indoor and outdoor, which is much
more challenging than scenery datasets used before. Due to the difficulty of
extending multiple complex objects in intricate scenes, no image outpainting
work has been done on this dataset, to the best of our knowledge.

Cityscapes [6] contains 2,975 training images and 500 validation images
recorded in street scenes from 50 different cities in Germany. The images are
finely annotated with 34 semantic categories. This dataset is easier than
ADE20K, and some previous work [36] has performed experiments on this dataset.

We adopt settings similar to [32]: aiming to fill in masked out image content
where the rightmost 25% and 50% of pixels in the image are respectively
masked. Specifically, for ADE20K [42] dataset, the input image to be masked is
$256\times 256$. For Cityscapes [6] dataset, we crop the input 1:2 street-view
image from middle-line to two 1:1 images, and flip the left one horizontally,
which together compose the training set. During inference, we concatenate two
extended images back to one 1:2 images (flipping the left image) to get the
final extended image. This setting is reasonable because street-view images
are symmetric and collected from the perspective of driving position.

During training, we use two common data augmentation methods for both
datasets: random crop and random flip. Input images are resized to $286\times
286$ first and randomly cropped to $256\times 256$. We flip images
horizontally with a probability of 0.5. Our models are trained on training set
and all reported results are evaluated on validation set, following the
official train/val split.

###  4.2 Implementation Details

We implement two alternatives of our method for ablation study. The first
baseline is a one-stage and end-to-end model which shares similar structures
as $\mathbf{G_{seg}}$ without the input of segmentation maps (NoSeg). The
second one is similar to the first one while concatenating masked segmentation
map with masked image as input (SegConcat).

We use pre-trained ResNeSt-200 [40] and HRNet-OCR [31] for $\mathbf{SegNet}$.
In training phase of image synthesis stage, we use $S_{orig}$ instead of
$S_{out}$ since there is no extended RGB image to match $S_{out}$. We use
$I_{orig}$ and $S_{orig}$ pairs to conduct supervised learning and train
$\mathbf{G_{img}}$ to learn the mapping relationship between semantic
segmentation maps and RGB pixels.We set $\lambda_{ce}=100$,
$\lambda_{perc}=10$ and $\lambda_{L1}=100$ as trade-off parameters. We perform
$300$ epochs of training for each model and dataset. Leaky-ReLU is used as the
activation function both in generators and discriminators. Learning rates for
generators and discriminators are initially set to $0.0001$ and $0.0004$
respectively, which decay linearly to $0$ after $200$ epochs. In addition, we
apply spectral norm [24], synchronized BatchNorm [16] and use Adam optimizer
[18] with $\beta_{1}=0,\beta_{2}=0.9$ to minimize the loss functions 7 and 13.
All models are implemented with Pytorch.

For comparison, we train state-of-the-art image outpainting models to compare
with ours: Boundless [32] and NS-Outpainting [37]. We re-implement Boundless
with PyTorch as there are no official training codes. For NS-Outpainting, we
adopt their official implements and modify the data loader and input/output
ratio of RCT module to adapt to our task settings. All experiments are
conducted on NVIDIA 32GB V100 GPUs.

Method | FID (25%) | FID (50%)
---|---|---
Boundless [32] | 12.82 | 38.51
NS-Outpainting [37] | 14.42 | 42.29
NoSeg | 11.69 | 34.95
SegConcat | 10.11 | 35.18
Ours | 8.26 | 27.67
(a) Results on ADE20K dataset.

Method | FID (25%) | FID (50%)
---|---|---
Boundless [32] | 22.44 | 41.08
NS-Outpainting [37] | 20.04 | 43.36
NoSeg | 20.84 | 45.11
SegConcat | 19.81 | 47.71
Ours | 15.28 | 33.80
(b) Results on Cityscapes dataset.

Table 1: FID results on ADE20K and Cityscapes validation sets. FID is computed
on the whole validation set. We compare Boundless [32], NS-Outpainting [37]
and our methods. Some possible alternatives are compared for ablation study.

Figure 3: Qualitative results for proposed models on ADE20K dataset [42] of
25% and 50% masks. From left to right: input image $I_{masked}$, input image’s
segmentation map $S_{masked}$, extended segmentation map $S_{out}$, Boundless
[32], NS-Outpainting [37], our model and ground truth. Please zoom in the
images for better view.

Figure 4: Qualitative results for proposed models on Cityscapes [6] of 25% and
50% masks. From top to bottom: input image $I_{masked}$, input image’s
segmentation map $S_{masked}$, extended segmentation map $S_{out}$, Boundless
[32], NS-Outpainting [37], our model and ground truth. Please zoom in the
images for better view.

###  4.3 Comparison with Previous Works

Datasets | Boundless | NS-Outpaint | Ours
---|---|---|---
ADE20K (25%) | 4.17% | 7.92% | 87.92%
ADE20K (50%) | 5.42% | 2.50% | 92.08%
Cityscapes (25%) | 14.17% | 19.17% | 66.67%
Cityscapes (50%) | 15.42% | 6.25% | 78.33%
Table 2: User study statistics. Each represents the percentage of cases where
that method is the best, under the ratio setting.

Quantitative Results In order to quantitatively measure the quality of
generated images, one universally acknowledged metric is Fréchet Inception
Distance (FID) [14], which measures the distance in feature space between
generated and real images.

We compute FID scores on the whole validation sets of ADE20K [42] and
Cityscapes [6], using two mask ratio (25% and 50%) settings. The results of
two datasets are shown in Table 1. We can observe that our method
significantly (more than 20%) outperforms other methods on both datasets and
both ratio settings, which proves the effectiveness of our framework.

Considering the limitation of FID, we also conduct a user study to evaluate
human perception preference. We randomly select 40 images (20 from ADE20K and
20 from Cityscapes) and invite 24 participants who are not involved in this
project to the user study. Each participant is asked to pick the best image of
the three (Boundless, NS-Outpainting and ours). Images are shuffled in a blind
random order to avoid guessing sequences of the three. Specific statistics are
shown in Table 2, which indicates consistent preference of our method over
other baseline methods.

Qualitative Results Figure 3 and Figure 4 give qualitative comparison of
various outpainting methods on ADE20K and Cityscapes dataset respectively. We
can see that in complex scenarios, especially ADE20K, our method generates
clearer and more reasonable extended images compared to others. Other methods
usually degenerate when objects stand on the boundary to extend, while ours
works well for exploiting semantic information. Specifically, Boundless [32]
and NS-Outpainting [37] both tend to extend boundary pixels horizontally. This
seems helpful for natural scenery images but usually leads to fake artifacts
for images containing complex objects.

Summary Both qualitative and quantitative results show that our method
outperforms others by a large margin, especially in complex scenes. Our
extended semantic segmentation maps are geometrically reasonable and
symmetric, consistent with the distribution of real images. In addition, there
is a strong correlation between the extended image and the extended
segmentation map, which coincides with our motivation of decomposing the
outpainting task, proving the effectiveness of our two-stage method.

Figure 5: Ablation results. Our two-stage method produces higher quality
images with sharper boundaries and more consistent semantics.

###  4.4 Ablation Studies and Discussion

We conduct ablation studies on the effectiveness of semantic information and
the necessariness of our two-stage strategy.

As mentioned in Section 4.2, we implement two weaker models (NoSeg, SegConcat)
to prove the necessity of our proposed method for outpainting. We also report
their quantitative results in Table 1. Not surprisingly, ours gets the best
FID value. SegConcat performs better than NoSeg at the mask of 25% ratio, but
is even slightly worse than NoSeg at a larger ratio 50%. This is due to the
limited guidance provided by partial semantic segmentation maps, which only
helps on small-ratio outpainting.

We also give a qualitative comparison of the three methods in Figure 5. It
shows that the NoSeg method can hardly control the edge of objects well,
generating fuzzy and low-quality images. Especially when there are objects at
the boundary, it is impossible to achieve semantic consistency without any
guidance. The other two methods both leverage semantic segmentation
information for image outpainting. For SegConcat model, it involves masked
semantic segmentation map as part of input, relying on model itself to extend
semantic information implicitly. But as for ours, we explicitly exploit the
extended semantic segmentation map and generate images under the guidance of
it. Obviously, our method produces compelling extended semantic layouts and
semantically consistent extended images.

Our experiments prove that involving semantic segmentation map facilitates the
outpainting task a lot and the decomposition of the task is effective.

Figure 6: Editable outpainting: designers could conveniently edit extended
semantic layout as they wish based on $S_{out}$, generating diverse convincing
and reasonable outpainting images $I^{\prime}_{out}$.

###  4.5 Applications and Limitations

Image outpainting could be widely used in applications such as automatic image
editing (Figure 1). In addition, since we decompose the image outpainting task
to two stages and exploit semantic layout explicitly, this naturally gives us
the chance to manually adjust outpainting results, which means we could
conveniently modify extended semantic layout to generate diverse outpainting
images, as shown in Figure 6. This capacity is beneficial for interactive
image editing tools but could not be easily adopted by previous methods.

Although our method performs obviously better than related work in complex
scene datasets, it still suffers from some structural objects such as humans.
This may be caused by the uneven distribution of the objects in complex scenes
and our dataset is not so big ($20$k) compared to others ($1$M), leading to
poor knowledge of structure and texture information.

##  5 Conclusion

In this paper, we propose a two-stage image outpainting framework to generate
high-quality extended images. The main idea is to disassemble the outpainting
task into two parts: semantic segmentation outpainting and semantic image
synthesis. The former focuses on semantic and structural consistency at the
boundary while the latter ensures realistic texture, color and lighting. Our
method behaves significantly better when handling complex scenarios,
outperforming state-of-the-art methods by a large margin.

## References

  * [1] Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera.  Filling-in by joint interpolation of vector fields and gray levels.  IEEE transactions on image processing, 10(8):1200–1211, 2001.
  * [2] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Transactions on Graphics (ToG), 28(3), July 2009.
  * [3] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester.  Image inpainting.  In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’00, page 417–424, USA, 2000\. ACM Press/Addison-Wesley Publishing Co.
  * [4] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In International Conference on Learning Representations, 2019.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [6] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [7] Alexey Dosovitskiy and Thomas Brox.  Generating images with perceptual similarity metrics based on deep networks.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [8] Alexei A. Efros and William T. Freeman.  Image quilting for texture synthesis and transfer.  In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’01, page 341–346, New York, NY, USA, 2001. Association for Computing Machinery.
  * [9] Alexei A Efros and Thomas K Leung.  Texture synthesis by non-parametric sampling.  In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1033–1038. IEEE, 1999.
  * [10] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  arXiv preprint arXiv:1406.2661, 2014.
  * [11] Jinjin Gu, Yujun Shen, and Bolei Zhou.  Image processing using multi-code gan prior.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [12] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [15] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.
  * [16] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [17] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
  * [18] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [19] Johannes Kopf, Wolf Kienzle, Steven Drucker, and Sing Bing Kang.  Quality prediction for image completion.  ACM Transactions on Graphics (ToG), 31(6), Nov. 2012.
  * [20] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [21] Yandong Li, Yu Cheng, Zhe Gan, Licheng Yu, Liqiang Wang, and Jingjing Liu.  Bachgan: High-resolution image synthesis from salient object layout.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [22] Jae Hyun Lim and Jong Chul Ye.  Geometric gan.  arXiv preprint arXiv:1705.02894, 2017.
  * [23] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [24] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  arXiv preprint arXiv:1802.05957, 2018.
  * [25] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [26] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [27] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [28] Jimmy SJ Ren, Li Xu, Qiong Yan, and Wenxiu Sun.  Shepard convolutional neural networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [29] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  Computer Science, 2014.
  * [30] Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe.  Local class-specific and global image-level generative adversarial networks for semantic-guided scene generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7870–7879, 2020.
  * [31] Andrew Tao, Karan Sapra, and Bryan Catanzaro.  Hierarchical multi-scale attention for semantic segmentation.  arXiv preprint arXiv:2005.10821, 2020.
  * [32] Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T. Freeman.  Boundless: Generative adversarial networks for image extension.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [33] Miao Wang, Yu-Kun Lai, Yuan Liang, Ralph R. Martin, and Shi-Min Hu.  Biggerpicture: Data-driven image extrapolation using graph matching.  ACM Transactions on Graphics (ToG), 33(6), Nov. 2014.
  * [34] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [35] Weiyue Wang, Qiangui Huang, Suya You, Chao Yang, and Ulrich Neumann.  Shape inpainting using 3d generative adversarial network and recurrent convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [36] Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia.  Wide-context semantic image extrapolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [37] Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan.  Very long natural scenery image prediction by outpainting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [38] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang.  Generative image inpainting with contextual attention.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [39] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang.  Free-form image inpainting with gated convolution.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [40] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al.  Resnest: Split-attention networks.  arXiv preprint arXiv:2004.08955, 2020.
  * [41] Yinda Zhang, Jianxiong Xiao, James Hays, and Ping Tan.  Framebreak: Dramatic image extrapolation by guided shift-maps.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
  * [42] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [43] Tong Zhou, Changxing Ding, Shaowen Lin, Xinchao Wang, and Dacheng Tao.  Learning oracle attention for high-fidelity face completion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

##  Appendix A Additional Implementation Details

We give a detailed description of our network structure. Table 3 shows the
multi-scale discriminator we use which is proposed in [34]. $\mathbf{G_{img}}$
and $\mathbf{G_{seg}}$ both follow an encoder-decoder structure.
$\mathbf{G_{img}}$ employs the same encoder with $\mathbf{G_{seg}}$, while
equipped with SPADE modules in each block of its decoder. The common encoder
structure and the decoder structures are shown in Table 4, Table 5 and Table 6
respectively.

Type | Kernel Size | Stride | # Channels
---|---|---|---
Conv. | $4\times 4$ | $2$ | $64$
Conv. | $4\times 4$ | $2$ | $128$
Conv. | $4\times 4$ | $2$ | $256$
Conv. | $4\times 4$ | $1$ | $512$
Conv. | $4\times 4$ | $1$ | $1$
Table 3: Architecture details of discriminators. We apply synchronized BatchNorm [16] and Spectral Norm [24] to the convolutional layers except the first and last layers. We use LeakyReLU as activation function for all convolutional layers. Type | Kernel Size | Stride | # Channels
---|---|---|---
Conv. | $7\times 7$ | $1$ | $64$
Conv. | $3\times 3$ | 1 | $128$
Conv. | $3\times 3$ | 2 | $128$
Conv. | $3\times 3$ | 1 | $256$
Conv. | $3\times 3$ | 2 | $256$
Conv. | $3\times 3$ | 1 | $512$
Conv. | $3\times 3$ | 2 | $512$
Conv. | $3\times 3$ | 1 | $1024$
Conv. | $3\times 3$ | 2 | $1024$
Conv. | $3\times 3$ | 1 | $1024$
Conv. | $3\times 3$ | 2 | $1024$
Table 4: Architecture details of encoder. We apply SyncBatchNorm [16], LeakyReLU and Spectral Norm [24] to all the convolutional layers. Type | Kernel Size | Stride | # Channels
---|---|---|---
ResnetBlock | $3\times 3$ | $1$ | $1024$
Upsample | None | None | $1024$
ResnetBlock | $3\times 3$ | 1 | $1024$
ResnetBlock | $3\times 3$ | 1 | $1024$
Upsample | None | None | $1024$
ResnetBlock | $3\times 3$ | 1 | $512$
Upsample | None | None | $512$
ResnetBlock | $3\times 3$ | 1 | $256$
Upsample | None | None | $256$
ResnetBlock | $3\times 3$ | 1 | $128$
Upsample | None | None | $128$
ResnetBlock | $3\times 3$ | 1 | $64$
Conv. | $3\times 3$ | 1 | out_nc
Table 5: Architecture details of decoder in $\mathbf{G_{seg}}$. For upsample we use the “nearest” mode and set scale_factor to 2. We don’t apply any activation function after last convolutional layer and the decoder generates a semantic segmentation map Type | Kernel Size | Stride | # Channels
---|---|---|---
SPADEBlock | $3\times 3$ | $1$ | $1024$
Upsample | None | None | $1024$
SPADEBlock | $3\times 3$ | 1 | $1024$
SPADEBlock | $3\times 3$ | 1 | $1024$
Upsample | None | None | $1024$
SPADEBlock | $3\times 3$ | 1 | $512$
Upsample | None | None | $512$
SPADEBlock | $3\times 3$ | 1 | $256$
Upsample | None | None | $256$
SPADEBlock | $3\times 3$ | 1 | $128$
Upsample | None | None | $128$
SPADEBlock | $3\times 3$ | 1 | $64$
Conv. | $3\times 3$ | 1 | out_nc
Table 6: Architecture details of decoder in $\mathbf{G_{img}}$. For upsample
we use the “nearest” mode and set scale_factor to 2. We apply tanh after last
convolutional layer and the decoder generates a real image.

Figure 7: Additional qualitative results for proposed models on ADE20K dataset
[42] of 25% masks. From left to right: input image $I_{masked}$, input image’s
segmentation map $S_{masked}$, extended segmentation map $S_{out}$, Boundless
[32], NS-Outpainting [37], our model and ground truth. Please zoom in the
images for better view.

Figure 8: Additional qualitative results for proposed models on ADE20K dataset
[42] of 50% masks. From left to right: input image $I_{masked}$, input image’s
segmentation map $S_{masked}$, extended segmentation map $S_{out}$, Boundless
[32], NS-Outpainting [37], our model and ground truth. Please zoom in the
images for better view.

Figure 9: Additional qualitative results for proposed models on Cityscapes [6]
of 25% masks. From top to bottom: input image $I_{masked}$, input image’s
segmentation map $S_{masked}$, extended segmentation map $S_{out}$, Boundless
[32], NS-Outpainting [37], our model and ground truth. Please zoom in the
images for better view.

Figure 10: Additional qualitative results for proposed models on Cityscapes
[6] of 50% masks. From top to bottom: input image $I_{masked}$, input image’s
segmentation map $S_{masked}$, extended segmentation map $S_{out}$, Boundless
[32], NS-Outpainting [37], our model and ground truth. Please zoom in the
images for better view.

Figure 11: Some bad cases for proposed models on ADE20K dataset [42] of 25%
and 50% masks. From left to right: input image $I_{masked}$, input image’s
segmentation map $S_{masked}$, extended segmentation map $S_{out}$, Boundless
[32], NS-Outpainting [37], our model and ground truth. Please zoom in the
images for better view.

##  Appendix B Additional Qualitative Results

Figure 7, Figure 8 and Figure 9, Figure 10 give additional qualitative
comparisons of various outpainting methods under two mask ratio (25% and 50%)
settings on ADE20K [42] and Cityscapes [6] dataset respectively.

##  Appendix C Bad Cases

Our model still suffers from some structural objects such as vehicles and
humans. We show some bad cases in Figure 11 and this is what we are to address
in our future work.