input
---
$y^{\prime 1}$
---
SSS
---
$y^{\prime 2}$
---
$y^{\prime 3}$
---
$y^{\prime 3}$
---
$y^{\prime 3}$
---

Our system, SSS, takes input vector cartographic data (left), and generates
realistic and seamless high-resolution satellite images (right; the image
$y^{\prime 3}$ is 8,192 pixels wide, approximately $67$ megapixels) which are
continuous over space and scale. Given the input, SSS generates massive
spatially seamless images over a range of scales ($y^{\prime 1}$, $y^{\prime
2}$, and $y^{\prime 3}$) by blending the results of lower resolution networks.

# Seamless Satellite-image Synthesis

Jialin Zhu0000-0002-1826-6566 and Tom Kelly0000-0002-6575-3682
University of Leeds

###### Abstract

We introduce Seamless Satellite-image Synthesis (SSS), a novel neural
architecture to create scale-and-space continuous satellite textures from
cartographic data. While 2D map data is cheap and easily synthesized, accurate
satellite imagery is expensive and often unavailable or out of date. Our
approach generates seamless textures over arbitrarily large spatial extents
which are consistent through scale-space. To overcome tile size limitations in
image-to-image translation approaches, SSS learns to remove seams between
tiled images in a semantically meaningful manner. Scale-space continuity is
achieved by a hierarchy of networks conditioned on style and cartographic
data. Our qualitative and quantitative evaluations show that our system
improves over the state-of-the-art in several key areas. We show applications
to texturing procedurally generation maps and interactive satellite image
manipulation.

<ccs2012> <concept>
<concept_id>10010147.10010371.10010382.10010384</concept_id>
<concept_desc>Computing methodologies Texturing</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010371.10010382.10010383</concept_id>
<concept_desc>Computing methodologies Image processing</concept_desc>
<concept_significance>100</concept_significance> </concept> </ccs2012>

[500]Computing methodologies Texturing [100]Computing methodologies Image
processing

††volume: 40††issue: 7

##  1 Introduction

Satellite images have revolutionized the way we visualise our environment.
Massive, high-resolution, satellite images are essential to many modern
endeavors including realistic virtual environments and regional- or city-
planning. However, these images can be prohibitively expensive, out of date,
or of low quality. Further, continuous satellite images covering large areas
are often unavailable due to limited sensor resolution or orbital mechanics.
In contrast, cartographic (map) information is frequently accessible and
available over massive continuous scales; further, it can be easily manually
edited or synthesized. In this paper we address the problem of generating
satellite images from map data at arbitrarily large resolutions.

Convolutional neural networks are state of the art for image to image
translation; such networks can translate from cartographic images to satellite
images. However, these networks are unable to generate outputs of arbitrary
areas – the resolution of these synthetic satellite image tiles remains
limited by available memory and compute. Arranging many such small tiles to
cover a large map results in inconsistencies (seams) between the tiles.
Further, if a larger number of tiles are joined together and viewed at
multiple scales, the result lacks continuity and style coordination.

Inspired by convolutional image-to-image translation systems such as Pix2Pix
[IZZE17] and SPADE [PLWZ19], _Seamless Satellite-image Synthesis_ (SSS, Figure
Seamless Satellite-image Synthesis) creates satellite image tiles by
conditioning generation on cartographic data. These tiles are of fixed
resolution (256 $\times$ 256 pixels in our implementation) and exhibit good
continuity internally, but would contain spatial and scale artifacts if
naïvely tiled over a larger area. _Spatial discontinuities_ between adjacent
tiles are caused by  a lack of coordination between network evaluations,
resulting in lack of continuity for properties not determined by the
cartographic input (e.g., color of building roof or species of tree). We
introduce a network which removes such spatial discontinuities by leveraging
learned semantic domain knowledge as well as the cartographic data. _Scale
discontinuities_ between a larger set of synthetic satellite tiles arise from
unnatural style variations over larger areas (e.g., distribution of cars
throughout a city or different camera sensors). To address this, SSS uses a
hierarchy of networks synchronized by color guidance.

The SSS system contains three components – two convolutional neural networks
to generate tiles of different resolutions (_map2sat_), and remove seams
(_seam2cont_) between those tiles, as well as an interface allowing the
generated textures to be explored interactively. SSS contributes:

  * •

A neural approach to generate massive scale-and-space continuous images
conditioned on cartographic data.

  * •

A novel architecture to train and evaluate image-to-image generation networks
at high resolutions with reduced memory use.

  * •

An application to texture procedurally generated cartographic maps – allowing
the the synthesis of textures for entirely novel areas following user style
guidance.

  * •

An application to interactive exploration of endless satellite images
generated on-the-fly from cartographic data.

The textures generated by SSS have many applications, including interactive
city planning, virtual environments, and procedural modeling. During city
planning, the textures may help stakeholders (e.g., homeowners, planners, and
motorists) understand proposed developments. For example, sit-down sessions in
which stakeholders interactively edit a map and are shown birds-eye-view
(satellite) “photos” within seconds. Further, virtual environments may be
constructed using SSS’s output (e.g., a flight simulator) to texture massive
terrains on-the-fly, or the streets and open areas of urban procedural models
[PM01].

In the remainder of this work we will discuss the corpus surrounding our work
in Section 2, introduce our system in Section 3, give the details of the
neural architectures and their coordination in section 4 before evaluating our
results qualitatively and quantitatively in section 5. Source code and weights
for our system are available online at https://github.com/Misaliet/Seamless-
Satellite-image-Synthesis.

##  2 Related Work

vector map
---
image map
---
_map2sat_
---
+
---
_seam2cont_
---
tiled result
---
seamless
---
color-guidance
---
(when $z>1$)
satellite texture
---
Figure 1: SSS uses a hierarchy of networks for different scales. A pair of
networks is shown above, which process input cartographic maps to large
seamless images at a single scale level, $z$. First, a vector map of arbitrary
extent is processed to create an image tile of map data. The _map2sat_ network
then performs a translation to the satellite texture domain, before the
_seam2cont_ network removes visible tilling artifacts to create a seamless
image and the result is combined with neighboring tiles to create an image of
arbitrary resolution (right). Subsequent, smaller scale, _map2sat_ networks
use previous outputs as scale-guidance (bottom left).

Texture synthesis is the generation of texture images from exemplars or lower
dimension representations. We refer the reader to Wei et al. [WLKT09] for an
overview. Non-neural approaches from the AI Winter create images with local
and stationary properties using techniques such as Markov Random Fields to
generate pixels [EL99, WL00] or patches [LLX∗01, EF01] from an exemplar image.
Variations on these themes generate larger, more varied textures [DSB∗12].
Procedural languages for texturing [MZWVG07] are well adapted for urban
domains and can be used to guide synthesis [GAD∗20].

With the advent of Convolutional Neural Networks (CNNs), texture synthesis has
increasingly become data-driven, creating images by gradient descent in
feature space [GEB15] or with Generative Adversarial Networks (GANs) [ZZB∗18].
Another approach to texture generation is to transform an input image to a
different style or domain while retaining the structure. Such _image-to-image_
translation is implemented by the influential UNet architecture [RFB15] which
passes the input through an “hourglass” shaped network of convolutional
layers. This allows the network to aggregate and coordinate features within a
_bottleneck_. UNets have been applied to generate satellite images [IZZE17]
from map data, however the resolution of the output tiles was limited by
available memory. Image-to-image networks typically disentangle the structure
from style to enable style transfer [GEB16]. These have been applied to
generate terrains [GDG∗17] and within chained sequences [KGS∗18] for
architectural texture pipelines.

Label-to-image translation is a variation of image-to-image translation; the
input is a label-map with each pixel having a specific class, such as
cartographic data containing, streets, buildings, and vegetation. Again, we
note the application of hourglass networks [WLZ∗18]. We build on Semantic
Image Synthesis with Spatially-Adaptive Normalization (SPADE) [PLWZ19] which
uses a simpler V-shaped architecture with a bottleneck at the start, rather
than center, of the network. From a latent code, successive _SPADE Residual
Blocks_ generate feature-maps of increasing resolution. Within these blocks, a
SPADE unit introduces learned per-label style information using Spatially-
Adaptive Normalization.

Texture blending or inpainting approaches give a way to combine multiple
smaller images into a larger one. These may match small image patches [LLX∗01,
EF01, BSFG09], or minimize a function over a grid of pixels using seam energy
minimization [AS07, KSE∗03, RKB04], exemplars [CPT03], or a Poisson
formulation [PGB03]. Although such local approaches produce excellent pixel-
level results, they do not contain the higher, semantic-level knowledge to
blend textures and images with complex structures. We may overcome this
limitation with user input [ADA∗04], large datasets [HE07], and distributions
learned by neural architectures [ZZL19, ZWS20, BSP∗19]. These learn
distributions for the blended area using techniques such as guided Poisson
constraints [WZZH19] and symmetry [DCX∗18]. Domain-specific image blending has
been studied for domains such as microscopy [LCH∗15], façade texturing
[KGS∗18], and satellite generation. TileGAN [FAW19] searches a library of
latent encodings of adjacent micro-tiles to create seamless images, rather
than learning a domain specific blending function.

In contrast to relatively recent satellite images, applications of cartography
date back to antiquity. Today, there are high-quality, publicly available maps
of much of the world [Fou21, Ser21]. We may also generate maps with a variety
of techniques, including procedural systems; see Vanegas et al. [VAW∗10] for
an overview. In particular, we may use rule and parameter controlled geometric
algorithms to recreate the distribution of streets [CEW∗08], buildings [PM01],
and city parcels [VKW∗12]. These procedural systems can generate massive
quantities of 3D city geometry from random seeds; such geometry can be used to
generate cartographic maps by projecting the 3D meshes to 2D images and
applying a constant color for each category (e.g., red buildings).
Procedurally generated cartography has the advantage that it can be easily
edited with a user-interface or created from databases.

##  3 Overview

We propose a novel method for map-to-satellite-image translation over
arbitrarily large areas. Two neural networks work together to first generate
satellite image tiles from map data, and then remove seams and discontinuities
from these images. To generate an area, SSS (Figure 1) first generates two
overlapping sets of satellite image tiles with the _map2sat_ network. Then the
_seam2cont_ network removes the discontinuities and seams between adjacent
tiles by learning to generate a mask to blend the overlapping satellite
images. Joining together such tiles results in a single seamless image with a
large resolution and scale.

To achieve our goals, we solve problems such as the training of a mask
generator to remove seams while realistically blending tiles, the coordination
of networks over different areas to create continuous results in scale-space,
and reducing the memory usage for large image synthesis networks.

##  4 Method

$t$
---
$s$
---
_map2sat_
---
Figure 2: Creation of _seam2cont_ input tiles. We obtain one image with seams,
$s$ and one without seams, $t$, from the _map2sat_ network run over map data
(left). Note that the corners of $s$ are seamless against adjacent $s$ tiles
as they come from the same _map2sat_ execution.

SSS uses convolution neural networks (CNNs) to learn to synthesize satellite
image by learning over datasets. We prepare a dataset of paired map and
satellite images for each tile at a fixed resolution of 256 $\times$ 256
pixels. Vector cartographic data is processed to image tiles – each GeoPackage
[geo] file contains a list of polygons, each with one of 13 semantic class
labels (e.g., _building, road or track_ , or _natural environment_). The
polygons are ordered canonically and rendered into an image, assigning every a
pixel a label. The corresponding satellite images have 8-bit color depth, a
resolution 0.25 meters per pixel, and metadata containing their location and
extent in the local Coordinate Reference System; we use this transform to
align them to the map data.

The paired dataset is used to train our image-to-image translation network,
_map2sat_ , based on the SPADE[PLWZ19] architecture. The network learns to
translate our semantic cartographic label image tiles into corresponding
satellite image tiles. However, to synthesize massive areas two problems must
be solved.

The first problem is maintaining a realistic style over different scales of
tiled areas: _scale-space continuity_. If our tiles are too large, we are able
to synthesize a large area, but the limited fixed resolution provides poor
detail (see additional materials _variants_8k_images/large_scale_only.jpg_).
On the other hand, if the tiles are too small, we see unrealistic variation
over the tiled image – this may be too little variation, resulting in a
homogeneous image (e.g., very similar trees over an entire city); see
additional materials _variants_8k_images/small_scale_only.jpg_ – or too much
variation, leading to a chaotic swings in is variation between adjacent tiles
(e.g., roads suddenly changing color). Additionally, our interactive system
allows users to explore a dynamically generated satellite image at a range of
scales; in this application, it is necessary to generate imagery at a given
scale without overly onerous number of generator network applications. SSS
uses a scale-space hierarchy of generator networks to create realistic
variation over areas at multiple scales (Section 4.1). By evaluating the
hierarchy tiles lazily, the interactive system is able to quickly evaluate
tiles at arbitrary scales.

The second problem is that simple tiling of the synthetic satellite images to
cover a large area does not produce natural images, but creates an image with
seams; state-of-the-art CNN architectures do not provide _spatial-continuity_
between adjacent tiles. This is largely because of their reliance on spatially
narrow bottleneck layers, which allow tile-wide coordination of features at
the cost of limiting maximum tile size. Adding additional layers to increase
the tile size becomes prohibitively expensive due to the memory requirements
of these large layers. Another source of seams is the padding [AKM∗21] around
layer boundaries, where synthetic values (i.e., zero or flipped) are used to
pad convolution input; these introduce seam artifacts around the perimeter of
network outputs. We address this spatial-continuity problem in the following
section.

###  4.1 Spatial Continuity: _seam2cont_ Network

The limited resolution of CNN image synthesis techniques, such as those we use
in the _map2sat_ network, leads to seam discontinuities when we join multiple
tiles together. Here we introduce the _seam2cont_ network which removes the
seams, allowing tiling  over an arbitrarily large area. Given tiles generated
by our image-to-image network we introduce a novel architecture to blend
pixels from these existing tiles to create a seam-free image. We found
blending (masking) a sufficient approach to seam removal as the tile content
is well aligned by strong cartographic conditioning, and blending over a wide
area allows the network to identify semantic plausible locations for each
transition.

For the area of one tile $i$, we use two independent, overlapping layers of
_map2sat_ evaluation to synthesize tiles $s_{i}$ and $t_{i}$. The areas of
$s_{i}$ and $t_{i}$ are offset so for each tile $i$, there is a seam-free
option, as Figure 2. Our goal is to blend $s_{i}$ and $t_{i}$ such that the
result has no internal seams (as $t_{i}$), but also has no seams with
neighboring tiles (as $s_{i}$). The quarters of $s_{i}$ are continuous across
the tile boundary because they have come from the same output tile of a single
_map2sat_ evaluation. Thus, the problem of removing seams is transformed to a
masking (image blending) problem – where the mask predominantly selects pixels
from $t_{i}$ in the center and from $s_{i}$ in the periphery. In-between it
should select pixels which maximize realism. We propose a method to learn to
generate the mask, $m$, which selects how to blend $s_{i}$ and $t_{i}$. By
learning a mask generator, _seam2cont_ can learn the semantic meanings of the
inputs, producing higher quality results than optimizing for gradient [KSE∗03]
or search [FAW19].

For one tile area $i$, _seam2cont_ takes the map image $x_{i}$ (including
label and instance data), the generated satellite image with seams ($s_{i}$),
and the generated satellite image without seams ($t_{i}$). The output of the
_seam2cont_ generator is a mask, $m_{i}$, which is used to blend the input
satellite images. The output of the system is seamlessly tillable satellite
image, $y^{\prime}_{i}$, with a feature distribution similar to the seamless
ground-truth images.

$\mathcal{L}_{seams}$
---
$\mathcal{L}_{realism}$
---
$M$
---
$D_{real}$
---
$D_{seams}$
---
$x_{i}$
---
$t_{i}$
---
$s_{i}$
---
$c$
---
$m_{i}$
---
$m^{\prime}_{i}$
---
$y^{\prime}_{i}$
---
Figure 3: The data-flow when testing the _seam2cont_ network (black arrows).
The inputs are map label images $x_{i}$, as well as generated satellite images
$s_{i}$ and $t_{i}$. $M$ generates a mask, $m_{i}$ which is multiplied by a
fixed circle image $c$ to create $m^{\prime}_{i}$. This final mask blends
$s_{i}$ and $t_{i}$ to create our spatially continuous seamless image
$y^{\prime}_{i}$. Cyan arrows: the two discriminators contribute losses when
training $M$.

Figure 3 shows the architecture of the _seam2cont_ network. We use a
UNet[RFB15, ZZP∗17] as our generator $M$ to create a mask, $m_{i}$, from
inputs $x_{i}$, $s_{i}$ and $t_{i}$:

$m_{i}=M(x_{i},s_{i},t_{i})$ (1)

To evaluate the quality of the generated mask, an output tile,
$y^{\prime}_{i}$ is blended from $s_{i}$ and $t_{i}$ by $B$:

$y^{\prime}_{i}=B(s_{i},t_{i},m_{i})=s_{i}*m_{i}+t_{i}*(1-m_{i})$ (2)

$M$ is a 8 layer UNet which we train with three goals for the blended image
$y^{\prime}_{i}$: (1) tileable – with $s_{i}$ near the borders, (2) realistic,
and (3) internally seamless. We construct a differentiable loss function to
achieve these three goals which can be backpropogated to improve $M$ during
training.

To address goal (1) the mask should select $s_{i}$ at the edge of the tile to
ensure continuity with adjacent tiles. We implement this by forcing the
outside part of generated mask $m_{i}$ to be white (to have a value of $1$) in
$F$, guided by a constant circular mask $c$ (Figure 3):

$m^{\prime}_{i}=F(m_{i})=m_{i}*(1-c)+c$ (3)

and we can modify equation 2 to give us the masked output:

$y^{\prime}_{i}=B(s_{i},t_{i},m^{\prime}_{i})=s_{i}*m^{\prime}_{i}+t_{i}*(1-m^{\prime}_{i})$
(4)

To support goal (2) we use a discriminator $D_{realism}$, a PatchGAN [IZZE17],
to reward $M$ for the realism of the masked output. $D_{realism}$ is trained
to discriminate between generated images and those from the ground-truth
(without seams). When training $M$, it takes results $y^{\prime}_{i}$ and the
constant circular mask $c$. These conditions allow the discriminator to locate
the output seams [TSK∗19] and increase accuracy. The loss $D_{realism}$ is
then:

$\mathcal{L}_{realism}=\mathbb{E}_{x_{i},s_{i},t_{i}}[-D_{realism}(y^{\prime}_{i},c)]$
(5)

Following the standard GAN architecture, this loss using $D_{realism}$ is used
during training to update $M$ to increase realism.

Goal (3) is to remove internal seams from $y^{\prime}_{i}$. To achieve this,
we train an image-to-image CNN to locate seams and use it as a discriminator,
$D_{seams}$. It encourages $M$ to generate a mask $m_{i}$ which removes seams
from the blended result. It is an image-to-image translation network which
learns to identify seams in patches of $y^{\prime}_{i}$, outputting a
grayscale image that is white (value of 1) where there is no seam and black
(value of 0) where there is a seam. $D_{seams}$ is used as an adversarial loss
when training $M$ by constructing a L2 loss between the discriminator’s output
and the desired (seamless; value of 1) result. Following CUT [PAAEZ20], we
extract random patches, $P$, from the output of mask to contribute to the
loss:

$\mathcal{L}_{seams}=||D_{seams}(P(y^{\prime}_{i}))-1||$ (6)

Finally, as Pix2Pix[IZZE17], we add a L1 loss to the ground-truth in order to
improve the training stability:

$\mathcal{L}_{recons}=|y^{\prime}_{i}-y_{i}|$ (7)

The total loss function that is propagated through $M$ during training is
then:

$\mathcal{L}_{M}=\lambda_{1}\mathcal{L}_{realism}+\lambda_{2}\mathcal{L}_{seams}+\lambda_{3}\mathcal{L}_{recons}$
(8)

We train the model with Adam [KB15] with a learning rate of $0.0002$. We may
write the complete _seam2cont_ network, $S$, which generates creating seamless
image, $y^{\prime}_{i}$, as:

$S(x_{i},s_{i},t_{i})=B(s_{i},t_{i},F(M(x_{i},s_{i},t_{i})))=y^{\prime}_{i}$
(9)

We train the neural networks $M$, $D_{realism}$, and $D_{seams}$ together,
performing one SGD update for each network at every iteration. $D_{realism}$
is trained to discriminate between blended result images ($y^{\prime}_{i}$)
and image without seams ($t_{i}$):

$\mathcal{L}_{d-realism}=\mathbb{E}_{x_{i},s_{i},t_{i}}[D_{realism}(s_{i},c)+(1-D_{realism}(y^{\prime}_{i},c))]$
(10)

$D_{seams}$ is trained to identify seam locations in images. We find that the
simplicity of this image based seams/no-seams formulation complements
$D_{realism}$, which works on more complex image space features. We use random
patches with $D_{seams}$ as they are smaller and faster to train. Random
patches also have non-constant output (with seams in different locations) as
they are unaffected by boundary layer padding. We train $D_{seams}$ to
discriminate between the output $n$ and a seams mask image showing the
position of seams, $l$, with the loss:

$\mathcal{L}_{d-seams}=||D_{seams}(P(y^{\prime}_{i}))-P(l)||$ (11)

where $P$ is a particular random crop of resolution 64 $\times$ 64 pixels.

The complete loss function of _seam2cont_ network that is backpropagated
through $M$ when training is:

$\displaystyle\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{realism}+\lambda_{2}\mathcal{L}_{seams}+\lambda_{3}\mathcal{L}_{recons}$
(12)
$\displaystyle+\lambda_{4}\mathcal{L}_{d-realism}+\lambda_{5}\mathcal{L}_{d-seams}$

The training data for the whole _seam2cont_ network contains seams images and
non-seams images, both of them are generated by _map2sat_ at different scales.
We train _seam2cont_ network with a dataset of around 3,000 images on a Nvidia
Titan V GPU for 200 epochs using hyperparameters $\lambda_{1}=1.0$,
$\lambda_{2}=1.0$, $\lambda_{3}=10.0$, $\lambda_{4}=1.0$, and
$\lambda_{5}=1.0$.

The primary resolution of all networks in SSS is 256 $\times$ 256 pixels.
Larger resolutions are possible, but would require significantly increased
memory. This limited resolution allows us to study impact of seam-removal with
reasonable compute resources.

###  4.2 Scale-Space Continuity: _map2sat_ Network

$e$
---
$\emph{map2sat}^{1}$
---
$\emph{map2sat}^{2}$
---
$\emph{map2sat}^{z_{max}}$
---
…
---
Figure 4: The _map2sat_ sub-networks can be trained independently or end-to-
end. This figure illustrates the organization of SPADE residual blocks
(magenta, Figure 5). $\emph{map2sat}^{1}$ is a SPADE network taking latent
input $e$. The output of $\emph{map2sat}^{1}$ is cropped (purple) to provide
color guidance to the $\emph{map2sat}^{2}$ sub-network. $\emph{map2sat}^{2}$
and later sub-networks have the same architecture. conv
---
conv
---
conv
---
batch
---
norm
SPADE
---
conv
---
conv
---
SPADE
---
relu
---
relu
---
Figure 5: Top (green): Extending the original SPADE block [PLWZ19], we
concatenate the RGB color guidance from the previous map2sat scale and the map
image. The concatenated tensor is used to learn convolutions to manipulate the
channels after batch-normalisation. Bottom: SPADE residual blocks are combined
with ReLU and convolutional layers to create the magenta blocks shown in
Figure 4.

The _seam2cont_ network allows us to build large composite images from
generated tiles. To generate these tiles, we build on SPADE [PLWZ19] to create
realistic individual tiles from map label images. These tiles can be assembled
into an arbitrarily large image. However, there is no coordination between
tiles which are far apart. To address this scale-space problem, we introduce
the _map2sat_ network to generate synthetic image tiles with scale-space
continuity.

To provide consistency at different scales, we would ideally use a single
large tile, however such high resolution is prohibitively computationally
expensive. Instead, _map2sat_ uses multiple sub-networks to generate tiles at
different _scales_ , retaining a fixed _resolution_ of 256 $\times$ 256
pixels. For example, at the largest scale, $z=1$, map image tiles are 1
kilometer wide (3.9 meters per pixel), while at a smaller scale, $z=2$, a map
tile is four times larger: 250 meters wide (0.98 meters per pixel). Using
multiple scales allows large scale tiles to provide scale-space
synchronization and small scale tiles to provide high detail ; however, they
must be consistent.

In the case of our interactive exploration system, we can achieve the
necessary speed by limiting the number of tile evaluations to balance speed
against scale or detail. For example, we do not need to generate small scale
tiles if the user is viewing a large area. However, small and large scales
must be consistent – the same areas must have the same content and style. To
achieve this scale-space continuity, each _map2sat_ sub-network is conditioned
on a _guidance_ subsection of the larger scale tile.

The input to _map2sat_ at each scale is a map label image and an optional
3-channel color guidance image which is only applied when scale level $z>1$.
The color guidance image is from a larger scale in our hierarchy satellite
image that provides color conditioning to ensure consistency between scales.
The output of each $\emph{map2sat}^{z}$ sub-network is a RGB satellite image
consistent with both the map images and the larger-scale color guidance (if
provided). The map image, color guidance, and output satellite image have the
same resolution.

The principle issue when adapting a CNN generator architecture to an arbitrary
resolution is avoiding the memory requirements of large layers. Using the
_seam2cont_ network we can refactor the network into sub-networks,
$\emph{map2sat}^{z}$, for each scale $z\in 1...z^{max}$ (see Figure 4). By
cropping each sub-network’s input, the memory and data requirements are
reduced considerably. With this cropped architecture we are able to train the
whole network end-to-end when required, but each sub-network remains
independent when evaluated. This sub-network independence allows portions of
scale-space to be evaluated to accelerate our interactive editor.

To implement this refactoring, a spatial crop from 256 $\times$ 256 pixels to
64 $\times$ 64 pixels is inserted between each adjacent sub-network (see
Figure 4); this provides a $4\times$ decrease in scale per sub-network. A 3
$\times$ 3 convolution layer to 512 channels is positioned at the start of
each sub-network to preserve the original SPADE network architecture as much
as possible – we found this arrangement did not impact generation quality.

The inputs to each sub-network are map label images with optional color
guidance images from the previous (larger) scale for sub-networks $z>1$. The
color guidance images coordinate style and content consistency between sub-
networks at scales $\emph{map2sat}^{n}$ and $\emph{map2sat}^{(n+1)}$.
Experimentally, we observe that the SPADE network takes the majority of its
guidance from the label image which overpowers the spatial convolutional input
guidance. To compensate, we append additional color guidance to the semantic
label channel in the spatially-adaptive denormalization blocks (Figure 5). We
found that this approach better preserves color guidance information. The
color guidance image is blurred with a Gaussian filter ($\sigma=8$ at 256
$\times$ 256 pixels) to only keep the color information and reduce other high
frequency features. The spatial input to the largest scale
$\emph{map2sat}^{1}$ at training time is from an encoder that is trained to
create a latent encoding, $e$, from a satellite image; at test time we use a
normally distributed $e$ to create variety over our results.

The complete _map2sat_ network is a streamlined version of a high-resolution
SPADE with considerably lower memory usage. For instance, the on-disk weights
of a full SPADE network trained at 256 $\times$ 256 pixels are 436.9MB  while
the weights for a cropped sub-network, $\emph{map2sat}^{z>1}$, are 31.5MB. For
a 8,192 $\times$ 8,192 pixels image the full SPADE GPU memory requirement
exceeds 32GB, while the factored weights for the two networks in the SSS
system are less that 12GB.

Real data is used to train each of the _map2sat_ sub-networks independently.
End-to-end fine-tuning over the whole network results distributions were
sufficiently close to realistic data that at test-time these networks would
accept synthetic inputs from the previous _map2sat_ when $z>1$. The training
data of sub-network $\emph{map2sat}^{n}$ is semantic label map images and
color guidance images generated by scale level $\emph{map2sat}^{n-1}$. We use
around 3,000 images to train the sub-networks. We implement our pipeline with
$z^{max}$ of 3 and train every $\emph{map2sat}^{z}$ network for 70 epochs.

###  4.3 Complete SSS Pipeline

Our entire system is a combination of the _seam2cont_ and _map2sat_ networks
(Figure 1) over the scale levels. The architecture is shown in Figure 6 and
the tile geometry in Figure 7. At the $z^{1}$ scale level, we use
$\emph{map2sat}^{1}$ with random latent encoding $e$, but without color
guidance. The satellite images tiles $s^{1}_{i}$ and $t^{1}_{i}$ are generated
and processed by the $\emph{seam2cont}^{1}$ network for spatial continuity,
resulting in $y^{\prime 1}_{i}$. The set of seamless tiles, $y^{\prime
1}_{*}$, are stitched together to create the final image at scale $z^{1}$; we
crop the this result to use as the color guidance of $z^{2}$. This process
continues over all scales $z\in 1...z^{max}$ to generate satellite images of
different scales over an area.

$\emph{map2sat}^{1}$
---
$\emph{map2sat}^{2}$
---
$\emph{map2sat}^{3}$
---
$\emph{seam2cont}^{1}$
---
$\emph{seam2cont}^{2}$
---
$\emph{seam2cont}^{3}$
---
$e$
---
Figure 6: SSS combines a series of _map2sat_ and _seam2cont_ networks. Here we
show a 3-scale ($z^{max}=3$) pipeline. _map2sat_ sub-networks generate
satellite image tiles which are processed by the _seam2cont_ networks to
remove spatial discontinuities. $s^{n}$
---
$s^{n+1}$
---
$y^{\prime n+1}$
---
$x^{n}$
---
$x^{n+1}$
---
$t^{n}$
---
$y^{\prime n}$
---
$t^{n+1}$
---
blur
---
blur
---
Figure 7: The geometry of SSS tiles. Left, right columns: network outputs at
scales $z=n$ and $z=n+1$. Scale-space continuity is provided by passing a
blurred crop of $y^{n}$ (green bottom left) to $\emph{map2sat}^{n+1}$ (cyan
mid-right).

Because our method generates tiles and stitches them together to create huge
images, we can generate satellite images of arbitrary size from cartographic
map data. Furthermore, we can enforce scale continuity over a wide range of
scales. Another advantage of our SSS pipeline is that the two independent
network architectures and independent networks at each scale can be used for
custom applications; for example, the $z^{2}$ _seam2cont_ network may be used
to remove seams in real-world satellite images, or we may insert additional
sub-modules to create satellite images with smaller scales.

The general complexity of the entire SSS pipeline depends on the scale level,
$z=k$, and the linear scale factor, $f$, used between each sub-network. For
example, a scale factor of $f=4$ between each sub-network implies that the
smaller scale sub-network has $f^{2}=16$ tiles covering the area of a single
larger scale tile. To generate one scale, all larger scales must also be
generated. The worst-case number of _map2sat_ evaluations, assuming we
generate $4$ tiles at $z=k$ is:

$\sum_{i=1}^{k}4/(f^{2})^{i-1}$ (13)

and worst-case number of _seam2cont_ evaluations is:

$\sum_{i=1}^{k}(\sqrt{4/(f^{2})^{i-1}}-1)^{2}$ (14)

##  5 Results

In the following results we use SSS with $z^{max}=3$ and a scale factor of
$f=4$ to create a 512$\times$512 pixels image at $z^{1}$, a 2,048$\times$2,048
pixels image at $z^{2}$, and a 8,192$\times$8,192 pixels image at $z^{3}$.

Figure 8: Results from the SSS pipeline. Rows: examples of generated satellite
images from real-world map data. Columns: left to right show scales $z=1,2,3$.
Inset are the corresponding map inputs.  Figure 9: Result of our system on
synthetic procedurally generated map data. Figure 10: Style generation of
$z^{1}$ (left) and $z^{3}$ (right) images in a 3 scale level SSS system. Each
row has a different latent encoding, $e$, input to $\emph{map2sat}^{1}$,
giving different styles which remain consistent throughout the scale levels.

Figure 8 shows the output of our technique for real-world map guidance images;
further results are given in our additional materials (see folder
_sss_8k_images/_). Figure 9 shows the output of our technique for synthetic
map labels generated using a procedural modeling system. Figure 10 illustrates
the style continuity of SSS pipeline.

Figure 11: User interface for our interactive system. In this figure, we show
a result of generated satellite image (right) by corresponding map (left).
Here the user has edited the map by rotating the central tile. Our system is
able to generate realistic results even with the discontinuities in the map
data.

Finally, our interactive system is introduced in figure 11. We allow users to
move around in the map, zoom in or out, and generate corresponding satellite
images of different styles. We also provide some operations to modify the map,
such as randomly replacing a single tile in the map input. All networks
(_map2sat_ and _seam2cont_) for 3 scales can be loaded into the 12GB memory of
a Nvidia Titan V graphics card. Generation times for a 512 $\times$ 512 pixels
window vary between 2.4 and 3.42 seconds depending on the scale. Smaller
scales require additional network evaluations.

##  6 Evaluation

###  6.1 Methodology

Evaluating the quality of composite synthesized images for neural networks is
a difficult problem; there is no canonical method to assess the performance of
image stitching. Traditional edge detection technology, such as Sobel filter
detection and Canny edge detection, can be used for seams detection but are
not optimized for tile-based systems where the seam locations can be
anticipated. In addition, these algorithms output images with edges rather
than comparable values.

To evaluate our _seam2cont_ network results, we use two methods. First, we
design a qualitative user study survey. We ask users to identify the best
system for removal of seams between tiles to examine the effectiveness of our
_seam2cont_ network against other algorithms. Second, we introduce a novel
algorithm, MoT (Mean over Tiles), to measure the ability to remove seams in
images. We evaluate against the following baselines: traditional image
blending with a soft mask, Graphcut[KSE∗03], Generative Image
Inpainting[YLY∗19], and TileGAN[FAW19].

(b)
---
(a)
---
(d)
---
(c)
---
(e)
---
(f)
---
Figure 12: This figure illustrates results from selected baseline techniques.
(a) $t$ from our _map2sat_ network; (b) the output of our _seam2cont_ network;
(b, inset) the calculated mask, $m$; (c) the result of fixed-mask (c, inset)
image blending. (d) the Graphcut algorithm calculates four minimum energy
seams within margins (d, inset); (e) Generative Image Inpainting is not
conditioned on map data, but is used to inpaint a diamond area (e, inset); (f)
TileGAN conditioned on a lower-resolution satellite image.

For traditional image blending, we use $s$ as input and blend $s$ with $t$ by
a soft mask shown in the right bottom corner of Figure 12 (c) to obtain an
image without seams. Graphcut removes seams by splicing the center part of $t$
into $s$ selecting a cut through a buffer area. We use a small area, 10
$\times$ 10 pixels, shown in the right bottom corner of Figure 12 (d) as our
buffer area. We train the Generative Image Inpainting model, which uses gated
convolutions[YLY∗19] for the image inpainting task; it is trained with our
generated seams dataset $s$ and the diamond shape mask shown in the right
bottom corner of Figure 12 (e). This is to let the model learn to generate the
center diamond area which covers the seams. Generative Image Inpainting does
synthesize reasonable results, but it often changes the original content
structure in $s$ as it is not conditioned on map data. Last, we train PGGAN
[KALL17] with ground-truth satellite images of 256$\times$256 pixels
resolution for TileGAN. TileGAN can generate large seamless images by
searching the latent space of the pre-trained PGGAN model to create tiles that
are stitched into a large image. It is not strictly an image-to-image
translation system, but can take a small, low-resolution, image as guidance to
generate large, high-resolution, images.

Figure 12 shows selected representative output tiles for each of the
baselines. Traditional image blending in Figure 12 (c) does a good job
generally, but since it uses a fixed mask, the rough shape of the mask can
sometimes be observed. This is more obvious when we apply image blending on
green fields. Figure 12 (d) is created by the Graphcut algorithm, which
clearly shows visible seams. Figure 12 (e) is generated by Generative Image
Inpainting; it does synthesize good results, however, it changes the original
structure of Figure 12 (a); traces of the diamond mask can also be observed.

Figure 12 (f) is a tile from a large image synthesized by TileGAN; an 8,320
$\times$ 8,320 pixel image is guided by a low-resolution image which is
cropped into the corresponding tile. TileGAN creates an overly “blobby” result
with poor large scale features. While TileGAN is true to the guidance image,
the guidance is too low resolution to effectively guide the generation of
crisp man-made features such as streets. When TileGAN is trained on satellite
images of urban areas, the results can be poor because of the difficulty in
searching for a latent encoding which matches large, sharp features over tile
boundaries. This is illustrated in the additional materials file
_baselines_8k_images/tilegan.jpg_. We provide large area results (8k images)
of all baselines in _baselines_8k_images/_.

We also evaluate against the following variants and later use these to form an
ablation study: photographic ground-truth, ground-truth with seams (created by
flipping each tile along the diagonal), original SPADE[IZZE17] with no
_seam2cont_ networks, our _map2sat_ results without _seam2cont_ network, and
our results which are generated by procedural map data.

0%
---
20%
---
40%
---
60%
---
80%
---
_seam2cont_
---
Graphcut
---
$t$
---
100%
---
GII
---
Image blending
---
_seam2cont_
---
vs.
Graphcut
_seam2cont_
---
vs.
GII
_seam2cont_
---
vs.
Image
blending
_seam2cont_
---
vs.
$t$
Figure 13: User study results. We show results of SSS (our approach),
Graphcut, Generative Image Inpainting (GII) and $t$ (_map2sat_ output, without
_seam2cont_). In all scenarios, users rate our seam-removal as more natural
than alternatives.

The User study survey evaluates user’s perceived quality of seam removal. We
designed a user questionnaire which shows an image with seams, and ask which
of two techniques remove the seams in the most natural way. We compare against
4 baselines: Graphcut, Generative Image Inpainting, image blending with a soft
mask, and the dataset $t$ without seams that we use for training. Comparison
trials for one baseline are performed over 10 pairs of images. We do not
include TileGAN in the survey because it does not support generating satellite
imagery of a specific area. We asked a total 20 participants “Which of these
two images below removes the seams in the most natural way?” (see
_misc/user_study.png_ in additional materials). We also collected information
on participant’s digital image editing experience. Among our participants,
55$\%$ process digital images every month, while the remaining 45$\%$ have no
relevant experience.

Figure 13 shows the results of our user study. SSS removes seams in a more
natural way than all the baselines in the majority of trials. Our advantage is
particularly strong against Graphcut and Generative Image Inpainting. We
believe this is because Graphcut cannot complete the stitching task well over
a limited buffer area, and Generative Image Inpainting removes seams while
changing the original content of the image. We found it surprising that when
our results are compared with $t$, we still have an advantage. This may be
because the tile generation near the border ($t$) may be less accurate, so
showing more images from the center of adjacent tiles ($y^{\prime}$) may
create more realistic results.

128
---
ground-truth with seams
---
ground-truth
---
_map2sat_ without _seam2cont_
---
whole pipeline
---
Pixel position
---
0
---
256
---
difference
---
0.005176
---
0.000006
---
Figure 14: Selected MoT algorithm results: difference value response for
ground-truth, ground-truth with seams, our _map2sat_ with no _seam2cont_ and
our whole pipeline. Images with strong vertical and horizontal seams have a
strong response on the tile boundary at 128 pixels.

The MoT algorithm compares differences between horizontally and vertically
adjacent pixels over generated images to highlight any sudden color
differences at known tile boundaries.

To compute the MoT response we firstly crop the input image by removing
($tileWidth/2$) pixels from each edge. Let $C_{a}$ be all pixels in columns
$\\{0+a,tileWidth+a,tileWidth*2+a$,…,$imageWidth-tileWidth+a\\}$ from a
cropped image. Similarly $R_{a}$ for all rows. The MoT response at pixel
position, $x,y$, is the absolute difference of the averages of differences of
adjacent pixels:

$D(x,y)=|\overline{C_{x}}-\overline{C_{x+1}}|+|\overline{R_{y}}-\overline{R_{y+1}}|$
(15)

We use $tileWidth=256$ to match our generated tile size, $x,y\in
0...tileWidth-1$, and a 8,192 $\times$ 8,192 pixel image to calculate the MoT
values.

Figure 14 shows the results of the MoT algorithm. The horizontal axis shows
discontinuities at each pixel. The _map2sat_ results show an obvious response
peak at tile boundaries, while after the _seam2cont_ network our results do
not have this peak. This illustrates that our network has learned to remove
the seams over the image. This algorithm also works for other baselines and
variants; table 1 shows the absolute difference between of $max(D(x,y))$ and
$mean(D(x,y))$ over $x,y$ for the baselines and variants. When the difference
value is greater than 0.002 a sharp vertical or horizontal seam is present.

Model, Baseline, variant | difference value
---|---
Image blending with a soft mask | 0.000341
Graphcut | 0.000344
Generative Image Inpainting | 0.001911
TileGAN | 0.001336
Ground-truth | 0.000291
Ground-truth with seams | 0.003214
Original SPADE without _seam2cont_ | 0.002723
The _map2sat_ without _seam2cont_ | 0.004724
The SSS whole pipeline | 0.000730
The SSS whole pipeline with procedural map data | 0.000845
Table 1: The difference between max and mean values from our MoT algorithm. We
found that images with difference values greater than 0.002 contained obvious
seams.

###  6.2 Ablation Study

We use variants’ results from the previous section and some new variants to
present a visual ablation study. First, we create an image of 512$\times$512
pixels resolution at scale $z^{1}$ and upscale it with simple Lanczos
interpolation to 8,192 $\times$ 8,192 pixels. Such an approach loses its
detail, appearing very soft (see additional materials,
_variants_8k_images/large_scale_only.jpg_). Second, we create an 8k image only
at $z^{3}$, without color guidance, and note that this image looks very
homogeneous. For example, most natural environmental areas are covered with
the same color and often have similar patterns (see
_variants_8k_images/small_scale_only.jpg_). Third, we generate an 8k image
with _map2sat_ , but no _seam2cont_ network (see
_variants_8k_images/no_sat2tile_network.jpg_). This gives us an image with
seams, but because this image is generated using color guidance, the style of
each tile also shows scale-space consistency; it is also better than a large
image stitched together by the results of SPADE alone (see
_baselines_8k_images/seams.jpg_).

##  7 Limitations

There are several limitations of our SSS pipeline. First, our _map2sat_ is
based on SPADE[PLWZ19] and Pix2Pix[IZZE17] which means that the quality of the
generated satellite images is limited by these architectures. We also tried
Progressive Growing of GANs when we implementing TileGAN for comparison, but
there is still a gap between the generated satellite image and the ground-
truth satellite image. Another limitation is that we only remove the linear
seams between adjacent tiles. The _seam2cont_ network also leaves theoretical
singular point discontinuities at the midpoints of each tile edge after
blending between $s$ and $t$. We found this had minimal impact on the quality
of seam removal in practice, but can envisage an additional network which
removes such singular discontinuities. Finally, we produce strong results
because SSS is conditioned on cartographic data, but compare to some baselines
which can not use this data fully (e.g., TileGAN), or were not designed for
this task (e.g., Graphcut).

##  8 Conclusion

We have proposed the SSS pipeline to synthesize large satellite images that
are continuous both  spatially and throughout scale-space at arbitrary size
for given cartographic map data while not requiring large memory use. It also
supports procedurally generated cartographic maps for entirely novel areas
which allow users to create textures on demand. This is achieved by the
_map2sat_ network architecture that can keep style continuity in specified
scale-space and _seam2cont_ network architecture to splice small images
seamlessly into one large image. We also implement a interactive system to let
users explore of an boundless satellite images generated conditioned on
cartographic data.

In future work, we would like to we would like to study the impact of
different edge constraint images ($c$) in our _seam2cont_ network by replacing
$c$ with blob-shaped masks and using other conditioning for _map2sat_ , such
as multi-spectral images, pencil sketches, and low resolution images. Another
avenue of exploration is the scale factor of 4 between $z$; we would like to
try different values with a perceptual evaluation. In addition, we suspect
that using different offsets at different scales would reduce regularity and
improve the quality of results. We also want to try to modify the neural
network structure used; for example, adding adaptive discriminator
augmentation[KAH∗20]. Finally, we want to extend our work to 3D voxel based
networks using transformers to synthesize a 3D virtual environment from
cartographic map data.

##  9 Acknowledgments

We would like to thank Nvidia Corporation for hardware and Ordnance Survey
Mapping[Ser21, Mas21] for map data which made this project possible. This work
was undertaken on ARC4, part of the High Performance Computing facilities at
the University of Leeds, UK. This work made use of the facilities of the N8
Centre of Excellence in Computationally Intensive Research (N8 CIR) provided
and funded by the N8 research partnership and EPSRC (Grant No. EP/T022167/1).

## References

  * [ADA∗04] Agarwala A., Dontcheva M., Agrawala M., Drucker S., Colburn A., Curless B., Salesin D., Cohen M.:  Interactive digital photomontage.  In _ACM SIGGRAPH_. 2004, pp. 294–302.
  * [AKM∗21] Alsallakh B., Kokhlikyan N., Miglani V., Yuan J., Reblitz-Richardson O.:  Mind the pad – {cnn}s can develop blind spots.  In _ICLR_ (2021).  URL: https://openreview.net/forum?id=m1CD7tPubNy.
  * [AS07] Avidan S., Shamir A.:  Seam carving for content-aware image resizing.  In _ACM SIGGRAPH_ (New York, NY, USA, 2007), Association for Computing Machinery, p. 10–es.  URL: https://doi.org/10.1145/1275808.1276390, doi:10.1145/1275808.1276390.
  * [BSFG09] Barnes C., Shechtman E., Finkelstein A., Goldman D. B.:  PatchMatch: A randomized correspondence algorithm for structural image editing.  _ACM SIGGRAPH 28_ , 3 (Aug. 2009).
  * [BSP∗19] Bau D., Strobelt H., Peebles W., Wulff J., Zhou B., Zhu J.-Y., Torralba A.:  Semantic photo manipulation with a generative image prior.  _ACM TOG 38_ , 4 (July 2019).  URL: https://doi.org/10.1145/3306346.3323023, doi:10.1145/3306346.3323023.
  * [CEW∗08] Chen G., Esch G., Wonka P., Müller P., Zhang E.:  Interactive procedural street modeling.  In _ACM SIGGRAPH_. 2008, pp. 1–10.
  * [CPT03] Criminisi A., Perez P., Toyama K.:  Object removal by exemplar-based inpainting.  In _IEEE CVPR_ (2003), vol. 2, IEEE, pp. II–II.
  * [DCX∗18] Deng J., Cheng S., Xue N., Zhou Y., Zafeiriou S.:  Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition.  In _IEEE ICCV_ (2018), pp. 7093–7102.
  * [DSB∗12] Darabi S., Shechtman E., Barnes C., Goldman D. B., Sen P.:  Image melding: combining inconsistent images using patch-based synthesis.  _ACM SIGGRAPH 31_ , 4 (2012), 1–10.
  * [EF01] Efros A. A., Freeman W. T.:  Image quilting for texture synthesis and transfer.  In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ (2001), pp. 341–346.
  * [EL99] Efros A. A., Leung T. K.:  Texture synthesis by non-parametric sampling.  In _IEEE ICCV_ (1999), vol. 2, IEEE, pp. 1033–1038.
  * [FAW19] Frühstück A., Alhashim I., Wonka P.:  Tilegan: synthesis of large-scale non-homogeneous textures.  _ACM SIGGRAPH 38_ , 4 (2019), 1–11.
  * [Fou21] Foundation O.:  OpenStreetMap, 2021.  URL: https://www.openstreetmap.org.
  * [GAD∗20] Guehl P., Allegre R., Dischler J.-M., Benes B., Galin E.:  Semi-procedural textures using point process texture basis functions.  In _CGF_ (2020), vol. 39, Wiley Online Library, pp. 159–171.
  * [GDG∗17] Guérin É., Digne J., Galin É., Peytavie A., Wolf C., Benes B., Martinez B.:  Interactive example-based terrain authoring with conditional generative adversarial networks.  _ACM TOG 36_ , 6 (2017), 1–13.
  * [GEB15] Gatys L., Ecker A. S., Bethge M.:  Texture synthesis using convolutional neural networks.  In _NIPS_ (2015), pp. 262–270.
  * [GEB16] Gatys L. A., Ecker A. S., Bethge M.:  Image style transfer using convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ (2016), pp. 2414–2423.
  * [geo] Geopackage.  https://www.geopackage.org/.  Accessed: 2010-09-30.
  * [HE07] Hays J., Efros A. A.:  Scene completion using millions of photographs.  _ACM SIGGRAPH 26_ , 3 (July 2007), 4–es.  URL: https://doi.org/10.1145/1276377.1276382, doi:10.1145/1276377.1276382.
  * [IZZE17] Isola P., Zhu J.-Y., Zhou T., Efros A. A.:  Image-to-image translation with conditional adversarial networks.  _IEEE CVPR_ (2017).
  * [KAH∗20] Karras T., Aittala M., Hellsten J., Laine S., Lehtinen J., Aila T.:  Training generative adversarial networks with limited data.  _arXiv preprint arXiv:2006.06676_ (2020).
  * [KALL17] Karras T., Aila T., Laine S., Lehtinen J.:  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ (2017).
  * [KB15] Kingma D. P., Ba J.:  Adam: A method for stochastic optimization.  In _ICLR_ (2015), Bengio Y., LeCun Y., (Eds.).  URL: http://arxiv.org/abs/1412.6980.
  * [KGS∗18] Kelly T., Guerrero P., Steed A., Wonka P., Mitra N. J.:  Frankengan: guided detail synthesis for building mass models using style-synchonized gans.  _ACM TOG 37_ , 6 (2018), 1–14.
  * [KSE∗03] Kwatra V., Schödl A., Essa I., Turk G., Bobick A.:  Graphcut textures: image and video synthesis using graph cuts.  _ACM SIGGRAPH 22_ , 3 (2003), 277–286.
  * [LCH∗15] Legesse F., Chernavskaia O., Heuke S., Bocklitz T., Meyer T., Popp J., Heintzmann R.:  Seamless stitching of tile scan microscope images.  _Journal of microscopy 258_ , 3 (2015), 223–232.
  * [LLX∗01] Liang L., Liu C., Xu Y.-Q., Guo B., Shum H.-Y.:  Real-time texture synthesis by patch-based sampling.  _ACM Trans. Graph. 20_ , 3 (July 2001), 127–150.  URL: https://doi.org/10.1145/501786.501787, doi:10.1145/501786.501787.
  * [Mas21] Mastermap O. S.:  Ordnance, 2021.  URL: https://www.ordnancesurvey.co.uk/business-government/products/mastermap-topography.
  * [MZWVG07] Müller P., Zeng G., Wonka P., Van Gool L.:  Image-based procedural modeling of facades.  _ACM SIGGRAPH 26_ , 3 (2007), 85.
  * [PAAEZ20] Park T., Alexei A. Efros R. Z., Zhu J.-Y.:  Contrastive learning for unpaired image-to-image translation.  _ECCV_ (2020).
  * [PGB03] Pérez P., Gangnet M., Blake A.:  Poisson image editing.  In _ACM SIGGRAPH 2003 Papers_. 2003, pp. 313–318.
  * [PLWZ19] Park T., Liu M.-Y., Wang T.-C., Zhu J.-Y.:  Semantic image synthesis with spatially-adaptive normalization.  In _IEEE CVPR_ (2019), pp. 2337–2346.
  * [PM01] Parish Y. I. H., Müller P.:  Procedural modeling of cities.  In _Proceedings of SIGGRAPH 2001_ (2001), pp. 301–308.  doi:10.1145/383259.383292.
  * [RFB15] Ronneberger O., Fischer P., Brox T.:  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ (2015), Springer, pp. 234–241.
  * [RKB04] Rother C., Kolmogorov V., Blake A.:  " grabcut" interactive foreground extraction using iterated graph cuts.  _ACM SIGGRAPH 23_ , 3 (2004), 309–314.
  * [Ser21] Service D. O. S. W. M.:  Digimap, 2021.  URL: https://digimap.edina.ac.uk/.
  * [TSK∗19] Teterwak P., Sarna A., Krishnan D., Maschinot A., Belanger D., Liu C., Freeman W. T.:  Boundless: Generative adversarial networks for image extension.  In _IEEE ICCV_ (2019), pp. 10521–10530.
  * [VAW∗10] Vanegas C. A., Aliaga D. G., Wonka P., Müller P., Waddell P., Watson B.:  Modelling the appearance and behaviour of urban spaces.  In _CGF_ (2010), vol. 29, Wiley Online Library, pp. 25–42.
  * [VKW∗12] Vanegas C. A., Kelly T., Weber B., Halatsch J., Aliaga D. G., Müller P.:  Procedural generation of parcels in urban modeling.  In _CGF Eurographics_ (2012), vol. 31, Wiley Online Library, pp. 681–690.
  * [WL00] Wei L.-Y., Levoy M.:  Fast texture synthesis using tree-structured vector quantization.  In _Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques_ (USA, 2000), SIGGRAPH ’00, ACM Press/Addison-Wesley Publishing Co., p. 479–488.  URL: https://doi.org/10.1145/344779.345009, doi:10.1145/344779.345009.
  * [WLKT09] Wie L.-Y., Lefebvre S., Kwatra V., Turk G.:  State of the Art in Example-based Texture Synthesis.  In _CGF Eurographics_ (2009), Pauly M., Greiner G., (Eds.), The Eurographics Association.  doi:10.2312/egst.20091063.
  * [WLZ∗18] Wang T.-C., Liu M.-Y., Zhu J.-Y., Tao A., Kautz J., Catanzaro B.:  High-resolution image synthesis and semantic manipulation with conditional gans.  In _IEEE CVPR_ (2018), pp. 8798–8807.
  * [WZZH19] Wu H., Zheng S., Zhang J., Huang K.:  Gp-gan: Towards realistic high-resolution image blending.  In _Proceedings of the 27th ACM International Conference on Multimedia_ (2019), pp. 2487–2495.
  * [YLY∗19] Yu J., Lin Z., Yang J., Shen X., Lu X., Huang T. S.:  Free-form image inpainting with gated convolution.  In _IEEE ICCV_ (2019), pp. 4471–4480.
  * [ZWS20] Zhang L., Wen T., Shi J.:  Deep image blending.  In _The IEEE Winter Conference on Applications of Computer Vision_ (2020), pp. 231–240.
  * [ZZB∗18] Zhou Y., Zhu Z., Bai X., Lischinski D., Cohen-Or D., Huang H.:  Non-stationary texture synthesis by adversarial expansion.  _ACM SIGGRAPH 37_ , 4 (July 2018).  URL: https://doi.org/10.1145/3197517.3201285, doi:10.1145/3197517.3201285.
  * [ZZL19] Zhan F., Zhu H., Lu S.:  Spatial fusion gan for image synthesis.  In _IEEE CVPR_ (2019), pp. 3653–3662.
  * [ZZP∗17] Zhu J.-Y., Zhang R., Pathak D., Darrell T., Efros A. A., Wang O., Shechtman E.:  Toward multimodal image-to-image translation.  In _NIPS_ (2017).