11institutetext:  University of Rochester 22institutetext: Adobe Research

# CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware
Training

Haitian Zheng1 Zhe Lin2 Jingwan Lu2 Scott Cohen2 Eli Shechtman2 Connelly
Barnes2 Jianming Zhang2 Ning Xu2 Sohrab Amirghodsi2 Jiebo Luo1

###### Abstract

Recent image inpainting methods have made great progress but often struggle to
generate plausible image structures when dealing with large holes in complex
images. This is partially due to the lack of effective network structures that
can capture both the long-range dependency and high-level semantics of an
image. To address these problems, we propose cascaded modulation GAN (CM-GAN),
a new network design consisting of an encoder with Fourier convolution blocks
that extract multi-scale feature representations from the input image with
holes and a StyleGAN-like decoder with a novel cascaded global-spatial
modulation block at each scale level. In each decoder block, global modulation
is first applied to perform coarse semantic-aware structure synthesis, then
spatial modulation is applied on the output of global modulation to further
adjust the feature map in a spatially adaptive fashion. In addition, we design
an object-aware training scheme to prevent the network from hallucinating new
objects inside holes, fulfilling the needs of object removal tasks in real-
world scenarios. Extensive experiments are conducted to show that our method
significantly outperforms existing methods in both quantitative and
qualitative evaluation.

###### Keywords:

Image Inpainting, Generative Adversarial Networks

Figure 1:  Results of CM-GAN in comparison to state of the art models: ProFill
[1], LaMa [2] and CoModGAN [3]. CM-GAN generates more plausible and realistic
results for the distractor removal scenario (1st row) and large holes (2nd
row).

##  1 Introduction

Image inpainting refers to the task of completing missing regions of an image
as shown in Fig. 1. It is one of the fundamental tasks in computer vision and
has many practical applications, such as object removal [1, 4] and
manipulation [5, 6], image re-targeting [7, 8, 9], image compositing [10], and
3D photo effects [11, 12].

Early inpainting methods leverage patch-based synthesis [9, 13, 14] or color
diffusion [15, 16, 17, 18] to fill the holes by propagating repeating textures
and patterns from the visible regions. To facilitate the completion of more
complex image structures, recent research efforts have shifted to adopting a
data-driven scheme where deep generative networks are learned to predict
visual content and appearance [19, 4, 1, 2, 3]. By training on a large corpus
of images and with assist of both reconstruction and adversarial losses,
generative inpainting models have shown to produce more visually appealing
results on various types of input data including natural images and faces.

While existing works have shown promising results on completing simple image
structures, generating complex holistic structures and image contents with
high-fidelity details remains a huge challenge, espeically when the holes are
large. Essentially, how to 1) _accurately propagate global context into the
incomplete region_ while 2) _synthesizing realistic local details that are
coherent to the global clue_ is the key question for image inpainting. To
tackle global context propagation, existing networks leverage encoder-decoder
structure [20, 21, 22], dilated convolution [23, 4], contextual attention [19,
4, 24], or Fourier convolution [2] to incorporate long range feature
dependency for expanding the effective receptive field [25]. Furthermore, two-
stage approaches [26, 19, 4, 27, 28] and iterative hole filling [1] predict a
coarse result such as a smoothed image, edge/semantic maps or partial
completion to enhance the global structure. However, those models lack a
mechanism to capture high-level semantics in the unmasked region and
effectively propagate them into the hole to synthesize a holistic global
structure. With the typical shallow bottleneck designs, the designed feature
propagation layer is less aware of global semantics and more prone to
generating incoherent local details potentially leading to visual artifacts.

More recently, feature modulation-based methods [29, 30, 22] have shown very
promising results on controlling image generation with a global style code.
Benefiting from a global code that captures the context of the entire image,
CoModGAN [3] attempts to inject global context into the generator for filling
in very large holes [3]. However, due to the lack of spatial adaptation,
global modulation is sensitive to the corrupted encoding feature inside the
inpainting region (shown in Fig. 3). Their results show that passing global
information in this way is insufficient for synthesizing high quality global
structures which may lead to severe artifacts such as the large unseen color
blobs [3] or inconsistent visual appearances such as distorted structures, cf.
Fig. 5.

To seek a better way to inject global context into the missing region in
inpainting, we investigate a new modulation scheme by cascading global and
spatial modulations. We propose Cascaded Modulation GAN (CM-GAN), a new
generative network that can synthesize better holistic structure and local
details, cf. Fig. 1 and Fig. 5. Different from [3] that attempts to globally
modulates the partially invalid encoder feature as shown Fig. 3, our
spatially-adaptive modulation scheme cascaded after a global modulation is
much more effective in processing invalid features inside the hole. Although
several spatially-adaptive modulation schemes [31, 32] have been proposed in
the past to tackle inpainting, our cascaded modulation approach is
significantly different from those works in that: i) our spatial code comes
from the _decoding stage_ rather than the encoding stage to avoid modulating
the decoder with invalid encoding feature (cf. Fig. 3), 2) we incorporate the
global code into spatial modulation for enforcing the global-local
consistency, 3) we introduce a dual-branch design that decouples global and
local features for structure-details separation, and 4) we design spatially-
aware demodulation instead of instance or batch normalization to avoid
potential ‘droplet artifact” [30]. Furthermore, on the encoder side, we inject
the Fast Fourier convolution [33] at each stage of the encoder network to
expand the receptive field of the encoder at early stages, allowing the
network to capture long-range correlations across the image.

Another design aspect to consider is how to generate synthetic masks used
during inpainting training. We need to design the masks tailored for real-
world inpainting use cases, such as object removal and partial object
completion. Previous methods generate training data by randomly locating
rectangular [20, 21] or irregularly-shaped [4, 3] masks. However, the masks
users draw in common use cases likely have a shape of an existing object, or
part of an object or other relatively simple shapes such as a scribble or a
blob. Moreover, users usually expect the removed objects to be filled by
background textures and structures, thus new objects are not expected to
appear inside the holes. However, models trained with randomly located masks
tend to have the color-bleeding effect across object boundaries and generate
object-like artifact blobs inside the hole [3]. In a better attempt, a recent
work [1] leverages saliency annotation to simulate the holes left by
distracting objects occluded by the foreground salient objects. However,
saliency annotations only capture large dominant foreground objects, thus
resulting in the algorithm constructing large holes that include most of the
other less salient objects. This is different from the real use cases where
only a few distracting objects need to be removed.

We found that the tendency to generate spurious objects, blobs and color-
bleeding effects across object boundaries can be addressed by a proper and
more carefully designed object-aware training scheme. In terms of mask
sampling for training, different from random mask [4, 3, 2, 34, 35] or
saliency-based mask [1], we leverage an instance-level panoptic segmentation
model [36] to generate object-aware masks that better simulate real
distractors or clutter removal use cases. To avoid generating distorted
objects or color blobs inside the hole, we filter out cases where the entire
object or a large part of the object is covered by the mask. Furthermore, the
panoptic segmentation provides precise object boundaries and thus prevents the
trained model from leaking colors at object boundaries.

Finally, for the training losses, we propose a masked $R_{1}$ regularization
specifically designed for inpainting and augment the adversarial loss with a
perceptual loss extracted by segmentation model for improving robustness. The
new regularization avoids penalizing the model outside the mask and thus
imposes a better separation of input condition from the generated contents.
Consequently, the new regularization eliminates the potential harmful impact
of computing regularization on the background. Our contributions are four-
fold:

  * •

Cascaded Modulation GAN, a new inpainting network architecture formed by a
masked image encoder with Fourier convolution blocks and a cascaded global-
spatial modulation-based decoder.

  * •

An object-aware mask generation scheme preventing the model from generating
new objects inside holes and mimicking realistic inpainting use cases.

  * •

A masked $R_{1}$ regularization loss to stabilize the adversarial training for
the inpainting tasks.

  * •

State-of-the-art results on the Places2 dataset for various types of masks.

##  2 Related works

###  2.1 Image Inpainting

Image inpainting has a long standing history. Traditionally, the patch-based
methods [37, 13, 9, 14] search and copy-paste patches from known region to
progressively fill in the target hole. Meanwhile, the diffusion-based methods
[15, 16, 17, 18] describe and solve the color propagation inside the hole via
partial differential equation. The above methods can produce high-quality
stationary textures while completing simple shapes, but they lack the
mechanisms to model the high-level semantics for completing new semantic
structure inside the hole. Recently, inpainting methods have shifted to a
data-driven scheme where deep generative models are learned to directly
predict the filled in content inside the hole in an end-to-end fashion. By
training deep generative models via adversarial training [38], the learned
model can capture higher-level representation of images and hence can generate
more visually plausible results. Specifically, Pathak et al. [20] first
leverage an encoder-decoder network with a bottleneck layer to predict the
missing structures for hole filling. Iizuka et al. [21] propose a two
discriminator design to encourage the global and local consistency separately.
To make the generator better at capturing the global context, several attempts
have been made. Motivated by the structure-texture decomposition principle
[39, 40], two-stage networks predict an intermediate representation of image
with smoothed image [19, 41, 24, 42, 1], edge [27, 28], gradient [43] or
segmentation map [44] for enhancing the final output. Yu et al. [19] design
contextual attention to explicitly let the network borrow patch features at a
global scale. Aiming at expanding the receptive field of the network, Iizuka
et al. [21] Yu et al. [19] incorporate dilated convolutions to the generator.
Likewise, Suvorov et al. [2] leverage Fourier convolution [33] to acquire a
global receptive field. Furthermore, feature gating such as partial
convolution [34] and gated convolution [4] is proposed to handle invalid
features inside the hole. To enhance global prediction capacity, Zhao et al.
[3] propose an encoder-decoder network that leverages style code modulation
for global-level structure inpainting. To augment the adversarial loss and to
suppress artifacts, existing works [19, 4, 2, 1, 27] often train the generator
with additional reconstruction objectives such as $\ell_{1}$, perceptual [45]
or contextual [20] loss. Recently, Suvorov et al. [2] propose to use
segmentation networks to compute perceptual loss which achieve better
performance.

###  2.2 Feature Modulation for Image Synthesis

Originating from style transfer [46], feature modulation [47, 29, 30] has been
widely adopted to incorporate input conditions for controlled image generation
[48, 22, 49, 3, 31, 50, 51]. Existing modulation methods usually leverage
batch normalization or instance normalization to normalize the input feature.
The modulation is then achieved by scaling and shifting the normalized
activation according to affine parameters predicted from input conditions.
Recently, Karras et al. [30] find that normalization would cause the “droplet
artifact” as the network can create a strong activation spike to sneak signal
through normalization layer. Consequently, StyleGAN2 [30] replaces the feature
normalization with a proposed demodulation step [30] for better image
synthesis. To modulate the input feature according to a spatially-varying
feature map, spatial modulation [31, 32, 49, 48] are proposed. Essentially,
those methods leverage convolutional layers to predict 2d affine parameters
for spatially-controlled modulation. However, feature normalization makes the
existing approaches [49, 48] less consistent with the design principle of
StyleGAN2.

###  2.3 Regularization for Adversarial Training

Adversarial training is known to be challenging [52] as it is hard for the
adversarial networks to reach global Nash equilibrium [53]. Consequently,
various regularization are proposed to stabilize the GAN training. In
particular, weight normalization [54] and spectrum normalization [55] are
proposed to constrain the the Lipschitz continuity of the discriminator.
Likewise, Gulrajani et al. [56] propose a gradient penalty to impose a
$K$-Lipschitz constraint to the discriminator. Mescheder et al. [52] propose
$R_{1}$ regularization to penalize the discriminator gradient on real data,
which is later used by [30, 3, 2]. Karras et al. [30] propose perceptual path
length regularization on the generator to ensure smoothness mappings and lazy
regularization to optimize the training efficiency.

##  3 Method

###  3.1 Cascaded Modulation GAN

To better model the global context for image completion [19, 4, 1, 57, 24, 3,
2], we propose a novel mechanism that _cascades global code modulation with
spatial code modulation_ to facilitate the processing of the partially invalid
feature while better injecting the global context into the spatial region. It
leads to a new architecture named Cascaded Modulation GAN (CM-GAN), which can
synthesize holistic structures and local details surprisingly well as shown in
Fig. 1.

Figure 2:  Left: The CM-GAN architecture, which consists of an encoder with
FFC blocks and a two-stream decoder with a cascade of global modulation block
(GB) and subsequent spatial modulation block (SB). This cascaded modulation
scheme extracts spatial style codes from the globally modulated feature map
(instead of from the encoder feature map used in the previous work) to make
spatial modulation more effective for inpainting. Right: Cascaded modulation
at each scale. GB and SB take ${\bm{\mathsfit{F}}}_{g}^{\text{in}}$ and
${\bm{\mathsfit{F}}}_{s}^{\text{in}}$, respectively, as inputs and produce the
upsampled feature ${\bm{\mathsfit{F}}}_{g}^{\text{out}}$ and
${\bm{\mathsfit{F}}}_{s}^{\text{out}}$. Specifically, we apply joint global-
spatial modulation to ensure the generation consistency both at the global and
local scales.

Network Overview. As shown in Fig. 2 (left), CM-GAN is based on an encoder
branch and two parallel cascaded decoder branches to generate visual output.
Specifically, it starts with an encoder that takes the partial image and the
mask as inputs to produce multi-scale feature maps
${\bm{\mathsfit{F}}}_{e}^{(1)},\cdots,{\bm{\mathsfit{F}}}_{e}^{(L)}$ at each
scale $1\leq i\leq L$ ($L$ is the highest level with the smallest spatial
size). Unlike most encoder-decoder methods [19, 2] and to facilitate the
completion of holistic structure, we extract a global style code ${\bm{s}}$
from the highest level feature ${\bm{F}}_{e}^{(L)}$ with a fully connected
layer followed by a $\ell_{2}$ normalization. Furthermore, an MLP-based
mapping network [30] is used to generate a style code ${\bm{w}}$ from noise,
simulating the stochasticity of image generation. The code ${\bm{w}}$ is
joined with ${\bm{s}}$ to produce a global code
${\bm{g}}=[{\bm{s}};\,{\bm{w}}]$ for the consequent decoding steps.

Global-Spatial Cascaded Modulation. To better bridge the global context at the
decoding stage, we propose _global-spatial Cascaded Modulation_ (CM). As shown
in Fig. 2 (right), the decoding stage is based on two branches of Global
Modulation Block (GB) and Spatial Modulation Block (SB) to respectively
upsample global feature ${\bm{\mathsfit{F}}}_{g}$ and local features
${\bm{\mathsfit{F}}}_{s}$ in parallel. Different from existing approaches [4,
1, 42, 2, 3], the CM design introduces a new way to inject the global context
into the hole region. At a conceptual level, it consists of a cascade of
global and spatial modulations between features at each scale and naturally
integrates three compensating mechanisms for global context modeling: 1)
_feature upsampling_ allows both GB and SB to utilize the global context from
the low-resolution features generated by both of the previous blocks; 2) the
_global modulation_ (cyan arrows of Fig. 2) allows both GB and SB to leverage
the global code ${\bm{g}}$ for generating better global structure; and 3)
_spatial modulation_ (blue arrows of Fig. 2) leverages spatial code
(intermediate feature output of GB) to further inject fine-grained visual
details to SB.

More specifically, as shown in Fig. 2 (right), CM at each level of the decoder
consists of the paralleled GB block (yellow) and SB block (blue) bridged by
spatial modulation. Such parallel blocks takes
${\bm{\mathsfit{F}}}_{g}^{\text{in}}$ and
${\bm{\mathsfit{F}}}_{s}^{\text{in}}$ as input and output
${\bm{\mathsfit{F}}}_{g}^{\text{out}}$ and
${\bm{\mathsfit{F}}}_{s}^{\text{out}}$. In particular, GB leverages an initial
upsampling layer following a convolution layer to generate the intermediate
feature ${\bm{\mathsfit{X}}}$ and global output
${\bm{\mathsfit{F}}}_{g}^{\text{out}}$, respectively. Both layers are
modulated by the global code ${\bm{g}}$ [30] to capture the global context.

Due to the limited expressive power of the global code ${\bm{g}}$ to represent
a $2$-d scene, and the noisy invalid features inside the inpainting hole [4,
34], the global modulation alone generates distorted features inconsistent
with the context as shown in Fig. 3 and leads to visual artifacts such as
large color blobs and incorrect structure as demonstrated in Fig. 7. To
address this critical issue, we cascade GB with an SB to correct invalid
features while further injecting spatial details. SB also takes the global
code ${\bm{g}}$ to synthesize local details while respecting global context.
Specifically, taking the spatial feature ${\bm{\mathsfit{F}}}_{s}^{\text{in}}$
as input, SB first produces an initial upsampled feature ${\bm{\mathsfit{Y}}}$
with an upsampling layer modulated by global code ${\bm{g}}$. Next,
${\bm{\mathsfit{Y}}}$ is jointly modulated by ${\bm{\mathsfit{X}}}$ and
${\bm{g}}$ in a spatially adaptive fashion following the _modulation-
convolution-demodulation_ principle [30]:

  * •

_Global-spatial feature modulation._ A spatial tensor
${\bm{\mathsfit{A}}}_{0}=\operatorname{APN}({\bm{\mathsfit{X}}})$ is produced
from feature ${\bm{\mathsfit{X}}}$ by a 2-layer convolutional affine parameter
network (APN). Meanwhile, a global vector
${\bm{\alpha}}=\operatorname{fc}({\bm{g}})$ is produced from the global code
${\bm{g}}$ with a fully connected layer ($\operatorname{fc}$) to incorporate
the global context. Finally, a fused spatial tensor
${\bm{\mathsfit{A}}}={\bm{\mathsfit{A}}}_{0}+{\bm{\alpha}}$ leverages both
global and spatial information extracted from ${\bm{g}}$ and
${\bm{\mathsfit{X}}}$, respectively, to scale the intermediate feature
${\bm{\mathsfit{Y}}}$ with element-wise product $\odot$:

$\displaystyle\bar{{\bm{\mathsfit{Y}}}}$
$\displaystyle={\bm{\mathsfit{Y}}}\odot{\bm{\mathsfit{A}}}.$ (1)

  * •

_Convolution_. The modulated tensor $\bar{{\bm{\mathsfit{Y}}}}$ is then
convolved with a $3\times 3$ learnable kernel ${\bm{\mathsfit{K}}}$, resulting
in $\hat{{\bm{\mathsfit{Y}}}}$

$\displaystyle\hat{{\bm{\mathsfit{Y}}}}$
$\displaystyle=\bar{{\bm{\mathsfit{Y}}}}\ast{\bm{\mathsfit{K}}}.$ (2)

  * •

_Spatially-aware demodulation_. Different from existing spatial modulation
methods [31, 32, 49], we discard instance or batch normalization to avoid the
known “water droplet” artifact [30] and propose a spatially-aware demodulation
step to produce normalized output $\widetilde{{\bm{\mathsfit{Y}}}}$.
Specifically, we assume that the input features ${\bm{\mathsfit{Y}}}$ are
independent random variables with unit variance and after the modulation, the
expected variance of the output does not change, i.e.
$\displaystyle\mathbb{E}_{{\textnormal{y}}\in\widetilde{{\bm{\mathsfit{Y}}}}}[\mathrm{Var}({\textnormal{y}})]=1$.
This assumption gives the demodulation computation:

$\displaystyle\widetilde{{\bm{\mathsfit{Y}}}}$
$\displaystyle=\hat{{\bm{\mathsfit{Y}}}}\odot{\bm{\mathsfit{D}}},$ (3)

where
${\bm{\mathsfit{D}}}=1/\sqrt{{\bm{\mathsfit{K}}}^{2}\odot\mathbb{E}_{{\bm{a}}\in{\bm{\mathsfit{A}}}}[{\bm{a}}^{2}]}$
is the demodulation coefficient. Eq. 3 is implemented with standard tensor
operations as elaborated in the supplementary material.

  * •

_Adding spatial bias and broadcast noise_. To introduce further spatial
variation from feature ${\bm{\mathsfit{X}}}$, the normalized feature
$\widetilde{{\bm{\mathsfit{Y}}}}$ is added to a shifting tensor
${\bm{\mathsfit{B}}}=\operatorname{APN}({\bm{\mathsfit{X}}})$ produced by
another affine parameter network from feature ${\bm{\mathsfit{X}}}$ along with
the broadcast noise ${\bm{n}}$ to generate the new local feature
${\bm{\mathsfit{F}}}_{s}^{\text{out}}$:

$\displaystyle{\bm{\mathsfit{F}}}_{s}^{\text{out}}=\widetilde{{\bm{\mathsfit{Y}}}}+{\bm{\mathsfit{B}}}+{\bm{n}}.$
(4)

As shown in the 4th column of Fig. 3, the cascaded SB block helps generate
fine-grained visual details and improves the consistency of feature values
inside and outside the hole.

Figure 3:  Visualization of the intermediate features for inpainting. From
left to right are the incomplete image, encoded feature, our globally
modulated feature and spatially modulated features at the $256\times 256$
layer and the output image.

Expanding the Receptive Field at Early Stages. The fully convolutional models
suffer from slow growth of the effective receptive field [25], especially at
early stages of the network. For this reason, an encoder based on strided
convolution usually generates invalid features inside the hole region, making
the feature correction at the decoding stage more challenging. A recent work
[2] shows that fast Fourier convolution (FFC) [33] can help early layers
achieve large receptive fields that cover the entire image. However, the work
[2] stacks FFC at the bottleneck layer and is computationally demanding.
Moreover, like many other works [4], due to the shallow bottleneck layers, [2]
cannot capture global semantics effectively, limiting its ability to handle
large holes. We propose to replace every convolutional blocks of the CNN
encoder with FFC. By adopting FFC at all scale levels, we enable the encoder
to propagate features at early stages and thus address the issue of generating
invalid features inside the holes, helping improve the results as shown in the
ablation study in Tab. 2.

Figure 4:  Examples of our object-aware masks generated for training (right)
in comparison to the real inpainting requests (left) and the masks generated
by CoModGAN [3] (middle). Note that our masks are more consistent with real
user requests.

###  3.2 Object-aware Training

The algorithm to generate masks for training is crucial. In essence, the
sampled masks should be similar to the masks drawn in realistic use-cases.
Moreover, the masks should avoid covering an entire object or most of any new
object to discourage model from generating object-like patterns. Previous
works generate mask with square-shaped masks [20, 21] or use random strokes
[34] or a mixture of both [4, 3] for training. The oversimplified mask schemes
may cause casual artifacts such as suspicious objects or color blobs.

To better support realistic object removal use cases while preventing the
model from trying to synthesize new objects inside the holes, we propose an
object-aware training scheme that generate more realistic masks during
training as shown in Fig. 4. Specifically, we first pass the training images
to PanopticFCN [36] to generate highly accurate instance-level segmentation
annotations. Next, we sample a mixture of free-form holes [3] and object holes
as the initial mask. Finally, we compute the overlapping ratio between a hole
and each instance from an image. If the overlapping ratio is larger than a
threshold, we exclude the foreground instance from the hole. Otherwise, the
hole is unchanged to mimic object completion. We set the threshold to $0.5$.
We dilate and translate the object masks randomly to avoid overfitting. We
also dilate the hole on the instance segmentation boundary to avoid leaking
background pixels near the hole into the inpainting region.

###  3.3 Training Objective and Masked-$R_{1}$ Regularization

Our model is trained with a combination of adversarial loss [3] and
segmentation-based perceptual loss [2]. The experiments show that our method
can also achieve good results when purely using the adversarial loss, but
adding the perceptual loss can further improve the performance. In addition,
we propose a masked-$R_{1}$ regularization tailored to stabilize the
adversarial training for the inpainting task. Different from [3, 2] that
naively apply $R_{1}$ regularization [52], we leverage the mask $m$ to avoid
computing the gradient penalty outside the mask, specifically:

$\displaystyle\bar{R}_{1}=\frac{\gamma}{2}\displaystyle\mathbb{E}_{p_{\rm{data}}}[\|m\odot\nabla
D(x)\|^{2}],$ (5)

where $m$ is the mask indicating the hole region and $\gamma$ is a balancing
weight. The new loss eliminates the potential harmful impact of computing
gradient on real pixels, and therefore stabilizes the training.

Table 1:  Quantitative evaluation of inpainting on the Places evaluation set.
We report FID [53], LPIPS [58], U-IDS [3] and P-IDS [3] scores.

Methods |  FID$\downarrow$ |  LPIPS$\downarrow$ |  U-IDS$\uparrow$ |  P-IDS$\uparrow$ | Methods |  FID$\downarrow$ |  LPIPS$\downarrow$ |  U-IDS$\uparrow$ |  P-IDS$\uparrow$
---|---|---|---|---|---|---|---|---|---
CM-GAN | 1.628 | 0.189 | 37.42 | 20.96  |  DS [59] | 16.003 | 0.399 | 10.72 | 0.47
CoModGAN[3] | 3.724 | 0.229 | 32.38 |  14.68  |  EC [27] | 12.086 | 0.414 | 9.21 | 0.28
Lama [2] | 3.864 | 0.195 | 29.57 |  10.08  |  ICT [42] | 16.405 | 0.424 | 8.12 | 0.25
ProFill [1] | 7.700 | 0.230 | 21.19 |  3.87  |  HiFill [24] | 37.484 | 0.336 | 9.86 | 0.96
CRFill [57] | 9.657 | 0.233 | 22.90 |  5.53  |  SF [35] | 28.252 | 0.489 | 6.05 | 0.13
DeepFillv2 [4] | 13.597 | 0.371 | 14.23 |  1.67  |  MEDFE [41] | 35.454 | 0.445 | 7.10 | 0.35

##  4 Experiments

Implementation Details. We conduct the image inpainting experiment at
resolution $512\times 512$ on the Places2 dataset [60]. Our model is trained
with Adam optimizer [61]. The learning rate and batch size are set to 0.001
and 32, respectively. Our network takes the resized image as input, so that
the model can predict the global structure of an image. We apply flip
augmentation to increase the training samples.

|
---|---
|
|
|
|
|
|

Figure 5: Visual comparisons on Places2 with our synthesized masks including
large random masks (left) and masks for the distractor removal scenario
(right). We show the input images and the results of ProFill [1], Lama [2],
CoModGAN [3] and CM-GAN (ours). Best viewed by zoom-in on screen.

|
---|---

Figure 6: Visual comparisons on Places2 with the masks proposed by Lama [2].

Evaluation Metrics. We report the numerical metrics on the validation set of
Places2 which contains 36.5k images. For the numerical evaluation, we compute
_Frchet Inception Distance_ (FID) [53] and _Perceptual Image Patch Similarity
Distance_ (LPIPS) [58]. We also adopt the _Paired/Unpaired Inception
Discriminative Score_ (P-IDS/U-IDS) [3] for evaluation.

###  4.1 Comparisons to Existing Methods

We set channel numbers of our network to have a similar model capacity as
CoModGAN and LaMa as shown in Tab. 4.
Quantitative Evaluation. Tab. 1 presents the comparison of our method against
a number of recent methods using our masks. Results show that our method
significantly outperforms all other methods in terms of FID, LPIPS, U-IDS and
P-IDS. We notice that with the assist of perceptual loss, LaMa [2] and our CM-
GAN achieve significantly better LPIPS score than CoModGAN and other methods,
attributing to additional semantic guidance provided of the pre-trained
perceptual model. Compared to LaMa, our CM-GAN reduces FID by over 50% from
3.864 to 1.628, which can be explained by the typically blurry results of LaMa
versus ours which tend to be sharper.

We evaluate generalization of CM-GAN to other types of masks including the
wide mask [2] and the mask of CoModGAN [3]. We also fine-tune CM-GAN with
masks of [2] and [3] (denoted by CM-GAN✝) and report the results. As shown in
Tab. 3, our models with and without fine-tuning achieve clear performance gain
and demonstrate its generalization ability. Notably, CM-GAN trained on our
object-aware masks outperforms CoModGAN on the CoModGAN mask, confirming the
better generation capacity of CM-GAN. The strong capacity of CM-GAN brings
further performance gain after fine-tuning.

Qualitative Evaluation. Fig. 5 and Fig. 6 presents visual comparisons of our
method with state-of-the-art methods on our synthesized masks and other types
of mask introduced by [2, 3]. ProFill [1] generates incoherent global
structures such as the smoothed building and tends to bleed color to the
background for the object-removal case. CoModGAN [3] produces structural
artifacts and color blobs. LaMa [2] is superior on repeating structures, but
tends to generate blurry results on large holes, especially on nature scenes.
In contrast, our method produces more coherent semantic structures and
hallucinates cleaner textures for various types of scenarios.

###  4.2 Ablation Study

We perform a set of ablation experiments to show the importance of each
component of our model. All ablated models are trained and evaluated on the
Places2 dataset. Results of the ablations are shown in Tab. 2. Below we
describe the ablations from the following aspects:

Masked-$R_{1}$ regularization We start from a baseline with a simple encoder-
decoder structure based on global-vector modulation [3] and skip connection.
We compare the baseline trained with $R_{1}$ regularization with the model
trained with masked $R_{1}$ regularization regularization (baseline+m$R_{1}$).
From the result, the masked $R_{1}$ regularization improves the numerical
metrics as the designed loss avoids computing gradient at the fixed input
region.

Cascaded Modulation We next evaluate the cascaded modulation design on top of
the baseline network and m$R_{1}$ loss. Specifically, we cascade global and
spatial modulation on the baseline to get the main CM model baseline+CM (g-s)
ours which improves all numerical metrics. To better understand CM, we
visually compare baseline with baseline+CM (g-s) ours in Fig. 7 and find that
CM significantly improves the synthesized color, texture and global semantic
and corrects the color blob artifact [3], which confirms the effectiveness of
CM on correcting the incoherent feature in a global-semantic aware fashion.

Choices of Second-stage Modulation We evaluate other variant of the second-
stage modulation choices: i) we replace our StyleGAN2-compatible spatial with
skip connection, resulting in a model that cascades global modulation twice,
i.e. baseline+CM(g-g), ii) we test the CM baseline with the spatial modulation
of [32], i.e. baseline+CM(g-[32]). From the results, our spatial modulation
outperforms the global modulation version as we modulate feature using both
global and spatial code. We found [32] does not improve CM as the instance
normalization of [32] is designed for StyleGAN and is less compatible with our
baseline. For the same reason, we found SPADE [31] not compatible with our
baseline due to the use of batch normalization.

Figure 7:  Compared to a global modulation baseline, CM significantly improves
the coherence of the synthesized color, textures, global structures and
objects.

Perceptual Loss The perceptual loss (perc.) provides additional semantic
supervision to the network and can significantly improve the FID metrics.
However, it slightly decreases the discriminative U/P-IDS scores as perceptual
loss may lead to certain visual patterns that are imperceptible to human.

Fast Fourier Convolutions Encoder The Fast Fourier Convolution (FFC) encoder
further brings significant performance gains on top of the cascaded modulation
and perceptual loss as shown in the table, which validates the importance of
more effective encoder with wider receptive fields in early encoding stages.

Object-aware Training To study the effect of object-aware training (OT), we
retrain LaMa [2] and CoModGAN [3] on our object-aware masks. The results show
that object-aware training improves the performance of both of these models
consistently. However, our full model still outperforms these retrained models
significantly. Notably, our model reduces FID of the retrained CoModGAN (with
OT) by 40% from 2.599 of to 1.628.

Table 2:  Ablation study of our model design (architecture, loss, training
scheme) including masked-$R_{1}$ loss (m$R_{1}$), cascaded modulation option
(CM), Fourier convolution (FFC), perceptual loss (perc.) and object-aware
training (OT). We report FID [53], LPIPS [58], U-IDS [3] and P-IDS [3] scores.

Ablations | Methods |  FID$\downarrow$ |  LPIPS$\downarrow$ |  U-IDS$\uparrow$ |  P-IDS$\uparrow$
---|---|---|---|---|---
Masked-$R_{1}$(m$R_{1}$) | Baseline | 2.530 | 0.221 | 36.59 | 21.10
Baseline + m$\bm{R_{1}}$ | 2.475 | 0.221 | 36.58 | 21.55
Cascaded Modulation (CM) | Baseline + CM (g-g) | 2.247 | 0.226 | 37.12 | 22.70
Baseline + CM (g-[32]) | 2.915 | 0.221 | 35.72 | 20.44
Baseline + CM (g-s) | 2.187 | 0.225 | 37.76 | 23.86
FFC and perc. | Baseline + CM + perc. | 1.730 | 0.195 | 36.12 | 19.73
Baseline + CM + perc. + FFC | 1.628 | 0.189 | 37.42 | 20.96
Object-aware Training (OT) |  CoModGAN [3] | 3.724 | 0.229 | 32.38 | 14.68
CoModGAN [3] \+ OT | 2.599 | 0.222 | 35.45 | 20.08
Lama [2] | 3.864 | 0.195 | 29.57 | 10.08
Lama [2] \+ OT | 2.884 | 0.192 | 31.32 | 15.10
CM-GAN (full) | 1.628 | 0.189 | 37.42 | 20.96

Table 3:  Generalization evaluation on other types of mask including wide
masks [2] and CoMoGAN masks [3]. CMGAN✝ are models fine-tuned on the two types
of mask.

Wide masks [2] | CoModGAN masks [3]
---|---
Methods |  FID$\downarrow$ |  LPIPS$\downarrow$ |  U-IDS $\uparrow$ |  P-IDS $\uparrow$ | Methods |  FID$\downarrow$ |  LPIPS$\downarrow$ |  U-IDS $\uparrow$ |  P-IDS $\uparrow$
CM-GAN | 1.521 | 0.129 | 39.24 |  23.24 | CM-GAN | 6.811 | 0.313 | 26.13 | 10.84
CM-GAN✝ | 1.329 | 0.126 | 40.20 | 25.59  | CM-GAN✝ | 5.863 | 0.310 | 27.92 | 12.45
LaMa [2] | 1.838 | 0.123 | 35.00 |  15.12  |  CoModGAN[3] | 7.790 | 0.344 | 24.87 | 10.47
CoModGAN[3] | 1.964 | 0.140 | 37.69 |  21.42  |  LaMa [2] | 12.442 | 0.316 | 18.71 | 4.36
ProFill [1] | 3.333 | 0.142 | 29.12 |  8.39  |  ProFill [1] | 20.314 | 0.352 | 11.21 | 1.23

Table 4:  The inference complexities. Our model has a similar number of parameters and FLOPs as other recent models.  Models |  #Params of $\mathcal{G}$ |  #Params of $\mathcal{D}$ | FLOPs
---|---|---|---
CoModGAN [3] | 79.79M | 28.98M | 345.54G
LaMa [2] | 51.25M | 9.258M | 395.30G
CM-GAN (ours) | 75.28M | 28.98M | 373.60G

###  4.3 User Study

We conduct a user study to better evaluate the visual quality of our method.
Specifically, we generate samples for evaluation using the Places2 evalation
set and three types of masks: our object-aware mask, wide mask [2] and mask
from real user request. The former data class contains 30 samples while the
latter contains 13 real inpainting requests online following [57]. Each input
image with the region to be removed and the results of different methods are
presented to online users who are asked to select the best result. Finally, we
collect votes from all users. Results in Tab. 5 shows that our method receives
the majority of votes on both the synthetic data and realistic object removal
requests.

Table 5:  The user study. For each mask type, we show the number of votes and percentages for different methods (ProFill, LaMa, CoModGAN, and ours)  Masks |  ProFill [1] |  Lama [2] |  CoModGAN [3] | CM-GAN
---|---|---|---|---
Our mask | 20 (5%) | 68 (17%) | 83 (20%) | 234 (58%)
Wide mask | 45 (10%) | 107 (25%) | 94 (22%) | 186 (43%)
User mask | 15 (5%) | 101 (35%) | 55 (19%) | 120 (41%)

##  5 Conclusion

In this paper, we present a new approach tailored to real-world image
inpainting. Our method is based on a new modulation block that cascades global
modulation with spatial modulation for better pass global context into the
hole region. We further propose a training scheme based on object-aware mask
sampling to improve generalization to real use cases. Finally, we propose
specifically designed masked $R_{1}$ regularization to stabilize the
adversarial training of image inpainting networks. Our method achieves the new
state-of-the-art performance on the Places2 dataset and better visual quality.

Currently, our model is still limited in synthesizing large objects like
humans or animals. One possible solution is to train a specialist inpainting
model for specific types of objects. Another direction is to leverage depth
and semantic segmentation for more precise structure-aware inpainting.

Appendix

We provided more analysis, visual results and implementation details in the
appendix, including the supplemental ablation studies (Appendix 0.A), analysis
of object-aware training (Appendix 0.B), effect of the masked-$R_{1}$
regularization (Appendix 0.C), visual comparison to other methods (Appendix
0.D), visual comparison on other types of masks (Appendix 0.E) and more
implementation details (Appendix 0.F). We also introduce comparisons to the
recent transformer-based approach TFill [62]. All visual results are in high-
resolution and best viewed by zoom-in on screen.

##  Appendix 0.A Supplemental Ablation Studies

To complement the main ablation study in Table 2 of the main paper, Table 6
provides the additional ablation results on spatial modulation choices.

Baseline with Spatial Modulation We evaluate a baseline model with spatial
modulation instead of the global code modulation, i.e. Baseline + mR1 + s.
Spatial modulation enhances the global modulation model, i.e. baseline + mR1
as spatial modulation provides more spatial adaptation.

Spatial Modulation without Demodulation baseline + CM(g-s) plain is a cascaded
modulation model with a plain spatial modulation operation, i.e. without the
demodulation step (Eq. 3). We empirically found such a model is not as good as
baseline + CM(g-s) ours, as the demodulation step can improve the results by
regularizing the intermediate feature activation.

Table 6:  Supplemental results to the main ablation study (table 5 of the main
paper) The supplemental results are denoted in blue color.

Ablations | Methods |  FID$\downarrow$ |  LPIPS$\downarrow$ |  U-IDS$\uparrow$ |  P-IDS$\uparrow$
---|---|---|---|---|---
Baseline |  Baseline + m$\bm{R_{1}}$ | 2.475 | 0.221 | 36.58 | 21.55
Baseline + m$\bm{R_{1}}$ \+ s | 2.398 | 0.218 | 36.72 | 21.94
Cascaded Modulation (CM) | Baseline + CM (g-g) | 2.247 | 0.226 | 37.12 | 22.70
Baseline + CM (g-s) plain | 2.392 | 0.224 | 37.23 | 22.67
Baseline + CM (g-s) stylemapgan[32] | 2.915 | 0.221 | 35.72 | 20.44
Baseline + CM (g-s) ours | 2.187 | 0.225 | 37.76 | 23.86

##  Appendix 0.B Visual Effects of Object-aware Training on Other Models

To analyze the generalization of object-aware training to other recent
inpainting methods [2, 3] while complementing the numerical results in Table 2
of the main paper, we provide the supplemental visual effect of object-aware
training in Figure 8 and Figure 9.

Figure 8 presents the visual comparison of LaMa [2] and CoModGAN [3] trained
without or with object-aware training (OT). Object-aware training in general
improves other state-of-the-art models including LaMa and CoModGAN on
retaining object boundaries and background under the distractor removal
scenario.

Figure 8:  The effect of object-aware training on other models. Object-aware
training (OT) improves other models including LaMa [2] and CoModGAN [3] on
achieving sharper boundaries and clearer background. Best viewed by zoom-in on
screen.

Furthermore, Figure 9 presents the visual comparison of our method and the
state-of-the-art models trained with object-aware training, including LaMa-OT
and CoModGAN-OT. CM-GAN with object-aware training achieves better performance
than other state-of-the-art models trained with object-aware training,
validating the strong generation capacity of CM-GAN.

Figure 9:  Results of CM-GAN in comparison to LaMa [2] and CoModGAN [3]
trained with object-aware training (OT). Best viewed by zoom-in on screen.

##  Appendix 0.C The Effect of The Masked-$R_{1}$ Regularization

Figure 10 visualizes the effect of masked-$R_{1}$ during the training.
Specifically, we visualize the baseline model trained with masked-$R_{1}$
regularization (red) and $R_{1}$ regularization (orange). The masked-$R_{1}$
regularization helps the model achieve lower FID scores, higher discriminator
classification loss and makes discriminator harder to distinguish fake
samples.

Figure 10:  The convergence curves of the baseline models trained with
masked-$R_{1}$ regularization (red) and $R_{1}$ regularization (orange). The
masked-$R_{1}$ regularization help the model achieve lower FID scores, higher
discriminator classification loss and makes the discriminator harder to
distinguish fake samples.

##  Appendix 0.D Additional Qualitative Results

In the following, we provide the supplementary qualitative results of methods
evaluated in Table 1 of the main paper. In addition, we include evaluation of
a recent transformer-based approach, i.e. TFill [62] in Section 0.D.2.

###  0.D.1 More Visual Comparisons with ProFill, LaMa, and CoModGAN

We present additional visual comparisons to ProFill [1], LaMa [2] and CoModGAN
[3] in Figures 11, 12, 13 and 14 to supplement Figure 5 of the main paper.

Figure 11:  Results of CM-GAN in comparison to ProFill [1], LaMa [2] and
CoModGAN [3]. Best viewed by zoom-in on screen.  Figure 12:  Results of CM-GAN
in comparison to ProFill [1], LaMa [2] and CoModGAN [3]. Best viewed by zoom-
in on screen.  Figure 13:  Results of CM-GAN in comparison to ProFill [1],
LaMa [2] and CoModGAN [3]. Best viewed by zoom-in on screen.  Figure 14:
Results of CM-GAN in comparison to ProFill [1], LaMa [2] and CoModGAN [3].
Best viewed by zoom-in on screen.

###  0.D.2 Visual Comparisons to Transformer-based Methods

CM-GAN is based on the GAN framework. With transformers becoming popular in
computer vision, several recent works [59, 42, 62] leverage transformer-based
architectures for inpainting. In this section, we present the visual
comparison to DS [59], ICT [42] and the recently proposed TFill [62] to
analyze the visual quality of those approaches. As observed in Figure 15, CM-
GAN achieves consistently better visual results in terms of holistic
structures and local textures, which is coherent to the FID scores reported in
Table 1 of the main paper 111The FID score of TFill is 7.435..

Figure 15:  Results of CM-GAN in comparison to transformer-based approaches
including DS [59], ICT [42] and TFill [62]. Best viewed by zoom-in on screen.

###  0.D.3 Visual Comparisons to Other Remaining Methods

Figure 16 presents the visual comparisons of CM-GAN to other methods including
EdgeConnect [27], MEDEF [41], DeepFillv2 [4], HiFill [24], CRFill [57]. Our
method achieves substantially better visual quality than all these compared
methods.

Figure 16:  Results of CM-GAN in comparison to other methods including
EdgeConnect [27], MEDEF [41], DeepFillv2 [4], HiFill [24], CRFill [57].  Best
viewed by zoom-in on screen.

##  Appendix 0.E Visual Comparisons on Other Types of Masks

To supplement Table 3 of the main paper, we provide the visual comparisons on
the LaMa mask [2] in Figure 17 and CoModGAN mask [3] in Figure 18,
respectively. Consistent to the numerical result in the main paper, our method
generates better global structure and more coherent textures than others,
demonstrating robustness of CM-GAN on different mask types.

Figure 17:  Visual comparison on the mask of LaMa [2].  Best viewed by zoom-in
on screen.

Figure 18:  Visual comparison on the mask of CoModGAN [3].  Best viewed by
zoom-in on screen.

##  Appendix 0.F Implementation Details

###  0.F.1 The Details of Spatial Modulation

We provide the detail implementation of the Affine Parameters Networks (APN)
and our spatial modulation operation in pseudo code.

The Affine Parameters Network (APN). The affine parameters network (APN) is
implemented as a stack of convolutional layer that takes ${\bm{\mathsfit{X}}}$
as input to generate scaling parameters ${\bm{\mathsfit{A}}}$ and shifting
parameters ${\bm{\mathsfit{B}}}$.

⬇

def APN(X):

# the 1x1 input layer

t1 = self.conv1_1x1(X)

# the 3x3+1x1 middle layer

t2 = self.conv2_3x3(t1)

t2 = t2 + self.conv2_1x1(t1)

# the 1x1 output layer

A = self.conv_A_1x1(t)

B = self.conv_B_1x1(t)

return A, B

Spatial Modulation. Next, the spatial modulation takes feature maps
${\bm{\mathsfit{X}}}$, ${\bm{\mathsfit{Y}}}$, global code ${\bm{g}}$, the
convolutional kernel weight ${\bm{w}}$ and the noise ${\bm{n}}$ as inputs to
modulate ${\bm{\mathsfit{Y}}}$:

⬇

import torch.nn.functional as F

def spatial_mod_ops(X, Y, g, w, noise):

bs = X.size(0) # batch size

# predicting the spatial code

A0, B = self.APN(X)

# merging with the global code

A = A0 + self.fc(g).reshape(bs,-1,1,1)

# spatial modulation

Y = Y.mul(A)

# convolution

Y = F.conv2d(Y, w)

# spatially-aware demodulation

w = w.unsqueeze(0)

A_avg_var = A.square().mean([2,3]).reshape(bs,1,-1,1,1)

D = (w.square().mul(A_avg_var).sum(dim=[2,3,4]) + 1e-8).rsqrt()

Y = Y.mul(D.reshape(bs, -1, 1, 1))

# adding bias and noise

Y = Y + B + noise

return Y

###  0.F.2 Details of the Object-Aware Mask Generation Procedure

Our object-aware mask generation scheme is based on the following pipeline to
sample a mask for image $x$:

  1. 1.

_Generating the initial mask_. We sample an initial mask ${\bm{m}}$ with
either irregular masks proposed by [3], object masks proposed by [1] or random
overlapping rectangle with probablity 0.45, 0.45 and 0.1, respectively. We
further augment the object mask by random circular translation and dilate the
mask using random width.

  2. 2.

_Occluding foreground instance_. For each object instance ${\bm{s}}_{i}$ from
image $x$, we compute the overlapping ratio
$r_{i}=\operatorname{Area}({\bm{m}},{\bm{s}}_{i})/\operatorname{Area}({\bm{s}}_{i})$
between the instance and the initial mask. If the overlapping ratio $r_{i}$ is
larger than 0.5, we exclude instance ${\bm{s}}_{i}$ from the initial mask
${\bm{m}}$, namely ${\bm{m}}\leftarrow{\bm{m}}-{\bm{s}}_{i}$, to mimic the
distractor removal use case.

  3. 3.

_Rejecting small mask._ If the area of the mask is less than 0.05 of the area
of the entire image, we repeating the sampling procedure, until the maximal
sampling iteration 5 is reached. Otherwise, the sampled mask is returned.

## References

  * [1] Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu.  High-resolution image inpainting with iterative confidence feedback and guided upsampling.  arXiv preprint arXiv:2005.11742, 2020.
  * [2] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions.  arXiv preprint arXiv:2109.07161, 2021.
  * [3] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu.  Large scale image completion via co-modulated generative adversarial networks.  arXiv preprint arXiv:2103.10428, 2021.
  * [4] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4471–4480, 2019.
  * [5] Evangelos Ntavelis, Andrés Romero, Iason Kastanis, Luc Van Gool, and Radu Timofte.  Sesame: Semantic editing of scenes by adding, manipulating or erasing objects.  In European Conference on Computer Vision, pages 394–411. Springer, 2020.
  * [6] Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand.  Image-based modeling and photo editing.  In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 433–442, 2001.
  * [7] Vidya Setlur, Saeko Takagi, Ramesh Raskar, Michael Gleicher, and Bruce Gooch.  Automatic image retargeting.  In Proceedings of the 4th international conference on Mobile and ubiquitous multimedia, pages 59–68, 2005.
  * [8] Daniel Vaquero, Matthew Turk, Kari Pulli, Marius Tico, and Natasha Gelfand.  A survey of image retargeting techniques.  In Applications of Digital Image Processing XXXIII, volume 7798, pages 328–342. SPIE, 2010.
  * [9] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Trans. Graph., 28(3):24, 2009.
  * [10] Bor-Chun Chen and Andrew Kae.  Toward realistic image compositing with adversarial learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8415–8424, 2019.
  * [11] Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu.  3d ken burns effect from a single image.  ACM Transactions on Graphics, 38(6):184:1–184:15, 2019.
  * [12] Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, Peizhao Zhang, Zijian He, Peter Vajda, Ayush Saraf, and Michael Cohen.  One shot 3d photography.  39(4), 2020.
  * [13] Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra.  Texture optimization for example-based synthesis.  In ACM SIGGRAPH 2005 Papers, pages 795–802. 2005.
  * [14] Taeg Sang Cho, Moshe Butman, Shai Avidan, and William T Freeman.  The patch transform and its applications to image editing.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [15] Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera.  Filling-in by joint interpolation of vector fields and gray levels.  IEEE transactions on image processing, 10(8):1200–1211, 2001.
  * [16] Tony F Chan and Jianhong Shen.  Nontexture inpainting by curvature-driven diffusions.  Journal of visual communication and image representation, 12(4):436–449, 2001.
  * [17] Jianhong Shen and Tony F Chan.  Mathematical models for local nontexture inpaintings.  SIAM Journal on Applied Mathematics, 62(3):1019–1043, 2002.
  * [18] Antonio Criminisi, Patrick Pérez, and Kentaro Toyama.  Region filling and object removal by exemplar-based image inpainting.  IEEE Transactions on image processing, 13(9):1200–1212, 2004.
  * [19] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505–5514, 2018.
  * [20] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [21] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.
  * [22] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A Efros, and Richard Zhang.  Swapping autoencoder for deep image manipulation.  arXiv preprint arXiv:2007.00653, 2020.
  * [23] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [24] Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu.  Contextual residual aggregation for ultra high-resolution image inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7508–7517, 2020.
  * [25] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 4905–4913, 2016.
  * [26] Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li.  High-resolution image inpainting using multi-scale neural patch synthesis.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 4076–4084, 2017.
  * [27] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi.  Edgeconnect: Generative image inpainting with adversarial edge learning.  arXiv preprint arXiv:1901.00212, 2019.
  * [28] Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo.  Foreground-aware image inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5840–5848, 2019.
  * [29] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [30] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In Proc. CVPR, 2020.
  * [31] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [32] Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh.  Exploiting spatial dimensions of latent in gan for real-time image editing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [33] Lu Chi, Borui Jiang, and Yadong Mu.  Fast fourier convolution.  Advances in Neural Information Processing Systems, 33, 2020.
  * [34] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 85–100, 2018.
  * [35] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, and Ge Li.  Structureflow: Image inpainting via structure-aware appearance flow.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [36] Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia.  Fully convolutional networks for panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 214–223, 2021.
  * [37] Alexei A Efros and William T Freeman.  Image quilting for texture synthesis and transfer.  In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 341–346. ACM, 2001.
  * [38] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [39] Jean-François Aujol, Guy Gilboa, Tony Chan, and Stanley Osher.  Structure-texture image decomposition—modeling, algorithms, and parameter selection.  International journal of computer vision, 67(1):111–136, 2006.
  * [40] Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, and Stanley Osher.  Simultaneous structure and texture image inpainting.  IEEE transactions on image processing, 12(8):882–889, 2003.
  * [41] Yibing Song Wei Huang Hongyu Liu, Bin Jiang and Yang Chao.  Rethinking image inpainting via a mutual encoder-decoder with feature equalizations.  In Proceedings of the European Conference on Computer Vision, 2020\.
  * [42] Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao.  High-fidelity pluralistic image completion with transformers.  arXiv preprint arXiv:2103.14031, 2021.
  * [43] Jie Yang, Zhiquan Qi, and Yong Shi.  Learning to incorporate structure knowledge for image inpainting.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12605–12612, 2020.
  * [44] Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo.  Spg-net: Segmentation prediction and guidance network for image inpainting.  arXiv preprint arXiv:1805.03356, 2018.
  * [45] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [46] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1501–1510, 2017.
  * [47] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In Proceedings of the European conference on computer vision (ECCV), pages 172–189, 2018.
  * [48] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 606–615, 2018.
  * [49] Badour AlBahar, Jingwan Lu, Jimei Yang, Zhixin Shu, Eli Shechtman, and Jia-Bin Huang.  Pose with Style: Detail-preserving pose-guided image synthesis with conditional stylegan.  ACM Transactions on Graphics, 2021.
  * [50] Haitian Zheng, Haofu Liao, Lele Chen, Wei Xiong, Tianlang Chen, and Jiebo Luo.  Example-guided image synthesis using masked spatial-channel attention and self-supervision.  In European Conference on Computer Vision, pages 422–439. Springer, 2020.
  * [51] Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Gang Hua, and Nenghai Yu.  Semantic image synthesis via efficient class-adaptive normalization.  arXiv preprint arXiv:2012.04644, 2020.
  * [52] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  In International conference on machine learning, pages 3481–3490. PMLR, 2018.
  * [53] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems, pages 6626–6637, 2017.
  * [54] Tim Salimans and Durk P Kingma.  Weight normalization: A simple reparameterization to accelerate training of deep neural networks.  Advances in neural information processing systems, 29:901–909, 2016\.
  * [55] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  arXiv preprint arXiv:1802.05957, 2018.
  * [56] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville.  Improved training of wasserstein gans.  arXiv preprint arXiv:1704.00028, 2017.
  * [57] Yu Zeng, Zhe Lin, Huchuan Lu, and Vishal M. Patel.  Cr-fill: Generative image inpainting with auxiliary contextual reconstruction.  In Proceedings of the IEEE International Conference on Computer Vision, 2021.
  * [58] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  * [59] Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li.  Generating diverse structure for image inpainting with hierarchical vq-vae.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10775–10784, 2021.
  * [60] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  * [61] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [62] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  Tfill: Image completion via a transformer-based architecture.  arXiv preprint arXiv:2104.00845, 2021.