# Wavefront shaping through a free-form scattering object

Alfredo Rates  1,* Ad Lagendijk  1 Aurèle J. L. Adam  2 Wilbert L. IJzerman
3,4 and Willem L. Vos1  1Complex Photonic Systems (COPS), MESA+ Institute for
Nanotechnology, University of Twente, P.O. Box 217, 7500AE Enschede, The
Netherlands
2Optics Research Group, Department of Imaging Physics, Delft University of
Technology, Lorentzweg 1, 2628 CJ Delft, The Netherlands
3CASA, Department of Mathematics and Computer Science, Eindhoven University of
Technology, PO Box 513, 5600MB Eindhoven, The Netherlands
4Signify Research, High Tech Campus 7, 5656AE Eindhoven, The Netherlands
<EMAIL_ADDRESS>
††journal: opticajournal††articletype: Research Article

Wavefront shaping is a technique to study and control light transport inside
scattering media. Wavefront shaping is considered to be applicable to any
complex material, yet in most previous studies, the only sample geometries
that are studied are slabs or wave-guides. In this paper, we study how
macroscopic changes in the sample shape affect light scattering using the
wavefront shaping technique. Using a flexible scattering material, we optimize
the intensity of light in a focusing spot using wavefront shaping and record
the optimized pattern, comparing the enhancement for different curvatures and
beam radii. We validate our hypothesis that wavefront shaping has a similar
enhancement regardless of the free-form shape of the sample and thus offers
relevant potential for industrial applications. We propose a new figure of
merit to evaluate the performance of wavefront shaping for different shapes.
Surprisingly, based on this figure of merit, we observe that for this
particular sample, wavefront shaping has a slightly better performance for a
free-form shape than for a slab shape.

##  1 Introduction

How to transport light efficiently from A to B? This deceivingly simple
question is the central challenge in the functionality of daily used devices
such as cameras, projectors, lighting systems, and even optically secured bank
cards. When light travels from point A to B through a transparent medium with
macroscopic, optionally curved, surfaces it follows a path that — following
the famous principle of Fermat (1658) — has an optical length which is an
extremum [1]. This changes fundamentally when the medium contains microscopic
particles that scatter light. The technologically relevant question “what
happens when both macroscopic curved interfaces and nanophotonic scattering
media appear simultaneously", has remained unaddressed and unanswered yet. To
date, free-form optics and nanophotonics have developed separately from each
other, the former originating from optical engineering and the latter from
condensed matter physics. The interplay between microscopic scattering and
refraction or reflection by macroscopic free-form surfaces offers new, largely
unexplored, opportunities and solutions to diverse technological problems.

The question of how light travels efficiently from A to B is central to many
advanced technologies that play important roles in modern society, such as
high-precision metrology for integrated nano-circuits in the Internet of
Things (IoT) applications and our smartphones, or detectors in earth-observing
satellites. In modern optics, the question is addressed by custom-designing
free-form components – lenses and curved mirrors – that transfer a given
distribution of light at the source to a desired distribution at the target in
real space, or a sold angle in angle space. Over the last decade, free-form
optics have been used in the development of versatile, miniature, and
efficient devices that appear in daily life [1, 2].

Figure 1:  Tailored-made holder and sample used for experiments. a) 3D model
of the sample holder. b) Commercial smart light bulb, Philips HUE, from which
the free-form shape of our sample is inspired. c) The sample in the sample
holder in the flat position, and d) the sample in the curved position. The
sample is attached horizontally to the holder (red arrow) and vertically to
two movable metal bars (yellow arrow). The bars move in the range delimited by
the rails on the side of the holder.

While nanophotonic media that scatter light cannot be understood with current
free-form design methodologies, many modern devices greatly benefit from these
media. For instance, in our homes, offices, and streets, a quiet revolution is
taking place as white LEDs are replacing energy-inefficient light sources. A
white LED consists of a blue diode [3], whose output is converted and diffused
in a layer of phosphor particles [4, 5]. The presence of the light-scattering
(and also absorbing and re-emitting) phosphor layer is essential to the
functionality of a white LED. Other applications of nanophotonic media include
high-precision metrology tools for nanolithography, the calibration of space
instruments for earth observations, and novel (quantum) optically secured
information technology for privacy. Currently, it is not feasible to describe
light-scattering coatings, diffusers, or suspensions with free-form optics.
Conversely, in nanophotonics, analytic solutions exist only for simple sample
geometries such as a sphere, slab, and plane. No analytic solutions exist for
arbitrary free forms. The presence of microscopically structured materials in
macroscopic free forms implies a huge difference in scale to which
conventional optical models cannot be applied. Hence current industry
solutions invoke shortcuts, including untested assumptions (e.g. material
homogeneity or disregard wavelength dependence). Today’s lack of knowledge on
free-shape scattering optics hampers fast, efficient, and systematic design
progress as well as the development of new optical architectures.

One method extensively used for studying nanophotonic scattering media is
wavefront shaping (WFS), where the light propagation through a scattering
medium is controlled by interference [6, 7, 8, 9, 10]. These interferences
generate a random pattern in the far field, called the speckle pattern.
Recently, the potential of WFS has been extended to, for instance, time-
varying samples [11, 12, 13], and periodic samples such as 2D (and 3D)
photonic crystals [14, 15]. The interferences inside a scattering media are
considered random. But still, when certain properties are slightly changed, a
correlation in the speckle pattern persists. The phenomenon of persisting
correlations is called the memory effect and has been studied by changing the
angle of the incident beam, its position on the sample, and its wavelength
[16, 17, 18, 19]. But for all these experiments, the sample’s shape was kept
constant.

In most cases to date, WFS has been done on the quintessential scattering
sample shape, namely in slabs. However, as previously illustrated, real-world
applications require samples to have a free-form shape and be finite. Although
many robustness studies have been made in WFS, as far as we know, no study
includes changes in the sample’s shape. Because of that, the impact of the
macroscopic characteristics of the object is not addressed in current
theories.

Here, we present the study of an opaque sample of TiO2 particles suspended in
silicone, as shown in Fig. 1. Exploiting the mechanical flexibility of
silicone, we modify the shape of the sample and measure the enhancement of the
intensity $\eta$ in a portion of the speckle pattern in angle space, i.e., a
solid angle. We used a tailored-made sample holder to change both the
curvature of the flexible sample and a focusing lens with a linear stage to
change the beam radius on the sample. When it is curved, the sample has a
curvature radius of approximately $R_{\rm c}=15$ mm. The thickness and the
radius of curvature of the slab are inspired by the dimensions used in, e.g.,
the Philips HUE bulb, see Fig. 1b). In this product, a set of colored LEDs is
used to create both colored and white light at will using wireless control. To
mix the light of these LEDs, the outer bulb plays an instrumental role. The
typical thickness of the bulb is close to 2 mm, and the radius varies with
position, of which we considered $15$ mm a typical value. We compare the
performance of WFS in a flat and a free-form sample as a function of beam
radius. With this, we want to draw the attention of the community to study the
properties of industrially relevant free-form scattering objects both in
theory and experiments.

##  2 Experimental Methods

The optical setup is depicted in Fig. 2. We use a green laser (Coherent
COMPASS 315M-100, $\lambda=532$ nm) as a source, and we modulate the wavefront
using a digital micromirror device (DMD, DLP7000 Texas Instruments). To
convert the binary amplitude modulation into a phase modulation, we use the
Lee Holography technique [20, 21] implemented with a 4f system, where we
filter the $-1$ order in the Fourier plane with an iris. The beam is then
focused into the scattering material, placed in a tailored-made holder, and a
CCD camera (Guppy PRO F-125b) collects the light transmitted through the
sample. The focusing lens L5 is placed on top of a linear stage to have
control over the position of the sample related to the focal length. The
information collected from the CCD camera is used to optimize the DMD pattern
following the sequential algorithm [6, 22].

Figure 2:  Experimental setup for Wavefront Shaping. We use a DMD device
combined with the Lee holography method to achieve phase-only modulation. Lens
L5 is placed on top of a linear stage, to control the distance between L5 and
the sample. We collect the speckle pattern in transmission. Inset: diagram of
the curvature of the sample $R_{\rm c}$. The sample is curved in such a way
that the position of the center of the sample is constant. The sample is
curved using the holder shown in Fig. 1. (L: lens; M: mirror).

The sample consists of TiO2 particles suspended in silicone with a weight
concentration of 0.1 wt%. The sample has a wafer-like shape with a thickness
of $d=2$ mm. To study the free-form sample, we exploit the flexibility of
silicone to do measurements when the sample is flat and perpendicular to the
incident beam and when the sample is curved. To curve the sample in a
controlled manner, we designed a special sample holder, shown in Fig. 1. The
sample is attached to the holder from the side with clamps and on the top and
bottom to sliding metallic bars. These fixed points are highlighted in Fig.
1c) and Fig. 1d) with yellow and red arrows, respectively. When the bars are
close to the front, the sample is completely flat, and when the bars are fixed
close to the back, the sample is curved. With this holder, we control the
curvature of the sample during scattering and wavefront shaping experiments
while other physical parameters are constant.

##  3 Enhancement in curved objects

A relevant parameter to study in WFS experiments is the intensity enhancement
$\eta$. The enhancement is defined as

$\displaystyle\eta\equiv\frac{I_{\rm opt}}{\langle I_{\rm 0}\rangle},$ (1)

where $I_{\rm opt}$ is the intensity at a solid angle after wavefront
optimization and $\langle I_{\rm 0}\rangle$ is the ensemble average intensity
of the pattern before optimization [23]. The ensemble average intensity is
calculated by averaging the total counts on the detector at the target
position over multiple random wavefronts. The enhancement increases when the
number of segments $N$ increases. A segment is a finite area of the incoming
modulated wavefront that has the same phase value.

We compare a free-form sample with a slab-like shape by measuring the
enhancement $\eta$ for a different number of segments $N_{\rm s}$. In Fig. 3,
we plot $\eta(N_{\rm s})$ for both a flat and a curved sample. Each symbol is
composed of the maximum enhancement obtained with the corresponding number of
segments, averaged over three iterations of the optimization loop. Every
iteration uses a flat incident wavefront as the initial value, meaning each
optimization is independent of the others but starts from the same initial
condition. Fig. 3 also includes two curves corresponding to theoretical
limits. We see that the maximum theoretical limit is considerably above the
experimental data. The discrepancy between the maximum limit and experimental
data is because the limit does not take into account variations in the speckle
size. In section 4 we will elaborate an extension of the theoretical limit,
which yields values much closer to the experimental data.

Fig. 3 shows that both shapes give the same trend in enhancement, suggesting
that the performance of the wavefront shaping technique is the same regardless
of the macroscopic properties of the object. This performance persistence is
in agreement with our hypothesis and current theory of wavefront shaping [24,
25]. If we represent the scattering object as a random transmission matrix,
regardless of the value of each coefficient, the efficiency of the wavefront
optimization is unchanged. A possible effect that would cause a change in the
total enhancement is if the number of open channels changes, changing the
saturation point of the enhancement. Our data shows no sign of a change in
saturation, meaning the number of open channels is seemingly unchanged.

Figure 3:  Enhancement versus the number of active modulating segments. The
diameter of the illumination area is $2r_{\rm b}=4.5mm$. The blue circles
represent the enhancement when the sample is flat and the red squares are for
the curved sample. Each symbol is an average from three independent
repetitions of the optimizations. The dashed line is the theoretical curve
after correcting for the variable speckle size, while the solid line is the
maximum theoretical curve without correction. Wavefront shaping is as
effective in optimizing the intensity of a curved sample as it is with a flat
sample.  Figure 4:  Correlation distribution between optimized phase patterns.
The blue distribution corresponds to the correlation between patterns from
different shapes, while the orange and yellow distribution corresponds to the
correlation between the same sample’s shape (curve and flat, respectively).

An alternative interpretation of these results is that the curvature is simply
not affecting the light transport inside the material or that it only affects
the border of the optimized wavefront with a curvature effect. To test this
new hypothesis, we calculate the correlation between the optimized wavefront
of different measurements. For each number of segments, we have a total of 6
measurements, three from a curved sample and three from a flat sample. We take
the optimized wavefront for each measurement, and we calculate the Pearson
correlation coefficient between the 6 cases. We use the SciKit python library
to calculate the correlation, which is defined as [26]

$\displaystyle C_{\rm P}\equiv\frac{\sum_{\rm i,j}(x_{\rm
i,j}-\overline{x})(y_{\rm i,j}-\overline{y})}{\sqrt{(\sum_{\rm i,j}(x_{\rm
i,j}-\overline{x})^{2}(y_{\rm i,j}-\overline{y})^{2}}},$ (2)

with $x_{\rm i,j}$ the value of segment $(i,j)$ of the first wavefront,
$y_{\rm i,j}$ the value of segment $(i,j)$ of the second wavefront, and
$\overline{x}$,$\overline{y}$ the average segment values of the first and
second wavefront, respectively.

We separate the correlations of wavefronts into three classes: the cross-
correlations between wavefronts for a flat and a curved sample $C_{\rm
cross}$, the correlation between wavefronts with flat shape $C_{\rm flat}$,
and the correlation between wavefronts with curved shape $C_{\rm curve}$. For
the cases where the sample’s shape is not changed ($C_{\rm flat}$ and $C_{\rm
curve}$), we expect the optimized wavefronts to converge to a similar
configuration, therefore, a high mutual correlation, regardless of the
hypothesis we mentioned before.

If the macroscopic shape does affect the light transport when we change the
sample’s shape, we expect the optimized wavefront to converge to a new
configuration, therefore $C_{\rm cross}$ to be small. On the other hand, if
the macroscopic changes do not affect the light transport, then we expect the
same value of $C_{\rm cross}$ regardless of the sample’s shape.

The probability distribution of the correlations is shown in Fig, 4. We see
that $C_{\rm cross}$ is centered at 0 and extends from about $-0.3$ to $+0.3$.
In contrast, both $C_{\rm flat}$ and $C_{\rm curve}$ have higher and mostly
positive correlations, between $0$ and $1$ and centered near $0.5$. From this
difference, we conclude that the macroscopic shape of the object affects the
light transport inside it, and to control the scattered light in a free-form
shape, we need a new optimized wavefront compared to the flat shape.

In the field of light scattering, the memory effect is a widely known property
[16, 25, 18, 27]. The memory effect shows that when the light transport inside
the scattering media is slightly changed, the speckle (and thus the optimized
wavefront) are correlated with the previous one. As both speckle patterns are
correlated, the same optimized wavefront can be used to enhance the intensity.
From our observation that curved wavefronts are nearly uncorrelated with flat
wavefronts, we conclude that the change in curvature is beyond the range of
the memory effect. In previous studies, the light transport is changed by
displacing the scattering media, tilting the incident beam, or changing the
wavelength of the source. We thus believe our work is the first step towards
studying the free-form memory effect, i.e., the memory effect when changing
the shape of the scattering media.

In theory, we expect the distributions of $C_{\rm flat}$ and $C_{\rm curve}$
to be centered at 1, while they are centered at 0.5. We attribute this to the
fact that the DMD is not evenly illuminated because the laser beam has a
Gaussian profile. Thus, the segments at the edges do not affect the
optimization significantly and hence the $(x_{\rm i,j},y_{\rm i,j})$ from Eq.
2 are different between one iteration and the other, thereby decreasing the
correlation.

From a geometrical-optics perspective, one might argue that even in the
absence of scatterers, the curvature of the sample could sufficiently impact
the wavefront to induce decorrelation of the optimized wavefronts. If this
were indeed the case, one would anticipate that any decorrelation induced by
the geometry would exhibit a pattern consistent with the curvature, i.e., the
central region of the wavefront being less affected than its edges. However,
upon closer examination of each individual segment, we observe that the
decorrelation is due to random changes in the complete wavefront, rather than
following the expected curvature-related trend. Thus, we dismiss this
possibility.

Figure 5:  a) Cartoon diagram of the illumination area, defined by its radius
$r_{\rm b}$. This depends on the distance between lens L5 and the sample. The
curvature of the diagram is exaggerated for a didactic purpose. b) Diagram of
the optimization area on the camera, defined by its radius $r_{\rm o}$. This
depends on the speckle size at the detector. To maintain a standard
comparison, we keep $r_{\rm o}$ constant in our experiments.

##  4 Enhancement in presence of variable speckle size

To estimate the maximum enhancement possible $\eta_{0}$ in WFS, one generally
invokes wave-guide approximations typical of random matrix theory [24, 28, 29,
30] and simplify the scattering medium as a set of scattering channels. For
this study, We will focus on the intensity enhancement of a single speckle
spot as the figure of merit (other types of optimizations are, e.g., enhance
the total transmission of the system). The maximum enhancement $\eta_{0}$ is
obtained when all the channels are optimized. $\eta_{0}$ expressed as

$\displaystyle\eta_{0}=\frac{\pi}{4}(N_{\rm s}-1)+1,$ (3)

where $N_{\rm s}$ is the total number of segments used in the modulator. This
equation is valid for phase-only modulation and does not consider the
saturation point when $N_{\rm s}$ is equal to the number of transmission
channels inside the sample $N_{\rm c}$.

To compare $\eta_{0}$ with our experiments, we need to distinguish between the
beam radius $r_{\rm b}$ and the optimization radius $r_{\rm o}$. This
difference is illustrated in Fig. 5, where $r_{\rm b}$ is the radius of the
illumination area at the sample surface that depends on the focusing lens L5
and the position of the sample. Conversely, $r_{\rm o}$ is the radius of the
area of the speckle pattern that we observe at the detector, and it is
relative to the speckle size of the speckle pattern. Because we have a lens-
less system, this area represents a solid angle.

Although the optimization process does not depend on $r_{\rm b}$ directly, it
depends on $r_{\rm o}$: to get the maximum enhancement, the optimization area
must have the same size as a single speckle $r_{\rm o}=r_{\rm s}$. The size of
this single speckle in the detector plane depends on the design of the
experiment, particularly if we use or not a lens at the detector. For our
measurements, we use a lens-less system, hence the size of a single speckle is
in solid angle and equal to [31].

Figure 6:  Enhancement versus the number of active modulating segments for a)
flat shape and b) curve shape. On each plot, we show the enhancement for the
maximum ($2r_{\rm b}=4.5mm$, blue circles) and minimum ($2r_{\rm b}=3.8mm$,
red squares) illumination radius. The dashed lines show the theoretical
enhancement corrected based on $r_{\rm b}$.

$\displaystyle r_{\rm s}=\frac{2.44\cdot\lambda}{4r_{\rm b}}d,$ (4)

with $\lambda$ the wavelength of the incident light and $d$ the thickness of
the sample. We infer from Eq 4 that $r_{\rm o}$ and $r_{\rm b}$ are inversely
related, which means the optimization process depends on both parameters.

In Ref. [32], Ojambati and co-workers studied how the maximum enhancement
changes when changing $r_{\rm o}$, thus
$\eta_{0}^{\prime}=\eta_{0}^{\prime}(r_{\rm o})$. This dependency is expressed
as [32]

$\displaystyle\eta_{0}^{\prime}(r_{\rm o})=\frac{K}{r_{\rm o}^{2}}+1,$ (5)

where the correction factor $K$ depends on the number of segments $N_{\rm s}$
and on the speckle size. The theoretical limit $\eta_{0}$ is reached when
$r_{\rm o}$ approaches the speckle size $r_{\rm s}(r_{\rm b})$, $\lim_{r_{\rm
o}\to r_{\rm s}(r_{\rm b})}\eta_{0}^{\prime}(r_{\rm o})=\eta_{0}$. If we take
the limiting value of $r_{\rm o}=r_{\rm s}(r_{\rm b})$ in Eq. 5 and we replace
it with Eq. 4, and we take the limiting value of $\eta_{0}^{\prime}(r_{\rm
o})=\eta_{0}$ and we replace it with Eq. 3, we obtain an extended correction
factor $K(N_{\rm s},r_{\rm b})$:

$\displaystyle K(N_{\rm s},r_{\rm
b})=\frac{2.44^{2}\pi\lambda^{2}}{4^{3}r_{\rm b}^{2}}d^{2}(N_{\rm s}-1).$ (6)

Finally, the theoretical enhancement corrected for optimization and beam areas
is given by

$\displaystyle\eta_{0}^{\prime}(r_{\rm o},r_{\rm
b})=\frac{2.44^{2}\pi\lambda^{2}}{4^{3}r_{\rm b}^{2}r_{\rm o}^{2}}d^{2}(N_{\rm
s}-1)+1.$ (7)

In Eq. 7 we see that the enhancement is inversely proportional to $r_{\rm
b}^{2}$ and $r_{\rm o}^{2}$, and directly proportional to $d^{2}$. Both
dependencies are in accordance with usual WFS experiments, where one focuses
the incident beam to a narrow spot and reduces the optimization area to a
single speckle, while placing the detector far from the sample to increase the
size of the speckle on the detector. In our experiments $r_{\rm o}$ is fixed,
and we compare the measurements while varying $r_{\rm b}$.

##  5 Effect of sample curvature

We study the impact of the ratio between the beam radius $r_{\rm b}$ and the
radius of curvature of the sample $R_{\rm c}$. We vary the radius $r_{\rm b}$
in a range between 1.9 mm and 2.3 mm (hence diameters $2r_{\rm b}=3.8$ to
$4.6$ mm) by changing the position of the focusing lens L5 (see Fig. 2). We
measure the enhancement versus number of segments $N_{\rm s}$ (as in Fig 3)
for six different beam radii $r_{\rm b}$ and both flat and curved sample’s
shapes.

Fig. 6 shows the enhancements for the lowest and highest $r_{\rm b}$ for both
shapes, along with the maximum theoretical limit for each case, calculated
from Eq. 7. We see that the enhancement decreases when increasing the beam
radius, and so does the theoretical limit, in accordance with our theory (see
Eq. 7). When increasing $r_{\rm b}$, the speckle size is smaller. If $r_{\rm
o}$ is constant, there are more speckles inside the optimization area, thus
the maximum enhancement is lower.

Finally, we compare the enhancement of a curved and a flat sample for
different $r_{\rm b}$. For this comparison, we propose the following figure of
merit $F$, defined as:

$\displaystyle F\equiv\langle\frac{\eta_{\rm curve}}{\eta_{\rm
flat}}\rangle_{N},$ (8)

with $\eta_{\rm curve}$ the enhancement for the curve shape, $\eta_{\rm flat}$
the enhancement for the flat shape, and $\langle.\rangle_{N}$ the geometric
mean over the number of segments $N$. This figure of merit is applicable
regardless of the shift shown in Fig. 6 because we expect this shift to occur
both in the flat and the curved shape.

Figure 7:  The figure of merit ($F$) versus beam diameter $2r_{\rm b}$. The
figure of merit is based on comparing a curved sample with $R_{\rm c}=20$ mm
and a flat sample ($R_{\rm c}=\infty$). The light blue line corresponds to the
case when the curvature is not changed, i.e., $FoM=1$.

We show $F$ in Fig. 7 for different beam radii. We see that $F$ is constant at
about $F=1.08$ and it is independent of $r_{\rm b}$.

Finally, we see that all data are larger than $F=1$, showing that, on average,
we achieve a larger enhancement with a curve shape instead of a flat shape.
This challenges the current theoretical framework of wavefront shaping because
no fundamental scattering property was changed except the macroscopic sample’s
shape and the position of the lens, yet based on this figure of merit, the
enhancement difference is statistically significant.

##  6 Conclusions

In this paper, we studied how the intensity enhancement obtained using
wavefront shaping changes when we change the macroscopic properties of the
object. We compare the case of two macroscopic shapes, a free-form curved
sample and a flat sample with a slab shape. We used a tailored-made sample
holder to change the curvature of a flexible sample and a focusing lens with a
linear stage to change the beam radius on the sample.

We see a low correlation between the optimal wavefronts for different
macroscopic shapes, suggesting a change in the light transport inside the
scattering media at a mesoscopic level. Meanwhile, when optimizing the
intensity using the wavefront shaping technique, we reach a similar
enhancement for a curved sample as for a flat sample. This performance
persistence is in agreement with our hypothesis; regardless of the changes in
light transport, the number of open channels is unchanged, thus the maximum
enhancement is not affected by these changes. Our results confirm the idea
that wavefront shaping is applicable regardless of the form of the object.

We derive an extension of the theory for maximum enhancement based on the beam
and optimization radius. Our experiments are in agreement with our theoretical
prediction, and our theory explains the trend for both a curved and flat
sample. Furthermore, we propose a figure of merit to compare the enhancement
between different shapes. Based on this figure of merit, we see an increase in
the total enhancement by 10% for every beam radius. This increase is not
explained using current theories based on slab shapes only.

With this paper, we aim to encourage the community to consider free-form
objects both in experiments and theory of light scattering, as they have a
larger impact on industrial applications. The next steps in this project
include studying speckle decorrelation in real-time while changing the form of
the sample and measuring samples with various densities of scatterers.

Funding Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO-TTW
P15-36).

Acknowledgments We thank Cornelis Harteveld for expert technical support and
sample preparation, and we thank Mario Vretenar for the fabrication of the
sample holder. This work was supported by the MESA+ Institute section Applied
Nanophotonics (ANP).

Disclosures The authors declare no conflicts of interest.

Data availability The data used for this publication are publicly available in
the Zenodo database in Ref. [33].

## References

  * [1] R. Winston, J. C. Miñano, and P. Benítez, eds., _Nonimaging Optics_ (Academic Press, Burlington, 2005).
  * [2] F. Fang, X. Zhang, A. Weckenmann, G. Zhang, and C. Evans, “Manufacturing and measurement of freeform optics,” CIRP Annals 62, 823–846 (2013).
  * [3] S. Nakamura, “Background story of the invention of efficient blue ingan light emitting diodes (nobel lecture),” Annalen der Physik 527, 335–349 (2015).
  * [4] E. F. Schubert, _Light-Emitting Diodes_ (Cambridge University Press, 2006), 2nd ed.
  * [5] M. R. Krames, O. B. Shchekin, R. Mueller-Mach, G. O. Mueller, L. Zhou, G. Harbers, and M. G. Craford, “Status and future of high-power light-emitting diodes for solid-state lighting,” Journal of Display Technology 3, 160–175 (2007).
  * [6] I. M. Vellekoop and A. P. Mosk, “Focusing coherent light through opaque strongly scattering media,” Opt. Lett. 32, 2309–2311 (2007).
  * [7] A. P. Mosk, A. Lagendijk, G. Lerosey, and M. Fink, “Controlling waves in space and time for imaging and focusing in complex media,” Nature Photon. 6, 283––292 (2012).
  * [8] S. M. Popoff, A. Goetschy, S. F. Liew, A. D. Stone, and H. Cao, “Coherent control of total transmission of light through disordered media,” Phys. Rev. Lett. 112, 133903 (2014).
  * [9] I. M. Vellekoop, “Feedback-based wavefront shaping,” Opt. Express 23, 12189–12206 (2015).
  * [10] S. Rotter and S. Gigan, “Light fields in complex media: Mesoscopic scattering meets wave control,” Rev. Mod. Phys. 89, 015005 (2017).
  * [11] M. Nixon, O. Katz, E. Small, Y. Bromberg, A. A. Friesem, Y. Silberberg, and N. Davidson, “Real-time wavefront shaping through scattering media by all-optical feedback,” Nature Photonics 7, 919–924 (2013).
  * [12] R. Horstmeyer, H. Ruan, and C. Yang, “Guidestar-assisted wavefront-shaping methods for focusing light into biological tissue,” Nature Photonics 9, 563–571 (2015).
  * [13] S. Gigan, O. Katz, H. B. de Aguiar, E. R. Andresen, A. Aubry, J. Bertolotti, E. Bossy, D. Bouchet, J. Brake, S. Brasselet, Y. Bromberg, H. Cao, T. Chaigne, Z. Cheng, W. Choi, T. Čižmár, M. Cui, V. R. Curtis, H. Defienne, M. Hofer, R. Horisaki, R. Horstmeyer, N. Ji, A. K. LaViolette, J. Mertz, C. Moser, A. P. Mosk, N. C. Pégard, R. Piestun, S. Popoff, D. B. Phillips, D. Psaltis, B. Rahmani, H. Rigneault, S. Rotter, L. Tian, I. M. Vellekoop, L. Waller, L. Wang, T. Weber, S. Xiao, C. Xu, A. Yamilov, C. Yang, and H. Yılmaz, “Roadmap on wavefront shaping and deep imaging in complex media,” Journal of Physics: Photonics 4, 042501 (2022).
  * [14] R. Sarma, A. Yamilov, and H. Cao, “Enhancing light transmission through a disordered waveguide with inhomogeneous scattering and loss,” Applied Physics Letters 110, 021103 (2017).
  * [15] R. Uppu, M. Adhikary, C. A. M. Harteveld, and W. L. Vos, “Spatially shaping waves to penetrate deep inside a forbidden gap,” Phys. Rev. Lett. 126, 177402 (2021).
  * [16] I. Freund, M. Rosenbluh, and S. Feng, “Memory effects in propagation of optical waves through disordered media,” Phys. Rev. Lett. 61, 2328–2331 (1988).
  * [17] S. Feng, C. Kane, P. A. Lee, and A. D. Stone, “Correlations and fluctuations of coherent wave transmission through disordered media,” Phys. Rev. Lett. 61, 834–837 (1988).
  * [18] B. Judkewitz, R. Horstmeyer, I. M. Vellekoop, I. N. Papadopoulos, and C. Yang, “Translation correlations in anisotropically scattering media,” Nature Physics 11, 684–689 (2015).
  * [19] F. van Beijnum, E. G. van Putten, A. Lagendijk, and A. P. Mosk, “Frequency bandwidth of light focused through turbid media,” Opt. Lett. 36, 373–375 (2011).
  * [20] W.-H. Lee, “III computer-generated holograms: Techniques and applications,” in _Progress in Optics,_ vol. 16 E. Wolf, ed. (Elsevier, 1978), pp. 119–232.
  * [21] D. B. Conkey, A. M. Caravaca-Aguirre, and R. Piestun, “High-speed scattering medium characterization with application to focusing light through turbid media,” Opt. Express 20, 1733–1740 (2012).
  * [22] I. M. Vellekoop and A. P. Mosk, “Universal optimal transmission of light through disordered materials,” Phys. Rev. Lett. 101, 120601 (2008).
  * [23] J. Aulbach, B. Gjonaj, P. Johnson, and A. Lagendijk, “Spatiotemporal focusing in opaque scattering media by wave front shaping with nonlinear feedback,” Opt. Express 20, 29237–29251 (2012).
  * [24] I. Freund, “Looking through walls and around corners,” Physica A 168, 49––65 (1990).
  * [25] M. C. W. van Rossum and T. M. Nieuwenhuizen, “Multiple scattering of classical waves: microscopy, mesoscopy, and diffusion,” Rev. Mod. Phys. 71, 313–371 (1999).
  * [26] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research 12, 2825–2830 (2011).
  * [27] M. Adhikary, “Controlled light propagation in random, periodic, and superperiodic silicon nanophotonic materials,” Ph.D. thesis, University of Twente, Netherlands (2021).
  * [28] E. P. Wigner, “On the statistical distribution of the widths and spacings of nuclear resonance levels,” Mathematical Proceedings of the Cambridge Philosophical Society 47, 790–798 (1951).
  * [29] T. Guhr, A. Müller–Groeling, and H. A. Weidenmüller, “Random-matrix theories in quantum physics: common concepts,” Physics Reports 299, 189–425 (1998).
  * [30] K. L. van der Molen, R. W. Tjerkstra, A. P. Mosk, and A. Lagendijk, “Spatial extent of random laser modes,” Phys. Rev. Lett. 98, 143901 (2007).
  * [31] E. Fujiwara, J. A. Fracarolli, and C. M. B. Cordeiro, “Didactic laser speckle experiments with a lensless camera,” European Journal of Physics 42, 065303 (2021).
  * [32] O. S. Ojambati, J. T. Hosmer-Quint, K.-J. Gorter, A. P. Mosk, and W. L. Vos, “Controlling the intensity of light in large areas at the interfaces of a scattering medium,” Phys. Rev. A 94, 043834 (2016).
  * [33] A. Rates, “Wavefront shaping through a free-form scattering object,” .


# ReGO: Reference-Guided Outpainting for Scenery Image

Yaxiong Wang, Yunchao Wei, Xueming Qian, Li Zhu and Yi Yang Y. Wang is with
the School of Software Engineering, Xi’an Jiaotong University, Xi’an, 710049,
China.
E-mail<EMAIL_ADDRESS>Y. Wei is with the Centre for Artificial
Intelligence, University of Technology Sydney, Ultimo, NSW 2007, Australia.
E-mail<EMAIL_ADDRESS>Qian is with the Key Laboratory for
Intelligent Networks and Network Security, Ministry of Education, Xi’an
Jiaotong University, Xi’an 710049, China, also with the SMILES Laboratory,
Xi’an Jiaotong University, Xi’an 710049,China, and also with Zhibian
Technology Co. Ltd., Taizhou 317000, China.
<EMAIL_ADDRESS>Zhu is with the School of Software, Xi’an
Jiaotong University, Xi’an 710049, China.
E-mail<EMAIL_ADDRESS>Yi Yang is with the Centre for Artificial
Intelligence, University of Technology Sydney, Ultimo, NSW 2007, Australia.
<EMAIL_ADDRESS>received …; revised ….

###### Abstract

We aim to tackle the challenging yet practical scenery image outpainting task
in this work. Recently, generative adversarial learning has significantly
advanced the image outpainting by producing semantic consistent content for
the given image. However, the existing methods always suffer from the blurry
texture and the artifacts of the generative part, making the overall
outpainting results lack authenticity. To overcome the weakness, this work
investigates a principle way to synthesize texture-rich results by borrowing
pixels from its neighbors (_i.e._ , reference images), named Reference-Guided
Outpainting (ReGO). Particularly, the ReGO designs an Adaptive Content
Selection (ACS) module to transfer the pixel of reference images for texture
compensating of the target one. To prevent the style of the generated part
from being affected by the reference images, a style ranking loss is further
proposed to augment the ReGO to synthesize style-consistent results. Extensive
experiments on two popular benchmarks, NS6K [3] and NS8K [9], well demonstrate
the effectiveness of our ReGO.

###### Index Terms:

Image outpainting, GAN, Generation model, Adversarial Learning.

##  1 Introduction

Given an input image, image outpainting aims at generating plausible visual
content outside the image boundary. Traditional approaches [4, 5, 6, 7] employ
a simple searching and stitching pipeline, the extrapolation is achieved by
stitching the picked image patches to the input image. However, these
solutions are too inflexible to meet the practical requirements. Recently,
inspired by the success of the Generative Adversarial Networks (GANs) [15],
researchers are making efforts to synthesize the unseen content for the input
image by adversarial learning [1, 2, 3, 8, 45, 46]. For example, Yang et al.
[3] propose a framework that could recurrently predict new content for the
given image patch. In [2], Piotr et al.translate the input image to a larger
picture with new content beyond the boundary. Wang et al. [9] take the image
outpainting one step forward by introducing the sketch clues to control the
synthesis procedure.

(a) Input

(b) Yang et al. [3]

(c) Piotr et al. [2]

(d) Wang et al. [9]

(e) Ours

(f) original image

Figure 1: Comparisons of sketch-guided image outpainting. All methods except
ours suffer from the lack of the texture details and blurry boundaries of
different semantic regions. The dashed boxes indicate the blurry regions.

(a) Inputs

(b) Reference Images

(c) w/o references

(d) w/ references

(e) Groundtruth

Figure 2: The outpainting examples with (w/) and without (w/o) the reference
images. Although the model abandoning the reference images could predict
reasonable pixels for the inputs, but its outpainting results suffer from the
lack of textural details. Since the neighbor images share many pixels with the
image to be extended, by borrowing some valuable pixels from the reference
images, the model could synthesize outpainting results with rich texture.

Although existing methods could produce coherent content for the given image
patch, results are still not satisfactory due to the lack of texture details.
In Fig. 10(d)\- 1(d), we visualize the outpaining results produced by current
state-of-the-art methods [3, 2, 9]. Generally, these methods could
successfully synthesize the desired images matching the guiding sketches.
However, if we look closely at the details, many poor generated regions such
as pixels with fewer texture particulars and blurry boundaries between
different semantic regions can be observed. As a consequence, the overall
outpainting results are not authentic enough.

Intuitively, an landscape photo is usually with similar layout and appearance
to those photos in the same scene. As shown in the top row of Fig. 2, both the
input patch and the reference image show the sunset related scene, and there
are many valuable pixels in the reference image to help synthesize the high-
quality content for the input patch. Therefore, if we can successfully
transfer the knowledge from similar photos to compensate the textural details
of the predicted content, the authenticity of the generated part may be
significantly improved. Straightforwardly, the input image itself is a natural
choice for serving as the reference, since it often contains content-
consistent pixels to the outpainting part. However, simply adopting the input
image for referring often limits the diversity of sketch layout or content
pattern of the outpainting part, leading to poor generalization ability,
especially for the free-form outpainting.

Motivated by the above observations and considerations, in this work, we
investigate a principle to synthesize detailed outpainting results by taking
the pixels from the neighbors (_i.e._ , reference images) of the given input
as the guidance, named Reference-Guided Outpainting (ReGO). Since the
reference images inevitably include some inconsistent content, simply
performing the transferring process without adaptive filtering will introduce
abrupt pixels, reducing the quality of the generated part accordingly. Thus,
the main challenge of ReGO lies in how to appropriately transfer pixels from
neighbors while maintain the style consistent with the input image.

To this end, an Adaptive Content Selection (ACS) module is first proposed to
augment our ReGO. Concretely, an image-guided convolution is first conducted
on the reference image to select the compensatory features, and two feature
fusion blocks are followed for guiding sketch fusing and boundary stitching,
respectively. With the ACS module, our ReGO can effectively filter out the
abrupt or profitless contents and only adopt the useful features to synthesize
texture-rich results. Besides, the introduced reference image is only
responsible for contributing contents to enrich the final outpainting results,
while the style of the synthesized part should keep consistent with the input,
instead of being affected by the reference. To achieve the style consistency,
we further utilize a hinge-based ranking loss to pull the style of generated
part close to the input image and prevent the style of generated image from
biasing to the reference image. Particularly, the generated part and the input
patch are treated as the positive pair, while the reference image is regarded
as the negative sample, then, the triplet loss is employed to constrain their
style representations. The style ranking loss could make our system avoid some
abrupt pixels and synthesize the results with smooth style. We perform
experiments on two popular benchmarks, NS6K [3] and NS8K [9]. Extensive
quantitative and qualitative comparisons can well demonstrate the superiority
of our ReGO over other state-of-the-art approaches.

In sum, we highlight the contributions of this work as follows:

  * •

We propose an effective outpainting method named ReGO, which can transfer
content-consistent pixels from reference images to synthesize texture-rich
results.

  * •

A novel adaptive content selection module is designed to pick up the
beneficial features from the reference image, making ReGO could filter out
some abrupt pixels and generate semantic-consistent results.

  * •

A style ranking loss is proposed to restrain the style of the synthesized part
and enables the system to synthesize style-consistent results.

  * •

State-of-the-art performance on both random outpainitng and sketch-guided
outpainting tasks.

Figure 3: The overview of the proposed ReGO. The ReGO system takes the left
image & the sketch as inputs, and synthesizes the additional right half new
content for the input image. The overall architecture follows an encoder-
decoder paradigm, where the encoder compresses the inputs and obtain the
hidden feature $F$, and the decoder is responsible to rebuild the complete
image from $F$. Meanwhile, our Adaptive Content Selection (ACS) module could
be equipped in each decoder layer, a reference image is first selected from
the training samples, whose right half is further cut and fed into the
proposed ACS module together with the guiding sketch to compensate the hidden
features. As for the guiding sketch, we use the groundtruth from the right
half image during training. At the testing stage, the guiding sketch could be
manually drawn or borrowed from other image, as shown in Fig.8.

##  2 Related Work

###  2.1 Image Inpainting

The image inpainting has been well explored recent years, whose target is to
restore the missing or corrupt regions in images [30, 31, 32, 35, 22, 27, 34].
Benefit from the tremendous success of the deep learning and the generation
adversarial networks (GANs) [15], the image inpainting has made great advances
these years. Yu et al.attempt to predict the missing pixels from similar
regions and design a contextual attention module to achieve the purpose [32].
Iizuka et al.propose two types of discriminators, _i.e._ , global and local
context discriminators, and employ a fully-convolutional neural network to
train the inpainting system [36]. Instead of filling the missing regions with
formal shape, recently, researcher have started to pay attention to the more
challenging cases, _i.e._ , the free-form inpainting [32, 38, 39, 30, 33]. Liu
et al.develop a novel operation named partial convolution, which could
iteratively predict the missing pixels by collecting information from the
surrounding content [33]. Yu et al.propose the gate convolution as a soft
masking to conduct the feature selection, the feature map could be adaptively
filtered by the produced soft masking [32]. In [39], Xie et al.propose a
feature re-normalization to adapt to the irregular holes. To compensate more
details for the corrupted areas. Guo et al.propose a full-resolution residual
network (FRRN) to restore the missing pixels with irregular shape. Ren et
al.regard the inpainting task as two sub-tasks, _i.e._ , structure
reconstruction and texture synthesizing, consequently, the authors develop a
two-stage method to synthesize the texture-detailed results. Sagong et
al.argue that the two-stage methods are resource-consuming and hard to applied
in the practical situation, therefore, they propose a parallel architecture
including a sharing encoder and two separate decoders. One decoding path is
responsible to conduct a coarse prediction while another attempts to generate
high-quality inpainting based on the preliminary results [42]. In [43], Ding
et al.try to first select the candidates for each target patch according to
the proposed texture similarity measure, and then inpaint the target regions
based on the candidates.

###  2.2 Image Outpainting

Conventional image outpainting methods follow a search-and-compose pipeline,
where the potential patches are first selected from an external library and
then stitched with the input image to conduct extrapolation [13, 10, 12, 11,
5]. For example, Wang et al. [10] first retrieve candidate images using
subgraph matching, and stitch the wrapped images into the input by smoothing
the boundary. Efros et al. [11] also expand the input image with similar
patches, and design an optimal boundary by optimizing the composition cost. In
[12], Zhang et al.take the image category into consideration and propose a
guided shift-map method to extrapolate the input image patch. However, the
performances of these non-parametric methods depend on the results of the
picked candidate images and the size of pre-constructed database, which causes
these methods inflexible and hard to extend to more complex situations.
Different from extrapolating the input from existing image patches,
researchers recently attempt to predict new content beyond the boundaries
using the generative adversarial networks (GANs) [15]. Piotr et al. [2]
develop an encoder-decoder based generator to predict the unseen pixels. The
corrupt image is first compressed by the encoder and restored by the decoder.
Yang et al. [3] utilize the recurrent neural network [14] to iteratively
produce new content, the proposed algorithm could synthesize very long
outpainting results. Wang et al. [1] propose to synthesize the results whose
size matches the desired margins of four directions. Comparing to generate
random extrapolation, controllable image outpainting has more practical
application value. Wang et al. [9] develop a network that allows users to
guide the final synthesis by free-form sketches. In [8], the authors propose
to utilize the language and the position clues to control the outpainting
results. GANs-based outpaiting methods are more flexible, and the generality
is also better than the conventional search-based approaches. However,
existing generative models only focus on producing semantic-consistent content
while ignore the fine-grained details of generated patches, which causes the
outpainting results lack authenticity.

Figure 4: The architecture of our proposed ACS module. The profitable features
are first distilled from the reference image, and then used to compensate the
predicted new features. The sketch fusing block is followed to fuse the sketch
clues, to build a controllable outpainting system. Finally, a seaming block is
applied to achieve the smooth transition from the left to the right content.

##  3 Methodology

The Overview. Fig. 3 exhibits the pipeline of our outpainting system, whose
architecture is designed in an encoder-decoder paradigm, the proposed Adaptive
Content Selection (ACS) module is plugged into each decoder layer.
Particularly, the encoder takes the left half image and the sketch as inputs
and outputs the hidden feature map $F$, which is fed into subsequent decoder
layers to rebuild the complete image. To synthesize the texture-rich results,
the reference image is first chosen from the searching space. _i.e._ ,
training samples and fed into the ACS module to distill its content for
compensating purpose. Besides, to allow the user to harvest the freestyle
outpainting, the guiding sketch clue is also integrated to build a flexible
system. Fig. 4 shows the details of the ACS module. The image-guided
convolution is first employed to distill the beneficial features from the
reference image, then the selected features are integrated with the hidden
representations of the synthesized part to compensate the texture details. In
addition, the style ranking loss is designed to encourage the generator to
produce style-consistent content.

Hereinafter, we first introduce the data structure used in this work in
subsection 3.1, and then the details of proposed ACS module and style ranking
loss are subsequently presented in subsection 3.2 and subsection 3.3,
respectively.

###  3.1 Data Preparation

For an image $I\in\mathbb{R}^{H\times W\times 3}$ from training set $X$, the
corresponding sketch and the reference image are two necessary auxiliary data
for our system. The sketch is treated as a conditional clue to guide the
synthesis procedure as shown in Fig. 1, while the reference image is
responsible to provide rich detailed features for the predicted new content.

To obtain the sketch, we first extract the edge map using the HED edge
detector [16], and binarize the generated edge map with a pre-defined
threshold (0.6 as adopted in our experiment), to acquire the binary sketch
$S\in\mathbb{R}^{H\times W\times 1}$. In the training stage, we adopt the
original sketches of the missing parts from training samples as inputs and
force the network to restore the ground-truth parts. At the testing stage,
users could input the manually drawn free-form sketches to synthesize desired
results.

To obtain reference images for the input $I$, we first extract feature
representations using a pre-trained Places 365 [17] model, and then apply the
cosine similarity to identify the similar samples. Based on our observation,
the visual neighbors often share similar content with the target image and
have more beneficial pixels, thus they are reasonable to be treated as the
reference images. In our practice, we select multiple neighbors for each
sample and randomly pick up one in each training iteration as the reference
image $G$. Besides, since we only need to compensate the details of
synthesized part, the right half of the reference image is only considered for
further processing, _i.e._ , $G\in\mathbb{R}^{H\times W/2\times 3}$.

###  3.2 ACS Module

With the reference image and the guiding sketch, the responsibilities of our
ACS module are two-fold, _i.e._ , compensating the texture details of the
synthesized new content and fusing the conditional information from the
guiding sketch. Consequently, an image-guided convolution is designed for
detail enriching, and a sketch fusing block is employed to integrate the
sketch clue. Additionally, we further apply a seaming block to smooth the
boundary between the original left feature and the predicted new
representation. The details of each block are subsequently presented in the
following.

####  3.2.1 Image-Guided Convolution

The proposed Image-Guided Convolution (IGConv) aims to help the network
compensate texture details for the new content using the distilled the
beneficial features from the reference image. Intuitively, the left part of
the input could directly serve as the reference image. However, based on our
experience, such a strategy often results in an insufficiently diverse
training set. As a consequence, the trained model would have a deep impression
on the original sketch layout and content pattern and fail to generalize to
the freestyle outpainting. Therefore, in our Image-Guided Convolution, we
first search multiple neighbors for the reference image and randomly select a
neighbor in each iteration as the reference. Such a training fashion allows
the model to see diverse input-reference pairs, thus, the generality of the
final model could be boosted accordingly.

Formally, let $F_{L}\in\mathbb{R}^{h\times w\times c}$ be the features encoded
from the image to be extended, $F_{R}$ represents the hidden features for the
predicted new content. $F_{L}$ and $F_{R}$ form the complete hidden features
of the overall image $F\in\mathbb{R}^{h\times 2w\times c}$. And the features
of $G$, which are encoded from a reference image encoder, are denoted as
$F^{G}\in\mathbb{R}^{h\times w\times c}$. The designed image-guided
convolution aims to compensate $F_{R}$ by extracting helpful information from
reference features $F^{G}$. A group of dynamic filters are conditionally
produced based on the features of input and reference images, making the
network adaptively collect the beneficial content from the reference image. To
be specific, a dynamic kernel is produced based on the concatenation of
$F^{G}$ and $F_{L}$ via a simple feed-forward procedure:

$k=\Psi(F^{G},F_{L}),$ (1)

where $k\in\mathbb{R}^{3\times 3\times c}$, 3 and $c$ indicates the kernel
size and channel number, respectively. The $\Psi$ can be modeled as the neural
network, which takes the features of the reference and the input and
recurrently use the convolution with stride=2, batch normalization and
ReLU111ReLU is not applied in the last layer to get the features.

The dynamic kernel in Eq. 1 targets on providing guidance to distill the
content of the reference image. To adaptively pick up the beneficial pixels
and restrain the unhelpful content, we conduct the channel-wise normalization
to update the dynamic kernel:

$k^{n}_{ijk}=\frac{\exp{(k_{ijk})}}{\sum_{h}{\exp{(k_{ijh)}}}}.$ (2)

Thus, the distilled $i$-th channel map can be obtained as follow:

$\widetilde{F}^{G}_{:,:,i}=F^{G}*P(k^{n}_{:,:,i}),i=1,2,...,c,$ (3)

where the $*$ denotes the convolution operation, $P(\cdot)$ is an operation to
repeat $k^{n}_{:,:,i}$ c times along the channel dimension. All of the
$\widetilde{F}^{G}_{:,:,i}|^{c}_{i=1}$ are channel-wise stacked to get the
distilled feature map $\widetilde{F}^{G}\in\mathcal{R}^{h\times w\times c}$.

The dynamic kernel in Eq. 2 focuses on automatically highlighting the
beneficial features and suppressing the profitless content, we expect the
network could assign lower enough weights for the profitless featurs while
give higher weights for the helpful contents via the softmax operation. And
the following distilled convolution in Eq. 3 attempts to greedily summarize
the profitable semantic regions across feature channels from each dynamic
kernel’s perspective. With such a procedure, the helpful features from the
reference image are effectively aggregated into the map $\widetilde{F}^{G}$,
which are further added to the feature $F_{R}$ to achieve the compensatory
purpose: $F_{R}^{*}=F_{R}+\widetilde{F}^{G}$.

Besides the reference image, the input image itself could also provide
valuable pixels. It is because the synthesized part contains the same objects
as the input image with high probability. Therefore, the features from the
input image are also integrated and the $F_{R}^{*}$ is updated as follow:

$F_{R}^{*}=F_{R}+\widetilde{F}^{G}+F_{L}\times\sigma(\rho(F_{L})\times
F_{R}),$ (4)

where $\rho(\cdot)$ is the horizontally flip operation, $\sigma(\cdot)$
denotes the sigmoid function, which is introduced to learn a dynamic feature
selection mechanism.

####  3.2.2 Sketch Fusing Block

Besides synthesizing the unseen part with thriving and realistic details, our
ReGO should also be equipped with a practical mechanism, _i.e._ , allowing
users to acquire personal custom outpainting results using their preferred
sketches as the guidance. To this end, we introduce a controllable sketch
fusion block to achieve the target. To make the final results exactly match
the guiding sketch, the sketch fusion block additionally integrates the sketch
feature to emphasize the desired shape in the restoring procedure, as shown in
Fig. 4.

Concretely, only the right half sketch $S^{r}\in\mathbb{R}^{H\times W/2\times
3}$ serves as the guiding clues, and its feature maps $F^{s}$ are first
encoded by a sketch encoder $E^{S}$. Then, the compressed sketch features are
channel-wise concatenated with the compensated feature $F_{R}^{*}$, and fed
forward a residual block [21] style structure to get the fused output
$F_{R}^{s}$.

####  3.2.3 Seaming Block

Our seaming block is responsible to fuse the raw left half features $F_{L}$
and the compensated features $F_{R}^{s}$, which in fact attempts to smooth the
boundary between the raw features from the input image and the compensated
right half features. As shown in Fig. 4, the seaming block consists of two
global residual blocks (GRB) [3] and a residual block [21]. We alternately
utilize the 1$\times$3 and 7$\times$1 convolution in GRB to strengthen the
connection between the original and the predicted regions, especially the
boundary between the map from the input image and the compensated map of the
predicted new content. Particularly, the $F_{L}$ and $F_{R}^{s}$ are first
concatenated along the width dimension, and then sequentially fed through two
GRBs and a residual block to get the output $F^{{}^{\prime}}$, which is also
the final output of our ACS module.

###  3.3 Style Ranking Loss

The reference image adopted by ReGO is only expected to contribute texture
details, its style should not be reflected on the synthesized content. To
reduce the artifacts, the model should 1) only transfer the texture details
from the reference image, 2) keep a consistent style between the given part
and the generated part.  Given the above considerations, we find the hinge
based ranking is quite in line with our requirement, we take the synthesized
part and the input image as the positive pair while treat the reference image
as the negative sample, then the hinge ranking loss could conduct on their
style representations to enforce the style of the input and the new content to
be closer than the reference image.

Following previous practices [23, 25, 24], we utilize the second-order
statistics of convolutional feature as style representation. Particularly, the
style features of generated part $\hat{I}^{r}\in\mathbb{R}^{H\times W/2\times
3}$, which is only the right half of the image reconstruction
$\hat{I}\in\mathbb{R}^{H\times W\times 3}$, is given by the Gram matrix
$R^{d}\in\mathcal{R}^{N_{d}\times N_{d}}$:

$R^{d}_{ij}=\sum_{k}M^{d}_{ik}M^{d}_{jk},$ (5)

where $M^{d}_{i}$ is the vectorised $i$-th feature map in layer $d$ from a
convolutional neural network like VGG19 [26], $N_{d}$ indicates the channel
number of layer $d$.

Analogously, the style representations of the reference image and the input
image can be extracted, and our style ranking loss is defined as:

$\mathcal{L}^{d}_{s}=[\alpha-SM(R^{d},L^{d})+SM(R^{d},G^{d})]_{+},$ (6)

where $L^{d}$ and $G^{d}$ represent the Garm matrixs of the left input and the
reference image, respectively, $SM(\cdot,\cdot)$ is the cosine similarity,
$\alpha\in\mathcal{R}$ is the scalar margin, and $[\cdot]_{+}=max(\cdot,0)$.

By including the feature correlations of multiple layers, the multi-scale
style representations are obtained, and the total style loss can be calculated
accordingly:

$\mathcal{L}_{s}=\sum_{d\in D}w_{d}\mathcal{L}^{d}_{s},$ (7)

where $D$ is the index collection of selected activation layers, and $w_{d}$
is the trade-off weight. In our experiments, the activated output of layer
relu_Y_1(Y=1,2,3,4,5) of VGG19 network [26] are taken for style
representation, _i.e._ , $|D|=5$. The designed style ranking loss is equipped
to the generator loss to train the network.

TABLE I: Performance comparisons on two datasets under criteria IS, FID and MSD. | Sketch-Guided Outpainting | Random Outpainting
---|---|---
NS6K | NS8K | NS6K | NS8K
IS${\color[rgb]{1,0,0}\uparrow}$ | FID${\color[rgb]{1,0,0}\downarrow}$ | MSD${\color[rgb]{1,0,0}\uparrow}$ | IS${\color[rgb]{1,0,0}\uparrow}$ | FID${\color[rgb]{1,0,0}\uparrow}$ | MSD${\color[rgb]{1,0,0}\uparrow}$ | IS${\color[rgb]{1,0,0}\uparrow}$ | FID${\color[rgb]{1,0,0}\downarrow}$ | IS${\color[rgb]{1,0,0}\uparrow}$ | FID${\color[rgb]{1,0,0}\downarrow}$
NSIO [3] | 2.891 | 12.87 | 0.649 | 3.254 | 10.813 | 0.837 | 2.883 | 13.612 | 3.123 | 12.871
SGIO [9] | 2.920 | 10.998 | 1.01 | 3.321 | 10.390 | 1.17 | 2.951 | 15.857 | 3.178 | 12.316
BDIE [2] | 3.002 | 11.021 | 0.963 | 3.323 | 9.639 | 0.892 | 2.880 | 13.252 | 3.155 | 11.373
$\text{ReGO}_{\text{NSIO}}$ | 2.923 | 12.03 | 0.839 | 3.329 | 10.232 | 0.966 | 2.792 | 14.224 | 2.948 | 13.230
$\text{ReGO}_{\text{SGIO}}$ | 2.924 | 10.104 | 1.201 | 3.387 | 9.787 | 1.293 | 2.848 | 15.396 | 3.204 | 13.748
$\text{ReGO}_{\text{BDIE}}$ | 3.126 | 10.052 | 1.357 | 3.444 | 8.738 | 1.396 | 3.243 | 12.606 | 3.573 | 10.586

##  4 Experiment

###  4.1 Experiment Setup

####  4.1.1 Dataset

This work focuses on performing image outpainting for scenery images. We
conduct extensive experiments on two benchmarks, _i.e._ , NS6K [3] and NS8K
[9], to validate the effectiveness of our ReGO. The NS6K dataset contains 6040
scenery images in total, and 5040 images are treated as training data while
the rest is used for testing [3]. The NS8K, which consists of 8115 images,
contains much more diverse scenery images comparing to NS6K. Of these, 6115
images are taken as training data, the rest is used for testing.

####  4.1.2 Implement Details

Our proposed ReGO provides a model-agnostic solution and could be plugged into
many off-the-shelf outpainting models. In this work, we apply our ReGO to
three state-of-the-art outpainting methods, including NSIO [3], BDIE [2], and
SGIO [9], to validate its superiority:

NSIO [3] is originally designed for random content prediction. To achieve the
sketch-guided outpainting, we make some modifications for NSIO as follow: the
left half sketch is channel-wise concatenated with the input, while the right
half sketch is encoded and fed as the initial state of LSTM [14] decoder to
predict the hidden feature of the full images. Our ACS module is plugged after
each decoding layer except the last one, and the style ranking loss is
weighted by 0.5 and added into the generator loss to train the network.

BDIE [2] is a random-outpainting model as well, and the sketch is concatenated
with the input to perform the sketch-guided outpainting. Besides, the
conditional skip connection and the position channels in SGIO [9] are also
equipped to BDIE [2], to build a stronger baseline. The ACS module and the
style ranking loss are equipped in an analogous way with NSIO [3].

SGIO [9] is the first attempt for the sketch-guided outpainting. The ACS
module is also employed after each decoding layer for texture compensating,
and the style ranking loss is added to the generator loss with weight 0.5 to
ensure the style consistency.

To make a fair comparison, the loss functions, the hyperparameters and the
training details all follow the same settings of their original papers. The
sketch augmentation strategy [9] is also employed for all methods to enhance
the free-form outpainting. For our study, the style ranking loss is equipped
to the generator loss, and the weights of style ranking loss in multiple
layers are all set as 0.2, _i.e._ , $w_{d}=0.2$. During the training stage,
five neighbors are employed in our baseline methods, and the impact of
neighbor number will be discussed in the following experiment. At the testing
stage, only the most similar neighbor is used to synthesize the outpainting.

###  4.2 Evaluation Metric

Following Wang’s [9] setting, three metrics, _i.e._ , Fr$\acute{\rm e}$chet
Inception Distance (FID) [28], the Inception Score (IS) [29] and Mean
Satisfactory Degree (MSD) [9], are employed for evaluation. We feed the
original sketches from the test dataset to rebuild the corresponding images,
and FID and IS can be calculated according to the ground-truth data for
objective comparison:

$\displaystyle\text{IS}=\exp(\mathbb{E}_{\hat{I}\sim\mathbb{P}_{f}}D_{KL}(p(y|\hat{I})||p(y))),$
(8)
$\displaystyle\text{FID}=||\mu_{r}-\mu_{f}||^{2}+Tr(\Sigma_{r}+\Sigma_{f}-2(\Sigma_{r}\Sigma_{f})^{1/2}),$
(9)

where IS is defined based on the KL-divergence between the classification
distribution of the fake sample and the mean probability on each class, while
the FID first employs the Inception-V3 network [44] to extract the 2048-d
features for the groundturth image and the synthesized one and then computes
the statistics distance to evaluate the generation model, the
$\mu_{f}\text{and }\Sigma_{f}$ are the mean vector and the covariance matrix
of the fake features, $Tr$ is the trace operation.

To evaluate the free-form outpainting results, we randomly select 555 images
from test data and replace the original sketches with manually drawn free-form
ones, 89 different types of sketches are collected in total. 20 volunteers are
invited to label the free-form outpainting results as three levels: 0-poor,
1-ordinary and 2-good, and the mean value of all labels on selected images are
taken as the mean satisfaction degree (MSD), which is taken for subjective
comparison since there is no groundtruth available. Comparing to the FID and
the IS, MSD directly reflects the performance in practical situations,
therefore, it is a critical metric to evaluate the generalization ability on
free-form sketches.

###  4.3 Quantitative Comparison

The performance of both sketch-guided outpainting and random outpainting on
datasets NS6K and NS8K is reported in Table I, where
$\text{ReGO}_{\text{NSIO}}$ denotes the NSIO model with our proposed ReGO
equipped. From Table I, we can observe that our proposed ReGO module could
simultaneously enhance the sketch-guided outpainting and the random
outpainting.

(a) BDIE [2]

(b) +RI

(c) +SRL

(d) +ACS

(e) Groundtruth

I: Visual ablation on image rebuilding.

(a) Inputs

(b) BDIE [2]

(c) +RI

(d) +SRL

(e) +ACS

II: Visual ablation on free-form outpainting.

Figure 5: The visual abaltion of each component in our method on image
rebuilding and free-form outpainting, where RI, SRL, and ACS indicate the
reference image, styel ranking loss, and adaptive content selection module,
respectively.

####  4.3.1 Sketch-Guided Outpainting

Our proposed ReGO module could boost the performance of three state-of-the-art
outpainting methods on both NS6K and NS8K. For example, the FID of BDIE [2] is
11.021 on NS6K, when our ReGO module is equipped, the FID could reach 10.052.
Besides the image restoring according to the original sketches, the free-form
outpainting can also be improved by the ReGO module. The MSD of
$\text{ReGO}_{\text{SGIO}}$ can reach 1.201 on NS6K, while the original SGIO’s
is only 1.01. Our best performance is achieved based on the BDIE [2], which
could reach 10.052 FID and 1.357 MSD on NS6K. It is clear from Table I that
our proposed ReGO module can both improve the image rebuilding and free-form
outpainting on three backbones, which validates the effectiveness of our
method.

In our ReGO, the searched neighbors of the input serve as the reference, while
the input image itself is also an intuitive choice since it naturally shares
the highest similarity with the input. To study the effects of two types of
references, we conduct experiments with BDIE backbone on NS6K dataset to
investigate the performance difference. Let $\text{ReGO}_{\text{BDIE}}$-SR
(Self-Reference) denotes the $\text{ReGO}_{\text{BDIE}}$ with the input as
reference. In our experiment, $\text{ReGO}_{\text{BDIE}}$-SR could also
achieve acceptable performance on image rebuilding according to the original
sketches, however, it shows poor generality when encountering the free-style
sketches. For example, the FID of $\text{ReGO}_{\text{BDIE}}$-SR could reach
10.561 on NS6K and surpasses the method BDIE, however, its MDS for free-form
outpainting is only 1.012, which is much worse than
$\text{ReGO}_{\text{BDIE}}$. We guess the barren sketch layout and content
pattern cause somewhat overfitting, consequently, the model trained with self-
reference could not well generalize to the free-style outpainting. In
contrast, When the neighbors serves as the reference, the model could see
diverse training pairs, as a result, the trained model could perform well on
both the image rebuilding and free-form outpainting.

####  4.3.2 Random Outpainting

Besides providing the sketches to harvest the desired outpainting, another
possible scenario is that the users may refuse to drawn any guiding sketches
and only attempt to obtain the random results. How would our system perform if
no guiding sketches are fed? To validate the effectiveness of our method under
such a scenario, we conduct experiments to predict random results and report
the performance on both datasets. Since the NSIO [3] and BDIE [2] are
originally designed for random outpainting, we follow the same pipelines as
their original papers [3, 2] to train the networks. For sketch-guided
outpainting systems, we directly set the right half sketch as zeros to conduct
random outpainting.

TABLE II: The contributions of each part in our method. RI, ACS, and SRL indicate reference image, ACS module, and style ranking loss respectively. | RI | ACS | SRL | IS${\color[rgb]{1,0,0}\uparrow}$ | FID${\color[rgb]{1,0,0}\downarrow}$ | MSD${\color[rgb]{1,0,0}\uparrow}$
---|---|---|---|---|---|---
BDIE [2] |  |  |  | 3.002 | 11.021 | 0.963
BDIE [2] | ✓ |  |  | 2.918 | 10.991 | 1.034
BDIE [2] | ✓ | ✓ |  | 3.038 | 10.269 | 1.221
BDIE [2] | ✓ | ✓ | ✓ | 3.126 | 10.052 | 1.357
TABLE III: The performances of the style ranking loss and the style regression loss. Method | IS${\color[rgb]{1,0,0}\uparrow}$ | FID${\color[rgb]{1,0,0}\downarrow}$ | MSD${\color[rgb]{1,0,0}\uparrow}$
---|---|---|---
$\text{ReGO}_{\text{BDIE}}$-Reg | 3.089 | 10.956 | 1.163
$\text{ReGO}_{\text{BDIE}}$ | 3.126 | 10.052 | 1.357

The results are also reported in Table I, we can observe that abandoning the
original guiding sketches significantly damnify the performance of the sketch-
guided outpainting systems. For example, the $\text{ReGO}_{\text{SGIO}}$ with
guiding sketches could reach 10.104 FID on NS6K, while its FID w/o the guiding
sketches deteriorates to 15.396, this is because the systems are trained with
the original sketches. The performance of SGIO with our ReGO module is
comparable with its original method SGIO [9]. For the random prediction
methods, $\text{ReGO}_{\text{NSIO}}$ performs slightly worse than the NSIO
[3], while $\text{ReGO}_{\text{BDIE}}$ is more outstanding comparing to the
BDIE [2] and achieve the new state-of-the-art performance on random
outpainting. From Table I, we can see that even though no guiding sketches are
provided, the methods with our ReGO module could also produce comparable
results with the original methods. With the designed ACS module, we can
develop a unified framework that could simultaneously deal with the random
prediction and the sketch-guided outpainting. what’s more, the BDIE with ReGO
module could achieve the state-of-the-art performance on both tasks.

###  4.4 Ablation Study

Figure 6: The IS, FID and MSD tendencies of different reference numbers, where
the x-axis indicates the reference number and the y-axis is the corresponding
values. We scale the FID by logarithmic function to make the value of three
criteria close and exhibit clearer tendency.

(a) NSIO [3]

(b) SGIO [9]

(c) BDIE [2]

(d) $\text{ReGO}_{\text{BDIE}}$

(e) Groundtruth

I: Image rebuilding according to the original sketches.

(a) NSIO [3]

(b) SGIO [9]

(c) BDIE [2]

(d) $\text{ReGO}_{\text{BDIE}}$

(e) Groundtruth

II: Random outpainting

Figure 7: The results for the image rebuilding and the random outpainting,
where the dotted red line indicates the imperfect region. Part I shows the
results on image rebuilding according to the original sketches. The comparison
methods could predict reasonable pixels for the input, however, they all
suffer from the blurry synthesized content. While our methods could synthesize
texture-rich content. The part II exhibits the random outpainting. Even though
no sketch is provided for guidance, $\text{ReGO}_{\text{BDIE}}$ could also
synthesise texture-rich results and performs much better than the methods
originally designed for random outpainting, _i.e._ , NSIO [3] and BDIE [2].Red
dashed frames indicate the blurry regions.

(a) Inputs

(b) NSIO [3]

(c) SGIO [9]

(d) BDIE [2]

(e) $\text{ReGO}_{\text{BDIE}}$

I: Outpainting using manually drawn free-form sketches.

(a) Inputs

(b) Outputs

(c) Inputs

(d) Outputs

(e) Original images

Figure 8: Outpainting results according to the manually drawn free-form
sketches and the sketch from other images, where the dotted red line indicates
the imperfect region. Part (I) shows the results for free-form outpainting.
While the part (II) exhibits the outpainting using sketches from the other
images as guidance. The inputs in (a) and (c) use the same reference images
(lower left in (a)) but different sketches. The sketches in (a) are directly
from the reference images, while the sketches in (c) are extracted from a
randomly selected image (shown in lower left). The corresponding predictions
are shown in (b) and (d), respectively. The results are produced by
$\text{ReGO}_{\text{BDIE}}$. Red dashed frames indicate the blurry regions.

II: Outpainting using sketches from the other images as guidance.

(a) Input

(b) Output

(c) Input

(d) Output

(e) Groundtruth

Figure 9: The results for multi-direction prediction. Based on the BDIE
backbone, our method $\text{ReGO}_{\text{BDIE}}$ could predict the content for
multiple directions.

(a) Inputs

(b) Outputs

(c) Inputs

(d) Outputs

Figure 10: Exhibition of failure cases. Although the model could synthesize
contents matching the guiding sketch but fail in predicting reasonable pixels
for the desired shape.

####  4.4.1 Validate the Components of ReGO

To validate the contributions of each component in our proposed ReGO, we
employ the BDIE [2] as backbone and conduct ablations on the NS6K dataset.
Quantitative results are reported in Table II.

As shown in Table II, when the reference image is introduced, the FID of BDIE
could be improved from 11.02 to 10.99 and the MSD could also be boosted, which
reveals compensating the texture details from the neighbors is a promising
idea. However, only the reference image does not make the performance
outstanding enough. With the proposed ACS module adopted, the network could
adaptively block the profitless content and distill the beneficial pixels, and
we can observe a significant performance improvement where the MSD and the FID
could reach 1.221 and 10.269, respectively. When the style ranking loss is
further equipped to ensure the style consistency, the performance step
further. The model with three parts simultaneously utilized achieves the best
performance, and when a new mechanism is equipped, the performance gets
improved, which validates the contributions of each component.

Fig. 5 exhibits the visual results of the ablation comparison on image
rebuilding and free-form outpainting. From Fig. 5, the contribution of each
component can be clearly observed.

####  4.4.2 Validate the Style Ranking Loss

To produce the style-consistent results, we design the style ranking loss to
prevent the style of the synthesized content being affected by the reference
image. Intuitively, the close style between the synthesized content and the
input can also be achieved by directly conducting a regression procedure. In
this subsection, we study the impact of these two solutions.

Table III shows the performance under IS, FID, and MSD, where
$\text{ReGO}_{\text{BDIE}}$-Reg indicates $\text{ReGO}_{\text{BDIE}}$ using
the $l_{2}$ style reconstruction loss instead of the style ranking loss. From
Table III, the $\text{ReGO}_{\text{BDIE}}$ with the proposed style ranking
loss performs much better on both image restoring and free-form outpainting.
The reasons for such results may stem from the overfitting. Although the left
half and the right half parts are from the same image, the style
representations captured by the Gram matrices are still different. Therefore,
stiffly conducting such a regression procedure is easy to cause overfitting.
Besides, the main goal of our style ranking loss is to block the style of the
reference image being reflected on the extended content, the style consistency
between the input and the synthesized part is not necessary to enhance, since
it can be achieved by the pixel-wise reconstruction and the adversarial
training [2, 3, 9].

####  4.4.3 Discuss the Number of Reference Image

In our baseline methods, five neighbors for each training sample are selected,
and we randomly pick up one in each iteration to serve as the reference image.
This subsection investigates the impacts of the number of the reference image.

The performance tendencies with five different reference numbers are shown in
Fig. 6, where the FID is scaled by the logarithmic function. As shown in Fig.
6, there are two important observations. First, Comparing to employing only
one reference image, using multiple references could enhance the model
generality and train a more robust generation model. The FID of
$\text{ReGO}_{\text{BDIE}}$ with only one reference image is 10.728, when the
reference number increases to 5, the FID could be improved to 10.052. Second,
more reference images do not make the performance step further, which is clear
from the performance tendencies when the reference number ranges from 5 to 20.
The situation with five reference images achieves the best performance on
average.

###  4.5 Qualitative Results

Fig. 7 provides the visualizations of the rebuilding results according to the
original sketches and random outpainting for all methods. To ease the visual
exhibition as well as saving some space, we only exhibit the outpainting
results of our best model $\text{ReGO}_{\text{BDIE}}$. It can be observed that
the results of $\text{ReGO}_{\text{BDIE}}$ are more authentic and natural due
to the richer textural details.

From Fig. 7 I, the comparison methods, NSIO [3], SGIO [9] and BDIE [2], could
extend reasonable pixels for the input image, but the predicted content is
blurry and lacks textural details, which makes the overall image not authentic
enough. While $\text{ReGO}_{\text{BDIE}}$ could produce texture-rich
outpainting results. The results of random prediction are exhibited in Fig. 7
II, comparing to the competing methods, $\text{ReGO}_{\text{BDIE}}$ could also
successfully synthesize the results with more textural details when no
sketches are fed, and the synthetic images are even more satisfactory than the
methods original designed for the random prediction, _i.e._ , NSIO [3] and
BDIE [2]. From Fig. 7, we could find that the guiding sketch is not one of
requisite inputs for our system, when the users do not provide the guiding
sketch, ours system could also produce satisfactory random outpainitng
results.

The comparison for free-form outpainting is exhibited in Fig. 8 I,
$\text{ReGO}_{\text{BDIE}}$ could not only synthesize the expected content
matching the guiding sketch but achieve authentic and natural enough results.
Especially the boundaries of different semantic regions are much clearer than
the competing methods. Additionally, we surprisingly find that the reference
image could also help fill reasonable pixels for the free-form outpainting, as
shown in the bottom row in Fig. 8 I. Besides the manually drawn sketches, we
could also use the sketch from another image to guide the outpainting, as
shown in Fig. 8 II. The inputs in Fig. 8 II(a) directly use the sketches of
the reference images to control the outpainting, while the ones in Fig. 8
II(c) use the sketches from randomly selected images, two types of cases serve
as the simple and the difficult cases, respectively. From Fig. 8 II, our
method could not only predict new content matching the guiding sketches but
achieve satisfactory style-consistency for both simple and difficult cases. It
is worthy noting that this paper mainly focuses on synthesizing new content
along left to right, however, the prediction of other directions could also be
performed based on BDIE backbone, just as shown in Fig 9. In this task, our
$\text{ReGO}_{\text{BDIE}}$ could also achieve more outstanding performance
comparing to the BDIE model, 10.817(FID) 3.004(IS)-$\text{ReGO}_{\text{BDIE}}$
VS 12.132(FID), 2.893(IS)-BDIE.

From the above analysis, it can be seen that our developed system is an
unified framework and is flexible enough, the proposed framework allows users
to harvest three types of results: random outpainting, free-form outpainting
from manually drawn sketches and controllable outpainting using sketch from
another image. Therefore, our proposed method is with higher practical value.
We also show some failure cases in Fig. 10, the model fails to predict
reasonable pixels or generate natural enough contents for the guiding
sketches. In our experiment, we find most of failed samples are from those
whose guiding shapes are in the top part of the images, we guess the reason
stems from the barren sketch patterns of training images. This reveals that
there are still many challenges standing for this task. In the future, we will
continue to explore this task and attempt to address more issues of this
field.

##  5 Conclusion and Future Work

This work develops a novel ReGO module that improves the outpainting quality
by borrowing pixels from its neighbors. The proposed method could effectively
boost the results of the sketch-guided image outpainting by enriching the
textual details. An ACS module is proposed to distill the beneficial pixels
and suppress the profitless contents, which could help the generator pick up
the helpful pixels. To prevent the style of the synthesized content being
affected by the reference image, we design a style ranking loss to enforce the
generator to produce the style-consistent contents. Experiments on two
benchmarks based on three backbones demonstrate the effectiveness of the
proposed method. The idea that enriches the details from neighbors may also
work for other generation tasks, we will continue to explore the effectiveness
of the proposed method in the future.

## References

  * [1] Wang Yi, Tao Xin, Shen Xiaoyong, and Jia, Jiaya, “Wide-Context Semantic Image Extrapolation”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019: 1399-1408.
  * [2] Teterwak Piotr, Sarna Aaron, Krishnan Dilip, Maschinot Aaron, Belanger David, Liu Ce, and Freeman William T, ”Boundless: Generative Adversarial Networks for Image Extension”, In _IEEE International Conference on Computer Vision (ICCV)_ , 2019: 10520-10529.
  * [3] Yang Zongxin, Dong Jian, Liu Ping, Yang Yi, and Yan Shuicheng, ”Very Long Natural Scenery Image Prediction by Outpainting”. In _IEEE International Conference on Computer Vision (ICCV)_ , 2019: 10560-10569.
  * [4] Johannes Kopf, Wolf Kienzle, Steven M. Drucker, and Sing Bing Kang, ”Quality prediction for image completion”. _ACM Trans. Graph_ , vol.31, no.6, pages: 131:1–131:8, 2012\.
  * [5] Josef Sivic, Biliana Kaneva, Antonio Torralba, Shai Avidan, and William T. Freeman, ”Creating and exploring a large photorealistic virtual space”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2008: 1-8.
  * [6] Yinda Zhang, Jianxiong Xiao, James Hays, and Ping Tan, ”FrameBreak: Dramatic Image Extrapolation by Guided Shift-Maps”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2013: 1171–1178.
  * [7] Miao Wang, Yu-Kun Lai, Yuan Liang, Ralph R. Martin, and Shi-Min Hu, ”BiggerPicture: data-driven image extrapolation using graph matching”. _ACM Trans. Graph_ , vol.33, no.6, 2014, pages: 173:1–173:13.
  * [8] Yijun Li, Lu Jiang, and Ming-Hsuan Yang, ”Controllable and Progressive Image Extrapolation”. In _CoRR_ , abs/1912.11711, 2019.
  * [9] Yaxiong Wang, Xueming Qian, Yunchao Wei, Li Zhu, Yi Yang and Tao Dai, ”Sketch-Guided Image Outpainting”. In _IEEE Trans. on Image Processing_ : 2021, vol.30, pp:2643-2655.
  * [10] Miao Wang, Yu-Kun Lai, Yuan Liang, Ralph R. Martin, and Shi-Min Hu, ”BiggerPicture: data-driven image extrapolation using graph matching”. _ACM Trans. Graph_ , vol.33, no.6, 2014, pages: 173:1–173:13.
  * [11] Alexei A. Efros, and William T. Freeman, ”Image quilting for texture synthesis and transfer”. _SIGGRAPH_ , 2001: 341–346.
  * [12] Yinda Zhang, Jianxiong Xiao, James Hays, and Ping Tan, ”FrameBreak: Dramatic Image Extrapolation by Guided Shift-Maps”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2013: 1171-1178.
  * [13] Qi Shan, Brian Curless, Yasutaka Furukawa, Carlos Hernández, and Steven M. Seitz, ”Photo Uncrop”. In _European Conference on Computer Vision (ECCV)_ , 2014: 16–31.
  * [14] Sepp Hochreiter, and Jürgen Schmidhuber, ”Long Short-Term Memory”. _Neural Computation_ , vol. 9, no.8, 1997, pages: 1735-1780.
  * [15] Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, ”Generative Adversarial nets”. In _Advances in Neural Information Processing Systems (NIPS)_ , 2014: 2672-2680.
  * [16] Xie Saining, and Tu Zhuowen, ”Holistically-Nested Edge Detection”. In _IEEE International Conference on Computer Vision (ICCV)_ , 2015: 1395-1403.
  * [17] Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba, ”Places: A 10 Million Image Database for Scene Recognition”. _IEEE Trans. Pattern Anal. Mach. Intell_ , vol. 40, no. 6, 2018, pages: 1452–1464.
  * [18] Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville, ”Improved Training of Wasserstein GANs”. In _Advances in Neural Information Processing Systems (NIPS)_ , 2017: 5767–5777.
  * [19] Karen Simonyan, and Andrew Zisserman, ”Very Deep Convolutional Networks for Large-Scale Image Recognition”. In _International Conference on Learning Representations (ICLR)_ , 2015.
  * [20] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter, ”ast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)”. In _International Conference on Learning Representations (ICLR)_ , 2016.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, ”Deep Residual Learning for Image Recognition”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016: 770–778.
  * [22] Thomas Schoenemann, Fredrik Kahl, Simon Masnou, and Daniel Cremers, ”A Linear Framework for Region-Based Image Segmentation and Inpainting Involving Curvature Penalization”. _International Journal of Computer Vision (IJCV)_ , 99(1):53-68, 2012.
  * [23] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge, ”Image Style Transfer Using Convolutional Neural Networks”. In _IEEE Conference on Computer Vision and Pattern Recognition, (CVPR)_ , 2016, 2414–2423.
  * [24] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala, ”Deep Photo Style Transfer”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017:6997–7005.
  * [25] Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, and Jun Wang, ”Attention-Aware Multi-Stroke Style Transfer”,In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019: 1467–1475.
  * [26] Karen Simonyan, and Andrew Zisserman, ”Very Deep Convolutional Networks for Large-Scale Image Recognition”. In _International Conference on Learning Representations (ICLR)_ , 2015.
  * [27] Pablo Arias, Gabriele Facciolo, Vicent Caseles, and Guilermo Spairo, ”A Variational Framework for Exemplar-Based Image Inpainting”. _International Journal of Computer Vision (IJCV)_ , 93(3):319-347, 2011.
  * [28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, ”GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium”. In _Advances in Neural Information Processing Systems (NIPS)_ , 2017: 6626-6637.
  * [29] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen, ”Improved Techniques for Training GANs”. In _Advances in Neural Information Processing Systems (NIPS)_ , 2016: 2226-2234.
  * [30] Zongyu Guo, Zhibo Chen, Tao Yu, Jiale Chen, and Sen Liu, ”Progressive Image Inpainting with Full-Resolution Residual”. In _ACM Multimedia_ , 2019: 2496-2504.
  * [31] Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding, ”Image Inpainting with Learnable Bidirectional Attention Maps”. In _IEEE International Conference on Computer Vision (ICCV)_ , 8857-8866, 2019.
  * [32] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang, ”Free-Form Image Inpainting with Gated Convolution”. In _IEEE International Conference on Computer Vision (ICCV)_ , 2018:4470-4479.
  * [33] Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro, ”Image Inpainting for Irregular Holes Using Partial Convolutions”. In _European Conference on Computer Vision (ECCV)_ , 89-105 2018.
  * [34] Jooyound Hahn, Xue-CHeng Tai, Sofia Borok, and Alfred Bruckstein, ”Orientation-Matching Minimization for Image Denoising and Inpainting”, _International Journal of Computer Vision (IJCV)_ , 92(3): 308-324, 2011.
  * [35] Deepak Pathak, Phillipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros, ”Context Encoders: Feature Learning by Inpainting”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2536-2544, 2017.
  * [36] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa, ”Globally and Locally Consistent Image Completion”. _ACM Trans. Graph_ , vol.36, no.4, pages: 107:1:107:14, 2017.
  * [37] Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li, ”High-Resolution Image Inpainting Using Multi-Scale Neural Patch Synthesis”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017:4076-4084.
  * [38] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang, ”Generative Image Inpainting With Contextual Attention”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018: 5505-5514.
  * [39] Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shiei Wen, and Errui Ding, ”Image Inpainting with Learnable Bidirectional Attention Maps”. In _IEEE International Conference on Computer Vision (ICCV)_ , 2019: 8857-8866.
  * [40] Xintong Han, Zhuxuan Wu, Wweilin Huang, Matthew Scott, and Larry Davis,”FiNet: Compatible and Diverse Fashion Image Inpainting”. In _IEEE International Conference on Computer Vision (ICCV)_ , 2019: 4480-4490.
  * [41] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, and Ge Li, ”StructureFlow: Image Inpainting via Structure-Aware Appearance Flow”, In _IEEE International Conference on Computer Vision (ICCV)_ , 2019:181-190.
  * [42] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor S. Lempitsky, ”Coordinate-Based Texture Inpainting for Pose-Guided Human Image Generation”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019: 12135-12144.
  * [43] Ding Ding, Ssundaresh Ram, and Jeffrey Rodríguez, ”Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering”, _IEEE Trans. Image Process_. vol.28, no.4, pages: 1705-1719, 2019.
  * [44] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ”Rethinking the Inception Architecture for Computer Vision”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016: 2818-2816.
  * [45] Wengling Chen, and James Hays, ”SketchyGan: Towards Diverse and Realistic Sketch to Image Synthesis”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018: 9416-9425.
  * [46] Satoshi Lizuka, Edgar Simo-Serra, ”DeepReamaster: temporal source-refrence attention networks for comprehensive video enhancement”. _ACM Trans. Graph_ , vol.38, no.6: 176:1-176:13, 2019.