# Determination of the neutron skin of 208Pb from ultrarelativistic nuclear
collisions

Giuliano Giacalone  Institut für Theoretische Physik, Universität Heidelberg,
Philosophenweg 16, 69120 Heidelberg, Germany  Govert Nijs  Center for
Theoretical Physics, Massachusetts Institute of Technology, Cambridge, MA
02139, USA  Wilke van der Schee  Theoretical Physics Department, CERN, CH-1211
Genève 23, Switzerland  Institute for Theoretical Physics, Utrecht University,
3584 CC Utrecht, The Netherlands

###### Abstract

Emergent bulk properties of matter governed by the strong nuclear force give
rise to physical phenomena across vastly different scales, ranging from the
shape of atomic nuclei to the masses and radii of neutron stars. They can be
accessed on Earth by measuring the spatial extent of the outer skin made of
neutrons that characterises the surface of heavy nuclei. The isotope 208Pb,
owing to its simple structure and neutron excess, has been in this context the
target of many dedicated efforts. Here, we determine the neutron skin from
measurements of particle distributions and their collective flow in
208Pb+208Pb collisions at ultrarelativistic energy performed at the Large
Hadron Collider, which are mediated by interactions of gluons and thus
sensitive to the overall size of the colliding 208Pb ions. By means of state-
of-the-art global analysis tools within the hydrodynamic model of heavy-ion
collisions, we infer a neutron skin $\Delta r_{np}=0.217\pm 0.058$ fm,
consistent with nuclear theory predictions, and competitive in accuracy with a
recent determination from parity-violating asymmetries in polarised electron
scattering. We establish thus a new experimental method to systematically
measure neutron distributions in the ground state of atomic nuclei.

††preprint: CERN-TH-2023-069//MIT-CTP/5558

Understanding the distribution of neutrons within heavy atomic nuclei has
profound implications for our knowledge of the neutron-rich matter that shapes
exotic astrophysical objects such as neutron stars. The neutron skin that
forms on the surface of heavy nuclei, whereby neutrons are located more
diffusely and more on the outside [1, 2], represents in particular a sensitive
probe of the equation of state (EOS) of neutron matter, whose pressure
determines the spatial extent of the neutron distributions. Indeed, nuclear
models predict a strong correlation between the neutron skins of heavy nuclei
and the masses and radii of neutron stars [3, 4].

While proton distributions in nuclei can be determined in a model-independent
way from electron scattering experiments [5], accessing neutron distributions
poses a far greater challenge. As a consequence, we have only limited
experimental constraints on the neutron skin of nuclei, $\Delta r_{np}$,
defined as the difference in root mean square (rms) radii between protons and
neutrons. The doubly-magic nucleus 208Pb ($Z=82$, $N=126$) has both protons
and neutrons filling up their respective shells and represents an optimal
study subject in this context. A recent, precise deduction of the neutron skin
of 208Pb has been achieved by the PREX collaboration [6] from the measurement
of parity-violating asymmetries in polarised electron scattering. On the side
of theory, the first calculation of 208Pb and its neutron skin in the context
of ab initio nuclear theory was also recently performed [7]. These results,
along with information coming from pulsar and gravitational wave observations,
portray a picture of nuclear matter that hints at potential tensions [8, 9,
10].

In this article, we determine the neutron skin of 208Pb from a new type of
probe. We use data collected in 208Pb+208Pb collisions performed at
ultrarelativistic energy at the CERN Large Hadron Collider (LHC). These
collisions produce short-lived quark-gluon plasma [11, 12, 13] (QGP), the hot
phase of quantum chromodynamics (QCD), which behaves like a near-ideal
relativistic fluid [14, 15] before fragmenting into observable particles. In
high-energy scattering, interactions are mediated by gluons, such that the
combined distribution of protons and neutrons (altogether called nucleons)
within the colliding 208Pb ions determines the shape and the size of the
created QGP. Employing the latest advances in simulation and Bayesian
inference tools within the hydrodynamic framework of heavy-ion collisions we
reconstruct the geometry of the QGP by using the detected particle
distributions. In conjunction with the precise knowledge of the proton density
this enables us to place a tight constraint on the neutron skin of 208Pb.

Figure 1: Neutron skin and collective flow in relativistic nuclear collisions.
a: Two ions collide with impact parameter $b=8\,$fm. Both ions are Lorentz-
contracted by a factor $\gamma\approx 2500$, and the relevant dynamics hence
effectively takes place in the transverse plane, $x_{\perp}=(x,\,y)$. b: The
collision deposits energy in the interaction region depending on the extent of
the neutron skin of the 208Pb nuclei. We consider $\Delta r_{np}=0.086\,$fm
(top) and $\Delta r_{np}=0.384$ (bottom). The neutron skin is varied by
keeping the half-width neutron radius, $R_{n}$, constant while changing the
neutron diffuseness, as displayed by the dotted lines (see also Eqn. (2)
below). A larger neutron skin leads to a considerably larger total hadronic
cross section, $\sigma_{\rm tot}$, and the resulting QGP is in addition more
diffuse and less elliptical. c: We show a single QGP evolving hydrodynamically
and being converted into particles (marked in the figure with their respective
symbols) as it cools, while expanding both in $z$ and in the transverse plane.
The observation of millions such events leads to characteristic azimuthal
anisotropies in the momentum distribution of the produced particles, the most
important of which is quantified by the rms value of its second Fourier
component, the elliptic flow $v_{2}\\{2\\}$, which reflects the ellipticity of
the QGP.

### The neutron skin and the quark-gluon plasma -

Our understanding of the QGP formed in 208Pb+208Pb collisions is highly
developed thanks to the wealth of experimental data collected in the past
decade by all LHC experiments, and in particular by the ALICE experiment
dedicated to nuclear physics [16]. Following Fig. 1, in an ultrarelativistic
heavy-ion collision in the lab frame (Fig. 1a), interactions of gluons deposit
energy density in the area of overlap in the so-called transverse plane,
perpendicular to the beam direction (Fig. 1b). The deposition of energy
density depends on the collision’s impact parameter $b$, on the structure of
the colliding nuclei and on the dynamics of the interaction itself.

Phenomenological studies have established a picture where the colliding ions
are treated, in each collision (or event), as a superposition of nucleons that
participate in the interaction. Both boosted nuclei are thus associated with a
profile of matter in the transverse plane, $\mathcal{T_{L,R}}(x_{\perp})$,
given as the sum of their participant nucleon profiles, typically taken as
Gaussians with a width denoted by $w$. The interaction process and the
subsequent energy depositions are then parameterised following some flexible
prescription which can be fine-tuned directly from experimental data. Here we
use a TRENTo-type Ansatz for the energy density of the QGP [17, 18],

$e(x_{\perp})\propto\left(\frac{\mathcal{T}_{L}(x_{\perp}-b/2)^{p}+\mathcal{T}_{R}(x_{\perp}+b/2)^{p}}{2}\right)^{q/p},$
(1)

where $L,R$ denote the two colliding ions, while $p$ and $q$ are model
parameters. As the positions of the participant nucleons shaping the functions
$\mathcal{T}_{L,R}$ are sampled in each collision from the neutron and proton
densities in the ground state of the scattering ions, the energy density
$e(x_{\perp})$ is sensitive to their spatial distribution. This can be seen by
eye in the density plot of Fig. 1b, representing an average energy density
over many collisions. The scenario where the colliding 208Pb nuclei have a
narrower neutron skin leads to a QGP with a sharper profile over the plane and
a higher density peak.

Starting from the initial condition discussed in Fig. 1b, the QGP then evolves
as a relativistic viscous fluid (with transport properties, such as shear and
bulk viscosities, that are also model parameters). For a single event,
snapshots of the hydrodynamic expansion obtained using our hydrodynamic code
are depicted in Fig. 1c. Cooling of the QGP lasts until the confinement
crossover is reached, after which at a fixed switching temperature the fluid
is converted into a gas of QCD resonance states that can further re-scatter or
decay to stable particles. Out of this process, experiments can only detect
final event-by-event stable particle spectra, typically denoted by:

$\frac{d^{3}N_{\rm ch}}{d^{2}{\boldsymbol{p}}_{T}\,d\eta}=\frac{d^{2}N_{\rm
ch}}{dp_{T}\,d\eta}\frac{1}{2\pi}\left(1+2\sum_{n=1}^{\infty}v_{n}\cos
n(\phi-\phi_{n})\right),$

where $\boldsymbol{p}_{T}$ is the transverse momentum, $\eta$ is the particle
pseudorapidity ($\eta\equiv-\ln\tan(\theta/2)$ with $\theta$ the polar angle
in the $(x_{\perp},\,z)$ plane of Fig. 1a), and the subscript ch indicates
that only charged particles are included. We have conveniently decoupled the
spectrum into a distribution of transverse momenta,
$p_{T}\equiv|\boldsymbol{p}_{T}|$, which quantifies the explosiveness of the
QGP expansion, and an azimuthal component developed in Fourier modes, where
$v_{n}$ are the so-called anisotropic flow coefficients that quantify the
anisotropy of the particle emission.

Figure 2: Signature of the neutron skin on bulk particle production in
ultrarelativisitic 208Pb+208Pb collisions. Varying only the neutron skin size
at our optimal parameter settings we show the charged particle multiplicity
(left), the mean transverse momentum (middle) and the elliptic flow as
measured by $v_{2}\\{k\\}$ (right) with a comparison to ALICE data [19, 20]. A
larger neutron skin leads to more collisions, but per collision the
multiplicity is lower at larger centralities. The larger size of the QGP leads
to a reduced transverse momentum on average. Smearing of the elliptical shape
leads to reduced elliptic flow as measured by $v_{2}\\{2\\}$ and
$v_{2}\\{4\\}$. Theoretical error bars are statistical only, experimental
uncertainties include systematics as well.

Experimentally the first step is to sort the collisions in centrality classes
based on the number of particles that they produce, where 0% centrality
corresponds to events with the highest number of particles at almost zero
impact parameter. As a function of centrality one can then measure among
others the distributions of $p_{T}$ and $v_{n}$ coefficients for different
particle species (pions, kaons, protons and more). This generates a wealth of
experimental information from which the hydrodynamic model parameters (here,
we have 26 in total) can be inferred. The central idea of this manuscript is
that of promoting the neutron skin of 208Pb to a model parameter that we
constrain from LHC data.

The neutron skin is introduced by considering variations in the neutron
diffuseness, $a_{n}$, in the two-parameter Fermi distributions that model the
point-neutron and point-proton densities in the colliding 208Pb nuclei:

$\rho_{n,p}(r)\propto\left[1+\exp\left(\frac{r-R_{n,p}}{a_{n,p}}\right)\right]^{-1}.$
(2)

We take $a_{p}=0.448$ fm, $R_{p}=6.680$ fm (corresponding to an rms proton
radius $r_{p}=5.436$ fm), and $R_{n}=6.690$ fm, which is motivated by the
experimental result that the neutron skin is caused by a more diffuse profile
rather than a larger half-width radius [1, 2].

Before proceeding with a full Bayesian analysis we simulate the QGP formation
and evolution for three different values of $\Delta r_{np}$ while keeping all
other model parameters fixed. First, a larger neutron skin leads to a larger
total hadronic cross section, $\sigma_{\rm tot}$ (see Fig. 1b for an increase
from 7.75 to 8.67 b), because it increases the overall number of events
occurring at higher impact parameters.

We follow now Fig. 2, showing experimental and model results for quantities
that characterise the bulk of particle production from the measured spectra.
The larger $\sigma_{\rm tot}$ for the larger neutron skin induces larger
impact parameters at the same centrality. As a consequence, fewer particles
are produced for larger values of $\Delta r_{np}$, as clearly visible in the
total multiplicity in Fig. 2 (left panel). A second effect of a larger skin,
highlighted in Fig. 1b, is that it leads to more diffuse QGP droplets, which
leads to weaker pressure gradients and a slower hydrodynamic expansion. This
translates into a lower average momentum of the detected particles, as seen in
the middle panel of Fig. 2. In addition Fig. 1 shows that a larger neutron
skin reduces the ellipticity of the QGP. This leads to a reduction of the
elliptic flow, measured in experiment as a two-particle azimuthal correlation
($v_{2}\\{2\\}$, the rms value of the distribution of $v_{2}$) or as a four-
particle correlation ($v_{2}\\{4\\}$). Indeed Fig. 2 (right) shows the
expected reduction and moreover we find that a larger neutron skin enhances
the difference between $v_{2}\\{2\\}$ and $v_{2}\\{4\\}$, which corresponds to
larger elliptic flow fluctuations.

Figure 3: Inferred neutron skin and energy-deposition parameters. We show the
posterior distribution of the neutron skin $\Delta r_{np}$, the nucleon width
$w$ and the energy deposition parameters $p$ and $q$, together with their
expectation values (see top) and correlations. Uncertainties correspond to the
standard deviations of the posterior distributions. Especially the $p$
parameter (see Eqn. (1)) is highly anti-correlated with $\Delta r_{np}$, as
both have a strong effect on the centrality dependence of observables (see
also Fig. 2).

### Bayesian inference of the 208Pb neutron skin -

Due to the interplay and cross-correlations among parameters and observables,
constraining the model from experiment requires advanced Bayesian analysis
tools as pioneered in earlier works [21, 15]. Our analysis makes use of 653
data points in 208Pb+208Pb collisions and a single data point (the total cross
section) of proton-nucleus ($p$+208Pb) collisions. We use 3000 design points
for the Gaussian Processes to emulate our collisions as a function of the
26-dimensional parameter space. See the Supplemental Material (SM) for a
specification of all data, parameters and their inferred distributions.

The posterior distributions are displayed in Fig. 3 for a subset of parameters
that correlate highly with $\Delta r_{np}$. These are the parameters appearing
in the energy deposition formula, Eqn. (1), namely, the energy deposition
parameters $p$ and $q$, as well as the nucleon size, $w$. In fact, the $p$
parameter and $\Delta r_{np}$ are the most negatively correlated across our
entire parameter space. This is not surprising, as both parameters strongly
influence the centrality dependence of observables, whereby a larger neutron
skin in particular affects off-central collisions by increasing the total
cross section.

Here we briefly revisit Fig. 2, where the middle curve represents our most
likely value estimate. In the SM we present the full posterior distributions
of our set of 653 data points. There, it can also be seen that the reason for
the mismatch between the computed $\langle p_{T}\rangle$ and experimental data
in Fig. 2 lies in a slight overestimate of yield of protons, which comes with
a larger $p_{T}$. There is also a significant posterior uncertainty in the
anisotropic flow, which is dominated by the emulator uncertainty.

In Fig. 4 we put our new result in context of other state-of-the-art
determinations of the skin of 208Pb. From the posterior distribution we obtain
$\Delta r_{np}=0.217\pm 0.058\,$fm, corresponding to a point-like rms neutron
radius $r_{n}=5.653\pm 0.058$ fm. Our result is compatible with both the ab
initio determination [7] and the PREX result [6], which is competitive in
precision. With regards to the EOS of neutron matter, from the relation
between $\Delta r_{np}$ and the slope parameter, $L$, of the symmetry energy
around the nuclear saturation density [22], we obtain $L=79\pm 39\,$MeV,
representing the first collider-based constraint on this parameter from high-
energy data.

We comment now on the robustness of this result. The total 208Pb+208Pb and
$p$+208Pb cross sections [23, 24] pose important constraints on the neutron
skin. Indeed, excluding these two measurements we obtain $\Delta
r_{np}=0.31\pm 0.10\,$fm, whereas using exclusively these two data points
results in $\Delta r_{np}=0.03\pm 0.12\,$fm. Our result comes hence from
constraints due to a combination of observables, where the cross section
prefers a smaller neutron skin, while other observables prefer a larger value
(this is similar for $w$ [25]). For the first time in Bayesian analyses we
include the $\rho_{2}$ observable [26, 27], a sensitive probe of the initial
conditions [28, 29, 30, 31, 25] which measures the correlation between
$v_{2}\\{2\\}$ and $\langle p_{T}\rangle$. Without this observable, the
analysis yields a consistent result, $\Delta r_{np}=0.243\pm 0.059\,$fm. Also,
as introduced in Ref. [25], we weight the targeted observables according to a
prescription that models unknown theoretical uncertainty with respect to
$p_{T}$-differential observables in particular. Turning this weighting off, we
find a consistent albeit slightly smaller neutron skin, $\Delta
r_{np}=0.160\pm 0.057\,$fm.

Further indication of the robustness of our finding comes from the fact that
targeting a subset of $p_{T}$-integrated-only observables, corresponding to
233 ALICE data points, we obtain $\Delta r_{np}=0.216\pm 0.057\,$fm. This
suggests that the extraction of $\Delta r_{np}$ is likely insensitive to
theoretical uncertainties in the particlisation of the QGP at the switching
temperature [32]. Lastly, we note that our TRENTo Ansatz of Eqn. (1) is very
versatile, and may lead to a relatively conservative estimate of the
uncertainty on $\Delta r_{np}$. Implementing in the future a model of initial
conditions motivated by first-principles arguments and with fewer parameters
[33], may lead to stronger constraints than presented here.

Figure 4: State-of-the-art determinations of the neutron skin of 208Pb. We
show the final likelihood distribution of the neutron skin as determined from
the LHC data as compared to the values obtained experimentally by the PREX
collaboration (including both experimental and theoretical uncertainties in
the extraction) [6] and the estimate of _ab initio_ nuclear theory (with an
error bar corresponding to a 68% credibility interval) [7].

### Future skin determinations at the LHC -

Albeit computationally expensive, it would be interesting to vary the full
neutron profile, by changing $R_{n}$, or via a multi-parameter function as in
Ref. [2]. The SM presents an exploratory study of this kind. Increasing the
neutron half-width radius does not affect the average multiplicity and the
elliptic flow, but leads to a decrease in the mean transverse momentum and a
higher hadronic cross section. We speculate, thus, that a more complete
analysis could eventually lead to a slightly smaller neutron skin.

We expect our analysis to trigger a program of complementary measurements of
skin effects at the LHC. A method pioneered by the STAR collaboration utilises
the photo-production of vector mesons in ultra-peripheral nucleus-nucleus
collisions to infer the average gluon density in the colliding nuclei, and
hence the neutron skins [34]. The extracted skin of 197Au is in good agreement
with nuclear theory predictions [35]. Therefore, the same method could be
exploited at the LHC to perform an independent extraction of the skin of
208Pb.

In addition, the global analysis presented here uses so-called _soft_
observables that depend on particles with transverse momentum of order of the
QCD deconfinement temperature, around 150 MeV. With high-luminosity LHC runs
it may be possible to constrain the neutron skin as well via _hard_
observables, such as high transverse momentum electroweak bosons [36, 37]. The
charge of the produced electroweak bosons can serve as a direct probe of the
number of neutron-neutron interactions. By selecting collisions at relatively
large impact parameter, it is then possible to determine the dominance of
neutrons at the outer edges of the 208Pb nucleus.

It is likely that the nucleus 48Ca and other ions will be collided at the
high-luminosity LHC in the next decade [38]. This will enable an extended
analysis that in particular can be compared with the dedicated CREX
measurement of the neutron skin of 48Ca [39]. Comparing many different
collision systems will furthermore permit us to study ratios of observables
that cancel most of the systematic theoretical uncertainties [40, 41, 42, 43],
leading to improved determinations of $\Delta r_{np}$ across the nuclear
chart.

### Acknowledgments -

We acknowledge discussions with the participants of INT-23-1a, “Intersection
of nuclear structure and high‐energy nuclear collisions”, and the hospitality
of the Institute for Nuclear Theory, Seattle. We thank in particular Rituparna
Kanungo for interesting discussions. We thank Krishna Rajagopal for comments
on the manuscript. G.G. is supported by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) under Germany’s Excellence Strategy EXC
2181/1 - 390900948 (the Heidelberg STRUCTURES Excellence Cluster), SFB 1225
(ISOQUANT) and FL 736/3-1. G.N. is supported by the U.S. Department of Energy,
Office of Science, Office of Nuclear Physics under grant Contract Number DE-
SC0011090.

## References

  * Trzcinska  _et al._ [2001] A. Trzcinska, J. Jastrzebski, P. Lubinski, F. J. Hartmann, R. Schmidt, T. von Egidy, and B. Klos, Phys. Rev. Lett. 87, 082501 (2001).
  * Zenihiro  _et al._ [2010] J. Zenihiro _et al._ , Phys. Rev. C 82, 044611 (2010).
  * Brown [2000] B. A. Brown, Phys. Rev. Lett. 85, 5296 (2000).
  * Horowitz and Piekarewicz [2001] C. J. Horowitz and J. Piekarewicz, Phys. Rev. Lett. 86, 5647 (2001), arXiv:astro-ph/0010227 .
  * De Vries  _et al._ [1987] H. De Vries, C. W. De Jager, and C. De Vries, Atom. Data Nucl. Data Tabl. 36, 495 (1987).
  * Adhikari  _et al._ [2021] D. Adhikari _et al._ (PREX), Phys. Rev. Lett. 126, 172502 (2021), arXiv:2102.10767 [nucl-ex] .
  * Hu  _et al._ [2022] B. Hu _et al._ , Nature Phys. 18, 1196 (2022), arXiv:2112.01125 [nucl-th] .
  * Fattoyev  _et al._ [2018] F. J. Fattoyev, J. Piekarewicz, and C. J. Horowitz, Phys. Rev. Lett. 120, 172702 (2018), arXiv:1711.06615 [nucl-th] .
  * Reed  _et al._ [2021] B. T. Reed, F. J. Fattoyev, C. J. Horowitz, and J. Piekarewicz, Phys. Rev. Lett. 126, 172503 (2021), arXiv:2101.03193 [nucl-th] .
  * Essick  _et al._ [2021] R. Essick, I. Tews, P. Landry, and A. Schwenk, Phys. Rev. Lett. 127, 192701 (2021), arXiv:2102.10074 [nucl-th] .
  * Braun-Munzinger and Stachel [2007] P. Braun-Munzinger and J. Stachel, Nature 448, 302 (2007).
  * Teaney [2010] D. A. Teaney, “Viscous Hydrodynamics and the Quark Gluon Plasma,” in  _Quark-gluon plasma 4_, edited by R. C. Hwa and X.-N. Wang (2010) pp. 207–266, arXiv:0905.2433 [nucl-th] .
  * Gardim  _et al._ [2020] F. G. Gardim, G. Giacalone, M. Luzum, and J.-Y. Ollitrault, Nature Phys. 16, 615 (2020), arXiv:1908.09728 [nucl-th] .
  * Romatschke and Romatschke [2019] P. Romatschke and U. Romatschke,  _Relativistic Fluid Dynamics In and Out of Equilibrium_, Cambridge Monographs on Mathematical Physics (Cambridge University Press, 2019) arXiv:1712.05815 [nucl-th] .
  * Bernhard  _et al._ [2019] J. E. Bernhard, J. S. Moreland, and S. A. Bass, Nature Phys. 15, 1113 (2019).
  * ALICE [2022a] ALICE, (2022a), arXiv:2211.04384 [nucl-ex] .
  * Moreland  _et al._ [2015] J. S. Moreland, J. E. Bernhard, and S. A. Bass, Phys. Rev. C 92, 011901 (2015), arXiv:1412.4708 [nucl-th] .
  * Nijs and van der Schee [2023a] G. Nijs and W. van der Schee, (2023a), arXiv:2304.06191 [nucl-th] .
  * Acharya  _et al._ [2019] S. Acharya _et al._ (ALICE), Phys. Lett. B 793, 420 (2019), arXiv:1805.05212 [nucl-ex] .
  * Acharya  _et al._ [2018] S. Acharya _et al._ (ALICE), JHEP 07, 103 (2018), arXiv:1804.02944 [nucl-ex] .
  * Sangaline and Pratt [2016] E. Sangaline and S. Pratt, Phys. Rev. C 93, 024908 (2016), arXiv:1508.07017 [nucl-th] .
  * Viñas  _et al._ [2014] X. Viñas, M. Centelles, X. Roca-Maza, and M. Warda, Eur. Phys. J. A 50, 27 (2014), arXiv:1308.1008 [nucl-th] .
  * ALICE [2022b] ALICE, (2022b), arXiv:2204.10148 [nucl-ex] .
  * Khachatryan  _et al._ [2016] V. Khachatryan _et al._ (CMS), Phys. Lett. B 759, 641 (2016), arXiv:1509.03893 [hep-ex] .
  * Nijs and van der Schee [2022] G. Nijs and W. van der Schee, Phys. Rev. Lett. 129, 232301 (2022), arXiv:2206.13522 [nucl-th] .
  * Bozek [2016] P. Bozek, Phys. Rev. C 93, 044908 (2016), arXiv:1601.04513 [nucl-th] .
  * ATLAS [2022] ATLAS, (2022), arXiv:2205.00039 [nucl-ex] .
  * Bozek and Mehrabpour [2020] P. Bozek and H. Mehrabpour, Phys. Rev. C 101, 064902 (2020), arXiv:2002.08832 [nucl-th] .
  * Schenke  _et al._ [2020] B. Schenke, C. Shen, and D. Teaney, Phys. Rev. C 102, 034905 (2020), arXiv:2004.00690 [nucl-th] .
  * Giacalone  _et al._ [2021] G. Giacalone, F. G. Gardim, J. Noronha-Hostler, and J.-Y. Ollitrault, Phys. Rev. C 103, 024909 (2021), arXiv:2004.01765 [nucl-th] .
  * Giacalone  _et al._ [2022] G. Giacalone, B. Schenke, and C. Shen, Phys. Rev. Lett. 128, 042301 (2022), arXiv:2111.02908 [nucl-th] .
  * Everett  _et al._ [2021] D. Everett _et al._ (JETSCAPE), Phys. Rev. C 103, 054904 (2021), arXiv:2011.01430 [hep-ph] .
  * Heffernan  _et al._ [2023] M. R. Heffernan, C. Gale, S. Jeon, and J.-F. Paquet, (2023), arXiv:2302.09478 [nucl-th] .
  * Abdallah  _et al._ [2023] M. Abdallah _et al._ (STAR), Sci. Adv. 9, eabq3903 (2023), arXiv:2204.01625 [nucl-ex] .
  * Bally  _et al._ [2023] B. Bally, G. Giacalone, and M. Bender, Eur. Phys. J. A 59, 58 (2023), arXiv:2301.02420 [nucl-th] .
  * Paukkunen [2015] H. Paukkunen, Phys. Lett. B 745, 73 (2015), arXiv:1503.02448 [hep-ph] .
  * Aad  _et al._ [2019] G. Aad _et al._ (ATLAS), Eur. Phys. J. C 79, 935 (2019), arXiv:1907.10414 [nucl-ex] .
  * Citron  _et al._ [2019] Z. Citron _et al._ , CERN Yellow Rep. Monogr. 7, 1159 (2019), arXiv:1812.06772 [hep-ph] .
  * Adhikari  _et al._ [2022] D. Adhikari _et al._ (CREX), Phys. Rev. Lett. 129, 042501 (2022), arXiv:2205.11593 [nucl-ex] .
  * Xu  _et al._ [2021] H.-j. Xu, H. Li, X. Wang, C. Shen, and F. Wang, Phys. Lett. B 819, 136453 (2021), arXiv:2103.05595 [nucl-th] .
  * Jia and Zhang [2023] J. Jia and C. Zhang, Phys. Rev. C 107, L021901 (2023), arXiv:2111.15559 [nucl-th] .
  * Nijs and van der Schee [2023b] G. Nijs and W. van der Schee, SciPost Phys. 15, 041 (2023b), arXiv:2112.13771 [nucl-th] .
  * Jia  _et al._ [2023] J. Jia, G. Giacalone, and C. Zhang, Phys. Rev. Lett. 131, 022301 (2023), arXiv:2206.10449 [nucl-th] .
  * Weil  _et al._ [2016] J. Weil _et al._ (SMASH), Phys. Rev. C 94, 054905 (2016), arXiv:1606.06642 [nucl-th] .
  * Oliinychenko  _et al._ [2020] D. Oliinychenko, V. Steinberg, J. Weil, M. Kretz, J. Staudenmaier, S. Ryu, A. Schäfer, J. Rothermel, J. Mohs, F. Li, H. E. (Petersen), L. Pang, D. Mitrovic, A. Goldschmidt, L. Geiger, J.-B. Rose, J. Hammelmann, and L. Prinz, “smash-transport/smash: Smash-1.8,” (2020).
  * Sjostrand  _et al._ [2008] T. Sjostrand, S. Mrenna, and P. Z. Skands, Comput. Phys. Commun. 178, 852 (2008), arXiv:0710.3820 [hep-ph] .
  * Bally  _et al._ [2022] B. Bally, M. Bender, G. Giacalone, and V. Somà, Phys. Rev. Lett. 128, 082301 (2022), arXiv:2108.09578 [nucl-th] .
  * Cooper and Frye [1974] F. Cooper and G. Frye, Phys. Rev. D 10, 186 (1974).
  * Pratt and Torrieri [2010] S. Pratt and G. Torrieri, Phys. Rev. C 82, 044901 (2010), arXiv:1003.0413 [nucl-th] .
  * Aamodt  _et al._ [2011] K. Aamodt _et al._ (ALICE), Phys. Rev. Lett. 106, 032301 (2011), arXiv:1012.1657 [nucl-ex] .
  * Adam  _et al._ [2016a] J. Adam _et al._ (ALICE), Phys. Rev. Lett. 116, 222302 (2016a), arXiv:1512.06104 [nucl-ex] .
  * Abelev  _et al._ [2013] B. Abelev _et al._ (ALICE), Phys. Rev. C 88, 044910 (2013), arXiv:1303.0737 [hep-ex] .
  * Adam  _et al._ [2016b] J. Adam _et al._ (ALICE), Phys. Rev. C 94, 034903 (2016b), arXiv:1603.04775 [nucl-ex] .
  * Abelev  _et al._ [2014] B. B. Abelev _et al._ (ALICE), Eur. Phys. J. C 74, 3077 (2014), arXiv:1407.5530 [nucl-ex] .
  * Adam  _et al._ [2016c] J. Adam _et al._ (ALICE), Phys. Rev. Lett. 116, 132302 (2016c), arXiv:1602.01119 [nucl-ex] .
  * Adam  _et al._ [2016d] J. Adam _et al._ (ALICE), JHEP 09, 164 (2016d), arXiv:1606.06057 [nucl-ex] .
  * Aaboud  _et al._ [2020] M. Aaboud _et al._ (ATLAS), JHEP 01, 051 (2020), arXiv:1904.04808 [nucl-ex] .
  * Earl and Deem [2005] D. J. Earl and M. W. Deem, Phys. Chem. Chem. Phys. 7, 3910 (2005).
  * Foreman-Mackey  _et al._ [2013] D. Foreman-Mackey, D. W. Hogg, D. Lang, and J. Goodman, Publ. Astron. Soc. Pac. 125, 306 (2013), arXiv:1202.3665 [astro-ph.IM] .
  * ALI [2022] (2022), arXiv:2204.10148 [nucl-ex] .
  * Garcia-Montero  _et al._ [2022] O. Garcia-Montero, J. Staudenmaier, A. Schäfer, J. M. Torres-Rincon, and H. Elfner, Phys. Rev. C 105, 064906 (2022), arXiv:2107.08812 [hep-ph] .

## Supplemental - Bayesian analysis in the _Trajectum_ framework

The _Trajectum_ framework consists of an initial stage, a hydrodynamic stage
and finally a freeze-out to hadrons that are then evolved by the SMASH code
[44, 45, 46]. Here we give a brief summary of all parameters involved, how
they are constrained, and by which experimental data. All parameters are
displayed in boldface. Full details can be found in Ref. [18].

For the initial stage the nucleons are distributed within the colliding nuclei
according to Eqn. 2 with parameter $\boldsymbol{a_{n}}$, whereby each sampled
nucleon pair has a minimum distance of $\boldsymbol{d_{\rm min}}$. In
addition, in Eqn. 2, we consider the half-width radius expanded in spherical
harmonics up to the axial quadrupole deformation,
$R(\theta)=R(1+\beta_{2}Y_{2}^{0}(\theta))$, where $\beta_{2}$ quantifies the
magnitude of the ellipsoidal deformation. Given the structure of 208Pb [47],
we let $\beta_{2}$ fluctuate around zero with a standard deviation
$\boldsymbol{\sqrt{\langle\beta_{2}^{2}\rangle-\langle\beta_{2}\rangle^{2}}}$.
The nucleons that collide are determined by the measured $\sigma_{\rm NN}$
cross section as in [17, 25] and are called participants. Each participant
then consists of $\boldsymbol{n_{c}}$ constituents, each associated with a
transverse Gaussian profile with width
$v=0.2\,\text{fm}+\boldsymbol{\chi_{\textbf{struct}}}(\boldsymbol{w}-0.2\,\text{fm})$.
The center coordinates of the nucleon constituents are distributed according
to a Gaussian distribution. This leads to an average Gaussian nucleon profile
with a width $\boldsymbol{w}$. Superimposition of the nucleon constituent
profiles then leads to the thickness functions $\mathcal{T}$ in Eqn. 3. The
normalisation of each Gaussian profile fluctuates and is equal to
$\boldsymbol{N}\gamma/\boldsymbol{n_{c}}$, where $\gamma$ is sampled from a
gamma distribution with mean 1 and width
$\boldsymbol{\sigma_{\textbf{fluct}}}\sqrt{\boldsymbol{n_{c}}}$. The final
energy density is then given by Eqn. 3 (main text) with parameters
$\boldsymbol{p}$ and $\boldsymbol{q}$.

For a time $\boldsymbol{\tau_{\textbf{hyd}}}$ this energy density then either
evolves via free streaming or according to a holographic prescription that
starts assuming viscous hydrodynamics (see [18]). The way this is chosen is
with a continuous parameter $\boldsymbol{r_{\textbf{hyd}}}$ between 0 and 1
which interpolates between free streaming ($\boldsymbol{r_{\textbf{hyd}}}=0$)
and the holographic prescription ($\boldsymbol{r_{\textbf{hyd}}}=1$).

Next the stress-energy tensor is evolved according to second order
hydrodynamics with 9 parameters that determine the transport coefficients,
namely $\boldsymbol{\overline{\eta/s}}$,
$\boldsymbol{(\eta/s)_{\textbf{slope}}}$,
$\boldsymbol{(\eta/s)_{\delta\textbf{slope}}}$,
$\boldsymbol{(\eta/s)_{0.8\,\textbf{GeV}}}$,
$\boldsymbol{(\zeta/s)_{\textbf{max}}}$,
$\boldsymbol{(\zeta/s)_{\textbf{m}\times\textbf{w}}}$,
$\boldsymbol{(\zeta/s)_{T_{0}}}$, $\boldsymbol{\frac{\tau_{\pi}sT}{\eta}}$ and
$\boldsymbol{\frac{\tau_{\pi\pi}}{\tau_{\pi}}}$. Of these the first four
determine the shear viscosity over the entropy density, $\eta/s$, as a
function of the temperature, by fixing the average value, the slope between
temperatures of 150 and 300 MeV, the difference in slope at higher
temperatures and its value at 800 MeV and higher. The next three determine the
maximum of the bulk viscosity over the entropy density, $\zeta/s$, the width
of its peak times the maximum, and the location of the maximum, respectively.
The last two parameters are dimensionless combinations of first and second
order transport coefficients, whereby in particular the first determines the
relaxation rate towards first order viscous hydrodynamics. In the hydrodynamic
phase the equation of state can vary with the parameter
$\boldsymbol{a_{\textbf{EOS}}}$, which varies $a_{n}$ in the EOS prescription
of [15] (not to be confused with the parameter $\boldsymbol{a_{n}}$ discussed
in the main text) while keeping the degrees of freedom at high temperatures
fixed.

At the switching temperature $\boldsymbol{T_{\textbf{switch}}}$ the fluid is
frozen out to hadrons according to the Cooper-Frye procedure [48] with the
Pratt-Torrieri-Bernhard (PTB) prescription for viscous corrections [49, 15].
Lastly, the hadrons are evolved with the SMASH code, whereby we scale all
interaction cross sections by a factor $\boldsymbol{f_{\textbf{SMASH}}}$.

Finally, in the analysis the number of events can vary with a normalising
parameter $\boldsymbol{cent_{\textbf{norm}}}$, which is also the only
parameter that has a non-trivial prior likelihood distribution, namely a
Gaussian of width 1%.

Figure 5: Correlation matrix among all 26 model parameters. Detailed
information about the prior ranges is provided in Tab. 1.  Figure 6: Neutron
density measured in polarised proton scattering from Zenihiro et al [2]
compared to that extracted from LHC data. Since we use a single parameter
variation ($a_{n}$ in Eqn. 2) our profile does not vary in the central region.
Nevertheless, around the skin at $r\approx 7\,$fm our likelihood band is
consistent with [2]. The accurately known proton densities (green) are also in
good agreement.

We then compare the output of the code with experimental data. In this work we
use the hadronic cross section measurements for 208Pb+208Pb and $p$+208Pb
collisions [23, 24], the charged particle yields at 2.76 [50] and
$5.02\,\text{TeV}$ [51], the identified particle yields $dN_{\text{ch}}/dy$
and mean transverse momentum $\langle p_{T}\rangle$ for pions, kaons and
protons at $2.76\,\text{TeV}$ [52], as well as unidentified transverse energy
$E_{T}$ [53] and fluctuations of mean $p_{T}$ [54] at $2.76\,\text{TeV}$. We
also include the integrated anisotropic flow coefficients $v_{2}\\{2\\}$,
$v_{2}\\{4\\}$, $v_{3}\\{2\\}$ and $v_{4}\\{2\\}$ at both 2.76 and
$5.02\,\text{TeV}$ [55] and $p_{T}$-differential observables with bin
boundaries at $(0.25,0.5,0.75,1.0,1.4,1.8,2.2,3.0)\,\text{GeV}$. In
particular, this includes spectra for pions, kaons and protons at
$2.76\,\text{TeV}$ [52], as well as $v_{2}\\{2\\}(p_{T})$ for pions, kaons and
protons, and $v_{3}\\{2\\}(p_{T})$ for pions (these data are only available
for $p_{T}>0.5\,$GeV) [56]. Finally we also include “statistically difficult”
observables being the $\rho_{2}(v_{2}\\{2\\},\langle p_{T}\rangle)$,
$NSC(2,3)$ and $NSC(2,4)$ correlators, where $NSC(i,j)$ measures the
normalised correlations between $v_{i}^{2}$ and $v_{j}^{2}$ [27, 57].

The posterior for our parameters is then given by Bayes’ formula

$\mathcal{P}(\boldsymbol{x}|\boldsymbol{y}_{\exp})=\frac{e^{-\Delta^{2}/2}}{\sqrt{(2\pi)^{n}\det\left(\Sigma(\boldsymbol{x})\right)}}\mathcal{P}(\boldsymbol{x})$
(3)

with $\mathcal{P}(\boldsymbol{x})$ the (flat) prior probability density and
where

$\Delta^{2}=\left(\boldsymbol{y}(\boldsymbol{x})-\boldsymbol{y}_{\rm
exp}\right)\cdot\Sigma(\boldsymbol{x})^{-1}\cdot\left(\boldsymbol{y}(\boldsymbol{x})-\boldsymbol{y}_{\rm
exp}\right),$

with $\boldsymbol{y}(\boldsymbol{x})$ the predicted data for parameters
$\boldsymbol{x}$, $\boldsymbol{y}_{\rm exp}$ the $n$ experimental data points
and $\Sigma(\boldsymbol{x})$ is the sum of the experimental and theoretical
covariance matrices. The covariance matrices are constructed as in [15].

The standard procedure is then to run the model at a number of design points
in the parameter space as determined by a latin hypercube and use those design
points to construct an emulator for $\boldsymbol{y}(\boldsymbol{x})$ to
evaluate Eqn. (3) using the parallel tempered emcee code [58, 59]. In this
work we used 3000 design points whereby each design point has 1M initial
stages of which we evolve 18k using hydrodynamics and finally simulate about
100k SMASH events to get high statistics even for ultracentral collisions.
Every 1 in 15 design points uses 10 times more statistics to allow for
“statistically difficult” observables that are then emulated using 200 design
points.

The final posterior distributions and their correlations for the standard
settings of our fitting procedure are displayed in Fig. 5. We refer to the
main text of the paper for details on variations in these settings.

parameter | lower | upper | MAP value
---|---|---|---
Norm [2.76 TeV] | 10 | 32 | 17.9
$w$ (fm) | 0.4 | 1 | 0.608
centnorm (%) | 96 | 104 | 99.6
Norm [5.02 TeV] | 10 | 32 | 22.1
$\sigma_{\rm fluct}$ | 0.1 | 1 | 0.463
$p$ | -0.4 | 0.4 | -0.0526
$q$ | 0.8 | 1.6 | 1.21
$d_{\rm min}$ (fm) | 0 | 1.5 | 0.298
$T_{\rm switch}$ (MeV) | 144 | 162 | 154
$n_{\rm c}$ | 1 | 8 | 2.88
$\chi_{\rm struct}$ | 0 | 1 | 0.932
$\tau_{\rm hyd}$ (fm/$c$) | 0.1 | 0.8 | 0.397
$(\eta/s)_{\rm ave}$ | 0.1 | 0.3 | 0.191
$(\eta/s)_{\rm slope}$ (GeV-1) | -1 | 2 | 0.855
$(\eta/s)_{\delta\rm slope}$ (GeV-1) | -1 | 1 | -0.393
$(\eta/s)_{\rm 0.8~{}GeV}$ | 0.08 | 0.4 | 0.344
$(\zeta/s)_{\rm max}$ | 0 | 0.12 | 0.072
$(\zeta/s)_{\rm max\times width}$ | 0 | 0.01 | 0.00865
$(\zeta/s)_{T_{0}}$ | 0.14 | 0.5 | 0.440
$\tau_{\pi}T$ | 1 | 10 | 4.06
$\tau_{\pi\pi}/\tau_{\pi}$ | 0.8 | 6 | 2.52
$r_{\rm hyd}$ | 0 | 1 | 0.798
$a_{n}$ (fm) | 0.45 | 0.75 | 0.612
$\sqrt{\langle\beta_{2}^{2}\rangle-\langle\beta_{2}\rangle^{2}}$ | 0 | 0.1 | 0.0217
$a_{\rm EOS}$ | -9.5 | -8.4 | -9.35
$f_{\rm SMASH}$ | 0.6 | 1.2 | 0.953
Table 1: List of all model parameters constituting the hydrodynamic model used
in this paper. A total of 26 parameters are employed. For each parameter, we
display the lower and upper boundaries of their prior distributions. The MAP
values represent the individual combination of parameters that provides the
best fit of the experimental data.

In Fig. 6 we compare the resulting posterior likelihood distributions for the
proton and neutron densities to measurements, as presented in [2]. Unlike the
analysis of [2], we vary only $a_{n}$, so that consequently our neutron
density is much more restricted and the proton density is fixed. Nevertheless,
in the region where our variation is relevant (i.e., where the diffuseness of
the neutron skin matters) the results of the two methods agree remarkably well
both in values and in uncertainties.

It is an important question what could happen when varying the neutron profile
with more parameters, such as done in [2]. An indication can be found in Fig.
7, where we vary $R_{n}$ in the Woods-Saxon profile such that the total
$\Delta r_{\rm np}$ variation matches Fig. 2 in the main text of the paper. It
can be seen that the average multiplicity and the elliptic flow remain
unaffected, while a smaller $R_{n}$ results in a larger mean transverse
momentum. This is in agreement with the impact of this parameter on the number
of participant nucleons and initial energy density eccentricity, which we
verified to be also unaffected (note, however, that a smaller nucleus leads to
a higher number of binary collisions). It is important that increasing $R_{n}$
in this figure increases as well the total hadronic cross section from $7.94$
to $8.40\,$b, so that even if the multiplicity and the anisotropic flow remain
unaffected, a potential Bayesian analysis would penalize the large $R_{n}$ by
the measurement of $7.67\,$b [25, 60]. Based on this figure a somewhat smaller
$\Delta r_{\rm np}$ would be possible, but perhaps with a larger bulk
viscosity to compensate for the higher mean transverse momentum. A full
analysis is however beyond the scope of this work.

Finally, in Fig. 8 we show the posterior results for all the considered
observables. Particularly important in relation to the main text is that the
mean transverse momentum for identified pions, kaons and protons have an
excellent description (row 4, column 3) as opposed to the charged hadron mean
transverse momentum in the main text. The reason here is that our model
overestimates the number of protons (row 5, column 1) by about 10%. Given the
high proton $p_{T}$, this small excess leads to the overestimate of
experimental data observed in Fig. 2 of the main text. This issue is mainly
due to effects in the hadronic phase, such as proton-antiproton annihilation
and regeneration (see e.g. [61]), whose detailed inclusion is beyond the scope
of _Trajectum_.

Figure 7: We vary $R_{n}$ such that the total $\Delta r_{\rm np}$ variation
matches Fig. 2 in the main text, keeping all other parameters fixed. The size
itself has a visible effect on the mean transverse momentum, but little effect
on either multiplicity or anisotropic flow. Figure 8: We show all the
observables used in our analysis from measurements by the ALICE collaboration
[50, 51, 52, 53, 54, 55, 52, 56] and the ATLAS collaboration [27, 57], with
the exception of the total hadronic cross sections (see [25]). Theoretical
uncertainties come from the posterior distribution from Fig. 5, which includes
an emulator uncertainty that is dominant for most flow observables.


# Semi-parametric Image Inpainting

Karim Iskakov
MIPT
<EMAIL_ADDRESS>
###### Abstract

This paper introduces a semi-parametric approach to image inpainting for
irregular holes. The nonparametric part consists of an external image
database. During test time database is used to retrieve a supplementary image,
similar to the input masked picture, and utilize it as auxiliary information
for the deep neural network. Further, we propose a novel method of generating
masks with irregular holes and present public dataset with such masks.
Experiments on CelebA-HQ dataset show that our semi-parametric method yields
more realistic results than previous approaches, which is confirmed by the
user study.

(a) Original

(b) Masked

(c) Telea

(d) PConv

(e) Sim-0

(f) Sim-1 (ours)

(g) Most similar

Figure 1: Comparison between (c) Telea, (d) PConv, (e, f) our model with 0 and
1 similar image as auxiliary input. (a, b) original and masked input, (g) most
similar image which is fed to Sim-1. For details — zoom in.

##  1 Introduction

Inpainting is an important computer vision task, where the goal is to restore
masked parts of an image. Inpainting finds wide application in image
processing problems like unwanted content removal, red-eye fixing [30] and
even eye opening [4].

Non-learnable methods that use statistics of not masked pieces of the image,
like PatchMatch [1], produce good inpainting results on images with monotonous
or repetitious content (sceneries, textures, _etc_.). In many ways, this
happens due to the fact, that human eye is not sensitive to little
discrepancies in images with such content. But inconsistencies in pictures of
special domains, like face photos, are very quickly detected by the human eye.
This happens because of the so-called ”uncanny valley” [17] effect.

But recently, deep convolution networks (DCNs) have managed to produce
photorealistic results in image inpainting [10, 16]. The main difference
between DCNs and previous methods is that DCNs learn semantic prior from
training data in an end-to-end manner. All the knowledge of DCN is contained
in the learned model’s parameters and it’s convolution architecture, which is
a good prior for real-world images, as it has been shown in [27]. But, what if
supply DCN with accessory information during inference?

_E.g_., for an artist it’s easier to draw a portrait of the person, while that
person is sitting in front of him, instead of drawing whole painting from
memory. An artist can refer to the origin to refine details or to pick correct
color. So, we decided to provide DCN with such ability.

In this paper, we focus on image inpainting with auxiliary images. Similar
images as auxiliary input can help modern deep neural networks in producing
detailed outputs by copying and fitting separate pieces of similar pictures.
We suppose that stage of finding similar images is not a barrier, because of
rapid progress in large-scale content-based image retrieval [32] and high
availability of a large number of images online.

Another problem which slows down the appearance of inpainting solutions in
real-life applications, like smart image editing, is that current research in
inpainting is focused on masks with regular forms. Often masks are assumed to
be fixed-size squares located in the center of the image [20, 29].
Fortunately, in the recent paper Liu _et al_. [16] proposed method, how to
generate irregular masks using two consecutive video frames. However, their
dataset has a critical disadvantage — there is nothing ”human” in the utilized
source of irregular patterns. Also, this dataset has unnatural sharp crops
close to the borders of masks and it’s not publicly available.

We propose our irregular mask dataset based on the ”Quick, Draw!” dataset [7]
(a collection of 50 million human drawings in vector format). We think that
the dataset of human drawings is a good source for irregular forms.
Combination of strokes from ”Quick, Draw!” dataset is effectively infinite,
that’s why our method can be used as an inexhaustible mask generator.
Moreover, the generation process is fully customizable. With bravery, we call
it ”Quick Draw Irregular Mask Dataset” (QD-IMD).

In summary, our contributions are as follows:

  * •

we propose the use of auxiliary images in inpainting task and show that it
significantly boosts visual quality

  * •

we propose publicly available irregular mask dataset, whose source of
irregular patterns is human drawings; introduced mask generation process is
fully customizable and effectively inexhaustible

##  2 Related Work

Non-learning models often use interpolation methods that rely on pixels of the
remaining not-masked parts of an input image. Alexandru Telea developed
inpainting technique based on the fast marching [26] which is now the default
inpainting method in OpenCV [2]. Another popular nonparametric method
PatchMatch [1] iteratively searches for the most suitable patches to fill in
the holes. It is one of the state-of-the-art approaches in such class of
algorithms.

Deep learning based approaches often use encoder-decoder architecture. Pathak
_et al_. [20] used inpainting task for feature learning with proposed so-
called Content Encoders. Yang _et al_. [29], additionally to Content Encoders,
uses refinement texture network as a postprocessing stage. Another popular
architecture for inpainting is Generative Adversarial Network [6]. GAN-like
architectures managed to solve some problems with blurry and unrealistic
outputs [18, 4, 11].

The recent paper by Liu _et al_. [16] showed stunning results, archiving
state-of-the-art in image inpainting for irregular masks. Authors use
modification of common convolution layer — partial convolution. Such
convolution condition output on valid cells in the mask.

In our research, we use auxiliary images retrieved from the external database.
But, an idea of using an external database of images in the inpainting task is
not novel. Hays and Efros [8] used a large collection of images to complete
missing parts. Another usage of feeding additional data to a network was
effectively applied to the closed-to-open eye inpainting in natural pictures
[4]. Dolhansky _et al_. use an extra photo of the same person with opened eyes
to consistently inpaint given image. Also, such an approach was successfully
applied to image synthesis from semantic maps [21]. Authors combine the
parametric and nonparametric techniques, where the nonparametric component is
a memory bank of image segments constructed from a training set of images.

##  3 Method

In the classical problem statement of inpainting, masked image and mask are
used as input data. Many algorithms use only this information for
reconstruction, but in addition, we can retrieve some useful data from the
external database to help the algorithm to fill in the missing parts. We’re
sure that such problem setup is reasonable, because of high availability of a
variety of images online and rapid progress in image retrieval techniques.

In our approach, we propose a UNet-like [22] neural network similar to [16]
(but with simple convolution layers) which takes masked image, mask and fixed
number of similar images as input ($1$ in our case).

Figure 2: Schematic representation of our method at test time. In the
beginning Original masked image is processed by fully-parametric inpainting
method to produce Coarse inpainting approximation. Then it’s fed to the
feature extractor and resulting Feature vector is used to retrieve K most
similar images from the image database. In the end, coarse approximation and
retrieved similar images are fed to our model Sim-K to get final Output.

###  3.1 Inpainting with auxiliary examples

Inpainting task can be formally stated like this: given a masked image
$I_{in}$ and corresponding mask $M$, one should reconstruct the initial image
$I_{gt}$.

In our approach we additionally find $k$ similar to $I_{in}$ images
$\\{I^{sim}_{1},\ldots,I^{sim}_{k}\\}$ in the external database, and then use
them as extra information.

###  3.2 Network design

Following [16] we build similar UNet-like architecture, but instead of using
partial convolutions we utilize simple ones. As input we stack masked image
$I_{in}$, corresponding mask $M$ and $k$ similar image
$\\{I^{sim}_{1},\ldots,I^{sim}_{k}\\}$. As a result we obtain a tensor with
$3+1+3\times k$ channels.

Unlike [16] our input image size is $256\times 256$ (due to the lack of
resources), hence the network’s depth is less by one. We don’t use batch
normalization layers, because they neither accelerate the convergence nor
improve final quality in our case.

###  3.3 Similar images retrieval

To retrieve similar images from the database we use kNN-classifier with cosine
similarity on images’ descriptors. As descriptors we use last convolution
layer features from VGG-16 [24] trained on ImageNet [3].

The main difficulty is that input images are corrupted by masks and it’s
merely impossible to directly extract proper features from them. That’s why we
use a two-step approach for retrieval: 1) inpaint input masked image with a
pre-trained fully-parametric model, 2) find an image in a database similar to
the inpainted image. As a pre-trained model, we used an architecture proposed
in [16].

During testing, we use described two-step approach, but it’s problematically
to use it during training due to its long running time. So, for training phase
we precalculated top $k$ similar images for all images in the training
dataset, using not-masked images for feature extraction. It’s not completely
fair, but training time reduces dramatically.

###  3.4 Loss functions

We follow [16] and use same objectives. Let’s say that model’s output is
$I_{out}$, than per-pixel losses are defined as
$L_{hole}={\lVert(1-M)\odot(I_{out}-I_{gt})\rVert}_{L_{1}}$ and
$L_{valid}={\lVert M\odot(I_{out}-I_{gt})\rVert}_{L_{1}}$.

For calculating perceptual losses we define $I_{comp}$ which is same as
$I_{out}$, but with not-masked pixels explicitly filled with ground truth. We
calculate perceptual loss [5] for these two outputs $L^{out}_{perc}$ and
$L^{comp}_{perc}$ using features from VGG-16 pool1, pool2 and pool3 layers.

Than we calculate two style losses [5]: $L^{out}_{style}$ and
$L^{comp}_{style}$. We use same layers’ outputs from VGG-16 as for perceptual
loss.

And the last component is TV-loss [12] $L_{tv}$ calculated for $I_{comp}$ with
the $1$-pixel dilation of the hole region.

For the total loss, we sum up all the components with weights. In all
experiments we use weights proposed in [16]:

$\begin{split}L_{total}=L_{valid}+6L_{hole}+0.1L_{tv}+\\\
0.05(L^{out}_{perc}+L^{comp}_{perc})+\\\
120(L^{out}_{style}+L^{comp}_{style})\end{split}$ (1)

##  4 Quick Draw Irregular Mask Dataset

Many recent approaches focus on rectangular shaped holes, often assumed to be
the center of the image. These limitations are absolutely not practical
because we often need to inpaint something with irregular form. That’s why we
need a dataset with masks of irregular forms.

Liu _et al_. [16] proposed such a dataset, where the source of irregular
patterns were the results of occlusion/dis-occlusion mask estimation method
between two consecutive frames for videos described in [25]. That work showed
good results in inpainting, but we think their irregular mask dataset has some
weaknesses:

  * •

There is nothing ”human” in generating such masks

  * •

Masks often have sharp edges because of rough crops close to borders

  * •

It’s not public (though authors claimed, they were going to release it)

We decided to fight these problems and generated Quick Draw Irregular Mask
Dataset (QD-IMD). Our dataset is based on Quick Draw dataset [7] (a collection
of 50 million human drawings). Our hypothesis is that a combination of strokes
drawn by a human hand is a good source of patterns for irregular masks.

(a) Examples of masks from [16] irregular mask dataset

(b) Examples of masks from our QD-IMD

Figure 3: Random masks from (a) [16] irregular mask dataset and from (b) our
QD-IMD. As it can be seen, masks from our dataset are more natural and smooth.

All the parameters of the generation process (number of strokes per image,
line width, etc.) are customizable. The number of possible masks is
effectively infinite, and the dataset can be used as an inexhaustible mask
generator. Algorithm 1 shows details of the generation process. Note, that we
make a central crop at the end of the procedure to get masks with holes, which
touches the boundaries of the image.

1 Randomly choose number of strokes for mask from $\mathcal{N}(4,2)$

2 Randomly sample strokes from ”Quick, Draw” dataset

3 forall  _strokes_ do

4 sample line width (px) from $U(5,15)$

5 draw it on the canvas

6

7 end forall

8Randomly sample upscale rate from $U(1.0,1.5)$ and upscale canvas
correspondingly

9 Make central crop of target shape $256\times 256$

10 Binarize resulting mask

11

Algorithm 1 How QD-IMD is generated

All code and $256\times 256$ dataset with $50$k training and $10$k testing
masks can be found here: https://github.com/karfly/qd-imd.

| $[0.01,0.1]$ | $(0.1,0.2]$ | $(0.2,0.3]$ | $(0.3,0.4]$ | $(0.4,0.5]$ | $(0.5,0.6]$
---|---|---|---|---|---|---
$L_{1}$(Telea)(%) | $28.51\pm 6.80$ | $28.47\pm 6.87$ | $27.93\pm 7.04$ | $28.37\pm 6.83$ | $27.07\pm 6.17$ | $25.48\pm 4.25$
$L_{1}$(PConv)(%) | $0.60\pm 0.15$ | $0.93\pm 0.30$ | $1.51\pm 0.62$ | $3.05\pm 0.89$ | $3.99\pm 1.07$ | $5.14\pm 1.12$
$L_{1}$(Sim-0)(%) | $0.87\pm 0.19$ | $1.20\pm 0.30$ | $1.75\pm 0.51$ | $2.86\pm 0.66$ | $3.46\pm 0.88$ | $4.92\pm 1.14$
$L_{1}$(Sim-1)(%) | $\mathbf{0.55\pm 0.17}$ | $\mathbf{0.89\pm 0.28}$ | $\mathbf{1.41\pm 0.48}$ | $\mathbf{2.52\pm 0.66}$ | $\mathbf{3.12\pm 0.83}$ | $\mathbf{4.41\pm 1.05}$
PSNR(Telea) | $9.12\pm 2.09$ | $9.15\pm 2.28$ | $9.38\pm 2.54$ | $9.13\pm 1.88$ | $9.49\pm 1.81$ | $9.91\pm 1.41$
PSNR(PConv) | $\mathbf{36.94\pm 3.16}$ | $\mathbf{32.67\pm 2.93}$ | $28.84\pm 3.38$ | $23.38\pm 2.86$ | $21.54\pm 2.55$ | $20.37\pm 2.13$
PSNR(Sim-0) | $35.34\pm 2.77$ | $31.52\pm 2.58$ | $28.14\pm 2.80$ | $24.02\pm 2.13$ | $22.76\pm 2.21$ | $20.60\pm 2.29$
PSNR(Sim-1) | $36.70\pm 3.26$ | $32.41\pm 2.79$ | $\mathbf{28.85\pm 2.88}$ | $\mathbf{24.58\pm 2.40}$ | $\mathbf{23.30\pm 2.40}$ | $\mathbf{21.24\pm 1.91}$
SSIM(Telea) | $0.369\pm 0.074$ | $0.371\pm 0.079$ | $0.379\pm 0.092$ | $0.375\pm 0.066$ | $0.365\pm 0.065$ | $0.364\pm 0.025$
SSIM(PConv) | $\mathbf{0.986\pm 0.006}$ | $\mathbf{0.969\pm 0.013}$ | $\mathbf{0.943\pm 0.023}$ | $0.884\pm 0.030$ | $0.850\pm 0.034$ | $\mathbf{0.804\pm 0.044}$
SSIM(Sim-0) | $0.981\pm 0.008$ | $0.961\pm 0.014$ | $0.932\pm 0.024$ | $0.877\pm 0.033$ | $0.849\pm 0.036$ | $0.783\pm 0.043$
SSIM(Sim-1) | $0.985\pm 0.007$ | $0.967\pm 0.013$ | $0.939\pm 0.023$ | $\mathbf{0.886\pm 0.033}$ | $\mathbf{0.858\pm 0.037}$ | $0.801\pm 0.047$
IScore(Telea) | $1.389\pm 0.934$ | $1.397\pm 0.958$ | $1.399\pm 0.976$ | $1.399\pm 1.063$ | $1.492\pm 1.183$ | $2.276\pm 1.185$
IScore(PConv) | $1.316\pm 0.914$ | $1.358\pm 0.970$ | $1.402\pm 1.069$ | $1.410\pm 1.038$ | $1.742\pm 1.394$ | $\mathbf{1.387\pm 0.800}$
IScore(Sim-0) | $1.314\pm 0.921$ | $\mathbf{1.353\pm 0.975}$ | $\mathbf{1.368\pm 1.057}$ | $\mathbf{1.367\pm 1.029}$ | $1.391\pm 0.990$ | $1.600\pm 0.578$
IScore(Sim-1) | $\mathbf{1.308\pm 0.908}$ | $1.353\pm 0.979$ | $1.370\pm 1.058$ | $1.426\pm 1.129$ | $\mathbf{1.263\pm 0.927}$ | $1.782\pm 0.667$
Table 1: Quantitative comparison results on 6 buckets, split according to the
hole-to-image area ratio. Our model Sim-1 outperforms other methods comparing
with $L_{1}$ distance and, according to other metrics, our model works better
when hole-to-image area ratio is bigger.

##  5 Experiments

###  5.1 Dataset

We conduct all experiments on CelebA-HQ [13] dataset downsampled to $256\times
256$ size. Faces in this dataset are aligned, so that eyes, mouth and other
face parts are located on the same fixed positions. We randomly split it into
training ($27$k images) and testing ($3$k images) sets.

Irregular masks for training we generate on the fly using our QD-IMD mask
generator. For testing, we sampled $3$k random masks from QD-IMD test dataset.

We don’t use any augmentations.

###  5.2 Training details

We initialize weights as proposed in [9]. For optimization we use Adam [14]
with $0.0001$ learning rate and batch size $12$. All models were trained on
two NVidia Tesla K40m GPU (12 Gb) until convergence. Training took about 2
days per one model. As a framework we use PyTorch [19].

###  5.3 Comparisons

We compare 4 methods:

  1. 1.

Telea: non-learning based approach proposed in [26]. This method is based on
the fast marching technique, which evaluates masked pixels using weighted
averages of already estimated pixels in one pass.

  2. 2.

PConv: Method proposed by Liu _et al_. in [26]. It’s a UNet-like neural
network, which uses partial convolutions instead of common ones.

  3. 3.

Sim-0: Our method with $0$ similar images given.

  4. 4.

Sim-1: Our method with $1$ similar image given.

We include Telea in the comparison list, because it is best non-learning based
approach available in OpenCV [2]. PConv is a state-of-the-art inpainting
method, according to results in [26]. Sim-0 is included into comparison models
to clearly demonstrate how visual quality boosts because of using auxiliary
information (Sim-1).

###  5.4 Quantitative evaluation

There is still no good metric for numerical evaluation of image inpainting
because the task is ill-posed and has many solutions. Following previous
papers [31, 29, 16] we calculate 4 metrics to compare methods: 1)
$L1$-distance, 2) PSNR, 3) SSIM [28] and 4) the inception score (IScore) [23].

Table 1 shows quantitative results on 6 buckets, split according to the hole-
to-image area ratio. You can see that our method Sim-1 outperforms other
models comparing with $L_{1}$ distance. According to other metrics, our model
works better when hole-to-image area ratio is bigger.

###  5.5 User study

In addition to quantitative metrics, we conducted the side-by-side user study
experiment. We randomly chose $30$ images from testing dataset, manually
created masks and processed them with $4$ models: Telea, PConv, Sim-1, Sim-0.
Then we constructed an online survey with $30$ questions, where a user had to
answer which picture seemed most realistic for him. Besides answers,
corresponding to all models, there was an available option — ”Can’t decide”.

$137$ users took part in our experiment. For every question, we calculated the
win ratio of each model. Results can be seen in the box plot figure 4. On
average our model (Sim-1) was chosen by users in $51\%$ of cases. The second
most chosen model was PConv with $31\%$ mean win ratio.

In the table 2 we show the overall number of wins of every model.

Method | Wins
---|---
Telea | $0/30$
PConv | $11/30$
Sim-0 | $4/30$
Sim-1 | $\mathbf{15/30}$
Table 2: Number of wins per each model in user study. Our model Sim-1 wins in
most of questions. Figure 4: Box plot with win ratio results for each model.
Our model Sim-1 is chosen much more frequently, than others.

###  5.6 Different images as auxiliary input

To ensure, that Sim-1 model actually uses the auxiliary image as useful extra
information for reconstruction, we passed different non-relevant images as
auxiliary input.

First with passed not similar to the input image, but a random picture from
the database. In many cases, the output changed a bit, adopting features of
the given random picture. You can see an example at figure 5 (a).

Also, we experimented with noise images as auxiliary images. The neural
network tries to copy the high-frequency structure of noise to the output, and
it results in bad inpainting outcome 5 (b).

These experiments visually show that auxiliary input contributes a lot to the
resulting inpainting output.

###  5.7 Failure cases

Our model Sim-1 is not perfect and sometimes it fails to produce reasonable
results. In this subsection, we explore some failure cases and try to
understand why it happens.

Sim-1 model’s failures can be divided into three big categories: blurry
results, unrealistic details and inconsistent/asymmetrical outputs. Several
examples for each category can be seen in the figure 7.

Blurriness is a common problem in ill-posed generation tasks. In many cases,
blurry results appear because of per-pixel loss terms. GANs successfully deal
with this problem, because it’s easy for discriminator to distinguish blurry
outputs from real.

The $3$-rd and $4$-th rows in the figure 7 show examples of unrealistic
results. _E.g_. in the $4$-th row you can see a creepy eye drawn above the
hair. We think this effect appears because of lack of variety in the training
dataset. For a neural network it’s more difficult to learn context utilization
for inpainting, than just spatial information about face parts’ locations.

And the last, but not the least issue is related to inconsistent outputs. In
the last row in the figure 7, form and color of the inpainted eye are
convenient, but the neural network didn’t manage to copy makeup style from the
not masked eye. We suppose this problem can be solved by using better image
retrieval and using global adversarial loss [15].

##  6 Conclusions and Future Work

We found out that additionally feeding similar images to the input of the
neural network can be effective in inpainting task. But there are still many
dimensions of improvement of such an idea.

In our paper, we chose the simplest network’s architecture as a proof-of-
concept. We’re sure, that better results can be obtained with more appropriate
architecture. _E.g_. it’s a good idea to hybridize our approach with [16]
model that utilizes partial convolutions.

Another side of the algorithm, that can be improved, is similar images
retrieval. Though VGG-16 features as images’ descriptors produce good results,
the quality of descriptors can be significantly boosted by applying, _e.g_.
metric learning techniques.

In our research, we didn’t tune weights in the total loss. Proper tuning for
each model and dataset can considerably improve final visual quality.

Due to the lack of resources, our models were trained only on CelebA-HQ
dataset. The future plan is to reproduce the research on other larger datasets
like ImageNet.

(a) Random image as auxiliary input

(b) Random noise as auxiliary input

Figure 5: From left to right: original image, masked image, Sim-1 with similar
image, Sim-1 with random/noise image. This example shows that our model Sim-1
uses auxiliary input to copy details.

## References

  * [1] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Transactions on Graphics (ToG), 28(3):24, 2009.
  * [2] G. Bradski and A. Kaehler.  Opencv.  Dr. Dobb’s journal of software tools, 2000.
  * [3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. Ieee, 2009.
  * [4] B. Dolhansky and C. C. Ferrer.  Eye in-painting with exemplar generative adversarial networks.  arXiv preprint arXiv:1712.03999, 2017.
  * [5] L. A. Gatys, A. S. Ecker, and M. Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016.
  * [6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [7] D. Ha and D. Eck.  A neural representation of sketch drawings.  arXiv preprint arXiv:1704.03477, 2017.
  * [8] J. Hays and A. A. Efros.  Scene completion using millions of photographs.  In ACM Transactions on Graphics (TOG), volume 26, page 4. ACM, 2007\.
  * [9] K. He, X. Zhang, S. Ren, and J. Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [10] S. Iizuka, E. Simo-Serra, and H. Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (TOG), 36(4):107, 2017.
  * [11] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks.  arXiv preprint, 2017.
  * [12] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European Conference on Computer Vision, pages 694–711. Springer, 2016.
  * [13] T. Karras, T. Aila, S. Laine, and J. Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [14] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [15] P. Li, Y. Hu, Q. Li, R. He, and Z. Sun.  Global and local consistent age generative adversarial networks.  arXiv preprint arXiv:1801.08390, 2018.
  * [16] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro.  Image inpainting for irregular holes using partial convolutions.  arXiv preprint arXiv:1804.07723, 2018.
  * [17] M. Mori.  The uncanny valley.  Energy, 7(4):33–35, 1970.
  * [18] A. Nguyen, J. Yosinski, Y. Bengio, A. Dosovitskiy, and J. Clune.  Plug & play generative networks: Conditional iterative generation of images in latent space.  arXiv preprint, 2017.
  * [19] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [20] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2536–2544, 2016.
  * [21] X. Qi, Q. Chen, J. Jia, and V. Koltun.  Semi-parametric image synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8808–8816, 2018.
  * [22] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [23] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen.  Improved techniques for training gans.  In Advances in Neural Information Processing Systems, pages 2234–2242, 2016.
  * [24] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [25] N. Sundaram, T. Brox, and K. Keutzer.  Dense point trajectories by gpu-accelerated large displacement optical flow.  In European conference on computer vision, pages 438–451. Springer, 2010.
  * [26] A. Telea.  An image inpainting technique based on the fast marching method.  Journal of graphics tools, 9(1):23–34, 2004.
  * [27] D. Ulyanov, A. Vedaldi, and V. Lempitsky.  Deep image prior.  arXiv preprint arXiv:1711.10925, 2017.
  * [28] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [29] C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang, and H. Li.  High-resolution image inpainting using multi-scale neural patch synthesis.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, page 3, 2017.
  * [30] S. Yoo and R.-H. Park.  Red-eye detection and correction using inpainting in digital photographs.  IEEE Transactions on Consumer Electronics, 55(3), 2009.
  * [31] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang.  Generative image inpainting with contextual attention.  arXiv preprint, 2018.
  * [32] W. Zhou, H. Li, and Q. Tian.  Recent advance in content-based image retrieval: A literature survey.  arXiv preprint arXiv:1706.06064, 2017.

## Appendix

In this section we present more visual examples of our model (Sim-1). In the
figure 6 you can see comparison examples between two best models Sim-1 and
PConv [16]. Additionally in the figure 7 we show failure cases of Sim-1.

[figure]justification=centering

(a) Original

(b) Masked

(c) PConv

(d) Sim-1 (ours)

(e) Most similar

Figure 6: Comparison between (c) PConv and (d) our model Sim-1. (a, b)
original and masked input, (e) most similar image which is fed to Sim-1. Sim-1
often results in more sharp and realistic details, unlike PConv.

(a) Original

(b) Masked

(c) Sim-1 (ours)

(d) Most similar

Figure 7: Failure cases of (c) our Sim-1 model. (a, b) original and masked
input, (d) most similar image which is fed to Sim-1. Sim-1 model’s failures
can be divided into three groups: (1, 2) blurry results, (3, 4) unrealistic
details and (5, 6) inconsistent/asymmetrical outputs.