**footnotetext: First Author. Work done during an internship at NAVER
Cloud.$\dagger$$\dagger$footnotetext: Co-corresponding authors.

# Noise Map Guidance: Inversion with Spatial Context for Real Image Editing

Hansam Cho1,2∗, Jonghyun Lee1,2, Seoung Bum Kim1, Tae-Hyun Oh3,4†, Yonghyun
Jeong2†
1School of Industrial and Management Engineering, Korea University, 2NAVER
Cloud
3Dept. of Electrical Engineering and Grad. School of Artificial Intelligence,
POSTECH
4Institute for Convergence Research and Education in Advanced
Technology,Yonsei University
{chosam95, tomtom1103<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Text-guided diffusion models have become a popular tool in image synthesis,
known for producing high-quality and diverse images. However, their
application to editing real images often encounters hurdles primarily due to
the text condition deteriorating the reconstruction quality and subsequently
affecting editing fidelity. Null-text Inversion (NTI) has made strides in this
area, but it fails to capture spatial context and requires computationally
intensive per-timestep optimization. Addressing these challenges, we present
Noise Map Guidance (NMG), an inversion method rich in a spatial context,
tailored for real-image editing. Significantly, NMG achieves this without
necessitating optimization, yet preserves the editing quality. Our empirical
investigations highlight NMG’s adaptability across various editing techniques
and its robustness to variants of DDIM inversions.

Figure 1: Compared to other inversion methods, NMG (a) demonstrates high
fidelity editing when paired with Prompt-to-Prompt, (b) successfully conducts
viewpoint alteration via MasaCtrl, and (c) preserves the spatial context of
the input image while performing zero-shot image-to-image translation with
pix2pix-zero. Text prompt corresponding to each input image is presented
beneath each sample, with words introduced for image editing distinctly
highlighted in green.

##  1 Introduction

Text-guided diffusion models (Rombach et al., 2022; Saharia et al., 2022;
Ramesh et al., 2022; Chang et al., 2023; Podell et al., 2023) have recently
emerged as a powerful tool in image synthesis, widely adapted for their
ability to generate images of exceptional visual quality and diversity. Owing
to their impressive performance, numerous studies have leveraged these models
for image editing (Hertz et al., 2022; Cao et al., 2023; Parmar et al., 2023;
Epstein et al., 2023). While these models excel in editing synthesized images,
they often produce sub-par results when editing real images. Hertz et al.
(2022) point out that this challenge mostly originates from the reliance on
classifier-free guidance (CFG) (Ho & Salimans, 2021), a method ubiquitously
adopted to increase fidelity

Editing real images within a diffusion framework follows a twofold approach.
Firstly, an image is inverted into a latent variable via DDIM inversion. This
latent variable is then channeled into two paths: reconstruction and editing.
While the editing path modifies the latent towards the desired outcome, it
also integrates information from the reconstruction path to retain the
original identity of the image. As a result, the quality of the edit is highly
dependent on the reconstruction path. However, extrapolating towards the
condition via CFG introduces errors in each timestep. This deviation pushes
the reconstruction path away from the DDIM inversion trajectory, hindering
accurate reconstruction and ultimately diminishing the editing fidelity.

Addressing this limitation, Mokady et al. (2023) propose Null-text Inversion
(NTI) leading to notable achievements in real image editing. This approach
optimizes the null-text embedding used in CFG on a per-timestep basis,
correcting the reconstruction path to the desired DDIM inversion trajectory.
By leveraging the optimized null-text embedding and integrating it with
Prompt-to-Prompt (Hertz et al., 2022) editing, NTI conducts real image
editing. While NTI offers certain benefits, its use of per-timestep
optimization can lead to increased computational demands in practical
applications. As a solution to this time-intensive optimization method, NPI
(Miyake et al., 2023) and ProxNPI (Han et al., 2023) attempt to approximate
the optimized null-text embedding without optimization. Although these methods
succeed in correcting the reconstruction path, they often struggle to capture
the spatial context in the input images.

To address these challenges, we introduce Noise Map Guidance (NMG), an
inversion methodology enriched with spatial context for real image editing. To
capture spatial context, we employ the latent variables from DDIM inversion,
which we refer to as noise maps. These noise maps, essentially noisy
representations of images, inherently encapsulate the spatial context. To
eliminate the need for an optimization phase, we condition the noise maps to
the reverse process. Rather than solely depending on text embeddings for image
editing, our methodology harnesses both noise maps and text embeddings,
drawing from their spatial and semantic guidance to perform faithful editing.
To accommodate our dual-conditioning strategy, we reformulate the reverse
process by leveraging the guidance technique proposed by Zhao et al. (2022).

Our experimental results highlight NMG’s capacity to preserve the spatial
context of the input image during real image editing. Figure 1 (a) and (c)
reveal that NTI, NPI, and ProxNPI often struggle to capture the spatial
context of the input image. Furthermore, Figure 1 (b) highlights a scenario
where spatial context is essential for effective editing, and in such context,
NMG consistently outperforms other methods. By utilizing guidance techniques
of diffusion models, our optimization-free method achieves speeds
substantially faster than NTI without compromising editing quality. The
versatility of NMG is further emphasized by its integration with various
editing techniques, e.g., Prompt-to-Prompt (Hertz et al., 2022), MasaCtrl (Cao
et al., 2023) and pix2pix-zero (Parmar et al., 2023), each grounded in an
inversion methodology. Moreover, we demonstrate NMG’s resilience to variations
of DDIM inversion. Our main contributions are summarized as follows:

  * •

We present Noise Map Guidance (NMG), an inversion method rich in spatial
context, specifically tailored for real-image editing.

  * •

Although formulated as an optimization-free method, we show that NMG maintains
editing quality without compromise, even achieving superior performance over
competing methods.

  * •

We demonstrate NMG’s adaptability by combining it with different editing
methodologies and by highlighting its consistent robustness across variations
of DDIM inversion.

##  2 Related Work

#### Inversion with Diffusion Models

The initial stage of image editing typically consists of encoding the input
image into a latent space, referred to as inversion. DDIM (Song et al., 2020a)
first introduces the concept of inversion within diffusion models by
formulating the sampling process as an ordinary differential equation (ODE)
and inverting it to recover a latent noise as a starting point for the
sampling process. However, the precision of DDIM’s reconstruction deteriorates
when being integrated with a text-guided diffusion model (Hertz et al., 2022).
In response, recent studies propose the addition of an auxiliary diffusion
state (Wallace et al., 2023) or a different inversion framework (Huberman-
Spiegelglas et al., 2023). Recently, Mokady et al. (2023) proposes Null-text
Inversion (NTI), which tailors inversion techniques specifically for text-
guided diffusion models. NTI refines the null-text embedding used in text-
guided diffusion models, achieving accurate image reconstruction. However, the
intensive computation required for optimization constrains its broader
applicability. To mitigate this drawback, Negative-prompt Inversion (NPI)
(Miyake et al., 2023) approximates the optimized null-text embedding to
achieve optimization-free inversion, albeit with compromised performance
relative to NTI. Subsequently, Han et al. (2023) incorporate proximal guidance
into NPI, enhancing its quality. Like NPI, we introduce an optimization-free
inversion technique, but uniquely, our approach robustly maintains the spatial
context of the original image.

#### Editing with Inversion methods

Despite the emergence of numerous studies in the field of editing via
diffusion models, they often encounter challenges including the need for extra
training (Kawar et al., 2023; Valevski et al., 2023; Bar-Tal et al., 2022), or
the incorporation of additional conditional signals (Avrahami et al., 2022;
2023; Nichol et al., 2022; Rombach et al., 2022). Recently, inversion-based
editing methods (Hertz et al., 2022; Cao et al., 2023; Parmar et al., 2023)
have shown promising results that do not require additional training or
conditional signals. These methods typically involve two parallel phases: a
reconstruction sequence and an editing sequence. In the reconstruction
sequence, critical information from the input image is extracted and fed into
the editing sequence for manipulation. Notably, Prompt-to-Prompt (Hertz et
al., 2022) leverages cross-attention maps to guide the editing sequence, while
MasaCtrl (Cao et al., 2023) introduces mutual self-attention to facilitate
non-rigid editing. pix2pix-zero (Parmar et al., 2023) utilizes cross-attention
map guidance to perform zero-shot image-to-image translation. Because
obtaining precise information from the reconstruction path is crucial for
reliable image editing, the efficiency of these methods heavily relies on the
performance of the inversion methods they use. By integrating our inversion
method with existing inversion-based editing methods, we demonstrate
significant improvement in preserving the spatial context of input images.

##  3 Method

###  3.1 Background

#### Text-guided Diffusion Model

Text-guided diffusion models (Rombach et al., 2022) are designed to map a
random Gaussian noise vector ${\bm{z}}_{T}$ into an image ${\bm{z}}_{0}$ while
aligning with the given text condition $c_{T}$, typically text embeddings
derived from text encoders like CLIP (Radford et al., 2021). This is achieved
through a sequential denoising operation, commonly termed as the reverse
process. This process is driven by a noise prediction network
$\epsilon_{\theta}$, which is optimized by the loss:

$L_{simple}=E_{{\bm{z}}_{0},\epsilon\sim N(0,I),t\sim
U(1,T)}\|{\epsilon-\epsilon_{\theta}({\bm{z}}_{t},t,c_{T})}\|_{2}^{2}.$ (1)

Although $\epsilon_{\theta}$ is conditioned on the timestep $t$, denoted as
$\epsilon_{\theta}({\bm{z}}_{t},t,c_{T})$, we omit the timestep condition for
the output of the network as $\epsilon_{\theta}({\bm{z}}_{t},c_{T})$ for
brevity. Text-guided diffusion models typically utilize classifier-free
guidance (CFG) (Ho & Salimans, 2021) to incorporate text conditions during
image generation. CFG is represented as follows:

$\tilde{\epsilon}_{\theta}({\bm{z}}_{t},c_{T})=\epsilon_{\theta}\left({\bm{z}}_{t},\emptyset\right)+w\cdot(\epsilon_{\theta}\left({\bm{z}}_{t},c_{T}\right)-\epsilon_{\theta}\left({\bm{z}}_{t},\emptyset\right)).$
(2)

Here, $w$ signifies the text guidance scale (controlling the impact of the
text condition), and $\emptyset$ denotes the null text embedding (the
embedding vector for a null-text “”).

Figure 2: As seen in (a), naive reconstruction often fails due to the
reconstruction path diverging from the original inversion path. Achieving
reliable reconstruction necessitates realigning the reconstruction path with
the inversion path. As depicted in (b), NTI achieves this alignment by
optimizing the null-text embedding, thereby reducing the error between the
inversion and reconstruction paths. Conversely, NMG, as shown in (c),
conditions the reconstruction process based on the divergence between the two
paths, leveraging this variance to refine the reconstruction path.

#### DDIM Inversion

To edit a real image in the framework of diffusion models, an image firstly
has to be converted into a latent variable ${\bm{z}}^{*}_{T}$, a process known
as inversion. Fundamentally, the latent variable ${\bm{z}}^{*}_{T}$ is the
original starting noise that reconstructs into the image when denoised. To
invert an image into its latent variable, DDIM inversion (Song et al., 2020a)
is predominantly utilized. DDIM inversion is derived from the reverse process
of DDIM that is formulated as:

${\bm{z}}_{t-1}=\sqrt{\tfrac{\alpha_{t-1}}{\alpha_{t}}}{\bm{z}}_{t}+\sqrt{\alpha_{t-1}}\left(\sqrt{\tfrac{1}{\alpha_{t-1}}-1}-\sqrt{\tfrac{1}{\alpha_{t}}-1}\right)\epsilon_{\theta}({\bm{z}}_{t},c_{T})$
(3)

with $\\{\alpha_{t}\\}_{t=0}^{T}$ as a predefined noise schedule. Because this
DDIM reverse process can be formulated as an ordinary differential equation
(ODE), the DDIM inversion process can be obtained by reversing said ODE as:

${\bm{z}}_{t+1}=\sqrt{\tfrac{\alpha_{t+1}}{\alpha_{t}}}{\bm{z}}_{t}+\sqrt{\alpha_{t+1}}\left(\sqrt{\tfrac{1}{\alpha_{t+1}}-1}-\sqrt{\tfrac{1}{\alpha_{t}}-1}\right)\epsilon_{\theta}({\bm{z}}_{t},c_{T}).$
(4)

#### Null-text Inversion

Inversion-based editing methods follow a twofold approach. First, an image is
converted into its latent variable through inversion. This is followed by a
simultaneous two-phase procedure: reconstruction and editing. While the
editing phase alters the latent variable towards the desired modification, it
concurrently draws on information from the reconstruction phase to maintain
the image’s characteristics. During editing, a substantial CFG guidance value
of $w>1$ is crucial for generating high-fidelity images. However, the
extrapolation toward the condition introduces errors at every timestep. As
illustrated in Figure 2 (a), a large CFG guidance scale misguides the
reconstruction path away from the inversion path. This divergence leads to an
imprecise reconstructed image, subsequently degrading the final quality of the
edited image. To address this, Null-text Inversion (NTI) (Mokady et al., 2023)
optimizes null-text embeddings used in Eq. 2 to align with the DDIM inversion
trajectory. Initially, DDIM inversion is performed with $w=1$ to compute the
latent variables $\\{{\bm{z}}_{t}^{*}\\}_{t=1}^{T}$, termed the noise maps. As
the reverse process progresses from $t$ to $t-1$, ${\bm{z}}_{t-1}$ is derived
from ${\bm{z}}_{t}$ with Eq. 2 and Eq. 3:

${\bm{z}}_{t-1}=\sqrt{\tfrac{\alpha_{t-1}}{\alpha_{t}}}{\bm{z}}_{t}+\sqrt{\alpha_{t-1}}\left(\sqrt{\tfrac{1}{\alpha_{t-1}}-1}-\sqrt{\tfrac{1}{\alpha_{t}}-1}\right)\tilde{\epsilon}_{\theta}({\bm{z}}_{t},c_{T}).$
(5)

With both ${\bm{z}}_{t-1}$ and a single noise map ${\bm{z}}^{*}_{t-1}$
available, the null-text embedding $\emptyset_{t}$ is optimized every step
using the loss function:

$\min_{\emptyset_{t}}\|{\bm{z}}_{t-1}-{\bm{z}}_{t-1}^{*}\|_{2}^{2}.$ (6)

By optimizing the null-text embeddings, NTI reduces the error of the
reconstruction path. Overall process of NTI is depicted in Figure 2 (b).
However, since the null-text embedding is represented as a single-dimensional
vector in $\mathbb{R}^{d}$, it struggles to capture the image’s spatial
context.

###  3.2 Noise Map Guidance

Unlike null-text embeddings, noise maps $\\{{\bm{z}}_{t}^{*}\\}_{t=1}^{T}$
naturally capture the spatial context of the input image. This is due to the
fact that noise maps originate from infusing a small amount of noise into the
input image and have the same spatial dimensions as the input image.
Leveraging this attractive trait, we directly employ noise maps to preserve
the spatial context of an image. As a part of this approach, we condition the
reverse process on noise maps. Given that current text-guided diffusion models
are conditioned solely on text embeddings, our initial step is to reformulate
the reverse process to account for both text and noise maps.

To reformulate the reverse process to a conditional form, we introduce the
score function $\nabla_{{\bm{z}}_{t}}\log p({\bm{z}}_{t})$ with the relation
$\epsilon_{\theta}({\bm{z}}_{t})\approx-\sqrt{1-\alpha_{t}}\nabla_{{\bm{z}}_{t}}\log
p({\bm{z}}_{t})$. By replacing $\nabla_{{\bm{z}}_{t}}\log p({\bm{z}}_{t})$
with $\nabla_{{\bm{z}}_{t}}\log p({\bm{z}}_{t}|c)$, we enable the network to
produce outputs based on specific conditions, leading to the equation
$\epsilon_{\theta}({\bm{z}}_{t},c)\approx-\sqrt{1-\alpha_{t}}\nabla_{{\bm{z}}_{t}}\log
p({\bm{z}}_{t}|c)$. Applying Bayes’s rule, $\nabla_{{\bm{z}}_{t}}\log
p({\bm{z}}_{t}|c)=\nabla_{{\bm{z}}_{t}}\log
p({\bm{z}}_{t})+\nabla_{{\bm{z}}_{t}}\log p(c|{\bm{z}}_{t})$, the network’s
conditional output is then formulated as:

$\epsilon_{\theta}({\bm{z}}_{t},c)=-\sqrt{1-\alpha_{t}}(\nabla_{{\bm{z}}_{t}}\log
p({\bm{z}}_{t})+\nabla_{{\bm{z}}_{t}}\log p(c|{\bm{z}}_{t}))$ (7)

In this context, $\nabla_{{\bm{z}}_{t}}\log p(c|{\bm{z}}_{t})$ is a crucial
component to condition the diffusion model. We introduce energy guidance (Zhao
et al., 2022), which serves as a flexible conditioning format in our
formulation. With energy guidance, Eq. 7 is revised as follows:

$\epsilon_{\theta}({\bm{z}}_{t},c)=-\sqrt{1-\alpha_{t}}(\nabla_{{\bm{z}}_{t}}\log
p({\bm{z}}_{t})-\nabla_{{\bm{z}}_{t}}\mathcal{E}({\bm{z}}_{t},c,t)).$ (8)

NTI (Mokady et al., 2023) performs one-step denoising, transitioning from
$z_{t}$ to $z_{t-1}$, and compares the noise map $z^{*}_{t-1}$ with latent
variable $z_{t-1}$ for optimization. To align with this process, we define the
energy function $\mathcal{E}({\bm{z}}_{t},c,t)=\|{z}_{t-1}-z_{t-1}^{*}\|_{1}$
to condition the noise map for each timestep. Note that unlike NTI, we employ
the L1 distance to compute the distance between the noise map and the current
latent variable. Based on the definition of the energy function, Eq. 8 is
revised as:

$\displaystyle\epsilon_{\theta}(z_{t},c_{N})=-\sqrt{1-\alpha_{t}}(\nabla_{{\bm{z}}_{t}}\log
p({\bm{z}}_{t})-s_{g}\cdot\nabla_{z_{t}}\|{\bm{z}}^{\prime}_{t-1}-{\bm{z}}_{t-1}^{*}\|_{1})$
(9)
$\displaystyle\text{where}\quad{\bm{z}}^{\prime}_{t-1}=\sqrt{\tfrac{\alpha_{t-1}}{\alpha_{t}}}{\bm{z}}_{t}+\sqrt{\alpha_{t-1}}\left(\sqrt{\tfrac{1}{\alpha_{t-1}}-1}-\sqrt{\tfrac{1}{\alpha_{t}}-1}\right)\epsilon_{\theta}({\bm{z}}_{t},\emptyset).$

We term the noise map $c_{N}$ conditioned to produce the network output as
$\epsilon_{\theta}(z_{t},c_{N})$ in Eq. 9. Additionally, we introduce a
hyperparameter $s_{g}$ termed the gradient scale in Eq. 9 to compensate for
the scale difference between the gradient and the network’s output. By
adjusting the magnitude of $s_{g}$, we modulate the degree of the edit from
the original reverse process to the DDIM inversion trajectory. Building on the
findings that a substantial guidance scale can yield high-quality results
(Dhariwal & Nichol, 2021; Ho & Salimans, 2021), we also introduce a guidance
technique for noise map condition similar to Eq. 2 as follows:

$\tilde{\epsilon_{\theta}}({\bm{z}}_{t},c_{N})=\epsilon_{\theta}({\bm{z}}_{t},\emptyset)+s_{N}\cdot(\epsilon_{\theta}({\bm{z}}_{t},c_{N})-\epsilon_{\theta}({\bm{z}}_{t},\emptyset)),$
(10)

where $s_{N}$ is the guidance scale of the noise map. To derive a latent
variable conditioned on the noise map, we execute a one-step reverse process
with $\tilde{\epsilon_{\theta}}({\bm{z}}_{t},c_{N})$ as follows:

${\bm{z}}^{NM}_{t-1}=\sqrt{\tfrac{\alpha_{t-1}}{\alpha_{t}}}{\bm{z}}_{t}+\sqrt{\alpha_{t-1}}\left(\sqrt{\tfrac{1}{\alpha_{t-1}}-1}-\sqrt{\tfrac{1}{\alpha_{t}}-1}\right)\tilde{\epsilon_{\theta}}({\bm{z}}_{t},c_{N}).$
(11)

By performing a text-conditioned reverse process with ${\bm{z}}^{NM}_{t-1}$, a
latent variable conditioned on a noise map, we derive our final latent
variable that is conditioned on both the noise map and the text condition.
Empirically, we find that we can approximate
${\bm{z}}^{NM}_{t}\approx{\bm{z}}^{NM}_{t-1}$. Thus, a latent ${\bm{z}}_{t-1}$
both conditioned on noise map and text embedding is determined as follows:

$\displaystyle\tilde{\epsilon_{\theta}}({\bm{z}}^{NM}_{t},c_{T})=\epsilon_{\theta}({\bm{z}}^{NM}_{t},\emptyset)+s_{T}\cdot(\epsilon_{\theta}({\bm{z}}^{NM}_{t},c_{T})-\epsilon_{\theta}({\bm{z}}^{NM}_{t},\emptyset))$
(12)
$\displaystyle{\bm{z}}_{t-1}=\sqrt{\tfrac{\alpha_{t-1}}{\alpha_{t}}}{\bm{z}}^{NM}_{t}+\sqrt{\alpha_{t-1}}\left(\sqrt{\tfrac{1}{\alpha_{t-1}}-1}-\sqrt{\tfrac{1}{\alpha_{t}}-1}\right)\tilde{\epsilon_{\theta}}({\bm{z}}^{NM}_{t},c_{T}).$
(13)

To maintain consistent notation conventions with $s_{N}$ in Eq. 10, we
designate $s_{T}$ to represent the text guidance scale, instead of $w$ in Eq.
2. In Figure 2 (c), we display the overall process of NMG. We note that NMG is
a sequential process of first conditioning the noise map and conditioning the
text embedding on the outcome of the step before.

##  4 Experiments

Figure 3: Image editing results using Prompt-to-Prompt are shown in (a) for
local editing and (b) for global editing. Results show that DDIM lacks in
preserving details of the input image, both NTI and NPI face challenges in
maintaining spatial context, and ProxNPI exhibits limited editing
capabilities. In contrast, NMG consistently produces robust results for both
local and global edits.

Our method is tailored to ensure that the reconstruction path remains closely
aligned with the inversion trajectory throughout the image editing process. In
the section below, we compare NMG with several methods, including DDIM (Song
et al., 2020a), NTI (Mokady et al., 2023), NPI (Miyake et al., 2023), and
ProxNPI (Han et al., 2023). While DDIM is not inherently formulated to prevent
reconstruction divergence, we include it in our comparisons to highlight how
image quality deteriorates when the reconstruction path strays from the
desired trajectory.

###  4.1 Qualitative Comparison

Given that the primary goal of inversion methods is image editing, we first
integrate NMG into the widely adopted Prompt-to-Prompt (Hertz et al., 2022)
editing method to empirically validate our approach. For the integration with
Prompt-to-Prompt, we incorporate NMG within the reconstruction path, rather
than the editing path. This approach stems from NMG’s characteristic of
aligning a pathway with the inversion trajectory. Such an alignment can be
detrimental to the image’s editing.

Prompt-to-Prompt performs diverse tasks by controlling attention maps from
reconstruction and editing paths. Our experiments encompass four local editing
tasks: object swapping, contextual alteration, facial attribute editing, and
color change, as well as four global style transfer tasks to styles of oil
paintings, van Gogh, pointillism, and watercolor. Figures 3 depict our results
for both local and global editing using Prompt-to-Prompt. Due to its limited
reconstruction capability, DDIM struggles to integrate details of the input
image into the edited image. Both NTI and NPI leverage null-text embeddings to
retain input image details, but since the null-text embedding is a one-
dimensional vector, it inherently struggles to preserve spatial context.
ProxNPI, due to the inversion guidance, utilizes the noise maps to follow an
inversion trajectory and excels in retaining spatial context. However, this
guidance is also applied to editing paths, producing constrained results. For
instance, the first row of Figure 3 (b) illustrates an attempt to transition
the style of the image toward “van Gogh style”. ProxNPI tends to adhere
closely to the original context, whereas our proposed NMG effectively
transforms the style.

Figure 4: Image editing outcomes are presented using (a) MasaCtrl and (b)
pix2pix-zero. NMG’s proficiency in retaining spatial context is highlighted in
(a), while its resilience to variations of DDIM inversion is showcased in (b).

###  4.2 Evaluating Property with Additional Editing Methods

#### Leveraging spatial context

The distinct advantage of NMG lies in its direct utilization of the noise
maps, allowing it to preserve spatial context compared to other inversion
methods. To demonstrate this capability, we integrate NMG with MasaCtrl (Cao
et al., 2023), an editing approach that utilizes mutual self-attention for
non-rigid editing tasks. Because MasaCtrl relies on the DDIM to reconstruct an
input image during real image editing, we substitute the DDIM with NMG and
other comparison methods to demonstrate the effectiveness of our approach. For
spatially demanding edits, we undertake pose modification and viewpoint
alternation. Figure 4 (a) showcases the results achieved with MasaCtrl. Due to
NMG’s capability to access spatial context directly through noise maps, it
yields unparalleled editing results. The effectiveness of our method in
preserving spatial context is further highlighted in Section 4.1. Although
ProxNPI employs inversion guidance to capture spatial context, its reliance on
a single gradient descent step to align with the inversion trajectory
occasionally results in inconsistent outputs. We also compare results with
ProxMasaCtrl, a method proposed by Han et al. (2023) that integrates ProxNPI
with MasaCtrl using a different integration approach than our main comparison.
For details and experiments, see Appendix A.2.

#### Robustness to variations of DDIM inversion

Most inversion techniques leverage DDIM inversion as its foundation to encode
the image into the latent. However, DDIM inversion can be modified to meet
certain goals. To explore the robustness of our method to variations on DDIM
inversion, we incorporate NMG with pix2pix-zero (Parmar et al., 2023).
Pix2pix-zero, designed for zero-shot image-to-image translation, modifies DDIM
inversion by adding a regularization term to calibrate the initial noise to
more closely resemble Gaussian noise. In the inversion phase, we utilize the
modified DDIM inversion proposed in pix2pix-zero. For the reconstruction
phase, similar to our experiments with MasaCtrl, we integrate NMG and other
comparison methods. Figure 4 (b) presents the editing outcomes achieved using
pix2pix-zero on tasks like cat-to-dog, and dog-to-cat conversions. The results
indicate that our method produces robust results with spatially coherent
samples, even under variations of DDIM inversion.

###  4.3 Quantitative Comparison

Models | P2P (Local) | P2P (Global) | MasaCtrl | User Study
---|---|---|---|---
CLIP$\uparrow$ | TIFA$\uparrow$ | CLIP$\uparrow$ | TIFA$\uparrow$ | CLIP$\uparrow$ | TIFA$\uparrow$ |
DDIM | 0.2977 | 0.8436 | 0.3066 | 0.8349 | 0.2825 | 0.8253 | -
NTI | 0.2983 | 0.9125 | 0.3202 | 0.8302 | 0.2903 | 0.8277 | 10.0%
NPI | 0.2982 | 0.9076 | 0.3157 | 0.8117 | 0.2921 | 0.8188 | 5.0%
ProxNPI | 0.2951 | 0.8947 | 0.3006 | 0.8463 | 0.2922 | 0.8188 | 12.5%
NMG(Ours) | 0.3007 | 0.8955 | 0.3221 | 0.8991 | 0.2955 | 0.8548 | 72.5%
Table 1: Quantitative evaluation of image editing using local and global
editing with Prompt-to-Prompt, MasaCtrl, and user study reveals that NMG
consistently surpasses other baseline methods in editing performance.

To quantitatively measure the quality of image editing paired with our method,
we utilize two metrics: CLIPScore (Hessel et al., 2021) and TIFA (Hu et al.,
2023). CLIPScore gauges how closely the edited image aligns with the target
text prompt in the CLIP (Radford et al., 2021) embedding space. TIFA, on the
other hand, evaluates the semantic alignment of given image and text prompt
pairs based on visual question-answering accuracy. Our evaluation, utilizing
all the tasks described in Section 4.1, is conducted on four local and global
editing results via Prompt-to-Prompt and two tasks for non-rigid image editing
via MasaCtrl. We edit 20 images for each task and report the averaging scores
in Table 1. It can be seen that NMG consistently surpasses competing methods,
an efficacy that stems from NMG’s capability to retain the spatial context of
the input image without imposing constraints on the editing path.

While the metrics previously discussed are commonly employed, their reliance
on specific models often causes them to misalign with human perception. To
address this and evaluate visual quality from a human-centric perspective, we
additionally conduct a user study. Fifty participants were recruited through
Prolific (Prolific, 2023). For the study, 40 sets, each containing four images
edited using NTI, NPI, ProxNPI, and NMG, were presented to the participants.
Each set was paired with a specific editing instruction and an input image.
Participants chose the image with the highest fidelity that best met the
editing instructions. We note that images within each set were displayed in a
randomized order. In our analysis, we collect the method most often selected
in each comparison and report the selection ratio in the final column of Table
1. Notably, our NMG method was the most preferred by participants. These
findings underscore the efficacy of our method in aligning closely with human
perception of image quality. A print screen of the user study is provided in
Figure 12.

| Method | Reconstruction
---|---|---
| optimized | conditional | MSE$\downarrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Time$\downarrow$
NTI | ✓ |  | 0.0127 | 0.7292 | 0.1588 | 104.85
NMG |  | ✓ | 0.0124 | 0.7296 | 0.1673 | 5.97
NTI + NMG | ✓ | ✓ | 0.0124 | 0.7302 | 0.1668 | 107.72
Table 2: A quantitative evaluation of reconstruction assesses how closely each
method aligns with the inversion trajectory. While combining both strategies
yields the best results, NMG stands out as an efficient approach when
considering both time and performance. Figure 5: Ablation results of (a)
guidance scales and (b) gradient scales. In (a), we demonstrate that the noise
map guidance scale governs the influence of input image nuances, while the
text guidance scale steers the extent of edits in the desired direction. In
(b), we demonstrate that the gradient scale regulates the degree of alignment
with the inversion trajectory.

###  4.4 Ablation Study

#### Reconstruction

For reconstruction of real images, NMG adheres to the trajectory of DDIM
inversion, conditioned by noise maps. To demonstrate the effectiveness of our
approach, we compare it with the optimization-based method, NTI. Our
evaluation utilizes the MS-COCO validation set Lin et al. (2014), from which
we randomly select 100 images. We assess the reconstruction quality of images
using metrics such as MSE, SSIM (Wang et al., 2004), and LPIPS (Zhang et al.,
2018). The results in Table 2 indicate that our method performs comparably to
the optimization-based approach. Notably, the best results are achieved when
both methods are used together. However, regarding reconstruction speed, our
strategy outpaces its optimization counterpart by nearly a factor of 20.
Taking both speed and performance into account, NMG emerges as the efficient
choice for aligning with the inversion trajectory during reconstruction.

#### Guidance scale

NMG utilizes a dual-conditioning strategy comprising noise maps and text
prompts. To discern the effects of each condition, we modulate their
respective scales of $s_{N}$ and $s_{T}$ in Eq.10 and Eq. 12. Figure 5 (a)
illustrates the influence of each guidance scale. Prioritizing text
conditioning aligns outcomes more closely to the intended edits. In contrast,
a prominent noise map conditioning scale retains the image’s context. However,
achieving a balance between the two scales is essential for a desirable
outcome. Over-relying on text conditioning, evident in the grid image’s lower
left triangle, erodes the nuance of the input image. Conversely, an excessive
emphasis on noise map conditioning, seen in the grid image’s upper right
triangle, may hinder successful editing. The grid image’s diagonal showcases
results from balanced scaling, indicating that appropriate scaling yields the
best outcomes. Importantly, our experiments maintained a consistent guidance
scale, underscoring its robustness across varied samples.

#### Gradient scale

To regulate the adherence to the trajectory of DDIM inversion, we employ a
gradient scale $s_{g}$ in Eq. 9. The effects of this scale are demonstrated by
varying its magnitude and presenting the samples in Figure 5 (b). A smaller
gradient scale offers limited alignment with the inversion trajectory, leading
to edited outputs closely mirroring the result using DDIM in the
reconstruction path. Conversely, an overly large gradient scale results in
pronounced trajectory alignments, degrading the editing quality. Therefore,
the proper selection of the gradient scale is vital. Analogous to the guidance
scale, we maintain a consistent gradient scale across all experiments,
ensuring it remains universally effective rather than overly sensitive to
individual samples.

##  5 Conclusion

In the evolving field of image editing, Noise Map Guidance (NMG) offers a
notable advancement. By addressing the challenges present in current real-
image editing methods using text-guided diffusion models, NMG introduces an
inversion method rich in spatial context. NMG directly conditions noise maps
to the reverse process, which captures the spatial nuance of the input image
and ensures the preservation of its spatial context. Experimental results
demonstrate NMG’s capability to preserve said spatial context, and this
property is furthermore highlighted with spatially intensive edits. NMG is
also designed as an optimization-free approach that prioritizes speed without
compromising quality. NMG represents a significant step forward, suggesting
further investigation and refinement in real-image editing techniques.

## Acknowledgements

We thank the ImageVision team of NAVER Cloud for their thoughtful advice and
discussions. Training and experiments were done on the Naver Smart Machine
Learning (NSML) platform (Kim et al., 2018). This study was supported by BK21
FOUR. T.-H. Oh was partially supported by Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government(MSIT) (No.2021-0-02068, Artificial Intelligence Innovation
Hub; No.RS-2023-00225630, Development of Artificial Intelligence for Text-
based 3D Movie Generation).

## Ethics statement

Generative models for synthesizing images carry with them several ethical
concerns, and these concerns are shared by (or perhaps exacerbated in) any
generative models such as ours. Generative models, in the hands of bad actors,
could be abused to generate disinformation. Generative models such as ours may
have the potential to displace creative workers via automation. That said,
these tools may also enable growth and improve accessibility for the creative
industry.

## Reproducibility statement

The source code can be found at https://github.com/hansam95/NMG.

## References

  * Avrahami et al. (2022) Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18208–18218, 2022.
  * Avrahami et al. (2023) Omri Avrahami, Ohad Fried, and Dani Lischinski.  Blended latent diffusion.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–11, 2023.
  * Bansal et al. (2023) Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 843–852, 2023.
  * Bar-Tal et al. (2022) Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In _European conference on computer vision_ , pp. 707–723. Springer, 2022.
  * Cao et al. (2023) Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  _arXiv preprint arXiv:2304.08465_ , 2023.
  * Chang et al. (2023) Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al.  Muse: Text-to-image generation via masked generative transformers.  _arXiv preprint arXiv:2301.00704_ , 2023.
  * Chung et al. (2022) Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye.  Improving diffusion models for inverse problems using manifold constraints.  _Advances in Neural Information Processing Systems_ , 35:25683–25696, 2022.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Epstein et al. (2023) Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski.  Diffusion self-guidance for controllable image generation.  _arXiv preprint arXiv:2306.00986_ , 2023.
  * Han et al. (2023) Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Yuxiao Chen, Di Liu, Qilong Zhangli, et al.  Improving negative-prompt inversion via proximal guidance.  _arXiv preprint arXiv:2306.05414_ , 2023.
  * Hertz et al. (2022) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or.  Prompt-to-prompt image editing with cross-attention control.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Hessel et al. (2021) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 7514–7528, 2021.
  * Ho & Salimans (2021) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_ , 2021.
  * Hu et al. (2023) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith.  Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.  _arXiv preprint arXiv:2303.11897_ , 2023.
  * Huberman-Spiegelglas et al. (2023) Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli.  An edit friendly ddpm noise space: Inversion and manipulations.  _arXiv preprint arXiv:2304.06140_ , 2023.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1125–1134, 2017.
  * Kawar et al. (2023) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6007–6017, 2023.
  * Kim et al. (2018) Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al.  Nsml: Meet the mlaas platform with a real-world case study.  _arXiv preprint arXiv:1810.09957_ , 2018.
  * Kwon & Ye (2022) Gihyun Kwon and Jong Chul Ye.  Diffusion-based image translation using disentangled style and content representation.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pp. 740–755. Springer, 2014.
  * Miyake et al. (2023) Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka.  Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models.  _arXiv preprint arXiv:2305.16807_ , 2023.
  * Mokady et al. (2023) Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6038–6047, 2023.
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning_ , pp. 16784–16804. PMLR, 2022.
  * Parmar et al. (2023) Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pp. 1–11, 2023\.
  * Pathak et al. (2016) Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2536–2544, 2016.
  * Podell et al. (2023) Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Prolific (2023) Prolific, 2023.  URL https://www.prolific.co.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022\.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Song et al. (2020a) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2020a.
  * Song et al. (2020b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2020b.
  * Valevski et al. (2023) Dani Valevski, Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning a diffusion model on a single image.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–10, 2023.
  * Wallace et al. (2023) Bram Wallace, Akash Gokul, and Nikhil Naik.  Edict: Exact diffusion inversion via coupled transformations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22532–22541, 2023.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Yu et al. (2023) Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang.  Freedom: Training-free energy-guided conditional diffusion model.  _arXiv preprint arXiv:2303.09833_ , 2023.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 586–595, 2018.
  * Zhang et al. (2022) Zhongping Zhang, Huiwen He, Bryan A. Plummer, Z. Liao, and Huayan Wang.  Complex scene image editing by scene graph comprehension.  2022\.  URL https://api.semanticscholar.org/CorpusID:247627800.
  * Zhao et al. (2022) Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu.  Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations.  _Advances in Neural Information Processing Systems_ , 35:3609–3623, 2022.

Noise Map Guidance: Inversion with Spatial Context for Real Image Editing

– Supplementary Material –

##  Appendix A Additional Experiments

Figure 6: Ablation results of (a) conditioning order and (b) energy function

###  A.1 Additional Ablation Study

#### Conditioning Order

As detailed in Section 3.2, NMG employs a dual-conditioning approach that
leverages both the noise map and text conditions sequentially. While our
primary strategy conditions the noise map before the text embeddings, it is
feasible to reverse this order. Figure 6 (a) presents the outcomes of local
editing under varying conditioning sequences: the second column depicts the
canonical NMG sequence—initially the noise map, then text; the third column
illustrates the reverse order. Both configurations yield promising results,
underscoring NMG’s robustness to conditioning sequence variations.

#### Energy Function

In Section 3.2, we formulate the energy function using the L1 distance,
diverging from the L2 distance employed by NTI (Mokady et al., 2023). Figure 6
(b) showcases editing outcomes derived from these distinct energy functions:
the second column illustrates results using L1 distance, while the third
column represents those from L2 distance. Both methodologies adeptly guide the
editing direction. However, the L2-based results manifest a noticeable
blurring in the image background. These observations align with the findings
of prior research (Isola et al., 2017; Pathak et al., 2016) that L2 distance
often produces blurred results. Hence, for NMG, we intentionally select the L1
distance for energy function formulation.

Figure 7: Comparison with ProxMasaCtrl

###  A.2 Comparisons with ProxMasaCtrl

ProxiNPI (Han et al., 2023) introduces an integration with MasaCtrl (Cao et
al., 2023), termed ProxMasaCtrl. This approach incorporates NPI (Miyake et
al., 2023) in the reconstruction path, while leveraging proximal guidance in
the editing path to enhance reliability. Figure 7 compares the results from
ProxMasaCtrl with our integration, labeled as ”NMG + MasaCtrl”. Despite
ProxMasaCtrl’s mitigation of unintended changes through proximal guidance, its
use of NPI in reconstruction means it often misses spatial context. This lack
of spatial context is evident in Figure 7 (a): the first row highlights
ProxMasaCtrl’s inability to alter poses accurately, and the second row reveals
missing spatial components, like a kitten’s tail, during editing. Similarly,
when observing viewpoint alternation results in Figure 7 (b), ProxMasaCtrl
struggles to deliver convincing outcomes, whereas our NMG-integrated solution,
drawing on its rich spatial context, consistently achieves superior results.

###  A.3 Experiments Details

Within our experimental framework, we employ Stable Diffusion (Rombach et al.,
2022), standardizing the diffusion steps to $T=50$ across all experiments. For
the editing tasks, the parameters are set as follows: noise map guidance
$s_{N}=10$, text guidance $s_{T}=10$, and guidance scale $s_{g}=5000$. For the
reconstruction tasks, the configurations are set to $s_{N}=10$, $s_{T}=7.5$,
and $s_{g}=10000$.

Figure 8: Additional editing results of (a) global editing, (b) non-rigid
editing, (c) combine NMG with NTI, and (d) additional comparison with ProxNPI

###  A.4 Additional Results

#### Local and Global Editing

We present supplementary comparison results in Figure 10 and Figure 11,
showcasing local and global editing outcomes achieved with Prompt-to-Prompt
(Hertz et al., 2022). These additional results further underscore NMG’s
proficiency in preserving spatial context without diminishing the quality of
the editing output. Furthermore, we present the results of diverse stylization
tasks in Figure 8(a), demonstrating NMG’s versatility. This evidence suggests
that NMG’s capabilities extend beyond painting stylizations, encompassing a
broad spectrum of styles, including anime and mosaic.

#### Non-rigid Editing

In conjunction with MasaCtrl (Cao et al., 2023), NMG demonstrates its
capability for non-rigid editing. Figure 8(b) exhibits additional experimental
results of pose modifications. It is evident that NMG effectively conducts
edits with pronounced spatial information changes.

#### Editing with NTI + NMG

In image reconstruction, the combined use of NTI and NMG has been shown to
yield superior quality. To ascertain whether this synergy also extends to
image editing, we have undertaken a series of editing experiments utilizing
both NTI and NMG. Figure 8(c) demonstrates that while NMG alone provides
dependable editing, it occasionally lacks specific spatial details.
Integrating additional information from NTI effectively compensates for this
shortfall, restoring the spatial context of the input image. This integration
potentially represents a significant advancement in augmenting the
capabilities of NMG for image editing tasks.

#### Comparisons with ProxNPI

For a further comparative analysis with ProxNPI, we replicated the experiment
using the same image featured in the ProxNPI study. Figure 8(d) presents these
additional comparative results. The outcomes demonstrate that NMG yields
results comparable to the ProxNPI. However, it is essential to note that
ProxNPI primarily concentrates on object swapping and does not extensively
explore various editing scenarios, such as contextual alterations or
stylization. As discussed in Section 4.1, ProxNPI’s capabilities are somewhat
limited due to its reliance on inversion guidance. In contrast, NMG is not
subject to these constraints, demonstrating its more extensive utility in
various image editing tasks.

#### Reconstruction

We offer an additional quantitative comparison of reconstruction results. The
left side of Table 3 presents a quantitative comparison of reconstruction
capabilities between NMG and other methods. While NMG demonstrates comparable
quantitative results in reconstruction to these methods, it is important to
emphasize that NMG’s primary focus is on image editing. It should be noted
that high-quality reconstruction does not invariably equate to superior
editing performance. ProxNPI exhibits a commendable ability to reconstruct the
image in Table 3. However, as illustrated in the sixth row of Figure 11, its
editing capabilities are somewhat limited, underscoring the distinction
between reconstruction proficiency and editing versatility.

#### Additional User Study

Evaluating edited images based on their alignment with editing instructions is
crucial, yet it is equally essential to delve into more detailed aspects of
evaluation. For example, in local editing tasks, it is essential to maintain
the integrity of unedited regions, while in global editing, preserving the
overall structure of the image is essential. To explore these aspects, we
conducted an additional user study similar to the approach in Section 4.3.
Thirty participants, recruited via Prolific (Prolific, 2023), were presented
with ten sets of images. Participants were instructed to assess the
preservation of unselected regions in local editing and the retention of the
overall structure in global editing. The results of the study, displayed on
the right side of Table 3, indicate that NMG performs well in both preserving
unselected regions during local editing and maintaining the overall structure
in global editing scenarios.

Method | Reconstruction | User Study
---|---|---
MSE$\downarrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
DDIM | 0.159 | 0.368 | 0.521 | -
NTI | 0.013 | 0.729 | 0.159 | 0.0%
NPI | 0.021 | 0.661 | 0.230 | 0.0%
ProxNPI | 0.012 | 0.737 | 0.153 | 10.0%
NMG(Ours) | 0.012 | 0.723 | 0.167 | 90.0%
Table 3: Quantitative comparison of reconstruction and additional user study
Figure 9: Illustrations of limitations

##  Appendix B Limitations

Our research endeavors to enhance inversion-based editing methods, focusing on
improving spatial context during the editing process. Nevertheless, our
current methodology is limited to compatibility with methods that do not
conform to the inversion-based framework. For instance, SGC-Net (Zhang et al.,
2022), which operates on a text-guided diffusion model for relationship change
tasks, proves challenging to integrate with NMG due to its deviation from the
inversion-based editing paradigm. Consequently, applying NMG in conjunction
with SGC-Net for tasks involving relationship changes faces significant
hurdles. As an alternative, we attempted relationship change tasks using
MasaCtrl (Cao et al., 2023). However, Figure 9(a) depicts the ineffectiveness
of this approach, as MasaCtrl is not inherently designed for relationship
change tasks.

Additionally, the editing capabilities of NMG are intrinsically linked to the
limitations of existing inversion-based methods. For example, Prompt-to-Prompt
(Hertz et al., 2022) editing involves the swapping of cross-attention maps
between source and target texts, which restricts the possibility of structural
changes in the edited object to the constraints of the original image’s
object. This limitation is evident in Figure 9(b), which illustrates an object
swap task between a car and a bicycle. The disparate structures of these
objects underscore the challenges faced by NMG in ensuring reliable editing
under these conditions.

Moreover, NMG’s dependence on text for image editing hinders its ability to
perform precise spatial changes. As shown on the left side of Figure 9(c), NMG
can effectively remove an object, such as a man, from an image. However, it
lacks the precision to identify and remove a specific individual, making
targeted removals unfeasible. Similarly, as seen on the right side of Figure
9(c), NMG struggles to follow exact location directives when adding new
objects. For instance, despite the instruction to place a boat on the right
side of the beach, the edited image fails to place it on the right. Addressing
these challenges to enable more accurate spatial control in real image editing
with NMG is a vital area for future research and development.

Figure 10: Local editing with Prompt-to-Prompt Figure 11: Global editing with
Prompt-to-Prompt

##  Appendix C Extended Related Work

#### Guidance in Diffusion Models

To generate samples that abide by a certain condition, Score-SDE (Song et al.,
2020b) initially introduces a formulation for conditional diffusion models.
Following this formulation, Dhariwal & Nichol (2021) proposes classifier
guidance, which leverages additional classifiers to enhance the quality of
class conditional image synthesis. Building upon classifier guidance, Ho &
Salimans (2021) proposes classifier-free guidance, which obviates the need for
an additional classifier. However, these guidance methods are often restricted
to class or text conditions. EGSDE (Zhao et al., 2022) advances the field by
introducing energy guidance techniques. Although EGSDE primarily focuses on
image-to-image translation tasks, they notably expand the flexibility in
conditioning formats of diffusion models. Similarly, DiffuseIT (Kwon & Ye,
2022), inspired by MCG (Chung et al., 2022), focuses on image translation.
Successive methodologies, such as FreeDoM (Yu et al., 2023) and universal
guidance (Bansal et al., 2023), broaden the scope of conditions to encompass a
diverse range of generative tasks. These methodologies are adaptable to
various conditional signals, including segmentation maps, style images, and
facial identities. Aligning with these advancements, our approach also
leverages the versatile framework of energy guidance, targeting the accurate
reconstruction of input images.

##  Appendix D User Study

A screenshot of our user study conducted through Prolific (Prolific, 2023) is
depicted in Figure 12. Note that Prolific receives a pre-built Google form,
and distributes it to participants of the study.

Figure 12: User study screenshot


* Jayasinghe et al. (2018) Jayasinghe T., et al., 2018, MNRAS, 477, 3145
  * Jayasinghe et al. (2019a) Jayasinghe T., et al., 2019a, MNRAS, 485, 961
  * Jayasinghe et al. (2019b) Jayasinghe T., et al., 2019b, MNRAS, 486, 1907
  * Jean-Baptiste et al. (2017) Jean-Baptiste I., Di Matteo P., Haywood M., Gómez A., Montuori M., Combes F., Semelin B., 2017, A&A, 604, A106
  * Jurcsik & Kovacs (1996) Jurcsik J., Kovacs G., 1996, A&A, 312, 111
  * Jurić et al. (2008) Jurić M., et al., 2008, ApJ, 673, 864
  * Karczmarek et al. (2017) Karczmarek P., Wiktorowicz G., Iłkiewicz K., Smolec R., Stępień K., Pietrzyński G., Gieren W., Belczynski K., 2017, MNRAS, 466, 2842
  * Kervella et al. (2019) Kervella P., et al., 2019, A&A, 623, A117
  * Kinman et al. (1966) Kinman T. D., Wirtanen C. A., Janes K. A., 1966, ApJS, 13, 379
  * Koposov et al. (2017) Koposov S. E., Belokurov V., Torrealba G., 2017, MNRAS, 470, 2702
  * Koposov et al. (2019) Koposov S. E., et al., 2019, MNRAS, 485, 4726
  * Koppelman et al. (2018) Koppelman H., Helmi A., Veljanoski J., 2018, ApJ, 860, L11
  * Koppelman et al. (2020) Koppelman H. H., Bos R. O. Y., Helmi A., 2020, arXiv e-prints, p. arXiv:2006.07620
  * Kormendy & Kennicutt (2004) Kormendy J., Kennicutt Robert C. J., 2004, ARA&A, 42, 603
  * Kukarkin (1949) Kukarkin B. V., 1949, The study of the structure and evolution of stellar systems
  * Kunder et al. (2016) Kunder A., et al., 2016, ApJ, 821, L25
  * Kunder et al. (2017) Kunder A., et al., 2017, AJ, 153, 75
  * Kunder et al. (2020) Kunder A., et al., 2020, AJ, 159, 270
  * Lacey (1984) Lacey C. G., 1984, MNRAS, 208, 687
  * Lancaster et al. (2019) Lancaster L., Koposov S. E., Belokurov V., Evans N. W., Deason A. J., 2019, MNRAS, 486, 378
  * Laporte et al. (2018) Laporte C. F. P., Johnston K. V., Gómez F. A., Garavito-Camargo N., Besla G., 2018, MNRAS, 481, 286
  * Laporte et al. (2019) Laporte C. F. P., Minchev I., Johnston K. V., Gómez F. A., 2019, MNRAS, 485, 3134
  * Layden (1994) Layden A. C., 1994, AJ, 108, 1016
  * Layden (1995a) Layden A. C., 1995a, AJ, 110, 2288
  * Layden (1995b) Layden A. C., 1995b, AJ, 110, 2312
  * Lee et al. (1994) Lee Y.-W., Demarque P., Zinn R., 1994, ApJ, 423, 248
  * Lindegren et al. (2018) Lindegren L., et al., 2018, A&A, 616, A2
  * Liu (1991) Liu T., 1991, PASP, 103, 205
  * Liu et al. (2020) Liu G. C., et al., 2020, ApJS, 247, 68
  * López-Corredoira & Molgó (2014) López-Corredoira M., Molgó J., 2014, A&A, 567, A106
  * Mackereth et al. (2019a) Mackereth J. T., et al., 2019a, MNRAS, 482, 3426
  * Mackereth et al. (2019b) Mackereth J. T., et al., 2019b, MNRAS, 489, 176
  * Magurno et al. (2018) Magurno D., et al., 2018, ApJ, 864, 57
  * Maiolino et al. (2017) Maiolino R., et al., 2017, Nature, 544, 202
  * Majewski et al. (2017) Majewski S. R., et al., 2017, AJ, 154, 94
  * Marsakov et al. (2018) Marsakov V. A., Gozha M. L., Koval V. V., 2018, Astronomy Reports, 62, 50
  * Marsakov et al. (2019) Marsakov V. A., Gozha M. L., Koval’ V. V., 2019, Astronomy Reports, 63, 203
  * Martig et al. (2014) Martig M., Minchev I., Flynn C., 2014, MNRAS, 443, 2452
  * Mateu & Vivas (2018) Mateu C., Vivas A. K., 2018, MNRAS, 479, 211
  * Mateu et al. (2018) Mateu C., Read J. I., Kawata D., 2018, MNRAS, 474, 4112
  * McWilliam & Zoccali (2010) McWilliam A., Zoccali M., 2010, ApJ, 724, 1491
  * Miceli et al. (2008) Miceli A., et al., 2008, ApJ, 678, 865
  * Michel-Dansac et al. (2011) Michel-Dansac L., Abadi M. G., Navarro J. F., Steinmetz M., 2011, MNRAS, 414, L1
  * Minchev et al. (2009) Minchev I., Quillen A. C., Williams M., Freeman K. C., Nordhaus J., Siebert A., Bienaymé O., 2009, MNRAS, 396, L56
  * Moetazedian & Just (2016) Moetazedian R., Just A., 2016, MNRAS, 459, 2905
  * Morrison et al. (2009) Morrison H. L., et al., 2009, ApJ, 694, 130
  * Muraveva et al. (2018) Muraveva T., Delgado H. E., Clementini G., Sarro L. M., Garofalo A., 2018, MNRAS, 481, 1195
  * Myeong et al. (2018a) Myeong G. C., Evans N. W., Belokurov V., Sand ers J. L., Koposov S. E., 2018a, ApJ, 856, L26
  * Myeong et al. (2018b) Myeong G. C., Evans N. W., Belokurov V., Sand ers J. L., Koposov S. E., 2018b, ApJ, 863, L28
  * Naidu et al. (2020) Naidu R. P., Conroy C., Bonaca A., Johnson B. D., Ting Y.-S., Caldwell N., Zaritsky D., Cargile P. A., 2020, arXiv e-prints, p. arXiv:2006.08625
  * Necib et al. (2019) Necib L., Lisanti M., Belokurov V., 2019, ApJ, 874, 3
  * Nemec et al. (1994) Nemec J. M., Nemec A. F. L., Lutz T. E., 1994, AJ, 108, 222
  * Nemec et al. (2011) Nemec J. M., et al., 2011, MNRAS, 417, 1022
  * Nemec et al. (2013) Nemec J. M., Cohen J. G., Ripepi V., Derekas A., Moskalik P., Sesar B., Chadid M., Bruntt H., 2013, ApJ, 773, 181
  * Ness et al. (2013) Ness M., et al., 2013, MNRAS, 430, 836
  * Nissen & Schuster (2010) Nissen P. E., Schuster W. J., 2010, A&A, 511, L10
  * Oort & Plaut (1975) Oort J. H., Plaut L., 1975, A&A, 41, 71
  * Oosterhoff (1939) Oosterhoff P. T., 1939, The Observatory, 62, 104
  * Oosterhoff (1944) Oosterhoff P. T., 1944, Bull. Astron. Inst. Netherlands, 10, 55
  * Pedregosa et al. (2011) Pedregosa F., et al., 2011, Journal of Machine Learning Research, 12, 2825
  * Pietrukowicz et al. (2015) Pietrukowicz P., et al., 2015, ApJ, 811, 113
  * Pietrzyński et al. (2012) Pietrzyński G., et al., 2012, Nature, 484, 75
  * Preston (1959) Preston G. W., 1959, ApJ, 130, 507
  * Price-Whelan et al. (2015) Price-Whelan A. M., Johnston K. V., Sheffield A. A., Laporte C. F. P., Sesar B., 2015, MNRAS, 452, 676
  * Pritzl et al. (2000) Pritzl B., Smith H. A., Catelan M., Sweigart A. V., 2000, ApJ, 530, L41
  * Prudil et al. (2019a) Prudil Z., Dékány I., Catelan M., Smolec R., Grebel E. K., Skarka M., 2019a, MNRAS, 484, 4833
  * Prudil et al. (2019b) Prudil Z., Skarka M., Liška J., Grebel E. K., Lee C. U., 2019b, MNRAS, 487, L1
  * Prudil et al. (2019c) Prudil Z., Dékány I., Grebel E. K., Catelan M., Skarka M., Smolec R., 2019c, MNRAS, 487, 3270
  * Prudil et al. (2020) Prudil Z., Dékány I., Grebel E. K., Kunder A., 2020, MNRAS, 492, 3408
  * Ramos et al. (2020) Ramos P., Mateu C., Antoja T., Helmi A., Castro-Ginard A., Balbinot E., Carrasco J. M., 2020, A&A, 638, A104
  * Renaud et al. (2020) Renaud F., Agertz O., Read J. I., Ryde N., Andersson E. P., Bensby T., Rey M. P., Feuillet D. K., 2020, arXiv e-prints, p. arXiv:2006.06011
  * Rimoldini et al. (2019) Rimoldini L., et al., 2019, A&A, 625, A97
  * Robin et al. (2012) Robin A. C., Marshall D. J., Schultheis M., Reylé C., 2012, A&A, 538, A106
  * Saha (1985) Saha A., 1985, ApJ, 289, 310
  * Salvatier et al. (2016) Salvatier J., Wiecki T., Fonnesbeck C., 2016, PeerJ Computer Science, 2, e55
  * Sandage (1982) Sandage A., 1982, ApJ, 252, 553
  * Sanders & Das (2018) Sanders J. L., Das P., 2018, MNRAS, 481, 4093
  * Savino et al. (2020) Savino A., Koch A., Prudil Z., Kunder A., Smolec R., 2020, arXiv e-prints, p. arXiv:2006.12507
  * Schlegel et al. (1998) Schlegel D. J., Finkbeiner D. P., Davis M., 1998, ApJ, 500, 525
  * Schönrich (2012) Schönrich R., 2012, MNRAS, 427, 274
  * Schönrich & Dehnen (2018) Schönrich R., Dehnen W., 2018, MNRAS, 478, 3809
  * Schönrich et al. (2010) Schönrich R., Binney J., Dehnen W., 2010, MNRAS, 403, 1829
  * Schönrich et al. (2011) Schönrich R., Asplund M., Casagrande L., 2011, MNRAS, 415, 3807
  * Schönrich et al. (2012) Schönrich R., Binney J., Asplund M., 2012, MNRAS, 420, 1281
  * Searle & Zinn (1978) Searle L., Zinn R., 1978, ApJ, 225, 357
  * Sellwood & Carlberg (1984) Sellwood J. A., Carlberg R. G., 1984, ApJ, 282, 61
  * Sesar et al. (2007) Sesar B., et al., 2007, AJ, 134, 2236
  * Sesar et al. (2013) Sesar B., et al., 2013, ApJ, 776, 26
  * Sesar et al. (2017) Sesar B., et al., 2017, AJ, 153, 204
  * Sharma et al. (2020) Sharma S., et al., 2020, arXiv e-prints, p. arXiv:2004.06556
  * Simion et al. (2014) Simion I. T., Belokurov V., Irwin M., Koposov S. E., 2014, MNRAS, 440, 161
  * Simion et al. (2019) Simion I. T., Belokurov V., Koposov S. E., 2019, MNRAS, 482, 921
  * Sit & Ness (2020) Sit T., Ness M., 2020, arXiv e-prints, p. arXiv:2006.01158
  * Skowron et al. (2019) Skowron D. M., et al., 2019, Science, 365, 478
  * Smith (1984) Smith H. A., 1984, PASP, 96, 505
  * Smith et al. (2009) Smith M. C., et al., 2009, MNRAS, 399, 1223
  * Smolec (2005) Smolec R., 2005, Acta Astron., 55, 59
  * Soszyński et al. (2009) Soszyński I., et al., 2009, Acta Astron., 59, 1
  * Soszyński et al. (2014) Soszyński I., et al., 2014, Acta Astron., 64, 177
  * Spitzer & Schwarzschild (1951) Spitzer Lyman J., Schwarzschild M., 1951, ApJ, 114, 385
  * Stetson et al. (2014) Stetson P. B., Fiorentino G., Bono G., Bernard E. J., Monelli M., Iannicola G., Gallart C., Ferraro I., 2014, PASP, 126, 616
  * Strömberg (1946) Strömberg G., 1946, ApJ, 104, 12
  * Suntzeff et al. (1991) Suntzeff N. B., Kinman T. D., Kraft R. P., 1991, ApJ, 367, 528
  * Taam et al. (1976) Taam R. E., Kraft R. P., Suntzeff N., 1976, ApJ, 207, 201
  * Thomas et al. (2019) Thomas G. F., et al., 2019, MNRAS, 483, 3119
  * Tian et al. (2019) Tian H., Liu C., Xu Y., Xue X., 2019, ApJ, 871, 184
  * Ting & Rix (2019) Ting Y.-S., Rix H.-W., 2019, ApJ, 878, 21
  * Torrealba et al. (2015) Torrealba G., et al., 2015, MNRAS, 446, 2251
  * Torrealba et al. (2019) Torrealba G., et al., 2019, MNRAS, 488, 2743
  * Veilleux et al. (2020) Veilleux S., Maiolino R., Bolatto A. D., Aalto S., 2020, A&ARv, 28, 2
  * Velazquez & White (1999) Velazquez H., White S. D. M., 1999, MNRAS, 304, 254
  * Venn et al. (2004) Venn K. A., Irwin M., Shetrone M. D., Tout C. A., Hill V., Tolstoy E., 2004, AJ, 128, 1177
  * Vivas & Zinn (2006) Vivas A. K., Zinn R., 2006, AJ, 132, 714
  * Vivas et al. (2001) Vivas A. K., et al., 2001, ApJ, 554, L33
  * Walker & Terndrup (1991) Walker A. R., Terndrup D. M., 1991, ApJ, 378, 119
  * Watkins et al. (2009) Watkins L. L., et al., 2009, MNRAS, 398, 1757
  * Wegg & Gerhard (2013) Wegg C., Gerhard O., 2013, MNRAS, 435, 1874
  * Wegg et al. (2019) Wegg C., Gerhard O., Bieth M., 2019, MNRAS, 485, 3296
  * Wenger et al. (2000) Wenger M., et al., 2000, A&AS, 143, 9
  * Wetzel et al. (2016) Wetzel A. R., Hopkins P. F., Kim J.-h., Faucher-Giguère C.-A., Kereš D., Quataert E., 2016, ApJ, 827, L23
  * Widrow et al. (2012) Widrow L. M., Gardner S., Yanny B., Dodelson S., Chen H.-Y., 2012, ApJ, 750, L41
  * Wielen (1977) Wielen R., 1977, A&A, 60, 263
  * Xu et al. (2015) Xu Y., Newberg H. J., Carlin J. L., Liu C., Deng L., Li J., Schönrich R., Yanny B., 2015, ApJ, 801, 105
  * Xue et al. (2015) Xue X.-X., Rix H.-W., Ma Z., Morrison H., Bovy J., Sesar B., Janesh W., 2015, ApJ, 809, 144
  * Yu et al. (2020) Yu S., et al., 2020, MNRAS, 494, 1539
  * Zinn & West (1984) Zinn R., West M. J., 1984, ApJS, 55, 45
  * Zinn et al. (2014) Zinn R., Horowitz B., Vivas A. K., Baltay C., Ellman N., Hadjiyska E., Rabinowitz D., Miller L., 2014, ApJ, 781, 22
  * Zinn et al. (2020) Zinn R., Chen X., Layden A. C., Casetti-Dinescu D. I., 2020, MNRAS, 492, 2161
  * Zoccali et al. (2003) Zoccali M., et al., 2003, A&A, 399, 931
  * de Boer et al. (2018) de Boer T. J. L., Belokurov V., Koposov S. E., 2018, MNRAS, 473, 647

##  Appendix A Photometric metallicity estimate

Most of the stars in the SOS Gaia catalogue have photometric metallicities
(Clementini et al., 2019) estimated through the non-linear relation by Nemec
et al. (2013). The Nemec et al. (2013) relation has been fitted to a small
sample of stars and it does not seem to generalise well enough on larger
sample. In particular, it assigns to a group of RRL with intermediate-large
periods and large $\Phi_{31}$ high metallicities ($[Fe/H]\gtrsim-0.5$) that
are likely artefacts (see e.g. Fig. 17). Moreover, the relation is based on
the Kepler magnitude band and a number of auxiliary relations have to be used
to translate the $\Phi_{31}$ from the original band to the Gaia one
(Clementini et al., 2019); additionally the value of $\Phi_{31}$ can change if
we use a different number of harmonics to decompose the light curve. For all
these reasons, we decide to find a relation based solely on the light curve
properties reported in the Gaia SOS catalogue. For the purpose of our analysis
we cross-matched the subsample of RRab stars with complete SOS light curve
information in our Gclean catalogue (see Sec. 2) with different spectroscopic
sample of RRab stars with spectroscopic metallicity estimate: Layden (1994)
(84 stars), Marsakov et al. (2018) (76 stars), Nemec et al. (2013) (21 stars),
Zinn et al. (2020) (149 stars, mostly based on the sample by Dambis et al.
2013 containing also the 84 stars in Layden 1994). Concerning the RRc stars we
follow Nemec et al. (2013) considering the RRL in globular clusters (50 stars)
assigning them the metallicity of the cluster they belong. We use the catalog
of Gaia objected associated with Globular Clusters in Gaia Collaboration et
al. (2018d), while the Globular Cluster metallicity are taken from Harris
(1996). We consider the old Harris (1996) compilation because the
metallicities are reported in the Zinn & West (1984) metallicity scale instead
of the Carretta et al. (2009) scale used in the more recent Harris (2010)
catalogue. The Zinn & West (1984) scale is the same metallicity scale of
spectroscopic catalogs and the absolute magnitude-metallicity relation used in
this work has been calibrated on this same scale (Muraveva et al., 2018).

Figure 16: Best fit linear relation $[Fe/H]\propto
a\times~{}P+b\times\Phi_{31}$ for RRab (top panel) and RRc stars (bottom
panel). The spectroscopic metallicities are from Layden (1994) and Harris
(1996) for RRab and RRc stars, respectively. Periods and phase difference
$\Phi_{31}$ values are from the SOS Gaia catalogue. The solid black lines show
the median of the posterior distributions of the relations, while the gray
lines are randomly sampled from the same distributions. The black dashed lines
indicate the intrinsic scatter. The best fit relations are given in Equations
3 and 4.

We perform a large number of tests using both linear (e.g. Jurcsik & Kovacs
1996; Smolec 2005) and non linear relations (e.g. Nemec et al. 2013) and
investigating different combinations of light curve and stellar properties.
Initially, we evaluate the feature relevance through a random forest
regression of the metallicity using the scikit-learn python module (Pedregosa
et al., 2011). In practice we consider as feature: the period $P$ (fundamental
period for RRab and first overtone period for RRc), the phase difference
between the third or second light curve harmonics with respect to the
fundamental one, the amplitude, the ratio between the amplitude of third or
second light curve harmonics with respect to the fundamental one and the
stellar color. In order to check possible biases and artefacts we also add the
number of Gaia observations, the mean $G$ magnitude and the $RUWE$ to the
group of features. For both RRab and RRc samples the most relevant feature is
by far the period $P$, followed by the phase difference $\Phi_{31}$. We do not
use the random forest method to estimate the metallicity since our training
sample is relatively small and, considering the large number of parameters
involved, it is very likely to produce a significant variance or overfit
problem. Instead we fit the relations using a Bayesian approach taking into
account the uncertainties of all the used features. In each tested relation we
consider also the presence of an intrinsic scatter. We sample the posterior of
the relation parameters exploiting the Hamiltonian MCMC technique making use
of the python module PYMC3 (Salvatier et al., 2016). The performance of the
various relations are analysed considering: $i$) fit residuals, $ii$)
comparison with metallicities of RRL stars in Globular clusters (association
with GC from Gaia Collaboration et al. 2018d, metallicites estimate from
Harris 1996), $iii$) comparison with the spectroscopic metallicities of the
RRL stars in the solar neighbours,the halo and the bulge taken from the
crossmatch with the Magurno et al. (2018), Liu et al. (2020) and Savino et al.
(2020) samples (see Fig. 17), $iv)$ comparison of the distance moduli derived
using the $M_{G}-[Fe/H]$ relation by Muraveva et al. (2018) with the distance
moduli of the Magellanic Clouds141414We used the median of the distance moduli
estimates taken from NED (NASA/IPAC Extragalactic Database,
http://ned.ipac.caltech.edu).. We conclude that the optimal fit, both for RRab
and RRc stars, is obtained with a linear relation with $P$ and $\Phi_{31}$,
very little improvements can be obtained using non-linearity or adding
parameters to the relation. As already noted by Jurcsik & Kovacs (1996);
Smolec (2005); Nemec et al. (2011), the major issue is a moderate systematic
trend of the residuals as a function of the spectroscopic metallicities: the
relation tends to overestimate (underestimate) the metallicity at the metal-
poor (metal-rich) end. Anyhow, this problem is present with the same
significance also with more complex models. This is likely due to the lack of
calibrators at both ends of the metallicity distribution. Among the various
samples of RRab, the results of the fit are very similar except for the Nemec
sample, but it contains a small number of stars covering a narrower range of
metallicites with respect to the other samples. Therefore, we adopt as final
relations (Equation 3 and 4), the linear relation in $P$ and $\Phi_{31}$
obtained with the Layden (1994) sample (for RRab stars). This choice is
motivated by the fact that it is not a collection of different catalogues and
it reports a metallicities uncertainty for each star. Fig. 16 shows the best-
fit relations. The metallicity interval of the fit training set ranges from
-2.51 to 0.08 for the RRabs stars and from -2.37 to -0.55 for the RRc stars.
Only a very small portion (mostly RRc stars) of our Gclean sample (see Sec.
2.2) has metallicities extrapolated outside these ranges: 396 at the metal-
poor tail (93 RRab, 303 RRc, 295 in the halo subsample, 6 in the disc
subsample), 105 at the metal-rich end (26 RRab, 79 RRc, 15 in the halo
subsample, 42 in the disc subsample). These numbers are small enough to have
negligible effects on our outcomes as confirmed by the results obtained with
the SA sample (see e.g. Fig. 12 and Fig. 7) that contains only $0.3\%$ of
stars with extrapolated metallicities. Moreover, the fit procedure “naturally"
assigns larger errors to extrapolated metallicities and the implemented linear
function limits uncontrolled behaviour outside the range of calibrators.

Compared to the photometric metallicities reported in the Gaia SOS catalogue
our estimate perform better both on estimating the absolute magnitude of the
stars in the Magellanic Clouds (using the $M_{G}-[Fe/H]$ relation by Muraveva
et al. 2018) and compared to the RRL sample of spectroscopic metallicity
obtained by Savino et al. (2020), Liu et al. (2020) and Magurno et al. (2018).
Fig. 17 shows that the distribution of SOS photometric metallicities
significantly differs from the spectroscopic ones in both shape and centroid
position (see also Hajdu 2019). In particular, considering the bulge sample,
the SOS distribution peaks at a very metal-rich value of $[Fe/H]\approx-0.5$,
while the peak of the spectroscopic metallicity is $[Fe/H]\approx-1.5$. The
photometric metallicity estimated with our relation shows a more similar
distribution with a coincident but narrower peak. The narrow distribution of
the photometric metallicities is due to the already discussed problem of
overestimating/underestimating the metallicities at the edge of the
distribution. Considering the Liu et al. (2020) sample, our metallicity
distribution is slightly offset from the spectroscopic distribution, but
overall the distribution widths are very similar. On the contrary, the SOS
distribution is much more spread containing a significant number of metal-rich
stars ($[Fe/H]>-1$). The peak of the distribution of our photometric
metallicities is consistent with the peak of the high resolution spectroscopic
metallicities in Magurno et al. (2018), but in this case the differences in
the tails are more significant. For the same sample, the SOS photometric
metallicities cover the same range of the spectroscopic metallicities, but
their distribution is much flatter without a clear peak and with an over-
abundance of very metal-rich stars.

Figure 17: Comparison between the distribution of photometric (this work,
blue; Gaia SOS orange) and spectroscopic (dashed-black) metallicity values for
two samples of RRL. Top panel: cross-match between the bulge RRL sample in
Savino et al. (2020) and Gaia SOS with lightcurve information (212 stars).
Middle panel: cross-match between the RRL sample (mostly in the halo) from Liu
et al. (2020) and Gaia SOS with lightcurve information (3153 stars). Bottom
panel: cross-match between the RRL sample (local field) from Magurno et al.
(2018) and the Gaia SOS with lightcurve information (64 stars). Vertical lines
indicate the median of each distribution.

Finally, we test that the use of the constant absolute magnitude
$M_{G}=0.64\pm 0.25$ for both RRab and RRc stars (see e.g. Iorio & Belokurov
2019) is a good approximation when light curve properties are not
available.The associated error $\delta M_{G}=0.25$ is a robust and
conservative estimate that can absorb both random and systematic uncertainties
(e.g. RRL type, metallicity) giving a error on heliocentrinc distance of about
$13\%$.

##  Appendix B Rotation Matrix

The rotation matrix $\mathbf{R}$ to pass from velocities in Spherical
$\bm{V}_{\mathrm{sph}}=(V_{\mathrm{r}},V_{\mathrm{\theta}},V_{\mathrm{\phi}})$
or Cylindrical
$\bm{V}_{\mathrm{cyl}}=(V_{\mathrm{R}},V_{\mathrm{z}},V_{\mathrm{\phi}})$
Galactocentric coordinates to the velocities in the observed frame of
reference $\bm{V}_{\mathrm{sky}}=(V_{\mathrm{los}},V_{\ell},V_{b})$ can be
obtained with the matrix product

$\mathbf{R}=\mathbf{R}_{\mathrm{c}}\cdot\mathbf{R}_{\mathrm{s,sph/cyl}}$ (12)

where $\mathbf{R}_{\mathrm{c}}$ is the rotation matrix to pass from the
Galactic cartesian velocities
$\bm{V}_{\mathrm{car}}=(V_{\mathrm{x}},V_{\mathrm{y}},V_{\mathrm{z}})$ to the
the observed velocities, while $\mathbf{R}_{\mathrm{s,sph}}$ and
$\mathbf{R}_{\mathrm{s,cyl}}$ are the rotation matrix to pass from Galactic
cartesian velocities to Galactic spherical and cylidrincal velocities,
respectively. The matrix $\mathbf{R}_{\mathrm{c}}$ is defined as

$\mathbf{R}_{\mathrm{c}}=\begin{bmatrix}\cos b\cos\ell&\cos b\sin\ell&\sin
b\\\ -\sin\ell&\cos\ell&0\\\ -\sin b\cos\ell&-\sin b\sin\ell&\cos
b\end{bmatrix},$ (13)

while the matrices $\mathbf{R}_{\mathrm{s}}$ are defined as

$\mathbf{R}_{\mathrm{s,sph}}=\begin{bmatrix}\Gamma\cos\theta\cos\phi&-\Gamma\sin\theta\cos\phi&-\Gamma\sin\phi\\\
\cos\theta\sin\phi&-\sin\theta\sin\phi&-\cos\theta\\\
\sin\theta&\cos\theta&0\end{bmatrix}$ (14)

and

$\mathbf{R}_{\mathrm{s,cyl}}=\begin{bmatrix}\Gamma\cos\phi&0&-\Gamma\sin\phi\\\
\sin\phi&0&-\cos\phi\\\ 0&1&0\end{bmatrix}.$ (15)

The factor $\Gamma$ is equal to 1 for a right-handed Galactocentric frame of
reference or to -1 for a left-handed Galactocentric frame of reference (as the
one used in this work). The angular coordinates $\theta$ and $\phi$ are the
zenithal and azimuthal angle respectively, while $b$ and $\ell$ are the
Galactic sky coordinates (see Sec. 2.1).