# Personalized Restoration via Dual-Pivot Tuning

Pradyumna Chari1,† Sizhuo Ma2 Daniil Ostashev2 Achuta Kadambi1 Gurunandan
Krishnan2 Jian Wang2,⋆ Kfir Aberman2 1University of California, Los Angeles
2Snap Inc.

###### Abstract

Generative diffusion models can serve as a prior which ensures that solutions
of image restoration systems adhere to the manifold of natural images.
However, for restoring facial images, a personalized prior is necessary to
accurately represent and reconstruct unique facial features of a given
individual. In this paper, we propose a simple, yet effective, method for
personalized restoration, called Dual-Pivot Tuning — a two-stage approach that
personalize a blind restoration system while maintaining the integrity of the
general prior and the distinct role of each component. Our key observation is
that for optimal personalization, the generative model should be tuned around
a fixed text pivot, while the guiding network should be tuned in a generic
(non-personalized) manner, using the personalized generative model as a fixed
“pivot”. This approach ensures that personalization does not interfere with
the restoration process, resulting in a natural appearance with high fidelity
to the person’s identity and the attributes of the degraded image. We
evaluated our approach both qualitatively and quantitatively through extensive
experiments with images of widely recognized individuals, comparing it against
relevant baselines. Surprisingly, we found that our personalized prior not
only achieves higher fidelity to identity with respect to the person’s
identity, but also outperforms state-of-the-art generic priors in terms of
general image quality. Project webpage: https://personalized-
restoration.github.io

Figure 1:  Given a degraded image of an individual’s face (a), diffusion-based
blind restoration approaches [25] may not retain the individual’s identity
(b). However, with a few reference images (bottom right), our _dual-pivot
tuning_ technique (c) can reconstruct the face while maintaining high identity
fidelity to the individual and without any perceivable loss in fidelity to the
degraded input.  $\dagger$$\dagger$footnotetext: Work partially done during an
internship at Snap Inc.$\star$$\star$footnotetext: Corresponding author

##  1 Introduction

(a) Input

(b) DiffBIR [25]

(c) Baseline 1 (Pivot 1 only)

(d) Baseline 2 (Pivot 2 only)

(e) Ours

(f) Reference image

Figure 2: Restoration Baselines. Given a real, degraded input image (a)
features a person whose identity is referenced in another image (f), a
diffusion-based blind restoration method [25] with a general (non-
personalized) prior, fails to preserve the person’s identity (b). When fine-
tuning the text-to-image prior with a text pivot only [35], the system ignores
the personalized generator when reintegrated into the system (c), while
personalizing the feature encoder with the generative model as a pivot only
leads to lack of detail in the result (tattoo in the first row, detail such as
beard, general high frequencies in the second row), along with absence of
generalization across identities (d). In contrast, our method (e) effectively
incorporates the individual’s identity into the restoration process while
retaining a quality comparable to the base model (b).

Image restoration is an extensively researched problem, characterized by its
inherent ill-posed nature. The goal of the restoration task is to find
visually plausible and natural images that maintain perceptual fidelity to the
degraded input image [47]. In blind restoration scenarios where no prior
information about the subject or the degradation is available, a prior
describing the manifold of natural images is needed. However, for face image
restoration, having an identity prior is necessary to ensure that the output
image remains within a manifold that accurately represents the distinctive
facial features of the individual in the degraded image. This sets the basis
for prior work in reference-based face image restoration [23, 24, 29].

Text-to-Image Diffusion Models [34, 37] have revolutionized image synthesis,
enabling the generation of images from textual descriptions. These models act
as versatile generative priors for various downstream tasks, and were recently
explored in the context of blind image restoration, enhancing the naturalness
of restored images [25, 43]. Moreover, the ability to personalize these
diffusion models with just a few reference images opens new avenues for
tailored content creation [35, 13]. Despite these advancements, effectively
integrating personalization into a diffusion-based blind restoration systems
remains an open challenge.

To highlight the challenges associated with personalizing a blind restoration
system, Fig. 2 demonstrates how naive personalization approaches fall short.
As shown in Fig. 2 (c), a personalized text-to-image model that simply
replaces the general one in a blind restoration system can be disregarded by
the unconditional system. Alternatively, naive personalization of the image
conditioning encoder (from [25], for example) disrupts the natural image
prior, leading to a lack of detail in the restored image as demonstrated in
Fig. 2 (d).

In this paper, we propose a simple technique for personalizing a text-to-image
prior within the context of a blind image restoration framework. Given a small
set ($\sim$10) of high-quality images of a person, our goal is to restore
their other degraded images, ensuring that the results exhibit (1) high
fidelity to the identity of the person, (2) high fidelity to the degraded
image, (3) natural appearance.

Our key idea, coined dual-pivot tuning, is to fine-tune the prior within the
context of a blind image restoration system, while preserving the integrity of
the general prior and the distinct roles of each component within the system.
Specifically, we propose a two-step solution, to personalize a general blind
image restoration system which consists of two main components: a diffusion-
based generative prior, and a guiding image encoder.

In the first step, we personalize the diffusion-based generative prior in the
context of the blind restoration system, while using a text prompt (for
example, “a photo of a [v] man”) as a _textual pivot_ \- a fixed token that is
held constant during the fine-tuning process [35]. However, in our case the
fine-tuning is performed with the guiding image encoder such that the
personalized prior learns to respect the attributes of the guiding image. In
the second step, we fine-tune the guiding encoder to align with the
personalized, “shifted”, prior of the diffusion model, thereby retargeting its
functionality. During this phase, we aim to maintain the identity-agnostic
nature of the encoder, allowing its applicability across various personalized
models. Therefore, we fine-tune it with general face images (not of the
specific individual). We refer to the fixed personalized network as a pivot,
around which we adjust the weights of the guiding encoder.

We find both of these operations, in this sequence, to be essential. While the
textual pivot enables identity injection without losing the general face prior
of the base model in the restoration system, fine-tuning around the network
pivot leads to better utilization of the guiding encoder and higher fidelity
to the input image features. Furthermore, we leverage the diffusion process’s
characteristics, noting that identity formation occurs later in the process,
where it primarily shapes finer details rather than the initial, noisier phase
that sets coarse image features. Our approach demonstrates that
personalization can be effectively applied at specific stages of the diffusion
process which can reduce the expansive fine-tuning time by approximately
$\times 2$.

We compare our method against few-shot learning baselines and evaluate them
using publicly available images of well-known figures, leveraging our pre-
existing knowledge of their features. Our experimenters shows that our method
reconstruct key facial features of the subject in the reference images while
maintaining high fidelity to the original degraded image, outperforming others
quantitatively and qualitatively. Moreover, user studies, interestingly,
confirm that this personalization aspect contributes significantly not only to
identity preservation but also to the perceived overall quality improvement of
the generated images. Fig. 1 shows one example that demonstrates how our
method is able to surpass existing diffusion-based blind face restoration
methods [25] in terms of identity retention while maintaining the same quality
of face image restoration.

To the best of our knowledge, this is the first approach to leverage a
personalized diffusion prior for restoration tasks.

##  2 Related Work

###  2.1 Blind Face Image Restoration

Distinct from general scene image restoration, face image restoration
typically leverages facial priors to achieve superior results. Depending on
how these facial priors are utilized, previous methods can be broadly
categorized into three main classes. (1) Geometric prior: These approaches
incorporate geometric cues, such as facial landmarks [9, 5, 19], parsing maps
[7, 38, 50], and component heatmaps [52], into the network’s design. (2)
Dictionary prior: In this method, a dictionary is first learned from a
collection of face images, either in the image space [21] or feature space
[48, 55, 15, 56]. Subsequently, a degraded image is reconstructed using high-
quality words from this dictionary. (3) Generative prior: This category
encompasses techniques like GAN (Generative Adversarial Network) prior [6, 51,
46, 27] or diffusion prior [53, 25, 49]. Among these methods, approaches
falling under categories (2) and (3) have demonstrated the most promising
results. Notable algorithms in this context include GFP-GAN [46], CodeFormer
[56], and DiffBIR [25].

However, blind face image restoration faces a significant challenge known as
the quality-fidelity tradeoff [56, 25]. This arises from the inherent
limitations in the information available in the original image. Striking the
right balance between generating high-quality results while staying faithful
to the original image can be a delicate task. Generating results with too
little modification may not yield a significant improvement in quality, while
excessive generation can lead to a departure from the identity of the original
image. Our proposed method differs from these methods: using personalized
diffusion-based methods, we are able to push the tradeoff frontiers by
achieving restoration quality while retaining fidelity with respect to
identity.

###  2.2 Reference-Based Face Image Restoration

A high-quality reference image of the same person can greatly benefit face
image restoration and help avoid the need of the tradeoff. Depending on the
number of references used, such methods can be divided into two categories.
(1) Single-reference methods: Examples include GFRNet [20] and GWAINet [12].
(2) Multi-reference methods: Examples include ASFFNet [23], Wang _et al_.
[44], and DMDNet [24] ($\leq$ 10 reference images), and MyStyle [29] ($\sim$
100 images). It is evident that multi-reference approaches yield superior
results as they leverage more information. Notably, ASFFNet [23] selects an
optimal guidance, Wang _et al_. [44] employs pixel-wise weights for multiple
references, and DMDNet [24] constructs a dictionary from multiple references;
MyStyle [29], on the other hand, fine-tunes StyleGAN based on personal images.

Our proposed method also utilizes multiple reference images to aid
personalized restoration. However, there are several distinctions. We use a
diffusion-based personalized generative prior, while [20, 12, 23, 44] use
feedforward architectures. On the other hand, [29] uses a GAN-based prior,
however it requires on the order of 100 images for effective personalization
and strict spatial alignment of the face landmarks within each image, while
our method is able to operate with 10 reference images and has no restriction
on the alignment, operating directly on images in the wild. These distinctions
lead to higher quality restoration with less restrictions for our proposed
method.

###  2.3 Personalized Diffusion Models

Diffusion models [17, 39, 11] have notably excelled in the area of generating
images from text (T2I) [32, 37, 34] and many other visual computing tasks
[31]. Recent advancements in this field involve customization of these
established model through fine-tuning, aiming to enhance features like
controllability, customization, or to cater to specific applications. One
approach to customization involves modifying the T2I model itself [35, 8, 41,
40] or the text embedding process [13, 1, 42], using selected images. This
allows to personalize the generation of images based on a particular subject
or style, driven by textual input. More recently, neurons in a model specific
to a concept can be identified and manipulated specifically to enable sparse
personalization [26], methods for fast personalization of the model have been
proposed [14, 2, 36], and approaches for multisubject personalization [30, 3].
Alternatively, other methods involve adapting the T2I model to introduce new
conditioning factors. These modifications are either for purposes of image
modification [18, 4, 45] or for generating more controlled images [54, 28].

In this work, we tackle the task of contextual customization, where the goal
is to fine-tune a prior within the context of a system, preserving the
distinct roles of each component within the system, while customizing the
image prior to a specific subject.

##  3 Method

In this section, we outline our method for personalizing guided diffusion
models. We start by recapitulating the personalization process for text-to-
image models and the application of diffusion priors in blind image
restoration. Subsequently, we present our dual-pivot tuning: firstly,
employing text-based fine-tuning to embed identity-specific information within
diffusion priors, and secondly, the necessity of model-centric pivoting to
harmonize the guiding image encoder with the integrated personalized priors.
This is followed by an exposition on how our method’s inherent characteristics
facilitate accelerated per-identity personalization. A high-level summary of
the main idea behind our approach is illustrated in Fig. 3.

###  3.1 Personalizing Text-Guided Diffusion Models

Figure 3: High-level overview. Our approach aims to personalize a blind face
restoration system (left) and consists of two main steps: (1) text-anchored
personalization of the generative prior $\bf{G}$ within the context of the
system, and (2) adjusting $\mathbf{E}$ in retarget the encoder $\mathbf{E}$ in
the presence of the strong personalized prior in $\mathbf{G}_{p}$, while
guiding the network. Then, at inference time (3), our system embeds the
personalized prior and can generate output images with high fidelity to the
individual appearing in the reference images.

Given a text prompt $\mathbf{p}$, text-guided denoising diffusion models learn
to sequentially denoise samples of random noise $\epsilon\leavevmode\nobreak\
\sim\leavevmode\nobreak\ \mathcal{N}(\mathbf{0},\mathbf{I})$ into samples of
images $\bf{x}$. During training, a neural network
$\epsilon_{\theta}(\mathbf{x}_{t},\mathbf{p})$ is trained to predict
$\epsilon$ from a noisy version of the image
$\mathbf{x}_{t}=\alpha_{t}\mathbf{x}+\sigma_{t}\epsilon$, where $\alpha_{t}$
and $\sigma_{t}$ are noise scheduling parameters, and $t$ refers to the time
step in the diffusion process.

A common way to sample the network at inference time is by using the
classifier-free guidance technique [16], that sums up a conditional instance
of the model together with an unconditional one:

$\mathbf{G}_{\theta}(\mathbf{x},\mathbf{p})=(1+w)\epsilon_{\theta}(\mathbf{x}_{t},\mathbf{p})-w\epsilon_{\theta}(\mathbf{x}_{t},\varnothing),$
(1)

where $w$ is a guidance scale, and $\varnothing$ represents a null-text
prompt. These conditional and unconditional branches aim to strike a balance
between prompt-fidelity and diversity within the generated images.

(a) Input

(b) Seed 1

(c) Seed 2

(d) Seed 3

(e) Seed 4

(f) Seed 5

Figure 4: Identity variation. Given a fixed degraded input image, it can be
seen that different seeds applied to [25] result in images with different
identities. Please zoom in closer to observe differences, especially in terms
of eyes, teeth and so on.

Recent work by Ruiz et al. [35] introduced the ability to personalize text-
guided diffusion models using a few reference images of a subject, where the
personalization is performed by fine-tuning the model around a text anchor $p$
(typically designed as a “rare token”+“class noun”, such as ’a [v] dog’), so
the identity of the subject can be embedded into the diffusion prior, and
utilized once $\mathbf{p}$ appears in the condition during inference. In
practice, the weights, $\theta$, are optimized by minimizing

$\mathcal{L}_{\text{DB}}=\mathbb{E}_{\mathbf{x},\mathbf{p},\epsilon,\epsilon^{\prime},t}\left[\lvert\lvert\mathbf{G}_{\theta}(\mathbf{x}_{t},\mathbf{p})-\epsilon\lvert\lvert_{2}^{2}\right.\\\
\left.+\lvert\lvert\mathbf{G}_{\theta}(\mathbf{x}^{\text{pr}}_{t},\mathbf{p}^{\text{pr}})-\epsilon^{\prime}\lvert\lvert_{2}^{2}\right],$
(2)

where $\mathbf{x}^{\text{pr}}_{t}$ is drawn from a separate, prior
preservation dataset, and $\mathbf{p}^{\text{pr}}$ is the prior prompt (“class
noun”, such as ’a dog’). This structure allows the optimization to revolve
around a text pivot $\mathbf{p}$, without ruining the general prior of the
model [33].

We define the personalization operator of text-to-image diffusion models by

$\mathbf{M}_{\mathcal{W}}=\mathcal{P}\\{\mathbf{M},\mathcal{W}\\},$ (3)

where $\mathcal{P}\\{\cdot\\}$ is the operator, and $\mathbf{M}_{\mathcal{W}}$
is the personalized version of a model $\mathbf{M}$ which is fine-tuned with
$\mathcal{W}$ as a pivot. In our case, this pivot can be both a text prompt or
a network, as discussed subsequently.

###  3.2 Diffusion-Guided Image Restoration

Recent advances in blind face image restoration [25, 43] have attempted to
utilize a pretrained diffusion model (such as Stable Diffusion [34]) as
generative prior to guide the restoration process. Commonly, these pipelines
contain a diffusion-guided image restoration step that consist of two
components: (1) A general image prior in the form of text-to-image diffusion
model $\mathbf{G}(\cdot)$, that encodes the manifold of natural images and (2)
an encoder $\mathbf{E}(\cdot)$, that captures context information from the
degraded image and guides the generation process such that it maintains high
fidelity to the visual attributes of the degraded image. Guiding the diffusion
process with spatial features extracted from an image encoder E is a common
way to control the diffusion processes [54]. It should be noted that in case
of blind restoration, the general prior G plays a key role in dictating the
identity of the person. As demonstrated in Fig. 4, for a given degraded input
image (namely, fixed set of guiding features that are extracted from E),
different seeds in the input of $\mathbf{G}(\cdot)$, lead to different
identities in the output.

Our goal in this work is to personalize the general prior G such that the
restored face maintains high fidelity to the identity of the individual
without degrading the integrity of the general image prior and the distinct
roles of each component in the system.

We consider a general setting, where the encoder $\mathbf{E}(\cdot)$ takes a
low-quality image as an input and provides guidance for the diffusion model.
That is, the diffusion model receives conditioning from the encoder as well,
such that

$\hat{I}=\mathbf{G}(\epsilon,\mathbf{p},\mathbf{E}(I_{LQ})),$ (4)

where $I_{\text{LQ}}$ is a low quality image as input and $\mathbf{p}$ is the
text conditioning. Note that $I_{\text{LQ}}$ could be a degraded image [43],
or the output of a preliminary restoration model, like in [25]. In the context
of the blind image restoration setting, we will use the following notation:

$\hat{I}=\mathbf{B}(\mathbf{G}(\epsilon,\varnothing),\mathbf{E}(I_{\text{LQ}})),$
(5)

where $\mathbf{B}$ represents the blind restoration system. In this blind
restoration setting, the text conditioning is null (as in [25]).

###  3.3 Dual-Pivot Tuning

#### Step-1 - Textual Pivoting.

Theoretically, personalization of a generative prior can be decoupled from the
context of the system. In our case, we can personalize G with a pivot text
prompt $p$ independently of the blind restoration system and plug it back to
the framework, namely,

$\mathbf{B}_{p}=\\{\mathbf{E},\mathbf{G}_{p}\\},$ (6)

where $\mathbf{G}_{p}=\mathcal{P}\\{\mathbf{G},p\\}$. However, as demonstrated
in Fig. 2(c), if we do so, the restored images are unable to imbibe identity
information and the outputs resemble the images generated with G, the non-
personalized version of the model (Fig. 2(b)).

The occurrence of this issue can be attributed to the initial training of
$\mathbf{E}$, where it was coupled with the unconditional branch of
$\mathbf{G}$. As a result, $\mathbf{E}$ cannot exert influence on the
conditional branch of $\mathbf{G}$, which is responsible to contribute the
personalized features around the anchor prompt $p$.

We therefore find that for effective personalization of face image
restoration, text-based pivoting alone is not enough, and that $\mathbf{E}$
must be fine-tuned in conjunction with $\mathbf{G}_{p}$.

#### Step-2 - Model-based Pivoting.

We are introducing a second fine-tuning step for $\mathbf{E}$ within the
context of $\mathbf{G}_{p}$. Essentially, our objective is to fine-tune
$\mathbf{E}$ in a way that it relinquishes identity cues to the personalized
prior in $\mathbf{G}_{p}$ while focusing on other detail cues. In this
scenario, our pivot is not a text but a network ($\mathbf{G}_{p}$), which we
keep fixed during the optimization process to align the encoder with the
personalized prior.

To preserve the identity-agnostic role of the guiding encoder, we
intentionally aim to avoid personalizing it. One approach to achieve this is
by updating $\mathbf{E}$ across different individuals, allowing it to adapt to
the conditional branch of $\mathbf{G}_{p}$ while remaining agnostic to fine-
grained identity features, which should be determined by $\textbf{G}_{p}$, as
demonstrated in Fig. 4. This identity-agnostic retargeting concept bears
similarities to the Prior Preservation loss [35] in the context of pivotal-
tuning around a model.

Mathematically, with $\mathbf{G}_{p}$ as our starting point, we perform fine-
tuning on $\mathbf{E}$ as follows:

$\mathbf{B}_{p}=\mathcal{P}\\{\mathbf{B}_{\textbf{E}}=\\{\mathbf{G}_{p},\bf{E}\\},\mathbf{G}_{p}\\},$
(7)

where $\mathbf{B}_{\textbf{E}}$ represents the restoration system (with the
embedded personalized prior), and we are now optimizing the weights of E.

###  3.4 Speeding-Up Dual-Pivot Tuning

In general, personalization of generative models is a time-consuming endeavor.
Given our dual pivot tuning approach, personalization involves two finetuning
steps per identity: textual pivoting, followed by model-based pivoting. Hence
we next suggest steps to significantly reduce (by about 2x) fine-tuning time
of each pivot tuning.

#### Speeding up textual pivoting.

The presence of both conditional and unconditional branches within our
inference pipeline affords an additional unique opportunity. We noticed that
the initial steps of the restoration do not rely on identity as much, as these
steps largely focus on coarse geometry and semantic detail [42]. Therefore,
during inference, we observe that high-quality, identity-preserving
restoration can be achieved even when the initial denoising steps are only
unconditional, using a guidance scale of 0, followed by text-guided denoising
for later steps with a non-zero guidance scale. This is mathematically
represented as follows:

$\begin{split}\epsilon_{t+1}=&\mathbf{B}_{p}(\mathbf{x}_{t},\varnothing)\,\,|\,\,t<\gamma,\\\
\epsilon_{t+1}=w\cdot\mathbf{B}_{p}(\mathbf{x}_{t},p)+&(1-w)\mathbf{B}_{p}(\mathbf{x}_{t},\varnothing)\,\,|\,\,\gamma\leq
t<1.0,\end{split}$ (8)

where $\gamma$ denotes the fraction of denoising steps for which unconditional
inference is carried out. Note that in the above expression, we denote the
initial noise to be at $t=0$ and the restored image to be at $t=1$.

We demonstrate this in Fig. 5, where we restore the degraded image (a)
unconditionally for the first $\gamma=50\%$ of steps, and conditionally for
the remaining steps (c). Despite this, we find the restored image being
faithful to the reference image (d), as well as to the restored image obtained
through conditional restoration for all the steps (b). $\gamma=50\%$ is
identified experimentally. This observation enables us to reduce
personalization time for the textual pivoting by about 2x since the denoiser
no longer needs to be personalized for higher noise levels (since the
unconditional model suffices for those), and can, therefore, be trained with a
focus on relatively lower noise levels.

#### Speeding up model-based pivoting.

Coupling the generative prior $\mathbf{G}$ and the guiding encoder
$\mathbf{E}$ during the textual pivoting step can provide enocder context to
the personalized generative prior. To achieve this, we propose to fine-tune
$\mathbf{G}$ within the context of $\mathbf{B}$, while pivoting around the
prompt $p$, such that it leverages the conditioning cues from $\mathbf{E}$,
namely,

$\mathbf{B}_{p}=\mathcal{P}\\{\mathbf{B_{G}},p\\},$ (9)

where $\mathbf{B_{G}}$ denotes that we modify the weights of $\mathbf{G}$
within the context of $\mathbf{B}$. Specifically, this personalization pivots
around the text prompt $p$ in the context of image restoration. The
performance of this so-called in-context textual pivoting, and comparison with
the previously discussed out-of-context textual pivoting is shown in Fig. 10
and discussed in detail later: while in-context textual pivoting improves
identity information in the restored images, model-based pivoting is still
required to retarget the encoder.

With the personalization of the generative prior being in-context in our
proposed pipeline, we find that the model-based pivoting is feasible with half
the number of finetuning steps, when compared with an out-of-context textual
pivoting pipeline (Fig. 10). On an NVIDIA Tesla V100 GPU, this leads to a
model-based pivoting time of about 10 minutes, as compared to 20 minutes in
the case of out-of-context textual pivoting.

In practice, we have observed that the personalized prior in $\mathbf{G}_{p}$
is sufficiently strong, such that even when performing the model-based
pivoting of $\mathbf{E}$ on a single individual, $\mathbf{E}$ is retargeted to
capture general cues and can be shared across various identities, as
demonstrated in Fig. 6. This is critical from a training time perspective: the
model-based pivoting, once performed on one identity, can be shared across
models for different identities. Therefore, the only personalization step
needed for each identity is the textual pivoting.

(a)

(b)

(c)

(d)

Figure 5: Unconditional Sampling. We found that during inference, identity-
preserved restoration is possible even with unconditional denoising for
initial steps, followed by conditional denoising for the remaining steps. For
a degraded input (a), we use unconditional sampling for the first 50% of
steps, followed by conditional sampling for the remaining steps (c). We find
this to still be consistent with the reference image (d), both in terms of
identity as well as faithfulness to the original image, as well as with the
restored image using conditional restoration for all the steps (b).

(a) Input

(b) $\mathbf{E}$, same id.

(c) $\mathbf{E}$, diff. id.

(d) Id. Reference

Figure 6: Finetuning $\mathbf{E}$ on different identities. We show that for
the model-based pivot tuning $\mathbf{E}$ can be finetuned on the same
identity (b), as well as on different identities as in the inset (c), while
providing similar plausible restorations with respect to the identity in the
reference image (d).

##  4 Experiments

(a) Degraded Image

(b) ASFFNet [22]

(c) DMDNet [24]

(d) DiffBIR [25]

(e) Ours

(f) Ground Truth

Figure 7: Results on synthetically degraded images. A considerable shift in
identity can be observed between our proposed method and alternative
baselines. Identity preservation can be observed in terms of overall geometric
features, as well as attributes: eye shape (row 1), unnatural mole on the chin
in the DiffBIR result (row 2), teeth structure and shape (row 3), eye color
and feature between eyes (row 4), nose and nostrill shape (row 5), eye color
(row 6).

(a) Degraded Image

(b) ASFFNet [22]

(c) DMDNet [24]

(d) DiffBIR [25]

(e) Ours

(f) Id. Reference

Figure 8: Results on real degraded images. It can be seen that even in images
in the wild with real, unknown degradation kernels, our proposed method is
superior to the baselines in terms of identity retention while maintaining
high fidelity to the degraded input.

In this section, we show through qualitative and quantitative analysis that
the proposed method is superior to prior reference-based and blind image
restoration methods.

###  4.1 Training Process and Datasets

We use images from CelebRef-HQ dataset [24] to personalize the model; more
specifically, this dataset has $>10$ $512\times 512$ images of the same person
with $500$ identities, and each time we use one identity’s images. Our method
is trained on synthetic data that covers a wide range of degradation similar
to the real world.

We follow the same second-order degradation strategy as [25]. At each stage we
first convolve the image with a blur kernel $k_{\sigma}$, and downsample with
a scale factor $r$. Following that, additive noise $n_{\delta}$ is added, and
finally the image is JPEG-compressed with quality $q$. Formally, a single
stage can be described as

$\mathbf{x}_{d}=[(\mathbf{x}\circledast
k_{\sigma})\downarrow_{r}+{n}_{\delta}]_{\mathtt{JPEG}_{q}},$ (10)

where $\circledast$ is the convolution operator. The final image is obtained
after applying Eq. 10 twice. We refer you to [25] for more details on the
degradation process.

Test data We did test on three datasets: (1) CelebRef-HQ test set, (2) Google
searched images including low-quality and high-quality images from the same
person, (3) our collected real data from acquaintance.

###  4.2 Comparisons with other strategies

We show the comparisons of our personalization strategy to the alternative
strategies in Fig. 2. As can be seen, the proposed strategy maximizes identity
preservation through the restoration process, while comparison strategies,
namely personalizing when pivoting only on the text condition (c) or
personalizing when pivoting only on the generative model both are unable to
effectively incorporate identity while retaining image fidelity.

Table 1: Fidelity and idenitity metrics. The proposed method generates images
with high fidelity, while showing superior identity retention, when
personalized on 10 images. We compare along PSNR and SSIM as fidelity metrics,
while the ArcFace similarity serves as an identity metric.

Method | PSNR (dB) | SSIM | ArcFace (Identity)
---|---|---|---
GFPGAN [46] | 23.67 | 0.58 | 0.75
CodeFormer [56] | 23.98 | 0.57 | 0.76
MyStyle [29] | 17.24 | 0.53 | 0.68
DR2 [49] | 21.56 | 0.54 | 0.40
ASFFNet [22] | 11.21 | 0.40 | 0.55
DMDNet [24] | 11.33 | 0.40 | 0.61
DiffBIR [25] | 24.27 | 0.58 | 0.76
Ours | 23.72 | 0.55 | 0.88

###  4.3 Comparisons with SOTA methods

We perform qualitative and quantitative comparisons between our method and
several state-of-the-art reference-based and blind face image restoration
methods. For reference-based methods, we chose ASFFNet [23], DMDNet [24], and
MyStyle [29]; for a fair comparison, all methods use 10 reference images for
guidance. For blind face image restoration, we chose DiffBIR [25], DR2 [49],
GFP-GAN [46], and CodeFormer [56]. For ease of reading, we put qualitative
results from MyStyle, DR2, GFP-GAN, and CodeFormer in the appendix.

(a) Degraded Image

(b) ASFFNet [22]

(c) DMDNet [24]

(d) DiffBIR [25]

(e) Ours

(f) Ground Truth

Figure 9: Our approach is agnostic to different types of degradation, such as
blur (top row), compression (middle row). Prior face image restoration methods
(b, c) have their estimates considerably affected by the nature of the
degradation (note artifacts near eyes and mouth). Prior unconditional
diffusion methods (d) have more consistent performance, but with lost identity
information (note eye shape, nose shape). Our proposed method provides
consistent restoration across a range of degradations, while retaining
identity.

Comparison on synthetic degradations. Fig. 7 and Fig. E show qualitative
comparisons of the methods on the CelebRef-HQ test set subject to the
synthetic degradations described above. It is noteworthy that ASFFNet and
DMDNet are trained on perfectly aligned face images. During test time, highly
accurate face alignment is required to eliminate domain gap and get high-
quality restoration results, which is hard to achieve on severely degraded
images. As a result, the results of both methods are underwhelming. On the
other hand, diffusion-based methods (DiffBIR and ours) are possess a
generative prior trained on huge amount of unaligned faces and can generalize
well on test images without specific alignment, which leads to significantly
higher restoration quality. MyStyle, which relies on a personalized generative
prior, shows very good identity retention, however the restored images lack
fidelity to the input degraded image. Blind restoration techniques, such as
CodeFormer, GFP-GAN and DiffBIR lead to significant identity drifts, while
retaining fidelity to the input image. Compared to DiffBIR, our method
achieves better fidelity to the ground truth face details, especially in terms
of eye shape, eye color, nose shapes and eyebrows, demonstrating that the
proposed method can indeed better preserve identities.

In addition to qualitative comparisons, we also report the quantitative
results on the entire CelebRef-HQ test set in Tab. 1. We report PSNR and SSIM
as full-reference metrics. To quantify the identity disparity from the ground
truth image, we use ArcFace [10] similarity. As observed in the qualitative
comparisons, ASFFNet and DMDNet cannot restore the images effectively,
resulting in low scores on all the metrics. On the other hand, MyStyle results
in poor quantitative performance, even through the qualitative results show
restored images with high fidelity. This can be attributed to the lack of
faithfulness of the restored image to the input image. The blind restoration
methods (GFP-GAN, CodeFormer, DR2 and DiffBIR) show varying degrees of
performance, however DiffBIR is found to perform the best among these. Our
method achieves higher identity preservation, at the cost of slightly lower
PSNR and SSIM to DiffBIR. One explanation is that while DiffBIR tends to
generate smooth restorations to which PSNR and SSIM are not sensitive, our
method injects realistic high-frequency details that may not be perfectly
aligned with the ground truth image, thus decreasing the scores. In summary,
across all compared methods, the proposed method is still able to provide high
image fidelity, while retaining identity. Some additional notes on the
baseline qualitative metrics may be found in Sec. B.

We also conduct a user study to identify the perceptual benefit of our
contextual personalized prior (Appendix Sec. B). We make two important
observations: first, comparing unpersonalized diffusion-based face restoration
[25] with our personalized method, a large majority of partipants in our study
note that personalization in fact improves identity-independent image quality
as well. Second, we analyze the benefit of our personalized prior towards
perceived identity retention while being faithful to the input image.
Specifically, we note that the proposed method is predominatly found to be
superior, when compared with unconditional diffusion-based image restoration
[25], reference-based image restoration [56], and GAN-based personalized
priors [29].

Comparison on real degradations. Fig. 8 and Fig. F shows the performance of
each method on real-world images. For each test image, the model is
personalized with 10 reference images of the same identity from CelebRef-HQ.
We also provide a reference image in column (f) for better evaluation of the
identity preservation. This set of experiments show that our personalized
model generalizes well to real-world degraded images. In all cases, our method
achieves significantly better image quality than ASFFNet DMDNet, GFP-GAN and
CodeFormer, and preserves identity better than DiFFBIR. When compared with
MyStyle, in this operating regime the proposed method is considerably more
faithful to the input degraded image, while being able to retain identity.

Consistent restoration through personalized prior. To show the strength of our
personalized prior, we conduct an experiment by applying different synthetic
degradations on the same image and observe how each method restores them, as
shown in Fig. 9. The upper row shows a blurring degradation, while the lower
row shows a compression degradation. For ASFFNet and DMDNet, the quality of
the restored images varies with the degradations being applied. DiffBIR
generates higher quality images, but the identity remains sensitive to the
input degraded image. For example, in Fig. 9(d), note the inconsistent eye and
nose shape, when compared with Fig. 9(f). Our personalized model effectively
restores the images with consistent perceptual quality and identity fidelity.
This indicates the stability of the contextual prior and the reliability of
the encoder conditioning, across degradation types.

Deg. Image

(a) In-Context Textual Pivoting

(b) \+ Model-based Pivoting

Id. Reference

(c) Out-of-Context Textual Pivoting

(d) \+ Model-based Pivoting

Figure 10: Ablation: understanding dual-pivot tuning. (a) In-context textual
pivoting enables identity injection, however there still remains a gap in both
identity and quality. This is bridged through the model-based pivoting (b).
Note the difference in texture information within the insets, for features
such as hair (yellow) and beard (blue, brown). Please zoom in for a clearer
distinction. This can be compared with the alternative proposed approach. By
beginning with out-of-context textual pivoting (c), significant drifts in
identity are seen (note the eye shape) as the restoration pipeline is unable
to leverage the personalized prior. However in this setting, the effect of the
model-based pivoting is much more pronounced (d): alignment of $\mathbf{E}$
with the personalized prior leads to significant improvement in identity
preservation, and comparable performance with the upper row.

(a) Degraded inp.

(b) CFG=2.0

(c) CFG=4.0

(d) CFG=6.0

Figure 11: Effect of classifier free guidance (CFG). Sweeping across CFG
enables a fidelity-diversity tradeoff, also manifesting as a tradeoff between
image sharpness with realism (higher the CFG, higher the sharpness). This
provides a degree of user control.

#### Ablation: Understanding Dual-Pivot Tuning.

We find the proposed dual-pivot tuning approach to be a general technique for
personalization of guided diffusion models. Fig. 10 qualitatively explores the
benefits of this method. Specifically in our case, we find that alternate
strategies for personalization exist. The first is our chosen strategy: in-
context textual pivoting (Fig. 10(a)), which injects identity information,
followed by model-based pivoting (Fig. 10(b)), which enables better
utilization of the general restoration prior to get high-fidelity restored
images. The alternate approach involves beginning with out-of-context textual
pivoting (where $\mathbf{G}$ is personalized outside the context of
$\mathbf{B}$). As shown in Fig. 10(c), this step leads to significant gap in
identity as the restoration pipeline is unable to utilize the newly injected
identity information. However, post the model-based pivoting (Fig. 10(d)),
this is resolved, leading to high fidelity, identity preserving image
restoration. The dual-pivot tuning approach successfully personalized the
diffusion model in both these settings. As discussed previously, we find the
in-context textual pivoting to enable faster personalization in terms of the
the model-based pivoting step.

Classifier-free guidance scale. Fig. 11 shows an ablation study on the effect
of classifier-free guidance scale. As can be seen, the parameter allows for
trading off image sharpness with realism, allowing for pereference-based
tuning. Specifically, we find that increasing the CFG scale makes the restored
images sharper, at the cost of realism (additional quantitative analysis in
Appendix Sec. D). We also note in Fig. 5 that the proposed method does not
require conditional inference across all denoising steps. That is, to enable
personalized restoration, we find that even though initial denoising steps are
carried out unconditionally, effective personalized restoration is possible.

Additional Applications: Fig. 12 shows the ability of our personalized
generative prior to enable applications such as face swapping (top row) and
text-guided editing (bottom row). Specifically, we blur the source image to
obscure identity. Then, for face swapping we follow by restoration using the
personalized prior for a new identity, then stitching the restored face back
to the unblurred, original image. In the text-guided editing application we
utilize the semantic prior of the diffusion model for text-conditional editing
of the restored image. For example, with the term “smiling” as part of the
prompt, we see a smile in the expression of the individual in the output
image. In addition, with the phrase “blue”, “green” or “yellow eyes”, we see a
clear variation in the eye color. Please zoom in for clearer visualization.

Face Swapping

Input

Induced blur

Face swap

Reference

Text-guided Editing

Deg. Inp.

No edit

“smile”

“blue eyes”

“green eyes”

“yellow eyes”

Figure 12: Editing applications: face swapping text-guided editing. The upper
row shows the application of face swapping, while the lower row shows
performance on text-guided editing.

##  5 Conclusion

We propose a technique for personalizing a diffusion prior for face image
restoration, leveraging the capabilities of few-shot fine-tuning based on a
set of example images of the person. Our method achieves high fidelity to both
the input image and the identity of the person. We conduct extensive
experiments to demonstrate the superiority of our method in comparison to
various state-of-the-art alternatives for both blind and few-shot personalized
face image restoration methods.

#### Limitations and Future Work.

Although we make the very first step in using contextually personalized
diffusion prior for face image restoration, it requires a computationally
consuming fine-tuning process for each identity, which limits its
deployability in large-scale systems. An important research direction is to
inject few-shot identity in a feed-forward way, which we leave for future
work. Furthermore, we would like to point out that, while the personalized
model has improved fidelity to the identity, its general fidelity and quality
are fundamentally limited by the underlying restoration method. Even though
improving general quality is not the focus of this work, we believe the idea
of contextual personalization holds promise for application in forthcoming
restoration methods, achieving improved performance in both image quality and
identity fidelity. Additionally, potential limitations native to
personalization [35] on a small set of images can find their way to our method
as well: namely overfitting towards features such as open eyes and smiling
mouth if most of the training images show this, and the input degraded image
is ambiguous in this aspect.

#### Acknowledgements.

We thank Jackson Wang and Fangzhou Mu for helpful discussions and feedback on
this manuscript.

## References

  * Alaluf et al. [2023] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or.  A neural space-time representation for text-to-image personalization.  _arXiv preprint arXiv:2305.15391_ , 2023.
  * Arar et al. [2023] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H Bermano.  Domain-agnostic tuning-encoder for fast personalization of text-to-image models.  _arXiv preprint arXiv:2307.06925_ , 2023.
  * Avrahami et al. [2023] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  _arXiv preprint arXiv:2305.16311_ , 2023.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18392–18402, 2023.
  * Bulat and Tzimiropoulos [2018] Adrian Bulat and Georgios Tzimiropoulos.  Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 109–117, 2018.
  * Chan et al. [2021] Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy.  Glean: Generative latent bank for large-factor image super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14245–14254, 2021.
  * Chen et al. [2021] Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K Wong.  Progressive semantic-aware style transformation for blind face restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11896–11905, 2021.
  * Chen et al. [2023] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  _arXiv preprint arXiv:2304.00186_ , 2023.
  * Chen et al. [2018] Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang.  Fsrnet: End-to-end learning face super-resolution with facial priors.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2492–2501, 2018.
  * Deng et al. [2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4690–4699, 2019.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dogan et al. [2019] Berk Dogan, Shuhang Gu, and Radu Timofte.  Exemplar guided face image super-resolution without facial landmarks.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2019.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Gal et al. [2023] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–13, 2023.
  * Gu et al. [2022] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng.  Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder.  _arXiv preprint arXiv:2205.06803_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017, 2023.
  * Kim et al. [2019] Deokyun Kim, Minseon Kim, Gihyun Kwon, and Dae-Shik Kim.  Progressive face super-resolution via attention to facial landmark.  _arXiv preprint arXiv:1908.08239_ , 2019.
  * Li et al. [2018] Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang.  Learning warped guidance for blind face restoration.  In _The European Conference on Computer Vision (ECCV)_ , 2018.
  * Li et al. [2020a] Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang.  Blind face restoration via deep multi-scale component dictionaries.  In _European Conference on Computer Vision_ , pages 399–415. Springer, 2020a.
  * Li et al. [2020b] Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo.  Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion.  In _CVPR_ , 2020b.
  * Li et al. [2020c] Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo.  Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2706–2715, 2020c.
  * Li et al. [2022] Xiaoming Li, Shiguang Zhang, Shangchen Zhou, Lei Zhang, and Wangmeng Zuo.  Learning dual memory dictionaries for blind face restoration.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Lin et al. [2023] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong.  Diffbir: Towards blind image restoration with generative diffusion prior.  _arXiv preprint arXiv:2308.15070_ , 2023.
  * Liu et al. [2023] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones: Concept neurons in diffusion models for customized generation.  _arXiv preprint arXiv:2303.05125_ , 2023.
  * Luo et al. [2021] Xuan Luo, Xuaner Zhang, Paul Yoo, Ricardo Martin-Brualla, Jason Lawrence, and Steven M Seitz.  Time-travel rephotography.  _ACM Transactions on Graphics (TOG)_ , 40(6):1–12, 2021.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Nitzan et al. [2022] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or.  Mystyle: A personalized generative prior.  _ACM Transactions on Graphics (TOG)_ , 41(6):1–10, 2022.
  * Po et al. [2023a] Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein.  Orthogonal adaptation for modular customization of diffusion models.  _arXiv preprint arXiv:2312.02432_ , 2023a.
  * Po et al. [2023b] Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit H Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al.  State of the art on diffusion models for visual computing.  _arXiv preprint arXiv:2310.07204_ , 2023b.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Roich et al. [2022] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  _ACM Transactions on graphics (TOG)_ , 42(1):1–13, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023a] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023a.
  * Ruiz et al. [2023b] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  _arXiv preprint arXiv:2307.06949_ , 2023b.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Shen et al. [2018] Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang.  Deep semantic face deblurring.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8260–8269, 2018.
  * Song et al. [2021] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2021.
  * Tang et al. [2023] Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, et al.  Realfill: Reference-driven generation for authentic image completion.  _arXiv preprint arXiv:2309.16668_ , 2023.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Voynov et al. [2023] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$: Extended textual conditioning in text-to-image generation.  _arXiv preprint arXiv:2303.09522_ , 2023.
  * Wang et al. [2023a] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy.  Exploiting diffusion prior for real-world image super-resolution.  _arXiv preprint arXiv:2305.07015_ , 2023a.
  * Wang et al. [2020a] Kaili Wang, Jose Oramas, and Tinne Tuytelaars.  Multiple exemplars-based hallucination for face super-resolution and editing.  In _Proceedings of the Asian Conference on Computer Vision_ , 2020a.
  * Wang et al. [2023b] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al.  Imagen editor and editbench: Advancing and evaluating text-guided image inpainting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18359–18369, 2023b.
  * Wang et al. [2021] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan.  Towards real-world blind face restoration with generative facial prior.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Wang et al. [2020b] Zhihao Wang, Jian Chen, and Steven CH Hoi.  Deep learning for image super-resolution: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(10):3365–3387, 2020b.
  * Wang et al. [2022] Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo.  Restoreformer: High-quality blind face restoration from undegraded key-value pairs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17512–17521, 2022.
  * Wang et al. [2023c] Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang.  Dr2: Diffusion-based robust degradation remover for blind face restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1704–1713, 2023c.
  * Yang et al. [2020] Lingbo Yang, Shanshe Wang, Siwei Ma, Wen Gao, Chang Liu, Pan Wang, and Peiran Ren.  Hifacegan: Face renovation via collaborative suppression and replenishment.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1551–1560, 2020.
  * Yang et al. [2021] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang.  Gan prior embedded network for blind face restoration in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 672–681, 2021.
  * Yu et al. [2018] Xin Yu, Basura Fernando, Bernard Ghanem, Fatih Porikli, and Richard Hartley.  Face super-resolution guided by facial component heatmaps.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 217–233, 2018.
  * Yue and Loy [2022] Zongsheng Yue and Chen Change Loy.  Difface: Blind face restoration with diffused error contraction.  _arXiv preprint arXiv:2212.06512_ , 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.
  * Zhao et al. [2022] Yang Zhao, Yu-Chuan Su, Chun-Te Chu, Yandong Li, Marius Renn, Yukun Zhu, Changyou Chen, and Xuhui Jia.  Rethinking deep face restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7652–7661, 2022.
  * Zhou et al. [2022] Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy.  Towards robust blind face restoration with codebook lookup transformer.  _Advances in Neural Information Processing Systems_ , 35:30599–30611, 2022.

Appendix

##  A Appendix Contents

This appendix contains the following contents:

  1. 1.

Additional quantitative analysis (Sec. B)

  2. 2.

Additional qualitative analysis (Sec. C)

  3. 3.

Additional observations (Sec. D)

  4. 4.

Ethical considerations (Sec. E)

##  B Additional Quantitative Analysis

#### Additional notes on Tab. 1.

An interesting result to note is the performance of [49]. We note that DR2
allows for tunable parameters $(N,\tau)$ selection to tradeoff restoration
with fidelity to the original image. We choose $(N,\tau)=(16,35)$ as we find
this combination able to restore the degradations in our synthetic images.
However, this does come at a cost of fidelity to the input - this is reflected
by low fidelity as well as identity metrics for DR2.

Another interesting observation is the performance of MyStyle [29] in terms of
identity retention. We see that the ArcFace metric for MyStyle is similar to
other blind restoration techniques. This is a somewhat surprising observation,
since MyStyle uses a personalized prior for restoration. This may be explained
as follows: while MyStyle indeed shows strong indentity retention (as
evidenced by qualitative results in Figs. E, F, G and H), the method is unable
to retain fidelity (in terms of color, lighting, texture, makeup and even
pose). All these factors can affect the perceived identity of the image from
the perspective of the ArcFace metric, leading the this anomaly. On the other
hand, the ArcFace metric is a valid comparison of identity retention with all
other comparison methods, since they are able to retain fidelity with the
input degraded image.

(a) Example survey question

(b) Survey Result

Figure A: User study: effect of personalization on perceived image quality.
When asked to choose the image with better perceived quality, we find users
predominantly choosing the images with our personalized restoration. Our
method is indicated in blue, while DiffBIR [25] is shown in red, in the pie
chart.

(a) Survey Result (quality)

(b) DiffBIR [25]

(c) Ours

Figure B: Two specific real degradation restorations, from the lens of image
quality. The upper row shows a case where respondent opinion is split -
DiffBIR provides specific detail like wrinkles, while our approach provides
structure and identity. The lower row shows a case with unanimous favor
towards our method. This arises from specific artifacts in the DiffBIR output,
which is avoided by having a strong personalized prior. Our method is
indicated in blue, while DiffBIR [25] is shown in red, in the pie charts.

(a) Example survey question

(b) Survey Result

Figure C: User study: the effect of personalization on perceived image
identity retention. When asked to choose the image with the better-looking
image, while prioritizing identity and faithfulness to input degraded image,
we find users predominantly choosing the images with our personalized
restoration. Our method is indicated in blue, DiffBIR [25] is shown in yellow,
MyStyle [29] is shown in red, and DMDNet [24] is shown in green in the pie
chart.

(a) Degraded Image

(b) DifFace [53]

(c) DR2 [49]

(d) DiffBIR [25]

(e) Ours

(f) Ground Truth

Figure D: Comparison with existing diffusion-based blind image restoration
methods. We find that for the degree of degradation we deal with in our
experiments, both DifFace [53] as well as DR2 [49] show unreliable
performance, losing both identity and fidelity at different instances. DiffBIR
[25], on the other hand, while not performing as well as our method in terms
of identity and fidelity retention, is the best performer among the prior
diffusion-based methods. We therefore choose DiffBIR as our base model and
comparison benchmark for subsequent experiments.

#### User study.

We conduct a user study as a measure of perceptual quality and comparison with
prior methods. We focus on two questions as part of the study: (a) can
personalization help improve perceived image quality as well?, and (b) how
does our method compare with prior methods in terms of identity-aware
restoration. We perform our study across 30 participants. The survey consists
of randomized, anonymized options that they can choose amongst. Figs. A and C
(a) show the exact guidelines provided as part of the study. For part (a),
image quality, we compare against the base unconditional restoration method
(DiffBIR [25]), with the objective being to choose the better-quality image,
irrespective of identity. For this case, we compare across 11 image pairs,
containing both real and synthetic degradations. For part (b), identity-aware
restoration, we compare with three methods: (i) DMDNet [24] (better-performing
method both qualitatively and quantitatively when compared with ASFFNet [22]);
(ii) MyStyle [29], a personalized generative prior; (iii) DiffBIR [25], a
unconditional diffusion-based method. For this case, we compare across 10
image sets, consists of synthetically degraded images.

We begin with discussing the first part of the user study: effect of
personalization on identity-independent perceived image quality. Fig. A shows
the results of this part of the study. We observe that across the 30 study
participants, a clear majority of the participants indicate the quality
improvement that arises out of personalization. This is an unexpected result,
and we explore this further by analyzing two specific instances, in Fig. B.
The upper row shows a case where the users are broadly split between the
candidate options. The unconditional method provides a restored image with
detailed facial features such as wrinkles, while the proposed method provides
less of such detail, with stronger identity cues. Both images look viable as
natural images with good quality. However, the lower row in Fig. B shows an
instance where the study participants almost unanimously prefer our method.
The reason for this is evident: the baselines method has considerable
artifacts, especially near the eyes, while our method leads to a good quality,
realistic face image. Through these observations, we can understand the effect
of our personalization method on identity-independent image quality. In cases
where the unconditional comparison method is able to perform, our personalized
model remains stable and provides realistic looking faces. However, in cases
where the comparison method fails, such as with high degradations, our method,
throgh the strength of the identity prior, still results in realistic restored
images. A combination of these two factors leads to superior perceived
identity-independent image quality.

We next analyze the second part of the user study: the perceived strength of
our identity-aware image restoration. Fig. C shows the results of this part of
the study. We see that, perhaps per expectation, participants rate our method
as the predominant favorite in terms of identity-aware image restoration,
while retaining faithfulness to the input degraded image. This can be seen
across our various qualitative results and speaks to the strength and
reliability of our personalized prior across identities and degradations.
However, an interesting insight is the relative placement of the comparison
methods. Specifically, study participants rate DiffBIR [25] to be the second-
best method, despite not retaining identity, as a result of its strong
correlation to the input degraded image. On the other hand, MyStyle [29],
while having a strong identity prior, is the third-best preferred method on
average, as a result of it not being faithful to the input image. That is,
perceptually, faithfulness to the input degraded image is given a higher
priority by participants, despite the study guidelines placing both identity
and faithfulness to input image at the same priority.

##  C Additional Qualitative Analysis

We now discuss additional qualitative results, comparing with a range of
baselines. We first qualitatively compare our proposed method with existing
diffusion-based blind restoration techniques. Specifically, we compare with
DifFace [53], DR2 [49] and DiffBIR [25]. Fig. D highlights these observations.
We note that in our operating regime for degradation, both DR2 and DifFace
result in inaccurate and unreliable restoration (in both these cases, the
models trade off fidelity with restoration and need to be used accordingly
depending on the degree of degradation). In constrast, DiffBIR, while worse in
both detail and identity when compared with our method, proves to be a much
more reliable blind restoration technique. We therefore use DiffBIR as our
base model for our method, and use it as the representative diffusion-based
blind restoration technique in our analysis.

We next describe qualitative performance in comparision with GFPGAN [46] and
CodeFormer [56] as blind image restoration methods, and MyStyle [29] as a
method with a personalized generative prior, in additional detail to the
paper. Fig. E shows results on synthetic degradations and corresponding
restorations across six different identities. As can be seen, the blind
restoration methods show clear drifts in identity across all six examples,
while retaining fidelity to the input image. On the other hand, Nitzan et al.
[29], while being able to retain a strong identity prior, sees a significant
deviation in the restored image, from the input degraded image (potentially as
a result of personalization on a small number of images (10)). In contrast,
our method is able to achieve the best of both worlds: while maintaining a
high degreee of faithfulness to the input image, we also see consistent
identity retention across all examples. This leads to our results being
closest to the reference image, despite the input having severe degradation in
several cases.

We next look at Fig. F, for an analysis on real degraded images when compared
with these additional baselines. Again, we note consistent observation. The
blind restoration methods remain faithful to the input image, however they
result in considerable identity drifts and artifacts in the restored images.
MyStyle is able to continue retaining a strong personalized prior, however at
the cost of losing all context relating to the input degraded image. Again,
our method is able to retain identity, while being faithful to the input
image.

Further, we supply additional synthetic (Fig. G) as well as real (Fig. H)
degradation results across all baselines. We wish to highlight the robustness
of our proposed method, across identites and degradations, over a larger
number of image settings. These aspects can specifically be seen through
identifying features in the participants, such as hair, teeth, ears, eye color
and so on. The overall trends remain the same: prior methods are either able
to retain strong identity without artifacts, or retaining faithfulness to the
input image. It is through our personalzation regime that we achieve the best
of both these worlds, getting identity-consistent restored images with high
fidelity.

(a) Degraded Image

(b) GFPGAN [46]

(c) CodeFormer [56]

(d) MyStyle [29]

(e) Ours

(f) Ground Truth

Figure E: Additional baseline methods on synthetically degraded images (in
addition to the results in the main paper). Similar observations to the
synthetically degraded results can be made. The blind image restoration
methods suffer both on identity and quality, while MyStyle is able to retain
identity at the cost of losing faithfulness to the input image. Our method is
able to retain identity in the restoration, while being faithful to the input
image. In terms of specifics for each row, note the eye and eyebrow shape in
row 1, eye shape, bags under the eyes, and teeth in row 2, eye shape and ears
(specifically, left ear) in row 3, mole on the left cheek and mark between the
eyes in row 4, eyes and mouth expression in row 5, and eye color in row 6.
Please zoom in to observe these details more easily.

(a) Degraded Image

(b) GFPGAN [46]

(c) CodeFormer [56]

(d) MyStyle [29]

(e) Ours

(f) Id. Reference

Figure F: Additional baseline methods on real degraded images (in addition to
the results in the main paper). It can be seen that even in images in the wild
with real, unknown degradation kernels, our proposed method is superior to the
baselines in terms of identity retention while maintaining high fidelity to
the degraded input. Comparison methods either result in artifacts or in an
image that is not faithful to the input, while the proposed method is able to
stably inject relevant identity information while retaining fidelity to the
input degraded image. Please zoom in to the Figure for a clearer view.

Deg. Image

GFPGAN [46]

CdFrmer [56]

MyStyle [29]

ASFFNet [22]

DMDNet [24]

DiffBIR [25]

Ours

Id. Reference

(a)

Deg. Image

GFPGAN [46]

CdFrmer [56]

MyStyle [29]

ASFFNet [22]

DMDNet [24]

DiffBIR [25]

Ours

Id. Reference

(b)

Deg. Image

GFPGAN [46]

CdFrmer [56]

MyStyle [29]

ASFFNet [22]

DMDNet [24]

DiffBIR [25]

Ours

Id. Reference

(c)

Figure G: Additional results on synthetic degraded images, comparing with all
discussed major comparison methods. For identity (a), zooming in shows that
all baseline methods either lead to considerable artifacts in the restored
image, or through identity drifts (eye color) and lack of fidelity with the
input image. For identity (b) in cases where the restored image does not have
significant artifacts, identity drifts can be noted in the form of nose shape
(as highlighted by the shadow on the nose), eye color and ear shape. For
identity (c), the mark between the eyes and the mole on the left cheek are
identifying features that show the superiority of the proposed method.

Degraded Image

GFPGAN [46]

CodeFormer [56]

Degraded Image

GFPGAN [46]

CodeFormer [56]

MyStyle [29]

ASFFNet [22]

DMDNet [24]

MyStyle [29]

ASFFNet [22]

DMDNet [24]

DiffBIR [25]

Ours

Id. Reference

DiffBIR [25]

Ours

Id. Reference

Figure H: Additional results on real degraded images, comparing with all
discussed major comparison methods. For both identities, zooming in shows that
comparison methods (both prior reference-based methods and blind restoration
methods) result in strong artifacts, identity drifts, and in the case of
MyStyle [29], lack of fidelity with the input degraded image. The proposed
method is the only one consistently able to incorporate identity in the
restoration in a stable manner. Please zoom in to the image for a clearer
visualization.

##  D Additional Observations

#### The Effect of Classifier Free Guidance.

The qualitative effect of classifier free guidance (CFG) was discussed in the
main paper (Figure 8). Here, we perform a quantitative analysis, through Tab.
A. We note that traditional fidelity metrics worsen as the CFG is increased.
This is consistent with our other observations, where the unconditional method
(DiffBIR) shows slightly better performance on these metrics. As we increase
the CFG, we move farther from the unconditional model and therefore see these
effects. In terms of identity, we see a small reduction with increased CFG.
This is also consistent with our observations from the main paper. As we
increase the CFG, the sharpness of the restored image increases and may lead
it to look unrealistic. Overall, in terms of tradional metrics, a lower CFG
values is optimal. However, visually, the CFG can serve as a useful control
knob for restored image style and quality, as discussed in Figure 8, main
paper.

#### Dealing with Heavy Degradations.

Fig. I shows a potentially interesting use setting for the proposed method.
Namely, in the case of very heavy degradations, multiple passes through the
restoration model may be performed. As seen in the figure, (b) is able to
obtain coarse details as well as shape and strcuture information. Through a
second pass, (c) is able to improve of texutre and detail, in addtion to
obtaining greater identity injection, leading to a better restored image than
(b).

(a)

(b)

(c)

(d)

Figure I: Heavy degradation restoration. Given a highly degraded image (a), a
first pass (b) through our restoration pipeline provides a coarse estimate
that pulls the image closer to the input domain of the model. The second pass
(c) injects texture and identity information, leading to a better restored
estimate with respect to the ground truth (d).

##  E Ethical Considerations

The use of generative models, while being extremely helpful for challenging
tasks such as identity-aware image restoration, can also potentially have
harmful effects. Specifically, generative models can be used for immoral
tasks. In our case, applications that we discussed, such as face swapping as
well as text-guided editing, can lead to generation of fake images, that may
be used without the consent of the person whose image it is. We strongly
condemn such and any other use cases of the proposed method.

Table A: Ablation: effect of classifier-free guidance scale on personalized
image restoration. We compare along PSNR and SSIM as fidelity metrics, while
the ArcFace similarity serves as an identity metric.

CFG | PSNR (dB) | SSIM | ArcFace (Identity)
---|---|---|---
1.0 | 24.78 | 0.70 | 0.90
2.0 | 24.54 | 0.69 | 0.89
3.0 | 24.17 | 0.68 | 0.87
4.0 | 23.73 | 0.67 | 0.85
5.0 | 23.12 | 0.65 | 0.84