# PFStorer: Personalized Face Restoration and Super-Resolution

Tuomas Varanka∗1 Tapani Toivonen2 Soumya Tripathy2 Guoying Zhao1 Erman Acar2
1 University of Oulu 2 Huawei Finland
<EMAIL_ADDRESS>
###### Abstract

Recent developments in face restoration have achieved remarkable results in
producing high-quality and lifelike outputs. The stunning results however
often fail to be faithful with respect to the identity of the person as the
models lack necessary context. In this paper, we explore the potential of
personalized face restoration with diffusion models. In our approach a
restoration model is personalized using a few images of the identity, leading
to tailored restoration with respect to the identity while retaining fine-
grained details. By using independent trainable blocks for personalization,
the rich prior of a base restoration model can be exploited to its fullest. To
avoid the model relying on parts of identity left in the conditioning low-
quality images, a generative regularizer is employed. With a learnable
parameter, the model learns to balance between the details generated based on
the input image and the degree of personalization. Moreover, we improve the
training pipeline of face restoration models to enable an alignment-free
approach. We showcase the robust capabilities of our approach in several real-
world scenarios with multiple identities, demonstrating our method’s ability
to generate fine-grained details with faithful restoration. In the user study
we evaluate the perceptual quality and faithfulness of the genereated details,
with our method being voted best 61% of the time compared to the second best
with 25% of the votes.

Figure 1: Imagine wanting to restore a photo of yourself, only for the
resulting image to not be you, but someone else! By utilizing a few high-
quality reference images, we can faithfully restore images with fine-grained
details. Best viewed by zooming in.

††footnotetext: ∗This research was performed during an internship at Huawei
Finland.

##  1 Introduction

Face restoration aims to recover HQ (high-quality) face images from degraded
observations, such as blur, low-resolution, noise and compression artifacts.
In real-world scenarios, the task is even more challenging, due to more
complex degradations and variations in illumination and pose.

Figure 2: Results under increasing levels of degradation. a) With only minor
degradation, both base and personalized model are capable of restoration. b)
The base model incorrectly restores fine-grained details such as the nose and
skin texture. c) More identity details such as eyes and facial hair are lost.
d) Base model outputs a completely different identity, while the personalized
model retains details of the identity, even if the semantics are not entirely
correct due to the extreme low-quality input image. Best viewed by zooming in.

Restoration of faces is a highly ill-posed problem with multiple solutions to
a given LQ (low-quality) input. Compared to natural images humans are very
sensitive to subtle differences with facial images. Even small variations in
the shape, size or color of eyes, nose, lips, etc., can cause a shift in the
identity of the person, see top and middle row of Fig. 2. Furthermore, if we
are familiar with identity of a specific person, we are even more prone to
spotting subtle differences. We show an example in Fig. 1. A restoration model
needs to not only output a realistic image, but also one that is faithful to
the identity of the person.

Recent research on face restoration has seen great progress towards higher
visual quality results. Many of the techniques exploit a generative prior such
as GAN [40, 3, 8], codebooks [24, 48, 13] or diffusion models [43, 41].
Generative prior methods have been trained under a generative task prior to
being modified to restoration models and as a result they are capable of
outputting realistic face images. However, often the outputs can be
inauthentic as the models lack crucial context about the identity. To combat
this, a reference prior has been used [25, 26], which uses a HQ reference
image of the same identity, leading in theory to high fidelity. However, in
practice transferring the identity from a reference image is difficult due to
differences in pose, illumination and semantics between the LQ and reference
image.

To alleviate the ill-posedness of the restoration problem we fully exploit the
reference images by creating a neural representation of the identity. We
propose PFStorer (Personalized Face Restorer) to restore LQ face images while
retaining the identity by personalization. Given a few HQ reference images
(_e.g_. 3-5 selfies from a photo gallery) a restoration model is fine-tuned to
a personalized restoration model. The reference images can have significantly
different illumination, pose, expression and do not have to be aligned with
the LQ image. Our goal is to personalize a restoration model such that it is
able to restore person specific images while producing realistic images and
being faithful to the identity.

As opposed to personalized generative models that personalize a generative
model, we use a base face restoration model as our foundation. The base model
is capable of realistic outputs, however the fidelity may suffer due to the
ill-posedness of the task. Our strategy is to perform personalized restoration
by fine-tuning a base restoration model with a few HQ reference images.
However, a naive fine-tuning strategy can destroy the strong existing priors
present in the base restoration model. To avoid catastrophic forgetting when
performing fine-tuning for personalization, an adapter is used to keep the
priors intact. Adapters are trainable blocks that can be used to adapt the
flow of a model. By freezing the base model and only training the adapter
blocks, existing priors can be preserved. To avoid the rapid change of
intermediate outputs caused by adapters a learnable parameter is used. The
learnable parameter also controls the amount of injection of personalization
for different layers of the network, leading to more fine-grained control.

During training, we observe an issue where the model learns to rely too much
on the LQ image ignoring the reference images. This is due to the majority of
the training samples having low degredation, which can preserve identity
information sufficient to restore the face without using the reference images.
To alleviate the issue we design a generative regularizer, in which no
conditional LQ image is given and the model is forced to generate the identity
using only reference images. This approach encourages the model to learn a
robust neural representation of the identity, as in generative
personalization.

Furthermore, for the base face restoration model we fine-tune a general
purpose restoration model on a face dataset to improve its restorative
capabilities. During training, instead of resizing all images to a specific
size and aligning them, random crops of the face images are used. This has
several benefits: 1) The model has access to higher resolution patches. 2) The
model is more robust to varying poses. 3) The model is capable of super-
resolution through the use of tiling.

We experiment with our technique using both synthetic and real-world data. The
user study confirms that our method is able to improve results over previous
methods.

##  2 Related work

Figure 3: (Left) PFStorer restores an image with a diffusion process
conditioned on the LQ and the reference image. Base Model blocks are
visualized in green and Personlization blocks in purple. StableDiffusion [32]
is used to extract features $F^{i}_{Ref}$ from the reference image. During
training the reference image is randomly sampled from a set of reference
images for each training iteration. During inference, no reference images are
required as the identity is learned in the personalization blocks as a neural
representation. (Right) $i$th UNet block containing the Base Model Block [38]
and Personalization Block [14]. The Base Model Blocks contain the normal
Stable Diffusion blocks with SFT (spatial feature transformation) [39] blocks
from StableSR [38]. After the Base Model block, the intermediate features
$F^{i}$ go to a trainable Personalization Block, which contains cross-
attention between the text-embedding and reference image features
$F^{i}_{Ref}$. A learnable adapter vector $\gamma^{i}$ balances the
contribution between the base model and personalization.

#### Face Restoration

Most recent face restoration approaches include a generative prior, where the
model has been trained in a generative manner prior to training it for
restoration. GFP-GAN [40] uses the generator of StyleGAN2 [19] that has been
trained on facial images to generate high-quality features. More recently,
methods such as VQFR [13] and CodeFormer [48] have been using discrete
codebooks with vector quantization and adversarial training [9] to “store”
high-quality data. Some latest approaches use diffusion models [16] for
restoration. DifFace [43] transforms the LQ input to the manifold of high-
quality images with an arbitrary restoration model, which is followed by a
forward and backward diffusion, bringing in fine-grained details. DR2 [41]
uses a similar approach of first performing forward diffusion, after which
backward diffusion is performed with guidance from the matching steps of
forward diffusion. Zhao _et al_. [46] note that around 15% of the commonly
used training data from FFHQ [18] is not necessarily high-quality. To improve
the data quality they propose a two-step training process where after the
first step, the training data is enhanced, using the model trained in the
first step. Concurrent to our work is DiffBFR [31], which separates identity
and texture restoration using cascaded diffusion models. Another concurrent
work [5], also emphasizes the ill-posedness of the problem, but goes in the
opposite direction to us, encouraging diversity, instead of personalization.

#### Personalization

With the seminal work of DreamBooth [33] personalization of generative models
with just a few images was made possible. DreamBooth can generate novel scenes
of a specific object or a concept. It achieves this by fine-tuning a text-to-
image diffusion model while overwriting a rare text embedding. Several works
have since followed [21, 14, 22, 2, 28, 34, 42, 37, 12, 11] that attempt to
improve personalization. Custom-Diffusion [21] only fine-tunes the attention
layers present in the UNet architecture of StableDiffusion [32], significantly
reducing the training time and model size. Perfusion [11] goes further and
only does rank-1 updates of the attention layers, while locking the keys of
cross-attention layers, significantly reducing compute and memory. ViCo [14]
adds image-attention adapters to the cross-attention layers to learn cross-
attention between the reference images and predicted image. It also learns a
text embedding and applies a regularization using the class-token to avoid
overfitting, leading to high-quality results.

On a high-level similar to our approach is RealFill [36], which personalizes a
pre-trained in-painting model to perform authentic in- and out-painting. Both
MyStyle [29] and IdentityEncoder [35] first personalize the model and then
transform it to perform tasks such as face in-painting, super-resolution and
semantic editing. Compared to their approach, we transform a restoration model
to a personalized restoration model as opposed to transforming a personalized
model to a personalized super-resolution model.

#### Reference-Based Face Restoration

Reference-based approaches use reference images from the same identity in the
restoration process. GFRNet [23] uses a single reference image and learns a
warping between the LQ and reference image. ASFFNet [25] selects the most
similar reference image to reduce misalignment and uses adaptive feature
fusion for the restoration. DMDNet [26] constructs a dictionary of deep
features from important cropped regions (_e.g_., eyes, nose, mouth). An
alignment module is then used to align the features of the input and reference
images, resulting in a fusion of the features to the output image. These
methods however struggle when the reference image and LQ input are not aligned
or not similar enough. Compared to these approaches we learn a neural
representation of the identity, enabling more robust restoration.

##  3 Method

We design a face restoration method capable of generating realistic imagery,
while still being faithful to the identity of the person in a given image. We
begin by analyzing the situation formally (Sec. 3.1) and conclude that a
personal prior is required for faithful reconstruction in certain situations.
Next, we present a method (Sec. 3.2) that preserves existing priors by
utilizing adapters for personalized face restoration. To further enhance the
results a generative regularizer is proposed to enable robust fine-grained
restoration. We name this method PFStorer (ours). Beyond personalization (Sec.
3.3), we show simple modifications to the training pipeline of general face
restoration methods that enable super-resolution and an alignment-free
approach. We refer to this improved restoration model without personalization
as the Base Model, which is used as a base for personalization. Background for
diffusion models and personalization is given in the supplementary material.

###  3.1 The Need for a Personal Prior

Restoration of low quality images is naturally an ill-posed problem. Assume a
degradation function
$\mathcal{D}:\mathcal{I}\times\mathbb{R}\rightarrow\mathcal{I}$ that takes in
a face image $I\in\mathcal{I}$ and a value of degradation $d\in\mathbb{R}$. A
higher degradation value $d$ indicates a higher degraded output image. When
$d$ approaches infinity the resulting image will be close to pure noise and
restoring the image faithfully is no longer possible,
$id(\mathcal{R}(\mathcal{D}(I;d)))\neq id(I)$, where $id$ is a function that
returns the identity of a face image and $\mathcal{R}$ is a restoration model.
There exists a value $d_{f}<\infty$ after which faithful restoration is no
longer possible. However, with additional personal prior $p_{id}$ the
restoration can be made faithfully:

$id\left(\mathcal{R}(\mathcal{D}(I;d_{f});p_{id})\right)=id(I),$ (1)

as $p_{id}$ is unchanged with any value of degradation $d$. In this paper the
personal prior $p_{id}$ is learned from a set of reference images using a
diffusion model.

###  3.2 Personalized Face Restoration

The main idea is to use high-quality images of an individual in aid when
restoring LQ images. We start with a restoration model, which is fine-tuned
with a personalization technique using the reference images. The
personalization is performed for each individual once, after which it can be
used for inference as many times as wanted. In essence, the model is trained
to add personal details, when the base restoration model is insufficient, due
to the ill-posed nature of the problem. The architecture of the model can be
seen from Fig. 3.

During the personalization fine-tuning, the model takes as input a synthesized
LQ image $I_{LQ}$ and a reference image $I_{Ref}$ sampled from the set of
reference images $\\{I_{Ref}^{k}\\}$. A modified diffusion model loss

$\mathcal{L}_{Diff}=\mathbb{E}_{z,t,I_{LQ},I_{Ref},{\epsilon}}\|{\epsilon}-{\epsilon}_{\theta}(z_{t},c,I_{LQ},I_{Ref})\|_{2},$
(2)

with the addition of the LQ and reference image, is used. Here
${\epsilon}_{\theta}$ is the diffusion model, $z_{t}$ the latent code at time
$t$, $c$ the conditioning text embedding and ${\epsilon}$ the sampled noise
from an Isotropic Gaussian distribution.

Figure 4: 20x Super-resolution of a low-quality image. Super-resolution for
images larger than $512\times 512$ using a tiling approach from [38]. Image
edited from Vecteezy.com.

#### Personalization

We initially attempt to fine-tune with prior-preservation regularization [33],
but find that it fails to properly capture the fine-grained identity details
as well as diminishes the results from restoration due to modifying existing
priors. This motives the need for preserving the priors completely, leaving
the priors untouched. Therefore, we prefer to utilize adapter blocks, which do
not modify the existing priors at all, retaining their rich abilities to
restore and generate. In order to implement this, we employ text and image
cross-attentions between the learnable text-embedding [10], reference image
features $F^{i}_{Ref}$ and intermediate restored image features $F^{i}$ of the
layer $i$ , as used similarly in [14] and shown on Fig. 3 right. The reference
image features $F^{i}_{Ref}$ are obtained from a frozen StableDiffusion [32],
in practice they are fed through part of the Base Model in same batch as the
LQ image. We refer to this as the Personalization Block (see Fig. 3 right).

#### Controlled Adaptation

The simple addition of the personalization block however results in distorted
outputs. This is due to the sudden additional data being added to the
intermediate features of the Base Model from the personalization block. In
order to avoid the personalization block from changing the outputs too much, a
learnable vector $\gamma=\textbf{0}$ can be used to initialize the outputs
from the adapter, as in [4]. To further control the effect of personalization
we introduce seperate $\gamma$ for each personalization block applied at
different resolution of PFStorer. Mathematically, each layer’s output can be
expressed as:

$\hat{F}^{i}=F^{i}+\gamma^{i}\odot\text{Personalization-
Block}(F^{i},F_{ref}^{i}),$ (3)

where Personalization-Block is the adapter, consisting of cross-attentions as
shown on right of Fig. 3.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
Input | CodeFormer | DMDNet | DR2 + SPAR | Ours | Pseudo-GT
Figure 5:  Qualitative comparison with state-of-the-art restoration models on
real-world images. Images from Wikimedia Commons.

#### Generative Regularization

Compared to personalized generative models our personalized restoration model
has one additional signal, the low-quality image $I_{LQ}$. It guides the
general structure of the restoration output and it may contain some
information from the identity depending on the severity of the degradation.
During training, the additional input can make the task of outputting
personalized restored images easier, but it can also introduce shortcuts for
the model as the model can rely on information from the additional input. This
leaking of identity information from the input can lead to the model not fully
learning a representation of the identity during training, hence leading to
poor performance on difficult unseen cases, _e.g_. atmospheric turbulence.

To mitigate the above issue, we propose a generative regularizer that
encourages the model to learn a more robust identity representation. A
regularizing loss

$\mathcal{L}_{Gen}=\mathbb{E}_{z,t,I_{LQ},I_{Ref},{\epsilon}}\|{\epsilon}-{\epsilon}_{\theta}(z_{t},c,\varnothing,I_{Ref})\|_{2}.$
(4)

is added to the original training loss, where a null input $\varnothing$ is
given as the conditioning LQ image. This forces the model to fully hallucinate
the identity without any help from a conditioning image, encouraging a more
robust representation of the identity. The final loss is then

$\mathcal{L}=\mathcal{L}_{Diff}+\lambda_{Gen}\mathcal{L}_{Gen}+\lambda_{Pers}\mathcal{L}_{Pers}$
(5)

where $\lambda_{Gen}$ controls the weight of the generative term and
$\lambda_{Pers}\mathcal{L}_{Pers}$ regularizes the cross-attention maps for
the learnable text embedding token, which enforces personalization [14] (see
the supplementary for $\mathcal{L}_{Pers}$). The trainable parameters $\theta$
from ${\epsilon}_{\theta}$ consist of the personalization blocks and their
accompanying vectors $\gamma^{i}$.

###  3.3 Improving Face Restoration Diffusion Models

To integrate personalization into a restoration model, we first need a strong
base restoration model. We train our model with the facial dataset FFHQ using
the steps described below, which is initialized from the pre-trained StableSR
[38]. We refer to the trained model as Base Model, as it has not been
personalized to any specific person.

#### Existing Priors

Many recent face restoration methods have used generative priors [40, 48]. We
go further, and start our training on face images with a restoration model
pre-trained on generic natural images, namely StableSR [38]. As the model is
not trained from scratch on a new task, the training time is decreased and the
model is more robust.

#### Alignment Free Approach

Cropping and alignment is commonly used in face processing for standardizing
input. However, delicate cropping and alignment using facial landmarks is
prone to errors when face detection models fail. This is especially true in
real world images. To avoid such approach we train our technique with a
combination of random crops and resizing, following the training strategy of
[38]. The random crops make the model more robust while also providing higher
resolution inputs as details are not lost in the resizing operation.

#### Synthetic Noise Generation

In order to generate LQ images for training, most previous face restoration
approaches have used a simple first-order degradation that may not encompass
all noises present in real-world images. We use a second order noise model
from [38], ISP model from [44] and add motion blur and median blur to better
simulate real-world conditions. As noted in [46], given a high-quality input,
a restoration model should not lose details in the restoration process. We
enforce this by directly feeding the high-quality input as is with a
probability of $p_{HQ}$, which is set to a low value of 0.03 in all of our
experiments.

##  4 Experiments

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
Input | CodeFormer | DMDNet | DR2+SPAR | Ours | GT
Figure 6:  Qualitative comparison with state-of-the-art restoration models on
Celeb-Ref dataset [26] with heavy synthetic degradation. Best viewed zoomed
in.

#### Datasets

For evaluation we use Celeb-Ref [26] and real-world images collected from the
internet. Due to the large computational cost of diffusion models we choose a
small subsection of the original Celeb-Ref. For synthetic data evaluation that
contains the ground truth, we randomly choose 20 identities with at least 10
images each, for a total of 342 images. For each identity we reserve 5 images
for the personalization, leaving a total 242 images for the testing. We
further use two variations, light and heavy degradation sets, see the
supplementary for details. For real-world data we again randomly choose 20
identities from Celeb-Ref, reverse search the identities using LAION-5B-KNN
[1] and collect one image for each identity from online. We focus on high-
quality images, where the subject is far away and/or out of focus and/or with
poor illumination to best simulate real-world applications.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
Input | CodeFormer | DMDNet | DR2 + SPAR | Ours | GT
Figure 7:  Qualitative comparison with state-of-the-art restoration models on
Celeb-Ref dataset [26] with light synthetic degradation.

#### Baselines

CodeFormer [48] is state-of-the-art technique for face restoration and it uses
a codebook. DR2 [41] is based on a diffusion model and is meant for extreme
degradations. For DR2, we use the provided SPAR enhancer and empirically find
the optimal hyperparameters. DMDNet [26] is state-of-the-art method for
reference-based face restoration, for which we use the same set of 5 reference
images as for the proposed method.

#### Evaluation Metrics

For quantitative evaluation we use PSNR, SSIM, LPIPS [45], MUSIQ (KonIQ) [20],
LMSE (Landmark MSE) [47], and ID (cosine similarity with ArcFace [7]) as
metrics.

#### Settings

For methods that use reference images, 5 images are randomly sampled. For
PFStorer, the personalization fine-tuning is done for 500 iterations, which
corresponds to 10 minutes on a single A100. For all of our experiments we set
the same settings, hyperparameters and a single seed. For detailed
experimental settings see the supplementary material.

###  4.1 Comparisons

|
---|---
Figure 8:  User study results.

#### Qualitative

To evaluate the effectiveness of the proposed method we show visual results in
Figs. 4, 5, 6 and 7, for real-world, low-quality images collected from real-
world, corrupted with heavy and light degradations. For the real-world sample
we provide a pseudo-GT that can be used to compare with the identity. It can
be observed from Fig. 5 that the baseline methods fail in preserving the
identity and producing a high-quality image. Despite the difficult case on
first row Fig. 5, where the head pose is atypical, the proposed method is able
to restore the image faithfully, thanks to the learned representation of the
identity. Figure 6 shows examples with heavy synthetic degradation. Even under
heavy degradation the proposed method is able to restore the image faithfully,
while other methods struggle with retaining the identity and outputting a
realistic image. Under light degradation in Fig. 7, CodeFormer is able to
output a high-quality image while mostly retaining the identity. Our method is
able to retain even small details such as the wrinkles and skin texture.

#### Quantitative

Quantitative results on the heavily degraded images can be seen from Tab. 1.
The pixel-wise metrics PSNR and SSIM as well as the perceptual metric LPIPS
have relatively similar values across the best performing methods, with slight
differences. Notably, the big difference is in the ID metric, where the
proposed method obtains a similarity of 57.18%, almost 20 percentage points
higher than the next best performing method. This result showcases the benefit
of personalization for retaining identity features. Another major improvement
can be seen in the LMSE with almost half the error compared to CodeFormer.
This is due to the combination of a strong base model and personalization. See
supplementary for the real-world and lightly degraded samples.

Table 1: Quantitative results for images with heavy degradation.  Red
indicates the best and  blue  indicates the second best. Ref indicates whether
the model uses reference images

Methods | Ref | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | MUSIQ $\uparrow$ | LMSE $\downarrow$ | ID $\uparrow$
---|---|---|---|---|---|---|---
Input |  | 22.56 | 0.719 | 0.615 | 58.83 | 80.98 | 21.85
DMDNet [26] | ✓ | 22.64 | 0.684 | 0.491 | 47.17 | 89.26 | 29.51
DR2 + SPAR [41] |  | 22.17 | 0.701 | 0.449 | 47.36 | 40.82 | 30.01
CodeFormer [48] |  | 22.26 | 0.642 | 0.422 | 60.92 | 33.34 | 38.33
PFStorer (Ours) | ✓ | 22.62 | 0.679 | 0.414 | 64.04 | 18.37 | 57.18
GT |  | $\infty$ | 1 | 0 | 62.37 | 0 | 100

#### User Study

As the quantitative metrics are not fully able to capture the nuances of human
preferred perceptual quality, a user study is conducted. We use all three
partitions of the data. We randomly pick 100 images. To attain statistical
significance we recruit 40 users, following [30]. With two questions we have a
total of 8000 answers from users. We compare our method to only CodeFormer and
DMDNet, as DR2 often produces low-quality images. We ask users to choose
between the best image in terms of quality and identity with respect to a
reference image.

The results are shown in Fig. 8. Our method obtains the highest number of
votes in both perceived identity and quality. Our method is especially good in
capturing the identity, gaining 36.6 percentage points over the next best
method, CodeFormer. This result resonates with both the qualitative results
and quantitative metrics.

|  |  |  |
---|---|---|---|---
|  |  |  |
Input | $\lambda_{Gen}=0$ | $\lambda_{Gen}=0.1$ | $\lambda_{Gen}=0.5$ | $\lambda_{Gen}=1$
Figure 9:  (Top) In the presence of heavy degradation a larger $\lambda_{Gen}$
is able to improve results. (Bottom) With minor degradation, a larger
$\lambda_{Gen}$ can deteriorate results.

###  4.2 Further analysis

#### Personalization

Table 2 demonstrates the improvements of the proposed method for
personalization. Without personalization, the Base Model with the improved
training mechanism is able to improve over StableSR [38] in all metrics.
However, the results fall behind largely when personalization is added. Base
Model + DreamBooth [33] and Base Model + ViCo [14] attain similar metrics,
however a drop in the PSNR value even below StableSR [38] and the increase in
LMSE compared to Base Model, signifies how fine-tuning the whole model can
hurt the existing priors. For a fair comparison Base + ViCo also contains
generative regularization and other proposed training method proposed and only
lacks the learnable $\gamma$ compared to PFStorer. The $\gamma$ provides
important balance over the personalized and restored features.

Table 2: Quantitative results for different personalization methods on the
heavy portion.  Red  indicates the best and  blue  indicates the second best

Methods | Ref | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | MUSIQ $\uparrow$ | LMSE $\downarrow$ | ID $\uparrow$
---|---|---|---|---|---|---|---
Input |  | 22.56 | 0.719 | 0.615 | 58.83 | 80.98 | 21.85
StableSR [38] |  | 21.68 | 0.601 | 0.605 | 38.55 | 93.82 | 22.87
Base Model |  | 22.15 | 0.661 | 0.449 | 64.33 | 32.83 | 33.90
Base + DreamBooth [33] | ✓ | 21.13 | 0.659 | 0.487 | 62.65 | 37.48 | 52.72
Base + ViCo [14] | ✓ | 22.14 | 0.664 | 0.423 | 65.23 | 20.26 | 53.92
PFStorer (Ours) | ✓ | 22.62 | 0.679 | 0.414 | 64.04 | 18.37 | 57.18
GT |  | $\infty$ | 1 | 0 | 62.37 | 0 | 100

#### Alignment-Free Training and Existing Priors

An immediate benefit to our landmark- and alignment-free approach is that it
can be run even when the landmark model fails, as can be seen from the top row
of Fig. 6. Furthermore, due to the existing priors of the Base Model, the
model is able to restore details from the full head and not only the face, see
the result from CodeFormer from Fig. 5 top.

#### Generative Regularization

Figure 9 showcases results with different values of the weight $\lambda_{Gen}$
of generative regularization. A larger $\lambda_{Gen}$ encourages more
hallucination, which is beneficial for unseen cases, while a smaller
$\lambda_{Gen}$ focuses more on the restoration. To balance the effects we use
a default $\lambda_{Gen}=0.1$ for all of our experiments based on empirical
observations.

###  4.3 Limitations

We show an example of a limitation in Fig. 10. The output is faithful to the
given reference images, hence if there are changes in the appearance between
references and the input the result may be unwanted. As the model is based on
Stable Diffusion it inherits its limitations of slow sampling speed and
occasional unwanted artifacts and hallucinations due to the stochasticity. As
a possible solution to stochasticity, concurrent work [6] guides the model
towards visually appealing results.

|  |
---|---
|
|  |
References | Input | PFStorer | GT
Figure 10:  The output is as accurate as the given reference images are.

##  5 Conclusions

In this work, we introduce the use of personalization for the task of face
restoration, where a restoration model is personalized using a few images of a
person. We postulate that the problem of face restoration is an ill-posed
problem and requires the use of a personal prior for faithful results. We
propose the use of a personalization adapter that preserves existing priors of
the base restoration model. To enhance the training generative regularization
is designed. We showcase our method’s abilities through qualitative,
quantitative and a user study.

#### Acknowledgements

This work was supported by the Research Council of Finland Academy Professor
project EmotionAI (grants 336116, 345122), ICT 2023 project TrustFace (grant
345948), the University of Oulu & Research Council of Finland Profi 7 (grant
352788), and by Infotech Oulu.

## References

  * [1] LAION AI.  Laion-knn api.  https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn.laion.ai&index=laion5B-H-14&useMclip=false.  Accessed 10-10-2023.
  * Alaluf et al. [2023] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or.  A neural space-time representation for text-to-image personalization, 2023.
  * Chen et al. [2021] Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K Wong.  Progressive semantic-aware style transformation for blind face restoration.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11896–11905, 2021.
  * Chen et al. [2023] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision transformer adapter for dense predictions, 2023.
  * Cohen et al. [2023] Noa Cohen, Hila Manor, Yuval Bahat, and Tomer Michaeli.  From posterior sampling to meaningful diversity in image restoration, 2023.
  * Dai et al. [2023] Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al.  Emu: Enhancing image generation models using photogenic needles in a haystack.  _arXiv preprint arXiv:2309.15807_ , 2023.
  * Deng et al. [2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4690–4699, 2019.
  * Dogan et al. [2019] Berk Dogan, Shuhang Gu, and Radu Timofte.  Exemplar guided face image super-resolution without facial landmarks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 0–0, 2019.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Gal et al. [2023] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–13, 2023.
  * Gu et al. [2023] Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, et al.  Photoswap: Personalized subject swapping in images.  _arXiv preprint arXiv:2305.18286_ , 2023.
  * Gu et al. [2022] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng.  Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder.  In _European Conference on Computer Vision_ , pages 126–143. Springer, 2022.
  * Hao et al. [2023] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan-Yee K Wong.  Vico: Detail-preserving visual condition for personalized text-to-image generation.  _arXiv preprint arXiv:2306.00971_ , 2023.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Huang et al. [2008] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller.  Labeled faces in the wild: A database forstudying face recognition in unconstrained environments.  In _Workshop on faces in’Real-Life’Images: detection, alignment, and recognition_ , 2008.
  * Karras et al. [2020a] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8110–8119, 2020a.
  * Karras et al. [2020b] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8110–8119, 2020b.
  * Ke et al. [2021] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang.  Musiq: Multi-scale image quality transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5148–5157, 2021.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Li et al. [2023] Dongxu Li, Junnan Li, and Steven CH Hoi.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  _arXiv preprint arXiv:2305.14720_ , 2023.
  * Li et al. [2018] Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang.  Learning warped guidance for blind face restoration.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 272–289, 2018.
  * Li et al. [2020a] Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang.  Blind face restoration via deep multi-scale component dictionaries.  In _European conference on computer vision_ , pages 399–415. Springer, 2020a.
  * Li et al. [2020b] Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo.  Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2706–2715, 2020b.
  * Li et al. [2022] Xiaoming Li, Shiguang Zhang, Shangchen Zhou, Lei Zhang, and Wangmeng Zuo.  Learning dual memory dictionaries for blind face restoration.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5):5904–5917, 2022.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pages 3730–3738, 2015.
  * Ma et al. [2023] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu.  Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning.  _arXiv preprint arXiv:2307.11410_ , 2023.
  * Nitzan et al. [2022] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or.  Mystyle: A personalized generative prior.  _ACM Transactions on Graphics (TOG)_ , 41(6):1–10, 2022.
  * Otani et al. [2023] Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin’ichi Satoh.  Toward verifiable and reproducible human evaluation for text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14277–14286, 2023.
  * Qiu et al. [2023] Xinmin Qiu, Congying Han, ZiCheng Zhang, Bonan Li, Tiande Guo, and Xuecheng Nie.  Diffbfr: Bootstrapping diffusion model towards blind face restoration.  _arXiv preprint arXiv:2305.04517_ , 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023a] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023a.
  * Ruiz et al. [2023b] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  _arXiv preprint arXiv:2307.06949_ , 2023b.
  * Su et al. [2023] Yu-Chuan Su, Kelvin CK Chan, Yandong Li, Yang Zhao, Han Zhang, Boqing Gong, Huisheng Wang, and Xuhui Jia.  Identity encoder for personalized diffusion.  _arXiv preprint arXiv:2304.07429_ , 2023.
  * Tang et al. [2023] Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E. Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, and Michael Rubinstein.  Realfill: Reference-driven generation for authentic image completion, 2023.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Wang et al. [2023a] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin C. K. Chan, and Chen Change Loy.  Exploiting diffusion prior for real-world image super-resolution, 2023a.
  * Wang et al. [2018] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 606–615, 2018.
  * Wang et al. [2021] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan.  Towards real-world blind face restoration with generative facial prior.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9168–9178, 2021.
  * Wang et al. [2023b] Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang.  Dr2: Diffusion-based robust degradation remover for blind face restoration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1704–1713, 2023b.
  * Xiao et al. [2023] Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han.  Fastcomposer: Tuning-free multi-subject image generation with localized attention.  _arXiv preprint arXiv:2305.10431_ , 2023.
  * Yue and Loy [2022] Zongsheng Yue and Chen Change Loy.  Difface: Blind face restoration with diffused error contraction.  _arXiv preprint arXiv:2212.06512_ , 2022.
  * Zhang et al. [2023] Kai Zhang, Yawei Li, Jingyun Liang, Jiezhang Cao, Yulun Zhang, Hao Tang, Deng-Ping Fan, Radu Timofte, and Luc Van Gool.  Practical blind image denoising via swin-conv-UNet and data synthesis.  _Machine Intelligence Research_ , 2023.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhao et al. [2023] Yang Zhao, Tingbo Hou, Yu-Chuan Su, Xuhui Jia, Yandong Li, and Matthias Grundmann.  Towards authentic face restoration with iterative diffusion models and beyond.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7312–7322, 2023.
  * Zheng et al. [2022] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen.  General facial representation learning in a visual-linguistic manner.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18697–18709, 2022.
  * Zhou et al. [2022] Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy.  Towards robust blind face restoration with codebook lookup transformer.  _Advances in Neural Information Processing Systems_ , 35:30599–30611, 2022.

Supplementary Material

Figure 11: User interface used in the user study.

This supplementary material contains the following sections. First, background
is provided for the used models. Next, further details of the user study are
provided. Then, full experimental details and additional experiments of the
personalized model are shown. Finally, the training details and further
experiments of the non-personalized base restoration model are discussed. As
the last section, societal impact is discussed.

## Background

In this section we provide sufficient background to keep the paper self-
sustained. We first introduce latent diffusion models, namely Stable Diffusion
[32], as both the used methods, StableSR [38] and ViCo [14] are based off of
it. Next we provide further details of the base model, StableSR and the
personalization technique ViCo.

#### Latent Diffusion Models

As oppose to diffusion models [16], LDMs [32] (latent diffusion models)
perform the diffusion steps in a latent space. In Stable Diffusion [32] an
encoder $\mathcal{E}$ is first trained to map input images
$x\in\mathbb{R}^{H\times W\times 3}$ to a latent code
$z=\mathcal{E}(x)\in\mathbb{R}^{(H/8)\times(W/8)\times 4}$, which can be
approximately reconstructed with a decoder $\mathcal{D}$. The diffusion
forward and backward steps are then performed within the latent space. To
perform conditional generation using text $y$, it is first transformed to an
embedding $c(y)$ with a text embedder. The training loss is then given by:

$\mathcal{L}=\mathbb{E}_{z\sim\mathcal{E}(x),y,{\epsilon}\sim\mathcal{N}(0,1),t}\left[||{\epsilon}-{\epsilon}_{\theta}(z_{t},c(y),t)||^{2}_{2}\right].$
(6)

Above, at timestep $t$ the diffusion model ${\epsilon}_{theta}$ denoises the
added noise from $z_{t}$ conditioned on the text embedding $c(y)$ and the
timestep $t$. After training the model can be used to generate images with a
text prompt $y$ and starting from a $z_{t}\sim\mathcal{N}(0,1)$ iteratively
until all noise is removed at $t=0$.

#### StableSR

To exploit the rich generative prior in Stable Diffusion [32] for image
restoration, StableSR [38] uses conditioning of low-quality images. The entire
Stable Diffusion is kept frozen, while a time-aware encoder and spatial
feature transformations are added as adapters to condition the low-quality
images. The encoder takes in a low-quality image and the current time-step of
the diffusion model and outputs feature maps at different resolutions,
corresponding to the ones in Stable Diffusion’s UNet. The features are then
fed through spatial feature transforms and added to the output of the original
UNet layer’s intermediate outputs.

The model is trained on high-quality generic natural images of 2k and 8k
resolutions, which are randomly cropped to $512\times 512$. This training
enables the use of arbitrary size super-resolution using aggregation sampling.
Here, the input is split to overlapping tiles, which are processed by the
model independently. To avoid border artifacts a Gaussian kernel is used for
the fusion of the tiles.

#### ViCo

For efficient and accurate personalization, ViCo [14] also uses Stable
Diffusion as its base. Similarly to StableSR, ViCo keeps the entire Stable
Diffusion frozen. Only added adapter blocks and a single text-embedding are
trained. Similar to [10], a text-embedding is made learnable that can be
associated with the subject. Additionally, image cross-attention adapters are
added to four blocks of the UNet. These cross-attention layers take the
current time-step’s intermediate result and the intermediate features of a
reference image, which has also gone through the UNet. To enhance the result,
a mask is used to ignore the background. The mask is obtained from an
attention map $A=\mathrm{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)$,
which is a product of the reference image and the text-embedding. A
regularizer

$\mathcal{L}_{Pers}=||A_{\star}/\max(A_{\star})-A_{\mathrm{EOT}}/\max(A_{\mathrm{EOT}})||_{2}^{2},$
(7)

where $A_{\star}$ are the similarity logits corresponding to the learnable
token of the text-embedding and $A_{\mathrm{EOT}}$ is the end-of-text token,
is used to avoid overfitting. The end-of-text token $\mathrm{<|EOT|>}$
captures a global representation, which retains good semantics of the
personalizable object through training.

## User Study Details

From the light and heavy partitions we randomly select two images for each
identity. From the real data, we select all of the images. In total we have
$20\times 2+20\times 2+20\times 1=100$ images. With 40 users and two tasks we
have a total of 8000 unique answers. As identifying fine-grained details of an
identity, especially not a familiar one, can be difficult, we chose four
images from each identity (light and heavy). This way the users can get more
accustomed to the identities and make more accurate evaluations for the
similarity of the identity features.

To ensure that the users are being accurate with their annotations, we use
five control tasks. Here a ground-truth image is paired with extremely poor
quality images. If the user fails in these tasks, their annotations are likely
to be inaccurate and their results can be potentially invalidated. To avoid
biases with users always choosing A, B or C, we randomly shuffle the model’s
positions.

Figure 11 displays the user interface used for the study. Users have to read
the full instructions before taking the study. The instructions detail the two
different tasks and how they should be evaluated.

Amazon Mechanical Turk is used for the study. We follow principles from [30].
We filter users based on the Master certificate to ensure quality annotations.
For each task we pay $ 0.04 as suggested in [30].

## Personalized Model Additional Experiments

In this section we detail the full experimental settings and conduct
additional experiments of hyperparameters for the personalized model. In all
experiments, excluding the parameter to be studied other hyperparameters are
kept constant. With further fine-tuning of hyperparameters, results for
specific individuals and inputs can be improved.

#### Settings

Followed by community findings that prompts can improve quality of the
restored image, we use both a positive and a negative prompt. For the positive
prompt we use ”a Photo of * , masterpiece, best quality, realistic, very
clear, professional” and for the negative prompt we use ”3d, cartoon, anime,
sketches, worst quality, low quality”. We note that including semantic changes
in the prompt like ”red hair” does not have an effect. This is due to the
restoration blocks fusing the low-quality image with the denoised image
directly. This is in line with the goal of the paper, as it is a restoration
method, not an editing method.

For the classifier-free guidance value we set 4 as a default for all
experiments. Standard DDPM [16] sampling is used with 200 steps as in StableSR
[38]. As the personalization fine-tuning approach is about learning a single
identity and not multiple parts of an identity, we find that not using the 50%
random crops improves the results slightly. For the baseline method DR2 + SPAR
[41] we empirically experimented with several hyperparameters values of $N$
and $\tau$ that are crucial for the performance of the method. $N$ is a
downsampling factor and $\tau$ is the output step after which generation is
started. We set $N=8$ and $\tau=40$ as we found it performed the best across
different levels of degradations. For LMSE (Landmark MSE) [47] was used to
obtain landmarks. In cases where landmarks could not be found due to the image
being severely degraded the MSE was set to 128. Similarly in cases where the
MSE was for an image was more than 128 it was capped to 128 to avoid outliers
due to numerical errors or other errors.

#### Degradations

During testing we synthesized a light and a heavy degradation to better
evaluate our algorithm in different situations. During training we use the
heavy setting. We use the settings from StableSR as a base and modify them. To
better suit for real-world applications we include motion and median blur, as
well as adding ISP (Image Signal Processing) noise [44].

To ensure our method works in less severe cases, we also include a light
partition during testing. Here, we only include a first-order noise similar to
CodeFormer.

The light portion follows:

$I_{D}=\\{[(I\circledast
k_{\sigma})_{\downarrow_{r}}+n_{\delta}]_{\text{JPEG}_{q}}\\}_{\uparrow_{r}},$
(8)

where $k_{\sigma}$ is Gaussian blur kernel, $\downarrow_{r}$ and
$\uparrow_{r}$ are the downsamping and upsampling operators, $n_{\delta}$
additive Gaussian noise and $[\cdot]_{\text{JPEG}_{q}}$ is JPEG compression.
We sample uniformly $\sigma$, $r$, $\delta$ and $q$ from $[0.1,10]$, $[1,4]$,
$[0,2]$ and $[30,100]$, respectively. The additive Gaussian noise has a
probability of 40% and downsampling a probablity of 70%, while filtering and
JPEG compression occur always.

The heavy portion first applies ISP model [44] with a 50% probability,
followed by motion and median blur with 5% and 10% probabilities. Next, we use
equation 8 and the same settings except, $r$ and $\delta$ are chosen from
$[1,10]$ and $[0,15]$, respectively, followed by a sinc filter [38]. Finally,
equation 8 is applied a second time with a 90% probability.

#### Classifier-Free Guidance Value

To further emphasize the conditional element, CFG [15] can be used to guide
the denoising process. As mentioned earlier, we use a negative prompt instead
of a null one. The formula is given by

$\tilde{X}=X+\lambda_{cfg}(X(p_{pos},I_{LQ})-X(p_{neg},I_{LQ})),$ (9)

where $p_{neg}$ and $p_{pos}$ correspond to the positive and negative prompts.
We also experimented with null conditioning the low-quality image

$\tilde{X}=X+\lambda_{cfg}(X(p_{pos},I_{LQ})-X(p_{neg},\varnothing)),$ (10)

but found the results to be of lower-quality, as emphasizing the low-quality
image may exaggerate blurry features.

We experiment using Eq. 9 with different CFG values $\lambda_{cfg}$ in Fig. 12
and note that $\lambda_{cfg}=1$ corresponds to not using guidance at all. It
can be seen that a higher $\lambda_{cfg}$ can oversature, as in the top row.
In the lower row, a low $\lambda_{cfg}$ loses identity features, whereas in
the top row it is more subtle. From our experiments we observe that different
identities behave differently with different $\lambda_{cfg}$. A common value
in text-to-image applications is $\lambda_{cfg}=7.5$, but to avoid saturation
we default to $\lambda_{cfg}=4$ in all of our other experiments.

|  |  |  |
---|---|---|---|---
$\lambda_{cfg}=1$ | $\lambda_{cfg}=2$ | $\lambda_{cfg}=4$ | $\lambda_{cfg}=7.5$ | GT
|  |  |  |
$\lambda_{cfg}=1$ | $\lambda_{cfg}=2$ | $\lambda_{cfg}=4$ | $\lambda_{cfg}=7.5$ | GT
Figure 12:  Experiments with different classifier-free guidance values.

#### Controlling Identity

The used personalization technique consists of two parts. 1) a learnable text-
embedding and 2. image cross-attention. We experiment with different values of
$\lambda_{att}$, which controls the weight of the cross-attention layers, in
Fig. 13. The sample with $\lambda_{att}=0$ corresponds to only using the
learnable text-embedding. We can see that it contains some identity at a high-
level but is missing details such as wrinkles and dip in the chin. With
increasing $\lambda_{att}$ the identity features become more prominent, even
to a degree of exaggeration. Similar to CFG values, we have observed that for
different individuals and depending on the noise levels of the input,
$\lambda_{att}$ acts differently. We chose $\lambda_{att}=1$ as a default
value, although in some cases, like this sample, the optimal results can be
something different.

|  |  |  |
---|---|---|---|---
Default | $\lambda_{att}=-0.5$ | $\lambda_{att}=0$ | $\lambda_{att}=0.1$ | $\lambda_{att}=0.25$
|  |  |  |
$\lambda_{att}=0.5$ | $\lambda_{att}=0.75$ | $\lambda_{att}=1$ | $\lambda_{att}=1.5$ | GT
Figure 13:  Controlling identity. The default corresponds to non-personalized
output. Samples with $\lambda_{att}$, use the personalized token in the
prompt.

#### Number of Reference Images

In Fig. 14 we experiment with how many reference images are required to
accurately capture the identity. With $n_{img}=0$, i.e. no personalization,
high-level features matching the input can be observed. With just one
reference image, the eyes, eyebrows and other finer details start to appear.
We default to using $n_{img}=5$ as it often performs sufficiently and the
addition of more images has less noticeable effect. For some individuals we
found that even three images can be sufficient, but it should be noted that
the similarities between input image and the reference images affect the
results.

|  |  |
---|---|---|---
Input | $n_{img}=0$ | $n_{img}=1$ | $n_{img}=2$
|  |  |
$n_{img}=3$ | $n_{img}=5$ | $n_{img}=10$ | GT
Figure 14:  Number of images used for personalization. $n_{img}=0$ refers to
no personalization.

#### Randomness with Different Seeds

Diffusion models are stochastic and notorious for unsatisfactory results with
different random initializations. Figure 15 contains results for an image with
light and heavy degradations with four different seeds. For the light portion,
the outputs tend to be mostly similar with small differences like skin
texture. With the heavy portion, there are noticeable differences in the
mouth, eyes and colors, although the identity is kept the same. Interestingly
the background logo and text deviate largely, as they are not part of the
learned personalization.

|  |  |  |  |
---|---|---|---|---|---
LQ-Light | Seed = 0 | Seed = 1 | Seed = 2 | Seed = 3 | GT
|  |  |  |  |
LQ-Heavy | Seed = 0 | Seed = 1 | Seed = 2 | Seed = 3 | GT
Figure 15:  Random seed effect.

#### Additional Qualitative Results

Here we provide additional results on the light, heavy and real portions of
the Celeb-Ref [26] dataset. From Fig. 17 row three, we can see that the DMDNet
is able to better preserver the identity compared to CodeFormer, but compared
to ours it is still missing fine-grained details such as the notch in the
chin. Despite achieving good results with light degradations, DMDNet struggles
with the heavy and real degradations in Figs. 18 and 19. Although DR2 provides
poor results in several cases, it works well on row 4 of Fig. 18.

Despite the input being very noisy and small in size, our result is faithful
with the identity, while codeformer struggles due to requiring alignment.
Figure 16 contains a qualitative comparison between different personalization
techniques and more results are provided in Figs. 20 and 21.

|  |  |  |
---|---|---|---|---
|  |  |  |
Input | Base Model | Base Model | Ours | GT
| \+ DreamBooth | \+ ViCo |  |
Figure 16:  Results using different personalization techniques combined with a
base restoration model. DreamBooth [33] is not able to capture all the details
and can result in poor-quality images. ViCo is better able to capture most
details, but can still result in blurry images. Ours is able to capture fine-
grained details without hurting the restoration performance of the Base Model.
Zoom in for best view.

#### Additional Quantitative Results

To complete the quantitative results of heavy portion from Tab. 1, the results
of light and real portions are presented.

Table 3: Quantitative results for the real portion of the data.  Red
indicates the best and  blue  indicates the second best

Methods | Ref |  MUSIQ $\uparrow$ |  ∗ID $\uparrow$
---|---|---|---
Input |  | 24.84 | 20.37
StableSR [38] |  | 51.59 | 23.39
Base Model |  | 60.27 | 24.29
Base Model + DreamBooth | ✓ | 55.31 | 29.78
Base Model + ViCo | ✓ | 57.67 | 24.71
DMDNet [26] | ✓ | 58.36 | 22.29
DR2 [41] |  | 29.18 | 18.38
CodeFormer [48] |  | 44.60 | 22.90
PFStorer (Ours) | ✓ | 60.11 | 25.01

∗ Compare with a reference image.

Table 3 tabulates the results for the real portion. As no GT is available, we
only use MUSIQ [20] and ID [7] as metrics. For the ID we use a reference image
of the same person. As can be seen from the results, the ID metric drops
significantly compared to the heavy portion, where ID used GT image. Despite
this the rankings of the results remain similar with ours as first and DMDNet
and CodeFormer being close with similar results and DR2 achieving the lowest
due to blurry results. Base Model + DreamBooth [33] achieves the best result
in ID which is likely due to overfitting to the identity, with poor
restoration results. Table 4 presents results for the light partition. Our
method is consistently among the second best performers, although the
differences between the methods are minor.

#### Value of Learnable $\gamma$ after training

Each layer $l$ has vector $\gamma^{l}$ with the size depending on the layer
hidden dimension. The values of the mean of the vector for each layer is
around 0.2 and 0.5. The higher importance of 0.5 values is from the middle of
the UNet layers, where the resolution is lowest and the lower values of 0.2 at
the higher resolution layers.

## Non-Personalized Base Model Experiments

In this section we cover the training details and results with the Base Model
without personalization. The model is a pre-trained StableSR [38] without any
modifications to the architecture. The personalized models all use the fine-
tuned Base Model described in this section as their starting point.

#### Training

The training is performed on a facial dataset FFHQ [18], which contains 70,000
facial images in the resolution of $1024\times 1024$. 50% of the data is
resized randomly to $512\times 512$ and the other 50% are taken as random
crops of the same resolution. Fine-tuning is performed for 12 epochs. At this
moment the personalizion adapter is not attached to the model. We synthesize
training data in the same manner as the personalized model with the heavy
degradation.

Table 4: Quantitative results for the light portion of the data.  Red
indicates the best and  blue  indicates the second best

Methods | Ref |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  MUSIQ $\uparrow$ |  LMSE $\downarrow$ |  ID $\uparrow$
---|---|---|---|---|---|---|---
Input |  | 22.56 | 0.719 | 0.615 | 58.83 | 9.74 | 21.85
StableSR |  | 27.18 | 0.767 | 0.337 | 62.96 | 5.73 | 70.62
Base Model |  | 27.72 | 0.767 | 0.318 | 64.16 | 4.93 | 72.43
Base Model + DreamBooth | ✓ | 24.77 | 0.721 | 0.419 | 62.01 | 14.29 | 62.57
Base Model + ViCo | ✓ | 27.58 | 0.765 | 0.325 | 63.04 | 4.88 | 73.26
DMDNet [26] | ✓ | 27.72 | 0.780 | 0.312 | 63.07 | 6.43 | 72.66
DR2 [41] |  | 22.17 | 0.701 | 0.449 | 47.36 | 13.13 | 30.01
CodeFormer [48] |  | 27.19 | 0.759 | 0.293 | 66.00 | 5.91 | 69.13
PFStorer (Ours) | ✓ | 27.71 | 0.767 | 0.309 | 63.31 | 4.79 | 75.39
GT |  | $\infty$ | 1 | 0 | 62.37 | 0 | 100

#### Qualitative Results

For qualitative results on synthetic degradation on CelebA-Test split [27],
see Fig. 22. The synthetic degradation for Celeba-Test is obtained from [40].
Compared to CodeFormer [48] our method is able to generate more fine-grained
details, while being more faithful to the low-quality image, _e.g_., the color
of facial hair on top. Results from real-world datasets, LFW [17], WebPhoto
[40] and Wider-Test [48] are shown in Fig. 23. In LFW, which contains less
severe degradations, compared to CodeFormer our method is able to generate
more details with sharper textures. Our method struggles with WebPhoto, as it
contains old images with scratches, color degradation and other untypical
degradations. With severe degradation on Wider-Test, our method is able to
generate realistic images, while CodeFormer struggles with artifacts.

#### Quantitative Results

We provide quantitative results with standard metrics. Table 5 tabulates
results from CelebA-Test, where the results are taken from [40], except for
CodeFormer and ours. In most of the metrics the results are similar between
GFP-GAN, CodeFormer and ours. In the real-world datasets, Tab. 6, our method
obtains the best FID for LFW and WIDER.

Table 5: Quantitative results for CelebA-Test with non-personalized model.
Red  indicates the best and  blue  indicates the second best

Methods |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  MUSIQ $\uparrow$ |  FID $\downarrow$ |  ID $\uparrow$
---|---|---|---|---|---|---
Input | 25.35 | 0.684 | 0.486 | 58.83 | 143.98 | 52.06
DFDNet | 23.68 | 0.662 | 0.4341 | N/A | 59.08 | 59.69
PULSE | 21.61 | 0.620 | 0.4851 | N/A | 67.56 | 30.45
GFP-GAN | 25.08 | 0.677 | 0.3646 | N/A | 42.62 | 65.40
CodeFormer | 26.77 | 0.719 | 0.343 | 66.54 | 52.44 | 62.73
Base Model | 26.03 | 0.680 | 0.392 | 66.57 | 40.36 | 63.89
GT | $\infty$ | 1 | 0 | 63.43 | 43.43 | 1

Table 6: Quantitative results for real-world datasets with non-personalized
model.  Red  indicates the best and  blue  indicates the second best

Dataset | LFW-Test | WebPhoto-Test | WIDER-Test
---|---|---|---
Degradation | mild | medium | heavy
Methods |  FID$\downarrow$ |  MUSIQ$\uparrow$ |  FID$\downarrow$ |  MUSIQ$\uparrow$ |  FID$\downarrow$ |  MUSIQ$\uparrow$
Input | 137.56 | 25.05 | 170.11 | 19.24 | 202.06 | 15.57
PULSE | 64.86 | 66.98 | 86.45 | 66.57 | 73.59 | 65.36
DFDNet | 62.57 | 67.95 | 100.68 | 63.81 | 57.84 | 59.34
GFP-GAN [40] | 49.96 | 68.95 | 87.35 | 68.04 | 40.59 | 68.26
CodeFormer [48] | 52.02 | 71.43 | 78.87 | 70.51 | 39.06 | 69.31
Base Model | 44.11 | 66.57 | 80.90 | 62.69 | 34.72 | 63.91
Light degradation | 44.02 | 62.69 | 84.81 | 57.64 | 82.93 | 51.66

#### Ablation: Heavy Degradation

We show that with more complex degradations the method is able to perform
better in cases with severe degradation. The results are tabulated in bottom
of Tab. 6. Base Model uses the heavy degradation, where as the Light
degradation does not. For LFW, which has relatively mild degradations, the
performance between Base Model and simple degradation does not change
drastically as expected. However, for WIDER-Test we can see a large difference
as the FID more than doubles from 34.72 to 82.93, meaning a significant
decrease in quality. Using heavy degradation results in higher quality outputs
under severe degradation, while having minimal effect on mild cases.

## Societal Impact

Machine learning models can learn biases from their datasets. We show that our
model is capable of working with different ethnicities and skin tones, while
acknowledging that the testing is limited. We also note that since our model
is built upon previous models, it inherits any biases these models may
contain. To avoid misunderstanding of the capabilites of our models, e.g.,
using it for enhancing security footage for criminal investigations, we have
shown the limitations in experiments and emphasize that the identity of the
restored individual needs to be known beforehand. Malicious users may want to
mislead viewers with generated images, which is a common common issue with
existing similar methods. However, recent approaches in detecting fake imagery
are improving rapidly.

#### Privacy and Image Copyrights

In this paper we showcase several pictures of individuals. Several images are
from the publicly available Celeb-Ref dataset [26]. Images shown from the
collected 20 image dataset are of well-known celebrities and are under a
Creative Commons license. Real world images not part of the collected dataset
are under public domain or a Creative Commons license.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Input | CodeFormer | DMDNet | DR2+SPAR | Ours | GT
Figure 17:  Qualitative comparison with state-of-the-art restoration models on Celeb-Ref dataset [26] with synthetic light degradation.  |  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Input | CodeFormer | DMDNet | DR2+SPAR | Ours | GT
Figure 18:  Qualitative comparison with state-of-the-art restoration models on Celeb-Ref dataset [26] with synthetic heavy degradation.  |  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Input | CodeFormer | DMDNet | DR2+SPAR | Ours | Pseudo-GT
Figure 19:  Qualitative comparison with state-of-the-art restoration models on Celeb-Ref dataset [26] with real degradation.  |  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Input | Base Model | Base Model | Ours | GT
| \+ DreamBooth | \+ ViCo |  |
Figure 20:  Results using different personalization techniques combined with a base restoration model with heavy degradation.  |  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Input | Base Model | Base Model | Ours | GT
| \+ DreamBooth | \+ ViCo |  |
Figure 21:  Results using different personalization techniques combined with a base restoration model with light degradation.  |  |  |
---|---|---|---
|  |  |
Input | CodeFormer | Base Model | GT
Figure 22:  Qualitative comparison with state-of-the-art restoration models on CelebA-Test [27] with synthetic degradation.  |  |  |  |  |
---|---|---|---|---|---
Input | CodeFormer | Base Model) | Input | CodeFormer | Base Model
LFW-Test | LFW-Test
|  |  |  |  |
Input | CodeFormer | Base Model | Input | CodeFormer | Base Model
WebPhoto | WebPhoto
|  |  |  |  |
Input | CodeFormer | Base Model | Input | CodeFormer | Base Model)
Wider-Test | Wider-Test
Figure 23:  Qualitative comparison with state-of-the-art restoration models on
LFW [17], WebPhoto [40] and Wider-Test [48].