Synthesizing Realistic Image Restoration Training Pairs: A Diffusion Approach

Tao Yang[1], Peiran Ren[1], Xuansong Xie[1], and Lei Zhang[2]

[1]DAMO Academy, Alibaba Group

[2]Department of Computing, The Hong Kong Polytechnic University

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
In supervised image restoration tasks, one key issue is how to obtain the aligned high-quality (HQ) and low-quality (LQ) training image pairs. Unfortunately, such HQ-LQ training pairs are hard to capture in practice, and hard to synthesize due to the complex unknown degradation in the wild. While several sophisticated degradation models have been manually designed to synthesize LQ images from their HQ counterparts, the distribution gap between the synthesized and real-world LQ images remains large. We propose a new approach to synthesizing realistic image restoration training pairs using the emerging denoising diffusion probabilistic model (DDPM).
First, we train a DDPM, which could convert a noisy input into the desired LQ image, with a large amount of collected LQ images, which define the target data distribution. Then, for a given HQ image, we synthesize an initial LQ image by using an off-the-shelf degradation model, and iteratively add proper Gaussian noises to it. Finally, we denoise the noisy LQ image using the pre-trained DDPM to obtain the final LQ image, which falls into the target distribution of real-world LQ images. Thanks to the strong capability of DDPM in distribution approximation, the synthesized HQ-LQ image pairs can be used to train robust models for real-world image restoration tasks, such as blind face image restoration and blind image super-resolution. Experiments demonstrated the superiority of our proposed approach to existing degradation models. Code and data will be released.

§ INTRODUCTION

Deep neural networks (DNNs) [24] have been successfully used in a variety of computer vision tasks, including image classification [12], object detection [34], segmentation [11], as well as image restoration [6]. In supervised learning, the amount and quality of labeled training data will largely affect the practical performance of trained DNN models. This problem becomes more crucial in real-world image restoration tasks [51, 52], where the aligned high-quality (HQ) and low-quality (LQ) training image pairs are very difficult to capture in practice.
Early works mainly resort to using bicubic downsampling or some simple degradation models [23] to synthesize LQ images from their HQ counterparts, which however can only cover a very small set of degradation types. The real-world LQ images can suffer from many factors, including but not limited to low resolution, blur, noise, compression, ., which are too complex to be explicitly modeled. As a result, the DNN models trained on synthesized HQ-LQ training pairs can hardly perform well on real-world LQ images.

To alleviate the above difficulties, some works have been proposed to first predict the degradation parameters [9, 10, 18] and then restore the HQ image with them in a non-blind way. They work well on some specific non-blind deblurring  [9], denoising [10] and JPEG artifacts removal [18] tasks. However, the degradation of real-world LQ images are often unknown and cannot be pre-defined, making it is hard, if not possible, to predict accurate degradation parameters. Some researchers attempted to collect real-world LQ-HQ pairs [2, 48] by using long-short camera focal lengths, which only work in applications where similar photographing devices are used. Recently, a couple of handcrafted degradation models [52, 45] have been proposed to model complex real-world degradations. Zhang [52] randomly shuffled blur, downsampling and noise degradations to form a complex combination. Wang [45] developed a high-order degradation model. While these methods can simulate a much larger scope of degradation types and have shown impressive progress in handling LQ images in the wild, the distribution gap between the synthesized and real-world LQ images remains large [26].

With the rapid advancement of deep generative networks [8, 20, 15], methods have been developed to learn how to synthesize LQ images from their HQ counterparts. Lugmayr [29] learned a domain distribution network using unpaired data and then built HQ-LQ pairs with it. Similarly, Fritsche [7] synthesized LQ images by using DSGAN to introduce natural image characteristics. Luo [30] proposed a probabilistic degradation model (PDM) to describe different degradations. Most recently, Li [26] developed the ReDegNet to model real-world degradations using paired face images and transfer them to produce LQ natural images. However, this method largely relies on the HQ faces generated by blind face restoration models [51, 44], which limits its applications.

In this work, we revisit the problem of HQ-LQ image pair synthesis, which is critical to the many image restoration tasks such as blind face restoration and real-world image super-resolution. Our idea is to seamlessly integrate the advantages of handcrafted degradation models and deep generative networks. We first train a generator, , the emerging denoising diffusion probabilistic model (DDPM) [15], by using a large amount of real-world LQ images collected from the Internet. The trained DDPM can be used to generate realistic LQ images. When building HQ-LQ image pairs, we first adopt a handcrafted degradation model to synthesize initial LQ images from the input HQ images, which may fall into the distribution of real-world LQ images. To reduce the distribution gap, the initially synthesized LQ images are added with Gaussian noise and then denoised by the pre-trained DDPM. After a few steps, the distribution of synthesized LQ images will become closer and closer to the distribution of real-world LQ images. Finally, a set of realistic HQ-LQ training pairs can be synthesized, and they can be used to train robust DNN models for image restoration tasks.

In summary, in this work we present a novel diffusion approach to synthesizing realistic training pairs for image restoration, targeting at shortening the distribution gap between synthetic and real-world LQ data. The synthesized HQ-LQ training pairs can be used for various downstream real-world image restoration tasks, as validated in our experiments of blind image restoration and blind image super-resolution. To the best of our knowledge, this is the first image degradation modeling method based on diffusion models. Codes will be made publicly available.

§ RELATED WORK

§.§ Degradation Modeling

Image degradation modeling is of great importance for many downstream tasks such as blind face restoration (BFR) [51] and blind image super-resolution (BISR) [52].
Bicubic downsampling has been popularly adopted in the research of single image super-resolution (SISR) [6, 25, 46]. Though it provides a common platform for comparison of SISR algorithms, models trained with this degradation strategy are of little avail, especially for real-world applications. Some works turned to use the classical degradation models [23], which take the commonly observed noise, blur, and downsampling degradations into consideration. Unfortunately, they are far from enough to describe the complex unknown degradations in real-world LQ images.

Zhang [52] proposed a random shuffling strategy to construct more complex degradations of blur, downsampling and noise. Concurrently, Wang [45] designed a high-order degradation model with several repeated degradation processes using the classical degradation model. These two methods managed to simulate diverse degradation combinations.
Considering that real-world degradations are too complex to be explicitly modeled, some researchers attempted to learn a network to implicitly approximate the degradation process [29, 7, 31, 49, 42, 43, 30, 26]. Nonetheless, it is challenging to learn such models due to the high illness of the problem.

Blind Face Restoration.
Face image restoration has attracted a lot of attentions [28, 16, 51, 44, 54]. Since facial images have specific structures, it is feasible to restore a clear face image from real-world degraded observations [51, 44, 54]. Many BFR methods have been proposed by resorting to exemplar images [28], 3D facial priors [16], and facial component dictionaries [27]. Recently, generative face prior has been widely used and shown powerful capability in BFR tasks [51, 44]. It has been demonstrated that many LQ face images in the wild can be robustly restored. Researchers have also found that a learned discrete codebook prior can better reduce the uncertainty and ambiguity of face restoration mapping [54].

Blind Image Super-resolution.
While significant progress has been achieved for non-blind SISR [6, 25, 46], blind image super-resolution (BISR) remains very difficult. Bell-Kligler [1] introduced KernelGAN to estimate the blur kernel and then restore images based on it, which is prone to errors of estimated kernels. Following works jointly predicted the blur kernel and the HQ image. Gu [9] suggested an iterative correction scheme to achieve better results.
Those methods focused on blur kernel estimation, while images in the wild suffer from much more complex degradations other than blur. Cai [2] and Wei et al. [48] respectively established an SISR dataset with paired LQ-HQ images collected by zooming camera lens. However, the model can be hardly generalized to other photographing devices. Some works exploited DNNs to learn the degradation process with unpaired data [29, 7, 31, 49, 42, 43, 30, 26]. Lugmayr et al. [29] employed a cycle consistency loss to learn a distribution mapping network. Fritsche et al. [7] proposed DSGAN to generate LQ images. Luo [30] modeled the degradation as a random variable and learned its distribution. Li et al. [26] transferred the real-world degradations learned from face images to natural images. Although those methods have shown impressive results in some cases, their overall generalization performance in the wild remains limited.

§.§ Deep Generative Network

The generative adversarial networks (GANs) [8] have demonstrated much more powerful capability to synthesize HQ images than likelihood-based methods such as variational autoencoders (VAE) [22], autoregressive models [41] and flows [35].
Recently, diffusion probabilistic models (DPMs) have emerged and shown promising performance in tasks of image generation [15], image inpainting [36], image-to-image translation [37], text-to-image generation [33], text-to-video generation [14], etc. In particular, high quality image synthesis results were presented in [15], which was extended and improved in [32, 5]. Despite the great success, DPMs require hundreds of steps to simulate a Markov chain. Song et al. [39] proposed a denoising diffusion implicit model (DDIM) to accelerate the sampling speed. Very recently, DPMs have shown impressive results in text-to-image/video generation [33, 38, 14].

Illustration of the proposed HQ-LQ image pair synthesis process. Please refer to Sec. <ref> for details.

§ PROPOSED METHOD

§.§ Problem Formulation and Framework

Let's denote by $\mathcal{Y}$ the space of original HQ images, by $\mathcal{X}$ the space of synthetic LQ images, and by $\mathcal{R}$ the space of real-world LQ images. Generally speaking, the spaces $\mathcal{X}$ and $\mathcal{R}$ are partially overlapped because  synthetic LQ images can have similar statistics to the real-world LQ images. How large the overlapped subspace between $\mathcal{X}$ and $\mathcal{R}$ will be depends on the employed degradation model, denoted by $d$. For example, Zhang et al. [52] and Wang et al. [45] handcrafted sophisticated models to synthesize LQ data from their corresponding HQ images. Some works resorted to learning a mapping network from $\mathcal{Y}$ to $\mathcal{R}$ by unsupervised learning [29, 7, 31, 49, 30] or transferring degradations [26]. Nonetheless, the distributions gap between synthesized and real-world LQ images remains large.

Different from these previous efforts [29, 49, 30, 52, 45, 26], we aim to integrate the advantages of handcrafted degradation models and deep generative networks, more specifically, the denoising diffusion probabilistic models (DDPM) [15], to synthesize realistic training pairs. As shown in Fig. <ref>, given an HQ image $y\in\mathcal{Y}$, we first adopt a state-of-the-art degradation model [52, 45] $d$ to generate an initial LQ image $x\in\mathcal{X}$, , $x=d(y)$. Usually, $x$ shares some similar statistics and appearances to images in $\mathcal{R}$, while it still has certain distance to space $\mathcal{R}$. To make $x$ closer to $\mathcal{R}$, we train a DDPM, which defines a Markov chain of diffusion steps to add random noise to data and then reverses the diffusion process to construct desired data samples from noise, to further transform $x$ into $r\in\mathcal{R}$.

The diffusion process will gradually convert the data of a complex distribution into the data of a simpler prior distribution $\mathcal{P}$, , the isotropic Gaussian distribution. In other words, the distributions of images in $\mathcal{X}$ and $\mathcal{R}$ would become closer and closer during the diffusion process, and approach to the same isotropic Gaussian distribution finally. As depicted in Fig. <ref>, we denote by $\mathcal{X}_t$ and $\mathcal{R}_t$ respectively the latent distributions of $\mathcal{X}$ and $\mathcal{R}$ after diffusing $t$ steps. One can see that the domain gap between $\mathcal{X}_t$ and $\mathcal{R}_t$ becomes more and more indistinguishable.

Based on the above consideration, we first diffuse $x$ by $t$ steps to generate $z$, , $z=q(x)$, which is more likely to fall into space $\mathcal{R}_t$. We then reverse $z$ to obtain the final LQ image $r$, , $r=p(z)$, using DDPM pre-trained on real-world LQ images. The whole HQ to LQ image synthesis process is illustrated by the solid red line in Fig. <ref>. Due to the powerful distribution approximation capability of DDPM, it is anticipated that the final LQ image $r$ will fall into the space $\mathcal{R}$ of real-world LQ images. Consequently, aligned HQ-LQ image pairs, , $(y,r)$, can be obtained and used to train more robust image restoration models than those trained with $(y,x)$.

Sample face and natural scene images in our collected degraded image dataset (DID).

Left: illustration of the training process of DDPM (refer to Sec. <ref> for details). Right: illustration of the HQ-LQ image pair generation process by using the pre-trained DDPM (refer to Sec. <ref> for details).

§.§ DDPM Model Training

Degraded Image Dataset. Our method employs a DDPM to convert the HQ image into the LQ one, which is expected to meet the distribution of real-world LQ images. Therefore, a large-scale real-world LQ image dataset is needed to train the DDPM model first. Since most of the publicly available datasets are composed of HQ images [40, 20] or synthetic LQ ones [53], we build a large-scale degraded image dataset (DID) from scratch, which consists of $70,000$ degraded face images and $30,000$ degraded natural images. Some example images are shown in Fig. <ref>.

We first collected a large number of images from the Internet, where each image contains at least one face. Then the facial portions are automatically detected, cropped and aligned [4]. Every face image is resized to have a resolution of $512\times 512$. We carefully prune the data to exclude occasional statues, paintings, . In addition, we tend to not harvest too many faces in a single group of photos in order to diversify the degradation types. Finally, $70,000$ real-world degraded faces are collected.

To tackle more general image restoration problems such as blind image super-resolution, we crawled another set of $30,000$ real-world natural images with diverse sizes and contents from the Internet. Specifically, we use more than $200$ keywords, which cover a large scope of categories, including human, animal, landscape, indoor scene, ., to search via Google Images. Different from face images, we keep the crawled images unchanged.

Denoising Diffusion Probabilistic Models.
Given a data point from a real data distribution $x_0\sim q(x)$, it is interesting to learn a model distribution $p(x)$ that could approximate $q(x)$. To achieve this goal, DDPM [15] defines a forward diffusion process $q$, which produces a sequence of latents $\{x_1,x_2,...,x_T\}$ by adding Gaussian noise in $T$ steps. The step sizes are controlled by a variance schedule $\beta_t\in(0,1)$, where $t\in\{1,2,...,T\}$. There is:

\begin{align}
\end{align}

In particular, the above equation allows us to sample a latent at an arbitrary step $t$ directly, conditioned on the input $x_0$. Let $\alpha_t=1-\beta_t$ and $\bar{\alpha}_t=\prod_{i=1}^t\alpha_i$, we can write the marginal distribution as follows:

\begin{equation}
\end{equation}

With the help of reparameterization techniques [22], $x_t$ can be formulated as follows:

\begin{equation}
x_t=\sqrt{\bar{\alpha}_t}x_0+\sqrt{1-\bar{\alpha}_t}\epsilon, ~where~\epsilon\sim \mathcal{N}(\mathbf{0}, \mathbf{I}).
\label{eqn:x_02x_t}
\end{equation}

Since $\alpha_t\in(0,1)$, $x_T$ is equivalent to an isotropic Gaussian distribution when $T\to\infty$. Usually, $T$ is set to $1,000$.

If the exact reverse distribution $q(x_{t-1}|x_t)$ is known, we will be able to recreate the original samples from a Gaussian noise input. Unfortunately, $q(x_{t-1}|x_t)$ depends on the entire data distribution. We need to learn a model $p_\theta$ to approximate it. It is worth noting that if $\beta_t$ is small enough, $q(x_{t-1}|x_t)$ will also be Gaussian, ,

\begin{align}
\end{align}

Model Training.
With the collected DID, we train a DDPM to approximate its distribution. As illustrated on the left side of Fig. <ref>, during each optimization iteration, we diffuse a randomly sampled LQ image $x_0$ from DID for $t$ steps, where $t$ is randomly chosen from $\{0,1,,...,T-1\}$. According to Eqn. <ref>, we can easily compute $x_t$ with a randomly generated Gaussian noise $\epsilon$. In order to reverse the diffusing process, DDPM is designed to predict the noise $\epsilon$ on top of $x_t$ and $t$. To this end, we can parameterize $\epsilon_\theta(x_t,t)$ with a simplified loss function:

\begin{equation}
\end{equation}

Different from Ho [15], we use $L_1$ instead of $L_2$ loss here because $L_1$ loss is usually more resistant to overfitting and encourages less predictive features. Particularly, for the task of BISR, we randomly crop patches of resolution $256\times 256$ from the original image as input $x_0$.

§.§ HQ-LQ Image Pair Synthesis

With the DDPM model trained in Sec. <ref>, we are able to synthesize realistic LQ images from the HQ ones, as illustrated on the right side of Fig. <ref>. We first apply a handcrafted degradation model to the input HQ image to obtain an initial LQ image, and then iteratively add proper Gaussian noises to the initially synthesized LQ image for $t$ steps, obtaining a noisy LQ image $x_t$. Note that $x_t$ can be directly calculated according to Eqn. <ref> without time-consuming iterations. This is because mathematically, the merge of two Gaussian noises will result in another Gaussian noise.

We then employ the pre-trained DDPM to predict the Gaussian noise $\epsilon$ added to $x_t$. With the predicted noise, we obtain a predicted $x_0$, which incorporates $x_t$ to produce $x_{t-1}$. We iteratively perform the above operations for $t$ times and finally obtain a synthesized LQ image. Generally speaking, the quality of the output LQ image depends on the setting of $t$. Since the DDPM model is trained to generate samples that meet the target distribution of real-world LQ images, our synthesized LQ images are more realistic than the manually synthesized ones, as we discussed in Sec. <ref>.

In the aforementioned synthesizing process, one can see that the handcrafted degradation model $d$ and the diffusion step $t$ are important factors to the final results. In our experiments, we adopt the degradation model proposed by Wang [45] and randomly sample $t$ from $[0, 500]$ and $[0, 250]$ for BFR and BISR, respectively.

xlabel=Diffusion step $t$,
xmin=0, xmax=950,
ymin=20, ymax=110,
legend pos=north east,
grid style=dashed,

ticklabel style=font=,
legend style=font=

xlabel=Diffusion step $t$,
ylabel=PSNR (dB),
xmin=0, xmax=950,
ymin=10, ymax=35,
legend pos=north east,
grid style=dashed,

ticklabel style=font=,
legend style=font=

Top: FID versus diffusion step $t$. Bottom: PSNR versus diffusion step $t$.

(a) HQ image; (b) initial LQ image; (c)-(e) synthesized LQ images by our method when $t=\{250, 500, 750\}$.

§ EXPERIMENTS

§.§ Experimental Setup

To comprehensively evaluate the effectiveness of the proposed HQ-LQ training pair synthesis method, we perform experiments on both synthetic data (for quantitative evaluation) and real-world data (for qualitative evaluation). For each of the BFR and BISR tasks, we employ several open-sourced models and re-train them on our training pairs, and then apply them to the synthetic test data for objective evaluation. On the real-world test data, we invite seventeen human subjects to perform user-study and compare the visual quality of the restored HQ images.

Test data.
For the task of BFR, we use the first $5,000$ HQ face images in the CelebA-HQ dataset [19] to simulate LQ images, while for the task of BISR, we use the $100$ HQ images in the DIV2K validation dataset  [40] to simulate LQ images.
Apart from the synthetic test data, we also select $100$ real-world LQ face images and $100$ LQ natural images (excluded from DDPM pre-training) from the DID to qualitatively evaluate the performance of different BFR and BISR models in the wild.

Evaluation metrics. As for the quantitative evaluation, the Fréchet Inception Distances (FID) [13], the Learned Perceptual Image Patch Similarity (LPIPS) [53], the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM) [47] indices are used to measure the distance between the model output and ground-truth.

Training details.
When training a DDPM, we adopt the Adam optimizer [21] with a batch size of $16$. The learning rate is fixed as $8\times 10^{-5}$. In particular, we adopt the exponential moving average (EMA) method with a decay coefficient of $0.995$ during optimization to ensure the training stability. The model is updated for $700K$ iterations.

In regarding to the training of downstreaming task (, BFR and BISR) models, we simply keep the training configurations in their original papers [3, 50, 44, 51, 45] unchanged, except when we train these models by using the HQ-LQ training pairs synthesized by our method.

The FID, LPIPS, PSNR and SSIM indices of original BFR models and their re-trained counterparts (denoted by “+") on the synthesized test data.

Metric     PSFRGAN [3]     PSFRGAN+     HiFaceGAN [50]     HiFaceGAN+     GFPGAN [44]     GFPGAN+     GPEN [51]     GPEN+

FID$\downarrow$     13.7655     14.9023     17.6288     12.4294     10.1204     10.1057     8.0718     6.8306

LPIPS$\downarrow$     0.2720     0.2717     0.2783     0.2588     0.2252     0.2161     0.2618     0.2299

PSNR$\uparrow$     25.5152     25.7500     26.3227     25.1443     25.1169     25.7269     24.9363     26.2205

SSIM$\uparrow$     0.6496     0.6633     0.6505     0.6802     0.6739     0.6813     0.6276     0.6775

Blind face restoration results of original BFR models and their re-trained counterparts (denoted by “+") by our synthesized training pairs.

§.§ Selections of Initial Handcrafted Degradation and Diffusion Step

In order to find out the effects of handcrafted degradation model $d$ and diffusion step $t$ on the synthesized HQ-LQ pairs, we conduct ablation studies by applying four representative handcrafted degradation models, , bicubic downsampling (denoted by Bicubic), the classical degradation model (denoted by Classical, see [23, 51]), the one proposed by Zhang [52] and the one proposed by Wang [45], to our method.

We use the first $5,000$ face images in FFHQ [20] as the HQ inputs, and generate four groups of LQ counterparts by our method with the four handcrafted degradation models. We calculate the FID [13] between each group of generated LQ images and the real degraded faces in our collected DID to evaluate the distribution distances.

Handcrafted Degradation Model.
The curves of FID versus diffusion step $t$ are plotted on the top of Fig. <ref>. We see that when $t=0$ (, without applying the pre-trained DDPM), Bicubic has the worst FID, while Zhang [52] and Wang [45] are among the best. This indicates that the degradation models proposed by Zhang et al. [52] and Wang et al. [45] can better simulate real-world degraded data than bicubic downsampling and the classical degradation model. It is worth mentioning that all the four models can produce realistic LQ images (, low FID value) with a large enough step number $t$; however, a good degradation model should be able to generate LQ images that cover a large scope of LQ space. The models developed by Zhang [52] and Wang [45] demonstrate their superiority to bicubic downsampling and the classical degradation model. We choose the model proposed by Wang [45] due to its efficient implementation.

Diffusion Step. A qualified HQ-LQ image pair should share the same content and structure in general. We calculate the PSNR of generated HQ-LQ pairs and plot the curve on the bottom of Fig. <ref>.
One can see that all curves converge with the increase of diffusion step $t$. This is because the distributions of LQ images and real-world degraded images are getting closer during the diffusion process, and they approach to the same isotropic Gaussian distribution. In other words, with a large enough diffusion step $t$, we can produce LQ images with lower FID values. However, a large $t$ will destroy the structure of synthesized LQs compared with their HQ counterparts. As shown in Fig. <ref>, the PSNR decreases when $t$ grows. By our experiments, when the PSNR of an HQ-LQ pair is smaller than $24$, the general structure of them will become inconsistent. Fig. <ref> shows some synthesized LQ images by our method with different steps $t$. One can see that the structures of synthesized LQ images become inconsistent with the HQ ones when $t>500$ / $t>250$ for face/natural images.
We therefore randomly sample $t$ from $[0, 500]$ for the training pair synthesis of BFR, and sample $t$ from $[0, 250]$ for BISR.

The FID, LPIPS, PSNR and SSIM indices of different BISR models on the synthesized test data. RRDB+ refer to the RRDB-based model trained on our training pairs.

Metric     ESRGAN-FS [7]     DASR [49]     Wang [43]     PDM-SRGAN [30]     F2N-ESRGAN [26]     RealSR [17]     BSRGAN [52]     Real-ESRGAN [45]     RRDB+

FID$\downarrow$     106.1637     100.9717     110.1927     102.8514     100.3145     115.2377     93.0567     91.9363     89.2679

LPIPS$\downarrow$     0.5168     0.4789     0.5232     0.4621     0.4349     0.6655     0.4287     0.4223     0.4248

PSNR$\uparrow$     20.9974     21.1608     20.2800     21.0749     21.132     21.0939     21.8632     21.3071     22.1809

SSIM$\uparrow$     0.5194     0.5099     0.4927     0.5226     0.5417     0.4629     0.5439     0.5389     0.5599

Blind image super-resolution results on synthesized degraded images. All models use the RRDB [46] backbone. The performance differences are mainly caused by different training pairs.

Blind face restoration results on real degraded faces in the wild.

User preferences of original BFR models versus the re-trained models on our synthesized training pairs.

Blind image super-resolution results on degraded images in the wild.

User study results of different BISR methods.

§.§ Experiments on Synthetic Test Images

Blind Face Restoration.
We validate the effectiveness of our approach by using four representative BFR models, , PSFRGAN [3], HiFaceGAN [50], GFPGAN [44] and GPEN [51], whose training codes are publicly available so that we can re-train them by using the HQ-LQ image pairs synthesized by our method with their original training settings. We denote by PSFRGAN+, HiFaceGAN+, GFPGAN+ and GPEN+ the re-trained models on our training data. By applying those original and re-trained models on the synthetic LQ face images, we can calculate the FID, LPIPS, PSNR and SSIM indices by comparing the reconstructed HQ images with the ground-truth HQ images. The results are listed in Table <ref>. One can see that for each BFR method, the model trained on our synthesized training pairs can achieve better FID/LPIPS/PSNR/SSIM indices than its original counterpart in most cases.

Fig. <ref> compares the BFR results on some degraded face images by original BFR models and their re-trained counterparts. It can be seen that GFPGAN [44] and GPEN [51] can produce reasonable face reconstructions while failing to tackle faces that are severely compressed or with large viewpoint variation. After re-training GFPGAN and GPEN with HQ-LQ training pairs synthesized by the proposed method, GFPGAN+ and GPEN+ succeed in restoring clearer faces with more vivid details in comparison with their original counterparts.

More visual comparison results can be found in Sec. <ref>.

Blind Image Super-Resolution.
BISR aims to reconstruct HQ images with perceptually realistic details from the input LQ image with unknown degradation. We employ two categories of state-of-the-art algorithms in this experiment. The first category is GAN-based methods trained on unpaired real data, including ESRGAN-FS [7], DASR [49], Wang [43], PDM-SRGAN [30], F2N-ESRGAN [26]. All methods are re-trained on our DID. The second category is models trained with handcrafted HQ-LQ image pairs, including RealSR [17], BSRGAN [52], Real-ESRGAN [45]. All these three methods use RRDB [46] as the backbone with similar losses. The key difference among them lies in the training data. Therefore, we re-train a RRDB-based model [46] by using our synthesized training pairs with the training settings of Real-ESRGAN [45], resulting in the RRDB+ model.  The quantitative evaluation results on the test data are presented in Table <ref>. One can see that RRDB+ outperforms the competing methods in most of the metrics. In particular, compared with the original Real-ESRGAN model, our re-trained  RRDB+ achieves similar LPIPS index but significantly better FID, PSNR and SSIM indices.

Fig. <ref> presents the visual comparison of competing BISR methods on several synthesized LQ natural images. One can see that our method can better reconstruct realistic details and preserve the general structures, demonstrating the effectiveness of our HQ-LQ training pair synthesis methods. Comparing to Real-ESRGAN [45], RRDB+ has clear advantages in inhibiting artifacts.

§.§ Experiments on Images in the Wild

Since our method targets at synthesizing realistic image restoration training pairs, it is necessary to conduct experiments on images in the wild. We select 100 face and 100 natural images from DID for evaluation.

Blind Face Restoration.
We apply the same four BFR methods as in Sec. <ref> to the 100 real-world LQ face images. Fig. <ref> shows the BFR results on two images. Due to the limit of space, we only show the results of GFPGAN [44], GPEN [51] and GPEN+ here. More results can be found in Sec. <ref>.
As can be seen, GPEN+ can better handle real-world degradations and produce more realistic results than GPEN and GFPGAN, mainly due to the more realistic image training pairs.

To better evaluate the advantage of the proposed method, we conduct a user study as subjective assessment. For each of the four groups of BFR methods, , PSFRGAN vs. PSFRGAN+, HiFaceGAN vs. HiFaceGAN+, GFPGAN vs. GFPGAN+, GPEN vs. GPEN+, the BFR results are presented in pairs to $17$ volunteers in a random order. The volunteers are asked to pick the better BFR image from each pair according to their perceptual quality. The statistics are presented in Fig. <ref>. One can see that in each group, the model trained with our synthesized data is more preferred.

Blind Image Super-Resolution.
Fig. <ref> shows the BISR results of ESRGAN-FS [7], DASR [49], BSRGAN [52], Real-ESRGAN [45], PDM-SRGAN [30], F2N-ESRGAN [26], and our RRDB+. It can be seen that the competing methods yield dirty outputs, , the second and third columns of Fig. <ref>, or tend to produce over-smoothed results, , the frog in the first row of Fig. <ref>, or fail to reconstruct photo-realistic details, , the eye in the second row of Fig. <ref>. With the help of the realistic HQ-LQ training pairs generated by our method, RRDB+ is capable of better handling complex degradations in the wild. More visual results can be found in Sec. <ref>.

Since the commonly used quantitative metrics do not correlate well with human visual perception to image quality, we conduct a user study as subjective assessment. The BISR results of BSRGAN [52], Real-ESRGAN [45], PDM-SRGAN [30], F2N-ESRGAN [26] and RRDB+ on $100$ natural images from DID are presented to $17$ volunteers in random order. The volunteers are asked to rank the five BISR outputs of each input image according to their perceptual quality. As presented in Fig. <ref>, our method RRDB+ receives the most rank-1 votes and the least rank-5 votes.

§ CONCLUSION AND DISCUSSION

We proposed, for the first time to our best knowledge, to train a DDPM to synthesize realistic image restoration training pairs. With a collected LQ image dataset, a DDPM was first trained to approximate its distribution. The pre-trained DDPM was then employed to convert the initially degraded image from its HQ counterpart into the desired LQ image, which fell into the distribution of real-world LQ images. With the synthesized realistic HQ-LQ image pairs, we re-trained the many state-of-the-art BFR and BISR models, and the re-trained models demonstrated much better realistic image restoration performance than their original counterparts both quantitatively and qualitatively.

It should be noted that in our experiments, we collected a large scale degraded face dataset and a natural image dataset, respectively, as the target distributions to train the DDPM and synthesize HQ-LQ training pairs. In practice, the users can build their own LQ image dataset according to their needs, train the corresponding DDPM models and synthesize training pairs for different image restoration tasks.

[1]
Sefi Bell-Kligler, Assaf Shocher, and Michal Irani.
Blind super-resolution kernel estimation using an internal-gan.
In NeurIPS, pages 284–293, 2019.

[2]
Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang.
Toward real-world single image super-resolution: A new benchmark and
a new model.
In ICCV, 2019.

[3]
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and KKY Wong.
Progressive semantic-aware style transformation for blind face

[4]
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos
Retinaface: Single-shot multi-level face localisation in the wild.
In CVPR, 2020.

[5]
Prafulla Dhariwal and Alex Nichol.
Diffusion models beat gans on image synthesis.
In Arxiv, 2021.

[6]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.
Image super-resolution using deep convolutional networks.
In ECCV, 2014.

[7]
Manuel Fritsche, Shuhang Gu, and Radu Timofte.
Frequency separation for real-world super-resolutio.

[8]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial networks.
In NeurIPS, pages 2672–2680, 2014.

[9]
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong.
Blind super-resolution with iterative kernel correction.
In CVPR, 2019.

[10]
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.
Toward convolutional blind denoising of real photographs.
In CVPR, 2019.

[11]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.
Mask r-cnn.
In ICCV, 2017.

[12]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In CVPR, pages 770––778, 2016.

[13]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
Gans trained by a two time-scale update rule converge to a local nash
In NeurIPS, 2017.

[14]
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey
Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet,
and Tim Salimans.
Imagen video: High definition video generation with diffusion models.
In Arxiv, 2022.

[15]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In NeurIPS, pages 6840–6851, 2020.

[16]
Xiaobin Hu, Wenqi Ren, John Lamaster, Xiaochun Cao, Xiaoming Li, Zechao Li,
Bjoern Menze, and Wei Liu.
Face super-resolution guided by 3d facial priors.
In ECCV, 2020.

[17]
Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang.
Real-world super-resolution via kernel estimation and noise
In CVPRW, 2020.

[18]
Jiaxi Jiang, Kai Zhang, and Radu Timofte.
Towards flexible blind jpeg artifacts removal.
In ICCV, pages 4997–5006, 2021.

[19]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of gans for improved quality, stability, and

[20]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and
Timo Aila.
Analyzing and improving the image quality of StyleGAN.
In CVPR, 2020.

[21]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In Arxiv, 2015.

[22]
Diederik P. Kingma and Max Welling.
Auto-encoding variational bayes.
In Arxiv, 2014.

[23]
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.
Deep laplacian pyramid networks for fast and accurate
In CVPR, 2017.

[24]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.
Deep learning.
Nature, 7553(521):436–444, 2015.

[25]
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham,
Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang,
and Wenzhe Shi.
Photo-realistic single image super-resolution using a generative
adversarial network.
In CVPR, 2017.

[26]
Xiaoming Li, Chaofeng Chen, Xianhui Lin, Wangmeng Zuo, and Lei Zhang.
From face to natural image: Learning real degradation for blind image
In ECCV, 2022.

[27]
Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei
Blind face restoration via deep multi-scale component dictionaries.
In ECCV, 2020.

[28]
Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang.
Learning warped guidance for blind face restoration.
In ECCV, 2018.

[29]
Andreas Lugmayr, Martin Danelljan, and Radu Timofte.
Unsupervised learning for real-world super-resolution.

[30]
Zhengxiong Luo, Yan Huang, , Shang Li, Liang Wang, and Tieniu Tan.
Learning the degradation distribution for blind image

[31]
Shunta Maeda.
Unpaired image super-resolution using pseudo-supervision.

[32]
Alex Nichol and Prafulla Dhariwal.
Improved denoising diffusion probabilistic models.
In Arxiv, 2021.

[33]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
In Arxiv, 2022.

[34]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region proposal
In NeurIPS, 2015.

[35]
Danilo Jimenez Rezende and Shakir Mohamed.
Variational inference with normalizing flows.

[36]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
High-resolution image synthesis with latent diffusion models.
In CVPR, 2022.

[37]
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim
Salimans, David Fleet, and Mohammad Norouzi.
Palette: Image-to-image diffusion models.
In Arxiv, 2022.

[38]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily
Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi,
Rapha Gontijo Lopes, Jonathan Ho Tim Salimans, David J Fleet, and Mohammad
Photorealistic text-to-image diffusion models with deep language
In Arxiv, 2022.

[39]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
In ICLR, 2021.

[40]
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang.
Ntire 2017 challenge on single image super-resolution: Methods and
In CVPRW, pages 114–125, 2017.

[41]
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.
Pixel recurrent neural networks.

[42]
Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An,
and Yulan Guo.
Unsupervised degradation representation learning for blind
In CVPR, 2021.

[43]
Wei Wang, Haochen Zhang, Zehuan Yuan, and Changhu Wang.
Unsupervised real-world super-resolution: A domain adaptation

[44]
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan.
Towards real-world blind face restoration with generative facial
In CVPR, 2021.

[45]
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan.
Real-esrgan: Training real-world blind super-resolution with pure
synthetic data.

[46]
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and
Chen Change Loy.
Esrgan: Enhanced super-resolution generative adversarial networks.

[47]
Zhou Wang, Alan Conrad Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.
Image quality assessment: from error visibility to structural
IEEE TIP, 13:600–612, 2004.

[48]
Pengxu Wei, Ziwei Xie, Hannan Lu, ZongYuan Zhan, Qixiang Ye, Wangmeng Zuo, and
Liang Lin.
Component divide-and-conquer for real-world image super-resolution.
In ECCV, 2020.

[49]
Yunxuan Wei, Shuhang Gu, Yawei Li, Radu Timofte, Longcun Jin, and Hengjie Song.
Unsupervised real-world image super resolution via domain-distance
aware training.

[50]
Lingbo Yang, C. Liu, P. Wang, Shanshe Wang, P. Ren, Siwei Ma, and W. Gao.
Hifacegan: Face renovation via collaborative suppression and
ACM MM, 2020.

[51]
Tao Yang, Peiran Ren, Xuansong Xie, , and Lei Zhang.
Gan prior embedded network for blind face restoration in the wild.
In CVPR, 2021.

[52]
Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte.
Designing a practical degradation model for deep blind image
In ICCV, pages 4791–4800, 2021.

[53]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual
In CVPR, 2018.

[54]
Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy.
Towards robust blind face restoration with codebook lookup
In NeurIPS, 2022.

§ APPENDICES

§.§ More BFR Results on Synthetic Faces

This section shows more visual results of competing BFR methods on synthetic face images. As in the main paper, we compare the BFR results on degraded face images by the original BFR models, , GFPGAN [44] and GPEN [51], and their re-trained counterparts on our training pairs, , GFPGAN+ and GPEN+. The visual comparisons are presented in Fig. <ref>. One can see that GFPGAN+ and GPEN+ generate superior results with finer details to their original counterparts.

§.§ More BFR Results on Faces in the Wild

This section shows more visual results of competing BFR methods on face images in the wild. As in the main paper, we compare our method with PSFRGAN [3], HiFaceGAN [50], GFPGAN [44] and GPEN [51]. The visual comparisons are presented in Fig. <ref>. It can be seen that our method can produce more realistic results.

§.§ More BISR Results on Images in the Wild

In this section, we show more visual comparison between our method with state-of-the-art BISR methods, including DASR [49], PDM-SRGAN [30], BSRGAN [52] and Real-ESRGAN [45], in Fig. <ref>, from which we can see the superior performance of our method on BISR tasks.

§.§ Comparison between GAN-based methods and our diffusion method on LQ image synthesis

In order to demonstrate the superior performance of our proposed diffusion-based method in synthesizing LQ images to existing GAN-based methods [7, 49, 30, 26], we compute the FID values to evaluate the distribution distances between synthesized LQs and real LQs in DID. The quantitative results are presented in Table <ref>. It can be seen that our diffusion-based method outperforms existing GAN-based methods significantly. This is because GAN-based methods learn a direct mapping from HQs to LQs, which is hard to train due to the large distribution gap and unstable adversarial training, while our method employs an initial LQ image as the starting point, and use a pre-trained diffusion model to stably generate the final LQ image. During the reverse process of DDPM, the images are naturally constrained in the LQ space. In contrast, GAN-based methods only constrain the final outputs, and hence they are less stable in LQ image synthesis.

Comparison (FID) of different methods in synthesizing LQ images.

Method     ESRGAN-FS [7]     DASR [49]     PDM-SRGAN [30]     F2N-ESRGAN [26]     Ours

FID$\downarrow$     79.5057     77.7190     68.8146     70.4900     49.2831

Blind face restoration results of original BFR models and their re-trained counterparts (denoted by “+") by our synthesized training pairs.

Blind image super-resolution results on real degraded faces in the wild.

Blind image super-resolution results on degraded images in the wild.