# Superresolution Reconstruction of Single Image for Latent features

Xin Wang,Jing-Ke Yan,Jing-Ye Cai,Jian-Hua Deng,Qin Qin,Qin Wang,Heng Xiao,Yao
Cheng,Peng-Fei Ye This work is supported by Guangxi Science and Technology
Major Project (AA19254016), Beihai city science and technology planning
project (202082033), Beihai city science and technology planning project
(202082023), Guang xi graduate student innovation project(YCSW2021174). Xin
Wang is 1st author, Jing-ke Yan and Jing-Ye Cai are corresponding authors. Xin
Wang is With the School of information and software engineering ,University of
Electronic Science and Technology of China, Chengdu 610000, China;Guilin
University of Electronic Technology, School of Computer Engineering, Beihai,
536000, China;Guilin University of Electronic Technology, School of Computer
Science and Information Security, Guilin, 541004, China(email:
<EMAIL_ADDRESS>Jing-Ke Yan is With the School of Marine Engineering,Guilin
University of Electronic Technology.No. 1 Jinji Road, GuiLin, 541000,
China(email: 592499985@qq.com). Jing-Ye Cai is With the School of information
and software engineering ,University of Electronic Science and Technology of
China, Chengdu 610000, China(email<EMAIL_ADDRESS>
###### Abstract

In recent years, Deep Learning has shown good results in the Single Image
Superresolution Reconstruction (SISR) task, thus becoming the most widely used
methods in this field. The SISR task is a typical problem solving task where
there may be an infinite number of High-resolution (HR) images corresponding
to a single Low-resolution (LR) image. Therefore, it is often challenging to
meet the requirements of high-quality sampling, fast sampling, and diversity
of details and texture after Sampling simultaneously in a SISR task.It leads
to model collapse, lack of details and texture features after Sampling, and
too long Sampling time in HR image reconstruction methods. This paper proposes
Denoising Diffusion Probabilistic model for Latent features (LDDPM) to solve
these problems. Firstly, a Conditional Encoder is designed to effectively
encode LR images, thereby reducing the solution space of reconstructed images
to improve the performance of reconstructed images. Then, the Normalized Flow
and Multi-modal adversarial training are used to model the denoising
distribution with complex Multi-modal distribution so that the Generative
Modeling ability of the model can be improved with a small number of Sampling
steps. Experimental results on mainstream datasets demonstrate that our
proposed model reconstructs more realistic HR images and obtains better PSNR
and SSIM performance compared to existing SISR tasks, thus providing a new
idea for SISR tasks.

###### Index Terms:

Image Superresolution Reconstruction, Denoising Diffusion Probabilistic model,
Normalized Flow, Adversarial Neural Network,Variational Auto-Encoder.

##  I Introduction

Single-image super-resolution reconstruction (SISR) tasks are critical in
research areas such as Computer Vision [1] and Image Processing [2][3]. The
SISR task is to reconstruct the corresponding HR image using an LR image.
Because LR images lose a lot of details and texture features in image
degradation, the reconstructed HR images must have rich image details and
clear textures. However, an LR image may correspond to infinitely many HR
images, and various degraded LR images can also restore a single HR image.
Hence, the SISR task is typical for solving the uncertainty problem. In the
SISR task, Researchers have successively proposed various traditional methods,
such as Iterative Back Projection [4], Convex Set Projection [5] and Sparse
Representation [6], Etc. However, the traditional methods usually explicitly
estimate the fuzzy kernel and then reconstruct the HR image. Therefore, the
traditional methods will lead to the error of the estimated fuzzy kernel, so
the HR image reconstruction effect is not ideal.

SISR task can also be regarded as a typical generation task. The so-called
generation task is to effectively fit the probability distribution of the data
through the generator to make the generated probability distribution as close
as possible to the real data distribution. Deep learning-based methods in this
task can be divided into five categories: CNN-based methods, Generative
Adversarial Network (GAN) based methods [7], Flow based methods [8],
Variational Auto-Encoder (VAE) based methods [9] and Denoising Diffusion
Probabilistic model (DDPM) based methods [10]. However, these generative
models face three major difficulties in the SISR task: high quality Sampling,
fast sampling and diversity of details after Sampling. The method based on CNN
can fit any Function, but cannot fit any probability distribution. Therefore,
the method based on CNN alone is difficult to solve the problems of unreal
perception and artifacts in the reconstructed results. Gan-based methods are
also commonly used in SISR. They use Perceptual Loss and Adversarial Loss to
reconstruct images. Although they can provide fast sampling, there are
problems such as pattern collapse and training instability. The Flow based
methods can improve the diversity of the generated images by using the Log-
Likelihood Function to infer the latent variables accurately, but the
generated images are too smooth. The VAE based method not only generates more
diverse data using additional conditions but also provides relatively fast
sampling. However, the VAE based method does not sample with high quality, and
there is a Loss of detail and texture in the HR images.

Recently, DDPM has achieved good results in Image Synthesis [11] and Speech
Synthesis [12]. DDPM uses the Markov chain to transform latent variables in
Gaussian distribution into data in complex distribution, thus solving the ”one
to many” problem in the SISR task and improving the quality of reconstructed
data. However, SISR tasks are different from other generation tasks. Applying
DDPM to SISR tasks requires solving the following problems:

  1. 1.

The inverse diffusion process of DDPM on the SISR task requires a complex
probability distribution to model the denoising distribution, so DDPM requires
thousands of evaluation steps in the forward diffusion process to sample a
sample feature. If DDPM uses a small number of sampling steps, the generated
images after DDPM sampling are not of high quality.

  2. 2.

DDPM is based on unconditional or simple conditions for model input. At the
same time, SISR tasks often need to fully use LR images as conditions for
model input to constrain the solution space of HR images.

Therefore, this paper proposes a novel Denoising Diffusion Probabilistic model
for Latent features (LDDPM) to solve the problem faced by DDPM in SISR task:

  1. 1.

To ensure high-quality sampling by DDPM with a small number of sampling steps,
this paper designs a Multimodal distribution based on GAN and Normalized Flow
to model HR images, which enables LDDPM to focus on reconstructing high-
frequency details of HR images with fewer diffusion steps.

  2. 2.

To ensure high-quality sampling by DDPM with a small number of sampling steps,
this paper designs a Multimodal distribution based on GAN and normalized flow
to model HR images, which enables LDDPM to focus on reconstructing high-
frequency details of HR images with fewer diffusion steps.

The model in this paper has the following advantages:

Fast and high quality sampling: We have reconstructed HR images by Markov
chains and complex multimodal distribution modeling, which enables fast model
sampling while reducing the negative impact of model collapse on modeled HR
images, thus producing complex and diverse HR images with high quality.

Stable style and content consistency: Although the probability distribution of
HR images is difficult to predict, this paper limits the effect of prediction
randomness caused by the maximally variable lower bound in DDPM by designing a
new conditional encoder, so that the model is trained stably and can generate
images with the same style and content as the original HR images.

The LDDPM proposed in this paper is experimentally proved on many datasets.
The experimental results show that the proposed model outperforms most of the
methods in SISR tasks on multiple datasets. In addition, the code of LDDPM
will be open source soon: https://github.com/yanjingke/Image-Super-
ResolutionLDDPM.

##  II Related Work

In this section, we discuss the Convolutional Neural Network (CNN) based
methods, Generative Adversarial Network (GAN) based methods, Flow based
methods, Variational Auto-Encoder (VAE) based methods, and Denoising Diffusion
Probabilistic model (DDPM) based methods in the generative model as shown in
Figure 1.

###  II-A Single Image Superresolution Based on Traditional generation model

CNN based methods:Due to the rapid development of Deep Learning, many Deep
Learning-based methods have been proposed in SISR. Most of these methods are
based on Convolutional Neural networks (CNN), which use an end-to-end to learn
the probability mapping relationship between LR and HR images. For example,
Zhang et al [13] found that most CNN based method not only did not fully
explore the contextual information of LR images during feature extraction but
also paid little attention to the reconstruction steps of the final HR images,
so they proposed a two-stage single image reconstruction method (TSAN) based
on an Attention Mechanism, thus achieving accurate HR image reconstruction
using a coarse-to-fine approach. However, TSAN rarely explores the feature
correlation between layers, which reduces the ability of CNN to learn
probabilistic mapping relationships between LR images and HR images.Dai et
al.[14] capture long-distance spatial contextual information between features
by using a Second-order feature statistics module and a Non-locally augmented
residual module so that the model can learn abstract probabilistic mapping
representations. Although the Second-order feature statistics module can
effectively extract features with rich information in each layer, the module
is processing the features of each Convolutional layer independently, thus
ignoring the correlation of features between different layers. Therefore, Niu
et al. [15] proposed a Holistic Attention Network (HAN) based on CNN, which
not only considers the correlation between layers to adaptively emphasize the
features between layers but also can learn the confidence of each channel so
that the model can carry out complex probability distribution mapping. In the
SSIR task, there are some similar Patches in the image. The similar Patches
can provide information to each other, which can help CNN learn the
probabilistic mapping relationship between the LR image and HR image.
Therefore, Zhou et al. [16] divided the LR image into multiple Patches and
used each Patch to search for K nearest adjacent features to construct a
cross-scale map matrix dynamically. The probability distribution in the HR
image can be transferred to the query Patch of the LR image in the above way,
thus helping to recover more complex Detail and texture features. However, if
only the CNN-based methods are used to generate the HR image, the perception
is not real, and there are artifacts. A breakthrough solution to this problem
is to use GAN-based methods.

GAN based methods:GAN obtains Content Loss and Discrimination Loss through the
Generator and Discriminator to make the generated image distribution as close
as possible to the real image distribution. For example, Wang et al. [17]
found that if the goal of Perception Loss is to minimize the error in pixel
space rather than the error in feature space, the generated HR image tends to
output excessively smooth results, thus missing enough High-Frequency details.
They then proposed the SRGAN network, which uses activation features to
improve Perceptual Loss, thus providing a stronger supervisory signal for
luminance consistency and texture recovery. However, SRGAN has a limited
ability to reconstruct Spectral–Spatial invariance, which may lead to
Spectral–Spatial distortion in the generated HR image, especially when the
image magnification factor is enormous. Therefore, Shi et al. [7] mapped the
generated Spectral–Spatial features from the image space to the latent space,
thereby generating a coupling component to regularize the generated samples.
Although the research results of GAN are applied to the SISR task, GAN is
based on data-driven, which leads to fundamental limitations of GAN in
reconstructing high-frequency information features of unknown images in the
testing phase. Therefore, Liu et al. [18] seamlessly integrated the advantages
of CNN-based methods into GAN-based methods based on the fact that CNN-based
methods have advantages in adaptive aspects, thus using the detailed features
captured by CNNs as prior knowledge to help GANs generate more realistic
details. The above methods reconstruct HR images with fewer artifacts and more
realistic perceptions. Still, they are prone to model collapse. They cannot
effectively solve the problems of ”one-to-many” uncertainty and the inability
to determine the distribution of true samples in the hidden space.

VAE based methods:The VAE based methods and Flow based methods are also
generative models. The VAE based approach first maps the input to the hidden
space for probability density estimation. Then VAE assumes the prior
distribution as a standard Gaussian distribution and trains a probability
decoder to achieve the mapping from the hidden space to the real data
distribution. For example, Gatopoulos et al. [19], according to the feature
that neurons in human vision can continuously add new information to enhance
existing signals after adapting to light, used the image downsampling
representation as a random variable based on the VAE and continuously added
random variables into the model for training. However, Gatopoulos et al.’s
method tended to generate blurred images, so Liu et al.[20] added a
Conditional sampling mechanism to reduce the potential subspace for
reconstruction. Although Liu’s method can reconstruct some HR images with
simple backgrounds, they use Mean Square Error(MSE) for model optimization,
which tends to cause blurring of the edges of some complex background images.
Liu et al. [9] considered searching similar style images from reference images
to guide the reconstruction of HR images. They use Conditional Variational
Auto-Encoder (CVAE) to compress various reference images into a compact hidden
space to learn the explicit distribution and sample corresponding style
features from this distribution as conditions or priors, which are used to
solve the problem of complex background edge blurring in reconstruction.

Flow based methods:Flow based methods use bijective Functions to learn the
posterior distribution from the prior distribution through a series of
reversible transformation Functions to generate HR images based on the
posterior distribution. For example, Liang et al.[8] found that the
Normalizing Flow can predict detail-rich HR images from LR images using
Downsampling and Upsampling by a joint modeling method. They then modeled the
LR image and the remaining high-frequency components so that the model uses
the bijective mapping between the HR and LR images for learning the lost high-
frequency information.Xiang et al. [21] used a Flow-based model for intra-flow
feature extraction, inter-flow dependency extraction, and joint feature
learning, which resulted in the better reconstruction of HR images. However,
the Flow-based model of the above methods only used a small number of
Convolutional Layers, which led to the limited perceptual field of the model.
Therefore, Jo et al. [22] stack more Convolutional Layers through affine
coupling to expand the receptive field and obtain more vital feature
expression ability. The Flow based and VAE based methods can not only
effectively learn the distribution of samples in hidden space but also solve
the ”one-to-many” uncertainty problem. However, the detailed features of HR
images generated by the above methods are too smooth and require high training
time.

###  II-B Single Image Superresolution Based on Denoising Diffusion
Probabilistic model

The recent Denoising Diffusion Probabilistic model (DDPM) was used for the
SISR task. The DDPM is composed of two parametric Markov chains (forward and
inverse chains) and uses variational inference to generate samples in finite
time that are consistent with the original data distribution. The forward
chain functions as a perturbation of the data by gradually adding Gaussian
noise to the data according to a predesigned noise schedule until the
distribution of the data converges to the prior distribution (standard
Gaussian distribution). The reverse chain learns to gradually recover the
original data distribution by iterating from a given prior distribution and
using a parameterized Gaussian transformation kernel. Thus DDPM is a highly
flexible and easy-to-compute generative model that not only effectively avoids
the model collapse encountered by GAN but also generates High-quality images.
For example, Li et al. [10] designed the SRDiff model based on DDPM, which
gradually transformed Gaussian noise into HR images through Markov chains with
residuals. Saharia et al. [23] designed based on a repeat detailed image
Super-Resolution model (SR3). Firstly, white Gaussian noise is added to the
image, and then various noisy images are used to train the UNet model to
refine the noise output iteratively. Ryu et al. [24] proposed a Pyramid
Denoising Diffusion Probabilistic model. In addition to DDPM, this model uses
the Position Embedding training score Function to make LR image gradually
generate HR image. Although the above DDPM-based models show strong
performance on different super-resolution datasets, DDPM’s high-quality
sampling, diversity of samples, and small computational overhead on SISR tasks
are still worth investigating.

Figure 1: Mainstream generative models

##  III Methods

This section introduces the proposed Denoising Diffusion Probabilistic model
for Latent features (LDDPM) for the SISR task. Firstly, We briefly introduce
the basic architecture of the model. Secondly, We review the Denoising
Diffusion Probabilistic model is review. Then, the critical components in
LDDPM are described in detail. Finally, the Loss Function of LDDPM is
introduced.

###  III-A Denoising Diffusion Probabilistic model for Latent features

In the SISR task, a given LR image $X\in{R^{w\times h\times c}}$ is restored
to the corresponding HR image
$Y\in{R^{{w_{s\uparrow}}\times{h_{{}_{s\uparrow}}}\times c}}$ . Where $w,h,c$
are the width, length, and the number of channels of image $X$ , respectively,
and $s\uparrow$ is the Upsampling factor. Therefore, the Superresolution
problem of a single image can be described in Eq. (1).

$\begin{array}[]{l}X=k\otimes Y+n,\end{array}$ (1)

Where $n$ represents Gaussian white noise, $k$ represents the Convolution
Kernel of the subsampling, and $\otimes$ represents the Subsampling of the
Convolution. The SISR task aims to model Eq. (1) as a maximum posterior
probability problem, as shown in Eq. (2).

$\begin{array}[]{l}\widehat{Y}=\mathop{\arg\max}\limits_{Y}\log
q\left({Y}\right)+\log q\left({{X}\left|{Y}\right.}\right),\end{array}$ (2)

Where $\widehat{Y}$ represents the reconstructed HR image, $\log q({Y})$
represents the model-optimized HR image, and $\log q({X}|{Y})$ represents the
Log-Likelihood of the LR image under given HR image conditions. However, in
the Traditional SISR task, the model is not only easy to collapse but also
cannot recover the image details well. The Denoising Diffusion Probabilistic
model [25][26] transforms the standard normal distribution into empirical data
distribution (similar to Langevin dynamics) through a series of refinement
steps. The Denoising Diffusion Probabilistic model can reduce model collapse
and retain more image details.Therefore, in this paper, the parameters of
$\log q({X_{1}},{X_{2}}...{X_{T}}|{Y})$ are learned to approximate
$\widehat{Y}$ by a random iterative refinement process in the way of DDPM. The
process is gradually mapping the Source image ${X_{1}},{X_{2}}...{X_{T}}$ to
the Target image ${Y}$ to achieve a ”one-to-many” mapping. The Target image
${Y}$ is gradually consistent with multiple Source images
${X_{1}},{X_{2}}...{X_{T}}$ as far as possible. Therefore, this paper can
change Eq.(2) into a modeling method based on DDPM, as shown in Eq. (3)).

$\begin{array}[]{l}\widehat{Y}=\mathop{\arg\max}\limits_{Y}\log q({Y})+\log
q({X_{1}},{X_{2}}...{X_{T}}|{Y}),\end{array}$ (3)

Where ${X_{i}}$ contains the LR image $X$ and the addition of ${X_{i-1}}$ to
Gaussian noise a t step $i$ , and $T$ is the total diffusion step.In DDPM,$Y$
gradually adds Gaussian noise to generate latent variables
${X_{1}},{X_{2}}...{X_{T}}$. The LDPM in this paper is shown in Figure 2, and
the LDPM is built on the DDPM of the $T$ Steps. Instead of directly
reconstructing the HR image at each iteration step of LDDPM, the UNet network
is used to predict the noise $\varepsilon$ in ${X_{i}}$ at the current $i$-th
step. In the LDDPM model, we add Conditional Encoding Mechanism, divided into
Conditional Encoding based on an Adaptive Multi-Head Attention Mechanism and
Conditional Encoding based on VAE. In the Conditional Encoding based on
Adaptive Multi-Head Attention Mechanism, we map the LR image features encoded
by the Conditional Encoder to the middle layer of UNet using the Multi-Head
Attention Mechanism to guide the UNet network to learn more latent features in
LR images.In the Conditional Encoding based on VAE, we use VAE to sample
random feature vectors from LR image $X$ as conditional feature ${F_{R}}$ and
combine the mean map ${F_{\mu}}$ and variance map ${F_{\sigma}}$ decomposed by
the feature vector ${F_{X}}$ of UNet encoder to transfer the conditional
features to the hidden space. VAE not only effectively fills in the missing
information of LR image amplification but also constrains the solution space
of the reconstructed HR image, making it easier for the model to learn the
noise at the current moment.

For the encoder output feature ${F_{g}}$ of the UNet network, we use
Normalized Flow, which can better make the model induce a more complex
probability distribution bias. During training, to ensure high-quality
sampling of the LDDPM, GAN is adopted to learn the Multi-Modal distribution of
${{\bf{X}}_{i-1}},{{\bf{X}}_{i}}$. The Multi-Modal distribution replaces the
simple Gaussian distribution learned by the original DDPM. Thus, the Kullback-
Leibler Divergence (KL Divergence) of the noise probability distributions of
the denoised model and the real model can be reduced.

Figure 2: Network architecture of the LDDPM model. Our model can be viewed as
a Conditional Denoising Diffusion Probabilistic model.

###  III-B Denoising Diffusion Probabilistic model

In DDPM, the HR image $Y$ is defined as the Target variable, and
$q\left(Y\right)$ is the probability distribution of the Target variable. As
shown in Figure 3, DDPM consists of forward and reverse diffusion processes.
The forward diffusion process of DDPM aims to map $Y$ to a Multidimensional
normal distribution (Gaussian noise) through a Markov chain, and the
calculation method is shown in Eq. (4).

$\begin{array}[]{l}q\left({{X_{1}},\cdots,{X_{T}}\left|{{X_{0}}}\right.}\right)=\prod\limits_{i=1}^{T}q\left({{X_{i}}\left|{{X_{i-1}}}\right.}\right),\end{array}$
(4)

Where we define ${X_{0}}$ as $Y$,${X_{i}}$ and $Y$ are variables of the same
dimension,$T$ is the number of diffusion steps, and
$q\left({{X_{i}}\left|{{X_{i-1}}}\right.}\right)$ is defined as the Gaussian
distribution ${\cal
N}\left({{X_{i}};\sqrt{1-{\beta_{i}}}{X_{i-1}},{\beta_{i}}I}\right)$
associated with the constant ${\beta_{i}}$. A small amount of Gaussian noise
is added at each diffusion step in the process, and the final HR image is
transformed into a Multidimensional Gaussian distribution with different
dimensions independent of each other. The inverse diffusion process of DDPM is
based on sampling the Gaussian distribution to generate HR images, which is
calculated as shown in Eq. 5.

$\begin{array}[]{l}{p_{\theta}}\left({{X_{0}},\cdots,{X_{T-1}}\left|{{X_{T}}}\right.}\right)=\prod\limits_{i=1}^{T}{{p_{\theta}}}\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right),\\\
{p_{\theta}}\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right){\rm{=}}{\cal
N}\left({{X_{i-1}};{\mu_{\theta}}\left({{X_{i}},i}\right),{\sigma_{\theta}}{{\left({{X_{i}},i}\right)}^{2}}I}\right),\\\
where\quad p\left({{X_{T}}}\right)={\cal N}\left({0,I}\right),\end{array}$ (5)

The process model can gradually eliminate Gaussian noise and generate HR
images matching the Target distribution. It is worth noting that we train only
the mean Function ${\mu_{\theta}}$ and the variance Function
${\sigma_{\theta}}$ in the model training so that the model can be sampled to
generate HR images. In addition, we set ${\sigma_{\theta}}$ as a constant so
that ${\mu_{\theta}}$ can be rewritten, as shown in Eq. 6, according to the
parameter re-referencing.

$\begin{array}[]{l}{\mu_{\theta}}\left({{X_{i}},i}\right)=\frac{1}{{\sqrt{{a_{i}}}}}\left({{X_{i}}-\frac{{{\beta_{i}}}}{{\sqrt{1-{{\bar{\alpha}}_{i}}}}}{\varepsilon_{\theta}}\left({{X_{i}},i}\right)}\right),\\\
where\quad{\alpha_{i}}=1-{\beta_{i}},\overline{{\alpha_{i}}}=\prod\limits_{s=1}^{i}{{\alpha_{s}}},\end{array}$
(6)

Ultimately, DDPM can be interpreted as abstracting the noise $\varepsilon$
added at step $i$ from ${X_{i}}$ given an image $Y$, noise $\varepsilon$, and
step $i$. To achieve this, the model needs to learn valid feature information
from ${X_{i}}$ ,$\varepsilon$, and step $i$ in order to allow the HR image $Y$
to be gradually mapped to the corresponding noise values according to the
specified rules and to generate a distribution similar to HR image $Y$ based
on the noise values during the Reverse diffusion process. Therefore, the Loss
Function of the DDPM is defined as shown in Eq. (7).

$\begin{array}[]{l}{L_{DDPM}}={\mathbb{E}_{i,{X_{{}_{0}}},\varepsilon}}\left[{{{\left\|{\varepsilon-{\varepsilon_{\theta}}\left({\sqrt{{{\bar{\alpha}}_{i}}}{X_{0}}+\sqrt{1-{{\bar{\alpha}}_{i}}}\varepsilon,i}\right)}\right\|}^{2}}}\right],\end{array}$
(7)

Figure 3: Overview of the forward and inverse diffusion processes of the
Denoising Diffusion Probabilistic model. The forward diffusion process is from
left to right, and the inverse diffusion process is from right to left and
$\theta$ denotes the learnable parameters.

###  III-C Conditional Encoding Mechanism

The purpose of the LDDPM on the SISR task is to model the conditional
distribution $P(X|Y)$. Therefore we can control the synthesis process of HR
images by inputting the LR image $X$ encoding as a condition to the Function
${\varepsilon_{\theta}}\left(\cdot\right)$.However, if $X$ and the noisy image
${X_{i}}$ at the current moment are directly stacked together in the UNet
network for conditional sampling, UNet not only tends to ignore detailed
features related to the perceptual context, but also requires the use of
expensive Function evaluation in pixel space to better extract the noisy
features. Therefore, encoding LR image $X$ in DDPM and using it as a
conditional input to the model to better learn the noise distribution deserves
further investigation.

Conditional Encoding based on Adaptive Multi-Headed Attention Mechanism:For
the modeling condition of LDDPM, we not only stack the noisy image ${X_{i}}$
and LR image $X$ at the current moment but also design a Conditional Encoding
method based on an adaptive Multi-Headed Attention Mechanism inspired by
Rombach et al. [27] and Qin et al. [28] in this paper. First, an Encoder
${{\cal T}_{\theta}}\left(\cdot\right)$ based on an LR image is designed, and
the LR image $X$ is projected to the same dimension of Unet’s middle layer
features using Encoder ${{\cal T}_{\theta}}\left(\cdot\right)$; then, the
features of the middle layer and the features projected by ${{\cal
T}_{\theta}}\left(\cdot\right)$ are points multiplied by the Multi-Headed
Attention Mechanism in UNet. Thus, the Conditional Adaptive Multi-Headed
Attention Mechanism is calculated as shown in Eq.(8).

$\begin{array}[]{l}\left({Q,K,V}\right)=softmax\left({\frac{{Q{K^{T}}}}{{\sqrt{d}}}}\right)\cdot
V\\\ where\quad
Q=W_{Q}^{\left(k\right)}\cdot{\varphi_{k}}\left({{X_{i}}}\right),\\\
\quad\quad\quad\quad
K=W_{K}^{\left(k\right)}\cdot{\tau_{\theta}}\left(X\right),\\\
\quad\quad\quad\quad
V=W_{V}^{\left(k\right)}\cdot{\tau_{\theta}}\left(X\right)\end{array}$ (8)

Where ${\varphi_{k}}\left(\cdot\right)$ denotes the flat feature operation and
$W_{Q}^{\left(k\right)}$,$W_{K}^{\left(k\right)}$ and $W_{V}^{\left(k\right)}$
are the projection matrices of the $k$-th intermediate layer of UNet. It is
worth noting that in the Glow model based on Normalized Flow, we replace the
Mask matrix in Glow with the feature matrix output from the Conditional
Encoder ${{\cal T}_{\theta}}\left(\cdot\right)$ so that the model learns the
noise $\varepsilon$ of the current step $i$ better.

Conditional Encoding based on Variational Auto-Encoder:For Conditional
Encoding in DDPM, we also designed the Conditional Variational Auto-Encoder
(CVAE) [29]. The CVAE is a variational inference modeling of the hidden $Z$
and observed variables $C\in{X_{1,2...T}}$ using a reparameterization
technique. The CVAE can project the conditional ${X_{i}}$ into the latent
space and thus learn the latent conditional probability distribution. The
decoder of UNet will refer to the conditional features ${F_{c}}$, which are
output from CVAE, to learn the feature map ${F_{g}}$ for the prediction of
UNet noise $\varepsilon$. The CVAE model is mainly divided into Feature
Encoder, Hidden Variable $Z$, and Feature Decoder, as shown in Fig. 2. the
Feature Encoder of CVAE is mainly used to fit the Likelihood Function, which
is calculated as shown in Eq. (9).

$\begin{array}[]{l}{P_{\theta}}(C\left|Z\right.)=\prod\limits_{i=1}^{T}{\cal
N}\left({{X_{i}};{\mu_{\theta}}\left({{Z_{i}}}\right),{\sigma_{\theta}}\left({{Z_{i}}}\right)I}\right),\end{array}$
(9)

The values of the Likelihood Functions depend on the results of the
${\mu_{\theta}}$ and ${\sigma_{\theta}}$ Functions
calculations.${\mu_{\theta}}$ and ${\sigma_{\theta}}$ learn the mean and
variance of the Gaussian distribution, respectively, and they thus learn the
relationships between pixels and represent them in a probabilistic model. The
hidden variable $Z$ can be denoted by ${\mu_{\theta}}$ and ${\sigma_{\theta}}$
respectively in CVAE,which is calculated as shown in Eq. (10).

$\begin{array}[]{l}{Z_{i}}={\mu_{\theta}}\left({{X_{i}}}\right)+\delta\cdot{\sigma_{\theta}}\left({{X_{i}}}\right),\\\
where\quad\delta\sim{\cal N}\left({0,I}\right)\end{array}$ (10)

Where the hidden variable $Z$ is sampled from the Gaussian distribution
$Q(Z)=N\sim(0,I)$, as shown in Figure 2. To ensure that randomness is
introduced in the sampling process and that the probability distribution
learned by CVAE is close to a Gaussian distribution, we use KL Divergence to
optimize CVAE, which is calculated as shown in Eq. (11).

$\begin{array}[]{l}{D_{KL}}\left({P\left({Z\left|C\right.}\right)\left\|{Q\left(Z\right)}\right.}\right)=E\left[{\log
P(Z\left|C\right.)-Q(Z)}\right]\\\
{\rm{}}=\frac{1}{2}\left({-\sum\limits_{i}{\left({\log\sigma_{i}^{2}+1}\right)}+\sum\limits_{i}{\sigma_{i}^{2}}+\sum\limits_{i}{\mu_{i}^{2}}}\right)\end{array}$
(11)

In the LDDPM forward diffusion process, first, we replace the Gaussian
distribution of the Feature Decoder input using the hidden variable $Z$. Then,
we use the convolutional layer to project the probability distribution of the
output of the Feature Decoder into the spatial domain to obtain the
conditional probability mapping feature ${F_{R}}$. Finally, to map the
conditional probability ${F_{R}}$ to the output of the UNet encoder, we use
the convolutional layer to learn the mean ${F_{\mu}}$ and variance
${F_{\delta}}$ of the feature mapping ${F_{X}}$ of the output of the UNet
encoder, and the conditional probability ${F_{R}}$ and ${F_{X}}$ are fused to
obtain the fused features ${F_{g}}$. It is worth noting that the mean
${F_{\mu}}$ and variance ${F_{\delta}}$ are the spatial variables of the
feature mapping, not the variables of the Gaussian distribution. In addition,
we can remove the Feature Encoder of CVAE in the inverse diffusion process of
LDDPM and use the random Gaussian distribution as the input hidden variable
$Z$ of the Feature Decoder.

###  III-D Optimized Denoising Diffusion Probabilistic model

Glow-based model Optimization:In the LDDPM-based SISR task’s problem setting,
this paper aims to make the Posterior Encoder of LDDPM able to reconstruct the
HR image accurately. However, the Prior Encoder of LDDPM can learn better
probability distributions for the Posterior Encoder to perform better
reconstruction of HR images. The Gaussian distribution is utilized in the CVAE
of the LDDPM to parameterize the Prior and Posterior Encoders of the UNet, so
we applied the Glow model based on Normalized Flow [30] to the feature map
${F_{g}}$. Glow is a type of Flow model that consists of a combination of
multiple Superficial Layers, each of which consists of a Squeeze Function and
a Flow step. Each Flow step contains ActNorm, 1x1 convolutional layer, and
Coupling Layer. We designed Glow to be able to convert the simple distribution
into a more complex distribution based on the Gaussian distribution of the
CVAE output, according to the law of the changing noise, using the bijection
Function ${f_{\theta}}\left(\cdot\right)$. This enables the Posterior Encoder
to reconstruct the complex probability distribution of the HR image more
easily. Glow is calculated as shown in Eq. (12).

$\begin{array}[]{l}{p_{\theta}}(C\left|{{F_{g}}}\right.){\rm{}}=N\left({{f_{\theta}}(C);{\mu_{\theta}}({F_{g}}),{\sigma_{\theta}}({F_{g}})}\right)\left|{det\frac{{\partial{f_{\theta}}(C)}}{{\partial
C}}}\right|\end{array}$ (12)

GAN-based model Optimization:The KL Divergence
${D_{KL}}(C,Y)={\mathbb{E}_{X}}\log\frac{C}{Y}$ in the DDPM describes the
degree of information loss by replacing the $C$ distribution with the $Y$
distribution. Therefore, the LDDPM should ensure that the KL Divergence of the
probability distribution
${p_{\theta}}\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)$ of the denoising
model in the inverse diffusion process and the probability distribution
$q\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)$ of the denoising model in
the forward diffusion process is as small as possible to ensure a better match
between the probability distribution of the real HR image and the
reconstructed HR image. Xiao et al.[31] demonstrated that the data
distribution would approach a unimodal Gaussian distribution as Gaussian noise
is added incrementally in the forward diffusion process, while the data
distribution will become more complex from a Gaussian distribution as the step
size increases in the forward diffusion process. Therefore, in this paper, we
design Conditional GAN to estimate the true denoising distribution
$q\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)$, thus enabling LDDPM to
model Multimodal denoising distributions with stronger expressive power. The
goal of our conditional GAN is to minimize the Adversarial Loss Function, thus
minimizing ${D_{KL}}(C,Y)$ and improving the match between the probability
distribution ${p_{\theta}}\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)$ of
the inverse diffusion process of LDDPM and the true denoising distribution
$q\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)$ of the forward diffusion
process. The Conditional GAN proposed in this paper, shown in Figure 4, sets
up a time-dependent Discriminator ${D_{\phi}}\left(\cdot\right)$ . The input
of the Discriminator is $\widehat{{X_{i-1}}}$ and $\widehat{{X_{i}}}$ computed
by the noise $\varepsilon$, and the output is the confidence of
$\widehat{{X_{i-1}}}$ and $\widehat{{X_{i}}}$. The Discriminator is trained by
Eq. (13).

$\begin{array}[]{l}{{L_{adv}}={{\min}_{\phi}}\sum\limits_{i\geq
1}{{\mathbb{E}_{q\left({{X_{i}}}\right)}}}\left[{{\mathbb{E}_{q\left({\widehat{{X_{i-1}}}\left|{{{\widehat{X}}_{i}}}\right.}\right)}}\left[{-\log\left({{D_{\phi}}\left({\widehat{{X_{i-1}}},\widehat{{X_{i}}},i}\right)}\right)}\right]}\right.}\\\
{\left.{+{\mathbb{E}_{{p_{\theta}}\left({\widehat{{X_{i-1}}}\left|{{{\widehat{X}}_{i}}}\right.}\right)}}\left[{-\log\left({1-{D_{\phi}}\left({\widehat{{X_{i-1}}},\widehat{{X_{i}}},i}\right)}\right)}\right]}\right]\par}\end{array}$
(13)

It is worth noting that the output of the GAN generator is the noise
$\varepsilon$ distribution, so this paper can use Equation (14) to calculate
$\widehat{{X_{i-1}}}$ and $\widehat{{X_{i}}}$.

$\begin{array}[]{l}\widehat{{X_{i-1}}}=\sqrt{{{\bar{\alpha}}_{i-1}}}{X_{0}}+\sqrt{1-{{\bar{\alpha}}_{i-1}}}{\varepsilon_{\theta}},\\\
\widehat{{X_{i}}}=\sqrt{{\alpha_{i-1}}}\widehat{{X_{i-1}}}+\sqrt{1-{\alpha_{i-1}}}\varepsilon\\\
where\quad\varepsilon\sim{\cal N}\left({0,I}\right)\end{array}$ (14)

Compared with DDPM, the distribution of HR images reconstructed by the inverse
diffusion process of LDDPM now in this paper is more complex, and LDDPM is an
implicit model. the forward diffusion process of LDDPM is still an additive
Gaussian noise process, so no matter how long the step length or how complex
the data distribution is, the forward diffusion process
$q\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)$ obeys the nature of Gaussian
distribution. Therefore, the inverse diffusion process
${p_{\theta}}\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)$ of LDDPM can be
expressed by Eq. (15).

$\begin{array}[]{l}{p_{\theta}}\left({{X_{i-1}}\left|{{X_{i}}}\right.}\right)=\int{{p_{\theta}}}\left({{\varepsilon_{\theta}}\left|{{X_{i}}}\right.}\right)q\left({\widehat{{X_{i-1}}}\left|{\widehat{{X_{i}}}}\right.,{\varepsilon_{\theta}}}\right)d{\varepsilon_{\theta}}\\\
{\rm{}}=\int
p(Z)q\left({\widehat{{X_{i-1}}}\left|{\widehat{{X_{i}}}}\right.,{\varepsilon_{\theta}}={G_{\theta}}\left({{X_{i}},Z,i}\right)}\right)dZ\end{array}$
(15)

where
${p_{\theta}}\left({{\varepsilon_{\theta}}\left|{{X_{i}}}\right.}\right)$ is
an implicit distribution added by the generator
${G_{\theta}}\left(\cdot\right)$ of the GAN, and the generator inputs are
${X_{i}}$, $Z\sim p(Z)={\cal N}(Z;0,I)$ and the number of diffusion steps $i$
.

Figure 4: Denoising process of LDDP Mbased on Generative Adversarial Network

###  III-E Loss Function for Training

During the training process, the LDDPM gradually maps $Y$ to a Gaussian
distribution through a Markov chain. However, the error of noise increases
with the number of iteration steps, and in order to reduce the noise error
during the training process bring about an increase in the perceived distance
between the reconstructed HR image $\widehat{Y}$ and the real HR image $Y$, we
utilize Content-Aware, Style-Aware for guiding LDDPM to reconstruct HR images.
The idea is that the real denoised image ${X_{i}}$ for each iteration step
needs to pass the style information to the predicted denoised image
$\widehat{{X_{i}}}$ while retaining the content information.

Content Loss:Ledig et al. [32] proved that the Mean Square Error(MSE) Loss
Function is prone to a lack of High-frequency details, which leads to
excessive smooth textures in the reconstructed images in the SISR task.
Therefore, we calculated the loss values of image ${X_{i}}$ and
${\widehat{X}_{i}}$ according to the Content Perception Loss based on VGG-19
so as to retain more detailed features of image ${X_{i}}$. Different from
Ledig et al., we also calculated the loss value of real HR image $Y$ and
reconstructed HR image $\widehat{Y}$ for the difference between pixels.
Therefore, the Content Loss of LDDPM is calculated as shown in Eq. (16).

$\begin{array}[]{l}{L_{content}}=\frac{1}{{{W_{i,j}}{H_{i,j}}}}\sum\limits_{x=1}^{{W_{i,j}}}{\sum\limits_{y=1}^{{H_{i,j}}}{\left({vg{g_{i,j}}{{\left({{X_{i}}}\right)}_{x,y}}}\right.}}\\\
-{\left.{vg{g_{i,j}}{{\left({{G_{{\theta_{G}}}}\left({{{\widehat{X}}_{i}}}\right)}\right)}_{x,y}}}\right)^{2}}{\rm{+}}\left\|{\bf{Y}}\right.-{\left.{\widehat{\bf{Y}}}\right\|^{1}}\\\
where\quad\widehat{\bf{Y}}=({X_{i}}-\sqrt{1-{{\bar{\alpha}}_{i}}}{\varepsilon_{\theta}})\times\frac{1}{{\sqrt{{{\bar{\alpha}}_{i}}}}}\end{array}$
(16)

Style Loss: Park et al.[33] used VGG-19 to extract feature maps and calculate
the mean and variance of feature maps so as to reduce the Style Loss of
feature maps. Therefore, LDDPM refers to Park et al. ’s method to calculate
the Style Loss value of ${X_{i}}$ and $\widehat{{X_{i}}}$,as shown in Eq.
(LABEL:eq17).

$\begin{array}[]{l}{L_{{\rm{style}}}}=\frac{1}{{{W_{i,j}}{H_{i,j}}}}\sum\limits_{x=1}^{{W_{i,j}}}{\sum\limits_{y=1}^{{H_{i,j}}}{\left({{\mu_{\theta}}\left({vg{g_{i,j}}{{\left({{X_{i}}}\right)}_{x,y}}}\right)}\right.}}\\\
{\rm{}}-{\left.{{\mu_{\theta}}\left({vg{g_{i,j}}{{\left({{G_{{\theta_{G}}}}\left({{{\widehat{X}}_{i}}}\right)}\right)}_{x,y}}}\right)}\right)^{2}}\\\
{\rm{}}+{\left({{\sigma_{\theta}}\left({vg{g_{i,j}}{{\left({{X_{i}}}\right)}_{x,y}}}\right)-\left.{{\sigma_{\theta}}\left({vg{g_{i,j}}{{\left({{G_{{\theta_{G}}}}\left({{{\widehat{X}}_{i}}}\right)}\right)}_{x,y}}}\right)}\right)}\right)^{2}}\\\
\par{\rm{}}\end{array}$ (17)

Therefore, the total Loss Function of LDDPM model is shown in Eq. (18).

$\begin{array}[]{l}{L_{total=}}{L_{DDPM}}+{D_{KL}}\left({P\left({Z\left|C\right.}\right)\left\|{Q\left(Z\right)}\right.}\right)\\\
{\rm{}}+{L_{adv}}+{L_{content}}+{L_{{\rm{style}}}\par\par}\end{array}$ (18)

##  IV Experiments and Analysis

In this section, the experimental details of the model are presented, and the
applicability of the proposed model for SISR tasks is proved. Firstly, the
experimental setup of the model is introduced in detail. Then, the
experimental results of the proposed model and other advanced models are
introduced. Finally, the network structure of the model is verified by
ablation experiments, and it is proved that the network structure designed in
this paper can recover image details and texture features well and reconstruct
realistic High-resolution images on the SISR task.

###  IV-A Experimental Settings

Datasets: We validated the applicability of LDDPM on a face-based dataset (8×)
and a dataset for general tasks (2×). For the face-based dataset, this paper
used the Flickr-Faces- High-quality (FFHQ) [Karras et al. [34], 2019] and
CelebFaces Attribute (CelebA) datasets [Liu et al. [35], 2018]. FFHQ is a
High-quality face dataset containing 70K face datasets. The dataset is not
only rich and distinct in age, race, and image context, but also possesses a
very large number of variations in face attributes. The training set of FFHQ
contains 30K images, and the test set consists of 2000 images. The CelebA
dataset is a face attribute dataset with 200K. The dataset images cover a
large range of pose variations and background clutter. The training set of
CelebA in this experiment comprises 54K images, and the test set comprises
5,000 images. For model training and prediction, we resized the images in the
CelebA and FFHQ datasets to HR images of 128×128 size and downsampled the HR
using dual cubic kernels to generate LR images of 16×16 size.

For the general task of image super-resolution datasets, we used the DIV2K
dataset [Agustsson et al. [36], 2017] and the Flickr2K dataset [Lim et al.
[37], 2017] together for training and testing. We selected 900 images on the
DIV2K dataset as the training set and 2650 images on the Flick2K dataset as
the training set. To ensure the generalization of the model, Set5, Set14,
Urban100, and Manga109 were chosen as the test sets. In addition, we cropped
each image in the dataset into 128 × 128 sizes to obtain the HR image and
downsampled the HR image using a double kernel to generate a 64 × 64 size LR
image. Finally, we used the image degradation algorithm of Zhang et al. [38]
for the LR images to improve the model’s robustness, which contains fuzzy
degradation, downsampling degradation, and random permutation of the noise
degradation. Two homogeneous and heterogeneous Gaussian blurs simulate blur
degradation. Downsampling degradation is image degradation by random selection
methods from nearest neighbor and bilinear and cubic spline interpolation.
Noise degradation is the addition of different noise levels of Gaussian noise,
JPEG compression of different quality, and reversal of ISP-generated sensor
noise to the LR image.

Experimental parameters:This paper used graphics cards for model training were
8 GeForce RTX 3090 24GB and 4 TITAN V 16GB. Model parameter settings for
training and testing on the CelebA, CelebA+FFHQ(CeleHQ) , DIV2K and
DIV2K+Flick2K(DIFL2K) datasets are provided in Table I. These settings are
used for the main results in the whole table. The same settings were used for
all variants of the LDDPM in the ablation experiments.We chose Peak Signal-to-
Noise Ratio (PSNR) and Structural Similarity (SSIM) as the evaluation metrics
for the experiment. Where PSNR measures the difference between the pixel
points of the reconstructed HR image and the original HR image by calculating
the peak signal-to-noise ratio, while SSIM is a metric that calculates the
structural similarity between the two images to calculate the similarity of
the two images.

TABLE I: Training parameter Settings of the model Training Config | DIV2K/DIFL2K | CelebA/CeleHQ
---|---|---
High-Resolution Size | 128×128 | 128×128
Low-Resolution Size | 64×64 | 16×16
Inner Channel | 64 | 64
Channel Multiplier | (1,2,4,8,8) | (1,2,4,8,8)
Dropout | 0.2 | 0.2
Timestep | 1000 | 1000
Base learning rate | 3E-05 | 1E-04
Learning Rate Schedule | Cosine Decay | Cosine Decay
Batch size | 16 | 20
Training epochs | 50 | 50
Exponential Moving Average (EMA) | 0.9999 | 0.9999
Optimizer | Adamw[39] | Adamw
OptimizeR Momentum | ${\beta_{1}},{\beta_{2}}=0.9,0.999$ | ${\beta_{1}},{\beta_{2}}=0.9,0.999$
Layer Scale | 1E-06 | 1E-06

###  IV-B Comparison of general Image Super-Resolution Experiments

In this section, we evaluate the reconstruction effect of LDDPM on the face
image super-resolution defense dataset (8×) and image super-resolution dataset
(2×) of the general dataset by comparing LDDPM with the advanced image super-
resolution model.

For the general image super-resolution dataset (2×), we first trained on the
DIV2K dataset the EDSR [Lim et al. [37],2017], RCAN [Zhang et al. [40], 2018],
SAN [Dai et al.[14], 2019], IGN [Zhou et al. [16], 2020], HAN [Niu et al.
[15], 2020], NLSA [Mei et al. [41], 2021], and LDDPM (Our) models. Then, the
SwinIR [liang et al.[42],2021], SwinFIR [Zhang et al. [43],2022], EDT [Li et
al. [44],2022], and LDDPM models were trained on the DIFL2K dataset. Finally,
we use the CelebA dataset as a pre-training dataset to train SR3 [Saharia et
al. [23], 2022] and LDDPM and will fine-tune the model parameters on the
DIFL2K dataset. EDSR, RCAN, SAN, IGNN, and HAN are mainly CNN-based models.
swinIR, EDT, and SwinFIR are mainly Transformer-based models. SR3 and LDDPM
are mainly DDPM-based models. Table II shows the experimental results of the
classical single-image super-resolution model. Compared with other advanced
models, the LDDPM in this paper achieves higher performance in reconstructing
HR images on several test sets. In particular, LDDPM improves PSNR and SSIM by
2.07 dB and 1.95%, respectively, compared with the EDT model on Urban100,
which proves the effectiveness of LDDPM and provides a new idea for the SISR
task. Meanwhile, we used CelebA as the LDDPM and SR3 pre-training dataset in
the comparison experiments. Table II shows that the LDDPM has a PSNR of 42.96
dB and SSIM of 97% on the Manga109 dataset. Compared with the DDPM-based SR3
model, PSNR improved by 6.57 dB, and SSIM improved by 1.75%, indicating that
adding a pre-training dataset significantly improved the reconstruction effect
of LDDPM.

Finally, we visualized the HR images recovered by some advanced models, and
the visualization results are shown in Figure 5. From Fig. 5, we can see that
the current Transformer and CNN-based models still have much room for
improvement in reconstructing details and textures of complex images. At the
same time, LDDPM can well solve the problems of the above Transformer and CNN-
based models and reconstruct HR images with High-frequency details.

TABLE II: Quantitative comparison of LDDPM models with advanced models on classical image hyper-resolution data (2×) Models | Training Dataset | Set5 | Set14 | Urban100 | Manga109
---|---|---|---|---|---
PSNR(dB) | SSIM(%) | PSNR(dB) | SSIM(%) | PSNR(dB) | SSIM(%) | PSNR(dB) | SSIM(%)
RCAN(ECCV,2018) | DIV2K | 38.11 | 96.02 | 33.92 | 91.95 | 32.93 | 93.51 | 39.1 | 97.73
EDSR(CVPR,2017) | DIV2K | 38.27 | 96.14 | 34.12 | 92.16 | 33.34 | 93.84 | 39.44 | 97.86
SAN(CVPR,2019) | DIV2K | 38.31 | 96.2 | 34.07 | 92.13 | 33.1 | 93.7 | 39.32 | 97.92
IGNN(NIPS,2020) | DIV2K | 38.24 | 96.13 | 34.07 | 92.17 | 33.23 | 93.83 | 39.35 | 97.86
HAN(ECCV,2020) | DIV2K | 38.27 | 96.14 | 34.16 | 92.17 | 33.35 | 93.85 | 39.46 | 97.85
NLSN(CVPR,2021) | DIV2K | 38.34 | 96.18 | 34.08 | 92.31 | 33.42 | 93.94 | 39.59 | 97.89
LDDPM | DIV2K | 36.75 | 94.08 | 32.89 | 91.42 | 31.3 | 92.02 | 36.21 | 96.36
SwinIR(CVPR,2021) | DIFL2K | 38.42 | 96.23 | 34.46 | 92.5 | 33.81 | 94.27 | 39.92 | 97.97
SwinFIR(arXiv,2022) | DIFL2K | 28.57 | 96.3 | 34.66 | 92.63 | 34.3 | 94.59 | 40.3 | 98.09
EDT(arXiv,2022) | DIFL2K | 38.63 | 96.32 | 34.8 | 92.73 | 34.27 | 94.56 | 40.37 | 98.11
LDDPM | DIFL2K | 37.17 | 96.1 | 36.87 | 94.68 | 34.44 | 96.02 | 41.48 | 97.21
SR3(T-PAMI,2022) | CelebA | 32.89 | 95.78 | 36.55 | 92.8 | 34.06 | 95.94 | 36.39 | 95.25
LDDPM | CelebA | 38.91 | 96.39 | 36.27 | 95.25 | 35.14 | 96.1 | 42.96 | 97
Figure 5: The results obtained on Urban100 and Set14 datasets (2×) are
visualized by different models.

###  IV-C Comparison of experimental results of Super-resolution of face
images

In this section, LDDPM is experimentally compared with ESRGAN [Wang et al.
[17],2018], ProgFSR [Kim et al. [45], 2019], SRFlow [Lugmayr et al. [46],
2020], SRDiff [Lin et al. [10], 2022] and SR3 models on the CeleHQ dataset ,
as shown in Table III. Where LDDPM, SRDiff, and SR3 are DDPM-based models,
respectively, RRDB and ProgFSR are CNN-based models, ESRGAN is a (GAN)-based
model, and SRFlow is a normalized Flow based model. From Table III, it can be
seen that LDDPM outperforms all the above models in terms of evaluation
metrics, and it improves PSNR by 0.96 dB and SSIM by 2.31% compared with the
advanced SRDiff, indicating that LDDPM is able to generate High-quality and
diverse HR images with strong uniformity with LR. As seen in Figure 6,
compared with other models, the LDDPM reconstructed images of the wrinkles on
the forehead of the elderly and the hair of the woman look more natural and
have rich details and textures.

In addition, LDDPM (43M) uses fewer model parameters than SR3 (98M) and SRDiff
models (52M) and takes only about 20 hours to converge on the CeleHQ dataset,
compared to 34 hours for SRDiff and 40 hours for SR3, indicating that LDDPM is
training efficient and can be used with a smaller computational overhead of
getting better performance.

As shown in Figure 7, this paper visualizes the important detail pixels of the
woman’s face part in the HR image reconstructed by LDDPM with SR3 and SRDiff
using histogram. It can be seen from Fig. 7 that LDDPM can learn more regular
feature distribution to capture good detail and texture features and obtain
better performance.

TABLE III: Quantitative comparison of LDDPM with the state-of-the-art model on the CeleHQ face dataset (8×). Models | PSNR(dB) | SSIM(%)
---|---|---
ESRGAN(ECCV,2018) | 23.24 | 66.45
ProgFSR(arXiv,2019) | 24.21 | 72.24
SRFlow(ECCV,2019) | 25.32 | 72.45
SRDiff(NC,2019) | 25.38 | 74.21
SR3(T-PAMI,2022) | 24.92 | 70.95
LDDPM | 26.07 | 76.52
Figure 6: Face SR (8×) visualization results on CeleHQ dataset. Figure 7: The
grayscale histograms of pixel features of different models are visualized on
the CeleHQ dataset.

###  IV-D Ablation experiments

In this section, to demonstrate the effectiveness of the module added by LDPM,
we perform extensive ablation experiments to validate the proposed module on
the CeleHQ dataset.

Condition Encode:In Conditional Encoding, three models are defined in this
paper, and the effect of Conditional Encoding on the LDDPM is discussed. The
first model (V1) of Conditional Encoding is to stack the low-resolution LR
images and the noisy images at each stage, thus reconstructing HR images. The
second model (V2) adds Conditional Encoding based on the Adaptive Multi-Headed
Attention Mechanism to the V1 model. The third model (V3) adds Conditional
Encoding based on the Variational Auto-Encoder(VAE) on top of V2. As can be
seen from Table IV, the V2 model with the addition of the Adaptive Multi-
Headed Attention Mechanism improves the PSNR and SSIM by 1.05 dB and 1.23 %,
respectively, over the V1 model, indicating that the Adaptive Multi-Headed
Attention Mechanism can provide more conditional features to guide the model
to learn the probability distribution of HR images, thus making the model
reconstructed HR consistent with the real HR. The V3 model with VAE added
improves the PSNR and SSIM by 1.14 dB and 0.83 %, respectively, compared with
V2, indicating that the addition of VAE to the LDDPM enables the model to
learn more latent conditional features in the LR images and use the latent
conditional features to further reduce the solution space of the HR images,
thus constraining the feature information in the image space.

Model optimization based on Glow and GAN:From row 2 of Table V, it can be seen
that the addition of Glow to the LDDPM increases the PSNR and SSIM by 1.03 dB
and 3.19%, respectively, indicating that the addition of Glow to the LDDPM
enables the LDDPM to capture a more complex noise probability distribution. As
can be seen from row 3 of Table V, the PSNR and SSIM distributions rise by
0.87 dB, and 0.97% for the GAN added by LDDPM, indicating that the multimodal
distribution of GAN learning can make the HR images reconstructed by LDDPM
more realistic in the inverse diffusion process. In Figure 8, we visualize the
features extracted by LDDPM adding Glow and GAN. As seen in Figure 8, compared
to the original LDDPM, the LDDPM with Glow and GAN can learn better
probability distributions with a relatively small number of total steps to be
sampled.

Optimization of Experimental Hyperparameters:We performed hyperparametric
ablation experiments to investigate the effect of total diffusion step and
Loss Function on LDDPM. As shown in Figure 9, with the increase in total
diffusion steps, the quality of the images in this paper is enhanced. However,
larger total diffusion steps slow down the training and inference of the
model, so is chosen as the default parameter setting in this paper. Finally,
we compare the effects of Content Loss and Style Loss on the experimental
results in this paper. From Table VI, we can see that adding Content Loss (CL)
to LDDPM increases the PSNR and SSIM by 0.56 dB and 0.49%, respectively,
compared to row 1 of Table VI. Moreover, adding Style Loss (SL) to LDDPM
increases 0.18 dB and 0.64% compared to row 2 of Table VI. The above results
illustrate that adding content loss and style loss to LDDPM can better guide
LDDPM to learn more information of image features, which leads to more stable
training of LDDPM.

TABLE IV: Comparison of PSNR (dB) and SSIM (%) metrics for LDDPM added Conditional Encoding module, with the best results bolded. Models | PSNR(dB) | SSIM(%)
---|---|---
V1 | 23.1 | 71.09
V2 | 24.15 | 72.32
V3 | 25.29 | 73.15
TABLE V: Comparison of PSNR (dB) and SSIM (%) of LDDPM with Glow and GAN, the best results are shown in bold. Models | PSNR(dB) | SSIM(%)
---|---|---
LDDPM | 23.43 | 71.23
LDDPM+Glow | 24.46 | 74.42
LDDPM+Glow+GAN | 25.33 | 75.39
TABLE VI: Comparison of PSNR (dB) and SSIM (%) for LDDPM with content loss and style loss, the best results are shown in bold. Models | PSNR(dB) | SSIM(%)
---|---|---
LDDPM | 25.33 | 75.39
LDDPM+CL | 25.89 | 75.88
LDDPM+CL+SL | 26.07 | 76.52
Figure 8: Visualization of feature sampling by LDDPM with the same number of
steps Figure 9: LDDPM generates HR images under different steps, where t is
the time to generate HR images in seconds.

###  IV-E Experimental comparison of Real Datasets

To evaluate the performance of the model of LDDPM more comprehensively, we
collected some Low-resolution images in the real world. As shown in Figure 10,
the quality of the reconstructed HR images from LDDPM is better than that of
the reconstructed SR3, EDF, and SRDiff. Specifically, the images reconstructed
by SR3, EDF, and SRDiff in Figure 10 are blurred and have missing details and
textures, while the LDDPM model can reconstruct not only clear images but also
reconstructed images with complete details and textures. Experiments on Real-
world datasets demonstrate that LDDPM generalizes well and can be applied well
to SISR tasks in natural environments.

Figure 10: LDDPM generates HR images under different steps, where t is the
time to generate HR images in seconds.

##  V Conclusion

We designed Denoising Diffusion Probabilistic model for Latent features
(LDDPM) to address the ”one-to-many” uncertainty of the SISR task, which
solves the problems of missing details and textures in the reconstructed HR
images, slow sampling speed of the model, and ineffective use of degraded
images in the existing Image Super-resolution Reconstruction methods. LDDPM
mainly uses Markov chains to convert HR images into simple Gaussian
probability distributions and then uses the inverse diffusion process to
reconstruct HR images gradually. We used Conditional Encoder in the forward
and reverse processes of LDDPM. The Conditional Encoder encodes the LR image
using an adaptive Multi-Headed Attention Mechanism and Variational Auto-
Encoder, which significantly constrains the solution space of the
reconstructed image. In addition, to accelerate the convergence speed and
stable training of LDDPM, we add Normalized Flow and Multimodal Adversarial
training to the model. These ways utilize a complex distribution to model each
denoising process, enabling the model to learn the probability distribution of
more complex HR images efficiently and significantly reducing the number of
diffusion steps of LDDPM. Through extensive experiments, it has been
demonstrated that LDDPM can better utilize the LR image feature information to
generate HR images with better perceptual quality at a smaller number of
diffusion steps.

Our work has many shortcomings, such as LDDPM still requires a large number of
sampling steps and whether the number of sampling steps can be further reduced
by Score-based DDPM. In addition, in the forward diffusion process of the
Markov chain, is it possible for LDDPM to add Gaussian white noise along with
JPEG compressed noise of different quality and reversed ISP-generated sensor
noise, Etc., for training to improve the robustness of the model.

Our future work focuses on two research areas:

  1. 1.

To greatly reduce the number of sampling steps in LDDPM by investigating the
Score-based DDPM using the Score-matching technique.

  2. 2.

To improve the quality of LDDPM reconstructed HR images by adding different
kinds of noise to the LDDPM forward diffusion process.

## References

  * [1] X. Jin, J. Hou, S.-J. Lee, and D. Zhou, “Recent advances in artificial neural networks and embedded systems for multi-source image fusion,” _Frontiers in Neurorobotics_ , vol. 16, 2022.
  * [2] M. Wang, Z. Xu, X. Liu, J. Xiong, and W. Xie, “Perceptually quasi-lossless compression of screen content data via visibility modeling and deep forecasting,” _IEEE Trans. Ind. Informatics_ , vol. 18, no. 10, pp. 6865–6875, 2022.
  * [3] J. Zhang, J. Song, L. Gao, Y. Liu, and H. T. Shen, “Progressive meta-learning with curriculum,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2022.
  * [4] T. Ma and W. Tian, “Back-projection-based progressive growing generative adversarial network for single image super-resolution,” _Vis. Comput._ , vol. 37, no. 5, pp. 925–938, 2021.
  * [5] N. Karimi and M. R. Taban, “A convex variational method for super resolution of SAR image with speckle noise,” _Signal Process. Image Commun._ , vol. 90, p. 116061, 2021.
  * [6] H. Zhou, C. Huang, S. Gao, and X. Zhuang, “Vspsr: Explorable super-resolution via variational sparse representation,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 373–381.
  * [7] Y. Shi, L. Han, L. Han, S. Chang, T. Hu, and D. Dancey, “A latent encoder coupled generative adversarial network (LE-GAN) for efficient hyperspectral image super-resolution,” _IEEE Trans. Geosci. Remote. Sens._ , vol. 60, pp. 1–19, 2022.
  * [8] J. Liang, A. Lugmayr, K. Zhang, M. Danelljan, L. V. Gool, and R. Timofte, “Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 4056–4065.
  * [9] Z. Liu, W. Siu, and L. Wang, “Variational autoencoder for reference based image super-resolution,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 516–525.
  * [10] H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Srdiff: Single image super-resolution with diffusion probabilistic models,” _Neurocomputing_ , vol. 479, pp. 47–59, 2022.
  * [11] S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, “Vector quantized diffusion model for text-to-image synthesis,” _CoRR_ , vol. abs/2111.14822, 2021.
  * [12] J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 5530–5540.
  * [13] J. Zhang, C. Long, Y. Wang, H. Piao, H. Mei, X. Yang, and B. Yin, “A two-stage attentive network for single image super-resolution,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 32, no. 3, pp. 1020–1033, 2022.
  * [14] T. Dai, J. Cai, Y. Zhang, S. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2019, pp. 11 065–11 074.
  * [15] B. Niu, W. Wen, W. Ren, X. Zhang, L. Yang, S. Wang, K. Zhang, X. Cao, and H. Shen, “Single image super-resolution via a holistic attention network,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII_ , ser. Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12357\. Springer, 2020, pp. 191–207.
  * [16] S. Zhou, J. Zhang, W. Zuo, and C. C. Loy, “Cross-scale internal graph neural network for image super-resolution,” in _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
  * [17] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. C. Loy, “ESRGAN: enhanced super-resolution generative adversarial networks,” in _Computer Vision - ECCV 2018 Workshops - Munich, Germany, September 8-14, 2018, Proceedings, Part V_ , ser. Lecture Notes in Computer Science, L. Leal-Taixé and S. Roth, Eds., vol. 11133. Springer, 2018, pp. 63–79.
  * [18] Z. Liu, Z. Li, X. Wu, Z. Liu, and W. Chen, “Dsrgan: Detail prior-assisted perceptual single image super-resolution via generative adversarial networks,” _IEEE Transactions on Circuits and Systems for Video Technology_ , pp. 1–1, 2022.
  * [19] I. Gatopoulos, M. Stol, and J. M. Tomczak, “Super-resolution variational auto-encoders,” _CoRR_ , vol. abs/2006.05218, 2020.
  * [20] Z.-S. Liu, W.-C. Siu, and Y.-L. Chan, “‘,” _IEEE Transactions on Circuits and Systems for video Technology_ , vol. 31, no. 4, pp. 1351–1365, 2020\.
  * [21] X. Xiang, L. Zhu, J. Li, Y. Wang, T. Huang, and Y. Tian, “Learning super-resolution reconstruction for high temporal resolution spike stream,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2021.
  * [22] Y. Jo, S. Yang, and S. J. Kim, “Srflow-da: Super-resolution using normalizing flow with deep convolutional block,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 364–372.
  * [23] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” _CoRR_ , vol. abs/2104.07636, 2021.
  * [24] D. Ryu and J. C. Ye, “Pyramidal denoising diffusion probabilistic models,” _CoRR_ , vol. abs/2208.01864, 2022.
  * [25] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _CoRR_ , vol. abs/2006.11239, 2020.
  * [26] A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139\. PMLR, 2021, pp. 8162–8171.
  * [27] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” _CoRR_ , vol. abs/2112.10752, 2021.
  * [28] Q. Qin, J. Yan, X. Wang, Q. Wang, M. Li, and Y. Wang, “Etdnet: An efficient transformer deraining model,” _IEEE Access_ , vol. 9, pp. 119 881–119 893, 2021.
  * [29] D. Liang, R. G. Krishnan, M. D. Hoffman, and T. Jebara, “Variational autoencoders for collaborative filtering,” in _Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018_ , P. Champin, F. Gandon, M. Lalmas, and P. G. Ipeirotis, Eds. ACM, 2018, pp. 689–698.
  * [30] D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” in _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp. 10 236–10 245.
  * [31] Z. Xiao, K. Kreis, and A. Vahdat, “Tackling the generative learning trilemma with denoising diffusion gans,” in _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
  * [32] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 105–114.
  * [33] D. Y. Park and K. H. Lee, “Arbitrary style transfer with style-attentional networks,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2019, pp. 5880–5888.
  * [34] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 43, no. 12, pp. 4217–4228, 2021.
  * [35] Z. Liu, P. Luo, X. Wang, and X. Tang, “Large-scale celebfaces attributes (celeba) dataset,” _Retrieved August_ , vol. 15, no. 2018, p. 11, 2018.
  * [36] E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 1122–1131.
  * [37] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual networks for single image super-resolution,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 1132–1140.
  * [38] K. Zhang, J. Liang, L. V. Gool, and R. Timofte, “Designing a practical degradation model for deep blind image super-resolution,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 4771–4780.
  * [39] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * [40] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII_ , ser. Lecture Notes in Computer Science, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol. 11211. Springer, 2018, pp. 294–310.
  * [41] Y. Mei, Y. Fan, and Y. Zhou, “Image super-resolution with non-local sparse attention,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 3517–3526.
  * [42] J. Liang, J. Cao, G. Sun, K. Zhang, L. V. Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, October 11-17, 2021_. IEEE, 2021, pp. 1833–1844.
  * [43] D. Zhang, F. Huang, S. Liu, X. Wang, and Z. Jin, “Swinfir: Revisiting the swinir with fast fourier convolution and improved training for image super-resolution,” _CoRR_ , vol. abs/2208.11247, 2022.
  * [44] W. Li, X. Lu, J. Lu, X. Zhang, and J. Jia, “On efficient transformer and image pre-training for low-level vision,” _CoRR_ , vol. abs/2112.10175, 2021.
  * [45] D. Kim, M. Kim, G. Kwon, and D. Kim, “Progressive face super-resolution via attention to facial landmark,” in _30th British Machine Vision Conference 2019, BMVC 2019, Cardiff, UK, September 9-12, 2019_. BMVA Press, 2019, p. 192.
  * [46] A. Lugmayr, M. Danelljan, L. V. Gool, and R. Timofte, “Srflow: Learning the super-resolution space with normalizing flow,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V_ , ser. Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12350. Springer, 2020, pp. 715–732.