11institutetext: East China Normal University, Shanghai 200241, China
11email: {52275904013,hfsong}@stu.ecnu.edu.cn,11email:
qlli@cs.ecnu.edu.cn,11email<EMAIL_ADDRESS>
# Spatially-Variant Degradation Model for Dataset-free Super-resolution

Shaojie Guo 11 0009-0008-3955-7240 Haofei Song 11 0009-0006-4926-7525 Qingli
Li 11 0000-0001-5063-8801 Yan Wang Corresponding author11 0000-0002-1592-9627

###### Abstract

This paper focuses on the dataset-free Blind Image Super-Resolution (BISR).
Unlike existing dataset-free BISR methods that focus on obtaining a
degradation kernel for the entire image, we are the first to explicitly design
a spatially-variant degradation model for each pixel. Our method also benefits
from having a significantly smaller number of learnable parameters compared to
data-driven spatially-variant BISR methods. Concretely, each pixel’s
degradation kernel is expressed as a linear combination of a learnable
dictionary composed of a small number of spatially-variant atom kernels. The
coefficient matrices of the atom degradation kernels are derived using
membership functions of fuzzy set theory. We construct a novel Probabilistic
BISR model with tailored likelihood function and prior terms. Subsequently, we
employ the Monte Carlo EM algorithm to infer the degradation kernels for each
pixel. Our method achieves a significant improvement over other state-of-the-
art BISR methods, with an average improvement of 1 dB ($2\times$).Code will be
released at https://github.com/shaojieguoECNU/SVDSR

###### Keywords:

Super-Resolution Dataset-free Degradation Model

##  1 Introduction

Super-Resolution (SR) aims to reconstruct high-resolution (HR) images from
their degradation red, noisy low-resolution (LR) counterparts. Due to the
challenge of obtaining the degradation kernel in real-world scenarios, Blind
Image Super-Resolution (BISR) has recently garnered significant research
interest. Mathematically, SR degradation model can be formulated as:

$\boldsymbol{y}=(\mathcal{D}\boldsymbol{x}){\downarrow_{s}}+\boldsymbol{n},$
(1)

where $\boldsymbol{x}$ denotes the HR image, $\boldsymbol{y}$ denotes the LR
image, $\mathcal{D}$ denotes degradation operator, $\downarrow_{s}$ denotes
subsampling operator with scale factor $s$, and $\boldsymbol{n}$ denotes the
noise.

Numerous BISR methods have been proposed in recent years to reconstruct images
afflicted by complex and unknown degradation operators. These methods can be
broadly classified into two categories. Methods[13, 17, 10, 23] belonging to
the first category directly use complex deep neural networks (DNNs) to map the
observed LR images to HR images, which do not effectively utilize the physical
information (_e.g._ , edge and texture) of image degradation in Eq. 1 and may
yield suboptimal results. Methods belonging to the second category first
estimate the degradation operator $\mathcal{D}$ from the LR images and then
transform the blind super-resolution problem into a non-blind SR problem. The
challenge with this approach lies in how to accurately estimate the unknown
degradation operator. Estimating the degradation operator directly from the LR
image is a highly ill-posed problem. Many priors for degradation operator have
been proposed to better estimate the degradation kernel. [41, 21, 31, 2]
designed explicit or implicit kernel priors for the entire image to better
utilize the information within LR images, these priors are all spatially-
invariant. [4, 15, 20, 45, 19] formulated a unique degradation kernel for each
patch or even each pixel of the image. These spatially-variant adaptive
degradation kernels can fully utilize the structural information differently
for each pixel location, thereby obtaining richer visual details in the
reconstructed image. However, learning its own degenerate kernel for each
pixel is computationally expensive. [45, 19] use a linear combination of 72
pre-defined atom kernels to represent the degenerate kernels of different
pixels, which greatly reduces the computational burden. They meticulously
designed the networks to learn the huge coefficient matrix of the atom kernel
for each pixel, rather than focusing on learning the shape of the entire
degradation kernel. But these networks still need to be trained on large
paired artificially generated dataset, which significantly reduces the
practicality of the proposed methods.

This work introduces a novel spatially-variant degradation model for dataset-
free BISR, which enjoys the benefit of (1) dealing with spatially-variant
degradation model for dataset-free BISR and (2) much smaller numbers of
learnable parameters than data-driven BISR. Our approach enables the
determination of individual degradation kernels for each pixel in the image
without the requirement of a dataset. Each pixel’s degradation kernel in our
model is expressed as a linear combination of a dictionary of learnable atom
kernels. In accordance with the observation in [19], the shape of the
degradation kernel is highly correlated with the texture density of the area
where its corresponding pixel is located. Therefore, we innovatively use fuzzy
set theory to derive the coefficient matrix of the atom kernels based on the
texture information of the images. Different from the fixed atom kernel
dictionary used in previous works, various degradation kernels are enabled to
be more flexibly represented thanks to the learnability and adaptability of
the proposed learnable dictionary. Our approach eliminates the need to learn
the shape of the entire atom kernels. Each atom kernel in the proposed
dictionary is uniquely determined by only three learnable parameters, which
are theoretically supported by high-dimensional Gaussian kernel decomposition
technology [12], significantly reducing the number of learning parameters.
Additionally, we meticulously design the likelihood function and priors under
the Maximum A Posteriori (MAP) framework to further enhance the performance of
the proposed model.

Our main contributions can be summarized as follows:

  * •

The first dataset-free deep learning method for BISR with an explicit
spatially-variant degradation model. To the best of our knowledge, all
existing spatially-variant degradation models require extensive training on
large datasets.

  * •

A novel Probabilistic BISR Model with elaborately designed likelihood function
and prior terms. An inference algorithm for the proposed model is also derived
based on the Monte Carlo Expectation Maximization (MCEM) algorithm.

  * •

Extensive experiments on synthetic datasets and real images verify the
effectiveness of our method. Compared to sota dataset-free BISR methods, there
is an average improvement of 1 dB ($2\times$) achieved.

##  2 Related Works

####  2.0.1 Non-learning BISR Methods:

Most existing non-learning methods adopt an iterative approach to search the
HR image. These models are designed to incorporate various handcrafted image
priors, such as sparsity priors [32, 34, 16, 14] and non-local similarity
priors [5, 7], to address the ill-posed nature of the BISR problem. However,
these methods often yield poor performance due to the inherent structural
complexity of natural images.

####  2.0.2 Data-driven Deep Learning Methods:

Many deep learning methods [6, 8, 11, 13, 22, 24, 28, 35, 38, 40, 44, 19] have
been proposed for BISR task. Here, we provide a brief introduction to
spatially-variant degradation-based models. Shocher _et al_. [33] propose a
Zero-Shot SR (ZSSR) model to leverage the recurring patterns of image patches
across different scales to achieve SR. Liang _et al_. [20] partition images
into patches and suggests a Mutual Affine Network (MANet) to emulate space-
variant blur for each patch. Kim _et al_. [15] design the Kernel-Oriented
Adaptive Local Adjustment (KOALA) method for jointly learning spatially-
variant degradation and restoration kernels. Chen _et al_. [4] design a Cross-
MOdal fuSion network (CMOS) that estimate both blur and semantics
simultaneously for images with out-of-focus blur. Zhou _et al_. [45] improve
SR results by adjusting degradation to known degradation using the proposed
linearly-assembled pixel degradation-adaptive regression module (DARM). [45,
19] utilize a dictionary of multiple pre-defined filter bases to transform the
complex degradation kernel prediction task into a linear coefficient
regression task. [37, 43, 29] use the HR datasets and their degradation
algorithm to generate the HR-LR dataset to train their model. These methods
all require training on large generated datasets and do not fully exploit
physical priors for kernel inference.

####  2.0.3 Dataset-free deep learning Methods:

Training a neural network for BISR without a dataset is challenging. Existing
dataset-free methods all assume that the priors proposed are spatially-
invariant. Ulyanov _et al_. [36] propose the Deep Image Prior (DIP) framework,
which leverages a generator network as an implicit prior for SR tasks without
the need for a dataset. Liang _et al_. [21] and Yue _et al_. [41] respectively
introduce implicit flow-based kernel prior (FKP) and explicit kernel prior
(EKP) into DIP to enhance the performance of BISR. Our method also falls into
this category. However, in contrast to previous approaches which all assume
that the degradation kernel is spatially-invariant, we are a pioneer in
proposing the spatially-invariant degradation model using deep learning
methods without relying on datasets. The unique degradation kernel for each
pixel is established by leveraging the texture information in its local
neighborhood. This approach tailors the degradation process for each pixel
based on its individual characteristics. Besides, the degeneration kernel in
Yue et al. [41] is assumed to follow a simple Gaussian distribution, whereas
our model extremity enhances its representation through the utilization of a
Gaussian mixture model.

##  3 The Proposed Model

In this section, we first suggest a spatially-variant degradation model with
the HR image $\boldsymbol{x}$ and variables related to the degradation
operator (see Sec. 3.1). Then, to model the variables in the degradation
model, a probabilistic model is employed under the Maximum A Posteriori (MAP)
framework, which incorporates uncertainty and probability distributions to
represent and analyze the variables (see Sec. 3.2). In the next section, we
will utilize the MCEM algorithm to solve the proposed probability model (see
Sec. 4). The entire model and its solution are shown in Fig. 1.

###  3.1 Spatially-Variant Degradation Model

Figure 1: The framework of the entire model and its solution (MCEM inference
algorithm). The proposed Spatially-Variant Degradation Model consists of of a
dictionary of learnable atom operators
$\\{\boldsymbol{D}_{i}\\}_{i=1}^{N_{\mathcal{D}}}$ and corresponding
coefficient matrices $\\{\boldsymbol{W}_{i}\\}_{i=1}^{N_{\mathcal{D}}}$ . Each
atom degradation operator is determined by three learnable parameters
$\theta_{i}$, $\sigma_{i,1}$ and $\sigma_{i,2}$. The coefficient matrices
$\\{\boldsymbol{W}_{i}\\}_{i=1}^{N_{\mathcal{D}}}$ are obtained from the
tentative HR image $\tilde{\boldsymbol{x}}$. Image prior, kernel prior and
likelihood are suggested to solve the proposed spatially-variant degradation
model under the MAP framework. In the inference process, the latent variable
$\boldsymbol{z}$ is sampled in the E-Step and parameters
$\\{\boldsymbol{\Gamma}\\}_{i=1}^{N_{\mathcal{D}}}$ and weights $\phi$ of G
are updated in M-step.

Inferring the degradation kernel for each pixel is computationally expensive
and challenging, especially in the absence of a dataset. Therefore, having
reasonable prior assumptions is crucial for improving the computational
efficiency and practicality of spatially-variant degradation models.
Considering that each pixel in the same photo undergoes a similar imaging
process, a lower-dimensional manifold should exist in the degradation operator
space of a whole photo. We utilize the O’Leary [30] formula to decompose
degradation operator as follows:

$(\mathcal{D}\boldsymbol{x})[h,w]:=\sum_{r,c}\sum_{i=1}^{N_{\mathcal{D}}}\boldsymbol{W}_{i}[h,w]\mathcal{D}_{i}\boldsymbol{x}[h-r,w-c],$
(2)

where $\\{\mathcal{D}_{i}\\}^{N_{\mathcal{D}}}_{i=1}$ is a learnable
dictionary composed of a set of atom degradation kernels,
$\\{\boldsymbol{W}_{i}\\}^{N_{\mathcal{D}}}_{i=1}$ is a set of coefficient
matrices, $N_{\mathcal{D}}$ is the number of atom degradation kernels, $[h,w]$
denotes the location of each pixel, $r,c$ denotes the location of the
surrounding pixels. More than that, each element of the degradation kernel
should obey the physical constraints that it is non-negative and the sum of
all elements is equal to one, the constraints
$\sum_{i=1}^{N_{\mathcal{D}}}\boldsymbol{W}_{i}[h,w]=1$,
$\boldsymbol{W}_{i}\geq 0$ are also imposed on Eq. 2. Note that in previous
linearly-assembled degradation models[19], the dictionary
$\\{\mathcal{D}_{i}\\}^{N_{\mathcal{D}}}_{i=1}$ is pre-defined and the
coefficient matrices $\\{\boldsymbol{W}_{i}\\}^{N_{\mathcal{D}}}_{i=1}$ need
to be learned and its size is usually related to the size of the image. We
further model $\mathcal{D}_{i}$ and $\boldsymbol{W}_{i}$ separately.

##### Atom degradation Operator $\mathcal{D}_{i}$:

Inspired by the the fact that any continuous distribution can be approximated
with any specific non-zero amount of error by a Gaussian mixture model with
sufficient components [27]. The atom degradation operator $\mathcal{D}_{i}$ is
expressed through convolution with an anisotropic Gaussian degradation kernel
$\boldsymbol{g}_{i}$ in Eq. 2. Additionally, the anisotropic Gaussian
degradation kernel can be decomposed [12] as follows:

$\displaystyle\mathcal{D}_{i}\triangleq\boldsymbol{g}_{i}\left(x_{1},x_{2}\right)=\prod_{l=1}^{2}\frac{1}{\sqrt{2\pi}\sigma_{i,l}}\exp\left(-\frac{\left(\boldsymbol{x}^{\top}\boldsymbol{\nu}_{l}\right)^{2}}{2\sigma_{i,l}^{2}}\right),$
(3)

where $\boldsymbol{x}=\left({x}_{1},{x}_{2}\right)^{\top}\in\mathbb{R}^{2}$,
$\boldsymbol{\nu}_{1}$$=(\cos(\theta_{i}),\sin(\theta_{i}))$ and
$\boldsymbol{\nu}_{2}$$=(-\sin(\theta_{i}),\cos(\theta_{i}))^{\top}$,
$\sigma_{i,l}$ are the variances of the two isotropic Gaussian kernels
obtained by decomposing $\boldsymbol{g}_{i}$, and $\theta_{i}\in[0,\pi)$ is
the rotation angle of the coordinate axis. Note that for the given
$\boldsymbol{W}_{i}$, Eq. 2 is differentiable _w.r.t._ the set of free
parameters $\boldsymbol{\Gamma}=\\{\theta_{i},\sigma_{i,1},\sigma_{i,2}\\}$,
indicating that $\\{\mathcal{D}_{i}\\}$ are learnable. The learnable atom
degradation operator is in stark contrast to previous spatially-variant
degradation models, where the atom degradation kernel is fixed.

##### Coefficient Matrices $\boldsymbol{W}_{i}$:

The shape of the degradation kernel is significantly different between pixels
in flat areas and high texture density areas[19]. Inspired by the correlation
between the shape of the degradation kernel and the texture richness of its
region, we propose to estimate the set of coefficient matrices
$\boldsymbol{W}_{i}$ for the atom degradation kernel through a sequence of
membership functions of fuzzy set theory, utilizing the tentative HR image
$\tilde{\boldsymbol{x}}$ as follows:

$\boldsymbol{W}_{i}=\frac{\boldsymbol{\mu}_{i}(\tilde{\boldsymbol{x}})}{\sum_{i=1}^{N_{\mathcal{D}}}\boldsymbol{\mu}_{i}(\tilde{\boldsymbol{x}})}.$
(4)

Eq. 4 is a sum normalization operation, aiming to ensure the physical prior
that the sum of all elements is equal to one. As a reminder, when
$N_{\mathcal{D}}=1$, the coefficient matrix corresponding to the only
degradation kernel is an all-ones matrix, which makes the proposed degradation
model naturally degenerates into a conventional spatially-invariant
degradation model. When $N_{\mathcal{D}}\geq 2$, membership function
$\\{\boldsymbol{\mu}_{i}\\}^{N_{\mathcal{D}}}_{i=1}$ are defined as follows:

$\boldsymbol{\mu}_{i}(\tilde{\boldsymbol{x}})=\text{exp}\left(-\frac{\left(N_{\mathcal{D}}-1\right)}{2\sigma_{g}^{2}}\left(\boldsymbol{h}\left(\tilde{\boldsymbol{x}}\right)-\frac{i-1}{N_{\mathcal{D}}-1}\right)^{2}\right),$
(5)

where $\sigma_{g}$ is the shape parameter. $\boldsymbol{h}(\cdot)$ is the
feature extraction function used to extract the texture features of
$\tilde{\boldsymbol{x}}$, _i.e._ ,

$\boldsymbol{h}\left(\tilde{\boldsymbol{x}}\right)=\boldsymbol{H}*(\nabla\tilde{\boldsymbol{x}}),$
(6)

where $*$ denotes the convolution operator, $\boldsymbol{H}$ is a median
filter with window size $P\times P$, $\nabla$ is a first-order derivative used
to extract the texture features of $\tilde{\boldsymbol{x}}$.

Remark 1. The proposed degradation model offers a more compact yet expressive
approach compared to previous spatially-invariant degradation models [41, 21,
20, 15, 19], which fully utilizes the low dimensional manifold information in
the imaging process and greatly reduces the number of learned parameters. The
learnable degradation kernel dictionary ensures the expressive ability of the
proposed degradation model, and the coefficient matrix obtained through fuzzy
set theory makes full use of the physical information in the image. This
allows the proposed spatially-variant degradation model to be effectively
utilized on a single LR image without the need for extensive dataset training.

###  3.2 Probabilistic BISR Model

We construct a probabilistic BISR model based on the proposed Spatially-
Variant degradation model illustrated in Sec. 3.1. Under the MAP framework,
Eq. 1 can be modeled as a MAP estimation as follows:

$\max_{\boldsymbol{D},\boldsymbol{x}}\log
p(\boldsymbol{y}\mid\boldsymbol{D},\boldsymbol{x})+\log p(\boldsymbol{x})+\log
p(\boldsymbol{D}),$ (7)

where $p(\boldsymbol{y}\mid\boldsymbol{D},\boldsymbol{x})$ represents the
likelihood, while $p(\boldsymbol{x})$ and $p(\boldsymbol{D})$ denote image
prior and kernel prior. We elaborately design these three terms to achieve
accurate HR reconstruction without relying on a dataset.

##### Likelihood:

Unlike most existing methods that only consider the likelihood
$p(\boldsymbol{y}\mid\boldsymbol{D},\boldsymbol{x})$ in the spatial domain, in
this work we define a likelihood in both the spatial domain and the frequency
domain as:

$y\sim\mathcal{N}(\boldsymbol{y}\mid\left(\mathcal{D}\boldsymbol{x}\right)\boldsymbol{\downarrow}_{s},\sigma_{y}\boldsymbol{1})\mathcal{N}(\mathcal{F}\left(\boldsymbol{y}\right)\mid\mathcal{F}\left(\left(\mathcal{D}\boldsymbol{x}\right)\boldsymbol{\downarrow}_{s}\right),\sigma_{f}\boldsymbol{1}),$
(8)

where $\boldsymbol{1}$ represents the identity matrix, $\sigma_{y}$ and
$\sigma_{f}$ are hyper-parameters which represent the variances of Gaussian
white noise in the spatial and frequency domains, respectively. The likelihood
in the frequency domain enhances the reconstruction effectiveness of the
proposed model. An ablation study of the frequency domain likelihood can be
found in Sec. 5.

##### Kernel Prior:

Based on the aforementioned implicit constraints priors and physical priors, a
spatially-variant degradation model is plugged into Eq. 7 as the kernel prior.
Next, each element of $\boldsymbol{\Gamma}$ can be naturally defined as Dirac
distributions with the ground truth
$\hat{\boldsymbol{\Gamma}}=\\{\hat{\theta_{i}},\hat{\sigma}_{i,1},\hat{\sigma}_{i,2}\\}$,
_i.e._ ,

$\boldsymbol{\Gamma}\sim\prod_{i=1}^{h\times
w}\operatorname{Dirac}(\theta_{i}\mid\hat{\theta_{i}})\prod_{i=1}^{h\times
w}\prod_{l=1}^{2}\operatorname{Dirac}(\sigma_{i,l}\mid\hat{\sigma}_{i,l}).$
(9)

Since the ground truth ${\hat{\theta_{i}}}$ are unknowns in BISR, we adopt
Gaussian distributions with zero mean and variance $\sigma_{\gamma}$ to
represent Eq. 9 as:

$\boldsymbol{\Gamma}\sim\prod_{i=1}^{h\times w}\mathcal{N}(\theta_{i}\mid
0,\sigma_{\gamma})\prod_{i=1}^{h\times
w}\prod_{l=1}^{2}\mathcal{N}(\sigma_{i,l}\mid 0,\sigma_{\gamma}).$ (10)

Remark 2. By employing Eq. 9 and Eq. 10, our model is versatile, capable of
accommodating both blind SR with Dirac distributions using the ground truth
$\hat{\boldsymbol{\Gamma}}$ and non-blind SR with Gaussian distributions
featuring a zero mean and variance $\sigma_{\gamma}$.

##### Image Prior:

We approximately characterize the complex structural nature of natural images
by reparametrizing the latent space using a network $G$ with 3-layer encoder-
decoder structure U-net, _i.e._ ,

$\boldsymbol{x}=G\left(\boldsymbol{z};\boldsymbol{\phi}\right),$ (11)

where $\boldsymbol{z}$ denotes the latent variable, $\boldsymbol{\phi}$
denotes the deterministic weights of $G$. The joint distribution of
$\boldsymbol{\phi}$ and $\boldsymbol{z}$ is defined as:

$\displaystyle(\boldsymbol{\phi},\boldsymbol{z})\sim
p(\boldsymbol{\phi},\boldsymbol{z})=p(\boldsymbol{\phi}\mid\boldsymbol{z})p(\boldsymbol{z}).$
(12)

Considering the powerful fitting capability of CNN, conventional Laplacian
prior with scale $\sigma_{x}$ and Gaussian prior with variance $\sigma_{z}$
are to mitigate the overfitting phenomenon. These priors are defined as:

$\displaystyle p(\boldsymbol{\phi}\mid\boldsymbol{z})$
$\displaystyle=\mathcal{L}(\nabla
G\left(\boldsymbol{z};\boldsymbol{\phi}\right)\mid
0,\sigma_{x}\boldsymbol{1}),$ (13) $\displaystyle p(\boldsymbol{z})$
$\displaystyle\sim\mathcal{N}(\boldsymbol{z}\mid 0,\sigma_{z}\boldsymbol{1}).$
(14)

where $\boldsymbol{z}$ denotes the first coefficient matrix used to emphasize
texture information. Additionally, we incorporate instance normalization
layers on $G$ and skip connection structure in the frequency domain like [25]
to alleviate overfitting in a dataset-free setting.

##  4 MCEM Inference Algorithm

Algorithm 1 MCEM Inference Algorithm

0: Observed LR image $\boldsymbol{y}$, hyper-paramter settings.

0: the estimated HR image $\tilde{\boldsymbol{x}}$.

1: while not converged do

2: E-Step: Sample the latent variable $\boldsymbol{z}$ from Eq. 16

3: M-Step Update parameter $\boldsymbol{\Gamma}$ and $\boldsymbol{\phi}$ based
on Eq. 18

4: end while

5: $\tilde{\boldsymbol{x}}=\boldsymbol{G}(\boldsymbol{z};\phi)$

In this section, we utilize the MCEM inference algorithm to solve the proposed
probabilistic BISR model in Sec. 3.2. First, we remodel BISR according to the
preview equation. Then, in the E-step, we adopt Langevin dynamics (LD) as a
Monte Carlo sampler to search for the optimal distribution of $\boldsymbol{z}$
and maximize the Evidence Lower Bound (ELBO) with respect to the model
parameters in the M-step. According to Eqs. 7, 8, 9, 10, 11, 12, 13 and 14, we
re-model BISR with an additional inference of parameters:

$\displaystyle\underset{\boldsymbol{\Gamma},\boldsymbol{\phi}}{\max}$
$\displaystyle
p(\boldsymbol{\Gamma},\boldsymbol{\phi}\mid\boldsymbol{y})=\log\int
p(\boldsymbol{y}\mid\boldsymbol{\Gamma},\boldsymbol{\phi},\boldsymbol{z})p(\boldsymbol{\Gamma}\mid\boldsymbol{y})p(\boldsymbol{\phi}\mid\boldsymbol{z})p(\boldsymbol{z})\mathrm{d}\boldsymbol{z}+\text{const}.$
(15)

##### E-Step:

In the E-Step, we fix the parameters in Eq. 15 and search for the optimal
distribution of $\boldsymbol{z}$. We denote it as
$p(\boldsymbol{z}\mid\boldsymbol{y},\boldsymbol{\Gamma}_{\text{old}},\boldsymbol{\phi}_{\text{old}})$,
where $\\{\boldsymbol{\Gamma}_{\text{old}},\boldsymbol{\phi}_{\text{old}}\\}$
represents the current parameters. $\boldsymbol{z}$ can be approximately
sampled using Stochastic Gradient Langevin Dynamics (SGLD) for
$\tau=1,2,\ldots,n_{z}$.

$\boldsymbol{z}^{\tau+1}=\boldsymbol{z}^{\tau}+\alpha\frac{\partial}{\partial\boldsymbol{z}}\log
p(\boldsymbol{z}^{\tau}\mid\boldsymbol{y},\boldsymbol{\Gamma}_{old},\boldsymbol{\phi}_{old})+\sqrt{2\alpha}\boldsymbol{\zeta},$
(16)

where $\alpha$ denotes the step size satisfying Robbins-Monro condition,
$\zeta\sim\mathcal{N}(0,\boldsymbol{1})$ represents Gaussian white noise used
to prevent trapping into local modes.

The posterior of $\boldsymbol{z}$ can be calculated according to Eqs. 8, 13
and 14 as:

$\displaystyle
p(\boldsymbol{z}\mid\boldsymbol{y},\boldsymbol{\Gamma}_{\text{old}},\boldsymbol{\phi}_{\text{old}})$
$\displaystyle=\frac{1}{2{\sigma}_{y}^{2}}\left\|\boldsymbol{y}-\mathcal{D}_{\text{old}}\boldsymbol{x}_{\text{old}}\right\|_{2}^{2}+\frac{1}{2\sigma_{x}}\left\|\nabla\boldsymbol{x}_{\text{old}}\right\|$
(17)
$\displaystyle+\frac{1}{2{\sigma}_{f}^{2}}\left\|\mathcal{F}\left(\boldsymbol{y}\right)-\mathcal{F}\left(\mathcal{D}_{\text{old}}\boldsymbol{x}_{\text{old}}\right)\right\|_{2}^{2}+\frac{1}{2\sigma_{z}^{2}}\left\|\boldsymbol{z}\right\|_{2}^{2},$

where $\mathcal{D}_{\text{old}}$ denotes the degradation operator determined
by $\boldsymbol{\Gamma}_{\text{old}}$, and $\boldsymbol{x}_{\text{old}}$
denotes the HR image generated by
$G\left(\boldsymbol{z};\boldsymbol{\phi}_{\text{old}}\right)$.

##### M-Step:

The M-step aims to maximize the Evidence Lower Bound (ELBO) _w.r.t._ the model
parameters. Fixing the sampled latent variable $\boldsymbol{z}$ in the E-step,
the objective function for optimizing model parameters $\boldsymbol{\Gamma}$
and $\boldsymbol{\phi}$ can be formulated based on Eqs. 8, 10 and 13 as:

$\displaystyle\max_{\boldsymbol{\Gamma},\boldsymbol{\phi}}Q(\boldsymbol{\Gamma},\boldsymbol{\phi})$
$\displaystyle\Rightarrow\min_{\boldsymbol{\Gamma},\boldsymbol{\phi}}E(\boldsymbol{\Gamma},\boldsymbol{\phi})\
\frac{a\ln(1+\boldsymbol{W}^{down}_{N_{D}})+b}{2{\sigma_{y}}^{2}}\left\|\boldsymbol{y}-\mathcal{D}\boldsymbol{x}\right\|_{2}^{2}+\frac{1}{2{\sigma_{\gamma}}^{2}}\sum_{i=1}^{h\times
w}\left\|\theta_{i}\right\|_{2}^{2}$ (18)
$\displaystyle+\frac{1}{2{\sigma_{f}}^{2}}\left\|\mathcal{F}\left(\boldsymbol{y}\right)-\mathcal{F}\left(\mathcal{D}\boldsymbol{x}\right)\right\|_{2}^{2}+\frac{1}{2\sigma_{x}}\left\|\nabla\boldsymbol{x}\right\|+\frac{1}{2{\sigma_{\gamma}}^{2}}\sum_{i=1}^{h\times
w}\sum_{l=1}^{2}\left\|\sigma_{i,l}\right\|_{2}^{2}.$

where a and b are empirically set to 45000 and
8000.$\boldsymbol{W}^{down}_{N_{D}}$ denotes the last layer coefficient matrix
after sampling. Eq. 18 can be solved via ADAM optimizer. The whole MCEM
Inference Algorithm is illustrated in Algorithm 1.

##  5 Experiments

In the rest of this paper, we refer to our Spatially-Variant Degradation Model
for dataset-free Super-Resolution as SVDSR.

###  5.1 Experimental Settings

##### Parameter Settings:

Throughout the experiments, we set the number of atom degradation kernels
$N_{\mathcal{D}}$ as 5. For the membership functions, a median filter with a
square window size of 15 is utilized to extract the texture information from
the image, and the shape parameter of the membership function $\sigma_{g}$ is
empirically set to 0.5. For simplicity, the variance $\sigma_{y}$ and
$\sigma_{f}$ of the spatial domain and frequency domain in likelihood are set
to 1 and 2, respectively. A generation network $G$ with 3-layer encoder-
decoder structure U-net is employed to predict the HR image, and the Laplace
distribution prior with a variance $\sigma_{x}$ of 2.5 and a Gaussian
distribution prior with a variance $\sigma_{z}$ of 1 are used to alleviate the
occurrence of overfitting. Since the real degradation kernels are unknown in
the BISR task, we empirically use a Gaussian distribution with a variance
$\sigma_{\gamma}$ of 1.5 to simulate their distribution. The hyperparameters
$\alpha$ and $n_{z}$ for $\boldsymbol{z}$ sampling are set to 1.5 and 5. For
the sake of fairness in comparison, the learning rates for $\gamma$ and $\phi$
are set to $2\times 10^{-3}$ and $5\times 10^{-3}$, respectively, which are
the same as BSRDM [41]. The maximum iteration number of the EM algorithm is
capped at 5,000.

##### Comparison Methods:

To evaluate our model, we compare it against five methods, namely RCAN [44],
ZSSR [33], DoubleDIP [18], DIPFKP [21] and BSRDM [41]. RCAN is a residual
channel attention networks with residual in residual (RIR) structure.
LARPAR[19] is not used for comparison due to its degradation kernel and
downsampling methods are different compared to common dataset-free methods.
ZSSR is a zero-shot method that exploits the patch recurrence within the same
image scale, as well as across different scales in a single image. Since the
degradation models of ZSSR and RCAN are both assumed to be a bicubic
downsampler, it is unfair to compare with their methods on datasets generated
using Gaussian degradation kernels. In addition to testing the above methods
in the BISR, we also test the effect of ZSSR in the non-blind case. The ground
truth of the degradation kernel is provided by us. By incorporating the non-
blind method, different models can be better evaluated. DoubleDIP, DIPFKP, and
BSRDM are the same type of comparison methods as our method, which propose
different degradation models to better reconstruct image details. Among them,
the degradation model of BSRDM is spatially-invariant and BSRDM is known as a
state-of-the-art (SOTA) method in Dataset-free BISR. All the code used for
comparison comes from their official website.

###  5.2 Evaluation on Synthetic Data

Table 1: Quantitative comparison on various datasets. The gray results indicate unfair comparisons. (The degradation models of ZSSR and RCAN both assume bicubic downsampling, while ZSSR-NB operates in a non-blind setting with access to the ground truth of the degradation kernel.) The best results of fair method comparison are highlighted in bold. Method | Scale | Set5 | Set14 | Urban100 | Manga109 | DIV2K100
---|---|---|---|---|---|---
| Factor | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM
RCAN [44] |  $\times$2 |  26.75 / 0.81 |  24.77 / 0.70 |  22.46 / 0.66 |  23.56 / 0.78 |  26.05 / 0.74
ZSSR [33] |  $\times$2 | 26.72 / 0.81 | 24.46 / 0.70 | 22.47 / 0.66 | 23.53 / 0.78 | 26.03 / 0.74
ZSSR-NB [33] |  $\times$2 | 33.40 / 0.91 | 30.08 / 0.85 | 27.67 / 0.83 | 32.19 / 0.91 | 31.24 / 0.87
DoubleDIP [31] |  $\times$2 | 18.57 / 0.48 | 18.90 / 0.44 | 17.18 / 0.41 | 18.44 / 0.58 | 20.40 / 0.50
DIPFKP [21] |  $\times$2 | 28.76 / 0.86 | 26.38 / 0.75 | 24.60 / 0.72 | 27.75 / 0.85 | 27.19 / 0.75
BSRDM [41] |  $\times$2 | 32.76 / 0.91 | 28.65 / 0.81 | 25.46 / 0.76 | 28.49 / 0.87 | 28.32 / 0.78
Our |  $\times$2 |  33.51 / 0.92 |  29.61 / 0.83 |  26.40 / 0.79 |  29.89 / 0.89 |  29.46 / 0.82
RCAN [44] |  $\times$3 | 23.12 / 0.68 | 21.71 / 0.55 | 19.71 / 0.51 | 20.13 / 0.64 | 23.05 / 0.61
ZSSR [33] |  $\times$3 | 23.02 / 0.68 | 21.54 / 0.55 | 19.93 / 0.52 | 20.21 / 0.65 | 23.14 / 0.62
ZSSR-NB [33] |  $\times$3 | 26.88 / 0.78 | 24.19 / 0.66 | 22.43 / 0.63 | 24.05 / 0.75 | 26.11 / 0.71
DoubleDIP [31] |  $\times$3 | 17.39 / 0.42 | 18.22 / 0.41 | 16.96 / 0.40 | 18.19 / 0.56 | 20.10 / 0.50
DIPFKP [21] |  $\times$3 | 25.47 / 0.82 | 25.06 / 0.72 | 23.51 / 0.69 | 26.54 / 0.83 | 26.16 / 0.72
BSRDM [41] |  $\times$3 | 30.96 / 0.88 | 27.67 / 0.77 | 24.66 / 0.72 | 27.44 / 0.85 | 27.67 / 0.76
Our |  $\times$3 |  31.37 / 0.89 |  28.14 / 0.79 |  25.26 / 0.75 |  28.50 / 0.87 |  28.51 / 0.79
RCAN [44] |  $\times$4 | 20.59 / 0.57 | 19.91 / 0.47 | 17.82 / 0.41 | 17.90 / 0.55 | 21.18 / 0.54
ZSSR [33] |  $\times$4 | 20.63 / 0.57 | 20.02 / 0.48 | 18.39 / 0.43 | 18.29 / 0.56 | 21.45 / 0.55
ZSSR-NB [33] |  $\times$4 | 23.24 / 0.62 | 23.19 / 0.60 | 21.89 / 0.59 | 23.60 / 0.72 | 25.81 / 0.69
DoubleDIP [31] |  $\times$4 | 17.53 / 0.43 | 18.24 / 0.41 | 17.25 / 0.40 | 17.49 / 0.53 | 20.05 / 0.50
DIPFKP [21] |  $\times$4 | 24.32 / 0.81 | 23.93 / 0.68 | 22.56 / 0.66 | 24.86 / 0.79 | 25.16 / 0.70
BSRDM [41] |  $\times$4 | 29.02 / 0.85 | 26.63 / 0.73 | 23.47 / 0.68 | 25.77 / 0.81 | 26.74 / 0.72
Our |  $\times$4 |  29.29 / 0.85 |  26.81 / 0.73 |  23.90 / 0.70 |  26.44 / 0.83 |  27.34 / 0.75
|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
Bilinear | ZSSR-NB | DoubleDIP | DIPFKP | BSRDM | Our | GT
Figure 2: Visualization of the synthetic images. Each row, from top to bottom,
contains super resolved images from Urban100, DIV2K100 and Manga109, under the
scale factor of 2, 3 and 4, respectively.

We adopt five widely-used datasets to synthesize degradation images for
evaluation, including Set5 [3], Set14 [42], Urban100 [9], Manga109 [26] and
DIV2K100 [1]. The LR image is first randomly blurred by six degradation
kernels, and then downsampled with scale factors 2, 3, and 4 respectively. In
addition, 1% Gaussian noise is also added to the LR image to better simulate
degradation images in real scenes. The six degradation kernels used are
sourced from BSRDM, which includes two isotropic Gaussian kernels with
different widths and four isotropic Gaussian kernels. To address GPU memory
limitations, all images larger than 1024 $\times$ 1024 are cropped below the
center. The results are evaluated by the peak signal-to-noise ratio (PSNR,
measured in dB) and structural similarity index (SSIM) [39]. PSNR and SSIM are
evaluated on the Y channel of the transformed YCbCr color space using MATLAB.
The PSNR and SSIM of different methods under different datasets are shown in
Tab. 1.

---
(a)

---
(b)

---
(c)

Figure 3: (a) Visualization of the coefficient matrices and the corresponding
learned atom degradation kernels of the image Baby under the scale factor of 2
in Set 5. (b) Visualization of the synthetic images with the spatially-
invariant degradation model (top) and spatially-variant degradation model
(bottom). (c) Convergence curves.

It is evident that our proposed method SVDSR outperforms the SOTA method BSRDM
in 2$\times$ SR,, and also has certain advantages in 4$\times$SR. This is
because compared with the spatially-invariant degradation model, the proposed
spatially-variant degradation model fully utilizes the physical information in
the image, but at 4$\times$SR, due to the excessive loss of texture
information in the image, the superiority of the SVDSR decreases to a certain
extent. The performance of Data-driven SR methods RCAN and ZSSR significantly
degrades when the degradation kernel deviates from their bicubic degradation
assumptions. In addition, SVDSR remains competitive with the non-blind methods
ZSSR-NB, which strongly demonstrates the effectiveness of the proposed
spatially-variant degradation model in estimating the degradation kernel. The
superior performance on the natural image dataset DIV2K100 and the cartoon
image dataset Manga109 shows the good adaptability of SVDSR on different
images.

Fig. 2 compares the visual effects with varying scale factors. It is obvious
that our method can achieve favorable results in photos under different
scenarios, especially in areas with complex details. This is attributed to the
substantial differences in degradation kernel information between flat areas
and densely textured areas in the image. Our proposed SVDSR leverages each
pixel in the image to estimate its own degradation kernel, utilizing the
texture information of the region in which the pixel is located. This approach
enhances our ability to model image degradation, resulting in superior
performance in BISR. Fig. 3(a) shows the derived coefficient matrices and the
corresponding learned atom degradation kernels exported from the baby image.
It can be seen that the derived coefficient matrices well reflect the texture
information of the image and the detail reconstruction effect under the
spatially-variant model in Fig. 3(b) is remarkable. Moreover, there are
significant differences in the shape of atom degradation kernels in different
regions, which is exactly the same phenomenon observed in LARPAR[19]. Fig.
3(c) demonstrates the convergence on different datasets.

###  5.3 Evaluation on Real Data

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
Bilinear | RCAN | ZSSR | DoubleDIP | DIPFKP | BSRDM | Our
Figure 4: Visualization of the RealSRSet under scale factor of 2(top) and
4(bottom).

To further demonstrate the applicability of our method to real images, we
evaluated data-driven and dataset-free methods using a small dataset called
RealSRSet [43], which consists of 20 real images. As ground truth for real-
world images is unavailable, we performed visual comparisons only on the HR
images generated by different methods. A subset of estimated HR images is
depicted in Fig. 4. It is evident that our proposed method outperforms other
dataset-free methods significantly in terms of detail restoration and is
comparable to data-driven methods. Given that the SOTA method BSRDM also
incorporates denoising during BISR, detailed areas such as the tiger’s hair
exhibit varying degrees of blurring in BISR for real images.

###  5.4 Comparison on Model Size and Runtime

We further compare different models in terms of running time and memory. The
comparison results are shown in Tab. 3. All tests were conducted on the
GeForce RTX 3090 GPU. We test the running time of super-resolution with scale
factors of 2, 3, and 4 for 256 $\times$ 256 images. For the sake of fairness,
only four methods based on traditional degradation models are considered,
namely ZSSR, DoubleDIP, DIPFKP, and BSRDM. It can be seen that even though our
proposed SVDSR estimates its own degradation kernel for each pixel, its time
and memory consumption do not exceed BSRDM significantly. This suggests that
our model has greater potential for practical utilization than these methods
in real-world scenarios.

###  5.5 Ablation Studies and Analysis

Ablation Studies  To assess the efficiency of each module in our proposed
method, we construct three baselines on Set 14 under the scale factor of 2 to
analyze the effectiveness of the innovative likelihood and image prior in our
proposed SVDSR.

  * •

Case1: Likelihood only contains the spatial domain, and the frequency domain
term is removed.

  * •

Case2: The results of the encoder in the generation network is directly
concatenated with the input of the corresponding decoder, instead of using
FFT(Fast Fourier Transform) and IFFT(Inverse Fast Fourier Transform).

  * •

Case3: Instance layer in the network is removed to verify the Instance layer’s
alleviating effect on the overfitting phenomenon of the generation network.

As shown in Tab. 3, when the skip structure of the generation net and the
likelihood function in the frequency domain are removed, the PSNR index of the
data results is reduced to some extent. After removing the instance layer used
to alleviate the overfitting phenomenon of the generation net, the results of
our model are significantly diminished. This study demonstrates that our
proposed novel likelihood function, and image prior all contribute to
improving the performance of our method and are applicable to other network
architectures.

Analysis of The Atom Kernel Dictionary We investigated the impact of the
number of atom degradation kernels on our proposed spatially-variant
degradation model. We found that the optimal number of atom degradation kernel
varies for different images, but it is usually a single digit. This indicates
that the 72 pre-defined atom kernel used in the previous method[19, 45] are
redundant. Due to the good consistency of image characteristics in the set5,
the experiment was conducted on set 5 under the scale factor of 2. Tab. 5
shows the effect of our model with different number $N_{\mathcal{D}}$ of atom
degradation kernels and non-blind setting. When $N_{\mathcal{D}}$=1, our fuzzy
degradation model is equivalent to the spatially-invariant degradation model.
The effectiveness of our model improves initially as $N_{\mathcal{D}}$
increases. However, as $N_{\mathcal{D}}$ exceeds 5, the performance of the
proposed model starts to decline. Tab. 5 demonstrates the impact of
hyperparameters. We follow BSRDM for hyperparameter settings for fairness. We
can even achieve better results in other parameter settings, _e.g._ ,
$\sigma_{f}$ = 20, $\sigma_{x}$=25.

Table 2: Comparison of different methods on model size, runtime and iterations
for 2$\times$ BISR..

Methods | ZSSR | DoubleDIP | DIPFKP | BSRDM | SVDSR
---|---|---|---|---|---
Time | 63s | 220s | 68s | 30s | 33s
Parameters | 225k | 2396k | 2396k | 762k | 850k
Iterations | 3000 | 2500 | 1000 | 5000 | 5000

Table 3: PSNR of different case.

Case 1 | 29.61
---|---
Case 2 | 29.52
Case 3 | 29.28

Table 4: The impact of the number of atom blur kernels(PSNR).

Number |  $N_{\mathcal{D}}$=1 |  $N_{\mathcal{D}}$=3 |  $N_{\mathcal{D}}$=5 |  $N_{\mathcal{D}}$=7 |  $N_{\mathcal{D}}$=9 | NB
---|---|---|---|---|---|---
Set5 | 33.19 | 33.29 | 33.51 | 33.43 | 33.27 | 34.44
Set14 | 29.13 | 29.40 | 29.61 | 29.58 | 29.59 | 30.32
Urban100 | 25.90 | 26.11 | 26.40 | 26.21 | 26.20 | 27.14
Manga109 | 29.23 | 29.53 | 29.89 | 29.67 | 29.74 | 31.13
DIV2K100 | 29.05 | 29.30 | 29.46 | 29.52 | 29.54 | 30.06

Table 5: The impact of hyperparameter.

$\sigma_{x}$ |  | $\sigma_{y}$ |
---|---|---|---
0.25 | 27.06 | 0.1 | 29.56
25 | 29.74 | 10 | 28.30
$\sigma_{\gamma}$ |  | $\sigma_{f}$ |
0.15 | 29.53 | 0.2 | 29.39
15 | 29.60 | 20 | 29.63

##  6 Conclusion and Limitations

This paper introduces a fuzzy degradation model to deal with the ambiguity of
dataset-free Blind Image Super-Resolution (BISR) solutions in different
regions. We present the degradation kernel of each pixel as a linear
combination of a learnable dictionary consisting of a small number of fuzzy
atom kernels. The coefficients of these atom degradation kernels are derived
using membership functions, allowing us to capture the uncertainty and
variability in the degradation process. We propose a Probabilistic BISR
framework with customized likelihood and prior terms. The Monte Carlo EM
algorithm, which iteratively estimates the degradation kernels of each pixel
while considering the probabilistic nature of the problem is employed.
Extensive experiments show the superiority of our method. Color distortion is
a challenge faced by most SR methods due to strong noise in several images.
Similarly, our method also encounters this problem, and our future goal is to
achieve better denoising while preserving more details.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China
(Grant No. 62101191), Shanghai Natural Science Foundation (Grant No.
21ZR1420800), and the Science and Technology Commission of Shanghai
Municipality (Grant No. 22DZ2229004).

## References

  * [1] Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 126–135 (2017)
  * [2] Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems 32 (2019)
  * [3] Bevilacqua, M., Roumy, A., Guillemot, C., Morel, A.: Low-complexity single image super-resolution based on nonnegative neighbor embedding. In: British Machine Vision Conference (2012)
  * [4] Chen, X., Zhang, J., Xu, C., Wang, Y., Wang, C., Liu, Y.: Better" cmos" produces clearer images: Learning space-variant blur estimation for blind image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1651–1661 (2023)
  * [5] Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE transactions on Image Processing 22(4), 1620–1630 (2012)
  * [6] Fritsche, M., Gu, S., Timofte, R.: Frequency separation for real-world super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). pp. 3599–3608. IEEE (2019)
  * [7] He, W., Yao, Q., Li, C., Yokoya, N., Zhao, Q., Zhang, H., Zhang, L.: Non-local meets global: An iterative paradigm for hyperspectral image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(4), 2089–2107 (2020)
  * [8] He, X., Mo, Z., Wang, P., Liu, Y., Yang, M., Cheng, J.: Ode-inspired network design for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1732–1741 (2019)
  * [9] Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5197–5206 (2015)
  * [10] Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th acm international conference on multimedia. pp. 2024–2032 (2019)
  * [11] Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., Huang, F.: Real-world super-resolution via kernel estimation and noise injection. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 466–467 (2020)
  * [12] Keilmann, A., Godehardt, M., Moghiseh, A., Redenbach, C., Schladitz, K.: Improved anisotropic gaussian filters. arXiv preprint arXiv:2303.13278 (2023)
  * [13] Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1646–1654 (2016)
  * [14] Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE transactions on pattern analysis and machine intelligence 32(6), 1127–1133 (2010)
  * [15] Kim, S.Y., Sim, H., Kim, M.: Koalanet: Blind super-resolution using kernel-oriented adaptive local adjustment. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10611–10620 (2021)
  * [16] Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-laplacian priors. Advances in neural information processing systems 22 (2009)
  * [17] Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE transactions on pattern analysis and machine intelligence 41(11), 2599–2613 (2018)
  * [18] Li, J., Wang, W., Nan, Y., Ji, H.: Self-supervised blind motion deblurring with deep expectation maximization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13986–13996 (2023)
  * [19] Li, W., Zhou, K., Qi, L., Jiang, N., Lu, J., Jia, J.: Lapar: Linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond. Advances in Neural Information Processing Systems 33, 20343–20355 (2020)
  * [20] Liang, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Mutual affine network for spatially variant kernel estimation in blind image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4096–4105 (2021)
  * [21] Liang, J., Zhang, K., Gu, S., Van Gool, L., Timofte, R.: Flow-based kernel prior with application to blind super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10601–10610 (2021)
  * [22] Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2359–2368 (2020)
  * [23] Lu, Z., Li, J., Liu, H., Huang, C., Zhang, L., Zeng, T.: Transformer for single image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 457–466 (2022)
  * [24] Maeda, S.: Unpaired image super-resolution using pseudo-supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 291–300 (2020)
  * [25] Mao, X., Liu, Y., Liu, F., Li, Q., Shen, W., Wang, Y.: Intriguing findings of frequency selection for image deblurring. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1905–1913 (2023)
  * [26] Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76, 21811–21838 (2017)
  * [27] McLachlan, G.J., Lee, S.X., Rathnayake, S.I.: Finite mixture models. Annual review of statistics and its application 6, 355–378 (2019)
  * [28] Mei, Y., Fan, Y., Zhou, Y., Huang, L., Huang, T.S., Shi, H.: Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5690–5699 (2020)
  * [29] Mou, C., Wu, Y., Wang, X., Dong, C., Zhang, J., Shan, Y.: Metric learning based interactive modulation for real-world super-resolution. In: European Conference on Computer Vision. pp. 723–740. Springer (2022)
  * [30] Nagy, J.G., O’Leary, D.P.: Restoring images degraded by spatially variant blur. SIAM Journal on Scientific Computing 19(4), 1063–1082 (1998)
  * [31] Ren, D., Zhang, K., Wang, Q., Hu, Q., Zuo, W.: Neural blind deconvolution using deep priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3341–3350 (2020)
  * [32] Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena 60(1-4), 259–268 (1992)
  * [33] Shocher, A., Cohen, N., Irani, M.: “zero-shot” super-resolution using deep internal learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3118–3126 (2018)
  * [34] Sun, J., Xu, Z., Shum, H.Y.: Image super-resolution using gradient profile prior. In: 2008 IEEE conference on computer vision and pattern recognition. pp. 1–8. IEEE (2008)
  * [35] Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: Proceedings of the IEEE international conference on computer vision. pp. 4539–4547 (2017)
  * [36] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 9446–9454 (2018)
  * [37] Wang, X., Xie, L., Dong, C., Shan, Y.: Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1905–1914 (2021)
  * [38] Wang, X., Yu, K., Dong, C., Loy, C.C.: Recovering realistic texture in image super-resolution by deep spatial feature transform. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 606–615 (2018)
  * [39] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)
  * [40] Xie, J., Gao, R., Zheng, Z., Zhu, S.C., Wu, Y.N.: Learning dynamic generator model by alternating back-propagation through time. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 5498–5507 (2019)
  * [41] Yue, Z., Zhao, Q., Xie, J., Zhang, L., Meng, D., Wong, K.Y.K.: Blind image super-resolution with elaborate degradation modeling on noise and kernel. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2128–2138 (2022)
  * [42] Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7. pp. 711–730. Springer (2012)
  * [43] Zhang, K., Liang, J., Van Gool, L., Timofte, R.: Designing a practical degradation model for deep blind image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4791–4800 (2021)
  * [44] Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV). pp. 286–301 (2018)
  * [45] Zhou, H., Zhu, X., Zhu, J., Han, Z., Zhang, S.X., Qin, J., Yin, X.C.: Learning correction filter via degradation-adaptive regression for blind single image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12365–12375 (2023)